pyPdf ignores newlines in PDF file

pdf, pypdf, python, string, unicode

Solution

I don't know much about PDF encoding, but I think you can solve your particular problem by modifying `pdf.py`. In the `PageObject.extractText` method, you see what's going on:

def extractText(self):
    [...]
    for operands,operator in content.operations:
        if operator == "Tj":
            _text = operands[0]
            if isinstance(_text, TextStringObject):
                text += _text
        elif operator == "T*":
            text += "\n"
        elif operator == "'":
            text += "\n"
            _text = operands[0]
            if isinstance(_text, TextStringObject):
                text += operands[0]
        elif operator == '"':
            _text = operands[2]
            if isinstance(_text, TextStringObject):
                text += "\n"
                text += _text
        elif operator == "TJ":
            for i in operands[0]:
                if isinstance(i, TextStringObject):
                    text += i

If the operator is `Tj` or `TJ` (it's Tj in your example PDF) then the text is simply appended and no newline is added. Now you wouldn't necessarily want to add a newline, at least if I'm reading the PDF reference right: `Tj/TJ` are simply the single and multiple show-string operators, and the existence of a separator of some kind isn't mandatory.

Anyway, if you modify this code to be something like

def extractText(self, Tj_sep="", TJ_sep=""):

[...]

        if operator == "Tj":
            _text = operands[0]
            if isinstance(_text, TextStringObject):
                text += Tj_sep
                text += _text

[...]

        elif operator == "TJ":
            for i in operands[0]:
                if isinstance(i, TextStringObject):
                    text += TJ_sep
                    text += i

then the default behaviour should be the same:

In [1]: pdf.getPage(1).extractText()[1120:1250]
Out[1]: u'ing an individual which, because of name, identifyingnumber, mark or description can be readily associated with a particular indiv'

but you can change it when you want to:

In [2]: pdf.getPage(1).extractText(Tj_sep=" ")[1120:1250]
Out[2]: u'ta" means any information concerning an individual which, because of name, identifying number, mark or description can be readily '

or

In [3]: pdf.getPage(1).extractText(Tj_sep="\n")[1120:1250]
Out[3]: u'ta" means any information concerning an individual which, because of name, identifying\nnumber, mark or description can be readily '

Alternatively, you could simply add the separators yourself by modifying the operands themselves in-place, but that could break something else (methods like `get_original_bytes` make me nervous).

Finally, you don't have to edit `pdf.py` itself if you don't want to: you could simply pull out this method into a function.

Problem

I'm trying to extract each page of a PDF as a string: ``` import pyPdf pages = [] pdf = pyPdf.PdfFileReader(file('g-reg-101.pdf', 'rb')) for i in range(0, pdf.getNumPages()): this_page = pdf.getPage(i).extractText() + "\n" this_page = " ".join(this_page.replace(u"\xa0", " ").strip().split()) pages.append(this_page.encode("ascii", "xmlcharrefreplace")) for page in pages: print '*' * 80 print page ``` But this script ignore newline characters, leaving me with messy strings like `information concerning an individual which, because of name, identifyingnumber, mark or description` (i.e, this should read `identifying number`, not `identifyingumber`). Here's an example of the type of PDF I'm trying to parse.

Original source