NLTK - How to use NER

Question

How can I invoke NER from NLTK to get all of its results on the first two hundred characters of each of all the TXT files located in the same directory? When I try this code: ``` for filename in os.listdir(ebooksFolder): fname, fextension = os.path.splitext(filename) if (fextension == '.txt'): newName = 'ner_' + filename file = open(ebooksFolder + '\' + filename) rawFile = file.read() partToUse = rawFile[:50] segmentedSentences = nltk.sent_tokenize(partToUse) tokenizedSentences = [nltk.word_tokenize(sent) for sent in segmentedSentences] posTaggedSentences = [nltk.pos_tag(sent) for sent in tokenizedSentences] nerResult = nltk.ne_chunk(posTaggedSentences) pathToCopy = 'C:\Users\Felipe\Desktop\books_txt\' nameToSave = os.path.join(pathToCopy, newName + '.txt') newFile = open(nameToSave, 'w') newFile.write(nerResult) newFile.close() ``` I get these errors: ``` Traceback (most recent call last): File "", line 11, in nerResult = nltk.ne_chunk(posTaggedSentences) F…

Accepted Answer

Having tokenized the text to sentences and then to POS tags, you need to iterate over the list of tagged sentences like so:

```
nerResult = [nltk.ne_chunk(pts) for pts in posTaggedSentences]
```

instead of like so:

```
nerResult = nltk.ne_chunk(posTaggedSentences)
```

NLTK - How to use NER

Solution

Problem