Does NLTK parts of speech tagger use global information or just the word that is being tagged?
nlp, nltk, python
Solution
The `pos_tag` function makes a call to load the `pickle` at `_POS_TAGGER`. This is a maximum entropy tagger probably trained on Penn Treebank POS annotated text. The information that a MaxEnt tagger uses to determine the part of speech will be based on the feature set used in training. That means, it could technically only use features of individual words but this is unlikely as the tagger would be inaccurate (and it would not take full advantage of using machine learning to generate a tagger). Consider an example given in Chapter 5 in Natural Language Processing with Python,
>>> text = nltk.word_tokenize("They refuse to permit us to obtain the refuse permit")
>>> nltk.pos_tag(text)
[('They', 'PRP'), ('refuse', 'VBP'), ('to', 'TO'), ('permit', 'VB'), ('us', 'PRP'),
('to', 'TO'), ('obtain', 'VB'), ('the', 'DT'), ('refuse', 'NN'), ('permit', 'NN')]
Since "refuse" and "permit" are each given different tags depending on the context, we can say for certain that it does use features of previous words (like their POS tags).
Problem
I am currently doing some parts of speech tagging using NLTK's "nltk.pos_tag". I was wondering does NLTK's tagger use information beyond the word that is currently being tagged to determine the POS of the word? If not does NLTK have a tagger that would do this? Thanks in advance for any info!