Classify a noun into abstract or concrete using NLTK or similar

nlp, nltk, python

Solution

I would suggest training a classifier using pretrained word vectors.

You need two libraries: `spacy` for tokenizing text and extracting word vectors, and `scikit-learn` for machine learning:

import spacy
from sklearn.linear_model import LogisticRegression
import numpy as np
nlp = spacy.load("en_core_web_md")

Distinguishing concrete and abstract nouns is a simple task, so you can train a model with very few examples:

classes = ['concrete', 'abstract']
# todo: add more examples
train_set = [
    ['apple', 'owl', 'house'],
    ['agony', 'knowledge', 'process'],
]
X = np.stack([list(nlp(w))[0].vector for part in train_set for w in part])
y = [label for label, part in enumerate(train_set) for _ in part]
classifier = LogisticRegression(C=0.1, class_weight='balanced').fit(X, y)

When you have a trained model, you can apply it to any text:

for token in nlp("Have a seat in that chair with comfort and drink some juice to soothe your thirst."):
    if token.pos_ == 'NOUN':
        print(token, classes[classifier.predict([token.vector])[0]])

The result looks satisfying:

# seat concrete
# chair concrete
# comfort abstract
# juice concrete
# thirst abstract

You can improve the model by applying it to different nouns, spotting the errors and adding them to the training set under the correct label.

Problem

Original source