Custom Solr TokenFilter lemmatizer

java, solr

Solution

Next, you want to `replace` or `append` the current token `termAtt` with your word.

Sample replace semantics

termAtt.setEmpty();
termAtt.copyBuffer(word.toCharArray(), 0, word.length());

Sample semantics of adding new tokens

For each token you want to add, the `CharTermAttribute` attribute must be set and the `incrementToken` routine to return true.

private List<String> extraTokens = ...
public boolean incrementToken() { 
  if (input.incrementToken()){ 
    // ... 
    return true; 
  } else if (!extraTokens.isEmtpy()) { 
    // set the added token and return true
    termAtt.setTerm(extraTokens.remove(0)); 
    return true; 
  } 
  return false; 
}

Problem

I'm trying to write a simple Solr lemmatizer for use in a field type, but I can't seem to find any information about writing a TokenFilter so I'm kind of lost. Here is the code I have so far. ``` import java.io.IOException; import java.util.List; import org.apache.lucene.analysis.TokenFilter; import org.apache.lucene.analysis.TokenStream; import org.apache.lucene.analysis.tokenattributes.CharTermAttribute; import org.apache.lucene.analysis.tokenattributes.PositionIncrementAttribute; import org.slf4j.Logger; import org.slf4j.LoggerFactory; class FooFilter extends TokenFilter { private static final Logger log = LoggerFactory.getLogger(FooFilter.class); private final CharTermAttribute termAtt = addAttribute(CharTermAttribute.class); private final PositionIncrementAttribute posAtt = addAttribute(PositionIncrementAttribute.class); public FooFilter(TokenStream input) { super(input); } @Override public boolean incrementToken() throws IOException { if (!input.incrementToken()) { return false; } char termBuffer[] = termAtt.buffer(); List<String> allForms = Lemmatize.getAllForms(new String(termBuffer)); if (allForms.size() > 0) { for (String word : allForms) { // Now what? } } return true; } } ```

Original source