Sphinx Pygments lexer filter extension?

pygments, python-sphinx

Solution

The reason is that the `NameHighlighFilter` only converts tokens that the lexer categorizes as `Token.Name`, but the `CommonLispLexer` categorizes almost everything as `Name.Variable`. This is the filter function of the `NameHighlightFilter`, from the Pygments source code:

def filter(self, lexer, stream):
    for ttype, value in stream:
        if ttype is Name and value in self.names:
            yield self.tokentype, value
        else:
            yield ttype, value

My only workaround was to write my own filter. This function gave me the look I wanted.

def filter(self, lexer, stream):
    define = False
    for ttype, value in stream:
        if value in self.tl_toplevel_forms:
            ttype = Name.Builtin
            define = True
        elif define and ttype == Name.Variable:
            define = False
            ttype = Name.Function
        elif value in self.tl_special_forms:
            ttype = Name.Variable
        # the Common Lisp lexer highlights everything else as
        # variables, which isn't the look I want.  Instead
        # highlight all non-special things as text.
        elif ttype == Name.Variable:
            ttype = Name.Text
        yield ttype, value

As a note to Pygments developers, perhaps the `NameHighlightFilter` could take an optional argument representing the token type(s) to be converted (currently it only takes the output token type).

Problem

I have a Lisp-like language I would like to highlight using Pygments in Sphinx code snippet documentation. My approach is to extend the existing CommonLispLexer to add the built-in names using a NameHighlightFilter. However, it is not working, so I must be missing something obvious. I have added the following to my conf.py: ``` def setup(app): from sphinx.highlighting import lexers from pygments.lexers import CommonLispLexer from pygments.token import Name from pygments.filters import NameHighlightFilter tl_lexer = CommonLispLexer() tl_lexer.add_filter(NameHighlightFilter( names=['define-function', 'define-macro', 'define-variable', 'define-constant'], tokentype=Name.Builtin, )) app.add_lexer('tl', tl_lexer) highlight_language = 'tl' ``` But the NameHighlightFilter has no effect. Code blocks are highlighted as if they were Lisp, but my new builtin names have no special highlighting.

Original source