Parsing namespaces with clang: AST differences in when including a header in another source file or parsing it directly
c++, clang, parsing, python
Solution
Since you aren't explicitly specifying a language, Clang determines the language from the file extension, resulting in `"example.h"` being parsed as C, not C++. Thus the file is largely ill-formed, and the indexer tries to recover as well as it can. `namespace Geom` is being treated as a variable declaration for `Geom` with an unknown type `namespace`, and the following unexpected `{` ... `}` block is skipped.
Try:
tu = index.parse(sys.argv[1], args=['-x', 'c++'])
Problem
Sorry for the verbose question, but I can't see any other way to make it clear. I am writing a tool to transform C++ header files to SWIG interface files as a starter for further fine-tuning. In the process of doing this, I've noticed some strange behavior by clang (v3.0). If I parse the header file I get a significantly different AST than if I parse a source file that includes the header. For purposes of illustration, here are some sample source files: Source file: ``` // example.cpp: Test case for nsbug.py // #include "example.h" ``` Header: ``` // example.h: Test case for nsbug.py // namespace Geom { struct Location { double x, y; }; class Shape { public: Shape(); void set_location(const Location &where) { m_pos = where; }; const Location &get_location() const // Draw it... virtual void draw() const = 0; protected: Location m_pos; }; class Circle : public Shape { Circle(); virtual void draw() const; }; } // namespace Geom ``` I've used the following Python code to parse it and dump the AST: ``` # Usage: python nsbug.py <file> import sys import clang.cindex def indent(level): """ Indentation string for pretty-printing """ return ' '*level def output_cursor(cursor, level): """ Low level cursor output """ spelling = '' displayname = '' if cursor.spelling: spelling = cursor.spelling if cursor.displayname: displayname = cursor.displayname kind = cursor.kind; print indent(level) + spelling, '<' + str(kind) + '>' print indent(level+1) + '"' + displayname + '"' def output_cursor_and_children(cursor, level=0): """ Output this cursor and its children with minimal formatting. """ output_cursor(cursor, level) if cursor.kind.is_reference(): print indent(level) + 'reference to:' output_cursor(clang.cindex.Cursor_ref(cursor), level+1) # Recurse for children of this cursor has_children = False; for c in cursor.get_children(): if not has_children: print indent(level) + '{' has_children = True output_cursor_and_children(c, level+1) if has_children: print indent(level) + '}' index = clang.cindex.Index.create() tu = index.parse(sys.argv[1], options=1) output_cursor_and_children(tu.cursor) ``` When I run this on example.cpp I get (correctly I think): ``` <CursorKind.TRANSLATION_UNIT> "example.cpp" { (Deleted lots of clang-generated declarations such as __VERSION__) Geom <CursorKind.NAMESPACE> "Geom" { Location <CursorKind.STRUCT_DECL> "Location" { x <CursorKind.FIELD_DECL> "x" y <CursorKind.FIELD_DECL> "y" } Shape <CursorKind.CLASS_DECL> "Shape" { <CursorKind.CXX_ACCESS_SPEC_DECL> "" <CursorKind.CXX_ACCESS_SPEC_DECL> "" Shape <CursorKind.CONSTRUCTOR> "Shape()" set_location <CursorKind.CXX_METHOD> "set_location(const Geom::Location &)" { where <CursorKind.PARM_DECL> "where" { <CursorKind.TYPE_REF> "struct Geom::Location" reference to: Location <CursorKind.STRUCT_DECL> "Location" } <CursorKind.COMPOUND_STMT> "" { <CursorKind.CALL_EXPR> "operator=" { <CursorKind.MEMBER_REF_EXPR> "m_pos" <CursorKind.UNEXPOSED_EXPR> "operator=" { <CursorKind.DECL_REF_EXPR> "operator=" } <CursorKind.DECL_REF_EXPR> "where" } } } get_location <CursorKind.CXX_METHOD> "get_location()" { <CursorKind.TYPE_REF> "struct Geom::Location" reference to: Location <CursorKind.STRUCT_DECL> "Location" } <CursorKind.CXX_ACCESS_SPEC_DECL> "" <CursorKind.CXX_ACCESS_SPEC_DECL> "" m_pos <CursorKind.FIELD_DECL> "m_pos" { <CursorKind.TYPE_REF> "struct Geom::Location" reference to: Location <CursorKind.STRUCT_DECL> "Location" } } Circle <CursorKind.CLASS_DECL> "Circle" { <CursorKind.CXX_BASE_SPECIFIER> "class Geom::Shape" reference to: Shape <CursorKind.CLASS_DECL> "Shape" { <CursorKind.TYPE_REF> "class Geom::Shape" reference to: Shape <CursorKind.CLASS_DECL> "Shape" } Circle <CursorKind.CONSTRUCTOR> "Circle()" draw <CursorKind.CXX_METHOD> "draw()" } } } ``` But when I try it on the header file along with `python nsbug.py example.py` I only get: ``` <CursorKind.TRANSLATION_UNIT> "example.h" { (deleted lots of clang-generated definitions such as __VERSION__) Geom <CursorKind.VAR_DECL> "Geom" } ``` Why does is the `Geom` name space in the AST as a VAR_DECL? I would have expected no difference, except in the preprocessor cursors. The work-around is obvious--just make a temp file in memory that includes the header--but that isn't very satisfying. Can someone enlighten me?