Adding output file to Python extension

distutils, pip, python, python-extensions, setuptools

Solution

In order to have distutils install a Python package, you need to pass `packages=['foo']`, and if you’re putting it somewhere that is not the root level of the project (I mean the foo directory next to the setup.py script), like you seem to do here, you also have to pass `package_dir={'foo': 'Source'}` or to use a simpler layout. If your setup.py script contains this `packages` parameter then the build command will invoke the build_py command to move the Python source files (and directories) to the build directory, which will be copied later by the install command.

The problem here is that your `foo/__init__.py` file is built by the build_ext command, which runs after build_py. You need to override that with a custom build command:

class MyBuild(build):
  sub_commands = [('build_clib', build.has_c_libraries),
                  ('build_ext', build.has_ext_modules),
                  ('build_py', build.has_pure_modules),
                  ('build_scripts', build.has_scripts),
                 ]

setup(..., cmdclass={'build': MyBuild, 'build_ext': MyBuildExt})

The elements in the sub_commands attribute are tuples of (command name, function to call to decide whether to run the command or not); this is documented in the source code but I don’t remember if it’s explained in the doc. In the standard build class build_py precedes build_clib. I may change this in the next version of Python 2.7, as it was reported that it interacted badly with 2to3 conversion.

Problem

I've defined a custom `build_ext` to build a funky extension that I'm trying to make pip-friendly. The following is a trimmed version of what I'm doing. ``` foo_ext = Extension( name='foo/_foo', sources=foo_sources, ) class MyBuildExt(build_ext): def build_extension(self, ext): # This standalone script builds the __init__.py file # and some .h files for the extension check_call(['python', 'build_init_file.py']) # Now that we've created all the init and .h code # build the C/C++ extension using setuptools/distutils build_ext.build_extension(self, ext) # Include the generated __init__.py in the build directory # which is something like `build/lib.linux-x86/foo/`. # How can I get setuptools/distutils to install the # generated file automatically?! generated_file = 'Source/foo/__init__.py' output_path = '/'.join(self.get_outputs()[0].split('/')[:-1]) self.move_file(generated_file, output_path) setup( ..., ext_modules = [foo_ext], cmdclass={'build_ext' : MyBuildExt}, ) ``` After packaging this module up and installing it with pip, I have a module `foo` in my virtualenv's site-packages directory. The directory structure looks like the following. ``` foo/ foo/__init__.py foo/_foo.so ``` The `egg-info/SOURCES.txt` file does not include the `__init__.py` file that I manually created/moved. When I do a `pip uninstall foo` the command leaves `foo/__init__.py` in my virtualenv's site-packages. I would like pip to delete the entire package. How can I add the generated `__init__.py` file that I manually move into the build directory to the list of output files that are installed? I realize this is disgusting and hacky, so I welcome disgusting and hacky answers! Attempts: - Added `packages=['foo']` -- When I do, pip doesn't build extension. Also tried adjusting file path/namespace versions of the package name -- no difference.

Original source