How to organize Python modules for PyPI to support 2.x and 3.x

python, python-2.x, python-3.x, software-distribution

Solution

I found that `setup.py` for `httplib2` seems to have an elegant way to support Python 2.x and 3.x. So I decided to copy that method.

The task is to craft a single `setup.py` for the package distribution that works with all the supported Python distributions. Then with the same `setup.py`, you can do:

python2 setup.py install

as well as

python3 setup.py install

It should be possible to keep `setup.py` simple enough to be parsed with all the supported Python distributions. I've successfully done so with a package `cobs` that supports 2.4 through 2.6 as well as 3.1. That package includes pure Python code (separate code for Python 2.x and 3.x) and C extensions, written separately for 2.x and 3.x.

To do it:

1) I put the Python 2.x code into a `python2` subdirectory, and Python 3.x code in a `python3` subdirectory.

2) I put the C extension code for 2.x and 3.x in a `src` directory under `python2` and `python3`.

So, the directory structure is:

root
  |
  +--python2
  |     |
  |     +--src
  |
  +--python3
  |     |
  |     +--src
  |
  +--setup.py
  +--MANIFEST.in

3) In the `setup.py`, I had these lines near the top:

if sys.version_info[0] == 2:
    base_dir = 'python2'
elif sys.version_info[0] == 3:
    base_dir = 'python3'

4) In the call to `setup`, I specified the packages as normal:

setup(
    ...
    packages=[ 'cobs', 'cobs.cobs', 'cobs.cobsr', ],

5) I specified the base directory for the Python code using a `package_dir` option (refer to step 3 for `base_dir`):

    package_dir={
        'cobs' : base_dir + '/cobs',
    },

6) For the C extensions, I gave the path:

    ext_modules=[
        Extension('cobs.cobs._cobs_ext', [ base_dir + '/src/_cobs_ext.c', ]),
        Extension('cobs.cobsr._cobsr_ext', [ base_dir + '/src/_cobsr_ext.c', ]),
    ],

That was about it for `setup.py`. The `setup.py` file is parsable by both Python 2.x and 3.x.

7) Finally, if you build a source distribution using:

python2 setup.py sdist

then it will by default pull in only the files that are specifically needed to build for that Python. E.g. in the above case, you would only get the files under `python2` in the source distribution, but not those under `python3`. But for a complete source distribution, you want to include the files for both 2.x and 3.x. To do that, create a `MANIFEST.in` file that contains something like this:

include *.txt
recursive-include python2 *
recursive-include python3 *

To see what I did, see the `cobs` source code on PyPI or BitBucket.

Problem

I have a Python module that I would like to upload to PyPI. So far, it is working for Python 2.x. It shouldn't be too hard to write a version for 3.x now. But, after following guidelines for making modules in these places: - Distributing Python Modules - The Hitchhiker’s Guide to Packaging it's not clear to me how to support multiple source distributions for different versions of Python, and it's not clear if/how PyPI could support it. I envisage I would have separate code for: - 2.x - 2.6 (maybe, as a special case to use the new buffer API) - 3.x How is it possible to set up a Python module in PyPI so that someone can do: ``` easy_install modulename ``` and it will install the right thing whether the user is using 2.x or 3.x?

Original source