How to run a Python project using __pycache__ folder?

compilation, cpython, pyc, python, python-3.x

Solution

You can enforce the same layout of pyc-files in the folders as in Python2 by using:

python3 -m compileall -b test3

The option `-b` triggers the output of `pyc`-files to their legacy-locations (i.e. the same as in Python2).

After that you can once again use the compiled cache via:

python3 main.pyc

The way the loading of modules works since PEP-3147, it is impossible to use pyc-files from `__pycache__` folder in the way you intend: If there is no `*.py`-file, the content of the `__pycache__` is never looked-up. Here is the most important part of the workflow:

   import foo
     |
     |
     -- >  [foo.py exists?]  --- NO ----> [foo.pyc exists?]  -- NO --> [ImportError] 
            |                                     |
            |                                    YES
           YES                                    |--> [load foo.pyc]
            |
            |-> [look up in __pycache__]

That means, files from `__pycache__` are only looked up, when a corresponding `*.py`-file can be found.

Obviously, building python scripts with a Python version 3.X in this way and trying to run the resulting pyc-files with another Python version 3.Y will not work: Different Python versions need different pyc-files, this is the whole point behind PEP-3147.

Problem

I want to run a Pythonic project using Python compilation (`.pyc` or `__pycache__`). In order to do that in Python2, I haven't any problem. Here is a simplified example in a Python2 project: Project tree: ``` test2 ├── main.py └── subfolder ├── __init__.py └── sub.py ``` Compile: ``` python -m compileall test2 ``` Project tree after the compile: ``` test2 ├── main.py ├── main.pyc └── subfolder ├── __init__.py ├── __init__.pyc ├── sub.py └── sub.pyc ``` As you can see, several `.pyc` manually generated. Now I can run this project using `main.pyc` as fine, which has a relation with the `sub.py`: ``` python main.pyc ``` Out: ``` Hi Bye ``` main.py content: ``` from subfolder import sub print('Bye') ``` sub.py content: ``` print('Hi') ``` Now I want to retry this behavior in a Python3 project. Here is a simplified `asyncio` (available in Python3) project: Project tree: ``` test3 ├── main.py └── subfolder ├── __init__.py └── sub.py ``` Compile: ``` python3 -m compileall test3 ``` Project tree after the compile: ``` test3 ├── main.py ├── __pycache__ │   └── main.cpython-36.pyc └── subfolder ├── __init__.py ├── __pycache__ │   ├── __init__.cpython-36.pyc │   └── sub.cpython-36.pyc └── sub.py ``` As you can see, `__pycache__` folders manually generated. But I cannot run this project using `main.cpython-36.pyc` which has a relation with `subfolder`: ``` cd test3/__pycache__ python3 main.cpython-36.pyc ``` Out (I expected that produced the `Hi Bye` message): ``` Traceback (most recent call last): File "test3/main.py", line 2, in <module> ModuleNotFoundError: No module named 'subfolder' ``` main.py content: ``` import asyncio from subfolder import sub async def myCoroutine(): print("Bye") def main(): loop = asyncio.get_event_loop() loop.run_until_complete(myCoroutine()) loop.close() main() ``` sub.py content: ``` print('Hi') ``` Question: How can I run this project (above Python3 project) using `__pycache__` folder? Or How can I run a Python3 project with the relation between subfolders using python compilation? [NOTE]: I cannot use the `python compileall` (Python2 compile) in the above `Python3` project due to the `asyncio` method. My Python(s) version is Python2.7 and Python3.6

Original source