POSTS

Managing paths in python virtual environments

There are many ways to fix your path in python virtual environments (and many ways to mess them up).

Imagine you have created a virtual environment via something like:

mkdir /tmp/my_project     # Create directory for project,
cd /tmp/my_project        # enter the new project directory,
python3 -m venv venv      # create your virtual environment
. venv/bin/activate       # and activate it

and then create an example python file via something like

echo 'print("hello world")' > example.py

If you want python files and packages inside your project to be easily importable (e.g., so tools like pylint, ruff, etc. can see them), you may need to adjust your paths. Probably an Editable pip install is the best approach although the easiest is to just Manually set PYTHONPATH.

The following is a survey of various methods:

Editable pip install

You can create a setup.py or pyproject.toml file and do

pip install -e .

to get your project into your venv. This is relatively robust and fairly straightforward to see later if you wonder why your path is the way it is. For example, you can see the setup.py or pyproject.toml file and doing pip freeze will show the installed package.

For pyproject.toml, you can create a minimal version via something like:

cat > pyproject.toml<<EOF
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
EOF

For setup.py, you can create a minimal version via something like:

cat > setup.py <<EOF
from setuptools import setup, find_packages
setup(name='MyPackageName', packages=find_packages())
EOF

In either case, you can do the local install via

pip install -e .

as mentioned above. After that point, your packages should be included in sys.path (although probably at the end which could cause problems depending on your setup).

You can verify that your path correctly includes your project by doing something like cd /tmp to change to a directory outside your project and then python -c 'import example' to import something from your project or do python -c 'import sys; print(sys.path)' to see your sys.path.

The main downsides of an editable install are that it is a minor hassle to create the setup.py or pyproject.toml file. Another issue to keep in mind is that the way your build system makes this work is probably to Add a pth file, so arguably you could just do that yourself and save a few lines of typing.

Manually set PYTHONPATH

You can manually set your PYTHONPATH to include your project via something like

export PYTHONPATH=my_project:${PYTHONPATH}

This is easy but annoying to have to do each time. Also, if you are working on multiple projects, it can clutter your PYTHONPATH

Add a pth file

You can add a .pth file in your site-packages via something like

echo "/tmp/my_project" > \
  /tmp/my_project/venv/lib/python3.10/site-packages/my_project.pth

The above .pth file should be automatically parsed by python whenever the venv is activated and so your path will get appended to sys.path.

This is somewhat clean in the sense that you get the desired path somewhere in sys.path but do not have to clutter your environment variables.

One downside is that you can’t get your desired path to show up earlier in sys.path or PYTHONPATH using the above. Depending on your setup, that may or may not cause trouble. Another problem is that you may later forget about your .pth file and pull your hair out trying to figure out what is mucking with your path. Finally, if the path you put into your .pth file does not exist (e.g., due to a typo), it will be silently ignored (which can be hard to debug).

If you are feeling ambitious, you can actually put executable python code in your .pth file. For example, if you put something like

import sys; sys.path.insert(0, '/tmp/my_project')

into your pth file, that will add /tmp/my_project to the start of your sys.path (at least as of the time when your .pth file is processed). Unfortunately, .pth files are brittle and do not have the best documentation. Errors in the above trick can be hard to see and debug.

Use sitecustomize

In theory, you could use the sitecustomize feature to add a sitecustomize.py in your site-packages directory. Unfortunately, this seems poorly documented and buggy.

Use virtualenvwrapper

If you are a fan of virtualenvwrapper, you can use the add2virtualenv command which will create/update a .pth file for you. This may be slightly better than manually managing your .pth file but still has the drawbacks of .pth files.

Edit activate script

You can edit the activate script in venv/bin/activate to update your PYTHONPATH (and possibly print messages to the console for added clarity on what is happening). One minor downside to this approach is that you should ideally also remember to update the deactivate function in your activate system to restore your original PYTHONPATH.