Python packaging, dependency management, and development tools

For most of the time I've been using Python, I've managed dependencies either with my system package manager or with a simple pip requirements.txt file and venv (or virtualenv in Python 2). This has served me well: it's simple and covers most use cases.

In the past few years, there has been a surge of interest in improving Python dependency management. Many dependency management tools are now available, some old, some new, including Pipenv, pyenv, Poetry, and conda. The PyPA working group recommends Pipenv (as of 2020-05-30), but I found it to be prohibitively slow last time I tried it. I decided to give Poetry a try on a small project after reading positive reviews of it online. So far I'm happy with it. It's fast, easy enough to use, and hasn't gotten in the way too much. I can see myself using it on future projects.

I also gave tox a try, having previously only used unittest or pytest directly. tox is not an alternative to unittest or pytest, but rather a tool for running your tests in a number of different environments. In my case, I only have one environment I wish to test in (Python 3.8), but I can see it being useful when authoring programs that need to run under many different Python versions.

Black is a tool to standardize Python code formatting. Similar tools for other languages include gofmt and rustfmt. I would recommend automatic code formatters to everyone. Not having to think about code style reduces unnecessary mental load when programming, allowing you to focus on the functionality of your software rather than its appearance.

Here is how I set up the environment for my latest project:

poetry new myproject
cd myproject
git init
curl -O .gitignore https://raw.githubusercontent.com/github/gitignore/master/Python.gitignore
poetry add -D black tox
poetry add dependency1 dependency2 ....

If you need to run a command inside the project's environment, you can use poetry run mycommand or poetry shell. This is similar to sourcing the activate script in a venv.

Here is how I configured tox (see here):

# tox.ini
[tox]
isolated_build = true
envlist = py38

[testenv]
whitelist_externals = poetry
commands =
    poetry install
    poetry run black --check .
    poetry run pytest tests/

Before committing any changes, I run Black to reformat my code and tox to run tests:

poetry run black .
poetry run tox

I'm satisfied with this new workflow. It's simple enough to set up and use that I can see myself using it instead of pip+venv in the future. Poetry has the added benefit of handling all the packaging work you would otherwise have to do manually, though I haven't needed that for this project.