Virtualenv Workflow

Over the past couple of days, I had gained an appreciation for the tooling that has developed around the python ecosystem for package management, from having to develop and deploy several python applications.

I’d like to share some guides and tips to maintaining python environments.

pyenv

I highly recommend using pyenv to manage your python environments.(and while we’re on that topict, rvm for Ruby, nenv for node). What these tools have in common is that it makes sure that you can maintain a separate set of dependencies for different projects.

The canonical example of where this useful is if you have two projects, A and B, that both depend on a library, let’s call it unicorn. While working on project A, you realize you need the absolute latest release of the unicorn library. But when you upgrade, project B breaks, because there was some code that made assumptions on the previous version of unicorn. This is a problem if there is a single global installation that both project share. This is why the “global” installation of dependencies can be dangerous.

After installing pyenv, switching python versions becomes as easy as:

1

pyenv shell anaconda-2.4.0

I also highly recommend the pyenv-virtualenv tool. This allows an activation of the virtual environment based on the directory you’re in. The syntax for virtualenv is like this

1
2
3
4
5
6
7

# this creates a new virtualenv managed by pyenv-virtualenv
pyenv virtualenv <python-version> <env-name>

# this creates a .python-version file in the local
# directory, which will instruct the pyenv-virtualenv plugin to activate
# the env whenever you switch to this directory
pyenv local <env-name>

Based on this, I recommend the following naming convention for the env name <python-version>_<project-name>. This is because BOTH the python version and the environment name is captured. WHen using pyenv virtualenv, one of the downsides is that all the envs are stored next to each other, so they somehow have to be namespaced.

So for example, I would do:

1
2

pyenv virtualenv 3.5.1 3_5_1_myproject # assuming python 3.5.1
pyenv local 3_5_1_myproject

Now the local .python-version can be committed to source code, and if someone else needs to recreate it, they just need to make sure that it’s done in the context of a python 3.5.1 environment.

pipenv

I hadn’t heard about pipenv until several days ago, when I had to deploy a python application. I had become so used to npm’s --save-dev option, that I wondered, what was the equivalent for python?

All virtual environments come bundled with a python package manager called pip. Pip is pretty wonderful, but it has many limitations, one of which is the lack of the concept of dev packages. For me, I wanted to install jupyter notebook for development but not for deployment, so this was a dealbreaker.

Enter pipenv. Pipenv allows for specifying dependencies and locking them, like in most other languages(package.json/package-lock.json, Podfile/Podfile.lock, etc)

1
2

pipenv install selenium
pipenv install --dev jupyter

gives us this Pipfile:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[packages]

selenium = "*"


[dev-packages]

jupyter = "*"

which is really neat. Furthermore, if you have an environment set up using pyenv, pipenv will happily use it.

pipenv install will download dependencies only for the production version, which greatly simplifies deployment.