When I started to learn Python a few years ago, I often wondered about what’s the “correct” or “best” way to prepare your system’s Python environment for the requirements your software project or some Python-based application you’d like to start using may have: Should I install modules using the package manager of my OS? Or by using Python tools for it like pip? What are “virtual environments” and how do I utilize these for my projects? What’s all this pyenv, pip, pipenv, easy_install, setuptools, anaconda, conda, miniconda …

In this article series, I’d like to introduce the most common tools and techniques on how to do this in the Python world.
At the end of the series, I will share some of my thoughts, doubts, and questions I had back then, tell about some experiences I gathered in the meantime and generally share the outcome of this journey and what my Python-Workflow looks like, nowadays.

Introduction to pyenv ?

This first article is about pyenv, a lightweight, yet powerful, Python version management tool that works in user – scope and does stay out of the way of systems global Python interpreters.

Installing pyenv

On my development workstations, the first thing I usually do when preparing my Python development environment is to install pyenv. pyenv lets you easily install and switch between multiple versions of Python. There are no administrative permissions required, since everything happens in your user context and $HOME  directory (~). This way, it is even an option for multi-user environments like a shared system at work or one of those you get when renting cheap hosting for your homepage, in which you do not have root permissions.
It can be installed with one single command like this (make sure to met prerequisites and build dependencies first):

curl https://pyenv.run | bash

Without any changes, this clones the very latest version of pyenv into the directory ~/.pyenv.
Next, this needs to be loaded in any newly launched shell. pyenv is sharing advice on how to do this on its own just after the install-command has been executed:

# Load pyenv automatically by adding
# the following to ~/.bashrc:
export PATH="/home/mrichter/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

?Note that the PATH defined is prefixed with this new folder. This way, it has precedence over any global interpreter version installed in a different location (like those installed by the PMS) for your own shell only, making sure that executing python and pip named commands will always use the versions pointed to from pyenv.

As soon as that code has been entered into your shell configuration file, spawn a subshell to have these loaded in your current environment:

$ exec $SHELL

Now the command pyenv is available in your PATH, as well as any Python interpreter installed by it in the future.
It is recommended to search for updates right away; even when there should be none after a fresh install we just did, since you see that pyenv is working fine at least:

$ pyenv update
Updating /home/mrichter/.pyenv...
...

Using pyenv to install some Python interpreters

Let’s install a few Python interpreters, shouldn’t we?

“A few?!?” – I can hear you say, already ?
Yeah – welcome to the easy-as-f**k – world of pyenv! ? Let’s go for having these versions as an example:

• 2.7.17
• 3.6.9
• 3.8.0

? For a list of all available interpreters, execute pyenv install –list.

Thanks to pyenv, this is as easy as this now:

$ pyenv install 2.7.17
Downloading Python-2.7.17.tar.xz... -> https://www.python.org/ftp/python/2.7.17/Python-2.7.17.tar.xz
Installing Python-2.7.17...
Installed Python-2.7.17 to /home/mrichter/.pyenv/versions/2.7.17
$

BAM – you now have Python 2.7.17 available in your environment! And none is cluttering the global environment since they got installed to ~/.pyenv/versions.
Repeat that for the formerly mentioned versions and you are done.

Watch out for the dependencies!

?Note: You still need to provide Python’s build dependencies for your OS yourself! If you see something like this during interpreter installation, you are most certainly missing any of them (zlib in this example):

$ pyenv install 3.7.5
Downloading Python-3.7.5.tar.xz...
-> https://www.python.org/ftp/python/3.7.5/Python-3.7.5.tar.xz
Installing Python-3.7.5...

BUILD FAILED (Ubuntu 18.04 using python-build 20180424)

Inspect or clean up the working tree at /tmp/python-build.20191126215827.6156
Results logged to /tmp/python-build.20191126215827.6156.log

Last 10 log lines:
    sys.exit(ensurepip._main())
  File "/tmp/python-build.20191126215827.6156/Python-3.7.5/Lib/ensurepip/__init__.py", line 204, in _main
    default_pip=args.default_pip,
  File "/tmp/python-build.20191126215827.6156/Python-3.7.5/Lib/ensurepip/__init__.py", line 117, in _bootstrap
    return _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "/tmp/python-build.20191126215827.6156/Python-3.7.5/Lib/ensurepip/__init__.py", line 27, in _run_pip
    import pip._internal
zipimport.ZipImportError: can't decompress data; zlib not available
Makefile:1141: recipe for target 'install' failed
make: *** [install] Error 1
$

Installing them by adding the appropriate “source” URI in Ubuntu plus executing apt-get build-dep python3.6, as the Python’s build dependencies page suggests, solves this issue in Ubuntu Linux. Please help yourself with any other OS distribution.

Switching between Python Interpreters

You now have several Python interpreters available – and now? How to use and switch between them? Seems a bit unhandy to change all your shebangs to something like ~/.pyenv/versions/3.8.0/bin/python, isn’t it?

Don’t worry – it’s far easier than that!
Let’s say, you are about to start your Python project in the empty/new directory ~/project1. All you need to do is to switch to that directory and enable the interpreter of your choice “locally” (explained in a minute) using pyenv:

~$ cd ~/project1
~/project1$ ls -al
total 0
drwxrwxrwx 1 mrichter mrichter 512 Nov 26 23:39 .
drwxr-xr-x 1 mrichter mrichter 512 Nov 27 00:01 ..
~/project1$ python -V
Python 2.7.15+
~/project1$ pyenv local 3.6.9
~/project1$ python -V
Python 3.6.9
~/project1$ ls -al
total 0
drwxrwxrwx 1 mrichter mrichter 512 Nov 27 00:18 .
drwxr-xr-x 1 mrichter mrichter 512 Nov 27 00:01 ..
-rw-rw-rw- 1 mrichter mrichter   6 Nov 27 00:18 .python-version
~/project1$ cat .python-version
3.6.9
~/project1$

How cool is that?? You can switch your pre-installed environments now with a single command, that only creates a single text file, called .python-version in your projects folder!
That pretty much reduces the shebang – issue to use #!/usr/bin/env python for all your scripts; no matter which version your projects are using and without the need to change it for future version changes ?

About pyenv scopes

?Note: pyenv supports two scopes:

1. local
   The scope of the current project/directory and its subdirectories is called “the local scope”. When you change the directory to a directory for which no local version is defined, this scope automatically changes to what is defined globally.
   As shown in a minute, this scope’s version can be set using the pyenv local command.
2. global
   The scope if no specific local scope has been defined (default) is called “the global scope”. It is defined in the file ~/.pyenv/version and can be set and changed using the pyenv global  command.

The difference of these defined scopes is shown in this short shell-session:

~/test$ pyenv versions
* system (set by /home/mrichter/.pyenv/version)
  2.7.17
  3.6.9
  3.7.5
  3.8.0
~/test$ pyenv global
system
~/test$ pyenv local
pyenv: no local version configured for this directory
~/test$ python -V
Python 2.7.15+
~/test$ /usr/bin/python -V
Python 2.7.15+
~/test$ pyenv local 3.8.0
~/test$ python -V
Python 3.8.0
~/test$ cd ..
~$ python -V
Python 2.7.15+
~$ cd test/
~/test$ python -V
Python 3.8.0
~/test$ mkdir subdir
~/test$ cd subdir/
~/test/subdir$ python -V
Python 3.8.0
~/test/subdir$

• Lines 1-6: In the output of pyenv versions, we see all Python interpreters installed by pyenv. The global one (default) is marked with an asterisk (*); system in this example.
  • system is a special case: It circumvents pyenv installed interpreters and results in the executable python found in the remaining parts of the $PATH variable. This means, that it normally resolves to the Python interpreter that didn’t get installed by pyenv but by your PMS (like brew, apt or yum) or similar.
• Line 7-8: pyenv global again shows that global scope is set to system.
• Line 9-10: pyenv local prints what the current directory’s local scope is set to (nothing so far; falling back to global scope).
• Lines 11-12: The first python -V shows the version of Python in effect. It is the system’s default version of Python (2.7.15+).
  • Lines 13-14: /usr/bin/python -V shows that this is identical to the full path to the system’s default python executable.
• Lines 15-17: pyenv local 3.8.0 sets the local scope to be Python 3.8.0, as the next execution of python -V shows.
  ?Note that this very same command yielded a different result before we set the local scope!
• Lines 18-20: As soon as we leave the directory again, we are back to the global scope’s version 2.7.15+.
• Lines 21-28: Going back into the project directory with its local version defined (containing the file .python-version) immediately enables that locally defined version again, without anything else to care about.
  • This even stays active for subdirectories.

This is pretty much all you need to know about pyenv to get started! It covers 100% of what I need in my day-to-day work, so there should not be missing too much in order to get you started with it ?
Feel free to investigate additional details from the project’s documentation resources.

Next time

I hope you enjoyed this first article!
I always welcome comments of any kind, so feel invited to leave yours in the comment section below ❤️

In the next part of this series, I will introduce pipenv – an advanced package-, dependency- and virtualenv-manager for Python.

Stay tuned to get your head wrapped around the workflow this awesome piece of software brings to your Python dependency management workflow!