Given you have a python installed on your system (let us reference it as "system python"), and you want to install for example tensorflow, you would install it in a global enviroment:
which python3
would return on a ubuntu /usr/bin/python3 what is the acual system python 3.
which pip3
would return on a ubuntu /usr/bin/pip3 what is the acual system pip 3
pip3 install tensorflow
This is now installed globaly (not recommended) with the latest version of tensorflow.
When you install now python using pip3 install tensorflow this will the latest tensorflow in your global python environment. I'm sure you have the question, what is wrong with this type of installation - the unhappy path begins, when you need a tensorflow 1.x for a specific project, and the 2.0 branch for a new prototype or other packages with different versions. THey can't be installed in parallel in the global scope - they will overwrite each other. Also the related dependencies of each tensorflow are different and can be incompatible. That's why you should be using a virtual environment. This environments will move the denendencies from the global scope, to a directory scope that have no dependency on other libraries you installed.
To get this directory based scoping, I use :code: pipenv. Let's call it pipenv environment. You can install it using pip3 as we did before with tensorflow. "But this will install it in the global scope" - thats ok for this case as this is the only think (well, with pyenv, thats comming next) that should be installed in global scope. Lets read our previous example and add the required installation of pipenv and tensorflow inside of the pipenv environment:
pip3 install pipenv
mkdir tensorflow_project
cd tensorflow_project
pipenv install tensorflow
As pip can use a requirements.txt file to manage your dependencies, pipenv uses a Pipfile. The main difference is that a Pipfile can have production dependencies and development dependeniies. An empty Pipfile will look like this:
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
[packages]
[requires]
python_version = "3.6"
Now inside the dev-packages section you define your development only dependencies. I put here pytest, a very common testing library for python. And inside our packages section, I will add tensorflow. Now the file looks like this:
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
pytest = "*"
[packages]
tensorflow = "*"
[requires]
python_version = "3.6"
Having * as version number, pipenv will try to fetch the latest possible, compatible versions with all your dependencies. You can also force a specific version here. Having such a file in your project and a simple pipenv install will install the dependencies from the just created Pipfile. Note here, that by default, packages inside the dev-packages section will not be installed, as the system can't guess if this system is for development or for production. To install the dev packages as well, I write instead pipenv install --dev. To execute any Console command from your just installed dependencies, run pipenv shell after your pipenv environment was created.
But as I mention at the beginning, still we can get issues when we need to handle different python versions. Some dependencies are not developed any more or other circumstances require us to use a different python version than the one we have as default. To handle different python versions, I recommend pyenv
Pyenv handles your different python versions for you and work great in combination with pipenv that I just told you about. To install it, I will share this script for you, originally posted on liquidweb.com . Let's first install the dependencies of pyenv, using here sudo as we need to install new libraries (using ubuntu 18):
sudo apt update -y
sudo apt install -y make build-essential libssl-dev zlib1g-dev libbz2-dev
sudo apt install -y libreadline-dev libsqlite3-dev wget curl llvm
sudo apt install -y libncurses5-dev libncursesw5-dev xz-utils tk-dev
sudo apt install -y libffi-dev liblzma-dev python-openssl git
A hint for mac users: Just get pyenv using homebrew: brew install pyenv. Now lets clone the repository using git and set some environment variables for pyenv, to make sure the correct python version can be set.
This commands can be run using your local (non root) user, as no root priviliges are required:
git clone https://github.com/pyenv/pyenv.git ~/.pyenv
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n eval "$(pyenv init -)"\nfi' >> ~/.bashrc
This commands will clone the pyenv repository from github, set a new environment variable in your bashrc file and add the pyenv root to your path, to allow pyenv appear in front of any existing python installation. The last command allow you to initialize your shell with pyenv - please remember that this examples are for ubuntu only. As last command, you need to restart your current shell session by using exec "$SHELL".
Now we are ready to instal some stuff with pyenv:
cd ~
mkdir -p tensorflow392
cd tensorflow392
pyenv install 3.9.2
pyenv local 3.9.2
pyenv rehash
python3 -m pip install --upgrade pip
pip3 install pipenv
We go to our home directory, create a directory named tensorflow392, ignore the case that it may exist, and change to this directory. Than we install python 3.9.2. With pyenv local 3.9.2 we set the current working directory to work with python 3.9.2. After you install any new executables, you should run pyenv rehash to make sure everything is in sync.
We update the preinstalled pip with version. Note that this will upgrade pip inside this particullar version globally (you could call pyenv global) - that is what we want. And the final command will install pipenv.
Now we have a prepared setup, to lunch a new environment with our desired version.
pyenv rehash
pipenv install tensorflow
Oh, it did not work. tensorflow is currently not supporting the python 3.9.x branch, so we need an older version. Let's try out with version 3.8.8
cd ~
mkdir -p tensorflow388
cd tensorflow388
pyenv install 3.8.8
pyenv local 3.8.8
python3 -m pip install --upgrade pip
pip3 install pipenv
pyenv rehash
pipenv install tensorflow
And this is our last code snippet for this post, a full working example for installing tensorflow with a specific version. Remember to use pyenv rehash before you run the pipenv command, otherwise the global pipenv may be fetched - but thats not what you want. You will get a hit that the versions are not matching from pipenv that you should not ignore. And the last command will as well create a Pipfile for you with the correct python version set and tensorflow as dependency. You can add more to the Pipfile as you like and than remove the environment by writing pipenv --rm
When you use docker, you can do everything described here as well, but you wil just have another layer of abstraction. To have a good development experience and not much suprises, use pipenv in both cases and let them share a Pipfile and the corresponding Pipfile.lock - File. Use pyenv to set your local version and a docker base image with the required python version as you need.