Building a numerical Python environment in Debian

Published October 22, 2013, by .

Copied from http://eliteraspberries.com/blog/2013/10/building-a-numerical-python-environment-in-debian.html, now missing, and copied from the Wayback Machine on 18 Feb 2017.

This also can apply to Ubuntu.

The Python programming language is great for prototyping applications involving some numerical processing. In this post I describe how to build an optimized environment for numerical applications using Python, with the Debian GNU/Linux operating system.

Following these instructions will provide the following list of software:

This setup is useful for numerical applications generally, and more specifically applications making heavy use of the FFT and some multimedia processing, such as computer vision applications.

The advantage of building from source (compiling) rather than using generic binary packages from the package manager is that libraries like FFTW will be faster when optimized specifically for your machine. Packages in the Debian package manager are compiled for generic processors within a certain architecture (x86, ARM, etc.) without making use of any processor-specific extensions.

The disadvantage of compiling is that it takes some time. In order to help save time, I have collected all the commands from the instructions below, into a single script that can be run unsupervised; see the section Shortcut: build script at the end of this post. However, first ensure that the CFLAGS environment variable is set, as described below.

Shortcut: binary packages

If you do not wish to take the time to compile software, use the Debian package manager, APT, to install these binary packages:

like so:

$ sudo apt-get install python-minimal python-numpy ...

However, for numerically-intensive applications it is best to build these packages from source, so that the compiler makes the best use of the available hardware.

Set CFLAGS

Before building software from source (specifically C and C++ programs) it is best to have set the correct CFLAGS environment variable in the file .bashrc in the user’s home directory.

In previous posts, I describe CFLAGS for the x86 and x86_64 architectures, for the Raspberry Pi, and for the Beaglebone Black. Look up the correct CFLAGS for your processor before proceeding.

In order to set the CFLAGS environment variable, edit the file .bashrc in the user’s home directory, so that it contains the lines:

CC="gcc"
export CC

CFLAGS="..."
export CFLAGS

CXXFLAGS="$CFLAGS"
export CXXFLAGS

Replace “gcc” above with “clang” if you are using the Clang compiler, and set the CFLAGS line(s) according to your hardware.

Ensure the changes are in effect:

$ source ~/.bashrc

Check that the environment variables are set properly:

$ echo $CFLAGS
$ echo $CXXFLAGS

Update and configure APT

APT is the Debian package manager. APT can be used to install binary packages, or to build and install packages from source.

In either case, ensure that APT is up to date before using it:

$ sudo apt-get update
$ sudo apt-get install apt-src
$ sudo apt-src update

Then edit the file buildflags.conf in the directory .config/dpkg in the user’s home directory to prepend the CFLAGS set above to the CFLAGS used during the building of packages, as follows:

$ mkdir -p ~/.config/dpkg
$ echo "PREPEND CFLAGS $CFLAGS" >> ~/.config/dpkg/buildflags.conf
$ echo "PREPEND CXXFLAGS $CXXFLAGS" >> ~/.config/dpkg/buildflags.conf

Check that the new CFLAGS are taken into account:

$ dpkg-buildflags --get CFLAGS
$ dpkg-buildflags --get CXXFLAGS

Install compilers

Building software from source requires compilers. Install C, C++, and Fortran compilers, as well as linkers and other necessary tools:

$ sudo apt-get install build-essential gcc g++ gfortran

If you prefer to use the Clang compiler instead of GCC, replace the gcc and g++ packages above with the clang package (the latter provides both C and C++ compilers).

Finally, install the pkg-config tool:

$ sudo apt-get install pkg-config

Install Python

The Python interpreter itself and the Python library can be installed as binaries from the package manager:

$ sudo apt-get install python-minimal python-dev

The Python interpreter is unlikely to be a bottleneck in numerical code because most people will use specialized libraries like NumPy instead of the built-in mathematics library, so it’s unnecessary to build these from source.

On Debian Wheezy, the python2 shortcut may be absent. Check for it:

$ python2 --version

If a “command not found” error is printed instead of a version number, create the shortcut:

$ PYTHON2=$(python -c 'import sysconfig; print "python"+sysconfig.get_config_var("VERSION")')
$ sudo ln -s $PYTHON2 /usr/bin/python2

Check that Python version 2 is installed:

$ python2 --version

Build and install NumPy

NumPy is the de facto standard library for Python numerical computing.

Download, build and install the latest stable version of NumPy:

$ wget https://pypi.python.org/packages/source/n/numpy/numpy-1.7.1.tar.gz
$ gunzip < numpy-1.7.1.tar.gz | tar -xf -
$ cd numpy-1.7.1
$ python setup.py build
$ sudo python setup.py install
$ cd

Check that NumPy is installed:

$ python2 -c 'import numpy; print numpy.__version__'

Then place a hold on the python-numpy package to prevent APT from installing the binary package over the version just installed:

$ sudo apt-mark hold python-numpy

Build and install matplotlib

The matplotlib package is a great plotting library for Python.

First install its requirements: the libpng development library; the FreeType 2 development library; and the dateutil and pyparsing packages. Do so using the package manager:

$ sudo apt-get install libpng-dev libfreetype6-dev
$ sudo apt-get install python-dateutil python-pyparsing

Then download, build and install the latest stable version of matplotlib:

$ wget https://downloads.sourceforge.net/project/matplotlib/matplotlib/matplotlib-1.3.1/matplotlib-1.3.1.tar.gz
$ gunzip < matplotlib-1.3.1.tar.gz | tar -xf -
$ cd matplotlib-1.3.1
$ python setup.py build
$ sudo python setup.py install
$ cd

Check that matplotlib is installed:

$ python2 -c 'import matplotlib; print matplotlib.__version__'

Place a hold on the python-matplotlib package with APT:

$ sudo apt-mark hold python-matplotlib

Build and install FFTW and pyFFTW

FFTW is an excellent FFT library, and pyFFTW is a Python interface to FFTW.

To install FFTW, use apt-src which will download the source code as well as install any build dependencies:

$ apt-src install libfftw3-dev

Then compile it:

$ apt-src build libfftw3-dev

Then install the compiled packages:

$ sudo dpkg --install libfftw3-*.deb

Place a hold on the package, to prevent APT from upgrading the package to a later binary version:

$ sudo apt-mark hold libfftw3-dev

Now getversion 0.9.2 of pyFFTW:

$ wget https://pypi.python.org/packages/source/p/pyFFTW/pyFFTW-0.9.2.tar.gz
$ gunzip < pyFFTW-0.9.2.tar.gz | tar -xf -
$ cd pyFFTW-0.9.2

On some Debian systems, such as the armhf port, FFTW is built without any ‘long double’ precision support. In that case it is necessary to patch pyFFTW to remove long double support, as follows:

$ wget http://www.eliteraspberries.com/files/pyFFTW-0.9.2-no-long-double.patch
$ patch -p1 < pyFFTW-0.9.2-no-long-double.patch

Next install the Cython compiler and compile the patched Cython code:

$ sudo apt-get install cython
$ cython pyfftw/pyfftw.pyx

Finally, build and install pyFFTW:

$ python setup.py build
$ sudo python setup.py install
$ cd

Check that pyFFTW is installed:

$ python2 -c 'import pyfftw; print pyfftw.version'

Build and install OpenCV

OpenCV is a computer vision library including a Python interface.

First download, build and install the FFMPEG development libraries, which are dependencies of OpenCV:

$ sudo apt-get install yasm
$ wget http://www.ffmpeg.org/releases/ffmpeg-2.0.2.tar.gz
$ gunzip < ffmpeg-2.0.2.tar.gz | tar -xf -
$ cd ffmpeg-2.0.2
$ ./configure --enable-shared --enable-pic
$ make
$ sudo make install
$ cd

Mark the FFMPEG libraries and command-line tools as on hold:

$ sudo apt-mark hold libavdevice-dev libavfilter-dev libavformat-dev
$ sudo apt-mark hold libavcodec-dev libswscale-dev libavutil-dev ffmpeg

Then download, build and install OpenCV:

$ sudo apt-get install libgtk2.0-dev cmake
$ wget http://sourceforge.net/projects/opencvlibrary/files/opencv-unix/2.4.6.1/opencv-2.4.6.1.tar.gz
$ gunzip < opencv-2.4.6.1.tar.gz | tar -xf -
$ cd opencv-2.4.6.1
$ mkdir -p build
$ cd build
$ cmake -DCMAKE_BUILD_TYPE=RELEASE ..
$ make
$ sudo make install

Check that OpenCV and its Python interface are installed:

$ python2 -c 'import cv2; print cv2.__version__'

Finally, mark the libopencv-dev and python-opencv packages as on hold:

$ sudo apt-mark hold libopencv-dev python-opencv

Shortcut: build script

I have collected all the commands outlined above into a single file so that it is easier to launch the entire the process and let it run unsupervised.

Because all these commands will take some time to complete, configure the sudo utility so that it remembers passwords indefinitely. To do so, edit the configuration file for the sudo command:

$ sudo visudo

and set the timestamp_timeout option to -1, by adding a line as follows:

Defaults	timestamp_timeout=-1

Save the changes and continue.

Download this list of commands as a script and run it, like so:

$ wget http://www.eliteraspberries.com/files/build-num-python-debian.sh
$ chmod a+x ./build-num-python-debian.sh
$ ./build-num-python-debian.sh

The script will ask for a password only once at the beginning. If it completes successfully, a list of version numbers will be written to the file build-num-python-debian.log.

Once the script is finished edit the sudo configuration again, setting the value of timestamp_timeout to a reasonable number in minutes.

← Return to blog index