Building a numerical Python environment in Debian
Published October 22, 2013, by Mansour Moufid.
Copied from http://eliteraspberries.com/blog/2013/10/building-a-numerical-python-environment-in-debian.html, now missing, and copied from the Wayback Machine on 18 Feb 2017.
This also can apply to Ubuntu.
The Python programming language is great for prototyping applications involving some numerical processing. In this post I describe how to build an optimized environment for numerical applications using Python, with the Debian GNU/Linux operating system.
Following these instructions will provide the following list of software:
- Python version 2;
- the OpenCV Python bindings; and
- all of the above’s prerequisites.
This setup is useful for numerical applications generally, and more specifically applications making heavy use of the FFT and some multimedia processing, such as computer vision applications.
The advantage of building from source (compiling) rather than using generic binary packages from the package manager is that libraries like FFTW will be faster when optimized specifically for your machine. Packages in the Debian package manager are compiled for generic processors within a certain architecture (x86, ARM, etc.) without making use of any processor-specific extensions.
The disadvantage of compiling is that it takes some time. In order to help save time, I have collected all the commands from the instructions below, into a single script that can be run unsupervised; see the section Shortcut: build script at the end of this post. However, first ensure that the CFLAGS environment variable is set, as described below.
Shortcut: binary packages
If you do not wish to take the time to compile software, use the Debian package manager, APT, to install these binary packages:
$ sudo apt-get install python-minimal python-numpy ...
However, for numerically-intensive applications it is best to build these packages from source, so that the compiler makes the best use of the available hardware.
Before building software from source (specifically C and C++ programs) it is best to have set the correct CFLAGS environment variable in the file
.bashrc in the user’s home directory.
In order to set the CFLAGS environment variable, edit the file
.bashrc in the user’s home directory, so that it contains the lines:
CC="gcc" export CC CFLAGS="..." export CFLAGS CXXFLAGS="$CFLAGS" export CXXFLAGS
Replace “gcc” above with “clang” if you are using the Clang compiler, and set the CFLAGS line(s) according to your hardware.
Ensure the changes are in effect:
$ source ~/.bashrc
Check that the environment variables are set properly:
$ echo $CFLAGS $ echo $CXXFLAGS
Update and configure APT
APT is the Debian package manager. APT can be used to install binary packages, or to build and install packages from source.
In either case, ensure that APT is up to date before using it:
$ sudo apt-get update $ sudo apt-get install apt-src $ sudo apt-src update
Then edit the file
buildflags.conf in the directory
.config/dpkg in the user’s home directory to prepend the CFLAGS set above to the CFLAGS used during the building of packages, as follows:
$ mkdir -p ~/.config/dpkg $ echo "PREPEND CFLAGS $CFLAGS" >> ~/.config/dpkg/buildflags.conf $ echo "PREPEND CXXFLAGS $CXXFLAGS" >> ~/.config/dpkg/buildflags.conf
Check that the new CFLAGS are taken into account:
$ dpkg-buildflags --get CFLAGS $ dpkg-buildflags --get CXXFLAGS
Building software from source requires compilers. Install C, C++, and Fortran compilers, as well as linkers and other necessary tools:
$ sudo apt-get install build-essential gcc g++ gfortran
If you prefer to use the Clang compiler instead of GCC, replace the
g++ packages above with the
clang package (the latter provides both C and C++ compilers).
Finally, install the
$ sudo apt-get install pkg-config
The Python interpreter itself and the Python library can be installed as binaries from the package manager:
$ sudo apt-get install python-minimal python-dev
The Python interpreter is unlikely to be a bottleneck in numerical code because most people will use specialized libraries like NumPy instead of the built-in mathematics library, so it’s unnecessary to build these from source.
On Debian Wheezy, the
python2 shortcut may be absent. Check for it:
$ python2 --version
If a “command not found” error is printed instead of a version number, create the shortcut:
$ PYTHON2=$(python -c 'import sysconfig; print "python"+sysconfig.get_config_var("VERSION")') $ sudo ln -s $PYTHON2 /usr/bin/python2
Check that Python version 2 is installed:
$ python2 --version
Build and install NumPy
NumPy is the de facto standard library for Python numerical computing.
Download, build and install the latest stable version of NumPy:
$ wget https://pypi.python.org/packages/source/n/numpy/numpy-1.7.1.tar.gz $ gunzip < numpy-1.7.1.tar.gz | tar -xf - $ cd numpy-1.7.1 $ python setup.py build $ sudo python setup.py install $ cd
Check that NumPy is installed:
$ python2 -c 'import numpy; print numpy.__version__'
Then place a hold on the
python-numpy package to prevent APT from installing the binary package over the version just installed:
$ sudo apt-mark hold python-numpy
Build and install matplotlib
The matplotlib package is a great plotting library for Python.
First install its requirements: the libpng development library; the FreeType 2 development library; and the dateutil and pyparsing packages. Do so using the package manager:
$ sudo apt-get install libpng-dev libfreetype6-dev $ sudo apt-get install python-dateutil python-pyparsing
Then download, build and install the latest stable version of matplotlib:
$ wget https://downloads.sourceforge.net/project/matplotlib/matplotlib/matplotlib-1.3.1/matplotlib-1.3.1.tar.gz $ gunzip < matplotlib-1.3.1.tar.gz | tar -xf - $ cd matplotlib-1.3.1 $ python setup.py build $ sudo python setup.py install $ cd
Check that matplotlib is installed:
$ python2 -c 'import matplotlib; print matplotlib.__version__'
Place a hold on the
python-matplotlib package with APT:
$ sudo apt-mark hold python-matplotlib
Build and install FFTW and pyFFTW
To install FFTW, use
apt-src which will download the source code as well as install any build dependencies:
$ apt-src install libfftw3-dev
Then compile it:
$ apt-src build libfftw3-dev
Then install the compiled packages:
$ sudo dpkg --install libfftw3-*.deb
Place a hold on the package, to prevent APT from upgrading the package to a later binary version:
$ sudo apt-mark hold libfftw3-dev
Now getversion 0.9.2 of pyFFTW:
$ wget https://pypi.python.org/packages/source/p/pyFFTW/pyFFTW-0.9.2.tar.gz $ gunzip < pyFFTW-0.9.2.tar.gz | tar -xf - $ cd pyFFTW-0.9.2
On some Debian systems, such as the armhf port, FFTW is built without any ‘long double’ precision support. In that case it is necessary to patch pyFFTW to remove long double support, as follows:
$ wget http://www.eliteraspberries.com/files/pyFFTW-0.9.2-no-long-double.patch $ patch -p1 < pyFFTW-0.9.2-no-long-double.patch
Next install the Cython compiler and compile the patched Cython code:
$ sudo apt-get install cython $ cython pyfftw/pyfftw.pyx
Finally, build and install pyFFTW:
$ python setup.py build $ sudo python setup.py install $ cd
Check that pyFFTW is installed:
$ python2 -c 'import pyfftw; print pyfftw.version'
Build and install OpenCV
OpenCV is a computer vision library including a Python interface.
First download, build and install the FFMPEG development libraries, which are dependencies of OpenCV:
$ sudo apt-get install yasm $ wget http://www.ffmpeg.org/releases/ffmpeg-2.0.2.tar.gz $ gunzip < ffmpeg-2.0.2.tar.gz | tar -xf - $ cd ffmpeg-2.0.2 $ ./configure --enable-shared --enable-pic $ make $ sudo make install $ cd
Mark the FFMPEG libraries and command-line tools as on hold:
$ sudo apt-mark hold libavdevice-dev libavfilter-dev libavformat-dev $ sudo apt-mark hold libavcodec-dev libswscale-dev libavutil-dev ffmpeg
Then download, build and install OpenCV:
$ sudo apt-get install libgtk2.0-dev cmake $ wget http://sourceforge.net/projects/opencvlibrary/files/opencv-unix/188.8.131.52/opencv-184.108.40.206.tar.gz $ gunzip < opencv-220.127.116.11.tar.gz | tar -xf - $ cd opencv-18.104.22.168 $ mkdir -p build $ cd build $ cmake -DCMAKE_BUILD_TYPE=RELEASE .. $ make $ sudo make install
Check that OpenCV and its Python interface are installed:
$ python2 -c 'import cv2; print cv2.__version__'
Finally, mark the
python-opencv packages as on hold:
$ sudo apt-mark hold libopencv-dev python-opencv
Shortcut: build script
I have collected all the commands outlined above into a single file so that it is easier to launch the entire the process and let it run unsupervised.
Because all these commands will take some time to complete, configure the
sudo utility so that it remembers passwords indefinitely. To do so, edit the configuration file for the
$ sudo visudo
and set the
timestamp_timeout option to
-1, by adding a line as follows:
Save the changes and continue.
Download this list of commands as a script and run it, like so:
$ wget http://www.eliteraspberries.com/files/build-num-python-debian.sh $ chmod a+x ./build-num-python-debian.sh $ ./build-num-python-debian.sh
The script will ask for a password only once at the beginning. If it completes successfully, a list of version numbers will be written to the file
Once the script is finished edit the
sudo configuration again, setting the value of
timestamp_timeout to a reasonable number in minutes.