Building Google OR-Tools from Source on Debian Jessie
Disclaimer: I can only claim that these instructions work for compiling the current version of Google OR-Tools on Debian Jessie for use as a python 2.7 package (although the instructions for Python 3 should be similar, I have yet to figure that one out). As I understand it, there are a LOT of nuances to compiling this tool which unfortunately means that even a small deviation from the formula described below will not produce the desired results. Most notably these instructions may not work on Ubuntu because they use different package names and because of several other differences.
Introduction:
I was working on a school project and wanted to learn Google OR-Tools because as of right now, the only constraint programming languages/APIs that I have experience with are MiniZinc and Objective-CP so I figured that I should expand my world a little bit. Little did I know that compiling one of these tools from scratch is an unreliable process at best. I had to go through several different issues before, during, and after installation and I do not think I could've done it without the help of lprerron who manages the Google OR-Tools repository so kudos to him.
For those who are not aware, Google OR-Tools is an API for C++, Java, Python, and C# that can be used for Constraint Programming and several other related approaches to optimization technology. If you do not know what Constraint Programming is, you probably will not find the rest of this post useful, but I strongly recommend that you learn as it is an excellent and extremely practical tool for solving some of the most difficult problems in computer science and by extension in the real world. They can help a business become very efficient and, although few people know it, none of the large corporations functioning today would've ever been able to get where they are without these tools (which is not to say that they are all using Google OR-Tools specifically, just that they are using some kind of optimization software).
A general guide to Google OR-Tools can be found here. It is an open source tool (of course) under the Apache 2.0 license. There is a set of instructions here that shows how to compile the software (which is what these instructions are based on), but it is not always clear, and there are instructions here on how to install it without building from source for whatever language (C++, C#, Python, or Java) that you want to use it with. Unfortunately I had multiple serious issues with getting it to work using the "normal" installation procedure, in one case, it would even install properly and I could run some of the test examples, but then I quickly discovered that not all of the modules installed and I could not run some of the modules required for solving the Traveling Salesman problem. That's why I figured I would write up this little page.
I should point out that one of the issues I had was that the program seems to experience a lot of problems with access permissions when installed "normally". I do not mind requiring root permissions to install a program (for that matter that's how it should be), but after installation I discovered that I needed root priveleges just to access the package. The most common solution on the internet was to simply run all of your scripts as root which I found to be absolutely unacceptable. This guide therefore describes how to compile and install OR-Tools in such a way as to not need root priveleges for running the sripts. You will however need root privileges to install.
Installing Pre-Requisites:
I am going to pretend like this is being done on a completely clean Debian system (for that matter I created a VM to make sure the process was reproduceable from scratch). You should begin with a complete update to make sure all the packages are up to date (Note that the $ prompt implies doing something as a normal user, while the # implies doing something as root, although I am going to assume that the user in question has sudo privileges and that sudo is installed):
$ su
# apt-get update; apt-get upgrade -y; apt-get dist-upgrade -y
Then we can install the pre-requisites for compiling these tools from scratch:
# apt-get install bison flex python-setuptools python-dev autoconf libtool texinfo help2man texlive gawk g++ zlib1g-dev cmake subversion git curl
This ends up being like 700MB+ of packages to download so it might take a little while. That's OK. This compilation is going to be a long process, so settle in.
You're also going to need to use a text editor. You can use nano or vim which should come pre-installed, but I prefer emacs:
# apt-get install emacs
Downloading and Extracting the Source Code:
I downloaded the latest release of Google OR-Tools that I could find at the time, which is the 2015-12 version which can be found here: https://github.com/google/or-tools/archive/v2015-12.tar.gz although since this article was posted, a newer version may have been released.
You can download and extract the source code like so (preferably not as root):
$ wget https://github.com/google/or-tools/archive/v2015-12.tar.gz
$ tar -xpf v2015-12.tar.gz
Compiling the Source Code
Now that you've extracted the contents of the archive, you should see a folder called or-tools-2015-12. You need to go into this folder and then start the compilation. This is gonna take a little while, you're going to compile the relevant third-party tools before compiling OR-Tools itself:
# cd or-tools-2015-12
# make third_party
During this process, at some point the compilation program will say something like this:
Error validating server certificate for 'https://projects.coin-or.org:443':
- The certificate is not issued by a trusted authority. Use the
fingerprint to validate the certificate manually!
Certificate information:
- Hostname: *.coin-or.org
- Valid: from Apr 11 20:44:41 2014 GMT unil Apr 11 20:44:41 2017 GMT
- Issuer: http://certs.godaddy.com/repository/, GoDaddy.com, Inc., Sottsdale, Arizona, US
- Fingerprint: E7:5B:E2:82:44:7F:EF:52:8B:E2:07:B0:D9:B4:27:0C:0E::64:11:56
(R)eject, accept (t)emporarily or accept (p)ermanently?
In this case I recommend using t to temporarily accept the certificate so the program can finish the download. Notably if you take too long to press t you get the following message:
...
... Error running context: The server unexpectedly closed the connection.
...
Do not panic, it simply means you took too long to reply with t which is understandable if you stepped away from your computer while it was compiling. If you simply run make third_party again it should get to the prompt above very quickly and then you can type in t and be on your merry way again.
Its possible that building flex will fail, that's fine. That's why we installed it as a pre-requisite in the beginning.
Now you can do:
$ make install_python modules
That should be relatively quick.
Building and Installing the Python Modules
Start by executing (takes a while, feel free to step away from your computer without having to worry about pressing t like in the previous section):
$ make python
The next step is to test the output of the previous command, which should be done like so:
$ make test_python
If the tests complete successfully, it should be a strong indicator that you have done everything correctly up to this point and then you can try to compile the PyPi archive:
$ make pypi_archive
Its possible (as in it did happen for me) that that command will fail. Do not panic, simply edit the file: makefiles/Makefile.python.mk to comment out the following line: -rm temp/ortools/setup.py-e (for me it was line 287, but it might be different depending on the version, so look for the content, not the line number):
$ emacs makefiles/Makefile.python.mk
283 $(SED) -i -e 's/\.dll/\.so/' temp/ortools/setup.py
284 $(SED) -i -e 's/DELETEWIN //g' temp/ortools/setup.py
285 $(SED) -i -e '/DELETEUNIX/d' temp/ortools/setup.py
286 $(SED) -i -e 's/DLL/$(DYNAMIC_LIB_SUFFIX)/g' temp/ortools/setup.py
287 # -rm temp/ortools/setup.py-e
288 ifeq ($(PLATFORM),MACOSX)
289 tools/fix_python_libraries_on_mac.sh
290 endif
291 ifeq ($(PLATFORM),LINUX)
Note that this file format is apparently tab deliniated, and if you end up editing any of the other lines (either by accident, like me, or on purpose) just make sure to put tabs to deliniate lines that need them, and not spaces. I ran into a problem with that.
Once you have edited the file, try running the previous command again. This time it should work:
$ make pypi_archive
if that step completed successfully, the PyPi archive should be present in the temp/ortools/ directory. If you go in there, you can install it as a python module accessible to all the python scripts run on your system:
$ cd temp/ortools
$ sudo python setup.py install
Now you should be good to go! You can run a quick test to make sure by going into the examples directory and trying out the Traveling Salesman example:
$ cd ../../examples/python
$ python tsp.py
Note that you do not need to run the examples as root, everything works exactly as it should with the proper permissions. You can place that scrip in any directory and run it anywhere on your system. All is as it should be. You can clean up now by deleting the or-tools-2015-12 directory from wherever you downloaded it to on your system. Don't worry, the compiled modules have been installed and should still be available.
I recommend saving a copy of the examples directory because those are important for understanding how to model properly using Google OR-Tools. In case you lose it, you can always find the examples here: https://github.com/google/or-tools/tree/master/examples/python and there are even more examples on this wonderful site: http://hakank.org/google_or_tools/.
I hope you have fun! Please feel free to contact me at eugene at kovalev dot systems if you find any issues with these instructions or have changes I could add that would allow these instructions to work on other operating systems as well.