As a ROS developer, I find myself wanting to do clean builds of my ROS workspace
pretty frequently. This can take a really long time if I’m compiling things like
GTSAM and g2o or visual odometry algorithms. ccache is a compiler cache which
hashes the source files to determine if it can use a cached version of the object
file when doing compilation. The authors claim that it produces identical results
to normal compilation. distcc is a way to distribute compilation across several
computers simultaneously, so I can do things like make -j12 -l12 with my laptop, using my desktop computer as a compilation server
Installing ccache and distcc
This is easy
sudo apt install ccache distcc
Setting up ccache
So, what you end up doing is pointing to ccache itself as your compiler. It in
turn knows where the actual gcc and g++ binaries are, and will use them if it
needs to. All you do is prepend /usr/lib/ccache to your PATH variable.
Do this by adding the following line to the bottom of your ~/.bashrc file
export PATH=/usr/lib/ccache:$PATH
Next, source your ~/.bashrc and check that it worked with echo $PATH. You
should see something like this
superjax ~ $ echo $PATH
/usr/lib/ccache:/opt/ros/kinetic/bin:/home/superjax/bin:/home/superjax/.local/bin:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:
/snap/bin:/opt/sublime_text:/opt/gcc-arm-none-eabi-5_4-2016q3/bin:/opt/sublime-text
where \usr\lib\ccache is at the start.
Another check you can do is which gcc which should return the following:
superjax ~ $ which gcc
/usr/lib/ccache/gcc
Setting up distcc (server)
So, go ahead and install distcc on the server computer (in my case, the desktop)
sudo apt install distcc
And then spin up the server. (The following assumes I’m on the 10.8.33.0 subnet, and I’m allowing all hosts on that subnet to send jobs)
sudo distccd --log-file=/tmp/distccd.log --daemon -a 10.8.33.0/24
Setting up distcc (client)
So, now you have to tell ccache to use the distcc distributed compilation servers.
To do this, just add the following line to your ~/.bashrc file.
export CCACHE_PREFIX=distcc
Next, we have to tell distcc where to go to find the server. This also, just add
to the bottom of your ~/.bashrc (my desktop is at 10.8.33.182 and it has 8 cores,
my laptop is at localhost and has 4)
export DISTCC_HOSTS='localhost/4 10.8.33.182/8'
Another useful thing if you’re using ROS, is to tell ROS automatically how many
jobs it has available during a catkin_make. This is easily done by adding the
following to your ~/.bashrc.
export ROS_PARALLEL_JOBS='-j'$(distcc -j)' -l'$(distcc -j)
Test Configuration
So, I just went ahead and compiled g2o from source. To check that ccache is working just run
watch -n1 'ccache -s'
in a separate terminal, and you should see the statistics change. You can also
ssh into your build server, and run top or htop to see if the distcc process
starts spooling up its threads. If all goes well, you should have vastly
improved compile times.