johnSo let's say you're a retrocomputing aficionado who enjoys running modern Unix on classic hardware. You're running NetBSD (after all, it's one of very few OSes which doesn't treat alternate architectures with disdain) but you cringe whenever Perl is updated because it'll take something like a week to update everything on your m68030. Sure, you already know how to crosscompile the OS, but that doesn't work for pkgsrc packages. On the other hand, you're not one to knowingly run insecure software, so waiting for binary packages for your platform isn't an option.
This guide will help you set up distcc to distribute pkgsrc compile tasks to another machine over your network.
These days you can pick up a Raspberry Pi for $40 USD, a BeagleBone Black for $50 or a PogoPlug for $20 or less. I like PogoPlugs because they're physically robust and are really hard to accidentally unplug, whereas if you sneeze the wrong way around a Pi you might be waiting for it to finish a reboot (Ok, it's not really that bad, but you get the point).
http://www.netbsd.org/docs/guide/en/chap-fetch.html
In my examples I'm using a PogoPlug Mobile with a rather modest 16 gig USB flash drive. It has an 800 MHz StrongARM (Kirkwood ARMv5), 128 megs of memory and gigabit ethernet and cost $14 USD when Radio Shack still had them. See my article on how to install NetBSD without needing a serial console if you're interested here:
The retrocomputer this machine will assist is a VAXstation 4000/60 with 56 megs of memory. I'll refer to them as retro and modern. retro in these examples is at 10.12.26.190 and modern is at 10.12.26.189.
Start by installing devel/distcc from pkgsrc on both retro and modern machines (or use whatever package method you like to install distcc if you're running GNU/Linux or something else). If you'd prefer to use distcc 3.2rc1 instead of 3.1, remove patches/patch-ad, patches/patch-src_emaillog.c and patches/patch-src_dotd.c, then change PKGREVISION in Makefile to 0 and DISTCC_VERSION to 3.2rc1 in Makefile.common, then run make with NO_CHECKSUM=yes.
While that's running, we'll compile a toolchain targetting retro on modern using NetBSD's build.sh. The locations below are chosen simply because they're easy to remember. Here's how we make a VAX toolchain. In /usr/src/:
./build.sh -O /usr/obj-vax -T /usr/tools -m vax tools
Note that in this case, "vax" is the same for both machine and architecture. If running m68k, you'd select a machine (-m) name such as sun3, netx68k, mac68k or amiga, not m68k, but the path would end up being /usr/tools/m68k--netbsdelf/bin/.
A little while later you'll have everything you need in /usr/tools/. If you look in /usr/tools/bin/, you'll see vax--netbsdelf-c++, vax--netbsdelf-gcc and friends, which are all (in our example) StrongARM binaries which target VAX. Building the VAX toolchain on the PogoPlug took about two and a quarter hours, and on an eight core AMD about five minutes.
Many online guides talk about symlinking stuff to the distcc binary or creating a wrapper script, but those methods are confusing and don't seem best for running a different architecture's toolchain while still calling the compilers with their standard names. Instead, to make things as simple as possible I created a /usr/tools/vax/bin/ directory with four shell scripts named cc, gcc, c++ and g++. Sure, we could use one script for all four, but distcc only ever uses these four programs remotely so let's just make /usr/tools/vax/bin/ and make four scripts. Note that there is no vax--netbsdelf-cc.
cc:
#!/bin/sh exec /usr/tools/bin/vax--netbsdelf-gcc "$@"
gcc:
#!/bin/sh exec /usr/tools/bin/vax--netbsdelf-gcc "$@"
c++:
#!/bin/sh exec /usr/tools/bin/vax--netbsdelf-c++ "$@"
g++:
#!/bin/sh exec /usr/tools/bin/vax--netbsdelf-g++ "$@"
Next, we will make a little shell script which we'll use to launch distccd on modern with the proper path. It looks like this:
#!/bin/csh setenv PATH "/usr/tools/vax/bin:$PATH" distccd --allow 10.12.26.0/24 --listen 10.12.26.189 --port 3632 \ --daemon --jobs 2 --nice 0 \ --stats --stats-port 3633 \ --log-file=/var/log/distccd/vax.log
The options in order are:
--allow 10.12.26.0/24: Accept connections from this subnet.
--listen 10.12.26.189: Listen on this specific address.
--port 3632: This is the default port. Increment by two for each architecture if you want to use one device for multiple architectures.
--daemon: Should be obvious.
--jobs 2: Sets the number of simultaneous jobs distcc will accept. Documentations suggest number of CPUs plus two, but since we're not running more than one job at a time I selected two so that there will always be a distcc instance waiting even when closing up a previous instance.
--nice 0: Sets the niceness of distcc tasks. We don't want them to be nice. Default niceness is 5.
--stats: Allow querying of distcc statistics. Visit http://10.12.26.189:3633/ to see stats. Note that in order to see stats, the querying IP must be included in the allow statement above.
--stats-port 3633: This is the default port for the http daemon.
--log-file=/var/log/distccd/vax.log: Should be obvious.
The distcc package requires the addition of the distcc user and group, so the log must be writable by one or the other (mkdir /var/log/distccd ; chgrp distcc /var/log/distccd ; chmod g+w /var/log/distccd).
Go ahead and run the script to start the daemons. You should see three distccd processes. If you plan to support more than one architecture, follow the directions above for additional architectures and increment the port numbers by two for each architecture. If you're supporting multiple retrocomputers of the same architecture, increase the number of allowed simultaneous jobs above.
Since you started the compile of devel/distcc on retro before compiling the toolchain on modern, it should be done by now. You did start that first, right? Using distcc on retro is as simple as adding these lines to /etc/mk.conf:
PKGSRC_COMPILER=distcc gcc MAKE_JOBS=1 DISTCC_HOSTS=10.12.26.189:3632
If your machine isn't quite as slow as a VAXstation or an m68030, you may want to use the following:
PKGSRC_COMPILER=distcc gcc MAKE_JOBS=2 DISTCC_HOSTS=10.12.26.189:3632 localhost
This will, for packages which can run parallel jobs, split the work between retro and modern machines. The reason I'm not recommending this as the default is that really slow machines will spend all their time just preparing for compiles and will be slowed by compiling anything at all by themselves.
Now that the toolchain is built, distccd is running on modern and /etc/mk.conf has been edited on retro, we're ready to compile!
To test the speed improvement, I did a small test with shells/tcsh. First I ran a "make fetch" ahead of time for the shells/tcsh package on this VAXstation 4000/60. Then I ran "time make", both with and without distcc.
Native compiling with no assistance:
4225.653u 958.195s 1:29:51.07 96.1%0+0k 63+2055io 143pf+0w
With distcc:
2196.750u 964.451s 59:58.47 87.8%0+0k 0+2376io 48pf+0w
This shows an almost halving of the time! Just to see what kind of a difference a faster compile system or running multiple jobs simultaneously would have, I tried with -j 2 and -j 4, and the differences in speed were less than 2%. Compiling on a 4 GHz AMD system versus an 800 MHz PogoPlug showed no noticeable differences at all, which suggests that the bottlenecks are all in the VAX.
Enjoy!
(Please note that there are formatting issues which will be fixed when I get to know this site better. I'll also add pictures, soon, too. Thanks!)