Setting up a network buildd with pbuilder ... continued

Last year I'd described my setup of a local network-buildd using pbuilder, ccache, inoticoming and NFS. One then-still-open goal was to support different Debian releases. This is especially necessary for backports of e.g. bluefish. The rules to contribute backports require to include all changelog entries since the last version on debian-backports or since stable if it's the first version in an uploaded package. Therefor one needs to know the last version in e.g. wheezy-backports. Because I'm not typing the command (the source package only gets uploaded and inoticoming starts the build process) I was looking for a way to automatically retrieve that version and add the relevant -vX.Y-Z switch to dpkg-buildpackage.

The solution I found requires aptitude and a sources.list entry for the relevant release. If you are only interested in the solution, just jump to the end :)

I'm going to add the version switch to the DEBBUILDOPTS variable of pbuilder. In my setup I have a common (shared) snippet called /etc/pbuilderrc.inc and one configuration file per release and architecture, say /etc/pbuilderrc.amd64.stable. Now the first already contains ...

DEBBUILDOPTS="-us -uc -j2"

... and DEBBUILDOPTS can be extended in the latter:

DEBBUILDOPTS+="..."

Because the config file is parsed pretty early in the process the package name has not yet been assigned to any variable. The last argument to pbuilder is the .dsc file. So I use the last argument and parse the file to retrieve the source package name.

cat ${@: -1} | grep -e ^Source | awk -F\  '{ print $2 }'

The solution above works, because pbuilder is a BASH script. Otherwise it maybe needs some tweaking. I use the source package name, because it is unique and there is just one :) Now with this name I check for all versions in wheezy* and stable* and sort them. The sort order of aptitude is from low to high, so the last line shopuld contain the highest version. This covers the possibility that there has not yet been a backport or that there is one:

aptitude versions -F '%p' --show-package-names=never --group-by=none --sort=version \
   "?narrow(?source-package(^PACKAGE\$), ?or(?archive(^wheezy.*), ?archive(^stable.*)))" |\
   tail -n 1 | sed -e 's#~bpo.*$##g'

The sed-part is necessary because otherwise dh_genchanges will add a superfluous changelog entry (the last one of the last upload). To make things easier, I assign the name and version to variables. So this is the complete solution:

[..]
MYPACKAGE="`cat ${@: -1} | grep -e ^Source | awk -F\  '{ print $2 }'`"
MYBPOVERS="`aptitude versions -F '%p' --show-package-names=never --group-by=none --sort=version "?narrow(?source-package(^$MYPACKAGE\$), ?or(?archive(^wheezy.*), ?archive(^stable.*)))" | tail -n 1 | sed -e 's#~bpo.*$##g'`"

log "I: Package is $MYPACKAGE and last stable/bpo version is $MYBPOVERS"

DEBBUILDOPTS+=" -v$MYBPOVERS"
[..]

Examples

I've recently built a new bluefish backport. The last backport version is 2.2.6-1~bpo70+1. There is also the stable version 2.2.3-4. So the version I need is 2.2.6-1 (2.2.6-1~bpo70+1 < 2.2.6-1!!). Checking the log it works:

I: Package is bluefish and last stable/bpo version is 2.2.6-1

A different example is rsync. I've recently locally rebuilt it for a stable system (wanted to make use of the --chown switch). There is not yet a backport. So the version I (would) need is 3.0.9-4. Checking the logs again and works too:

I: Package is rsync and last stable/bpo version is 3.0.9-4

Feedback appreciated ...

Links

• aptitude: Search term reference
• aptitude: Customizing the package list

Setting up a network buildd with pbuilder, ccache, inoticoming and NFS

I recently bought an N54L HP microserver, which shall act as a build-daemon (buildd) for Debian packaging. The idea of my workflow is the following:

1. do the maintenance tasks locally, usually using svn-buildpackage, quilt etc.pp
2. build a source package locally
3. upload the source to an incoming directory on the server
4. watch this incoming directory and start a build process as soon as a package is uploaded
5. put the result into an accessible directory
6. support different vendors (debian, ubuntu)
7. support different releases (experimental, unstable, stable, oldstable)
8. support different architectures (amd64, i386)
9. speed up the build process

Setting up the building environment

My first idea was to use sbuild as described in several places. However I went with pbuilder for a long time having a setup I personally like. So I decided to stay with it. I reduce the time and work to update the configure files by using

• one central file /etc/pbuilderrc.inc defining all the common/shared settings and
• small configure files defining vendor, release, architecture and mirrors

Here are the relevant parts of my central configure snippet:

[..]
EXTRAPACKAGES="apt-utils debconf debconf-utils libfile-fcntllock-perl"
[..]
BASETGZ="/var/cache/pbuilder/base.$DISTRIBUTION.$ARCHITECTURE.tgz"
APTCACHE="/var/cache/pbuilder/aptcache/$DISTRIBUTION/$ARCHITECTURE/"
BUILDRESULT="/var/cache/pbuilder/result/$DISTRIBUTION/$ARCHITECTURE/"
[..]
case "$VENDOR" in
ubuntu)
MIRRORSITE="http://archive.ubuntu.com/ubuntu"
COMPONENTS="main restricted universe multiverse"
OTHERMIRROR="\
deb http://de.archive.ubuntu.com/ubuntu $DISTRIBUTION main restricted universe multiverse|\
deb http://archive.ubuntu.com/ubuntu $DISTRIBUTION-updates main restricted universe multiverse|\
deb http://security.ubuntu.com/ubuntu $DISTRIBUTION-security main restricted universe multiverse|\
"
;;
*)
MIRRORSITE="http://ftp.debian.org/debian"
COMPONENTS="main contrib non-free"
OTHERMIRROR="\
deb http://ftp.de.debian.org/debian $DISTRIBUTION main contrib non-free|\
"
;;
esac
[..]
DEBBUILDOPTS="-us -uc -j2"
AUTO_DEBSIGN=no
USE_PDEBUILD_INTERNAL=yes
[..]

You get the main idea, right? Create own files and directories for each architecture, release and vendor. Further I only want to pull build related packages into the build environment, not into the server environment. So I enabled USE_PDEBUILD_INTERNAL. I don't want to sign anything on the buildd itself and use both cores. That's what DEBBUILDOPTS and AUTO_DEBSIGN are used for. Further I saw some warnings reported during build processes. I therefor install the packages mentioned in EXTRAPACKAGES to every build enviroment. The following are the configure files for the unstable amd64 (default) and the stable i386 build environments:

VENDOR="debian"
DISTRIBUTION="sid"
ARCHITECTURE="amd64"

. /etc/pbuilderrc.inc

VENDOR="debian"
DISTRIBUTION="wheezy"
ARCHITECTURE="i386"

. /etc/pbuilderrc.inc

OTHERMIRROR+="\
deb http://security.debian.org $DISTRIBUTION/updates main contrib non-free|\
"

Now to not having to update all of these environments manually there is a small parallelized script /usr/local/sbin/update-pbuilder-chroots:

#!/bin/sh

t=$(tempfile -p .upc. -s .list) || exit 1

cat > $t << EOF
/etc/pbuilderrc.amd64.sid
/etc/pbuilderrc.i386.sid
/etc/pbuilderrc.amd64.wheezy
/etc/pbuilderrc.i386.wheezy
/etc/pbuilderrc.amd64.squeeze
/etc/pbuilderrc.i386.squeeze
EOF

parallel /usr/sbin/pbuilder update --override-config --configfile :::: $t | tee -a /var/log/pbuilder.log

rm -f $t

exit 0

The script will make sure, that every change to the configuration file(s) will be considered (--override-config) and is run once a week via cron. The default build environment is updated on a daily base without this restriction:

@reboot root    rm -rf /var/cache/pbuilder/build/*/ >> /dev/null 2>&1
@daily  root    test -x /usr/sbin/pbuilder && /usr/sbin/pbuilder update --configfile /etc/pbuilderrc.amd64.sid | tee -a /var/log/pbuilder.log >> /dev/null 2>&1
@weekly root    test -x /usr/local/sbin/update-pbuilder-chroots && /usr/local/sbin/update-pbuilder-chroots >> /dev/null 2>&1

Speeding up the build process

Sometimes it is necessary to recompile a package. The tool to speed up this process is ccache, which can be easily integrated into pbuilder by setting CCACHEDIR in /etc/pbuilderrc.inc. It acts like a cache for compiled files and avoids recompilation in several buildd runs.

CCACHEDIR=/var/cache/pbuilder/ccache

It is also possible to use a TMPFS for BUILDPLACE to speed up the build process itself. In this case APTCACHEHARDLINK must be disabled. Well, yeah, I don't do this.

Watch an incoming directory and start the build process

Not much to say about this except that the user nobody is used to run the pbuilder command. Therefor it is necessary to mention this in /etc/sudoers to allow him to do this via sudo command:

Defaults        env_keep = "DH_OPTIONS DH_VERBOSE DEB_BUILD_OPTIONS DH_OPTIONS MAKEFLAGS CC CXX CPP F77 F90"
[..]
nobody          ALL=NOPASSWD:/usr/sbin/pbuilder

The first line is only necessary, if a pbuilder command is started with one of these environmental variables set. This can be useful when e.g. debugging a build failure with a future gcc version that is not yet the default or when debugging a clang build issue or when creating non-stripped binary packages for creating a backtrace (BTW: When there is a common repository of all debugging symbols created?).

Now to start the build process, I created an incoming directory with mods 777 at /var/cache/pbuilder/incoming/ and use inoticoming to watch it. This is started during boot via cron. The following shows the entry for the default unstable amd64 build environment:

@reboot root    /usr/bin/inoticoming --logfile /var/log/inoticoming.log --pid-file /var/run/inoticoming.pid /var/cache/pbuilder/incoming --chdir /var/cache/pbuilder/incoming --suffix .dsc --stderr-to-log --stdout-to-log /bin/su -c '/usr/bin/sudo /usr/sbin/pbuilder build --autocleanaptcache {}' nobody \;

Done. That's how the basic buildd works.

Getting the results

The pbuilder BUILDRESULT directory with mods 777 is shared via NFS mount to the workstation. After successfully building the package, debsign and dput can be used as usual.

Setting up the workstation

I use svn-buildpackage for most of my packages. Because pbuilder won't download source tarballs on demand by default (maybe via hook?), the tarball must always be included building the source package, which then gets uploaded via dput to the buildd. This is considered in the command listed in svn-builder in ~/.svn-buildpackage.conf (note, below shows my personal layout):

svn-builder=debuild --post-dpkg-buildpackage-hook=\"dput -f buildd /usr/local/src/packages/$PACKAGE/%p_%s_source.changes\" --no-lintian -d -sa -us -uc -S
svn-override=origDir=/usr/local/src/packages/$PACKAGE
svn-override=buildArea=/usr/local/src/packages/$PACKAGE
[..]

The host buildd has been added to ~/.dput.cf:

[buildd]
fqdn = mybuildd
login = mylogin
method = scp
incoming = /var/cache/pbuilder/incoming
allow_unsigned_uploads = 1
run_lintian = 0

Done. Now the svn-buildpackage command will create a source package including the source tarball and upload it to the buildd, where the package building starts right after the upload.

Conclusions

This is the main setup which works nicely for the default build environment so far. On the TODO list is to extend this setup so I can easily deal with all the architectures and releases (point 6..8 at the beginning).

Update 17.02.2015

The original article suggested to use the %v variable in the hook to transfer the _sources.changes file to the build-daemon. This will fail if the source version e.g. is X:Y-Z in which case the file created is package_Y-Z_source.changes but the command to execute is:

dput -f buildd package_X:Y-Z_source.changes

and thus will fail. According to dpkg-buildpackage(1) the hook command also accepts a %s variable, which seems to extract the superflous version characters.

Re: Making pbuilder just that little bit faster

It is true that making /var/cache/pbuilder/build a tmpfs makes a noticeable speed difference. But it seems you cannot cache the downloaded debs: ln: creating hard link ...: Invalid cross-device link.