Dockerising apt-cacher-ng

Docker's own example how to dockerise apt-cacher-ng is spectacularly simplified to avoid demonstrating Docker's pitfalls and challenges.

As per our evaluation of Docker one of the first services we dockerised was apt-cacher-ng, partially because official example seemed so straightforward. In this article we describe some deployment challenges that are not covered by Docker's apt-cacher-ng dockerising instructions.

First challenge is to figure out data storage model.

Docker containers are volatile therefore persistent data should be stored either in special data container or on the host machine. We have chosen the latter because we could use shared network folder for data and therefore move container between servers. Storing data on host requires to deal with synchronisation between host's and container's user id for ownership of data.

Docker have no native support for shared volume's uid configuration.

It is not hard to choose user (g)id on host machine and update apt-cacher-ng user's (g)id in container but it is an effort without clear instructions regarding the best way to do so. We ended up running container with parameters -v ${HOSTDATA}/cache:/var/cache/apt-cacher-ng -v ${HOSTDATA}/log:/var/log/apt-cacher-ng to place apt-cacher-ng data and logs to location on host.

Second challenge is to make sure that container runs cron.

Apt-cacher-ng needs periodic maintenance of its data and Debian package installs corresponding daily cron job. Probably we could create a separate container specifically for cron job but that would be crazy because of added complexity. Running cron inside container can be useful for services that use logrotate to manage their logs. So we need container that runs at least two daemons: apt-cacher-ng and cron. Unfortunately Docker is well suited to run only one process so we need workaround to run dockerised container/mini-VM with multiple daemons. Basically we need Docker friendly init system that can propagate signals and terminate all children processes when container is stopping. There is a project to do exactly that but unfortunately it is based on Ubuntu and uses 3rd party software therefore we have decided to try our own solution using only software from Debian repositories. Neither Systemd nor SysV init systems work in Docker so we tried runit which worked fairly well except that we had to re-write daemons' init scripts from scratch -- at this point we appreciated package maintainers work which makes it so easy to deploy services by installing OS packages.
Instead of runit we could use supervisor which uses slightly more memory.

Third challenge is to configure container to use static IP.

There are number of reasons for that. Primarily we wanted to minimise changes to our infrastructure. Since we have dedicated IP for apt-cacher-ng it does not matter where we start VM which claims this IP. We need no DNS updates or load balancers.
But the real problem with Dockerised apt-cacher-ng is that it needs to know its publicly accessible IP address. Not only it is advertised on web page but also apt-cacher-ng announces its services through Avahi for autodiscovery. If package "squid-deb-proxy-client" is installed Apt will automatically find and use apt-cacher-ng services in local network provided that "avahi-daemon" runs on the same host with apt-cacher-ng. Avahi-daemon depends on dbus and they are all should be running on the machine with public IP.
Docker do not support assignment of static IPs.
When container is started Docker chooses random IP for container and overriding that requires a bit of hacking. First thing we tried is to map IP using docker run argument as follows: --net=bridge -p 192.168.0.249:3142:3142. It almost worked but not quite as daemons have to know their public IPs to work properly. After a while we figured out that we can configure docker daemon to start with DOCKER_OPTS="--bridge=br0 --ip-masq=false --iptables=false" to minimise Docker's interference with network configuration (yeah, we had to set-up network bridge on host). Docker still assigns random IP to container on start but we could override this by the following configuration of /etc/network/interfaces inside container:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    pre-up ip addr flush dev eth0
    address 192.168.0.249
    netmask 255.255.255.0
    gateway 192.168.0.1

pre-up ip addr flush dev eth0 is necessary to dismiss Docker-assigned IP. To make it work container entry script should begin with /etc/init.d/networking start and container should be started with --cap-add=NET_ADMIN --net=bridge. Finally from container entry script we had to re-create Docker-generated /etc/hosts file to remove references to Docker-assigned IP. This configuration avoids collisions with existing LAN IPs (see docker bug 11199) and uses static IP assigned on container's build time. Such network configuration can be trivially modified to use DHCP instead of static IP.

Resulting container runs four daemons (apt-cacher-ng, cron, dbus and avahi-daemon) and claims its own IP. All daemons except cron are running without root privileges.

Probably it would be easier to set up networking if we were using rkt container runtime but the latter is not available in Debian yet.
If you need rkt in Debian then please consider sponsoring our work.

Conclusion

Docker introduces multiple problems that did not exist before Docker.

Docker adds significant complexity to application deployment. Where deploying applications with packages takes minutes, dokerisation takes hours or even days of effort and experimentation.

Docker is young and there is no established industry best practice how to manage dockerised applications.

runit "run" scripts

cron:

exec cron -f -l

apt-cacher-ng:

mkdir /var/run/apt-cacher-ng/
chown -c apt-cacher-ng /var/run/apt-cacher-ng/
exec chpst -u apt-cacher-ng /usr/sbin/apt-cacher-ng SocketPath=/var/run/apt-cacher-ng/socket -c /etc/apt-cacher-ng ForeGround=1

avahi-daemon:

sv start dbus || exit 1
exec /usr/sbin/avahi-daemon --no-rlimits

We had to add "--no-rlimits" because for unknown reason avahi-daemon fails to start on some (seemingly identically configured) machines as follows:

chroot.c: fork() failed: Resource temporarily unavailable
failed to start chroot() helper daemon.

This is not a rlimit-nproc < 3 problem.

dbus:

[ -d "/var/run/dbus" ] || mkdir /var/run/dbus >>/dev/null
rm -fv /var/run/dbus/* >>/dev/null
chown -R messagebus:messagebus /var/run/dbus
dbus-uuidgen --ensure
exec dbus-daemon --system --nofork

Tags: