Build your own FreeBSD update server

Posted on Fri 30 January 2015 in FreeBSD

FreeBSD is an operating system that we use to power various things at work. Its most significant advantage lays in the advanced filesystem ZFS and it's proven stability. As of this writing we run 39 servers with FreeBSD and we update them regularly. While normal patches are quite fast to fetch and install, the upgrade from version 9.2 to 9.3 took a significant amount of time to fetch all the necessary files from the official servers.

That is why we started to run our own internal FreeBSD update server. There exists some documentation for this inside the official handbook which formed the basis for this blog post.


Important to know is that the update server has to run a newer version than the version you are trying to distribute. Thus I first upgraded the update server machine to 9.3 via the freebsd-update -r 9.3-RELEASE upgrade command as described in point 24.2.3 in the handbook.

When running on the newest version, proceed with checking out the SVN repository that holds all the configuration and scripts for the update server via:

cd /usr/local/
svn co freebsd-update-server

I suggest to use /usr/local as this is the 'standard' path of most scripts within the repository.

Building update packages

For upgrading a system from 9.2 to 9.3 we need both versions available via our update server. This is necessary since the system has to determine the difference between two versions. For this we have to create the amd64 directory in the scripts folder and copy over the build.conf from the i386 folder:

cd /usr/local/freebsd-update-server/scripts/9.2-RELEASE
mkdir amd64
cp i386/build.conf amd64/build.conf

Next we edit the amd64/build.conf file and adjust the SHA256 hash and the EOL date in it to reflect the amd64 version. Also we need to point it towards the FTP server where to fetch the ISO image of the release from. The final file should look somewhat like this:

# SHA256 hash of disc1.iso image.
export RELH=a8c1751b83646530148766618a89a97009e7500e7057a5cbe3afd74ef480c915

# Components of the world, source, and kernels
export WORLDPARTS="base doc games"
export SOURCEPARTS="src"
export KERNELPARTS="kernel"

# EOL date
export EOL=1419944400

# Location from which to fetch releases
export FTP=

FreeBSD upgrades are signed by the update server. This is done via a key that is generated via the first run of the make script, it will ask you for a password to protect your key. Note that password down somewhere safe, you will need it to sign all updates in the future:

cd /usr/local/freebsd-update-server
sh scripts/

Important! The command finishes with the output of the public key fingerprint. Note this down somewhere as well, you will need this fingerprint on every client so that it can verify the packages from the update server later on:

Public key fingerprint:

The next step is to run our initial build. This can take some time thus I recommend starting a tmux (or screen if you like) session first:

tmux new -s upgrade_server
cd /usr/local/freebsd-update-server

After that we can start our first build:

sh scripts/ amd64 9.2-RELEASE

This process can take up to several hours, depending on the hardware it is run on. It finishes with the following output:

FreeBSD/amd64 9.2-RELEASE initialization build complete.  Please
review the list of build stamps printed above to confirm that
they look sensible, then run
# sh -e amd64 9.2-RELEASE
to sign the release.

First, we mount our signing key with the password that we encrypted it with, then we sign our release:

sh -e scripts/
sh -e scripts/ amd64 9.2-RELEASE

This finishes with:

The FreeBSD/amd64 9.2-RELEASE update build has been signed and is
ready to be uploaded.  Remember to run
# sh -e
to unmount the decrypted key once you have finished signing all
the new builds.

Were done! No - actually we now have to build our 9.3-RELEASE upgrade files. We will copy over the configuration folder from 9.2 and adjust it to our needs:

cp -r /usr/local/freebsd-update-server/scripts/9.2-RELEASE /usr/local/freebsd-update-server/scripts/9.3-RELEASE

Next, as we want to build and distribute only an amd64 version, we delete the i386 folder:

cd /usr/local/freebsd-update-server/scripts/9.3-RELEASE
rm -rf i386

Let's adjust the amd64/build.conf file for 9.3:

# SHA256 hash of disc1.iso image.
export RELH=5a3c82653d77bba7d7ded8bd7efbedc09d52cf4045d98ce52a82c9e0f8fa9b0e

# Components of the world, source, and kernels
export WORLDPARTS="base doc games"
export SOURCEPARTS="src"
export KERNELPARTS="kernel"

# EOL date
export EOL=1483142400

# Location from which to fetch releases
export FTP=

And fire off our build of the 9.3-RELEASE:

cd /usr/local/freebsd-update-server
sh scripts/ amd64 9.3-RELEASE

This will complete with:

FreeBSD/amd64 9.3-RELEASE initialization build complete.  Please
review the list of build stamps printed above to confirm that
they look sensible, then run
# sh -e amd64 9.3-RELEASE

To sign the release run the approve script again (if you unmounted the key in the meantime don't forget to mount it again via sh -e scripts/

sh -e scripts/ amd64 9.3-RELEASE

Building newest patches

After having build our initial version it is time to bring both to their newest level. The 17 as a parameter is the desired patchlevel in the following command:

sh scripts/ amd64 9.2-RELEASE 17

This will conclude with:

FreeBSD/amd64 9.2-RELEASE update build complete.  Please review
the list of build stamps printed above and the list of updated
files to confirm that they look sensible, then run
# sh -e amd64 9.2-RELEASE
to sign the build.

Next step is to approve the 9.2 build followed by starting the build of the newest upgrade patches for 9.3. At the moment (April 2015) the patchlevel for this version is 13, but you should check here for any new security advisory that announces a new patchlevel. To use the newest patchlevel currently checked in to the SVN look into the folder /usr/local/freebsd-update-server/patches/9.3-RELEASE. The first number of a patchfile tells you its patchlevel. In my case the highest number is 13 and therefore this is the highest patchlevel currently available via the SVN.

Approving the patches for 9.2 and starting the differential build is done via:

sh -e scripts/ amd64 9.2-RELEASE
sh scripts/ amd64 9.3-RELEASE 13

The last step is to approve the 9.3 patches and build the upgrade patches between 9.2 and 9.3 via:

sh -e scripts/ amd64 9.3-RELEASE
sh tools/ pub amd64 9.3-RELEASE 9.2-RELEASE

This command should finish without any output. Finally we can umount our signing key via:

sh scripts/

We now have all necessary files to upgrade a 9.2 system to a 9.3 system. To distribute these we will use the webserver nginx.

Installing and configuring nginx

Installing nginx via the pkg package manager starts with:

pkg install nginx

To enable it add

nginx_enable="YES" to

to the \etc\rc.conf file. The standard configuration leaves the server open to the public, thus we will adjust the nginx configuration under \usr\local\etc\nginx\nginx.conf, to make the server only available within our internal network. In our case this is the following address:

server {

Also we change the root directory of nginx to point to the update files we just built:

root /usr/local/freebsd-update-server/pub

The last step is to start it via:

service nginx start

Configuring the clients

We now have a working update server that is able to serve our clients, but they are not yet aware of the fast machine in their neighbourhood. To change this we have to adjust the \etc\freebsd-update.conf:

# Trusted keyprint.  Changing this is a Bad Idea unless you've received
# a PGP-signed email from <[email protected]> telling you to
# change it and explaining why.
# KeyPrint 800651ef4b4c71c27e60786d7b487188970f4b4169cc055784e21eb71d410cc5
KeyPrint 5552a36eb246be88412c56eb6ee7edccab50ec9b2fe18c90767853e90ad52a0a

# Server or server pool from which to fetch updates.  You can change
# this to point at a specific server if you want, but in most cases
# using a "nearby" server won't provide a measurable improvement in
# performance.
# ServerName

As you can see we used the KeyPrint that the script generated for us. You can of course use the DNS name instead of the IP as the ServerName, but be sure to adjust the /etc/hosts file on each client so that it can resolve local names.