Build your own FreeBSD update server
Posted on Fri 30 January 2015 in FreeBSD
FreeBSD is an operating system that we use to power various things at work. Its most significant advantage lays in the advanced filesystem ZFS and it's proven stability. As of this writing we run 39 servers with FreeBSD and we update them regularly. While normal patches are quite fast to fetch and install, the upgrade from version 9.2 to 9.3 took a significant amount of time to fetch all the necessary files from the official servers.
That is why we started to run our own internal FreeBSD update server. There exists some documentation for this inside the official handbook which formed the basis for this blog post.
Prerequisites
Important to know is that the update server has to run a newer version than the
version you are trying to distribute. Thus I first upgraded the update server
machine to 9.3 via the freebsd-update -r 9.3-RELEASE upgrade command as
described in point 24.2.3 in the handbook.
When running on the newest version, proceed with checking out the SVN repository that holds all the configuration and scripts for the update server via:
cd /usr/local/
svn co http://svn.freebsd.org/base/user/cperciva/freebsd-update-build freebsd-update-server
I suggest to use /usr/local as this is the 'standard' path of most scripts
within the repository.
Building update packages
For upgrading a system from 9.2 to 9.3 we need both versions available via our
update server. This is necessary since the system has to determine the difference
between two versions. For this we have to create the amd64 directory in the scripts
folder and copy over the build.conf from the i386 folder:
cd /usr/local/freebsd-update-server/scripts/9.2-RELEASE
mkdir amd64
cp i386/build.conf amd64/build.conf
Next we edit the amd64/build.conf file and adjust
the SHA256 hash and the EOL date in it to reflect the amd64 version. Also we need to
point it towards the FTP server where to fetch the ISO image of the release from.
The final file should look somewhat like this:
# SHA256 hash of disc1.iso image.
export RELH=a8c1751b83646530148766618a89a97009e7500e7057a5cbe3afd74ef480c915
# Components of the world, source, and kernels
export WORLDPARTS="base doc games"
export SOURCEPARTS="src"
export KERNELPARTS="kernel"
# EOL date
export EOL=1419944400
# Location from which to fetch releases
export FTP=ftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/9.2/
FreeBSD upgrades are signed by the update server. This is done via a key that is generated via the first run of the make script, it will ask you for a password to protect your key. Note that password down somewhere safe, you will need it to sign all updates in the future:
cd /usr/local/freebsd-update-server
sh scripts/make.sh
Important! The command finishes with the output of the public key fingerprint. Note this down somewhere as well, you will need this fingerprint on every client so that it can verify the packages from the update server later on:
Public key fingerprint:
5552a36eb246be88412c56eb6ee7edccab50ec9b2fe18c90767853e90ad52a0a
The next step is to run our initial build. This can take some time thus I recommend starting a tmux (or screen if you like) session first:
tmux new -s upgrade_server
cd /usr/local/freebsd-update-server
After that we can start our first build:
sh scripts/init.sh amd64 9.2-RELEASE
This process can take up to several hours, depending on the hardware it is run on. It finishes with the following output:
FreeBSD/amd64 9.2-RELEASE initialization build complete. Please
review the list of build stamps printed above to confirm that
they look sensible, then run
# sh -e approve.sh amd64 9.2-RELEASE
to sign the release.
First, we mount our signing key with the password that we encrypted it with, then we sign our release:
sh -e scripts/mountkey.sh
sh -e scripts/approve.sh amd64 9.2-RELEASE
This finishes with:
The FreeBSD/amd64 9.2-RELEASE update build has been signed and is
ready to be uploaded. Remember to run
# sh -e umountkey.sh
to unmount the decrypted key once you have finished signing all
the new builds.
Were done! No - actually we now have to build our 9.3-RELEASE upgrade files.
We will copy over the configuration folder from 9.2 and adjust it to our needs:
cp -r /usr/local/freebsd-update-server/scripts/9.2-RELEASE /usr/local/freebsd-update-server/scripts/9.3-RELEASE
Next, as we want to build and distribute only an amd64 version, we delete the i386 folder:
cd /usr/local/freebsd-update-server/scripts/9.3-RELEASE
rm -rf i386
Let's adjust the amd64/build.conf file for 9.3:
# SHA256 hash of disc1.iso image.
export RELH=5a3c82653d77bba7d7ded8bd7efbedc09d52cf4045d98ce52a82c9e0f8fa9b0e
# Components of the world, source, and kernels
export WORLDPARTS="base doc games"
export SOURCEPARTS="src"
export KERNELPARTS="kernel"
# EOL date
export EOL=1483142400
# Location from which to fetch releases
export FTP=ftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/9.3/
And fire off our build of the 9.3-RELEASE:
cd /usr/local/freebsd-update-server
sh scripts/init.sh amd64 9.3-RELEASE
This will complete with:
FreeBSD/amd64 9.3-RELEASE initialization build complete. Please
review the list of build stamps printed above to confirm that
they look sensible, then run
# sh -e approve.sh amd64 9.3-RELEASE
To sign the release run the approve script again (if you unmounted the
key in the meantime don't forget to mount it again via sh -e scripts/mountkey.sh:
sh -e scripts/approve.sh amd64 9.3-RELEASE
Building newest patches
After having build our initial version it is time to bring both to their newest level.
The 17 as a parameter is the desired patchlevel in the following command:
sh scripts/diff.sh amd64 9.2-RELEASE 17
This will conclude with:
FreeBSD/amd64 9.2-RELEASE update build complete. Please review
the list of build stamps printed above and the list of updated
files to confirm that they look sensible, then run
# sh -e approve.sh amd64 9.2-RELEASE
to sign the build.
Next step is to approve the 9.2 build followed by starting the build of the newest
upgrade patches for 9.3. At the moment (April 2015) the patchlevel for this version is 13,
but you should check here
for any new security advisory that announces a new patchlevel. To use the newest
patchlevel currently checked in to the SVN look into the folder
/usr/local/freebsd-update-server/patches/9.3-RELEASE.
The first number of a patchfile tells you its patchlevel. In my case the highest number is
13 and therefore this is the highest patchlevel currently available via the SVN.
Approving the patches for 9.2 and starting the differential build is done via:
sh -e scripts/approve.sh amd64 9.2-RELEASE
sh scripts/diff.sh amd64 9.3-RELEASE 13
The last step is to approve the 9.3 patches and build the upgrade patches between 9.2 and 9.3 via:
sh -e scripts/approve.sh amd64 9.3-RELEASE
sh tools/build-upgrade-patches.sh pub amd64 9.3-RELEASE 9.2-RELEASE
This command should finish without any output. Finally we can umount our signing key via:
sh scripts/umountkey.sh
We now have all necessary files to upgrade a 9.2 system to a 9.3 system. To distribute these we will use the webserver nginx.
Installing and configuring nginx
Installing nginx via the pkg package manager starts with:
pkg install nginx
To enable it add
nginx_enable="YES" to
to the \etc\rc.conf file. The standard configuration leaves the server
open to the public, thus we will adjust the nginx configuration
under \usr\local\etc\nginx\nginx.conf, to make the server only available
within our internal network. In our case this is the following address:
server {
listen 10.5.128.1:80;
Also we change the root directory of nginx to point to the update files we just built:
root /usr/local/freebsd-update-server/pub
The last step is to start it via:
service nginx start
Configuring the clients
We now have a working update server that is able to serve our clients, but
they are not yet aware of the fast machine in their neighbourhood. To change
this we have to adjust the \etc\freebsd-update.conf:
# Trusted keyprint. Changing this is a Bad Idea unless you've received
# a PGP-signed email from <[email protected]> telling you to
# change it and explaining why.
# KeyPrint 800651ef4b4c71c27e60786d7b487188970f4b4169cc055784e21eb71d410cc5
KeyPrint 5552a36eb246be88412c56eb6ee7edccab50ec9b2fe18c90767853e90ad52a0a
# Server or server pool from which to fetch updates. You can change
# this to point at a specific server if you want, but in most cases
# using a "nearby" server won't provide a measurable improvement in
# performance.
# ServerName update.FreeBSD.org
ServerName 10.5.128.1
As you can see we used the KeyPrint that the make.sh script generated for us.
You can of course use the DNS name instead of the IP as the ServerName,
but be sure to adjust the /etc/hosts file on each client so that it can resolve local
names.