Too big to fail - a Beam Pattern for enriching a Stream using State and Timers

2023-08-01T00:00:00+02:00

The recording of my talk at the Beam Summit 2023, discussing a pattern for enriching streaming data using state and timers, is now available:

Playing the Long Game: Transforming Ricardo's Data Infrastructure with Apache Beam

2022-08-03T00:00:00+02:00

The recording of my talk at the Beam Summit 2022 about building real-time data pipelines, from running our own Apache Flink cluster on-premise to the fully managed GCP Dataflow service, is now available:

Python script to control BI Engine Reservations

2022-05-07T00:00:00+02:00

I wrote a small Python script to be able to create and delete BI Engine reservations for Google Cloud BigQuery. I used this script in a cronjob to disable BI Engine on the weekends/during the nights to save some costs and see how well the system is with and without pre-warmed caches. To run it you need to install the google-cloud-bigquery-reservation library.

Apache Beam Case Study

2021-12-03T14:05:00+01:00

In a case study for Apache Beam, I described how the framework enables Ricardo to evolve into a smarter second-hand marketplace. You can find it on the Apache Beam website.

You belong together - detecting linked accounts at Ricardo

2021-08-06T00:00:00+02:00

The recording of my talk at the Beam Summit 2021 about one of our first production pipelines created with the Python SDK is now available on YouTube:

GCP Zurich Meetup #8

2021-03-16T00:00:00+01:00

At the GCP Meetup #8 in Zurich, I've talked about how the Google BigQuery data warehouse can benefit from the NoSQL database Cloud Bigtable, by using Bigtable as a fast key-based lookup store for point queries:

8 Years of Event Streaming with Apache Kafka

2021-02-02T00:00:00+01:00

A post for the Confluent blog about my personal history with Apache Kafka is availabe here.

Why we chose Bigtable to complement BigQuery

2021-01-14T12:03:00+01:00

I've contributed a blog post for the GCP Blog about our use of Bigtable at Ricardo. You can read it here.

Four Apache Technologies Combined for Fun and Profit

2020-08-26T00:00:00+02:00

The recording of my talk at the Online Beam Summit 2020 is now available on YouTube:

From database dumps to streaming - Ricardo.ch's Beam journey

2019-06-18T00:00:00+02:00

My talk at the Beam Summit Europe 2019 is now available on YouTube:

Taking Data from Somewhere and Put It Somewhere Else

2018-06-14T00:00:00+02:00

I gave a talk at the Tamedia TX conference about how we approached the modernization of the Data Intelligence architecture at Ricardo:

Build your own FreeBSD update server

2015-01-30T16:01:00+01:00

FreeBSD is an operating system that we use to power various things at work. Its most significant advantage lays in the advanced filesystem ZFS and it's proven stability. As of this writing we run 39 servers with FreeBSD and we update them regularly. While normal patches are quite fast to fetch and install, the upgrade from version 9.2 to 9.3 took a significant amount of time to fetch all the necessary files from the official servers.

That is why we started to run our own internal FreeBSD update server. There exists some documentation for this inside the official handbook which formed the basis for this blog post.

Prerequisites

Important to know is that the update server has to run a newer version than the version you are trying to distribute. Thus I first upgraded the update server machine to 9.3 via the freebsd-update -r 9.3-RELEASE upgrade command as described in point 24.2.3 in the handbook.

When running on the newest version, proceed with checking out the SVN repository that holds all the configuration and scripts for the update server via:

cd /usr/local/
svn co http://svn.freebsd.org/base/user/cperciva/freebsd-update-build freebsd-update-server

I suggest to use /usr/local as this is the 'standard' path of most scripts within the repository.

Building update packages

For upgrading a system from 9.2 to 9.3 we need both versions available via our update server. This is necessary since the system has to determine the difference between two versions. For this we have to create the amd64 directory in the scripts folder and copy over the build.conf from the i386 folder:

cd /usr/local/freebsd-update-server/scripts/9.2-RELEASE
mkdir amd64
cp i386/build.conf amd64/build.conf

Next we edit the amd64/build.conf file and adjust the SHA256 hash and the EOL date in it to reflect the amd64 version. Also we need to point it towards the FTP server where to fetch the ISO image of the release from. The final file should look somewhat like this:

# SHA256 hash of disc1.iso image.
export RELH=a8c1751b83646530148766618a89a97009e7500e7057a5cbe3afd74ef480c915

# Components of the world, source, and kernels
export WORLDPARTS="base doc games"
export SOURCEPARTS="src"
export KERNELPARTS="kernel"

# EOL date
export EOL=1419944400

# Location from which to fetch releases
export FTP=ftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/9.2/

FreeBSD upgrades are signed by the update server. This is done via a key that is generated via the first run of the make script, it will ask you for a password to protect your key. Note that password down somewhere safe, you will need it to sign all updates in the future:

cd /usr/local/freebsd-update-server
sh scripts/make.sh

Important! The command finishes with the output of the public key fingerprint. Note this down somewhere as well, you will need this fingerprint on every client so that it can verify the packages from the update server later on:

Public key fingerprint:
5552a36eb246be88412c56eb6ee7edccab50ec9b2fe18c90767853e90ad52a0a

The next step is to run our initial build. This can take some time thus I recommend starting a tmux (or screen if you like) session first:

tmux new -s upgrade_server
cd /usr/local/freebsd-update-server

After that we can start our first build:

sh scripts/init.sh amd64 9.2-RELEASE

This process can take up to several hours, depending on the hardware it is run on. It finishes with the following output:

FreeBSD/amd64 9.2-RELEASE initialization build complete.  Please
review the list of build stamps printed above to confirm that
they look sensible, then run
# sh -e approve.sh amd64 9.2-RELEASE
to sign the release.

First, we mount our signing key with the password that we encrypted it with, then we sign our release:

sh -e scripts/mountkey.sh
sh -e scripts/approve.sh amd64 9.2-RELEASE

This finishes with:

The FreeBSD/amd64 9.2-RELEASE update build has been signed and is
ready to be uploaded.  Remember to run
# sh -e umountkey.sh
to unmount the decrypted key once you have finished signing all
the new builds.

Were done! No - actually we now have to build our 9.3-RELEASE upgrade files. We will copy over the configuration folder from 9.2 and adjust it to our needs:

cp -r /usr/local/freebsd-update-server/scripts/9.2-RELEASE /usr/local/freebsd-update-server/scripts/9.3-RELEASE

Next, as we want to build and distribute only an amd64 version, we delete the i386 folder:

cd /usr/local/freebsd-update-server/scripts/9.3-RELEASE
rm -rf i386

Let's adjust the amd64/build.conf file for 9.3:

# SHA256 hash of disc1.iso image.
export RELH=5a3c82653d77bba7d7ded8bd7efbedc09d52cf4045d98ce52a82c9e0f8fa9b0e

# Components of the world, source, and kernels
export WORLDPARTS="base doc games"
export SOURCEPARTS="src"
export KERNELPARTS="kernel"

# EOL date
export EOL=1483142400

# Location from which to fetch releases
export FTP=ftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/9.3/

And fire off our build of the 9.3-RELEASE:

cd /usr/local/freebsd-update-server
sh scripts/init.sh amd64 9.3-RELEASE

This will complete with:

FreeBSD/amd64 9.3-RELEASE initialization build complete.  Please
review the list of build stamps printed above to confirm that
they look sensible, then run
# sh -e approve.sh amd64 9.3-RELEASE

To sign the release run the approve script again (if you unmounted the key in the meantime don't forget to mount it again via sh -e scripts/mountkey.sh:

sh -e scripts/approve.sh amd64 9.3-RELEASE

Building newest patches

After having build our initial version it is time to bring both to their newest level. The 17 as a parameter is the desired patchlevel in the following command:

sh scripts/diff.sh amd64 9.2-RELEASE 17

This will conclude with:

FreeBSD/amd64 9.2-RELEASE update build complete.  Please review
the list of build stamps printed above and the list of updated
files to confirm that they look sensible, then run
# sh -e approve.sh amd64 9.2-RELEASE
to sign the build.

Next step is to approve the 9.2 build followed by starting the build of the newest upgrade patches for 9.3. At the moment (April 2015) the patchlevel for this version is 13, but you should check here for any new security advisory that announces a new patchlevel. To use the newest patchlevel currently checked in to the SVN look into the folder /usr/local/freebsd-update-server/patches/9.3-RELEASE. The first number of a patchfile tells you its patchlevel. In my case the highest number is 13 and therefore this is the highest patchlevel currently available via the SVN.

Approving the patches for 9.2 and starting the differential build is done via:

sh -e scripts/approve.sh amd64 9.2-RELEASE
sh scripts/diff.sh amd64 9.3-RELEASE 13

The last step is to approve the 9.3 patches and build the upgrade patches between 9.2 and 9.3 via:

sh -e scripts/approve.sh amd64 9.3-RELEASE
sh tools/build-upgrade-patches.sh pub amd64 9.3-RELEASE 9.2-RELEASE

This command should finish without any output. Finally we can umount our signing key via:

sh scripts/umountkey.sh

We now have all necessary files to upgrade a 9.2 system to a 9.3 system. To distribute these we will use the webserver nginx.

Installing and configuring nginx

Installing nginx via the pkg package manager starts with:

pkg install nginx

To enable it add

nginx_enable="YES" to

to the \etc\rc.conf file. The standard configuration leaves the server open to the public, thus we will adjust the nginx configuration under \usr\local\etc\nginx\nginx.conf, to make the server only available within our internal network. In our case this is the following address:

server {
     listen       10.5.128.1:80;

Also we change the root directory of nginx to point to the update files we just built:

root /usr/local/freebsd-update-server/pub

The last step is to start it via:

service nginx start

Configuring the clients

We now have a working update server that is able to serve our clients, but they are not yet aware of the fast machine in their neighbourhood. To change this we have to adjust the \etc\freebsd-update.conf:

# Trusted keyprint.  Changing this is a Bad Idea unless you've received
# a PGP-signed email from <security-officer@FreeBSD.org> telling you to
# change it and explaining why.
# KeyPrint 800651ef4b4c71c27e60786d7b487188970f4b4169cc055784e21eb71d410cc5
KeyPrint 5552a36eb246be88412c56eb6ee7edccab50ec9b2fe18c90767853e90ad52a0a

# Server or server pool from which to fetch updates.  You can change
# this to point at a specific server if you want, but in most cases
# using a "nearby" server won't provide a measurable improvement in
# performance.
# ServerName update.FreeBSD.org
ServerName 10.5.128.1

As you can see we used the KeyPrint that the make.sh script generated for us. You can of course use the DNS name instead of the IP as the ServerName, but be sure to adjust the /etc/hosts file on each client so that it can resolve local names.

Tobias Kaymak"s Blog