Task 1

Infrastructure

PXE-Boot/

A diskless Raspberry Pi cluster: one Pi 5 as the control-plane / boot server, eight Pi 3s that network-boot over PXE with no SD cards at all.

What was asked

Develop and set up an infrastructure of one or more sensor nodes with single-board computers (Raspberry Pi 5), deploying operating systems and object-detection software. Investigate consolidating Pi 3 OS images onto Pi 4/5 and enable PXE boot over LAN for simplified cluster administration.

What was built

Instead of individual SD cards per worker, every Pi 3 fetches its bootloader and mounts its entire root filesystem over the network from the Pi 5:

  1. DHCP discovery — a worker broadcasts a network-boot request; dnsmasq on the Pi 5 answers.
  2. TFTP transfer — the worker downloads its kernel and boot config from the Pi 5's TFTP server.
  3. NFS root mount — the worker mounts its root filesystem (/) from /nfs/<serial> on the Pi 5, read-write, over NFS.

Server-side setup

ServiceRole
dnsmasqDHCP server (leases 10.0.0.10–10.0.0.50) + built-in TFTP server, root at /tftpboot
nfs-kernel-serverexports /tftpboot and one /nfs/<serial> directory per worker
nftables + ip_forwardNAT gateway so workers reach the internet through the Pi 5's Wi-Fi uplink
persistent journaldlogs survive reboot, moved off the default RAM-only journal

Per-worker image provisioning

A single base image is rsync'd (or loop-mounted from an .img file) into /nfs/base_image, then cloned per worker under /nfs/<serial>. Each clone is de-identified so eight Pi 3s booting the "same" image don't collide on the network:

  • cmdline.txt and /etc/fstab rewritten to point at that worker's own NFS root (base_image → <serial>)
  • /etc/hostname and /etc/hosts set to pi-<serial>
  • /etc/machine-id truncated so it regenerates on first boot; the D-Bus symlink to it removed
  • SSH host keys deleted so each worker regenerates its own identity

Raspberry Pi 3s need a one-time, irreversible OTP bit flipped (program_usb_boot_mode=1) to enable network boot at all; Pi 4 and later just need BOOT_ORDER set via raspi-config.

Automated provisioning

A systemd service (pi-provisioner.service) tails the dnsmasq journal for new TFTP boot requests, extracts the requesting board's 8-hex-digit serial number, and — if it's not already provisioned — runs the whole clone / de-identify / export sequence automatically. New Pi 3s can be racked and network-booted with zero manual per-node setup.

Result on real hardware

All eight Pi 3 workers join the network via PXE with no SD cards. This same diskless design is what later drives the storage risk analysis in Task 10 — the honest risk register: because the workers have no physical disk, everything they think is "local storage" is really a directory on the Pi 5's single SSD, reached over NFS.

Known gap, reported in the risk register

The NFS exports use sec=sys (no cryptographic authentication) and a wildcard host list (*) with no_root_squash — required for a diskless worker to boot at all, but it means any host that can reach the Pi 5's NFS port can mount any worker's entire root filesystem read-write as root. Scoped to the isolated 10.0.0.0/24 switch, but nfsd binds all interfaces including Wi-Fi and Tailscale. See Task 10 for the full writeup.

Raspberry Pi PXE Network boot

This directory contains notes, configuration blueprints, and scripts required to set up a diskless (SD-card-free) Preboot Execution Environment (PXE) cluster using Raspberry Pis.

Architecture Overview

Instead of booting from individual SD cards, worker nodes fetch their bootloader via the network from a designated Server Node.

  1. DHCP Discovery: The worker node broadcasts a network boot request.
  2. DHCP Offer: The Server Node serves an IP address.
  3. TFTP Transfer: The worker connects to the Master via Trivial File Transfer Protocol (TFTP) to download the kernel and boot configuration.
  4. NFS Mount: The worker mounts its root file system (/) from a shared directory on the Master Node over Network File System (NFS).

Server Raspberry Pi Setup

We will at first look at setting up our Server Raspberry Pi, where the DHCP server, that will handle the allocation of IP addresses, the NFS server, responsilbe for sharing files located on our Pi, and the TFTP Server, responsible for sending and receiving files across the network, will run.

Install OS & Enable SSH

Use the Raspberry Pi Imager to flash Raspberry Pi OS onto your primary server storage. Ensure SSH is enabled during the customization step.

If SSH needs to be enabled manually later via terminal, use:

sudo raspi-config
# Navigate to: Interfacing Options -> SSH -> Choose "Yes"

Accessing the Pi through SSH can be done through the following commands

ssh [username]@[IP address]
# or
ssh [username]@[hostname].local

Enable persistent logging

By default, the Raspberry Pi will save all logs into RAM under /run/log/journal, which is not retained upon reboot. As we will need to look into error messages in case something goes wrong, we will first change our setup, so the log messages will be saved on our persistent storage. To do that we will have to change our journal.conf file. Normally the file would be saved in /etc/systemd/journald.conf, but for some reason the Raspberry Pi OS stores the needed file in /usr/lib/systemd/journald.conf.d/40-rpi-volatile-storage.conf, that overrides the journald.conf file. Inside that file we change the Storage option to persistent and if needed, add a maximum file size. It should look something like

[Journal]
Storage=persistent
SystemMaxUse=200M

If we want to keep using the /etc/systemd/ file path, we can add a file to a folder /etc/systemd/journald.conf.d/ that has a prefix higher than the 40 that in 40-rpi-volatile-storage.conf. The new file will then be loaded last and overwrites the other file.

sudo mkdir -p /etc/systemd/journald.conf.d/
sudo touch /etc/systemd/journald.conf.d/99-persistent-journal.conf

The file contents are the same as earlier.

After our updates we can restart the journald process

sudo systemctl restart systemd-journald

and if we want to write the logs already written onto RAM into our persistent log, we can use:

sudo journalctl --flush

To see if everything worked, we can use

sudo systemctl status systemd-journald --no-pager | grep -I 'Journal ('

and we will see something like:

image

The directory /var/log/journal/ should have been created and the logs will be saved on the persistent storage.

We will go over the most important functions of the journalctl command:

journalctl [options] [unit]
  • Just using journalctl will display the recent log messages from all units starting from the most recent entries
  • -r: Reverses the log order.
  • -n: Specify a specific number of log entries to be shown.
  • -f: Continuously print new entries when they are appended to the journal.
  • -u: Display logs for a specific systemd unit or service.
  • -p: Filters the output by message priorities.
    • "emerg" (0)
    • "alert" (1)
    • "crit" (2)
    • "err" (3)
    • "warning" (4)
    • "notice" (5)
    • "info" (6)
    • "debug" (7)
  • --list-boots: View information about system boots.
  • -b: Show messages from a specific boot, for the last boot use journalctl -b -1.
  • -g: Filter Message field that matches specified regular expression.
  • -o verbose: shows the full-structured entry items with all fields.

See all options under https://man7.org/linux/man-pages/man1/journalctl.1.html

DHCP Server

We will set up our Raspberry Pi to act as a Dynamic Host Configuration(DHCP) server, that we will use to automate the allocation of IP addresses for our worker Pi's.

A succesful DHCP process has four steps:

  • Discover
    The device that tries to connect to the network, in our case the worker Pi's, broadcasts a message to all devices inside the network to see wether a DHCP server is available.
  • Offer
    The DHCP server replies to the client and offer an available IP address and configuration options
  • Request
    The client condfirms the offer and tells the selected DHCP server to oficially allocate the IP. In case there are multiple DHCP servers, only one of the offers will be accepted.
  • Acknowledgment
    The server sends the acknowledgment after receiving the request and officially assigns the IP address to the client, as well as providing the full set of network configuration parameters

To configure a DHCP server on our Pi, we will use dnsmasq and we can use tcpdump to monitor and analyze packets that pass through our system.

sudo apt install tcpdump dnsmasq
sudo systemctl enable dnsmasq

To start looking for bootpc (Bootstrap protocol client) packages, used to request for IP addresses, on the ethernet port using tcpdump, we can use

sudo tcpdump -i eth0 port bootpc

Adding a client to the network should now result in us seeing a request:

image

Now we need to to update the dnsmasq.conf file, generally located at /etc/dnsmasq.conf. The default file can be quite verbose, if that is a problem, we can use the following command to clear it, or create it if it does not already exist.

echo | sudo tee /etc/dnsmasq.conf

And we replace the contents of dnsmasq.conf with

# Only listen on the Ethernet port (Cluster network)
interface=eth0

log-dhcp

# Provide a dynamic range of IPs for the workers
# They will get IPs from 10.0.0.10 to 10.0.0.50
dhcp-range=10.0.0.10,10.0.0.50,255.255.255.0,12h

# Set the Master (this Pi) as the default gateway/router for the workers
dhcp-option=3,10.0.0.1
dhcp-option=66,10.0.0.1
dhcp-option=67,bootcode.bin

# Enable the built-in TFTP server for network booting
enable-tftp
tftp-root=/tftpboot

# PXE Booting specific for Raspberry Pi
pxe-service=0,"Raspberry Pi Boot"

# Set static IP address for switch
dhcp-host=8c:86:dd:44:82:68,10.0.0.2,Switch

Descriptions of our configurations:

  • interface=eth0:
    This line binds the dnsmasq service to a specific physical or virtual network interface, here eth0. We are only looking at our local private network.
  • log-dhcp: Enables more detailed logging.
  • dhcp-range: Takes four arguments:
    • The start IP for the range: 10.0.0.10
    • The end IP for the range: 10.0.0.50
    • The Subnet Mask, that defines the boundaries of the local network: 255.255.255.0
    • The lease time, meaning the amount of time the IP is lent to a device. After half the time, the lease is renewed.
  • dhcp-option: This option allows us to set specific network parameters to pass to clients when they request for an IP address.
    • 3 DHCP code for the Router/Gateway.
    • 66 DHCP code for TFTP Server Name / Boot Server Host Name.
    • 67 Defines the bootfile name.
  • enable-tftp: This option turns on the built-in TFTP server.
  • tftp-root: Sets the folder directory the TFTP service is allowed to look for files.
  • pxe-service: An option to send a PXE boot service block to any client, that identifies as PXE network-booting device.
    • 0: Represents the architecture type, 0 indicates x86/client architecture.
    • "Raspberry Pi Boot" This is the name of the service option provided to clients.

On newer Raspberry Pi OS versions, NetworkManager is used for network management, which might try to manage eth0 and run internal DHCP tasks, which could strip away our statc IP or it might conflict with dnsmasq. We can disable NetworkManager from trying to do dns tasks by adding the line dns=systemd-resolved to /etc/NetworkManager/NetworkManager.conf. The file contents should look like the following:

[main]
plugins=ifupdown,keyfile
dns=systemd-resolved

[ifupdown]
managed=false

This also means we can use the NetworkManager to set the static IP of our server node. To do that we first need to know the name of our ethernet connection, which we can identify using the command:

nmcli connection show

image

We can see that our name in this instance is netplan-eth0, which we can use to add an IP address to our ethernet port, we will set the IP of our device as 10.0.0.1:

sudo nmcli connection modify "netplan-eth0" ipv4.addresses 10.0.0.1/24 ipv4.method manual
sudo nmcli connection up "netplan-eth0"

To check that our IP address is set as expected we can use

ip a

that returns the addresses on our device:

image

We should see our set IP address, as well as its status, if it is up or down. It is possible that even though we set the connection it might still be down. If that is the case we can use nmclito set the connection to up:

sudo nmcli connection up "netplan-eth0"

Lastly we add reliable DNS servers, in this case Google(8.8.8.8) and Cloudflare(1.1.1.1), to our /etc/resolv.conf, as the NetworkManager might no longer do this automatically.

nameserver 8.8.8.8
nameserver 1.1.1.1

Trivial File Transfer Protocol(TFTP)

The trivial file transfer protocol is a simplified version of the file transfer protocol and is used to send and receive files between devices on a local network. It uses the User Datagram Protocol to send small data packets, called datagrams, without establishing a prior connection. TFTP can be used before the main processor has loaded the operating system, which means we can use it to fetch the initial lightweight boot files from our server node.

We already enabled tftp boot and set the directory for all boot files in /tftpboot, so all we have to do now is create the corresponding directory and add the needed permissions

sudo mkdir /tftpboot
sudo chmod -R 777 /tftpboot

Network file system(NFS)

To allow booting of Raspberry Pi's through the network, we need to setup a shared filesystem, that allows sharing of a directory located on one device inside the network, with other users on the same network. The device that hosts the directory is generally called the server, while the devices accessing it are called the clients. We install the nfs service using

sudo apt install nfs-kernel-server

and add to the now created /etc/exports file the directroy names we want to make available through the network. We will already add the /tftpboot folder to our exports and later we will add the root file systems for each of our worker nodes.

echo "/tftpboot *(rw,sync,no_subtree_check,no_root_squash)" | sudo tee -a /etc/exports

Now we need to make sure that RCP-bind and the NFS-server are up and running.

sudo systemctl enable rpcbind
sudo systemctl restart rpcbind
sudo systemctl enable nfs-kernel-server
sudo systemctl restart nfs-kernel-server

Create Base Image

For our worker nodes to be able to boot through the network, we need to provide a base image of our operating system, which we will copy to create a unique folder for each worker node. There are several possibilities to deploy an image.

Use rsync

First we will look at using rsync to copy our current operating system into a folder we will later allow access to through NFS.

sudo apt install rsync

We create a folder where the root of our worker operating system will be set:

sudo mkdir -p /nfs/base_image

And copy all files into the folder using rsync.

sudo rsync -xa --progress --exclude /nfs / /nfs/base_image

The -x option tells rsync to not cross filesystem boundaries, which is important, as temporary virtual filesystems created by the operating system, like /dev or /sys, will not be copied. A problem is, that the /boot folder is also in a separate FAT32 partition and we have to manually copy the boot folder as well.

sudo rsync -xa --progress /boot/firmware/ /nfs/base_image/boot/

Now we will also regenerate the SSH host key for our client filesystem

cd /nfs/base_image
sudo mount --bind /dev dev
sudo mount --bind /dev/pts dev/pts
sudo mount --bind /sys sys
sudo mount --bind /proc proc
sudo chroot .
rm /etc/ssh/ssh_host_*
dpkg-reconfigure openssh-server
exit
sudo umount dev sys proc dev/pts

As already mentioned, the /dev, /sys and /proc are virtual folders that are not copied, so we mount our original folders to the folders inside our base image. The chroot . command then changes the root '/' to the current directory, and any command we run will now use the binaries and libraries inside our base image. For now, we will only regenerate the ssh host key, but we can also use the same commands to install further software and packages needed on the base image. Do note that to use chroot, the cpu architectures must match, which can be a problem when we want to setup an image that differs from the image of our server node.

Now we still have to update our base images /etc/fstab file, where we will remove, or comment out, both PARTUUID partitions and add the tftp boot partition.

proc            /proc           proc    defaults          0       0
# PARTUUID=53c92d33-01  /boot/firmware  vfat    defaults          0       2
# PARTUUID=53c92d33-02  /               ext4    defaults,noatime  0       1
10.0.0.1:/tftpboot/base_image /boot/firmware/ nfs defaults,vers=3 0 0

In this example PARTUUID=53c92d33-01 is the boot partition(FAT32) for SD cards, and PARTUUID=53c92d33-02 is the root partition(EXT4). The fstab file tells Linux which physical hardware partition mounts to folders like /boot and the root /. If the entries remain, our system will look for a non-existing SD card and the boot sequence will freeze. The last line mounts the server's TFTP folder onto the worker Pi's /boot/firmware/ directory. This allows to run system updates directly on the worker Pi, as the boot files on the server will be updated rather than writing into an empty folder and to directly change settings in config.txt or cmdline.txt.

Instead of using fstab, during network boot the Raspberry Pi's firmware reads the cmdline.txt file, that tells the kernel to look across the network to find the operating system. The cmdline.txt file is located in our base image under /boot/cmdline.txt and we will change the contents of the file to the following:

console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=10.0.0.1:/nfs/base_image fsck.repair=yes rw rootwait quiet splash plymouth.ignore-serial-consoles

It is important, that each of the options is in one line! For the options we have:

  • console=serial0,115200: This option tells the kernel to send boot messages and system logs to the Raspberry Pi's physical serial pins (UART GPIO pins) at a baud rate of 115200.
  • console=tty1: This option tells the kernel to send boot messages to the first virtual terminal screen.
  • root=/dev/nfs: Instructs the kernel that the root filesystem is in a network file server.
  • nfsroot=10.0.0.1:/nfs/base_image: Tells the kernel exactly where the network file server is located.
  • fsck.repair=yes: Run a filesystem check to look for corrupted files and automatically fix errors.
  • rw: The network operating system folder will be mounted with full read and write access.
  • rootwait: Tells kernel to pause and wait indefinetly for the root filesystem.
  • quiet: Hides some of the scrolling kernel text messages during boot.
  • splash Show splash screen while loading.
  • plymouth.ignore-serial-consoles: Forces to ignore serial port and display graphical splash screen.

Mount image file directly to Pi 5

If we want to use a different image from our system, we can also directly mount a different image using the losetup (loop setup) command, that allows us to setup and use loop devices. As loop device we understand a block device that maps its datablocks to the blocks of a regular file inside a filesystem or another block device, instead of mapping them to a hard disk or optical disk drive. This means we can use a loop device to make our linux distribution treat the .img file as if it were a physical hard drive. We use losetup to set up our image, here called raspberry-pi-os.img, as a loop device.

# Set up the image as a loop device
sudo losetup -fP raspberry-pi-os.img

This creates the devices for the boot partition, like /dev/loop0p1, and root partition, like /dev/loop0p2. The -f option tells losetup to use the device found under the file name as loop device, -P forces the kernel to scan the partition table on a newly created loop device,, that tells the operating system how the storage space inside that drive is partitioned. Now we can mount the partitions inside our filesystem:

sudo mkdir -p /mnt/img_root /mnt/img_boot
sudo mount /dev/loop0p2 /mnt/img_root
sudo mount /dev/loop0p1 /mnt/img_boot

Copy them to our folder that should contain the base image

sudo mkdir -p /nfs/base_image
sudo cp -a /mnt/img_root/. /nfs/base_image/

And put the boot files in the right folder structure.

sudo mkdir -p /nfs/base_image/boot
sudo cp -a /mnt/img_boot/. /nfs/base_image/boot/

Then we can use the same steps as our previous section, to ready the base image to be used as blueprint for our worker nodes.

Add images for worker nodes

Each worker will use TFTP to try and get the boot files from our server Node. Now, if each worker accesses the same image, there might be problems when multiple worker nodes want to access the same files. For this reason we will create multiple folders containing an image for each of our worker nodes. TFTP boot for Raspberry Pi's automatically ask for the boot files in /tftpboot/$SERIAL_NUMBER first, and if it does not find the files it is looking for inside the folder with its serial number, it defaults to looking in /tftpboot for the necessary files. This means we want a custom image for each of our worker nodes based on its serial number.

First we create a folder to store our filesystem and boot files, which we will just copy from our base image

# create directory
mkdir -p "/nfs/$SERIAL_NUMBER"		
# Copy the base image
cp -a /nfs/base_image/. "/nfs/$SERIAL_NUMBER/"

Then we will create a link from /tftpboot/$SERIAL_NUMBER to the boot files inside our nfs partition

ln -s "/nfs/$SERIAL/boot" "/tftpboot/$SERIAL"

and add the folder of the root file system to /etc/exports, using eportsfs -ra to reexport(-r) all(-a) directories listed in /etc/exports.

echo "/nfs/$SERIAL *(rw,sync,no_subtree_check,no_root_squash)" >> /etc/exports
exportfs -ra

We will also have to cutomize the /etc/fstab file and the cmdline.txt. We can use the sed command to easily change strings in our files. Sed is a stream editor and allows us to perform basic text transformation on an input stream, in this case a file.

sed -i "s/base_image/$SERIAL_NUMBER/g" /tftpboot/$SERIAL_NUMBER/cmdline.txt
sed -i "s/base_image/$SERIAL_NUMBER/g" /nfs/$SERIAL_NUMBER/etc/fstab
  • -i: This option allows in-place editing, sed will overwrite the file directly.
  • s/search/replace/g:
    • s/: The s/ at the beginning of the expression initiates the substitution command
    • search: This is the target string we are looking for and which we want to replace, in this case it is 'base_image'
    • replace: The string used to replace our target string, here it is the serial number of a worker node $SERIAL_NUMBER
    • /g*: This is the global option, that tells sed to replace every occurence of our target string inside our file
  • target-folder:

Further we will have to change the hostname, as if every worker node has the same name, it might lead to problems. We will have to change the hostname in /etc/hostname

echo "$NODE_NAME" | sudo tee /nfs/$SERIAL_NUMBER/etc/hostname

and we have to update the hosts file, in /etc/hosts, changing the entry that is mapped to 127.0.1.1 to the new node name. The new File should look something like this, where $NODE_NAME$ is a placeholder for the hostname.

127.0.1.1 $NODE_NAME$
127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

The machine-id in /etc/machine-id and /var/lib/dbus/machine-id is a unique id that is generally set during either installation or booting and does not change after. This ID is supposed to be unique to each system and might lead to problems if multiple devices inside our network have the same, which is why we want our system to generate a new one at first boot. To do that we can empty the /etc/machine-id file.

sudo truncate -s 0 /etc/machine-id

Deleting the file outright might lead to problems at boot, while most operating system regenerate the file, the more secure route is to just empty the file. The /var/lib/dbus/machine-id is often a link to /etc/machine-id, and if it is missing, the link will be replaced, meaning we can just delete the link.

sudo rm -f /var/lib/dbus/machine-id

We already regenerated the ssh files in our setup, but if there are problems, it is also possible to just delete the files related to our ssh host keys, which will also be regenerated at startup.

sudo rm -f /nfs/$NEW_SERIAL/etc/ssh/ssh_host_*_key*

Lastly, for older Pi units, it first needs to succesfully download a bootcode.bin directly from /tftpboot for it to start the whole needed boot sequence, so we will copy bootcode.bin from our base image boot folder right into /tftpboot

sudo cp /nfs/base_image/boot/bootcode.bin /tftpboot/

With all that work, we have successfully setup the boot files and root system for one worker node on our server node.

Reroute internet through server node

To be able to access the internet using our worker nodes, we will set up ip forwarding and NAT in our server note, that will forward the packets being sent. First we enable the packet forwarding on our server using the commands:

echo "net.ipv4.ip_forward=1" | sudo tee /etc/sysctl.d/30-ipforward.conf
sudo sysctl -p /etc/sysctl.d/30-ipforward.conf

This will create a file with the content net.ipv4.ip_forward=1 that will run during booting and set the option we want. To check if our commands worked, we can look if the net.ipv4.ip_forward variable is indeed 1.

sysctl net.ipv4.ip_forward

Then we have to update /etc/nftables.conf and add

# Share Wi-Fi internet connection with the Ethernet cluster
table ip nat {
    chain postrouting {
        type nat hook postrouting priority srcnat;
        oifname "wlan0" masquerade
    }
}

and enable nftables.

sudo systemctl enable nftables
sudo systemctl restart nftables

Ready Worker nodes

To use network boot on our worker nodes, it might have to be enabled first. There are different methods for Raspberry Pi 3 and for Raspberry Pi 4 and higher.

Raspberry Pi 3

To allow the Raspberry Pi 3 to network boot we have to flip a permanent internal switch, that is calle a One-Time Programmable(OTP) bit. We first have to boot it using a SD card and to see if Network boot is already enabled, we can use the command vcgencmd, that communicates with the VideoCoreGPU firmware to report system information.

vcgencmd otp_dump | grep 17:

If the output is : 17:3020000a, then the network boot is already enabled, if on the other hand the output is 17:1020000a, the network boot is disabled and we have to set the internal bit. Do note, that the operation cannot be undone!

To enable usb and network boot we add the option program_usb_boot_mode=1 to out boot config:

echo program_usb_boot_mode=1 | sudo tee -a /boot/firmware/config.txt

After rebooting the OTP bit should be set.

Raspberry Pi 4 and higher

Raspberry Pi 4 and higher use a reprogrammable EEPROM(Electrically Erasable Programmable Read-Only Memory) to manage the boot order.

Using the following command we can look at the eeprom config

rpi-eeprom-config

If it is has not been changed and is still the default, we will not see a BOOT_ORDER value. changing it will make it appear and we will see a value like BOOT_ORDER=0xf21. The 2 signifies the Network boot, while 1 stands for the SD card boot. To enable it we first launch the configuration menu

sudo raspi-config

In the opened window we then

  • Navigate to Advanced options
  • Select Boot Order
  • Choose network boot
  • Select finish

The Pi will reboot and Network boot will be enabled.

Automate the process

Adding each Pi manually can be quite tedious, so we try to automate the process. Each worker node first looks for the folder containing its serial number, which will be logged inside dnsmasq. This means we can create a script, that reads the dnsmasq logger and does the following steps when a dnsmasq request comes in:

  • Look for serial number inside dnsmasq log
  • create folders in /nfs/ based on serial number /nfs/$SERIAL_NUMBER only if folder does not already exist
  • Add the base image to the folder
  • Create symlink from boot files to /tftpboot/$SERIAL_NUMBER
  • Update /etc/exports
  • restart NFS server
  • Update cmdline.txt, fstab, hosname, and machine-id
  • delete link to machine id
#!/bin/bash
# run through dnsmasq log to find serial number
journalctl -u dnsmasq -f -o cat | while read -r line; do
    case "$line" in
        *tftpboot*)
            SERIAL=$(echo "$line" | sed -n 's/.*tftpboot\/\([^/]*\)\/.*/\1/p')

            # Only proceed if we found a valid 8-digit hex serial
            if [[ $SERIAL =~ ^[0-9a-fA-F]{8}$ ]]; then

                # Check if we directory already exists
                if [ ! -d "/nfs/$SERIAL" ]; then
                    logger "Provisioner: Initializing $SERIAL..."

                    # Create the directory FIRST
                    if mkdir -p "/nfs/$SERIAL"; then
                        logger "created directory /nfs/$SERIAL"
                    else
                        logger "Error: could not create directory /nfs/$SERIAL"
                    fi
                    # Copy the template
                    if cp -a /nfs/base_image/. "/nfs/$SERIAL/"; then
                        logger "copied template to /nfs/$SERIAL"
                        sync
                    else
                        logger "Error: could not copy template"
                    fi

                    # Create the boot link
                    if ln -s "/nfs/$SERIAL/boot" "/tftpboot/$SERIAL"; then
                        logger "created symbolic link from /nfs/$SERIAL/boot to /tftpboot/$SERIAL"
                    else
                        logger "Error creating symlink for serial"
                    fi

                    # Automatically update the cmdline.txt
                    if sed -i "s/base_image/$SERIAL/g" /nfs/$SERIAL/boot/cmdline.txt; then
                        logger "Updated cmdline.txt"
                    else
                        logger "Error: could not update cmdline.txt"
                    fi

                    # Automatically update fstab
                    if sed -i "s/base_image/$SERIAL/g" /nfs/$SERIAL/etc/fstab; then
                        logger "Updated fstab"
                    else
                        logger "Error: could not update /etc/fstab"
                    fi

                    # Empty /etc/machine-id
                    if truncate -s 0 /nfs/$SERIAL/etc/machine-id; then
                        logger "truncated machine-id"
                    else
                        logger "Error: could not truncate machine-id"
                    fi

                    # Remove machine-id symlink
                    if rm -f /nfs/$SERIAL/var/lib/dbus/machine-id; then
                        logger "deleted /nfs/$SERIAL/var/lib/dbus/machine-id"
                    else
                        logger "Error: could not delete /nfs/$SERIAL/var/lib/dbus/machine-id"
                    fi

					# Remove ssh keys
                    if rm -f //nfs/$SERIAL/etc/ssh/ssh_host_*_key*; then
                        logger "deleted /nfs/$NEW_SERIAL/etc/ssh/ssh_host_*_key*"
                    else
                        logger "Error: could not delete /nfs/$NEW_SERIAL/etc/ssh/ssh_host_*_key*"
                    fi

                   #Set hostname
                   echo "pi-$SERIAL" > /nfs/$SERIAL/etc/hostname

                   # set hosts file
                   if sed -i "s/base_image/pi-$SERIAL/g" /nfs/$SERIAL/etc/hosts; then
                       logger "Updated hosts"
                   else
                       logger "Error: could not update /etc/hosts"
                   fi
                   # Add to exports if it's not already there
                    if ! grep -q "/nfs/$SERIAL" /etc/exports; then
                        echo "/nfs/$SERIAL *(rw,sync,no_subtree_check,no_root_squash)" >> /etc/exports
                        exportfs -ra
                        logger "Added /nfs/$SERIAL *(rw,sync,no_subtree_check,no_root_squash) to "
                    fi
                    logger "Provisioner: $SERIAL is ready."
                fi
            fi
            ;;
    esac
done

Most of our script mirrors already covered topics, so we will only look at new additions. Our first line

journalctl -u dnsmasq -f -o cat | while read -r line; do

queries the logs for messages from the dnsmasq service, the option -f keeps the command running and streams new log-entries in real time. The script keeps running in the background. -cat stripps out unneeded information, and we are left only with the log message test.

Using

case "$line" in
        *tftpboot*)

we are looking for lines that contain 'tftpboot' and using

SERIAL=$(echo "$line" | sed -n 's/.*tftpboot\/\([^/]*\)\/.*/\1/p')

we get the serial number that is between the two / after tftpboot. Looking closer at the sed command:

sed -n 's/.*tftpboot\/\([^/]*\)\/.*/\1/p')
  • -n: Option to suppress automatic printing of files
  • s/target/replace/p
    • s/ signifies substitution
    • /p prints the replaced line
    • target: .*tftpboot\/\([^/]*\)\/.*
      • .*: Looks for any characters
      • tftpboot\/: match the literal text tftpboot/
      • \([^/]*\): Is a capture group, denoted by \(...\), [^/]* matches any character that is not forward slash
        • captures the folder name right after tftpboot/
    • replace: \1 is a backreference that returns what our first capture group captured, namely the serial number we are looking for.
  • target-folder:

After checking if we have a valid serial number of length 8, we are basically mirroring the steps of adding images for our worker nodes and we completed our automation script.

Do note that the copying of the files is very slow! It takes several minutes to finish copying.

Create Systemd service

As systemd service, or daemon, we consider a type of background process that systemd is responsible for starting, stopping, and monitoring. We already created the script, now we want it to start after booting, and specifically after the network is fully active and the dnsmasq service has started.

To do that we create a systemd unit configuration file in /etc/systemd/system/ called pi-provisioner.service and we named the file our bashscript is in provisioner.sh.

The contents of pi-provisioner.service are

[Unit]
Description=Raspberry Pi PXE Auto-Provisioner
After=network.target dnsmasq.service

[Service]
ExecStart=/usr/local/bin/provisioner.sh
Restart=always
User=root

[Install]
WantedBy=multi-user.target

that is divided into three sections:

  • The unit section:
    The unit section defines the metadata for services and tells systemd when in the boot sequence the process is started
    • Description: Label(Name) for the service.
    • After=network.target dnsmasq.service: Defines which services have to run earlier.
  • The service section: Defines how the script is executed and managed.
    • ExecStart: Points to path of the bash script. Systemd will execute this command when starting the service.
    • Restart: Defines behaviour when service stops, in this case it will always restart.
    • User: Sets privileges based on user, here we have full root privileges.
  • The install section: Defines behaviour upon running sudo systemctl enable, for the service, meaning what happens after it is enabled.
    • WantedBy: This directive specifies the relationship between this service and other services. multi-user.target is the state where the system can accept multiple non-graphical user sessions.

After creating our file, we need to maek it executable and we have to enable the daemon, which we can do using the following commands:

sudo chmod +x /usr/local/bin/provisioner.sh
sudo systemctl daemon-reload
sudo systemctl enable pi-provisioner.service
sudo systemctl start pi-provisioner.service

And if we want to monitor our service we can use

journalctl -u pi-provisioner.service -f
← System Architecture Task 2 — Benchmarking →