Diskless PXE boot on NFS root

When you have a whole rack of machines and don't want to install and manage separate OS installs on all of them, setting up to boot via PXE with a shared NFS root can be a major time-saver. Here are some notes on how to set up both a host machine to act as the DHCP, TFTP, and NFS server, as well as how to set up the PXE-booted image. In this configuration, the control server will also act as a gateway machine for the PXE booted hosts.

Set up ubuntu as a net boot source (for ubuntu) (ref: https://help.ubuntu.com/community/DisklessUbuntuHowto)

Install ubuntu

Install necessary packages

# apt install isc-dhcp-server tftpd-hpa pxelinux nfs-kernel-server debootstrap

Configure network interface (ref: https://help.ubuntu.com/lts/serverguide/network-configuration.html.en). In this case, eno1 is the external interface (facing the internet) and eno2 is the internal interface (facing the PXE booted hosts).

Edit /etc/netplan/99_config.yaml

network:
  version: 2
  renderer: networkd
  ethernets:
    eno1:
      dhcp4: true
    eno2:
      addresses:
        - 10.144.0.1/16

Configure DHCP (ref: https://help.ubuntu.com/community/isc-dhcp-server)

Edit /etc/dhcp/dhcpd.conf

option domain-name "your-domain-name";
option domain-name-servers 8.8.8.8, 9.9.9.9;

default-lease-time 600;
max-lease-time 7200;

subnet 10.144.0.0 netmask 255.255.0.0 {
  option routers 10.144.0.1;
  option broadcast-address 10.144.255.255;
  option ntp-servers 1.2.3.4;
}

include "/etc/dhcp/reactor-host-dhcp.conf";
include "/etc/dhcp/reactor-ilo-dhcp.conf";

Edit /etc/dhcp/reactor-host-dhcp.conf

host reactor0 {
  hardware ethernet de:ad:be:ef:ca:fe;
  fixed-address 10.144.0.100;
  option host-name "reactor0";
  filename "/pxelinux.0";
  next-server 10.144.0.1;
}

# etc.

Edit /etc/dhcp/reactor-ilo-dhcp.conf

host reactor0-mgt {
  hardware ethernet de:ad:be:ef:ca:fe;
  fixed-address 10.144.1.100;
  option host-name "reactor0-mgt";
}

# etc.

Edit /etc/default/isc-dhcp-server

INTERFACESv4="eno2"

Start DHCP server

Run systemctl enable –now isc-dhcp-server

Configure TFTP

Should be good to go already once tftpd-hpa is installed

Configure PXE

# mkdir /srv/nfs/ubuntu
# cd /var/lib/tftpboot
# mkdir pxelinux.cfg boot
# cp /usr/lib/PXELINUX/pxelinux.0 /var/lib/tftpboot
# cp -r /usr/lib/syslinux/modules/bios /var/lib/tftpboot/boot/isolinux

Edit /var/lib/tftpboot/pxelinux.cfg/default

SERIAL 1 115200
UI menu.c32
TIMEOUT 100
ONTIMEOUT linux

LABEL linux
  MENU LABEL Ubuntu Linux
  KERNEL ubuntu/vmlinuz
  APPEND root=/dev/nfs initrd=ubuntu/initrd.img nfsroot=10.144.0.1:/srv/nfs/ubuntu ip=dhcp rw console=ttyS1 console=tty0

LABEL memtest
  MENU LABEL Memtest86+
  KERNEL memtest

LABEL memtest-console
  MENU LABEL Memtest86+ (serial console)
  KERNEL memtest
  APPEND console=ttyS1,115200n8

# mkdir ubuntu
# mount --bind /srv/nfs/ubuntu ubuntu

Edit /etc/fstab to append

# tftp doesn't follow symlinks
/srv/nfs/ubuntu        /var/lib/tftpboot/ubuntu           none    bind            0       0

Configure NFS

(ref:https://help.ubuntu.com/community/SettingUpNFSHowTo)

Export install and required mounts

Edit /etc/exports

/home           10.144.0.0/24(rw,no_root_squash,async,insecure)
/opt            10.144.0.0/24(rw,no_root_squash,async,insecure)
/srv/nfs/ubuntu 10.144.0.0/24(rw,no_root_squash,async,insecure)

Run exportfs -rv

Install ubuntu for net boot clients (ref: https://help.ubuntu.com/community/Installation/OnNFSDrive)

# mkdir /srv/nfs
# cd /srv/nfs

Edit run_debootstrap.sh

#!/bin/sh

debootstrap --arch amd64 bionic /srv/nfs/ubuntu/ http://archive.ubuntu.com/ubuntu

Run bash run_debootstrap.sh

Do not set /srv/nfs/ubuntu/etc/hostname, hostname will come from DHCP

# mount --bind /proc ubuntu/proc
# chroot ubuntu

Set up locales

Edit /etc/locale.gen, uncomment locale (en_US.UTF-8)

Run locale-gen

Install kernel, nfs-common, and SSH server Do not install grub if prompted to do so

# apt install linux-image-generic nfs-common openssh-server
# chmod a+r /boot/vmlinuz*
# systemctl enable ssh

Enable serial console by running systemctl enable getty@ttyS1

Set up fstab

Edit /etc/fstab

# /etc/fstab: static file system information.
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
/dev/nfs        /               nfs     defaults        1       1
none            /tmp            tmpfs   defaults        0       0
none            /var/tmp        tmpfs   defaults        0       0
10.144.0.1:/home /home          nfs     defaults        0       0
10.144.0.1:/opt /opt            nfs     defaults        0       0

Workaround for unreadable kernel image

Edit /etc/kernel/postinst.d/chmod-vmlinuz

#!/bin/sh -e

chmod 0644 /boot/vmlinuz-*

Run chmod a+x /etc/kernel/postinst.d/chmod-vmlinuz

Edit initramfs settings for NFS boot

Edit /etc/initramfs-tools/initramfs.conf

MODULES=netboot

Regenerate initramfs by running update-initramfs -u

Set root password and/or add users, install packages, etc. Then:

# exit
# umount ubuntu/proc

At this point, it should be possible to boot a system over the network.

(ref: https://help.ubuntu.com/lts/serverguide/firewall.html#ip-masquerading)

Note: make sure console is accessable over iLO just in case networking gets broken

Edit /etc/default/ufw

DEFAULT_FORWARD_POLICY="ACCEPT"

Edit /etc/ufw/sysctl.conf

net/ipv4/ip_forward=1

Edit /etc/ufw/before.rules [place immediately after header comment]

# nat Table rules
*nat
:POSTROUTING ACCEPT [0:0]

# Forward traffic from eno2 through eno1.
-A POSTROUTING -s 10.144.0.0/16 -o eno1 -j MASQUERADE

# don't delete the 'COMMIT' line or these nat table rules won't be processed
COMMIT

# ufw allow ssh
# ufw allow in on eno2
# ufw enable