Big Bubbles (no troubles)

What sucks, who sucks and you suck

Secure Boot With Cobbler

It doesn’t appear to be much discussed but you can perform secure boot of UEFI clients from a Cobbler build server. We managed it recently, under pain of needing to install Ubuntu on 400 Dell laptops before an imminent start-of-term deadline, and with only a modicum of dirty hacks.

I’m not claiming to be an expert on how Secure Boot works (all my knowledge was gained on a JIT basis), but my understanding of it thus far is contained within the following:

  • Secure Boot only generally works on UEFI firmware.
  • UEFI supports PXE network booting the same as BIOS, although you might have to explicitly enable it first in Setup, so Cobbler compatibility isn’t generally an issue (but see below). You will need to use an EFI-compliant boot loader though (i.e. GRUB), rather than PXELINUX .
  • Secure Boot firmware requires all the binaries it loads (i.e. the boot loader and initial OS bootstrap files) to be cryptographically signed and checksummed by a trusted key.
  • The default keys in most UEFI images belong to Microsoft, so the boot loader at least must be signed by a Microsoft key.
  • The firmware will also import and trust keys that have been signed by a Microsoft key. Hence the approach for booting a Linux-based OS is to first load a shim file which is verified by MS and signed with one of their trusted keys. This in turn provides a distro-specific CA key, which has been used to sign the further stages of the boot process (GRUB loader, kernel, initial ram disk, etc.). The implication of this is that the shim, loader, kernel and ram disk files must all come from the same distribution. Under the standard non-secure client boot, Cobbler relies on a common GRUB loader to boot any supported distro kernel. This won’t work with secure boot; you need to use the shim and loader file specific to the vendor and OS distribution you’re booting. This has implications if you’re using Cobbler for DHCP management, as the DHCP template in Cobbler serves a generic EFI boot loader by default, so either you hack the template to differentiate secure boot clients some way or you stick with secure booting only one supported OS flavour.
  • Only GRUB2 supports secure boot. I don’t believe Cobbler currently supports GRUB2, in terms of generating the correct configuration files (there is a merged PR but it isn’t in a release yet), but it can be hacked to do so as we’ve done here.
  • GRUB as utilised by Cobbler relies on the GRUB loader searching for a client-specific configuration file named after the MAC address, e.g. something like 01-00-50-DE-AD-BE-EF. Unfortunately, it turns out that this behaviour was added to GRUB by Red Hat in a downstream patch and is specific to their version. Remember I said you have to use the distro-specific GRUB loader signed with the same key as everything else? So yep, this won’t work for non-EL distros. Instead, we’ll need to load a common grub.cfg configuration file that then sources a second configuration named after the client MAC address. (Needless to say, the MAC address format used by Red Hat’s GRUB and thus generated by Cobbler differs from the one returned in the standard GRUB2 client variables. Hack, hack.)
    Also, we discovered that some of the typical GRUB2 loadable modules (such as regexp) are blacklisted in a secure boot environment, which further limits the kind of manipulations you can perform within the config.
  • IMPORTANT: Secure boot also means that any third party kernel modules - such as, say, the proprietary NVIDIA driver - must also be signed with a trusted key. As an end user can’t use the distro key to do this, DKMS typically generates a new Machine Owner Key (MOK) at install time using mokutil. It then prompts you interactively for a password, which must be entered on the next boot to confirm import of the new key into the firmware database so that the driver will be authenticated to load. As far as I can tell, this process cannot be automated, at least via Ansible (doubtless you could build a custom integrated distro instead but…). Despite all our effort to make Secure Boot work, this caught us out in the end and resulted in us disabling it on each client. But up until that point, it worked so I’m providing the recipe here for anyone with more modest requirements.

In our case, PXE client configuration is a little simpler or at least less of a concern as we use external DHCP servers and configure the client boot parameters such as the initial filename separately. (If you’re using Cobbler for DHCP: sorry, you’re on your own but see the notes above.) As mentioned, our specific use case was to boot Ubuntu Bionic 18.04 LTS on Dell 5490 laptops from a CentOS 7 host running the cobbler-2.8.4-4 package from EPEL. In the end, we used the method suggested by this Russian blog post (Google translation - don’t copy the shell source from this link as it will be corrupted). The post suggests adding a Cobbler post-sync trigger script to create the required GRUB configurations with the correct filenames by copying and renaming the ones generated by Cobbler. However, in our case we also need to convert the GRUB-legacy configurations to GRUB2 syntax and reformat the client MAC addresses to be compatible with that used in the GRUB2 $net_default_mac variable, which is colon- rather than hyphen-separated. The revised configurations are written to a uefi/ subdirectory under the TFTP boot folder, along with a default (initial) config that simply sources the appropriate client MAC-specific file.

Step-by-step then, here’s what to do:

  1. Ensure your distro signatures are up to date in Cobbler.
  2. Import the Ubuntu server edition ISO into Cobbler (don’t use a live image, it won’t work) and set up a boot profile and client systems as normal. Cobbler includes an example Debian preseed file called sample.seed (and good luck with it because the syntax isn’t well documented).
  3. On the Cobbler server, create the directory /var/lib/tftpboot/uefi/.
  4. From an existing installed Ubuntu system, ensure the packages shim-signed and grub-efi-amd64-signed are installed and copy the files /usr/lib/shim/shim64.efi and /usr/lib/grub/x86_64-efi-signed/grubnetx64.efi. Copy them to the uefi/ folder above, renaming as shim64.efi and grubx64.efi respectively.
  5. Download this Cobbler post-sync trigger script, make it executable and place it in /var/lib/cobbler/triggers/sync/post/
  6. Configure the DHCP PXE settings for your Ubuntu client to boot from the Cobbler host as the TFTP server using the filename uefi/shimx64.efi. As I noted above, if you’re using Cobbler as the DHCP server, this may require a bit more hacking of /etc/cobbler/dhcp.template.
  7. Run cobbler sync and check the generated files under /var/lib/tftpboot/uefi/.

(With thanks to all the authors of the various blog, forum and GitHub posts I googled to figure all this out.)

Other bubbles

  • Secure Boot info from Debian
  • Linked from the above, more than you wanted to know about Shims; see particularly the concluding thoughts about the likely utility of all this for Linux users.