[Bug 1981487] Re: kernel BUG/Oops errors from modprobe while the DRBG has not yet initialized (focal/fips-updates)

This bug is awaiting verification that the linux-gcp-
fips/6.8.0-1017.19+fips1 kernel in -proposed solves the problem. Please
test the kernel and update this bug with the results. If the problem is
solved, change the tag 'verification-needed-noble-linux-gcp-fips' to
'verification-done-noble-linux-gcp-fips'. If the problem still exists,
change the tag 'verification-needed-noble-linux-gcp-fips' to
'verification-failed-noble-linux-gcp-fips'.


If verification is not done by 5 working days from today, this fix will
be dropped from the source code, and this bug will be closed.


See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how
to enable and use -proposed. Thank you!


** Tags added: kernel-spammed-noble-linux-azure-fips-v2 verification-needed-noble-linux-azure-fips

--
You received this bug notification because you are subscribed to linux
in Ubuntu.
Matching subscriptions: Bgg, Bmail, Nb
https://bugs.launchpad.net/bugs/1981487

Title:
kernel BUG/Oops errors from modprobe while the DRBG has not yet
initialized (focal/fips-updates)

Status in linux package in Ubuntu:
Invalid
Status in linux source package in Bionic:
Invalid
Status in linux source package in Focal:
Fix Released
Status in linux source package in Jammy:
Invalid

Bug description:
[Impact]

 * The Focal FIPS kernel in fips-updates hits kernel BUG/Oops
   errors during boot with the FIPS OpenSSL library installed
   (but those don't cause issues), when it runs modprobe with
   request_module() when looking up crypto algorithms/modules.

 * The modprobe command happens to call the OpenSSL library,
   and the FIPS version of OpenSSL calls getrandom(),
   and the FIPS kernel calls the DRBG for that,
   BUT it's _not yet_ initialized that early during boot
   when the kernel can run modprobe via request_module().
   (e.g., IPv6 initialization time.)

 * The issue impacts the kernels in fips-updates only, per:
   "UBUNTU: SAUCE: random: Use Crypto API DRBG for urandom in FIPS mode"
   which exists in Focal, but not in Xenial/Bionic/Jammy.

 * The issue only happens with the crypto algorithms, even
   if they're built-in (i.e., modprobe is not needed).

[Fix]

 * Fall back to CRNG while the DRBG is not yet initialized.
   (Marcelo Cerri confirmed it's OK per other discussions.)

 * The fix doesn't change the list and details of algorithms
   as in /proc/crypto (e.g., name, driver, module, priority)
   by the time the DRBG is initialized / initramfs started,
   so even though behavior changes, the net effect doesn't.

 * (Note: it's not possible to just use an initcall level
    earlier than rootfs_initcall() so modprobe isn't there,
    because fips_drbg_init() must run _after_ module_init()
    level so that crypto_rng_reset() works, even though its
    required module is built-in too.

[Test Steps]

 * Install the kernel and openssl from fips-updates,
   boot with fips=1, check dmesg for BUG/Oops errors:

   $ sudo apt install linux-image-fips libssl1.1 # fips-updates
   $ sudo vim /etc/default/grub # append fips=1 boot option
   $ sudo update-grub && sudo reboot
   $ sudo dmesg | grep BUG:

 * Check/store the /proc/crypto file for comparisons.
   You can boot with break=top as well, to check that
   as early as possible, and copy into /run/initramfs/
   then exit, to get it later in the rootfs.

[Regression Potential]

 * The fix falls back to the regular CRNG for a while
   in early boot. The CRNG is used permanently in the
   non-FIPS kernels (and in FIPS kernels w/out fips=1),
   so the code path is exercised/tested frequently.

 * Regressions would most likely occur in calls to
   getrandom() before the DRBG is initialized, but
   that currently hits a BUG/Oops anyway.

[Original Bug Description]

$ sudo apt install --yes linux-image-fips # fips-updates
$ sudo vim /etc/default/grub # fips=1
$ sudo update-grub && sudo reboot

$ uname -r
5.4.0-1056-fips

$ cat /proc/cmdline
... fips=1

No errors with the original/non-FIPS openssl, because it does NOT call
getrandom():

$ dmesg | grep -c BUG:
0

$ dpkg -s libssl1.1 | grep ^Version:
Version: 1.1.1f-1ubuntu2.15

$ strace -e getrandom modprobe --version
kmod version 27
+ZSTD +XZ -ZLIB +LIBCRYPTO -EXPERIMENTAL
+++ exited with 0 +++

But if you install the FIPS openssl, it calls getrandom(), then
BUG/Oops happen:

$ sudo apt install libssl1.1 # updates initramfs

$ dpkg -s libssl1.1 | grep ^Version:
Version: 1.1.1f-1ubuntu2.fips.13.1

$ strace -e getrandom modprobe --version
getrandom("\xc4\x84\x26\x25\x6f\xd4\xed\x38\xdf\xa9\x67\xee\x15\x1c\xe3\x98", 16, GRND_NONBLOCK) = 16
getrandom("\xa1\xd6\x67\x3e\xe4\x90\xb3\x8b\xdf\xe6\x34\x2a\xa7\x50\xbc\x2f", 16, GRND_NONBLOCK) = 16
getrandom("\xf1\x3e\xe4\x27\x9d\x47\x8c\x4b\x8a\x39\x8c\xe1\x2e\xee\xfa\x45", 16, GRND_NONBLOCK) = 16
getrandom("\xfb\x34\x18\x44\xd8\x23\x4c\x87\x13\x2e\x6b\x03\x79\xa7\x99\xf8", 16, 0) = 16
getrandom("\xdd\x83\xa7\x02\x10\x51\x2b\x4f\x21\x6b\xc1\xf1\x0d\xe7\x44\xb7", 16, 0) = 16
kmod version 27
+ZSTD +XZ -ZLIB +LIBCRYPTO -EXPERIMENTAL
+++ exited with 0 +++

$ sudo reboot

$ dmesg | grep -c BUG:
22

$ dmesg
...
[ 1.595759] NET: Registered protocol family 10
[ 1.600256] BUG: kernel NULL pointer dereference, address: 0000000000000038
...
[ 1.603829] CPU: 2 PID: 137 Comm: modprobe Not tainted 5.4.0-1056-fips #64-Ubuntu
[ 1.603829] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1.1 04/01/2014
[ 1.603829] RIP: 0010:urandom_read+0x268/0x480
...
[ 1.603829] Call Trace:
[ 1.603829] __x64_sys_getrandom+0x7f/0x130
[ 1.603829] do_syscall_64+0x57/0x190
[ 1.603829] entry_SYSCALL_64_after_hwframe+0x44/0xa9
...

All BUG/Oops errors are the same:

$ dmesg | grep BUG: | sed 's/^.*BUG:/BUG:/' | uniq -c
     22 BUG: kernel NULL pointer dereference, address: 0000000000000038

And they stop after the DRBG is initialized:

[ 3.651566] random: DRBG (drbg_nopr_ctr_aes256) initialized!

[Fix Impact Analysis]

The patch adds a dynamic debug message that can be enabled
in the kernel cmdline, for comparisons (dmesg); we can also
compare /proc/crypto for no changes.

dyndbg="func urandom_read +p"

Also add break=top, so we can copy /proc/crypto and dmesg
right after DRBG is initialized (when initramfs is started).

@ break=top time

 suffix=original # or modified
 cat /proc/crypto > /run/initramfs/proc-crypto.$suffix
 dmesg > /run/initramfs/dmesg.$suffix
 exit

@ login time
 sudo -s
 cp /run/initramfs/{proc-crypto.*,dmesg.*} .
 reboot # next test

There's no difference in the list/details of loaded crypto algorithms
at all, with any combination:

# md5sum proc-crypto.*
0b91bd619078fa342c6b4da039cb1582 proc-crypto.modified
0b91bd619078fa342c6b4da039cb1582 proc-crypto.original

The kernel with the fix does not hit BUG/Oops errors:

# grep ^ -m1 dmesg.*
dmesg.modified:[ 0.000000] Linux version 5.4.0-1060-fips ... #68+fipsdrbgnullcheck ...
dmesg.original:[ 0.000000] Linux version 5.4.0-1060-fips ... #68-Ubuntu ...

# grep -c BUG: dmesg.*
dmesg.modified:0
dmesg.original:22

# grep -c 'random: DRBG uninitialized! crng fallback' dmesg.*
dmesg.modified:110
dmesg.original:0

The 110 number is 5 * 22 calls as modprobe calls getrandom() 5 times (see strace above).
The original kernel has 1 BUG/Oops only because that kills the modprobe task.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1981487/+subscriptions