[Bug 2074215] Re: [SRU] UBSAN warnings in bnx2x kernel driver

** Changed in: linux (Ubuntu Noble)
Status: Fix Committed => Fix Released

--
You received this bug notification because you are subscribed to linux
in Ubuntu.
Matching subscriptions: Bgg, Bmail, Nb
https://bugs.launchpad.net/bugs/2074215

Title:
[SRU] UBSAN warnings in bnx2x kernel driver

Status in linux package in Ubuntu:
Fix Released
Status in linux source package in Focal:
Fix Released
Status in linux source package in Jammy:
Fix Released
Status in linux source package in Noble:
Fix Released
Status in linux source package in Oracular:
Fix Released

Bug description:
[impact]

Currently in the bnx2x kernel driver there are reads/writes that occur out of bounds that have the possibility to cause kernel crashes. No meaningful impact has been observed yet other than UBSAN stack traces.
I have posted a patch upstream to resolve this issue (134061163ee5 bnx2x: Fix multiple UBSAN array-index-out-of-bounds) and it has been accepted and merged. Although these traces appear only on linux version 6.5 and up, this bug also affects kernels 6.x and 5.x as well but no UBSAN warnings will be printed on these kernels since they were not enforced in these kernels.

[Test Plan]

There are multiple ways to reproduce the issue. But the most hands
free way to reproduce it would be to utilize a Qlogic NIC that makes
use of the E2 controller on a system with more than 32 cores. Below
are both ways this can be reproduced. Please note that both will
require a NIC that makes use of the bnx2x driver.

* Normal Reproduction:

1. start a machine running kernel 6.5 or higher with a a number of cores above 32. Please note that these need to be physical cores not threads. The machine also needs to be using a NIC that utilizes an E2 controller.
2. In dmesg the following UBSAN warnings can be seen:

UBSAN: array-index-out-of-bounds in
       drivers/net/ethernet/broadcom/bnx2x/bnx2x_stats.c:1529:11
index 20 is out of range for type 'stats_query_entry [19]'
CPU: 12 PID: 858 Comm: systemd-network Not tainted 6.9.0-060900rc7-generic
      #202405052133
Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9,
        BIOS P89 10/21/2019
Call Trace:
 <TASK>
 dump_stack_lvl+0x76/0xa0
 dump_stack+0x10/0x20
 __ubsan_handle_out_of_bounds+0xcb/0x110
 bnx2x_prep_fw_stats_req+0x2e1/0x310 [bnx2x]
 bnx2x_stats_init+0x156/0x320 [bnx2x]
 bnx2x_post_irq_nic_init+0x81/0x1a0 [bnx2x]
 bnx2x_nic_load+0x8e8/0x19e0 [bnx2x]
 bnx2x_open+0x16b/0x290 [bnx2x]
 __dev_open+0x10e/0x1d0
RIP: 0033:0x736223927a0a
Code: d8 64 89 02 48 c7 c0 ff ff ff ff eb b8 0f 1f 00 f3 0f 1e fa 41 89 ca
      64 8b 04 25 18 00 00 00 85 c0 75 15 b8 2c 00 00 00 0f 05 <48> 3d 00
      f0 ff ff 77 7e c3 0f 1f 44 00 00 41 54 48 83 ec 30 44 89
RSP: 002b:00007ffc0bb2ada8 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
RAX: ffffffffffffffda RBX: 0000583df50f9c78 RCX: 0000736223927a0a
RDX: 0000000000000020 RSI: 0000583df50ee510 RDI: 0000000000000003
RBP: 0000583df50d4940 R08: 00007ffc0bb2adb0 R09: 0000000000000080
R10: 0000000000000000 R11: 0000000000000246 R12: 0000583df5103ae0
R13: 000000000000035a R14: 0000583df50f9c30 R15: 0000583ddddddf00
</TASK>
---[ end trace ]---
------------[ cut here ]------------
UBSAN: array-index-out-of-bounds in
       drivers/net/ethernet/broadcom/bnx2x/bnx2x_stats.c:1546:11
index 28 is out of range for type 'stats_query_entry [19]'
CPU: 12 PID: 858 Comm: systemd-network Not tainted 6.9.0-060900rc7-generic
      #202405052133
Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9,
        BIOS P89 10/21/2019
Call Trace:
<TASK>
dump_stack_lvl+0x76/0xa0
dump_stack+0x10/0x20
__ubsan_handle_out_of_bounds+0xcb/0x110
bnx2x_prep_fw_stats_req+0x2fd/0x310 [bnx2x]
bnx2x_stats_init+0x156/0x320 [bnx2x]
bnx2x_post_irq_nic_init+0x81/0x1a0 [bnx2x]
bnx2x_nic_load+0x8e8/0x19e0 [bnx2x]
bnx2x_open+0x16b/0x290 [bnx2x]
__dev_open+0x10e/0x1d0
RIP: 0033:0x736223927a0a
Code: d8 64 89 02 48 c7 c0 ff ff ff ff eb b8 0f 1f 00 f3 0f 1e fa 41 89 ca
      64 8b 04 25 18 00 00 00 85 c0 75 15 b8 2c 00 00 00 0f 05 <48> 3d 00
      f0 ff ff 77 7e c3 0f 1f 44 00 00 41 54 48 83 ec 30 44 89
RSP: 002b:00007ffc0bb2ada8 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
RAX: ffffffffffffffda RBX: 0000583df50f9c78 RCX: 0000736223927a0a
RDX: 0000000000000020 RSI: 0000583df50ee510 RDI: 0000000000000003
RBP: 0000583df50d4940 R08: 00007ffc0bb2adb0 R09: 0000000000000080
R10: 0000000000000000 R11: 0000000000000246 R12: 0000583df5103ae0
R13: 000000000000035a R14: 0000583df50f9c30 R15: 0000583ddddddf00
 </TASK>
---[ end trace ]---
------------[ cut here ]------------
UBSAN: array-index-out-of-bounds in
       drivers/net/ethernet/broadcom/bnx2x/bnx2x_sriov.c:1895:8
index 29 is out of range for type 'stats_query_entry [19]'
CPU: 13 PID: 163 Comm: kworker/u96:1 Not tainted 6.9.0-060900rc7-generic
      #202405052133
Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9,
        BIOS P89 10/21/2019
Workqueue: bnx2x bnx2x_sp_task [bnx2x]
Call Trace:
 <TASK>
 dump_stack_lvl+0x76/0xa0
 dump_stack+0x10/0x20
 __ubsan_handle_out_of_bounds+0xcb/0x110
 bnx2x_iov_adjust_stats_req+0x3c4/0x3d0 [bnx2x]
 bnx2x_storm_stats_post.part.0+0x4a/0x330 [bnx2x]
 ? bnx2x_hw_stats_post+0x231/0x250 [bnx2x]
 bnx2x_stats_start+0x44/0x70 [bnx2x]
 bnx2x_stats_handle+0x149/0x350 [bnx2x]
 bnx2x_attn_int_asserted+0x998/0x9b0 [bnx2x]
 bnx2x_sp_task+0x491/0x5c0 [bnx2x]
 process_one_work+0x18d/0x3f0
 </TASK>
---[ end trace ]---

* Forced reproducer:

1. Make sure you have a machine running kernel 6.5 and higher with any
NIC that makes use of the bnx2x driver (No need for a NIC that
utilizes the E2 controller). Also the number of cores the machine has
is not important.

2. once the machine is booted unload the bnx2x module from the kernel:
$ sudo modprobe -r bnx2x

3. then load back the driver but while specifying the number of ethernet queues to a value above 16:
$ sudo modprobe bnx2x num_queues=20

4. The same stack traces shown above will show up in dmesg.

[Fix]

The fix already upstream and provided by:

* 134061163ee5 bnx2x: Fix multiple UBSAN array-index-out-of-bounds

[where problems could occur]

* Since the patch increases the firmware stats array size, the driver
will utilize slightly more memory, however this is still an
insignificant amount.

* Since no logic change has been done to the driver the regression
risk is minimal

[workaround]

As stated earlier I have already written a patch to solve the issue, but in the meantime one way to avoid this problem would be to unload the driver and then load it back with a value for num_queues below 16:
$ sudo modprobe bnx2x num_queues=15

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2074215/+subscriptions