[Bug 2081685] Re: [Ubuntu 24.04-generic Kernel-6.8]Hard lockup on 8 Socket System, ThinkSystem SR950 V3.

Hi Amy,

here is another kernel for you to test:
https://people.canonical.com/~ppisati/lp2081685/7f31caaf412e/

We are bisecting the Noble kernel starting from the good working one
(6.8.0-31.31, see your comment in #20) and the first one exhibiting the
issue (6.8.0-32.32), and the kernel you are about to test was produced
starting at commit 7f31caaf412e (while the previous one was at
83e6ebf6e740, etcetc).

Let me know how it goes.

--
You received this bug notification because you are subscribed to linux
in Ubuntu.
Matching subscriptions: Bgg, Bmail, Nb
https://bugs.launchpad.net/bugs/2081685

Title:
[Ubuntu 24.04-generic Kernel-6.8]Hard lockup on 8 Socket System,
ThinkSystem SR950 V3.

Status in linux package in Ubuntu:
New
Status in linux source package in Noble:
In Progress
Status in linux source package in Oracular:
New

Bug description:
There is CPU hard Lockup detected under Ubuntu 24.04 LTS (kernel
6.8.0-38). see attachment"dmesg0723-Lockup-Ubuntu24.04.log"

ubuntu@SR950V3:~$ cat /var/log/dmesg | grep -i lockup

[ 15.241164] kernel: watchdog: Watchdog detected hard LOCKUP on cpu
124

[ 15.241164] kernel: ? watchdog_hardlockup_check+0x1cb/0x3b0


Besides, the issue does not occur on upstream kernel 6.8，6.9, 6.10, 6.11-rc*, then only ubuntu kernel issue. see attachment "dmesg0923-No-Lockup-Kernel 6-10.log".
According to the dmesg log, the "hard lockup" is not a real lockup,
Because many CPU try to get cache_disable_lock spin lock at the same time when kernel boot. And competition has occurred here.
Every CPU's TLB will be flushed in the critical zone, the flushing TLB is a time-consuming operation, and there are so many CPUs,
so the false "hard lockup" was detected by kernel. To avoid customer confuse, when Canonical do the fix?


HW Config:
ThinkSystem SR950 V3

CPU: 8* Intel(R) Xeon(R) Platinum 8490H 60 Core 3.5GHz

MEM: 2TB = SK Hynix 356GB DDR5 4800MHz 3DS (2015.1GB)

Raid: ThinkSystem RAID 940-8i 4GB Flash PCIe Gen4 12Gb Adapter

Storage: Micron_7450_MTFDKBA960TFR *1

Samsung 30.7TB 24Gbps SAS 2.5" SSD

NIC: ThinkSystem Intel X710-T4L 10GBASE-T 4-Port OCP Ethernet Adapter

OS: ubuntu 24.04 LTS( kernel 6.8.0-38-generic)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2081685/+subscriptions