[Bug 2081685] Re: [Ubuntu 24.04-generic Kernel-6.8]Hard lockup on 8 Socket System, ThinkSystem SR950 V3.

Hi Paolo.

Here is the feedback for the question:
I cannot commit it occurs 100%, with the latest Noble kernel is Ubuntu-6.8.0-48.48, the issue occurs 7/10 times. There is no change on HW config nor FW code, just Reboot. The issue does not occur in random, but once it occurs, it always later then reboot.

Besides, for kernel 31.31, it does not occur in 5/5 times.

Hope it clarifies and move forward on the root cause.

--
You received this bug notification because you are subscribed to linux
in Ubuntu.
Matching subscriptions: Bgg, Bmail, Nb
https://bugs.launchpad.net/bugs/2081685

Title:
[Ubuntu 24.04-generic Kernel-6.8]Hard lockup on 8 Socket System,
ThinkSystem SR950 V3.

Status in linux package in Ubuntu:
New
Status in linux source package in Noble:
In Progress
Status in linux source package in Oracular:
New

Bug description:
There is CPU hard Lockup detected under Ubuntu 24.04 LTS (kernel
6.8.0-38). see attachment"dmesg0723-Lockup-Ubuntu24.04.log"

ubuntu@SR950V3:~$ cat /var/log/dmesg | grep -i lockup

[ 15.241164] kernel: watchdog: Watchdog detected hard LOCKUP on cpu
124

[ 15.241164] kernel: ? watchdog_hardlockup_check+0x1cb/0x3b0


Besides, the issue does not occur on upstream kernel 6.8，6.9, 6.10, 6.11-rc*, then only ubuntu kernel issue. see attachment "dmesg0923-No-Lockup-Kernel 6-10.log".
According to the dmesg log, the "hard lockup" is not a real lockup,
Because many CPU try to get cache_disable_lock spin lock at the same time when kernel boot. And competition has occurred here.
Every CPU's TLB will be flushed in the critical zone, the flushing TLB is a time-consuming operation, and there are so many CPUs,
so the false "hard lockup" was detected by kernel. To avoid customer confuse, when Canonical do the fix?


HW Config:
ThinkSystem SR950 V3

CPU: 8* Intel(R) Xeon(R) Platinum 8490H 60 Core 3.5GHz

MEM: 2TB = SK Hynix 356GB DDR5 4800MHz 3DS (2015.1GB)

Raid: ThinkSystem RAID 940-8i 4GB Flash PCIe Gen4 12Gb Adapter

Storage: Micron_7450_MTFDKBA960TFR *1

Samsung 30.7TB 24Gbps SAS 2.5" SSD

NIC: ThinkSystem Intel X710-T4L 10GBASE-T 4-Port OCP Ethernet Adapter

OS: ubuntu 24.04 LTS( kernel 6.8.0-38-generic)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2081685/+subscriptions