[Bug 2060039] Comment bridged from LTC Bugzilla

------- Comment From kowshik.jois@in.ibm.com 2024-10-16 07:47 EDT-------
I have verified bug on 6.8.0-48-generic kernel and the issue is still getting recreated.

My system had 15G Memory and according to the recommendation here:
https://wiki.ubuntu.com/ppc64el/Recommendations I had reserved 512M for
the crashkernel. I had enabled FADump and triggered sysrq crash and
observed the L1 hang issue again.

I have attached the console log next comment. Below are the ENV details
on which I had verified this scenario

root@ubuntu2404:~# uname -ar
Linux ubuntu2404 6.8.0-48-generic #48-Ubuntu SMP Fri Sep 27 14:02:57 UTC 2024 ppc64le ppc64le ppc64le GNU/Linux

root@ubuntu2404:~# cat /proc/cmdline
BOOT_IMAGE=/vmlinux-6.8.0-48-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro fadump=on fadump_reserve_mem=512M

root@ubuntu2404:~# dmesg | grep -i reser
[ 0.000000] fadump: 'fadump_reserve_mem=' parameter is deprecated in favor of 'crashkernel=' parameter.
[ 0.000000] fadump: Reserved 512MB of memory at 0x00000020000000 (System RAM: 16384MB)
[ 0.000000] fadump: Initialized 0x20000000 bytes cma area at 512MB from 0x20030000 bytes of memory reserved for firmware-assisted dump
[ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinux-6.8.0-48-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro fadump=on fadump_reserve_mem=512M
[ 0.000000] Memory: 16040704K/16777216K available (23680K kernel code, 4096K rwdata, 25472K rodata, 8832K init, 1901K bss, 212224K reserved, 524288K cma-reserved)
[ 0.278704] ibmvscsi 30000069: Client reserve enabled

root@ubuntu2404:~# kdump-config show
DUMP_MODE: fadump
USE_KDUMP: 1
KDUMP_COREDIR: /var/crash
/var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinux-6.8.0-48-generic
kdump initrd:
/var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-6.8.0-48-generic
current state: ready to fadump

--
You received this bug notification because you are subscribed to linux
in Ubuntu.
Matching subscriptions: Bgg, Bmail, Nb
https://bugs.launchpad.net/bugs/2060039

Title:
[Ubuntu-24.04] FADump with recommended crash size is making the L1
hang

Status in The Ubuntu-power-systems project:
Fix Committed
Status in linux package in Ubuntu:
Fix Released
Status in linux source package in Noble:
Fix Committed
Status in linux source package in Oracular:
Fix Released

Bug description:
SRU Justification:

[Impact]
 * L1 host hangs when triggering FADump that results in crash

[Fix]
 * 353d7a84c214f184d5a6b62acdec8b4424159b7c 353d7a84c214 "powerpc/64s/radix/kfence: map __kfence_pool at page granularity"

[Test Case]
 * Have a Ubuntu Server 24.04 LTS installation on ppc64el.
 * Enable FADump with 1GB: fadump=on crashkernel=1024M
 * A kernel panic will happen when dump got triggered

[Regression Potential]
* There is a certain risk of a regression, but it is mapping only the memory
  allocated for KFENCE pool at page granularity, reducing memory consumption
  when KFENCE is used.

* On top the commit is already upstream reviewed and accepted.

* The modifications were done and tested by IBM.

* The fadump feature is supported only on IBM POWER systems.

[Other]
* The fix/commit got upstream accepted with kernel v6.11-rc4,
  hence Oracular (with a planned kernel of 6.11) is not affected.

.......................

Problem description :
======================

Triggered FADump with the recommended crash. L1 host got hung.

As per the public document
https://wiki.ubuntu.com/ppc64el/Recommendations recommended crash
kernel size is 1024M for the system. But with 1024M and 2048M, the L1
is getting hanged. with 4096, crash is generated and collected.

root@ubuntu2404:~# uname -ar
Linux ubuntu2404 6.8.0-11-generic #11-Ubuntu SMP Wed Feb 14 00:33:03 UTC 2024 ppc64le ppc64le ppc64le GNU/Linux

root@ubuntu2404:~# free -h
               total used free shared buff/cache available
Mem: 48Gi 1.7Gi 46Gi 13Mi 687Mi 46Gi
Swap: 8.0Gi 0B 8.0Gi

root@ubuntu2404:~# cat /proc/cmdline
BOOT_IMAGE=/vmlinux-6.8.0-11-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro fadump=on crashkernel=1024M

root@ubuntu2404:~# dmesg | grep -i reser
[ 0.000000] fadump: Reserved 1024MB of memory at 0x00000040000000 (System RAM: 51200MB)
[ 0.000000] fadump: Initialized 0x40000000 bytes cma area at 1024MB from 0x40070000 bytes of memory reserved for firmware-assisted dump
[ 0.000000] Memory: 49316672K/52428800K available (23616K kernel code, 4096K rwdata, 25536K rodata, 8832K init, 2487K bss, 2063552K reserved, 1048576K cma-reserved)
[ 0.396408] ibmvscsi 30000066: Client reserve enabled

root@ubuntu2404:~# kdump-config show
DUMP_MODE: fadump
USE_KDUMP: 1
KDUMP_COREDIR: /var/crash
   /var/lib/kdump/vmlinuz
kdump initrd:
   /var/lib/kdump/initrd.img
current state: ready to fadump

IBM is looking to update the crash kernel reservations section of the
wiki for Power.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-power-systems/+bug/2060039/+subscriptions