Public bug reported: I updated my DL360 gen8 server to the newest kernel and got a non functioning P420i due to massive errors in dmesg. Since I'm not that familiar with kernel stuff and storage drivers I've asked an LLM to provide all necessary information to file a bug against the ubuntu kernel package. This is the result of the information that was already found. Let me know if you need any further information on this issue. # [regression] 6.8.0-134: IOVA allocator hands out addresses inside BIOS RMRR regions, causing DMA mapping failures and storage controller loss (HP ProLiant Gen8 / P420i) **Package:** linux (Ubuntu Noble 24.04) **Broken:** 6.8.0-134.134 (6.8.12) **Last good:** 6.8.0-124.124 **Also present in:** 6.8.0-136.136 (per changelog; no fix/revert present) **Severity:** total storage loss on affected hosts — 32 TB RAID volume dropped offline, 10 production VMs lost their disks --- > **Note on how this report was produced.** The diagnosis, log analysis and the > text of this report were assembled with the help of an AI assistant (Claude) > working on the affected machine under my supervision. I have reviewed it and > can vouch for the underlying facts; the raw logs are attached so every claim > can be checked independently. The address/RMRR correlation below was verified > programmatically against the raw log rather than by eye. > > **I am happy to supply any additional information, run diagnostics, or test a > proposed kernel on request** — see the "Control machine" note near the end. --- ## Summary On 6.8.0-134, the Intel IOMMU IOVA allocator hands out IOVAs that fall inside BIOS-declared RMRR regions. Those regions are already identity-mapped, so `__domain_mapping()` finds an existing PTE, throws `WARNING at drivers/iommu/intel/iommu.c:2227`, and the DMA mapping fails: ``` WARNING: CPU: 32 PID: 562 at drivers/iommu/intel/iommu.c:2227 __domain_mapping+0x2b5/0x320 DMAR: ERROR: DMA PTE for vPFN 0xbdf81 already set (to bdf81003 not 128759003) ``` Note `bdf81003` — vPFN 0xbdf81 is mapped to PFN 0xbdf81, i.e. an identity mapping, which is what RMRR regions get. This is **not driver-specific**. On this host it hit three drivers within 90 seconds: `nvme` first, then `hpsa`, then `tg3`. The HP Smart Array P420i was simply the one that could not recover: its logical volume reset failed, the device went offline (`DID_NO_CONNECT`), and the 32 TB RAID-5 volume backing our LVM volume group disappeared, taking 10 production VMs with it. ## Root cause evidence The BIOS declares these RMRR regions (identical on both kernels, see below): ``` DMAR: RMRR base: 0x000000bdf6e000 end: 0x000000bdf6efff -> PFN 0xbdf6e DMAR: RMRR base: 0x000000bdf6f000 end: 0x000000bdf7efff -> PFN 0xbdf6f-0xbdf7e DMAR: RMRR base: 0x000000bdf7f000 end: 0x000000bdf82fff -> PFN 0xbdf7f-0xbdf82 DMAR: RMRR base: 0x000000bdf83000 end: 0x000000bdf84fff -> PFN 0xbdf83-0xbdf84 union -> PFN 0xbdf6e-0xbdf84 ``` The complete set of colliding vPFNs observed across all 66 errors: ``` 0xbdf6e 0xbdf6f 0xbdf70 0xbdf71 0xbdf72 0xbdf73 0xbdf74 0xbdf75 0xbdf76 0xbdf77 0xbdf78 0xbdf79 0xbdf7a 0xbdf7b 0xbdf7c 0xbdf7d 0xbdf7e 0xbdf7f 0xbdf80 0xbdf81 0xbdf82 0xbdf83 0xbdf84 ``` That is `0xbdf6e`-`0xbdf84` — an exact, complete match with the RMRR union. All 23 distinct colliding IOVAs lie inside an RMRR region; none lies outside one. (Checked programmatically over the attached log, not by inspection.) The RMRR reserved-region exclusion in the IOVA allocator appears not to be in effect. ## This is not a firmware or BIOS problem The RMRR regions declared by the BIOS are **byte-for-byte identical** on 6.8.0-124 and 6.8.0-134 (diffed; both logs attached). Same machine, same BIOS (P71, 05/24/2019), same controller firmware. The only variable is the kernel. 6.8.0-124 handles these exact regions correctly. ## This is not passthrough-specific No VFIO/PCI passthrough is configured on this host (`lsmod | grep vfio` -> 0). This is a plain KVM/libvirt host using virtio and LVM-backed disks. Reports elsewhere tie the 6.8.12 IOMMU regression to VFIO group assignment; this reproducer has no VFIO involvement at all, which suggests the underlying defect is broader than the passthrough symptom it was first noticed through. ## A/B evidence (same physical machine, consecutive boots) | Boot | Kernel | Uptime | DMAR errors | I/O errors | |------|--------|--------|-------------|------------| | -2 | 6.8.0-124 | 2026-06-13 -> 2026-07-14 (31 days) | 0 | 0 | | -1 | 6.8.0-134 | ~10 min to controller loss | 66 | 4701 | | 0 | 6.8.0-124 | since revert, stable | 0 | 0 | Booting 6.8.0-124 restored full function immediately: controller status OK, volume group back, all VMs recovered. ## Timeline (boot -1, kernel 6.8.0-134) ``` 12:32:34 boot 12:41:24 first WARNING - nvme_map_data (mdraid rebuild driving heavy DMA) 12:42:46 hpsa_scatter_gather hits the same warning 12:42:50 tg3_start_xmit hits it too (network path) 12:43:23 hpsa: resetting logical Direct-Access HP LOGICAL VOLUME RAID-5 12:44:32 hpsa: reset logical failed / sd 3:1:0:0: rejecting I/O to offline device sda offline -> volume group gone -> 10 VMs lose their disks 13:25:05 reboot into 6.8.0-124 -> fully recovered ``` The first failure came ~9 minutes in, under sustained DMA load from an mdraid rebuild — consistent with the allocator only reaching the RMRR range once lower IOVA space was under pressure. Hosts may therefore appear fine after boot and fail later under load. ## Representative stack trace ``` WARNING: CPU: 32 PID: 562 at drivers/iommu/intel/iommu.c:2227 __domain_mapping+0x2b5/0x320 CPU: 32 PID: 562 Comm: kworker/32:1H Tainted: G I 6.8.0-134-generic #134-Ubuntu Hardware name: HP ProLiant DL360p Gen8, BIOS P71 05/24/2019 Workqueue: kblockd blk_mq_run_work_fn RIP: 0010:__domain_mapping+0x2b5/0x320 Call Trace: <TASK> intel_iommu_map_pages+0xe1/0x140 __iommu_map+0x121/0x280 iommu_map_sg+0xbf/0x1f0 iommu_dma_map_sg+0x463/0x4f0 __dma_map_sg_attrs+0x35/0xd0 dma_map_sgtable+0x1d/0x40 nvme_map_data+0xd6/0x480 [nvme] nvme_prep_rq.part.0+0x35/0x120 [nvme] nvme_queue_rq+0x7e/0x207 [nvme] blk_mq_dispatch_rq_list+0x137/0x520 __blk_mq_sched_dispatch_requests+0x118/0x190 blk_mq_sched_dispatch_requests+0x2f/0x80 blk_mq_run_work_fn+0x72/0x90 process_one_work+0x184/0x3a0 worker_thread+0x18b/0x330 kthread+0xf2/0x120 ret_from_fork+0x47/0x70 ret_from_fork_asm+0x1b/0x30 </TASK> ---[ end trace 0000000000000000 ]--- DMAR: ERROR: DMA PTE for vPFN 0xbdf81 already set (to bdf81003 not 128759003) ``` The same warning with `hpsa_scatter_gather+0x3e/0x310 [hpsa]` and with `__tg3_start_xmit+0x1f0/0xd60 [tg3]` in place of the nvme frames is in the attached log (46 nvme, 18 hpsa, 4 tg3 occurrences). ## Environment ``` Hardware: HP ProLiant DL360p Gen8 BIOS: HP P71, 05/24/2019 CPU: Intel Xeon E5-2697 v2 @ 2.70GHz (48 threads) RAM: 503 GB Controller: Smart Array P420i [103c:323b] rev 01, subsystem [103c:3354] PCI 0000:02:00.0, IOMMU group 49 Driver: hpsa 3.4.20-200 OS: Ubuntu 24.04.4 LTS Cmdline: BOOT_IMAGE=/boot/vmlinuz-6.8.0-134-generic root=UUID=<redacted> ro maybe-ubiquity (no intel_iommu= or iommu= parameter set; IOMMU on by default, 132 groups) Workload: KVM/libvirt host, ~30 VMs, LVM on hpsa RAID-5 + LVM on NVMe mdraid1 ``` **Control machine:** a second, identical DL360p Gen8 (same BIOS, same P420i) running 6.8.0-101 has 18+ weeks uptime with zero DMAR errors. It has not been exposed to -134 and is available to test a proposed kernel. I can also reproduce on the primary host in a maintenance window if that is more useful. ## Related reports Same signature (`DMA PTE ... already set` from a RAID driver's scatter-gather mapping, leading to controller resets) reported downstream on Proxmox, where a Proxmox developer states "the -29 one had an IOMMU related regression that is fixed there", referring to their 6.8.12-30 kernel: - https://forum.proxmox.com/threads/has-anyone-seen-dma-pte-megaraid-controller-resets-after-upgrading-to-recent-proxmox-8-kernels.184389/ - https://forum.proxmox.com/threads/proxmox-kernel-6-8-12-29-causing-array-controller-errors.184181/ Proxmox 6.8.12-29/-30 share the 6.8.12 base with Ubuntu's 6.8.0-134. If the fix there is a revert of an upstream stable commit, the same revert would plausibly apply here. **This is an inference from the shared code base and matching signature, not something I was able to confirm** — the referenced upstream report (kernel bugzilla #221234) is behind an anti-bot gate, and I could not identify the offending commit. An older, still-untriaged Ubuntu bug with the same signature: https://bugs.launchpad.net/bugs/1970453 ## Workarounds - Boot 6.8.0-124 (what we did; fully effective) - `iommu=pt` or `intel_iommu=off` reported effective by others (untested here) None of these are acceptable long-term for hosts that must stay on a supported kernel. ## Request Please identify and revert/backport the fix for the RMRR reserved-region handling in the 6.8.0 Noble series. ## Attachments and how they were prepared The attached logs have been **filtered before upload** — please read this so you know what you are and are not looking at: - `[UFW BLOCK]` firewall lines were **removed entirely**. They carried third-party source IP addresses (personal data under GDPR) and were unrelated to the fault. - Hostname, IP addresses, MAC addresses and UUIDs were replaced with placeholders. - Full `dmidecode` output is **not attached**: it embeds the chassis serial number three times over (including as a raw hex dump), and the system UUID is a hex encoding of that same serial. `hardware-summary.txt` contains the model-level fields instead. No kernel-relevant line was altered. The evidence was counted before and after filtering to confirm nothing was lost: 66 DMAR errors, 66 `iommu.c:2227` warnings, 132 `__domain_mapping` frames, 23 distinct vPFNs, 9 RMRR regions and the complete hpsa reset-to-offline chain are all present in both. **If you need an unfiltered log, ask me and I will provide one through a private channel.** - `dmesg-6.8.0-134-BAD.log` — full kernel log of the failing boot - `dmesg-6.8.0-124-GOOD-boot.log` — boot log of the working kernel, showing the identical RMRR regions and no collisions - `lspci-vvnn.txt` — full PCI topology - `hardware-summary.txt` — curated hardware/firmware details - `version.txt` — kernel, cmdline, distro, hpsa module info ** Affects: linux (Ubuntu) Importance: Undecided Status: New ** Tags: iommu kernel-bug noble regression-update -- You received this bug notification because you are subscribed to linux in Ubuntu. Matching subscriptions: Bgg, Bmail, Nb https://bugs.launchpad.net/bugs/2160616 Title: [regression] 6.8.0-134: IOVA allocator hands out addresses inside BIOS RMRR regions, causing DMA mapping failures and storage controller loss (HP ProLiant Gen8 / P420i) Status in linux package in Ubuntu: New Bug description: I updated my DL360 gen8 server to the newest kernel and got a non functioning P420i due to massive errors in dmesg. Since I'm not that familiar with kernel stuff and storage drivers I've asked an LLM to provide all necessary information to file a bug against the ubuntu kernel package. This is the result of the information that was already found. Let me know if you need any further information on this issue. # [regression] 6.8.0-134: IOVA allocator hands out addresses inside BIOS RMRR regions, causing DMA mapping failures and storage controller loss (HP ProLiant Gen8 / P420i) **Package:** linux (Ubuntu Noble 24.04) **Broken:** 6.8.0-134.134 (6.8.12) **Last good:** 6.8.0-124.124 **Also present in:** 6.8.0-136.136 (per changelog; no fix/revert present) **Severity:** total storage loss on affected hosts — 32 TB RAID volume dropped offline, 10 production VMs lost their disks --- > **Note on how this report was produced.** The diagnosis, log analysis and the > text of this report were assembled with the help of an AI assistant (Claude) > working on the affected machine under my supervision. I have reviewed it and > can vouch for the underlying facts; the raw logs are attached so every claim > can be checked independently. The address/RMRR correlation below was verified > programmatically against the raw log rather than by eye. > > **I am happy to supply any additional information, run diagnostics, or test a > proposed kernel on request** — see the "Control machine" note near the end. --- ## Summary On 6.8.0-134, the Intel IOMMU IOVA allocator hands out IOVAs that fall inside BIOS-declared RMRR regions. Those regions are already identity-mapped, so `__domain_mapping()` finds an existing PTE, throws `WARNING at drivers/iommu/intel/iommu.c:2227`, and the DMA mapping fails: ``` WARNING: CPU: 32 PID: 562 at drivers/iommu/intel/iommu.c:2227 __domain_mapping+0x2b5/0x320 DMAR: ERROR: DMA PTE for vPFN 0xbdf81 already set (to bdf81003 not 128759003) ``` Note `bdf81003` — vPFN 0xbdf81 is mapped to PFN 0xbdf81, i.e. an identity mapping, which is what RMRR regions get. This is **not driver-specific**. On this host it hit three drivers within 90 seconds: `nvme` first, then `hpsa`, then `tg3`. The HP Smart Array P420i was simply the one that could not recover: its logical volume reset failed, the device went offline (`DID_NO_CONNECT`), and the 32 TB RAID-5 volume backing our LVM volume group disappeared, taking 10 production VMs with it. ## Root cause evidence The BIOS declares these RMRR regions (identical on both kernels, see below): ``` DMAR: RMRR base: 0x000000bdf6e000 end: 0x000000bdf6efff -> PFN 0xbdf6e DMAR: RMRR base: 0x000000bdf6f000 end: 0x000000bdf7efff -> PFN 0xbdf6f-0xbdf7e DMAR: RMRR base: 0x000000bdf7f000 end: 0x000000bdf82fff -> PFN 0xbdf7f-0xbdf82 DMAR: RMRR base: 0x000000bdf83000 end: 0x000000bdf84fff -> PFN 0xbdf83-0xbdf84 union -> PFN 0xbdf6e-0xbdf84 ``` The complete set of colliding vPFNs observed across all 66 errors: ``` 0xbdf6e 0xbdf6f 0xbdf70 0xbdf71 0xbdf72 0xbdf73 0xbdf74 0xbdf75 0xbdf76 0xbdf77 0xbdf78 0xbdf79 0xbdf7a 0xbdf7b 0xbdf7c 0xbdf7d 0xbdf7e 0xbdf7f 0xbdf80 0xbdf81 0xbdf82 0xbdf83 0xbdf84 ``` That is `0xbdf6e`-`0xbdf84` — an exact, complete match with the RMRR union. All 23 distinct colliding IOVAs lie inside an RMRR region; none lies outside one. (Checked programmatically over the attached log, not by inspection.) The RMRR reserved-region exclusion in the IOVA allocator appears not to be in effect. ## This is not a firmware or BIOS problem The RMRR regions declared by the BIOS are **byte-for-byte identical** on 6.8.0-124 and 6.8.0-134 (diffed; both logs attached). Same machine, same BIOS (P71, 05/24/2019), same controller firmware. The only variable is the kernel. 6.8.0-124 handles these exact regions correctly. ## This is not passthrough-specific No VFIO/PCI passthrough is configured on this host (`lsmod | grep vfio` -> 0). This is a plain KVM/libvirt host using virtio and LVM-backed disks. Reports elsewhere tie the 6.8.12 IOMMU regression to VFIO group assignment; this reproducer has no VFIO involvement at all, which suggests the underlying defect is broader than the passthrough symptom it was first noticed through. ## A/B evidence (same physical machine, consecutive boots) | Boot | Kernel | Uptime | DMAR errors | I/O errors | |------|--------|--------|-------------|------------| | -2 | 6.8.0-124 | 2026-06-13 -> 2026-07-14 (31 days) | 0 | 0 | | -1 | 6.8.0-134 | ~10 min to controller loss | 66 | 4701 | | 0 | 6.8.0-124 | since revert, stable | 0 | 0 | Booting 6.8.0-124 restored full function immediately: controller status OK, volume group back, all VMs recovered. ## Timeline (boot -1, kernel 6.8.0-134) ``` 12:32:34 boot 12:41:24 first WARNING - nvme_map_data (mdraid rebuild driving heavy DMA) 12:42:46 hpsa_scatter_gather hits the same warning 12:42:50 tg3_start_xmit hits it too (network path) 12:43:23 hpsa: resetting logical Direct-Access HP LOGICAL VOLUME RAID-5 12:44:32 hpsa: reset logical failed / sd 3:1:0:0: rejecting I/O to offline device sda offline -> volume group gone -> 10 VMs lose their disks 13:25:05 reboot into 6.8.0-124 -> fully recovered ``` The first failure came ~9 minutes in, under sustained DMA load from an mdraid rebuild — consistent with the allocator only reaching the RMRR range once lower IOVA space was under pressure. Hosts may therefore appear fine after boot and fail later under load. ## Representative stack trace ``` WARNING: CPU: 32 PID: 562 at drivers/iommu/intel/iommu.c:2227 __domain_mapping+0x2b5/0x320 CPU: 32 PID: 562 Comm: kworker/32:1H Tainted: G I 6.8.0-134-generic #134-Ubuntu Hardware name: HP ProLiant DL360p Gen8, BIOS P71 05/24/2019 Workqueue: kblockd blk_mq_run_work_fn RIP: 0010:__domain_mapping+0x2b5/0x320 Call Trace: <TASK> intel_iommu_map_pages+0xe1/0x140 __iommu_map+0x121/0x280 iommu_map_sg+0xbf/0x1f0 iommu_dma_map_sg+0x463/0x4f0 __dma_map_sg_attrs+0x35/0xd0 dma_map_sgtable+0x1d/0x40 nvme_map_data+0xd6/0x480 [nvme] nvme_prep_rq.part.0+0x35/0x120 [nvme] nvme_queue_rq+0x7e/0x207 [nvme] blk_mq_dispatch_rq_list+0x137/0x520 __blk_mq_sched_dispatch_requests+0x118/0x190 blk_mq_sched_dispatch_requests+0x2f/0x80 blk_mq_run_work_fn+0x72/0x90 process_one_work+0x184/0x3a0 worker_thread+0x18b/0x330 kthread+0xf2/0x120 ret_from_fork+0x47/0x70 ret_from_fork_asm+0x1b/0x30 </TASK> ---[ end trace 0000000000000000 ]--- DMAR: ERROR: DMA PTE for vPFN 0xbdf81 already set (to bdf81003 not 128759003) ``` The same warning with `hpsa_scatter_gather+0x3e/0x310 [hpsa]` and with `__tg3_start_xmit+0x1f0/0xd60 [tg3]` in place of the nvme frames is in the attached log (46 nvme, 18 hpsa, 4 tg3 occurrences). ## Environment ``` Hardware: HP ProLiant DL360p Gen8 BIOS: HP P71, 05/24/2019 CPU: Intel Xeon E5-2697 v2 @ 2.70GHz (48 threads) RAM: 503 GB Controller: Smart Array P420i [103c:323b] rev 01, subsystem [103c:3354] PCI 0000:02:00.0, IOMMU group 49 Driver: hpsa 3.4.20-200 OS: Ubuntu 24.04.4 LTS Cmdline: BOOT_IMAGE=/boot/vmlinuz-6.8.0-134-generic root=UUID=<redacted> ro maybe-ubiquity (no intel_iommu= or iommu= parameter set; IOMMU on by default, 132 groups) Workload: KVM/libvirt host, ~30 VMs, LVM on hpsa RAID-5 + LVM on NVMe mdraid1 ``` **Control machine:** a second, identical DL360p Gen8 (same BIOS, same P420i) running 6.8.0-101 has 18+ weeks uptime with zero DMAR errors. It has not been exposed to -134 and is available to test a proposed kernel. I can also reproduce on the primary host in a maintenance window if that is more useful. ## Related reports Same signature (`DMA PTE ... already set` from a RAID driver's scatter-gather mapping, leading to controller resets) reported downstream on Proxmox, where a Proxmox developer states "the -29 one had an IOMMU related regression that is fixed there", referring to their 6.8.12-30 kernel: - https://forum.proxmox.com/threads/has-anyone-seen-dma-pte-megaraid-controller-resets-after-upgrading-to-recent-proxmox-8-kernels.184389/ - https://forum.proxmox.com/threads/proxmox-kernel-6-8-12-29-causing-array-controller-errors.184181/ Proxmox 6.8.12-29/-30 share the 6.8.12 base with Ubuntu's 6.8.0-134. If the fix there is a revert of an upstream stable commit, the same revert would plausibly apply here. **This is an inference from the shared code base and matching signature, not something I was able to confirm** — the referenced upstream report (kernel bugzilla #221234) is behind an anti-bot gate, and I could not identify the offending commit. An older, still-untriaged Ubuntu bug with the same signature: https://bugs.launchpad.net/bugs/1970453 ## Workarounds - Boot 6.8.0-124 (what we did; fully effective) - `iommu=pt` or `intel_iommu=off` reported effective by others (untested here) None of these are acceptable long-term for hosts that must stay on a supported kernel. ## Request Please identify and revert/backport the fix for the RMRR reserved-region handling in the 6.8.0 Noble series. ## Attachments and how they were prepared The attached logs have been **filtered before upload** — please read this so you know what you are and are not looking at: - `[UFW BLOCK]` firewall lines were **removed entirely**. They carried third-party source IP addresses (personal data under GDPR) and were unrelated to the fault. - Hostname, IP addresses, MAC addresses and UUIDs were replaced with placeholders. - Full `dmidecode` output is **not attached**: it embeds the chassis serial number three times over (including as a raw hex dump), and the system UUID is a hex encoding of that same serial. `hardware-summary.txt` contains the model-level fields instead. No kernel-relevant line was altered. The evidence was counted before and after filtering to confirm nothing was lost: 66 DMAR errors, 66 `iommu.c:2227` warnings, 132 `__domain_mapping` frames, 23 distinct vPFNs, 9 RMRR regions and the complete hpsa reset-to-offline chain are all present in both. **If you need an unfiltered log, ask me and I will provide one through a private channel.** - `dmesg-6.8.0-134-BAD.log` — full kernel log of the failing boot - `dmesg-6.8.0-124-GOOD-boot.log` — boot log of the working kernel, showing the identical RMRR regions and no collisions - `lspci-vvnn.txt` — full PCI topology - `hardware-summary.txt` — curated hardware/firmware details - `version.txt` — kernel, cmdline, distro, hpsa module info To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2160616/+subscriptions
Комментариев нет:
Отправить комментарий