Thanks for confirming this. I can also confirm that the equivalent interrupt-ordering workaround works on Ubuntu 26.04.1 with 7.0.0-31-generic on my MSI Vector with Intel VMD 8086:ad0b (Arrow Lake-HX). I am currently running 7.0.0-31-generic with the patched VMD module via DKMS. The recurring nvme: I/O ... timeout, completion polled events are gone and NVMe performance is normal. My original workaround was developed for 8086:ad0b / ARL004 and performs the same essential operation: a PCI configuration-space read from the MSI-initiating NVMe device before dispatching the child interrupt. So we now have successful results on both 8086:7d0b and 8086:ad0b under Ubuntu kernel 7.0.0-31-generic. С уважением Владимир Недошивин Tel: +79656931076 > > Вторник, 8 сентября 2026, 00:00 +04:00 от Stas Safari <2166325@bugs.launchpad.net > >: > Confirming the same issue on an Arrow Lake laptop with VMD 8086:7d0b > (Ubuntu 26.04.1, kernel 7.0.0-31-generic): frequent "nvme: I/O ... > timeout, completion polled" events causing 30 s – 2 min application > stalls. APST off, C-state limits, kernel/BIOS/firmware updates did not > help. > > Applying Rickey Bartlett's "PCI: vmd: Flush initiator posted writes before > demuxing interrupts on Meteor Lake" (pci.git controller/vmd) to Ubuntu's > vmd.c via DKMS fixed it completely: 0 events, I/O pressure from ~40% to > ~0%. Please consideran SRU cherry-pick for 7.0.x. > > -- > You received this bug notification because you are subscribed to the bug > report. > https://bugs.launchpad.net/bugs/2166325 > > Title: > Intel VMD 8086:ad0b: Bus Offset 3 prevents NVMe enumeration and ARL004 > causes NVMe completion stalls > > Status in linux package in Ubuntu: > Confirmed > > Bug description: > Hardware > ======== > > Laptop: MSI Vector 17 HX AI > Platform: Intel Arrow Lake > Intel VMD: 8086:ad0b > VMD PCI address: 0000:00:0e.0 > Subsystem: 1462:149c > > Internal NVMe SSD: > Phison 1TB ESR01TBYCCA4-EDJ-2MS > Controller: Phison PS5029-E29T PCIe 4.0 NVMe > PCI ID: 1987:5029 (rev 01) > Firmware: ETFM50.0 > > Ubuntu: 26.04 > Current/final tested kernel: 7.0.0-30-generic > Secure Boot: enabled > > > Problem 1: Intel VMD Bus Offset 3 > ================================ > > With the stock Ubuntu VMD driver the internal NVMe SSD is not exposed. > > The kernel reports: > > vmd 0000:00:0e.0: Unknown Bus Offset Setting (3) > > and no internal /dev/nvme* device is available. > > Linux 6.14 was also tested on this machine and did not solve the VMD > problem. > > The system firmware does not expose a usable BIOS option to disable Intel > VMD/RST. > > Intel's public VMD second-rootbus patch series was integrated into the > Ubuntu VMD source: > > [PATCH v3 0/8] VMD add second rootbus support > > https://www.spinics.net/lists/linux-pci/msg163096.html > > After integrating the second-rootbus support, Linux successfully > enumerates > the internal NVMe controller and SSD. > > > Problem 2: NVMe completion stalls > ================================= > > After the SSD became accessible through the modified VMD driver, a second > problem became visible. > > Kernel messages repeatedly contained: > > nvme nvme0: I/O tag ... timeout, completion polled > > The stalls occurred at approximately 30-second intervals. > > A diagnostic sequential read before the workaround: > > sudo dd if=/dev/nvme0n1 of=/dev/null bs=16M count=64 > status=progress > > Result: > > 1 GiB in approximately 121.339 seconds > approximately 8.8 MB/s > > CPU and memory were not saturated. > > Disabling ASPM was tested and did not solve the problem. > > > ARL004 investigation > ==================== > > The official Intel RST/VMD Windows driver distributed by MSI for this > machine was examined by static analysis for comparison. > > The relevant Windows driver path performs a PCI configuration-space read > after MSI handling when the completion state requires ordering. > > This behavior is consistent with the Intel Arrow Lake ARL004 erratum, > where > an MSI from a VMD-owned device may pass a preceding memory write. > > An experimental Linux workaround was implemented for the tested > 8086:ad0b VMD/NVMe path. It performs a dummy PCI configuration-space read > before the subsequent interrupt handling path. > > > Result > ====== > > With the experimental ARL004 workaround: > > sudo dd if=/dev/nvme0n1 of=/dev/null bs=16M count=64 > status=progress > > completed in: > > 1 GiB in 0.88206 seconds > approximately 1.2 GB/s > > A subsequent 10 GiB sequential diagnostic read also completed > normally. > > After booting with the final tested module: > > sudo journalctl -k -b | grep -E 'timeout|completion polled' > > produced no matching messages. > > The approximately 136x difference above is only a diagnostic comparison on > > this specific machine and is not intended as a general SSD benchmark. > > > Reproduction / reference implementation > ======================================= > > The complete investigation, tested source, separate patches, combined > patch, > known-working module, SHA256 checksums, installation/bootstrap procedure > and > recovery procedure are published here: > > https://github.com/gvozd188/vmd-arl004 > > Exact reference commit: > > https://github.com/gvozd188/vmd-arl004/commit/5438c83 > > Relevant files: > > 0001-vmd-second-rootbus-intel.patch > Integration of Intel's second-rootbus support. > > 0002-vmd-ad0b-arl004-workaround.patch > Experimental ARL004 workaround. > > vmd-arl004.patch > Combined patch. > > vmd-ubuntu-7.0.0-30.c > Ubuntu VMD source used as the patch base. > > vmd.c > Final tested source. > > vmd.ko > Known-working reference module for 7.0.0-30-generic. > > > Known-working source SHA256: > > 0a27aa1379e8b7fc22f509b8d06ed3676a211d7bb3226a3f9b3dadfc208be7a3 > > Known-working module SHA256: > > b2c9eacb720d45fcaab15ff09ad3fdc2e97b8a6dd215c80fa282090ff8d2b2cd > > > Installation history > ==================== > > Because the stock VMD driver could not expose the internal SSD, an > existing > Ubuntu HDD from another laptop was connected to the MSI Vector through a > SATA-to-USB adapter. > > The MSI Vector was booted directly from that HDD. That Ubuntu environment > was running kernel 7.0.0-30-generic. > > The modified VMD driver was used there to expose the internal NVMe > SSD. > > Ubuntu was then manually deployed onto the dedicated Ubuntu partition of > the > internal SSD without modifying the existing Windows and data partitions. > > The manually deployed SSD installation initially contained kernel > 7.0.0-14-generic. This kernel was not intentionally selected as a VMD > workaround; it was simply the kernel present in the manually deployed > system > at that stage. > > A compatible custom VMD module was installed for 7.0.0-14-generic and > added > to its initramfs, allowing the first independent boot from the internal > SSD. > > Only after that successful SSD boot was DKMS configured. > > The SSD installation was subsequently updated to 7.0.0-30-generic, where > the > final VMD/ARL004 development and testing was performed. > > > Expected result > =============== > > The stock Ubuntu kernel should: > > 1. correctly enumerate the second-rootbus / Bus Offset 3 topology on Intel > > VMD 8086:ad0b; > 2. expose the internal NVMe SSD; > 3. handle NVMe completion ordering without repeated approximately > 30-second > completion timeouts. > > > Actual result > ============= > > With the stock VMD driver, the internal SSD is not exposed because Bus > Offset Setting 3 is rejected. > > After adding second-rootbus support alone, the SSD becomes visible but > repeated NVMe completion stalls occur on this hardware. > > The experimental ARL004 ordering workaround eliminates the observed stalls > > on the tested system. > > > Notes > ===== > > The ARL004 workaround is experimental and platform-specific. I am > reporting > the observed hardware behavior and the tested workaround rather than > claiming that this implementation is the appropriate final upstream fix. > > The Windows driver binary is not redistributed in the GitHub repository. > Only identification information, hashes and static-analysis notes are > provided. > > ProblemType: Bug > DistroRelease: Ubuntu 26.04 > Package: linux-image-7.0.0-30-generic 7.0.0-30.30 > ProcVersionSignature: Ubuntu 7.0.0-30.30-generic 7.0.12 > Uname: Linux 7.0.0-30-generic x86_64 > ApportVersion: 2.34.1-0ubuntu0.1 > Architecture: amd64 > AudioDevicesInUse: > USER PID ACCESS COMMAND > /dev/snd/controlC1: gvozd188 2323 F.... pipewire > gvozd188 2360 F.... wireplumber > /dev/snd/controlC0: gvozd188 2360 F.... wireplumber > /dev/snd/seq: gvozd188 2323 F.... pipewire > CasperMD5CheckResult: unknown > CurrentDesktop: ubuntu:GNOME > Date: Thu Sep 3 13:08:10 2026 > MachineType: Micro-Star International Co., Ltd. Vector 17 HX AI A2XWIG > ProcFB: 0 i915drmfb > ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-7.0.0-30-generic > root=UUID=fef65fb9-b1c4-4bbe-9c02-92a4a7280fee ro quiet splash iommu=pt > crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M > > SourcePackage: linux > UpgradeStatus: No upgrade log present (probably fresh install) > WifiSyslog: > > dmi.bios.date: 04/20/2026 > dmi.bios.release: 1.18 > dmi.bios.vendor: American Megatrends International, LLC. > dmi.bios.version: E17S3IMS.112 > dmi.board.asset.tag: Default string > dmi.board.name: MS-17S3 > dmi.board.vendor: Micro-Star International Co., Ltd. > dmi.board.version: REV:1.0 > dmi.chassis.asset.tag: No Asset Tag > dmi.chassis.type: 10 > dmi.chassis.vendor: Micro-Star International Co., Ltd. > dmi.chassis.version: N/A > dmi.modalias: > dmi:bvnAmericanMegatrendsInternational,LLC.:bvrE17S3IMS.112:bd04/20/2026:br1.18:svnMicro-StarInternationalCo.,Ltd.:pnVector17HXAIA2XWIG:pvrREV1.0:rvnMicro-StarInternationalCo.,Ltd.:rnMS-17S3:rvrREV1.0:cvnMicro-StarInternationalCo.,Ltd.:ct10:cvrN/A:sku17S3.1:pfaVector: > > dmi.product.family: Vector > dmi.product.name: Vector 17 HX AI A2XWIG > dmi.product.sku: 17S3.1 > dmi.product.version: REV:1.0 > dmi.sys.vendor: Micro-Star International Co., Ltd. > > To manage notifications about this bug go to: > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2166325/+subscriptions > > -- You received this bug notification because you are subscribed to linux in Ubuntu. Matching subscriptions: Bgg, Bmail, Nb https://bugs.launchpad.net/bugs/2166325 Title: Intel VMD 8086:ad0b: Bus Offset 3 prevents NVMe enumeration and ARL004 causes NVMe completion stalls Status in linux package in Ubuntu: Confirmed Bug description: Hardware ======== Laptop: MSI Vector 17 HX AI Platform: Intel Arrow Lake Intel VMD: 8086:ad0b VMD PCI address: 0000:00:0e.0 Subsystem: 1462:149c Internal NVMe SSD: Phison 1TB ESR01TBYCCA4-EDJ-2MS Controller: Phison PS5029-E29T PCIe 4.0 NVMe PCI ID: 1987:5029 (rev 01) Firmware: ETFM50.0 Ubuntu: 26.04 Current/final tested kernel: 7.0.0-30-generic Secure Boot: enabled Problem 1: Intel VMD Bus Offset 3 ================================ With the stock Ubuntu VMD driver the internal NVMe SSD is not exposed. The kernel reports: vmd 0000:00:0e.0: Unknown Bus Offset Setting (3) and no internal /dev/nvme* device is available. Linux 6.14 was also tested on this machine and did not solve the VMD problem. The system firmware does not expose a usable BIOS option to disable Intel VMD/RST. Intel's public VMD second-rootbus patch series was integrated into the Ubuntu VMD source: [PATCH v3 0/8] VMD add second rootbus support https://www.spinics.net/lists/linux-pci/msg163096.html After integrating the second-rootbus support, Linux successfully enumerates the internal NVMe controller and SSD. Problem 2: NVMe completion stalls ================================= After the SSD became accessible through the modified VMD driver, a second problem became visible. Kernel messages repeatedly contained: nvme nvme0: I/O tag ... timeout, completion polled The stalls occurred at approximately 30-second intervals. A diagnostic sequential read before the workaround: sudo dd if=/dev/nvme0n1 of=/dev/null bs=16M count=64 status=progress Result: 1 GiB in approximately 121.339 seconds approximately 8.8 MB/s CPU and memory were not saturated. Disabling ASPM was tested and did not solve the problem. ARL004 investigation ==================== The official Intel RST/VMD Windows driver distributed by MSI for this machine was examined by static analysis for comparison. The relevant Windows driver path performs a PCI configuration-space read after MSI handling when the completion state requires ordering. This behavior is consistent with the Intel Arrow Lake ARL004 erratum, where an MSI from a VMD-owned device may pass a preceding memory write. An experimental Linux workaround was implemented for the tested 8086:ad0b VMD/NVMe path. It performs a dummy PCI configuration-space read before the subsequent interrupt handling path. Result ====== With the experimental ARL004 workaround: sudo dd if=/dev/nvme0n1 of=/dev/null bs=16M count=64 status=progress completed in: 1 GiB in 0.88206 seconds approximately 1.2 GB/s A subsequent 10 GiB sequential diagnostic read also completed normally. After booting with the final tested module: sudo journalctl -k -b | grep -E 'timeout|completion polled' produced no matching messages. The approximately 136x difference above is only a diagnostic comparison on this specific machine and is not intended as a general SSD benchmark. Reproduction / reference implementation ======================================= The complete investigation, tested source, separate patches, combined patch, known-working module, SHA256 checksums, installation/bootstrap procedure and recovery procedure are published here: https://github.com/gvozd188/vmd-arl004 Exact reference commit: https://github.com/gvozd188/vmd-arl004/commit/5438c83 Relevant files: 0001-vmd-second-rootbus-intel.patch Integration of Intel's second-rootbus support. 0002-vmd-ad0b-arl004-workaround.patch Experimental ARL004 workaround. vmd-arl004.patch Combined patch. vmd-ubuntu-7.0.0-30.c Ubuntu VMD source used as the patch base. vmd.c Final tested source. vmd.ko Known-working reference module for 7.0.0-30-generic. Known-working source SHA256: 0a27aa1379e8b7fc22f509b8d06ed3676a211d7bb3226a3f9b3dadfc208be7a3 Known-working module SHA256: b2c9eacb720d45fcaab15ff09ad3fdc2e97b8a6dd215c80fa282090ff8d2b2cd Installation history ==================== Because the stock VMD driver could not expose the internal SSD, an existing Ubuntu HDD from another laptop was connected to the MSI Vector through a SATA-to-USB adapter. The MSI Vector was booted directly from that HDD. That Ubuntu environment was running kernel 7.0.0-30-generic. The modified VMD driver was used there to expose the internal NVMe SSD. Ubuntu was then manually deployed onto the dedicated Ubuntu partition of the internal SSD without modifying the existing Windows and data partitions. The manually deployed SSD installation initially contained kernel 7.0.0-14-generic. This kernel was not intentionally selected as a VMD workaround; it was simply the kernel present in the manually deployed system at that stage. A compatible custom VMD module was installed for 7.0.0-14-generic and added to its initramfs, allowing the first independent boot from the internal SSD. Only after that successful SSD boot was DKMS configured. The SSD installation was subsequently updated to 7.0.0-30-generic, where the final VMD/ARL004 development and testing was performed. Expected result =============== The stock Ubuntu kernel should: 1. correctly enumerate the second-rootbus / Bus Offset 3 topology on Intel VMD 8086:ad0b; 2. expose the internal NVMe SSD; 3. handle NVMe completion ordering without repeated approximately 30-second completion timeouts. Actual result ============= With the stock VMD driver, the internal SSD is not exposed because Bus Offset Setting 3 is rejected. After adding second-rootbus support alone, the SSD becomes visible but repeated NVMe completion stalls occur on this hardware. The experimental ARL004 ordering workaround eliminates the observed stalls on the tested system. Notes ===== The ARL004 workaround is experimental and platform-specific. I am reporting the observed hardware behavior and the tested workaround rather than claiming that this implementation is the appropriate final upstream fix. The Windows driver binary is not redistributed in the GitHub repository. Only identification information, hashes and static-analysis notes are provided. ProblemType: Bug DistroRelease: Ubuntu 26.04 Package: linux-image-7.0.0-30-generic 7.0.0-30.30 ProcVersionSignature: Ubuntu 7.0.0-30.30-generic 7.0.12 Uname: Linux 7.0.0-30-generic x86_64 ApportVersion: 2.34.1-0ubuntu0.1 Architecture: amd64 AudioDevicesInUse: USER PID ACCESS COMMAND /dev/snd/controlC1: gvozd188 2323 F.... pipewire gvozd188 2360 F.... wireplumber /dev/snd/controlC0: gvozd188 2360 F.... wireplumber /dev/snd/seq: gvozd188 2323 F.... pipewire CasperMD5CheckResult: unknown CurrentDesktop: ubuntu:GNOME Date: Thu Sep 3 13:08:10 2026 MachineType: Micro-Star International Co., Ltd. Vector 17 HX AI A2XWIG ProcFB: 0 i915drmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-7.0.0-30-generic root=UUID=fef65fb9-b1c4-4bbe-9c02-92a4a7280fee ro quiet splash iommu=pt crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) WifiSyslog: dmi.bios.date: 04/20/2026 dmi.bios.release: 1.18 dmi.bios.vendor: American Megatrends International, LLC. dmi.bios.version: E17S3IMS.112 dmi.board.asset.tag: Default string dmi.board.name: MS-17S3 dmi.board.vendor: Micro-Star International Co., Ltd. dmi.board.version: REV:1.0 dmi.chassis.asset.tag: No Asset Tag dmi.chassis.type: 10 dmi.chassis.vendor: Micro-Star International Co., Ltd. dmi.chassis.version: N/A dmi.modalias: dmi:bvnAmericanMegatrendsInternational,LLC.:bvrE17S3IMS.112:bd04/20/2026:br1.18:svnMicro-StarInternationalCo.,Ltd.:pnVector17HXAIA2XWIG:pvrREV1.0:rvnMicro-StarInternationalCo.,Ltd.:rnMS-17S3:rvrREV1.0:cvnMicro-StarInternationalCo.,Ltd.:ct10:cvrN/A:sku17S3.1:pfaVector: dmi.product.family: Vector dmi.product.name: Vector 17 HX AI A2XWIG dmi.product.sku: 17S3.1 dmi.product.version: REV:1.0 dmi.sys.vendor: Micro-Star International Co., Ltd. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2166325/+subscriptions
Комментариев нет:
Отправить комментарий