** Description changed: Package: linux (Ubuntu) Source Package: linux - Binary: linux-image-7.0.0-31-generic (Ubuntu 26.04 LTS / 24.04 HWE / 24.10) + Binary: linux-image-7.0.0-31-generic (Ubuntu 26.04 LTS Resolute / 24.04 HWE) Upstream Subsystem: drivers/thunderbolt/ (Native Host Interface & Software Connection Manager) - Affected Hardware: Intel Meteor Lake / Arrow Lake USB4 Host Interface [8086:7ec2 / 8086:7ec4], AMD Ryzen 6000/7000/8000 USB4 routers, ASMedia ASM2464PD, Intel Goshen Ridge / Titan Ridge bridges, and external PCIe NVMe direct-boot topologies. + Affected Hardware: Intel Meteor Lake / Arrow Lake USB4 Host Interface [8086:7ec2 / 8086:7ec4], AMD Hawk Point / Phoenix USB4 Host Interface [1022:1502 / 1022:1669], ASMedia ASM2464PD, and external PCIe NVMe direct-boot topologies. + Related Bug Trackers: Launchpad Bug #2078573 (Dell Latitude 5550), Bug #2159575 (ASUS Zenbook 14, dracut), Bug #2167764. + Upstream Commits: 59a54c5f3dbd & 0fc70886569c (Stable backport cc4c94a5f6c4). + Related Issue: CVE-2024-53194 (Use-after-free in pciehp on hot remove). ================================================================================ - 1. SUMMARY OF THE DEFECT + 1. SUMMARY OF THE OBSERVATION ================================================================================ - When direct-booting Linux from an external NVMe SSD over a USB4/Thunderbolt 4 PCIe Gen 4 x4 tunnel, motherboard UEFI firmware negotiates the link and builds the PCIe tunnel. GRUB2 executes and loads vmlinuz and initrd into host RAM across this tunnel. + When booting Linux directly from an external NVMe SSD over a USB4/Thunderbolt 4 PCIe Gen 4 x4 tunnel, motherboard UEFI firmware negotiates the link and establishes the PCIe tunnel. GRUB2 executes and loads vmlinuz and initrd into host RAM across this tunnel. However, during early kernel initialization inside the initramfs, - thunderbolt.ko issues an unconditional Host Router Reset - (host_reset=true). This severs the pre-boot PCIe tunnel mid-boot, - causing nvme_probe() to encounter Master Abort (0xFFFFFFFF) and return - terminal error -ENODEV. Because the Linux driver core does not re-probe - devices that failed with -ENODEV, the root filesystem device disappears - permanently from the kernel bus, causing an initramfs timeout and - emergency rescue shell drop ("Gave up waiting for root file system - device"). + thunderbolt.ko issues a Host Router Reset (host_reset=true). This drops + the pre-boot PCIe tunnel mid-boot, causing nvme_probe() to encounter + Master Abort (0xFFFFFFFF) and return terminal error -ENODEV. Because the + storage controller is disconnected during initial bus enumeration, the + root filesystem is not discovered, dropping the system into an emergency + rescue shell ("Gave up waiting for root file system device" / "Warning: + /dev/disk/by-uuid/<UUID> does not exist"). ================================================================================ - 2. FORENSIC ROOT CAUSE: THE TEARDOWN CASCADE + 2. TECHNICAL SEQUENCE DURING EARLY BOOT ================================================================================ - Tracing drivers/thunderbolt/nhi.c and drivers/thunderbolt/tb.c isolates the exact sequence: - + Tracing drivers/thunderbolt/nhi.c and drivers/thunderbolt/tb.c details the timing: 1. nhi_probe() (drivers/thunderbolt/nhi.c:1249): - Calls nhi_reset(nhi). On USB4 v2 controllers (REG_CAPS >= 0x40), because module parameter 'host_reset' defaults to true, it writes REG_RESET_HRR (BIT 0) to memory-mapped register REG_RESET (0x39898): - iowrite32(REG_RESET_HRR, nhi->iobase + REG_RESET); - This asserts a hardware Host Router Reset. Register ADP_PCIE_CS_0 bit ADP_PCIE_CS_0_PE (Path Enable, BIT 31) is de-asserted, physically cutting the PCIe tunnel. - + Calls nhi_reset(nhi). On USB4 v2 controllers (REG_CAPS >= 0x40), because module parameter 'host_reset' defaults to true, it writes REG_RESET_HRR (BIT 0) to memory-mapped register REG_RESET (0x39898). Register ADP_PCIE_CS_0 bit ADP_PCIE_CS_0_PE (Path Enable, BIT 31) is cleared, dropping the pre-boot PCIe tunnel. The Root Port clears Presence Detect State (PDS) and DL_Active. 2. tb_start() (drivers/thunderbolt/tb.c:3066-3070): - nhi_probe() invokes tb_domain_add(tb, host_reset), which calls tb_start(tb, reset = true). - tb_start() enforces: - if (reset && tb_switch_is_usb4(tb->root_switch)) { - discover = false; - if (usb4_switch_version(tb->root_switch) == 1) - tb_switch_reset(tb->root_switch); - } - Because discover is set to false, tb_discover_tunnels() and tb_scan_switch() are completely bypassed. - - 3. Asynchronous Driver Collision (drivers/nvme/host/pci.c): - Concurrently, nvme_probe() attempts to enumerate the storage controller at the pre-boot ACPI/PCI address. Because the tunnel has been severed: - nvme 0000:06:00.0: Unable to change power state from D3cold to D0, device inaccessible - nvme 0000:06:00.0: error -ENODEV: probe failed - Under Linux driver core semantics, an endpoint that fails probe with -ENODEV is never re-probed. Even when thunderbolt.ko eventually re-enumerates the enclosure seconds later, it generates thunderbolt bus uevents, not PCI bus uevents. The root partition UUID is never detected by dracut/systemd, rendering 'rootdelay=' parameters ineffective. + Because reset == true, tb_start() sets discover = false, bypassing native tunnel discovery (tb_discover_tunnels()). + 3. Driver Probing Timing (drivers/nvme/host/pci.c): + Concurrently, nvme_probe() attempts to enumerate the storage controller at the pre-boot address while the link is down. It receives Master Abort and returns terminal error -ENODEV. Under standard Linux driver core semantics, an endpoint that fails probing with -ENODEV is not automatically re-probed. ================================================================================ - 3. EMPIRICAL HARDWARE PROOF: BUILT-IN ADOPTION ALREADY EXISTS + 3. OBSERVATIONS ON USERSPACE AUTHORIZATION VS. IN-KERNEL TUNNEL PRESERVATION ================================================================================ - Empirical testing on live physical hardware (Intel Core Ultra 9 275HX Arrow Lake-HX with WD_BLACK SN7100 in ASM2464PD) proves that the Thunderbolt driver ALREADY POSSESSES complete architectural infrastructure to support pre-boot boot tunnels. + In Launchpad Bug #2078573, an initial hypothesis was considered where userspace tooling (such as boltd or udev rules inside the initramfs) might handle re-authorizing the device after the reset. - When booted with 'thunderbolt.host_reset=0': - - nhi_reset() is bypassed: the hardware PCIe tunnel remains uninterrupted. - - tb_start() retains discover = true. - - tb_discover_tunnels() executes tb_tunnel_discover_pci(), locates the active pre-boot PCIe tunnel, and marks intermediate switches as parent->boot = true. - - In tb_scan_finalize_switch(), sw->boot triggers automated switch authorization (sw->authorized = 1) before userspace uevents fire. - - Live sysfs telemetry confirms pre-boot adoption: - $ cat /sys/bus/thunderbolt/devices/0-1/boot - 1 - $ cat /sys/bus/thunderbolt/devices/0-1/authorized - 1 - - Performance & Flash Endurance when tunnel is preserved: - - PCIe Gen 4.0 x4 at 16.0 GT/s (~64 Gbps physical link). - - Buffered disk read: 3,587.60 MB/s; Direct sequential write: 2,024.33 MB/s. - - Host Memory Buffer (HMB): 64 MB host DDR5 RAM allocated via Intel VT-d IOMMU (Write Amplification Factor drops from 6.80 under UASP to 1.88 under native NVMe, extending NAND endurance by 72%). - - The default 'host_reset=true' policy simply short-circuits this - functional subsystem. + However, subsequent testing from duplicate Bug #2159575 (ASUS Zenbook 14 running Ubuntu 26.04 LTS Resolute on dracut) provides useful insights: + 1. Testing on Dracut: When testing Ubuntu 26.04 with dracut 110-11, the same early boot timeout occurred out of the box, showing that initramfs tooling updates alone do not resolve the timing race. + 2. PCI Driver Core Probe Lifecycle: In the initramfs emergency shell of Bug #2159575, reporter Lucas observed that manually authorizing the switch (echo 1 > /sys/bus/thunderbolt/.../authorized) did not bring up the NVMe device until followed by an explicit 'echo 1 > /sys/bus/pci/rescan'. Because bolt focuses strictly on domain security (/sys/bus/thunderbolt) and does not issue PCI bus rescans, handling this in userspace requires coordinating udev rules, D-Bus, and secondary bus rescans during early boot. + 3. In-Tree Discovery Logic: drivers/thunderbolt/tb.c already contains native infrastructure to discover pre-boot tunnels (tb_discover_tunnels()), which marks sw->boot = true and auto-authorizes devices in-kernel (sw->authorized = 1). When booted with 'thunderbolt.host_reset=0', this discovery path executes naturally, preserving the boot device without requiring userspace services. ================================================================================ - 4. UPSTREAM COMMIT GENEALOGY & MAINTAINER ASSUMPTIONS + 4. HARDWARE VERIFICATION & SILICON TELEMETRY ================================================================================ - - Commit 0fc70886569c (Mika Westerberg, Dec 2022): Introduced USB4 v2 host router reset via REG_RESET_HRR. - - Commit 59a54c5f3dbd (Sanath S / Mario Limonciello, Jan 2024): Defaulted host_reset to true to clear suboptimal boot firmware DisplayPort bandwidth tunnels (HBR2 vs HBR3) and reclaim exhausted AMD PCIe BAR space for docking stations. - - Commit 6faa39eea953 (Mika Westerberg, Feb 2024): Cemented 'discover = false' for USB4 host routers. - - Upstream maintainers operated under the unexamined assumption that all - USB4 devices are secondary, hotpluggable peripherals mounted after the - OS has booted from internal storage. They were unaware that the running - root filesystem could reside on the pre-boot PCIe tunnel. - - Related Vulnerability: CVE-2024-53194 documents how commits 0fc70886569c - and 59a54c5f3dbd cause unexpected PCIe presence drops triggering use- - after-free crashes in pciehp. + Testing on live physical hardware (Intel Core Ultra 9 275HX Arrow Lake-HX with WD_BLACK SN7100 in ASMedia ASM2464PD) confirms that preserving pre-boot tunnels maintains complete stability: + - With 'thunderbolt.host_reset=0': + * nhi_reset() is bypassed; hardware PCIe tunnel remains uninterrupted. + * tb_start() retains discover = true. + * tb_discover_tunnels() detects the pre-boot PCIe tunnel, sets parent->boot = true, and auto-authorizes the switch (sw->authorized = 1). + - Telemetry: + $ cat /sys/bus/thunderbolt/devices/0-1/boot -> 1 + $ cat /sys/bus/thunderbolt/devices/0-1/authorized -> 1 + - Performance: + * PCIe Gen 4 x4 link at 16.0 GT/s (~64 Gbps physical link). + * Buffered read: 3,587.60 MB/s; Direct write: 2,024.33 MB/s. + * Host Memory Buffer (HMB): 64 MB host DDR5 RAM cleanly allocated via Intel VT-d IOMMU (WAF dropped from 6.80 to 1.88, significantly extending NAND lifespan). ================================================================================ - 5. TESTED WORKAROUND (IMMEDIATE MITIGATION FOR USERS) + 5. PROPOSED UPSTREAM LINUX KERNEL PATCH ================================================================================ - Affected users direct-booting over USB4/TB4 can immediately work around the failure without recompiling the kernel by appending the following to GRUB_CMDLINE_LINUX in /etc/default/grub (or /etc/default/grub.d/99-usb4.cfg): + To allow the driver to distinguish between hotpluggable accessories (which benefit from a clean reset for DisplayPort renegotiation or MMIO allocation) and active boot storage (which must not be severed), drivers/thunderbolt/ can check whether an active pre-boot PCIe tunnel is present before issuing the reset: - thunderbolt.host_reset=0 thunderbolt.clx=0 pcie_port_pm=off - - Notes on parameters: - - thunderbolt.host_reset=0: Preserves the pre-boot UEFI PCIe tunnel across kernel handover. - - thunderbolt.clx=0 & pcie_port_pm=off: Prevents low-power link state retraining drops during early initqueue settling (may increase idle power draw by 1–3W on battery). - - ASM2464PD Retimer Cold Reset: If transitioning between ports or warm reboots fails to detect the drive in BIOS, a 30-second flea-power discharge (unplug AC, hold power button 30s) resets the high-speed retimer PHY state. - - ================================================================================ - 6. PROPOSED UPSTREAM LINUX KERNEL PATCH - ================================================================================ - We propose a two-tier safety guard in drivers/thunderbolt/: - 1. In nhi.c (nhi_has_active_boot_device()): Before issuing REG_RESET_HRR, check if sibling PCIe bridges on the root bus (external-facing or Thunderbolt ports) have active child devices populated by boot firmware. If active downstream devices exist, skip nhi_reset() and clear host_reset. - 2. In tb.c (tb_switch_has_active_pcie_tunnel()): In tb_start(), inspect whether any PCIe downstream adapter is enabled by boot firmware (tb_pci_port_is_enabled()). If active PCIe boot tunnels exist, do not set discover = false and do not reset the router. Allow tb_discover_tunnels() to adopt and authorize the boot storage. - - Engineering Suite & LKML Proposal Repository: - - GitHub: https://github.com/StickwoodJr/usb4-nvme-direct-boot - - Full Patch: https://github.com/StickwoodJr/usb4-nvme-direct-boot/blob/main/patches/0001-thunderbolt-preserve-pre-boot-pcie-tunnels.patch - - Automated Patch Verification Tool: scripts/apply_kernel_patch.sh (--check / --apply / --reverse) - - ```diff --- a/drivers/thunderbolt/nhi.c +++ b/drivers/thunderbolt/nhi.c @@ -1158,6 +1158,11 @@ static void nhi_reset(struct tb_nhi *nhi) return; } + if (nhi_has_active_boot_device(nhi)) { + dev_info(nhi->dev, "preserving pre-boot PCIe tunnel for active boot device\n"); + return; + } + iowrite32(REG_RESET_HRR, nhi->iobase + REG_RESET); msleep(100); - + } --- a/drivers/thunderbolt/tb.c +++ b/drivers/thunderbolt/tb.c @@ -3059,6 +3077,11 @@ static int tb_start(struct tb *tb, bool reset) tb_switch_tmu_enable(tb->root_switch); + if (tb_switch_has_active_pcie_tunnel(tb->root_switch)) { + tb_info(tb, "active PCIe boot tunnel detected, preserving topology\n"); + reset = false; + } + if (reset && tb_switch_is_usb4(tb->root_switch)) { discover = false; + if (usb4_switch_version(tb->root_switch) == 1) + + Reference standalone patch, tests, and packaging: + https://github.com/StickwoodJr/usb4-nvme-direct-boot + + ================================================================================ + 6. SUGGESTIONS FOR THE UBUNTU KERNEL TEAM + ================================================================================ + 1. Re-evaluate Launchpad Bug #2078573 under linux (Ubuntu): + Consider re-opening the kernel task in light of the -ENODEV probe timing findings and dracut test results, which indicate that kernel-side tunnel preservation is more robust than userspace initramfs hooks. + 2. Consider In-Kernel Tunnel Preservation: + Evaluate adopting conditional checks for active boot tunnels or documenting thunderbolt.host_reset=0 as the recommended setting for external direct-boot environments. + 3. Documentation: + Help provide guidance in Ubuntu release notes or documentation for users running external direct-boot NVMe configurations over USB4 / Thunderbolt. -- You received this bug notification because you are subscribed to linux in Ubuntu. Matching subscriptions: Bgg, Bmail, Nb https://bugs.launchpad.net/bugs/2167764 Title: thunderbolt.host_reset=1 default tears down pre-boot UEFI PCIe tunnels, breaking external NVMe direct boot over USB4 Status in linux package in Ubuntu: New Bug description: Package: linux (Ubuntu) Source Package: linux Binary: linux-image-7.0.0-31-generic (Ubuntu 26.04 LTS Resolute / 24.04 HWE) Upstream Subsystem: drivers/thunderbolt/ (Native Host Interface & Software Connection Manager) Affected Hardware: Intel Meteor Lake / Arrow Lake USB4 Host Interface [8086:7ec2 / 8086:7ec4], AMD Hawk Point / Phoenix USB4 Host Interface [1022:1502 / 1022:1669], ASMedia ASM2464PD, and external PCIe NVMe direct-boot topologies. Related Bug Trackers: Launchpad Bug #2078573 (Dell Latitude 5550), Bug #2159575 (ASUS Zenbook 14, dracut), Bug #2167764. Upstream Commits: 59a54c5f3dbd & 0fc70886569c (Stable backport cc4c94a5f6c4). Related Issue: CVE-2024-53194 (Use-after-free in pciehp on hot remove). ================================================================================ 1. SUMMARY OF THE OBSERVATION ================================================================================ When booting Linux directly from an external NVMe SSD over a USB4/Thunderbolt 4 PCIe Gen 4 x4 tunnel, motherboard UEFI firmware negotiates the link and establishes the PCIe tunnel. GRUB2 executes and loads vmlinuz and initrd into host RAM across this tunnel. However, during early kernel initialization inside the initramfs, thunderbolt.ko issues a Host Router Reset (host_reset=true). This drops the pre-boot PCIe tunnel mid-boot, causing nvme_probe() to encounter Master Abort (0xFFFFFFFF) and return terminal error -ENODEV. Because the storage controller is disconnected during initial bus enumeration, the root filesystem is not discovered, dropping the system into an emergency rescue shell ("Gave up waiting for root file system device" / "Warning: /dev/disk/by-uuid/<UUID> does not exist"). ================================================================================ 2. TECHNICAL SEQUENCE DURING EARLY BOOT ================================================================================ Tracing drivers/thunderbolt/nhi.c and drivers/thunderbolt/tb.c details the timing: 1. nhi_probe() (drivers/thunderbolt/nhi.c:1249): Calls nhi_reset(nhi). On USB4 v2 controllers (REG_CAPS >= 0x40), because module parameter 'host_reset' defaults to true, it writes REG_RESET_HRR (BIT 0) to memory-mapped register REG_RESET (0x39898). Register ADP_PCIE_CS_0 bit ADP_PCIE_CS_0_PE (Path Enable, BIT 31) is cleared, dropping the pre-boot PCIe tunnel. The Root Port clears Presence Detect State (PDS) and DL_Active. 2. tb_start() (drivers/thunderbolt/tb.c:3066-3070): Because reset == true, tb_start() sets discover = false, bypassing native tunnel discovery (tb_discover_tunnels()). 3. Driver Probing Timing (drivers/nvme/host/pci.c): Concurrently, nvme_probe() attempts to enumerate the storage controller at the pre-boot address while the link is down. It receives Master Abort and returns terminal error -ENODEV. Under standard Linux driver core semantics, an endpoint that fails probing with -ENODEV is not automatically re-probed. ================================================================================ 3. OBSERVATIONS ON USERSPACE AUTHORIZATION VS. IN-KERNEL TUNNEL PRESERVATION ================================================================================ In Launchpad Bug #2078573, an initial hypothesis was considered where userspace tooling (such as boltd or udev rules inside the initramfs) might handle re-authorizing the device after the reset. However, subsequent testing from duplicate Bug #2159575 (ASUS Zenbook 14 running Ubuntu 26.04 LTS Resolute on dracut) provides useful insights: 1. Testing on Dracut: When testing Ubuntu 26.04 with dracut 110-11, the same early boot timeout occurred out of the box, showing that initramfs tooling updates alone do not resolve the timing race. 2. PCI Driver Core Probe Lifecycle: In the initramfs emergency shell of Bug #2159575, reporter Lucas observed that manually authorizing the switch (echo 1 > /sys/bus/thunderbolt/.../authorized) did not bring up the NVMe device until followed by an explicit 'echo 1 > /sys/bus/pci/rescan'. Because bolt focuses strictly on domain security (/sys/bus/thunderbolt) and does not issue PCI bus rescans, handling this in userspace requires coordinating udev rules, D-Bus, and secondary bus rescans during early boot. 3. In-Tree Discovery Logic: drivers/thunderbolt/tb.c already contains native infrastructure to discover pre-boot tunnels (tb_discover_tunnels()), which marks sw->boot = true and auto-authorizes devices in-kernel (sw->authorized = 1). When booted with 'thunderbolt.host_reset=0', this discovery path executes naturally, preserving the boot device without requiring userspace services. ================================================================================ 4. HARDWARE VERIFICATION & SILICON TELEMETRY ================================================================================ Testing on live physical hardware (Intel Core Ultra 9 275HX Arrow Lake-HX with WD_BLACK SN7100 in ASMedia ASM2464PD) confirms that preserving pre-boot tunnels maintains complete stability: - With 'thunderbolt.host_reset=0': * nhi_reset() is bypassed; hardware PCIe tunnel remains uninterrupted. * tb_start() retains discover = true. * tb_discover_tunnels() detects the pre-boot PCIe tunnel, sets parent->boot = true, and auto-authorizes the switch (sw->authorized = 1). - Telemetry: $ cat /sys/bus/thunderbolt/devices/0-1/boot -> 1 $ cat /sys/bus/thunderbolt/devices/0-1/authorized -> 1 - Performance: * PCIe Gen 4 x4 link at 16.0 GT/s (~64 Gbps physical link). * Buffered read: 3,587.60 MB/s; Direct write: 2,024.33 MB/s. * Host Memory Buffer (HMB): 64 MB host DDR5 RAM cleanly allocated via Intel VT-d IOMMU (WAF dropped from 6.80 to 1.88, significantly extending NAND lifespan). ================================================================================ 5. PROPOSED UPSTREAM LINUX KERNEL PATCH ================================================================================ To allow the driver to distinguish between hotpluggable accessories (which benefit from a clean reset for DisplayPort renegotiation or MMIO allocation) and active boot storage (which must not be severed), drivers/thunderbolt/ can check whether an active pre-boot PCIe tunnel is present before issuing the reset: --- a/drivers/thunderbolt/nhi.c +++ b/drivers/thunderbolt/nhi.c @@ -1158,6 +1158,11 @@ static void nhi_reset(struct tb_nhi *nhi) return; } + if (nhi_has_active_boot_device(nhi)) { + dev_info(nhi->dev, "preserving pre-boot PCIe tunnel for active boot device\n"); + return; + } + iowrite32(REG_RESET_HRR, nhi->iobase + REG_RESET); msleep(100); } --- a/drivers/thunderbolt/tb.c +++ b/drivers/thunderbolt/tb.c @@ -3059,6 +3077,11 @@ static int tb_start(struct tb *tb, bool reset) tb_switch_tmu_enable(tb->root_switch); + if (tb_switch_has_active_pcie_tunnel(tb->root_switch)) { + tb_info(tb, "active PCIe boot tunnel detected, preserving topology\n"); + reset = false; + } + if (reset && tb_switch_is_usb4(tb->root_switch)) { discover = false; if (usb4_switch_version(tb->root_switch) == 1) Reference standalone patch, tests, and packaging: https://github.com/StickwoodJr/usb4-nvme-direct-boot ================================================================================ 6. SUGGESTIONS FOR THE UBUNTU KERNEL TEAM ================================================================================ 1. Re-evaluate Launchpad Bug #2078573 under linux (Ubuntu): Consider re-opening the kernel task in light of the -ENODEV probe timing findings and dracut test results, which indicate that kernel-side tunnel preservation is more robust than userspace initramfs hooks. 2. Consider In-Kernel Tunnel Preservation: Evaluate adopting conditional checks for active boot tunnels or documenting thunderbolt.host_reset=0 as the recommended setting for external direct-boot environments. 3. Documentation: Help provide guidance in Ubuntu release notes or documentation for users running external direct-boot NVMe configurations over USB4 / Thunderbolt. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2167764/+subscriptions
Комментариев нет:
Отправить комментарий