LP#2167595 - second real-world failure, now with the v3 patch on 7.0.0-38 == Environment == - Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic - v3 patch built out of tree and loaded from /lib/modules/7.0.0-38-generic/updates/ (mt792x_usb srcversion BF5F60AFAFBFC93CA662D26, mt7921_common srcversion 9336B003C308D2F9FDFC700) - Adapter: MediaTek MT7921AU USB (0e8d:7961) on xHCI port 1-9 - Firmware: WM build 20260224110949 (HW/SW 0x8a108a10), Bluetooth build 20260224111243 (HW/SW 0x008a008a). This is the latest MT7921 firmware available in Ubuntu (linux-firmware-mediatek 20260319.git217ca6e4-0ubuntu1.2, resolute-updates) and in upstream linux-firmware as of 2026-10-15. - Driver: mt76 from Ubuntu 7.0.0-38.38 (upstream 7.0.14) plus the v3 patch, which is the newest mt76 in Ubuntu 26.04 (generic/HWE; the OEM 7.0.0-1015 kernel carries the same mt76 changes), while upstream the USB hang hardening by Sean Wang landed in v7.3-rc1 (e137e5fd245c "mt792x: stop USB register access after bus hang", 161596f033e8 "mt7921u: escalate broken USB transport to device reset", 7910bd565d8b "mt792x: quiesce USB paths on disconnect") and is not in any Ubuntu 26.04 kernel yet. - Use: managed station to a phone hotspot (2.4 GHz, channel 11) plus a concurrent AP interface (hostapd, same channel) on the same radio. The AP had no associated clients at the time. == 1. Timeline (local time, 2026-10-14) == 07:46 system suspended 14:22 resumed; station reconnected, AP interface recreated 14:30 Bluetooth HID gamepad connected to the same combo chip (about 60 ACL packets/s from then on) 15:23:16 last periodic sample before the failure, nothing unusual: about 2 MPDU/s, about 10% failed MPDUs, RTT to the gateway 9 ms, no packet loss 15:23:34 kernel: mt7921u 1-9:1.3: vendor request req:63 off:d02c failed:-110 15:23:37 kernel: mt7921u 1-9:1.3: vendor request req:63 off:d054 failed:-110 15:23:40 kernel: mt7921u 1-9:1.3: vendor request req:63 off:d058 failed:-110 15:23:44 kernel: mt7921u 1-9:1.3: vendor request req:63 off:53b8 failed:-110 15:23:46 kernel: Bluetooth: hci0: Opcode 0x0401 failed: -110 15:23:46 kernel: Bluetooth: hci0: command 0x0401 tx timeout 15:23:47 kernel: mt7921u 1-9:1.3: vendor request req:63 off:53c4 failed:-110 15:23:50 kernel: mt7921u 1-9:1.3: vendor request req:66 off:53c4 failed:-110 15:23:54 kernel: mt7921u 1-9:1.3: vendor request req:63 off:d02c failed:-110 15:23:57 kernel: mt7921u 1-9:1.3: vendor request req:63 off:d054 failed:-110 15:23:57 kernel: Bluetooth: hci0: Failed to write uhw reg(-110) 15:24:04 kernel: wlx...: deauthenticating from [BSSID] by local choice (Reason: 3=DEAUTH_LEAVING) 15:24:17 kernel: wlx...: failed to remove key (2, ff:ff:ff:ff:ff:ff) from hardware (-110) 15:24:18 kernel: wlx...: failed to remove key (5, ff:ff:ff:ff:ff:ff) from hardware (-110) 15:24:19 kernel: mt7921u 1-9:1.3: timed out waiting for pending tx 15:24:20 kernel: usb 1-9: reset high-speed USB device number 3 using xhci_hcd 15:24:26 kernel: usb 1-9: device descriptor read/64, error -110 15:25:08 kernel: xhci_hcd 0000:00:14.0: Timeout while waiting for setup device command 15:25:14 kernel: usb 1-9: device not accepting address 3, error -62 15:25:26 kernel: usb 1-9: USB disconnect, device number 3 15:26:09 kernel: usb usb1-port9: attempt power cycle 15:26:20 kernel: usb 1-9: device not accepting address 7, error -62 15:31:30 kernel: usb usb1-port9: attempt power cycle 15:31:41 kernel: usb 1-9: device not accepting address 11, error -62 15:31:46 manual reboot. The shutdown tail was 5 s, with no hung tasks and no "blocked for more than" messages. After the reboot the adapter enumerated normally and has worked since. == 2. Comparison with earlier events == 2026-09-26 (v2 patch, 7.0.0-34): the same failure, described in comment #7. The register sequence is identical (d02c -> d054 -> d058 -> 53b8 -> BT 0x0401 -> 53c4 -> 53c4 -> d02c -> d054 -> BT uhw reg), with the same spacing of about 3 s, followed by "timed out waiting for pending tx" and a USB reset loop the device never recovered from. At that time no AP interface existed; the adapter was a plain station. 2026-10-02 05:48: this was NOT a field failure. It was produced by my own test script (sysfs unbind under TX load), which yielded "failed to remove key -110" and a "rx urb mismatch" WARNING. Please do not count it as a spontaneous event. == 3. What this shows == 1. The deadlock fix holds in the field. This is the second real failure with the patch (v2 on 26.09, v3 now). Neither had a hung task, and the reboot was clean (5 s shutdown tail on 14.10). 2. The underlying failure is outside the disconnect path the patch touches. The whole chip stops answering on USB: WiFi and Bluetooth time out at the same moment. A port power cycle does not bring it back, and re-enumeration fails with -110/-62 until the machine is rebooted. This looks like a firmware or USB-level hang rather than a driver logic error. 3. Running the AP and the station concurrently is not required to trigger it: 26.09 happened without any AP interface. Both failures happened under low traffic. New in this occurrence: an active Bluetooth HID device on the same chip, and the failure came about one hour after resume from suspend. I do not know the Bluetooth or suspend state for 26.09. == 4. Questions == - Is a firmware hang of this kind known for the 20260224 WM/BT firmware builds on MT7921AU? - Is there any debug output (mt76 debugfs, firmware log, usbmon, a module parameter) worth enabling permanently, so the next occurrence captures more than the -110 sequence? Failures so far are about 18 days apart, so a low-overhead option would be preferred. == 5. Request == The USB hang hardening by Sean Wang (MediaTek), merged in v7.3-rc1 (d23353688dd0, 161596f033e8, e137e5fd245c, 7910bd565d8b, a4803d1801c7, b994cb409f21, plus the dependency 5323d3e50c20 "mt76u: use a threaded NAPI for the RX path"), targets exactly this failure: e137e5fd245c describes the same pattern of "vendor request ... failed:-110" every ~3 s. None of it is in any Ubuntu 26.04 kernel yet. Could the kernel team consider an SRU of this series to resolute? I am happy to test a PPA or -proposed kernel on the affected hardware and report back. See also LP#2161448, which looks like the same issue on noble HWE. -- You received this bug notification because you are subscribed to linux in Ubuntu. Matching subscriptions: Bgg, Bmail, Nb https://bugs.launchpad.net/bugs/2167595 Title: mt7921u: USB reset after -110 timeouts deadlocks in mt7921_abort_roc, blocks rtnl_lock and shutdown Status in linux package in Ubuntu: Confirmed Bug description: [ Impact ] When an MT7921AU/MT7925 USB wireless adapter (e.g. 0e8d:7961) experiences communication stalls (USB -110 ETIMEDOUT errors on MT_MIB_SDR9 register reads), mt792x_mac_work holds dev->mt76.mutex while looping on register reads. When USB core detects the stall and triggers unbind, mt792xu_disconnect calls mt76_unregister_device, which takes rtnl_lock and calls mt7921_abort_roc (or mt7925_abort_roc). mt7921_abort_roc then tries to acquire dev->mt76.mutex, which is already held by the stuck mac_work thread. This creates an unrecoverable ABBA deadlock between rtnl_lock and dev->mt76.mutex. As a result: - mac_work waits on USB timeouts with dev->mt76.mutex held. - mt792xu_disconnect waits for dev->mt76.mutex under rtnl_lock. - All network operations (NetworkManager, ip, dev_close) hang forever in D state waiting for rtnl_lock. - The system cannot power off or reboot cleanly without a hard reset. [ Fix ] 1. In mt792xu_disconnect(), set MT76_REMOVED, MT76_RESET, and MT76_MCU_RESET immediately, wake MCU waitqueues, purge res_q, and cancel reset_work and init_work FIRST, followed by mac_work, ps_work, and wake_work before invoking mt76_unregister_device(). Setting MT76_REMOVED causes all pending and subsequent USB register requests to fail immediately with -EIO instead of blocking on 3-second timeouts. 2. In mt7921_mac_reset_work() and mt7925_mac_reset_work(), check MT76_REMOVED: if the device was disconnected, abort reset immediately and skip waking queues or interface iterations on a dead adapter. 3. In mt7921_abort_roc() and mt7925_abort_roc(), check if MT76_REMOVED is set. If the device was removed, clear MT76_STATE_ROC and return 0 immediately without waiting on dev->mt76.mutex or sending MCU commands over a dead bus. 4. In mt7921_mcu_parse_response() and mt7925_mcu_parse_response(), suppress timeout error logging and mt792x_reset() when MT76_MCU_RESET or MT76_REMOVED is set, avoiding spurious teardown noise and preventing reset_work re-queueing during unbind. [ Test Plan ] 1. Boot kernel (e.g. 7.0.0-34-generic) with patched mt7921u / mt792x-usb modules. 2. Insert MT7921AU USB Wi-Fi adapter (0e8d:7961) and connect to an AP. 3. Physically disconnect the USB adapter while network traffic is active (~750 pkt/s TX load). 4. Verify in dmesg: - Device disconnect completes cleanly without call traces, WARNINGs, or blocked tasks. - rtnl_lock is not blocked; NetworkManager and ip link operate normally. - Re-plugging the adapter re-probes cleanly and reconnects. - System reboots and powers off cleanly without hanging in D state. Real hardware verification performed on Dell Precision 3650 (7.0.0-34-generic): - 2x rmmod/insmod cycles: MCU teardown noise reduced to 0 lines, 0 warnings. - Unplug under active TX load (~750 pkt/s): clean teardown, only "USB disconnect" and benign "tx urb failed: -71". Clean re-probe and reconnection on replug. [ Where problems could occur ] - If abort_roc returns early without taking mutex on device removal, any cleanup that relied on mutex serialization must be safe. Since MT76_REMOVED is set, hardware registers cannot be accessed anyway, so skipping mutex and clearing local state flag is safe. - Worker cancellations prior to unregistration prevent concurrent execution during teardown, which is the desired behavior on unbind. - Suppressing MCU timeout logs when MT76_REMOVED/MT76_MCU_RESET is set only silences noise during expected teardown; genuine runtime MCU timeouts with no flags set continue to log and trigger recovery reset as before. [ Other Info ] Target: Ubuntu Resolute (26.04) and upstream linux-wireless. Patch: Fix mt7921u/mt792xu USB disconnect deadlock on timeout v3 (Attachment on Launchpad). Verified on hardware by: Piotr Kalinski <callynsky.linux@proton.me> (Tested-by: Piotr Kalinski <callynsky.linux@proton.me>). --- [ Original Report ] Dell Precision 3650 Tower, Ubuntu 26.04 dev kernel 7.0.0-31-generic. Alfa AWUS036AXM (MT7921AU, USB ID 0e8d:7961). Under heavy traffic or weak signal, the adapter disconnects and reconnects rapidly. Eventually kernel dmesg shows: [ 342.112004] mt7921u 1-2:1.0: Message 00000040 (seq 4) timeout [ 345.184002] mt7921u 1-2:1.0: Message 00000040 (seq 5) timeout [ 348.256011] mt7921u 1-2:1.0: Failed to get patch sem [ 351.328008] mt7921u 1-2:1.0: hardware init failed After that, NetworkManager stops responding. Running 'ip link' hangs indefinitely in D state. Rebooting hangs on 'A stop job is running for Network Manager' and requires SysRq+B or power button. SysRq-t output shows mt7921_abort_roc blocked waiting on mutex while held by mac_work: [ 420.100012] task:kworker/u16:3 blocked for more than 120 seconds. [ 420.100020] Call Trace: [ 420.100025] __schedule+0x345/0x890 [ 420.100030] schedule+0x5a/0xc0 [ 420.100035] schedule_preempt_disabled+0x18/0x30 [ 420.100040] __mutex_lock.isra.0+0x28a/0x4b0 [ 420.100045] mt7921_abort_roc+0x2d/0x80 [mt7921_common] [ 420.100050] ieee80211_set_disassoc+0x62/0x90 [cfg80211] [ 420.100055] mt76_unregister_device+0x48/0x90 [mt76] [ 420.100060] mt792xu_disconnect+0x3c/0x70 [mt792x_usb] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2167595/+subscriptions
Комментариев нет:
Отправить комментарий