Reproduced in a VM and used to verify the patch proposed by Viktor Pashaiev. Reproducer and both boot logs attached. All runs on 7.0.0-30-generic, the kernel the original crash happened on. ## 1. Mechanism attr_data_get_block_locked() duplicates the resident attribute: *res = kmemdup(resident_data(attr_b), data_size, GFP_KERNEL); The pointer becomes iomap->inline_data, iomap->length becomes data_size. iomap_write_end_inline() then asserts iomap->length <= PAGE_SIZE - offset_in_page(iomap->inline_data) A slab object rarely starts on a page boundary. data_size 129..192 comes from kmalloc-192: order=1 slab, 42 objects, 192-byte stride. Object #21 starts at 4032, and 4032 + (129..192) = 4161..4224 > 4096. The other 41 objects pass. ## 2. The guest needs >= 16 vCPUs calculate_order() in mm/slub.c: nr_cpus = num_present_cpus(); min_objects = 4 * (fls(nr_cpus) + 1); min_order = max(slub_min_order, get_order(min_objects * size)); size=192, 15 CPUs: fls(15)=4, min_objects=20, 20*192=3840, get_order(3840)=0, min_order=0. calc_slab_order() at order 0: rem = 4096 % 192 = 64, threshold slab_size/16 = 256, 64 <= 256 so it stops. order=0: one page, 21 objects, offsets 0..3840, and 3840 + 192 = 4032 <= 4096. Nothing straddles a page boundary; the BUG_ON is unreachable. size=192, 16 CPUs: fls(16)=5, min_objects=24, 24*192=4608, get_order(4608)=1, min_order=1. order=1: 42 objects, object #21 at 4032. Threshold is exactly 16 present CPUs. Verify in the guest (mode 0400, root): # cat /sys/kernel/slab/kmalloc-192/order -> 1 # cat /sys/kernel/slab/kmalloc-192/objs_per_slab -> 42 Below that, a reproducer reports "not reproducible" on a broken kernel too. ## 3. Workload The original crash was a 137-byte write(2) to fd 1, rsync writing its log to an ntfs3 volume. attr_set_size_res() only goes non-resident once used + dsize > sbi->max_bytes_per_attr (~700 bytes at a 1024-byte MFT record), so a small new file stays resident and its first write takes the resident branch. Loop: create file, one write() of 129..192 bytes, unlink. Two constraints, both of which produce zero hits if violated: * Appending (open(p,'ab')) writes past i_size, goes through ntfs_extend() -> ntfs_set_size(), and once non-resident kmemdup() is never called again. * Unrelated kmalloc-192 traffic is required. The buffer lives only from iomap_begin to iomap_end, so SLUB returns the same object LIFO and object #21 never surfaces. Setup: qemu/KVM, 16 vCPUs, 4 GiB RAM, 4 GiB NTFS image, files seeded through ntfs-3g, then remounted with ntfs3 rw. ## 4. Result on an unmodified kernel Shipped ntfs3 module, BUG_ON 68.4 s after boot: kernel BUG at fs/iomap/buffered-io.c:1061! Oops: invalid opcode: 0000 [#1] SMP NOPTI CPU: 10 UID: 0 PID: 1788 Comm: python3 Not tainted 7.0.0-30-generic #30-Ubuntu RIP: 0010:iomap_write_end+0x1e0/0x1f0 RAX: 0000000000000040 RBX: fffffb5d41097280 RCX: fffffb5d41097280 RDX: 0000000000000fc0 RSI: 00000000000000b8 RDI: ffff8dc4cd852fc0 Call Trace: iomap_write_iter+0x171/0x340 iomap_file_buffered_write+0xa6/0x110 ntfs_file_write_iter+0x267/0x310 [ntfs3] vfs_write+0x25b/0x490 ksys_write+0x71/0xf0 __x64_sys_write+0x19/0x30 Against the crash originally reported here: original (rsync) reproducer RDX = offset_in_page 0xfc0 = 4032 0xfc0 = 4032 RAX = PAGE_SIZE - off 0x40 = 64 0x40 = 64 write length 0x89 = 137 0xb8 = 184 inline_data low bits ...c55acfc0 ...cd852fc0 Same signature in bugzilla.kernel.org #221446 (Arch, 7.0.2, process `git`): RDX 0xfc0, RAX 0x40, length 145. Three independent hits on offset 4032, all lengths inside the kmalloc-192 range. ## 5. Patch verification Built out of tree from linux-source-7.0.0 against 7.0.0-30-generic headers as two modules differing only by Viktor's patch, which applied with no offsets and no fuzz. Confirmed at machine-code level: in attr_data_get_block_locked() the unpatched build calls kmemdup_noprof, the patched build calls alloc_pages_noprof plus memcpy. An unpatched out-of-tree build was run first as a control. Without it, "no panic with the patch" would not distinguish a working fix from an out-of-tree build that differs from the shipped module. Each run verified the live module by comparing /sys/module/ntfs3/srcversion against the expected value. Same kernel, same image, same workload. Only the module differs. control patched live srcversion 99295717... (ok) A80EE935... (ok) kmalloc-192 geometry order=1 objs=42 order=1 objs=42 run length 60 s 840 s inline mappings observed 56,013,951 864,253,125 offset_in_page values seen 76 distinct only 0 mappings violating the check 1 0 BUG_ON yes, at 29.4 s none in 840 s The @DANGER counter and the oops agree on the same mapping. In the control run the probe fired on the return from ntfs_iomap_begin() with [DANGER] off=4032 len=160 sum=4192 -- BUG_ON next and the oops milliseconds later reports RDX=0xfc0 (4032), RSI=0xa0 (160), RAX=0x40 (4096 - 4032). In boot-vanilla.log that printf is interleaved character by character with the oops text, since bpftrace and the kernel share the console. So the counter that reads 0 across 864,253,125 mappings on the patched build is measuring the exact condition that crashes. The offset row is the load-bearing one. With the patch every inline mapping was page aligned, so length <= PAGE_SIZE - 0 holds for any resident size (at most ~700 bytes against 4096). The failure mode is removed structurally, not made rarer. The BUG_ON itself is probabilistic, which matters for SRU verification. Four unpatched runs here fired at 68.4 s, 25.7 s and 29.4 s after boot, and one did not fire at all in 360 s despite 319,225,379 inline mappings and 85 distinct offsets in that run. A violating mapping turns up on the order of once per a few hundred million inline mappings, since it needs the allocation to land in object #21 of a kmalloc-192 slab specifically. Absence of a BUG_ON in a single run is therefore not evidence of a fix on its own; the offset distribution is the reliable signal. Instrumentation caveat: offset 4032 alone is not a fault indicator. It is also the last object of a kmalloc-64 slab (63 * 64 = 4032), where 4032 + 64 = 4096 fits exactly. In longer runs most landings on 4032 were harmless for exactly that reason. The condition to test is length > PAGE_SIZE - offset_in_page(inline_data), which is what the @DANGER counter in the attached bpftrace script evaluates. ## 6. Upstream 70d3855594cf6e8791970714b65cac3202d6160e "ntfs3: Allocate iomap inline_data using alloc_page" Mihai Brodschi Fixes: 099ef9ab9203 ("fs/ntfs3: implement iomap-based file operations") fs/ntfs3/attrib.c and fs/ntfs3/inode.c. Mainline v7.2, stable v7.1.5. Checked both stable tags: fs/ntfs3/attrib.c in v7.1.4 still has kmemdup(resident_data(...)), in v7.1.5 it has alloc_page(). 099ef9ab9203, the iomap conversion, is what shipped in 7.0. 7.0.0-31.31 is at "Upstream stable to v6.18.39, v7.1.4". ## Attachments * ntfs3-2165844-reproducer.tar.gz - workload scripts, bpftrace script, runner, module build script, README with the recipe and the prerequisites. * boot-vanilla.log - control run, unpatched (next comment). * boot-patched.log - patched run (next comment). ** Attachment added: "Reproducer: workload scripts, bpftrace instrumentation, VM runner, module build script, README" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2165844/+attachment/6000954/+files/ntfs3-2165844-reproducer.tar.gz -- You received this bug notification because you are subscribed to linux in Ubuntu. Matching subscriptions: Bgg, Bmail, Nb https://bugs.launchpad.net/bugs/2165844 Title: linux 7.0.0-30: kernel BUG at fs/iomap/buffered-io.c:1061 in iomap_write_end() on ntfs3 buffered write Status in linux package in Ubuntu: Confirmed Bug description: [Summary] Any buffered write() to an ntfs3-mounted volume can hit a BUG_ON in the generic iomap write path and panic the kernel. An unprivileged process (rsync, uid 1000) took the whole machine down; the system froze instantly, then kdump rebooted into the capture kernel. [Impact] This is reachable from unprivileged userspace with a plain write(2) syscall. Whatever the state of the NTFS volume, the correct outcome is -EIO returned to the caller, not a kernel panic. Reproducing it needs nothing but a writable NTFS mount, which udisks/GNOME Files creates by default when a user clicks a Windows partition. [Environment] Ubuntu 26.04, linux-image-7.0.0-30-generic 7.0.0-30.30, x86_64 ASUS ROG STRIX B550-XE GAMING WIFI, BIOS 3607 NTFS volume: 1.8 TB Windows system partition, mounted read-write by udisks (driver ntfs3, POSIX ACLs enabled) [What happened] Writing to the NTFS mount, the kernel hit the assertion and died: kernel BUG at fs/iomap/buffered-io.c:1061! Oops: invalid opcode: 0000 [#1] SMP NOPTI CPU: 2 UID: 1000 PID: 18561 Comm: rsync Kdump: loaded Tainted: G O Hardware name: ASUS System Product Name/ROG STRIX B550-XE GAMING WIFI RIP: 0010:iomap_write_end+0x1e0/0x1f0 Call Trace: iomap_write_iter+0x171/0x340 iomap_file_buffered_write+0xa6/0x110 ntfs_file_write_iter+0x267/0x310 [ntfs3] vfs_write+0x25b/0x490 ksys_write+0x71/0xf0 __x64_sys_write+0x19/0x30 do_syscall_64+0x105/0x5a0 entry_SYSCALL_64_after_hwframe+0x76/0x7e The faulting syscall was a 137-byte write(2) to fd 1 (ORIG_RAX 0x1, RDX 0x89) — an ordinary small buffered write, not anything exotic. The G/O taint is the out-of-tree NVIDIA module (595.84). It is unrelated to this code path; ntfs3, iomap and the write path are all in-tree. [Preceding symptom] About 14 minutes before the panic, ntfs3 logged a large burst of metadata repairs on the same volume: ntfs3(nvme0n1p2): ino=..., Correct links count -> N ntfs3: 15286 callbacks suppressed So the volume was not perfectly clean. That may well be what steers the write path into the bad state — but an inconsistent filesystem must not be able to panic the kernel, so it looks like a missing error path rather than a disk problem. [Reproducer] Not deterministic here, but the shape is simple: 1. Mount an NTFS volume read-write with ntfs3 (udisksctl mount is enough). 2. Run sustained buffered I/O against it (rsync of a large tree). The other reports linked below describe the same trace from ordinary file writes, including games writing save data. [Already reported elsewhere] Same trace, same line, on 7.0.x kernels since roughly April 2026: - https://github.com/CachyOS/linux-cachyos/issues/841 - https://discuss.cachyos.org/t/kernel-bug-in-iomap-write-end-triggered-by-ntfs3-buffered-write-linux-7-0-1-cachyos/28546 - https://bbs.archlinux.org/viewtopic.php?pid=2296728 - https://forum.manjaro.org/t/total-system-hangs-and-fs-iomap-kernel-panic-on-7-0-10-stable-on-6-18-lts/188283 I could not find an upstream fix. Upstream appears to be replacing ntfs3 with the new ntfsplus driver in 7.1 rather than fixing this one. [Suggestion for Ubuntu] Since 26.04 ships 7.0 and a user can trigger a kernel panic by copying files onto a Windows partition from the file manager, it may be worth either backporting a fix or having udisks default NTFS mounts to read-only until 7.1 lands via HWE. [Attached] Full kernel ring buffer from the kdump vmcore (VmCoreDmesg), plus the usual apport-collected hardware and package data. Sanitisation applied before upload: firewall log lines were dropped, and MAC addresses, the wireless BSSID/SSID and LAN IPv4 addresses were replaced with placeholders in VmCoreDmesg, CurrentDmesg and WifiSyslog. Nothing else was altered; the panic trace is untouched. An incomplete vmcore (2.3 GB, truncated because the machine was power-cycled during the dump) is available if it would help. ProblemType: Bug DistroRelease: Ubuntu 26.04 Package: linux-image-7.0.0-30-generic 7.0.0-30.30 ProcVersionSignature: Ubuntu 7.0.0-30.30-generic 7.0.12 Uname: Linux 7.0.0-30-generic x86_64 ApportVersion: 2.34.1-0ubuntu0.1 Architecture: amd64 CasperMD5CheckResult: pass Date: Sun Aug 30 11:45:03 2026 InstallationDate: Installed on 2026-07-12 (49 days ago) InstallationMedia: Ubuntu 26.04 "Resolute Raccoon" - Release amd64 (20260423.1) MachineType: ASUS System Product Name PackageArchitecture: amd64 ProcFB: 0 nvidia-drmdrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-7.0.0-30-generic root=UUID=5778b2dd-b496-4525-b52f-223e5be557bc ro quiet splash crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M SourcePackage: linux Title: linux 7.0.0-30: kernel BUG at fs/iomap/buffered-io.c:1061 in iomap_write_end() on ntfs3 buffered write UpgradeStatus: No upgrade log present (probably fresh install) _MarkForUpload: True dmi.bios.date: 03/18/2024 dmi.bios.release: 5.17 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 3607 dmi.board.asset.tag: Default string dmi.board.name: ROG STRIX B550-XE GAMING WIFI dmi.board.vendor: ASUSTeK COMPUTER INC. dmi.board.version: Rev X.0x dmi.chassis.asset.tag: Default string dmi.chassis.type: 3 dmi.chassis.vendor: Default string dmi.chassis.version: Default string dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr3607:bd03/18/2024:br5.17:svnASUS:pnSystemProductName:pvrSystemVersion:rvnASUSTeKCOMPUTERINC.:rnROGSTRIXB550-XEGAMINGWIFI:rvrRevX.0x:cvnDefaultstring:ct3:cvrDefaultstring:skuSKU:pfaTobefilledbyO.E.M.: dmi.product.family: To be filled by O.E.M. dmi.product.name: System Product Name dmi.product.sku: SKU dmi.product.version: System Version dmi.sys.vendor: ASUS To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2165844/+subscriptions
Комментариев нет:
Отправить комментарий