пятница

[Bug 2174483] Re: [resolute] .NET processes crash with general protection fault in libc since linux 7.0.0-42 (7.0.0-38 OK)

Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: linux (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are subscribed to linux in Ubuntu. Matching subscriptions: Bgg, Bmail, Nb https://bugs.launchpad.net/bugs/2174483 Title: [resolute] .NET processes crash with general protection fault in libc since linux 7.0.0-42 (7.0.0-38 OK) Status in linux package in Ubuntu: Confirmed Bug description: == Summary == After upgrading Ubuntu 26.04 (resolute) amd64 hosts from linux 7.0.0-38 to 7.0.0-42 / 7.0.0-43, .NET processes running in containers crash randomly with SIGSEGV (exit code 139). The kernel logs a general protection fault on .NET runtime threads ("TP Worker" thread pool, "BGC" background GC) inside the container's libc. Rebooting the same VM into an older kernel makes the crashes stop completely. The kernel is the only thing that changes. == Affected / not affected kernels (same VMs, same container images) == linux-image-7.0.0-31-generic 7.0.0-31.31 OK (0 faults in ~2 months of CI use; 0 faults after rollback) linux-image-7.0.0-38-generic 7.0.0-38.38 OK (host running 9 .NET services, 0 faults) linux-image-7.0.0-42-generic 7.0.0-42.42 BROKEN linux-image-7.0.0-43-generic 7.0.0-43.43 BROKEN (resolute-updates) linux-image-7.0.0-44-generic 7.0.0-44.44 BROKEN (resolute-proposed, signed, tested 2026-11-14) The regression was introduced somewhere between 7.0.0-38.38 and 7.0.0-42.42. == System info == lsb_release -rd: Ubuntu 26.04 LTS / Release 26.04 /proc/version_signature (current, rolled back): Ubuntu 7.0.0-31.31-generic 7.0.14 lspci: no PCI devices listed (Hyper-V Generation 2 VM, synthetic VMBus devices only) == Environment == - Ubuntu 26.04 LTS amd64, Hyper-V Generation 2 VMs (UEFI, Secure Boot enabled), 8 vCPU, 16 GB RAM - CPU: Intel(R) Xeon(R) Platinum 8276 CPU @ 2.20GHz (flags include avx2 avx512f avx512vl erms xsaves) - Container runtimes: containerd 2.1.5 (RKE2 Kubernetes v1.34.4) and Docker 29.6.0. Both are affected. - .NET: SDK 9.0.320 / runtime 9.0.22, ASP.NET Core 9 runtime images, and .NET 10 SDK. All are affected. - Container libc: affected with both Ubuntu 24.04 glibc 2.39-0ubuntu8.9 (mcr.microsoft.com/dotnet/sdk:9.0-noble, 10.0-noble) and Debian 12 glibc 2.36-9+deb12u14 (ASP.NET 9 runtime image). This is not specific to one libc build. == Kernel log (examples) == Ubuntu 24.04 glibc 2.39 in the container (host kernel 7.0.0-43, and the same with 7.0.0-44): traps: .NET BGC[39263] general protection fault ip:754c651919a2 sp:75441e735a20 error:0 in libc.so.6[289a2,754c65191000+189000] traps: .NET TP Worker[213956] general protection fault ip:76786d3879a2 sp:7676caffbe60 error:0 in libc.so.6[289a2,76786d387000+189000] traps: .NET TP Worker[12529] general protection fault ip:7a6d09bee9a2 sp:7a68aac9be50 error:0 (7.0.0-44) Debian 12 glibc 2.36 in the container (host kernel 7.0.0-42): traps: .NET TP Worker[940249] general protection fault ip:70ce23ea950f sp:70cc4900ea10 error:0 in libc.so.6[2650f,70ce23ea9000+156000] traps: .NET BGC[946081] general protection fault ip:71f63264a50f sp:71f44cff8a30 error:0 in libc.so.6[2650f,71f63264a000+156000] On each host the faulting offset is always identical (0x289a2 in one libc build, 0x2650f in the other). == Analysis of the faulting IP == In both libc builds the reported IP is NOT on an instruction boundary: - glibc 2.39-0ubuntu8.9 (build-id a4a7992a8e66555c8141ab2a08a8465ff6e0ea65): the text segment is mapped from file offset 0x28000, so the IP is at file address 0x509a2. That is 1 byte into the instruction "mov %r11,-0x448(%rbp)" at 0x509a1. - glibc 2.36-9+deb12u14 (build-id 93ac61ec5a8eb1396f9fbd350e3169a558528a40): the text segment is mapped from 0x26000, so the IP is at file address 0x4c50f. That is the last byte of the padding "nopl" between wcstombs and the next function, 1 byte before "push %rbx" at 0x4c510 and 2 bytes before the next instruction at 0x4c511. A user RIP that lands off an instruction boundary in two unrelated libc builds suggests that the user-mode instruction pointer is corrupted by the kernel, for example on signal delivery or return, or by syscall-restart IP adjustment. It does not look like a libc or .NET bug. The .NET runtime uses signals heavily on these threads (thread suspension for GC, activation injection) and also uses membarrier. == Reproduction == - The most reliable reproducer is a real CI job running "dotnet sonarscanner begin" followed by "dotnet restore" on a mid-size solution, in a container limited to 4 CPUs / 4 GiB. It failed 3/3 on 7.0.0-43 and 1/1 on 7.0.0-44, and passed every time on 7.0.0-31 and 7.0.0-38. - A long-running ASP.NET Core 9 service (HTTP client + EF Core / Npgsql) crashed 3 times in about 1 hour on 7.0.0-42. - A synthetic loop of "dotnet new webapi && dotnet build" reproduces rarely (1 fault in about 80 builds), so the trigger seems to depend on load or timing. Fault frequency on an affected CI host: 17 faults in 8 hours. == Workaround == Pin GRUB_DEFAULT to 7.0.0-38 (or 7.0.0-31) and hold linux-generic / linux-image-generic / linux-headers-generic. == Notes == - I can test candidate kernels or bisect between 7.0.0-38.38 and 7.0.0-42.42 if packages are provided. The intermediate ABIs 7.0.0-39/-40/-41 were not published to resolute-updates. - Production nodes were rolled back, so apport data from an affected boot is not attached. I can boot a test node into 7.0.0-43 to collect it on request. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2174483/+subscriptions

Комментариев нет:

Отправить комментарий