[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

I push a large amount of Files (36TB) to a new Backup Server.
This happened to me twice in the last 18 hours.


Bug Description:
nfs-server hangs after some hours
all nfs-clients are affected.
nfs-server can`t be stopped
only power cycle helps

--> DMESG nfs-server:
[33547.322122] INFO: task nfsd:1020 blocked for more than 122 seconds.
[33547.328449] Tainted: G C E 6.8.0-1013-raspi #14-Ubuntu
[33547.335379] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[33547.343255] task:nfsd state:D stack:0 pid:1020 tgid:1020 ppid:2 flags:0x00000008
[33547.343264] Call trace:
[33547.343266] __switch_to+0xb8/0xd8
[33547.343275] __schedule+0x2f0/0x8a0
[33547.343280] schedule+0x3c/0x138
[33547.343284] schedule_timeout+0x1b0/0x1d0
[33547.343290] wait_for_completion+0xcc/0x178
[33547.343294] __flush_workqueue+0x110/0x410
[33547.343300] nfsd4_probe_callback_sync+0x24/0x38 [nfsd]
[33547.343371] nfsd4_destroy_session+0x168/0x228 [nfsd]
[33547.343436] nfsd4_proc_compound+0x4c0/0x770 [nfsd]
[33547.343502] nfsd_dispatch+0xc8/0x278 [nfsd]
[33547.343573] svc_process_common+0x44c/0x720 [sunrpc]
[33547.343670] svc_process+0xec/0x168 [sunrpc]
[33547.343760] svc_handle_xprt+0x3e0/0x5f0 [sunrpc]
[33547.343850] svc_recv+0x17c/0x338 [sunrpc]
[33547.343939] nfsd+0xc0/0x1d0 [nfsd]
[33547.344010] kthread+0xf4/0x108
[33547.344015] ret_from_fork+0x10/0x20


--> DMESG nfs-client:
[52581.872230] RPC: Could not send backchannel reply error: -110

--> systemctl status nfs-kernel-server after attemting to stop service:
× nfs-server.service - NFS server and services
Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; preset: enabled)
Drop-In: /run/systemd/generator/nfs-server.service.d
└─order-with-mounts.conf
Active: failed (Result: timeout) since Sat 2024-10-19 11:17:06 CEST; 11min ago
Duration: 11h 11min 55.954s
Main PID: 1000 (code=exited, status=0/SUCCESS)
Tasks: 2 (limit: 9375)
Memory: 336.0K (peak: 492.0K)
CPU: 4ms
CGroup: /system.slice/nfs-server.service
├─8086 /usr/sbin/rpc.nfsd 0
└─8107 /usr/sbin/exportfs -au

systemd[1]: nfs-server.service: Processes still around after SIGKILL. Ignoring.
systemd[1]: nfs-server.service: State 'stop-post' timed out. Terminating.
systemd[1]: nfs-server.service: State 'final-sigterm' timed out. Killing.
systemd[1]: nfs-server.service: Killing process 8107 (exportfs) with signal SIGKILL.
systemd[1]: nfs-server.service: Killing process 8086 (rpc.nfsd) with signal SIGKILL.
systemd[1]: nfs-server.service: Processes still around after final SIGKILL. Entering failed mode.
systemd[1]: nfs-server.service: Failed with result 'timeout'.
systemd[1]: nfs-server.service: Unit process 8086 (rpc.nfsd) remains running after unit stopped.
systemd[1]: nfs-server.service: Unit process 8107 (exportfs) remains running after unit stopped.
systemd[1]: Stopped nfs-server.service - NFS server and services.


Specs:
Raspberry Pi 5 8GB
Ubuntu 24.04.1 LTS

--
You received this bug notification because you are subscribed to linux
in Ubuntu.
Matching subscriptions: Bgg, Bmail, Nb
https://bugs.launchpad.net/bugs/2062568

Title:
nfsd gets unresponsive after some hours of operation

Status in linux package in Ubuntu:
In Progress
Status in nfs-utils package in Ubuntu:
Incomplete
Status in linux source package in Noble:
In Progress
Status in nfs-utils source package in Noble:
Incomplete

Bug description:
I installed the 24.04 Beta on two test machines that were running
22.04 without issues before. One of them exports two volumes that are
mounted by the other machine, which primarily uses them as a secondary
storage for ccache.

After being up for a couple of hours (happened twice since yesterday
evening) it seems that nfsd on the machine exporting the volumes hangs
on something.

From dmesg on the server (repeated a few times):

[11183.290548] INFO: task nfsd:1419 blocked for more than 1228 seconds.
[11183.290558] Not tainted 6.8.0-22-generic #22-Ubuntu
[11183.290563] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[11183.290582] task:nfsd state:D stack:0 pid:1419 tgid:1419 ppid:2 flags:0x00004000
[11183.290587] Call Trace:
[11183.290602] <TASK>
[11183.290606] __schedule+0x27c/0x6b0
[11183.290612] schedule+0x33/0x110
[11183.290615] schedule_timeout+0x157/0x170
[11183.290619] wait_for_completion+0x88/0x150
[11183.290623] __flush_workqueue+0x140/0x3e0
[11183.290629] nfsd4_probe_callback_sync+0x1a/0x30 [nfsd]
[11183.290689] nfsd4_destroy_session+0x186/0x260 [nfsd]
[11183.290744] nfsd4_proc_compound+0x3af/0x770 [nfsd]
[11183.290798] nfsd_dispatch+0xd4/0x220 [nfsd]
[11183.290851] svc_process_common+0x44d/0x710 [sunrpc]
[11183.290924] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[11183.290976] svc_process+0x132/0x1b0 [sunrpc]
[11183.291041] svc_handle_xprt+0x4d3/0x5d0 [sunrpc]
[11183.291105] svc_recv+0x18b/0x2e0 [sunrpc]
[11183.291168] ? __pfx_nfsd+0x10/0x10 [nfsd]
[11183.291220] nfsd+0x8b/0xe0 [nfsd]
[11183.291270] kthread+0xef/0x120
[11183.291274] ? __pfx_kthread+0x10/0x10
[11183.291276] ret_from_fork+0x44/0x70
[11183.291279] ? __pfx_kthread+0x10/0x10
[11183.291281] ret_from_fork_asm+0x1b/0x30
[11183.291286] </TASK>

From dmesg on the client (repeated a number of times):
[ 6596.911785] RPC: Could not send backchannel reply error: -110
[ 6596.972490] RPC: Could not send backchannel reply error: -110
[ 6837.281307] RPC: Could not send backchannel reply error: -110

ProblemType: Bug
DistroRelease: Ubuntu 24.04
Package: nfs-kernel-server 1:2.6.4-3ubuntu5
ProcVersionSignature: Ubuntu 6.8.0-22.22-generic 6.8.1
Uname: Linux 6.8.0-22-generic x86_64
.etc.request-key.d.id_resolver.conf: create id_resolver * * /usr/sbin/nfsidmap -t 600 %k %d
ApportVersion: 2.28.1-0ubuntu1
Architecture: amd64
CasperMD5CheckResult: pass
Date: Fri Apr 19 14:10:25 2024
InstallationDate: Installed on 2024-04-16 (3 days ago)
InstallationMedia: Ubuntu-Server 24.04 LTS "Noble Numbat" - Beta amd64 (20240410.1)
NFSMounts:

NFSv4Mounts:

ProcEnviron:
 LANG=en_US.UTF-8
 PATH=(custom, no user)
 SHELL=/bin/bash
 TERM=xterm-256color
 XDG_RUNTIME_DIR=<set>
SourcePackage: nfs-utils
UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2062568/+subscriptions