Public bug reported: BugLink: https://bugs.launchpad.net/bugs/2149767 [Impact] An unprivileged user doing a simple system wide grep can cause a NULL pointer dereference and oops in the SUNRPC subsystem, leading to a local Denial Of Service. A user doing a grep such as $ grep -R "something" / will eventually make its way to /sys/kernel/sunrpc/, where it can hit a race where ->sock in SUNRPC can be set to NULL, like if a network was going down and up again, or a nfs server was being restarted, leading to the following oops. BUG: kernel NULL pointer dereference, address: 0000000000000020 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 0 P4D 0 Oops: 0000 SMP NOPTI CPU: 2 PID: 4933 Comm: grep Not tainted 5.15.0-176-generic #186-Ubuntu RIP: 0010:kernel_getsockname+0x6/0x20 Code: 00 00 00 00 0f 1f 44 00 00 55 48 8b 47 20 48 8b 40 60 48 89 e5 ff d0 0f 1f 00 5d c3 cc cc cc cc 0f 1f 40 00 0f 1f 44 00 00 55 <48> 8b 47 20 31 d2 48 8b 40 38 48 89 e5 ff d0 0f 1f 00 5d c3 cc cc Call Trace: <TASK> ? xs_sock_getport+0x2b/0x70 [sunrpc] ? kvmalloc_node+0x28/0xa0 ? memcg_slab_post_alloc_hook+0x19e/0x210 get_srcport+0x15/0x20 [sunrpc] rpc_sysfs_xprt_info_show+0x110/0x130 [sunrpc] kobj_attr_show+0xf/0x30 sysfs_kf_seq_show+0xa2/0x100 kernfs_seq_show+0x24/0x30 seq_read_iter+0x121/0x4b0 ? _copy_to_user+0x20/0x30 ? cp_new_stat+0x152/0x180 kernfs_fop_read_iter+0x30/0x40 new_sync_read+0x10a/0x190 vfs_read+0x106/0x1a0 ksys_read+0x67/0xf0 __x64_sys_read+0x19/0x20 x64_sys_call+0x1dba/0x1fa0 do_syscall_64+0x56/0xb0 ? do_syscall_64+0x63/0xb0 ? do_syscall_64+0x63/0xb0 ? arch_exit_to_user_mode_prepare.constprop.0+0x1e/0xc0 ? syscall_exit_to_user_mode+0x41/0x80 ? do_syscall_64+0x63/0xb0 entry_SYSCALL_64_after_hwframe+0x6c/0xd6 A workaround is to exclude /sys from your grep or find commands. [Fix] The fix is to ensure that SUNRPC holds the ->recv_mutex during sysfs reads. This makes sure that ->sock cannot be modified by an external change, e.g. a nfs server being restarted. The fix, and their dependencies and fixes are: commit 17f09d3f619a7ad2d2b021b4e5246f08225b1b0f Author: Anna Schumaker <Anna.Schumaker@Netapp.com> Date: Thu Oct 28 15:17:41 2021 -0400 Subject: SUNRPC: Check if the xprt is connected before handling sysfs reads commit b49ea673e119f59c71645e2f65b3ccad857c90ee Author: NeilBrown <neil@brown.name> Date: Mon Jan 17 16:36:53 2022 +1100 Subject: SUNRPC: lock against ->sock changing during sysfs read commit 421ab1be43bd015ffe744f4ea25df4f19d1ce6fe Author: Trond Myklebust <trond.myklebust@hammerspace.com> Date: Fri Mar 25 10:37:31 2022 -0400 Subject: SUNRPC: Do not dereference non-socket transports in sysfs These landed during 5.16-rc1 and 5.18-rc1. [Testcase] Create a fresh jammy VM. Create a NFS share: $ sudo apt install nfs-kernel-server $ sudo mkdir -p /mnt/nfs_share $ sudo chown nobody:nogroup /mnt/nfs_share $ sudo chmod 777 /mnt/nfs_share $ sudo vim /etc/exports /mnt/nfs_share 192.168.1.0/24(rw,sync,no_subtree_check) $ sudo exportfs -a $ sudo systemctl restart nfs-kernel-server Set up a loop where we grep the SUNRPC sysfs interface, causing a read to happen, and some of these reads will happen when the ->recv_mutex is not held. $ while true; do grep -Rr . /sys/kernel/sunrpc/xprt-switches/; done Set up a loop where we mount and unmount the nfs share. This triggers the SUNRPC transport to disconnect or change states. $ sudo -s # while true; do mount -t nfs 192.168.122.126:/mnt/nfs_share /mnt/test umount /mnt/test done Wait a few seconds and the kernel will oops with a null pointer dereference. There is a test kernel available in the following PPA: https://launchpad.net/~mruffell/+archive/ubuntu/sf434838-test If you install the test kernel, you can keep running the loops forever without any kernel oops. [Where problems can occur] We are changing how SUNRPC protects sysfs reads, ensuring we take a mutex to protect the socket transport from changing due to external factors. Taking the mutex might take time, and slow down sysfs read operations, or cause deadlocks in other places in SUNRPC if not done correctly. The patch "SUNRPC: Do not dereference non-socket transports in sysfs" is quite a refactor, but the risk to RDMA users not having the patch is higher than carrying the patch. If a regression were to occur, users could likely work around the issue by not using system wide grep or find commands that parse sysfs entries. [Other info] This is known as CVE-2022-48816. https://ubuntu.com/security/CVE-2022-48816 https://nvd.nist.gov/vuln/detail/cve-2022-48816 ** Affects: linux (Ubuntu) Importance: Undecided Status: Fix Released ** Affects: linux (Ubuntu Jammy) Importance: Undecided Status: In Progress ** Tags: sts ** Description changed: - BugLink: https://bugs.launchpad.net/bugs/ + BugLink: https://bugs.launchpad.net/bugs/2149767 [Impact] An unprivileged user doing a simple system wide grep can cause a NULL pointer - dereference and oops in the SUNRPC subsystem, leading to a local Denial Of + dereference and oops in the SUNRPC subsystem, leading to a local Denial Of Service. A user doing a grep such as $ grep -R "something" / will eventually make its way to /sys/kernel/sunrpc/, where it can hit a race where ->sock in SUNRPC can be set to NULL, like if a network was going down and up again, or a nfs server was being restarted, leading to the following oops. BUG: kernel NULL pointer dereference, address: 0000000000000020 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page - PGD 0 P4D 0 + PGD 0 P4D 0 Oops: 0000 SMP NOPTI CPU: 2 PID: 4933 Comm: grep Not tainted 5.15.0-176-generic #186-Ubuntu RIP: 0010:kernel_getsockname+0x6/0x20 Code: 00 00 00 00 0f 1f 44 00 00 55 48 8b 47 20 48 8b 40 60 48 89 e5 ff d0 0f 1f 00 5d c3 cc cc cc cc 0f 1f 40 00 0f 1f 44 00 00 55 <48> 8b 47 20 31 d2 48 8b 40 38 48 89 e5 ff d0 0f 1f 00 5d c3 cc cc Call Trace: - <TASK> - ? xs_sock_getport+0x2b/0x70 [sunrpc] - ? kvmalloc_node+0x28/0xa0 - ? memcg_slab_post_alloc_hook+0x19e/0x210 - get_srcport+0x15/0x20 [sunrpc] - rpc_sysfs_xprt_info_show+0x110/0x130 [sunrpc] - kobj_attr_show+0xf/0x30 - sysfs_kf_seq_show+0xa2/0x100 - kernfs_seq_show+0x24/0x30 - seq_read_iter+0x121/0x4b0 - ? _copy_to_user+0x20/0x30 - ? cp_new_stat+0x152/0x180 - kernfs_fop_read_iter+0x30/0x40 - new_sync_read+0x10a/0x190 - vfs_read+0x106/0x1a0 - ksys_read+0x67/0xf0 - __x64_sys_read+0x19/0x20 - x64_sys_call+0x1dba/0x1fa0 - do_syscall_64+0x56/0xb0 - ? do_syscall_64+0x63/0xb0 - ? do_syscall_64+0x63/0xb0 - ? arch_exit_to_user_mode_prepare.constprop.0+0x1e/0xc0 - ? syscall_exit_to_user_mode+0x41/0x80 - ? do_syscall_64+0x63/0xb0 - entry_SYSCALL_64_after_hwframe+0x6c/0xd6 + <TASK> + ? xs_sock_getport+0x2b/0x70 [sunrpc] + ? kvmalloc_node+0x28/0xa0 + ? memcg_slab_post_alloc_hook+0x19e/0x210 + get_srcport+0x15/0x20 [sunrpc] + rpc_sysfs_xprt_info_show+0x110/0x130 [sunrpc] + kobj_attr_show+0xf/0x30 + sysfs_kf_seq_show+0xa2/0x100 + kernfs_seq_show+0x24/0x30 + seq_read_iter+0x121/0x4b0 + ? _copy_to_user+0x20/0x30 + ? cp_new_stat+0x152/0x180 + kernfs_fop_read_iter+0x30/0x40 + new_sync_read+0x10a/0x190 + vfs_read+0x106/0x1a0 + ksys_read+0x67/0xf0 + __x64_sys_read+0x19/0x20 + x64_sys_call+0x1dba/0x1fa0 + do_syscall_64+0x56/0xb0 + ? do_syscall_64+0x63/0xb0 + ? do_syscall_64+0x63/0xb0 + ? arch_exit_to_user_mode_prepare.constprop.0+0x1e/0xc0 + ? syscall_exit_to_user_mode+0x41/0x80 + ? do_syscall_64+0x63/0xb0 + entry_SYSCALL_64_after_hwframe+0x6c/0xd6 A workaround is to exclude /sys from your grep or find commands. [Fix] The fix is to ensure that SUNRPC holds the ->recv_mutex during sysfs reads. This makes sure that ->sock cannot be modified by an external change, e.g. a nfs server being restarted. The fix, and their dependencies and fixes are: commit 17f09d3f619a7ad2d2b021b4e5246f08225b1b0f Author: Anna Schumaker <Anna.Schumaker@Netapp.com> Date: Thu Oct 28 15:17:41 2021 -0400 Subject: SUNRPC: Check if the xprt is connected before handling sysfs reads commit b49ea673e119f59c71645e2f65b3ccad857c90ee Author: NeilBrown <neil@brown.name> Date: Mon Jan 17 16:36:53 2022 +1100 Subject: SUNRPC: lock against ->sock changing during sysfs read commit 421ab1be43bd015ffe744f4ea25df4f19d1ce6fe Author: Trond Myklebust <trond.myklebust@hammerspace.com> Date: Fri Mar 25 10:37:31 2022 -0400 Subject: SUNRPC: Do not dereference non-socket transports in sysfs These landed during 5.16-rc1 and 5.18-rc1. [Testcase] Create a fresh jammy VM. Create a NFS share: $ sudo apt install nfs-kernel-server $ sudo mkdir -p /mnt/nfs_share $ sudo chown nobody:nogroup /mnt/nfs_share $ sudo chmod 777 /mnt/nfs_share $ sudo vim /etc/exports /mnt/nfs_share 192.168.1.0/24(rw,sync,no_subtree_check) $ sudo exportfs -a $ sudo systemctl restart nfs-kernel-server Set up a loop where we grep the SUNRPC sysfs interface, causing a read to happen, and some of these reads will happen when the ->recv_mutex is not held. $ while true; do - grep -Rr . /sys/kernel/sunrpc/xprt-switches/; + grep -Rr . /sys/kernel/sunrpc/xprt-switches/; done Set up a loop where we mount and unmount the nfs share. This triggers the SUNRPC transport to disconnect or change states. $ sudo -s # while true; do - mount -t nfs 192.168.122.126:/mnt/nfs_share /mnt/test - umount /mnt/test + mount -t nfs 192.168.122.126:/mnt/nfs_share /mnt/test + umount /mnt/test done Wait a few seconds and the kernel will oops with a null pointer dereference. There is a test kernel available in the following PPA: https://launchpad.net/~mruffell/+archive/ubuntu/sf434838-test If you install the test kernel, you can keep running the loops forever without any kernel oops. [Where problems can occur] We are changing how SUNRPC protects sysfs reads, ensuring we take a mutex to protect the socket transport from changing due to external factors. Taking the mutex might take time, and slow down sysfs read operations, or cause deadlocks in other places in SUNRPC if not done correctly. The patch "SUNRPC: Do not dereference non-socket transports in sysfs" is quite a refactor, but the risk to RDMA users not having the patch is higher than carrying the patch. If a regression were to occur, users could likely work around the issue by not using system wide grep or find commands that parse sysfs entries. [Other info] This is known as CVE-2022-48816. https://ubuntu.com/security/CVE-2022-48816 https://nvd.nist.gov/vuln/detail/cve-2022-48816 ** Also affects: linux (Ubuntu Jammy) Importance: Undecided Status: New ** Changed in: linux (Ubuntu) Status: New => Fix Released ** Changed in: linux (Ubuntu Jammy) Status: New => In Progress ** Tags added: sts -- You received this bug notification because you are subscribed to linux in Ubuntu. Matching subscriptions: Bgg, Bmail, Nb https://bugs.launchpad.net/bugs/2149767 Title: SUNRPC: System wide grep leads to NULL pointer deference in sysfs reads Status in linux package in Ubuntu: Fix Released Status in linux source package in Jammy: In Progress Bug description: BugLink: https://bugs.launchpad.net/bugs/2149767 [Impact] An unprivileged user doing a simple system wide grep can cause a NULL pointer dereference and oops in the SUNRPC subsystem, leading to a local Denial Of Service. A user doing a grep such as $ grep -R "something" / will eventually make its way to /sys/kernel/sunrpc/, where it can hit a race where ->sock in SUNRPC can be set to NULL, like if a network was going down and up again, or a nfs server was being restarted, leading to the following oops. BUG: kernel NULL pointer dereference, address: 0000000000000020 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 0 P4D 0 Oops: 0000 SMP NOPTI CPU: 2 PID: 4933 Comm: grep Not tainted 5.15.0-176-generic #186-Ubuntu RIP: 0010:kernel_getsockname+0x6/0x20 Code: 00 00 00 00 0f 1f 44 00 00 55 48 8b 47 20 48 8b 40 60 48 89 e5 ff d0 0f 1f 00 5d c3 cc cc cc cc 0f 1f 40 00 0f 1f 44 00 00 55 <48> 8b 47 20 31 d2 48 8b 40 38 48 89 e5 ff d0 0f 1f 00 5d c3 cc cc Call Trace: <TASK> ? xs_sock_getport+0x2b/0x70 [sunrpc] ? kvmalloc_node+0x28/0xa0 ? memcg_slab_post_alloc_hook+0x19e/0x210 get_srcport+0x15/0x20 [sunrpc] rpc_sysfs_xprt_info_show+0x110/0x130 [sunrpc] kobj_attr_show+0xf/0x30 sysfs_kf_seq_show+0xa2/0x100 kernfs_seq_show+0x24/0x30 seq_read_iter+0x121/0x4b0 ? _copy_to_user+0x20/0x30 ? cp_new_stat+0x152/0x180 kernfs_fop_read_iter+0x30/0x40 new_sync_read+0x10a/0x190 vfs_read+0x106/0x1a0 ksys_read+0x67/0xf0 __x64_sys_read+0x19/0x20 x64_sys_call+0x1dba/0x1fa0 do_syscall_64+0x56/0xb0 ? do_syscall_64+0x63/0xb0 ? do_syscall_64+0x63/0xb0 ? arch_exit_to_user_mode_prepare.constprop.0+0x1e/0xc0 ? syscall_exit_to_user_mode+0x41/0x80 ? do_syscall_64+0x63/0xb0 entry_SYSCALL_64_after_hwframe+0x6c/0xd6 A workaround is to exclude /sys from your grep or find commands. [Fix] The fix is to ensure that SUNRPC holds the ->recv_mutex during sysfs reads. This makes sure that ->sock cannot be modified by an external change, e.g. a nfs server being restarted. The fix, and their dependencies and fixes are: commit 17f09d3f619a7ad2d2b021b4e5246f08225b1b0f Author: Anna Schumaker <Anna.Schumaker@Netapp.com> Date: Thu Oct 28 15:17:41 2021 -0400 Subject: SUNRPC: Check if the xprt is connected before handling sysfs reads commit b49ea673e119f59c71645e2f65b3ccad857c90ee Author: NeilBrown <neil@brown.name> Date: Mon Jan 17 16:36:53 2022 +1100 Subject: SUNRPC: lock against ->sock changing during sysfs read commit 421ab1be43bd015ffe744f4ea25df4f19d1ce6fe Author: Trond Myklebust <trond.myklebust@hammerspace.com> Date: Fri Mar 25 10:37:31 2022 -0400 Subject: SUNRPC: Do not dereference non-socket transports in sysfs These landed during 5.16-rc1 and 5.18-rc1. [Testcase] Create a fresh jammy VM. Create a NFS share: $ sudo apt install nfs-kernel-server $ sudo mkdir -p /mnt/nfs_share $ sudo chown nobody:nogroup /mnt/nfs_share $ sudo chmod 777 /mnt/nfs_share $ sudo vim /etc/exports /mnt/nfs_share 192.168.1.0/24(rw,sync,no_subtree_check) $ sudo exportfs -a $ sudo systemctl restart nfs-kernel-server Set up a loop where we grep the SUNRPC sysfs interface, causing a read to happen, and some of these reads will happen when the ->recv_mutex is not held. $ while true; do grep -Rr . /sys/kernel/sunrpc/xprt-switches/; done Set up a loop where we mount and unmount the nfs share. This triggers the SUNRPC transport to disconnect or change states. $ sudo -s # while true; do mount -t nfs 192.168.122.126:/mnt/nfs_share /mnt/test umount /mnt/test done Wait a few seconds and the kernel will oops with a null pointer dereference. There is a test kernel available in the following PPA: https://launchpad.net/~mruffell/+archive/ubuntu/sf434838-test If you install the test kernel, you can keep running the loops forever without any kernel oops. [Where problems can occur] We are changing how SUNRPC protects sysfs reads, ensuring we take a mutex to protect the socket transport from changing due to external factors. Taking the mutex might take time, and slow down sysfs read operations, or cause deadlocks in other places in SUNRPC if not done correctly. The patch "SUNRPC: Do not dereference non-socket transports in sysfs" is quite a refactor, but the risk to RDMA users not having the patch is higher than carrying the patch. If a regression were to occur, users could likely work around the issue by not using system wide grep or find commands that parse sysfs entries. [Other info] This is known as CVE-2022-48816. https://ubuntu.com/security/CVE-2022-48816 https://nvd.nist.gov/vuln/detail/cve-2022-48816 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2149767/+subscriptions
Комментариев нет:
Отправить комментарий