[Bug 2084979] Re: Device Mapper accepts writes even when the underlying devices are dead (offline or transport-offline)

Hi Miroslaw,

I finally got around to trying this out. I reproduced on:

Focal with 5.4.0-200-generic
Jammy with 5.15.0-122-generic
Noble with 6.8.0-49-generic
Plucky with 6.12.0-3-generic

They all behave the same, and they all behave as you report in the bug
report.

I will note that every dmesg will report that the underlying disk is
offline:

[ 108.055848] I/O error, dev sda, sector 2048 op 0x1:(WRITE) flags 0x104000 phys_seg 32 prio class 0
[ 108.056279] Buffer I/O error on dev dm-0, logical block 0, lost async page write
[ 108.056612] Buffer I/O error on dev dm-0, logical block 1, lost async page write
[ 108.056996] Buffer I/O error on dev dm-0, logical block 2, lost async page write
[ 108.057335] Buffer I/O error on dev dm-0, logical block 3, lost async page write
[ 108.057694] Buffer I/O error on dev dm-0, logical block 4, lost async page write
[ 108.058013] Buffer I/O error on dev dm-0, logical block 5, lost async page write
[ 108.058342] Buffer I/O error on dev dm-0, logical block 6, lost async page write
[ 108.058703] Buffer I/O error on dev dm-0, logical block 7, lost async page write
[ 108.059023] Buffer I/O error on dev dm-0, logical block 8, lost async page write
[ 108.059359] Buffer I/O error on dev dm-0, logical block 9, lost async page write
[ 108.060034] I/O error, dev sda, sector 2304 op 0x1:(WRITE) flags 0x104000 phys_seg 32 prio class 0
[ 108.060549] I/O error, dev sda, sector 2560 op 0x1:(WRITE) flags 0x104000 phys_seg 32 prio class 0
[ 108.061012] I/O error, dev sda, sector 2816 op 0x1:(WRITE) flags 0x104000 phys_seg 32 prio class 0
[ 108.061465] I/O error, dev sda, sector 3072 op 0x1:(WRITE) flags 0x104000 phys_seg 32 prio class 0
[ 108.061908] I/O error, dev sda, sector 3328 op 0x1:(WRITE) flags 0x104000 phys_seg 32 prio class 0
[ 108.062318] I/O error, dev sda, sector 3584 op 0x1:(WRITE) flags 0x104000 phys_seg 32 prio class 0
[ 108.062744] I/O error, dev sda, sector 3840 op 0x1:(WRITE) flags 0x104000 phys_seg 32 prio class 0
[ 108.063188] I/O error, dev sda, sector 4096 op 0x1:(WRITE) flags 0x104000 phys_seg 32 prio class 0
[ 108.063616] I/O error, dev sda, sector 4352 op 0x1:(WRITE) flags 0x104000 phys_seg 32 prio class 0

Its just that dd succeeds, and exits 0. A bit frustrating I suppose.

Since this affects the very latest mainline, we should probably ask upstream
about it.

Will you write to the upstream device mapper maintainers?

Thanks,
Matthew

--
You received this bug notification because you are subscribed to linux
in Ubuntu.
Matching subscriptions: Bgg, Bmail, Nb
https://bugs.launchpad.net/bugs/2084979

Title:
Device Mapper accepts writes even when the underlying devices are dead
(offline or transport-offline)

Status in linux package in Ubuntu:
New

Bug description:
**********
#### Steps to reproduce
**********

root@ubusrv:~# lsblk /dev/sda
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 2G 0 disk
└─sda1 8:1 0 2G 0 part
root@ubusrv:~#

root@ubusrv:~# pvcreate /dev/sda1
Physical volume "/dev/sda1" successfully created.
root@ubusrv:~# vgcreate vgtmp /dev/sda1
Volume group "vgtmp" successfully created
root@ubusrv:~# lvcreate -n lvtmp -l 100%free vgtmp
Logical volume "lvtmp" created.
root@ubusrv:~# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
ubuntu-lv ubuntu-vg -wi-ao---- 18.22g
lvtmp vgtmp -wi-a----- <2.00g
root@ubusrv:~#

root@ubusrv:~# cat /sys/block/sda/device/state
running
root@ubusrv:~# echo offline > /sys/block/sda/device/state
root@ubusrv:~# cat /sys/block/sda/device/state
offline
root@ubusrv:~#

root@ubusrv:~# l file.tar
-rw-r--r-- 1 root root 2.4G Oct 19 14:12 file.tar
root@ubusrv:~#


----------------------
1. Reading from faulty DM device: OK - expected behavior
root@ubusrv:~# dd if=/dev/mapper/vgtmp-lvtmp
dd: error reading '/dev/mapper/vgtmp-lvtmp': Input/output error
0+0 records in
0+0 records out
0 bytes copied, 0.00298229 s, 0.0 kB/s
root@ubusrv:~#

2. Reading from faulty/offline underlying sda: OK - expected behavior
root@ubusrv:~# dd if=/dev/sda1
dd: failed to open '/dev/sda1': No such device or address
root@ubusrv:~#

3. Writing to faulty/offline underlying sda: OK - expected behavior
root@ubusrv:~# dd of=/dev/sda1
dd: failed to open '/dev/sda1': No such device or address
root@ubusrv:~#

4. And writing to faulty DM device: not OK - UNEXPECTED behavior
root@ubusrv:~# dd if=file.tar of=/dev/mapper/vgtmp-lvtmp bs=10M count=204
204+0 records in
204+0 records out
2139095040 bytes (2.1 GB, 2.0 GiB) copied, 4.57038 s, 468 MB/s <<<<<<<<
root@ubusrv:~#
----------------------


******** Where this data has been stored?
Not in memory - not enough space:
root@ubusrv:~# free
total used free shared buff/cache available
Mem: 961Mi 361Mi 200Mi 1.3Mi 556Mi 599Mi <<<<
Swap: 2.0Gi 0B 2.0Gi
root@ubusrv:~#


And the same applies to iSCSI+MPIO.
----------------------
root@ubusrv:~# multipath -ll
mpatha (3600140582d6e7c50c0347908f0d094c2) dm-2 LIO-ORG,ubusrv-wrbk
size=4.4G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 8:0:0:135 sdc 8:32 active ready running
`- 7:0:0:135 sdb 8:16 active ready running
mpathb (3600140512e2d3ad7ae048aba1ee0a33a) dm-3 LIO-ORG,ubusrv-wrth
size=3.9G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 8:0:0:235 sdd 8:48 active ready running
`- 7:0:0:235 sde 8:64 active ready running
root@ubusrv:~#

#### poweroff iSCSI target/server

root@ubusrv:~# egrep 'mpath[ab]' /var/log/syslog | tail
2024-10-19T15:07:01.810687+02:00 ubusrv multipathd[382]: checker failed path 8:64 in map mpathb
2024-10-19T15:07:01.811658+02:00 ubusrv multipathd[382]: mpathb: remaining active paths: 1
2024-10-19T15:07:02.810464+02:00 ubusrv multipathd[382]: checker failed path 8:16 in map mpatha
2024-10-19T15:07:02.813151+02:00 ubusrv multipathd[382]: mpatha: Entering recovery mode: max_retries=12
2024-10-19T15:07:02.813182+02:00 ubusrv multipathd[382]: mpatha: remaining active paths: 0
2024-10-19T15:07:04.811717+02:00 ubusrv multipathd[382]: checker failed path 8:48 in map mpathb
2024-10-19T15:07:04.813169+02:00 ubusrv multipathd[382]: mpathb: Entering recovery mode: max_retries=12
2024-10-19T15:07:04.813206+02:00 ubusrv multipathd[382]: mpathb: remaining active paths: 0
2024-10-19T15:08:02.844668+02:00 ubusrv multipathd[382]: mpatha: Disable queueing
2024-10-19T15:08:04.845680+02:00 ubusrv multipathd[382]: mpathb: Disable queueing
root@ubusrv:~#

root@ubusrv:~# iscsiadm -m session -P 3 | grep Att
Attached SCSI devices:
Attached scsi disk sdb State: transport-offline
Attached scsi disk sde State: transport-offline
Attached SCSI devices:
Attached scsi disk sdc State: transport-offline
Attached scsi disk sdd State: transport-offline
root@ubusrv:~#

root@ubusrv:~# multipath -ll
mpatha (3600140582d6e7c50c0347908f0d094c2) dm-2 LIO-ORG,ubusrv-wrbk
size=4.4G features='0' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=0 status=active
|- 8:0:0:135 sdc 8:32 failed faulty running
`- 7:0:0:135 sdb 8:16 failed faulty running
mpathb (3600140512e2d3ad7ae048aba1ee0a33a) dm-3 LIO-ORG,ubusrv-wrth
size=3.9G features='0' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=0 status=active
|- 8:0:0:235 sdd 8:48 failed faulty running
`- 7:0:0:235 sde 8:64 failed faulty running
root@ubusrv:~#

root@ubusrv:~# cat /sys/block/sd[b-e]/device/state
transport-offline
transport-offline
transport-offline
transport-offline
root@ubusrv:~#

root@ubusrv:~# l /dev/mapper/mpath*
lrwxrwxrwx 1 root root 7 Oct 19 13:35 /dev/mapper/mpatha -> ../dm-2
lrwxrwxrwx 1 root root 7 Oct 19 13:35 /dev/mapper/mpatha-part1 -> ../dm-4
lrwxrwxrwx 1 root root 7 Oct 19 13:35 /dev/mapper/mpathb -> ../dm-3
lrwxrwxrwx 1 root root 7 Oct 19 13:35 /dev/mapper/mpathb-part1 -> ../dm-5
root@ubusrv:~#

********
root@ubusrv:~# dd if=file.tar of=/dev/mapper/mpatha-part1 bs=10M
240+1 records in
240+1 records out
2526904320 bytes (2.5 GB, 2.4 GiB) copied, 5.65064 s, 447 MB/s <<<<<<<<
root@ubusrv:~#
********
----------------------


Is it a Device Mapper feature?


--
Regards,
Mirek

ProblemType: Bug
DistroRelease: Ubuntu 24.04
Package: linux-image-6.8.0-47-generic 6.8.0-47.47
ProcVersionSignature: Ubuntu 6.8.0-47.47-generic 6.8.12
Uname: Linux 6.8.0-47-generic x86_64
AlsaVersion: Advanced Linux Sound Architecture Driver Version k6.8.0-47-generic.
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.28.1-0ubuntu3.1
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/timer', '/dev/snd/seq', '/dev/snd/pcmC0D0p', '/dev/snd/pcmC0D0c', '/dev/snd/hwC0D0', '/dev/snd/controlC0', '/dev/snd/by-path'] failed with exit code 1:
CRDA: N/A
Card0.Amixer.info: Error: [Errno 2] No such file or directory: 'amixer'
Card0.Amixer.values: Error: [Errno 2] No such file or directory: 'amixer'
CasperMD5CheckResult: pass
Date: Sat Oct 19 15:39:00 2024
InstallationDate: Installed on 2024-09-03 (46 days ago)
InstallationMedia: Ubuntu-Server 24.04 LTS "Noble Numbat" - Release amd64 (20240423)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb:
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd QEMU Tablet
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Lsusb-t:
/: Bus 001.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/15p, 480M
|__ Port 001: Dev 002, If 0, Class=Human Interface Device, Driver=usbhid, 480M
/: Bus 002.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/15p, 5000M
MachineType: QEMU Standard PC (Q35 + ICH9, 2009)
ProcEnviron:
LANG=en_US.UTF-8
PATH=(custom, no user)
SHELL=/bin/bash
TERM=xterm-256color
ProcFB: 0 qxldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-6.8.0-47-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro
RelatedPackageVersions:
linux-restricted-modules-6.8.0-47-generic N/A
linux-backports-modules-6.8.0-47-generic N/A
linux-firmware 20240318.git3b128b60-0ubuntu2.4
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 04/01/2014
dmi.bios.release: 0.0
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.16.3-debian-1.16.3-2
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-q35-8.2
dmi.modalias: dmi:bvnSeaBIOS:bvr1.16.3-debian-1.16.3-2:bd04/01/2014:br0.0:svnQEMU:pnStandardPC(Q35+ICH9,2009):pvrpc-q35-8.2:cvnQEMU:ct1:cvrpc-q35-8.2:sku:
dmi.product.name: Standard PC (Q35 + ICH9, 2009)
dmi.product.version: pc-q35-8.2
dmi.sys.vendor: QEMU

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2084979/+subscriptions