5.4.20: cannot mount device that blipped off the bus: duplicate device fsid:devid for
/dev/sde blipped off the bus (hardware issue?) and came back as /dev/sdq.
Except btrfs won't let me scan or mount it.
I was able to btrfs check it though and that came back clean.
gargamel:~# ls -l /dev/sde ls: cannot access '/dev/sde': No such file or directory
gargamel:~# mount /dev/sdq1 /mnt/mnt mount: /mnt/mnt: mount(2) system call failed: File exists. gargamel:~# dmesg |tail -1 [2560371.195249] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdq1
gargamel:~# btrfs device scan Scanning for Btrfs filesystems ERROR: device scan failed on '/dev/sdq1': File exists ERROR: there are 1 errors while registering devices gargamel:~# dmesg |tail -1 [2560416.434529] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdq1
gargamel:~# grep sde /proc/mounts cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 gargamel:~#
gargamel:~# lsblk -f |grep 727c7ba3-f6f9-462a-8472-453dd7d46d8a └─sdq1 btrfs btrfs_space 727c7ba3-f6f9-462a-8472-453dd7d46d8a gargamel:~#
So, that FS isn't a duplicate anymore and I see to have no way out except reboot which I'll do now.
Was there another way around it? Obviously this is not desirable behaviour, in the past, I was able to remount the device when it came back.
Thanks,
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Quoted 30 lines · Marc MERLIN
/dev/sdeblipped off the bus (hardware issue?) and came back as/dev/sdq.
Except btrfs won't let me scan or mount it.I was able to btrfs check it though and that came back clean.
gargamel:~# ls -l /dev/sde ls: cannot access '/dev/sde': No such file or directorygargamel:~# mount /dev/sdq1 /mnt/mnt mount: /mnt/mnt: mount(2) system call failed: File exists. gargamel:~# dmesg |tail -1 [2560371.195249] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdq1gargamel:~# btrfs device scan Scanning for Btrfs filesystems ERROR: device scan failed on '/dev/sdq1': File exists ERROR: there are 1 errors while registering devices gargamel:~# dmesg |tail -1 [2560416.434529] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdq1gargamel:~# grep sde /proc/mounts cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 gargamel:~#gargamel:~# lsblk -f |grep 727c7ba3-f6f9-462a-8472-453dd7d46d8a └─sdq1 btrfs btrfs_space 727c7ba3-f6f9-462a-8472-453dd7d46d8a gargamel:~#
Quoted 7 lines
So, that FS isn't a duplicate anymore and I see to have no way out except reboot which I'll do now.
Was there another way around it? Obviously this is not desirable behaviour, in the past, I was able to remount the device when it came back.
Presumably you could have used the device forget functionality that got introduced in 5.1, i.e the BTRFS_IOC_FORGET_DEV ioctl. For more info check out: 228a73abde5c04428678e917b271f8526cfd90ed
Quoted 3 lines
Thanks,
Marc
Thanks for the suggestion Nikolay
Dear Anand, David,
I see that
https://gitlab.freedesktop.org/seanpaul/dpu-staging/commit/228a73abde5c04428678e917b271f8526cfd90ed
may have helped, but is this really something a user should know/do?
Why does a device that disappeared from the bus, need to be manually unregistered?
Are users really supposed to know this?
Why does btrfs device scan not invalidate the cache of devices and keep remembering a device that's gone (not visible in new scan)?
Thanks,
Marc
Quoted 49 lines · Nikolay Borisov
Quoted 30 lines · Marc MERLIN
/dev/sdeblipped off the bus (hardware issue?) and came back as/dev/sdq.
Except btrfs won't let me scan or mount it.I was able to btrfs check it though and that came back clean.
gargamel:~# ls -l /dev/sde ls: cannot access '/dev/sde': No such file or directorygargamel:~# mount /dev/sdq1 /mnt/mnt mount: /mnt/mnt: mount(2) system call failed: File exists. gargamel:~# dmesg |tail -1 [2560371.195249] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdq1gargamel:~# btrfs device scan Scanning for Btrfs filesystems ERROR: device scan failed on '/dev/sdq1': File exists ERROR: there are 1 errors while registering devices gargamel:~# dmesg |tail -1 [2560416.434529] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdq1gargamel:~# grep sde /proc/mounts cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 gargamel:~#gargamel:~# lsblk -f |grep 727c7ba3-f6f9-462a-8472-453dd7d46d8a └─sdq1 btrfs btrfs_space 727c7ba3-f6f9-462a-8472-453dd7d46d8a gargamel:~#Quoted 7 lines
So, that FS isn't a duplicate anymore and I see to have no way out except reboot which I'll do now.
Was there another way around it? Obviously this is not desirable behaviour, in the past, I was able to remount the device when it came back.
Presumably you could have used the device forget functionality that got introduced in 5.1, i.e the
BTRFS_IOC_FORGET_DEVioctl. For more info check out:228a73abde5c04428678e917b271f8526cfd90edQuoted 3 lines
Thanks,
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Hi Marc,
Quoted 8 lines · Marc MERLIN
Thanks for the suggestion Nikolay
Dear Anand, David,
I see that
https://gitlab.freedesktop.org/seanpaul/dpu-staging/commit/228a73abde5c04428678e917b271f8526cfd90ed
may have helped, but is this really something a user should know/do?
Quoted 6 lines
Why does a device that disappeared from the bus, need to be manually unregistered?
Are users really supposed to know this?
Why does btrfs device scan not invalidate the cache of devices and keep remembering a device that's gone (not visible in new scan)?
btrfs device scan --forget is only useful to cleanup the unmounted devices, per the logs below the device was mounted when it disappeared. More below.
Quoted 4 lines
Thanks,
Marc
Quoted 18 lines
Quoted 17 lines · Nikolay Borisov
Quoted 14 lines · Marc MERLIN
/dev/sdeblipped off the bus (hardware issue?) and came back as/dev/sdq.
Except btrfs won't let me scan or mount it.I was able to btrfs check it though and that came back clean.
gargamel:~# ls -l /dev/sde ls: cannot access '/dev/sde': No such file or directorygargamel:~# mount /dev/sdq1 /mnt/mnt mount: /mnt/mnt: mount(2) system call failed: File exists. gargamel:~# dmesg |tail -1 [2560371.195249] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdq1
This indicates the device was mounted when it disappeared. So it re-appears with the new path, but as its fsid+uuid+devid matches with the old still mounted device we rightly consider it as an alien device and fail the mount.
To avoid assigning new path to the reappearing device we need to close/pause the device path when it disappears. I need to figure out if there is any KPI from the block layer to help doing that.
Anyone- any idea if there is anything in the block layer which can do the callback into the filesystem if the device disappears?
Thanks, Anand
Quoted 33 lines
Quoted 32 lines
Quoted 16 lines
gargamel:~# btrfs device scan Scanning for Btrfs filesystems ERROR: device scan failed on '/dev/sdq1': File exists ERROR: there are 1 errors while registering devices gargamel:~# dmesg |tail -1 [2560416.434529] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdq1gargamel:~# grep sde /proc/mounts cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 gargamel:~#gargamel:~# lsblk -f |grep 727c7ba3-f6f9-462a-8472-453dd7d46d8a └─sdq1 btrfs btrfs_space 727c7ba3-f6f9-462a-8472-453dd7d46d8a gargamel:~#Quoted 7 lines
So, that FS isn't a duplicate anymore and I see to have no way out except reboot which I'll do now.
Was there another way around it? Obviously this is not desirable behaviour, in the past, I was able to remount the device when it came back.
Presumably you could have used the device forget functionality that got introduced in 5.1, i.e the
BTRFS_IOC_FORGET_DEVioctl. For more info check out:228a73abde5c04428678e917b271f8526cfd90edQuoted 3 lines
Thanks,
Marc
Quoted 7 lines · Anand Jain
Quoted 3 lines
Are users really supposed to know this?
Why does btrfs device scan not invalidate the cache of devices and keep remembering a device that's gone (not visible in new scan)?btrfs device scan --forget is only useful to cleanup the unmounted devices, per the logs below the device was mounted when it disappeared. More below.
I'm confused: why is --forget even needed? Why would it remember devices that were unmounted and not part of a new scan?
And yes, the device was not unmounted. The sata layer failed, device disappeared while mounted and then re-appeared
I was able to force umount the mountpoints, so maybe --forget would have helped, but I'm confused as to why it even exists.
Quoted 4 lines
This indicates the device was mounted when it disappeared. So it re-appears with the new path, but as its fsid+uuid+devid matches with the old still mounted device we rightly consider it as an alien device and fail the mount.
It was unmounted after disappearing, see the 'grep sde /proc/mounts' showing that it wasn't mounted anymore, so it seems that even that part didn't work as intended?
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Quoted 12 lines · Marc MERLIN
Quoted 7 lines · Anand Jain
Quoted 3 lines
Are users really supposed to know this?
Why does btrfs device scan not invalidate the cache of devices and keep remembering a device that's gone (not visible in new scan)?btrfs device scan --forget is only useful to cleanup the unmounted devices, per the logs below the device was mounted when it disappeared. More below.
I'm confused: why is --forget even needed? Why would it remember devices that were unmounted and not part of a new scan?
Quoted 7 lines
And yes, the device was not unmounted. The sata layer failed, device disappeared while mounted and then re-appeared
I was able to force umount the mountpoints, so maybe --forget would have helped, but I'm confused as to why it even exists.
We would log the below only if the old device sde is still in mounted state. Unfortunately we don't have the unmount event log yet (patches are in the ML) so we don't know if unmount was successful.
[2560416.434529] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdq1
If the device is unmounted, the scan would have replaced the sde to sdi, unless the sde (stale) generation is > generation in sdi (lost commit). In which case the --forget is useful to remove the state device entry (provided device is unmounted).
Quoted 8 lines
Quoted 4 lines
This indicates the device was mounted when it disappeared. So it re-appears with the new path, but as its fsid+uuid+devid matches with the old still mounted device we rightly consider it as an alien device and fail the mount.
It was unmounted after disappearing, see the 'grep sde
/proc/mounts' showing that it wasn't mounted anymore, so it seems that even that part didn't work as intended?
Its strange /proc/mounts doesn't list sde. Could you please send me complete kernel logs. Lets try if there is any clue.
I tried to reproduce.. but in my case the unmount was successful.
$ mkfs.btrfs -fq /dev/sdc && mount /dev/sdc /btrfs $ devmgt show | grep sdc host2 sdc $ devmgt detach /dev/sdc :: detach /dev/sdc successful $ devmgt attach host2 :: sd 2:0:0:0: [sdb] Attached SCSI disk :: BTRFS warning (device sdc): duplicate device fsid:devid for dcbc5603-e1cf-4d8d-9ec2-832bc3ac4e36:1 old:/dev/sdc new:/dev/sdb ------------------ attach host2 successful
mounts shows sdc is in ro.
$ cat /proc/mounts | grep sdc /dev/sdc /btrfs btrfs ro,relatime,noacl,space_cache,subvolid=5,subvol=/ 0 0
$ dmesg -k | tail
[ 1427.268767] BTRFS warning (device sdc): duplicate device fsid:devid for dcbc5603-e1cf-4d8d-9ec2-832bc3ac4e36:1 old:/dev/sdc new:/dev/sdb
$ umount /dev/sdc
Unfortunately there is no log about the unmount :-(.
And the following device scan replaces the sdc with sdb.
$ btrfs dev scan Scanning for Btrfs filesystems
$ cat /proc/fs/btrfs/devlist | grep sdc $ cat /proc/fs/btrfs/devlist | grep sdb device: /dev/sdb $
And mount is successful.
$ mount /dev/sdb /btrfs
Thanks, Anand
Quoted 1 line
Marc
Quoted 11 lines · Anand Jain
We would log the below only if the old device sde is still in mounted state. Unfortunately we don't have the unmount event log yet (patches are in the ML) so we don't know if unmount was successful.
[2560416.434529] BTRFS warning (device sde1): duplicate device fsid:devidfor 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdq1
If the device is unmounted, the scan would have replaced the sde to sdi, unless the sde (stale) generation is > generation in sdi (lost commit). In which case the --forget is useful to remove the state device entry (provided device is unmounted).
Well, the device did disappear, wouldn't that cause the in memory version to be more recent than the disk version?
Quoted 2 lines
Its strange
/proc/mountsdoesn't list sde. Could you please send me complete kernel logs. Lets try if there is any clue.
Sure https://pastebin.com/SWAfYxV8
Quoted 10 lines
I tried to reproduce.. but in my case the unmount was successful.
$ mkfs.btrfs -fq /dev/sdc && mount /dev/sdc /btrfs $ devmgt show | grep sdc host2 sdc $ devmgt detach /dev/sdc :: detach /dev/sdc successful $ devmgt attach host2
That's probably too clean.
Can you
1) write to device in a loop
2) pull power from SATA device (in this case it was an ssd)
3) plug device back in
Quoted 3 lines
$ umount /dev/sdcUnfortunately there is no log about the unmount :-(.
Maybe worth adding to help debug later?
I looked in my bash history, and it shows this:
37280 mount | grep sde 37281 umount /mnt/btrfs_space 37282 umount /var/local/space 37283 umount /var/cache/zoneminder 37284 fuser -vm /var/cache/zoneminder 37285 fuser -vkm /var/cache/zoneminder 37286 umount /var/cache/zoneminder 37287 umount /var/lib/mysql 37288 mount | grep sde1
First time, I got output to the mount command.
Second time I did not.
other commands I typed:
37289 mount /dev/sdq1 /mnt/btrfs_space 37296 btrfs device scan 37301 grep sde /etc/* 2>/dev/null 37303 mount /dev/sdq1 /mnt/btrfs_space 37308 grep -r /mnt/btrfs_space /etc 2>/dev/null 37311 btrfs device scan 37312 l /sys/block/sde/ 37313 btrfs check /dev/sdq1 37314 btrfs device scan 37320 mount /dev/sdq1 /mnt/mnt 37323 btrfs device scan 37324 dmesg |tail -1 37326 lsblk -v 37327 lsblk 37328 grep sde /proc/mounts
Hope this helps.
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Anaud, I had this happen agin with 5.5.11, and it was impossible to do anything to fix it, I had to reboot again.
btrfs device scan --forget
did nothing.
See details:
BTRFS: device label btrfs_space devid 1 transid 35178413 /dev/sde1
BTRFS info (device sde1): use lzo compression, level 0
BTRFS info (device sde1): disk space caching is enabled
BTRFS info (device sde1): has skinny extents
BTRFS info (device sde1): enabling ssd optimizations
sd 6:1:3:0: [sde] tag#642 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=2s sd 6:1:3:0: [sde] tag#640 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=2s sd 6:1:3:0: [sde] tag#702 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=2s sd 6:1:3:0: [sde] tag#702 CDB: Write(16) 8a 00 00 00 00 00 f1 a7 3a 68 00 00 01 f0 00 00blk_update_request: I/O error, dev sde, sector 4054268520 op 0x1:(WRITE) flags 0x100000 phys_seg 62 prio class 0 sd 6:1:3:0: [sde] tag#701 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=2s sd 6:1:3:0: [sde] tag#701 CDB: Write(16) 8a 00 00 00 00 00 f1 a7 38 68 00 00 02 00 00 00blk_update_request: I/O error, dev sde, sector 4054268008 op 0x1:(WRITE) flags 0x104000 phys_seg 64 prio class 0 sd 6:1:3:0: [sde] tag#700 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=2s sd 6:1:3:0: [sde] tag#700 CDB: Write(16) 8a 00 00 00 00 00 f1 a7 36 68 00 00 02 00 00 00blk_update_request: I/O error, dev sde, sector 4054267496 op 0x1:(WRITE) flags 0x104000 phys_seg 64 prio class 0 BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
sd 6:1:3:0: [sde] tag#641 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=10s sd 6:1:3:0: [sde] tag#641 CDB: Unmap/Read sub-channel 42 00 00 00 00 00 00 00 18 00
BTRFS info (device sde1): forced readonly
BTRFS warning (device sde1): Skipping commit of aborted transaction.
BTRFS: error (device sde1) in cleanup_transaction:1894: errno=-5 IO failure
BTRFS info (device sde1): delayed_refs has NO entrybtrfs_dev_stat_print_on_error: 244 callbacks suppressed
gargamel:~# dmtail 3 [1887142.765448] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4529, flush 0, corrupt 0, gen 0 [1887142.795820] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4530, flush 0, corrupt 0, gen 0 [1887142.826176] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4531, flush 0, corrupt 0, gen 0 gargamel:~# cat /proc/partitions |grep sd[ep] 8 240 3750738264 sdp 8 241 3750737223 sdp1 gargamel:~# mount | grep sde /dev/sde1 on /mnt/btrfs_space type btrfs (ro,noatime,compress=lzo,ssd,discard,space_cache,skip_balance,subvolid=5,subvol=/) /dev/sde1 on /var/local/space type btrfs (ro,noexec,noatime,compress=lzo,ssd,discard,space_cache,skip_balance,subvolid=257,subvol=/varlocalspace) /dev/sde1 on /var/cache/zoneminder type btrfs (ro,nosuid,nodev,noatime,compress=lzo,ssd,discard,space_cache,skip_balance,subvolid=257,subvol=/varlocalspace/zoneminder) /dev/sde1 on /var/lib/mysql type btrfs (ro,nosuid,nodev,noatime,compress=lzo,ssd,discard,space_cache,skip_balance,subvolid=3648,subvol=/mysql) gargamel:~# umount /mnt/btrfs_space; umount /var/local/space; umount /var/cache/zoneminder; umount /var/lib/mysql gargamel:~# mount | grep sde
gargamel:~# mount /dev/sdp1 /mnt/mnt mount: /mnt/mnt: mount(2) system call failed: File exists. gargamel:~# dmtail 2 [1887142.826176] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4531, flush 0, corrupt 0, gen 0 [1887453.610947] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdp1
gargamel:/usr/local/bin# btrfs device scan --forget gargamel:/usr/local/bin# mount /dev/sdp1 /mnt/mnt mount: /mnt/mnt: mount(2) system call failed: File exists.
After reboot, I made sure sde is not used by anything weird, just simple mounts:
gargamel:~# lsblk | grep sde sde 8:64 1 931.5G 0 disk ├─sde1 8:65 1 488.3M 0 part ├─sde2 8:66 1 14.9G 0 part ├─sde3 8:67 1 80G 0 part └─sde4 8:68 1 836.1G 0 part
Any ideas?
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Quoted 27 lines · Marc MERLIN
Anaud, I had this happen agin with 5.5.11, and it was impossible to do anything to fix it, I had to reboot again.
btrfs device scan --forget
did nothing.See details:
BTRFS: device labelbtrfs_spacedevid 1 transid 35178413/dev/sde1
BTRFS info (device sde1): use lzo compression, level 0
BTRFS info (device sde1): disk space caching is enabled
BTRFS info (device sde1): has skinny extents
BTRFS info (device sde1): enabling ssd optimizations
sd 6:1:3:0: [sde] tag#642 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=2s sd 6:1:3:0: [sde] tag#640 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=2s sd 6:1:3:0: [sde] tag#702 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=2s sd 6:1:3:0: [sde] tag#702 CDB: Write(16) 8a 00 00 00 00 00 f1 a7 3a 68 00 00 01 f0 00 00blk_update_request: I/O error, dev sde, sector 4054268520 op 0x1:(WRITE) flags0x100000phys_seg62 prio class 0 sd 6:1:3:0: [sde] tag#701 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=2s sd 6:1:3:0: [sde] tag#701 CDB: Write(16) 8a 00 00 00 00 00 f1 a7 38 68 00 00 02 00 00 00blk_update_request: I/O error, dev sde, sector 4054268008 op 0x1:(WRITE) flags0x104000phys_seg64 prio class 0 sd 6:1:3:0: [sde] tag#700 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=2s sd 6:1:3:0: [sde] tag#700 CDB: Write(16) 8a 00 00 00 00 00 f1 a7 36 68 00 00 02 00 00 00blk_update_request: I/O error, dev sde, sector 4054267496 op 0x1:(WRITE) flags0x104000phys_seg64 prio class 0 BTRFS error (device sde1): bdev/dev/sde1errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
sd 6:1:3:0: [sde] tag#641 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=10s sd 6:1:3:0: [sde] tag#641 CDB: Unmap/Read sub-channel 42 00 00 00 00 00 00 00 18 00
Quoted 1 line
BTRFS info (device sde1): forced readonly
Unfortunately that's the only thing we do as of now.
Quoted 5 lines
BTRFS warning (device sde1): Skipping commit of aborted transaction.
BTRFS: error (device sde1) in cleanup_transaction:1894: errno=-5 IO failure BTRFS info (device sde1):delayed_refshas NO entrybtrfs_dev_stat_print_on_error: 244 callbacks suppressed
Quoted 8 lines
gargamel:~# dmtail 3 [1887142.765448] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4529, flush 0, corrupt 0, gen 0 [1887142.795820] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4530, flush 0, corrupt 0, gen 0 [1887142.826176] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4531, flush 0, corrupt 0, gen 0gargamel:~# cat /proc/partitions |grep sd[ep] 8 240 3750738264 sdp 8 241 3750737223 sdp1
So the same device reappears as sdp. But btrfs does not close a failed device yet (patches are in the mailing list) the old path sde is still in the block layer and opened. I guess /proc/partitions
doesn't show non working sde.
Quoted 10 lines
gargamel:~# mount | grep sde /dev/sde1 on /mnt/btrfs_space type btrfs (ro,noatime,compress=lzo,ssd,discard,space_cache,skip_balance,subvolid=5,subvol=/) /dev/sde1 on /var/local/space type btrfs (ro,noexec,noatime,compress=lzo,ssd,discard,space_cache,skip_balance,subvolid=257,subvol=/varlocalspace) /dev/sde1 on /var/cache/zoneminder type btrfs (ro,nosuid,nodev,noatime,compress=lzo,ssd,discard,space_cache,skip_balance,subvolid=257,subvol=/varlocalspace/zoneminder) /dev/sde1 on /var/lib/mysql type btrfs (ro,nosuid,nodev,noatime,compress=lzo,ssd,discard,space_cache,skip_balance,subvolid=3648,subvol=/mysql)gargamel:~# umount /mnt/btrfs_space; umount /var/local/space; umount /var/cache/zoneminder; umount /var/lib/mysqlgargamel:~# mount | grep sde
better to have grep-ed sdp also, here.
And /proc/self/mounts will be more accurate as it probes the fs module.
Quoted 6 lines
gargamel:~# mount /dev/sdp1 /mnt/mnt mount: /mnt/mnt: mount(2) system call failed: File exists.gargamel:~# dmtail 2 [1887142.826176] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4531, flush 0, corrupt 0, gen 0 [1887453.610947] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdp1
Unmount wasn't successful above. Or it was remounted by automount? just guessing.
Quoted 5 lines
gargamel:/usr/local/bin# btrfs device scan --forget gargamel:/usr/local/bin# mount /dev/sdp1 /mnt/mnt mount: /mnt/mnt: mount(2) system call failed: File exists.
Can you please send a complete kernel logs.
Quoted 8 lines
After reboot, I made sure sde is not used by anything weird, just simple mounts:
gargamel:~# lsblk | grep sde sde 8:64 1 931.5G 0 disk ├─sde1 8:65 1 488.3M 0 part ├─sde2 8:66 1 14.9G 0 part ├─sde3 8:67 1 80G 0 part └─sde4 8:68 1 836.1G 0 part
So in summary the chronological order of events are...
sde disappears.
btrfs does not close the device.
block layer creates sdp when the disappeared device reappears. unmount of sde was tried but it might not have completely successful
we don't have sufficient logs to prove it.
mount of sdp fails per log indicates that sde is still mounted.
So thing(s) to fix is/are:
The root of the issue - When sde fails we need to close the device so that block layer can reuse sde when it reappears (not sdp). In btrfs as we have closed the failed device btrfs dev scan --forget can work to cleanup the stale entries left behind during unmount.
We can do something better here:
When two different device with same fsid uuid and devid and one of it is mounted we have to fail the scan/mount of the newer device for obvious reasons. That's when we get the log - 'duplicate device fsid'. But here the case it bit skewed that both are same device with same major number but different minor number (sde sdp). I need to figure out a way so that we don't treat these two device paths as different device. Probably should check the guid/wwid assigned by the block layer which should be same for both of these devices, or in the last resort check scsi inquiry_VPD page and get the serial number but its going too much beyond what FS should do. Let me check with block layer experts what they suggest.
We might need a workaround tool to force clean a given FSID to avoid reboot.
Still unknown:
unmount is successful? And mount logs shows that device sde still
exists in btrfs.
Sorry I was diverted into other stuffs when you reported last time, let me take a fresh look.
Thanks, Anand
Quoted 4 lines
Any ideas?
Marc
Quoted 3 lines · Anand Jain
Quoted 1 line
BTRFS info (device sde1): forced readonly
Unfortunately that's the only thing we do as of now.
Of course, and that's fine, but I don't understand why after unmounting the filesystem cleanly, the references aren't freed.
That part really seems like a bug to me.
Quoted 5 lines
So the same device reappears as sdp. But btrfs does not close a failed device yet (patches are in the mailing list) the old path sde is still in the block layer and opened. I guess
/proc/partitionsdoesn't show non working sde.
Correct on all points
Quoted 2 lines
Quoted 1 line
gargamel:~# mount | grep sdebetter to have grep-ed sdp also, here.
it was not mounted yet, I checked that.
Quoted 1 line
And
/proc/self/mountswill be more accurate as it probes the fs module.
Noted.
Quoted 5 lines
Quoted 2 lines
[1887142.826176] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4531, flush 0, corrupt 0, gen 0 [1887453.610947] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdp1Unmount wasn't successful above. Or it was remounted by automount? just guessing.
umount was successful and automount does not handle this device. /dev/sde was not mounted for sure and /dev/sdp was unmountable
Quoted 5 lines
Quoted 3 lines
gargamel:/usr/local/bin# btrfs device scan --forget gargamel:/usr/local/bin# mount /dev/sdp1 /mnt/mnt mount: /mnt/mnt: mount(2) system call failed: File exists.Can you please send a complete kernel logs.
They contain a lot of crap that wouldn't fit on the list, but I pasted everything relevant.
Quoted 2 lines
sde disappears.
btrfs does not close the device.
it remounts the mountpoints read only, which is fine (they can't be unmounted because they are in use).
Quoted 3 lines
block layer creates sdp when the disappeared device reappears. unmount of sde was tried but it might not have completely successful we
don't have sufficient logs to prove it.
umount looked complete on my side, there is nothing in the logs that shows otherwise, but as you said, unmount does not log anything.
Quoted 1 line
mount of sdp fails per log indicates that sde is still mounted.
correct.
Quoted 5 lines
So thing(s) to fix is/are:
The root of the issue - When sde fails we need to close the device so that block layer can reuse sde when it reappears (not sdp). In btrfs as we have closed the failed device btrfs dev scan --forget can work to cleanup the stale entries left behind during unmount.
ideally "btrfs dev scan --forget" should be automatic. It feels like a weird command for an admin to know or have to use. Other filesystems do not need it.
Quoted 4 lines
We can do something better here:
When two different device with same fsid uuid and devid and one of it is mounted we have to fail the scan/mount of the newer device for obvious reasons. That's when we get the log - 'duplicate device fsid'.
Quoted 8 lines
But here the case it bit skewed that both are same device with same major number but different minor number (sde sdp). I need to figure out a way so that we don't treat these two device paths as different device. Probably should check the guid/wwid assigned by the block layer which should be same for both of these devices, or in the last resort check scsi
inquiry_VPDpage and get the serial number but its going too much beyond what FS should do. Let me check with block layer experts what they suggest.
defense in depth sounds great here, if any of those can work too, that'd be great.
Quoted 3 lines
Still unknown:
unmount is successful? And mount logs shows that device sde still exists in
btrfs.
It failed while mountpoints were still it use, and after the correct fuser -kvm /path
umount worked great and the device disappeared from /proc/mounts. As you said, there are no kernel logs on unmount, so it's hard to say more.
If you want me to apply a patch that puts more logging on unmount (against 5.5 or 5.6), please let me know, but of course, it could be weeks or months before I get that blip again.
I think this could be reproduced by simply having a drive mounted, and unplugging it while the machine is live, and plugging it back in at runtime. I could technically do it with my hardware, but it happens on a a database I don't really want to lose or corrupt.
Quoted 2 lines
Sorry I was diverted into other stuffs when you reported last time, let me take a fresh look.
No worries, we've all been there :)
Also, it's not like I can get a refund on my support contract I don't have ;)
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
The steps below are they in the chronological order?
Quoted 15 lines
gargamel:~# dmtail 3
[1887142.765448] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4529, flush 0, corrupt 0, gen 0
[1887142.795820] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4530, flush 0, corrupt 0, gen 0
[1887142.826176] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4531, flush 0, corrupt 0, gen 0
gargamel:~# cat /proc/partitions |grep sd[ep]
8 240 3750738264 sdp
8 241 3750737223 sdp1
gargamel:~# mount | grep sde
/dev/sde1 on /mnt/btrfs_space type btrfs (ro,noatime,compress=lzo,ssd,discard,space_cache,skip_balance,subvolid=5,subvol=/)
/dev/sde1 on /var/local/space type btrfs (ro,noexec,noatime,compress=lzo,ssd,discard,space_cache,skip_balance,subvolid=257,subvol=/varlocalspace)
/dev/sde1 on /var/cache/zoneminder type btrfs (ro,nosuid,nodev,noatime,compress=lzo,ssd,discard,space_cache,skip_balance,subvolid=257,subvol=/varlocalspace/zoneminder)
/dev/sde1 on /var/lib/mysql type btrfs (ro,nosuid,nodev,noatime,compress=lzo,ssd,discard,space_cache,skip_balance,subvolid=3648,subvol=/mysql)
gargamel:~# umount /mnt/btrfs_space; umount /var/local/space; umount /var/cache/zoneminder; umount /var/lib/mysql
gargamel:~# mount | grep sdeQuoted 5 lines
gargamel:~# mount /dev/sdp1 /mnt/mnt mount: /mnt/mnt: mount(2) system call failed: File exists. gargamel:~# dmtail 2 [1887142.826176] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4531, flush 0, corrupt 0, gen 0 [1887453.610947] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdp1
Before and after --forget command
btrfs fi show -m
could have told us what devices are still mounted.
I will send a boilerplate code to dump device list from the kernel it will help to debug. As of now this boilderplate code which I have been using is too localized needs a lot of cleanups, will take sometime.
Quoted 3 lines
gargamel:/usr/local/bin# btrfs device scan --forget gargamel:/usr/local/bin# mount /dev/sdp1 /mnt/mnt mount: /mnt/mnt: mount(2) system call failed: File exists.
Thanks, Anand
Quoted 13 lines
After reboot, I made sure sde is not used by anything weird, just simple mounts:
gargamel:~# lsblk | grep sde sde 8:64 1 931.5G 0 disk ├─sde1 8:65 1 488.3M 0 part ├─sde2 8:66 1 14.9G 0 part ├─sde3 8:67 1 80G 0 part └─sde4 8:68 1 836.1G 0 partAny ideas?
Marc
Quoted 1 line · Anand Jain
The steps below are they in the chronological order?
That is my recollection, yes.
Quoted 3 lines
Before and after --forget command
btrfs fi show -m
could have told us what devices are still mounted.
Oh, I didn't know about this. If/when it happens next, I'll run this to show btrfs' understanding of what's mounted instead of the kernel's understanding (/proc/self/mounts)
Quoted 3 lines
I will send a boilerplate code to dump device list from the kernel it will help to debug. As of now this boilderplate code which I have been using is too localized needs a lot of cleanups, will take sometime.
Sounds good.
Thanks,
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Marc,
Could you please use the kernel patch (sent to the list or at [email protected]:asj/btrfs-boilerplate.git boilerplate-v5.6) it can dump the btrfs kernel device_list into the user space using procfs. (This patch is only for debugging).
I tried test (as below) if there will be any availability issue (that is requiring to reboot) steps used are as below, and I am unable to reproduce. When it happens again at your end, these insight into the kernel might shed some more light on the issue.
--------------------------------
$ fillfs /btrfs 10000 $ devmgt detach /dev/sda
[65985.636630] BTRFS: error (device sda) in btrfs_commit_transaction:2345: errno=-5 IO failure (Error while writing out transaction) [65985.636631] BTRFS info (device sda): forced readonly [65985.636633] BTRFS warning (device sda): Skipping commit of aborted transaction. [65985.636634] BTRFS: error (device sda) in cleanup_transaction:1894: errno=-5 IO failure [65985.636636] BTRFS info (device sda): delayed_refs has NO entry
$ devmgt attach host0
[66501.910237] BTRFS warning (device sda): duplicate device fsid:devid for 8cc98c45-1a11-4a30-bca8-9760c246ccb4:1 old:/dev/sda new:/dev/sdb
$ btrfs fi show -m Label: none uuid: 8cc98c45-1a11-4a30-bca8-9760c246ccb4 Total devices 1 FS bytes used 16.06MiB *** Some devices missing
above -m option reads the device path from the kernel which does provide as /dev/sda but as we check its access in the user-space and as its not accessible so we report missing.
$ cat /proc/fs/btrfs/devlist :: device: /dev/sda :: generation: 10 :: dev_state: |WRITEABLE|IN_FS_METADATA|dev_stats_valid bdev: not_null
$ mount /dev/sdb /btrfs1 mount: /btrfs1: mount(2) system call failed: File exists.
The above mount fails because we find the same fs signature on both /dev/sda (stale) and /dev/sdb and further the generation number on both of these devices are same.
$ btrfs in dump-super /dev/sdb | grep ^generation generation 10
$ btrfs dev scan --forget $ cat /proc/fs/btrfs/devlist :: device: /dev/sda
--forget option can't clean the device because its still mounted.
$ umount /btrfs $ cat /proc/fs/btrfs/devlist | egrep 'device:|bdev' device: /dev/sda bdev: null
unmount is successful and bdev is null. Now --forget should work.
$ btrfs dev scan --forget $ cat /proc/fs/btrfs/devlist | egrep 'device:|bdev' $
Now as there isn't any stale device in the kernel and mount will be successful.
$ mount /dev/sdb /btrfs $ cat /proc/fs/btrfs/devlist | egrep 'device:|bdev' device: /dev/sdb bdev: not_null
So reboot was required.
---------------------
Thanks, Anand
Quoted 22 lines · Marc MERLIN
Quoted 1 line · Anand Jain
The steps below are they in the chronological order?
That is my recollection, yes.
Quoted 3 lines
Before and after --forget command
btrfs fi show -m
could have told us what devices are still mounted.Oh, I didn't know about this. If/when it happens next, I'll run this to show btrfs' understanding of what's mounted instead of the kernel's understanding (
/proc/self/mounts)Quoted 3 lines
I will send a boilerplate code to dump device list from the kernel it will help to debug. As of now this boilderplate code which I have been using is too localized needs a lot of cleanups, will take sometime.
Sounds good.
Thanks,
Marc
Quoted 5 lines · Anand Jain
Marc,
Could you please use the kernel patch (sent to the list or at [email protected]:asj/btrfs-boilerplate.git boilerplate-v5.6) it can dump
Thanks.
Since it might not be super obvious to someone else how to get it, the actual patch is:
https://github.com/asj/btrfs-boilerplate/commit/f6874237fbe66c281b5bf84ead41983fb43ed9b9
or wget https://github.com/asj/btrfs-boilerplate/commit/f6874237fbe66c281b5bf84ead41983fb43ed9b9.patch
Quoted 1 line
$ cat /proc/fs/btrfs/devlist
I patched this in my 5.6 kernel, installed, rebooted, and I can confirm it's working.
I'll report back in due time :)
Thanks,
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
PATCH 1/1 btrfs: boilerplate: devlist and fsinfo
In reply to Marc MERLINFrom: Anand Jain <[email protected]>
** This patch is not for integration but to debug/visualize the btrfs device tree and fs_info. **
usage:
cat /proc/fs/btrfs/devlist
cat /proc/fs/btrfs/fsinfo
---
This patch can also be pulled from the branch boilerplate-v5.6
[email protected]:asj/btrfs-boilerplate.git boilerplate-v5.6
fs/btrfs/Makefile | 2 +- fs/btrfs/procfs.c | 600 ++++++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/procfs.h | 2 + fs/btrfs/super.c | 6 +- 4 files changed, 608 insertions(+), 2 deletions(-) create mode 100644 fs/btrfs/procfs.c create mode 100644 fs/btrfs/procfs.h
fs/btrfs/Makefile +1 -1
@@ -11,7 +11,7 @@ btrfs-y += super.o ctree.o extent-tree.o print-tree.o root-tree.o dir-item.o \ compression.o delayed-ref.o relocation.o delayed-inode.o scrub.o \ reada.o backref.o ulist.o qgroup.o send.o dev-replace.o raid56.o \ uuid-tree.o props.o free-space-tree.o tree-checker.o space-info.o \ - block-rsv.o delalloc-space.o block-group.o discard.o + block-rsv.o delalloc-space.o block-group.o discard.o procfs.o btrfs-$(CONFIG_BTRFS_FS_POSIX_ACL) += acl.o btrfs-$(CONFIG_BTRFS_FS_CHECK_INTEGRITY) += check-integrity.o
fs/btrfs/procfs.c new file +600 -0
600 lines not on this page · raw message
fs/btrfs/procfs.h new file +2 -0
@@ -0,0 +1,2 @@ +void btrfs_exit_procfs (void ); +void btrfs_init_procfs (void );
fs/btrfs/super.c +5 -1
@@ -47,7 +47,7 @@#include "tests/btrfs-tests.h" #include "block-group.h" #include "discard.h" - +#include "procfs.h" #include "qgroup.h" #define CREATE_TRACE_POINTS #include <trace/events/btrfs.h>
@@ -2393,6 +2393,8 @@ static int __init init_btrfs_fs(void)if (err)return err; +btrfs_init_procfs (); +btrfs_init_compress (); err =btrfs_init_cachep ();
@@ -2477,6 +2479,7 @@ static int __init init_btrfs_fs(void)btrfs_destroy_cachep (); free_compress:btrfs_exit_compress (); +btrfs_exit_procfs ();btrfs_exit_sysfs ();return err;
@@ -2496,6 +2499,7 @@ static void __exit exit_btrfs_fs(void)btrfs_interface_exit ();btrfs_end_io_wq_exit ();unregister_filesystem (&btrfs_fs_type); +btrfs_exit_procfs ();btrfs_exit_sysfs ();btrfs_cleanup_fs_uuids ();btrfs_exit_compress ();
Signature · 1 line
2.23.0