Skip to contentSkip to conversation

5.4.20: cannot mount device that blipped off the bus: duplicate device fsid:devid for

PATCH New Marc MERLIN 3 participants 15 messages Mar 21, 2020 – Apr 22, 2020 Btrfs file system

lore
Share & tools
lore
#0 lore raw

/dev/sde blipped off the bus (hardware issue?) and came back as /dev/sdq.
Except btrfs won't let me scan or mount it.

I was able to btrfs check it though and that came back clean.

gargamel:~# ls -l /dev/sde
ls: cannot access '/dev/sde': No such file or directory
gargamel:~# mount /dev/sdq1 /mnt/mnt
mount: /mnt/mnt: mount(2) system call failed: File exists.
gargamel:~# dmesg |tail -1
[2560371.195249] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdq1
gargamel:~# btrfs device scan
Scanning for Btrfs filesystems
ERROR: device scan failed on '/dev/sdq1': File exists
ERROR: there are 1 errors while registering devices
gargamel:~# dmesg |tail -1
[2560416.434529] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdq1
gargamel:~# grep sde /proc/mounts 
cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0
gargamel:~# 
gargamel:~# lsblk -f |grep 727c7ba3-f6f9-462a-8472-453dd7d46d8a
└─sdq1                            btrfs             btrfs_space                 727c7ba3-f6f9-462a-8472-453dd7d46d8a   
gargamel:~# 

So, that FS isn't a duplicate anymore and I see to have no way out except reboot which I'll do now.

Was there another way around it? Obviously this is not desirable behaviour, in the past, I was able to remount the device when it came back.

Thanks,
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#1 lore raw
Quoted 30 lines · Marc MERLIN

/dev/sde blipped off the bus (hardware issue?) and came back as /dev/sdq.
Except btrfs won't let me scan or mount it.

I was able to btrfs check it though and that came back clean.

gargamel:~# ls -l /dev/sde
ls: cannot access '/dev/sde': No such file or directory
gargamel:~# mount /dev/sdq1 /mnt/mnt
mount: /mnt/mnt: mount(2) system call failed: File exists.
gargamel:~# dmesg |tail -1
[2560371.195249] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdq1
gargamel:~# btrfs device scan
Scanning for Btrfs filesystems
ERROR: device scan failed on '/dev/sdq1': File exists
ERROR: there are 1 errors while registering devices
gargamel:~# dmesg |tail -1
[2560416.434529] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdq1
gargamel:~# grep sde /proc/mounts 
cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0
gargamel:~# 
gargamel:~# lsblk -f |grep 727c7ba3-f6f9-462a-8472-453dd7d46d8a
└─sdq1                            btrfs             btrfs_space                 727c7ba3-f6f9-462a-8472-453dd7d46d8a   
gargamel:~# 
Quoted 7 lines

So, that FS isn't a duplicate anymore and I see to have no way out except reboot which I'll do now.

Was there another way around it? Obviously this is not desirable behaviour, in the past, I was able to remount the device when it came back.

Presumably you could have used the device forget functionality that got introduced in 5.1, i.e the BTRFS_IOC_FORGET_DEV ioctl. For more info check out: 228a73abde5c04428678e917b271f8526cfd90ed

Quoted 3 lines

Thanks,
Marc

#2 lore raw

Thanks for the suggestion Nikolay

Dear Anand, David,

I see that
https://gitlab.freedesktop.org/seanpaul/dpu-staging/commit/228a73abde5c04428678e917b271f8526cfd90ed
may have helped, but is this really something a user should know/do?

Why does a device that disappeared from the bus, need to be manually unregistered?
Are users really supposed to know this?
Why does btrfs device scan not invalidate the cache of devices and keep remembering a device that's gone (not visible in new scan)?

Thanks,
Marc

Quoted 49 lines · Nikolay Borisov
Quoted 30 lines · Marc MERLIN

/dev/sde blipped off the bus (hardware issue?) and came back as /dev/sdq.
Except btrfs won't let me scan or mount it.

I was able to btrfs check it though and that came back clean.

gargamel:~# ls -l /dev/sde
ls: cannot access '/dev/sde': No such file or directory
gargamel:~# mount /dev/sdq1 /mnt/mnt
mount: /mnt/mnt: mount(2) system call failed: File exists.
gargamel:~# dmesg |tail -1
[2560371.195249] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdq1
gargamel:~# btrfs device scan
Scanning for Btrfs filesystems
ERROR: device scan failed on '/dev/sdq1': File exists
ERROR: there are 1 errors while registering devices
gargamel:~# dmesg |tail -1
[2560416.434529] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdq1
gargamel:~# grep sde /proc/mounts 
cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0
gargamel:~# 
gargamel:~# lsblk -f |grep 727c7ba3-f6f9-462a-8472-453dd7d46d8a
└─sdq1                            btrfs             btrfs_space                 727c7ba3-f6f9-462a-8472-453dd7d46d8a   
gargamel:~# 
Quoted 7 lines

So, that FS isn't a duplicate anymore and I see to have no way out except reboot which I'll do now.

Was there another way around it? Obviously this is not desirable behaviour, in the past, I was able to remount the device when it came back.

Presumably you could have used the device forget functionality that got introduced in 5.1, i.e the BTRFS_IOC_FORGET_DEV ioctl. For more info check out: 228a73abde5c04428678e917b271f8526cfd90ed

Quoted 3 lines

Thanks,
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#3 lore raw

Hi Marc,

Quoted 8 lines · Marc MERLIN

Thanks for the suggestion Nikolay

Dear Anand, David,

I see that
https://gitlab.freedesktop.org/seanpaul/dpu-staging/commit/228a73abde5c04428678e917b271f8526cfd90ed
may have helped, but is this really something a user should know/do?

Quoted 6 lines

Why does a device that disappeared from the bus, need to be manually unregistered?

Are users really supposed to know this?
Why does btrfs device scan not invalidate the cache of devices and keep remembering a device that's gone (not visible in new scan)?

btrfs device scan --forget is only useful to cleanup the unmounted devices, per the logs below the device was mounted when it disappeared. More below.

Quoted 4 lines

Thanks,
Marc

Quoted 18 lines
Quoted 17 lines · Nikolay Borisov
Quoted 14 lines · Marc MERLIN

/dev/sde blipped off the bus (hardware issue?) and came back as /dev/sdq.
Except btrfs won't let me scan or mount it.

I was able to btrfs check it though and that came back clean.

gargamel:~# ls -l /dev/sde
ls: cannot access '/dev/sde': No such file or directory
gargamel:~# mount /dev/sdq1 /mnt/mnt
mount: /mnt/mnt: mount(2) system call failed: File exists.
gargamel:~# dmesg |tail -1
[2560371.195249] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdq1

This indicates the device was mounted when it disappeared. So it re-appears with the new path, but as its fsid+uuid+devid matches with the old still mounted device we rightly consider it as an alien device and fail the mount.

To avoid assigning new path to the reappearing device we need to close/pause the device path when it disappears. I need to figure out if there is any KPI from the block layer to help doing that.

Anyone- any idea if there is anything in the block layer which can do the callback into the filesystem if the device disappears?

Thanks, Anand

Quoted 33 lines
Quoted 32 lines
Quoted 16 lines
gargamel:~# btrfs device scan
Scanning for Btrfs filesystems
ERROR: device scan failed on '/dev/sdq1': File exists
ERROR: there are 1 errors while registering devices
gargamel:~# dmesg |tail -1
[2560416.434529] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdq1
gargamel:~# grep sde /proc/mounts
cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0
gargamel:~#
gargamel:~# lsblk -f |grep 727c7ba3-f6f9-462a-8472-453dd7d46d8a
└─sdq1                            btrfs             btrfs_space                 727c7ba3-f6f9-462a-8472-453dd7d46d8a
gargamel:~#
Quoted 7 lines

So, that FS isn't a duplicate anymore and I see to have no way out except reboot which I'll do now.

Was there another way around it? Obviously this is not desirable behaviour, in the past, I was able to remount the device when it came back.

Presumably you could have used the device forget functionality that got introduced in 5.1, i.e the BTRFS_IOC_FORGET_DEV ioctl. For more info check out: 228a73abde5c04428678e917b271f8526cfd90ed

Quoted 3 lines

Thanks,
Marc

#4 lore raw
Quoted 7 lines · Anand Jain
Quoted 3 lines

Are users really supposed to know this?
Why does btrfs device scan not invalidate the cache of devices and keep remembering a device that's gone (not visible in new scan)?

btrfs device scan --forget is only useful to cleanup the unmounted devices, per the logs below the device was mounted when it disappeared. More below.

I'm confused: why is --forget even needed? Why would it remember devices that were unmounted and not part of a new scan?

And yes, the device was not unmounted. The sata layer failed, device disappeared while mounted and then re-appeared
I was able to force umount the mountpoints, so maybe --forget would have helped, but I'm confused as to why it even exists.

Quoted 4 lines

This indicates the device was mounted when it disappeared. So it re-appears with the new path, but as its fsid+uuid+devid matches with the old still mounted device we rightly consider it as an alien device and fail the mount.

It was unmounted after disappearing, see the 'grep sde /proc/mounts' showing that it wasn't mounted anymore, so it seems that even that part didn't work as intended?

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#5 lore raw
Quoted 12 lines · Marc MERLIN
Quoted 7 lines · Anand Jain
Quoted 3 lines

Are users really supposed to know this?
Why does btrfs device scan not invalidate the cache of devices and keep remembering a device that's gone (not visible in new scan)?

btrfs device scan --forget is only useful to cleanup the unmounted devices, per the logs below the device was mounted when it disappeared. More below.

I'm confused: why is --forget even needed? Why would it remember devices that were unmounted and not part of a new scan?

Quoted 7 lines

And yes, the device was not unmounted. The sata layer failed, device disappeared while mounted and then re-appeared

I was able to force umount the mountpoints, so maybe --forget would have helped, but I'm confused as to why it even exists.

We would log the below only if the old device sde is still in mounted state. Unfortunately we don't have the unmount event log yet (patches are in the ML) so we don't know if unmount was successful.

[2560416.434529] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdq1

If the device is unmounted, the scan would have replaced the sde to sdi, unless the sde (stale) generation is > generation in sdi (lost commit). In which case the --forget is useful to remove the state device entry (provided device is unmounted).

Quoted 8 lines
Quoted 4 lines

This indicates the device was mounted when it disappeared. So it re-appears with the new path, but as its fsid+uuid+devid matches with the old still mounted device we rightly consider it as an alien device and fail the mount.

It was unmounted after disappearing, see the 'grep sde /proc/mounts' showing that it wasn't mounted anymore, so it seems that even that part didn't work as intended?

Its strange /proc/mounts doesn't list sde. Could you please send me complete kernel logs. Lets try if there is any clue.

I tried to reproduce.. but in my case the unmount was successful.

$ mkfs.btrfs -fq /dev/sdc && mount /dev/sdc /btrfs
$ devmgt show | grep sdc
host2 sdc
$ devmgt detach /dev/sdc
::
detach /dev/sdc successful
$ devmgt attach host2
::
	sd 2:0:0:0: [sdb] Attached SCSI disk
::
	BTRFS warning (device sdc): duplicate device fsid:devid for 
dcbc5603-e1cf-4d8d-9ec2-832bc3ac4e36:1 old:/dev/sdc new:/dev/sdb
------------------
attach host2 successful

mounts shows sdc is in ro.

$ cat /proc/mounts | grep sdc
/dev/sdc /btrfs btrfs ro,relatime,noacl,space_cache,subvolid=5,subvol=/ 0 0
$ dmesg -k | tail
[ 1427.268767] BTRFS warning (device sdc): duplicate device fsid:devid for dcbc5603-e1cf-4d8d-9ec2-832bc3ac4e36:1 old:/dev/sdc new:/dev/sdb
$ umount /dev/sdc

Unfortunately there is no log about the unmount :-(.

And the following device scan replaces the sdc with sdb.

$ btrfs dev scan
Scanning for Btrfs filesystems
$ cat /proc/fs/btrfs/devlist | grep sdc
$ cat /proc/fs/btrfs/devlist | grep sdb
		device:		/dev/sdb
$

And mount is successful.

$ mount /dev/sdb /btrfs

Thanks, Anand

Quoted 1 line

Marc

#6 lore raw
Quoted 11 lines · Anand Jain

We would log the below only if the old device sde is still in mounted state. Unfortunately we don't have the unmount event log yet (patches are in the ML) so we don't know if unmount was successful.

[2560416.434529] BTRFS warning (device sde1): duplicate device fsid:devid

for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdq1

If the device is unmounted, the scan would have replaced the sde to sdi, unless the sde (stale) generation is > generation in sdi (lost commit). In which case the --forget is useful to remove the state device entry (provided device is unmounted).

Well, the device did disappear, wouldn't that cause the in memory version to be more recent than the disk version?

Quoted 2 lines

Its strange /proc/mounts doesn't list sde. Could you please send me complete kernel logs. Lets try if there is any clue.

Sure https://pastebin.com/SWAfYxV8

Quoted 10 lines

I tried to reproduce.. but in my case the unmount was successful.

$ mkfs.btrfs -fq /dev/sdc && mount /dev/sdc /btrfs
$ devmgt show | grep sdc
host2 sdc
$ devmgt detach /dev/sdc
::
detach /dev/sdc successful
$ devmgt attach host2

That's probably too clean.
Can you
1) write to device in a loop
2) pull power from SATA device (in this case it was an ssd)
3) plug device back in

Quoted 3 lines
$ umount /dev/sdc

Unfortunately there is no log about the unmount :-(.

Maybe worth adding to help debug later?

I looked in my bash history, and it shows this:

37280  mount | grep sde
37281  umount /mnt/btrfs_space
37282  umount /var/local/space
37283  umount /var/cache/zoneminder
37284  fuser -vm /var/cache/zoneminder
37285  fuser -vkm /var/cache/zoneminder
37286  umount /var/cache/zoneminder
37287  umount /var/lib/mysql
37288  mount | grep sde1

First time, I got output to the mount command.
Second time I did not.

other commands I typed:

37289  mount /dev/sdq1 /mnt/btrfs_space
37296  btrfs device scan
37301  grep sde /etc/* 2>/dev/null
37303  mount /dev/sdq1 /mnt/btrfs_space
37308  grep -r /mnt/btrfs_space /etc 2>/dev/null
37311  btrfs device scan 
37312  l /sys/block/sde/
37313  btrfs check /dev/sdq1
37314  btrfs device scan 
37320  mount /dev/sdq1 /mnt/mnt
37323  btrfs device scan
37324  dmesg |tail -1
37326  lsblk -v
37327  lsblk 
37328  grep sde /proc/mounts 

Hope this helps.

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#7 lore raw

Anaud, I had this happen agin with 5.5.11, and it was impossible to do anything to fix it, I had to reboot again.
btrfs device scan --forget
did nothing.

See details:
BTRFS: device label btrfs_space devid 1 transid 35178413 /dev/sde1
BTRFS info (device sde1): use lzo compression, level 0
BTRFS info (device sde1): disk space caching is enabled
BTRFS info (device sde1): has skinny extents
BTRFS info (device sde1): enabling ssd optimizations
sd 6:1:3:0: [sde] tag#642 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=2s sd 6:1:3:0: [sde] tag#640 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=2s sd 6:1:3:0: [sde] tag#702 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=2s sd 6:1:3:0: [sde] tag#702 CDB: Write(16) 8a 00 00 00 00 00 f1 a7 3a 68 00 00 01 f0 00 00
blk_update_request: I/O error, dev sde, sector 4054268520 op 0x1:(WRITE) flags 0x100000 phys_seg 62 prio class 0 sd 6:1:3:0: [sde] tag#701 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=2s sd 6:1:3:0: [sde] tag#701 CDB: Write(16) 8a 00 00 00 00 00 f1 a7 38 68 00 00 02 00 00 00
blk_update_request: I/O error, dev sde, sector 4054268008 op 0x1:(WRITE) flags 0x104000 phys_seg 64 prio class 0 sd 6:1:3:0: [sde] tag#700 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=2s sd 6:1:3:0: [sde] tag#700 CDB: Write(16) 8a 00 00 00 00 00 f1 a7 36 68 00 00 02 00 00 00
blk_update_request: I/O error, dev sde, sector 4054267496 op 0x1:(WRITE) flags 0x104000 phys_seg 64 prio class 0 BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
sd 6:1:3:0: [sde] tag#641 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=10s sd 6:1:3:0: [sde] tag#641 CDB: Unmap/Read sub-channel 42 00 00 00 00 00 00 00 18 00
BTRFS info (device sde1): forced readonly
BTRFS warning (device sde1): Skipping commit of aborted transaction.
BTRFS: error (device sde1) in cleanup_transaction:1894: errno=-5 IO failure
BTRFS info (device sde1): delayed_refs has NO entry
btrfs_dev_stat_print_on_error: 244 callbacks suppressed

gargamel:~# dmtail 3
[1887142.765448] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4529, flush 0, corrupt 0, gen 0
[1887142.795820] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4530, flush 0, corrupt 0, gen 0
[1887142.826176] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4531, flush 0, corrupt 0, gen 0
gargamel:~# cat /proc/partitions  |grep sd[ep]
   8      240 3750738264 sdp
   8      241 3750737223 sdp1
gargamel:~# mount | grep sde
/dev/sde1 on /mnt/btrfs_space type btrfs (ro,noatime,compress=lzo,ssd,discard,space_cache,skip_balance,subvolid=5,subvol=/)
/dev/sde1 on /var/local/space type btrfs (ro,noexec,noatime,compress=lzo,ssd,discard,space_cache,skip_balance,subvolid=257,subvol=/varlocalspace)
/dev/sde1 on /var/cache/zoneminder type btrfs (ro,nosuid,nodev,noatime,compress=lzo,ssd,discard,space_cache,skip_balance,subvolid=257,subvol=/varlocalspace/zoneminder)
/dev/sde1 on /var/lib/mysql type btrfs (ro,nosuid,nodev,noatime,compress=lzo,ssd,discard,space_cache,skip_balance,subvolid=3648,subvol=/mysql)
gargamel:~# umount /mnt/btrfs_space; umount /var/local/space; umount /var/cache/zoneminder; umount /var/lib/mysql
gargamel:~# mount | grep sde
gargamel:~# mount /dev/sdp1 /mnt/mnt
mount: /mnt/mnt: mount(2) system call failed: File exists.
gargamel:~# dmtail 2
[1887142.826176] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4531, flush 0, corrupt 0, gen 0
[1887453.610947] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdp1
gargamel:/usr/local/bin# btrfs device scan --forget 
gargamel:/usr/local/bin# mount /dev/sdp1 /mnt/mnt
mount: /mnt/mnt: mount(2) system call failed: File exists.

After reboot, I made sure sde is not used by anything weird, just simple mounts:

gargamel:~# lsblk  | grep sde
sde                                 8:64   1 931.5G  0 disk  
├─sde1                              8:65   1 488.3M  0 part  
├─sde2                              8:66   1  14.9G  0 part  
├─sde3                              8:67   1    80G  0 part  
└─sde4                              8:68   1 836.1G  0 part 

Any ideas?

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#8 lore raw
Quoted 27 lines · Marc MERLIN

Anaud, I had this happen agin with 5.5.11, and it was impossible to do anything to fix it, I had to reboot again.
btrfs device scan --forget
did nothing.

See details:
BTRFS: device label btrfs_space devid 1 transid 35178413 /dev/sde1
BTRFS info (device sde1): use lzo compression, level 0
BTRFS info (device sde1): disk space caching is enabled
BTRFS info (device sde1): has skinny extents
BTRFS info (device sde1): enabling ssd optimizations
sd 6:1:3:0: [sde] tag#642 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=2s sd 6:1:3:0: [sde] tag#640 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=2s sd 6:1:3:0: [sde] tag#702 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=2s sd 6:1:3:0: [sde] tag#702 CDB: Write(16) 8a 00 00 00 00 00 f1 a7 3a 68 00 00 01 f0 00 00
blk_update_request: I/O error, dev sde, sector 4054268520 op 0x1:(WRITE) flags 0x100000 phys_seg 62 prio class 0 sd 6:1:3:0: [sde] tag#701 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=2s sd 6:1:3:0: [sde] tag#701 CDB: Write(16) 8a 00 00 00 00 00 f1 a7 38 68 00 00 02 00 00 00
blk_update_request: I/O error, dev sde, sector 4054268008 op 0x1:(WRITE) flags 0x104000 phys_seg 64 prio class 0 sd 6:1:3:0: [sde] tag#700 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=2s sd 6:1:3:0: [sde] tag#700 CDB: Write(16) 8a 00 00 00 00 00 f1 a7 36 68 00 00 02 00 00 00
blk_update_request: I/O error, dev sde, sector 4054267496 op 0x1:(WRITE) flags 0x104000 phys_seg 64 prio class 0 BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
sd 6:1:3:0: [sde] tag#641 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=10s sd 6:1:3:0: [sde] tag#641 CDB: Unmap/Read sub-channel 42 00 00 00 00 00 00 00 18 00

Quoted 1 line

BTRFS info (device sde1): forced readonly

Unfortunately that's the only thing we do as of now.

Quoted 5 lines

BTRFS warning (device sde1): Skipping commit of aborted transaction.
BTRFS: error (device sde1) in cleanup_transaction:1894: errno=-5 IO failure BTRFS info (device sde1): delayed_refs has NO entry btrfs_dev_stat_print_on_error: 244 callbacks suppressed

Quoted 8 lines
gargamel:~# dmtail 3
[1887142.765448] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4529, flush 0, corrupt 0, gen 0
[1887142.795820] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4530, flush 0, corrupt 0, gen 0
[1887142.826176] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4531, flush 0, corrupt 0, gen 0
gargamel:~# cat /proc/partitions  |grep sd[ep]
    8      240 3750738264 sdp
    8      241 3750737223 sdp1

So the same device reappears as sdp. But btrfs does not close a failed device yet (patches are in the mailing list) the old path sde is still in the block layer and opened. I guess /proc/partitions
doesn't show non working sde.

Quoted 10 lines
gargamel:~# mount | grep sde /dev/sde1 on /mnt/btrfs_space type btrfs (ro,noatime,compress=lzo,ssd,discard,space_cache,skip_balance,subvolid=5,subvol=/)
/dev/sde1 on /var/local/space type btrfs (ro,noexec,noatime,compress=lzo,ssd,discard,space_cache,skip_balance,subvolid=257,subvol=/varlocalspace)
/dev/sde1 on /var/cache/zoneminder type btrfs (ro,nosuid,nodev,noatime,compress=lzo,ssd,discard,space_cache,skip_balance,subvolid=257,subvol=/varlocalspace/zoneminder)
/dev/sde1 on /var/lib/mysql type btrfs (ro,nosuid,nodev,noatime,compress=lzo,ssd,discard,space_cache,skip_balance,subvolid=3648,subvol=/mysql)
gargamel:~# umount /mnt/btrfs_space; umount /var/local/space; umount /var/cache/zoneminder; umount /var/lib/mysql
gargamel:~# mount | grep sde

better to have grep-ed sdp also, here.
And /proc/self/mounts will be more accurate as it probes the fs module.

Quoted 6 lines
gargamel:~# mount /dev/sdp1 /mnt/mnt
mount: /mnt/mnt: mount(2) system call failed: File exists.
gargamel:~# dmtail 2
[1887142.826176] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4531, flush 0, corrupt 0, gen 0
[1887453.610947] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdp1

Unmount wasn't successful above. Or it was remounted by automount? just guessing.

Quoted 5 lines
gargamel:/usr/local/bin# btrfs device scan --forget
gargamel:/usr/local/bin# mount /dev/sdp1 /mnt/mnt
mount: /mnt/mnt: mount(2) system call failed: File exists.

Can you please send a complete kernel logs.

Quoted 8 lines

After reboot, I made sure sde is not used by anything weird, just simple mounts:

gargamel:~# lsblk  | grep sde
sde                                 8:64   1 931.5G  0 disk
├─sde1                              8:65   1 488.3M  0 part
├─sde2                              8:66   1  14.9G  0 part
├─sde3                              8:67   1    80G  0 part
└─sde4                              8:68   1 836.1G  0 part

So in summary the chronological order of events are...

sde disappears.
btrfs does not close the device.
block layer creates sdp when the disappeared device reappears. unmount of sde was tried but it might not have completely successful
we don't have sufficient logs to prove it.
mount of sdp fails per log indicates that sde is still mounted.

So thing(s) to fix is/are:
The root of the issue - When sde fails we need to close the device so that block layer can reuse sde when it reappears (not sdp). In btrfs as we have closed the failed device btrfs dev scan --forget can work to cleanup the stale entries left behind during unmount.

We can do something better here:
When two different device with same fsid uuid and devid and one of it is mounted we have to fail the scan/mount of the newer device for obvious reasons. That's when we get the log - 'duplicate device fsid'. But here the case it bit skewed that both are same device with same major number but different minor number (sde sdp). I need to figure out a way so that we don't treat these two device paths as different device. Probably should check the guid/wwid assigned by the block layer which should be same for both of these devices, or in the last resort check scsi inquiry_VPD page and get the serial number but its going too much beyond what FS should do. Let me check with block layer experts what they suggest.

We might need a workaround tool to force clean a given FSID to avoid reboot.

Still unknown:
unmount is successful? And mount logs shows that device sde still
exists in btrfs.

Sorry I was diverted into other stuffs when you reported last time, let me take a fresh look.

Thanks, Anand

Quoted 4 lines

Any ideas?

Marc

#9 lore raw
Quoted 3 lines · Anand Jain
Quoted 1 line

BTRFS info (device sde1): forced readonly

Unfortunately that's the only thing we do as of now.

Of course, and that's fine, but I don't understand why after unmounting the filesystem cleanly, the references aren't freed.
That part really seems like a bug to me.

Quoted 5 lines

So the same device reappears as sdp. But btrfs does not close a failed device yet (patches are in the mailing list) the old path sde is still in the block layer and opened. I guess /proc/partitions doesn't show non working sde.

Correct on all points

Quoted 2 lines
Quoted 1 line
gargamel:~# mount | grep sde

better to have grep-ed sdp also, here.

it was not mounted yet, I checked that.

Quoted 1 line

And /proc/self/mounts will be more accurate as it probes the fs module.

Noted.

Quoted 5 lines
Quoted 2 lines
[1887142.826176] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4531, flush 0, corrupt 0, gen 0
[1887453.610947] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdp1

Unmount wasn't successful above. Or it was remounted by automount? just guessing.

umount was successful and automount does not handle this device. /dev/sde was not mounted for sure and /dev/sdp was unmountable

Quoted 5 lines
Quoted 3 lines
gargamel:/usr/local/bin# btrfs device scan --forget
gargamel:/usr/local/bin# mount /dev/sdp1 /mnt/mnt
mount: /mnt/mnt: mount(2) system call failed: File exists.

Can you please send a complete kernel logs.

They contain a lot of crap that wouldn't fit on the list, but I pasted everything relevant.

Quoted 2 lines

sde disappears.
btrfs does not close the device.

it remounts the mountpoints read only, which is fine (they can't be unmounted because they are in use).

Quoted 3 lines

block layer creates sdp when the disappeared device reappears. unmount of sde was tried but it might not have completely successful we
don't have sufficient logs to prove it.

umount looked complete on my side, there is nothing in the logs that shows otherwise, but as you said, unmount does not log anything.

Quoted 1 line

mount of sdp fails per log indicates that sde is still mounted.

correct.

Quoted 5 lines

So thing(s) to fix is/are:
The root of the issue - When sde fails we need to close the device so that block layer can reuse sde when it reappears (not sdp). In btrfs as we have closed the failed device btrfs dev scan --forget can work to cleanup the stale entries left behind during unmount.

ideally "btrfs dev scan --forget" should be automatic. It feels like a weird command for an admin to know or have to use. Other filesystems do not need it.

Quoted 4 lines

We can do something better here:
When two different device with same fsid uuid and devid and one of it is mounted we have to fail the scan/mount of the newer device for obvious reasons. That's when we get the log - 'duplicate device fsid'.

Quoted 8 lines

But here the case it bit skewed that both are same device with same major number but different minor number (sde sdp). I need to figure out a way so that we don't treat these two device paths as different device. Probably should check the guid/wwid assigned by the block layer which should be same for both of these devices, or in the last resort check scsi inquiry_VPD page and get the serial number but its going too much beyond what FS should do. Let me check with block layer experts what they suggest.

defense in depth sounds great here, if any of those can work too, that'd be great.

Quoted 3 lines

Still unknown:
unmount is successful? And mount logs shows that device sde still exists in
btrfs.

It failed while mountpoints were still it use, and after the correct fuser -kvm /path
umount worked great and the device disappeared from /proc/mounts. As you said, there are no kernel logs on unmount, so it's hard to say more.
If you want me to apply a patch that puts more logging on unmount (against 5.5 or 5.6), please let me know, but of course, it could be weeks or months before I get that blip again.
I think this could be reproduced by simply having a drive mounted, and unplugging it while the machine is live, and plugging it back in at runtime. I could technically do it with my hardware, but it happens on a a database I don't really want to lose or corrupt.

Quoted 2 lines

Sorry I was diverted into other stuffs when you reported last time, let me take a fresh look.

No worries, we've all been there :)
Also, it's not like I can get a refund on my support contract I don't have ;)

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#10 lore raw
In reply to Marc MERLIN

The steps below are they in the chronological order?

Quoted 15 lines
gargamel:~# dmtail 3
[1887142.765448] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4529, flush 0, corrupt 0, gen 0
[1887142.795820] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4530, flush 0, corrupt 0, gen 0
[1887142.826176] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4531, flush 0, corrupt 0, gen 0
gargamel:~# cat /proc/partitions  |grep sd[ep]
    8      240 3750738264 sdp
    8      241 3750737223 sdp1
gargamel:~# mount | grep sde
/dev/sde1 on /mnt/btrfs_space type btrfs (ro,noatime,compress=lzo,ssd,discard,space_cache,skip_balance,subvolid=5,subvol=/)
/dev/sde1 on /var/local/space type btrfs (ro,noexec,noatime,compress=lzo,ssd,discard,space_cache,skip_balance,subvolid=257,subvol=/varlocalspace)
/dev/sde1 on /var/cache/zoneminder type btrfs (ro,nosuid,nodev,noatime,compress=lzo,ssd,discard,space_cache,skip_balance,subvolid=257,subvol=/varlocalspace/zoneminder)
/dev/sde1 on /var/lib/mysql type btrfs (ro,nosuid,nodev,noatime,compress=lzo,ssd,discard,space_cache,skip_balance,subvolid=3648,subvol=/mysql)
gargamel:~# umount /mnt/btrfs_space; umount /var/local/space; umount /var/cache/zoneminder; umount /var/lib/mysql
gargamel:~# mount | grep sde
Quoted 5 lines
gargamel:~# mount /dev/sdp1 /mnt/mnt
mount: /mnt/mnt: mount(2) system call failed: File exists.
gargamel:~# dmtail 2
[1887142.826176] BTRFS error (device sde1): bdev /dev/sde1 errs: wr 1038, rd 4531, flush 0, corrupt 0, gen 0
[1887453.610947] BTRFS warning (device sde1): duplicate device fsid:devid for 727c7ba3-f6f9-462a-8472-453dd7d46d8a:1 old:/dev/sde1 new:/dev/sdp1

Before and after --forget command
btrfs fi show -m
could have told us what devices are still mounted.

I will send a boilerplate code to dump device list from the kernel it will help to debug. As of now this boilderplate code which I have been using is too localized needs a lot of cleanups, will take sometime.

Quoted 3 lines
gargamel:/usr/local/bin# btrfs device scan --forget
gargamel:/usr/local/bin# mount /dev/sdp1 /mnt/mnt
mount: /mnt/mnt: mount(2) system call failed: File exists.

Thanks, Anand

Quoted 13 lines

After reboot, I made sure sde is not used by anything weird, just simple mounts:

gargamel:~# lsblk  | grep sde
sde                                 8:64   1 931.5G  0 disk
├─sde1                              8:65   1 488.3M  0 part
├─sde2                              8:66   1  14.9G  0 part
├─sde3                              8:67   1    80G  0 part
└─sde4                              8:68   1 836.1G  0 part

Any ideas?

Marc

#11 lore raw
Quoted 1 line · Anand Jain

The steps below are they in the chronological order?

That is my recollection, yes.

Quoted 3 lines

Before and after --forget command
btrfs fi show -m
could have told us what devices are still mounted.

Oh, I didn't know about this. If/when it happens next, I'll run this to show btrfs' understanding of what's mounted instead of the kernel's understanding (/proc/self/mounts)

Quoted 3 lines

I will send a boilerplate code to dump device list from the kernel it will help to debug. As of now this boilderplate code which I have been using is too localized needs a lot of cleanups, will take sometime.

Sounds good.

Thanks,
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#14 lore raw

PATCH 1/1 btrfs: boilerplate: devlist and fsinfo

In reply to Marc MERLIN

From: Anand Jain <[email protected]>

** This patch is not for integration but to debug/visualize the btrfs device tree and fs_info. **

usage:
cat /proc/fs/btrfs/devlist
cat /proc/fs/btrfs/fsinfo

---

This patch can also be pulled from the branch boilerplate-v5.6
[email protected]:asj/btrfs-boilerplate.git boilerplate-v5.6

 fs/btrfs/Makefile |   2 +-
 fs/btrfs/procfs.c | 600 ++++++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/procfs.h |   2 +
 fs/btrfs/super.c  |   6 +-
 4 files changed, 608 insertions(+), 2 deletions(-)
 create mode 100644 fs/btrfs/procfs.c
 create mode 100644 fs/btrfs/procfs.h
fs/btrfs/Makefile +1 -1
index 9a0ff3384381..8a894ee1d27b 100644
@@ -11,7 +11,7 @@ btrfs-y += super.o ctree.o extent-tree.o print-tree.o root-tree.o dir-item.o \
 	   compression.o delayed-ref.o relocation.o delayed-inode.o scrub.o \
 	   reada.o backref.o ulist.o qgroup.o send.o dev-replace.o raid56.o \
 	   uuid-tree.o props.o free-space-tree.o tree-checker.o space-info.o \
-	   block-rsv.o delalloc-space.o block-group.o discard.o
+	   block-rsv.o delalloc-space.o block-group.o discard.o procfs.o
 
 btrfs-$(CONFIG_BTRFS_FS_POSIX_ACL) += acl.o
 btrfs-$(CONFIG_BTRFS_FS_CHECK_INTEGRITY) += check-integrity.o
fs/btrfs/procfs.c new file +600 -0

600 lines not on this page · raw message

fs/btrfs/procfs.h new file +2 -0
new file mode 100644
index 000000000000..f7b712b58a5d
@@ -0,0 +1,2 @@
+void btrfs_exit_procfs(void);
+void btrfs_init_procfs(void);
fs/btrfs/super.c +5 -1
index 67c63858812a..c9dfceff6aa4 100644
@@ -47,7 +47,7 @@
 #include "tests/btrfs-tests.h"
 #include "block-group.h"
 #include "discard.h"
-
+#include "procfs.h"
 #include "qgroup.h"
 #define CREATE_TRACE_POINTS
 #include <trace/events/btrfs.h>
@@ -2393,6 +2393,8 @@ static int __init init_btrfs_fs(void)
 	if (err)
 		return err;
 
+	btrfs_init_procfs();
+
 	btrfs_init_compress();
 
 	err = btrfs_init_cachep();
@@ -2477,6 +2479,7 @@ static int __init init_btrfs_fs(void)
 	btrfs_destroy_cachep();
 free_compress:
 	btrfs_exit_compress();
+	btrfs_exit_procfs();
 	btrfs_exit_sysfs();
 
 	return err;
@@ -2496,6 +2499,7 @@ static void __exit exit_btrfs_fs(void)
 	btrfs_interface_exit();
 	btrfs_end_io_wq_exit();
 	unregister_filesystem(&btrfs_fs_type);
+	btrfs_exit_procfs();
 	btrfs_exit_sysfs();
 	btrfs_cleanup_fs_uuids();
 	btrfs_exit_compress();
Signature · 1 line
2.23.0
#12 lore raw
In reply to Marc MERLIN

Marc,

Could you please use the kernel patch (sent to the list or at [email protected]:asj/btrfs-boilerplate.git boilerplate-v5.6) it can dump the btrfs kernel device_list into the user space using procfs. (This patch is only for debugging).

I tried test (as below) if there will be any availability issue (that is requiring to reboot) steps used are as below, and I am unable to reproduce. When it happens again at your end, these insight into the kernel might shed some more light on the issue.

--------------------------------

$ fillfs /btrfs 10000
$ devmgt detach /dev/sda
[65985.636630] BTRFS: error (device sda) in btrfs_commit_transaction:2345: errno=-5 IO failure (Error while writing out transaction)
[65985.636631] BTRFS info (device sda): forced readonly
[65985.636633] BTRFS warning (device sda): Skipping commit of aborted transaction.
[65985.636634] BTRFS: error (device sda) in cleanup_transaction:1894: errno=-5 IO failure
[65985.636636] BTRFS info (device sda): delayed_refs has NO entry
$ devmgt attach host0
[66501.910237] BTRFS warning (device sda): duplicate device fsid:devid for 8cc98c45-1a11-4a30-bca8-9760c246ccb4:1 old:/dev/sda new:/dev/sdb
$ btrfs fi show -m
Label: none  uuid: 8cc98c45-1a11-4a30-bca8-9760c246ccb4
	Total devices 1 FS bytes used 16.06MiB
	*** Some devices missing

above -m option reads the device path from the kernel which does provide as /dev/sda but as we check its access in the user-space and as its not accessible so we report missing.

$ cat /proc/fs/btrfs/devlist
::
		device:		/dev/sda
::
		generation:	10
::
		dev_state:	|WRITEABLE|IN_FS_METADATA|dev_stats_valid
		bdev:		not_null
$ mount /dev/sdb /btrfs1
mount: /btrfs1: mount(2) system call failed: File exists.

The above mount fails because we find the same fs signature on both /dev/sda (stale) and /dev/sdb and further the generation number on both of these devices are same.

$ btrfs in dump-super /dev/sdb | grep ^generation
generation		10
$ btrfs dev scan --forget
$ cat /proc/fs/btrfs/devlist
::
		device:		/dev/sda

--forget option can't clean the device because its still mounted.

$ umount /btrfs
$ cat /proc/fs/btrfs/devlist | egrep 'device:|bdev'
		device:		/dev/sda
		bdev:		null

unmount is successful and bdev is null. Now --forget should work.

$ btrfs dev scan --forget
$ cat /proc/fs/btrfs/devlist | egrep 'device:|bdev'
$

Now as there isn't any stale device in the kernel and mount will be successful.

$ mount /dev/sdb /btrfs
$ cat /proc/fs/btrfs/devlist | egrep 'device:|bdev'
		device:		/dev/sdb
		bdev:		not_null

So reboot was required.
---------------------

Thanks, Anand

Quoted 22 lines · Marc MERLIN
Quoted 1 line · Anand Jain

The steps below are they in the chronological order?

That is my recollection, yes.

Quoted 3 lines

Before and after --forget command
btrfs fi show -m
could have told us what devices are still mounted.

Oh, I didn't know about this. If/when it happens next, I'll run this to show btrfs' understanding of what's mounted instead of the kernel's understanding (/proc/self/mounts)

Quoted 3 lines

I will send a boilerplate code to dump device list from the kernel it will help to debug. As of now this boilderplate code which I have been using is too localized needs a lot of cleanups, will take sometime.

Sounds good.

Thanks,
Marc

#13 lore raw
Quoted 5 lines · Anand Jain

Marc,

Could you please use the kernel patch (sent to the list or at [email protected]:asj/btrfs-boilerplate.git boilerplate-v5.6) it can dump

Thanks.
Since it might not be super obvious to someone else how to get it, the actual patch is:
https://github.com/asj/btrfs-boilerplate/commit/f6874237fbe66c281b5bf84ead41983fb43ed9b9

or wget https://github.com/asj/btrfs-boilerplate/commit/f6874237fbe66c281b5bf84ead41983fb43ed9b9.patch

Quoted 1 line
$ cat /proc/fs/btrfs/devlist

I patched this in my 5.6 kernel, installed, rebooted, and I can confirm it's working.

I'll report back in due time :)

Thanks,
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08