btrfs fails to mount on kernel 5.11.4 but works on 5.10.19
Dear all
(please cc)
not sure this is the right mailing list, but I cannot boot into 5.11.4 it gives me
devid 9 uui ..... failed to read the system array: -2 open_ctree failed
(only partial, typed in from photo)
OTOH, 5.10.19 boots without a hinch
$ btrfs fi show /
Label: none uuid: 911600cb-bd76-4299-9445-666382e8ad20
Total devices 8 FS bytes used 3.28TiB
devid 1 size 899.01GiB used 670.00GiB path /dev/sdb3
devid 2 size 489.05GiB used 271.00GiB path /dev/sdd
devid 3 size 1.82TiB used 1.58TiB path /dev/sde1
devid 4 size 931.51GiB used 708.00GiB path /dev/sdf1
devid 5 size 1.82TiB used 1.58TiB path /dev/sdc1
devid 7 size 931.51GiB used 675.00GiB path /dev/nvme2n1p1
devid 8 size 931.51GiB used 680.03GiB path /dev/nvme1n1p1
devid 9 size 931.51GiB used 678.03GiB path /dev/nvme0n1p1That is a multi disk array with all data duplicated data/sys:
$ btrfs fi us -T /
Overall:
Device size: 8.63TiB
Device allocated: 6.76TiB
Device unallocated: 1.87TiB
Device missing: 0.00B
Used: 6.57TiB
Free (estimated): 1.03TiB (min: 1.03TiB)
Free (statfs, df): 1.01TiB
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Multiple profiles: no Data Metadata System
Id Path RAID1 RAID1 RAID1 Unallocated
-- -------------- --------- -------- --------- -----------
1 /dev/sdb3 662.00GiB 8.00GiB - 229.01GiB 2 /dev/sdd 271.00GiB - - 1.55TiB 3 /dev/sde1 1.58TiB 7.00GiB - 241.02GiB 4 /dev/sdf1 701.00GiB 7.00GiB - 223.51GiB 5 /dev/sdc1 1.57TiB 10.00GiB - 241.02GiB 7 /dev/nvme2n1p1 675.00GiB - - 256.51GiB 8 /dev/nvme1n1p1 673.00GiB 7.00GiB 32.00MiB 251.48GiB 9 /dev/nvme0n1p1 671.00GiB 7.00GiB 32.00MiB 253.48GiB
-- -------------- --------- -------- --------- -----------
Total 3.36TiB 23.00GiB 32.00MiB 3.21TiB
Used 3.27TiB 15.70GiB 528.00KiB
$
Is there something wrong with the filesystem? Or the kernel? Any hint how to debug this?
(Please cc)
Thanks
Norbert
--
PREINING Norbert https://www.preining.info
Fujitsu Research Labs + IFMGA Guide + TU Wien + TeX Live + Debian Dev
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Hi,
If this is a boot btrfs filesystem, please try to mount it manually with a live linux with the kernel 5.11.4?
When a boot btrfs filesystem with multiple disks, I have a question for a little long time.
If some but not all of the disks are scaned, systemd will try to mount it? and then btrfs mount will try to wait other disks to be scaned?
some but not all disks already make '/dev/disk/by-uuid/' or '/dev/disk/by-label/' ready, and then systemd will try to mount it?
I have another problem about boot btrfs filesystem with multiple disks.
https://github.com/systemd-rhel/rhel-7/issues/129
Best Regards
Wang Yugui ([email protected])
2021/03/08
Quoted 69 lines
Dear all
(please cc)
not sure this is the right mailing list, but I cannot boot into 5.11.4 it gives me
devid 9 uui ..... failed to read the system array: -2 open_ctree failed(only partial, typed in from photo)
OTOH, 5.10.19 boots without a hinch
$ btrfs fi show / Label: none uuid: 911600cb-bd76-4299-9445-666382e8ad20 Total devices 8 FS bytes used 3.28TiB devid 1 size 899.01GiB used 670.00GiB path /dev/sdb3 devid 2 size 489.05GiB used 271.00GiB path /dev/sdd devid 3 size 1.82TiB used 1.58TiB path /dev/sde1 devid 4 size 931.51GiB used 708.00GiB path /dev/sdf1 devid 5 size 1.82TiB used 1.58TiB path /dev/sdc1 devid 7 size 931.51GiB used 675.00GiB path /dev/nvme2n1p1 devid 8 size 931.51GiB used 680.03GiB path /dev/nvme1n1p1 devid 9 size 931.51GiB used 678.03GiB path /dev/nvme0n1p1That is a multi disk array with all data duplicated data/sys:
$ btrfs fi us -T / Overall: Device size: 8.63TiB Device allocated: 6.76TiB Device unallocated: 1.87TiB Device missing: 0.00B Used: 6.57TiB Free (estimated): 1.03TiB (min: 1.03TiB) Free (statfs, df): 1.01TiB Data ratio: 2.00 Metadata ratio: 2.00 Global reserve: 512.00MiB (used: 0.00B) Multiple profiles: noData Metadata System
Id Path RAID1 RAID1 RAID1 Unallocated
-- -------------- --------- -------- --------- -----------1 /dev/sdb3 662.00GiB 8.00GiB - 229.01GiB 2 /dev/sdd 271.00GiB - - 1.55TiB 3 /dev/sde1 1.58TiB 7.00GiB - 241.02GiB 4 /dev/sdf1 701.00GiB 7.00GiB - 223.51GiB 5 /dev/sdc1 1.57TiB 10.00GiB - 241.02GiB 7 /dev/nvme2n1p1 675.00GiB - - 256.51GiB 8 /dev/nvme1n1p1 673.00GiB 7.00GiB 32.00MiB 251.48GiB 9 /dev/nvme0n1p1 671.00GiB 7.00GiB 32.00MiB 253.48GiB-- -------------- --------- -------- --------- -----------
Total 3.36TiB 23.00GiB 32.00MiB 3.21TiB
Used 3.27TiB 15.70GiB 528.00KiB
$Is there something wrong with the filesystem? Or the kernel? Any hint how to debug this?
(Please cc)
Thanks
Norbert
--
PREINING Norbert https://www.preining.info
Fujitsu Research Labs + IFMGA Guide + TU Wien + TeX Live + Debian Dev
GPG:0x860CDC13fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Hi
(please cc)
thanks for your email. First some additional information. Since this happened I searched and realized that there seem to have been a problem with 5.12-rc1, which I tried for short time (checking whether AMD-GPU hangs are fixed). Now I read that -rc1 is a btrfs-killer. I have swap partition, not swap file, and 64G or RAM, so normally swap is not used, though.
Then for your email:
Quoted 2 lines · Wang Yugui
If this is a boot btrfs filesystem, please try to mount it manually with a live linux with the kernel 5.11.4?
Hmmm, good idea, but how do I get a live system with that kernel? Not that I know any having this as default by now.
I have booted systemrescue and run now
btrfsck --check-data-csum
to see whether there are some other errors.
Quoted 2 lines
When a boot btrfs filesystem with multiple disks, I have a question for a little long time.
What kind of problems do you see?
Quoted 2 lines
If some but not all of the disks are scaned, systemd will try to mount it? and then btrfs mount will try to wait other disks to be scaned?
Hmm, but nothing has changed in the underlying boot configuration (Debian/sid). Same system only kernel changes.
(Please cc)
Thanks and all the best
Norbert
Quoted 52 lines
Quoted 52 lines
not sure this is the right mailing list, but I cannot boot into 5.11.4 it gives me
devid 9 uui ..... failed to read the system array: -2 open_ctree failed(only partial, typed in from photo)
OTOH, 5.10.19 boots without a hinch
$ btrfs fi show / Label: none uuid: 911600cb-bd76-4299-9445-666382e8ad20 Total devices 8 FS bytes used 3.28TiB devid 1 size 899.01GiB used 670.00GiB path /dev/sdb3 devid 2 size 489.05GiB used 271.00GiB path /dev/sdd devid 3 size 1.82TiB used 1.58TiB path /dev/sde1 devid 4 size 931.51GiB used 708.00GiB path /dev/sdf1 devid 5 size 1.82TiB used 1.58TiB path /dev/sdc1 devid 7 size 931.51GiB used 675.00GiB path /dev/nvme2n1p1 devid 8 size 931.51GiB used 680.03GiB path /dev/nvme1n1p1 devid 9 size 931.51GiB used 678.03GiB path /dev/nvme0n1p1That is a multi disk array with all data duplicated data/sys:
$ btrfs fi us -T / Overall: Device size: 8.63TiB Device allocated: 6.76TiB Device unallocated: 1.87TiB Device missing: 0.00B Used: 6.57TiB Free (estimated): 1.03TiB (min: 1.03TiB) Free (statfs, df): 1.01TiB Data ratio: 2.00 Metadata ratio: 2.00 Global reserve: 512.00MiB (used: 0.00B) Multiple profiles: noData Metadata System
Id Path RAID1 RAID1 RAID1 Unallocated
-- -------------- --------- -------- --------- -----------1 /dev/sdb3 662.00GiB 8.00GiB - 229.01GiB 2 /dev/sdd 271.00GiB - - 1.55TiB 3 /dev/sde1 1.58TiB 7.00GiB - 241.02GiB 4 /dev/sdf1 701.00GiB 7.00GiB - 223.51GiB 5 /dev/sdc1 1.57TiB 10.00GiB - 241.02GiB 7 /dev/nvme2n1p1 675.00GiB - - 256.51GiB 8 /dev/nvme1n1p1 673.00GiB 7.00GiB 32.00MiB 251.48GiB 9 /dev/nvme0n1p1 671.00GiB 7.00GiB 32.00MiB 253.48GiB-- -------------- --------- -------- --------- -----------
Total 3.36TiB 23.00GiB 32.00MiB 3.21TiB
Used 3.27TiB 15.70GiB 528.00KiB
$
Quoted 2 lines
Quoted 2 lines
Is there something wrong with the filesystem? Or the kernel? Any hint how to debug this?
--
PREINING Norbert https://www.preining.info
Fujitsu Research Labs + IFMGA Guide + TU Wien + TeX Live + Debian Dev
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Quoted 11 lines · Norbert Preining
Hi
(please cc)
thanks for your email. First some additional information. Since this happened I searched and realized that there seem to have been a problem with 5.12-rc1, which I tried for short time (checking whether AMD-GPU hangs are fixed). Now I read that -rc1 is a btrfs-killer. I have swap partition, not swap file, and 64G or RAM, so normally swap is not used, though.
That bug should not have affected the dedication swap partition case.
Signature · 1 line
Chris Murphy
Hi Chris,
thanks a lot for your email!
Quoted 2 lines
Post the photo? This is a generic message and we need to see more information. Is devid 9 missing?
Yes it is missing. I attach the screenshot.
Quoted 2 lines
Does the initrd on this system contain?
/usr/lib/udev/rules.d/64-btrfs.rules
Good question, I don't have access now since I am doing the
btrfsck --check-data-csum
which takes a bit of time. On a similar system (but without multiple disks and no raid), but otherwise the same Debian/sid, I don't have the rules file in the initrd (only in /lib/udev/rules.d/64-btrfs.rules on the running system)
Quoted 4 lines
That will wait until all devices are available before attempting to mount. If it's not in the initrd, it won't wait and it's prone to races, and you can often get mount failures because not all devices are ready to be mounted.
Ahh, ok that could be an issue - I will try to include the udev rules and try again with 5.11.4. Strange that it didn't before, though.
Quoted 2 lines
This seems to be a somewhat risky setup or at least highly performance variable. Any single device that fails will result in boot failure.
I had that, and it can normally be easily recovered via mounting in degraded mode, removing or replacing the device, and reboot. Worked last time ;-) What would your suggestion be instead?
Quoted 3 lines
Quoted 1 line
hangs are fixed). Now I read that -rc1 is a btrfs-killer. I have swap
That bug should not have affected the dedication swap partition case.
Good to hear, thanks!
Best
Norbert
--
PREINING Norbert https://www.preining.info
Fujitsu Research Labs + IFMGA Guide + TU Wien + TeX Live + Debian Dev
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Hi Chris,
once more ..
Quoted 2 lines
Quoted 2 lines
Does the initrd on this system contain?
/usr/lib/udev/rules.d/64-btrfs.rules
No, it didn't.
Now I added it, and with 64-btrfs.rules available in the initrd I still get the same error (see previous screenshot) :-(
Best
Norbert
--
PREINING Norbert https://www.preining.info
Fujitsu Research Labs + IFMGA Guide + TU Wien + TeX Live + Debian Dev
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Quoted 5 lines · Norbert Preining
Hi Chris,
once more ..
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Does the initrd on this system contain?
/usr/lib/udev/rules.d/64-btrfs.rulesNo, it didn't.
Now I added it, and with 64-btrfs.rules available in the initrd I still get the same error (see previous screenshot) :-(
I suspect something is wrong with devid 9 in that case. If it's a dracut system, then it waits indefinitely for sysroot. You'll need to boot with something like rd.break=pre-mount and see first if you can mount normally to /sysroot, but if devid 9 is still missing then mount degraded and replace that device. Or otherwise find out why it's missing.
I don't think the scrub helps right now, the issue is the device is missing. Where scrub does help is if the device reappears for normal mount following previous degraded mount - the scrub is needed to get the missing device caught up with the rest.
Signature · 1 line
Chris Murphy
Hi Chris,
Quoted 6 lines · Chris Murphy
I suspect something is wrong with devid 9 in that case. If it's a dracut system, then it waits indefinitely for sysroot. You'll need to boot with something like rd.break=pre-mount and see first if you can mount normally to /sysroot, but if devid 9 is still missing then mount degraded and replace that device. Or otherwise find out why it's missing.
Ok, I will see what I can do. It isn't a dracut system, but initramfs-tools AFAIS (not that I ever touched or decided anything in this respect).
I will try to get to that state. (Only that USB devices seem to be hosed in Debian's initramfs ... no kbd atm :-(
As said, it is just puzzling that 5.10.19 boots without a hinch and without a warning, just running it now.
Quoted 1 line
I don't think the scrub helps right now, the issue is the device is
Ok, I have already killed it ;-)
Thanks a lot
Norbert
--
PREINING Norbert https://www.preining.info
Fujitsu Research Labs + IFMGA Guide + TU Wien + TeX Live + Debian Dev
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Hi Chris, hi all,
case closed. The reason was that some NVMe devices weren't recognized. I don't have any idea **why** this happened, but I cleaned out the whole models tree, made a complete rebuild of the kernel, and now it works.
Sorry for the noise.
And I also switched to dracut on the go, thanks for the suggestions.
Best
Norbert
--
PREINING Norbert https://www.preining.info
Fujitsu Research Labs + IFMGA Guide + TU Wien + TeX Live + Debian Dev
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Quoted 11 lines · Norbert Preining
Dear all
(please cc)
not sure this is the right mailing list, but I cannot boot into 5.11.4 it gives me
devid 9 uui .....
failed to read the system array: -2
open_ctreefailed
(only partial, typed in from photo)
Post the photo? This is a generic message and we need to see more information. Is devid 9 missing?
Does the initrd on this system contain?
/usr/lib/udev/rules.d/64-btrfs.rules
That will wait until all devices are available before attempting to mount. If it's not in the initrd, it won't wait and it's prone to races, and you can often get mount failures because not all devices are ready to be mounted.
Quoted 13 lines
OTOH, 5.10.19 boots without a hinch
$ btrfs fi show / Label: none uuid: 911600cb-bd76-4299-9445-666382e8ad20 Total devices 8 FS bytes used 3.28TiB devid 1 size 899.01GiB used 670.00GiB path /dev/sdb3 devid 2 size 489.05GiB used 271.00GiB path /dev/sdd devid 3 size 1.82TiB used 1.58TiB path /dev/sde1 devid 4 size 931.51GiB used 708.00GiB path /dev/sdf1 devid 5 size 1.82TiB used 1.58TiB path /dev/sdc1 devid 7 size 931.51GiB used 675.00GiB path /dev/nvme2n1p1 devid 8 size 931.51GiB used 680.03GiB path /dev/nvme1n1p1 devid 9 size 931.51GiB used 678.03GiB path /dev/nvme0n1p1
This seems to be a somewhat risky setup or at least highly performance variable. Any single device that fails will result in boot failure.
Signature · 1 line
Chris Murphy