btrfs dev del not transaction protected?
Hi!
I used btrfs del /somedevice /mountpoint to remove a device, and then typed sync. A short time later the system had a hard reset.
Now the file system doesn't mount read-write anymore because it complains about a missing device (linux 5.4.5):
[ 247.385346] BTRFS error (device dm-32): devid 1 uuid f5c3dc63-1fac-45b3-b9ba-ed1ec5f92403 is missing [ 247.386942] BTRFS error (device dm-32): failed to read chunk tree: -2 [ 247.462693] BTRFS error (device dm-32): open_ctree failed
The thing is, the device is still there and accessible, but btrfs no longer recognises it, as it already deleted it before the crash.
I can mount the filesystem in degraded mode, and I have a backup in case somehting isn't readable, so this is merely a costly inconvinience for me (it's a 40TB volume), but this seems very unexpected, both that device dels apparently have a race condition and that sync doesn't actually synchronise the filesystem - I naively expected that btrfs dev del doesn't cause the loss of the filesystem due to a system crash.
Probably nbot related, but maybe worth mentioning: I found that system crashes (resets, not power failures) cause btrfs to not mount the first time a mount is attempted, but it always succeeds the second time, e.g.:
# mount /device /mnt ... no errors or warnings in kernel log, except: BTRFS error (device dm-34): open_ctree failed # mount /device /mnt magically succeeds
The typical symptom here is that systemd goes into emergency mode on mount failure, but simpyl rebooting, or executing the mount manually then succeeds.
Greetings,
Marc
Signature · 6 lines
The choice of a Deliantra, the free code+content MORPG
-----==- _GNU_ http://www.deliantra.net
----==-- _ generation
---==---(_)__ __ ____ __ Marc Lehmann
--==---/ / _ \/ // /\ \/ / [email protected]
-=====/_/_//_/\_,_/ /_/\_\Quoted 4 lines · Marc Lehmann
Hi!
I used btrfs del /somedevice /mountpoint to remove a device, and then typed sync. A short time later the system had a hard reset.
Then it doesn't look like the title.
Normally for sync, btrfs will commit transaction, thus even something like the title happened, you shouldn't be affected at all.
Quoted 7 lines
Now the file system doesn't mount read-write anymore because it complains about a missing device (linux 5.4.5):
[ 247.385346] BTRFS error (device dm-32): devid 1 uuid f5c3dc63-1fac-45b3-b9ba-ed1ec5f92403 is missing [ 247.386942] BTRFS error (device dm-32): failed to read chunk tree: -2 [ 247.462693] BTRFS error (device dm-32): open_ctree failed
Is that devid 1 the device you tried to deleted?
Or some unrelated device?
Quoted 3 lines
The thing is, the device is still there and accessible, but btrfs no longer recognises it, as it already deleted it before the crash.
I think it's not what you thought, but btrfs device scan is not properly triggered.
Would you please give some more dmesg? As each scanned btrfs device will show up in dmesg.
That would help us to pin down the real cause.
Quoted 12 lines
I can mount the filesystem in degraded mode, and I have a backup in case somehting isn't readable, so this is merely a costly inconvinience for me (it's a 40TB volume), but this seems very unexpected, both that device dels apparently have a race condition and that sync doesn't actually synchronise the filesystem - I naively expected that btrfs dev del doesn't cause the loss of the filesystem due to a system crash.
Probably nbot related, but maybe worth mentioning: I found that system crashes (resets, not power failures) cause btrfs to not mount the first time a mount is attempted, but it always succeeds the second time, e.g.:
Quoted 5 lines
# mount /device /mnt ... no errors or warnings in kernel log, except: BTRFS error (device dm-34): open_ctree failed # mount /device /mnt magically succeeds
Yep, this makes it sound more like a scan related bug.
Thanks,
Qu
Quoted 7 lines
The typical symptom here is that systemd goes into emergency mode on mount failure, but simpyl rebooting, or executing the mount manually then succeeds.
Greetings,
Marc
Quoted 4 lines · Qu Wenruo
Quoted 2 lines
I used btrfs del /somedevice /mountpoint to remove a device, and then typed sync. A short time later the system had a hard reset.
Then it doesn't look like the title.
Hmm, I am not sure I understand: do you mean the subject? The command here is obviously not copied and pasted, and when typing it into my mail client, I forgot the "dev" part. The exact command, I think, was this:
btrfs dev del /dev/mapper/xmnt-cold13 /oldcold
Quoted 2 lines
Normally for sync, btrfs will commit transaction, thus even something like the title happened, you shouldn't be affected at all.
Exactly, that is my expectation.
Quoted 6 lines
Quoted 3 lines
[ 247.385346] BTRFS error (device dm-32): devid 1 uuid f5c3dc63-1fac-45b3-b9ba-ed1ec5f92403 is missing [ 247.386942] BTRFS error (device dm-32): failed to read chunk tree: -2 [ 247.462693] BTRFS error (device dm-32): open_ctree failedIs that devid 1 the device you tried to deleted?
Or some unrelated device?
I think the device I removed had devid 1. I am not 100% sure, but I am reasonably sure because I had "watch -n10 btrfs dev us" running while waiting for the removal to finish and not being able to control the device ids triggers my ocd reflexes (mostly because btrfs fi res needs the device id even for some single-device filesystems :), so I kind of memorised them.
Quoted 5 lines
Quoted 2 lines
The thing is, the device is still there and accessible, but btrfs no longer recognises it, as it already deleted it before the crash.
I think it's not what you thought, but btrfs device scan is not properly triggered.
Quite possible - I based my statement that it is no longer recognized based on the fact that a) blkid also didn't recognize a filesystem on the removed device anymore and b) btrfs found the other two remaining devices, so if btrfs scan is not properly triggered, then this is a serious issue in current GNU/Linux distributions (I use debian buster on that server).
I assume that the device is not recognised as btrfs by blkid anymore because the signature had been wiped by btrfs dev del, based on previous experience, but I of course can't exactly know it's not, say, a hardware error that wiped that disk, although I would find that hard to believe :)
Quoted 2 lines
Would you please give some more dmesg? As each scanned btrfs device will show up in dmesg.
Here should be all btrfs-related messages for this (from grep -i btrfs):
[ 10.288533] BTRFS: device label ROOT devid 1 transid 2106939 /dev/mapper/vg_doom-root [ 10.314498] BTRFS info (device dm-0): disk space caching is enabled [ 10.316488] BTRFS info (device dm-0): has skinny extents [ 10.900930] BTRFS info (device dm-0): enabling ssd optimizations [ 10.902741] BTRFS info (device dm-0): disk space caching is enabled [ 11.524129] BTRFS info (device dm-0): device fsid bb3185c8-19f0-4018-b06f-38678c06c7c2 devid 1 moved old:/dev/mapper/vg_doom-root new:/dev/dm-0 [ 11.528554] BTRFS info (device dm-0): device fsid bb3185c8-19f0-4018-b06f-38678c06c7c2 devid 1 moved old:/dev/dm-0 new:/dev/mapper/vg_doom-root [ 42.273530] BTRFS: device label LOCALVOL3 devid 1 transid 1240483 /dev/dm-28 [ 42.312354] BTRFS info (device dm-28): enabling auto defrag [ 42.314152] BTRFS info (device dm-28): force zstd compression, level 12 [ 42.315938] BTRFS info (device dm-28): using free space tree [ 42.317696] BTRFS info (device dm-28): has skinny extents [ 49.115007] BTRFS: device label LOCALVOL5 devid 1 transid 146201 /dev/dm-29 [ 49.138816] BTRFS info (device dm-29): using free space tree [ 49.140590] BTRFS info (device dm-29): has skinny extents [ 102.348872] BTRFS info (device dm-29): checking UUID tree [ 102.393185] BTRFS: device label COLD1 devid 5 transid 1876906 /dev/dm-30 [ 109.626550] BTRFS: device label COLD1 devid 4 transid 1876907 /dev/dm-32 [ 109.654401] BTRFS: device label COLD1 devid 3 transid 1876907 /dev/dm-31 [ 109.656171] BTRFS info (device dm-32): use zstd compression, level 12 [ 109.657924] BTRFS info (device dm-32): using free space tree [ 109.660917] BTRFS info (device dm-32): has skinny extents [ 109.662687] BTRFS error (device dm-32): devid 1 uuid f5c3dc63-1fac-45b3-b9ba-ed1ec5f92403 is missing [ 109.664832] BTRFS error (device dm-32): failed to read chunk tree: -2 [ 109.742501] BTRFS error (device dm-32): open_ctree failed
At this point, /dev/mapper/xmnt-cold11 (dm-32), /dev/mapper/xmnt-oldcold12 (dm-31) and /dev/mapper/xmnt-cold14 (dm-30) were the remaining disks in the filesystem, while xmnt-cold13 was the device I had formerly removed (which doesn't show up).
(There are two btrfs filesystems with the COLD1 label in this machine at the moment, as I was migrating the fs, but the above COLD1 messages should all relate to the same fs).
"blkid -o value -s TYPE /dev/mapper/xmnt-cold13" didn't give any output (the mounting script checks for that and pauses to make provisioning of new disks easier), while normally it would give "btrfs" on volume members. This, I think, would be normal behaviour for devices that have been removed from a btrfs.
BTW, the four devices in question are all dmcrypt-on-lvm and are single devices in a hardware raid controller (a perc h740).
Quoted 11 lines
Quoted 9 lines
Probably nbot related, but maybe worth mentioning: I found that system crashes (resets, not power failures) cause btrfs to not mount the first time a mount is attempted, but it always succeeds the second time, e.g.:
# mount /device /mnt ... no errors or warnings in kernel log, except: BTRFS error (device dm-34): open_ctree failed # mount /device /mnt magically succeedsYep, this makes it sound more like a scan related bug.
BTW, this (second issue) also happens with filesystems that are not multi-device. Not sure if that menas that btrfs scan would be involved, as I would assume the only device btrfs would need in such cases is the one given to mount, but maybe that also needs a working btrfs scan?
Thanks for your working on btrfs btw. :)
Signature · 6 lines
The choice of a Deliantra, the free code+content MORPG
-----==- _GNU_ http://www.deliantra.net
----==-- _ generation
---==---(_)__ __ ____ __ Marc Lehmann
--==---/ / _ \/ // /\ \/ / [email protected]
-=====/_/_//_/\_,_/ /_/\_\Quoted 7 lines · Marc Lehmann
Quoted 4 lines · Qu Wenruo
Quoted 2 lines
I used btrfs del /somedevice /mountpoint to remove a device, and then typed sync. A short time later the system had a hard reset.
Then it doesn't look like the title.
Hmm, I am not sure I understand: do you mean the subject?
Oh, sorry, I mean subject line "btrfs dev del not transaction protected".
Quoted 3 lines
The command here
is obviously not copied and pasted, and when typing it into my mail client, I forgot the "dev" part. The exact command, I think, was this:
No big deal, as we all get the point.
Quoted 14 lines
btrfs dev del
/dev/mapper/xmnt-cold13/oldcold>Quoted 2 lines
Normally for sync, btrfs will commit transaction, thus even something like the title happened, you shouldn't be affected at all.
Exactly, that is my expectation.
Quoted 6 lines
Quoted 3 lines
[ 247.385346] BTRFS error (device dm-32): devid 1 uuid f5c3dc63-1fac-45b3-b9ba-ed1ec5f92403 is missing [ 247.386942] BTRFS error (device dm-32): failed to read chunk tree: -2 [ 247.462693] BTRFS error (device dm-32): open_ctree failedIs that devid 1 the device you tried to deleted?
Or some unrelated device?
Quoted 6 lines
I think the device I removed had devid 1. I am not 100% sure, but I am reasonably sure because I had "watch -n10 btrfs dev us" running while waiting for the removal to finish and not being able to control the device ids triggers my ocd reflexes (mostly because btrfs fi res needs the device id even for some single-device filesystems :), so I kind of memorised them.
Then it looks like a big deal.
After looking into the code (at least v5.5-rc kernel), btrfs will commit transaction after deleting the device item in btrfs_rm_dev_item().
So even no manual sync is called, as long as there is no error report from "btrfs dev del", such case shouldn't happen.
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
The thing is, the device is still there and accessible, but btrfs no longer recognises it, as it already deleted it before the crash.
I think it's not what you thought, but btrfs device scan is not properly triggered.
Quoted 6 lines
Quite possible - I based my statement that it is no longer recognized based on the fact that a) blkid also didn't recognize a filesystem on the removed device anymore and b) btrfs found the other two remaining devices, so if btrfs scan is not properly triggered, then this is a serious issue in current GNU/Linux distributions (I use debian buster on that server).
a) means btrfs has wiped the superblock, which happens after btrfs_rm_dev_item().
Something is not sane now.
Quoted 20 lines
I assume that the device is not recognised as btrfs by blkid anymore because the signature had been wiped by btrfs dev del, based on previous experience, but I of course can't exactly know it's not, say, a hardware error that wiped that disk, although I would find that hard to believe :)
Quoted 2 lines
Would you please give some more dmesg? As each scanned btrfs device will show up in dmesg.
Here should be all btrfs-related messages for this (from grep -i btrfs):
[ 10.288533] BTRFS: device label ROOT devid 1 transid 2106939 /dev/mapper/vg_doom-root [ 10.314498] BTRFS info (device dm-0): disk space caching is enabled [ 10.316488] BTRFS info (device dm-0): has skinny extents [ 10.900930] BTRFS info (device dm-0): enabling ssd optimizations [ 10.902741] BTRFS info (device dm-0): disk space caching is enabled [ 11.524129] BTRFS info (device dm-0): device fsid bb3185c8-19f0-4018-b06f-38678c06c7c2 devid 1 moved old:/dev/mapper/vg_doom-root new:/dev/dm-0 [ 11.528554] BTRFS info (device dm-0): device fsid bb3185c8-19f0-4018-b06f-38678c06c7c2 devid 1 moved old:/dev/dm-0 new:/dev/mapper/vg_doom-root [ 42.273530] BTRFS: device label LOCALVOL3 devid 1 transid 1240483 /dev/dm-28 [ 42.312354] BTRFS info (device dm-28): enabling auto defrag
Quoted 8 lines
[ 42.314152] BTRFS info (device dm-28): force zstd compression, level 12 [ 42.315938] BTRFS info (device dm-28): using free space tree [ 42.317696] BTRFS info (device dm-28): has skinny extents [ 49.115007] BTRFS: device label LOCALVOL5 devid 1 transid 146201 /dev/dm-29 [ 49.138816] BTRFS info (device dm-29): using free space tree [ 49.140590] BTRFS info (device dm-29): has skinny extents [ 102.348872] BTRFS info (device dm-29): checking UUID tree [ 102.393185] BTRFS: device label COLD1 devid 5 transid 1876906 /dev/dm-30
dm-30 is one transaction older than other devices.
Is that expected? If not, it may explain why we got the dead device. As we're using older superblock, which may points to older chunk tree which has the device item.
Quoted 2 lines
[ 109.626550] BTRFS: device label COLD1 devid 4 transid 1876907 /dev/dm-32 [ 109.654401] BTRFS: device label COLD1 devid 3 transid 1876907 /dev/dm-31
And I'm also curious about the 7s delay between devid5 and devid 3/4 detection.
Can you find a way to make devid 3/4 show up before devid 5 and try again?
And if you find a way to mount the volume RW, please write a single empty file, and sync the fs, then umount the fs, ensure "btrfs ins dump-super" gives the same transid of all 3 related disks.
Then the problem *may* be gone if it matches my assumption. (After all these assumed success, please to do an unmounted btrfs check just to make sure nothing is wrong)
Quoted 35 lines
[ 109.656171] BTRFS info (device dm-32): use zstd compression, level 12 [ 109.657924] BTRFS info (device dm-32): using free space tree [ 109.660917] BTRFS info (device dm-32): has skinny extents [ 109.662687] BTRFS error (device dm-32): devid 1 uuid f5c3dc63-1fac-45b3-b9ba-ed1ec5f92403 is missing [ 109.664832] BTRFS error (device dm-32): failed to read chunk tree: -2 [ 109.742501] BTRFS error (device dm-32): open_ctree failedAt this point,
/dev/mapper/xmnt-cold11(dm-32),/dev/mapper/xmnt-oldcold12(dm-31) and/dev/mapper/xmnt-cold14(dm-30) were the remaining disks in the filesystem, while xmnt-cold13 was the device I had formerly removed (which doesn't show up).(There are two btrfs filesystems with the COLD1 label in this machine at the moment, as I was migrating the fs, but the above COLD1 messages should all relate to the same fs).
"blkid -o value -s TYPE
/dev/mapper/xmnt-cold13" didn't give any output (the mounting script checks for that and pauses to make provisioning of new disks easier), while normally it would give "btrfs" on volume members. This, I think, would be normal behaviour for devices that have been removed from a btrfs.BTW, the four devices in question are all dmcrypt-on-lvm and are single devices in a hardware raid controller (a perc h740).
Quoted 10 lines
Quoted 9 lines
Probably nbot related, but maybe worth mentioning: I found that system crashes (resets, not power failures) cause btrfs to not mount the first time a mount is attempted, but it always succeeds the second time, e.g.:
# mount /device /mnt ... no errors or warnings in kernel log, except: BTRFS error (device dm-34): open_ctree failed # mount /device /mnt magically succeeds
Quoted 4 lines
Quoted 1 line
Yep, this makes it sound more like a scan related bug.
BTW, this (second issue) also happens with filesystems that are not multi-device.
Single device btrfs doesn't need device scan. If that happened, something insane happened again... Thanks,
Qu
Quoted 6 lines
Not sure if that menas that btrfs scan would be involved, as I would assume the only device btrfs would need in such cases is the one given to mount, but maybe that also needs a working btrfs scan?
Thanks for your working on btrfs btw. :)
Quoted 7 lines
Quoted 1 line
[ 102.393185] BTRFS: device label COLD1 devid 5 transid 1876906 /dev/dm-30dm-30 is one transaction older than other devices.
Is that expected? If not, it may explain why we got the dead device. As we're using older superblock, which may points to older chunk tree which has the device item.
Well, not that my expectation here would mean anything, but no, from experience I have never seen the transids to disagree, or bad thingsa will happen...
Quoted 5 lines
Quoted 2 lines
[ 109.626550] BTRFS: device label COLD1 devid 4 transid 1876907 /dev/dm-32 [ 109.654401] BTRFS: device label COLD1 devid 3 transid 1876907 /dev/dm-31And I'm also curious about the 7s delay between devid5 and devid 3/4 detection.
That is about the time it takes the disk to wake up when its spinned down, so maybe that was the case - the disks are used for archiving ("cold" storage), have a short spin-down and btrfs filesystems can takes ages to mount. The real question is why the fortuh disk was already spun up then, but the disks do not apply time outs very exactly.
Quoted 1 line
Can you find a way to make devid 3/4 show up before devid 5 and try again?
Unfortunately, I had to start restoring from backup a while ago, as I need the machine up and restoring takes days.
How would I go about making it show up in different orders though? If these messages come up independently, I could have spun down some of the disks, right?
Quoted 3 lines
And if you find a way to mount the volume RW, please write a single empty file, and sync the fs, then umount the fs, ensure "btrfs ins dump-super" gives the same transid of all 3 related disks.
I tried -o degraded followed by remounting rw, but couldn't get it to mount rw. I tried to mount/remount, though:
04:48:45 doom kernel: BTRFS error (device dm-32): devid 1 uuid f5c3dc63-1fac-45b3-b9ba-ed1ec5f92403 is missing 04:48:45 doom kernel: BTRFS error (device dm-32): failed to read chunk tree: -2
04:48:45 doom kernel: BTRFS error (device dm-32): open_ctree failed
04:49:37 doom kernel: BTRFS warning (device dm-31): devid 1 uuid f5c3dc63-1fac-45b3-b9ba-ed1ec5f92403 is missing 04:52:30 doom kernel: BTRFS warning (device dm-31): chunk 12582912 missing 1 devices, max tolerance is 0 for writable mount 04:52:30 doom kernel: BTRFS warning (device dm-31): writable mount is not allowed due to too many missing devices 04:52:30 doom kernel: BTRFS error (device dm-31): open_ctree failed
04:54:01 doom kernel: BTRFS warning (device dm-32): devid 1 uuid f5c3dc63-1fac-45b3-b9ba-ed1ec5f92403 is missing 04:54:45 doom kernel: BTRFS warning (device dm-32): chunk 12582912 missing 1 devices, max tolerance is 0 for writable mount 04:54:45 doom kernel: BTRFS warning (device dm-32): too many missing devices, writable remount is not allowed
Since (in theory :) the filesystemw a completely backed up, I didn't bother with further recovery after I made sure the physical disk is actually there and was unlocked (cryptsetup), so it wasn't a case of an actual missing disk.
Quoted 6 lines
Quoted 2 lines
BTW, this (second issue) also happens with filesystems that are not multi-device.
Single device btrfs doesn't need device scan. If that happened, something insane happened again... Thanks,
It happens since at least 4.14 on at least four machines, but I haven't seen it recently, after I switched to 5.2.21 one some machines (post-4.4 kernels have this habit of freezing under memory pressure, and 5.2.21 has greatly improved in this regard). That also means I had far fewer hard resets with 5.2.21, but the problem did not happen on the last resets in 5.2.21 and 5.4.5.
I originally reported it below, with some evidence that it isn't a hardware issue (no reset needed, just wipe the dm table while the device is mounted which should cleanly "cut off" the write stream):
https://bugzilla.kernel.org/show_bug.cgi?id=204083
Since multiple scrubs and full reads of the volumes didn't show up any issues, I didn't think much of it.
And if you want to hear more "insane" things, after I hard-reset my desktop machine (5.2.21) two days ago I had to "btrfs rescue fix-device-size" to be able to mount (can't find the kernel error atm.).
Greetings,
Signature · 6 lines
The choice of a Deliantra, the free code+content MORPG
-----==- _GNU_ http://www.deliantra.net
----==-- _ generation
---==---(_)__ __ ____ __ Marc Lehmann
--==---/ / _ \/ // /\ \/ / [email protected]
-=====/_/_//_/\_,_/ /_/\_\Quoted 24 lines · Marc Lehmann
Quoted 7 lines
Quoted 1 line
[ 102.393185] BTRFS: device label COLD1 devid 5 transid 1876906 /dev/dm-30dm-30 is one transaction older than other devices.
Is that expected? If not, it may explain why we got the dead device. As we're using older superblock, which may points to older chunk tree which has the device item.
Well, not that my expectation here would mean anything, but no, from experience I have never seen the transids to disagree, or bad thingsa will happen...
Quoted 5 lines
Quoted 2 lines
[ 109.626550] BTRFS: device label COLD1 devid 4 transid 1876907 /dev/dm-32 [ 109.654401] BTRFS: device label COLD1 devid 3 transid 1876907 /dev/dm-31And I'm also curious about the 7s delay between devid5 and devid 3/4 detection.
That is about the time it takes the disk to wake up when its spinned down, so maybe that was the case - the disks are used for archiving ("cold" storage), have a short spin-down and btrfs filesystems can takes ages to mount. The real question is why the fortuh disk was already spun up then, but the disks do not apply time outs very exactly.
Quoted 8 lines
Quoted 1 line
Can you find a way to make devid 3/4 show up before devid 5 and try again?
Unfortunately, I had to start restoring from backup a while ago, as I need the machine up and restoring takes days.
How would I go about making it show up in different orders though? If these messages come up independently, I could have spun down some of the disks, right?
You could utilize the latest "forget" feature, to make btrfs kernel module forget that device, provided by "btrfs device scan -u".
So the plan would be something like:
- Forget all devices of that volume
- Scan the two disks with higher transid
- Scan the disk with mismatched transid
Then try to mount the volume.
Quoted 5 lines
Quoted 4 lines
And if you find a way to mount the volume RW, please write a single empty file, and sync the fs, then umount the fs, ensure "btrfs ins dump-super" gives the same transid of all 3 related disks.
Quoted 8 lines
I tried -o degraded followed by remounting rw, but couldn't get it to mount rw. I tried to mount/remount, though:
04:48:45 doom kernel: BTRFS error (device dm-32): devid 1 uuid f5c3dc63-1fac-45b3-b9ba-ed1ec5f92403 is missing 04:48:45 doom kernel: BTRFS error (device dm-32): failed to read chunk tree: -2
04:48:45 doom kernel: BTRFS error (device dm-32):open_ctreefailed
04:49:37 doom kernel: BTRFS warning (device dm-31): devid 1 uuid f5c3dc63-1fac-45b3-b9ba-ed1ec5f92403 is missing 04:52:30 doom kernel: BTRFS warning (device dm-31): chunk 12582912 missing 1 devices, max tolerance is 0 for writable mount
BTW, that chunk number is very small, and since it has 0 tolerance, it looks like to be SINGLE chunk.
In that case, it looks like a temporary chunk from older mkfs, and it should contain no data/metadata at all, thus brings no data loss.
Quoted 10 lines
04:52:30 doom kernel: BTRFS warning (device dm-31): writable mount is not allowed due to too many missing devices 04:52:30 doom kernel: BTRFS error (device dm-31):
open_ctreefailed
04:54:01 doom kernel: BTRFS warning (device dm-32): devid 1 uuid f5c3dc63-1fac-45b3-b9ba-ed1ec5f92403 is missing 04:54:45 doom kernel: BTRFS warning (device dm-32): chunk 12582912 missing 1 devices, max tolerance is 0 for writable mount 04:54:45 doom kernel: BTRFS warning (device dm-32): too many missing devices, writable remount is not allowedSince (in theory :) the filesystemw a completely backed up, I didn't bother with further recovery after I made sure the physical disk is actually there and was unlocked (cryptsetup), so it wasn't a case of an actual missing disk.
BTW, "btrfs ins dump-tree -t chunk <dev>" would help a lot. That would directly tell us if the devid 1 device is in chunk tree.
If passing different <dev> would cause different output, please also provide all different versions.
Quoted 19 lines
Quoted 7 lines
Quoted 3 lines
BTW, this (second issue) also happens with filesystems that are not multi-device.
Single device btrfs doesn't need device scan. If that happened, something insane happened again... Thanks,
It happens since at least 4.14 on at least four machines, but I haven't seen it recently, after I switched to 5.2.21 one some machines (post-4.4 kernels have this habit of freezing under memory pressure, and 5.2.21 has greatly improved in this regard). That also means I had far fewer hard resets with 5.2.21, but the problem did not happen on the last resets in 5.2.21 and 5.4.5.
I originally reported it below, with some evidence that it isn't a hardware issue (no reset needed, just wipe the dm table while the device is mounted which should cleanly "cut off" the write stream):
Quoted 8 lines
https://bugzilla.kernel.org/show_bug.cgi?id=204083
Since multiple scrubs and full reads of the volumes didn't show up any issues, I didn't think much of it.
And if you want to hear more "insane" things, after I hard-reset my desktop machine (5.2.21) two days ago I had to "btrfs rescue fix-device-size" to be able to mount (can't find the kernel error atm.).
Consider all these insane things, I tend to believe there is some FUA/FLUSH related hardware problem.
E.g. the HDD/SSD controller reports FUA/FLUSH finished way before it really write data into the disk or non-volatile cache, or the non-volatile cache recovery is not implemented properly...
Thanks,
Qu
Quoted 3 lines
Greetings,
Quoted 5 lines · Qu Wenruo
BTW, that chunk number is very small, and since it has 0 tolerance, it looks like to be SINGLE chunk.
In that case, it looks like a temporary chunk from older mkfs, and it should contain no data/metadata at all, thus brings no data loss.
Well, there indeed should not have been any data or metadata left as the btrfs dev del succeeded after lengthy copying.
Quoted 2 lines
BTW, "btrfs ins dump-tree -t chunk <dev>" would help a lot. That would directly tell us if the devid 1 device is in chunk tree.
Apologies if I wasn't too clear about it - I already had to mkfs and redo the filesystem. I understand that makes tracking this down hard or impossible, but I did need that machine and filesystem.
Quoted 6 lines
Quoted 3 lines
And if you want to hear more "insane" things, after I hard-reset my desktop machine (5.2.21) two days ago I had to "btrfs rescue fix-device-size" to be able to mount (can't find the kernel error atm.).
Consider all these insane things, I tend to believe there is some FUA/FLUSH related hardware problem.
Please don't - I honestly think btrfs developers are way to fast to blame hardware for problems. I currently lose btrfs filesystems about once every 6 months, and other than the occasional user error, it's always the kernel (e.g. 4.11 corrupting data, dmcache and/or bcache corrupting things, low-memory situations etc. - none of these seem to be centric to btrfs, but none of those are hardware errors either). I know its the kernel in most cases because in those cases, I can identify the fix in a later kernel, or the mitigating circumstances don't appear (e.g. freezes).
In any case if it is a hardware problem, then linux and/or btrfs has to work around them, because it affects many different controllers on different boards:
- dell perc h740 on "doom" and "cerebro"
- intel series 9 controller on "doom'" and "cerebro".
- samsung nvme controller on "yoyo" and "yuna".
- marvell sata controller on "doom".
Just while I was writing this mail, on 5.4.5, the _newly created_ btrfs filesystem I restored to went into readonly mode with ENOSPC. Another hardware problem?
[41801.618772] ------------[ cut here ]------------ [41801.618776] BTRFS: Transaction aborted (error -28) [41801.618843] WARNING: CPU: 2 PID: 5713 at fs/btrfs/inode.c:3159 btrfs_finish_ordered_io+0x730/0x820 [btrfs] [41801.618844] Modules linked in: nfsv3 nfs fscache nvidia_modeset(POE) nvidia(POE) btusb algif_skcipher af_alg dm_crypt nfsd auth_rpcgss nfs_acl lockd grace cls_fw sch_htb sit tunnel4 ip_tunnel hidp act_police cls_u32 sch_ingress sch_tbf 8021q garp mrp stp llc ip6t_REJECT nf_reject_ipv6 xt_CT xt_MASQUERADE xt_nat xt_REDIRECT nft_chain_nat nf_nat xt_owner xt_TCPMSS xt_DSCP xt_mark nf_log_ipv4 nf_log_common xt_LOG xt_state xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipt_REJECT nf_reject_ipv4 xt_length xt_mac xt_tcpudp nft_compat nft_counter nf_tables xfrm_user xfrm_algo nfnetlink cmac uhid bnep tda10021 snd_hda_codec_hdmi binfmt_misc nls_iso8859_1 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass tda827x tda10023 crct10dif_pclmul mei_hdcp crc32_pclmul btrtl btbcm rc_tt_1500 ghash_clmulni_intel snd_emu10k1 btintel snd_util_mem snd_ac97_codec aesni_intel bluetooth snd_hda_intel budget_av snd_rawmidi snd_intel_nhlt crypto_simd saa7 146_vv [41801.618864] snd_hda_codec videobuf_dma_sg budget_ci videobuf_core snd_seq_device budget_core cryptd ttpci_eeprom glue_helper snd_hda_core saa7146 dvb_core intel_cstate ac97_bus snd_hwdep rc_core snd_pcm intel_rapl_perf mxm_wmi cdc_acm pcspkr videodev snd_timer ecdh_generic snd emu10k1_gp ecc mc gameport soundcore mei_me mei mac_hid acpi_pad tcp_bbr drm_kms_helper drm fb_sys_fops syscopyarea sysfillrect sysimgblt ipmi_devintf ipmi_msghandler hid_generic usbhid hid usbkbd coretemp nct6775 hwmon_vid sunrpc parport_pc ppdev lp parport msr ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 multipath linear dm_cache_smq dm_cache dm_persistent_data dm_bio_prison dm_bufio libcrc32c ahci megaraid_sas i2c_i801 libahci lpc_ich r8169 realtek wmi video [last unloaded: nvidia] [41801.618887] CPU: 2 PID: 5713 Comm: kworker/u8:15 Tainted: P OE 5.4.5-050405-generic #201912181630 [41801.618888] Hardware name: MSI MS-7816/Z97-G43 (MS-7816), BIOS V17.8 12/24/2014 [41801.618903] Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs] [41801.618916] RIP: 0010:btrfs_finish_ordered_io+0x730/0x820 [btrfs] [41801.618917] Code: 49 8b 46 50 f0 48 0f ba a8 40 ce 00 00 02 72 1c 8b 45 b0 83 f8 fb 0f 84 d4 00 00 00 89 c6 48 c7 c7 48 33 62 c0 e8 eb 9c 91 d5 <0f> 0b 8b 4d b0 ba 57 0c 00 00 48 c7 c6 40 67 61 c0 4c 89 f7 bb 01 [41801.618918] RSP: 0018:ffffc18b40edfd80 EFLAGS: 00010282 [41801.618921] BTRFS: error (device dm-35) in btrfs_finish_ordered_io:3159: errno=-28 No space left [41801.618922] RAX: 0000000000000000 RBX: ffff9f8b7b2e3800 RCX: 0000000000000006 [41801.618922] BTRFS info (device dm-35): forced readonly [41801.618924] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff9f8bbeb17440 [41801.618924] RBP: ffffc18b40edfdf8 R08: 00000000000005a6 R09: ffffffff979a4d90 [41801.618925] R10: ffffffff97983fa8 R11: ffffc18b40edfbe8 R12: ffff9f8ad8b4ab60 [41801.618926] R13: ffff9f867ddb53c0 R14: ffff9f8bbb0446e8 R15: 0000000000000000 [41801.618927] FS: 0000000000000000(0000) GS:ffff9f8bbeb00000(0000) knlGS:0000000000000000 [41801.618928] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41801.618929] CR2: 00007f8ab728fc30 CR3: 000000049080a002 CR4: 00000000001606e0 [41801.618930] Call Trace: [41801.618943] finish_ordered_fn+0x15/0x20 [btrfs] [41801.618957] normal_work_helper+0xbd/0x2f0 [btrfs] [41801.618959] ? __schedule+0x2eb/0x740 [41801.618973] btrfs_endio_write_helper+0x12/0x20 [btrfs] [41801.618975] process_one_work+0x1ec/0x3a0 [41801.618977] worker_thread+0x4d/0x400 [41801.618979] kthread+0x104/0x140 [41801.618980] ? process_one_work+0x3a0/0x3a0 [41801.618982] ? kthread_park+0x90/0x90 [41801.618984] ret_from_fork+0x1f/0x40 [41801.618985] ---[ end trace 35086266bf39c897 ]--- [41801.618987] BTRFS: error (device dm-35) in btrfs_finish_ordered_io:3159: errno=-28 No space left
unmount/remount seems to make it work again, and it is full (df) yet has 3TB of unallocated space left. No clue what to do now, do I have to start over restoring again?
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/xmnt-cold15 27T 23T 0 100% /cold1
Overall:
Device size: 24216.49GiB
Device allocated: 20894.89GiB
Device unallocated: 3321.60GiB
Device missing: 0.00GiB
Used: 20893.68GiB
Free (estimated): 3322.73GiB (min: 1661.93GiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 0.50GiB (used: 0.00GiB)
Data,single: Size:20839.01GiB, Used:20837.88GiB (99.99%)
/dev/mapper/xmnt-cold15 9288.01GiB
/dev/mapper/xmnt-cold12 7427.00GiB
/dev/mapper/xmnt-cold13 4124.00GiBMetadata,RAID1: Size:27.91GiB, Used:27.90GiB (99.97%)
/dev/mapper/xmnt-cold15 25.44GiB
/dev/mapper/xmnt-cold12 24.46GiB
/dev/mapper/xmnt-cold13 5.91GiBSystem,RAID1: Size:0.03GiB, Used:0.00GiB (6.69%)
/dev/mapper/xmnt-cold15 0.03GiB
/dev/mapper/xmnt-cold12 0.03GiBUnallocated:
/dev/mapper/xmnt-cold15 0.01GiB
/dev/mapper/xmnt-cold12 0.00GiB
/dev/mapper/xmnt-cold13 3321.59GiBPlease, don't always chalk it up to hardware problems - btrfs is a wonderful filesystem for many reasons, one reason I like is that it can detect corruption much earlier than other filesystems. This featire alone makes it impossible for me to go back to xfs. However, I had corruption on ext4, xfs, reiserfs over the years, but btrfs *is* simply way buggier still than those - before btrfs (and even now) I kept md5sums of all archived files (~200TB), and xfs and ext4 _do_ a much better job at not corrupting data than btrfs on the same hardware - while I get filesystem problems about every half a year with btrfs, I had (silent) corruption problems maybe once every three to four years with xfs or ext4 (and not yet on the bxoes I use currently).
Please take these issues seriously - the trend of "it's a hardware problem" will not remove the "unstable" stigma from btrfs as long as btrfs is clearly more buggy then other filesystems.
Sorry to be so blunt, but I am a bit sensitive with always being told "it's probably a hardware problem" when it clearly affects practically any server and any laptop I administrate. I believe in btrfs, and detecting corruption early is a feature to me.
I understand it can be frustrating to be confronted with hard to explain accidents, and I understand if you can't find the bug with the sparse info I gave, especially as the bug might not even be in btrfs. But keep in mind that the people who boldly/dumbly use btrfs in production and restore dozens of terabytes from backup every so and so many months are also being frustrated if they present evidence from multiple machines and get told "its probably a hardware problem".
Signature · 6 lines
The choice of a Deliantra, the free code+content MORPG
-----==- _GNU_ http://www.deliantra.net
----==-- _ generation
---==---(_)__ __ ____ __ Marc Lehmann
--==---/ / _ \/ // /\ \/ / [email protected]
-=====/_/_//_/\_,_/ /_/\_\Quoted 25 lines · Marc Lehmann
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/xmnt-cold15 27T 23T 0 100% /cold1
Overall:
Device size: 24216.49GiB
Device allocated: 20894.89GiB
Device unallocated: 3321.60GiB
Device missing: 0.00GiB
Used: 20893.68GiB
Free (estimated): 3322.73GiB (min: 1661.93GiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 0.50GiB (used: 0.00GiB)
Data,single: Size:20839.01GiB, Used:20837.88GiB (99.99%)
/dev/mapper/xmnt-cold15 9288.01GiB
/dev/mapper/xmnt-cold12 7427.00GiB
/dev/mapper/xmnt-cold13 4124.00GiB
Metadata,RAID1: Size:27.91GiB, Used:27.90GiB (99.97%)
/dev/mapper/xmnt-cold15 25.44GiB
/dev/mapper/xmnt-cold12 24.46GiB
/dev/mapper/xmnt-cold13 5.91GiBQuoted 9 lines
System,RAID1: Size:0.03GiB, Used:0.00GiB (6.69%)
/dev/mapper/xmnt-cold15 0.03GiB
/dev/mapper/xmnt-cold12 0.03GiB
Unallocated:
/dev/mapper/xmnt-cold15 0.01GiB
/dev/mapper/xmnt-cold12 0.00GiB
/dev/mapper/xmnt-cold13 3321.59GiBYou don't need hints, the problem is right here.
Your Metadata is Raid 1, (which requires minimum of 2 devices,) Your allocated metadata is full (27.90GB / 27.91 GB) and you only have 1 device left with unallocated space, so no new metadata space can be allocated until you fix that.
Quoted 1 line · Remi Gauvin
You don't need hints, the problem is right here.
Yes, I already guessed that (see my other mail). I fortunately can add two more devices. However:
Quoted 2 lines
device left with unallocated space, so no new metadata space can be allocated until you fix that.
I think it really shouldn't be up to me to second guess btrfs's not very helpful error messages "and fix things". And if I couldn't add another device, I would be pretty much fucked - btrfs balance does not allow me to move any chunks to the other device, I tried balancing 10 data chunks and 10 metadata chunks - the data chunks balanced successfully but nothing changed, and the metadata chunks instantly hit the ENOSPC problem.
Pushing "fix things" at users without giving them the ability to do so is rather poor.
So is there a legit fix for this? The tools don't allow me to rebalance the filesystem so there is more space on the drives and deleting data and writing it again doesn't seem to help - btrfs still wants to write to the nearly full disks. I could probably convert the metadata to single and back, but as long as btrfs has no way orf moving data form one disk to another, that's going to be tough. Maybe converting to single and resizing would do the trick - seriously, though, btrfs shouldn't force users to jump through such hoops.
Signature · 6 lines
The choice of a Deliantra, the free code+content MORPG
-----==- _GNU_ http://www.deliantra.net
----==-- _ generation
---==---(_)__ __ ____ __ Marc Lehmann
--==---/ / _ \/ // /\ \/ / [email protected]
-=====/_/_//_/\_,_/ /_/\_\Quoted 2 lines · Remi Gauvin
You don't need hints, the problem is right here.
Your Metadata is Raid 1, (which requires minimum of 2 devices,) Your
Guess I found another bug - three disks with >>3tb free space, but df still shows 0 available bytes. Sure I can probably work around it somehow, but no, I refuse to accept that this is supposedly a user problem - surely btrfs could create more raid1 metadata with _three disks with lots of free space_.
doom ~# df /cold1
Filesystem Size Used Avail Use% Mounted on/dev/mapper/xmnt-cold15 43T 23T 0 100% /cold1
doom ~# btrfs dev us /cold1/dev/mapper/xmnt-cold15, ID: 1
Device size: 9.09TiB
Device slack: 0.00B
Data,single: 9.07TiB
Metadata,RAID1: 25.46GiB
System,RAID1: 32.00MiB
Unallocated: 1.00MiB
/dev/mapper/xmnt-cold12, ID: 2
Device size: 7.28TiB
Device slack: 0.00B
Data,single: 7.25TiB
Metadata,RAID1: 24.46GiB
System,RAID1: 32.00MiB
Unallocated: 1.00MiB
/dev/mapper/xmnt-cold13, ID: 3
Device size: 7.28TiB
Device slack: 0.00B
Data,single: 4.03TiB Metadata,RAID1: 5.92GiB Unallocated: 3.24TiB
/dev/mapper/xmnt-cold14, ID: 4
Device size: 7.28TiB
Device slack: 0.00B
Unallocated: 7.28TiB
/dev/mapper/xmnt-cold11, ID: 5
Device size: 7.28TiB
Device slack: 0.00B
Unallocated: 7.28TiB
Signature · 6 lines
The choice of a Deliantra, the free code+content MORPG
-----==- _GNU_ http://www.deliantra.net
----==-- _ generation
---==---(_)__ __ ____ __ Marc Lehmann
--==---/ / _ \/ // /\ \/ / [email protected]
-=====/_/_//_/\_,_/ /_/\_\In general df will only ever be an approximation on btrfs filesystems since the different profiles use different amounts of space, and it does have bugs from time to time. If you untar a mail spool on the filesystem the metadata usage may shoot way up when only a small amount of additional data is needed. So on btrfs filesystems I really just ignore df, and use btrfs filesystem usage -T almost exclusively. The table format of -T does make it much more readable for an admin.
Quoted 55 lines · Marc Lehmann
Quoted 2 lines · Remi Gauvin
You don't need hints, the problem is right here.
Your Metadata is Raid 1, (which requires minimum of 2 devices,) YourGuess I found another bug - three disks with >>3tb free space, but df still shows 0 available bytes. Sure I can probably work around it somehow, but no, I refuse to accept that this is supposedly a user problem - surely btrfs could create more raid1 metadata with
_threedisks with lots of freespace_.doom ~# df /cold1
Filesystem Size Used Avail Use% Mounted on/dev/mapper/xmnt-cold1543T 23T 0 100% /cold1
doom ~# btrfs dev us /cold1/dev/mapper/xmnt-cold15, ID: 1
Device size: 9.09TiB
Device slack: 0.00B
Data,single: 9.07TiB
Metadata,RAID1: 25.46GiB
System,RAID1: 32.00MiB
Unallocated: 1.00MiB
/dev/mapper/xmnt-cold12, ID: 2
Device size: 7.28TiB
Device slack: 0.00B
Data,single: 7.25TiB
Metadata,RAID1: 24.46GiB
System,RAID1: 32.00MiB
Unallocated: 1.00MiB
/dev/mapper/xmnt-cold13, ID: 3
Device size: 7.28TiB
Device slack: 0.00BData,single: 4.03TiB Metadata,RAID1: 5.92GiB Unallocated: 3.24TiB
/dev/mapper/xmnt-cold14, ID: 4
Device size: 7.28TiB
Device slack: 0.00B
Unallocated: 7.28TiB
/dev/mapper/xmnt-cold11, ID: 5
Device size: 7.28TiB
Device slack: 0.00B
Unallocated: 7.28TiB--
The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / [email protected] -=====/_/_//_/\_,_/ /_/\_\
Quoted 8 lines · Marc Lehmann
Quoted 2 lines · Qu Wenruo
Consider all these insane things, I tend to believe there is some FUA/FLUSH related hardware problem.
Please don't - I honestly think btrfs developers are way to fast to blame hardware for problems.
That's because they have a lot of evidence of this, in a way that's only inferable with other file systems. This has long been suspected by, and demonstrated, well before Btrfs with ZFS development.
A reasonable criticism of Btrfs development is the state of the file system check repair, which still has danger warnings. But it's also a case of damned if they do, and damned if they don't provide it. It might be the best chance of recovery, so why not provide it? Conversely, the reality is that the file system is complicated enough, and the file system checker too slow, that the effort needs to be on (what I call) file system autopsy tools, to figure out why the corruption happened, and prevent that from happening. The repair is often too difficult.
Take, for example, the recent 5.2.0-5.2.14 corruption bug. That was self-reported once it was discovered and fixed, which took longer than usual, and developers apologized. What else can they do? It's not like the developers are blaming hardware for their own bugs. They have consistently taken responsibility for Btrfs bugs.
Quoted 7 lines
I currently lose btrfs filesystems about once every
6 months, and other than the occasional user error, it's always the kernel (e.g. 4.11 corrupting data, dmcache and/or bcache corrupting things, low-memory situations etc. - none of these seem to be centric to btrfs, but none of those are hardware errors either). I know its the kernel in most cases because in those cases, I can identify the fix in a later kernel, or the mitigating circumstances don't appear (e.g. freezes).
Usually Btrfs developers do mention the possibility of other software layers contributing to the problem, it's a valid observation that this possibility be stated.
However, if it's exclusively a software problem, then it should be reproducible on other systems.
Quoted 3 lines
In any case if it is a hardware problem, then linux and/or btrfs has to work around them, because it affects many different controllers on different boards:
How do you propose Btrfs work around it? In particular when there are additional software layers over which it has no control?
Have you tried disabling the (drives') write cache?
Quoted 5 lines
Just while I was writing this mail, on 5.4.5, the
_newlycreated_btrfs filesystem I restored to went into readonly mode with ENOSPC. Another hardware problem?[41801.618887] CPU: 2 PID: 5713 Comm: kworker/u8:15 Tainted: P OE 5.4.5-050405-generic #201912181630
Why is this kernel tainted? The point of pointing this out isn't to blame whatever it tainting the kernel, but to point out that identifying the cause of your problems is made a lot more difficult. I think you need to simplify the setup, a lot, in order to reduce the surface area of possible problems. Any bug hunt is made way harder when there's complication.
Quoted 29 lines
[41801.618888] Hardware name: MSI MS-7816/Z97-G43 (MS-7816), BIOS V17.8 12/24/2014 [41801.618903] Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs] [41801.618916] RIP: 0010:btrfs_finish_ordered_io+0x730/0x820 [btrfs] [41801.618917] Code: 49 8b 46 50 f0 48 0f ba a8 40 ce 00 00 02 72 1c 8b 45 b0 83 f8 fb 0f 84 d4 00 00 00 89 c6 48 c7 c7 48 33 62 c0 e8 eb 9c 91 d5 <0f> 0b 8b 4d b0 ba 57 0c 00 00 48 c7 c6 40 67 61 c0 4c 89 f7 bb 01 [41801.618918] RSP: 0018:ffffc18b40edfd80 EFLAGS: 00010282 [41801.618921] BTRFS: error (device dm-35) in btrfs_finish_ordered_io:3159: errno=-28 No space left [41801.618922] RAX: 0000000000000000 RBX: ffff9f8b7b2e3800 RCX: 0000000000000006 [41801.618922] BTRFS info (device dm-35): forced readonly [41801.618924] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff9f8bbeb17440 [41801.618924] RBP: ffffc18b40edfdf8 R08: 00000000000005a6 R09: ffffffff979a4d90 [41801.618925] R10: ffffffff97983fa8 R11: ffffc18b40edfbe8 R12: ffff9f8ad8b4ab60 [41801.618926] R13: ffff9f867ddb53c0 R14: ffff9f8bbb0446e8 R15: 0000000000000000 [41801.618927] FS: 0000000000000000(0000) GS:ffff9f8bbeb00000(0000) knlGS:0000000000000000 [41801.618928] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41801.618929] CR2: 00007f8ab728fc30 CR3: 000000049080a002 CR4: 00000000001606e0 [41801.618930] Call Trace: [41801.618943] finish_ordered_fn+0x15/0x20 [btrfs] [41801.618957] normal_work_helper+0xbd/0x2f0 [btrfs] [41801.618959] ? __schedule+0x2eb/0x740 [41801.618973] btrfs_endio_write_helper+0x12/0x20 [btrfs] [41801.618975] process_one_work+0x1ec/0x3a0 [41801.618977] worker_thread+0x4d/0x400 [41801.618979] kthread+0x104/0x140 [41801.618980] ? process_one_work+0x3a0/0x3a0 [41801.618982] ? kthread_park+0x90/0x90 [41801.618984] ret_from_fork+0x1f/0x40 [41801.618985] ---[ end trace 35086266bf39c897 ]--- [41801.618987] BTRFS: error (device dm-35) in btrfs_finish_ordered_io:3159: errno=-28 No space left
Quoted 6 lines
unmount/remount seems to make it work again, and it is full (df) yet has 3TB of unallocated space left. No clue what to do now, do I have to start over restoring again?
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/xmnt-cold1527T 23T 0 100% /cold1
Clearly a bug, possibly more than one. This problem is being discussed in other threads on df misreporting with recent kernels, and a fix is pending.
As for the ENOSPC, also clearly a bug. But not clear why or where.
Quoted 11 lines
Please, don't always chalk it up to hardware problems - btrfs is a wonderful filesystem for many reasons, one reason I like is that it can detect corruption much earlier than other filesystems. This featire alone makes it impossible for me to go back to xfs. However, I had corruption on ext4, xfs, reiserfs over the years, but btrfs *is* simply way buggier still than those - before btrfs (and even now) I kept md5sums of all archived files (~200TB), and xfs and ext4 _do_ a much better job at not corrupting data than btrfs on the same hardware - while I get filesystem problems about every half a year with btrfs, I had (silent) corruption problems maybe once every three to four years with xfs or ext4 (and not yet on the bxoes I use currently).
I can't really parse the suggestion that you are seeing md5 mismatches (indicating data changes) on Btrfs, where Btrfs doesn't produce a csum warning along with EIO on those files? Are these files nodatacow, either by mount option nodatasum or nodatacow, or using chattr +C on these files?
A mechanism explaining this anecdote isn't clear. Not even crc32c checksum collision would explain more than maybe one instance of it.
I'm curious what Zygo thinks about this.
Quoted 9 lines
Please take these issues seriously - the trend of "it's a hardware problem" will not remove the "unstable" stigma from btrfs as long as btrfs is clearly more buggy then other filesystems.
Sorry to be so blunt, but I am a bit sensitive with always being told "it's probably a hardware problem" when it clearly affects practically any server and any laptop I administrate. I believe in btrfs, and detecting corruption early is a feature to me.
The problem with the anecdotal method of arguing in favor of software bugs as the explanation? It directly goes against my own experience, also anecdote. I've had no problems that I can attribute to Btrfs. All were hardware or user sabotage. And I've had zero data loss, outside of user sabotage.
I have seen device UNC read errors, corrected automatically by Btrfs. And I have seen devices return bad data that Btrfs caught, that would otherwise have been silent corruption of either metadata or data, and this was corrected in the raid1 cases, and merely reported in the non-raid cases. And I've also seen considerable corruption reported upon SD Cards in the midst of implosion and becoming read only. But even read only, I was able to get all the data out.
But in your case, practically ever server and laptop? That's weird and unexpected. And it makes me wonder what's in common. Btrfs is much fussier than other file systems because the by far largest target for corruption, isn't file system metadata, but data. The actual payload of a file system isn't the file system. And Btrfs is the only Linux native file system that checksums data. The other file systems check only metadata, and only somewhat recently, depending on the distribution you're using.
Quoted 7 lines
I understand it can be frustrating to be confronted with hard to explain accidents, and I understand if you can't find the bug with the sparse info I gave, especially as the bug might not even be in btrfs. But keep in mind that the people who boldly/dumbly use btrfs in production and restore dozens of terabytes from backup every so and so many months are also being frustrated if they present evidence from multiple machines and get told "its probably a hardware problem".
For sure. But take the contrary case that other file systems have depended on for more than a decade: assuming the hardware is returning valid data. This is intrinsic to their design. And go back before they had metadata checksumming, and you'd see it stated on their lists that they do assume this, and if your devices return any bad data, it's not the file system's fault at all. Not even the lack of reporting any kind of problem whatsoever. How is that better?
Well indeed, not long after Btrfs was demonstrating these are actually more common problems that suspected, metadata checksumming started creeping into other file systems, finally becoming the default (a while ago on XFS, and very recently on ext4). And they are catching a lot of these same kinds of layer and hardware bugs. Hardware does not just mean the drive, it can be power supply, logic board, controller, cables, drive write caches, drive firmware, and other drive internals.
And the only way any problem can be fixed, is to understand how, when and where it happened.
--
Chris Murphy
Quoted 6 lines · Chris Murphy
Quoted 2 lines
Please don't - I honestly think btrfs developers are way to fast to blame hardware for problems.
That's because they have a lot of evidence of this, in a way that's only inferable with other file systems. This has long been suspected by, and demonstrated, well before Btrfs with ZFS development.
But they don't - when I report that on different machines I see this reproducible behaviour, what is that lot of evidence? At the least they could inquire
Quoted 4 lines
A reasonable criticism of Btrfs development is the state of the file system check repair, which still has danger warnings. But it's also a case of damned if they do, and damned if they don't provide it. It might be the best chance of recovery, so why not provide it?
Note that I have not asked for a better fsck or anything of the sort.
Quoted 3 lines
usual, and developers apologized. What else can they do? It's not like the developers are blaming hardware for their own bugs. They have consistently taken responsibility for Btrfs bugs.
That's not the reality I live in, though. Most of my bug reports on btrfs have either been completely ignored or "oh, I can't reproduce it today anymore, maybe its fixed itself".
Sure, some of my bug reports have been taken seriously as well, and btrfs has advanced considerably over the years.
I am a software developer myself, and I understand that not every bug report can be acted upon, and sometimes you need to be sceptic for other reasons then the assumed reported ones.
Quoted 3 lines
Usually Btrfs developers do mention the possibility of other software layers contributing to the problem, it's a valid observation that this possibility be stated.
That's probably why I stated it, yes.
Your mail doesn't really apply to much of what I wrote - have you really read my bug report, or is this the pre-canned response you send out for criticism? Sorry to be so blunt, but thats pretty much how your mail feels to me, as it doesn't seem to take into account what I reported.
Quoted 2 lines
However, if it's exclusively a software problem, then it should be reproducible on other systems.
Which in this case, it is.
Even hardware problems can be reproduced on other systems, when it's say, a controller problem, so reproducability of a problem does not mean it's a software bug. But likewise jumping to conclusions because it is convenient is also a non-sequitur.
Quoted 6 lines
Quoted 3 lines
In any case if it is a hardware problem, then linux and/or btrfs has to work around them, because it affects many different controllers on different boards:
How do you propose Btrfs work around it? In particular when there are additional software layers over which it has no control?
Why do I suddenly have to propose btrfs to work around it? I said if its a hardware problem with practically every current controller, then linux and/or btrfs have to work around them, otherwise it becomes useless. And if other filesystems can keep data safe when btrfs can't, then clearly btrfs can be improved to do likewise.
Quoted 1 line
Have you tried disabling the (drives') write cache?
The write caches of all drives have been off during the lifetime of the filesystem.
Nevertheless, whats your basis for asking for write caches to be turned off? Is there any evidence that drives lose their caches without being turned off? Maybe such drives exist, but I have not heard of them - have you?
The reason why the write cache is off is because the stupid lsi raid controllers I use _do_ lose data on power outages when the drive cache is on, something that 3ware controllers didn't do. Not that power outages (actual brownouts or manually induced via the power switch) are something that really happens here.
I do, however, expect that current filesystems properly flush data to disk, and outside the raid controllers, I do not disable the write cache.
Quoted 3 lines
Quoted 1 line
[41801.618887] CPU: 2 PID: 5713 Comm: kworker/u8:15 Tainted: P OE 5.4.5-050405-generic #201912181630Why is this kernel tainted?
non-free nvidia driver
Quoted 3 lines
The point of pointing this out isn't to blame whatever it tainting the kernel, but to point out that identifying the cause of your problems is made a lot more difficult.
I think you are wildly exaggerating. Are there any reports of nvidia drivers actually corrupting filesystems? I am genuinely curious.
Other than that, your claim that a tainted kernel some makes identifying the problem a lot more difficult is just taken out of the blue, isn't it?
Quoted 3 lines
I think you need to simplify the setup, a lot, in order to reduce the surface area of possible problems. Any bug hunt is made way harder when there's complication.
Well, sure, you can hide behind the kernel taint. I am sorry, in that case, I just cannot provide you with any reports anymore, if that is what you really want.
Note, however, that this is simply an excuse "oh, a raid controller, you need to simplify your setup first". "oh, a tainted driver, you need to simplify your setup first". What's next, "oh, you used a sata disk, you need to simplify the setup first to see if filesystem corruption happens without the disk frst".
Debugging real world problems is hard, and just ignoring the real world because it doesn't fit into a lab doesn't work.
Besides, I already simplified the setup for you - my dell laptop only uses certiefied genuine non-tainted in-kernel drivers and an nvme controller, and still szuffered form the open_ctree on first reboot problem once.
Quoted 6 lines
Quoted 2 lines
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/xmnt-cold1527T 23T 0 100% /cold1Clearly a bug, possibly more than one. This problem is being discussed in other threads on df misreporting with recent kernels, and a fix is pending.
As it turns out, it's not df misreporting, it's simply very bad error reporting on the side of btrfs, requiring a lot of guesswork on the side of the user, followed by "you need to fix that problem first" from the btrfs mailing list.
In any case, I am sorry I was triggered and brought this up - this last oops report wqas not meant as a request to help me solve my problem, but to show how bad the user experience really is, both with btrfs and with this list.
Seriously, when I mention I have a reproducible problem on multiple kernel versions on multiple very different computers (that I reported in may) then it is simply not appreciated to tell me its probably a hardware problem, even if, however inconceivable it might be, it possibly could be a hardware problem.
Quoted 1 line
As for the ENOSPC, also clearly a bug. But not clear why or where.
So at least it wasn't immediately obvious to you either - it took me a while to figure out the "obvious", namely that one disk with free space is not enough for raid1 metadata. The issue here is not df potentially misreporting, but the fatc that btrfs simply has no tool to do much about it in obvious ways, yet the btrfs lsit tells me I need to fix thigs first. Great advice.
Quoted 12 lines
Quoted 11 lines
Please, don't always chalk it up to hardware problems - btrfs is a wonderful filesystem for many reasons, one reason I like is that it can detect corruption much earlier than other filesystems. This featire alone makes it impossible for me to go back to xfs. However, I had corruption on ext4, xfs, reiserfs over the years, but btrfs *is* simply way buggier still than those - before btrfs (and even now) I kept md5sums of all archived files (~200TB), and xfs and ext4 _do_ a much better job at not corrupting data than btrfs on the same hardware - while I get filesystem problems about every half a year with btrfs, I had (silent) corruption problems maybe once every three to four years with xfs or ext4 (and not yet on the bxoes I use currently).
Quoted 2 lines
I can't really parse the suggestion that you are seeing md5 mismatches (indicating data changes) on Btrfs,
Me neither, where do you think I suggested that?
Quoted 2 lines
where Btrfs doesn't produce a csum warning along with EIO on those files? Are these files nodatacow,
Well, first of all, the default is a relatively weak crc checksum (fortunately, current kernels already offer more, another good ppint in favour of btrfs). second, nocow data isn't checksummed. Third, I didn't say btrfs claims good checksums on md5 mismatches, I claimed it corrupts data.
For me, btrfs telling me instantly that a file is unreadable due to a checksum error is much preferable to me having to manually checksum it to find it, something I do maybe once a year.
What I am saying is that I lose way more files and data on btrfs than on any other filesystem I used or use. And that's a fact - we cna now speculate on why that is.
I have a few data points - most of the problems I ran into have been other kernel bugs (such as the 4.11 corruption bug, various dmcache corruption bugs and so on). Some of these have been btrfs bugs that have been fixed. Some of these have been operator errors. And some are unknown, but it's easy to scratch it up to bugs in, say, dmcache, which is another big unknown. I don't report any of these issues because I have no useful data to report. (And before you ask, the write cache of the dmcache backing ssd is switche doff, although I believe the crucial ssds that I use are some of the rare devices which actually honor FUA, and have data at rest protection).
The big advantage of btrfs is that I can often mount it and make a full backup (which only stats files to see if they changed) of the disk before recreating the filesystem, and even if it fails, it usually can and does loudly tell me so. I had way worse experience with, say, reiserfs long ago, for example :)
Quoted 4 lines
A mechanism explaining this anecdote isn't clear. Not even crc32c checksum collision would explain more than maybe one instance of it.
I'm curious what Zygo thinks about this.
I think you are reading way too much extra stuff into what I wrote - I really feel you are replying to a superficial versiob fo what I actually reported, because you are used to other such reports maybe?
Quoted 2 lines
The problem with the anecdotal method of arguing in favor of software bugs as the explanation?
Is this your method? Because it is certainly not mine, or where do you see that?
I have carefully presented facts together with supporting evidence that it is unlikely to be a hardware problem. Nowhere did I exclude hardware problems, nowhere did I exclude operator errors, nowhere did I exclude errors in other parts of the kernel (the most common source for corruption problems I have encountered).
Nowhere have I claimed it must be a software problem.
Quoted 3 lines
also anecdote. I've had no problems that I can attribute to Btrfs. All were hardware or user sabotage. And I've had zero data loss, outside of user sabotage.
That's great for you. I have ~100 active disks here with >>200TB of data, the vast majority nowadays runs with btrfs. The filesystems tend to be very busy, with very different use cases, and I think as far as anecdotes go, I have far better chances of runn inginto btrfs (or other!) bugs than most other people, possibly (don't know) even you.
Quoted 1 line
I have seen device UNC read errors, corrected automatically by Btrfs.
Yeah, me too, but only on a USB stick that I knew was badly borked.
I currently see about one unreadable sector every 1-1.5 years, and in most cases, raid5 scrubbing takes care of that, or I identify the file, delete it, and move on. I had a single case of a sudden drive death in my whole 3x years career (maybe I am lucky).
Of course these are just anecdotes. But hundreds of disks give a much better statistical base then, say, a single drive in somebodies home computer.
But of course I only have as single multi-device btrfs filesystem (as an experiment), so my statistical basis here is very thin...
Quoted 2 lines
And I have seen devices return bad data that Btrfs caught, that would otherwise have been silent corruption of either metadata or data, and
Me too, me too. Wonderful feature, that.
I have also seen btrfs fail to use the mirror copy because of a broken block, even though the mirror would have been fine - quite the number of these have been fixed in recent years, did you know that? Until quite recently, btrfs only believed in the checksum, and if that was good, dup/raid1 was of no use...
Until recently, btrfs did practically nothing about corruption introduced by itself or the kernel (or bad ram for example). It's great that this changed, even though I had a few filesystems that the new stricter 5.4.x checker refused to mount.
It's painful, but clearly progress.
I really think you confuse me with some other person that mindlessly complains. I think my complaints do have substance, though, and you chide unfairly :)
Quoted 2 lines
But in your case, practically ever server and laptop? That's weird and unexpected.
Not if you have some google fu. btrfs corruption problems are extremely common, but it's often very hard to guess what caused it.
I also think you are super badly informed about btrfs (or pretend to to defend btrfs against all reason) - recent kernels report a lot of scary corruption messages and refuse mounts, with no hint as to what could be the problem (a stricter checker - your fs might work fine under previous kernels).
If btrfs declares my fs as corrupt, I consider that filesystem corruption. It took me quite a while to realise its stricter checking and not neccessarily an indication of a real problem. I quietly reformatted those affected partitions.
Seriosly, if claims that btrfs corruption is unexpected is so far fetched and unexpected you live in a parallel universe.
Just go through kernel changelogs and count the btrfs bugfixes that cold somehow lead to corruption on somebidies system - btrfs received a lotm opf bugfixes (which is good!) but it also means there certaionly were a lot of bugs in it.
And must I remind you of the raid5 issues - I never ran into these, because careful reading of the documentation clearly told me to not use it, but it cetrainly caused a lot of btrfs corruption - let's chalk that up to user errors, though.
Quoted 3 lines
And it makes me wonder what's in common. Btrfs is much fussier than other file systems because the by far largest target for corruption, isn't file system metadata, but data. The actual payload
I don'Ät think this is true. File data might offer more surface, but there are many workloads (wsome of mine included) where metadata is shuffled around a lot more than data, and there is a lot less that can go wrong with actual data - btrfs just has to copy and checksum it - while for netadata, evry complicated algorithms are in use.
Maybe actual data is the largest target, but I don't think you can substantiate that claim in this generality.
Quoted 2 lines
of a file system isn't the file system. And Btrfs is the only Linux native file system that checksums data. The other file systems check
Which is exactly why I am using it. I had a single case of a file that was silently corrupted on xfs on teh last decade, and I only had a backup of it because it was silently corrupted and the backup had a good copy, also practically proving that it was silent data corruption.
Quoted 2 lines
only metadata, and only somewhat recently, depending on the distribution you're using.
I think that is a clear case of fanboyism - ext4 has had metadata checksums for almost 8 years in the standard kernel now, and is probably the most commonly used fs. XFS has had metadata checksums in the standard kernel for more than 6 ysears. I am not sure how stable for production btrfs was when ext4 introduced these.
Sure, metadata checksums are for noobs, but let's not make other filesystems look worse than they really are.
Quoted 7 lines
For sure. But take the contrary case that other file systems have depended on for more than a decade: assuming the hardware is returning valid data. This is intrinsic to their design. And go back before they had metadata checksumming, and you'd see it stated on their lists that they do assume this, and if your devices return any bad data, it's not the file system's fault at all. Not even the lack of reporting any kind of problem whatsoever. How is that better?
Sure, but I have considerable performance data about devices returning bad data over decades, as I, remember, keep md5 sums of practically all my files and more or less regulalry compare them, and in many cases, have backups so I can even investigate what exactly was corrupted how.
I am sorry to bring you the bad news, but outside of known broken hardware (e.g. the cmd640 corruption, which I suffered from if somebody is old enough to remember those), devices returning bad data happens, but is _exceedingly_ rare. Unreadable sectors are by far more common on spinning rust, and in my experience, quite rare unless there as "an incident" (such as a headcrash).
The most common sources of data corruption is not bad hardware, especially on hardware that otherwise works fine (e.g. survives a few hours of memtest etc. and keeps file data in general), but software bugs. The, by far, most common source of data loss for me is kernel bugs, especially in recent years. The second most common source of data loss is operator error, at least for me. Having backups certainly made me careless.
Quoted 2 lines
Well indeed, not long after Btrfs was demonstrating these are actually more common problems that suspected, metadata checksumming started
Sure they are more common than suspected, but thats trivial since practically nobody expected them.
And I know these problems exist, having suffered from them. But they are still an insignificant issue.
Maybe we come from different backgrounds - practically all my data is on hardware raid5 or better, and an unreadable sector is not something my filesystem usually has to take care of. They happen. Also, hardware has become both better (e.g. checksumming on transfer) and worse (asi n, cheper and much closer to physical limits).
Yet still, disks silently returning other data is exceedingly rare (even if you include controller firmware problems - I have no doubt that lsi controller firmwares are a complete bugfest).
However, what you are seeing is "btrfs is reporting bad checksums", and you wrongly seem to ascribe all these cases to hardware, while probably many of these cases are driver bugs, mm bugs or even filesystem bugs (a checksum will also fail if a metadata block is outdated or points to the wrong data block for example, which can easiily happen if something goes wrong during tree management).
I think that is not warranted without further evidence, which you don't seem to have.
Quoted 4 lines
while ago on XFS, and very recently on ext4). And they are catching a lot of these same kinds of layer and hardware bugs. Hardware does not just mean the drive, it can be power supply, logic board, controller, cables, drive write caches, drive firmware, and other drive internals.
And it can also be software. And tghe filesystem. On what grounds do you exclude btrfs from this list, for example? It clearly had a lot of bugs, and like every complex piece of softwrae, it surely has a lot of bugs left.
Quoted 2 lines
And the only way any problem can be fixed, is to understand how, when and where it happened.
Yes, and you can't understand how if xyou simply exclude the filesystem because it prpobably was the hardware anyway and ignore the problem. "Could not reproduce this in the current kernel anymore, maye its fixed, closing bug".
a disclaimer: this mail (and your mail!) was way too long. When I can't fully participate in any (potential) further disucssion, it is because I lack the time, not for other reasons.
Greetings,
Signature · 6 lines
The choice of a Deliantra, the free code+content MORPG
-----==- _GNU_ http://www.deliantra.net
----==-- _ generation
---==---(_)__ __ ____ __ Marc Lehmann
--==---/ / _ \/ // /\ \/ / [email protected]
-=====/_/_//_/\_,_/ /_/\_\Quoted 39 lines · Chris Murphy
Quoted 8 lines · Marc Lehmann
Quoted 2 lines · Qu Wenruo
Consider all these insane things, I tend to believe there is some FUA/FLUSH related hardware problem.
Please don't - I honestly think btrfs developers are way to fast to blame hardware for problems.
That's because they have a lot of evidence of this, in a way that's only inferable with other file systems. This has long been suspected by, and demonstrated, well before Btrfs with ZFS development.
A reasonable criticism of Btrfs development is the state of the file system check repair, which still has danger warnings. But it's also a case of damned if they do, and damned if they don't provide it. It might be the best chance of recovery, so why not provide it? Conversely, the reality is that the file system is complicated enough, and the file system checker too slow, that the effort needs to be on (what I call) file system autopsy tools, to figure out why the corruption happened, and prevent that from happening. The repair is often too difficult.
Take, for example, the recent 5.2.0-5.2.14 corruption bug. That was self-reported once it was discovered and fixed, which took longer than usual, and developers apologized. What else can they do? It's not like the developers are blaming hardware for their own bugs. They have consistently taken responsibility for Btrfs bugs.
Quoted 7 lines
I currently lose btrfs filesystems about once every
6 months, and other than the occasional user error, it's always the kernel (e.g. 4.11 corrupting data, dmcache and/or bcache corrupting things, low-memory situations etc. - none of these seem to be centric to btrfs, but none of those are hardware errors either). I know its the kernel in most cases because in those cases, I can identify the fix in a later kernel, or the mitigating circumstances don't appear (e.g. freezes).
Quoted 3 lines
Usually Btrfs developers do mention the possibility of other software layers contributing to the problem, it's a valid observation that this possibility be stated.
Also note that not all btrfs developers will agree on a failure analysis. Some patience is required. Be prepared to support your bug report with working reproducers and relevant evidence, possibly many times, with fresh backtraces on each new kernel release in which the bug still appears.
Quoted 4 lines
However, if it's exclusively a software problem, then it should be reproducible on other systems.
Quoted 8 lines
Quoted 3 lines
In any case if it is a hardware problem, then linux and/or btrfs has to work around them, because it affects many different controllers on different boards:
How do you propose Btrfs work around it? In particular when there are additional software layers over which it has no control?
Have you tried disabling the (drives') write cache?
Apparently many sysadmins disable write cache proactively on all drives, instead of waiting until the drive drops some data to learn that there's a problem with the firmware. That's a reasonable tradeoff for btrfs, which already has a heavily optimized write path (most of the IO time in btrfs commit is spent _reading_ metadata).
Quoted 76 lines
Quoted 5 lines
Just while I was writing this mail, on 5.4.5, the
_newlycreated_btrfs filesystem I restored to went into readonly mode with ENOSPC. Another hardware problem?[41801.618887] CPU: 2 PID: 5713 Comm: kworker/u8:15 Tainted: P OE 5.4.5-050405-generic #201912181630Why is this kernel tainted? The point of pointing this out isn't to blame whatever it tainting the kernel, but to point out that identifying the cause of your problems is made a lot more difficult. I think you need to simplify the setup, a lot, in order to reduce the surface area of possible problems. Any bug hunt is made way harder when there's complication.
Quoted 29 lines
[41801.618888] Hardware name: MSI MS-7816/Z97-G43 (MS-7816), BIOS V17.8 12/24/2014 [41801.618903] Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs] [41801.618916] RIP: 0010:btrfs_finish_ordered_io+0x730/0x820 [btrfs] [41801.618917] Code: 49 8b 46 50 f0 48 0f ba a8 40 ce 00 00 02 72 1c 8b 45 b0 83 f8 fb 0f 84 d4 00 00 00 89 c6 48 c7 c7 48 33 62 c0 e8 eb 9c 91 d5 <0f> 0b 8b 4d b0 ba 57 0c 00 00 48 c7 c6 40 67 61 c0 4c 89 f7 bb 01 [41801.618918] RSP: 0018:ffffc18b40edfd80 EFLAGS: 00010282 [41801.618921] BTRFS: error (device dm-35) in btrfs_finish_ordered_io:3159: errno=-28 No space left [41801.618922] RAX: 0000000000000000 RBX: ffff9f8b7b2e3800 RCX: 0000000000000006 [41801.618922] BTRFS info (device dm-35): forced readonly [41801.618924] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff9f8bbeb17440 [41801.618924] RBP: ffffc18b40edfdf8 R08: 00000000000005a6 R09: ffffffff979a4d90 [41801.618925] R10: ffffffff97983fa8 R11: ffffc18b40edfbe8 R12: ffff9f8ad8b4ab60 [41801.618926] R13: ffff9f867ddb53c0 R14: ffff9f8bbb0446e8 R15: 0000000000000000 [41801.618927] FS: 0000000000000000(0000) GS:ffff9f8bbeb00000(0000) knlGS:0000000000000000 [41801.618928] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41801.618929] CR2: 00007f8ab728fc30 CR3: 000000049080a002 CR4: 00000000001606e0 [41801.618930] Call Trace: [41801.618943] finish_ordered_fn+0x15/0x20 [btrfs] [41801.618957] normal_work_helper+0xbd/0x2f0 [btrfs] [41801.618959] ? __schedule+0x2eb/0x740 [41801.618973] btrfs_endio_write_helper+0x12/0x20 [btrfs] [41801.618975] process_one_work+0x1ec/0x3a0 [41801.618977] worker_thread+0x4d/0x400 [41801.618979] kthread+0x104/0x140 [41801.618980] ? process_one_work+0x3a0/0x3a0 [41801.618982] ? kthread_park+0x90/0x90 [41801.618984] ret_from_fork+0x1f/0x40 [41801.618985] ---[ end trace 35086266bf39c897 ]--- [41801.618987] BTRFS: error (device dm-35) in btrfs_finish_ordered_io:3159: errno=-28 No space leftQuoted 6 lines
unmount/remount seems to make it work again, and it is full (df) yet has 3TB of unallocated space left. No clue what to do now, do I have to start over restoring again?
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/xmnt-cold1527T 23T 0 100% /cold1Clearly a bug, possibly more than one. This problem is being discussed in other threads on df misreporting with recent kernels, and a fix is pending.
As for the ENOSPC, also clearly a bug. But not clear why or where.
Quoted 11 lines
Please, don't always chalk it up to hardware problems - btrfs is a wonderful filesystem for many reasons, one reason I like is that it can detect corruption much earlier than other filesystems. This featire alone makes it impossible for me to go back to xfs. However, I had corruption on ext4, xfs, reiserfs over the years, but btrfs *is* simply way buggier still than those - before btrfs (and even now) I kept md5sums of all archived files (~200TB), and xfs and ext4 _do_ a much better job at not corrupting data than btrfs on the same hardware - while I get filesystem problems about every half a year with btrfs, I had (silent) corruption problems maybe once every three to four years with xfs or ext4 (and not yet on the bxoes I use currently).
I can't really parse the suggestion that you are seeing md5 mismatches (indicating data changes) on Btrfs, where Btrfs doesn't produce a csum warning along with EIO on those files? Are these files nodatacow, either by mount option nodatasum or nodatacow, or using chattr +C on these files?
Quoted 4 lines
A mechanism explaining this anecdote isn't clear. Not even crc32c checksum collision would explain more than maybe one instance of it.
I'm curious what Zygo thinks about this.
Hardware bugs and failures are certainly common, and fleetwide hardware failures do happen. They're also recognizable as hardware bugs--some specific failure modes (e.g. single-bit data value errors, parent transid verify failure after crashes) are definitely hardware and can be easily spotted with only a few lines of kernel logs. Some components of btrfs (e.g. scrubs, csum verification, raid1 corruption recovery) are very reliable detectors of hardware or firmware misbehavior (although sometimes it is not trivial to identify _which_ hardware is at fault). Some parts of btrfs (like free space management) are completely btrfs, and cannot be affected by hardware failures without destroying the entire filesystem.
On the other hand, it's not like btrfs or the Linux kernel has been bug free either, and a lot of serious but hard to detect bugs are 5-10 years old when they get fixed. All kernels before 5.1 had silent data corruption bugs for compressed data at hole boundaries. Kernels 5.1 to 5.4 have use-after-free bugs in btrfs that lead to metadata corruption (5.1), transaction aborts due to self-detected metadata corruption (5.2), and crashes (5.3 and 5.4). 5.2 also had a second metadata corruption with deadlock bug. Other parts of the kernel are hard on data as well: somewhere around 4.7 a year-old kernel memory corruption bug was found in the r8169 network driver, and 4.0, 4.19, and 5.1 all had famous block-layer bugs that would destroy any filesystem under certain conditions.
I test every upstream kernel release thoroughly before deploying to production, because every upstream Linux kernel release has thousands of bugs (btrfs is usually about 1-2% of those). I am still waiting for the very first upstream kernel release for btrfs that can run our full production stress test workload without any backported fixes and without crashing or corrupting data or metadata for 30 days. So far that goal has never been met. We upgrade kernels when a new release gets better than an old one, but the median uptime under stress is still an order of magnitude short of the 30 day mark, and our testing on 5.4.5+fixes isn't done yet.
Unfortunately, due to the nature of crashing bugs, we can only work on the most frequently occurring bug at any time, and each one has to be fixed before the next most frequently occurring bug can be discovered, making these fixes a very sequential process. Then there's the two-month lag to get patches from the mailing list into stable kernels, which is plenty of time for new regressions to appear, and we start over again with a fresh set of bugs to fix.
btrfs dev del bugs are not crashing bugs, so they are so far down my priority list that I haven't bothered to test for them, or even to report them when I find one accidentally. There are a few bugs there though, especially if you are low on metadata space (which is a likely event if you just removed an entire disk's worth of storage) or btrfs has a bug in that kernel version that just makes btrfs _think_ it is low on metadata space, and the transaction aborts during the delete. Occasionally I hit one of these in an array and work around it with a patch like this one:
fs/btrfs/volumes.c +2 -0
@@ -7350,6 +7350,8 @@ int btrfs_read_chunk_tree(struct btrfs_fs_info *fs_info)#if 0 ret = -EINVAL ; goto error; +#else +btrfs_set_super_num_devices (fs_info->super_copy, total_dev);#endif } if (btrfs_super_total_bytes(fs_info->super_copy) <
Probably not a good idea for general use, but it may solve an immediate problem if the problem is simply that the wrong number of devices is stored in the superblock.
Quoted 16 lines
Quoted 7 lines
Quoted 8 lines
Please take these issues seriously - the trend of "it's a hardware problem" will not remove the "unstable" stigma from btrfs as long as btrfs is clearly more buggy then other filesystems.
Sorry to be so blunt, but I am a bit sensitive with always being told "it's probably a hardware problem" when it clearly affects practically any server and any laptop I administrate. I believe in btrfs, and detecting corruption early is a feature to me.
Quoted 5 lines
The problem with the anecdotal method of arguing in favor of software bugs as the explanation? It directly goes against my own experience, also anecdote. I've had no problems that I can attribute to Btrfs. All were hardware or user sabotage. And I've had zero data loss, outside of user sabotage.
You are definitely not testing hard enough. ;)
At one point in 2016 there were 145 active bugs known today. About 10 of those 145 were discovered in the last few months alone (i.e. it was broken in 2016, and we only know now how broken it was then after 3 years of hindsight). https://imgur.com/a/A2sXcQB
Thankfully, many of those bugs were mostly harmless, but some were not:
I've found at least 5 distinct data or metadata corrupting bugs since 2014, and confirmed the existence of several more in regression testing.
Quoted 7 lines
I have seen device UNC read errors, corrected automatically by Btrfs. And I have seen devices return bad data that Btrfs caught, that would otherwise have been silent corruption of either metadata or data, and this was corrected in the raid1 cases, and merely reported in the non-raid cases. And I've also seen considerable corruption reported upon SD Cards in the midst of implosion and becoming read only. But even read only, I was able to get all the data out.
btrfs data recovery on raid1 from csum and UNC sector failures is excellent. I've seen no issues there since 3.18ish. I do test that from time to time with VMs and fault injection and also with real disk failures.
btrfs on raid5 (internal or external raid5 implementation), device delete, and some unfortunate degraded mode behaviors still need some work.
Quoted 8 lines
But in your case, practically ever server and laptop? That's weird and unexpected. And it makes me wonder what's in common. Btrfs is much fussier than other file systems because the by far largest target for corruption, isn't file system metadata, but data. The actual payload of a file system isn't the file system. And Btrfs is the only Linux native file system that checksums data. The other file systems check only metadata, and only somewhat recently, depending on the distribution you're using.
If the "corruption" consists of large quantities of zeros, the problem might be using the (default) noflushoncommit mount option, or using applications that don't fsync() religiously. This is correct filesystem behavior, though maybe not behavior any application developer wants.
If the corruption affects compressed data adjacent to holes, then it's a known problem fixed in 5.1 and later.
If the corruption is specifically and only parent transid verify failures after a crash, UNC sector read, or power failure, then we'd be looking for drive firmware issues or non-default kernel settings to get a fleetwide effect.
If the corruption is general metadata corruption without metadata page csum failures, then it could be host RAM failure, general kernel memory corruption (i.e. you have to look at all the other device drivers in the system), or known bugs in btrfs kernel 5.1 and later.
If the corruption is all csum failures, then there's a long list of drive issues that could cause it, or the partition could be trampled by other software (BIOSes are sometimes surprisingly bad at this).
Quoted 30 lines
Quoted 7 lines
I understand it can be frustrating to be confronted with hard to explain accidents, and I understand if you can't find the bug with the sparse info I gave, especially as the bug might not even be in btrfs. But keep in mind that the people who boldly/dumbly use btrfs in production and restore dozens of terabytes from backup every so and so many months are also being frustrated if they present evidence from multiple machines and get told "its probably a hardware problem".
For sure. But take the contrary case that other file systems have depended on for more than a decade: assuming the hardware is returning valid data. This is intrinsic to their design. And go back before they had metadata checksumming, and you'd see it stated on their lists that they do assume this, and if your devices return any bad data, it's not the file system's fault at all. Not even the lack of reporting any kind of problem whatsoever. How is that better?
Well indeed, not long after Btrfs was demonstrating these are actually more common problems that suspected, metadata checksumming started creeping into other file systems, finally becoming the default (a while ago on XFS, and very recently on ext4). And they are catching a lot of these same kinds of layer and hardware bugs. Hardware does not just mean the drive, it can be power supply, logic board, controller, cables, drive write caches, drive firmware, and other drive internals.
And the only way any problem can be fixed, is to understand how, when and where it happened.
--
Chris Murphy
Quoted 42 lines · Marc Lehmann
Quoted 5 lines · Qu Wenruo
BTW, that chunk number is very small, and since it has 0 tolerance, it looks like to be SINGLE chunk.
In that case, it looks like a temporary chunk from older mkfs, and it should contain no data/metadata at all, thus brings no data loss.
Well, there indeed should not have been any data or metadata left as the btrfs dev del succeeded after lengthy copying.
Quoted 2 lines
BTW, "btrfs ins dump-tree -t chunk <dev>" would help a lot. That would directly tell us if the devid 1 device is in chunk tree.
Apologies if I wasn't too clear about it - I already had to mkfs and redo the filesystem. I understand that makes tracking this down hard or impossible, but I did need that machine and filesystem.
Quoted 6 lines
Quoted 3 lines
And if you want to hear more "insane" things, after I hard-reset my desktop machine (5.2.21) two days ago I had to "btrfs rescue fix-device-size" to be able to mount (can't find the kernel error atm.).
Consider all these insane things, I tend to believe there is some FUA/FLUSH related hardware problem.
Please don't - I honestly think btrfs developers are way to fast to blame hardware for problems. I currently lose btrfs filesystems about once every 6 months, and other than the occasional user error, it's always the kernel (e.g. 4.11 corrupting data, dmcache and/or bcache corrupting things, low-memory situations etc. - none of these seem to be centric to btrfs, but none of those are hardware errors either). I know its the kernel in most cases because in those cases, I can identify the fix in a later kernel, or the mitigating circumstances don't appear (e.g. freezes).
In any case if it is a hardware problem, then linux and/or btrfs has to work around them, because it affects many different controllers on different boards:
- dell perc h740 on "doom" and "cerebro"
- intel series 9 controller on "doom'" and "cerebro".
- samsung nvme controller on "yoyo" and "yuna".
- marvell sata controller on "doom".
Quoted 6 lines
Just while I was writing this mail, on 5.4.5, the
_newlycreated_btrfs filesystem I restored to went into readonly mode with ENOSPC. Another hardware problem?[41801.618772] ------------[ cut here ]------------ [41801.618776] BTRFS: Transaction aborted (error -28)
According to your later replies, this bug turns out to be a problem in over-commit calculation.
It doesn't really take disk requirement into consideration, thus can't handle cases like 3 disks RAID1 with 2 full disks.
Now it acts just like we're using DUP profiles, thus causing the problem.
To Josef, any idea to fix it?
I guess we could go the complex statfs() way to do a calculation on how many bytes can really be allocated.
Or hugely reduce the over-commit threshold?
Thanks,
Qu
Quoted 99 lines
[41801.618843] WARNING: CPU: 2 PID: 5713 at fs/btrfs/inode.c:3159 btrfs_finish_ordered_io+0x730/0x820 [btrfs] [41801.618844] Modules linked in: nfsv3 nfs fscache nvidia_modeset(POE) nvidia(POE) btusb algif_skcipher af_alg dm_crypt nfsd auth_rpcgss nfs_acl lockd grace cls_fw sch_htb sit tunnel4 ip_tunnel hidp act_police cls_u32 sch_ingress sch_tbf 8021q garp mrp stp llc ip6t_REJECT nf_reject_ipv6 xt_CT xt_MASQUERADE xt_nat xt_REDIRECT nft_chain_nat nf_nat xt_owner xt_TCPMSS xt_DSCP xt_mark nf_log_ipv4 nf_log_common xt_LOG xt_state xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipt_REJECT nf_reject_ipv4 xt_length xt_mac xt_tcpudp nft_compat nft_counter nf_tables xfrm_user xfrm_algo nfnetlink cmac uhid bnep tda10021 snd_hda_codec_hdmi binfmt_misc nls_iso8859_1 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass tda827x tda10023 crct10dif_pclmul mei_hdcp crc32_pclmul btrtl btbcm rc_tt_1500 ghash_clmulni_intel snd_emu10k1 btintel snd_util_mem snd_ac97_codec aesni_intel bluetooth snd_hda_intel budget_av snd_rawmidi snd_intel_nhlt crypto_simd saa7146_vv [41801.618864] snd_hda_codec videobuf_dma_sg budget_ci videobuf_core snd_seq_device budget_core cryptd ttpci_eeprom glue_helper snd_hda_core saa7146 dvb_core intel_cstate ac97_bus snd_hwdep rc_core snd_pcm intel_rapl_perf mxm_wmi cdc_acm pcspkr videodev snd_timer ecdh_generic snd emu10k1_gp ecc mc gameport soundcore mei_me mei mac_hid acpi_pad tcp_bbr drm_kms_helper drm fb_sys_fops syscopyarea sysfillrect sysimgblt ipmi_devintf ipmi_msghandler hid_generic usbhid hid usbkbd coretemp nct6775 hwmon_vid sunrpc parport_pc ppdev lp parport msr ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 multipath linear dm_cache_smq dm_cache dm_persistent_data dm_bio_prison dm_bufio libcrc32c ahci megaraid_sas i2c_i801 libahci lpc_ich r8169 realtek wmi video [last unloaded: nvidia] [41801.618887] CPU: 2 PID: 5713 Comm: kworker/u8:15 Tainted: P OE 5.4.5-050405-generic #201912181630 [41801.618888] Hardware name: MSI MS-7816/Z97-G43 (MS-7816), BIOS V17.8 12/24/2014 [41801.618903] Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs] [41801.618916] RIP: 0010:btrfs_finish_ordered_io+0x730/0x820 [btrfs] [41801.618917] Code: 49 8b 46 50 f0 48 0f ba a8 40 ce 00 00 02 72 1c 8b 45 b0 83 f8 fb 0f 84 d4 00 00 00 89 c6 48 c7 c7 48 33 62 c0 e8 eb 9c 91 d5 <0f> 0b 8b 4d b0 ba 57 0c 00 00 48 c7 c6 40 67 61 c0 4c 89 f7 bb 01 [41801.618918] RSP: 0018:ffffc18b40edfd80 EFLAGS: 00010282 [41801.618921] BTRFS: error (device dm-35) in btrfs_finish_ordered_io:3159: errno=-28 No space left [41801.618922] RAX: 0000000000000000 RBX: ffff9f8b7b2e3800 RCX: 0000000000000006 [41801.618922] BTRFS info (device dm-35): forced readonly [41801.618924] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff9f8bbeb17440 [41801.618924] RBP: ffffc18b40edfdf8 R08: 00000000000005a6 R09: ffffffff979a4d90 [41801.618925] R10: ffffffff97983fa8 R11: ffffc18b40edfbe8 R12: ffff9f8ad8b4ab60 [41801.618926] R13: ffff9f867ddb53c0 R14: ffff9f8bbb0446e8 R15: 0000000000000000 [41801.618927] FS: 0000000000000000(0000) GS:ffff9f8bbeb00000(0000) knlGS:0000000000000000 [41801.618928] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41801.618929] CR2: 00007f8ab728fc30 CR3: 000000049080a002 CR4: 00000000001606e0 [41801.618930] Call Trace: [41801.618943] finish_ordered_fn+0x15/0x20 [btrfs] [41801.618957] normal_work_helper+0xbd/0x2f0 [btrfs] [41801.618959] ? __schedule+0x2eb/0x740 [41801.618973] btrfs_endio_write_helper+0x12/0x20 [btrfs] [41801.618975] process_one_work+0x1ec/0x3a0 [41801.618977] worker_thread+0x4d/0x400 [41801.618979] kthread+0x104/0x140 [41801.618980] ? process_one_work+0x3a0/0x3a0 [41801.618982] ? kthread_park+0x90/0x90 [41801.618984] ret_from_fork+0x1f/0x40 [41801.618985] ---[ end trace 35086266bf39c897 ]--- [41801.618987] BTRFS: error (device dm-35) in btrfs_finish_ordered_io:3159: errno=-28 No space leftunmount/remount seems to make it work again, and it is full (df) yet has 3TB of unallocated space left. No clue what to do now, do I have to start over restoring again?
Filesystem Size Used Avail Use% Mounted on /dev/mapper/xmnt-cold15 27T 23T 0 100% /cold1 Overall: Device size: 24216.49GiB Device allocated: 20894.89GiB Device unallocated: 3321.60GiB Device missing: 0.00GiB Used: 20893.68GiB Free (estimated): 3322.73GiB (min: 1661.93GiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 0.50GiB (used: 0.00GiB) Data,single: Size:20839.01GiB, Used:20837.88GiB (99.99%) /dev/mapper/xmnt-cold15 9288.01GiB /dev/mapper/xmnt-cold12 7427.00GiB /dev/mapper/xmnt-cold13 4124.00GiB Metadata,RAID1: Size:27.91GiB, Used:27.90GiB (99.97%) /dev/mapper/xmnt-cold15 25.44GiB /dev/mapper/xmnt-cold12 24.46GiB /dev/mapper/xmnt-cold13 5.91GiB System,RAID1: Size:0.03GiB, Used:0.00GiB (6.69%) /dev/mapper/xmnt-cold15 0.03GiB /dev/mapper/xmnt-cold12 0.03GiB Unallocated: /dev/mapper/xmnt-cold15 0.01GiB /dev/mapper/xmnt-cold12 0.00GiB /dev/mapper/xmnt-cold13 3321.59GiBPlease, don't always chalk it up to hardware problems - btrfs is a wonderful filesystem for many reasons, one reason I like is that it can detect corruption much earlier than other filesystems. This featire alone makes it impossible for me to go back to xfs. However, I had corruption on ext4, xfs, reiserfs over the years, but btrfs *is* simply way buggier still than those - before btrfs (and even now) I kept md5sums of all archived files (~200TB), and xfs and ext4 _do_ a much better job at not corrupting data than btrfs on the same hardware - while I get filesystem problems about every half a year with btrfs, I had (silent) corruption problems maybe once every three to four years with xfs or ext4 (and not yet on the bxoes I use currently).
Please take these issues seriously - the trend of "it's a hardware problem" will not remove the "unstable" stigma from btrfs as long as btrfs is clearly more buggy then other filesystems.
Sorry to be so blunt, but I am a bit sensitive with always being told "it's probably a hardware problem" when it clearly affects practically any server and any laptop I administrate. I believe in btrfs, and detecting corruption early is a feature to me.
I understand it can be frustrating to be confronted with hard to explain accidents, and I understand if you can't find the bug with the sparse info I gave, especially as the bug might not even be in btrfs. But keep in mind that the people who boldly/dumbly use btrfs in production and restore dozens of terabytes from backup every so and so many months are also being frustrated if they present evidence from multiple machines and get told "its probably a hardware problem".
Quoted 3 lines
Just while I was writing this mail, on 5.4.5, the
_newlycreated_btrfs filesystem I restored to went into readonly mode with ENOSPC. Another hardware problem?
btrfs check gave me a possible hint:
Checking filesystem on /dev/mapper/xmnt-cold15
UUID: 6e035cfe-5b47-406a-998f-b8ee6567abbc
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space tree
cache and super generation don't match, space cache will be invalidated [4/7] checking fs roots
[no other errors]
But mounting with clear_cache,space_cache=v2 didn't help, df still shows 0 bytes free, "btrfs f us" still shows 3tb unallocated. I'll play around with it more...
Signature · 6 lines
The choice of a Deliantra, the free code+content MORPG
-----==- _GNU_ http://www.deliantra.net
----==-- _ generation
---==---(_)__ __ ____ __ Marc Lehmann
--==---/ / _ \/ // /\ \/ / [email protected]
-=====/_/_//_/\_,_/ /_/\_\Quoted 4 lines · Marc Lehmann
Quoted 3 lines
Just while I was writing this mail, on 5.4.5, the
_newlycreated_btrfs filesystem I restored to went into readonly mode with ENOSPC. Another hardware problem?
Quoted 8 lines
btrfs check gave me a possible hint:
Checking filesystem on
/dev/mapper/xmnt-cold15
UUID: 6e035cfe-5b47-406a-998f-b8ee6567abbc
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space tree
cache and super generation don't match, space cache will be invalidated
That's common, and not a problem at all.
Btrfs will rebuild the free space tree.
Quoted 6 lines
[4/7] checking fs roots
[no other errors]But mounting with clear_cache,space_cache=v2 didn't help, df still shows 0 bytes free, "btrfs f us" still shows 3tb unallocated. I'll play around with it more...
Df reports 0 available is a bug and caused pinned down. It's btrfs_statfs() can't co-operate with latest over-commit behavior.
This happens when there are some metadata operation queued. It's completely a runtime false alert, had nothing incorrect on-disk.
I had a fix submitted for it.
https://patchwork.kernel.org/patch/11293419/
Thanks,
Qu
Quoted 1 line
Quoted 7 lines
Quoted 3 lines
Just while I was writing this mail, on 5.4.5, the
_newlycreated_btrfs filesystem I restored to went into readonly mode with ENOSPC. Another hardware problem?But mounting with clear_cache,space_cache=v2 didn't help, df still shows 0 bytes free, "btrfs f us" still shows 3tb unallocated. I'll play around with it more...
clear_cache didn't work, but btrfsck --clear-space-cache v1 and .. v2 did work:
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/xmnt-cold15 27T 23T 3.6T 87% /cold1
Which is rather insane, as I can't see how this filesystem was ever not mounted without -o space_cache=v2.
Looking at btrfs f u again...
Metadata,single: Size:1.22GiB, Used:0.00B (0.00%)
/dev/mapper/xmnt-cold13 1.22GiB
Metadata,RAID1: Size:27.92GiB, Used:27.90GiB (99.91%)
/dev/mapper/xmnt-cold15 25.46GiB
/dev/mapper/xmnt-cold12 24.46GiB
/dev/mapper/xmnt-cold13 5.92GiBSystem,RAID1: Size:32.00MiB, Used:2.16MiB (6.74%)
/dev/mapper/xmnt-cold15 32.00MiB
/dev/mapper/xmnt-cold12 32.00MiBUnallocated:
/dev/mapper/xmnt-cold15 1.00MiB
/dev/mapper/xmnt-cold12 1.00MiB
/dev/mapper/xmnt-cold13 3.24TiBDid this happen because metadata is raid1 and two of the disks were full, and for some reason, btrfsck freed up a tiny bit of space somewhere?
Signature · 6 lines
The choice of a Deliantra, the free code+content MORPG
-----==- _GNU_ http://www.deliantra.net
----==-- _ generation
---==---(_)__ __ ____ __ Marc Lehmann
--==---/ / _ \/ // /\ \/ / [email protected]
-=====/_/_//_/\_,_/ /_/\_\