Deleted files cause btrfs-send to fail

Hi all,

Starting today I have an interesting problem: I deleted some files as part (old fcrontabs), which now persistently causes btrfs-send to fail. The error message I get is:

Aug 12 23:32:24 thetick make_backups.sh[1059]: ERROR: send ioctl failed with -2: No such file or directory
Aug 12 23:32:25 thetick make_backups.sh[1059]: ERROR: unexpected EOF in stream.

There is nothing in the dmesg output.

Since this is the root file system, I haven't gotten a copy of the actual output of "btrfs check", though I have run it from an initramfs rescue shell. The output I saw there was much like the following (taken from an Email by Roman Mamedov from 2014-12-28):

root 22730 inode 6236418 errors 2000, link count wrong unresolved ref dir 105512 index 586340 namelen 48 name [redacted].dat.bak filetype 0 errors 3, no dir item, no dir index

Only in my case, it's "root 5" and "root 4" (I think), and the file names (and other file system specifics) are of course different. I definitely saw "errors 2000" (I take it that's supposed to be an error code?).

Is this something that "btrfs check --repair" (or something else) can safely fix?

# uname -a
Linux thetick 4.1.4-gentoo #1 SMP PREEMPT Tue Aug 4 21:58:41 CEST 2015 x86_64 AMD Athlon(tm) 64 X2 Dual Core Processor 4200+ AuthenticAMD GNU/Linux
# btrfs --version
btrfs-progs v4.1.2
# btrfs filesystem show
Label: 'MARCEC_ROOT'  uuid: 0267d8b3-a074-460a-832d-5d5fd36bae64
        Total devices 1 FS bytes used 59.30GiB
        devid    1 size 107.79GiB used 74.03GiB path /dev/sda1

Label: 'MARCEC_STORAGE' uuid: 472c9290-3ff2-4096-9c47-0612d3a52cef

        Total devices 2 FS bytes used 597.75GiB
        devid    1 size 931.51GiB used 600.03GiB path /dev/sdc
        devid    2 size 931.51GiB used 600.03GiB path /dev/sdb

Label: 'MARCEC_BACKUP' uuid: f97b3cda-15e8-418b-bb9b-235391ef2a38
Total devices 1 FS bytes used 810.35GiB
devid 1 size 976.56GiB used 837.06GiB path /dev/sdd2

btrfs-progs v4.1.2

# btrfs filesystem df /
Data, single: total=70.00GiB, used=57.53GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=4.00GiB, used=1.77GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

Greetings

Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

Am Thu, 13 Aug 2015 00:34:19 +0200
schrieb Marc Joliet <[email protected]>:

[...]

[...]

Here's the actual output now, obtained via btrfs-progs 4.0.1 from an initramfs emergency shell:

checking extents
checking free space cache
checking fs roots
root 5 inode 8338813 errors 2000, link count wrong
unresolved ref dir 26699 index 50500 namelen 4 name root filetype 0 errors 3, no dir item, no dir index
root 5 inode 8338814 errors 2000, link count wrong
unresolved ref dir 26699 index 50502 namelen 6 name marcec filetype 0 errors 3, no dir item, no dir index
root 5 inode 8338815 errors 2000, link count wrong
unresolved ref dir 26699 index 50504 namelen 6 name systab filetype 0 errors 3, no dir item, no dir index
root 5 inode 8710030 errors 2000, link count wrong
unresolved ref dir 26699 index 59588 namelen 6 name marcec filetype 0 errors 3, no dir item, no dir index
root 5 inode 8710031 errors 2000, link count wrong
unresolved ref dir 26699 index 59590 namelen 4 name root filetype 0 errors 3, no dir item, no dir index
Checking filesystem on /dev/sda1
UUID: 0267d8b3-a074-460a-832d-5d5fd36bae64
found 63467610172 bytes used err is 1
total csum bytes: 59475016
total tree bytes: 1903411200
total fs tree bytes: 1691504640
total extent tree bytes: 130322432
btree space waste bytes: 442495212
file data blocks allocated: 555097092096
referenced 72887840768
btrfs-progs v4.0.1

Again: is this fixable?

Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

Marc Joliet posted on Thu, 13 Aug 2015 09:05:41 +0200 as excerpted:

FWIW, root 5 (which you asked about upthread) is the main filesystem root. So all these appear to be on the main filesystem, not on snapshots/ subvolumes.

As for the problem itself, noting that I'm not a dev, just a user/admin following the list, I believe...

There was a recent bug (early 4.0 or 4.1, IDR which) that (as I recall understanding it) would fail to decrement link count and would thus leave unnamed inodes hanging around in directories with no way to delete them. That looks very much like what you're seeing. The bug has indeed been fixed in current, and a current btrfs check should fix it, but I don't believe that v4.0.1 userspace from the initramfs is new enough to have that fix. The 4.1.2 userspace on your main system (from the first post) is current and should fix it, I believe, however.

But if it's critical, you may wish to wait and have someone else confirm that before acting on it, just in case I have it wrong.

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Am Thu, 13 Aug 2015 08:29:19 +0000 (UTC)
schrieb Duncan <[email protected]>:

OK

Now that you mention it, I think I remember seeing that patch (series?).

I have updated the initramfs in the meantime. (Funny: I *just* started using one, mainly to be able to use btrfstune on /, but now I have a genuine necessity for it.)

I can wait until tonight, at least. The FS still mounts, and it's just the root subvolume that's affected; running btrfs-send on the /home subvolume still works.

Greetings

Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

Am Thu, 13 Aug 2015 10:54:58 +0200
schrieb Marc Joliet <[email protected]>:

[...]

Well, I got impatient, and just went ahead and did it (I have backups, after all). It looks like it worked: the affected files were moved to /lost+found/, where I deleted them again, and btrfs-send works again. The output of "btrfs check" after --repair:

checking extents
checking free space cache
checking fs roots
checking csums
There are no extents for csum range 0-69632
Csum exists for 0-69632 but there is no extent record
Checking filesystem on /dev/sda1
UUID: 0267d8b3-a074-460a-832d-5d5fd36bae64
block group 274307481600 has wrong amount of free spacefailed to load free space cache for block group 274307481600 found 60980420666 bytes used err is 1
total csum bytes: 57521732
total tree bytes: 1996800000
total fs tree bytes: 1791721472
total extent tree bytes: 127942656
btree space waste bytes: 460072661
file data blocks allocated: 478650343424
referenced 73326161920
btrfs-progs v4.1.2

If I notice anything amiss, I'll report back.

(One other thing I found interesting was that "btrfs scrub" didn't care about the link count errors.)

Greetings.

Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

Marc Joliet posted on Fri, 14 Aug 2015 23:37:37 +0200 as excerpted:

A lot of people are confused about exactly what btrfs scrub does, and expect it to detect and possibly fix stuff it has nothing to do with. It's *not* an fsck.

Scrub does one very useful, but limited, thing. It systematically verifies that the computed checksums for all data and metadata covered by checksums match the corresponding recorded checksums. For dup/raid1/ raid10 modes, if there's a match failure, it will look up the other copy and see if it matches, replacing the invalid block with a new copy of the other one, assuming it's valid. For raid56 modes, it attempts to compute the valid copy from parity and, again assuming a match after doing so, does the replace. If a valid copy cannot be found or computed, either because it's damaged too or because there's no second copy or parity to fall back on (single and raid0 modes), then scrub will detect but cannot correct the error.

In routine usage, btrfs automatically does the same thing if it happens to come across checksum errors in its normal IO stream, but it has to come across them first. Scrub's benefit is that it systematically verifies (and corrects errors where it can) checksums on the entire filesystem, not just the parts that happen to appear in the normal IO stream.

Such checksum errors can be for a few reasons...

I have one ssd that's gradually failing and returns checksum errors fairly regularly. Were I using a normal filesystem I'd have had to replace it some time ago. But with btrfs in raid1 mode and regular scrubs (and backups, should they be needed; sometimes I let them get a bit stale, but I do have them and am prepared to live with the stale restored data if I have to), I've been able to keep using the failing device. When the scrubs hit errors and btrfs does the rewrite from the good copy, a block relocation on the failing device is triggered as well, with the bad block taken out of service and a new one from the set of spares all modern devices have takes its place. Currently, smartctl -A reports 904 reallocated sectors raw value, with a standardized value of
92. Before the first reallocated sector, the standardized value was 253, perfect. With the first reallocated sector, it immediately dropped to 100, apparently the rounded percentage of spare sectors left. It has gradually dropped since then to its current 92, with a threshold value of
36. So while it's gradually failing, there's still plenty of spare sectors left. Normally I would have replaced the device even so, but I've never actually had the opportunity to actually watch a slow failure continue to get worse over time, and now that I do I'm a bit curious how things will go, so I'm just letting it happen, tho I do have a replacement device already purchased and ready, when the time comes.

So real media failure, bitrot, is one reason for bad checksums. The data read back from the device simply isn't the same data that was stored to it, and the checksum fails as a result.

Of course bad connector cables or storage chipset firmware or hardware is another "hardware" cause.

Sudden reboot or power loss, with data being actively written and one copy either already updated or not yet touched, while the other is actually being written at the time of the crash so the write isn't completed, is yet another reason for checksum failure. This one is actually why a scrub can appear to do so much more than it does, because where there's a second copy (or parity) of the data available, scrub can use it to recover the partially written copy (which being partially written fails its checksum verification) to either the completed write state, if the other copy was already written, or the pre-write state, if the other copy hadn't been written at all, yet. In this way the result is often the same one an fsck would normally produce, detecting and fixing the error, but the mechanism is entirely different -- it only detected and fixed the error because the checksum was bad and it had a good copy it could replace it with, not because it had any smarts about how the filesystem actually worked, and could actually tell what the error was and correct it by actually correcting it.

Meanwhile, in your case the problem was an actual btrfs logic bug -- it didn't track the inode ref-counts correctly, and didn't remove the inode when the last reference to it was deleted, because it still thought there were more references. So the metadata actually written to storage was incorrect due to the logic flaw, but the checksum covering it was indeed the correct checksum for that metadata, as wrong as the metadata actually happened to be. So scrub couldn't detect the error, because it was an error not in checksum, which was computed correctly over the metadata, but in the logic of the metadata itself as it was written. Scrub therefore had nothing to do with that error and was in fact totally oblivious to the fact that the valid checksum covered flawed data in the first place. Only a tool that could follow the actual logic, send in this case, since it has to follow the logic in ordered to properly send it, could detect the error, and only btrfs check knew enough about the logic to both detect the problem and correct it -- tho even then, it couldn't totally fix it, as part of the metadata was irretrievably missing, so it simply dropped what it could retrieve in lost-and-found.

That should make the answer to the question of why scrub couldn't detect and fix the problem clearer -- scrub only detects and possibly fixes a very specific problem. checksum verification failure, and that's not the problem you had. As far as scrub was concerned, the checksums were fine, and that's all it knows about, so to it, the data and metadata were fine.

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Am Sat, 15 Aug 2015 05:10:57 +0000 (UTC)
schrieb Duncan <[email protected]>:

I know all that, I just thought it was interesting and wanted to remark as such. After thinking about it a bit, of course, it makes perfect sense and is not very interesting at all: scrub will just verify that the checksums match, no matter whether the underlying (meta)data is valid or not.

I'm curious how that will pan out. My experience with HDDs is that at some point the sector reallocations start picking up at a somewhat constant (maybe even accelerating) rate. I wonder how SSDs behave in this regard.

Yeah, that's a more verbose way to put it :) . Thanks anyway.

Greetings

Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

Am Fri, 14 Aug 2015 23:37:37 +0200
schrieb Marc Joliet <[email protected]>:

I haven't noticed anything amiss, so I'm marking this thread as SOLVED.

Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup