btrfs check segfaults after flipping 2 Bytes
Hello,
I was trying to determine how btrfs reacts to disk errors, when I discovered, that flipping two Bytes, supposedly inside of a file can render the filesystem unusable. Here is what I did:
1. dd if=/dev/zero of=/dev/sdg2 bs=1M
2. mkfs.btrfs /dev/sdg2
3. mount /dev/sdg2 /tmp/btrfs
4. echo "hello world this is some text" > /tmp/btrfs/hello
5. umount /dev/sdg2
this should result in this image [1]
in the following steps /dev/sdg2 is altered. Basically, we want to apply s/world/wirld/g
6. dd if=/dev/sdg2 of=/tmp/sdg2 bs=1M
7. edit /tmp/sdg2 in ghex, find & replace all occurences of "world" with "wirld" (two occurences found), save as /tmp/sdg2_new
8. dd if=/tmp/sdg2_new of=/dev/sdg2 bs=1M
/dev/sdg2 now looks like this [2]
now, when I try mounting /dev/sdg2:
"""22:21 root@localhost ~#
LANG=EN mount /dev/sdg2 /tmp/btrfs/
mount: mount /dev/sdg2 on /tmp/btrfs failed: Cannot allocate memory"""
ok, so let us try btrfs check
"""22:21 root@localhost ~#
LANG=EN btrfs check /dev/sdg2
checksum verify failed on 4222976 found F22E71BD wanted CE334502 checksum verify failed on 4222976 found F22E71BD wanted CE334502 Csum didn't match
Checking filesystem on /dev/sdg2
UUID: ea2843f1-e8aa-420f-acf8-6ef13f6c6753
checking extents
Speicherzugriffsfehler (Speicherabzug geschrieben)"""
that last line reads "Segfault, memory dumped". Is this behaviour expected?
Kind regards,
Niklas
[1] tgz: https://gigamove.rz.rwth-aachen.de/d/id/R5CKikxbYTBdqs
raw: https://gigamove.rz.rwth-aachen.de/d/id/VgMXAicS7CGEjt [2] tgz: https://gigamove.rz.rwth-aachen.de/d/id/cEebNYne9ppFy4 raw: https://gigamove.rz.rwth-aachen.de/d/id/ATJa7wpfEWdfL2
What I forgot to mention:
uname -r: 3.17.0-0.rc6.git2.1.fc22.x86_64btrfs --version: Btrfs v3.16
regards,
Niklas
Am 01.10.2014 um 22:29 schrieb Niklas Fischer:
Quoted 52 lines
Hello,
I was trying to determine how btrfs reacts to disk errors, when I discovered, that flipping two Bytes, supposedly inside of a file can render the filesystem unusable. Here is what I did:
1. dd if=/dev/zero of=/dev/sdg2 bs=1M
2. mkfs.btrfs/dev/sdg2
3. mount/dev/sdg2/tmp/btrfs
4. echo "hello world this is some text" >/tmp/btrfs/hello
5. umount/dev/sdg2this should result in this image [1]
in the following steps
/dev/sdg2is altered. Basically, we want to apply s/world/wirld/g6. dd if=/dev/sdg2 of=/tmp/sdg2 bs=1M
7. edit/tmp/sdg2in ghex, find & replace all occurences of "world" with "wirld" (two occurences found), save as/tmp/sdg2_new
8. dd if=/tmp/sdg2_new of=/dev/sdg2 bs=1M
/dev/sdg2now looks like this [2]now, when I try mounting
/dev/sdg2:"""22:21 root@localhost ~#
LANG=EN mount/dev/sdg2/tmp/btrfs/
mount: mount/dev/sdg2on/tmp/btrfsfailed: Cannot allocate memory"""ok, so let us try btrfs check
"""22:21 root@localhost ~#
LANG=EN btrfs check/dev/sdg2
checksum verify failed on 4222976 found F22E71BD wanted CE334502 checksum verify failed on 4222976 found F22E71BD wanted CE334502 Csum didn't match
Checking filesystem on/dev/sdg2
UUID: ea2843f1-e8aa-420f-acf8-6ef13f6c6753
checking extents
Speicherzugriffsfehler (Speicherabzug geschrieben)"""that last line reads "Segfault, memory dumped". Is this behaviour expected?
Kind regards,
Niklas[1] tgz: https://gigamove.rz.rwth-aachen.de/d/id/R5CKikxbYTBdqs
raw: https://gigamove.rz.rwth-aachen.de/d/id/VgMXAicS7CGEjt [2] tgz: https://gigamove.rz.rwth-aachen.de/d/id/cEebNYne9ppFy4 raw: https://gigamove.rz.rwth-aachen.de/d/id/ATJa7wpfEWdfL2
Niklas Fischer posted on Wed, 01 Oct 2014 22:29:55 +0200 as excerpted:
Quoted 9 lines
I was trying to determine how btrfs reacts to disk errors, when I discovered, that flipping two Bytes, supposedly inside of a file can render the filesystem unusable. Here is what I did:
1. dd if=/dev/zero of=/dev/sdg2 bs=1M
2. mkfs.btrfs/dev/sdg2
3. mount/dev/sdg2/tmp/btrfs
4. echo "hello world this is some text" >/tmp/btrfs/hello
5. umount/dev/sdg2
Keep in mind that on btrfs, small enough files will not be written to file extents but instead will be written directly into the metadata.
That's a small enough file I guess that's what you were seeing, which would explain the two instances of the string, since on a single device btrfs, metadata is dup mode by default.
That metadata block would then fail checksum, and an attempt would be made to use the second copy, which of course would fail it the same way.
And that being the only file in the filesystem, I'd /guess/ (not being a developer myself, just a btrfs testing admin and list regular) that metadata block is still the original one, which very likely contains critical filesystem information as well, thus explaining the mount failure when the block failed checksum verify.
In theory at least, with a less synthetic test case there'd be enough more metadata on the filesystem that the affected metadata block would be further down the chain, and corrupting it wouldn't corrupt critical filesystem information as it wouldn't be in the same block.
That might explain the problem, but I don't know enough about btrfs to know how reasonable a solution would be. I guess a btrfs dev should know.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 14 lines · Duncan
Niklas Fischer posted on Wed, 01 Oct 2014 22:29:55 +0200 as excerpted:
Quoted 9 lines
I was trying to determine how btrfs reacts to disk errors, when I discovered, that flipping two Bytes, supposedly inside of a file can render the filesystem unusable. Here is what I did:
1. dd if=/dev/zero of=/dev/sdg2 bs=1M
2. mkfs.btrfs/dev/sdg2
3. mount/dev/sdg2/tmp/btrfs
4. echo "hello world this is some text" >/tmp/btrfs/hello
5. umount/dev/sdg2Keep in mind that on btrfs, small enough files will not be written to file extents but instead will be written directly into the metadata.
Quoted 6 lines
That's a small enough file I guess that's what you were seeing, which would explain the two instances of the string, since on a single device btrfs, metadata is dup mode by default.
That metadata block would then fail checksum, and an attempt would be made to use the second copy, which of course would fail it the same way.
At least a very unlikely scenario in production.
Quoted 5 lines
And that being the only file in the filesystem, I'd /guess/ (not being a developer myself, just a btrfs testing admin and list regular) that metadata block is still the original one, which very likely contains critical filesystem information as well, thus explaining the mount failure when the block failed checksum verify.
This is a possible use-case for an equivalent to ZFS's ditto blocks. An alternative strategy would be to purposefully "sparsify" early metadata blocks (this is thinking out loud - whether or not that is a viable or easy strategy is debatable).
Quoted 8 lines
In theory at least, with a less synthetic test case there'd be enough more metadata on the filesystem that the affected metadata block would be further down the chain, and corrupting it wouldn't corrupt critical filesystem information as it wouldn't be in the same block.
That might explain the problem, but I don't know enough about btrfs to know how reasonable a solution would be.
[snip]
A reasonable workaround to get the filesystem back into a usable or recoverable state might be to mount read-only and ignore checksums. That would keep the filesystem intact, though the system has no way to know whether or not the folder structures are also corrupt.
I'm not sure if there is a mount option for this use case however. The option descriptions for "nodatasum" and "nodatacow" imply that *new* checksums are not generated. In this case the checksums already exist.
Signature · 4 lines
__________ Brendan Hide http://swiftspirit.co.za/ http://www.webafrica.co.za/?AFF1E97
Brendan Hide posted on Thu, 02 Oct 2014 07:51:08 +0200 as excerpted:
Quoted 10 lines
A reasonable workaround to get the filesystem back into a usable or recoverable state might be to mount read-only and ignore checksums. That would keep the filesystem intact, though the system has no way to know whether or not the folder structures are also corrupt.
I'm not sure if there is a mount option for this use case however. The option descriptions for "nodatasum" and "nodatacow" imply that *new* checksums are not generated. In this case the checksums already exist.
From a certain viewpoint that's sort of what btrfs restore does, except
that it doesn't mount the filesystem; it simply lets you retrieve files already there without mounting, if the filesystem isn't mountable.
Also see btrfs check --init-csum-tree, which basically wipes out the csums, after which the filesystem should mount, but entirely without checksums. Of course this option kills all the csums and there's presently nothing that actually recalculates and rebuilds the csum tree (tho there's a very recent patch that I believe adds that functionality), so once they're gone, there' gone.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 5 lines · Brendan Hide
On 2014/10/02 01:31, Duncan wrote:
[snip]I'm not sure if there is a mount option for this use case however. The option descriptions for "nodatasum" and "nodatacow" imply that *new* checksums are not generated. In this case the checksums already exist.
Looks like btrfsck has a relevant option, albeit likely more destructive than absolutely necessary:
--init-csum-tree
create a new CRC tree.
^ Also, mail was sent as HTML 12 hours ago thus was never delivered. Thunderbird has been disciplined.
Signature · 4 lines
__________ Brendan Hide http://swiftspirit.co.za/ http://www.webafrica.co.za/?AFF1E97
-------- Original Message --------
Subject: btrfs check segfaults after flipping 2 Bytes
From: Niklas Fischer <[email protected]>
To: <[email protected]>
Date: 2014年10月02日 04:29
Quoted 12 lines
Hello,
I was trying to determine how btrfs reacts to disk errors, when I discovered, that flipping two Bytes, supposedly inside of a file can render the filesystem unusable. Here is what I did:
1. dd if=/dev/zero of=/dev/sdg2 bs=1M
2. mkfs.btrfs/dev/sdg2
3. mount/dev/sdg2/tmp/btrfs
4. echo "hello world this is some text" >/tmp/btrfs/hello
5. umount/dev/sdg2
Quoted 8 lines
this should result in this image [1]
in the following steps
/dev/sdg2is altered. Basically, we want to apply s/world/wirld/g6. dd if=/dev/sdg2 of=/tmp/sdg2 bs=1M
7. edit/tmp/sdg2in ghex, find & replace all occurences of "world" with "wirld" (two occurences found), save as/tmp/sdg2_new
2 problems here.
[1] csum mismatch
As already mentioned by Ducan and Brendan, the csum does not match.
What makes thing much worse, since small file's extent is inlined, the data is stored in metadata tree blocks, and the file system is almost empty so the inline extent lies in the *root* leaf of fs_tree. These two unfortunate facts makes the whole fs_tree corrupted(only one leaf, and its cusm dismatch), which cause btrfs-progs segfault.
The good news is that, the bug in btrfs-progs is already fixed by Wang's patch:
https://patchwork.kernel.org/patch/4254631/
So at least, btrfs-progs will not segfault anymore.
[2] two occurences?
So you definitely changed something you should not touch... maybe another tree root?
Anyway, almost everything in btrfs is csums, so any manual modification may cause problem.
Quoted 24 lines
8. dd if=/tmp/sdg2_new of=/dev/sdg2 bs=1M
/dev/sdg2now looks like this [2]now, when I try mounting
/dev/sdg2:"""22:21 root@localhost ~#
LANG=EN mount/dev/sdg2/tmp/btrfs/
mount: mount/dev/sdg2on/tmp/btrfsfailed: Cannot allocate memory"""ok, so let us try btrfs check
"""22:21 root@localhost ~#
LANG=EN btrfs check/dev/sdg2
checksum verify failed on 4222976 found F22E71BD wanted CE334502 checksum verify failed on 4222976 found F22E71BD wanted CE334502 Csum didn't match
Checking filesystem on/dev/sdg2
UUID: ea2843f1-e8aa-420f-acf8-6ef13f6c6753
checking extents
Speicherzugriffsfehler (Speicherabzug geschrieben)"""that last line reads "Segfault, memory dumped". Is this behaviour expected?
Quoted 7 lines
Kind regards,
Niklas[1] tgz: https://gigamove.rz.rwth-aachen.de/d/id/R5CKikxbYTBdqs
raw: https://gigamove.rz.rwth-aachen.de/d/id/VgMXAicS7CGEjt [2] tgz: https://gigamove.rz.rwth-aachen.de/d/id/cEebNYne9ppFy4 raw: https://gigamove.rz.rwth-aachen.de/d/id/ATJa7wpfEWdfL2
BTW, [2]'s tgz is in fact raw, and raw is tgz....
Thanks,
Qu
Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Qu Wenruo posted on Mon, 06 Oct 2014 10:29:53 +0800 as excerpted:
Quoted 5 lines
2 problems here.
[1] csum mismatch As already mentioned by Ducan and Brendan, the csum does not match.
Quoted 7 lines
What makes thing much worse, since small file's extent is inlined, the data is stored in metadata tree blocks,
and the file system is almost empty so the inline extent lies in the *root* leaf offs_tree.
These two unfortunate facts makes the wholefs_treecorrupted(only one leaf, and its cusm dismatch),
which cause btrfs-progs segfault.
So I nailed it. =:^)
Quoted 8 lines
The good news is that, the bug in btrfs-progs is already fixed by Wang's patch:
https://patchwork.kernel.org/patch/4254631/
So at least, btrfs-progs will not segfault anymore.[2] two occurences?
So you definitely changed something you should not touch... maybe another tree root?
1) Single device btrfs, therefore...
2) DUP metadata by default.
3) Small file, therefore...
4) Inlined in (DUP) metadata, therefore...
5) Two occurrences, each a copy of the (metadata-inlined) file in its own instance of the (duped) metadata block.
I nailed it again. =:^)
But the connection between the mentioned tree root patch and this particular bug had escaped me, so that's useful new information to me too, resolving the problem that I spotted but had no clue whether it was even fixable. Thanks. =:^)
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
-------- Original Message --------
Subject: Re: btrfs check segfaults after flipping 2 Bytes
From: Duncan <[email protected]>
To: <[email protected]>
Date: 2014年10月06日 12:10
Quoted 21 lines
Qu Wenruo posted on Mon, 06 Oct 2014 10:29:53 +0800 as excerpted:
Quoted 5 lines
2 problems here.
[1] csum mismatch As already mentioned by Ducan and Brendan, the csum does not match.
Quoted 7 lines
What makes thing much worse, since small file's extent is inlined, the data is stored in metadata tree blocks,
and the file system is almost empty so the inline extent lies in the *root* leaf offs_tree.
These two unfortunate facts makes the wholefs_treecorrupted(only one leaf, and its cusm dismatch),
which cause btrfs-progs segfault.So I nailed it. =:^)
Quoted 5 lines
The good news is that, the bug in btrfs-progs is already fixed by Wang's patch:
https://patchwork.kernel.org/patch/4254631/
So at least, btrfs-progs will not segfault anymore.
Quoted 5 lines
Quoted 3 lines
[2] two occurences?
So you definitely changed something you should not touch... maybe another tree root?1) Single device btrfs, therefore...
2) DUP metadata by default.
Oh, I forgot this, default DUP for metadata...
Thanks
Qu.
Quoted 14 lines
3) Small file, therefore...
4) Inlined in (DUP) metadata, therefore...5) Two occurrences, each a copy of the (metadata-inlined) file in its own instance of the (duped) metadata block.
I nailed it again. =:^)
But the connection between the mentioned tree root patch and this particular bug had escaped me, so that's useful new information to me too, resolving the problem that I spotted but had no clue whether it was even fixable. Thanks. =:^)