bitflip in extent tree questions
User reports transient Btrfs corruption.
$ sudo btrfs check --readonly /dev/nvme0n1p3 Opening filesystem to check... Checking filesystem on /dev/nvme0n1p3 UUID: 73b0efc1-1bbf-44a9-b61c-f4b8936608ae [1/8] checking log skipped (none written) [2/8] checking root items [3/8] checking extents data extent[3068469248, 4096] referencer count mismatch (root 257 owner 3844574 offset 382279680) wanted 0 have 1 data extent[3068469248, 4096] bytenr mimsmatch, extent item bytenr 3068469248 file item bytenr 0 data extent[3068469248, 4096] referencer count mismatch (root 257 owner 3844574 offset 382541824) wanted 1 have 0 backpointer mismatch on [3068469248 4096] ERROR: errors found in extent allocation tree or chunk allocation [4/8] checking free space tree [5/8] checking fs roots [6/8] checking only csums items (without verifying data) [7/8] checking root refs [8/8] checking quota groups skipped (not enabled on this FS) found 851404390400 bytes used, error(s) found total csum bytes: 828557228 total tree bytes: 2640494592 total fs tree bytes: 1551499264 total extent tree bytes: 167936000 btree space waste bytes: 398250390 file data blocks allocated: 10191907336192 referenced 895043592192
Clear bitflip:
382279680 = 16C92000 = 10110110010010010000000000000
382541824 = 16CD2000 = 10110110011010010000000000000
The questions:
1. In 3/8 checking extents, do the 1st and 3rd lines tell us that the metadata profile is DUP?
2. Why does the kernel not report a checksum mismatch? What does this tell us about the nature of the bitflip?
3. Is this almost certainly logic board memory induced or is it possible it's a drive cache induced? Seems the lack of a checksum error tells us the checksum was computed after the bitflip? Therefore not drive DRAM?
What else?
Thanks!
--
Chris Murphy
4. Is it safe to run --repair?
I'm uncertain if btrfs check repair has the ability to safely fix this type of error.
Chris Murphy
Quoted 46 lines · Chris Murphy
User reports transient Btrfs corruption.
$ sudo btrfs check --readonly /dev/nvme0n1p3 Opening filesystem to check... Checking filesystem on /dev/nvme0n1p3 UUID: 73b0efc1-1bbf-44a9-b61c-f4b8936608ae [1/8] checking log skipped (none written) [2/8] checking root items [3/8] checking extents data extent[3068469248, 4096] referencer count mismatch (root 257 owner 3844574 offset 382279680) wanted 0 have 1 data extent[3068469248, 4096] bytenr mimsmatch, extent item bytenr 3068469248 file item bytenr 0 data extent[3068469248, 4096] referencer count mismatch (root 257 owner 3844574 offset 382541824) wanted 1 have 0 backpointer mismatch on [3068469248 4096] ERROR: errors found in extent allocation tree or chunk allocation [4/8] checking free space tree [5/8] checking fs roots [6/8] checking only csums items (without verifying data) [7/8] checking root refs [8/8] checking quota groups skipped (not enabled on this FS) found 851404390400 bytes used, error(s) found total csum bytes: 828557228 total tree bytes: 2640494592 total fs tree bytes: 1551499264 total extent tree bytes: 167936000 btree space waste bytes: 398250390 file data blocks allocated: 10191907336192 referenced 895043592192Clear bitflip:
382279680 = 16C92000 = 10110110010010010000000000000382541824 = 16CD2000 = 10110110011010010000000000000
The questions:
1. In 3/8 checking extents, do the 1st and 3rd lines tell us that the metadata profile is DUP?
2. Why does the kernel not report a checksum mismatch? What does this tell us about the nature of the bitflip?
3. Is this almost certainly logic board memory induced or is it possible it's a drive cache induced? Seems the lack of a checksum error tells us the checksum was computed after the bitflip? Therefore not drive DRAM?
Signature · 1 line
Chris Murphy
Quoted 1 line · Chris Murphy
User reports transient Btrfs corruption.
[...]
Quoted 4 lines
The questions:
1. In 3/8 checking extents, do the 1st and 3rd lines tell us that the metadata profile is DUP?
Nope, it's just how the extent item cross-check works.
From the backref item in the extent tree, and from the file extent in the fs tree.
Quoted 1 line
2. Why does the kernel not report a checksum mismatch? What does this tell us about the nature of the bitflip?
Bitflip happened in the RAM hardware. Btrfs calculate the csum when committing, so it has no way to know if the RAM is reliable or not.
Quoted 1 line
3. Is this almost certainly logic board memory induced or is it possible it's a drive cache induced? Seems the lack of a checksum error tells us the checksum was computed after the bitflip? Therefore not drive DRAM?
I believe it's the system DRAM.
A memtest is always recommended for DDR4.
DDR5 has some extra checks, although not as good as full ECC, it should still reduce the chance.
Quoted 1 line
4. Is it safe to run --repair?
Yes. I believe so.
Quoted 7 lines
What else?
Thanks!
--
Chris Murphy
Quoted 14 lines · Qu Wenruo
Quoted 1 line · Chris Murphy
User reports transient Btrfs corruption.
[...]
Quoted 4 lines
The questions:
1. In 3/8 checking extents, do the 1st and 3rd lines tell us that the metadata profile is DUP?
Nope, it's just how the extent item cross-check works.
From the backref item in the extent tree, and from the file extent in
the fs tree.
Quoted 8 lines
Quoted 1 line
2. Why does the kernel not report a checksum mismatch? What does this tell us about the nature of the bitflip?
Bitflip happened in the RAM hardware. Btrfs calculate the csum when committing, so it has no way to know if the RAM is reliable or not.
Quoted 1 line
3. Is this almost certainly logic board memory induced or is it possible it's a drive cache induced? Seems the lack of a checksum error tells us the checksum was computed after the bitflip? Therefore not drive DRAM?
I believe it's the system DRAM.
I've also seen system crashes where a bitflip happened in CPU registers. DRAM may still be reliable but CPU failing.
--nX
Quoted 17 lines
A memtest is always recommended for DDR4.
DDR5 has some extra checks, although not as good as full ECC, it should still reduce the chance.Quoted 1 line
4. Is it safe to run --repair?
Yes. I believe so.
Quoted 7 lines
What else?
Thanks!
--
Chris Murphy
Quoted 20 lines · Daniel Vacek
Quoted 19 lines · Qu Wenruo
Quoted 1 line · Chris Murphy
User reports transient Btrfs corruption.
[...]
Quoted 4 lines
The questions:
1. In 3/8 checking extents, do the 1st and 3rd lines tell us that the metadata profile is DUP?
Nope, it's just how the extent item cross-check works.
From the backref item in the extent tree, and from the file extent in
the fs tree.Quoted 1 line
2. Why does the kernel not report a checksum mismatch? What does this tell us about the nature of the bitflip?
Bitflip happened in the RAM hardware. Btrfs calculate the csum when committing, so it has no way to know if the RAM is reliable or not.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
3. Is this almost certainly logic board memory induced or is it possible it's a drive cache induced? Seems the lack of a checksum error tells us the checksum was computed after the bitflip? Therefore not drive DRAM?
I believe it's the system DRAM.
I've also seen system crashes where a bitflip happened in CPU registers. DRAM may still be reliable but CPU failing.
That's damn cool!
And that also explains a few cases where obvious bitflip is observed but memtest returned clean.
BTW, how did you debug such situation? Regular core dump?
Thanks,
Qu
Quoted 21 lines
--nX
Quoted 17 lines
A memtest is always recommended for DDR4.
DDR5 has some extra checks, although not as good as full ECC, it should still reduce the chance.Quoted 1 line
4. Is it safe to run --repair?
Yes. I believe so.
Quoted 7 lines
What else?
Thanks!
--
Chris Murphy
Quoted 30 lines · Qu Wenruo
Quoted 20 lines · Daniel Vacek
Quoted 19 lines · Qu Wenruo
Quoted 1 line · Chris Murphy
User reports transient Btrfs corruption.
[...]
Quoted 4 lines
The questions:
1. In 3/8 checking extents, do the 1st and 3rd lines tell us that the metadata profile is DUP?
Nope, it's just how the extent item cross-check works.
From the backref item in the extent tree, and from the file extent in
the fs tree.Quoted 1 line
2. Why does the kernel not report a checksum mismatch? What does this tell us about the nature of the bitflip?
Bitflip happened in the RAM hardware. Btrfs calculate the csum when committing, so it has no way to know if the RAM is reliable or not.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
3. Is this almost certainly logic board memory induced or is it possible it's a drive cache induced? Seems the lack of a checksum error tells us the checksum was computed after the bitflip? Therefore not drive DRAM?
I believe it's the system DRAM.
I've also seen system crashes where a bitflip happened in CPU registers. DRAM may still be reliable but CPU failing.
Quoted 6 lines
That's damn cool!
And that also explains a few cases where obvious bitflip is observed but memtest returned clean.
BTW, how did you debug such situation? Regular core dump?
Yup, vmcores from customers. And examining the data in memory you could see the correct value. Just the reg was off by 1 bit flip. Sometimes I even got multiple dumps where I could confirm it was always the same CPU core failing (matching the logical CPU or it's HT sibling).
--nX
Quoted 25 lines
Thanks,
QuQuoted 21 lines
--nX
Quoted 17 lines
A memtest is always recommended for DDR4.
DDR5 has some extra checks, although not as good as full ECC, it should still reduce the chance.Quoted 1 line
4. Is it safe to run --repair?
Yes. I believe so.
Quoted 7 lines
What else?
Thanks!
--
Chris Murphy