Skip to contentSkip to conversation

bitflip in extent tree questions

DISCUSSION Chris Murphy 4 participants 6 messages Nov 3 – Nov 4 Btrfs file system

lore
Share & tools
lore
#0 lore raw

User reports transient Btrfs corruption.

$ sudo btrfs check --readonly /dev/nvme0n1p3
Opening filesystem to check...
Checking filesystem on /dev/nvme0n1p3
UUID: 73b0efc1-1bbf-44a9-b61c-f4b8936608ae
[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
data extent[3068469248, 4096] referencer count mismatch (root 257 owner 3844574 offset 382279680) wanted 0 have 1
data extent[3068469248, 4096] bytenr mimsmatch, extent item bytenr 3068469248 file item bytenr 0
data extent[3068469248, 4096] referencer count mismatch (root 257 owner 3844574 offset 382541824) wanted 1 have 0
backpointer mismatch on [3068469248 4096]
ERROR: errors found in extent allocation tree or chunk allocation
[4/8] checking free space tree
[5/8] checking fs roots
[6/8] checking only csums items (without verifying data)
[7/8] checking root refs
[8/8] checking quota groups skipped (not enabled on this FS)
found 851404390400 bytes used, error(s) found
total csum bytes: 828557228
total tree bytes: 2640494592
total fs tree bytes: 1551499264
total extent tree bytes: 167936000
btree space waste bytes: 398250390
file data blocks allocated: 10191907336192
 referenced 895043592192

Clear bitflip:
382279680 = 16C92000 = 10110110010010010000000000000

382541824 = 16CD2000 = 10110110011010010000000000000

The questions:

1. In 3/8 checking extents, do the 1st and 3rd lines tell us that the metadata profile is DUP?
2. Why does the kernel not report a checksum mismatch? What does this tell us about the nature of the bitflip?
3. Is this almost certainly logic board memory induced or is it possible it's a drive cache induced? Seems the lack of a checksum error tells us the checksum was computed after the bitflip? Therefore not drive DRAM?

What else?

Thanks!

--
Chris Murphy

#1 lore raw

4. Is it safe to run --repair?

I'm uncertain if btrfs check repair has the ability to safely fix this type of error.

Chris Murphy

Quoted 46 lines · Chris Murphy

User reports transient Btrfs corruption.

$ sudo btrfs check --readonly /dev/nvme0n1p3
Opening filesystem to check...
Checking filesystem on /dev/nvme0n1p3
UUID: 73b0efc1-1bbf-44a9-b61c-f4b8936608ae
[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
data extent[3068469248, 4096] referencer count mismatch (root 257 owner 
3844574 offset 382279680) wanted 0 have 1
data extent[3068469248, 4096] bytenr mimsmatch, extent item bytenr 
3068469248 file item bytenr 0
data extent[3068469248, 4096] referencer count mismatch (root 257 owner 
3844574 offset 382541824) wanted 1 have 0
backpointer mismatch on [3068469248 4096]
ERROR: errors found in extent allocation tree or chunk allocation
[4/8] checking free space tree
[5/8] checking fs roots
[6/8] checking only csums items (without verifying data)
[7/8] checking root refs
[8/8] checking quota groups skipped (not enabled on this FS)
found 851404390400 bytes used, error(s) found
total csum bytes: 828557228
total tree bytes: 2640494592
total fs tree bytes: 1551499264
total extent tree bytes: 167936000
btree space waste bytes: 398250390
file data blocks allocated: 10191907336192
 referenced 895043592192

Clear bitflip:
382279680 = 16C92000 = 10110110010010010000000000000

382541824 = 16CD2000 = 10110110011010010000000000000

The questions:

1. In 3/8 checking extents, do the 1st and 3rd lines tell us that the metadata profile is DUP?
2. Why does the kernel not report a checksum mismatch? What does this tell us about the nature of the bitflip?
3. Is this almost certainly logic board memory induced or is it possible it's a drive cache induced? Seems the lack of a checksum error tells us the checksum was computed after the bitflip? Therefore not drive DRAM?

Signature · 1 line
Chris Murphy
#2 lore raw
In reply to Chris Murphy
Quoted 1 line · Chris Murphy

User reports transient Btrfs corruption.

[...]

Quoted 4 lines

The questions:

1. In 3/8 checking extents, do the 1st and 3rd lines tell us that the metadata profile is DUP?

Nope, it's just how the extent item cross-check works.
From the backref item in the extent tree, and from the file extent in the fs tree.

Quoted 1 line

2. Why does the kernel not report a checksum mismatch? What does this tell us about the nature of the bitflip?

Bitflip happened in the RAM hardware. Btrfs calculate the csum when committing, so it has no way to know if the RAM is reliable or not.

Quoted 1 line

3. Is this almost certainly logic board memory induced or is it possible it's a drive cache induced? Seems the lack of a checksum error tells us the checksum was computed after the bitflip? Therefore not drive DRAM?

I believe it's the system DRAM.

A memtest is always recommended for DDR4.
DDR5 has some extra checks, although not as good as full ECC, it should still reduce the chance.

Quoted 1 line

4. Is it safe to run --repair?

Yes. I believe so.

Quoted 7 lines

What else?

Thanks!

--
Chris Murphy

#3 lore raw
Quoted 14 lines · Qu Wenruo
Quoted 1 line · Chris Murphy

User reports transient Btrfs corruption.

[...]

Quoted 4 lines

The questions:

1. In 3/8 checking extents, do the 1st and 3rd lines tell us that the metadata profile is DUP?

Nope, it's just how the extent item cross-check works.
From the backref item in the extent tree, and from the file extent in
the fs tree.

Quoted 8 lines
Quoted 1 line

2. Why does the kernel not report a checksum mismatch? What does this tell us about the nature of the bitflip?

Bitflip happened in the RAM hardware. Btrfs calculate the csum when committing, so it has no way to know if the RAM is reliable or not.

Quoted 1 line

3. Is this almost certainly logic board memory induced or is it possible it's a drive cache induced? Seems the lack of a checksum error tells us the checksum was computed after the bitflip? Therefore not drive DRAM?

I believe it's the system DRAM.

I've also seen system crashes where a bitflip happened in CPU registers. DRAM may still be reliable but CPU failing.

--nX

Quoted 17 lines

A memtest is always recommended for DDR4.
DDR5 has some extra checks, although not as good as full ECC, it should still reduce the chance.

Quoted 1 line

4. Is it safe to run --repair?

Yes. I believe so.

Quoted 7 lines

What else?

Thanks!

--
Chris Murphy

#4 lore raw
Quoted 20 lines · Daniel Vacek
Quoted 19 lines · Qu Wenruo
Quoted 1 line · Chris Murphy

User reports transient Btrfs corruption.

[...]

Quoted 4 lines

The questions:

1. In 3/8 checking extents, do the 1st and 3rd lines tell us that the metadata profile is DUP?

Nope, it's just how the extent item cross-check works.
From the backref item in the extent tree, and from the file extent in
the fs tree.

Quoted 1 line

2. Why does the kernel not report a checksum mismatch? What does this tell us about the nature of the bitflip?

Bitflip happened in the RAM hardware. Btrfs calculate the csum when committing, so it has no way to know if the RAM is reliable or not.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

3. Is this almost certainly logic board memory induced or is it possible it's a drive cache induced? Seems the lack of a checksum error tells us the checksum was computed after the bitflip? Therefore not drive DRAM?

I believe it's the system DRAM.

I've also seen system crashes where a bitflip happened in CPU registers. DRAM may still be reliable but CPU failing.

That's damn cool!

And that also explains a few cases where obvious bitflip is observed but memtest returned clean.

BTW, how did you debug such situation? Regular core dump?

Thanks,
Qu

Quoted 21 lines

--nX

Quoted 17 lines

A memtest is always recommended for DDR4.
DDR5 has some extra checks, although not as good as full ECC, it should still reduce the chance.

Quoted 1 line

4. Is it safe to run --repair?

Yes. I believe so.

Quoted 7 lines

What else?

Thanks!

--
Chris Murphy

#5 lore raw
Quoted 30 lines · Qu Wenruo
Quoted 20 lines · Daniel Vacek
Quoted 19 lines · Qu Wenruo
Quoted 1 line · Chris Murphy

User reports transient Btrfs corruption.

[...]

Quoted 4 lines

The questions:

1. In 3/8 checking extents, do the 1st and 3rd lines tell us that the metadata profile is DUP?

Nope, it's just how the extent item cross-check works.
From the backref item in the extent tree, and from the file extent in
the fs tree.

Quoted 1 line

2. Why does the kernel not report a checksum mismatch? What does this tell us about the nature of the bitflip?

Bitflip happened in the RAM hardware. Btrfs calculate the csum when committing, so it has no way to know if the RAM is reliable or not.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

3. Is this almost certainly logic board memory induced or is it possible it's a drive cache induced? Seems the lack of a checksum error tells us the checksum was computed after the bitflip? Therefore not drive DRAM?

I believe it's the system DRAM.

I've also seen system crashes where a bitflip happened in CPU registers. DRAM may still be reliable but CPU failing.

Quoted 6 lines

That's damn cool!

And that also explains a few cases where obvious bitflip is observed but memtest returned clean.

BTW, how did you debug such situation? Regular core dump?

Yup, vmcores from customers. And examining the data in memory you could see the correct value. Just the reg was off by 1 bit flip. Sometimes I even got multiple dumps where I could confirm it was always the same CPU core failing (matching the logical CPU or it's HT sibling).

--nX

Quoted 25 lines

Thanks,
Qu

Quoted 21 lines

--nX

Quoted 17 lines

A memtest is always recommended for DDR4.
DDR5 has some extra checks, although not as good as full ECC, it should still reduce the chance.

Quoted 1 line

4. Is it safe to run --repair?

Yes. I believe so.

Quoted 7 lines

What else?

Thanks!

--
Chris Murphy