Skip to contentSkip to conversation

checksum error...

DISCUSSION Scott E. Blomquist 3 participants 4 messages Apr 8, 2019 – Apr 15, 2019 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi All,

The weekend btrfs scrub/balance came back with this following...

[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271471, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 62378, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 270894, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271453, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271449, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271435, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271432, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271421, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271411, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271401, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271391, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271345, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271253, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271163, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271074, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 270983, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Sun Apr  7 06:57:10 2019] BTRFS error (device sdb1): unable to fixup (regular) error at logical 274820497408 on dev /dev/sda1

Here is what I have...

    root@cbmm-fsb:~# uname -a
    Linux cbmm-fsb 4.14.24-custom #1 SMP Mon Mar 5 10:10:39 EST 2018 x86_64 x86_64 x86_64 GNU/Linux
    root@cbmm-fsb:~# btrfs --version
    btrfs-progs v4.15.1
    root@cbmm-fsb:~# btrfs fi show
    Label: none  uuid: d83b1e28-db27-4035-8638-d4b2eb824ff2
           Total devices 2 FS bytes used 80.09TiB
           devid    1 size 76.40TiB used 62.49TiB path /dev/sda1
           devid    2 size 32.74TiB used 18.83TiB path /dev/sdb1
    root@cbmm-fsb:~# btrfs fi df /home/cbcl
    Data, single: total=79.80TiB, used=79.80TiB
    System, RAID1: total=32.00MiB, used=9.09MiB
    Metadata, RAID1: total=757.00GiB, used=281.34GiB
    Metadata, DUP: total=22.50GiB, used=19.27GiB
    GlobalReserve, single: total=512.00MiB, used=0.00B

sda and sdb are megaraid raid6 with BBU and both are optimal.

Any tips? Thanks.

sb. Scott Blomquist

#1 lore raw
Quoted 6 lines · Scott E. Blomquist

Hi All,

The weekend btrfs scrub/balance came back with this following...

[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271471, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)

[snip]

Since there doesn't seem to be anything else wrong (no messages
without a filename, which would imply metadata corruption), this is most likely a simple case of on-device corruption.

Delete yyyy/yyyyy and restore it from backups. At least, do so in
the working copy; The snapshots of it can safely remain until they get rotated out normally.

Check your SMART statistics and see if anything looks wrong there
on the hardware side. Also check dmesg and earlier kernel logs for signs of the hardware showing an error on read -- it may have tried several times to read that location before giving up and/or returning bad data.

Hugo.

Quoted 28 lines

Here is what I have...

    root@cbmm-fsb:~# uname -a
    Linux cbmm-fsb 4.14.24-custom #1 SMP Mon Mar 5 10:10:39 EST 2018 x86_64 x86_64 x86_64 GNU/Linux
    root@cbmm-fsb:~# btrfs --version
    btrfs-progs v4.15.1
    root@cbmm-fsb:~# btrfs fi show
    Label: none  uuid: d83b1e28-db27-4035-8638-d4b2eb824ff2
           Total devices 2 FS bytes used 80.09TiB
           devid    1 size 76.40TiB used 62.49TiB path /dev/sda1
           devid    2 size 32.74TiB used 18.83TiB path /dev/sdb1
    root@cbmm-fsb:~# btrfs fi df /home/cbcl
    Data, single: total=79.80TiB, used=79.80TiB
    System, RAID1: total=32.00MiB, used=9.09MiB
    Metadata, RAID1: total=757.00GiB, used=281.34GiB
    Metadata, DUP: total=22.50GiB, used=19.27GiB
    GlobalReserve, single: total=512.00MiB, used=0.00B

sda and sdb are megaraid raid6 with BBU and both are optimal.

Any tips? Thanks.

sb. Scott Blomquist

Signature · 4 lines
Hugo Mills             | If it ain't broke, hit it again.
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |                                                  Foon
#2 lore raw
Quoted 17 lines · Hugo Mills
Quoted 6 lines · Scott E. Blomquist

Hi All,

The weekend btrfs scrub/balance came back with this following...

[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271471, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)

[snip]

Since there doesn't seem to be anything else wrong (no messages
without a filename, which would imply metadata corruption), this is most likely a simple case of on-device corruption.

Delete yyyy/yyyyy and restore it from backups. At least, do so in
the working copy; The snapshots of it can safely remain until they get rotated out normally.

Quoted 7 lines

Check your SMART statistics and see if anything looks wrong there
on the hardware side. Also check dmesg and earlier kernel logs for signs of the hardware showing an error on read -- it may have tried several times to read that location before giving up and/or returning bad data.

Hugo.

Thanks, Hugo. Very helpful.

Turns out event log from MegaCli is showing some unexpected sense in the eventlog.

Cheers,

sb. Scott Blomquist

#3 lore raw
In reply to Scott E. Blomquist
Quoted 9 lines · Scott E. Blomquist
    root@cbmm-fsb:~# btrfs fi df /home/cbcl
    Data, single: total=79.80TiB, used=79.80TiB
    System, RAID1: total=32.00MiB, used=9.09MiB
    Metadata, RAID1: total=757.00GiB, used=281.34GiB
    Metadata, DUP: total=22.50GiB, used=19.27GiB
    GlobalReserve, single: total=512.00MiB, used=0.00B

sda and sdb are megaraid raid6 with BBU and both are optimal.

You should convert the remaining "Metadata, DUP" to RAID1.

btrfs balance start -v -mconvert=raid1,soft /home/cbcl