Skip to contentSkip to conversation

btrfs dev sta not updating

DISCUSSION Russell Coker 4 participants 11 messages Jun 23, 2020 – Jun 24, 2020 Btrfs file system

lore
Share & tools
lore
#0 lore raw
[395198.926320] BTRFS warning (device sdc1): csum failed root 5 ino 276 off 
19267584 csum 0x8941f998 expected csum 0xccd545e0 mirror 1
[395199.147439] BTRFS warning (device sdc1): csum failed root 5 ino 276 off 
20611072 csum 0x8941f998 expected csum 0xdaf657cb mirror 1
[395199.183680] BTRFS warning (device sdc1): csum failed root 5 ino 276 off 
24190976 csum 0x8941f998 expected csum 0xcddce0b1 mirror 1
[395199.185172] BTRFS warning (device sdc1): csum failed root 5 ino 276 off 
19267584 csum 0x8941f998 expected csum 0xccd545e0 mirror 1
[395199.330841] BTRFS warning (device sdc1): csum failed root 5 ino 277 off 0 

csum 0x8941f998 expected csum 0xa54d865c mirror 1

I have a USB stick that's corrupted, I get the above kernel messages when I try to copy files from it. But according to btrfs dev sta it has had 0 read and 0 corruption errors.

root@xev:/mnt/tmp# btrfs dev sta .
[/dev/sdc1].write_io_errs    0
[/dev/sdc1].read_io_errs     0
[/dev/sdc1].flush_io_errs    0
[/dev/sdc1].corruption_errs  0
[/dev/sdc1].generation_errs  0
root@xev:/mnt/tmp# uname -a
Linux xev 5.6.0-2-amd64 #1 SMP Debian 5.6.14-1 (2020-05-23) x86_64 GNU/Linux
Signature · 2 lines
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/
#1 lore raw
Quoted 15 lines · Russell Coker
[395198.926320] BTRFS warning (device sdc1): csum failed root 5 ino 276 off 
19267584 csum 0x8941f998 expected csum 0xccd545e0 mirror 1
[395199.147439] BTRFS warning (device sdc1): csum failed root 5 ino 276 off 
20611072 csum 0x8941f998 expected csum 0xdaf657cb mirror 1
[395199.183680] BTRFS warning (device sdc1): csum failed root 5 ino 276 off 
24190976 csum 0x8941f998 expected csum 0xcddce0b1 mirror 1
[395199.185172] BTRFS warning (device sdc1): csum failed root 5 ino 276 off 
19267584 csum 0x8941f998 expected csum 0xccd545e0 mirror 1
[395199.330841] BTRFS warning (device sdc1): csum failed root 5 ino 277 off 0 

csum 0x8941f998 expected csum 0xa54d865c mirror 1

I have a USB stick that's corrupted, I get the above kernel messages when I try to copy files from it. But according to btrfs dev sta it has had 0 read and 0 corruption errors.

Quoted 9 lines
root@xev:/mnt/tmp# btrfs dev sta .
[/dev/sdc1].write_io_errs    0
[/dev/sdc1].read_io_errs     0
[/dev/sdc1].flush_io_errs    0
[/dev/sdc1].corruption_errs  0
[/dev/sdc1].generation_errs  0
root@xev:/mnt/tmp# uname -a
Linux xev 5.6.0-2-amd64 #1 SMP Debian 5.6.14-1 (2020-05-23) x86_64 GNU/Linux

The read/write io err counters are updated when even repair bio have failed. So in your case you had some checksum errors, but btrfs managed to repair them by reading from a different mirror. In this case those aren't really counted as io errs since in the end you did get the correct data.

#2 lore raw
Quoted 28 lines · Nikolay Borisov
Quoted 15 lines · Russell Coker
[395198.926320] BTRFS warning (device sdc1): csum failed root 5 ino 276 off
19267584 csum 0x8941f998 expected csum 0xccd545e0 mirror 1
[395199.147439] BTRFS warning (device sdc1): csum failed root 5 ino 276 off
20611072 csum 0x8941f998 expected csum 0xdaf657cb mirror 1
[395199.183680] BTRFS warning (device sdc1): csum failed root 5 ino 276 off
24190976 csum 0x8941f998 expected csum 0xcddce0b1 mirror 1
[395199.185172] BTRFS warning (device sdc1): csum failed root 5 ino 276 off
19267584 csum 0x8941f998 expected csum 0xccd545e0 mirror 1
[395199.330841] BTRFS warning (device sdc1): csum failed root 5 ino 277 off 0

csum 0x8941f998 expected csum 0xa54d865c mirror 1

I have a USB stick that's corrupted, I get the above kernel messages when I try to copy files from it. But according to btrfs dev sta it has had 0 read and 0 corruption errors.

Quoted 9 lines
root@xev:/mnt/tmp# btrfs dev sta .
[/dev/sdc1].write_io_errs    0
[/dev/sdc1].read_io_errs     0
[/dev/sdc1].flush_io_errs    0
[/dev/sdc1].corruption_errs  0
[/dev/sdc1].generation_errs  0
root@xev:/mnt/tmp# uname -a
Linux xev 5.6.0-2-amd64 #1 SMP Debian 5.6.14-1 (2020-05-23) x86_64 GNU/Linux
Quoted 6 lines

The read/write io err counters are updated when even repair bio have failed. So in your case you had some checksum errors, but btrfs managed to repair them by reading from a different mirror. In this case those aren't really counted as io errs since in the end you did get the correct data.

I don't think this is what most people expect.
A simple way to solve this could be to put the non-fatal errors in parentheses if this can be done easily.

For example:
[/dev/sdc1].write_io_errs 0 (5)

IMHO this would be more readable and more useful.

#3 lore raw
Quoted 45 lines · waxhead
Quoted 36 lines · Nikolay Borisov
Quoted 22 lines · Russell Coker
[395198.926320] BTRFS warning (device sdc1): csum failed root 5 ino
276 off
19267584 csum 0x8941f998 expected csum 0xccd545e0 mirror 1
[395199.147439] BTRFS warning (device sdc1): csum failed root 5 ino
276 off
20611072 csum 0x8941f998 expected csum 0xdaf657cb mirror 1
[395199.183680] BTRFS warning (device sdc1): csum failed root 5 ino
276 off
24190976 csum 0x8941f998 expected csum 0xcddce0b1 mirror 1
[395199.185172] BTRFS warning (device sdc1): csum failed root 5 ino
276 off
19267584 csum 0x8941f998 expected csum 0xccd545e0 mirror 1
[395199.330841] BTRFS warning (device sdc1): csum failed root 5 ino

277 off 0
csum 0x8941f998 expected csum 0xa54d865c mirror 1

I have a USB stick that's corrupted, I get the above kernel messages when I
try to copy files from it.  But according to btrfs dev sta it has had 0 read
and 0 corruption errors.

Quoted 10 lines
root@xev:/mnt/tmp# btrfs dev sta .
[/dev/sdc1].write_io_errs    0
[/dev/sdc1].read_io_errs     0
[/dev/sdc1].flush_io_errs    0
[/dev/sdc1].corruption_errs  0
[/dev/sdc1].generation_errs  0
root@xev:/mnt/tmp# uname -a
Linux xev 5.6.0-2-amd64 #1 SMP Debian 5.6.14-1 (2020-05-23) x86_64
GNU/Linux
Quoted 6 lines

The read/write io err counters are updated when even repair bio have failed. So in your case you had some checksum errors, but btrfs managed to repair them by reading from a different mirror. In this case those aren't really counted as io errs since in the end you did get the correct data.

Quoted 8 lines

I don't think this is what most people expect.
A simple way to solve this could be to put the non-fatal errors in parentheses if this can be done easily.

For example:
[/dev/sdc1].write_io_errs    0 (5)

IMHO this would be more readable and more useful.

Frankly just by looking at this example output, without having read any accompanying documentation it would be hard to deduce what's the difference between the numbers. Furthermore, those error numbers are persisted on disk, so if we want to add new persistent error numbers the disk format would have to be changed. On the other hand we *could* make even transient errors be counted as persistent ones e.g. in read_io_errs. But this leads to a different can of worms - if a user sees read_io_errs should they be worried because potentially some data is stale or not (give we won't be distinguishing between unrepairable vs transient ones).

Weighing pros and cons of adding "transient" errors I'd say the effort would be better invested if instead we clearly document how errors are counted, admittedly that's a department we are severely lacking in!

#4 lore raw
In reply to Nikolay Borisov
Quoted 14 lines · AEST Nikolay Borisov
Quoted 5 lines

I have a USB stick that's corrupted, I get the above kernel messages when I
try to copy files from it. But according to btrfs dev sta it has had 0 read and 0 corruption errors.

Quoted 9 lines
root@xev:/mnt/tmp# btrfs dev sta .
[/dev/sdc1].write_io_errs    0
[/dev/sdc1].read_io_errs     0
[/dev/sdc1].flush_io_errs    0
[/dev/sdc1].corruption_errs  0
[/dev/sdc1].generation_errs  0
root@xev:/mnt/tmp# uname -a
Linux xev 5.6.0-2-amd64 #1 SMP Debian 5.6.14-1 (2020-05-23) x86_64
GNU/Linux
Quoted 5 lines

The read/write io err counters are updated when even repair bio have failed. So in your case you had some checksum errors, but btrfs managed to repair them by reading from a different mirror. In this case those aren't really counted as io errs since in the end you did get the correct data.

In this case I'm getting application IO errors and lost data, so if the error count is designed to not count recovered errors then it's still not doing the right thing.

# md5sum *
md5sum: 'Rise of the Machines S1 Ep6 - Mega Digger-qcOpMtIWsrgN.mp4': Input/
output error
md5sum: 'Rise of the Machines S1 Ep7 - Ultimate Dragster-Ke9hMplfQAWF.mp4': 
Input/output error
md5sum: 'Rise of the Machines S1 Ep8 - Aircraft Carrier-Qxht6qMEwMKr.mp4': 
Input/output error
^C
# btrfs dev sta .
[/dev/sdc1].write_io_errs    0
[/dev/sdc1].read_io_errs     0
[/dev/sdc1].flush_io_errs    0
[/dev/sdc1].corruption_errs  0
[/dev/sdc1].generation_errs  0
# tail /var/log/kern.log
Jun 23 17:48:40 xev kernel: [417603.547748] BTRFS warning (device sdc1): csum 
failed root 5 ino 275 off 59580416 csum 0x8941f998 expected csum 0xb5b581fc 
mirror 1
Jun 23 17:48:40 xev kernel: [417603.609861] BTRFS warning (device sdc1): csum 
failed root 5 ino 275 off 60628992 csum 0x8941f998 expected csum 0x4b6c9883 
mirror 1
Jun 23 17:48:40 xev kernel: [417603.672251] BTRFS warning (device sdc1): csum 
failed root 5 ino 275 off 61677568 csum 0x8941f998 expected csum 0x89f5fb68 
mirror 1
# uname -a
Linux xev 5.6.0-2-amd64 #1 SMP Debian 5.6.14-1 (2020-05-23) x86_64 GNU/Linux
Quoted 3 lines · AEST waxhead

I don't think this is what most people expect.
A simple way to solve this could be to put the non-fatal errors in parentheses if this can be done easily.

I think that most people would expect a "device stats" command to just give stats of the device and not refer to what happens at the higher level. If a device is giving corruption or read errors then "device stats" should tell that.

Quoted 4 lines · AEST Nikolay Borisov

read_io_errs. But this leads to a different can of worms - if a user sees read_io_errs should they be worried because potentially some data is stale or not (give we won't be distinguishing between unrepairable vs transient ones).

If a user sees errors reported their degree of worry should be based on the degree to which they use RAID and have decent backups. If you have RAID-1 and only 1 device has errors then you are OK. If you have 2 devices with errors then you have a problem.

Below is an example of a zpool having errors that were corrected. The DEVICE had an unrecoverable error, but the RAID-Z pool recovered it from other devices. It states that "Applications are unaffected" so the user knows the degree of worry that should be involved.

# zpool status
  pool: pet630
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: scrub repaired 380K in 156h39m with 0 errors on Sat Jun 20 13:03:26 
2020
config:
        NAME           STATE     READ WRITE CKSUM
        pet630         ONLINE       0     0     0
          raidz1-0     ONLINE       0     0     0
            sdf        ONLINE       0     0     0
            sdq        ONLINE       0     0     0
            sdd        ONLINE       0     0     0
            sdh        ONLINE       0     0     0
            sdi        ONLINE      41     0     1
Signature · 2 lines
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/
#5 lore raw
Quoted 21 lines · Russell Coker
Quoted 14 lines · AEST Nikolay Borisov
Quoted 5 lines

I have a USB stick that's corrupted, I get the above kernel messages when I
try to copy files from it. But according to btrfs dev sta it has had 0 read and 0 corruption errors.

Quoted 9 lines
root@xev:/mnt/tmp# btrfs dev sta .
[/dev/sdc1].write_io_errs    0
[/dev/sdc1].read_io_errs     0
[/dev/sdc1].flush_io_errs    0
[/dev/sdc1].corruption_errs  0
[/dev/sdc1].generation_errs  0
root@xev:/mnt/tmp# uname -a
Linux xev 5.6.0-2-amd64 #1 SMP Debian 5.6.14-1 (2020-05-23) x86_64
GNU/Linux
Quoted 5 lines

The read/write io err counters are updated when even repair bio have failed. So in your case you had some checksum errors, but btrfs managed to repair them by reading from a different mirror. In this case those aren't really counted as io errs since in the end you did get the correct data.

Quoted 3 lines

In this case I'm getting application IO errors and lost data, so if the error count is designed to not count recovered errors then it's still not doing the right thing.

In this case yes, however this was utterly not clear from your initial email. In fact it seems you have omitted quite a lot of information. So let's step back and start afresh. So first give information about your current btrfs setup by giving the output of:

btrfs fi usage /path/to/btrfs

From the output provided it seems the affected mirror is '1', which to me implies you have at least another disk containing the same data. So unless you have errors in mirror 0 as well those errors should be recovered from by simply reading from that mirror.

Quoted 9 lines
# md5sum *
md5sum: 'Rise of the Machines S1 Ep6 - Mega Digger-qcOpMtIWsrgN.mp4': Input/
output error
md5sum: 'Rise of the Machines S1 Ep7 - Ultimate Dragster-Ke9hMplfQAWF.mp4': 
Input/output error
md5sum: 'Rise of the Machines S1 Ep8 - Aircraft Carrier-Qxht6qMEwMKr.mp4': 
Input/output error
^C

You are trying to md5sum 3 distinct files....

Quoted 8 lines
# btrfs dev sta .
[/dev/sdc1].write_io_errs    0
[/dev/sdc1].read_io_errs     0
[/dev/sdc1].flush_io_errs    0
[/dev/sdc1].corruption_errs  0
[/dev/sdc1].generation_errs  0
# tail /var/log/kern.log
Jun 23 17:48:40 xev kernel: [417603.547748] BTRFS warning (device sdc1): csum 
Quoted 8 lines
failed root 5 ino 275 off 59580416 csum 0x8941f998 expected csum 0xb5b581fc 
mirror 1
Jun 23 17:48:40 xev kernel: [417603.609861] BTRFS warning (device sdc1): csum 
failed root 5 ino 275 off 60628992 csum 0x8941f998 expected csum 0x4b6c9883 
mirror 1
Jun 23 17:48:40 xev kernel: [417603.672251] BTRFS warning (device sdc1): csum 
failed root 5 ino 275 off 61677568 csum 0x8941f998 expected csum 0x89f5fb68 
mirror 1

Yet here all the errors happen in one inode, namely 275. So the md5sum commands do not correspond to those errors specifically. Also provide the name of inode 275. And for good measure also provide the output of "btrfs check /dev/sdc1" - this is a read only command so if there is some metadata corruption it will at least not make it any worse.

Quoted 4 lines
# uname -a
Linux xev 5.6.0-2-amd64 #1 SMP Debian 5.6.14-1 (2020-05-23) x86_64 GNU/Linux

On Tuesday, 23 June 2020 4:17:55 PM AEST waxhead wrote:

Quoted 8 lines
Quoted 3 lines

I don't think this is what most people expect.
A simple way to solve this could be to put the non-fatal errors in parentheses if this can be done easily.

I think that most people would expect a "device stats" command to just give stats of the device and not refer to what happens at the higher level. If a device is giving corruption or read errors then "device stats" should tell that.

That's a fair point.

Quoted 12 lines · AEST Nikolay Borisov
Quoted 4 lines

read_io_errs. But this leads to a different can of worms - if a user sees read_io_errs should they be worried because potentially some data is stale or not (give we won't be distinguishing between unrepairable vs transient ones).

If a user sees errors reported their degree of worry should be based on the degree to which they use RAID and have decent backups. If you have RAID-1 and only 1 device has errors then you are OK. If you have 2 devices with errors then you have a problem.

Quoted 4 lines

Below is an example of a zpool having errors that were corrected. The DEVICE had an unrecoverable error, but the RAID-Z pool recovered it from other devices. It states that "Applications are unaffected" so the user knows the degree of worry that should be involved.

BTRFS' internal structure is very different from ZFS' so we don't have this notion of vdev, consisting of multiple child devices. And so each physical + vdev can be considered a separate device. So again, without extending the on-disk format i.e introducing new items or changing the format of existing ones we can't accommodate the exact same reports. And while the on-disk format can be changed (which of course comes with added complexity) there should be a very good reason to do so. Clearly something is amiss in your case, but I would like to first properly root cause it before jumping to conclusions.

Quoted 23 lines
# zpool status
  pool: pet630
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: scrub repaired 380K in 156h39m with 0 errors on Sat Jun 20 13:03:26 
2020
config:
        NAME           STATE     READ WRITE CKSUM
        pet630         ONLINE       0     0     0
          raidz1-0     ONLINE       0     0     0
            sdf        ONLINE       0     0     0
            sdq        ONLINE       0     0     0
            sdd        ONLINE       0     0     0
            sdh        ONLINE       0     0     0
            sdi        ONLINE      41     0     1
#6 lore raw
Quoted 10 lines · AEST Nikolay Borisov
Quoted 3 lines

In this case I'm getting application IO errors and lost data, so if the error count is designed to not count recovered errors then it's still not doing the right thing.

In this case yes, however this was utterly not clear from your initial email. In fact it seems you have omitted quite a lot of information. So let's step back and start afresh. So first give information about your current btrfs setup by giving the output of:

btrfs fi usage /path/to/btrfs

# btrfs fi usa .
Overall:
    Device size:                  62.50GiB
    Device allocated:             19.02GiB
    Device unallocated:           43.48GiB
    Device missing:                  0.00B
    Used:                         16.26GiB
    Free (estimated):             44.25GiB      (min: 22.51GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:               17.06MiB      (used: 0.00B)

Data,single: Size:17.01GiB, Used:16.23GiB (95.43%)
/dev/sdc1 17.01GiB

Metadata,DUP: Size:1.00GiB, Used:17.19MiB (1.68%)
/dev/sdc1 2.00GiB

System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
/dev/sdc1 16.00MiB

Unallocated:
/dev/sdc1 43.48GiB

Quoted 15 lines

From the output provided it seems the affected mirror is '1', which to me implies you have at least another disk containing the same data. So unless you have errors in mirror 0 as well those errors should be recovered from by simply reading from that mirror.

Quoted 9 lines
# md5sum *
md5sum: 'Rise of the Machines S1 Ep6 - Mega Digger-qcOpMtIWsrgN.mp4':
Input/ output error
md5sum: 'Rise of the Machines S1 Ep7 - Ultimate
Dragster-Ke9hMplfQAWF.mp4':
Input/output error
md5sum: 'Rise of the Machines S1 Ep8 - Aircraft Carrier-Qxht6qMEwMKr.mp4':
Input/output error
^C
Quoted 1 line

You are trying to md5sum 3 distinct files....

There's more files, some of the files were read correctly.

Quoted 17 lines
Quoted 8 lines
# btrfs dev sta .
[/dev/sdc1].write_io_errs    0
[/dev/sdc1].read_io_errs     0
[/dev/sdc1].flush_io_errs    0
[/dev/sdc1].corruption_errs  0
[/dev/sdc1].generation_errs  0
# tail /var/log/kern.log
Jun 23 17:48:40 xev kernel: [417603.547748] BTRFS warning (device sdc1):
Quoted 8 lines
csum failed root 5 ino 275 off 59580416 csum 0x8941f998 expected csum
0xb5b581fc mirror 1
Jun 23 17:48:40 xev kernel: [417603.609861] BTRFS warning (device sdc1):
csum failed root 5 ino 275 off 60628992 csum 0x8941f998 expected csum
0x4b6c9883 mirror 1
Jun 23 17:48:40 xev kernel: [417603.672251] BTRFS warning (device sdc1):
csum failed root 5 ino 275 off 61677568 csum 0x8941f998 expected csum
0x89f5fb68 mirror 1
Quoted 5 lines

Yet here all the errors happen in one inode, namely 275. So the md5sum commands do not correspond to those errors specifically. Also provide the name of inode 275. And for good measure also provide the output of "btrfs check /dev/sdc1" - this is a read only command so if there is some metadata corruption it will at least not make it any worse.

# ls -li /mnt/tmp|grep 275
275 -rw-r--r--. 1 root root  507979219 Jun  3 11:05 Rise of the Machines S1 
Ep8 - Aircraft Carrier-Qxht6qMEwMKr.mp4
# umount /mnt/tmp
# btrfs check /dev/sdc1
Opening filesystem to check...
Checking filesystem on /dev/sdc1
UUID: 841b569f-63ab-477f-b603-64e4e4339146
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 17446019072 bytes used, no error found
total csum bytes: 17014904
total tree bytes: 18038784
total fs tree bytes: 81920
total extent tree bytes: 114688
btree space waste bytes: 669647
file data blocks allocated: 17427980288
 referenced 17427980288

I don't mind about making problems worse, there is no precious data on that device, just downloads of some TV shows which are also stored elsewhere. But I don't want to have such problems happen to more important data.

Quoted 17 lines
Quoted 12 lines · AEST Nikolay Borisov
Quoted 4 lines

read_io_errs. But this leads to a different can of worms - if a user sees read_io_errs should they be worried because potentially some data is stale or not (give we won't be distinguishing between unrepairable vs transient ones).

If a user sees errors reported their degree of worry should be based on the
degree to which they use RAID and have decent backups. If you have RAID-1 and only 1 device has errors then you are OK. If you have 2 devices with errors then you have a problem.

Quoted 4 lines

Below is an example of a zpool having errors that were corrected. The DEVICE had an unrecoverable error, but the RAID-Z pool recovered it from other devices. It states that "Applications are unaffected" so the user knows the degree of worry that should be involved.

Quoted 9 lines

BTRFS' internal structure is very different from ZFS' so we don't have this notion of vdev, consisting of multiple child devices. And so each physical + vdev can be considered a separate device. So again, without extending the on-disk format i.e introducing new items or changing the format of existing ones we can't accommodate the exact same reports. And while the on-disk format can be changed (which of course comes with added complexity) there should be a very good reason to do so. Clearly something is amiss in your case, but I would like to first properly root cause it before jumping to conclusions.

ZFS gives fewer numbers when asking for device status, but the numbers provide more useful information.

Also it is possible to have numbers stored only in RAM that get lost when the filesystem is umounted. That would still be very useful for monitoring systems. I want to know about problems and replace disks before the problems become significant enough to lose data.

Signature · 2 lines
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/
#7 lore raw
Quoted 27 lines · Russell Coker
Quoted 10 lines · AEST Nikolay Borisov
Quoted 3 lines

In this case I'm getting application IO errors and lost data, so if the error count is designed to not count recovered errors then it's still not doing the right thing.

In this case yes, however this was utterly not clear from your initial email. In fact it seems you have omitted quite a lot of information. So let's step back and start afresh. So first give information about your current btrfs setup by giving the output of:

btrfs fi usage /path/to/btrfs

# btrfs fi usa .
Overall:
    Device size:                  62.50GiB
    Device allocated:             19.02GiB
    Device unallocated:           43.48GiB
    Device missing:                  0.00B
    Used:                         16.26GiB
    Free (estimated):             44.25GiB      (min: 22.51GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:               17.06MiB      (used: 0.00B)

Data,single: Size:17.01GiB, Used:16.23GiB (95.43%)
/dev/sdc1 17.01GiB

Quoted 8 lines

Metadata,DUP: Size:1.00GiB, Used:17.19MiB (1.68%)
/dev/sdc1 2.00GiB

System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
/dev/sdc1 16.00MiB

Unallocated:

   /dev/sdc1      43.48GiB

Do you use compression on this filesystem i.e have you mounted with -ocompression= option ?

Based on this data alone it's evident that you don't really have mirrors of the data, in this case having experienced the checksum errors should have indeed resulted in error counters being incremented. I'll look into this.

<snip>

#8 lore raw
Quoted 18 lines · AEST Nikolay Borisov
Quoted 18 lines
# btrfs fi usa .
Overall:
Device size:                  62.50GiB
Device allocated:             19.02GiB
Device unallocated:           43.48GiB
Device missing:                  0.00B
Used:                         16.26GiB
Free (estimated):             44.25GiB      (min: 22.51GiB)
Data ratio:                       1.00
Metadata ratio:                   2.00
Global reserve:               17.06MiB      (used: 0.00B)

Data,single: Size:17.01GiB, Used:16.23GiB (95.43%)
/dev/sdc1 17.01GiB

Metadata,DUP: Size:1.00GiB, Used:17.19MiB (1.68%)
/dev/sdc1 2.00GiB

Quoted 8 lines
Quoted 5 lines

System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
/dev/sdc1 16.00MiB

Unallocated:
/dev/sdc1 43.48GiB

Do you use compression on this filesystem i.e have you mounted with -ocompression= option ?

No, used the default mount with the Debian build of kernel 5.6.14. Everything was pretty much default with it. Made a filesystem, copied a bunch of large files to it, tried to read it, got problems.

It was a storage device I suspected of having errors, copying files to/from it with BTRFS is a good way of exposing errors.

Quoted 4 lines

Based on this data alone it's evident that you don't really have mirrors of the data, in this case having experienced the checksum errors should have indeed resulted in error counters being incremented. I'll look into this.

Thanks.

Signature · 2 lines
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/
#9 lore raw
In reply to Nikolay Borisov
Quoted 39 lines · Nikolay Borisov
Quoted 27 lines · Russell Coker
Quoted 10 lines · AEST Nikolay Borisov
Quoted 3 lines

In this case I'm getting application IO errors and lost data, so if the error count is designed to not count recovered errors then it's still not doing the right thing.

In this case yes, however this was utterly not clear from your initial email. In fact it seems you have omitted quite a lot of information. So let's step back and start afresh. So first give information about your current btrfs setup by giving the output of:

btrfs fi usage /path/to/btrfs

# btrfs fi usa .
Overall:
    Device size:                  62.50GiB
    Device allocated:             19.02GiB
    Device unallocated:           43.48GiB
    Device missing:                  0.00B
    Used:                         16.26GiB
    Free (estimated):             44.25GiB      (min: 22.51GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:               17.06MiB      (used: 0.00B)

Data,single: Size:17.01GiB, Used:16.23GiB (95.43%)
/dev/sdc1 17.01GiB

Quoted 8 lines

Metadata,DUP: Size:1.00GiB, Used:17.19MiB (1.68%)
/dev/sdc1 2.00GiB

System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
/dev/sdc1 16.00MiB

Unallocated:

   /dev/sdc1      43.48GiB
Quoted 7 lines

Do you use compression on this filesystem i.e have you mounted with -ocompression= option ?

Based on this data alone it's evident that you don't really have mirrors of the data, in this case having experienced the checksum errors should have indeed resulted in error counters being incremented. I'll look into this.

In commit 0cc068e6ee59 "btrfs: don't report readahead errors and don't update statistics" we stopped counting errors if they occur during readahead. If there's a mirror available, we do still correct errors in that case. Errors in readahead are fairly common, e.g. there are usually a few during lvm pvmove operations, so it maybe makes sense not to count them; however, if the errors are not counted, they should also not be repaired. Instead, they should be repaired only during non-readahead reads (i.e. when the repairs will be counted in dev stats). Repairing errors without counting is bad because it hides an important indicator of device failure.

This thread might be a different issue since there aren't any mirrors with single data, but if you're look at dev stats correctness anyway...

Quoted 1 line

<snip>

#10 lore raw
Quoted 59 lines · Zygo Blaxell
Quoted 39 lines · Nikolay Borisov
Quoted 27 lines · Russell Coker
Quoted 10 lines · AEST Nikolay Borisov
Quoted 3 lines

In this case I'm getting application IO errors and lost data, so if the error count is designed to not count recovered errors then it's still not doing the right thing.

In this case yes, however this was utterly not clear from your initial email. In fact it seems you have omitted quite a lot of information. So let's step back and start afresh. So first give information about your current btrfs setup by giving the output of:

btrfs fi usage /path/to/btrfs

# btrfs fi usa .
Overall:
    Device size:                  62.50GiB
    Device allocated:             19.02GiB
    Device unallocated:           43.48GiB
    Device missing:                  0.00B
    Used:                         16.26GiB
    Free (estimated):             44.25GiB      (min: 22.51GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:               17.06MiB      (used: 0.00B)

Data,single: Size:17.01GiB, Used:16.23GiB (95.43%)
/dev/sdc1 17.01GiB

Quoted 8 lines

Metadata,DUP: Size:1.00GiB, Used:17.19MiB (1.68%)
/dev/sdc1 2.00GiB

System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
/dev/sdc1 16.00MiB

Unallocated:

   /dev/sdc1      43.48GiB
Quoted 7 lines

Do you use compression on this filesystem i.e have you mounted with -ocompression= option ?

Based on this data alone it's evident that you don't really have mirrors of the data, in this case having experienced the checksum errors should have indeed resulted in error counters being incremented. I'll look into this.

In commit 0cc068e6ee59 "btrfs: don't report readahead errors and don't update statistics" we stopped counting errors if they occur during readahead. If there's a mirror available, we do still correct errors in that case. Errors in readahead are fairly common, e.g. there are usually a few during lvm pvmove operations, so it maybe makes sense not to count them; however, if the errors are not counted, they should also not be repaired. Instead, they should be repaired only during non-readahead reads (i.e. when the repairs will be counted in dev stats). Repairing errors without counting is bad because it hides an important indicator of device failure.

Quoted 2 lines

This thread might be a different issue since there aren't any mirrors with single data, but if you're look at dev stats correctness anyway...

Turns out this is a genueine bug, namely errors stats are only ever updated in btrfs_end_bio which happens well before checksums are checked. In fact at the time when we are checking checksums end_bio_extent_readpage->readpage_end_io_hook (btrfs_readpage_end_io_hook) we don't (currently) have enough context to increment the errors. I'm currently testing a tentative fix for this.

Quoted 3 lines
Quoted 2 lines

<snip>