Skip to contentSkip to conversation

btrfs errors

DISCUSSION Konstantin V. Gavrilenko 2 participants 5 messages Aug 13, 2019 – Aug 14, 2019 3 messages not in the archive Btrfs file system

lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive

Hi list

I have run the btrfs check, and that reported multiple errors on the FS.

# btrfs check --readonly --force /dev/kubuntu-vg/lv_root
<SKIP>
root 9214 inode 6850330 errors 2001, no inode item, link count wrong
        unresolved ref dir 266982 index 22459 namelen 36 name 9621041045a17a475428a26fcfb5982f.png filetype 1 errors 6, no dir index, no inode ref
        unresolved ref dir 226516 index 9 namelen 28 name GYTSPMxjwCVp8kXB7+j91O8kcq4= filetype 1 errors 4, no inode ref
root 9214 inode 6877070 errors 2001, no inode item, link count wrong
        unresolved ref dir 226516 index 11 namelen 28 name VSqlYzl4pFqJpvC3GA9bQ0mZK8A= filetype 1 errors 4, no inode ref
root 9214 inode 6878054 errors 2001, no inode item, link count wrong
        unresolved ref dir 266982 index 22460 namelen 36 name 52e74e9d2b6f598038486f90f8f911c4.png filetype 1 errors 4, no inode ref
root 9214 inode 6888414 errors 2001, no inode item, link count wrong
        unresolved ref dir 226391 index 122475 namelen 14 name the-real-index filetype 1 errors 4, no inode ref
root 9214 inode 6889202 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 4096
root 9214 inode 6889203 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 4096
ERROR: errors found in fs roots
found 334531551237 bytes used, error(s) found
total csum bytes: 291555464
total tree bytes: 1004404736
total fs tree bytes: 411713536
total extent tree bytes: 242974720
btree space waste bytes: 186523621
file data blocks allocated: 36730163200
 referenced 42646511616

However, scrub and badblock find no errors.

# btrfs scrub status /mnt/
scrub status for 7b19fb5e-4e16-4b62-b803-f59364fd61a2
        scrub started at Tue Aug 13 07:31:38 2019 and finished after 00:02:47
        total bytes scrubbed: 311.15GiB with 0 errors
# badblocks -sv /dev/kubuntu-vg/lv_root 
Checking blocks 0 to 352133119
Checking for bad blocks (read-only test):  done                                                 
Pass completed, 0 bad blocks found. (0/0/0 errors)
# btrfs dev stats /dev/kubuntu-vg/lv_root                                                                                                                                                       
[/dev/mapper/kubuntu--vg-lv_root].write_io_errs    0
[/dev/mapper/kubuntu--vg-lv_root].read_io_errs     0
[/dev/mapper/kubuntu--vg-lv_root].flush_io_errs    0
[/dev/mapper/kubuntu--vg-lv_root].corruption_errs  0
[/dev/mapper/kubuntu--vg-lv_root].generation_errs  0

FS mount fine, and is operational.
would you recommend executing the btrfs check --repair option or is there something else that I can try.

# uname -a Linux xps 5.1.16-050116-generic #201907031232 SMP Wed Jul 3 12:35:21 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

# btrfs --version                                                                                                                                                                               
btrfs-progs v4.15.1

mount options
on / type btrfs (rw,relatime,compress-force=lzo,ssd,discard,space_cache=v2,autodefrag,subvol=/@)

thanks,
Konstantin

#2 lore raw
Quoted 6 lines · Konstantin V. Gavrilenko

Hi list

I have run the btrfs check, and that reported multiple errors on the FS.

# btrfs check --readonly --force /dev/kubuntu-vg/lv_root
<SKIP>

Please don't skip the output, especially for btrfs check.

The first tree btrfs check checks is extent tree, if we have anything wrong in extent tree, it's way serious than the later part.

And I understand you want to check your root fs, thus you have to use --force, but I'd recommend to go whatever distro you like, use its liveCD/USB to check your root fs.

It looks like that since your fs is still mounted, the data structure changed during the btrfs check run, it's possible to cause false alert.

Quoted 26 lines

root 9214 inode 6850330 errors 2001, no inode item, link count wrong
unresolved ref dir 266982 index 22459 namelen 36 name 9621041045a17a475428a26fcfb5982f.png filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 226516 index 9 namelen 28 name GYTSPMxjwCVp8kXB7+j91O8kcq4= filetype 1 errors 4, no inode ref
root 9214 inode 6877070 errors 2001, no inode item, link count wrong
unresolved ref dir 226516 index 11 namelen 28 name VSqlYzl4pFqJpvC3GA9bQ0mZK8A= filetype 1 errors 4, no inode ref
root 9214 inode 6878054 errors 2001, no inode item, link count wrong

        unresolved ref dir 266982 index 22460 namelen 36 name 52e74e9d2b6f598038486f90f8f911c4.png filetype 1 errors 4, no inode ref

root 9214 inode 6888414 errors 2001, no inode item, link count wrong
unresolved ref dir 226391 index 122475 namelen 14 name the-real-index filetype 1 errors 4, no inode ref
root 9214 inode 6889202 errors 100, file extent discount
Found file extent holes:

        start: 0, len: 4096
root 9214 inode 6889203 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 4096

ERROR: errors found in fs roots
found 334531551237 bytes used, error(s) found
total csum bytes: 291555464
total tree bytes: 1004404736
total fs tree bytes: 411713536
total extent tree bytes: 242974720
btree space waste bytes: 186523621
file data blocks allocated: 36730163200
referenced 42646511616

Quoted 6 lines

However, scrub and badblock find no errors.

# btrfs scrub status /mnt/
scrub status for 7b19fb5e-4e16-4b62-b803-f59364fd61a2
        scrub started at Tue Aug 13 07:31:38 2019 and finished after 00:02:47
        total bytes scrubbed: 311.15GiB with 0 errors

Scrub only checks checksum, doesn't care the content. (Kernel newer than v5.0 will care the content, but doesn't do full cross-check, unlike btrfs-check)

Quoted 15 lines
# badblocks -sv /dev/kubuntu-vg/lv_root 
Checking blocks 0 to 352133119
Checking for bad blocks (read-only test):  done                                                 
Pass completed, 0 bad blocks found. (0/0/0 errors)
# btrfs dev stats /dev/kubuntu-vg/lv_root                                                                                                                                                       
[/dev/mapper/kubuntu--vg-lv_root].write_io_errs    0
[/dev/mapper/kubuntu--vg-lv_root].read_io_errs     0
[/dev/mapper/kubuntu--vg-lv_root].flush_io_errs    0
[/dev/mapper/kubuntu--vg-lv_root].corruption_errs  0
[/dev/mapper/kubuntu--vg-lv_root].generation_errs  0
Quoted 2 lines

FS mount fine, and is operational.
would you recommend executing the btrfs check --repair option or is there something else that I can try.

Don't do anything stupid yet.
Just go LiveCD/USB and check again.

Quoted 2 lines

# uname -a Linux xps 5.1.16-050116-generic #201907031232 SMP Wed Jul 3 12:35:21 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Since v5.2 introduced a lot of new restrict check, I'd recommend to go mount with latest Archiso, btrfs-check first, if no problem, mount and scrub again just in case.

Quoted 2 lines
# btrfs --version                                                                                                                                                                               
btrfs-progs v4.15.1

Big nope. You won't really want to run btrfs check --repair on such old and buggy progs. Unless recent releases (5.2?) btrfs-progs has a bug that transaction is not committed correctly, thus if something wrong happened like BUG_ON() or transaction aborted, the fs can easily be screwed up.

Thanks,
Qu

Quoted 8 lines

mount options
on / type btrfs (rw,relatime,compress-force=lzo,ssd,discard,space_cache=v2,autodefrag,subvol=/@)

thanks,
Konstantin

#4 lore raw
In reply to a message not in the archive
Quoted 22 lines · Konstantin V. Gavrilenko

Hi Qu,

thanks for the quick response. so I've booted into the latest Archiso (kernel 5.2.xx, btrfs 5.2.xx) and rerun the # btrfs check and # btrfs scrub. The btrfs check output is rather large and is included as an attachment to this message.

It seems that the problem lies in fs_roots.
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
root 257 inode 6360900 errors 1000, some csum missing
root 258 inode 364233 errors 1040, bad file extent, some csum missing root 258 inode 364234 errors 1040, bad file extent, some csum missing
....
root 258 inode 5074178 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096

root 258 inode 5386921 errors 1040, bad file extent, some csum missing
...
ERROR: errors found in fs roots

Quoted 12 lines

Opening filesystem to check...
Checking filesystem on /dev/kubuntu-vg/lv_root
UUID: 7b19fb5e-4e16-4b62-b803-f59364fd61a2
cache and super generation don't match, space cache will be invalidated found 335628447749 bytes used, error(s) found
total csum bytes: 292272496
total tree bytes: 5118279680
total fs tree bytes: 4523163648
total extent tree bytes: 244498432
btree space waste bytes: 1186351226
file data blocks allocated: 536719597568
referenced 630181720064

The result indeeds shows some *minor* problem.

One is "bad file extent" normally related to some extent too large for compressed inlined extent.

Considering you're using lzo compression, it looks like some older kernel behavior which is no longer considered sane nowadays.

You don't need to panic (never run btrfs check --repair yet), as your fs is mostly fine, no data loss or whatever.

At least in recent kernel releases, you won't hit a problem.

Quoted 12 lines

so I've mounted the FS and run scrub, which resulted in "ALL OK" again.

UUID:             7b19fb5e-4e16-4b62-b803-f59364fd61a2
Scrub started:    Tue Aug 13 13:01:26 2019
Status:           finished
Duration:         0:02:44
Total to scrub:   312.59GiB
Rate:             1.91GiB/s
Error summary:    no errors found
Quoted 2 lines

I have backed up all the important data on the external disc just now, and no errors in the dmesg were reported, so I assume the data is OK. I also have snapshots of this system stored on the external disc dating back to Apr'19.

Currently it looks like false alert.

But to be sure, please do me a favor by running lowmem mode check, which should output more useful info other than "bad file extent".

# btrfs check --mode=lowmem <dev>

It may take a longer time to finish. But should be more useful.

Quoted 2 lines

So I guess the two important questions are
- is it possible to reliable recover FS, or at least find out which files were affected at the reported inode location.

If it's a false alert, you don't need to recover anything.

If it's too strict btrfs check, and you want to follow latest btrfs *too sane* behavior, you can just try copy the old data to a newly created btrfs.

Thanks,
Qu

Quoted 121 lines

- is it possible to # btrfs check <snapshot> without copying it back on the main disk. maybe loopdevice?

thanks,
Konstantin

----- Original Message -----
From: "Qu Wenruo" <[email protected]>
To: "Konstantin V. Gavrilenko" <[email protected]>, "linux-btrfs" <[email protected]>
Sent: Tuesday, 13 August, 2019 12:55:47 PM
Subject: Re: btrfs errors

Quoted 6 lines · Konstantin V. Gavrilenko

Hi list

I have run the btrfs check, and that reported multiple errors on the FS.

# btrfs check --readonly --force /dev/kubuntu-vg/lv_root
<SKIP>

Please don't skip the output, especially for btrfs check.

The first tree btrfs check checks is extent tree, if we have anything wrong in extent tree, it's way serious than the later part.

And I understand you want to check your root fs, thus you have to use --force, but I'd recommend to go whatever distro you like, use its liveCD/USB to check your root fs.

It looks like that since your fs is still mounted, the data structure changed during the btrfs check run, it's possible to cause false alert.

Quoted 26 lines

root 9214 inode 6850330 errors 2001, no inode item, link count wrong
unresolved ref dir 266982 index 22459 namelen 36 name 9621041045a17a475428a26fcfb5982f.png filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 226516 index 9 namelen 28 name GYTSPMxjwCVp8kXB7+j91O8kcq4= filetype 1 errors 4, no inode ref
root 9214 inode 6877070 errors 2001, no inode item, link count wrong
unresolved ref dir 226516 index 11 namelen 28 name VSqlYzl4pFqJpvC3GA9bQ0mZK8A= filetype 1 errors 4, no inode ref
root 9214 inode 6878054 errors 2001, no inode item, link count wrong

        unresolved ref dir 266982 index 22460 namelen 36 name 52e74e9d2b6f598038486f90f8f911c4.png filetype 1 errors 4, no inode ref

root 9214 inode 6888414 errors 2001, no inode item, link count wrong
unresolved ref dir 226391 index 122475 namelen 14 name the-real-index filetype 1 errors 4, no inode ref
root 9214 inode 6889202 errors 100, file extent discount
Found file extent holes:

        start: 0, len: 4096
root 9214 inode 6889203 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 4096

ERROR: errors found in fs roots
found 334531551237 bytes used, error(s) found
total csum bytes: 291555464
total tree bytes: 1004404736
total fs tree bytes: 411713536
total extent tree bytes: 242974720
btree space waste bytes: 186523621
file data blocks allocated: 36730163200
referenced 42646511616

Quoted 6 lines

However, scrub and badblock find no errors.

# btrfs scrub status /mnt/
scrub status for 7b19fb5e-4e16-4b62-b803-f59364fd61a2
        scrub started at Tue Aug 13 07:31:38 2019 and finished after 00:02:47
        total bytes scrubbed: 311.15GiB with 0 errors

Scrub only checks checksum, doesn't care the content. (Kernel newer than v5.0 will care the content, but doesn't do full cross-check, unlike btrfs-check)

Quoted 15 lines
# badblocks -sv /dev/kubuntu-vg/lv_root 
Checking blocks 0 to 352133119
Checking for bad blocks (read-only test):  done                                                 
Pass completed, 0 bad blocks found. (0/0/0 errors)
# btrfs dev stats /dev/kubuntu-vg/lv_root                                                                                                                                                       
[/dev/mapper/kubuntu--vg-lv_root].write_io_errs    0
[/dev/mapper/kubuntu--vg-lv_root].read_io_errs     0
[/dev/mapper/kubuntu--vg-lv_root].flush_io_errs    0
[/dev/mapper/kubuntu--vg-lv_root].corruption_errs  0
[/dev/mapper/kubuntu--vg-lv_root].generation_errs  0
Quoted 2 lines

FS mount fine, and is operational.
would you recommend executing the btrfs check --repair option or is there something else that I can try.

Don't do anything stupid yet.
Just go LiveCD/USB and check again.

Quoted 2 lines

# uname -a Linux xps 5.1.16-050116-generic #201907031232 SMP Wed Jul 3 12:35:21 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Since v5.2 introduced a lot of new restrict check, I'd recommend to go mount with latest Archiso, btrfs-check first, if no problem, mount and scrub again just in case.

Quoted 2 lines
# btrfs --version                                                                                                                                                                               
btrfs-progs v4.15.1

Big nope. You won't really want to run btrfs check --repair on such old and buggy progs. Unless recent releases (5.2?) btrfs-progs has a bug that transaction is not committed correctly, thus if something wrong happened like BUG_ON() or transaction aborted, the fs can easily be screwed up.

Thanks,
Qu

Quoted 8 lines

mount options
on / type btrfs (rw,relatime,compress-force=lzo,ssd,discard,space_cache=v2,autodefrag,subvol=/@)

thanks,
Konstantin

#6 lore raw
In reply to a message not in the archive

Thanks for the dump.

Now it's very clear what the problem is.

It's an old kernel behavior which allows compressed file extents to have NODATASUM flag.

The behavior is going to be fixed in v5.3 by commit 42c16da6d684 ("btrfs: inode: Don't compress if NODATASUM or NODATACOW set").

Currently you can just dismiss the false alert.
The only minor downside of the current behavior is, if one copy of your data is corrupted, there is no chance to recover the data even you're using DUP/RAID1/RAID5/RAID6/RAID10.

Or you can wait for v5.3 kernel and copy old data back to a newly created fs which only modified by v5.3 kernel.

Thanks,
Qu

Quoted 280 lines · Konstantin V. Gavrilenko

Yours sincerely,
Konstantin V. Gavrilenko

Director
Arhont Services Ltd

web:    http://www.arhont.com
e-mail: [email protected]

tel: +44 (0) 870 44 31337
fax: +44 (0) 208 429 3111

PGP: Key ID - 0xE81824F4
PGP: Server - keyserver.pgp.com

----- Original Message -----
From: "Qu Wenruo" <[email protected]>
To: "Konstantin V. Gavrilenko" <[email protected]>
Cc: "linux-btrfs" <[email protected]>
Sent: Tuesday, 13 August, 2019 4:01:30 PM
Subject: Re: btrfs errors

Quoted 22 lines · Konstantin V. Gavrilenko

Hi Qu,

thanks for the quick response. so I've booted into the latest Archiso (kernel 5.2.xx, btrfs 5.2.xx) and rerun the # btrfs check and # btrfs scrub. The btrfs check output is rather large and is included as an attachment to this message.

It seems that the problem lies in fs_roots.
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
root 257 inode 6360900 errors 1000, some csum missing
root 258 inode 364233 errors 1040, bad file extent, some csum missing root 258 inode 364234 errors 1040, bad file extent, some csum missing
....
root 258 inode 5074178 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096

root 258 inode 5386921 errors 1040, bad file extent, some csum missing
...
ERROR: errors found in fs roots

Quoted 12 lines

Opening filesystem to check...
Checking filesystem on /dev/kubuntu-vg/lv_root
UUID: 7b19fb5e-4e16-4b62-b803-f59364fd61a2
cache and super generation don't match, space cache will be invalidated found 335628447749 bytes used, error(s) found
total csum bytes: 292272496
total tree bytes: 5118279680
total fs tree bytes: 4523163648
total extent tree bytes: 244498432
btree space waste bytes: 1186351226
file data blocks allocated: 536719597568
referenced 630181720064

The result indeeds shows some *minor* problem.

One is "bad file extent" normally related to some extent too large for compressed inlined extent.

Considering you're using lzo compression, it looks like some older kernel behavior which is no longer considered sane nowadays.

You don't need to panic (never run btrfs check --repair yet), as your fs is mostly fine, no data loss or whatever.

At least in recent kernel releases, you won't hit a problem.

KVG: Good to hear that. Actually this system is about 1.5 y/o, since the release of Ubuntu 18.04LTS that was shipped with 4.15 and the mount options remained the same from the start. the kernel upgrade path was 4.18 in Feb'19, then 5.0 in Apr'19 followed by 5.1 in May'19.

I also used to run on a monthly basis for about a year.

btrfs filesystem defrag -r -t 32m $MP 2> /dev/null btrfs balance start -musage=35 -dusage=55 $MP

but once I started doing daily snapshost, I stopped doing it a it was messing the free space calculations.

Quoted 12 lines

so I've mounted the FS and run scrub, which resulted in "ALL OK" again.

UUID:             7b19fb5e-4e16-4b62-b803-f59364fd61a2
Scrub started:    Tue Aug 13 13:01:26 2019
Status:           finished
Duration:         0:02:44
Total to scrub:   312.59GiB
Rate:             1.91GiB/s
Error summary:    no errors found
Quoted 2 lines

I have backed up all the important data on the external disc just now, and no errors in the dmesg were reported, so I assume the data is OK. I also have snapshots of this system stored on the external disc dating back to Apr'19.

Currently it looks like false alert.

But to be sure, please do me a favor by running lowmem mode check, which should output more useful info other than "bad file extent".

# btrfs check --mode=lowmem <dev>

It may take a longer time to finish. But should be more useful.

KVG: Indeed, the btrfs check generated 150k lines of text :) so the more detailed errors fall into the following categories

326 lines similar to
ERROR: root 258 INODE[5074178] size 162 should have a file extent hole
ERROR: root 258 INODE[5711285] size 586 should have a file extent hole
ERROR: root 258 INODE[5761076] size 215 should have a file extent hole

75584 lines similar to
ERROR: root 258 EXTENT_DATA[364233 0] is compressed, but inode flag doesn't allow it
ERROR: root 258 EXTENT_DATA[364233 131072] is compressed, but inode flag doesn't allow it
ERROR: root 258 EXTENT_DATA[364233 262144] is compressed, but inode flag doesn't allow it

75693 lines similar to
ERROR: root 258 EXTENT_DATA[364233 0] compressed extent must have csum, but only 0 bytes have, expect 4096
ERROR: root 258 EXTENT_DATA[364233 131072] compressed extent must have csum, but only 0 bytes have, expect 16384
ERROR: root 258 EXTENT_DATA[364233 262144] compressed extent must have csum, but only 0 bytes have, expect 4096

The complete output is xz'ed and attached.

Quoted 2 lines

So I guess the two important questions are
- is it possible to reliable recover FS, or at least find out which files were affected at the reported inode location.

If it's a false alert, you don't need to recover anything.

If it's too strict btrfs check, and you want to follow latest btrfs *too sane* behavior, you can just try copy the old data to a newly created btrfs.

KVG: Sure, thats one of the possibilities. What about if I try a forceful recompression of the complete FS via

# btrfs filesystem defragment -c $MP

or alternatively I can try to remove compression alltogether and run a defrag to see if that helps remove the problem. What do you think?

thanks,
Kos

Thanks,
Qu

Quoted 121 lines

- is it possible to # btrfs check <snapshot> without copying it back on the main disk. maybe loopdevice?

thanks,
Konstantin

----- Original Message -----
From: "Qu Wenruo" <[email protected]>
To: "Konstantin V. Gavrilenko" <[email protected]>, "linux-btrfs" <[email protected]>
Sent: Tuesday, 13 August, 2019 12:55:47 PM
Subject: Re: btrfs errors

Quoted 6 lines · Konstantin V. Gavrilenko

Hi list

I have run the btrfs check, and that reported multiple errors on the FS.

# btrfs check --readonly --force /dev/kubuntu-vg/lv_root
<SKIP>

Please don't skip the output, especially for btrfs check.

The first tree btrfs check checks is extent tree, if we have anything wrong in extent tree, it's way serious than the later part.

And I understand you want to check your root fs, thus you have to use --force, but I'd recommend to go whatever distro you like, use its liveCD/USB to check your root fs.

It looks like that since your fs is still mounted, the data structure changed during the btrfs check run, it's possible to cause false alert.

Quoted 26 lines

root 9214 inode 6850330 errors 2001, no inode item, link count wrong
unresolved ref dir 266982 index 22459 namelen 36 name 9621041045a17a475428a26fcfb5982f.png filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 226516 index 9 namelen 28 name GYTSPMxjwCVp8kXB7+j91O8kcq4= filetype 1 errors 4, no inode ref
root 9214 inode 6877070 errors 2001, no inode item, link count wrong
unresolved ref dir 226516 index 11 namelen 28 name VSqlYzl4pFqJpvC3GA9bQ0mZK8A= filetype 1 errors 4, no inode ref
root 9214 inode 6878054 errors 2001, no inode item, link count wrong

        unresolved ref dir 266982 index 22460 namelen 36 name 52e74e9d2b6f598038486f90f8f911c4.png filetype 1 errors 4, no inode ref

root 9214 inode 6888414 errors 2001, no inode item, link count wrong
unresolved ref dir 226391 index 122475 namelen 14 name the-real-index filetype 1 errors 4, no inode ref
root 9214 inode 6889202 errors 100, file extent discount
Found file extent holes:

        start: 0, len: 4096
root 9214 inode 6889203 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 4096

ERROR: errors found in fs roots
found 334531551237 bytes used, error(s) found
total csum bytes: 291555464
total tree bytes: 1004404736
total fs tree bytes: 411713536
total extent tree bytes: 242974720
btree space waste bytes: 186523621
file data blocks allocated: 36730163200
referenced 42646511616

Quoted 6 lines

However, scrub and badblock find no errors.

# btrfs scrub status /mnt/
scrub status for 7b19fb5e-4e16-4b62-b803-f59364fd61a2
        scrub started at Tue Aug 13 07:31:38 2019 and finished after 00:02:47
        total bytes scrubbed: 311.15GiB with 0 errors

Scrub only checks checksum, doesn't care the content. (Kernel newer than v5.0 will care the content, but doesn't do full cross-check, unlike btrfs-check)

Quoted 15 lines
# badblocks -sv /dev/kubuntu-vg/lv_root 
Checking blocks 0 to 352133119
Checking for bad blocks (read-only test):  done                                                 
Pass completed, 0 bad blocks found. (0/0/0 errors)
# btrfs dev stats /dev/kubuntu-vg/lv_root                                                                                                                                                       
[/dev/mapper/kubuntu--vg-lv_root].write_io_errs    0
[/dev/mapper/kubuntu--vg-lv_root].read_io_errs     0
[/dev/mapper/kubuntu--vg-lv_root].flush_io_errs    0
[/dev/mapper/kubuntu--vg-lv_root].corruption_errs  0
[/dev/mapper/kubuntu--vg-lv_root].generation_errs  0
Quoted 2 lines

FS mount fine, and is operational.
would you recommend executing the btrfs check --repair option or is there something else that I can try.

Don't do anything stupid yet.
Just go LiveCD/USB and check again.

Quoted 2 lines

# uname -a Linux xps 5.1.16-050116-generic #201907031232 SMP Wed Jul 3 12:35:21 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Since v5.2 introduced a lot of new restrict check, I'd recommend to go mount with latest Archiso, btrfs-check first, if no problem, mount and scrub again just in case.

Quoted 2 lines
# btrfs --version                                                                                                                                                                               
btrfs-progs v4.15.1

Big nope. You won't really want to run btrfs check --repair on such old and buggy progs. Unless recent releases (5.2?) btrfs-progs has a bug that transaction is not committed correctly, thus if something wrong happened like BUG_ON() or transaction aborted, the fs can easily be screwed up.

Thanks,
Qu

Quoted 8 lines

mount options
on / type btrfs (rw,relatime,compress-force=lzo,ssd,discard,space_cache=v2,autodefrag,subvol=/@)

thanks,
Konstantin

#7 lore raw

Thanks for the help and clarification Qu.

I will wait for the 5.3 and see what it brings.

Best regars,
Konstantin

----- Original Message -----
From: "Qu Wenruo" <[email protected]>
To: "Konstantin V. Gavrilenko" <[email protected]>, "linux-btrfs" <[email protected]>
Sent: Wednesday, 14 August, 2019 1:24:42 AM
Subject: Re: btrfs errors

Thanks for the dump.

Now it's very clear what the problem is.

It's an old kernel behavior which allows compressed file extents to have NODATASUM flag.

The behavior is going to be fixed in v5.3 by commit 42c16da6d684 ("btrfs: inode: Don't compress if NODATASUM or NODATACOW set").

Currently you can just dismiss the false alert.
The only minor downside of the current behavior is, if one copy of your data is corrupted, there is no chance to recover the data even you're using DUP/RAID1/RAID5/RAID6/RAID10.

Or you can wait for v5.3 kernel and copy old data back to a newly created fs which only modified by v5.3 kernel.

Thanks,
Qu

Quoted 280 lines · Konstantin V. Gavrilenko

Yours sincerely,
Konstantin V. Gavrilenko

Director
Arhont Services Ltd

web:    http://www.arhont.com
e-mail: [email protected]

tel: +44 (0) 870 44 31337
fax: +44 (0) 208 429 3111

PGP: Key ID - 0xE81824F4
PGP: Server - keyserver.pgp.com

----- Original Message -----
From: "Qu Wenruo" <[email protected]>
To: "Konstantin V. Gavrilenko" <[email protected]>
Cc: "linux-btrfs" <[email protected]>
Sent: Tuesday, 13 August, 2019 4:01:30 PM
Subject: Re: btrfs errors

Quoted 22 lines · Konstantin V. Gavrilenko

Hi Qu,

thanks for the quick response. so I've booted into the latest Archiso (kernel 5.2.xx, btrfs 5.2.xx) and rerun the # btrfs check and # btrfs scrub. The btrfs check output is rather large and is included as an attachment to this message.

It seems that the problem lies in fs_roots.
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
root 257 inode 6360900 errors 1000, some csum missing
root 258 inode 364233 errors 1040, bad file extent, some csum missing root 258 inode 364234 errors 1040, bad file extent, some csum missing
....
root 258 inode 5074178 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096

root 258 inode 5386921 errors 1040, bad file extent, some csum missing
...
ERROR: errors found in fs roots

Quoted 12 lines

Opening filesystem to check...
Checking filesystem on /dev/kubuntu-vg/lv_root
UUID: 7b19fb5e-4e16-4b62-b803-f59364fd61a2
cache and super generation don't match, space cache will be invalidated found 335628447749 bytes used, error(s) found
total csum bytes: 292272496
total tree bytes: 5118279680
total fs tree bytes: 4523163648
total extent tree bytes: 244498432
btree space waste bytes: 1186351226
file data blocks allocated: 536719597568
referenced 630181720064

The result indeeds shows some *minor* problem.

One is "bad file extent" normally related to some extent too large for compressed inlined extent.

Considering you're using lzo compression, it looks like some older kernel behavior which is no longer considered sane nowadays.

You don't need to panic (never run btrfs check --repair yet), as your fs is mostly fine, no data loss or whatever.

At least in recent kernel releases, you won't hit a problem.

KVG: Good to hear that. Actually this system is about 1.5 y/o, since the release of Ubuntu 18.04LTS that was shipped with 4.15 and the mount options remained the same from the start. the kernel upgrade path was 4.18 in Feb'19, then 5.0 in Apr'19 followed by 5.1 in May'19.

I also used to run on a monthly basis for about a year.

btrfs filesystem defrag -r -t 32m $MP 2> /dev/null btrfs balance start -musage=35 -dusage=55 $MP

but once I started doing daily snapshost, I stopped doing it a it was messing the free space calculations.

Quoted 12 lines

so I've mounted the FS and run scrub, which resulted in "ALL OK" again.

UUID:             7b19fb5e-4e16-4b62-b803-f59364fd61a2
Scrub started:    Tue Aug 13 13:01:26 2019
Status:           finished
Duration:         0:02:44
Total to scrub:   312.59GiB
Rate:             1.91GiB/s
Error summary:    no errors found
Quoted 2 lines

I have backed up all the important data on the external disc just now, and no errors in the dmesg were reported, so I assume the data is OK. I also have snapshots of this system stored on the external disc dating back to Apr'19.

Currently it looks like false alert.

But to be sure, please do me a favor by running lowmem mode check, which should output more useful info other than "bad file extent".

# btrfs check --mode=lowmem <dev>

It may take a longer time to finish. But should be more useful.

KVG: Indeed, the btrfs check generated 150k lines of text :) so the more detailed errors fall into the following categories

326 lines similar to
ERROR: root 258 INODE[5074178] size 162 should have a file extent hole
ERROR: root 258 INODE[5711285] size 586 should have a file extent hole
ERROR: root 258 INODE[5761076] size 215 should have a file extent hole

75584 lines similar to
ERROR: root 258 EXTENT_DATA[364233 0] is compressed, but inode flag doesn't allow it
ERROR: root 258 EXTENT_DATA[364233 131072] is compressed, but inode flag doesn't allow it
ERROR: root 258 EXTENT_DATA[364233 262144] is compressed, but inode flag doesn't allow it

75693 lines similar to
ERROR: root 258 EXTENT_DATA[364233 0] compressed extent must have csum, but only 0 bytes have, expect 4096
ERROR: root 258 EXTENT_DATA[364233 131072] compressed extent must have csum, but only 0 bytes have, expect 16384
ERROR: root 258 EXTENT_DATA[364233 262144] compressed extent must have csum, but only 0 bytes have, expect 4096

The complete output is xz'ed and attached.

Quoted 2 lines

So I guess the two important questions are
- is it possible to reliable recover FS, or at least find out which files were affected at the reported inode location.

If it's a false alert, you don't need to recover anything.

If it's too strict btrfs check, and you want to follow latest btrfs *too sane* behavior, you can just try copy the old data to a newly created btrfs.

KVG: Sure, thats one of the possibilities. What about if I try a forceful recompression of the complete FS via

# btrfs filesystem defragment -c $MP

or alternatively I can try to remove compression alltogether and run a defrag to see if that helps remove the problem. What do you think?

thanks,
Kos

Thanks,
Qu

Quoted 121 lines

- is it possible to # btrfs check <snapshot> without copying it back on the main disk. maybe loopdevice?

thanks,
Konstantin

----- Original Message -----
From: "Qu Wenruo" <[email protected]>
To: "Konstantin V. Gavrilenko" <[email protected]>, "linux-btrfs" <[email protected]>
Sent: Tuesday, 13 August, 2019 12:55:47 PM
Subject: Re: btrfs errors

Quoted 6 lines · Konstantin V. Gavrilenko

Hi list

I have run the btrfs check, and that reported multiple errors on the FS.

# btrfs check --readonly --force /dev/kubuntu-vg/lv_root
<SKIP>

Please don't skip the output, especially for btrfs check.

The first tree btrfs check checks is extent tree, if we have anything wrong in extent tree, it's way serious than the later part.

And I understand you want to check your root fs, thus you have to use --force, but I'd recommend to go whatever distro you like, use its liveCD/USB to check your root fs.

It looks like that since your fs is still mounted, the data structure changed during the btrfs check run, it's possible to cause false alert.

Quoted 26 lines

root 9214 inode 6850330 errors 2001, no inode item, link count wrong
unresolved ref dir 266982 index 22459 namelen 36 name 9621041045a17a475428a26fcfb5982f.png filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 226516 index 9 namelen 28 name GYTSPMxjwCVp8kXB7+j91O8kcq4= filetype 1 errors 4, no inode ref
root 9214 inode 6877070 errors 2001, no inode item, link count wrong
unresolved ref dir 226516 index 11 namelen 28 name VSqlYzl4pFqJpvC3GA9bQ0mZK8A= filetype 1 errors 4, no inode ref
root 9214 inode 6878054 errors 2001, no inode item, link count wrong

        unresolved ref dir 266982 index 22460 namelen 36 name 52e74e9d2b6f598038486f90f8f911c4.png filetype 1 errors 4, no inode ref

root 9214 inode 6888414 errors 2001, no inode item, link count wrong
unresolved ref dir 226391 index 122475 namelen 14 name the-real-index filetype 1 errors 4, no inode ref
root 9214 inode 6889202 errors 100, file extent discount
Found file extent holes:

        start: 0, len: 4096
root 9214 inode 6889203 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 4096

ERROR: errors found in fs roots
found 334531551237 bytes used, error(s) found
total csum bytes: 291555464
total tree bytes: 1004404736
total fs tree bytes: 411713536
total extent tree bytes: 242974720
btree space waste bytes: 186523621
file data blocks allocated: 36730163200
referenced 42646511616

Quoted 6 lines

However, scrub and badblock find no errors.

# btrfs scrub status /mnt/
scrub status for 7b19fb5e-4e16-4b62-b803-f59364fd61a2
        scrub started at Tue Aug 13 07:31:38 2019 and finished after 00:02:47
        total bytes scrubbed: 311.15GiB with 0 errors

Scrub only checks checksum, doesn't care the content. (Kernel newer than v5.0 will care the content, but doesn't do full cross-check, unlike btrfs-check)

Quoted 15 lines
# badblocks -sv /dev/kubuntu-vg/lv_root 
Checking blocks 0 to 352133119
Checking for bad blocks (read-only test):  done                                                 
Pass completed, 0 bad blocks found. (0/0/0 errors)
# btrfs dev stats /dev/kubuntu-vg/lv_root                                                                                                                                                       
[/dev/mapper/kubuntu--vg-lv_root].write_io_errs    0
[/dev/mapper/kubuntu--vg-lv_root].read_io_errs     0
[/dev/mapper/kubuntu--vg-lv_root].flush_io_errs    0
[/dev/mapper/kubuntu--vg-lv_root].corruption_errs  0
[/dev/mapper/kubuntu--vg-lv_root].generation_errs  0
Quoted 2 lines

FS mount fine, and is operational.
would you recommend executing the btrfs check --repair option or is there something else that I can try.

Don't do anything stupid yet.
Just go LiveCD/USB and check again.

Quoted 2 lines

# uname -a Linux xps 5.1.16-050116-generic #201907031232 SMP Wed Jul 3 12:35:21 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Since v5.2 introduced a lot of new restrict check, I'd recommend to go mount with latest Archiso, btrfs-check first, if no problem, mount and scrub again just in case.

Quoted 2 lines
# btrfs --version                                                                                                                                                                               
btrfs-progs v4.15.1

Big nope. You won't really want to run btrfs check --repair on such old and buggy progs. Unless recent releases (5.2?) btrfs-progs has a bug that transaction is not committed correctly, thus if something wrong happened like BUG_ON() or transaction aborted, the fs can easily be screwed up.

Thanks,
Qu

Quoted 8 lines

mount options
on / type btrfs (rw,relatime,compress-force=lzo,ssd,discard,space_cache=v2,autodefrag,subvol=/@)

thanks,
Konstantin