Skip to contentSkip to conversation

bad tree block start, want 705757184 have 82362368

DISCUSSION Stephan Olbrich 4 participants 9 messages Nov 16, 2018 – Nov 19, 2018 Btrfs file system

lore
Share & tools
lore

Stephan Olbrich

author
#0 lore raw

Hi,

a few days ago my root file system (simple btrfs on a SSD, no RAID or anything) suddenly became read only. Looking at dmsg, I found this:

[   19.285020] BTRFS error (device sda2): bad tree block start, want 705757184 have 82362368
[   19.285042] BTRFS: error (device sda2) in __btrfs_free_extent:6804: errno=-5 IO failure
[   19.285048] BTRFS info (device sda2): forced readonly
[   19.285051] BTRFS: error (device sda2) in btrfs_run_delayed_refs:2934: errno=-5 IO failure
[   19.287213] BTRFS error (device sda2): pending csums is 41889792

Late on I got the same errors for my /home partition (on the same drive) as well.
I have snapshots of all partitions on another drive made by btrbk. To get a working system, I made new (rw) snapshots of the most recent backup and setup grub and fstab, so my system would boot from the other drive.
Unfortunately now I got the "bad tree block start" error again at least once in dmesg but I didn't save it and it's not in syslog :-( What I remember is, that it was followed by other btrfs error messages saying something about correcting something. And the filesystem was still read/write this time.
At the moment I can't reproduce it.

Is there any way to find out, which files are affected by the errors above? I don't really trust the data on the drive I'm using at the moment, as it has shown errors as well, but I have a less current backup on yet another drive but at it is a few weeks old, I don't want to use it to setup the system on the SSD again, but just copy the relevant files if possible.
Or is it possible to repair the original file system?

Some information about my system:
Kubuntu 18.04
Kernel 4.19.1 when the problem occured, now 4.19.2 btrfs-tools 4.15.1

Regards,
Stephan

#1 lore raw
Quoted 24 lines · Stephan Olbrich

Hi,

a few days ago my root file system (simple btrfs on a SSD, no RAID or anything) suddenly became read only. Looking at dmsg, I found this:

[   19.285020] BTRFS error (device sda2): bad tree block start, want 705757184 have 82362368
[   19.285042] BTRFS: error (device sda2) in __btrfs_free_extent:6804: errno=-5 IO failure
[   19.285048] BTRFS info (device sda2): forced readonly
[   19.285051] BTRFS: error (device sda2) in btrfs_run_delayed_refs:2934: errno=-5 IO failure
[   19.287213] BTRFS error (device sda2): pending csums is 41889792

Late on I got the same errors for my /home partition (on the same drive) as well.
I have snapshots of all partitions on another drive made by btrbk. To get a working system, I made new (rw) snapshots of the most recent backup and setup grub and fstab, so my system would boot from the other drive.
Unfortunately now I got the "bad tree block start" error again at least once in dmesg but I didn't save it and it's not in syslog :-( What I remember is, that it was followed by other btrfs error messages saying something about correcting something. And the filesystem was still read/write this time.
At the moment I can't reproduce it.

Is there any way to find out, which files are affected by the errors above? I don't really trust the data on the drive I'm using at the moment, as it has shown errors as well, but I have a less current backup on yet another drive but at it is a few weeks old, I don't want to use it to setup the system on the SSD again, but just copy the relevant files if possible.
Or is it possible to repair the original file system?

Quoted 4 lines

Some information about my system:
Kubuntu 18.04
Kernel 4.19.1 when the problem occured, now 4.19.2 btrfs-tools 4.15.1

What is the SMART status of your SSD, how old is the ssd. This really sounds like the drive going to lalal land.

Quoted 9 lines

Regards,
Stephan

author
#2 lore raw

Am Freitag, 16. November 2018, 17:44:35 CET schrieb Nikolay Borisov:

Quoted 32 lines · Stephan Olbrich
Quoted 31 lines

Hi,

a few days ago my root file system (simple btrfs on a SSD, no RAID or anything) suddenly became read only. Looking at dmsg, I found this:

[   19.285020] BTRFS error (device sda2): bad tree block start, want
705757184 have 82362368 [   19.285042] BTRFS: error (device sda2) in
__btrfs_free_extent:6804: errno=-5 IO failure [   19.285048] BTRFS info
(device sda2): forced readonly
[   19.285051] BTRFS: error (device sda2) in btrfs_run_delayed_refs:2934:

errno=-5 IO failure [ 19.287213] BTRFS error (device sda2): pending
csums is 41889792

Late on I got the same errors for my /home partition (on the same drive) as well. I have snapshots of all partitions on another drive made by btrbk. To get a working system, I made new (rw) snapshots of the most recent backup and setup grub and fstab, so my system would boot from the other drive. Unfortunately now I got the "bad tree block start" error again at least once in dmesg but I didn't save it and it's not in syslog :-( What I remember is, that it was followed by other btrfs error messages saying something about correcting something. And the filesystem was still read/write this time.
At the moment I can't reproduce it.

Is there any way to find out, which files are affected by the errors above? I don't really trust the data on the drive I'm using at the moment, as it has shown errors as well, but I have a less current backup on yet another drive but at it is a few weeks old, I don't want to use it to setup the system on the SSD again, but just copy the relevant files if possible. Or is it possible to repair the original file system?

Quoted 7 lines
Quoted 4 lines

Some information about my system:
Kubuntu 18.04
Kernel 4.19.1 when the problem occured, now 4.19.2 btrfs-tools 4.15.1

What is the SMART status of your SSD, how old is the ssd. This really sounds like the drive going to lalal land.

The SSD is 3.5 years old and was never really full, so I wouldn't expect it to die. SMART shows no errors. All values are way above the threshold and selftests show no errors.
I checked RAM as well. No errors there, so I have no clue, where these errors come from.

Regards,
Stephan

#3 lore raw
In reply to Stephan Olbrich
Quoted 6 lines · Stephan Olbrich

Hi,

a few days ago my root file system (simple btrfs on a SSD, no RAID or anything) suddenly became read only. Looking at dmsg, I found this:

[   19.285020] BTRFS error (device sda2): bad tree block start, want 705757184 have 82362368

Does this only occurs once?

If only once, are you using SINGLE metadata profile (default for SSD)? If using DUP/RAID1 it may have a chance to recover.

Quoted 1 line
[   19.285042] BTRFS: error (device sda2) in __btrfs_free_extent:6804: errno=-5 IO failure

The problem is in extent tree.

If there is no other problem, your data should be OK.

You could still try to mount the fs RO to salvage data.

Quoted 12 lines
[   19.285048] BTRFS info (device sda2): forced readonly
[   19.285051] BTRFS: error (device sda2) in btrfs_run_delayed_refs:2934: errno=-5 IO failure
[   19.287213] BTRFS error (device sda2): pending csums is 41889792

Late on I got the same errors for my /home partition (on the same drive) as well.
I have snapshots of all partitions on another drive made by btrbk. To get a working system, I made new (rw) snapshots of the most recent backup and setup grub and fstab, so my system would boot from the other drive.
Unfortunately now I got the "bad tree block start" error again at least once in dmesg but I didn't save it and it's not in syslog :-( What I remember is, that it was followed by other btrfs error messages saying something about correcting something. And the filesystem was still read/write this time.
At the moment I can't reproduce it.

Quoted 1 line

Is there any way to find out, which files are affected by the errors above?

No files are affected, but an essential tree, extent tree, is corrupted.

Normally this may prevent RW mount, and even it mounts it can still cause problem when doing any write.
It could even prevent RO mount if the corrupted leaf contains block group item.

But your data should be OK if there is no other corruption, and in that case btrfs-restore should work well.

Quoted 4 lines

I don't really trust the data on the drive I'm using at the
moment, as it has shown errors as well, but I have a less current backup on yet another drive but at it is a few weeks old, I don't want to use it to setup the system on the SSD again, but just copy the relevant files if possible.
Or is it possible to repair the original file system?

At least we need "btrfs check" output.

Quoted 5 lines

Some information about my system:
Kubuntu 18.04
Kernel 4.19.1 when the problem occured, now 4.19.2 btrfs-tools 4.15.1

And "btrfs check" should be executed using latest version.

Thanks,
Qu

Quoted 8 lines

Regards,
Stephan

author
#4 lore raw

Am Samstag, 17. November 2018, 09:03:04 CET schrieb Qu Wenruo:

Quoted 6 lines · Stephan Olbrich
Quoted 5 lines

Hi,

a few days ago my root file system (simple btrfs on a SSD, no RAID or anything) suddenly became read only. Looking at dmsg, I found this:

Quoted 6 lines
Quoted 2 lines
[   19.285020] BTRFS error (device sda2): bad tree block start, want

705757184 have 82362368

Does this only occurs once?

If only once, are you using SINGLE metadata profile (default for SSD)? If using DUP/RAID1 it may have a chance to recover.

It shows up once after reboot for each partition.
I'm using SINGLE metadata.

Quoted 4 lines
Quoted 3 lines
[   19.285042] BTRFS: error (device sda2) in __btrfs_free_extent:6804:

errno=-5 IO failure

The problem is in extent tree.

I checked my logs again, the __btrfs_free_extent error is missing for the /home partition:

[ 3154.066544] BTRFS error (device sda4): bad tree block start, want 560119808 have 4330389667183373366
[ 3154.066558] BTRFS: error (device sda4) in btrfs_run_delayed_refs:2934: errno=-5 IO failure
[ 3154.066561] BTRFS info (device sda4): forced readonly
[ 3154.066904] BTRFS error (device sda4): pending csums is 360448
Quoted 4 lines

If there is no other problem, your data should be OK.

You could still try to mount the fs RO to salvage data.

I'm not so much worried about salvaging data, as I have a current backup. I'm more worried how reliable the backup is, as I'm not sure, when the errors first started. But if the data is OK, then the backup (snapshots on another drive) should be OK as well I guess.

Quoted 29 lines
Quoted 16 lines
[   19.285048] BTRFS info (device sda2): forced readonly
[   19.285051] BTRFS: error (device sda2) in btrfs_run_delayed_refs:2934:

errno=-5 IO failure [ 19.287213] BTRFS error (device sda2): pending
csums is 41889792

Late on I got the same errors for my /home partition (on the same drive) as well. I have snapshots of all partitions on another drive made by btrbk. To get a working system, I made new (rw) snapshots of the most recent backup and setup grub and fstab, so my system would boot from the other drive. Unfortunately now I got the "bad tree block start" error again at least once in dmesg but I didn't save it and it's not in syslog :-( What I remember is, that it was followed by other btrfs error messages saying something about correcting something. And the filesystem was still read/write this time.
At the moment I can't reproduce it.

Quoted 2 lines

Is there any way to find out, which files are affected by the errors above?

No files are affected, but an essential tree, extent tree, is corrupted.

Normally this may prevent RW mount, and even it mounts it can still cause problem when doing any write.
It could even prevent RO mount if the corrupted leaf contains block group item.

But your data should be OK if there is no other corruption, and in that case btrfs-restore should work well.

Quoted 7 lines
Quoted 5 lines

I don't really trust the data on the drive I'm using at the moment, as it has shown errors as well, but I have a less current backup on yet another drive but at it is a few weeks old, I don't want to use it to setup the system on the SSD again, but just copy the relevant files if possible. Or is it possible to repair the original file system?

At least we need "btrfs check" output.

I updated btrfs-progs and run btrfs check for / and /home No errors are found on / (sda2), but there are errors on /home ??

$ btrfs --version
btrfs-progs v4.19
$ btrfs check /dev/sda2
Opening filesystem to check...
Checking filesystem on /dev/sda2
UUID: 80368989-ffa8-463c-98fb-fe2e28ca7bf3
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 64816218112 bytes used, no error found
total csum bytes: 59518732
total tree bytes: 2180268032
total fs tree bytes: 1965965312
total extent tree bytes: 123289600
btree space waste bytes: 478665777
file data blocks allocated: 151083261952
 referenced 76879990784
$ btrfs check /dev/sda4
Opening filesystem to check...
Checking filesystem on /dev/sda4
UUID: 81c38df8-b7f9-412c-8c88-cfde8db68eb1
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
root 257 inode 7970563 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 20480
root 257 inode 7970564 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 77824
ERROR: errors found in fs roots
found 303386652672 bytes used, error(s) found
total csum bytes: 289501272
total tree bytes: 2336227328
total fs tree bytes: 1766473728
total extent tree bytes: 202014720
btree space waste bytes: 519245278
file data blocks allocated: 6851730792448
 referenced 533348069376

Thanks,
Stephan

Quoted 13 lines
Quoted 5 lines

Some information about my system:
Kubuntu 18.04
Kernel 4.19.1 when the problem occured, now 4.19.2 btrfs-tools 4.15.1

And "btrfs check" should be executed using latest version.

Thanks,
Qu

Quoted 2 lines

Regards,
Stephan

#5 lore raw
Quoted 29 lines · Stephan Olbrich

Am Samstag, 17. November 2018, 09:03:04 CET schrieb Qu Wenruo:

Quoted 6 lines · Stephan Olbrich
Quoted 5 lines

Hi,

a few days ago my root file system (simple btrfs on a SSD, no RAID or anything) suddenly became read only. Looking at dmsg, I found this:

Quoted 6 lines
Quoted 2 lines
[   19.285020] BTRFS error (device sda2): bad tree block start, want

705757184 have 82362368

Does this only occurs once?

If only once, are you using SINGLE metadata profile (default for SSD)? If using DUP/RAID1 it may have a chance to recover.

It shows up once after reboot for each partition.
I'm using SINGLE metadata.

Quoted 4 lines
Quoted 3 lines
[   19.285042] BTRFS: error (device sda2) in __btrfs_free_extent:6804:

errno=-5 IO failure

The problem is in extent tree.

I checked my logs again, the __btrfs_free_extent error is missing for the /home partition:

[ 3154.066544] BTRFS error (device sda4): bad tree block start, want 560119808 have 4330389667183373366
[ 3154.066558] BTRFS: error (device sda4) in btrfs_run_delayed_refs:2934: errno=-5 IO failure
[ 3154.066561] BTRFS info (device sda4): forced readonly
[ 3154.066904] BTRFS error (device sda4): pending csums is 360448
Quoted 1 line
Quoted 12 lines
Quoted 3 lines

If there is no other problem, your data should be OK.

You could still try to mount the fs RO to salvage data.

I'm not so much worried about salvaging data, as I have a current backup. I'm more worried how reliable the backup is, as I'm not sure, when the errors first started. But if the data is OK, then the backup (snapshots on another drive) should be OK as well I guess.

Quoted 4 lines
Quoted 4 lines
[   19.285048] BTRFS info (device sda2): forced readonly
[   19.285051] BTRFS: error (device sda2) in btrfs_run_delayed_refs:2934:

errno=-5 IO failure [ 19.287213] BTRFS error (device sda2): pending
csums is 41889792

Still extent tree.

Quoted 49 lines
Quoted 25 lines
Quoted 12 lines

Late on I got the same errors for my /home partition (on the same drive) as well. I have snapshots of all partitions on another drive made by btrbk. To get a working system, I made new (rw) snapshots of the most recent backup and setup grub and fstab, so my system would boot from the other drive. Unfortunately now I got the "bad tree block start" error again at least once in dmesg but I didn't save it and it's not in syslog :-( What I remember is, that it was followed by other btrfs error messages saying something about correcting something. And the filesystem was still read/write this time.
At the moment I can't reproduce it.

Quoted 2 lines

Is there any way to find out, which files are affected by the errors above?

No files are affected, but an essential tree, extent tree, is corrupted.

Normally this may prevent RW mount, and even it mounts it can still cause problem when doing any write.
It could even prevent RO mount if the corrupted leaf contains block group item.

But your data should be OK if there is no other corruption, and in that case btrfs-restore should work well.

Quoted 7 lines
Quoted 5 lines

I don't really trust the data on the drive I'm using at the moment, as it has shown errors as well, but I have a less current backup on yet another drive but at it is a few weeks old, I don't want to use it to setup the system on the SSD again, but just copy the relevant files if possible. Or is it possible to repair the original file system?

At least we need "btrfs check" output.

I updated btrfs-progs and run btrfs check for / and /home No errors are found on / (sda2), but there are errors on /home ??

$ btrfs --version
btrfs-progs v4.19
$ btrfs check /dev/sda2
Opening filesystem to check...
Checking filesystem on /dev/sda2
UUID: 80368989-ffa8-463c-98fb-fe2e28ca7bf3
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
Quoted 8 lines
found 64816218112 bytes used, no error found
total csum bytes: 59518732
total tree bytes: 2180268032
total fs tree bytes: 1965965312
total extent tree bytes: 123289600
btree space waste bytes: 478665777
file data blocks allocated: 151083261952
 referenced 76879990784

This fs is completely fine, including your data.

Quoted 16 lines
$ btrfs check /dev/sda4
Opening filesystem to check...
Checking filesystem on /dev/sda4
UUID: 81c38df8-b7f9-412c-8c88-cfde8db68eb1
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
root 257 inode 7970563 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 20480
root 257 inode 7970564 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 77824
ERROR: errors found in fs roots

These are just minor errors, won't even causing any data mismatch.

So all your fses should be mostly OK.

Would you please try to use v4.19-rc* to see if it changes anything?

Thanks,
Qu

Quoted 29 lines

found 303386652672 bytes used, error(s) found
total csum bytes: 289501272
total tree bytes: 2336227328
total fs tree bytes: 1766473728
total extent tree bytes: 202014720
btree space waste bytes: 519245278
file data blocks allocated: 6851730792448
referenced 533348069376

Thanks,
Stephan

Quoted 13 lines
Quoted 5 lines

Some information about my system:
Kubuntu 18.04
Kernel 4.19.1 when the problem occured, now 4.19.2 btrfs-tools 4.15.1

And "btrfs check" should be executed using latest version.

Thanks,
Qu

Quoted 2 lines

Regards,
Stephan

author
#6 lore raw

Am Sonntag, 18. November 2018, 01:30:14 CET schrieb Qu Wenruo:

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

Late on I got the same errors for my /home partition (on the same drive) as well. I have snapshots of all partitions on another drive made by btrbk. To get a working system, I made new (rw) snapshots of the most recent backup and setup grub and fstab, so my system would boot from the other drive. Unfortunately now I got the "bad tree block start" error again at least once in dmesg but I didn't save it and it's not in syslog :-( What I remember is, that it was followed by other btrfs error messages saying something about correcting something. And the filesystem was still read/write this time.
At the moment I can't reproduce it.

Today it happend again (sdb is my backup drive, which is my main drive at the moment):

[  286.325857] BTRFS error (device sdb1): bad tree block start, want 787719208960 have 11268016545161247416
[  286.363245] BTRFS info (device sdb1): read error corrected: ino 0 off 787719208960 (dev /dev/sdb1 sector 1243815072)
[  286.364087] BTRFS info (device sdb1): read error corrected: ino 0 off 787719213056 (dev /dev/sdb1 sector 1243815080)
[  286.425946] BTRFS info (device sdb1): read error corrected: ino 0 off 787719217152 (dev /dev/sdb1 sector 1243815088)
[  286.427530] BTRFS info (device sdb1): read error corrected: ino 0 off 787719221248 (dev /dev/sdb1 sector 1243815096)
Quoted 70 lines
Quoted 48 lines
Quoted 21 lines
Quoted 2 lines

Is there any way to find out, which files are affected by the errors above?

No files are affected, but an essential tree, extent tree, is corrupted.

Normally this may prevent RW mount, and even it mounts it can still cause problem when doing any write.
It could even prevent RO mount if the corrupted leaf contains block group item.

But your data should be OK if there is no other corruption, and in that case btrfs-restore should work well.

Quoted 7 lines

I don't really trust the data on the drive I'm using at the moment, as it has shown errors as well, but I have a less current backup on yet another drive but at it is a few weeks old, I don't want to use it
to setup the system on the SSD again, but just copy the relevant files if
possible. Or is it possible to repair the original file system?

Quoted 1 line

At least we need "btrfs check" output.

I updated btrfs-progs and run btrfs check for / and /home No errors are found on / (sda2), but there are errors on /home ??

$ btrfs --version
btrfs-progs v4.19
$ btrfs check /dev/sda2
Opening filesystem to check...
Checking filesystem on /dev/sda2
UUID: 80368989-ffa8-463c-98fb-fe2e28ca7bf3
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 64816218112 bytes used, no error found
total csum bytes: 59518732
total tree bytes: 2180268032
total fs tree bytes: 1965965312
total extent tree bytes: 123289600
btree space waste bytes: 478665777
file data blocks allocated: 151083261952
Quoted 1 line

referenced 76879990784

This fs is completely fine, including your data.

Quoted 18 lines
$ btrfs check /dev/sda4
Opening filesystem to check...
Checking filesystem on /dev/sda4
UUID: 81c38df8-b7f9-412c-8c88-cfde8db68eb1
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
root 257 inode 7970563 errors 100, file extent discount

Found file extent holes:

        start: 0, len: 20480

root 257 inode 7970564 errors 100, file extent discount

Found file extent holes:

        start: 0, len: 77824
Quoted 7 lines
Quoted 1 line

ERROR: errors found in fs roots

These are just minor errors, won't even causing any data mismatch.

So all your fses should be mostly OK.

Would you please try to use v4.19-rc* to see if it changes anything?

v4.19-rc1 is the only rc I found, but that is older than v4.19, right? Anyway, here is the output:

$ btrfs --version
btrfs-progs v4.19-rc1
$ btrfs check /dev/sda2
Opening filesystem to check...
Checking filesystem on /dev/sda2
UUID: 80368989-ffa8-463c-98fb-fe2e28ca7bf3
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 64816218112 bytes used, no error found
total csum bytes: 59518732
total tree bytes: 2180268032
total fs tree bytes: 1965965312
total extent tree bytes: 123289600
btree space waste bytes: 478665777
file data blocks allocated: 151083261952
 referenced 76879990784
$ btrfs check /dev/sda4
Opening filesystem to check...
Checking filesystem on /dev/sda4
UUID: 81c38df8-b7f9-412c-8c88-cfde8db68eb1
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
root 257 inode 7970563 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 20480
root 257 inode 7970564 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 77824
ERROR: errors found in fs roots
found 303386652672 bytes used, error(s) found
total csum bytes: 289501272
total tree bytes: 2336227328
total fs tree bytes: 1766473728
total extent tree bytes: 202014720                                                                                                                                                                        
btree space waste bytes: 519245278                                                                                                                                                                        
file data blocks allocated: 6851730792448                                                                                                                                                                 
 referenced 533348069376

Thanks,
Stephan

Quoted 28 lines

Thanks,
Qu

Quoted 25 lines

found 303386652672 bytes used, error(s) found
total csum bytes: 289501272
total tree bytes: 2336227328
total fs tree bytes: 1766473728
total extent tree bytes: 202014720
btree space waste bytes: 519245278
file data blocks allocated: 6851730792448

referenced 533348069376

Thanks,
Stephan

Quoted 12 lines
Quoted 4 lines

Some information about my system:
Kubuntu 18.04
Kernel 4.19.1 when the problem occured, now 4.19.2 btrfs-tools 4.15.1

And "btrfs check" should be executed using latest version.

Thanks,
Qu

Quoted 2 lines

Regards,
Stephan

#7 lore raw
Quoted 12 lines · Stephan Olbrich

Am Sonntag, 18. November 2018, 01:30:14 CET schrieb Qu Wenruo:

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

Late on I got the same errors for my /home partition (on the same drive) as well. I have snapshots of all partitions on another drive made by btrbk. To get a working system, I made new (rw) snapshots of the most recent backup and setup grub and fstab, so my system would boot from the other drive. Unfortunately now I got the "bad tree block start" error again at least once in dmesg but I didn't save it and it's not in syslog :-( What I remember is, that it was followed by other btrfs error messages saying something about correcting something. And the filesystem was still read/write this time.
At the moment I can't reproduce it.

Quoted 6 lines

Today it happend again (sdb is my backup drive, which is my main drive at the moment):

[  286.325857] BTRFS error (device sdb1): bad tree block start, want 787719208960 have 11268016545161247416
[  286.363245] BTRFS info (device sdb1): read error corrected: ino 0 off 787719208960 (dev /dev/sdb1 sector 1243815072)
[  286.364087] BTRFS info (device sdb1): read error corrected: ino 0 off 787719213056 (dev /dev/sdb1 sector 1243815080)
[  286.425946] BTRFS info (device sdb1): read error corrected: ino 0 off 787719217152 (dev /dev/sdb1 sector 1243815088)
[  286.427530] BTRFS info (device sdb1): read error corrected: ino 0 off 787719221248 (dev /dev/sdb1 sector 1243815096)

Was there any hardware issues? How about the following data from the
system..

btrfs fi df <mnt>
btrfs dev stat <mnt>

Thanks, Anand

Quoted 165 lines
Quoted 71 lines
Quoted 49 lines
Quoted 22 lines
Quoted 3 lines

Is there any way to find out, which files are affected by the errors above?

No files are affected, but an essential tree, extent tree, is corrupted.

Normally this may prevent RW mount, and even it mounts it can still cause problem when doing any write.
It could even prevent RO mount if the corrupted leaf contains block group item.

But your data should be OK if there is no other corruption, and in that case btrfs-restore should work well.

Quoted 7 lines

I don't really trust the data on the drive I'm using at the moment, as it has shown errors as well, but I have a less current backup on yet another drive but at it is a few weeks old, I don't want to use it
to setup the system on the SSD again, but just copy the relevant files if
possible. Or is it possible to repair the original file system?

Quoted 1 line

At least we need "btrfs check" output.

I updated btrfs-progs and run btrfs check for / and /home No errors are found on / (sda2), but there are errors on /home ??

$ btrfs --version
btrfs-progs v4.19
$ btrfs check /dev/sda2
Opening filesystem to check...
Checking filesystem on /dev/sda2
UUID: 80368989-ffa8-463c-98fb-fe2e28ca7bf3
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 64816218112 bytes used, no error found
total csum bytes: 59518732
total tree bytes: 2180268032
total fs tree bytes: 1965965312
total extent tree bytes: 123289600
btree space waste bytes: 478665777
file data blocks allocated: 151083261952
Quoted 1 line

referenced 76879990784

This fs is completely fine, including your data.

Quoted 18 lines
$ btrfs check /dev/sda4
Opening filesystem to check...
Checking filesystem on /dev/sda4
UUID: 81c38df8-b7f9-412c-8c88-cfde8db68eb1
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
root 257 inode 7970563 errors 100, file extent discount

Found file extent holes:

         start: 0, len: 20480

root 257 inode 7970564 errors 100, file extent discount

Found file extent holes:

         start: 0, len: 77824
Quoted 7 lines
Quoted 1 line

ERROR: errors found in fs roots

These are just minor errors, won't even causing any data mismatch.

So all your fses should be mostly OK.

Would you please try to use v4.19-rc* to see if it changes anything?

v4.19-rc1 is the only rc I found, but that is older than v4.19, right? Anyway, here is the output:

$ btrfs --version
btrfs-progs v4.19-rc1
$ btrfs check /dev/sda2
Opening filesystem to check...
Checking filesystem on /dev/sda2
UUID: 80368989-ffa8-463c-98fb-fe2e28ca7bf3
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 64816218112 bytes used, no error found
total csum bytes: 59518732
total tree bytes: 2180268032
total fs tree bytes: 1965965312
total extent tree bytes: 123289600
btree space waste bytes: 478665777
file data blocks allocated: 151083261952
  referenced 76879990784
$ btrfs check /dev/sda4
Opening filesystem to check...
Checking filesystem on /dev/sda4
UUID: 81c38df8-b7f9-412c-8c88-cfde8db68eb1
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
root 257 inode 7970563 errors 100, file extent discount
Found file extent holes:
         start: 0, len: 20480
root 257 inode 7970564 errors 100, file extent discount
Found file extent holes:
         start: 0, len: 77824
ERROR: errors found in fs roots
found 303386652672 bytes used, error(s) found
total csum bytes: 289501272
total tree bytes: 2336227328
total fs tree bytes: 1766473728
total extent tree bytes: 202014720
btree space waste bytes: 519245278
file data blocks allocated: 6851730792448
  referenced 533348069376

Thanks,
Stephan

Quoted 28 lines

Thanks,
Qu

Quoted 25 lines

found 303386652672 bytes used, error(s) found
total csum bytes: 289501272
total tree bytes: 2336227328
total fs tree bytes: 1766473728
total extent tree bytes: 202014720
btree space waste bytes: 519245278
file data blocks allocated: 6851730792448

referenced 533348069376

Thanks,
Stephan

Quoted 12 lines
Quoted 4 lines

Some information about my system:
Kubuntu 18.04
Kernel 4.19.1 when the problem occured, now 4.19.2 btrfs-tools 4.15.1

And "btrfs check" should be executed using latest version.

Thanks,
Qu

Quoted 2 lines

Regards,
Stephan

author
#8 lore raw

Am Sonntag, 18. November 2018, 14:31:36 CET schrieb Anand Jain:

Quoted 29 lines · Stephan Olbrich
Quoted 18 lines

Am Sonntag, 18. November 2018, 01:30:14 CET schrieb Qu Wenruo:

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

Late on I got the same errors for my /home partition (on the same drive)
as well. I have snapshots of all partitions on another drive made by btrbk. To get a working system, I made new (rw) snapshots of the most recent backup and setup grub and fstab, so my system would boot from the
other drive. Unfortunately now I got the "bad tree block start" error again at least once in dmesg but I didn't save it and it's not in syslog

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

:-( What I remember is, that it was followed by other btrfs error

messages saying something about correcting something. And the filesystem
was still read/write this time.
At the moment I can't reproduce it.

Quoted 10 lines

Today it happend again (sdb is my backup drive, which is my main drive at the moment): [ 286.325857] BTRFS error (device sdb1): bad tree block start, want 787719208960 have 11268016545161247416 [ 286.363245] BTRFS info (device sdb1): read error corrected: ino 0 off 787719208960 (dev /dev/sdb1 sector 1243815072) [ 286.364087] BTRFS info (device sdb1):
read error corrected: ino 0 off 787719213056 (dev /dev/sdb1 sector 1243815080) [ 286.425946] BTRFS info (device sdb1): read error
corrected: ino 0 off 787719217152 (dev /dev/sdb1 sector 1243815088) [ 286.427530] BTRFS info (device sdb1): read error corrected: ino 0 off 787719221248 (dev /dev/sdb1 sector 1243815096)

Quoted 5 lines

Was there any hardware issues? How about the following data from the
system..

btrfs fi df <mnt>
btrfs dev stat <mnt>

I didn't have any hardware issues (that I know of).

$ btrfs fi df /
Data, single: total=841.00GiB, used=791.92GiB                                                                                                                                                              
System, DUP: total=8.00MiB, used=112.00KiB                                                                                                                                                                 
System, single: total=4.00MiB, used=0.00B                                                                                                                                                                  
Metadata, DUP: total=10.50GiB, used=8.44GiB                                                                                                                                                                
Metadata, single: total=8.00MiB, used=0.00B                                                                                                                                                                
GlobalReserve, single: total=512.00MiB, used=0.00B
$ btrfs dev stat /                                                                                                                                                             
[/dev/sdb1].write_io_errs    0                                                                                                                                                                             
[/dev/sdb1].read_io_errs     0                                                                                                                                                                             
[/dev/sdb1].flush_io_errs    0                                                                                                                                                                             
[/dev/sdb1].corruption_errs  0                                                                                                                                                                             
[/dev/sdb1].generation_errs  0

Also the numbers in the error messages keep changing:

[ 1577.302959] BTRFS error (device sdb1): bad tree block start, want 788086226944 have 411602213549769407
[ 1577.369083] BTRFS info (device sdb1): read error corrected: ino 0 off 788086226944 (dev /dev/sdb1 sector 1244531904)
[ 1577.428139] BTRFS info (device sdb1): read error corrected: ino 0 off 788086231040 (dev /dev/sdb1 sector 1244531912)
[ 1577.429091] BTRFS info (device sdb1): read error corrected: ino 0 off 788086235136 (dev /dev/sdb1 sector 1244531920)
[ 1577.478249] BTRFS info (device sdb1): read error corrected: ino 0 off 788086239232 (dev /dev/sdb1 sector 1244531928)

Thanks,
Stephan

Quoted 169 lines

Thanks, Anand

Quoted 167 lines
Quoted 72 lines
Quoted 50 lines
Quoted 23 lines
Quoted 2 lines

Is there any way to find out, which files are affected by the errors above?

No files are affected, but an essential tree, extent tree, is corrupted.

Normally this may prevent RW mount, and even it mounts it can still cause problem when doing any write.
It could even prevent RO mount if the corrupted leaf contains block group item.

But your data should be OK if there is no other corruption, and in that case btrfs-restore should work well.

Quoted 8 lines

I don't really trust the data on the drive I'm using at the moment, as it has shown errors as well, but I have a less current backup
on yet another drive but at it is a few weeks old, I don't want to use it
to setup the system on the SSD again, but just copy the relevant files if
possible. Or is it possible to repair the original file system?

Quoted 1 line

At least we need "btrfs check" output.

I updated btrfs-progs and run btrfs check for / and /home No errors are found on / (sda2), but there are errors on /home ??

$ btrfs --version
btrfs-progs v4.19
$ btrfs check /dev/sda2
Opening filesystem to check...
Checking filesystem on /dev/sda2
UUID: 80368989-ffa8-463c-98fb-fe2e28ca7bf3
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 64816218112 bytes used, no error found
total csum bytes: 59518732
total tree bytes: 2180268032
total fs tree bytes: 1965965312
total extent tree bytes: 123289600
btree space waste bytes: 478665777
file data blocks allocated: 151083261952
Quoted 1 line

referenced 76879990784

This fs is completely fine, including your data.

Quoted 18 lines
$ btrfs check /dev/sda4
Opening filesystem to check...
Checking filesystem on /dev/sda4
UUID: 81c38df8-b7f9-412c-8c88-cfde8db68eb1
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
root 257 inode 7970563 errors 100, file extent discount

Found file extent holes:

         start: 0, len: 20480

root 257 inode 7970564 errors 100, file extent discount

Found file extent holes:

         start: 0, len: 77824
Quoted 7 lines
Quoted 1 line

ERROR: errors found in fs roots

These are just minor errors, won't even causing any data mismatch.

So all your fses should be mostly OK.

Would you please try to use v4.19-rc* to see if it changes anything?

v4.19-rc1 is the only rc I found, but that is older than v4.19, right? Anyway, here is the output:

$ btrfs --version
btrfs-progs v4.19-rc1
$ btrfs check /dev/sda2
Opening filesystem to check...
Checking filesystem on /dev/sda2
UUID: 80368989-ffa8-463c-98fb-fe2e28ca7bf3
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 64816218112 bytes used, no error found
total csum bytes: 59518732
total tree bytes: 2180268032
total fs tree bytes: 1965965312
total extent tree bytes: 123289600
btree space waste bytes: 478665777
file data blocks allocated: 151083261952

referenced 76879990784

$ btrfs check /dev/sda4
Opening filesystem to check...
Checking filesystem on /dev/sda4
UUID: 81c38df8-b7f9-412c-8c88-cfde8db68eb1
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
root 257 inode 7970563 errors 100, file extent discount

Found file extent holes:

         start: 0, len: 20480

root 257 inode 7970564 errors 100, file extent discount

Found file extent holes:

         start: 0, len: 77824

ERROR: errors found in fs roots
found 303386652672 bytes used, error(s) found
total csum bytes: 289501272
total tree bytes: 2336227328
total fs tree bytes: 1766473728
total extent tree bytes: 202014720
btree space waste bytes: 519245278
file data blocks allocated: 6851730792448

referenced 533348069376

Thanks,
Stephan

Quoted 28 lines

Thanks,
Qu

Quoted 25 lines

found 303386652672 bytes used, error(s) found
total csum bytes: 289501272
total tree bytes: 2336227328
total fs tree bytes: 1766473728
total extent tree bytes: 202014720
btree space waste bytes: 519245278
file data blocks allocated: 6851730792448

referenced 533348069376

Thanks,
Stephan

Quoted 12 lines
Quoted 4 lines

Some information about my system:
Kubuntu 18.04
Kernel 4.19.1 when the problem occured, now 4.19.2 btrfs-tools 4.15.1

And "btrfs check" should be executed using latest version.

Thanks,
Qu

Quoted 2 lines

Regards,
Stephan