Skip to contentSkip to conversation

scrub "correcting" tons of errors ?

DISCUSSION Swâmi Petaramesh 5 participants 12 messages Mar 29, 2013 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi there,

I've started "btrfs scrub start /" on one of my machines (Kernel 3.8.0-15 Ubuntu AMD64), which typically "behaves well" so I wasn't suspected any disk issue.

After having ran for only 165 seconds, "scrub status" shows it pretends having found and corrected 22926 CSUM errors ??!?!?!?!!???

This is a rather new HDD, in perfect shape (SMART all OK, never reallocated a single sector, less than 200 hours total runtime...)

WTF ?!?

I've cancelled scrub for now, until I get further understanding of what can be happening...

Signature · 2 lines
Swâmi Petaramesh <[email protected]> http://petaramesh.org PGP 9076E32E
Ne cherchez pas : Je ne suis pas sur Facebook.
#1 lore raw
Quoted 9 lines · Swâmi Petaramesh

Hi there,

I've started "btrfs scrub start /" on one of my machines (Kernel 3.8.0-15 Ubuntu AMD64), which typically "behaves well" so I wasn't suspected any disk issue.

After having ran for only 165 seconds, "scrub status" shows it pretends having found and corrected 22926 CSUM errors ??!?!?!?!!???

Quoted 8 lines

This is a rather new HDD, in perfect shape (SMART all OK, never reallocated a single sector, less than 200 hours total runtime...)

WTF ?!?

I've cancelled scrub for now, until I get further understanding of what can be happening...

So this is probably because of the extent tree corruption you had, it's just cleaning things up and you should be fine once it finishes. Thanks,

Josef

#2 lore raw

Hi Josef,

Quoted 2 lines · Josef Bacik

So this is probably because of the extent tree corruption you had, it's just cleaning things up and you should be fine once it finishes. Thanks,

Er... It's on a different machine !

Current (at the time I write) status is :

# btrfs scrub status /
scrub status for 346b81b2-0735-4c4d-a137-1995bc78ad70
        scrub resumed at Fri Mar 29 11:52:43 2013 and finished after
7470 seconds
        total bytes scrubbed: 231.96GB with 149691 errors
        error details: csum=149691
        corrected errors: 149691, uncorrectable errors: 0, unverified
errors: 0

I have to say that scrub completely froze the machine at least 4 times (disk had ceased activity and any command that would imply a disk access would hang forever), but at least after a (quite brutal) reboot it could be resumed...

The only thing about this FS is that it had been imaged, then restored, using partclone.btrfs (which itself is supposed to use the BTRFS libraries).

I have a screenshot of "last thing I saw when it hanged", I can upload it somewhere, should it be relevant...

Kind regards.

Signature · 2 lines
Swâmi Petaramesh <[email protected]> http://petaramesh.org PGP 9076E32E
Ne cherchez pas : Je ne suis pas sur Facebook.
#10 lore raw
In reply to Josef Bacik

On that note, is btrfs doing automatic background scrubs of its own or do I have to use crontab to schedule scrubs?

Thanks!

Quoted 27 lines · Josef Bacik
Quoted 9 lines · Swāmi Petaramesh

Hi there,

I've started "btrfs scrub start /" on one of my machines (Kernel 3.8.0-15 Ubuntu AMD64), which typically "behaves well" so I wasn't suspected any disk issue.

After having ran for only 165 seconds, "scrub status" shows it pretends having found and corrected 22926 CSUM errors ??!?!?!?!!???

Quoted 8 lines

This is a rather new HDD, in perfect shape (SMART all OK, never reallocated a single sector, less than 200 hours total runtime...)

WTF ?!?

I've cancelled scrub for now, until I get further understanding of what can be happening...

So this is probably because of the extent tree corruption you had, it's just cleaning things up and you should be fine once it finishes. Thanks,

Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#11 lore raw
Quoted 2 lines · Harald Glatt

On that note, is btrfs doing automatic background scrubs of its own or do I have to use crontab to schedule scrubs?

If you want a full-disk scrub, you'll need to schedule it yourself
with cron (I run mine once a month). However, if a problem is detected during normal operation -- e.g. you read a piece of data and it's got bad checksums -- then the FS will fix it if it can, in the same way that it would with a scrub.

Hugo.

Quoted 26 lines

Thanks!

Quoted 23 lines · Josef Bacik
Quoted 9 lines · Swāmi Petaramesh

Hi there,

I've started "btrfs scrub start /" on one of my machines (Kernel 3.8.0-15 Ubuntu AMD64), which typically "behaves well" so I wasn't suspected any disk issue.

After having ran for only 165 seconds, "scrub status" shows it pretends having found and corrected 22926 CSUM errors ??!?!?!?!!???

Quoted 8 lines

This is a rather new HDD, in perfect shape (SMART all OK, never reallocated a single sector, less than 200 hours total runtime...)

WTF ?!?

I've cancelled scrub for now, until I get further understanding of what can be happening...

So this is probably because of the extent tree corruption you had, it's just cleaning things up and you should be fine once it finishes. Thanks,

Josef

Signature · 4 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
      --- Eighth Army Push Bottles Up Germans -- WWII newspaper ---      
                     headline (possibly apocryphal)                      
#3 lore raw
In reply to Swâmi Petaramesh
Quoted 19 lines · Swâmi Petaramesh

Hi Josef,

Quoted 2 lines · Josef Bacik

So this is probably because of the extent tree corruption you had, it's just cleaning things up and you should be fine once it finishes. Thanks,

Er... It's on a different machine !

Current (at the time I write) status is :

# btrfs scrub status /
scrub status for 346b81b2-0735-4c4d-a137-1995bc78ad70
        scrub resumed at Fri Mar 29 11:52:43 2013 and finished after
7470 seconds
        total bytes scrubbed: 231.96GB with 149691 errors
        error details: csum=149691
        corrected errors: 149691, uncorrectable errors: 0, unverified
errors: 0
Quoted 8 lines

I have to say that scrub completely froze the machine at least 4 times (disk had ceased activity and any command that would imply a disk access would hang forever), but at least after a (quite brutal) reboot it could be resumed...

The only thing about this FS is that it had been imaged, then restored, using partclone.btrfs (which itself is supposed to use the BTRFS libraries).

This is where I go "AHA!" and just assume that it wasn't our fault ;).

Quoted 3 lines

I have a screenshot of "last thing I saw when it hanged", I can upload it somewhere, should it be relevant...

Screenshots are welcome, I have no doubt scrub is fixing actual problems, but it definitely shouldn't be hanging the box so I'd like to get those fixed if possible. Sysrq+w during hangs are very usefull but may be too much output for screenshots, netconsole works very nicely for this

http://fedoraproject.org/wiki/Netconsole

Thanks,

Josef

#4 lore raw
Quoted 1 line · Josef Bacik

Screenshots are welcome,

I posted one to http://dl.free.fr/jsRQ8JXZh (use your email address or the list's one to fetch it)

It may or may not be very interesting, but that's all I got.

Quoted 1 line

I have no doubt scrub is fixing actual problems

Looks like it actually is. First times it hanged, I restarted it from the start and it wasn't finding errors during the first GBs anymore, so I assumed it has fixed them in the previous pass (even though it eventually crashed the disk susbsystem).

Quoted 4 lines

, but it
definitely shouldn't be hanging the box so I'd like to get those fixed if possible. Sysrq+w during hangs are very usefull but may be too much output for screenshots, netconsole works very nicely for this

I'll restart the complete scrub (now everything is supposedly fixed...?) and let you know if it hangs agan and I can put the hand on something.

Kind regards.

Signature · 2 lines
Swâmi Petaramesh <[email protected]> http://petaramesh.org PGP 9076E32E
Ne cherchez pas : Je ne suis pas sur Facebook.
#6 lore raw
In reply to Swâmi Petaramesh
Quoted 24 lines · Swâmi Petaramesh

Hi Josef,

Quoted 2 lines · Josef Bacik

So this is probably because of the extent tree corruption you had, it's just cleaning things up and you should be fine once it finishes. Thanks,

Er... It's on a different machine !

Current (at the time I write) status is :

# btrfs scrub status /
scrub status for 346b81b2-0735-4c4d-a137-1995bc78ad70
        scrub resumed at Fri Mar 29 11:52:43 2013 and finished after
7470 seconds
        total bytes scrubbed: 231.96GB with 149691 errors
        error details: csum=149691
        corrected errors: 149691, uncorrectable errors: 0, unverified
errors: 0

I have to say that scrub completely froze the machine at least 4 times (disk had ceased activity and any command that would imply a disk access would hang forever), but at least after a (quite brutal) reboot it could be resumed...

Quoted 6 lines

The only thing about this FS is that it had been imaged, then restored, using partclone.btrfs (which itself is supposed to use the BTRFS libraries).

I have a screenshot of "last thing I saw when it hanged", I can upload it somewhere, should it be relevant...

Actually instead of netconsole we have an awesome service provided by Carey, you can just do

nc cwillu.com 10101 < /dev/kmsg

after you've run sysrq+w and then reply with the URL it spits out. Thanks,

Josef

#7 lore raw
Quoted 4 lines

Actually instead of netconsole we have an awesome service provided by Carey, you can just do

nc cwillu.com 10101 < /dev/kmsg

... at a root prompt.

Quoted 1 line

after you've run sysrq+w and then reply with the URL it spits out. Thanks,

#8 lore raw
In reply to Josef Bacik
Quoted 1 line · Josef Bacik

after you've run sysrq+w and then reply with the URL it spits out. Thanks,

I'm afraid I won't be able to do this this afternoon : I also need to work on my machine ;-) so for now I will avoid to restart a scrub that would possibly crash it once more...

I'll hopefully be back on this soon.

Kind regards.

Signature · 2 lines
Swâmi Petaramesh <[email protected]> http://petaramesh.org PGP 9076E32E
Ne cherchez pas : Je ne suis pas sur Facebook.
#9 lore raw
Quoted 8 lines · Swâmi Petaramesh
Quoted 1 line · Josef Bacik

after you've run sysrq+w and then reply with the URL it spits out. Thanks,

I'm afraid I won't be able to do this this afternoon : I also need to work on my machine ;-) so for now I will avoid to restart a scrub that would possibly crash it once more...

I'll hopefully be back on this soon.

Yeah that picture was enough, I see what's going on, I'll send a patch. Thanks,

Josef