scrub "correcting" tons of errors ?
Hi there,
I've started "btrfs scrub start /" on one of my machines (Kernel 3.8.0-15 Ubuntu AMD64), which typically "behaves well" so I wasn't suspected any disk issue.
After having ran for only 165 seconds, "scrub status" shows it pretends having found and corrected 22926 CSUM errors ??!?!?!?!!???
This is a rather new HDD, in perfect shape (SMART all OK, never reallocated a single sector, less than 200 hours total runtime...)
WTF ?!?
I've cancelled scrub for now, until I get further understanding of what can be happening...
Signature · 2 lines
Swâmi Petaramesh <[email protected]> http://petaramesh.org PGP 9076E32E Ne cherchez pas : Je ne suis pas sur Facebook.
Quoted 9 lines · Swâmi Petaramesh
Hi there,
I've started "btrfs scrub start /" on one of my machines (Kernel 3.8.0-15 Ubuntu AMD64), which typically "behaves well" so I wasn't suspected any disk issue.
After having ran for only 165 seconds, "scrub status" shows it pretends having found and corrected 22926 CSUM errors ??!?!?!?!!???
Quoted 8 lines
This is a rather new HDD, in perfect shape (SMART all OK, never reallocated a single sector, less than 200 hours total runtime...)
WTF ?!?
I've cancelled scrub for now, until I get further understanding of what can be happening...
So this is probably because of the extent tree corruption you had, it's just cleaning things up and you should be fine once it finishes. Thanks,
Josef
Hi Josef,
Quoted 2 lines · Josef Bacik
So this is probably because of the extent tree corruption you had, it's just cleaning things up and you should be fine once it finishes. Thanks,
Er... It's on a different machine !
Current (at the time I write) status is :
# btrfs scrub status /
scrub status for 346b81b2-0735-4c4d-a137-1995bc78ad70
scrub resumed at Fri Mar 29 11:52:43 2013 and finished after
7470 seconds
total bytes scrubbed: 231.96GB with 149691 errors
error details: csum=149691
corrected errors: 149691, uncorrectable errors: 0, unverified
errors: 0I have to say that scrub completely froze the machine at least 4 times (disk had ceased activity and any command that would imply a disk access would hang forever), but at least after a (quite brutal) reboot it could be resumed...
The only thing about this FS is that it had been imaged, then restored, using partclone.btrfs (which itself is supposed to use the BTRFS libraries).
I have a screenshot of "last thing I saw when it hanged", I can upload it somewhere, should it be relevant...
Kind regards.
Signature · 2 lines
Swâmi Petaramesh <[email protected]> http://petaramesh.org PGP 9076E32E Ne cherchez pas : Je ne suis pas sur Facebook.
On that note, is btrfs doing automatic background scrubs of its own or do I have to use crontab to schedule scrubs?
Thanks!
Quoted 27 lines · Josef Bacik
Quoted 9 lines · Swāmi Petaramesh
Hi there,
I've started "btrfs scrub start /" on one of my machines (Kernel 3.8.0-15 Ubuntu AMD64), which typically "behaves well" so I wasn't suspected any disk issue.
After having ran for only 165 seconds, "scrub status" shows it pretends having found and corrected 22926 CSUM errors ??!?!?!?!!???
Quoted 8 lines
This is a rather new HDD, in perfect shape (SMART all OK, never reallocated a single sector, less than 200 hours total runtime...)
WTF ?!?
I've cancelled scrub for now, until I get further understanding of what can be happening...
So this is probably because of the extent tree corruption you had, it's just cleaning things up and you should be fine once it finishes. Thanks,
Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 2 lines · Harald Glatt
On that note, is btrfs doing automatic background scrubs of its own or do I have to use crontab to schedule scrubs?
If you want a full-disk scrub, you'll need to schedule it yourself
with cron (I run mine once a month). However, if a problem is detected during normal operation -- e.g. you read a piece of data and it's got bad checksums -- then the FS will fix it if it can, in the same way that it would with a scrub.
Hugo.
Quoted 26 lines
Thanks!
Quoted 23 lines · Josef Bacik
Quoted 9 lines · Swāmi Petaramesh
Hi there,
I've started "btrfs scrub start /" on one of my machines (Kernel 3.8.0-15 Ubuntu AMD64), which typically "behaves well" so I wasn't suspected any disk issue.
After having ran for only 165 seconds, "scrub status" shows it pretends having found and corrected 22926 CSUM errors ??!?!?!?!!???
Quoted 8 lines
This is a rather new HDD, in perfect shape (SMART all OK, never reallocated a single sector, less than 200 hours total runtime...)
WTF ?!?
I've cancelled scrub for now, until I get further understanding of what can be happening...
So this is probably because of the extent tree corruption you had, it's just cleaning things up and you should be fine once it finishes. Thanks,
Josef
Signature · 4 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- Eighth Army Push Bottles Up Germans -- WWII newspaper ---
headline (possibly apocryphal) Quoted 19 lines · Swâmi Petaramesh
Hi Josef,
Quoted 2 lines · Josef Bacik
So this is probably because of the extent tree corruption you had, it's just cleaning things up and you should be fine once it finishes. Thanks,
Er... It's on a different machine !
Current (at the time I write) status is :
# btrfs scrub status / scrub status for 346b81b2-0735-4c4d-a137-1995bc78ad70 scrub resumed at Fri Mar 29 11:52:43 2013 and finished after 7470 seconds total bytes scrubbed: 231.96GB with 149691 errors error details: csum=149691 corrected errors: 149691, uncorrectable errors: 0, unverified errors: 0
Quoted 8 lines
I have to say that scrub completely froze the machine at least 4 times (disk had ceased activity and any command that would imply a disk access would hang forever), but at least after a (quite brutal) reboot it could be resumed...
The only thing about this FS is that it had been imaged, then restored, using partclone.btrfs (which itself is supposed to use the BTRFS libraries).
This is where I go "AHA!" and just assume that it wasn't our fault ;).
Quoted 3 lines
I have a screenshot of "last thing I saw when it hanged", I can upload it somewhere, should it be relevant...
Screenshots are welcome, I have no doubt scrub is fixing actual problems, but it definitely shouldn't be hanging the box so I'd like to get those fixed if possible. Sysrq+w during hangs are very usefull but may be too much output for screenshots, netconsole works very nicely for this
http://fedoraproject.org/wiki/Netconsole
Thanks,
Josef
Quoted 1 line · Josef Bacik
Screenshots are welcome,
I posted one to http://dl.free.fr/jsRQ8JXZh (use your email address or the list's one to fetch it)
It may or may not be very interesting, but that's all I got.
Quoted 1 line
I have no doubt scrub is fixing actual problems
Looks like it actually is. First times it hanged, I restarted it from the start and it wasn't finding errors during the first GBs anymore, so I assumed it has fixed them in the previous pass (even though it eventually crashed the disk susbsystem).
Quoted 4 lines
, but it
definitely shouldn't be hanging the box so I'd like to get those fixed if possible. Sysrq+w during hangs are very usefull but may be too much output for screenshots, netconsole works very nicely for this
I'll restart the complete scrub (now everything is supposedly fixed...?) and let you know if it hangs agan and I can put the hand on something.
Kind regards.
Signature · 2 lines
Swâmi Petaramesh <[email protected]> http://petaramesh.org PGP 9076E32E Ne cherchez pas : Je ne suis pas sur Facebook.
Quoted 24 lines · Swâmi Petaramesh
Hi Josef,
Quoted 2 lines · Josef Bacik
So this is probably because of the extent tree corruption you had, it's just cleaning things up and you should be fine once it finishes. Thanks,
Er... It's on a different machine !
Current (at the time I write) status is :
# btrfs scrub status / scrub status for 346b81b2-0735-4c4d-a137-1995bc78ad70 scrub resumed at Fri Mar 29 11:52:43 2013 and finished after 7470 seconds total bytes scrubbed: 231.96GB with 149691 errors error details: csum=149691 corrected errors: 149691, uncorrectable errors: 0, unverified errors: 0I have to say that scrub completely froze the machine at least 4 times (disk had ceased activity and any command that would imply a disk access would hang forever), but at least after a (quite brutal) reboot it could be resumed...
Quoted 6 lines
The only thing about this FS is that it had been imaged, then restored, using partclone.btrfs (which itself is supposed to use the BTRFS libraries).
I have a screenshot of "last thing I saw when it hanged", I can upload it somewhere, should it be relevant...
Actually instead of netconsole we have an awesome service provided by Carey, you can just do
nc cwillu.com 10101 < /dev/kmsg
after you've run sysrq+w and then reply with the URL it spits out. Thanks,
Josef
Quoted 4 lines
Actually instead of netconsole we have an awesome service provided by Carey, you can just do
nc cwillu.com 10101 <
/dev/kmsg
... at a root prompt.
Quoted 1 line
after you've run sysrq+w and then reply with the URL it spits out. Thanks,
Quoted 1 line · Josef Bacik
Screenshots are welcome
This time I good a real nice kernel Ooops during scrub...
(use your email address or the list's one to fetch it)
Signature · 2 lines
Swâmi Petaramesh <[email protected]> http://petaramesh.org PGP 9076E32E Ne cherchez pas : Je ne suis pas sur Facebook.
Quoted 1 line · Josef Bacik
after you've run sysrq+w and then reply with the URL it spits out. Thanks,
I'm afraid I won't be able to do this this afternoon : I also need to work on my machine ;-) so for now I will avoid to restart a scrub that would possibly crash it once more...
I'll hopefully be back on this soon.
Kind regards.
Signature · 2 lines
Swâmi Petaramesh <[email protected]> http://petaramesh.org PGP 9076E32E Ne cherchez pas : Je ne suis pas sur Facebook.
Quoted 8 lines · Swâmi Petaramesh
Quoted 1 line · Josef Bacik
after you've run sysrq+w and then reply with the URL it spits out. Thanks,
I'm afraid I won't be able to do this this afternoon : I also need to work on my machine ;-) so for now I will avoid to restart a scrub that would possibly crash it once more...
I'll hopefully be back on this soon.
Yeah that picture was enough, I see what's going on, I'll send a patch. Thanks,
Josef