File system stuck in scrub
Hello,
I started a scrub of one of my btrfs filesystem and then had to restart the system. systemctl restart seemed to terminate all processes, but then got stuck at the end. The disk activity led was still flashing rapidly at that point, so I assume that the active scrub was preventing the reboot (is that a bug or a feature?).
In any case, I could not wait for that so I power cycled. But now my file system seems to be stuck in a scrub that can neither be completed nor cancelled:
$ sudo btrfs scrub status /home/nikratio/
scrub status for 8742472d-a9b0-4ab6-b67a-5d21f14f7a38
scrub started at Sun Aug 10 18:36:43 2014, running for 1562 seconds
total bytes scrubbed: 209.97GiB with 0 errors$ date Sun Aug 10 22:00:44 PDT 2014
$ sudo btrfs scrub cancel /home/nikratio/ ERROR: scrub cancel failed on /home/nikratio/: not running
$ sudo btrfs scrub start /home/nikratio/ ERROR: scrub is already running. To cancel use 'btrfs scrub cancel /home/nikratio/'. To see the status use 'btrfs scrub status [-d] /home/nikratio/'.
Note that the scrub was started more than 3 hours ago, but claims to have been running for only 1562 seconds.
I then figured that maybe I need to run btrfsck. This gave the following output:
checking extents
checking free space cache
checking fs roots
root 5 inode 3149791 errors 400, nbytes wrong
root 5 inode 3150233 errors 400, nbytes wrong
root 5 inode 3150238 errors 400, nbytes wrong
[102 similar lines]
Checking filesystem on /dev/mapper/vg0-nikratio_crypt
UUID: 8742472d-a9b0-4ab6-b67a-5d21f14f7a38
free space inode generation (0) did not match free space cache generation (161262) free space inode generation (0) did not match free space cache generation (75485) free space inode generation (0) did not match free space cache generation (79599) free space inode generation (0) did not match free space cache generation (72280) free space inode generation (0) did not match free space cache generation (79599) free space inode generation (0) did not match free space cache generation (25866) free space inode generation (0) did not match free space cache generation (12255) free space inode generation (0) did not match free space cache generation (72521) free space inode generation (0) did not match free space cache generation (161286) free space inode generation (0) did not match free space cache generation (28716) free space inode generation (0) did not match free space cache generation (161481) found 216444746042 bytes used err is 1
total csum bytes: 383160676
total tree bytes: 875753472
total fs tree bytes: 284246016
total extent tree bytes: 69320704
btree space waste bytes: 205021777
file data blocks allocated: 3701556121600
referenced 388107321344
Btrfs v3.14.1
So nothing about the scrub, but apparently some other errors.
Can someone tell me:
* Should I be able to restart while a scrub is in progress, or is that deliberately prevented by btrfs?
* How can I resume or cancel the scrub?
* Is it more risky to leave the above errors uncorrected, or to run btrfsck with --repair?
I'm using kernel 3.14.
Thanks!
-Nikolaus
Signature · 4 lines
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«Quoted 5 lines · Nikolaus Rath
I started a scrub of one of my btrfs filesystem and then had to restart the system.
systemctl restartseemed to terminate all processes, but then got stuck at the end. The disk activity led was still flashing rapidly at that point, so I assume that the active scrub was preventing the reboot (is that a bug or a feature?).
Shouldn't have stopped it.
Quoted 15 lines
In any case, I could not wait for that so I power cycled. But now my file system seems to be stuck in a scrub that can neither be completed nor cancelled:
$ sudo btrfs scrub status /home/nikratio/ scrub status for 8742472d-a9b0-4ab6-b67a-5d21f14f7a38 scrub started at Sun Aug 10 18:36:43 2014, running for 1562 seconds total bytes scrubbed: 209.97GiB with 0 errors$ date Sun Aug 10 22:00:44 PDT 2014$ sudo btrfs scrub cancel /home/nikratio/ ERROR: scrub cancel failed on /home/nikratio/: not running
Quoted 7 lines
$ sudo btrfs scrub start /home/nikratio/ ERROR: scrub is already running. To cancel use 'btrfs scrub cancel /home/nikratio/'. To see the status use 'btrfs scrub status [-d] /home/nikratio/'.Note that the scrub was started more than 3 hours ago, but claims to have been running for only 1562 seconds.
This is a regrettably common problem -- fortunately with a simple
solution. The userspace scrub monitor died in the reboot, leaving the status file present. If you delete the status file, which is in /var/lib/btrfs/, that should allow you to start a new scrub.
Quoted 13 lines
I then figured that maybe I need to run btrfsck. This gave the following output:
checking extents
checking free space cache
checking fs roots
root 5 inode 3149791 errors 400, nbytes wrong
root 5 inode 3150233 errors 400, nbytes wrong
root 5 inode 3150238 errors 400, nbytes wrong
[102 similar lines]
Checking filesystem on/dev/mapper/vg0-nikratio_crypt
UUID: 8742472d-a9b0-4ab6-b67a-5d21f14f7a38
free space inode generation (0) did not match free space cache generation (161262)
[snip]
Quoted 11 lines
found 216444746042 bytes used err is 1
total csum bytes: 383160676
total tree bytes: 875753472
total fs tree bytes: 284246016
total extent tree bytes: 69320704
btree space waste bytes: 205021777
file data blocks allocated: 3701556121600
referenced 388107321344
Btrfs v3.14.1So nothing about the scrub, but apparently some other errors.
The free space inode generation errors are harmless. The wrong
nbytes is probably not horrifically damaging, but I don't know so much about that one.
Quoted 4 lines
Can someone tell me:
* Should I be able to restart while a scrub is in progress, or is that deliberately prevented by btrfs?
Restart the machine? Yes.
Quoted 1 line
* How can I resume or cancel the scrub?
It's probably simply not running -- see above.
Quoted 2 lines
* Is it more risky to leave the above errors uncorrected, or to run btrfsck with --repair?
I would, I think, leave them.
Hugo.
Signature · 4 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- We are all lying in the gutter, but some of us are looking ---
at the stars. Hi,
Quoted 10 lines · Nikolaus Rath
Hello,
I started a scrub of one of my btrfs filesystem and then had to restart
the system.systemctl restartseemed to terminate all processes, but
then got stuck at the end. The disk activity led was still flashing rapidly at that point, so I assume that the active scrub was preventing
the reboot (is that a bug or a feature?).
This sounds like a bug - I know that e.g. the rebalance operation is designed so that you can shutdown/reboot during the operation, and it will complete following a reboot. But I'm not familiar with the code in question.
Quoted 14 lines
In any case, I could not wait for that so I power cycled. But now my file system seems to be stuck in a scrub that can neither be completed
nor cancelled:$ sudo btrfs scrub status /home/nikratio/ scrub status for 8742472d-a9b0-4ab6-b67a-5d21f14f7a38 scrub started at Sun Aug 10 18:36:43 2014, running for 1562 seconds total bytes scrubbed: 209.97GiB with 0 errors$ date Sun Aug 10 22:00:44 PDT 2014
Quoted 7 lines
$ sudo btrfs scrub cancel /home/nikratio/ ERROR: scrub cancel failed on /home/nikratio/: not running$ sudo btrfs scrub start /home/nikratio/ ERROR: scrub is already running. To cancel use 'btrfs scrub cancel /home/nikratio/'. To see the status use 'btrfs scrub status [-d] /home/nikratio/'.
My guess is that this is a mismatch between some state stored by the userspace tools and the state in the kernel. One of the things you can try is to delete the files /var/lib/btrfs/scrub.status.* - that will force the btrfs tools to get the current status from the kernel (you will lose some statistics and scrub history.)
Running 'btrfs scrub status /home/nikratio/' after this should simply say 'no stats available', and you can start a new scrub later if you like.
Quoted 3 lines
I then figured that maybe I need to run btrfsck. This gave the following
output:
As long as you didn't use --repair, this shouldn't break anything... Note that btrfs has to be run on an *unmounted* filesystem to give useful results.
Quoted 2 lines
* Is it more risky to leave the above errors uncorrected, or to run btrfsck with --repair?
There probably aren't any issues on the filesystem that the runtime btrfs code can't handle. Don't run with --repair, at least not yet.
Quoted 8 lines
I'm using kernel 3.14.
Thanks!
-Nikolaus
Signature · 1 line
Calvin Walton <[email protected]>
Quoted 9 lines · Calvin Walton
Quoted 4 lines
$ sudo btrfs scrub start /home/nikratio/ ERROR: scrub is already running. To cancel use 'btrfs scrub cancel /home/nikratio/'. To see the status use 'btrfs scrub status [-d] /home/nikratio/'.My guess is that this is a mismatch between some state stored by the userspace tools and the state in the kernel. One of the things you can try is to delete the files
/var/lib/btrfs/scrub.status.*- that will force the btrfs tools to get the current status from the kernel (you will lose some statistics and scrub history.)
No need to really delete it, just changing one character will do :)
Cheers,
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901