Skip to contentSkip to conversation

deleting a dead device

DISCUSSION Russell Coker 3 participants 5 messages Sep 21, 2014 – Sep 24, 2014 Btrfs file system

lore
Share & tools
lore
#0 lore raw

On a system running the Debian 3.14.15-2 kernel I added a new drive to a RAID-1 array. My aim was to add a device and remove one of the old devices.

Sep 21 11:26:51 server kernel: [2070145.375221] BTRFS: lost page write due to 
I/O error on /dev/sdc3
Sep 21 11:26:51 server kernel: [2070145.375225] BTRFS: bdev /dev/sdc3 errs: wr 
269, rd 0, flush 0, corrupt 0, gen 0
Sep 21 11:27:21 server kernel: [2070175.517691] BTRFS: lost page write due to 
I/O error on /dev/sdc3
Sep 21 11:27:21 server kernel: [2070175.517699] BTRFS: bdev /dev/sdc3 errs: wr 
270, rd 0, flush 0, corrupt 0, gen 0
Sep 21 11:27:21 server kernel: [2070175.517712] BTRFS: lost page write due to 
I/O error on /dev/sdc3
Sep 21 11:27:21 server kernel: [2070175.517715] BTRFS: bdev /dev/sdc3 errs: wr 
271, rd 0, flush 0, corrupt 0, gen 0
Sep 21 11:27:51 server kernel: [2070205.665947] BTRFS: lost page write due to 
I/O error on /dev/sdc3
Sep 21 11:27:51 server kernel: [2070205.665955] BTRFS: bdev /dev/sdc3 errs: wr 
272, rd 0, flush 0, corrupt 0, gen 0
Sep 21 11:27:51 server kernel: [2070205.665967] BTRFS: lost page write due to 
I/O error on /dev/sdc3
Sep 21 11:27:51 server kernel: [2070205.665971] BTRFS: bdev /dev/sdc3 errs: wr 

273, rd 0, flush 0, corrupt 0, gen 0

Anyway the new drive turned out to have some errors, writes failed and I've got a heap of errors such as the above. The errors started immediately after adding the drive and the system wasn't actively writing to the filesystem. So very few (if any) writes made it to the device.

# btrfs device delete /dev/sdc3 /
ERROR: error removing the device '/dev/sdc3' - Invalid argument

It seems that I can't remove the device because removing requires writing.

# btrfs device delete /dev/sdc3 /
ERROR: error removing the device '/dev/sdc3' - No such file or directory
# btrfs device stats /
[/dev/sda3].write_io_errs   0
[/dev/sda3].read_io_errs    0
[/dev/sda3].flush_io_errs   0
[/dev/sda3].corruption_errs 57
[/dev/sda3].generation_errs 0
[/dev/sdb3].write_io_errs   0
[/dev/sdb3].read_io_errs    0
[/dev/sdb3].flush_io_errs   0
[/dev/sdb3].corruption_errs 0
[/dev/sdb3].generation_errs 0
[/dev/sdc3].write_io_errs   267
[/dev/sdc3].read_io_errs    0
[/dev/sdc3].flush_io_errs   0
[/dev/sdc3].corruption_errs 0
[/dev/sdc3].generation_errs 0

The drive is attached by USB so I turned off the USB device and then got the above result. So it still seems impossible to remove the device even though it's physically not present. I've connected a new USB disk which is now /dev/sdd, so it seems that BTRFS is keeping the name /dev/sdc locked.

Should there be a way to fix this without rebooting or anything?

Also as an aside, while the stats about write errors are useful, in this case it would be really good if there was a count of successful writes, it would be useful to know if the successful write count was close to 0. My understanding of the BTRFS design is that there would be no performance penalty for adding counts of the number of successful reads and writes to the superblock. Could this be done?

Signature · 2 lines
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/
#1 lore raw

Russell Coker posted on Sun, 21 Sep 2014 11:39:17 +1000 as excerpted:

Quoted 3 lines

On a system running the Debian 3.14.15-2 kernel I added a new drive to a RAID-1 array. My aim was to add a device and remove one of the old devices.

That's an old kernel and presumably an old btrfs-progs. Quite a number of device management fixes have gone in recently, and you'd likely not be in quite that predicament were you running a current kernel (make sure it's 3.16.2+ or 3.17-rc2+ to get the fix for the workqueues bug that affected 3.15 and thru 3.16.1 and 3.17-rc1).

And the recommended way to handle a device replace now would be btrfs replace, doing the add and delete in a single (albeit possibly long) step instead of separately.

[snip most of the problem description]

Quoted 7 lines

The drive is attached by USB so I turned off the USB device and then got the above result. So it still seems impossible to remove the device even though it's physically not present. I've connected a new USB disk which is now /dev/sdd, so it seems that BTRFS is keeping the name /dev/sdc locked.

Should there be a way to fix this without rebooting or anything?

Did you try btrfs device delete missing? It's documented on the wiki but apparently not yet on the manpage. According to the wiki that deletes the first device that was in the metadata but not found when booting, so you may have to reboot to do it, but it should work. Tho with the recent stale-devices fixes, were that a current kernel you may not actually have to reboot to have delete missing work. But you probably will on 3.14, and of course to upgrade kernels you'd have to reboot anyway, so...

Quoted 6 lines

Also as an aside, while the stats about write errors are useful, in this case it would be really good if there was a count of successful writes, it would be useful to know if the successful write count was close to 0.
My understanding of the BTRFS design is that there would be no
performance penalty for adding counts of the number of successful reads and writes to the superblock. Could this be done?

Not necessarily for reads, consider the case when the filesystem is read- only as my btrfs root filesystem is by default -- lots of reads but likely no writes and no super-block updates for the entire uptime. But I believe you're correct for writes, since they'd ultimately update the superblocks anyway.

Signature · 3 lines
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman
#2 lore raw
Quoted 10 lines · Duncan

Russell Coker posted on Sun, 21 Sep 2014 11:39:17 +1000 as excerpted:

Quoted 3 lines

On a system running the Debian 3.14.15-2 kernel I added a new drive to a RAID-1 array. My aim was to add a device and remove one of the old devices.

That's an old kernel and presumably an old btrfs-progs. Quite a number of device management fixes have gone in recently, and you'd likely not be in quite that predicament were you running a current kernel (make sure it's 3.16.2+ or 3.17-rc2+ to get the fix for the workqueues bug that affected 3.15 and thru 3.16.1 and 3.17-rc1).

3.16.2 is in Debian, I'm in the process of upgrading to it.

Quoted 3 lines

And the recommended way to handle a device replace now would be btrfs replace, doing the add and delete in a single (albeit possibly long) step instead of separately.

I'm changing from a 500G single filesystem to a 200G RAID-1 (there's only 150G of data). The change from 500G to 200G can't be done with a replace as a replace requires an equal or greater size.

I did a 2 step process, add/delete to go to a 200G USB attached device for half the array and then replace to go from 200G on USB to 200G internal.

Quoted 10 lines
Quoted 7 lines

The drive is attached by USB so I turned off the USB device and then got the above result. So it still seems impossible to remove the device even though it's physically not present. I've connected a new USB disk which is now /dev/sdd, so it seems that BTRFS is keeping the name /dev/sdc locked.

Should there be a way to fix this without rebooting or anything?

Did you try btrfs device delete missing? It's documented on the wiki but apparently not yet on the manpage.

I did that after rebooting. It didn't occur to me to try a "missing" operation when the drive really wasn't missing.

Quoted 3 lines

According to the wiki that deletes
the first device that was in the metadata but not found when booting, so you may have to reboot to do it, but it should work.

That would be a bug. There's no reason a reboot should be required if we can remove a drive and add a new one with the kernel recognising it. Hot-swap disks aren't any sort of new feature.

Quoted 4 lines

Tho with the recent
stale-devices fixes, were that a current kernel you may not actually have to reboot to have delete missing work. But you probably will on 3.14, and of course to upgrade kernels you'd have to reboot anyway, so...

Yes a reboot was needed anyway. But I'd have liked to delay that.

Quoted 6 lines
Quoted 6 lines

Also as an aside, while the stats about write errors are useful, in this case it would be really good if there was a count of successful writes, it would be useful to know if the successful write count was close to 0.

My understanding of the BTRFS design is that there would be no

Quoted 8 lines
Quoted 2 lines

performance penalty for adding counts of the number of successful reads and writes to the superblock. Could this be done?

Not necessarily for reads, consider the case when the filesystem is read- only as my btrfs root filesystem is by default -- lots of reads but likely no writes and no super-block updates for the entire uptime. But I believe you're correct for writes, since they'd ultimately update the superblocks anyway.

For the case of a read-only filesystem it's OK to skip read stats. It would also be a bad idea to update read stats without writing data. But there's no reason why read stats couldn't be accumulated in-memory and written out the next time something was written to disk. That would give a slight inaccuracy in the case where there was a power failure after some period of reading without writing, but that's an unusual corner case.

Signature · 2 lines
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/
#3 lore raw
In reply to Russell Coker
Quoted 3 lines · Russell Coker

Anyway the new drive turned out to have some errors, writes failed and I've got a heap of errors such as the above.

I'm curious if smartctl -t conveyance reveals any problems, it's not a full surface test but is designed to be a test for (typical?) problems drives have due to shipment damage, and doesn't take very long.

Quoted 8 lines

The errors started immediately after
adding the drive and the system wasn't actively writing to the filesystem. So very few (if any) writes made it to the device.

# btrfs device delete /dev/sdc3 /
ERROR: error removing the device '/dev/sdc3' - Invalid argument

It seems that I can't remove the device because removing requires writing.

What kernel message do you get associated with this? Try using the devid instead of /dev/.

For future reference, btrfs replace start is better to use than add+delete. It's an optimization but it also makes it possible to ignore the device being replaced for reads; and you can also get a status on the progress with "btrfs replace status". And it looks like it does some additional error checking.

Quoted 20 lines
# btrfs device delete /dev/sdc3 /
ERROR: error removing the device '/dev/sdc3' - No such file or directory
# btrfs device stats /
[/dev/sda3].write_io_errs   0
[/dev/sda3].read_io_errs    0
[/dev/sda3].flush_io_errs   0
[/dev/sda3].corruption_errs 57
[/dev/sda3].generation_errs 0
[/dev/sdb3].write_io_errs   0
[/dev/sdb3].read_io_errs    0
[/dev/sdb3].flush_io_errs   0
[/dev/sdb3].corruption_errs 0
[/dev/sdb3].generation_errs 0
[/dev/sdc3].write_io_errs   267
[/dev/sdc3].read_io_errs    0
[/dev/sdc3].flush_io_errs   0
[/dev/sdc3].corruption_errs 0
[/dev/sdc3].generation_errs 0
Quoted 4 lines

The drive is attached by USB so I turned off the USB device and then got the above result. So it still seems impossible to remove the device even though it's physically not present. I've connected a new USB disk which is now /dev/sdd, so it seems that BTRFS is keeping the name /dev/sdc locked.

Pretty sure kernel assignment is major:minor, and anything under /dev/ is udev. What do you get for btrfs fi show

Unfortunately this won't show devid for missing devices, so you might have to infer this. But you can use btrfs replace start -r <devid> /dev/sddX <mountpoint>

Quoted 4 lines

Also as an aside, while the stats about write errors are useful, in this case it would be really good if there was a count of successful writes, it would be useful to know if the successful write count was close to 0.

I think this is for other tools. Btrfs is a file system its responsible for the integrity of the data it writes, I don't think it's responsible for prequalifying drives.

Even a simple dd if=/dev/zero of=/dev/sdc bs=64k count=1600 will write out 100MB, and dmesg will show if there are any controller or drive problems on writes. You may have to do more than 100MB for problems to show up but you get the idea.

You can also use badblocks -swv (progress, destructive write/read, verbose) which will also show writes the drive says succeeded but are actually corrupt.

Use smartctl -t conveyance/short/long to isolate the drive mechanism itself. This obviously doesn't test writes.

Consumer drives should fairly quickly report persistent write failures, which libata will report in dmesg. A common problem though, is they try to do reads much longer than the linux SCSI layer default timeout. Either the SCT ERC timeout of the drive needs to be reduced below 30 seconds; or the linux SCSI layer timeout needs to be raised above the drive SCT ERC timeout. Otherwise the drive keeps trying to do reads, the linux SCSI layer gives up on the non-communicating drive (which is busy recovering) and resets the link. Now the read error doesn't actually happen, doesn't report the offending sector, and Btrfs can't fix the problem.

Chris Murphy

#4 lore raw
Quoted 8 lines · On Sun, 21 Sep 2014 11:05:46 Chris Murphy wrote:
Quoted 3 lines · Russell Coker

Anyway the new drive turned out to have some errors, writes failed and I've
got a heap of errors such as the above.

I'm curious if smartctl -t conveyance reveals any problems, it's not a full surface test but is designed to be a test for (typical?) problems drives have due to shipment damage, and doesn't take very long.

Unfortunately I got to your message after sending the defective drive to e- waste. But I expect that any test that involves real reads/writes would report a failure (if the USB-SATA device supported passing them through) as the drive seemed to fail for everything.

Quoted 7 lines
Quoted 4 lines
# btrfs device delete /dev/sdc3 /
ERROR: error removing the device '/dev/sdc3' - Invalid argument

It seems that I can't remove the device because removing requires writing.

What kernel message do you get associated with this? Try using the devid instead of /dev/.

I'll keep that in mine.

device delete <dev> [<dev>...] <path>
Remove device(s) from a filesystem identified by <path>.

The man page has the above text which makes no mention of devid, so I think we need a documentation patch for this.

Quoted 5 lines

For future reference, btrfs replace start is better to use than add+delete. It's an optimization but it also makes it possible to ignore the device being replaced for reads; and you can also get a status on the progress with "btrfs replace status". And it looks like it does some additional error checking.

Oh yes, I've done this and it works well. However it doesn't work if the replacement is smaller than the device being replaced.

Quoted 7 lines
Quoted 3 lines

Also as an aside, while the stats about write errors are useful, in this case it would be really good if there was a count of successful writes, it would be useful to know if the successful write count was close to 0.

I think this is for other tools. Btrfs is a file system its responsible for the integrity of the data it writes, I don't think it's responsible for prequalifying drives.

I agree that it doesn't have to prequalify drives. But it should expose all data it has which can be of use to the sysadmin. After it was too late I realised that I could have used iostat to get stats for the block device. But it would still be nice to have stats from btrfs.

Also btrfs has to deal with the fact that drives may fail at any time. Admittedly I was using a drive I knew to be slightly sub-standard (I got it free because it gave an error in a client's RAID-Z array). But sometimes drives like that last for years, it's difficult to predict.

Quoted 4 lines

Even a simple dd if=/dev/zero of=/dev/sdc bs=64k count=1600 will write out 100MB, and dmesg will show if there are any controller or drive problems on writes. You may have to do more than 100MB for problems to show up but you get the idea.

True. But a drive can fail after 101MB.

Signature · 2 lines
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/