size 2.73TiB used 240.97GiB after balance

DISCUSSION Hendrik Friedel 4 participants 7 messages Jul 6, 2015 Btrfs file system

Share & tools

Hendrik Friedel

author

2015-07-06 19:20 UTC #0 lore raw

Hello,

I started with a raid1:
devid 1 size 2.73TiB used 2.67TiB path /dev/sdd
devid 2 size 2.73TiB used 2.67TiB path /dev/sdb
Then I added a third device, /dev/sdc1 and a balance
btrfs balance start -dconvert=raid5 -mconvert=raid5 /mnt/__Complete_Disk/

Now the file-system looks like this:
Total devices 3 FS bytes used 4.68TiB
devid 1 size 2.73TiB used 2.67TiB path /dev/sdd
devid 2 size 2.73TiB used 2.67TiB path /dev/sdb
devid 3 size 2.73TiB used 240.97GiB path /dev/sdc1

I am surprised by the 240.97GiB...

In the syslog and dmesg I find several:

[108274.415499] btrfs_dev_stat_print_on_error: 8 callbacks suppressed
[108279.840334] btrfs_dev_stat_print_on_error: 12 callbacks suppressed

What's wrong here?

Regards,
Hendrik

---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus

Hendrik Friedel

author

2015-07-06 19:44 UTC #1 lore raw

Hello,

ok, sdc seems to have failed (sorry, I checked only sdd and sdb SMART values, as sdc is brand new. Maybe a bad assumption, from my side.

I have mounted the device
mount -o recovery,ro

So, what should I do now:
btrfs device delete /dev/sdc /mnt

mount -o degraded /dev/sdb /mnt
btrfs device delete missing /mnt

I do have a backup of the most valuable data.
But if you consider one of the above options risky, I might better get a new drive before, but this might take a couple of days (in which sdc could further degrade). What is your recommendation?

Regards,
Hendrik

---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus

Hugo Mills

2015-07-06 19:49 UTC #2 lore raw

Quoted 14 lines · Hendrik Friedel

Hello,
ok, sdc seems to have failed (sorry, I checked only sdd and sdb SMART values, as sdc is brand new. Maybe a bad assumption, from my side.
I have mounted the device
mount -o recovery,ro
So, what should I do now:
btrfs device delete /dev/sdc /mnt
or

Quoted 8 lines

mount -o degraded /dev/sdb /mnt
btrfs device delete missing /mnt
I do have a backup of the most valuable data.
But if you consider one of the above options risky, I might better get a new drive before, but this might take a couple of days (in which sdc could further degrade).
What is your recommendation?

Physically remove the device from the array, mount with -o
degraded, optionally add the new device, and run a balance.

Hugo.

Signature · 4 lines

Hugo Mills             | "I lost my leg in 1942. Some bastard stole it in a
hugo@... carfax.org.uk | pub in Pimlico."
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

Donald Pearson

2015-07-06 20:01 UTC #3 lore raw

Based on my experience Hugo's advice is critical, get the bad drive out of the pool when in raid56 and do not try to replace or delete it while it's still attached and recognized.

If you add a new device, mount degraded and rebalance. If you don't, mount degraded then device delete missing.

Quoted 34 lines · Hugo Mills

Quoted 14 lines · Hendrik Friedel
Hello,
ok, sdc seems to have failed (sorry, I checked only sdd and sdb SMART values, as sdc is brand new. Maybe a bad assumption, from my side.
I have mounted the device
mount -o recovery,ro
So, what should I do now:
btrfs device delete /dev/sdc /mnt
or
Quoted 8 lines
mount -o degraded /dev/sdb /mnt
btrfs device delete missing /mnt
I do have a backup of the most valuable data.
But if you consider one of the above options risky, I might better get a new drive before, but this might take a couple of days (in which sdc could further degrade).
What is your recommendation?
Physically remove the device from the array, mount with -o
degraded, optionally add the new device, and run a balance.
Hugo.
--
Hugo Mills             | "I lost my leg in 1942. Some bastard stole it in a
hugo@... carfax.org.uk | pub in Pimlico."
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

Omar Sandoval

2015-07-06 20:53 UTC #4 lore raw

Quoted 7 lines · Donald Pearson

Based on my experience Hugo's advice is critical, get the bad drive out of the pool when in raid56 and do not try to replace or delete it while it's still attached and recognized.
If you add a new device, mount degraded and rebalance. If you don't, mount degraded then device delete missing.

Watch out, replacing a missing device in RAID 5/6 currently doesn't work and will cause a kernel BUG(). See my patch series here:
http://www.spinics.net/lists/linux-btrfs/msg44874.html

Signature · 1 line

Omar

Hendrik Friedel

author

2015-07-06 21:12 UTC #5 lore raw

Hello,

oh dear, I fear I am in trouble:
recovery-mounted, I tried to save some data, but the system hung. So I re-booted and sdc is now physically disconnected.

Label: none uuid: b4a6cce6-dc9c-4a13-80a4-ed6bc5b40bb8
Total devices 3 FS bytes used 4.67TiB
devid 1 size 2.73TiB used 2.67TiB path /dev/sdc
devid 2 size 2.73TiB used 2.67TiB path /dev/sdb
*** Some devices missing

I try to mount the rest again:
mount -o recovery,ro /dev/sdb /mnt/__Complete_Disk
mount: wrong fs type, bad option, bad superblock on /dev/sdb,
missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so

root@homeserver:~# dmesg | tail

[  447.059275] BTRFS info (device sdc): enabling auto recovery
[  447.059280] BTRFS info (device sdc): disk space caching is enabled
[  447.086844] BTRFS: failed to read chunk tree on sdc
[  447.110588] BTRFS: open_ctree failed
[  474.496778] BTRFS info (device sdc): enabling auto recovery
[  474.496781] BTRFS info (device sdc): disk space caching is enabled
[  474.519005] BTRFS: failed to read chunk tree on sdc
[  474.540627] BTRFS: open_ctree failed

mount -o degraded,ro /dev/sdb /mnt/__Complete_Disk Does work now though.

So, how can I remove the reference to the failed disk and check the data for consistency (scrub I suppose, but is it safe?)?

Regards,
Hendrik

Quoted 13 lines · Omar Sandoval

Quoted 7 lines · Donald Pearson
Based on my experience Hugo's advice is critical, get the bad drive out of the pool when in raid56 and do not try to replace or delete it while it's still attached and recognized.
If you add a new device, mount degraded and rebalance. If you don't, mount degraded then device delete missing.
Watch out, replacing a missing device in RAID 5/6 currently doesn't work and will cause a kernel BUG(). See my patch series here:
http://www.spinics.net/lists/linux-btrfs/msg44874.html

Signature · 9 lines

Hendrik Friedel
Auf dem Brink 12
28844 Weyhe
Tel. 04203 8394854
Mobil 0178 1874363

---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus

Donald Pearson

2015-07-06 21:49 UTC #6 lore raw

If you can mount it RO, first thing to do is back up any data that you care about.

According to the bug that Omar posted you should not try a device replace and you should not try a scrub with a missing device.

You may be able to just do a device delete missing, then separately do a device add of a new drive, or rebalance back in to raid1.

Quoted 72 lines · Hendrik Friedel

Hello,
oh dear, I fear I am in trouble:
recovery-mounted, I tried to save some data, but the system hung. So I re-booted and sdc is now physically disconnected.
Label: none uuid: b4a6cce6-dc9c-4a13-80a4-ed6bc5b40bb8
Total devices 3 FS bytes used 4.67TiB
devid 1 size 2.73TiB used 2.67TiB path /dev/sdc
devid 2 size 2.73TiB used 2.67TiB path /dev/sdb
*** Some devices missing
I try to mount the rest again:
mount -o recovery,ro /dev/sdb /mnt/__Complete_Disk
mount: wrong fs type, bad option, bad superblock on /dev/sdb,
missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so
root@homeserver:~# dmesg | tail
[  447.059275] BTRFS info (device sdc): enabling auto recovery
[  447.059280] BTRFS info (device sdc): disk space caching is enabled
[  447.086844] BTRFS: failed to read chunk tree on sdc
[  447.110588] BTRFS: open_ctree failed
[  474.496778] BTRFS info (device sdc): enabling auto recovery
[  474.496781] BTRFS info (device sdc): disk space caching is enabled
[  474.519005] BTRFS: failed to read chunk tree on sdc
[  474.540627] BTRFS: open_ctree failed
mount -o degraded,ro /dev/sdb /mnt/__Complete_Disk Does work now though.
So, how can I remove the reference to the failed disk and check the data for consistency (scrub I suppose, but is it safe?)?
Regards,
Hendrik
Quoted 7 lines · Omar Sandoval
Quoted 5 lines · Donald Pearson
Based on my experience Hugo's advice is critical, get the bad drive out of the pool when in raid56 and do not try to replace or delete it while it's still attached and recognized.
Quoted 8 lines
Quoted 3 lines
If you add a new device, mount degraded and rebalance. If you don't, mount degraded then device delete missing.
Watch out, replacing a missing device in RAID 5/6 currently doesn't work and will cause a kernel BUG(). See my patch series here:
http://www.spinics.net/lists/linux-btrfs/msg44874.html
--
Hendrik Friedel
Auf dem Brink 12
28844 Weyhe
Tel. 04203 8394854
Mobil 0178 1874363
---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus