Skip to contentSkip to conversation

Errors after successful disk replace

DISCUSSION Emil Heimpel 2 participants 11 messages Oct 19, 2021 – Oct 26, 2021 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi all,

One of my drives of a raid 5 btrfs array failed (was dead completely) so I installed an identical replacement drive. The dead drive was devid 1 and the new drive /dev/sde. I used the following to replace the missing drive:

sudo btrfs replace start -B 1 /dev/sde1 /mnt/btrfsrepair/

and it completed successfully without any reported errors (took around 2 weeks though...).

I then tried to see my array with filesystem show, but it hung (or took longer than I wanted to wait), so I did a reboot.

It showed up after a reboot as followed:

Label: 'BlueButter'  uuid: 7e3378e6-da46-4a60-b9b8-1bcc306986e3 Total devices 6 FS bytes used 20.96TiB
        devid    0 size 7.28TiB used 5.46TiB path /dev/sde1 devid    2 size 7.28TiB used 5.46TiB path /dev/sdb1 devid    3 size 2.73TiB used 2.73TiB path /dev/sdg1 devid    4 size 2.73TiB used 2.73TiB path /dev/sdd1 devid    5 size 7.28TiB used 4.81TiB path /dev/sdf1 devid    6 size 7.28TiB used 5.33TiB path /dev/sdc1

I then tried to mount it, but it failed, so I run a readonly check and it reported the following problem:

[...]
[2/7] checking extents
ERROR: super total bytes 38007432437760 smaller than real device(s) size 46008994590720
ERROR: mounting this fs may fail for newer kernels
ERROR: this can be fixed by 'btrfs rescue fix-device-size'
[3/7] checking free space tree
[...]

So I followed that advice but got the following error:

sudo btrfs rescue fix-device-size /dev/sde1
ERROR: devid 1 is missing or not writeable
ERROR: fixing device size needs all device(s) to be present and writeable

So it seems something went wrong or didn't complete fully. What can I do to fix this problem?

uname -a
Linux BlueQ 5.14.12-arch1-1 #1 SMP PREEMPT Wed, 13 Oct 2021 16:58:16 +0000 x86_64 GNU/Linux

btrfs --version
btrfs-progs v5.14.2

Regards,
Emil

P.S.: Yes, I know, raid5 isn't stable but it works good enough for me ;) Metadata is raid1 btw...

#1 lore raw
Quoted 9 lines · Emil Heimpel

Hi all,

One of my drives of a raid 5 btrfs array failed (was dead completely) so I installed an identical replacement drive. The dead drive was devid 1 and the new drive /dev/sde. I used the following to replace the missing drive:

sudo btrfs replace start -B 1 /dev/sde1 /mnt/btrfsrepair/

and it completed successfully without any reported errors (took around 2 weeks though...).

I then tried to see my array with filesystem show, but it hung (or took longer than I wanted to wait), so I did a reboot.

Any dmesg of that time?

Quoted 12 lines

It showed up after a reboot as followed:

Label: 'BlueButter'  uuid: 7e3378e6-da46-4a60-b9b8-1bcc306986e3
        Total devices 6 FS bytes used 20.96TiB
        devid    0 size 7.28TiB used 5.46TiB path /dev/sde1 devid    2 size 7.28TiB used 5.46TiB path /dev/sdb1 devid    3 size 2.73TiB used 2.73TiB path /dev/sdg1 devid    4 size 2.73TiB used 2.73TiB path /dev/sdd1 devid    5 size 7.28TiB used 4.81TiB path /dev/sdf1 devid    6 size 7.28TiB used 5.33TiB path /dev/sdc1

Quoted 1 line

I then tried to mount it, but it failed, so I run a readonly check and it reported the following problem:

And dmesg for the failed mount?

Thanks,
Qu

Quoted 30 lines

[...]
[2/7] checking extents
ERROR: super total bytes 38007432437760 smaller than real device(s) size 46008994590720
ERROR: mounting this fs may fail for newer kernels
ERROR: this can be fixed by 'btrfs rescue fix-device-size'
[3/7] checking free space tree
[...]

So I followed that advice but got the following error:

sudo btrfs rescue fix-device-size /dev/sde1
ERROR: devid 1 is missing or not writeable
ERROR: fixing device size needs all device(s) to be present and writeable

So it seems something went wrong or didn't complete fully. What can I do to fix this problem?

uname -a
Linux BlueQ 5.14.12-arch1-1 #1 SMP PREEMPT Wed, 13 Oct 2021 16:58:16 +0000 x86_64 GNU/Linux

btrfs --version
btrfs-progs v5.14.2

Regards,
Emil

P.S.: Yes, I know, raid5 isn't stable but it works good enough for me ;) Metadata is raid1 btw...

#2 lore raw

Oct 19, 2021 07:35:54 Qu Wenruo <[email protected]>:

Quoted 7 lines · Emil Heimpel
Quoted 4 lines

Hi all,

One of my drives of a raid 5 btrfs array failed (was dead completely) so I installed an identical replacement drive. The dead drive was devid 1 and the new drive /dev/sde. I used the following to replace the missing drive:

Quoted 8 lines
Quoted 5 lines

sudo btrfs replace start -B 1 /dev/sde1 /mnt/btrfsrepair/

and it completed successfully without any reported errors (took around 2 weeks though...).

I then tried to see my array with filesystem show, but it hung (or took longer than I wanted to wait), so I did a reboot.

Any dmesg of that time?

Nothing after the replace finished:

1634463961.245751 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663044222976 for dev (efault) 1634463961.255819 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663045795840 for dev (efault) 1634463961.275815 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663046582272 for dev (efault) 1634463961.275922 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663047368704 for dev (efault) 1634463961.339074 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663048155136 for dev (efault) 1634463961.339248 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663048941568 for dev (efault) 1634475910.611261 BlueQ kernel: sd 9:0:2:0: attempting task abort!scmd(0x0000000046fead3f), outstanding for 7120 ms & timeout 7000 ms 1634475910.615126 BlueQ kernel: sd 9:0:2:0: [sdd] tag#840 CDB: ATA command pass through(16) 85 08 2e 00 00 00 01 00 00 00 00 00 00 00 ec 00 1634475910.615429 BlueQ kernel: scsi target9:0:2: handle(0x000b), sas_address(0x4433221105000000), phy(5) 1634475910.615691 BlueQ kernel: scsi target9:0:2: enclosure logical id(0x590b11c022f3fb00), slot(6) 1634475910.787911 BlueQ kernel: sd 9:0:2:0: task abort: SUCCESS scmd(0x0000000046fead3f)
1634475910.807083 BlueQ kernel: sd 9:0:2:0: Power-on or device reset occurred
1634475949.877998 BlueQ kernel: sd 9:0:2:0: Power-on or device reset occurred
1634525944.213931 BlueQ kernel: perf: interrupt took too long (3138 > 3137), lowering kernel.perf_event_max_sample_rate to 63600 1634533791.168760 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 22996545634304 for dev (efault) 1634552685.203559 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 23816815706112 for dev (efault) 1634558977.979621 BlueQ kernel: BTRFS info (device sdb1): dev_replace from <missing disk> (devid 1) to /dev/sde1 finished 1634560793.132731 BlueQ kernel: zram0: detected capacity change from 32610864 to 0
1634560793.169379 BlueQ kernel: zram: Removed device: zram0
1634560883.549481 BlueQ kernel: watchdog: watchdog0: watchdog did not stop!
1634560883.556038 BlueQ systemd-shutdown[1]: Syncing filesystems and block devices.
1634560883.572840 BlueQ systemd-shutdown[1]: Sending SIGTERM to remaining processes...

Quoted 12 lines
Quoted 12 lines

It showed up after a reboot as followed:

Label: 'BlueButter'  uuid: 7e3378e6-da46-4a60-b9b8-1bcc306986e3 Total devices 6 FS bytes used 20.96TiB
        devid    0 size 7.28TiB used 5.46TiB path /dev/sde1 devid    2 size 7.28TiB used 5.46TiB path /dev/sdb1 devid    3 size 2.73TiB used 2.73TiB path /dev/sdg1 devid    4 size 2.73TiB used 2.73TiB path /dev/sdd1 devid    5 size 7.28TiB used 4.81TiB path /dev/sdf1 devid    6 size 7.28TiB used 5.33TiB path /dev/sdc1

Quoted 4 lines
Quoted 1 line

I then tried to mount it, but it failed, so I run a readonly check and it reported the following problem:

And dmesg for the failed mount?

Oops, I must have missed that it failed because of missing devid 1 too...

1634562944.145383 BlueQ kernel: BTRFS info (device sde1): flagging fs with big metadata feature 1634562944.145529 BlueQ kernel: BTRFS info (device sde1): force zstd compression, level 2 1634562944.145650 BlueQ kernel: BTRFS info (device sde1): using free space tree
1634562944.145697 BlueQ kernel: BTRFS info (device sde1): has skinny extents
1634562944.148709 BlueQ kernel: BTRFS error (device sde1): devid 1 uuid 51645efd-bf95-458d-b5ae-b31623533abb is missing 1634562944.148764 BlueQ kernel: BTRFS error (device sde1): failed to read chunk tree: -2 1634562944.185369 BlueQ kernel: BTRFS error (device sde1): open_ctree failed

Quoted 32 lines

Thanks,
Qu

Quoted 30 lines

[...]
[2/7] checking extents
ERROR: super total bytes 38007432437760 smaller than real device(s) size 46008994590720
ERROR: mounting this fs may fail for newer kernels
ERROR: this can be fixed by 'btrfs rescue fix-device-size'
[3/7] checking free space tree
[...]

So I followed that advice but got the following error:

sudo btrfs rescue fix-device-size /dev/sde1
ERROR: devid 1 is missing or not writeable
ERROR: fixing device size needs all device(s) to be present and writeable

So it seems something went wrong or didn't complete fully. What can I do to fix this problem?

uname -a
Linux BlueQ 5.14.12-arch1-1 #1 SMP PREEMPT Wed, 13 Oct 2021 16:58:16 +0000 x86_64 GNU/Linux

btrfs --version
btrfs-progs v5.14.2

Regards,
Emil

P.S.: Yes, I know, raid5 isn't stable but it works good enough for me ;) Metadata is raid1 btw...

#3 lore raw
Quoted 19 lines · Emil Heimpel

Oct 19, 2021 07:35:54 Qu Wenruo <[email protected]>:

Quoted 7 lines · Emil Heimpel
Quoted 4 lines

Hi all,

One of my drives of a raid 5 btrfs array failed (was dead completely) so I installed an identical replacement drive. The dead drive was devid 1 and the new drive /dev/sde. I used the following to replace the missing drive:

Quoted 8 lines
Quoted 5 lines

sudo btrfs replace start -B 1 /dev/sde1 /mnt/btrfsrepair/

and it completed successfully without any reported errors (took around 2 weeks though...).

I then tried to see my array with filesystem show, but it hung (or took longer than I wanted to wait), so I did a reboot.

Any dmesg of that time?

Quoted 8 lines

Nothing after the replace finished:

1634463961.245751 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663044222976 for dev (efault) 1634463961.255819 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663045795840 for dev (efault) 1634463961.275815 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663046582272 for dev (efault) 1634463961.275922 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663047368704 for dev (efault) 1634463961.339074 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663048155136 for dev (efault) 1634463961.339248 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663048941568 for dev (efault)

*failed*...

Quoted 4 lines

1634475910.611261 BlueQ kernel: sd 9:0:2:0: attempting task abort!scmd(0x0000000046fead3f), outstanding for 7120 ms & timeout 7000 ms 1634475910.615126 BlueQ kernel: sd 9:0:2:0: [sdd] tag#840 CDB: ATA command pass through(16) 85 08 2e 00 00 00 01 00 00 00 00 00 00 00 ec 00 1634475910.615429 BlueQ kernel: scsi target9:0:2: handle(0x000b), sas_address(0x4433221105000000), phy(5) 1634475910.615691 BlueQ kernel: scsi target9:0:2: enclosure logical id(0x590b11c022f3fb00), slot(6)

And ATA commands failure.

I don't believe the replace finished without problem, and the involved device is /dev/sdd.

Quoted 6 lines

1634475910.787911 BlueQ kernel: sd 9:0:2:0: task abort: SUCCESS scmd(0x0000000046fead3f)
1634475910.807083 BlueQ kernel: sd 9:0:2:0: Power-on or device reset occurred
1634475949.877998 BlueQ kernel: sd 9:0:2:0: Power-on or device reset occurred
1634525944.213931 BlueQ kernel: perf: interrupt took too long (3138 > 3137), lowering kernel.perf_event_max_sample_rate to 63600 1634533791.168760 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 22996545634304 for dev (efault) 1634552685.203559 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 23816815706112 for dev (efault)

You won't want to see this message at all.

This means, you're running RAID56, as btrfs has write-hole problem, which will degrade the robust of RAID56 byte by byte for each unclean shutdown.

I guess the write hole problem has already make the repair failed for the replace.

Thus after a successful mount, scrub and manually file checking is almost a must.

Quoted 29 lines

1634558977.979621 BlueQ kernel: BTRFS info (device sdb1): dev_replace from <missing disk> (devid 1) to /dev/sde1 finished 1634560793.132731 BlueQ kernel: zram0: detected capacity change from 32610864 to 0
1634560793.169379 BlueQ kernel: zram: Removed device: zram0
1634560883.549481 BlueQ kernel: watchdog: watchdog0: watchdog did not stop!
1634560883.556038 BlueQ systemd-shutdown[1]: Syncing filesystems and block devices.
1634560883.572840 BlueQ systemd-shutdown[1]: Sending SIGTERM to remaining processes...

Quoted 12 lines
Quoted 12 lines

It showed up after a reboot as followed:

Label: 'BlueButter'  uuid: 7e3378e6-da46-4a60-b9b8-1bcc306986e3
        Total devices 6 FS bytes used 20.96TiB
        devid    0 size 7.28TiB used 5.46TiB path /dev/sde1 devid    2 size 7.28TiB used 5.46TiB path /dev/sdb1 devid    3 size 2.73TiB used 2.73TiB path /dev/sdg1 devid    4 size 2.73TiB used 2.73TiB path /dev/sdd1 devid    5 size 7.28TiB used 4.81TiB path /dev/sdf1 devid    6 size 7.28TiB used 5.33TiB path /dev/sdc1

Quoted 4 lines
Quoted 1 line

I then tried to mount it, but it failed, so I run a readonly check and it reported the following problem:

And dmesg for the failed mount?

Oops, I must have missed that it failed because of missing devid 1 too...

Quoted 7 lines

1634562944.145383 BlueQ kernel: BTRFS info (device sde1): flagging fs with big metadata feature 1634562944.145529 BlueQ kernel: BTRFS info (device sde1): force zstd compression, level 2 1634562944.145650 BlueQ kernel: BTRFS info (device sde1): using free space tree
1634562944.145697 BlueQ kernel: BTRFS info (device sde1): has skinny extents
1634562944.148709 BlueQ kernel: BTRFS error (device sde1): devid 1 uuid 51645efd-bf95-458d-b5ae-b31623533abb is missing 1634562944.148764 BlueQ kernel: BTRFS error (device sde1): failed to read chunk tree: -2 1634562944.185369 BlueQ kernel: BTRFS error (device sde1): open_ctree failed

This doesn't sound correct.

If a device is properly replaced, it should have the same devid number.

I guess you have tried to add a new device before, and then tried to replace the missing device, right?

Anyway, have you tried to mount it degraded and then remove the missing device?

Since you're using RAID56, I guess degrade mount should work.

Thanks,
Qu

Quoted 34 lines
Quoted 33 lines

Thanks,
Qu

Quoted 30 lines

[...]
[2/7] checking extents
ERROR: super total bytes 38007432437760 smaller than real device(s) size 46008994590720
ERROR: mounting this fs may fail for newer kernels
ERROR: this can be fixed by 'btrfs rescue fix-device-size'
[3/7] checking free space tree
[...]

So I followed that advice but got the following error:

sudo btrfs rescue fix-device-size /dev/sde1
ERROR: devid 1 is missing or not writeable
ERROR: fixing device size needs all device(s) to be present and writeable

So it seems something went wrong or didn't complete fully. What can I do to fix this problem?

uname -a
Linux BlueQ 5.14.12-arch1-1 #1 SMP PREEMPT Wed, 13 Oct 2021 16:58:16 +0000 x86_64 GNU/Linux

btrfs --version
btrfs-progs v5.14.2

Regards,
Emil

P.S.: Yes, I know, raid5 isn't stable but it works good enough for me ;) Metadata is raid1 btw...

#4 lore raw

Oct 19, 2021 13:37:09 Qu Wenruo <[email protected]>:

Quoted 50 lines · Emil Heimpel
Quoted 19 lines

Oct 19, 2021 07:35:54 Qu Wenruo <[email protected]>:

Quoted 7 lines · Emil Heimpel
Quoted 4 lines

Hi all,

One of my drives of a raid 5 btrfs array failed (was dead completely) so I installed an identical replacement drive. The dead drive was devid 1 and the new drive /dev/sde. I used the following to replace the missing drive:

Quoted 8 lines
Quoted 5 lines

sudo btrfs replace start -B 1 /dev/sde1 /mnt/btrfsrepair/

and it completed successfully without any reported errors (took around 2 weeks though...).

I then tried to see my array with filesystem show, but it hung (or took longer than I wanted to wait), so I did a reboot.

Any dmesg of that time?

Quoted 8 lines

Nothing after the replace finished:

1634463961.245751 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663044222976 for dev (efault) 1634463961.255819 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663045795840 for dev (efault) 1634463961.275815 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663046582272 for dev (efault) 1634463961.275922 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663047368704 for dev (efault) 1634463961.339074 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663048155136 for dev (efault) 1634463961.339248 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663048941568 for dev (efault)

*failed*...

Quoted 4 lines

1634475910.611261 BlueQ kernel: sd 9:0:2:0: attempting task abort!scmd(0x0000000046fead3f), outstanding for 7120 ms & timeout 7000 ms 1634475910.615126 BlueQ kernel: sd 9:0:2:0: [sdd] tag#840 CDB: ATA command pass through(16) 85 08 2e 00 00 00 01 00 00 00 00 00 00 00 ec 00 1634475910.615429 BlueQ kernel: scsi target9:0:2: handle(0x000b), sas_address(0x4433221105000000), phy(5) 1634475910.615691 BlueQ kernel: scsi target9:0:2: enclosure logical id(0x590b11c022f3fb00), slot(6)

And ATA commands failure.

I don't believe the replace finished without problem, and the involved device is /dev/sdd.

Quoted 6 lines

1634475910.787911 BlueQ kernel: sd 9:0:2:0: task abort: SUCCESS scmd(0x0000000046fead3f)
1634475910.807083 BlueQ kernel: sd 9:0:2:0: Power-on or device reset occurred
1634475949.877998 BlueQ kernel: sd 9:0:2:0: Power-on or device reset occurred
1634525944.213931 BlueQ kernel: perf: interrupt took too long (3138 > 3137), lowering kernel.perf_event_max_sample_rate to 63600 1634533791.168760 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 22996545634304 for dev (efault) 1634552685.203559 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 23816815706112 for dev (efault)

Quoted 9 lines

You won't want to see this message at all.

This means, you're running RAID56, as btrfs has write-hole problem, which will degrade the robust of RAID56 byte by byte for each unclean shutdown.

I guess the write hole problem has already make the repair failed for the replace.

Hm, I never checked dmesg for the replace, because btrfs replace always showed 0 errors... I may have to check my sata-controller as I don't think it is a problem with sdd. I found these same errors for other drives as well....

Quoted 40 lines

Thus after a successful mount, scrub and manually file checking is almost a must.

Quoted 29 lines

1634558977.979621 BlueQ kernel: BTRFS info (device sdb1): dev_replace from <missing disk> (devid 1) to /dev/sde1 finished 1634560793.132731 BlueQ kernel: zram0: detected capacity change from 32610864 to 0
1634560793.169379 BlueQ kernel: zram: Removed device: zram0
1634560883.549481 BlueQ kernel: watchdog: watchdog0: watchdog did not stop!
1634560883.556038 BlueQ systemd-shutdown[1]: Syncing filesystems and block devices.
1634560883.572840 BlueQ systemd-shutdown[1]: Sending SIGTERM to remaining processes...

Quoted 12 lines
Quoted 12 lines

It showed up after a reboot as followed:

Label: 'BlueButter'  uuid: 7e3378e6-da46-4a60-b9b8-1bcc306986e3 Total devices 6 FS bytes used 20.96TiB
        devid    0 size 7.28TiB used 5.46TiB path /dev/sde1 devid    2 size 7.28TiB used 5.46TiB path /dev/sdb1 devid    3 size 2.73TiB used 2.73TiB path /dev/sdg1 devid    4 size 2.73TiB used 2.73TiB path /dev/sdd1 devid    5 size 7.28TiB used 4.81TiB path /dev/sdf1 devid    6 size 7.28TiB used 5.33TiB path /dev/sdc1

Quoted 4 lines
Quoted 1 line

I then tried to mount it, but it failed, so I run a readonly check and it reported the following problem:

And dmesg for the failed mount?

Oops, I must have missed that it failed because of missing devid 1 too...

Quoted 7 lines

1634562944.145383 BlueQ kernel: BTRFS info (device sde1): flagging fs with big metadata feature 1634562944.145529 BlueQ kernel: BTRFS info (device sde1): force zstd compression, level 2 1634562944.145650 BlueQ kernel: BTRFS info (device sde1): using free space tree
1634562944.145697 BlueQ kernel: BTRFS info (device sde1): has skinny extents
1634562944.148709 BlueQ kernel: BTRFS error (device sde1): devid 1 uuid 51645efd-bf95-458d-b5ae-b31623533abb is missing 1634562944.148764 BlueQ kernel: BTRFS error (device sde1): failed to read chunk tree: -2 1634562944.185369 BlueQ kernel: BTRFS error (device sde1): open_ctree failed

Quoted 7 lines

This doesn't sound correct.

If a device is properly replaced, it should have the same devid number.

I guess you have tried to add a new device before, and then tried to replace the missing device, right?

No. Never added a drive to that array, never tried to remove one. Only replace! This is the 3rd time I was replacing a drive, the first two times the faulty drive was readable and still in the system. This was the first time I had to do a replace without the source drive.

I did replace the first two drives with bigger ones though and did a resize after the replace...

Quoted 6 lines

Anyway, have you tried to mount it degraded and then remove the missing device?

Since you're using RAID56, I guess degrade mount should work.

So, mount degraded and then remove devid 1? I'll try that and report back, thanks!

Emil

Quoted 37 lines

Thanks,
Qu

Quoted 34 lines
Quoted 33 lines

Thanks,
Qu

Quoted 30 lines

[...]
[2/7] checking extents
ERROR: super total bytes 38007432437760 smaller than real device(s) size 46008994590720
ERROR: mounting this fs may fail for newer kernels
ERROR: this can be fixed by 'btrfs rescue fix-device-size'
[3/7] checking free space tree
[...]

So I followed that advice but got the following error:

sudo btrfs rescue fix-device-size /dev/sde1
ERROR: devid 1 is missing or not writeable
ERROR: fixing device size needs all device(s) to be present and writeable

So it seems something went wrong or didn't complete fully. What can I do to fix this problem?

uname -a
Linux BlueQ 5.14.12-arch1-1 #1 SMP PREEMPT Wed, 13 Oct 2021 16:58:16 +0000 x86_64 GNU/Linux

btrfs --version
btrfs-progs v5.14.2

Regards,
Emil

P.S.: Yes, I know, raid5 isn't stable but it works good enough for me ;) Metadata is raid1 btw...

#5 lore raw
In reply to Qu Wenruo

Color me suprised:

[74713.072745] BTRFS info (device sde1): flagging fs with big metadata feature
[74713.072755] BTRFS info (device sde1): allowing degraded mounts
[74713.072758] BTRFS info (device sde1): using free space tree
[74713.072760] BTRFS info (device sde1): has skinny extents
[74713.104297] BTRFS warning (device sde1): devid 1 uuid 51645efd-bf95-458d-b5ae-b31623533abb is missing
[74714.675001] BTRFS info (device sde1): bdev (efault) errs: wr 52950, rd 8161, flush 0, corrupt 1221, gen 0
[74714.675015] BTRFS info (device sde1): bdev /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 228, gen 0
[74714.675025] BTRFS info (device sde1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 140, gen 0
[74751.033383] BTRFS info (device sde1): continuing dev_replace from <missing disk> (devid 1) to target /dev/sde1 @74%

[bluemond@BlueQ ~]$ sudo btrfs replace status  -1 /mnt/btrfsrepair/ 74.9% done, 0 write errs, 0 uncorr. read errs

I guess I just wait?

Oct 19, 2021 13:37:09 Qu Wenruo <[email protected]>:

Quoted 149 lines · Emil Heimpel
Quoted 19 lines

Oct 19, 2021 07:35:54 Qu Wenruo <[email protected]>:

Quoted 7 lines · Emil Heimpel
Quoted 4 lines

Hi all,

One of my drives of a raid 5 btrfs array failed (was dead completely) so I installed an identical replacement drive. The dead drive was devid 1 and the new drive /dev/sde. I used the following to replace the missing drive:

Quoted 8 lines
Quoted 5 lines

sudo btrfs replace start -B 1 /dev/sde1 /mnt/btrfsrepair/

and it completed successfully without any reported errors (took around 2 weeks though...).

I then tried to see my array with filesystem show, but it hung (or took longer than I wanted to wait), so I did a reboot.

Any dmesg of that time?

Quoted 8 lines

Nothing after the replace finished:

1634463961.245751 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663044222976 for dev (efault) 1634463961.255819 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663045795840 for dev (efault) 1634463961.275815 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663046582272 for dev (efault) 1634463961.275922 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663047368704 for dev (efault) 1634463961.339074 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663048155136 for dev (efault) 1634463961.339248 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663048941568 for dev (efault)

*failed*...

Quoted 4 lines

1634475910.611261 BlueQ kernel: sd 9:0:2:0: attempting task abort!scmd(0x0000000046fead3f), outstanding for 7120 ms & timeout 7000 ms 1634475910.615126 BlueQ kernel: sd 9:0:2:0: [sdd] tag#840 CDB: ATA command pass through(16) 85 08 2e 00 00 00 01 00 00 00 00 00 00 00 ec 00 1634475910.615429 BlueQ kernel: scsi target9:0:2: handle(0x000b), sas_address(0x4433221105000000), phy(5) 1634475910.615691 BlueQ kernel: scsi target9:0:2: enclosure logical id(0x590b11c022f3fb00), slot(6)

And ATA commands failure.

I don't believe the replace finished without problem, and the involved device is /dev/sdd.

Quoted 6 lines

1634475910.787911 BlueQ kernel: sd 9:0:2:0: task abort: SUCCESS scmd(0x0000000046fead3f)
1634475910.807083 BlueQ kernel: sd 9:0:2:0: Power-on or device reset occurred
1634475949.877998 BlueQ kernel: sd 9:0:2:0: Power-on or device reset occurred
1634525944.213931 BlueQ kernel: perf: interrupt took too long (3138 > 3137), lowering kernel.perf_event_max_sample_rate to 63600 1634533791.168760 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 22996545634304 for dev (efault) 1634552685.203559 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 23816815706112 for dev (efault)

You won't want to see this message at all.

This means, you're running RAID56, as btrfs has write-hole problem, which will degrade the robust of RAID56 byte by byte for each unclean shutdown.

I guess the write hole problem has already make the repair failed for the replace.

Thus after a successful mount, scrub and manually file checking is almost a must.

Quoted 29 lines

1634558977.979621 BlueQ kernel: BTRFS info (device sdb1): dev_replace from <missing disk> (devid 1) to /dev/sde1 finished 1634560793.132731 BlueQ kernel: zram0: detected capacity change from 32610864 to 0
1634560793.169379 BlueQ kernel: zram: Removed device: zram0
1634560883.549481 BlueQ kernel: watchdog: watchdog0: watchdog did not stop!
1634560883.556038 BlueQ systemd-shutdown[1]: Syncing filesystems and block devices.
1634560883.572840 BlueQ systemd-shutdown[1]: Sending SIGTERM to remaining processes...

Quoted 12 lines
Quoted 12 lines

It showed up after a reboot as followed:

Label: 'BlueButter'  uuid: 7e3378e6-da46-4a60-b9b8-1bcc306986e3 Total devices 6 FS bytes used 20.96TiB
        devid    0 size 7.28TiB used 5.46TiB path /dev/sde1 devid    2 size 7.28TiB used 5.46TiB path /dev/sdb1 devid    3 size 2.73TiB used 2.73TiB path /dev/sdg1 devid    4 size 2.73TiB used 2.73TiB path /dev/sdd1 devid    5 size 7.28TiB used 4.81TiB path /dev/sdf1 devid    6 size 7.28TiB used 5.33TiB path /dev/sdc1

Quoted 4 lines
Quoted 1 line

I then tried to mount it, but it failed, so I run a readonly check and it reported the following problem:

And dmesg for the failed mount?

Oops, I must have missed that it failed because of missing devid 1 too...

Quoted 7 lines

1634562944.145383 BlueQ kernel: BTRFS info (device sde1): flagging fs with big metadata feature 1634562944.145529 BlueQ kernel: BTRFS info (device sde1): force zstd compression, level 2 1634562944.145650 BlueQ kernel: BTRFS info (device sde1): using free space tree
1634562944.145697 BlueQ kernel: BTRFS info (device sde1): has skinny extents
1634562944.148709 BlueQ kernel: BTRFS error (device sde1): devid 1 uuid 51645efd-bf95-458d-b5ae-b31623533abb is missing 1634562944.148764 BlueQ kernel: BTRFS error (device sde1): failed to read chunk tree: -2 1634562944.185369 BlueQ kernel: BTRFS error (device sde1): open_ctree failed

This doesn't sound correct.

If a device is properly replaced, it should have the same devid number.

I guess you have tried to add a new device before, and then tried to replace the missing device, right?

Anyway, have you tried to mount it degraded and then remove the missing device?

Since you're using RAID56, I guess degrade mount should work.

Thanks,
Qu

Quoted 34 lines
Quoted 33 lines

Thanks,
Qu

Quoted 30 lines

[...]
[2/7] checking extents
ERROR: super total bytes 38007432437760 smaller than real device(s) size 46008994590720
ERROR: mounting this fs may fail for newer kernels
ERROR: this can be fixed by 'btrfs rescue fix-device-size'
[3/7] checking free space tree
[...]

So I followed that advice but got the following error:

sudo btrfs rescue fix-device-size /dev/sde1
ERROR: devid 1 is missing or not writeable
ERROR: fixing device size needs all device(s) to be present and writeable

So it seems something went wrong or didn't complete fully. What can I do to fix this problem?

uname -a
Linux BlueQ 5.14.12-arch1-1 #1 SMP PREEMPT Wed, 13 Oct 2021 16:58:16 +0000 x86_64 GNU/Linux

btrfs --version
btrfs-progs v5.14.2

Regards,
Emil

P.S.: Yes, I know, raid5 isn't stable but it works good enough for me ;) Metadata is raid1 btw...

#6 lore raw
Quoted 12 lines · Emil Heimpel

Color me suprised:

[74713.072745] BTRFS info (device sde1): flagging fs with big metadata feature
[74713.072755] BTRFS info (device sde1): allowing degraded mounts
[74713.072758] BTRFS info (device sde1): using free space tree
[74713.072760] BTRFS info (device sde1): has skinny extents
[74713.104297] BTRFS warning (device sde1): devid 1 uuid 51645efd-bf95-458d-b5ae-b31623533abb is missing
[74714.675001] BTRFS info (device sde1): bdev (efault) errs: wr 52950, rd 8161, flush 0, corrupt 1221, gen 0
[74714.675015] BTRFS info (device sde1): bdev /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 228, gen 0
[74714.675025] BTRFS info (device sde1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 140, gen 0
[74751.033383] BTRFS info (device sde1): continuing dev_replace from <missing disk> (devid 1) to target /dev/sde1 @74%
Quoted 4 lines

[bluemond@BlueQ ~]$ sudo btrfs replace status  -1 /mnt/btrfsrepair/ 74.9% done, 0 write errs, 0 uncorr. read errs

I guess I just wait?

Yep, wait and stay alert, better to also keep an eye on the dmesg.

But this also means, previous replace didn't really finish, which may mean the replace ioctl is not reporting the proper status, and can be a possible bug.

Thanks,
Qu

Quoted 152 lines

Oct 19, 2021 13:37:09 Qu Wenruo <[email protected]>:

Quoted 149 lines · Emil Heimpel
Quoted 19 lines

Oct 19, 2021 07:35:54 Qu Wenruo <[email protected]>:

Quoted 7 lines · Emil Heimpel
Quoted 4 lines

Hi all,

One of my drives of a raid 5 btrfs array failed (was dead completely) so I installed an identical replacement drive. The dead drive was devid 1 and the new drive /dev/sde. I used the following to replace the missing drive:

Quoted 8 lines
Quoted 5 lines

sudo btrfs replace start -B 1 /dev/sde1 /mnt/btrfsrepair/

and it completed successfully without any reported errors (took around 2 weeks though...).

I then tried to see my array with filesystem show, but it hung (or took longer than I wanted to wait), so I did a reboot.

Any dmesg of that time?

Quoted 8 lines

Nothing after the replace finished:

1634463961.245751 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663044222976 for dev (efault) 1634463961.255819 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663045795840 for dev (efault) 1634463961.275815 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663046582272 for dev (efault) 1634463961.275922 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663047368704 for dev (efault) 1634463961.339074 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663048155136 for dev (efault) 1634463961.339248 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663048941568 for dev (efault)

*failed*...

Quoted 4 lines

1634475910.611261 BlueQ kernel: sd 9:0:2:0: attempting task abort!scmd(0x0000000046fead3f), outstanding for 7120 ms & timeout 7000 ms 1634475910.615126 BlueQ kernel: sd 9:0:2:0: [sdd] tag#840 CDB: ATA command pass through(16) 85 08 2e 00 00 00 01 00 00 00 00 00 00 00 ec 00 1634475910.615429 BlueQ kernel: scsi target9:0:2: handle(0x000b), sas_address(0x4433221105000000), phy(5) 1634475910.615691 BlueQ kernel: scsi target9:0:2: enclosure logical id(0x590b11c022f3fb00), slot(6)

And ATA commands failure.

I don't believe the replace finished without problem, and the involved device is /dev/sdd.

Quoted 6 lines

1634475910.787911 BlueQ kernel: sd 9:0:2:0: task abort: SUCCESS scmd(0x0000000046fead3f)
1634475910.807083 BlueQ kernel: sd 9:0:2:0: Power-on or device reset occurred
1634475949.877998 BlueQ kernel: sd 9:0:2:0: Power-on or device reset occurred
1634525944.213931 BlueQ kernel: perf: interrupt took too long (3138 > 3137), lowering kernel.perf_event_max_sample_rate to 63600 1634533791.168760 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 22996545634304 for dev (efault) 1634552685.203559 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 23816815706112 for dev (efault)

You won't want to see this message at all.

This means, you're running RAID56, as btrfs has write-hole problem, which will degrade the robust of RAID56 byte by byte for each unclean shutdown.

I guess the write hole problem has already make the repair failed for the replace.

Thus after a successful mount, scrub and manually file checking is almost a must.

Quoted 29 lines

1634558977.979621 BlueQ kernel: BTRFS info (device sdb1): dev_replace from <missing disk> (devid 1) to /dev/sde1 finished 1634560793.132731 BlueQ kernel: zram0: detected capacity change from 32610864 to 0
1634560793.169379 BlueQ kernel: zram: Removed device: zram0
1634560883.549481 BlueQ kernel: watchdog: watchdog0: watchdog did not stop!
1634560883.556038 BlueQ systemd-shutdown[1]: Syncing filesystems and block devices.
1634560883.572840 BlueQ systemd-shutdown[1]: Sending SIGTERM to remaining processes...

Quoted 12 lines
Quoted 12 lines

It showed up after a reboot as followed:

Label: 'BlueButter'  uuid: 7e3378e6-da46-4a60-b9b8-1bcc306986e3
        Total devices 6 FS bytes used 20.96TiB
        devid    0 size 7.28TiB used 5.46TiB path /dev/sde1 devid    2 size 7.28TiB used 5.46TiB path /dev/sdb1 devid    3 size 2.73TiB used 2.73TiB path /dev/sdg1 devid    4 size 2.73TiB used 2.73TiB path /dev/sdd1 devid    5 size 7.28TiB used 4.81TiB path /dev/sdf1 devid    6 size 7.28TiB used 5.33TiB path /dev/sdc1

Quoted 4 lines
Quoted 1 line

I then tried to mount it, but it failed, so I run a readonly check and it reported the following problem:

And dmesg for the failed mount?

Oops, I must have missed that it failed because of missing devid 1 too...

Quoted 7 lines

1634562944.145383 BlueQ kernel: BTRFS info (device sde1): flagging fs with big metadata feature 1634562944.145529 BlueQ kernel: BTRFS info (device sde1): force zstd compression, level 2 1634562944.145650 BlueQ kernel: BTRFS info (device sde1): using free space tree
1634562944.145697 BlueQ kernel: BTRFS info (device sde1): has skinny extents
1634562944.148709 BlueQ kernel: BTRFS error (device sde1): devid 1 uuid 51645efd-bf95-458d-b5ae-b31623533abb is missing 1634562944.148764 BlueQ kernel: BTRFS error (device sde1): failed to read chunk tree: -2 1634562944.185369 BlueQ kernel: BTRFS error (device sde1): open_ctree failed

This doesn't sound correct.

If a device is properly replaced, it should have the same devid number.

I guess you have tried to add a new device before, and then tried to replace the missing device, right?

Anyway, have you tried to mount it degraded and then remove the missing device?

Since you're using RAID56, I guess degrade mount should work.

Thanks,
Qu

Quoted 34 lines
Quoted 33 lines

Thanks,
Qu

Quoted 30 lines

[...]
[2/7] checking extents
ERROR: super total bytes 38007432437760 smaller than real device(s) size 46008994590720
ERROR: mounting this fs may fail for newer kernels
ERROR: this can be fixed by 'btrfs rescue fix-device-size'
[3/7] checking free space tree
[...]

So I followed that advice but got the following error:

sudo btrfs rescue fix-device-size /dev/sde1
ERROR: devid 1 is missing or not writeable
ERROR: fixing device size needs all device(s) to be present and writeable

So it seems something went wrong or didn't complete fully. What can I do to fix this problem?

uname -a
Linux BlueQ 5.14.12-arch1-1 #1 SMP PREEMPT Wed, 13 Oct 2021 16:58:16 +0000 x86_64 GNU/Linux

btrfs --version
btrfs-progs v5.14.2

Regards,
Emil

P.S.: Yes, I know, raid5 isn't stable but it works good enough for me ;) Metadata is raid1 btw...

#7 lore raw

So it finished after 2 minutes?

[Tue Oct 19 14:13:51 2021] BTRFS info (device sde1): continuing dev_replace from <missing disk> (devid 1) to target /dev/sde1 @74%
[Tue Oct 19 14:15:39 2021] BTRFS info (device sde1): dev_replace from <missing disk> (devid 1) to /dev/sde1 finished

Now I at least have an expected filesystem show:

Label: 'BlueButter'  uuid: 7e3378e6-da46-4a60-b9b8-1bcc306986e3 Total devices 6 FS bytes used 20.96TiB
        devid    1 size 7.28TiB used 5.46TiB path /dev/sde1 devid    2 size 7.28TiB used 5.46TiB path /dev/sdb1 devid    3 size 2.73TiB used 2.73TiB path /dev/sdg1 devid    4 size 2.73TiB used 2.73TiB path /dev/sdd1 devid    5 size 7.28TiB used 4.81TiB path /dev/sdf1 devid    6 size 7.28TiB used 5.33TiB path /dev/sdc1

And a nondegraded remount worked too.

Thanks,
Emil

Oct 19, 2021 14:20:21 Qu Wenruo <[email protected]>:

Quoted 132 lines · Emil Heimpel
Quoted 12 lines

Color me suprised:

[74713.072745] BTRFS info (device sde1): flagging fs with big metadata feature
[74713.072755] BTRFS info (device sde1): allowing degraded mounts
[74713.072758] BTRFS info (device sde1): using free space tree
[74713.072760] BTRFS info (device sde1): has skinny extents
[74713.104297] BTRFS warning (device sde1): devid 1 uuid 51645efd-bf95-458d-b5ae-b31623533abb is missing
[74714.675001] BTRFS info (device sde1): bdev (efault) errs: wr 52950, rd 8161, flush 0, corrupt 1221, gen 0
[74714.675015] BTRFS info (device sde1): bdev /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 228, gen 0
[74714.675025] BTRFS info (device sde1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 140, gen 0
[74751.033383] BTRFS info (device sde1): continuing dev_replace from <missing disk> (devid 1) to target /dev/sde1 @74%
Quoted 4 lines

[bluemond@BlueQ ~]$ sudo btrfs replace status  -1 /mnt/btrfsrepair/ 74.9% done, 0 write errs, 0 uncorr. read errs

I guess I just wait?

Yep, wait and stay alert, better to also keep an eye on the dmesg.

But this also means, previous replace didn't really finish, which may mean the replace ioctl is not reporting the proper status, and can be a possible bug.

Thanks,
Qu

Quoted 103 lines

Oct 19, 2021 13:37:09 Qu Wenruo <[email protected]>:

Quoted 100 lines · Emil Heimpel
Quoted 11 lines

Oct 19, 2021 07:35:54 Qu Wenruo <[email protected]>:

Quoted 7 lines · Emil Heimpel
Quoted 1 line

…

Any dmesg of that time?

Quoted 8 lines

Nothing after the replace finished:

1634463961.245751 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663044222976 for dev (efault) 1634463961.255819 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663045795840 for dev (efault) 1634463961.275815 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663046582272 for dev (efault) 1634463961.275922 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663047368704 for dev (efault) 1634463961.339074 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663048155136 for dev (efault) 1634463961.339248 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663048941568 for dev (efault)

*failed*...

Quoted 4 lines

1634475910.611261 BlueQ kernel: sd 9:0:2:0: attempting task abort!scmd(0x0000000046fead3f), outstanding for 7120 ms & timeout 7000 ms 1634475910.615126 BlueQ kernel: sd 9:0:2:0: [sdd] tag#840 CDB: ATA command pass through(16) 85 08 2e 00 00 00 01 00 00 00 00 00 00 00 ec 00 1634475910.615429 BlueQ kernel: scsi target9:0:2: handle(0x000b), sas_address(0x4433221105000000), phy(5) 1634475910.615691 BlueQ kernel: scsi target9:0:2: enclosure logical id(0x590b11c022f3fb00), slot(6)

And ATA commands failure.

I don't believe the replace finished without problem, and the involved device is /dev/sdd.

Quoted 6 lines

1634475910.787911 BlueQ kernel: sd 9:0:2:0: task abort: SUCCESS scmd(0x0000000046fead3f)
1634475910.807083 BlueQ kernel: sd 9:0:2:0: Power-on or device reset occurred
1634475949.877998 BlueQ kernel: sd 9:0:2:0: Power-on or device reset occurred
1634525944.213931 BlueQ kernel: perf: interrupt took too long (3138 > 3137), lowering kernel.perf_event_max_sample_rate to 63600 1634533791.168760 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 22996545634304 for dev (efault) 1634552685.203559 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 23816815706112 for dev (efault)

You won't want to see this message at all.

This means, you're running RAID56, as btrfs has write-hole problem, which will degrade the robust of RAID56 byte by byte for each unclean shutdown.

I guess the write hole problem has already make the repair failed for the replace.

Thus after a successful mount, scrub and manually file checking is almost a must.

Quoted 17 lines

1634558977.979621 BlueQ kernel: BTRFS info (device sdb1): dev_replace from <missing disk> (devid 1) to /dev/sde1 finished 1634560793.132731 BlueQ kernel: zram0: detected capacity change from 32610864 to 0
1634560793.169379 BlueQ kernel: zram: Removed device: zram0
1634560883.549481 BlueQ kernel: watchdog: watchdog0: watchdog did not stop!
1634560883.556038 BlueQ systemd-shutdown[1]: Syncing filesystems and block devices.
1634560883.572840 BlueQ systemd-shutdown[1]: Sending SIGTERM to remaining processes...

Quoted 4 lines
Quoted 1 line

…

And dmesg for the failed mount?

Oops, I must have missed that it failed because of missing devid 1 too...

Quoted 7 lines

1634562944.145383 BlueQ kernel: BTRFS info (device sde1): flagging fs with big metadata feature 1634562944.145529 BlueQ kernel: BTRFS info (device sde1): force zstd compression, level 2 1634562944.145650 BlueQ kernel: BTRFS info (device sde1): using free space tree
1634562944.145697 BlueQ kernel: BTRFS info (device sde1): has skinny extents
1634562944.148709 BlueQ kernel: BTRFS error (device sde1): devid 1 uuid 51645efd-bf95-458d-b5ae-b31623533abb is missing 1634562944.148764 BlueQ kernel: BTRFS error (device sde1): failed to read chunk tree: -2 1634562944.185369 BlueQ kernel: BTRFS error (device sde1): open_ctree failed

This doesn't sound correct.

If a device is properly replaced, it should have the same devid number.

I guess you have tried to add a new device before, and then tried to replace the missing device, right?

Anyway, have you tried to mount it degraded and then remove the missing device?

Since you're using RAID56, I guess degrade mount should work.

Thanks,
Qu

Quoted 5 lines
Quoted 4 lines

Thanks,
Qu

Quoted 1 line

…

#8 lore raw
Quoted 4 lines · Emil Heimpel

So it finished after 2 minutes?

[Tue Oct 19 14:13:51 2021] BTRFS info (device sde1): continuing dev_replace from <missing disk> (devid 1) to target /dev/sde1 @74%
[Tue Oct 19 14:15:39 2021] BTRFS info (device sde1): dev_replace from <missing disk> (devid 1) to /dev/sde1 finished

Then this means it previously only have the dev replace status not cleaned up.

Thus it ends pretty quick.

Quoted 13 lines

Now I at least have an expected filesystem show:

Label: 'BlueButter'  uuid: 7e3378e6-da46-4a60-b9b8-1bcc306986e3
        Total devices 6 FS bytes used 20.96TiB
        devid    1 size 7.28TiB used 5.46TiB path /dev/sde1 devid    2 size 7.28TiB used 5.46TiB path /dev/sdb1 devid    3 size 2.73TiB used 2.73TiB path /dev/sdg1 devid    4 size 2.73TiB used 2.73TiB path /dev/sdd1 devid    5 size 7.28TiB used 4.81TiB path /dev/sdf1 devid    6 size 7.28TiB used 5.33TiB path /dev/sdc1

Quoted 1 line

And a nondegraded remount worked too.

And it's time for a full fs scrub to find out how consistent the fs is.

For btrfs RAID56, every time a unexpected shutdown is hit, a scrub is strongly recommended.

And even routine scrub would be a plus for btrfs RAID56.

Thanks,
Qu

Quoted 138 lines

Thanks,
Emil

Oct 19, 2021 14:20:21 Qu Wenruo <[email protected]>:

Quoted 132 lines · Emil Heimpel
Quoted 12 lines

Color me suprised:

[74713.072745] BTRFS info (device sde1): flagging fs with big metadata feature
[74713.072755] BTRFS info (device sde1): allowing degraded mounts
[74713.072758] BTRFS info (device sde1): using free space tree
[74713.072760] BTRFS info (device sde1): has skinny extents
[74713.104297] BTRFS warning (device sde1): devid 1 uuid 51645efd-bf95-458d-b5ae-b31623533abb is missing
[74714.675001] BTRFS info (device sde1): bdev (efault) errs: wr 52950, rd 8161, flush 0, corrupt 1221, gen 0
[74714.675015] BTRFS info (device sde1): bdev /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 228, gen 0
[74714.675025] BTRFS info (device sde1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 140, gen 0
[74751.033383] BTRFS info (device sde1): continuing dev_replace from <missing disk> (devid 1) to target /dev/sde1 @74%
Quoted 4 lines

[bluemond@BlueQ ~]$ sudo btrfs replace status  -1 /mnt/btrfsrepair/ 74.9% done, 0 write errs, 0 uncorr. read errs

I guess I just wait?

Yep, wait and stay alert, better to also keep an eye on the dmesg.

But this also means, previous replace didn't really finish, which may mean the replace ioctl is not reporting the proper status, and can be a possible bug.

Thanks,
Qu

Quoted 103 lines

Oct 19, 2021 13:37:09 Qu Wenruo <[email protected]>:

Quoted 100 lines · Emil Heimpel
Quoted 11 lines

Oct 19, 2021 07:35:54 Qu Wenruo <[email protected]>:

Quoted 7 lines · Emil Heimpel
Quoted 1 line

…

Any dmesg of that time?

Quoted 8 lines

Nothing after the replace finished:

1634463961.245751 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663044222976 for dev (efault) 1634463961.255819 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663045795840 for dev (efault) 1634463961.275815 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663046582272 for dev (efault) 1634463961.275922 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663047368704 for dev (efault) 1634463961.339074 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663048155136 for dev (efault) 1634463961.339248 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 17663048941568 for dev (efault)

*failed*...

Quoted 4 lines

1634475910.611261 BlueQ kernel: sd 9:0:2:0: attempting task abort!scmd(0x0000000046fead3f), outstanding for 7120 ms & timeout 7000 ms 1634475910.615126 BlueQ kernel: sd 9:0:2:0: [sdd] tag#840 CDB: ATA command pass through(16) 85 08 2e 00 00 00 01 00 00 00 00 00 00 00 ec 00 1634475910.615429 BlueQ kernel: scsi target9:0:2: handle(0x000b), sas_address(0x4433221105000000), phy(5) 1634475910.615691 BlueQ kernel: scsi target9:0:2: enclosure logical id(0x590b11c022f3fb00), slot(6)

And ATA commands failure.

I don't believe the replace finished without problem, and the involved device is /dev/sdd.

Quoted 6 lines

1634475910.787911 BlueQ kernel: sd 9:0:2:0: task abort: SUCCESS scmd(0x0000000046fead3f)
1634475910.807083 BlueQ kernel: sd 9:0:2:0: Power-on or device reset occurred
1634475949.877998 BlueQ kernel: sd 9:0:2:0: Power-on or device reset occurred
1634525944.213931 BlueQ kernel: perf: interrupt took too long (3138 > 3137), lowering kernel.perf_event_max_sample_rate to 63600 1634533791.168760 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 22996545634304 for dev (efault) 1634552685.203559 BlueQ kernel: BTRFS error (device sdb1): failed to rebuild valid logical 23816815706112 for dev (efault)

You won't want to see this message at all.

This means, you're running RAID56, as btrfs has write-hole problem, which will degrade the robust of RAID56 byte by byte for each unclean shutdown.

I guess the write hole problem has already make the repair failed for the replace.

Thus after a successful mount, scrub and manually file checking is almost a must.

Quoted 17 lines

1634558977.979621 BlueQ kernel: BTRFS info (device sdb1): dev_replace from <missing disk> (devid 1) to /dev/sde1 finished 1634560793.132731 BlueQ kernel: zram0: detected capacity change from 32610864 to 0
1634560793.169379 BlueQ kernel: zram: Removed device: zram0
1634560883.549481 BlueQ kernel: watchdog: watchdog0: watchdog did not stop!
1634560883.556038 BlueQ systemd-shutdown[1]: Syncing filesystems and block devices.
1634560883.572840 BlueQ systemd-shutdown[1]: Sending SIGTERM to remaining processes...

Quoted 4 lines
Quoted 1 line

…

And dmesg for the failed mount?

Oops, I must have missed that it failed because of missing devid 1 too...

Quoted 7 lines

1634562944.145383 BlueQ kernel: BTRFS info (device sde1): flagging fs with big metadata feature 1634562944.145529 BlueQ kernel: BTRFS info (device sde1): force zstd compression, level 2 1634562944.145650 BlueQ kernel: BTRFS info (device sde1): using free space tree
1634562944.145697 BlueQ kernel: BTRFS info (device sde1): has skinny extents
1634562944.148709 BlueQ kernel: BTRFS error (device sde1): devid 1 uuid 51645efd-bf95-458d-b5ae-b31623533abb is missing 1634562944.148764 BlueQ kernel: BTRFS error (device sde1): failed to read chunk tree: -2 1634562944.185369 BlueQ kernel: BTRFS error (device sde1): open_ctree failed

This doesn't sound correct.

If a device is properly replaced, it should have the same devid number.

I guess you have tried to add a new device before, and then tried to replace the missing device, right?

Anyway, have you tried to mount it degraded and then remove the missing device?

Since you're using RAID56, I guess degrade mount should work.

Thanks,
Qu

Quoted 5 lines
Quoted 4 lines

Thanks,
Qu

Quoted 1 line

…

#9 lore raw

After reapplying the thermal paste and adding a fan to my storage controller (LSI SAS2008) I haven't encountered any abnormal behavior of my drives. Scrub is running (disk by disk) and everything looks fine so far.

I only got one task blocked message in dmesg, is that anything I should worry about?

Sun Oct 24 18:22:11 2021] BTRFS info (device sdg1): scrub: started on devid 5

[Mon Oct 25 15:31:23 2021] BTRFS info (device sdg1): scrub: finished on devid 5 with status: 0
[Mon Oct 25 15:31:26 2021] BTRFS info (device sdg1): scrub: started on devid 3
[Mon Oct 25 21:12:34 2021] perf: interrupt took too long (2501 > 2500), lowering kernel.perf_event_max_sample_rate to 79800
[Mon Oct 25 23:01:43 2021] hrtimer: interrupt took 13811 ns
[Tue Oct 26 02:58:17 2021] BTRFS info (device sdg1): scrub: finished on devid 3 with status: 0
[Tue Oct 26 02:58:21 2021] BTRFS info (device sdg1): scrub: started on devid 6
[Tue Oct 26 12:23:36 2021] INFO: task btrfs:341674 blocked for more than 122 seconds.
[Tue Oct 26 12:23:36 2021]       Tainted: G           OE     5.14.14-arch1-1 #1
[Tue Oct 26 12:23:36 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Tue Oct 26 12:23:36 2021] task:btrfs           state:D stack:    0 pid:341674 ppid:341658 flags:0x00004000
[Tue Oct 26 12:23:36 2021] Call Trace:
[Tue Oct 26 12:23:36 2021]  __schedule+0x333/0x1530
[Tue Oct 26 12:23:36 2021]  ? psi_task_switch+0xc2/0x1f0
[Tue Oct 26 12:23:36 2021]  ? autoremove_wake_function+0x2c/0x50
[Tue Oct 26 12:23:36 2021]  schedule+0x59/0xc0
[Tue Oct 26 12:23:36 2021]  __scrub_blocked_if_needed+0xa0/0xf0 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:23:36 2021]  ? do_wait_intr_irq+0xa0/0xa0
[Tue Oct 26 12:23:36 2021]  scrub_pause_off+0x21/0x50 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:23:36 2021]  scrub_stripe+0x452/0x1580 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:23:36 2021]  ? do_wait_intr_irq+0xa0/0xa0
[Tue Oct 26 12:23:36 2021]  ? __btrfs_end_transaction+0xf6/0x210 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:23:36 2021]  ? scrub_chunk+0xcd/0x130 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:23:36 2021]  scrub_chunk+0xcd/0x130 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:23:36 2021]  scrub_enumerate_chunks+0x354/0x790 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:23:36 2021]  ? do_wait_intr_irq+0xa0/0xa0
[Tue Oct 26 12:23:36 2021]  btrfs_scrub_dev+0x23d/0x570 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:23:36 2021]  btrfs_ioctl+0x1410/0x2df0 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:23:36 2021]  ? __x64_sys_ioctl+0x82/0xb0
[Tue Oct 26 12:23:36 2021]  __x64_sys_ioctl+0x82/0xb0
[Tue Oct 26 12:23:36 2021]  do_syscall_64+0x5c/0x80
[Tue Oct 26 12:23:36 2021]  ? create_task_io_context+0xc7/0x110
[Tue Oct 26 12:23:36 2021]  ? get_task_io_context+0x48/0x80
[Tue Oct 26 12:23:36 2021]  ? set_task_ioprio+0x97/0xa0
[Tue Oct 26 12:23:36 2021]  ? __do_sys_ioprio_set+0x5e/0x300
[Tue Oct 26 12:23:36 2021]  ? syscall_exit_to_user_mode+0x23/0x40
[Tue Oct 26 12:23:36 2021]  ? syscall_exit_to_user_mode+0x23/0x40
[Tue Oct 26 12:23:36 2021]  ? do_syscall_64+0x69/0x80
[Tue Oct 26 12:23:36 2021]  ? exit_to_user_mode_prepare+0x77/0x170
[Tue Oct 26 12:23:36 2021]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[Tue Oct 26 12:23:36 2021] RIP: 0033:0x7fba22e2559b
[Tue Oct 26 12:23:36 2021] RSP: 002b:00007fba22cf7c98 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Tue Oct 26 12:23:36 2021] RAX: ffffffffffffffda RBX: 000056511ecc6bc0 RCX: 00007fba22e2559b
[Tue Oct 26 12:23:36 2021] RDX: 000056511ecc6bc0 RSI: 00000000c400941b RDI: 0000000000000003
[Tue Oct 26 12:23:36 2021] RBP: 0000000000000000 R08: 00007fba22cf8640 R09: 0000000000000000
[Tue Oct 26 12:23:36 2021] R10: 00007fba22cf8640 R11: 0000000000000246 R12: 00007ffec33d061e
[Tue Oct 26 12:23:36 2021] R13: 00007ffec33d061f R14: 0000000000000000 R15: 00007fba22cf8640
[Tue Oct 26 12:27:42 2021] INFO: task btrfs:341674 blocked for more than 122 seconds.
[Tue Oct 26 12:27:42 2021]       Tainted: G           OE     5.14.14-arch1-1 #1
[Tue Oct 26 12:27:42 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Tue Oct 26 12:27:42 2021] task:btrfs           state:D stack:    0 pid:341674 ppid:341658 flags:0x00004000
[Tue Oct 26 12:27:42 2021] Call Trace:
[Tue Oct 26 12:27:42 2021]  __schedule+0x333/0x1530
[Tue Oct 26 12:27:42 2021]  ? autoremove_wake_function+0x2c/0x50
[Tue Oct 26 12:27:42 2021]  schedule+0x59/0xc0
[Tue Oct 26 12:27:42 2021]  __scrub_blocked_if_needed+0xa0/0xf0 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:27:42 2021]  ? do_wait_intr_irq+0xa0/0xa0
[Tue Oct 26 12:27:42 2021]  scrub_pause_off+0x21/0x50 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:27:42 2021]  scrub_stripe+0x452/0x1580 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:27:42 2021]  ? __update_load_avg_cfs_rq+0x27d/0x2e0
[Tue Oct 26 12:27:42 2021]  ? __btrfs_end_transaction+0xf6/0x210 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:27:42 2021]  ? kmem_cache_free+0x107/0x410
[Tue Oct 26 12:27:42 2021]  ? scrub_chunk+0xcd/0x130 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:27:42 2021]  scrub_chunk+0xcd/0x130 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:27:42 2021]  scrub_enumerate_chunks+0x354/0x790 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:27:42 2021]  ? do_wait_intr_irq+0xa0/0xa0
[Tue Oct 26 12:27:42 2021]  btrfs_scrub_dev+0x23d/0x570 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:27:42 2021]  btrfs_ioctl+0x1410/0x2df0 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:27:42 2021]  ? __x64_sys_ioctl+0x82/0xb0
[Tue Oct 26 12:27:42 2021]  __x64_sys_ioctl+0x82/0xb0
[Tue Oct 26 12:27:42 2021]  do_syscall_64+0x5c/0x80
[Tue Oct 26 12:27:42 2021]  ? create_task_io_context+0xc7/0x110
[Tue Oct 26 12:27:42 2021]  ? get_task_io_context+0x48/0x80
[Tue Oct 26 12:27:42 2021]  ? set_task_ioprio+0x97/0xa0
[Tue Oct 26 12:27:42 2021]  ? __do_sys_ioprio_set+0x5e/0x300
[Tue Oct 26 12:27:42 2021]  ? syscall_exit_to_user_mode+0x23/0x40
[Tue Oct 26 12:27:42 2021]  ? syscall_exit_to_user_mode+0x23/0x40
[Tue Oct 26 12:27:42 2021]  ? do_syscall_64+0x69/0x80
[Tue Oct 26 12:27:42 2021]  ? exit_to_user_mode_prepare+0x77/0x170
[Tue Oct 26 12:27:42 2021]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[Tue Oct 26 12:27:42 2021] RIP: 0033:0x7fba22e2559b
[Tue Oct 26 12:27:42 2021] RSP: 002b:00007fba22cf7c98 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Tue Oct 26 12:27:42 2021] RAX: ffffffffffffffda RBX: 000056511ecc6bc0 RCX: 00007fba22e2559b
[Tue Oct 26 12:27:42 2021] RDX: 000056511ecc6bc0 RSI: 00000000c400941b RDI: 0000000000000003
[Tue Oct 26 12:27:42 2021] RBP: 0000000000000000 R08: 00007fba22cf8640 R09: 0000000000000000
[Tue Oct 26 12:27:42 2021] R10: 00007fba22cf8640 R11: 0000000000000246 R12: 00007ffec33d061e
[Tue Oct 26 12:27:42 2021] R13: 00007ffec33d061f R14: 0000000000000000 R15: 00007fba22cf8640

Thanks,
Emil

Oct 19, 2021 14:46:46 Qu Wenruo <[email protected]>:

Quoted 122 lines · Emil Heimpel
Quoted 4 lines

So it finished after 2 minutes?

[Tue Oct 19 14:13:51 2021] BTRFS info (device sde1): continuing dev_replace from <missing disk> (devid 1) to target /dev/sde1 @74%
[Tue Oct 19 14:15:39 2021] BTRFS info (device sde1): dev_replace from <missing disk> (devid 1) to /dev/sde1 finished

Then this means it previously only have the dev replace status not cleaned up.

Thus it ends pretty quick.

Quoted 13 lines

Now I at least have an expected filesystem show:

Label: 'BlueButter'  uuid: 7e3378e6-da46-4a60-b9b8-1bcc306986e3 Total devices 6 FS bytes used 20.96TiB
        devid    1 size 7.28TiB used 5.46TiB path /dev/sde1 devid    2 size 7.28TiB used 5.46TiB path /dev/sdb1 devid    3 size 2.73TiB used 2.73TiB path /dev/sdg1 devid    4 size 2.73TiB used 2.73TiB path /dev/sdd1 devid    5 size 7.28TiB used 4.81TiB path /dev/sdf1 devid    6 size 7.28TiB used 5.33TiB path /dev/sdc1

Quoted 1 line

And a nondegraded remount worked too.

And it's time for a full fs scrub to find out how consistent the fs is.

For btrfs RAID56, every time a unexpected shutdown is hit, a scrub is strongly recommended.

And even routine scrub would be a plus for btrfs RAID56.

Thanks,
Qu

Quoted 85 lines

Thanks,
Emil

Oct 19, 2021 14:20:21 Qu Wenruo <[email protected]>:

Quoted 79 lines · Emil Heimpel
Quoted 12 lines

Color me suprised:

[74713.072745] BTRFS info (device sde1): flagging fs with big metadata feature
[74713.072755] BTRFS info (device sde1): allowing degraded mounts
[74713.072758] BTRFS info (device sde1): using free space tree
[74713.072760] BTRFS info (device sde1): has skinny extents
[74713.104297] BTRFS warning (device sde1): devid 1 uuid 51645efd-bf95-458d-b5ae-b31623533abb is missing
[74714.675001] BTRFS info (device sde1): bdev (efault) errs: wr 52950, rd 8161, flush 0, corrupt 1221, gen 0
[74714.675015] BTRFS info (device sde1): bdev /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 228, gen 0
[74714.675025] BTRFS info (device sde1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 140, gen 0
[74751.033383] BTRFS info (device sde1): continuing dev_replace from <missing disk> (devid 1) to target /dev/sde1 @74%
Quoted 4 lines

[bluemond@BlueQ ~]$ sudo btrfs replace status  -1 /mnt/btrfsrepair/ 74.9% done, 0 write errs, 0 uncorr. read errs

I guess I just wait?

Yep, wait and stay alert, better to also keep an eye on the dmesg.

But this also means, previous replace didn't really finish, which may mean the replace ioctl is not reporting the proper status, and can be a possible bug.

Thanks,
Qu

Quoted 50 lines

Oct 19, 2021 13:37:09 Qu Wenruo <[email protected]>:

Quoted 47 lines · Emil Heimpel
Quoted 1 line

…

*failed*...

Quoted 1 line

…

And ATA commands failure.

I don't believe the replace finished without problem, and the involved device is /dev/sdd.

Quoted 1 line

…

You won't want to see this message at all.

This means, you're running RAID56, as btrfs has write-hole problem, which will degrade the robust of RAID56 byte by byte for each unclean shutdown.

I guess the write hole problem has already make the repair failed for the replace.

Thus after a successful mount, scrub and manually file checking is almost a must.

Quoted 1 line

…

This doesn't sound correct.

If a device is properly replaced, it should have the same devid number.

I guess you have tried to add a new device before, and then tried to replace the missing device, right?

Anyway, have you tried to mount it degraded and then remove the missing device?

Since you're using RAID56, I guess degrade mount should work.

Thanks,
Qu

Quoted 1 line

…

#10 lore raw
Quoted 3 lines · Emil Heimpel

After reapplying the thermal paste and adding a fan to my storage controller (LSI SAS2008) I haven't encountered any abnormal behavior of my drives. Scrub is running (disk by disk) and everything looks fine so far.

I only got one task blocked message in dmesg, is that anything I should worry about?

Sometimes we can also hit such blocked message for certain test cases.

As long as the fs is still submitting IO, and you find proper CPU usage, it should be fine.

Thanks,
Qu

Quoted 217 lines

Sun Oct 24 18:22:11 2021] BTRFS info (device sdg1): scrub: started on devid 5

[Mon Oct 25 15:31:23 2021] BTRFS info (device sdg1): scrub: finished on devid 5 with status: 0
[Mon Oct 25 15:31:26 2021] BTRFS info (device sdg1): scrub: started on devid 3
[Mon Oct 25 21:12:34 2021] perf: interrupt took too long (2501 > 2500), lowering kernel.perf_event_max_sample_rate to 79800
[Mon Oct 25 23:01:43 2021] hrtimer: interrupt took 13811 ns
[Tue Oct 26 02:58:17 2021] BTRFS info (device sdg1): scrub: finished on devid 3 with status: 0
[Tue Oct 26 02:58:21 2021] BTRFS info (device sdg1): scrub: started on devid 6
[Tue Oct 26 12:23:36 2021] INFO: task btrfs:341674 blocked for more than 122 seconds.
[Tue Oct 26 12:23:36 2021]       Tainted: G           OE     5.14.14-arch1-1 #1
[Tue Oct 26 12:23:36 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Tue Oct 26 12:23:36 2021] task:btrfs           state:D stack:    0 pid:341674 ppid:341658 flags:0x00004000
[Tue Oct 26 12:23:36 2021] Call Trace:
[Tue Oct 26 12:23:36 2021]  __schedule+0x333/0x1530
[Tue Oct 26 12:23:36 2021]  ? psi_task_switch+0xc2/0x1f0
[Tue Oct 26 12:23:36 2021]  ? autoremove_wake_function+0x2c/0x50
[Tue Oct 26 12:23:36 2021]  schedule+0x59/0xc0
[Tue Oct 26 12:23:36 2021]  __scrub_blocked_if_needed+0xa0/0xf0 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:23:36 2021]  ? do_wait_intr_irq+0xa0/0xa0
[Tue Oct 26 12:23:36 2021]  scrub_pause_off+0x21/0x50 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:23:36 2021]  scrub_stripe+0x452/0x1580 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:23:36 2021]  ? do_wait_intr_irq+0xa0/0xa0
[Tue Oct 26 12:23:36 2021]  ? __btrfs_end_transaction+0xf6/0x210 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:23:36 2021]  ? scrub_chunk+0xcd/0x130 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:23:36 2021]  scrub_chunk+0xcd/0x130 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:23:36 2021]  scrub_enumerate_chunks+0x354/0x790 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:23:36 2021]  ? do_wait_intr_irq+0xa0/0xa0
[Tue Oct 26 12:23:36 2021]  btrfs_scrub_dev+0x23d/0x570 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:23:36 2021]  btrfs_ioctl+0x1410/0x2df0 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:23:36 2021]  ? __x64_sys_ioctl+0x82/0xb0
[Tue Oct 26 12:23:36 2021]  __x64_sys_ioctl+0x82/0xb0
[Tue Oct 26 12:23:36 2021]  do_syscall_64+0x5c/0x80
[Tue Oct 26 12:23:36 2021]  ? create_task_io_context+0xc7/0x110
[Tue Oct 26 12:23:36 2021]  ? get_task_io_context+0x48/0x80
[Tue Oct 26 12:23:36 2021]  ? set_task_ioprio+0x97/0xa0
[Tue Oct 26 12:23:36 2021]  ? __do_sys_ioprio_set+0x5e/0x300
[Tue Oct 26 12:23:36 2021]  ? syscall_exit_to_user_mode+0x23/0x40
[Tue Oct 26 12:23:36 2021]  ? syscall_exit_to_user_mode+0x23/0x40
[Tue Oct 26 12:23:36 2021]  ? do_syscall_64+0x69/0x80
[Tue Oct 26 12:23:36 2021]  ? exit_to_user_mode_prepare+0x77/0x170
[Tue Oct 26 12:23:36 2021]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[Tue Oct 26 12:23:36 2021] RIP: 0033:0x7fba22e2559b
[Tue Oct 26 12:23:36 2021] RSP: 002b:00007fba22cf7c98 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Tue Oct 26 12:23:36 2021] RAX: ffffffffffffffda RBX: 000056511ecc6bc0 RCX: 00007fba22e2559b
[Tue Oct 26 12:23:36 2021] RDX: 000056511ecc6bc0 RSI: 00000000c400941b RDI: 0000000000000003
[Tue Oct 26 12:23:36 2021] RBP: 0000000000000000 R08: 00007fba22cf8640 R09: 0000000000000000
[Tue Oct 26 12:23:36 2021] R10: 00007fba22cf8640 R11: 0000000000000246 R12: 00007ffec33d061e
[Tue Oct 26 12:23:36 2021] R13: 00007ffec33d061f R14: 0000000000000000 R15: 00007fba22cf8640
[Tue Oct 26 12:27:42 2021] INFO: task btrfs:341674 blocked for more than 122 seconds.
[Tue Oct 26 12:27:42 2021]       Tainted: G           OE     5.14.14-arch1-1 #1
[Tue Oct 26 12:27:42 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Tue Oct 26 12:27:42 2021] task:btrfs           state:D stack:    0 pid:341674 ppid:341658 flags:0x00004000
[Tue Oct 26 12:27:42 2021] Call Trace:
[Tue Oct 26 12:27:42 2021]  __schedule+0x333/0x1530
[Tue Oct 26 12:27:42 2021]  ? autoremove_wake_function+0x2c/0x50
[Tue Oct 26 12:27:42 2021]  schedule+0x59/0xc0
[Tue Oct 26 12:27:42 2021]  __scrub_blocked_if_needed+0xa0/0xf0 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:27:42 2021]  ? do_wait_intr_irq+0xa0/0xa0
[Tue Oct 26 12:27:42 2021]  scrub_pause_off+0x21/0x50 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:27:42 2021]  scrub_stripe+0x452/0x1580 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:27:42 2021]  ? __update_load_avg_cfs_rq+0x27d/0x2e0
[Tue Oct 26 12:27:42 2021]  ? __btrfs_end_transaction+0xf6/0x210 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:27:42 2021]  ? kmem_cache_free+0x107/0x410
[Tue Oct 26 12:27:42 2021]  ? scrub_chunk+0xcd/0x130 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:27:42 2021]  scrub_chunk+0xcd/0x130 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:27:42 2021]  scrub_enumerate_chunks+0x354/0x790 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:27:42 2021]  ? do_wait_intr_irq+0xa0/0xa0
[Tue Oct 26 12:27:42 2021]  btrfs_scrub_dev+0x23d/0x570 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:27:42 2021]  btrfs_ioctl+0x1410/0x2df0 [btrfs 796168a0fefcfb1a0bc1ff664fee7292a977f032]
[Tue Oct 26 12:27:42 2021]  ? __x64_sys_ioctl+0x82/0xb0
[Tue Oct 26 12:27:42 2021]  __x64_sys_ioctl+0x82/0xb0
[Tue Oct 26 12:27:42 2021]  do_syscall_64+0x5c/0x80
[Tue Oct 26 12:27:42 2021]  ? create_task_io_context+0xc7/0x110
[Tue Oct 26 12:27:42 2021]  ? get_task_io_context+0x48/0x80
[Tue Oct 26 12:27:42 2021]  ? set_task_ioprio+0x97/0xa0
[Tue Oct 26 12:27:42 2021]  ? __do_sys_ioprio_set+0x5e/0x300
[Tue Oct 26 12:27:42 2021]  ? syscall_exit_to_user_mode+0x23/0x40
[Tue Oct 26 12:27:42 2021]  ? syscall_exit_to_user_mode+0x23/0x40
[Tue Oct 26 12:27:42 2021]  ? do_syscall_64+0x69/0x80
[Tue Oct 26 12:27:42 2021]  ? exit_to_user_mode_prepare+0x77/0x170
[Tue Oct 26 12:27:42 2021]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[Tue Oct 26 12:27:42 2021] RIP: 0033:0x7fba22e2559b
[Tue Oct 26 12:27:42 2021] RSP: 002b:00007fba22cf7c98 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Tue Oct 26 12:27:42 2021] RAX: ffffffffffffffda RBX: 000056511ecc6bc0 RCX: 00007fba22e2559b
[Tue Oct 26 12:27:42 2021] RDX: 000056511ecc6bc0 RSI: 00000000c400941b RDI: 0000000000000003
[Tue Oct 26 12:27:42 2021] RBP: 0000000000000000 R08: 00007fba22cf8640 R09: 0000000000000000
[Tue Oct 26 12:27:42 2021] R10: 00007fba22cf8640 R11: 0000000000000246 R12: 00007ffec33d061e
[Tue Oct 26 12:27:42 2021] R13: 00007ffec33d061f R14: 0000000000000000 R15: 00007fba22cf8640

Thanks,
Emil

Oct 19, 2021 14:46:46 Qu Wenruo <[email protected]>:

Quoted 122 lines · Emil Heimpel
Quoted 4 lines

So it finished after 2 minutes?

[Tue Oct 19 14:13:51 2021] BTRFS info (device sde1): continuing dev_replace from <missing disk> (devid 1) to target /dev/sde1 @74%
[Tue Oct 19 14:15:39 2021] BTRFS info (device sde1): dev_replace from <missing disk> (devid 1) to /dev/sde1 finished

Then this means it previously only have the dev replace status not cleaned up.

Thus it ends pretty quick.

Quoted 13 lines

Now I at least have an expected filesystem show:

Label: 'BlueButter'  uuid: 7e3378e6-da46-4a60-b9b8-1bcc306986e3
        Total devices 6 FS bytes used 20.96TiB
        devid    1 size 7.28TiB used 5.46TiB path /dev/sde1 devid    2 size 7.28TiB used 5.46TiB path /dev/sdb1 devid    3 size 2.73TiB used 2.73TiB path /dev/sdg1 devid    4 size 2.73TiB used 2.73TiB path /dev/sdd1 devid    5 size 7.28TiB used 4.81TiB path /dev/sdf1 devid    6 size 7.28TiB used 5.33TiB path /dev/sdc1

Quoted 1 line

And a nondegraded remount worked too.

And it's time for a full fs scrub to find out how consistent the fs is.

For btrfs RAID56, every time a unexpected shutdown is hit, a scrub is strongly recommended.

And even routine scrub would be a plus for btrfs RAID56.

Thanks,
Qu

Quoted 85 lines

Thanks,
Emil

Oct 19, 2021 14:20:21 Qu Wenruo <[email protected]>:

Quoted 79 lines · Emil Heimpel
Quoted 12 lines

Color me suprised:

[74713.072745] BTRFS info (device sde1): flagging fs with big metadata feature
[74713.072755] BTRFS info (device sde1): allowing degraded mounts
[74713.072758] BTRFS info (device sde1): using free space tree
[74713.072760] BTRFS info (device sde1): has skinny extents
[74713.104297] BTRFS warning (device sde1): devid 1 uuid 51645efd-bf95-458d-b5ae-b31623533abb is missing
[74714.675001] BTRFS info (device sde1): bdev (efault) errs: wr 52950, rd 8161, flush 0, corrupt 1221, gen 0
[74714.675015] BTRFS info (device sde1): bdev /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 228, gen 0
[74714.675025] BTRFS info (device sde1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 140, gen 0
[74751.033383] BTRFS info (device sde1): continuing dev_replace from <missing disk> (devid 1) to target /dev/sde1 @74%
Quoted 4 lines

[bluemond@BlueQ ~]$ sudo btrfs replace status  -1 /mnt/btrfsrepair/ 74.9% done, 0 write errs, 0 uncorr. read errs

I guess I just wait?

Yep, wait and stay alert, better to also keep an eye on the dmesg.

But this also means, previous replace didn't really finish, which may mean the replace ioctl is not reporting the proper status, and can be a possible bug.

Thanks,
Qu

Quoted 50 lines

Oct 19, 2021 13:37:09 Qu Wenruo <[email protected]>:

Quoted 47 lines · Emil Heimpel
Quoted 1 line

…

*failed*...

Quoted 1 line

…

And ATA commands failure.

I don't believe the replace finished without problem, and the involved device is /dev/sdd.

Quoted 1 line

…

You won't want to see this message at all.

This means, you're running RAID56, as btrfs has write-hole problem, which will degrade the robust of RAID56 byte by byte for each unclean shutdown.

I guess the write hole problem has already make the repair failed for the replace.

Thus after a successful mount, scrub and manually file checking is almost a must.

Quoted 1 line

…

This doesn't sound correct.

If a device is properly replaced, it should have the same devid number.

I guess you have tried to add a new device before, and then tried to replace the missing device, right?

Anyway, have you tried to mount it degraded and then remove the missing device?

Since you're using RAID56, I guess degrade mount should work.

Thanks,
Qu

Quoted 1 line

…