Skip to contentSkip to conversation

Cannot 'mount -o degraded /dev/replacement' after a replace

DISCUSSION Jakob Schöttl 4 participants 8 messages Feb 9, 2019 – Feb 11, 2019 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi,

I've setup a RAID1 with two disks (disk1 and disk2) and I'm testing the btrfs replace command.

After replacing disk2 with disk3, I can only mount
(a) disk1 or disk3 (if both disk are plugged) and
(b) the original disk1 (degraded, if disk3 is unplugged).

I cannot mount the replacement disk3 if disk1 is unplugged.

Quoted 1 line

mount: /mnt: wrong fs type, bad option, bad superblock on /dev/loop3, missing codepage or helper program, or other error.

What I expect is that both disk1 and disk3 are fully valid and working after a replace.

Steps to reproduce:

  dd if=/dev/zero of=/vdisk1 bs=1024 count=300000
  losetup /dev/loop1 /vdisk1
  dd if=/dev/zero of=/vdisk2 bs=1024 count=300000
  losetup /dev/loop2 /vdisk2
  dd if=/dev/zero of=/vdisk3 bs=1024 count=300000

  # losetup /dev/loop3 /vdisk3    # don't plug this device yet

Create RAID1 file system:

  mkfs.btrfs -L datavol -m raid1 -d raid1 /dev/loop1 /dev/loop2

Unplug device 2 to simulate a defect:

  losetup -d /dev/loop2

Plug device 3:

  losetup /dev/loop3 /vdisk3

Replace device 2 with device 3:

  mount -o degraded /dev/loop1 /mnt
  btrfs filesystem show   # to get devid of device 2 btrfs replace start -Br 2 /dev/loop3 /mnt btrfs replace status /mnt   # check success umount /mnt

Unplug the original device 1 to see if device 3 has really replaced device 2:

  losetup -d /dev/loop1
  mount -o degraded /dev/loop3 /mnt

The mount fails with this error:

Quoted 1 line

mount: /mnt: wrong fs type, bad option, bad superblock on /dev/loop3, missing codepage or helper program, or other error.

In this situation, btrfs device scan does not change anything and btrfs filesystem show shows:

Quoted 10 lines

warning, device 1 is missing
warning, device 1 is missing
warning, device 1 is missing
warning, device 1 is missing
bad tree block 198180864, bytenr mismatch, want=198180864, have=0
ERROR: cannot read chunk root
Label: 'datavol'  uuid: 640e45d3-e741-4a78-a24e-2d8a41c6b8c3 Total devices 2 FS bytes used 128.00KiB
    devid    2 size 292.97MiB used 104.00MiB path /dev/loop3 *** Some devices missing

Is this a known problem? Can you reproduce it? Am I doing something wrong?

Regards, Jakob

#1 lore raw
Quoted 10 lines · Jakob Schöttl

Hi,

I've setup a RAID1 with two disks (disk1 and disk2) and I'm testing the btrfs replace command.

After replacing disk2 with disk3, I can only mount
(a) disk1 or disk3 (if both disk are plugged) and
(b) the original disk1 (degraded, if disk3 is unplugged).

I cannot mount the replacement disk3 if disk1 is unplugged.

Sounds like there is one single chunk on disk1, which caused the problem.

Quoted 3 lines
Quoted 2 lines

mount: /mnt: wrong fs type, bad option, bad superblock on /dev/loop3,

missing codepage or helper program, or other error.

dmesg please.

And btrfs-progs version please.

Maybe mkfs is too old to leave SINGLE profile chunks on the original fs.

And you could verify the chunk mapping by executing 'btrfs ins dump-tree -t chunk <device>' and paste the output.

Thanks,
Qu

Quoted 62 lines

What I expect is that both disk1 and disk3 are fully valid and working after a replace.

Steps to reproduce:

  dd if=/dev/zero of=/vdisk1 bs=1024 count=300000
  losetup /dev/loop1 /vdisk1
  dd if=/dev/zero of=/vdisk2 bs=1024 count=300000
  losetup /dev/loop2 /vdisk2
  dd if=/dev/zero of=/vdisk3 bs=1024 count=300000

  # losetup /dev/loop3 /vdisk3    # don't plug this device yet

Create RAID1 file system:

  mkfs.btrfs -L datavol -m raid1 -d raid1 /dev/loop1 /dev/loop2

Unplug device 2 to simulate a defect:

  losetup -d /dev/loop2

Plug device 3:

  losetup /dev/loop3 /vdisk3

Replace device 2 with device 3:

  mount -o degraded /dev/loop1 /mnt
  btrfs filesystem show   # to get devid of device 2 btrfs replace start -Br 2 /dev/loop3 /mnt btrfs replace status /mnt   # check success umount /mnt

Unplug the original device 1 to see if device 3 has really replaced device 2:

  losetup -d /dev/loop1
  mount -o degraded /dev/loop3 /mnt

The mount fails with this error:

Quoted 1 line

mount: /mnt: wrong fs type, bad option, bad superblock on /dev/loop3,

missing codepage or helper program, or other error.

In this situation, btrfs device scan does not change anything and btrfs filesystem show shows:

Quoted 10 lines

warning, device 1 is missing
warning, device 1 is missing
warning, device 1 is missing
warning, device 1 is missing
bad tree block 198180864, bytenr mismatch, want=198180864, have=0
ERROR: cannot read chunk root
Label: 'datavol'  uuid: 640e45d3-e741-4a78-a24e-2d8a41c6b8c3 Total devices 2 FS bytes used 128.00KiB
    devid    2 size 292.97MiB used 104.00MiB path /dev/loop3 *** Some devices missing

Is this a known problem? Can you reproduce it? Am I doing something wrong?

Regards, Jakob

#2 lore raw

Thanks Qu,

Am 09.02.19 um 13:16 schrieb Qu Wenruo:

Quoted 10 lines · Jakob Schöttl
Quoted 9 lines

Hi,

I've setup a RAID1 with two disks (disk1 and disk2) and I'm testing the btrfs replace command.

After replacing disk2 with disk3, I can only mount
(a) disk1 or disk3 (if both disk are plugged) and
(b) the original disk1 (degraded, if disk3 is unplugged).

Quoted 6 lines
Quoted 1 line

I cannot mount the replacement disk3 if disk1 is unplugged.

Sounds like there is one single chunk on disk1, which caused the problem.

Quoted 2 lines
Quoted 1 line

mount: /mnt: wrong fs type, bad option, bad superblock on /dev/loop3,

missing codepage or helper program, or other error.

dmesg please.

[60456.856883] BTRFS: device label datavol devid 1 transid 5 /dev/loop1
[60456.940785] BTRFS: device label datavol devid 2 transid 5 /dev/loop2
[60525.211389] BTRFS info (device loop1): allowing degraded mounts
[60525.211395] BTRFS info (device loop1): disk space caching is enabled
[60525.211398] BTRFS info (device loop1): has skinny extents
[60525.211401] BTRFS info (device loop1): flagging fs with big metadata feature
[60525.213854] BTRFS warning (device loop1): devid 2 uuid 0b4e0b31-e2b1-40a0-8360-09978f58a2e4 is missing
[60525.214639] BTRFS info (device loop1): checking UUID tree
[60525.386695] BTRFS info (device loop1): dev_replace from <missing disk> (devid 2) to /dev/loop3 started
[60525.394403] BTRFS info (device loop1): dev_replace from <missing disk> (devid 2) to /dev/loop3 finished
[60533.721841] BTRFS info (device loop3): allowing degraded mounts
[60533.721846] BTRFS info (device loop3): disk space caching is enabled
[60533.721850] BTRFS info (device loop3): has skinny extents
[60533.723703] BTRFS error (device loop3): failed to read chunk root
[60533.773553] BTRFS error (device loop3): open_ctree failed
Quoted 1 line

And btrfs-progs version please.

$ pacman -Q btrfs-progs
btrfs-progs 4.20.1-2
$ uname -a
Linux jathink 4.20.7-arch1-1-ARCH #1 SMP PREEMPT Wed Feb 6 18:42:40 UTC 2019 x86_64 GNU/Linux
Quoted 4 lines

Maybe mkfs is too old to leave SINGLE profile chunks on the original fs.

And you could verify the chunk mapping by executing 'btrfs ins dump-tree -t chunk <device>' and paste the output.

When only /dev/loop3 is plugged:

# btrfs inspect-internal dump-tree -t chunk /dev/loop3
btrfs-progs v4.20.1
warning, device 1 is missing
warning, device 1 is missing
warning, device 1 is missing
warning, device 1 is missing
bad tree block 198180864, bytenr mismatch, want=198180864, have=0
ERROR: cannot read chunk root
ERROR: unable to open /dev/loop3

When only /dev/loop1 is plugged:

# btrfs inspect-internal dump-tree -t chunk /dev/loop1
btrfs-progs v4.20.1
warning, device 2 is missing
chunk tree
leaf 198180864 items 10 free space 15005 generation 8 owner CHUNK_TREE
leaf 198180864 flags 0x1(WRITTEN) backref revision 1
fs uuid 005a8d59-a561-4371-869e-b0ccc4a4862b
chunk uuid b3e609f1-a7fe-4add-bc51-6231f0bbf320
    item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
        devid 1 total_bytes 307200000 bytes_used 306053120
        io_align 4096 io_width 4096 sector_size 4096 type 0
        generation 0 start_offset 0 dev_group 0
        seek_speed 0 bandwidth 0
        uuid 043443c7-ac91-4085-a5e4-983b59dd0803
        fsid 005a8d59-a561-4371-869e-b0ccc4a4862b
    item 1 key (DEV_ITEMS DEV_ITEM 2) itemoff 16087 itemsize 98
        devid 2 total_bytes 307200000 bytes_used 109051904
        io_align 4096 io_width 4096 sector_size 4096 type 0
        generation 0 start_offset 0 dev_group 0
        seek_speed 0 bandwidth 0
        uuid 0b4e0b31-e2b1-40a0-8360-09978f58a2e4
        fsid 005a8d59-a561-4371-869e-b0ccc4a4862b
    item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975 itemsize 112
        length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
        io_align 65536 io_width 65536 sector_size 4096
        num_stripes 2 sub_stripes 0
            stripe 0 devid 2 offset 1048576
            dev_uuid 0b4e0b31-e2b1-40a0-8360-09978f58a2e4
            stripe 1 devid 1 offset 22020096
            dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
    item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863 itemsize 112
        length 33554432 owner 2 stripe_len 65536 type METADATA|RAID1
        io_align 65536 io_width 65536 sector_size 4096
        num_stripes 2 sub_stripes 0
            stripe 0 devid 2 offset 9437184
            dev_uuid 0b4e0b31-e2b1-40a0-8360-09978f58a2e4
            stripe 1 devid 1 offset 30408704
            dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
    item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 63963136) itemoff 15751 itemsize 112
        length 67108864 owner 2 stripe_len 65536 type DATA|RAID1
        io_align 65536 io_width 65536 sector_size 4096
        num_stripes 2 sub_stripes 0
            stripe 0 devid 2 offset 42991616
            dev_uuid 0b4e0b31-e2b1-40a0-8360-09978f58a2e4
            stripe 1 devid 1 offset 63963136
            dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
    item 5 key (FIRST_CHUNK_TREE CHUNK_ITEM 131072000) itemoff 15671 itemsize 80
        length 33554432 owner 2 stripe_len 65536 type METADATA
        io_align 65536 io_width 65536 sector_size 4096
        num_stripes 1 sub_stripes 1
            stripe 0 devid 1 offset 131072000
            dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
    item 6 key (FIRST_CHUNK_TREE CHUNK_ITEM 164626432) itemoff 15591 itemsize 80
        length 33554432 owner 2 stripe_len 65536 type SYSTEM
        io_align 65536 io_width 65536 sector_size 4096
        num_stripes 1 sub_stripes 1
            stripe 0 devid 1 offset 164626432
            dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
    item 7 key (FIRST_CHUNK_TREE CHUNK_ITEM 198180864) itemoff 15479 itemsize 112
        length 33554432 owner 2 stripe_len 65536 type SYSTEM|DUP
        io_align 65536 io_width 65536 sector_size 4096
        num_stripes 2 sub_stripes 1
            stripe 0 devid 1 offset 198180864
            dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
            stripe 1 devid 1 offset 231735296
            dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
    item 8 key (FIRST_CHUNK_TREE CHUNK_ITEM 231735296) itemoff 15367 itemsize 112
        length 20905984 owner 2 stripe_len 65536 type METADATA|DUP
        io_align 65536 io_width 65536 sector_size 4096
        num_stripes 2 sub_stripes 1
            stripe 0 devid 1 offset 265289728
            dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
            stripe 1 devid 1 offset 286195712
            dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
    item 9 key (FIRST_CHUNK_TREE CHUNK_ITEM 252641280) itemoff 15255 itemsize 112
        length 10485760 owner 2 stripe_len 65536 type DATA|DUP
        io_align 65536 io_width 65536 sector_size 4096
        num_stripes 2 sub_stripes 1
            stripe 0 devid 1 offset 1048576
            dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
            stripe 1 devid 1 offset 11534336
            dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
#3 lore raw

09.02.2019 16:45, Jakob Schöttl пишет:

Quoted 147 lines

Thanks Qu,

Am 09.02.19 um 13:16 schrieb Qu Wenruo:

Quoted 10 lines · Jakob Schöttl
Quoted 9 lines

Hi,

I've setup a RAID1 with two disks (disk1 and disk2) and I'm testing the btrfs replace command.

After replacing disk2 with disk3, I can only mount
(a) disk1 or disk3 (if both disk are plugged) and
(b) the original disk1 (degraded, if disk3 is unplugged).

Quoted 6 lines
Quoted 1 line

I cannot mount the replacement disk3 if disk1 is unplugged.

Sounds like there is one single chunk on disk1, which caused the problem.

Quoted 2 lines
Quoted 1 line

mount: /mnt: wrong fs type, bad option, bad superblock on /dev/loop3,

missing codepage or helper program, or other error.

dmesg please.

[60456.856883] BTRFS: device label datavol devid 1 transid 5 /dev/loop1
[60456.940785] BTRFS: device label datavol devid 2 transid 5 /dev/loop2
[60525.211389] BTRFS info (device loop1): allowing degraded mounts
[60525.211395] BTRFS info (device loop1): disk space caching is enabled
[60525.211398] BTRFS info (device loop1): has skinny extents
[60525.211401] BTRFS info (device loop1): flagging fs with big metadata
feature
[60525.213854] BTRFS warning (device loop1): devid 2 uuid
0b4e0b31-e2b1-40a0-8360-09978f58a2e4 is missing
[60525.214639] BTRFS info (device loop1): checking UUID tree
[60525.386695] BTRFS info (device loop1): dev_replace from <missing
disk> (devid 2) to /dev/loop3 started
[60525.394403] BTRFS info (device loop1): dev_replace from <missing
disk> (devid 2) to /dev/loop3 finished
[60533.721841] BTRFS info (device loop3): allowing degraded mounts
[60533.721846] BTRFS info (device loop3): disk space caching is enabled
[60533.721850] BTRFS info (device loop3): has skinny extents
[60533.723703] BTRFS error (device loop3): failed to read chunk root
[60533.773553] BTRFS error (device loop3): open_ctree failed
Quoted 1 line

And btrfs-progs version please.

$ pacman -Q btrfs-progs
btrfs-progs 4.20.1-2
$ uname -a
Linux jathink 4.20.7-arch1-1-ARCH #1 SMP PREEMPT Wed Feb 6 18:42:40 UTC
2019 x86_64 GNU/Linux
Quoted 4 lines

Maybe mkfs is too old to leave SINGLE profile chunks on the original fs.

And you could verify the chunk mapping by executing 'btrfs ins dump-tree -t chunk <device>' and paste the output.

When only /dev/loop3 is plugged:

# btrfs inspect-internal dump-tree -t chunk /dev/loop3
btrfs-progs v4.20.1
warning, device 1 is missing
warning, device 1 is missing
warning, device 1 is missing
warning, device 1 is missing
bad tree block 198180864, bytenr mismatch, want=198180864, have=0
ERROR: cannot read chunk root
ERROR: unable to open /dev/loop3

When only /dev/loop1 is plugged:

# btrfs inspect-internal dump-tree -t chunk /dev/loop1
btrfs-progs v4.20.1
warning, device 2 is missing
chunk tree
leaf 198180864 items 10 free space 15005 generation 8 owner CHUNK_TREE
leaf 198180864 flags 0x1(WRITTEN) backref revision 1
fs uuid 005a8d59-a561-4371-869e-b0ccc4a4862b
chunk uuid b3e609f1-a7fe-4add-bc51-6231f0bbf320
    item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
        devid 1 total_bytes 307200000 bytes_used 306053120
        io_align 4096 io_width 4096 sector_size 4096 type 0
        generation 0 start_offset 0 dev_group 0
        seek_speed 0 bandwidth 0
        uuid 043443c7-ac91-4085-a5e4-983b59dd0803
        fsid 005a8d59-a561-4371-869e-b0ccc4a4862b
    item 1 key (DEV_ITEMS DEV_ITEM 2) itemoff 16087 itemsize 98
        devid 2 total_bytes 307200000 bytes_used 109051904
        io_align 4096 io_width 4096 sector_size 4096 type 0
        generation 0 start_offset 0 dev_group 0
        seek_speed 0 bandwidth 0
        uuid 0b4e0b31-e2b1-40a0-8360-09978f58a2e4
        fsid 005a8d59-a561-4371-869e-b0ccc4a4862b
    item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975
itemsize 112
        length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
        io_align 65536 io_width 65536 sector_size 4096
        num_stripes 2 sub_stripes 0
            stripe 0 devid 2 offset 1048576
            dev_uuid 0b4e0b31-e2b1-40a0-8360-09978f58a2e4
            stripe 1 devid 1 offset 22020096
            dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
    item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863
itemsize 112
        length 33554432 owner 2 stripe_len 65536 type METADATA|RAID1
        io_align 65536 io_width 65536 sector_size 4096
        num_stripes 2 sub_stripes 0
            stripe 0 devid 2 offset 9437184
            dev_uuid 0b4e0b31-e2b1-40a0-8360-09978f58a2e4
            stripe 1 devid 1 offset 30408704
            dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
    item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 63963136) itemoff 15751
itemsize 112
        length 67108864 owner 2 stripe_len 65536 type DATA|RAID1
        io_align 65536 io_width 65536 sector_size 4096
        num_stripes 2 sub_stripes 0
            stripe 0 devid 2 offset 42991616
            dev_uuid 0b4e0b31-e2b1-40a0-8360-09978f58a2e4
            stripe 1 devid 1 offset 63963136
            dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
    item 5 key (FIRST_CHUNK_TREE CHUNK_ITEM 131072000) itemoff 15671
itemsize 80
        length 33554432 owner 2 stripe_len 65536 type METADATA
        io_align 65536 io_width 65536 sector_size 4096
        num_stripes 1 sub_stripes 1
            stripe 0 devid 1 offset 131072000
            dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
    item 6 key (FIRST_CHUNK_TREE CHUNK_ITEM 164626432) itemoff 15591
itemsize 80
        length 33554432 owner 2 stripe_len 65536 type SYSTEM
        io_align 65536 io_width 65536 sector_size 4096
        num_stripes 1 sub_stripes 1
            stripe 0 devid 1 offset 164626432
            dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
    item 7 key (FIRST_CHUNK_TREE CHUNK_ITEM 198180864) itemoff 15479
itemsize 112
        length 33554432 owner 2 stripe_len 65536 type SYSTEM|DUP
        io_align 65536 io_width 65536 sector_size 4096
        num_stripes 2 sub_stripes 1
            stripe 0 devid 1 offset 198180864
            dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
            stripe 1 devid 1 offset 231735296
            dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
    item 8 key (FIRST_CHUNK_TREE CHUNK_ITEM 231735296) itemoff 15367
itemsize 112
        length 20905984 owner 2 stripe_len 65536 type METADATA|DUP
        io_align 65536 io_width 65536 sector_size 4096
        num_stripes 2 sub_stripes 1
            stripe 0 devid 1 offset 265289728
            dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
            stripe 1 devid 1 offset 286195712
            dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
    item 9 key (FIRST_CHUNK_TREE CHUNK_ITEM 252641280) itemoff 15255
Quoted 10 lines
itemsize 112
        length 10485760 owner 2 stripe_len 65536 type DATA|DUP
        io_align 65536 io_width 65536 sector_size 4096
        num_stripes 2 sub_stripes 1
            stripe 0 devid 1 offset 1048576
            dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
            stripe 1 devid 1 offset 11534336
            dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803

Yes, I can trivially reproduce it using commands in original post.

Immediately after mkfs:

10:/home/bor # btrfs inspect-internal dump-tree -t chunk /dev/sdb1 btrfs-progs v4.15
warning, device 2 is missing
chunk tree
leaf 22036480 items 5 free space 15626 generation 5 owner 3 leaf 22036480 flags 0x1(WRITTEN) backref revision 1
fs uuid d33e6bff-5903-449d-b49c-f2d992676e30
chunk uuid ed55b2ad-66c4-4764-acd5-7fa3a9fe619e

	item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
		devid 1 total_bytes 5367640064 bytes_used 1081868288
		io_align 4096 io_width 4096 sector_size 4096 type 0
		generation 0 start_offset 0 dev_group 0
		seek_speed 0 bandwidth 0
		uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
		fsid d33e6bff-5903-449d-b49c-f2d992676e30
	item 1 key (DEV_ITEMS DEV_ITEM 2) itemoff 16087 itemsize 98
		devid 2 total_bytes 5367640064 bytes_used 1081868288
		io_align 4096 io_width 4096 sector_size 4096 type 0
		generation 0 start_offset 0 dev_group 0
		seek_speed 0 bandwidth 0
		uuid 1e91c346-eba9-4a41-aeb2-d7efef64a5ea
		fsid d33e6bff-5903-449d-b49c-f2d992676e30
	item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975
itemsize 112
		length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 0
			stripe 0 devid 2 offset 1048576
			dev_uuid 1e91c346-eba9-4a41-aeb2-d7efef64a5ea
			stripe 1 devid 1 offset 22020096
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
	item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863
itemsize 112
		length 536739840 owner 2 stripe_len 65536 type METADATA|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 0
			stripe 0 devid 2 offset 9437184
			dev_uuid 1e91c346-eba9-4a41-aeb2-d7efef64a5ea
			stripe 1 devid 1 offset 30408704
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
	item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 567148544) itemoff 15751
itemsize 112
		length 536739840 owner 2 stripe_len 65536 type DATA|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 0
			stripe 0 devid 2 offset 546177024
			dev_uuid 1e91c346-eba9-4a41-aeb2-d7efef64a5ea
			stripe 1 devid 1 offset 567148544
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6

Immediately after degraded mount

btrfs-progs v4.15
warning, device 2 is missing
chunk tree
leaf 22020096 items 8 free space 15311 generation 6 owner 3 leaf 22020096 flags 0x1(WRITTEN) backref revision 1 fs uuid e69b93ef-d852-4afa-8c74-50fa07d33a01
chunk uuid 74317bdd-f011-44a8-9c62-0cce665f1b52

	item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
		devid 1 total_bytes 5368709120 bytes_used 2994733056
		io_align 4096 io_width 4096 sector_size 4096 type 0
		generation 0 start_offset 0 dev_group 0
		seek_speed 0 bandwidth 0
		uuid 85365dff-ceb8-46bb-a4c5-78ce422a32f8
		fsid e69b93ef-d852-4afa-8c74-50fa07d33a01
	item 1 key (DEV_ITEMS DEV_ITEM 2) itemoff 16087 itemsize 98
		devid 2 total_bytes 5368709120 bytes_used 2155872256
		io_align 4096 io_width 4096 sector_size 4096 type 0
		generation 0 start_offset 0 dev_group 0
		seek_speed 0 bandwidth 0
		uuid 1989d9cd-9f82-4d3a-a577-c7e583157ed3
		fsid e69b93ef-d852-4afa-8c74-50fa07d33a01
	item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975
itemsize 112
		length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 0
			stripe 0 devid 2 offset 1048576
			dev_uuid 1989d9cd-9f82-4d3a-a577-c7e583157ed3
			stripe 1 devid 1 offset 22020096
			dev_uuid 85365dff-ceb8-46bb-a4c5-78ce422a32f8
	item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863
itemsize 112
		length 1073741824 owner 2 stripe_len 65536 type METADATA|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 0
			stripe 0 devid 2 offset 9437184
			dev_uuid 1989d9cd-9f82-4d3a-a577-c7e583157ed3
			stripe 1 devid 1 offset 30408704
			dev_uuid 85365dff-ceb8-46bb-a4c5-78ce422a32f8
	item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 1104150528) itemoff 15751
itemsize 112
		length 1073741824 owner 2 stripe_len 65536 type DATA|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 0
			stripe 0 devid 2 offset 1083179008
			dev_uuid 1989d9cd-9f82-4d3a-a577-c7e583157ed3
			stripe 1 devid 1 offset 1104150528
			dev_uuid 85365dff-ceb8-46bb-a4c5-78ce422a32f8
	item 5 key (FIRST_CHUNK_TREE CHUNK_ITEM 2177892352) itemoff 15671
itemsize 80
		length 268435456 owner 2 stripe_len 65536 type METADATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 2177892352
			dev_uuid 85365dff-ceb8-46bb-a4c5-78ce422a32f8
	item 6 key (FIRST_CHUNK_TREE CHUNK_ITEM 2446327808) itemoff 15591
itemsize 80
		length 33554432 owner 2 stripe_len 65536 type SYSTEM
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 2446327808
			dev_uuid 85365dff-ceb8-46bb-a4c5-78ce422a32f8
	item 7 key (FIRST_CHUNK_TREE CHUNK_ITEM 2479882240) itemoff 15511
itemsize 80
		length 536870912 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 2479882240
			dev_uuid 85365dff-ceb8-46bb-a4c5-78ce422a32f8

After device replacement completed

10:/home/bor # btrfs replace status /mnt
Started on 9.Feb 18:07:18, finished on 9.Feb 18:07:19, 0 write errs, 0 uncorr. read errs
10:/home/bor # btrfs inspect-internal dump-tree -t chunk /dev/sdb1 btrfs-progs v4.15
chunk tree
leaf 1942749184 items 11 free space 14900 generation 9 owner 3 leaf 1942749184 flags 0x1(WRITTEN) backref revision 1
fs uuid d33e6bff-5903-449d-b49c-f2d992676e30
chunk uuid ed55b2ad-66c4-4764-acd5-7fa3a9fe619e

	item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
		devid 1 total_bytes 5367640064 bytes_used 2793144320
		io_align 4096 io_width 4096 sector_size 4096 type 0
		generation 0 start_offset 0 dev_group 0
		seek_speed 0 bandwidth 0
		uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
		fsid d33e6bff-5903-449d-b49c-f2d992676e30
	item 1 key (DEV_ITEMS DEV_ITEM 2) itemoff 16087 itemsize 98
		devid 2 total_bytes 5367640064 bytes_used 1081868288
		io_align 4096 io_width 4096 sector_size 4096 type 0
		generation 0 start_offset 0 dev_group 0
		seek_speed 0 bandwidth 0
		uuid 1e91c346-eba9-4a41-aeb2-d7efef64a5ea
		fsid d33e6bff-5903-449d-b49c-f2d992676e30
	item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975
itemsize 112
		length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 0
			stripe 0 devid 2 offset 1048576
			dev_uuid 1e91c346-eba9-4a41-aeb2-d7efef64a5ea
			stripe 1 devid 1 offset 22020096
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
	item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863
itemsize 112
		length 536739840 owner 2 stripe_len 65536 type METADATA|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 0
			stripe 0 devid 2 offset 9437184
			dev_uuid 1e91c346-eba9-4a41-aeb2-d7efef64a5ea
			stripe 1 devid 1 offset 30408704
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
	item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 567148544) itemoff 15751
itemsize 112
		length 536739840 owner 2 stripe_len 65536 type DATA|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 0
			stripe 0 devid 2 offset 546177024
			dev_uuid 1e91c346-eba9-4a41-aeb2-d7efef64a5ea
			stripe 1 devid 1 offset 567148544
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
	item 5 key (FIRST_CHUNK_TREE CHUNK_ITEM 1103888384) itemoff 15671
itemsize 80
		length 268435456 owner 2 stripe_len 65536 type METADATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 1103888384
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
	item 6 key (FIRST_CHUNK_TREE CHUNK_ITEM 1372323840) itemoff 15591
itemsize 80
		length 33554432 owner 2 stripe_len 65536 type SYSTEM
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 1372323840
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
	item 7 key (FIRST_CHUNK_TREE CHUNK_ITEM 1405878272) itemoff 15511
itemsize 80
		length 536870912 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 1405878272
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
	item 8 key (FIRST_CHUNK_TREE CHUNK_ITEM 1942749184) itemoff 15399
itemsize 112
		length 33554432 owner 2 stripe_len 65536 type SYSTEM|DUP
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 1
			stripe 0 devid 1 offset 1942749184
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
			stripe 1 devid 1 offset 1976303616
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
	item 9 key (FIRST_CHUNK_TREE CHUNK_ITEM 1976303616) itemoff 15287
itemsize 112
		length 134217728 owner 2 stripe_len 65536 type METADATA|DUP
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 1
			stripe 0 devid 1 offset 2009858048
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
			stripe 1 devid 1 offset 2144075776
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
	item 10 key (FIRST_CHUNK_TREE CHUNK_ITEM 2110521344) itemoff 15175
itemsize 112
		length 268435456 owner 2 stripe_len 65536 type DATA|DUP
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 1
			stripe 0 devid 1 offset 2278293504
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
			stripe 1 devid 1 offset 2546728960
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6

10:/home/bor #

Running

btrfs balance start -f -ssoft,convert=raid1 -dsoft,convert=raid1 -msoft,convert=raid1 /mnt

fixes it. I am not sure whether btrfs is expected to do it automatically as of now.

#5 lore raw
In reply to Jakob Schöttl

09.02.2019 13:36, Jakob Schöttl пишет:

Quoted 1 line

  btrfs filesystem show   # to get devid of device 2

Funny because for me it just says "Some devices missing" without giving exact device id. May be has been fixed in recent versions but for a long time it really annoyed me.

OK, when I just unplug device it does not say it - but it also does not say *which* device is missing, it happily continues to show old device names at the time of mount, be they actually present or not.

I always wondered how users are supposed to know which device to replace.

#4 lore raw
In reply to Andrei Borzenkov

Am 09.02.19 um 16:32 schrieb Andrei Borzenkov:

Quoted 7 lines

Running

btrfs balance start -f -ssoft,convert=raid1 -dsoft,convert=raid1 -msoft,convert=raid1 /mnt

fixes it. I am not sure whether btrfs is expected to do it automatically as of now.

This fix seems to work for me, too. Output was:

Quoted 1 line

Done, had to relocate 5 out of 8 chunks

Tested without any subvolumes or data.

#6 lore raw
In reply to Jakob Schöttl
Quoted 8 lines · Jakob Schöttl

Replace device 2 with device 3:

mount -o degraded /dev/loop1 /mnt
btrfs filesystem show # to get devid of device 2
btrfs replace start -Br 2 /dev/loop3 /mnt
btrfs replace status /mnt # check success
umount /mnt

The first command causes the creation of metadata and system chunk using single profile, on loop1, and even after a replace, both devices know that there is a single and required chunk on loop1. So subsequent degraded mount fails because the volume was not fully redundant.

The missing step is between 'btrfs replaced start' completing, and umount. There needed to be at least a 'btrfs fi us' to check if there are any single chunks and to convert them with 'btrfs balance start -dconvert=raid1,soft -mconvert=raid1,soft' where soft is only possible if there were no data writes while degraded; if there were degraded writes, a scrub is also necessary to make sure any single mirror raid1 chunk writes to the degraded device are replicated to the replacement.

Yep - kinda ugly. And not intuitive. But that's the present status.

Signature · 1 line
Chris Murphy
#7 lore raw
Quoted 23 lines · Chris Murphy
Quoted 8 lines · Jakob Schöttl

Replace device 2 with device 3:

mount -o degraded /dev/loop1 /mnt
btrfs filesystem show # to get devid of device 2
btrfs replace start -Br 2 /dev/loop3 /mnt
btrfs replace status /mnt # check success
umount /mnt

The first command causes the creation of metadata and system chunk using single profile, on loop1, and even after a replace, both devices know that there is a single and required chunk on loop1. So subsequent degraded mount fails because the volume was not fully redundant.

The missing step is between 'btrfs replaced start' completing, and umount. There needed to be at least a 'btrfs fi us' to check if there are any single chunks and to convert them with 'btrfs balance start -dconvert=raid1,soft -mconvert=raid1,soft' where soft is only possible if there were no data writes while degraded; if there were degraded writes, a scrub is also necessary to make sure any single mirror raid1 chunk writes to the degraded device are replicated to the replacement.

Quoted 1 line

Yep - kinda ugly. And not intuitive. But that's the present status.

Not exactly.

Currently the problem is, there is extra DUP sys/meta/data chunks created. So despite the SINGLE chunks, we still can't mount it degraded.

IIRC there is some patch trying to address the SINGLE problem, but the DUP one seems to be something new.

Thanks,
Qu

Quoted 2 lines