Skip to contentSkip to conversation

Some file lost when send back snapshot.

DISCUSSION MASAKI Haruka 3 participants 8 messages Apr 25, 2019 – Apr 27, 2019 Btrfs file system

lore
Share & tools
lore
#0 lore raw

I send snapshot from backup volume, and send new snaopshot to backup volume, and some file lost in received volume.

In detail:

1. Host A has new Btrfs volume and Host B have two backuped readonly subvolumes (190312-010100, 190315-001000).
2. Host B send 190315-001000 subvolume to Host A.
3. Host A completely received (all files exist.)
4. Host B took new readonly snapshot "back1" on 190315-001000 (all files exist.)
5. Host B send back1 with -p 190315-001000 to Host A (some file lost.)
6. Host B took new read-writable snapshot "world" on back1 (all files exist.)
7. Host B modified world.
8. Host B took new readonly snapshot "back2" on world (all files exist.)
9. Host B send back2 with -p 190315-001000 to Host A (some file lost.)
10. Deleted received snapshot and Host B send back2 with -p back1 to Host A (some file lost.)

Kernel
5.0.5-1-MANJARO

Btrfs progs
btrfs-progs 4.20.2-1

What I should do?

Thanks.

Masaki Haruka

#1 lore raw

25.04.2019 16:34, MASAKI Haruka пишет:

Quoted 5 lines

I send snapshot from backup volume, and send new snaopshot to backup volume, and some file lost in received volume.

In detail:

Quoted 11 lines

1. Host A has new Btrfs volume and Host B have two backuped readonly subvolumes (190312-010100, 190315-001000).
2. Host B send 190315-001000 subvolume to Host A.
3. Host A completely received (all files exist.)
4. Host B took new readonly snapshot "back1" on 190315-001000 (all files exist.)
5. Host B send back1 with -p 190315-001000 to Host A (some file lost.)
6. Host B took new read-writable snapshot "world" on back1 (all files exist.)
7. Host B modified world.
8. Host B took new readonly snapshot "back2" on world (all files exist.)
9. Host B send back2 with -p 190315-001000 to Host A (some file lost.)
10. Deleted received snapshot and Host B send back2 with -p back1 to Host A (some file lost.)

Please show

1. Full protocol of commands you executed (commands and their output), e.g. captured using script.
2. Commands used to determine missing files on both source and receive sides.
3. Listing of subvolumes on both source and receive sides with

btrfs sub li -Rqu /mount/point

Quoted 15 lines

Kernel
5.0.5-1-MANJARO

Btrfs progs
btrfs-progs 4.20.2-1

What I should do?

Thanks.

Masaki Haruka

#2 lore raw

Thank you for your reply.

# What does

@Host A ("lily" master, new volume)
aki@lily /mnt % socat TCP-LISTEN:25500 STDOUT | sudo btrfs receive -vv .

@Host B ("daisy" backup, having volumes)
aki@daisy /HySlave % sudo btrfs send -v 190315-001000 | socat STDIN TCP:lily.local:25500

( ...Transfared 11TB... )

@Host A
aki@lily /mnt % sudo btrfs subvolume snapshot 190315-001000 world Create a snapshot of '190315-001000' in './world' aki@lily /mnt % sudo btrfs subvolume snapshot -r world back1 Create a readonly snapshot of 'world' in './back1'

@Host A
aki@lily /mnt % sudo btrfs send -vv -p 190315-001000 back1 | socat STDIN tcp-connect:192.168.1.17:25555 At subvol back1
BTRFS_IOC_SEND returned 0
joining genl thread

@Host B
root@daisy /HySlave # socat TCP-LISTEN:25555 STDOUT | btrfs receive -vv .
At snapshot back1
receiving snapshot back1 uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, ctransid=2658 parent_uuid=9c10b77f-74fb-f94a-9a24-3c6d30c00ca2, parent_ctransid=2658 BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, stransid=2658

# Host A current

aki@lily /mnt % ls
190315-001000 back1 back2 world
aki@lily /mnt % sudo btrfs sub li -Rqu .
ID 258 gen 2698 top level 5 parent_uuid - received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid bc90d05c-7430-7b42-b683-702d906bb993 path 190315-001000

ID 2139 gen 2696 top level 5 parent_uuid bc90d05c-7430-7b42-b683-702d906bb993 received_uuid -                                    uuid 23c5fede-a338-d14a-afcd-677db2e97637 path world
ID 2140 gen 2660 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid -                                    uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 path back1
ID 2142 gen 2672 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid -                                    uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b path back2

# Host B current

aki@daisy /HySlave % pwd
/HySlave
aki@daisy /HySlave % ls
190312-010100 190315-001000 back1 back2
[aki@daisy HySlave]$ sudo btrfs sub li -Rqu .
ID 256 gen 36238 top level 5 parent_uuid - received_uuid - uuid 4e42555d-a715-0e47-bd0e-85d6eb8911e0 path slave
ID 259 gen 35116 top level 256 parent_uuid - received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 path 190312-010100
ID 3059 gen 7273 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 114cd74f-80d8-af44-a9d0-fd8ce3c991b4 path 190315-001000 ID 3072 gen 35112 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 uuid fe7281d1-8b3e-8f46-a0b4-cdc52a1bb7aa path back1 ID 3073 gen 36236 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b uuid 65206183-f20d-4648-83ab-e0485b16f7af path back2

# Files missing

aki@daisy /HySlave % ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CD
Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

aki@daisy /HySlave % ls /HySlave/back1/local/pub/media/sound/notmusic_in_CD
Little_My_Maid el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

aki@lily /mnt % ls /mnt/190315-001000/local/pub/media/sound/notmusic_in_CD
Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

ki@lily /mnt % ls /mnt/back1/local/pub/media/sound/notmusic_in_CD
Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

aki@lily /mnt % ls /mnt/world/local/pub/media/sound/notmusic_in_CD
Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

SMEE is missing.

Thanks.

Quoted 15 lines · On Thu, 25 Apr 2019 20:55:59 +0300 Andrei Borzenkov <[email protected]> wrote:

25.04.2019 16:34, MASAKI Haruka пишет:
[...]

Please show

1. Full protocol of commands you executed (commands and their output), e.g. captured using script.
2. Commands used to determine missing files on both source and receive sides.
3. Listing of subvolumes on both source and receive sides with

btrfs sub li -Rqu /mount/point

[...]

#3 lore raw

26.04.2019 7:50, MASAKI Haruka пишет:

Quoted 43 lines

Thank you for your reply.

# What does

@Host A ("lily" master, new volume)
aki@lily /mnt % socat TCP-LISTEN:25500 STDOUT | sudo btrfs receive -vv .

@Host B ("daisy" backup, having volumes)
aki@daisy /HySlave % sudo btrfs send -v 190315-001000 | socat STDIN TCP:lily.local:25500

( ...Transfared 11TB... )

@Host A
aki@lily /mnt % sudo btrfs subvolume snapshot 190315-001000 world Create a snapshot of '190315-001000' in './world' aki@lily /mnt % sudo btrfs subvolume snapshot -r world back1 Create a readonly snapshot of 'world' in './back1'

@Host A
aki@lily /mnt % sudo btrfs send -vv -p 190315-001000 back1 | socat STDIN tcp-connect:192.168.1.17:25555 At subvol back1
BTRFS_IOC_SEND returned 0
joining genl thread

@Host B
root@daisy /HySlave # socat TCP-LISTEN:25555 STDOUT | btrfs receive -vv .
At snapshot back1
receiving snapshot back1 uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, ctransid=2658 parent_uuid=9c10b77f-74fb-f94a-9a24-3c6d30c00ca2, parent_ctransid=2658 BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, stransid=2658

# Host A current

aki@lily /mnt % ls
190315-001000 back1 back2 world
aki@lily /mnt % sudo btrfs sub li -Rqu .
ID 258 gen 2698 top level 5 parent_uuid - received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid bc90d05c-7430-7b42-b683-702d906bb993 path 190315-001000

ID 2139 gen 2696 top level 5 parent_uuid bc90d05c-7430-7b42-b683-702d906bb993 received_uuid -                                    uuid 23c5fede-a338-d14a-afcd-677db2e97637 path world
ID 2140 gen 2660 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid -                                    uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 path back1
ID 2142 gen 2672 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid -                                    uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b path back2

# Host B current

Quoted 8 lines

aki@daisy /HySlave % pwd
/HySlave
aki@daisy /HySlave % ls
190312-010100 190315-001000 back1 back2
[aki@daisy HySlave]$ sudo btrfs sub li -Rqu .
ID 256 gen 36238 top level 5 parent_uuid - received_uuid - uuid 4e42555d-a715-0e47-bd0e-85d6eb8911e0 path slave
ID 259 gen 35116 top level 256 parent_uuid - received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 path 190312-010100
ID 3059 gen 7273 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 114cd74f-80d8-af44-a9d0-fd8ce3c991b4 path 190315-001000

You have two subvolumes with the same received_uuid. Is the content of these subvolumes the same? Is "missing" file present in both subvolumes?

Quoted 47 lines

ID 3072 gen 35112 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 uuid fe7281d1-8b3e-8f46-a0b4-cdc52a1bb7aa path back1 ID 3073 gen 36236 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b uuid 65206183-f20d-4648-83ab-e0485b16f7af path back2

# Files missing

aki@daisy /HySlave % ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CD
Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

aki@daisy /HySlave % ls /HySlave/back1/local/pub/media/sound/notmusic_in_CD
Little_My_Maid el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

aki@lily /mnt % ls /mnt/190315-001000/local/pub/media/sound/notmusic_in_CD
Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

ki@lily /mnt % ls /mnt/back1/local/pub/media/sound/notmusic_in_CD
Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

aki@lily /mnt % ls /mnt/world/local/pub/media/sound/notmusic_in_CD
Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

SMEE is missing.

Thanks.

Quoted 15 lines · On Thu, 25 Apr 2019 20:55:59 +0300 Andrei Borzenkov <[email protected]> wrote:

25.04.2019 16:34, MASAKI Haruka пишет:
[...]

Please show

1. Full protocol of commands you executed (commands and their output), e.g. captured using script.
2. Commands used to determine missing files on both source and receive sides.
3. Listing of subvolumes on both source and receive sides with

btrfs sub li -Rqu /mount/point

[...]

#4 lore raw
Quoted 2 lines

You have two subvolumes with the same received_uuid. Is the content of these subvolumes the same? Is "missing" file present in both subvolumes?

No, each subvolume name (190312-010100, 190315-001000) is timestamp.
These snapshot volumes were created by job scheduler on original (already lost) volume, so 190315-001000 is modified from 190312-010100.

190315-001000 has lost file,
% ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CD/
Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

but 190312-010100 doesn't.
% ls /HySlave/190312-010100/local/pub/media/sound/notmusic_in_CD/
Little_My_Maid el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

Quoted 104 lines · On Fri, 26 Apr 2019 20:30:07 +0300 Andrei Borzenkov <[email protected]> wrote:

26.04.2019 7:50, MASAKI Haruka пишет:

Quoted 43 lines

Thank you for your reply.

# What does

@Host A ("lily" master, new volume)
aki@lily /mnt % socat TCP-LISTEN:25500 STDOUT | sudo btrfs receive -vv .

@Host B ("daisy" backup, having volumes)
aki@daisy /HySlave % sudo btrfs send -v 190315-001000 | socat STDIN TCP:lily.local:25500

( ...Transfared 11TB... )

@Host A
aki@lily /mnt % sudo btrfs subvolume snapshot 190315-001000 world Create a snapshot of '190315-001000' in './world' aki@lily /mnt % sudo btrfs subvolume snapshot -r world back1 Create a readonly snapshot of 'world' in './back1'

@Host A
aki@lily /mnt % sudo btrfs send -vv -p 190315-001000 back1 | socat STDIN tcp-connect:192.168.1.17:25555 At subvol back1
BTRFS_IOC_SEND returned 0
joining genl thread

@Host B
root@daisy /HySlave # socat TCP-LISTEN:25555 STDOUT | btrfs receive -vv .
At snapshot back1
receiving snapshot back1 uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, ctransid=2658 parent_uuid=9c10b77f-74fb-f94a-9a24-3c6d30c00ca2, parent_ctransid=2658 BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, stransid=2658

# Host A current

aki@lily /mnt % ls
190315-001000 back1 back2 world
aki@lily /mnt % sudo btrfs sub li -Rqu .
ID 258 gen 2698 top level 5 parent_uuid - received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid bc90d05c-7430-7b42-b683-702d906bb993 path 190315-001000

ID 2139 gen 2696 top level 5 parent_uuid bc90d05c-7430-7b42-b683-702d906bb993 received_uuid -                                    uuid 23c5fede-a338-d14a-afcd-677db2e97637 path world
ID 2140 gen 2660 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid -                                    uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 path back1
ID 2142 gen 2672 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid -                                    uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b path back2

# Host B current

Quoted 8 lines

aki@daisy /HySlave % pwd
/HySlave
aki@daisy /HySlave % ls
190312-010100 190315-001000 back1 back2
[aki@daisy HySlave]$ sudo btrfs sub li -Rqu .
ID 256 gen 36238 top level 5 parent_uuid - received_uuid - uuid 4e42555d-a715-0e47-bd0e-85d6eb8911e0 path slave
ID 259 gen 35116 top level 256 parent_uuid - received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 path 190312-010100
ID 3059 gen 7273 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 114cd74f-80d8-af44-a9d0-fd8ce3c991b4 path 190315-001000

You have two subvolumes with the same received_uuid. Is the content of these subvolumes the same? Is "missing" file present in both subvolumes?

Quoted 47 lines

ID 3072 gen 35112 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 uuid fe7281d1-8b3e-8f46-a0b4-cdc52a1bb7aa path back1 ID 3073 gen 36236 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b uuid 65206183-f20d-4648-83ab-e0485b16f7af path back2

# Files missing

aki@daisy /HySlave % ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CD
Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

aki@daisy /HySlave % ls /HySlave/back1/local/pub/media/sound/notmusic_in_CD
Little_My_Maid el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

aki@lily /mnt % ls /mnt/190315-001000/local/pub/media/sound/notmusic_in_CD
Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

ki@lily /mnt % ls /mnt/back1/local/pub/media/sound/notmusic_in_CD
Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

aki@lily /mnt % ls /mnt/world/local/pub/media/sound/notmusic_in_CD
Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

SMEE is missing.

Thanks.

Quoted 15 lines · On Thu, 25 Apr 2019 20:55:59 +0300 Andrei Borzenkov <[email protected]> wrote:

25.04.2019 16:34, MASAKI Haruka пишет:
[...]

Please show

1. Full protocol of commands you executed (commands and their output), e.g. captured using script.
2. Commands used to determine missing files on both source and receive sides.
3. Listing of subvolumes on both source and receive sides with

btrfs sub li -Rqu /mount/point

[...]

#5 lore raw

26.04.2019 21:12, MASAKI Haruka пишет:

Quoted 6 lines
Quoted 2 lines

You have two subvolumes with the same received_uuid. Is the content of these subvolumes the same? Is "missing" file present in both subvolumes?

No, each subvolume name (190312-010100, 190315-001000) is timestamp.
These snapshot volumes were created by job scheduler on original (already lost) volume, so 190315-001000 is modified from 190312-010100.

Which is your problem. Two subvolumes with the same received_uuid are expected to have exactly the same content (received_uuid can be viewed as dataset UUID). You send incremental stream against base uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2; on receiving side btrfs receive looks for matching base subvolume and happens to pick 190312-010100.

There were patches to automatically remove received_uuid when subvolume is changed to read-write. I do not know whether they were applied.

btrfs send/receive expects subvolumes to remain read-only. As soon as you start fiddling with read-only status you already lost. btrfs should have never allowed to change read-only into read-write in the first place.

Quoted 118 lines

190315-001000 has lost file,
% ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CD/
Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

but 190312-010100 doesn't.
% ls /HySlave/190312-010100/local/pub/media/sound/notmusic_in_CD/
Little_My_Maid el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

Quoted 104 lines · On Fri, 26 Apr 2019 20:30:07 +0300 Andrei Borzenkov <[email protected]> wrote:

26.04.2019 7:50, MASAKI Haruka пишет:

Quoted 43 lines

Thank you for your reply.

# What does

@Host A ("lily" master, new volume)
aki@lily /mnt % socat TCP-LISTEN:25500 STDOUT | sudo btrfs receive -vv .

@Host B ("daisy" backup, having volumes)
aki@daisy /HySlave % sudo btrfs send -v 190315-001000 | socat STDIN TCP:lily.local:25500

( ...Transfared 11TB... )

@Host A
aki@lily /mnt % sudo btrfs subvolume snapshot 190315-001000 world Create a snapshot of '190315-001000' in './world' aki@lily /mnt % sudo btrfs subvolume snapshot -r world back1 Create a readonly snapshot of 'world' in './back1'

@Host A
aki@lily /mnt % sudo btrfs send -vv -p 190315-001000 back1 | socat STDIN tcp-connect:192.168.1.17:25555 At subvol back1
BTRFS_IOC_SEND returned 0
joining genl thread

@Host B
root@daisy /HySlave # socat TCP-LISTEN:25555 STDOUT | btrfs receive -vv .
At snapshot back1
receiving snapshot back1 uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, ctransid=2658 parent_uuid=9c10b77f-74fb-f94a-9a24-3c6d30c00ca2, parent_ctransid=2658 BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, stransid=2658

# Host A current

aki@lily /mnt % ls
190315-001000 back1 back2 world
aki@lily /mnt % sudo btrfs sub li -Rqu .
ID 258 gen 2698 top level 5 parent_uuid - received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid bc90d05c-7430-7b42-b683-702d906bb993 path 190315-001000

ID 2139 gen 2696 top level 5 parent_uuid bc90d05c-7430-7b42-b683-702d906bb993 received_uuid -                                    uuid 23c5fede-a338-d14a-afcd-677db2e97637 path world
ID 2140 gen 2660 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid -                                    uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 path back1
ID 2142 gen 2672 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid -                                    uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b path back2

# Host B current

Quoted 8 lines

aki@daisy /HySlave % pwd
/HySlave
aki@daisy /HySlave % ls
190312-010100 190315-001000 back1 back2
[aki@daisy HySlave]$ sudo btrfs sub li -Rqu .
ID 256 gen 36238 top level 5 parent_uuid - received_uuid - uuid 4e42555d-a715-0e47-bd0e-85d6eb8911e0 path slave
ID 259 gen 35116 top level 256 parent_uuid - received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 path 190312-010100
ID 3059 gen 7273 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 114cd74f-80d8-af44-a9d0-fd8ce3c991b4 path 190315-001000

You have two subvolumes with the same received_uuid. Is the content of these subvolumes the same? Is "missing" file present in both subvolumes?

Quoted 47 lines

ID 3072 gen 35112 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 uuid fe7281d1-8b3e-8f46-a0b4-cdc52a1bb7aa path back1 ID 3073 gen 36236 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b uuid 65206183-f20d-4648-83ab-e0485b16f7af path back2

# Files missing

aki@daisy /HySlave % ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CD
Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

aki@daisy /HySlave % ls /HySlave/back1/local/pub/media/sound/notmusic_in_CD
Little_My_Maid el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

aki@lily /mnt % ls /mnt/190315-001000/local/pub/media/sound/notmusic_in_CD
Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

ki@lily /mnt % ls /mnt/back1/local/pub/media/sound/notmusic_in_CD
Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

aki@lily /mnt % ls /mnt/world/local/pub/media/sound/notmusic_in_CD
Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

SMEE is missing.

Thanks.

Quoted 15 lines · On Thu, 25 Apr 2019 20:55:59 +0300 Andrei Borzenkov <[email protected]> wrote:

25.04.2019 16:34, MASAKI Haruka пишет:
[...]

Please show

1. Full protocol of commands you executed (commands and their output), e.g. captured using script.
2. Commands used to determine missing files on both source and receive sides.
3. Listing of subvolumes on both source and receive sides with

btrfs sub li -Rqu /mount/point

[...]

#6 lore raw
Quoted 14 lines · On Fri, 26 Apr 2019 21:27:52 +0300 Andrei Borzenkov <[email protected]> wrote:

26.04.2019 21:12, MASAKI Haruka пишет:

Quoted 6 lines
Quoted 2 lines

You have two subvolumes with the same received_uuid. Is the content of these subvolumes the same? Is "missing" file present in both subvolumes?

No, each subvolume name (190312-010100, 190315-001000) is timestamp.
These snapshot volumes were created by job scheduler on original (already lost) volume, so 190315-001000 is modified from 190312-010100.

Which is your problem. Two subvolumes with the same received_uuid are expected to have exactly the same content (received_uuid can be viewed as dataset UUID). You send incremental stream against base uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2; on receiving side btrfs receive looks for matching base subvolume and happens to pick 190312-010100.

Quoted 7 lines

There were patches to automatically remove received_uuid when subvolume is changed to read-write. I do not know whether they were applied.

btrfs send/receive expects subvolumes to remain read-only. As soon as you start fiddling with read-only status you already lost. btrfs should have never allowed to change read-only into read-write in the first place.

Thanks it was solved!

I remember change attributes on original volume.

I deleted 190312-010100 and send back1, it's OK.
I learned that shouldn't change subvolume read-only to read-write.

Thank you very much!!!

Quoted 119 lines
Quoted 118 lines

190315-001000 has lost file,
% ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CD/
Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

but 190312-010100 doesn't.
% ls /HySlave/190312-010100/local/pub/media/sound/notmusic_in_CD/
Little_My_Maid el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

Quoted 104 lines · On Fri, 26 Apr 2019 20:30:07 +0300 Andrei Borzenkov <[email protected]> wrote:

26.04.2019 7:50, MASAKI Haruka пишет:

Quoted 43 lines

Thank you for your reply.

# What does

@Host A ("lily" master, new volume)
aki@lily /mnt % socat TCP-LISTEN:25500 STDOUT | sudo btrfs receive -vv .

@Host B ("daisy" backup, having volumes)
aki@daisy /HySlave % sudo btrfs send -v 190315-001000 | socat STDIN TCP:lily.local:25500

( ...Transfared 11TB... )

@Host A
aki@lily /mnt % sudo btrfs subvolume snapshot 190315-001000 world Create a snapshot of '190315-001000' in './world' aki@lily /mnt % sudo btrfs subvolume snapshot -r world back1 Create a readonly snapshot of 'world' in './back1'

@Host A
aki@lily /mnt % sudo btrfs send -vv -p 190315-001000 back1 | socat STDIN tcp-connect:192.168.1.17:25555 At subvol back1
BTRFS_IOC_SEND returned 0
joining genl thread

@Host B
root@daisy /HySlave # socat TCP-LISTEN:25555 STDOUT | btrfs receive -vv .
At snapshot back1
receiving snapshot back1 uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, ctransid=2658 parent_uuid=9c10b77f-74fb-f94a-9a24-3c6d30c00ca2, parent_ctransid=2658 BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, stransid=2658

# Host A current

aki@lily /mnt % ls
190315-001000 back1 back2 world
aki@lily /mnt % sudo btrfs sub li -Rqu .
ID 258 gen 2698 top level 5 parent_uuid - received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid bc90d05c-7430-7b42-b683-702d906bb993 path 190315-001000

ID 2139 gen 2696 top level 5 parent_uuid bc90d05c-7430-7b42-b683-702d906bb993 received_uuid -                                    uuid 23c5fede-a338-d14a-afcd-677db2e97637 path world
ID 2140 gen 2660 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid -                                    uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 path back1
ID 2142 gen 2672 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid -                                    uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b path back2

# Host B current

Quoted 8 lines

aki@daisy /HySlave % pwd
/HySlave
aki@daisy /HySlave % ls
190312-010100 190315-001000 back1 back2
[aki@daisy HySlave]$ sudo btrfs sub li -Rqu .
ID 256 gen 36238 top level 5 parent_uuid - received_uuid - uuid 4e42555d-a715-0e47-bd0e-85d6eb8911e0 path slave
ID 259 gen 35116 top level 256 parent_uuid - received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 path 190312-010100
ID 3059 gen 7273 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 114cd74f-80d8-af44-a9d0-fd8ce3c991b4 path 190315-001000

You have two subvolumes with the same received_uuid. Is the content of these subvolumes the same? Is "missing" file present in both subvolumes?

Quoted 47 lines

ID 3072 gen 35112 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 uuid fe7281d1-8b3e-8f46-a0b4-cdc52a1bb7aa path back1 ID 3073 gen 36236 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b uuid 65206183-f20d-4648-83ab-e0485b16f7af path back2

# Files missing

aki@daisy /HySlave % ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CD
Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

aki@daisy /HySlave % ls /HySlave/back1/local/pub/media/sound/notmusic_in_CD
Little_My_Maid el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

aki@lily /mnt % ls /mnt/190315-001000/local/pub/media/sound/notmusic_in_CD
Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

ki@lily /mnt % ls /mnt/back1/local/pub/media/sound/notmusic_in_CD
Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

aki@lily /mnt % ls /mnt/world/local/pub/media/sound/notmusic_in_CD
Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala

SMEE is missing.

Thanks.

Quoted 15 lines · On Thu, 25 Apr 2019 20:55:59 +0300 Andrei Borzenkov <[email protected]> wrote:

25.04.2019 16:34, MASAKI Haruka пишет:
[...]

Please show

1. Full protocol of commands you executed (commands and their output), e.g. captured using script.
2. Commands used to determine missing files on both source and receive sides.
3. Listing of subvolumes on both source and receive sides with

btrfs sub li -Rqu /mount/point

[...]

#7 lore raw
In reply to Andrei Borzenkov
Quoted 4 lines · Andrei Borzenkov

btrfs send/receive expects subvolumes to remain read-only. As soon as you start fiddling with read-only status you already lost. btrfs should have never allowed to change read-only into read-write in the first place.

I'm in agreement, and would like to see 'btrfs property set <> ro false> removed. It's confusing. Once a subvolume is read-only, it should remain read-only, and if the use case requires rw, then just snapshot it (default rw).

Signature · 1 line
Chris Murphy