Some file lost when send back snapshot.
I send snapshot from backup volume, and send new snaopshot to backup volume, and some file lost in received volume.
In detail:
1. Host A has new Btrfs volume and Host B have two backuped readonly subvolumes (190312-010100, 190315-001000).
2. Host B send 190315-001000 subvolume to Host A.
3. Host A completely received (all files exist.)
4. Host B took new readonly snapshot "back1" on 190315-001000 (all files exist.)
5. Host B send back1 with -p 190315-001000 to Host A (some file lost.)
6. Host B took new read-writable snapshot "world" on back1 (all files exist.)
7. Host B modified world.
8. Host B took new readonly snapshot "back2" on world (all files exist.)
9. Host B send back2 with -p 190315-001000 to Host A (some file lost.)
10. Deleted received snapshot and Host B send back2 with -p back1 to Host A (some file lost.)
Kernel
5.0.5-1-MANJARO
Btrfs progs
btrfs-progs 4.20.2-1
What I should do?
Thanks.
Masaki Haruka
25.04.2019 16:34, MASAKI Haruka пишет:
Quoted 5 lines
I send snapshot from backup volume, and send new snaopshot to backup volume, and some file lost in received volume.
In detail:
Quoted 11 lines
1. Host A has new Btrfs volume and Host B have two backuped readonly subvolumes (190312-010100, 190315-001000).
2. Host B send 190315-001000 subvolume to Host A.
3. Host A completely received (all files exist.)
4. Host B took new readonly snapshot "back1" on 190315-001000 (all files exist.)
5. Host B send back1 with-p 190315-001000to Host A (some file lost.)
6. Host B took new read-writable snapshot "world" on back1 (all files exist.)
7. Host B modified world.
8. Host B took new readonly snapshot "back2" on world (all files exist.)
9. Host B send back2 with-p 190315-001000to Host A (some file lost.)
10. Deleted received snapshot and Host B send back2 with-p back1to Host A (some file lost.)
Please show
1. Full protocol of commands you executed (commands and their output), e.g. captured using script.
2. Commands used to determine missing files on both source and receive sides.
3. Listing of subvolumes on both source and receive sides with
btrfs sub li -Rqu /mount/point
Quoted 15 lines
Kernel
5.0.5-1-MANJAROBtrfs progs
btrfs-progs 4.20.2-1What I should do?
Thanks.
Masaki Haruka
Thank you for your reply.
# What does
@Host A ("lily" master, new volume)
aki@lily /mnt % socat TCP-LISTEN:25500 STDOUT | sudo btrfs receive -vv .
@Host B ("daisy" backup, having volumes)
aki@daisy /HySlave % sudo btrfs send -v 190315-001000 | socat STDIN TCP:lily.local:25500
( ...Transfared 11TB... )
@Host A
aki@lily /mnt % sudo btrfs subvolume snapshot 190315-001000 world Create a snapshot of '190315-001000' in './world' aki@lily /mnt % sudo btrfs subvolume snapshot -r world back1 Create a readonly snapshot of 'world' in './back1'
@Host A
aki@lily /mnt % sudo btrfs send -vv -p 190315-001000 back1 | socat STDIN tcp-connect:192.168.1.17:25555 At subvol back1BTRFS_IOC_SEND returned 0
joining genl thread
@Host B
root@daisy /HySlave # socat TCP-LISTEN:25555 STDOUT | btrfs receive -vv .
At snapshot back1
receiving snapshot back1 uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, ctransid=2658 parent_uuid=9c10b77f-74fb-f94a-9a24-3c6d30c00ca2, parent_ctransid=2658 BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, stransid=2658
# Host A current
aki@lily /mnt % ls
190315-001000 back1 back2 world
aki@lily /mnt % sudo btrfs sub li -Rqu .
ID 258 gen 2698 top level 5 parent_uuid - received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid bc90d05c-7430-7b42-b683-702d906bb993 path 190315-001000
ID 2139 gen 2696 top level 5 parent_uuid bc90d05c-7430-7b42-b683-702d906bb993 received_uuid - uuid 23c5fede-a338-d14a-afcd-677db2e97637 path world ID 2140 gen 2660 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid - uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 path back1 ID 2142 gen 2672 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid - uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b path back2
# Host B current
aki@daisy /HySlave % pwd
/HySlave
aki@daisy /HySlave % ls
190312-010100 190315-001000 back1 back2
[aki@daisy HySlave]$ sudo btrfs sub li -Rqu .
ID 256 gen 36238 top level 5 parent_uuid - received_uuid - uuid 4e42555d-a715-0e47-bd0e-85d6eb8911e0 path slave
ID 259 gen 35116 top level 256 parent_uuid - received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 path 190312-010100
ID 3059 gen 7273 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 114cd74f-80d8-af44-a9d0-fd8ce3c991b4 path 190315-001000 ID 3072 gen 35112 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 uuid fe7281d1-8b3e-8f46-a0b4-cdc52a1bb7aa path back1 ID 3073 gen 36236 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b uuid 65206183-f20d-4648-83ab-e0485b16f7af path back2
# Files missing
aki@daisy /HySlave % ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CDLittle_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala
aki@daisy /HySlave % ls /HySlave/back1/local/pub/media/sound/notmusic_in_CDLittle_My_Maid el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala
aki@lily /mnt % ls /mnt/190315-001000/local/pub/media/sound/notmusic_in_CDLittle_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala
ki@lily /mnt % ls /mnt/back1/local/pub/media/sound/notmusic_in_CDLittle_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala
aki@lily /mnt % ls /mnt/world/local/pub/media/sound/notmusic_in_CDLittle_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala
SMEE is missing.
Thanks.
Quoted 15 lines · On Thu, 25 Apr 2019 20:55:59 +0300 Andrei Borzenkov <[email protected]> wrote:
25.04.2019 16:34, MASAKI Haruka пишет:
[...]Please show
1. Full protocol of commands you executed (commands and their output), e.g. captured using script.
2. Commands used to determine missing files on both source and receive sides.
3. Listing of subvolumes on both source and receive sides withbtrfs sub li -Rqu /mount/point
[...]
26.04.2019 7:50, MASAKI Haruka пишет:
Quoted 43 lines
Thank you for your reply.
# What does
@Host A ("lily" master, new volume)
aki@lily /mnt % socat TCP-LISTEN:25500 STDOUT | sudo btrfs receive -vv .@Host B ("daisy" backup, having volumes)
aki@daisy /HySlave % sudo btrfs send -v 190315-001000 | socat STDIN TCP:lily.local:25500( ...Transfared 11TB... )
@Host A
aki@lily /mnt % sudo btrfs subvolume snapshot 190315-001000 world Create a snapshot of '190315-001000' in './world' aki@lily /mnt % sudo btrfs subvolume snapshot -r world back1 Create a readonly snapshot of 'world' in './back1'@Host A
aki@lily /mnt % sudo btrfs send -vv -p 190315-001000 back1 | socat STDIN tcp-connect:192.168.1.17:25555 At subvol back1BTRFS_IOC_SENDreturned 0
joining genl thread@Host B
root@daisy /HySlave # socat TCP-LISTEN:25555 STDOUT | btrfs receive -vv .
At snapshot back1
receiving snapshot back1 uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, ctransid=2658 parent_uuid=9c10b77f-74fb-f94a-9a24-3c6d30c00ca2, parent_ctransid=2658BTRFS_IOC_SET_RECEIVED_SUBVOLuuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, stransid=2658# Host A current
aki@lily /mnt % ls
190315-001000 back1 back2 world
aki@lily /mnt % sudo btrfs sub li -Rqu .
ID 258 gen 2698 top level 5parent_uuid-received_uuid9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid bc90d05c-7430-7b42-b683-702d906bb993 path 190315-001000ID 2139 gen 2696 top level 5 parent_uuid bc90d05c-7430-7b42-b683-702d906bb993 received_uuid - uuid 23c5fede-a338-d14a-afcd-677db2e97637 path world ID 2140 gen 2660 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid - uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 path back1 ID 2142 gen 2672 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid - uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b path back2# Host B current
Quoted 8 lines
aki@daisy /HySlave % pwd
/HySlave
aki@daisy /HySlave % ls
190312-010100 190315-001000 back1 back2
[aki@daisy HySlave]$ sudo btrfs sub li -Rqu .
ID 256 gen 36238 top level 5parent_uuid-received_uuid- uuid 4e42555d-a715-0e47-bd0e-85d6eb8911e0 path slave
ID 259 gen 35116 top level 256parent_uuid-received_uuid9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 path 190312-010100
ID 3059 gen 7273 top level 256parent_uuid4a91d585-ab25-0c49-b183-ae951b5377e1received_uuid9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 114cd74f-80d8-af44-a9d0-fd8ce3c991b4 path 190315-001000
You have two subvolumes with the same received_uuid. Is the content of these subvolumes the same? Is "missing" file present in both subvolumes?
Quoted 47 lines
ID 3072 gen 35112 top level 256
parent_uuid4a91d585-ab25-0c49-b183-ae951b5377e1received_uuid32dda1a2-07bd-684f-b773-c5b4cd5f29c1 uuid fe7281d1-8b3e-8f46-a0b4-cdc52a1bb7aa path back1 ID 3073 gen 36236 top level 256parent_uuid4a91d585-ab25-0c49-b183-ae951b5377e1received_uuida8e3e041-c1fc-e443-9baf-ab481ae32b8b uuid 65206183-f20d-4648-83ab-e0485b16f7af path back2# Files missing
aki@daisy /HySlave % ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CD
Little_My_MaidSMEEel_louisfirstkiss gakuensai infinity-kid kid-memories-offojyo_tokkyusamusupiwizards_harmonyyuukyuu-chara-sealaaki@daisy /HySlave % ls /HySlave/back1/local/pub/media/sound/notmusic_in_CD
Little_My_Maidel_louisfirstkiss gakuensai infinity-kid kid-memories-offojyo_tokkyusamusupiwizards_harmonyyuukyuu-chara-sealaaki@lily /mnt % ls
/mnt/190315-001000/local/pub/media/sound/notmusic_in_CDLittle_My_MaidSMEEel_louisfirstkiss gakuensai infinity-kid kid-memories-offojyo_tokkyusamusupiwizards_harmonyyuukyuu-chara-sealaki@lily /mnt % ls
/mnt/back1/local/pub/media/sound/notmusic_in_CDLittle_My_MaidSMEEel_louisfirstkiss gakuensai infinity-kid kid-memories-offojyo_tokkyusamusupiwizards_harmonyyuukyuu-chara-sealaaki@lily /mnt % ls
/mnt/world/local/pub/media/sound/notmusic_in_CDLittle_My_MaidSMEEel_louisfirstkiss gakuensai infinity-kid kid-memories-offojyo_tokkyusamusupiwizards_harmonyyuukyuu-chara-sealaSMEE is missing.
Thanks.
Quoted 15 lines · On Thu, 25 Apr 2019 20:55:59 +0300 Andrei Borzenkov <[email protected]> wrote:
25.04.2019 16:34, MASAKI Haruka пишет:
[...]Please show
1. Full protocol of commands you executed (commands and their output), e.g. captured using script.
2. Commands used to determine missing files on both source and receive sides.
3. Listing of subvolumes on both source and receive sides withbtrfs sub li -Rqu /mount/point
[...]
Quoted 2 lines
You have two subvolumes with the same
received_uuid. Is the content of these subvolumes the same? Is "missing" file present in both subvolumes?
No, each subvolume name (190312-010100, 190315-001000) is timestamp.
These snapshot volumes were created by job scheduler on original (already lost) volume, so 190315-001000 is modified from 190312-010100.
190315-001000 has lost file,
% ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CD/Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala
but 190312-010100 doesn't.
% ls /HySlave/190312-010100/local/pub/media/sound/notmusic_in_CD/Little_My_Maid el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala
Quoted 104 lines · On Fri, 26 Apr 2019 20:30:07 +0300 Andrei Borzenkov <[email protected]> wrote:
26.04.2019 7:50, MASAKI Haruka пишет:
Quoted 43 lines
Thank you for your reply.
# What does
@Host A ("lily" master, new volume)
aki@lily /mnt % socat TCP-LISTEN:25500 STDOUT | sudo btrfs receive -vv .@Host B ("daisy" backup, having volumes)
aki@daisy /HySlave % sudo btrfs send -v 190315-001000 | socat STDIN TCP:lily.local:25500( ...Transfared 11TB... )
@Host A
aki@lily /mnt % sudo btrfs subvolume snapshot 190315-001000 world Create a snapshot of '190315-001000' in './world' aki@lily /mnt % sudo btrfs subvolume snapshot -r world back1 Create a readonly snapshot of 'world' in './back1'@Host A
aki@lily /mnt % sudo btrfs send -vv -p 190315-001000 back1 | socat STDIN tcp-connect:192.168.1.17:25555 At subvol back1BTRFS_IOC_SENDreturned 0
joining genl thread@Host B
root@daisy /HySlave # socat TCP-LISTEN:25555 STDOUT | btrfs receive -vv .
At snapshot back1
receiving snapshot back1 uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, ctransid=2658 parent_uuid=9c10b77f-74fb-f94a-9a24-3c6d30c00ca2, parent_ctransid=2658BTRFS_IOC_SET_RECEIVED_SUBVOLuuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, stransid=2658# Host A current
aki@lily /mnt % ls
190315-001000 back1 back2 world
aki@lily /mnt % sudo btrfs sub li -Rqu .
ID 258 gen 2698 top level 5parent_uuid-received_uuid9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid bc90d05c-7430-7b42-b683-702d906bb993 path 190315-001000ID 2139 gen 2696 top level 5 parent_uuid bc90d05c-7430-7b42-b683-702d906bb993 received_uuid - uuid 23c5fede-a338-d14a-afcd-677db2e97637 path world ID 2140 gen 2660 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid - uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 path back1 ID 2142 gen 2672 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid - uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b path back2# Host B current
Quoted 8 lines
aki@daisy /HySlave % pwd
/HySlave
aki@daisy /HySlave % ls
190312-010100 190315-001000 back1 back2
[aki@daisy HySlave]$ sudo btrfs sub li -Rqu .
ID 256 gen 36238 top level 5parent_uuid-received_uuid- uuid 4e42555d-a715-0e47-bd0e-85d6eb8911e0 path slave
ID 259 gen 35116 top level 256parent_uuid-received_uuid9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 path 190312-010100
ID 3059 gen 7273 top level 256parent_uuid4a91d585-ab25-0c49-b183-ae951b5377e1received_uuid9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 114cd74f-80d8-af44-a9d0-fd8ce3c991b4 path 190315-001000You have two subvolumes with the same
received_uuid. Is the content of these subvolumes the same? Is "missing" file present in both subvolumes?Quoted 47 lines
ID 3072 gen 35112 top level 256
parent_uuid4a91d585-ab25-0c49-b183-ae951b5377e1received_uuid32dda1a2-07bd-684f-b773-c5b4cd5f29c1 uuid fe7281d1-8b3e-8f46-a0b4-cdc52a1bb7aa path back1 ID 3073 gen 36236 top level 256parent_uuid4a91d585-ab25-0c49-b183-ae951b5377e1received_uuida8e3e041-c1fc-e443-9baf-ab481ae32b8b uuid 65206183-f20d-4648-83ab-e0485b16f7af path back2# Files missing
aki@daisy /HySlave % ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CD
Little_My_MaidSMEEel_louisfirstkiss gakuensai infinity-kid kid-memories-offojyo_tokkyusamusupiwizards_harmonyyuukyuu-chara-sealaaki@daisy /HySlave % ls /HySlave/back1/local/pub/media/sound/notmusic_in_CD
Little_My_Maidel_louisfirstkiss gakuensai infinity-kid kid-memories-offojyo_tokkyusamusupiwizards_harmonyyuukyuu-chara-sealaaki@lily /mnt % ls
/mnt/190315-001000/local/pub/media/sound/notmusic_in_CDLittle_My_MaidSMEEel_louisfirstkiss gakuensai infinity-kid kid-memories-offojyo_tokkyusamusupiwizards_harmonyyuukyuu-chara-sealaki@lily /mnt % ls
/mnt/back1/local/pub/media/sound/notmusic_in_CDLittle_My_MaidSMEEel_louisfirstkiss gakuensai infinity-kid kid-memories-offojyo_tokkyusamusupiwizards_harmonyyuukyuu-chara-sealaaki@lily /mnt % ls
/mnt/world/local/pub/media/sound/notmusic_in_CDLittle_My_MaidSMEEel_louisfirstkiss gakuensai infinity-kid kid-memories-offojyo_tokkyusamusupiwizards_harmonyyuukyuu-chara-sealaSMEE is missing.
Thanks.
Quoted 15 lines · On Thu, 25 Apr 2019 20:55:59 +0300 Andrei Borzenkov <[email protected]> wrote:
25.04.2019 16:34, MASAKI Haruka пишет:
[...]Please show
1. Full protocol of commands you executed (commands and their output), e.g. captured using script.
2. Commands used to determine missing files on both source and receive sides.
3. Listing of subvolumes on both source and receive sides withbtrfs sub li -Rqu /mount/point
[...]
26.04.2019 21:12, MASAKI Haruka пишет:
Quoted 6 lines
Quoted 2 lines
You have two subvolumes with the same
received_uuid. Is the content of these subvolumes the same? Is "missing" file present in both subvolumes?No, each subvolume name (190312-010100, 190315-001000) is timestamp.
These snapshot volumes were created by job scheduler on original (already lost) volume, so 190315-001000 is modified from 190312-010100.
Which is your problem. Two subvolumes with the same received_uuid are expected to have exactly the same content (received_uuid can be viewed as dataset UUID). You send incremental stream against base uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2; on receiving side btrfs receive looks for matching base subvolume and happens to pick 190312-010100.
There were patches to automatically remove received_uuid when subvolume is changed to read-write. I do not know whether they were applied.
btrfs send/receive expects subvolumes to remain read-only. As soon as you start fiddling with read-only status you already lost. btrfs should have never allowed to change read-only into read-write in the first place.
Quoted 118 lines
190315-001000 has lost file,
% ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CD/Little_My_MaidSMEEel_louisfirstkiss gakuensai infinity-kid kid-memories-offojyo_tokkyusamusupiwizards_harmonyyuukyuu-chara-sealabut 190312-010100 doesn't.
% ls /HySlave/190312-010100/local/pub/media/sound/notmusic_in_CD/Little_My_Maidel_louisfirstkiss gakuensai infinity-kid kid-memories-offojyo_tokkyusamusupiwizards_harmonyyuukyuu-chara-sealaQuoted 104 lines · On Fri, 26 Apr 2019 20:30:07 +0300 Andrei Borzenkov <[email protected]> wrote:
26.04.2019 7:50, MASAKI Haruka пишет:
Quoted 43 lines
Thank you for your reply.
# What does
@Host A ("lily" master, new volume)
aki@lily /mnt % socat TCP-LISTEN:25500 STDOUT | sudo btrfs receive -vv .@Host B ("daisy" backup, having volumes)
aki@daisy /HySlave % sudo btrfs send -v 190315-001000 | socat STDIN TCP:lily.local:25500( ...Transfared 11TB... )
@Host A
aki@lily /mnt % sudo btrfs subvolume snapshot 190315-001000 world Create a snapshot of '190315-001000' in './world' aki@lily /mnt % sudo btrfs subvolume snapshot -r world back1 Create a readonly snapshot of 'world' in './back1'@Host A
aki@lily /mnt % sudo btrfs send -vv -p 190315-001000 back1 | socat STDIN tcp-connect:192.168.1.17:25555 At subvol back1BTRFS_IOC_SENDreturned 0
joining genl thread@Host B
root@daisy /HySlave # socat TCP-LISTEN:25555 STDOUT | btrfs receive -vv .
At snapshot back1
receiving snapshot back1 uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, ctransid=2658 parent_uuid=9c10b77f-74fb-f94a-9a24-3c6d30c00ca2, parent_ctransid=2658BTRFS_IOC_SET_RECEIVED_SUBVOLuuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, stransid=2658# Host A current
aki@lily /mnt % ls
190315-001000 back1 back2 world
aki@lily /mnt % sudo btrfs sub li -Rqu .
ID 258 gen 2698 top level 5parent_uuid-received_uuid9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid bc90d05c-7430-7b42-b683-702d906bb993 path 190315-001000ID 2139 gen 2696 top level 5 parent_uuid bc90d05c-7430-7b42-b683-702d906bb993 received_uuid - uuid 23c5fede-a338-d14a-afcd-677db2e97637 path world ID 2140 gen 2660 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid - uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 path back1 ID 2142 gen 2672 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid - uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b path back2# Host B current
Quoted 8 lines
aki@daisy /HySlave % pwd
/HySlave
aki@daisy /HySlave % ls
190312-010100 190315-001000 back1 back2
[aki@daisy HySlave]$ sudo btrfs sub li -Rqu .
ID 256 gen 36238 top level 5parent_uuid-received_uuid- uuid 4e42555d-a715-0e47-bd0e-85d6eb8911e0 path slave
ID 259 gen 35116 top level 256parent_uuid-received_uuid9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 path 190312-010100
ID 3059 gen 7273 top level 256parent_uuid4a91d585-ab25-0c49-b183-ae951b5377e1received_uuid9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 114cd74f-80d8-af44-a9d0-fd8ce3c991b4 path 190315-001000You have two subvolumes with the same
received_uuid. Is the content of these subvolumes the same? Is "missing" file present in both subvolumes?Quoted 47 lines
ID 3072 gen 35112 top level 256
parent_uuid4a91d585-ab25-0c49-b183-ae951b5377e1received_uuid32dda1a2-07bd-684f-b773-c5b4cd5f29c1 uuid fe7281d1-8b3e-8f46-a0b4-cdc52a1bb7aa path back1 ID 3073 gen 36236 top level 256parent_uuid4a91d585-ab25-0c49-b183-ae951b5377e1received_uuida8e3e041-c1fc-e443-9baf-ab481ae32b8b uuid 65206183-f20d-4648-83ab-e0485b16f7af path back2# Files missing
aki@daisy /HySlave % ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CD
Little_My_MaidSMEEel_louisfirstkiss gakuensai infinity-kid kid-memories-offojyo_tokkyusamusupiwizards_harmonyyuukyuu-chara-sealaaki@daisy /HySlave % ls /HySlave/back1/local/pub/media/sound/notmusic_in_CD
Little_My_Maidel_louisfirstkiss gakuensai infinity-kid kid-memories-offojyo_tokkyusamusupiwizards_harmonyyuukyuu-chara-sealaaki@lily /mnt % ls
/mnt/190315-001000/local/pub/media/sound/notmusic_in_CDLittle_My_MaidSMEEel_louisfirstkiss gakuensai infinity-kid kid-memories-offojyo_tokkyusamusupiwizards_harmonyyuukyuu-chara-sealaki@lily /mnt % ls
/mnt/back1/local/pub/media/sound/notmusic_in_CDLittle_My_MaidSMEEel_louisfirstkiss gakuensai infinity-kid kid-memories-offojyo_tokkyusamusupiwizards_harmonyyuukyuu-chara-sealaaki@lily /mnt % ls
/mnt/world/local/pub/media/sound/notmusic_in_CDLittle_My_MaidSMEEel_louisfirstkiss gakuensai infinity-kid kid-memories-offojyo_tokkyusamusupiwizards_harmonyyuukyuu-chara-sealaSMEE is missing.
Thanks.
Quoted 15 lines · On Thu, 25 Apr 2019 20:55:59 +0300 Andrei Borzenkov <[email protected]> wrote:
25.04.2019 16:34, MASAKI Haruka пишет:
[...]Please show
1. Full protocol of commands you executed (commands and their output), e.g. captured using script.
2. Commands used to determine missing files on both source and receive sides.
3. Listing of subvolumes on both source and receive sides withbtrfs sub li -Rqu /mount/point
[...]
Quoted 14 lines · On Fri, 26 Apr 2019 21:27:52 +0300 Andrei Borzenkov <[email protected]> wrote:
26.04.2019 21:12, MASAKI Haruka пишет:
Quoted 6 lines
Quoted 2 lines
You have two subvolumes with the same
received_uuid. Is the content of these subvolumes the same? Is "missing" file present in both subvolumes?No, each subvolume name (190312-010100, 190315-001000) is timestamp.
These snapshot volumes were created by job scheduler on original (already lost) volume, so 190315-001000 is modified from 190312-010100.Which is your problem. Two subvolumes with the same
received_uuidare expected to have exactly the same content (received_uuidcan be viewed as dataset UUID). You send incremental stream against base uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2; on receiving side btrfs receive looks for matching base subvolume and happens to pick 190312-010100.
Quoted 7 lines
There were patches to automatically remove
received_uuidwhen subvolume is changed to read-write. I do not know whether they were applied.btrfs send/receive expects subvolumes to remain read-only. As soon as you start fiddling with read-only status you already lost. btrfs should have never allowed to change read-only into read-write in the first place.
Thanks it was solved!
I remember change attributes on original volume.
I deleted 190312-010100 and send back1, it's OK.
I learned that shouldn't change subvolume read-only to read-write.
Thank you very much!!!
Quoted 119 lines
Quoted 118 lines
190315-001000 has lost file,
% ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CD/Little_My_MaidSMEEel_louisfirstkiss gakuensai infinity-kid kid-memories-offojyo_tokkyusamusupiwizards_harmonyyuukyuu-chara-sealabut 190312-010100 doesn't.
% ls /HySlave/190312-010100/local/pub/media/sound/notmusic_in_CD/Little_My_Maidel_louisfirstkiss gakuensai infinity-kid kid-memories-offojyo_tokkyusamusupiwizards_harmonyyuukyuu-chara-sealaQuoted 104 lines · On Fri, 26 Apr 2019 20:30:07 +0300 Andrei Borzenkov <[email protected]> wrote:
26.04.2019 7:50, MASAKI Haruka пишет:
Quoted 43 lines
Thank you for your reply.
# What does
@Host A ("lily" master, new volume)
aki@lily /mnt % socat TCP-LISTEN:25500 STDOUT | sudo btrfs receive -vv .@Host B ("daisy" backup, having volumes)
aki@daisy /HySlave % sudo btrfs send -v 190315-001000 | socat STDIN TCP:lily.local:25500( ...Transfared 11TB... )
@Host A
aki@lily /mnt % sudo btrfs subvolume snapshot 190315-001000 world Create a snapshot of '190315-001000' in './world' aki@lily /mnt % sudo btrfs subvolume snapshot -r world back1 Create a readonly snapshot of 'world' in './back1'@Host A
aki@lily /mnt % sudo btrfs send -vv -p 190315-001000 back1 | socat STDIN tcp-connect:192.168.1.17:25555 At subvol back1BTRFS_IOC_SENDreturned 0
joining genl thread@Host B
root@daisy /HySlave # socat TCP-LISTEN:25555 STDOUT | btrfs receive -vv .
At snapshot back1
receiving snapshot back1 uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, ctransid=2658 parent_uuid=9c10b77f-74fb-f94a-9a24-3c6d30c00ca2, parent_ctransid=2658BTRFS_IOC_SET_RECEIVED_SUBVOLuuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, stransid=2658# Host A current
aki@lily /mnt % ls
190315-001000 back1 back2 world
aki@lily /mnt % sudo btrfs sub li -Rqu .
ID 258 gen 2698 top level 5parent_uuid-received_uuid9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid bc90d05c-7430-7b42-b683-702d906bb993 path 190315-001000ID 2139 gen 2696 top level 5 parent_uuid bc90d05c-7430-7b42-b683-702d906bb993 received_uuid - uuid 23c5fede-a338-d14a-afcd-677db2e97637 path world ID 2140 gen 2660 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid - uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 path back1 ID 2142 gen 2672 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid - uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b path back2# Host B current
Quoted 8 lines
aki@daisy /HySlave % pwd
/HySlave
aki@daisy /HySlave % ls
190312-010100 190315-001000 back1 back2
[aki@daisy HySlave]$ sudo btrfs sub li -Rqu .
ID 256 gen 36238 top level 5parent_uuid-received_uuid- uuid 4e42555d-a715-0e47-bd0e-85d6eb8911e0 path slave
ID 259 gen 35116 top level 256parent_uuid-received_uuid9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 path 190312-010100
ID 3059 gen 7273 top level 256parent_uuid4a91d585-ab25-0c49-b183-ae951b5377e1received_uuid9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 114cd74f-80d8-af44-a9d0-fd8ce3c991b4 path 190315-001000You have two subvolumes with the same
received_uuid. Is the content of these subvolumes the same? Is "missing" file present in both subvolumes?Quoted 47 lines
ID 3072 gen 35112 top level 256
parent_uuid4a91d585-ab25-0c49-b183-ae951b5377e1received_uuid32dda1a2-07bd-684f-b773-c5b4cd5f29c1 uuid fe7281d1-8b3e-8f46-a0b4-cdc52a1bb7aa path back1 ID 3073 gen 36236 top level 256parent_uuid4a91d585-ab25-0c49-b183-ae951b5377e1received_uuida8e3e041-c1fc-e443-9baf-ab481ae32b8b uuid 65206183-f20d-4648-83ab-e0485b16f7af path back2# Files missing
aki@daisy /HySlave % ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CD
Little_My_MaidSMEEel_louisfirstkiss gakuensai infinity-kid kid-memories-offojyo_tokkyusamusupiwizards_harmonyyuukyuu-chara-sealaaki@daisy /HySlave % ls /HySlave/back1/local/pub/media/sound/notmusic_in_CD
Little_My_Maidel_louisfirstkiss gakuensai infinity-kid kid-memories-offojyo_tokkyusamusupiwizards_harmonyyuukyuu-chara-sealaaki@lily /mnt % ls
/mnt/190315-001000/local/pub/media/sound/notmusic_in_CDLittle_My_MaidSMEEel_louisfirstkiss gakuensai infinity-kid kid-memories-offojyo_tokkyusamusupiwizards_harmonyyuukyuu-chara-sealaki@lily /mnt % ls
/mnt/back1/local/pub/media/sound/notmusic_in_CDLittle_My_MaidSMEEel_louisfirstkiss gakuensai infinity-kid kid-memories-offojyo_tokkyusamusupiwizards_harmonyyuukyuu-chara-sealaaki@lily /mnt % ls
/mnt/world/local/pub/media/sound/notmusic_in_CDLittle_My_MaidSMEEel_louisfirstkiss gakuensai infinity-kid kid-memories-offojyo_tokkyusamusupiwizards_harmonyyuukyuu-chara-sealaSMEE is missing.
Thanks.
Quoted 15 lines · On Thu, 25 Apr 2019 20:55:59 +0300 Andrei Borzenkov <[email protected]> wrote:
25.04.2019 16:34, MASAKI Haruka пишет:
[...]Please show
1. Full protocol of commands you executed (commands and their output), e.g. captured using script.
2. Commands used to determine missing files on both source and receive sides.
3. Listing of subvolumes on both source and receive sides withbtrfs sub li -Rqu /mount/point
[...]
Quoted 4 lines · Andrei Borzenkov
btrfs send/receive expects subvolumes to remain read-only. As soon as you start fiddling with read-only status you already lost. btrfs should have never allowed to change read-only into read-write in the first place.
I'm in agreement, and would like to see 'btrfs property set <> ro false> removed. It's confusing. Once a subvolume is read-only, it should remain read-only, and if the use case requires rw, then just snapshot it (default rw).
Signature · 1 line
Chris Murphy