btrfs send / receive via netcat, fails halfway?
I'm trying to transfer a btrfs snapshot via the network.
First attempt: Both NC programs don't exit after the transfer is complete. When I ctrl-C the sending side, the receiving side exits OK.
btrfs subvolume delete /mnt/rec/snapshots/*
receive side:
# nc -l -p 6790 | btrfs receive /mnt/rec/snapshots At subvol 0
sending side:
# btrfs send /mnt/send/snapshots/0 | nc -v 127.0.0.1 6790 At subvol /mnt/send/snapshots/0 localhost [127.0.0.1] 6790 (hnmp) open
Second attempt: both nc programs exit ok at snapshot 0,1,2, but snapshot3 fails halfway, and 4 fails, as 3 is not complete. receive side:
# nc -l -p 6790 | btrfs receive /mnt/rec/snapshots At subvol 0 # nc -l -p 6790 | btrfs receive /mnt/rec/snapshots At snapshot 1 # nc -l -p 6790 | btrfs receive /mnt/rec/snapshots At snapshot 2 # nc -l -p 6790 | btrfs receive /mnt/rec/snapshots At snapshot 3 read(net): Connection reset by peer ERROR: short read from stream: expected 49183 read 10450 # nc -l -p 6790 | btrfs receive /mnt/rec/snapshots At snapshot 4 ERROR: cannot find parent subvolume write(stdout): Broken pipe
sending side:
# btrfs send /mnt/send/snapshots/0 | nc -v -c 127.0.0.1 6790 At subvol /mnt/send/snapshots/0 localhost [127.0.0.1] 6790 (hnmp) open # btrfs send -p /mnt/send/snapshots/0 /mnt/send/snapshots/1 | nc -v -c 127.0.0.1 6790 At subvol /mnt/send/snapshots/1 localhost [127.0.0.1] 6790 (hnmp) open # btrfs send -p /mnt/send/snapshots/1 /mnt/send/snapshots/2 | nc -v -c 127.0.0.1 6790 At subvol /mnt/send/snapshots/2 localhost [127.0.0.1] 6790 (hnmp) open # btrfs send -p /mnt/send/snapshots/2 /mnt/send/snapshots/3 | nc -v -c 127.0.0.1 6790 At subvol /mnt/send/snapshots/3 localhost [127.0.0.1] 6790 (hnmp) open # btrfs send -p /mnt/send/snapshots/3 /mnt/send/snapshots/4 | nc -v -c 127.0.0.1 6790 At subvol /mnt/send/snapshots/4 localhost [127.0.0.1] 6790 (hnmp) open write(net): Connection reset by peer
---
Take your mailboxes with you. Free, fast and secure Mail & Cloud: https://www.eclipso.eu - Time to change!
Quoted 34 lines · On 2021-01-10 11:34, [email protected] wrote:
I'm trying to transfer a btrfs snapshot via the network.
First attempt: Both NC programs don't exit after the transfer is complete. When I ctrl-C the sending side, the receiving side exits OK.
btrfs subvolume delete
/mnt/rec/snapshots/*
receive side:# nc -l -p 6790 | btrfs receive /mnt/rec/snapshots At subvol 0sending side:
# btrfs send /mnt/send/snapshots/0 | nc -v 127.0.0.1 6790 At subvol /mnt/send/snapshots/0 localhost [127.0.0.1] 6790 (hnmp) openSecond attempt: both nc programs exit ok at snapshot 0,1,2, but snapshot3 fails halfway, and 4 fails, as 3 is not complete. receive side:
# nc -l -p 6790 | btrfs receive /mnt/rec/snapshots At subvol 0 # nc -l -p 6790 | btrfs receive /mnt/rec/snapshots At snapshot 1 # nc -l -p 6790 | btrfs receive /mnt/rec/snapshots At snapshot 2 # nc -l -p 6790 | btrfs receive /mnt/rec/snapshots At snapshot 3 read(net): Connection reset by peer ERROR: short read from stream: expected 49183 read 10450 # nc -l -p 6790 | btrfs receive /mnt/rec/snapshots At snapshot 4 ERROR: cannot find parent subvolume write(stdout): Broken pipesending side:
Quoted 19 lines
# btrfs send /mnt/send/snapshots/0 | nc -v -c 127.0.0.1 6790 At subvol /mnt/send/snapshots/0 localhost [127.0.0.1] 6790 (hnmp) open # btrfs send -p /mnt/send/snapshots/0 /mnt/send/snapshots/1 | nc -v -c 127.0.0.1 6790 At subvol /mnt/send/snapshots/1 localhost [127.0.0.1] 6790 (hnmp) open # btrfs send -p /mnt/send/snapshots/1 /mnt/send/snapshots/2 | nc -v -c 127.0.0.1 6790 At subvol /mnt/send/snapshots/2 localhost [127.0.0.1] 6790 (hnmp) open # btrfs send -p /mnt/send/snapshots/2 /mnt/send/snapshots/3 | nc -v -c 127.0.0.1 6790 At subvol /mnt/send/snapshots/3 localhost [127.0.0.1] 6790 (hnmp) open # btrfs send -p /mnt/send/snapshots/3 /mnt/send/snapshots/4 | nc -v -c 127.0.0.1 6790 At subvol /mnt/send/snapshots/4 localhost [127.0.0.1] 6790 (hnmp) open write(net): Connection reset by peer
Hi,
Haven't used netcat for btrfs send, so I am not sure why you have these issues. Have you tried mbuffer instead? This is what I use when I don't need to go over ssh.
Mbuffer[1] allows you to specify remote host and port. It also has an async buffer so it should be a bit faster too. In fact, it is useful on a localhost too, for example when sending snaps to a slower local backup media.
Receiving side:
# mbuffer -I port | btrfs receive /mnt/rec/snapshots
Sending side:
# btrfs send /mnt/send/snapshots/0 | mbuffer -O remotehost:port
Good Luck!
//Forza
Quoted 3 lines · On Sun, 10 Jan 2021 11:34:27 +0100 " " <[email protected]> wrote:
I'm trying to transfer a btrfs snapshot via the network.
First attempt: Both NC programs don't exit after the transfer is complete. When I ctrl-C the sending side, the receiving side exits OK.
It is a common annoyance that NC doesn't exit in such scenario and needs to be Ctrl-C'ed after verifying that the transfer is over.
Instead, at host2 try:
ssh host1 "btrfs send ..." | btrfs receive ...
Also much more secure.
Signature · 2 lines
With respect, Roman
It is a common annoyance that NC doesn't exit in such scenario and needs to be
Ctrl-C'ed after verifying that the transfer is over.
Instead, at host2 try:
ssh host1 "btrfs send ..." | btrfs receive ...
Also much more secure.
Signature · 9 lines
With respect, Roman I was practicing on a test system, the real transfer was supposed to be between two systems on a LAN. I didn't want encryption, as my system then only is capable of sending 30MB/sec, instead of the 110MB/sec over the 1GB network. --- Take your mailboxes with you. Free, fast and secure Mail & Cloud: https://www.eclipso.eu - Time to change!
Quoted 6 lines · On Sun, Jan 10, 2021 at 11:34:27AM +0100, wrote:
I'm trying to transfer a btrfs snapshot via the network.
First attempt: Both NC programs don't exit after the transfer is complete. When I ctrl-C the sending side, the receiving side exits OK.
btrfs subvolume delete
/mnt/rec/snapshots/*
receive side:
Quoted 7 lines
# nc -l -p 6790 | btrfs receive /mnt/rec/snapshots At subvol 0sending side:
# btrfs send /mnt/send/snapshots/0 | nc -v 127.0.0.1 6790 At subvol /mnt/send/snapshots/0 localhost [127.0.0.1] 6790 (hnmp) open
Use -q 15 on the sending side. That will exit after 15 seconds of
no activity from the send process.
Quoted 12 lines
Second attempt: both nc programs exit ok at snapshot 0,1,2, but snapshot3 fails halfway, and 4 fails, as 3 is not complete. receive side:
# nc -l -p 6790 | btrfs receive /mnt/rec/snapshots At subvol 0 # nc -l -p 6790 | btrfs receive /mnt/rec/snapshots At snapshot 1 # nc -l -p 6790 | btrfs receive /mnt/rec/snapshots At snapshot 2 # nc -l -p 6790 | btrfs receive /mnt/rec/snapshots At snapshot 3 read(net): Connection reset by peer ERROR: short read from stream: expected 49183 read 10450
This failed because of a network disconnect.
Quoted 4 lines
# nc -l -p 6790 | btrfs receive /mnt/rec/snapshots At snapshot 4 ERROR: cannot find parent subvolume write(stdout): Broken pipe
This is expected because the previous one failed.
There's no btrfs problem here. You just need better error handling
(to retry a failed transfer, for example).
Hugo.
Signature · 4 lines
Hugo Mills | How do you become King? You stand in the marketplace hugo@... carfax.org.uk | and announce you're going to tax everyone. If you http://carfax.org.uk/ | get out alive, you're King. PGP: E2AB1DE4 | Harry Harrison
By the way, Cedric, your SMTP server is rejecting mail from mine
with a "forged HELO" error. I'm not sure why, but I've not knowingly encountered this with anyone else's mail server.
Hugo.
Signature · 4 lines
Hugo Mills | The last man on Earth sat in a room. Suddenly, there hugo@... carfax.org.uk | was a knock at the door. http://carfax.org.uk/ | PGP: E2AB1DE4 | Frederic Brown