Incremental send robustness question
Hi, all. I have a question about a backup plan I have involving send/receive. As far as I can tell, there's no way to to resume a send that has been interrupted. In this case, my interruption comes from an overbearing firewall that doesn't like long-lived connections. I'm trying to do the initial (non-incremental) sync of the first snapshot from my main server to my backup endpoint. The snapshot is ~900 GiB, and the internet link is 25 Mbps, so this'll be going for quite a long time.
What I would like to do is "fake" the first snapshot transfer by rsync-ing the files over. So my question is this: if I rsync a subvolume (with the -a option to make all file times, permissions, ownerships, etc. the same), is that good enough to then be used as a parent for future incremental sends?
And while we're at it, what are the failure modes for incremental sends? Will it throw an error if the parents don't match, or will there just be silent failures? I would imagine receive would barf if it was told to reference a file that didn't exist, but what if the referenced file is there but contains different data? Are there checks for this sort of thing, or is it always assumed that the parent subvols are identical and if they're not, you're in undefined behavior land?
Thanks,
--Sean
Quoted 8 lines · Sean Greenslade
Hi, all. I have a question about a backup plan I have involving send/receive. As far as I can tell, there's no way to to resume a send that has been interrupted. In this case, my interruption comes from an overbearing firewall that doesn't like long-lived connections. I'm trying to do the initial (non-incremental) sync of the first snapshot from my main server to my backup endpoint. The snapshot is ~900 GiB, and the internet link is 25 Mbps, so this'll be going for quite a long time.
Quoted 5 lines
What I would like to do is "fake" the first snapshot transfer by rsync-ing the files over. So my question is this: if I rsync a subvolume (with the -a option to make all file times, permissions, ownerships, etc. the same), is that good enough to then be used as a parent for future incremental sends?
No. There will be allocation differences in inode and number of extents, and whether they're shared. But even before that, incremental receive will fail immediately because it will see the parent isn't really on the receive side.
Quoted 3 lines
And while we're at it, what are the failure modes for incremental sends? Will it throw an error if the parents don't match, or will there just be silent failures?
It will give you an error that it can't find the parent.
Quoted 5 lines
I would imagine receive would barf if it was told to reference a file that didn't exist, but what if the referenced file is there but contains different data? Are there checks for this sort of thing, or is it always assumed that the parent subvols are identical and if they're not, you're in undefined behavior land?
I haven't tried to break this too much so I'm not sure.
https://btrfs.wiki.kernel.org/index.php/Design_notes_on_Send/Receive
Signature · 1 line
Chris Murphy
Quoted 8 lines · Sean Greenslade
Hi, all. I have a question about a backup plan I have involving send/receive. As far as I can tell, there's no way to to resume a send that has been interrupted. In this case, my interruption comes from an overbearing firewall that doesn't like long-lived connections. I'm trying to do the initial (non-incremental) sync of the first snapshot from my main server to my backup endpoint. The snapshot is ~900 GiB, and the internet link is 25 Mbps, so this'll be going for quite a long time.
Quoted 5 lines
What I would like to do is "fake" the first snapshot transfer by rsync-ing the files over. So my question is this: if I rsync a subvolume (with the -a option to make all file times, permissions, ownerships, etc. the same), is that good enough to then be used as a parent for future incremental sends?
No, it needs some additional subvol metadata to be set so that the
receiver can detect which subvol to snapshot when restoring an incremental.
Depending on how much local storage you have, you could do
something like write the send stream to disk (split into pieces), and then copy each piece to the receiver, and cat those all together to replay them. It'd be a bit hairy to get the timing right, though, without emptying or filling the various pipes.
Quoted 3 lines
And while we're at it, what are the failure modes for incremental sends? Will it throw an error if the parents don't match, or will there just be silent failures?
It'll throw an error, as far as I know.
Quoted 5 lines
I would imagine receive would barf if it was told to reference a file that didn't exist, but what if the referenced file is there but contains different data? Are there checks for this sort of thing, or is it always assumed that the parent subvols are identical and if they're not, you're in undefined behavior land?
The parent subvols have UUIDs in them which should act as a
sufficient check for any accidental errors. (See the received-subvol option for btrfs sub list). It's not robust against an attacker that can control the UUIDs... but then, if they can do that, there's a whole load more problems you've got.
Hugo.
Signature · 4 lines
Hugo Mills | Great oxymorons of the world, no. 8: hugo@... carfax.org.uk | The Latest In Proven Technology http://carfax.org.uk/ | PGP: E2AB1DE4 |
Quoted 7 lines · Sean Greenslade
Hi, all. I have a question about a backup plan I have involving send/receive. As far as I can tell, there's no way to to resume a send that has been interrupted. In this case, my interruption comes from an overbearing firewall that doesn't like long-lived connections. I'm trying to do the initial (non-incremental) sync of the first snapshot from my main server to my backup endpoint. The snapshot is ~900 GiB, and the internet link is 25 Mbps, so this'll be going for quite a long time.
You can't resume an interrupted send. You'll have to remove the target subvolume on the destination and start again.
Pipe the send into a local file, and then use any tool that can reliably resume interrupted transfers to get it to the other side.
Or, if faster, put in on a disk and drive there with your car. :)
At the receiving side, cat the file into btrfs receive.
Quoted 5 lines
What I would like to do is "fake" the first snapshot transfer by rsync-ing the files over. So my question is this: if I rsync a subvolume (with the -a option to make all file times, permissions, ownerships, etc. the same), is that good enough to then be used as a parent for future incremental sends?
No. btrfs send/receive works on a different level than rsync. Because btrfs internally tags all things that happen with a "generation" number, it can compare two snapshots which were taken from the same subvolume at different moments, and just walk the btrees and see which parts of it were updated between the previous and current snapshot.
The first send needs a read-only snapshot of a subvolume and if that one made it to the remote, you can send incrementals by taking another read only snapshot and letting btrfs decide what changed between them very efficiently.
That's why btrfs send -p can do very fast transfers of differences for snapshots, while rsync would need to compare all metadata of all files over and over again.
Quoted 3 lines
And while we're at it, what are the failure modes for incremental sends? Will it throw an error if the parents don't match, or will there just be silent failures?
Create a list of possibilities, create some test filesystems, try it.
Quoted 5 lines
I would imagine receive would barf if it was told to reference a file that didn't exist, but what if the referenced file is there but contains different data? Are there checks for this sort of thing, or is it always assumed that the parent subvols are identical and if they're not, you're in undefined behavior land?
btrfs send/receive is not rsync :-)
Signature · 1 line
Hans van Kranenburg
Quoted 9 lines · Hans van Kranenburg
Quoted 7 lines · Sean Greenslade
Hi, all. I have a question about a backup plan I have involving send/receive. As far as I can tell, there's no way to to resume a send that has been interrupted. In this case, my interruption comes from an overbearing firewall that doesn't like long-lived connections. I'm trying to do the initial (non-incremental) sync of the first snapshot from my main server to my backup endpoint. The snapshot is ~900 GiB, and the internet link is 25 Mbps, so this'll be going for quite a long time.
Quoted 7 lines
You can't resume an interrupted send. You'll have to remove the target subvolume on the destination and start again.
Pipe the send into a local file, and then use any tool that can reliably resume interrupted transfers to get it to the other side.
Or, if faster, put in on a disk and drive there with your car. :)
I may just end up doing that. Hugo's responce gave me some crazy ideas involving a custom build of split that waits for a command after each output file fills, which would of course require an equally weird build of cat that would stall the pipe indefinitely until all the files showed up. Driving the HDD over would probably be a little simpler. =P
Quoted 5 lines
Quoted 3 lines
And while we're at it, what are the failure modes for incremental sends? Will it throw an error if the parents don't match, or will there just be silent failures?
Create a list of possibilities, create some test filesystems, try it.
I may just do that, presuming I can find the spare time. Given that I'm building a backup solution around this tech, it would definitely bolster my confidence in it if I knew what its failure modes looked like.
Thanks to everyone for your fast responses.
--Sean
Quoted 11 lines · Sean Greenslade
Quoted 6 lines · Hans van Kranenburg
Quoted 3 lines · Sean Greenslade
And while we're at it, what are the failure modes for incremental sends? Will it throw an error if the parents don't match, or will there just be silent failures?
Create a list of possibilities, create some test filesystems, try it.
I may just do that, presuming I can find the spare time. Given that I'm building a backup solution around this tech, it would definitely bolster my confidence in it if I knew what its failure modes looked like.
Exactly :) Try to break it yourself before you have to start finding out how things behave 3AM monday morning under pressure.
Signature · 1 line
Hans van Kranenburg
Quoted 5 lines · Sean Greenslade
I may just end up doing that. Hugo's responce gave me some crazy ideas involving a custom build of split that waits for a command after each output file fills, which would of course require an equally weird build of cat that would stall the pipe indefinitely until all the files showed up. Driving the HDD over would probably be a little simpler. =P
I am sure it is, if that is an option. I had considered doing something
similar: doing an initial send to a big file on a spare disk, then sending the disk to Amazon to import using their Import/Export Disk service, then creating a server and a filesystem in AWS and doing a btrfs receive from the imported file. The plan would then be to do incremental sends over my home broadband line for subsequent backups.
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
And while we're at it, what are the failure modes for incremental sends? Will it throw an error if the parents don't match, or will there just be silent failures?
Create a list of possibilities, create some test filesystems, try it.
I may just do that, presuming I can find the spare time. Given that I'm building a backup solution around this tech, it would definitely bolster my confidence in it if I knew what its failure modes looked like.
That's a good idea. In the end I decided that relying on btrfs send for my offsite cold storage backups was probably not a good idea. Btrfs is great, and I heavily use snapshotting and send/receive locally. But there is always a small nagging fear that a btrfs bug could introduce some problem which can survive send/receive and mean the backup was corrupted as well (or that an incremental won't load for some reason). For that reason I decided to deliberately use a different technology for backups. I now use dar to create backups and then upload the files to a cloud cold storage service for safekeeping.
There are other reasons as well: encryption is easier to handle, cold storage for files is cheaper than having disk images which need to be online to load the incrementals, no need for a virtual server, handling security for backups from different servers with different levels of risk is easier, etc. There are also downsides: verifying the backups are readable/restorable is harder, bandwidth usage is less efficient (dar sends more data than btrfs send would as it is working at the file level, not the extent level).
By the way, to test out your various failure modes I recommend creating some small btrfs filesystems on loop devices -- just be careful to make sure you create each one from scratch and do not copy disk images (so that they all have unique UIDs).
Sean Greenslade posted on Wed, 12 Oct 2016 18:29:55 -0400 as excerpted:
Quoted 8 lines
Hi, all. I have a question about a backup plan I have involving send/receive. As far as I can tell, there's no way to to resume a send that has been interrupted. In this case, my interruption comes from an overbearing firewall that doesn't like long-lived connections. I'm trying to do the initial (non-incremental) sync of the first snapshot from my main server to my backup endpoint. The snapshot is ~900 GiB, and the internet link is 25 Mbps, so this'll be going for quite a long time.
Quoted 6 lines
What I would like to do is "fake" the first snapshot transfer by rsync-ing the files over. So my question is this: if I rsync a subvolume (with the -a option to make all file times, permissions, ownerships, etc. the same),
is that good enough to then be used as a parent for future incremental sends?
I see the specific questions have been answered, and alternatives explored in one direction, but I've another alternative, in a different direction, to suggest.
First a disclaimer. I'm a btrfs user/sysadmin and regular on the list, but I'm not a dev, and my own use-case doesn't involve send/receive, so what I know regarding send/receive is from the list and manpages, not personal experience. With that in mind...
It's worth noting that send/receive are subvolume-specific -- a send won't continue down into a subvolume.
Also note that in addition to -p/parent, there's -s/clone-src. The latter is more flexible than the super-strict parent option, at the expense of a fatter send-stream as additional metadata is sent that specifies which clone the instructions are relative to.
It should be possible to use the combination of these two facts to split and recombine your send stream in a firewall-timeout-friendly manner, as long as no individual files are so big that sending an individual file exceeds the timeout.
1) Start by taking a read-only snapshot of your intended source subvolume, so you have an unchanging reference.
2) Take multiple writable snapshots of it, and selectively delete subdirs (and files if necessary) from each writable snapshot, trimming each one to a size that should pass the firewall without interruption, so that the combination of all these smaller subvolumes contains the content of the single larger one.
3) Take read-only snapshots of each of these smaller snapshots, suitable for sending.
4) Do a non-incremental send of each of these smaller snapshots to the remote.
If it's practical to keep the subvolume divisions, you can simply split the working tree into subvolumes and send those individually instead of doing the snapshot splitting above, in which case you can then use -p/ parent on each as you were trying to do on the original, and you can stop here.
If you need/prefer the single subvolume, continue...
5) Do an incremental send of the original full snapshot, using multiple -c <src> options to list each of the smaller snapshots. Since all the data has already been transferred in the smaller snapshot sends, this send should be all metadata, no actual data. It'll simply be combining the individual reference subvolumes into a single larger subvolume once again.
6) Once you have the single larger subvolume on the receive side, you can delete the smaller snapshots as you now have a copy of the larger subvolume on each side to do further incremental sends of the working copy against.
7) I believe the first incremental send of the full working copy against the original larger snapshot will still have to use -c, while incremental sends based on that first one will be able to use the stricter but slimmer send-stream -p, with each one then using the previous one as the parent. However, I'm not sure on that. It may be that you have to continue using the fatter send-stream -c each time.
Again, I don't have send/receive experience of my own, so hopefully someone who does can reply either confirming that this should work and whether or not -p can be used after the initial setup, or explaining why the idea won't work, but at this point based on my own understanding, it seems like it should be perfectly workable to me. =:^)
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 83 lines · Duncan
I see the specific questions have been answered, and alternatives explored in one direction, but I've another alternative, in a different
direction, to suggest.
First a disclaimer. I'm a btrfs user/sysadmin and regular on the list,
but I'm not a dev, and my own use-case doesn't involve send/receive, so
what I know regarding send/receive is from the list and manpages, not personal experience. With that in mind...
It's worth noting that send/receive are subvolume-specific -- a send won't continue down into a subvolume.
Also note that in addition to -p/parent, there's -s/clone-src. The latter is more flexible than the super-strict parent option, at the expense of a fatter send-stream as additional metadata is sent that specifies which clone the instructions are relative to.
It should be possible to use the combination of these two facts to split
and recombine your send stream in a firewall-timeout-friendly manner, as
long as no individual files are so big that sending an individual file exceeds the timeout.1) Start by taking a read-only snapshot of your intended source subvolume, so you have an unchanging reference.
2) Take multiple writable snapshots of it, and selectively delete subdirs
(and files if necessary) from each writable snapshot, trimming each oneto a size that should pass the firewall without interruption, so that the
combination of all these smaller subvolumes contains the content of thesingle larger one.
3) Take read-only snapshots of each of these smaller snapshots, suitable
for sending.4) Do a non-incremental send of each of these smaller snapshots to the remote.
If it's practical to keep the subvolume divisions, you can simply split
the working tree into subvolumes and send those individually instead of
doing the snapshot splitting above, in which case you can then use -p/ parent on each as you were trying to do on the original, and you can stop
here.If you need/prefer the single subvolume, continue...
5) Do an incremental send of the original full snapshot, using multiple -c <src> options to list each of the smaller snapshots. Since all the data has already been transferred in the smaller snapshot sends, this send should be all metadata, no actual data. It'll simply be combining
the individual reference subvolumes into a single larger subvolume once
again.
6) Once you have the single larger subvolume on the receive side, you can
delete the smaller snapshots as you now have a copy of the larger subvolume on each side to do further incremental sends of the working copy against.7) I believe the first incremental send of the full working copy against
the original larger snapshot will still have to use -c, while incremental
sends based on that first one will be able to use the stricter but slimmer send-stream -p, with each one then using the previous one as the
parent. However, I'm not sure on that. It may be that you have to continue using the fatter send-stream -c each time.
Quoted 7 lines
Again, I don't have send/receive experience of my own, so hopefully someone who does can reply either confirming that this should work and whether or not -p can be used after the initial setup, or explaining why
the idea won't work, but at this point based on my own understanding, it
seems like it should be perfectly workable to me. =:^)
I was considering doing something like this, but the simple solution of "just bring the disk over" won out. If that hadn't been possible, I might have done something like that, and I'm still mulling over possible solutions to similar / related problems.
I think the biggest solution would be support for partial / resuming receives. That'll probably go on my ever-growing list of things to possibly look into when I happen upon some free time. It sounds quite complicated, though...
--Sean