btrfs double send
Hello,
I would like to do backups based on btrfs send/receive.
So I though I would do a transfer over portable HDD and then incremental sends (using -p) over network.
Initial :
btrfs send "all" (primary HDD) → btrfs receive (portable HDD) → btrfs send/receive (final HDD)
Later :
btrfs send -p "all" new → ssh newhost "btrfs receive … "
But I found out that does not work. It complains parent not found.
Am I doing something wrong, or is it intended behaviour?
Best regards,
Lubos
Quoted 12 lines · On Sat, Oct 24, 2015 at 6:36 PM, <[email protected]> wrote:
Hello,
I would like to do backups based on btrfs send/receive.
So I though I would do a transfer over portable HDD and then incremental sends (using -p) over network.
Initial :
btrfs send "all" (primary HDD) → btrfs receive (portable HDD) → btrfs send/receive (final HDD)
Quoted 7 lines
Later :
btrfs send -p "all" new → ssh newhost "btrfs receive … "
But I found out that does not work. It complains parent not found.
Am I doing something wrong, or is it intended behaviour?
It's a bug, we got it recently reported and fixed.
The fix is in Chris' integration branch for the upcoming 4.4 merge window:
cheers
Quoted 8 lines
Best regards,
Lubos
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 5 lines
Filipe David Manana, "Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men."
Filipe Manana posted on Sat, 24 Oct 2015 18:52:21 +0100 as excerpted:
Quoted 19 lines · On Sat, Oct 24, 2015 at 6:36 PM, <[email protected]> wrote:
Quoted 18 lines
Hello,
I would like to do backups based on btrfs send/receive.
So I though I would do a transfer over portable HDD and then incremental sends (using -p) over network.
Initial :
btrfs send "all" (primary HDD) → btrfs receive (portable HDD) → btrfs send/receive (final HDD)
Later :
btrfs send -p "all" new → ssh newhost "btrfs receive … "
But I found out that does not work. It complains parent not found.
Quoted 7 lines
Quoted 1 line
Am I doing something wrong, or is it intended behaviour?
It's a bug, we got it recently reported and fixed.
The fix is in Chris' integration branch for the upcoming 4.4 merge window:http://git.kernel.org/cgit/linux/kernel/git/mason/linux-btrfs.git/
commit/?h=integration-4.4&id=b96b1db039ebc584d03a9933b279e0d3e704c528
As a current workaround...
It's worth noting that btrfs send outputs a stream, while btrfs receives a stream, but they don't have to be hooked up directly in real-time. Both send and receive have parameters that allow sending output to, and receiving input from, a file, and of course since they work with STDOUT/ STDIN, even if explicit file options weren't supplied, shell redirection can be used to do it.
So it's quite possible to send to a file on the the portable device instead of to btrfs receive, then unplug the device and plug it in on the target, receiving from the file there. That avoids the intermediate receive/send and thus the problem, since it's ultimately (a copy of) the original send stream now being received by receive on the target machine.
That also allows additional flexibility, since the intermediate/portable device now doesn't have to have btrfs on it; it can be ext* or whatever. In fact, the portable device shouldn't even need a filesystem at all, as sending directly to the device-file and receiving from it should work. Send/receive works with a stream after all, and anything that can accurately store and replay that stream should be sufficient. =:^)
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Filipe,
Its still not perfect. Here I can do sequential sends a few times then I get something like this:
[root@grover snap]# sh -x brh
+ base=/snap/shot
++ date +%Y-%V-%u_%m-%d_%H:%M
+ stamp=2015-43-6_10-24_10:24
+ btrfs subv snapshot -r /snap/homevol /snap/shot.2015-43-6_10-24_10:24
Create a readonly snapshot of '/snap/homevol' in '/snap/shot.2015-43-6_10-24_10:24'
+ sync
+ /usr/bin/time -f 'send %E' nice -19 ionice -c3 btrfs send -v -p /snap/shot /snap/shot.2015-43-6_10-24_10:24
+ btrfs receive -v /backup/snap
At subvol /snap/shot.2015-43-6_10-24_10:24
At snapshot shot.2015-43-6_10-24_10:24
receiving snapshot shot.2015-43-6_10-24_10:24 uuid=cb3ad856-ca0f-f744-b7c3-b33f2d5bc8d3, ctransid=625267 parent_uuid=cb3ad856-ca0f-f744-b7c3-b33f2d5bc8d3, parent_ctransid=619893
ERROR: unlink home/ed/Maildir/.spam/dovecot.index.log.2 failed. No such file or directory
Command terminated by signal 13
send 1:49.64
+ sync
+ btrfs subv delete /snap/shot
Delete subvolume (no-commit): '/snap/shot'
+ sync
+ mv /snap/shot.2015-43-6_10-24_10:24 /snap/shot
so there is another bug hiding in the code, its usually something in my Maildir that shows in the log.
Please let me know if there is anything you would like me to try. I am running 4.2 with the 4.3 for-linus tree applied and the 4.2.x patches with btrfs fixes removed. On top of this are a few patches from this list.
TIA
Ed Tomlinson
Quoted 27 lines · Filipe Manana
Quoted 8 lines · On Sat, Oct 24, 2015 at 6:36 PM, <[email protected]> wrote:
Hello,
I would like to do backups based on btrfs send/receive.
So I though I would do a transfer over portable HDD and then incremental sends (using -p) over network.
Initial : ...
It's a bug, we got it recently reported and fixed.
The fix is in Chris' integration branch for the upcoming 4.4 merge window:cheers
Quoted 8 lines
Best regards,
Lubos
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 32 lines · Ed Tomlinson
Filipe,
Its still not perfect. Here I can do sequential sends a few times then I get something like this:
[root@grover snap]# sh -x brh
+ base=/snap/shot
++ date +%Y-%V-%u_%m-%d_%H:%M
+ stamp=2015-43-6_10-24_10:24
+ btrfs subv snapshot -r /snap/homevol /snap/shot.2015-43-6_10-24_10:24 Create a readonly snapshot of '/snap/homevol' in '/snap/shot.2015-43-6_10-24_10:24'
+ sync
+/usr/bin/time-f 'send %E' nice -19 ionice -c3 btrfs send -v -p /snap/shot /snap/shot.2015-43-6_10-24_10:24
+ btrfs receive -v /backup/snap
At subvol /snap/shot.2015-43-6_10-24_10:24
At snapshot shot.2015-43-6_10-24_10:24
receiving snapshot shot.2015-43-6_10-24_10:24 uuid=cb3ad856-ca0f-f744-b7c3-b33f2d5bc8d3, ctransid=625267 parent_uuid=cb3ad856-ca0f-f744-b7c3-b33f2d5bc8d3, parent_ctransid=619893
ERROR: unlink home/ed/Maildir/.spam/dovecot.index.log.2 failed. No such file or directory
Command terminated by signal 13
send 1:49.64
+ sync
+ btrfs subv delete /snap/shot
Delete subvolume (no-commit): '/snap/shot'
+ sync
+ mv /snap/shot.2015-43-6_10-24_10:24 /snap/shot
Quoted 2 lines
so there is another bug hiding in the code, its usually something in my Maildir that shows in the log.
Different and unrelated problem.
Yes, we know there are still some problems regarding send issuing invalid/outdated paths to the send stream. Nothing to do with incorrect uuids in the send stream.
Quoted 43 lines
Please let me know if there is anything you would like me to try. I am running 4.2 with the 4.3 for-linus tree applied and the 4.2.x patches with btrfs fixes removed. On top of this are a few patches from this list.
TIA
Ed TomlinsonQuoted 33 lines · Filipe Manana
Quoted 9 lines · On Sat, Oct 24, 2015 at 6:36 PM, <[email protected]> wrote:
Hello,
I would like to do backups based on btrfs send/receive.
So I though I would do a transfer over portable HDD and then incremental sends (using -p) over network.
Initial : ...
It's a bug, we got it recently reported and fixed.
The fix is in Chris' integration branch for the upcoming 4.4 merge window:cheers
Quoted 9 lines
Best regards,
Lubos
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 5 lines
Filipe David Manana, "Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men."
Filipe,
I realize its another bug. Two additional pieces of info that might help.
One, btrfs-progs was at 4.1.2 (I missed a tag= in my git pull). Second
is that I have been able to recreate this issue three times over a period of
two to three days (tring again with 4.2.3). My fs is probably a good testcase for any patches that appear.
Meanwhile I've been falling back to rsync which always works but is so much slower.
TIA
Ed
Quoted 23 lines · Filipe Manana
Quoted 7 lines · Ed Tomlinson
Filipe,
Its still not perfect. Here I can do sequential sends a few times then I get something like this:
[root@grover snap]# sh -x brh
+ base=/snap/shot ...Different and unrelated problem.
Yes, we know there are still some problems regarding send issuing invalid/outdated paths to the send stream. Nothing to do with incorrect uuids in the send stream.Quoted 6 lines
Please let me know if there is anything you would like me to try. I am running 4.2 with the 4.3 for-linus tree applied and the 4.2.x patches with btrfs fixes removed. On top of this are a few patches from this list.
TIA
Ed Tomlinson ...
Quoted 4 lines · Ed Tomlinson
Filipe,
I realize its another bug. Two additional pieces of info that might help. One, btrfs-progs was at 4.1.2 (I missed a tag= in my git pull).
Btrfs-progs' version is irrelevant. The problem is the send side generating incorrect paths in the send stream (as I mentioned earlier), so it's exclusively a problem in the kernel.
Quoted 4 lines
Second
is that I have been able to recreate this issue three times over a period of two to three days (tring again with 4.2.3). My fs is probably a good testcase for any patches that appear.
Yeah.
If you can find out what happened to the directory/file in question between the parent and send snapshots (was it renamed/moved? was it deleted and then a new one with the same name/location created?) that helps. Otherwise looking at btrfs-debug-tree and see the trees for each snapshot helps too, but you need to figure out the number of the inode where it's failing (sometimes btrfs receive -vv is enough to figure this out).
thanks
Quoted 36 lines
Meanwhile I've been falling back to rsync which always works but is so much slower.
TIA
EdQuoted 28 lines · Filipe Manana
Quoted 8 lines · Ed Tomlinson
Filipe,
Its still not perfect. Here I can do sequential sends a few times then I get something like this:
[root@grover snap]# sh -x brh
+ base=/snap/shot ...Different and unrelated problem.
Yes, we know there are still some problems regarding send issuing invalid/outdated paths to the send stream. Nothing to do with incorrect uuids in the send stream.Quoted 7 lines
Please let me know if there is anything you would like me to try. I am running 4.2 with the 4.3 for-linus tree applied and the 4.2.x patches with
btrfs fixes removed. On top of this are a few patches from this list.TIA
Ed Tomlinson ...
Signature · 5 lines
Filipe David Manana, "Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men."