Incremental backup for a raid1
My backup use case is different from the what has been recently discussed in another thread. I'm trying to guard against hardware failure and other causes of destruction.
I have a btrfs raid1 filesystem spread over two disks. I want to backup this filesystem regularly and efficiently to an external disk (same model as the ones in the raid) in such a way that
* when one disk in the raid fails, I can substitute the backup and rebalancing from the surviving disk to the substitute only applies the missing changes.
* when the entire raid fails, I can re-build a new one from the backup.
The filesystem is mounted at its root and has several nested subvolumes and snapshots (in a .snapshots subdir on each subvol).
Is it possible to do what I'm looking for?
Michael
Signature · 3 lines
Michael Schuerig mailto:[email protected] http://www.schuerig.de/michael/
Quoted 13 lines · Michael Schuerig
My backup use case is different from the what has been recently discussed in another thread. I'm trying to guard against hardware failure and other causes of destruction.
I have a btrfs raid1 filesystem spread over two disks. I want to backup this filesystem regularly and efficiently to an external disk (same model as the ones in the raid) in such a way that
* when one disk in the raid fails, I can substitute the backup and rebalancing from the surviving disk to the substitute only applies the missing changes.
Quoted 6 lines
* when the entire raid fails, I can re-build a new one from the backup.
The filesystem is mounted at its root and has several nested subvolumes and snapshots (in a .snapshots subdir on each subvol).
Is it possible to do what I'm looking for?
For point 2, yes. (Add new disk, balance -oconvert from single to
raid1).
For point 1, not really. It's a different filesystem, so it'll have
a different UUID. You *might* be able to get away with rsync of one of the block devices in the array to the backup block device, but you'd have to unmount the FS (or halt all writes to it) for the period of the rsync to ensure a consistent image, and the rsync would have to read all the data in the device being synced to work out what to send. Probably not what you want.
Hugo.
Signature · 4 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- Do not meddle in the affairs of system administrators, for ---
they are subtle, and quick to anger. Quoted 32 lines · Hugo Mills
Quoted 13 lines · Michael Schuerig
My backup use case is different from the what has been recently discussed in another thread. I'm trying to guard against hardware failure and other causes of destruction.
I have a btrfs raid1 filesystem spread over two disks. I want to backup this filesystem regularly and efficiently to an external disk (same model as the ones in the raid) in such a way that
* when one disk in the raid fails, I can substitute the backup and rebalancing from the surviving disk to the substitute only applies the missing changes.
Quoted 6 lines
* when the entire raid fails, I can re-build a new one from the backup.
The filesystem is mounted at its root and has several nested subvolumes and snapshots (in a .snapshots subdir on each subvol).
Is it possible to do what I'm looking for?
For point 2, yes. (Add new disk, balance -oconvert from single to
raid1).For point 1, not really. It's a different filesystem, so it'll have
a different UUID. You *might* be able to get away with rsync of one of the block devices in the array to the backup block device, but you'd have to unmount the FS (or halt all writes to it) for the period of the rsync to ensure a consistent image, and the rsync would have to read all the data in the device being synced to work out what to send. Probably not what you want.
Quoted 2 lines
Hugo.
I'm new; btrfs noob; completely unqualified to write intelligently on this topic, nevertheless:
I understand your setup to be btrfs RAID1 with /dev/A /dev/B, and a backup device someplace /dev/C
Could you, at the time you wanted to backup the filesystem:
1) in the filesystem, break RAID1: /dev/A /dev/B <-- remove /dev/B
2) reestablish RAID1 to the backup device: /dev/A /dev/C <-- added
3) balance to effect the backup (i.e. rebuilding the RAID1 onto /dev/C)
4) break/reconnect the original devices: remove /dev/C; re-add /dev/B to the fs
I think this could be done online. Any one device [ABC] surviving is sufficient to rebuild a RAID1 of the filesystem, or be accessed alone in degraded fashion for disaster recovery purposes.
I think that would address point 1. Is my thinking horrible on this? (again, noob to btrfs)
Quoted 6 lines · Andrew Skretvedt
Quoted 5 lines · Hugo Mills
Quoted 4 lines · Michael Schuerig
I have a btrfs raid1 filesystem spread over two disks. I want to backup this filesystem regularly and efficiently to an external disk (same model as the ones in the raid) in such a way that
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
* when one disk in the raid fails, I can substitute the backup and rebalancing from the surviving disk to the substitute only applies the missing changes.
For point 1, not really. It's a different filesystem
[snip]
Hugo.I'm new
We all start somewhere. ;)
Quoted 3 lines
Could you, at the time you wanted to backup the filesystem:
1) in the filesystem, break RAID1:/dev/A/dev/B<-- remove/dev/B
2) reestablish RAID1 to the backup device:/dev/A/dev/C<-- added
Its this step that won't work "as is" and, from an outsider's perspective, it is not obvious why:
As Hugo mentioned, "It's a different filesystem". The two disks don't have any "co-ordinating" record of data and don't have any record indicating that the other disk even exists. The files they store might even be identical - but there's a lot of missing information that would be necessary to tell them they can work together. All this will do is reformat /dev/C and then it will be rewritten again by the balance operation in step 3) below.
Quoted 2 lines
3) balance to effect the backup (i.e. rebuilding the RAID1 onto
/dev/C)
4) break/reconnect the original devices: remove/dev/C; re-add/dev/Bto the fs
Again, as with 2), /dev/A is now synchronised with (for all intents and purposes) a new disk. If you want to re-add /dev/B, you're going to lose any data on /dev/B (view this in the sense that, if you wiped the disk, the end-result would be the same) and then you would be re-balancing new data onto it from scratch.
Before removing /dev/B:
Disk A: abdeg__cf__
Disk B: abc_df_ge__ <- note that data is *not* necessarily stored in the exact same position on both disks Disk C: gbfc_d__a_e
All data is available on all disks. Disk C has no record indicating that disks A and B exist.
Disk A and B have a record indicating that the other disk is part of the same FS. These two disks have no record indicating disk C exists.
1. Remove /dev/B:
Disk A: abdeg__cf__
Disk C: gbfc_d__a_e
2. Add /dev/C to /dev/A as RAID1:
Disk A: abdeg__cf__
Disk C: _########## <- system reformats /dev/C and treats the old data as garbage
3. Balance /dev/{A,C}:
Disk A: abdeg__cf__
Disk C: abcdefg____
Both disks now have a full record of where the data is supposed to be and have a record indicating that the other disk is part of the FS. Notice that, though Disk C has the exact same files as it did before step 1, the on-disk filesystem looks very different.
4. Follow steps 1, 2, and 3 above - but with different disks - similar end-result.
Signature · 4 lines
__________ Brendan Hide http://swiftspirit.co.za/ http://www.webafrica.co.za/?AFF1E97
Quoted 19 lines · On Thursday 13 March 2014 14:48:55 Andrew Skretvedt wrote:
Quoted 18 lines · Hugo Mills
Quoted 17 lines · Michael Schuerig
My backup use case is different from the what has been recently discussed in another thread. I'm trying to guard against hardware failure and other causes of destruction.
I have a btrfs raid1 filesystem spread over two disks. I want to backup this filesystem regularly and efficiently to an external disk (same model as the ones in the raid) in such a way that
* when one disk in the raid fails, I can substitute the backup and rebalancing from the surviving disk to the substitute only applies the missing changes.
* when the entire raid fails, I can re-build a new one from the backup.
The filesystem is mounted at its root and has several nested subvolumes and snapshots (in a .snapshots subdir on each subvol).
[...]
Quoted 11 lines
I'm new; btrfs noob; completely unqualified to write intelligently on this topic, nevertheless:
I understand your setup to be btrfs RAID1 with/dev/A/dev/B, and a backup device someplace/dev/CCould you, at the time you wanted to backup the filesystem:
1) in the filesystem, break RAID1:/dev/A/dev/B<-- remove/dev/B
2) reestablish RAID1 to the backup device:/dev/A/dev/C<-- added
3) balance to effect the backup (i.e. rebuilding the RAID1 onto/dev/C) 4) break/reconnect the original devices: remove/dev/C; re-add/dev/Bto the fs
I've thought of this but don't dare try it without approval from the experts. At any rate, for being practical, this approach hinges on an ability to rebuild the raid1 incrementally. That is, the rebuild would have to start from what already is present on disk B (or C, when it is re-added). Starting from an effectively blank disk each time would be prohibitive.
Even if this would work, I'd much prefer keeping the original raid1 intact and to only temporarily add another mirror: "lazy mirroring", to give the thing a name.
Michael
Signature · 3 lines
Michael Schuerig mailto:[email protected] http://www.schuerig.de/michael/
Quoted 41 lines · Michael Schuerig
Quoted 19 lines · On Thursday 13 March 2014 14:48:55 Andrew Skretvedt wrote:
Quoted 18 lines · Hugo Mills
Quoted 17 lines · Michael Schuerig
My backup use case is different from the what has been recently discussed in another thread. I'm trying to guard against hardware failure and other causes of destruction.
I have a btrfs raid1 filesystem spread over two disks. I want to backup this filesystem regularly and efficiently to an external disk (same model as the ones in the raid) in such a way that
* when one disk in the raid fails, I can substitute the backup and rebalancing from the surviving disk to the substitute only applies the missing changes.
* when the entire raid fails, I can re-build a new one from the backup.
The filesystem is mounted at its root and has several nested subvolumes and snapshots (in a .snapshots subdir on each subvol).
[...]
Quoted 11 lines
I'm new; btrfs noob; completely unqualified to write intelligently on this topic, nevertheless:
I understand your setup to be btrfs RAID1 with/dev/A/dev/B, and a backup device someplace/dev/CCould you, at the time you wanted to backup the filesystem:
1) in the filesystem, break RAID1:/dev/A/dev/B<-- remove/dev/B
2) reestablish RAID1 to the backup device:/dev/A/dev/C<-- added
3) balance to effect the backup (i.e. rebuilding the RAID1 onto/dev/C) 4) break/reconnect the original devices: remove/dev/C; re-add/dev/Bto the fsI've thought of this but don't dare try it without approval from the experts. At any rate, for being practical, this approach hinges on an ability to rebuild the raid1 incrementally. That is, the rebuild would have to start from what already is present on disk B (or C, when it is re-added). Starting from an effectively blank disk each time would be prohibitive.
Quoted 3 lines
Even if this would work, I'd much prefer keeping the original raid1 intact and to only temporarily add another mirror: "lazy mirroring", to give the thing a name.
At best this seems fragile, but I don't think it works and is an edge case from the start. This is what send/receive is for.
In the btrfs replace scenario, the missing device is removed from the volume. It's like a divorce. Missing device 2 is replaced by a different physical device also called device 2. If you then removed 2b and readd (formerly replaced) device 2a, what happens? I don't know, I'm pretty sure the volume knows this is not device 2b as it should be, and won't accept formerly replaced device 2a. But it's an edge case to do this because you've said "device replace". So lexicon wise, I wouldn't even want this to work, we'd need a different command even if not different logic.
In the btfs device add case, you now have a three disk raid1 which is a whole different beast. Since this isn't n-way raid1, each disk is not stand alone. You're only assured data survives a one disk failure meaning you must have two drives. You've just increased your risk by doing this, not reduced it. It further proposes running an (ostensibly) production workflow with an always degraded volume, mounted with -o degraded, on an on-going basis. So it's three strikes. It's not n-way, you have no uptime if you lose one of two disks onsite, you's have to go get the offsite/onshelf disk to keep working. Plus that offsite disk isn't stand alone, so why even have it offsite? This is a fail.
So the btrfs replace scenario might work but it seems like a bad idea. And overall it's a use case for which send/receive was designed anyway so why not just use that?
Chris Murphy
Quoted 1 line · On Thursday 13 March 2014 16:04:33 Chris Murphy wrote:
On Mar 13, 2014, at 3:14 PM, Michael Schuerig
Quoted 49 lines · <[email protected]> wrote:
Quoted 49 lines · On Thursday 13 March 2014 14:48:55 Andrew Skretvedt wrote:
Quoted 21 lines · Hugo Mills
Quoted 20 lines · Michael Schuerig
Quoted 19 lines
My backup use case is different from the what has been recently discussed in another thread. I'm trying to guard against hardware failure and other causes of destruction.
I have a btrfs raid1 filesystem spread over two disks. I want to backup this filesystem regularly and efficiently to an external disk (same model as the ones in the raid) in such a way that
* when one disk in the raid fails, I can substitute the backup and
rebalancing from the surviving disk to the substitute only applies
the missing changes.* when the entire raid fails, I can re-build a new one from the backup.
The filesystem is mounted at its root and has several nested subvolumes and snapshots (in a .snapshots subdir on each subvol).
[...]
Quoted 6 lines
I'm new; btrfs noob; completely unqualified to write intelligently on
this topic, nevertheless:
I understand your setup to be btrfs RAID1 with/dev/A/dev/B, and a backup device someplace/dev/CQuoted 6 lines
Could you, at the time you wanted to backup the filesystem:
1) in the filesystem, break RAID1:/dev/A/dev/B<-- remove/dev/B
2) reestablish RAID1 to the backup device:/dev/A/dev/C<-- added
3) balance to effect the backup (i.e. rebuilding the RAID1 onto/dev/C) 4) break/reconnect the original devices: remove/dev/C; re-add/dev/Bto the fsI've thought of this but don't dare try it without approval from the experts. At any rate, for being practical, this approach hinges on an
ability to rebuild the raid1 incrementally. That is, the rebuild would have to start from what already is present on disk B (or C, when it is re-added). Starting from an effectively blank disk each time would be prohibitive.Even if this would work, I'd much prefer keeping the original raid1 intact and to only temporarily add another mirror: "lazy mirroring", to give the thing a name.
[...]
Quoted 4 lines
In the btfs device add case, you now have a three disk raid1 which is a whole different beast. Since this isn't n-way raid1, each disk is not stand alone. You're only assured data survives a one disk failure meaning you must have two drives.
Yes, I understand that. Unless someone convinces me that it's a bad idea, I keep wishing for a feature that allows to intermittently add a third disk to a two disk raid1 and update that disk so that it could replace one of the others.
Quoted 3 lines
So the btrfs replace scenario might work but it seems like a bad idea. And overall it's a use case for which send/receive was designed anyway so why not just use that?
Because it's not "just". Doing it right doesn't seem trivial. For one thing, there are multiple subvolumes; not at the top-level but nested inside a root subvolume. Each of them already has snapshots of its own. If there already is a send/receive script that can handle such a setup I'll happily have a look at it.
Michael
Signature · 3 lines
Michael Schuerig mailto:[email protected] http://www.schuerig.de/michael/
Quoted 63 lines · Michael Schuerig
Quoted 1 line · On Thursday 13 March 2014 16:04:33 Chris Murphy wrote:
On Mar 13, 2014, at 3:14 PM, Michael Schuerig
Quoted 47 lines · <[email protected]> wrote:
Quoted 47 lines · On Thursday 13 March 2014 14:48:55 Andrew Skretvedt wrote:
Quoted 21 lines · Hugo Mills
Quoted 20 lines · Michael Schuerig
Quoted 19 lines
My backup use case is different from the what has been recently discussed in another thread. I'm trying to guard against hardware failure and other causes of destruction.
I have a btrfs raid1 filesystem spread over two disks. I want to backup this filesystem regularly and efficiently to an external disk (same model as the ones in the raid) in such a way that
* when one disk in the raid fails, I can substitute the backup and
rebalancing from the surviving disk to the substitute only applies
the missing changes.* when the entire raid fails, I can re-build a new one from the backup.
The filesystem is mounted at its root and has several nested subvolumes and snapshots (in a .snapshots subdir on each subvol).
[...]
Quoted 6 lines
I'm new; btrfs noob; completely unqualified to write intelligently on
this topic, nevertheless:
I understand your setup to be btrfs RAID1 with/dev/A/dev/B, and a backup device someplace/dev/CQuoted 6 lines
Could you, at the time you wanted to backup the filesystem:
1) in the filesystem, break RAID1:/dev/A/dev/B<-- remove/dev/B
2) reestablish RAID1 to the backup device:/dev/A/dev/C<-- added
3) balance to effect the backup (i.e. rebuilding the RAID1 onto/dev/C) 4) break/reconnect the original devices: remove/dev/C; re-add/dev/Bto the fsI've thought of this but don't dare try it without approval from the experts. At any rate, for being practical, this approach hinges on an
ability to rebuild the raid1 incrementally. That is, the rebuild would have to start from what already is present on disk B (or C, when it is re-added). Starting from an effectively blank disk each time would be prohibitive.Even if this would work, I'd much prefer keeping the original raid1 intact and to only temporarily add another mirror: "lazy mirroring", to give the thing a name.
[...]
Quoted 4 lines
In the btfs device add case, you now have a three disk raid1 which is a whole different beast. Since this isn't n-way raid1, each disk is not stand alone. You're only assured data survives a one disk failure meaning you must have two drives.
Yes, I understand that. Unless someone convinces me that it's a bad idea, I keep wishing for a feature that allows to intermittently add a third disk to a two disk raid1 and update that disk so that it could replace one of the others.
Quoted 3 lines
So the btrfs replace scenario might work but it seems like a bad idea. And overall it's a use case for which send/receive was designed anyway so why not just use that?
Quoted 8 lines
Because it's not "just". Doing it right doesn't seem trivial. For one thing, there are multiple subvolumes; not at the top-level but nested inside a root subvolume. Each of them already has snapshots of its own. If there already is a send/receive script that can handle such a setup I'll happily have a look at it.
Michael
I think the closest thing there will ever be to this is n-way mirroring. I currently use rsync to a separate drive to maintain a backup copy, but it is not integrated into the array like n-way would be, and is definitely not a perfect solution. But a 3 drive 3-way would require the 3rd drive to be in the array the whole time or it would run into the same problem requiring a complete rebuild rather than an incremental when reintroduced, UNLESS such a feature was specifically included in the design, and even then, in a 3-way configuration, you would end up simplex on at least some data until the partial rebuild was completed. Personally, I will be DELIGHTED when n-way appears simply because basic 3-way gets us out of the dreaded simplex trap.
See comments at the bottom:
Quoted 70 lines · George Mitchell
Quoted 69 lines · Michael Schuerig
Quoted 1 line · On Thursday 13 March 2014 16:04:33 Chris Murphy wrote:
On Mar 13, 2014, at 3:14 PM, Michael Schuerig
Quoted 47 lines · <[email protected]> wrote:
Quoted 47 lines · On Thursday 13 March 2014 14:48:55 Andrew Skretvedt wrote:
Quoted 21 lines · Hugo Mills
Quoted 20 lines · Michael Schuerig
Quoted 19 lines
My backup use case is different from the what has been recently discussed in another thread. I'm trying to guard against hardware failure and other causes of destruction.
I have a btrfs raid1 filesystem spread over two disks. I want to backup this filesystem regularly and efficiently to an external disk (same model as the ones in the raid) in such a way that
* when one disk in the raid fails, I can substitute the backup and
rebalancing from the surviving disk to the substitute only applies
the missing changes.* when the entire raid fails, I can re-build a new one from the backup.
The filesystem is mounted at its root and has several nested subvolumes and snapshots (in a .snapshots subdir on each subvol).
[...]
Quoted 6 lines
I'm new; btrfs noob; completely unqualified to write intelligently on
this topic, nevertheless:
I understand your setup to be btrfs RAID1 with/dev/A/dev/B, and a backup device someplace/dev/CQuoted 6 lines
Could you, at the time you wanted to backup the filesystem:
1) in the filesystem, break RAID1:/dev/A/dev/B<-- remove/dev/B
2) reestablish RAID1 to the backup device:/dev/A/dev/C<-- added
3) balance to effect the backup (i.e. rebuilding the RAID1 onto/dev/C) 4) break/reconnect the original devices: remove/dev/C; re-add/dev/Bto the fsI've thought of this but don't dare try it without approval from the experts. At any rate, for being practical, this approach hinges on an
ability to rebuild the raid1 incrementally. That is, the rebuild would have to start from what already is present on disk B (or C, when it is re-added). Starting from an effectively blank disk each time would be prohibitive.Even if this would work, I'd much prefer keeping the original raid1 intact and to only temporarily add another mirror: "lazy mirroring", to give the thing a name.
[...]
Quoted 4 lines
In the btfs device add case, you now have a three disk raid1 which is a whole different beast. Since this isn't n-way raid1, each disk is not stand alone. You're only assured data survives a one disk failure meaning you must have two drives.
Yes, I understand that. Unless someone convinces me that it's a bad idea, I keep wishing for a feature that allows to intermittently add a third disk to a two disk raid1 and update that disk so that it could replace one of the others.
Quoted 3 lines
So the btrfs replace scenario might work but it seems like a bad idea. And overall it's a use case for which send/receive was designed anyway so why not just use that?
Because it's not "just". Doing it right doesn't seem trivial. For one thing, there are multiple subvolumes; not at the top-level but nested inside a root subvolume. Each of them already has snapshots of its own. If there already is a send/receive script that can handle such a setup I'll happily have a look at it.
Quoted 8 lines
Quoted 2 lines
Michael
I think the closest thing there will ever be to this is n-way mirroring. I currently use rsync to a separate drive to maintain a backup copy, but it is not integrated into the array like n-way would be, and is definitely not a perfect solution. But a 3 drive 3-way would require the 3rd drive to be in the array the whole time or it would run into the same problem requiring a complete rebuild rather than an incremental when reintroduced, UNLESS such a feature was specifically included in the design, and even then, in a 3-way configuration, you would end up simplex on at least some data until the partial rebuild was completed. Personally, I will be DELIGHTED when n-way appears simply because basic 3-way gets us out of the dreaded simplex trap.
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
I'm coming from ZFS land, am a BTRFS newbie, and I don't understand this discussion, at all. I'm assuming that BTRFS send/receive works similar to ZFS's similarly named feature. We use snapshots and ZFS send/receive to a remote server to do our backups. To do an rsync of our production file store takes days because there are so many files, while snapshotting and using ZFS send/receive takes tens of minutes at local (Gbit) speeds, and a few hours at WAN speeds, nearly all of that time being transfer time.
So just I don't get the "backup" problem. Place btrfs' equivalent of a pool on the external drive, and use send/receive of the filesystem or snapshot(s). Does BTRFS work so differently in this regard? If so, I'd like to know what's different.
My primary interest in BTRFS vs ZFS is two-fold:
1) ZFS has a couple of limitations that I find disappointing, that don't appear to be present in BTRFS.
A) Inability to upgrade a non-redundant ZFS pool/vdev to raidz or
increase the raidz (redundancy) level after creation. (Yes, you can plan around this, but I see no good reason to HAVE to)
B) Inability to remove a vdev once added to a pool.
2) Licensing: ZFS on Linux is truly great so far in all my testing, can't throw enough compliments their way, but I would really like to rely on a "first class citizen" as far as the Linux kernel is concerned.
-Ben
Quoted 2 lines · Lists
I'm assuming that BTRFS send/receive works similar to ZFS's similarly named feature.
Similar yes but not all options are the same between them. e.g. zfs send -R replicates all descendent file systems. I don't think zfs requires volumes, filesystems, or snapshots to be read-only, whereas btrfs send only works on read only snapshot-subvolumes. There has been some suggestion of a recursive snapshot creation and recursive send for btrfs.
Quoted 1 line
So just I don't get the "backup" problem. Place btrfs' equivalent of a pool on the external drive, and use send/receive of the filesystem or snapshot(s). Does BTRFS work so differently in this regard? If so, I'd like to know what's different.
Top most thing in zfs is the pool, which on btrfs is the volume. Neither zfs send or btrfs send works on this level to send everything within a pool/volume. zfs has the file system and btrfs has the subvolume which can be snapshot. Either (or both) can be used with send.
zfs also has the volume which is a block device that can be snapshot, there isn't yet a btrfs equivalent.
Btrfs and zfs have clones but the distinction is stronger with zfs. Like zfs snapshots can't be deleted unless its clones are deleted. Btrfs send has a -c clone-src option that I don't really understand, and also the --reflink which is a clone at the file level.
Anyway there are a lot of similarities but also quite a few differences. Basic functionality seems pretty much the same.
Quoted 8 lines
My primary interest in BTRFS vs ZFS is two-fold:
1) ZFS has a couple of limitations that I find disappointing, that don't appear to be present in BTRFS.
A) Inability to upgrade a non-redundant ZFS pool/vdev to raidz or increase the raidz (redundancy) level after creation. (Yes, you can plan around this, but I see no good reason to HAVE to)
B) Inability to remove a vdev once added to a pool.2) Licensing: ZFS on Linux is truly great so far in all my testing, can't throw enough compliments their way, but I would really like to rely on a "first class citizen" as far as the Linux kernel is concerned.
3. On btrfs you can delete a parent subvolume and the children remain. On zfs, you can't destroy a zfs filesystem/volume unless its snapshots are deleted, and you can't delete snapshots unless their clones are deleted.
Chris Murphy
Michael Schuerig posted on Thu, 13 Mar 2014 20:12:44 +0100 as excerpted:
Quoted 12 lines
My backup use case is different from the what has been recently discussed in another thread. I'm trying to guard against hardware failure and other causes of destruction.
I have a btrfs raid1 filesystem spread over two disks. I want to backup this filesystem regularly and efficiently to an external disk (same model as the ones in the raid) in such a way that
* when one disk in the raid fails, I can substitute the backup and rebalancing from the surviving disk to the substitute only applies the missing changes.
Quoted 6 lines
* when the entire raid fails, I can re-build a new one from the backup.
The filesystem is mounted at its root and has several nested subvolumes and snapshots (in a .snapshots subdir on each subvol).
Is it possible to do what I'm looking for?
AFAICS, as mentioned down the other subthread, the closest thing to this would be N-way mirroring, a coming feature on the roadmap for introduction after raid5/6 mode[1] gets completed. The current raid1 mode is 2-way-mirroring only, regardless of the number of devices.
N-way-mirroring is actually my most hotly anticipated feature for a different reason[2], but for you it would work like this:
1) Setup the 3-way (or 4-way if preferred) mirroring and balance to ensured copies of all data on all devices.
2) Optionally scrub to ensure the integrity of all copies.
3) Disconnect the backup device(s). (Don't btrfs device delete, this would remove the copy. Just disconnect.)
4) Store the backups.
5) Periodically get them out and reconnect.
6) Rebalance to update. (Since the devices remain members of the mirror, simply outdated, the balance should only update, not rewrite the entire thing.)
7) Optionally scrub to verify.
8) Repeat steps 3-7 as necessary.
If you went 4-way so two backups and alternated the one you plugged in, it'd also protect against mishap that might take out all devices during steps 5-7 when the backup is connected as well, since you'd still have that other backup available.
Unfortunately, completing raid5/6 support is still an ongoing project, and as a result, fully functional and /reasonably/ tested N-way-mirroring remains the same 6-months-minimum away that it has been for over a year now. But I sure am anticipating that day!
---
[1] Currently, the raid5/6 support is incomplete, the parity is calculated and writes are done, but some restore scenarios aren't yet properly supported and raid5/6-mode scrub isn't complete either, so the current code is considered testing-only, not for deployment where the raid5/6 feature would actually be relied on. That has remained the raid5/6 status for several kernels now, as the focus has been on bugfixing other areas including snapshot-aware defrag which is currently deactivated due to horrible scaling issues (current defrag COWS the operational mount only, duplicating previously shared blocks), send/ receive.
[2] In addition to loss of N-1 device-protection, I really love btrfs' data integrity features and the ability to recover from other copies if the one is found to be corrupted, which is why I'm running raid1 mode here. But currently, there's only the two copies and if both get corrupted... My sweet spot would be three copies, allowing corruption of two and recovery from the third, which is why I personally am so hotly anticipating N-way-mirroring, but unfortunately, it's looking a bit like the proverbial carrot on the stick in front of the donkey, these days.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 15 lines · On Friday 14 March 2014 06:42:27 Duncan wrote:
N-way-mirroring is actually my most hotly anticipated feature for a different reason[2], but for you it would work like this:
1) Setup the 3-way (or 4-way if preferred) mirroring and balance to ensured copies of all data on all devices.
2) Optionally scrub to ensure the integrity of all copies.
3) Disconnect the backup device(s). (Don't btrfs device delete, this would remove the copy. Just disconnect.)
4) Store the backups.
5) Periodically get them out and reconnect.
Quoted 7 lines
6) Rebalance to update. (Since the devices remain members of the mirror, simply outdated, the balance should only update, not rewrite the entire thing.)
7) Optionally scrub to verify.
8) Repeat steps 3-7 as necessary.
Judging from your description, N-way mirroring is (going to be) exactly what I was hoping for.
Michael
Signature · 3 lines
Michael Schuerig mailto:[email protected] http://www.schuerig.de/michael/
Michael Schuerig posted on Fri, 14 Mar 2014 09:56:20 +0100 as excerpted:
[Duncan posted...]
Quoted 2 lines
Quoted 2 lines
3) Disconnect the backup device(s). (Don't btrfs device delete, this would remove the copy. Just disconnect.)
Hmm... Looking back at what I wrote...
Presumably either have the filesystem unmounted for the disconnect (and ideally, the system off, tho with modern drives in theory that's not an issue, but still good if it can be done), or at least remounted read-only.
I had guessed that was implicit, but making it explicit is probably best all around, just in case. At least I can rest better with it, having made that explicit.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Actually, an interesting concept would be to have the initial two drive RAID 1 mirrored by 2 additional drives in 4-way configuration on a second machine at a remote location on a private high speed network with both machines up 24/7. In that case, if such a configuration would work, either machine could be obliterated and the data would survive fully intact in full duplex mode. It would just need to be remounted from the backup system and away it goes. Just thinking of interesting possibilities with n-way mirroring. Oh how I would love to have n-way mirroring to play with!
Quoted 16 lines · Duncan
Michael Schuerig posted on Fri, 14 Mar 2014 09:56:20 +0100 as excerpted:
[Duncan posted...]
Quoted 2 lines
Quoted 2 lines
3) Disconnect the backup device(s). (Don't btrfs device delete, this would remove the copy. Just disconnect.)
Hmm... Looking back at what I wrote...
Presumably either have the filesystem unmounted for the disconnect (and ideally, the system off, tho with modern drives in theory that's not an issue, but still good if it can be done), or at least remounted read-only.
I had guessed that was implicit, but making it explicit is probably best all around, just in case. At least I can rest better with it, having made that explicit.
George Mitchell posted on Fri, 14 Mar 2014 06:46:19 -0700 as excerpted:
Quoted 9 lines
Actually, an interesting concept would be to have the initial two drive RAID 1 mirrored by 2 additional drives in 4-way configuration on a second machine at a remote location on a private high speed network with both machines up 24/7. In that case, if such a configuration would work, either machine could be obliterated and the data would survive fully intact in full duplex mode. It would just need to be remounted from the backup system and away it goes. Just thinking of interesting possibilities with n-way mirroring. Oh how I would love to have n-way mirroring to play with!
In terms of raid1, mdraid already supports such a concept with its "write mostly" component device designation. A component device designated "write mostly" is never read from unless it becomes the only device available, so it's perfect for such an "over-the-net real-time-online- backup" solution.
The other half of the solution are the various block-device-over-network drivers such as BLK_DEV_NBD (see Documentation/blockdev/nbd.txt) for the client side, the server-side of which is in userspace. That lets you have what appears to be a block-device routed over the inet to that remote location.
Of course mdraid is lacking btrfs' data integrity features, etc, with its raid1 implementation entirely lacking any data integrity or real-time cross-checking at all, but unlike btrfs' N-way-mirroring it gets points for actually being available right now, so as they say, YMMV.
Of course the other notable issue with your idea is that while it DOES address the real-time remote redundancy issue, that doesn't (by itself) deal with fat-fingering or similar issues where real-time actually means the same problem's duplicated to the backup as well.
But btrfs snapshots address the fat-fingering issue and can be done on the partially-remote filesystem solution as well, and local or remote- local solutions (like periodic btrfs send to a separate local filesystem at both ends) can deal with the filesystem damage possibilities.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 9 lines · George Mitchell
Actually, an interesting concept would be to have the initial two drive RAID 1 mirrored by 2 additional drives in 4-way configuration on a second machine at a remote location on a private high speed network with both machines up 24/7. In that case, if such a configuration would work, either machine could be obliterated and the data would survive fully intact in full duplex mode. It would just need to be remounted from the backup system and away it goes. Just thinking of interesting possibilities with n-way mirroring. Oh how I would love to have n-way mirroring to play with!
That can already be done, albeit slightly differently by stacking btrfs RAID 1 on top of a pair of DRBD devices. Of course, this doesn't provide quite the same degree of safety as your suggestion, but it does work (and DRBD makes the remote copy write-mostly for the local system automatically).
Quoted 3 lines · On Thursday 13 March 2014 17:29:11 George Mitchell wrote:
I currently use rsync to a separate drive to maintain a backup copy, but it is not integrated into the array like n-way would be, and is definitely not a perfect solution.
Could you explain how you're using rsync? I was just about to copy a btrfs filesystem to another disk. That filesystem has several subvolumes and about 100 snapshots overall. Owing to COW, this amounts to about 1.2TB. However, I reckon that rsync doesn't know anything about COW and accordingly would blow up my data immensely on the destination disk.
How do I copy a btrfs filesystem preserving its complete contents? How do I update such a copy?
Yes, I want to keep the subvolume layout of the original and I want to copy all snapshots. I don't think send/receive is the answer, but it's likey I don't understand it well enough. I'm concerned, that a send/receive-based approach is not robust against mishaps.
Consider: I want to incrementally back-up a filesystem to two external disks. For this I'd have to for each subvolume keep a snapshot corresponding to its state on the backup disk. If I make any mistake in managing these snapshots, I can't update the external backup anymore.
Also, I don't understand whether send/receive would allow me to copy/update a subvolume *including* its snapshots.
Things have become a little more complicated than I had hoped for, but I've only been using btrfs for a couple of weeks.
Michael
Signature · 3 lines
Michael Schuerig mailto:[email protected] http://www.schuerig.de/michael/
Quoted 11 lines · Michael Schuerig
Quoted 3 lines · On Thursday 13 March 2014 17:29:11 George Mitchell wrote:
I currently use rsync to a separate drive to maintain a backup copy, but it is not integrated into the array like n-way would be, and is definitely not a perfect solution.
Could you explain how you're using rsync? I was just about to copy a btrfs filesystem to another disk. That filesystem has several subvolumes and about 100 snapshots overall. Owing to COW, this amounts to about 1.2TB. However, I reckon that rsync doesn't know anything about COW and accordingly would blow up my data immensely on the destination disk.
Quoted 7 lines
How do I copy a btrfs filesystem preserving its complete contents? How do I update such a copy?
Yes, I want to keep the subvolume layout of the original and I want to copy all snapshots. I don't think send/receive is the answer, but it's likey I don't understand it well enough. I'm concerned, that a send/receive-based approach is not robust against mishaps.
send/receive is the answer, but it's going to be a bit more
complicated to manage *all* of the snapshots. (Questions -- do you actually need them all backed up? Can you instead do incremental backups of the "main" subvol and keep each of those independently on the backup machine instead?)
Quoted 4 lines
Consider: I want to incrementally back-up a filesystem to two external disks. For this I'd have to for each subvolume keep a snapshot corresponding to its state on the backup disk. If I make any mistake in managing these snapshots, I can't update the external backup anymore.
Correct (I got bitten by this last week with my fledgling backup
process). You need a place that stores the "current state" subvolumes that's not going to be touched by anything else, and you can't clean up any given base until you're certain that there's a good new one available on both sides. One thing that helps here is that send requires the snapshot being sent to be marked read-only, so it's not possible to change it at all -- but you can delete them.
Quoted 2 lines
Also, I don't understand whether send/receive would allow me to copy/update a subvolume *including* its snapshots.
Snapshots aren't owned by subvolumes. Once you've made a snapshot,
that snapshot is a fully equal partner of the subvol that it was a snapshot of -- there is no hierarchy of ownership. This means that you will have to send each snapshot independently.
What send allows you to do is to specify that one or more
subvolumes on the send side can be assumed to exist on the receive side (via -p and -c). If you do that, the stream can then use them as clone sources (i.e. should make shared CoW copies from them, rather than sending all the data).
Hugo.
Signature · 4 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- ... one ping(1) to rule them all, and in the ---
darkness bind(2) them. Michael, I am currently using rsync INSTEAD of btrfs backup tools. I really don't see anyway that it could be compatible with the backup features of btrfs. As I noted in my post, it is definitely not a perfect solution, but it is doing the job for me. What I REALLY want in this regard is n-way mirroring to get me out of the simplex trap completely. At that point, I can have more confidence in btrfs snapshop capability.
Quoted 31 lines · Michael Schuerig
Quoted 3 lines · On Thursday 13 March 2014 17:29:11 George Mitchell wrote:
I currently use rsync to a separate drive to maintain a backup copy, but it is not integrated into the array like n-way would be, and is definitely not a perfect solution.
Could you explain how you're using rsync? I was just about to copy a btrfs filesystem to another disk. That filesystem has several subvolumes and about 100 snapshots overall. Owing to COW, this amounts to about 1.2TB. However, I reckon that rsync doesn't know anything about COW and accordingly would blow up my data immensely on the destination disk.
How do I copy a btrfs filesystem preserving its complete contents? How do I update such a copy?
Yes, I want to keep the subvolume layout of the original and I want to copy all snapshots. I don't think send/receive is the answer, but it's likey I don't understand it well enough. I'm concerned, that a send/receive-based approach is not robust against mishaps.
Consider: I want to incrementally back-up a filesystem to two external disks. For this I'd have to for each subvolume keep a snapshot corresponding to its state on the backup disk. If I make any mistake in managing these snapshots, I can't update the external backup anymore.
Also, I don't understand whether send/receive would allow me to copy/update a subvolume *including* its snapshots.
Things have become a little more complicated than I had hoped for, but I've only been using btrfs for a couple of weeks.
Michael