Storage and snapshots as historical yearly
I am trying to implement a system that stores the data in a unit (A) with BTRFS format that is untouchable and that future files and folders created or modified are stored in another physical unit (B) with BTRFS format. Each year the new files will be moved to store A and start over.
The idea is that a duplicate of disk A can be made to keep it in a safe place and that the files stored there can not be modified until the mixture of (A) and
(B) is made.
I have looked for information on this but I do not see it very clear. Both "SEND" and "RECEIVE" do the opposite case to what interests me.
I have also seen that you could try to get a RAID 0 but I am afraid that the data in A will not remain intact if the system performs a "BALANCE" at some point and mixes data between (A) and (B).
It is assumed that both the (A) and (B) data will be displayed in the same structure transparently, for example in "/ home".
-------------------------------
| HOME | | ------------ ------------ | | | | | | | | | | | | | | | DISK (A) | | DISK (B) | | | | BTRFS | | BTRFS | | | | | | | | | ------------ ------------ |
-------------------------------
Quoted 8 lines · Senén Vidal Blanco
I am trying to implement a system that stores the data in a unit (A) with BTRFS format that is untouchable and that future files and folders created or modified are stored in another physical unit (B) with BTRFS format. Each year the new files will be moved to store A and start over.
The idea is that a duplicate of disk A can be made to keep it in a safe place and that the files stored there can not be modified until the mixture of (A) and
(B) is made.
Before I get into anything further, I would like to comment that this is a very odd use case. Yearly granularity doesn't make sense for backups unless you generate very little data throughout the year (otherwise backups will take forever) and can afford to lose multiple months of that data.
The timescale you're talking about combined with the requirement that files not be modifiable on A except during the times when you sync changes indicates you will probably be much better served by proper off-line archival storage than some online configuration as you appear to be trying to create.
Quoted 3 lines
I have looked for information on this but I do not see it very clear. Both "SEND" and "RECEIVE" do the opposite case to what interests me.
If you can afford to operate at a shorter timescale (even monthly), then BTRFS snapshots plus send and receive probably are one of the best options. Perhaps you could explain your understanding of send and receive and why you think it won't work, and I (and possibly other people on the list as well) could confirm whether or not you understand correctly.
Quoted 17 lines
I have also seen that you could try to get a RAID 0 but I am afraid that the data in A will not remain intact if the system performs a "BALANCE" at some point and mixes data between (A) and (B).
It is assumed that both the (A) and (B) data will be displayed in the same structure transparently, for example in "/ home".
-------------------------------
| HOME | | ------------ ------------ | | | | | | | | | | | | | | | DISK (A) | | DISK (B) | | | | BTRFS | | BTRFS | | | | | | | | | ------------ ------------ |
Quoted 1 line
-------------------------------
What you are describing here is called an overlay or union mount. Source A would be your lower directory, and B would be your upper directory. Unmodified data is passed through from the lower directory until a file is changed. When changes are made to the overlay mount, they are reflected on the upper directory. Special files called 'whiteout' files are used in the upper directory to represent deleted items.
Unfortunately, I don't know of any overlay mount implementation that works correctly and reliably with BTRFS. I know for a fact that OverlayFS (the upstream in-kernel implementation) does not work, and I believe that AUFS3 and UnionFS (the third-party options that are used by most LiveCD's) don't work either. UnionFS-FUSE (a userspace implementation completely unrelated to UnionFS) might work, but I've never tested it and it will likely have performance issues because it's implemented in userspace. As far as I know, whiteout support is the primary missing piece here, but I may be mistaken.
Alternatively, this could be done with a seed device. The concept is pretty similar to an overlay mount, but it operates at a lower level, and it's a BTRFS specific feature. Unfortunately, it's not well documented, and I'm not confident that I could explain how to do it correctly.
Quoted 10 lines · Austin S. Hemmelgarn
Unfortunately, I don't know of any overlay mount implementation that works correctly and reliably with BTRFS. I know for a fact that OverlayFS (the upstream in-kernel implementation) does not work, and I believe that AUFS3 and UnionFS (the third-party options that are used by most LiveCD's) don't work either. UnionFS-FUSE (a userspace implementation completely unrelated to UnionFS) might work, but I've never tested it and it will likely have performance issues because it's implemented in userspace. As far as I know, whiteout support is the primary missing piece here, but I may be mistaken.
Diverting away from the original topic, what issues with overlayfs and btrfs?
I'm using btrfs to create 'base' operating system containers (btrfs) and then using overlayfs for a few 'upper' containers for specific applications, so the upper parts of the overlays contain only the config and data files and I can apply OS updates only on the lower ones.
I do note that changes in the 'base' os can take time to propagate to the upper containers and I'm probably not being sensible in not stopping the upper containers when updating the lower ones. This is also does not seem to be what overlaysfs is intended for. However, for my light usage it generally works OK and is useful to me.
Pete
Quoted 13 lines · Pete
Quoted 10 lines · Austin S. Hemmelgarn
Unfortunately, I don't know of any overlay mount implementation that works correctly and reliably with BTRFS. I know for a fact that OverlayFS (the upstream in-kernel implementation) does not work, and I believe that AUFS3 and UnionFS (the third-party options that are used by most LiveCD's) don't work either. UnionFS-FUSE (a userspace implementation completely unrelated to UnionFS) might work, but I've never tested it and it will likely have performance issues because it's implemented in userspace. As far as I know, whiteout support is the primary missing piece here, but I may be mistaken.
Quoted 2 lines
Diverting away from the original topic, what issues with overlayfs and btrfs?
As mentioned, I thought whiteout support was missing, but if you're using it without issue, I might be wrong.
Quoted 11 lines
I'm using btrfs to create 'base' operating system containers (btrfs) and then using overlayfs for a few 'upper' containers for specific applications, so the upper parts of the overlays contain only the config and data files and I can apply OS updates only on the lower ones.
I do note that changes in the 'base' os can take time to propagate to the upper containers and I'm probably not being sensible in not stopping the upper containers when updating the lower ones. This is also does not seem to be what overlaysfs is intended for. However, for my light usage it generally works OK and is useful to me.
Actually, this is pretty well in-line with one of the intended use cases (it was mostly designed for efficient multiple instantiation of Docker or LXC containers). The other big use case is 'live' systems that only retain state while powered on, like most install images.
Quoted 4 lines · Austin S. Hemmelgarn
Quoted 2 lines
Diverting away from the original topic, what issues with overlayfs and btrfs?
As mentioned, I thought whiteout support was missing, but if you're using it without issue, I might be wrong.
Whiteout works fine. Upper and lower layers and working directory are all on btrfs subvolumes. Snapshotting seems fine.
Quoted 10 lines
Quoted 10 lines
I'm using btrfs to create 'base' operating system containers (btrfs) and then using overlayfs for a few 'upper' containers for specific applications, so the upper parts of the overlays contain only the config and data files and I can apply OS updates only on the lower ones.
I do note that changes in the 'base' os can take time to propagate to the upper containers and I'm probably not being sensible in not stopping the upper containers when updating the lower ones. This is also does not seem to be what overlaysfs is intended for. However, for my light usage it generally works OK and is useful to me.
Quoted 4 lines
Actually, this is pretty well in-line with one of the intended use cases (it was mostly designed for efficient multiple instantiation of Docker or LXC containers). The other big use case is 'live' systems that only retain state while powered on, like most install images.
OK, I only spotted the latter use case when reading up, apart from one website which seemed to mention using it for containers.
Quoted 9 lines · Pete
Quoted 4 lines · Austin S. Hemmelgarn
Quoted 2 lines
Diverting away from the original topic, what issues with overlayfs and btrfs?
As mentioned, I thought whiteout support was missing, but if you're using it without issue, I might be wrong.
Whiteout works fine. Upper and lower layers and working directory are all on btrfs subvolumes. Snapshotting seems fine.
Hmm, just double checked myself. Apparently I was operating based on old information.
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
I'm using btrfs to create 'base' operating system containers (btrfs) and then using overlayfs for a few 'upper' containers for specific applications, so the upper parts of the overlays contain only the config and data files and I can apply OS updates only on the lower ones.
I do note that changes in the 'base' os can take time to propagate to the upper containers and I'm probably not being sensible in not stopping the upper containers when updating the lower ones. This is also does not seem to be what overlaysfs is intended for. However, for my light usage it generally works OK and is useful to me.
Quoted 7 lines
Quoted 4 lines
Actually, this is pretty well in-line with one of the intended use cases (it was mostly designed for efficient multiple instantiation of Docker or LXC containers). The other big use case is 'live' systems that only retain state while powered on, like most install images.
OK, I only spotted the latter use case when reading up, apart from one website which seemed to mention using it for containers.
Yeah, containers are the other big one. It's not unusual for someone to need to spin up a dozen or more instances of essentially the same container (think large build systems), and without an overlay mount, you end up multiplying the space for the base image times the number of containers.
Hi Austin,
Thank you very much for your answer.
I comment a little on your suggestions in the context.
Quoted 16 lines · El lunes, 11 de septiembre de 2017 14:49:05 (CEST) Austin S. Hemmelgarn…
Quoted 8 lines · Senén Vidal Blanco
I am trying to implement a system that stores the data in a unit (A) with BTRFS format that is untouchable and that future files and folders created or modified are stored in another physical unit (B) with BTRFS format. Each year the new files will be moved to store A and start over.
The idea is that a duplicate of disk A can be made to keep it in a safe place and that the files stored there can not be modified until the mixture of (A) and (B) is made.
Before I get into anything further, I would like to comment that this is a very odd use case. Yearly granularity doesn't make sense for backups unless you generate very little data throughout the year (otherwise backups will take forever) and can afford to lose multiple months of that data.
Quoted 5 lines
The timescale you're talking about combined with the requirement that files not be modifiable on A except during the times when you sync changes indicates you will probably be much better served by proper off-line archival storage than some online configuration as you appear to be trying to create.
Totally agree with you, although it really is not the objective, for that I use Bareos as backup system. It is simply because I need to have stored almost 2 Terabytes and be able to have a copy of the data outside the enclosure and 3 years of duplicate disks and be able to access almost instantaneously the old data.
With a backup system it would take several days to restore the entire system or several hours to rebuild the file system and restore the file you need. While with an image from last year you would only have to restore a year at best.
Quoted 10 lines
Quoted 3 lines
I have looked for information on this but I do not see it very clear. Both "SEND" and "RECEIVE" do the opposite case to what interests me.
If you can afford to operate at a shorter timescale (even monthly), then BTRFS snapshots plus send and receive probably are one of the best options. Perhaps you could explain your understanding of send and receive and why you think it won't work, and I (and possibly other people on the list as well) could confirm whether or not you understand correctly.
That is another long-term goal that I would like to implement, since I have seen that it can be combined with external storage and able to move the data through snapshots. It uses little bandwidth and seems to be more immediate than the Bareos, although it would leave both systems for security, since I speak of very sensitive and important data to lose them.
Quoted 39 lines
Quoted 20 lines
I have also seen that you could try to get a RAID 0 but I am afraid that the data in A will not remain intact if the system performs a "BALANCE" at some point and mixes data between (A) and (B).
It is assumed that both the (A) and (B) data will be displayed in the same structure transparently, for example in "/ home".
-------------------------------
| HOME | | | ------------ ------------ | | | | DISK (A) | | DISK (B) | | | | | | BTRFS | | BTRFS | | | | ------------ ------------ |Quoted 1 line
-------------------------------
What you are describing here is called an overlay or union mount. Source A would be your lower directory, and B would be your upper directory. Unmodified data is passed through from the lower directory until a file is changed. When changes are made to the overlay mount, they are reflected on the upper directory. Special files called 'whiteout' files are used in the upper directory to represent deleted items.
Unfortunately, I don't know of any overlay mount implementation that works correctly and reliably with BTRFS. I know for a fact that OverlayFS (the upstream in-kernel implementation) does not work, and I believe that AUFS3 and UnionFS (the third-party options that are used by most LiveCD's) don't work either. UnionFS-FUSE (a userspace implementation completely unrelated to UnionFS) might work, but I've never tested it and it will likely have performance issues because it's implemented in userspace. As far as I know, whiteout support is the primary missing piece here, but I may be mistaken.
Quoted 5 lines
Alternatively, this could be done with a seed device. The concept is pretty similar to an overlay mount, but it operates at a lower level, and it's a BTRFS specific feature. Unfortunately, it's not well documented, and I'm not confident that I could explain how to do it correctly.
This part does seem to me very interesting, since one of your colleagues has commented on the topic of using "Seed" with BTRFS, which I am seeing its operation; and I can see how this would be OverlayFS, since I think the other options are not very generalized.
Thank you for the explanation and I will keep you informed if I can get something clean, because if documenting my experience I can contribute something and I will be satisfied.
Greetings.
Signature · 6 lines
Senén Vidal Blanco - SGISoft S.L. Tlf.: 986413322 - 660923711 GPG ID 466431A8AF01F99A http://www.sgisoft.com/ --
11.09.2017 21:17, Senén Vidal Blanco пишет:
Quoted 9 lines
I am trying to implement a system that stores the data in a unit (A) with BTRFS format that is untouchable and that future files and folders created or modified are stored in another physical unit (B) with BTRFS format. Each year the new files will be moved to store A and start over.
The idea is that a duplicate of disk A can be made to keep it in a safe place and that the files stored there can not be modified until the mixture of (A) and
(B) is made.
This can probably be achieved using seed device. Mark original device as seed and all changes will go to another writable device, similar to overlay; then remove seed bit from original device, "btrfs device remove writable" device and it should relocate its content back. Rinse and repeat.
Perfect!! Just what I was looking for.
Sorry for the delay, because before doing so, I preferred to test to see if it actually worked.
I have a doubt. The system works perfectly, but at the time of deleting the writing disk and merging the data on the read-only disk I fail to understand the process.
I have tried to remove the seed bit on disk A and delete the write B as you mention, and so move the data to A, but tells me that disk B does not exist. These are the orders I have made:
md127-> A
md126-> B
btrfstune -S 0 /dev /md127
mount /dev/md127 /mnt (I mount this disk since the md126 gives error) btrfs device delete /dev/md126 /mnt
ERROR: error removing device '/dev/md126': No such file or directory
Another thing I've tried is to remove disk B without removing the seed bit, but it gives me the error:
ERROR: error removing device '/dev/md126': unable to remove the only writeable device.
Any ideas about it?
Thank you very much for the reply.
Greetings.
Quoted 14 lines · El martes, 12 de septiembre de 2017 6:34:15 (CEST) Andrei Borzenkov escribió:
11.09.2017 21:17, Senén Vidal Blanco пишет:
Quoted 8 lines
I am trying to implement a system that stores the data in a unit (A) with BTRFS format that is untouchable and that future files and folders created or modified are stored in another physical unit (B) with BTRFS format. Each year the new files will be moved to store A and start over.
The idea is that a duplicate of disk A can be made to keep it in a safe place and that the files stored there can not be modified until the mixture of (A) and (B) is made.
This can probably be achieved using seed device. Mark original device as seed and all changes will go to another writable device, similar to overlay; then remove seed bit from original device, "btrfs device remove writable" device and it should relocate its content back. Rinse and repeat.
Signature · 6 lines
Senén Vidal Blanco - SGISoft S.L. Tlf.: 986413322 - 660923711 GPG ID 466431A8AF01F99A http://www.sgisoft.com/ --
19.09.2017 14:49, Senén Vidal Blanco пишет:
Quoted 20 lines
Perfect!! Just what I was looking for.
Sorry for the delay, because before doing so, I preferred to test to see if it actually worked.I have a doubt. The system works perfectly, but at the time of deleting the writing disk and merging the data on the read-only disk I fail to understand the process.
I have tried to remove the seed bit on disk A and delete the write B as you mention, and so move the data to A, but tells me that disk B does not exist. These are the orders I have made:
md127-> A
md126-> Bbtrfstune -S 0 /dev /md127
mount/dev/md127/mnt (I mount this disk since the md126 gives error) btrfs device delete/dev/md126/mnt
ERROR: error removing device '/dev/md126': No such file or directory
Quoted 7 lines
Another thing I've tried is to remove disk B without removing the seed bit, but it gives me the error:
ERROR: error removing device '
/dev/md126': unable to remove the only writeable device.Any ideas about it?
Yes, sorry about it. Clearing seed flag on device invalidates filesystem. What you can do, is to rotate devices. I.e. remove /dev/md126, set seed flag on md127 and add md126 back.
I actually tested it and it works for me.
Quoted 19 lines
Thank you very much for the reply.
Greetings.Quoted 14 lines · El martes, 12 de septiembre de 2017 6:34:15 (CEST) Andrei Borzenkov escribió:
11.09.2017 21:17, Senén Vidal Blanco пишет:
Quoted 8 lines
I am trying to implement a system that stores the data in a unit (A) with BTRFS format that is untouchable and that future files and folders created or modified are stored in another physical unit (B) with BTRFS format. Each year the new files will be moved to store A and start over.
The idea is that a duplicate of disk A can be made to keep it in a safe place and that the files stored there can not be modified until the mixture of (A) and (B) is made.
This can probably be achieved using seed device. Mark original device as seed and all changes will go to another writable device, similar to overlay; then remove seed bit from original device, "btrfs device remove writable" device and it should relocate its content back. Rinse and repeat.
Quoted 27 lines · Andrei Borzenkov
19.09.2017 14:49, Senén Vidal Blanco пишет:
Quoted 26 lines
Perfect!! Just what I was looking for.
Sorry for the delay, because before doing so, I preferred to test to see if it actually worked.I have a doubt. The system works perfectly, but at the time of deleting the writing disk and merging the data on the read-only disk I fail to understand the process.
I have tried to remove the seed bit on disk A and delete the write B as you mention, and so move the data to A, but tells me that disk B does not exist. These are the orders I have made:
md127-> A
md126-> Bbtrfstune -S 0 /dev /md127
mount/dev/md127/mnt (I mount this disk since the md126 gives error) btrfs device delete/dev/md126/mnt
ERROR: error removing device '/dev/md126': No such file or directoryAnother thing I've tried is to remove disk B without removing the seed bit, but it gives me the error:
ERROR: error removing device '
/dev/md126': unable to remove the only writeable device.
Quoted 7 lines
Quoted 1 line
Any ideas about it?
Yes, sorry about it. Clearing seed flag on device invalidates filesystem. What you can do, is to rotate devices. I.e. remove
/dev/md126, set seed flag on md127 and add md126 back.I actually tested it and it works for me.
Same. FWIW, this (like many other things in BTRFS) could stand to be better documented, especially since it's not likely to be intuitive for most people.
Quoted 22 lines
Quoted 20 lines
Thank you very much for the reply.
Greetings.Quoted 14 lines · El martes, 12 de septiembre de 2017 6:34:15 (CEST) Andrei Borzenkov escribió:
11.09.2017 21:17, Senén Vidal Blanco пишет:
Quoted 8 lines
I am trying to implement a system that stores the data in a unit (A) with BTRFS format that is untouchable and that future files and folders created or modified are stored in another physical unit (B) with BTRFS format. Each year the new files will be moved to store A and start over.
The idea is that a duplicate of disk A can be made to keep it in a safe place and that the files stored there can not be modified until the mixture of (A) and (B) is made.
This can probably be achieved using seed device. Mark original device as seed and all changes will go to another writable device, similar to overlay; then remove seed bit from original device, "btrfs device remove writable" device and it should relocate its content back. Rinse and repeat.
Quoted 30 lines · El martes, 19 de septiembre de 2017 21:33:31 (CEST) Andrei Borzenkov escribió:
19.09.2017 14:49, Senén Vidal Blanco пишет:
Quoted 29 lines
Perfect!! Just what I was looking for.
Sorry for the delay, because before doing so, I preferred to test to see if it actually worked.I have a doubt. The system works perfectly, but at the time of deleting the
writing disk and merging the data on the read-only disk I fail to understand the process.I have tried to remove the seed bit on disk A and delete the write B as you
mention, and so move the data to A, but tells me that disk B does not exist. These are the orders I have made:md127-> A
md126-> Bbtrfstune -S 0 /dev /md127
mount/dev/md127/mnt (I mount this disk since the md126 gives error) btrfs device delete/dev/md126/mnt
ERROR: error removing device '/dev/md126': No such file or directoryAnother thing I've tried is to remove disk B without removing the seed bit,
but it gives me the error:ERROR: error removing device '
/dev/md126': unable to remove the only writeable device.
Quoted 8 lines
Quoted 1 line
Any ideas about it?
Yes, sorry about it. Clearing seed flag on device invalidates filesystem. What you can do, is to rotate devices. I.e. remove
/dev/md126, set seed flag on md127 and add md126 back.I actually tested it and it works for me.
OK thanks
Now I see how it works :))
With the commands:
mount /dev/md126 /mnt
btrfs device remove /dev/md127 /mnt
We remove the read-only array (A) from the BTRFS system and in doing so pass all the information from (A) to (B) read-write to mix them.
From what I see is not bad since both (A) and (B) are still operational. (A) with last year and (B) with everything current.
Finally with this other commands:
btrfstune -S 1 /dev/md126
mount /dev/md126 /mnt
btrfs device add -f /dev/md127 /mnt
we activate the seed bit in md126 (B) and add the (A) in read-write mode, where the new files will be archived and (B) as store until the following year and (A) do clean to fill in it new data.
I have tried to rotate twice to see if it goes well and smoothly.
Just comment that I see two small problems to this:
1. The transfer of data from (A) to (B) when removing the read-only disk takes quite a while and more the more it has stored in the history. It would be nice if the process were reversed, since in (B) there are fewer "data" stored. Also, I could not use it monthly or daily for this reason.
2. My idea was to have a larger A-disk than B where I would save the historical ones, because so in B I could put a smaller disk and something faster. If the decoupling process outside read-write rather than read-only and passed the data to A would be ideal for this case.
On the other hand, as an anecdote only, and perhaps for lack of experience or knowledge, I have used the entire linux system in BTRFS (@ and @home) format and a single partition md126 to have the system bootable and running simply by attaching the disk to the computer in degraded mode (swap outside the raid , which I'm not so bad: P). This has made that by rotating disks A and B I have had some problems with grub and fstab at boot, which I had to overcome by making changes to the boot configurations and some more botches.
I'm going to see a couple more things and if there's any way I can combine this with snapshots and see if the bulb will light up. If I do not get it I will try with the other filesystems that you have suggested to me. Although honestly, I like BTRFS more than the other alternatives, I already use BTRFS on 5 computers and it goes very well.
Greetings.
Quoted 4 lines
Quoted 4 lines
Thank you very much for the reply.
Greetings.El martes, 12 de septiembre de 2017 6:34:15 (CEST) Andrei Borzenkov
Quoted 17 lines · escribió:
Quoted 17 lines
Quoted 17 lines
11.09.2017 21:17, Senén Vidal Blanco пишет:
Quoted 10 lines
I am trying to implement a system that stores the data in a unit (A) with
BTRFS format that is untouchable and that future files and folders created
or modified are stored in another physical unit (B) with BTRFS format. Each year the new files will be moved to store A and start over.The idea is that a duplicate of disk A can be made to keep it in a safe place and that the files stored there can not be modified until the mixture of (A) and (B) is made.
This can probably be achieved using seed device. Mark original device as seed and all changes will go to another writable device, similar to overlay; then remove seed bit from original device, "btrfs device remove writable" device and it should relocate its content back. Rinse and repeat.
Signature · 6 lines
Senén Vidal Blanco - SGISoft S.L. Tlf.: 986413322 - 660923711 GPG ID 466431A8AF01F99A http://www.sgisoft.com/ --