Snapshots – noob questions
I have been trying to understand how snapshots work (in BRTFS and in general), but I still have some questions, and would appreciate if someone could clear them for me. To make things easier, I tried to make most of them Yes/No questions:
1. When creating the filesystem, I only made btrfs partition with no subvolumes. Do I have to use subvolumes, or can I snapshot entire “/“?
2. Can I take a snapshot of a working / root filesystem?
3. If I make a snapshot of / and there are some separate partitions mounted under /mnt/ or /home/, will snapshot skip them?
4. If I have a snapshot of /, can I completely erase this partition and later restore it in full form that snapshot, or do snapshots work only if a limited number of files has been changed? If the former, then does it mean that snapshot size will be comparable to the original data size?
5. Can a snapshot be stored on a different partition? And if so, does that partition have to be BTRFS too? I ask, because I would also want to store snapshots in a cloud storage.
Thanks.
Quoted 8 lines · KC
I have been trying to understand how snapshots work (in BRTFS and in general), but I still have some questions, and would appreciate if someone could clear them for me.
To make things easier, I tried to make most of them Yes/No questions:1. When creating the filesystem, I only made btrfs partition with no subvolumes. Do I have to use subvolumes, or can I snapshot entire “/“?
/ (the top level of the FS) is a subvolume as well -- it's the only
one that can't be renamed or deleted, though.
Quoted 1 line
2. Can I take a snapshot of a working / root filesystem?
Yes.
Quoted 2 lines
3. If I make a snapshot of / and there are some separate partitions mounted under
/mnt/or/home/, will snapshot skip them?
Yes. Snapshots work on a single subvolume at a time.
Quoted 2 lines
4. If I have a snapshot of /, can I completely erase this partition and later restore it in full form that snapshot,
If you have your root filesystem as the top level of the btrfs
filesystem, then no, you can't. I would recommend using the filesystem structure shown here
https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Managing_snapshots
because it makes making, managing and rolling back the snapshots much easier.
Quoted 2 lines
or do snapshots
work only if a limited number of files has been changed?
Snapshots will continue to work regardless of how many files have
been changed. The only thing to note is that as you make more modifications (either to the snapshot or the original, or both), the amount of storage requires goes up. Note that once you've made a snapshot, there's no distinction between the snapshot and the original -- they're completely equal partners. This means that you can delete either one and the other one will remain.
Quoted 2 lines
If the former, then does it mean that snapshot size will be comparable to the original data size?
When you first make a snapshot, the total storage space required isunchanged(*). As you make modifications, the two subvolumes will diverge from each other, and more and more space is needed. Eventually, when you've modified every file (on one side or the other), you will end up needing enough space to store each one separately. Basically, they share as many data extents as possible, and use separate ones only when needed.
(*) OK, that's a lie -- you need an extra tree root for the snapshot, but that's a whole 4k-16k, depending on the size of your tree nodes.
Quoted 3 lines
5. Can a snapshot be stored on a different partition? And if so, does that partition have to be BTRFS too? I ask, because I would also want to store snapshots in a cloud storage.
No, they can't.
Hugo.
Signature · 5 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- For months now, we have been making triumphant retreats ---
before a demoralised enemy who is advancing
in utter disorder. Well... you can't /create/ snapshots on different partitions / cloud storage / whatever, but you can certainly /send/ them there once you've created them.
Generally the best way to do this kind of thing is with incremental replication of snapshots to another btrfs filesystem, which may be local or cloud.
you@box1:~$ sudo btrfs send /.snapshots/snapshot1 | ssh you@box2 sudo btrfs receive /.snapshots
This sends the snapshot to the remote machine, where it will end up in /.snapshots. This is a FULL send - you'll be sending a MB of data for every MB stored on disk locally. But the next time:
you@box1:~$ sudo btrfs send -p /.snapshots/snapshot1 /.snapshots/snapshot2 | ssh you@box2 sudo btrfs receive /.snapshots
Assuming you still have a copy of snapshot1 on box2, the next time, you can do an INCREMENTAL send as shown above. This tells btrfs send to use snapshot1 as a "parent" - which means it only sends the blocks in snapshot2 that are different from the blocks in snapshot1. For example, this might be a filesystem with 500GB of data - but if you only changed 10MB of that data between snapshot1 and snapshot2, you only have to send that 10MB of data to box2 with this incremental send.
Now you have snapshot1 and snapshot2 on both the local and the remote machine - and, best part, they're deduplicated, meaning that even though each snapshot is a fully browseable copy of your (example) 500GB of data, they only require (example) 500.01GB of storage - the 500GB of data they share in common, plus the 10MB of stuff that's different between snapshot1 and snapshot2.
Quoted 4 lines · Hugo Mills
Quoted 3 lines
5. Can a snapshot be stored on a different partition? And if so, does that partition have to be BTRFS too? I ask, because I would also want to store snapshots in a cloud storage.
No, they can't.
Quoted 6 lines · Hugo Mills
Quoted 5 lines · KC
I have been trying to understand how snapshots work (in BRTFS and in general), but I still have some questions, and would appreciate if someone could clear them for me.
To make things easier, I tried to make most of them Yes/No questions:
Quoted 6 lines
Quoted 3 lines
1. When creating the filesystem, I only made btrfs partition with no subvolumes. Do I have to use subvolumes, or can I snapshot entire “/“?
/ (the top level of the FS) is a subvolume as well -- it's the only
one that can't be renamed or deleted, though.
I'll point out a source of confusion I've had, and see new users have the same confusion quite often also. Within the file system hierarchy, / is a mount point. It could be that subvolume ID 5 is mounted there, which is how opensuse's default rootfs on Btrfs works. However, Fedora's / is a subvolume named root, the top level ID 5 subvolume isn't mounted by default.
So / is not necessarily the top level of the file system. I'm not sure what to call the undeleteable ID 5 subvolume. It's the default subvolume, but if the user changes the default subvolume then it isn't. Calling it the default default is maybe more confusing. And calling it subvolume ID 5 is a bit confusing because subvolume ID 0 maps to 5, and many users are more familiar with subvol ID 0 than ID 5. And then if I have a subvolume within a subvolume mounted and do a btrfs list, it shows top level ID with a value other than 5.
Anyway I really don't know what else to call the undeletable first subvolume.
Chris Murphy
Quoted 13 lines · Chris Murphy
Quoted 10 lines · Hugo Mills
Quoted 8 lines · KC
I have been trying to understand how snapshots work (in BRTFS and in general), but I still have some questions, and would appreciate if someone could clear them for me.
To make things easier, I tried to make most of them Yes/No questions:1. When creating the filesystem, I only made btrfs partition with no subvolumes. Do I have to use subvolumes, or can I snapshot entire “/“?
Quoted 5 lines
Quoted 2 lines
/ (the top level of the FS) is a subvolume as well -- it's the only
one that can't be renamed or deleted, though.I'll point out a source of confusion I've had, and see new users have the same confusion quite often also. Within the file system hierarchy, / is a mount point. It could be that subvolume ID 5 is mounted there, which is how opensuse's default rootfs on Btrfs works. However, Fedora's / is a subvolume named root, the top level ID 5 subvolume isn't mounted by default.
I agree the terminology gets confusing. I try to be consistent with
the set of words used in the SysadminGuide and (at least some of) the rest of the website:
Subvolume id=5: "top level of the FS"
The top of the current visible UNIX filesystem structure: "/" Root's home directory: "~root"
I always try to avoid using the word "root" for any of them, as
it's rather heavily overloaded at this point.
Quoted 3 lines
So / is not necessarily the top level of the file system. I'm not sure what to call the undeleteable ID 5 subvolume. It's the default subvolume, but if the user changes the default subvolume then it isn't. Calling it the default default is maybe more confusing. And calling it subvolume ID 5 is a bit confusing because subvolume ID 0 maps to 5, and many users are more familiar with subvol ID 0 than ID 5. And then if I have a subvolume within a subvolume mounted and do a btrfs list, it shows top level ID with a value other than 5.
Anyway I really don't know what else to call the undeletable first subvolume.
I call it "the top level of the btrfs filesystem", as that's what
it is. (Even if you don't always mount the top level of the FS).
Hugo.
Signature · 4 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- I always felt that as a C programmer, I ---
was becoming typecast. Quoted 2 lines · KC
3. If I make a snapshot of / and there are some separate partitions mounted under
/mnt/or/home/, will snapshot skip them?
A snapshot of a subvolume only snapshots that subvolume, not other subvolumes contained in it or other partitions/file systems.
Quoted 3 lines
4. If I have a snapshot of /, can I completely erase this partition and later restore it in full form that snapshot, or do snapshots work only if a limited number of files has been changed? If the former, then does it mean that snapshot size will be comparable to the original data size?
I'm confused by what you mean by completely erase this partitions and later restore it. What do you mean by partition? Taken literally I'd say no because a snapshot is a subvolume, it's a function of a Btrfs file system, and if you erase the partition you've erased the file system and everything on it.
Otherwise, either the snapshot or its parent subvolume can be deleted at any time. At the moment the snapshot is taken, they are clones, but they behave as completely separate file systems. Any changes to one do not affect the other.
Quoted 1 line
5. Can a snapshot be stored on a different partition?
Yes Btrfs send/receive.
Quoted 1 line
And if so, does that partition have to be BTRFS too?
You can send a snapshot with -f <filename> to a non-Btrfs file system. But it's a file. You can't interact with it, you can only restore it using btrfs receive, to another Btrfs file system.
Quoted 1 line
I ask, because I would also want to store snapshots in a cloud storage.
Yes although a possible drawback is that I think the resulting snapshot as a file, doesn't contain checksums. So silent data corruption detection isn't possible. Yes you can and should shasum the resulting file but if the sums don't match you don't necessarily know what files are affected.
Chris Murphy
Quoted 8 lines · Chris Murphy
Quoted 2 lines · KC
4. If I have a snapshot of /, can I completely erase this partition and later restore it in full form that snapshot, or do snapshots work only if a limited number of files has been changed? If the former, then does it mean that snapshot size will be comparable to the original data size?
I'm confused by what you mean by completely erase this partitions and later restore it. What do you mean by partition? Taken literally I'd say no because a snapshot is a subvolume, it's a function of a Btrfs file system, and if you erase the partition you've erased the file system and everything on it.
Otherwise, either the snapshot or its parent subvolume can be deleted at any time. At the moment the snapshot is taken, they are clones, but they behave as completely separate file systems. Any changes to one do not affect the other.
Probably slightly more accurate to say "completely different file
trees", as they're part of the same filesystem (the single btrfs FS that contains all of the subvolumes in this example).
Quoted 3 lines
Quoted 1 line
5. Can a snapshot be stored on a different partition?
Yes Btrfs send/receive.
Mmmm... debatable semantics again. You can transfer the data with
btrfs send/receive, but after that it's not really a snapshot in the sense of being a CoW copy on the same filesystem...
Hugo.
Signature · 4 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- I always felt that as a C programmer, I ---
was becoming typecast. Quoted 6 lines · Hugo Mills
Quoted 5 lines · Chris Murphy
Quoted 2 lines · KC
4. If I have a snapshot of /, can I completely erase this partition and later restore it in full form that snapshot, or do snapshots work only if a limited number of files has been changed? If the former, then does it mean that snapshot size will be comparable to the original data size?
Quoted 7 lines
Quoted 3 lines
I'm confused by what you mean by completely erase this partitions and later restore it. What do you mean by partition? Taken literally I'd say no because a snapshot is a subvolume, it's a function of a Btrfs file system, and if you erase the partition you've erased the file system and everything on it.
Otherwise, either the snapshot or its parent subvolume can be deleted at any time. At the moment the snapshot is taken, they are clones, but they behave as completely separate file systems. Any changes to one do not affect the other.
Probably slightly more accurate to say "completely different file
trees", as they're part of the same filesystem (the single btrfs FS that contains all of the subvolumes in this example).
Yes good point. File system = volume; file tree = subvolume.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
5. Can a snapshot be stored on a different partition?
Yes Btrfs send/receive.
Mmmm... debatable semantics again. You can transfer the data with
btrfs send/receive, but after that it's not really a snapshot in the sense of being a CoW copy on the same filesystem…
Ahh yes I was being very literal rather than thinking of a magical hard link that can work across file system boundaries.
Maybe the question needs to be phrased in terms of a use case question, and possibly a seed-device would fit?
https://btrfs.wiki.kernel.org/index.php/Seed-device
Chris Murphy
Quoted 6 lines · Chris Murphy
Quoted 2 lines · KC
3. If I make a snapshot of / and there are some separate partitions mounted under
/mnt/or/home/, will snapshot skip them?A snapshot of a subvolume only snapshots that subvolume, not other subvolumes contained in it or other partitions/file systems.
Thanks. That's a critical detail to know.
A few follow-on questions... Are taking snapshots:
"atomic"?
Do they hold up or lock out other disk activity?
Is there an overhead to keeping many snapshots other than the storage any changed files take?
Can we now consider making and deleting snapshots a "debugged stable feature"?
Thanks,
Martin
Quoted 8 lines · Martin
Quoted 6 lines · Chris Murphy
Quoted 2 lines · KC
3. If I make a snapshot of / and there are some separate partitions mounted under
/mnt/or/home/, will snapshot skip them?A snapshot of a subvolume only snapshots that subvolume, not other subvolumes contained in it or other partitions/file systems.
Quoted 7 lines
Thanks. That's a critical detail to know.
A few follow-on questions... Are taking snapshots:
"atomic"?
Yes. The state of the snapshot is that of a single point in the
filesystem's history.
Quoted 1 line
Do they hold up or lock out other disk activity?
Yes, but only very briefly -- no more than any other simple
filesystem operation.
Quoted 2 lines
Is there an overhead to keeping many snapshots other than the storage any changed files take?
The storage the changed metadata takes, but that's all.
Quoted 2 lines
Can we now consider making and deleting snapshots a "debugged stable feature"?
I think so, yes, although there are still cases where having lots
of snapshots (thousands and upwards) can cause problems. I'm thinking specifically of deletion of snapshots, where the system can bog down quite heavily doing all the deletions.
Hugo.
Signature · 3 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- Great oxymorons of the world, no. 10: Business Ethics --- Hugo Mills posted on Tue, 28 Jan 2014 19:16:16 +0000 as excerpted:
Quoted 7 lines
Quoted 2 lines
Can we now consider making and deleting snapshots a "debugged stable feature"?
I think so, yes, although there are still cases where having lots of snapshots (thousands and upwards) can cause problems. I'm thinking specifically of deletion of snapshots, where the system can bog down quite heavily doing all the deletions.
What about the recent pathologic large (half-gig-plus) frequently- internally-rewritten and thus without NOCOW highly fragmented file case, made pathologic (even on SSD, it's a CPU not an IO bottleneck) by thousands of snapshots?
To my knowledge that's speculatively pinned down to the not all that long ago snapshot-aware-defrag code (which itself was in response to the problem of defrag not being snapshot aware before that), but while I'm not a dev and thus don't/can't-easily follow developing patches in code- level detail (tho I routinely read both intros and followups), I'm not aware of pending patches that address that yet.
Altho recommended procedure is to NOCOW such files at creation (generally by setting NOCOW on the directory they're to be placed in, before the file is written at all), based on reports here, quite a few people get caught in the trap of not knowing about that until it's too late, and then are stuck with a file that it can literally take *DAYS* to do anything with due to the pathelogic lack of btrfs ability to scale or make pretty much any progress at all in attempting to move/defrag/delete such files (altho I believe they can be renamed in-place on the same snapshot).
That said, in general I'd agree, snapshots can practically be considered /reasonably/ debugged-stable, as long as you're doing reasonable snapshot management and don't get thousands (and preferably not too many hundreds) of them to deal with on the same filesystem.
An automated script that say takes a snapshot a minute, but keeps only say 30 one-minute snapshots, keeping a snapshot every half hour for a day (47, 30+47=77), keeping a snapshot every hour for another two days (48, total three days covered, 77+48=125), keeping a snapshot every 6 hours to fill out the week ((7-3)*4=16, 125+16=141), keeping a snapshot a day to fill out the month (31-7=24, 142+24=165), keeping a snapshot a week to fill out the year (52-4=48, 166+48=213), and finally keeping a snapshot a quarter (13 weeks) out to say a decade (40-4=36, 213+36=249), if the filesystem remains in use that long... should be reasonable, and assuming the math is correct, that's under 250 snapshots total, with coverage of a snapshot a minute out to a half hour, a snapshot a half-hour out to a day, a snapshot a day out to a month, a snapshot a week out to a year, and a snapshot a quarter out to a decade (!!). If you need history back more than a decade and you don't have some sort of longer-term storage archiving involved... well let's just say I seriously doubt the quality of your decisions in that case.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 4 lines · KC
Quoted 3 lines · Chris Murphy
Quoted 1 line · KC
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
3. If I make a snapshot of / and there are some separate partitions mounted under
/mnt/or/home/, will snapshot skip them?A snapshot of a subvolume only snapshots that subvolume, not other subvolumes contained in it or other partitions/file systems.
Excellent, thank you.
If currently my /home is just a folder on the / subvolume, is there any way to promote existing /home to a subvolume status so that if I take a snapshot of /, the /home gets skipped?
Multiple ways. I think the easiest is for the subvolumes to be at the top level of the Btrfs volume, and mount them where you want with fstab. So for example on Fedora 20 my top level subvolumes are:
boot
root
home
root is mounted at / using mount option subvol=root boot is mounted at /boot using mount option subvol=boot home is mounted at /home using mount option subvol=home
I've also decided to put systemd-journals on a separate subvolume:
journald is mounted at /var/log/journal using mount option subvol=journald
That way I can snapshot root, and rollback but have a persistent journal rather than a snapshot specific journal. Of course /home is also persistent because I don't roll it back.
You could also rename /home, create a subvolume home within current subvolume root, then move the files from directory /home_old to the subvolume /home. Now when you snapshot rootfs, /home isn't snapshot. In this case you don't need to separately mount the home subvolume because it's available as if it were a directory. It's an organizational thing I don't know that there's a right or wrong here.
One thing that came up recently on the Fedora devel list though, with respect to roofs snapshots available in the mounted fs hierarchy, is that an old rootfs snapshot could contain vulnerabilities. So two ways to handle it are to keep snapshots in the top level of the fs outside of the mounted fs hierarchy (only mount it as needed when taking snapshots); and/or mount that snapshots subvolume with noexec or nosuid mount option. That way its executables aren't root executable.
Chris Murphy
A message that is not in the archive (sent off-list or not yet received): [email protected]
readded list
Quoted 7 lines · KC
Quoted 2 lines · Chris Murphy
You could also rename /home, create a subvolume home within current subvolume root, then move the files from directory /home_old to the subvolume /home. Now when you snapshot rootfs, /home isn't snapshot. In this case you don't need to separately mount the home subvolume because it's available as if it were a directory. It's an organizational thing I don't know that there's a right or wrong here.
So I did
btrfs subvolume create /home
Quoted 6 lines
and rsynced all the files from old home into it. The system boots and works fine. Now I get:
$ sudo btrfs subvolume list . ID 351 gen 25880 top level 5 path homeDoes this mean that /home is now subvolume of root?
Yes, but more correctly it means it's in the top level of the file system. This also means your layout is mounting the top level of the file system, rather than some other subvolume. I find this layout confusing now that I'm used to a different layout: by default Fedora creates boot [1], root and home subvolumes, which then in fstab are mounted using subvol=. Therefore the top level of the file system is not mounted.
Naturally I found this quite confusing when I first encountered it, so it's understandable why other distros do things differently. It's perhaps easier to see the fstab, sub list, and the kernel boot param.
UUID=xxxxxx / btrfs subvol=root,compress=lzo,ssd 0 0 UUID=xxxxxx /boot btrfs subvol=boot,compress=lzo,ssd 0 0 UUID=yy /boot/efi vfat umask=0077,shortname=winnt 0 0 UUID=xxxxxx /home btrfs subvol=home,compress=lzo,ssd 0 0 UUID=xxxxxx /var/log/journal btrfs subvol=journald,compress=lzo,ssd 0 0
# btrfs sub list / ID 256 gen 505 top level 5 path home ID 258 gen 510 top level 5 path root ID 259 gen 511 top level 5 path journald ID 263 gen 507 top level 5 path boot ID 264 gen 192 top level 5 path boot.clean ID 265 gen 193 top level 5 path root.clean ID 266 gen 194 top level 5 path home.clean ID 269 gen 189 top level 5 path boot.1 ID 270 gen 190 top level 5 path root.1 ID 271 gen 191 top level 5 path home.1 ID 272 gen 192 top level 5 path boot.clean_ro ID 273 gen 193 top level 5 path root.clean_ro ID 274 gen 194 top level 5 path home.clean_ro
linuxefi /boot/vmlinuz-3.12.8-300.fc20.x86_64 root=UUID=xxxxxx ro rootflags=subvol=root vconsole.font=latarcyrheb-sun16 elevator=noop 3
The boot param "rootflags=subvol=root" is the same as fstab's mount option subvol=root. This tells the kernel that the root file system isn't merely at UUID=xxxxxx, but it's within subvolume root.
Quoted 1 line
And how do I take a snapshot of root, if it is not listed in this command?
btrfs sub snaps -r / root.1
At the top level of the file system is a subvolume, ID 5. You can't rename it, and you can't delete it. If the top level subvolume is mounted, then you are snapshotting it by referring to the mount point, in this case /. And in this example root.1 will be a read-only snapshot of /.
I think snapper on opensuse puts snapshots into a subvolume called .snapshots. You could do the same thing, and then with a file listing of / you won't see a bunch of accumulated snapshots. They could just as well go in a directory called .snapshots, off hand I'm not thinking of an advantage of putting snapshots into a subvolume.
Chris Murphy
Quoted 1 line · I wrote:
They could just as well go in a directory called .snapshots, off hand I'm not thinking of an advantage of putting snapshots into a subvolume.
There isn't an advantage with a top level 5 subvolume being used for rootfs. But if it's not being mounted, rather other named subvolumes are, then the advantage of a subvolume for snapshots is that it can be independently mounted to provide access to all snapshots within from which to create derivative snapshots. And it can also have a nosuid or noexec mount option to prevent old vulnerabilities from being root executable.
Chris Murphy