subvols and parents - how?

Hey.

I'd have a, mainly administrative, question.

When I use subvolumes than these have always a parent subvolume (except ID5), so I can basically decide between two ways:

a) make child subvolumes, e.g.
5
|

+-root   (=subvol, mountpoint /)
  +-boot/
  +-root/
  +-lib/
  +-home/ (=subvolume)

and soon on... perhaps the whole thing without the dedicated root- subovlume (although that's probably not so smart, I guess).

b) place at least some of the subvolumes directly below the top-level and mount them e.g. via /etc/fstab, e.g.
5
|

+-root   (=subvol, mountpoint /)
| +-boot/
| +-root/
| +-lib/
+-home/ (=subvolume, mountpoint /home)

Now I wondered whether this has any technical implications, but neither the wiki, nor the manpages seem to explain a lot here.

The "differences", AFAIU, are the follows:
- When I mount a given subvolume,.. it's childs are automatically "there".
  Whereas when I don't have them as childs (as in (b)) I must of course mount them somehow manually.
- Analogously for umounting.
- I can move existing subvols to higher/lower levels, and the parent IDs will change accordingly.

So basically it makes no difference, right? Or is there anything more technical going on? E.g. with the ref-links or so?
Right now, there are, AFAIK, neither recursive snapshots (and especially not atomic ones) nor recursive send/receive, right? If that should ever be implemented, would I perhaps have problems with
(a) or (b)?

Thanks,
Chris.

btw, for a developer:

$ btrfs subvolume list /mnt/ -pacguqt
ID      gen     cgen    parent  top level       parent_uuid     uuid    path    
--      ---     ----    ------  ---------       -----------     ----    ----    
257     16      8       5       5               -       64f8a75b-5cb5-6e4d-9f30-d45fe3d9e060    <FS_TREE>/root

There seem to be some colum mis-aglinment issues in the output =) btrfsprogs 4.3

Christoph Anton Mitterer posted on Tue, 24 Nov 2015 05:56:00 +0100 as excerpted:

Very astute question! =:^)

Somewhere on the wiki I believe there's a recommendation to use (b) layout, but to some extent it depends on why you're actually doing subvolumes.

OK, found it on the wiki. It wasn't under use-cases, where I initially thought to look, but under sysadmin guide. Specifically, see section 4.2, managing snapshots, but I'd suggest reading the entire subvolumes discussion, section 4, or even most/all of the page.

https://btrfs.wiki.kernel.org/index.php/SysadminGuide

(More below.)

If you're doing subvolumes for snapshotting and potential rollback purposes, layout (b) can be preferable as it allows a more direct mix and match rollback.

Suppose you only want to rollback /, because some update screwed you up, but not /home, which is fine. If /home is a nested subvolume, then you're now mounting the nested home subvolume from some other nesting tree entirely, whereas if they're all under top-level, you simply mount the /home subvolume under whatever snapshot of / you are currently booting.

Of course the reverse applies as well, if / is fine but you want to rollback /home. Again, with nesting you're reaching into some other nesting to mount what you want, and it can get a bit unintuitive and difficult to track, particularly if you go more than the two levels deep, but if all the snapshots are direct children of the top level ID 5, it's a lot easier.

Tho I'd actually suggest a variant of the flat layout they suggest in the sysadmin's guide. What I'd do is something like this (using your tree drawing style):

5
|

+-roots (dir not subvol, note the s, rootS, plural)
| +-root (subvol, mountpoint /)
| | +-boot/ (dir)
| | +-root/ (dir)
| | +-lib/  (dir)
| | +-home/ (empty dir as mountpoint)
| +-root-snapshot-2015.0301 (dated snapshot of root)
| +-root-snapshot-2015.0601 (dated snapshot of root)
| +-root-snapshot-2015.0901 (dated snapshot of root)
+-homes (dir not subvol)
| +-home (subvol, mountpoint /home)
| +-home-snapshot-2015.0301 (dated snapshot of home)

...

Of course, you might also organize by date instead of subvol...

5
|

+- heads (dir, headS plural)
| +-root (subvol)
| +-home (subvol)
| +-whatever (subvol)
+-snapshots-2015.0301 (dir, snapshotS plural)
| +-root-2015.0301 (snapshot of heads/root)
| +-home-2015.0301 (snapshot of heads/home)
| ...
+-snapshots-2015.0601 (dir)
| +-root-2015.0601 (snapshot)
| ...
+-snapshots-2015.0901 (dir)
| +-root-2015.0901 (snapshot)

...

Either of these would make finding a desired snapshot to rollback to much easier than a pure flat subvols/snapshots layout, with the preferred one depending on whether you want subvols/snapshots grouped by date or by snapshotted mountpoint.

The dates organization would make cleaning up old snapshots by date, and visually checking that the snapshot cleanup script (if automated) is working as intended, somewhat easier, however.

Meanwhile, if the intention for a subvolume is simply to exclude that subtree from snapshotting of the parent, as might be the case for example if you have a VMs subvol, with the VM image files set NOCOW to avoid fragmentation, since snapshotting nocow files forces cow1 (a cow at the first write of that block, before returning to nocow, because a snapshot locks the existing extents in place for the snapshot, so initial writes to a block after a snapshot /can't be nocow or it'd change the snapshot too...), and it's not going to be snapshotted on perhaps a different schedule than the parent...

... then it can make more sense to use a nested subvolume, as the primary and perhaps only reason for the subvolume at all is to prevent the subtree from being included in the parent snapshot, and as such, it's unlikely to be separately mounted.

OTOH, if there's a chance you might want to mount that subvolume in a roll-back scenario, under the snapshot you're rolling back to, then it makes sense to put it directly under ID 5 again, and mount it in any case.

Then there's the security angle to consider. With the (basically, possibly modified as I suggested) flat layout, mounting something doesn't automatically give people in-tree access to nested subvolumes (subject to normal file permissions, of course), like nested layout does. And with (possibly modified) flat layout, the whole subvolume tree doesn't need to be mounted all the time either, only when you're actually working with subvolumes.

In general, I believe the nested layout is easier to setup initially, because you don't have to worry about actually mounting the subvolumes to make them accessible. But flat (or modified flat, as I suggest) is easier for continued administration, particularly if you're doing snapshots and potentially rollbacks, because while doing the initial mount setup is more work, once the mount layout is established, mixing and matching various heads with snapshots of varying ages is a relatively simple matter of pointing each mount at the appropriate snapshot instead of head. Additionally, (possibly modified) flat layout is easier to do subvolume/snapshot management on, because they're all in the same place, instead of scattered at random places within the tree. In my quarterly snapshots example, for instance, suppose 2016 rolls around and you decide you don't need 2014's snapshots any longer. Which layout is going to make it easiest to delete them all, while conversely, making it least likely to fat-finger things and delete the wrong ones?

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Well I've had read that, but it's pretty vague and especially doesn't mentioned any of the filesystem internal implications (if there are any)... like I wondered before, whether this has effects on ref-links not being used when e.g. sending/recieving ... or on future planned extensions like recursive snapshots.

That's a bit unclear to me,... I thought when I make a snapshot, any nested subvols wouldn't be snapshotted and thus be empty dirs. So I'd have rather that if I would simply have no /home (if I didn't move it to the rolled-back subvol manually)

That's more what I've had in mind...
Actually something like this:
5
|

+-root       (=subvol)
| +-boot
| +-home     (subvo=/home being mounted heron)
| +-lib
+-home       (subvol, the current version)
+-snapshots  (=dir)
  +-root
  | +-2015-01-14 (subvol, snapshot)
  | +-2015-09-30 (subvol, snapshot)
  +-home
    +-2015-06-04 (subvol, snapshot)
    +-2015-09-01 (subvol, snapshot)

And it once more points to the problem of the wiki... anyone can write (I think even I) and it's totally unclear at the first glance how "[non-]official" and outdated something may be.
Apart from the problem that many important questions (from the PoV of an more advanced admin that doesn't just do mkfs.btrfs and then never again thinks about it) :-(

that is in fact also use case.. so in practise I'll probably have a mix of (a) and (b).

Ah that's good to know how that works (or better said, that it works at all)... I've already wondered myself several times what happens when I snapshot NOCOW files, ... something that's I guess also missing from the (missing ;-) ) btrfs-end-user overview and details documentation.

Yes.

Uhm, I don't get the big security advantage here... whether nested or manually mounted to a subdir,... if the permissions are insecure I'll have a problem... if they're secure, than not.

Of course I use insecure permissions and don't mount the subvols then I'd be safe in flat setup, but not in a nested setup...(where they subvol is "auto-mounted")...

But that seems pretty awkward.

Mhh I think my main question turns out to be whether the different layouts would have any technical (i.e. not administrative) effects... like the aforementioned stuff of recursive snapshots (should such thing ever come to life).
But at least from the userspace tools it seems that I can move subvols arbitrarily and they adapt their parent IDs accordingly...

So I guess whatever setup one uses nested/flat/mixed... doesn't rule out any functionalities for the future?!

thx,
Chris.

No, there's no particular implications one way or the other in
terms of reflinks. Obviously, if recursive snapshots get implemented, it'll remove some of the current awkwardness with nested subvols, but it won't invalidate any existing setups that use the recommendations in the Sysadmin Guide.

Correct. This is actually a use-case for nested subvols, and one
which snapper uses -- the target for snapshots of /foo is /foo/.snapshots/$date, where /foo/.snapshots is a subvol in its own right. So, if you have a subdir which you won't want to include in snapshots of a subvol, make it a subvol itself.

Yup, this is one of the main reasons for not nesting subvols.

In practice, new content is checked by a number of people when it's
put in, so the case of someone putting random poorly-thought-out crap in the wiki isn't particularly likely to happen.

Please feel free to add the things you'd like to see. As I said
above, we do check the docs on the wiki as they're changed, so if you're wrong on some details, it won't be a major issue for very long. If you want to discuss details before you write something, start a conversation -- either on here, or on IRC (or even on the Talk pages of the wiki).

Pretty much, yes.

Note that the "parent" of send -p and of snapshots is not the same
relationship as the "parent" (containing subvol) of the tree structure. This is an awkward nomenclature problem, and I'm not sure how to fix it.

The first meaning is fixed between a subvol and its snapshot when
the snapshot is created, and won't change, regardless of how the snapshots are renamed/moved.

The second meaning... actually, that doesn't change either, because
you can't rename a subvol across another subvol boundary. (I only just realised this, halfway through the explanation).

They're still different things, though. :)

Correct, it doesn't.

Hugo.

Hugo Mills             | Anyone who claims their cryptographic protocol is
hugo@... carfax.org.uk | secure is either a genius or a fool. Given the
http://carfax.org.uk/  | genius/fool ratio for our species, the odds aren't
PGP: E2AB1DE4          | good.                                  Bruce Schneier

Well... it may work in 99% cases... but there could something slip through, which isn't as easy the case in manpages, which also tend to be less messy than the huge pile of wiki pages where similar/related things are described on different pages.

Imagine a case, a non-experienced user update the wiki saying that -- repair should be used, he may not even doing it in bad faith, perhaps he had success with it and now writes a recipe.
It may take a while until someone of the more experienced guys notices that and corrects it.
But if ", in the meantime had some fs corruptions,... I may experience already severe problems by following that suggestion... (and while I do have many backups of all my data, others may not, and if their life's data is concerned, they'd be screwed).

So even if it takes you just a few hours to correct such rubbish, you know that Murphy's law may still hit n people during that time ;-)

Well I can write a list together of things which I think should be part of some more general documentation (i.e. less documentation about options of the tools)... questions a complete newcomer to btrfs may have who needs however more than "just a filesystem".

Yeah, that was clear... :-)
Maybe call the "parent" from send -p "base" or something like that... IMHO that would fit more as the parent there is more like a "fundament".

Anyway, it's still not as bad as the usage of "RAID1" ;-)

That's not quite clear to me... I had subvols like that:
/top/root/below-root
/top/below-top
and was able to move that to:
/top/root/below-top
/top/below-root

i.e. not just changing names but swapping as in:
mv /top/root/below-top /top/tmp
mv /top/below-root /top/root/below-root
mv /top/tmp /top/below-top

with top, root, below-top and below-root all being the same subvols

Thanks a lot for your explanations :)

Chris.

You can get update notifications, with the diffs for each page, via
RSS. (At least, I do, and I think David and a few others monitor it in the same way). The window of failure is fairly small, particularly in view of the number of such "dangerous" changes made. The total vulnerability is measured in hours per year...

Yes, that makes sense.

With a recent mv, it'll be doing doing reflink copies followed by
delete for all of the contents, which makes it pretty efficient, but I think you'll find that the UUID of the subvols changes. (At least, I hope it does. If it doesn't, then my mental model of what the FS is doing is *really* screwed up).

Hugo.

Hugo Mills             | Never underestimate the bandwidth of a Volvo filled
hugo@... carfax.org.uk | with backup tapes.
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

Feel free to shamelessly use my idea (as well as the one to call btrfs' RAID1 replica2 or something else)
:-O

root@heisenberg:/mnt# mv --version
mv (GNU coreutils) 8.23

=> not recent enough...

Well... see below:

root@heisenberg:~# truncate  -s 2G image
root@heisenberg:~# losetup -f image 
root@heisenberg:~# mkfs.btrfs /dev/loop0 
btrfs-progs v4.3
See http://btrfs.wiki.kernel.org for more information.

Performing full device TRIM (2.00GiB) ... Label:              (null)
UUID:               10e1a55c-448a-4f37-ae5c-6a7801a7f202 Node size:          16384
Sector size:        4096
Filesystem size:    2.00GiB
Block group profiles:
  Data:             single            8.00MiB Metadata:         DUP             110.38MiB System:           DUP              12.00MiB SSD detected:       no
Incompat features:  extref, skinny-metadata Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     2.00GiB  /dev/loop0

root@heisenberg:~# mount /dev/loop0 /mnt/
root@heisenberg:/mnt# btrfs subvolume create root
Create subvolume './root'
root@heisenberg:/mnt# btrfs subvolume create below-top
Create subvolume './below-top'
root@heisenberg:/mnt# cd root/
root@heisenberg:/mnt/root# btrfs subvolume create below-root
Create subvolume './below-root'
root@heisenberg:/mnt# btrfs subvolume list /mnt/ -pacguqt
ID	gen	cgen	parent	top level	parent_uuid	uuid	path	
--	---	----	------	---------	-----------	----	----	
257	9	7	5	5		-	8fbf521e-77f9-0d49-9891-87767f98c655	root
258	8	8	5	5		-	b49131e9-4207-aa42-8195-c50de5f06136	below-top
259	9	9	257	257		-	20c042be-ead8-204a-a684-94c1a770e739	<FS_TREE>/root/below-root

root@heisenberg:/mnt# mv root/below-root/ tmp
root@heisenberg:/mnt# mv below-top/ root/
root@heisenberg:/mnt# mv tmp/ below-root
root@heisenberg:/mnt# btrfs subvolume list /mnt/ -pacguqt
ID	gen	cgen	parent	top level	parent_uuid	uuid	path	
--	---	----	------	---------	-----------	----	----	
257	9	7	5	5		-	8fbf521e-77f9-0d49-9891-87767f98c655	root
258	8	8	257	257		-	b49131e9-4207-aa42-8195-c50de5f06136	<FS_TREE>/root/below-top
259	9	9	5	5		-	20c042be-ead8-204a-a684-94c1a770e739	below-root
root@heisenberg:/mnt# 

So the UUIDs seem to stay the same (or are these other UUIDs?)

Hope I haven't ruined your day now ;-)

Cheers,
Chris.

Christoph Anton Mitterer posted on Tue, 24 Nov 2015 22:25:50 +0100 as excerpted:

What I was intending to convey but apparently failed to be quite clear enough, suppose:

5
|

+-+ subvols (dir)
  |
  +-+ root (subvol)
  | |
  | + home (nested subvol)
  |
  +-+ snaps-2015.0901 (dir)
    |
    +-+ root-2015.0901 (subvol)

As long as you're on the working /, then /home is a nested subvol, and you don't have to mount it to access, tho you can if you want.

But now, you roll back to snaps-2015.0901/root-2015.0901.

It won't have /home nested underneath, as you correctly pointed out, but in ordered to access it, you now MUST mount /home, which...

#1 could be a pain to setup if you weren't actually mounting it previously, just relying on the nested tree, AND...

#2 The point I was trying to make, now, to mount it you'll mount not a native nested subvol, and not a directly available sibling 5/subvols/home, but you'll actually be reaching into an entirely different nesting structure to grab something down inside, mounting 5/subvols/root/home subvolume nesting down inside the direct 5/subvols/root sibling subvol.

With just one level of nesting and one additional mount, it's not too hard to keep track of, but if you're dealing with four or five levels of subvol nesting and some of them you're mounting the working head copy while others you're rolling back, it could get difficult to keep straight in your head what's going on.

Consider a layout like this:

5

+-+ subvols (dir)
  |
  +-+ root (subvol)
  | |
  | +-+ home (subvol)
  | | |
  | | +-+ henry (dir, no subvol for henry)
  | | |
  | | +-+ fred (subvol)
  | | | |
  | | | +-+ vms (subvol)
  | | | 
  | | +-+ betty (subvol)
  | |
  | +-+ svr (subvol)
  |   |
  |   +-+ vms (subvol)
  |
  +-+ snaps-2015.0901 (dir)
    |
    +-+ root-2015.0901 (subvol here and below)
    |
    +-+ home-2015.0901
    |
    +-+ fred-2015.0901
    |
    +-+ fred-vms-2015.0901
    |
    +-+ betty-2015.0901
    |
    +-+ svr-2015.0901
    |
    +-+ svr-vms-2015.0901

Now, you were hacked and they encrypted a bunch of stuff, but you were lucky and caught them before they got everything. You need to roll back root but not home, fred is fine, but his vms subvol needs rolled back, betty needs rolled back, svr needs rolled back, but svr's vms are fine.

Try to sort THAT out along with the nesting, and keep it all straight while under the severe pressure of trying to recover from a hack in time for those svr things to go live for Black Friday in a few hours, where in a single day you expect to make as much as you normally do in a month, the rest of the year! The pressure is on!

Oh, and you weren't actually doing the mounts as you were depending on the nested tree, so you have to actually setup the mounts as well, not just switch them to point to the appropriate location.

OK, so that's a bit contrived, but server encryption for ransom hacks are in the news, black Friday starts in a few hours here, and I think the point should be obvious! =:^)

(Some years ago, before btrfs, I had something similar setup but with partitions. Disaster struck and I ended up with / from one backup, /usr from another, and /var, with the package database of what was installed on the other two, from current, or something like that. Needless to say I learned quite some lessons from that, one of which was that everything that the package manager installs should be on the same partition with the installed-package database, so if it has to be restored from backup, at least if it's all old, at least it's all equally old, and the package database actually matches what's on the system because it's in the same backup! That partition and btrfs, along with each of its various backups, are now 8 GiB each, so it's not like I'll run out of room with several levels of backup. I went mdraid after that, but after an initial experiment with lvm on top of the raid, I decided that was too complex to deal with in the pressure of a disaster and redid it to multiple raids on parallel partitioned hardware. In a disaster the raid would be bad enough to deal with but tolerable, but I did NOT need the complexity of lvm on top of raid, and after dealing with the parts of three different installs mess, I had the hard-earned wisdom to realize it!

The same idea applies here. Once you start reaching into nested subvols to get the deeper nested subvols you're trying to mount, it's too much and you're just begging to get it wrong under the extreme pressures of a disaster recovery. Do it right and keep it mostly flat, and you'll not only know your basic mounts are already setup and you only have to point them at the rollbacks if appropriate, but you won't be having to reach deep into multiple arbitrary nestings to grab subvols from weird spots. Disaster recovery you might well be thanking me, come a few months or a few years from now! =:^)

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

On Fri, 2015-11-27 at 01:02 +0000, Duncan wrote:
[snip snap]

Okay so your main point was basically "keeping things administrable"...

I basically agree, though I'd allow few exceptions, like database-like data that is stored in /var/ sometimes and that doesn't need to be consistent with anything but iself... e.g. static web pages (/var/www)... postgresl DB, or sks keyserver DB... and so on.

btw: What's the proper way for merging / splitting into subvols. E.g. consider I have:
5
|

+--root (subvol)
   |
   +-- var (no subvol)

And say I would want to split of var/www into a subvol.
Well one obvious way would be with mv (and AFAIU that would keep my ref-links with clones, if any) but that also means that anything that accesses /var/www probably needs a downtime.
Is it planned to have a special function that basically says:
"make dir foo and anything below (except nested subvols) a subvol named foo, immediately and atomically"?

And similar vice-versa... a special function that says:
"make subvol foo and anything below (except nested subvols) a dir of the parent subvol named foo, immediately and atomically"?

Could be handy for real world administration, especially when one want's
to avoid downtimes.

btw: Few days ago, either Hugo or your thought that mv'ing a subvol would change it's UUID, but my try (which was with coreutils 8.3 -> no reflinked mv) seemed to show it wouldn't but there was no further reply then... so am I right that the UUID wouldn't change?

Well apparently you oversaw the extremely simple and reliable solution:
leaving a tiny little note on your desk saying something like: "dear boss, things are screwed up, I'm on vacation now..." ;-)

Thanks,
Chris.

Christoph Anton Mitterer posted on Wed, 09 Dec 2015 05:36:37 +0100 as excerpted:

=:^)

Of course. But such data isn't usually managed by the package manager, so splitting off a partition or subvolume or whatever, is fair game.

IOW, /var can and should still be on /, but /var/www and the like can be on some other subvolume or filesystem.

AFAIK, no.

There's a similar, but not identical, question covered on the wiki, use- cases page (link very likely wrapped as I'm not bothering to jump thru the hoops to unwrap while keeping everything else formatted, ATM).

https://btrfs.wiki.kernel.org/index.php/
UseCases#Can_I_take_a_snapshot_of_a_directory.3F

If you use the recipe (subvol create, cp with reflink) it suggests there, you'll end up with the reflinked copy in a subvol.

You can then mount that subvol over top of the existing dir, and *new* file opens will access the new subvol, tho existing open files will of course continue to access the files/reflinks to which they have a reference, "underneath" the new mount.

For some services it's possible to signal them to reload their files. Where this is possible, you can do the overmount trick and then signal them to reload, and they should keep running, otherwise undisturbed (altho any changes between the reflink and processing the signal will still go to the existing open files, I don't believe there's a way around that).

But AFAIK that's the closest it gets, and nothing more along that line is planned.

In general, keep in mind that subvolumes work in most respects very much like normal directories do, except where they don't. =:^) If it can't be done with normal directories and it's not specifically part of subvol/ snapshot functionality, it very likely can't be done with subvols either. From my read, that's the general rule that applies here.

That was Hugo. I read your reply then with interest, but my own use-case doesn't involve subvols, so I'd be unlikely to know that level of specific detail, and even more unlikely to post about it unless I was quite sure on it (as I am about the above due to the wiki reference and previous list discussion).

=:^)

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Agreed. It's not too bad fixing a Gentoo system (as long as /var/lib/portage/world is still correct, you can just nuke the installed package database and emerge world, it'll take time, but it will get your system in a guaranteed consistent state). On something using dpkg or rpm though, it's got the potential to be almost impossible to recover from without a significant amount of low-level knowledge of the package manager's installation database.

Austin S Hemmelgarn posted on Wed, 09 Dec 2015 14:04:06 -0500 as excerpted:

For sufficiently loose values of "consistent", yes, as I found out by experience. But it can be done, and I do have the experience to prove it.

What happens in practice is that while yes, as long as @world is correct you can install to current and have all those files tracked again as appropriate, if your package installation database is missing or out of sync with what's actually on your filesystem(s), where the new version of various packages will replace older files as they come across them during the install process (subject to CONFIG_PROTECT of course, this part isn't the problem), the problem is actually where the files of the actually installed but untracked version differ from those of the version you're installing.

I think the last bug I tracked down to having a stale file from an old version still around, was nearly two years after the initial disaster and recovery. It was some shell script (something to do with eselect IIRC) that had changed installed bindir location between package versions, and the bug was that due to path ordering, the stale version that hadn't been replaced by the reinstall and wasn't removed like it normally would have been when the old version was uninstalled, because my installed package database was out of sync with the backup I had used, was being executed as that bindir came earlier in the path, than the bindir for the updated version that was in the current package.

So yes it did work and I was back in business, but I was tracing bugs down due to stray orphan files that hadn't been cleaned up by the out of sync installation database package uninstall, for two years!

I wouldn't exactly call that "consistent", tho it /was/ consistent /enough/ to get back in business, if with lingering bugs to track down to still stale orphan files for years afterward.

(FWIW, that'd be less of an issue on my own installation now, even if the installed package database got similarly out of sync, because on my system I'm running a unified root/usr and bin/sbin, now, with symlinks /usr -> . and /sbin -> bin, so all normal executables ultimately end up in /bin, and all normal libraries in /lib64, with /lib -> lib64, as well. No-multilib so I don't have to worry about that angle. Portage handles the symlinks just fine, tho I've filed I believe three bugs in total due to individual ebuild scripts (1, resolved/fixed) or cmake (2, one resolved/obsolete as the new version works, one still open, that I'm having to work around) doing the wrong thing.)

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Oh, definately, it's a useable system short term, but not something you should be depending on. The other big difference though is that it's then trivial to bootstrap a clean install on the same system if you have the space for it (which is what I usually end up doing).

Sure, which would mean however that a downtime is still necessary.

Yep,.. but as you say,... it doesn't really help to avoid the downtime,... rather lead to data corruption (not on the filesystem level, of course, but within the application).

I've been so free to add that idea to the project ideas:
https://btrfs.wiki.kernel.org/index.php/Project_ideas#.28Atomically.29_convert_directories_into_subvolumes_and_vice_versa

any developer or more experienced user than me is of course free to remove that again, if it seems not so important or isn't possible to be implemented.

:-P :-P

Cheers,
Chris.

Christoph Anton Mitterer posted on Tue, 24 Nov 2015 22:25:50 +0100 as excerpted:

Consider a setuid-root binary with a recently publicized but patched on your system vuln. But if you have root snapshots from before the patch and those snapshots are nested below root, then they're always accessible. If the path to the vulnerable setuid is as user accessible as it likely was in its original location, then anyone with login access to the system is likely to be able to run it from the snapshot... and will be able to get root due to the vuln.

On a flat layout, a snapshot with the vuln would have to be mounted before it could be accessed, as otherwise it'd be outside the mounted tree.

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Hmm good point... I think it would be great if you could add that scenario somewhere to the documentation. :-)
Based on that one can easily think about more/similar examples... device file that had too permissive modes set, and where snapshotted like that... and so on.

I think that's another example why it would be nice if btrfs had something (per subvolume) like ext4's default mount options (I mean the ones stored in the superblock).

Not only would it allow the userland tools to do things like "adding notatime" per default on snapshots (at least ro snapshot), so that one can have them nested and still doesn't suffer from the previously discussed writes-on-read-amplifications... it would also allow to set things like nodev, noexec, nosuid and that like on subvols... and again it would make the whole thing practically usable with nested subvols.

Where would be the appropriate place to record that as a feature request?
Simply here on the list?

Cheers,
Chris.

Christoph Anton Mitterer posted on Wed, 09 Dec 2015 05:38:02 +0100 as excerpted:

FWIW, I (personally, not sure if that "you" was singular or plural) am much more comfortable on the lists than on wikis, which I tend to treat much as I did printed encyclopedias back in the day -- as great references, but read-only from my perspective. So while I know they can be edited in theory, I just don't tend to get around to it, and don't believe I've ever even gotten a wiki login, on /any/ wiki, let alone the btrfs wiki. But as you've likely noticed, I do find them great references to read and to post links to. =:^)

So while I'm not opposed in principle to editing the wiki, in practice I'm never going to do it. I'll reply here with the info, and if it gets to the wiki, it's almost certainly because someone else posted it there, based on my post here, or not.

Exactly...

While the appropriate place would be on the wiki, where there's a page for that (here, the request is likely to be lost to history, while on the wiki it's in a nice list that both devs and other requesters can look at, a year, five years, a decade... into the future), in this case...

There's already a framework for it and a limited implementation, tho only a small subset of possible options are yet available. The UI exposure is as btrfs property (see the btrfs-property manpage), with the properties, in general, implemented as extented attributes.

You can browse what's currently available, but as I said, it's pretty limited ATM, pretty much just enough to prove the concept and serve as a placeholder for more to be added, later.

Of course other informational and sometimes configurable bits are exposed in the developing btrfs sysfs tree, which is where most of the work of that sort has been focused recently. But it too is somewhat immature and developing, tho at this point it seems to be further along than properties, partly because btrfs' own tools are using it at times.

Basically, both properties and the sysfs interface demonstrate btrfs' status as stabilizING and maturING, as that sort of thing really can't be implemented until the basics have settled down far enough to form a stable reference point, but not yet fully stable or mature, as both interfaces are as yet not fully developed, with more to be added.

But in both cases there's enough there to browse and get a reasonable idea how things are being exposed and how they'll work once they are. =:^)

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Hey.

I'd have an additional question about subvols O:-)

Given the following setup:
5
|

+--root (subvol, /)
   +-- mnt (dir)

with the following done:
- init 1
- remount,ro / (i.e. the subvol root)
- mount /dev/btrfs-device /mnt (i.e. mount the top subvol at /mnt)

The following happened:
- / was ro-mounted (obviously, at least one thing that I had expected correctly)
- /mnt was ro-mounted either (and the /mnt/root/ nested subvol then as well).
  => why is /mnt (i.e. the top level subvol) mounted ro?? => I would have expected that, since / (i.e. the subvol "root" is ro mounted), it's also ro mounted as the nested subvol below 5, i.e. my naive thinking was in terms of logic:
     "/ mounted ro" => "subvol root is mounted ro (everywhere)" => "thus /mnt/root/ is mounted ro as well"

However, the later doesn't seem to be true, cause then I did:
- remount,rw /mnt
=> now /mnt/*, including /mnt/root/* was rw moutned

So I guess my assumption of subvols behaving more or less as if they'd be a fs (and thus mounted at one place ro => everywhere ro) is not true, is it?

Do, ro,rw (and possibly others) instead only affect the respective mountpoint?
And automatically any nested subvols of that mountpoint?

So I could have basically:
/mount-point1/subvol-a ; ro, noexec
/mount-point2/subvol-a ; rw, compress=yes
/root ; rw, compress=no
/root/here/it/is/nested/subvol-a ; (no mountpoint)

(with subvol-a being the same subvol)

And when I write via mount-point1 I'd get an error, but via mount- point2 it works and in addition I get compression, while when writing via the /root mountpoint, where it is nested, I'd get the rw and compress=no from the "parent" mountpoint /root

Does that sounds correct?
It seems to make sense actually, though it's a bit unfamiliar... if I'm not correctly wrong, than e.g. in terms of ext* I cannot have the same fs mounted with different settings,... of course I cannot have it mounted twice at all, but speaking of bind mounts.

So I guess, that when I'd do --bind mounts instead, I actually do get the "old" behaviour, i.e. when the source is ro, then the --bind mount's target is also forcibly ro.

Still, one unclear thing, why got /mnt mounted ro very above?

Thanks,
Chris.

btw: Not sure if I just missed it, but I guess the above should be more or less documented, showing people that mounting subvols (especially when mounting the same several times, either directly or as nested subvol) has these implications.

Quote:

" Most mount options apply to the whole filesystem, and only the options for the first subvolume to be mounted will take effect. This is due to lack of implementation and may change in the future. "

from https://btrfs.wiki.kernel.org/index.php/Mount_options in a red box on the top.

I've had read that, but it doesn't really make clear that that options can effectively differ for the *same* subvol, when mounted several times (or when appearing additionally as nested subvolume).

Chris.

That seems due for a revision because I do rw, ro, rw, rw, ro mounts in sequence and they stick fine. In fact they stick with the same subvolume.

[root@f23m ]# mount /dev/sda7 /mnt/1 -o subvol=home [root@f23m ]# mount /dev/sda7 /mnt/2 -o subvol=home,ro [root@f23m ]# mount /dev/sda7 /mnt/3 -o subvol=home [root@f23m ]# mount
[...snip...]
/dev/sda7 on /mnt/1 type btrfs (rw,relatime,seclabel,ssd,space_cache,subvolid=258,subvol=/home) /dev/sda7 on /mnt/2 type btrfs (ro,relatime,seclabel,ssd,space_cache,subvolid=258,subvol=/home) /dev/sda7 on /mnt/3 type btrfs (rw,relatime,seclabel,ssd,space_cache,subvolid=258,subvol=/home)

And Project Atomic, a.k.a. ostree and rpm-ostree etc., depend on mounting different parts of the same fs volume to different mounts points with different read and read/write settings (bind mounts), and that works too. http://projectatomic.io/

Chris Murphy

Not sure what you mean with "stick" here... I'd say the above has simply the following semantics:
- the default for mounts is rw
- thus /mnt/1 and /mnt/3 are rw, and 3 isn't rw, because 1 was

In other words, if you change that to the following:

# mount /dev/sda7 /mnt/1 -o subvol=home,ro
# mount /dev/sda7 /mnt/2 -o subvol=home,ro
# mount /dev/sda7 /mnt/3 -o subvol=home
I'd expect that you get
ro
ro
rw

At least based on how I understood the whole system now. That was actually my question here:
Q: In other words does mounting the same subvol *again* behave like -- bind mounts, i.e. the further mounts would get the options from the first mount?

And I guess the A:(nswer) is: no, mount options affect the respective mountpoint (including any nested subvols below that) only (except of course it's a --bind mount.

If one of the devs could confirm that semantics, I may find some time to update the wiki accordingly.

Cheers,
Chris.

Well you (since it was your idea and I'd consider you "senior" enough to do it, at least more senior than myself ;) )... or any developer of course ;)

I did that now, or better said I largely rewrote:
https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Subvolumes
which I think should rather be its own wiki article.

One of the devs/experts... please double check it and pick/drop what you like/dislike.
Especially notice that I've changed what was in the wiki, namely subvolid=0 would mount the toplevel subovl,... I changed that to 5. Also I assumed the manpage would be correct and subvol= is always relative to the top level subvol, thus subvol=/ should mount that.

What's still missing now, IMHO, is:
- the snapshots subchapter itself is not complete (especially ro vs. rw snapshots)... and implications like not mounting snapshots noatime, and write amplification effects
- a guide when one should make subvole (e.g. keeping things on the root fs together, unless it's separate like /var/www is usually, but /var/lib typically "corresponds" to a state of /etc and /usr.
- what I've asked in another mail,.. about subvols and mountopts.

That rework also contains your security idea... shamelessly not quoting you there O:-)

Well, I hope me editing such central part of the wiki wasn't to bold,... but if it was, then I guess Chris Mason placing me on the pillory would be nearly as big of an honour as being insulted by Linus ;-)

I think I'm simply going to add it there, in another bold step... O:-)

Mhh,... I've only recently learned about that properties... well if that's technically possible to cleanly solve it with XATTRS... fine for me.

Cheers,
Chris.

I just added that:
https://btrfs.wiki.kernel.org/index.php/SysadminGuide#When_To_Make_Subvolumes

Chris.

Christoph Anton Mitterer posted on Sat, 12 Dec 2015 03:32:57 +0100 as excerpted:

I'd hesitate to call myself a real "expert" on subvolumes because I don't run them here (tho I did play with them a bit, before deciding the additional benefit for my use-case wasn't worth the additional complexity), so nearly all of what I post on the subject is simply restatement of other people's posts on the subject[1], but...

Looks quite reasonable to me... and /much/ more filled out than the last time I read it. =:^)

On that particular point, subvolid=0 is a special-case alias to subvolid=5, for those for which 0 as reference to the toplevel subvolume makes more sense than 5.

So 0 or 5, doesn't particularly matter in practice. Tho 5 is the actual tree ID (AFAIK) with 0 simply an alias to it, so 5 is arguably technically slightly more correct, while 0 could be argued to be humanly slightly more correct as the logical/expected ID for the toplevel subvolume.

You might put that there too, somewhere, if you have a mind to do it...

Haven't the foggiest...

Yes.

I see you added that (as mentioned in a followup). Looks good.

That looks good too... tho to be fair that too was simply reposting someone else's idea... which was both *entirely* new to me when I read it as I'd simply not considered that before, and equally immediately obvious in hindsight, which is why it stuck with me so well, because the combination of entirely new viewpoint on something and "Duh, he's absolutely right and I should have seen it myself" tends to /be/ pretty memorable! =:^)

Anyway, glad it's mentioned on the wiki, now. =:^)

---
[1] No claim of subvolume expert: And also because I've a bit of the sysadmin's healthy superstition of ever claiming to be an expert until it's so clearly the case it's becoming ridiculous to try to claim otherwise, since if there's the slightest doubt, Murphy will surely see/ hear and assert his law shortly thereafter, demonstrating the fallacy of the claim, likely with a brown-bag level error! =:^\ Of course, without anything but the toplevel subvolumes here, that might make the unwary sysadmin unwisely bold, but lest I be unpleasantly reminded of the fact that the top-level's a subvolume too, I'm still wary of tempting either Murphy or fate with such a claim of expertise! =:^\

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Talking about security: If you split the "var" parts from the root, you get a nice side-effect that the generation of the root subvolume does not change until you update your system (e.g. by installing/updating packages).

Consider this setup:

5
|

+--root (subvol)
|  |
|  +--var (subvol, or mounted)
|  +--tmp (subvol, or tmpfs)
|  +--run (subvol, or tmpfs)
|  +--home (subvol, or mounted)
|
+--snapshots (dir, uid=root,mode=700)
   |
   +--root.20150101
   +--root.20150102
   +--...

Now if you take a look at the output of "btrfs sub list -c /", you can see exactly when your root subvolume has changed. This means if you always create a new snapshot of "root" after you update your system, you are able to spot changes to the system easily:

# btrfs sub list -c /
ID 100 gen 4242 cgen 50 top level 5 path root
ID 101 gen 4500 cgen 53 top level 5 path var
[...]
ID 5000 gen 4111 cgen 4111 top level 5 path snapshots/root.20151203
ID 5001 gen 4242 cgen 4242 top level 5 path snapshots/root.20151205
[...]

In this example, I can see that "root" has not been altered since 20151205, as "cgen" of the snapshot matches the "gen" of the root subvolume. This is the last time I updated this system. If the "gen" of the root subvolume would be greater than 4242, I know some changes were made to the system root, and I can start tracking it down using "btrfs subvol find-new".