syntax for deleting subvolumes?

I'm having trouble deleting a subvolume.

[root@new-alfred ~]# uname -a
Linux new-alfred.corp.graphitesystems.com 3.10.0-123.el7.x86_64 #1 SMP Mon May 5 11:16:57 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@new-alfred ~]# btrfs --version
Btrfs v3.16.2
[root@new-alfred ~]# btrfs fi show
Label: 'rhel_bark' uuid: d3c6c783-c190-4675-99f2-b83f3c462656

        Total devices 2 FS bytes used 14.25GiB
        devid    1 size 461.38GiB used 7.03GiB path /dev/sda2
        devid    2 size 465.76GiB used 12.01GiB path /dev/sdb1

Btrfs v3.16.2
[root@new-alfred ~]# btrfs fi df /
Data, single: total=15.01GiB, used=13.19GiB System, RAID1: total=8.00MiB, used=16.00KiB System, single: total=4.00MiB, used=0.00 Metadata, RAID1: total=2.00GiB, used=1.07GiB Metadata, single: total=8.00MiB, used=0.00 [root@new-alfred ~]# btrfs subvolume list / ID 257 gen 1483 top level 5 path root
ID 258 gen 1466 top level 5 path home [root@new-alfred ~]# btrfs subvolume delete home Transaction commit: none (default)
ERROR: error accessing 'home' [root@new-alfred ~]# btrfs subvolume delete /home Transaction commit: none (default)
ERROR: error accessing '/home'

What format do I need to use to delete the "home" subvolume?

--rich

If highly recommend that you check out elrepo.org for a newer kernel than this. Currently they have kernel-ml-3.19.1-1.el7.elrepo.x86_64.rpm available.

The Fedora 21 package for btrfs-progs-3.19 also works on RHEL 7/CentOS 7, which you can find in koji.fedoraproject.org, or build it from source if you prefer.

# mkdir  /mnt/btr
# mount /dev/sda2 /mnt/btr
# btrfs subvolume delete /mnt/btr/home

The Fedora/RHEL/CentOS installer creates two subvolumes: root and home. If you check out fstab, those subvolumes are mounted at / and /home. Therefore the top level subvolume (id 5) is not mounted by default, so there's no way to delete subvolumes in the top level.

Chris Murphy

Ah! Thank you. That's the piece I was missing.

IMO, someone needs to take a clue-by-four to the heads of the Fedora/RHEL/CentOS installer folks. I see no reason for this with btrfs.

--rich

Actually, it's the recommended approach.

We've found over time that nesting subvolumes inside each other is,
in general, more trouble than it's worth. It makes it harder to do things like take snapshots and then replace the original with a snapshot (e.g. rollback to an earlier state).

Arranging the subvolumes with the top level (subvolid=0) as nothing
but a store for subvolumes means that you don't get the subvolumes tangled up with each other, so it's much easier to manage them in general.

There's a few cases where nesting subvolumes is helpful, but that's
more to do with preventing snapshots being taken of some pieces of the filesystem.

Hugo.

Hugo Mills             | "There's a Martian war machine outside -- they want
hugo@... carfax.org.uk | to talk to you about a cure for the common cold."
http://carfax.org.uk/  |
PGP: 65E74AC0          |                           Stephen Franklin, Babylon 5

Yes, I understand about the flat arrangement. It's been that way for years now.

My complaint is a) about multiple subvols and b) about an unnecessary and redundant subvol for the top level file system.

The redundancy should be obvious. And the problem with multiple subvols is that they aren't needed and make snapshotting an entire system more difficult than it needs to be. I want to be able to snapshot and chroot. The traditional reason for partitioning a disk was to isolate errors and to firewall "out of space" usage. Neither of those really apply to btrfs as far as I can see.

I would much rather the installer simply create a single file system and mount the partition directly rather than through a superfluous subvol. I've never seen it done any other way as you might guess from my confusion above.

--rich

The current granularity supplied by root and home subvolumes is minor. Eventually there'd also be a boot subvolume too, but that's not supported yet due to a very old grubby bug that's like a booger that can't be flicked off.

openSUSE uses ~13 subvolumes by default, for an idea of much finer granularity (which I don't like, personally, it's too much and really is unnecessary).

There's rarely a use case where you want to rollback root and home at the same time anyway, therefore making them separate subvolumes makes sense. There could be an argument that /boot can be a directory rather than a subvolume since it probably shouldn't get separated from its root anyway due to kernel+kernel-module pairing and bootloader stuff.

A separate home subvolume is consistent with the RHEL/CentOS/Fedora installer (Anaconda) default of putting home on a separate volume. It makes home easier to backup, restore, migrate, and upgrade systems. Btrfs support is uniquely handled by Anaconda too, in that it doesn't require Btrfs volumes be reformat to do a clean install like for other file systems. It does require a new subvolume, however, for the "rootxx" subvolume.

It's the same on Btrfs in that regard as any other. Some people prefer partitions, others prefer quotas.

The confusion is understandable. But there's an express intent behind the layout. Maybe the installer should include a brief explanatory readme.txt somewhere? Or maybe integrate it as comments in /etc/fstab?

Chris Murphy

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

We use this layout in SLES too and it's necessary for both compliance and principle-of-lease-surprise purposes in concert with our snapshot-rollback facility. If you roll back your root file system, would you really expect to lose all your logs? How about your mail? Do you want snapshots of /tmp to hang around eating up disk space, causing you to reduce the number of snapshots you can retain? Keeping these as separate subvolumes also allows us to mount them seamlessly when you boot from a snapshot to either recover or reset your system.

Does it look nice and neat? That's a matter of style. But the reasons for having separate subvolumes are well thought out and completely necessary.

BTW, going back to the original issue of why you'd even have a separate non-top-level subvolume for root - if you boot from a snapshot and choose to continue from there, you wouldn't be able to remove the old "root" if it weren't a separate subvolume.

- -Jeff

- --
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)

iQIcBAEBAgAGBQJVChyhAAoJEB57S2MheeWyQoAQAKVJmrh+BA8MBApVqmpOADUi o7RvsSx0D/2lReqep9p2zCyLq0RCorClP7ESKqu+VXCl4gLkZmLszr1G5btTthk8 524vN0ErR5vb2L7hbZ/Rexkd5+H3CRmKsgLPeHeO8ladx57k1lF3bDEX4J6ssfd/ 4MbIhZ/B3Xu8kxe7Te69T/86kL8A0teVesBARRlxewtm1AzMn6ABXVrAIWhJU9rk vOe+8L/kMXTqOod0cv9wQNKWu5y587dHrrjGsTQ9qcrdTrudbFq0GKzO5gDaFTm+ RZ8LWucjMKdTDzLVxdmt0KxcBCS/fLTWjSoKeVjHk3If/ciFRYM7DZeCxKsmLU1j 5Fwtlm3VjcquzLRyAZST3buJ0atuP9w1Onu2LNKn4xymQlLvoF2U/Fp/gerNiDxe ZxVBA6JuwG8PktAFqKNYZC9Wd3+BBmYUTyV7JXM5qXbuXZ8uN1O0e0I3XcN3zgh/ 56FPQ7cKIe9AxCjxJzlho8ZDQ8kpYB1p+NiGAs7qiCIfsNKwH3ZfuwoBizdagM8M KmFUVLC/y7b9vVQW5bQX+4NPJU9dmAYe0/dVD68ZK6ROF4b+pV5baP+qlFu8Hp1+ 8DFxlG3wJ51WSoZuIvAWV+9y0drhamOONAQ0D1smddnbWvG1HoiL+Z4v36iYc9+3 kGXR3LtcuNWaV5rCjZ3i
=dToy
-----END PGP SIGNATURE-----

I understand the logic, I'm just not convinced it's the only way to achieve that. It's kinda like 28 GPT partitions on the typical Android/Cyanogen mobile phone — as long as everything works and you don't interact with it, it's fine. But as soon as you need to understand it, I think this layout and what's going on is non-obvious.

After a clean installation, and updates, it looks like this:

# btrfs sub list /
ID 257 gen 36 top level 5 path boot/grub2/i386-pc
ID 258 gen 36 top level 5 path boot/grub2/x86_64-efi
ID 259 gen 24 top level 5 path opt
ID 260 gen 36 top level 5 path srv
ID 261 gen 101 top level 5 path tmp
ID 262 gen 36 top level 5 path usr/local
ID 263 gen 36 top level 5 path var/crash
ID 264 gen 36 top level 5 path var/lib/mailman
ID 265 gen 36 top level 5 path var/lib/named
ID 266 gen 36 top level 5 path var/lib/pgsql
ID 267 gen 101 top level 5 path var/log
ID 268 gen 36 top level 5 path var/opt
ID 269 gen 101 top level 5 path var/spool
ID 270 gen 101 top level 5 path var/tmp
ID 275 gen 101 top level 5 path .snapshots
ID 276 gen 38 top level 275 path .snapshots/1/snapshot
ID 277 gen 41 top level 275 path .snapshots/2/snapshot
ID 278 gen 43 top level 275 path .snapshots/3/snapshot
ID 279 gen 46 top level 275 path .snapshots/4/snapshot
ID 282 gen 81 top level 275 path .snapshots/5/snapshot
ID 283 gen 97 top level 275 path .snapshots/6/snapshot

I reckognize that these subvolumes exist to *exclude* them from the snapshotting policies, rather than to include them. That's part of the typical user confusion, since subvolumes are associated with making snapshots rather than avoiding them.

Plus it also confuses the hell out of the RH/Fedora installer which ends up seeing every snapshot as a completely separate unique openSUSE installation - because the distros haven't had any sort of conversation about interoperability in this brave new world.
https://bugzilla.redhat.com/attachment.cgi?id=983116

Of course not. I just disagree that there's only one way to manage this.

For the most part the "OS" is in /usr, and users' stuff is in /home, and then there's everything else which includes /etc, /var, and /boot being the things that change state the most at a system level. I would like to see applications have an explicit home. This is a huge problem on linux as they spread themselves throughout the filesystem. And mobile platforms, apps are in a separate place, on Linux desktop/server I should be able to revert applications separate from an OS update, those two things aren't the same thing.

I'd also prefer seeing updates/upgrades handled differently. Create the proper snapshots, upgrade the snapshots in a container, boot the upgraded snapshot in a container to test its viability, and if successful then make it a new boot option at next boot; if it fails then obliterate the snapshot. This makes for atomic upgrades that aren't yanking or modifying files out from running processes, and permits indefinitely delayed reboots.

And eventually I really want to get to stateless systems on Linux. Mobile platforms have surpassed desktop/server Linux in this regard. Even Windows 8+ have surpassed it, with system refresh and reset options.

I'm all for refining the granularity, but I think the subvolume as snapshot exclusion is messy and confusing. I have to use snapper to figure out the relationship of anything in .snapshot. And anything in .snapshot is accessible from the normally mounted fs hierarchy. AND when doing backups I have to make really really sure that I exclude .snapshot or I've got a very good chance of encountering an ENOSPC on the backup target due to what will appear to be dozens or hundreds of OS's - the deduplication aspect of those snapshots is lost

Well I put /tmp on tmpfs so I don't worry about that.

I understand the logic, I just happen to disagree this is the only way to achieve the desired outcome. And I also think it's possible to come up with one layout that is compatible with both snapshot+rollback, and stateless systems paradigms.

Chris Murphy

Other than the technical reasons Hugo mentions regarding nesting...

The problem with the "install normally to top level with Linux FHS" approach like Ubuntu and openSUSE follow now, is that snapshots then have to go in the mounted path. This arguably exposes old binaries in that mounted path and is a possible security risk. There are some ways to mitigate that, but better when it's simply not in the mounted path, sorta like a chroot.

It's also a better way to organize stateless systems. Myriad trees that can be used to form a stateless system existing "out of tree" and mounted either by path or subvolid is more sane (or at least less madness inducing) than alternatives. See under "what we propose" for the subvol naming convention:
http://0pointer.net/blog/revisiting-how-we-put-together-linux-systems.html

This is also compatible with delivery of such systems with a btrfs seed device.

Chris Murphy

I see. Thanks for the education.

I'm not sure what I think about the possible security risk, but I hear the concern.

Most of the uses I have for btrfs involve fairly dynamic use of snapshots, typically by non-root users. That's what brought me to btrfs in the first place and continues to be the biggest driver for me.

Because of this, the top level file system would need to be mounted pretty much constantly, which essentially removes any benefit from the redundant top level subvol. It's just a nuisance for my applications. And most of my applications try very hard to avoid mounting the snapshots. That takes too much time and isn't reentrant.

It seems to me that it depends on whether you think of snapshots as a system admin sort of facility or as a user facility. As a system admin facility, you're probably right. But as a user level facility, I want to be able to snapshot before making a change to a tree full of source code and (re)building it all over again. I may want to keep my new build, but I may want to flush it and return to known good state. It's pretty easy to open that facility up to non-root users but the easiest way to do that that I've found is to use a single file system on root mounted directly. For an individual user, this can easily save hours and hundreds of gigabytes. For automated build systems, it can mean a few orders of magnitude difference in typical build times.

It's not clear to me yet how to set machines up this way from kickstart, which makes this scheme look like an impediment, rather than a feature. But maybe all I need is an easy way to shut it off and get the more familiar arrangement.

--rich

By default, none privileged users can create subvolumes and snapshots, but they can't delete them. User deletion of subvolumes requires a mount option to enable. I'm not sure if this mount option exposes all subvolumes to none privileged users' ability to delete. Eventually, I think the idea is to make subvolumes behave even more like directories. Right now that's largely the case but the line needs to be blurred a bit more.

User subvolumes/snapshots nested under the home subvolume aren't being proscribed. It's just that doing this out of the gate with an installation could lead to unexpected results.

Realize that snapshots are non-recursive for any nested subvol/snapshots. So by virtue of your users creating subvols or snapshots in their own home directory, if you snapshot home, *none* of their subvolumes (including their snapshots) will be snapshot. And that means they're excluded from btrfs send/receive as well. So long as you and users know this, it's fine to nest them.

Sure. The near equivalent to a btrfs subvolume is a new file system, and creating new file systems is not typically a user facility. A btrfs subvolume is a separate fs tree, not a completely separate file system, so it's sorta something in between that of a directory and a new separate file system. For example inode counts start over. So two different files in two subvolumes can have the same inode number.

So I'm not disputing the use case you've come up with, just that it's going to take some consideration of all the myriad consequences of the current state.

And for that matter, definitely don't depend on EL kernels for serious Btrfs use. Btrfs is a tech preview. The latest features and bug fixes will be in much newer stable kernels, and there's been a metric shitton of changes since 3.10.

I'm fairly certain Anaconda considers any defined mount point to be cause to create a subvolume. Just like for on LVM, a specific mount point causes an LV to be created; and for Standard Partitions each mount point is a separate partition.

Chris Murphy

You may want to check out cp --reflink=always (different from cp --link), which creates copy-on-write copy of the data. It isn't quite as fast as snapshots to create, but it's still plenty fast and without the downsides of subvolumes.

  _____________________________________________________________________
     / __// /__ ____  __               http://www.modeemi.fi/~flux/\   \
    / /_ / // // /\ \/ /                                            \  /
   /_/  /_/ \___/ /_/\_\@modeemi.fi                                  \/

In which situations is --reflink=always expected to fail on BTRFS?

I recently needed to move /home on one system to a subvol (instead of being in the root subvol) and used cp --reflink=always which gave a few errors when running with BTRFS on kernel 3.16.7-ckt7-1. I was in a hurry and didn't try to track down the caue, I just ran it again with --reflink=auto which gave no errors and completed very quickly (obviously the files which couldn't be reflinked were small).

My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/

Another thing. Some distros behave this way:

chris@linux-6gc0:~> btrfs sub list /
Absolute path to 'btrfs' is '/usr/sbin/btrfs', so running it may require superuser privileges (eg. root).

As is the case on openSUSE. On Fedora, any user can use the btrfs command, without sudo, and even without being in group wheel. Kindof a stark difference, I'm vaguely curious about the security implications. So this normal/non-admin user I'm logged in right now can create subvolumes and snapshots at will, but I can't delete them (as previously mentioned).

Erkki's cp --reflink idea is a good one. I've often wondered if it's a good idea, and possible, to eventually make --reflink the default behavior with Btrfs copies (I think some things probably first need enhancement like the cross subvolume stuff, because otherwise cp could fail inexplicably).

Chris Murphy

Chris Murphy posted on Thu, 19 Mar 2015 13:09:06 -0600 as excerpted:

If cp --reflink=auto was the default, it'd "just work", making a reflink where possible, falling back to a normal copy where not possible to reflink.

However, I'd be wary of such a change, because admins are used to cp creating a separate copy which may well be intended as a backup, guarding against I/O errors on the original. Btrfs does do checksum checking to ensure validity, but if that fails, do you really want it failing for BOTH copies, including the one the admin made specifically to AVOID such a problem, because he knows the file is likely to be open at the time of a crash or some such and did a cp to have a backup, just in case? I know as an admin, *I* wouldn't like such behavior. Far better to simply have the option there should I deliberately invoke it, but not invoke it by default, in ordered to avoid unhappy surprises!

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

If you have a second copy on the same filesystem then an error on any of the metadata that is a parent will corrupt both. EG if you have 2 copies under your home directory then a directory error for /, /home, or /home/$USER will make both copies unavailable.

Also any errors on superblocks or other essential filesystem metadata risks losing both copies.

If BTRFS was to adopt something equivalent to the ZFS "copies=" feature then the sysadmin could specify that certain subtrees would have extra copies of data AND each metadata block would have 1 more copy than the data blocks it refers to.

In conclusion I think that the ZFS copies= feature is the correct solution to this problem. Until/unless the BFTRFS developers copy that design concept the thing to do if you want backups on the same storage hardware is to copy the data to a filesystem on a different partition - the NetApp research shows that disk read errors tend to be correlated by location on disk.

My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/

I'm guessing you've got the home subvolume mounted, something like:

# mount /dev/sda2 /home -o subvol=home

You can't delete that while it's mounted, because you'd be trying
to delete the subvolume out from under the mountpoint. So the first thing is to unmount it.

Then the syntax to delete a subvolume is:

# btrfs sub del /path/to/subvolume

... but at the moment, you can't see the subvolume, because it's not mounted, and there's no way to see it. The trick here it to mount something higher than the subvolume you want to operate on, and access it through that mount path. So, mount the top level subvolume somewhere:

# mount /dev/sda2 /media/btrfs -o subvolid=0

then delete the subvolume through that:

# btrfs sub del /media/btrfs/home

In fact, I'd recommend doing _all_ subvolume management operations
(and nothing else) through the top-level mount at /media/btrfs.

Hugo.

Hugo Mills             | "There's a Martian war machine outside -- they want
hugo@... carfax.org.uk | to talk to you about a cure for the common cold."
http://carfax.org.uk/  |
PGP: 65E74AC0          |                           Stephen Franklin, Babylon 5