btrfs-progs: my libbtrfsutil patch queue
cover btrfs-progs: my libbtrfsutil patch queue
From: Omar Sandoval <[email protected]>
Hi,
This series contains my backlog of libbtrfsutil changes which I've been collecting over the past few weeks.
Patches 1-4 are fixes. Patches 5-6 add functionality to the unit tests which is needed for patches 7-8. Patches 7-8 add support for the unprivileged ioctls added in Linux 4.18; more on those below. Patch 9 bumps the library version. Patch 10 adds documentation for the available API along with examples.
Patches 7-8 are based on Misono Tomohiro's previous patch series [1], with a few important changes.
- Both subvolume_info() and create_subvolume_iterator() now have unit tests for the unprivileged case.
- Both no longer explicitly check that top == 0 in the unprivileged case, since that will already fail with a clear permission error.
- Unprivileged iteration is much simpler: it uses openat() instead of fchdir() and is based more closely on the original tree search variant. This fixes a bug in post-order iteration in Misono's version.
- Unprivileged iteration does _not_ support passing in a non-subvolume path; if this behavior is desired, I'd like it to be a separate change with an explicit flag.
Please take a look.
Thanks!
1: https://www.spinics.net/lists/linux-btrfs/msg79285.html
Omar Sandoval (10):
libbtrfsutil: use top=0 as default for SubvolumeIterator()
libbtrfsutil: change async parameters to async_ in Python bindings
libbtrfsutil: document qgroup_inherit parameter in Python bindings
libbtrfsutil: use SubvolumeIterator as context manager in tests
libbtrfsutil: add test helpers for dropping privileges
libbtrfsutil: allow tests to create multiple Btrfs instances
libbtrfsutil: relax the privileges of subvolume_info()
libbtrfsutil: relax the privileges of subvolume iterator
libbtrfsutil: bump version to 1.1.0
libbtrfsutil: document API in README
libbtrfsutil/README.md | 422 +++++++++++++++++++- libbtrfsutil/btrfsutil.h | 21 +- libbtrfsutil/errors.c | 8 + libbtrfsutil/python/module.c | 17 +- libbtrfsutil/python/subvolume.c | 6 +- libbtrfsutil/python/tests/__init__.py | 66 ++- libbtrfsutil/python/tests/test_subvolume.py | 215 +++++++--- libbtrfsutil/subvolume.c | 407 ++++++++++++++++--- 8 files changed, 1029 insertions(+), 133 deletions(-)
Signature · 1 line
2.19.1
PATCH 1/10 libbtrfsutil: use top=0 as default for SubvolumeIterator()
From: Omar Sandoval <[email protected]>
Right now, we're defaulting to top=5 (i.e, all subvolumes). The documented default is top=0 (i.e, only beneath the given path). This is the expected behavior. Fix it and make the test cases cover it.
--- libbtrfsutil/python/subvolume.c | 2 +- libbtrfsutil/python/tests/test_subvolume.py | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-)
libbtrfsutil/python/subvolume.c +1 -1
@@ -525,7 +525,7 @@ static int SubvolumeIterator_init(SubvolumeIterator *self, PyObject *args,static char *keywords[] = {"path" ,"top" ,"info" ,"post_order" ,NULL };struct path_arg path = {.allow_fd = true};enum btrfs_util_error err; -unsigned long long top =5 ; +unsigned long long top =0 ;int info =0 ;int post_order =0 ;int flags =0 ;
libbtrfsutil/python/tests/test_subvolume.py +2 -0
@@ -353,6 +353,7 @@ class TestSubvolume(BtrfsTestCase): with self.subTest(type=type(arg)): self.assertEqual(list(btrfsutil.SubvolumeIterator(arg)), subvols) self.assertEqual(list(btrfsutil.SubvolumeIterator('.', top=0)), subvols) + self.assertEqual(list(btrfsutil.SubvolumeIterator('foo', top=5)), subvols) self.assertEqual(list(btrfsutil.SubvolumeIterator('.', post_order=True)), [('foo/bar/baz', 258),
@@ -365,6 +366,7 @@ class TestSubvolume(BtrfsTestCase): ] self.assertEqual(list(btrfsutil.SubvolumeIterator('.', top=256)), subvols) + self.assertEqual(list(btrfsutil.SubvolumeIterator('foo')), subvols) self.assertEqual(list(btrfsutil.SubvolumeIterator('foo', top=0)), subvols) os.rename('foo/bar/baz', 'baz')
Signature · 1 line
2.19.1
PATCH 2/10 libbtrfsutil: change async parameters to async_ in Python bindings
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
async became a keyword in Python 3.7, so, e.g., create_subvolume('foo', async=True) is now a syntax error. Fix it with the Python convention of adding a trailing underscore to the keyword (async -> async_). This is what several other Python libraries did to handle this.
--- libbtrfsutil/python/module.c | 8 ++++---- libbtrfsutil/python/subvolume.c | 4 ++-- libbtrfsutil/python/tests/test_subvolume.py | 4 ++-- 3 files changed, 8 insertions(+), 8 deletions(-)
libbtrfsutil/python/module.c +4 -4
@@ -233,22 +233,22 @@ static PyMethodDef btrfsutil_methods[] = { "this ID instead of the given path"}, {"create_subvolume" , (PyCFunction )create_subvolume,METH_VARARGS |METH_KEYWORDS , -"create_subvolume(path, async=False)\n\n" +"create_subvolume(path, async_=False)\n\n" "Create a new subvolume.\n\n" "Arguments:\n" "path -- string, bytes, or path-like object\n" -"async -- create the subvolume without waiting for it to commit to\n" +"async_ -- create the subvolume without waiting for it to commit to\n" "disk and return the transaction ID"}, {"create_snapshot", (PyCFunction)create_snapshot,METH_VARARGS |METH_KEYWORDS , -"create_snapshot(source, path, recursive=False, read_only=False, async=False)\n\n" +"create_snapshot(source, path, recursive=False, read_only=False, async_=False)\n\n" "Create a new snapshot.\n\n" "Arguments:\n" "source -- string, bytes, path-like object, or open file descriptor\n" "path -- string, bytes, or path-like object\n" "recursive -- also snapshot child subvolumes\n" "read_only -- create a read-only snapshot\n" -"async -- create the subvolume without waiting for it to commit to\n" +"async_ -- create the subvolume without waiting for it to commit to\n" "disk and return the transaction ID"}, {"delete_subvolume", (PyCFunction)delete_subvolume, METH_VARARGS | METH_KEYWORDS,
libbtrfsutil/python/subvolume.c +2 -2
@@ -322,7 +322,7 @@ PyObject *set_default_subvolume(PyObject *self, PyObject *args, PyObject *kwds)PyObject *create_subvolume (PyObject *self,PyObject *args,PyObject *kwds) { -static char *keywords[] = {"path" ,"async" ,"qgroup_inherit" ,NULL }; +static char *keywords[] = {"path" ,"async_" ,"qgroup_inherit" ,NULL };struct path_arg path = {.allow_fd = false};enum btrfs_util_error err;int async =0 ;
@@ -352,7 +352,7 @@ PyObject *create_subvolume(PyObject *self, PyObject *args, PyObject *kwds)PyObject *create_snapshot (PyObject *self,PyObject *args,PyObject *kwds) {static char *keywords[] = { -"source" ,"path" ,"recursive" ,"read_only" ,"async" , +"source" ,"path" ,"recursive" ,"read_only" ,"async_" ,"qgroup_inherit" ,NULL , };struct path_arg src = {.allow_fd = true}, dst = {.allow_fd = false};
libbtrfsutil/python/tests/test_subvolume.py +2 -2
@@ -202,7 +202,7 @@ class TestSubvolume(BtrfsTestCase): btrfsutil.create_subvolume(subvol + '6//') self.assertTrue(btrfsutil.is_subvolume(subvol + '6')) - transid = btrfsutil.create_subvolume(subvol + '7', async=True) + transid = btrfsutil.create_subvolume(subvol + '7', async_=True) self.assertTrue(btrfsutil.is_subvolume(subvol + '7')) self.assertGreater(transid, 0)
@@ -265,7 +265,7 @@ class TestSubvolume(BtrfsTestCase): btrfsutil.create_snapshot(subvol, snapshot + '2', recursive=True) self.assertTrue(os.path.exists(os.path.join(snapshot + '2', 'nested/more_nested/nested_dir'))) - transid = btrfsutil.create_snapshot(subvol, snapshot + '3', recursive=True, async=True) + transid = btrfsutil.create_snapshot(subvol, snapshot + '3', recursive=True, async_=True) self.assertTrue(os.path.exists(os.path.join(snapshot + '3', 'nested/more_nested/nested_dir'))) self.assertGreater(transid, 0)
Signature · 1 line
2.19.1
PATCH 3/10 libbtrfsutil: document qgroup_inherit parameter in Python bindings
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
This has been supported since day one, but it wasn't documented.
--- libbtrfsutil/python/module.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
libbtrfsutil/python/module.c +9 -4
@@ -233,15 +233,18 @@ static PyMethodDef btrfsutil_methods[] = { "this ID instead of the given path"}, {"create_subvolume" , (PyCFunction )create_subvolume,METH_VARARGS |METH_KEYWORDS , -"create_subvolume(path, async_=False)\n\n" +"create_subvolume(path, async_=False, qgroup_inherit=None)\n\n" "Create a new subvolume.\n\n" "Arguments:\n" "path -- string, bytes, or path-like object\n" "async_ -- create the subvolume without waiting for it to commit to\n" - "disk and return the transaction ID"}, +"disk and return the transaction ID\n" +"qgroup_inherit -- optional QgroupInherit object of qgroups to\n" + "inherit from"}, {"create_snapshot", (PyCFunction)create_snapshot,METH_VARARGS |METH_KEYWORDS , -"create_snapshot(source, path, recursive=False, read_only=False, async_=False)\n\n" +"create_snapshot(source, path, recursive=False, read_only=False,\n" +" async_=False, qgroup_inherit=None)\n\n" "Create a new snapshot.\n\n" "Arguments:\n" "source -- string, bytes, path-like object, or open file descriptor\n"
@@ -249,7 +252,9 @@ static PyMethodDef btrfsutil_methods[] = {"recursive -- also snapshot child subvolumes\n" "read_only -- create a read-only snapshot\n" "async_ -- create the subvolume without waiting for it to commit to\n" - "disk and return the transaction ID"}, +"disk and return the transaction ID\n" +"qgroup_inherit -- optional QgroupInherit object of qgroups to\n" + "inherit from"}, {"delete_subvolume", (PyCFunction)delete_subvolume,METH_VARARGS |METH_KEYWORDS ,"delete_subvolume(path, recursive=False)\n\n"
Signature · 1 line
2.19.1
PATCH 4/10 libbtrfsutil: use SubvolumeIterator as context manager in tests
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
We're leaking file descriptors, which makes it impossible to clean up the temporary mount point created by the test.
--- libbtrfsutil/python/tests/test_subvolume.py | 51 ++++++++++++--------- 1 file changed, 30 insertions(+), 21 deletions(-)
libbtrfsutil/python/tests/test_subvolume.py +30 -21
@@ -334,11 +334,13 @@ class TestSubvolume(BtrfsTestCase): os.chdir(self.mountpoint) btrfsutil.create_subvolume('foo') - path, subvol = next(btrfsutil.SubvolumeIterator('.', info=True)) - self.assertEqual(path, 'foo') - self.assertIsInstance(subvol, btrfsutil.SubvolumeInfo) - self.assertEqual(subvol.id, 256) - self.assertEqual(subvol.parent_id, 5) + with btrfsutil.SubvolumeIterator('.', info=True) as it: + path, subvol = next(it) + self.assertEqual(path, 'foo') + self.assertIsInstance(subvol, btrfsutil.SubvolumeInfo) + self.assertEqual(subvol.id, 256) + self.assertEqual(subvol.parent_id, 5) + self.assertRaises(StopIteration, next, it) btrfsutil.create_subvolume('foo/bar') btrfsutil.create_subvolume('foo/bar/baz')
@@ -350,30 +352,37 @@ class TestSubvolume(BtrfsTestCase): ] for arg in self.path_or_fd('.'): - with self.subTest(type=type(arg)): - self.assertEqual(list(btrfsutil.SubvolumeIterator(arg)), subvols) - self.assertEqual(list(btrfsutil.SubvolumeIterator('.', top=0)), subvols) - self.assertEqual(list(btrfsutil.SubvolumeIterator('foo', top=5)), subvols) - - self.assertEqual(list(btrfsutil.SubvolumeIterator('.', post_order=True)), - [('foo/bar/baz', 258), - ('foo/bar', 257), - ('foo', 256)]) + with self.subTest(type=type(arg)), btrfsutil.SubvolumeIterator(arg) as it: + self.assertEqual(list(it), subvols) + with btrfsutil.SubvolumeIterator('.', top=0) as it: + self.assertEqual(list(it), subvols) + with btrfsutil.SubvolumeIterator('foo', top=5) as it: + self.assertEqual(list(it), subvols) + + with btrfsutil.SubvolumeIterator('.', post_order=True) as it: + self.assertEqual(list(it), + [('foo/bar/baz', 258), + ('foo/bar', 257), + ('foo', 256)]) subvols = [ ('bar', 257), ('bar/baz', 258), ] - self.assertEqual(list(btrfsutil.SubvolumeIterator('.', top=256)), subvols) - self.assertEqual(list(btrfsutil.SubvolumeIterator('foo')), subvols) - self.assertEqual(list(btrfsutil.SubvolumeIterator('foo', top=0)), subvols) + with btrfsutil.SubvolumeIterator('.', top=256) as it: + self.assertEqual(list(it), subvols) + with btrfsutil.SubvolumeIterator('foo') as it: + self.assertEqual(list(it), subvols) + with btrfsutil.SubvolumeIterator('foo', top=0) as it: + self.assertEqual(list(it), subvols) os.rename('foo/bar/baz', 'baz') - self.assertEqual(sorted(btrfsutil.SubvolumeIterator('.')), - [('baz', 258), - ('foo', 256), - ('foo/bar', 257)]) + with btrfsutil.SubvolumeIterator('.') as it: + self.assertEqual(sorted(it), + [('baz', 258), + ('foo', 256), + ('foo/bar', 257)]) with btrfsutil.SubvolumeIterator('.') as it: self.assertGreaterEqual(it.fileno(), 0)
Signature · 1 line
2.19.1
PATCH 5/10 libbtrfsutil: add test helpers for dropping privileges
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
These will be used for testing some upcoming changes which allow unprivileged operations.
--- libbtrfsutil/python/tests/__init__.py | 31 ++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)
libbtrfsutil/python/tests/__init__.py +30 -1
@@ -15,14 +15,44 @@ # You should have received a copy of the GNU Lesser General Public License # along with libbtrfsutil. If not, see <http://www.gnu.org/licenses/>. +import contextlib import os from pathlib import PurePath +import pwd import subprocess import tempfile import unittest HAVE_PATH_LIKE = hasattr(PurePath, '__fspath__') +try: + NOBODY_UID = pwd.getpwnam('nobody').pw_uid + skipUnlessHaveNobody = lambda func: func +except KeyError: + NOBODY_UID = None + skipUnlessHaveNobody = unittest.skip('must have nobody user') + + [email protected] +def drop_privs(): + try: + os.seteuid(NOBODY_UID) + yield + finally: + os.seteuid(0) + + [email protected] +def regain_privs(): + uid = os.geteuid() + if uid: + try: + os.seteuid(0) + yield + finally: + os.seteuid(uid) + else: + yield @unittest.skipIf(os.geteuid() != 0, 'must be run as root')
@@ -67,4 +97,3 @@ class BtrfsTestCase(unittest.TestCase): yield fd finally: os.close(fd) -
Signature · 1 line
2.19.1
PATCH 6/10 libbtrfsutil: allow tests to create multiple Btrfs instances
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
Some upcoming tests will need to create a second Btrfs filesystem, so add support for this to the test helpers.
--- libbtrfsutil/python/tests/__init__.py | 35 +++++++++++++++++---------- 1 file changed, 22 insertions(+), 13 deletions(-)
libbtrfsutil/python/tests/__init__.py +22 -13
@@ -57,14 +57,18 @@ def regain_privs(): @unittest.skipIf(os.geteuid() != 0, 'must be run as root') class BtrfsTestCase(unittest.TestCase): - def setUp(self): - self.mountpoint = tempfile.mkdtemp() + def __init__(self, *args, **kwds): + super().__init__(*args, **kwds) + self._mountpoints = [] + + def mount_btrfs(self): + mountpoint = tempfile.mkdtemp() try: with tempfile.NamedTemporaryFile(delete=False) as f: os.truncate(f.fileno(), 1024 * 1024 * 1024) - self.image = f.name + image = f.name except Exception as e: - os.rmdir(self.mountpoint) + os.rmdir(mountpoint) raise e if os.path.exists('../../mkfs.btrfs'):
@@ -72,19 +76,24 @@ class BtrfsTestCase(unittest.TestCase): else: mkfs = 'mkfs.btrfs' try: - subprocess.check_call([mkfs, '-q', self.image]) - subprocess.check_call(['mount', '-o', 'loop', '--', self.image, self.mountpoint]) + subprocess.check_call([mkfs, '-q', image]) + subprocess.check_call(['mount', '-o', 'loop', '--', image, mountpoint]) except Exception as e: - os.remove(self.image) - os.rmdir(self.mountpoint) + os.rmdir(mountpoint) + os.remove(image) raise e + self._mountpoints.append((mountpoint, image)) + return mountpoint, image + + def setUp(self): + self.mountpoint, self.image = self.mount_btrfs() + def tearDown(self): - try: - subprocess.check_call(['umount', self.mountpoint]) - finally: - os.remove(self.image) - os.rmdir(self.mountpoint) + for mountpoint, image in self._mountpoints: + subprocess.call(['umount', '-R', mountpoint]) + os.rmdir(mountpoint) + os.remove(image) @staticmethod def path_or_fd(path, open_flags=os.O_RDONLY):
Signature · 1 line
2.19.1
PATCH 7/10 libbtrfsutil: relax the privileges of subvolume_info()
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
Attempt to use the BTRFS_IOC_GET_SUBVOL_INFO ioctl (added in kernel 4.18) for subvolume_info() if not root. Also, rename get_subvolume_info_root() -> get_subvolume_info_privileged() for consistency with further changes.
This is based on a patch from Misono Tomohiro.
--- libbtrfsutil/btrfsutil.h | 4 +- libbtrfsutil/errors.c | 2 + libbtrfsutil/python/tests/test_subvolume.py | 42 ++++++++++++---- libbtrfsutil/subvolume.c | 53 +++++++++++++++++++-- 4 files changed, 89 insertions(+), 12 deletions(-)
libbtrfsutil/btrfsutil.h +3 -1
@@ -63,6 +63,7 @@ enum btrfs_util_error {BTRFS_UTIL_ERROR_SYNC_FAILED ,BTRFS_UTIL_ERROR_START_SYNC_FAILED ,BTRFS_UTIL_ERROR_WAIT_SYNC_FAILED , + BTRFS_UTIL_ERROR_GET_SUBVOL_INFO_FAILED, };/**
@@ -266,7 +267,8 @@ struct btrfs_util_subvolume_info {* to check whether the subvolume exists; %BTRFS_UTIL_ERROR_SUBVOLUME_NOT_FOUND * will be returned if it does not. * -* This requires appropriate privilege (CAP_SYS_ADMIN). +* This requires appropriate privilege (CAP_SYS_ADMIN) unless @id is zero and +* the kernel supports BTRFS_IOC_GET_SUBVOL_INFO (kernel >= 4.18). * * Return: %BTRFS_UTIL_OK on success, non-zero error code on failure. */
libbtrfsutil/errors.c +2 -0
@@ -45,6 +45,8 @@ static const char * const error_messages[] = { [BTRFS_UTIL_ERROR_SYNC_FAILED] = "Could not sync filesystem", [BTRFS_UTIL_ERROR_START_SYNC_FAILED] = "Could not start filesystem sync", [BTRFS_UTIL_ERROR_WAIT_SYNC_FAILED] = "Could not wait for filesystem sync", + [BTRFS_UTIL_ERROR_GET_SUBVOL_INFO_FAILED ] = + "Could not get subvolume information with BTRFS_IOC_GET_SUBVOL_INFO", }; PUBLIC const char *btrfs_util_strerror(enum btrfs_util_error err)
libbtrfsutil/python/tests/test_subvolume.py +34 -8
83 lines not on this page · raw message
libbtrfsutil/subvolume.c +50 -3
83 lines not on this page · raw message
Signature · 1 line
2.19.1
PATCH 8/10 libbtrfsutil: relax the privileges of subvolume iterator
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
We can use the new BTRFS_IOC_GET_SUBVOL_ROOTREF and BTRFS_IOC_INO_LOOKUP_USER ioctls to allow non-root users to list subvolumes.
This is based on a patch from Misono Tomohiro but takes a different approach (mainly, this approach is more similar to the existing tree search approach).
--- libbtrfsutil/btrfsutil.h | 15 +- libbtrfsutil/errors.c | 6 + libbtrfsutil/python/tests/test_subvolume.py | 180 +++++++--- libbtrfsutil/subvolume.c | 354 +++++++++++++++++--- 4 files changed, 450 insertions(+), 105 deletions(-)
libbtrfsutil/btrfsutil.h +13 -2
@@ -64,6 +64,9 @@ enum btrfs_util_error {BTRFS_UTIL_ERROR_START_SYNC_FAILED ,BTRFS_UTIL_ERROR_WAIT_SYNC_FAILED ,BTRFS_UTIL_ERROR_GET_SUBVOL_INFO_FAILED , +BTRFS_UTIL_ERROR_GET_SUBVOL_ROOTREF_FAILED , +BTRFS_UTIL_ERROR_INO_LOOKUP_USER_FAILED , + BTRFS_UTIL_ERROR_FS_INFO_FAILED, };/**
@@ -507,6 +510,12 @@ struct btrfs_util_subvolume_iterator;* @flags: Bitmask of BTRFS_UTIL_SUBVOLUME_ITERATOR_* flags. * @ret: Returned iterator. * +* Subvolume iterators require appropriate privilege (CAP_SYS_ADMIN) unless @top +* is zero and the kernel supports BTRFS_IOC_GET_SUBVOL_ROOTREF and +* BTRFS_IOC_INO_LOOKUP_USER (kernel >= 4.18). In this case, subvolumes which +* cannot be accessed (e.g., due to permissions or other mounts) will be +* skipped. +* * The returned iterator must be freed with * btrfs_util_destroy_subvolume_iterator(). *
@@ -555,7 +564,8 @@ int btrfs_util_subvolume_iterator_fd(const struct btrfs_util_subvolume_iterator* Must be freed with free(). * @id_ret: Returned subvolume ID. May be %NULL. * -* This requires appropriate privilege (CAP_SYS_ADMIN). +* This requires appropriate privilege (CAP_SYS_ADMIN) for kernels < 4.18. See +* btrfs_util_create_subvolume_iterator(). * * Return: %BTRFS_UTIL_OK on success, %BTRFS_UTIL_ERROR_STOP_ITERATION if there * are no more subvolumes, non-zero error code on failure.
@@ -574,7 +584,8 @@ enum btrfs_util_error btrfs_util_subvolume_iterator_next(struct btrfs_util_subvo* This convenience function basically combines * btrfs_util_subvolume_iterator_next() and btrfs_util_subvolume_info(). * -* This requires appropriate privilege (CAP_SYS_ADMIN). +* This requires appropriate privilege (CAP_SYS_ADMIN) for kernels < 4.18. See +* btrfs_util_create_subvolume_iterator(). * * Return: See btrfs_util_subvolume_iterator_next(). */
libbtrfsutil/errors.c +6 -0
@@ -47,6 +47,12 @@ static const char * const error_messages[] = { [BTRFS_UTIL_ERROR_WAIT_SYNC_FAILED] = "Could not wait for filesystem sync", [BTRFS_UTIL_ERROR_GET_SUBVOL_INFO_FAILED ] = "Could not get subvolume information with BTRFS_IOC_GET_SUBVOL_INFO", + [BTRFS_UTIL_ERROR_GET_SUBVOL_ROOTREF_FAILED ] = + "Could not get rootref information with BTRFS_IOC_GET_SUBVOL_ROOTREF", + [BTRFS_UTIL_ERROR_INO_LOOKUP_USER_FAILED ] = + "Could not resolve subvolume path with BTRFS_IOC_INO_LOOKUP_USER", + [BTRFS_UTIL_ERROR_FS_INFO_FAILED ] = + "Could not get filesystem information", }; PUBLIC const char *btrfs_util_strerror(enum btrfs_util_error err)
libbtrfsutil/python/tests/test_subvolume.py +125 -55
206 lines not on this page · raw message
libbtrfsutil/subvolume.c +306 -48
466 lines not on this page · raw message
Signature · 1 line
2.19.1
PATCH 9/10 libbtrfsutil: bump version to 1.1.0
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
With the previous few fixes and features, we should bump the minor version.
--- libbtrfsutil/btrfsutil.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
libbtrfsutil/btrfsutil.h +1 -1
@@ -26,7 +26,7 @@#include <sys/time.h> #define BTRFS_UTIL_VERSION_MAJOR 1 -#define BTRFS_UTIL_VERSION_MINOR 0 +#define BTRFS_UTIL_VERSION_MINOR 1#define BTRFS_UTIL_VERSION_PATCH 0#ifdef __cplusplus
Signature · 1 line
2.19.1
PATCH 10/10 libbtrfsutil: document API in README
In reply to Omar SandovalFrom: Omar Sandoval <[email protected]>
btrfsutil.h and the Python docstrings are thorough, but I've gotten a couple of requests for a high-level overview of the available interfaces and example usages. Add them to README.md.
--- libbtrfsutil/README.md | 422 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 421 insertions(+), 1 deletion(-)
libbtrfsutil/README.md +421 -1
@@ -6,6 +6,425 @@ the LGPL. libbtrfsutil provides interfaces for a subset of the operations offered by the `btrfs` command line utility. It also includes official Python bindings (Python 3 only). +API Overview +------------ + +This section provides an overview of the interfaces available in libbtrfsutil +as well as example usages. Detailed documentation for the C API can be found in +[`btrfsutil.h`](btrfsutil.h). Detailed documentation for the Python bindings is +available with `pydoc3 btrfsutil` or in the interpreter: + +``` +>>> import btrfsutil +>>> help(btrfsutil) +``` + +Many functions in the C API have a variant taking a path and a variant taking a +file descriptor. The latter has the same name as the former with an `_fd` +suffix. The Python bindings for these functions can take a path, a file object, +or a file descriptor. + +Error handling is omitted from most of these examples for brevity. Please +handle errors in production code. + +### Error Handling + +In the C API, all functions that can return an error return an `enum +btrfs_util_error` and set `errno`. `BTRFS_UTIL_OK` (zero) is returned on +success. `btrfs_util_strerror()` converts an error code to a string +description suitable for human-friendly error reporting. + +```c +enum btrfs_util_err err; + +err = btrfs_util_sync("/"); +if (err) + fprintf("stderr, %s: %m\n", btrfs_util_strerror(err)); +``` + +In the Python bindings, functions may raise a `BtrfsUtilError`, which is a +subclass of `OSError` with an added `btrfsutilerror` error code member. Error +codes are available as `ERROR_*` constants. + +```python +try: + btrfsutil.sync('/') +except btrfsutil.BtrfsUtilError as e: + print(e, file=sys.stderr) +``` + +### Filesystem Operations + +There are several operations which act on the entire filesystem. + +#### Sync + +Btrfs can commit all caches for a specific filesystem to disk. + +`btrfs_util_sync()` forces a sync on the filesystem containing the given file +and waits for it to complete. + +`btrfs_wait_sync()` waits for a previously started transaction to complete. The +transaction is specified by ID, which may be zero to indicate the current +transaction. + +`btrfs_start_sync()` asynchronously starts a sync and returns a transaction ID +which can then be passed to `btrfs_wait_sync()`. + +```c +uint64_t transid; +btrfs_util_sync("/"); +btrfs_util_start_sync("/", &transid); +btrfs_util_wait_sync("/", &transid); +btrfs_util_wait_sync("/", 0); +``` + +```python +btrfsutil.sync('/') +transid = btrfsutil.start_sync('/') +btrfsutil.wait_sync('/', transid) +btrfsutil.wait_sync('/') # equivalent to wait_sync('/', 0) +``` + +All of these functions have `_fd` variants. + +The equivalent `btrfs-progs` command is `btrfs filesystem sync`. + +### Subvolume Operations + +Functions which take a file and a subvolume ID can be used in two ways. If zero +is given as the subvolume ID, then the given file is used as the subvolume. +Otherwise, the given file can be any file in the filesystem, and the subvolume +with the given ID is used. + +#### Subvolume Information + +`btrfs_util_is_subvolume()` returns whether a given file is a subvolume. + +`btrfs_util_subvolume_id()` returns the ID of the subvolume containing the +given file. + +```c +enum btrfs_util_error err; +err = btrfs_util_is_subvolume("/subvol"); +if (!err) + printf("Subvolume\n"); +else if (err == BTRFS_UTIL_ERROR_NOT_BTRFS || err == BTRFS_UTIL_ERROR_NOT_SUBVOLUME) + printf("Not subvolume\n"); +uint64_t id; +btrfs_util_subvolume_id("/subvol", &id); +``` + +```python +if btrfsutil.is_subvolume('/subvol'): + print('Subvolume') +else: + print('Not subvolume') +id_ = btrfsutil.subvolume_id('/subvol') +``` + +`btrfs_util_subvolume_path()` returns the path of the subvolume with the given +ID relative to the filesystem root. This requires `CAP_SYS_ADMIN`. The path +must be freed with `free()`. + +```c +char *path; +btrfs_util_subvolume_path("/", 256, &path); +free(path); +btrfs_util_subvolume_path("/subvol", 0, &path); +free(path); +``` + +```python +path = btrfsutil.subvolume_path('/', 256) +path = btrfsutil.subvolume_path('/subvol') # equivalent to subvolume_path('/subvol', 0) +``` + +`btrfs_util_subvolume_info()` returns information (including ID, parent ID, +UUID) about a subvolume. In the C API, this is returned as a `struct +btrfs_util_subvolume_info`. The Python bindings use a `SubvolumeInfo` object. + +This requires `CAP_SYS_ADMIN` unless the given subvolume ID is zero and the +kernel supports the `BTRFS_IOC_GET_SUBVOL_INFO` ioctl (added in 4.18). + +The equivalent `btrfs-progs` command is `btrfs subvolume show`. + +```c +struct btrfs_util_subvolume_info info; +btrfs_util_subvolume_info("/", 256, &info); +btrfs_util_subvolume_info("/subvol", 0, &info); +``` + +```python +info = btrfsutil.subvolume_info('/', 256) +info = btrfsutil.subvolume_info('/subvol') # equivalent to subvolume_info('/subvol', 0) +``` + +All of these functions have `_fd` variants. + +#### Enumeration + +An iterator interface is provided for enumerating subvolumes on a filesystem. +In the C API, a `struct btrfs_util_subvolume_iterator` is initialized by +`btrfs_util_create_subvolume_iterator()`, which takes a top subvolume to +enumerate under and flags. Currently, the only flag is to specify post-order +traversal instead of the default pre-order. This function has an `_fd` variant. + +`btrfs_util_destroy_subvolume_iterator()` must be called to free a previously +created `struct btrfs_util_subvolume_iterator`. + +`btrfs_util_subvolume_iterator_fd()` returns the file descriptor opened by +`btrfs_util_create_subvolume_iterator()` which can be used for other functions. + +`btrfs_util_subvolume_iterator_next()` returns the path (relative to the top +subvolume that the iterator was created with) and ID of the next subvolume. +`btrfs_util_subvolume_iterator_next_info()` returns a `struct +btrfs_subvolume_info` instead of the ID. It is slightly more efficient than +doing separate `btrfs_util_subvolume_iterator_next()` and +`btrfs_util_subvolume_info()` calls if the subvolume information is needed. The +path returned by these functions must be freed with `free()`. When there are no +more subvolumes, they return `BTRFS_UTIL_ERROR_STOP_ITERATION`. + +```c +struct btrfs_util_subvolume_iterator *iter; +enum btrfs_util_error err; +char *path; +uint64_t id; +struct btrfs_util_subvolume_info info; + +btrfs_util_create_subvolume_iterator("/", 256, 0, &iter); +/* + * This is just an example use-case for btrfs_util_subvolume_iterator_fd(). It + * is not necessary. + */ +btrfs_util_sync_fd(btrfs_util_subvolume_iterator_fd(iter)); +while (!(err = btrfs_util_subvolume_iterator_next(iter, &path, &id))) { + printf("%" PRIu64 " %s\n", id, path); + free(path); +} +btrfs_util_destroy_subvolume_iterator(iter); + +btrfs_util_create_subvolume_iterator("/subvol", 0, + BTRFS_UTIL_SUBVOLUME_ITERATOR_POST_ORDER, + &iter); +while (!(err = btrfs_util_subvolume_iterator_next_info(iter, &path, &info))) { + printf("%" PRIu64 " %" PRIu64 " %s\n", info.id, info.parent_id, path); + free(path); +} +btrfs_util_destroy_subvolume_iterator(iter); +``` + +The Python bindings provide this interface as an iterable `SubvolumeIterator` +class. It should be used as a context manager to ensure that the underlying +file descriptor is closed. Alternatively, it has a `close()` method for closing +explicitly. It also has a `fileno()` method to get the underlying file +descriptor. + +```python +with btrfsutil.SubvolumeIterator('/', 256) as it: + # This is just an example use-case for fileno(). It is not necessary. + btrfsutil.sync(it.fileno()) + for path, id_ in it: + print(id_, path) + +it = btrfsutil.SubvolumeIterator('/subvol', info=True, post_order=True) +try: + for path, info in it: + print(info.id, info.parent_id, path) +finally: + it.close() +``` + +This interface requires `CAP_SYS_ADMIN` unless the given top subvolume ID is +zero and the kernel supports the `BTRFS_IOC_GET_SUBVOL_ROOTREF` and +`BTRFS_IOC_INO_LOOKUP_USER` ioctls (added in 4.18). In the unprivileged case, +subvolumes which cannot be accessed are skipped. + +The equivalent `btrfs-progs` command is `btrfs subvolume list`. + +#### Creation + +`btrfs_util_create_subvolume()` creates a new subvolume at the given path. The +subvolume can be created asynchronously and inherit from quota groups +(qgroups). + +Qgroups to inherit are specified with a `struct btrfs_util_qgroup_inherit`, +which is created by `btrfs_util_create_qgroup_inherit()` and freed by +`btrfs_util_destroy_qgroup_inherit()`. Qgroups are added with +`btrfs_util_qgroup_inherit_add_group()`. The list of added groups can be +retrieved with `btrfs_util_qgroup_inherit_get_groups()`; note that the returned +array does not need to be freed and is no longer valid when the `struct +btrfs_util_qgroup_inherit` is freed. + +The Python bindings provide a `QgroupInherit` class. It has an `add_group()` +method and a `groups` member, which is a list of ints. + +```c +btrfs_util_create_subvolume("/subvol2", 0, NULL, NULL); + +uint64_t async_transid; +btrfs_util_create_subvolume("/subvol2", 0, &async_transid, NULL); +btrfs_util_wait_sync("/", async_transid); + +struct btrfs_util_qgroup_inherit *qgroups; +btrfs_util_create_qgroup_inherit(0, &qgroups); +btrfs_util_qgroup_inherit_add_group(&qgroups, 256); +btrfs_util_create_subvolume("/subvol2", 0, NULL, qgroups); +btrfs_util_destroy_qgroup_inherit(qgroups); +``` + +```python +btrfsutil.create_subvolume('/subvol2') + +async_transid = btrfsutil.create_subvolume('/subvol2', async_=True) +btrfsutil.wait_sync('/', async_transid) + +qgroups = btrfsutil.QgroupInherit() +qgroups.add_group(256) +btrfsutil.create_subvolume('/subvol2', qgroup_inherit=qgroups) +``` + +The C API has an `_fd` variant which takes a name and a file descriptor +referring to the parent directory. + +The equivalent `btrfs-progs` command is `btrfs subvolume create`. + +#### Snapshotting + +Snapshots are created with `btrfs_util_create_snapshot()`, which takes a source +path, a destination path, and flags. It can also be asynchronous and inherit +from quota groups; see [subvolume creation](#Creation). + +Snapshot creation can be recursive, in which case subvolumes underneath the +subvolume being snapshotted will also be snapshotted onto the same location in +the new snapshot (note that this is implemented in userspace non-atomically and +has the same capability requirements as a [subvolume iterator](#Enumeration)). +The newly created snapshot can also be read-only, but not if doing a recursive +snapshot. + +```c +btrfs_util_create_snapshot("/subvol", "/snapshot", 0, NULL, NULL); +btrfs_util_create_snapshot("/nested_subvol", "/nested_snapshot", + BTRFS_UTIL_CREATE_SNAPSHOT_RECURSIVE, NULL, NULL); +btrfs_util_create_snapshot("/subvol", "/rosnapshot", + BTRFS_UTIL_CREATE_SNAPSHOT_READ_ONLY, NULL, NULL); +``` + +```python +btrfsutil.create_snapshot('/subvol', '/snapshot') +btrfsutil.create_snapshot('/nested_subvol', '/nested_snapshot', recursive=True) +btrfsutil.create_snapshot('/subvol', '/rosnapshot', read_only=True) +``` + +The C API has two `_fd` variants. `btrfs_util_create_snapshot_fd()` takes the +source subvolume as a file descriptor. `btrfs_util_create_snapshot_fd2()` takes +the source subvolume as a file descriptor and the destination as a name and +parent file descriptor. + +The equivalent `btrfs-progs` command is `btrfs subvolume snapshot`. + +#### Deletion + +`btrfs_util_delete_subvolume()` takes a subvolume to delete and flags. This +requires `CAP_SYS_ADMIN` if the filesystem was not mounted with +`user_subvol_rm_allowed`. Deletion may be recursive, in which case all +subvolumes beneath the given subvolume are deleted before the given subvolume +is deleted. This is implemented in user-space non-atomically and has the same +capability requirements as a [subvolume iterator](#Enumeration). + +```c +btrfs_util_delete_subvolume("/subvol", 0); +btrfs_util_delete_subvolume("/nested_subvol", + BTRFS_UTIL_DELETE_SUBVOLUME_RECURSIVE); +``` + +```python +btrfsutil.delete_subvolume('/subvol') +btrfsutil.delete_subvolume('/nested_subvol', recursive=True) +``` + +The C API has an `_fd` variant which takes a name and a file descriptor +referring to the parent directory. + +The equivalent `btrfs-progs` command is `btrfs subvolume delete`. + +#### Deleted Subvolumes + +Btrfs lazily cleans up deleted subvolumes. `btrfs_util_deleted_subvolumes()` +returns an array of subvolume IDs which have been deleted but not yet cleaned +up. The returned array should be freed with `free()`. +```c +uint64_t *ids; +size_t n; /* Number of returned IDs. */ +btrfs_util_deleted_subvolumes("/", &ids, &n); +free(ids); +``` + +The Python binding returns a list of ints. + +```python +ids = btrfsutil.deleted_subvolumes('/') +``` + +This function also has an `_fd` variant. It requires `CAP_SYS_ADMIN`. + +The closest `btrfs-progs` command is `btrfs subvolume sync`, which waits for +deleted subvolumes to be cleaned up. + +#### Read-Only Flag + +Subvolumes can be set to read-only. `btrfs_util_get_subvolume_read_only()` +returns whether a subvolume is read-only. +`btrfs_util_set_subvolume_read_only()` sets the read-only flag to the desired +value. + +```c +bool read_only; +btrfs_util_get_subvolume_read_only("/subvol", &read_only); +btrfs_util_set_subvolume_read_only("/subvol", true); +btrfs_util_set_subvolume_read_only("/subvol", false); +``` + +```python +read_only = btrfsutil.get_subvolume_read_only('/subvol') +btrfsutil.set_subvolume_read_only('/subvol', True) +btrfsutil.set_subvolume_read_only('/subvol', False) +``` + +Both of these functions have `_fd` variants. + +The equivalent `btrfs-progs` commands are `btrfs property get` and `btrfs +property set` with the `ro` property. + +#### Default Subvolume + +The default subvolume of a filesystem is the subvolume which is mounted when no +`subvol` or `subvolid` mount option is passed. + +`btrfs_util_get_default_subvolume()` gets the ID of the default subvolume for +the filesystem containing the given file. + +`btrfs_util_set_default_subvolume()` sets the default subvolume. + +```c +uint64_t id; +btrfs_util_get_default_subvolume("/", &id); +btrfs_util_set_default_subvolume("/", 256); +btrfs_util_set_default_subvolume("/subvol", 0); +``` + +```python +id = btrfsutil.get_default_subvolume('/') +btrfsutil.set_default_subvolume('/', 256) +btrfsutil.set_default_subvolume('/subvol') # equivalent to set_default_subvolume('/subvol', 0) +``` + +Both of these functions have an `_fd` variant. They both require +`CAP_SYS_ADMIN`. + +The equivalent `btrfs-progs` commands are `btrfs subvolume get-default` and +`btrfs subvolume set-default`. + Development -----------
@@ -24,7 +443,8 @@ release of btrfs-progs). A few guidelines: -* All interfaces must be documented in `btrfsutil.h` using the kernel-doc style +* All interfaces must be documented in this README and in `btrfsutil.h` using + the kernel-doc style * Error codes should be specific about what _exactly_ failed * Functions should have a path and an fd variant whenever possible * Spell out terms in function names, etc. rather than abbreviating whenever
Signature · 1 line
2.19.1
Quoted 16 lines · Omar Sandoval
From: Omar Sandoval <[email protected]>
Hi,
This series contains my backlog of libbtrfsutil changes which I've been collecting over the past few weeks.
Patches 1-4 are fixes. Patches 5-6 add functionality to the unit tests which is needed for patches 7-8. Patches 7-8 add support for the unprivileged ioctls added in Linux 4.18; more on those below. Patch 9 bumps the library version. Patch 10 adds documentation for the available API along with examples.
Patches 7-8 are based on Misono Tomohiro's previous patch series [1], with a few important changes.
Quoted 10 lines
- Both
subvolume_info()andcreate_subvolume_iterator()now have unit tests for the unprivileged case.
- Both no longer explicitly check that top == 0 in the unprivileged case, since that will already fail with a clear permission error.
- Unprivileged iteration is much simpler: it usesopenat()instead offchdir()and is based more closely on the original tree search variant. This fixes a bug in post-order iteration in Misono's version.
- Unprivileged iteration does _not_ support passing in a non-subvolume path; if this behavior is desired, I'd like it to be a separate change with an explicit flag.
Series merged to devel, thanks. I've added link from the main README now that there's the API documentation.
The test-libbtrfsutil is missing from the travis CI for some reason, I was about to add it. So far the testing environment does not provide 'umount' that knows about '-R' so the tests fail. I'll have a look if there's a newer base image provided, otherwise a workaround would be necessary.
As for the unprivileged subvolume listing ioctls, the functionality in the util library is self-contained and the interface is up to you to design properly, so this does not depend on the 'btrfs subvolume list' command. That one has unfortunately not bubbled high enough in my todo.
Quoted 28 lines · David Sterba
Quoted 16 lines · Omar Sandoval
From: Omar Sandoval <[email protected]>
Hi,
This series contains my backlog of libbtrfsutil changes which I've been collecting over the past few weeks.
Patches 1-4 are fixes. Patches 5-6 add functionality to the unit tests which is needed for patches 7-8. Patches 7-8 add support for the unprivileged ioctls added in Linux 4.18; more on those below. Patch 9 bumps the library version. Patch 10 adds documentation for the available API along with examples.
Patches 7-8 are based on Misono Tomohiro's previous patch series [1], with a few important changes.
Quoted 10 lines
- Both
subvolume_info()andcreate_subvolume_iterator()now have unit tests for the unprivileged case.
- Both no longer explicitly check that top == 0 in the unprivileged case, since that will already fail with a clear permission error.
- Unprivileged iteration is much simpler: it usesopenat()instead offchdir()and is based more closely on the original tree search variant. This fixes a bug in post-order iteration in Misono's version.
- Unprivileged iteration does _not_ support passing in a non-subvolume path; if this behavior is desired, I'd like it to be a separate change with an explicit flag.
Quoted 1 line
Series merged to devel, thanks.
Thanks!
Quoted 2 lines
I've added link from the main README now
that there's the API documentation.
Ah, great idea.
Quoted 5 lines
The test-libbtrfsutil is missing from the travis CI for some reason, I was about to add it. So far the testing environment does not provide 'umount' that knows about '-R' so the tests fail. I'll have a look if there's a newer base image provided, otherwise a workaround would be necessary.
It looks like it was added to util-linux in v2.23 back in 2013. Or maybe the base image uses busybox? I believe that umount from busybox doesn't have -R.
Quoted 4 lines
As for the unprivileged subvolume listing ioctls, the functionality in the util library is self-contained and the interface is up to you to design properly, so this does not depend on the 'btrfs subvolume list' command. That one has unfortunately not bubbled high enough in my todo.
That comment is mostly for Misono, since the original version had that functionality, probably for the subvolume list command.
Quoted 46 lines
-----Original Message-----
From: David Sterba [mailto:[email protected]]
Sent: Tuesday, November 27, 2018 1:18 AM
To: Omar Sandoval <[email protected]>
Cc: [email protected]; [email protected]; Misono, Tomohiro <[email protected]>
Subject: Re: [PATCH 00/10] btrfs-progs: my libbtrfsutil patch queueQuoted 10 lines · Omar Sandoval
From: Omar Sandoval <[email protected]>
Hi,
This series contains my backlog of libbtrfsutil changes which I've been collecting over the past few weeks.
Patches 1-4 are fixes. Patches 5-6 add functionality to the unit tests which is needed for patches 7-8. Patches 7-8 add support for the unprivileged ioctls added in Linux 4.18; more on those below. Patch
9
Quoted 6 lines
bumps the library version. Patch 10 adds documentation for the available API along with examples.
Patches 7-8 are based on Misono Tomohiro's previous patch series [1], with a few important changes.
Quoted 10 lines
- Both
subvolume_info()andcreate_subvolume_iterator()now have unit tests for the unprivileged case.
- Both no longer explicitly check that top == 0 in the unprivileged case, since that will already fail with a clear permission error.
- Unprivileged iteration is much simpler: it usesopenat()instead offchdir()and is based more closely on the original tree search variant. This fixes a bug in post-order iteration in Misono's version.
- Unprivileged iteration does _not_ support passing in a non-subvolume path; if this behavior is desired, I'd like it to be a separate change with an explicit flag.Series merged to devel, thanks. I've added link from the main README now that there's the API documentation.
The test-libbtrfsutil is missing from the travis CI for some reason, I was about to add it. So far the testing environment does not provide 'umount' that knows about '-R' so the tests fail. I'll have a look if there's a newer base image provided, otherwise a workaround would be necessary.
Quoted 4 lines
As for the unprivileged subvolume listing ioctls, the functionality in the util library is self-contained and the interface is up to you to design properly, so this does not depend on the 'btrfs subvolume list' command. That one has unfortunately not bubbled high enough in my todo.
Hello,
I missed the mails and am sorry for late response.
As mentioned libbtrfsuitl and other progs are mostly independent, the patches I submitted (once in devel with your review and some modification) can be cleanly applied to this version of libbtrfsutil.
I will resend them for easier review/comment.
Thanks,
Misono