Hello world, python-btrfs
Hi!
After playing around a bit for a few months with a bunch of proof-of-concept level scripts to be able to debug my btrfs file systems, the inevitable happened:
https://github.com/knorrie/python-btrfs/
Currently, the primary goal of this module is to be able to inspect the internals of an existing filesystem for educational purposes.
In the last week I started sorting out the puzzle pieces I already assembled, and put most of them together into a proper python library to query a btrfs filesystem.
The python module acts as a wrapper around the low level kernel calls and btrfs data structures, presenting them as python objects with interesting attributes and references to other objects.
Using these helpers, it should be fairly easy to reimplement the same functionality as for example btrfs fi df, btrfs fi show and btrfs inspect-internal provide.
For example:
import btrfs
fs = btrfs.FileSystem('/')
for space in fs.space_info():
print("{0}, {1}: total={2}, used={3}".format(
btrfs.utils.block_group_type_str(space.flags),
btrfs.utils.block_group_profile_str(space.flags),
btrfs.utils.pretty_size(space.total_bytes),
btrfs.utils.pretty_size(space.used_bytes)))results in:
Data, single: total=15.94GiB, used=6.30GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=768.00MiB, used=158.72MiB
GlobalReserve, single: total=64.00MiB, used=0.00B
The btrfs directory in the source is the python module which is to be imported as btrfs.
The examples directory contains example programs that use the library. For all functionality that's added in the lib, there must be an example program that uses it.
Currently, the code can look at devices, chunks, block groups with their usage and lists of data extents. I usually write quite elaborate git commit messages, so when looking through my commits or doing a git blame on code, you should (now and later) find a nice description about the hows whys etc of the code.
Why?!
* Because it's fun!
* Because I get sick of using horrible ducttape regex solutions to programmatically parse human readable output of external tools to implement monitoring tools~~!11one.
* Because I might to be able to help other people that also want to learn the internals of btrfs.
* Much more...
How?
* It's a python library, so the code will have a decent level of pythonicity. The ioctl search returns a generator, the search key is a nice object you can just increment etc...
* This is a work in progress. I can only add support for parts of btrfs that I understand myself first.
Moo!
Hans van Kranenburg
Signature · 2 lines
Hans van Kranenburg - System / Network Engineer T +31 (0)10 2760434 | [email protected] | www.mendix.com
Quoted 7 lines · Hans van Kranenburg
proof-of-concept level scripts to be able to debug my btrfs file systems, the inevitable happened:
https://github.com/knorrie/python-btrfs/
Currently, the primary goal of this module is to be able to inspect the internals of an existing filesystem for educational purposes.
Looks great, I like. Python is better language for prototyping various things, also more convenient to write one-shot scripts to do specific rescue tasks.
Quoted 3 lines
The python module acts as a wrapper around the low level kernel calls and btrfs data structures, presenting them as python objects with interesting attributes and references to other objects.
I guess it could be extended to read the filesystem objects directly from the image, so it's not only built around the search ioctl.
Quoted 3 lines
Why?!
* Because it's fun!
Best reason ever!
Quoted 7 lines
* Because I get sick of using horrible ducttape regex solutions to programmatically parse human readable output of external tools to implement monitoring tools~~!11one.
* Because I might to be able to help other people that also want to learn the internals of btrfs.
* Much more...
Quoted 7 lines
How?
* It's a python library, so the code will have a decent level of pythonicity. The ioctl search returns a generator, the search key is a nice object you can just increment etc...
* This is a work in progress. I can only add support for parts of btrfs that I understand myself first.
My question is whether it sounds like a good idea to host the python bindings inside btrfs-progs or not. I can imagine creating scripts to craft images for testing or verifying results of fsck, so it would be more convenient to keep them in one place (but using a git submodule instead would not be a big deal).
I'm mostly interested in the object representation of the filesystem objects and basic API around that, so if you have other plans with python-btrfs, we can at least share some of the code. My python-fu is not that strong so I'd rely on you or others to maintain that if you're interested.
Hi,
Quoted 10 lines · David Sterba
Quoted 7 lines · Hans van Kranenburg
proof-of-concept level scripts to be able to debug my btrfs file systems, the inevitable happened:
https://github.com/knorrie/python-btrfs/
Currently, the primary goal of this module is to be able to inspect the internals of an existing filesystem for educational purposes.
Looks great, I like.
Thanks, much appreciated!
Quoted 8 lines
Python is better language for prototyping various
things, also more convenient to write one-shot scripts to do specific rescue tasks.Quoted 3 lines
The python module acts as a wrapper around the low level kernel calls and btrfs data structures, presenting them as python objects with interesting attributes and references to other objects.
Quoted 8 lines
I guess it could be extended to read the filesystem objects directly from the image, so it's not only built around the search ioctl.
Quoted 3 lines
Why?!
* Because it's fun!
Best reason ever!
Moo!
Quoted 3 lines
Quoted 3 lines
* Because I get sick of using horrible ducttape regex solutions to programmatically parse human readable output of external tools to implement monitoring tools~~!11one.
This is actually the main reason I started it. Or was it being able to see where balance got the usage information from...
Quoted 6 lines
Quoted 6 lines
* Because I might to be able to help other people that also want to learn the internals of btrfs.
* Much more...How?
Quoted 8 lines
Quoted 5 lines
* It's a python library, so the code will have a decent level of pythonicity. The ioctl search returns a generator, the search key is a nice object you can just increment etc...
* This is a work in progress. I can only add support for parts of btrfs that I understand myself first.My question is whether it sounds like a good idea to host the python bindings inside btrfs-progs or not.
Whoa, hold your horses!
Quoted 4 lines
I can imagine creating scripts to
craft images for testing or verifying results of fsck, so it would be more convenient to keep them in one place (but using a git submodule instead would not be a big deal).
python-btrfs is just a library. You can just clone the git somewhere, start hacking on it and put it in your python path, or pip install or apt-get install python-btrfs etc...
Programs that use it can be anywhere. What I have in examples/ now should probably end up either in howto style documentation or in proper tests for the lib (which requires keeping known-state filesystem images to test on etc, exploding into a new level of whoa).
Quoted 5 lines
I'm mostly interested in the object representation of the filesystem objects and basic API around that, so if you have other plans with python-btrfs, we can at least share some of the code. My python-fu is not that strong so I'd rely on you or others to maintain that if you're interested.
Ok, I added a little TODO:
https://github.com/knorrie/python-btrfs/blob/master/TODO.md
There's a few things I want to do first at least, which are completing to support the full set of data structures using the kernel API, finishing to move existing code that I have lying around to use the new lib and a bit of extra info to help interested sysadmin/programmers to start using the library to peek around in their own filesystem a bit to see what's going on.
I haven't looked at how the btrfs-progs treat disks directly, but would certainly be interested in learning about that.
From what you say, I feel that you're thinking about not only looking around in filesystems, but having ways to manipulate them?
Can you provide a more concrete example of "craft images for testing" or related ideas about how this project could help during regular btrfs development?
Signature · 2 lines
Hans van Kranenburg - System / Network Engineer T +31 (0)10 2760434 | [email protected] | www.mendix.com
Quoted 8 lines · Hans van Kranenburg
Quoted 4 lines
I can imagine creating scripts to
craft images for testing or verifying results of fsck, so it would be more convenient to keep them in one place (but using a git submodule instead would not be a big deal).python-btrfs is just a library. You can just clone the git somewhere, start hacking on it and put it in your python path, or pip install or apt-get install python-btrfs etc...
So it sounds more appropriate to keep them separate.
Quoted 14 lines
Programs that use it can be anywhere. What I have in examples/ now should probably end up either in howto style documentation or in proper tests for the lib (which requires keeping known-state filesystem images to test on etc, exploding into a new level of whoa).
Quoted 5 lines
I'm mostly interested in the object representation of the filesystem objects and basic API around that, so if you have other plans with python-btrfs, we can at least share some of the code. My python-fu is not that strong so I'd rely on you or others to maintain that if you're interested.
Ok, I added a little TODO:
https://github.com/knorrie/python-btrfs/blob/master/TODO.md
Quoted 6 lines
There's a few things I want to do first at least, which are completing to support the full set of data structures using the kernel API, finishing to move existing code that I have lying around to use the new lib and a bit of extra info to help interested sysadmin/programmers to start using the library to peek around in their own filesystem a bit to see what's going on.
So this will be mostly ioctl based.
Quoted 2 lines
I haven't looked at how the btrfs-progs treat disks directly, but would certainly be interested in learning about that.
The core work lies in the b-tree item manipulation, free block allocation and maybe more.
Quoted 2 lines
From what you say, I feel that you're thinking about not only looking
around in filesystems, but having ways to manipulate them?
Yes. Initially it can be only in-place modification, like rewrite a specific member of an item and recalculate the block checksum. We can do that with btrfs-corrupt-block but the command line interface is not that great.
Quoted 3 lines
Can you provide a more concrete example of "craft images for testing" or related ideas about how this project could help during regular btrfs development?
With a good support of python interface to btrfs structures, we can eg. lookup specific extent items, swap some pointers and see how fsck survives. Writing that as a C code is possible, but IMO more tedious compared to python.
2016-06-18 0:01 GMT+02:00 Hans van Kranenburg <[email protected]>:
Quoted 8 lines
Hi!
After playing around a bit for a few months with a bunch of proof-of-concept level scripts to be able to debug my btrfs file systems, the inevitable happened:
Hi, I want just to inform you that I've submitted your module to Fedora.
Thanks for your work.