Announcing btrfs-gui

Over the last few weeks, I've been playing with a foolish idea,
mostly triggered by a cluster of people being confused by btrfs's free space reporting (df vs btrfs fi df vs btrfs fi show). I also wanted an excuse, and some code, to mess around in the depths of the FS data structures.

Like all silly ideas, this one got a bit out of hand, and seems to
have turned into something vaguely useful. I'm therefore pleased to announce the first major public release of btrfs-gui[1]: a point-and- click tool for managing btrfs filesystems.

The tool currently can scan for and list btrfs filesystems and the
volumes they live on. It can show the allocation and usage of data in a selected filesystem, categorised by use, replication, and device. It can show and manipulate subvolumes and snapshots: creation, deletion, and setting the default. For those with servers to manage, it also has the ability to ssh into a remote machine, and manage its filesystems remotely (so you don't have to have X installed on your servers just to use btrfs-gui on them).

You can get the latest version from git[2] (or gitweb[3]), or
tarball download of the sources from [4]. To install and run the GUI, you will need python3 and the python tk libraries (package python3-tk on my Debian system). The root helper component (which can be installed independently on an X-less server) will run under python2 or python3, depending on how it's installed. Installation instructions can be found on the main web page, and in the README file.

Unless the traffic gets too high-volume, or unless someone
important objects, I'm going to suggest that bug reports should go to this list for now (cc'd me, if you like). Note that this isn't an "offical" btrfs project -- it's just something I knocked together on my own.

Finally, I'd like to thank David Sterba for testing the pre-release
versions, reporting bugs, and making many good suggestions for improvements. Any deviation from his instructions is entirely my fault. :)

Hugo.

[1] http://carfax.org.uk/btrfs-gui/
[2] http://git.darksatanic.net/repo/btrfs-gui.git/
[3] http://git.darksatanic.net/cgi/gitweb.cgi?p=btrfs-gui.git;a=summary
[4] http://carfax.org.uk/node/79

=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
      --- Great oxymorons of the world, no. 7: The Simple Truth ---      

e

n

o

e

t

Some comments:
(1) Currently it needs to be run from the directory where it's downloaded, even after a "python3 setup.py install". When run from other directory, it bails with

Traceback (most recent call last):
File "/usr/local/bin/btrfs-gui", line 5, in <module>
btrfsgui.main.main()
File "/usr/local/lib/python3.2/dist-packages/btrfsgui/main.py", line
24, in main
subproc =3D init_root_process(options)
File "/usr/local/lib/python3.2/dist-packages/btrfsgui/sudo.py", line
31, in init_root_process
stdin=3Dsubprocess.PIPE, stdout=3Dsubprocess.PIPE)
File "/usr/lib/python3.2/subprocess.py", line 736, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.2/subprocess.py", line 1330, in _execute_child
raise child_exception_type(errno_num, err_msg)
OSError: [Errno 2] No such file or directory: './btrfs-gui-helper'

Is this intentional?

(2) When showing space usage for a single-device FS, selecting "Show unallocated space as raw space", why is the top and bottom graph different? Shouldn't it be the same, since there's only one device?

(3) Not directly related to btrfs-gui, but I've been wondering what's the correct way to SHOW the current default subvolume?

--=20
=46ajar
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

No, and will be fixed later today. I forsee an emergency 0.2.1
coming shortly. :)

Good question. I shall investigate what's going on.

The btrfs-progs tools don't have a way of doing this. It's
relatively easy to do: use the tree-search ioctl to search for a DIR_ITEM key in the tree of tree roots with the name "default". The corresponding FS tree is the default subvolume. It's just not implemented yet.

Thanks for the feedback.

Hugo.

=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
              --- Welcome to Rivendell,  Mr Anderson... ---              

[snip]

OK, it's fixed in git in the stable-v0.2 branch. Unless anyone else
reports something that needs fixing over the weekend, I'll tag it as 0.2.1 on Sunday and roll another release tarball.

(The fix is actually pretty ugly, and has some poor UX in it for
one case, but I've run out of brain this evening, and can't face the shell hackery necessary to do it nicely right now.)

OK, on investigation and reflection, it shouldn't be identical,
because metadata is DUP. The per-disk displays show actual physical disk usage; the filesystem display at the top shows unique data. Therefore, the top display will show half the amount of metadata than the bottom.

Hugo.

=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
   --- Comic Sans goes into a bar,  and the barman says, "We don't ---   
                         serve your type here."                          

Excerpts from Hugo Mills's message of 2011-06-01 19:20:58 -0400:

This is really interesting. I'm updating my local install of python and friends so it can run properly. But I like the idea and will check it out.

-chris

On 02-06-11 01:20, Hugo Mills wrote:

*snip*

*snip*

I seem to have stumbled upon a bug.
During install, Ubuntu gives the user the option to use btrfs as the root filesystem. It automatically creates two subvolumes, @ (for /) and @home (for /home, only if no other partition is used for /home), like this:

mount | grep btrfs
/dev/sda6 on / type btrfs (rw,subvol=@)
/dev/sda6 on /home type btrfs (rw,subvol=@home)
/dev/sda6 on /tmp/btrfs-gui-kl6zx1/12624/eba4f1e9-5b55-4e14-abb8-5a3cf3625c56 type btrfs (rw,subvolid=0)

When gtrfs-gui scans for filesystems, it gives the error Root helper exception: b'@' is not JSON serializable

This is the relevant console output:

Helper: found label None, UUID eba4f1e9-5b55-4e14-abb8-5a3cf3625c56
Helper: found dev 1 = /dev/sda6
Helper: Created private directory /tmp/btrfs-gui-kl6zx1
Helper: Mounted filesystem UUID=eba4f1e9-5b55-4e14-abb8-5a3cf3625c56 at /tmp/btrfs-gui-kl6zx1/12624/eba4f1e9-5b55-4e14-abb8-5a3cf3625c56 Traceback (most recent call last):
File "/usr/local/lib/python3.2/dist-packages/btrfsgui/hlp/helper.py",
line 40, in main
COMMANDS[command](parameters)
File "/usr/local/lib/python3.2/dist-packages/btrfsgui/hlp/subvol.py",
line 126, in sv_list
sys.stdout.write(json.dumps(res))
File "/usr/lib/python3.2/json/__init__.py", line 224, in dumps
return _default_encoder.encode(obj)
File "/usr/lib/python3.2/json/encoder.py", line 188, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python3.2/json/encoder.py", line 246, in iterencode
return _iterencode(o, 0)
File "/usr/lib/python3.2/json/encoder.py", line 170, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: b'@' is not JSON serializable

Kind regards,
Amedee Van Gasse
[email protected]

[snip]

Mmm... That could be fun. :)

Thanks for the bug report. I hope I'll have time to look into this
next week.

Hugo.

=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
   --- The English language has the mot juste for every occasion. ---    

meone

to

n

this

Looks like a unicode screwup; python 3 doesn't do implicit unicode conversions anymore, so passing a bytestring to the json module isn't supported. The solution will be something along the lines of "subvol.decode('ascii')" (or possibly utf-8 if you're willing to silently break on the admittedly obscure case of a non-ascii subvolume name in an installation using an encoding other than utf-8). --
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

This is a nice little tool. The one suggestion that I have is that it display the actual chunks and where they are located. It seems that right now it uses the same ioctl that btrfs fi df uses, and that just gets the total combined size for all chunks of a given type. It would be nice if the gui actually parsed the chunk tree and showed each individual chunk with lines showing how they are mapped to the physical disks.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I managed to cobble together the following patches to implement this. The one thing that I still don't like is that the radio knob is in the data replication and allocation box, instead of the volumes box. My python and tkinter skills are too lacking to figure out how to move it down there.

Phillip Susi (3):
Changed volume_df() to return all chunks with their offsets Update UsageDisplay to be capable of displaying all chunks by
position
Add radio knob to show space by position or combined

 btrfsgui/gui/usagedisplay.py |   98
++++++++++++++++++++++++++++--------------
 btrfsgui/hlp/size.py         |   18 +++----
 2 files changed, 73 insertions(+), 43 deletions(-)

- --
1.7.5.4

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk7MSIoACgkQJ4UciIs+XuLyjQCeI4m7+u75R863B2RY3hkFELbP iJ8AoJwzVdiYZqgE1tXvHEOHz+gciDgj
=dfd1
-----END PGP SIGNATURE-----

Thanks for this. I hope I'll have time to look at it next week.

Hugo.

=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
   --- The English language has the mot juste for every occasion. ---    

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

volume_df() used to return a structure containing a dictionary named "usage" that contained 3 chunks, keyed by their usage type ( sys, meta, data ). I changed this to an array named "chunks" that contains one entry for every chunk found on the disk. Each chunk still is a dictionary that contains flags, size, and used, but now also contains voffset and poffset entries giving their starting offset relative to the filesystem and disk respectively. The poffset is intended to be used to show the chunk at the correct position in the disk graph, and the voffset is intended to allow correlation with the filesystem graph.
- ---

 btrfsgui/gui/usagedisplay.py |   14 ++++++++++----
 btrfsgui/hlp/size.py         |   18 ++++++++----------
 2 files changed, 18 insertions(+), 14 deletions(-)

index ccfc148..e6ae9b4 100644

- --- a/btrfsgui/gui/usagedisplay.py
+++ b/btrfsgui/gui/usagedisplay.py
@@ -347,13 +347,19 @@ class UsageDisplay(Frame, Requester):
max_space = 0
for dev in self.fs["vols"]:
rv, text, obj = self.request("vol_df", self.fs["uuid"], dev["id"])
- - dev["usage"] = obj
+ dev["vol_df"] = obj
if obj["size"] > max_space:
max_space = obj["size"]

y = 4
for i, dev in enumerate(self.fs["vols"]):
- - obj = dev["usage"]
+ # Combine chunks with same flags
+ chunks = {}
+ for chunk in dev["vol_df"]["chunks"]:
+ if chunk["flags"] in chunks:
+ chunks[chunk["flags"]]["size"] += chunk["size"]
+ chunks[chunk["flags"]]["used"] += chunk["used"]
+ else: chunks[chunk["flags"]] = chunk
frame = LabelFrame(self.per_disk,
text=dev["path"])
frame.grid(sticky=N+S+E+W)
@@ -371,8 +377,8 @@ class UsageDisplay(Frame, Requester):
bbox = self.per_disk.bbox(container)
y = bbox[3] + 4
raw_free += self.create_usage_box(canvas,
- - obj["usage"].values(),
- - size=obj["size"],
+ chunks.values(),
+ size=dev["vol_df"]["size"],
max_size=max_space)
self.per_disk.configure(
scrollregion=(0, 0,

index 0ec98c3..5ac89f4 100644

- --- a/btrfsgui/hlp/size.py
+++ b/btrfsgui/hlp/size.py
@@ -69,7 +69,7 @@ def volume_df(params):
res["size"] = data[1] res["used"] = data[2] res["uuid"] = btrfs.format_uuid(data[12])
- - res["usage"] = {}
+ res["chunks"] = []

# Now, collect data on the block group types in use
last_offset = 0
@@ -124,20 +124,18 @@ def volume_df(params):
if header[2] != chunk_length:
raise HelperException("Chunk length inconsistent: chunk tree says {0} bytes, extent tree says {1} bytes".format(chunk_length, header[2]))
chunk_used = extent_info[0]
- -
- - if chunk_type not in res["usage"]:
- - res["usage"][chunk_type] = {
- - "flags": chunk_type,
- - "size": 0,
- - "used": 0,
- - }
- - res["usage"][chunk_type]["size"] += ext_length
# We have a total of chunk_used space used, out of

 				# chunk_length in this block group. So
 				# chunk_used/chunk_length is the proportion of the BG
 				# used. We multiply that by the length of the dev_extent
 				# to get the amount of space used in the dev_extent.
- -				res["usage"][chunk_type]["used"] += chunk_used * ext_length / chunk_length
+				res["chunks"].append({
+								"flags": chunk_type,
+								"size": ext_length,
+								"used": chunk_used * ext_length / chunk_length,
+								"voffset": chunk_offset,
+								"poffset": ext_offset,
+								})

sys.stdout.write(json.dumps(res))
sys.stdout.write("\n")
- --
1.7.5.4

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk7MSJMACgkQJ4UciIs+XuK8wwCghzZA+5DZGxSd3mnm5gY7pCs1 WncAn3nTtOsUswDaz2SC+Hxmp702/aQ2
=OpKV
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Previously the input_data to create_usage_box was assumed to be a list of 3 chunks, one of each type: data, meta, sys. Now the list can contain any number of entries and they will each be displayed. If the entries contain the key "offset", then they will be shown at the appropriate location based on that offset, with any gaps filled in by unused space ( they are thus assumed to be in order ). Without the offset key, they will be displayed in order, with no gaps.
- ---

 btrfsgui/gui/usagedisplay.py |   65 ++++++++++++++++++++++-------------------
 1 files changed, 35 insertions(+), 30 deletions(-)

index e6ae9b4..aff24da 100644

- --- a/btrfsgui/gui/usagedisplay.py
+++ b/btrfsgui/gui/usagedisplay.py
@@ -243,52 +243,48 @@ class UsageDisplay(Frame, Requester):
'size' or 'free' should be provided. If 'size' is set, the amount of free space computed is returned; otherwise the return value is arbitrary."""
- -
# Calculate the overall width of the box we are going to draw width = DF_BOX_WIDTH
if max_size is not None:
width = width * size / max_size
- -
+ box = SplitBox(orient=SplitBox.HORIZONTAL)
+ nextpos = 0
# Categorise the data
- - data = SplitBox(orient=SplitBox.VERTICAL)
- - meta = SplitBox(orient=SplitBox.VERTICAL)
- - sys = SplitBox(orient=SplitBox.VERTICAL)
- - freebox = SplitBox(orient=SplitBox.VERTICAL)
- - for bg_type in input_data:
- - repl = btrfs.replication_type(bg_type["flags"])
- - usage = btrfs.usage_type(bg_type["flags"])
+ for chunk in input_data:
+ if not "offset" in chunk:
+ chunk["offset"] = nextpos
+ if nextpos <= chunk["offset"]:
+ freesize = chunk["offset"] - nextpos
+ freebox = SplitBox(orient=SplitBox.VERTICAL)
+ freebox.append((freesize, { "fill": COLOUR_UNUSED }))
+ nextpos = chunk["offset"] + chunk["size"]
+ if size is not None:
+ size -= chunk["size"]
+ chunkbox = SplitBox(orient=SplitBox.VERTICAL)
+
+ repl = btrfs.replication_type(chunk["flags"])
+ usage = btrfs.usage_type(chunk["flags"])

if usage == "data":
- - destination = data
col = COLOURS[repl][0]
if usage == "meta":
- - destination = meta
col = COLOURS[repl][1]
if usage == "sys":
- - destination = sys
col = COLOURS[repl][2]

usedfree = SplitBox(orient=SplitBox.HORIZONTAL)
- - usedfree.append((bg_type["used"],
+ usedfree.append((chunk["used"],
{ "fill": col }))
- - usedfree.append((bg_type["size"]-bg_type["used"],
+ usedfree.append((chunk["size"]-chunk["used"],
{ "fill": col, "stripe": fade(col) }))
- - destination.append((usedfree.total, usedfree))
- - if size is not None:
- - size -= bg_type["size"]
+ chunkbox.append((usedfree.total, usedfree))
+ box.append((chunkbox.total, chunkbox))

- - if size is not None:
- - freebox.append((size, { "fill": COLOUR_UNUSED }))
- - elif free is not None:
+ if size is not None and nextpos < size:
+ free = size - nextpos
+ if free is not None:
+ freebox = SplitBox(orient=SplitBox.VERTICAL)
freebox.append((free, { "fill": COLOUR_UNUSED }))
- -
- - # total is our whole block
- - # *_total are the three main divisions
- - box = SplitBox(orient=SplitBox.HORIZONTAL)
- - box.append((sys.total, sys))
- - box.append((meta.total, meta))
- - box.append((data.total, data))
- - if size is not None or free is not None:
box.append((freebox.total, freebox))

box.set_position(DF_BOX_PADDING, DF_BOX_PADDING,
@@ -329,6 +325,15 @@ class UsageDisplay(Frame, Requester):

@ex_handler
def update_display(self):
+ def usage_sort(chunk):
+ usage = btrfs.usage_type(chunk["flags"])
+ if usage == "sys":
+ return 0
+ if usage == "meta":
+ return 1
+ if usage == "data":
+ return 2
+
if not self.stale:
return

@@ -377,7 +382,7 @@ class UsageDisplay(Frame, Requester):
bbox = self.per_disk.bbox(container)
y = bbox[3] + 4
raw_free += self.create_usage_box(canvas,
- - chunks.values(),
+ sorted(chunks.values(), key=usage_sort),
size=dev["vol_df"]["size"], max_size=max_space)
self.per_disk.configure(
@@ -389,4 +394,4 @@ class UsageDisplay(Frame, Requester):
kwargs = {}
if self.df_selection.get() == "raw":
kwargs["free"] = raw_free
- - self.create_usage_box(self.df_display, obj, **kwargs)
+ self.create_usage_box(self.df_display, sorted(obj, key=usage_sort), **kwargs)
- --
1.7.5.4

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk7MSJoACgkQJ4UciIs+XuInWQCgunltRDcrr+wVzlSUzVXiXEjw /nIAn2la0tdjlZdkSYcChg87Tw/JS0Ek
=8XRk
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

The previous method was to show chunks combined by type. This knob allows the user to choose to show each individual chunk in its actual position.
- ---

 btrfsgui/gui/usagedisplay.py |   35 ++++++++++++++++++++++++++++-------
 1 files changed, 28 insertions(+), 7 deletions(-)

index aff24da..fd8b1e2 100644

- --- a/btrfsgui/gui/usagedisplay.py
+++ b/btrfsgui/gui/usagedisplay.py
@@ -213,6 +213,22 @@ class UsageDisplay(Frame, Requester):
but.grid(row=2, column=1, sticky=W) self.df_selection.set("alloc")

+ self.df_byposition = StringVar()
+ Label(box, text="Show allocated space").grid(row=3,column=0)
+ but = Radiobutton(
+ box, text="Combined",
+ variable=self.df_byposition,
+ command=self.change_display,
+ value="combined")
+ but.grid(row=3, column=1, sticky=W)
+ but = Radiobutton(
+ box, text="Individual",
+ variable=self.df_byposition,
+ command=self.change_display,
+ value="individual")
+ but.grid(row=4, column=1, sticky=W)
+ self.df_byposition.set("combined")
+
frm = LabelFrame(self, text="Volumes")
frm.columnconfigure(0, weight=1)
frm.rowconfigure(0, weight=1)
@@ -359,12 +375,17 @@ class UsageDisplay(Frame, Requester):
y = 4
for i, dev in enumerate(self.fs["vols"]):
# Combine chunks with same flags
- - chunks = {}
- - for chunk in dev["vol_df"]["chunks"]:
- - if chunk["flags"] in chunks:
- - chunks[chunk["flags"]]["size"] += chunk["size"]
- - chunks[chunk["flags"]]["used"] += chunk["used"]
- - else: chunks[chunk["flags"]] = chunk
+ if self.df_byposition.get() == "combined":
+ chunks = {}
+ for chunk in dev["vol_df"]["chunks"]:
+ if chunk["flags"] in chunks:
+ chunks[chunk["flags"]]["size"] += chunk["size"]
+ chunks[chunk["flags"]]["used"] += chunk["used"]
+ else: chunks[chunk["flags"]] = chunk
+ chunks = sorted(chunks.values(), key=usage_sort)
+ else:
+ chunks = [{"size": x["size"], "used": x["used"], "flags": x["flags"],
+ "offset": x["poffset"]} for x in dev["vol_df"]["chunks"]]
frame = LabelFrame(self.per_disk,
text=dev["path"])
frame.grid(sticky=N+S+E+W)
@@ -382,7 +403,7 @@ class UsageDisplay(Frame, Requester):
bbox = self.per_disk.bbox(container)
y = bbox[3] + 4
raw_free += self.create_usage_box(canvas,
- - sorted(chunks.values(), key=usage_sort),
+ chunks,
size=dev["vol_df"]["size"], max_size=max_space)
self.per_disk.configure(
- --
1.7.5.4

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk7MSKIACgkQJ4UciIs+XuIx5QCfSHc4/8bPkQuiTGs0R3D6SyPK 6+cAn1n7HcLowmobT48hQc+iPjUqJEer
=Ljl6
-----END PGP SIGNATURE-----