Skip to contentSkip to conversation

libfsimage

PATCH In review John Levon 3 participants 8 messages Oct 27, 2006 – Oct 31, 2006

lore
Share & tools
lore

John Levon

author
#0 lore raw

PATCH 1/2 libfsimage

These two patches re-work pygrub to use a generic C library for reading filesystem images. An immediate benefit is that there's no longer any external dependencies, so all the filesystems always 'just work'.

Slightly longer term, providing a stable C API for filesystem support adds a considerable amount of future proofing. It's easy for FS developers to add support for their FS in a stable manner. Additionally, it's worth noting that the facility provided by libfsimage is a generic one, and could prove useful in circumstances outside pygrub.

It also adds Solaris UFS support. The current plugins are all based on slightly modified Grub legacy code (so far it's taken about an hour to get a new FS from the grub code working).

Ideally, this library would be provided by the OS vendors themselves, as there's nothing Xen-specific to it, but for now I've provided it as a patch to xen-unstable.

Comments?

regards,
john

# HG changeset patch
# User [email protected]
# Date 1161961324 25200
# Node ID 5ff4ddea38e2cd18a10cb66040879ab7eca6797a
# Parent eff2bb014e46dd9745ff30e43a5c35cf908d8c58
Add libfsimage, a C library for reading files from filesystem images. Initial support is provided for Solaris UFS, ext2, and reiserfs.

tools/Makefile +1 -0
@@ -18,6 +18,7 @@ SUBDIRS-y += xenstat
 SUBDIRS-y += xenstat
 SUBDIRS-$(CONFIG_Linux) += libaio
 SUBDIRS-$(CONFIG_Linux) += blktap
+SUBDIRS-y += libfsimage
 
 # These don't cross-compile
 ifeq ($(XEN_COMPILE_ARCH),$(XEN_TARGET_ARCH))
tools/libfsimage/Makefile new file +12 -0
new file mode 100644
@@ -0,0 +1,12 @@
+XEN_ROOT = ../..
+include $(XEN_ROOT)/tools/Rules.mk
+
+SUBDIRS-y = common ufs ext2fs reiserfs
+
+.PHONY: all
+all install clean:
+	@set -e; for subdir in $(SUBDIRS-y); do \
+		$(MAKE) -C $$subdir $@; \
+	done
+
+distclean: clean
tools/libfsimage/Rules.mk new file +32 -0
new file mode 100644
@@ -0,0 +1,32 @@
+include $(XEN_ROOT)/tools/Rules.mk
+
+DEPS = .*.d
+
+CFLAGS += -I$(XEN_ROOT)/tools/libfsimage/common/ -Werror -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -Wp,-MD,.$(@F).d
+LDFLAGS += -L../common/
+
+PIC_OBJS := $(patsubst %.c,%.opic,$(LIB_SRCS-y))
+
+FSDIR-$(CONFIG_Linux) = $(LIBDIR)/fs/$(FS)
+FSDIR-$(CONFIG_SunOS)-x86_64 = lib/fs/$(FS)/64
+FSDIR-$(CONFIG_SunOS)-x86_32 = lib/fs/$(FS)/
+FSDIR-$(CONFIG_SunOS) = $(FSDIR-$(CONFIG_SunOS)-$(XEN_TARGET_ARCH))
+FSDIR = $(FSDIR-y)
+
+FSLIB = fsimage.so
+
+.PHONY: fs-all
+fs-all: $(FSLIB)
+
+.PHONY: fs-install
+fs-install: fs-all
+	$(INSTALL_DIR) $(DESTDIR)/usr/$(FSDIR)
+	$(INSTALL_PROG) $(FSLIB) $(DESTDIR)/usr/$(FSDIR)
+
+$(FSLIB): $(PIC_OBJS)
+	$(CC) $(CFLAGS) $(LDFLAGS) $(SHLIB_CFLAGS) -o $@ $^ -lfsimage
+
+clean distclean:
+	rm -f $(PIC_OBJS) $(FSLIB)
+
+-include $(DEPS)
tools/libfsimage/common/Makefile new file +45 -0

45 lines not on this page · raw message

tools/libfsimage/common/fsimage.c new file +140 -0

140 lines not on this page · raw message

tools/libfsimage/common/fsimage.h new file +52 -0

52 lines not on this page · raw message

tools/libfsimage/common/fsimage_grub.c new file +277 -0

277 lines not on this page · raw message

tools/libfsimage/common/fsimage_grub.h new file +92 -0

92 lines not on this page · raw message

tools/libfsimage/common/fsimage_plugin.c new file +208 -0

208 lines not on this page · raw message

tools/libfsimage/common/fsimage_plugin.h new file +64 -0

64 lines not on this page · raw message

tools/libfsimage/common/fsimage_priv.h new file +62 -0

62 lines not on this page · raw message

tools/libfsimage/common/mapfile new file +34 -0
new file mode 100644
@@ -0,0 +1,34 @@
+libfsimage.so.1.1 {
+	global:
+		fsi_open_fsimage;
+		fsi_close_fsimage;
+		fsi_file_exists;
+		fsi_open_file;
+		fsi_close_file;
+		fsi_read_file;
+		fsi_pread_file;
+
+		fsip_fs_set_data;
+		fsip_fs_free;
+		fsip_file_alloc;
+		fsip_file_free;
+		fsip_fs;
+		fsip_fs_data;
+		fsip_file_data;
+
+		fsig_init;
+		fsig_devread;
+		fsig_substring;
+		fsig_fs_buf;
+		fsig_file_alloc;
+		fsig_file_buf;
+		fsig_filepos;
+		fsig_filemax;
+		fsig_int1;
+		fsig_int2;
+		fsig_errnum;
+		fsig_disk_read_junk;
+
+	local:
+		*;
+};
tools/libfsimage/ext2fs/Makefile new file +13 -0
new file mode 100644
@@ -0,0 +1,13 @@
+XEN_ROOT = ../../..
+
+LIB_SRCS-y = fsys_ext2fs.c
+
+FS = ext2fs
+
+.PHONY: all
+all: fs-all
+
+.PHONY: install
+install: fs-install
+
+include $(XEN_ROOT)/tools/libfsimage/Rules.mk
tools/libfsimage/ext2fs/fsys_ext2fs.c new file +804 -0

804 lines not on this page · raw message

tools/libfsimage/reiserfs/Makefile new file +13 -0
new file mode 100644
@@ -0,0 +1,13 @@
+XEN_ROOT = ../../..
+
+LIB_SRCS-y = fsys_reiserfs.c
+
+FS = reiserfs
+
+.PHONY: all
+all: fs-all
+
+.PHONY: install
+install: fs-install
+
+include $(XEN_ROOT)/tools/libfsimage/Rules.mk
tools/libfsimage/reiserfs/fsys_reiserfs.c new file +1254 -0

1,145 lines not on this page · raw message

tools/libfsimage/ufs/Makefile new file +13 -0
new file mode 100644

Lines of this file are not shown. View the raw message.

tools/libfsimage/ufs/fsys_ufs.c new file +276 -0

276 lines not on this page · raw message

tools/libfsimage/ufs/ufs.h new file +228 -0

228 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Part of the diff is not shown here. View the raw message.

author
#1 lore raw

PATCH 2/2 libfsimage - use for pygrub

# HG changeset patch
# User [email protected]
# Date 1161961342 25200
# Node ID c2a9cf36d6850ca07b069a43d588241f7b3d640e # Parent 5ff4ddea38e2cd18a10cb66040879ab7eca6797a Use libfsimage for reading filesystem images.

tools/pygrub/setup.py +10 -31
@@ -5,46 +5,25 @@ import sys
 
 extra_compile_args  = [ "-static-libgcc", "-fno-strict-aliasing", "-Wall", "-Werror" ]
 
-fsys_mods = []
-fsys_pkgs = []
+XEN_ROOT = "../.."
 
-if os.path.exists("/usr/include/ext2fs/ext2_fs.h"):
-    ext2defines = []
-    cc = new_compiler()
-    cc.add_library("ext2fs")
-    if hasattr(cc, "has_function") and cc.has_function("ext2fs_open2"):
-        ext2defines.append( ("HAVE_EXT2FS_OPEN2", None) )
-    else:
-        sys.stderr.write("WARNING: older version of e2fsprogs installed, not building full\n")
-        sys.stderr.write("         disk support for ext2.\n")
-        
-    ext2 = Extension("grub.fsys.ext2._pyext2",
-                     extra_compile_args = extra_compile_args,
-                     libraries = ["ext2fs"],
-                     define_macros = ext2defines,
-                     sources = ["src/fsys/ext2/ext2module.c"])
-    fsys_mods.append(ext2)
-    fsys_pkgs.append("grub.fsys.ext2")
+fsimage = Extension("fsimage",
+    extra_compile_args = extra_compile_args,
+    include_dirs = [ XEN_ROOT + "/tools/libfsimage/common/" ],
+    library_dirs = [ XEN_ROOT + "/tools/libfsimage/common/" ],
+    libraries = ["fsimage"],
+    sources = ["src/fsimage/fsimage.c"])
 
-if os.path.exists("/usr/include/reiserfs/reiserfs.h"):
-    reiser = Extension("grub.fsys.reiser._pyreiser",
-                     extra_compile_args = extra_compile_args,
-                     libraries = ["reiserfs"],
-                     sources = ["src/fsys/reiser/reisermodule.c"])
-    fsys_mods.append(reiser)
-    fsys_pkgs.append("grub.fsys.reiser")
+pkgs = [ 'grub' ]
 
-pkgs = ['grub', 'grub.fsys']
-pkgs.extend(fsys_pkgs)
 setup(name='pygrub',
       version='0.3',
       description='Boot loader that looks a lot like grub for Xen',
       author='Jeremy Katz',
       author_email='[email protected]',
       license='GPL',
-      package_dir={'grub': 'src'},
+      package_dir={'grub': 'src', 'fsimage': 'src'},
       scripts = ["src/pygrub"],
       packages=pkgs,
-      ext_modules = fsys_mods
+      ext_modules = [ fsimage ]
       )
-               
tools/pygrub/src/fsimage/fsimage.c new file +299 -0

299 lines not on this page · raw message

tools/pygrub/src/fsys/__init__.py deleted +0 -64

64 lines not on this page · raw message

tools/pygrub/src/fsys/ext2/__init__.py deleted +0 -38
deleted file mode 100644
@@ -1,38 +0,0 @@
-# Copyright 2005 Red Hat, Inc.
-# Jeremy Katz <[email protected]>
-#
-# This software may be freely redistributed under the terms of the GNU
-# general public license.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program; if not, write to the Free Software
-# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
-#
-
-from grub.fsys import register_fstype, FileSystemType
-from _pyext2 import *
-
-import os, struct
-
-class Ext2FileSystemType(FileSystemType):
-    def __init__(self):
-        FileSystemType.__init__(self)
-        self.name = "ext2"
-
-    def sniff_magic(self, fn, offset = 0):
-        fd = os.open(fn, os.O_RDONLY)
-        os.lseek(fd, offset, 0)
-        buf = os.read(fd, 2048)
-        os.close(fd)        
-        if len(buf) > 1082 and \
-               struct.unpack("<H", buf[1080:1082]) == (0xef53,):
-            return True
-        return False
-
-    def open_fs(self, fn, offset = 0):
-        if not self.sniff_magic(fn, offset):
-            raise ValueError, "Not an ext2 filesystem"
-        return Ext2Fs(fn, offset = offset)
-
-register_fstype(Ext2FileSystemType())
-
tools/pygrub/src/fsys/ext2/ext2module.c deleted +0 -387

387 lines not on this page · raw message

tools/pygrub/src/fsys/ext2/test.py deleted +0 -15
deleted file mode 100644
@@ -1,15 +0,0 @@
-#!/usr/bin/python
-
-
-import _pyext2
-import struct, os, sys
-
-fs = _pyext2.Ext2Fs("test.img")
-
-f = fs.open_file("/boot/vmlinuz-2.6.11-1.1177_FC4")
-buf = f.read()
-o = open("vmlinuz", "wb+")
-o.write(buf)
-o.close()
-
-f.close()
tools/pygrub/src/fsys/reiser/__init__.py deleted +0 -40
deleted file mode 100644
@@ -1,40 +0,0 @@
-# 
-# Copyright (C) 2005 Nguyen Anh Quynh <[email protected]>
-#
-# This software may be freely redistributed under the terms of the GNU
-# general public license.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program; if not, write to the Free Software
-# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
-#
-
-from grub.fsys import register_fstype, FileSystemType
-from _pyreiser import *
-
-import os
-
-FSMAGIC2 = 'ReIsEr2'
-FSMAGIC3 = 'ReIsEr3'
-
-class ReiserFileSystemType(FileSystemType):
-    def __init__(self):
-        FileSystemType.__init__(self)
-        self.name = "reiser"
-
-    def sniff_magic(self, fn, offset = 0):
-        fd = os.open(fn, os.O_RDONLY)
-        os.lseek(fd, 0x10000, 0)
-        buf = os.read(fd, 0x40)
-        os.close(fd)
-        if len(buf) == 0x40 and (buf[0x34:0x3B] in [FSMAGIC2, FSMAGIC3]) :
-            return True
-        return False
-
-    def open_fs(self, fn, offset = 0):
-        if not self.sniff_magic(fn, offset):
-            raise ValueError, "Not a reiserfs filesystem"
-        return ReiserFs(fn)
-
-register_fstype(ReiserFileSystemType())
-
tools/pygrub/src/fsys/reiser/reisermodule.c deleted +0 -345

345 lines not on this page · raw message

tools/pygrub/src/pygrub +6 -17
@@ -22,8 +22,8 @@ import getopt
 
 sys.path = [ '/usr/lib/python' ] + sys.path
 
+import fsimage
 import grub.GrubConf
-import grub.fsys
 
 PYGRUB_VER = 0.5
 
@@ -313,25 +313,21 @@ class Grub:
                 raise RuntimeError, "Unable to find active partition on disk"
 
         # open the image and read the grub config
-        fs = None
-        for fstype in grub.fsys.fstypes.values():
-            if fstype.sniff_magic(fn, offset):
-                fs = fstype.open_fs(fn, offset)
-                break
+        fs = fsimage.open(fn, offset)
 
         if fs is not None:
             grubfile = None
             for f in ("/boot/grub/menu.lst", "/boot/grub/grub.conf",
                       "/grub/menu.lst", "/grub/grub.conf"):
-                if fs.file_exist(f):
+                if fs.file_exists(f):
                     grubfile = f
                     break
             if grubfile is None:
                 raise RuntimeError, "we couldn't find grub config file in the image provided."
             f = fs.open_file(grubfile)
             buf = f.read()
-            f.close()
-            fs.close()
+            del f
+            del fs
             # then parse the grub config
             self.cf.parse(buf)
         else:
@@ -511,14 +507,7 @@ if __name__ == "__main__":
             raise RuntimeError, "Unable to find active partition on disk"
 
     # read the kernel and initrd onto the hostfs
-    fs = None
-    for fstype in grub.fsys.fstypes.values():
-        if fstype.sniff_magic(file, offset):
-            fs = fstype.open_fs(file, offset)
-            break
-
-    if fs is None:
-        raise RuntimeError, "Unable to open filesystem"
+    fs = fsimage.open(file, offset)
 
     kernel = fs.open_file(img.kernel[1],).read()
     (tfd, fn) = tempfile.mkstemp(prefix="vmlinuz.", dir="/var/lib/xen")
#2 lore raw
In reply to John Levon

Hi,

Quoted 3 lines · John Levon

These two patches re-work pygrub to use a generic C library for reading filesystem images. An immediate benefit is that there's no longer any external dependencies, so all the filesystems always 'just work'.

The downside is that it no longer picks up updates, fixes etc. from the official upstream support for libext2fs / libreiserfs, so we're just duplicating the places where filesystem support needs to be maintained.

If xen packages are properly built, filesystems will Just Work anyway.

Quoted 2 lines

Slightly longer term, providing a stable C API for filesystem support adds a considerable amount of future proofing.

That I can agree with. But I'd rather see that API done as a wrapper around existing upstream filesystem libraries, rather than forking fs userspace support entirely.

--Stephen

author
#3 lore raw
Quoted 8 lines · Stephen C. Tweedie
Quoted 3 lines · John Levon

These two patches re-work pygrub to use a generic C library for reading filesystem images. An immediate benefit is that there's no longer any external dependencies, so all the filesystems always 'just work'.

The downside is that it no longer picks up updates, fixes etc. from the official upstream support for libext2fs / libreiserfs, so we're just duplicating the places where filesystem support needs to be maintained.

You mean upstream grub, not upstream libraries. It's not hard to sync against grub. When grub2 is up and running, it should be truly trivial, as no source changes should be required; the FS drivers there were developed in userspace, so it's not even a 'fork'.

Quoted 1 line

If xen packages are properly built, filesystems will Just Work anyway.

On what operating system would that be?

Quoted 6 lines
Quoted 2 lines

Slightly longer term, providing a stable C API for filesystem support adds a considerable amount of future proofing.

That I can agree with. But I'd rather see that API done as a wrapper around existing upstream filesystem libraries, rather than forking fs userspace support entirely.

Vendors are free to write such plugins if they'd prefer it that way. As it is with using grub code, you get UFS support for free, we get ext2 and reiserfs, and it's trivially extendible to anything that grub needs to boot...

regards
john

#4 lore raw

Hi,

Quoted 7 lines · John Levon
Quoted 5 lines · Stephen C. Tweedie
Quoted 3 lines · John Levon

These two patches re-work pygrub to use a generic C library for reading filesystem images. An immediate benefit is that there's no longer any external dependencies, so all the filesystems always 'just work'.

Quoted 5 lines
Quoted 3 lines

The downside is that it no longer picks up updates, fixes etc. from the official upstream support for libext2fs / libreiserfs, so we're just duplicating the places where filesystem support needs to be maintained.

You mean upstream grub, not upstream libraries.

No, existing pygrub uses upstream libext2fs and libreiserfs, which are the official maintained fs libraries, not extracted from grub. It's that relationship that I don't want to lose.

Having the same drivers in multiple places is just an unnecessary maintenance burden. It is somewhat justified in grub, which is a boot loader and which by definition has to run before the full fs is up and running. But pygrub has no such excuse.

Quoted 3 lines
Quoted 1 line

If xen packages are properly built, filesystems will Just Work anyway.

On what operating system would that be?

On any OSes which can represent the relationship that the xen package requires libext2fs and libreiserfs. You missed out the quote that I was replying to:

"An immediate benefit is that there's no longer any external dependencies, so all the filesystems always 'just work'.

and I was pointing out that dealing with external dependencies is a packaging issue, not an excuse for copying that external functionality into the Xen tree.

Quoted 8 lines
Quoted 3 lines

That I can agree with. But I'd rather see that API done as a wrapper around existing upstream filesystem libraries, rather than forking fs userspace support entirely.

Vendors are free to write such plugins if they'd prefer it that way. As it is with using grub code, you get UFS support for free, we get ext2 and reiserfs, and it's trivially extendible to anything that grub needs to boot...

Vendors are free to write extensions to pygrub already --- reiserfs and ext2fs are already supported in such a way, through a modular python interface.

I'd really far rather see a UFS module added for pygrub which uses an external libUFS, rather than moving all fs-reading code into the xen tree. where I just don't think it belongs.

--Stephen

author
#5 lore raw
Quoted 10 lines · Stephen C. Tweedie
Quoted 1 line

You mean upstream grub, not upstream libraries.

No, existing pygrub uses upstream libext2fs and libreiserfs, which are the official maintained fs libraries, not extracted from grub. It's that relationship that I don't want to lose.

Having the same drivers in multiple places is just an unnecessary maintenance burden. It is somewhat justified in grub, which is a boot loader and which by definition has to run before the full fs is up and running. But pygrub has no such excuse.

I do not see much of a maintenance burden, and it does apparently solve a number of problems with Linux distros and those libraries as has been noted before. Much more importantly, it provides a simple way to extend beyond just Linux FS's on Linux, both shorter term (e.g. ZFS) and in the long term (random fs's on random OS types).

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

If xen packages are properly built, filesystems will Just Work anyway.

On what operating system would that be?

On any OSes which can represent the relationship that the xen package requires libext2fs and libreiserfs.

Which don't "just work" on Solaris afaik, for example.

Quoted 3 lines

and I was pointing out that dealing with external dependencies is a packaging issue, not an excuse for copying that external functionality into the Xen tree.

I would absolutely like to see libfsimage be vendor-supplied eventually. This would solve all of these issues of needing certain revisions of libext2fs, etc. The vendor can just ship what they need to. It's the primary reason that I've gone this way. The existing grub-based implementations are just what we're using; vendors are free to choose other methods.

As I said it's provided as a patch to the Xen tree right now for convenience's sake.

Quoted 3 lines

Vendors are free to write extensions to pygrub already --- reiserfs and ext2fs are already supported in such a way, through a modular python interface.

I don't think python is perfect for providing a future-proof API/ABI in this case, never mind that it forces all users to be python-based.

regards,
john

#6 lore raw
In reply to John Levon
Quoted 3 lines · John Levon

These two patches re-work pygrub to use a generic C library for reading filesystem images. An immediate benefit is that there's no longer any external dependencies, so all the filesystems always 'just work'.

This is a big step in the right direction. Especially for pygrub, using the same fs codebase that normal grub does seems like the right thing to do.

Quoted 5 lines

It also adds Solaris UFS support. The current plugins are all based on slightly modified Grub legacy code (so far it's taken about an hour to get a new FS from the grub code working).

From a quick glance through, the changes you've made to the grub fs code

seem to be mostly eliminating grub's world of static block-access variables and functions. Could that be abstracted out into a wrapper file that would let any grub filesystem code be used verbatim? That would reduce the maintenance load even further (and give us FAT and ISO9660 for free).

Alternatively, how far away from stable enough are the grub2 drivers?

Cheers,

Tim.

author
#7 lore raw
Quoted 10 lines · Tim Deegan
Quoted 5 lines

It also adds Solaris UFS support. The current plugins are all based on slightly modified Grub legacy code (so far it's taken about an hour to get a new FS from the grub code working).

From a quick glance through, the changes you've made to the grub fs code

seem to be mostly eliminating grub's world of static block-access variables and functions. Could that be abstracted out into a wrapper file that would let any grub filesystem code be used verbatim? That would reduce the maintenance load even further (and give us FAT and ISO9660 for free).

That was my initial aim, but translating the FSYS_BUF define into the file-local buffer seemed non-trivial. And I had to make some changes to allow for 64-bit usage; a lot of the grub legacy code is not 64-bit clean :/

I would expect FAT and iso9660 (and ZFS) to be just easy as as the others though, around an hour's work to fix and test...

Quoted 1 line

Alternatively, how far away from stable enough are the grub2 drivers?

I spoke to the grub devs; they're still not recommending people use grub2. reiserfs support isn't even integrated and has only just had patches posted, and no-one's ported ZFS yet...

regards,
john