New image target (x86/x64): raw disk image

Hi list,

These 3 patches allow the building of a new target for x86 and x64. The "raw" image is a disk image, with a valid partition table and MBR, that can directly written to a disk and will boot, or the file is bootable in QEMU. Also cloud providers who use KVM (like Google) should accept this as their VM image format.

Regards,
Karoly

Changes v1 -> v2:
- package/util-linux: minimal wording
- fs/rawimg: separated script into new host package like mke2img
- package/mkrawimg: new host package

Karoly Kasza (3):
util-linux: enable host fdisk
package/mkrawimg: new package
fs/rawimg: new rootfs target

 fs/Config.in                                       |    1 +
 fs/rawimg/Config.in                                |   15 ++++++++
 fs/rawimg/rawimg.mk                                |   13 +++++++
 package/Config.in.host                             |    1 +
 package/mkrawimg/Config.in.host                    |   13 +++++++
 package/mkrawimg/mkrawimg                          |   38 ++++++++++++++++++++
 package/mkrawimg/mkrawimg.mk                       |   14 ++++++++
 package/util-linux/Config.in.host                  |    9 +++++
 .../util-linux/util-linux-004-enable-fdisk.patch   |   34 ++++++++++++++++++
 package/util-linux/util-linux.mk                   |    5 +++
 10 files changed, 143 insertions(+)
 create mode 100644 fs/rawimg/Config.in
 create mode 100644 fs/rawimg/rawimg.mk
 create mode 100644 package/mkrawimg/Config.in.host
 create mode 100755 package/mkrawimg/mkrawimg
 create mode 100644 package/mkrawimg/mkrawimg.mk
 create mode 100644 package/util-linux/util-linux-004-enable-fdisk.patch

1.7.10.4

Enable the building of fdisk utility for host.

---

Changes v1 -> v2:
- Wording.

 package/util-linux/Config.in.host                  |    9 ++++++
 .../util-linux/util-linux-004-enable-fdisk.patch   |   34 ++++++++++++++++++++
 package/util-linux/util-linux.mk                   |    5 +++
 3 files changed, 48 insertions(+)
 create mode 100644 package/util-linux/util-linux-004-enable-fdisk.patch

index f73dc43..65f4a87 100644
@@ -6,3 +6,12 @@ config BR2_PACKAGE_HOST_UTIL_LINUX
 	  Things like mkfs, mkswap, swapon, fdisk, mount, dmesg, etc...
 
 	  http://www.kernel.org/pub/linux/utils/util-linux/
+
+if BR2_PACKAGE_HOST_UTIL_LINUX
+
+config BR2_PACKAGE_HOST_UTIL_LINUX_FDISK
+	bool "fdisk"
+	help
+	  Enable compiling fdisk for the host
+
+endif

new file mode 100644
index 0000000..5faf1bb
@@ -0,0 +1,34 @@
+Add options to enable/disable libfdisk and fdisk
+
+This is needed to be able to just build an fdisk binary
+for the host package.
+
+Signed-off-by: Karoly Kasza <[email protected]>
+
+--- util-linux-2.25.1.orig/configure.ac	2014-10-11 13:47:52.000000000 +0200
++++ util-linux-2.25.1/configure.ac	2014-10-11 13:45:50.065076171 +0200
+@@ -820,12 +820,22 @@
+ dnl
+ dnl libfdisk is enabled at all times if possible
+ dnl
+-UL_BUILD_INIT([libfdisk], [check])
++# this behaviour changed for buildroot, along with fdisk
++AC_ARG_ENABLE([libfdisk],
++  AS_HELP_STRING([--disable-libfdisk], [do not build libfdisk]),
++  [], [UL_DEFAULT_ENABLE([libfdisk], [check])]
++)
++UL_BUILD_INIT([libfdisk])
+ UL_REQUIRES_BUILD([libfdisk], [libuuid])
+ UL_REQUIRES_BUILD([libfdisk], [libsmartcols])
+ AM_CONDITIONAL([BUILD_LIBFDISK], [test "x$build_libfdisk" = xyes])
+ 
+-UL_BUILD_INIT([fdisk], [check])
++
++AC_ARG_ENABLE([fdisk],
++  AS_HELP_STRING([--disable-fdisk], [do not build fdisk]),
++  [], [UL_DEFAULT_ENABLE([fdisk], [check])]
++)
++UL_BUILD_INIT([fdisk])
+ UL_REQUIRES_BUILD([fdisk], [libfdisk])
+ AM_CONDITIONAL([BUILD_FDISK], [test "x$build_fdisk" = xyes])
+ 

index dc10193..9f70a11 100644
@@ -97,6 +97,11 @@ HOST_UTIL_LINUX_CONF_OPTS += \
 	--disable-libblkid --disable-libmount \
 	--disable-all-programs --without-ncurses
 
+# Build host fdisk if selected
+ifeq ($(BR2_PACKAGE_HOST_UTIL_LINUX_FDISK),y)
+HOST_UTIL_LINUX_CONF_OPTS += --enable-fdisk --enable-libfdisk --enable-libsmartcols
+endif
+
 # Avoid building the tools if they are disabled since we can't install on
 # a per-directory basis.
 ifeq ($(BR2_PACKAGE_UTIL_LINUX_BINARIES),)

1.7.10.4

Karoly, All,

On 2014-12-09 17:03 +0100, Karoly Kasza spake thusly:

We've reviewed your patch series yesterday during the Patchwork cleanup session, and cam e to this conclusion about this patch: we think there is a better solution as proposed by Thierry Bultel in:

    http://patchwork.ozlabs.org/patch/419247/

The patch from Thierry is better because it is simpler and does not need patching util-linux.

The solution to build all or nothing is an acceptable trade-off for simplicity. Also, if later the need arise to install yet another host util-linux program, we'd have to add yet another option and further patch util-linux. The gain is not that important when compared to maintainability and simplicity.

So, we marked this patch as Rejected in Patchwork.

Thank you for contribution!

Regards,
Yann E. MORIN.

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

mkrawimg is a helper script like mke2img.
It a host package, useful to build a complete raw disk image, with grub in the MBR and one ext* partition. Creates a bootable raw disk image as a new rootfs image. Boots in QEMU or on bare metal after written (dd) to disk.

---

Changes v1 -> v2:
- Created a new package from the script in v1.
- Fdisk now specifies head and sector.
- Sector size of ext* image is calculated properly.

 package/Config.in.host          |    1 +
 package/mkrawimg/Config.in.host |   13 +++++++++++++
 package/mkrawimg/mkrawimg       |   38 ++++++++++++++++++++++++++++++++++++++
 package/mkrawimg/mkrawimg.mk    |   14 ++++++++++++++
 4 files changed, 66 insertions(+)
 create mode 100644 package/mkrawimg/Config.in.host
 create mode 100755 package/mkrawimg/mkrawimg
 create mode 100644 package/mkrawimg/mkrawimg.mk

index 0b7bc36..4fa58f8 100644
@@ -11,6 +11,7 @@ menu "Host utilities"
 	source "package/genpart/Config.in.host"
 	source "package/lpc3250loader/Config.in.host"
 	source "package/mke2img/Config.in.host"
+	source "package/mkrawimg/Config.in.host"
 	source "package/mtd/Config.in.host"
 	source "package/mtools/Config.in.host"
 	source "package/omap-u-boot-utils/Config.in.host"

new file mode 100644
index 0000000..f5083b4
@@ -0,0 +1,12 @@
+config BR2_PACKAGE_HOST_MKRAWIMG
+	bool "host mkrawimg"
+	select BR2_TARGET_GRUB
+	select BR2_TARGET_GRUB_FS_EXT2
+	select BR2_PACKAGE_HOST_UTIL_LINUX
+	select BR2_PACKAGE_HOST_UTIL_LINUX_FDISK
+	help
+	  Easily create raw disk images.
+
+	  This tool is bundled by, and specific to Buildroot.
+	  It does not make much sense, to use it without
+	  the "raw disk image" rootfs target.

new file mode 100755
index 0000000..dcc5d06
@@ -0,0 +1,38 @@
+#!/bin/bash
+
+# Buildroot tool to generate bootable raw disk image
+# GRUB in MBR, boot first primary partition (ext*)
+
+set -e
+
+BINARIES_DIR="$1"
+OUTPUT_IMG="$2"
+# Size of generated ext* fs in sectors
+SECTORS=`fdisk -l $BINARIES_DIR/rootfs.ext2 | grep $BINARIES_DIR/rootfs.ext2 | cut -d" " -f7`
+# Size of image with MBR and partition table (512 block size with fdisk image file)
+SECTOR_COUNT=$(($SECTORS+2048))
+
+# Create an empty file, final size
+dd if=/dev/zero of=$OUTPUT_IMG bs=512 count=$SECTOR_COUNT 2>/dev/null
+
+# Create a partition table - call our own fdisk in $PATH from host-util-linux
+fdisk -H 16 -S 63 $OUTPUT_IMG >/dev/null 2>/dev/null <<EOB
+n
+p
+1
+
+
+w
+EOB
+
+# Write the rootfs to the 1st partition
+dd if=$BINARIES_DIR/rootfs.ext2 of=$OUTPUT_IMG conv=notrunc bs=512 iflag=fullblock seek=2048 skip=0 2>/dev/null
+
+# Write MBR, grub is also called from $PATH
+grub --no-floppy --batch >/dev/null 2>/dev/null <<EOB
+device (hd0) $OUTPUT_IMG
+geometry (hd0)
+root (hd0,0)
+setup (hd0)
+quit
+EOB

new file mode 100644
index 0000000..858db79
@@ -0,0 +1,14 @@
+################################################################################
+#
+# mkrawimg
+#
+################################################################################
+
+HOST_MKRAWIMG_SOURCE =
+HOST_MKRAWIMG_DEPENDENCIES = host-util-linux grub
+
+define HOST_MKRAWIMG_INSTALL_CMDS
+	$(INSTALL) -D -m 0755 package/mkrawimg/mkrawimg $(HOST_DIR)/usr/bin/mkrawimg
+endef
+
+$(eval $(host-generic-package))

1.7.10.4

Rootfs target to call the new mkrawimg package.

---

Changes v1 -> v2:
- Removed script into new mkrawimg package.

 fs/Config.in        |    1 +
 fs/rawimg/Config.in |   15 +++++++++++++++
 fs/rawimg/rawimg.mk |   13 +++++++++++++
 3 files changed, 29 insertions(+)
 create mode 100644 fs/rawimg/Config.in
 create mode 100644 fs/rawimg/rawimg.mk

index 5853113..36961db 100644
@@ -8,6 +8,7 @@ source "fs/initramfs/Config.in"
 source "fs/iso9660/Config.in"
 source "fs/jffs2/Config.in"
 source "fs/romfs/Config.in"
+source "fs/rawimg/Config.in"
 source "fs/squashfs/Config.in"
 source "fs/tar/Config.in"
 source "fs/ubifs/Config.in"

new file mode 100644
index 0000000..68b518e
@@ -0,0 +1,15 @@
+config BR2_TARGET_ROOTFS_RAWIMG
+	bool "raw disk image"
+	depends on BR2_i386 || BR2_x86_64
+	depends on BR2_LINUX_KERNEL
+	select BR2_PACKAGE_HOST_MKRAWIMG
+	select BR2_LINUX_KERNEL_INSTALL_TARGET
+	select BR2_TARGET_ROOTFS_EXT2
+	help
+	  Build a bootable raw disk image, that can be booted in a VM (e.g. QEMU),
+	  or directly written to disk. Creates a valid partition table with MBR,
+	  and one primary ext* partition.
+
+comment "raw disk image needs a Linux kernel to be built"
+	depends on BR2_i386 || BR2_x86_64
+	depends on !BR2_LINUX_KERNEL

new file mode 100644
index 0000000..c0a720d
@@ -0,0 +1,13 @@
+################################################################################
+#
+# Build the raw disk image
+#
+################################################################################
+
+ROOTFS_RAWIMG_DEPENDENCIES = rootfs-ext2 host-mkrawimg
+
+define ROOTFS_RAWIMG_CMD
+	PATH=$(BR_PATH) mkrawimg $(BINARIES_DIR) $@
+endef
+
+$(eval $(call ROOTFS_TARGET,rawimg))

1.7.10.4

Karoly, All,

On 2014-12-09 17:03 +0100, Karoly Kasza spake thusly:

We (Peter, Thomas P., Samuel and I) discussed your patch during the Patchwork cleanup session yesterday. We came up to the following conclusion.

We believe this is trying to cover a very specific use-case, and should better be done in a post-image script.

Basically, we believe that Buildroot should not try to be too smart about the filesystem generation. We offer a set of options to build basic filesystem images for basic use-cases, as a mean to get kick-started more easily and for simple systems.

However, more complex filesystem layouts ought to be outside the scope of what Buildroot should try to automate, given the unlimited combinations that may exist.

We are however willing to add host packages that provide the tools required for generating various filesystems or manipulating various partition types (some of them are already present in the "Host utilities" sub-menu) and delegate to the user to provide one or more post-image scripts that can use those tools and assemble the required filesystem and/or disk images from the rootfs.tar tarball.

Another point in the conclusion was that we should provide some sample post-image scripts that users can have a look at to see how to write their own.

As an aside, I add a personal note that was not discussed yesterday: we should document in the manual all of the above.

As a consequence, we've marked your two patches as Rejected on Patchwork:
- [2/3] package/mkrawimg: new package
- [3/3] fs/rawimg: new rootfs target

Thank you for your numerous contributions so far, they are much appreciated! :-)

Regards,
Yann E. MORIN.

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Hi Yann, list,

We believe this is trying to cover a very specific use-case, and should

However, more complex filesystem layouts ought to be outside the scope

OK

Another point in the conclusion was that we should provide some sample

Should I just create a mock-up of this script into the support/scripts directory or so?
By the way, the build-ext3-img script does something very similar, but needs root privileges. It also creates a vmx file but no vmdk conversion is involved, so I think it is broken.

As an aside, I add a personal note that was not discussed yesterday: we

You are right - just the manual seems to follow the development of BR with some delay :)

Some explanation from my perspective regarding this patch:

I am building BR for various VMs. I use VMware (that's why I sent in openvmtools obviously), QEMU, KVM and various Cloud providers based on KVM. I think that BR has a great potential in purpose built VMs (vApps), because of a few reasons:

- the user is completely in control what runs in his/her VM, this adds security
- the current VM model is one App per one VM - for which BR is a good choice as it contains no bloatware like a general purpose Linux distro
- it is sparing with the resources, which means two things: greater VM density and lower pay-per-go costs in case of hosted VMs (like in Google Cloud's payment model)

So I started this series, and planned to add at least 2 other functionalities later:
- build as an OVA file (with the use of host-qemu's "qemu-img convert" command)
- build directly for Google Cloud Compute Engine, which is a special KVM (thus QEMU) image

If you guys think, that this could be a fair use-case for BR, I would be happy to continue to provide patches or post build scripts for at least these VM formats.

Best regards,
Karoly
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20141211/167d025f/attachment.html>