Skip to contentSkip to conversation

Review Request for Build Appliance

RFC In review Saul Wold 2 participants 9 messages Mar 26, 2012

lore
Share & tools
lore

Saul Wold

author
#0 lore raw

cover Review Request for Build Appliance

Richard,

This is an updated set of changes for the Build Appliance to use directdisk. I am submitting this even as I am finding out that there is an issue with genext2fs building Large FS Size (5GB and greater).

The other option is to add a newer version of dosfstools, which is not fully vetted yet, but may allow us to use bootimg instead of driectdisk.

Please review and comment the builder, image-vmdk and boot-directdisk patches ready to go, the VMDK, update for directdisk is questionable until we can solve the large image issues.

Sau!

The following changes since commit 647595821870693186c9ecad1bdfa12fe4ea334d:

core-image-minimal-mtdutils.bb: Simple DESCRIPTION grammar fix. (2012-03-25 12:27:38 +0100)

are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib sgw/self
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=sgw/oe-self

Dexuan Cui (1):
self-hosted-image: pre-populate the builder user with poky source

Saul Wold (4):
image-vmdk: Add symbolic link
boot-directdisk: Fix Block Calcuation
vmdk: Update for direstdisk
builder: Enable auto starting of Hob

 meta/classes/boot-directdisk.bbclass               |   19 ++++++---
 meta/classes/image-vmdk.bbclass                    |   17 +++++++-
 meta/recipes-core/images/self-hosted-image.bb      |   41 +++++++++++++++++++-
 meta/recipes-graphics/builder/builder_0.1.bb       |    2 +-
 .../builder/files/builder_hob_start.sh             |   10 +++-
 5 files changed, 74 insertions(+), 15 deletions(-)
Signature · 1 line
1.7.7.6
author
#1 lore raw

RFC 1/5 self-hosted-image: pre-populate the builder user with poky source

From: Dexuan Cui <[email protected]>

This patch installs the poky source into the /home/builder/poky/ of the self-hosted-image.
This makes the user of self-hosted-image easier to start a build.

I think the recent poky master is stable enough, so I specify a commit number by SRCREV -- we may want to update this number before releasing 1.2.

This patch fixes [YOCTO #2065]

Added code for supporting target based pseudo
fixed indentation

---
 meta/recipes-core/images/self-hosted-image.bb |   41 +++++++++++++++++++++++-
 1 files changed, 39 insertions(+), 2 deletions(-)
meta/recipes-core/images/self-hosted-image.bb +39 -2
index d56c2cb..5aa670d 100644
@@ -4,7 +4,7 @@ LICENSE = "MIT"
 LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=3f40d7994397109285ec7b81fdeb3b58 \
                     file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
 
-PR = "r5"
+PR = "r6"
 
 CORE_IMAGE_EXTRA_INSTALL = "\
     task-self-hosted \
@@ -13,7 +13,10 @@ CORE_IMAGE_EXTRA_INSTALL = "\
 IMAGE_FEATURES += "x11-mini package-management"
 
 # Ensure there's enough space to do a core-image-minimal build, with rm_work enabled
-IMAGE_ROOTFS_EXTRA_SPACE = "2621440"
+IMAGE_ROOTFS_EXTRA_SPACE = "1048576"
+#IMAGE_ROOTFS_EXTRA_SPACE = "2621440"
+#IMAGE_ROOTFS_EXTRA_SPACE = "20971520"
+#IMAGE_ROOTFS_EXTRA_SPACE = "5242880"
 
 # Do a quiet boot with limited console messages
 APPEND += "quiet"
@@ -21,3 +24,37 @@ APPEND += "quiet"
 IMAGE_FSTYPES = "vmdk"
 
 inherit core-image
+
+SRCREV = "26a46938d3ea1821e7bec4fa6cc8379babad238b"
+SRC_URI = "git://git.yoctoproject.org/poky;protocol=git"
+
+fakeroot do_populate_poky_src () {
+	# Because fetch2's git's unpack uses -s cloneflag, the unpacked git repo
+	# will become invalid in the target.
+	rm -rf ${WORKDIR}/git/.git
+	rm -f ${WORKDIR}/git/.gitignore
+
+	cp -Rp ${WORKDIR}/git ${IMAGE_ROOTFS}/home/builder/poky
+
+	mkdir -p ${IMAGE_ROOTFS}/home/builder/poky/build/conf
+	cp -Rp ${DL_DIR} ${IMAGE_ROOTFS}/home/builder/poky/build
+	echo "/usr/bin" > ${IMAGE_ROOTFS}/home/builder/poky/build/pseudodone
+	echo "BB_NO_NETWORK = \"1\"" > ${IMAGE_ROOTFS}/home/builder/poky/build/conf/auto.conf
+	echo "INHERIT += \"rm_work\"" >> ${IMAGE_ROOTFS}/home/builder/poky/build/conf/auto.conf
+        mkdir -p ${IMAGE_ROOTFS}/home/builder/pseudo
+        echo "export PSEUDO_PREFIX=/usr" >> ${IMAGE_ROOTFS}/home/builder/.bashrc
+        echo "export PSEUDO_LOCALSTATEDIR=/home/builder/pseudo" >> ${IMAGE_ROOTFS}/home/builder/.bashrc
+        echo "export PSEUDO_LIBDIR=/usr/lib/pseudo/lib64" >> ${IMAGE_ROOTFS}/home/builder/.bashrc
+
+        chown builder.builder ${IMAGE_ROOTFS}/home/builder/pseudo
+
+	chown -R builder.builder  ${IMAGE_ROOTFS}/home/builder/poky
+}
+
+IMAGE_PREPROCESS_COMMAND += "do_populate_poky_src; "
+
+python do_get_poky_src () {
+    bb.build.exec_func('base_do_fetch', d)
+    bb.build.exec_func('base_do_unpack', d)
+}
+addtask do_get_poky_src before do_rootfs
Signature · 1 line
1.7.7.6
author
#2 lore raw

RFC 2/5 image-vmdk: Add symbolic link

In reply to Saul Wold
---
 meta/classes/image-vmdk.bbclass |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
meta/classes/image-vmdk.bbclass +2 -0
index 3e63006..9d596cd 100644
@@ -10,6 +10,8 @@ inherit image-live
 
 create_vmdk_image () {
 	qemu-img convert -O vmdk ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.vmdk
+	ln -s ${IMAGE_NAME}.vmdk ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.vmdk
+
 }
 
 python do_vmdkimg() {
Signature · 1 line
1.7.7.6
#3 lore raw

Re: [Build App RFC 2/5] image-vmdk: Add symbolic link

The idea being what? To keep a short-name symlink to the longer named actual image?

Please include a commit message.

Quoted 18 lines · Saul Wold
---
 meta/classes/image-vmdk.bbclass |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
meta/classes/image-vmdk.bbclass +2 -0
index 3e63006..9d596cd 100644
@@ -10,6 +10,8 @@ inherit image-live
 
 create_vmdk_image () {
 	qemu-img convert -O vmdk ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.vmdk
+	ln -s ${IMAGE_NAME}.vmdk ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.vmdk
+
 }
 
 python do_vmdkimg() {
Signature · 3 lines
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel
author
#4 lore raw

RFC 3/5 boot-directdisk: Fix Block Calcuation

In reply to Saul Wold

This also changes the timeout to be settable

The block calcuation was not correctly rounding, see comment

Thanks to Darren Hart for fixing this.

---
 meta/classes/boot-directdisk.bbclass |   19 +++++++++++++------
 1 files changed, 13 insertions(+), 6 deletions(-)
meta/classes/boot-directdisk.bbclass +13 -6
index 893164f..e9e1ba3 100644
@@ -37,9 +37,8 @@ BOOTDD_EXTRA_SPACE ?= "16384"
 # Get the build_syslinux_cfg() function from the syslinux class
 
 AUTO_SYSLINUXCFG = "1"
-LABELS = "boot"
 SYSLINUX_ROOT ?= "root=/dev/sda2"
-SYSLINUX_TIMEOUT = "10" # 1 second
+SYSLINUX_TIMEOUT ?= "10"
 
 SYSLINUXCFG  = "${HDDDIR}/syslinux.cfg"
 SYSLINUXMENU = "${HDDDIR}/menu"
@@ -55,15 +54,23 @@ build_boot_dd() {
 	install -m 444 ${STAGING_LIBDIR}/syslinux/ldlinux.sys ${HDDDIR}/ldlinux.sys
 
 	BLOCKS=`du -bks ${HDDDIR} | cut -f 1`
-	SIZE=`expr $BLOCKS + ${BOOTDD_EXTRA_SPACE}`
+	BLOCKS=`expr $BLOCKS + ${BOOTDD_EXTRA_SPACE}`
+
+	# Ensure total sectors is an integral number of sectors per
+	# track or mcopy will complain. Sectors are 512 bytes, and we
+	# generate images with 32 sectors per track. This calculation is
+	# done in blocks, thus the mod by 16 instead of 32.
+	BLOCKS=$(expr $BLOCKS + $(expr 16 - $(expr $BLOCKS % 16)))
+
+	mkdosfs -n ${BOOTDD_VOLUME_ID} -S 512 -C ${HDDIMG} $BLOCKS 
+	mcopy -i ${HDDIMG} -s ${HDDDIR}/* ::/
 
-	mkdosfs -n ${BOOTDD_VOLUME_ID} -d ${HDDDIR} -C ${HDDIMG} $SIZE 
 	syslinux ${HDDIMG}
 	chmod 644 ${HDDIMG}
 
 	ROOTFSBLOCKS=`du -Lbks ${ROOTFS} | cut -f 1`
-	TOTALSIZE=`expr $SIZE + $ROOTFSBLOCKS`
-	END1=`expr $SIZE \* 1024`
+	TOTALSIZE=`expr $BLOCKS + $ROOTFSBLOCKS`
+	END1=`expr $BLOCKS \* 1024`
 	END2=`expr $END1 + 512`
 	END3=`expr \( $ROOTFSBLOCKS \* 1024 \) + $END1`
 
Signature · 1 line
1.7.7.6
#5 lore raw

Re: [Build App RFC 3/5] boot-directdisk: Fix Block Calcuation

Quoted 7 lines · Saul Wold

This also changes the timeout to be settable

The block calcuation was not correctly rounding, see comment

Thanks to Darren Hart for fixing this.

Cc Darren Hart <[email protected]>

Thanks, but be sure to use the option to get that added to actual email header :-) That's "-c" for the send-pull-request script.

--
Darren

Quoted 48 lines
---
 meta/classes/boot-directdisk.bbclass |   19 +++++++++++++------
 1 files changed, 13 insertions(+), 6 deletions(-)
meta/classes/boot-directdisk.bbclass +13 -6
index 893164f..e9e1ba3 100644
@@ -37,9 +37,8 @@ BOOTDD_EXTRA_SPACE ?= "16384"
 # Get the build_syslinux_cfg() function from the syslinux class
 
 AUTO_SYSLINUXCFG = "1"
-LABELS = "boot"
 SYSLINUX_ROOT ?= "root=/dev/sda2"
-SYSLINUX_TIMEOUT = "10" # 1 second
+SYSLINUX_TIMEOUT ?= "10"
 
 SYSLINUXCFG  = "${HDDDIR}/syslinux.cfg"
 SYSLINUXMENU = "${HDDDIR}/menu"
@@ -55,15 +54,23 @@ build_boot_dd() {
 	install -m 444 ${STAGING_LIBDIR}/syslinux/ldlinux.sys ${HDDDIR}/ldlinux.sys
 
 	BLOCKS=`du -bks ${HDDDIR} | cut -f 1`
-	SIZE=`expr $BLOCKS + ${BOOTDD_EXTRA_SPACE}`
+	BLOCKS=`expr $BLOCKS + ${BOOTDD_EXTRA_SPACE}`
+
+	# Ensure total sectors is an integral number of sectors per
+	# track or mcopy will complain. Sectors are 512 bytes, and we
+	# generate images with 32 sectors per track. This calculation is
+	# done in blocks, thus the mod by 16 instead of 32.
+	BLOCKS=$(expr $BLOCKS + $(expr 16 - $(expr $BLOCKS % 16)))
+
+	mkdosfs -n ${BOOTDD_VOLUME_ID} -S 512 -C ${HDDIMG} $BLOCKS 
+	mcopy -i ${HDDIMG} -s ${HDDDIR}/* ::/
 
-	mkdosfs -n ${BOOTDD_VOLUME_ID} -d ${HDDDIR} -C ${HDDIMG} $SIZE 
 	syslinux ${HDDIMG}
 	chmod 644 ${HDDIMG}
 
 	ROOTFSBLOCKS=`du -Lbks ${ROOTFS} | cut -f 1`
-	TOTALSIZE=`expr $SIZE + $ROOTFSBLOCKS`
-	END1=`expr $SIZE \* 1024`
+	TOTALSIZE=`expr $BLOCKS + $ROOTFSBLOCKS`
+	END1=`expr $BLOCKS \* 1024`
 	END2=`expr $END1 + 512`
 	END3=`expr \( $ROOTFSBLOCKS \* 1024 \) + $END1`
 
Signature · 3 lines
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel
author
#6 lore raw

RFC 4/5 vmdk: Update for direstdisk

In reply to Saul Wold
---
 meta/classes/image-vmdk.bbclass |   15 ++++++++++++---
 1 files changed, 12 insertions(+), 3 deletions(-)
meta/classes/image-vmdk.bbclass +12 -3
index 9d596cd..736d1d7 100644
@@ -1,15 +1,23 @@
 
 NOISO = "1"
+
+SYSLINUX_ROOT = "root=/dev/hda2 "
 SYSLINUX_PROMPT = "0"
 SYSLINUX_TIMEOUT = "1"
 SYSLINUX_LABELS = "boot"
+LABELS_append = " ${SYSLINUX_LABELS} "
+
+# need to define the dependency and the ROOTFS for directdisk
+do_bootdirectdisk[depends] += "${IMAGE_BASENAME}:do_rootfs"
+ROOTFS ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_BASENAME}-${MACHINE}.ext3"
 
 # creating VMDK relies on having a live hddimg so ensure we
 # inherit it here.
-inherit image-live
+#inherit image-live
+inherit boot-directdisk
 
 create_vmdk_image () {
-	qemu-img convert -O vmdk ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.vmdk
+	qemu-img convert -O vmdk ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hdddirect ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.vmdk
 	ln -s ${IMAGE_NAME}.vmdk ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.vmdk
 
 }
@@ -18,7 +26,8 @@ python do_vmdkimg() {
         bb.build.exec_func('create_vmdk_image', d)
 }
 
-addtask vmdkimg after do_bootimg before do_build
+#addtask vmdkimg after do_bootimg before do_build
+addtask vmdkimg after do_bootdirectdisk before do_build
 do_vmdkimg[nostamp] = "1"
 
 do_vmdkimg[depends] += "qemu-native:do_populate_sysroot" 
Signature · 1 line
1.7.7.6
#7 lore raw

Re: [Build App RFC 4/5] vmdk: Update for direstdisk

The conversion to directdisk is necessary due to the limitation in the mkdosfs version we carry and larger filesystems. Please note in the commit message so readers can know WHY the change was made.

Quoted 17 lines · Saul Wold
---
 meta/classes/image-vmdk.bbclass |   15 ++++++++++++---
 1 files changed, 12 insertions(+), 3 deletions(-)
meta/classes/image-vmdk.bbclass +2 -0
index 9d596cd..736d1d7 100644
@@ -1,15 +1,23 @@
 
 NOISO = "1"
+
+SYSLINUX_ROOT = "root=/dev/hda2 "
 SYSLINUX_PROMPT = "0"
 SYSLINUX_TIMEOUT = "1"
 SYSLINUX_LABELS = "boot"

meta/classes/image-vmdk.bbclass line 16

Quoted 10 lines
meta/classes/image-vmdk.bbclass +6 -1
+LABELS_append = " ${SYSLINUX_LABELS} "
+
+# need to define the dependency and the ROOTFS for directdisk
+do_bootdirectdisk[depends] += "${IMAGE_BASENAME}:do_rootfs"
+ROOTFS ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_BASENAME}-${MACHINE}.ext3"
 
 # creating VMDK relies on having a live hddimg so ensure we
 # inherit it here.
-inherit image-live
+#inherit image-live

A development artifact I presume? No need to leave the old image type commented out. Should probably also remove the comment as with directdisk that is no longer the case - right?

Quoted 9 lines
meta/classes/image-vmdk.bbclass +2 -1
+inherit boot-directdisk
 
 create_vmdk_image () {
-	qemu-img convert -O vmdk ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.vmdk
+	qemu-img convert -O vmdk ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hdddirect ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.vmdk
 	ln -s ${IMAGE_NAME}.vmdk ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.vmdk
 
 }

meta/classes/image-vmdk.bbclass python do_vmdkimg() { · line 29

Quoted 5 lines
meta/classes/image-vmdk.bbclass +1 -1
@@ -18,7 +26,8 @@ python do_vmdkimg() {
         bb.build.exec_func('create_vmdk_image', d)
 }
 
-addtask vmdkimg after do_bootimg before do_build
+#addtask vmdkimg after do_bootimg before do_build

Ditto.

meta/classes/image-vmdk.bbclass python do_vmdkimg() { · line 33

Quoted 4 lines
meta/classes/image-vmdk.bbclass +1 -0
+addtask vmdkimg after do_bootdirectdisk before do_build
 do_vmdkimg[nostamp] = "1"
 
 do_vmdkimg[depends] += "qemu-native:do_populate_sysroot" 

Thanks,

Signature · 3 lines
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel
author
#8 lore raw

RFC 5/5 builder: Enable auto starting of Hob

In reply to Saul Wold
---
 meta/recipes-graphics/builder/builder_0.1.bb       |    2 +-
 .../builder/files/builder_hob_start.sh             |   10 +++++++---
 2 files changed, 8 insertions(+), 4 deletions(-)
meta/recipes-graphics/builder/builder_0.1.bb +1 -1
index ceb9612..1324e5b 100644
@@ -1,7 +1,7 @@
 SUMMARY = "New user to do specific job"
 DESCRIPTION = "This recipe create a new user named ${PN}, who is used for specific jobs like building. The task can be auto started via mini X"
 SECTION = "x11"
-PR = "r0"
+PR = "r1"
 LICENSE = "MIT"
 LIC_FILES_CHKSUM = "file://builder_hob_start.sh;endline=5;md5=84796c3c41785d86100fdabcbdade00e"
 
meta/recipes-graphics/builder/files/builder_hob_start.sh +7 -3
index fdafc4e..f43e6c5 100644
@@ -4,8 +4,12 @@
 #be put here
 
 # start hob here
-#cd /intel/poky/poky
-#. ./oe-init-build-env
-#../scripts/hob
+export PSEUDO_PREFIX=/usr
+export PSEUDO_LOCALSTATEDIR=/home/builder/pseudo
+export PSEUDO_LIBDIR=/usr/lib/pseudo/lib64
+
+cd /home/builder/poky
+. ./oe-init-build-env
+hob &
 
 matchbox-terminal&
Signature · 1 line
1.7.7.6