bmaptool support
v2 cover bmaptool support
Hi,
This patchset adds ability to use bmaptool to flash images faster than using traditional methods.
Bmaptool is a generic tool for creating the block map (bmap) for a file and and copying files using the block map. The idea is that large file containing unused blocks, like raw system image files, can be copied or flashed a lot faster with bmaptool than with traditional tools like "dd" or "cp".
Here is an example of flashing image using dd and bmaptool showing around 25% performance gain in flashing time:
Quoted 1 line
ls -slh
19M -rw-r--r-- 1 ed ed 26M Apr 26 11:36 core-image-minimal-qemux86-64-20160426083554.rootfs.ext4
4.0K -rw-r--r-- 1 ed ed 2.7K Apr 26 11:36 core-image-minimal-qemux86-64-20160426083554.rootfs.ext4.bmap
Quoted 1 line
time dd if=core-image-minimal-qemux86-64-20160426083554.rootfs.ext4 of=/dev/sdb
52302+0 records in
52302+0 records out
26778624 bytes (27 MB) copied, 2.51183 s, 10.7 MB/s
real 0m2.515s user 0m0.014s sys 0m0.329s
Quoted 1 line
time bmaptool copy core-image-minimal-qemux86-64-20160426083554.rootfs.ext4
/dev/sdb
bmaptool: info: discovered bmap file 'core-image-minimal-qemux86-64-20160426083554.rootfs.ext4.bmap'
bmaptool: info: block map format version 2.0
bmaptool: info: 6538 blocks of size 4096 (25.5 MiB), mapped 4799 blocks (18.7 MiB or 73.4%)
bmaptool: info: copying image 'core-image-minimal-qemux86-64-20160426083554.rootfs.ext4' to block device '/dev/sdb' using bmap file 'core-image-minimal-qemux86-64-20160426083554.rootfs.ext4.bmap'
bmaptool: info: 100% copied
bmaptool: info: synchronizing '/dev/sdb'
bmaptool: info: copying time: 1.8s, copying speed 10.3 MiB/sec
real 0m1.909s user 0m0.226s sys 0m0.052s
The test was repeated 5 times with the same image and the same usb stick device.
Changes in v2: Combined Alexander's and my bmap-tools recipes.
Got rid of generating standalone script as it breaks build of bmap-tools-native.
The following changes since commit a9b503b268e94d311f892fa00c5d6bd9ffdb228e:
license.bbclass: make sure that image manifest dir exists (2016-04-22 16:28:57 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib ed/oe-core/bmap-tools-9414
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/oe-core/bmap-tools-9414
Alexander D. Kanevskiy (1):
image types: add bmap generation option
Ed Bartosh (2):
bmap-tools: initial commit, version 3.2
selftest: add bmap test
meta/classes/image_types.bbclass | 4 +++- meta/lib/oeqa/selftest/imagefeatures.py | 27 +++++++++++++++++++++++ meta/recipes-support/bmap-tools/bmap-tools_3.2.bb | 24 ++++++++++++++++++++ 3 files changed, 54 insertions(+), 1 deletion(-) create mode 100644 meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
Signature · 1 line
2.1.4
PATCH v2 1/3 bmap-tools: initial commit, version 3.2
Bmap-tools - tools to generate block map (AKA bmap) and flash images using bmap. Bmaptool is a generic tool for creating the block map (bmap) for a file and copying files using the block map.
The idea is that large file containing unused blocks, like raw system image files, can be copied or flashed a lot faster with bmaptool than with traditional tools like "dd" or "cp".
[YOCTO #9414]
--- meta/recipes-support/bmap-tools/bmap-tools_3.2.bb | 24 +++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
meta/recipes-support/bmap-tools/bmap-tools_3.2.bb new file +24 -0
@@ -0,0 +1,24 @@ +SUMMARY = "Tools to generate block map (AKA bmap) and flash images using bmap" +DESCRIPTION = "Bmap-tools - tools to generate block map (AKA bmap) and flash images using \ +bmap. Bmaptool is a generic tool for creating the block map (bmap) for a file, \ +and copying files using the block map. The idea is that large file containing \ +unused blocks, like raw system image files, can be copied or flashed a lot \ +faster with bmaptool than with traditional tools like "dd" or "cp"." +HOMEPAGE = "http://git.infradead.org/users/dedekind/bmap-tools.git" +SECTION = "console/utils" +LICENSE = "GPLv2" +LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263" + +SRC_URI = "ftp://ftp.infradead.org/pub/${BPN}/${BPN}-${PV}.tgz" +SRC_URI[md5sum] = "92cdad1cb4dfa0cca7176c8e22752616" +SRC_URI[sha256sum] = "cc6c7f7dc0a37e2a32deb127308e24e6c4b80bfb54f3803c308efab02bf2d434" + +RDEPENDS_${PN} = "python-core python-compression" + +inherit setuptools + +BBCLASSEXTEND = "native" + +do_install_append_class-native() { + sed -i -e 's|^#!.*/usr/bin/env python|#! /usr/bin/env nativepython|' ${D}${bindir}/bmaptool +}
Signature · 1 line
2.1.4
Re: [PATCH v2 1/3] bmap-tools: initial commit, version 3.2
Quoted 53 lines · Ed Bartosh
Bmap-tools - tools to generate block map (AKA bmap) and flash images using bmap. Bmaptool is a generic tool for creating the block map (bmap) for a file and copying files using the block map.
The idea is that large file containing unused blocks, like raw system image files, can be copied or flashed a lot faster with bmaptool than with traditional tools like "dd" or "cp".
[YOCTO #9414]
Signed-off-by Ed Bartosh <[email protected]>--- meta/recipes-support/bmap-tools/bmap-tools_3.2.bb | 24 +++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 meta/recipes-support/bmap-tools/bmap-tools_3.2.bbmeta/recipes-support/bmap-tools/bmap-tools_3.2.bb new file +1
-0@@ -0,0 +1,24 @@ +SUMMARY = "Tools to generate block map (AKA bmap) and flash images usingbmap"
+DESCRIPTION = "Bmap-tools - tools to generate block map (AKA bmap) and flash images using \
+bmap. Bmaptool is a generic tool for creating the block map (bmap) for a file, \
+and copying files using the block map. The idea is that large file containing \
+unused blocks, like raw system image files, can be copied or flashed a lot \meta/recipes-support/bmap-tools/bmap-tools_3.2.bb new file +9
-0+faster with bmaptool than with traditional tools like "dd" or "cp"." +HOMEPAGE = "http://git.infradead.org/users/dedekind/bmap-tools.git" +SECTION = "console/utils" +LICENSE = "GPLv2" +LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263" + +SRC_URI = "ftp://ftp.infradead.org/pub/${BPN}/${BPN}-${PV}.tgz" +SRC_URI[md5sum] = "92cdad1cb4dfa0cca7176c8e22752616" +SRC_URI[sha256sum] ="
cc6c7f7dc0a37e2a32deb127308e24e6c4b80bfb54f3803c308efab02bf2d434"meta/recipes-support/bmap-tools/bmap-tools_3.2.bb new file +9
-0+ +RDEPENDS_${PN} = "python-core python-compression" + +inherit setuptools + +BBCLASSEXTEND = "native" + +do_install_append_class-native() { + sed -i -e 's|^#!.*/usr/bin/env python|#! /usr/bin/env nativepython|'
Quoted 3 lines
${D}${bindir}/bmaptool
+}
better to use recipe that would create in deploy/tools standalone version of the tool.
Otherwise, there is no easy way for user to utilize generated bmap files.
--
Quoted 8 lines
2.1.4
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Signature · 1 line
br, Alexander Kanevskiy
Re: [PATCH v2 1/3] bmap-tools: initial commit, version 3.2
Quoted 57 lines · On Fri, 29 Apr 2016 01:07:56 Alexander Kanevskiy wrote:
On Wed, Apr 27, 2016 at 12:36 PM, Ed Bartosh <[email protected]>
Quoted 54 lines · wrote:
Bmap-tools - tools to generate block map (AKA bmap) and flash images using bmap. Bmaptool is a generic tool for creating the block map (bmap) for a file and copying files using the block map.
The idea is that large file containing unused blocks, like raw system image files, can be copied or flashed a lot faster with bmaptool than with traditional tools like "dd" or "cp".
[YOCTO #9414]
Signed-off-by Ed Bartosh <[email protected]>---
meta/recipes-support/bmap-tools/bmap-tools_3.2.bb | 24+++++++++++++++++++++++
1 file changed, 24 insertions(+) create mode 100644 meta/recipes-support/bmap-tools/bmap-tools_3.2.bbmeta/recipes-support/bmap-tools/bmap-tools_3.2.bb new file +1
-0@@ -0,0 +1,24 @@ +SUMMARY = "Tools to generate block map (AKA bmap) and flash images usingbmap"
+DESCRIPTION = "Bmap-tools - tools to generate block map (AKA bmap) and flash images using \
+bmap. Bmaptool is a generic tool for creating the block map (bmap) for a file, \
+and copying files using the block map. The idea is that large file containing \
+unused blocks, like raw system image files, can be copied or flashed a lot \meta/recipes-support/bmap-tools/bmap-tools_3.2.bb new file +9
-0+faster with bmaptool than with traditional tools like "dd" or "cp"." +HOMEPAGE = "http://git.infradead.org/users/dedekind/bmap-tools.git" +SECTION = "console/utils" +LICENSE = "GPLv2" +LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263" + +SRC_URI = "ftp://ftp.infradead.org/pub/${BPN}/${BPN}-${PV}.tgz" +SRC_URI[md5sum] = "92cdad1cb4dfa0cca7176c8e22752616" +SRC_URI[sha256sum] ="
cc6c7f7dc0a37e2a32deb127308e24e6c4b80bfb54f3803c308efab02bf2d434"meta/recipes-support/bmap-tools/bmap-tools_3.2.bb new file +7
-0+ +RDEPENDS_${PN} = "python-core python-compression" + +inherit setuptools + +BBCLASSEXTEND = "native" +
Quoted 8 lines
Quoted 4 lines
meta/recipes-support/bmap-tools/bmap-tools_3.2.bb new file +2
-0+do_install_append_class-native() { + sed -i -e 's|^#!.*/usr/bin/env python|#! /usr/bin/env nativepython|'${D}${bindir}/bmaptool
+}better to use recipe that would create in deploy/tools standalone version of the tool.
Otherwise, there is no easy way for user to utilize generated bmap files.
You both probably have had notification already, but just to close the loop I found the issue with the do_deploy version, see the comments on bug 9414:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=9414
Cheers,
Paul
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Re: [PATCH v2 1/3] bmap-tools: initial commit, version 3.2
Quoted 21 lines · Paul Eggleton
Quoted 12 lines · On Fri, 29 Apr 2016 01:07:56 Alexander Kanevskiy wrote:
On Wed, Apr 27, 2016 at 12:36 PM, Ed Bartosh <[email protected]
Quoted 8 lines · wrote:
Bmap-tools - tools to generate block map (AKA bmap) and flash images using bmap. Bmaptool is a generic tool for creating the block map (bmap) for a file and copying files using the block map.
The idea is that large file containing unused blocks, like raw system image files, can be copied or flashed a lot faster with bmaptool than with traditional tools like "dd" or "cp".
Quoted 7 lines
Quoted 3 lines
[YOCTO #9414]
Signed-off-by Ed Bartosh <[email protected]>better to use recipe that would create in deploy/tools standalone version of the tool.
Otherwise, there is no easy way for user to utilize generated bmap files.
Quoted 7 lines
You both probably have had notification already, but just to close the loop I
found the issue with thedo_deployversion, see the comments on bug 9414:
Thank you Paul for your help! Your fix mentioned in the bug for do_depoly is indeed the best solution.
Signature · 1 line
br, Alexander Kanevskiy
Re: [PATCH v2 1/3] bmap-tools: initial commit, version 3.2
Quoted 28 lines · Alexander Kanevskiy
Quoted 25 lines · Paul Eggleton
Quoted 12 lines · On Fri, 29 Apr 2016 01:07:56 Alexander Kanevskiy wrote:
On Wed, Apr 27, 2016 at 12:36 PM, Ed Bartosh <[email protected]
Quoted 8 lines · wrote:
Bmap-tools - tools to generate block map (AKA bmap) and flash images using bmap. Bmaptool is a generic tool for creating the block map (bmap) for a file and copying files using the block map.
The idea is that large file containing unused blocks, like raw system image files, can be copied or flashed a lot faster with bmaptool than with traditional tools like "dd" or "cp".
Quoted 7 lines
Quoted 3 lines
[YOCTO #9414]
Signed-off-by Ed Bartosh <[email protected]>better to use recipe that would create in deploy/tools standalone version of the tool.
Otherwise, there is no easy way for user to utilize generated bmap files.You both probably have had notification already, but just to close the loop I
found the issue with thedo_deployversion, see the comments on bug 9414:
Quoted 6 lines
Quoted 3 lines
Thank you Paul for your help! Your fix mentioned in the bug for
do_depolyis indeed the best solution.
OK. I'll add this functionality back in a separate patch.
--
Regards,
Ed
PATCH v2 2/3 image types: add bmap generation option
In reply to Ed BartoshFrom: "Alexander D. Kanevskiy" <[email protected]>
bmap image conversion type allows to create block map files for sparse images. Bmap file can be used together with bmap-tools for efficiently flash images to raw devices (hdd or usb drive)
[YOCTO #9414]
--- meta/classes/image_types.bbclass | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
meta/classes/image_types.bbclass +3 -1
@@ -262,7 +262,7 @@ IMAGE_TYPES = " \ wic wic.gz wic.bz2 wic.lzma \ " -COMPRESSIONTYPES = "gz bz2 lzma xz lz4 sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum" +COMPRESSIONTYPES = "gz bz2 lzma xz lz4 sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum bmap" COMPRESS_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}" COMPRESS_CMD_gz = "gzip -f -9 -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz" COMPRESS_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
@@ -275,12 +275,14 @@ COMPRESS_CMD_sha224sum = "sha224sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > COMPRESS_CMD_sha256sum = "sha256sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha256sum" COMPRESS_CMD_sha384sum = "sha384sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha384sum" COMPRESS_CMD_sha512sum = "sha512sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha512sum" +COMPRESS_CMD_bmap = "bmaptool create ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} -o ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.bmap" COMPRESS_DEPENDS_lzma = "xz-native" COMPRESS_DEPENDS_gz = "" COMPRESS_DEPENDS_bz2 = "pbzip2-native" COMPRESS_DEPENDS_xz = "xz-native" COMPRESS_DEPENDS_lz4 = "lz4-native" COMPRESS_DEPENDS_sum = "mtd-utils-native" +COMPRESS_DEPENDS_bmap = "bmap-tools-native" RUNNABLE_IMAGE_TYPES ?= "ext2 ext3 ext4" RUNNABLE_MACHINE_PATTERNS ?= "qemu"
Signature · 1 line
2.1.4
PATCH v2 3/3 selftest: add bmap test
In reply to Ed BartoshAdded test_bmap to imagefeatures tests.
It tests if bmap file is generated for the images and if the image is sparse.
[YOCTO #9414]
--- meta/lib/oeqa/selftest/imagefeatures.py | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
meta/lib/oeqa/selftest/imagefeatures.py +27 -0
@@ -98,3 +98,30 @@ class ImageFeatures(oeSelfTest): # Build a core-image-weston bitbake('core-image-weston') + def test_bmap(self): + """ + Summary: Check bmap support + Expected: 1. core-image-minimal can be build with bmap support + 2. core-image-minimal is sparse + Product: oe-core + Author: Ed Bartosh <[email protected]> + """ + + features = 'IMAGE_FSTYPES += " ext4 ext4.bmap"' + self.write_config(features) + + image_name = 'core-image-minimal' + bitbake(image_name) + + deploy_dir = get_bb_var('DEPLOY_DIR_IMAGE') + link_name = get_bb_var('IMAGE_LINK_NAME', image_name) + image_path = os.path.join(deploy_dir, "%s.ext4" % link_name) + bmap_path = "%s.bmap" % image_path + + # check if result image and bmap file are in deploy directory + self.assertTrue(os.path.exists(image_path)) + self.assertTrue(os.path.exists(bmap_path)) + + # check if result image is sparse + image_stat = os.stat(image_path) + self.assertTrue(image_stat.st_size > image_stat.st_blocks * 512)
Signature · 1 line
2.1.4