package/bmap-tools: new package
PATCH 1/2 package/bmap-tools: new package
--- DEVELOPERS | 3 +++ package/Config.in | 1 + package/bmap-tools/Config.in | 13 +++++++++++++ package/bmap-tools/bmap-tools.hash | 2 ++ package/bmap-tools/bmap-tools.mk | 15 +++++++++++++++ 5 files changed, 34 insertions(+) create mode 100644 package/bmap-tools/Config.in create mode 100644 package/bmap-tools/bmap-tools.hash create mode 100644 package/bmap-tools/bmap-tools.mk
DEVELOPERS +3 -0
@@ -1725,6 +1725,9 @@ F: package/libevdev/ N: Nicola Di Lieto <[email protected]> F: package/uacme/ +N: Nicolas Carrier <[email protected]> +F: package/bmap-tools/ + N: Nicolas Cavallari <[email protected]> F: package/libgit2/
package/Config.in +1 -0
@@ -184,6 +184,7 @@ menu "Filesystem and flash utilities" source "package/aufs/Config.in" source "package/aufs-util/Config.in" source "package/autofs/Config.in" + source "package/bmap-tools/Config.in" source "package/btrfs-progs/Config.in" source "package/cifs-utils/Config.in" source "package/cpio/Config.in"
package/bmap-tools/Config.in new file +13 -0
@@ -0,0 +1,13 @@ +config BR2_PACKAGE_BMAP_TOOLS + bool "bmap-tools" + depends on BR2_PACKAGE_PYTHON3 || \ + BR2_PACKAGE_PYTHON + select BR2_PACKAGE_PYTHON_SIX # runtime + select BR2_PACKAGE_PYTHON_SETUPTOOLS + help + Tool to flash image files to block devices using the block map + bmaptool is a generic tool for creating the block map (bmap) + for a file, and copying files using the block map. The idea is + that large file containing unused blocks, like raw system + image files, can be copied or flashed a lot faster with + bmaptool than with traditional tools like "dd" or "cp".
package/bmap-tools/bmap-tools.hash new file +2 -0
@@ -0,0 +1,2 @@ +sha256 d410e2d97192d0fc2f88ef160a0bb6ed83fce99da97a606d7f6890cc654ec594 bmap-tools-v3.5.tar.gz +sha256 8177f97513213526df2cf6184d8ff986c675afb514d4e68a404010521b880643 COPYING
package/bmap-tools/bmap-tools.mk new file +15 -0
@@ -0,0 +1,15 @@ +################################################################################ +# +# bmap-tools +# +################################################################################ + +BMAP_TOOLS_VERSION = v3.5 +BMAP_TOOLS_SITE = $(call github,intel,bmap-tools,$(BMAP_TOOLS_VERSION)) +BMAP_TOOLS_LICENSE = GPLv2 +BMAP_TOOLS_LICENSE_FILES = COPYING +BMAP_TOOLS_SETUP_TYPE = setuptools +BMAP_TOOLS_DEPENDENCIES = \ + python-setuptools + +$(eval $(python-package))
Signature · 1 line
2.20.1
PATCH 2/2 support/testing/tests/package/: runtime test
This patch implements a simple test in which a dummy file system image is created, then bmaptool create and bmaptool copy are used to copy it to another file.
--- .../tests/package/sample_bmap_tools.sh | 15 ++++++ .../testing/tests/package/test_bmap_tools.py | 54 +++++++++++++++++++ 2 files changed, 69 insertions(+) create mode 100755 support/testing/tests/package/sample_bmap_tools.sh create mode 100644 support/testing/tests/package/test_bmap_tools.py
support/testing/tests/package/sample_bmap_tools.sh new file +15 -0
@@ -0,0 +1,15 @@ +#!/bin/sh +# simple test which creates a dummy file system image, then use bmaptool create +# and bmaptool copy to copy it to another file + +set -xeu + +# create the necessary test files +dd if=/dev/zero of=disk.img bs=2M count=1 +mkfs.ext4 disk.img +fallocate -d disk.img +dd if=/dev/zero of=copy.img bs=2M count=1 + +# do a test copy of the file system image +bmaptool create -o disk.img.bmap disk.img +bmaptool copy disk.img copy.img
support/testing/tests/package/test_bmap_tools.py new file +54 -0
54 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH 2/2 support/testing/tests/package/: runtime test
Hello Nicolas,
Quoted 5 lines · On Sat, 26 Oct 2019 14:41:21 +0000 Nicolas Carrier…
This patch implements a simple test in which a dummy file system image is created, then
bmaptool createandbmaptool copyare used to copy it to another file.Signed-off-by Nicolas Carrier <[email protected]>
The title should be: support/testing/tests/package/test_bmap_tools: new test
Also, please add an entry in the DEVELOPERS file for those two new files you are doing. This will make sure you get notified when there is a failure of your test in our CI.
support/testing/tests/package/test_bmap_tools.py line 6
Quoted 12 lines
support/testing/tests/package/test_bmap_tools.py new file +6
-0@@ -0,0 +1,54 @@ +import os +import sys +import infra + +from infra.basetest import BRTest +from abc import ABC, abstractproperty
I don't think we really need this, it's not used anywhere else.
support/testing/tests/package/test_bmap_tools.py line 8
Quoted 2 lines
support/testing/tests/package/test_bmap_tools.py new file +2
-0+ +class AbstractBmapToolsTest(BRTest, ABC):
Name this just:
class TestBmapTools(infra.basetest.BRTest)
support/testing/tests/package/test_bmap_tools.py line 11
Quoted 3 lines
support/testing/tests/package/test_bmap_tools.py new file +3
-0+ __test__ = False + sample_script = "tests/package/sample_bmap_tools.sh" +
Maybe this empty line is not really needed.
BTW, did you run flake8 on your Python code to make sure it is compliant with Buildroot's Python coding style ?
support/testing/tests/package/test_bmap_tools.py line 14
Quoted 3 lines
support/testing/tests/package/test_bmap_tools.py new file +3
-0+ copy_script = 'tests/package/copy-sample-script-to-target.sh' + config = f''' + {infra.basetest.BASIC_TOOLCHAIN_CONFIG}
I didn't know about this syntax. I don't think we're using it elsewhere, but that's OK for me, this syntax is quite nice.
Quoted 4 lines
support/testing/tests/package/test_bmap_tools.py new file +4
-0+ BR2_TARGET_ROOTFS_CPIO=y + BR2_PACKAGE_BMAP_TOOLS=y + BR2_ROOTFS_POST_BUILD_SCRIPT="{infra.filepath(copy_script)}" + BR2_ROOTFS_POST_SCRIPT_ARGS="{infra.filepath(sample_script)}"
support/testing/tests/package/test_bmap_tools.py line 28
Quoted 10 lines
support/testing/tests/package/test_bmap_tools.py new file +10
-0+ # BR2_TARGET_ROOTFS_TAR is not set + BR2_PACKAGE_UTIL_LINUX=y + BR2_PACKAGE_UTIL_LINUX_FALLOCATE=y + BR2_PACKAGE_E2FSPROGS=y + BR2_PACKAGE_UTIL_LINUX_LIBUUID=y + ''' + + def __init__(self, names): + super(AbstractBmapToolsTest, self).__init__(names) + self.config += f"BR2_PACKAGE_PYTHON{self.python_version}=y"
I don't think you need this.
support/testing/tests/package/test_bmap_tools.py line 32
Quoted 3 lines
support/testing/tests/package/test_bmap_tools.py new file +3
-0+ @abstractproperty + def python_version(self): + pass
or this.
Quoted 8 lines
support/testing/tests/package/test_bmap_tools.py new file +8
-0+ + def login(self): + cpio_file = os.path.join(self.builddir, "images", "rootfs.cpio") + self.emulator.boot(arch="armv5", + kernel="builtin", + options=["-initrd", cpio_file]) + self.emulator.login() +
support/testing/tests/package/test_bmap_tools.py line 50
Quoted 10 lines
support/testing/tests/package/test_bmap_tools.py new file +10
-0+ def test_run(self): + print(self.config, file=sys.stderr) + self.login() + cmd = f"/root/{os.path.basename(self.sample_script)}" + _, exit_code = self.emulator.run(cmd, timeout=10) + self.assertEqual(exit_code, 0) + +class TestPy2BmapTools(AbstractBmapToolsTest): + __test__ = True + python_version = ""
Just do this:
class TestBmapToolsPython2(TestBmapTools):
__test__ = True config = TestBmapTools.config + \ """ BR2_PACKAGE_PYTHON=y """
and ditto for Python 3.x of course. See test_python_treq.py for an example of how we are doing it.
I understand what you're proposing is a different way of doing the same thing, we really value consistency across the code base. So we want a given thing to always be done the same way.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 2/2 support/testing/tests/package/: runtime test
Quoted 5 lines · Thomas Petazzoni
Hello Nicolas,
On Sat, 26 Oct 2019 14:41:21 +0000
Nicolas Carrier <[email protected]> wrote:
Quoted 7 lines
Quoted 5 lines
This patch implements a simple test in which a dummy file system image is created, then
bmaptool createandbmaptool copyare used to copy it to another file.Signed-off-by Nicolas Carrier <[email protected]>The title should be: support/testing/tests/package/test_bmap_tools: new test
Not that I have the power to veto, but I veto this :-) That's a ridiculously
long title. I think it should be "support/testing: add bmap_tools test". That is BTW what 90% of the titles of patches in support/testing are now.
Quoted 5 lines
Also, please add an entry in the DEVELOPERS file for those two new files you are doing. This will make sure you get notified when there is a failure of your test in our CI.
Quoted 14 lines
support/testing/tests/package/test_bmap_tools.py line 6
Quoted 12 lines
support/testing/tests/package/test_bmap_tools.py new file +6
-0@@ -0,0 +1,54 @@ +import os +import sys +import infra + +from infra.basetest import BRTest +from abc import ABC, abstractpropertyI don't think we really need this, it's not used anywhere else.
If we're using ABC, then BRTest should definitely be an abstract base class.
But indeed, for consistency, it would be better to stick to the patterns we are using now, or convert things globally.
Quoted 17 lines
support/testing/tests/package/test_bmap_tools.py line 8
Quoted 3 lines
support/testing/tests/package/test_bmap_tools.py new file +2
-0+ +class AbstractBmapToolsTest(BRTest, ABC):Name this just:
class TestBmapTools(infra.basetest.BRTest)
support/testing/tests/package/test_bmap_tools.py line 11
Quoted 3 lines
support/testing/tests/package/test_bmap_tools.py new file +3
-0+ __test__ = False + sample_script = "tests/package/sample_bmap_tools.sh" +Maybe this empty line is not really needed.
BTW, did you run flake8 on your Python code to make sure it is compliant with Buildroot's Python coding style ?
Quoted 6 lines
support/testing/tests/package/test_bmap_tools.py line 14
Quoted 3 lines
support/testing/tests/package/test_bmap_tools.py new file +3
-0+ copy_script = 'tests/package/copy-sample-script-to-target.sh' + config = f''' + {infra.basetest.BASIC_TOOLCHAIN_CONFIG}I didn't know about this syntax. I don't think we're using it elsewhere, but that's OK for me, this syntax is quite nice.
f-strings were introduced in Python 3.5. I'm a big fan of f-string too, but
then we should document that we require Python 3.5 for running the tests.
Quoted 16 lines
Quoted 5 lines
support/testing/tests/package/test_bmap_tools.py new file +4
-0+ BR2_TARGET_ROOTFS_CPIO=y + BR2_PACKAGE_BMAP_TOOLS=y + BR2_ROOTFS_POST_BUILD_SCRIPT="{infra.filepath(copy_script)}" + BR2_ROOTFS_POST_SCRIPT_ARGS="{infra.filepath(sample_script)}"support/testing/tests/package/test_bmap_tools.py line 28
Quoted 10 lines
support/testing/tests/package/test_bmap_tools.py new file +10
-0+ # BR2_TARGET_ROOTFS_TAR is not set + BR2_PACKAGE_UTIL_LINUX=y + BR2_PACKAGE_UTIL_LINUX_FALLOCATE=y + BR2_PACKAGE_E2FSPROGS=y + BR2_PACKAGE_UTIL_LINUX_LIBUUID=y + ''' + + def __init__(self, names): + super(AbstractBmapToolsTest, self).__init__(names) + self.config += f"BR2_PACKAGE_PYTHON{self.python_version}=y"
Quoted 1 line
I don't think you need this.
I think to deal with variations like this, it would be better to have support
for formatting the config string. I.e., before writing the config file, we pass it through format() with a subclass-provided dict as argument.
But again, that's a bigger change. For now, you can just copy the self.config
+= to the different subclasses, it's just a tiny increase in the number of lines.
Quoted 35 lines
support/testing/tests/package/test_bmap_tools.py line 32
Quoted 4 lines
support/testing/tests/package/test_bmap_tools.py new file +3
-0+ @abstractproperty + def python_version(self): + passor this.
Quoted 8 lines
support/testing/tests/package/test_bmap_tools.py new file +8
-0+ + def login(self): + cpio_file = os.path.join(self.builddir, "images", "rootfs.cpio") + self.emulator.boot(arch="armv5", + kernel="builtin", + options=["-initrd", cpio_file]) + self.emulator.login() +support/testing/tests/package/test_bmap_tools.py line 50
Quoted 10 lines
support/testing/tests/package/test_bmap_tools.py new file +10
-0+ def test_run(self): + print(self.config, file=sys.stderr) + self.login() + cmd = f"/root/{os.path.basename(self.sample_script)}" + _, exit_code = self.emulator.run(cmd, timeout=10) + self.assertEqual(exit_code, 0) + +class TestPy2BmapTools(AbstractBmapToolsTest): + __test__ = True + python_version = ""Just do this:
class TestBmapToolsPython2(TestBmapTools):
__test__ = True config = TestBmapTools.config + \ """ BR2_PACKAGE_PYTHON=y """
Quoted 6 lines
and ditto for Python 3.x of course. See
test_python_treq.pyfor an example of how we are doing it.I understand what you're proposing is a different way of doing the same thing, we really value consistency across the code base. So we want a given thing to always be done the same way.
+1
Regards,
Arnout
Quoted 5 lines
Best regards,
Thomas
PATCH 1/2 package/bmap-tools: new package
In reply to Nicolas CarrierHello Nicolas,
package/bmap-tools/Config.in line 4
Quoted 10 lines · On Sat, 26 Oct 2019 14:41:18 +0000 Nicolas Carrier…
package/bmap-tools/Config.in new file +4
-0@@ -0,0 +1,13 @@ +config BR2_PACKAGE_BMAP_TOOLS + bool "bmap-tools" + depends on BR2_PACKAGE_PYTHON3 || \ + BR2_PACKAGE_PYTHON
For some packages like this, we also do a simple select, because it's not obvious you need Python.
package/bmap-tools/Config.in line 6
Quoted 2 lines
package/bmap-tools/Config.in new file +2
-0+ select BR2_PACKAGE_PYTHON_SIX # runtime + select BR2_PACKAGE_PYTHON_SETUPTOOLS
Do you really need setuptools on the target? This seems odd.
package/bmap-tools/Config.in line 7
Quoted 1 line
package/bmap-tools/Config.in new file +1
-0+BMAP_TOOLS_VERSION = v3.5
The "v" should be in BMAP_TOOLS_SITE, not BMAP_TOOLS_VERSION.
package/bmap-tools/Config.in line 9
Quoted 2 lines
package/bmap-tools/Config.in new file +2
-0+BMAP_TOOLS_SITE = $(call github,intel,bmap-tools,$(BMAP_TOOLS_VERSION)) +BMAP_TOOLS_LICENSE = GPLv2
Should be GPL-2.0
package/bmap-tools/Config.in line 13
Quoted 4 lines
package/bmap-tools/Config.in new file +4
-0+BMAP_TOOLS_LICENSE_FILES = COPYING +BMAP_TOOLS_SETUP_TYPE = setuptools +BMAP_TOOLS_DEPENDENCIES = \ + python-setuptools
Why do you need python-setuptools on the target, and be a build time dependency ? BMAP_TOOLS_SETUP_TYPE = setuptools already says that this package needs setuptools on the host for its build/installation.
Thanks,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 1/2 package/bmap-tools: new package
Quoted 4 lines · Thomas Petazzoni
...
Do you really need setuptools on the target? This seems odd.
Apparently yes, otherwise I have this failure when running the test:
Traceback (most recent call last):
File "/usr/bin/bmaptool", line 6, in <module>
from pkg_resources import load_entry_point
ModuleNotFoundError: No module named 'pkg_resources'
And a bit of searching on SO showed that having setuptools was the missing package.
Quoted 4 lines
Quoted 2 lines
+BMAP_TOOLS_VERSION = v3.5
The "v" should be in
BMAP_TOOLS_SITE, notBMAP_TOOLS_VERSION.
Ok, will fix in v3.
Quoted 6 lines
Quoted 4 lines
+BMAP_TOOLS_SITE = $(call github,intel,bmap-
tools,$(BMAP_TOOLS_VERSION))
+BMAP_TOOLS_LICENSE = GPLv2Should be GPL-2.0
Ok, will fix in v3.
Quoted 10 lines
package/bmap-tools/bmap-tools.mk line 13
Quoted 5 lines
package/bmap-tools/bmap-tools.mk new file +4
-0+BMAP_TOOLS_LICENSE_FILES = COPYING +BMAP_TOOLS_SETUP_TYPE = setuptools +BMAP_TOOLS_DEPENDENCIES = \ + python-setuptoolsWhy do you need python-setuptools on the target, and be a build time dependency ?
BMAP_TOOLS_SETUP_TYPE= setuptools already says that this
package needs setuptools on the host for its build/installation.
As said before, it appears to be a runtime dependency too.
Quoted 11 lines
Thanks,
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
ATTENTION: This email came from an external source.
Do not open attachments or click on links from unknown senders or unexpected emails.
PATCH 1/2 package/bmap-tools: new package
In reply to Nicolas Carrierpackage/bmap-tools/Config.in line 13
Quoted 6 lines · On Sat, 26 Oct 2019 14:41:18 +0000 Nicolas Carrier…
package/bmap-tools/Config.in new file +6
-0+ Tool to flash image files to block devices using the block map + bmaptool is a generic tool for creating the block map (bmap) + for a file, and copying files using the block map. The idea is + that large file containing unused blocks, like raw system + image files, can be copied or flashed a lot faster with + bmaptool than with traditional tools like "dd" or "cp".
Another question is that you're packaging this for the target, but it sounds like a tool that would be mainly useful on the host to prepare/flash a SD card. Do you have a use case for having this tool on the target ?
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 1/2 package/bmap-tools: new package
Quoted 13 lines · Thomas Petazzoni
package/bmap-tools/Config.in line 13
Quoted 9 lines · On Sat, 26 Oct 2019 14:41:18 +0000 Nicolas Carrier…
+ Tool to flash image files to block devices using the block
map
+ bmaptool is a generic tool for creating the block map
(bmap)
+ for a file, and copying files using the block map. The idea
ispackage/bmap-tools/Config.in new file +3
-0+ that large file containing unused blocks, like raw system + image files, can be copied or flashed a lot faster with + bmaptool than with traditional tools like "dd" or "cp".
Quoted 5 lines
Another question is that you're packaging this for the target, but it sounds like a tool that would be mainly useful on the host to prepare/flash a SD card. Do you have a use case for having this tool on
the target ?
Yes, I use it to speed up the flashing of the EMMC too, it's part of our update or bootstrap procedures.
Quoted 9 lines
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
ATTENTION: This email came from an external source.
Do not open attachments or click on links from unknown senders or unexpected emails.