Skip to contentSkip to conversation

package/bmap-tools: new package

PATCH Superseded Nicolas Carrier 4 participants 8 messages Oct 27, 2019 – Nov 5, 2019

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/4 package/bmap-tools: new package

---

I need bmap-tools on the target since it's what our flashing / upgrade procudure uses to speed up the process (flashing ~8GB disk images full of zeroes).

Changes v2 -> v3:
- double-checked that setuptools is required by bmap-tools at runtime too
- moved the "v" from BMAP_TOOLS_VERSION to BMAP_TOOLS_SITE (in the .hash too)
- changed license name from GPLv2 to GPL-2.0

Changes v1 -> v2:
- no change

---
 DEVELOPERS                         |  3 +++
 package/Config.in                  |  1 +
 package/bmap-tools/Config.in       | 13 +++++++++++++
 package/bmap-tools/bmap-tools.hash |  2 ++
 package/bmap-tools/bmap-tools.mk   | 15 +++++++++++++++
 5 files changed, 34 insertions(+)
 create mode 100644 package/bmap-tools/Config.in
 create mode 100644 package/bmap-tools/bmap-tools.hash
 create mode 100644 package/bmap-tools/bmap-tools.mk
DEVELOPERS +3 -0
index 6efe52f19f..322201301c 100644
@@ -1725,6 +1725,9 @@ F:	package/libevdev/
 N:	Nicola Di Lieto <[email protected]>
 F:	package/uacme/
 
+N:	Nicolas Carrier <[email protected]>
+F:	package/bmap-tools/
+
 N:	Nicolas Cavallari <[email protected]>
 F:	package/libgit2/
 
package/Config.in +1 -0
index 901c25fe02..951342042a 100644
@@ -184,6 +184,7 @@ menu "Filesystem and flash utilities"
 	source "package/aufs/Config.in"
 	source "package/aufs-util/Config.in"
 	source "package/autofs/Config.in"
+	source "package/bmap-tools/Config.in"
 	source "package/btrfs-progs/Config.in"
 	source "package/cifs-utils/Config.in"
 	source "package/cpio/Config.in"
package/bmap-tools/Config.in new file +13 -0
new file mode 100644
index 0000000000..63301f2558
@@ -0,0 +1,13 @@
+config BR2_PACKAGE_BMAP_TOOLS
+	bool "bmap-tools"
+	depends on BR2_PACKAGE_PYTHON3 || \
+		BR2_PACKAGE_PYTHON
+	select BR2_PACKAGE_PYTHON_SIX # runtime
+	select BR2_PACKAGE_PYTHON_SETUPTOOLS
+	help
+	  Tool to flash image files to block devices using the block map
+	  bmaptool is a generic tool for creating the block map (bmap)
+	  for a file, and copying files using the block map. The idea is
+	  that large file containing unused blocks, like raw system
+	  image files, can be copied or flashed a lot faster with
+	  bmaptool than with traditional tools like "dd" or "cp".
package/bmap-tools/bmap-tools.hash new file +2 -0
new file mode 100644
index 0000000000..794765b633
@@ -0,0 +1,2 @@
+sha256  d410e2d97192d0fc2f88ef160a0bb6ed83fce99da97a606d7f6890cc654ec594  bmap-tools-3.5.tar.gz
+sha256  8177f97513213526df2cf6184d8ff986c675afb514d4e68a404010521b880643  COPYING
package/bmap-tools/bmap-tools.mk new file +15 -0
new file mode 100644
index 0000000000..4c2465876e
@@ -0,0 +1,15 @@
+################################################################################
+#
+# bmap-tools
+#
+################################################################################
+
+BMAP_TOOLS_VERSION = 3.5
+BMAP_TOOLS_SITE = $(call github,intel,bmap-tools,v$(BMAP_TOOLS_VERSION))
+BMAP_TOOLS_LICENSE = GPL-2.0
+BMAP_TOOLS_LICENSE_FILES = COPYING
+BMAP_TOOLS_SETUP_TYPE = setuptools
+BMAP_TOOLS_DEPENDENCIES = \
+	python-setuptools
+
+$(eval $(python-package))
Signature · 1 line
2.20.1
#1 lore raw

PATCH v3 2/4 support/docker: add python3

From: Ricardo Martincoski <[email protected]>

The test infra will soon be converted to Python 3 only.
So add the interpreter and also the Python 3 variant of modules nose2 and pexpect to the docker image used to run runtime tests.

Keep the Python 2 variant of those modules to allow a gradual transition.

---

This is a re-submit of Ricardo Martincoski's original patch since it's a requirement the next patch (from Ricardo) which itself is required for the last patch of the series, which relies on python 3.6+ features (notably f-strings).

The 3 first patches of Ricardo's original series have already been applied, the two resubmitted here are the only ones left.

---
 support/docker/Dockerfile | 3 +++
 1 file changed, 3 insertions(+)
support/docker/Dockerfile +3 -0
index 36cd3b28a4..03acde85d2 100644
@@ -39,6 +39,9 @@ RUN apt-get install -y --no-install-recommends \
         python-flake8 \
         python-nose2 \
         python-pexpect \
+        python3 \
+        python3-nose2 \
+        python3-pexpect \
         qemu-system-arm \
         qemu-system-x86 \
         rsync \
Signature · 1 line
2.20.1
#2 lore raw

PATCH v3 2/4 support/docker: add python3

Quoted 6 lines · On Sun, 27 Oct 2019 13:36:53 +0000 Nicolas Carrier…

From: Ricardo Martincoski <[email protected]>

The test infra will soon be converted to Python 3 only.
So add the interpreter and also the Python 3 variant of modules nose2 and pexpect to the docker image used to run runtime tests.

Quoted 8 lines

Keep the Python 2 variant of those modules to allow a gradual transition.

---

Applied to master, thanks.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#3 lore raw

PATCH v3 3/4 support/testing: switch to Python 3 only

In reply to Nicolas Carrier

From: Ricardo Martincoski <[email protected]>

Python 2.7 will not be maintained past 2020.

Many scripts on the tree are used during the build and should keep Python 2 compatibility for a while.
This is not the case for the runtime test infra. It's meant to be run in modern distros only, so it can safely switch to support Python 3 only.

An advantage of this approach is to have less scenarios to test in. Otherwise every change to the test infra or runtime tests would need to be tested against both versions of the interpreter, increasing the effort of the developers, to ensure the compatibility to Python 2 was not broken.

In order to accomplish the change to Python 3:
- change the shebang for run-tests;
- use Python 3 urllib as a drop-in replacement for Python 2 urllib2;
- when writing the downloaded binary files, explicitly open the output file as binary;
- when subprocess is used to retrieve the text output from commands, explicitly ask for text output. For this, use 'universal_newlines' because 'text' was added only on Python 3.7;
- when pexpect is used to retrieve the text output from qemu or git, explicitly ask for text output using 'encoding';
- the code using csv currently follows the example in the documentation for the Python 2 module, change it to follow the example in the documentation for the Python 3 module;
- fix the relative import for test_git.py to be Python 3 compliant.

---

This is a re-submit of Ricardo Martincoski's original patch since it's required for the next (and last patch) of the series, which relies on python 3.6+ features (notably f-strings).

Original comment of Romain Naour on the original series:

Quoted 2 lines

I'm using Fedora 30 and there no python2-nose2 package anymore. So, I'm using this patch to avoid installing python2-nose2 from pypi.

I added my Tested-by tag since I used it for running the tests that are in the next patch.

---
 support/testing/infra/__init__.py                 | 8 +++++---
 support/testing/infra/emulator.py                 | 1 +
 support/testing/run-tests                         | 2 +-
 support/testing/tests/core/test_post_scripts.py   | 2 +-
 support/testing/tests/download/gitremote.py       | 3 ++-
 support/testing/tests/download/test_git.py        | 2 +-
 support/testing/tests/utils/test_check_package.py | 3 ++-
 7 files changed, 13 insertions(+), 8 deletions(-)
support/testing/infra/__init__.py +5 -3
index 43045d0173..6392aa679b 100644
@@ -3,7 +3,8 @@ import re
 import sys
 import tempfile
 import subprocess
-from urllib2 import urlopen, HTTPError, URLError
+from urllib.request import urlopen
+from urllib.error import HTTPError, URLError
 
 ARTIFACTS_URL = "http://autobuild.buildroot.net/artefacts/"
 BASE_DIR = os.path.realpath(os.path.join(os.path.dirname(__file__), "../../.."))
@@ -44,7 +45,7 @@ def download(dldir, filename):
 
     try:
         url_fh = urlopen(os.path.join(ARTIFACTS_URL, filename))
-        with open(tmpfile, "w+") as tmpfile_fh:
+        with open(tmpfile, "w+b") as tmpfile_fh:
             tmpfile_fh.write(url_fh.read())
     except (HTTPError, URLError) as err:
         os.unlink(tmpfile)
@@ -60,7 +61,8 @@ def run_cmd_on_host(builddir, cmd):
     out = subprocess.check_output(cmd,
                                   stderr=open(os.devnull, "w"),
                                   cwd=builddir,
-                                  env={"LANG": "C"})
+                                  env={"LANG": "C"},
+                                  universal_newlines=True)
     return out
 
 
support/testing/infra/emulator.py +1 -0
index 093a643a8b..5611ec96e8 100644
@@ -76,6 +76,7 @@ class Emulator(object):
         self.logfile.write("> starting qemu with '%s'\n" % " ".join(qemu_cmd))
         self.qemu = pexpect.spawn(qemu_cmd[0], qemu_cmd[1:],
                                   timeout=5 * self.timeout_multiplier,
+                                  encoding='utf-8',
                                   env={"QEMU_AUDIO_DRV": "none"})
         # We want only stdout into the log to avoid double echo
         self.qemu.logfile_read = self.logfile
support/testing/run-tests +1 -1
index 813b927045..74741aee1b 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2
+#!/usr/bin/env python3
 import argparse
 import sys
 import os
support/testing/tests/core/test_post_scripts.py +1 -1
index 40a36b7904..bc61f4af58 100644
@@ -20,7 +20,7 @@ class TestPostScripts(infra.basetest.BRTest):
 
     def check_post_log_file(self, f, what, target_dir):
         lines = {}
-        with open(os.path.join(self.builddir, "build", f), 'rb') as csvfile:
+        with open(os.path.join(self.builddir, "build", f), newline='') as csvfile:
             r = csv.reader(csvfile, delimiter=',')
             for row in r:
                 lines[row[0]] = row[1]
support/testing/tests/download/gitremote.py +2 -1
index 3b35456dd1..7df252d031 100644
@@ -32,7 +32,8 @@ class GitRemote(object):
         for port in range(GIT_REMOTE_PORT_INITIAL, GIT_REMOTE_PORT_LAST + 1):
             cmd = daemon_cmd + ["--port={port}".format(port=port)]
             self.logfile.write("> starting git remote with '{}'\n".format(" ".join(cmd)))
-            self.daemon = pexpect.spawn(cmd[0], cmd[1:], logfile=self.logfile)
+            self.daemon = pexpect.spawn(cmd[0], cmd[1:], logfile=self.logfile,
+                                        encoding='utf-8')
             ret = self.daemon.expect(["Ready to rumble",
                                       "Address already in use"])
             if ret == 0:
support/testing/tests/download/test_git.py +1 -1
index 2455557298..ec5b8f3fdd 100644
@@ -1,7 +1,7 @@
 import os
 import shutil
 
-from gitremote import GitRemote
+from tests.download.gitremote import GitRemote
 
 import infra
 
support/testing/tests/utils/test_check_package.py +2 -1
index 17c2fcf3bc..c70ba02324 100644
@@ -16,7 +16,8 @@ import infra
 def call_script(args, env, cwd):
     """Call a script and return stdout and stderr as lists."""
     out, err = subprocess.Popen(args, cwd=cwd, stdout=subprocess.PIPE,
-                                stderr=subprocess.PIPE, env=env).communicate()
+                                stderr=subprocess.PIPE, env=env,
+                                universal_newlines=True).communicate()
     return out.splitlines(), err.splitlines()
 
 
Signature · 1 line
2.20.1
#4 lore raw

PATCH v3 3/4 support/testing: switch to Python 3 only

Quoted 30 lines · Nicolas Carrier

From: Ricardo Martincoski <[email protected]>

Python 2.7 will not be maintained past 2020.

Many scripts on the tree are used during the build and should keep Python 2 compatibility for a while.
This is not the case for the runtime test infra. It's meant to be run in modern distros only, so it can safely switch to support Python 3 only.

An advantage of this approach is to have less scenarios to test in. Otherwise every change to the test infra or runtime tests would need to be tested against both versions of the interpreter, increasing the effort of the developers, to ensure the compatibility to Python 2 was not broken.

In order to accomplish the change to Python 3:
- change the shebang for run-tests;
- use Python 3 urllib as a drop-in replacement for Python 2 urllib2;
- when writing the downloaded binary files, explicitly open the output file as binary;
- when subprocess is used to retrieve the text output from commands, explicitly ask for text output. For this, use 'universal_newlines' because 'text' was added only on Python 3.7;
- when pexpect is used to retrieve the text output from qemu or git, explicitly ask for text output using 'encoding';
- the code using csv currently follows the example in the documentation for the Python 2 module, change it to follow the example in the documentation for the Python 3 module;
- fix the relative import for test_git.py to be Python 3 compliant.

Quoted 7 lines

Applied to master, thanks.

Regards,
Arnout

#5 lore raw

PATCH v3 4/4 support/testing: add bmap_tools test

In reply to Nicolas Carrier

This patch implements a simple test in which a dummy file system image is created, then bmaptool create and bmaptool copy are used to copy it to another file.

---

Changes v2 -> v3:
- title changed to "support/testing: add bmap_tools test"
I followed Arnout's suggestion, but if there's a veto on that I can still change...
- added myself in the DEVELOPERS file for both test scripts
- implemented genericity following Thomas' comment, i.e. without ABC, which results in a simpler, cleaner and shorter code (thank you btw ^^)
- fixed all the defects reported by flake8
- increased timeout to 20s

I saw that most of the python tests duplicate code between tests for v2 and v3. I think we should factor that, even if only small portions of code are concerned and, of course, I'm willing to do that if it's considered a good idea.

---

 DEVELOPERS                                    |  2 +
 .../tests/package/sample_bmap_tools.sh        | 15 ++++++
 .../testing/tests/package/test_bmap_tools.py  | 47 +++++++++++++++++++
 3 files changed, 64 insertions(+)
 create mode 100755 support/testing/tests/package/sample_bmap_tools.sh
 create mode 100644 support/testing/tests/package/test_bmap_tools.py
DEVELOPERS +2 -0
index 322201301c..7e8f537588 100644
@@ -1727,6 +1727,8 @@ F:	package/uacme/
 
 N:	Nicolas Carrier <[email protected]>
 F:	package/bmap-tools/
+F:	support/testing/tests/package/sample_bmap_tools.sh
+F:	support/testing/tests/package/test_bmap_tools.py
 
 N:	Nicolas Cavallari <[email protected]>
 F:	package/libgit2/
support/testing/tests/package/sample_bmap_tools.sh new file +15 -0
new file mode 100755
index 0000000000..7c90368c17
@@ -0,0 +1,15 @@
+#!/bin/sh
+# simple test which creates a dummy file system image, then use bmaptool create
+# and bmaptool copy to copy it to another file
+
+set -xeu
+
+# create the necessary test files
+dd if=/dev/zero of=disk.img bs=2M count=1
+mkfs.ext4 disk.img
+fallocate -d disk.img
+dd if=/dev/zero of=copy.img bs=2M count=1
+
+# do a test copy of the file system image
+bmaptool create -o disk.img.bmap disk.img
+bmaptool copy disk.img copy.img
support/testing/tests/package/test_bmap_tools.py new file +47 -0

47 lines not on this page · raw message

Signature · 1 line
2.20.1
#6 lore raw

PATCH v3 1/4 package/bmap-tools: new package

In reply to Nicolas Carrier

Hi Nicolas,

Le 27/10/2019 ? 14:36, Nicolas Carrier a ?crit?:

Quoted 61 lines

---

I need bmap-tools on the target since it's what our flashing / upgrade procudure uses to speed up the process (flashing ~8GB disk images full of zeroes).

Changes v2 -> v3:
- double-checked that setuptools is required by bmap-tools at runtime too
- moved the "v" from BMAP_TOOLS_VERSION to BMAP_TOOLS_SITE (in the .hash too)
- changed license name from GPLv2 to GPL-2.0

Changes v1 -> v2:
- no change

---
 DEVELOPERS                         |  3 +++
 package/Config.in                  |  1 +
 package/bmap-tools/Config.in       | 13 +++++++++++++
 package/bmap-tools/bmap-tools.hash |  2 ++
 package/bmap-tools/bmap-tools.mk   | 15 +++++++++++++++
 5 files changed, 34 insertions(+)
 create mode 100644 package/bmap-tools/Config.in
 create mode 100644 package/bmap-tools/bmap-tools.hash
 create mode 100644 package/bmap-tools/bmap-tools.mk
DEVELOPERS +3 -0
index 6efe52f19f..322201301c 100644
@@ -1725,6 +1725,9 @@ F:	package/libevdev/
 N:	Nicola Di Lieto <[email protected]>
 F:	package/uacme/
 
+N:	Nicolas Carrier <[email protected]>
+F:	package/bmap-tools/
+
 N:	Nicolas Cavallari <[email protected]>
 F:	package/libgit2/
 
package/Config.in +1 -0
index 901c25fe02..951342042a 100644
@@ -184,6 +184,7 @@ menu "Filesystem and flash utilities"
 	source "package/aufs/Config.in"
 	source "package/aufs-util/Config.in"
 	source "package/autofs/Config.in"
+	source "package/bmap-tools/Config.in"
 	source "package/btrfs-progs/Config.in"
 	source "package/cifs-utils/Config.in"
 	source "package/cpio/Config.in"
package/bmap-tools/Config.in new file +3 -0
new file mode 100644
index 0000000000..63301f2558
@@ -0,0 +1,13 @@
+config BR2_PACKAGE_BMAP_TOOLS
+	bool "bmap-tools"
+	depends on BR2_PACKAGE_PYTHON3 || \

package/bmap-tools/Config.in line 13

Quoted 10 lines
package/bmap-tools/Config.in new file +10 -0
+		BR2_PACKAGE_PYTHON
+	select BR2_PACKAGE_PYTHON_SIX # runtime
+	select BR2_PACKAGE_PYTHON_SETUPTOOLS
+	help
+	  Tool to flash image files to block devices using the block map
+	  bmaptool is a generic tool for creating the block map (bmap)
+	  for a file, and copying files using the block map. The idea is
+	  that large file containing unused blocks, like raw system
+	  image files, can be copied or flashed a lot faster with
+	  bmaptool than with traditional tools like "dd" or "cp".

We usually add a link to the upstream project in the last line of the help text:
https://github.com/intel/bmap-tools

Best regards,
Romain

Quoted 30 lines
package/bmap-tools/bmap-tools.hash new file +2 -0
new file mode 100644
index 0000000000..794765b633
@@ -0,0 +1,2 @@
+sha256  d410e2d97192d0fc2f88ef160a0bb6ed83fce99da97a606d7f6890cc654ec594  bmap-tools-3.5.tar.gz
+sha256  8177f97513213526df2cf6184d8ff986c675afb514d4e68a404010521b880643  COPYING
package/bmap-tools/bmap-tools.mk new file +15 -0
new file mode 100644
index 0000000000..4c2465876e
@@ -0,0 +1,15 @@
+################################################################################
+#
+# bmap-tools
+#
+################################################################################
+
+BMAP_TOOLS_VERSION = 3.5
+BMAP_TOOLS_SITE = $(call github,intel,bmap-tools,v$(BMAP_TOOLS_VERSION))
+BMAP_TOOLS_LICENSE = GPL-2.0
+BMAP_TOOLS_LICENSE_FILES = COPYING
+BMAP_TOOLS_SETUP_TYPE = setuptools
+BMAP_TOOLS_DEPENDENCIES = \
+	python-setuptools
+
+$(eval $(python-package))
#7 lore raw

PATCH v3 1/4 package/bmap-tools: new package

Quoted 2 lines · Romain Naour

Hi Nicolas,

Hello,

Quoted 8 lines

package/bmap-tools/Config.in line 13

Quoted 3 lines
package/bmap-tools/Config.in new file +3 -0
+       that large file containing unused blocks, like raw system
+       image files, can be copied or flashed a lot faster with
+       bmaptool than with traditional tools like "dd" or "cp".

We usually add a link to the upstream project in the last line of the help text:
https://github.com/intel/bmap-tools

Thank you, I fixed it in v4.

Quoted 5 lines

Best regards,
Romain