package/bmap-tools: new package
PATCH v3 1/4 package/bmap-tools: new package
---
I need bmap-tools on the target since it's what our flashing / upgrade procudure uses to speed up the process (flashing ~8GB disk images full of zeroes).
Changes v2 -> v3:
- double-checked that setuptools is required by bmap-tools at runtime too
- moved the "v" from BMAP_TOOLS_VERSION to BMAP_TOOLS_SITE (in the .hash too)
- changed license name from GPLv2 to GPL-2.0
Changes v1 -> v2:
- no change
--- DEVELOPERS | 3 +++ package/Config.in | 1 + package/bmap-tools/Config.in | 13 +++++++++++++ package/bmap-tools/bmap-tools.hash | 2 ++ package/bmap-tools/bmap-tools.mk | 15 +++++++++++++++ 5 files changed, 34 insertions(+) create mode 100644 package/bmap-tools/Config.in create mode 100644 package/bmap-tools/bmap-tools.hash create mode 100644 package/bmap-tools/bmap-tools.mk
DEVELOPERS +3 -0
@@ -1725,6 +1725,9 @@ F: package/libevdev/ N: Nicola Di Lieto <[email protected]> F: package/uacme/ +N: Nicolas Carrier <[email protected]> +F: package/bmap-tools/ + N: Nicolas Cavallari <[email protected]> F: package/libgit2/
package/Config.in +1 -0
@@ -184,6 +184,7 @@ menu "Filesystem and flash utilities" source "package/aufs/Config.in" source "package/aufs-util/Config.in" source "package/autofs/Config.in" + source "package/bmap-tools/Config.in" source "package/btrfs-progs/Config.in" source "package/cifs-utils/Config.in" source "package/cpio/Config.in"
package/bmap-tools/Config.in new file +13 -0
@@ -0,0 +1,13 @@ +config BR2_PACKAGE_BMAP_TOOLS + bool "bmap-tools" + depends on BR2_PACKAGE_PYTHON3 || \ + BR2_PACKAGE_PYTHON + select BR2_PACKAGE_PYTHON_SIX # runtime + select BR2_PACKAGE_PYTHON_SETUPTOOLS + help + Tool to flash image files to block devices using the block map + bmaptool is a generic tool for creating the block map (bmap) + for a file, and copying files using the block map. The idea is + that large file containing unused blocks, like raw system + image files, can be copied or flashed a lot faster with + bmaptool than with traditional tools like "dd" or "cp".
package/bmap-tools/bmap-tools.hash new file +2 -0
@@ -0,0 +1,2 @@ +sha256 d410e2d97192d0fc2f88ef160a0bb6ed83fce99da97a606d7f6890cc654ec594 bmap-tools-3.5.tar.gz +sha256 8177f97513213526df2cf6184d8ff986c675afb514d4e68a404010521b880643 COPYING
package/bmap-tools/bmap-tools.mk new file +15 -0
@@ -0,0 +1,15 @@ +################################################################################ +# +# bmap-tools +# +################################################################################ + +BMAP_TOOLS_VERSION = 3.5 +BMAP_TOOLS_SITE = $(call github,intel,bmap-tools,v$(BMAP_TOOLS_VERSION)) +BMAP_TOOLS_LICENSE = GPL-2.0 +BMAP_TOOLS_LICENSE_FILES = COPYING +BMAP_TOOLS_SETUP_TYPE = setuptools +BMAP_TOOLS_DEPENDENCIES = \ + python-setuptools + +$(eval $(python-package))
Signature · 1 line
2.20.1
PATCH v3 2/4 support/docker: add python3
From: Ricardo Martincoski <[email protected]>
The test infra will soon be converted to Python 3 only.
So add the interpreter and also the Python 3 variant of modules nose2 and pexpect to the docker image used to run runtime tests.
Keep the Python 2 variant of those modules to allow a gradual transition.
---
This is a re-submit of Ricardo Martincoski's original patch since it's a requirement the next patch (from Ricardo) which itself is required for the last patch of the series, which relies on python 3.6+ features (notably f-strings).
The 3 first patches of Ricardo's original series have already been applied, the two resubmitted here are the only ones left.
--- support/docker/Dockerfile | 3 +++ 1 file changed, 3 insertions(+)
support/docker/Dockerfile +3 -0
@@ -39,6 +39,9 @@ RUN apt-get install -y --no-install-recommends \ python-flake8 \ python-nose2 \ python-pexpect \ + python3 \ + python3-nose2 \ + python3-pexpect \ qemu-system-arm \ qemu-system-x86 \ rsync \
Signature · 1 line
2.20.1
Thomas Petazzoni
AppliedPATCH v3 2/4 support/docker: add python3
Quoted 6 lines · On Sun, 27 Oct 2019 13:36:53 +0000 Nicolas Carrier…
From: Ricardo Martincoski <[email protected]>
The test infra will soon be converted to Python 3 only.
So add the interpreter and also the Python 3 variant of modules nose2 and pexpect to the docker image used to run runtime tests.
Quoted 8 lines
Keep the Python 2 variant of those modules to allow a gradual transition.
Signed-off-by Ricardo Martincoski <[email protected]>Signed-off-by Nicolas Carrier <[email protected]>Cc Yann E. MORIN <[email protected]>---
Applied to master, thanks.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH v3 3/4 support/testing: switch to Python 3 only
In reply to Nicolas CarrierFrom: Ricardo Martincoski <[email protected]>
Python 2.7 will not be maintained past 2020.
Many scripts on the tree are used during the build and should keep Python 2 compatibility for a while.
This is not the case for the runtime test infra. It's meant to be run in modern distros only, so it can safely switch to support Python 3 only.
An advantage of this approach is to have less scenarios to test in. Otherwise every change to the test infra or runtime tests would need to be tested against both versions of the interpreter, increasing the effort of the developers, to ensure the compatibility to Python 2 was not broken.
In order to accomplish the change to Python 3:
- change the shebang for run-tests;
- use Python 3 urllib as a drop-in replacement for Python 2 urllib2;
- when writing the downloaded binary files, explicitly open the output file as binary;
- when subprocess is used to retrieve the text output from commands, explicitly ask for text output. For this, use 'universal_newlines' because 'text' was added only on Python 3.7;
- when pexpect is used to retrieve the text output from qemu or git, explicitly ask for text output using 'encoding';
- the code using csv currently follows the example in the documentation for the Python 2 module, change it to follow the example in the documentation for the Python 3 module;
- fix the relative import for test_git.py to be Python 3 compliant.
---
This is a re-submit of Ricardo Martincoski's original patch since it's required for the next (and last patch) of the series, which relies on python 3.6+ features (notably f-strings).
Original comment of Romain Naour on the original series:
Quoted 2 lines
I'm using Fedora 30 and there no python2-nose2 package anymore. So, I'm using this patch to avoid installing python2-nose2 from pypi.
I added my Tested-by tag since I used it for running the tests that are in the next patch.
--- support/testing/infra/__init__.py | 8 +++++--- support/testing/infra/emulator.py | 1 + support/testing/run-tests | 2 +- support/testing/tests/core/test_post_scripts.py | 2 +- support/testing/tests/download/gitremote.py | 3 ++- support/testing/tests/download/test_git.py | 2 +- support/testing/tests/utils/test_check_package.py | 3 ++- 7 files changed, 13 insertions(+), 8 deletions(-)
support/testing/infra/__init__.py +5 -3
@@ -3,7 +3,8 @@ import re import sys import tempfile import subprocess -from urllib2 import urlopen, HTTPError, URLError +from urllib.request import urlopen +from urllib.error import HTTPError, URLError ARTIFACTS_URL = "http://autobuild.buildroot.net/artefacts/" BASE_DIR = os.path.realpath(os.path.join(os.path.dirname(__file__), "../../.."))
@@ -44,7 +45,7 @@ def download(dldir, filename): try: url_fh = urlopen(os.path.join(ARTIFACTS_URL, filename)) - with open(tmpfile, "w+") as tmpfile_fh: + with open(tmpfile, "w+b") as tmpfile_fh: tmpfile_fh.write(url_fh.read()) except (HTTPError, URLError) as err: os.unlink(tmpfile)
@@ -60,7 +61,8 @@ def run_cmd_on_host(builddir, cmd): out = subprocess.check_output(cmd, stderr=open(os.devnull, "w"), cwd=builddir, - env={"LANG": "C"}) + env={"LANG": "C"}, + universal_newlines=True) return out
support/testing/infra/emulator.py +1 -0
@@ -76,6 +76,7 @@ class Emulator(object): self.logfile.write("> starting qemu with '%s'\n" % " ".join(qemu_cmd)) self.qemu = pexpect.spawn(qemu_cmd[0], qemu_cmd[1:], timeout=5 * self.timeout_multiplier, + encoding='utf-8', env={"QEMU_AUDIO_DRV": "none"}) # We want only stdout into the log to avoid double echo self.qemu.logfile_read = self.logfile
support/testing/run-tests +1 -1
@@ -1,4 +1,4 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 import argparse import sys import os
support/testing/tests/core/test_post_scripts.py +1 -1
@@ -20,7 +20,7 @@ class TestPostScripts(infra.basetest.BRTest): def check_post_log_file(self, f, what, target_dir): lines = {} - with open(os.path.join(self.builddir, "build", f), 'rb') as csvfile: + with open(os.path.join(self.builddir, "build", f), newline='') as csvfile: r = csv.reader(csvfile, delimiter=',') for row in r: lines[row[0]] = row[1]
support/testing/tests/download/gitremote.py +2 -1
@@ -32,7 +32,8 @@ class GitRemote(object): for port in range(GIT_REMOTE_PORT_INITIAL, GIT_REMOTE_PORT_LAST + 1): cmd = daemon_cmd + ["--port={port}".format(port=port)] self.logfile.write("> starting git remote with '{}'\n".format(" ".join(cmd))) - self.daemon = pexpect.spawn(cmd[0], cmd[1:], logfile=self.logfile) + self.daemon = pexpect.spawn(cmd[0], cmd[1:], logfile=self.logfile, + encoding='utf-8') ret = self.daemon.expect(["Ready to rumble", "Address already in use"]) if ret == 0:
support/testing/tests/download/test_git.py +1 -1
@@ -1,7 +1,7 @@ import os import shutil -from gitremote import GitRemote +from tests.download.gitremote import GitRemote import infra
support/testing/tests/utils/test_check_package.py +2 -1
@@ -16,7 +16,8 @@ import infra def call_script(args, env, cwd): """Call a script and return stdout and stderr as lists.""" out, err = subprocess.Popen(args, cwd=cwd, stdout=subprocess.PIPE, - stderr=subprocess.PIPE, env=env).communicate() + stderr=subprocess.PIPE, env=env, + universal_newlines=True).communicate() return out.splitlines(), err.splitlines()
Signature · 1 line
2.20.1
Arnout Vandecappelle
AppliedPATCH v3 3/4 support/testing: switch to Python 3 only
Quoted 30 lines · Nicolas Carrier
From: Ricardo Martincoski <[email protected]>
Python 2.7 will not be maintained past 2020.
Many scripts on the tree are used during the build and should keep Python 2 compatibility for a while.
This is not the case for the runtime test infra. It's meant to be run in modern distros only, so it can safely switch to support Python 3 only.An advantage of this approach is to have less scenarios to test in. Otherwise every change to the test infra or runtime tests would need to be tested against both versions of the interpreter, increasing the effort of the developers, to ensure the compatibility to Python 2 was not broken.
In order to accomplish the change to Python 3:
- change the shebang for run-tests;
- use Python 3 urllib as a drop-in replacement for Python 2 urllib2;
- when writing the downloaded binary files, explicitly open the output file as binary;
- when subprocess is used to retrieve the text output from commands, explicitly ask for text output. For this, use 'universal_newlines' because 'text' was added only on Python 3.7;
- when pexpect is used to retrieve the text output from qemu or git, explicitly ask for text output using 'encoding';
- the code using csv currently follows the example in the documentation for the Python 2 module, change it to follow the example in the documentation for the Python 3 module;
- fix the relative import fortest_git.pyto be Python 3 compliant.
Quoted 7 lines
Signed-off-by Ricardo Martincoski <[email protected]>Cc Arnout Vandecappelle <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Yann E. MORIN <[email protected]>Tested-by Romain Naour <[email protected]>Tested-by Nicolas Carrier <[email protected]>Signed-off-by Nicolas Carrier <[email protected]>
Applied to master, thanks.
Regards,
Arnout
PATCH v3 4/4 support/testing: add bmap_tools test
In reply to Nicolas CarrierThis patch implements a simple test in which a dummy file system image is created, then bmaptool create and bmaptool copy are used to copy it to another file.
---
Changes v2 -> v3:
- title changed to "support/testing: add bmap_tools test"
I followed Arnout's suggestion, but if there's a veto on that I can still change...
- added myself in the DEVELOPERS file for both test scripts
- implemented genericity following Thomas' comment, i.e. without ABC, which results in a simpler, cleaner and shorter code (thank you btw ^^)
- fixed all the defects reported by flake8
- increased timeout to 20s
I saw that most of the python tests duplicate code between tests for v2 and v3. I think we should factor that, even if only small portions of code are concerned and, of course, I'm willing to do that if it's considered a good idea.
---
DEVELOPERS | 2 + .../tests/package/sample_bmap_tools.sh | 15 ++++++ .../testing/tests/package/test_bmap_tools.py | 47 +++++++++++++++++++ 3 files changed, 64 insertions(+) create mode 100755 support/testing/tests/package/sample_bmap_tools.sh create mode 100644 support/testing/tests/package/test_bmap_tools.py
DEVELOPERS +2 -0
@@ -1727,6 +1727,8 @@ F: package/uacme/ N: Nicolas Carrier <[email protected]> F: package/bmap-tools/ +F: support/testing/tests/package/sample_bmap_tools.sh +F: support/testing/tests/package/test_bmap_tools.py N: Nicolas Cavallari <[email protected]> F: package/libgit2/
support/testing/tests/package/sample_bmap_tools.sh new file +15 -0
@@ -0,0 +1,15 @@ +#!/bin/sh +# simple test which creates a dummy file system image, then use bmaptool create +# and bmaptool copy to copy it to another file + +set -xeu + +# create the necessary test files +dd if=/dev/zero of=disk.img bs=2M count=1 +mkfs.ext4 disk.img +fallocate -d disk.img +dd if=/dev/zero of=copy.img bs=2M count=1 + +# do a test copy of the file system image +bmaptool create -o disk.img.bmap disk.img +bmaptool copy disk.img copy.img
support/testing/tests/package/test_bmap_tools.py new file +47 -0
47 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH v3 1/4 package/bmap-tools: new package
In reply to Nicolas CarrierHi Nicolas,
Le 27/10/2019 ? 14:36, Nicolas Carrier a ?crit?:
Quoted 61 lines
Signed-off-by Nicolas Carrier <[email protected]>---
I need bmap-tools on the target since it's what our flashing / upgrade procudure uses to speed up the process (flashing ~8GB disk images full of zeroes).
Changes v2 -> v3:
- double-checked that setuptools is required by bmap-tools at runtime too
- moved the "v" fromBMAP_TOOLS_VERSIONtoBMAP_TOOLS_SITE(in the .hash too)
- changed license name from GPLv2 to GPL-2.0Changes v1 -> v2:
- no change--- DEVELOPERS | 3 +++ package/Config.in | 1 + package/bmap-tools/Config.in | 13 +++++++++++++ package/bmap-tools/bmap-tools.hash | 2 ++ package/bmap-tools/bmap-tools.mk | 15 +++++++++++++++ 5 files changed, 34 insertions(+) create mode 100644 package/bmap-tools/Config.in create mode 100644 package/bmap-tools/bmap-tools.hash create mode 100644 package/bmap-tools/bmap-tools.mkDEVELOPERS +3
-0@@ -1725,6 +1725,9 @@ F: package/libevdev/ N: Nicola Di Lieto <[email protected]> F: package/uacme/ +N: Nicolas Carrier <[email protected]> +F: package/bmap-tools/ + N: Nicolas Cavallari <[email protected]> F: package/libgit2/package/Config.in +1
-0@@ -184,6 +184,7 @@ menu "Filesystem and flash utilities" source "package/aufs/Config.in" source "package/aufs-util/Config.in" source "package/autofs/Config.in" + source "package/bmap-tools/Config.in" source "package/btrfs-progs/Config.in" source "package/cifs-utils/Config.in" source "package/cpio/Config.in"package/bmap-tools/Config.in new file +3
-0@@ -0,0 +1,13 @@ +config BR2_PACKAGE_BMAP_TOOLS + bool "bmap-tools" + depends on BR2_PACKAGE_PYTHON3 || \
package/bmap-tools/Config.in line 13
Quoted 10 lines
package/bmap-tools/Config.in new file +10
-0+ BR2_PACKAGE_PYTHON + select BR2_PACKAGE_PYTHON_SIX # runtime + select BR2_PACKAGE_PYTHON_SETUPTOOLS + help + Tool to flash image files to block devices using the block map + bmaptool is a generic tool for creating the block map (bmap) + for a file, and copying files using the block map. The idea is + that large file containing unused blocks, like raw system + image files, can be copied or flashed a lot faster with + bmaptool than with traditional tools like "dd" or "cp".
We usually add a link to the upstream project in the last line of the help text:
https://github.com/intel/bmap-tools
Best regards,
Romain
Quoted 30 lines
package/bmap-tools/bmap-tools.hash new file +2
-0@@ -0,0 +1,2 @@ +sha256 d410e2d97192d0fc2f88ef160a0bb6ed83fce99da97a606d7f6890cc654ec594 bmap-tools-3.5.tar.gz +sha256 8177f97513213526df2cf6184d8ff986c675afb514d4e68a404010521b880643 COPYINGpackage/bmap-tools/bmap-tools.mk new file +15
-0@@ -0,0 +1,15 @@ +################################################################################ +# +# bmap-tools +# +################################################################################ + +BMAP_TOOLS_VERSION = 3.5 +BMAP_TOOLS_SITE = $(call github,intel,bmap-tools,v$(BMAP_TOOLS_VERSION)) +BMAP_TOOLS_LICENSE = GPL-2.0 +BMAP_TOOLS_LICENSE_FILES = COPYING +BMAP_TOOLS_SETUP_TYPE = setuptools +BMAP_TOOLS_DEPENDENCIES = \ + python-setuptools + +$(eval $(python-package))
PATCH v3 1/4 package/bmap-tools: new package
Quoted 2 lines · Romain Naour
Hi Nicolas,
Hello,
Quoted 8 lines
package/bmap-tools/Config.in line 13
Quoted 3 lines
package/bmap-tools/Config.in new file +3
-0+ that large file containing unused blocks, like raw system + image files, can be copied or flashed a lot faster with + bmaptool than with traditional tools like "dd" or "cp".We usually add a link to the upstream project in the last line of the help text:
https://github.com/intel/bmap-tools
Thank you, I fixed it in v4.
Quoted 5 lines
Reviewed-by Romain Naour <[email protected]>Best regards,
Romain