fdt: Fix up tests and build warnings on newer distros
cover fdt: Fix up tests and build warnings on newer distros
At present the code coverage for test_fdt is not working. This series fixes that and takes the opportunity to fix the pylint warnings in that file.
Some newer Linux distributions are producing warnings about using distutils, so this series moves things over to setuptools.
Simon Glass (7):
dtoc: Tidy up fdt_tests RunTestCoverage() args
dtoc: Tidy up fdt_tests RunTests()
dtoc: Fix fdt test coverage
dtoc: Move main program into its own function
test_fdt: Convert to use argparse
dtoc: Correct remaining pylint problems in test_fdt
fdt: Move to setuptools
scripts/dtc/README | 106 +++++++++++ scripts/dtc/pylibfdt/Makefile | 5 +- scripts/dtc/pylibfdt/setup.py | 60 ++++++- scripts/pylint.base | 2 +- tools/dtoc/test_fdt.py | 326 +++++++++++++++++++--------------- 5 files changed, 345 insertions(+), 154 deletions(-) create mode 100644 scripts/dtc/README
Signature · 1 line
2.37.1.455.g008518b4e5-goog
PATCH 1/7 dtoc: Tidy up fdt_tests RunTestCoverage() args
Pass the options args in rather than using the global various. Use snake case and fix up comments to make pylint happy.
---
tools/dtoc/test_fdt.py | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
tools/dtoc/test_fdt.py +8 -4
@@ -768,10 +768,14 @@ class TestFdtUtil(unittest.TestCase): tools.outdir= old_outdir -def RunTestCoverage(): - """Run the tests and check that we get 100% coverage""" +def run_test_coverage(build_dir): + """Run the tests and check that we get 100% coverage + + Args: + build_dir (str): Directory containing the build output + """ test_util.run_test_coverage('tools/dtoc/test_fdt.py', None, - ['tools/patman/*.py', '*test_fdt.py'], options.build_dir) + ['tools/patman/*.py', '*test_fdt.py'], build_dir) def RunTests(args):
@@ -811,4 +815,4 @@ if options.test: ret_code = RunTests(args) sys.exit(ret_code) elif options.test_coverage: - RunTestCoverage() + run_test_coverage(options.build_dir)
Signature · 1 line
2.37.1.455.g008518b4e5-goog
Re: [PATCH 1/7] dtoc: Tidy up fdt_tests RunTestCoverage() args
Pass the options args in rather than using the global various. Use snake case and fix up comments to make pylint happy.
---
tools/dtoc/test_fdt.py | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
Applied to u-boot-dm, thanks!
PATCH 2/7 dtoc: Tidy up fdt_tests RunTests()
In reply to Simon GlassPass the options args in rather than using the global variables. Use snake case, fix up comments and use a ternary operator to make pylint happy.
---
tools/dtoc/test_fdt.py | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
tools/dtoc/test_fdt.py +9 -6
@@ -778,17 +778,20 @@ def run_test_coverage(build_dir): ['tools/patman/*.py', '*test_fdt.py'], build_dir) -def RunTests(args): +def run_tests(args, processes): """Run all the test we have for the fdt model Args: - args: List of positional args provided to fdt. This can hold a test - name to execute (as in 'fdt -t testFdt', for example) + args (list or str): List of positional args provided. This can hold a + test name to execute (as in 'test_fdt -t testFdt', for example) + processes (int): Number of processes to use (None means as many as there + are CPUs on the system. This must be set to 1 when running under + the python3-coverage tool Returns: - Return code, 0 on success + int: Return code, 0 on success """ - test_name = args and args[0] or None + test_name = args[0] if args else None result = test_util.run_test_suites( 'test_fdt', False, False, False, None, test_name, None, [TestFdt, TestNode, TestProp, TestFdtUtil])
@@ -812,7 +815,7 @@ parser.add_option('-T', '--test-coverage', action='store_true', # Run our meagre tests if options.test: - ret_code = RunTests(args) + ret_code = run_tests(args, options.processes) sys.exit(ret_code) elif options.test_coverage: run_test_coverage(options.build_dir)
Signature · 1 line
2.37.1.455.g008518b4e5-goog
Re: [PATCH 2/7] dtoc: Tidy up fdt_tests RunTests()
Pass the options args in rather than using the global variables. Use snake case, fix up comments and use a ternary operator to make pylint happy.
---
tools/dtoc/test_fdt.py | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
Applied to u-boot-dm, thanks!
PATCH 3/7 dtoc: Fix fdt test coverage
In reply to Simon GlassFix a bug that the --processes option was ignored, thus resulting in no test coverage information being generated.
---
tools/dtoc/test_fdt.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/dtoc/test_fdt.py +1 -1
@@ -793,7 +793,7 @@ def run_tests(args, processes): """ test_name = args[0] if args else None result = test_util.run_test_suites( - 'test_fdt', False, False, False, None, test_name, None, + 'test_fdt', False, False, False, processes, test_name, None, [TestFdt, TestNode, TestProp, TestFdtUtil]) return (0 if result.wasSuccessful() else 1)
Signature · 1 line
2.37.1.455.g008518b4e5-goog
Re: [PATCH 3/7] dtoc: Fix fdt test coverage
Fix a bug that the --processes option was ignored, thus resulting in no test coverage information being generated.
---
tools/dtoc/test_fdt.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Applied to u-boot-dm, thanks!
PATCH 4/7 dtoc: Move main program into its own function
In reply to Simon GlassUse a function for the main program so everything there doesn't look like a global variable to pylint.
---
tools/dtoc/test_fdt.py | 44 +++++++++++++++++++++++------------------- 1 file changed, 24 insertions(+), 20 deletions(-)
tools/dtoc/test_fdt.py +24 -20
@@ -799,23 +799,27 @@ def run_tests(args, processes): return (0 if result.wasSuccessful() else 1) -if __name__ != '__main__': - sys.exit(1) - -parser = OptionParser() -parser.add_option('-B', '--build-dir', type='string', default='b', - help='Directory containing the build output') -parser.add_option('-P', '--processes', type=int, - help='set number of processes to use for running tests') -parser.add_option('-t', '--test', action='store_true', dest='test', - default=False, help='run tests') -parser.add_option('-T', '--test-coverage', action='store_true', - default=False, help='run tests and check for 100% coverage') -(options, args) = parser.parse_args() - -# Run our meagre tests -if options.test: - ret_code = run_tests(args, options.processes) - sys.exit(ret_code) -elif options.test_coverage: - run_test_coverage(options.build_dir) +def main(): + """Main program for this tool""" + parser = OptionParser() + parser.add_option('-B', '--build-dir', type='string', default='b', + help='Directory containing the build output') + parser.add_option('-P', '--processes', type=int, + help='set number of processes to use for running tests') + parser.add_option('-t', '--test', action='store_true', dest='test', + default=False, help='run tests') + parser.add_option('-T', '--test-coverage', action='store_true', + default=False, help='run tests and check for 100% coverage') + (options, args) = parser.parse_args() + + # Run our meagre tests + if options.test: + ret_code = run_tests(args, options.processes) + return ret_code + if options.test_coverage: + run_test_coverage(options.build_dir) + return 0 + +if __name__ == '__main__': + sys.exit(main()) +sys.exit(1)
Signature · 1 line
2.37.1.455.g008518b4e5-goog
Re: [PATCH 4/7] dtoc: Move main program into its own function
Use a function for the main program so everything there doesn't look like a global variable to pylint.
---
tools/dtoc/test_fdt.py | 44 +++++++++++++++++++++++------------------- 1 file changed, 24 insertions(+), 20 deletions(-)
Applied to u-boot-dm, thanks!
PATCH 5/7 test_fdt: Convert to use argparse
In reply to Simon GlassDrop the deprecated OptionParser.
---
tools/dtoc/test_fdt.py | 39 ++++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 19 deletions(-)
tools/dtoc/test_fdt.py +20 -19
@@ -4,7 +4,7 @@ # Written by Simon Glass <[email protected]> # -from optparse import OptionParser +from argparse import ArgumentParser import glob import os import shutil
@@ -778,12 +778,11 @@ def run_test_coverage(build_dir): ['tools/patman/*.py', '*test_fdt.py'], build_dir) -def run_tests(args, processes): +def run_tests(names, processes): """Run all the test we have for the fdt model Args: - args (list or str): List of positional args provided. This can hold a - test name to execute (as in 'test_fdt -t testFdt', for example) + names (list of str): List of test names provided. Only the first is used processes (int): Number of processes to use (None means as many as there are CPUs on the system. This must be set to 1 when running under the python3-coverage tool
@@ -791,7 +790,7 @@ def run_tests(args, processes): Returns: int: Return code, 0 on success """ - test_name = args[0] if args else None + test_name = names[0] if names else None result = test_util.run_test_suites( 'test_fdt', False, False, False, processes, test_name, None, [TestFdt, TestNode, TestProp, TestFdtUtil])
@@ -801,23 +800,25 @@ def run_tests(args, processes): def main(): """Main program for this tool""" - parser = OptionParser() - parser.add_option('-B', '--build-dir', type='string', default='b', - help='Directory containing the build output') - parser.add_option('-P', '--processes', type=int, - help='set number of processes to use for running tests') - parser.add_option('-t', '--test', action='store_true', dest='test', - default=False, help='run tests') - parser.add_option('-T', '--test-coverage', action='store_true', - default=False, help='run tests and check for 100% coverage') - (options, args) = parser.parse_args() + parser = ArgumentParser() + parser.add_argument('-B', '--build-dir', type=str, default='b', + help='Directory containing the build output') + parser.add_argument('-P', '--processes', type=int, + help='set number of processes to use for running tests') + parser.add_argument('-t', '--test', action='store_true', dest='test', + default=False, help='run tests') + parser.add_argument('-T', '--test-coverage', action='store_true', + default=False, + help='run tests and check for 100% coverage') + parser.add_argument('name', nargs='*') + args = parser.parse_args() # Run our meagre tests - if options.test: - ret_code = run_tests(args, options.processes) + if args.test: + ret_code = run_tests(args.name, args.processes) return ret_code - if options.test_coverage: - run_test_coverage(options.build_dir) + if args.test_coverage: + run_test_coverage(args.build_dir) return 0 if __name__ == '__main__':
Signature · 1 line
2.37.1.455.g008518b4e5-goog
Re: [PATCH 5/7] test_fdt: Convert to use argparse
Drop the deprecated OptionParser.
---
tools/dtoc/test_fdt.py | 39 ++++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 19 deletions(-)
Applied to u-boot-dm, thanks!
PATCH 6/7 dtoc: Correct remaining pylint problems in test_fdt
In reply to Simon GlassFix various camel-case and other naming problems. Update the pylint base file to avoid regressions.
---
scripts/pylint.base | 2 +- tools/dtoc/test_fdt.py | 254 +++++++++++++++++++++++------------------ 2 files changed, 142 insertions(+), 114 deletions(-)
scripts/pylint.base +1 -1
@@ -167,7 +167,7 @@ tools_binman_etype_x86_reset16_tpl -15.71 tools_binman_etype_x86_start16 -15.71 tools_binman_etype_x86_start16_spl -15.71 tools_binman_etype_x86_start16_tpl -15.71 -tools_binman_fdt_test 3.23 +tools_binman_fdt_test 10.00 tools_binman_fip_util 9.85 tools_binman_fip_util_test 10.00 tools_binman_fmap_util 6.88
tools/dtoc/test_fdt.py +141 -113
594 lines not on this page · raw message
Signature · 1 line
2.37.1.455.g008518b4e5-goog
Re: [PATCH 6/7] dtoc: Correct remaining pylint problems in test_fdt
Fix various camel-case and other naming problems. Update the pylint base file to avoid regressions.
---
scripts/pylint.base | 2 +- tools/dtoc/test_fdt.py | 254 +++++++++++++++++++++++------------------ 2 files changed, 142 insertions(+), 114 deletions(-)
Applied to u-boot-dm, thanks!
PATCH 7/7 fdt: Move to setuptools
In reply to Simon GlassThe distutils package is deprecated. The upstream libfdt repo uses setuptools for building the pylibfdt module, so bring in that code, suitably modified for U-Boot. Also bring in the README.
The modifications include setting the version correctly, making use of the environment variables provided by the Makefile and various tweaks to the directories.
Note that the version omits the minus character at the start of EXTRAVERSION, since this creates a warning. The build is really just used within U-Boot itself, so it doesn't matter too much if the version matches upstream, or exactly matches U-Boot.
---
scripts/dtc/README | 106 ++++++++++++++++++++++++++++++++++ scripts/dtc/pylibfdt/Makefile | 5 +- scripts/dtc/pylibfdt/setup.py | 60 ++++++++++++++++--- 3 files changed, 161 insertions(+), 10 deletions(-) create mode 100644 scripts/dtc/README
scripts/dtc/README new file +106 -0
@@ -0,0 +1,106 @@ +The source tree contains the Device Tree Compiler (dtc) toolchain for +working with device tree source and binary files and also libfdt, a +utility library for reading and manipulating the binary format. + +DTC and LIBFDT are maintained by: + +David Gibson <[email protected]> +Jon Loeliger <[email protected]> + + +Python library +-------------- + +A Python library is also available. To build this you will need to install +swig and Python development files. On Debian distributions: + + sudo apt-get install swig python3-dev + +The library provides an Fdt class which you can use like this: + +$ PYTHONPATH=../pylibfdt python3 +>>> import libfdt +>>> fdt = libfdt.Fdt(open('test_tree1.dtb', mode='rb').read()) +>>> node = fdt.path_offset('/subnode@1') +>>> print(node) +124 +>>> prop_offset = fdt.first_property_offset(node) +>>> prop = fdt.get_property_by_offset(prop_offset) +>>> print('%s=%s' % (prop.name, prop.as_str())) +compatible=subnode1 +>>> node2 = fdt.path_offset('/') +>>> print(fdt.getprop(node2, 'compatible').as_str()) +test_tree1 + +You will find tests in tests/pylibfdt_tests.py showing how to use each +method. Help is available using the Python help command, e.g.: + + $ cd pylibfdt + $ python3 -c "import libfdt; help(libfdt)" + +If you add new features, please check code coverage: + + $ sudo apt-get install python3-coverage + $ cd tests + # It's just 'coverage' on most other distributions + $ python3-coverage run pylibfdt_tests.py + $ python3-coverage html + # Open 'htmlcov/index.html' in your browser + + +The library can be installed with pip from a local source tree: + + pip install . [--user|--prefix=/path/to/install_dir] + +Or directly from a remote git repo: + + pip install git+git://git.kernel.org/pub/scm/utils/dtc/dtc.git@main + +The install depends on libfdt shared library being installed on the host system +first. Generally, using --user or --prefix is not necessary and pip will use the +default location for the Python installation which varies if the user is root or +not. + +You can also install everything via make if you like, but pip is recommended. + +To install both libfdt and pylibfdt you can use: + + make install [PREFIX=/path/to/install_dir] + +To disable building the python library, even if swig and Python are available, +use: + + make NO_PYTHON=1 + + +More work remains to support all of libfdt, including access to numeric +values. + + +Adding a new function to libfdt.h +--------------------------------- + +The shared library uses libfdt/version.lds to list the exported functions, so +add your new function there. Check that your function works with pylibfdt. If +it cannot be supported, put the declaration in libfdt.h behind #ifndef SWIG so +that swig ignores it. + + +Tests +----- + +Test files are kept in the tests/ directory. Use 'make check' to build and run +all tests. + +If you want to adjust a test file, be aware that tree_tree1.dts is compiled +and checked against a binary tree from assembler macros in trees.S. So +if you change that file you must change tree.S also. + + +Mailing list +------------ +The following list is for discussion about dtc and libfdt implementation +mailto:[email protected] + +Core device tree bindings are discussed on the devicetree-spec list: +mailto:[email protected]
scripts/dtc/pylibfdt/Makefile +4 -1
@@ -13,11 +13,14 @@ include $(LIBFDT_srcdir)/Makefile.libfdt PYLIBFDT_srcs = $(addprefix $(LIBFDT_srcdir)/,$(LIBFDT_SRCS)) \ $(obj)/libfdt.i +# create a version string compliant with PEP 440 +PEP_VERSION = $(VERSION)$(if $(PATCHLEVEL),.$(PATCHLEVEL)$(if $(SUBLEVEL),.$(SUBLEVEL)))$(subst -,,$(EXTRAVERSION)) + quiet_cmd_pymod = PYMOD $@ cmd_pymod = unset CROSS_COMPILE; unset CFLAGS; \ CC="$(HOSTCC)" LDSHARED="$(HOSTCC) -shared " \ LDFLAGS="$(HOSTLDFLAGS)" \ - VERSION="u-boot-$(UBOOTVERSION)" \ + VERSION="$(PEP_VERSION)" \ CPPFLAGS="$(HOSTCFLAGS) -I$(LIBFDT_srcdir)" OBJDIR=$(obj) \ SOURCES="$(PYLIBFDT_srcs)" \ SWIG_OPTS="-I$(LIBFDT_srcdir) -I$(LIBFDT_srcdir)/.." \
scripts/dtc/pylibfdt/setup.py +51 -9
105 lines not on this page · raw message
Signature · 1 line
2.37.1.455.g008518b4e5-goog
Re: [PATCH 7/7] fdt: Move to setuptools
Quoted 13 lines · Simon Glass
The distutils package is deprecated. The upstream libfdt repo uses setuptools for building the pylibfdt module, so bring in that code, suitably modified for U-Boot. Also bring in the README.
The modifications include setting the version correctly, making use of the environment variables provided by the
Makefileand various tweaks to the directories.Note that the version omits the minus character at the start of EXTRAVERSION, since this creates a warning. The build is really just used within U-Boot itself, so it doesn't matter too much if the version matches upstream, or exactly matches U-Boot.
Quoted 8 lines
Signed-off-by Simon Glass <[email protected]>---
scripts/dtc/README | 106 ++++++++++++++++++++++++++++++++++ scripts/dtc/pylibfdt/Makefile | 5 +- scripts/dtc/pylibfdt/setup.py | 60 ++++++++++++++++--- 3 files changed, 161 insertions(+), 10 deletions(-) create mode 100644 scripts/dtc/README
This isn't new enough, however:/usr/lib/python3/dist-packages/setuptools/installer.py:27:
SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
when trying in Ubuntu 22.04 (the trini/u-boot-gitlab-ci-runner:jammy-20220531-13Jul2022 docker specifically).
Signature · 1 line
Tom
Re: [PATCH 7/7] fdt: Move to setuptools
Hi Tom,
Quoted 25 lines · Tom Rini
Quoted 13 lines · Simon Glass
The distutils package is deprecated. The upstream libfdt repo uses setuptools for building the pylibfdt module, so bring in that code, suitably modified for U-Boot. Also bring in the README.
The modifications include setting the version correctly, making use of the environment variables provided by the
Makefileand various tweaks to the directories.Note that the version omits the minus character at the start of EXTRAVERSION, since this creates a warning. The build is really just used within U-Boot itself, so it doesn't matter too much if the version matches upstream, or exactly matches U-Boot.
Quoted 8 lines
Signed-off-by Simon Glass <[email protected]>---
scripts/dtc/README | 106 ++++++++++++++++++++++++++++++++++ scripts/dtc/pylibfdt/Makefile | 5 +- scripts/dtc/pylibfdt/setup.py | 60 ++++++++++++++++--- 3 files changed, 161 insertions(+), 10 deletions(-) create mode 100644 scripts/dtc/README
Quoted 6 lines
This isn't new enough, however:
/usr/lib/python3/dist-packages/setuptools/installer.py:27:
SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
when trying in Ubuntu 22.04 (the trini/u-boot-gitlab-ci-runner:jammy-20220531-13Jul2022 docker specifically).
+Rob Herring
Yes I see that. But it is a start!
Also I was about to send a pull request with this...should I take it out?
Regards,
Simon
Re: [PATCH 7/7] fdt: Move to setuptools
Quoted 35 lines · Simon Glass
Hi Tom,
Quoted 25 lines · Tom Rini
Quoted 13 lines · Simon Glass
The distutils package is deprecated. The upstream libfdt repo uses setuptools for building the pylibfdt module, so bring in that code, suitably modified for U-Boot. Also bring in the README.
The modifications include setting the version correctly, making use of the environment variables provided by the
Makefileand various tweaks to the directories.Note that the version omits the minus character at the start of EXTRAVERSION, since this creates a warning. The build is really just used within U-Boot itself, so it doesn't matter too much if the version matches upstream, or exactly matches U-Boot.
Quoted 8 lines
Signed-off-by Simon Glass <[email protected]>---
scripts/dtc/README | 106 ++++++++++++++++++++++++++++++++++ scripts/dtc/pylibfdt/Makefile | 5 +- scripts/dtc/pylibfdt/setup.py | 60 ++++++++++++++++--- 3 files changed, 161 insertions(+), 10 deletions(-) create mode 100644 scripts/dtc/READMEQuoted 6 lines
This isn't new enough, however:
/usr/lib/python3/dist-packages/setuptools/installer.py:27:
SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
when trying in Ubuntu 22.04 (the trini/u-boot-gitlab-ci-runner:jammy-20220531-13Jul2022 docker specifically).
Quoted 5 lines
+Rob Herring
Yes I see that. But it is a start!
Also I was about to send a pull request with this...should I take it out?
Well, it brings us in sync with upstream I see, so yes, we should take this now and work on getting the next warning resolved. For why I hit this again, I'm just shuffling around where we set -e/+e.
Signature · 1 line
Tom