Skip to contentSkip to conversation

fdt: Fix up tests and build warnings on newer distros

PATCH In review Simon Glass 2 participants 17 messages Jul 31, 2022 – Aug 9, 2022 Device tree

lore
Share & tools
lore
#0 lore raw

cover fdt: Fix up tests and build warnings on newer distros

At present the code coverage for test_fdt is not working. This series fixes that and takes the opportunity to fix the pylint warnings in that file.

Some newer Linux distributions are producing warnings about using distutils, so this series moves things over to setuptools.

Simon Glass (7):
dtoc: Tidy up fdt_tests RunTestCoverage() args
dtoc: Tidy up fdt_tests RunTests()
dtoc: Fix fdt test coverage
dtoc: Move main program into its own function
test_fdt: Convert to use argparse
dtoc: Correct remaining pylint problems in test_fdt
fdt: Move to setuptools

 scripts/dtc/README            | 106 +++++++++++
 scripts/dtc/pylibfdt/Makefile |   5 +-
 scripts/dtc/pylibfdt/setup.py |  60 ++++++-
 scripts/pylint.base           |   2 +-
 tools/dtoc/test_fdt.py        | 326 +++++++++++++++++++---------------
 5 files changed, 345 insertions(+), 154 deletions(-)
 create mode 100644 scripts/dtc/README
Signature · 1 line
2.37.1.455.g008518b4e5-goog
#1 lore raw

PATCH 1/7 dtoc: Tidy up fdt_tests RunTestCoverage() args

Pass the options args in rather than using the global various. Use snake case and fix up comments to make pylint happy.

---

 tools/dtoc/test_fdt.py | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
tools/dtoc/test_fdt.py +8 -4
index 3baf4437cdd..ec257552efc 100755
@@ -768,10 +768,14 @@ class TestFdtUtil(unittest.TestCase):
             tools.outdir= old_outdir
 
 
-def RunTestCoverage():
-    """Run the tests and check that we get 100% coverage"""
+def run_test_coverage(build_dir):
+    """Run the tests and check that we get 100% coverage
+
+    Args:
+        build_dir (str): Directory containing the build output
+    """
     test_util.run_test_coverage('tools/dtoc/test_fdt.py', None,
-            ['tools/patman/*.py', '*test_fdt.py'], options.build_dir)
+            ['tools/patman/*.py', '*test_fdt.py'], build_dir)
 
 
 def RunTests(args):
@@ -811,4 +815,4 @@ if options.test:
     ret_code = RunTests(args)
     sys.exit(ret_code)
 elif options.test_coverage:
-    RunTestCoverage()
+    run_test_coverage(options.build_dir)
Signature · 1 line
2.37.1.455.g008518b4e5-goog
#2 lore raw

Re: [PATCH 1/7] dtoc: Tidy up fdt_tests RunTestCoverage() args

Pass the options args in rather than using the global various. Use snake case and fix up comments to make pylint happy.

---

 tools/dtoc/test_fdt.py | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

Applied to u-boot-dm, thanks!

#3 lore raw

PATCH 2/7 dtoc: Tidy up fdt_tests RunTests()

In reply to Simon Glass

Pass the options args in rather than using the global variables. Use snake case, fix up comments and use a ternary operator to make pylint happy.

---

 tools/dtoc/test_fdt.py | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
tools/dtoc/test_fdt.py +9 -6
index ec257552efc..b48819831d6 100755
@@ -778,17 +778,20 @@ def run_test_coverage(build_dir):
             ['tools/patman/*.py', '*test_fdt.py'], build_dir)
 
 
-def RunTests(args):
+def run_tests(args, processes):
     """Run all the test we have for the fdt model
 
     Args:
-        args: List of positional args provided to fdt. This can hold a test
-            name to execute (as in 'fdt -t testFdt', for example)
+        args (list or str): List of positional args provided. This can hold a
+            test name to execute (as in 'test_fdt -t testFdt', for example)
+        processes (int): Number of processes to use (None means as many as there
+            are CPUs on the system. This must be set to 1 when running under
+            the python3-coverage tool
 
     Returns:
-        Return code, 0 on success
+        int: Return code, 0 on success
     """
-    test_name = args and args[0] or None
+    test_name = args[0] if args else None
     result = test_util.run_test_suites(
         'test_fdt', False, False, False, None, test_name, None,
         [TestFdt, TestNode, TestProp, TestFdtUtil])
@@ -812,7 +815,7 @@ parser.add_option('-T', '--test-coverage', action='store_true',
 
 # Run our meagre tests
 if options.test:
-    ret_code = RunTests(args)
+    ret_code = run_tests(args, options.processes)
     sys.exit(ret_code)
 elif options.test_coverage:
     run_test_coverage(options.build_dir)
Signature · 1 line
2.37.1.455.g008518b4e5-goog
#4 lore raw

Re: [PATCH 2/7] dtoc: Tidy up fdt_tests RunTests()

Pass the options args in rather than using the global variables. Use snake case, fix up comments and use a ternary operator to make pylint happy.

---

 tools/dtoc/test_fdt.py | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

Applied to u-boot-dm, thanks!

#5 lore raw

PATCH 3/7 dtoc: Fix fdt test coverage

In reply to Simon Glass

Fix a bug that the --processes option was ignored, thus resulting in no test coverage information being generated.

Fixes 42ae363ddd9 ("dtoc: Update fdt tests to use test_util")

---

 tools/dtoc/test_fdt.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/dtoc/test_fdt.py +1 -1
index b48819831d6..afa0bb58850 100755
@@ -793,7 +793,7 @@ def run_tests(args, processes):
     """
     test_name = args[0] if args else None
     result = test_util.run_test_suites(
-        'test_fdt', False, False, False, None, test_name, None,
+        'test_fdt', False, False, False, processes, test_name, None,
         [TestFdt, TestNode, TestProp, TestFdtUtil])
 
     return (0 if result.wasSuccessful() else 1)
Signature · 1 line
2.37.1.455.g008518b4e5-goog
#6 lore raw

Re: [PATCH 3/7] dtoc: Fix fdt test coverage

Fix a bug that the --processes option was ignored, thus resulting in no test coverage information being generated.

Fixes 42ae363ddd9 ("dtoc: Update fdt tests to use test_util")

---

 tools/dtoc/test_fdt.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Applied to u-boot-dm, thanks!

#7 lore raw

PATCH 4/7 dtoc: Move main program into its own function

In reply to Simon Glass

Use a function for the main program so everything there doesn't look like a global variable to pylint.

---

 tools/dtoc/test_fdt.py | 44 +++++++++++++++++++++++-------------------
 1 file changed, 24 insertions(+), 20 deletions(-)
tools/dtoc/test_fdt.py +24 -20
index afa0bb58850..e10fb528e81 100755
@@ -799,23 +799,27 @@ def run_tests(args, processes):
     return (0 if result.wasSuccessful() else 1)
 
 
-if __name__ != '__main__':
-    sys.exit(1)
-
-parser = OptionParser()
-parser.add_option('-B', '--build-dir', type='string', default='b',
-        help='Directory containing the build output')
-parser.add_option('-P', '--processes', type=int,
-                  help='set number of processes to use for running tests')
-parser.add_option('-t', '--test', action='store_true', dest='test',
-                  default=False, help='run tests')
-parser.add_option('-T', '--test-coverage', action='store_true',
-                default=False, help='run tests and check for 100% coverage')
-(options, args) = parser.parse_args()
-
-# Run our meagre tests
-if options.test:
-    ret_code = run_tests(args, options.processes)
-    sys.exit(ret_code)
-elif options.test_coverage:
-    run_test_coverage(options.build_dir)
+def main():
+    """Main program for this tool"""
+    parser = OptionParser()
+    parser.add_option('-B', '--build-dir', type='string', default='b',
+            help='Directory containing the build output')
+    parser.add_option('-P', '--processes', type=int,
+                      help='set number of processes to use for running tests')
+    parser.add_option('-t', '--test', action='store_true', dest='test',
+                      default=False, help='run tests')
+    parser.add_option('-T', '--test-coverage', action='store_true',
+                    default=False, help='run tests and check for 100% coverage')
+    (options, args) = parser.parse_args()
+
+    # Run our meagre tests
+    if options.test:
+        ret_code = run_tests(args, options.processes)
+        return ret_code
+    if options.test_coverage:
+        run_test_coverage(options.build_dir)
+    return 0
+
+if __name__ == '__main__':
+    sys.exit(main())
+sys.exit(1)
Signature · 1 line
2.37.1.455.g008518b4e5-goog
#8 lore raw

Re: [PATCH 4/7] dtoc: Move main program into its own function

Use a function for the main program so everything there doesn't look like a global variable to pylint.

---

 tools/dtoc/test_fdt.py | 44 +++++++++++++++++++++++-------------------
 1 file changed, 24 insertions(+), 20 deletions(-)

Applied to u-boot-dm, thanks!

#9 lore raw

PATCH 5/7 test_fdt: Convert to use argparse

In reply to Simon Glass

Drop the deprecated OptionParser.

---

 tools/dtoc/test_fdt.py | 39 ++++++++++++++++++++-------------------
 1 file changed, 20 insertions(+), 19 deletions(-)
tools/dtoc/test_fdt.py +20 -19
index e10fb528e81..c38158edf32 100755
@@ -4,7 +4,7 @@
 # Written by Simon Glass <[email protected]>
 #
 
-from optparse import OptionParser
+from argparse import ArgumentParser
 import glob
 import os
 import shutil
@@ -778,12 +778,11 @@ def run_test_coverage(build_dir):
             ['tools/patman/*.py', '*test_fdt.py'], build_dir)
 
 
-def run_tests(args, processes):
+def run_tests(names, processes):
     """Run all the test we have for the fdt model
 
     Args:
-        args (list or str): List of positional args provided. This can hold a
-            test name to execute (as in 'test_fdt -t testFdt', for example)
+        names (list of str): List of test names provided. Only the first is used
         processes (int): Number of processes to use (None means as many as there
             are CPUs on the system. This must be set to 1 when running under
             the python3-coverage tool
@@ -791,7 +790,7 @@ def run_tests(args, processes):
     Returns:
         int: Return code, 0 on success
     """
-    test_name = args[0] if args else None
+    test_name = names[0] if names else None
     result = test_util.run_test_suites(
         'test_fdt', False, False, False, processes, test_name, None,
         [TestFdt, TestNode, TestProp, TestFdtUtil])
@@ -801,23 +800,25 @@ def run_tests(args, processes):
 
 def main():
     """Main program for this tool"""
-    parser = OptionParser()
-    parser.add_option('-B', '--build-dir', type='string', default='b',
-            help='Directory containing the build output')
-    parser.add_option('-P', '--processes', type=int,
-                      help='set number of processes to use for running tests')
-    parser.add_option('-t', '--test', action='store_true', dest='test',
-                      default=False, help='run tests')
-    parser.add_option('-T', '--test-coverage', action='store_true',
-                    default=False, help='run tests and check for 100% coverage')
-    (options, args) = parser.parse_args()
+    parser = ArgumentParser()
+    parser.add_argument('-B', '--build-dir', type=str, default='b',
+                        help='Directory containing the build output')
+    parser.add_argument('-P', '--processes', type=int,
+                        help='set number of processes to use for running tests')
+    parser.add_argument('-t', '--test', action='store_true', dest='test',
+                        default=False, help='run tests')
+    parser.add_argument('-T', '--test-coverage', action='store_true',
+                        default=False,
+                        help='run tests and check for 100% coverage')
+    parser.add_argument('name', nargs='*')
+    args = parser.parse_args()
 
     # Run our meagre tests
-    if options.test:
-        ret_code = run_tests(args, options.processes)
+    if args.test:
+        ret_code = run_tests(args.name, args.processes)
         return ret_code
-    if options.test_coverage:
-        run_test_coverage(options.build_dir)
+    if args.test_coverage:
+        run_test_coverage(args.build_dir)
     return 0
 
 if __name__ == '__main__':
Signature · 1 line
2.37.1.455.g008518b4e5-goog
#10 lore raw

Re: [PATCH 5/7] test_fdt: Convert to use argparse

Drop the deprecated OptionParser.

---

 tools/dtoc/test_fdt.py | 39 ++++++++++++++++++++-------------------
 1 file changed, 20 insertions(+), 19 deletions(-)

Applied to u-boot-dm, thanks!

#11 lore raw

PATCH 6/7 dtoc: Correct remaining pylint problems in test_fdt

In reply to Simon Glass

Fix various camel-case and other naming problems. Update the pylint base file to avoid regressions.

---

 scripts/pylint.base    |   2 +-
 tools/dtoc/test_fdt.py | 254 +++++++++++++++++++++++------------------
 2 files changed, 142 insertions(+), 114 deletions(-)
scripts/pylint.base +1 -1
index 9ebebf47ab9..5203f2bb492 100644
@@ -167,7 +167,7 @@ tools_binman_etype_x86_reset16_tpl -15.71
 tools_binman_etype_x86_start16 -15.71
 tools_binman_etype_x86_start16_spl -15.71
 tools_binman_etype_x86_start16_tpl -15.71
-tools_binman_fdt_test 3.23
+tools_binman_fdt_test 10.00
 tools_binman_fip_util 9.85
 tools_binman_fip_util_test 10.00
 tools_binman_fmap_util 6.88
tools/dtoc/test_fdt.py +141 -113

594 lines not on this page · raw message

Signature · 1 line
2.37.1.455.g008518b4e5-goog
#12 lore raw

Re: [PATCH 6/7] dtoc: Correct remaining pylint problems in test_fdt

Fix various camel-case and other naming problems. Update the pylint base file to avoid regressions.

---

 scripts/pylint.base    |   2 +-
 tools/dtoc/test_fdt.py | 254 +++++++++++++++++++++++------------------
 2 files changed, 142 insertions(+), 114 deletions(-)

Applied to u-boot-dm, thanks!

#13 lore raw

PATCH 7/7 fdt: Move to setuptools

In reply to Simon Glass

The distutils package is deprecated. The upstream libfdt repo uses setuptools for building the pylibfdt module, so bring in that code, suitably modified for U-Boot. Also bring in the README.

The modifications include setting the version correctly, making use of the environment variables provided by the Makefile and various tweaks to the directories.

Note that the version omits the minus character at the start of EXTRAVERSION, since this creates a warning. The build is really just used within U-Boot itself, so it doesn't matter too much if the version matches upstream, or exactly matches U-Boot.

---

 scripts/dtc/README            | 106 ++++++++++++++++++++++++++++++++++
 scripts/dtc/pylibfdt/Makefile |   5 +-
 scripts/dtc/pylibfdt/setup.py |  60 ++++++++++++++++---
 3 files changed, 161 insertions(+), 10 deletions(-)
 create mode 100644 scripts/dtc/README
scripts/dtc/README new file +106 -0
new file mode 100644
index 00000000000..a48312a422c
@@ -0,0 +1,106 @@
+The source tree contains the Device Tree Compiler (dtc) toolchain for
+working with device tree source and binary files and also libfdt, a
+utility library for reading and manipulating the binary format.
+
+DTC and LIBFDT are maintained by:
+
+David Gibson <[email protected]>
+Jon Loeliger <[email protected]>
+
+
+Python library
+--------------
+
+A Python library is also available. To build this you will need to install
+swig and Python development files. On Debian distributions:
+
+   sudo apt-get install swig python3-dev
+
+The library provides an Fdt class which you can use like this:
+
+$ PYTHONPATH=../pylibfdt python3
+>>> import libfdt
+>>> fdt = libfdt.Fdt(open('test_tree1.dtb', mode='rb').read())
+>>> node = fdt.path_offset('/subnode@1')
+>>> print(node)
+124
+>>> prop_offset = fdt.first_property_offset(node)
+>>> prop = fdt.get_property_by_offset(prop_offset)
+>>> print('%s=%s' % (prop.name, prop.as_str()))
+compatible=subnode1
+>>> node2 = fdt.path_offset('/')
+>>> print(fdt.getprop(node2, 'compatible').as_str())
+test_tree1
+
+You will find tests in tests/pylibfdt_tests.py showing how to use each
+method. Help is available using the Python help command, e.g.:
+
+    $ cd pylibfdt
+    $ python3 -c "import libfdt; help(libfdt)"
+
+If you add new features, please check code coverage:
+
+    $ sudo apt-get install python3-coverage
+    $ cd tests
+    # It's just 'coverage' on most other distributions
+    $ python3-coverage run pylibfdt_tests.py
+    $ python3-coverage html
+    # Open 'htmlcov/index.html' in your browser
+
+
+The library can be installed with pip from a local source tree:
+
+    pip install . [--user|--prefix=/path/to/install_dir]
+
+Or directly from a remote git repo:
+
+    pip install git+git://git.kernel.org/pub/scm/utils/dtc/dtc.git@main
+
+The install depends on libfdt shared library being installed on the host system
+first. Generally, using --user or --prefix is not necessary and pip will use the
+default location for the Python installation which varies if the user is root or
+not.
+
+You can also install everything via make if you like, but pip is recommended.
+
+To install both libfdt and pylibfdt you can use:
+
+    make install [PREFIX=/path/to/install_dir]
+
+To disable building the python library, even if swig and Python are available,
+use:
+
+    make NO_PYTHON=1
+
+
+More work remains to support all of libfdt, including access to numeric
+values.
+
+
+Adding a new function to libfdt.h
+---------------------------------
+
+The shared library uses libfdt/version.lds to list the exported functions, so
+add your new function there. Check that your function works with pylibfdt. If
+it cannot be supported, put the declaration in libfdt.h behind #ifndef SWIG so
+that swig ignores it.
+
+
+Tests
+-----
+
+Test files are kept in the tests/ directory. Use 'make check' to build and run
+all tests.
+
+If you want to adjust a test file, be aware that tree_tree1.dts is compiled
+and checked against a binary tree from assembler macros in trees.S. So
+if you change that file you must change tree.S also.
+
+
+Mailing list
+------------
+The following list is for discussion about dtc and libfdt implementation
+mailto:[email protected]
+
+Core device tree bindings are discussed on the devicetree-spec list:
+mailto:[email protected]
scripts/dtc/pylibfdt/Makefile +4 -1
index 493995e3038..e442d5c2420 100644
@@ -13,11 +13,14 @@ include $(LIBFDT_srcdir)/Makefile.libfdt
 PYLIBFDT_srcs = $(addprefix $(LIBFDT_srcdir)/,$(LIBFDT_SRCS)) \
 		$(obj)/libfdt.i
 
+# create a version string compliant with PEP 440
+PEP_VERSION = $(VERSION)$(if $(PATCHLEVEL),.$(PATCHLEVEL)$(if $(SUBLEVEL),.$(SUBLEVEL)))$(subst -,,$(EXTRAVERSION))
+
 quiet_cmd_pymod = PYMOD   $@
       cmd_pymod = unset CROSS_COMPILE; unset CFLAGS; \
 		CC="$(HOSTCC)" LDSHARED="$(HOSTCC) -shared " \
 		LDFLAGS="$(HOSTLDFLAGS)" \
-		VERSION="u-boot-$(UBOOTVERSION)" \
+		VERSION="$(PEP_VERSION)" \
 		CPPFLAGS="$(HOSTCFLAGS) -I$(LIBFDT_srcdir)" OBJDIR=$(obj) \
 		SOURCES="$(PYLIBFDT_srcs)" \
 		SWIG_OPTS="-I$(LIBFDT_srcdir) -I$(LIBFDT_srcdir)/.." \
scripts/dtc/pylibfdt/setup.py +51 -9

105 lines not on this page · raw message

Signature · 1 line
2.37.1.455.g008518b4e5-goog
#14 lore raw

Re: [PATCH 7/7] fdt: Move to setuptools

Quoted 13 lines · Simon Glass

The distutils package is deprecated. The upstream libfdt repo uses setuptools for building the pylibfdt module, so bring in that code, suitably modified for U-Boot. Also bring in the README.

The modifications include setting the version correctly, making use of the environment variables provided by the Makefile and various tweaks to the directories.

Note that the version omits the minus character at the start of EXTRAVERSION, since this creates a warning. The build is really just used within U-Boot itself, so it doesn't matter too much if the version matches upstream, or exactly matches U-Boot.

Quoted 8 lines

---

 scripts/dtc/README            | 106 ++++++++++++++++++++++++++++++++++
 scripts/dtc/pylibfdt/Makefile |   5 +-
 scripts/dtc/pylibfdt/setup.py |  60 ++++++++++++++++---
 3 files changed, 161 insertions(+), 10 deletions(-)
 create mode 100644 scripts/dtc/README

This isn't new enough, however:
/usr/lib/python3/dist-packages/setuptools/installer.py:27:
SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
when trying in Ubuntu 22.04 (the trini/u-boot-gitlab-ci-runner:jammy-20220531-13Jul2022 docker specifically).

Signature · 1 line
Tom
#15 lore raw

Re: [PATCH 7/7] fdt: Move to setuptools

Hi Tom,

Quoted 25 lines · Tom Rini
Quoted 13 lines · Simon Glass

The distutils package is deprecated. The upstream libfdt repo uses setuptools for building the pylibfdt module, so bring in that code, suitably modified for U-Boot. Also bring in the README.

The modifications include setting the version correctly, making use of the environment variables provided by the Makefile and various tweaks to the directories.

Note that the version omits the minus character at the start of EXTRAVERSION, since this creates a warning. The build is really just used within U-Boot itself, so it doesn't matter too much if the version matches upstream, or exactly matches U-Boot.

Quoted 8 lines

---

 scripts/dtc/README            | 106 ++++++++++++++++++++++++++++++++++
 scripts/dtc/pylibfdt/Makefile |   5 +-
 scripts/dtc/pylibfdt/setup.py |  60 ++++++++++++++++---
 3 files changed, 161 insertions(+), 10 deletions(-)
 create mode 100644 scripts/dtc/README
Quoted 6 lines

This isn't new enough, however:
/usr/lib/python3/dist-packages/setuptools/installer.py:27:
SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
when trying in Ubuntu 22.04 (the trini/u-boot-gitlab-ci-runner:jammy-20220531-13Jul2022 docker specifically).

+Rob Herring

Yes I see that. But it is a start!

Also I was about to send a pull request with this...should I take it out?

Regards,
Simon

#16 lore raw

Re: [PATCH 7/7] fdt: Move to setuptools

Quoted 35 lines · Simon Glass

Hi Tom,

Quoted 25 lines · Tom Rini
Quoted 13 lines · Simon Glass

The distutils package is deprecated. The upstream libfdt repo uses setuptools for building the pylibfdt module, so bring in that code, suitably modified for U-Boot. Also bring in the README.

The modifications include setting the version correctly, making use of the environment variables provided by the Makefile and various tweaks to the directories.

Note that the version omits the minus character at the start of EXTRAVERSION, since this creates a warning. The build is really just used within U-Boot itself, so it doesn't matter too much if the version matches upstream, or exactly matches U-Boot.

Quoted 8 lines

---

 scripts/dtc/README            | 106 ++++++++++++++++++++++++++++++++++
 scripts/dtc/pylibfdt/Makefile |   5 +-
 scripts/dtc/pylibfdt/setup.py |  60 ++++++++++++++++---
 3 files changed, 161 insertions(+), 10 deletions(-)
 create mode 100644 scripts/dtc/README
Quoted 6 lines

This isn't new enough, however:
/usr/lib/python3/dist-packages/setuptools/installer.py:27:
SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
when trying in Ubuntu 22.04 (the trini/u-boot-gitlab-ci-runner:jammy-20220531-13Jul2022 docker specifically).

Quoted 5 lines

+Rob Herring

Yes I see that. But it is a start!

Also I was about to send a pull request with this...should I take it out?

Well, it brings us in sync with upstream I see, so yes, we should take this now and work on getting the next warning resolved. For why I hit this again, I'm just shuffling around where we set -e/+e.

Signature · 1 line
Tom