New CI image and fixes
cover New CI image and fixes
Hi all,
This series build a new CI image based on Ubuntu focal with LoongArch64 support, fixed various python scripts for python 3.12, fixed various problems popped up when testing againt latest software.
This change must be combined with test hook changes at [1].
Last two commits are for demonstration purpose and not for commit into repo.
CI runs passed at azure [2] and public gitlab.com runner [3].
Thanks
[1]: https://lore.kernel.org/u-boot/[email protected]/
[2]: https://gitlab.com/FlyGoat/u-boot/-/pipelines/1327832544
[3]: https://flygoat.visualstudio.com/u-boot/_build/results?buildId=63&view=results
---
Jiaxun Yang (20):
py: Replace deprecated unittest APIs
binman: Replace pkg_resources with importlib.resources
py: Replace distutils.core with setuptools
py: Replace usage of configparser.read_fp doc/sphinx: Remove usage of six
py: Remove unused entries in requirements.txt
py: Bump requirements versions
py: Bump pylint version and clear warnings
binman: Workaround lz4 cli padding in test cases tests/test_event_dump: Relax match rule for output lib/charset & efi: Fix possible unaligned accesses
cyclic: Rise default CYCLIC_MAX_CPU_TIME_US to 5000
CI: Ensure pip install is always performed in venv
CI: GitLab: Split build_world tasks
CI: Dockerfile: Set global git name & email config
CI: Dockerfile: Bump various software version
CI: Dockerfile: Add LoongArch64 support
doc: ci: Document how to run pipeline on gitlab.com [NFC] Use Jiaxun's CI Image
[NFC] CI: Dockerfile: Replace some URL with mirror sites
.azure-pipelines.yml | 22 +++-- .gitlab-ci.yml | 122 ++++++++++++++---------- common/Kconfig | 2 +- configs/octeon_nic23_defconfig | 1 - doc/develop/ci_testing.rst | 12 ++- doc/develop/python_cq.rst | 4 +- doc/sphinx/kfigure.py | 3 +- doc/sphinx/requirements.txt | 1 - lib/charset.c | 21 ++-- lib/efi_loader/efi_device_path.c | 11 +-- test/py/requirements.txt | 24 +---- test/py/tests/test_event_dump.py | 10 +- test/py/tests/test_ums.py | 1 + test/py/tests/test_usb.py | 1 + tools/binman/control.py | 18 ++-- tools/binman/entry_test.py | 6 +- tools/binman/etype/fdtmap.py | 1 + tools/binman/etype/fit.py | 1 + tools/binman/etype/image_header.py | 1 + tools/binman/etype/pre_load.py | 2 + tools/binman/etype/ti_board_config.py | 1 + tools/binman/etype/x509_cert.py | 1 + tools/binman/fdt_test.py | 48 +++++----- tools/binman/ftest.py | 50 +++++----- tools/binman/setup.py | 2 +- tools/binman/state.py | 1 + tools/binman/test/184_compress_section_size.dts | 1 + tools/buildman/bsettings.py | 2 +- tools/buildman/builder.py | 2 + tools/buildman/func_test.py | 74 +++++++------- tools/buildman/requirements.txt | 7 +- tools/buildman/test.py | 2 +- tools/docker/Dockerfile | 75 +++++++++------ tools/dtoc/setup.py | 2 +- tools/microcode-tool.py | 1 + tools/patman/settings.py | 9 +- tools/patman/test_checkpatch.py | 2 + tools/qconfig.py | 1 + 38 files changed, 293 insertions(+), 252 deletions(-)
---
base-commit: 1ebd659cf020843fd8e8ef90d85a66941cbab6ec
change-id: 20240610-docker-image-868126a1a929
Best regards,
Signature · 1 line
Jiaxun Yang <[email protected]>
PATCH 1/20 py: Replace deprecated unittest APIs
assertEquals -> assertEqual
assertRegexpMatches -> assertRegex
Those APIs were deprecated long ago and being removed in latest python.
--- tools/binman/entry_test.py | 6 ++-- tools/binman/fdt_test.py | 48 ++++++++++++++--------------- tools/binman/ftest.py | 42 ++++++++++++------------- tools/buildman/func_test.py | 74 ++++++++++++++++++++++----------------------- tools/buildman/test.py | 2 +- 5 files changed, 86 insertions(+), 86 deletions(-)
tools/binman/entry_test.py +3 -3
@@ -103,7 +103,7 @@ class TestEntry(unittest.TestCase): ent = entry.Entry.Create(None, self.GetNode(), 'missing', missing_etype=True) self.assertTrue(isinstance(ent, Entry_blob)) - self.assertEquals('missing', ent.etype) + self.assertEqual('missing', ent.etype) def testDecompressData(self): """Test the DecompressData() method of the base class"""
@@ -111,8 +111,8 @@ class TestEntry(unittest.TestCase): base.compress = 'lz4' bintools = {} base.comp_bintool = base.AddBintool(bintools, '_testing') - self.assertEquals(tools.get_bytes(0, 1024), base.CompressData(b'abc')) - self.assertEquals(tools.get_bytes(0, 1024), base.DecompressData(b'abc')) + self.assertEqual(tools.get_bytes(0, 1024), base.CompressData(b'abc')) + self.assertEqual(tools.get_bytes(0, 1024), base.DecompressData(b'abc')) def testLookupOffset(self): """Test the lookup_offset() method of the base class"""
tools/binman/fdt_test.py +24 -24
67 lines not on this page · raw message
tools/binman/ftest.py +21 -21
136 lines not on this page · raw message
tools/buildman/func_test.py +37 -37
159 lines not on this page · raw message
tools/buildman/test.py +1 -1
@@ -584,7 +584,7 @@ class TestBuild(unittest.TestCase): if use_network: with test_util.capture_sys_output() as (stdout, stderr): url = self.toolchains.LocateArchUrl('arm') - self.assertRegexpMatches(url, 'https://www.kernel.org/pub/tools/' + self.assertRegex(url, 'https://www.kernel.org/pub/tools/' 'crosstool/files/bin/x86_64/.*/' 'x86_64-gcc-.*-nolibc[-_]arm-.*linux-gnueabi.tar.xz')
Signature · 1 line
2.43.0
PATCH 2/20 binman: Replace pkg_resources with importlib.resources
In reply to Jiaxun Yangpkg_resources is deprecated long ago and being removed in python 3.12.
Reimplement functions with importlib.resources.
--- tools/binman/control.py | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
tools/binman/control.py +9 -9
@@ -8,12 +8,11 @@ from collections import OrderedDict import glob try: - import importlib.resources -except ImportError: # pragma: no cover + from importlib import resources +except ImportError: # for Python 3.6 - import importlib_resources + import importlib_resources as resources import os -import pkg_resources import re import sys
@@ -96,12 +95,12 @@ def _ReadMissingBlobHelp(): msg = '' return tag, msg - my_data = pkg_resources.resource_string(__name__, 'missing-blob-help') + my_data = resources.files(__package__).joinpath('missing-blob-help').read_text() re_tag = re.compile('^([-a-z0-9]+):$') result = {} tag = None msg = '' - for line in my_data.decode('utf-8').splitlines(): + for line in my_data.splitlines(): if not line.startswith('#'): m_tag = re_tag.match(line) if m_tag:
@@ -151,8 +150,9 @@ def GetEntryModules(include_testing=True): Returns: Set of paths to entry class filenames """ - glob_list = pkg_resources.resource_listdir(__name__, 'etype') - glob_list = [fname for fname in glob_list if fname.endswith('.py')] + directory = resources.files("binman.etype") + glob_list = [entry.name for entry in directory.iterdir() + if entry.name.endswith('.py')] return set([os.path.splitext(os.path.basename(item))[0] for item in glob_list if include_testing or '_testing' not in item])
@@ -735,7 +735,7 @@ def Binman(args): global state if args.full_help: - with importlib.resources.path('binman', 'README.rst') as readme: + with resources.path('binman', 'README.rst') as readme: tools.print_full_help(str(readme)) return 0
Signature · 1 line
2.43.0
PATCH 3/20 py: Replace distutils.core with setuptools
In reply to Jiaxun Yangdistutils is deprecated long ago and being removed in python 3.12.
--- tools/binman/setup.py | 2 +- tools/buildman/requirements.txt | 1 + tools/dtoc/setup.py | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-)
tools/binman/setup.py +1 -1
@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0+ -from distutils.core import setup +from setuptools import setup setup(name='binman', version='1.0', license='GPL-2.0+',
tools/buildman/requirements.txt +1 -0
@@ -1,3 +1,4 @@ jsonschema==4.17.3 pyyaml==6.0 yamllint==1.26.3 +setuptools==65.5.1
tools/dtoc/setup.py +1 -1
@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0+ -from distutils.core import setup +from setuptools import setup setup(name='dtoc', version='1.0', license='GPL-2.0+',
Signature · 1 line
2.43.0
PATCH 4/20 py: Replace usage of configparser.read_fp
In reply to Jiaxun Yangconfigparser.read_fp is deprecated long ago.
Replace with relevant API.
--- tools/buildman/bsettings.py | 2 +- tools/patman/settings.py | 9 ++++----- 2 files changed, 5 insertions(+), 6 deletions(-)
tools/buildman/bsettings.py +1 -1
@@ -29,7 +29,7 @@ def setup(fname=''): settings.read(config_fname) def add_file(data): - settings.readfp(io.StringIO(data)) + settings.read_file(io.StringIO(data), data) def get_items(section): """Get the items from a section of the config.
tools/patman/settings.py +4 -5
@@ -41,7 +41,6 @@ class _ProjectConfigParser(ConfigParser.ConfigParser): - Merge general default settings/aliases with project-specific ones. # Sample config used for tests below... - >>> from io import StringIO >>> sample_config = ''' ... [alias] ... me: Peter P. <[email protected]>
@@ -59,25 +58,25 @@ class _ProjectConfigParser(ConfigParser.ConfigParser): # Check to make sure that bogus project gets general alias. >>> config = _ProjectConfigParser("zzz") - >>> config.readfp(StringIO(sample_config)) + >>> config.read_string(sample_config) >>> str(config.get("alias", "enemies")) 'Evil <[email protected]>' # Check to make sure that alias gets overridden by project. >>> config = _ProjectConfigParser("sm") - >>> config.readfp(StringIO(sample_config)) + >>> config.read_string(sample_config) >>> str(config.get("alias", "enemies")) 'Green G. <[email protected]>' # Check to make sure that settings get merged with project. >>> config = _ProjectConfigParser("linux") - >>> config.readfp(StringIO(sample_config)) + >>> config.read_string(sample_config) >>> sorted((str(a), str(b)) for (a, b) in config.items("settings")) [('am_hero', 'True'), ('check_patch_use_tree', 'True'), ('process_tags', 'False')] # Check to make sure that settings works with unknown project. >>> config = _ProjectConfigParser("unknown") - >>> config.readfp(StringIO(sample_config)) + >>> config.read_string(sample_config) >>> sorted((str(a), str(b)) for (a, b) in config.items("settings")) [('am_hero', 'True')] """
Signature · 1 line
2.43.0
PATCH 5/20 doc/sphinx: Remove usage of six
In reply to Jiaxun YangWe don't support python2 any more so there is no point to use six here.
--- doc/sphinx/kfigure.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
doc/sphinx/kfigure.py +1 -2
@@ -58,7 +58,6 @@ from docutils.parsers.rst.directives import images import sphinx from sphinx.util.nodes import clean_astext -from six import iteritems import kernellog
@@ -540,7 +539,7 @@ def add_kernel_figure_to_std_domain(app, doctree): docname = app.env.docname labels = std.data["labels"] - for name, explicit in iteritems(doctree.nametypes): + for name, explicit in doctree.nametypes.items(): if not explicit: continue labelid = doctree.nameids[name]
Signature · 1 line
2.43.0
PATCH 6/20 py: Remove unused entries in requirements.txt
In reply to Jiaxun YangMany packages in requirements.txt are not actually imported/used by any test case, some of them was for python2 compatibility.
Remove them to keep our environment clean.
--- doc/sphinx/requirements.txt | 1 - test/py/requirements.txt | 18 ------------------ 2 files changed, 19 deletions(-)
doc/sphinx/requirements.txt +0 -1
@@ -10,7 +10,6 @@ MarkupSafe==2.1.3 packaging==23.2 Pygments==2.17.2 requests==2.31.0 -six==1.16.0 snowballstemmer==2.2.0 Sphinx==7.2.6 sphinx-prompt==1.8.0
test/py/requirements.txt +0 -18
@@ -1,30 +1,12 @@ -atomicwrites==1.4.1 -attrs==19.3.0 concurrencytest==0.1.2 coverage==4.5.4 -extras==1.0.0 filelock==3.0.12 -fixtures==3.0.0 -importlib-metadata==0.23 -linecache2==1.0.0 -more-itertools==7.2.0 packaging==23.2 -pbr==5.4.3 -pluggy==0.13.0 -py==1.11.0 pycryptodomex==3.19.1 pyelftools==0.27 pygit2==1.13.3 pyparsing==3.0.7 pytest==6.2.5 pytest-xdist==2.5.0 -python-mimeparse==1.6.0 -python-subunit==1.3.0 requests==2.31.0 setuptools==65.5.1 -six==1.16.0 -testtools==2.3.0 -traceback2==1.4.0 -unittest2==1.1.0 -wcwidth==0.1.7 -zipp==0.6.0
Signature · 1 line
2.43.0
PATCH 7/20 py: Bump requirements versions
In reply to Jiaxun YangBump python packages to reasonable versions to fix build error with Python 3.12.
--- test/py/requirements.txt | 6 +++--- tools/buildman/requirements.txt | 8 ++++---- 2 files changed, 7 insertions(+), 7 deletions(-)
test/py/requirements.txt +3 -3
@@ -6,7 +6,7 @@ pycryptodomex==3.19.1 pyelftools==0.27 pygit2==1.13.3 pyparsing==3.0.7 -pytest==6.2.5 -pytest-xdist==2.5.0 +pytest==7.4.4 +pytest-xdist==3.6.1 requests==2.31.0 -setuptools==65.5.1 +setuptools==70
tools/buildman/requirements.txt +4 -4
@@ -1,4 +1,4 @@ -jsonschema==4.17.3 -pyyaml==6.0 -yamllint==1.26.3 -setuptools==65.5.1 +jsonschema==4.22.0 +pyyaml==6.0.1 +yamllint==1.35.1 +setuptools==70
Signature · 1 line
2.43.0
PATCH 8/20 py: Bump pylint version and clear warnings
In reply to Jiaxun YangBump pylint to 3.2.3 as old versions are not working with python 3.12.
Clear warnings, mostly E0606: (possibly-used-before-assignment).
--- .azure-pipelines.yml | 2 +- .gitlab-ci.yml | 2 +- doc/develop/python_cq.rst | 4 ++-- test/py/tests/test_ums.py | 1 + test/py/tests/test_usb.py | 1 + tools/binman/etype/fdtmap.py | 1 + tools/binman/etype/fit.py | 1 + tools/binman/etype/image_header.py | 1 + tools/binman/etype/pre_load.py | 2 ++ tools/binman/etype/ti_board_config.py | 1 + tools/binman/etype/x509_cert.py | 1 + tools/binman/ftest.py | 1 + tools/binman/state.py | 1 + tools/buildman/builder.py | 2 ++ tools/microcode-tool.py | 1 + tools/patman/test_checkpatch.py | 2 ++ tools/qconfig.py | 1 + 17 files changed, 21 insertions(+), 4 deletions(-)
.azure-pipelines.yml +1 -1
@@ -152,7 +152,7 @@ stages: export USER=azure pip install -r test/py/requirements.txt pip install -r tools/buildman/requirements.txt - pip install asteval pylint==2.12.2 pyopenssl + pip install asteval pylint==3.2.3 pyopenssl export PATH=${PATH}:~/.local/bin echo "[MASTER]" >> .pylintrc echo "load-plugins=pylint.extensions.docparams" >> .pylintrc
.gitlab-ci.yml +1 -1
@@ -214,7 +214,7 @@ Run pylint: - git config --global --add safe.directory "${CI_PROJECT_DIR}" - pip install -r test/py/requirements.txt - pip install -r tools/buildman/requirements.txt - - pip install asteval pylint==2.12.2 pyopenssl + - pip install asteval pylint==3.2.3 pyopenssl - export PATH=${PATH}:~/.local/bin - echo "[MASTER]" >> .pylintrc - echo "load-plugins=pylint.extensions.docparams" >> .pylintrc
doc/develop/python_cq.rst +2 -2
@@ -23,7 +23,7 @@ regressions in any module. To run this locally you should use this version of pylint:: # pylint --version - pylint 2.11.1 + pylint 3.2.3 astroid 2.8.6 Python 3.8.10 (default, Sep 28 2021, 16:10:42) [GCC 9.3.0]
@@ -31,7 +31,7 @@ To run this locally you should use this version of pylint:: You should be able to select and this install other required tools with:: - pip install pylint==2.11.1 + pip install pylint==3.2.3 pip install -r test/py/requirements.txt pip install asteval pyopenssl
test/py/tests/test_ums.py +1 -0
@@ -118,6 +118,7 @@ def test_ums(u_boot_console, env__usb_dev_port, env__block_devs): test_f = u_boot_utils.PersistentRandomFile(u_boot_console, 'ums.bin', 1024 * 1024); + mounted_test_fn = None if have_writable_fs_partition: mounted_test_fn = mount_point + '/' + mount_subdir + test_f.fn
test/py/tests/test_usb.py +1 -0
@@ -564,6 +564,7 @@ def test_usb_load(u_boot_console): part_detect = 1 addr = u_boot_utils.find_ram_base(u_boot_console) + file, size = 0, 0 if fs == 'fat': file, size = test_usb_fatload_fatwrite(u_boot_console) elif fs == 'ext4':
tools/binman/etype/fdtmap.py +1 -0
@@ -106,6 +106,7 @@ class Entry_fdtmap(Entry): Returns: FDT map binary data """ + fsw = None def _AddNode(node): """Add a node to the FDT map""" for pname, prop in node.props.items():
tools/binman/etype/fit.py +1 -0
@@ -808,6 +808,7 @@ class Entry_fit(Entry_section): data_size = fdt_util.GetInt(node, "data-size") # Contents are inside the FIT + offset, size = 0, 0 if data_prop is not None: # GetOffset() returns offset of a fdt_property struct, # which has 3 fdt32_t members before the actual data.
tools/binman/etype/image_header.py +1 -0
@@ -62,6 +62,7 @@ class Entry_image_header(Entry): def _GetHeader(self): image_pos = self.GetSiblingImagePos('fdtmap') + offset = 0xffffffff if image_pos == False: self.Raise("'image_header' section must have an 'fdtmap' sibling") elif image_pos is None:
tools/binman/etype/pre_load.py +2 -0
@@ -112,6 +112,8 @@ class Entry_pre_load(Entry_collection): # Compute the signature if padding_name is None: padding_name = "pkcs-1.5" + + padding, padding_args = None, {} if padding_name == "pss": salt_len = key.size_in_bytes() - hash_image.digest_size - 2 padding = pss
tools/binman/etype/ti_board_config.py +1 -0
@@ -118,6 +118,7 @@ class Entry_ti_board_config(Entry_section): Returns: array of bytes representing value """ + br = None size = 0 if (data_type == '#/definitions/u8'): size = 1
tools/binman/etype/x509_cert.py +1 -0
@@ -83,6 +83,7 @@ class Entry_x509_cert(Entry_collection): output_fname = tools.get_output_filename('cert.%s' % uniq) input_fname = tools.get_output_filename('input.%s' % uniq) config_fname = tools.get_output_filename('config.%s' % uniq) + stdout = None tools.write_file(input_fname, input_data) if type == 'generic': stdout = self.openssl.x509_cert(
tools/binman/ftest.py +1 -0
@@ -6293,6 +6293,7 @@ fdt fdtmap Extract the devicetree blob from the fdtmap re_name = re.compile('_binman_(u_boot_(.*))_prop_(.*)') for name, sym in syms.items(): msg = 'test' + expect_val = None val = elf.GetSymbolValue(sym, edata, msg) entry_m = re_name.match(name) if entry_m:
tools/binman/state.py +1 -0
@@ -406,6 +406,7 @@ def CheckSetHashValue(node, get_data_func): hash_node = node.FindNode('hash') if hash_node: algo = hash_node.props.get('algo').value + data = None if algo == 'sha256': m = hashlib.sha256() m.update(get_data_func())
tools/buildman/builder.py +2 -0
@@ -1066,6 +1066,7 @@ class Builder: printed_target = False for name in sorted(result): diff = result[name] + color = None if name.startswith('_'): continue if diff != 0:
@@ -1326,6 +1327,7 @@ class Builder: for line in lines: if not line: continue + col = None if line[0] == '+': col = self.col.GREEN elif line[0] == '-':
tools/microcode-tool.py +1 -0
@@ -277,6 +277,7 @@ def MicrocodeTool(): if cmd not in commands: parser.error("Unknown command '%s'" % cmd) + date, license_text, microcodes = None, None, None if (not not options.mcfile) != (not not options.mcfile): parser.error("You must specify either header files or a microcode file, not both") if options.headerfile:
tools/patman/test_checkpatch.py +2 -0
@@ -530,4 +530,6 @@ index 0000000..2234c87 if __name__ == "__main__": unittest.main() + # pylint doesn't seem to find this + # pylint: disable=E1101 gitutil.RunTests()
tools/qconfig.py +1 -0
@@ -873,6 +873,7 @@ def read_database(): all_defconfigs = set() defconfig_db = collections.defaultdict(set) + defconfig = None for line in read_file(CONFIG_DATABASE): line = line.rstrip() if not line: # Separator between defconfigs
Signature · 1 line
2.43.0
PATCH 9/20 binman: Workaround lz4 cli padding in test cases
In reply to Jiaxun YangNewer lz4 util is not happy with any padding at end of file, it would abort with error message like:
Stream followed by undecodable data at position 43.
Workaround by skipping testCompUtilPadding test case and manually strip padding in testCompressSectionSize test case.
--- tools/binman/ftest.py | 7 +++++-- tools/binman/test/184_compress_section_size.dts | 1 + 2 files changed, 6 insertions(+), 2 deletions(-)
tools/binman/ftest.py +5 -2
@@ -4518,6 +4518,8 @@ class TestFunctional(unittest.TestCase): dtb.Scan() props = self._GetPropTree(dtb, ['offset', 'image-pos', 'size', 'uncomp-size']) + data = data[:0x30] + data = data.rstrip(b'\xff') orig = self._decompress(data) self.assertEqual(COMPRESS_DATA + U_BOOT_DATA, orig) expected = {
@@ -6118,8 +6120,9 @@ fdt fdtmap Extract the devicetree blob from the fdtmap def testCompUtilPadding(self): """Test padding of compression algorithms""" - # Skip zstd because it doesn't support padding - for bintool in [v for k,v in self.comp_bintools.items() if k != 'zstd']: + # Skip zstd and lz4 because they doesn't support padding + for bintool in [v for k,v in self.comp_bintools.items() + if not k in ['zstd', 'lz4']]: self._CheckBintool(bintool) data = bintool.compress(COMPRESS_DATA) self.assertNotEqual(COMPRESS_DATA, data)
tools/binman/test/184_compress_section_size.dts +1 -0
@@ -6,6 +6,7 @@ section { size = <0x30>; compress = "lz4"; + pad-byte = <0xff>; blob { filename = "compress"; };
Signature · 1 line
2.43.0
PATCH 10/20 tests/test_event_dump: Relax match rule for output
In reply to Jiaxun Yangevent_dump.py relies on addr2line to obtain source location information, however newer addr2line is unable to determine line numbers for some functions.
With addr2line from binutils 2.34 we got:
Event type Id Source location -------------------- ------------------------------ ------------------------------ EVT_FT_FIXUP bootmeth_vbe_ft_fixup :? EVT_FT_FIXUP bootmeth_vbe_simple_ft_fixup :? EVT_LAST_STAGE_INIT install_smbios_table :? EVT_MISC_INIT_F sandbox_early_getopt_check arch/sandbox/cpu/start.c:61 EVT_TEST h_adder_simple :?
Which will fail the test.
Relax the source location regex to .*:.*, this is sufficent to show that addr2line is being called and returned a possible line number.
--- test/py/tests/test_event_dump.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
test/py/tests/test_event_dump.py +5 -5
@@ -16,9 +16,9 @@ def test_event_dump(u_boot_console): out = util.run_and_log(cons, ['scripts/event_dump.py', sandbox]) expect = '''.*Event type Id Source location -------------------- ------------------------------ ------------------------------ -EVT_FT_FIXUP bootmeth_vbe_ft_fixup .*boot/vbe_request.c:.* -EVT_FT_FIXUP bootmeth_vbe_simple_ft_fixup .*boot/vbe_simple_os.c:.* -EVT_LAST_STAGE_INIT install_smbios_table .*lib/efi_loader/efi_smbios.c:.* -EVT_MISC_INIT_F sandbox_early_getopt_check .*arch/sandbox/cpu/start.c:.* -EVT_TEST h_adder_simple .*test/common/event.c:''' +EVT_FT_FIXUP bootmeth_vbe_ft_fixup .*:.* +EVT_FT_FIXUP bootmeth_vbe_simple_ft_fixup .*:.* +EVT_LAST_STAGE_INIT install_smbios_table .*:.* +EVT_MISC_INIT_F sandbox_early_getopt_check .*:.* +EVT_TEST h_adder_simple .*:''' assert re.match(expect, out, re.MULTILINE) is not None
Signature · 1 line
2.43.0
PATCH 11/20 lib/charset & efi: Fix possible unaligned accesses
In reply to Jiaxun YangAs per armv7 arch spec, for A-profile CPU if translation is disabled, then the default memory type is Device(-nGnRnE) instead of Normal, which requires that alignment be enforced.
This means in some cases we can't perform unaligned access even after allow_unaligned is called.
We do have many platforms that didn't have translation enabled in U-Boot, and QEMU started to enforce this since 9.0.
Fix by using unaligned access helper for UTF-16 memory read/write to ensure we don't do any unaligned access in U-Boot.
--- lib/charset.c | 21 ++++++++++++--------- lib/efi_loader/efi_device_path.c | 11 ++--------- 2 files changed, 14 insertions(+), 18 deletions(-)
lib/charset.c +12 -9
@@ -11,6 +11,7 @@#include <efi_loader.h> #include <errno.h> #include <malloc.h> +#include <asm/unaligned.h> /** * codepage_437 - Unicode to codepage 437 translation table
@@ -215,7 +216,7 @@ s32 utf16_get(const u16 **src)return -1 ;if (!**src)return 0 ; - code = **src; + code =get_unaligned_le16 (*src); ++*src;if (code >=0xDC00 && code <=0xDFFF )return -1 ;
@@ -242,12 +243,12 @@ int utf16_put(s32 code, u16 **dst)if ((code >=0xD800 && code <=0xDFFF ) || code >=0x110000 )return -1 ;if (code <0x10000 ) { - **dst = code; +put_unaligned_le16 (code, *dst); }else { code -=0x10000 ; - **dst = code >>10 |0xD800 ; +put_unaligned_le16 (code >>10 |0xD800 , *dst); ++*dst; - **dst = (code &0x3ff ) |0xDC00 ; +put_unaligned_le16 ((code &0x3ff ) |0xDC00 , *dst); } ++*dst;return 0 ;
@@ -392,7 +393,7 @@ int __efi_runtime u16_strncmp(const u16 *s1, const u16 *s2, size_t n)int ret =0 ;for (; n; --n, ++s1, ++s2) { - ret = *s1 - *s2; + ret =get_unaligned_le16 (s1) -get_unaligned_le16 (s2);if (ret || !*s1)break ; }
@@ -403,7 +404,7 @@ int __efi_runtime u16_strncmp(const u16 *s1, const u16 *s2, size_t n) size_t __efi_runtime u16_strnlen(const u16 *in, size_t count) {size_t i; -for (i =0 ; count-- && in[i]; i++); +for (i =0 ; count-- &&get_unaligned_le16 (in + i); i++);return i; }
@@ -417,8 +418,10 @@ u16 *u16_strcpy(u16 *dest, const u16 *src)u16 *tmp = dest;for (;; dest++, src++) { - *dest = *src; -if (!*src) +u16 code =get_unaligned_le16 (src); + +put_unaligned_le16 (code, dest); +if (!code)break ; }
@@ -463,7 +466,7 @@ uint8_t *utf16_to_utf8(uint8_t *dest, const uint16_t *src, size_t size)uint32_t code_high =0 ;while (size--) { -uint32_t code = *src++; +uint32_t code =get_unaligned_le16 (src++);if (code_high) {if (code >=0xDC00 && code <=0xDFFF ) {
lib/efi_loader/efi_device_path.c +2 -9
@@ -18,7 +18,7 @@#include <efi_loader.h> #include <part.h> #include <uuid.h> -#include <asm-generic/unaligned.h> +#include <asm/unaligned.h> #include <linux/compat.h> /* U16_MAX */ /* template END node: */
@@ -867,13 +867,6 @@ static void path_to_uefi(void *uefi, const char *src) {u16 *pos = uefi; -/* -* efi_set_bootdev() calls this routine indirectly before the UEFI -* subsystem is initialized. So we cannot assume unaligned access to be -* enabled. -*/ -allow_unaligned (); -while (*src) {s32 code =utf8_get (&src);
@@ -883,7 +876,7 @@ static void path_to_uefi(void *uefi, const char *src) code ='\\' ;utf16_put (code, &pos); } - *pos =0 ; +put_unaligned_le16 (0 , pos); }/**
Signature · 1 line
2.43.0
PATCH 12/20 cyclic: Rise default CYCLIC_MAX_CPU_TIME_US to 5000
In reply to Jiaxun YangThe default value CYCLIC_MAX_CPU_TIME_US was 1000, which is a little bit too low for slower hardware and sandbox.
On my MIPS Boston FPGA board with interaptiv CPU, wdt_cyclic can easily take 3200 us to run.
On azure pipeline sandbox_clang, wdt_cyclic some times goes beyond 1300 us.
Raise default value to 5000, which is the value already taken by octeon_nic32. This is still sufficent to maintain system responsiveness.
--- common/Kconfig | 2 +- configs/octeon_nic23_defconfig | 1 - 2 files changed, 1 insertion(+), 2 deletions(-)
common/Kconfig +1 -1
@@ -628,7 +628,7 @@ if CYCLIC config CYCLIC_MAX_CPU_TIME_US int "Sets the max allowed time for a cyclic function in us" - default 1000 + default 5000 help The max allowed time for a cyclic function in us. If a functions takes longer than this duration this function will get unregistered
configs/octeon_nic23_defconfig +0 -1
@@ -25,7 +25,6 @@ CONFIG_SYS_PBSIZE=276 CONFIG_SYS_CONSOLE_ENV_OVERWRITE=y # CONFIG_SYS_DEVICE_NULLDEV is not set CONFIG_CYCLIC=y -CONFIG_CYCLIC_MAX_CPU_TIME_US=5000 CONFIG_ARCH_MISC_INIT=y CONFIG_BOARD_EARLY_INIT_F=y CONFIG_BOARD_LATE_INIT=y
Signature · 1 line
2.43.0
Re: [PATCH 12/20] cyclic: Rise default CYCLIC_MAX_CPU_TIME_US to 5000
Quoted 13 lines · Jiaxun Yang
The default value
CYCLIC_MAX_CPU_TIME_USwas 1000, which is a little bit too low for slower hardware and sandbox.On my MIPS Boston FPGA board with interaptiv CPU,
wdt_cycliccan easily take 3200 us to run.On azure pipeline
sandbox_clang,wdt_cyclicsome times goes beyond 1300 us.Raise default value to 5000, which is the value already taken by
octeon_nic32. This is still sufficent to maintain system responsiveness.
Quoted 5 lines
Signed-off-by Jiaxun Yang <[email protected]>--- common/Kconfig | 2 +- configs/octeon_nic23_defconfig | 1 - 2 files changed, 1 insertion(+), 2 deletions(-)
This seems similar to:
https://patchwork.ozlabs.org/project/uboot/patch/[email protected]/
at least for CI. And for outside CI, I'm OK with just having the value be changed in the defconfig as needed. We do support using config fragments, so keeping such changes locally isn't too hard.
Signature · 1 line
Tom
Re: [PATCH 12/20] cyclic: Rise default CYCLIC_MAX_CPU_TIME_US to 5000
在2024年6月12日六月 下午5:00,Tom Rini写道:
[...]
Quoted 9 lines
Quoted 2 lines
configs/octeon_nic23_defconfig | 1 - 2 files changed, 1 insertion(+), 2 deletions(-)This seems similar to:
https://patchwork.ozlabs.org/project/uboot/patch/[email protected]/at least for CI. And for outside CI, I'm OK with just having the value be changed in the defconfig as needed. We do support using config fragments, so keeping such changes locally isn't too hard.
So the default value is a little bit too hard even for some of the actual hardware.
I think being permissive here can prevent people hit into the problem only after enabling cyclic on their own board.
Thanks
Quoted 6 lines
Signature · 5 lines
-- Tom 附件: * signature.asc
Signature · 1 line
- Jiaxun
Re: [PATCH 12/20] cyclic: Rise default CYCLIC_MAX_CPU_TIME_US to 5000
Quoted 10 lines · Jiaxun Yang
在2024年6月12日六月 下午5:00,Tom Rini写道:
[...]Quoted 6 lines
Quoted 2 lines
configs/octeon_nic23_defconfig | 1 - 2 files changed, 1 insertion(+), 2 deletions(-)This seems similar to:
https://patchwork.ozlabs.org/project/uboot/patch/[email protected]/
Quoted 6 lines
Quoted 3 lines
at least for CI. And for outside CI, I'm OK with just having the value be changed in the defconfig as needed. We do support using config fragments, so keeping such changes locally isn't too hard.
So the default value is a little bit too hard even for some of the actual hardware.
Right, there's some platforms where it's too small and we should just bump it up. I think for now the default is what we want it to be for most platforms.
Signature · 1 line
Tom
Stefan Roese
Acked-byRe: [PATCH 12/20] cyclic: Rise default CYCLIC_MAX_CPU_TIME_US to 5000
Quoted 15 lines · Tom Rini
Quoted 14 lines · Jiaxun Yang
在2024年6月12日六月 下午5:00,Tom Rini写道:
[...]Quoted 9 lines
Quoted 2 lines
configs/octeon_nic23_defconfig | 1 - 2 files changed, 1 insertion(+), 2 deletions(-)This seems similar to:
https://patchwork.ozlabs.org/project/uboot/patch/[email protected]/at least for CI. And for outside CI, I'm OK with just having the value be changed in the defconfig as needed. We do support using config fragments, so keeping such changes locally isn't too hard.
Quoted 6 lines
Quoted 2 lines
So the default value is a little bit too hard even for some of the actual hardware.
Right, there's some platforms where it's too small and we should just bump it up. I think for now the default is what we want it to be for most platforms.
The current default value is definitely too small, especially when CI is involved (I did not have this in mind when implementing), so:
Thanks,
Stefan
Re: [PATCH 12/20] cyclic: Rise default CYCLIC_MAX_CPU_TIME_US to 5000
Quoted 19 lines · Stefan Roese
Quoted 18 lines · Tom Rini
Quoted 16 lines · Jiaxun Yang
在2024年6月12日六月 下午5:00,Tom Rini写道:
[...]Quoted 9 lines
Quoted 2 lines
configs/octeon_nic23_defconfig | 1 - 2 files changed, 1 insertion(+), 2 deletions(-)This seems similar to:
https://patchwork.ozlabs.org/project/uboot/patch/[email protected]/at least for CI. And for outside CI, I'm OK with just having the value be changed in the defconfig as needed. We do support using config fragments, so keeping such changes locally isn't too hard.
So the default value is a little bit too hard even for some of the actual hardware.
Quoted 8 lines
Quoted 3 lines
Right, there's some platforms where it's too small and we should just bump it up. I think for now the default is what we want it to be for most platforms.
The current default value is definitely too small, especially when CI is involved (I did not have this in mind when implementing), so:
Acked-by Stefan Roese <[email protected]>
Can we please get either this, or
https://patchwork.ozlabs.org/project/uboot/patch/[email protected]/
merged for master? The number of false negatives in CI due to this is big issue for getting more contributors to use CI. Thanks.
Signature · 1 line
Tom
Re: [PATCH 12/20] cyclic: Rise default CYCLIC_MAX_CPU_TIME_US to 5000
Quoted 27 lines · Tom Rini
Quoted 26 lines · Stefan Roese
Quoted 15 lines · Tom Rini
Quoted 14 lines · Jiaxun Yang
在2024年6月12日六月 下午5:00,Tom Rini写道:
[...]Quoted 9 lines
Quoted 2 lines
configs/octeon_nic23_defconfig | 1 - 2 files changed, 1 insertion(+), 2 deletions(-)This seems similar to:
https://patchwork.ozlabs.org/project/uboot/patch/[email protected]/at least for CI. And for outside CI, I'm OK with just having the value be changed in the defconfig as needed. We do support using config fragments, so keeping such changes locally isn't too hard.
Quoted 6 lines
Quoted 2 lines
So the default value is a little bit too hard even for some of the actual hardware.
Right, there's some platforms where it's too small and we should just bump it up. I think for now the default is what we want it to be for most platforms.
The current default value is definitely too small, especially when CI is involved (I did not have this in mind when implementing), so:
Quoted 6 lines
Quoted 1 line
Acked-by Stefan Roese <[email protected]>Can we please get either this, or
https://patchwork.ozlabs.org/project/uboot/patch/[email protected]/
merged for master? The number of false negatives in CI due to this is big issue for getting more contributors to use CI. Thanks.
Yes.
Applied to u-boot-watchdog/master
Thanks,
Stefan
Re: [PATCH 12/20] cyclic: Rise default CYCLIC_MAX_CPU_TIME_US to 5000
In reply to Tom Rini在2024年6月18日六月 上午12:29,Tom Rini写道:
[...]
Quoted 6 lines
Quoted 1 line
Acked-by Stefan Roese <[email protected]>Can we please get either this, or
https://patchwork.ozlabs.org/project/uboot/patch/[email protected]/
merged for master? The number of false negatives in CI due to this is big issue for getting more contributors to use CI. Thanks.
It seems like CYCLIC doesn't have it's own custodian tree. How are we going to handle it in this case?
Thanks
- Jiaxun
Quoted 6 lines
Signature · 5 lines
-- Tom 附件: * signature.asc
Signature · 1 line
- Jiaxun
Re: [PATCH 12/20] cyclic: Rise default CYCLIC_MAX_CPU_TIME_US to 5000
Hi Jiaxun,
Quoted 6 lines · Jiaxun Yang
在2024年6月18日六月 上午12:29,Tom Rini写道:
[...]Quoted 2 lines
Quoted 1 line
Acked-by Stefan Roese <[email protected]>
Quoted 7 lines
Quoted 4 lines
Can we please get either this, or
https://patchwork.ozlabs.org/project/uboot/patch/[email protected]/
merged for master? The number of false negatives in CI due to this is big issue for getting more contributors to use CI. Thanks.It seems like CYCLIC doesn't have it's own custodian tree. How are we going to handle it in this case?
I've already pulled this patch. I introduced the cyclic framework some time ago and as it's closely related to the watchdog stuff, I'm currently handling these patches as well.
Thanks,
Stefan
Re: [PATCH 12/20] cyclic: Rise default CYCLIC_MAX_CPU_TIME_US to 5000
In reply to Jiaxun YangQuoted 6 lines · Jiaxun Yang
在2024年6月18日六月 上午12:29,Tom Rini写道:
[...]Quoted 2 lines
Quoted 1 line
Acked-by Stefan Roese <[email protected]>
Quoted 7 lines
Quoted 4 lines
Can we please get either this, or
https://patchwork.ozlabs.org/project/uboot/patch/[email protected]/
merged for master? The number of false negatives in CI due to this is big issue for getting more contributors to use CI. Thanks.It seems like CYCLIC doesn't have it's own custodian tree. How are we going to handle it in this case?
Note that in U-Boot it's less important to have a tree per sub-section and more important that custodians have a tree (and then use appropriate tags when sending pull requests).
Signature · 1 line
Tom
Re: [PATCH 12/20] cyclic: Rise default CYCLIC_MAX_CPU_TIME_US to 5000
Quoted 16 lines · Tom Rini
Quoted 6 lines · Jiaxun Yang
在2024年6月18日六月 上午12:29,Tom Rini写道:
[...]Quoted 2 lines
Quoted 1 line
Acked-by Stefan Roese <[email protected]>Quoted 7 lines
Quoted 4 lines
Can we please get either this, or
https://patchwork.ozlabs.org/project/uboot/patch/[email protected]/
merged for master? The number of false negatives in CI due to this is big issue for getting more contributors to use CI. Thanks.It seems like CYCLIC doesn't have it's own custodian tree. How are we going to handle it in this case?
Quoted 6 lines
Note that in U-Boot it's less important to have a tree per sub-section and more important that custodians have a tree (and then use appropriate tags when sending pull requests).
--
Tom
Stefan and Tom,
I'm seeing CI issues here even with 5000us [1]:
host bind 0 /tmp/sandbox/persistent-cyclic function wdt-gpio-level took too long: 5368us vs 5000us max
Best Regards,
Re: [PATCH 12/20] cyclic: Rise default CYCLIC_MAX_CPU_TIME_US to 5000
Quoted 8 lines · Tim Harvey
Quoted 1 line · Tom Rini
Stefan and Tom,
I'm seeing CI issues here even with 5000us [1]:
host bind 0/tmp/sandbox/persistent-cyclicfunction wdt-gpio-level took too long: 5368us vs 5000us max
Yes, 5ms is way too little when you're not the only thing running on the cpu, which is why I went with 100ms.
Random thoughts and questions:
(1) Do we have any way to programmatically grab all the logs from azure CI, so we can get some kind of objective statistics on the number after "took too long:". Clicking through the web interface and randomly searching is too painful.
It would also be helpful to know what percentage of CI runs have failed due to that, versus due to some genuine error.
(2) I considered a patch that just added a
default $something big if SANDBOX
to config CYCLIC_MAX_CPU_TIME_US, but since the problem also hit qemu, I dropped that. But, if my patch is too ugly (and I might tend to think that myself...), perhaps at least this would be an added improvement over the generic bump to 5000us.
(3) I also thought that perhaps for sandbox, we should simply measure the time using clock_gettime(CLOCK_PROCESS_CPUTIME_ID), instead of wallclock time. But it's a little ugly to implement since the "now" variable is both used to decide if its time to run the callback, and as a starting point for measuring cpu time, and we probably still want the "is it time" to be measured on wallclock and not however much cpu-time the u-boot process has been given. Or maybe we don't, and CLOCK_PROCESS_CPUTIME_ID would simply be a better backend for os_get_nsec(). Sure, time in the sandbox would progress slower than on the host, but does that actually matter?
(4) Btw., what kind of clock tick do we even get when run under qemu? I don't have much experience with qemu, but from quick googling it seems that -icount would be interesting. Also see
https://github.com/zephyrproject-rtos/zephyr/issues/14173 . From quick reading it seems there were some issues back in 2019, but that today it mostly works for them, except some SMP issues (that are certainly not relevant to U-Boot).
The current situation is a frustrating waste of developer and maintainer time and CI resources.
Rasmus
Re: [PATCH 12/20] cyclic: Rise default CYCLIC_MAX_CPU_TIME_US to 5000
Quoted 15 lines · Rasmus Villemoes
Quoted 8 lines · Tim Harvey
Quoted 1 line · Tom Rini
Stefan and Tom,
I'm seeing CI issues here even with 5000us [1]:
host bind 0/tmp/sandbox/persistent-cyclicfunction wdt-gpio-level took too long: 5368us vs 5000us maxYes, 5ms is way too little when you're not the only thing running on the cpu, which is why I went with 100ms.
Random thoughts and questions:
Quoted 7 lines
(1) Do we have any way to programmatically grab all the logs from azure CI, so we can get some kind of objective statistics on the number after "took too long:". Clicking through the web interface and randomly searching is too painful.
It would also be helpful to know what percentage of CI runs have failed due to that, versus due to some genuine error.
I don't think we can easily grab logs via API. And at least for
https://dev.azure.com/u-boot/u-boot/_build?definitionId=2&_a=summary a lot of the failures have rolled off already because we only get so many logs and I try and mark full release logs as keep forever. But anecdotally I can say 75%+ of the Azure runs fail at least once due to this, and most fail enough to fail the pipeline (we get 2 tries per job).
Quoted 8 lines
(2) I considered a patch that just added a
default $something big if SANDBOX
to config
CYCLIC_MAX_CPU_TIME_US, but since the problem also hit qemu, I dropped that. But, if my patch is too ugly (and I might tend to think that myself...), perhaps at least this would be an added improvement over the generic bump to 5000us.
I was fine with your approach really, maybe a bit bigger of a comment and note under doc/ as well why we do it?
Quoted 11 lines
(3) I also thought that perhaps for sandbox, we should simply measure the time using
clock_gettime(CLOCK_PROCESS_CPUTIME_ID), instead of wallclock time. But it's a little ugly to implement since the "now" variable is both used to decide if its time to run the callback, and as a starting point for measuring cpu time, and we probably still want the "is it time" to be measured on wallclock and not however much cpu-time the u-boot process has been given. Or maybe we don't, andCLOCK_PROCESS_CPUTIME_IDwould simply be a better backend foros_get_nsec(). Sure, time in the sandbox would progress slower than on the host, but does that actually matter?
Quoted 7 lines
(4) Btw., what kind of clock tick do we even get when run under qemu? I don't have much experience with qemu, but from quick googling it seems that -icount would be interesting. Also see
https://github.com/zephyrproject-rtos/zephyr/issues/14173 . From quick reading it seems there were some issues back in 2019, but that today it mostly works for them, except some SMP issues (that are certainly not relevant to U-Boot).
That could be interesting too, yeah. I do worry it might open its own set of problems to figure out tho.
Signature · 1 line
Tom
PATCH 13/20 CI: Ensure pip install is always performed in venv
In reply to Jiaxun YangSince Ubuntu focal it's nolonger permitted to perform global pip install.
Ensure that pip install is always performed in venv. For buildman alone, all dependencies are already in docker so there is no need to perform pip install.
--- .azure-pipelines.yml | 16 ++++++++++------ .gitlab-ci.yml | 13 ++++++++----- 2 files changed, 18 insertions(+), 11 deletions(-)
.azure-pipelines.yml +10 -6
@@ -150,6 +150,8 @@ stages: - script: | git config --global --add safe.directory $(work_dir) export USER=azure + virtualenv -p /usr/bin/python3 /tmp/venv + . /tmp/venv/bin/activate pip install -r test/py/requirements.txt pip install -r tools/buildman/requirements.txt pip install asteval pylint==3.2.3 pyopenssl
@@ -183,7 +185,10 @@ stages: image: $(ci_runner_image) options: $(container_option) steps: - - script: make pip + - script: | + virtualenv -p /usr/bin/python3 /tmp/venv + . /tmp/venv/bin/activate + make pip - job: create_test_py_wrapper_script displayName: 'Create and stage a wrapper for test.py runs'
@@ -217,7 +222,11 @@ stages: if [ -n "\${BUILD_ENV}" ]; then export \${BUILD_ENV}; fi + virtualenv -p /usr/bin/python3 /tmp/venv + . /tmp/venv/bin/activate pip install -r tools/buildman/requirements.txt + pip install -r test/py/requirements.txt + pip install pytest-azurepipelines tools/buildman/buildman -o \${UBOOT_TRAVIS_BUILD_DIR} -w -E -W -e --board \${TEST_PY_BD} \${OVERRIDE} cp ~/grub_x86.efi \${UBOOT_TRAVIS_BUILD_DIR}/ cp ~/grub_x64.efi \${UBOOT_TRAVIS_BUILD_DIR}/
@@ -241,10 +250,6 @@ stages: /opt/coreboot/cbfstool \${UBOOT_TRAVIS_BUILD_DIR}/coreboot.rom remove -n fallback/payload; /opt/coreboot/cbfstool \${UBOOT_TRAVIS_BUILD_DIR}/coreboot.rom add-flat-binary -f \${UBOOT_TRAVIS_BUILD_DIR}/u-boot.bin -n fallback/payload -c LZMA -l 0x1110000 -e 0x1110000; fi - virtualenv -p /usr/bin/python3 /tmp/venv - . /tmp/venv/bin/activate - pip install -r test/py/requirements.txt - pip install pytest-azurepipelines export PATH=/opt/qemu/bin:/tmp/uboot-test-hooks/bin:\${PATH} export PYTHONPATH=/tmp/uboot-test-hooks/py/travis-ci # "\${var:+"-k \$var"}" expands to "" if \$var is empty, "-k \$var" if not
@@ -504,7 +509,6 @@ stages: # make environment variables available as tests are running inside a container export BUILDMAN="${BUILDMAN}" git config --global --add safe.directory ${WORK_DIR} - pip install -r tools/buildman/requirements.txt EOF cat << "EOF" >> build.sh if [[ "${BUILDMAN}" != "" ]]; then
.gitlab-ci.yml +8 -5
@@ -50,6 +50,10 @@ stages: - if [ -n "${BUILD_ENV}" ]; then export ${BUILD_ENV}; fi + - virtualenv -p /usr/bin/python3 /tmp/venv + - . /tmp/venv/bin/activate + - pip install -r tools/buildman/requirements.txt + - pip install -r test/py/requirements.txt - tools/buildman/buildman -o ${UBOOT_TRAVIS_BUILD_DIR} -w -E -W -e --board ${TEST_PY_BD} ${OVERRIDE} - cp ~/grub_x86.efi $UBOOT_TRAVIS_BUILD_DIR/
@@ -74,9 +78,6 @@ stages: /opt/coreboot/cbfstool ${UBOOT_TRAVIS_BUILD_DIR}/coreboot.rom remove -n fallback/payload; /opt/coreboot/cbfstool ${UBOOT_TRAVIS_BUILD_DIR}/coreboot.rom add-flat-binary -f ${UBOOT_TRAVIS_BUILD_DIR}/u-boot.bin -n fallback/payload -c LZMA -l 0x1110000 -e 0x1110000; fi - - virtualenv -p /usr/bin/python3 /tmp/venv - - . /tmp/venv/bin/activate - - pip install -r test/py/requirements.txt # "${var:+"-k $var"}" expands to "" if $var is empty, "-k $var" if not - export PATH=/opt/qemu/bin:/tmp/uboot-test-hooks/bin:${PATH}; export PYTHONPATH=/tmp/uboot-test-hooks/py/travis-ci;
@@ -100,7 +101,6 @@ build all 32bit ARM platforms: script: - ret=0; git config --global --add safe.directory "${CI_PROJECT_DIR}"; - pip install -r tools/buildman/requirements.txt; ./tools/buildman/buildman -o /tmp -PEWM arm -x aarch64 || ret=$?; if [[ $ret -ne 0 ]]; then ./tools/buildman/buildman -o /tmp -seP;
@@ -114,7 +114,6 @@ build all 64bit ARM platforms: - . /tmp/venv/bin/activate - ret=0; git config --global --add safe.directory "${CI_PROJECT_DIR}"; - pip install -r tools/buildman/requirements.txt; ./tools/buildman/buildman -o /tmp -PEWM aarch64 || ret=$?; if [[ $ret -ne 0 ]]; then ./tools/buildman/buildman -o /tmp -seP;
@@ -212,6 +211,8 @@ Run pylint: extends: .testsuites script: - git config --global --add safe.directory "${CI_PROJECT_DIR}" + - virtualenv -p /usr/bin/python3 /tmp/venv; + - . /tmp/venv/bin/activate; - pip install -r test/py/requirements.txt - pip install -r tools/buildman/requirements.txt - pip install asteval pylint==3.2.3 pyopenssl
@@ -240,6 +241,8 @@ Check for pre-schema tags: Check packing of Python tools: extends: .testsuites script: + - virtualenv -p /usr/bin/python3 /tmp/venv; + - . /tmp/venv/bin/activate; - make pip # Test sandbox with test.py
Signature · 1 line
2.43.0
Re: [PATCH 13/20] CI: Ensure pip install is always performed in venv
Quoted 8 lines · Jiaxun Yang
Since Ubuntu focal it's nolonger permitted to perform global pip install.
Ensure that pip install is always performed in venv. For buildman alone, all dependencies are already in docker so there is no need to perform pip install.
Signed-off-by Jiaxun Yang <[email protected]>
Signature · 1 line
Tom
PATCH 14/20 CI: GitLab: Split build_world tasks
In reply to Jiaxun YangCurrent build_world task runs for too long on public gitlab runner.
Split the job as what we've done to azure pipeline.
--- .gitlab-ci.yml | 103 +++++++++++++++++++++++++++++++++------------------------ 1 file changed, 59 insertions(+), 44 deletions(-)
.gitlab-ci.yml +59 -44
@@ -91,56 +91,71 @@ stages: - "*.css" expire_in: 1 week -.world_build: +.world_build_template: &world_build_dfn stage: world build rules: - when: always - -build all 32bit ARM platforms: - extends: .world_build - script: - - ret=0; - git config --global --add safe.directory "${CI_PROJECT_DIR}"; - ./tools/buildman/buildman -o /tmp -PEWM arm -x aarch64 || ret=$?; - if [[ $ret -ne 0 ]]; then - ./tools/buildman/buildman -o /tmp -seP; - exit $ret; - fi; - -build all 64bit ARM platforms: - extends: .world_build - script: - - virtualenv -p /usr/bin/python3 /tmp/venv - - . /tmp/venv/bin/activate - - ret=0; - git config --global --add safe.directory "${CI_PROJECT_DIR}"; - ./tools/buildman/buildman -o /tmp -PEWM aarch64 || ret=$?; - if [[ $ret -ne 0 ]]; then - ./tools/buildman/buildman -o /tmp -seP; - exit $ret; - fi; - -build all PowerPC platforms: - extends: .world_build script: - ret=0; git config --global --add safe.directory "${CI_PROJECT_DIR}"; - ./tools/buildman/buildman -o /tmp -P -E -W powerpc || ret=$?; - if [[ $ret -ne 0 ]]; then - ./tools/buildman/buildman -o /tmp -seP; - exit $ret; - fi; - -build all other platforms: - extends: .world_build - script: - - ret=0; - git config --global --add safe.directory "${CI_PROJECT_DIR}"; - ./tools/buildman/buildman -o /tmp -PEWM -x arm,powerpc || ret=$?; - if [[ $ret -ne 0 ]]; then - ./tools/buildman/buildman -o /tmp -seP; - exit $ret; - fi; + if [[ "${BUILDMAN}" != "" ]]; then + ret=0; + tools/buildman/buildman -o /tmp -PEWM ${BUILDMAN} ${OVERRIDE} || ret=$?; + if [[ $ret -ne 0 ]]; then + tools/buildman/buildman -o /tmp -seP ${BUILDMAN}; + exit $ret; + fi; + fi + +am33xx_at91_kirkwood_mvebu_omap: + variables: + BUILDMAN: "am33xx at91_kirkwood mvebu omap -x siemens" + <<: *world_build_dfn + +amlogic_bcm_boundary_engicam_siemens_technexion_oradex: + variables: + BUILDMAN: "amlogic bcm boundary engicam siemens technexion toradex -x mips" + <<: *world_build_dfn + +arm_nxp_minus_imx: + variables: + BUILDMAN: "freescale -x powerpc,m68k,imx,mx" + <<: *world_build_dfn + +imx: + variables: + BUILDMAN: "mx imx -x boundary,engicam,technexion,toradex" + <<: *world_build_dfn + +rk: + variables: + BUILDMAN: "rk" + <<: *world_build_dfn + +sunxi: + variables: + BUILDMAN: "sunxi" + <<: *world_build_dfn + +powerpc: + variables: + BUILDMAN: "powerpc" + <<: *world_build_dfn + +arm_catch_all: + variables: + BUILDMAN: "arm -x aarch64,am33xx,at91,bcm,ls1,kirkwood,mvebu,omap,rk,siemens,mx,sunxi,technexion,toradex" + <<: *world_build_dfn + +aarch64_catch_all: + variables: + BUILDMAN: "aarch64 -x amlogic,bcm,engicam,imx,ls1,ls2,lx216,mvebu,rk,siemens,sunxi,toradex" + <<: *world_build_dfn + +everything_but_arm_and_powerpc: + variables: + BUILDMAN: "-x arm,powerpc" + <<: *world_build_dfn .testsuites: stage: testsuites
Signature · 1 line
2.43.0
Re: [PATCH 14/20] CI: GitLab: Split build_world tasks
Quoted 9 lines · Jiaxun Yang
Current
build_worldtask runs for too long on public gitlab runner.Split the job as what we've done to azure pipeline.
Signed-off-by Jiaxun Yang <[email protected]>--- .gitlab-ci.yml | 103 +++++++++++++++++++++++++++++++++------------------------ 1 file changed, 59 insertions(+), 44 deletions(-)
I don't like this. The list in Azure is because of the time limits there and in turn we:
a) Have to tweak it periodically to keep things from running too long
b) Have to tweak it to ensure that we don't miss some new SoC/etc
Signature · 1 line
Tom
Re: [PATCH 14/20] CI: GitLab: Split build_world tasks
Quoted 12 lines · 在2024年6月12日六月 下午5:01,Tom Rini写道:
Quoted 9 lines · Jiaxun Yang
Current
build_worldtask runs for too long on public gitlab runner.Split the job as what we've done to azure pipeline.
Signed-off-by Jiaxun Yang <[email protected]>--- .gitlab-ci.yml | 103 +++++++++++++++++++++++++++++++++------------------------ 1 file changed, 59 insertions(+), 44 deletions(-)
Quoted 4 lines
I don't like this. The list in Azure is because of the time limits there and in turn we:
a) Have to tweak it periodically to keep things from running too long
b) Have to tweak it to ensure that we don't miss some new SoC/etc
Then it will render running CI test on gitlab.com impossible again :-(
Thanks
Quoted 6 lines
Signature · 5 lines
-- Tom 附件: * signature.asc
Signature · 1 line
- Jiaxun
Re: [PATCH 14/20] CI: GitLab: Split build_world tasks
Quoted 15 lines · Jiaxun Yang
Quoted 12 lines · 在2024年6月12日六月 下午5:01,Tom Rini写道:
Quoted 9 lines · Jiaxun Yang
Current
build_worldtask runs for too long on public gitlab runner.Split the job as what we've done to azure pipeline.
Signed-off-by Jiaxun Yang <[email protected]>--- .gitlab-ci.yml | 103 +++++++++++++++++++++++++++++++++------------------------ 1 file changed, 59 insertions(+), 44 deletions(-)
Quoted 6 lines
Quoted 4 lines
I don't like this. The list in Azure is because of the time limits there and in turn we:
a) Have to tweak it periodically to keep things from running too long
b) Have to tweak it to ensure that we don't miss some new SoC/etcThen it will render running CI test on gitlab.com impossible again :-(
Yeah, it's not something I'm the happiest with. Looking around a bit, I see a blog post that talks about dealing with dynamic variables, in Azure. So we could, I think, figure out some logic to have each build stage say what platforms it covers. And then have a final step that compares all of the platforms built vs the global list (just tools/buildman/buildman --dry-run -v) to make sure nothing was missed. With something like that, and assuming GitLab can do it too (it probably can), I'm OK with having the world build be broken down to 10 groups (maximum number of parallel jobs in Azure CI for free) since we'll know if we miss something too.
So lets set this patch (and the doc update) aside for now, unless you want to look at the above. I'll look at the above soon.
Signature · 1 line
Tom
Re: [PATCH 14/20] CI: GitLab: Split build_world tasks
Hi Tom,
Quoted 35 lines · Tom Rini
Quoted 15 lines · Jiaxun Yang
Quoted 12 lines · 在2024年6月12日六月 下午5:01,Tom Rini写道:
Quoted 9 lines · Jiaxun Yang
Current
build_worldtask runs for too long on public gitlab runner.Split the job as what we've done to azure pipeline.
Signed-off-by Jiaxun Yang <[email protected]>--- .gitlab-ci.yml | 103 +++++++++++++++++++++++++++++++++------------------------ 1 file changed, 59 insertions(+), 44 deletions(-)Quoted 6 lines
Quoted 4 lines
I don't like this. The list in Azure is because of the time limits there and in turn we:
a) Have to tweak it periodically to keep things from running too long
b) Have to tweak it to ensure that we don't miss some new SoC/etcThen it will render running CI test on gitlab.com impossible again :-(
Yeah, it's not something I'm the happiest with. Looking around a bit, I see a blog post that talks about dealing with dynamic variables, in Azure. So we could, I think, figure out some logic to have each build stage say what platforms it covers. And then have a final step that compares all of the platforms built vs the global list (just
tools/buildman/buildman--dry-run -v) to make sure nothing was missed. With something like that, and assuming GitLab can do it too (it probably can), I'm OK with having the world build be broken down to 10 groups (maximum number of parallel jobs in Azure CI for free) since we'll know if we miss something too.
Quoted 2 lines
So lets set this patch (and the doc update) aside for now, unless you want to look at the above. I'll look at the above soon.
Could we hook up one of our machines as a public runner somehow?
Regards,
Simon
Re: [PATCH 14/20] CI: GitLab: Split build_world tasks
Quoted 38 lines · Simon Glass
Hi Tom,
Quoted 35 lines · Tom Rini
Quoted 15 lines · Jiaxun Yang
Quoted 12 lines · 在2024年6月12日六月 下午5:01,Tom Rini写道:
Quoted 9 lines · Jiaxun Yang
Current
build_worldtask runs for too long on public gitlab runner.Split the job as what we've done to azure pipeline.
Signed-off-by Jiaxun Yang <[email protected]>--- .gitlab-ci.yml | 103 +++++++++++++++++++++++++++++++++------------------------ 1 file changed, 59 insertions(+), 44 deletions(-)Quoted 6 lines
Quoted 4 lines
I don't like this. The list in Azure is because of the time limits there and in turn we:
a) Have to tweak it periodically to keep things from running too long
b) Have to tweak it to ensure that we don't miss some new SoC/etcThen it will render running CI test on gitlab.com impossible again :-(
Yeah, it's not something I'm the happiest with. Looking around a bit, I see a blog post that talks about dealing with dynamic variables, in Azure. So we could, I think, figure out some logic to have each build stage say what platforms it covers. And then have a final step that compares all of the platforms built vs the global list (just
tools/buildman/buildman--dry-run -v) to make sure nothing was missed. With something like that, and assuming GitLab can do it too (it probably can), I'm OK with having the world build be broken down to 10 groups (maximum number of parallel jobs in Azure CI for free) since we'll know if we miss something too.
Quoted 4 lines
Quoted 2 lines
So lets set this patch (and the doc update) aside for now, unless you want to look at the above. I'll look at the above soon.
Could we hook up one of our machines as a public runner somehow?
It's a pool of free runners, ala Azure. I don't think we can, no, and I don't want to make all of our runners available on public CI either, they're a bit over-whelmed as it is I think once custodians start working.
Signature · 1 line
Tom
PATCH 15/20 CI: Dockerfile: Set global git name & email config
In reply to Jiaxun YangSet global git name & email config so we don't have to setup it for every project.
--- tools/docker/Dockerfile | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
tools/docker/Dockerfile +4 -5
@@ -121,6 +121,10 @@ RUN apt-get update && apt-get install -y \ zip \ && rm -rf /var/lib/apt/lists/* +# Setup Git +RUN git config --global user.name "U-Boot CI" && \ + git config --global user.email [email protected] + # Make kernels readable for libguestfs tools to work correctly RUN chmod +r /boot/vmlinu*
@@ -128,8 +132,6 @@ RUN chmod +r /boot/vmlinu* RUN git clone git://git.savannah.gnu.org/grub.git /tmp/grub && \ cd /tmp/grub && \ git checkout grub-2.06 && \ - git config --global user.name "GitLab CI Runner" && \ - git config --global user.email [email protected] && \ git cherry-pick 049efdd72eb7baa7b2bf8884391ee7fe650da5a0 && \ git cherry-pick 403d6540cd608b2706cfa0cb4713f7e4b490ff45 && \ ./bootstrap && \
@@ -180,9 +182,6 @@ RUN git clone git://git.savannah.gnu.org/grub.git /tmp/grub && \ RUN git clone https://gitlab.com/qemu-project/qemu.git /tmp/qemu && \ cd /tmp/qemu && \ git checkout v8.2.0 && \ - # config user.name and user.email to make 'git am' happy - git config user.name u-boot && \ - git config user.email [email protected] && \ git format-patch 0c7ffc977195~..0c7ffc977195 && \ git am 0001-hw-net-cadence_gem-Fix-MDIO_OP_xxx-values.patch && \ git cherry-pick d3c79c3974 && \
Signature · 1 line
2.43.0
PATCH 16/20 CI: Dockerfile: Bump various software version
In reply to Jiaxun YangBump base OS to Ubuntu noble, grub to 2.12, QEMU to 9.0 (drop cherry-pick for m68k as it's already in 9.0) and other tools to latest version.
Remove unsupported python2.
Install some required python packages for buildman/binman.
--- tools/docker/Dockerfile | 35 +++++++++++++++++------------------ 1 file changed, 17 insertions(+), 18 deletions(-)
tools/docker/Dockerfile +17 -18
@@ -2,7 +2,7 @@ # This Dockerfile is used to build an image containing basic stuff to be used # to build U-Boot and run our test suites. -FROM ubuntu:jammy-20240227 +FROM ubuntu:noble MAINTAINER Tom Rini <[email protected]> LABEL Description=" This image is for building U-Boot inside a container"
@@ -12,7 +12,7 @@ ENV DEBIAN_FRONTEND=noninteractive # Add LLVM repository RUN apt-get update && apt-get install -y gnupg2 wget xz-utils && rm -rf /var/lib/apt/lists/* RUN wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - -RUN echo deb http://apt.llvm.org/jammy/ llvm-toolchain-jammy-17 main | tee /etc/apt/sources.list.d/llvm.list +RUN echo deb http://apt.llvm.org/noble/ llvm-toolchain-noble-17 main | tee /etc/apt/sources.list.d/llvm.list # Manually install the kernel.org "Crosstool" based toolchains for gcc-13.2.0 RUN wget -O - https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.2.0/x86_64-gcc-13.2.0-nolibc-aarch64-linux.tar.xz | tar -C /opt -xJ
@@ -55,6 +55,7 @@ RUN apt-get update && apt-get install -y \ gawk \ gdisk \ git \ + gnat \ gnu-efi \ gnutls-dev \ graphviz \
@@ -95,13 +96,14 @@ RUN apt-get update && apt-get install -y \ parted \ pkg-config \ python-is-python3 \ - python2.7 \ python3 \ python3-dev \ + python3-jsonschema \ python3-pip \ python3-pyelftools \ python3-sphinx \ python3-virtualenv \ + python3-yaml \ rpm2cpio \ sbsigntool \ socat \
@@ -118,6 +120,7 @@ RUN apt-get update && apt-get install -y \ vboot-utils \ xilinx-bootgen \ xxd \ + yamllint \ zip \ && rm -rf /var/lib/apt/lists/*
@@ -131,9 +134,7 @@ RUN chmod +r /boot/vmlinu* # Build GRUB UEFI targets for ARM & RISC-V, 32-bit and 64-bit RUN git clone git://git.savannah.gnu.org/grub.git /tmp/grub && \ cd /tmp/grub && \ - git checkout grub-2.06 && \ - git cherry-pick 049efdd72eb7baa7b2bf8884391ee7fe650da5a0 && \ - git cherry-pick 403d6540cd608b2706cfa0cb4713f7e4b490ff45 && \ + git checkout grub-2.12 && \ ./bootstrap && \ mkdir -p /opt/grub && \ ./configure --target=aarch64 --with-platform=efi \
@@ -181,10 +182,7 @@ RUN git clone git://git.savannah.gnu.org/grub.git /tmp/grub && \ RUN git clone https://gitlab.com/qemu-project/qemu.git /tmp/qemu && \ cd /tmp/qemu && \ - git checkout v8.2.0 && \ - git format-patch 0c7ffc977195~..0c7ffc977195 && \ - git am 0001-hw-net-cadence_gem-Fix-MDIO_OP_xxx-values.patch && \ - git cherry-pick d3c79c3974 && \ + git checkout v9.0.0 && \ ./configure --prefix=/opt/qemu --target-list="aarch64-softmmu,arm-softmmu,i386-softmmu,m68k-softmmu,mips-softmmu,mips64-softmmu,mips64el-softmmu,mipsel-softmmu,ppc-softmmu,riscv32-softmmu,riscv64-softmmu,sh4-softmmu,x86_64-softmmu,xtensa-softmmu" && \ make -j$(nproc) all install && \ rm -rf /tmp/qemu
@@ -192,7 +190,7 @@ RUN git clone https://gitlab.com/qemu-project/qemu.git /tmp/qemu && \ # Build fiptool RUN git clone https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git /tmp/tf-a && \ cd /tmp/tf-a/ && \ - git checkout v2.10.0 && \ + git checkout lts-v2.10.4 && \ cd tools/fiptool && \ make && \ mkdir -p /usr/local/bin && \
@@ -200,12 +198,12 @@ RUN git clone https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git /tmp/t rm -rf /tmp/tf-a # Build genimage (required by some targets to generate disk images) -RUN wget -O - https://github.com/pengutronix/genimage/releases/download/v14/genimage-14.tar.xz | tar -C /tmp -xJ && \ - cd /tmp/genimage-14 && \ +RUN wget -O - https://github.com/pengutronix/genimage/releases/download/v17/genimage-17.tar.xz | tar -C /tmp -xJ && \ + cd /tmp/genimage-17 && \ ./configure && \ make -j$(nproc) && \ make install && \ - rm -rf /tmp/genimage-14 + rm -rf /tmp/genimage-17 # Build libtpms RUN git clone https://github.com/stefanberger/libtpms /tmp/libtpms && \
@@ -236,22 +234,23 @@ RUN mkdir /tmp/trace && \ cd /tmp/trace/libtracefs && \ make -j$(nproc) && \ sudo make install && \ - git clone https://github.com/rostedt/trace-cmd.git /tmp/trace/trace-cmd && \ + git clone https://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git /tmp/trace/trace-cmd && \ cd /tmp/trace/trace-cmd && \ make -j$(nproc) && \ sudo make install && \ rm -rf /tmp/trace # Build coreboot -RUN wget -O - https://coreboot.org/releases/coreboot-4.22.01.tar.xz | tar -C /tmp -xJ && \ - cd /tmp/coreboot-4.22.01 && \ +RUN wget -O - https://coreboot.org/releases/coreboot-24.05.tar.xz | tar -C /tmp -xJ && \ + cd /tmp/coreboot-24.05 && \ make crossgcc-i386 CPUS=$(nproc) && \ make -C payloads/coreinfo olddefconfig && \ make -C payloads/coreinfo && \ make olddefconfig && \ make -j $(nproc) && \ sudo mkdir /opt/coreboot && \ - sudo cp build/coreboot.rom build/cbfstool /opt/coreboot/ + sudo cp build/coreboot.rom build/cbfstool /opt/coreboot/ && \ + rm -rf /tmp/coreboot-24.05 # Create our user/group RUN echo uboot ALL=NOPASSWD: ALL > /etc/sudoers.d/uboot
Signature · 1 line
2.43.0
Re: [PATCH 16/20] CI: Dockerfile: Bump various software version
Quoted 18 lines · Jiaxun Yang
Bump base OS to Ubuntu noble, grub to 2.12, QEMU to 9.0 (drop cherry-pick for m68k as it's already in 9.0) and other tools to latest version.
Remove unsupported python2.
Install some required python packages for buildman/binman.
Signed-off-by Jiaxun Yang <[email protected]>--- tools/docker/Dockerfile | 35 +++++++++++++++++------------------ 1 file changed, 17 insertions(+), 18 deletions(-)tools/docker/Dockerfile +0
-0
tools/docker/Dockerfile line 8
Quoted 8 lines
tools/docker/Dockerfile +1
-1@@ -2,7 +2,7 @@ # This Dockerfile is used to build an image containing basic stuff to be used # to build U-Boot and run our test suites. -FROM ubuntu:jammy-20240227 +FROM ubuntu:noble MAINTAINER Tom Rini <[email protected]> LABEL Description=" This image is for building U-Boot inside a container"
For reproducibility I keep us on date-tags and not just the raw release tag/latest. Also, I'm not yet ready to move us past Jammy without a big reason. Once 24.04.1 comes out, perhaps. So it's good to know at least this wasn't too much work to move forward. Also, we should keep the bumping of various tools to their own commit, each.
Signature · 1 line
Tom
Re: [PATCH 16/20] CI: Dockerfile: Bump various software version
在2024年6月12日六月 下午5:02,Tom Rini写道:
[...]
Quoted 6 lines
Quoted 6 lines
tools/docker/Dockerfile +0
-0@@ -2,7 +2,7 @@ # This Dockerfile is used to build an image containing basic stuff to be used # to build U-Boot and run our test suites.
Quoted 11 lines
tools/docker/Dockerfile line 8
Quoted 5 lines
tools/docker/Dockerfile +1
-1-FROM ubuntu:jammy-20240227 +FROM ubuntu:noble MAINTAINER Tom Rini <[email protected]> LABEL Description=" This image is for building U-Boot inside a container"For reproducibility I keep us on date-tags and not just the raw release tag/latest. Also, I'm not yet ready to move us past Jammy without a big reason. Once 24.04.1 comes out, perhaps. So it's good to know at least this wasn't too much work to move forward. Also, we should keep the bumping of various tools to their own commit, each.
Ok, I'll add tag back.
My purpose to move CI to noble is to test against all new pieces of software. I was hit by those issues when I was trying to run tests locally with my ArchLinux PC, so I decided to fix them and bump CI to ensure we don't introduce more in future.
I think as we can pass all CI runs it's good to move forward :-)
Thanks
Quoted 6 lines
Signature · 5 lines
-- Tom 附件: * signature.asc
Signature · 1 line
- Jiaxun
PATCH 17/20 CI: Dockerfile: Add LoongArch64 support
In reply to Jiaxun YangInstall LoongArch64 toolchains, build LoongArch64 QEMU, build LoongArch64 GRUB.
--- tools/docker/Dockerfile | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-)
tools/docker/Dockerfile +19 -2
@@ -19,6 +19,7 @@ RUN wget -O - https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_ RUN wget -O - https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.2.0/x86_64-gcc-13.2.0-nolibc-arc-linux.tar.xz | tar -C /opt -xJ RUN wget -O - https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.2.0/x86_64-gcc-13.2.0-nolibc-arm-linux-gnueabi.tar.xz | tar -C /opt -xJ RUN wget -O - https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.2.0/x86_64-gcc-13.2.0-nolibc-i386-linux.tar.xz | tar -C /opt -xJ +RUN wget -O - https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.2.0/x86_64-gcc-13.2.0-nolibc-loongarch64-linux.tar.xz | tar -C /opt -xJ RUN wget -O - https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.2.0/x86_64-gcc-13.2.0-nolibc-m68k-linux.tar.xz | tar -C /opt -xJ RUN wget -O - https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.2.0/x86_64-gcc-13.2.0-nolibc-mips-linux.tar.xz | tar -C /opt -xJ RUN wget -O - https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.2.0/x86_64-gcc-13.2.0-nolibc-microblaze-linux.tar.xz | tar -C /opt -xJ
@@ -131,7 +132,7 @@ RUN git config --global user.name "U-Boot CI" && \ # Make kernels readable for libguestfs tools to work correctly RUN chmod +r /boot/vmlinu* -# Build GRUB UEFI targets for ARM & RISC-V, 32-bit and 64-bit +# Build GRUB UEFI targets for ARM & LoongArch64 & RISC-V, 32-bit and 64-bit RUN git clone git://git.savannah.gnu.org/grub.git /tmp/grub && \ cd /tmp/grub && \ git checkout grub-2.12 && \
@@ -165,6 +166,20 @@ RUN git clone git://git.savannah.gnu.org/grub.git /tmp/grub && \ search search_fs_file search_fs_uuid search_label serial sleep test \ true && \ make clean && \ + ./configure --target=loongarch64 --with-platform=efi \ + CC=gcc \ + TARGET_CC=/opt/gcc-13.2.0-nolibc/loongarch64-linux/bin/loongarch64-linux-gcc \ + TARGET_OBJCOPY=/opt/gcc-13.2.0-nolibc/loongarch64-linux/bin/loongarch64-linux-objcopy \ + TARGET_STRIP=/opt/gcc-13.2.0-nolibc/loongarch64-linux/bin/loongarch64-linux-strip \ + TARGET_NM=/opt/gcc-13.2.0-nolibc/loongarch64-linux/bin/loongarch64-linux-nm \ + TARGET_RANLIB=/opt/gcc-13.2.0-nolibc/loongarch64-linux/bin/loongarch64-linux-ranlib && \ + make && \ + ./grub-mkimage -O loongarch64-efi -o /opt/grub/grubloongarch64.efi --prefix= -d \ + grub-core cat chain configfile echo efinet ext2 fat halt help linux \ + lsefisystab loadenv lvm minicmd normal part_msdos part_gpt reboot \ + search search_fs_file search_fs_uuid search_label serial sleep test \ + true && \ + make clean && \ ./configure --target=riscv64 --with-platform=efi \ CC=gcc \ TARGET_CC=/opt/gcc-13.2.0-nolibc/riscv64-linux/bin/riscv64-linux-gcc \
@@ -183,7 +198,9 @@ RUN git clone git://git.savannah.gnu.org/grub.git /tmp/grub && \ RUN git clone https://gitlab.com/qemu-project/qemu.git /tmp/qemu && \ cd /tmp/qemu && \ git checkout v9.0.0 && \ - ./configure --prefix=/opt/qemu --target-list="aarch64-softmmu,arm-softmmu,i386-softmmu,m68k-softmmu,mips-softmmu,mips64-softmmu,mips64el-softmmu,mipsel-softmmu,ppc-softmmu,riscv32-softmmu,riscv64-softmmu,sh4-softmmu,x86_64-softmmu,xtensa-softmmu" && \ + git cherry-pick 16b1ecee52effa3346fb34dcc351e4645e4ab53e && \ + git cherry-pick 085446905000d6b80978815594a7cd34d54ff46b && \ + ./configure --prefix=/opt/qemu --target-list="aarch64-softmmu,arm-softmmu,i386-softmmu,loongarch64-softmmu,m68k-softmmu,mips-softmmu,mips64-softmmu,mips64el-softmmu,mipsel-softmmu,ppc-softmmu,riscv32-softmmu,riscv64-softmmu,sh4-softmmu,x86_64-softmmu,xtensa-softmmu" && \ make -j$(nproc) all install && \ rm -rf /tmp/qemu
Signature · 1 line
2.43.0
PATCH 18/20 doc: ci: Document how to run pipeline on gitlab.com
In reply to Jiaxun YangIt's now possible to run CI job on gitlab.com.
Document the process.
--- doc/develop/ci_testing.rst | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
doc/develop/ci_testing.rst +9 -3
@@ -46,9 +46,10 @@ resources the project has available. For Custodians, it is a matter of enabling the pipeline feature in your project repository following the standard GitLab documentation. For non-custodians, the pipeline itself is part of the tree and should be able to be used on any GitLab instance, with whatever -runners you are able to provide. While it is intended to be able to run this -pipeline on the free public instances provided at https://gitlab.com/ a problem -with our squashfs tests currently prevents this. +runners you are able to provide. To run this pipeline on the free public +instances provided at https://gitlab.com/ you will need to fork the repository, +enable the CI/CD feature [1]_ for the repository, rise pipeline timeout [2]_ to +at least 2 hours and then push your changes to the repository. To push to Gitlab without triggering a pipeline use:
@@ -74,3 +75,8 @@ developing features. In that case, it can be useful as part of your own testing cycle to edit these pipelines in separate local commits to pair them down to just the jobs you're interested in. These changes must be removed prior to submission. + +References +---------- +.. [1] https://docs.gitlab.com/ee/ci/quick_start/ +.. [2] https://docs.gitlab.com/ee/ci/pipelines/settings.html#set-a-limit-for-how-long-jobs-can-run
Signature · 1 line
2.43.0
PATCH 19/20 Use Jiaxun's CI Image
In reply to Jiaxun YangUse Jiaxun's CI Image for demonstration.
NOT FOR COMMIT!
--- .azure-pipelines.yml | 4 ++-- .gitlab-ci.yml | 4 ++-- tools/docker/Dockerfile | 6 +++--- 3 files changed, 7 insertions(+), 7 deletions(-)
.azure-pipelines.yml +2 -2
@@ -2,7 +2,7 @@ variables: windows_vm: windows-2019 ubuntu_vm: ubuntu-22.04 macos_vm: macOS-12 - ci_runner_image: trini/u-boot-gitlab-ci-runner:jammy-20240227-14Mar2024 + ci_runner_image: ghcr.io/flygoat/u-boot-ci-docker:main # Add '-u 0' options for Azure pipelines, otherwise we get "permission # denied" error when it tries to "useradd -m -u 1001 vsts_azpcontainer", # since our $(ci_runner_image) user is not root.
@@ -203,7 +203,7 @@ stages: # the below corresponds to .gitlab-ci.yml "before_script" cd \${WORK_DIR} git config --global --add safe.directory \${WORK_DIR} - git clone --depth=1 https://source.denx.de/u-boot/u-boot-test-hooks /tmp/uboot-test-hooks + git clone --depth=1 https://github.com/FlyGoat/u-boot-test-hooks.git /tmp/uboot-test-hooks ln -s travis-ci /tmp/uboot-test-hooks/bin/\`hostname\` ln -s travis-ci /tmp/uboot-test-hooks/py/\`hostname\` grub-mkimage --prefix=\"\" -o ~/grub_x86.efi -O i386-efi normal echo lsefimmap lsefi lsefisystab efinet tftp minicmd
.gitlab-ci.yml +2 -2
@@ -10,7 +10,7 @@ default: # Grab our configured image. The source for this is found # in the u-boot tree at tools/docker/Dockerfile -image: ${MIRROR_DOCKER}/trini/u-boot-gitlab-ci-runner:jammy-20240227-14Mar2024 +image: ghcr.io/flygoat/u-boot-ci-docker:main # We run some tests in different order, to catch some failures quicker. stages:
@@ -26,7 +26,7 @@ stages: before_script: # Clone uboot-test-hooks - git config --global --add safe.directory "${CI_PROJECT_DIR}" - - git clone --depth=1 https://source.denx.de/u-boot/u-boot-test-hooks /tmp/uboot-test-hooks + - git clone --depth=1 https://github.com/FlyGoat/u-boot-test-hooks.git /tmp/uboot-test-hooks - ln -s travis-ci /tmp/uboot-test-hooks/bin/`hostname` - ln -s travis-ci /tmp/uboot-test-hooks/py/`hostname` - grub-mkimage --prefix="" -o ~/grub_x86.efi -O i386-efi normal echo lsefimmap lsefi lsefisystab efinet tftp minicmd
tools/docker/Dockerfile +3 -3
@@ -276,9 +276,9 @@ USER uboot:uboot # Populate the cache for pip to use. Get these via wget as the # COPY / ADD directives don't work as we need them to. -RUN wget -O /tmp/pytest-requirements.txt https://source.denx.de/u-boot/u-boot/-/raw/master/test/py/requirements.txt -RUN wget -O /tmp/sphinx-requirements.txt https://source.denx.de/u-boot/u-boot/-/raw/master/doc/sphinx/requirements.txt -RUN wget -O /tmp/buildman-requirements.txt https://source.denx.de/u-boot/u-boot/-/raw/master/tools/buildman/requirements.txt +RUN wget -O /tmp/pytest-requirements.txt https://gitlab.com/FlyGoat/u-boot/-/raw/b4/docker-image/test/py/requirements.txt +RUN wget -O /tmp/sphinx-requirements.txt https://gitlab.com/FlyGoat/u-boot/-/raw/b4/docker-image/doc/sphinx/requirements.txt +RUN wget -O /tmp/buildman-requirements.txt https://gitlab.com/FlyGoat/u-boot/-/raw/b4/docker-image/tools/buildman/requirements.txt RUN virtualenv -p /usr/bin/python3 /tmp/venv && \ . /tmp/venv/bin/activate && \ pip install -r /tmp/pytest-requirements.txt \
Signature · 1 line
2.43.0
PATCH 20/20 CI: Dockerfile: Replace some URL with mirror sites
In reply to Jiaxun YangSomehow when I'm trying to build dockerimage on 11 June 2024 multiple upstream sites are down.
Arm site is returning 500 error, nasm.us domain expired, ftpmirror.gnu.org is down worldwide.
I belive those problems are not permanent so made this change NFC.
NOT FOR COMMIT!
--- tools/docker/Dockerfile | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
tools/docker/Dockerfile +3 -1
@@ -205,7 +205,7 @@ RUN git clone https://gitlab.com/qemu-project/qemu.git /tmp/qemu && \ rm -rf /tmp/qemu # Build fiptool -RUN git clone https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git /tmp/tf-a && \ +RUN git clone https://github.com/ARM-software/arm-trusted-firmware.git /tmp/tf-a && \ cd /tmp/tf-a/ && \ git checkout lts-v2.10.4 && \ cd tools/fiptool && \
@@ -260,6 +260,8 @@ RUN mkdir /tmp/trace && \ # Build coreboot RUN wget -O - https://coreboot.org/releases/coreboot-24.05.tar.xz | tar -C /tmp -xJ && \ cd /tmp/coreboot-24.05 && \ + sed -i 's,https://ftpmirror.gnu.org,https://ftp.gnu.org/gnu,g' ./util/crossgcc/buildgcc && \ + sed -i 's,NASM_BASE_URL=.*,NASM_BASE_URL="https://distfiles.macports.org/nasm",g' ./util/crossgcc/buildgcc && \ make crossgcc-i386 CPUS=$(nproc) && \ make -C payloads/coreinfo olddefconfig && \ make -C payloads/coreinfo && \
Signature · 1 line
2.43.0
Quoted 6 lines · Jiaxun Yang
Hi all,
This series build a new CI image based on Ubuntu focal with LoongArch64 support, fixed various python scripts for python 3.12, fixed various problems popped up when testing againt latest software.
Quoted 6 lines
This change must be combined with test hook changes at [1].
Last two commits are for demonstration purpose and not for commit into repo.
CI runs passed at azure [2] and public gitlab.com runner [3].
Thanks for doing all of the python updates.
Signature · 1 line
Tom