Misc utils/genrandconfig improvements
v2 cover Misc utils/genrandconfig improvements
Hello,
My starting point for this series was that I wanted to create some configurations that have BR2_BACKUP_SITE="", so that we do not use the Buildroot backup mirror, to ensure that upstream locations are still around.
While implementing this, I realized that all the semi-random configuration tweaks that we do in gen_config() are in fact completely overridden by "make randconfig", when genrandconfig is used without any toolchain CSV.
So I thought it was time for a bit of clean-up:
- Drop some fixups related to toolchains that are no longer tested (patch 1)
- Completely remove the support for using a toolchain CSV file. "make randconfig" support has been around for quite a while, it gives good results, so let's use that as the default now and only use fully random configurations (patch 2)
- Adapt the fine-tuning logic to the fact that "make randconfig" overwrites everything: move the fine-tuning logic *after* "make randconfig", but keep only what makes sense, since "make randconfig" already randomizes everything (patch 3)
- Implement what I wanted: set BR2_BACKUP_SITE="" for some builds (patch 4)
- Rework the logging of genrandconfig, as it is quite noisy right now, but only with irrelevant messages (patch 5)
Changes since v1:
- Fix PATCH 3/5, which was breaking the randomness. Indeed the tuning done *after* randconfig was done by overwriting the .config entirely, instead of appending to it (open mode was "w+", while it should have been "a")
Let me know what you think.
Thomas
Thomas Petazzoni (5):
utils/genrandconfig: remove fixups related to untested CT-NG
toolchains
utils/genrandconfig: remove support for toolchain CSV utils/genrandconfig: rework fine-tuning logic utils/genrandconfig: do not use BR2_BACKUP_SITE for some builds utils/genrandconfig: improve logging
utils/genrandconfig | 268 ++++++++------------------------------------ 1 file changed, 44 insertions(+), 224 deletions(-)
Signature · 6 lines
2.46.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH v2 1/5 utils/genrandconfig: remove fixups related to untested CT-NG toolchains
We have accumulated a whole bunch of very old fixups to avoid issues with super old CT-NG toolchains, which we are not testing anymore, so remove those fixups.
--- utils/genrandconfig | 69 --------------------------------------------- 1 file changed, 69 deletions(-)
utils/genrandconfig +0 -69
@@ -181,43 +181,8 @@ async def fixup_config(sysinfo, configfile): ROOTFS_SIZE = '5G' - BR2_TOOLCHAIN_EXTERNAL_URL = 'BR2_TOOLCHAIN_EXTERNAL_URL="http://autobuild.buildroot.org/toolchains/tarballs/' - if "BR2_NEEDS_HOST_JAVA=y\n" in configlines and not await sysinfo.has("java"): return False - # The ctng toolchain is affected by PR58854 - if 'BR2_PACKAGE_LTTNG_TOOLS=y\n' in configlines and \ - BR2_TOOLCHAIN_EXTERNAL_URL + 'armv5-ctng-linux-gnueabi.tar.xz"\n' in configlines: - return False - # The ctng toolchain tigger an assembler error with guile package when compiled with -Os (same issue as for CS ARM 2014.05-29) - if 'BR2_PACKAGE_GUILE=y\n' in configlines and \ - 'BR2_OPTIMIZE_S=y\n' in configlines and \ - BR2_TOOLCHAIN_EXTERNAL_URL + 'armv5-ctng-linux-gnueabi.tar.xz"\n' in configlines: - return False - # The ctng toolchain is affected by PR58854 - if 'BR2_PACKAGE_LTTNG_TOOLS=y\n' in configlines and \ - BR2_TOOLCHAIN_EXTERNAL_URL + 'armv6-ctng-linux-uclibcgnueabi.tar.xz"\n' in configlines: - return False - # The ctng toolchain is affected by PR58854 - if 'BR2_PACKAGE_LTTNG_TOOLS=y\n' in configlines and \ - BR2_TOOLCHAIN_EXTERNAL_URL + 'armv7-ctng-linux-gnueabihf.tar.xz"\n' in configlines: - return False - # The ctng toolchain is affected by PR60155 - if 'BR2_PACKAGE_SDL=y\n' in configlines and \ - BR2_TOOLCHAIN_EXTERNAL_URL + 'powerpc-ctng-linux-uclibc.tar.xz"\n' in configlines: - return False - # The ctng toolchain is affected by PR60155 - if 'BR2_PACKAGE_LIBMPEG2=y\n' in configlines and \ - BR2_TOOLCHAIN_EXTERNAL_URL + 'powerpc-ctng-linux-uclibc.tar.xz"\n' in configlines: - return False - # This MIPS toolchain uses eglibc-2.18 which lacks SYS_getdents64 - if 'BR2_PACKAGE_STRONGSWAN=y\n' in configlines and \ - BR2_TOOLCHAIN_EXTERNAL_URL + 'mips64el-ctng_n64-linux-gnu.tar.xz"\n' in configlines: - return False - # This MIPS toolchain uses eglibc-2.18 which lacks SYS_getdents64 - if 'BR2_PACKAGE_PYTHON3=y\n' in configlines and \ - BR2_TOOLCHAIN_EXTERNAL_URL + 'mips64el-ctng_n64-linux-gnu.tar.xz"\n' in configlines: - return False # libffi not available on ARMv7-M, but propagating libffi arch # dependencies in Buildroot is really too much work, so we handle # this here.
@@ -232,40 +197,6 @@ async def fixup_config(sysinfo, configfile): if 'BR2_PACKAGE_SUNXI_BOARDS=y\n' in configlines: configlines.remove('BR2_PACKAGE_SUNXI_BOARDS_FEX_FILE=""\n') configlines.append('BR2_PACKAGE_SUNXI_BOARDS_FEX_FILE="a10/hackberry.fex"\n') - # This MIPS uClibc toolchain fails to build the gdb package - if 'BR2_PACKAGE_GDB=y\n' in configlines and \ - BR2_TOOLCHAIN_EXTERNAL_URL + 'mipsel-ctng-linux-uclibc.tar.xz"\n' in configlines: - return False - # This MIPS uClibc toolchain fails to build the rt-tests package - if 'BR2_PACKAGE_RT_TESTS=y\n' in configlines and \ - BR2_TOOLCHAIN_EXTERNAL_URL + 'mipsel-ctng-linux-uclibc.tar.xz"\n' in configlines: - return False - # This MIPS uClibc toolchain fails to build the civetweb package - if 'BR2_PACKAGE_CIVETWEB=y\n' in configlines and \ - BR2_TOOLCHAIN_EXTERNAL_URL + 'mipsel-ctng-linux-uclibc.tar.xz"\n' in configlines: - return False - # This MIPS ctng toolchain fails to build the python3 package - if 'BR2_PACKAGE_PYTHON3=y\n' in configlines and \ - BR2_TOOLCHAIN_EXTERNAL_URL + 'mips64el-ctng_n64-linux-gnu.tar.xz"\n' in configlines: - return False - # This MIPS uClibc toolchain fails to build the strace package - if 'BR2_PACKAGE_STRACE=y\n' in configlines and \ - BR2_TOOLCHAIN_EXTERNAL_URL + 'mipsel-ctng-linux-uclibc.tar.xz"\n' in configlines: - return False - # This MIPS uClibc toolchain fails to build the cdrkit package - if 'BR2_PACKAGE_CDRKIT=y\n' in configlines and \ - 'BR2_STATIC_LIBS=y\n' in configlines and \ - BR2_TOOLCHAIN_EXTERNAL_URL + 'mipsel-ctng-linux-uclibc.tar.xz"\n' in configlines: - return False - # uClibc vfork static linking issue - if 'BR2_PACKAGE_ALSA_LIB=y\n' in configlines and \ - 'BR2_STATIC_LIBS=y\n' in configlines and \ - BR2_TOOLCHAIN_EXTERNAL_URL + 'i486-ctng-linux-uclibc.tar.xz"\n' in configlines: - return False - # This MIPS uClibc toolchain fails to build the weston package - if 'BR2_PACKAGE_WESTON=y\n' in configlines and \ - BR2_TOOLCHAIN_EXTERNAL_URL + 'mipsel-ctng-linux-uclibc.tar.xz"\n' in configlines: - return False # No C library for internal toolchain if 'BR2_TOOLCHAIN_BUILDROOT_NONE=y\n' in configlines: return False
Signature · 6 lines
2.46.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH v2 2/5 utils/genrandconfig: remove support for toolchain CSV
In reply to Thomas Petazzoni via buildrootNow that the support for generating a fully random configuration has been well-tested, the whole mechanism based on a toolchain CSV isn't really useful anymore, so let's drop it to simplify the logic.
--- utils/genrandconfig | 99 +-------------------------------------------- 1 file changed, 2 insertions(+), 97 deletions(-)
utils/genrandconfig +2 -97
@@ -20,7 +20,6 @@ from binascii import hexlify import asyncio -import csv import os from random import randint import sys
@@ -100,73 +99,6 @@ class SystemInfo: return not missing_requirements -def get_toolchain_configs(toolchains_csv, buildrootdir): - """Fetch and return the possible toolchain configurations - - This function returns an array of toolchain configurations. Each - toolchain configuration is itself an array of lines of the defconfig. - """ - - with open(toolchains_csv) as r: - # filter empty lines and comments - lines = [t for t in r.readlines() if len(t.strip()) > 0 and t[0] != '#'] - toolchains = lines - configs = [] - - (_, _, _, _, hostarch) = os.uname() - # ~2015 distros report x86 when on a 32bit install - if hostarch == 'i686' or hostarch == 'i386' or hostarch == 'x86': - hostarch = 'x86' - - for row in csv.reader(toolchains): - config = {} - configfile = row[0] - config_hostarch = row[1] - keep = False - - # Keep all toolchain configs that work regardless of the host - # architecture - if config_hostarch == "any": - keep = True - - # Keep all toolchain configs that can work on the current host - # architecture - if hostarch == config_hostarch: - keep = True - - # Assume that x86 32 bits toolchains work on x86_64 build - # machines - if hostarch == 'x86_64' and config_hostarch == "x86": - keep = True - - if not keep: - continue - - if not os.path.isabs(configfile): - configfile = os.path.join(buildrootdir, configfile) - - with open(configfile) as r: - config = r.readlines() - configs.append(config) - return configs - - -async def is_toolchain_usable(configfile, config): - """Check if the toolchain is actually usable.""" - - with open(configfile) as configf: - configlines = configf.readlines() - - # Check that the toolchain configuration is still present - for toolchainline in config: - if toolchainline not in configlines: - print("WARN: toolchain can't be used", file=sys.stderr) - print(" Missing: %s" % toolchainline.strip(), file=sys.stderr) - return False - - return True - - async def fixup_config(sysinfo, configfile): """Finalize the configuration and reject any problematic combinations
@@ -581,24 +513,11 @@ async def fixup_config(sysinfo, configfile): async def gen_config(args): """Generate a new random configuration - - This function generates the configuration, by choosing a random - toolchain configuration and then generating a random selection of - packages. """ sysinfo = SystemInfo() - if args.toolchains_csv: - # Select a random toolchain configuration - configs = get_toolchain_configs(args.toolchains_csv, args.buildrootdir) - - i = randint(0, len(configs) - 1) - toolchainconfig = configs[i] - else: - toolchainconfig = [] - - configlines = list(toolchainconfig) + configlines = list() # Combine with the minimal configuration minimalconfigfile = os.path.join(args.buildrootdir,
@@ -658,9 +577,6 @@ async def gen_config(args): if ret: return ret - if not await is_toolchain_usable(configfile, toolchainconfig): - return 2 - # Now, generate the random selection of packages, and fixup # things if needed. # Safe-guard, in case we can not quickly come to a valid
@@ -676,7 +592,7 @@ async def gen_config(args): "make", "O=%s" % args.outputdir, "-C", args.buildrootdir, "KCONFIG_SEED=0x%s" % hexlify(os.urandom(4)).decode("ascii").upper(), "KCONFIG_PROBABILITY=%d" % randint(1, 20), - "randpackageconfig" if args.toolchains_csv else "randconfig") + "randconfig") ret = await proc.wait() if ret: return ret
@@ -711,17 +627,6 @@ if __name__ == '__main__': help="Buildroot directory (relative to current directory)", type=str, default='.') - toolchains_csv = parser.add_mutually_exclusive_group(required=False) - toolchains_csv.add_argument("--toolchains-csv", - dest="toolchains_csv", - help="Path of the toolchain configuration file", - type=str) - toolchains_csv.add_argument("--no-toolchains-csv", - dest="toolchains_csv", - help="Generate random toolchain configuration", - action='store_false') - parser.set_defaults(toolchains_csv="support/config-fragments/autobuild/toolchain-configs.csv") - args = parser.parse_args() # We need the absolute path to use with O=, because the relative
Signature · 6 lines
2.46.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Yann E. MORIN
AppliedRe: [Buildroot] [PATCH v2 2/5] utils/genrandconfig: remove support for toolchain CSV
Thomas, All,
On 2024-08-18 11:03 +0200, Thomas Petazzoni via buildroot spake thusly:
Quoted 3 lines
Now that the support for generating a fully random configuration has been well-tested, the whole mechanism based on a toolchain CSV isn't really useful anymore, so let's drop it to simplify the logic.
As discussed on IRC, the autobuilder code still uses --{,no-}toolchains-csv, so dopping those flags would instaly break the autobuilders.
So, I've kept them and docuemnted them to be legacy.
Applied to master, thanks.
Regards,
Yann E. MORIN.
Quoted 161 lines
Signed-off-by Thomas Petazzoni <[email protected]>--- utils/genrandconfig | 99 +-------------------------------------------- 1 file changed, 2 insertions(+), 97 deletions(-)utils/genrandconfig +2
-97@@ -20,7 +20,6 @@ from binascii import hexlify import asyncio -import csv import os from random import randint import sys@@ -100,73 +99,6 @@ class SystemInfo: return not missing_requirements -def get_toolchain_configs(toolchains_csv, buildrootdir): - """Fetch and return the possible toolchain configurations - - This function returns an array of toolchain configurations. Each - toolchain configuration is itself an array of lines of the defconfig. - """ - - with open(toolchains_csv) as r: - # filter empty lines and comments - lines = [t for t in r.readlines() if len(t.strip()) > 0 and t[0] != '#'] - toolchains = lines - configs = [] - - (_, _, _, _, hostarch) = os.uname() - # ~2015 distros report x86 when on a 32bit install - if hostarch == 'i686' or hostarch == 'i386' or hostarch == 'x86': - hostarch = 'x86' - - for row in csv.reader(toolchains): - config = {} - configfile = row[0] - config_hostarch = row[1] - keep = False - - # Keep all toolchain configs that work regardless of the host - # architecture - if config_hostarch == "any": - keep = True - - # Keep all toolchain configs that can work on the current host - # architecture - if hostarch == config_hostarch: - keep = True - - # Assume that x86 32 bits toolchains work on x86_64 build - # machines - if hostarch == 'x86_64' and config_hostarch == "x86": - keep = True - - if not keep: - continue - - if not os.path.isabs(configfile): - configfile = os.path.join(buildrootdir, configfile) - - with open(configfile) as r: - config = r.readlines() - configs.append(config) - return configs - - -async def is_toolchain_usable(configfile, config): - """Check if the toolchain is actually usable.""" - - with open(configfile) as configf: - configlines = configf.readlines() - - # Check that the toolchain configuration is still present - for toolchainline in config: - if toolchainline not in configlines: - print("WARN: toolchain can't be used", file=sys.stderr) - print(" Missing: %s" % toolchainline.strip(), file=sys.stderr) - return False - - return True - - async def fixup_config(sysinfo, configfile): """Finalize the configuration and reject any problematic combinations@@ -581,24 +513,11 @@ async def fixup_config(sysinfo, configfile): async def gen_config(args): """Generate a new random configuration - - This function generates the configuration, by choosing a random - toolchain configuration and then generating a random selection of - packages. """ sysinfo = SystemInfo() - if args.toolchains_csv: - # Select a random toolchain configuration - configs = get_toolchain_configs(args.toolchains_csv, args.buildrootdir) - - i = randint(0, len(configs) - 1) - toolchainconfig = configs[i] - else: - toolchainconfig = [] - - configlines = list(toolchainconfig) + configlines = list() # Combine with the minimal configuration minimalconfigfile = os.path.join(args.buildrootdir,@@ -658,9 +577,6 @@ async def gen_config(args): if ret: return ret - if not await is_toolchain_usable(configfile, toolchainconfig): - return 2 - # Now, generate the random selection of packages, and fixup # things if needed. # Safe-guard, in case we can not quickly come to a valid@@ -676,7 +592,7 @@ async def gen_config(args): "make", "O=%s" % args.outputdir, "-C", args.buildrootdir, "KCONFIG_SEED=0x%s" % hexlify(os.urandom(4)).decode("ascii").upper(), "KCONFIG_PROBABILITY=%d" % randint(1, 20), - "randpackageconfig" if args.toolchains_csv else "randconfig") + "randconfig") ret = await proc.wait() if ret: return ret@@ -711,17 +627,6 @@ if __name__ == '__main__': help="Buildroot directory (relative to current directory)", type=str, default='.') - toolchains_csv = parser.add_mutually_exclusive_group(required=False) - toolchains_csv.add_argument("--toolchains-csv", - dest="toolchains_csv", - help="Path of the toolchain configuration file", - type=str) - toolchains_csv.add_argument("--no-toolchains-csv", - dest="toolchains_csv", - help="Generate random toolchain configuration", - action='store_false') - parser.set_defaults(toolchains_csv="support/config-fragments/autobuild/toolchain-configs.csv") - args = parser.parse_args() # We need the absolute path to use with O=, because the relativeSignature · 6 lines
2.46.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH v2 3/5 utils/genrandconfig: rework fine-tuning logic
In reply to Thomas Petazzoni via buildrootBefore calling randpackageconfig/randconfig, we were pre-generating a snippet of .config with:
(1) minimal.config
(2) BR2_CURL/BR2_WGET settings
(3) some random selection of init system, debug, runtime debug, etc
(4) enabling BR2_REPRODUCIBLE=y when diffoscope was found
Now that we only use randconfig, this whole fine-tuning is completely irrelevant, as it gets overridden by "make randconfig".
(1) and (3) above are useless, as randconfig does all the randomization that is needed.
However, we want to preserve (2) and (4) above, so we re-implement those fixups, but *after* randconfig has done its job.
---
Changes since v1:
- Restore proper randomness. Indeed the tuning done *after* randconfig was done by overwriting the .config entirely, instead of appending to it (open mode was "w+", while it should have been "a")
utils/genrandconfig | 71 ++++++++++++--------------------------------- 1 file changed, 18 insertions(+), 53 deletions(-)
utils/genrandconfig +18 -53
@@ -517,65 +517,15 @@ async def gen_config(args): sysinfo = SystemInfo() - configlines = list() - - # Combine with the minimal configuration - minimalconfigfile = os.path.join(args.buildrootdir, - 'support/config-fragments/minimal.config') - with open(minimalconfigfile) as minimalf: - configlines += minimalf.readlines() - - # Allow hosts with old certificates to download over https - configlines.append("BR2_WGET=\"wget -nd -t 3 --no-check-certificate\"\n") - configlines.append("BR2_CURL=\"curl --ftp-pasv --retry 3 --insecure\"\n") - - # Per-package folder - if randint(0, 15) == 0: - configlines.append("BR2_PER_PACKAGE_DIRECTORIES=y\n") - - # Amend the configuration with a few things. - if randint(0, 20) == 0: - configlines.append("BR2_ENABLE_DEBUG=y\n") - if randint(0, 20) == 0: - configlines.append("BR2_ENABLE_RUNTIME_DEBUG=y\n") - if randint(0, 1) == 0: - configlines.append("BR2_INIT_BUSYBOX=y\n") - elif randint(0, 15) == 0: - configlines.append("BR2_INIT_SYSTEMD=y\n") - elif randint(0, 10) == 0: - configlines.append("BR2_ROOTFS_DEVICE_CREATION_DYNAMIC_EUDEV=y\n") - if randint(0, 20) == 0: - configlines.append("BR2_STATIC_LIBS=y\n") - if randint(0, 20) == 0: - configlines.append("BR2_PACKAGE_PYTHON3_PY_ONLY=y\n") - if randint(0, 5) == 0: - configlines.append("BR2_OPTIMIZE_2=y\n") - if randint(0, 4) == 0: - configlines.append("BR2_SYSTEM_ENABLE_NLS=y\n") - if randint(0, 4) == 0: - configlines.append("BR2_FORTIFY_SOURCE_2=y\n") - - # Randomly enable BR2_REPRODUCIBLE 10% of times - # also enable tar filesystem images for testing - if await sysinfo.has("diffoscope") and randint(0, 10) == 0: - configlines.append("BR2_REPRODUCIBLE=y\n") - configlines.append("BR2_TARGET_ROOTFS_TAR=y\n") - - # Write out the configuration file + # Create output directory if not os.path.exists(args.outputdir): os.makedirs(args.outputdir) + + # Calculate path to config file if args.outputdir == os.path.abspath(os.path.join(args.buildrootdir, "output")): configfile = os.path.join(args.buildrootdir, ".config") else: configfile = os.path.join(args.outputdir, ".config") - with open(configfile, "w+") as configf: - configf.writelines(configlines) - - proc = await asyncio.create_subprocess_exec( - "make", "O=%s" % args.outputdir, "-C", args.buildrootdir, "olddefconfig") - ret = await proc.wait() - if ret: - return ret # Now, generate the random selection of packages, and fixup # things if needed.
@@ -600,6 +550,21 @@ async def gen_config(args): if await fixup_config(sysinfo, configfile): break + configlines = list() + + # Allow hosts with old certificates to download over https + configlines.append("BR2_WGET=\"wget -nd -t 3 --no-check-certificate\"\n") + configlines.append("BR2_CURL=\"curl --ftp-pasv --retry 3 --insecure\"\n") + + # Randomly enable BR2_REPRODUCIBLE 10% of times + # also enable tar filesystem images for testing + if await sysinfo.has("diffoscope") and randint(0, 10) == 0: + configlines.append("BR2_REPRODUCIBLE=y\n") + configlines.append("BR2_TARGET_ROOTFS_TAR=y\n") + + with open(configfile, "a") as configf: + configf.writelines(configlines) + proc = await asyncio.create_subprocess_exec( "make", "O=%s" % args.outputdir, "-C", args.buildrootdir, "olddefconfig") ret = await proc.wait()
Signature · 6 lines
2.46.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH v2 4/5 utils/genrandconfig: do not use BR2_BACKUP_SITE for some builds
In reply to Thomas Petazzoni via buildrootIn order to test that upstream sites are still working, we need to NOT fallback to sources.buildroot.net for some builds.
As there is anyway a local cache in the autobuilder instances, we need to do quite a lot of builds without any BR2_BACKUP_SITE configured to have a chance to catch issues, which is why a 50% chance is used to unset BR2_BACKUP_SITE.
--- utils/genrandconfig | 5 +++++ 1 file changed, 5 insertions(+)
utils/genrandconfig +5 -0
@@ -562,6 +562,11 @@ async def gen_config(args): configlines.append("BR2_REPRODUCIBLE=y\n") configlines.append("BR2_TARGET_ROOTFS_TAR=y\n") + # From time to time, ignore sources.buildroot.net to really fetch + # from upstream + if randint(0, 1) == 0: + configlines.append("""BR2_BACKUP_SITE=""\n""") + with open(configfile, "a") as configf: configf.writelines(configlines)
Signature · 6 lines
2.46.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH v2 5/5 utils/genrandconfig: improve logging
In reply to Thomas Petazzoni via buildrootRight now, genrandconfig just spits out the random messages from the different make invocations, which isn't terribly useful. Instead, let's redirect the output of make invocations to oblivion, and add some more high level logging.
As part of this logging, we're interested to see how many iterations were needed to find a valid configuration, so changed the loop logic to count from 0 to 100 instead of from 100 to 0 so that we can easily show the iteration number.
--- utils/genrandconfig | 28 +++++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-)
utils/genrandconfig +21 -7
@@ -531,23 +531,27 @@ async def gen_config(args): # things if needed. # Safe-guard, in case we can not quickly come to a valid # configuration: allow at most 100 (arbitrary) iterations. - bounded_loop = 100 + loop = 0 while True: - if bounded_loop == 0: + if loop == 100: print("ERROR: cannot generate random configuration after 100 iterations", file=sys.stderr) return 1 - bounded_loop -= 1 + print("Generating configuration, loop %d" % loop) + loop += 1 proc = await asyncio.create_subprocess_exec( "make", "O=%s" % args.outputdir, "-C", args.buildrootdir, "KCONFIG_SEED=0x%s" % hexlify(os.urandom(4)).decode("ascii").upper(), "KCONFIG_PROBABILITY=%d" % randint(1, 20), - "randconfig") + "randconfig", + stdout=asyncio.subprocess.DEVNULL, + stderr=asyncio.subprocess.DEVNULL) ret = await proc.wait() if ret: return ret if await fixup_config(sysinfo, configfile): + print(" configuration valid") break configlines = list()
@@ -570,20 +574,30 @@ async def gen_config(args): with open(configfile, "a") as configf: configf.writelines(configlines) + print(" olddefconfig") proc = await asyncio.create_subprocess_exec( - "make", "O=%s" % args.outputdir, "-C", args.buildrootdir, "olddefconfig") + "make", "O=%s" % args.outputdir, "-C", args.buildrootdir, "olddefconfig", + stdout=asyncio.subprocess.DEVNULL, + stderr=asyncio.subprocess.DEVNULL) ret = await proc.wait() if ret: return ret + print(" savedefconfig") proc = await asyncio.create_subprocess_exec( - "make", "O=%s" % args.outputdir, "-C", args.buildrootdir, "savedefconfig") + "make", "O=%s" % args.outputdir, "-C", args.buildrootdir, "savedefconfig", + stdout=asyncio.subprocess.DEVNULL, + stderr=asyncio.subprocess.DEVNULL) + ret = await proc.wait() if ret: return ret + print(" dependencies") proc = await asyncio.create_subprocess_exec( - "make", "O=%s" % args.outputdir, "-C", args.buildrootdir, "dependencies") + "make", "O=%s" % args.outputdir, "-C", args.buildrootdir, "dependencies", + stdout=asyncio.subprocess.DEVNULL, + stderr=asyncio.subprocess.DEVNULL) return await proc.wait()
Signature · 6 lines
2.46.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Thomas, All,
On 2024-08-18 11:03 +0200, Thomas Petazzoni via buildroot spake thusly:
Quoted 11 lines
Hello,
My starting point for this series was that I wanted to create some configurations that have BR2_BACKUP_SITE="", so that we do not use the Buildroot backup mirror, to ensure that upstream locations are still around.
While implementing this, I realized that all the semi-random configuration tweaks that we do in
gen_config()are in fact completely overridden by "make randconfig", when genrandconfig is used without any toolchain CSV.
Series applied to master, thanks. I did some changes on patch 2, so I'll reply there as well..
Regards,
Yann E. MORIN.
Quoted 50 lines
So I thought it was time for a bit of clean-up:
- Drop some fixups related to toolchains that are no longer tested (patch 1)
- Completely remove the support for using a toolchain CSV file. "make randconfig" support has been around for quite a while, it gives good results, so let's use that as the default now and only use fully random configurations (patch 2)
- Adapt the fine-tuning logic to the fact that "make randconfig" overwrites everything: move the fine-tuning logic *after* "make randconfig", but keep only what makes sense, since "make randconfig" already randomizes everything (patch 3)
- Implement what I wanted: set BR2_BACKUP_SITE="" for some builds (patch 4)
- Rework the logging of genrandconfig, as it is quite noisy right now, but only with irrelevant messages (patch 5)
Changes since v1:
- Fix PATCH 3/5, which was breaking the randomness. Indeed the tuning done *after* randconfig was done by overwriting the .config entirely, instead of appending to it (open mode was "w+", while it should have been "a")
Let me know what you think.
Thomas
Thomas Petazzoni (5):
utils/genrandconfig: remove fixups related to untested CT-NG
toolchains
utils/genrandconfig: remove support for toolchain CSV utils/genrandconfig: rework fine-tuning logic utils/genrandconfig: do not useBR2_BACKUP_SITEfor some builds utils/genrandconfig: improve loggingutils/genrandconfig | 268 ++++++++------------------------------------ 1 file changed, 44 insertions(+), 224 deletions(-)Signature · 6 lines
2.46.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot