Skip to contentSkip to conversation

support/scripts: add check-dotconfig.py

PATCH Superseded Romain Naour 3 participants 6 messages Apr 20, 2020

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 support/scripts: add check-dotconfig.py

For the same reason as for 50b747f212be2c9c0f7cf10c674ed488d042715c, we need to check if the generated configuration file (.config) contains all symbols present in the defconfig file.

If not there is an issue with the defconfig.

This script will be used in .gitlab-ci.yml.

Inspired by is_toolchain_usable() function from genrandconfig:
https://git.busybox.net/buildroot/tree/utils/genrandconfig?h=2020.02#n164

Cc Thomas Petazzoni <[email protected]>
---
 support/scripts/check-dotconfig.py | 37 ++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)
 create mode 100755 support/scripts/check-dotconfig.py
support/scripts/check-dotconfig.py new file +37 -0
new file mode 100755
index 0000000000..9e60810c1d
@@ -0,0 +1,37 @@
+#!/usr/bin/env python3
+
+# This script check if the .config is contains all lines present in the defconfig.
+
+import sys
+
+
+def main():
+    if not (len(sys.argv) == 3):
+        print("Error: incorrect number of arguments")
+        print("""Usage: check-dotconfig <configfile> <defconfig>""")
+        sys.exit(1)
+
+    configfile = sys.argv[1]
+    defconfig = sys.argv[2]
+
+    with open(configfile) as configf:
+        configlines = configf.readlines()
+
+    defconfiglines = []
+    with open(defconfig) as defconfigf:
+        for line in defconfigf.readlines():
+            if line.startswith("BR2_"):
+                defconfiglines.append(line)
+
+    # Check that all the defconfig lines are still present
+    for defconfigline in defconfiglines:
+        if defconfigline not in configlines:
+            print("WARN: defconfig can't be used\n")
+            print("      Missing: %s\n" % defconfigline.strip())
+            return False
+
+    return True
+
+
+if __name__ == "__main__":
+    main()
Signature · 1 line
2.25.3
#1 lore raw

PATCH 2/2 gitlab-ci: check generated config files

Use the script added by the previous patch to check generated config files.

Tested on gitlab:
https://gitlab.com/kubu93/buildroot/pipelines/137597966

Cc Thomas Petazzoni <[email protected]>

---
No particular issues was found on existing defconfigs.

---
 .gitlab-ci.yml    | 1 +
 .gitlab-ci.yml.in | 1 +
 2 files changed, 2 insertions(+)
.gitlab-ci.yml +1 -0
index fa8e077a07..0d06a1b7cf 100644
@@ -47,6 +47,7 @@ check-package:
     script:
         - echo "Configure Buildroot for ${DEFCONFIG_NAME}"
         - make ${DEFCONFIG_NAME}
+        - ./support/scripts/check-dotconfig.py .config ./configs/${DEFCONFIG_NAME}
         - echo 'Build buildroot'
         - |
             make > >(tee build.log |grep '>>>') 2>&1 || {
.gitlab-ci.yml.in +1 -0
index 6b09730a65..413c6d2956 100644
@@ -47,6 +47,7 @@ check-package:
     script:
         - echo "Configure Buildroot for ${DEFCONFIG_NAME}"
         - make ${DEFCONFIG_NAME}
+        - ./support/scripts/check-dotconfig.py .config ./configs/${DEFCONFIG_NAME}
         - echo 'Build buildroot'
         - |
             make > >(tee build.log |grep '>>>') 2>&1 || {
Signature · 1 line
2.25.3
#2 lore raw

PATCH 1/2 support/scripts: add check-dotconfig.py

In reply to Romain Naour

Hello,

Quoted 3 lines · On Mon, 20 Apr 2020 11:17:56 +0200 Romain Naour <[email protected]> wrote:

For the same reason as for 50b747f212be2c9c0f7cf10c674ed488d042715c, we need to check if the generated configuration file (.config) contains all symbols present in the defconfig file.

Then can we also make the runtime-test infrastructure use this check-dotconfig.py script ?

No need to do it right now, in this patch or patch series, but just something that would be good to have.

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#3 lore raw

PATCH 1/2 support/scripts: add check-dotconfig.py

Hello Thomas,

Le 20/04/2020 ? 13:52, Thomas Petazzoni a ?crit?:

Quoted 11 lines

Hello,

Quoted 3 lines · On Mon, 20 Apr 2020 11:17:56 +0200 Romain Naour <[email protected]> wrote:

For the same reason as for 50b747f212be2c9c0f7cf10c674ed488d042715c, we need to check if the generated configuration file (.config) contains all symbols present in the defconfig file.

Then can we also make the runtime-test infrastructure use this check-dotconfig.py script ?

Not with the current implementation of the runtime-test infrastructure since the defconfig is bundled in the test python script.

The defconfig is written to .config but then we run make olddefconfig to generate the real .config. So we can't compare the .config file with check-dotconfig.py without creating a defconfig file.

Quoted 3 lines

No need to do it right now, in this patch or patch series, but just something that would be good to have.

This can be easily be done in a followup patch.

Best regards,
Romain

Quoted 5 lines

Thanks!

Thomas

#4 lore raw

PATCH 1/2 support/scripts: add check-dotconfig.py

Quoted 6 lines · On Mon, 20 Apr 2020 14:19:55 +0200 Romain Naour <[email protected]> wrote:

Not with the current implementation of the runtime-test infrastructure since the defconfig is bundled in the test python script.

The defconfig is written to .config but then we run make olddefconfig to generate the real .config. So we can't compare the .config file with check-dotconfig.py without creating a defconfig file.

Well, we can always have the defconfig written to a different file, then copied to .config, that's not a big change to introduce.

Quoted 1 line

This can be easily be done in a followup patch.

Excellent, thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#5 lore raw

PATCH 1/2 support/scripts: add check-dotconfig.py

In reply to Romain Naour

Romain, All,

On 2020-04-20 11:17 +0200, Romain Naour spake thusly:

Quoted 38 lines

For the same reason as for 50b747f212be2c9c0f7cf10c674ed488d042715c, we need to check if the generated configuration file (.config) contains all symbols present in the defconfig file.

If not there is an issue with the defconfig.

This script will be used in .gitlab-ci.yml.

Inspired by is_toolchain_usable() function from genrandconfig:
https://git.busybox.net/buildroot/tree/utils/genrandconfig?h=2020.02#n164

Cc Thomas Petazzoni <[email protected]>
---
 support/scripts/check-dotconfig.py | 37 ++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)
 create mode 100755 support/scripts/check-dotconfig.py
support/scripts/check-dotconfig.py new file +14 -0
new file mode 100755
index 0000000000..9e60810c1d
@@ -0,0 +1,37 @@
+#!/usr/bin/env python3
+
+# This script check if the .config is contains all lines present in the defconfig.
+
+import sys
+
+
+def main():
+    if not (len(sys.argv) == 3):
+        print("Error: incorrect number of arguments")
+        print("""Usage: check-dotconfig <configfile> <defconfig>""")
+        sys.exit(1)
+
+    configfile = sys.argv[1]

support/scripts/check-dotconfig.py line 24

Quoted 10 lines
support/scripts/check-dotconfig.py new file +10 -0
+    defconfig = sys.argv[2]
+
+    with open(configfile) as configf:
+        configlines = configf.readlines()
+
+    defconfiglines = []
+    with open(defconfig) as defconfigf:
+        for line in defconfigf.readlines():
+            if line.startswith("BR2_"):
+                defconfiglines.append(line)

And what about lines that match '# foo is not set' ? We also want to ensure those are still present too.

support/scripts/check-dotconfig.py line 31

Quoted 6 lines
support/scripts/check-dotconfig.py new file +6 -0
+    # Check that all the defconfig lines are still present
+    for defconfigline in defconfiglines:
+        if defconfigline not in configlines:
+            print("WARN: defconfig can't be used\n")
+            print("      Missing: %s\n" % defconfigline.strip())
+            return False

This only reports a single missing variable, while there can more than one.

Here is a proposal for an updated script:

#!/usr/bin/env python3

# This scripts check that all lines present in the defconfig are # still present in the .config

import sys

def main():
if not (len(sys.argv) == 3):
print("Error: incorrect number of arguments") print("""Usage: check-dotconfig <configfile> <defconfig>""") sys.exit(1)

configfile = sys.argv[1]
defconfig = sys.argv[2]

# strip() to get rid of trailing \n
with open(configfile) as configf:
configlines = [l.strip() for l in configf.readlines()]

defconfiglines = []
with open(defconfig) as defconfigf:
for line in defconfigf.readlines():
# strip() to get rid of trailing \n
line = line.strip()
if line.startswith("BR2_"):
defconfiglines.append(line)
elif line.startswith('# BR2_') and line.endswith(' is not set'):
defconfiglines.append(line)

# Check that all the defconfig lines are still present missing = [defconfigline.strip() for defconfigline in defconfiglines
if defconfigline not in configlines]

if len(missing):
print("WARN: defconfig {} can't be used:".format(defconfig)) for m in missing:
print(" Missing: {}".format(m))
sys.exit(1 if len(missing) else 0)

if __name__ == "__main__":
main()

And with this script, I ran:

    $ for cfg in $(make list-defconfigs |sed -r -e '/ - /!d; s/  (.*) - .*/\1/'); do
        make "${cfg}" >/dev/null 2>&1
        support/scripts/check-dotconfig.py .config configs/"${cfg}"
    done
    WARN: defconfig configs/amarula_a64_relic_defconfig can't be used:
          Missing: BR2_PACKAGE_HOST_ANDROID_TOOLS_FASTBOOT=y
    WARN: defconfig configs/beaglebone_qt5_defconfig can't be used:
          Missing: BR2_PACKAGE_QT5=y
          Missing: BR2_PACKAGE_QT5BASE_EXAMPLES=y
          Missing: BR2_PACKAGE_QT5BASE_EGLFS=y
          Missing: BR2_PACKAGE_QT5BASE_DEFAULT_QPA="wayland"
          Missing: BR2_PACKAGE_QT5QUICKCONTROLS=y
          Missing: BR2_PACKAGE_QT5WAYLAND=y
          Missing: BR2_PACKAGE_QT5WAYLAND_COMPOSITOR=y
    WARN: defconfig configs/engicam_imx6qdl_icore_qt5_defconfig can't be used:
          Missing: BR2_PACKAGE_QT5=y
          Missing: BR2_PACKAGE_QT5BASE_LICENSE_APPROVED=y
          Missing: BR2_PACKAGE_QT5BASE_OPENGL_LIB=y
          Missing: BR2_PACKAGE_QT5BASE_LINUXFB=y
          Missing: BR2_PACKAGE_QT5BASE_FONTCONFIG=y
          Missing: BR2_PACKAGE_QT5BASE_GIF=y
          Missing: BR2_PACKAGE_QT5BASE_JPEG=y
          Missing: BR2_PACKAGE_GLMARK2=y
          Missing: BR2_PACKAGE_QT5CINEX=y
          Missing: BR2_PACKAGE_QT5CINEX_HD=y
    WARN: defconfig configs/freescale_imx28evk_defconfig can't be used:
          Missing: BR2_TARGET_ROOTFS_EXT4=y
    WARN: defconfig configs/imx23evk_defconfig can't be used:
          Missing: BR2_TARGET_ROOTFS_EXT4=y
    WARN: defconfig configs/imx6-sabresd_qt5_defconfig can't be used:
          Missing: BR2_PACKAGE_QT5=y
          Missing: BR2_PACKAGE_QT5BASE_LICENSE_APPROVED=y
          Missing: BR2_PACKAGE_QT5BASE_OPENGL_LIB=y
          Missing: BR2_PACKAGE_QT5BASE_LINUXFB=y
          Missing: BR2_PACKAGE_QT5BASE_FONTCONFIG=y
          Missing: BR2_PACKAGE_QT5BASE_GIF=y
          Missing: BR2_PACKAGE_QT5BASE_JPEG=y
          Missing: BR2_PACKAGE_QT5CINEX=y
          Missing: BR2_PACKAGE_QT5CINEX_HD=y
    WARN: defconfig configs/minnowboard_max-graphical_defconfig can't be used:
          Missing: BR2_PACKAGE_GLMARK2=y
          Missing: BR2_PACKAGE_MESA3D_DEMOS=y
    WARN: defconfig configs/nanopi_r1_defconfig can't be used:
          Missing: BR2_TARGET_UBOOT_BOARD_DEFCONFIG="nanopi_r1"
    WARN: defconfig configs/olimex_a20_olinuxino_lime2_defconfig can't be used:
          Missing: BR2_PACKAGE_SUNXI_MALI_MAINLINE=y
          Missing: BR2_PACKAGE_SUNXI_MALI_MAINLINE_DRIVER=y
    WARN: defconfig configs/olimex_a20_olinuxino_lime_defconfig can't be used:
          Missing: BR2_PACKAGE_SUNXI_MALI_MAINLINE=y
          Missing: BR2_PACKAGE_SUNXI_MALI_MAINLINE_DRIVER=y
    WARN: defconfig configs/olimex_imx233_olinuxino_defconfig can't be used:
          Missing: BR2_TARGET_ROOTFS_EXT4=y
    WARN: defconfig configs/qemu_ppc_virtex_ml507_defconfig can't be used:
          Missing: BR2_SOFT_FLOAT=y
    WARN: defconfig configs/qemu_riscv32_virt_defconfig can't be used:
          Missing: BR2_TARGET_OPENSBI_USE_PLAT=y
    WARN: defconfig configs/qemu_riscv64_virt_defconfig can't be used:
          Missing: BR2_TARGET_OPENSBI_USE_PLAT=y
    WARN: defconfig configs/raspberrypi3_qt5we_defconfig can't be used:
          Missing: BR2_PACKAGE_QT5=y
          Missing: BR2_PACKAGE_QT5BASE_EXAMPLES=y
          Missing: BR2_PACKAGE_QT5BASE_GIF=y
          Missing: BR2_PACKAGE_QT5BASE_JPEG=y
          Missing: BR2_PACKAGE_QT5BASE_PNG=y
          Missing: BR2_PACKAGE_QT5WEBENGINE=y
          Missing: BR2_PACKAGE_QT5WEBENGINE_PROPRIETARY_CODECS=y
Quoted 12 lines
support/scripts/check-dotconfig.py new file +5 -0
+    return True
+
+
+if __name__ == "__main__":
+    main()
Signature · 6 lines
2.25.3

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'