ffmpeg: nasm is x86 only, so only DEPEND if x86
PATCH v2 1/11 ffmpeg: nasm is x86 only, so only DEPEND if x86
No need to depend on nasm if we're not going to use it.
--- meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb +2 -1
@@ -46,7 +46,8 @@ ARM_INSTRUCTION_SET:armv6 = "arm" # libpostproc was previously packaged from a separate recipe PROVIDES = "libav libpostproc" -DEPENDS = "nasm-native" +DEPENDS:append:x86 = " nasm-native" +DEPENDS:append:x86-64 = " nasm-native" inherit autotools pkgconfig
Signature · 1 line
2.34.1
PATCH v2 2/11 ffmpeg: no need for textrel INSANE_SKIP
It appears in testing that modern ffmpeg no longer needs to disable PIC, so there's no need to ignore textrel warnings.
--- meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb | 10 ---------- 1 file changed, 10 deletions(-)
meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb +0 -10
@@ -181,13 +181,3 @@ FILES:libpostproc = "${libdir}/libpostproc${SOLIBS}" FILES:libswresample = "${libdir}/libswresample${SOLIBS}" FILES:libswscale = "${libdir}/libswscale${SOLIBS}" FILES:${PN}-examples = "${datadir}/${PN}/examples" - -# ffmpeg disables PIC on some platforms (e.g. x86-32) -INSANE_SKIP:${MLPREFIX}libavcodec = "textrel" -INSANE_SKIP:${MLPREFIX}libavdevice = "textrel" -INSANE_SKIP:${MLPREFIX}libavfilter = "textrel" -INSANE_SKIP:${MLPREFIX}libavformat = "textrel" -INSANE_SKIP:${MLPREFIX}libavutil = "textrel" -INSANE_SKIP:${MLPREFIX}libswscale = "textrel" -INSANE_SKIP:${MLPREFIX}libswresample = "textrel" -INSANE_SKIP:${MLPREFIX}libpostproc = "textrel"
Signature · 1 line
2.34.1
Re: [OE-core] [PATCH v2 02/11] ffmpeg: no need for textrel INSANE_SKIP
The textrels still occur on qemux86 when using clang+lld
https://errors.yoctoproject.org/Errors/Details/809513/
Quoted 39 lines · Ross Burton via lists.openembedded.org
It appears in testing that modern ffmpeg no longer needs to disable PIC, so there's no need to ignore textrel warnings.
Signed-off-by Ross Burton <[email protected]>--- meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb | 10 ---------- 1 file changed, 10 deletions(-)meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb +0
-11@@ -181,13 +181,3 @@ FILES:libpostproc = "${libdir}/libpostproc${SOLIBS}" FILES:libswresample = "${libdir}/libswresample${SOLIBS}" FILES:libswscale = "${libdir}/libswscale${SOLIBS}" FILES:${PN}-examples = "${datadir}/${PN}/examples" - -# ffmpeg disables PIC on some platforms (e.g. x86-32) -INSANE_SKIP:${MLPREFIX}libavcodec = "textrel" -INSANE_SKIP:${MLPREFIX}libavdevice = "textrel" -INSANE_SKIP:${MLPREFIX}libavfilter = "textrel" -INSANE_SKIP:${MLPREFIX}libavformat = "textrel" -INSANE_SKIP:${MLPREFIX}libavutil = "textrel" -INSANE_SKIP:${MLPREFIX}libswscale = "textrel" -INSANE_SKIP:${MLPREFIX}libswresample = "textrel" -INSANE_SKIP:${MLPREFIX}libpostproc = "textrel" --2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#205411): https://lists.openembedded.org/g/openembedded-core/message/205411
Mute This Topic: https://lists.openembedded.org/mt/108932124/1997914
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
PATCH v2 3/11 libjpeg-turbo: remove redundant RUNPATH entries
In reply to Ross BurtonOur RPATH sanity checks currently don't check RUNPATH (patch incoming), but the libjpeg objects are built with a RUNPATH set to ${libdir}.
RUNPATH is less of a problem than RPATH (because it is used after the standard search), but redundant entries are still redundant so lets remove them anyway.
--- meta/recipes-graphics/jpeg/libjpeg-turbo_3.0.1.bb | 5 +++++ 1 file changed, 5 insertions(+)
meta/recipes-graphics/jpeg/libjpeg-turbo_3.0.1.bb +5 -0
@@ -55,4 +55,9 @@ FILES:jpeg-tools = "${bindir}/*" DESCRIPTION:libturbojpeg = "A SIMD-accelerated JPEG codec which provides only TurboJPEG APIs" FILES:libturbojpeg = "${libdir}/libturbojpeg.so.*" +do_install:append() { + # The binaries have RUNPATH=$libdir, which is redundant + chrpath -d ${D}/${bindir}/* ${D}${libdir}/*${SOLIBS} +} + BBCLASSEXTEND = "native nativesdk"
Signature · 1 line
2.34.1
PATCH v2 4/11 oe/elf: don't regenerate machine data on every call
In reply to Ross BurtonEvery time oe.elf.machine_dict() is called a large dictionary is created and returned. However, the "arch" package_qa test will call this method for every binary in a package, which results in a lot of dictionary creation.
Concrete exmaple: in running ltp:do_package_qa, the arch test takes 25% of the runtime, and opitimising the machine_dict() call to cache the generated dictionary reduces the runtime from 57s to 44s.
--- meta/lib/oe/elf.py | 264 +++++++++++++++++++++++---------------------- 1 file changed, 133 insertions(+), 131 deletions(-)
meta/lib/oe/elf.py +133 -131
@@ -5,141 +5,143 @@ # def machine_dict(d): -# TARGET_OS TARGET_ARCH MACHINE, OSABI, ABIVERSION, Little Endian, 32bit? - machdata = { - "darwin9" : { - "arm" : (40, 0, 0, True, 32), - }, - "eabi" : { - "arm" : (40, 0, 0, True, 32), - }, - "elf" : { - "aarch64" : (183, 0, 0, True, 64), - "aarch64_be" :(183, 0, 0, False, 64), - "i586" : (3, 0, 0, True, 32), - "i686" : (3, 0, 0, True, 32), - "x86_64": (62, 0, 0, True, 64), - "epiphany": (4643, 0, 0, True, 32), - "lm32": (138, 0, 0, False, 32), - "loongarch64":(258, 0, 0, True, 64), - "mips": ( 8, 0, 0, False, 32), - "mipsel": ( 8, 0, 0, True, 32), - "microblaze": (189, 0, 0, False, 32), - "microblazeel":(189, 0, 0, True, 32), - "powerpc": (20, 0, 0, False, 32), - "riscv32": (243, 0, 0, True, 32), - "riscv64": (243, 0, 0, True, 64), - }, - "linux" : { - "aarch64" : (183, 0, 0, True, 64), - "aarch64_be" :(183, 0, 0, False, 64), - "arm" : (40, 97, 0, True, 32), - "armeb": (40, 97, 0, False, 32), - "powerpc": (20, 0, 0, False, 32), - "powerpc64": (21, 0, 0, False, 64), - "powerpc64le": (21, 0, 0, True, 64), - "i386": ( 3, 0, 0, True, 32), - "i486": ( 3, 0, 0, True, 32), - "i586": ( 3, 0, 0, True, 32), - "i686": ( 3, 0, 0, True, 32), - "x86_64": (62, 0, 0, True, 64), - "ia64": (50, 0, 0, True, 64), - "alpha": (36902, 0, 0, True, 64), - "hppa": (15, 3, 0, False, 32), - "loongarch64":(258, 0, 0, True, 64), - "m68k": ( 4, 0, 0, False, 32), - "mips": ( 8, 0, 0, False, 32), - "mipsel": ( 8, 0, 0, True, 32), - "mips64": ( 8, 0, 0, False, 64), - "mips64el": ( 8, 0, 0, True, 64), - "mipsisa32r6": ( 8, 0, 0, False, 32), - "mipsisa32r6el": ( 8, 0, 0, True, 32), - "mipsisa64r6": ( 8, 0, 0, False, 64), - "mipsisa64r6el": ( 8, 0, 0, True, 64), - "nios2": (113, 0, 0, True, 32), - "riscv32": (243, 0, 0, True, 32), - "riscv64": (243, 0, 0, True, 64), - "s390": (22, 0, 0, False, 32), - "sh4": (42, 0, 0, True, 32), - "sparc": ( 2, 0, 0, False, 32), - "microblaze": (189, 0, 0, False, 32), - "microblazeel":(189, 0, 0, True, 32), - }, - "linux-android" : { - "aarch64" : (183, 0, 0, True, 64), - "i686": ( 3, 0, 0, True, 32), - "x86_64": (62, 0, 0, True, 64), - }, - "linux-androideabi" : { - "arm" : (40, 97, 0, True, 32), - }, - "linux-musl" : { - "aarch64" : (183, 0, 0, True, 64), - "aarch64_be" :(183, 0, 0, False, 64), - "arm" : ( 40, 97, 0, True, 32), - "armeb": ( 40, 97, 0, False, 32), - "powerpc": ( 20, 0, 0, False, 32), - "powerpc64": ( 21, 0, 0, False, 64), - "powerpc64le": (21, 0, 0, True, 64), - "i386": ( 3, 0, 0, True, 32), - "i486": ( 3, 0, 0, True, 32), - "i586": ( 3, 0, 0, True, 32), - "i686": ( 3, 0, 0, True, 32), - "x86_64": ( 62, 0, 0, True, 64), - "mips": ( 8, 0, 0, False, 32), - "mipsel": ( 8, 0, 0, True, 32), - "mips64": ( 8, 0, 0, False, 64), - "mips64el": ( 8, 0, 0, True, 64), - "microblaze": (189, 0, 0, False, 32), - "microblazeel":(189, 0, 0, True, 32), - "riscv32": (243, 0, 0, True, 32), - "riscv64": (243, 0, 0, True, 64), - "sh4": ( 42, 0, 0, True, 32), - }, - "uclinux-uclibc" : { - "bfin": ( 106, 0, 0, True, 32), - }, - "linux-gnueabi" : { - "arm" : (40, 0, 0, True, 32), - "armeb" : (40, 0, 0, False, 32), - }, - "linux-musleabi" : { - "arm" : (40, 0, 0, True, 32), - "armeb" : (40, 0, 0, False, 32), - }, - "linux-gnuspe" : { - "powerpc": (20, 0, 0, False, 32), - }, - "linux-muslspe" : { - "powerpc": (20, 0, 0, False, 32), - }, - "linux-gnu" : { - "powerpc": (20, 0, 0, False, 32), - "sh4": (42, 0, 0, True, 32), - }, - "linux-gnu_ilp32" : { - "aarch64" : (183, 0, 0, True, 32), - }, - "linux-gnux32" : { - "x86_64": (62, 0, 0, True, 32), - }, - "linux-muslx32" : { - "x86_64": (62, 0, 0, True, 32), - }, - "linux-gnun32" : { - "mips64": ( 8, 0, 0, False, 32), - "mips64el": ( 8, 0, 0, True, 32), - "mipsisa64r6": ( 8, 0, 0, False, 32), - "mipsisa64r6el":( 8, 0, 0, True, 32), - }, - } + # Generating this data is slow, so cache it + if not hasattr(machine_dict, "machdata"): + machine_dict.machdata = { + # TARGET_OS TARGET_ARCH MACHINE, OSABI, ABIVERSION, Little Endian, 32bit? + "darwin9" : { + "arm" : (40, 0, 0, True, 32), + }, + "eabi" : { + "arm" : (40, 0, 0, True, 32), + }, + "elf" : { + "aarch64" : (183, 0, 0, True, 64), + "aarch64_be" :(183, 0, 0, False, 64), + "i586" : (3, 0, 0, True, 32), + "i686" : (3, 0, 0, True, 32), + "x86_64": (62, 0, 0, True, 64), + "epiphany": (4643, 0, 0, True, 32), + "lm32": (138, 0, 0, False, 32), + "loongarch64":(258, 0, 0, True, 64), + "mips": ( 8, 0, 0, False, 32), + "mipsel": ( 8, 0, 0, True, 32), + "microblaze": (189, 0, 0, False, 32), + "microblazeel":(189, 0, 0, True, 32), + "powerpc": (20, 0, 0, False, 32), + "riscv32": (243, 0, 0, True, 32), + "riscv64": (243, 0, 0, True, 64), + }, + "linux" : { + "aarch64" : (183, 0, 0, True, 64), + "aarch64_be" :(183, 0, 0, False, 64), + "arm" : (40, 97, 0, True, 32), + "armeb": (40, 97, 0, False, 32), + "powerpc": (20, 0, 0, False, 32), + "powerpc64": (21, 0, 0, False, 64), + "powerpc64le": (21, 0, 0, True, 64), + "i386": ( 3, 0, 0, True, 32), + "i486": ( 3, 0, 0, True, 32), + "i586": ( 3, 0, 0, True, 32), + "i686": ( 3, 0, 0, True, 32), + "x86_64": (62, 0, 0, True, 64), + "ia64": (50, 0, 0, True, 64), + "alpha": (36902, 0, 0, True, 64), + "hppa": (15, 3, 0, False, 32), + "loongarch64":(258, 0, 0, True, 64), + "m68k": ( 4, 0, 0, False, 32), + "mips": ( 8, 0, 0, False, 32), + "mipsel": ( 8, 0, 0, True, 32), + "mips64": ( 8, 0, 0, False, 64), + "mips64el": ( 8, 0, 0, True, 64), + "mipsisa32r6": ( 8, 0, 0, False, 32), + "mipsisa32r6el": ( 8, 0, 0, True, 32), + "mipsisa64r6": ( 8, 0, 0, False, 64), + "mipsisa64r6el": ( 8, 0, 0, True, 64), + "nios2": (113, 0, 0, True, 32), + "riscv32": (243, 0, 0, True, 32), + "riscv64": (243, 0, 0, True, 64), + "s390": (22, 0, 0, False, 32), + "sh4": (42, 0, 0, True, 32), + "sparc": ( 2, 0, 0, False, 32), + "microblaze": (189, 0, 0, False, 32), + "microblazeel":(189, 0, 0, True, 32), + }, + "linux-android" : { + "aarch64" : (183, 0, 0, True, 64), + "i686": ( 3, 0, 0, True, 32), + "x86_64": (62, 0, 0, True, 64), + }, + "linux-androideabi" : { + "arm" : (40, 97, 0, True, 32), + }, + "linux-musl" : { + "aarch64" : (183, 0, 0, True, 64), + "aarch64_be" :(183, 0, 0, False, 64), + "arm" : ( 40, 97, 0, True, 32), + "armeb": ( 40, 97, 0, False, 32), + "powerpc": ( 20, 0, 0, False, 32), + "powerpc64": ( 21, 0, 0, False, 64), + "powerpc64le": (21, 0, 0, True, 64), + "i386": ( 3, 0, 0, True, 32), + "i486": ( 3, 0, 0, True, 32), + "i586": ( 3, 0, 0, True, 32), + "i686": ( 3, 0, 0, True, 32), + "x86_64": ( 62, 0, 0, True, 64), + "mips": ( 8, 0, 0, False, 32), + "mipsel": ( 8, 0, 0, True, 32), + "mips64": ( 8, 0, 0, False, 64), + "mips64el": ( 8, 0, 0, True, 64), + "microblaze": (189, 0, 0, False, 32), + "microblazeel":(189, 0, 0, True, 32), + "riscv32": (243, 0, 0, True, 32), + "riscv64": (243, 0, 0, True, 64), + "sh4": ( 42, 0, 0, True, 32), + }, + "uclinux-uclibc" : { + "bfin": ( 106, 0, 0, True, 32), + }, + "linux-gnueabi" : { + "arm" : (40, 0, 0, True, 32), + "armeb" : (40, 0, 0, False, 32), + }, + "linux-musleabi" : { + "arm" : (40, 0, 0, True, 32), + "armeb" : (40, 0, 0, False, 32), + }, + "linux-gnuspe" : { + "powerpc": (20, 0, 0, False, 32), + }, + "linux-muslspe" : { + "powerpc": (20, 0, 0, False, 32), + }, + "linux-gnu" : { + "powerpc": (20, 0, 0, False, 32), + "sh4": (42, 0, 0, True, 32), + }, + "linux-gnu_ilp32" : { + "aarch64" : (183, 0, 0, True, 32), + }, + "linux-gnux32" : { + "x86_64": (62, 0, 0, True, 32), + }, + "linux-muslx32" : { + "x86_64": (62, 0, 0, True, 32), + }, + "linux-gnun32" : { + "mips64": ( 8, 0, 0, False, 32), + "mips64el": ( 8, 0, 0, True, 32), + "mipsisa64r6": ( 8, 0, 0, False, 32), + "mipsisa64r6el":( 8, 0, 0, True, 32), + }, + } # Add in any extra user supplied data which may come from a BSP layer, removing the # need to always change this class directly extra_machdata = (d and d.getVar("PACKAGEQA_EXTRA_MACHDEFFUNCS" or None) or "").split() for m in extra_machdata: call = m + "(machdata, d)" - locs = { "machdata" : machdata, "d" : d} - machdata = bb.utils.better_eval(call, locs) + locs = { "machdata" : machine_dict.machdata, "d" : d} + machine_dict.machdata = bb.utils.better_eval(call, locs) - return machdata + return machine_dict.machdata
Signature · 1 line
2.34.1
PATCH v2 5/11 insane: only parse ELFs if they're files, not symlinks
In reply to Ross BurtonThis reduces the number of files that need to be swept by not scanning eg the library symlinks, and means we can remove the explicit islink() checks in many of the tests.
--- meta/classes-global/insane.bbclass | 21 ++------------------- 1 file changed, 2 insertions(+), 19 deletions(-)
meta/classes-global/insane.bbclass +2 -19
@@ -122,9 +122,6 @@ def package_qa_check_rpath(file,name, d, elf): if not elf: return - if os.path.islink(file): - return - bad_dirs = [d.getVar('BASE_WORKDIR'), d.getVar('STAGING_DIR_TARGET')] phdrs = elf.run_objdump("-p", d)
@@ -150,9 +147,6 @@ def package_qa_check_useless_rpaths(file, name, d, elf): if not elf: return - if os.path.islink(file): - return - libdir = d.getVar("libdir") base_libdir = d.getVar("base_libdir")
@@ -337,11 +331,6 @@ def package_qa_check_arch(path,name,d, elf): oe.qa.handle_error("arch", pn + ": Recipe inherits the allarch class, but has packaged architecture-specific binaries", d) return - # avoid following links to /usr/bin (e.g. on udev builds) - # we will check the files pointed to anyway... - if os.path.islink(path): - return - #if this will throw an exception, then fix the dict above (machine, osabi, abiversion, littleendian, bits) \ = oe.elf.machine_dict(d)[host_os][host_arch]
@@ -383,9 +372,6 @@ def package_qa_textrel(path, name, d, elf): if not elf: return - if os.path.islink(path): - return - phdrs = elf.run_objdump("-p", d) import re
@@ -405,9 +391,6 @@ def package_qa_hash_style(path, name, d, elf): if not elf: return - if os.path.islink(path): - return - gnu_hash = "--hash-style=gnu" in d.getVar('LDFLAGS') if not gnu_hash: gnu_hash = "--hash-style=both" in d.getVar('LDFLAGS')
@@ -601,7 +584,7 @@ def check_32bit_symbols(path, packagename, d, elf): ) # elf is a oe.qa.ELFFile object - if elf is not None: + if elf: phdrs = elf.run_objdump("-tw", d) syms = re.finditer(ptrn, phdrs) usedapis = {sym.group('notag') for sym in syms}
@@ -789,7 +772,7 @@ def package_qa_walk(checkfuncs, package, d): elves = {} for path in pkgfiles[package]: elf = None - if os.path.isfile(path): + if os.path.isfile(path) and not os.path.islink(path): elf = oe.qa.ELFFile(path) try: elf.open()
Signature · 1 line
2.34.1
PATCH v2 6/11 insane: check for RUNPATH as well as RPATH
In reply to Ross BurtonSince oe-core 66f8a7 merged in 2023[1], ld sets DT_RUNPATH instead of DT_RPATH when -rpath is specified, which we don't check for.
Update the insane tests to look at both RPATH and RUNPATH.
[1] oe-core 66f8a745668a067d8d763fa2af3e65f26c9c1ebe
--- meta/classes-global/insane.bbclass | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
meta/classes-global/insane.bbclass +4 -4
@@ -115,7 +115,7 @@ def package_qa_check_libexec(path,name, d, elf): oe.qa.handle_error("libexec", "%s: %s is using libexec please relocate to %s" % (name, package_qa_clean_path(path, d, name), libexec), d) QAPATHTEST[rpaths] = "package_qa_check_rpath" -def package_qa_check_rpath(file,name, d, elf): +def package_qa_check_rpath(file, name, d, elf): """ Check for dangerous RPATHs """
@@ -127,14 +127,14 @@ def package_qa_check_rpath(file,name, d, elf): phdrs = elf.run_objdump("-p", d) import re - rpath_re = re.compile(r"\s+RPATH\s+(.*)") + rpath_re = re.compile(r"\s+(?:RPATH|RUNPATH)\s+(.*)") for line in phdrs.split("\n"): m = rpath_re.match(line) if m: rpath = m.group(1) for dir in bad_dirs: if dir in rpath: - oe.qa.handle_error("rpaths", "package %s contains bad RPATH %s in file %s" % (name, rpath, file), d) + oe.qa.handle_error("rpaths", "%s: %s contains bad RPATH %s" % (name, package_qa_clean_path(file, d, name), rpath), d) QAPATHTEST[useless-rpaths] = "package_qa_check_useless_rpaths" def package_qa_check_useless_rpaths(file, name, d, elf):
@@ -153,7 +153,7 @@ def package_qa_check_useless_rpaths(file, name, d, elf): phdrs = elf.run_objdump("-p", d) import re - rpath_re = re.compile(r"\s+RPATH\s+(.*)") + rpath_re = re.compile(r"\s+(?:RPATH|RUNPATH)\s+(.*)") for line in phdrs.split("\n"): m = rpath_re.match(line) if m:
Signature · 1 line
2.34.1
PATCH v2 7/11 insane: tidy up objdump preloading in package_qa_walk()
In reply to Ross BurtonMove the prepopulate function out of global scope, and access the dictionary once instead of repeatedly.
This still results in each ELF being opened twice, but this avoids opening all of the files at once and the ELFFile.open() call is fairly fast.
--- meta/classes-global/insane.bbclass | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-)
meta/classes-global/insane.bbclass +11 -10
@@ -763,10 +763,6 @@ def qa_check_staged(path,d): if not skip_shebang_size: package_qa_check_shebang_size(path, "", d, None) -def prepopulate_objdump_p(elf, d): - output = elf.run_objdump("-p", d) - return (elf.name, output) - # Walk over all files in a directory and call func def package_qa_walk(checkfuncs, package, d): elves = {}
@@ -782,17 +778,22 @@ def package_qa_walk(checkfuncs, package, d): if elf: elves[path] = elf + def prepopulate_objdump_p(elf, d): + output = elf.run_objdump("-p", d) + return (elf.name, output) + results = oe.utils.multiprocess_launch(prepopulate_objdump_p, elves.values(), d, extraargs=(d,)) for item in results: elves[item[0]].set_objdump("-p", item[1]) for path in pkgfiles[package]: - if path in elves: - elves[path].open() - for func in checkfuncs: - func(path, package, d, elves.get(path)) - if path in elves: - elves[path].close() + elf = elves.get(path) + if elf: + elf.open() + for func in checkfuncs: + func(path, package, d, elf) + if elf: + elf.close() def package_qa_check_rdepends(pkg, pkgdest, skip, taskdeps, packages, d): # Don't do this check for kernel/module recipes, there aren't too many debug/development
Signature · 1 line
2.34.1
PATCH v2 8/11 insane: micro-optimise the sweep of pkgfiles
In reply to Ross BurtonDon't actively do more work:
- Exit early if there are no packages being generated
- Don't iterate repeatedly when removing CONTROL and DEBIAN
- Extend a list with another list instead of appending item by item
- Remove unused variables
--- meta/classes-global/insane.bbclass | 27 +++++++++++---------------- 1 file changed, 11 insertions(+), 16 deletions(-)
meta/classes-global/insane.bbclass +11 -16
@@ -1084,13 +1084,8 @@ parse_test_matrix[vardepsexclude] = "ERROR_QA WARN_QA" # The PACKAGE FUNC to scan each package python do_package_qa () { - import subprocess import oe.packagedata - bb.note("DO PACKAGE QA") - - main_lic = d.getVar('LICENSE') - # Check for obsolete license references in main LICENSE (packages are checked below for any changes) main_licenses = oe.license.list_licenses(d.getVar('LICENSE')) obsolete = set(oe.license.obsolete_license_list()) & main_licenses
@@ -1106,27 +1101,27 @@ python do_package_qa () { pn = d.getVar('PN') # Scan the packages... - pkgdest = d.getVar('PKGDEST') packages = set((d.getVar('PACKAGES') or '').split()) + # no packages should be scanned + if not packages: + return global pkgfiles pkgfiles = {} + pkgdest = d.getVar('PKGDEST') for pkg in packages: - pkgfiles[pkg] = [] pkgdir = os.path.join(pkgdest, pkg) + pkgfiles[pkg] = [] for walkroot, dirs, files in os.walk(pkgdir): # Don't walk into top-level CONTROL or DEBIAN directories as these # are temporary directories created by do_package. if walkroot == pkgdir: - for control in ("CONTROL", "DEBIAN"): - if control in dirs: - dirs.remove(control) - for file in files: - pkgfiles[pkg].append(os.path.join(walkroot, file)) - - # no packages should be scanned - if not packages: - return + for removedir in ("CONTROL", "DEBIAN"): + try: + dirs.remove(removedir) + except ValueError: + pass + pkgfiles[pkg].extend((os.path.join(walkroot, f) for f in files)) import re # The package name matches the [a-z0-9.+-]+ regular expression
Signature · 1 line
2.34.1
PATCH v2 9/11 insane: optimise test in package_qa_check_shebang_size
In reply to Ross BurtonCheck whether the elf is not None first, before doing os.stat() calls on disk. Also don't check anything that isn't a file, not just FIFOs.
--- meta/classes-global/insane.bbclass | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
meta/classes-global/insane.bbclass +1 -2
@@ -82,8 +82,7 @@ def package_qa_clean_path(path, d, pkg=None): QAPATHTEST[shebang-size] = "package_qa_check_shebang_size" def package_qa_check_shebang_size(path, name, d, elf): - import stat - if os.path.islink(path) or stat.S_ISFIFO(os.stat(path).st_mode) or elf: + if elf or os.path.islink(path) or not os.path.isfile(path): return try:
Signature · 1 line
2.34.1
PATCH v2 10/11 insane: use oe.cachedpath.CachedPath instead of os.path
In reply to Ross BurtonThe insane QAPATHTESTs make many os.stat() calls, the majority of which are redundant with caching as the initial sweep does a stat() on every entry to determine if it is a file or a directory, and from then on each test that does further stat()s is redundant as the tree doesn't change.
Switch os.stat() and friends (os.path.isfile(), etc) to use a common oe.cachedpath.CachedPath() instance that is shared between all of the functions, meaning only one stat is done.
In my test case of ltp:do_package_qa, this reduces the time taken from 44s to 37s.
--- meta/classes-global/insane.bbclass | 42 ++++++++++++++++++++---------- 1 file changed, 28 insertions(+), 14 deletions(-)
meta/classes-global/insane.bbclass +28 -14
@@ -82,7 +82,9 @@ def package_qa_clean_path(path, d, pkg=None): QAPATHTEST[shebang-size] = "package_qa_check_shebang_size" def package_qa_check_shebang_size(path, name, d, elf): - if elf or os.path.islink(path) or not os.path.isfile(path): + global cpath + + if elf or cpath.islink(path) or not cpath.isfile(path): return try:
@@ -167,8 +169,8 @@ def package_qa_check_dev(path, name, d, elf): """ Check for ".so" library symlinks in non-dev packages """ - - if not name.endswith("-dev") and not name.endswith("-dbg") and not name.endswith("-ptest") and not name.startswith("nativesdk-") and path.endswith(".so") and os.path.islink(path): + global cpath + if not name.endswith("-dev") and not name.endswith("-dbg") and not name.endswith("-ptest") and not name.startswith("nativesdk-") and path.endswith(".so") and cpath.islink(path): oe.qa.handle_error("dev-so", "non -dev/-dbg/nativesdk- package %s contains symlink .so '%s'" % \ (name, package_qa_clean_path(path, d, name)), d)
@@ -179,7 +181,8 @@ def package_qa_check_dev_elf(path, name, d, elf): check that the file is not a link and is an ELF object as some recipes install link-time .so files that are linker scripts. """ - if name.endswith("-dev") and path.endswith(".so") and not os.path.islink(path) and elf: + global cpath + if name.endswith("-dev") and path.endswith(".so") and not cpath.islink(path) and elf: oe.qa.handle_error("dev-elf", "-dev package %s contains non-symlink .so '%s'" % \ (name, package_qa_clean_path(path, d, name)), d)
@@ -422,9 +425,9 @@ def package_qa_check_buildpaths(path, name, d, elf): explicitly ignored. """ import stat - + global cpath # Ignore symlinks/devs/fifos - mode = os.lstat(path).st_mode + mode = cpath.lstat(path).st_mode if stat.S_ISLNK(mode) or stat.S_ISBLK(mode) or stat.S_ISFIFO(mode) or stat.S_ISCHR(mode) or stat.S_ISSOCK(mode): return
@@ -469,7 +472,8 @@ def package_qa_check_symlink_to_sysroot(path, name, d, elf): """ Check that the package doesn't contain any absolute symlinks to the sysroot. """ - if os.path.islink(path): + global cpath + if cpath.islink(path): target = os.readlink(path) if os.path.isabs(target): tmpdir = d.getVar('TMPDIR')
@@ -760,14 +764,19 @@ def qa_check_staged(path,d): oe.qa.handle_error("pkgconfig", error_msg, d) if not skip_shebang_size: + global cpath + cpath = oe.cachedpath.CachedPath() package_qa_check_shebang_size(path, "", d, None) + cpath = None # Walk over all files in a directory and call func def package_qa_walk(checkfuncs, package, d): + global cpath + elves = {} for path in pkgfiles[package]: elf = None - if os.path.isfile(path) and not os.path.islink(path): + if cpath.isfile(path) and not cpath.islink(path): elf = oe.qa.ELFFile(path) try: elf.open()
@@ -915,11 +924,12 @@ def package_qa_check_deps(pkg, pkgdest, d): QAPKGTEST[usrmerge] = "package_qa_check_usrmerge" def package_qa_check_usrmerge(pkg, d): + global cpath pkgdest = d.getVar('PKGDEST') pkg_dir = pkgdest + os.sep + pkg + os.sep merged_dirs = ['bin', 'sbin', 'lib'] + d.getVar('MULTILIB_VARIANTS').split() for f in merged_dirs: - if os.path.exists(pkg_dir + f) and not os.path.islink(pkg_dir + f): + if cpath.exists(pkg_dir + f) and not cpath.islink(pkg_dir + f): msg = "%s package is not obeying usrmerge distro feature. /%s should be relocated to /usr." % (pkg, f) oe.qa.handle_error("usrmerge", msg, d) return
@@ -985,10 +995,11 @@ def package_qa_check_empty_dirs(pkg, d): empty. """ + global cpath pkgd = oe.path.join(d.getVar('PKGDEST'), pkg) for dir in (d.getVar('QA_EMPTY_DIRS') or "").split(): empty_dir = oe.path.join(pkgd, dir) - if os.path.exists(empty_dir) and os.listdir(empty_dir): + if cpath.exists(empty_dir) and os.listdir(empty_dir): recommendation = (d.getVar('QA_EMPTY_DIRS_RECOMMENDATION:' + dir) or "but it is expected to be empty") msg = "%s installs files in %s, %s" % (pkg, dir, recommendation)
@@ -1018,8 +1029,9 @@ HOST_USER_GID := "${@os.getgid()}" QAPATHTEST[host-user-contaminated] = "package_qa_check_host_user" def package_qa_check_host_user(path, name, d, elf): """Check for paths outside of /home which are owned by the user running bitbake.""" + global cpath - if not os.path.lexists(path): + if not cpath.lexists(path): return dest = d.getVar('PKGDEST')
@@ -1029,7 +1041,7 @@ def package_qa_check_host_user(path, name, d, elf): return try: - stat = os.lstat(path) + stat = cpath.lstat(path) except OSError as exc: import errno if exc.errno != errno.ENOENT:
@@ -1105,13 +1117,14 @@ python do_package_qa () { if not packages: return - global pkgfiles + global pkgfiles, cpath pkgfiles = {} + cpath = oe.cachedpath.CachedPath() pkgdest = d.getVar('PKGDEST') for pkg in packages: pkgdir = os.path.join(pkgdest, pkg) pkgfiles[pkg] = [] - for walkroot, dirs, files in os.walk(pkgdir): + for walkroot, dirs, files in cpath.walk(pkgdir): # Don't walk into top-level CONTROL or DEBIAN directories as these # are temporary directories created by do_package. if walkroot == pkgdir:
@@ -1159,6 +1172,7 @@ python do_package_qa () { package_qa_check_libdir(d) + cpath = None oe.qa.exit_if_errors(d) }
Signature · 1 line
2.34.1
Re: [OE-core] [PATCH v2 10/11] insane: use oe.cachedpath.CachedPath instead of os.path
After this merged, seemingly random package_qa errors started appearing, check sub-fails here:
https://valkyrie.yoctoproject.org/#/builders/29/builds/257
File: '/srv/pokybuild/yocto-worker/musl-qemux86/build/meta/classes-global/insane.bbclass',
lineno: 439, function: package_qa_check_buildpaths
0435: """
0436: import stat
0437: global cpath
0438: # Ignore symlinks/devs/fifos
*** 0439: mode = cpath.lstat(path).st_mode
0440: if stat.S_ISLNK(mode) or stat.S_ISBLK(mode) or
stat.S_ISFIFO(mode) or stat.S_ISCHR(mode) or stat.S_ISSOCK(mode):
0441: return
0442:
0443: tmpdir = bytes(d.getVar('TMPDIR'), encoding="utf-8")Exception: AttributeError: 'bool' object has no attribute 'st_mode'
Alex
Quoted 181 lines · Ross Burton via lists.openembedded.org
The insane QAPATHTESTs make many
os.stat()calls, the majority of which are redundant with caching as the initial sweep does astat()on every entry to determine if it is a file or a directory, and from then on each test that does further stat()s is redundant as the tree doesn't change.Switch
os.stat()and friends (os.path.isfile(), etc) to use a commonoe.cachedpath.CachedPath()instance that is shared between all of the functions, meaning only one stat is done.In my test case of ltp:do_package_qa, this reduces the time taken from 44s to 37s.
Signed-off-by Ross Burton <[email protected]>--- meta/classes-global/insane.bbclass | 42 ++++++++++++++++++++---------- 1 file changed, 28 insertions(+), 14 deletions(-)meta/classes-global/insane.bbclass +28
-15@@ -82,7 +82,9 @@ def package_qa_clean_path(path, d, pkg=None): QAPATHTEST[shebang-size] = "package_qa_check_shebang_size" def package_qa_check_shebang_size(path, name, d, elf): - if elf or os.path.islink(path) or not os.path.isfile(path): + global cpath + + if elf or cpath.islink(path) or not cpath.isfile(path): return try:@@ -167,8 +169,8 @@ def package_qa_check_dev(path, name, d, elf): """ Check for ".so" library symlinks in non-dev packages """ - - if not name.endswith("-dev") and not name.endswith("-dbg") and not name.endswith("-ptest") and not name.startswith("nativesdk-") and path.endswith(".so") and os.path.islink(path): + global cpath + if not name.endswith("-dev") and not name.endswith("-dbg") and not name.endswith("-ptest") and not name.startswith("nativesdk-") and path.endswith(".so") and cpath.islink(path): oe.qa.handle_error("dev-so", "non -dev/-dbg/nativesdk- package %s contains symlink .so '%s'" % \ (name, package_qa_clean_path(path, d, name)), d)@@ -179,7 +181,8 @@ def package_qa_check_dev_elf(path, name, d, elf): check that the file is not a link and is an ELF object as some recipes install link-time .so files that are linker scripts. """ - if name.endswith("-dev") and path.endswith(".so") and not os.path.islink(path) and elf: + global cpath + if name.endswith("-dev") and path.endswith(".so") and not cpath.islink(path) and elf: oe.qa.handle_error("dev-elf", "-dev package %s contains non-symlink .so '%s'" % \ (name, package_qa_clean_path(path, d, name)), d)@@ -422,9 +425,9 @@ def package_qa_check_buildpaths(path, name, d, elf): explicitly ignored. """ import stat - + global cpath # Ignore symlinks/devs/fifos - mode = os.lstat(path).st_mode + mode = cpath.lstat(path).st_mode if stat.S_ISLNK(mode) or stat.S_ISBLK(mode) or stat.S_ISFIFO(mode) or stat.S_ISCHR(mode) or stat.S_ISSOCK(mode): return@@ -469,7 +472,8 @@ def package_qa_check_symlink_to_sysroot(path, name, d, elf): """ Check that the package doesn't contain any absolute symlinks to the sysroot. """ - if os.path.islink(path): + global cpath + if cpath.islink(path): target = os.readlink(path) if os.path.isabs(target): tmpdir = d.getVar('TMPDIR')@@ -760,14 +764,19 @@ def qa_check_staged(path,d): oe.qa.handle_error("pkgconfig", error_msg, d) if not skip_shebang_size: + global cpath + cpath = oe.cachedpath.CachedPath() package_qa_check_shebang_size(path, "", d, None) + cpath = None # Walk over all files in a directory and call func def package_qa_walk(checkfuncs, package, d): + global cpath + elves = {} for path in pkgfiles[package]: elf = None - if os.path.isfile(path) and not os.path.islink(path): + if cpath.isfile(path) and not cpath.islink(path): elf = oe.qa.ELFFile(path) try: elf.open()@@ -915,11 +924,12 @@ def package_qa_check_deps(pkg, pkgdest, d): QAPKGTEST[usrmerge] = "package_qa_check_usrmerge" def package_qa_check_usrmerge(pkg, d): + global cpath pkgdest = d.getVar('PKGDEST') pkg_dir = pkgdest + os.sep + pkg + os.sep merged_dirs = ['bin', 'sbin', 'lib'] + d.getVar('MULTILIB_VARIANTS').split() for f in merged_dirs: - if os.path.exists(pkg_dir + f) and not os.path.islink(pkg_dir + f): + if cpath.exists(pkg_dir + f) and not cpath.islink(pkg_dir + f): msg = "%s package is not obeying usrmerge distro feature. /%s should be relocated to /usr." % (pkg, f) oe.qa.handle_error("usrmerge", msg, d) return@@ -985,10 +995,11 @@ def package_qa_check_empty_dirs(pkg, d): empty. """ + global cpath pkgd = oe.path.join(d.getVar('PKGDEST'), pkg) for dir in (d.getVar('QA_EMPTY_DIRS') or "").split(): empty_dir = oe.path.join(pkgd, dir) - if os.path.exists(empty_dir) and os.listdir(empty_dir): + if cpath.exists(empty_dir) and os.listdir(empty_dir): recommendation = (d.getVar('QA_EMPTY_DIRS_RECOMMENDATION:' + dir) or "but it is expected to be empty") msg = "%s installs files in %s, %s" % (pkg, dir, recommendation)@@ -1018,8 +1029,9 @@ HOST_USER_GID := "${@os.getgid()}" QAPATHTEST[host-user-contaminated] = "package_qa_check_host_user" def package_qa_check_host_user(path, name, d, elf): """Check for paths outside of /home which are owned by the user running bitbake.""" + global cpath - if not os.path.lexists(path): + if not cpath.lexists(path): return dest = d.getVar('PKGDEST')@@ -1029,7 +1041,7 @@ def package_qa_check_host_user(path, name, d, elf): return try: - stat = os.lstat(path) + stat = cpath.lstat(path) except OSError as exc: import errno if exc.errno != errno.ENOENT:@@ -1105,13 +1117,14 @@ python do_package_qa () { if not packages: return - global pkgfiles + global pkgfiles, cpath pkgfiles = {} + cpath = oe.cachedpath.CachedPath() pkgdest = d.getVar('PKGDEST') for pkg in packages: pkgdir = os.path.join(pkgdest, pkg) pkgfiles[pkg] = [] - for walkroot, dirs, files in os.walk(pkgdir): + for walkroot, dirs, files in cpath.walk(pkgdir): # Don't walk into top-level CONTROL or DEBIAN directories as these # are temporary directories created by do_package. if walkroot == pkgdir:@@ -1159,6 +1172,7 @@ python do_package_qa () { package_qa_check_libdir(d) + cpath = None oe.qa.exit_if_errors(d) } --2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#205419): https://lists.openembedded.org/g/openembedded-core/message/205419
Mute This Topic: https://lists.openembedded.org/mt/108932135/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v2 10/11] insane: use oe.cachedpath.CachedPath instead of os.path
Quoted 4 lines · Alexander Kanavin
After this merged, seemingly random
package_qaerrors started appearing, check sub-fails here:
https://valkyrie.yoctoproject.org/#/builders/29/builds/257
Quoted 15 lines
File: '
/srv/pokybuild/yocto-worker/musl-qemux86/build/meta/classes-global/insane.bbclass',
lineno: 439, function:package_qa_check_buildpaths
0435: """
0436: import stat
0437: global cpath
0438: # Ignore symlinks/devs/fifos
*** 0439: mode = cpath.lstat(path).st_mode
0440: ifstat.S_ISLNK(mode)orstat.S_ISBLK(mode)orstat.S_ISFIFO(mode)orstat.S_ISCHR(mode)orstat.S_ISSOCK(mode):
0441: return
0442:
0443: tmpdir =bytes(d.getVar('TMPDIR'), encoding="utf-8")
Exception: AttributeError: 'bool' object has no attribute 'st_mode'
They're not random. I spotted a potential issue and have put a potential fix in master-next.
Cheers,
Richard
PATCH v2 11/11 insane: rewrite package_qa_check_arch
In reply to Ross BurtonReorder and comment the architecture checks to make it clearer what they are actually checking.
--- meta/classes-global/insane.bbclass | 41 ++++++++++++++++++------------ 1 file changed, 25 insertions(+), 16 deletions(-)
meta/classes-global/insane.bbclass +25 -16
@@ -325,32 +325,41 @@ def package_qa_check_arch(path,name,d, elf): host_os = d.getVar('HOST_OS') host_arch = d.getVar('HOST_ARCH') - provides = d.getVar('PROVIDES') - bpn = d.getVar('BPN') + provides = d.getVar('PROVIDES') if host_arch == "allarch": - pn = d.getVar('PN') - oe.qa.handle_error("arch", pn + ": Recipe inherits the allarch class, but has packaged architecture-specific binaries", d) + oe.qa.handle_error("arch", "%s: inherits the allarch class, but has architecture-specific binaries %s" % \ + (name, package_qa_clean_path(path, d, name)), d) return - #if this will throw an exception, then fix the dict above - (machine, osabi, abiversion, littleendian, bits) \ + # If this throws an exception, the machine_dict needs expanding + (expected_machine, expected_osabi, expected_abiversion, expected_littleendian, expected_bits) \ = oe.elf.machine_dict(d)[host_os][host_arch] + actual_machine = elf.machine() + actual_bits = elf.abiSize() + actual_littleendian = elf.isLittleEndian() + + # BPF don't match the target + if oe.qa.elf_machine_to_string(actual_machine) == "BPF": + return + + # These targets have 32-bit userspace but 64-bit kernel, so fudge the expected values + if (("virtual/kernel" in provides) or bb.data.inherits_class("module", d)) and (host_os in ("linux-gnux32", "linux-muslx32", "linux-gnu_ilp32") or re.match(r'mips64.*32', d.getVar('DEFAULTTUNE'))): + expected_bits = 64 + # Check the architecture and endiannes of the binary - is_32 = (("virtual/kernel" in provides) or bb.data.inherits_class("module", d)) and \ - (host_os == "linux-gnux32" or host_os == "linux-muslx32" or \ - host_os == "linux-gnu_ilp32" or re.match(r'mips64.*32', d.getVar('DEFAULTTUNE'))) - is_bpf = (oe.qa.elf_machine_to_string(elf.machine()) == "BPF") - if not ((machine == elf.machine()) or is_32 or is_bpf): + if expected_machine != actual_machine: oe.qa.handle_error("arch", "Architecture did not match (%s, expected %s) in %s" % \ - (oe.qa.elf_machine_to_string(elf.machine()), oe.qa.elf_machine_to_string(machine), package_qa_clean_path(path, d, name)), d) - elif not ((bits == elf.abiSize()) or is_32 or is_bpf): + (oe.qa.elf_machine_to_string(actual_machine), oe.qa.elf_machine_to_string(expected_machine), package_qa_clean_path(path, d, name)), d) + + if expected_bits != actual_bits: oe.qa.handle_error("arch", "Bit size did not match (%d, expected %d) in %s" % \ - (elf.abiSize(), bits, package_qa_clean_path(path, d, name)), d) - elif not ((littleendian == elf.isLittleEndian()) or is_bpf): + (actual_bits, expected_bits, package_qa_clean_path(path, d, name)), d) + + if expected_littleendian != actual_littleendian: oe.qa.handle_error("arch", "Endiannes did not match (%d, expected %d) in %s" % \ - (elf.isLittleEndian(), littleendian, package_qa_clean_path(path, d, name)), d) + (actual_littleendian, expected_littleendian, package_qa_clean_path(path, d, name)), d) package_qa_check_arch[vardepsexclude] = "DEFAULTTUNE" QAPATHTEST[desktop] = "package_qa_check_desktop"
Signature · 1 line
2.34.1