Skip to contentSkip to conversation

python3-kirk: Add version 4.0.0

PATCH In review [email protected] 2 participants 8 messages Mar 27

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/6 python3-kirk: Add version 4.0.0

From: Daniel Turull <[email protected]>

Kirk application is a fork of runltp-ng and it’s the official LTP tests executor.

---

v2: use pypi bbclass instead of github repository

---
 meta/conf/distro/include/maintainers.inc         |  1 +
 .../python3-kirk/python3-kirk_4.0.0.bb           | 16 ++++++++++++++++
 2 files changed, 17 insertions(+)
 create mode 100644 meta/recipes-extended/python3-kirk/python3-kirk_4.0.0.bb
meta/conf/distro/include/maintainers.inc +1 -0
index f9a600ff11..9cd58dbac0 100644
@@ -665,6 +665,7 @@ RECIPE_MAINTAINER:pn-python3-jinja2 = "Richard Purdie <richard.purdie@linuxfound
 RECIPE_MAINTAINER:pn-python3-jsonpointer = "Bruce Ashfield <[email protected]>"
 RECIPE_MAINTAINER:pn-python3-jsonschema = "Bruce Ashfield <[email protected]>"
 RECIPE_MAINTAINER:pn-python3-jsonschema-specifications = "Bruce Ashfield <[email protected]>"
+RECIPE_MAINTAINER:pn-python3-kirk = "Daniel Turull <[email protected]>"
 RECIPE_MAINTAINER:pn-python3-libarchive-c = "Joshua Watt <[email protected]>"
 RECIPE_MAINTAINER:pn-python3-license-expression = "Wang Mingyu <[email protected]>"
 RECIPE_MAINTAINER:pn-python3-lxml = "Khem Raj <[email protected]>"
meta/recipes-extended/python3-kirk/python3-kirk_4.0.0.bb new file +16 -0
new file mode 100644
index 0000000000..af53d2edf3
@@ -0,0 +1,16 @@
+SUMMARY = "Linux Test Project tests executor"
+HOMEPAGE = "https://github.com/linux-test-project/kirk"
+DESCRIPTION = "Kirk application is a fork of runltp-ng and it’s the official \
+LTP tests executor. It provides support for remote testing via Qemu, SSH, \
+LTX, parallel execution and much more."
+SECTION = "console/utils"
+LICENSE = "GPL-2.0-only"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=39bba7d2cf0ba1036f2a6e2be52fe3f0"
+
+SRC_URI[sha256sum] = "4606f46dfdb7be1e1c2de8d8f040037459a9de8a07fa1123cee4b68c7f7cb764"
+
+inherit pypi python_setuptools_build_meta
+
+PYPI_PACKAGE = "kirk"
+
+RDEPENDS:${PN} += "python3-asyncio python3-json"
#1 lore raw

PATCH v2 2/6 ltp: disable min_free_kbytes

From: Daniel Turull <[email protected]>

It casues OOM regardless of the configuration

There is a bug in ltp, fixed in a newer version
https://github.com/linux-test-project/ltp/commit/355124aea99c7c7d40140c969a53ed892421dee9

min_free_kbytes: Connection to 192.168.7.2 closed by remote host.

NOTE: test_ltp_runltp_cve (ltp.LtpTest.test_ltp_runltp_cve)
NOTE: ... skipped "Test case ltp.LtpTest.test_ltp_runltp_cve depends on ltp.LtpTest.test_ltp_groups but it didn't pass/run." Test case ltp.LtpTest.test_ltp_runltp_cve depends on ltp.LtpTest.test_ltp_groups but it didn't pass/run.
NOTE: ======================================================================
NOTE: FAIL: test_ltp_groups (ltp.LtpTest.test_ltp_groups)
NOTE: ----------------------------------------------------------------------

---
If it needs to be added, we can revert this and have the backported patch. We need further testing to enable this again

---
 meta/recipes-extended/ltp/ltp_20260130.bb | 3 +++
 1 file changed, 3 insertions(+)
meta/recipes-extended/ltp/ltp_20260130.bb +3 -0
index dcd1e81398..0b4f9c1ad5 100644
@@ -78,6 +78,9 @@ do_install(){
     # The controllers memcg_stree test seems to cause us hangs and takes 900s
     # (maybe we expect more regular output?), anyhow, skip it
     sed -e '/^memcg_stress/d' -i ${D}${prefix}/runtest/controllers
+
+    # min_free_kbytes can be disruptive on constrained targets
+    sed -e '/^min_free_kbytes/d' -i ${D}${prefix}/runtest/mm
 }
 
 RDEPENDS:${PN} = "\
#2 lore raw

PATCH v2 3/6 ltp: disable pty testcase

In reply to [email protected]

From: Daniel Turull <[email protected]>

On qemux86-64 with 1 CPU, 4GB causes an OOM on the following test cases: pty07 and ptem02

---
We further investigate the root cause and fix the causing issue

---
 meta/lib/oeqa/runtime/cases/ltp.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oeqa/runtime/cases/ltp.py +1 -1
index 0ffdbe23e4..4f7498fcbd 100644
@@ -57,7 +57,7 @@ class LtpTestBase(OERuntimeTestCase):
 
 class LtpTest(LtpTestBase):
 
-    ltp_groups = ["math", "syscalls", "dio", "mm", "sched", "nptl", "pty", "containers", "controllers", "fcntl-locktests", "commands", "net.ipv6_lib", "input","fs_perms_simple", "cve", "crypto", "ima", "net.nfs", "net_stress.ipsec_icmp", "net.ipv6", "numa", "uevent", "ltp-aiodio.part1", "ltp-aiodio.part2", "ltp-aiodio.part3", "ltp-aiodio.part4"]
+    ltp_groups = ["math", "syscalls", "dio", "mm", "sched", "nptl", "containers", "controllers", "fcntl-locktests", "commands", "net.ipv6_lib", "input","fs_perms_simple", "cve", "crypto", "ima", "net.nfs", "net_stress.ipsec_icmp", "net.ipv6", "numa", "uevent", "ltp-aiodio.part1", "ltp-aiodio.part2", "ltp-aiodio.part3", "ltp-aiodio.part4"]
 
     ltp_fs = ["fs", "fs_bind"]
     # skip kernel cpuhotplug
#3 lore raw

PATCH v2 4/6 ltp: disable cve failing testcases

In reply to [email protected]

From: Daniel Turull <[email protected]>

On a qemux86-64 with 1 CPU the following testcases causes an OOM

CVE-2020-36557 and CVE-2018-13405

Test cases triggers memory deadlock kernel panic with CVE-2018-13405

---
We will investigate why to be able to enable them again

---
 meta/recipes-extended/ltp/ltp_20260130.bb | 6 ++++++
 1 file changed, 6 insertions(+)
meta/recipes-extended/ltp/ltp_20260130.bb +6 -0
index 0b4f9c1ad5..e5da94855f 100644
@@ -81,6 +81,12 @@ do_install(){
 
     # min_free_kbytes can be disruptive on constrained targets
     sed -e '/^min_free_kbytes/d' -i ${D}${prefix}/runtest/mm
+
+    # cve-2018-13405 triggers memory deadlock kernel panic on constrained targets
+    sed -e '/^cve-2018-13405/d' -i ${D}${prefix}/runtest/cve
+
+    # cve-2020-36557 triggers memory deadlock kernel panic on constrained targets
+    sed -e '/^cve-2020-36557/d' -i ${D}${prefix}/runtest/cve
 }
 
 RDEPENDS:${PN} = "\
#4 lore raw

PATCH v2 5/6 oeqa: replace runltp with kirk

In reply to [email protected]

From: Daniel Turull <[email protected]>

runltp has been removed from ltp and the official tool to invoke ltp test is kirk.

Message when running runltp:

"INFO: runltp script is deprecated, try kirk
https://github.com/linux-test-project/kirk"

See also:
https://github.com/linux-test-project/ltp/commit/6efd3605dc005c3ed135b463f182174e24bdce1b

The test coverage is the same since the logic there is not touched but the output is different. The return value from oeqa are the same as before.

Now it is a json file with the results, which is machine readable without any extra parsing. Two files are created by test suite <test>.json and <test>-raw.log

Output example of math.json:

{
    "results": [
        {
            "test_fqn": "abs01",
            "status": "pass",
            "test": {
                "command": "abs01",
                "arguments": [],
                "log": "abs01       1  TPASS  :  Test passed\nabs01       2  TPASS  :  Test passed\nabs01       3  TPASS  :  Test passed\n",
                "retval": [
                    "0"
                ],
                "duration": 0.002702474594116211,
                "failed": 0,
                "passed": 3,
                "broken": 0,
                "skipped": 0,
                "warnings": 0,
                "result": "pass"
            }
        }, [...]
    ],
    "stats": {
        "runtime": 1.4888691902160645,
        "passed": 22,
        "failed": 0,
        "broken": 0,
        "skipped": 0,
        "warnings": 0
    },
    "environment": {
        "distribution": "nodistro",
        "distribution_version": "nodistro.0",
        "kernel": "Linux 6.18.13-yocto-standard #1 SMP PREEMPT_DYNAMIC Tue Mar  3 16:48:55 UTC 2026",
        "cmdline": "root=/dev/vda rw  ip=192.168.7.2::192.168.7.1:255.255.255.0::eth0:off:8.8.8.8 net.ifnames=0 console=ttyS0 console=ttyS1 oprofile.timer=1 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 swiotlb=0  printk.time=1",
        "arch": "x86_64",
        "cpu": "unknown",
        "swap": "0 kB",
        "RAM": "222368 kB"
    }
}

Add in configuration file (local.conf)

IMAGE_CLASSES += "testimage"
CORE_IMAGE_EXTRA_INSTALL += "ltp openssh"
TEST_SUITES = "ping ssh ltp"
QB_MEM = <mem size>
QB_CPU_KVM = "-cpu host -smp <cpus>"
QB_SMP = "-smp <cpus>"
QB_KVM = "1"
IMAGE_ROOTFS_EXTRA_SPACE = "2097152"

Tested with different VM sizes and architectures:

qemux86-64 MEM: 4096, CPU: 2 (all ok)
qemux86-64 MEM: 1024, CPU: 1 (all ok)
qemux86-64 MEM: 1024, CPU: 2 (all ok)
qemux86-64 MEM: 1024, CPU: 4 (all ok)
qemuarm64: MEM: 4096, CPU: 1 (all ok)
qemuriscv64: MEM 4096, CPU 4 (all ok)

Then:
bitbake core-image-minimal
bitbake core-image-minimal -c testimage

With these configs we didn't see any testimage failures with kirk and run completes.

If we revert the ltp disable patches below, the testimage does not fail but it gives warnings, since the ssh connection gets droped from the DUT because of the OOM killer. Then in continues with the next testsuite.

* ltp: disable cve failing testcases
* ltp: disable pty testcase
* ltp: disable min_free_kbytes

Assisted-by Claude, Anthropic

---

v2:
- replace LtpParser with functionality to parse output from kirk
- improve commit message
- add timeout to kirk for 20 minutes, same as the ssh connection
- add try, except to handle exceptions in the run and be able to continue with the next test suite
- tested with more configurations

---
 meta/lib/oeqa/runtime/cases/ltp.py        | 21 ++++++----
 meta/lib/oeqa/runtime/cases/ltp_stress.py | 12 +++++-
 meta/lib/oeqa/utils/logparser.py          | 51 ++++++++---------------
 meta/recipes-extended/ltp/ltp_20260130.bb |  1 +
 4 files changed, 42 insertions(+), 43 deletions(-)
meta/lib/oeqa/runtime/cases/ltp.py +13 -8
index 4f7498fcbd..b855e76907 100644
@@ -66,9 +66,9 @@ class LtpTest(LtpTestBase):
 
     def runltp(self, ltp_group):
             # LTP appends to log files, so ensure we start with a clean log
-            self.target.deleteFiles("/opt/ltp/results/", ltp_group)
+            self.target.deleteFiles("/opt/ltp/results/", "%s.json" % ltp_group)
 
-            cmd = '/opt/ltp/runltp -f %s -q -r /opt/ltp -l /opt/ltp/results/%s -I 1 -d /opt/ltp' % (ltp_group, ltp_group)
+            cmd = 'kirk --run-suite %s --json-report /opt/ltp/results/%s.json -n -d /opt/ltp --exec-timeout 20m' % (ltp_group, ltp_group)
 
             starttime = time.time()
             (status, output) = self.target.run(cmd, timeout=1200)
@@ -87,8 +87,8 @@ class LtpTest(LtpTestBase):
             self.extras['ltpresult.rawlogs']['log'] = self.extras['ltpresult.rawlogs']['log'] + output
 
             # Copy the machine-readable test results locally so we can parse it
-            dst = os.path.join(self.ltptest_log_dir, ltp_group)
-            remote_src = "/opt/ltp/results/%s" % ltp_group 
+            dst = os.path.join(self.ltptest_log_dir, "%s.json" % ltp_group)
+            remote_src = "/opt/ltp/results/%s.json" % ltp_group
             (status, output) = self.target.copyFrom(remote_src, dst, True)
             if status:
                 msg = 'File could not be copied. Output: %s' % output
@@ -113,16 +113,21 @@ class LtpTest(LtpTestBase):
 
     # LTP runtime tests
     @OETestDepends(['ssh.SSHTest.test_ssh'])
-    @OEHasPackage(["ltp"])
+    @OEHasPackage(["ltp", "python3-kirk"])
     def test_ltp_help(self):
-        (status, output) = self.target.run('/opt/ltp/runltp --help')
+        (status, output) = self.target.run('kirk --help')
         msg = 'Failed to get ltp help. Output: %s' % output
         self.assertEqual(status, 0, msg=msg)
 
     @OETestDepends(['ltp.LtpTest.test_ltp_help'])
     def test_ltp_groups(self):
-        for ltp_group in self.ltp_groups: 
-            self.runltp(ltp_group)
+        for ltp_group in self.ltp_groups:
+            try:
+                self.runltp(ltp_group)
+            except Exception as e:
+                self.extras['ltpresult.%s' % ltp_group] = {'status': 'FAILED'}
+                self.failmsg = self.failmsg + "Suite %s crashed: %s\n" % (ltp_group, e)
+                self.target.logger.warning("Suite %s crashed, continuing: %s" % (ltp_group, e))
 
     @OETestDepends(['ltp.LtpTest.test_ltp_groups'])
     def test_ltp_runltp_cve(self):
meta/lib/oeqa/runtime/cases/ltp_stress.py +10 -2
index ce6f4bf59d..c902486eab 100644
@@ -60,7 +60,7 @@ class LtpStressBase(OERuntimeTestCase):
 class LtpStressTest(LtpStressBase):
 
     def runltp(self, stress_group):
-            cmd = '/opt/ltp/runltp -f %s -p -q 2>@1 | tee /opt/ltp/results/%s' % (stress_group, stress_group)
+            cmd = 'kirk --run-suite %s --json-report /opt/ltp/results/%s.json -n -d /opt/ltp' % (stress_group, stress_group)
             starttime = time.time()
             (status, output) = self.target.run(cmd)
             endtime = time.time()
@@ -69,8 +69,16 @@ class LtpStressTest(LtpStressBase):
 
             self.extras['ltpstressresult.rawlogs']['log'] = self.extras['ltpstressresult.rawlogs']['log'] + output
 
+            # Copy kirk JSON report from target
+            dst = os.path.join(self.ltptest_log_dir, "%s.json" % stress_group)
+            remote_src = "/opt/ltp/results/%s.json" % stress_group
+            (status, output) = self.target.copyFrom(remote_src, dst, True)
+            if status:
+                msg = 'File could not be copied. Output: %s' % output
+                self.target.logger.warning(msg)
+
             parser = LtpParser()
-            results, sections  = parser.parse(os.path.join(self.ltptest_log_dir, "%s" % stress_group))
+            results, sections  = parser.parse(dst)
 
             runtime = int(endtime-starttime)
             sections['duration'] = runtime
meta/lib/oeqa/utils/logparser.py +18 -33

69 lines not on this page · raw message

meta/recipes-extended/ltp/ltp_20260130.bb +1 -0
index e5da94855f..4183b5d497 100644
@@ -113,6 +113,7 @@ RDEPENDS:${PN} = "\
     net-tools \
     perl \
     python3-core \
+    python3-kirk \
     procps \
     quota \
     unzip \
#5 lore raw

PATCH v2 6/6 oeqa: ltp - add LTP_WORKERS to allow parallelization

In reply to [email protected]

From: Daniel Turull <[email protected]>

Add new configurable variable to execute multiple workers in parallel. kirk adds the option that was not available in runltp.

Tested in qemux86-64 with configuration:

IMAGE_CLASSES += "testimage"
CORE_IMAGE_EXTRA_INSTALL += "ltp openssh"
TEST_SUITES = "ping ssh ltp"
QB_MEM = "32768"
QB_CPU_KVM = "-cpu host -smp 16"
QB_SMP = "-smp 16"
LTP_WORKERS = "16"
IMAGE_ROOTFS_EXTRA_SPACE = "2097152"

bitbake core-image-minimal
bitbake core-image-minimal -c testimage

---
 meta/lib/oeqa/runtime/cases/ltp.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
meta/lib/oeqa/runtime/cases/ltp.py +3 -1
index b855e76907..ae6a275110 100644
@@ -68,7 +68,9 @@ class LtpTest(LtpTestBase):
             # LTP appends to log files, so ensure we start with a clean log
             self.target.deleteFiles("/opt/ltp/results/", "%s.json" % ltp_group)
 
-            cmd = 'kirk --run-suite %s --json-report /opt/ltp/results/%s.json -n -d /opt/ltp --exec-timeout 20m' % (ltp_group, ltp_group)
+            # LTP_WORKERS: number of parallel test workers (default: 1)
+            workers = self.td.get('LTP_WORKERS', '1')
+            cmd = 'kirk --run-suite %s --json-report /opt/ltp/results/%s.json -n -d /opt/ltp --exec-timeout 20m --workers %s' % (ltp_group, ltp_group, workers)
 
             starttime = time.time()
             (status, output) = self.target.run(cmd, timeout=1200)
#6 lore raw

Re: [PATCH v2 6/6] oeqa: ltp - add LTP_WORKERS to allow parallelization

Quoted 16 lines · On Fri, 2026-03-27 at 13:27 +0100, [email protected] wrote:

From: Daniel Turull <[email protected]>

Add new configurable variable to execute multiple workers in parallel. kirk adds the option that was not available in runltp.

Tested in qemux86-64 with configuration:

IMAGE_CLASSES += "testimage"
CORE_IMAGE_EXTRA_INSTALL += "ltp openssh"
TEST_SUITES = "ping ssh ltp"
QB_MEM = "32768"
QB_CPU_KVM = "-cpu host -smp 16"
QB_SMP = "-smp 16"
LTP_WORKERS = "16"
IMAGE_ROOTFS_EXTRA_SPACE = "2097152"

Quoted 7 lines

bitbake core-image-minimal
bitbake core-image-minimal -c testimage

---
 meta/lib/oeqa/runtime/cases/ltp.py | 4 +++-

 1 file changed, 3 insertions(+), 1 deletion(-)

Thanks for looking at this, it is interesting it now has the parallelism support. I'll probably hold off this initially just so that we can transition successfully but it then makes sense to tweak things.

I checked yocto-autobuilder-helper's config.json for the config we use on the autobuilder and it is:

"IMAGE_INSTALL:append = ' ltp'",
"TEST_SUITES = 'ping ssh ltp ltp_compliance'",
"IMAGE_OVERHEAD_FACTOR = '1.0'",
"IMAGE_ROOTFS_EXTRA_SPACE = '1324288'",
"QB_MEM:forcevariable = '-m 1024'"

so we're running with a lot less memory than you configured but you also have a lot more threads!

Quoted 7 lines
meta/lib/oeqa/runtime/cases/ltp.py +0 -0
index b855e76907..ae6a275110 100644
@@ -68,7 +68,9 @@ class LtpTest(LtpTestBase):
              # LTP appends to log files, so ensure we start with a clean log
              self.target.deleteFiles("/opt/ltp/results/", "%s.json" % ltp_group)
Quoted 5 lines


-            cmd = 'kirk --run-suite %s --json-report /opt/ltp/results/%s.json -n -d /opt/ltp --exec-timeout 20m' % (ltp_group, ltp_group) +            # LTP_WORKERS: number of parallel test workers (default: 1)
+            workers = self.td.get('LTP_WORKERS', '1')
+            cmd = 'kirk --run-suite %s --json-report /opt/ltp/results/%s.json -n -d /opt/ltp --exec-timeout 20m --workers %s' % (ltp_group, ltp_group, workers)

We probably need to namespace the variable a little more, such as OEQA_LTP_WORKERS so that people have a hint on where it is used.

Cheers,

Richard

#7 lore raw

RE: [PATCH v2 6/6] oeqa: ltp - add LTP_WORKERS to allow parallelization

Sure, we can keep this for later after the release.
I can have another one with the ltp_groups to be configurable and not hard code in the ltp.py

I'll send a new series for the patches that do not make if for this release.

Daniel

Quoted 76 lines

-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: Friday, 27 March 2026 16:37
To: Daniel Turull <[email protected]>; openembedded- [email protected]
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH v2 6/6] oeqa: ltp - add LTP_WORKERS to allow parallelization

Quoted 16 lines · On Fri, 2026-03-27 at 13:27 +0100, [email protected] wrote:

From: Daniel Turull <[email protected]>

Add new configurable variable to execute multiple workers in parallel. kirk adds the option that was not available in runltp.

Tested in qemux86-64 with configuration:

IMAGE_CLASSES += "testimage"
CORE_IMAGE_EXTRA_INSTALL += "ltp openssh"
TEST_SUITES = "ping ssh ltp"
QB_MEM = "32768"
QB_CPU_KVM = "-cpu host -smp 16"
QB_SMP = "-smp 16"
LTP_WORKERS = "16"
IMAGE_ROOTFS_EXTRA_SPACE = "2097152"

Quoted 7 lines

bitbake core-image-minimal
bitbake core-image-minimal -c testimage

---
 meta/lib/oeqa/runtime/cases/ltp.py | 4 +++-

 1 file changed, 3 insertions(+), 1 deletion(-)

Thanks for looking at this, it is interesting it now has the parallelism support. I'll probably hold off this initially just so that we can transition successfully but it then makes sense to tweak things.

I checked yocto-autobuilder-helper's config.json for the config we use on the autobuilder and it is:

"IMAGE_INSTALL:append = ' ltp'",
"TEST_SUITES = 'ping ssh ltp ltp_compliance'", "IMAGE_OVERHEAD_FACTOR = '1.0'", "IMAGE_ROOTFS_EXTRA_SPACE = '1324288'", "QB_MEM:forcevariable = '- m 1024'"

so we're running with a lot less memory than you configured but you also have a lot more threads!

Quoted 11 lines
meta/lib/oeqa/runtime/cases/ltp.py +0 -0
index b855e76907..ae6a275110 100644
@@ -68,7 +68,9 @@ class LtpTest(LtpTestBase):
              # LTP appends to log files, so ensure we start with a

clean log
             self.target.deleteFiles("/opt/ltp/results/", "%s.json" % ltp_group)

Quoted 9 lines

-            cmd = 'kirk --run-suite %s --json-report /opt/ltp/results/%s.json -n -d /opt/ltp --exec-timeout 20m' % (ltp_group, ltp_group)
+            # LTP_WORKERS: number of parallel test workers (default:
+1)
+            workers = self.td.get('LTP_WORKERS', '1')
+            cmd = 'kirk --run-suite %s --json-report +/opt/ltp/results/%s.json -n -d /opt/ltp --exec-timeout 20m --workers +%s' % (ltp_group, ltp_group, workers)

We probably need to namespace the variable a little more, such as OEQA_LTP_WORKERS so that people have a hint on where it is used.

Cheers,

Richard