Skip to contentSkip to conversation

ltp: upgrade 20250530 -> 20250930

PATCH Petr Vorel 3 participants 9 messages Oct 13, 2025 – Oct 21, 2025 1 message not in the archive

lore
Share & tools
lore

Petr Vorel

author
#0 lore raw

Hi Richard, all,

[ I accidentally deleted mail thread. Unfortunately I don't see Message-id in web UI [1] therefore I cannot set In-Reply-To:. Due this also any reply to other LTP developers about runltp vs. kirk would not be in the thread ]

Quoted 16 lines
Quoted 15 lines
$ . oe-init-build-env

.../build $ bitbake ltp
ERROR: Error importing OE modules: module 'bb.parse' has no attribute 'vardepsexclude'
ERROR: Unable to parse /home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py Traceback (most recent call last):
File "/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py", line 372, in eval

      layerid, fragment_name = f.split('/', 1)
	      ^^^^^^^^^^^^^^^^^^^^^^
		  ValueError: not enough values to unpack (expected 2, got 1)
Quoted 5 lines
Quoted 1 line

FYI I also plan to get rid of some patches posted.

Are you setting OE_FRAGMENTS to something in a config file? It shouldn't traceback like this so that is a bug but something is triggering it...

No, simple clone and the 2 commands above. I'll later try it again on fresh git clone.

Quoted 2 lines

I did send some questions and had some discussion on kirk a while ago. Quite simply, it isn't useful/interesting to Yocto Project.

I see your points [2] [3].

Quoted 5 lines

What we want to test with is our images and our kernel, as we build it. kirk, as far as I understand it has gone a different route where there isn't really any userspace left and it simply tests against a kernel binary. We'd no longer be testing our build artefacts but some more artificial construct.

I don't understand "any userspace left and it simply tests against a kernel binary". LTP tests are mostly focused on the kernel (+ it's modules). And you can run individual tests by just executing them (+ handle environment variables) or use runltp or use kirk. The executor does not matter that much. In the end we at SUSE also test with LTP our built image :). (LTP is used by mainline folks and by distro folks).

FYI although kirk is meant to be run on the host, doing a different connections, it can also be run on SUT. Sure, there is then python3 dependency on SUT (heavier than shell + it's dependencies), but still kind of runltp replacement.

Quoted 9 lines

We're trying to test what we build. You're trying to test the kernel for regressions. They're two different things.

I totally understand why you've gone that direction with kirk but I also did spell out at the time that it wasn't something which really fits in with the way we run tests, or what we actually aim to test. I was told at the time that basically, nobody is interested in what we want/do.

...

Quoted 7 lines
Quoted 4 lines

I see in meta/lib/oeqa/runtime/cases/ltp.py the deprecated /opt/ltp/runltp is still being used. We want to remove it (not sure when, but it will happen sooner or later). Any change somebody would submit a patch to switch to kirk?

It is more likely that when you drop runltp, we'll just have to drop ltp. Sorry :(. I did explain this at while ago :/.

It's a question if any of the users really need LTP. If yes, you could vendor runltp.

Or, write a simple script which parses the content of the runtest files and execute them. FYI for part of openSUSE testing we still use our custom openQA runner which does exactly this. It would be very light: -d and -r can be replaced by TMPDIR and LTPROOT, -I is supported by all tests. The only missing thing will be generating of results (if you really need it, I'd recommend kirk and it's JSON output).

The reason we go to kirk + ltx is that in the future we plan to get rid of runtests, replace them with metadata info (that will allow many things [4], but you were Cc at the discussion [5]). Once this happen, runltp (or any custom script / framework runner) will be broken. But this likely take long time.

Quoted 3 lines

I did also flag the issue to our own wider community but nobody has stepped up to do anything about it. I've way too many other things to work on so it is very unlikely I have any time to look at it.

Understand, everybody is busy with the

Kind regards,
Petr

[1] https://lists.openembedded.org/g/openembedded-core/message/224749
[2] https://lore.kernel.org/ltp/c8d4ee181809c4bbf5e21bf355c241eeb540e9a5.camel@linuxfoundation.org/
[3] https://lore.kernel.org/ltp/8043628a6eed94e788f9fedbf6c8b264ebfbae15.camel@linuxfoundation.org/
[4] https://www.youtube.com/watch?v=T1ImuNr9Oxo
[5] https://lore.kernel.org/ltp/ZmbFyjuXndeXCLp8@rei/

#1 lore raw
Quoted 22 lines · Petr Vorel

Hi Richard, all,

[ I accidentally deleted mail thread. Unfortunately I don't see Message-id in web UI [1] therefore I cannot set In-Reply-To:. Due this also any reply to other LTP developers about runltp vs. kirk would not be in the thread ]

Quoted 15 lines
Quoted 15 lines
$ . oe-init-build-env

.../build $ bitbake ltp
ERROR: Error importing OE modules: module 'bb.parse' has no attribute 'vardepsexclude'
ERROR: Unable to parse /home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py Traceback (most recent call last):
  File "/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py", line 372, in eval
      layerid, fragment_name = f.split('/', 1)

	      ^^^^^^^^^^^^^^^^^^^^^^
		  ValueError: not enough values to unpack (expected 2, got 1)
Quoted 8 lines
Quoted 5 lines
Quoted 1 line

FYI I also plan to get rid of some patches posted.

Are you setting OE_FRAGMENTS to something in a config file? It shouldn't traceback like this so that is a bug but something is triggering it...

No, simple clone and the 2 commands above. I'll later try it again on fresh git clone.

If you could share the repo/branch you cloned (poky?) I'd appreciate it as I did try and reproduce that and couldn't. I will write some patches to improve the code/error regardless as that much is clear from the failure. It really shouldn't fail like that, it is embarrassing :/.

Quoted 11 lines
Quoted 2 lines

I did send some questions and had some discussion on kirk a while ago. Quite simply, it isn't useful/interesting to Yocto Project.

I see your points [2] [3].

Quoted 5 lines

What we want to test with is our images and our kernel, as we build it. kirk, as far as I understand it has gone a different route where there isn't really any userspace left and it simply tests against a kernel binary. We'd no longer be testing our build artefacts but some more artificial construct.

Quoted 6 lines

I don't understand "any userspace left and it simply tests against a kernel binary". LTP tests are mostly focused on the kernel (+ it's modules). And you can run individual tests by just executing them (+ handle environment variables) or use runltp or use kirk. The executor does not matter that much. In the end we at SUSE also test with LTP our built image :). (LTP is used by mainline folks and by distro folks).

Sorry, I'm not really being clear. The Yocto Project is in the middle of some huge changes and I'm struggling with those. Equally, I did want to at least given some response to your upgrade which is appreciated.

I guess we're one of the few ltp users with a cross environment. We have a way to launch target images under emulation and our own methods for controlling them and transferring files. We do all this without special permissions or access other than ability to use kvm. As such we liked being able to just run runltp on the target in the environment we already have.

kirk, at least as I understand it, wants to do much of this for itself. That means we end up with two ways of running the target emulation which may or may not match. We'd much prefer just having one so we either have issues there or we do not but our test environment is consistent across all tests.

Quoted 26 lines

FYI although kirk is meant to be run on the host, doing a different connections, it can also be run on SUT. Sure, there is then python3 dependency on SUT (heavier than shell + it's dependencies), but still kind of runltp replacement.

Quoted 9 lines

We're trying to test what we build. You're trying to test the kernel for regressions. They're two different things.

I totally understand why you've gone that direction with kirk but I also did spell out at the time that it wasn't something which really fits in with the way we run tests, or what we actually aim to test. I was told at the time that basically, nobody is interested in what we want/do.

...

Quoted 7 lines
Quoted 4 lines

I see in meta/lib/oeqa/runtime/cases/ltp.py the deprecated /opt/ltp/runltp is still being used. We want to remove it (not sure when, but it will happen sooner or later). Any change somebody would submit a patch to switch to kirk?

It is more likely that when you drop runltp, we'll just have to drop ltp. Sorry :(. I did explain this at while ago :/.

It's a question if any of the users really need LTP. If yes, you could vendor runltp.

Quoted 6 lines

Or, write a simple script which parses the content of the runtest files and execute them. FYI for part of openSUSE testing we still use our custom openQA runner which does exactly this. It would be very light: -d and -r can be replaced by TMPDIR and LTPROOT, -I is supported by all tests. The only missing thing will be generating of results (if you really need it, I'd recommend kirk and it's JSON output).

We'd appreciate and be able to use the json output, we already have json output for most of our other tests. Beyond that, as you say, we don't really need much beyond what runltp does though. kirk brings with it a number of things we definitely do not want (as mentioned above). I don't know if we can avoid those or not.

Quoted 4 lines

The reason we go to kirk + ltx is that in the future we plan to get rid of runtests, replace them with metadata info (that will allow many things [4], but you were Cc at the discussion [5]). Once this happen, runltp (or any custom script / framework runner) will be broken. But this likely take long time.

Right, using a vendored runltp would just buy us a small amount of time so likely isn't a long term viable solution sadly. I'd consider it if it were but it doesn't seem a good investment of time.

Cheers,

Richard

author
#2 lore raw

Hi Richard,

Quoted 29 lines · Petr Vorel
Quoted 27 lines

Hi Richard, all,

[ I accidentally deleted mail thread. Unfortunately I don't see Message-id in web UI [1] therefore I cannot set In-Reply-To:. Due this also any reply to other LTP developers about runltp vs. kirk would not be in the thread ]

Quoted 21 lines
Quoted 12 lines
$ . oe-init-build-env

.../build $ bitbake ltp
ERROR: Error importing OE modules: module 'bb.parse' has no attribute 'vardepsexclude'
ERROR: Unable to parse /home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py Traceback (most recent call last):
ďż˝ File
ďż˝ "/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py", ďż˝ line 372, in eval
����� layerid, fragment_name = f.split('/', 1)

Quoted 5 lines
	����� ^^^^^^^^^^^^^^^^^^^^^^
		ďż˝ ValueError: not enough values to unpack (expected 2, got 1)

FYI I also plan to get rid of some patches posted.

Are you setting OE_FRAGMENTS to something in a config file? It shouldn't traceback like this so that is a bug but something is triggering it...

Quoted 7 lines
Quoted 2 lines

No, simple clone and the 2 commands above. I'll later try it again on fresh git clone.

If you could share the repo/branch you cloned (poky?) I'd appreciate it as I did try and reproduce that and couldn't. I will write some patches to improve the code/error regardless as that much is clear from the failure. It really shouldn't fail like that, it is embarrassing :/.

Master branch from my fork [1] do:

... $ . oe-init-build-env
./build $ bitbake ltp

ERROR: Error importing OE modules: module 'bb.parse' has no attribute 'vardepsexclude'
ERROR: Unable to parse /home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py Traceback (most recent call last):
File "/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py", line 372, in eval

    layerid, fragment_name = f.split('/', 1)
    ^^^^^^^^^^^^^^^^^^^^^^

ValueError: not enough values to unpack (expected 2, got 1)

Kind regards,
Petr

[1] https://github.com/pevik/openembedded-core

#3 lore raw
Quoted 9 lines · Petr Vorel via lists.openembedded.org

Master branch from my fork [1] do:

... $ . oe-init-build-env
./build $ bitbake ltp

ERROR: Error importing OE modules: module 'bb.parse' has no attribute 'vardepsexclude'
ERROR: Unable to parse /home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py Traceback (most recent call last):
File "/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py", line 372, in eval

Quoted 8 lines
    layerid, fragment_name = f.split('/', 1)
    ^^^^^^^^^^^^^^^^^^^^^^

ValueError: not enough values to unpack (expected 2, got 1)

Kind regards,
Petr

[1] https://github.com/pevik/openembedded-core

Which revision/branch of bitbake repo are you using? Note that bitbake and oe-core repositories are tightly coupled, and if branches are different, or branches are same but one is behind the other, you will get errors such as above, e.g. "ERROR: Error importing OE modules:
module 'bb.parse' has no attribute 'vardepsexclude'" is pretty much saying that oe-core is expecting something bitbake on your machine doesn't have.

Alex

#4 lore raw
Quoted 20 lines · Alexander Kanavin
Quoted 13 lines · Petr Vorel via lists.openembedded.org

Master branch from my fork [1] do:

... $ . oe-init-build-env
./build $ bitbake ltp

ERROR: Error importing OE modules: module 'bb.parse' has no attribute 'vardepsexclude'
ERROR: Unable to parse /home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py Traceback (most recent call last):
  File "/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py", line 372, in eval layerid, fragment_name = f.split('/', 1)
    ^^^^^^^^^^^^^^^^^^^^^^
ValueError: not enough values to unpack (expected 2, got 1)

Quoted 4 lines

Kind regards,
Petr

[1] https://github.com/pevik/openembedded-core

Quoted 7 lines

Which revision/branch of bitbake repo are you using? Note that bitbake and oe-core repositories are tightly coupled, and if branches are different, or branches are same but one is behind the other, you will get errors such as above, e.g. "ERROR: Error importing OE modules:
module 'bb.parse' has no attribute 'vardepsexclude'" is pretty much saying that oe-core is expecting something bitbake on your machine doesn't have.

Just to be clear, try cloning git://git.yoctoproject.org/poky which contains both OE-Core and bitbake. This does look like a bitbake/oecore version mismatch. I am curious which bitbake version that is? Was it distro provided?

Cheers,

Richard

author
#5 lore raw

Hi Richard, all,

Quoted 30 lines · Richard Purdie
Quoted 20 lines · Alexander Kanavin
Quoted 9 lines · Petr Vorel via lists.openembedded.org

Master branch from my fork [1] do:

... $ . oe-init-build-env
./build $ bitbake ltp

ERROR: Error importing OE modules: module 'bb.parse' has no attribute 'vardepsexclude'
ERROR: Unable to parse /home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py Traceback (most recent call last):
File "/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py", line 372, in eval

Quoted 8 lines
    layerid, fragment_name = f.split('/', 1)
    ^^^^^^^^^^^^^^^^^^^^^^

ValueError: not enough values to unpack (expected 2, got 1)

Kind regards,
Petr

[1] https://github.com/pevik/openembedded-core

Quoted 7 lines

Which revision/branch of bitbake repo are you using? Note that bitbake and oe-core repositories are tightly coupled, and if branches are different, or branches are same but one is behind the other, you will get errors such as above, e.g. "ERROR: Error importing OE modules:
module 'bb.parse' has no attribute 'vardepsexclude'" is pretty much saying that oe-core is expecting something bitbake on your machine doesn't have.

Quoted 4 lines

Just to be clear, try cloning git://git.yoctoproject.org/poky which contains both OE-Core and bitbake. This does look like a bitbake/oecore version mismatch. I am curious which bitbake version that is? Was it distro provided?

I'm sorry for wasting your time, my fault. I used
git://git.openembedded.org/openembedded-core.git with
git://git.openembedded.org/bitbake.git, which was outdated 2024-09-25:
29e67acb8 ("tests/parse: Add test for unclosed functions") I forget there are two repos. But lesson learned :).
And thanks for pointing out poky repo.

Kind regards,
Petr

Quoted 3 lines

Cheers,

Richard

#6 lore raw
Quoted 6 lines · Petr Vorel

I'm sorry for wasting your time, my fault. I used
git://git.openembedded.org/openembedded-core.git with
git://git.openembedded.org/bitbake.git, which was outdated 2024-09-25:
29e67acb8 ("tests/parse: Add test for unclosed functions") I forget there are two repos. But lesson learned :).
And thanks for pointing out poky repo.

As chance would have it, poky repo is soon discontinued, and everyone will be using separate repos. Bitbake-setup will help somewhat with avoiding the mismatches, but bitbake should probably have checks that it isn't used with a wildly mismatching layer metadata.

Maybe add an equivalent of LAYERSERIES_COMPAT declaration to bitbake itself, so it doesn't just check that the layers are matching, but also that they match bitbake?

Alex

A message that is not in the archive (sent off-list or not yet received): [email protected]

#8 lore raw
Quoted 8 lines · Alexander Kanavin via lists.openembedded.org

As chance would have it, poky repo is soon discontinued, and everyone will be using separate repos. Bitbake-setup will help somewhat with avoiding the mismatches, but bitbake should probably have checks that it isn't used with a wildly mismatching layer metadata.

Maybe add an equivalent of LAYERSERIES_COMPAT declaration to bitbake itself, so it doesn't just check that the layers are matching, but also that they match bitbake?

Ah, there is such a check in meta/classes-global/sanity.bbclass:

# Check the bitbake version meets minimum requirements minversion = d.getVar('BB_MIN_VERSION')
if bb.utils.vercmp_string_op(bb.__version__, minversion, "<"):
status.addresult('Bitbake version %s is required and version
%s was found\n' % (minversion, bb.__version__))

The check seems to be happening later than the parse problem you've encountered (which is triggered by addpylib statement in meta/conf/layer.conf), and so it would be good to do something similar as early as possible. I'll prototype a patch.

Alex

author
#9 lore raw
In reply to Richard Purdie

Hi Richard, all,

[ I realized that we went quite far to kirk topic, which would be interesting to Cyril, Andrea and possible other people developers or users, therefore Cc them. Replying to this thread:
https://lists.openembedded.org/g/openembedded-core/message/224816
+ links to replies from Richardo previously in LTP ML
https://lore.kernel.org/ltp/c8d4ee181809c4bbf5e21bf355c241eeb540e9a5.camel@linuxfoundation.org/
https://lore.kernel.org/ltp/8043628a6eed94e788f9fedbf6c8b264ebfbae15.camel@linuxfoundation.org/
NOTE: Cc requires subscription.
]

I added few final points below.

Quoted 1 line · Petr Vorel

...

Quoted 68 lines
Quoted 11 lines
Quoted 2 lines

I did send some questions and had some discussion on kirk a while ago. Quite simply, it isn't useful/interesting to Yocto Project.

I see your points [2] [3].

Quoted 5 lines

What we want to test with is our images and our kernel, as we build it. kirk, as far as I understand it has gone a different route where there isn't really any userspace left and it simply tests against a kernel binary. We'd no longer be testing our build artefacts but some more artificial construct.

Quoted 6 lines

I don't understand "any userspace left and it simply tests against a kernel binary". LTP tests are mostly focused on the kernel (+ it's modules). And you can run individual tests by just executing them (+ handle environment variables) or use runltp or use kirk. The executor does not matter that much. In the end we at SUSE also test with LTP our built image :). (LTP is used by mainline folks and by distro folks).

Sorry, I'm not really being clear. The Yocto Project is in the middle of some huge changes and I'm struggling with those. Equally, I did want to at least given some response to your upgrade which is appreciated.

I guess we're one of the few ltp users with a cross environment. We have a way to launch target images under emulation and our own methods for controlling them and transferring files. We do all this without special permissions or access other than ability to use kvm. As such we liked being able to just run runltp on the target in the environment we already have.

kirk, at least as I understand it, wants to do much of this for itself. That means we end up with two ways of running the target emulation which may or may not match. We'd much prefer just having one so we either have issues there or we do not but our test environment is consistent across all tests.

Quoted 26 lines

FYI although kirk is meant to be run on the host, doing a different connections, it can also be run on SUT. Sure, there is then python3 dependency on SUT (heavier than shell + it's dependencies), but still kind of runltp replacement.

Quoted 8 lines

We're trying to test what we build. You're trying to test the kernel for regressions. They're two different things.

I totally understand why you've gone that direction with kirk but I also did spell out at the time that it wasn't something which really fits in with the way we run tests, or what we actually aim to test. I was told at the time that basically, nobody is interested in what we want/do.

...

Quoted 7 lines
Quoted 4 lines

I see in meta/lib/oeqa/runtime/cases/ltp.py the deprecated /opt/ltp/runltp is still being used. We want to remove it (not sure when, but it will happen sooner or later). Any change somebody would submit a patch to switch to kirk?

It is more likely that when you drop runltp, we'll just have to drop ltp. Sorry :(. I did explain this at while ago :/.

It's a question if any of the users really need LTP. If yes, you could vendor runltp.

Quoted 6 lines

Or, write a simple script which parses the content of the runtest files and execute them. FYI for part of openSUSE testing we still use our custom openQA runner which does exactly this. It would be very light: -d and -r can be replaced by TMPDIR and LTPROOT, -I is supported by all tests. The only missing thing will be generating of results (if you really need it, I'd recommend kirk and it's JSON output).

Quoted 5 lines

We'd appreciate and be able to use the json output, we already have json output for most of our other tests. Beyond that, as you say, we don't really need much beyond what runltp does though. kirk brings with it a number of things we definitely do not want (as mentioned above). I don't know if we can avoid those or not.

As I wrote previously, kirk is possible to be used as nearly drop-in runltp replacement run on the SUT. It's not the intended use-case (we aim it to be run on host), but still supported use case.

Quoted 8 lines
Quoted 4 lines

The reason we go to kirk + ltx is that in the future we plan to get rid of runtests, replace them with metadata info (that will allow many things [4], but you were Cc at the discussion [5]). Once this happen, runltp (or any custom script / framework runner) will be broken. But this likely take long time.

Right, using a vendored runltp would just buy us a small amount of time so likely isn't a long term viable solution sadly. I'd consider it if it were but it doesn't seem a good investment of time.

I would say adopting kirk in minimalistic way (or identify what you need and it's still missing) would be sure safer in a long term.

Kind regards,
Petr

Quoted 3 lines

Cheers,

Richard