ltp: upgrade 20250530 -> 20250930
Hi Richard, all,
[ I accidentally deleted mail thread. Unfortunately I don't see Message-id in web UI [1] therefore I cannot set In-Reply-To:. Due this also any reply to other LTP developers about runltp vs. kirk would not be in the thread ]
Quoted 16 lines
Quoted 15 lines
$ . oe-init-build-env.../build $ bitbake ltp
ERROR: Error importing OE modules: module 'bb.parse' has no attribute 'vardepsexclude'
ERROR: Unable to parse/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.pyTraceback (most recent call last):
File "/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py", line 372, in evallayerid, fragment_name = f.split('/', 1) ^^^^^^^^^^^^^^^^^^^^^^ ValueError: not enough values to unpack (expected 2, got 1)
Quoted 5 lines
Quoted 1 line
FYI I also plan to get rid of some patches posted.
Are you setting
OE_FRAGMENTSto something in a config file? It shouldn't traceback like this so that is a bug but something is triggering it...
No, simple clone and the 2 commands above. I'll later try it again on fresh git clone.
Quoted 2 lines
I did send some questions and had some discussion on kirk a while ago. Quite simply, it isn't useful/interesting to Yocto Project.
I see your points [2] [3].
Quoted 5 lines
What we want to test with is our images and our kernel, as we build it. kirk, as far as I understand it has gone a different route where there isn't really any userspace left and it simply tests against a kernel binary. We'd no longer be testing our build artefacts but some more artificial construct.
I don't understand "any userspace left and it simply tests against a kernel binary". LTP tests are mostly focused on the kernel (+ it's modules). And you can run individual tests by just executing them (+ handle environment variables) or use runltp or use kirk. The executor does not matter that much. In the end we at SUSE also test with LTP our built image :). (LTP is used by mainline folks and by distro folks).
FYI although kirk is meant to be run on the host, doing a different connections, it can also be run on SUT. Sure, there is then python3 dependency on SUT (heavier than shell + it's dependencies), but still kind of runltp replacement.
Quoted 9 lines
We're trying to test what we build. You're trying to test the kernel for regressions. They're two different things.
I totally understand why you've gone that direction with kirk but I also did spell out at the time that it wasn't something which really fits in with the way we run tests, or what we actually aim to test. I was told at the time that basically, nobody is interested in what we want/do.
...
Quoted 7 lines
Quoted 4 lines
I see in
meta/lib/oeqa/runtime/cases/ltp.pythe deprecated/opt/ltp/runltpis still being used. We want to remove it (not sure when, but it will happen sooner or later). Any change somebody would submit a patch to switch to kirk?It is more likely that when you drop runltp, we'll just have to drop ltp. Sorry :(. I did explain this at while ago :/.
It's a question if any of the users really need LTP. If yes, you could vendor runltp.
Or, write a simple script which parses the content of the runtest files and execute them. FYI for part of openSUSE testing we still use our custom openQA runner which does exactly this. It would be very light: -d and -r can be replaced by TMPDIR and LTPROOT, -I is supported by all tests. The only missing thing will be generating of results (if you really need it, I'd recommend kirk and it's JSON output).
The reason we go to kirk + ltx is that in the future we plan to get rid of runtests, replace them with metadata info (that will allow many things [4], but you were Cc at the discussion [5]). Once this happen, runltp (or any custom script / framework runner) will be broken. But this likely take long time.
Quoted 3 lines
I did also flag the issue to our own wider community but nobody has stepped up to do anything about it. I've way too many other things to work on so it is very unlikely I have any time to look at it.
Understand, everybody is busy with the
Kind regards,
Petr
[1] https://lists.openembedded.org/g/openembedded-core/message/224749
[2] https://lore.kernel.org/ltp/c8d4ee181809c4bbf5e21bf355c241eeb540e9a5.camel@linuxfoundation.org/
[3] https://lore.kernel.org/ltp/8043628a6eed94e788f9fedbf6c8b264ebfbae15.camel@linuxfoundation.org/
[4] https://www.youtube.com/watch?v=T1ImuNr9Oxo
[5] https://lore.kernel.org/ltp/ZmbFyjuXndeXCLp8@rei/
Quoted 22 lines · Petr Vorel
Hi Richard, all,
[ I accidentally deleted mail thread. Unfortunately I don't see Message-id in web UI [1] therefore I cannot set In-Reply-To:. Due this also any reply to other LTP developers about runltp vs. kirk would not be in the thread ]
Quoted 15 lines
Quoted 15 lines
$ . oe-init-build-env.../build $ bitbake ltp
ERROR: Error importing OE modules: module 'bb.parse' has no attribute 'vardepsexclude'
ERROR: Unable to parse/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.pyTraceback (most recent call last):
File "/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py", line 372, in eval
layerid,fragment_name=f.split('/', 1)^^^^^^^^^^^^^^^^^^^^^^ ValueError: not enough values to unpack (expected 2, got 1)
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
FYI I also plan to get rid of some patches posted.
Are you setting
OE_FRAGMENTSto something in a config file? It shouldn't traceback like this so that is a bug but something is triggering it...No, simple clone and the 2 commands above. I'll later try it again on fresh git clone.
If you could share the repo/branch you cloned (poky?) I'd appreciate it as I did try and reproduce that and couldn't. I will write some patches to improve the code/error regardless as that much is clear from the failure. It really shouldn't fail like that, it is embarrassing :/.
Quoted 11 lines
Quoted 2 lines
I did send some questions and had some discussion on kirk a while ago. Quite simply, it isn't useful/interesting to Yocto Project.
I see your points [2] [3].
Quoted 5 lines
What we want to test with is our images and our kernel, as we build it. kirk, as far as I understand it has gone a different route where there isn't really any userspace left and it simply tests against a kernel binary. We'd no longer be testing our build artefacts but some more artificial construct.
Quoted 6 lines
I don't understand "any userspace left and it simply tests against a kernel binary". LTP tests are mostly focused on the kernel (+ it's modules). And you can run individual tests by just executing them (+ handle environment variables) or use runltp or use kirk. The executor does not matter that much. In the end we at SUSE also test with LTP our built image :). (LTP is used by mainline folks and by distro folks).
Sorry, I'm not really being clear. The Yocto Project is in the middle of some huge changes and I'm struggling with those. Equally, I did want to at least given some response to your upgrade which is appreciated.
I guess we're one of the few ltp users with a cross environment. We have a way to launch target images under emulation and our own methods for controlling them and transferring files. We do all this without special permissions or access other than ability to use kvm. As such we liked being able to just run runltp on the target in the environment we already have.
kirk, at least as I understand it, wants to do much of this for itself. That means we end up with two ways of running the target emulation which may or may not match. We'd much prefer just having one so we either have issues there or we do not but our test environment is consistent across all tests.
Quoted 26 lines
FYI although kirk is meant to be run on the host, doing a different connections, it can also be run on SUT. Sure, there is then python3 dependency on SUT (heavier than shell + it's dependencies), but still kind of runltp replacement.
Quoted 9 lines
We're trying to test what we build. You're trying to test the kernel for regressions. They're two different things.
I totally understand why you've gone that direction with kirk but I also did spell out at the time that it wasn't something which really fits in with the way we run tests, or what we actually aim to test. I was told at the time that basically, nobody is interested in what we want/do.
...
Quoted 7 lines
Quoted 4 lines
I see in
meta/lib/oeqa/runtime/cases/ltp.pythe deprecated/opt/ltp/runltpis still being used. We want to remove it (not sure when, but it will happen sooner or later). Any change somebody would submit a patch to switch to kirk?It is more likely that when you drop runltp, we'll just have to drop ltp. Sorry :(. I did explain this at while ago :/.
It's a question if any of the users really need LTP. If yes, you could vendor runltp.
Quoted 6 lines
Or, write a simple script which parses the content of the runtest files and execute them. FYI for part of openSUSE testing we still use our custom openQA runner which does exactly this. It would be very light: -d and -r can be replaced by TMPDIR and LTPROOT, -I is supported by all tests. The only missing thing will be generating of results (if you really need it, I'd recommend kirk and it's JSON output).
We'd appreciate and be able to use the json output, we already have json output for most of our other tests. Beyond that, as you say, we don't really need much beyond what runltp does though. kirk brings with it a number of things we definitely do not want (as mentioned above). I don't know if we can avoid those or not.
Quoted 4 lines
The reason we go to kirk + ltx is that in the future we plan to get rid of runtests, replace them with metadata info (that will allow many things [4], but you were Cc at the discussion [5]). Once this happen, runltp (or any custom script / framework runner) will be broken. But this likely take long time.
Right, using a vendored runltp would just buy us a small amount of time so likely isn't a long term viable solution sadly. I'd consider it if it were but it doesn't seem a good investment of time.
Cheers,
Richard
Hi Richard,
Quoted 29 lines · Petr Vorel
Quoted 27 lines
Hi Richard, all,
[ I accidentally deleted mail thread. Unfortunately I don't see Message-id in web UI [1] therefore I cannot set In-Reply-To:. Due this also any reply to other LTP developers about runltp vs. kirk would not be in the thread ]
Quoted 21 lines
Quoted 12 lines
$ . oe-init-build-env.../build $ bitbake ltp
ERROR: Error importing OE modules: module 'bb.parse' has no attribute 'vardepsexclude'
ERROR: Unable to parse/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.pyTraceback (most recent call last):
ďż˝ File
ďż˝ "/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py", ďż˝ line 372, in eval
����� layerid,fragment_name=f.split('/', 1)Quoted 5 lines
����� ^^^^^^^^^^^^^^^^^^^^^^ � ValueError: not enough values to unpack (expected 2, got 1)FYI I also plan to get rid of some patches posted.
Are you setting
OE_FRAGMENTSto something in a config file? It shouldn't traceback like this so that is a bug but something is triggering it...
Quoted 7 lines
Quoted 2 lines
No, simple clone and the 2 commands above. I'll later try it again on fresh git clone.
If you could share the repo/branch you cloned (poky?) I'd appreciate it as I did try and reproduce that and couldn't. I will write some patches to improve the code/error regardless as that much is clear from the failure. It really shouldn't fail like that, it is embarrassing :/.
Master branch from my fork [1] do:
... $ . oe-init-build-env
./build $ bitbake ltp
ERROR: Error importing OE modules: module 'bb.parse' has no attribute 'vardepsexclude'
ERROR: Unable to parse /home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py Traceback (most recent call last):
File "/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py", line 372, in eval
layerid, fragment_name = f.split('/', 1)
^^^^^^^^^^^^^^^^^^^^^^ValueError: not enough values to unpack (expected 2, got 1)
Kind regards,
Petr
Quoted 9 lines · Petr Vorel via lists.openembedded.org
Master branch from my fork [1] do:
... $ . oe-init-build-env
./build $ bitbake ltpERROR: Error importing OE modules: module 'bb.parse' has no attribute 'vardepsexclude'
ERROR: Unable to parse/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.pyTraceback (most recent call last):
File "/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py", line 372, in eval
Quoted 8 lines
layerid, fragment_name = f.split('/', 1) ^^^^^^^^^^^^^^^^^^^^^^ValueError: not enough values to unpack (expected 2, got 1)
Kind regards,
Petr
Which revision/branch of bitbake repo are you using? Note that bitbake and oe-core repositories are tightly coupled, and if branches are different, or branches are same but one is behind the other, you will get errors such as above, e.g. "ERROR: Error importing OE modules:
module 'bb.parse' has no attribute 'vardepsexclude'" is pretty much saying that oe-core is expecting something bitbake on your machine doesn't have.
Alex
Quoted 20 lines · Alexander Kanavin
Quoted 13 lines · Petr Vorel via lists.openembedded.org
Master branch from my fork [1] do:
... $ . oe-init-build-env
./build $ bitbake ltpERROR: Error importing OE modules: module 'bb.parse' has no attribute 'vardepsexclude'
ERROR: Unable to parse/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.pyTraceback (most recent call last):
File "/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py", line 372, in eval layerid,fragment_name=f.split('/', 1)
^^^^^^^^^^^^^^^^^^^^^^
ValueError: not enough values to unpack (expected 2, got 1)Quoted 4 lines
Kind regards,
Petr
Quoted 7 lines
Which revision/branch of bitbake repo are you using? Note that bitbake and oe-core repositories are tightly coupled, and if branches are different, or branches are same but one is behind the other, you will get errors such as above, e.g. "ERROR: Error importing OE modules:
module 'bb.parse' has no attribute 'vardepsexclude'" is pretty much saying that oe-core is expecting something bitbake on your machine doesn't have.
Just to be clear, try cloning git://git.yoctoproject.org/poky which contains both OE-Core and bitbake. This does look like a bitbake/oecore version mismatch. I am curious which bitbake version that is? Was it distro provided?
Cheers,
Richard
Hi Richard, all,
Quoted 30 lines · Richard Purdie
Quoted 20 lines · Alexander Kanavin
Quoted 9 lines · Petr Vorel via lists.openembedded.org
Master branch from my fork [1] do:
... $ . oe-init-build-env
./build $ bitbake ltpERROR: Error importing OE modules: module 'bb.parse' has no attribute 'vardepsexclude'
ERROR: Unable to parse/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.pyTraceback (most recent call last):
File "/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py", line 372, in evalQuoted 8 lines
layerid, fragment_name = f.split('/', 1) ^^^^^^^^^^^^^^^^^^^^^^ValueError: not enough values to unpack (expected 2, got 1)
Kind regards,
PetrQuoted 7 lines
Which revision/branch of bitbake repo are you using? Note that bitbake and oe-core repositories are tightly coupled, and if branches are different, or branches are same but one is behind the other, you will get errors such as above, e.g. "ERROR: Error importing OE modules:
module 'bb.parse' has no attribute 'vardepsexclude'" is pretty much saying that oe-core is expecting something bitbake on your machine doesn't have.
Quoted 4 lines
Just to be clear, try cloning
git://git.yoctoproject.org/pokywhich contains both OE-Core and bitbake. This does look like a bitbake/oecore version mismatch. I am curious which bitbake version that is? Was it distro provided?
I'm sorry for wasting your time, my fault. I usedgit://git.openembedded.org/openembedded-core.git withgit://git.openembedded.org/bitbake.git, which was outdated 2024-09-25:29e67acb8 ("tests/parse: Add test for unclosed functions") I forget there are two repos. But lesson learned :).
And thanks for pointing out poky repo.
Kind regards,
Petr
Quoted 3 lines
Cheers,
Richard
Quoted 6 lines · Petr Vorel
I'm sorry for wasting your time, my fault. I used
git://git.openembedded.org/openembedded-core.gitwithgit://git.openembedded.org/bitbake.git, which was outdated 2024-09-25:29e67acb8("tests/parse: Add test for unclosed functions") I forget there are two repos. But lesson learned :).
And thanks for pointing out poky repo.
As chance would have it, poky repo is soon discontinued, and everyone will be using separate repos. Bitbake-setup will help somewhat with avoiding the mismatches, but bitbake should probably have checks that it isn't used with a wildly mismatching layer metadata.
Maybe add an equivalent of LAYERSERIES_COMPAT declaration to bitbake itself, so it doesn't just check that the layers are matching, but also that they match bitbake?
Alex
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 8 lines · Alexander Kanavin via lists.openembedded.org
As chance would have it, poky repo is soon discontinued, and everyone will be using separate repos. Bitbake-setup will help somewhat with avoiding the mismatches, but bitbake should probably have checks that it isn't used with a wildly mismatching layer metadata.
Maybe add an equivalent of
LAYERSERIES_COMPATdeclaration to bitbake itself, so it doesn't just check that the layers are matching, but also that they match bitbake?
Ah, there is such a check in meta/classes-global/sanity.bbclass:
# Check the bitbake version meets minimum requirements minversion = d.getVar('BB_MIN_VERSION')
if bb.utils.vercmp_string_op(bb.__version__, minversion, "<"):
status.addresult('Bitbake version %s is required and version
%s was found\n' % (minversion, bb.__version__))
The check seems to be happening later than the parse problem you've encountered (which is triggered by addpylib statement in meta/conf/layer.conf), and so it would be good to do something similar as early as possible. I'll prototype a patch.
Alex
Hi Richard, all,
[ I realized that we went quite far to kirk topic, which would be interesting to Cyril, Andrea and possible other people developers or users, therefore Cc them. Replying to this thread:
https://lists.openembedded.org/g/openembedded-core/message/224816
+ links to replies from Richardo previously in LTP ML
https://lore.kernel.org/ltp/c8d4ee181809c4bbf5e21bf355c241eeb540e9a5.camel@linuxfoundation.org/
https://lore.kernel.org/ltp/8043628a6eed94e788f9fedbf6c8b264ebfbae15.camel@linuxfoundation.org/
NOTE: Cc requires subscription.
]
I added few final points below.
Quoted 1 line · Petr Vorel
...
Quoted 68 lines
Quoted 11 lines
Quoted 2 lines
I did send some questions and had some discussion on kirk a while ago. Quite simply, it isn't useful/interesting to Yocto Project.
I see your points [2] [3].
Quoted 5 lines
What we want to test with is our images and our kernel, as we build it. kirk, as far as I understand it has gone a different route where there isn't really any userspace left and it simply tests against a kernel binary. We'd no longer be testing our build artefacts but some more artificial construct.
Quoted 6 lines
I don't understand "any userspace left and it simply tests against a kernel binary". LTP tests are mostly focused on the kernel (+ it's modules). And you can run individual tests by just executing them (+ handle environment variables) or use runltp or use kirk. The executor does not matter that much. In the end we at SUSE also test with LTP our built image :). (LTP is used by mainline folks and by distro folks).
Sorry, I'm not really being clear. The Yocto Project is in the middle of some huge changes and I'm struggling with those. Equally, I did want to at least given some response to your upgrade which is appreciated.
I guess we're one of the few ltp users with a cross environment. We have a way to launch target images under emulation and our own methods for controlling them and transferring files. We do all this without special permissions or access other than ability to use kvm. As such we liked being able to just run runltp on the target in the environment we already have.
kirk, at least as I understand it, wants to do much of this for itself. That means we end up with two ways of running the target emulation which may or may not match. We'd much prefer just having one so we either have issues there or we do not but our test environment is consistent across all tests.
Quoted 26 lines
FYI although kirk is meant to be run on the host, doing a different connections, it can also be run on SUT. Sure, there is then python3 dependency on SUT (heavier than shell + it's dependencies), but still kind of runltp replacement.
Quoted 8 lines
We're trying to test what we build. You're trying to test the kernel for regressions. They're two different things.
I totally understand why you've gone that direction with kirk but I also did spell out at the time that it wasn't something which really fits in with the way we run tests, or what we actually aim to test. I was told at the time that basically, nobody is interested in what we want/do.
...
Quoted 7 lines
Quoted 4 lines
I see in
meta/lib/oeqa/runtime/cases/ltp.pythe deprecated/opt/ltp/runltpis still being used. We want to remove it (not sure when, but it will happen sooner or later). Any change somebody would submit a patch to switch to kirk?It is more likely that when you drop runltp, we'll just have to drop ltp. Sorry :(. I did explain this at while ago :/.
It's a question if any of the users really need LTP. If yes, you could vendor runltp.
Quoted 6 lines
Or, write a simple script which parses the content of the runtest files and execute them. FYI for part of openSUSE testing we still use our custom openQA runner which does exactly this. It would be very light: -d and -r can be replaced by TMPDIR and LTPROOT, -I is supported by all tests. The only missing thing will be generating of results (if you really need it, I'd recommend kirk and it's JSON output).
Quoted 5 lines
We'd appreciate and be able to use the json output, we already have json output for most of our other tests. Beyond that, as you say, we don't really need much beyond what runltp does though. kirk brings with it a number of things we definitely do not want (as mentioned above). I don't know if we can avoid those or not.
As I wrote previously, kirk is possible to be used as nearly drop-in runltp replacement run on the SUT. It's not the intended use-case (we aim it to be run on host), but still supported use case.
Quoted 8 lines
Quoted 4 lines
The reason we go to kirk + ltx is that in the future we plan to get rid of runtests, replace them with metadata info (that will allow many things [4], but you were Cc at the discussion [5]). Once this happen, runltp (or any custom script / framework runner) will be broken. But this likely take long time.
Right, using a vendored runltp would just buy us a small amount of time so likely isn't a long term viable solution sadly. I'd consider it if it were but it doesn't seem a good investment of time.
I would say adopting kirk in minimalistic way (or identify what you need and it's still missing) would be sure safer in a long term.
Kind regards,
Petr
Quoted 3 lines
Cheers,
Richard