runltp-ng and possible ltp-pan deprecation
Hi!
As you may have know some time ago I have started to work on runltp-ng a script to replace runltp and ltp-pan.
The new executor is different in several key aspects, for example:
- it can and is supposed to run on a different machine from the system under test (although it still supports running on the same machine)
- there are different backends, it can run tests locally, over ssh, or start a VM with qemu
- produces different output formats currently html or json
We have been using runltp-ng in SUSE for a while and it works for us quite well and I think that we need a larger user base now to iron out all the quirks. So unless someone strongly disagrees I think that it's time to make it officialy supported test executor for LTP.
I would like to eventually get rid of runltp and ltp-pan, but if that happens it should be done once everyone switched over to a different solution and it's probably still at least a year or two from now.
To remind you the code still lives in my private repository at:
https://github.com/metan-ucw/runltp-ng
For a start I would like to move the code somewhere to the linux-test-project github just to make it more visible. Does everyone agree to that?
Also there are two different options how to do that. Either we put it into runltp-ng directory to the LTP git root or we may as well create a separate repository under the linux-test-project and make it a ltp git submodule. Any preferencies to that?
Signature · 2 lines
Cyril Hrubis [email protected]
Quoted 28 lines
-----Original Message-----
From: ltp <[email protected]> On Behalf Of Cyril HrubisHi!
As you may have know some time ago I have started to work on runltp-ng a script to replace runltp and ltp-pan.The new executor is different in several key aspects, for example:
- it can and is supposed to run on a different machine from the system under test (although it still supports running on the same machine)
- there are different backends, it can run tests locally, over ssh, or start a VM with qemu
- produces different output formats currently html or json
We have been using runltp-ng in SUSE for a while and it works for us quite well and I think that we need a larger user base now to iron out all the quirks. So unless someone strongly disagrees I think that it's time to make it officialy supported test executor for LTP.
I would like to eventually get rid of runltp and ltp-pan, but if that happens it should be done once everyone switched over to a different solution and it's probably still at least a year or two from now.
Quoted 7 lines
To remind you the code still lives in my private repository at:
https://github.com/metan-ucw/runltp-ng
For a start I would like to move the code somewhere to the linux-test-project github just to make it more visible. Does everyone agree to that?
Sounds great to me! I'm very interested in the new functionality.
It sounds like I might be interested in adding a backend for running tests over a Fuego transport. Fuego mostly ends up using ssh, but it provides other transports as well, so I'll be interested to see what your backend API for test-running looks like.
Quoted 5 lines
Also there are two different options how to do that. Either we put it into runltp-ng directory to the LTP git root or we may as well create a separate repository under the linux-test-project and make it a ltp git submodule. Any preferencies to that?
If this is intended to be run with LTP only, then my preference would be to put it into the LTP repository.
-- Tim
Hi!
Quoted 10 lines
Quoted 3 lines
For a start I would like to move the code somewhere to the linux-test-project github just to make it more visible. Does everyone agree to that?
Sounds great to me! I'm very interested in the new functionality.
It sounds like I might be interested in adding a backend for running tests over a Fuego transport. Fuego mostly ends up using ssh, but it provides other transports as well, so I'll be interested to see what your backend API for test-running looks like.
Backend is basically implemented with these functions:
* foo_init() that parses backend specific parameters and prepares a handle that is used later on
* foo_start() that boots machine, connects to SUT, ...
This gets SUT into a state where we can execute commands on (remote) shell and stores pair of file descriptors to the backend handle
for qemu it boots the machine and logs in
for ssh it connects to remote machine
for sh (local execution) it just forks shell
* foo_stop() that does the oposite of foo_start()
Closes connection, poweroff qemu OS, etc.
* foo_force_stop() that attempts to force stop the machine after it stopped responding so that we can bring it up again and continue with next test
for qemu it just kills the process, we do have overlay for the disks in place, so we always start with a clean state
for ssh it's expected that user supplies a script
- when I was testing the test runner I had a script that toggled relay to reset the machine
- for public cloud testing we pass a short script that requests the machine to be rebooted
If machine crashes we do force_stop() followed by a start() to bring it back to a defined state.
Everything else is build on a top of these functions. The testcases are executed using a (remote) shell and the output is parsed, which is not optimal for a several reasons and may change later on. I guess that the general API will not change in this case, the difference would be that once machine starts we would start a binary and talk to it via the file descriptors instead of shell.
We are also working on parallel test execution, which will require such binary running on the SUT that will dispatch testcases and report results, but that haven't been finished yet.
Also in the short term we would like to make use of the metadata.json produced by LTP build to make smarter decisions, probably easiest and quite usefull would be adjusting timeout expectations. LTP generally consists of two types of tests, most of the syscall test are fast and take only miliseconds, then there are tests that run for minutes or even scale its runtime based on test timeout. And this is all multiplied by tests that do several iterations, for instance for each filesystem. I really need to sit down and think how all this should work toghether.
Signature · 2 lines
Cyril Hrubis [email protected]
Hi Cyril,
Thank you for bringing the discussion to upstream runltp-ng.
I'd prefer to add the project to ltp git, in runltp-ng/ in root directory. And add either a symlink runltp-ng/runltp-ng to the root directory or have a shell script which runs it:
perl -I$(readlink -f $0)/runltp-ng $(readlink -f $0)/runltp-ng/runltp-ng $@
Kind regards,
Petr
Quoted 5 lines · Petr Vorel
Hi Cyril,
Thank you for bringing the discussion to upstream runltp-ng.
I'd prefer to add the project to ltp git, in runltp-ng/ in root directory.
+1
I also prefer to add it into ltp directly.
Best Regards,
Xiao Yang
Quoted 8 lines
And add either a symlink runltp-ng/runltp-ng to the root directory or have a shell script which runs it:
perl -I$(readlink -f $0)/runltp-ng $(readlink -f $0)/runltp-ng/runltp-ng $@Kind regards,
Petr
Quoted 6 lines · Petr Vorel
Hi Cyril,
Thank you for bringing the discussion to upstream runltp-ng.
I'd prefer to add the project to ltp git
I agree with that too.
Also if we can seamlessly move from runltp to runltp-ng, since we don't want the behavior to change.
Or we need to support both for a release or two and then remove runltp.
Cheers,
Anders
Quoted 11 lines
, in runltp-ng/ in root directory.
And add either a symlink runltp-ng/runltp-ng to the root directory or have a shell script which runs it:
perl -I$(readlink -f $0)/runltp-ng $(readlink -f $0)/runltp-ng/runltp-ng $@Kind regards,
Petr--
Mailing list info: https://lists.linux.it/listinfo/ltp
Hi!
Quoted 3 lines
Also if we can seamlessly move from runltp to runltp-ng, since we don't want the behavior to change.
Or we need to support both for a release or two and then remove runltp.
Do not worry, I do plan to carry both for more than one release in order to give everyone enough time for the transition.
Signature · 2 lines
Cyril Hrubis [email protected]
Hi Cyril,
Quoted 7 lines · Cyril Hrubis
Hi!
As you may have know some time ago I have started to work on runltp-ng a script to replace runltp and ltp-pan.The new executor is different in several key aspects, for example:
Quoted 8 lines
- it can and is supposed to run on a different machine from the system under test (although it still supports running on the same machine)
- there are different backends, it can run tests locally, over ssh, or start a VM with qemu
- produces different output formats currently html or json
Very nice features. Thanks!
Quoted 6 lines
Also there are two different options how to do that. Either we put it into runltp-ng directory to the LTP git root or we may as well create a separate repository under the linux-test-project and make it a ltp git submodule. Any preferencies to that?
I prefer the LTP git root as well.
Btw, I opened a PR for some tiny issues fixing, plz help review.
https://github.com/metan-ucw/runltp-ng/pull/28
Signature · 5 lines
Regards, Li Wang -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.linux.it/pipermail/ltp/attachments/20210426/57080db9/attachment.htm>