LTP old API conversion
Hi all,
so we still have ~200 patches to refactor and to move from old LTP API into the new LTP API. That would be really useful for tests maintenance and long term supports, such as the runtest removal and replacement with a smarter tests filtering/grouping.
This is a tedious task that requires a huge amount of work and in the past years we managed to convert hundreds of tests by hand, each one requiring multiple reviews iterations.
It was overwhelming not only for developers, but also for reviewers who were stucked by reviewing new patches + tests rewriting.
In 2026 we have the chance to accelerate this transition from old API to new API using LLMs and, as we discussed in the yesterday LTP after release meeting, we might be in the right way to start doing it (at least for smaller tests).
I created a set of configurations and skills in my personal repo which can be used to start this process: https://github.com/acerv/agents-config.
It's maily tested using Claude Code, since it's the model which perform the best (at the moment), but any other model can be used.
I experimented with a list of tests that can be obtained with the following command:
wc -l $(grep -R '"test\.h"' --include="*.c" testcases/kernel/ | cut -d: -f1) | sort -g
.. and tests conversion for tests which are smaller than 200 lines of code requires minimal (if no) edit. I will continue to adapt the ltp-convert skill in order to tweak and to improve this process for bigger tests.
~~ Said so..
.. since this process seems to be quite straight forward, and with the usage of LLM we could easily generate hundreds of patches per month, we don't really want to flood the ML with garbage and to overwhelm who's involved into maintenance review.
How we should organize this job?
Should we set a maximum amount of tests refactoring per month? How do we organize these patches? (i.e. with blocks of patches)
Kind regards,
--
Andrea Cervesato
SUSE QE Automation Engineer Linux
[email protected]
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Hi Andrea,
[ Cc others who were on LTP meeting + Sebastian who might be interested ]
Quoted 24 lines
Hi all,
so we still have ~200 patches to refactor and to move from old LTP API into the new LTP API. That would be really useful for tests maintenance and long term supports, such as the
runtestremoval and replacement with a smarter tests filtering/grouping.This is a tedious task that requires a huge amount of work and in the past years we managed to convert hundreds of tests by hand, each one requiring multiple reviews iterations.
It was overwhelming not only for developers, but also for reviewers who were stucked by reviewing new patches + tests rewriting.
In 2026 we have the chance to accelerate this transition from old API to new API using LLMs and, as we discussed in the yesterday LTP after release meeting, we might be in the right way to start doing it (at least for smaller tests).
I created a set of configurations and skills in my personal repo which can be used to start this process: https://github.com/acerv/agents-config.
It's maily tested using Claude Code, since it's the model which perform the best (at the moment), but any other model can be used.
Quoted 4 lines
I experimented with a list of tests that can be obtained with the following command:
wc -l $(grep -R '"test\.h"' --include="*.c" testcases/kernel/ | cut -d: -f1) | sort -g
wc -l $(grep -R '"test\.h"' --include="*.c" testcases/ | cut -d: -f1) | sort -g
Otherwise you miss 12 old API tests.
Quoted 6 lines
.. and tests conversion for tests which are smaller than 200 lines of code requires minimal (if no) edit. I will continue to adapt the ltp-convert skill in order to tweak and to improve this process for bigger tests.
~~ Said so..
Quoted 8 lines
.. since this process seems to be quite straight forward, and with the usage of LLM we could easily generate hundreds of patches per month, we don't really want to flood the ML with garbage and to overwhelm who's involved into maintenance review.
How we should organize this job?
Should we set a maximum amount of tests refactoring per month? How do we organize these patches? (i.e. with blocks of patches)
I'd vote for limiting patchsets to max tests in a single directory. Why? Smaller patchset is easier to review. And if just some of the commits are accepted then fewer commits need to be rebased.
Kind regards,
Petr
Quoted 1 line
Kind regards,
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Hi Petr,
Quoted 3 lines
I'd vote for limiting patchsets to max tests in a single directory. Why? Smaller patchset is easier to review. And if just some of the commits are accepted then fewer commits need to be rebased.
Do you mean one patch-set for multiple (unrelated) tests (for instance syscalls folder) [1], or one patch-set per testing suite [2] ?
[1] would be 20 random tests from testcases/kerne/syscalls
[2] i.e. all tests to refactor in testcases/kernel/mem/mmapstress
I would follow the first approach if we want to send a big amount of patches, or the second if we want to be more specific.
--
Andrea Cervesato
SUSE QE Automation Engineer Linux
[email protected]
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 8 lines
Hi Petr,
Quoted 3 lines
I'd vote for limiting patchsets to max tests in a single directory. Why? Smaller patchset is easier to review. And if just some of the commits are accepted then fewer commits need to be rebased.
Do you mean one patch-set for multiple (unrelated) tests (for instance syscalls folder) [1], or one patch-set per testing suite [2] ?
I meant [2], i.e. tests from certain folder (usually tests related to single syscall). But it's just a suggestion, feel free to do it differently if it's significantly simpler.
Thanks for this effort.
Kind regards,
Petr
Quoted 5 lines
[1] would be 20 random tests from testcases/kerne/syscalls
[2] i.e. all tests to refactor intestcases/kernel/mem/mmapstressI would follow the first approach if we want to send a big amount of patches, or the second if we want to be more specific.
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 6 lines · Petr Vorel
Quoted 5 lines
Hi Petr,
Quoted 3 lines
I'd vote for limiting patchsets to max tests in a single directory. Why? Smaller patchset is easier to review. And if just some of the commits are accepted then fewer commits need to be rebased.
Quoted 6 lines
Quoted 2 lines
Do you mean one patch-set for multiple (unrelated) tests (for instance syscalls folder) [1], or one patch-set per testing suite [2] ?
I meant [2], i.e. tests from certain folder (usually tests related to single syscall). But it's just a suggestion, feel free to do it differently if it's significantly simpler.
Some tests may relocate their folder after the refacoring.
Regardless of which method we ultimately adopt, I recommend that the AI send no more than 10 patches per week [1].
Furthermore, new patches should be sent only after the previous patchset has been fully processed.
[1] Given that the total number of legacy tests is only 242, the cleanup work will not take much time.
Signature · 6 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
Hi!
Quoted 3 lines
.. and tests conversion for tests which are smaller than 200 lines of code requires minimal (if no) edit. I will continue to adapt the ltp-convert skill in order to tweak and to improve this process for bigger tests.
I would disagree here withe the minimal edit. The old tests are quite often garbage. So the human in the process should asses if the test is actually doing anything useful and guide the machine to implement better test in a case that the original wasn't doing anything useful.
Quoted 6 lines
~~ Said so..
.. since this process seems to be quite straight forward, and with the usage of LLM we could easily generate hundreds of patches per month, we don't really want to flood the ML with garbage and to overwhelm who's involved into maintenance review.
I'm pretty sure that you will get this down to smaller set as long as you actually spend some time thinking about what the test does. At least for me I'm able to spend 30% time on a test conversion because LLM, but you have to spend some time understanding what the test does and making sure it makes sense. Otherwise it's garbage in - garbage out.
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Hi Cyril,
Quoted 9 lines
Hi!
Quoted 3 lines
.. and tests conversion for tests which are smaller than 200 lines of code requires minimal (if no) edit. I will continue to adapt the ltp-convert skill in order to tweak and to improve this process for bigger tests.
I would disagree here withe the minimal edit. The old tests are quite often garbage. So the human in the process should asses if the test is actually doing anything useful and guide the machine to implement better test in a case that the original wasn't doing anything useful.
That's indeed what I do. To blindly accept everything turns out to produce a lot of trash, but in general it's way more easy to refactor tests in this way than before, at least for the smallest tests. With a good set of rules given to the LLM we can produce good results at the very first try sometimes.
Quoted 8 lines
Quoted 7 lines
~~ Said so..
.. since this process seems to be quite straight forward, and with the usage of LLM we could easily generate hundreds of patches per month, we don't really want to flood the ML with garbage and to overwhelm who's involved into maintenance review.
Quoted 5 lines
I'm pretty sure that you will get this down to smaller set as long as you actually spend some time thinking about what the test does. At least for me I'm able to spend 30% time on a test conversion because LLM, but you have to spend some time understanding what the test does and making sure it makes sense. Otherwise it's garbage in - garbage out.
I counted a x4 times faster conversion compared to the previous manual process, at least for 100-200 lines tests. Bigger ones will be more challenging and after a few attempt the conversion time drops significantly.
An another thing is how we want to push these patches in the repository. At the moment kernel is using some special tags to identify if patches have been generated by LLMs or by humans. This is something we should think about, because we already have a few examples recently (i.e. your ulimit01, newuname01 tests and my listns() testing suite) where LLMs where doing most of the job. Also, we are receiving more and more patches which seem to be written by AI.
--
Andrea Cervesato
SUSE QE Automation Engineer Linux
[email protected]
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 6 lines
An another thing is how we want to push these patches in the repository. At the moment kernel is using some special tags to identify if patches have been generated by LLMs or by humans. This is something we should think about, because we already have a few examples recently (i.e. your ulimit01, newuname01 tests and my
listns()testing suite) where LLMs where doing most of the job. Also, we are receiving more and more patches which seem to be written by AI.
What about this ?
--
Andrea Cervesato
SUSE QE Automation Engineer Linux
[email protected]
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 17 lines · Cyril Hrubis
Hi!
Quoted 3 lines
.. and tests conversion for tests which are smaller than 200 lines of code requires minimal (if no) edit. I will continue to adapt the ltp-convert skill in order to tweak and to improve this process for bigger tests.
I would disagree here withe the minimal edit. The old tests are quite often garbage. So the human in the process should asses if the test is actually doing anything useful and guide the machine to implement better test in a case that the original wasn't doing anything useful.
Quoted 6 lines
~~ Said so..
.. since this process seems to be quite straight forward, and with the usage of LLM we could easily generate hundreds of patches per month, we don't really want to flood the ML with garbage and to overwhelm who's involved into maintenance review.
Quoted 5 lines
I'm pretty sure that you will get this down to smaller set as long as you actually spend some time thinking about what the test does. At least for me I'm able to spend 30% time on a test conversion because LLM, but you have to spend some time understanding what the test does and making sure it makes sense. Otherwise it's garbage in - garbage out.
Yes but not entirely. We can have AI summarize test methods within code comments, which helps us grasp the underlying logic much faster than reading through legacy code directly.
And, nowdays AI is gradually becoming the producer of code, while humans(specifically maintainers) assume the role of the ultimate guardians of code quality.
Signature · 6 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp