The Linux Test Project has been released for FEBRUARY 2011.

Hi,

The Linux Test Project test suite has been released for the month of FEBRUARY 2011. Please see ltp/INSTALL file carefully, as there has been multiple changes for building/installing the test suite after the recent changes in Makefile infrastructure.

The latest version of the test-suite contains 3000+ tests for the Linux OS and can be found at:
http://ltp.sourceforge.net/,
Latest happenings in LTP can also be found at:
http://ltp.sourceforge.net/wiki/,
http://ltp.sourceforge.net/wikiArchives.php, and,
IRC: irc.freenode.org #ltp.

Our web site also contains other information such as:
- A Linux test tools matrix
- Technical papers
- How To's on Linux testing
- Code coverage analysis tool.

We would encourage the community to post results to [email protected],
patches, new tests, bugs or comments/questions [email protected],
http://sourceforge.net/tracker/?func=add&group_id=3382&atid=103382 (for New Bug(s)),
http://sourceforge.net/tracker/?func=add&group_id=3382&atid=303382 (for New Patch(s)),
http://sourceforge.net/tracker/?func=add&group_id=3382&atid=353382 (for New Feature Request(s))

**Note: Some pending patches will be checked in for next monthś release.

Happy testing,
Regards,
Shubham

Wouldn't make sense to integrate this test suite in the kernel source tree?

Regards,
Paolo

There was discussion like this some few years back. The idea was to get some core tests from LTP to the kernel source tree. But then the idea was dropped probably to avoid maintenance overhead ;-)

Regards--
Subrata

Putting LTP in the kernel.org sources really doesn't make sense for the following reasons:

1. LTP isn't really tied to a single kernel release.
2. LTP isn't the only test project out there for Linux.
3. LTP has more stuff than it needs to have for testing out the kernel (well, it did more in the past before I started cleaning it up in the past couple of months).
4. Maintaining it will become a political bloodbath for both parties as Linux is loosely managed by Linus et all, and LTP has been largely developed by SGI and maintained by IBM and a few other parties like Fujitsu, Nokia, Redhat, etc.
5. Integrating LTP into Kbuild, etc would probably be non-trivial due to the size of LTP (but it might be easier after the Makefile restructuring I did a year and a half ago).

That being said, if Linux devs took the initiative and submitted testcases that either illustrated past regressions in the kernel, feature tested enhancements, and submitted documentation that actually described their changes to the Linux kernel and the manpages project, I would take this over having LTP in the linux sources because right now things largely work because the Linux sources haven't really been rototilled since 2.4 -> 2.6, but they are somewhat bitrotted and when Linux rototills its sources again, we'll have to go through rototilling our stuff as well.

Right now multiple QA engineers are sort of playing whack-a-mole trying to figure out requirements and submit tests to LTP, and since they don't have 100% context into the actual problem, some information is lost in translation when the tests are submitted. This isn't desirable.

Thanks,
-Garrett

these are all very good reasons. additional points:
- ltp is pretty fsckin huge
- ltp often times tests both sides of the userspace API/ABI -- between the
kernel and the C library, and the C library and end applications -mike

Where are the places in kernel source tree for those?

Those days, there just too many tests and testing projects for kernel like LTP, autotest, xfstests and so on. Why not have somewhere to collabrate and then to extract the best?

LTP has so many goals and focus which isn't going to be only to test kernel any more and it is increasing difficult to support so many distros, kernel versions, and so on.

There are some CORE tests like memory management tests, ksm, oom etc have benefit from the developers' bless and review. It also need to be updated to keep the tests relevant to the current git tree, since the features/specs are changing consistent inside the kernel.

This could be also useful to improve the kernel quality by providing test code inside the kernel tree that to be used during the code review process that for example, a ksm patchset needs to pass that particular sanity tests in order to catch the regression. It provide benefit that when the changelog said that it passed the ksm tests inside the kernel, we knew exactly what it is without needing to sync up with another project like LTP.

In term of maintenance, it needs to be selectively which tests need to be inside the kernel. There should ideally have a dedicated maintainer from the testing point of view to review them. The criteria can be something like,

1) purely purpose of the tests are to test kernel written in C with the kernel coding style. Userspace and integration tests should be better to put into LTP and other projects.
2) tests need to pass sub-system maintainers' review that for example, ksm tests need MM sub-system maintainers' review-by and sign-off-by and alike.
3) they need to be working with and sync up to the latest git version.
4) they have to be proved to be the best tests we can have to test those particular kernel code. There are many tests in LTP, but there are also many duplicated tests as well. Those need to be solved when considering to be moved inside the kernel source.
5) they should really be functional testing. Non-functional tests like stress or performance tests are usually more complex to setup hence defeat the purpose of quick regression checking.

Once we have those tests in-place, the next step to improve the kernel quality is to have more patches had Tested-by tags before been accepted in the kernel git tree. Those testers can simply use the non-ambiguious references for tests provided by those in-kernel tests.

In addition, those could be a follow-up items with the kernel regression reports that after fixed/analyzed a regression in kernel, the next natural thing to do is to fix/add missing tests to close this gap in the future by providing efficient tests to our users if all possible.

CAI Qian

------------------------------------------------------------------------------
Free Software Download: Index, Search & Analyze Logs and other IT data in Real-Time with Splunk. Collect, index and harness all the fast moving IT data generated by your applications, servers and devices whether physical, virtual or in the cloud. Deliver compliance at lower cost and gain new business insights. http://p.sf.net/sfu/splunk-dev2dev
_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list

Hi!
I'm not completely against this. At least LTP and possibly other testsuites would gain by attention from developers (it seems that the QA people themselves are the only ones interested). But some things are not just as simple as you see them (at least from my point of view).

That is IMHO just too much work. You would need somebody to extract it and then somebody to keep things in sync.

Frankly, LTP has maybe quite a lot of goals, but has a very little manpower, so just now it's mostly broken and rotten code. I would rather focus on cleaning up and fixing up the LTP, dropping ambiguous tests and so. We already did some progress in that area (I tend to say "we" but besides random contributions we have like three or five people).

That's true indeed.

Well, I don't see what would be gained by merging parts of the LTP into kernel tree. As I said before, this would probably lead to splitting of the forces (and not that we have a lot to split anyway). LTP already has directory called testcases/kernel/, LTP is in the git repository and we have a mailing list. All that is needed is people start noticing that we are here.

I don't think that it's easy to say if some tests are testing kernel/userspace. Sometimes the line isn't that clear.

Well, requiring maintainers to sign-off your tests is kind of dull. That would probably block the tests from being accepted just because maintainers don't care too much/have different things to do.

You can't easily prove that something is best ;).

Once again, LTP does exist so reference to LTP is not ambiguous. Yes, it's, for historical reasons, hosted on sourceforge rather than kernel.org. But there it is.

That's right thing to do, but once more, LTP is here for you ;). So my conclusion is that the major point here is make LTP more visible among linux hackers. On that note, I'm planing to prepare some presentation to let the people know what is the current state and what we are doing to get it better.

Cyril Hrubis
[email protected]

Then, the approach to merge parts of LTP to kernel is to say "Here we are, please accept our best". On the other hand, I have noticed that there are many developers tend to have test code in their kernel submit changelog which isn't it better to make life easier for them to add those testing code in a proper place in kernel which in-turn to benefit in a long run.

There are C code as in kernel coding style. Scripting code like Bash, Perl better to re-written in C that in a long run when there are something like thousands of tests to run that performance/scalabitlies/maintenence is going to matter just like to write an OS.

The idea is to raise a bar to get the best out of it. If maintainers don't care too much about the testing right now that is fine. There are many people they do care. A particular subsystem maintainer and its tests maintainer aren't necessary to be the same person because subsystem maintainer isn't necessary to be the best one to find/acknowledge defeats for code he maintained.

The best will at lest be reviewed by eyes from the kernel community and experts, and will be the one to be accepted by the community.

By accepted into the kernel, it certainly make it easier to reference without dealing with two projects and trees.

CAI Qian

I'm sorry for even starting this bikeshed discussion. Let's just bury the hatchet and get back to work on more fruitful things. Thanks,
-Garrett

Part of the problem is that every single testing project has different goals and
priorities. For example xfstests is maintained by the XFS folks, as well as people
from some of the other file system development efforts (the ext4 one in particular, thanks to people like Eric Sandeen), to test file systems.

At least at one point, I had heard a complaint that LTP was more focused on increasing test coverage as measured by a code coverage tool in the kernel than it was about about covering edge conditions, or races. There's nothing wrong with that, per se, and I don't know if it was true then or now, but it's a very different focus from one which is focused increasing the data reliability of file systems, quickly and efficiently.

And then there's the LSB test suites, which is really code at testing correctness from a standards perspective, which is a different focus yet again from the LTP and xfstests approach.

Bottom line, I'm a big fan of having different test suites, with different philosophies. Each philosophy has its strengths and blind spots, and so a problem that might be missed by one test suite might get caught by another.

The only real problem is an operational one. There are some programs which
are used by both LTP and xfstests, and changes that is made in one, don't necessarily get propagated to the other unless someone manually does it. But I think we can solve that without trying to merge all of these tests into a single Grand Unified Test Suite.

-- Ted

Perhaps we need to get developer's tests into the kernel. We now have a tools/testing directory lets use it.

How about having the developers write tests and place them in the tools/testing directory and then the folks at LTP and xfstests et al. can update their code with the code from the kernel.

Currently only ktest.pl exists in this directory. I use it constantly and post bugs that it finds. It focuses on just the building and booting of a kernel. It can run several randconfigs, do bisects and such, but it just has a single command to do any tests. This command is just a shell command the user can put in. I leave what tests to run to the user. Thus LTP could be the test that gets kicked off.

For example: I have this script I run on my x86 box:

TEST_START ITERATE 10
TEST_TYPE = test
BUILD_TYPE = randconfig
MIN_CONFIG = /home/rostedt/work/autotest/configs/mitest/config-mitest-net CHECKOUT = origin/master
TEST = ssh root@mitest /work/c/hackbench_32 50

TEST_START ITERATE 10
TEST_TYPE = boot
BUILD_TYPE = randconfig
MIN_CONFIG = /home/rostedt/work/autotest/configs/mitest/config-mitest-min MAKE_CMD = make ARCH=i386

DEFAULTS
REBOOT_ON_ERROR = 0
POWEROFF_ON_ERROR = 1
POWEROFF_ON_SUCCESS = 1
REBOOT_ON_SUCCESS = 0
DIE_ON_FAILURE = 0
STORE_FAILURES = /home/rostedt/work/autotest/nobackup/failures
POWEROFF_AFTER_HALT = 60
CLEAR_LOG = 1
MIN_CONFIG = /home/rostedt/work/autotest/configs/mitest/config-mitest-min
SSH_USER = root
BUILD_DIR = /home/rostedt/work/autotest/nobackup/linux-test.git
OUTPUT_DIR = /home/rostedt/work/autotest/nobackup/mitest
BUILD_TARGET = arch/x86/boot/bzImage
TARGET_IMAGE = /boot/vmlinuz-test
POWER_CYCLE = /home/rostedt/work/autotest/cycle-mxtest
CONSOLE = nc -d fedora 3001
LOCALVERSION = -test
GRUB_MENU = Test Kernel
MAKE_CMD = distmake-32 ARCH=i386
POWER_OFF = /home/rostedt/work/autotest/poweroff-mxtest
BUILD_OPTIONS = -j20
LOG_FILE = /home/rostedt/work/autotest/nobackup/mitest/mitest.log
TEST = ssh root@mitest cat /debug/tracing/trace
ADD_CONFIG = /home/rostedt/work/autotest/configs/config-broken /home/rostedt/work/autotest/config-general

Each command is documented in the samples.conf that is also in that directory.

The above test runs two sets of tests. The first runs randconfig 10 times with a minimum config that will allow my box to have a network connection, and after it boots, it runs hackbench.

The second test runs ten randconfig builds 10 times and just makes sure the box can boot.

I'm working on getting to run commands via the console so I do not need the network active to run tests.

Once could change the TEST = ... to run LTP tests, or anything else. Having tests to run that I can add to my automated testing would be helpful. I could build randconfigs and run these tests making sure that they work with different configurations.

Maybe it would also be helpful to have the CONFIGs that are needed by the tests. It would not make sense to test iptables if iptables is not configured ;)

-- Steve

There are a few test code span over other directories too. Documentation/vm/hugepage-mmap.c Documentation/vm/hugepage-shm.c Documentation/vm/map_hugetlb.c
...

CAI Qian

------------------------------------------------------------------------------
What You Don't Know About Data Connectivity CAN Hurt You This paper provides an overview of data connectivity, details its effect on application quality, and explores various alternative solutions. http://p.sf.net/sfu/progress-d2d
_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list

These all look like "Examples", which means they should probably be moved to samples? As "samples" can be compiled on boot. But perhaps because they are user space programs its fine to keep them here?

-- Steve

Yes, but also test cases to check if basic hugetlb code is working. Well, the problem is that they are all over the place, and some of good test cases are within commit logs as well which makes things harder to maintain, use...

CAI Qian

------------------------------------------------------------------------------
Colocation vs. Managed Hosting
A question and answer guide to determining the best fit for your organization - today and in the future.
http://p.sf.net/sfu/internap-sfd2d
_______________________________________________
Ltp-list mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list