Skip to contentSkip to conversation

minimal ltp test set

LTP Mark Ver 3 participants 4 messages Nov 20, 2009 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Mark Ver

author
#1 lore raw

The number of testcases over the past year had increased faster than could be properly debugged by our skeleton team for the s390x platform. Many of the testcases seemed to be designed for features on later kernel versions with were not yet fully integrated by our supported distributions.

Since we mostly used LTP for a quick regression test, I was discussing with Subrata about creating a minimal test set that we expected would run properly on a majority of the distributions with kernel 2.6 and not just the more recent versions of the kernel.

This is my first pass at the test set:
(See attached file: s390x_ltpcore_targets)

It is basically a subset of the default SCENFILES from the ltp-full-20090731 concatenated into one file.

Basic rundown of the contents ...
* Included tests:
syscalls
fs
fsx
dio
mm
ipc
sched
math
nptl
pty
containers
fs_bind
filecaps
fcntl-locktests
-----
* Excluded tests - new in 2009 (maybe late 2008) versions (appear to have build, run, or support issues depending on distribution release) io
cap_bounds
connectors
admin_tools
timers
power_management_tests
numa
hugetlb
commands
hyperthreading
-----
* Excluded from original set:
controllers
- notes: this feature was currently available only on SLES-11 and the build and execution of the testcase appeared to need more work for s390x
-----

Ideally I was hoping to come up with a set of tests that were so basic, they should be supported by practically every release and would be expected
to run correctly everywhere. They would only fail if major issues had
been introduced into the system build.

- Mark Ver

office: Building 710 / Room 2-RF-10
phone: (845) 435-7794 [tie 8 295-7794]

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july

_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list

#2 lore raw
Quoted 52 lines · Mark Ver

The number of testcases over the past year had increased faster than could be properly debugged by our skeleton team for the s390x platform. Many of the testcases seemed to be designed for features on later kernel versions with were not yet fully integrated by our supported distributions.

Since we mostly used LTP for a quick regression test, I was discussing with Subrata about creating a minimal test set that we expected would run properly on a majority of the distributions with kernel 2.6 and not just the more recent versions of the kernel.

This is my first pass at the test set:
(See attached file: s390x_ltpcore_targets)

It is basically a subset of the default SCENFILES from the ltp-full-20090731 concatenated into one file.

Basic rundown of the contents ...
* Included tests:
syscalls
fs
fsx
dio
mm
ipc
sched
math
nptl
pty
containers
fs_bind
filecaps
fcntl-locktests
-----
* Excluded tests - new in 2009 (maybe late 2008) versions (appear to have build, run, or support issues depending on distribution release) io
cap_bounds
connectors
admin_tools
timers
power_management_tests
numa
hugetlb
commands
hyperthreading
-----
* Excluded from original set:
controllers
- notes: this feature was currently available only on SLES-11 and the build and execution of the testcase appeared to need more work for s390x
-----

Quoted 4 lines

Ideally I was hoping to come up with a set of tests that were so basic, they should be supported by practically every release and would be expected to run correctly everywhere. They would only fail if major issues had been introduced into the system build.

Mark,
An ideal solution to this problem is a two-tiered solution:
1. Feature based conditional compilation.
2. Tool based conditional compilation (some tests require certain
tools, such as perl, python, and various Unix utilities that aren't a part of the minimal set of tools, e.g. useradd, userdel, etc).
The tool based condition compilation is somewhat simple at a 10k
ft level (I have perl, python, ruby, etc -- add / remove these tests / features) -- the feature based conditional compilation issue is a bit more difficult to deal with as it requires a more in-depth understanding of what code exists in the sourcebase, and what functional tests map to what functional blocks under test. This is ideal, but not trivial to perform.
My first goal is to deal with any fallout from the make
infrastructure commit(s), dealing with the make 3.80 incompatibility and other unforeseen usecases like you have with building and executing out of the same source tree. Next, I want to clean up any and all C warnings so the tests are more robust (Mike, Cyril, and I have done a pretty good job but there's a ways to go...). Then I'll undertake completing the ballista cleanup, which will also involve adding the shims for perl and python across the entire sourcebase. We'll need to revisit the feature based enabling a little later.
How does that sound?
Thanks,
-Garrett

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list

#3 lore raw

Hi Mark,

Quoted 6 lines · Garrett Cooper
Quoted 5 lines · Mark Ver

The number of testcases over the past year had increased faster than could be properly debugged by our skeleton team for the s390x platform. Many of the testcases seemed to be designed for features on later kernel versions with were not yet fully integrated by our supported distributions.

Quoted 7 lines
Quoted 7 lines

Since we mostly used LTP for a quick regression test, I was discussing with Subrata about creating a minimal test set that we expected would run properly on a majority of the distributions with kernel 2.6 and not just the more recent versions of the kernel.

This is my first pass at the test set:
(See attached file: s390x_ltpcore_targets)

We can definitely make this part of ltp/runtest for your convenience.

Garret,

Can you please check this in. I seemed to have not received Mark´s original mail in this ID.

Regards--
Subrata

Quoted 71 lines
Quoted 40 lines

It is basically a subset of the default SCENFILES from the ltp-full-20090731 concatenated into one file.

Basic rundown of the contents ...
* Included tests:
syscalls
fs
fsx
dio
mm
ipc
sched
math
nptl
pty
containers
fs_bind
filecaps
fcntl-locktests
-----
* Excluded tests - new in 2009 (maybe late 2008) versions (appear to have build, run, or support issues depending on distribution release) io
cap_bounds
connectors
admin_tools
timers
power_management_tests
numa
hugetlb
commands
hyperthreading
-----
* Excluded from original set:
controllers
- notes: this feature was currently available only on SLES-11 and the build and execution of the testcase appeared to need more work for s390x
-----

Quoted 4 lines

Ideally I was hoping to come up with a set of tests that were so basic, they should be supported by practically every release and would be expected to run correctly everywhere. They would only fail if major issues had been introduced into the system build.

Mark,
An ideal solution to this problem is a two-tiered solution:
1. Feature based conditional compilation.
2. Tool based conditional compilation (some tests require certain
tools, such as perl, python, and various Unix utilities that aren't a part of the minimal set of tools, e.g. useradd, userdel, etc).
The tool based condition compilation is somewhat simple at a 10k
ft level (I have perl, python, ruby, etc -- add / remove these tests / features) -- the feature based conditional compilation issue is a bit more difficult to deal with as it requires a more in-depth understanding of what code exists in the sourcebase, and what functional tests map to what functional blocks under test. This is ideal, but not trivial to perform.
My first goal is to deal with any fallout from the make
infrastructure commit(s), dealing with the make 3.80 incompatibility and other unforeseen usecases like you have with building and executing out of the same source tree. Next, I want to clean up any and all C warnings so the tests are more robust (Mike, Cyril, and I have done a pretty good job but there's a ways to go...). Then I'll undertake completing the ballista cleanup, which will also involve adding the shims for perl and python across the entire sourcebase. We'll need to revisit the feature based enabling a little later.
How does that sound?
Thanks,
-Garrett

Quoted 9 lines

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list

author
#4 lore raw
In reply to Garrett Cooper

Garrett Cooper <[email protected]> wrote on 11/19/2009 11:54:02 PM:

Quoted 17 lines

Mark,
An ideal solution to this problem is a two-tiered solution:
1. Feature based conditional compilation.
2. Tool based conditional compilation (some tests require certain
tools, such as perl, python, and various Unix utilities that aren't a part of the minimal set of tools, e.g. useradd, userdel, etc).
The tool based condition compilation is somewhat simple at a 10k
ft level (I have perl, python, ruby, etc -- add / remove these tests / features) -- the feature based conditional compilation issue is a bit more difficult to deal with as it requires a more in-depth understanding of what code exists in the sourcebase, and what functional tests map to what functional blocks under test. This is ideal, but not trivial to perform.
My first goal is to deal with any fallout from the make
infrastructure commit(s), dealing with the make 3.80 incompatibility and other unforeseen usecases like you have with building and executing out of the same source tree. Next, I want to clean up any

Quoted 8 lines

and all C warnings so the tests are more robust (Mike, Cyril, and I have done a pretty good job but there's a ways to go...). Then I'll undertake completing the ballista cleanup, which will also involve adding the shims for perl and python across the entire sourcebase. We'll need to revisit the feature based enabling a little later.
How does that sound?
Thanks,
-Garrett

Sounds good to me. Definitely getting the conditional compilation straightened out is priority one. Right now we end up having to remove testcases and modify makefiles just to get the test suite to build.

I guess I was thinking of the minimal test set for the cases when new testcases were misbehaving rather than when they couldn't run because they misjudged the environment. But I agree the 2 scenarios probably overlap 95% of the time.

- Mark

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july

_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list