Skip to contentSkip to conversation

Stress tests added to ctf testsuite

DISCUSSION Alexandre Montplaisir 2 participants 6 messages Nov 16, 2014 – Nov 18, 2014 6 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

Hi Mathieu,

Looks great! Preliminary testing shows that Trace Compass also blows up in many different ways, but hey that's expected ;)

If I may suggest, having a script, either shell or Python, to just generate/clean the test traces would be very useful.
We don't use the run.sh scripts in CI (we're within JUnit tests, there is no executable at that point), we just look at the trace directories and try to run everything in there. If there was a simple way to "create everything" and then "clean everything" it would make it easier to integrate in our (and I guess others') build system.

Cheers,
Alexandre

Quoted 15 lines · Mathieu Desnoyers

Hi!

If you look at
https://github.com/efficios/ctf-testsuite

I just pushed various CTF stress-tests, and did some general cleanup in the tree (e.g. renaming tests to meaningful names).

Have fun!

Thanks,

Mathieu

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw

Can you modify Junit, or use a different test system,
to allow calling scripts instead ? I expect we will end up having much more automatically generated test-cases over time, and the more we will have, the less sense it will make to generate them all at test start.

Thanks!

Mathieu

----- Original Message -----

Quoted 41 lines

From: "Alexandre Montplaisir" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>, "Matthew Khouzam" <[email protected]>
Cc: "lttng-dev" <[email protected]>, "Trace Compass Developer Discussions" <[email protected]>
Sent: Sunday, November 16, 2014 9:39:44 PM
Subject: Re: Stress tests added to ctf testsuite

Hi Mathieu,

Looks great! Preliminary testing shows that Trace Compass also blows up in many different ways, but hey that's expected ;)

If I may suggest, having a script, either shell or Python, to just generate/clean the test traces would be very useful.
We don't use the run.sh scripts in CI (we're within JUnit tests, there is no executable at that point), we just look at the trace directories and try to run everything in there. If there was a simple way to "create everything" and then "clean everything" it would make it easier to integrate in our (and I guess others') build system.

Cheers,
Alexandre

Quoted 15 lines · Mathieu Desnoyers

Hi!

If you look at
https://github.com/efficios/ctf-testsuite

I just pushed various CTF stress-tests, and did some general cleanup in the tree (e.g. renaming tests to meaningful names).

Have fun!

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

A message that is not in the archive (sent off-list or not yet received): [email protected]

#5 lore raw

We can run scripts, yeah. But, as far as I understand it, the ./run.sh scripts require passing a CTF_READER_BIN executable, which we don't have during the testing phase.

Another advantage of using parameterized tests is that it creates one JUnit test for every instance that is created at run-time (here, one per test trace to test). This way they show up individually in test reports [1]. And if one of them fails, at the end we know which one did, and not just "something failed in the script".

I see how the test files can be big, so generating *everything* may not be desirable. Could it be per test then? Or even part of the Python scripts themselves? (generate -> test -> clean, have a default array of sizes to generate, user can specify other sizes?)

Thanks,
Alexandre

[1] For example: https://hudson.eclipse.org/tracecompass/job/tracecompass-master-nightly/31/testReport/org.eclipse.tracecompass.ctf.core.tests.ctftestsuite/CtfTestSuiteTests/

Quoted 51 lines · Mathieu Desnoyers

Can you modify Junit, or use a different test system,
to allow calling scripts instead ? I expect we will end up having much more automatically generated test-cases over time, and the more we will have, the less sense it will make to generate them all at test start.

Thanks!

Mathieu

----- Original Message -----

Quoted 40 lines

From: "Alexandre Montplaisir" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>, "Matthew Khouzam" <[email protected]>
Cc: "lttng-dev" <[email protected]>, "Trace Compass Developer Discussions" <[email protected]>
Sent: Sunday, November 16, 2014 9:39:44 PM
Subject: Re: Stress tests added to ctf testsuite

Hi Mathieu,

Looks great! Preliminary testing shows that Trace Compass also blows up in many different ways, but hey that's expected ;)

If I may suggest, having a script, either shell or Python, to just generate/clean the test traces would be very useful.
We don't use the run.sh scripts in CI (we're within JUnit tests, there is no executable at that point), we just look at the trace directories and try to run everything in there. If there was a simple way to "create everything" and then "clean everything" it would make it easier to integrate in our (and I guess others') build system.

Cheers,
Alexandre

Quoted 15 lines · Mathieu Desnoyers

Hi!

If you look at
https://github.com/efficios/ctf-testsuite

I just pushed various CTF stress-tests, and did some general cleanup in the tree (e.g. renaming tests to meaningful names).

Have fun!

Thanks,

Mathieu

A message that is not in the archive (sent off-list or not yet received): [email protected]

#7 lore raw

The run.sh scripts are just a convenience for running test in every subdirectory with TAP output, and in the case of stress tests, calling test.py for every line of param-list.txt in each subdirectory.

AFAIU, the junit tests are the ones driving the traversal in your case, so you probably don't want run.sh at all. However, what would prevent a junit test from invoking e.g.

./test.py prepare 512000
[do test]
./test.py clean 512000

?

Thanks,

Mathieu

----- Original Message -----

Quoted 85 lines

From: "Alexandre Montplaisir" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "Matthew Khouzam" <[email protected]>, "lttng-dev" <[email protected]>, "Trace Compass Developer Discussions" <[email protected]>
Sent: Monday, November 17, 2014 12:12:25 AM
Subject: Re: Stress tests added to ctf testsuite

We can run scripts, yeah. But, as far as I understand it, the ./run.sh scripts require passing a CTF_READER_BIN executable, which we don't have during the testing phase.

Another advantage of using parameterized tests is that it creates one JUnit test for every instance that is created at run-time (here, one per test trace to test). This way they show up individually in test reports [1]. And if one of them fails, at the end we know which one did, and not just "something failed in the script".

I see how the test files can be big, so generating *everything* may not be desirable. Could it be per test then? Or even part of the Python scripts themselves? (generate -> test -> clean, have a default array of sizes to generate, user can specify other sizes?)

Thanks,
Alexandre

[1] For example:
https://hudson.eclipse.org/tracecompass/job/tracecompass-master-nightly/31/testReport/org.eclipse.tracecompass.ctf.core.tests.ctftestsuite/CtfTestSuiteTests/

Quoted 53 lines · Mathieu Desnoyers

Can you modify Junit, or use a different test system,
to allow calling scripts instead ? I expect we will end up having much more automatically generated test-cases over time, and the more we will have, the less sense it will make to generate them all at test start.

Thanks!

Mathieu

----- Original Message -----

Quoted 42 lines

From: "Alexandre Montplaisir" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>, "Matthew Khouzam" <[email protected]>
Cc: "lttng-dev" <[email protected]>, "Trace Compass Developer Discussions" <[email protected]>
Sent: Sunday, November 16, 2014 9:39:44 PM
Subject: Re: Stress tests added to ctf testsuite

Hi Mathieu,

Looks great! Preliminary testing shows that Trace Compass also blows up in many different ways, but hey that's expected ;)

If I may suggest, having a script, either shell or Python, to just generate/clean the test traces would be very useful.
We don't use the run.sh scripts in CI (we're within JUnit tests, there is no executable at that point), we just look at the trace directories and try to run everything in there. If there was a simple way to "create everything" and then "clean everything" it would make it easier to integrate in our (and I guess others') build system.

Cheers,
Alexandre

Quoted 15 lines · Mathieu Desnoyers

Hi!

If you look at
https://github.com/efficios/ctf-testsuite

I just pushed various CTF stress-tests, and did some general cleanup in the tree (e.g. renaming tests to meaningful names).

Have fun!

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

A message that is not in the archive (sent off-list or not yet received): [email protected]

#9 lore raw
Quoted 11 lines · Mathieu Desnoyers

The run.sh scripts are just a convenience for running test in every subdirectory with TAP output, and in the case of stress tests, calling test.py for every line of param-list.txt in each subdirectory.

AFAIU, the junit tests are the ones driving the traversal in your case, so you probably don't want run.sh at all. However, what would prevent a junit test from invoking e.g.

./test.py prepare 512000
[do test]
./test.py clean 512000

The way it works, we just look for all the traces in a directory (like let's say regression/metadata/pass/* ), and the parameters just create one test for every entry found. So the traces needs to be present on the filesystem before the test class is run.

Which is why I suggested a way to run all the "prepare 256", "prepare 512" etc. in one shot, so that we could do it as part of the test preparation phase (before the actual tests are run). If it's not deemed useful upstream, we could always do it on our side. But then we'd have to list every single one of those tests individually, which we were able to avoid doing so far. And we'd always be playing catch-up as new tests are added. But hey, such is life ;)

Hope this explains it.

Cheers,
Alexandre

Quoted 91 lines

?

Thanks,

Mathieu

----- Original Message -----

Quoted 83 lines

From: "Alexandre Montplaisir" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "Matthew Khouzam" <[email protected]>, "lttng-dev" <[email protected]>, "Trace Compass Developer Discussions" <[email protected]>
Sent: Monday, November 17, 2014 12:12:25 AM
Subject: Re: Stress tests added to ctf testsuite

We can run scripts, yeah. But, as far as I understand it, the ./run.sh scripts require passing a CTF_READER_BIN executable, which we don't have during the testing phase.

Another advantage of using parameterized tests is that it creates one JUnit test for every instance that is created at run-time (here, one per test trace to test). This way they show up individually in test reports [1]. And if one of them fails, at the end we know which one did, and not just "something failed in the script".

I see how the test files can be big, so generating *everything* may not be desirable. Could it be per test then? Or even part of the Python scripts themselves? (generate -> test -> clean, have a default array of sizes to generate, user can specify other sizes?)

Thanks,
Alexandre

[1] For example:
https://hudson.eclipse.org/tracecompass/job/tracecompass-master-nightly/31/testReport/org.eclipse.tracecompass.ctf.core.tests.ctftestsuite/CtfTestSuiteTests/

Quoted 52 lines · Mathieu Desnoyers

Can you modify Junit, or use a different test system,
to allow calling scripts instead ? I expect we will end up having much more automatically generated test-cases over time, and the more we will have, the less sense it will make to generate them all at test start.

Thanks!

Mathieu

----- Original Message -----

Quoted 41 lines

From: "Alexandre Montplaisir" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>, "Matthew Khouzam" <[email protected]>
Cc: "lttng-dev" <[email protected]>, "Trace Compass Developer Discussions" <[email protected]>
Sent: Sunday, November 16, 2014 9:39:44 PM
Subject: Re: Stress tests added to ctf testsuite

Hi Mathieu,

Looks great! Preliminary testing shows that Trace Compass also blows up in many different ways, but hey that's expected ;)

If I may suggest, having a script, either shell or Python, to just generate/clean the test traces would be very useful.
We don't use the run.sh scripts in CI (we're within JUnit tests, there is no executable at that point), we just look at the trace directories and try to run everything in there. If there was a simple way to "create everything" and then "clean everything" it would make it easier to integrate in our (and I guess others') build system.

Cheers,
Alexandre

Quoted 15 lines · Mathieu Desnoyers

Hi!

If you look at
https://github.com/efficios/ctf-testsuite

I just pushed various CTF stress-tests, and did some general cleanup in the tree (e.g. renaming tests to meaningful names).

Have fun!

Thanks,

Mathieu

A message that is not in the archive (sent off-list or not yet received): [email protected]

#11 lore raw

----- Original Message -----

Quoted 17 lines

From: "Alexandre Montplaisir" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "Matthew Khouzam" <[email protected]>, "lttng-dev" <[email protected]>, "Trace Compass Developer Discussions" <[email protected]>
Sent: Monday, November 17, 2014 10:41:10 PM
Subject: Re: Stress tests added to ctf testsuite

Quoted 8 lines · Mathieu Desnoyers

The run.sh scripts are just a convenience for running test in every subdirectory with TAP output, and in the case of stress tests, calling test.py for every line of param-list.txt in each subdirectory.

AFAIU, the junit tests are the ones driving the traversal in your case, so you probably don't want run.sh at all. However, what would prevent a junit test from invoking e.g.

Quoted 8 lines
Quoted 3 lines

./test.py prepare 512000
[do test]
./test.py clean 512000

The way it works, we just look for all the traces in a directory (like let's say regression/metadata/pass/* ), and the parameters just create one test for every entry found. So the traces needs to be present on the filesystem before the test class is run.

I understand that this is the way it is currently implemented in your test suite, but why can't it be changed ?

How about looking for:

stress/metadata/pass/*/test.py too ?

For every test.py found, you read param-list.txt in the same directory, and generate one test per parameter. Each test would be a sequence of ops:

test.py prepare $param
[run test on trace-$param]
test.py clean $param

On our side, we could ensure that we stay regular in the ctf-testsuite, always using test.py and param-list.txt with prepare/clean phases.

Quoted 8 lines

Which is why I suggested a way to run all the "prepare 256", "prepare 512" etc. in one shot, so that we could do it as part of the test preparation phase (before the actual tests are run). If it's not deemed useful upstream, we could always do it on our side. But then we'd have to list every single one of those tests individually, which we were able to avoid doing so far. And we'd always be playing catch-up as new tests are added. But hey, such is life ;)

You would not have to list each test if we enforce a regular structure in the test suite, as proposed above.

Or am I missing something ?

Thanks,

Mathieu

Quoted 100 lines

Hope this explains it.

Cheers,
Alexandre

Quoted 92 lines

?

Thanks,

Mathieu

----- Original Message -----

Quoted 84 lines

From: "Alexandre Montplaisir" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "Matthew Khouzam" <[email protected]>, "lttng-dev" <[email protected]>, "Trace Compass
Developer Discussions" <[email protected]>
Sent: Monday, November 17, 2014 12:12:25 AM
Subject: Re: Stress tests added to ctf testsuite

We can run scripts, yeah. But, as far as I understand it, the ./run.sh scripts require passing a CTF_READER_BIN executable, which we don't have during the testing phase.

Another advantage of using parameterized tests is that it creates one JUnit test for every instance that is created at run-time (here, one per test trace to test). This way they show up individually in test reports [1]. And if one of them fails, at the end we know which one did, and not just "something failed in the script".

I see how the test files can be big, so generating *everything* may not be desirable. Could it be per test then? Or even part of the Python scripts themselves? (generate -> test -> clean, have a default array of sizes to generate, user can specify other sizes?)

Thanks,
Alexandre

[1] For example:
https://hudson.eclipse.org/tracecompass/job/tracecompass-master-nightly/31/testReport/org.eclipse.tracecompass.ctf.core.tests.ctftestsuite/CtfTestSuiteTests/

Quoted 52 lines · Mathieu Desnoyers

Can you modify Junit, or use a different test system,
to allow calling scripts instead ? I expect we will end up having much more automatically generated test-cases over time, and the more we will have, the less sense it will make to generate them all at test start.

Thanks!

Mathieu

----- Original Message -----

Quoted 41 lines

From: "Alexandre Montplaisir" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>, "Matthew Khouzam" <[email protected]>
Cc: "lttng-dev" <[email protected]>, "Trace Compass Developer Discussions" <[email protected]>
Sent: Sunday, November 16, 2014 9:39:44 PM
Subject: Re: Stress tests added to ctf testsuite

Hi Mathieu,

Looks great! Preliminary testing shows that Trace Compass also blows up in many different ways, but hey that's expected ;)

If I may suggest, having a script, either shell or Python, to just generate/clean the test traces would be very useful.
We don't use the run.sh scripts in CI (we're within JUnit tests, there is no executable at that point), we just look at the trace directories and try to run everything in there. If there was a simple way to "create everything" and then "clean everything" it would make it easier to integrate in our (and I guess others') build system.

Cheers,
Alexandre

Quoted 15 lines · Mathieu Desnoyers

Hi!

If you look at
https://github.com/efficios/ctf-testsuite

I just pushed various CTF stress-tests, and did some general cleanup in the tree (e.g. renaming tests to meaningful names).

Have fun!

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com