CTF stress tests
CC-ing lttng-dev since this applies to Babeltrace.
Quoted 5 lines · Matthew Khouzam
Hi,
I was looking into the CTF stress tests.
They are good, but I don't know if we want them in our standard test cases. They basically check the scalability of the computer it is being run on in most cases and in all cases, the test of reading 12g of files
I respectfully disagree.
In most tests, there is definitely a smart way to deal with the trace. That is not to say that all implementations are expected to support every imaginable test cases. However, they should strive to, at the very least, gracefully handle failures to read traces and document their limitations as the spec imposes very few.
While I agree that some of these tests are far-fetched (you're not likely to see a CTF trace with a 32 768 level deep structure any time soon), traces with enough streams to exhaust the max fd count are not far-fetched at all. In fact, tracing any decently-sized cluster will bust that limit in no time.
Handling a multi-megabyte sequence (~100mb), something that Babeltrace can't do at the moment, may seem unreasonable at first. It quickly becomes very pertinent when people start talking of core dumping to a CTF trace.
Quoted 4 lines
is rather prohibitive. That being said, maybe having this as a weekly build could be an interesting idea. Also, I don't think the idea that busting the heap size is a good indication of a test failure. I can always allocate more heap at the problem. :)
Unfortunately, my motherboard's DIMM slots are already full. ;-)
Quoted 9 lines
Now for a breakdown of the tests:
├── gen.sh ├── metadata │ └── pass │ ├── large-metadata - interesting test, I think something likethis should be used to improve robustness
│ ├── long-identifier - this should be our approach here, I think
http://stackoverflow.com/questions/6007568/what-is-max-length-for-an-c-c-identifier-on-common-build-systems
If you mean that an implementation should document its limitations, I agree.
Quoted 5 lines
│ ├── many-callsites - works here until we oome but it's slow to
test and we don't have a real application to validate the data yet. So it's good, but we may wish to put efforts elsewhere for now.
│ ├── many-stream-class - This test is looking at the max size of
an array...
Not sure what you mean here although I agree that reading a trace with 16 million stream classes is not a "realistic" test.
That's not the point of the CTF stress test suite. This test suite will expose errors that are not handled gracefully. Implementations are expected to fail, but to do so in a controlled and documented way; not silently or by triggering the OOM-killer.
Quoted 5 lines
│ ├── many-typealias - interesting test, the parser will suffer.
│ └── many-typedef - ditto
└── stream
└── pass
├── array-large - Testing the max size of ram and arrays.There is no need to load the entire array in RAM. I, personally, intend on implementing a slow fallback to disk when arrays exceed a given size.
This will not work in live mode and that's completely okay. As long as it doesn't outright crash.
There is also a security aspect to this; an implementation shouldn't trust the relay daemon to send packets it can handle safely. This doesn't really apply to arrays as their length is statically known, but unchecked sequence length are definitely a security/stability concern.
Quoted 2 lines
├── many-events - This can be an interesting tool for profiling.
But it's not really a test for the reader...
Not sure why. Tracing multiple UST applications over a long time is bound to stress this at some point.
Just imagine a snapshot session running for a year, through multiple application updates. The number of UST tracepoints can quickly become humongous.
Quoted 4 lines
├── many-packets - This will be good to test distributed indexes
├── many-streams - Good, but even better is...
├── many-traces - interesting and a real problem
├── packet-large - this is actually easier than many-eventsSpeak for yourself! Babeltrace mmaps entire packets! ;-) We have our own share of embarrassing limitations :-P
Quoted 5 lines
├── sequence-large - see array-large ├── string-large - see array-large ├── struct-many-fields - see array-large ├── struct-nest-n-deep - I like this one, it highlights one ofour optimisations
Hmm, interesting! Care to explain the gist of it?
Quoted 1 line
└── variant-many-tags - see array-large
Again, this is something I can easily see being produced by a dynamically-typed language. Imagine a VM implementation that would trace every function call for every possible argument type permutation.
Quoted 2 lines
Why no deep variants? just curious.
Good point, we should add that!
We should also have nested sequences, arrays and mixes of variants, sequences and arrays.
Our assumption is that the nesting-handling code is shared between various types; the nested-structures test should ideally trigger it.
Perhaps we should not presume such implementation details... The test suite is still a work in progress; any weird test case you can think of is welcome!
Quoted 6 lines
so, tl;dr the tests are good at making a system suffer and qualifying its boundaries. I think this is great for profiling, but for actual testing, it should not be part of the per-patch system we have set up.
Any thoughts?
I think the real problem is assuming that a failing test is necessarily a deal-breaker.
I see this test suite as CTF's ACID equivalent [1]. Most web browsers knowingly fail this test, it doesn't make it any less relevant.
Jérémie
Signature · 8 lines
Jérémie Galarneau EfficiOS Inc. http://www.efficios.com _______________________________________________ lttng-dev mailing list [email protected] http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Great points!
I like the view: this is like the acid test, since that test is not run at every build, but rather periodically. That is much more reasonable.
The reason for my initial email is that when I ran the stress test on the parser, 147 tests failed. When I tweaked the timeouts and the stack and heap size, it went down to 7. As it is a stress test, it had repetitive data and was only testing narrow code paths.
The main point I got from the reply is that the program should be clear in why it failed, not just freeze or disappear. We are probably all in agreement on that. Efforts will be made on that front.
I still don't see this being run as a test on a per-patch basis, but rather a periodic activity on the weekend, lest our builds take 4-5 hours instead of 20 min.
Quoted 42 lines · Jérémie Galarneau
CC-ing lttng-dev since this applies to Babeltrace.
Quoted 5 lines · Matthew Khouzam
Hi,
I was looking into the CTF stress tests.
They are good, but I don't know if we want them in our standard test cases. They basically check the scalability of the computer it is being run on in most cases and in all cases, the test of reading 12g of filesI respectfully disagree.
In most tests, there is definitely a smart way to deal with the trace. That is not to say that all implementations are expected to support every imaginable test cases. However, they should strive to, at the very least, gracefully handle failures to read traces and document their limitations as the spec imposes very few.
While I agree that some of these tests are far-fetched (you're not likely to see a CTF trace with a 32 768 level deep structure any time soon), traces with enough streams to exhaust the max fd count are not far-fetched at all. In fact, tracing any decently-sized cluster will bust that limit in no time.
Handling a multi-megabyte sequence (~100mb), something that Babeltrace can't do at the moment, may seem unreasonable at first. It quickly becomes very pertinent when people start talking of core dumping to a CTF trace.
Quoted 4 lines
is rather prohibitive. That being said, maybe having this as a weekly build could be an interesting idea. Also, I don't think the idea that busting the heap size is a good indication of a test failure. I can always allocate more heap at the problem. :)
Unfortunately, my motherboard's DIMM slots are already full. ;-)
Quoted 8 lines
Now for a breakdown of the tests:
├── gen.sh ├── metadata │ └── pass │ ├── large-metadata - interesting test, I think something likethis should be used to improve robustness
│ ├── long-identifier - this should be our approach here, I think
http://stackoverflow.com/questions/6007568/what-is-max-length-for-an-c-c-identifier-on-common-build-systems
Quoted 2 lines
If you mean that an implementation should document its limitations, I agree.
+1
Quoted 11 lines
Quoted 6 lines
│ ├── many-callsites - works here until we oome but it's slow to
test and we don't have a real application to validate the data yet. So it's good, but we may wish to put efforts elsewhere for now.
│ ├── many-stream-class - This test is looking at the max size of
an array...Not sure what you mean here although I agree that reading a trace with 16 million stream classes is not a "realistic" test.
That's not the point of the CTF stress test suite. This test suite will expose errors that are not handled gracefully.
I am guessing this is the main issue, and should be addressed first. Thank you for highlighting it.
Quoted 38 lines
Implementations are
expected to fail, but to do so in a controlled and documented way; not silently or by triggering the OOM-killer.Quoted 5 lines
│ ├── many-typealias - interesting test, the parser will suffer. │ └── many-typedef - ditto └── stream └── pass ├── array-large - Testing the max size of ram and arrays.There is no need to load the entire array in RAM. I, personally, intend on implementing a slow fallback to disk when arrays exceed a given size.
This will not work in live mode and that's completely okay. As long as it doesn't outright crash.
There is also a security aspect to this; an implementation shouldn't trust the relay daemon to send packets it can handle safely. This doesn't really apply to arrays as their length is statically known, but unchecked sequence length are definitely a
security/stabilityconcern.Quoted 2 lines
├── many-events - This can be an interesting tool for profiling.
But it's not really a test for the reader...Not sure why. Tracing multiple UST applications over a long time is bound to stress this at some point.
Just imagine a snapshot session running for a year, through multiple application updates. The number of UST tracepoints can quickly become humongous.
Quoted 4 lines
├── many-packets - This will be good to test distributed indexes ├── many-streams - Good, but even better is... ├── many-traces - interesting and a real problem ├── packet-large - this is actually easier than many-eventsSpeak for yourself! Babeltrace mmaps entire packets! ;-) We have our own share of embarrassing limitations :-P
Quoted 6 lines
Quoted 5 lines
├── sequence-large - see array-large ├── string-large - see array-large ├── struct-many-fields - see array-large ├── struct-nest-n-deep - I like this one, it highlights one ofour optimisations
Hmm, interesting! Care to explain the gist of it?
We have a struct flatenner, if a struct only contains structs, the scope is maintained with a separator and the value are brought up a level. This means that your nest-n-deep tests are always of a depth or 1 for the reader. Please ping me for more details. :)
Reading the struct still does awful things to the stack though. Migrating to Antlr4 and Java 8 would help that probably.
Quoted 11 lines
Quoted 1 line
└── variant-many-tags - see array-large
Again, this is something I can easily see being produced by a dynamically-typed language. Imagine a VM implementation that would trace every function call for every possible argument type permutation.
Quoted 1 line
Why no deep variants? just curious.
Good point, we should add that!
We should also have nested sequences, arrays and mixes of variants, sequences and arrays.
Quoted 7 lines
Our assumption is that the nesting-handling code is shared between various types; the nested-structures test should ideally trigger it.
Perhaps we should not presume such implementation details... The test suite is still a work in progress; any weird test case you can think of is welcome!
A deep nested something that has an int(1 bit) at every level A deep nested something that has a sequence at every odd level and it's size at every even level.
Quoted 8 lines
Quoted 5 lines
so, tl;dr the tests are good at making a system suffer and qualifying its boundaries. I think this is great for profiling, but for actual testing, it should not be part of the per-patch system we have set up.
Any thoughts?
I think the real problem is assuming that a failing test is necessarily a deal-breaker.
Quoted 7 lines
I see this test suite as CTF's ACID equivalent [1]. Most web browsers knowingly fail this test, it doesn't make it any less relevant.
Jérémie
_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev