Skip to contentSkip to conversation

CTF stress tests

DISCUSSION Jérémie Galarneau 2 participants 2 messages Nov 20, 2014 – Nov 21, 2014 2 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

CC-ing lttng-dev since this applies to Babeltrace.

Quoted 5 lines · Matthew Khouzam

Hi,
I was looking into the CTF stress tests.
They are good, but I don't know if we want them in our standard test cases. They basically check the scalability of the computer it is being run on in most cases and in all cases, the test of reading 12g of files

I respectfully disagree.

In most tests, there is definitely a smart way to deal with the trace. That is not to say that all implementations are expected to support every imaginable test cases. However, they should strive to, at the very least, gracefully handle failures to read traces and document their limitations as the spec imposes very few.

While I agree that some of these tests are far-fetched (you're not likely to see a CTF trace with a 32 768 level deep structure any time soon), traces with enough streams to exhaust the max fd count are not far-fetched at all. In fact, tracing any decently-sized cluster will bust that limit in no time.

Handling a multi-megabyte sequence (~100mb), something that Babeltrace can't do at the moment, may seem unreasonable at first. It quickly becomes very pertinent when people start talking of core dumping to a CTF trace.

Quoted 4 lines

is rather prohibitive. That being said, maybe having this as a weekly build could be an interesting idea. Also, I don't think the idea that busting the heap size is a good indication of a test failure. I can always allocate more heap at the problem. :)

Unfortunately, my motherboard's DIMM slots are already full. ;-)

Quoted 9 lines

Now for a breakdown of the tests:

├── gen.sh
├── metadata
│   └── pass
│       ├── large-metadata - interesting test, I think something like

this should be used to improve robustness
│ ├── long-identifier - this should be our approach here, I think
http://stackoverflow.com/questions/6007568/what-is-max-length-for-an-c-c-identifier-on-common-build-systems

If you mean that an implementation should document its limitations, I agree.

Quoted 5 lines

│ ├── many-callsites - works here until we oome but it's slow to
test and we don't have a real application to validate the data yet. So it's good, but we may wish to put efforts elsewhere for now.
│ ├── many-stream-class - This test is looking at the max size of
an array...

Not sure what you mean here although I agree that reading a trace with 16 million stream classes is not a "realistic" test.

That's not the point of the CTF stress test suite. This test suite will expose errors that are not handled gracefully. Implementations are expected to fail, but to do so in a controlled and documented way; not silently or by triggering the OOM-killer.

Quoted 5 lines
│       ├── many-typealias - interesting test, the parser will suffer.
│       └── many-typedef - ditto
└── stream
    └── pass
        ├── array-large - Testing the max size of ram and arrays.

There is no need to load the entire array in RAM. I, personally, intend on implementing a slow fallback to disk when arrays exceed a given size.

This will not work in live mode and that's completely okay. As long as it doesn't outright crash.

There is also a security aspect to this; an implementation shouldn't trust the relay daemon to send packets it can handle safely. This doesn't really apply to arrays as their length is statically known, but unchecked sequence length are definitely a security/stability concern.

Quoted 2 lines

├── many-events - This can be an interesting tool for profiling.
But it's not really a test for the reader...

Not sure why. Tracing multiple UST applications over a long time is bound to stress this at some point.

Just imagine a snapshot session running for a year, through multiple application updates. The number of UST tracepoints can quickly become humongous.

Quoted 4 lines
        ├── many-packets - This will be good to test distributed indexes
        ├── many-streams - Good, but even better is...
        ├── many-traces - interesting and a real problem
        ├── packet-large - this is actually easier than many-events

Speak for yourself! Babeltrace mmaps entire packets! ;-) We have our own share of embarrassing limitations :-P

Quoted 5 lines
        ├── sequence-large - see array-large
        ├── string-large - see array-large
        ├── struct-many-fields - see array-large
        ├── struct-nest-n-deep - I like this one, it highlights one of

our optimisations

Hmm, interesting! Care to explain the gist of it?

Quoted 1 line

└── variant-many-tags - see array-large

Again, this is something I can easily see being produced by a dynamically-typed language. Imagine a VM implementation that would trace every function call for every possible argument type permutation.

Quoted 2 lines

Why no deep variants? just curious.

Good point, we should add that!
We should also have nested sequences, arrays and mixes of variants, sequences and arrays.

Our assumption is that the nesting-handling code is shared between various types; the nested-structures test should ideally trigger it.

Perhaps we should not presume such implementation details... The test suite is still a work in progress; any weird test case you can think of is welcome!

Quoted 6 lines

so, tl;dr the tests are good at making a system suffer and qualifying its boundaries. I think this is great for profiling, but for actual testing, it should not be part of the per-patch system we have set up.

Any thoughts?

I think the real problem is assuming that a failing test is necessarily a deal-breaker.

I see this test suite as CTF's ACID equivalent [1]. Most web browsers knowingly fail this test, it doesn't make it any less relevant.

Jérémie

[1] http://www.acidtests.org/

Signature · 8 lines
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): CA+jJMxuXX-h74Lxy1xrcQ5awJ_qQLVbuV0H_cjPKiYX_kArorQ@mail.gmail.com

#3 lore raw

Great points!

I like the view: this is like the acid test, since that test is not run at every build, but rather periodically. That is much more reasonable.

The reason for my initial email is that when I ran the stress test on the parser, 147 tests failed. When I tweaked the timeouts and the stack and heap size, it went down to 7. As it is a stress test, it had repetitive data and was only testing narrow code paths.

The main point I got from the reply is that the program should be clear in why it failed, not just freeze or disappear. We are probably all in agreement on that. Efforts will be made on that front.

I still don't see this being run as a test on a per-patch basis, but rather a periodic activity on the weekend, lest our builds take 4-5 hours instead of 20 min.

Quoted 42 lines · Jérémie Galarneau

CC-ing lttng-dev since this applies to Babeltrace.

Quoted 5 lines · Matthew Khouzam

Hi,
I was looking into the CTF stress tests.
They are good, but I don't know if we want them in our standard test cases. They basically check the scalability of the computer it is being run on in most cases and in all cases, the test of reading 12g of files

I respectfully disagree.

In most tests, there is definitely a smart way to deal with the trace. That is not to say that all implementations are expected to support every imaginable test cases. However, they should strive to, at the very least, gracefully handle failures to read traces and document their limitations as the spec imposes very few.

While I agree that some of these tests are far-fetched (you're not likely to see a CTF trace with a 32 768 level deep structure any time soon), traces with enough streams to exhaust the max fd count are not far-fetched at all. In fact, tracing any decently-sized cluster will bust that limit in no time.

Handling a multi-megabyte sequence (~100mb), something that Babeltrace can't do at the moment, may seem unreasonable at first. It quickly becomes very pertinent when people start talking of core dumping to a CTF trace.

Quoted 4 lines

is rather prohibitive. That being said, maybe having this as a weekly build could be an interesting idea. Also, I don't think the idea that busting the heap size is a good indication of a test failure. I can always allocate more heap at the problem. :)

Unfortunately, my motherboard's DIMM slots are already full. ;-)

Quoted 8 lines

Now for a breakdown of the tests:

├── gen.sh
├── metadata
│   └── pass
│       ├── large-metadata - interesting test, I think something like

this should be used to improve robustness
│ ├── long-identifier - this should be our approach here, I think
http://stackoverflow.com/questions/6007568/what-is-max-length-for-an-c-c-identifier-on-common-build-systems

Quoted 2 lines

If you mean that an implementation should document its limitations, I agree.

+1

Quoted 11 lines
Quoted 6 lines

│ ├── many-callsites - works here until we oome but it's slow to
test and we don't have a real application to validate the data yet. So it's good, but we may wish to put efforts elsewhere for now.
│ ├── many-stream-class - This test is looking at the max size of
an array...

Not sure what you mean here although I agree that reading a trace with 16 million stream classes is not a "realistic" test.

That's not the point of the CTF stress test suite. This test suite will expose errors that are not handled gracefully.

I am guessing this is the main issue, and should be addressed first. Thank you for highlighting it.

Quoted 38 lines

Implementations are
expected to fail, but to do so in a controlled and documented way; not silently or by triggering the OOM-killer.

Quoted 5 lines
│       ├── many-typealias - interesting test, the parser will suffer.
│       └── many-typedef - ditto
└── stream
    └── pass
        ├── array-large - Testing the max size of ram and arrays.

There is no need to load the entire array in RAM. I, personally, intend on implementing a slow fallback to disk when arrays exceed a given size.

This will not work in live mode and that's completely okay. As long as it doesn't outright crash.

There is also a security aspect to this; an implementation shouldn't trust the relay daemon to send packets it can handle safely. This doesn't really apply to arrays as their length is statically known, but unchecked sequence length are definitely a security/stability concern.

Quoted 2 lines

├── many-events - This can be an interesting tool for profiling.
But it's not really a test for the reader...

Not sure why. Tracing multiple UST applications over a long time is bound to stress this at some point.

Just imagine a snapshot session running for a year, through multiple application updates. The number of UST tracepoints can quickly become humongous.

Quoted 4 lines
        ├── many-packets - This will be good to test distributed indexes
        ├── many-streams - Good, but even better is...
        ├── many-traces - interesting and a real problem
        ├── packet-large - this is actually easier than many-events

Speak for yourself! Babeltrace mmaps entire packets! ;-) We have our own share of embarrassing limitations :-P

Quoted 6 lines
Quoted 5 lines
        ├── sequence-large - see array-large
        ├── string-large - see array-large
        ├── struct-many-fields - see array-large
        ├── struct-nest-n-deep - I like this one, it highlights one of

our optimisations

Hmm, interesting! Care to explain the gist of it?

We have a struct flatenner, if a struct only contains structs, the scope is maintained with a separator and the value are brought up a level. This means that your nest-n-deep tests are always of a depth or 1 for the reader. Please ping me for more details. :)

Reading the struct still does awful things to the stack though. Migrating to Antlr4 and Java 8 would help that probably.

Quoted 11 lines
Quoted 1 line

└── variant-many-tags - see array-large

Again, this is something I can easily see being produced by a dynamically-typed language. Imagine a VM implementation that would trace every function call for every possible argument type permutation.

Quoted 1 line

Why no deep variants? just curious.

Good point, we should add that!
We should also have nested sequences, arrays and mixes of variants, sequences and arrays.

Quoted 7 lines

Our assumption is that the nesting-handling code is shared between various types; the nested-structures test should ideally trigger it.

Perhaps we should not presume such implementation details... The test suite is still a work in progress; any weird test case you can think of is welcome!

A deep nested something that has an int(1 bit) at every level A deep nested something that has a sequence at every odd level and it's size at every even level.

Quoted 8 lines
Quoted 5 lines

so, tl;dr the tests are good at making a system suffer and qualifying its boundaries. I think this is great for profiling, but for actual testing, it should not be part of the per-patch system we have set up.

Any thoughts?

I think the real problem is assuming that a failing test is necessarily a deal-breaker.

Quoted 7 lines

I see this test suite as CTF's ACID equivalent [1]. Most web browsers knowingly fail this test, it doesn't make it any less relevant.

Jérémie

[1] http://www.acidtests.org/

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev