Skip to contentSkip to conversation

LTTng 0.27, vmap-less buffering and splice()

DISCUSSION Mathieu Desnoyers 2 participants 2 messages Oct 2, 2008 – Oct 3, 2008

lore
Share & tools
lore
#0 lore raw

Hi,

I have reworked the underlying buffering mechanism LTTng uses in the past week. I took "relay", changed its vmap() buffers for my own linked list of buffers (not vmaped), made read/write wrappers which support writing data larger than a page size and writing across page boundaries, and finally managed to create a splice_read file operation which supports that. I finally changed lttd to make it use splice() instead of a mmap() and... it worked! :-) (after a bit a debugging, clearly)

This is all in LTTng 0.27 and ltt-control 0.53 (for the lttd part). As this is an important change, testing is very welcome. If you are interested in looking in the inner details of the buffering mechanism I just did, you might also want to enable the "check for random buffer access" option in the menuconfig. It will generate warnings when offset accesses more than a page away from the previous done is requested by the client. Cases such as large data write and reentrancy over the tracer code will generate a few "cache misses", but it's supposed to be rare, and therefore not a performance concern. This self-checking feature has proven to be very useful in the early development stages, and I think it's also useful if any other tracer client want to use it :
it helps finding lack of reference locality a tracer client might have.

I am also very interested in getting numbers comparing the performance of the new buffering infrastructure with the previous one. Having much less TLB impact should improve performance.

It's all available in the git tree :
http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=summary

And the userspace packages available at http://ltt.polymtl.ca (see the "QUICKSTART")

Cheers,

Mathieu

Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
#1 lore raw

I did some performance testing between LTTng 0.26 (uses vmap buffers) and LTTng 0.31 (vmap-less buffers). I think performance got degraded in the process. Those tests only write trace data in circular buffers ("overwrite mode").

No tracing, kernel 2.6.27-rc8 :
tbench : 1862.24

LTTng 0.26, kernel 2.6.27-rc7, flight recorder, default size buffers :
tbench : 1156.37 MB/s

LTTng 0.31, kernel 2.6.27-rc8, flight recorder, default size buffers :
tbench : 942.72 MB/s

For those of the LTT community interested in LTTng performance, I think it's worth having a look at the ltt-relay-alloc.patch implementation to see if there is any obvious performance degradation source. I'll play a bit with the implementation to see where it comes from.

Mathieu

Quoted 46 lines · * Mathieu Desnoyers ([email protected]) wrote:

Hi,

I have reworked the underlying buffering mechanism LTTng uses in the past week. I took "relay", changed its vmap() buffers for my own linked list of buffers (not vmaped), made read/write wrappers which support writing data larger than a page size and writing across page boundaries, and finally managed to create a splice_read file operation which supports that. I finally changed lttd to make it use splice() instead of a mmap() and... it worked! :-) (after a bit a debugging, clearly)

This is all in LTTng 0.27 and ltt-control 0.53 (for the lttd part). As this is an important change, testing is very welcome. If you are interested in looking in the inner details of the buffering mechanism I just did, you might also want to enable the "check for random buffer access" option in the menuconfig. It will generate warnings when offset accesses more than a page away from the previous done is requested by the client. Cases such as large data write and reentrancy over the tracer code will generate a few "cache misses", but it's supposed to be rare, and therefore not a performance concern. This self-checking feature has proven to be very useful in the early development stages, and I think it's also useful if any other tracer client want to use it :
it helps finding lack of reference locality a tracer client might have.

I am also very interested in getting numbers comparing the performance of the new buffering infrastructure with the previous one. Having much less TLB impact should improve performance.

It's all available in the git tree :
http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=summary

And the userspace packages available at http://ltt.polymtl.ca (see the "QUICKSTART")

Cheers,

Mathieu

Signature · 7 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

_______________________________________________
ltt-dev mailing list
[email protected]
http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68