tracing/kmemtrace: normalize the raw tracer event to the unified tracing API

Pekka, note that I would be pleased to add statistical tracing on this tracer, but I would need a hashtable, or an array, or a list, or whatever iterable to insert the data into the stat tracing api.

But I don't know your projects about this... whether you wanted to use a section or something else...

Pekka, note that I would be pleased to add statistical tracing on this tracer, but I would need a hashtable, or an array, or a list, or whatever iterable to insert the data into the stat tracing api.

But I don't know your projects about this... whether you wanted to use a section or something else...

Hi Frederic,

Quoted 6 lines · Frederic Weisbecker

Pekka, note that I would be pleased to add statistical tracing on this tracer, but I would need a hashtable, or an array, or a list, or whatever iterable to insert the data into the stat tracing api.

But I don't know your projects about this... whether you wanted to use a section or something else...

It really depends on what we're tracing. If we're interested in just the allocation hotspots, a section will do just fine. However, if we're tracing memory footprint, we need to take into store the object pointer returned from kmalloc() and kmem_cache_alloc() so we can update call-site statistics properly upon kfree().

So I suppose we need both, a section for per call-site statistics and a hash table for the object -> call-site mapping.

Pekka

Hi Frederic,

Quoted 6 lines · Frederic Weisbecker

Pekka, note that I would be pleased to add statistical tracing on this tracer, but I would need a hashtable, or an array, or a list, or whatever iterable to insert the data into the stat tracing api.

But I don't know your projects about this... whether you wanted to use a section or something else...

It really depends on what we're tracing. If we're interested in just the allocation hotspots, a section will do just fine. However, if we're tracing memory footprint, we need to take into store the object pointer returned from kmalloc() and kmem_cache_alloc() so we can update call-site statistics properly upon kfree().

So I suppose we need both, a section for per call-site statistics and a hash table for the object -> call-site mapping.

3)

the most lowlevel (and hence most allocation-footprint sensitive) object to track would be the memory object itself. I think the best approach would be to do a static, limited size hash that could track up to N memory objects.

The advantage of such an approach is that it does not impact allocation patterns at all (besides the one-time allocation cost of the hash itself during tracer startup).

And this too would be driven from ftrace mainly - the SLAB code would only offer the alloc+free callbacks with the object IDs. [ and this means that we could detect memory leaks by looking at the hash table and print out the age of entries :-) ]

Quoted 10 lines · Ingo Molnar

3)

the most lowlevel (and hence most allocation-footprint sensitive) object to track would be the memory object itself. I think the best approach would be to do a static, limited size hash that could track up to N memory objects.

The advantage of such an approach is that it does not impact allocation patterns at all (besides the one-time allocation cost of the hash itself during tracer startup).

kmemtrace-user handles this by analysing offline :). I presume you could get around this by discarding every hash collision in a well-sized hashtable. The hashing algo in kmemtrace-user performs okay, considering it fills the hashtable almost entirely, but I presume you're doing that in-kernel and using other available code.

Quoted 4 lines

And this too would be driven from ftrace mainly - the SLAB code would only offer the alloc+free callbacks with the object IDs. [ and this means that we could detect memory leaks by looking at the hash table and print out the age of entries :-) ]

Some time ago I dropped timestamps because they were not providing a good way to reorder packets in userspace. We're currently relying on a sequence number to do that. You could take that as 'age', but it's not temporally-meaningful.

1)

i think the call_site based tracking should be a built-in capability - the branch tracer needs that too for example. That would also make it very simple on the usage place: you wouldnt have to worry about sections in slub.c/etc.

2)

i think a possibly useful intermediate object would be the slab cache itself, which could be the basis for some highlevel stats too. It would probably overlap /proc/slabinfo statistics but it's a natural part of this abstraction i think.

3)

the most lowlevel (and hence most allocation-footprint sensitive) object to track would be the memory object itself. I think the best approach would be to do a static, limited size hash that could track up to N memory objects.

The advantage of such an approach is that it does not impact allocation patterns at all (besides the one-time allocation cost of the hash itself during tracer startup).

The disadvantage is when an overflow happens: the sizing heuristics would get the size correct most of the time anyway, so it's not a practical issue. There would be some sort of sizing control similar to /debug/tracing/buffer_size_kb, and a special trace entry that signals an 'overflow' of the hash table. (in that case we wont track certain objects
- but it would be clear from the trace output what happens and the hash size can be adjusted.)

Another advantage would be that it would trivially not interact with any allocator - because the hash itself would never 'allocate' in any dynamic way. Either there are free entries available (in which case we use it), or not - in which case we emit an hash-overflow trace entry.

And this too would be driven from ftrace mainly - the SLAB code would only offer the alloc+free callbacks with the object IDs. [ and this means that we could detect memory leaks by looking at the hash table and print out the age of entries :-) ]

How does this sound to you?

Hi Ingo,

Quoted 7 lines · Ingo Molnar

1)

i think the call_site based tracking should be a built-in capability - the branch tracer needs that too for example. That would also make it very simple on the usage place: you wouldnt have to worry about sections in slub.c/etc.

Quoted 6 lines

2)

i think a possibly useful intermediate object would be the slab cache itself, which could be the basis for some highlevel stats too. It would probably overlap /proc/slabinfo statistics but it's a natural part of this abstraction i think.

Makes sense but keep in mind that this is really just an extension to SLUB statistics and is only good for detecting allocation hotspots, not for analyzing memory footprint.

Quoted 29 lines · Ingo Molnar

3)

the most lowlevel (and hence most allocation-footprint sensitive) object to track would be the memory object itself. I think the best approach would be to do a static, limited size hash that could track up to N memory objects.

The advantage of such an approach is that it does not impact allocation patterns at all (besides the one-time allocation cost of the hash itself during tracer startup).

The disadvantage is when an overflow happens: the sizing heuristics would get the size correct most of the time anyway, so it's not a practical issue. There would be some sort of sizing control similar to /debug/tracing/buffer_size_kb, and a special trace entry that signals an 'overflow' of the hash table. (in that case we wont track certain objects
- but it would be clear from the trace output what happens and the hash size can be adjusted.)

Another advantage would be that it would trivially not interact with any allocator - because the hash itself would never 'allocate' in any dynamic way. Either there are free entries available (in which case we use it), or not - in which case we emit an hash-overflow trace entry.

And this too would be driven from ftrace mainly - the SLAB code would only offer the alloc+free callbacks with the object IDs. [ and this means that we could detect memory leaks by looking at the hash table and print out the age of entries :-) ]

Quoted 1 line

How does this sound to you?

That will probably be okay for things like analyzing memory footprint immediately after boot. However, as soon as the amount of active memory objects increases (think dentry and inode cache), the numbers might get skewed. One option would be to let the user exclude some of the caches from tracing.

Quoted 10 lines · Pekka Enberg

Hi Frederic,

Quoted 6 lines · Frederic Weisbecker

Pekka, note that I would be pleased to add statistical tracing on this tracer, but I would need a hashtable, or an array, or a list, or whatever iterable to insert the data into the stat tracing api.

But I don't know your projects about this... whether you wanted to use a section or something else...

Quoted 8 lines

It really depends on what we're tracing. If we're interested in just the allocation hotspots, a section will do just fine. However, if we're tracing memory footprint, we need to take into store the object pointer returned from kmalloc() and kmem_cache_alloc() so we can update call-site statistics properly upon kfree().

So I suppose we need both, a section for per call-site statistics and a hash table for the object -> call-site mapping.

1)

i think the call_site based tracking should be a built-in capability - the branch tracer needs that too for example. That would also make it very simple on the usage place: you wouldnt have to worry about sections in slub.c/etc.

2)

i think a possibly useful intermediate object would be the slab cache itself, which could be the basis for some highlevel stats too. It would probably overlap /proc/slabinfo statistics but it's a natural part of this abstraction i think.

3)

the most lowlevel (and hence most allocation-footprint sensitive) object to track would be the memory object itself. I think the best approach would be to do a static, limited size hash that could track up to N memory objects.
The advantage of such an approach is that it does not impact allocation patterns at all (besides the one-time allocation cost of the hash itself during tracer startup).

The disadvantage is when an overflow happens: the sizing heuristics would get the size correct most of the time anyway, so it's not a practical issue. There would be some sort of sizing control similar to /debug/tracing/buffer_size_kb, and a special trace entry that signals an 'overflow' of the hash table. (in that case we wont track certain objects
- but it would be clear from the trace output what happens and the hash size can be adjusted.)

Another advantage would be that it would trivially not interact with any allocator - because the hash itself would never 'allocate' in any dynamic way. Either there are free entries available (in which case we use it), or not - in which case we emit an hash-overflow trace entry.

And this too would be driven from ftrace mainly - the SLAB code would only offer the alloc+free callbacks with the object IDs. [ and this means that we could detect memory leaks by looking at the hash table and print out the age of entries :-) ]

How does this sound to you?

Ingo

Quoted 22 lines · Ingo Molnar

Quoted 10 lines · Pekka Enberg

Hi Frederic,

Quoted 6 lines · Frederic Weisbecker

Pekka, note that I would be pleased to add statistical tracing on this tracer, but I would need a hashtable, or an array, or a list, or whatever iterable to insert the data into the stat tracing api.

But I don't know your projects about this... whether you wanted to use a section or something else...

Quoted 8 lines

It really depends on what we're tracing. If we're interested in just the allocation hotspots, a section will do just fine. However, if we're tracing memory footprint, we need to take into store the object pointer returned from kmalloc() and kmem_cache_alloc() so we can update call-site statistics properly upon kfree().

So I suppose we need both, a section for per call-site statistics and a hash table for the object -> call-site mapping.

Quoted 6 lines

1)

i think the call_site based tracking should be a built-in capability - the branch tracer needs that too for example. That would also make it very simple on the usage place: you wouldnt have to worry about sections in slub.c/etc.

I think that too. Can we use sections here? The traced functions are not directly kmalloc/kmem_cache_alloc and to use a section which contains the per site allocation requests, such a thing is required (we can't build a section with per site allocations requests by using intermediate level allocation function I fear...).

Quoted 31 lines · Frederic Weisbecker

Quoted 22 lines · Ingo Molnar

Quoted 10 lines · Pekka Enberg

Hi Frederic,

Quoted 6 lines · Frederic Weisbecker

Pekka, note that I would be pleased to add statistical tracing on this tracer, but I would need a hashtable, or an array, or a list, or whatever iterable to insert the data into the stat tracing api.

But I don't know your projects about this... whether you wanted to use a section or something else...

Quoted 8 lines

It really depends on what we're tracing. If we're interested in just the allocation hotspots, a section will do just fine. However, if we're tracing memory footprint, we need to take into store the object pointer returned from kmalloc() and kmem_cache_alloc() so we can update call-site statistics properly upon kfree().

So I suppose we need both, a section for per call-site statistics and a hash table for the object -> call-site mapping.

Quoted 6 lines

1)

i think the call_site based tracking should be a built-in capability - the branch tracer needs that too for example. That would also make it very simple on the usage place: you wouldnt have to worry about sections in slub.c/etc.

Quoted 5 lines

I think that too. Can we use sections here? The traced functions are not directly kmalloc/kmem_cache_alloc and to use a section which contains the per site allocation requests, such a thing is required (we can't build a section with per site allocations requests by using intermediate level allocation function I fear...).

i think initially this should be a fixed-size allocation array + hash as well. (like lockdep uses) The number of allocation sites is even the most extreme case at most a few thousand - and is typically at most a couple of hundred.

Ingo

Hi Frederic,

Quoted 6 lines · Frederic Weisbecker

Pekka, note that I would be pleased to add statistical tracing on this tracer, but I would need a hashtable, or an array, or a list, or whatever iterable to insert the data into the stat tracing api.

But I don't know your projects about this... whether you wanted to use a section or something else...

It really depends on what we're tracing. If we're interested in just the allocation hotspots, a section will do just fine. However, if we're tracing memory footprint, we need to take into store the object pointer returned from kmalloc() and kmem_cache_alloc() so we can update call-site statistics properly upon kfree().

So I suppose we need both, a section for per call-site statistics and a hash table for the object -> call-site mapping.

Pekka

Pekka, note that I would be pleased to add statistical tracing on this tracer, but I would need a hashtable, or an array, or a list, or whatever iterable to insert the data into the stat tracing api.

Impact: new tracer

This patch adapts kmemtrace raw events tracing to the unified tracing API. It applies on latest tip/tracing/kmemtrace To enable and use this tracer, just do the following:

echo kmemtrace > /debugfs/tracing/current_tracer
cat /debugfs/tracing/trace

 kernel/trace/trace_kmemtrace.c |  345 ++++++++++++++++++++++++++++++++++++++++

Quoted 1 line · Frederic Weisbecker

 kernel/trace/trace_kmemtrace.c |  345 ++++++++++++++++++++++++++++++++++++++++

btw., i renamed this to kernel/trace/kmemtrace.c. Mentioning 'trace' twice is enough already ;-)

Quoted 7 lines · Ingo Molnar

Quoted 1 line · Frederic Weisbecker

 kernel/trace/trace_kmemtrace.c |  345 ++++++++++++++++++++++++++++++++++++++++

btw., i renamed this to kernel/trace/kmemtrace.c. Mentioning 'trace' twice is enough already ;-)

Should we rename all the kernel/trace/trace_(.*)\.c to kernel/trace/\1.c ?

Although I do like the trace in the function name, because it groups them nicely together, and keeps ftrace.c and ring_buffer.c stand out from the rest.

-- Steve