Skip to contentSkip to conversation

LTTng Userspace Tracing, vDSO

RFC Mathieu Desnoyers 1 participant 1 message Jan 22, 2008

lore
Share & tools
lore
#0 lore raw

Hi,

I am poking in the userspace tracing problem again. I had two different implementations in LTTng, but I left them out so I could come back with a more satisfying solution.

Here are the ideas I would like to use in my forthcoming implementation. I submit them for comments, especially about limitations of vDSO I might not be aware of (code size... ?, max data size ?).

The idea is to fast-path tracing so a system call is not required. I have to determine if I put the tracing code in a normal shared object or if it belongs to a vDSO.

Basic ideas :

* Buffers
Per CPU buffers mapped by both userspace and the kernel menuconfig options for buffers
- per process (safe, a process cannot overwrite other processes'data)
- 4k default size, menuconfig configurable
- Total of 12.5MB for 200 traced processes * 4 CPUs.. not bad.
- system-wide (unsafe, low memory consumption)
- 1MB default size, menuconfig configurable

Depending on the system type, one could wish for efficiency (single-user server) or protection (multi-user system)

Since threads can be stopped by the OS when the buffers are filled, we can afford to have smaller buffers than the kernel buffers (4k vs 1MB).

Atomic space reservation to protect against traced signal handlers. Note that a full cmpxchg (synchronized wrt other CPUs) is required because the vgetcpu only returns a statistically correct cpu ID (might be migrated).

* Synchronization
- Use seqlock to synchronize vsyscall with kernel if required.

* time source
- vsyscall to get the raw cycle count
- fallback on kernel syscall

One could ask : why don't you stay in the kernel when you have to use a syscall to get the timestamps anyway ? Well, because passing a format string and var args from userspace to the kernel would not be "polite". ;) If userspace passes wrong format strings to the tracing code, I would rather prefer the segfault to happen in userspace and not involve the kernel at all.

* marker registration

System call to register markers in library init.
Markers in special section, modify the linker script. Keep a per-process data structure (in kernel) to keep track of the registered markers. When the last process using a marker set exits, remove those markers. (refcount)

* buffer switch

buffer switch performed by a system call.
Use inotify to tell lttd (trace reading daemon) that new debugfs files are created upon first buffer switch of a userspace process.

* Filtering

Export filter data and code to user-space. (eventually)

* Multiple traces

LTTng supports writing to multiple traces at the same time. Each of them can have different filtering expressions or be in different mode (extract the data to disk when the trace is running or flight recorder mode where the data is only kept in circular buffers and always overwritten). Can be useful for multiuser systems too.

Therefore, we would have to reserve enough virtual address space in the vDSO region to put roughly 16 per-cpu buffers (?). Then, upon buffer allocation for a new trace, we would have to iterate over each process in the system; if it is being traced, we would allocate extra buffers within the tracing vDSO data region.

Any thoughts ?

Mathieu

Signature · 3 lines
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68