ftrace: track dynamic ftrace update failures
cover ftrace: track dynamic ftrace update failures
Hi Steven/Ingo,
In dynamic ftrace any record which fails gets freed to be recycled. This happens normally during (although not limited to) tracing of init functions. In general, failures can happen due to multiple reason, such as external patching of kernel functions, instrumentation of the mcount calls-sites, hardware error etc. As an example, in the case of kprobes if a probe is installed on the mcount call site (before being converted to NOP or after) then eventually ftrace will detect it and free the corresponding record. Kprobes however will keep on single-sepping the instruction it installed the probe on.
These patches prevent freeing of records which have failed (except for init functions). This prevents re-recording of failed functions and allows tracking them so that they can be listed using a new debugfs file -> 'failures'. The main change here is preventing removal of ftrace records from ftace_hash. This way records can be looked up by their 'ip' at any time.
--
Regards,
Abhishek Sagar
Quoted 12 lines · Abhishek Sagar
Hi Steven/Ingo,
In dynamic ftrace any record which fails gets freed to be recycled. This happens normally during (although not limited to) tracing of init functions. In general, failures can happen due to multiple reason, such as external patching of kernel functions, instrumentation of the mcount calls-sites, hardware error etc. As an example, in the case of kprobes if a probe is installed on the mcount call site (before being converted to NOP or after) then eventually ftrace will detect it and free the corresponding record. Kprobes however will keep on single-sepping the instruction it installed the probe on.
Quoted 6 lines
These patches prevent freeing of records which have failed (except for init functions). This prevents re-recording of failed functions and allows tracking them so that they can be listed using a new debugfs file -> 'failures'. The main change here is preventing removal of ftrace records from
ftace_hash. This way records can be looked up by their 'ip' at any time.
Hi Abhishek,
Have you also taken into account adding and removing of modules? This can cause some funny behaviours with ftrace that can be rather dangerous.
-- Steve
Quoted 1 line · Steven Rostedt
Hi Abhishek,
Hi Steven,
Thanks for reviewing my recent patches.
Quoted 2 lines
Have you also taken into account adding and removing of modules? This can cause some funny behaviours with ftrace that can be rather dangerous.
I've tried to accommodate them (fixed the wrong use of kernel_text_address). This makes
init and module function records eligible to be freed. Only core kernel function update failures now get tracked/remembered.
There might still be a need to find a way to track external modifiers of mcount call-sites such as (but not limited to) kprobes. Consider this (rather long) contrived scenario:
1. Function foo gets recorded by ftrace.
2. Mcount call site of foo is patched with a NOP by ftraced.
3. foo gets 'enabled' by some ftrace filter rule. NOP changes to a call to the function tracer.
4. A kprobe is registered on the mcount call-site. It places a trap generating instruction at the
mcount call site and stores the call-to-function-tracer instruction elsewhere internally.
5. Tracing is disabled by end user.
6. Disabling of foo fails because Kprobe had placed a trap at the mcount call site.
7. foo still keeps getting traced because Kprobes will keep single-stepping the call-to-function-tracer instruction each time the mcount call site is executed.
Not sure what to do with such cases...
--
Regards,
Abhishek Sagar
Quoted 28 lines · Abhishek Sagar
Quoted 3 lines
Have you also taken into account adding and removing of modules? This can cause some funny behaviours with ftrace that can be rather dangerous.
I've tried to accommodate them (fixed the wrong use of
kernel_text_address). This makes
init and module function records eligible to be freed. Only core kernel function update failures now get tracked/remembered.There might still be a need to find a way to track external modifiers of mcount call-sites such as (but not limited to) kprobes. Consider this (rather long) contrived scenario:
1. Function foo gets recorded by ftrace.
2. Mcount call site of foo is patched with a NOP by ftraced.
3. foo gets 'enabled' by some ftrace filter rule. NOP changes to a call to the function tracer.
4. A kprobe is registered on the mcount call-site. It places a trap generating instruction at the
mcount call site and stores the call-to-function-tracer instruction elsewhere internally.
5. Tracing is disabled by end user.
6. Disabling of foo fails because Kprobe had placed a trap at the mcount call site.
7. foo still keeps getting traced because Kprobes will keep single-stepping the call-to-function-tracer instruction each time the mcount call site is executed.
Quoted 1 line
Not sure what to do with such cases...
Can we add an API to kprobes or perhaps a way to register code modification in ftrace, that ftrace can ignore? Let the two know about each other, but in a way that they dont need to know the internals of one another.
-- Steve
Quoted 4 lines · Steven Rostedt
Can we add an API to kprobes or perhaps a way to register code modification in ftrace, that ftrace can ignore? Let the two know about each other, but in a way that they dont need to know the internals of one another.
There's a get_kprobe() already which can tell if an address's kprobed or not. But still doesn't stop kprobe from forcing change on mcount sites and fooling ftrace like mentioned previously. The problem with maintaining a blacklist sort of thing for kprobed mcount sites may also be an
overkill since it can permanently make that function un-ftraceable.
One kprobe specific measure can be to provide notifiers in [un]register_kprobe(). Ftrace can
listen on register event to temporarily disable tracing of the corresponding function by forcing
a NOP at the mcount site just before a kprobe is placed on it, if it's already recorded/enabled.
The tracing can be enabled back when the corresponding unregister event is received. Worth a
shot?
--
Regards,
Abhishek Sagar