Djprobes questions
Hi Masami,
I recently had to add support for inline code patching on i386 to my marker infrastructure. Clearly, it looks like what is done in djprobes, with the main difference that I only patch the immediate value of a 2 bytes "load immediate" instruction.
I think I found a solution to one of the main issues with djprobes : it currently has to wait for each CPU to hit the probe before being sure that it's safe to patch the code with something else than an int3. This is due to PIII errata 49, which says that a CPU much execute a serializing instruction before executing cross-modified code.
Here is what I do : While I use a breakpoint to fall in a trap for the CPUs that hit the site currently being modified, I also send an IPI to all CPUs so they execute cpuid. Once it returns, I am sure that every CPU has executed a serializing instruction, which enables me to go on with the complete code modification, therefore removing the initial breakpoint.
Here is my code :
(Comments are welcome)
On a second note, looking at the djprobes code triggered some question in my mind about the safety of using a worker thread to "make sure" every interrupt context has returned (so there is no IP pointing into the modified code). The following scenario might be possible : an interrupt handler (or trap handler) reenables interrupts, does irq_exit() or nmi_exit() (which reenables preemption) but does not do iret yet. My understanding is that it could be scheduled and have a return IP pointing to the code that is being modified. Am I right ?
Regards,
Mathieu
Signature · 3 lines
Mathieu Desnoyers Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Hi Mathieu,
Quoted 6 lines · Mathieu Desnoyers
Hi Masami,
I recently had to add support for inline code patching on i386 to my marker infrastructure. Clearly, it looks like what is done in djprobes, with the main difference that I only patch the immediate value of a 2 bytes "load immediate" instruction.
That's interesting.
Quoted 5 lines
I think I found a solution to one of the main issues with djprobes : it currently has to wait for each CPU to hit the probe before being sure that it's safe to patch the code with something else than an int3. This is due to PIII errata 49, which says that a CPU much execute a serializing instruction before executing cross-modified code.
Hmm, djprobe already might not wait for each CPU to hit the probe point. It just wait scheduler synchronization instead of that. And after that, it issues cpuid for cache serialization before executing cross-modified code.
The most difficult point of the djprobe is that it has to replace "live" instructions. So we must check other processors not to run those instructions carefully.
Quoted 6 lines
Here is what I do : While I use a breakpoint to fall in a trap for the CPUs that hit the site currently being modified, I also send an IPI to all CPUs so they execute cpuid. Once it returns, I am sure that every CPU has executed a serializing instruction, which enables me to go on with the complete code modification, therefore removing the initial breakpoint.
I think its OK. That is the same way which I've done in djprobe.
Quoted 6 lines
Here is my code :
(Comments are welcome)
Quoted 8 lines
On a second note, looking at the djprobes code triggered some question in my mind about the safety of using a worker thread to "make sure" every interrupt context has returned (so there is no IP pointing into the modified code). The following scenario might be possible : an interrupt handler (or trap handler) reenables interrupts, does
irq_exit()ornmi_exit()(which reenables preemption) but does not do iret yet. My understanding is that it could be scheduled and have a return IP pointing to the code that is being modified. Am I right ?
Same idea was already discussed. It might work on normal kernel, but, unfortunately, it doesn't work on preemptive kernel. And actually, that idea is same as synchronize_sched(). So, I've used it on normal kernel. In the case of preemptive kernel, currently, I'm using freeze_processes() suggested by Ingo.
Anyway, I and Satoshi are developing a static analysis tool to check whether target instructions can be replaced by long jump. I'd like to release djprobe patch against latest kernel after developed it.
Best regards,
Signature · 4 lines
Masami HIRAMATSU Linux Technology Center Hitachi, Ltd., Systems Development Laboratory E-mail: [email protected]
Quoted 12 lines · * Masami Hiramatsu ([email protected]) wrote:
Hi Mathieu,
Quoted 6 lines · Mathieu Desnoyers
Hi Masami,
I recently had to add support for inline code patching on i386 to my marker infrastructure. Clearly, it looks like what is done in djprobes, with the main difference that I only patch the immediate value of a 2 bytes "load immediate" instruction.
That's interesting.
Quoted 8 lines
Quoted 5 lines
I think I found a solution to one of the main issues with djprobes : it currently has to wait for each CPU to hit the probe before being sure that it's safe to patch the code with something else than an int3. This is due to PIII errata 49, which says that a CPU much execute a serializing instruction before executing cross-modified code.
Hmm, djprobe already might not wait for each CPU to hit the probe point. It just wait scheduler synchronization instead of that.
Oh, I see. You can do it because you don't support fully preemptible kernels. It makes sense it that scenario. I could not use this scheme for my markers though, since I support fully preemptible kernels.
Quoted 3 lines
And after that, it issues cpuid for cache serialization before executing cross-modified code.
Ok, so you already used the IPI serialization I am talking about. Great :)
Quoted 28 lines
The most difficult point of the djprobe is that it has to replace "live" instructions. So we must check other processors not to run those instructions carefully.
Quoted 6 lines
Here is what I do : While I use a breakpoint to fall in a trap for the CPUs that hit the site currently being modified, I also send an IPI to all CPUs so they execute cpuid. Once it returns, I am sure that every CPU has executed a serializing instruction, which enables me to go on with the complete code modification, therefore removing the initial breakpoint.
I think its OK. That is the same way which I've done in djprobe.
Quoted 6 lines
Here is my code :
(Comments are welcome)
Quoted 8 lines
On a second note, looking at the djprobes code triggered some question in my mind about the safety of using a worker thread to "make sure" every interrupt context has returned (so there is no IP pointing into the modified code). The following scenario might be possible : an interrupt handler (or trap handler) reenables interrupts, does
irq_exit()ornmi_exit()(which reenables preemption) but does not do iret yet. My understanding is that it could be scheduled and have a return IP pointing to the code that is being modified. Am I right ?
Quoted 6 lines
Same idea was already discussed. It might work on normal kernel, but, unfortunately, it doesn't work on preemptive kernel. And actually, that idea is same as
synchronize_sched(). So, I've used it on normal kernel. In the case of preemptive kernel, currently, I'm usingfreeze_processes()suggested by Ingo.
It looks like a good way to know where the processes are stopped.
Quoted 5 lines
Anyway, I and Satoshi are developing a static analysis tool to check whether target instructions can be replaced by long jump. I'd like to release djprobe patch against latest kernel after developed it.
Great, continue your good work!
Regards,
Mathieu
Quoted 7 lines
Signature · 4 lines
Masami HIRAMATSU Linux Technology Center Hitachi, Ltd., Systems Development Laboratory E-mail: [email protected]
Signature · 3 lines
Mathieu Desnoyers Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68