kprobes: Fix to cancel optimizing/unoptimizing kprobes correctly
Masami Hiramatsu
maintainerPATCH 1/1 kprobes: Fix to cancel optimizing/unoptimizing kprobes correctly
optimize_kprobe() and unoptimize_kprobe() cancels if given kprobe is on the optimizing_list or unoptimizing_list. However, since commit f66c0447cca1 ("kprobes: Set unoptimized flag after unoptimizing code") modified the update timing of the KPROBE_FLAG_OPTIMIZED, it doesn't work as expected anymore.
The optimized_kprobe could be following states.
- [optimizing]: Before inserting jump instruction
op.kp->flags has KPROBE_FLAG_OPTIMIZED and
op->list is not empty.
- [optimized]: jump inserted
op.kp->flags has KPROBE_FLAG_OPTIMIZED and
op->list is empty.
- [unoptimizing]: Before removing jump instruction (including unused optprobe)
op.kp->flags has KPROBE_FLAG_OPTIMIZED and
op->list is not empty.
- [unoptimized]: jump removed
op.kp->flags doesn't have KPROBE_FLAG_OPTIMIZED and op->list is empty.
Current code mis-expects [unoptimizing] state doesn't have KPROBE_FLAG_OPTIMIZED, and that can cause wrong results.
This introduces optprobe_queued_unopt() to distinguish [optimizing] and [unoptimizing] states and fixes logics in optimize_kprobe() and unoptimize_kprobe().
--- kernel/kprobes.c | 65 ++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 41 insertions(+), 24 deletions(-)
kernel/kprobes.c +41 -24
@@ -612,6 +612,16 @@ void wait_for_kprobe_optimizer(void)mutex_unlock (&kprobe_mutex); } +static bool optprobe_queued_unopt (struct optimized_kprobe *op) +{ +struct optimized_kprobe *_op; + +list_for_each_entry (_op, &unoptimizing_list, list) +if (op == _op) +return true; +return false; +} +/* Optimize kprobe if p is ready to be optimized */ static void optimize_kprobe (struct kprobe *p) {
@@ -633,17 +643,21 @@ static void optimize_kprobe(struct kprobe *p)return ;/* Check if it is already optimized. */ -if (op->kp.flags &KPROBE_FLAG_OPTIMIZED ) +if (op->kp.flags &KPROBE_FLAG_OPTIMIZED ) { +if (optprobe_queued_unopt (op)) { +/* This is under unoptimizing. Just dequeue the probe */ +list_del_init (&op->list); + }return ; + } op->kp.flags |=KPROBE_FLAG_OPTIMIZED ; -if (!list_empty (&op->list)) -/* This is under unoptimizing. Just dequeue the probe */ -list_del_init (&op->list); -else { -list_add (&op->list, &optimizing_list); -kick_kprobe_optimizer (); - } +/* On unoptimizing/optimizing_list, op must have OPTIMIZED flag */ +if (WARN_ON_ONCE (!list_empty (&op->list))) +return ; + +list_add (&op->list, &optimizing_list); +kick_kprobe_optimizer (); }/* Short cut to direct unoptimizing */
@@ -665,30 +679,33 @@ static void unoptimize_kprobe(struct kprobe *p, bool force)return ;/* This is not an optprobe nor optimized */ op = container_of(p, struct optimized_kprobe, kp); -if (!kprobe_optimized (p)) { -/* Unoptimized or unoptimizing case */ -if (force && !list_empty (&op->list)) { -/* -* Only if this is unoptimizing kprobe and forced, -* forcibly unoptimize it. (No need to unoptimize -* unoptimized kprobe again :) -*/ -list_del_init (&op->list); -force_unoptimize_kprobe (op); - } +if (!kprobe_optimized (p))return ; - }if (!list_empty (&op->list)) { -/* Dequeue from the optimization queue */ -list_del_init (&op->list); +if (optprobe_queued_unopt (op)) { +/* Queued in unoptimizing queue */ +if (force) { +/* +* Forcibly unoptimize probe here, and queue it +* in freeing list for release probe afterwards. +*/ +force_unoptimize_kprobe (op); +list_move (&op->list, &freeing_list); + } + }else { +/* Dequeue from the optimizing queue */ +list_del_init (&op->list); + op->kp.flags &= ~KPROBE_FLAG_OPTIMIZED ; + }return ; } +/* Optimized kprobe case */ -if (force) +if (force) {/* Forcibly update the code: this is a special case */ force_unoptimize_kprobe (op); -else { + }else {list_add (&op->list, &unoptimizing_list);kick_kprobe_optimizer (); }
Quoted 28 lines · On Tue, 7 Jan 2020 23:42:24 +0900 Masami Hiramatsu <[email protected]> wrote:
optimize_kprobe()andunoptimize_kprobe()cancels if given kprobe is on theoptimizing_listorunoptimizing_list. However, since commitf66c0447cca1("kprobes: Set unoptimized flag after unoptimizing code") modified the update timing of theKPROBE_FLAG_OPTIMIZED, it doesn't work as expected anymore.The
optimized_kprobecould be following states.- [optimizing]: Before inserting jump instruction
op.kp->flagshasKPROBE_FLAG_OPTIMIZEDand
op->listis not empty.- [optimized]: jump inserted
op.kp->flagshasKPROBE_FLAG_OPTIMIZEDand
op->listis empty.- [unoptimizing]: Before removing jump instruction (including unused optprobe)
op.kp->flagshasKPROBE_FLAG_OPTIMIZEDand
op->listis not empty.- [unoptimized]: jump removed
op.kp->flagsdoesn't haveKPROBE_FLAG_OPTIMIZEDandop->listis empty.Current code mis-expects [unoptimizing] state doesn't have
KPROBE_FLAG_OPTIMIZED, and that can cause wrong results.
Quoted 6 lines
This introduces
optprobe_queued_unopt()to distinguish [optimizing] and [unoptimizing] states and fixes logics inoptimize_kprobe()andunoptimize_kprobe().Fixes f66c0447cca1 ("kprobes: Set unoptimized flag after unoptimizing code")Signed-off-by Masami Hiramatsu <[email protected]>
Looks good.
kernel/kprobes.c static void unoptimize_kprobe(struct kprobe *p, bool force) · line 711
Quoted 13 lines
kernel/kprobes.c +3
-2return ; } +/* Optimized kprobe case */ -if (force) +if (force) {/* Forcibly update the code: this is a special case */ force_unoptimize_kprobe (op); -else { + }else {list_add (&op->list, &unoptimizing_list);kick_kprobe_optimizer (); }
I see you added some clean up to this patch.
-- Steve
Masami Hiramatsu
maintainerQuoted 35 lines · On Tue, 7 Jan 2020 18:39:07 -0500 Steven Rostedt <[email protected]> wrote:
Quoted 32 lines · On Tue, 7 Jan 2020 23:42:24 +0900 Masami Hiramatsu <[email protected]> wrote:
optimize_kprobe()andunoptimize_kprobe()cancels if given kprobe is on theoptimizing_listorunoptimizing_list. However, since commitf66c0447cca1("kprobes: Set unoptimized flag after unoptimizing code") modified the update timing of theKPROBE_FLAG_OPTIMIZED, it doesn't work as expected anymore.The
optimized_kprobecould be following states.- [optimizing]: Before inserting jump instruction
op.kp->flagshasKPROBE_FLAG_OPTIMIZEDand
op->listis not empty.- [optimized]: jump inserted
op.kp->flagshasKPROBE_FLAG_OPTIMIZEDand
op->listis empty.- [unoptimizing]: Before removing jump instruction (including unused optprobe)
op.kp->flagshasKPROBE_FLAG_OPTIMIZEDand
op->listis not empty.- [unoptimized]: jump removed
op.kp->flagsdoesn't haveKPROBE_FLAG_OPTIMIZEDandop->listis empty.Current code mis-expects [unoptimizing] state doesn't have
KPROBE_FLAG_OPTIMIZED, and that can cause wrong results.This introduces
optprobe_queued_unopt()to distinguish [optimizing] and [unoptimizing] states and fixes logics inoptimize_kprobe()andunoptimize_kprobe().
Quoted 6 lines
Quoted 2 lines
Fixes f66c0447cca1 ("kprobes: Set unoptimized flag after unoptimizing code")Signed-off-by Masami Hiramatsu <[email protected]>Looks good.
Reviewed-by Steven Rostedt (VMware) <[email protected]>
Thank you!
Quoted 17 lines
Quoted 5 lines
kernel/kprobes.c +1
-0return ; } +kernel/kprobes.c static void unoptimize_kprobe(struct kprobe *p, bool force) · line 711
Quoted 10 lines
kernel/kprobes.c +2
-2/* Optimized kprobe case */ -if (force) +if (force) {/* Forcibly update the code: this is a special case */ force_unoptimize_kprobe (op); -else { + }else {list_add (&op->list, &unoptimizing_list);kick_kprobe_optimizer (); }I see you added some clean up to this patch.
Yeah, I felt somewhat uncomfortable for that.
Quoted 2 lines
-- Steve
Signature · 1 line
Masami Hiramatsu <[email protected]>
[tip: core/kprobes] kprobes: Fix optimize_kprobe()/unoptimize_kprobe() cancellation logic
In reply to Masami HiramatsuThe following commit has been merged into the core/kprobes branch of tip:
The following commit has been merged into the core/kprobes branch of tip:
Commit-ID: e4add247789e4ba5e08ad8256183ce2e211877d4 Gitweb: https://git.kernel.org/tip/e4add247789e4ba5e08ad8256183ce2e211877d4 Author: Masami Hiramatsu <[email protected]> AuthorDate: Tue, 07 Jan 2020 23:42:24 +09:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Thu, 09 Jan 2020 12:40:13 +01:00
kprobes: Fix optimize_kprobe()/unoptimize_kprobe() cancellation logic
optimize_kprobe() and unoptimize_kprobe() cancels if a given kprobe is on the optimizing_list or unoptimizing_list already. However, since the following commit:
f66c0447cca1 ("kprobes: Set unoptimized flag after unoptimizing code")
modified the update timing of the KPROBE_FLAG_OPTIMIZED, it doesn't work as expected anymore.
The optimized_kprobe could be in the following states:
- [optimizing]: Before inserting jump instruction
op.kp->flags has KPROBE_FLAG_OPTIMIZED and
op->list is not empty.
- [optimized]: jump inserted
op.kp->flags has KPROBE_FLAG_OPTIMIZED and
op->list is empty.
- [unoptimizing]: Before removing jump instruction (including unused optprobe)
op.kp->flags has KPROBE_FLAG_OPTIMIZED and
op->list is not empty.
- [unoptimized]: jump removed
op.kp->flags doesn't have KPROBE_FLAG_OPTIMIZED and op->list is empty.
Current code mis-expects [unoptimizing] state doesn't have KPROBE_FLAG_OPTIMIZED, and that can cause incorrect results.
To fix this, introduce optprobe_queued_unopt() to distinguish [optimizing] and [unoptimizing] states and fixes the logic in optimize_kprobe() and unoptimize_kprobe().
[ mingo: Cleaned up the changelog and the code a bit. ]
--- kernel/kprobes.c | 67 ++++++++++++++++++++++++++++++----------------- 1 file changed, 43 insertions(+), 24 deletions(-)
kernel/kprobes.c +43 -24
@@ -612,6 +612,18 @@ void wait_for_kprobe_optimizer(void)mutex_unlock (&kprobe_mutex); } +static bool optprobe_queued_unopt (struct optimized_kprobe *op) +{ +struct optimized_kprobe *_op; + +list_for_each_entry (_op, &unoptimizing_list, list) { +if (op == _op) +return true; + } + +return false; +} +/* Optimize kprobe if p is ready to be optimized */ static void optimize_kprobe (struct kprobe *p) {
@@ -633,17 +645,21 @@ static void optimize_kprobe(struct kprobe *p)return ;/* Check if it is already optimized. */ -if (op->kp.flags &KPROBE_FLAG_OPTIMIZED ) +if (op->kp.flags &KPROBE_FLAG_OPTIMIZED ) { +if (optprobe_queued_unopt (op)) { +/* This is under unoptimizing. Just dequeue the probe */ +list_del_init (&op->list); + }return ; + } op->kp.flags |=KPROBE_FLAG_OPTIMIZED ; -if (!list_empty (&op->list)) -/* This is under unoptimizing. Just dequeue the probe */ -list_del_init (&op->list); -else { -list_add (&op->list, &optimizing_list); -kick_kprobe_optimizer (); - } +/* On unoptimizing/optimizing_list, op must have OPTIMIZED flag */ +if (WARN_ON_ONCE (!list_empty (&op->list))) +return ; + +list_add (&op->list, &optimizing_list); +kick_kprobe_optimizer (); }/* Short cut to direct unoptimizing */
@@ -665,30 +681,33 @@ static void unoptimize_kprobe(struct kprobe *p, bool force)return ;/* This is not an optprobe nor optimized */ op = container_of(p, struct optimized_kprobe, kp); -if (!kprobe_optimized (p)) { -/* Unoptimized or unoptimizing case */ -if (force && !list_empty (&op->list)) { -/* -* Only if this is unoptimizing kprobe and forced, -* forcibly unoptimize it. (No need to unoptimize -* unoptimized kprobe again :) -*/ -list_del_init (&op->list); -force_unoptimize_kprobe (op); - } +if (!kprobe_optimized (p))return ; - }if (!list_empty (&op->list)) { -/* Dequeue from the optimization queue */ -list_del_init (&op->list); +if (optprobe_queued_unopt (op)) { +/* Queued in unoptimizing queue */ +if (force) { +/* +* Forcibly unoptimize the kprobe here, and queue it +* in the freeing list for release afterwards. +*/ +force_unoptimize_kprobe (op); +list_move (&op->list, &freeing_list); + } + }else { +/* Dequeue from the optimizing queue */ +list_del_init (&op->list); + op->kp.flags &= ~KPROBE_FLAG_OPTIMIZED ; + }return ; } +/* Optimized kprobe case */ -if (force) +if (force) {/* Forcibly update the code: this is a special case */ force_unoptimize_kprobe (op); -else { + }else {list_add (&op->list, &unoptimizing_list);kick_kprobe_optimizer (); }