bpf: defer bpf_link dealloc to after RCU grace period
Andrii Nakryiko
maintainerRFC PATCH 1/1 bpf: defer bpf_link dealloc to after RCU grace period
BPF link for some program types is passed as a "context" which can be used by those BPF programs to look up additional information. E.g., for BPF raw tracepoints, link is used to fetch BPF cookie value, similarly for BPF multi-kprobes and multi-uprobes.
Because of this runtime dependency, when bpf_link refcnt drops to zero that could be still active BPF programs running accessing link data (cookie, program pointer, etc).
This patch accommodates this by delaying freeing memory to after RCU GP, which will fix BPF raw tp, multi-kprobe, and non-sleepable multi-uprobe.
Perhaps a better approach would be to have a per-link flag specifying desired behavior: no delay, RCU delay, or task_trace RCU delay? So sending this as an RFC fix to discuss desired final solution.
--- include/linux/bpf.h | 8 +++++++- kernel/bpf/syscall.c | 12 ++++++++++-- 2 files changed, 17 insertions(+), 3 deletions(-)
include/linux/bpf.h +7 -1
@@ -1573,7 +1573,13 @@ struct bpf_link {enum bpf_link_type type;const struct bpf_link_ops *ops;struct bpf_prog *prog; -struct work_struct work; +/* rcu is used before freeing, work can be used to schedule that +* RCU-based freeing before that, so they never overlap +*/ +union { +struct rcu_head rcu; +struct work_struct work; + }; };struct bpf_link_ops {
kernel/bpf/syscall.c +10 -2
@@ -3024,6 +3024,14 @@ void bpf_link_inc(struct bpf_link *link)atomic64_inc (&link->refcnt); } +static void bpf_link_dealloc_deferred (struct rcu_head *rcu) +{ + struct bpf_link *link = container_of(rcu, struct bpf_link, rcu); + +/* free bpf_link and its containing memory */ + link->ops->dealloc (link); +} +/* bpf_link_free is guaranteed to be called from process context */ static void bpf_link_free (struct bpf_link *link) {
@@ -3033,8 +3041,8 @@ static void bpf_link_free(struct bpf_link *link) link->ops->release (link);bpf_prog_put (link->prog); } -/* free bpf_link and its containing memory */ - link->ops->dealloc (link); +/* schedule BPF link deallocation after RCU grace period */ +call_rcu (&link->rcu, bpf_link_dealloc_deferred); }static void bpf_link_put_deferred (struct work_struct *work)
Signature · 1 line
2.43.0
Quoted 66 lines · Andrii Nakryiko
BPF link for some program types is passed as a "context" which can be used by those BPF programs to look up additional information. E.g., for BPF raw tracepoints, link is used to fetch BPF cookie value, similarly for BPF multi-kprobes and multi-uprobes.
Because of this runtime dependency, when
bpf_linkrefcnt drops to zero that could be still active BPF programs running accessing link data (cookie, program pointer, etc).This patch accommodates this by delaying freeing memory to after RCU GP, which will fix BPF raw tp, multi-kprobe, and non-sleepable multi-uprobe.
Perhaps a better approach would be to have a per-link flag specifying desired behavior: no delay, RCU delay, or
task_traceRCU delay? So sending this as an RFC fix to discuss desired final solution.Fixes d4dfc5700e86 ("bpf: pass whole link instead of prog when triggering raw tracepoint")Reported-by [email protected]Reported-by [email protected]Reported-by [email protected]Signed-off-by Andrii Nakryiko <[email protected]>--- include/linux/bpf.h | 8 +++++++- kernel/bpf/syscall.c | 12 ++++++++++-- 2 files changed, 17 insertions(+), 3 deletions(-)include/linux/bpf.h +7
-1@@ -1573,7 +1573,13 @@ struct bpf_link {enum bpf_link_type type;const struct bpf_link_ops *ops;struct bpf_prog *prog; -struct work_struct work; +/* rcu is used before freeing, work can be used to schedule that +* RCU-based freeing before that, so they never overlap +*/ +union { +struct rcu_head rcu; +struct work_struct work; + }; };struct bpf_link_ops {kernel/bpf/syscall.c +8
-0@@ -3024,6 +3024,14 @@ void bpf_link_inc(struct bpf_link *link)atomic64_inc (&link->refcnt); } +static void bpf_link_dealloc_deferred (struct rcu_head *rcu) +{ + struct bpf_link *link = container_of(rcu, struct bpf_link, rcu); + +/* free bpf_link and its containing memory */ + link->ops->dealloc (link); +} +/* bpf_link_free is guaranteed to be called from process context */ static void bpf_link_free (struct bpf_link *link) {
kernel/bpf/syscall.c static void bpf_link_free(struct bpf_link *link) · line 3045
Quoted 7 lines
kernel/bpf/syscall.c +2
-2@@ -3033,8 +3041,8 @@ static void bpf_link_free(struct bpf_link *link) link->ops->release (link);bpf_prog_put (link->prog); } -/* free bpf_link and its containing memory */ - link->ops->dealloc (link); +/* schedule BPF link deallocation after RCU grace period */ +call_rcu (&link->rcu, bpf_link_dealloc_deferred);
Do we have any sleepable progs that access link as well? If so then call_rcu_tasks_trace ?
and check rcu_trace_implies_rcu_gp to avoid extra call_rcu.
Doing that for all link types like networking that don't look into link is not great. Probably needs to look into link type? Or look into prog->type and do the extra work only for raw_tp, multi-[ku]probe ?
Quoted 70 lines · Alexei Starovoitov
Quoted 68 lines · Andrii Nakryiko
BPF link for some program types is passed as a "context" which can be used by those BPF programs to look up additional information. E.g., for BPF raw tracepoints, link is used to fetch BPF cookie value, similarly for BPF multi-kprobes and multi-uprobes.
Because of this runtime dependency, when
bpf_linkrefcnt drops to zero that could be still active BPF programs running accessing link data (cookie, program pointer, etc).This patch accommodates this by delaying freeing memory to after RCU GP, which will fix BPF raw tp, multi-kprobe, and non-sleepable multi-uprobe.
Perhaps a better approach would be to have a per-link flag specifying desired behavior: no delay, RCU delay, or
task_traceRCU delay? So sending this as an RFC fix to discuss desired final solution.Fixes d4dfc5700e86 ("bpf: pass whole link instead of prog when triggering raw tracepoint")Reported-by [email protected]Reported-by [email protected]Reported-by [email protected]Signed-off-by Andrii Nakryiko <[email protected]>--- include/linux/bpf.h | 8 +++++++- kernel/bpf/syscall.c | 12 ++++++++++-- 2 files changed, 17 insertions(+), 3 deletions(-)include/linux/bpf.h +7
-1@@ -1573,7 +1573,13 @@ struct bpf_link {enum bpf_link_type type;const struct bpf_link_ops *ops;struct bpf_prog *prog; -struct work_struct work; +/* rcu is used before freeing, work can be used to schedule that +* RCU-based freeing before that, so they never overlap +*/ +union { +struct rcu_head rcu; +struct work_struct work; + }; };struct bpf_link_ops {kernel/bpf/syscall.c +8
-0@@ -3024,6 +3024,14 @@ void bpf_link_inc(struct bpf_link *link)atomic64_inc (&link->refcnt); } +static void bpf_link_dealloc_deferred (struct rcu_head *rcu) +{ + struct bpf_link *link = container_of(rcu, struct bpf_link, rcu); + +/* free bpf_link and its containing memory */ + link->ops->dealloc (link); +} +/* bpf_link_free is guaranteed to be called from process context */ static void bpf_link_free (struct bpf_link *link) {@@ -3033,8 +3041,8 @@ static void bpf_link_free(struct bpf_link *link) link->ops->release (link);bpf_prog_put (link->prog);
Quoted 8 lines
kernel/bpf/syscall.c static void bpf_link_free(struct bpf_link *link) · line 3045
Quoted 5 lines
kernel/bpf/syscall.c +2
-2} -/* free bpf_link and its containing memory */ - link->ops->dealloc (link); +/* schedule BPF link deallocation after RCU grace period */ +call_rcu (&link->rcu, bpf_link_dealloc_deferred);Do we have any sleepable progs that access link as well? If so then
call_rcu_tasks_trace?
yep, multi-uprobe, can be sleepable
Quoted 1 line
and check
rcu_trace_implies_rcu_gpto avoid extracall_rcu.
yep, saw that for maps
Quoted 5 lines
Doing that for all link types like networking that don't look into link is not great. Probably needs to look into link type? Or look into
prog->typeand do the extra work only forraw_tp, multi-[ku]probe ?
so I was thinking that bpf_link_init() can accept an enum or flags to specify what needs to be done, instead of trying to guess/derive that. That seems less error-prone, more explicit, easier to audit. WDYT?
This should work ok given this is normally dependent on program type, the only exception (that we currently don't have) would be if LINK_UPDATE command replaces non-sleepable BPF program (say uprobe) with sleepable one. But we don't support that for kprobe/uprobe and other tracing programs, so it's a future problem (but we'll be able to support it, when we get to this).
Quoted 89 lines · Andrii Nakryiko
Quoted 70 lines · Alexei Starovoitov
Quoted 68 lines · Andrii Nakryiko
BPF link for some program types is passed as a "context" which can be used by those BPF programs to look up additional information. E.g., for BPF raw tracepoints, link is used to fetch BPF cookie value, similarly for BPF multi-kprobes and multi-uprobes.
Because of this runtime dependency, when
bpf_linkrefcnt drops to zero that could be still active BPF programs running accessing link data (cookie, program pointer, etc).This patch accommodates this by delaying freeing memory to after RCU GP, which will fix BPF raw tp, multi-kprobe, and non-sleepable multi-uprobe.
Perhaps a better approach would be to have a per-link flag specifying desired behavior: no delay, RCU delay, or
task_traceRCU delay? So sending this as an RFC fix to discuss desired final solution.Fixes d4dfc5700e86 ("bpf: pass whole link instead of prog when triggering raw tracepoint")Reported-by [email protected]Reported-by [email protected]Reported-by [email protected]Signed-off-by Andrii Nakryiko <[email protected]>--- include/linux/bpf.h | 8 +++++++- kernel/bpf/syscall.c | 12 ++++++++++-- 2 files changed, 17 insertions(+), 3 deletions(-)include/linux/bpf.h +7
-1@@ -1573,7 +1573,13 @@ struct bpf_link {enum bpf_link_type type;const struct bpf_link_ops *ops;struct bpf_prog *prog; -struct work_struct work; +/* rcu is used before freeing, work can be used to schedule that +* RCU-based freeing before that, so they never overlap +*/ +union { +struct rcu_head rcu; +struct work_struct work; + }; };struct bpf_link_ops {kernel/bpf/syscall.c +8
-0@@ -3024,6 +3024,14 @@ void bpf_link_inc(struct bpf_link *link)atomic64_inc (&link->refcnt); } +static void bpf_link_dealloc_deferred (struct rcu_head *rcu) +{ + struct bpf_link *link = container_of(rcu, struct bpf_link, rcu); + +/* free bpf_link and its containing memory */ + link->ops->dealloc (link); +} +/* bpf_link_free is guaranteed to be called from process context */ static void bpf_link_free (struct bpf_link *link) {@@ -3033,8 +3041,8 @@ static void bpf_link_free(struct bpf_link *link) link->ops->release (link);bpf_prog_put (link->prog);Quoted 8 lines
kernel/bpf/syscall.c static void bpf_link_free(struct bpf_link *link) · line 3045
Quoted 5 lines
kernel/bpf/syscall.c +2
-2} -/* free bpf_link and its containing memory */ - link->ops->dealloc (link); +/* schedule BPF link deallocation after RCU grace period */ +call_rcu (&link->rcu, bpf_link_dealloc_deferred);Do we have any sleepable progs that access link as well? If so then
call_rcu_tasks_trace?yep, multi-uprobe, can be sleepable
Quoted 1 line
and check
rcu_trace_implies_rcu_gpto avoid extracall_rcu.yep, saw that for maps
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Doing that for all link types like networking that don't look into link is not great. Probably needs to look into link type? Or look into
prog->typeand do the extra work only forraw_tp, multi-[ku]probe ?so I was thinking that
bpf_link_init()can accept an enum or flags to specify what needs to be done, instead of trying to guess/derive that. That seems less error-prone, more explicit, easier to audit. WDYT?
yeah. makes sense.
Quoted 6 lines
This should work ok given this is normally dependent on program type, the only exception (that we currently don't have) would be if
LINK_UPDATEcommand replaces non-sleepable BPF program (say uprobe) with sleepable one. But we don't support that for kprobe/uprobe and other tracing programs, so it's a future problem (but we'll be able to support it, when we get to this).
that would be weird. future problem indeed.