Fix bpf_link grace period wait for tracepoints
Kumar Kartikeya Dwivedi
maintainerv2 cover Fix bpf_link grace period wait for tracepoints
A recent change to non-faultable tracepoints switched from preempt-disabled critical sections to SRCU-fast, which breaks assumptions in the bpf_link_free() path. Use call_srcu() to fix the breakage. While at it, retire rcu_trace_implies_rcu_gp() and clean up all in-tree users.
Changelog:
----------
v1 -> v2
v1: https://lore.kernel.org/bpf/[email protected]
* Add Reviewed-by tags. (Paul, Puranjay)
* Adjust commit descriptions and comments to clarify intent. (Puranjay)
Kumar Kartikeya Dwivedi (2):
bpf: Fix grace period wait for tracepoint bpf_link
bpf: Retire rcu_trace_implies_rcu_gp()
include/linux/rcupdate.h | 12 ----------- include/linux/tracepoint.h | 8 +++++++ kernel/bpf/core.c | 10 ++++----- kernel/bpf/helpers.c | 2 +- kernel/bpf/memalloc.c | 33 +++++++++------------------- kernel/bpf/syscall.c | 44 +++++++++++++++++++++++--------------- 6 files changed, 50 insertions(+), 59 deletions(-)
base-commit: c369299895a591d96745d6492d4888259b004a9e
Signature · 1 line
2.52.0
Kumar Kartikeya Dwivedi
maintainerPATCH v2 1/2 bpf: Fix grace period wait for tracepoint bpf_link
Recently, tracepoints were switched from using disabled preemption (which acts as RCU read section) to SRCU-fast when they are not faultable. This means that to do a proper grace period wait for programs running in such tracepoints, we must use SRCU's grace period wait. This is only for non-faultable tracepoints, faultable ones continue using RCU Tasks Trace.
However, bpf_link_free() currently does call_rcu() for all cases when the link is non-sleepable (hence, for tracepoints, non-faultable). Fix this by doing a call_srcu() grace period wait.
As far RCU Tasks Trace gp -> RCU gp chaining is concerned, it is deemed unnecessary for tracepoint programs. The link and program are either accessed under RCU Tasks Trace protection, or SRCU-fast protection now.
The earlier logic of chaining both RCU Tasks Trace and RCU gp waits was to generalize the logic, even if it conceded an extra RCU gp wait, however that is unnecessary for tracepoints even before this change. In practice no cost was paid since rcu_trace_implies_rcu_gp() was always true. Hence we need not chaing any RCU gp after the SRCU gp.
For instance, in the non-faultable raw tracepoint, the RCU read section of the program in __bpf_trace_run() is enclosed in the SRCU gp, likewise for faultable raw tracepoint, the program is under the RCU Tasks Trace protection. Hence, the outermost scope can be waited upon to ensure correctness.
--- include/linux/tracepoint.h | 8 ++++++++ kernel/bpf/syscall.c | 26 ++++++++++++++++++++++++-- 2 files changed, 32 insertions(+), 2 deletions(-)
include/linux/tracepoint.h +8 -0
@@ -113,6 +113,10 @@ void for_each_tracepoint_in_module(struct module *mod,*/ #ifdef CONFIG_TRACEPOINTS extern struct srcu_struct tracepoint_srcu; +static inline struct srcu_struct *tracepoint_srcu_ptr (void ) +{ +return &tracepoint_srcu; +}static inline void tracepoint_synchronize_unregister (void ) {synchronize_rcu_tasks_trace ();
@@ -123,6 +127,10 @@ static inline bool tracepoint_is_faultable(struct tracepoint *tp) return tp->ext && tp->ext->faultable; }#else +static inline struct srcu_struct *tracepoint_srcu_ptr (void ) +{ +return NULL ; +}static inline void tracepoint_synchronize_unregister (void ) { }static inline bool tracepoint_is_faultable (struct tracepoint *tp)
kernel/bpf/syscall.c +24 -2
@@ -3261,6 +3261,17 @@ static void bpf_link_defer_dealloc_rcu_gp(struct rcu_head *rcu)bpf_link_dealloc (link); } +static bool bpf_link_is_tracepoint (struct bpf_link *link) +{ +/* +* Only these combinations support a tracepoint bpf_link. +* BPF_LINK_TYPE_TRACING raw_tp progs are hardcoded to use +* bpf_raw_tp_link_lops, see bpf_raw_tp_link_attach(). +*/ +return link->type ==BPF_LINK_TYPE_RAW_TRACEPOINT || + (link->type ==BPF_LINK_TYPE_TRACING && link->attach_type ==BPF_TRACE_RAW_TP ); +} +static void bpf_link_defer_dealloc_mult_rcu_gp (struct rcu_head *rcu) {if (rcu_trace_implies_rcu_gp ())
@@ -3279,16 +3290,27 @@ static void bpf_link_free(struct bpf_link *link)if (link->prog) ops->release (link);if (ops->dealloc_deferred) { -/* Schedule BPF link deallocation, which will only then +struct srcu_struct *tp_srcu =tracepoint_srcu_ptr (); + +/* +* Schedule BPF link deallocation, which will only then * trigger putting BPF program refcount. * If underlying BPF program is sleepable or BPF link's target * attach hookpoint is sleepable or otherwise requires RCU GPs * to ensure link and its underlying BPF program is not * reachable anymore, we need to first wait for RCU tasks -* trace sync, and then go through "classic" RCU grace period +* trace sync, and then go through "classic" RCU grace period. +* +* For tracepoint BPF links, we need to go through SRCU grace +* period wait instead when non-faultable tracepoint is used. We +* don't need to chain SRCU grace period waits, however, for the +* faultable case, since it exclusively uses RCU Tasks Trace. */ if (link->sleepable || (link->prog && link->prog->sleepable))call_rcu_tasks_trace (&link->rcu, bpf_link_defer_dealloc_mult_rcu_gp); +/* We need to do a SRCU grace period wait for tracepoint-based BPF links. */ +else if (bpf_link_is_tracepoint (link) && tp_srcu) +call_srcu (tp_srcu, &link->rcu, bpf_link_defer_dealloc_rcu_gp);else call_rcu (&link->rcu, bpf_link_defer_dealloc_rcu_gp); }else if (ops->dealloc) {
Signature · 1 line
2.52.0
Re: [PATCH bpf v2 1/2] bpf: Fix grace period wait for tracepoint bpf_link
Quoted 101 lines · Kumar Kartikeya Dwivedi
Recently, tracepoints were switched from using disabled preemption (which acts as RCU read section) to SRCU-fast when they are not faultable. This means that to do a proper grace period wait for programs running in such tracepoints, we must use SRCU's grace period wait. This is only for non-faultable tracepoints, faultable ones continue using RCU Tasks Trace.
However,
bpf_link_free()currently doescall_rcu()for all cases when the link is non-sleepable (hence, for tracepoints, non-faultable). Fix this by doing acall_srcu()grace period wait.As far RCU Tasks Trace gp -> RCU gp chaining is concerned, it is deemed unnecessary for tracepoint programs. The link and program are either accessed under RCU Tasks Trace protection, or SRCU-fast protection now.
The earlier logic of chaining both RCU Tasks Trace and RCU gp waits was to generalize the logic, even if it conceded an extra RCU gp wait, however that is unnecessary for tracepoints even before this change. In practice no cost was paid since
rcu_trace_implies_rcu_gp()was always true. Hence we need not chaing any RCU gp after the SRCU gp.For instance, in the non-faultable raw tracepoint, the RCU read section of the program in
__bpf_trace_run()is enclosed in the SRCU gp, likewise for faultable raw tracepoint, the program is under the RCU Tasks Trace protection. Hence, the outermost scope can be waited upon to ensure correctness.Fixes a46023d5616e ("tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast")Reviewed-by Puranjay Mohan <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/tracepoint.h | 8 ++++++++ kernel/bpf/syscall.c | 26 ++++++++++++++++++++++++-- 2 files changed, 32 insertions(+), 2 deletions(-)include/linux/tracepoint.h +8
-0@@ -113,6 +113,10 @@ void for_each_tracepoint_in_module(struct module *mod,*/ #ifdef CONFIG_TRACEPOINTS extern struct srcu_struct tracepoint_srcu; +static inline struct srcu_struct *tracepoint_srcu_ptr (void ) +{ +return &tracepoint_srcu; +}static inline void tracepoint_synchronize_unregister (void ) {synchronize_rcu_tasks_trace ();@@ -123,6 +127,10 @@ static inline bool tracepoint_is_faultable(struct tracepoint *tp) return tp->ext && tp->ext->faultable; }#else +static inline struct srcu_struct *tracepoint_srcu_ptr (void ) +{ +return NULL ; +}static inline void tracepoint_synchronize_unregister (void ) { }static inline bool tracepoint_is_faultable (struct tracepoint *tp)kernel/bpf/syscall.c +17
-2@@ -3261,6 +3261,17 @@ static void bpf_link_defer_dealloc_rcu_gp(struct rcu_head *rcu)bpf_link_dealloc (link); } +static bool bpf_link_is_tracepoint (struct bpf_link *link) +{ +/* +* Only these combinations support a tracepoint bpf_link. +* BPF_LINK_TYPE_TRACING raw_tp progs are hardcoded to use +* bpf_raw_tp_link_lops, see bpf_raw_tp_link_attach(). +*/ +return link->type ==BPF_LINK_TYPE_RAW_TRACEPOINT || + (link->type ==BPF_LINK_TYPE_TRACING && link->attach_type ==BPF_TRACE_RAW_TP ); +} +static void bpf_link_defer_dealloc_mult_rcu_gp (struct rcu_head *rcu) {if (rcu_trace_implies_rcu_gp ())@@ -3279,16 +3290,27 @@ static void bpf_link_free(struct bpf_link *link)if (link->prog) ops->release (link);if (ops->dealloc_deferred) { -/* Schedule BPF link deallocation, which will only then +struct srcu_struct *tp_srcu =tracepoint_srcu_ptr (); + +/* +* Schedule BPF link deallocation, which will only then * trigger putting BPF program refcount. * If underlying BPF program is sleepable or BPF link's target * attach hookpoint is sleepable or otherwise requires RCU GPs * to ensure link and its underlying BPF program is not * reachable anymore, we need to first wait for RCU tasks -* trace sync, and then go through "classic" RCU grace period +* trace sync, and then go through "classic" RCU grace period. +*
kernel/bpf/syscall.c static void bpf_link_free(struct bpf_link *link) · line 3313
Quoted 10 lines
kernel/bpf/syscall.c +7
-0+* For tracepoint BPF links, we need to go through SRCU grace +* period wait instead when non-faultable tracepoint is used. We +* don't need to chain SRCU grace period waits, however, for the +* faultable case, since it exclusively uses RCU Tasks Trace. */ if (link->sleepable || (link->prog && link->prog->sleepable))call_rcu_tasks_trace (&link->rcu, bpf_link_defer_dealloc_mult_rcu_gp); +/* We need to do a SRCU grace period wait for tracepoint-based BPF links. */ +else if (bpf_link_is_tracepoint (link) && tp_srcu) +call_srcu (tp_srcu, &link->rcu, bpf_link_defer_dealloc_rcu_gp);
It bothers me that we have a special rule here just for tracepoints (and non-sleepable ones only), and fetch that srcu_struct... it feels like we are missing some more fundamental link-type-specific grace period callback that could be overridden by link ops. E.g., perhaps we should be having something like
if (ops->call_rcu)
ops->call_rcu(link, bpf_link_defer_dealloc_rcu_gp);
else
/* current sleepable or non-sleepable RCU stuff */and for non-sleepable rawtps we'll have link_ops->call_rcu set to something that would fetch that tracepoint_srcu_ptr() and call call_srcu().
I don't think we need to solve this right now, but if we ever have second case where link's attach hook has some extra lifetime nuances like this, we should generalize this. For now:
Quoted 6 lines
kernel/bpf/syscall.c +0
-1else call_rcu (&link->rcu, bpf_link_defer_dealloc_rcu_gp); }else if (ops->dealloc) { --2.52.0
Re: [PATCH bpf v2 1/2] bpf: Fix grace period wait for tracepoint bpf_link
Quoted 131 lines · Andrii Nakryiko
Quoted 101 lines · Kumar Kartikeya Dwivedi
Recently, tracepoints were switched from using disabled preemption (which acts as RCU read section) to SRCU-fast when they are not faultable. This means that to do a proper grace period wait for programs running in such tracepoints, we must use SRCU's grace period wait. This is only for non-faultable tracepoints, faultable ones continue using RCU Tasks Trace.
However,
bpf_link_free()currently doescall_rcu()for all cases when the link is non-sleepable (hence, for tracepoints, non-faultable). Fix this by doing acall_srcu()grace period wait.As far RCU Tasks Trace gp -> RCU gp chaining is concerned, it is deemed unnecessary for tracepoint programs. The link and program are either accessed under RCU Tasks Trace protection, or SRCU-fast protection now.
The earlier logic of chaining both RCU Tasks Trace and RCU gp waits was to generalize the logic, even if it conceded an extra RCU gp wait, however that is unnecessary for tracepoints even before this change. In practice no cost was paid since
rcu_trace_implies_rcu_gp()was always true. Hence we need not chaing any RCU gp after the SRCU gp.For instance, in the non-faultable raw tracepoint, the RCU read section of the program in
__bpf_trace_run()is enclosed in the SRCU gp, likewise for faultable raw tracepoint, the program is under the RCU Tasks Trace protection. Hence, the outermost scope can be waited upon to ensure correctness.Fixes a46023d5616e ("tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast")Reviewed-by Puranjay Mohan <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/tracepoint.h | 8 ++++++++ kernel/bpf/syscall.c | 26 ++++++++++++++++++++++++-- 2 files changed, 32 insertions(+), 2 deletions(-)include/linux/tracepoint.h +8
-0@@ -113,6 +113,10 @@ void for_each_tracepoint_in_module(struct module *mod,*/ #ifdef CONFIG_TRACEPOINTS extern struct srcu_struct tracepoint_srcu; +static inline struct srcu_struct *tracepoint_srcu_ptr (void ) +{ +return &tracepoint_srcu; +}static inline void tracepoint_synchronize_unregister (void ) {synchronize_rcu_tasks_trace ();@@ -123,6 +127,10 @@ static inline bool tracepoint_is_faultable(struct tracepoint *tp) return tp->ext && tp->ext->faultable; }#else +static inline struct srcu_struct *tracepoint_srcu_ptr (void ) +{ +return NULL ; +}static inline void tracepoint_synchronize_unregister (void ) { }static inline bool tracepoint_is_faultable (struct tracepoint *tp)kernel/bpf/syscall.c +17
-2@@ -3261,6 +3261,17 @@ static void bpf_link_defer_dealloc_rcu_gp(struct rcu_head *rcu)bpf_link_dealloc (link); } +static bool bpf_link_is_tracepoint (struct bpf_link *link) +{ +/* +* Only these combinations support a tracepoint bpf_link. +* BPF_LINK_TYPE_TRACING raw_tp progs are hardcoded to use +* bpf_raw_tp_link_lops, see bpf_raw_tp_link_attach(). +*/ +return link->type ==BPF_LINK_TYPE_RAW_TRACEPOINT || + (link->type ==BPF_LINK_TYPE_TRACING && link->attach_type ==BPF_TRACE_RAW_TP ); +} +static void bpf_link_defer_dealloc_mult_rcu_gp (struct rcu_head *rcu) {if (rcu_trace_implies_rcu_gp ())@@ -3279,16 +3290,27 @@ static void bpf_link_free(struct bpf_link *link)if (link->prog) ops->release (link);if (ops->dealloc_deferred) { -/* Schedule BPF link deallocation, which will only then +struct srcu_struct *tp_srcu =tracepoint_srcu_ptr (); + +/* +* Schedule BPF link deallocation, which will only then * trigger putting BPF program refcount. * If underlying BPF program is sleepable or BPF link's target * attach hookpoint is sleepable or otherwise requires RCU GPs * to ensure link and its underlying BPF program is not * reachable anymore, we need to first wait for RCU tasks -* trace sync, and then go through "classic" RCU grace period +* trace sync, and then go through "classic" RCU grace period. +* kernel/bpf/syscall.c static void bpf_link_free(struct bpf_link *link) · line 3313
Quoted 10 lines
kernel/bpf/syscall.c +7
-0+* For tracepoint BPF links, we need to go through SRCU grace +* period wait instead when non-faultable tracepoint is used. We +* don't need to chain SRCU grace period waits, however, for the +* faultable case, since it exclusively uses RCU Tasks Trace. */ if (link->sleepable || (link->prog && link->prog->sleepable))call_rcu_tasks_trace (&link->rcu, bpf_link_defer_dealloc_mult_rcu_gp); +/* We need to do a SRCU grace period wait for tracepoint-based BPF links. */ +else if (bpf_link_is_tracepoint (link) && tp_srcu) +call_srcu (tp_srcu, &link->rcu, bpf_link_defer_dealloc_rcu_gp);It bothers me that we have a special rule here just for tracepoints (and non-sleepable ones only), and fetch that
srcu_struct... it feels like we are missing some more fundamental link-type-specific grace period callback that could be overridden by link ops. E.g., perhaps we should be having something likeif (
ops->call_rcu)ops->call_rcu(link, bpf_link_defer_dealloc_rcu_gp); else /* current sleepable or non-sleepable RCU stuff */and for non-sleepable rawtps we'll have
link_ops->call_rcuset to something that would fetch thattracepoint_srcu_ptr()and callcall_srcu().
Quoted 5 lines
I don't think we need to solve this right now, but if we ever have second case where link's attach hook has some extra lifetime nuances like this, we should generalize this. For now:
Acked-by Andrii Nakryiko <[email protected]>
there is also that comment in bpf_link_ops next to dealloc_deferred, see if it should be updated a bit?
Quoted 8 lines
Quoted 8 lines
kernel/bpf/syscall.c +0
-1else call_rcu (&link->rcu, bpf_link_defer_dealloc_rcu_gp); }else if (ops->dealloc) { --2.52.0
Kumar Kartikeya Dwivedi
maintainerPATCH v2 2/2 bpf: Retire rcu_trace_implies_rcu_gp()
In reply to Kumar Kartikeya DwivediRCU Tasks Trace grace period implies RCU grace period, and this guarantee is expected to remain in the future. Only BPF is the user of this predicate, hence retire the API and clean up all in-tree users.
RCU Tasks Trace is now implemented on SRCU-fast and its grace period mechanism always has at least one call to synchronize_rcu() as it is required for SRCU-fast's correctness (it replaces the smp_mb() that SRCU-fast readers skip). So, RCU-tt GP will always imply RCU GP.
--- include/linux/rcupdate.h | 12 ------------ kernel/bpf/core.c | 10 ++++------ kernel/bpf/helpers.c | 2 +- kernel/bpf/memalloc.c | 33 ++++++++++----------------------- kernel/bpf/syscall.c | 24 ++++++------------------ 5 files changed, 21 insertions(+), 60 deletions(-)
include/linux/rcupdate.h +0 -12
@@ -205,18 +205,6 @@ static inline void exit_tasks_rcu_start(void) { }static inline void exit_tasks_rcu_finish (void ) { }#endif /* #else #ifdef CONFIG_TASKS_RCU_GENERIC */ -/** -* rcu_trace_implies_rcu_gp - does an RCU Tasks Trace grace period imply an RCU grace period? -* -* As an accident of implementation, an RCU Tasks Trace grace period also -* acts as an RCU grace period. However, this could change at any time. -* Code relying on this accident must call this function to verify that -* this accident is still happening. -* -* You have been warned! -*/ -static inline bool rcu_trace_implies_rcu_gp (void ) {return true; } -/** * cond_resched_tasks_rcu_qs - Report potential quiescent states to RCU *
kernel/bpf/core.c +4 -6
@@ -2641,14 +2641,12 @@ static void __bpf_prog_array_free_sleepable_cb(struct rcu_head *rcu) {struct bpf_prog_array *progs; -/* If RCU Tasks Trace grace period implies RCU grace period, there is -* no need to call kfree_rcu(), just call kfree() directly. +/* +* RCU Tasks Trace grace period implies RCU grace period, there is no +* need to call kfree_rcu(), just call kfree() directly. */ progs = container_of(rcu, struct bpf_prog_array, rcu); -if (rcu_trace_implies_rcu_gp ()) -kfree (progs); -else -kfree_rcu (progs, rcu); +kfree (progs); }void bpf_prog_array_free_sleepable (struct bpf_prog_array *progs)
kernel/bpf/helpers.c +1 -1
@@ -1272,7 +1272,7 @@ static void bpf_async_cb_rcu_tasks_trace_free(struct rcu_head *rcu)return ; } -/* rcu_trace_implies_rcu_gp() is true and will remain so */ +/* RCU Tasks Trace grace period implies RCU grace period. */ bpf_async_cb_rcu_free (rcu); }
kernel/bpf/memalloc.c +10 -23
@@ -284,17 +284,6 @@ static void __free_rcu(struct rcu_head *head)atomic_set (&c->call_rcu_ttrace_in_progress,0 ); } -static void __free_rcu_tasks_trace (struct rcu_head *head) -{ -/* If RCU Tasks Trace grace period implies RCU grace period, -* there is no need to invoke call_rcu(). -*/ -if (rcu_trace_implies_rcu_gp ()) -__free_rcu (head); -else -call_rcu (head, __free_rcu); -} -static void enque_to_free (struct bpf_mem_cache *c,void *obj) {struct llist_node *llnode = obj;
@@ -326,12 +315,12 @@ static void do_call_rcu_ttrace(struct bpf_mem_cache *c)return ; } -/* Use call_rcu_tasks_trace() to wait for sleepable progs to finish. -* If RCU Tasks Trace grace period implies RCU grace period, free -* these elements directly, else use call_rcu() to wait for normal -* progs to finish and finally do free_one() on each element. +/* +* Use call_rcu_tasks_trace() to wait for sleepable progs to finish. +* RCU Tasks Trace grace period implies RCU grace period, so pass +* __free_rcu directly as the callback. */ -call_rcu_tasks_trace (&c->rcu_ttrace, __free_rcu_tasks_trace); +call_rcu_tasks_trace (&c->rcu_ttrace, __free_rcu); }static void free_bulk (struct bpf_mem_cache *c)
@@ -696,20 +685,18 @@ static void free_mem_alloc_no_barrier(struct bpf_mem_alloc *ma)static void free_mem_alloc (struct bpf_mem_alloc *ma) { -/* waiting_for_gp[_ttrace] lists were drained, but RCU callbacks +/* +* waiting_for_gp[_ttrace] lists were drained, but RCU callbacks * might still execute. Wait for them. * * rcu_barrier_tasks_trace() doesn't imply synchronize_rcu_tasks_trace(), * but rcu_barrier_tasks_trace() and rcu_barrier() below are only used -* to wait for the pending __free_rcu_tasks_trace() and __free_rcu(), -* so if call_rcu(head, __free_rcu) is skipped due to -* rcu_trace_implies_rcu_gp(), it will be OK to skip rcu_barrier() by -* using rcu_trace_implies_rcu_gp() as well. +* to wait for the pending __free_by_rcu(), and __free_rcu(). RCU Tasks +* Trace grace period implies RCU grace period, so all __free_rcu don't +* need extra call_rcu() (and thus extra rcu_barrier() here). */ rcu_barrier ();/* wait for __free_by_rcu */ rcu_barrier_tasks_trace ();/* wait for __free_rcu */ -if (!rcu_trace_implies_rcu_gp ()) -rcu_barrier ();free_mem_alloc_no_barrier (ma); }
kernel/bpf/syscall.c +6 -18
@@ -941,14 +941,6 @@ static void bpf_map_free_rcu_gp(struct rcu_head *rcu)bpf_map_free_in_work (container_of(rcu ,struct bpf_map ,rcu )); } -static void bpf_map_free_mult_rcu_gp (struct rcu_head *rcu) -{ -if (rcu_trace_implies_rcu_gp ()) -bpf_map_free_rcu_gp (rcu); -else -call_rcu (rcu, bpf_map_free_rcu_gp); -} -/* decrement map refcnt and schedule it for freeing via workqueue * (underlying map implementation ops->map_free() might sleep) */
@@ -959,8 +951,9 @@ void bpf_map_put(struct bpf_map *map)bpf_map_free_id (map);WARN_ON_ONCE (atomic64_read (&map->sleepable_refcnt)); +/* RCU tasks trace grace period implies RCU grace period. */ if (READ_ONCE (map->free_after_mult_rcu_gp)) -call_rcu_tasks_trace (&map->rcu, bpf_map_free_mult_rcu_gp); +call_rcu_tasks_trace (&map->rcu, bpf_map_free_rcu_gp);else if (READ_ONCE (map->free_after_rcu_gp))call_rcu (&map->rcu, bpf_map_free_rcu_gp);else
@@ -3272,14 +3265,6 @@ static bool bpf_link_is_tracepoint(struct bpf_link *link) (link->type ==BPF_LINK_TYPE_TRACING && link->attach_type ==BPF_TRACE_RAW_TP ); } -static void bpf_link_defer_dealloc_mult_rcu_gp (struct rcu_head *rcu) -{ -if (rcu_trace_implies_rcu_gp ()) -bpf_link_defer_dealloc_rcu_gp (rcu); -else -call_rcu (rcu, bpf_link_defer_dealloc_rcu_gp); -} -/* bpf_link_free is guaranteed to be called from process context */ static void bpf_link_free (struct bpf_link *link) {
@@ -3305,9 +3290,12 @@ static void bpf_link_free(struct bpf_link *link)* period wait instead when non-faultable tracepoint is used. We * don't need to chain SRCU grace period waits, however, for the * faultable case, since it exclusively uses RCU Tasks Trace. +* +* RCU Tasks Trace grace period implies RCU grace period, hence +* pass bpf_link_defer_dealloc_rcu_gp as callback directly. */ if (link->sleepable || (link->prog && link->prog->sleepable)) -call_rcu_tasks_trace (&link->rcu, bpf_link_defer_dealloc_mult_rcu_gp); +call_rcu_tasks_trace (&link->rcu, bpf_link_defer_dealloc_rcu_gp);/* We need to do a SRCU grace period wait for tracepoint-based BPF links. */ else if (bpf_link_is_tracepoint (link) && tp_srcu)call_srcu (tp_srcu, &link->rcu, bpf_link_defer_dealloc_rcu_gp);
Signature · 1 line
2.52.0
Re: [PATCH bpf v2 2/2] bpf: Retire rcu_trace_implies_rcu_gp()
Quoted 31 lines · Kumar Kartikeya Dwivedi
RCU Tasks Trace grace period implies RCU grace period, and this guarantee is expected to remain in the future. Only BPF is the user of this predicate, hence retire the API and clean up all in-tree users.
RCU Tasks Trace is now implemented on SRCU-fast and its grace period mechanism always has at least one call to
synchronize_rcu()as it is required for SRCU-fast's correctness (it replaces thesmp_mb()that SRCU-fast readers skip). So, RCU-tt GP will always imply RCU GP.Reviewed-by Puranjay Mohan <[email protected]>Reviewed-by Paul E. McKenney <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/rcupdate.h | 12 ------------ kernel/bpf/core.c | 10 ++++------ kernel/bpf/helpers.c | 2 +- kernel/bpf/memalloc.c | 33 ++++++++++----------------------- kernel/bpf/syscall.c | 24 ++++++------------------ 5 files changed, 21 insertions(+), 60 deletions(-)include/linux/rcupdate.h +0
-2@@ -205,18 +205,6 @@ static inline void exit_tasks_rcu_start(void) { }static inline void exit_tasks_rcu_finish (void ) { }#endif /* #else #ifdef CONFIG_TASKS_RCU_GENERIC */ -/** -* rcu_trace_implies_rcu_gp - does an RCU Tasks Trace grace period imply an RCU grace period?
include/linux/rcupdate.h static inline void exit_tasks_rcu_start(void) { } · line 219
Quoted 10 lines
include/linux/rcupdate.h +0
-10-* -* As an accident of implementation, an RCU Tasks Trace grace period also -* acts as an RCU grace period. However, this could change at any time. -* Code relying on this accident must call this function to verify that -* this accident is still happening. -* -* You have been warned! -*/ -static inline bool rcu_trace_implies_rcu_gp (void ) {return true; } -
This is not a fix, so respin to bpf-next, and probably as separate patch.
pw-bot: cr