bpf: Do not allow deleting local storage in NMI
PATCH 1/1 bpf: Do not allow deleting local storage in NMI
Currently, local storage may deadlock when deferring freeing selem or local storage through kfree_rcu(), call_rcu() or call_rcu_tasks_trace() in NMI or reentrant. kfree_rcu() and call_rcu() have always been unsafe in NMI. call_rcu_tasks_trace() only became unsafe when it switched to call_srcu(). Since deleting selem in NMI is an unlikely use case, partially mitigate it by returning error when calling from bpf_xxx_storage_delete() helpers in NMI. The check is only added to bpf_selem_unlink(), which is used by helpers and syscalls. bpf_selem_unlink_nofail() is fine as it is called during map and owner tear down that never run in NMI or reentrant.
The long-term solution requires _nolock versions of call_rcu. Another possible solution is to defer the free through irq_work, but it would grow the size of selem, which is non-ideal.
--- kernel/bpf/bpf_local_storage.c | 3 +++ 1 file changed, 3 insertions(+)
kernel/bpf/bpf_local_storage.c +3 -0
@@ -393,6 +393,9 @@ int bpf_selem_unlink(struct bpf_local_storage_elem *selem)unsigned long flags;int err; +if (in_nmi ()) +return -EOPNOTSUPP ; +if (unlikely (!selem_linked_to_storage_lockless (selem)))/* selem has already been unlinked from sk */ return 0 ;
Signature · 1 line
2.52.0
Quoted 5 lines · Amery Hung
Currently, local storage may deadlock when deferring freeing selem or local storage through
kfree_rcu(),call_rcu()orcall_rcu_tasks_trace()in NMI or reentrant.kfree_rcu()andcall_rcu()have always been unsafe in NMI.call_rcu_tasks_trace()only became unsafe when it switched to
It has also always been unsafe in NMI, so the commit log needs an update. We should also note that the correct solution is irqs_disabled() to completely close the same context reentrancy, but it is too coarse-grained for this case, and we can revisit it once we convert to the _nolock() variant.
Quoted 7 lines
call_srcu(). Since deleting selem in NMI is an unlikely use case, partially mitigate it by returning error when calling frombpf_xxx_storage_delete()helpers in NMI. The check is only added tobpf_selem_unlink(), which is used by helpers and syscalls.bpf_selem_unlink_nofail()is fine as it is called during map and owner tear down that never run in NMI or reentrant.
Quoted 7 lines
The long-term solution requires
_nolockversions ofcall_rcu. Another possible solution is to defer the free throughirq_work, but it would grow the size of selem, which is non-ideal.Fixes a10787e6d58c ("bpf: Enable task local storage for tracing programs")Signed-off-by Amery Hung <[email protected]>---
Quoted 20 lines
kernel/bpf/bpf_local_storage.c | 3 +++ 1 file changed, 3 insertions(+)kernel/bpf/bpf_local_storage.c +3
-1@@ -393,6 +393,9 @@ int bpf_selem_unlink(struct bpf_local_storage_elem *selem)unsigned long flags;int err; +if (in_nmi ()) +return -EOPNOTSUPP ; +if (unlikely (!selem_linked_to_storage_lockless (selem)))/* selem has already been unlinked from sk */ return 0 ; --2.52.0
Quoted 8 lines · Kumar Kartikeya Dwivedi
Quoted 5 lines · Amery Hung
Currently, local storage may deadlock when deferring freeing selem or local storage through
kfree_rcu(),call_rcu()orcall_rcu_tasks_trace()in NMI or reentrant.kfree_rcu()andcall_rcu()have always been unsafe in NMI.call_rcu_tasks_trace()only became unsafe when it switched to
Quoted 5 lines
It has also always been unsafe in NMI, so the commit log needs an update. We should also note that the correct solution is
irqs_disabled()to completely close the same context reentrancy, but it is too coarse-grained for this case, and we can revisit it once we convert to the_nolock()variant.
Thanks for the clarification on the call_rcu_tasks_trace(). I will adjust the commit msg and send v2.
Quoted 38 lines
Quoted 8 lines
call_srcu(). Since deleting selem in NMI is an unlikely use case, partially mitigate it by returning error when calling frombpf_xxx_storage_delete()helpers in NMI. The check is only added tobpf_selem_unlink(), which is used by helpers and syscalls.bpf_selem_unlink_nofail()is fine as it is called during map and owner tear down that never run in NMI or reentrant.Quoted 7 lines
The long-term solution requires
_nolockversions ofcall_rcu. Another possible solution is to defer the free throughirq_work, but it would grow the size of selem, which is non-ideal.Fixes a10787e6d58c ("bpf: Enable task local storage for tracing programs")Signed-off-by Amery Hung <[email protected]>---
Quoted 20 lines
kernel/bpf/bpf_local_storage.c | 3 +++ 1 file changed, 3 insertions(+)kernel/bpf/bpf_local_storage.c +3
-1@@ -393,6 +393,9 @@ int bpf_selem_unlink(struct bpf_local_storage_elem *selem)unsigned long flags;int err; +if (in_nmi ()) +return -EOPNOTSUPP ; +if (unlikely (!selem_linked_to_storage_lockless (selem)))/* selem has already been unlinked from sk */ return 0 ; --2.52.0
Quoted 4 lines · Amery Hung
Currently, local storage may deadlock when deferring freeing selem or local storage through
kfree_rcu(),call_rcu()orcall_rcu_tasks_trace()in NMI or reentrant.
The fix looks correct. Adding the in_nmi() check in bpf_selem_unlink() rather than in each individual helper is the right approach, as it centralizes the protection at the single entry point used by all bpf_xxx_storage_delete() helpers.
As Kumar noted, the commit log should mention that kfree_rcu() and call_rcu() have always been unsafe in NMI, not just call_rcu_tasks_trace().
One question: should the return value be -EOPNOTSUPP or would -EBUSY be more appropriate here? -EOPNOTSUPP typically means the operation is not supported at all, while -EBUSY might better convey that the operation cannot be performed in the current context but could succeed otherwise.
Otherwise the patch looks good to me.