bpf: Avoid unnecessary -EBUSY from htab_lock_bucket
Song Liu
reviewerPATCH 1/1 bpf: Avoid unnecessary -EBUSY from htab_lock_bucket
htab_lock_bucket uses the following logic to avoid recursion:
1. preempt_disable();
2. check percpu counter htab->map_locked[hash] for recursion; 2.1. if map_lock[hash] is already taken, return -BUSY;
3. raw_spin_lock_irqsave();
However, if an IRQ hits between 2 and 3, BPF programs attached to the IRQ logic will not able to access the same hash of the hashtab and get -EBUSY. This -EBUSY is not really necessary. Fix it by disabling IRQ before checking map_locked:
1. preempt_disable();
2. local_irq_save();
3. check percpu counter htab->map_locked[hash] for recursion; 3.1. if map_lock[hash] is already taken, return -BUSY;
4. raw_spin_lock().
Similarly, use raw_spin_unlock() and local_irq_restore() in htab_unlock_bucket().
---
Resend to bpf tree to run CI with Ilya's s390 fixes.
--- kernel/bpf/hashtab.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
kernel/bpf/hashtab.c +5 -2
@@ -155,13 +155,15 @@ static inline int htab_lock_bucket(const struct bpf_htab *htab, hash = hash &min_t (u32,HASHTAB_MAP_LOCK_MASK , htab->n_buckets -1 );preempt_disable (); +local_irq_save (flags);if (unlikely (__this_cpu_inc_return (*(htab->map_locked[hash])) !=1 )) {__this_cpu_dec (*(htab->map_locked[hash])); +local_irq_restore (flags);preempt_enable ();return -EBUSY ; } -raw_spin_lock_irqsave (&b->raw_lock, flags); +raw_spin_lock (&b->raw_lock); *pflags = flags;return 0 ;
@@ -172,8 +174,9 @@ static inline void htab_unlock_bucket(const struct bpf_htab *htab, unsigned long flags) { hash = hash &min_t (u32,HASHTAB_MAP_LOCK_MASK , htab->n_buckets -1 ); -raw_spin_unlock_irqrestore (&b->raw_lock, flags); +raw_spin_unlock (&b->raw_lock);__this_cpu_dec (*(htab->map_locked[hash])); +local_irq_restore (flags);preempt_enable (); }
Signature · 1 line
2.34.1
This patch was applied to bpf/bpf.git (master) by Andrii Nakryiko <[email protected]>:
Hello:
This patch was applied to bpf/bpf.git (master)
by Andrii Nakryiko <[email protected]>:
Quoted 8 lines · On Wed, 11 Oct 2023 22:57:41 -0700 you wrote:
htab_lock_bucketuses the following logic to avoid recursion:1.
preempt_disable();
2. check percpu counter htab->map_locked[hash] for recursion; 2.1. if map_lock[hash] is already taken, return -BUSY;
3.raw_spin_lock_irqsave();[...]
Here is the summary with links:
- [bpf] bpf: Avoid unnecessary -EBUSY from htab_lock_bucket
https://git.kernel.org/bpf/bpf/c/a307f5a24924
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html