Skip to contentSkip to conversation

bpf: Avoid unnecessary -EBUSY from htab_lock_bucket

PATCH Superseded Song Liu 2 participants 3 messages Oct 3, 2023 BPF

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Song Liu

author reviewer
#0 lore raw

PATCH 1/1 bpf: Avoid unnecessary -EBUSY from htab_lock_bucket

htab_lock_bucket uses the following logic to avoid recursion:

1. preempt_disable();
2. check percpu counter htab->map_locked[hash] for recursion; 2.1. if map_lock[hash] is already taken, return -BUSY;
3. raw_spin_lock_irqsave();

However, if an IRQ hits between 2 and 3, BPF programs attached to the IRQ logic will not able to access the same hash of the hashtab and get -EBUSY. This -EBUSY is not really necessary. Fix it by disabling IRQ before checking map_locked:

1. preempt_disable();
2. local_irq_save();
3. check percpu counter htab->map_locked[hash] for recursion; 3.1. if map_lock[hash] is already taken, return -BUSY;
4. raw_spin_lock().

---
 kernel/bpf/hashtab.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/bpf/hashtab.c +3 -1
index a8c7e1c5abfa..347af4476662 100644
@@ -155,13 +155,15 @@ static inline int htab_lock_bucket(const struct bpf_htab *htab,
 	hash = hash & min_t(u32, HASHTAB_MAP_LOCK_MASK, htab->n_buckets - 1);
 
 	preempt_disable();
+	local_irq_save(flags);
 	if (unlikely(__this_cpu_inc_return(*(htab->map_locked[hash])) != 1)) {
 		__this_cpu_dec(*(htab->map_locked[hash]));
+		local_irq_restore(flags);
 		preempt_enable();
 		return -EBUSY;
 	}
 
-	raw_spin_lock_irqsave(&b->raw_lock, flags);
+	raw_spin_lock(&b->raw_lock);
 	*pflags = flags;
 
 	return 0;
Signature · 1 line
2.34.1
#1 lore raw
Quoted 35 lines · Song Liu

htab_lock_bucket uses the following logic to avoid recursion:

1. preempt_disable();
2. check percpu counter htab->map_locked[hash] for recursion; 2.1. if map_lock[hash] is already taken, return -BUSY;
3. raw_spin_lock_irqsave();

However, if an IRQ hits between 2 and 3, BPF programs attached to the IRQ logic will not able to access the same hash of the hashtab and get -EBUSY. This -EBUSY is not really necessary. Fix it by disabling IRQ before checking map_locked:

1. preempt_disable();
2. local_irq_save();
3. check percpu counter htab->map_locked[hash] for recursion; 3.1. if map_lock[hash] is already taken, return -BUSY;
4. raw_spin_lock().

---
 kernel/bpf/hashtab.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/bpf/hashtab.c +1 -0
index a8c7e1c5abfa..347af4476662 100644
@@ -155,13 +155,15 @@ static inline int htab_lock_bucket(const struct bpf_htab *htab,
        hash = hash & min_t(u32, HASHTAB_MAP_LOCK_MASK, htab->n_buckets - 1);
 
        preempt_disable();
+       local_irq_save(flags);
        if (unlikely(__this_cpu_inc_return(*(htab->map_locked[hash])) != 1)) {

kernel/bpf/hashtab.c static inline int htab_lock_bucket(const struct bpf_htab *htab, · line 168

Quoted 10 lines
kernel/bpf/hashtab.c +2 -1
                __this_cpu_dec(*(htab->map_locked[hash]));
+               local_irq_restore(flags);
                preempt_enable();
                return -EBUSY;
        }
 
-       raw_spin_lock_irqsave(&b->raw_lock, flags);
+       raw_spin_lock(&b->raw_lock);
        *pflags = flags;
 

I might be wrong, but I think it's dangerous to have raw_spin_lock() + raw_spin_unlock_irqrestore() (in htab_unlock_bucket). Looking at the implementation of raw_spin_lock_irqsave() and raw_spin_unlock_irqrestore(), they do their own preempt_disable/preempt_enable, and so with your change I think we have imbalance, one preempt_disable() in htab_lock_bucket(), but two preempt_enable (one explicit in htab_unlock_bucket, and one implicit inside raw_spin_unlock_irqrestore).

I'd say let's use plain raw_spin_unlock() + explicit local_irq_restore(flags) in htab_unlock_bucket?

Quoted 4 lines
kernel/bpf/hashtab.c +0 -1
        return 0;
--

2.34.1

author reviewer
#2 lore raw
Quoted 57 lines · Andrii Nakryiko
Quoted 35 lines · Song Liu

htab_lock_bucket uses the following logic to avoid recursion:

1. preempt_disable();
2. check percpu counter htab->map_locked[hash] for recursion; 2.1. if map_lock[hash] is already taken, return -BUSY;
3. raw_spin_lock_irqsave();

However, if an IRQ hits between 2 and 3, BPF programs attached to the IRQ logic will not able to access the same hash of the hashtab and get -EBUSY. This -EBUSY is not really necessary. Fix it by disabling IRQ before checking map_locked:

1. preempt_disable();
2. local_irq_save();
3. check percpu counter htab->map_locked[hash] for recursion; 3.1. if map_lock[hash] is already taken, return -BUSY;
4. raw_spin_lock().

---
 kernel/bpf/hashtab.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/bpf/hashtab.c +1 -0
index a8c7e1c5abfa..347af4476662 100644
@@ -155,13 +155,15 @@ static inline int htab_lock_bucket(const struct bpf_htab *htab,
        hash = hash & min_t(u32, HASHTAB_MAP_LOCK_MASK, htab->n_buckets - 1);
 
        preempt_disable();
+       local_irq_save(flags);
        if (unlikely(__this_cpu_inc_return(*(htab->map_locked[hash])) != 1)) {

kernel/bpf/hashtab.c static inline int htab_lock_bucket(const struct bpf_htab *htab, · line 168

Quoted 10 lines
kernel/bpf/hashtab.c +2 -1
                __this_cpu_dec(*(htab->map_locked[hash]));
+               local_irq_restore(flags);
                preempt_enable();
                return -EBUSY;
        }
 
-       raw_spin_lock_irqsave(&b->raw_lock, flags);
+       raw_spin_lock(&b->raw_lock);
        *pflags = flags;
 

I might be wrong, but I think it's dangerous to have raw_spin_lock() + raw_spin_unlock_irqrestore() (in htab_unlock_bucket). Looking at the implementation of raw_spin_lock_irqsave() and raw_spin_unlock_irqrestore(), they do their own preempt_disable/preempt_enable, and so with your change I think we have imbalance, one preempt_disable() in htab_lock_bucket(), but two preempt_enable (one explicit in htab_unlock_bucket, and one implicit inside raw_spin_unlock_irqrestore).

Quoted 2 lines

I'd say let's use plain raw_spin_unlock() + explicit local_irq_restore(flags) in htab_unlock_bucket?

Yeah, there is actually a similar window in htab_unlock_bucket(). Let's also close that.

Thanks,
Song

Quoted 6 lines
Quoted 6 lines
kernel/bpf/hashtab.c +0 -1
        return 0;
--

2.34.1