Skip to contentSkip to conversation

bpf: Avoid unnecessary -EBUSY from htab_lock_bucket

PATCH Superseded Song Liu 5 participants 12 messages Oct 4, 2023 – Oct 12, 2023 BPF

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Song Liu

author reviewer
#0 lore raw

PATCH v2 1/1 bpf: Avoid unnecessary -EBUSY from htab_lock_bucket

htab_lock_bucket uses the following logic to avoid recursion:

1. preempt_disable();
2. check percpu counter htab->map_locked[hash] for recursion; 2.1. if map_lock[hash] is already taken, return -BUSY;
3. raw_spin_lock_irqsave();

However, if an IRQ hits between 2 and 3, BPF programs attached to the IRQ logic will not able to access the same hash of the hashtab and get -EBUSY. This -EBUSY is not really necessary. Fix it by disabling IRQ before checking map_locked:

1. preempt_disable();
2. local_irq_save();
3. check percpu counter htab->map_locked[hash] for recursion; 3.1. if map_lock[hash] is already taken, return -BUSY;
4. raw_spin_lock().

Similarly, use raw_spin_unlock() and local_irq_restore() in htab_unlock_bucket().

---
Changes in v2:
1. Use raw_spin_unlock() and local_irq_restore() in htab_unlock_bucket(). (Andrii)

---
 kernel/bpf/hashtab.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
kernel/bpf/hashtab.c +5 -2
index a8c7e1c5abfa..fd8d4b0addfc 100644
@@ -155,13 +155,15 @@ static inline int htab_lock_bucket(const struct bpf_htab *htab,
 	hash = hash & min_t(u32, HASHTAB_MAP_LOCK_MASK, htab->n_buckets - 1);
 
 	preempt_disable();
+	local_irq_save(flags);
 	if (unlikely(__this_cpu_inc_return(*(htab->map_locked[hash])) != 1)) {
 		__this_cpu_dec(*(htab->map_locked[hash]));
+		local_irq_restore(flags);
 		preempt_enable();
 		return -EBUSY;
 	}
 
-	raw_spin_lock_irqsave(&b->raw_lock, flags);
+	raw_spin_lock(&b->raw_lock);
 	*pflags = flags;
 
 	return 0;
@@ -172,8 +174,9 @@ static inline void htab_unlock_bucket(const struct bpf_htab *htab,
 				      unsigned long flags)
 {
 	hash = hash & min_t(u32, HASHTAB_MAP_LOCK_MASK, htab->n_buckets - 1);
-	raw_spin_unlock_irqrestore(&b->raw_lock, flags);
+	raw_spin_unlock(&b->raw_lock);
 	__this_cpu_dec(*(htab->map_locked[hash]));
+	local_irq_restore(flags);
 	preempt_enable();
 }
 
Signature · 1 line
2.34.1
Acked-by on v2 1/1
#1 lore raw
Quoted 25 lines · Song Liu

htab_lock_bucket uses the following logic to avoid recursion:

1. preempt_disable();
2. check percpu counter htab->map_locked[hash] for recursion; 2.1. if map_lock[hash] is already taken, return -BUSY;
3. raw_spin_lock_irqsave();

However, if an IRQ hits between 2 and 3, BPF programs attached to the IRQ logic will not able to access the same hash of the hashtab and get -EBUSY. This -EBUSY is not really necessary. Fix it by disabling IRQ before checking map_locked:

1. preempt_disable();
2. local_irq_save();
3. check percpu counter htab->map_locked[hash] for recursion; 3.1. if map_lock[hash] is already taken, return -BUSY;
4. raw_spin_lock().

Similarly, use raw_spin_unlock() and local_irq_restore() in htab_unlock_bucket().

Quoted 8 lines

---
Changes in v2:
1. Use raw_spin_unlock() and local_irq_restore() in htab_unlock_bucket(). (Andrii)

---
 kernel/bpf/hashtab.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

Now it's more symmetrical and seems correct to me, thanks!

Quoted 35 lines
kernel/bpf/hashtab.c +5 -3
index a8c7e1c5abfa..fd8d4b0addfc 100644
@@ -155,13 +155,15 @@ static inline int htab_lock_bucket(const struct bpf_htab *htab,
        hash = hash & min_t(u32, HASHTAB_MAP_LOCK_MASK, htab->n_buckets - 1);
 
        preempt_disable();
+       local_irq_save(flags);
        if (unlikely(__this_cpu_inc_return(*(htab->map_locked[hash])) != 1)) {
                __this_cpu_dec(*(htab->map_locked[hash]));
+               local_irq_restore(flags);
                preempt_enable();
                return -EBUSY;
        }
 
-       raw_spin_lock_irqsave(&b->raw_lock, flags);
+       raw_spin_lock(&b->raw_lock);
        *pflags = flags;
 
        return 0;
@@ -172,8 +174,9 @@ static inline void htab_unlock_bucket(const struct bpf_htab *htab,
                                      unsigned long flags)
 {
        hash = hash & min_t(u32, HASHTAB_MAP_LOCK_MASK, htab->n_buckets - 1);
-       raw_spin_unlock_irqrestore(&b->raw_lock, flags);
+       raw_spin_unlock(&b->raw_lock);
        __this_cpu_dec(*(htab->map_locked[hash]));
+       local_irq_restore(flags);
        preempt_enable();
 }
 
--

2.34.1

#2 lore raw
Quoted 56 lines · Andrii Nakryiko
Quoted 25 lines · Song Liu

htab_lock_bucket uses the following logic to avoid recursion:

1. preempt_disable();
2. check percpu counter htab->map_locked[hash] for recursion; 2.1. if map_lock[hash] is already taken, return -BUSY;
3. raw_spin_lock_irqsave();

However, if an IRQ hits between 2 and 3, BPF programs attached to the IRQ logic will not able to access the same hash of the hashtab and get -EBUSY. This -EBUSY is not really necessary. Fix it by disabling IRQ before checking map_locked:

1. preempt_disable();
2. local_irq_save();
3. check percpu counter htab->map_locked[hash] for recursion; 3.1. if map_lock[hash] is already taken, return -BUSY;
4. raw_spin_lock().

Similarly, use raw_spin_unlock() and local_irq_restore() in htab_unlock_bucket().

Quoted 8 lines

---
Changes in v2:
1. Use raw_spin_unlock() and local_irq_restore() in htab_unlock_bucket(). (Andrii)

---
 kernel/bpf/hashtab.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

Now it's more symmetrical and seems correct to me, thanks!

Quoted 16 lines
kernel/bpf/hashtab.c +2 -0
index a8c7e1c5abfa..fd8d4b0addfc 100644
@@ -155,13 +155,15 @@ static inline int htab_lock_bucket(const struct bpf_htab *htab,
        hash = hash & min_t(u32, HASHTAB_MAP_LOCK_MASK, htab->n_buckets - 1);
 
        preempt_disable();
+       local_irq_save(flags);
        if (unlikely(__this_cpu_inc_return(*(htab->map_locked[hash])) != 1)) {
                __this_cpu_dec(*(htab->map_locked[hash]));
+               local_irq_restore(flags);
                preempt_enable();
                return -EBUSY;
        }
 
Quoted 2 lines

kernel/bpf/hashtab.c static inline int htab_lock_bucket(const struct bpf_htab *htab, · line 166

Quoted 2 lines
kernel/bpf/hashtab.c +1 -1
-       raw_spin_lock_irqsave(&b->raw_lock, flags);
+       raw_spin_lock(&b->raw_lock);

Song,

take a look at s390 crash in BPF CI.
I suspect this patch is causing it.

Ilya,

do you have an idea what is going on?

#3 lore raw
Quoted 62 lines · Alexei Starovoitov
Quoted 56 lines · Andrii Nakryiko
Quoted 25 lines · Song Liu

htab_lock_bucket uses the following logic to avoid recursion:

1. preempt_disable();
2. check percpu counter htab->map_locked[hash] for recursion; 2.1. if map_lock[hash] is already taken, return -BUSY;
3. raw_spin_lock_irqsave();

However, if an IRQ hits between 2 and 3, BPF programs attached to the IRQ logic will not able to access the same hash of the hashtab and get -EBUSY. This -EBUSY is not really necessary. Fix it by disabling IRQ before checking map_locked:

1. preempt_disable();
2. local_irq_save();
3. check percpu counter htab->map_locked[hash] for recursion; 3.1. if map_lock[hash] is already taken, return -BUSY;
4. raw_spin_lock().

Similarly, use raw_spin_unlock() and local_irq_restore() in htab_unlock_bucket().

Quoted 8 lines

---
Changes in v2:
1. Use raw_spin_unlock() and local_irq_restore() in htab_unlock_bucket(). (Andrii)
---
kernel/bpf/hashtab.c | 7 +++++--

1 file changed, 5 insertions(+), 2 deletions(-)

Now it's more symmetrical and seems correct to me, thanks!

Quoted 16 lines
kernel/bpf/hashtab.c +2 -0
index a8c7e1c5abfa..fd8d4b0addfc 100644
@@ -155,13 +155,15 @@ static inline int htab_lock_bucket(const struct bpf_htab *htab,
       hash = hash & min_t(u32, HASHTAB_MAP_LOCK_MASK, htab->n_buckets - 1);
 
       preempt_disable();
+       local_irq_save(flags);
       if (unlikely(__this_cpu_inc_return(*(htab->map_locked[hash])) != 1)) {
               __this_cpu_dec(*(htab->map_locked[hash]));
+               local_irq_restore(flags);
               preempt_enable();
               return -EBUSY;
       }
 
Quoted 7 lines
Quoted 2 lines

kernel/bpf/hashtab.c static inline int htab_lock_bucket(const struct bpf_htab *htab, · line 166

Quoted 2 lines
kernel/bpf/hashtab.c +1 -1
-       raw_spin_lock_irqsave(&b->raw_lock, flags);
+       raw_spin_lock(&b->raw_lock);

Song,

take a look at s390 crash in BPF CI.
I suspect this patch is causing it.

It indeed looks like triggered by this patch. But I haven't figured out why it happens. v1 seems ok for the same tests.

Song

Quoted 4 lines

Ilya,

do you have an idea what is going on?

#4 lore raw
Quoted 71 lines · Song Liu
Quoted 68 lines · Alexei Starovoitov
Quoted 56 lines · Andrii Nakryiko
Quoted 25 lines · Song Liu

htab_lock_bucket uses the following logic to avoid recursion:

1. preempt_disable();
2. check percpu counter htab->map_locked[hash] for recursion; 2.1. if map_lock[hash] is already taken, return -BUSY;
3. raw_spin_lock_irqsave();

However, if an IRQ hits between 2 and 3, BPF programs attached to the IRQ logic will not able to access the same hash of the hashtab and get -EBUSY. This -EBUSY is not really necessary. Fix it by disabling IRQ before checking map_locked:

1. preempt_disable();
2. local_irq_save();
3. check percpu counter htab->map_locked[hash] for recursion; 3.1. if map_lock[hash] is already taken, return -BUSY;
4. raw_spin_lock().

Similarly, use raw_spin_unlock() and local_irq_restore() in htab_unlock_bucket().

Quoted 8 lines

---
Changes in v2:
1. Use raw_spin_unlock() and local_irq_restore() in htab_unlock_bucket(). (Andrii)
---
kernel/bpf/hashtab.c | 7 +++++--

1 file changed, 5 insertions(+), 2 deletions(-)

Now it's more symmetrical and seems correct to me, thanks!

Quoted 16 lines
kernel/bpf/hashtab.c +2 -0
index a8c7e1c5abfa..fd8d4b0addfc 100644
@@ -155,13 +155,15 @@ static inline int htab_lock_bucket(const struct bpf_htab *htab,
      hash = hash & min_t(u32, HASHTAB_MAP_LOCK_MASK, htab->n_buckets - 1);
 
      preempt_disable();
+       local_irq_save(flags);
      if (unlikely(__this_cpu_inc_return(*(htab->map_locked[hash])) != 1)) {
              __this_cpu_dec(*(htab->map_locked[hash]));
+               local_irq_restore(flags);
              preempt_enable();
              return -EBUSY;
      }
 
Quoted 2 lines

kernel/bpf/hashtab.c static inline int htab_lock_bucket(const struct bpf_htab *htab, · line 166

Quoted 2 lines
kernel/bpf/hashtab.c +1 -1
-       raw_spin_lock_irqsave(&b->raw_lock, flags);
+       raw_spin_lock(&b->raw_lock);

Song,

Quoted 5 lines
Quoted 2 lines

take a look at s390 crash in BPF CI.
I suspect this patch is causing it.

It indeed looks like triggered by this patch. But I haven't figured out why it happens. v1 seems ok for the same tests.

I guess I finally figured out this (should be simple) bug. If I got it correctly, we need:

kernel/bpf/hashtab.c +1 -0
index fd8d4b0addfc..1cfa2329a53a 100644
@@ -160,6 +160,7 @@ static inline int htab_lock_bucket(const struct bpf_htab *htab,
                __this_cpu_dec(*(htab->map_locked[hash]));
                local_irq_restore(flags);
                preempt_enable();
+               *pflags = flags;
                return -EBUSY;
        }
 

Running CI tests here:

https://github.com/kernel-patches/bpf/pull/5769

If it works, I will send v3.

Thanks,
Song

PS: s390x CI is running slow. I got some jobs stayed in the queue for more than a hour.

Quoted 8 lines

Song

Quoted 4 lines

Ilya,

do you have an idea what is going on?

#5 lore raw
Quoted 89 lines · Song Liu
Quoted 72 lines · Song Liu
Quoted 68 lines · Alexei Starovoitov
Quoted 56 lines · Andrii Nakryiko
Quoted 25 lines · Song Liu

htab_lock_bucket uses the following logic to avoid recursion:

1. preempt_disable();
2. check percpu counter htab->map_locked[hash] for recursion; 2.1. if map_lock[hash] is already taken, return -BUSY;
3. raw_spin_lock_irqsave();

However, if an IRQ hits between 2 and 3, BPF programs attached to the IRQ logic will not able to access the same hash of the hashtab and get -EBUSY. This -EBUSY is not really necessary. Fix it by disabling IRQ before checking map_locked:

1. preempt_disable();
2. local_irq_save();
3. check percpu counter htab->map_locked[hash] for recursion; 3.1. if map_lock[hash] is already taken, return -BUSY;
4. raw_spin_lock().

Similarly, use raw_spin_unlock() and local_irq_restore() in htab_unlock_bucket().

Quoted 8 lines

---
Changes in v2:
1. Use raw_spin_unlock() and local_irq_restore() in htab_unlock_bucket(). (Andrii)
---
kernel/bpf/hashtab.c | 7 +++++--

1 file changed, 5 insertions(+), 2 deletions(-)

Now it's more symmetrical and seems correct to me, thanks!

Quoted 16 lines
kernel/bpf/hashtab.c +2 -0
index a8c7e1c5abfa..fd8d4b0addfc 100644
@@ -155,13 +155,15 @@ static inline int htab_lock_bucket(const struct bpf_htab *htab,
     hash = hash & min_t(u32, HASHTAB_MAP_LOCK_MASK, htab->n_buckets - 1);
 
     preempt_disable();
+       local_irq_save(flags);
     if (unlikely(__this_cpu_inc_return(*(htab->map_locked[hash])) != 1)) {
             __this_cpu_dec(*(htab->map_locked[hash]));
+               local_irq_restore(flags);
             preempt_enable();
             return -EBUSY;
     }
 
Quoted 2 lines

kernel/bpf/hashtab.c static inline int htab_lock_bucket(const struct bpf_htab *htab, · line 166

Quoted 2 lines
kernel/bpf/hashtab.c +1 -1
-       raw_spin_lock_irqsave(&b->raw_lock, flags);
+       raw_spin_lock(&b->raw_lock);

Song,

Quoted 5 lines
Quoted 2 lines

take a look at s390 crash in BPF CI.
I suspect this patch is causing it.

It indeed looks like triggered by this patch. But I haven't figured out why it happens. v1 seems ok for the same tests.

I guess I finally figured out this (should be simple) bug. If I got it correctly, we need:

kernel/bpf/hashtab.c +0 -0
index fd8d4b0addfc..1cfa2329a53a 100644

kernel/bpf/hashtab.c static inline int htab_lock_bucket(const struct bpf_htab *htab, · line 165

Quoted 6 lines
kernel/bpf/hashtab.c +1 -0
@@ -160,6 +160,7 @@ static inline int htab_lock_bucket(const struct bpf_htab *htab,
               __this_cpu_dec(*(htab->map_locked[hash]));
               local_irq_restore(flags);
               preempt_enable();
+               *pflags = flags;
               return -EBUSY;
       }

No... I was totally wrong. This is not needed.

Trying something different..

Thanks,
Song

#6 lore raw
Quoted 3 lines · Song Liu
Quoted 1 line · Song Liu

[...]

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 17 lines
Quoted 17 lines
kernel/bpf/hashtab.c +2 -0
index a8c7e1c5abfa..fd8d4b0addfc 100644
@@ -155,13 +155,15 @@ static inline int htab_lock_bucket(const struct bpf_htab *htab,
    hash = hash & min_t(u32, HASHTAB_MAP_LOCK_MASK, htab->n_buckets - 1);
 
    preempt_disable();
+       local_irq_save(flags);
    if (unlikely(__this_cpu_inc_return(*(htab->map_locked[hash])) != 1)) {
            __this_cpu_dec(*(htab->map_locked[hash]));
+               local_irq_restore(flags);
            preempt_enable();
            return -EBUSY;
    }
 
Quoted 2 lines

kernel/bpf/hashtab.c static inline int htab_lock_bucket(const struct bpf_htab *htab, · line 166

Quoted 2 lines
kernel/bpf/hashtab.c +1 -1
-       raw_spin_lock_irqsave(&b->raw_lock, flags);
+       raw_spin_lock(&b->raw_lock);

Song,

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

take a look at s390 crash in BPF CI.
I suspect this patch is causing it.

It indeed looks like triggered by this patch. But I haven't figured out why it happens. v1 seems ok for the same tests.

Update my findings today:

I tried to reproduce the issue locally with qemu on my server (x86_64). I got the following artifacts:

1. bzImage and selftests from CI: (need to login to GitHub)
https://github.com/kernel-patches/bpf/suites/16885416280/artifacts/964765766

2. cross compiler:
https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.2.0/x86_64-gcc-13.2.0-nolibc-s390-linux.tar.gz

3. root image:
https://libbpf-ci.s3.us-west-1.amazonaws.com/libbpf-vmtest-rootfs-2022.10.23-bullseye-s390x.tar.zst

With bzImage compiled in CI, I can reproduce the issue with qemu. However, if I compile the kernel locally with the cross compiler (with .config from CI, and then olddefconfig), the issue cannot be reproduced. I have attached the two .config files here. They look very similar, except the compiler version:

-CONFIG_CC_VERSION_TEXT="gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0" +CONFIG_CC_VERSION_TEXT="s390-linux-gcc (GCC) 13.2.0"

So far, I still think v2 is the right patch. But I really cannot explain the issue on the bzImage from the CI. I cannot make much sense out of the s390 assembly code either. (TIL: gdb on my x86 server can disassem s390 binary).

Ilya,

Could you please take a look at this?

Thanks,
Song

PS: the root image from the CI is not easy to use. Hopefully you have something better than that.

#7 lore raw
Quoted 70 lines · Song Liu
Quoted 4 lines
Quoted 2 lines · Song Liu

On Oct 4, 2023, at 5:11 PM, Song Liu <[email protected]> wrote:

[...]

Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 22 lines
Quoted 22 lines
kernel/bpf/hashtab.c +1 -0
index a8c7e1c5abfa..fd8d4b0addfc 100644
@@ -155,13 +155,15 @@ static inline int
 htab_lock_bucket(const struct bpf_htab *htab,
      hash = hash & min_t(u32, HASHTAB_MAP_LOCK_MASK,
 htab->n_buckets - 1);
 
      preempt_disable();
+       local_irq_save(flags);

     if (unlikely(__this_cpu_inc_return(*(htab-

Quoted 1 line

map_locked[hash])) != 1)) {

             __this_cpu_dec(*(htab->map_locked[hash])); +               local_irq_restore(flags); preempt_enable();
             return -EBUSY;
     }

-       raw_spin_lock_irqsave(&b->raw_lock, flags); +       raw_spin_lock(&b->raw_lock);

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

Song,

take a look at s390 crash in BPF CI.
I suspect this patch is causing it.

It indeed looks like triggered by this patch. But I haven't figured
out why it happens. v1 seems ok for the same tests.

Update my findings today:

I tried to reproduce the issue locally with qemu on my server (x86_64).
I got the following artifacts:

1. bzImage and selftests from CI: (need to login to GitHub)
https://github.com/kernel-patches/bpf/suites/16885416280/artifacts/964765766

2. cross compiler:
https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.2.0/x86_64-gcc-13.2.0-nolibc-s390-linux.tar.gz

3. root image:
https://libbpf-ci.s3.us-west-1.amazonaws.com/libbpf-vmtest-rootfs-2022.10.23-bullseye-s390x.tar.zst

With bzImage compiled in CI, I can reproduce the issue with qemu. However, if I compile the kernel locally with the cross compiler (with .config from CI, and then olddefconfig), the issue cannot be reproduced. I have attached the two .config files here. They look very similar, except the compiler version:

(quoted diff) +1 -1
-CONFIG_CC_VERSION_TEXT="gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0"
+CONFIG_CC_VERSION_TEXT="s390-linux-gcc (GCC) 13.2.0"

So far, I still think v2 is the right patch. But I really cannot explain the issue on the bzImage from the CI. I cannot make much sense out of the s390 assembly code either. (TIL: gdb on my x86 server can disassem s390 binary).

Ilya,

Quoted 7 lines

Could you please take a look at this?

Thanks,
Song

PS: the root image from the CI is not easy to use. Hopefully you have something better than that.

Hi,

sorry for the long silence, I was traveling and then sick. I'll look into this ASAP.

Best regards,
Ilya

#8 lore raw
In reply to Song Liu
Quoted 70 lines · Song Liu
Quoted 4 lines
Quoted 2 lines · Song Liu

On Oct 4, 2023, at 5:11 PM, Song Liu <[email protected]> wrote:

[...]

Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 22 lines
Quoted 22 lines
kernel/bpf/hashtab.c +1 -0
index a8c7e1c5abfa..fd8d4b0addfc 100644
@@ -155,13 +155,15 @@ static inline int
 htab_lock_bucket(const struct bpf_htab *htab,
      hash = hash & min_t(u32, HASHTAB_MAP_LOCK_MASK,
 htab->n_buckets - 1);
 
      preempt_disable();
+       local_irq_save(flags);

     if (unlikely(__this_cpu_inc_return(*(htab-

Quoted 1 line

map_locked[hash])) != 1)) {

             __this_cpu_dec(*(htab->map_locked[hash])); +               local_irq_restore(flags); preempt_enable();
             return -EBUSY;
     }

-       raw_spin_lock_irqsave(&b->raw_lock, flags); +       raw_spin_lock(&b->raw_lock);

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

Song,

take a look at s390 crash in BPF CI.
I suspect this patch is causing it.

It indeed looks like triggered by this patch. But I haven't figured
out why it happens. v1 seems ok for the same tests.

Update my findings today:

I tried to reproduce the issue locally with qemu on my server (x86_64).
I got the following artifacts:

1. bzImage and selftests from CI: (need to login to GitHub)
https://github.com/kernel-patches/bpf/suites/16885416280/artifacts/964765766

2. cross compiler:
https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.2.0/x86_64-gcc-13.2.0-nolibc-s390-linux.tar.gz

3. root image:
https://libbpf-ci.s3.us-west-1.amazonaws.com/libbpf-vmtest-rootfs-2022.10.23-bullseye-s390x.tar.zst

With bzImage compiled in CI, I can reproduce the issue with qemu. However, if I compile the kernel locally with the cross compiler (with .config from CI, and then olddefconfig), the issue cannot be reproduced. I have attached the two .config files here. They look very similar, except the compiler version:

(quoted diff) +1 -1
-CONFIG_CC_VERSION_TEXT="gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0"
+CONFIG_CC_VERSION_TEXT="s390-linux-gcc (GCC) 13.2.0"

So far, I still think v2 is the right patch. But I really cannot explain the issue on the bzImage from the CI. I cannot make much sense out of the s390 assembly code either. (TIL: gdb on my x86 server can disassem s390 binary).

Ilya,

Quoted 7 lines

Could you please take a look at this?

Thanks,
Song

PS: the root image from the CI is not easy to use. Hopefully you have something better than that.

Hi,

Thanks for posting the analysis and links to the artifacts, that saved me quite some time. The crash is caused by a backchain issue in the trampoline code and has nothing to do with your patch; I've posted the fix here [1].

The difference between compilers is most likely caused by different inlining decisions around lookup_elem_raw(). When it's inlined, the test becomes a no-op.

Regarding GDB, Debian and Ubuntu have gdb-multiarch package. On Fedora one has to build it from source; the magic binutils-gdb configure flag is --enable-targets=all.

Regarding the development environment, in this case I've sidestepped the need for a proper image by putting everything into initramfs:

workdir=$(mktemp -d)
tar -C "$workdir" -xf libbpf-vmtest-rootfs-2022.10.23-bullseye- s390x.tar.zst
rsync -a selftests "$workdir"
(cd "$workdir" && find . | cpio -o --format=newc -R root:root)

Quoted 1 line

initrd.cpio

qemu-system-s390x -accel kvm -smp 2 -m 4G -kernel kbuild- output/arch/s390/boot/bzImage -nographic -append 'nokaslr console=ttyS1 rdinit=/bin/sh' -initrd initrd.cpio -s

For the regular development I have a different setup, with a non-minimal Debian install inside the guest, and the testcases mounted from the host using 9p.

Best regards,
Ilya

[1]
https://lore.kernel.org/bpf/[email protected]/

author reviewer
#9 lore raw

Hi Ilya,

Quoted 1 line · Ilya Leoshkevich

[...]

Quoted 7 lines
Quoted 6 lines

Thanks,
Song

PS: the root image from the CI is not easy to use. Hopefully you have something better than that.

Quoted 6 lines

Hi,

Thanks for posting the analysis and links to the artifacts, that saved me quite some time. The crash is caused by a backchain issue in the trampoline code and has nothing to do with your patch; I've posted the fix here [1].

Thanks for the fix!

Song

Quoted 17 lines

The difference between compilers is most likely caused by different inlining decisions around lookup_elem_raw(). When it's inlined, the test becomes a no-op.

Regarding GDB, Debian and Ubuntu have gdb-multiarch package. On Fedora one has to build it from source; the magic binutils-gdb configure flag is --enable-targets=all.

Regarding the development environment, in this case I've sidestepped the need for a proper image by putting everything into initramfs:

workdir=$(mktemp -d)
tar -C "$workdir" -xf libbpf-vmtest-rootfs-2022.10.23-bullseye- s390x.tar.zst
rsync -a selftests "$workdir"
(cd "$workdir" && find . | cpio -o --format=newc -R root:root)

Quoted 4 lines
Quoted 1 line

initrd.cpio

qemu-system-s390x -accel kvm -smp 2 -m 4G -kernel kbuild- output/arch/s390/boot/bzImage -nographic -append 'nokaslr console=ttyS1 rdinit=/bin/sh' -initrd initrd.cpio -s

Nice trick!

Quoted 9 lines

For the regular development I have a different setup, with a non-minimal Debian install inside the guest, and the testcases mounted from the host using 9p.

Best regards,
Ilya

[1]
https://lore.kernel.org/bpf/[email protected]/

#10 lore raw
Quoted 15 lines · Song Liu

Hi Ilya,

Quoted 1 line · Ilya Leoshkevich

[...]

Quoted 9 lines
Quoted 6 lines

Thanks,
Song

PS: the root image from the CI is not easy to use. Hopefully you have something better than that.

Hi,

Quoted 8 lines
Quoted 4 lines

Thanks for posting the analysis and links to the artifacts, that saved me quite some time. The crash is caused by a backchain issue in the trampoline code and has nothing to do with your patch; I've posted the fix here [1].

Thanks for the fix!

Song

Song, can you please resend your patch so that CI can test it again on top of Ilya's changes? Thanks!

Quoted 34 lines
Quoted 18 lines

The difference between compilers is most likely caused by different inlining decisions around lookup_elem_raw(). When it's inlined, the test becomes a no-op.

Regarding GDB, Debian and Ubuntu have gdb-multiarch package. On Fedora one has to build it from source; the magic binutils-gdb configure flag is --enable-targets=all.

Regarding the development environment, in this case I've sidestepped the need for a proper image by putting everything into initramfs:

workdir=$(mktemp -d)
tar -C "$workdir" -xf libbpf-vmtest-rootfs-2022.10.23-bullseye- s390x.tar.zst
rsync -a selftests "$workdir"
(cd "$workdir" && find . | cpio -o --format=newc -R root:root)

Quoted 4 lines
Quoted 1 line

initrd.cpio

qemu-system-s390x -accel kvm -smp 2 -m 4G -kernel kbuild- output/arch/s390/boot/bzImage -nographic -append 'nokaslr console=ttyS1 rdinit=/bin/sh' -initrd initrd.cpio -s

Nice trick!

Quoted 9 lines

For the regular development I have a different setup, with a non-minimal Debian install inside the guest, and the testcases mounted from the host using 9p.

Best regards,
Ilya

[1]
https://lore.kernel.org/bpf/[email protected]/

#11 lore raw
Quoted 25 lines · Andrii Nakryiko
Quoted 20 lines · Song Liu

Hi Ilya,

Quoted 1 line · Ilya Leoshkevich

[...]

Quoted 13 lines
Quoted 6 lines

Thanks,
Song

PS: the root image from the CI is not easy to use. Hopefully you have something better than that.

Hi,

Thanks for posting the analysis and links to the artifacts, that saved me quite some time. The crash is caused by a backchain issue in the trampoline code and has nothing to do with your patch; I've posted the fix here [1].

Quoted 6 lines
Quoted 3 lines

Thanks for the fix!

Song

Song, can you please resend your patch so that CI can test it again on top of Ilya's changes? Thanks!

Sure. Just sent the patch to bpf tree.

Thanks,
Song