bpf: Avoid unnecessary -EBUSY from htab_lock_bucket
Song Liu
reviewerPATCH v2 1/1 bpf: Avoid unnecessary -EBUSY from htab_lock_bucket
htab_lock_bucket uses the following logic to avoid recursion:
1. preempt_disable();
2. check percpu counter htab->map_locked[hash] for recursion; 2.1. if map_lock[hash] is already taken, return -BUSY;
3. raw_spin_lock_irqsave();
However, if an IRQ hits between 2 and 3, BPF programs attached to the IRQ logic will not able to access the same hash of the hashtab and get -EBUSY. This -EBUSY is not really necessary. Fix it by disabling IRQ before checking map_locked:
1. preempt_disable();
2. local_irq_save();
3. check percpu counter htab->map_locked[hash] for recursion; 3.1. if map_lock[hash] is already taken, return -BUSY;
4. raw_spin_lock().
Similarly, use raw_spin_unlock() and local_irq_restore() in htab_unlock_bucket().
---
Changes in v2:
1. Use raw_spin_unlock() and local_irq_restore() in htab_unlock_bucket(). (Andrii)
--- kernel/bpf/hashtab.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
kernel/bpf/hashtab.c +5 -2
@@ -155,13 +155,15 @@ static inline int htab_lock_bucket(const struct bpf_htab *htab, hash = hash &min_t (u32,HASHTAB_MAP_LOCK_MASK , htab->n_buckets -1 );preempt_disable (); +local_irq_save (flags);if (unlikely (__this_cpu_inc_return (*(htab->map_locked[hash])) !=1 )) {__this_cpu_dec (*(htab->map_locked[hash])); +local_irq_restore (flags);preempt_enable ();return -EBUSY ; } -raw_spin_lock_irqsave (&b->raw_lock, flags); +raw_spin_lock (&b->raw_lock); *pflags = flags;return 0 ;
@@ -172,8 +174,9 @@ static inline void htab_unlock_bucket(const struct bpf_htab *htab, unsigned long flags) { hash = hash &min_t (u32,HASHTAB_MAP_LOCK_MASK , htab->n_buckets -1 ); -raw_spin_unlock_irqrestore (&b->raw_lock, flags); +raw_spin_unlock (&b->raw_lock);__this_cpu_dec (*(htab->map_locked[hash])); +local_irq_restore (flags);preempt_enable (); }
Signature · 1 line
2.34.1
Quoted 25 lines · Song Liu
htab_lock_bucketuses the following logic to avoid recursion:1.
preempt_disable();
2. check percpu counter htab->map_locked[hash] for recursion; 2.1. if map_lock[hash] is already taken, return -BUSY;
3.raw_spin_lock_irqsave();However, if an IRQ hits between 2 and 3, BPF programs attached to the IRQ logic will not able to access the same hash of the hashtab and get
-EBUSY. This-EBUSYis not really necessary. Fix it by disabling IRQ before checkingmap_locked:1.
preempt_disable();
2.local_irq_save();
3. check percpu counter htab->map_locked[hash] for recursion; 3.1. if map_lock[hash] is already taken, return -BUSY;
4.raw_spin_lock().Similarly, use
raw_spin_unlock()andlocal_irq_restore()inhtab_unlock_bucket().Suggested-by Tejun Heo <[email protected]>Signed-off-by Song Liu <[email protected]>
Quoted 8 lines
---
Changes in v2:
1. Useraw_spin_unlock()andlocal_irq_restore()inhtab_unlock_bucket(). (Andrii)--- kernel/bpf/hashtab.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
Now it's more symmetrical and seems correct to me, thanks!
Quoted 35 lines
kernel/bpf/hashtab.c +5
-3@@ -155,13 +155,15 @@ static inline int htab_lock_bucket(const struct bpf_htab *htab, hash = hash &min_t (u32,HASHTAB_MAP_LOCK_MASK , htab->n_buckets -1 );preempt_disable (); +local_irq_save (flags);if (unlikely (__this_cpu_inc_return (*(htab->map_locked[hash])) !=1 )) {__this_cpu_dec (*(htab->map_locked[hash])); +local_irq_restore (flags);preempt_enable ();return -EBUSY ; } -raw_spin_lock_irqsave (&b->raw_lock, flags); +raw_spin_lock (&b->raw_lock); *pflags = flags;return 0 ;@@ -172,8 +174,9 @@ static inline void htab_unlock_bucket(const struct bpf_htab *htab, unsigned long flags) { hash = hash &min_t (u32,HASHTAB_MAP_LOCK_MASK , htab->n_buckets -1 ); -raw_spin_unlock_irqrestore (&b->raw_lock, flags); +raw_spin_unlock (&b->raw_lock);__this_cpu_dec (*(htab->map_locked[hash])); +local_irq_restore (flags);preempt_enable (); } --2.34.1
Quoted 56 lines · Andrii Nakryiko
Quoted 25 lines · Song Liu
htab_lock_bucketuses the following logic to avoid recursion:1.
preempt_disable();
2. check percpu counter htab->map_locked[hash] for recursion; 2.1. if map_lock[hash] is already taken, return -BUSY;
3.raw_spin_lock_irqsave();However, if an IRQ hits between 2 and 3, BPF programs attached to the IRQ logic will not able to access the same hash of the hashtab and get
-EBUSY. This-EBUSYis not really necessary. Fix it by disabling IRQ before checkingmap_locked:1.
preempt_disable();
2.local_irq_save();
3. check percpu counter htab->map_locked[hash] for recursion; 3.1. if map_lock[hash] is already taken, return -BUSY;
4.raw_spin_lock().Similarly, use
raw_spin_unlock()andlocal_irq_restore()inhtab_unlock_bucket().Suggested-by Tejun Heo <[email protected]>Signed-off-by Song Liu <[email protected]>Quoted 8 lines
---
Changes in v2:
1. Useraw_spin_unlock()andlocal_irq_restore()inhtab_unlock_bucket(). (Andrii)--- kernel/bpf/hashtab.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)Now it's more symmetrical and seems correct to me, thanks!
Acked-by Andrii Nakryiko <[email protected]>Quoted 16 lines
kernel/bpf/hashtab.c +2
-0@@ -155,13 +155,15 @@ static inline int htab_lock_bucket(const struct bpf_htab *htab, hash = hash &min_t (u32,HASHTAB_MAP_LOCK_MASK , htab->n_buckets -1 );preempt_disable (); +local_irq_save (flags);if (unlikely (__this_cpu_inc_return (*(htab->map_locked[hash])) !=1 )) {__this_cpu_dec (*(htab->map_locked[hash])); +local_irq_restore (flags);preempt_enable ();return -EBUSY ; }
Quoted 2 lines
kernel/bpf/hashtab.c static inline int htab_lock_bucket(const struct bpf_htab *htab, · line 166
Quoted 2 lines
kernel/bpf/hashtab.c +1
-1-raw_spin_lock_irqsave (&b->raw_lock, flags); +raw_spin_lock (&b->raw_lock);
Song,
take a look at s390 crash in BPF CI.
I suspect this patch is causing it.
Ilya,
do you have an idea what is going on?
Quoted 62 lines · Alexei Starovoitov
Quoted 56 lines · Andrii Nakryiko
Quoted 25 lines · Song Liu
htab_lock_bucketuses the following logic to avoid recursion:1.
preempt_disable();
2. check percpu counter htab->map_locked[hash] for recursion; 2.1. if map_lock[hash] is already taken, return -BUSY;
3.raw_spin_lock_irqsave();However, if an IRQ hits between 2 and 3, BPF programs attached to the IRQ logic will not able to access the same hash of the hashtab and get
-EBUSY. This-EBUSYis not really necessary. Fix it by disabling IRQ before checkingmap_locked:1.
preempt_disable();
2.local_irq_save();
3. check percpu counter htab->map_locked[hash] for recursion; 3.1. if map_lock[hash] is already taken, return -BUSY;
4.raw_spin_lock().Similarly, use
raw_spin_unlock()andlocal_irq_restore()inhtab_unlock_bucket().Suggested-by Tejun Heo <[email protected]>Signed-off-by Song Liu <[email protected]>Quoted 8 lines
---
Changes in v2:
1. Useraw_spin_unlock()andlocal_irq_restore()inhtab_unlock_bucket(). (Andrii)
---kernel/bpf/hashtab.c| 7 +++++--1 file changed, 5 insertions(+), 2 deletions(-)Now it's more symmetrical and seems correct to me, thanks!
Acked-by Andrii Nakryiko <[email protected]>Quoted 16 lines
kernel/bpf/hashtab.c +2
-0@@ -155,13 +155,15 @@ static inline int htab_lock_bucket(const struct bpf_htab *htab, hash = hash &min_t (u32,HASHTAB_MAP_LOCK_MASK , htab->n_buckets -1 );preempt_disable (); +local_irq_save (flags);if (unlikely (__this_cpu_inc_return (*(htab->map_locked[hash])) !=1 )) {__this_cpu_dec (*(htab->map_locked[hash])); +local_irq_restore (flags);preempt_enable ();return -EBUSY ; }
Quoted 7 lines
Quoted 2 lines
kernel/bpf/hashtab.c static inline int htab_lock_bucket(const struct bpf_htab *htab, · line 166
Quoted 2 lines
kernel/bpf/hashtab.c +1
-1-raw_spin_lock_irqsave (&b->raw_lock, flags); +raw_spin_lock (&b->raw_lock);Song,
take a look at s390 crash in BPF CI.
I suspect this patch is causing it.
It indeed looks like triggered by this patch. But I haven't figured out why it happens. v1 seems ok for the same tests.
Song
Quoted 4 lines
Ilya,
do you have an idea what is going on?
Quoted 71 lines · Song Liu
Quoted 68 lines · Alexei Starovoitov
Quoted 56 lines · Andrii Nakryiko
Quoted 25 lines · Song Liu
htab_lock_bucketuses the following logic to avoid recursion:1.
preempt_disable();
2. check percpu counter htab->map_locked[hash] for recursion; 2.1. if map_lock[hash] is already taken, return -BUSY;
3.raw_spin_lock_irqsave();However, if an IRQ hits between 2 and 3, BPF programs attached to the IRQ logic will not able to access the same hash of the hashtab and get
-EBUSY. This-EBUSYis not really necessary. Fix it by disabling IRQ before checkingmap_locked:1.
preempt_disable();
2.local_irq_save();
3. check percpu counter htab->map_locked[hash] for recursion; 3.1. if map_lock[hash] is already taken, return -BUSY;
4.raw_spin_lock().Similarly, use
raw_spin_unlock()andlocal_irq_restore()inhtab_unlock_bucket().Suggested-by Tejun Heo <[email protected]>Signed-off-by Song Liu <[email protected]>Quoted 8 lines
---
Changes in v2:
1. Useraw_spin_unlock()andlocal_irq_restore()inhtab_unlock_bucket(). (Andrii)
---kernel/bpf/hashtab.c| 7 +++++--1 file changed, 5 insertions(+), 2 deletions(-)Now it's more symmetrical and seems correct to me, thanks!
Acked-by Andrii Nakryiko <[email protected]>Quoted 16 lines
kernel/bpf/hashtab.c +2
-0@@ -155,13 +155,15 @@ static inline int htab_lock_bucket(const struct bpf_htab *htab, hash = hash &min_t (u32,HASHTAB_MAP_LOCK_MASK , htab->n_buckets -1 );preempt_disable (); +local_irq_save (flags);if (unlikely (__this_cpu_inc_return (*(htab->map_locked[hash])) !=1 )) {__this_cpu_dec (*(htab->map_locked[hash])); +local_irq_restore (flags);preempt_enable ();return -EBUSY ; }Quoted 2 lines
kernel/bpf/hashtab.c static inline int htab_lock_bucket(const struct bpf_htab *htab, · line 166
Quoted 2 lines
kernel/bpf/hashtab.c +1
-1-raw_spin_lock_irqsave (&b->raw_lock, flags); +raw_spin_lock (&b->raw_lock);Song,
Quoted 5 lines
Quoted 2 lines
take a look at s390 crash in BPF CI.
I suspect this patch is causing it.It indeed looks like triggered by this patch. But I haven't figured out why it happens. v1 seems ok for the same tests.
I guess I finally figured out this (should be simple) bug. If I got it correctly, we need:
kernel/bpf/hashtab.c +1 -0
@@ -160,6 +160,7 @@ static inline int htab_lock_bucket(const struct bpf_htab *htab,__this_cpu_dec (*(htab->map_locked[hash]));local_irq_restore (flags);preempt_enable (); + *pflags = flags;return -EBUSY ; }
Running CI tests here:
https://github.com/kernel-patches/bpf/pull/5769
If it works, I will send v3.
Thanks,
Song
PS: s390x CI is running slow. I got some jobs stayed in the queue for more than a hour.
Quoted 8 lines
Song
Quoted 4 lines
Ilya,
do you have an idea what is going on?
Quoted 89 lines · Song Liu
Quoted 72 lines · Song Liu
Quoted 68 lines · Alexei Starovoitov
Quoted 56 lines · Andrii Nakryiko
Quoted 25 lines · Song Liu
htab_lock_bucketuses the following logic to avoid recursion:1.
preempt_disable();
2. check percpu counter htab->map_locked[hash] for recursion; 2.1. if map_lock[hash] is already taken, return -BUSY;
3.raw_spin_lock_irqsave();However, if an IRQ hits between 2 and 3, BPF programs attached to the IRQ logic will not able to access the same hash of the hashtab and get
-EBUSY. This-EBUSYis not really necessary. Fix it by disabling IRQ before checkingmap_locked:1.
preempt_disable();
2.local_irq_save();
3. check percpu counter htab->map_locked[hash] for recursion; 3.1. if map_lock[hash] is already taken, return -BUSY;
4.raw_spin_lock().Similarly, use
raw_spin_unlock()andlocal_irq_restore()inhtab_unlock_bucket().Suggested-by Tejun Heo <[email protected]>Signed-off-by Song Liu <[email protected]>Quoted 8 lines
---
Changes in v2:
1. Useraw_spin_unlock()andlocal_irq_restore()inhtab_unlock_bucket(). (Andrii)
---kernel/bpf/hashtab.c| 7 +++++--1 file changed, 5 insertions(+), 2 deletions(-)Now it's more symmetrical and seems correct to me, thanks!
Acked-by Andrii Nakryiko <[email protected]>Quoted 16 lines
kernel/bpf/hashtab.c +2
-0@@ -155,13 +155,15 @@ static inline int htab_lock_bucket(const struct bpf_htab *htab, hash = hash &min_t (u32,HASHTAB_MAP_LOCK_MASK , htab->n_buckets -1 );preempt_disable (); +local_irq_save (flags);if (unlikely (__this_cpu_inc_return (*(htab->map_locked[hash])) !=1 )) {__this_cpu_dec (*(htab->map_locked[hash])); +local_irq_restore (flags);preempt_enable ();return -EBUSY ; }Quoted 2 lines
kernel/bpf/hashtab.c static inline int htab_lock_bucket(const struct bpf_htab *htab, · line 166
Quoted 2 lines
kernel/bpf/hashtab.c +1
-1-raw_spin_lock_irqsave (&b->raw_lock, flags); +raw_spin_lock (&b->raw_lock);Song,
Quoted 5 lines
Quoted 2 lines
take a look at s390 crash in BPF CI.
I suspect this patch is causing it.It indeed looks like triggered by this patch. But I haven't figured out why it happens. v1 seems ok for the same tests.
I guess I finally figured out this (should be simple) bug. If I got it correctly, we need:
kernel/bpf/hashtab.c +0
-0
kernel/bpf/hashtab.c static inline int htab_lock_bucket(const struct bpf_htab *htab, · line 165
Quoted 6 lines
kernel/bpf/hashtab.c +1
-0@@ -160,6 +160,7 @@ static inline int htab_lock_bucket(const struct bpf_htab *htab,__this_cpu_dec (*(htab->map_locked[hash]));local_irq_restore (flags);preempt_enable (); + *pflags = flags;return -EBUSY ; }
No... I was totally wrong. This is not needed.
Trying something different..
Thanks,
Song
Quoted 3 lines · Song Liu
Quoted 1 line · Song Liu
[...]
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 17 lines
Quoted 17 lines
kernel/bpf/hashtab.c +2
-0@@ -155,13 +155,15 @@ static inline int htab_lock_bucket(const struct bpf_htab *htab, hash = hash &min_t (u32,HASHTAB_MAP_LOCK_MASK , htab->n_buckets -1 );preempt_disable (); +local_irq_save (flags);if (unlikely (__this_cpu_inc_return (*(htab->map_locked[hash])) !=1 )) {__this_cpu_dec (*(htab->map_locked[hash])); +local_irq_restore (flags);preempt_enable ();return -EBUSY ; }Quoted 2 lines
kernel/bpf/hashtab.c static inline int htab_lock_bucket(const struct bpf_htab *htab, · line 166
Quoted 2 lines
kernel/bpf/hashtab.c +1
-1-raw_spin_lock_irqsave (&b->raw_lock, flags); +raw_spin_lock (&b->raw_lock);Song,
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
take a look at s390 crash in BPF CI.
I suspect this patch is causing it.It indeed looks like triggered by this patch. But I haven't figured out why it happens. v1 seems ok for the same tests.
Update my findings today:
I tried to reproduce the issue locally with qemu on my server (x86_64). I got the following artifacts:
1. bzImage and selftests from CI: (need to login to GitHub)
https://github.com/kernel-patches/bpf/suites/16885416280/artifacts/964765766
2. cross compiler:
https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.2.0/x86_64-gcc-13.2.0-nolibc-s390-linux.tar.gz
3. root image:
https://libbpf-ci.s3.us-west-1.amazonaws.com/libbpf-vmtest-rootfs-2022.10.23-bullseye-s390x.tar.zst
With bzImage compiled in CI, I can reproduce the issue with qemu. However, if I compile the kernel locally with the cross compiler (with .config from CI, and then olddefconfig), the issue cannot be reproduced. I have attached the two .config files here. They look very similar, except the compiler version:
-CONFIG_CC_VERSION_TEXT="gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0" +CONFIG_CC_VERSION_TEXT="s390-linux-gcc (GCC) 13.2.0"
So far, I still think v2 is the right patch. But I really cannot explain the issue on the bzImage from the CI. I cannot make much sense out of the s390 assembly code either. (TIL: gdb on my x86 server can disassem s390 binary).
Ilya,
Could you please take a look at this?
Thanks,
Song
PS: the root image from the CI is not easy to use. Hopefully you have something better than that.
Quoted 70 lines · Song Liu
Quoted 4 lines
Quoted 2 lines · Song Liu
On Oct 4, 2023, at 5:11 PM, Song Liu <[email protected]> wrote:
[...]
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 22 lines
Quoted 22 lines
kernel/bpf/hashtab.c +1
-0@@ -155,13 +155,15 @@ static inline int htab_lock_bucket(const struct bpf_htab *htab, hash = hash & min_t(u32, HASHTAB_MAP_LOCK_MASK, htab->n_buckets -1 ); preempt_disable(); + local_irq_save(flags);if (unlikely(__this_cpu_inc_return(*(htab-
Quoted 1 line
map_locked[hash])) != 1)) {
__this_cpu_dec(*(htab->map_locked[hash])); +
local_irq_restore(flags);preempt_enable();
return-EBUSY;
}-
raw_spin_lock_irqsave(&b->raw_lock, flags); +raw_spin_lock(&b->raw_lock);Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Song,
take a look at s390 crash in BPF CI.
I suspect this patch is causing it.It indeed looks like triggered by this patch. But I haven't figured
out why it happens. v1 seems ok for the same tests.Update my findings today:
I tried to reproduce the issue locally with qemu on my server (
x86_64).
I got the following artifacts:1. bzImage and selftests from CI: (need to login to GitHub)
https://github.com/kernel-patches/bpf/suites/16885416280/artifacts/9647657662. cross compiler:
https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.2.0/x86_64-gcc-13.2.0-nolibc-s390-linux.tar.gz3. root image:
https://libbpf-ci.s3.us-west-1.amazonaws.com/libbpf-vmtest-rootfs-2022.10.23-bullseye-s390x.tar.zstWith bzImage compiled in CI, I can reproduce the issue with qemu. However, if I compile the kernel locally with the cross compiler (with .config from CI, and then olddefconfig), the issue cannot be reproduced. I have attached the two .config files here. They look very similar, except the compiler version:
(quoted diff) +1
-1-CONFIG_CC_VERSION_TEXT="gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0" +CONFIG_CC_VERSION_TEXT="s390-linux-gcc (GCC) 13.2.0"So far, I still think v2 is the right patch. But I really cannot explain the issue on the bzImage from the CI. I cannot make much sense out of the s390 assembly code either. (TIL: gdb on my x86 server can disassem s390 binary).
Ilya,
Quoted 7 lines
Could you please take a look at this?
Thanks,
SongPS: the root image from the CI is not easy to use. Hopefully you have something better than that.
Hi,
sorry for the long silence, I was traveling and then sick. I'll look into this ASAP.
Best regards,
Ilya
Quoted 70 lines · Song Liu
Quoted 4 lines
Quoted 2 lines · Song Liu
On Oct 4, 2023, at 5:11 PM, Song Liu <[email protected]> wrote:
[...]
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 22 lines
Quoted 22 lines
kernel/bpf/hashtab.c +1
-0@@ -155,13 +155,15 @@ static inline int htab_lock_bucket(const struct bpf_htab *htab, hash = hash & min_t(u32, HASHTAB_MAP_LOCK_MASK, htab->n_buckets -1 ); preempt_disable(); + local_irq_save(flags);if (unlikely(__this_cpu_inc_return(*(htab-
Quoted 1 line
map_locked[hash])) != 1)) {
__this_cpu_dec(*(htab->map_locked[hash])); +
local_irq_restore(flags);preempt_enable();
return-EBUSY;
}-
raw_spin_lock_irqsave(&b->raw_lock, flags); +raw_spin_lock(&b->raw_lock);Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Song,
take a look at s390 crash in BPF CI.
I suspect this patch is causing it.It indeed looks like triggered by this patch. But I haven't figured
out why it happens. v1 seems ok for the same tests.Update my findings today:
I tried to reproduce the issue locally with qemu on my server (
x86_64).
I got the following artifacts:1. bzImage and selftests from CI: (need to login to GitHub)
https://github.com/kernel-patches/bpf/suites/16885416280/artifacts/9647657662. cross compiler:
https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.2.0/x86_64-gcc-13.2.0-nolibc-s390-linux.tar.gz3. root image:
https://libbpf-ci.s3.us-west-1.amazonaws.com/libbpf-vmtest-rootfs-2022.10.23-bullseye-s390x.tar.zstWith bzImage compiled in CI, I can reproduce the issue with qemu. However, if I compile the kernel locally with the cross compiler (with .config from CI, and then olddefconfig), the issue cannot be reproduced. I have attached the two .config files here. They look very similar, except the compiler version:
(quoted diff) +1
-1-CONFIG_CC_VERSION_TEXT="gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0" +CONFIG_CC_VERSION_TEXT="s390-linux-gcc (GCC) 13.2.0"So far, I still think v2 is the right patch. But I really cannot explain the issue on the bzImage from the CI. I cannot make much sense out of the s390 assembly code either. (TIL: gdb on my x86 server can disassem s390 binary).
Ilya,
Quoted 7 lines
Could you please take a look at this?
Thanks,
SongPS: the root image from the CI is not easy to use. Hopefully you have something better than that.
Hi,
Thanks for posting the analysis and links to the artifacts, that saved me quite some time. The crash is caused by a backchain issue in the trampoline code and has nothing to do with your patch; I've posted the fix here [1].
The difference between compilers is most likely caused by different inlining decisions around lookup_elem_raw(). When it's inlined, the test becomes a no-op.
Regarding GDB, Debian and Ubuntu have gdb-multiarch package. On Fedora one has to build it from source; the magic binutils-gdb configure flag is --enable-targets=all.
Regarding the development environment, in this case I've sidestepped the need for a proper image by putting everything into initramfs:
workdir=$(mktemp -d)
tar -C "$workdir" -xf libbpf-vmtest-rootfs-2022.10.23-bullseye- s390x.tar.zst
rsync -a selftests "$workdir"
(cd "$workdir" && find . | cpio -o --format=newc -R root:root)
Quoted 1 line
initrd.cpio
qemu-system-s390x -accel kvm -smp 2 -m 4G -kernel kbuild- output/arch/s390/boot/bzImage -nographic -append 'nokaslr console=ttyS1 rdinit=/bin/sh' -initrd initrd.cpio -s
For the regular development I have a different setup, with a non-minimal Debian install inside the guest, and the testcases mounted from the host using 9p.
Best regards,
Ilya
Song Liu
reviewerHi Ilya,
Quoted 1 line · Ilya Leoshkevich
[...]
Quoted 7 lines
Quoted 6 lines
Thanks,
SongPS: the root image from the CI is not easy to use. Hopefully you have something better than that.
Quoted 6 lines
Hi,
Thanks for posting the analysis and links to the artifacts, that saved me quite some time. The crash is caused by a backchain issue in the trampoline code and has nothing to do with your patch; I've posted the fix here [1].
Thanks for the fix!
Song
Quoted 17 lines
The difference between compilers is most likely caused by different inlining decisions around
lookup_elem_raw(). When it's inlined, the test becomes a no-op.Regarding GDB, Debian and Ubuntu have gdb-multiarch package. On Fedora one has to build it from source; the magic binutils-gdb configure flag is --enable-targets=all.
Regarding the development environment, in this case I've sidestepped the need for a proper image by putting everything into initramfs:
workdir=$(mktemp -d)
tar -C "$workdir" -xf libbpf-vmtest-rootfs-2022.10.23-bullseye- s390x.tar.zst
rsync -a selftests "$workdir"
(cd "$workdir" && find . | cpio -o --format=newc -R root:root)
Quoted 4 lines
Quoted 1 line
initrd.cpio
qemu-system-s390x -accel kvm -smp 2 -m 4G -kernel kbuild- output/arch/s390/boot/bzImage -nographic -append 'nokaslr console=ttyS1 rdinit=/bin/sh' -initrd initrd.cpio -s
Nice trick!
Quoted 9 lines
For the regular development I have a different setup, with a non-minimal Debian install inside the guest, and the testcases mounted from the host using 9p.
Best regards,
Ilya
Quoted 15 lines · Song Liu
Hi Ilya,
Quoted 1 line · Ilya Leoshkevich
[...]
Quoted 9 lines
Quoted 6 lines
Thanks,
SongPS: the root image from the CI is not easy to use. Hopefully you have something better than that.
Hi,
Quoted 8 lines
Quoted 4 lines
Thanks for posting the analysis and links to the artifacts, that saved me quite some time. The crash is caused by a backchain issue in the trampoline code and has nothing to do with your patch; I've posted the fix here [1].
Thanks for the fix!
Song
Song, can you please resend your patch so that CI can test it again on top of Ilya's changes? Thanks!
Quoted 34 lines
Quoted 18 lines
The difference between compilers is most likely caused by different inlining decisions around
lookup_elem_raw(). When it's inlined, the test becomes a no-op.Regarding GDB, Debian and Ubuntu have gdb-multiarch package. On Fedora one has to build it from source; the magic binutils-gdb configure flag is --enable-targets=all.
Regarding the development environment, in this case I've sidestepped the need for a proper image by putting everything into initramfs:
workdir=$(mktemp -d)
tar -C "$workdir" -xf libbpf-vmtest-rootfs-2022.10.23-bullseye- s390x.tar.zst
rsync -a selftests "$workdir"
(cd "$workdir" && find . | cpio -o --format=newc -R root:root)Quoted 4 lines
Quoted 1 line
initrd.cpio
qemu-system-s390x -accel kvm -smp 2 -m 4G -kernel kbuild- output/arch/s390/boot/bzImage -nographic -append 'nokaslr console=ttyS1 rdinit=/bin/sh' -initrd initrd.cpio -s
Nice trick!
Quoted 9 lines
For the regular development I have a different setup, with a non-minimal Debian install inside the guest, and the testcases mounted from the host using 9p.
Best regards,
Ilya
Quoted 25 lines · Andrii Nakryiko
Quoted 20 lines · Song Liu
Hi Ilya,
Quoted 1 line · Ilya Leoshkevich
[...]
Quoted 13 lines
Quoted 6 lines
Thanks,
SongPS: the root image from the CI is not easy to use. Hopefully you have something better than that.
Hi,
Thanks for posting the analysis and links to the artifacts, that saved me quite some time. The crash is caused by a backchain issue in the trampoline code and has nothing to do with your patch; I've posted the fix here [1].
Quoted 6 lines
Quoted 3 lines
Thanks for the fix!
Song
Song, can you please resend your patch so that CI can test it again on top of Ilya's changes? Thanks!
Sure. Just sent the patch to bpf tree.
Thanks,
Song