perf test: Regression because of d6e6286a12e7
Hi Andrii,
I'm seeing perf test failure because of commit d6e6286a12e7 ("libbpf:
disassociate section handler on explicit bpf_program__set_type() call").
With 6.4 kernel and perf:
$ ./tools/perf/perf version perf version 6.4.g6995e2de6891 $ sudo ./tools/perf/perf test 42 42: BPF filter : 42.1: Basic BPF filtering : Ok 42.2: BPF pinning : Ok 42.3: BPF prologue generation : FAILED!
After reverting d6e6286a12e7 commit:
$ git revert d6e6286a12e7 $ make -C tools/perf/ $ sudo ./tools/perf/perf test 42 42: BPF filter : 42.1: Basic BPF filtering : Ok 42.2: BPF pinning : Ok 42.3: BPF prologue generation : Ok
Is it a known issue?
Thanks,
Ravi
Quoted 6 lines · Ravi Bangoria
Hi Andrii,
I'm seeing perf test failure because of commit
d6e6286a12e7("libbpf:
disassociate section handler on explicitbpf_program__set_type()call").
Yep, this commit would reset catch-all custom handler, which perf is setting. I've just sent a fix upstream ([0]). And once it lands, I'll cut a v1.2.1 libbpf bugfix release with just this fix on top of v1.2.
Can you please double-check that this patch indeed fixes the issue for you? I tried to do this locally, but for me perf test 42 fails both at current bpf-next, with the above commit reverted, and with my fix applied on top. So I can't be 100% sure.
Thanks!
[0] https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]/
Quoted 24 lines
With 6.4 kernel and perf:
$ ./tools/perf/perf version perf version 6.4.g6995e2de6891 $ sudo ./tools/perf/perf test 42 42: BPF filter : 42.1: Basic BPF filtering : Ok 42.2: BPF pinning : Ok 42.3: BPF prologue generation : FAILED!After reverting
d6e6286a12e7commit:$ git revert d6e6286a12e7 $ make -C tools/perf/ $ sudo ./tools/perf/perf test 42 42: BPF filter : 42.1: Basic BPF filtering : Ok 42.2: BPF pinning : Ok 42.3: BPF prologue generation : OkIs it a known issue?
Thanks,
Ravi
Quoted 12 lines · Andrii Nakryiko
Quoted 6 lines · Ravi Bangoria
Hi Andrii,
I'm seeing perf test failure because of commit
d6e6286a12e7("libbpf:
disassociate section handler on explicitbpf_program__set_type()call").Yep, this commit would reset catch-all custom handler, which perf is setting. I've just sent a fix upstream ([0]). And once it lands, I'll cut a v1.2.1 libbpf bugfix release with just this fix on top of v1.2.
Quoted 8 lines
Can you please double-check that this patch indeed fixes the issue for you? I tried to do this locally, but for me perf test 42 fails both at current bpf-next, with the above commit reverted, and with my fix applied on top. So I can't be 100% sure.
Thanks!
[0] https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]/
Thanks. A quick test seems to be working fine.
Ravi
Quoted 19 lines · Ravi Bangoria
Quoted 17 lines · Andrii Nakryiko
Quoted 6 lines · Ravi Bangoria
Hi Andrii,
I'm seeing perf test failure because of commit
d6e6286a12e7("libbpf:
disassociate section handler on explicitbpf_program__set_type()call").Yep, this commit would reset catch-all custom handler, which perf is setting. I've just sent a fix upstream ([0]). And once it lands, I'll cut a v1.2.1 libbpf bugfix release with just this fix on top of v1.2.
Can you please double-check that this patch indeed fixes the issue for you? I tried to do this locally, but for me perf test 42 fails both at current bpf-next, with the above commit reverted, and with my fix applied on top. So I can't be 100% sure.
Quoted 5 lines
Quoted 3 lines
Thanks!
[0] https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]/
Thanks. A quick test seems to be working fine.
Alright, thanks for confirming! I've just released v1.2.1 bug fix release with just this fix on top of v1.2.
Thanks for reporting!
But given v1.2 was cut on May 1st, and the offending commit landed some time late March, I wonder how did this slip through the cracks and go unreported for so long? Is there something we can do to catch these perf-only regressions a bit sooner?
Quoted 2 lines
Ravi
Quoted 29 lines · Andrii Nakryiko
Quoted 19 lines · Ravi Bangoria
Quoted 17 lines · Andrii Nakryiko
Quoted 6 lines · Ravi Bangoria
Hi Andrii,
I'm seeing perf test failure because of commit
d6e6286a12e7("libbpf:
disassociate section handler on explicitbpf_program__set_type()call").Yep, this commit would reset catch-all custom handler, which perf is setting. I've just sent a fix upstream ([0]). And once it lands, I'll cut a v1.2.1 libbpf bugfix release with just this fix on top of v1.2.
Can you please double-check that this patch indeed fixes the issue for you? I tried to do this locally, but for me perf test 42 fails both at current bpf-next, with the above commit reverted, and with my fix applied on top. So I can't be 100% sure.
Quoted 5 lines
Quoted 3 lines
Thanks!
[0] https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]/
Thanks. A quick test seems to be working fine.
Alright, thanks for confirming! I've just released v1.2.1 bug fix release with just this fix on top of v1.2.
Quoted 6 lines
Thanks for reporting!
But given v1.2 was cut on May 1st, and the offending commit landed some time late March, I wonder how did this slip through the cracks and go unreported for so long? Is there something we can do to catch these perf-only regressions a bit sooner?
I guess it got slipped because that patch went in via bpf tree. Would it be possible to run bpf related perf tests at the time of applying libbpf patches? Arnaldo might have better ideas :)
Thanks,
Ravi
Em Wed, Jul 12, 2023 at 07:38:58PM +0530, Ravi Bangoria escreveu:
Quoted 25 lines · Andrii Nakryiko
Quoted 23 lines · Ravi Bangoria
Quoted 9 lines · Andrii Nakryiko
Quoted 7 lines · Ravi Bangoria
Quoted 2 lines
I'm seeing perf test failure because of commit
d6e6286a12e7("libbpf:
disassociate section handler on explicitbpf_program__set_type()call").Yep, this commit would reset catch-all custom handler, which perf is setting. I've just sent a fix upstream ([0]). And once it lands, I'll cut a v1.2.1 libbpf bugfix release with just this fix on top of v1.2.
Quoted 8 lines
Quoted 6 lines
Can you please double-check that this patch indeed fixes the issue for you? I tried to do this locally, but for me perf test 42 fails both at current bpf-next, with the above commit reverted, and with my fix applied on top. So I can't be 100% sure.
[0] https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]/
Thanks. A quick test seems to be working fine.
Alright, thanks for confirming! I've just released v1.2.1 bug fix release with just this fix on top of v1.2.
Thanks for reporting!
Quoted 8 lines
Quoted 4 lines
But given v1.2 was cut on May 1st, and the offending commit landed some time late March, I wonder how did this slip through the cracks and go unreported for so long? Is there something we can do to catch these perf-only regressions a bit sooner?
I guess it got slipped because that patch went in via bpf tree. Would it be possible to run bpf related perf tests at the time of applying libbpf patches? Arnaldo might have better ideas :)
Right, perhaps the libbpf CI could try building perf, preferably with BUILD_BPF_SKEL=1, to enable these tools:
[acme@nine linux]$ ls -la tools/perf/util/bpf_skel/*.bpf.*
-rw-r--r--. 1 acme acme 5581 Jul 7 12:38 tools/perf/util/bpf_skel/bperf_cgroup.bpf.c -rw-r--r--. 1 acme acme 1764 Jul 7 12:38 tools/perf/util/bpf_skel/bperf_follower.bpf.c -rw-r--r--. 1 acme acme 1438 Jul 7 12:38 tools/perf/util/bpf_skel/bperf_leader.bpf.c -rw-r--r--. 1 acme acme 2290 Jul 7 12:38 tools/perf/util/bpf_skel/bpf_prog_profiler.bpf.c -rw-r--r--. 1 acme acme 2164 Jul 7 12:38 tools/perf/util/bpf_skel/func_latency.bpf.c -rw-r--r--. 1 acme acme 9017 Jul 7 12:38 tools/perf/util/bpf_skel/kwork_trace.bpf.c -rw-r--r--. 1 acme acme 10147 Jul 12 11:49 tools/perf/util/bpf_skel/lock_contention.bpf.c -rw-r--r--. 1 acme acme 6109 Jul 7 12:38 tools/perf/util/bpf_skel/off_cpu.bpf.c -rw-r--r--. 1 acme acme 4932 Jul 7 12:38 tools/perf/util/bpf_skel/sample_filter.bpf.c
[acme@nine linux]$
There are 'perf test' entries for the BPF support in perf that is pre libbpf skel functionality above and as well some for the libbpf based skel features, such as:
[root@quaco ~]# perf test -vvv contention
87: kernel lock contention analysis test :
--- start ---
test child forked, pid 213314
Testing perf lock record and perf lock contention
Testing perf lock contention --use-bpf
Testing perf lock record and perf lock contention at the same time Testing perf lock contention --threads
Testing perf lock contention --lock-addr
Testing perf lock contention --type-filter (w/ spinlock) Testing perf lock contention --lock-filter (w/ tasklist_lock) Testing perf lock contention --callstack-filter (w/ unix_stream) Testing perf lock contention --callstack-filter with task aggregation Testing perf lock contention CSV output
test child finished with 0
---- end ----
kernel lock contention analysis test: Ok
[root@quaco ~]#
We could perhaps add support for some kind of tags to the tests, to help run just the ones that use bpf and then you could use:
# perf test --uses bpf
At first we would just add this tag, then we would use it for other stuff as the need arises to run just a subset of the tests.
But for now you could start perhaps with:
[root@quaco ~]# perf test LLVM ; perf test "BPF " ; perf test "lock contention"
40: LLVM search and compile : 40.1: Basic BPF llvm compile : Ok 40: LLVM search and compile : 40.1: Basic BPF llvm compile : Ok 40.3: Compile source for BPF prologue generation : Ok 40.4: Compile source for BPF relocation : Ok 42: BPF filter : 42.1: Basic BPF filtering : Ok 42.2: BPF pinning : Ok 42.3: BPF prologue generation : FAILED! 87: kernel lock contention analysis test : Ok
[root@quaco ~]#
- Arnaldo
Quoted 3 lines · Arnaldo Carvalho de Melo
Right, perhaps the libbpf CI could try building perf, preferably with BUILD_BPF_SKEL=1, to enable these tools:
That would be great.
perf experts probably should do pull-req to bpf CI to enable that. See slides:
http://vger.kernel.org/bpfconf2022_material/lsfmmbpf2022-bpf-ci.pdf
"How to contribute?
Depending on what part of CI you are changing, you can create a pull request to
https://github.com/kernel-patches/vmtest/
https://github.com/libbpf/ci
"
Em Wed, Jul 12, 2023 at 11:20:27AM -0700, Alexei Starovoitov escreveu:
Quoted 12 lines
Quoted 3 lines · Arnaldo Carvalho de Melo
Right, perhaps the libbpf CI could try building perf, preferably with BUILD_BPF_SKEL=1, to enable these tools:
That would be great.
perf experts probably should do pull-req to bpf CI to enable that. See slides:
http://vger.kernel.org/bpfconf2022_material/lsfmmbpf2022-bpf-ci.pdf
Quoted 5 lines
"How to contribute?
Depending on what part of CI you are changing, you can create a pull request to
https://github.com/kernel-patches/vmtest/
https://github.com/libbpf/ci
"
Sure, I still recall Quentin's talk about CI, etc in Dublin, will come up with something and submit.
- Arnaldo
Hey Arnaldo,
Quoted 20 lines · Arnaldo Carvalho de Melo
!-------------------------------------------------------------------|
This Message Is From an External Sender|-------------------------------------------------------------------!
Em Wed, Jul 12, 2023 at 11:20:27AM -0700, Alexei Starovoitov escreveu:
Quoted 12 lines
Quoted 3 lines · Arnaldo Carvalho de Melo
Right, perhaps the libbpf CI could try building perf, preferably with BUILD_BPF_SKEL=1, to enable these tools:
That would be great.
perf experts probably should do pull-req to bpf CI to enable that. See slides:
http://vger.kernel.org/bpfconf2022_material/lsfmmbpf2022-bpf-ci.pdf
Quoted 8 lines
Quoted 5 lines
"How to contribute?
Depending on what part of CI you are changing, you can create a pull request to
https://github.com/kernel-patches/vmtest/
https://github.com/libbpf/ci
"Sure, I still recall Quentin's talk about CI, etc in Dublin, will come up with something and submit.
Thanks for looking at this!
If you will have any questions on how CI works, do not hesitate to join BPF office hours and we will do our best to answer.
Mykola
Quoted 2 lines
- Arnaldo
A message that is not in the archive (sent off-list or not yet received): SA1PR15MB46099ABDC08009096019B5B4CBF7A@SA1PR15MB4609.namprd15.prod.outlook.com
Quoted 7 lines · Manu Bretelle
Hi Arnaldo,
Checking back here to see if there is anything you need help in order to add perf support to BPF CI. Were you able to make progress and are hitting some issues along the way?
Separate from the CI issue there were some updates on perf:
- we're looking to re-enable BPF skeletons by default for 6.7:
https://lore.kernel.org/lkml/[email protected]/
- Ravi's original failure can no longer fail as we removed the BPF
filter events in favor of a BPF skeleton based --filter option:
https://lore.kernel.org/lkml/[email protected]/
As such these tests no longer exist. Other tests like kernel lock contention analysis implicitly use BPF and so we are still testing BPF.
Thanks,
Ian
Quoted 52 lines
Thanks,
Manu
From: Mykola Lysenko <[email protected]>
Date: Friday, July 14, 2023 at 11:15 AM
To: Arnaldo Carvalho de Melo <[email protected]>
Cc: Mykola Lysenko <[email protected]>, Alexei Starovoitov <[email protected]>, Ravi Bangoria <[email protected]>, Andrii Nakryiko <[email protected]>, Andrii Nakryiko <[email protected]>, Jiri Olsa <[email protected]>, Namhyung Kim <[email protected]>, Ian Rogers <[email protected]>, linux-perf-users <[email protected]>, bpf <[email protected]>, Manu Bretelle <[email protected]>, Daniel Müller <[email protected]>, Mykola Lysenko <[email protected]>
Subject: Re: [BUG] perf test: Regression because ofd6e6286a12e7Hey Arnaldo,
Quoted 17 lines · Arnaldo Carvalho de Melo
Em Wed, Jul 12, 2023 at 11:20:27AM -0700, Alexei Starovoitov escreveu:
Quoted 12 lines
Quoted 3 lines · Arnaldo Carvalho de Melo
Right, perhaps the libbpf CI could try building perf, preferably with BUILD_BPF_SKEL=1, to enable these tools:
That would be great.
perf experts probably should do pull-req to bpf CI to enable that. See slides:
http://vger.kernel.org/bpfconf2022_material/lsfmmbpf2022-bpf-ci.pdfQuoted 8 lines
Quoted 5 lines
"How to contribute?
Depending on what part of CI you are changing, you can create a pull request to
https://github.com/kernel-patches/vmtest/
https://github.com/libbpf/ci
"Sure, I still recall Quentin's talk about CI, etc in Dublin, will come up with something and submit.
Thanks for looking at this!
If you will have any questions on how CI works, do not hesitate to join BPF office hours and we will do our best to answer.
Mykola
Quoted 2 lines
- Arnaldo
Em Thu, Sep 14, 2023 at 09:43:41PM +0000, Manu Bretelle escreveu:
Quoted 5 lines
Hi Arnaldo,
Checking back here to see if there is anything you need help in order to add perf support to BPF CI. Were you able to make progress and are hitting some issues along the way?
I'm travelling now, but I saw that Ian replied.
No progress on my part, but I would start with a simple:
Build perf from the same kernel source tree used in the current CI tests, then run 'perf test'. It would test more than just the BPF part, but would be a good start.
A second step would be just disabling the tests that are failing and that are not BPF specific, using the --skip arg to 'perf test':
[root@quaco test]# perf test -h skip
Usage: perf test [<options>] [{list <test-name-fragment>|[<test-name-fragments>|<test-numbers>]}]
-s, --skip <tests> tests to skip
[root@quaco test]#
So something like:
-----------------
make -C tools/perf
perf test --skip list,of,tests,failing,that,are,not,BPF,specific
-----------------
The perf build dependencies should be similar to the ones needed to build libbpf or close to it, so the above would be a great first stab at it.
- Arnaldo
Quoted 50 lines
Thanks,
Manu
From: Mykola Lysenko <[email protected]>
Date: Friday, July 14, 2023 at 11:15 AM
To: Arnaldo Carvalho de Melo <[email protected]>
Cc: Mykola Lysenko <[email protected]>, Alexei Starovoitov <[email protected]>, Ravi Bangoria <[email protected]>, Andrii Nakryiko <[email protected]>, Andrii Nakryiko <[email protected]>, Jiri Olsa <[email protected]>, Namhyung Kim <[email protected]>, Ian Rogers <[email protected]>, linux-perf-users <[email protected]>, bpf <[email protected]>, Manu Bretelle <[email protected]>, Daniel Müller <[email protected]>, Mykola Lysenko <[email protected]>
Subject: Re: [BUG] perf test: Regression because ofd6e6286a12e7
Hey Arnaldo,Quoted 20 lines · Arnaldo Carvalho de Melo
!-------------------------------------------------------------------|
This Message Is From an External Sender|-------------------------------------------------------------------!
Em Wed, Jul 12, 2023 at 11:20:27AM -0700, Alexei Starovoitov escreveu:
Quoted 12 lines
Quoted 3 lines · Arnaldo Carvalho de Melo
Right, perhaps the libbpf CI could try building perf, preferably with BUILD_BPF_SKEL=1, to enable these tools:
That would be great.
perf experts probably should do pull-req to bpf CI to enable that. See slides:
http://vger.kernel.org/bpfconf2022_material/lsfmmbpf2022-bpf-ci.pdfQuoted 8 lines
Quoted 5 lines
"How to contribute?
Depending on what part of CI you are changing, you can create a pull request to
https://github.com/kernel-patches/vmtest/
https://github.com/libbpf/ci
"Sure, I still recall Quentin's talk about CI, etc in Dublin, will come up with something and submit.
Thanks for looking at this!
If you will have any questions on how CI works, do not hesitate to join BPF office hours and we will do our best to answer.
Mykola
Quoted 2 lines
- Arnaldo
Signature · 1 line
- Arnaldo