I don't know if it is still maintained, e.g. it includes DEBUG_SLAB option. But also, it enables options that are maybe not needed: GCOV? X86_DEBUG_FPU?
Maybe it is better not to use this .config file, no?

For our CI validating MPTCP tests in a "debug" mode, we use "debug.config" without "x86_debug.config". On top of that, we also disable "SLUB_DEBUG_ON", because the impact on the perf is too important, especially with slow environments. We think it is not worth it for our case. You don't have the same hardware, but if you have perf issues, don't hesitate to do the same ;)

Quoted 2 lines

Also feel free to request membership in the linux-netdev org to be able to edit!

Linked to the discussion we had last week, I can describe which patches are being tested if you think it can help others :)

Cheers,
Matt

Signature · 2 lines

Sponsored by the NGI0 Core fund.
(matttbe on GitHub)

Jakub Kicinski

author maintainer

2024-02-07 01:37 UTC #2 lore raw

Quoted 6 lines · On Mon, 5 Feb 2024 18:21:56 +0100 Matthieu Baerts wrote:

Thank you for this wiki page, and all the work with the CI infrastructure!
For the debug options, I see that you are using:
kernel/configs/x86_debug.config

Quoted 8 lines

It looks like this is specific for the 'tip' tree:
Debugging options for tip tree testing
I don't know if it is still maintained, e.g. it includes DEBUG_SLAB option. But also, it enables options that are maybe not needed: GCOV? X86_DEBUG_FPU?
Maybe it is better not to use this .config file, no?

I haven't looked to closely. I noticed that the basic debug config doesn't enable LOCKDEP ?! so I put the x86 one on top.

I added a local patch to cut out all the obviously pointless stuff from x86_debug.config, we should probably start our own config for networking at some stage.

Quoted 6 lines

For our CI validating MPTCP tests in a "debug" mode, we use "debug.config" without "x86_debug.config". On top of that, we also disable "SLUB_DEBUG_ON", because the impact on the perf is too important, especially with slow environments. We think it is not worth it for our case. You don't have the same hardware, but if you have perf issues, don't hesitate to do the same ;)

The mptcp tests take <60min to run with debug enabled, and just a single thread / VM. I think that's fine for now. But thanks for the heads up that SLUB_DEBUG_ON is problematic, for it may matter for forwarding or net tests.

Matthieu Baerts

2024-02-07 08:50 UTC #3 lore raw

Hi Jakub,

Thank you for your reply!

Quoted 11 lines · Jakub Kicinski

Quoted 10 lines · On Mon, 5 Feb 2024 18:21:56 +0100 Matthieu Baerts wrote:
Thank you for this wiki page, and all the work with the CI infrastructure!
For the debug options, I see that you are using:
kernel/configs/x86_debug.config
It looks like this is specific for the 'tip' tree:
Debugging options for tip tree testing

Quoted 7 lines

Quoted 4 lines
I don't know if it is still maintained, e.g. it includes DEBUG_SLAB option. But also, it enables options that are maybe not needed: GCOV? X86_DEBUG_FPU?
Maybe it is better not to use this .config file, no?
I haven't looked to closely. I noticed that the basic debug config doesn't enable LOCKDEP ?! so I put the x86 one on top.

I was surprised not to see LOCKDEP there, but in fact, it is: it enables PROVE_LOCKING, which selects LOCKDEP and a few other DEBUG_xxx ones.

So maybe it is not needed to include the x86 one?

Quoted 2 lines

I added a local patch to cut out all the obviously pointless stuff from x86_debug.config

Thank you!

Quoted 2 lines

we should probably start our own config for networking at some stage.

Good idea!

On our side, we always enable DEBUG_NET, and the "debug" environment also has NET_NS_REFCNT_TRACKER. We should probably enable NET_DEV_REFCNT_TRACKER too.

Do you want me to add a new file in "kernel/configs" for net including these 3 options?

Not directly related to "Net", we also enable DEBUG_INFO (+ compressed) everywhere + KFENCE in the "non-debug" env only, disable RETPOLINE (+ mitigations=off) not to slow down the tests in already slow envs, and disable a few components we don't need to accelerate the build and boot:
DRM, SOUND, etc.

https://github.com/multipath-tcp/mptcp-upstream-virtme-docker/blob/latest/entrypoint.sh#L284

It is also possible to add some kconfig in the selftests if preferred, e.g. in

./tools/testing/selftests/net/debug.config

Quoted 11 lines

Quoted 6 lines
For our CI validating MPTCP tests in a "debug" mode, we use "debug.config" without "x86_debug.config". On top of that, we also disable "SLUB_DEBUG_ON", because the impact on the perf is too important, especially with slow environments. We think it is not worth it for our case. You don't have the same hardware, but if you have perf issues, don't hesitate to do the same ;)
The mptcp tests take <60min to run with debug enabled, and just a single thread / VM. I think that's fine for now. But thanks for the heads up that SLUB_DEBUG_ON is problematic, for it may matter for forwarding or net tests.

The longest MPTCP selftest is currently stopped after 30 minutes due to the selftest timeout. I will see what we can do. That's not just because of SLUB_DEBUG_ON, that's normal to be very slow in such particular env.

Cheers,
Matt

Signature · 1 line

Sponsored by the NGI0 Core fund.

Jakub Kicinski

author maintainer

2024-02-07 15:22 UTC #4 lore raw

Quoted 17 lines · On Wed, 7 Feb 2024 09:50:48 +0100 Matthieu Baerts wrote:

Quoted 16 lines · Jakub Kicinski
Quoted 14 lines · On Mon, 5 Feb 2024 18:21:56 +0100 Matthieu Baerts wrote:
Thank you for this wiki page, and all the work with the CI infrastructure!
For the debug options, I see that you are using:
kernel/configs/x86_debug.config
It looks like this is specific for the 'tip' tree:
Debugging options for tip tree testing
I don't know if it is still maintained, e.g. it includes DEBUG_SLAB option. But also, it enables options that are maybe not needed: GCOV? X86_DEBUG_FPU?
Maybe it is better not to use this .config file, no?

Quoted 7 lines

Quoted 2 lines
I haven't looked to closely. I noticed that the basic debug config doesn't enable LOCKDEP ?! so I put the x86 one on top.
I was surprised not to see LOCKDEP there, but in fact, it is: it enables PROVE_LOCKING, which selects LOCKDEP and a few other DEBUG_xxx ones.
So maybe it is not needed to include the x86 one?

I'm confused. Now doing:

make O=built_test defconfig
make O=built_test debug.config

I don't get KASAN but I get LOCKDEP :S Bleh, maybe because of the options vng throws in?

Quoted 8 lines

Quoted 2 lines
I added a local patch to cut out all the obviously pointless stuff from x86_debug.config
Thank you!
Quoted 2 lines
we should probably start our own config for networking at some stage.

Quoted 8 lines

Good idea!
On our side, we always enable DEBUG_NET, and the "debug" environment also has NET_NS_REFCNT_TRACKER. We should probably enable NET_DEV_REFCNT_TRACKER too.
Do you want me to add a new file in "kernel/configs" for net including these 3 options?

Just the three? What about KASAN, OBJECT debug, DEBUG_VM etc? As much as we can without going over the 3h limit in the tests :)

Quoted 2 lines

Not directly related to "Net", we also enable DEBUG_INFO (+ compressed) everywhere

We have debug_info, maybe vng adds it..

Quoted 1 line

+ KFENCE in the "non-debug" env only,

We could do KFENCE I guess. I'm a bit surprised it's not on by default for x86. My first choice would be to try to change that..

Quoted 3 lines

disable RETPOLINE (+
mitigations=off) not to slow down the tests in already slow envs, and disable a few components we don't need to accelerate the build and boot:

RETPOLINE we could kill, agreed

Quoted 3 lines

DRM, SOUND, etc.
https://github.com/multipath-tcp/mptcp-upstream-virtme-docker/blob/latest/entrypoint.sh#L284

Yes, vng also adds some stuff we don't need in this area :(

Quoted 4 lines

It is also possible to add some kconfig in the selftests if preferred, e.g. in
./tools/testing/selftests/net/debug.config

Would be great if everyone didn't have to go thru this exercise. How about we start sending patches to kernel/configs/debug.config and see if anyone screams at us?

Quoted 7 lines

Quoted 7 lines
Quoted 6 lines
For our CI validating MPTCP tests in a "debug" mode, we use "debug.config" without "x86_debug.config". On top of that, we also disable "SLUB_DEBUG_ON", because the impact on the perf is too important, especially with slow environments. We think it is not worth it for our case. You don't have the same hardware, but if you have perf issues, don't hesitate to do the same ;)

Quoted 8 lines

Quoted 4 lines
The mptcp tests take <60min to run with debug enabled, and just a single thread / VM. I think that's fine for now. But thanks for the heads up that SLUB_DEBUG_ON is problematic, for it may matter for forwarding or net tests.
The longest MPTCP selftest is currently stopped after 30 minutes due to the selftest timeout. I will see what we can do. That's not just because of SLUB_DEBUG_ON, that's normal to be very slow in such particular env.

I'll 2x the timeouts before reporting debug to patchwork, so don't worry about it, yet.

Matthieu Baerts

2024-02-07 15:59 UTC #5 lore raw

Quoted 26 lines · Jakub Kicinski

Quoted 17 lines · On Wed, 7 Feb 2024 09:50:48 +0100 Matthieu Baerts wrote:
Quoted 16 lines · Jakub Kicinski
Quoted 14 lines · On Mon, 5 Feb 2024 18:21:56 +0100 Matthieu Baerts wrote:
Thank you for this wiki page, and all the work with the CI infrastructure!
For the debug options, I see that you are using:
kernel/configs/x86_debug.config
It looks like this is specific for the 'tip' tree:
Debugging options for tip tree testing
I don't know if it is still maintained, e.g. it includes DEBUG_SLAB option. But also, it enables options that are maybe not needed: GCOV? X86_DEBUG_FPU?
Maybe it is better not to use this .config file, no?
Quoted 7 lines
Quoted 2 lines
I haven't looked to closely. I noticed that the basic debug config doesn't enable LOCKDEP ?! so I put the x86 one on top.
I was surprised not to see LOCKDEP there, but in fact, it is: it enables PROVE_LOCKING, which selects LOCKDEP and a few other DEBUG_xxx ones.
So maybe it is not needed to include the x86 one?

Quoted 7 lines

I'm confused. Now doing:
make O=built_test defconfig
make O=built_test debug.config
I don't get KASAN but I get LOCKDEP :S Bleh, maybe because of the options vng throws in?

Strange, I get both of them on my side.

Same if I include the two targets:

make O=built_test defconfig debug.config

Quoted 14 lines

Quoted 14 lines
Quoted 2 lines
I added a local patch to cut out all the obviously pointless stuff from x86_debug.config
Thank you!
Quoted 2 lines
we should probably start our own config for networking at some stage.
Good idea!
On our side, we always enable DEBUG_NET, and the "debug" environment also has NET_NS_REFCNT_TRACKER. We should probably enable NET_DEV_REFCNT_TRACKER too.

Quoted 5 lines

Quoted 2 lines
Do you want me to add a new file in "kernel/configs" for net including these 3 options?
Just the three? What about KASAN, OBJECT debug, DEBUG_VM etc? As much as we can without going over the 3h limit in the tests :)

Sorry, I meant to say: we would use the existing "debug.config" + a new "net_debug.config" containing these 3 options (and maybe more that are specific to the net tree).

Quoted 4 lines

Quoted 2 lines
Not directly related to "Net", we also enable DEBUG_INFO (+ compressed) everywhere
We have debug_info, maybe vng adds it..

Great!

Quoted 4 lines

Quoted 1 line
+ KFENCE in the "non-debug" env only,
We could do KFENCE I guess. I'm a bit surprised it's not on by default for x86. My first choice would be to try to change that..

Good point. I can ask KFENCE maintainers what they think about that.

Quoted 5 lines

Quoted 3 lines
disable RETPOLINE (+
mitigations=off) not to slow down the tests in already slow envs, and disable a few components we don't need to accelerate the build and boot:
RETPOLINE we could kill, agreed

I still need to check vrg: is it easy to add kconfig you want to enable/disable? Do you need a new file? Should we maintain this file in the tree?

Quoted 5 lines

Quoted 3 lines
DRM, SOUND, etc.
https://github.com/multipath-tcp/mptcp-upstream-virtme-docker/blob/latest/entrypoint.sh#L284
Yes, vng also adds some stuff we don't need in this area :(

With the non-ng virtme, I disable those:

-d PCCARD -d MACINTOSH_DRIVERS -d SOUND -d USB_SUPPORT -d NEW_LEDS -d SURFACE_PLATFORMS -d DRM -d FB

I will check when I will switch to vng to see what else we can disable. But on your side, you probably have more CPU resources to compile the kernel, and that's fine to keep them :)

Quoted 8 lines

Quoted 4 lines
It is also possible to add some kconfig in the selftests if preferred, e.g. in
./tools/testing/selftests/net/debug.config
Would be great if everyone didn't have to go thru this exercise. How about we start sending patches to kernel/configs/debug.config and see if anyone screams at us?

I guess the 3 net debug ones could go there indeed.

Do you want me to try?

Quoted 12 lines

Quoted 12 lines
Quoted 11 lines
Quoted 6 lines
For our CI validating MPTCP tests in a "debug" mode, we use "debug.config" without "x86_debug.config". On top of that, we also disable "SLUB_DEBUG_ON", because the impact on the perf is too important, especially with slow environments. We think it is not worth it for our case. You don't have the same hardware, but if you have perf issues, don't hesitate to do the same ;)
The mptcp tests take <60min to run with debug enabled, and just a single thread / VM. I think that's fine for now. But thanks for the heads up that SLUB_DEBUG_ON is problematic, for it may matter for forwarding or net tests.

Quoted 6 lines

Quoted 3 lines
The longest MPTCP selftest is currently stopped after 30 minutes due to the selftest timeout. I will see what we can do. That's not just because of SLUB_DEBUG_ON, that's normal to be very slow in such particular env.
I'll 2x the timeouts before reporting debug to patchwork, so don't worry about it, yet.

Thanks :)

Cheers,
Matt

Signature · 1 line

Sponsored by the NGI0 Core fund.

Jakub Kicinski

author maintainer

2024-02-08 02:11 UTC #6 lore raw

Quoted 12 lines · On Wed, 7 Feb 2024 16:59:31 +0100 Matthieu Baerts wrote:

Quoted 11 lines · Jakub Kicinski
Quoted 4 lines · On Wed, 7 Feb 2024 09:50:48 +0100 Matthieu Baerts wrote:
I was surprised not to see LOCKDEP there, but in fact, it is: it enables PROVE_LOCKING, which selects LOCKDEP and a few other DEBUG_xxx ones.
So maybe it is not needed to include the x86 one?
I'm confused. Now doing:
make O=built_test defconfig
make O=built_test debug.config

Quoted 8 lines

Quoted 2 lines
I don't get KASAN but I get LOCKDEP :S Bleh, maybe because of the options vng throws in?
Strange, I get both of them on my side.
Same if I include the two targets:
make O=built_test defconfig debug.config

:o Maybe somehow doing it in two separate commands makes a difference?

I tried now and all that's changing without the x86 config is:

-CONFIG_DEBUG_PAGEALLOC=y
-# CONFIG_DEBUG_PAGEALLOC_ENABLE_DEFAULT is not set +# CONFIG_DEBUG_PAGEALLOC is not set

-CONFIG_LOCK_STAT=y
+# CONFIG_LOCK_STAT is not set

I don't know if PAGEALLOC is useful but let's assume it's not important enough to add the extra config.

Quoted 6 lines

Quoted 2 lines
Just the three? What about KASAN, OBJECT debug, DEBUG_VM etc? As much as we can without going over the 3h limit in the tests :)
Sorry, I meant to say: we would use the existing "debug.config" + a new "net_debug.config" containing these 3 options (and maybe more that are specific to the net tree).

Turns out that if I try with vng DEBUG_VM and KASAN _are_ enabled :(

Quoted 9 lines

Quoted 5 lines
Quoted 3 lines
disable RETPOLINE (+
mitigations=off) not to slow down the tests in already slow envs, and disable a few components we don't need to accelerate the build and boot:
RETPOLINE we could kill, agreed
I still need to check vrg: is it easy to add kconfig you want to enable/disable? Do you need a new file? Should we maintain this file in the tree?

Sorry for going in circles. Seems like you were right that we don't need the x86 debug, so on the debug side all we need is the two networking options? Can we try to add those to debug.config ?

Quoted 6 lines

Quoted 5 lines
Quoted 3 lines
DRM, SOUND, etc.
https://github.com/multipath-tcp/mptcp-upstream-virtme-docker/blob/latest/entrypoint.sh#L284
Yes, vng also adds some stuff we don't need in this area :(

Quoted 8 lines

With the non-ng virtme, I disable those:
-d PCCARD -d MACINTOSH_DRIVERS -d SOUND -d USB_SUPPORT -d NEW_LEDS -d SURFACE_PLATFORMS -d DRM -d FB
I will check when I will switch to vng to see what else we can disable. But on your side, you probably have more CPU resources to compile the kernel, and that's fine to keep them :)

Yup!

Quoted 5 lines

Quoted 5 lines
Quoted 4 lines
It is also possible to add some kconfig in the selftests if preferred, e.g. in
./tools/testing/selftests/net/debug.config

Quoted 7 lines

Quoted 3 lines
Would be great if everyone didn't have to go thru this exercise. How about we start sending patches to kernel/configs/debug.config and see if anyone screams at us?
I guess the 3 net debug ones could go there indeed.
Do you want me to try?

Yes, please! (Unless you don't like being screamed at, then I can :))

Simon Horman

reviewer

2024-02-09 10:25 UTC #7 lore raw

In reply to Jakub Kicinski

Quoted 5 lines · Jakub Kicinski

Hi folks!
We added a wiki page to the nipa repo on how to run the tests locally:
https://github.com/linux-netdev/nipa/wiki/How-to-test-netdev-selftests

Thanks for this.

I am wondering if the URL should be / is now:

https://github.com/linux-netdev/nipa/wiki/How-to-run-netdev-selftests-CI-style

...

Jakub Kicinski

author maintainer

2024-02-09 15:30 UTC #8 lore raw

Quoted 5 lines · On Fri, 9 Feb 2024 10:25:10 +0000 Simon Horman wrote:

Quoted 4 lines · Jakub Kicinski
Hi folks!
We added a wiki page to the nipa repo on how to run the tests locally:

Quoted 7 lines

Quoted 1 line
https://github.com/linux-netdev/nipa/wiki/How-to-test-netdev-selftests
Thanks for this.
I am wondering if the URL should be / is now:
https://github.com/linux-netdev/nipa/wiki/How-to-run-netdev-selftests-CI-style

Yes, I didn't realize editing the title would change the URL. There's no permanent link either, it seems.
I added some links to the home page:
https://github.com/linux-netdev/nipa/wiki
And also the contest page now links directly to the repro instructions (top right corner). Is there any other place we need to update?

Simon Horman

reviewer

2024-02-16 17:01 UTC #9 lore raw

Quoted 12 lines · Jakub Kicinski

Quoted 11 lines · On Fri, 9 Feb 2024 10:25:10 +0000 Simon Horman wrote:
Quoted 5 lines · Jakub Kicinski
Hi folks!
We added a wiki page to the nipa repo on how to run the tests locally:
https://github.com/linux-netdev/nipa/wiki/How-to-test-netdev-selftests
Thanks for this.
I am wondering if the URL should be / is now:

Quoted 8 lines

Quoted 1 line
https://github.com/linux-netdev/nipa/wiki/How-to-run-netdev-selftests-CI-style
Yes, I didn't realize editing the title would change the URL. There's no permanent link either, it seems.
I added some links to the home page:
https://github.com/linux-netdev/nipa/wiki
And also the contest page now links directly to the repro instructions (top right corner). Is there any other place we need to update?

Thanks, I'm not aware of any other places that need updating.