Skip to contentSkip to conversation

The no-kvm CI instances going away

TEST Jakub Kicinski 3 participants 12 messages Feb 6, 2024 – Feb 7, 2024

Subsystems (3)
Networking Network drivers Networking MPTCP
lore
Share & tools
lore

Jakub Kicinski

author maintainer
#0 lore raw

Hi,

because cloud computing is expensive I'm shutting down the instances which were running without KVM support. We're left with the KVM-enabled instances only (metal) - one normal and one with debug configs enabled.

maintainer
#1 lore raw

Hi Jakub,

Quoted 3 lines · Jakub Kicinski

because cloud computing is expensive I'm shutting down the instances which were running without KVM support. We're left with the KVM-enabled instances only (metal) - one normal and one with debug configs enabled.

Thank you for the notification!

It sounds like good news if the non-support of KVM was causing issues :)

I think we can then no longer ignore the two MPTCP tests that were unstable in the previous environment.

The results from the different tests running on the -dbg instances don't look good. Maybe some debug kconfig have a too big impact? [1]

For MPTCP, one test always hits the selftest timeout [2] when using a debug kconfig. I don't know what to do in this case: if we need to set a timeout value that is supported by debug environments, the value will be so high, it will no longer catch issues "early enough" in "normal" environments.
Or could it be possible to ignore or double the timeout value in this debug environment?

Also, what is the plan with this debug env? It looks like the results are not reported to patchwork for the moment. Maybe only "important" issues, like kernel warnings, could be reported? Failed tests could be reported as "Warning" instead of "Fail"?

[1]
https://lore.kernel.org/netdev/[email protected]/
[2]
https://netdev-3.bots.linux.dev/vmksft-mptcp-dbg/results/453502/1-mptcp-join-sh/stdout

Cheers,
Matt

Signature · 1 line
Sponsored by the NGI0 Core fund.
author maintainer
#2 lore raw
Quoted 9 lines · On Tue, 6 Feb 2024 12:16:43 +0100 Matthieu Baerts wrote:

Hi Jakub,

Quoted 3 lines · Jakub Kicinski

because cloud computing is expensive I'm shutting down the instances which were running without KVM support. We're left with the KVM-enabled instances only (metal) - one normal and one with debug configs enabled.

Thank you for the notification!

Quoted 7 lines

It sounds like good news if the non-support of KVM was causing issues :)

I think we can then no longer ignore the two MPTCP tests that were unstable in the previous environment.

The results from the different tests running on the -dbg instances don't look good. Maybe some debug kconfig have a too big impact? [1]

Sorry, I'm behind on the reading the list. FWIW if you want to reach me quickly make sure the To: doesn't include anyone else. That gets sorted to a higher prio folder :S

Quoted 8 lines

For MPTCP, one test always hits the selftest timeout [2] when using a debug kconfig. I don't know what to do in this case: if we need to set a timeout value that is supported by debug environments, the value will be so high, it will no longer catch issues "early enough" in "normal" environments.
Or could it be possible to ignore or double the timeout value in this debug environment?

Quoted 4 lines

Also, what is the plan with this debug env? It looks like the results are not reported to patchwork for the moment. Maybe only "important" issues, like kernel warnings, could be reported? Failed tests could be reported as "Warning" instead of "Fail"?

Unfortunately I'm really behind on my "real job". I don't have a clear plan. I think we should scale the timeout by 2x or so, but I haven't looked how to do that.

I wish the selftest subsystem had some basic guidance.

maintainer
#3 lore raw

Hi Jakub,

Quoted 15 lines · Jakub Kicinski
Quoted 14 lines · On Tue, 6 Feb 2024 12:16:43 +0100 Matthieu Baerts wrote:

Hi Jakub,

Quoted 3 lines · Jakub Kicinski

because cloud computing is expensive I'm shutting down the instances which were running without KVM support. We're left with the KVM-enabled instances only (metal) - one normal and one with debug configs enabled.

Thank you for the notification!

It sounds like good news if the non-support of KVM was causing issues :)

I think we can then no longer ignore the two MPTCP tests that were unstable in the previous environment.

Quoted 6 lines
Quoted 2 lines

The results from the different tests running on the -dbg instances don't look good. Maybe some debug kconfig have a too big impact? [1]

Sorry, I'm behind on the reading the list. FWIW if you want to reach me quickly make sure the To: doesn't include anyone else. That gets sorted to a higher prio folder :S

Sorry, there was no urgency, I only wanted to add a link to the previous discussion for those who wanted more details about that.

Thank you for the note!

Quoted 8 lines
Quoted 8 lines

For MPTCP, one test always hits the selftest timeout [2] when using a debug kconfig. I don't know what to do in this case: if we need to set a timeout value that is supported by debug environments, the value will be so high, it will no longer catch issues "early enough" in "normal" environments.
Or could it be possible to ignore or double the timeout value in this debug environment?

Quoted 8 lines
Quoted 4 lines

Also, what is the plan with this debug env? It looks like the results are not reported to patchwork for the moment. Maybe only "important" issues, like kernel warnings, could be reported? Failed tests could be reported as "Warning" instead of "Fail"?

Unfortunately I'm really behind on my "real job". I don't have a clear plan. I think we should scale the timeout by 2x or so, but I haven't looked how to do that.

No hurry, I understand.

It is not clear to me how the patches you add on top of the ones from patchwork are managed. Then, I don't know if it can help, but on the debug instance, this command could be launched before starting the tests to double the timeout values in all the "net" selftests:

  $ find tools/testing/selftests/net -name settings -print0 | xargs -0 \
       awk -i inplace -F '=' \
           '{if ($1 == "timeout") { print $1 "=" $2*2 } else { print }}'
Quoted 1 line

I wish the selftest subsystem had some basic guidance.

Me too :)

Cheers,
Matt

Signature · 1 line
Sponsored by the NGI0 Core fund.
author maintainer
#4 lore raw
Quoted 6 lines · On Wed, 7 Feb 2024 10:44:14 +0100 Matthieu Baerts wrote:
Quoted 3 lines

Unfortunately I'm really behind on my "real job". I don't have a clear plan. I think we should scale the timeout by 2x or so, but I haven't looked how to do that.

No hurry, I understand.

Quoted 8 lines

It is not clear to me how the patches you add on top of the ones from patchwork are managed. Then, I don't know if it can help, but on the debug instance, this command could be launched before starting the tests to double the timeout values in all the "net" selftests:

  $ find tools/testing/selftests/net -name settings -print0 | xargs -0 \
       awk -i inplace -F '=' \
           '{if ($1 == "timeout") { print $1 "=" $2*2 } else { print }}'

I'd rather not modify the tree. Poking around - this seems to work:

export kselftest_override_timeout=1

Now it's just a matter of finding 15min to code it up :)

maintainer
#5 lore raw
Quoted 12 lines · Jakub Kicinski
Quoted 11 lines · On Wed, 7 Feb 2024 10:44:14 +0100 Matthieu Baerts wrote:
Quoted 3 lines

Unfortunately I'm really behind on my "real job". I don't have a clear plan. I think we should scale the timeout by 2x or so, but I haven't looked how to do that.

No hurry, I understand.

It is not clear to me how the patches you add on top of the ones from patchwork are managed. Then, I don't know if it can help, but on the debug instance, this command could be launched before starting the tests to double the timeout values in all the "net" selftests:

Quoted 7 lines
Quoted 3 lines
  $ find tools/testing/selftests/net -name settings -print0 | xargs -0 \
       awk -i inplace -F '=' \
           '{if ($1 == "timeout") { print $1 "=" $2*2 } else { print }}'

I'd rather not modify the tree. Poking around - this seems to work:

export kselftest_override_timeout=1

Even better :)

  f=tools/testing/selftests/net/settings
  kselftest_override_timeout=$(awk -F = '/^timeout=/ {print $2*2}' $f)
Quoted 1 line

Now it's just a matter of finding 15min to code it up :)

I'm not sure if I can help here :)

Cheers,
Matt

Signature · 1 line
Sponsored by the NGI0 Core fund.
author maintainer
#6 lore raw
Quoted 11 lines · On Wed, 7 Feb 2024 15:37:22 +0100 Matthieu Baerts wrote:
Quoted 3 lines

I'd rather not modify the tree. Poking around - this seems to work:

export kselftest_override_timeout=1

Even better :)

  f=tools/testing/selftests/net/settings
  kselftest_override_timeout=$(awk -F = '/^timeout=/ {print $2*2}' $f)
Quoted 1 line

Now it's just a matter of finding 15min to code it up :)

I'm not sure if I can help here :)

If you're willing to touch my nasty Python - that'd be very welcome! :)

Right now I put this in the configs:

[vm]
exports=KSFT_MACHINE_SLOW=yes

We can leave the support for adding exports in place, but we should add a new config option like

[vm]
slowdown=2

(exact name up for debate), if it's present generate the KSFT_MACHINE_SLOW=yes export automatically, without the explicit entry in the config. And add the export for the timeout matching the logic you propose (but in Python).

FWIW here's the configs we currently use:

$ cat remote.config
[remote]
branches=https://netdev.bots.linux.dev/static/nipa/branches.json
filters=https://netdev.bots.linux.dev/contest/filters.json
[local]
results_path=results
json_path=jsons
[env]
paths=/home/virtme/virtme-ng/
[vm]
paths=/home/virtme/tools/fs/bin:/home/virtme/tools/fs/sbin:/home/virtme/tools/fs/usr/bin:/home/virtme/tools/fs/usr/sbin
ld_paths=/lib64/:/home/virtme/tools/fs/usr/lib/:/home/virtme/tools/fs/lib64/:/home/virtme/tools/fs/usr/lib64/
init_prompt=bash-5.2#
default_timeout=200
boot_timeout=45
$ cat ksft-mptcp-dbg.config
[executor]
name=vmksft-mptcp-dbg
[local]
tree_path=/home/virtme/testing-12/
base_path=/home/virtme/outputs-12/
[www]
url=https://netdev-3.bots.linux.dev/vmksft-mptcp-dbg
[vm]
exports=KSFT_MACHINE_SLOW=yes
configs=kernel/configs/debug.config,kernel/configs/x86_debug.config
default_timeout=300
cpus=4
[ksft]
target=net/mptcp
[cfg]
thread_cnt=1
maintainer
#7 lore raw
Quoted 7 lines · Jakub Kicinski
Quoted 6 lines · On Wed, 7 Feb 2024 15:37:22 +0100 Matthieu Baerts wrote:
Quoted 3 lines

I'd rather not modify the tree. Poking around - this seems to work:

export kselftest_override_timeout=1

Even better :)

Quoted 7 lines
Quoted 5 lines
  f=tools/testing/selftests/net/settings
  kselftest_override_timeout=$(awk -F = '/^timeout=/ {print $2*2}' $f)
Quoted 1 line

Now it's just a matter of finding 15min to code it up :)

I'm not sure if I can help here :)

If you're willing to touch my nasty Python - that'd be very welcome! :)

Sure, I can have a look (but not today) :)

Thank you for the guide! Like with my PR to support subtests, I might only test the code I modify not to have to set up everything, I hope it is fine :)

Cheers,
Matt

Signature · 1 line
Sponsored by the NGI0 Core fund.
#8 lore raw
In reply to Jakub Kicinski
Quoted 3 lines · Jakub Kicinski

because cloud computing is expensive I'm shutting down the instances which were running without KVM support. We're left with the KVM-enabled instances only (metal) - one normal and one with debug configs enabled.

who is covering the cost of the cloud VMs? Have you considered cheaper alternatives to AWS?

author maintainer
#9 lore raw
Quoted 6 lines · On Wed, 7 Feb 2024 10:45:26 -0700 David Ahern wrote:
Quoted 3 lines · Jakub Kicinski

because cloud computing is expensive I'm shutting down the instances which were running without KVM support. We're left with the KVM-enabled instances only (metal) - one normal and one with debug configs enabled.

who is covering the cost of the cloud VMs?

Meta

Quoted 1 line

Have you considered cheaper alternatives to AWS?

If I'm completely honest it's more a time thing than cost thing. I have set a budget for the project in internal tooling to 3x what I expected just the build bot to consume, so it can fit one large instance without me having to jump thru any hoops. I will slowly jump thru hoops to get more as time allows, but I figured the VM instance was a mistake in the first place, so I can as well just kill it off already. The -dbg runners are also slow. Or do you see benefit to running without KVM? Another potential extension is running on ARM.

And yes, it was much cheaper when the builder run in Digital Ocean.

But why do you ask? :) Just to offer cheaper alternatives or do you happen to have the ability to get a check written to support the effort? :)

#10 lore raw
Quoted 22 lines · Jakub Kicinski
Quoted 6 lines · On Wed, 7 Feb 2024 10:45:26 -0700 David Ahern wrote:
Quoted 3 lines · Jakub Kicinski

because cloud computing is expensive I'm shutting down the instances which were running without KVM support. We're left with the KVM-enabled instances only (metal) - one normal and one with debug configs enabled.

who is covering the cost of the cloud VMs?

Meta

Quoted 1 line

Have you considered cheaper alternatives to AWS?

If I'm completely honest it's more a time thing than cost thing. I have set a budget for the project in internal tooling to 3x what I expected just the build bot to consume, so it can fit one large instance without me having to jump thru any hoops. I will slowly jump thru hoops to get more as time allows, but I figured the VM instance was a mistake in the first place, so I can as well just kill it off already. The -dbg runners are also slow. Or do you see benefit to running without KVM? Another potential extension is running on ARM.

Quoted 5 lines

And yes, it was much cheaper when the builder run in Digital Ocean.

But why do you ask? :) Just to offer cheaper alternatives or do you happen to have the ability to get a check written to support the effort? :)

I have no such ability :-) I cover the costs myself when I use VMs on DigitalOcean and Vultr for Linux development and testing.

Kernel builds and selftests just need raw compute power, none of the fancy enterprise features that AWS provides (and bills accordingly).

The first question about who is covering the cost is to avoid assumptions and acknowledge the service (and costs) provided to the community. Having the selftests tied to patchsets is really helpful to proactively identify potential regressions.

For the second question I was just curious as to whether you had tried the cheaper options (DO, Vultr, Linode, ...) and if they worked ok for you. ie., why AWS. I like the range of OS versions that are accessible within minutes.

author maintainer
#11 lore raw
Quoted 10 lines · On Wed, 7 Feb 2024 12:45:00 -0700 David Ahern wrote:

I have no such ability :-) I cover the costs myself when I use VMs on DigitalOcean and Vultr for Linux development and testing.

Kernel builds and selftests just need raw compute power, none of the fancy enterprise features that AWS provides (and bills accordingly).

The first question about who is covering the cost is to avoid assumptions and acknowledge the service (and costs) provided to the community. Having the selftests tied to patchsets is really helpful to proactively identify potential regressions.

Thanks! Meta is pretty great at covering various community costs. I try to keep the asks reasonable, but I haven't been told "no", yet. The reason why it'd be great to have more supporters is that (at a cost of slightly more paperwork) we could externalize the whole shebang, and give out SSH keys more freely :(

I also heard that some clouds or LF have free cloud credits for open source CI. But IDK how I could connect those to my corporate account..

Anyway. In my ideal world we'd write a check for LF and get a bunch of VMs in DO or alike. In the present reality we have AWS which only I can access 🥴️

Good thing is the system is distributed. Hopefully over time more people will join in by reporting the results (like Mojatatu does with TCD).