Skip to contentSkip to conversation

OOMs on s390x runners

BPF CI Ihor Solodrai 2 participants 4 messages Jan 20 – Feb 10 BPF

lore
Share & tools
lore

Ihor Solodrai

author reviewer
#0 lore raw

Hi Ilya,

The BPF selftests regularly fail on s390x runners with OOMs.

The s390x hosts that I've been maintaining have 16G of memory, hosting 2 runners each. VMs (qemu instances) running the tests currently get 5G of memory each.

I noticed that the s390x runners didn't have swap set up, so I added an ansible config to set up a swapfile. This seems to have helped with OOM failures.

https://github.com/libbpf/ci/commit/8767dc05ab84c88da198af3c651511e731ddbac7

I spot checked a couple of recent failures, and all of them happened on the "ebpf1-worker-*" runners that you've been maintaining IIRC. Here is a couple of examples:
- https://github.com/kernel-patches/bpf/actions/runs/21178791628/job/60916006746
- https://github.com/kernel-patches/bpf/actions/runs/21163448380/job/60863171062
- https://github.com/kernel-patches/bpf/actions/runs/21157785072/job/60846570064

Could you please re-provision the hosts with the swap setup? Let's see if that helps.

Thanks!

#1 lore raw
Quoted 21 lines · Ihor Solodrai

Hi Ilya,

The BPF selftests regularly fail on s390x runners with OOMs.

The s390x hosts that I've been maintaining have 16G of memory, hosting 2 runners each. VMs (qemu instances) running the tests currently get 5G of memory each.

I noticed that the s390x runners didn't have swap set up, so I added an ansible config to set up a swapfile. This seems to have helped with OOM failures.

https://github.com/libbpf/ci/commit/8767dc05ab84c88da198af3c651511e731ddbac7

I spot checked a couple of recent failures, and all of them happened on the "ebpf1-worker-*" runners that you've been maintaining IIRC. Here is a couple of examples:
- https://github.com/kernel-patches/bpf/actions/runs/21178791628/job/60916006746
- https://github.com/kernel-patches/bpf/actions/runs/21163448380/job/60863171062
- https://github.com/kernel-patches/bpf/actions/runs/21157785072/job/60846570064

Quoted 4 lines

Could you please re-provision the hosts with the swap setup? Let's see if that helps.

Thanks!

Hi Ihor,

Thanks for the scripts!

I reprovisioned all 5 builders, everything looks healthy.

I will check on them again tomorrow morning.

Best regards,

Ilya

author reviewer
#2 lore raw
Quoted 28 lines · Ilya Leoshkevich
Quoted 21 lines · Ihor Solodrai

Hi Ilya,

The BPF selftests regularly fail on s390x runners with OOMs.

The s390x hosts that I've been maintaining have 16G of memory, hosting 2 runners each. VMs (qemu instances) running the tests currently get 5G of memory each.

I noticed that the s390x runners didn't have swap set up, so I added an ansible config to set up a swapfile. This seems to have helped with OOM failures.

https://github.com/libbpf/ci/commit/8767dc05ab84c88da198af3c651511e731ddbac7

I spot checked a couple of recent failures, and all of them happened on the "ebpf1-worker-*" runners that you've been maintaining IIRC. Here is a couple of examples:
   - https://github.com/kernel-patches/bpf/actions/runs/21178791628/job/60916006746
   - https://github.com/kernel-patches/bpf/actions/runs/21163448380/job/60863171062
   - https://github.com/kernel-patches/bpf/actions/runs/21157785072/job/60846570064

Quoted 4 lines

Could you please re-provision the hosts with the swap setup? Let's see if that helps.

Thanks!

Quoted 8 lines

Hi Ihor,

Thanks for the scripts!

I reprovisioned all 5 builders, everything looks healthy.

I will check on them again tomorrow morning.

Hi Ilya.

We ended up disabling KASAN for s390x and that helped with OOMs.

I noticed a couple "no space left on device" errors on your runners today, could you please take a look? Example:

https://github.com/kernel-patches/bpf/actions/runs/21732338737/job/62690585484

Thanks!

Quoted 6 lines

Best regards,

Ilya

#3 lore raw
Quoted 37 lines · Ihor Solodrai
Quoted 26 lines · Ilya Leoshkevich
Quoted 21 lines · Ihor Solodrai

Hi Ilya,

The BPF selftests regularly fail on s390x runners with OOMs.

The s390x hosts that I've been maintaining have 16G of memory, hosting 2 runners each. VMs (qemu instances) running the tests currently get 5G of memory each.

I noticed that the s390x runners didn't have swap set up, so I added an ansible config to set up a swapfile. This seems to have helped with OOM failures.

https://github.com/libbpf/ci/commit/8767dc05ab84c88da198af3c651511e731ddbac7

I spot checked a couple of recent failures, and all of them happened on the "ebpf1-worker-*" runners that you've been maintaining IIRC. Here is a couple of examples:
   - https://github.com/kernel-patches/bpf/actions/runs/21178791628/job/60916006746
   - https://github.com/kernel-patches/bpf/actions/runs/21163448380/job/60863171062
   - https://github.com/kernel-patches/bpf/actions/runs/21157785072/job/60846570064

Quoted 4 lines

Could you please re-provision the hosts with the swap setup? Let's see if that helps.

Thanks!

Quoted 8 lines

Hi Ihor,

Thanks for the scripts!

I reprovisioned all 5 builders, everything looks healthy.

I will check on them again tomorrow morning.

Hi Ilya.

Quoted 14 lines

We ended up disabling KASAN for s390x and that helped with OOMs.

I noticed a couple "no space left on device" errors on your runners today, could you please take a look? Example:

https://github.com/kernel-patches/bpf/actions/runs/21732338737/job/62690585484

Thanks!

Quoted 5 lines

Best regards,

Ilya

Hi Ihor,

Thanks for letting me know.

I deleted the unused volumes: actions-runner-kernel-patches-worker-02 and actions-runner-kernel-patches-worker-03 from all builders.

Now there should be enough space on all of them.

Best regards,

Ilya