OOMs on s390x runners
Ihor Solodrai
reviewerHi Ilya,
The BPF selftests regularly fail on s390x runners with OOMs.
The s390x hosts that I've been maintaining have 16G of memory, hosting 2 runners each. VMs (qemu instances) running the tests currently get 5G of memory each.
I noticed that the s390x runners didn't have swap set up, so I added an ansible config to set up a swapfile. This seems to have helped with OOM failures.
https://github.com/libbpf/ci/commit/8767dc05ab84c88da198af3c651511e731ddbac7
I spot checked a couple of recent failures, and all of them happened on the "ebpf1-worker-*" runners that you've been maintaining IIRC. Here is a couple of examples:
- https://github.com/kernel-patches/bpf/actions/runs/21178791628/job/60916006746
- https://github.com/kernel-patches/bpf/actions/runs/21163448380/job/60863171062
- https://github.com/kernel-patches/bpf/actions/runs/21157785072/job/60846570064
Could you please re-provision the hosts with the swap setup? Let's see if that helps.
Thanks!
Quoted 21 lines · Ihor Solodrai
Hi Ilya,
The BPF selftests regularly fail on s390x runners with OOMs.
The s390x hosts that I've been maintaining have 16G of memory, hosting 2 runners each. VMs (qemu instances) running the tests currently get 5G of memory each.
I noticed that the s390x runners didn't have swap set up, so I added an ansible config to set up a swapfile. This seems to have helped with OOM failures.
https://github.com/libbpf/ci/commit/8767dc05ab84c88da198af3c651511e731ddbac7
I spot checked a couple of recent failures, and all of them happened on the "ebpf1-worker-*" runners that you've been maintaining IIRC. Here is a couple of examples:
- https://github.com/kernel-patches/bpf/actions/runs/21178791628/job/60916006746
- https://github.com/kernel-patches/bpf/actions/runs/21163448380/job/60863171062
- https://github.com/kernel-patches/bpf/actions/runs/21157785072/job/60846570064
Quoted 4 lines
Could you please re-provision the hosts with the swap setup? Let's see if that helps.
Thanks!
Hi Ihor,
Thanks for the scripts!
I reprovisioned all 5 builders, everything looks healthy.
I will check on them again tomorrow morning.
Best regards,
Ilya
Ihor Solodrai
reviewerQuoted 28 lines · Ilya Leoshkevich
Quoted 21 lines · Ihor Solodrai
Hi Ilya,
The BPF selftests regularly fail on s390x runners with OOMs.
The s390x hosts that I've been maintaining have 16G of memory, hosting 2 runners each. VMs (qemu instances) running the tests currently get 5G of memory each.
I noticed that the s390x runners didn't have swap set up, so I added an ansible config to set up a swapfile. This seems to have helped with OOM failures.
https://github.com/libbpf/ci/commit/8767dc05ab84c88da198af3c651511e731ddbac7
I spot checked a couple of recent failures, and all of them happened on the "ebpf1-worker-*" runners that you've been maintaining IIRC. Here is a couple of examples:
- https://github.com/kernel-patches/bpf/actions/runs/21178791628/job/60916006746
- https://github.com/kernel-patches/bpf/actions/runs/21163448380/job/60863171062
- https://github.com/kernel-patches/bpf/actions/runs/21157785072/job/60846570064Quoted 4 lines
Could you please re-provision the hosts with the swap setup? Let's see if that helps.
Thanks!
Quoted 8 lines
Hi Ihor,
Thanks for the scripts!
I reprovisioned all 5 builders, everything looks healthy.
I will check on them again tomorrow morning.
Hi Ilya.
We ended up disabling KASAN for s390x and that helped with OOMs.
I noticed a couple "no space left on device" errors on your runners today, could you please take a look? Example:
https://github.com/kernel-patches/bpf/actions/runs/21732338737/job/62690585484
Thanks!
Quoted 6 lines
Best regards,
Ilya
Quoted 37 lines · Ihor Solodrai
Quoted 26 lines · Ilya Leoshkevich
Quoted 21 lines · Ihor Solodrai
Hi Ilya,
The BPF selftests regularly fail on s390x runners with OOMs.
The s390x hosts that I've been maintaining have 16G of memory, hosting 2 runners each. VMs (qemu instances) running the tests currently get 5G of memory each.
I noticed that the s390x runners didn't have swap set up, so I added an ansible config to set up a swapfile. This seems to have helped with OOM failures.
https://github.com/libbpf/ci/commit/8767dc05ab84c88da198af3c651511e731ddbac7
I spot checked a couple of recent failures, and all of them happened on the "ebpf1-worker-*" runners that you've been maintaining IIRC. Here is a couple of examples:
- https://github.com/kernel-patches/bpf/actions/runs/21178791628/job/60916006746
- https://github.com/kernel-patches/bpf/actions/runs/21163448380/job/60863171062
- https://github.com/kernel-patches/bpf/actions/runs/21157785072/job/60846570064Quoted 4 lines
Could you please re-provision the hosts with the swap setup? Let's see if that helps.
Thanks!
Quoted 8 lines
Hi Ihor,
Thanks for the scripts!
I reprovisioned all 5 builders, everything looks healthy.
I will check on them again tomorrow morning.
Hi Ilya.
Quoted 14 lines
We ended up disabling KASAN for s390x and that helped with OOMs.
I noticed a couple "no space left on device" errors on your runners today, could you please take a look? Example:
https://github.com/kernel-patches/bpf/actions/runs/21732338737/job/62690585484
Thanks!
Quoted 5 lines
Best regards,
Ilya
Hi Ihor,
Thanks for letting me know.
I deleted the unused volumes: actions-runner-kernel-patches-worker-02 and actions-runner-kernel-patches-worker-03 from all builders.
Now there should be enough space on all of them.
Best regards,
Ilya