CX7 timeouts on reconfig w/ page pool failure injection
Subsystems (2)
Jakub Kicinski
maintainerHi!
The CX7 testing in NIPA is looking pretty good, one major source of noise, tho, is that the pp-alloc test sometimes locks rtnl for a minute.
Here's a failure in that test itself:
# # Exception| subprocess.TimeoutExpired: Command '['ethtool', '-G', 'ens25f1np1', 'rx', '2048']' timed out after 20 seconds # # Exception|
# not ok 1 pp_alloc_fail.test_pp_alloc
Then the next test that runs also times out during setup:
# subprocess.TimeoutExpired: Command '['ip', '--json', '-d', 'link', 'show', 'dev', 'ens25f1np1']' timed out after 20 seconds not ok 1 selftests: drivers/net/hw: rss_api.py # exit=1
https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/763842/test-outputs/37-rss-api-py/stdout
pp_alloc test can fail, but spilling into the next test is a problem.
At the retry things clear up enough to return to happiness, tho:
I haven't try to dig into where it is blocked but does this ring any bells?
Quoted 21 lines · Jakub Kicinski
Hi!
The CX7 testing in NIPA is looking pretty good, one major source of noise, tho, is that the pp-alloc test sometimes locks rtnl for a minute.
Here's a failure in that test itself:
# # Exception| subprocess.TimeoutExpired: Command '['ethtool', '-G', 'ens25f1np1', 'rx', '2048']' timed out after 20 seconds # # Exception|
# not ok 1 pp_alloc_fail.test_pp_allocThen the next test that runs also times out during setup:
# subprocess.TimeoutExpired: Command '['ip', '--json', '-d', 'link', 'show', 'dev', 'ens25f1np1']' timed out after 20 seconds not ok 1 selftests:
drivers/net/hw:rss_api.py# exit=1https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/763842/test-outputs/37-rss-api-py/stdout
Quoted 8 lines
pp_alloctest can fail, but spilling into the next test is a problem.At the retry things clear up enough to return to happiness, tho:
I haven't try to dig into where it is blocked but does this ring any bells?
Not to me. We can try an internal reproduction.
Randomly clicking around the flakes page, I found another instance of pp_alloc_fail.py running and getting crashes in mlx5 during page deallocation:
https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/762402/sol-machine-2
Is this also something new?
Thanks,
Dragos
Jakub Kicinski
maintainerQuoted 6 lines · On Tue, 4 Aug 2026 19:16:19 +0200 Dragos Tatulea wrote:
Randomly clicking around the flakes page, I found another instance of
pp_alloc_fail.pyrunning and getting crashes in mlx5 during page deallocation:
https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/762402/sol-machine-2Is this also something new?
Ah, that was going to be my next ask. (there were some fake "crashes" reported in NIPA because the system was retrying and running out of time, that's hopefully fixed as of this morning so the signal will be cleaner)
The crash has been hitting since the beginning of July (if it gets line wrapped into oblivion the tl;dr is that first hit was net-next-hw-2026-07-03--00-00):
branch | test | crashes
-------------------------------+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------
net-next-hw-2026-08-04--16-00 | pp-alloc-fail-py | ["", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:mlx5e_free_rx_mpwqe:__napi_poll"] net-next-hw-2026-08-03--08-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"] net-next-hw-2026-08-02--16-00 | pp-alloc-fail-py | ["", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"] net-next-hw-2026-08-02--08-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"] net-next-hw-2026-07-31--00-00 | pp-alloc-fail-py | [""] net-next-hw-2026-07-28--08-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"] net-next-hw-2026-07-28--00-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"] net-next-hw-2026-07-26--16-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"] net-next-hw-2026-07-25--16-00 | pp-alloc-fail-py | ["", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"] net-next-hw-2026-07-25--08-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"] net-next-hw-2026-07-25--00-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_dealloc_rx_mpwqe:mlx5e_free_rx_descs:mlx5e_close_rq:mlx5e_close_queues"] net-next-hw-2026-07-23--00-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"] net-next-hw-2026-07-19--16-03 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"] net-next-hw-2026-07-19--08-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_dealloc_rx_mpwqe:mlx5e_free_rx_descs:mlx5e_close_rq:mlx5e_close_queues", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"] net-next-hw-2026-07-18--00-00 | pp-alloc-fail-py | [""] net-next-hw-2026-07-17--16-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"] net-next-hw-2026-07-17--00-00 | irq-py | ["mlx5e_free_rx_mpwqe:mlx5e_dealloc_rx_mpwqe:mlx5e_free_rx_descs:mlx5e_close_rq:mlx5e_close_queues"] net-next-hw-2026-07-15--08-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes"] net-next-hw-2026-07-13--16-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"] net-next-hw-2026-07-13--08-00 | pp-alloc-fail-py | ["", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"] net-next-hw-2026-07-13--00-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"] net-next-hw-2026-07-12--16-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"] net-next-hw-2026-07-11--00-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"] net-next-hw-2026-07-10--08-00 | pp-alloc-fail-py | ["", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"] net-next-hw-2026-07-10--00-00 | pp-alloc-fail-py | ["", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"] net-next-hw-2026-07-09--08-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:mlx5e_free_rx_mpwqe"] net-next-hw-2026-07-09--00-00 | pp-alloc-fail-py | ["handle_softirqs:run_ksoftirqd:mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"] net-next-hw-2026-07-08--08-00 | pp-alloc-fail-py | [""] net-next-hw-2026-07-07--16-00 | ethtool-rmon-sh | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"] net-next-hw-2026-07-07--08-00 | pp-alloc-fail-py | ["", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"] net-next-hw-2026-07-07--00-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"] net-next-hw-2026-07-06--16-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action", "ret_from_fork_asm"] net-next-hw-2026-07-06--00-00 | irq-py | ["mlx5e_free_rx_mpwqe:mlx5e_dealloc_rx_mpwqe:mlx5e_free_rx_descs:mlx5e_close_rq:mlx5e_close_queues"] net-next-hw-2026-07-05--16-00 | pp-alloc-fail-py | ["", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"] net-next-hw-2026-07-03--00-00 | pp-alloc-fail-py | ["", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]
(for my own record)
SELECT
r.branch,
x.test,
x.crashes
FROM results AS r
CROSS JOIN LATERAL jsonb_to_recordset(
r.json_full::jsonb -> 'results' ) AS x(test text, crashes jsonb)
WHERE r.remote = 'netdev-CX7-dbg'
AND x.crashes IS NOT NULL
AND x.crashes <> '["kernel crash during test"]'::jsonb
ORDER BY r.branch_date DESC;
Quoted 14 lines · Jakub Kicinski
Hi!
The CX7 testing in NIPA is looking pretty good, one major source of noise, tho, is that the pp-alloc test sometimes locks rtnl for a minute.
Here's a failure in that test itself:
# # Exception| subprocess.TimeoutExpired: Command '['ethtool', '-G', 'ens25f1np1', 'rx', '2048']' timed out after 20 seconds # # Exception|
# not ok 1 pp_alloc_fail.test_pp_alloc
Quoted 9 lines
Then the next test that runs also times out during setup:
# subprocess.TimeoutExpired: Command '['ip', '--json', '-d', 'link', 'show', 'dev', 'ens25f1np1']' timed out after 20 seconds not ok 1 selftests:
drivers/net/hw:rss_api.py# exit=1https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/763842/test-outputs/37-rss-api-py/stdout
pp_alloctest can fail, but spilling into the next test is a problem.
Seems like the bigger culprit is the earlier xdp test, which leaves MTU at 9000:
# # Defer Exception| net.lib.py.utils.CmdExitFailure: Command failed # # Defer Exception| CMD: ip link set dev ens25f1np1 mtu 1500 xdpdrv off # # Defer Exception| EXIT: -15 # # Defer Exception| # not ok 17 xdp.test_xdp_native_adjst_head_grow_data.ipv4 # # Totals: pass:14 fail:3 xfail:0 xpass:0 skip:0 error:0 not ok 1 selftests: drivers/net: xdp.py # TIMEOUT 360 seconds
I'll try to reproduce this internally and investigate what happened.
Also, I noticed that the pp-alloc test isn't properly restoring the rx ring size due to the timeout. In the subsequent retry, it's attempting to double it again, this time from 2K to 4K:
# # Exception| subprocess.TimeoutExpired: Command '['ethtool', '-G', 'ens25f1np1', 'rx', '4096']' timed out after 20 seconds
# # Exception|
# not ok 1 pp_alloc_fail.test_pp_alloc
The high MTU and ring size along with the heavy debug kernel likely make the pp-alloc test significantly slower. I'll also prepare a patch to fix this pp-alloc cleanup issue.
Thanks,
Nimrod
Jakub Kicinski
maintainerQuoted 36 lines · On Tue, 11 Aug 2026 10:20:52 +0300 Nimrod Oren wrote:
Quoted 14 lines · Jakub Kicinski
Hi!
The CX7 testing in NIPA is looking pretty good, one major source of noise, tho, is that the pp-alloc test sometimes locks rtnl for a minute.
Here's a failure in that test itself:
# # Exception| subprocess.TimeoutExpired: Command '['ethtool', '-G', 'ens25f1np1', 'rx', '2048']' timed out after 20 seconds # # Exception|
# not ok 1 pp_alloc_fail.test_pp_allocQuoted 9 lines
Then the next test that runs also times out during setup:
# subprocess.TimeoutExpired: Command '['ip', '--json', '-d', 'link', 'show', 'dev', 'ens25f1np1']' timed out after 20 seconds not ok 1 selftests:
drivers/net/hw:rss_api.py# exit=1https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/763842/test-outputs/37-rss-api-py/stdout
pp_alloctest can fail, but spilling into the next test is a problem.Seems like the bigger culprit is the earlier xdp test, which leaves MTU at 9000:
# # Defer Exception| net.lib.py.utils.CmdExitFailure: Command failed # # Defer Exception| CMD: ip link set dev ens25f1np1 mtu 1500 xdpdrv off # # Defer Exception| EXIT: -15 # # Defer Exception| # not ok 17 xdp.test_xdp_native_adjst_head_grow_data.ipv4 # # Totals: pass:14 fail:3 xfail:0 xpass:0 skip:0 error:0 not ok 1 selftests: drivers/net: xdp.py # TIMEOUT 360 seconds
Quoted 1 line
Ah, well spotted. That's the test timeout, I'll bump the timeout from the current 6min to 10min. Once it passes we'll know the true time needed, and we shall set the timeout to 150% of the expected case..
Quoted 9 lines
I'll try to reproduce this internally and investigate what happened.
Also, I noticed that the pp-alloc test isn't properly restoring the rx ring size due to the timeout. In the subsequent retry, it's attempting to double it again, this time from 2K to 4K:
# # Exception| subprocess.TimeoutExpired: Command '['ethtool', '-G', 'ens25f1np1', 'rx', '4096']' timed out after 20 seconds
# # Exception|
Quoted 7 lines
# not ok 1 pp_alloc_fail.test_pp_allocThe high MTU and ring size along with the heavy debug kernel likely make the pp-alloc test significantly slower. I'll also prepare a patch to fix this pp-alloc cleanup issue.
To be clear - if the commands time out - how can we fix the cleanup? Or do you mean some other bug?
Quoted 37 lines · Jakub Kicinski
Quoted 36 lines · On Tue, 11 Aug 2026 10:20:52 +0300 Nimrod Oren wrote:
Quoted 14 lines · Jakub Kicinski
Hi!
The CX7 testing in NIPA is looking pretty good, one major source of noise, tho, is that the pp-alloc test sometimes locks rtnl for a minute.
Here's a failure in that test itself:
# # Exception| subprocess.TimeoutExpired: Command '['ethtool', '-G', 'ens25f1np1', 'rx', '2048']' timed out after 20 seconds # # Exception|
# not ok 1 pp_alloc_fail.test_pp_allocQuoted 9 lines
Then the next test that runs also times out during setup:
# subprocess.TimeoutExpired: Command '['ip', '--json', '-d', 'link', 'show', 'dev', 'ens25f1np1']' timed out after 20 seconds not ok 1 selftests:
drivers/net/hw:rss_api.py# exit=1https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/763842/test-outputs/37-rss-api-py/stdout
pp_alloctest can fail, but spilling into the next test is a problem.Seems like the bigger culprit is the earlier xdp test, which leaves MTU at 9000:
# # Defer Exception| net.lib.py.utils.CmdExitFailure: Command failed # # Defer Exception| CMD: ip link set dev ens25f1np1 mtu 1500 xdpdrv off # # Defer Exception| EXIT: -15 # # Defer Exception| # not ok 17 xdp.test_xdp_native_adjst_head_grow_data.ipv4 # # Totals: pass:14 fail:3 xfail:0 xpass:0 skip:0 error:0 not ok 1 selftests: drivers/net: xdp.py # TIMEOUT 360 seconds
Quoted 6 lines
Quoted 1 line
Ah, well spotted. That's the test timeout, I'll bump the timeout from the current 6min to 10min. Once it passes we'll know the true time needed, and we shall set the timeout to 150% of the expected case..
Sounds good, thanks.
Quoted 11 lines
Quoted 11 lines
I'll try to reproduce this internally and investigate what happened.
Also, I noticed that the pp-alloc test isn't properly restoring the rx ring size due to the timeout. In the subsequent retry, it's attempting to double it again, this time from 2K to 4K:
# # Exception| subprocess.TimeoutExpired: Command '['ethtool', '-G', 'ens25f1np1', 'rx', '4096']' timed out after 20 seconds
# # Exception|# not ok 1 pp_alloc_fail.test_pp_alloc
Quoted 8 lines
Quoted 5 lines
The high MTU and ring size along with the heavy debug kernel likely make the pp-alloc test significantly slower. I'll also prepare a patch to fix this pp-alloc cleanup issue.
To be clear - if the commands time out - how can we fix the cleanup? Or do you mean some other bug?
In the pp-alloc test, the ring size command timed out, but the change was still applied in the end.
Currently that test only attempts to restore the original ring size if the set command returns successfully. This case makes me think that it should attempt the restoration regardless...
Jakub Kicinski
maintainerQuoted 16 lines · On Tue, 11 Aug 2026 19:57:31 +0300 Nimrod Oren wrote:
Quoted 2 lines · Jakub Kicinski
Quoted 1 line · On Tue, 11 Aug 2026 10:20:52 +0300 Nimrod Oren wrote:
On 04/08/2026 19:45, Jakub Kicinski wrote:
[...]
Quoted 12 lines
Quoted 12 lines
Seems like the bigger culprit is the earlier xdp test, which leaves MTU at 9000:
# # Defer Exception| net.lib.py.utils.CmdExitFailure: Command failed # # Defer Exception| CMD: ip link set dev ens25f1np1 mtu 1500 xdpdrv off # # Defer Exception| EXIT: -15 # # Defer Exception| # not ok 17 xdp.test_xdp_native_adjst_head_grow_data.ipv4 # # Totals: pass:14 fail:3 xfail:0 xpass:0 skip:0 error:0 not ok 1 selftests: drivers/net: xdp.py # TIMEOUT 360 seconds
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Ah, well spotted. That's the test timeout, I'll bump the timeout from the current 6min to 10min. Once it passes we'll know the true time needed, and we shall set the timeout to 150% of the expected case..
Sounds good, thanks.
10min wasn't enough, bumping to 15min :(
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
I'll try to reproduce this internally and investigate what happened.
Also, I noticed that the pp-alloc test isn't properly restoring the rx ring size due to the timeout. In the subsequent retry, it's attempting to double it again, this time from 2K to 4K:
# # Exception| subprocess.TimeoutExpired: Command '['ethtool', '-G', 'ens25f1np1', 'rx', '4096']' timed out after 20 seconds
# # Exception|# not ok 1 pp_alloc_fail.test_pp_allocQuoted 8 lines
Quoted 5 lines
The high MTU and ring size along with the heavy debug kernel likely make the pp-alloc test significantly slower. I'll also prepare a patch to fix this pp-alloc cleanup issue.
To be clear - if the commands time out - how can we fix the cleanup? Or do you mean some other bug?
Quoted 6 lines
In the pp-alloc test, the ring size command timed out, but the change was still applied in the end.
Currently that test only attempts to restore the original ring size if the set command returns successfully. This case makes me think that it should attempt the restoration regardless...
Would the defer() not time out as well if the command to apply the config timed out? Any idea what the main time consumer is for the reconfig? It can't be anything device related since the device isn't really impacted by kernel config options...
Quoted 19 lines · Jakub Kicinski
Quoted 18 lines · On Tue, 11 Aug 2026 19:57:31 +0300 Nimrod Oren wrote:
Quoted 2 lines · Jakub Kicinski
Quoted 1 line · On Tue, 11 Aug 2026 10:20:52 +0300 Nimrod Oren wrote:
On 04/08/2026 19:45, Jakub Kicinski wrote:
[...]
Quoted 14 lines
Quoted 13 lines
Seems like the bigger culprit is the earlier xdp test, which leaves MTU at 9000:
# # Defer Exception| net.lib.py.utils.CmdExitFailure: Command failed # # Defer Exception| CMD: ip link set dev ens25f1np1 mtu 1500 xdpdrv off # # Defer Exception| EXIT: -15 # # Defer Exception| # not ok 17 xdp.test_xdp_native_adjst_head_grow_data.ipv4 # # Totals: pass:14 fail:3 xfail:0 xpass:0 skip:0 error:0 not ok 1 selftests: drivers/net: xdp.py # TIMEOUT 360 seconds
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Ah, well spotted. That's the test timeout, I'll bump the timeout from the current 6min to 10min. Once it passes we'll know the true time needed, and we shall set the timeout to 150% of the expected case..
Sounds good, thanks.
10min wasn't enough, bumping to 15min :(
In these runs xdp-py took ~9.5 minutes. Both pp-alloc and rss-api tests passed:
https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/774403/test-outputs/
https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/774889/test-outputs/
https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/775361/test-outputs/
Quoted 23 lines
Quoted 23 lines
Quoted 11 lines
Quoted 11 lines
I'll try to reproduce this internally and investigate what happened.
Also, I noticed that the pp-alloc test isn't properly restoring the rx ring size due to the timeout. In the subsequent retry, it's attempting to double it again, this time from 2K to 4K:
# # Exception| subprocess.TimeoutExpired: Command '['ethtool', '-G', 'ens25f1np1', 'rx', '4096']' timed out after 20 seconds
# # Exception|# not ok 1 pp_alloc_fail.test_pp_allocQuoted 8 lines
Quoted 5 lines
The high MTU and ring size along with the heavy debug kernel likely make the pp-alloc test significantly slower. I'll also prepare a patch to fix this pp-alloc cleanup issue.
To be clear - if the commands time out - how can we fix the cleanup? Or do you mean some other bug?
In the pp-alloc test, the ring size command timed out, but the change was still applied in the end.
Quoted 8 lines
Quoted 3 lines
Currently that test only attempts to restore the original ring size if the set command returns successfully. This case makes me think that it should attempt the restoration regardless...
Would the
defer()not time out as well if the command to apply the config timed out? Any idea what the main time consumer is for the reconfig? It can't be anything device related since the device isn't really impacted by kernel config options...
It looks like most of the time is being spent in SLUB debugging. In the pp-alloc test, reconfiguration runs while background traffic is active. The SLUB overhead disappears a short while after the test traffic stops, so a deferred restore command is more likely to succeed.
Jakub Kicinski
maintainerQuoted 8 lines · On Wed, 12 Aug 2026 19:31:09 +0300 Nimrod Oren wrote:
Quoted 1 line
10min wasn't enough, bumping to 15min :(
In these runs xdp-py took ~9.5 minutes. Both pp-alloc and rss-api tests passed:
https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/774403/test-outputs/
https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/774889/test-outputs/
https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/775361/test-outputs/
Ack, I'll send a patch to bump the timeout shortly.
That said, the page pool warnings were probabilistic to begin with. And they were hit in page_pool_unref_netmem(), I think _some_ bug must still be there?
IOW mlx5e_page_release_fragmented() is draining more references than the pp page use counter, that's not normal whether some command timeout out or not.
Quoted 12 lines · Jakub Kicinski
Quoted 8 lines · On Wed, 12 Aug 2026 19:31:09 +0300 Nimrod Oren wrote:
Quoted 1 line
10min wasn't enough, bumping to 15min :(
In these runs xdp-py took ~9.5 minutes. Both pp-alloc and rss-api tests passed:
https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/774403/test-outputs/
https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/774889/test-outputs/
https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/775361/test-outputs/Ack, I'll send a patch to bump the timeout shortly.
Quoted 7 lines
That said, the page pool warnings were probabilistic to begin with. And they were hit in
page_pool_unref_netmem(), I think _some_ bug must still be there?IOW
mlx5e_page_release_fragmented()is draining more references than the pp page use counter, that's not normal whether some command timeout out or not.
Yes, this shouldn't happen.
You mentioned tag net-next-hw-2026-07-03--00-00 as the first where the crash occurred. Do you know if it ran before without crashing? I can't find older data in nipa. And there's nothing that seems relevant around that timeframe. I'm thinking that this issue could be older.
Thanks,
Dragos
Jakub Kicinski
maintainerQuoted 9 lines · On Wed, 12 Aug 2026 21:48:31 +0200 Dragos Tatulea wrote:
Quoted 9 lines
Ack, I'll send a patch to bump the timeout shortly.
That said, the page pool warnings were probabilistic to begin with. And they were hit in
page_pool_unref_netmem(), I think _some_ bug must still be there?IOW
mlx5e_page_release_fragmented()is draining more references than the pp page use counter, that's not normal whether some command timeout out or not.
Quoted 6 lines
Yes, this shouldn't happen.
You mentioned tag net-next-hw-2026-07-03--00-00 as the first where the crash occurred. Do you know if it ran before without crashing? I can't find older data in nipa. And there's nothing that seems relevant around that timeframe. I'm thinking that this issue could be older.
The data should be there but I think there may be limits on how much the DB returns. Anyway, querying directly - looks like until ~june there was something missing in the kconfig so the test was skipping. But here is a June 1st run which passes:
so either it was running and not crashing for all of june... or our crash tracking was broken in June :)
Hi,
Quoted 22 lines · Jakub Kicinski
Quoted 9 lines · On Wed, 12 Aug 2026 21:48:31 +0200 Dragos Tatulea wrote:
Quoted 9 lines
Ack, I'll send a patch to bump the timeout shortly.
That said, the page pool warnings were probabilistic to begin with. And they were hit in
page_pool_unref_netmem(), I think _some_ bug must still be there?IOW
mlx5e_page_release_fragmented()is draining more references than the pp page use counter, that's not normal whether some command timeout out or not.Quoted 6 lines
Yes, this shouldn't happen.
You mentioned tag net-next-hw-2026-07-03--00-00 as the first where the crash occurred. Do you know if it ran before without crashing? I can't find older data in nipa. And there's nothing that seems relevant around that timeframe. I'm thinking that this issue could be older.
The data should be there but I think there may be limits on how much the DB returns. Anyway, querying directly - looks like until ~june there was something missing in the kconfig so the test was skipping. But here is a June 1st run which passes:
Quoted 4 lines
so either it was running and not crashing for all of june... or our crash tracking was broken in June :)
There were 2 corner cases that were missed by the last fix in the area regarding handling of XDP buffer geometry changes (fragments get deleted):
- napi_alloc_skb() error path. Probably this is the one one that triggered the crash, it occurs when injecting an error there.
- XDP drop path. Doesn't make sense but it is possible.
We have a fix in the pipeline for these.
Thanks,
Dragos
Jakub Kicinski
maintainerHi!
Sorry to re-heat an old thread but FWIW I enabled HW-GRO recently, to get the zero-copy tests to work, finally(?) And it increased the reconfig time by ~2x :(
Quoted 5 lines · Jakub Kicinski
Hi!
Sorry to re-heat an old thread but FWIW I enabled HW-GRO recently, to get the zero-copy tests to work, finally(?) And it increased the reconfig time by ~2x :(
Yes this is a known issue: each ZC ndo queue cycle triggers setup and teardown of all queues. We want to improve that eventually.
I think in this particular case it is due to ncdevmem which takes half of the queues during probe.
Thanks,
Dragos