google runner needs nested cases
Subsystems (2)
Jakub Kicinski
maintainerHi!
Another ask, please parse the TAP output of the tests and provide a nested array of results per-subcase, see:
https://github.com/linux-netdev/nipa/wiki/Netdev-CI-system/#results
right now you only seem to report top level pass/fail but most tests have at least one sub-case the driver doesn't support.
Thanks, Jakub. We plan to merge the changes to support nested arrays for sub-test results by next week.
Quoted 10 lines · Jakub Kicinski
Hi!
Another ask, please parse the TAP output of the tests and provide a nested array of results per-subcase, see:
https://github.com/linux-netdev/nipa/wiki/Netdev-CI-system/#results
right now you only seem to report top level pass/fail but most tests have at least one sub-case the driver doesn't support.
Jakub Kicinski
maintainerQuoted 2 lines · On Thu, 23 Jul 2026 10:14:20 -0700 Sheena Mohan wrote:
Thanks, Jakub. We plan to merge the changes to support nested arrays for sub-test results by next week.
SG, please ping again when you do, so I can check and close the request on my side.
Hello Jakub,
The recent updates have been merged, and the tests should now report sub-case results.
Please let us know if any further improvements are needed.
Thanks,
Sheena
Quoted 7 lines · Jakub Kicinski
Quoted 2 lines · On Thu, 23 Jul 2026 10:14:20 -0700 Sheena Mohan wrote:
Thanks, Jakub. We plan to merge the changes to support nested arrays for sub-test results by next week.
SG, please ping again when you do, so I can check and close the request on my side.
Jakub Kicinski
maintainerQuoted 3 lines · On Tue, 28 Jul 2026 10:07:15 -0700 Sheena Mohan wrote:
The recent updates have been merged, and the tests should now report sub-case results.
Please let us know if any further improvements are needed.
Looks good, thanks!
I'm still waiting on the device node, but I assume we're stacking that separately.
Quoted 10 lines · Jakub Kicinski
Quoted 3 lines · On Tue, 28 Jul 2026 10:07:15 -0700 Sheena Mohan wrote:
The recent updates have been merged, and the tests should now report sub-case results.
Please let us know if any further improvements are needed.Looks good, thanks!
I'm still waiting on the device node, but I assume we're stacking that separately.
Yes, we are tracking that separately. I will share an update on devlink support once the patches are submitted for upstream review.
Jakub Kicinski
maintainerQuoted 7 lines · On Tue, 28 Jul 2026 15:38:56 -0700 Sheena Mohan wrote:
Quoted 6 lines · Jakub Kicinski
Quoted 3 lines · On Tue, 28 Jul 2026 10:07:15 -0700 Sheena Mohan wrote:
The recent updates have been merged, and the tests should now report sub-case results.
Please let us know if any further improvements are needed.
Quoted 6 lines
Quoted 4 lines
Looks good, thanks!
I'm still waiting on the device node, but I assume we're stacking that separately.
Yes, we are tracking that separately. I will share an update on devlink support once the patches are submitted for upstream review.
To be clear -- it's not required to be real devlink. The format has to be like devlink (JSON keys). For QEMU virtio for example we have a two liner bash script in NIPA:
qver=$(qemu-system-x86_64 --version | head -1)
echo '{"driver":"virtio_net","versions":{"fixed":{},"stored":{},"running":{"fw":"'"$qver"'"}}}'
https://github.com/linux-netdev/nipa/blob/main/contest/scripts/vm-virtio-dev-info.sh
Of course real devlink reporting is better because it may also help users correlate problems in production.
Quoted 21 lines · Jakub Kicinski
Quoted 7 lines · On Tue, 28 Jul 2026 15:38:56 -0700 Sheena Mohan wrote:
Quoted 6 lines · Jakub Kicinski
Quoted 3 lines · On Tue, 28 Jul 2026 10:07:15 -0700 Sheena Mohan wrote:
The recent updates have been merged, and the tests should now report sub-case results.
Please let us know if any further improvements are needed.Quoted 6 lines
Quoted 4 lines
Looks good, thanks!
I'm still waiting on the device node, but I assume we're stacking that separately.
Yes, we are tracking that separately. I will share an update on devlink support once the patches are submitted for upstream review.
To be clear -- it's not required to be real devlink. The format has to be like devlink (JSON keys). For QEMU virtio for example we have a two liner bash script in NIPA:
Quoted 8 lines
qver=$(qemu-system-x86_64 --version | head -1)
echo '{"driver":"virtio_net","versions":{"fixed":{},"stored":{},"running":{"fw":"'"$qver"'"}}}'
https://github.com/linux-netdev/nipa/blob/main/contest/scripts/vm-virtio-dev-info.sh
Of course real devlink reporting is better because it may also help users correlate problems in production.
Thanks for the clarification. Yes, we can unblock NIPA testing by providing the information via the script until the real devlink feature is supported in the drivers.
Jakub Kicinski
maintainerQuoted 3 lines · On Tue, 28 Jul 2026 10:07:15 -0700 Sheena Mohan wrote:
The recent updates have been merged, and the tests should now report sub-case results.
Please let us know if any further improvements are needed.
It doesn't look right.
test "napi-threaded-py"
group "selftests-drivers-net"
result "pass"
^^^^^^^^^^^^^
link "https://gve-netdev-nipa.static.usercontent.goog/results/202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a/napi-threaded-py/logs.html"
time 20.788391
# selftests: drivers/net: napi_threaded.py
not ok 1 selftests: drivers/net: napi_threaded.py # exit=1 # Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0
# not ok 1 napi_threaded.napi_init # not ok 2 napi_threaded.change_num_queues # not ok 3 napi_threaded.enable_dev_threaded_disable_napi_threaded
Does not look like a pass at all. Subcases not in JSON. Not sure why the subcases are listed under the main return in the output, I've never seen that.
test "psp-py"
group "selftests-drivers-net"
result "pass"
link "https://gve-netdev-nipa.static.usercontent.goog/results/202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a/psp-py/logs.html"
time 341.833939
# selftests: drivers/net: psp.py
not ok 1 selftests: drivers/net: psp.py # exit=1 # Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0
Again, doesn't seem to be a pass??
test "queues-py"
group "selftests-drivers-net"
result "pass"
link "https://gve-netdev-nipa.static.usercontent.goog/results/202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a/queues-py/logs.html"
time 686.483495
results
test "queues-check-xsk"
result "pass"
Okay, here we have a subcase..
# selftests: drivers/net: queues.py
ok 1 selftests: drivers/net: queues.py
# Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0 check_down
# ok 4 queues.check_xsk # # Totals: pass:4 fail:0 xfail:0 xpass:0 skip:0 error:0
But why only one? What happened to the other 3 that ksft clearly saw since it enumerated check_xsk as the fourth??
Something is mangling the output on your runner, it seems. And the parsing seems off.
Quoted 60 lines · Jakub Kicinski
Quoted 3 lines · On Tue, 28 Jul 2026 10:07:15 -0700 Sheena Mohan wrote:
The recent updates have been merged, and the tests should now report sub-case results.
Please let us know if any further improvements are needed.It doesn't look right.
test "napi-threaded-py" group "selftests-drivers-net" result "pass"^^^^^^^^^^^^^
link "https://gve-netdev-nipa.static.usercontent.goog/results/202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a/napi-threaded-py/logs.html"
time 20.788391# selftests:
drivers/net:napi_threaded.py
not ok 1 selftests:drivers/net:napi_threaded.py# exit=1 # Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0# not ok 1 napi_threaded.napi_init # not ok 2 napi_threaded.change_num_queues # not ok 3 napi_threaded.enable_dev_threaded_disable_napi_threadedDoes not look like a pass at all. Subcases not in JSON. Not sure why the subcases are listed under the main return in the output, I've never seen that.
test "psp-py" group "selftests-drivers-net" result "pass" link "https://gve-netdev-nipa.static.usercontent.goog/results/202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a/psp-py/logs.html" time 341.833939# selftests:
drivers/net:psp.py
not ok 1 selftests:drivers/net:psp.py# exit=1 # Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0Again, doesn't seem to be a pass??
test "queues-py" group "selftests-drivers-net" result "pass" link "https://gve-netdev-nipa.static.usercontent.goog/results/202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a/queues-py/logs.html" time 686.483495results
test "queues-check-xsk"
result "pass"Okay, here we have a subcase..
# selftests:
drivers/net:queues.py
ok 1 selftests:drivers/net:queues.py
# Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0check_down# ok 4 queues.check_xsk # # Totals: pass:4 fail:0 xfail:0 xpass:0 skip:0 error:0
Quoted 6 lines
But why only one? What happened to the other 3 that ksft clearly saw since it enumerated
check_xskas the fourth??Something is mangling the output on your runner, it seems. And the parsing seems off.
Thanks for bringing this to our attention. We will look into the parsing issues and work on getting this fixed.
Quoted 62 lines · Sheena Mohan
Quoted 60 lines · Jakub Kicinski
Quoted 3 lines · On Tue, 28 Jul 2026 10:07:15 -0700 Sheena Mohan wrote:
The recent updates have been merged, and the tests should now report sub-case results.
Please let us know if any further improvements are needed.It doesn't look right.
test "napi-threaded-py" group "selftests-drivers-net" result "pass"^^^^^^^^^^^^^
link "https://gve-netdev-nipa.static.usercontent.goog/results/202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a/napi-threaded-py/logs.html"
time 20.788391# selftests:
drivers/net:napi_threaded.py
not ok 1 selftests:drivers/net:napi_threaded.py# exit=1 # Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0# not ok 1 napi_threaded.napi_init # not ok 2 napi_threaded.change_num_queues # not ok 3 napi_threaded.enable_dev_threaded_disable_napi_threadedDoes not look like a pass at all. Subcases not in JSON. Not sure why the subcases are listed under the main return in the output, I've never seen that.
test "psp-py" group "selftests-drivers-net" result "pass" link "https://gve-netdev-nipa.static.usercontent.goog/results/202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a/psp-py/logs.html" time 341.833939# selftests:
drivers/net:psp.py
not ok 1 selftests:drivers/net:psp.py# exit=1 # Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0Again, doesn't seem to be a pass??
test "queues-py" group "selftests-drivers-net" result "pass" link "https://gve-netdev-nipa.static.usercontent.goog/results/202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a/queues-py/logs.html" time 686.483495results
test "queues-check-xsk"
result "pass"Okay, here we have a subcase..
# selftests:
drivers/net:queues.py
ok 1 selftests:drivers/net:queues.py
# Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0check_down# ok 4 queues.check_xsk # # Totals: pass:4 fail:0 xfail:0 xpass:0 skip:0 error:0
Quoted 8 lines
Quoted 6 lines
But why only one? What happened to the other 3 that ksft clearly saw since it enumerated
check_xskas the fourth??Something is mangling the output on your runner, it seems. And the parsing seems off.
Thanks for bringing this to our attention. We will look into the parsing issues and work on getting this fixed.
Latest test results should now cleanly report subtest results.