Skip to contentSkip to conversation

google runner needs nested cases

TEST Jakub Kicinski 2 participants 11 messages Jul 21 – Aug 13

Subsystems (2)
lore
Share & tools
lore
#1 lore raw

Thanks, Jakub. We plan to merge the changes to support nested arrays for sub-test results by next week.

Quoted 10 lines · Jakub Kicinski

Hi!

Another ask, please parse the TAP output of the tests and provide a nested array of results per-subcase, see:

https://github.com/linux-netdev/nipa/wiki/Netdev-CI-system/#results

right now you only seem to report top level pass/fail but most tests have at least one sub-case the driver doesn't support.

author maintainer
#2 lore raw
Quoted 2 lines · On Thu, 23 Jul 2026 10:14:20 -0700 Sheena Mohan wrote:

Thanks, Jakub. We plan to merge the changes to support nested arrays for sub-test results by next week.

SG, please ping again when you do, so I can check and close the request on my side.

#3 lore raw

Hello Jakub,

The recent updates have been merged, and the tests should now report sub-case results.
Please let us know if any further improvements are needed.

Thanks,
Sheena

Quoted 7 lines · Jakub Kicinski
Quoted 2 lines · On Thu, 23 Jul 2026 10:14:20 -0700 Sheena Mohan wrote:

Thanks, Jakub. We plan to merge the changes to support nested arrays for sub-test results by next week.

SG, please ping again when you do, so I can check and close the request on my side.

author maintainer
#4 lore raw
Quoted 3 lines · On Tue, 28 Jul 2026 10:07:15 -0700 Sheena Mohan wrote:

The recent updates have been merged, and the tests should now report sub-case results.
Please let us know if any further improvements are needed.

Looks good, thanks!

I'm still waiting on the device node, but I assume we're stacking that separately.

#5 lore raw
Quoted 10 lines · Jakub Kicinski
Quoted 3 lines · On Tue, 28 Jul 2026 10:07:15 -0700 Sheena Mohan wrote:

The recent updates have been merged, and the tests should now report sub-case results.
Please let us know if any further improvements are needed.

Looks good, thanks!

I'm still waiting on the device node, but I assume we're stacking that separately.

Yes, we are tracking that separately. I will share an update on devlink support once the patches are submitted for upstream review.

author maintainer
#6 lore raw
Quoted 7 lines · On Tue, 28 Jul 2026 15:38:56 -0700 Sheena Mohan wrote:
Quoted 6 lines · Jakub Kicinski
Quoted 3 lines · On Tue, 28 Jul 2026 10:07:15 -0700 Sheena Mohan wrote:

The recent updates have been merged, and the tests should now report sub-case results.
Please let us know if any further improvements are needed.

Quoted 6 lines
Quoted 4 lines

Looks good, thanks!

I'm still waiting on the device node, but I assume we're stacking that separately.

Yes, we are tracking that separately. I will share an update on devlink support once the patches are submitted for upstream review.

To be clear -- it's not required to be real devlink. The format has to be like devlink (JSON keys). For QEMU virtio for example we have a two liner bash script in NIPA:

qver=$(qemu-system-x86_64 --version | head -1)

echo '{"driver":"virtio_net","versions":{"fixed":{},"stored":{},"running":{"fw":"'"$qver"'"}}}'

https://github.com/linux-netdev/nipa/blob/main/contest/scripts/vm-virtio-dev-info.sh

Of course real devlink reporting is better because it may also help users correlate problems in production.

#7 lore raw
Quoted 21 lines · Jakub Kicinski
Quoted 7 lines · On Tue, 28 Jul 2026 15:38:56 -0700 Sheena Mohan wrote:
Quoted 6 lines · Jakub Kicinski
Quoted 3 lines · On Tue, 28 Jul 2026 10:07:15 -0700 Sheena Mohan wrote:

The recent updates have been merged, and the tests should now report sub-case results.
Please let us know if any further improvements are needed.

Quoted 6 lines
Quoted 4 lines

Looks good, thanks!

I'm still waiting on the device node, but I assume we're stacking that separately.

Yes, we are tracking that separately. I will share an update on devlink support once the patches are submitted for upstream review.

To be clear -- it's not required to be real devlink. The format has to be like devlink (JSON keys). For QEMU virtio for example we have a two liner bash script in NIPA:

Quoted 8 lines

qver=$(qemu-system-x86_64 --version | head -1)

echo '{"driver":"virtio_net","versions":{"fixed":{},"stored":{},"running":{"fw":"'"$qver"'"}}}'

https://github.com/linux-netdev/nipa/blob/main/contest/scripts/vm-virtio-dev-info.sh

Of course real devlink reporting is better because it may also help users correlate problems in production.

Thanks for the clarification. Yes, we can unblock NIPA testing by providing the information via the script until the real devlink feature is supported in the drivers.

author maintainer
#8 lore raw
In reply to Sheena Mohan
Quoted 3 lines · On Tue, 28 Jul 2026 10:07:15 -0700 Sheena Mohan wrote:

The recent updates have been merged, and the tests should now report sub-case results.
Please let us know if any further improvements are needed.

It doesn't look right.

https://gve-netdev-nipa.static.usercontent.goog/json/results-202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a.json

test "napi-threaded-py"
group "selftests-drivers-net"
result "pass"
^^^^^^^^^^^^^
link "https://gve-netdev-nipa.static.usercontent.goog/results/202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a/napi-threaded-py/logs.html"
time 20.788391

# selftests: drivers/net: napi_threaded.py
not ok 1 selftests: drivers/net: napi_threaded.py # exit=1 # Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0

# not ok 1 napi_threaded.napi_init
# not ok 2 napi_threaded.change_num_queues
# not ok 3 napi_threaded.enable_dev_threaded_disable_napi_threaded

Does not look like a pass at all. Subcases not in JSON. Not sure why the subcases are listed under the main return in the output, I've never seen that.

test "psp-py"
group "selftests-drivers-net"
result "pass"
link "https://gve-netdev-nipa.static.usercontent.goog/results/202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a/psp-py/logs.html"
time 341.833939

# selftests: drivers/net: psp.py
not ok 1 selftests: drivers/net: psp.py # exit=1 # Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0

Again, doesn't seem to be a pass??

test "queues-py"
group "selftests-drivers-net"
result "pass"
link "https://gve-netdev-nipa.static.usercontent.goog/results/202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a/queues-py/logs.html"
time 686.483495
results
test "queues-check-xsk"
result "pass"

Okay, here we have a subcase..

# selftests: drivers/net: queues.py
ok 1 selftests: drivers/net: queues.py
# Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0 check_down

# ok 4 queues.check_xsk
# # Totals: pass:4 fail:0 xfail:0 xpass:0 skip:0 error:0

But why only one? What happened to the other 3 that ksft clearly saw since it enumerated check_xsk as the fourth??

Something is mangling the output on your runner, it seems. And the parsing seems off.

#9 lore raw
Quoted 60 lines · Jakub Kicinski
Quoted 3 lines · On Tue, 28 Jul 2026 10:07:15 -0700 Sheena Mohan wrote:

The recent updates have been merged, and the tests should now report sub-case results.
Please let us know if any further improvements are needed.

It doesn't look right.

https://gve-netdev-nipa.static.usercontent.goog/json/results-202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a.json

test    "napi-threaded-py"
group   "selftests-drivers-net"
result  "pass"

^^^^^^^^^^^^^
link "https://gve-netdev-nipa.static.usercontent.goog/results/202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a/napi-threaded-py/logs.html"
time 20.788391

# selftests: drivers/net: napi_threaded.py
not ok 1 selftests: drivers/net: napi_threaded.py # exit=1 # Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0

# not ok 1 napi_threaded.napi_init
# not ok 2 napi_threaded.change_num_queues
# not ok 3 napi_threaded.enable_dev_threaded_disable_napi_threaded

Does not look like a pass at all. Subcases not in JSON. Not sure why the subcases are listed under the main return in the output, I've never seen that.

test    "psp-py"
group   "selftests-drivers-net"
result  "pass"
link    "https://gve-netdev-nipa.static.usercontent.goog/results/202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a/psp-py/logs.html"
time    341.833939

# selftests: drivers/net: psp.py
not ok 1 selftests: drivers/net: psp.py # exit=1 # Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0

Again, doesn't seem to be a pass??

test    "queues-py"
group   "selftests-drivers-net"
result  "pass"
link    "https://gve-netdev-nipa.static.usercontent.goog/results/202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a/queues-py/logs.html"
time    686.483495

results
test "queues-check-xsk"
result "pass"

Okay, here we have a subcase..

# selftests: drivers/net: queues.py
ok 1 selftests: drivers/net: queues.py
# Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0 check_down

# ok 4 queues.check_xsk
# # Totals: pass:4 fail:0 xfail:0 xpass:0 skip:0 error:0
Quoted 6 lines

But why only one? What happened to the other 3 that ksft clearly saw since it enumerated check_xsk as the fourth??

Something is mangling the output on your runner, it seems. And the parsing seems off.

Thanks for bringing this to our attention. We will look into the parsing issues and work on getting this fixed.

#10 lore raw
Quoted 62 lines · Sheena Mohan
Quoted 60 lines · Jakub Kicinski
Quoted 3 lines · On Tue, 28 Jul 2026 10:07:15 -0700 Sheena Mohan wrote:

The recent updates have been merged, and the tests should now report sub-case results.
Please let us know if any further improvements are needed.

It doesn't look right.

https://gve-netdev-nipa.static.usercontent.goog/json/results-202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a.json

test    "napi-threaded-py"
group   "selftests-drivers-net"
result  "pass"

^^^^^^^^^^^^^
link "https://gve-netdev-nipa.static.usercontent.goog/results/202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a/napi-threaded-py/logs.html"
time 20.788391

# selftests: drivers/net: napi_threaded.py
not ok 1 selftests: drivers/net: napi_threaded.py # exit=1 # Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0

# not ok 1 napi_threaded.napi_init
# not ok 2 napi_threaded.change_num_queues
# not ok 3 napi_threaded.enable_dev_threaded_disable_napi_threaded

Does not look like a pass at all. Subcases not in JSON. Not sure why the subcases are listed under the main return in the output, I've never seen that.

test    "psp-py"
group   "selftests-drivers-net"
result  "pass"
link    "https://gve-netdev-nipa.static.usercontent.goog/results/202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a/psp-py/logs.html"
time    341.833939

# selftests: drivers/net: psp.py
not ok 1 selftests: drivers/net: psp.py # exit=1 # Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0

Again, doesn't seem to be a pass??

test    "queues-py"
group   "selftests-drivers-net"
result  "pass"
link    "https://gve-netdev-nipa.static.usercontent.goog/results/202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a/queues-py/logs.html"
time    686.483495

results
test "queues-check-xsk"
result "pass"

Okay, here we have a subcase..

# selftests: drivers/net: queues.py
ok 1 selftests: drivers/net: queues.py
# Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0 check_down

# ok 4 queues.check_xsk
# # Totals: pass:4 fail:0 xfail:0 xpass:0 skip:0 error:0
Quoted 8 lines
Quoted 6 lines

But why only one? What happened to the other 3 that ksft clearly saw since it enumerated check_xsk as the fourth??

Something is mangling the output on your runner, it seems. And the parsing seems off.

Thanks for bringing this to our attention. We will look into the parsing issues and work on getting this fixed.

Latest test results should now cleanly report subtest results.