Skip to contentSkip to conversation

Weird RSS state on ice

TEST Jakub Kicinski 2 participants 4 messages Jun 24 – Jun 26

Subsystems (2)
lore
Share & tools
lore

Jakub Kicinski

author maintainer
#0 lore raw

Hi!

I noticed in the netdev CI that the ice runner fails to run the toeplitz tests because of the RSS config.

https://netdev-ci-results.intel.com/ice-results/net-next-hw-2026-06-23--00-00/ice-E810-CQ2/toeplitz.py/stdout

I added some extra debug on the branch:

net.lib.ynl.pyynl.lib.ynl.NlError: Netlink error: hash field config is not symmetric 16 304: Invalid argument {'bad-attr': '.input-xfrm'}

16, 304 means GTP flow, GTP_TEID field. So we are trying to disable symmetric RSS, but the field configuration contains TEID. The problem is this is an illegal configuration in the first place. We are _disabling_ symmetric RSS, but the kernel tries to make sure that both before and after states are correct (because the configuration involves multiple calls to the drivers and may fail half-way-thru). If the current config is illegal net/ethtool/ won't even let us restore it to sane state.

So the question is how we got into this state. It does not happen on netdev machines. And on Intel machines it happens randomly around 30% of the time.

I tried to look thru the driver code and I don't see how we could end up with such a config.

Could y'all have a look and figure out / fix this? This has been happening for a while back but I was waiting until the merge window to poke at it first.

#1 lore raw
Quoted 37 lines

-----Original Message-----
From: Intel-wired-lan <[email protected]> On Behalf Of Jakub Kicinski
Sent: Wednesday, June 24, 2026 5:30 PM
To: Pielech, Adrian <[email protected]>; Kitszel, Przemyslaw <[email protected]>
Cc: [email protected]; [email protected]
Subject: [Intel-wired-lan] [TEST] Weird RSS state on ice

Hi!

I noticed in the netdev CI that the ice runner fails to run the toeplitz tests because of the RSS config.

https://netdev-ci-results.intel.com/ice-results/net-next-hw-2026-06-
23--00-00/ice-E810-CQ2/toeplitz.py/stdout

I added some extra debug on the branch:

net.lib.ynl.pyynl.lib.ynl.NlError: Netlink error: hash field config is not symmetric 16 304: Invalid argument {'bad-attr': '.input-xfrm'}

16, 304 means GTP flow, GTP_TEID field. So we are trying to disable symmetric RSS, but the field configuration contains TEID. The problem is this is an illegal configuration in the first place. We are _disabling_ symmetric RSS, but the kernel tries to make sure that both before and after states are correct (because the configuration involves multiple calls to the drivers and may fail half-way-thru). If the current config is illegal net/ethtool/ won't even let us restore it to sane state.

So the question is how we got into this state. It does not happen on netdev machines. And on Intel machines it happens randomly around 30% of the time.

Quoted 6 lines

I tried to look thru the driver code and I don't see how we could end up with such a config.

Could y'all have a look and figure out / fix this? This has been happening for a while back but I was waiting until the merge window to poke at it first.

Good day, Jakub

The patchset didn't help?

[PATCH iwl-next v5 2/2] ice: implement symmetric RSS hash configuration

With the best regards
Alex

author maintainer
#2 lore raw
Quoted 3 lines · Aleksandr

The patchset didn't help?

[PATCH iwl-next v5 2/2] ice: implement symmetric RSS hash configuration

Not sure, it's not in tree, and lore doesn't want to point me at it either. What I don't get is how we get into the bad state in the first place.

Looking at other tests today I spotted that rss flow label test is also behaving oddly. Most of the time the first case fails and the second passes:

test "rss-flow-label-py"
group "selftests-drivers-net-hw"
result "fail"
link "https://netdev-ci-results.intel.com/ice-results/net-next-hw-2026-06-26--00-00/ice-E810-XXV4/rss_flow_label.py/stdout"
results
0
test "rss-flow-label-test-rss-flow-label"
result "fail"
1
test "rss-flow-label-test-rss-flow-label-6only"
result "pass"

But every now and then they skip:

ok 1 rss_flow_label.test_rss_flow_label # SKIP Device doesn't support Flow Label for UDP6 ok 2 rss_flow_label.test_rss_flow_label_6only # SKIP Device doesn't support Flow Label for UDP6

test "rss-flow-label-py"
group "selftests-drivers-net-hw"
result "skip"
link "https://netdev-ci-results.intel.com/ice-results/net-next-hw-2026-06-25--16-00/ice-E810-XXV4/rss_flow_label.py/stdout"
results
0
test "rss-flow-label-test-rss-flow-label"
result "skip"
1
test "rss-flow-label-test-rss-flow-label-6only"
result "skip"

The devlink info is identical so it must be that the device is in unclean state sometimes?? Do y'all power cycle these machines between runs?

#3 lore raw
Quoted 60 lines

-----Original Message-----
From: Jakub Kicinski <[email protected]>
Sent: Friday, June 26, 2026 4:06 AM
To: Loktionov, Aleksandr <[email protected]>
Cc: Pielech, Adrian <[email protected]>; Kitszel, Przemyslaw <[email protected]>; [email protected]; intel-wired- [email protected]
Subject: Re: [Intel-wired-lan] [TEST] Weird RSS state on ice

Quoted 4 lines · Aleksandr

The patchset didn't help?

[PATCH iwl-next v5 2/2] ice: implement symmetric RSS hash configuration

Not sure, it's not in tree, and lore doesn't want to point me at it either. What I don't get is how we get into the bad state in the first place.

Looking at other tests today I spotted that rss flow label test is also behaving oddly. Most of the time the first case fails and the second
passes:

test "rss-flow-label-py"
group "selftests-drivers-net-hw"
result "fail"
link "https://netdev-ci-results.intel.com/ice-results/net-next-hw-
2026-06-26--00-00/ice-E810-XXV4/rss_flow_label.py/stdout" results
0
test "rss-flow-label-test-rss-flow-label"
result "fail"
1
test "rss-flow-label-test-rss-flow-label-6only"
result "pass"

But every now and then they skip:

ok 1 rss_flow_label.test_rss_flow_label # SKIP Device doesn't support Flow Label for UDP6 ok 2 rss_flow_label.test_rss_flow_label_6only # SKIP Device doesn't support Flow Label for UDP6

test "rss-flow-label-py"
group "selftests-drivers-net-hw"
result "skip"
link "https://netdev-ci-results.intel.com/ice-results/net-next-hw-
2026-06-25--16-00/ice-E810-XXV4/rss_flow_label.py/stdout" results
0
test "rss-flow-label-test-rss-flow-label"
result "skip"
1
test "rss-flow-label-test-rss-flow-label-6only"
result "skip"

Quoted 3 lines

The devlink info is identical so it must be that the device is in unclean state sometimes?? Do y'all power cycle these machines between runs?

Good day, Jakub

I heard from @Pielech, Adrian that we experienced infrastructure issues, but reboots helped us. Please ask him about CI infrastructure.

About my v5 March 16 symmetric RSS fix, which worked for me, I've just resent it today, please bless it.

With the best regards
Alex