Skip to contentSkip to conversation

selftests: drv-net: rss: validate min RSS table size

PATCH Applied Jakub Kicinski 5 participants 12 messages Jan 31 – Feb 18

Subsystems (4)
lore
Share & tools
lore

Jakub Kicinski

author maintainer
#0 lore raw

PATCH v2 1/2 selftests: drv-net: rss: validate min RSS table size

Add a test which checks that the RSS table is at least 4x the max queue count supported by the device. The original RSS spec from Microsoft stated that the RSS indirection table should be 2 to 8 times the CPU count, presumably assuming queue per CPU. If the CPU count is not a power of two, however, a power-of-2 table 2x larger than queue count results in a 33% traffic imbalance. Validate that the indirection table is at least 4x the queue count. This lowers the imbalance to 16% which empirically appears to be more acceptable to memcache-like workloads.

---
v2:
- no changes (see patch 2)
v1: https://lore.kernel.org/[email protected]

---
 .../testing/selftests/drivers/net/hw/Makefile |  1 +
 .../selftests/drivers/net/hw/rss_drv.py       | 88 +++++++++++++++++++
 2 files changed, 89 insertions(+)
 create mode 100755 tools/testing/selftests/drivers/net/hw/rss_drv.py
tools/testing/selftests/drivers/net/hw/Makefile +1 -0
index 9c163ba6feee..a64140333a46 100644
@@ -35,6 +35,7 @@ TEST_PROGS = \
 	pp_alloc_fail.py \
 	rss_api.py \
 	rss_ctx.py \
+	rss_drv.py \
 	rss_flow_label.py \
 	rss_input_xfrm.py \
 	toeplitz.py \
tools/testing/selftests/drivers/net/hw/rss_drv.py new file +88 -0

88 lines not on this page · raw message

Signature · 1 line
2.52.0
author maintainer
#1 lore raw

PATCH v2 2/2 docs: networking: mention that RSS table should be 4x the queue count

Spell out the recommendation that the RSS table should be 4x the queue count to avoid traffic imbalance. Include minor rephrasing and removal of the explicit 128 entry example since a 128 entry table is inadequate on modern machines.

---
v2:
- new patch

---
 Documentation/networking/scaling.rst | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
Documentation/networking/scaling.rst +8 -4
index 99b6a61e5e31..0023afa530ec 100644
@@ -38,11 +38,15 @@ that is not the focus of these techniques.
 The filter used in RSS is typically a hash function over the network
 and/or transport layer headers-- for example, a 4-tuple hash over
 IP addresses and TCP ports of a packet. The most common hardware
-implementation of RSS uses a 128-entry indirection table where each entry
+implementation of RSS uses an indirection table where each entry
 stores a queue number. The receive queue for a packet is determined
-by masking out the low order seven bits of the computed hash for the
-packet (usually a Toeplitz hash), taking this number as a key into the
-indirection table and reading the corresponding value.
+by indexing the indirection table with the low order bits of the
+computed hash for the packet (usually a Toeplitz hash).
+
+The indirection table helps even out the traffic distribution when queue
+count is not a power of two. NICs should provide an indirection table
+at least 4 times larger than the queue count. 4x table results in ~16%
+imbalance between the queues, which is acceptable for most applications.
 
 Some NICs support symmetric RSS hashing where, if the IP (source address,
 destination address) and TCP/UDP (source port, destination port) tuples
Signature · 1 line
2.52.0
Reviewed-by on v2 2/2
#2 lore raw

Re: [PATCH net-next v2 2/2] docs: networking: mention that RSS table should be 4x the queue count

Quoted 7 lines · Jakub Kicinski

Spell out the recommendation that the RSS table should be 4x the queue count to avoid traffic imbalance. Include minor rephrasing and removal of the explicit 128 entry example since a 128 entry table is inadequate on modern machines.

Thanks Jakub.

#3 lore raw
In reply to Jakub Kicinski
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Quoted 11 lines · On Sat, 31 Jan 2026 14:54:53 -0800 you wrote:

Add a test which checks that the RSS table is at least 4x the max queue count supported by the device. The original RSS spec from Microsoft stated that the RSS indirection table should be 2 to 8 times the CPU count, presumably assuming queue per CPU. If the CPU count is not a power of two, however, a power-of-2 table 2x larger than queue count results in a 33% traffic imbalance. Validate that the indirection table is at least 4x the queue count. This lowers the imbalance to 16% which empirically appears to be more acceptable to memcache-like workloads.

[...]

Here is the summary with links:
- [net-next,v2,1/2] selftests: drv-net: rss: validate min RSS table size

    https://git.kernel.org/netdev/net-next/c/9e3d4dae9832
  - [net-next,v2,2/2] docs: networking: mention that RSS table should be 4x the queue count
    https://git.kernel.org/netdev/net-next/c/ba9c5611f088

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
#4 lore raw
In reply to Jakub Kicinski

tools/testing/selftests/drivers/net/hw/rss_drv.py

Quoted 42 lines · Jakub Kicinski

Add a test which checks that the RSS table is at least 4x the max queue count supported by the device. The original RSS spec from Microsoft stated that the RSS indirection table should be 2 to 8 times the CPU count, presumably assuming queue per CPU. If the CPU count is not a power of two, however, a power-of-2 table 2x larger than queue count results in a 33% traffic imbalance. Validate that the indirection table is at least 4x the queue count. This lowers the imbalance to 16% which empirically appears to be more acceptable to memcache-like workloads.

---
v2:
- no changes (see patch 2)
v1: https://lore.kernel.org/[email protected]

---
 .../testing/selftests/drivers/net/hw/Makefile |  1 +
 .../selftests/drivers/net/hw/rss_drv.py       | 88 +++++++++++++++++++
 2 files changed, 89 insertions(+)
 create mode 100755 tools/testing/selftests/drivers/net/hw/rss_drv.py
tools/testing/selftests/drivers/net/hw/Makefile +1 -0
index 9c163ba6feee..a64140333a46 100644
@@ -35,6 +35,7 @@ TEST_PROGS = \
 	pp_alloc_fail.py \
 	rss_api.py \
 	rss_ctx.py \
+	rss_drv.py \
 	rss_flow_label.py \
 	rss_input_xfrm.py \
 	toeplitz.py \
tools/testing/selftests/drivers/net/hw/rss_drv.py new file +0 -0
new file mode 100755
index 000000000000..2d1a33189076

Hi Jakub,
Thanks for the test addition. I wanted to raise a concern regarding the spread factor requirement that may apply to mlx5 and potentially other drivers as well.
The real issue arises when the hardware's maximum RQT (indirection table) size isn't large enough to accommodate both the desired number of channels and a spread factor of 4. RX queues/channels serve multiple purposes beyond RSS - they're also used for XDP, AF_XDP, and direct queue steering via ntuple filters or TC.
Artificially limiting the number of channels based solely on RSS spread requirements would be overly restrictive for these non-RSS use cases. In such scenarios, we'd rather have a slightly degraded spread factor (< 4) than limit channel availability.
We'd appreciate any feedback on this approach.
Thanks,
Yael

Quoted 89 lines
tools/testing/selftests/drivers/net/hw/rss_drv.py new file +88 -0
@@ -0,0 +1,88 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+"""
+Driver-related behavior tests for RSS.
+"""
+
+from lib.py import ksft_run, ksft_exit, ksft_ge
+from lib.py import ksft_variants, KsftNamedVariant, KsftSkipEx
+from lib.py import defer, ethtool
+from lib.py import EthtoolFamily, NlError
+from lib.py import NetDrvEnv
+
+
+def _is_power_of_two(n):
+    return n > 0 and (n & (n - 1)) == 0
+
+
+def _get_rss(cfg, context=0):
+    return ethtool(f"-x {cfg.ifname} context {context}", json=True)[0]
+
+
+def _test_rss_indir_size(cfg, qcnt, context=0):
+    """Test that indirection table size is at least 4x queue count."""
+    ethtool(f"-L {cfg.ifname} combined {qcnt}")
+
+    rss = _get_rss(cfg, context=context)
+    indir = rss['rss-indirection-table']
+    ksft_ge(len(indir), 4 * qcnt, "Table smaller than 4x")
+    return len(indir)
+
+
+def _maybe_create_context(cfg, create_context):
+    """ Either create a context and return its ID or return 0 for main ctx """
+    if not create_context:
+        return 0
+    try:
+        ctx = cfg.ethnl.rss_create_act({'header': {'dev-index': cfg.ifindex}})
+        ctx_id = ctx['context']
+        defer(cfg.ethnl.rss_delete_act,
+              {'header': {'dev-index': cfg.ifindex}, 'context': ctx_id})
+    except NlError:
+        raise KsftSkipEx("Device does not support additional RSS contexts")
+
+    return ctx_id
+
+
+@ksft_variants([
+    KsftNamedVariant("main", False),
+    KsftNamedVariant("ctx", True),
+])
+def indir_size_4x(cfg, create_context):
+    """
+    Test that the indirection table has at least 4 entries per queue.
+    Empirically network-heavy workloads like memcache suffer with the 33%
+    imbalance of a 2x indirection table size.
+    4x table translates to a 16% imbalance.
+    """
+    channels = cfg.ethnl.channels_get({'header': {'dev-index': cfg.ifindex}})
+    ch_max = channels.get('combined-max', 0)
+    qcnt = channels['combined-count']
+
+    if ch_max < 3:
+        raise KsftSkipEx(f"Not enough queues for the test: max={ch_max}")
+
+    defer(ethtool, f"-L {cfg.ifname} combined {qcnt}")
+    ethtool(f"-L {cfg.ifname} combined 3")
+
+    ctx_id = _maybe_create_context(cfg, create_context)
+
+    indir_sz = _test_rss_indir_size(cfg, 3, context=ctx_id)
+
+    # Test with max queue count (max - 1 if max is a power of two)
+    test_max = ch_max - 1 if _is_power_of_two(ch_max) else ch_max
+    if test_max > 3 and indir_sz < test_max * 4:
+        _test_rss_indir_size(cfg, test_max, context=ctx_id)
+
+
+def main() -> None:
+    """ Ksft boiler plate main """
+    with NetDrvEnv(__file__) as cfg:
+        cfg.ethnl = EthtoolFamily()
+        ksft_run([indir_size_4x], args=(cfg, ))
+    ksft_exit()
+
+
+if __name__ == "__main__":
+    main()
author maintainer
#5 lore raw
Quoted 5 lines · On Wed, 11 Feb 2026 22:10:56 +0200 Yael Chemla wrote:

Thanks for the test addition. I wanted to raise a concern regarding the spread factor requirement that may apply to mlx5 and potentially other drivers as well.
The real issue arises when the hardware's maximum RQT (indirection table) size isn't large enough to accommodate both the desired number of

Quoted 8 lines

channels and a spread factor of 4. RX queues/channels serve multiple purposes beyond RSS - they're also used for XDP, AF_XDP, and direct queue steering via ntuple filters or TC.
Artificially limiting the number of channels based solely on RSS spread requirements would be overly restrictive for these non-RSS use cases. In such scenarios, we'd rather have a slightly degraded spread factor (< 4) than limit channel availability.
We'd appreciate any feedback on this approach.

That's fine. In fact IIRC ixgbe (infamously) had more queues than it could fit in its RSS table. So none of this is new. At the same time if user _does_ want to use a lot of queues in the main context fewer than 4x entries in the indir table is inadequate.

The test is based on production experience, and provides valuable guidance to device developers.

I'm not sure what you want me to say here.

#6 lore raw
Quoted 20 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 11 Feb 2026 22:10:56 +0200 Yael Chemla wrote:

Thanks for the test addition. I wanted to raise a concern regarding the spread factor requirement that may apply to mlx5 and potentially other drivers as well.
The real issue arises when the hardware's maximum RQT (indirection table) size isn't large enough to accommodate both the desired number of

Quoted 8 lines

channels and a spread factor of 4. RX queues/channels serve multiple purposes beyond RSS - they're also used for XDP, AF_XDP, and direct queue steering via ntuple filters or TC.
Artificially limiting the number of channels based solely on RSS spread requirements would be overly restrictive for these non-RSS use cases. In such scenarios, we'd rather have a slightly degraded spread factor (< 4) than limit channel availability.
We'd appreciate any feedback on this approach.

That's fine. In fact IIRC ixgbe (infamously) had more queues than it could fit in its RSS table. So none of this is new. At the same time if user _does_ want to use a lot of queues in the main context fewer than 4x entries in the indir table is inadequate.

Quoted 5 lines

The test is based on production experience, and provides valuable guidance to device developers.

I'm not sure what you want me to say here.

No doubt that larger factors help overcome imbalance issues, and it's fine to recommend using 4x (or even larger) factors.

The point is, when this comes with a selftest, it's less of a recommendation/guidance anymore, it becomes kind of a requirement, an expected behavior. Otherwise the test fails.

This ignores multiple other considerations:

1. Existing behavior: In general, mlx5e today implies 2x factor, so it would fail this new test.

2. Device resources: In large scale (high num of channels, or high num of netdevs on the same chip, or both), it is not obvious that increasing the indirection table size is still desirable, or even possible. To pass the selftest, you'll have to limit the max number of channels.

3. ch_max should win: Related to point #2. Driver should not enforce limitations on supported ch_max just to fulfill the recommendation and pass the test. I prefer flexibility, give the admin the control. That means, driver would use 4x factor (or larger) whenever possible, but would not block configurations in which the 4x factor cannot be satisfied.

author maintainer
#7 lore raw
Quoted 43 lines · On Thu, 12 Feb 2026 11:41:19 +0200 Tariq Toukan wrote:
Quoted 20 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 11 Feb 2026 22:10:56 +0200 Yael Chemla wrote:

Thanks for the test addition. I wanted to raise a concern regarding the spread factor requirement that may apply to mlx5 and potentially other drivers as well.
The real issue arises when the hardware's maximum RQT (indirection table) size isn't large enough to accommodate both the desired number of

Quoted 8 lines

channels and a spread factor of 4. RX queues/channels serve multiple purposes beyond RSS - they're also used for XDP, AF_XDP, and direct queue steering via ntuple filters or TC.
Artificially limiting the number of channels based solely on RSS spread requirements would be overly restrictive for these non-RSS use cases. In such scenarios, we'd rather have a slightly degraded spread factor (< 4) than limit channel availability.
We'd appreciate any feedback on this approach.

That's fine. In fact IIRC ixgbe (infamously) had more queues than it could fit in its RSS table. So none of this is new. At the same time if user _does_ want to use a lot of queues in the main context fewer than 4x entries in the indir table is inadequate.

Quoted 4 lines

The test is based on production experience, and provides valuable guidance to device developers.

I'm not sure what you want me to say here.

No doubt that larger factors help overcome imbalance issues, and it's fine to recommend using 4x (or even larger) factors.

The point is, when this comes with a selftest, it's less of a recommendation/guidance anymore, it becomes kind of a requirement, an expected behavior. Otherwise the test fails.

This ignores multiple other considerations:

1. Existing behavior: In general, mlx5e today implies 2x factor, so it would fail this new test.

2. Device resources: In large scale (high num of channels, or high num of netdevs on the same chip, or both), it is not obvious that increasing the indirection table size is still desirable, or even possible. To pass the selftest, you'll have to limit the max number of channels.

Quoted 5 lines

3. ch_max should win: Related to point #2. Driver should not enforce limitations on supported ch_max just to fulfill the recommendation and pass the test. I prefer flexibility, give the admin the control. That means, driver would use 4x factor (or larger) whenever possible, but would not block configurations in which the 4x factor cannot be satisfied.

Oh I see.. I wasn't aware the CX7 has a limitation of the indirection table size. I wrote the test because of a similar limitation in a different NIC, but that one has been fixed.. I have limited access to CX7 NICs, the one I tested on maxed out at 63 queues so the test has passed.

Is it not possible to create an indirection table larger than 256 entries? 256 is not a lot, AMD Venice (to pick one) will have up to 256 CPU cores (not threads) in a single CPU package.

#8 lore raw
Quoted 44 lines · Jakub Kicinski
Quoted 43 lines · On Thu, 12 Feb 2026 11:41:19 +0200 Tariq Toukan wrote:
Quoted 20 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 11 Feb 2026 22:10:56 +0200 Yael Chemla wrote:

Thanks for the test addition. I wanted to raise a concern regarding the spread factor requirement that may apply to mlx5 and potentially other drivers as well.
The real issue arises when the hardware's maximum RQT (indirection table) size isn't large enough to accommodate both the desired number of

Quoted 8 lines

channels and a spread factor of 4. RX queues/channels serve multiple purposes beyond RSS - they're also used for XDP, AF_XDP, and direct queue steering via ntuple filters or TC.
Artificially limiting the number of channels based solely on RSS spread requirements would be overly restrictive for these non-RSS use cases. In such scenarios, we'd rather have a slightly degraded spread factor (< 4) than limit channel availability.
We'd appreciate any feedback on this approach.

That's fine. In fact IIRC ixgbe (infamously) had more queues than it could fit in its RSS table. So none of this is new. At the same time if user _does_ want to use a lot of queues in the main context fewer than 4x entries in the indir table is inadequate.

Quoted 4 lines

The test is based on production experience, and provides valuable guidance to device developers.

I'm not sure what you want me to say here.

No doubt that larger factors help overcome imbalance issues, and it's fine to recommend using 4x (or even larger) factors.

The point is, when this comes with a selftest, it's less of a recommendation/guidance anymore, it becomes kind of a requirement, an expected behavior. Otherwise the test fails.

This ignores multiple other considerations:

1. Existing behavior: In general, mlx5e today implies 2x factor, so it would fail this new test.

2. Device resources: In large scale (high num of channels, or high num of netdevs on the same chip, or both), it is not obvious that increasing the indirection table size is still desirable, or even possible. To pass the selftest, you'll have to limit the max number of channels.

Quoted 8 lines
Quoted 5 lines

3. ch_max should win: Related to point #2. Driver should not enforce limitations on supported ch_max just to fulfill the recommendation and pass the test. I prefer flexibility, give the admin the control. That means, driver would use 4x factor (or larger) whenever possible, but would not block configurations in which the 4x factor cannot be satisfied.

Oh I see.. I wasn't aware the CX7 has a limitation of the indirection table size.

There is a limitation, we read it from FW.
It's usually not small, much larger than 256.

But currently it can vary according to FW decisions in scale (resource management).

Quoted 7 lines

I wrote the test because of a similar limitation in a different NIC, but that one has been fixed.. I have limited access to CX7 NICs, the one I tested on maxed out at 63 queues so the test has passed.

Is it not possible to create an indirection table larger than 256 entries?

It is possible, depending on the exposed FW capability.
As of today, there are high-scale configurations (many VFs for example) where the FW exposed cap is lowered.

Quoted 2 lines

256 is not a lot, AMD Venice (to pick one) will have up to 256 CPU cores (not threads) in a single CPU package.

author maintainer
#9 lore raw
Quoted 30 lines · On Mon, 16 Feb 2026 10:28:52 +0200 Tariq Toukan wrote:
Quoted 10 lines
Quoted 10 lines

This ignores multiple other considerations:

1. Existing behavior: In general, mlx5e today implies 2x factor, so it would fail this new test.

2. Device resources: In large scale (high num of channels, or high num of netdevs on the same chip, or both), it is not obvious that increasing the indirection table size is still desirable, or even possible. To pass the selftest, you'll have to limit the max number of channels.

Quoted 8 lines
Quoted 5 lines

3. ch_max should win: Related to point #2. Driver should not enforce limitations on supported ch_max just to fulfill the recommendation and pass the test. I prefer flexibility, give the admin the control. That means, driver would use 4x factor (or larger) whenever possible, but would not block configurations in which the 4x factor cannot be satisfied.

Oh I see.. I wasn't aware the CX7 has a limitation of the indirection table size.

There is a limitation, we read it from FW.
It's usually not small, much larger than 256.

But currently it can vary according to FW decisions in scale (resource management).

Quoted 5 lines

I wrote the test because of a similar limitation in a different NIC, but that one has been fixed.. I have limited access to CX7 NICs, the one I tested on maxed out at 63 queues so the test has passed.

Quoted 6 lines
Quoted 2 lines

Is it not possible to create an indirection table larger than 256 entries?

It is possible, depending on the exposed FW capability.
As of today, there are high-scale configurations (many VFs for example) where the FW exposed cap is lowered.

Not entirely sure what you expect the outcome of this discussion to be.

The 2x indirection table has been proven inadequate for real production use. I'm not talking about some theory or benchmarks, actual workloads reported machines/NICs with such table as unusable (workload starts choking way before reaching expected machine capacity).

That said I just checked out of curiosity and the OCP NIC spec also states:

The minimum supported indirection table size MUST be 128. The minimum SHOULD be at least 4 times the number of supported receive queues.

so I guess the 4x isn't exactly a new recommendation.

#10 lore raw
Quoted 45 lines · Jakub Kicinski
Quoted 30 lines · On Mon, 16 Feb 2026 10:28:52 +0200 Tariq Toukan wrote:
Quoted 10 lines
Quoted 10 lines

This ignores multiple other considerations:

1. Existing behavior: In general, mlx5e today implies 2x factor, so it would fail this new test.

2. Device resources: In large scale (high num of channels, or high num of netdevs on the same chip, or both), it is not obvious that increasing the indirection table size is still desirable, or even possible. To pass the selftest, you'll have to limit the max number of channels.

Quoted 8 lines
Quoted 5 lines

3. ch_max should win: Related to point #2. Driver should not enforce limitations on supported ch_max just to fulfill the recommendation and pass the test. I prefer flexibility, give the admin the control. That means, driver would use 4x factor (or larger) whenever possible, but would not block configurations in which the 4x factor cannot be satisfied.

Oh I see.. I wasn't aware the CX7 has a limitation of the indirection table size.

There is a limitation, we read it from FW.
It's usually not small, much larger than 256.

But currently it can vary according to FW decisions in scale (resource management).

Quoted 5 lines

I wrote the test because of a similar limitation in a different NIC, but that one has been fixed.. I have limited access to CX7 NICs, the one I tested on maxed out at 63 queues so the test has passed.

Quoted 6 lines
Quoted 2 lines

Is it not possible to create an indirection table larger than 256 entries?

It is possible, depending on the exposed FW capability.
As of today, there are high-scale configurations (many VFs for example) where the FW exposed cap is lowered.

Not entirely sure what you expect the outcome of this discussion to be.

The 2x indirection table has been proven inadequate for real production use. I'm not talking about some theory or benchmarks, actual workloads reported machines/NICs with such table as unusable (workload starts choking way before reaching expected machine capacity).

Quoted 4 lines

That said I just checked out of curiosity and the OCP NIC spec also states:

The minimum supported indirection table size MUST be 128.

That should be fine. Max table size cap is always >= 256.

Quoted 4 lines

The minimum
SHOULD be at least 4 times the number of supported receive queues.

so I guess the 4x isn't exactly a new recommendation.

My position is as follows:
Provide the appropriate factor *if possible* (currently it is 2x, and we will likely increase it to 4x). However, do not limit the maximum nch if the factor cannot be satisfied - even if that results in the selftest failing.

author maintainer
#11 lore raw
Quoted 10 lines · On Wed, 18 Feb 2026 10:02:01 +0200 Tariq Toukan wrote:
Quoted 4 lines

The minimum
SHOULD be at least 4 times the number of supported receive queues.

so I guess the 4x isn't exactly a new recommendation.

My position is as follows:
Provide the appropriate factor *if possible* (currently it is 2x, and we will likely increase it to 4x).
However, do not limit the maximum nch if the factor cannot be satisfied
- even if that results in the selftest failing.

Yes, that's definitely fair. We can adjust the comment in the test to make that clear. The user may want to use the queues in a different RSS context / zero-copy so limiting queue count would be the wrong direction. The test, is only signaling practicality of using all queues in the main context.