mlx4_en: TX ring size default to 1024
Subsystems (3)
PATCH 1/3 mlx4_en: TX ring size default to 1024
--- drivers/net/ethernet/mellanox/mlx4/mlx4_en.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
drivers/net/ethernet/mellanox/mlx4/mlx4_en.h +1 -1
@@ -110,7 +110,7 @@ enum {#define MLX4_EN_NUM_TX_RINGS 8#define MLX4_EN_NUM_PPP_RINGS 8#define MAX_TX_RINGS (MLX4_EN_NUM_TX_RINGS + MLX4_EN_NUM_PPP_RINGS) -#define MLX4_EN_DEF_TX_RING_SIZE 512 +#define MLX4_EN_DEF_TX_RING_SIZE 1024#define MLX4_EN_DEF_RX_RING_SIZE 1024/* Target number of packets to coalesce with interrupt moderation */
Signature · 1 line
1.5.4.3
David Miller
maintainerFrom: Yevgeny Petrilin <[email protected]>
Date: Thu, 23 Feb 2012 15:34:05 +0200
Quoted 1 line
Signed-off-by Yevgeny Petrilin <[email protected]>
This is rediculious as a default, yes even for 10Gb.
Do you have any idea how high latency is going to be for packets trying to get into the transmit queue if there are already a thousand other frames in there?
Quoted 10 lines · David Miller
From: Yevgeny Petrilin <[email protected]>
Date: Thu, 23 Feb 2012 15:34:05 +0200Quoted 1 line
Signed-off-by Yevgeny Petrilin <[email protected]>This is rediculious as a default, yes even for 10Gb.
Do you have any idea how high latency is going to be for packets trying to get into the transmit queue if there are already a thousand other frames in there?
Before increasing TX ring sizes, a driver should implement BQL as a prereq.
Quoted 7 lines
Quoted 1 line
Signed-off-by Yevgeny Petrilin <[email protected]>This is rediculious as a default, yes even for 10Gb.
Do you have any idea how high latency is going to be for packets trying to get into the transmit queue if there are already a thousand other frames in there?
On the other hand, when having smaller queue with 1000 in-flight packets would mean queue would be stopped, how is it better?
Having bigger TX ring helps dealing better with bursts of TX packets, without the overhead of stopping and starting the queue, It also makes sense to have same size TX and RX queues, for example in case of traffic being forwarded from TX to RX.
I did find number of 10Gb vendors that have 1024 or more as the default size for TX queue.
Thanks,
Yevgeny
David Miller
maintainerFrom: Yevgeny Petrilin <[email protected]>
Date: Fri, 24 Feb 2012 19:35:45 +0000
Quoted 2 lines
On the other hand, when having smaller queue with 1000 in-flight packets would mean queue would be stopped, how is it better?
It's a thousand times better.
Because if a high priority packet gets queued up it won't have to wait for 1024 packets to hit the wire before it can go out.
You need to support byte queue limits before you jack things up so high like this, otherwise high priority packets are absolutely pointless and unusable.
Quoted 10 lines · Yevgeny Petrilin
Quoted 7 lines
Quoted 1 line
Signed-off-by Yevgeny Petrilin <[email protected]>This is rediculious as a default, yes even for 10Gb.
Do you have any idea how high latency is going to be for packets trying to get into the transmit queue if there are already a thousand other frames in there?
On the other hand, when having smaller queue with 1000 in-flight packets would mean queue would be stopped, how is it better?
Its better because you can have any kind of Qdisc setup to properly classify packets, with 100.000 total packets in queues if you wish.
TX ring is a single FIFO, and that is just horrible, especially with big packets...
Quoted 3 lines
Having bigger TX ring helps dealing better with bursts of TX packets, without the overhead of stopping and starting the queue, It also makes sense to have same size TX and RX queues, for example in case of traffic being forwarded from TX to RX.
Really I doubt people using forwarding setups use default qdiscs.
Instead of bigger TX rings, they need appropriate Qdiscs.
Quoted 1 line
I did find number of 10Gb vendors that have 1024 or more as the default size for TX queue.
Thats a shame.
Quoted 8 lines · Eric Dumazet
Quoted 7 lines · Yevgeny Petrilin
Quoted 7 lines
Quoted 1 line
Signed-off-by Yevgeny Petrilin <[email protected]>This is rediculious as a default, yes even for 10Gb.
Do you have any idea how high latency is going to be for packets trying to get into the transmit queue if there are already a thousand other frames in there?
For a GigE NIC with a typical ring size of 256, the serialization delay for 256 1500 byte packets is:
1500*8*256/10^9 = ~3.1 msec
For a 10-GigE NIC with a ring size of 1024, the serialization delay for 1024 1500 byte packets is:
1500*8*1024/10^10 = ~1.2 msec
So it's not immediately clear that a ring size of 1024 is unreasonable for 10-GigE.
It probably boils down to whether the default setting should be biased more toward low latency applications or high throughput bulk data applications. Determining the best happy medium is best decided by appropriate benchmark testing. Of course, anyone can change the settings to suit their purpose, so it's really just a question of what's best for the "usual" case.
Quoted 5 lines
Quoted 2 lines
On the other hand, when having smaller queue with 1000 in-flight packets would mean queue would be stopped, how is it better?
Its better because you can have any kind of Qdisc setup to properly classify packets, with 100.000 total packets in queues if you wish.
Not everyone wants to deal with the convoluted, arcane, and poorly documented qdisc machinery, especially with its current limitations at 10-GigE (or faster) line rates.
Quoted 6 lines
TX ring is a single FIFO, and that is just horrible, especially with big packets...
Quoted 2 lines
Having bigger TX ring helps dealing better with bursts of TX packets, without the overhead of stopping and starting the queue, It also makes sense to have same size TX and RX queues, for example in case of traffic being forwarded from TX to RX.
Really I doubt people using forwarding setups use default qdiscs.
I don't think it's necessarily that uncommon, such as a simple 10-GigE firewall setup.
Quoted 5 lines
Instead of bigger TX rings, they need appropriate Qdiscs.
Quoted 1 line
I did find number of 10Gb vendors that have 1024 or more as the default size for TX queue.
Thats a shame.
-Bill
Quoted 5 lines · Bill Fink
For a GigE NIC with a typical ring size of 256, the serialization delay for 256 1500 byte packets is:
1500*8*256/10^9 = ~3.1 msec
Quoted 8 lines
For a 10-GigE NIC with a ring size of 1024, the serialization delay for 1024 1500 byte packets is:
1500*8*1024/10^10 = ~1.2 msecSo it's not immediately clear that a ring size of 1024 is unreasonable for 10-GigE.
Its clear when you take into account packets of 64Kbytes (TSO)
With current hardware and state of linux software, you dont need anymore very big NIC queues since they bring known drawbacks.
It was true in the past with UP and some timer handlers that could hog cpu for long periods of time, and when TSO didnt exist.
Hopefully all these cpu hogs are not running in softirq handlers anymore.
If your workload needs more than ~500 slots, then something is wrong elsewhere and should be fixed. No more workarounds please.
Now BQL (Byte Queue Limits) is available, a driver should implement it first before considering big TX rings. Thats a 20 minutes change.