virtio_net: enable napi_tx by default
Subsystems (3)
PATCH 1/1 virtio_net: enable napi_tx by default
From: Willem de Bruijn <[email protected]>
NAPI tx mode improves TCP behavior by enabling TCP small queues (TSQ). TSQ reduces queuing ("bufferbloat") and burstiness.
Previous measurements have shown significant improvement for TCP_STREAM style workloads. Such as those in commit 86a5df1495cc ("Merge branch 'virtio-net-tx-napi'").
There has been uncertainty about smaller possible regressions in latency due to increased reliance on tx interrupts.
The above results did not show that, nor did I observe this when rerunning TCP_RR on Linux 5.1 this week on a pair of guests in the same rack. This may be subject to other settings, notably interrupt coalescing.
In the unlikely case of regression, we have landed a credible runtime solution. Ethtool can configure it with -C tx-frames [0|1] as of commit 0c465be183c7 ("virtio_net: ethtool tx napi configuration").
NAPI tx mode has been the default in Google Container-Optimized OS (COS) for over half a year, as of release M70 in October 2018, without any negative reports.
---
now that we have ethtool support and real production deployment, it seemed like a good time to revisit this discussion.
--- drivers/net/virtio_net.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/virtio_net.c +1 -1
@@ -26,7 +26,7 @@static int napi_weight =NAPI_POLL_WEIGHT ;module_param (napi_weight, int,0444 ); -static bool csum = true, gso = true, napi_tx; +static bool csum = true, gso = true, napi_tx = true;module_param (csum, bool,0444 );module_param (gso, bool,0444 );module_param (napi_tx, bool,0644 );
Signature · 1 line
2.22.0.rc2.383.gf4fbbf30c2-goog
Quoted 25 lines · Willem de Bruijn
From: Willem de Bruijn <[email protected]>
NAPI tx mode improves TCP behavior by enabling TCP small queues (TSQ). TSQ reduces queuing ("bufferbloat") and burstiness.
Previous measurements have shown significant improvement for
TCP_STREAMstyle workloads. Such as those in commit86a5df1495cc("Merge branch 'virtio-net-tx-napi'").There has been uncertainty about smaller possible regressions in latency due to increased reliance on tx interrupts.
The above results did not show that, nor did I observe this when rerunning
TCP_RRon Linux 5.1 this week on a pair of guests in the same rack. This may be subject to other settings, notably interrupt coalescing.In the unlikely case of regression, we have landed a credible runtime solution. Ethtool can configure it with -C tx-frames [0|1] as of commit
0c465be183c7("virtio_net: ethtool tx napi configuration").NAPI tx mode has been the default in Google Container-Optimized OS (COS) for over half a year, as of release M70 in October 2018, without any negative reports.
Quoted 8 lines
Signed-off-by Willem de Bruijn <[email protected]>---
now that we have ethtool support and real production deployment, it seemed like a good time to revisit this discussion.
I agree to enable it by default. Need inputs from Michael. One possible issue is we may get some regression on the old machine without APICV, but consider most modern CPU has this feature, it probably doesn't matter.
Thanks
Quoted 18 lines
--- drivers/net/virtio_net.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/net/virtio_net.c +1
-1@@ -26,7 +26,7 @@static int napi_weight =NAPI_POLL_WEIGHT ;module_param (napi_weight, int,0444 ); -static bool csum = true, gso = true, napi_tx; +static bool csum = true, gso = true, napi_tx = true;module_param (csum, bool,0444 );module_param (gso, bool,0444 );module_param (napi_tx, bool,0644 );
Michael S. Tsirkin
maintainerQuoted 37 lines · Jason Wang
Quoted 25 lines · Willem de Bruijn
From: Willem de Bruijn <[email protected]>
NAPI tx mode improves TCP behavior by enabling TCP small queues (TSQ). TSQ reduces queuing ("bufferbloat") and burstiness.
Previous measurements have shown significant improvement for
TCP_STREAMstyle workloads. Such as those in commit86a5df1495cc("Merge branch 'virtio-net-tx-napi'").There has been uncertainty about smaller possible regressions in latency due to increased reliance on tx interrupts.
The above results did not show that, nor did I observe this when rerunning
TCP_RRon Linux 5.1 this week on a pair of guests in the same rack. This may be subject to other settings, notably interrupt coalescing.In the unlikely case of regression, we have landed a credible runtime solution. Ethtool can configure it with -C tx-frames [0|1] as of commit
0c465be183c7("virtio_net: ethtool tx napi configuration").NAPI tx mode has been the default in Google Container-Optimized OS (COS) for over half a year, as of release M70 in October 2018, without any negative reports.
Quoted 8 lines
Signed-off-by Willem de Bruijn <[email protected]>---
now that we have ethtool support and real production deployment, it seemed like a good time to revisit this discussion.
Quoted 6 lines
I agree to enable it by default. Need inputs from Michael. One possible issue is we may get some regression on the old machine without APICV, but consider most modern CPU has this feature, it probably doesn't matter.
Thanks
Right. If the issue does arise we can always add e.g. a feature flag to control the default from the host.
Quoted 17 lines
Quoted 17 lines
--- drivers/net/virtio_net.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/net/virtio_net.c +1
-1@@ -26,7 +26,7 @@static int napi_weight =NAPI_POLL_WEIGHT ;module_param (napi_weight, int,0444 ); -static bool csum = true, gso = true, napi_tx; +static bool csum = true, gso = true, napi_tx = true;module_param (csum, bool,0444 );module_param (gso, bool,0444 );module_param (napi_tx, bool,0644 );
Jason Wang
Acked-byQuoted 36 lines · Michael S. Tsirkin
Quoted 35 lines · Jason Wang
Quoted 33 lines · Willem de Bruijn
From: Willem de Bruijn <[email protected]>
NAPI tx mode improves TCP behavior by enabling TCP small queues (TSQ). TSQ reduces queuing ("bufferbloat") and burstiness.
Previous measurements have shown significant improvement for
TCP_STREAMstyle workloads. Such as those in commit86a5df1495cc("Merge branch 'virtio-net-tx-napi'").There has been uncertainty about smaller possible regressions in latency due to increased reliance on tx interrupts.
The above results did not show that, nor did I observe this when rerunning
TCP_RRon Linux 5.1 this week on a pair of guests in the same rack. This may be subject to other settings, notably interrupt coalescing.In the unlikely case of regression, we have landed a credible runtime solution. Ethtool can configure it with -C tx-frames [0|1] as of commit
0c465be183c7("virtio_net: ethtool tx napi configuration").NAPI tx mode has been the default in Google Container-Optimized OS (COS) for over half a year, as of release M70 in October 2018, without any negative reports.
Signed-off-by Willem de Bruijn <[email protected]>---
now that we have ethtool support and real production deployment, it seemed like a good time to revisit this discussion.
Quoted 9 lines
Quoted 6 lines
I agree to enable it by default. Need inputs from Michael. One possible issue is we may get some regression on the old machine without APICV, but consider most modern CPU has this feature, it probably doesn't matter.
Thanks
Right. If the issue does arise we can always add e.g. a feature flag to control the default from the host.
Yes.
So
Quoted 21 lines · Willem de Bruijn
From: Willem de Bruijn <[email protected]>
NAPI tx mode improves TCP behavior by enabling TCP small queues (TSQ). TSQ reduces queuing ("bufferbloat") and burstiness.
Previous measurements have shown significant improvement for
TCP_STREAMstyle workloads. Such as those in commit86a5df1495cc("Merge branch 'virtio-net-tx-napi'").There has been uncertainty about smaller possible regressions in latency due to increased reliance on tx interrupts.
The above results did not show that, nor did I observe this when rerunning
TCP_RRon Linux 5.1 this week on a pair of guests in the same rack. This may be subject to other settings, notably interrupt coalescing.In the unlikely case of regression, we have landed a credible runtime solution. Ethtool can configure it with -C tx-frames [0|1] as of commit
0c465be183c7("virtio_net: ethtool tx napi configuration").
Quoted 7 lines
NAPI tx mode has been the default in Google Container-Optimized OS (COS) for over half a year, as of release M70 in October 2018, without any negative reports.
Quoted 24 lines
---
now that we have ethtool support and real production deployment, it seemed like a good time to revisit this discussion.
--- drivers/net/virtio_net.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/net/virtio_net.c +1
-1@@ -26,7 +26,7 @@static int napi_weight =NAPI_POLL_WEIGHT ;module_param (napi_weight, int,0444 ); -static bool csum = true, gso = true, napi_tx; +static bool csum = true, gso = true, napi_tx = true;module_param (csum, bool,0444 );module_param (gso, bool,0444 );module_param (napi_tx, bool,0644 );Signature · 1 line
2.22.0.rc2.383.gf4fbbf30c2-goog
From: Willem de Bruijn <[email protected]>
Date: Thu, 13 Jun 2019 12:24:57 -0400
Quoted 21 lines
From: Willem de Bruijn <[email protected]>
NAPI tx mode improves TCP behavior by enabling TCP small queues (TSQ). TSQ reduces queuing ("bufferbloat") and burstiness.
Previous measurements have shown significant improvement for
TCP_STREAMstyle workloads. Such as those in commit86a5df1495cc("Merge branch 'virtio-net-tx-napi'").There has been uncertainty about smaller possible regressions in latency due to increased reliance on tx interrupts.
The above results did not show that, nor did I observe this when rerunning
TCP_RRon Linux 5.1 this week on a pair of guests in the same rack. This may be subject to other settings, notably interrupt coalescing.In the unlikely case of regression, we have landed a credible runtime solution. Ethtool can configure it with -C tx-frames [0|1] as of commit
0c465be183c7("virtio_net: ethtool tx napi configuration").
Quoted 7 lines
NAPI tx mode has been the default in Google Container-Optimized OS (COS) for over half a year, as of release M70 in October 2018, without any negative reports.
Applied.