Skip to contentSkip to conversation

wireguard updates for 6.16, part 2, late

PATCH Applied Jason A. Donenfeld 3 participants 7 messages May 30, 2025 – Jun 5, 2025

Subsystems (3)
lore
Share & tools
lore

Jason A. Donenfeld

author maintainer
#0 lore raw

cover wireguard updates for 6.16, part 2, late

Hey Jakub/Paolo,

This one patch missed the cut off of the series I sent last week for net-next. It's a oneliner, almost trivial, and I suppose it could be a "net" patch, but we're still in the merge window. I was hoping that if you're planning on doing a net-next part 2 pull, you might include this. If not, I'll send it later in 6.16 as a "net" patch.

Thanks,
Jason

Mirco Barone (1):
wireguard: device: enable threaded NAPI

 drivers/net/wireguard/device.c | 1 +
 1 file changed, 1 insertion(+)
Signature · 1 line
2.48.1
author maintainer
#1 lore raw

PATCH 1/1 wireguard: device: enable threaded NAPI

From: Mirco Barone <[email protected]>

Enable threaded NAPI by default for WireGuard devices in response to low performance behavior that we observed when multiple tunnels (and thus multiple wg devices) are deployed on a single host. This affects any kind of multi-tunnel deployment, regardless of whether the tunnels share the same endpoints or not (i.e., a VPN concentrator type of gateway would also be affected).

The problem is caused by the fact that, in case of a traffic surge that involves multiple tunnels at the same time, the polling of the NAPI instance of all these wg devices tends to converge onto the same core, causing underutilization of the CPU and bottlenecking performance.

This happens because NAPI polling is hosted by default in softirq context, but the WireGuard driver only raises this softirq after the rx peer queue has been drained, which doesn't happen during high traffic. In this case, the softirq already active on a core is reused instead of raising a new one.

As a result, once two or more tunnel softirqs have been scheduled on the same core, they remain pinned there until the surge ends.

In our experiments, this almost always leads to all tunnel NAPIs being handled on a single core shortly after a surge begins, limiting scalability to less than 3× the performance of a single tunnel, despite plenty of unused CPU cores being available.

The proposed mitigation is to enable threaded NAPI for all WireGuard devices. This moves the NAPI polling context to a dedicated per-device kernel thread, allowing the scheduler to balance the load across all available cores.

On our 32-core gateways, enabling threaded NAPI yields a ~4× performance improvement with 16 tunnels, increasing throughput from ~13 Gbps to ~48 Gbps. Meanwhile, CPU usage on the receiver (which is the bottleneck) jumps from 20% to 100%.

We have found no performance regressions in any scenario we tested. Single-tunnel throughput remains unchanged.

More details are available in our Netdev paper.

---
 drivers/net/wireguard/device.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/wireguard/device.c +1 -0
index 3ffeeba5dccf..4a529f1f9bea 100644
@@ -366,6 +366,7 @@ static int wg_newlink(struct net_device *dev,
 	if (ret < 0)
 		goto err_free_handshake_queue;
 
+	dev_set_threaded(dev, true);
 	ret = register_netdevice(dev);
 	if (ret < 0)
 		goto err_uninit_ratelimiter;
Signature · 1 line
2.48.1
maintainer
#2 lore raw
In reply to Jason A. Donenfeld
Quoted 5 lines · Jason A. Donenfeld

This one patch missed the cut off of the series I sent last week for net-next. It's a oneliner, almost trivial, and I suppose it could be a "net" patch, but we're still in the merge window. I was hoping that if you're planning on doing a net-next part 2 pull, you might include this. If not, I'll send it later in 6.16 as a "net" patch.

We usually (always AFAIR) send a single PR for net-next, mostly because there is no additional material due to net-next being closed in the merge window.

Anyhow I can apply directly this patch to the net tree and it will be included in this week net PR.

Side note: no need for the cover letter for a single patch series, unless it's a formal PR.

Cheers,

Paolo

maintainer
#3 lore raw
Quoted 7 lines · Paolo Abeni
Quoted 5 lines · Jason A. Donenfeld

This one patch missed the cut off of the series I sent last week for net-next. It's a oneliner, almost trivial, and I suppose it could be a "net" patch, but we're still in the merge window. I was hoping that if you're planning on doing a net-next part 2 pull, you might include this. If not, I'll send it later in 6.16 as a "net" patch.

Quoted 6 lines

We usually (always AFAIR) send a single PR for net-next, mostly because there is no additional material due to net-next being closed in the merge window.

Anyhow I can apply directly this patch to the net tree and it will be included in this week net PR.

I'm sorry, I rushed my reply a bit. Could you please provide a suitable
Fixes: tag for this patch?

Thanks,

Paolo

author maintainer
#4 lore raw

PATCH v2 1/1 wireguard: device: enable threaded NAPI

From: Mirco Barone <[email protected]>

Enable threaded NAPI by default for WireGuard devices in response to low performance behavior that we observed when multiple tunnels (and thus multiple wg devices) are deployed on a single host. This affects any kind of multi-tunnel deployment, regardless of whether the tunnels share the same endpoints or not (i.e., a VPN concentrator type of gateway would also be affected).

The problem is caused by the fact that, in case of a traffic surge that involves multiple tunnels at the same time, the polling of the NAPI instance of all these wg devices tends to converge onto the same core, causing underutilization of the CPU and bottlenecking performance.

This happens because NAPI polling is hosted by default in softirq context, but the WireGuard driver only raises this softirq after the rx peer queue has been drained, which doesn't happen during high traffic. In this case, the softirq already active on a core is reused instead of raising a new one.

As a result, once two or more tunnel softirqs have been scheduled on the same core, they remain pinned there until the surge ends.

In our experiments, this almost always leads to all tunnel NAPIs being handled on a single core shortly after a surge begins, limiting scalability to less than 3× the performance of a single tunnel, despite plenty of unused CPU cores being available.

The proposed mitigation is to enable threaded NAPI for all WireGuard devices. This moves the NAPI polling context to a dedicated per-device kernel thread, allowing the scheduler to balance the load across all available cores.

On our 32-core gateways, enabling threaded NAPI yields a ~4× performance improvement with 16 tunnels, increasing throughput from ~13 Gbps to ~48 Gbps. Meanwhile, CPU usage on the receiver (which is the bottleneck) jumps from 20% to 100%.

We have found no performance regressions in any scenario we tested. Single-tunnel throughput remains unchanged.

More details are available in our Netdev paper.

---
Changes v1->v2:
- Add Fixes tag.

 drivers/net/wireguard/device.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/wireguard/device.c +1 -0
index 3ffeeba5dccf..4a529f1f9bea 100644
@@ -366,6 +366,7 @@ static int wg_newlink(struct net_device *dev,
 	if (ret < 0)
 		goto err_free_handshake_queue;
 
+	dev_set_threaded(dev, true);
 	ret = register_netdevice(dev);
 	if (ret < 0)
 		goto err_uninit_ratelimiter;
Signature · 1 line
2.48.1
#5 lore raw

Re: [PATCH net-next v2 1/1] wireguard: device: enable threaded NAPI

This patch was applied to netdev/net.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This patch was applied to netdev/net.git (main)
by Jakub Kicinski <[email protected]>:

Quoted 10 lines · On Thu, 5 Jun 2025 14:06:16 +0200 you wrote:

From: Mirco Barone <[email protected]>

Enable threaded NAPI by default for WireGuard devices in response to low performance behavior that we observed when multiple tunnels (and thus multiple wg devices) are deployed on a single host. This affects any kind of multi-tunnel deployment, regardless of whether the tunnels share the same endpoints or not (i.e., a VPN concentrator type of gateway would also be affected).

[...]

Here is the summary with links:
- [net-next,v2,1/1] wireguard: device: enable threaded NAPI

    https://git.kernel.org/netdev/net/c/db9ae3b6b43c

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
#6 lore raw
In reply to Jason A. Donenfeld
This patch was applied to netdev/net.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This patch was applied to netdev/net.git (main)
by Jakub Kicinski <[email protected]>:

Quoted 9 lines · On Fri, 30 May 2025 05:04:57 +0200 you wrote:

Hey Jakub/Paolo,

This one patch missed the cut off of the series I sent last week for net-next. It's a oneliner, almost trivial, and I suppose it could be a "net" patch, but we're still in the merge window. I was hoping that if you're planning on doing a net-next part 2 pull, you might include this. If not, I'll send it later in 6.16 as a "net" patch.

[...]

Here is the summary with links:
- [net-next,1/1] wireguard: device: enable threaded NAPI

    https://git.kernel.org/netdev/net/c/db9ae3b6b43c

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html