WireGuardNT, a high-performance WireGuard implementation for the Windows kernel
Hey everyone,
After many months of work, Simon and I are pleased to announce the WireGuardNT project, a native port of WireGuard to the Windows kernel. This has been a monumental undertaking, and if you've noticed that I haven't read emails in about two months, now you know why.
WireGuardNT, lower-cased as "wireguard-nt" like the other repos, began as a port of the Linux codebase, so that we could benefit from the analysis and scrutiny that that code has already received. After the initial porting efforts there succeeded, the NT codebase quickly diverged to fit well with native NTisms and NDIS (Windows networking stack) APIs. The end result is a deeply integrated and highly performant implementation of WireGuard for the NT kernel, that makes use of the full gamut of NT kernel and NDIS capabilities. You can read about the project and look at its source code here:
https://git.zx2c4.com/wireguard-nt/about/
For the Windows platform, this project is a big deal to me, as it marks the graduation of WireGuard to being a serious operating system component, meant for more serious usage. It's also a rather significant open source release, as there generally isn't so much (though there is some) open source crypto-NIC driver code already out there that does this kind of thing while pulling together various kernel capabilities in the process.
To frame what WireGuardNT is, a bit of background for how WireGuard on Windows _currently_ works, prior to this, might be in store. We currently have a cross-platform Go codebase, called wireguard-go, which uses a generic TUN driver we developed called Wintun (see wintun.net for info). The implementation lives in userspace, and shepherds packets to and from the Wintun interface. WireGuardNT will (eventually) replace that, placing all of the WireGuard protocol implementation directly into the networking stack for deeper integration, in the same way that it's done currently on Linux, OpenBSD, and FreeBSD.
With the old wireguard-go/Wintun implementation, the fact of being in userspace means that for each RX UDP packet that arrives in the kernel from the NIC and gets put in a UDP socket buffer, there's a context switch to userspace to receive it, and then a trip through the Go scheduler to decrypt it, and then it's written to Wintun's ring buffer, where it is then processed upon the next context switch. For TX, things happen in reverse: userspace sends a packet, and there's a context switch to the kernel to hand it off to Wintun, which places it into a ring buffer, and then there's another context switch to userspace, and a trip through the Go scheduler to encrypt it, and then it's sent through a socket, which involves another context switch to send it. All of the ring buffers -- Wintun's rings and Winsock's RIO rings -- amortize context switches as much as possible and make this decently fast, but all and all it still constitutes overhead and latency. WireGuardNT gets rid of all of that.
While performance is quite good right now (~7.5Gbps TX on my small test box), not a lot of effort has yet been spent on optimizing it, and there's still a lot more performance to eek out of it, I suspect, especially as we learn more about NT's scheduler and threading model particulars. Yet, by simply being in the kernel, we significantly reduce latency and do away with the context switch problems of wireguard-go/Wintun.
Most Windows users, however, don't really care what happens beyond 1Gbps, and this is where things get interesting. Windows users with an Ethernet connection generally haven't had much trouble getting close to 1Gbps or so with the old slow wireguard-go/Wintun, but over WiFi, those same users would commonly see massive slowdowns. With the significantly decreased latency of WireGuardNT, it appears that these slowdowns are no more. Jonathan Tooker reported to me that, on his system with an Intel AC9560 WiFi card, he gets ~600Mbps without WireGuard, ~600Mbps with wireguard-go/Wintun over Ethernet, ~95Mbps with wireguard-go/Wintun over WiFi, and ~600Mbps with WireGuardNT over WiFi. In other words, the WiFi performance hit from wireguard-go/Wintun has evaporated when using WireGuardNT. Power consumption, and hence battery usage, should be lower too.
And of course, on the multigig throughput side of things, Windows Server users will no doubt benefit.
The project is still at its early stages, and for now (August 2021; if you're reading this in the future this might not apply) this should be considered "experimental". There's a decent amount of new code on which I'd like to spend a bit more time scrutinizing and analyzing. And hopefully by putting the code online in an "earlier" stage of development, others might be interested in studying the source and reporting bugs in it.
Nonetheless, experimental or not, we still need people to test this and help shake out issues. To that end, WireGuardNT is now available in the ordinary WireGuard for Windows client -- https://www.wireguard.com/install/ -- with the 0.4.z series, in addition to having full support of the venerable wg(8) utility, but currently (August 2021; if you're reading this in the future this might not apply) it is behind a manually set registry knob. There will be three phases of the 0.4.z series:
Phase 1) WireGuardNT is hidden behind the "ExperimentalKernelDriver"
registry knob. If you don't manually tinker around to enable it, the client will continue to use wireguard-go/Wintun like before.
Phase 2) WireGuardNT is enabled by default and is no longer hidden.
However, in case there are late-stage problems that cause downtime for existing infrastructure, there'll be a new hidden knob called "UseUserspaceImplementation" that goes back to using wireguard-go/Wintun like before.
Phase 3) WireGuardNT is enabled, and wireguard-go/Wintun is removed from
the client. [Do note: as projects and codebases, both Wintun and wireguard-go will continue to be maintained, as they have applications and uses outside of our WireGuard client, and Wintun has uses outside of WireGuard in general.]
The leap between each phase is rather large, and I'll update this thread when each one happens. Moving from 1 to 2 will happen when things seem okay for general consumption and from 2 to 3 when we're reasonably sure there's the same level of stability. Since we don't include any telemetry in the client, a lot of this assessment will be a matter of you, mailing list readers, sending bug reports or not sending bug reports. And of course, having testers during the unstable phase 1 will be a great boon. Instructions on enabling these knobs can be found in the usual place:
https://git.zx2c4.com/wireguard-windows/about/docs/adminregistry.md
[ If you're reading this email in the future and that page either does not
exist or does not contain mention of "ExperimentalKernelDriver" or "UseUserspaceImplementation", then we have already moved to phase 3, as above, and none of this applies any more. ]
So, please do give it a whirl, check out the documentation and code, and let me know what you think. I'm looking forward to hearing your thoughts and receiving bug reports, experience reports, and overall feedback.
Enjoy!
Jason
Hi everyone,
Quoted 11 lines · Jason A. Donenfeld
Nonetheless, experimental or not, we still need people to test this and help shake out issues. To that end, WireGuardNT is now available in the ordinary WireGuard for Windows client -- https://www.wireguard.com/install/ -- with the 0.4.z series, in addition to having full support of the venerable wg(8) utility, but currently (August 2021; if you're reading this in the future this might not apply) it is behind a manually set registry knob. There will be three phases of the 0.4.z series:
Phase 1) WireGuardNT is hidden behind the "ExperimentalKernelDriver"
registry knob. If you don't manually tinker around to enable it, the client will continue to use wireguard-go/Wintun like before.
After the first few days, bug reports for WireGuardNT have died down considerably. It's hard to tell whether that's a sign that the software is stable or that nobody is using it. So, we're still in phase 1. However, with version 0.4.5 of the client, I've added a checkbox below the "activate" button in order to make the experimental testing more discoverable. I dislike such knobs, but I'd rather have more testing than less, before turning it on by default. Hopefully this will make it easier for alpha testers to give it a spin and report findings. If you have been testing the driver, and it's been working well for you, please write in to say so, simply as a signal that it is being tested. If things stay as quiet as they are now on the bug reporting front, we'll likely move into phase 2 -- enabling WireGuardNT by default, with a registry key to revert to the legacy implementation -- in 2 to 3 weeks.
Jason
Hello Jason,
thanks for your work on Wireguard.
I have just updated and activated the experimental driver. The fact that I do not need to activate it in the registry was decisive for me ;-)
I cannot report any issue so far.
Best regards,
Hendrik
------ Originalnachricht ------
Von: "Jason A. Donenfeld" <[email protected]>
An: "WireGuard mailing list" <[email protected]>
Gesendet: 13.08.2021 13:52:39
Betreff: Re: [ANNOUNCE] WireGuardNT, a high-performance WireGuard implementation for the Windows kernel
Quoted 31 lines
Hi everyone,
Quoted 11 lines · Jason A. Donenfeld
Nonetheless, experimental or not, we still need people to test this and help shake out issues. To that end, WireGuardNT is now available in the ordinary WireGuard for Windows client -- https://www.wireguard.com/install/ -- with the 0.4.z series, in addition to having full support of the venerable wg(8) utility, but currently (August 2021; if you're reading this in the future this might not apply) it is behind a manually set registry knob. There will be three phases of the 0.4.z series:
Phase 1) WireGuardNT is hidden behind the "ExperimentalKernelDriver"
registry knob. If you don't manually tinker around to enable it, the client will continue to use wireguard-go/Wintun like before.After the first few days, bug reports for WireGuardNT have died down considerably. It's hard to tell whether that's a sign that the software is stable or that nobody is using it. So, we're still in phase 1. However, with version 0.4.5 of the client, I've added a checkbox below the "activate" button in order to make the experimental testing more discoverable. I dislike such knobs, but I'd rather have more testing than less, before turning it on by default. Hopefully this will make it easier for alpha testers to give it a spin and report findings. If you have been testing the driver, and it's been working well for you, please write in to say so, simply as a signal that it is being tested. If things stay as quiet as they are now on the bug reporting front, we'll likely move into phase 2 -- enabling WireGuardNT by default, with a registry key to revert to the legacy implementation -- in 2 to 3 weeks.
Jason
Hi
I've been using it daily since 0.4.0 on a laptop and two servers in a mixed Linux/WNT/FreeBSD (pfsense) environment.
Since 0.4.1 it has worked perfectly for me.
Thanks for the effort. The tunnel is subjectively snappier than the pure Go implementation.
Best wishes
Peter Whisker
Quoted 30 lines · Jason A. Donenfeld
Hi everyone,
Quoted 11 lines · Jason A. Donenfeld
Nonetheless, experimental or not, we still need people to test this and help shake out issues. To that end, WireGuardNT is now available in the ordinary WireGuard for Windows client -- https://www.wireguard.com/install/ -- with the 0.4.z series, in addition to having full support of the venerable wg(8) utility, but currently (August 2021; if you're reading this in the future this might not apply) it is behind a manually set registry knob. There will be three phases of the 0.4.z series:
Phase 1) WireGuardNT is hidden behind the "ExperimentalKernelDriver"
registry knob. If you don't manually tinker around to enable it, the client will continue to use wireguard-go/Wintun like before.After the first few days, bug reports for WireGuardNT have died down considerably. It's hard to tell whether that's a sign that the software is stable or that nobody is using it. So, we're still in phase 1. However, with version 0.4.5 of the client, I've added a checkbox below the "activate" button in order to make the experimental testing more discoverable. I dislike such knobs, but I'd rather have more testing than less, before turning it on by default. Hopefully this will make it easier for alpha testers to give it a spin and report findings. If you have been testing the driver, and it's been working well for you, please write in to say so, simply as a signal that it is being tested. If things stay as quiet as they are now on the bug reporting front, we'll likely move into phase 2 -- enabling WireGuardNT by default, with a registry key to revert to the legacy implementation -- in 2 to 3 weeks.
Jason
I've been testing the experimental driver since the first announcement but no issues to date.
Quoted 32 lines · Jason A. Donenfeld
Hi everyone,
Quoted 11 lines · Jason A. Donenfeld
Nonetheless, experimental or not, we still need people to test this and help shake out issues. To that end, WireGuardNT is now available in the ordinary WireGuard for Windows client -- https://www.wireguard.com/install/ -- with the 0.4.z series, in addition to having full support of the venerable wg(8) utility, but currently (August 2021; if you're reading this in the future this might not apply) it is behind a manually set registry knob. There will be three phases of the 0.4.z series:
Phase 1) WireGuardNT is hidden behind the "ExperimentalKernelDriver"
registry knob. If you don't manually tinker around to enable it, the client will continue to use wireguard-go/Wintun like before.After the first few days, bug reports for WireGuardNT have died down considerably. It's hard to tell whether that's a sign that the software is stable or that nobody is using it. So, we're still in phase 1. However, with version 0.4.5 of the client, I've added a checkbox below the "activate" button in order to make the experimental testing more discoverable. I dislike such knobs, but I'd rather have more testing than less, before turning it on by default. Hopefully this will make it easier for alpha testers to give it a spin and report findings. If you have been testing the driver, and it's been working well for you, please write in to say so, simply as a signal that it is being tested. If things stay as quiet as they are now on the bug reporting front, we'll likely move into phase 2 -- enabling WireGuardNT by default, with a registry key to revert to the legacy implementation -- in 2 to 3 weeks.
Jason
Signature · 4 lines
Phillip McMahon https://flowcrypt.com/me/phillipmcmahon OpenPGP: EA04 83D4 C864 AA7C 1099 4BE6 A11E 70AD FDA6 0CF9
Den 13-08-2021 kl. 13:52 skrev Jason A. Donenfeld:
I have used it since first announcement. Not experienced any problems.
--
Morten Christensen
Quoted 30 lines
Hi everyone,
Quoted 11 lines · Jason A. Donenfeld
Nonetheless, experimental or not, we still need people to test this and help shake out issues. To that end, WireGuardNT is now available in the ordinary WireGuard for Windows client -- https://www.wireguard.com/install/ -- with the 0.4.z series, in addition to having full support of the venerable wg(8) utility, but currently (August 2021; if you're reading this in the future this might not apply) it is behind a manually set registry knob. There will be three phases of the 0.4.z series:
Phase 1) WireGuardNT is hidden behind the "ExperimentalKernelDriver"
registry knob. If you don't manually tinker around to enable it, the client will continue to use wireguard-go/Wintun like before.After the first few days, bug reports for WireGuardNT have died down considerably. It's hard to tell whether that's a sign that the software is stable or that nobody is using it. So, we're still in phase 1. However, with version 0.4.5 of the client, I've added a checkbox below the "activate" button in order to make the experimental testing more discoverable. I dislike such knobs, but I'd rather have more testing than less, before turning it on by default. Hopefully this will make it easier for alpha testers to give it a spin and report findings. If you have been testing the driver, and it's been working well for you, please write in to say so, simply as a signal that it is being tested. If things stay as quiet as they are now on the bug reporting front, we'll likely move into phase 2 -- enabling WireGuardNT by default, with a registry key to revert to the legacy implementation -- in 2 to 3 weeks.
Jason
Quoted 4 lines · Jason A. Donenfeld
this will make it easier for alpha testers to give it a spin and report findings. If you have been testing the driver, and it's been working well for you, please write in to say so, simply as a signal that it is being tested.
No issues on Windows 8.1 so far, using the NT kernel version from 0.4.5.
Signature · 1 line
Mark Lawrence
Hi again,
Quoted 16 lines · Jason A. Donenfeld
Hi everyone,
Quoted 11 lines · Jason A. Donenfeld
Nonetheless, experimental or not, we still need people to test this and help shake out issues. To that end, WireGuardNT is now available in the ordinary WireGuard for Windows client -- https://www.wireguard.com/install/ -- with the 0.4.z series, in addition to having full support of the venerable wg(8) utility, but currently (August 2021; if you're reading this in the future this might not apply) it is behind a manually set registry knob. There will be three phases of the 0.4.z series:
Phase 1) WireGuardNT is hidden behind the "ExperimentalKernelDriver"
registry knob. If you don't manually tinker around to enable it, the client will continue to use wireguard-go/Wintun like before.
Quoted 14 lines
After the first few days, bug reports for WireGuardNT have died down considerably. It's hard to tell whether that's a sign that the software is stable or that nobody is using it. So, we're still in phase 1. However, with version 0.4.5 of the client, I've added a checkbox below the "activate" button in order to make the experimental testing more discoverable. I dislike such knobs, but I'd rather have more testing than less, before turning it on by default. Hopefully this will make it easier for alpha testers to give it a spin and report findings. If you have been testing the driver, and it's been working well for you, please write in to say so, simply as a signal that it is being tested. If things stay as quiet as they are now on the bug reporting front, we'll likely move into phase 2 -- enabling WireGuardNT by default, with a registry key to revert to the legacy implementation -- in 2 to 3 weeks.
We're still technically in phase 1, but with the version that should be coming out later today, users who have enrolled in the Windows Insider Program have "ExperimentalKernelDriver" set to always on, unless the "IgnoreInsiderProgram" admin knob is set. In other words, we're gradually ramping up testing coverage. I still hope to move into phase 2 soon, but this seemed like a cautious intermediate step.
Regards,
Jason
Hi everyone,
As described in the original announcement:
Quoted 6 lines
There will be three phases of the 0.4.z series:
Phase 1) WireGuardNT is hidden behind the "ExperimentalKernelDriver"
registry knob. If you don't manually tinker around to enable it, the client will continue to use wireguard-go/Wintun like before.
Quoted 8 lines
Phase 2) WireGuardNT is enabled by default and is no longer hidden.
However, in case there are late-stage problems that cause downtime for existing infrastructure, there'll be a new hidden knob called "UseUserspaceImplementation" that goes back to using wireguard-go/Wintun like before.Phase 3) WireGuardNT is enabled, and wireguard-go/Wintun is removed from
the client.
After quite a bit of testing, the bug reports have dried up. We've gone through several subphases of phase 1, from being masked behind a registry knob, to introducing a checkbox, to opting in users part of the Windows Insider program. There's been a pretty heavy amount of testing, from what I can gather, and I think things have shaped up nicely.
Finally, tomorrow with the release of v0.4.8, we will move onto phase
2. That means that WireGuardNT will be on by default. If something goes wrong, you will still be able to disable it and revert to the older wireguard-go/Wintun using the "UseUserspaceImplementation" switch described on this page (as of September 2021; will be removed in the future):
https://git.zx2c4.com/wireguard-windows/about/docs/adminregistry.md
But note that if you /do/ revert to using wireguard-go/Wintun, you *must* absolutely email [email protected] with details as to why you've done so, so that if there is a bug, we can fix it. If you do not do this, it is possible that your bug will never be fixed, and in a month or so, we'll move onto phase 3, and you will lose the ability to revert to using wireguard-go/Wintun. Do not rely on other people to report your bug for you; Windows is weird and diverse and there are no guarantees that somebody else will run into your same problem. So
please: if you enable "UseUserspaceImplementation", send an email saying why.
Depending on how Monday morning goes, this should be released in around 12 hours from now. Enjoy!
Regards,
Jason
Hi Jason,
Quoted 10 lines · Jason A. Donenfeld
...
Finally, tomorrow with the release of v0.4.8, we will move onto phase
2. That means that WireGuardNT will be on by default. If something goes wrong, you will still be able to disable it and revert to the older wireguard-go/Wintun using the "UseUserspaceImplementation" switch described on this page (as of September 2021; will be removed in the future):https://git.zx2c4.com/wireguard-windows/about/docs/adminregistry.md
Quoted 10 lines
But note that if you /do/ revert to using wireguard-go/Wintun, you *must* absolutely email [email protected] with details as to why you've done so, so that if there is a bug, we can fix it. If you do not do this, it is possible that your bug will never be fixed, and in a month or so, we'll move onto phase 3, and you will lose the ability to revert to using wireguard-go/Wintun. Do not rely on other people to report your bug for you; Windows is weird and diverse and there are no guarantees that somebody else will run into your same problem. So
please: if you enable "UseUserspaceImplementation", send an email saying why.
One month to move into the next phase may be a bit tight for some folks. 30 days is probably fine for a developer or standalone installation, but some organizations cannot move that fast.
I've worked in US Financial and US Federal, and some changes take longer to approve. Some organizations have processes in place that require approvals from management. It may take months to get a Change Control Request approved.
When I worked at Treasury a trivial change could take two or three months and it required management signoffs and complete testing before being released to the production network. Nearly everyone dreaded a Change Control Request.
It may be noteworthy... on Windows OSes, the trend is to move stuff out of the kernel and into userspace to reduce risk. For example, Microsoft moved parts of the GDI out of the kernel and into userspace. So some folks may actually want the userland architecture to reduce risk.
Jeff
Hello Jeff,
Quoted 9 lines · Jeffrey Walton
One month to move into the next phase may be a bit tight for some folks. 30 days is probably fine for a developer or standalone installation, but some organizations cannot move that fast.
I've worked in US Financial and US Federal, and some changes take longer to approve. Some organizations have processes in place that require approvals from management. It may take months to get a Change Control Request approved.
Quoted 4 lines
When I worked at Treasury a trivial change could take two or three months and it required management signoffs and complete testing before being released to the production network. Nearly everyone dreaded a Change Control Request.
I'm not sure I really follow this line of reasoning here. You've divided the into two categories:
A) "a developer or standalone installation" that is presumably frequently upgrading in order to have the latest in greatest.
B) "US Financial and US Federal" and similar organizations for whom changes require extensive paperwork and reliability testing.
For group (A), the gradual three phase rollout outlined in the initial email works very well, as it means each component gets an appropriate amount of testing depending on the phase, with related rollback controls. It's group (B) that you've raised a concern about. But, if each change for (B) requires paperwork and reliability testing, wouldn't _that_ procedure also catch issues? Obviously a downside of not updating like (A) does is that you miss timely security updates, but I assume that (B) argues that their slow processes of sign-offs and testing and paperwork increases reliability and accountability, which may well be more important for (B). And it would seem that those processes basically accomplish the same thing as the three phase plan that's available to group (A) does. So it doesn't seem like anybody is left out. Unless, of course, all those sign-offs and testing aren't _actually_ testing anything meaningful, and they're just a Brazilesque paperwork nightmare, in which case, with regard for neither security nor reliability, what does it matter either way?
Anyway, there's no exact timeline of precisely 30 days. It will be a good time after nobody has reported a bug. While products have "go to market" deadlines, projects have the luxury of being on the "when it's ready" schedule, which means we don't need to rush. At the same time, I very much look forward to the reduced maintenance burden of no longer maintaining duplicate code paths in the wireguard-windows repo. The jd/remove-wintun branch has a commit that removes about a thousand lines, which isn't _that_ much, but those are lines that are a bit stressful in the sense that abstraction layers and duplicated code paths are common sources of bugs, so they require extra vigilance from me (and auditors).
Quoted 5 lines
It may be noteworthy... on Windows OSes, the trend is to move stuff out of the kernel and into userspace to reduce risk. For example, Microsoft moved parts of the GDI out of the kernel and into userspace. So some folks may actually want the userland architecture to reduce risk.
I don't think it actually makes a practical difference in this case:
the userspace tunnel service retained SeLoadDriverPrivilege so that it could *unload* Wintun when quitting. Besides, WireGuard was made to be implemented in operating system kernels; figuring out how to do that in a minimally scary way was an original thrust of the project. Linux, OpenBSD, FreeBSD, and now NT.
Jason
Hi folks,
As described in August:
Quoted 5 lines · Jason A. Donenfeld
Phase 3) WireGuardNT is enabled, and wireguard-go/Wintun is removed from
the client. [Do note: as projects and codebases, both Wintun and wireguard-go will continue to be maintained, as they have applications and uses outside of our WireGuard client, and Wintun has uses outside of WireGuard in general.]
After a considerable amount of testing and 11 releases, we are now in phase 3, and the transition is complete. WireGuard for Windows v0.5 is being released as we speak, and contains WireGuardNT 0.10.1, which is now fully WHQL Certified and will run on all flavors of Windows Server.
Thank you very much to everyone who helped test and stabilize this code.
Regards,
Jason