WireGuard for FreeBSD in development for 13.y – and a note of how we got here
Hi everybody,
I’m pleased to announce that WireGuard now runs inside the FreeBSD kernel, with a driver called if_wg. It has full support of wg(8) and wg-quick(8) [5], as well as general integration into FreeBSD userland. Performance should be decent. The implementation in FreeBSD’s main branch should pretty much work, though it’s something of a so-so work in progress. To learn what I mean there, read on…
Sometime ago, a popular firewall vendor tasked a developer with writing a WireGuard implementation for FreeBSD. They didn’t bother reaching out to the project. That’s okay, I figured, I’ll reach out and see if I can help and coordinate. What followed over the next year was a series of poor communications – messages unanswered, code reviews ignored, that kind of thing. Usually collaborations I’ve had with others have been full of excitement, but it just didn’t work out here. In the few discussions we were able to have, I did get across some key points, like, “you’ll save a bunch of time if you use the OpenBSD code as a starting point.” But mostly it seemed like a stop-and-go effort that the WireGuard project didn’t have much to do with. Then, at some point, whatever code laying around got merged into the FreeBSD tree and the developer tasked with writing it moved on.
Fortunately, two weeks before FreeBSD 13.0 was due to be released, FreeBSD core developer Kyle Evans emailed the list about integrating wireguard-tools (wg(8) and such). In the ensuing discussion I mentioned that we really need to get the actual if_wg kernel implementation up to snuff. We took the conversation to IRC, and agreed that we should work on figuring out what to do before the release date. At the same time, Matt Dunwoodie, who worked on the OpenBSD implementation, also took a look at what had become of that implementation in FreeBSD. Over the next week, the three of us dug in and completely reworked the implementation from top to bottom, each one of us pushing commits and taking passes through the code to ensure correctness. The result was [6]. It was an incredible effort. The collaboration was very fast paced and exciting. Matt and Kyle are terrific programmers and fun to work with too.
The first step was assessing the current state of the code the previous developer had dumped into the tree. It was not pretty. I imagined strange Internet voices jeering, “this is what gives C a bad name!” There were random sleeps added to “fix” race conditions, validation functions that just returned true, catastrophic cryptographic vulnerabilities, whole parts of the protocol unimplemented, kernel panics, security bypasses, overflows, random printf statements deep in crypto code, the most spectacular buffer overflows, and the whole litany of awful things that go wrong when people aren’t careful when they write C. Or, more simply, it seems typical of what happens when code ships that wasn’t meant to. It was essentially an incomplete half-baked implementation – nothing close to something anybody would want on a production machine. Matt had to talk me out of just insisting they pull the code entirely, and rework it more slowly and carefully for the next release cycle. And he was right: nobody would have agreed to do that, and it would only have fostered frustration from folks genuinely enthusiastic about if_wg. So our one and only option was to iteratively improve it as fast as we could during the two weeks before release, and try to make it as simple and close as possible to OpenBSD so that we could benefit from the previous analysis done there. With that as our mission, we set out auditing and rewriting code.
One curious thing of note is that there were 40,000 lines of optimized crypto implementations pulled out of the Linux kernel compat module but not really wired up correctly, and mangled beyond repair with mazes of Linux→FreeBSD ifdefs. I wound up replacing this with an 1,800 line file, crypto.c [1], containing all of the cryptographic primitives needed to implement WireGuard. Aside from its place in the FreeBSD story, this is kind of neat in its own right: these are simple, but fast enough, reference implementations. It’s not deliberately tiny or obfuscated like TweetNaCl is, yet is still just a single file, and the Curve25519 field arithmetic in it is formally verified. Maybe other projects will find use for it. Future releases will hopefully get rid of crypto.c and hook into FreeBSD’s already existing optimized implementations [4], which should give a nice performance boost, but given the time crunch, having something boring, safe, and simple seemed like the way to go.
We reduced the project structure down to four C files – the aforementioned crypto.c, two files copied verbatim from OpenBSD – wg_noise.c and wg_cookie.c – and if_wg.c, the actual interface device driver implementation and protocol logic. The IPC interface was reworked as well, and wg(8) in the wireguard-tools package grew support for it (also rewritten from the original attempt). The three of us spent countless hours across three time zones auditing state machine logic, running trials, and generally trying to get this working and workable. There are now even a few automated tests!
I think we’ve mostly succeeded in producing something that behaves like WireGuard. The net result certainly isn’t perfect, though – the Linux and OpenBSD implementations were long, careful, slow projects by comparison – but it is at least a base on which to build and improve over time. Going forward, I think there’ll be additional systems coding issues to work out – locking, lifetimes, races, and that sort of thing. But now that there’s at least a stable base, developers can work out remaining issues incrementally.
But perhaps this is a good moment to step back and ask how we got here, and what WireGuard itself really is.
Traditionally, network protocols are specified in a document of protocol behaviors. Then different organizations implement that specification. Then everybody interoperates and all goes well. In practice, it often doesn’t go well (see IPsec woes), but this at least has been the traditional way of doing this on the Internet, and in some ways it works.
But that is not the approach taken by the WireGuard project. In contrast, WireGuard is both a protocol and a set of implementations, implemented with a particular set of security and safety techniques. That’s a radical departure from the traditional model, and one surely to raise some grumbles amongst graybeards. But I believe this is a necessary and beneficial quality for having the types of high assurance software that is needed for core Internet security infrastructure. When you use WireGuard, you’re not just using some protocol that is capable of producing packets that are legible by others. You’re also using an implementation that’s been designed to avoid security pitfalls, and that provides interfaces for using it that mitigate footguns. In that way, the WireGuard project is more expansive than a mere protocol project or a mere software project or a mere cryptography project or a mere specification project or a mere interface project. It combines all of those things into a single unified approach. (For this same reason, the original WireGuard paper [2] has been difficult for folks to categorize. Is this a systems paper? A networking paper? A crypto paper?)
Because of that, I think this was an understandable predicament. After all, why shouldn’t a company be able to task a developer with writing some ring-0 WireGuard code in C? And why does it matter to me whether the code is garbage if it can at least produce protocol packets? The reason is that the WireGuard project’s mission is wider than that. We deeply care about code quality and implementation particulars.
While we now have the FreeBSD code in a maintainable state, there are other projects too that could use some attention from us. Looking forward, for example, we hope to be able to lend a hand similarly to the NetBSD developers soon to help them finish their implementation; this is long overdue on my part, and I owe them some time and energy there. And I hope that others don’t hesitate to email the list asking for collaboration. This kind of thing is, of course, one of the reasons that the project as an organization exists.
To return to the primary announcement, I had originally hoped to say that this would be shipping for 13.0, and have some instructions for setup there, but unfortunately, and contrary to our plans, it looks exceedingly likely that given the grave issues we found in the existing code, they’ll in the end just disable the module from the release, and revisit for 13.1, rather than merging our fixes a few short days before the release. That’s a bit of a bummer, given how hard we worked to get things done in the time crunch, but it’s also probably a very wise decision that takes some courage to make, and this will give us more time to really get this rock solid for 13.1.
As well, hopefully we’ll have backport modules for the 13.0 and 12.y release, making it as available as possible. Kyle or I will update the list when we’ve got a standalone backport module ready, with instructions, as well as updating [3] per usual. There’s also ongoing work to integrate WireGuard interface setup into rc, and hopefully that will land during the the next release window, as well as the aforementioned improvements to optimized crypto and systems issues.
Enjoy,
Jason
[1] https://cgit.freebsd.org/src/tree/sys/dev/if_wg/crypto.c?id=74ae3f3e33b810248da19004c58b3581cd367843
[2] https://www.wireguard.com/papers/wireguard.pdf
[3] https://www.wireguard.com/install/
[4] https://lists.freebsd.org/pipermail/freebsd-hackers/2021-March/057076.html
[5] https://lists.zx2c4.com/pipermail/wireguard/2021-March/006493.html
[6] https://cgit.freebsd.org/src/commit/?id=74ae3f3e33b810248da19004c58b3581cd367843
That's awesome!
Thanks a lot for your hard work and dedication on wireguard.
I really appreciate what you do including your willingness to work with non Linux OSes.
Cheers,
Reto
Hi Scott,
It's certainly disappointing to receive your email like that. I see that you're really upset, and I think we have a shared interest in deescalating this. You've mentioned the desire to talk about this in public, so I'm responding in the announcement thread to you, as this seems like the appropriate venue for discussion. I've also CC'd the FreeBSD Security Officer. I've responded to your email and its threats in line mailing-list style below:
Quoted 5 lines · Scott Long
What you and Kyle did was tell the world that there are a number of zero-day exploits in the code. You gave us no details until after the fact, gave us no time to mitigate, correct, and publish before your announcement and Kyle's code drop, and used the opportunity to bash the code, and by extension us, for your own self-gain.
I have no idea how you possibly have arrived at this here. FreeBSD 13 is currently at RC2 or RC3, if I recall correctly. Our whole motivation for working on this so tirelessly and with so much energy and devotion over the week was to get things fixed up before 13 was actually released. I dropped everything else I was working on to focus on this. I'm actually with my folks in Colorado and should be spending more time with them but the idea of FreeBSD 13 shipping with all those bugs seemed like a really important thing to address.
Here's what we did, step by step:
1. We made a clone of the freebsd git repo where we could all push changes to master, to git.zx2c4.com like everything else.
2. We took turns taking big passes at the codebase, reading it, auditing it, fixing things. We did a lot of syncing and comparison with the OpenBSD code base. For example, in one step, we reordered the functions to match the flow of OpenBSD so we could see it more easily side by side.
3. We kept pushing commits like this in that manner.
4. We didn't know where we'd end up, but after a few days of this, the number of changes we had accumulated and fishy business we had observed was massive.
5. When we thought we reached a good resting point -- and when it became clear that our rapid fire changes were becoming too numerous -- we squashed that down to a summary commit, and Kyle committed it.
6. Seeing that the crypto needed to be re-accelerated, I published a call for help here --
https://lists.freebsd.org/pipermail/freebsd-hackers/2021-March/057076.html
7. I published wireguard-tools with the improved wg(8) command here --
https://lists.zx2c4.com/pipermail/wireguard/2021-March/006493.html
8. I sent a summary of everything to the mailing list here --
https://lists.zx2c4.com/pipermail/wireguard/2021-March/006494.html
I also made sure to email a FreeBSD Security Officer, Colin, to keep him posted about the state of this, so he was kept apprised.
Again, the whole idea was: two weeks until release! If I suggest the code is just disabled, people will be really upset, so instead we'll try to fix as much as fast as possible before the release. And actually, really focused sprints like that are fun and stimulating. We were all really enjoying writing code until the torrents of anger started coming from you about our efforts. (And I also learned that this isn't the first time you guys tried to intimidate an open source project -- see this summary of the opnSense defamatory website Netgate created --
https://opnsense.org/opnsense-com/ .)
And in the process, too, I've tried to be in contact with you and Jim and let you know what our intentions are and to diffuse tensions. I spent time on a video call trying to describe to you some of the security things we found, in case it wasn't possible for you to use the new code right away. I've also made it abundantly clear to you how much I want to work WITH Netgate. When that Reddit thread cropped up, I offered to you multiple times to send a message to it telling people that we've spoken and it looks like you have a good plan and every things going to be okay, but you didn't respond to my offer.
Quoted 4 lines · Scott Long
None
of these actions reflect good-faith collaboration, and your statements that you think that our work is "really cool" ring hollow. Maybe you and Kyle are really naive and thought that this was how people
Quoted 8 lines
normally collaborate in the security and business worlds, and that everyone would rush to praise you and shower you with contracts and funding. And hey, maybe that’ll happen, you certainly have made a splash. That of course, comes at our expense, and it’s likely to be a pretty damaging one. Now with the Ars article, I’m starting to wonder if this was a coordinated smear campaign. Normally we don’t get this kind of attention. It’s going to be painful to navigate our way out.
I'm not sure how to address these types of... theories. I certainly AM acting in good faith. I can't possibly see how it benefits anybody to have WireGuard and Netgate at odds with one another. You guys make a neat product based on open source software, and I think it's great you're planning to add WireGuard to it. My interest is in making sure the underlying operating system -- FreeBSD 13 -- does that well. (Or, I think you mentioned to me this morning that you're actually planning to base it on FreeBSD 12 for some time, but same idea for the backport you're working on, I guess.)
And there's no "smear campaign" -- that's outrageous. I have no idea how that would help anybody at all. I should point out, again to you, how grateful I am that Netgate got the initial work on this started. Had you guys not gotten if_wg rolling, it's not clear to me that it'd exist in the first place.
From your end, it's very disappointing that you're (below) planning to attack security researchers and kernel programmers who took the time to rewrite code to make it better, hustling to get it in before FreeBSD 13 is released.
Quoted 8 lines · Scott Long
I’ll be writing an article tonight outlining the mistakes that we made and what we will be doing to correct it. I’ll also be highlighting that this incident has been a textbook example of the wrong way for people to collaborate in the security community, and that extreme caution should be taken in any future dealings with you. Your actions are self serving and in bad faith, and your words are hollow and untrustworthy. I don’t care if you disagree or don’t see it this way, this is the effect that you’ve had.
It makes sense to communicate with your customers about things with your upcoming products if you feel it's necessary. But threatening that you're going to highlight "that extreme caution should be taken in any future dealings with you" sounds to me like a threat of some intense slander. And again, attacking security researchers and kernel programmers who took time to rewrite code to make it better before a release deadline... That's ...wow. I wish you would not go on the attack like that.
Instead, it seems like we can both be glad that we're on the path to having better code! And that there's a team of people volunteering to do this and enjoying writing that code. We're all working together to make things better. We worked really super hard on fixing up this code base. It was a LOT of work. And I would like to keep working on it and feeling good about it.
So I'm just disappointed that somehow this has devolved for you into making threats like that.
Quoted 7 lines · Scott Long
I’ve also spoken with the FreeBSD Security Officer, and we’ve agreed that wireguard will be removed from all branches of FreeBSD until further notice. I’ve also informed Kyle of this. I do not support its reintroduction into FreeBSD, whether in the src tree or in the ports tree, at this time. As for pfSense, we are conducting an audit and will decide on the best course of action for our customers and our company.
I actually think that's a great idea. I was under the impression initially that removing it would be a total non-starter and would just make people upset, hence this hurried mad dash to turn things around. But having an opportunity to take it out of tree and develop it slowly and carefully, like we did for Linux and OpenBSD, sounds a lot better. This also meshes well with what we had planned to do, which I mentioned in the announcement email this morning -- have this available as an out of tree module that people can try out. Then, after we've got it really solid, reintroduce it in the normal FreeBSD ways of doing that. (I'm not so familiar with that latter process, but I trust that Kyle & folks know that part well.)
Quoted 2 lines · Scott Long
Scott
So, that's where we are. You're upset and making threats to slander security researchers and kernel programmers. That's really disappointing. All I can do is ask you: please don't? We have much more to gain by supporting each other than being in conflict. I just want to make sure FreeBSD 13 (or 14) gets a great WireGuard implementation. And you want to make sure your customers get great products. I see no reason why these are conflicting goals.
I hope this can deescalate, and we can work together on this. I'm confident that our goals are ultimately aligned very well.
Regards,
Jason
Quoted 8 lines · Jason A. Donenfeld
Hi everybody,
[... snip ...]
The first step was assessing the current state of the code the previous developer had dumped into the tree. It was not pretty. I imagined strange Internet voices jeering, “this is what gives C a bad name!”
This was a highly unnecessary jab.
Quoted 1 line
There were random sleeps added to “fix” race conditions
This is true.
Quoted 1 line
validation functions that just returned true
I looked back at the history here just now, and I count one, wg_allowedip_valid, that pretty much got ripped out anyways.
Quoted 2 lines
catastrophic cryptographic vulnerabilities
whole parts of the protocol unimplemented
I'm not qualified to speak about these two, which are perhaps the worst from the public's perspective.
Quoted 1 line
kernel panics
These are true.
Quoted 2 lines
security bypasses
overflows
I only recall one of each.
Quoted 1 line
random printf statements deep in crypto code
This is true.
Quoted 1 line
the most spectacular buffer overflows
English is a crappy language, but this sounds like you're advertising more than one. There was one, and for the record to anyone watching to dispel "the word on the street" that it's potentially an RCE: based on what's happening I cannot imagine how you could usefully turn it into an RCE. Local privileged execution, 100%, but looking at it further, you've got to be pretty skilled to make it before killing the kernel elsewhere.
Quoted 2 lines
and the whole litany of awful things that go wrong when people aren’t careful when they write C.
This is an exceedingly broad statement, and it would have been good to provide some pointers to these.
Quoted 1 line
[...]
You know that I don't appreciate how you handled this initial communication, as I've told you a number of times. Now that I look at the history again, I'm even more disappointed in how you handled this because there are some pretty broad statements in here and language that could go either way.
I've additionally recommended, as a developer and not in any kind of official capacity, that we can't include if_wg in any future version of base. We cannot have our users being put at the risk of this kind of publicity if we can't get security
advisories issued in time. Ports is a fine place, where security issues can be addressed expeditiously and more in line with how the WireGuard project chooses to handle them.
Thanks,
Kyle Evans
Quoted 41 lines · Kyle Evans
Quoted 8 lines · Jason A. Donenfeld
Hi everybody,
[... snip ...]
The first step was assessing the current state of the code the previous developer had dumped into the tree. It was not pretty. I imagined strange Internet voices jeering, “this is what gives C a bad name!”
This was a highly unnecessary jab.
Quoted 1 line
There were random sleeps added to “fix” race conditions
This is true.
Quoted 1 line
validation functions that just returned true
I looked back at the history here just now, and I count one,
wg_allowedip_valid, that pretty much got ripped out anyways.Quoted 2 lines
catastrophic cryptographic vulnerabilities
whole parts of the protocol unimplementedI'm not qualified to speak about these two, which are perhaps the worst from the public's perspective.
Quoted 1 line
kernel panics
These are true.
Quoted 2 lines
security bypasses
overflowsI only recall one of each.
Quoted 1 line
random printf statements deep in crypto code
This is true.
Quoted 9 lines
Quoted 1 line
the most spectacular buffer overflows
English is a crappy language, but this sounds like you're advertising more than one. There was one, and for the record to anyone watching to dispel "the word on the street" that it's potentially an RCE: based on what's happening I cannot imagine how you could usefully turn it into an RCE. Local privileged execution, 100%, but looking at it further, you've got to be pretty skilled to make it before killing the kernel elsewhere.
Ah, I have to clarify this one; I suspect if you're acting as a wg gateway, it's possible. It
still doesn't look like an easy one to exploit.
Quoted 24 lines
Quoted 2 lines
and the whole litany of awful things that go wrong when people aren’t careful when they write C.
This is an exceedingly broad statement, and it would have been good to provide some pointers to these.
Quoted 1 line
[...]
You know that I don't appreciate how you handled this initial communication, as I've told you a number of times. Now that I look at the history again, I'm even more disappointed in how you handled this because there are some pretty broad statements in here and language that could go either way.
I've additionally recommended, as a developer and not in any kind of official capacity, that we can't include
if_wgin any future version of base. We cannot have our users being put at the risk of this kind of publicity if we can't get security
advisories issued in time. Ports is a fine place, where security issues can be addressed expeditiously and more in line with how the WireGuard project chooses to handle them.Thanks,
Kyle Evans