Skip to contentSkip to conversation

Userspace Networking Stack + WireGuard + Go

DISCUSSION Jason A. Donenfeld 4 participants 9 messages Jan 13, 2021 – Jan 15, 2021

lore
Share & tools
lore
#0 lore raw

Hi,

Sometimes people say things like, "I want to make an application that connects to the server over WireGuard" or "I want to host this service application behind WireGuard" or "I want this small utility to initiate an SSH connection to a specific server over WireGuard." The usual answer to this is to simply bring up a WireGuard interface on the computer, and then do networking as usual. And making WireGuard easy enough that people can pull up ad-hoc interfaces trivially has been a big motivating factor for the simplicity of the tooling, such as ip link add wg0 type wireguard && wg set wg0 ... and similar commands.

On Linux, adding an interface is easy to do and is a solution that fits nearly all circumstances. Even if you're unprivileged and want a WireGuard interface for just a single application that's bound to the lifetime of that application, you can still use WireGuard's normal kernel interface inside of a user namespace + a network namespace, and get a private process-specific WireGuard interface. Pretty cool stuff. It's not like that's an accident, of course -- WireGuard was originally developed with Linux's model in mind.

But not all operating systems are as neat as Linux. To that end, we've tried very hard to make WireGuard integrate as natively as possible into other operating systems, with native clients for every platform, as well as APIs for every platform to allow application developers to embed
tunnels: https://www.wireguard.com/embedding/ . This is great and works well. WireGuard's design centers around the "network interface" as the central object, and that's what these libraries enable. But still, some people do not want to think about network interfaces or operating systems when writing code.

To that end, I've recently written some code that couples a userspace networking stack (from gVisor) with a userspace WireGuard implementation (from wireguard-go), along with a little custom DNS client implementation, so that Go applications can use an in-process WireGuard interface directly to make TCP and UDP connections and listeners. From the operating system's point of view, the application just sends and receives encrypted UDP packets, and never sees the inner TCP or UDP packets or knows about WireGuard at all.

Here's a quick example of what's possible:

    package main

    import (
        "io"
        "log"
        "net"
        "net/http"

        "golang.zx2c4.com/wireguard/device"
        "golang.zx2c4.com/wireguard/tun"
    )

    func main() {
        tun, tnet, err := tun.CreateNetTUN(
            []net.IP{net.ParseIP("192.168.4.29")},
	    []net.IP{net.ParseIP("8.8.8.8")},
	    1420)
        if err != nil {
            log.Panic(err)
        }
        dev := device.NewDevice(tun, &device.Logger{log.Default(), log.Default(), log.Default()})
        dev.IpcSet(`private_key=a8dac1d8a70a751f0f699fb14ba1cff7b79cf4fbd8f09f44c6e6a90d0369604f
    public_key=25123c5dcd3328ff645e4f2a3fce0d754400d3887a0cb7c56f0267e20fbf3c5b
    endpoint=163.172.161.0:12912
    allowed_ip=0.0.0.0/0
    `)
        dev.Up()

        client := http.Client{
            Transport: &http.Transport{
                DialContext: tnet.DialContext,
            },
        }
        resp, err := client.Get("https://www.zx2c4.com/ip")
        if err != nil {
            log.Panic(err)
        }
        body, err := io.ReadAll(resp.Body)
        if err != nil {
            log.Panic(err)
        }
        log.Println(string(body))
    }

This snippet prints out:

    163.172.161.0
    demo.wireguard.com
    Go-http-client/1.1

, because that HTTPS request is going through the demo box.

Other use cases of this include cloud VM providers to bundle an ssh client directly into their tooling that connects to a "cloud private network" over WireGuard. Or hosting utilities that allow people to run web apps from their own computers that are then accessible on the Internet by connecting in reverse through WireGuard. Or maybe other use cases.

The API and the code is new, and potentially incomplete. Please take it for a spin and let me know your feedback.

Enjoy,
Jason

PS: While the gophers on the list might be happy about this, the rustaceans may well be left wondering, "what about us?" I hope to have some positive news about wireguard-rs not before too long.

#1 lore raw
Quoted 5 lines · Jason A. Donenfeld

Even if you're unprivileged and want a WireGuard interface for just a single application that's bound to the lifetime of that application, you can still use WireGuard's normal kernel interface inside of a user namespace + a network namespace, and get a private process-specific WireGuard interface.

That's what my patches from back in 2018 were trying to accomplish. Unless I've missed something since, I do not see how what you're describing would work. Unless you also

- create a TUN device in the network namespace
- add a default route through that TUN device
- manually route all traffic between the init network namespace and your network namespace.

Is that what you meant or is there a simpler way?

#2 lore raw

network namespace wireguard routing [Was: Re: Userspace Networking Stack + WireGuard + Go]

In order to prevent this Go thread from being hijacked with Linux concerns, I've changed the Subject line of the email. Please keep follow ups in this thread rather than the other.

Response is in line below:

Quoted 13 lines · Julian Orth
Quoted 5 lines · Jason A. Donenfeld

Even if you're unprivileged and want a WireGuard interface for just a single application that's bound to the lifetime of that application, you can still use WireGuard's normal kernel interface inside of a user namespace + a network namespace, and get a private process-specific WireGuard interface.

That's what my patches from back in 2018 were trying to accomplish. Unless I've missed something since, I do not see how what you're describing would work. Unless you also

Quoted 6 lines

- create a TUN device in the network namespace
- add a default route through that TUN device
- manually route all traffic between the init network namespace and your network namespace.

Is that what you meant or is there a simpler way?

What I meant was:

1. User opens his shell and runs ./blah. That executes in the init namespace where all the physical interfaces are.
2. blah creates a wireguard interface.
3. blah creates a network namespace.
4. blah moves that wireguard interface into that network namespace.
5. blah calls setns() on one of its threads to use that network namespace.

Thinking about this in more detail, I'm guessing you take issue with step #2? Since that actually might require privileges in the init namespace?

Jason

#3 lore raw

Re: network namespace wireguard routing [Was: Re: Userspace Networking Stack + WireGuard + Go]

Quoted 36 lines · Jason A. Donenfeld

In order to prevent this Go thread from being hijacked with Linux concerns, I've changed the Subject line of the email. Please keep follow ups in this thread rather than the other.

Response is in line below:

Quoted 13 lines · Julian Orth
Quoted 5 lines · Jason A. Donenfeld

Even if you're unprivileged and want a WireGuard interface for just a single application that's bound to the lifetime of that application, you can still use WireGuard's normal kernel interface inside of a user namespace + a network namespace, and get a private process-specific WireGuard interface.

That's what my patches from back in 2018 were trying to accomplish. Unless I've missed something since, I do not see how what you're describing would work. Unless you also

Quoted 6 lines

- create a TUN device in the network namespace
- add a default route through that TUN device
- manually route all traffic between the init network namespace and your network namespace.

Is that what you meant or is there a simpler way?

What I meant was:

1. User opens his shell and runs ./blah. That executes in the init namespace where all the physical interfaces are.
2. blah creates a wireguard interface.
3. blah creates a network namespace.
4. blah moves that wireguard interface into that network namespace.
5. blah calls setns() on one of its threads to use that network namespace.

Quoted 3 lines

Thinking about this in more detail, I'm guessing you take issue with step #2? Since that actually might require privileges in the init namespace?

Exactly :). My patches in 2018 were trying to solve this by allowing the user to change the "transit" network namespace after the device has been created. The "transit" network namespace being the namespace in which the Wireguard UDP socket lives. This would not require privileges in the transit namespace, only some kind of proof that the user can create UDP sockets in said namespace.

Quoted 3 lines

Jason

#4 lore raw

Re: network namespace wireguard routing [Was: Re: Userspace Networking Stack + WireGuard + Go]

In reply to Jason A. Donenfeld
Quoted 36 lines · Jason A. Donenfeld

In order to prevent this Go thread from being hijacked with Linux concerns, I've changed the Subject line of the email. Please keep follow ups in this thread rather than the other.

Response is in line below:

Quoted 13 lines · Julian Orth
Quoted 5 lines · Jason A. Donenfeld

Even if you're unprivileged and want a WireGuard interface for just a single application that's bound to the lifetime of that application, you can still use WireGuard's normal kernel interface inside of a user namespace + a network namespace, and get a private process-specific WireGuard interface.

That's what my patches from back in 2018 were trying to accomplish. Unless I've missed something since, I do not see how what you're describing would work. Unless you also

Quoted 6 lines

- create a TUN device in the network namespace
- add a default route through that TUN device
- manually route all traffic between the init network namespace and your network namespace.

Is that what you meant or is there a simpler way?

What I meant was:

1. User opens his shell and runs ./blah. That executes in the init namespace where all the physical interfaces are.
2. blah creates a wireguard interface.
3. blah creates a network namespace.
4. blah moves that wireguard interface into that network namespace.
5. blah calls setns() on one of its threads to use that network namespace.

Quoted 3 lines

Thinking about this in more detail, I'm guessing you take issue with step #2? Since that actually might require privileges in the init namespace?

5. also requires CAP_SYS_ADMIN (and I think by extension, so does 3., and 4.). From 'man setns':

Network, IPC, time, and UTS namespaces
In order to reassociate itself with a new network, IPC, time, or UTS namespace, the caller must have the CAP_SYS_ADMIN capability both in its own user namespace and in the user namespace that owns the target namespace.

-Toke

#5 lore raw

Re: network namespace wireguard routing [Was: Re: Userspace Networking Stack + WireGuard + Go]

Quoted 8 lines · Toke Høiland-Jørgensen

5. also requires CAP_SYS_ADMIN (and I think by extension, so does 3., and 4.). From 'man setns':

Network, IPC, time, and UTS namespaces
In order to reassociate itself with a new network, IPC, time, or UTS namespace, the caller must have the CAP_SYS_ADMIN capability both in its own user namespace and in the user namespace that owns the target namespace.

For this, you just create a new user namespace first. You can try it yourself from the command line:

zx2c4@thinkpad ~ $ unshare -n
unshare: unshare failed: Operation not permitted zx2c4@thinkpad ~ $ unshare -Un
nobody@thinkpad ~ $ ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
#6 lore raw

Re: network namespace wireguard routing [Was: Re: Userspace Networking Stack + WireGuard + Go]

Quoted 13 lines · Jason A. Donenfeld
Quoted 8 lines · Toke Høiland-Jørgensen

5. also requires CAP_SYS_ADMIN (and I think by extension, so does 3., and 4.). From 'man setns':

Network, IPC, time, and UTS namespaces
In order to reassociate itself with a new network, IPC, time, or UTS namespace, the caller must have the CAP_SYS_ADMIN capability both in its own user namespace and in the user namespace that owns the target namespace.

For this, you just create a new user namespace first. You can try it yourself from the command line:

Quoted 6 lines

zx2c4@thinkpad ~ $ unshare -n
unshare: unshare failed: Operation not permitted zx2c4@thinkpad ~ $ unshare -Un
nobody@thinkpad ~ $ ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

Ah, right, of course, silly me :)

-Toke

#7 lore raw
In reply to Julian Orth

Hi Julian

Quoted 13 lines · Julian Orth
Quoted 5 lines · Jason A. Donenfeld

Even if you're unprivileged and want a WireGuard interface for just a single application that's bound to the lifetime of that application, you can still use WireGuard's normal kernel interface inside of a user namespace + a network namespace, and get a private process-specific WireGuard interface.

That's what my patches from back in 2018 were trying to accomplish. Unless I've missed something since, I do not see how what you're describing would work. Unless you also

Quoted 6 lines

- create a TUN device in the network namespace
- add a default route through that TUN device
- manually route all traffic between the init network namespace and your network namespace.

Is that what you meant or is there a simpler way?

I am not a network admin, but I agree. Setting up this kind of user network namespace isn't trivial and requires some privileges. It would be nice if the kernel or some services provided a simpler way. (fwiw, some time ago I did some experimental/research work for VM & containers at https://gitlab.freedesktop.org/elmarco/vnet)

Signature · 1 line
Marc-André Lureau
#8 lore raw
In reply to Jason A. Donenfeld

Another example, for the curious. This one hosts a web server entirely in userspace. All the kernel sees are incoming and outgoing encrypted WireGuard UDP packets.

    package main
    import (
        "io"
        "log"
        "net"
        "net/http"
        "golang.zx2c4.com/wireguard/device"
        "golang.zx2c4.com/wireguard/tun"
    )
    func main() {
        tun, tnet, err := tun.CreateNetTUN(
            []net.IP{net.ParseIP("192.168.4.29")},
            []net.IP{net.ParseIP("8.8.8.8"), net.ParseIP("8.8.4.4")},
            1420,
        )
        if err != nil {
            log.Panic(err)
        }
        dev := device.NewDevice(tun, &device.Logger{log.Default(), log.Default(), log.Default()})
        dev.IpcSet(`private_key=a8dac1d8a70a751f0f699fb14ba1cff7b79cf4fbd8f09f44c6e6a90d0369604f
    public_key=25123c5dcd3328ff645e4f2a3fce0d754400d3887a0cb7c56f0267e20fbf3c5b
    endpoint=163.172.161.0:12912
    allowed_ip=0.0.0.0/0
    persistent_keepalive_interval=25
    `)
        dev.Up()
        listener, err := tnet.ListenTCP(&net.TCPAddr{Port: 80})
        if err != nil {
            log.Panicln(err)
        }
        http.HandleFunc("/", func(writer http.ResponseWriter, request *http.Request) {
            log.Printf("> %s - %s - %s", request.RemoteAddr, request.URL.String(), request.UserAgent())
            io.WriteString(writer, "Hello from userspace TCP!")
        })
        err = http.Serve(listener, nil)
        if err != nil {
            log.Panicln(err)
        }
    }

Here's a gif: https://data.zx2c4.com/wireguard-go-userspace-networking-to-host-a-server.gif