Skip to contentSkip to conversation

net: WireGuard secure network tunnel

PATCH Applied Jason A. Donenfeld 3 participants 30 messages Dec 8, 2019 – Feb 20, 2020

Subsystems (5)
lore
Share & tools
lore

Jason A. Donenfeld

author maintainer
#0 lore raw

PATCH v2 1/1 net: WireGuard secure network tunnel

WireGuard is a layer 3 secure networking tunnel made specifically for the kernel, that aims to be much simpler and easier to audit than IPsec. Extensive documentation and description of the protocol and considerations, along with formal proofs of the cryptography, are available at:

* https://www.wireguard.com/
* https://www.wireguard.com/papers/wireguard.pdf

This commit implements WireGuard as a simple network device driver, accessible in the usual RTNL way used by virtual network drivers. It makes use of the udp_tunnel APIs, GRO, GSO, NAPI, and the usual set of networking subsystem APIs. It has a somewhat novel multicore queueing system designed for maximum throughput and minimal latency of encryption operations, but it is implemented modestly using workqueues and NAPI. Configuration is done via generic Netlink, and following a review from the Netlink maintainer a year ago, several high profile userspace tools have already implemented the API.

This commit also comes with several different tests, both in-kernel tests and out-of-kernel tests based on network namespaces, taking profit of the fact that sockets used by WireGuard intentionally stay in the namespace the WireGuard interface was originally created, exactly like the semantics of userspace tun devices. See wireguard.com/netns/ for pictures and examples.

The source code is fairly short, but rather than combining everything into a single file, WireGuard is developed as cleanly separable files, making auditing and comprehension easier. Things are laid out as follows:

* noise.[ch], cookie.[ch], messages.h: These implement the bulk of the cryptographic aspects of the protocol, and are mostly data-only in nature, taking in buffers of bytes and spitting out buffers of bytes. They also handle reference counting for their various shared pieces of data, like keys and key lists.

* ratelimiter.[ch]: Used as an integral part of cookie.[ch] for ratelimiting certain types of cryptographic operations in accordance with particular WireGuard semantics.

* allowedips.[ch], peerlookup.[ch]: The main lookup structures of WireGuard, the former being trie-like with particular semantics, an integral part of the design of the protocol, and the latter just being nice helper functions around the various hashtables we use.

* device.[ch]: Implementation of functions for the netdevice and for rtnl, responsible for maintaining the life of a given interface and wiring it up to the rest of WireGuard.

* peer.[ch]: Each interface has a list of peers, with helper functions available here for creation, destruction, and reference counting.

* socket.[ch]: Implementation of functions related to udp_socket and the general set of kernel socket APIs, for sending and receiving ciphertext UDP packets, and taking care of WireGuard-specific sticky socket routing semantics for the automatic roaming.

* netlink.[ch]: Userspace API entry point for configuring WireGuard peers and devices. The API has been implemented by several userspace tools and network management utility, and the WireGuard project distributes the basic wg(8) tool.

* queueing.[ch]: Shared function on the rx and tx path for handling the various queues used in the multicore algorithms.

* send.c: Handles encrypting outgoing packets in parallel on multiple cores, before sending them in order on a single core, via workqueues and ring buffers. Also handles sending handshake and cookie messages as part of the protocol, in parallel.

* receive.c: Handles decrypting incoming packets in parallel on multiple cores, before passing them off in order to be ingested via the rest of the networking subsystem with GRO via the typical NAPI poll function. Also handles receiving handshake and cookie messages as part of the protocol, in parallel.

* timers.[ch]: Uses the timer wheel to implement protocol particular event timeouts, and gives a set of very simple event-driven entry point functions for callers.

* main.c, version.h: Initialization and deinitialization of the module.

* selftest/*.h: Runtime unit tests for some of the most security sensitive functions.

* tools/testing/selftests/wireguard/netns.sh: Aforementioned testing script using network namespaces.

This commit aims to be as self-contained as possible, implementing WireGuard as a standalone module not needing much special handling or coordination from the network subsystem. I expect for future optimizations to the network stack to positively improve WireGuard, and vice-versa, but for the time being, this exists as intentionally standalone.

We introduce a menu option for CONFIG_WIREGUARD, as well as providing a verbose debug log and self-tests via CONFIG_WIREGUARD_DEBUG.

---

This applies on top of 5.5-rc1 and on net-next.

Changes v1->v2:
- Avoid touching skb->{next,prev} directly where possible.
- API changes for 5.5.

 MAINTAINERS                                  |   8 +
 drivers/net/Kconfig                          |  41 +
 drivers/net/Makefile                         |   1 +
 drivers/net/wireguard/Makefile               |  18 +
 drivers/net/wireguard/allowedips.c           | 381 +++++++++
 drivers/net/wireguard/allowedips.h           |  59 ++
 drivers/net/wireguard/cookie.c               | 236 ++++++
 drivers/net/wireguard/cookie.h               |  59 ++
 drivers/net/wireguard/device.c               | 458 ++++++++++
 drivers/net/wireguard/device.h               |  73 ++
 drivers/net/wireguard/main.c                 |  64 ++
 drivers/net/wireguard/messages.h             | 128 +++
 drivers/net/wireguard/netlink.c              | 642 ++++++++++++++
 drivers/net/wireguard/netlink.h              |  12 +
 drivers/net/wireguard/noise.c                | 828 +++++++++++++++++++
 drivers/net/wireguard/noise.h                | 137 +++
 drivers/net/wireguard/peer.c                 | 240 ++++++
 drivers/net/wireguard/peer.h                 |  83 ++
 drivers/net/wireguard/peerlookup.c           | 221 +++++
 drivers/net/wireguard/peerlookup.h           |  64 ++
 drivers/net/wireguard/queueing.c             |  53 ++
 drivers/net/wireguard/queueing.h             | 197 +++++
 drivers/net/wireguard/ratelimiter.c          | 223 +++++
 drivers/net/wireguard/ratelimiter.h          |  19 +
 drivers/net/wireguard/receive.c              | 595 +++++++++++++
 drivers/net/wireguard/selftest/allowedips.c  | 683 +++++++++++++++
 drivers/net/wireguard/selftest/counter.c     | 104 +++
 drivers/net/wireguard/selftest/ratelimiter.c | 226 +++++
 drivers/net/wireguard/send.c                 | 413 +++++++++
 drivers/net/wireguard/socket.c               | 437 ++++++++++
 drivers/net/wireguard/socket.h               |  44 +
 drivers/net/wireguard/timers.c               | 243 ++++++
 drivers/net/wireguard/timers.h               |  31 +
 drivers/net/wireguard/version.h              |   1 +
 include/uapi/linux/wireguard.h               | 196 +++++
 tools/testing/selftests/wireguard/netns.sh   | 537 ++++++++++++
 36 files changed, 7755 insertions(+)
 create mode 100644 drivers/net/wireguard/Makefile
 create mode 100644 drivers/net/wireguard/allowedips.c
 create mode 100644 drivers/net/wireguard/allowedips.h
 create mode 100644 drivers/net/wireguard/cookie.c
 create mode 100644 drivers/net/wireguard/cookie.h
 create mode 100644 drivers/net/wireguard/device.c
 create mode 100644 drivers/net/wireguard/device.h
 create mode 100644 drivers/net/wireguard/main.c
 create mode 100644 drivers/net/wireguard/messages.h
 create mode 100644 drivers/net/wireguard/netlink.c
 create mode 100644 drivers/net/wireguard/netlink.h
 create mode 100644 drivers/net/wireguard/noise.c
 create mode 100644 drivers/net/wireguard/noise.h
 create mode 100644 drivers/net/wireguard/peer.c
 create mode 100644 drivers/net/wireguard/peer.h
 create mode 100644 drivers/net/wireguard/peerlookup.c
 create mode 100644 drivers/net/wireguard/peerlookup.h
 create mode 100644 drivers/net/wireguard/queueing.c
 create mode 100644 drivers/net/wireguard/queueing.h
 create mode 100644 drivers/net/wireguard/ratelimiter.c
 create mode 100644 drivers/net/wireguard/ratelimiter.h
 create mode 100644 drivers/net/wireguard/receive.c
 create mode 100644 drivers/net/wireguard/selftest/allowedips.c
 create mode 100644 drivers/net/wireguard/selftest/counter.c
 create mode 100644 drivers/net/wireguard/selftest/ratelimiter.c
 create mode 100644 drivers/net/wireguard/send.c
 create mode 100644 drivers/net/wireguard/socket.c
 create mode 100644 drivers/net/wireguard/socket.h
 create mode 100644 drivers/net/wireguard/timers.c
 create mode 100644 drivers/net/wireguard/timers.h
 create mode 100644 drivers/net/wireguard/version.h
 create mode 100644 include/uapi/linux/wireguard.h
 create mode 100755 tools/testing/selftests/wireguard/netns.sh
MAINTAINERS +8 -0
index bd5847e802de..ce7fd2e7aa1a 100644
@@ -17850,6 +17850,14 @@ L:	[email protected]
 S:	Maintained
 F:	drivers/gpio/gpio-ws16c48.c
 
+WIREGUARD SECURE NETWORK TUNNEL
+M:	Jason A. Donenfeld <[email protected]>
+S:	Maintained
+F:	drivers/net/wireguard/
+F:	tools/testing/selftests/wireguard/
+L:	[email protected]
+L:	[email protected]
+
 WISTRON LAPTOP BUTTON DRIVER
 M:	Miloslav Trmac <[email protected]>
 S:	Maintained
drivers/net/Kconfig +41 -0

47 lines not on this page · raw message

drivers/net/Makefile +1 -0
index 0d3ba056cda3..953b7c12f0b0 100644
@@ -10,6 +10,7 @@ obj-$(CONFIG_BONDING) += bonding/
 obj-$(CONFIG_IPVLAN) += ipvlan/
 obj-$(CONFIG_IPVTAP) += ipvlan/
 obj-$(CONFIG_DUMMY) += dummy.o
+obj-$(CONFIG_WIREGUARD) += wireguard/
 obj-$(CONFIG_EQUALIZER) += eql.o
 obj-$(CONFIG_IFB) += ifb.o
 obj-$(CONFIG_MACSEC) += macsec.o
drivers/net/wireguard/Makefile new file +18 -0
new file mode 100644
index 000000000000..fc52b2cb500b
@@ -0,0 +1,18 @@
+ccflags-y := -O3
+ccflags-y += -D'pr_fmt(fmt)=KBUILD_MODNAME ": " fmt'
+ccflags-$(CONFIG_WIREGUARD_DEBUG) += -DDEBUG
+wireguard-y := main.o
+wireguard-y += noise.o
+wireguard-y += device.o
+wireguard-y += peer.o
+wireguard-y += timers.o
+wireguard-y += queueing.o
+wireguard-y += send.o
+wireguard-y += receive.o
+wireguard-y += socket.o
+wireguard-y += peerlookup.o
+wireguard-y += allowedips.o
+wireguard-y += ratelimiter.o
+wireguard-y += cookie.o
+wireguard-y += netlink.o
+obj-$(CONFIG_WIREGUARD) := wireguard.o
drivers/net/wireguard/allowedips.c new file +381 -0

381 lines not on this page · raw message

drivers/net/wireguard/allowedips.h new file +59 -0

59 lines not on this page · raw message

drivers/net/wireguard/cookie.c new file +236 -0

236 lines not on this page · raw message

drivers/net/wireguard/cookie.h new file +59 -0

59 lines not on this page · raw message

drivers/net/wireguard/device.c new file +458 -0

458 lines not on this page · raw message

drivers/net/wireguard/device.h new file +73 -0

73 lines not on this page · raw message

drivers/net/wireguard/main.c new file +64 -0

64 lines not on this page · raw message

drivers/net/wireguard/messages.h new file +128 -0

128 lines not on this page · raw message

drivers/net/wireguard/netlink.c new file +642 -0

642 lines not on this page · raw message

drivers/net/wireguard/netlink.h new file +12 -0
new file mode 100644
index 000000000000..15100d92e2e3
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#ifndef _WG_NETLINK_H
+#define _WG_NETLINK_H
+
+int wg_genetlink_init(void);
+void wg_genetlink_uninit(void);
+
+#endif /* _WG_NETLINK_H */
drivers/net/wireguard/noise.c new file +828 -0

802 lines not on this page · raw message

drivers/net/wireguard/noise.h new file +137 -0

137 lines not on this page · raw message

drivers/net/wireguard/peer.c new file +240 -0

240 lines not on this page · raw message

drivers/net/wireguard/peer.h new file +83 -0

83 lines not on this page · raw message

drivers/net/wireguard/peerlookup.c new file +221 -0

221 lines not on this page · raw message

drivers/net/wireguard/peerlookup.h new file +64 -0

64 lines not on this page · raw message

drivers/net/wireguard/queueing.c new file +53 -0

53 lines not on this page · raw message

drivers/net/wireguard/queueing.h new file +197 -0

197 lines not on this page · raw message

drivers/net/wireguard/ratelimiter.c new file +223 -0

223 lines not on this page · raw message

drivers/net/wireguard/ratelimiter.h new file +19 -0
new file mode 100644
index 000000000000..83067f71ea99

Lines of this file are not shown. View the raw message.

drivers/net/wireguard/receive.c new file +595 -0

595 lines not on this page · raw message

drivers/net/wireguard/selftest/allowedips.c new file +683 -0

683 lines not on this page · raw message

drivers/net/wireguard/selftest/counter.c new file +104 -0

104 lines not on this page · raw message

drivers/net/wireguard/selftest/ratelimiter.c new file +226 -0

226 lines not on this page · raw message

drivers/net/wireguard/send.c new file +413 -0

413 lines not on this page · raw message

drivers/net/wireguard/socket.c new file +437 -0

437 lines not on this page · raw message

drivers/net/wireguard/socket.h new file +44 -0

44 lines not on this page · raw message

drivers/net/wireguard/timers.c new file +243 -0

243 lines not on this page · raw message

drivers/net/wireguard/timers.h new file +31 -0
new file mode 100644
index 000000000000..f0653dcb1326

Lines of this file are not shown. View the raw message.

drivers/net/wireguard/version.h new file +1 -0
new file mode 100644
index 000000000000..a1a269a11634

Lines of this file are not shown. View the raw message.

include/uapi/linux/wireguard.h new file +196 -0

196 lines not on this page · raw message

tools/testing/selftests/wireguard/netns.sh new file +537 -0

537 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.24.0

Part of the diff is not shown here. View the raw message.

maintainer Applied
#1 lore raw

From: "Jason A. Donenfeld" <[email protected]>
Date: Mon, 9 Dec 2019 00:27:34 +0100

Quoted 8 lines

WireGuard is a layer 3 secure networking tunnel made specifically for the kernel, that aims to be much simpler and easier to audit than IPsec. Extensive documentation and description of the protocol and considerations, along with formal proofs of the cryptography, are available at:

* https://www.wireguard.com/
* https://www.wireguard.com/papers/wireguard.pdf

...

Applied, thanks Jason.

author maintainer
#2 lore raw
Quoted 10 lines · David Miller

From: "Jason A. Donenfeld" <[email protected]>
Date: Mon, 9 Dec 2019 00:27:34 +0100

Quoted 6 lines

WireGuard is a layer 3 secure networking tunnel made specifically for the kernel, that aims to be much simpler and easier to audit than IPsec. Extensive documentation and description of the protocol and considerations, along with formal proofs of the cryptography, are available at:

Quoted 5 lines
Quoted 2 lines

* https://www.wireguard.com/
* https://www.wireguard.com/papers/wireguard.pdf

...

Applied, thanks Jason.

Thank you, Dave!

#3 lore raw
In reply to Jason A. Donenfeld
Quoted 97 lines · Jason A. Donenfeld

WireGuard is a layer 3 secure networking tunnel made specifically for the kernel, that aims to be much simpler and easier to audit than IPsec. Extensive documentation and description of the protocol and considerations, along with formal proofs of the cryptography, are available at:

* https://www.wireguard.com/
* https://www.wireguard.com/papers/wireguard.pdf

This commit implements WireGuard as a simple network device driver, accessible in the usual RTNL way used by virtual network drivers. It makes use of the udp_tunnel APIs, GRO, GSO, NAPI, and the usual set of networking subsystem APIs. It has a somewhat novel multicore queueing system designed for maximum throughput and minimal latency of encryption operations, but it is implemented modestly using workqueues and NAPI. Configuration is done via generic Netlink, and following a review from the Netlink maintainer a year ago, several high profile userspace tools have already implemented the API.

This commit also comes with several different tests, both in-kernel tests and out-of-kernel tests based on network namespaces, taking profit of the fact that sockets used by WireGuard intentionally stay in the namespace the WireGuard interface was originally created, exactly like the semantics of userspace tun devices. See wireguard.com/netns/ for pictures and examples.

The source code is fairly short, but rather than combining everything into a single file, WireGuard is developed as cleanly separable files, making auditing and comprehension easier. Things are laid out as follows:

* noise.[ch], cookie.[ch], messages.h: These implement the bulk of the cryptographic aspects of the protocol, and are mostly data-only in nature, taking in buffers of bytes and spitting out buffers of bytes. They also handle reference counting for their various shared pieces of data, like keys and key lists.

* ratelimiter.[ch]: Used as an integral part of cookie.[ch] for ratelimiting certain types of cryptographic operations in accordance with particular WireGuard semantics.

* allowedips.[ch], peerlookup.[ch]: The main lookup structures of WireGuard, the former being trie-like with particular semantics, an integral part of the design of the protocol, and the latter just being nice helper functions around the various hashtables we use.

* device.[ch]: Implementation of functions for the netdevice and for rtnl, responsible for maintaining the life of a given interface and wiring it up to the rest of WireGuard.

* peer.[ch]: Each interface has a list of peers, with helper functions available here for creation, destruction, and reference counting.

* socket.[ch]: Implementation of functions related to udp_socket and the general set of kernel socket APIs, for sending and receiving ciphertext UDP packets, and taking care of WireGuard-specific sticky socket routing semantics for the automatic roaming.

* netlink.[ch]: Userspace API entry point for configuring WireGuard peers and devices. The API has been implemented by several userspace tools and network management utility, and the WireGuard project distributes the basic wg(8) tool.

* queueing.[ch]: Shared function on the rx and tx path for handling the various queues used in the multicore algorithms.

* send.c: Handles encrypting outgoing packets in parallel on multiple cores, before sending them in order on a single core, via workqueues and ring buffers. Also handles sending handshake and cookie messages as part of the protocol, in parallel.

* receive.c: Handles decrypting incoming packets in parallel on multiple cores, before passing them off in order to be ingested via the rest of the networking subsystem with GRO via the typical NAPI poll function. Also handles receiving handshake and cookie messages as part of the protocol, in parallel.

* timers.[ch]: Uses the timer wheel to implement protocol particular event timeouts, and gives a set of very simple event-driven entry point functions for callers.

* main.c, version.h: Initialization and deinitialization of the module.

* selftest/*.h: Runtime unit tests for some of the most security sensitive functions.

* tools/testing/selftests/wireguard/netns.sh: Aforementioned testing script using network namespaces.

This commit aims to be as self-contained as possible, implementing WireGuard as a standalone module not needing much special handling or coordination from the network subsystem. I expect for future optimizations to the network stack to positively improve WireGuard, and vice-versa, but for the time being, this exists as intentionally standalone.

Quoted 2 lines

We introduce a menu option for CONFIG_WIREGUARD, as well as providing a verbose debug log and self-tests via CONFIG_WIREGUARD_DEBUG.

Hi Jason, Dave,

Some late feedback on CONFIG_WIREGUARD_DEBUG.

Does it really do "verbose debug log"? I only see it is used for self-tests and debug checks:

linux$ grep DEBUG drivers/net/wireguard/*.c drivers/net/wireguard/allowedips.c: WARN_ON(IS_ENABLED(DEBUG) && *len >= 128); drivers/net/wireguard/allowedips.c: WARN_ON(IS_ENABLED(DEBUG) && len

Quoted 1 line

= 128); \

drivers/net/wireguard/main.c:#ifdef DEBUG drivers/net/wireguard/noise.c: WARN_ON(IS_ENABLED(DEBUG) &&

There are 3 different things:
- boot self-tests
- additional debug checks
- verbose logging

In different contexts one may enable different sets of these. In particular in fuzzing context one absolutely wants additional debug checks, but not self tests and definitely no verbose logging. CI and various manual scenarios will require different sets as well. If this does verbose logging, we won't get debug checks as well during fuzzing, which is unfortunate.
Can make sense splitting CONFIG_WIREGUARD_DEBUG into 2 or 3 separate configs (that's what I see frequently). Unfortunately there is no standard conventions for anything of this, so CIs will never find your boot tests and fuzzing won't find the additional checks...

author maintainer
#4 lore raw

Hi Dmitry,

Quoted 2 lines · Dmitry Vyukov

Does it really do "verbose debug log"? I only see it is used for self-tests and debug checks:

Yes, it does, via net_dbg and co. People with the Linux "dynamic debugging" feature turned also get the log by twiddling with some file at runtime.

Quoted 10 lines

In different contexts one may enable different sets of these. In particular in fuzzing context one absolutely wants additional debug checks, but not self tests and definitely no verbose logging. CI and various manual scenarios will require different sets as well. If this does verbose logging, we won't get debug checks as well during fuzzing, which is unfortunate.
Can make sense splitting CONFIG_WIREGUARD_DEBUG into 2 or 3 separate configs (that's what I see frequently). Unfortunately there is no standard conventions for anything of this, so CIs will never find your boot tests and fuzzing won't find the additional checks...

I agree that it might make sense to split this up at some point, but for now I think it might be a bit overkill. Both the self-tests and debug tests are *very* light at the moment. Down the road if these become heavier, I think it'd probably be a good idea, but for the time being it'd mostly be more complexity for nothing.

Another more interesting point, though, you wrote

Quoted 1 line

and definitely no verbose logging.

Actually with WireGuard, I think that's not the case. The WireGuard logging has been written with DoS in mind. You /should/ be able to safely run it on a production system exposed to the wild Internet, and while there will be some additional things in your dmesg, an attacker isn't supposed to be able to totally flood it without ratelimiting or inject malicious strings into it (such as ANSI escape sequence). In other words, I consider the logging to be fair game attack surface. If your fuzzer manages to craft some nasty sequence of packets that tricks some rate limiting logic and lets you litter all over dmesg totally unbounded, I'd consider that a real security bug worth stressing out about. So from the perspective of letting your fuzzers loose on WireGuard, I'd actually like to see this option kept on.

Jason

#5 lore raw
Quoted 31 lines · Jason A. Donenfeld

Hi Dmitry,

Quoted 2 lines · Dmitry Vyukov

Does it really do "verbose debug log"? I only see it is used for self-tests and debug checks:

Yes, it does, via net_dbg and co. People with the Linux "dynamic debugging" feature turned also get the log by twiddling with some file at runtime.

Quoted 10 lines

In different contexts one may enable different sets of these. In particular in fuzzing context one absolutely wants additional debug checks, but not self tests and definitely no verbose logging. CI and various manual scenarios will require different sets as well. If this does verbose logging, we won't get debug checks as well during fuzzing, which is unfortunate.
Can make sense splitting CONFIG_WIREGUARD_DEBUG into 2 or 3 separate configs (that's what I see frequently). Unfortunately there is no standard conventions for anything of this, so CIs will never find your boot tests and fuzzing won't find the additional checks...

I agree that it might make sense to split this up at some point, but for now I think it might be a bit overkill. Both the self-tests and debug tests are *very* light at the moment. Down the road if these become heavier, I think it'd probably be a good idea, but for the time being it'd mostly be more complexity for nothing.

Another more interesting point, though, you wrote

Quoted 1 line

and definitely no verbose logging.

Quoted 12 lines

Actually with WireGuard, I think that's not the case. The WireGuard logging has been written with DoS in mind. You /should/ be able to safely run it on a production system exposed to the wild Internet, and while there will be some additional things in your dmesg, an attacker isn't supposed to be able to totally flood it without ratelimiting or inject malicious strings into it (such as ANSI escape sequence). In other words, I consider the logging to be fair game attack surface. If your fuzzer manages to craft some nasty sequence of packets that tricks some rate limiting logic and lets you litter all over dmesg totally unbounded, I'd consider that a real security bug worth stressing out about. So from the perspective of letting your fuzzers loose on WireGuard, I'd actually like to see this option kept on.

This is the case even with CONFIG_WIREGUARD_DEBUG turned on, right? Or without?

Well, it may be able to trigger unbounded printing, but that won't be detected as a bug and won't be reported. To be reported it needs to fall into a set of predefined bug cases (e.g. "BUG:" or "WARNING:" on console). Unless of course it triggers total stall/hang. But I'm afraid it will just dirty dmesg, make reading crashes harder and slow down everything without benefit. O(1) output per test is generally not OK in heavy stressing scenario, even if it's overall bounded and rate limited.

author maintainer
#6 lore raw

Hi Dmitry,

Quoted 13 lines · Dmitry Vyukov
Quoted 4 lines

Actually with WireGuard, I think that's not the case. The WireGuard logging has been written with DoS in mind. You /should/ be able to safely run it on a production system exposed to the wild Internet, and while there will be some additional things in your dmesg, an attacker

Quoted 8 lines

isn't supposed to be able to totally flood it without ratelimiting or inject malicious strings into it (such as ANSI escape sequence). In other words, I consider the logging to be fair game attack surface. If your fuzzer manages to craft some nasty sequence of packets that tricks some rate limiting logic and lets you litter all over dmesg totally unbounded, I'd consider that a real security bug worth stressing out about. So from the perspective of letting your fuzzers loose on WireGuard, I'd actually like to see this option kept on.

Quoted 1 line

This is the case even with CONFIG_WIREGUARD_DEBUG turned on, right? Or without?

Turned on.

Quoted 4 lines

Well, it may be able to trigger unbounded printing, but that won't be detected as a bug and won't be reported. To be reported it needs to fall into a set of predefined bug cases (e.g. "BUG:" or "WARNING:" on console). Unless of course it triggers total stall/hang.

Bummer. Well, at least the stall case is interesting.

Quoted 3 lines

But I'm
afraid it will just dirty dmesg, make reading crashes harder and slow down everything without benefit.

Actually the point of the logging is usually to make it more obvious why a crash has come about, to provide some trail about the sequence of events. This was especially helpful in fixing old race conditions where subtle packet timing caused WireGuard's timer-based state machine to go haywire. Is syzkaller able to backtrack from crashes to the packets and packet timing that caused them, in order to make a test case to replay the crash? Is this precise enough for race condition bugs? If so, then when debugging the crashes I could always replay it later with logging turned on, in which case it might make sense to split out the debug logging into CONFIG_WIREGUARD_VERBOSE_LOG or similar (unless the logging itself changes the timing constraints and I can't repro that way). If this isn't possible, then it seems like logging might be something we would benefit from having in the crash reports, right? Or am I missing some other detail of how the system works?

Jason

#7 lore raw
Quoted 32 lines · Jason A. Donenfeld

Hi Dmitry,

Quoted 13 lines · Dmitry Vyukov
Quoted 4 lines

Actually with WireGuard, I think that's not the case. The WireGuard logging has been written with DoS in mind. You /should/ be able to safely run it on a production system exposed to the wild Internet, and while there will be some additional things in your dmesg, an attacker

Quoted 8 lines

isn't supposed to be able to totally flood it without ratelimiting or inject malicious strings into it (such as ANSI escape sequence). In other words, I consider the logging to be fair game attack surface. If your fuzzer manages to craft some nasty sequence of packets that tricks some rate limiting logic and lets you litter all over dmesg totally unbounded, I'd consider that a real security bug worth stressing out about. So from the perspective of letting your fuzzers loose on WireGuard, I'd actually like to see this option kept on.

Quoted 1 line

This is the case even with CONFIG_WIREGUARD_DEBUG turned on, right? Or without?

Turned on.

Quoted 4 lines

Well, it may be able to trigger unbounded printing, but that won't be detected as a bug and won't be reported. To be reported it needs to fall into a set of predefined bug cases (e.g. "BUG:" or "WARNING:" on console). Unless of course it triggers total stall/hang.

Bummer. Well, at least the stall case is interesting.

Quoted 3 lines

But I'm
afraid it will just dirty dmesg, make reading crashes harder and slow down everything without benefit.

Quoted 7 lines

Actually the point of the logging is usually to make it more obvious why a crash has come about, to provide some trail about the sequence of events. This was especially helpful in fixing old race conditions where subtle packet timing caused WireGuard's timer-based state machine to go haywire. Is syzkaller able to backtrack from crashes to the packets and packet timing that caused them, in order to make a test case to replay the crash?

Sometimes. You may sort by "Repro" column here to get the ratio:
https://syzkaller.appspot.com/upstream
https://syzkaller.appspot.com/upstream/fixed

Quoted 2 lines

Is this precise enough for race
condition bugs?

It's finding lots of race conditions provoked bugs (I would say it's the most common cause of kernel bugs).

Quoted 8 lines

If so, then when debugging the crashes I could always replay it later with logging turned on, in which case it might make sense to split out the debug logging into CONFIG_WIREGUARD_VERBOSE_LOG or similar (unless the logging itself changes the timing constraints and I can't repro that way). If this isn't possible, then it seems like logging might be something we would benefit from having in the crash reports, right? Or am I missing some other detail of how the system works?

Well, you are missing that wireguard is not the only subsystem syzkaller tests (in fact, it does not test it at all) and there are 3000 other subsystems :)
If we enable verbose debug logging for all of them, we will get storm of output and the wireguard logging you are interested in may simply be evicted from the 1MB buffer. Also, the expected case is that a program does not crash, in that case we waste performance for unnecessary logging (and not finding bugs because of that).

In some cases there are reproducers, in some cases a bug is trivial to debug based on the crash report (no tracing needed).

But additional debug checks are useful in any testing.

author maintainer
#8 lore raw
Quoted 5 lines · Dmitry Vyukov
Quoted 2 lines

Is this precise enough for race
condition bugs?

It's finding lots of race conditions provoked bugs (I would say it's the most common cause of kernel bugs).

I meant -- are the reproducers it makes precise enough to retrigger network-level race conditions?

Quoted 3 lines

Well, you are missing that wireguard is not the only subsystem syzkaller tests (in fact, it does not test it at all) and there are 3000 other subsystems :)

Oooo! Everything is tested at the same time. I understand now; that makes a lot more sense.

I'll look into splitting out the option, as you've asked. Note, though, that there are currently only three spots that have the "extra checks" at the moment, and one of them can be optimized out by the compiler with aggressive enough inlining added everywhere. The other two will result in an immediately corrupted stack frame that should be caught immediately by other things. So for now, I think you can get away with turning the debug option off, and you won't be missing much from the "extra checks", at least until we add more.

That's exciting about syzcaller having at it with WireGuard. Is there some place where I can "see" it fuzzing WireGuard, or do I just wait for the bug reports to come rolling in?

Jason

#10 lore raw
Quoted 10 lines · Jason A. Donenfeld
Quoted 5 lines · Jason A. Donenfeld

On Thu, Dec 19, 2019 at 10:35 AM Dmitry Vyukov <[email protected]> wrote:
That's exciting about syzcaller having at it with WireGuard. Is there some place where I can "see" it fuzzing WireGuard, or do I just wait for the bug reports to come rolling in?

Ahh, found it: https://storage.googleapis.com/syzkaller/cover/ci-upstream-net-kasan-gce.html
Looks like we're at 1% and counting. :)

Yes, that's it. But that's mostly stray coverage. wg_netdevice_notification I guess mostly because it tested _other_ device types. And a bit of netlink because it sends random garbage into netlink.

For netlink part it would require something along these lines:
https://github.com/google/syzkaller/blob/master/sys/linux/socket_netlink_generic_devlink.txt
https://github.com/google/syzkaller/blob/master/sys/linux/socket_netlink_crypto.txt
https://github.com/google/syzkaller/blob/master/sys/linux/socket_netlink_generic_fou.txt
https://github.com/google/syzkaller/blob/master/sys/linux/socket_netlink_generic_seg6.txt

And for device setup, harder to say. Either pre-create one here:
https://github.com/google/syzkaller/blob/79b211f74b08737aeb4934c6ff69a263b3c38013/executor/common_linux.h#L668
or teach it how to create them on the fly or both or something else.

Probably some wire packet formats here:
https://github.com/google/syzkaller/blob/79b211f74b08737aeb4934c6ff69a263b3c38013/sys/linux/vnet.txt

author maintainer
#11 lore raw
Quoted 23 lines · Dmitry Vyukov
Quoted 10 lines · Jason A. Donenfeld
Quoted 5 lines · Jason A. Donenfeld

On Thu, Dec 19, 2019 at 10:35 AM Dmitry Vyukov <[email protected]> wrote:
That's exciting about syzcaller having at it with WireGuard. Is there some place where I can "see" it fuzzing WireGuard, or do I just wait for the bug reports to come rolling in?

Ahh, found it: https://storage.googleapis.com/syzkaller/cover/ci-upstream-net-kasan-gce.html
Looks like we're at 1% and counting. :)

Yes, that's it. But that's mostly stray coverage. wg_netdevice_notification I guess mostly because it tested _other_ device types. And a bit of netlink because it sends random garbage into netlink.

For netlink part it would require something along these lines:
https://github.com/google/syzkaller/blob/master/sys/linux/socket_netlink_generic_devlink.txt
https://github.com/google/syzkaller/blob/master/sys/linux/socket_netlink_crypto.txt
https://github.com/google/syzkaller/blob/master/sys/linux/socket_netlink_generic_fou.txt
https://github.com/google/syzkaller/blob/master/sys/linux/socket_netlink_generic_seg6.txt

Quoted 6 lines

And for device setup, harder to say. Either pre-create one here:
https://github.com/google/syzkaller/blob/79b211f74b08737aeb4934c6ff69a263b3c38013/executor/common_linux.h#L668
or teach it how to create them on the fly or both or something else.

Probably some wire packet formats here:
https://github.com/google/syzkaller/blob/79b211f74b08737aeb4934c6ff69a263b3c38013/sys/linux/vnet.txt

Ahh, cool, okay. Netlink, device creation, and basic packet structure is a good start. What about the crypto, though?

#12 lore raw
Quoted 29 lines · Jason A. Donenfeld
Quoted 27 lines · Dmitry Vyukov
Quoted 10 lines · Jason A. Donenfeld
Quoted 5 lines · Jason A. Donenfeld

On Thu, Dec 19, 2019 at 10:35 AM Dmitry Vyukov <[email protected]> wrote:
That's exciting about syzcaller having at it with WireGuard. Is there some place where I can "see" it fuzzing WireGuard, or do I just wait for the bug reports to come rolling in?

Ahh, found it: https://storage.googleapis.com/syzkaller/cover/ci-upstream-net-kasan-gce.html
Looks like we're at 1% and counting. :)

Yes, that's it. But that's mostly stray coverage. wg_netdevice_notification I guess mostly because it tested _other_ device types. And a bit of netlink because it sends random garbage into netlink.

For netlink part it would require something along these lines:
https://github.com/google/syzkaller/blob/master/sys/linux/socket_netlink_generic_devlink.txt
https://github.com/google/syzkaller/blob/master/sys/linux/socket_netlink_crypto.txt
https://github.com/google/syzkaller/blob/master/sys/linux/socket_netlink_generic_fou.txt
https://github.com/google/syzkaller/blob/master/sys/linux/socket_netlink_generic_seg6.txt

And for device setup, harder to say. Either pre-create one here:
https://github.com/google/syzkaller/blob/79b211f74b08737aeb4934c6ff69a263b3c38013/executor/common_linux.h#L668
or teach it how to create them on the fly or both or something else.

Quoted 5 lines
Quoted 2 lines

Probably some wire packet formats here:
https://github.com/google/syzkaller/blob/79b211f74b08737aeb4934c6ff69a263b3c38013/sys/linux/vnet.txt

Ahh, cool, okay. Netlink, device creation, and basic packet structure is a good start. What about the crypto, though?

It depends. What exactly we need there?
syzkaller uses comparison operand interception which allows it e.g. to guess signatures/checksums in some cases.
We also have this special checksum type in descriptions:
https://github.com/google/syzkaller/blob/79b211f74b08737aeb4934c6ff69a263b3c38013/sys/linux/vnet.txt#L462
which means "fill in other data, then calculate the checksum and put in that location".
Other things can be done in a similar way, e.g. some crypto. But that requires more effort and leads to a tradeoff between effort/gain. Generally we like smaller set of more generic constructs in descriptions that can help with a set of different kernel interfaces. But if it's lots of custom support for a single very special case, then it becomes somewhat questionable.
But we have some ugly hack for such cases as well, e.g. for netfilter we just say "screw it, so many special one-off cases, so we will just write one-off procedural code":
https://github.com/google/syzkaller/blob/79b211f74b08737aeb4934c6ff69a263b3c38013/sys/linux/init_iptables.go
But generally we try to avoid that (and/or later replace ugly custom code with generic constructs).

author maintainer
#13 lore raw
Quoted 6 lines · Dmitry Vyukov
Quoted 2 lines

Ahh, cool, okay. Netlink, device creation, and basic packet structure is a good start. What about the crypto, though?

It depends. What exactly we need there?
syzkaller uses comparison operand interception which allows it e.g. to guess signatures/checksums in some cases.

I don't think you'll have too much luck with WireGuard here. Fuzzing your way to a valid handshake message involves guessing the 4th preimage of some elliptic curve scalar multiplication, with some random/changing data mixed in there every time you make a new try. There's a condensed protocol description here which should be less bad to glance at than the academic paper:
https://www.wireguard.com/protocol/#first-message-initiator-to-responder
. The fuzzers I've written for the crypto bits of WireGuard always involve taking a complete handshake implementation and mutating things from there. So maybe the "outer packet" won't be too fruitful without a bunch of work. At the very least, we can generate packets that have the right field sizes and such, and that should test the first level of error cases I guess.

However, there's still a decent amount of surface on the "inner packet". For this, we can set up a pair of wireguard interfaces that are preconfigured to talk to each other (in common_linux.h, right? Or do you have some Go file that'd be easier to do that initialization in?), and then syzkaller will figure out itself how to send nasty IP packets through them with send/recv and such. There's a bit of surface here because sending packets provokes the aforementioned handshake, and also moves around the timer state machine. The receiver also needs to do some minimal parsing of the received packet to check "allowedips". So, good fodder for fuzzing.

#14 lore raw
Quoted 9 lines · Jason A. Donenfeld
Quoted 6 lines · Dmitry Vyukov
Quoted 2 lines

Ahh, cool, okay. Netlink, device creation, and basic packet structure is a good start. What about the crypto, though?

It depends. What exactly we need there?
syzkaller uses comparison operand interception which allows it e.g. to guess signatures/checksums in some cases.

Quoted 13 lines

I don't think you'll have too much luck with WireGuard here. Fuzzing your way to a valid handshake message involves guessing the 4th preimage of some elliptic curve scalar multiplication, with some random/changing data mixed in there every time you make a new try. There's a condensed protocol description here which should be less bad to glance at than the academic paper:
https://www.wireguard.com/protocol/#first-message-initiator-to-responder
. The fuzzers I've written for the crypto bits of WireGuard always involve taking a complete handshake implementation and mutating things from there. So maybe the "outer packet" won't be too fruitful without a bunch of work. At the very least, we can generate packets that have the right field sizes and such, and that should test the first level of error cases I guess.

Yes, properly formed packets will probably require some procedural support. I can't say right away which exactly extension mechanism in syzkaller is the best for this.
But there is another one I forgot to mention - custom pseudo syscalls. For the generic fuzzer engine they look like normal syscalls that accept/return something:
https://github.com/google/syzkaller/blob/36650b4b2c942bc382314dce384d311fbadd1208/sys/linux/vnet.txt#L26
but the actual implementation is our custom C code that can do anything (augment data, form packets, call multiple syscalls, take result of one syscall, do something with it and use as argument for another, etc):
https://github.com/google/syzkaller/blob/36650b4b2c942bc382314dce384d311fbadd1208/executor/common_linux.h#L972-L1023
These make doing a complex thing trivial for fuzzer, but the more is hardcoded there, the less randomness we get from fuzzing (though that presudo-syscall can also accept some additional randomness and use it in some way).

Quoted 10 lines

However, there's still a decent amount of surface on the "inner packet". For this, we can set up a pair of wireguard interfaces that are preconfigured to talk to each other (in common_linux.h, right? Or do you have some Go file that'd be easier to do that initialization in?), and then syzkaller will figure out itself how to send nasty IP packets through them with send/recv and such. There's a bit of surface here because sending packets provokes the aforementioned handshake, and also moves around the timer state machine. The receiver also needs to do some minimal parsing of the received packet to check "allowedips". So, good fodder for fuzzing.

Yes, there is this location to pre-create some net devices in right states:
https://github.com/google/syzkaller/blob/79b211f74b08737aeb4934c6ff69a263b3c38013/executor/common_linux.h#L668
(not in Go, but in C because eventually it will become part of C reproducers). For example it creates vcan0 device and then syscall descriptions know this name and use it.

author maintainer
#15 lore raw

syzkaller wireguard key situation [was: Re: [PATCH net-next v2] net: WireGuard secure network tunnel]

Hey Dmitry,

I see you got wireguard's netlink stuff hooked up to syzkaller. Excellent work, and thanks! It's already finding bugs.

Right now it seems to know about 5 different keys you've come up with, and not much in the way of endpoints. I think we can improve this.

For keys, there are a few cases we care about:

1) Low order keys
2) Negative keys
3) Normal keys
4) Keys that correspond to other keys (private ==> public)

For this last point, if we just have a few with that correspondance quality in there, syzkaller will eventually wind up configuring two interfaces that can talk to each other, which is good. Here's a collection of keys you can use, in base64, that will cover those cases, if you want to add these instead of the current ones in there:

1)
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
4Ot6fDtBuK4WVuP68Z/EatoJjeucMrH9hmIFFl9JuAA=

2)
2/////////////////////////////////////////8=
TJyVvKNQjCSx0LFVnIPvWwREXMRYHI6G2CJO3dCfEdc=

3,4)
oFyoT2ycjjhT4v16cK4Psg+hUmAMsAhFF08IB2+NeEM=
l1ydgcmDyCCe54ElS4mfjtklrp8JI8I8YvU8V82/aRw=
sIBz6NROkePakiwiQ4JEu4hcaeJpyOnYNbEUKTpN3G4=
0XMomfYRzYmUA01/QT3JV2MOVJPChaykAGXLYxG+aWs=
oMuHmkf1vGRMDmk/ptAxx0oVU7bpAbn/L1GMeAQvtUI=
9E2jZ6iO5lZPAgIRRWcnCC9c6+6LG/Xrczc0G0WbOSI=

That's 10 keys total, which should be a decent collection to replace your current set of hard coded keys in there. You can unbase64 these into C format with commands like:

$ echo '2/////////////////////////////////////////8=' | base64 -d | xxd -i
  0xdb, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
  0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
  0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff

The second thing is getting two wireguard interfaces to talk to each other. This probably should happen over localhost. That means the listen port of one should be the endpoint of the other. So maybe you can get away fuzzing these with:

Listen ports:
51820
51821
51822
[randomly selected]

and

Endpoints:
127.0.0.1:51820
127.0.0.1:51821
127.0.0.1:51822
[::1]:51820
[::1]:51821
[::1]:51822
[randomly selected]

Finally the "allowed ips" for a peer, the routing table entry that points to wireguard, and the packet that's being sent, should all somehow correspond. But probably an allowed ips of 0.0.0.0/0 will eventually be fuzzed to, which covers everything for the first part, so let's see if the rest falls into place on its own.

What do you think of all that?

Jason

#16 lore raw

Re: syzkaller wireguard key situation [was: Re: [PATCH net-next v2] net: WireGuard secure network tunnel]

Quoted 75 lines · Jason A. Donenfeld

Hey Dmitry,

I see you got wireguard's netlink stuff hooked up to syzkaller. Excellent work, and thanks! It's already finding bugs.

Right now it seems to know about 5 different keys you've come up with, and not much in the way of endpoints. I think we can improve this.

For keys, there are a few cases we care about:

1) Low order keys
2) Negative keys
3) Normal keys
4) Keys that correspond to other keys (private ==> public)

For this last point, if we just have a few with that correspondance quality in there, syzkaller will eventually wind up configuring two interfaces that can talk to each other, which is good. Here's a collection of keys you can use, in base64, that will cover those cases, if you want to add these instead of the current ones in there:

1)
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
4Ot6fDtBuK4WVuP68Z/EatoJjeucMrH9hmIFFl9JuAA=

2)
2/////////////////////////////////////////8=
TJyVvKNQjCSx0LFVnIPvWwREXMRYHI6G2CJO3dCfEdc=

3,4)
oFyoT2ycjjhT4v16cK4Psg+hUmAMsAhFF08IB2+NeEM=
l1ydgcmDyCCe54ElS4mfjtklrp8JI8I8YvU8V82/aRw=
sIBz6NROkePakiwiQ4JEu4hcaeJpyOnYNbEUKTpN3G4=
0XMomfYRzYmUA01/QT3JV2MOVJPChaykAGXLYxG+aWs=
oMuHmkf1vGRMDmk/ptAxx0oVU7bpAbn/L1GMeAQvtUI=
9E2jZ6iO5lZPAgIRRWcnCC9c6+6LG/Xrczc0G0WbOSI=

That's 10 keys total, which should be a decent collection to replace your current set of hard coded keys in there. You can unbase64 these into C format with commands like:

$ echo '2/////////////////////////////////////////8=' | base64 -d | xxd -i
  0xdb, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
  0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
  0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff

The second thing is getting two wireguard interfaces to talk to each other. This probably should happen over localhost. That means the listen port of one should be the endpoint of the other. So maybe you can get away fuzzing these with:

Listen ports:
51820
51821
51822
[randomly selected]

and

Endpoints:
127.0.0.1:51820
127.0.0.1:51821
127.0.0.1:51822
[::1]:51820
[::1]:51821
[::1]:51822
[randomly selected]

Finally the "allowed ips" for a peer, the routing table entry that points to wireguard, and the packet that's being sent, should all somehow correspond. But probably an allowed ips of 0.0.0.0/0 will eventually be fuzzed to, which covers everything for the first part, so let's see if the rest falls into place on its own.

Quoted 3 lines

What do you think of all that?

Jason

Hi Jason,

[getting through backlog after a tip...]

I think you addressed all of this by now, right? And we got decent coverage of wireguard. Anything else low hanging left?

https://github.com/google/syzkaller/commit/2c71f1a9122cc3cb0abacbbec6359c40db02be35
https://github.com/google/syzkaller/commit/4d1ab643be2091f794ec55d83ec8acf7b0a60be3
https://github.com/google/syzkaller/commit/c5ed587f4af5e639f7373d8ebf10ac049cb9c71b

Thanks!

author maintainer
#17 lore raw

Re: syzkaller wireguard key situation [was: Re: [PATCH net-next v2] net: WireGuard secure network tunnel]

Hey Dmitry,

Yes! Our side discussions wound up getting everything pretty squared away, and coverage on syzkaller looks pretty good to me. By inference, I think we're hitting most code paths in WireGuard. Syzkaller, though, is missing non-userspace-process coverage from:

- workqueues
- napi callback
- timer callback
- udp tunnel callback

Seems like there might be some future research to be done on how we can track these. But tracking it or not, the fact that packets are flowing on some path implies that other code paths are being hit. So I feel pretty good about syzkaller's ability to dig up nice wireguard bugs.

Jason

author maintainer
#18 lore raw

Re: syzkaller wireguard key situation [was: Re: [PATCH net-next v2] net: WireGuard secure network tunnel]

Observation:

It seems to be starting to synthesize packets sent to the wireguard socket. These aren't the proper handshake packets generated internally by that triangle commit, but rather ones that syzkaller creates itself. That's why we have coverage on wg_receive, which otherwise wouldn't be called from a userspace process, since syzbot is sending its own packets to that function.

However, the packets it generates aren't getting very far, failing all of the tests in validate_header_len. None of those checks are at all cryptographic, which means it should be able to hit those eventually. Anything we should be doing to help it out? After it gets past that check, it'll wind up in the handshake queue or the data queue, and then (in theory) it should be rejected on a cryptographic basis. But maybe syzbot will figure out how to crash it instead :-P.

#19 lore raw

Re: syzkaller wireguard key situation [was: Re: [PATCH net-next v2] net: WireGuard secure network tunnel]

Quoted 10 lines · Jason A. Donenfeld

Observation:

It seems to be starting to synthesize packets sent to the wireguard socket. These aren't the proper handshake packets generated internally by that triangle commit, but rather ones that syzkaller creates itself. That's why we have coverage on wg_receive, which otherwise wouldn't be called from a userspace process, since syzbot is sending its own packets to that function.

Quoted 7 lines

However, the packets it generates aren't getting very far, failing all of the tests in validate_header_len. None of those checks are at all cryptographic, which means it should be able to hit those eventually. Anything we should be doing to help it out? After it gets past that check, it'll wind up in the handshake queue or the data queue, and then (in theory) it should be rejected on a cryptographic basis. But maybe syzbot will figure out how to crash it instead :-P.

Looking into this.

Found the program that gives wg_receive coverage:

r0 = openat$tun(0xffffffffffffff9c, &(0x7f0000000080)='/dev/net/tun\x00', 0x88002, 0x0) ioctl$TUNSETIFF(r0, 0x400454ca, &(0x7f00000000c0)={'syzkaller1\x00', 0x420000015001})
r1 = socket$netlink(0x10, 0x3, 0x0) ioctl$sock_inet_SIOCSIFADDR(r1, 0x8914, &(0x7f0000000140)={'syzkaller1\x00', {0x7, 0x0, @empty}}) write$tun(r0, &(0x7f00000002c0)={@void, @val, @ipv4=@udp={{0x5, 0x4, 0x0, 0x0, 0x1c, 0x0, 0x0, 0x0, 0x11, 0x0, @remote, @broadcast}, {0x0, 0x4e21, 0x8}}}, 0x26)

Checked that doing SIOCSIFADDR is also required, otherwise the packet does not reach wg_receive.

#20 lore raw

Re: syzkaller wireguard key situation [was: Re: [PATCH net-next v2] net: WireGuard secure network tunnel]

Quoted 35 lines · Dmitry Vyukov
Quoted 10 lines · Jason A. Donenfeld

Observation:

It seems to be starting to synthesize packets sent to the wireguard socket. These aren't the proper handshake packets generated internally by that triangle commit, but rather ones that syzkaller creates itself. That's why we have coverage on wg_receive, which otherwise wouldn't be called from a userspace process, since syzbot is sending its own packets to that function.

Quoted 7 lines

However, the packets it generates aren't getting very far, failing all of the tests in validate_header_len. None of those checks are at all cryptographic, which means it should be able to hit those eventually. Anything we should be doing to help it out? After it gets past that check, it'll wind up in the handshake queue or the data queue, and then (in theory) it should be rejected on a cryptographic basis. But maybe syzbot will figure out how to crash it instead :-P.

Looking into this.

Found the program that gives wg_receive coverage:

r0 = openat$tun(0xffffffffffffff9c, &(0x7f0000000080)='/dev/net/tun\x00', 0x88002, 0x0) ioctl$TUNSETIFF(r0, 0x400454ca, &(0x7f00000000c0)={'syzkaller1\x00', 0x420000015001})
r1 = socket$netlink(0x10, 0x3, 0x0) ioctl$sock_inet_SIOCSIFADDR(r1, 0x8914, &(0x7f0000000140)={'syzkaller1\x00', {0x7, 0x0, @empty}}) write$tun(r0, &(0x7f00000002c0)={@void, @val, @ipv4=@udp={{0x5, 0x4, 0x0, 0x0, 0x1c, 0x0, 0x0, 0x0, 0x11, 0x0, @remote, @broadcast}, {0x0, 0x4e21, 0x8}}}, 0x26)

Quoted 2 lines

Checked that doing SIOCSIFADDR is also required, otherwise the packet does not reach wg_receive.

All packets we inject with standard means (syz_emit_ethernet) get rejected on the following check:

static struct sk_buff *ip_rcv_core(struct sk_buff *skb, struct net *net) {
const struct iphdr *iph;
u32 len;

/* When the interface is in promisc. mode, drop all the crap
* that it receives, do not try to analyse it.
*/
if (skb->pkt_type == PACKET_OTHERHOST)
goto drop;

Even if we drop IFF_NAPI_FRAGS which diverges packets who-knows-where.

Somehow we need to get something other than PACKET_OTHERHOST... Why is it dropping all remote packets?...
How do remote packets get into stack then?...
_______________________________________________
WireGuard mailing list
[email protected]
https://lists.zx2c4.com/mailman/listinfo/wireguard

#21 lore raw

Re: syzkaller wireguard key situation [was: Re: [PATCH net-next v2] net: WireGuard secure network tunnel]

Quoted 51 lines · Dmitry Vyukov
Quoted 33 lines
Quoted 10 lines

Observation:

It seems to be starting to synthesize packets sent to the wireguard socket. These aren't the proper handshake packets generated internally by that triangle commit, but rather ones that syzkaller creates itself. That's why we have coverage on wg_receive, which otherwise wouldn't be called from a userspace process, since syzbot is sending its own packets to that function.

Quoted 7 lines

However, the packets it generates aren't getting very far, failing all of the tests in validate_header_len. None of those checks are at all cryptographic, which means it should be able to hit those eventually. Anything we should be doing to help it out? After it gets past that check, it'll wind up in the handshake queue or the data queue, and then (in theory) it should be rejected on a cryptographic basis. But maybe syzbot will figure out how to crash it instead :-P.

Looking into this.

Found the program that gives wg_receive coverage:

r0 = openat$tun(0xffffffffffffff9c, &(0x7f0000000080)='/dev/net/tun\x00', 0x88002, 0x0) ioctl$TUNSETIFF(r0, 0x400454ca, &(0x7f00000000c0)={'syzkaller1\x00', 0x420000015001})
r1 = socket$netlink(0x10, 0x3, 0x0) ioctl$sock_inet_SIOCSIFADDR(r1, 0x8914, &(0x7f0000000140)={'syzkaller1\x00', {0x7, 0x0, @empty}}) write$tun(r0, &(0x7f00000002c0)={@void, @val, @ipv4=@udp={{0x5, 0x4, 0x0, 0x0, 0x1c, 0x0, 0x0, 0x0, 0x11, 0x0, @remote, @broadcast}, {0x0, 0x4e21, 0x8}}}, 0x26)

Quoted 2 lines

Checked that doing SIOCSIFADDR is also required, otherwise the packet does not reach wg_receive.

All packets we inject with standard means (syz_emit_ethernet) get rejected on the following check:

static struct sk_buff *ip_rcv_core(struct sk_buff *skb, struct net *net) {
const struct iphdr *iph;
u32 len;

/* When the interface is in promisc. mode, drop all the crap
* that it receives, do not try to analyse it.
*/
if (skb->pkt_type == PACKET_OTHERHOST)
goto drop;

Quoted 5 lines

Even if we drop IFF_NAPI_FRAGS which diverges packets who-knows-where.

Somehow we need to get something other than PACKET_OTHERHOST... Why is it dropping all remote packets?...
How do remote packets get into stack then?...

I've managed to create a packet that reaches wg_receive, that is:

syz_emit_ethernet(AUTO, &AUTO={@local, @empty, @void, {@ipv4={AUTO, @udp={{AUTO, AUTO, 0x0, 0x0, AUTO, 0x0, 0x0, 0x0, AUTO, 0x0, @empty, @empty, {[]}}, {0x0, 0x4e22, AUTO, 0x0, [], ""/10}}}}}, 0x0)

Had to enumerate all possible combinations of local/remote mac, local/report ip, local/remote port.

However, this is only without IFF_NAPI_FRAGS. With IFF_NAPI_FRAGS it reaches udp_gro_receive, but does not get past:

if (!sk || NAPI_GRO_CB(skb)->encap_mark ||

    (skb->ip_summed != CHECKSUM_PARTIAL &&
     NAPI_GRO_CB(skb)->csum_cnt == 0 &&
     !NAPI_GRO_CB(skb)->csum_valid) ||
    !udp_sk(sk)->gro_receive)
    goto out;
#22 lore raw

Re: syzkaller wireguard key situation [was: Re: [PATCH net-next v2] net: WireGuard secure network tunnel]

Quoted 71 lines · Dmitry Vyukov
Quoted 51 lines · Dmitry Vyukov
Quoted 33 lines
Quoted 10 lines

Observation:

It seems to be starting to synthesize packets sent to the wireguard socket. These aren't the proper handshake packets generated internally by that triangle commit, but rather ones that syzkaller creates itself. That's why we have coverage on wg_receive, which otherwise wouldn't be called from a userspace process, since syzbot is sending its own packets to that function.

Quoted 7 lines

However, the packets it generates aren't getting very far, failing all of the tests in validate_header_len. None of those checks are at all cryptographic, which means it should be able to hit those eventually. Anything we should be doing to help it out? After it gets past that check, it'll wind up in the handshake queue or the data queue, and then (in theory) it should be rejected on a cryptographic basis. But maybe syzbot will figure out how to crash it instead :-P.

Looking into this.

Found the program that gives wg_receive coverage:

r0 = openat$tun(0xffffffffffffff9c, &(0x7f0000000080)='/dev/net/tun\x00', 0x88002, 0x0) ioctl$TUNSETIFF(r0, 0x400454ca, &(0x7f00000000c0)={'syzkaller1\x00', 0x420000015001})
r1 = socket$netlink(0x10, 0x3, 0x0) ioctl$sock_inet_SIOCSIFADDR(r1, 0x8914, &(0x7f0000000140)={'syzkaller1\x00', {0x7, 0x0, @empty}}) write$tun(r0, &(0x7f00000002c0)={@void, @val, @ipv4=@udp={{0x5, 0x4, 0x0, 0x0, 0x1c, 0x0, 0x0, 0x0, 0x11, 0x0, @remote, @broadcast}, {0x0, 0x4e21, 0x8}}}, 0x26)

Quoted 2 lines

Checked that doing SIOCSIFADDR is also required, otherwise the packet does not reach wg_receive.

All packets we inject with standard means (syz_emit_ethernet) get rejected on the following check:

static struct sk_buff *ip_rcv_core(struct sk_buff *skb, struct net *net) {
const struct iphdr *iph;
u32 len;

/* When the interface is in promisc. mode, drop all the crap
* that it receives, do not try to analyse it.
*/
if (skb->pkt_type == PACKET_OTHERHOST)
goto drop;

Quoted 5 lines

Even if we drop IFF_NAPI_FRAGS which diverges packets who-knows-where.

Somehow we need to get something other than PACKET_OTHERHOST... Why is it dropping all remote packets?...
How do remote packets get into stack then?...

I've managed to create a packet that reaches wg_receive, that is:

syz_emit_ethernet(AUTO, &AUTO={@local, @empty, @void, {@ipv4={AUTO, @udp={{AUTO, AUTO, 0x0, 0x0, AUTO, 0x0, 0x0, 0x0, AUTO, 0x0, @empty, @empty, {[]}}, {0x0, 0x4e22, AUTO, 0x0, [], ""/10}}}}}, 0x0)

Had to enumerate all possible combinations of local/remote mac, local/report ip, local/remote port.

However, this is only without IFF_NAPI_FRAGS. With IFF_NAPI_FRAGS it reaches udp_gro_receive, but does not get past:

Quoted 6 lines

if (!sk || NAPI_GRO_CB(skb)->encap_mark ||

    (skb->ip_summed != CHECKSUM_PARTIAL &&
     NAPI_GRO_CB(skb)->csum_cnt == 0 &&
     !NAPI_GRO_CB(skb)->csum_valid) ||
    !udp_sk(sk)->gro_receive)
    goto out;

I've added descriptions for wireguard packets:
https://github.com/google/syzkaller/commit/012fbc3229ebef871a201ea431b16610e6e0d345
It gives all reachable coverage (without breaking crypto).

Strictly saying, for tcp we experimented with receiving ACKs back from tun and exposing them to fuzzer to form proper SYNACKs:
https://github.com/google/syzkaller/blob/012fbc3229ebef871a201ea431b16610e6e0d345/executor/common_linux.h#L1390-L1441
https://github.com/google/syzkaller/blob/012fbc3229ebef871a201ea431b16610e6e0d345/sys/linux/vnet.txt#L24-L27

Theoretically, it could receive wireguard handshakes and form proper replies with valid signatures and stuff.

I disabled IFF_NAPI_FRAGS entirely, it seems to prevent from getting any meaningful coverage:
https://github.com/google/syzkaller/commit/39cd0f85a1ac60b88c793bd8f4a981227614da88

author maintainer
#23 lore raw

Re: syzkaller wireguard key situation [was: Re: [PATCH net-next v2] net: WireGuard secure network tunnel]

Quoted 3 lines · Dmitry Vyukov

I've added descriptions for wireguard packets:
https://github.com/google/syzkaller/commit/012fbc3229ebef871a201ea431b16610e6e0d345
It gives all reachable coverage (without breaking crypto).

Oh, great, that looks really good. It now fails at the index match, which is basically trying to brute force a 32-bit integer that's changing every 3 minutes, which syzkaller is probably too slow to do on it's own. But that's fine.

Your commit has some questions in it, like "# Not clear if these indexes are also generated randomly and we need to guess them or not.". Here's what's up with those indices:

Message message_handshake_initiation: the sender picks a random 32-bit number, and places it in the "sender_index" field.
Message message_handshake_response: the sender picks a random 32-bit number, and places it in the "sender_index" field. It places the 32-bit number that it received from message_handshake_initiation into the "receiver_index" field.
Message message_handshake_cookie: the sender places the 32-bit number that it received from message_handshake_initiation or message_handshake_response into the "receive_index" field. Message message_data: the sender places the 32-bit number that it picked for the message_handshake_initiation or message_handshake_response into the "key_idx" field.

I'm not sure it'll be too feasible to correlate these relations via fuzzing. And either way, I think we've got decent enough non-crypto coverage now on the receive path.

I noticed a small seemingly insignificant function with low coverage that's on the rtnl path that might benefit from some attention and also help find bugs in other devices: wg_netdevice_notification. This triggers on various things, but the only case it really cares about is NETDEV_REGISTER, which happens when the interface changes network namespaces. WireGuard holds a reference to its underlying creating namespace, and in order to avoid circular reference counting or UaF it needs to either relinquish or get a reference. There are other drivers with similar type of reference counting management, I would assume. This sort of thing seems up the ally of the types of bugs and races syzkaller likes to find. The way to trigger it is with ip link set dev wg0 netns blah, and then back to its original netns. That's this code in net/core/rtnetlink.c:

       if (tb[IFLA_NET_NS_PID] || tb[IFLA_NET_NS_FD] ||
tb[IFLA_TARGET_NETNSID]) {
               struct net *net = rtnl_link_get_net_capable(skb, dev_net(dev),
                                                           tb, CAP_NET_ADMIN);
               if (IS_ERR(net)) {
                       err = PTR_ERR(net);
                       goto errout;
               }

               err = dev_change_net_namespace(dev, net, ifname);
               put_net(net);
               if (err)
                       goto errout;
               status |= DO_SETLINK_MODIFIED;
       }

That seems to have decent coverage, but not over wireguard. Is that just a result of the syzkaller @devname not yet being expanded to wg{0,1,2}, and it'll take a few more weeks for it to learn that? @netns_id seems probably good, being restricted to 0:4; is it possible these weren't created though a priori?
_______________________________________________
WireGuard mailing list
[email protected]
https://lists.zx2c4.com/mailman/listinfo/wireguard

#24 lore raw

Re: syzkaller wireguard key situation [was: Re: [PATCH net-next v2] net: WireGuard secure network tunnel]

Quoted 62 lines · Jason A. Donenfeld
Quoted 3 lines · Dmitry Vyukov

I've added descriptions for wireguard packets:
https://github.com/google/syzkaller/commit/012fbc3229ebef871a201ea431b16610e6e0d345
It gives all reachable coverage (without breaking crypto).

Oh, great, that looks really good. It now fails at the index match, which is basically trying to brute force a 32-bit integer that's changing every 3 minutes, which syzkaller is probably too slow to do on it's own. But that's fine.

Your commit has some questions in it, like "# Not clear if these indexes are also generated randomly and we need to guess them or not.". Here's what's up with those indices:

Message message_handshake_initiation: the sender picks a random 32-bit number, and places it in the "sender_index" field.
Message message_handshake_response: the sender picks a random 32-bit number, and places it in the "sender_index" field. It places the 32-bit number that it received from message_handshake_initiation into the "receiver_index" field.
Message message_handshake_cookie: the sender places the 32-bit number that it received from message_handshake_initiation or message_handshake_response into the "receive_index" field. Message message_data: the sender places the 32-bit number that it picked for the message_handshake_initiation or message_handshake_response into the "key_idx" field.

I'm not sure it'll be too feasible to correlate these relations via fuzzing. And either way, I think we've got decent enough non-crypto coverage now on the receive path.

I noticed a small seemingly insignificant function with low coverage that's on the rtnl path that might benefit from some attention and also help find bugs in other devices: wg_netdevice_notification. This triggers on various things, but the only case it really cares about is NETDEV_REGISTER, which happens when the interface changes network namespaces. WireGuard holds a reference to its underlying creating namespace, and in order to avoid circular reference counting or UaF it needs to either relinquish or get a reference. There are other drivers with similar type of reference counting management, I would assume. This sort of thing seems up the ally of the types of bugs and races syzkaller likes to find. The way to trigger it is with ip link set dev wg0 netns blah, and then back to its original netns. That's this code in net/core/rtnetlink.c:

       if (tb[IFLA_NET_NS_PID] || tb[IFLA_NET_NS_FD] ||
tb[IFLA_TARGET_NETNSID]) {
               struct net *net = rtnl_link_get_net_capable(skb, dev_net(dev),
                                                           tb, CAP_NET_ADMIN);
               if (IS_ERR(net)) {
                       err = PTR_ERR(net);
                       goto errout;
               }

               err = dev_change_net_namespace(dev, net, ifname);
               put_net(net);
               if (err)
                       goto errout;
               status |= DO_SETLINK_MODIFIED;
       }
Quoted 5 lines

That seems to have decent coverage, but not over wireguard. Is that just a result of the syzkaller @devname not yet being expanded to wg{0,1,2}, and it'll take a few more weeks for it to learn that? @netns_id seems probably good, being restricted to 0:4; is it possible these weren't created though a priori?

Running an instance aimed at these functions:

"enable_syscalls": [
"socket$nl_generic",
"syz_genetlink_get_family_id$wireguard",
"sendmsg$WG_CMD*",
"ioctl$ifreq_SIOCGIFINDEX_wireguard*",
"socket$nl_route",
"sendmsg$nl_route",
"setsockopt",
"syz_emit_ethernet",
"syz_extract_tcp_res",
"socket$inet*",
"accept4$inet*",
"bind$inet*",
"connect$inet*",
"sendto$inet*",
"recvfrom$inet*",
"syz_genetlink_get_family_id$fou",
"sendmsg$FOU*",
"unshare",
"setns",
"socket",
"socketpair",
"getuid",
"getgid",
"bpf",
"setsockopt",
"getsockopt",
"ioctl",
"accept4",
"connect",
"bind",
"listen",
"sendmsg",
"recvmsg",
"syz_emit_ethernet",
"syz_extract_tcp_res",
"write$tun",
"syz_init_net_socket",
"syz_genetlink_get_family_id",
"syz_open_procfs$namespace",
"ioctl$NS_GET*",
"ioctl$TUN*",
"openat$tun",
"getpid",
"mmap"
]

I got some coverage in wg_netdevice_notification:
https://imgur.com/a/1sJZKtp

Or you mean the parts that are still red?

I think theoretically these parts should be reachable too because syzkaller can do unshare and obtain net ns fd's.

It's quite hard to test because it just crashes all the time on known bugs. So maybe the most profitable way to get more coverage throughout the networking subsystem now is to fix the top layer of crashers ;)

2020/02/20 16:29:08 vm-9: crash: BUG: MAX_LOCKDEP_ENTRIES too low! 2020/02/20 16:32:25 vm-6: crash: kernel BUG at net/core/skbuff.c:LINE! 2020/02/20 16:33:13 vm-16: crash: INFO: task hung in register_netdevice_notifier 2020/02/20 16:33:46 vm-5: crash: INFO: task hung in register_netdevice_notifier 2020/02/20 16:34:22 vm-2: crash: WARNING in restore_regulatory_settings 2020/02/20 16:34:27 vm-4: crash: unregister_netdevice: waiting for DEV to become free
2020/02/20 16:35:04 vm-26: crash: BUG: MAX_LOCKDEP_ENTRIES too low! 2020/02/20 16:35:17 vm-34: crash: no output from test machine 2020/02/20 16:35:20 vm-0: crash: INFO: task hung in register_netdevice_notifier 2020/02/20 16:36:42 vm-3: crash: WARNING in restore_regulatory_settings 2020/02/20 16:37:47 vm-29: crash: INFO: task hung in register_netdevice_notifier 2020/02/20 16:38:54 vm-31: crash: INFO: task hung in register_netdevice_notifier 2020/02/20 16:39:00 vm-9: crash: INFO: task hung in register_netdevice_notifier 2020/02/20 16:41:41 vm-4: crash: BUG: MAX_LOCKDEP_ENTRIES too low! 2020/02/20 16:43:00 vm-23: crash: BUG: MAX_LOCKDEP_ENTRIES too low! 2020/02/20 16:44:01 vm-2: crash: possible deadlock in htab_lru_map_delete_node 2020/02/20 16:44:06 vm-38: crash: INFO: task hung in register_netdevice_notifier 2020/02/20 16:47:13 vm-8: crash: unregister_netdevice: waiting for DEV to become free
2020/02/20 16:47:20 vm-25: crash: unregister_netdevice: waiting for DEV to become free
2020/02/20 16:47:33 vm-18: crash: BUG: using smp_processor_id() in preemptible [ADDR] code: syz-executor
2020/02/20 16:49:37 vm-37: crash: unregister_netdevice: waiting for DEV to become free
2020/02/20 16:51:51 vm-12: crash: WARNING in print_bfs_bug 2020/02/20 16:53:28 vm-4: crash: BUG: MAX_LOCKDEP_ENTRIES too low! 2020/02/20 16:55:24 vm-17: crash: INFO: task hung in register_netdevice_notifier 2020/02/20 16:56:26 vm-34: crash: BUG: MAX_LOCKDEP_KEYS too low! 2020/02/20 16:57:11 vm-3: crash: BUG: MAX_LOCKDEP_ENTRIES too low! 2020/02/20 16:57:44 vm-31: crash: BUG: MAX_LOCKDEP_ENTRIES too low! 2020/02/20 16:58:13 vm-33: crash: BUG: MAX_LOCKDEP_KEYS too low! 2020/02/20 16:58:35 vm-32: crash: BUG: using smp_processor_id() in preemptible [ADDR] code: syz-executor
2020/02/20 17:03:12 vm-9: crash: INFO: task hung in raw_release 2020/02/20 17:03:20 vm-3: crash: unregister_netdevice: waiting for DEV to become free
2020/02/20 17:03:26 vm-6: crash: BUG: using smp_processor_id() in preemptible [ADDR] code: syz-executor
2020/02/20 17:05:08 vm-31: crash: INFO: task hung in lock_sock_nested 2020/02/20 17:06:01 vm-16: crash: BUG: MAX_LOCKDEP_ENTRIES too low! 2020/02/20 17:06:24 vm-26: crash: INFO: task hung in bcm_release 2020/02/20 17:07:20 vm-27: crash: BUG: MAX_LOCKDEP_ENTRIES too low! 2020/02/20 17:08:03 vm-11: crash: BUG: MAX_LOCKDEP_ENTRIES too low! 2020/02/20 17:08:05 vm-17: crash: WARNING in hsr_addr_subst_dest 2020/02/20 17:09:12 vm-22: crash: BUG: MAX_LOCKDEP_ENTRIES too low!

author maintainer
#25 lore raw

Re: syzkaller wireguard key situation [was: Re: [PATCH net-next v2] net: WireGuard secure network tunnel]

Hi Dmitry,

Quoted 4 lines · Dmitry Vyukov

I got some coverage in wg_netdevice_notification:
https://imgur.com/a/1sJZKtp

Or you mean the parts that are still red?

Yes, it's the red parts that interest me. Intermixing those with various wireguard-specific netlink calls and setting devices up and down and putting traffic through those sockets, in weird ways, could dig up bugs.

Quoted 6 lines

I think theoretically these parts should be reachable too because syzkaller can do unshare and obtain net ns fd's.

It's quite hard to test because it just crashes all the time on known bugs. So maybe the most profitable way to get more coverage throughout the networking subsystem now is to fix the top layer of crashers ;)

Ahhh, interesting, so the issue is that syzkaller is finding too many other networking stack bugs before it gets to being able to play with wireguard. Shucks.

Jason

#26 lore raw

Re: syzkaller wireguard key situation [was: Re: [PATCH net-next v2] net: WireGuard secure network tunnel]

Quoted 17 lines · Jason A. Donenfeld

Hi Dmitry,

Quoted 4 lines · Dmitry Vyukov

I got some coverage in wg_netdevice_notification:
https://imgur.com/a/1sJZKtp

Or you mean the parts that are still red?

Yes, it's the red parts that interest me. Intermixing those with various wireguard-specific netlink calls and setting devices up and down and putting traffic through those sockets, in weird ways, could dig up bugs.

Quoted 3 lines

I think theoretically these parts should be reachable too because syzkaller can do unshare and obtain net ns fd's.

Quoted 7 lines
Quoted 3 lines

It's quite hard to test because it just crashes all the time on known bugs. So maybe the most profitable way to get more coverage throughout the networking subsystem now is to fix the top layer of crashers ;)

Ahhh, interesting, so the issue is that syzkaller is finding too many other networking stack bugs before it gets to being able to play with wireguard. Shucks.

If it's aimed only at, say, wireguard netlink interface, then it's not distracted by bugs in other parts. But as you add some ipv4/6 tcp/udp sockets, more netlink to change these net namespaces, namespaces related syscalls, packet injection, etc, in the end it covers quite a significant part of kernel. You know how fuzzing works, right. You really need to fix the current layer of bugs to get to the next one. And we accumulated 600+ open bugs. It still finds some new ones, but I guess these are really primitive ones (as compared to its full bug finding potential).

author maintainer
#27 lore raw

Re: syzkaller wireguard key situation [was: Re: [PATCH net-next v2] net: WireGuard secure network tunnel]

Quoted 9 lines · Dmitry Vyukov

If it's aimed only at, say, wireguard netlink interface, then it's not distracted by bugs in other parts. But as you add some ipv4/6 tcp/udp sockets, more netlink to change these net namespaces, namespaces related syscalls, packet injection, etc, in the end it covers quite a significant part of kernel. You know how fuzzing works, right. You really need to fix the current layer of bugs to get to the next one. And we accumulated 600+ open bugs. It still finds some new ones, but I guess these are really primitive ones (as compared to its full bug finding potential).

Yea, seems reasonable. I need to get a local syzkaller instance set up for customization and then start patching the things that seem to be standing in the way. Either way, so long as there isn't some implementation issue or logical problem getting in the way of calling that codepath, I'm satisfied in knowing that syzkaller will get there eventually.

#28 lore raw
In reply to Jason A. Donenfeld
Quoted 10 lines · Jason A. Donenfeld
Quoted 5 lines · Dmitry Vyukov
Quoted 2 lines

Is this precise enough for race
condition bugs?

It's finding lots of race conditions provoked bugs (I would say it's the most common cause of kernel bugs).

I meant -- are the reproducers it makes precise enough to retrigger network-level race conditions?

We provide a simple invariant: if it claims a reproducer, it was able to provide the exact reported crash using that exact program on a freshly booted kernel.
However, the crash may be reproducible on the first iteration, or may require running it for a few seconds/minutes. And obviously for race conditions the rate on your machine may be different (in both directions) from the rate on the syzbot machine. Reproducers don't try to fix the exact execution (generally not feasible), instead they are just threaded stress tests with some amount of natural randomness is execution. But as I notes, it was able to trigger that exact crash using that exact program.

A shorter version: it's good enough to tr-trigger lots of race conditions.

Quoted 6 lines
Quoted 3 lines

Well, you are missing that wireguard is not the only subsystem syzkaller tests (in fact, it does not test it at all) and there are 3000 other subsystems :)

Oooo! Everything is tested at the same time. I understand now; that makes a lot more sense.

Yes, it's generally whole kernel. Partitioning it into 3000 separate instances is lots of problems on multiple fronts and in the end it's not really possible to draw strict boundaries, in end whole kernel is tied via mm/fs/pipes/splice/vmslice/etc. E.g. what if you vmsplice some device-mapped memory into wireguard using io_uring and setup some bpf filter somewhere and ptrace it at the same time while sending a signal? :)

Quoted 8 lines

I'll look into splitting out the option, as you've asked. Note, though, that there are currently only three spots that have the "extra checks" at the moment, and one of them can be optimized out by the compiler with aggressive enough inlining added everywhere. The other two will result in an immediately corrupted stack frame that should be caught immediately by other things. So for now, I think you can get away with turning the debug option off, and you won't be missing much from the "extra checks", at least until we add more.

I see. Maybe something to keep in mind for future.

Quoted 3 lines

That's exciting about syzcaller having at it with WireGuard. Is there some place where I can "see" it fuzzing WireGuard, or do I just wait for the bug reports to come rolling in?

Well, unfortunately it does not test wireguard at the moment. I've enabled the config as I saw it appeared in linux-next:
https://github.com/google/syzkaller/commit/240ba66ba8a0a99f27e1aac01f376331051a65c2
but that's it for now.
There are 3000 subsystems, are you ready to describe precise interface for all of them with all the necessary setup and prerequisites? Nobody can do it for all subsystems. Developer of a particular subsystem is the best candidate for also describing what it takes to test it ;)

author maintainer
#29 lore raw
Quoted 4 lines · Dmitry Vyukov

Well, unfortunately it does not test wireguard at the moment. I've enabled the config as I saw it appeared in linux-next:
https://github.com/google/syzkaller/commit/240ba66ba8a0a99f27e1aac01f376331051a65c2
but that's it for now.

I do see that in the linux-next and net-next instances you have running, it's at least hitting interface creation and netlink.

Quoted 4 lines

There are 3000 subsystems, are you ready to describe precise interface for all of them with all the necessary setup and prerequisites? Nobody can do it for all subsystems. Developer of a particular subsystem is the best candidate for also describing what it takes to test it ;)

Sure, I'd be happy to help get things rolling on WireGuard. What do you need from me? A small shell script to set up a few interfaces that are peered with each other? Sample packets to begin mutations from? Since most of WireGuard's surface is behind constant-time/branchless crypto, I doubt you'll fuzz your way to having valid key agreement, which makes me think some sort of setup is necessary.