Skip to contentSkip to conversation

net,bpf: fix null-ptr-deref in xdp_master_redirect() for bonding and add selftest

PATCH Superseded Jiayuan Chen 2 participants 6 messages Apr 10 – Apr 11

Subsystems (7)

v6 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v6 cover net,bpf: fix null-ptr-deref in xdp_master_redirect() for bonding and add selftest

From: Jiayuan Chen <[email protected]>

This series has gone through several rounds of discussion and the maintainers hold different views on where the fix should live (in the generic xdp_master_redirect() path vs. inside bonding). I respect all of the suggestions, but I would like to get the crash fixed first, so this version takes the approach of checking whether the master device is up in xdp_master_redirect(), as suggested by Daniel Borkmann. If a different shape is preferred later it can be done as a follow-up, but the null-ptr-deref should not linger.

syzkaller reported a kernel panic, full decoded trace here:
https://syzkaller.appspot.com/bug?extid=80e046b8da2820b6ba73

Problem Description

bond_rr_gen_slave_id() dereferences bond->rr_tx_counter without a NULL check. rr_tx_counter is a per-CPU counter that bonding only allocates in bond_open() when the mode is round-robin. If the bond device was never brought up, rr_tx_counter stays NULL.

The XDP redirect path can still reach that code on a bond that was never opened: bpf_master_redirect_enabled_key is a global static key, so as soon as any bond device has native XDP attached, the XDP_TX -> xdp_master_redirect() interception is enabled for every slave system-wide. The path xdp_master_redirect() -> bond_xdp_get_xmit_slave() -> bond_xdp_xmit_roundrobin_slave_get() -> bond_rr_gen_slave_id() then runs against a bond that has no rr_tx_counter and crashes.

Solution

Patch 1: Fix this in the generic xdp_master_redirect() by skipping master interception when the master device is not running. Returning XDP_TX keeps the original XDP_TX behaviour on the receiving slave, and avoids calling into any master ->ndo_xdp_get_xmit_slave() on a device that has not fully initialized its XDP state. This is not specific to
bonding: any current or future master that defers XDP state allocation to ->ndo_open() is protected.
Patch 2: Add a selftest that reproduces the above scenario.

Changes since v5:
https://lore.kernel.org/netdev/[email protected]/
- Moved the fix back into xdp_master_redirect() and use netif_running() on the master device to decide whether to intercept (Suggested by Daniel Borkmann, seconded by Paolo Abeni and Eric Dumazet)

Changes since v4:
https://lore.kernel.org/netdev/[email protected]/
- Reverted unconditional alloc in bond_init(); instead add a NULL check with unlikely()/READ_ONCE() in bond_rr_gen_slave_id() and WRITE_ONCE() in bond_open(), avoiding memory waste for non-RR modes (Suggested by Nikolay Aleksandrov, patch by Jay Vosburgh)

Changes since v3:
https://lore.kernel.org/netdev/[email protected]/T/#t
- Added code comment and commit log explaining why rr_tx_counter is allocated unconditionally for all modes (Suggested by Jay Vosburgh)

Changes since v2:
https://lore.kernel.org/netdev/[email protected]/T/#t
- Moved allocation from bond_create_init() helper into bond_init() (ndo_init), which is the natural single point covering both creation paths and also handles post-creation mode changes to round-robin

Changes since v1:
https://lore.kernel.org/netdev/[email protected]/T/#t
- Moved the guard for NULL rr_tx_counter from xdp_master_redirect() into the bonding subsystem itself
(Suggested by Sebastian Andrzej Siewior <[email protected]>)

[1] https://syzkaller.appspot.com/bug?extid=80e046b8da2820b6ba73

Jiayuan Chen (2):
net, bpf: fix null-ptr-deref in xdp_master_redirect() for down master selftests/bpf: add test for xdp_master_redirect with bond not up

 net/core/filter.c                             |   2 +
 .../selftests/bpf/prog_tests/xdp_bonding.c    | 101 +++++++++++++++++-
 2 files changed, 101 insertions(+), 2 deletions(-)
Signature · 1 line
2.43.0
#1 lore raw

PATCH v6 1/2 net, bpf: fix null-ptr-deref in xdp_master_redirect() for down master

syzkaller reported a kernel panic in bond_rr_gen_slave_id() reached via xdp_master_redirect(). Full decoded trace:

https://syzkaller.appspot.com/bug?extid=80e046b8da2820b6ba73

bond_rr_gen_slave_id() dereferences bond->rr_tx_counter, a per-CPU counter that bonding only allocates in bond_open() when the mode is round-robin. If the bond device was never brought up, rr_tx_counter stays NULL.

The XDP redirect path can still reach that code on a bond that was never opened: bpf_master_redirect_enabled_key is a global static key, so as soon as any bond device has native XDP attached, the XDP_TX -> xdp_master_redirect() interception is enabled for every slave system-wide. The path xdp_master_redirect() -> bond_xdp_get_xmit_slave() -> bond_xdp_xmit_roundrobin_slave_get() -> bond_rr_gen_slave_id() then runs against a bond that has no rr_tx_counter and crashes.

Fix this in the generic xdp_master_redirect() by refusing to call into the master's ->ndo_xdp_get_xmit_slave() when the master device is not up. IFF_UP is only set after ->ndo_open() has successfully returned, so this reliably excludes masters whose XDP state has not been fully initialized. Drop the frame with XDP_ABORTED so the exception is visible via trace_xdp_exception() rather than silently falling through. This is not specific to bonding: any current or future master that defers XDP state allocation to ->ndo_open() is protected.

Fixes 879af96ffd72 ("net, core: Add support for XDP redirection to slave device")
Reported-by [email protected]
---
 net/core/filter.c | 2 ++
 1 file changed, 2 insertions(+)
net/core/filter.c +2 -0
index cf2113af4bc9..9ec70c4b7723 100644
@@ -4398,6 +4398,8 @@ u32 xdp_master_redirect(struct xdp_buff *xdp)
 	struct net_device *master, *slave;
 
 	master = netdev_master_upper_dev_get_rcu(xdp->rxq->dev);
+	if (unlikely(!(master->flags & IFF_UP)))
+		return XDP_ABORTED;
 	slave = master->netdev_ops->ndo_xdp_get_xmit_slave(master, xdp);
 	if (slave && slave != xdp->rxq->dev) {
 		/* The target device is different from the receiving device, so
Signature · 1 line
2.43.0
maintainer Acked-by on v6 1/2
#2 lore raw

Re: [PATCH net v6 1/2] net, bpf: fix null-ptr-deref in xdp_master_redirect() for down master

Quoted 28 lines · Jiayuan Chen

syzkaller reported a kernel panic in bond_rr_gen_slave_id() reached via xdp_master_redirect(). Full decoded trace:

   https://syzkaller.appspot.com/bug?extid=80e046b8da2820b6ba73

bond_rr_gen_slave_id() dereferences bond->rr_tx_counter, a per-CPU counter that bonding only allocates in bond_open() when the mode is round-robin. If the bond device was never brought up, rr_tx_counter stays NULL.

The XDP redirect path can still reach that code on a bond that was never opened: bpf_master_redirect_enabled_key is a global static key, so as soon as any bond device has native XDP attached, the XDP_TX -> xdp_master_redirect() interception is enabled for every slave system-wide. The path xdp_master_redirect() -> bond_xdp_get_xmit_slave() -> bond_xdp_xmit_roundrobin_slave_get() -> bond_rr_gen_slave_id() then runs against a bond that has no rr_tx_counter and crashes.

Fix this in the generic xdp_master_redirect() by refusing to call into the master's ->ndo_xdp_get_xmit_slave() when the master device is not up. IFF_UP is only set after ->ndo_open() has successfully returned, so this reliably excludes masters whose XDP state has not been fully initialized. Drop the frame with XDP_ABORTED so the exception is visible via trace_xdp_exception() rather than silently falling through. This is not specific to bonding: any current or future master that defers XDP state allocation to ->ndo_open() is protected.

Quoted 5 lines
Fixes 879af96ffd72 ("net, core: Add support for XDP redirection to slave device")
Reported-by [email protected]
#3 lore raw

PATCH v6 2/2 selftests/bpf: add test for xdp_master_redirect with bond not up

In reply to Jiayuan Chen

Add a selftest that reproduces the null-ptr-deref in bond_rr_gen_slave_id() when XDP redirect targets a bond device in round-robin mode that was never brought up. The test verifies the fix by ensuring no crash occurs.

Test setup:
- bond0: active-backup mode, UP, with native XDP (enables bpf_master_redirect_enabled_key globally)
- bond1: round-robin mode, never UP
- veth1: slave of bond1, with generic XDP (XDP_TX)
- BPF_PROG_TEST_RUN with live frames triggers the redirect path

---
 .../selftests/bpf/prog_tests/xdp_bonding.c    | 101 +++++++++++++++++-
 1 file changed, 99 insertions(+), 2 deletions(-)
tools/testing/selftests/bpf/prog_tests/xdp_bonding.c +99 -2
index e8ea26464349..0d4ec1e5b401 100644
@@ -191,13 +191,18 @@ static int bonding_setup(struct skeletons *skeletons, int mode, int xmit_policy,
 	return -1;
 }
 
-static void bonding_cleanup(struct skeletons *skeletons)
+static void link_cleanup(struct skeletons *skeletons)
 {
-	restore_root_netns();
 	while (skeletons->nlinks) {
 		skeletons->nlinks--;
 		bpf_link__destroy(skeletons->links[skeletons->nlinks]);
 	}
+}
+
+static void bonding_cleanup(struct skeletons *skeletons)
+{
+	restore_root_netns();
+	link_cleanup(skeletons);
 	ASSERT_OK(system("ip link delete bond1"), "delete bond1");
 	ASSERT_OK(system("ip link delete veth1_1"), "delete veth1_1");
 	ASSERT_OK(system("ip link delete veth1_2"), "delete veth1_2");
@@ -493,6 +498,95 @@ static void test_xdp_bonding_nested(struct skeletons *skeletons)
 	system("ip link del bond_nest2");
 }
 
+/*
+ * Test that XDP redirect via xdp_master_redirect() does not crash when
+ * the bond master device is not up. When bond is in round-robin mode but
+ * never opened, rr_tx_counter is NULL.
+ */
+static void test_xdp_bonding_redirect_no_up(struct skeletons *skeletons)
+{
+	struct nstoken *nstoken = NULL;
+	int xdp_pass_fd, xdp_tx_fd;
+	int veth1_ifindex;
+	int err;
+	char pkt[ETH_HLEN + 1];
+	struct xdp_md ctx_in = {};
+
+	DECLARE_LIBBPF_OPTS(bpf_test_run_opts, opts,
+			    .data_in = &pkt,
+			    .data_size_in = sizeof(pkt),
+			    .ctx_in = &ctx_in,
+			    .ctx_size_in = sizeof(ctx_in),
+			    .flags = BPF_F_TEST_XDP_LIVE_FRAMES,
+			    .repeat = 1,
+			    .batch_size = 1,
+		);
+
+	/* We can't use bonding_setup() because bond will be active */
+	SYS(out, "ip netns add ns_rr_no_up");
+	nstoken = open_netns("ns_rr_no_up");
+	if (!ASSERT_OK_PTR(nstoken, "open ns_rr_no_up"))
+		goto out;
+
+	/* bond0: active-backup, UP with slave veth0.
+	 * Attaching native XDP to bond0 enables bpf_master_redirect_enabled_key
+	 * globally.
+	 */
+	SYS(out, "ip link add bond0 type bond mode active-backup");
+	SYS(out, "ip link add veth0 type veth peer name veth0p");
+	SYS(out, "ip link set veth0 master bond0");
+	SYS(out, "ip link set bond0 up");
+	SYS(out, "ip link set veth0p up");
+
+	/* bond1: round-robin, never UP -> rr_tx_counter stays NULL */
+	SYS(out, "ip link add bond1 type bond mode balance-rr");
+	SYS(out, "ip link add veth1 type veth peer name veth1p");
+	SYS(out, "ip link set veth1 master bond1");
+
+	veth1_ifindex = if_nametoindex("veth1");
+	if (!ASSERT_GT(veth1_ifindex, 0, "veth1_ifindex"))
+		goto out;
+
+	/* Attach native XDP to bond0 -> enables global redirect key */
+	if (xdp_attach(skeletons, skeletons->xdp_tx->progs.xdp_tx, "bond0"))
+		goto out;
+
+	/* Attach generic XDP (XDP_TX) to veth1.
+	 * When packets arrive at veth1 via netif_receive_skb, do_xdp_generic()
+	 * runs this program. XDP_TX + bond slave triggers xdp_master_redirect().
+	 */
+	xdp_tx_fd = bpf_program__fd(skeletons->xdp_tx->progs.xdp_tx);
+	if (!ASSERT_GE(xdp_tx_fd, 0, "xdp_tx prog_fd"))
+		goto out;
+
+	err = bpf_xdp_attach(veth1_ifindex, xdp_tx_fd,
+			     XDP_FLAGS_SKB_MODE, NULL);
+	if (!ASSERT_OK(err, "attach generic XDP to veth1"))
+		goto out;
+
+	/* Run BPF_PROG_TEST_RUN with XDP_PASS live frames on veth1.
+	 * XDP_PASS frames become SKBs with skb->dev = veth1, entering
+	 * netif_receive_skb -> do_xdp_generic -> xdp_master_redirect.
+	 * Without the fix, bond_rr_gen_slave_id() dereferences NULL
+	 * rr_tx_counter and crashes.
+	 */
+	xdp_pass_fd = bpf_program__fd(skeletons->xdp_dummy->progs.xdp_dummy_prog);
+	if (!ASSERT_GE(xdp_pass_fd, 0, "xdp_pass prog_fd"))
+		goto out;
+
+	memset(pkt, 0, sizeof(pkt));
+	ctx_in.data_end = sizeof(pkt);
+	ctx_in.ingress_ifindex = veth1_ifindex;
+
+	err = bpf_prog_test_run_opts(xdp_pass_fd, &opts);
+	ASSERT_OK(err, "xdp_pass test_run should not crash");
+
+out:
+	link_cleanup(skeletons);
+	close_netns(nstoken);
+	SYS_NOFAIL("ip netns del ns_rr_no_up");
+}
+
 static void test_xdp_bonding_features(struct skeletons *skeletons)
 {
 	LIBBPF_OPTS(bpf_xdp_query_opts, query_opts);
@@ -738,6 +832,9 @@ void serial_test_xdp_bonding(void)
 	if (test__start_subtest("xdp_bonding_redirect_multi"))
 		test_xdp_bonding_redirect_multi(&skeletons);
 
+	if (test__start_subtest("xdp_bonding_redirect_no_up"))
+		test_xdp_bonding_redirect_no_up(&skeletons);
+
 out:
 	xdp_dummy__destroy(skeletons.xdp_dummy);
 	xdp_tx__destroy(skeletons.xdp_tx);
Signature · 1 line
2.43.0
maintainer
#4 lore raw

Re: [PATCH net v6 2/2] selftests/bpf: add test for xdp_master_redirect with bond not up

Quoted 5 lines · Jiayuan Chen

Add a selftest that reproduces the null-ptr-deref in bond_rr_gen_slave_id() when XDP redirect targets a bond device in round-robin mode that was never brought up. The test verifies the fix by ensuring no crash occurs.

Quoted 8 lines

Test setup:
- bond0: active-backup mode, UP, with native XDP (enables bpf_master_redirect_enabled_key globally)
- bond1: round-robin mode, never UP
- veth1: slave of bond1, with generic XDP (XDP_TX)
- BPF_PROG_TEST_RUN with live frames triggers the redirect path

I checked locally that this XDP test passes fine and triggers a NULL pointer deref without the fix.

[...]

Review

Quoted 7 lines
(quoted diff) +7 -0
+	/* Attach generic XDP (XDP_TX) to veth1.
+	 * When packets arrive at veth1 via netif_receive_skb, do_xdp_generic()
+	 * runs this program. XDP_TX + bond slave triggers xdp_master_redirect().
+	 */
+	xdp_tx_fd = bpf_program__fd(skeletons->xdp_tx->progs.xdp_tx);
+	if (!ASSERT_GE(xdp_tx_fd, 0, "xdp_tx prog_fd"))
+		goto out;

nit: no need for the ASSERT_GE given the skeleton loaded, see also the various other tests gathering bpf_program__fd().

Quoted 4 lines
(quoted diff) +4 -0
+	err = bpf_xdp_attach(veth1_ifindex, xdp_tx_fd,
+			     XDP_FLAGS_SKB_MODE, NULL);
+	if (!ASSERT_OK(err, "attach generic XDP to veth1"))
+		goto out;

Review

Quoted 10 lines
(quoted diff) +10 -0
+
+	/* Run BPF_PROG_TEST_RUN with XDP_PASS live frames on veth1.
+	 * XDP_PASS frames become SKBs with skb->dev = veth1, entering
+	 * netif_receive_skb -> do_xdp_generic -> xdp_master_redirect.
+	 * Without the fix, bond_rr_gen_slave_id() dereferences NULL
+	 * rr_tx_counter and crashes.
+	 */
+	xdp_pass_fd = bpf_program__fd(skeletons->xdp_dummy->progs.xdp_dummy_prog);
+	if (!ASSERT_GE(xdp_pass_fd, 0, "xdp_pass prog_fd"))
+		goto out;

ditto, can be simplified a bit into:

tools/testing/selftests/bpf/prog_tests/xdp_bonding.c +3 -8
index 0d4ec1e5b401..c42488e445c2 100644
@@ -506,7 +506,7 @@ static void test_xdp_bonding_nested(struct skeletons *skeletons)
 static void test_xdp_bonding_redirect_no_up(struct skeletons *skeletons)
 {
 	struct nstoken *nstoken = NULL;
-	int xdp_pass_fd, xdp_tx_fd;
+	int xdp_pass_fd;
 	int veth1_ifindex;
 	int err;
 	char pkt[ETH_HLEN + 1];
@@ -555,11 +555,8 @@ static void test_xdp_bonding_redirect_no_up(struct skeletons *skeletons)
 	 * When packets arrive at veth1 via netif_receive_skb, do_xdp_generic()
 	 * runs this program. XDP_TX + bond slave triggers xdp_master_redirect().
 	 */
-	xdp_tx_fd = bpf_program__fd(skeletons->xdp_tx->progs.xdp_tx);
-	if (!ASSERT_GE(xdp_tx_fd, 0, "xdp_tx prog_fd"))
-		goto out;
-
-	err = bpf_xdp_attach(veth1_ifindex, xdp_tx_fd,
+	err = bpf_xdp_attach(veth1_ifindex,
+			     bpf_program__fd(skeletons->xdp_tx->progs.xdp_tx),
 			     XDP_FLAGS_SKB_MODE, NULL);
 	if (!ASSERT_OK(err, "attach generic XDP to veth1"))
 		goto out;
@@ -571,8 +568,6 @@ static void test_xdp_bonding_redirect_no_up(struct skeletons *skeletons)
 	 * rr_tx_counter and crashes.
 	 */
 	xdp_pass_fd = bpf_program__fd(skeletons->xdp_dummy->progs.xdp_dummy_prog);
-	if (!ASSERT_GE(xdp_pass_fd, 0, "xdp_pass prog_fd"))
-		goto out;
 
 	memset(pkt, 0, sizeof(pkt));
 	ctx_in.data_end = sizeof(pkt);
#5 lore raw

Re: [PATCH net v6 2/2] selftests/bpf: add test for xdp_master_redirect with bond not up

Quoted 88 lines · Daniel Borkmann
Quoted 5 lines · Jiayuan Chen

Add a selftest that reproduces the null-ptr-deref in bond_rr_gen_slave_id() when XDP redirect targets a bond device in round-robin mode that was never brought up. The test verifies the fix by ensuring no crash occurs.

Quoted 8 lines

Test setup:
- bond0: active-backup mode, UP, with native XDP (enables bpf_master_redirect_enabled_key globally)
- bond1: round-robin mode, never UP
- veth1: slave of bond1, with generic XDP (XDP_TX)
- BPF_PROG_TEST_RUN with live frames triggers the redirect path

I checked locally that this XDP test passes fine and triggers a NULL pointer deref without the fix.

[...]

Quoted 9 lines

+    /* Attach generic XDP (XDP_TX) to veth1.
+     * When packets arrive at veth1 via netif_receive_skb, do_xdp_generic()
+     * runs this program. XDP_TX + bond slave triggers xdp_master_redirect().
+     */
+    xdp_tx_fd = bpf_program__fd(skeletons->xdp_tx->progs.xdp_tx); +    if (!ASSERT_GE(xdp_tx_fd, 0, "xdp_tx prog_fd"))
+        goto out;

nit: no need for the ASSERT_GE given the skeleton loaded, see also the various other tests gathering bpf_program__fd().

Quoted 7 lines

+    err = bpf_xdp_attach(veth1_ifindex, xdp_tx_fd,
+                 XDP_FLAGS_SKB_MODE, NULL);
+    if (!ASSERT_OK(err, "attach generic XDP to veth1")) +        goto out;
+
+    /* Run BPF_PROG_TEST_RUN with XDP_PASS live frames on veth1. +     * XDP_PASS frames become SKBs with skb->dev = veth1, entering

Quoted 8 lines

+     * netif_receive_skb -> do_xdp_generic -> xdp_master_redirect. +     * Without the fix, bond_rr_gen_slave_id() dereferences NULL +     * rr_tx_counter and crashes.
+     */
+    xdp_pass_fd = bpf_program__fd(skeletons->xdp_dummy->progs.xdp_dummy_prog); +    if (!ASSERT_GE(xdp_pass_fd, 0, "xdp_pass prog_fd")) +        goto out;

ditto, can be simplified a bit into:

tools/testing/selftests/bpf/prog_tests/xdp_bonding.c +1 -1
index 0d4ec1e5b401..c42488e445c2 100644
@@ -506,7 +506,7 @@ static void test_xdp_bonding_nested(struct 
 skeletons *skeletons)
  static void test_xdp_bonding_redirect_no_up(struct skeletons *skeletons)
  {
      struct nstoken *nstoken = NULL;
-    int xdp_pass_fd, xdp_tx_fd;
+    int xdp_pass_fd;
      int veth1_ifindex;
      int err;

     char pkt[ETH_HLEN + 1];

tools/testing/selftests/bpf/prog_tests/xdp_bonding.c +2 -5
@@ -555,11 +555,8 @@ static void 
 test_xdp_bonding_redirect_no_up(struct skeletons *skeletons)
       * When packets arrive at veth1 via netif_receive_skb, do_xdp_generic()
       * runs this program. XDP_TX + bond slave triggers xdp_master_redirect().
       */
-    xdp_tx_fd = bpf_program__fd(skeletons->xdp_tx->progs.xdp_tx);
-    if (!ASSERT_GE(xdp_tx_fd, 0, "xdp_tx prog_fd"))
-        goto out;
-
-    err = bpf_xdp_attach(veth1_ifindex, xdp_tx_fd,
+    err = bpf_xdp_attach(veth1_ifindex,
+ bpf_program__fd(skeletons->xdp_tx->progs.xdp_tx),
                   XDP_FLAGS_SKB_MODE, NULL);
      if (!ASSERT_OK(err, "attach generic XDP to veth1"))

         goto out;

tools/testing/selftests/bpf/prog_tests/xdp_bonding.c +0 -2
@@ -571,8 +568,6 @@ static void test_xdp_bonding_redirect_no_up(struct 
 skeletons *skeletons)
       * rr_tx_counter and crashes.
       */
      xdp_pass_fd = bpf_program__fd(skeletons->xdp_dummy->progs.xdp_dummy_prog);
-    if (!ASSERT_GE(xdp_pass_fd, 0, "xdp_pass prog_fd"))
-        goto out;
 
      memset(pkt, 0, sizeof(pkt));
Quoted 2 lines

     ctx_in.data_end = sizeof(pkt);

Hi daniel

Thanks for the review and for verifying the crash reproduces without the fix.

I've applied your suggestion.