Skip to contentSkip to conversation

Fix missing synack in BPF cgroup_skb filters

PATCH Applied Kui-Feng Lee 3 participants 5 messages Jun 24, 2023 – Jun 30, 2023

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

v4 cover Fix missing synack in BPF cgroup_skb filters

TCP SYN/ACK packets of connections from processes/sockets outside a cgroup on the same host are not received by the cgroup's installed cgroup_skb filters.

There were two BPF cgroup_skb programs attached to a cgroup named "my_cgroup".

    SEC("cgroup_skb/ingress")
    int ingress(struct __sk_buff *skb)
    {
        /* .... process skb ... */
        return 1;
    }

    SEC("cgroup_skb/egress")
    int egress(struct __sk_buff *skb)
    {
        /* .... process skb ... */
        return 1;
    
    }

We discovered that when running the command "nc -6 -l 8000" in "my_group" and connecting to it from outside of "my_cgroup" with the command "nc -6 localhost 8000", the egress filter did not detect the SYN/ACK packet. However, we did observe the SYN/ACK packet at the ingress when connecting from a socket in "my_cgroup" to a socket outside of it.

We came across BPF_CGROUP_RUN_PROG_INET_EGRESS(). This macro is responsible for calling BPF programs that are attached to the egress hook of a cgroup and it skips programs if the sending socket is not the owner of the skb. Specifically, in our situation, the SYN/ACK skb is owned by a struct request_sock instance, but the sending socket is the listener socket we use to receive incoming connections. The request_sock is created to manage an incoming connection.

It has been determined that checking the owner of a skb against the sending socket is not required. Removing this check will allow the filters to receive SYN/ACK packets.

To ensure that cgroup_skb filters can receive all signaling packets, including SYN, SYN/ACK, ACK, FIN, and FIN/ACK. A new self-test has been added as well.

Changes from v3:

- Check SKB ownership against full socket instead of just remove the check.

- Address the issue raised by Yonghong.

- Put more details down in the commit message.

Changes from v2:

- Remove redundant blank lines.

Changes from v1:

- Check the number of observed packets instead of just sleeping.

- Use ASSERT_XXX() instead of CHECK()/

[v1] https://lore.kernel.org/all/[email protected]/
[v2] https://lore.kernel.org/all/[email protected]/
[v3] https://lore.kernel.org/all/[email protected]/

Kui-Feng Lee (2):
net: bpf: Check SKB ownership against full socket. selftests/bpf: Verify that the cgroup_skb filters receive expected
packets.

 include/linux/bpf-cgroup.h                    |   4 +-
 tools/testing/selftests/bpf/cgroup_helpers.c  |  12 +
 tools/testing/selftests/bpf/cgroup_helpers.h  |   1 +
 tools/testing/selftests/bpf/cgroup_tcp_skb.h  |  35 ++
 .../selftests/bpf/prog_tests/cgroup_tcp_skb.c | 402 ++++++++++++++++++
 .../selftests/bpf/progs/cgroup_tcp_skb.c      | 382 +++++++++++++++++
 6 files changed, 834 insertions(+), 2 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/cgroup_tcp_skb.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/cgroup_tcp_skb.c
 create mode 100644 tools/testing/selftests/bpf/progs/cgroup_tcp_skb.c
Signature · 1 line
2.34.1
#1 lore raw

PATCH v4 1/2 net: bpf: Check SKB ownership against full socket.

Check SKB ownership of an SKB against full sockets instead of request_sock.

The filters were called only if an SKB is owned by the sock that the SKB is sent out through. In another words, skb->sk should point to the sock that it is sending through its egress. However, the filters would miss SYN/ACK SKBs that they are owned by a request_sock but sent through the listener sock, that is the socket listening incoming connections.

However, the listener socket is also the full socket of the request socket. We should use the full socket as the owner socket of an SKB instead.

What is the ownership check for?
====================================

BPF_CGROUP_RUN_PROG_INET_EGRESS() checked sk == skb->sk to ensure the ownership of an SKB. Alexei referred to a mailing list conversation that took place a few years ago. [1] In that conversation, Daniel Borkmann stated that:

Wouldn't that mean however, when you go through stacked devices that you'd run the same eBPF cgroup program for skb->sk multiple times?

According to what Daniel said, the ownership check mentioned earlier presumably prevents multiple calls of egress filters caused by an skb.

A test that reproduce this scenario shows that the BPF cgroup egress programs can be called multiple times for one SKB if this ownership check is not there. So, we can not just remove this check.

Test Stacked Devices
=======================

We use L2TP to build an environment of stacked devices. L2TP (Layer 2 Tunneling Protocol) is a tunneling protocol used to support virtual private networks (VPNs). It relays encapsulated packets; for example in UDP, to its peer by using a socket.

Using L2TP, packets are first sent through the IP stack and should then arrive at an L2TP device. The device will expand its SKB header to encapsulate the packet. The SKB will be sent back to the IP stack using the socket that was made for the L2TP session. After that, the routing process will occur once more, but this time for a new destination.

We changed tools/testing/selftests/net/l2tp.sh to set up a test environment using L2TP. The run_ping() function in l2tp.sh is where the main change occurred.

    run_ping()
    {
        local desc="$1"

        sleep 10
        run_cmd host-1 ${ping6} -s 227 -c 4 -i 10 -I fc00:101::1
        fc00:101::2
        log_test $? 0 "IPv6 route through L2TP tunnel ${desc}"
        sleep 10

    }

The test will use L2TP devices to send PING messages. These messages will have a message size of 227 bytes as a special label to distinguish them. This is not an ideal solution, but works.

During the execution of the test script, bpftrace was attached to ip6_finish_output() and l2tp_xmit_skb(). BPF

    bpftrace -e '
      kfunc:ip6_finish_output {
        time("%H:%M:%S: ");
        printf("ip6_finish_output skb=%p skb->len=%d cgroup=%p sk=%p
                skb->sk=%p\n", args->skb, args->skb->len,
               args->sk->sk_cgrp_data.cgroup, args->sk, args->skb->sk); }
      kfunc:l2tp_xmit_skb {
        time("%H:%M:%S: ");
        printf("l2tp_xmit_skb skb=%p sk=%p\n", args->skb,
	       args->session->tunnel->sock); }'

The following is part of the output messages printed by bpftrace.

    16:35:20: ip6_finish_output skb=0xffff888103d8e600 skb->len=275
              cgroup=0xffff88810741f800 sk=0xffff888105f3b900
              skb->sk=0xffff888105f3b900

    16:35:20: l2tp_xmit_skb skb=0xffff888103d8e600 sk=0xffff888103dd6300

    16:35:20: ip6_finish_output skb=0xffff888103d8e600 skb->len=337
              cgroup=0xffff88810741f800 sk=0xffff888103dd6300
              skb->sk=0xffff888105f3b900

    16:35:20: ip6_finish_output skb=0xffff888103d8e600 skb->len=337
              cgroup=(nil) sk=(nil) skb->sk=(nil)

    16:35:20: ip6_finish_output skb=0xffff888103d8e000 skb->len=275
              cgroup=0xffffffff837741d0 sk=0xffff888101fe0000
              skb->sk=0xffff888101fe0000

    16:35:20: l2tp_xmit_skb skb=0xffff888103d8e000 sk=0xffff888103483180

    16:35:20: ip6_finish_output skb=0xffff888103d8e000 skb->len=337
              cgroup=0xffff88810741f800 sk=0xffff888103483180
              skb->sk=0xffff888101fe0000

    16:35:20: ip6_finish_output skb=0xffff888103d8e000 skb->len=337
              cgroup=(nil) sk=(nil) skb->sk=(nil)

The first four entries describe a PING message that was sent using the ping command, whereas the following four entries describe the response received. Multiple sockets are used to send one skb, including the socket used by the L2TP session. This can be observed.

Based on this information, it seems that the ownership check is designed to avoid multiple calls of egress filters caused by a single skb.

[1] https://lore.kernel.org/all/[email protected]/,

---
 include/linux/bpf-cgroup.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
include/linux/bpf-cgroup.h +2 -2
index 57e9e109257e..8506690dbb9c 100644
@@ -199,9 +199,9 @@ static inline bool cgroup_bpf_sock_enabled(struct sock *sk,
 #define BPF_CGROUP_RUN_PROG_INET_EGRESS(sk, skb)			       \
 ({									       \
 	int __ret = 0;							       \
-	if (cgroup_bpf_enabled(CGROUP_INET_EGRESS) && sk && sk == skb->sk) { \
+	if (cgroup_bpf_enabled(CGROUP_INET_EGRESS) && sk) {		       \
 		typeof(sk) __sk = sk_to_full_sk(sk);			       \
-		if (sk_fullsock(__sk) &&				       \
+		if (sk_fullsock(__sk) && __sk == skb_to_full_sk(skb) &&	       \
 		    cgroup_bpf_sock_enabled(__sk, CGROUP_INET_EGRESS))	       \
 			__ret = __cgroup_bpf_run_filter_skb(__sk, skb,	       \
 						      CGROUP_INET_EGRESS); \
Signature · 1 line
2.34.1
maintainer
#2 lore raw

Re: [PATCH bpf-next v4 1/2] net: bpf: Check SKB ownership against full socket.

Quoted 105 lines · Kui-Feng Lee

Check SKB ownership of an SKB against full sockets instead of request_sock.

The filters were called only if an SKB is owned by the sock that the SKB is sent out through. In another words, skb->sk should point to the sock that it is sending through its egress. However, the filters would miss SYN/ACK SKBs that they are owned by a request_sock but sent through the listener sock, that is the socket listening incoming connections.

However, the listener socket is also the full socket of the request socket. We should use the full socket as the owner socket of an SKB instead.

What is the ownership check for?
====================================

BPF_CGROUP_RUN_PROG_INET_EGRESS() checked sk == skb->sk to ensure the ownership of an SKB. Alexei referred to a mailing list conversation that took place a few years ago. [1] In that conversation, Daniel Borkmann stated that:

Wouldn't that mean however, when you go through stacked devices that you'd run the same eBPF cgroup program for skb->sk multiple times?

According to what Daniel said, the ownership check mentioned earlier presumably prevents multiple calls of egress filters caused by an skb.

A test that reproduce this scenario shows that the BPF cgroup egress programs can be called multiple times for one SKB if this ownership check is not there. So, we can not just remove this check.

Test Stacked Devices
=======================

We use L2TP to build an environment of stacked devices. L2TP (Layer 2 Tunneling Protocol) is a tunneling protocol used to support virtual private networks (VPNs). It relays encapsulated packets; for example in UDP, to its peer by using a socket.

Using L2TP, packets are first sent through the IP stack and should then arrive at an L2TP device. The device will expand its SKB header to encapsulate the packet. The SKB will be sent back to the IP stack using the socket that was made for the L2TP session. After that, the routing process will occur once more, but this time for a new destination.

We changed tools/testing/selftests/net/l2tp.sh to set up a test environment using L2TP. The run_ping() function in l2tp.sh is where the main change occurred.

     run_ping()
     {
         local desc="$1"

         sleep 10
         run_cmd host-1 ${ping6} -s 227 -c 4 -i 10 -I fc00:101::1
         fc00:101::2
         log_test $? 0 "IPv6 route through L2TP tunnel ${desc}"
         sleep 10

     }

The test will use L2TP devices to send PING messages. These messages will have a message size of 227 bytes as a special label to distinguish them. This is not an ideal solution, but works.

During the execution of the test script, bpftrace was attached to ip6_finish_output() and l2tp_xmit_skb(). BPF

     bpftrace -e '
       kfunc:ip6_finish_output {
         time("%H:%M:%S: ");
         printf("ip6_finish_output skb=%p skb->len=%d cgroup=%p sk=%p
                 skb->sk=%p\n", args->skb, args->skb->len,
                args->sk->sk_cgrp_data.cgroup, args->sk, args->skb->sk); }
       kfunc:l2tp_xmit_skb {
         time("%H:%M:%S: ");
         printf("l2tp_xmit_skb skb=%p sk=%p\n", args->skb,
	       args->session->tunnel->sock); }'

The following is part of the output messages printed by bpftrace.

     16:35:20: ip6_finish_output skb=0xffff888103d8e600 skb->len=275
               cgroup=0xffff88810741f800 sk=0xffff888105f3b900
               skb->sk=0xffff888105f3b900

     16:35:20: l2tp_xmit_skb skb=0xffff888103d8e600 sk=0xffff888103dd6300

     16:35:20: ip6_finish_output skb=0xffff888103d8e600 skb->len=337
               cgroup=0xffff88810741f800 sk=0xffff888103dd6300
               skb->sk=0xffff888105f3b900

     16:35:20: ip6_finish_output skb=0xffff888103d8e600 skb->len=337
               cgroup=(nil) sk=(nil) skb->sk=(nil)

     16:35:20: ip6_finish_output skb=0xffff888103d8e000 skb->len=275
               cgroup=0xffffffff837741d0 sk=0xffff888101fe0000
               skb->sk=0xffff888101fe0000

     16:35:20: l2tp_xmit_skb skb=0xffff888103d8e000 sk=0xffff888103483180

     16:35:20: ip6_finish_output skb=0xffff888103d8e000 skb->len=337
               cgroup=0xffff88810741f800 sk=0xffff888103483180
               skb->sk=0xffff888101fe0000

     16:35:20: ip6_finish_output skb=0xffff888103d8e000 skb->len=337
               cgroup=(nil) sk=(nil) skb->sk=(nil)
Quoted 7 lines

The first four entries describe a PING message that was sent using the ping command, whereas the following four entries describe the response received. Multiple sockets are used to send one skb, including the socket used by the L2TP session. This can be observed.

Based on this information, it seems that the ownership check is designed to avoid multiple calls of egress filters caused by a single skb.

Thanks for this investigation and adding these details to the commit log, this is very useful to keep for the archive. I did some minor formatting and pushed it out to bpf-next.

#3 lore raw

PATCH v4 2/2 selftests/bpf: Verify that the cgroup_skb filters receive expected packets.

In reply to Kui-Feng Lee

This test case includes four scenarios:
1. Connect to the server from outside the cgroup and close the connection from outside the cgroup.
2. Connect to the server from outside the cgroup and close the connection from inside the cgroup.
3. Connect to the server from inside the cgroup and close the connection from outside the cgroup.
4. Connect to the server from inside the cgroup and close the connection from inside the cgroup.

The test case is to verify that cgroup_skb/{egress, ingress} filters receive expected packets including SYN, SYN/ACK, ACK, FIN, and FIN/ACK.

---
 tools/testing/selftests/bpf/cgroup_helpers.c  |  12 +
 tools/testing/selftests/bpf/cgroup_helpers.h  |   1 +
 tools/testing/selftests/bpf/cgroup_tcp_skb.h  |  35 ++
 .../selftests/bpf/prog_tests/cgroup_tcp_skb.c | 402 ++++++++++++++++++
 .../selftests/bpf/progs/cgroup_tcp_skb.c      | 382 +++++++++++++++++
 5 files changed, 832 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/cgroup_tcp_skb.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/cgroup_tcp_skb.c
 create mode 100644 tools/testing/selftests/bpf/progs/cgroup_tcp_skb.c
tools/testing/selftests/bpf/cgroup_helpers.c +12 -0
index 9e95b37a7dff..2caee8423ee0 100644
@@ -277,6 +277,18 @@ int join_cgroup(const char *relative_path)
 	return join_cgroup_from_top(cgroup_path);
 }
 
+/**
+ * join_root_cgroup() - Join the root cgroup
+ *
+ * This function joins the root cgroup.
+ *
+ * On success, it returns 0, otherwise on failure it returns 1.
+ */
+int join_root_cgroup(void)
+{
+	return join_cgroup_from_top(CGROUP_MOUNT_PATH);
+}
+
 /**
  * join_parent_cgroup() - Join a cgroup in the parent process workdir
  * @relative_path: The cgroup path, relative to parent process workdir, to join
tools/testing/selftests/bpf/cgroup_helpers.h +1 -0
index f099a166c94d..5c2cb9c8b546 100644
@@ -22,6 +22,7 @@ void remove_cgroup(const char *relative_path);
 unsigned long long get_cgroup_id(const char *relative_path);
 
 int join_cgroup(const char *relative_path);
+int join_root_cgroup(void);
 int join_parent_cgroup(const char *relative_path);
 
 int setup_cgroup_environment(void);
tools/testing/selftests/bpf/cgroup_tcp_skb.h new file +35 -0
new file mode 100644
index 000000000000..7f6b24f102fb
@@ -0,0 +1,35 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. */
+
+/* Define states of a socket to tracking messages sending to and from the
+ * socket.
+ *
+ * These states are based on rfc9293 with some modifications to support
+ * tracking of messages sent out from a socket. For example, when a SYN is
+ * received, a new socket is transiting to the SYN_RECV state defined in
+ * rfc9293. But, we put it in SYN_RECV_SENDING_SYN_ACK state and when
+ * SYN-ACK is sent out, it moves to SYN_RECV state. With this modification,
+ * we can track the message sent out from a socket.
+ */
+
+#ifndef __CGROUP_TCP_SKB_H__
+#define __CGROUP_TCP_SKB_H__
+
+enum {
+	INIT,
+	CLOSED,
+	SYN_SENT,
+	SYN_RECV_SENDING_SYN_ACK,
+	SYN_RECV,
+	ESTABLISHED,
+	FIN_WAIT1,
+	FIN_WAIT2,
+	CLOSE_WAIT_SENDING_ACK,
+	CLOSE_WAIT,
+	CLOSING,
+	LAST_ACK,
+	TIME_WAIT_SENDING_ACK,
+	TIME_WAIT,
+};
+
+#endif /* __CGROUP_TCP_SKB_H__ */
tools/testing/selftests/bpf/prog_tests/cgroup_tcp_skb.c new file +402 -0

402 lines not on this page · raw message

tools/testing/selftests/bpf/progs/cgroup_tcp_skb.c new file +382 -0

382 lines not on this page · raw message

Signature · 1 line
2.34.1
#4 lore raw
In reply to Kui-Feng Lee
This series was applied to bpf/bpf-next.git (master) by Daniel Borkmann <[email protected]>:

Hello:

This series was applied to bpf/bpf-next.git (master) by Daniel Borkmann <[email protected]>:

Quoted 8 lines · On Fri, 23 Jun 2023 18:45:58 -0700 you wrote:

TCP SYN/ACK packets of connections from processes/sockets outside a cgroup on the same host are not received by the cgroup's installed cgroup_skb filters.

There were two BPF cgroup_skb programs attached to a cgroup named "my_cgroup".

[...]

Here is the summary with links:
- [bpf-next,v4,1/2] net: bpf: Check SKB ownership against full socket.

    https://git.kernel.org/bpf/bpf-next/c/223f5f79f2ce
  - [bpf-next,v4,2/2] selftests/bpf: Verify that the cgroup_skb filters receive expected packets.
    https://git.kernel.org/bpf/bpf-next/c/539c7e67aa4a

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html