Fix missing synack in BPF cgroup_skb filters
Subsystems (4)
cover Fix missing synack in BPF cgroup_skb filters
TCP SYN/ACK packets of connections from processes/sockets outside a cgroup on the same host are not received by the cgroup's installed cgroup_skb filters.
There were two BPF cgroup_skb programs attached to a cgroup named "my_cgroup".
SEC("cgroup_skb/ingress")
int ingress(struct __sk_buff *skb)
{
/* .... process skb ... */
return 1;
}
SEC("cgroup_skb/egress")
int egress(struct __sk_buff *skb)
{
/* .... process skb ... */
return 1;
}We discovered that when running the command "nc -6 -l 8000" in "my_group" and connecting to it from outside of "my_cgroup" with the command "nc -6 localhost 8000", the egress filter did not detect the SYN/ACK packet. However, we did observe the SYN/ACK packet at the ingress when connecting from a socket in "my_cgroup" to a socket outside of it.
We came across BPF_CGROUP_RUN_PROG_INET_EGRESS(). This macro is responsible for calling BPF programs that are attached to the egress hook of a cgroup and it skips programs if the sending socket is not the owner of the sk_buff. Specifically, in our situation, the SYN/ACK sk_buff is owned by a struct request_sock instance, but the sending socket is the listener socket we use to receive incoming connections. The request_sock is created to manage an incoming connection.
It has been determined that checking the owner of a sk_buff against the sending socket is not required. Removing this check will allow the filters to receive SYN/ACK packets.
To ensure that cgroup_skb filters can receive all signaling packets, including SYN, SYN/ACK, ACK, FIN, and FIN/ACK. A new self-test has been added as well.
Kui-Feng Lee (2):
net: bpf: Always call BPF cgroup filters for egress. selftests/bpf: Verify that the cgroup_skb filters receive expected
packets.
include/linux/bpf-cgroup.h | 2 +- tools/testing/selftests/bpf/cgroup_helpers.c | 12 + tools/testing/selftests/bpf/cgroup_helpers.h | 1 + tools/testing/selftests/bpf/cgroup_tcp_skb.h | 35 ++ .../selftests/bpf/prog_tests/cgroup_tcp_skb.c | 363 ++++++++++++++++++ .../selftests/bpf/progs/cgroup_tcp_skb.c | 340 ++++++++++++++++ 6 files changed, 752 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/cgroup_tcp_skb.h create mode 100644 tools/testing/selftests/bpf/prog_tests/cgroup_tcp_skb.c create mode 100644 tools/testing/selftests/bpf/progs/cgroup_tcp_skb.c
Signature · 1 line
2.34.1
PATCH 1/2 net: bpf: Always call BPF cgroup filters for egress.
Always call BPF filters if CGROUP BPF is enabled for EGRESS without checking skb->sk against sk.
The filters were called only if sk_buff is owned by the sock that the sk_buff is sent out through. In another words, sk_buff::sk should point to the sock that it is sending through its egress. However, the filters would miss SYNACK sk_buffs that they are owned by a request_sock but sent through the listening sock, that is the socket listening incoming connections. This is an unnecessary restrict.
--- include/linux/bpf-cgroup.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/bpf-cgroup.h +1 -1
@@ -199,7 +199,7 @@ static inline bool cgroup_bpf_sock_enabled(struct sock *sk,#define BPF_CGROUP_RUN_PROG_INET_EGRESS (sk, skb) \ ({ \ int __ret = 0; \ - if (cgroup_bpf_enabled(CGROUP_INET_EGRESS) && sk && sk == skb->sk) { \ + if (cgroup_bpf_enabled(CGROUP_INET_EGRESS) && sk) { \ typeof(sk) __sk = sk_to_full_sk(sk); \ if (sk_fullsock(__sk) && \ cgroup_bpf_sock_enabled(__sk, CGROUP_INET_EGRESS)) \
Signature · 1 line
2.34.1
Re: [PATCH bpf-next 1/2] net: bpf: Always call BPF cgroup filters for egress.
Quoted 6 lines · Kui-Feng Lee
Always call BPF filters if CGROUP BPF is enabled for EGRESS without checking
skb->skagainst sk.The filters were called only if
sk_buffis owned by the sock that thesk_buffis sent out through. In another words,sk_buff::skshould point to
What is "sk_buff::sk" ? Did you mean skb->sk ?
Quoted 15 lines
the sock that it is sending through its egress. However, the filters would miss SYNACK
sk_buffsthat they are owned by arequest_sockbut sent through the listening sock, that is the socket listening incoming connections. This is an unnecessary restrict.Signed-off-by Kui-Feng Lee <[email protected]>--- include/linux/bpf-cgroup.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)include/linux/bpf-cgroup.h +0
-0
include/linux/bpf-cgroup.h static inline bool cgroup_bpf_sock_enabled(struct sock *sk, · line 202
Quoted 5 lines
include/linux/bpf-cgroup.h +1
-1@@ -199,7 +199,7 @@ static inline bool cgroup_bpf_sock_enabled(struct sock *sk,#define BPF_CGROUP_RUN_PROG_INET_EGRESS (sk, skb) \ ({ \ int __ret = 0; \ - if (cgroup_bpf_enabled(CGROUP_INET_EGRESS) && sk && sk == skb->sk) { \ + if (cgroup_bpf_enabled(CGROUP_INET_EGRESS) && sk) {
I did a bit of git-archeology.
That check was there since the beginning of cgroup-bpf and came as a suggestion to use 'sk' instead of 'skb->sk':
https://lore.kernel.org/all/[email protected]/
Using sk is certainly correct. It looks to me that the check was added just for a "piece of mind".
Re: [PATCH bpf-next 1/2] net: bpf: Always call BPF cgroup filters for egress.
Quoted 9 lines · Alexei Starovoitov
Quoted 6 lines · Kui-Feng Lee
Always call BPF filters if CGROUP BPF is enabled for EGRESS without checking
skb->skagainst sk.The filters were called only if
sk_buffis owned by the sock that thesk_buffis sent out through. In another words,sk_buff::skshould point toWhat is "
sk_buff::sk" ? Did you meanskb->sk?
Yes!
Quoted 28 lines
Quoted 16 lines
the sock that it is sending through its egress. However, the filters would miss SYNACK
sk_buffsthat they are owned by arequest_sockbut sent through the listening sock, that is the socket listening incoming connections. This is an unnecessary restrict.Signed-off-by Kui-Feng Lee <[email protected]>--- include/linux/bpf-cgroup.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)include/linux/bpf-cgroup.h +0
-0include/linux/bpf-cgroup.h static inline bool cgroup_bpf_sock_enabled(struct sock *sk, · line 202
Quoted 5 lines
include/linux/bpf-cgroup.h +1
-1@@ -199,7 +199,7 @@ static inline bool cgroup_bpf_sock_enabled(struct sock *sk,#define BPF_CGROUP_RUN_PROG_INET_EGRESS (sk, skb) \ ({ \ int __ret = 0; \ - if (cgroup_bpf_enabled(CGROUP_INET_EGRESS) && sk && sk == skb->sk) { \ + if (cgroup_bpf_enabled(CGROUP_INET_EGRESS) && sk) {I did a bit of git-archeology.
That check was there since the beginning of cgroup-bpf and came as a suggestion to use 'sk' instead of 'skb->sk':
https://lore.kernel.org/all/[email protected]/
Quoted 3 lines
Using sk is certainly correct. It looks to me that the check was added just for a "piece of mind".
Good to know that. Thank you for the confirmation.
PATCH 2/2 selftests/bpf: Verify that the cgroup_skb filters receive expected packets.
In reply to Kui-Feng LeeThis test case includes four scenarios:
1. Connect to the server from outside the cgroup and close the connection from outside the cgroup.
2. Connect to the server from outside the cgroup and close the connection from inside the cgroup.
3. Connect to the server from inside the cgroup and close the connection from outside the cgroup.
4. Connect to the server from inside the cgroup and close the connection from inside the cgroup.
The test case is to verify that cgroup_skb/{egress, ingress} filters receive expected packets including SYN, SYN/ACK, ACK, FIN, and FIN/ACK.
--- tools/testing/selftests/bpf/cgroup_helpers.c | 12 + tools/testing/selftests/bpf/cgroup_helpers.h | 1 + tools/testing/selftests/bpf/cgroup_tcp_skb.h | 35 ++ .../selftests/bpf/prog_tests/cgroup_tcp_skb.c | 363 ++++++++++++++++++ .../selftests/bpf/progs/cgroup_tcp_skb.c | 340 ++++++++++++++++ 5 files changed, 751 insertions(+) create mode 100644 tools/testing/selftests/bpf/cgroup_tcp_skb.h create mode 100644 tools/testing/selftests/bpf/prog_tests/cgroup_tcp_skb.c create mode 100644 tools/testing/selftests/bpf/progs/cgroup_tcp_skb.c
tools/testing/selftests/bpf/cgroup_helpers.c +12 -0
@@ -277,6 +277,18 @@ int join_cgroup(const char *relative_path)return join_cgroup_from_top (cgroup_path); } +/** +* join_root_cgroup() - Join the root cgroup +* +* This function joins the root cgroup. +* +* On success, it returns 0, otherwise on failure it returns 1. +*/ +int join_root_cgroup (void ) +{ +return join_cgroup_from_top (CGROUP_MOUNT_PATH ); +} +/** * join_parent_cgroup() - Join a cgroup in the parent process workdir * @relative_path: The cgroup path, relative to parent process workdir, to join
tools/testing/selftests/bpf/cgroup_helpers.h +1 -0
@@ -22,6 +22,7 @@ void remove_cgroup(const char *relative_path);unsigned long long get_cgroup_id (const char *relative_path);int join_cgroup (const char *relative_path); +int join_root_cgroup (void );int join_parent_cgroup (const char *relative_path);int setup_cgroup_environment (void );
tools/testing/selftests/bpf/cgroup_tcp_skb.h new file +35 -0
@@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2023 Facebook */ + +/* Define states of a socket to tracking messages sending to and from the +* socket. +* +* These states are based on rfc9293 with some modifications to support +* tracking of messages sent out from a socket. For example, when a SYN is +* received, a new socket is transiting to the SYN_RECV state defined in +* rfc9293. But, we put it in SYN_RECV_SENDING_SYN_ACK state and when +* SYN-ACK is sent out, it moves to SYN_RECV state. With this modification, +* we can track the message sent out from a socket. +*/ + +#ifndef __CGROUP_TCP_SKB_H__ +#define __CGROUP_TCP_SKB_H__ + +enum { +INIT , +CLOSED , +SYN_SENT , +SYN_RECV_SENDING_SYN_ACK , +SYN_RECV , +ESTABLISHED , +FIN_WAIT1 , +FIN_WAIT2 , +CLOSE_WAIT_SENDING_ACK , +CLOSE_WAIT , +CLOSING , +LAST_ACK , +TIME_WAIT_SENDING_ACK , +TIME_WAIT , +}; + +#endif /* __CGROUP_TCP_SKB_H__ */
tools/testing/selftests/bpf/prog_tests/cgroup_tcp_skb.c new file +363 -0
363 lines not on this page · raw message
tools/testing/selftests/bpf/progs/cgroup_tcp_skb.c new file +340 -0
340 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH bpf-next 2/2] selftests/bpf: Verify that the cgroup_skb filters receive expected packets.
Quoted 7 lines · Kui-Feng Lee
tools/testing/selftests/bpf/prog_tests/cgroup_tcp_skb.c new file +7
-0+static int close_connection (int *closing_fd,int *peer_fd,int *listen_fd) +{ +int err; + +/* Half shutdown to make sure the closing socket having a chance to +* receive a FIN from the client. +*/
tools/testing/selftests/bpf/prog_tests/cgroup_tcp_skb.c line 230
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/cgroup_tcp_skb.c new file +10
-0+ err =shutdown (*closing_fd,SHUT_WR ); +if (CHECK (err,"shutdown closing_fd" ,"failed: %d\n" , err)) +return -1 ; +usleep (100000 ); + err =close (*peer_fd); +if (CHECK (err,"close peer_fd" ,"failed: %d\n" , err)) +return -1 ; + *peer_fd =-1 ; +usleep (100000 ); + err =close (*closing_fd);
usleep() won't guarantee it. The test will be flaky. Can you make it reliable?
Quoted 14 lines
tools/testing/selftests/bpf/progs/cgroup_tcp_skb.c new file +14
-0+ +/* Run accept() on a socket in the cgroup to receive a new connection. */ +#define EGRESS_ACCEPT \ + case SYN_RECV_SENDING_SYN_ACK: \ + if (tcph.fin || !tcph.syn || tcph.rst || !tcph.ack) \ + g_unexpected++; \ + else \ + g_sock_state = SYN_RECV; \ + break + +#define INGRESS_ACCEPT \ + case INIT: \ + if (!tcph.syn || tcph.fin || tcph.rst || tcph.ack) \ + g_unexpected++; \
tools/testing/selftests/bpf/progs/cgroup_tcp_skb.c line 68
Quoted 10 lines
tools/testing/selftests/bpf/progs/cgroup_tcp_skb.c new file +10
-0+else \ + g_sock_state =SYN_RECV_SENDING_SYN_ACK ; \ +break ; \ +case SYN_RECV : \ +if (tcph.fin || tcph.syn || tcph.rst || !tcph.ack) \ + g_unexpected++; \ +else \ + g_sock_state =ESTABLISHED ; \ +break +
The macros make the code difficult to follow.
Could you do it with normal functions?
Re: [PATCH bpf-next 2/2] selftests/bpf: Verify that the cgroup_skb filters receive expected packets.
Quoted 13 lines · Alexei Starovoitov
Quoted 12 lines · Kui-Feng Lee
tools/testing/selftests/bpf/prog_tests/cgroup_tcp_skb.c new file +12
-0+static int close_connection (int *closing_fd,int *peer_fd,int *listen_fd) +{ +int err; + +/* Half shutdown to make sure the closing socket having a chance to +* receive a FIN from the client. +*/ + err =shutdown (*closing_fd,SHUT_WR ); +if (CHECK (err,"shutdown closing_fd" ,"failed: %d\n" , err)) +return -1 ; +usleep (100000 ); + err =close (*peer_fd);
Quoted 8 lines
tools/testing/selftests/bpf/prog_tests/cgroup_tcp_skb.c line 230
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/cgroup_tcp_skb.c new file +5
-0+if (CHECK (err,"close peer_fd" ,"failed: %d\n" , err)) +return -1 ; + *peer_fd =-1 ; +usleep (100000 ); + err =close (*closing_fd);
usleep()won't guarantee it. The test will be flaky. Can you make it reliable?
What if it checks a counter of packets going through the filter? Will try a couple times until it is too long, one second for example.
Quoted 20 lines
Quoted 20 lines
tools/testing/selftests/bpf/progs/cgroup_tcp_skb.c new file +19
-0+ +/* Run accept() on a socket in the cgroup to receive a new connection. */ +#define EGRESS_ACCEPT \ + case SYN_RECV_SENDING_SYN_ACK: \ + if (tcph.fin || !tcph.syn || tcph.rst || !tcph.ack) \ + g_unexpected++; \ + else \ + g_sock_state = SYN_RECV; \ + break + +#define INGRESS_ACCEPT \ + case INIT: \ + if (!tcph.syn || tcph.fin || tcph.rst || tcph.ack) \ + g_unexpected++; \ + else \ + g_sock_state = SYN_RECV_SENDING_SYN_ACK; \ + break; \ + case SYN_RECV: \ + if (tcph.fin || tcph.syn || tcph.rst || !tcph.ack) \
Quoted 9 lines
tools/testing/selftests/bpf/progs/cgroup_tcp_skb.c line 68
Quoted 5 lines
tools/testing/selftests/bpf/progs/cgroup_tcp_skb.c new file +5
-0+ g_unexpected++; \ + else \ + g_sock_state =ESTABLISHED ; \ +break +The macros make the code difficult to follow.
Could you do it with normal functions?
Sure!
Re: [PATCH bpf-next 2/2] selftests/bpf: Verify that the cgroup_skb filters receive expected packets.
Quoted 58 lines · Kui-Feng Lee
Quoted 13 lines · Alexei Starovoitov
Quoted 12 lines · Kui-Feng Lee
tools/testing/selftests/bpf/prog_tests/cgroup_tcp_skb.c new file +12
-0+static int close_connection (int *closing_fd,int *peer_fd,int *listen_fd) +{ +int err; + +/* Half shutdown to make sure the closing socket having a chance to +* receive a FIN from the client. +*/ + err =shutdown (*closing_fd,SHUT_WR ); +if (CHECK (err,"shutdown closing_fd" ,"failed: %d\n" , err)) +return -1 ; +usleep (100000 ); + err =close (*peer_fd);Quoted 8 lines
tools/testing/selftests/bpf/prog_tests/cgroup_tcp_skb.c line 230
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/cgroup_tcp_skb.c new file +5
-0+if (CHECK (err,"close peer_fd" ,"failed: %d\n" , err)) +return -1 ; + *peer_fd =-1 ; +usleep (100000 ); + err =close (*closing_fd);
usleep()won't guarantee it. The test will be flaky. Can you make it reliable?What if it checks a counter of packets going through the filter? Will try a couple times until it is too long, one second for example.
Quoted 20 lines
Quoted 20 lines
tools/testing/selftests/bpf/progs/cgroup_tcp_skb.c new file +19
-0+ +/* Run accept() on a socket in the cgroup to receive a new connection. */ +#define EGRESS_ACCEPT \ + case SYN_RECV_SENDING_SYN_ACK: \ + if (tcph.fin || !tcph.syn || tcph.rst || !tcph.ack) \ + g_unexpected++; \ + else \ + g_sock_state = SYN_RECV; \ + break + +#define INGRESS_ACCEPT \ + case INIT: \ + if (!tcph.syn || tcph.fin || tcph.rst || tcph.ack) \ + g_unexpected++; \ + else \ + g_sock_state = SYN_RECV_SENDING_SYN_ACK; \ + break; \ + case SYN_RECV: \ + if (tcph.fin || tcph.syn || tcph.rst || !tcph.ack) \Quoted 9 lines
tools/testing/selftests/bpf/progs/cgroup_tcp_skb.c line 68
Quoted 5 lines
tools/testing/selftests/bpf/progs/cgroup_tcp_skb.c new file +5
-0+ g_unexpected++; \ + else \ + g_sock_state =ESTABLISHED ; \ +break +The macros make the code difficult to follow.
Could you do it with normal functions?
Quoted 2 lines
Sure!
please don't use CHECK() macros for new code, use ASSERT_xxx() family
Re: [PATCH bpf-next 2/2] selftests/bpf: Verify that the cgroup_skb filters receive expected packets.
Quoted 59 lines · Andrii Nakryiko
Quoted 58 lines · Kui-Feng Lee
Quoted 13 lines · Alexei Starovoitov
Quoted 12 lines · Kui-Feng Lee
tools/testing/selftests/bpf/prog_tests/cgroup_tcp_skb.c new file +12
-0+static int close_connection (int *closing_fd,int *peer_fd,int *listen_fd) +{ +int err; + +/* Half shutdown to make sure the closing socket having a chance to +* receive a FIN from the client. +*/ + err =shutdown (*closing_fd,SHUT_WR ); +if (CHECK (err,"shutdown closing_fd" ,"failed: %d\n" , err)) +return -1 ; +usleep (100000 ); + err =close (*peer_fd);Quoted 8 lines
tools/testing/selftests/bpf/prog_tests/cgroup_tcp_skb.c line 230
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/cgroup_tcp_skb.c new file +5
-0+if (CHECK (err,"close peer_fd" ,"failed: %d\n" , err)) +return -1 ; + *peer_fd =-1 ; +usleep (100000 ); + err =close (*closing_fd);
usleep()won't guarantee it. The test will be flaky. Can you make it reliable?What if it checks a counter of packets going through the filter? Will try a couple times until it is too long, one second for example.
Quoted 29 lines
Quoted 15 lines
tools/testing/selftests/bpf/progs/cgroup_tcp_skb.c new file +14
-0+ +/* Run accept() on a socket in the cgroup to receive a new connection. */ +#define EGRESS_ACCEPT \ + case SYN_RECV_SENDING_SYN_ACK: \ + if (tcph.fin || !tcph.syn || tcph.rst || !tcph.ack) \ + g_unexpected++; \ + else \ + g_sock_state = SYN_RECV; \ + break + +#define INGRESS_ACCEPT \ + case INIT: \ + if (!tcph.syn || tcph.fin || tcph.rst || tcph.ack) \ + g_unexpected++; \tools/testing/selftests/bpf/progs/cgroup_tcp_skb.c line 68
Quoted 10 lines
tools/testing/selftests/bpf/progs/cgroup_tcp_skb.c new file +10
-0+else \ + g_sock_state =SYN_RECV_SENDING_SYN_ACK ; \ +break ; \ +case SYN_RECV : \ +if (tcph.fin || tcph.syn || tcph.rst || !tcph.ack) \ + g_unexpected++; \ +else \ + g_sock_state =ESTABLISHED ; \ +break +The macros make the code difficult to follow.
Could you do it with normal functions?
Quoted 4 lines
Quoted 2 lines
Sure!
please don't use
CHECK()macros for new code, useASSERT_xxx()family
Got it!