Skip to contentSkip to conversation

bpf, sockmap: handle spurious tcp_msg_wait_data() wakeup

PATCH In review Nnamdi Onyeyiri 4 participants 8 messages Jul 21 – Aug 5

Subsystems (7)
lore
Share & tools
lore
#0 lore raw

v7 cover bpf, sockmap: handle spurious tcp_msg_wait_data() wakeup

Spurious wakeups in tcp_msg_wait_data() aren't being handled by tcp_bpf_recvmsg() and tcp_bpf_recvmsg_parser(), leading to unexpected EAGAIN errors returned by recvfrom()/recv(). Adding handling for the wakeup and a selftest.

This issue was first discovered in an application that adds sockets to a sockmap as a way to view the received data. No redirects or any other operations are performed.

Sashiko has noted a few other pre-existing issues in the same area
(https://patch.msgid.link/[email protected])
that can lead to EAGAIN in tcp_bpf_recvmsg() and tcp_bpf_recvmsg_parser(). A local run of Sashiko also identified a potential issue in tcp_bpf_recvmsg() handling zero-data FIN packets when the MSG_PEEK flag is set. To prevent this patchset from growing too large, I intend to submit follow up patches to address these once this one has been accepted.

Changes in v7:
- In tcp_bpf_recvmsg check for data in the TCP_CLOSE, !timeo and signal_pending branches.
- Refactored changes to tcp_bpf_recvmsg to reduce duplication.
- Increase the number of iterations in the selftests.
- Documented selftests numeric constants.
- Link to v6: https://patch.msgid.link/[email protected]

Changes in v6:
- Closing the file descriptor in the selftest worker thread.
- Ensuring the selftest loop breaks early for an error in the worker thread.
- Added comments to selftest regarding focus on EAGAIN error.
- Updated selftest commit message to imperative mood.
- Link to v5: https://patch.msgid.link/[email protected]

Changes in v5:
- Move selftest from net into bpf/prog_tests/sockmap_basic.c
- Link to v4: https://patch.msgid.link/[email protected]

Changes in v4:
- Fix potential data loss in tcp_bpf_recvmsg() when a FIN or RST has been received.
- Check the return code of pthread_create() in the selftest.
- Fix race caused by using EXPECT macros in a thread in selftest.
- Link to v3: https://patch.msgid.link/[email protected]

Changes in v3:
- Added the sockmap_recvfrom selftest.
- Link to v2: https://patch.msgid.link/[email protected]

Changes in v2:
- In tcp_bpf_recvmsg, handle signals and the socket closing in the loop.
- Fix spurious wakeups when SO_RCVTIMEO has been set on the socket.
- Link to v1: https://patch.msgid.link/[email protected]

---
Nnamdi Onyeyiri (2):
bpf, sockmap: handle spurious tcp_msg_wait_data() wakeup selftests/bpf: add sockmap recvfrom EAGAIN selftest

 net/ipv4/tcp_bpf.c                            |  72 ++++++++--
 .../selftests/bpf/prog_tests/sockmap_basic.c  | 136 ++++++++++++++++++
 2 files changed, 199 insertions(+), 9 deletions(-)
Signature · 1 line
2.52.0
#1 lore raw

PATCH v7 1/2 bpf, sockmap: handle spurious tcp_msg_wait_data() wakeup

recvfrom()/recv() are documented as only returning EAGAIN for blocking sockets when they have a receive timeout configured. However, adding a blocking ipv4 tcp socket without a receive timeout to a sockmap will cause EAGAIN errors sporadically. A socket with a receive timeout may return EAGAIN before the timeout expires.

There are 2 code paths affected by this:

1. tcp_bpf_recvmsg() - Used when the socket has been added to a sockmap that has no verdict program attached.

2. tcp_bpf_recvmsg_parser() - Used when the socket has been added to a sockmap that has a verdict program. To reproduce this issue, it is enough for the verdict program to do nothing but return SK_PASS.

In both cases this happens when tcp_msg_wait_data() wakes spuriously (returning 0). To fix it, we now loop back to msg_bytes_ready instead of returning -EAGAIN on spurious wakeup.

To ensure the looping does not cause sockets with a SO_RCVTIMEO set to wait excessively long, tcp_msg_wait_data() now takes a pointer to timeo, allowing sk_wait_event() to update it as appropriate.

The logic in tcp_bpf_recvmsg_parser() that allow it to handle signals, socket errors and closuers in its loop was also added to tcp_bpf_recvmsg().

---
 net/ipv4/tcp_bpf.c | 72 ++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 63 insertions(+), 9 deletions(-)
net/ipv4/tcp_bpf.c +63 -9
index cc0bd73f36b6..1755fcf726fc 100644
@@ -179,7 +179,7 @@ EXPORT_SYMBOL_GPL(tcp_bpf_sendmsg_redir);
 
 #ifdef CONFIG_BPF_SYSCALL
 static int tcp_msg_wait_data(struct sock *sk, struct sk_psock *psock,
-			     long timeo)
+			     long *timeo)
 {
 	DEFINE_WAIT_FUNC(wait, woken_wake_function);
 	int ret = 0;
@@ -187,12 +187,12 @@ static int tcp_msg_wait_data(struct sock *sk, struct sk_psock *psock,
 	if (sk->sk_shutdown & RCV_SHUTDOWN)
 		return 1;
 
-	if (!timeo)
+	if (!*timeo)
 		return ret;
 
 	add_wait_queue(sk_sleep(sk), &wait);
 	sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
-	ret = sk_wait_event(sk, &timeo,
+	ret = sk_wait_event(sk, timeo,
 			    !list_empty(&psock->ingress_msg) ||
 			    !skb_queue_empty_lockless(&sk->sk_receive_queue), &wait);
 	sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk);
@@ -229,6 +229,7 @@ static int tcp_bpf_recvmsg_parser(struct sock *sk,
 	int copied_from_self = 0;
 	int copied = 0;
 	u32 seq;
+	long timeo;
 
 	if (unlikely(flags & MSG_ERRQUEUE))
 		return inet_recv_error(sk, msg, len);
@@ -262,6 +263,8 @@ static int tcp_bpf_recvmsg_parser(struct sock *sk,
 		}
 	}
 
+	timeo = sock_rcvtimeo(sk, flags & MSG_DONTWAIT);
+
 msg_bytes_ready:
 	copied = __sk_msg_recvmsg(sk, psock, msg, len, flags, &copied_from_self);
 	/* The typical case for EFAULT is the socket was gracefully
@@ -280,7 +283,6 @@ static int tcp_bpf_recvmsg_parser(struct sock *sk,
 	}
 	seq += copied_from_self;
 	if (!copied) {
-		long timeo;
 		int data;
 
 		if (sock_flag(sk, SOCK_DONE))
@@ -299,7 +301,6 @@ static int tcp_bpf_recvmsg_parser(struct sock *sk,
 			goto out;
 		}
 
-		timeo = sock_rcvtimeo(sk, flags & MSG_DONTWAIT);
 		if (!timeo) {
 			copied = -EAGAIN;
 			goto out;
@@ -310,13 +311,15 @@ static int tcp_bpf_recvmsg_parser(struct sock *sk,
 			goto out;
 		}
 
-		data = tcp_msg_wait_data(sk, psock, timeo);
+		data = tcp_msg_wait_data(sk, psock, &timeo);
 		if (data < 0) {
 			copied = data;
 			goto unlock;
 		}
 		if (data && !sk_psock_queue_empty(psock))
 			goto msg_bytes_ready;
+		if (!data && timeo > 0)
+			goto msg_bytes_ready;
 		copied = -EAGAIN;
 	}
 out:
@@ -355,6 +358,7 @@ static int tcp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 {
 	struct sk_psock *psock;
 	int copied, ret;
+	long timeo;
 
 	if (unlikely(flags & MSG_ERRQUEUE))
 		return inet_recv_error(sk, msg, len);
@@ -371,14 +375,52 @@ static int tcp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 		return tcp_recvmsg(sk, msg, len, flags);
 	}
 	lock_sock(sk);
+
+	timeo = sock_rcvtimeo(sk, flags & MSG_DONTWAIT);
+
 msg_bytes_ready:
 	copied = sk_msg_recvmsg(sk, psock, msg, len, flags);
 	if (!copied) {
-		long timeo;
 		int data;
 
-		timeo = sock_rcvtimeo(sk, flags & MSG_DONTWAIT);
-		data = tcp_msg_wait_data(sk, psock, timeo);
+		if (sock_flag(sk, SOCK_DONE)) {
+			ret = 0;
+			goto check_queues;
+		}
+
+		if (sk->sk_err) {
+			if (!sk_psock_queue_empty(psock))
+				goto msg_bytes_ready;
+			if (!skb_queue_empty(&sk->sk_receive_queue)) {
+				release_sock(sk);
+				sk_psock_put(sk, psock);
+				return tcp_recvmsg(sk, msg, len, flags);
+			}
+			ret = sock_error(sk);
+			goto unlock;
+		}
+
+		if (sk->sk_shutdown & RCV_SHUTDOWN) {
+			ret = 0;
+			goto check_queues;
+		}
+
+		if (sk->sk_state == TCP_CLOSE) {
+			ret = -ENOTCONN;
+			goto check_queues;
+		}
+
+		if (!timeo) {
+			ret = -EAGAIN;
+			goto check_queues;
+		}
+
+		if (signal_pending(current)) {
+			ret = sock_intr_errno(timeo);
+			goto check_queues;
+		}
+
+		data = tcp_msg_wait_data(sk, psock, &timeo);
 		if (data < 0) {
 			ret = data;
 			goto unlock;
@@ -390,6 +432,8 @@ static int tcp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 			sk_psock_put(sk, psock);
 			return tcp_recvmsg(sk, msg, len, flags);
 		}
+		if (!data && timeo > 0)
+			goto msg_bytes_ready;
 		copied = -EAGAIN;
 	}
 	ret = copied;
@@ -398,6 +442,16 @@ static int tcp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 	release_sock(sk);
 	sk_psock_put(sk, psock);
 	return ret;
+
+check_queues:
+	if (!sk_psock_queue_empty(psock))
+		goto msg_bytes_ready;
+	if (!skb_queue_empty(&sk->sk_receive_queue)) {
+		release_sock(sk);
+		sk_psock_put(sk, psock);
+		return tcp_recvmsg(sk, msg, len, flags);
+	}
+	goto unlock;
 }
 
 static int tcp_bpf_send_verdict(struct sock *sk, struct sk_psock *psock,
Signature · 1 line
2.52.0
#2 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] tcp_bpf_recvmsg_parser() fails to handle unparsed data in sk_receive_queue, returning premature EAGAIN on blocking sockets. --

--- Patch [1]: [PATCH v7 1/2] bpf, sockmap: handle spurious tcp_msg_wait_data() wakeup --- commit 7468e0f3ba324455b02b600d4c77965cd4caa92c
Author: Nnamdi Onyeyiri <[email protected]>

bpf, sockmap: handle spurious tcp_msg_wait_data() wakeup

This commit addresses spurious wakeups from tcp_msg_wait_data() by looping back to msg_bytes_ready instead of returning -EAGAIN. It also passes a pointer for timeo to allow sk_wait_event() to update the remaining timeout.

Quoted 4 lines
net/ipv4/tcp_bpf.c +0 -0
index 8e905b50deadb..87aef99a13792 100644

[ ... ]

Quoted 4 lines
net/ipv4/tcp_bpf.c +0 -0
@@ -310,13 +311,15 @@ static int tcp_bpf_recvmsg_parser(struct sock *sk,
 			goto out;
 		}
 

net/ipv4/tcp_bpf.c static int tcp_bpf_recvmsg_parser(struct sock *sk, · line 322

Quoted 10 lines
net/ipv4/tcp_bpf.c +3 -1
-		data = tcp_msg_wait_data(sk, psock, timeo);
+		data = tcp_msg_wait_data(sk, psock, &timeo);
 		if (data < 0) {
 			copied = data;
 			goto unlock;
 		}
 		if (data && !sk_psock_queue_empty(psock))
 			goto msg_bytes_ready;
+		if (!data && timeo > 0)
+			goto msg_bytes_ready;

[Severity: High]
This is a pre-existing issue, but could this return a premature -EAGAIN on a blocking socket when data is waiting to be parsed?

If data arrives and is added to sk_receive_queue, but a stream parser requires more bytes to complete a message (or the parser is deferred to a workqueue), tcp_msg_wait_data() will return 1.

Because psock->ingress_msg is still empty, both the existing condition:

    if (data && !sk_psock_queue_empty(psock))
and the new condition:
    if (!data && timeo > 0)

will evaluate to false.

Does this mean tcp_bpf_recvmsg_parser() falls through and returns -EAGAIN instead of continuing to wait for the remaining data to be parsed?

Quoted 2 lines
net/ipv4/tcp_bpf.c +0 -0
 		copied = -EAGAIN;
 	}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
maintainer
#3 lore raw
In reply to Nnamdi Onyeyiri
Quoted 138 lines · Nnamdi Onyeyiri

recvfrom()/recv() are documented as only returning EAGAIN for blocking sockets when they have a receive timeout configured. However, adding a blocking ipv4 tcp socket without a receive timeout to a sockmap will cause EAGAIN errors sporadically. A socket with a receive timeout may return EAGAIN before the timeout expires.

There are 2 code paths affected by this:

1. tcp_bpf_recvmsg() - Used when the socket has been added to a sockmap that has no verdict program attached.

2. tcp_bpf_recvmsg_parser() - Used when the socket has been added to a sockmap that has a verdict program. To reproduce this issue, it is enough for the verdict program to do nothing but return SK_PASS.

In both cases this happens when tcp_msg_wait_data() wakes spuriously (returning 0). To fix it, we now loop back to msg_bytes_ready instead of returning -EAGAIN on spurious wakeup.

To ensure the looping does not cause sockets with a SO_RCVTIMEO set to wait excessively long, tcp_msg_wait_data() now takes a pointer to timeo, allowing sk_wait_event() to update it as appropriate.

The logic in tcp_bpf_recvmsg_parser() that allow it to handle signals, socket errors and closuers in its loop was also added to tcp_bpf_recvmsg().

---
 net/ipv4/tcp_bpf.c | 72 ++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 63 insertions(+), 9 deletions(-)
net/ipv4/tcp_bpf.c +19 -9
index cc0bd73f36b6..1755fcf726fc 100644
@@ -179,7 +179,7 @@ EXPORT_SYMBOL_GPL(tcp_bpf_sendmsg_redir);
 
 #ifdef CONFIG_BPF_SYSCALL
 static int tcp_msg_wait_data(struct sock *sk, struct sk_psock *psock,
-			     long timeo)
+			     long *timeo)
 {
 	DEFINE_WAIT_FUNC(wait, woken_wake_function);
 	int ret = 0;
@@ -187,12 +187,12 @@ static int tcp_msg_wait_data(struct sock *sk, struct sk_psock *psock,
 	if (sk->sk_shutdown & RCV_SHUTDOWN)
 		return 1;
 
-	if (!timeo)
+	if (!*timeo)
 		return ret;
 
 	add_wait_queue(sk_sleep(sk), &wait);
 	sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
-	ret = sk_wait_event(sk, &timeo,
+	ret = sk_wait_event(sk, timeo,
 			    !list_empty(&psock->ingress_msg) ||
 			    !skb_queue_empty_lockless(&sk->sk_receive_queue), &wait);
 	sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk);
@@ -229,6 +229,7 @@ static int tcp_bpf_recvmsg_parser(struct sock *sk,
 	int copied_from_self = 0;
 	int copied = 0;
 	u32 seq;
+	long timeo;
 
 	if (unlikely(flags & MSG_ERRQUEUE))
 		return inet_recv_error(sk, msg, len);
@@ -262,6 +263,8 @@ static int tcp_bpf_recvmsg_parser(struct sock *sk,
 		}
 	}
 
+	timeo = sock_rcvtimeo(sk, flags & MSG_DONTWAIT);
+
 msg_bytes_ready:
 	copied = __sk_msg_recvmsg(sk, psock, msg, len, flags, &copied_from_self);
 	/* The typical case for EFAULT is the socket was gracefully
@@ -280,7 +283,6 @@ static int tcp_bpf_recvmsg_parser(struct sock *sk,
 	}
 	seq += copied_from_self;
 	if (!copied) {
-		long timeo;
 		int data;
 
 		if (sock_flag(sk, SOCK_DONE))
@@ -299,7 +301,6 @@ static int tcp_bpf_recvmsg_parser(struct sock *sk,
 			goto out;
 		}
 
-		timeo = sock_rcvtimeo(sk, flags & MSG_DONTWAIT);
 		if (!timeo) {
 			copied = -EAGAIN;
 			goto out;
@@ -310,13 +311,15 @@ static int tcp_bpf_recvmsg_parser(struct sock *sk,
 			goto out;
 		}
 
-		data = tcp_msg_wait_data(sk, psock, timeo);
+		data = tcp_msg_wait_data(sk, psock, &timeo);
 		if (data < 0) {
 			copied = data;
 			goto unlock;
 		}
 		if (data && !sk_psock_queue_empty(psock))
 			goto msg_bytes_ready;
+		if (!data && timeo > 0)
+			goto msg_bytes_ready;
 		copied = -EAGAIN;
 	}
 out:
@@ -355,6 +358,7 @@ static int tcp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 {
 	struct sk_psock *psock;
 	int copied, ret;
+	long timeo;
 
 	if (unlikely(flags & MSG_ERRQUEUE))
 		return inet_recv_error(sk, msg, len);
@@ -371,14 +375,52 @@ static int tcp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 		return tcp_recvmsg(sk, msg, len, flags);
 	}
 	lock_sock(sk);
+
+	timeo = sock_rcvtimeo(sk, flags & MSG_DONTWAIT);
+
 msg_bytes_ready:
 	copied = sk_msg_recvmsg(sk, psock, msg, len, flags);
 	if (!copied) {
-		long timeo;
 		int data;
 
-		timeo = sock_rcvtimeo(sk, flags & MSG_DONTWAIT);
-		data = tcp_msg_wait_data(sk, psock, timeo);
+		if (sock_flag(sk, SOCK_DONE)) {
+			ret = 0;
+			goto check_queues;
+		}
+
+		if (sk->sk_err) {

net/ipv4/tcp_bpf.c static int tcp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, · line 401

Quoted 10 lines
net/ipv4/tcp_bpf.c +10 -0
+			if (!sk_psock_queue_empty(psock))
+				goto msg_bytes_ready;
+			if (!skb_queue_empty(&sk->sk_receive_queue)) {
+				release_sock(sk);
+				sk_psock_put(sk, psock);
+				return tcp_recvmsg(sk, msg, len, flags);
+			}
+			ret = sock_error(sk);
+			goto unlock;
+		}

Isn't that branch the same as:

        if (sk->sk_err) {
                ret = sock_error(sk);
                goto check_queues;
        }

?

Quoted 51 lines
net/ipv4/tcp_bpf.c +34 -0
+
+		if (sk->sk_shutdown & RCV_SHUTDOWN) {
+			ret = 0;
+			goto check_queues;
+		}
+
+		if (sk->sk_state == TCP_CLOSE) {
+			ret = -ENOTCONN;
+			goto check_queues;
+		}
+
+		if (!timeo) {
+			ret = -EAGAIN;
+			goto check_queues;
+		}
+
+		if (signal_pending(current)) {
+			ret = sock_intr_errno(timeo);
+			goto check_queues;
+		}
+
+		data = tcp_msg_wait_data(sk, psock, &timeo);
 		if (data < 0) {
 			ret = data;
 			goto unlock;
@@ -390,6 +432,8 @@ static int tcp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 			sk_psock_put(sk, psock);
 			return tcp_recvmsg(sk, msg, len, flags);
 		}
+		if (!data && timeo > 0)
+			goto msg_bytes_ready;
 		copied = -EAGAIN;
 	}
 	ret = copied;
@@ -398,6 +442,16 @@ static int tcp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 	release_sock(sk);
 	sk_psock_put(sk, psock);
 	return ret;
+
+check_queues:
+	if (!sk_psock_queue_empty(psock))
+		goto msg_bytes_ready;
+	if (!skb_queue_empty(&sk->sk_receive_queue)) {
+		release_sock(sk);
+		sk_psock_put(sk, psock);
+		return tcp_recvmsg(sk, msg, len, flags);
+	}
+	goto unlock;
 }
 
 static int tcp_bpf_send_verdict(struct sock *sk, struct sk_psock *psock,
#4 lore raw
Quoted 152 lines · Jakub Sitnicki
Quoted 138 lines · Nnamdi Onyeyiri

recvfrom()/recv() are documented as only returning EAGAIN for blocking sockets when they have a receive timeout configured. However, adding a blocking ipv4 tcp socket without a receive timeout to a sockmap will cause EAGAIN errors sporadically. A socket with a receive timeout may return EAGAIN before the timeout expires.

There are 2 code paths affected by this:

1. tcp_bpf_recvmsg() - Used when the socket has been added to a sockmap that has no verdict program attached.

2. tcp_bpf_recvmsg_parser() - Used when the socket has been added to a sockmap that has a verdict program. To reproduce this issue, it is enough for the verdict program to do nothing but return SK_PASS.

In both cases this happens when tcp_msg_wait_data() wakes spuriously (returning 0). To fix it, we now loop back to msg_bytes_ready instead of returning -EAGAIN on spurious wakeup.

To ensure the looping does not cause sockets with a SO_RCVTIMEO set to wait excessively long, tcp_msg_wait_data() now takes a pointer to timeo, allowing sk_wait_event() to update it as appropriate.

The logic in tcp_bpf_recvmsg_parser() that allow it to handle signals, socket errors and closuers in its loop was also added to tcp_bpf_recvmsg().

---
 net/ipv4/tcp_bpf.c | 72 ++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 63 insertions(+), 9 deletions(-)
net/ipv4/tcp_bpf.c +19 -9
index cc0bd73f36b6..1755fcf726fc 100644
@@ -179,7 +179,7 @@ EXPORT_SYMBOL_GPL(tcp_bpf_sendmsg_redir);
 
 #ifdef CONFIG_BPF_SYSCALL
 static int tcp_msg_wait_data(struct sock *sk, struct sk_psock *psock,
-			     long timeo)
+			     long *timeo)
 {
 	DEFINE_WAIT_FUNC(wait, woken_wake_function);
 	int ret = 0;
@@ -187,12 +187,12 @@ static int tcp_msg_wait_data(struct sock *sk, struct sk_psock *psock,
 	if (sk->sk_shutdown & RCV_SHUTDOWN)
 		return 1;
 
-	if (!timeo)
+	if (!*timeo)
 		return ret;
 
 	add_wait_queue(sk_sleep(sk), &wait);
 	sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
-	ret = sk_wait_event(sk, &timeo,
+	ret = sk_wait_event(sk, timeo,
 			    !list_empty(&psock->ingress_msg) ||
 			    !skb_queue_empty_lockless(&sk->sk_receive_queue), &wait);
 	sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk);
@@ -229,6 +229,7 @@ static int tcp_bpf_recvmsg_parser(struct sock *sk,
 	int copied_from_self = 0;
 	int copied = 0;
 	u32 seq;
+	long timeo;
 
 	if (unlikely(flags & MSG_ERRQUEUE))
 		return inet_recv_error(sk, msg, len);
@@ -262,6 +263,8 @@ static int tcp_bpf_recvmsg_parser(struct sock *sk,
 		}
 	}
 
+	timeo = sock_rcvtimeo(sk, flags & MSG_DONTWAIT);
+
 msg_bytes_ready:
 	copied = __sk_msg_recvmsg(sk, psock, msg, len, flags, &copied_from_self);
 	/* The typical case for EFAULT is the socket was gracefully
@@ -280,7 +283,6 @@ static int tcp_bpf_recvmsg_parser(struct sock *sk,
 	}
 	seq += copied_from_self;
 	if (!copied) {
-		long timeo;
 		int data;
 
 		if (sock_flag(sk, SOCK_DONE))
@@ -299,7 +301,6 @@ static int tcp_bpf_recvmsg_parser(struct sock *sk,
 			goto out;
 		}
 
-		timeo = sock_rcvtimeo(sk, flags & MSG_DONTWAIT);
 		if (!timeo) {
 			copied = -EAGAIN;
 			goto out;
@@ -310,13 +311,15 @@ static int tcp_bpf_recvmsg_parser(struct sock *sk,
 			goto out;
 		}
 
-		data = tcp_msg_wait_data(sk, psock, timeo);
+		data = tcp_msg_wait_data(sk, psock, &timeo);
 		if (data < 0) {
 			copied = data;
 			goto unlock;
 		}
 		if (data && !sk_psock_queue_empty(psock))
 			goto msg_bytes_ready;
+		if (!data && timeo > 0)
+			goto msg_bytes_ready;
 		copied = -EAGAIN;
 	}
 out:
@@ -355,6 +358,7 @@ static int tcp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 {
 	struct sk_psock *psock;
 	int copied, ret;
+	long timeo;
 
 	if (unlikely(flags & MSG_ERRQUEUE))
 		return inet_recv_error(sk, msg, len);
@@ -371,14 +375,52 @@ static int tcp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 		return tcp_recvmsg(sk, msg, len, flags);
 	}
 	lock_sock(sk);
+
+	timeo = sock_rcvtimeo(sk, flags & MSG_DONTWAIT);
+
 msg_bytes_ready:
 	copied = sk_msg_recvmsg(sk, psock, msg, len, flags);
 	if (!copied) {
-		long timeo;
 		int data;
 
-		timeo = sock_rcvtimeo(sk, flags & MSG_DONTWAIT);
-		data = tcp_msg_wait_data(sk, psock, timeo);
+		if (sock_flag(sk, SOCK_DONE)) {
+			ret = 0;
+			goto check_queues;
+		}
+
+		if (sk->sk_err) {

net/ipv4/tcp_bpf.c static int tcp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, · line 401

Quoted 10 lines
net/ipv4/tcp_bpf.c +10 -0
+			if (!sk_psock_queue_empty(psock))
+				goto msg_bytes_ready;
+			if (!skb_queue_empty(&sk->sk_receive_queue)) {
+				release_sock(sk);
+				sk_psock_put(sk, psock);
+				return tcp_recvmsg(sk, msg, len, flags);
+			}
+			ret = sock_error(sk);
+			goto unlock;
+		}

Isn't that branch the same as:

Quoted 7 lines
        if (sk->sk_err) {
                ret = sock_error(sk);
                goto check_queues;
        }

?

sock_error clears the error in sk->sk_err. using check_queues means tcp_recvmsg would not see it. is that ok?

Quoted 51 lines
Quoted 51 lines
net/ipv4/tcp_bpf.c +34 -0
+
+		if (sk->sk_shutdown & RCV_SHUTDOWN) {
+			ret = 0;
+			goto check_queues;
+		}
+
+		if (sk->sk_state == TCP_CLOSE) {
+			ret = -ENOTCONN;
+			goto check_queues;
+		}
+
+		if (!timeo) {
+			ret = -EAGAIN;
+			goto check_queues;
+		}
+
+		if (signal_pending(current)) {
+			ret = sock_intr_errno(timeo);
+			goto check_queues;
+		}
+
+		data = tcp_msg_wait_data(sk, psock, &timeo);
 		if (data < 0) {
 			ret = data;
 			goto unlock;
@@ -390,6 +432,8 @@ static int tcp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 			sk_psock_put(sk, psock);
 			return tcp_recvmsg(sk, msg, len, flags);
 		}
+		if (!data && timeo > 0)
+			goto msg_bytes_ready;
 		copied = -EAGAIN;
 	}
 	ret = copied;
@@ -398,6 +442,16 @@ static int tcp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 	release_sock(sk);
 	sk_psock_put(sk, psock);
 	return ret;
+
+check_queues:
+	if (!sk_psock_queue_empty(psock))
+		goto msg_bytes_ready;
+	if (!skb_queue_empty(&sk->sk_receive_queue)) {
+		release_sock(sk);
+		sk_psock_put(sk, psock);
+		return tcp_recvmsg(sk, msg, len, flags);
+	}
+	goto unlock;
 }
 
 static int tcp_bpf_send_verdict(struct sock *sk, struct sk_psock *psock,
#5 lore raw

PATCH v7 2/2 selftests/bpf: add sockmap recvfrom EAGAIN selftest

In reply to Nnamdi Onyeyiri

These selftests exercise the tcp_bpf_recvmsg() and tcp_bpf_recvmsg_parser() functions, to ensure that they are properly handling spurious wakeups in tcp_msg_wait_data().

The expected behaviour is that recvfrom() does not return an EAGAIN error. If the spurious wakeups are incorrectly handled, this assertion will fail.

---
 .../selftests/bpf/prog_tests/sockmap_basic.c  | 136 ++++++++++++++++++
 1 file changed, 136 insertions(+)
tools/testing/selftests/bpf/prog_tests/sockmap_basic.c +136 -0
index cb3229711f93..afda4d90e573 100644
@@ -1373,6 +1373,138 @@ static void test_sockmap_multi_channels(int sotype)
 	test_sockmap_pass_prog__destroy(skel);
 }
 
+static void *test_sockmap_recvfrom_eagain_thread(void *arg)
+{
+	int fd = *(int *)arg;
+	char buf[1024];
+	void *result = NULL;
+
+	while (true) {
+		ssize_t len = recvfrom(fd, buf, sizeof(buf), 0, NULL, NULL);
+
+		if (len == -1) {
+			if (errno == EINTR)
+				continue;
+			result = (void *)1;
+			break;
+		}
+
+		if (!len || buf[len - 1] == 'e')
+			break;
+	}
+
+	send(fd, "test", 4, MSG_NOSIGNAL);
+
+	close(fd);
+
+	return result;
+}
+
+static void test_sockmap_recvfrom_eagain(bool with_verdict)
+{
+	struct test_sockmap_pass_prog *skel = NULL;
+	struct bpf_program *prog = NULL;
+	char *buf = NULL;
+	int map, err;
+
+	/*
+	 * the size of the buffer to send.  completion of the sending can trigger a spurious wake
+	 * up.  larger values make the issue more likely, but make successful test runs longer.
+	 */
+	const size_t buflen = 1024 * 1024 * 25;
+
+	/*
+	 * maximum number of attempts to reproduce EAGAIN on spurious wake up.  more attempts
+	 * increases the chances of triggering the issue, and the test ends as soon as we do.
+	 * however, successful runs would take longer to complete.
+	 */
+	const int max_attempts = 300;
+
+	skel = test_sockmap_pass_prog__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "open_and_load"))
+		return;
+
+	map = bpf_map__fd(skel->maps.sock_map_rx);
+
+	if (with_verdict) {
+		prog = skel->progs.prog_skb_verdict;
+		err = bpf_prog_attach(bpf_program__fd(prog), map, BPF_SK_SKB_STREAM_VERDICT, 0);
+		if (!ASSERT_OK(err, "bpf_prog_attach verdict"))
+			goto cleanup;
+	}
+
+	buf = malloc(buflen);
+	if (!ASSERT_OK_PTR(buf, "malloc buf"))
+		goto cleanup;
+	memset(buf, 0, buflen);
+	buf[buflen - 1] = 'e';
+
+	for (int i = 0; i < max_attempts; ++i) {
+		ssize_t len;
+		char ignored[128];
+		pthread_t thread;
+		bool thread_created = false;
+		size_t rem = buflen;
+		int c = -1, p = -1, zero = 0;
+		bool success = false;
+
+		err = create_pair(AF_INET, SOCK_STREAM, &c, &p);
+		if (!ASSERT_OK(err, "create_pair"))
+			goto end_attempt;
+
+		err = pthread_create(&thread, NULL, &test_sockmap_recvfrom_eagain_thread, &p);
+		if (!ASSERT_OK(err, "pthread_create"))
+			goto end_attempt;
+		thread_created = true;
+
+		err = bpf_map_update_elem(map, &zero, &c, BPF_ANY);
+		if (!ASSERT_OK(err, "bpf_map_update_elem"))
+			goto end_attempt;
+
+		while (rem) {
+			len = xsend(c, buf + (buflen - rem), rem, 0);
+			if (len == -1)
+				goto end_attempt;
+			rem -= len;
+		}
+
+		/* we cannot use recv_timeout(), otherwise EAGAIN would be an expected errno. */
+		len = recvfrom(c, ignored, sizeof(ignored), 0, NULL, NULL);
+
+		/*
+		 * we are checking for the invalid return of EAGAIN, any other return is considered
+		 * successful for the purposes of this test.
+		 */
+		if (len < 0 && !ASSERT_NEQ(errno, EAGAIN, "recvfrom eagain"))
+			goto end_attempt;
+
+		success = true;
+
+end_attempt:
+		if (c >= 0)
+			close(c);
+
+		if (thread_created) {
+			void *retval = NULL;
+
+			pthread_join(thread, &retval);
+			if (!ASSERT_NULL(retval, "retval"))
+				success = false;
+		}
+
+		if (!thread_created && p >= 0)
+			close(p);
+		if (!success)
+			break;
+	}
+
+cleanup:
+	if (buf)
+		free(buf);
+
+	test_sockmap_pass_prog__destroy(skel);
+}
+
 void test_sockmap_basic(void)
 {
 	if (test__start_subtest("sockmap create_update_free"))
@@ -1451,4 +1583,8 @@ void test_sockmap_basic(void)
 		test_sockmap_multi_channels(SOCK_STREAM);
 	if (test__start_subtest("sockmap udp multi channels"))
 		test_sockmap_multi_channels(SOCK_DGRAM);
+	if (test__start_subtest("sockmap recvfrom eagain"))
+		test_sockmap_recvfrom_eagain(false);
+	if (test__start_subtest("sockmap recvfrom eagain with verdict"))
+		test_sockmap_recvfrom_eagain(true);
 }
Signature · 1 line
2.52.0
#6 lore raw
In reply to Nnamdi Onyeyiri
Quoted 55 lines · Nnamdi Onyeyiri

Spurious wakeups in tcp_msg_wait_data() aren't being handled by tcp_bpf_recvmsg() and tcp_bpf_recvmsg_parser(), leading to unexpected EAGAIN errors returned by recvfrom()/recv(). Adding handling for the wakeup and a selftest.

This issue was first discovered in an application that adds sockets to a sockmap as a way to view the received data. No redirects or any other operations are performed.

Sashiko has noted a few other pre-existing issues in the same area
(https://patch.msgid.link/[email protected])
that can lead to EAGAIN in tcp_bpf_recvmsg() and tcp_bpf_recvmsg_parser(). A local run of Sashiko also identified a potential issue in tcp_bpf_recvmsg() handling zero-data FIN packets when the MSG_PEEK flag is set. To prevent this patchset from growing too large, I intend to submit follow up patches to address these once this one has been accepted.

Changes in v7:
- In tcp_bpf_recvmsg check for data in the TCP_CLOSE, !timeo and signal_pending branches.
- Refactored changes to tcp_bpf_recvmsg to reduce duplication.
- Increase the number of iterations in the selftests.
- Documented selftests numeric constants.
- Link to v6: https://patch.msgid.link/[email protected]

Changes in v6:
- Closing the file descriptor in the selftest worker thread.
- Ensuring the selftest loop breaks early for an error in the worker thread.
- Added comments to selftest regarding focus on EAGAIN error.
- Updated selftest commit message to imperative mood.
- Link to v5: https://patch.msgid.link/[email protected]

Changes in v5:
- Move selftest from net into bpf/prog_tests/sockmap_basic.c
- Link to v4: https://patch.msgid.link/[email protected]

Changes in v4:
- Fix potential data loss in tcp_bpf_recvmsg() when a FIN or RST has been received.
- Check the return code of pthread_create() in the selftest.
- Fix race caused by using EXPECT macros in a thread in selftest.
- Link to v3: https://patch.msgid.link/[email protected]

Changes in v3:
- Added the sockmap_recvfrom selftest.
- Link to v2: https://patch.msgid.link/[email protected]

Changes in v2:
- In tcp_bpf_recvmsg, handle signals and the socket closing in the loop.
- Fix spurious wakeups when SO_RCVTIMEO has been set on the socket.
- Link to v1: https://patch.msgid.link/[email protected]

Quoted 12 lines

---
Nnamdi Onyeyiri (2):
bpf, sockmap: handle spurious tcp_msg_wait_data() wakeup selftests/bpf: add sockmap recvfrom EAGAIN selftest

 net/ipv4/tcp_bpf.c                            |  72 ++++++++--
 .../selftests/bpf/prog_tests/sockmap_basic.c  | 136 ++++++++++++++++++
 2 files changed, 199 insertions(+), 9 deletions(-)
Signature · 1 line
2.52.0

Hi all, just wondering if you need anything more from me on this?

maintainer Reviewed-by
#7 lore raw
Quoted 61 lines · Nnamdi Onyeyiri
Quoted 60 lines · Nnamdi Onyeyiri

Spurious wakeups in tcp_msg_wait_data() aren't being handled by tcp_bpf_recvmsg() and tcp_bpf_recvmsg_parser(), leading to unexpected EAGAIN errors returned by recvfrom()/recv(). Adding handling for the wakeup and a selftest.

This issue was first discovered in an application that adds sockets to a sockmap as a way to view the received data. No redirects or any other operations are performed.

Sashiko has noted a few other pre-existing issues in the same area
(https://patch.msgid.link/[email protected])
that can lead to EAGAIN in tcp_bpf_recvmsg() and tcp_bpf_recvmsg_parser(). A local run of Sashiko also identified a potential issue in tcp_bpf_recvmsg() handling zero-data FIN packets when the MSG_PEEK flag is set. To prevent this patchset from growing too large, I intend to submit follow up patches to address these once this one has been accepted.

Changes in v7:
- In tcp_bpf_recvmsg check for data in the TCP_CLOSE, !timeo and signal_pending branches.
- Refactored changes to tcp_bpf_recvmsg to reduce duplication.
- Increase the number of iterations in the selftests.
- Documented selftests numeric constants.
- Link to v6: https://patch.msgid.link/[email protected]

Changes in v6:
- Closing the file descriptor in the selftest worker thread.
- Ensuring the selftest loop breaks early for an error in the worker thread.
- Added comments to selftest regarding focus on EAGAIN error.
- Updated selftest commit message to imperative mood.
- Link to v5: https://patch.msgid.link/[email protected]

Changes in v5:
- Move selftest from net into bpf/prog_tests/sockmap_basic.c
- Link to v4: https://patch.msgid.link/[email protected]

Changes in v4:
- Fix potential data loss in tcp_bpf_recvmsg() when a FIN or RST has been received.
- Check the return code of pthread_create() in the selftest.
- Fix race caused by using EXPECT macros in a thread in selftest.
- Link to v3: https://patch.msgid.link/[email protected]

Changes in v3:
- Added the sockmap_recvfrom selftest.
- Link to v2: https://patch.msgid.link/[email protected]

Changes in v2:
- In tcp_bpf_recvmsg, handle signals and the socket closing in the loop.
- Fix spurious wakeups when SO_RCVTIMEO has been set on the socket.
- Link to v1: https://patch.msgid.link/[email protected]

---
Nnamdi Onyeyiri (2):
bpf, sockmap: handle spurious tcp_msg_wait_data() wakeup selftests/bpf: add sockmap recvfrom EAGAIN selftest

Quoted 8 lines
Quoted 7 lines
  net/ipv4/tcp_bpf.c                            |  72 ++++++++--
  .../selftests/bpf/prog_tests/sockmap_basic.c  | 136 ++++++++++++++++++
  2 files changed, 199 insertions(+), 9 deletions(-)
Signature · 1 line
2.52.0

Hi all, just wondering if you need anything more from me on this?

targeting bpf-next without Fixes tags seems fine given the scope