Skip to contentSkip to conversation

bpf,ktls: Fix data corruption caused by using bpf_msg_pop_data() in ktls

PATCH Applied Jiayuan Chen 4 participants 6 messages Jun 9, 2025 – Jun 11, 2025

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

v2 cover bpf,ktls: Fix data corruption caused by using bpf_msg_pop_data() in ktls

Cong reported an issue where running 'test_sockmap' in the current bpf-next tree results in an error [1].

The specific test case that triggered the error is a combined test involving ktls and bpf_msg_pop_data().

Root Cause:
When sending plaintext data, we initially calculated the corresponding ciphertext length. However, if we later reduced the plaintext data length via socket policy, we failed to recalculate the ciphertext length.

This results in transmitting buffers containing uninitialized data during ciphertext transmission.

This causes uninitialized bytes to be appended after a complete "Application Data" packet, leading to errors on the receiving end when parsing TLS record.

This issue has existed for a long time but was only exposed after the following test code was merged.
commit 47eae080410b ("selftests/bpf: Add more tests for test_txmsg_push_pop in test_sockmap")

Although we already had tests for pop data before this commit, the pop data length was insufficient (less than 5 bytes). This meant that the corrupted TLS records with data length <5 bytes were cached without being parsed, resulting in no error being triggered.

After this fix, all tests pass.

 1/ 6  sockmap::txmsg test passthrough:OK
 2/ 6  sockmap::txmsg test redirect:OK
 3/ 2  sockmap::txmsg test redirect wait send mem:OK
 4/ 6  sockmap::txmsg test drop:OK
 5/ 6  sockmap::txmsg test ingress redirect:OK
 6/ 7  sockmap::txmsg test skb:OK
 7/12  sockmap::txmsg test apply:OK
 8/12  sockmap::txmsg test cork:OK
 9/ 3  sockmap::txmsg test hanging corks:OK
10/11  sockmap::txmsg test push_data:OK
11/17  sockmap::txmsg test pull-data:OK
12/ 9  sockmap::txmsg test pop-data:OK
13/ 6  sockmap::txmsg test push/pop data:OK
14/ 1  sockmap::txmsg test ingress parser:OK
15/ 1  sockmap::txmsg test ingress parser2:OK

16/ 6 sockhash::txmsg test passthrough:OK
17/ 6 sockhash::txmsg test redirect:OK
18/ 2 sockhash::txmsg test redirect wait send mem:OK 19/ 6 sockhash::txmsg test drop:OK
20/ 6 sockhash::txmsg test ingress redirect:OK 21/ 7 sockhash::txmsg test skb:OK
22/12 sockhash::txmsg test apply:OK
23/12 sockhash::txmsg test cork:OK
24/ 3 sockhash::txmsg test hanging corks:OK 25/11 sockhash::txmsg test push_data:OK
26/17 sockhash::txmsg test pull-data:OK
27/ 9 sockhash::txmsg test pop-data:OK
28/ 6 sockhash::txmsg test push/pop data:OK
29/ 1 sockhash::txmsg test ingress parser:OK 30/ 1 sockhash::txmsg test ingress parser2:OK 31/ 6 sockhash:ktls:txmsg test passthrough:OK 32/ 6 sockhash:ktls:txmsg test redirect:OK
33/ 2 sockhash:ktls:txmsg test redirect wait send mem:OK 34/ 6 sockhash:ktls:txmsg test drop:OK
35/ 6 sockhash:ktls:txmsg test ingress redirect:OK 36/ 7 sockhash:ktls:txmsg test skb:OK
37/12 sockhash:ktls:txmsg test apply:OK
38/12 sockhash:ktls:txmsg test cork:OK
39/ 3 sockhash:ktls:txmsg test hanging corks:OK 40/11 sockhash:ktls:txmsg test push_data:OK 41/17 sockhash:ktls:txmsg test pull-data:OK
42/ 9 sockhash:ktls:txmsg test pop-data:OK
43/ 6 sockhash:ktls:txmsg test push/pop data:OK 44/ 1 sockhash:ktls:txmsg test ingress parser:OK 45/ 0 sockhash:ktls:txmsg test ingress parser2:OK
Pass: 45 Fail: 0

[1]: https://lore.kernel.org/bpf/CAM_iQpU7=4xjbefZoxndKoX9gFFMOe7FcWMq5tHBsymbrnMHxQ@mail.gmail.com/
---
v2 -> v1: Removed WARN_ON() check and added Reviewed-by tag.
https://lore.kernel.org/bpf/[email protected]/

Jiayuan Chen (2):
bpf,ktls: Fix data corruption when using bpf_msg_pop_data() in ktls selftests/bpf: Add test to cover ktls with bpf_msg_pop_data

 net/tls/tls_sw.c                              | 13 +++
 .../selftests/bpf/prog_tests/sockmap_ktls.c   | 91 +++++++++++++++++++
 .../selftests/bpf/progs/test_sockmap_ktls.c   |  4 +
 3 files changed, 108 insertions(+)
Signature · 1 line
2.47.1
#1 lore raw

PATCH v2 1/2 bpf,ktls: Fix data corruption when using bpf_msg_pop_data() in ktls

When sending plaintext data, we initially calculated the corresponding ciphertext length. However, if we later reduced the plaintext data length via socket policy, we failed to recalculate the ciphertext length.

This results in transmitting buffers containing uninitialized data during ciphertext transmission.

This causes uninitialized bytes to be appended after a complete "Application Data" packet, leading to errors on the receiving end when parsing TLS record.

Fixes d3b18ad31f93 ("tls: add bpf support to sk_msg handling")
---
 net/tls/tls_sw.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
net/tls/tls_sw.c +13 -0
index fc88e34b7f33..549d1ea01a72 100644
@@ -872,6 +872,19 @@ static int bpf_exec_tx_verdict(struct sk_msg *msg, struct sock *sk,
 		delta = msg->sg.size;
 		psock->eval = sk_psock_msg_verdict(sk, psock, msg);
 		delta -= msg->sg.size;
+
+		if ((s32)delta > 0) {
+			/* It indicates that we executed bpf_msg_pop_data(),
+			 * causing the plaintext data size to decrease.
+			 * Therefore the encrypted data size also needs to
+			 * correspondingly decrease. We only need to subtract
+			 * delta to calculate the new ciphertext length since
+			 * ktls does not support block encryption.
+			 */
+			struct sk_msg *enc = &ctx->open_rec->msg_encrypted;
+
+			sk_msg_trim(sk, enc, enc->sg.size - delta);
+		}
 	}
 	if (msg->cork_bytes && msg->cork_bytes > msg->sg.size &&
 	    !enospc && !full_record) {
Signature · 1 line
2.47.1
#3 lore raw

PATCH v2 2/2 selftests/bpf: Add test to cover ktls with bpf_msg_pop_data

In reply to Jiayuan Chen

The selftest can reproduce an issue where using bpf_msg_pop_data() in ktls causes errors on the receiving end.

---
 .../selftests/bpf/prog_tests/sockmap_ktls.c   | 91 +++++++++++++++++++
 .../selftests/bpf/progs/test_sockmap_ktls.c   |  4 +
 2 files changed, 95 insertions(+)
tools/testing/selftests/bpf/prog_tests/sockmap_ktls.c +91 -0
index b6c471da5c28..b87e7f39e15a 100644
@@ -314,6 +314,95 @@ static void test_sockmap_ktls_tx_no_buf(int family, int sotype, bool push)
 	test_sockmap_ktls__destroy(skel);
 }
 
+static void test_sockmap_ktls_tx_pop(int family, int sotype)
+{
+	char msg[37] = "0123456789abcdefghijklmnopqrstuvwxyz\0";
+	int c = 0, p = 0, one = 1, sent, recvd;
+	struct test_sockmap_ktls *skel;
+	int prog_fd, map_fd;
+	char rcv[50] = {0};
+	int err;
+	int i, m, r;
+
+	skel = test_sockmap_ktls__open_and_load();
+	if (!ASSERT_TRUE(skel, "open ktls skel"))
+		return;
+
+	err = create_pair(family, sotype, &c, &p);
+	if (!ASSERT_OK(err, "create_pair()"))
+		goto out;
+
+	prog_fd = bpf_program__fd(skel->progs.prog_sk_policy);
+	map_fd = bpf_map__fd(skel->maps.sock_map);
+
+	err = bpf_prog_attach(prog_fd, map_fd, BPF_SK_MSG_VERDICT, 0);
+	if (!ASSERT_OK(err, "bpf_prog_attach sk msg"))
+		goto out;
+
+	err = bpf_map_update_elem(map_fd, &one, &c, BPF_NOEXIST);
+	if (!ASSERT_OK(err, "bpf_map_update_elem(c)"))
+		goto out;
+
+	err = init_ktls_pairs(c, p);
+	if (!ASSERT_OK(err, "init_ktls_pairs(c, p)"))
+		goto out;
+
+	struct {
+		int	pop_start;
+		int	pop_len;
+	} pop_policy[] = {
+		/* trim the start */
+		{0, 2},
+		{0, 10},
+		{1, 2},
+		{1, 10},
+		/* trim the end */
+		{35, 2},
+		/* New entries should be added before this line */
+		{-1, -1},
+	};
+
+	i = 0;
+	while (pop_policy[i].pop_start >= 0) {
+		skel->bss->pop_start = pop_policy[i].pop_start;
+		skel->bss->pop_end =  pop_policy[i].pop_len;
+
+		sent = send(c, msg, sizeof(msg), 0);
+		if (!ASSERT_EQ(sent, sizeof(msg), "send(msg)"))
+			goto out;
+
+		recvd = recv_timeout(p, rcv, sizeof(rcv), MSG_DONTWAIT, 1);
+		if (!ASSERT_EQ(recvd, sizeof(msg) - pop_policy[i].pop_len, "pop len mismatch"))
+			goto out;
+
+		/* verify the data
+		 * msg: 0123456789a bcdefghij klmnopqrstuvwxyz
+		 *                  |       |
+		 *                  popped data
+		 */
+		for (m = 0, r = 0; m < sizeof(msg);) {
+			/* skip checking the data that has been popped */
+			if (m >= pop_policy[i].pop_start &&
+			    m <= pop_policy[i].pop_start + pop_policy[i].pop_len - 1) {
+				m++;
+				continue;
+			}
+
+			if (!ASSERT_EQ(msg[m], rcv[r], "data mismatch"))
+				goto out;
+			m++;
+			r++;
+		}
+		i++;
+	}
+out:
+	if (c)
+		close(c);
+	if (p)
+		close(p);
+	test_sockmap_ktls__destroy(skel);
+}
+
 static void run_tests(int family, enum bpf_map_type map_type)
 {
 	int map;
@@ -338,6 +427,8 @@ static void run_ktls_test(int family, int sotype)
 		test_sockmap_ktls_tx_cork(family, sotype, true);
 	if (test__start_subtest("tls tx egress with no buf"))
 		test_sockmap_ktls_tx_no_buf(family, sotype, true);
+	if (test__start_subtest("tls tx with pop"))
+		test_sockmap_ktls_tx_pop(family, sotype);
 }
 
 void test_sockmap_ktls(void)
tools/testing/selftests/bpf/progs/test_sockmap_ktls.c +4 -0
index 8bdb9987c0c7..83df4919c224 100644
@@ -7,6 +7,8 @@ int cork_byte;
 int push_start;
 int push_end;
 int apply_bytes;
+int pop_start;
+int pop_end;
 
 struct {
 	__uint(type, BPF_MAP_TYPE_SOCKMAP);
@@ -22,6 +24,8 @@ int prog_sk_policy(struct sk_msg_md *msg)
 		bpf_msg_cork_bytes(msg, cork_byte);
 	if (push_start > 0 && push_end > 0)
 		bpf_msg_push_data(msg, push_start, push_end, 0);
+	if (pop_start >= 0 && pop_end > 0)
+		bpf_msg_pop_data(msg, pop_start, pop_end, 0);
 
 	return SK_PASS;
 }
Signature · 1 line
2.47.1
maintainer Acked-by on v2 1/2
#2 lore raw

Re: [PATCH bpf-next v2 1/2] bpf,ktls: Fix data corruption when using bpf_msg_pop_data() in ktls

In reply to Jiayuan Chen
Quoted 10 lines · On Mon, 9 Jun 2025 10:08:52 +0800 Jiayuan Chen wrote:

When sending plaintext data, we initially calculated the corresponding ciphertext length. However, if we later reduced the plaintext data length via socket policy, we failed to recalculate the ciphertext length.

This results in transmitting buffers containing uninitialized data during ciphertext transmission.

This causes uninitialized bytes to be appended after a complete "Application Data" packet, leading to errors on the receiving end when parsing TLS record.

maintainer Reviewed-by on v2 2/2
#4 lore raw

Re: [PATCH bpf-next v2 2/2] selftests/bpf: Add test to cover ktls with bpf_msg_pop_data

In reply to Jiayuan Chen
Quoted 5 lines · Jiayuan Chen

The selftest can reproduce an issue where using bpf_msg_pop_data() in ktls causes errors on the receiving end.

---

Quoted 58 lines
 .../selftests/bpf/prog_tests/sockmap_ktls.c   | 91 +++++++++++++++++++
 .../selftests/bpf/progs/test_sockmap_ktls.c   |  4 +
 2 files changed, 95 insertions(+)
tools/testing/selftests/bpf/prog_tests/sockmap_ktls.c +46 -0
index b6c471da5c28..b87e7f39e15a 100644
@@ -314,6 +314,95 @@ static void test_sockmap_ktls_tx_no_buf(int family, int sotype, bool push)
 	test_sockmap_ktls__destroy(skel);
 }
 
+static void test_sockmap_ktls_tx_pop(int family, int sotype)
+{
+	char msg[37] = "0123456789abcdefghijklmnopqrstuvwxyz\0";
+	int c = 0, p = 0, one = 1, sent, recvd;
+	struct test_sockmap_ktls *skel;
+	int prog_fd, map_fd;
+	char rcv[50] = {0};
+	int err;
+	int i, m, r;
+
+	skel = test_sockmap_ktls__open_and_load();
+	if (!ASSERT_TRUE(skel, "open ktls skel"))
+		return;
+
+	err = create_pair(family, sotype, &c, &p);
+	if (!ASSERT_OK(err, "create_pair()"))
+		goto out;
+
+	prog_fd = bpf_program__fd(skel->progs.prog_sk_policy);
+	map_fd = bpf_map__fd(skel->maps.sock_map);
+
+	err = bpf_prog_attach(prog_fd, map_fd, BPF_SK_MSG_VERDICT, 0);
+	if (!ASSERT_OK(err, "bpf_prog_attach sk msg"))
+		goto out;
+
+	err = bpf_map_update_elem(map_fd, &one, &c, BPF_NOEXIST);
+	if (!ASSERT_OK(err, "bpf_map_update_elem(c)"))
+		goto out;
+
+	err = init_ktls_pairs(c, p);
+	if (!ASSERT_OK(err, "init_ktls_pairs(c, p)"))
+		goto out;
+
+	struct {
+		int	pop_start;
+		int	pop_len;
+	} pop_policy[] = {
+		/* trim the start */
+		{0, 2},
+		{0, 10},
+		{1, 2},
+		{1, 10},
+		/* trim the end */
+		{35, 2},
+		/* New entries should be added before this line */
+		{-1, -1},

tools/testing/selftests/bpf/prog_tests/sockmap_ktls.c static void test_sockmap_ktls_tx_no_buf(int family, int sotype, bool push) · line 372

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/sockmap_ktls.c +10 -0
+	};
+
+	i = 0;
+	while (pop_policy[i].pop_start >= 0) {
+		skel->bss->pop_start = pop_policy[i].pop_start;
+		skel->bss->pop_end =  pop_policy[i].pop_len;
+
+		sent = send(c, msg, sizeof(msg), 0);
+		if (!ASSERT_EQ(sent, sizeof(msg), "send(msg)"))
+			goto out;

Its possible this could actually not send 38B (sent < 38), but then again it is only 38B so I guess it should never fail? Anyways we have this case in a few places already I think and its not tripping CI so lets go for it.

Thanks,
John

Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/sockmap_ktls.c +4 -0
+
+		recvd = recv_timeout(p, rcv, sizeof(rcv), MSG_DONTWAIT, 1);
+		if (!ASSERT_EQ(recvd, sizeof(msg) - pop_policy[i].pop_len, "pop len mismatch"))
+			goto out;
#5 lore raw
In reply to Jiayuan Chen
This series was applied to bpf/bpf-next.git (net) by Daniel Borkmann <[email protected]>:

Hello:

This series was applied to bpf/bpf-next.git (net)
by Daniel Borkmann <[email protected]>:

Quoted 6 lines · On Mon, 9 Jun 2025 10:08:51 +0800 you wrote:

Cong reported an issue where running 'test_sockmap' in the current bpf-next tree results in an error [1].

The specific test case that triggered the error is a combined test involving ktls and bpf_msg_pop_data().

Quoted 6 lines

Root Cause:
When sending plaintext data, we initially calculated the corresponding ciphertext length. However, if we later reduced the plaintext data length via socket policy, we failed to recalculate the ciphertext length.

[...]

Here is the summary with links:
- [bpf-next,v2,1/2] bpf,ktls: Fix data corruption when using bpf_msg_pop_data() in ktls

    https://git.kernel.org/bpf/bpf-next/c/178f6a5c8cb3
  - [bpf-next,v2,2/2] selftests/bpf: Add test to cover ktls with bpf_msg_pop_data
    https://git.kernel.org/bpf/bpf-next/c/f1c025773f25

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html