Skip to contentSkip to conversation

: A non NULL req->sk in tcp_rtx_synack. Not a fastopen connection.

Question Martin KaFai Lau 2 participants 4 messages Oct 4, 2024

Subsystems (4)
Networking Network drivers BPF TCP
lore
Share & tools
lore

Martin KaFai Lau

author reviewer
#0 lore raw

Hi,

We are seeing a use-after-free from a bpf prog attached to trace_tcp_retransmit_synack. The program passes the req->sk to the bpf_sk_storage_get_tracing kernel helper which does check for null before using it.

fastopen is not used.

We got a kfence report on use-after-free (pasted at the end). It is running with an older 6.4 kernel and we hardly hit this in production.

From the upstream code, del_timer_sync() should have been done by inet_csk_reqsk_queue_drop() before "req->sk = child;" is assigned in inet_csk_reqsk_queue_add(). My understanding is the req->rsk_timer should have been stopped before the "req->sk = child;" assignment.

or there are cases that req->sk is not NULL in the reqsk_timer_handler()?

BUG: KFENCE: use-after-free read in bpf_sk_storage_get_tracing+0x2e/0x1b0

Use-after-free read at 0x00000000a891fb3a (in kfence-#1):

bpf_sk_storage_get_tracing+0x2e/0x1b0
bpf_prog_5ea3e95db6da0438_tcp_retransmit_synack+0x1d20/0x1dda
bpf_trace_run2+0x4c/0xc0
tcp_rtx_synack+0xf9/0x100
reqsk_timer_handler+0xda/0x3d0
run_timer_softirq+0x292/0x8a0
irq_exit_rcu+0xf5/0x320
sysvec_apic_timer_interrupt+0x6d/0x80
asm_sysvec_apic_timer_interrupt+0x16/0x20
intel_idle_irq+0x5a/0xa0
cpuidle_enter_state+0x94/0x273
cpu_startup_entry+0x15e/0x260
start_secondary+0x8a/0x90
secondary_startup_64_no_verify+0xfa/0xfb

kfence-#1: 0x00000000a72cc7b6-0x00000000d97616d9, size=2376, cache=TCPv6

allocated by task 0 on cpu 9 at 260507.901592s:

sk_prot_alloc+0x35/0x140
sk_clone_lock+0x1f/0x3f0
inet_csk_clone_lock+0x15/0x160
tcp_create_openreq_child+0x1f/0x410
tcp_v6_syn_recv_sock+0x1da/0x700
tcp_check_req+0x1fb/0x510
tcp_v6_rcv+0x98b/0x1420
ipv6_list_rcv+0x2258/0x26e0
napi_complete_done+0x5b1/0x2990
mlx5e_napi_poll+0x2ae/0x8d0
net_rx_action+0x13e/0x590
irq_exit_rcu+0xf5/0x320
common_interrupt+0x80/0x90
asm_common_interrupt+0x22/0x40
cpuidle_enter_state+0xfb/0x273
cpu_startup_entry+0x15e/0x260
start_secondary+0x8a/0x90
secondary_startup_64_no_verify+0xfa/0xfb

freed by task 0 on cpu 9 at 260507.927527s:

rcu_core_si+0x4ff/0xf10
irq_exit_rcu+0xf5/0x320
sysvec_apic_timer_interrupt+0x6d/0x80
asm_sysvec_apic_timer_interrupt+0x16/0x20
cpuidle_enter_state+0xfb/0x273
cpu_startup_entry+0x15e/0x260
start_secondary+0x8a/0x90
secondary_startup_64_no_verify+0xfa/0xfb

Thanks,
Martin

#1 lore raw

From: Martin KaFai Lau <[email protected]>
Date: Thu, 3 Oct 2024 18:14:09 -0700

Quoted 8 lines

Hi,

We are seeing a use-after-free from a bpf prog attached to trace_tcp_retransmit_synack. The program passes the req->sk to the bpf_sk_storage_get_tracing kernel helper which does check for null before using it.

fastopen is not used.

Quoted 7 lines

We got a kfence report on use-after-free (pasted at the end). It is running with an older 6.4 kernel and we hardly hit this in production.

From the upstream code, del_timer_sync() should have been done by
inet_csk_reqsk_queue_drop() before "req->sk = child;" is assigned in inet_csk_reqsk_queue_add(). My understanding is the req->rsk_timer should have been stopped before the "req->sk = child;" assignment.

There seems to be a small race window in reqsk_queue_unlink().

expire_timers() first calls detach_timer(, true), which marks the timer as not pending, and then calls reqsk_timer_handler().

If reqsk_queue_unlink() calls timer_pending() just before expire_timers() calls reqsk_timer_handler(), reqsk_queue_unlink() could miss del_timer_sync() ?

---8<---

net/ipv4/inet_connection_sock.c +1 -1
index 2c5632d4fddb..4ba47ee6c9da 100644
@@ -1045,7 +1045,7 @@ static bool reqsk_queue_unlink(struct request_sock *req)
 		found = __sk_nulls_del_node_init_rcu(sk);
 		spin_unlock(lock);
 	}
-	if (timer_pending(&req->rsk_timer) && del_timer_sync(&req->rsk_timer))
+	if (del_timer_sync(&req->rsk_timer))
 		reqsk_put(req);
 	return found;
 }

---8<---

Quoted 55 lines

or there are cases that req->sk is not NULL in the reqsk_timer_handler()?

BUG: KFENCE: use-after-free read in bpf_sk_storage_get_tracing+0x2e/0x1b0

Use-after-free read at 0x00000000a891fb3a (in kfence-#1):

bpf_sk_storage_get_tracing+0x2e/0x1b0
bpf_prog_5ea3e95db6da0438_tcp_retransmit_synack+0x1d20/0x1dda
bpf_trace_run2+0x4c/0xc0
tcp_rtx_synack+0xf9/0x100
reqsk_timer_handler+0xda/0x3d0
run_timer_softirq+0x292/0x8a0
irq_exit_rcu+0xf5/0x320
sysvec_apic_timer_interrupt+0x6d/0x80
asm_sysvec_apic_timer_interrupt+0x16/0x20
intel_idle_irq+0x5a/0xa0
cpuidle_enter_state+0x94/0x273
cpu_startup_entry+0x15e/0x260
start_secondary+0x8a/0x90
secondary_startup_64_no_verify+0xfa/0xfb

kfence-#1: 0x00000000a72cc7b6-0x00000000d97616d9, size=2376, cache=TCPv6

allocated by task 0 on cpu 9 at 260507.901592s:

sk_prot_alloc+0x35/0x140
sk_clone_lock+0x1f/0x3f0
inet_csk_clone_lock+0x15/0x160
tcp_create_openreq_child+0x1f/0x410
tcp_v6_syn_recv_sock+0x1da/0x700
tcp_check_req+0x1fb/0x510
tcp_v6_rcv+0x98b/0x1420
ipv6_list_rcv+0x2258/0x26e0
napi_complete_done+0x5b1/0x2990
mlx5e_napi_poll+0x2ae/0x8d0
net_rx_action+0x13e/0x590
irq_exit_rcu+0xf5/0x320
common_interrupt+0x80/0x90
asm_common_interrupt+0x22/0x40
cpuidle_enter_state+0xfb/0x273
cpu_startup_entry+0x15e/0x260
start_secondary+0x8a/0x90
secondary_startup_64_no_verify+0xfa/0xfb

freed by task 0 on cpu 9 at 260507.927527s:

rcu_core_si+0x4ff/0xf10
irq_exit_rcu+0xf5/0x320
sysvec_apic_timer_interrupt+0x6d/0x80
asm_sysvec_apic_timer_interrupt+0x16/0x20
cpuidle_enter_state+0xfb/0x273
cpu_startup_entry+0x15e/0x260
start_secondary+0x8a/0x90
secondary_startup_64_no_verify+0xfa/0xfb

Thanks,
Martin

author reviewer
#2 lore raw
Quoted 18 lines · Kuniyuki Iwashima

From: Martin KaFai Lau <[email protected]>
Date: Thu, 3 Oct 2024 18:14:09 -0700

Quoted 8 lines

Hi,

We are seeing a use-after-free from a bpf prog attached to trace_tcp_retransmit_synack. The program passes the req->sk to the bpf_sk_storage_get_tracing kernel helper which does check for null before using it.

fastopen is not used.

Quoted 7 lines

We got a kfence report on use-after-free (pasted at the end). It is running with an older 6.4 kernel and we hardly hit this in production.

From the upstream code, del_timer_sync() should have been done by
inet_csk_reqsk_queue_drop() before "req->sk = child;" is assigned in inet_csk_reqsk_queue_add(). My understanding is the req->rsk_timer should have been stopped before the "req->sk = child;" assignment.

Quoted 8 lines

There seems to be a small race window in reqsk_queue_unlink().

expire_timers() first calls detach_timer(, true), which marks the timer as not pending, and then calls reqsk_timer_handler().

If reqsk_queue_unlink() calls timer_pending() just before expire_timers() calls reqsk_timer_handler(), reqsk_queue_unlink() could miss del_timer_sync() ?

This seems to explain it. :)

Does it mean there is a chance that the reqsk_timer_handler() may rearm the timer again and I guess only a few more synack will be sent in this case and should be no harm?

net/ipv4/inet_connection_sock.c static bool reqsk_queue_unlink(struct request_sock *req) · line 1048

Quoted 12 lines

---8<---

net/ipv4/inet_connection_sock.c +1 -1
index 2c5632d4fddb..4ba47ee6c9da 100644
@@ -1045,7 +1045,7 @@ static bool reqsk_queue_unlink(struct request_sock *req)
  		found = __sk_nulls_del_node_init_rcu(sk);
  		spin_unlock(lock);
  	}
-	if (timer_pending(&req->rsk_timer) && del_timer_sync(&req->rsk_timer))
+	if (del_timer_sync(&req->rsk_timer))

It seems the reqsk_timer_handler() will also call reqsk_queue_unlink() through inet_csk_reqsk_queue_drop_and_put(). Not sure if the reqsk_timer_handler() can del_timer_sync() itself.

Quoted 61 lines
net/ipv4/inet_connection_sock.c +0 -1
  		reqsk_put(req);
  	return found;
  }
---8<---
Quoted 57 lines

or there are cases that req->sk is not NULL in the reqsk_timer_handler()?

BUG: KFENCE: use-after-free read in bpf_sk_storage_get_tracing+0x2e/0x1b0

Use-after-free read at 0x00000000a891fb3a (in kfence-#1):

bpf_sk_storage_get_tracing+0x2e/0x1b0
bpf_prog_5ea3e95db6da0438_tcp_retransmit_synack+0x1d20/0x1dda
bpf_trace_run2+0x4c/0xc0
tcp_rtx_synack+0xf9/0x100
reqsk_timer_handler+0xda/0x3d0
run_timer_softirq+0x292/0x8a0
irq_exit_rcu+0xf5/0x320
sysvec_apic_timer_interrupt+0x6d/0x80
asm_sysvec_apic_timer_interrupt+0x16/0x20
intel_idle_irq+0x5a/0xa0
cpuidle_enter_state+0x94/0x273
cpu_startup_entry+0x15e/0x260
start_secondary+0x8a/0x90
secondary_startup_64_no_verify+0xfa/0xfb

kfence-#1: 0x00000000a72cc7b6-0x00000000d97616d9, size=2376, cache=TCPv6

allocated by task 0 on cpu 9 at 260507.901592s:

sk_prot_alloc+0x35/0x140
sk_clone_lock+0x1f/0x3f0
inet_csk_clone_lock+0x15/0x160
tcp_create_openreq_child+0x1f/0x410
tcp_v6_syn_recv_sock+0x1da/0x700
tcp_check_req+0x1fb/0x510
tcp_v6_rcv+0x98b/0x1420
ipv6_list_rcv+0x2258/0x26e0
napi_complete_done+0x5b1/0x2990
mlx5e_napi_poll+0x2ae/0x8d0
net_rx_action+0x13e/0x590
irq_exit_rcu+0xf5/0x320
common_interrupt+0x80/0x90
asm_common_interrupt+0x22/0x40
cpuidle_enter_state+0xfb/0x273
cpu_startup_entry+0x15e/0x260
start_secondary+0x8a/0x90
secondary_startup_64_no_verify+0xfa/0xfb

freed by task 0 on cpu 9 at 260507.927527s:

rcu_core_si+0x4ff/0xf10
irq_exit_rcu+0xf5/0x320
sysvec_apic_timer_interrupt+0x6d/0x80
asm_sysvec_apic_timer_interrupt+0x16/0x20
cpuidle_enter_state+0xfb/0x273
cpu_startup_entry+0x15e/0x260
start_secondary+0x8a/0x90
secondary_startup_64_no_verify+0xfa/0xfb

Thanks,
Martin

#3 lore raw

From: Martin KaFai Lau <[email protected]>
Date: Thu, 3 Oct 2024 21:00:20 -0700

Quoted 28 lines · Kuniyuki Iwashima
Quoted 18 lines

From: Martin KaFai Lau <[email protected]>
Date: Thu, 3 Oct 2024 18:14:09 -0700

Quoted 8 lines

Hi,

We are seeing a use-after-free from a bpf prog attached to trace_tcp_retransmit_synack. The program passes the req->sk to the bpf_sk_storage_get_tracing kernel helper which does check for null before using it.

fastopen is not used.

Quoted 7 lines

We got a kfence report on use-after-free (pasted at the end). It is running with an older 6.4 kernel and we hardly hit this in production.

From the upstream code, del_timer_sync() should have been done by
inet_csk_reqsk_queue_drop() before "req->sk = child;" is assigned in inet_csk_reqsk_queue_add(). My understanding is the req->rsk_timer should have been stopped before the "req->sk = child;" assignment.

Quoted 8 lines

There seems to be a small race window in reqsk_queue_unlink().

expire_timers() first calls detach_timer(, true), which marks the timer as not pending, and then calls reqsk_timer_handler().

If reqsk_queue_unlink() calls timer_pending() just before expire_timers() calls reqsk_timer_handler(), reqsk_queue_unlink() could miss del_timer_sync() ?

Quoted 5 lines

This seems to explain it. :)

Does it mean there is a chance that the reqsk_timer_handler() may rearm the timer again and I guess only a few more synack will be sent in this case and should be no harm?

Ah, it seems possible. I was wondering how the timer can be delayed until sk is freed. In such a case, the timer will just let the peer generate some challenge ACKs.

Quoted 17 lines

net/ipv4/inet_connection_sock.c static bool reqsk_queue_unlink(struct request_sock *req) · line 1048

Quoted 13 lines

---8<---

net/ipv4/inet_connection_sock.c +1 -1
index 2c5632d4fddb..4ba47ee6c9da 100644
@@ -1045,7 +1045,7 @@ static bool reqsk_queue_unlink(struct request_sock *req)
  		found = __sk_nulls_del_node_init_rcu(sk);
  		spin_unlock(lock);
  	}
-	if (timer_pending(&req->rsk_timer) && del_timer_sync(&req->rsk_timer))
+	if (del_timer_sync(&req->rsk_timer))

It seems the reqsk_timer_handler() will also call reqsk_queue_unlink() through inet_csk_reqsk_queue_drop_and_put(). Not sure if the reqsk_timer_handler() can del_timer_sync() itself.

Exactly, it seems illegal to call it from the timer. Then, we need a variant of inet_csk_reqsk_queue_drop() to see if the caller is tiemr or not. (compile-test only)

---8<---

net/ipv4/inet_connection_sock.c +16 -5
index 2c5632d4fddb..2623964d8817 100644
@@ -1045,21 +1045,31 @@ static bool reqsk_queue_unlink(struct request_sock *req)
 		found = __sk_nulls_del_node_init_rcu(sk);
 		spin_unlock(lock);
 	}
-	if (timer_pending(&req->rsk_timer) && del_timer_sync(&req->rsk_timer))
-		reqsk_put(req);
+
 	return found;
 }
 
-bool inet_csk_reqsk_queue_drop(struct sock *sk, struct request_sock *req)
+static bool __inet_csk_reqsk_queue_drop(struct sock *sk,
+					struct request_sock *req,
+					bool from_timer)
 {
 	bool unlinked = reqsk_queue_unlink(req);
 
+	if (!from_timer && del_timer_sync(&req->rsk_timer))
+		reqsk_put(req);
+
 	if (unlinked) {
 		reqsk_queue_removed(&inet_csk(sk)->icsk_accept_queue, req);
 		reqsk_put(req);
 	}
+
 	return unlinked;
 }
+
+bool inet_csk_reqsk_queue_drop(struct sock *sk, struct request_sock *req)
+{
+	return __inet_csk_reqsk_queue_drop(sk, req, false);
+}
 EXPORT_SYMBOL(inet_csk_reqsk_queue_drop);
 
 void inet_csk_reqsk_queue_drop_and_put(struct sock *sk, struct request_sock *req)

@@ -1152,7 +1162,7 @@ static void reqsk_timer_handler(struct timer_list *t)
 
 		if (!inet_ehash_insert(req_to_sk(nreq), req_to_sk(oreq), NULL)) {
 			/* delete timer */
-			inet_csk_reqsk_queue_drop(sk_listener, nreq);
+			__inet_csk_reqsk_queue_drop(sk_listener, nreq, true);
 			goto no_ownership;
 		}
 

@@ -1178,7 +1188,8 @@ static void reqsk_timer_handler(struct timer_list *t)
 	}
 
 drop:
-	inet_csk_reqsk_queue_drop_and_put(oreq->rsk_listener, oreq);
+	__inet_csk_reqsk_queue_drop(sk_listener, nreq, true);
+	reqsk_put(req);
 }
 
 static bool reqsk_queue_hash_req(struct request_sock *req,

---8<---