Skip to contentSkip to conversation

unregistering tcp_ca struct_ops can cause kernel page fault

DISCUSSION [email protected] 2 participants 4 messages Dec 8, 2024 – Dec 14, 2024

Subsystems (3)
Networking Network drivers BPF
lore
Share & tools
lore

[email protected]

author
#0 lore raw

The attached program, along with the attached bpf_cubic.o eBPF binary, uses bpftool to install a tcp_ca struct ops, creates a tcp connection, and then unregisters the struct ops while the tcp connection is still active. On my 6.13.0 system this causes a kernel fault due to tcp_tso_segs() calling through a de-allocated struct tcp_congestion_ops.

bpf_cubic.o came from

https://github.com/aroodgar/bpf-tcp-congestion-control-algorithm

Linux ubuntu66 6.13.0-rc1-00337-g7503345ac5f5 #11 SMP Sun Dec 8 08:37:57 EST 2024 x86_64 x86_64 x86_64 GNU/Linux

Oops: general protection fault, probably for non-canonical address 0x6b6b6b6b6b6b6b6b: 0000 [#1] SMP DEBUG_PAGEALLOC PTI
CPU: 6 UID: 0 PID: 1594 Comm: a.out Not tainted 6.13.0-rc1-00337-g7503345ac5f5 #11
Hardware name: FreeBSD BHYVE/BHYVE, BIOS 14.0 10/17/2021
RIP: 0010:__x86_indirect_thunk_array+0xa/0x20
Code: 66 0f 1f 00 31 ff e9 15 70 d6 fe cc cc cc cc cc cc cc cc cc cc cc cc cc cc
 cc cc cc cc cc cc cc e8 01 00 00 00 cc 48 89 04 24 <c3> cc cc cc cc 90 66 66 2e
 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00
RSP: 0018:ffffc90002037c60 EFLAGS: 00010202
RAX: 6b6b6b6b6b6b6b6b RBX: ffff88810afc0b00 RCX: 0000000000000018
RDX: 00000000077b668a RSI: 0000000000008000 RDI: ffff88810afc0b00
RBP: 0000000000008000 R08: 0000000000000000 R09: 0000000000008000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000008000 R14: 0000000000000000 R15: ffffffff842f6140
FS:  00007f5457b84740(0000) GS:ffff88842db80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5457d8c710 CR3: 000000011b4b8005 CR4: 00000000003706f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <TASK>
 ? die_addr+0x31/0x80
 ? exc_general_protection+0x1b4/0x3c0
 ? asm_exc_general_protection+0x26/0x30
 ? __x86_indirect_thunk_array+0xa/0x20
 ? tcp_tso_segs+0x1c/0x90
 ? tcp_write_xmit+0x74/0x1840
 ? __mod_memcg_state+0x91/0x190
 ? __tcp_push_pending_frames+0x31/0xc0
 ? tcp_sendmsg_locked+0xafc/0xd10
 ? tcp_sendmsg+0x26/0x40
 ? sock_write_iter+0x167/0x1a0
 ? vfs_write+0x35d/0x400
 ? ksys_write+0xc6/0xe0
 ? do_syscall_64+0x3f/0xd0
 ? entry_SYSCALL_64_after_hwframe+0x76/0x7e
 </TASK>

Robert Morris
[email protected]

reviewer
#1 lore raw
Quoted 4 lines · On 12/8/24 6:59 AM, [email protected] wrote:

The attached program, along with the attached bpf_cubic.o eBPF binary, uses bpftool to install a tcp_ca struct ops, creates a tcp connection, and then unregisters the struct ops while the tcp connection is still active. On my 6.13.0 system this causes a kernel fault due to

I tried the latest bpf-next/master (6.13-r2 + some recent bpf patches). I cannot reproduce. I also don't recall any recent fix related to struct_ops refcnt in the bpf_struct_ops or bpf_tcp_ca.

I don't see how this could happen now because the icsk->icsk_ca_ops is refcnt-ed. e.g. the setsockopt(TCP_CONGESTION) in your attached tcpbpf12a.c. bpf_try_module_get, which then calls bpf_struct_ops_get for the bpf written tcp-cc, should have refcnt-ed the icsk_ca_ops. I traced on the kernel code path triggered by your tcpbpf12a.c and the refcnt looks correct.

If you can reproduce consistently, please help to debug and narrow further why the bpf_struct_ops_map_free() cleanup is finished while a tcp_sock still has a hold of it in icsk->icsk_ca_ops.

Quoted 48 lines

tcp_tso_segs() calling through a de-allocated struct tcp_congestion_ops.

bpf_cubic.o came from

https://github.com/aroodgar/bpf-tcp-congestion-control-algorithm

Linux ubuntu66 6.13.0-rc1-00337-g7503345ac5f5 #11 SMP Sun Dec 8 08:37:57 EST 2024 x86_64 x86_64 x86_64 GNU/Linux

Oops: general protection fault, probably for non-canonical address 0x6b6b6b6b6b6b6b6b: 0000 [#1] SMP DEBUG_PAGEALLOC PTI
CPU: 6 UID: 0 PID: 1594 Comm: a.out Not tainted 6.13.0-rc1-00337-g7503345ac5f5 #11
Hardware name: FreeBSD BHYVE/BHYVE, BIOS 14.0 10/17/2021
RIP: 0010:__x86_indirect_thunk_array+0xa/0x20
Code: 66 0f 1f 00 31 ff e9 15 70 d6 fe cc cc cc cc cc cc cc cc cc cc cc cc cc cc
  cc cc cc cc cc cc cc e8 01 00 00 00 cc 48 89 04 24 <c3> cc cc cc cc 90 66 66 2e
  0f 1f 84 00 00 00 00 00 0f 1f 44 00 00
RSP: 0018:ffffc90002037c60 EFLAGS: 00010202
RAX: 6b6b6b6b6b6b6b6b RBX: ffff88810afc0b00 RCX: 0000000000000018
RDX: 00000000077b668a RSI: 0000000000008000 RDI: ffff88810afc0b00
RBP: 0000000000008000 R08: 0000000000000000 R09: 0000000000008000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000008000 R14: 0000000000000000 R15: ffffffff842f6140
FS:  00007f5457b84740(0000) GS:ffff88842db80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5457d8c710 CR3: 000000011b4b8005 CR4: 00000000003706f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
  <TASK>
  ? die_addr+0x31/0x80
  ? exc_general_protection+0x1b4/0x3c0
  ? asm_exc_general_protection+0x26/0x30
  ? __x86_indirect_thunk_array+0xa/0x20
  ? tcp_tso_segs+0x1c/0x90
  ? tcp_write_xmit+0x74/0x1840
  ? __mod_memcg_state+0x91/0x190
  ? __tcp_push_pending_frames+0x31/0xc0
  ? tcp_sendmsg_locked+0xafc/0xd10
  ? tcp_sendmsg+0x26/0x40
  ? sock_write_iter+0x167/0x1a0
  ? vfs_write+0x35d/0x400
  ? ksys_write+0xc6/0xe0
  ? do_syscall_64+0x3f/0xd0
  ? entry_SYSCALL_64_after_hwframe+0x76/0x7e
  </TASK>

Robert Morris
[email protected]

author
#2 lore raw

Martin,

When I build from bpf-next/master with a default .config, I do not get the crash.

When I disable CONFIG_MODULES, I do get a crash from tcpbps12a.c.

I've attached my .config.

Robert

Quoted 73 lines

Date: Mon, 9 Dec 2024 13:56:21 -0800
Subject: Re: unregistering tcp_ca struct_ops can cause kernel page fault
To: [email protected]
From: Martin KaFai Lau <[email protected]>
Cc: [email protected], [email protected]
X-Migadu-Flow: FLOW_OUT

Quoted 4 lines · On 12/8/24 6:59 AM, [email protected] wrote:

The attached program, along with the attached bpf_cubic.o eBPF binary, uses bpftool to install a tcp_ca struct ops, creates a tcp connection, and then unregisters the struct ops while the tcp connection is still active. On my 6.13.0 system this causes a kernel fault due to

I tried the latest bpf-next/master (6.13-r2 + some recent bpf patches). I cannot reproduce. I also don't recall any recent fix related to struct_ops refcnt in the bpf_struct_ops or bpf_tcp_ca.

I don't see how this could happen now because the icsk->icsk_ca_ops is refcnt-ed. e.g. the setsockopt(TCP_CONGESTION) in your attached tcpbpf12a.c. bpf_try_module_get, which then calls bpf_struct_ops_get for the bpf written tcp-cc, should have refcnt-ed the icsk_ca_ops. I traced on the kernel code path triggered by your tcpbpf12a.c and the refcnt looks correct.

If you can reproduce consistently, please help to debug and narrow further why the bpf_struct_ops_map_free() cleanup is finished while a tcp_sock still has a hold of it in icsk->icsk_ca_ops.

Quoted 45 lines

tcp_tso_segs() calling through a de-allocated struct tcp_congestion_ops. bpf_cubic.o came from
https://github.com/aroodgar/bpf-tcp-congestion-control-algorithm
Linux ubuntu66 6.13.0-rc1-00337-g7503345ac5f5 #11 SMP Sun Dec 8 08:37:57 EST 2024 x86_64 x86_64 x86_64 GNU/Linux

Oops: general protection fault, probably for non-canonical address
0x6b6b6b6b6b6b6b6b: 0000 [#1] SMP DEBUG_PAGEALLOC PTI
CPU: 6 UID: 0 PID: 1594 Comm: a.out Not tainted 6.13.0-rc1-00337-g7503345ac5f5 #11
Hardware name: FreeBSD BHYVE/BHYVE, BIOS 14.0 10/17/2021
RIP: 0010:__x86_indirect_thunk_array+0xa/0x20
Code: 66 0f 1f 00 31 ff e9 15 70 d6 fe cc cc cc cc cc cc cc cc cc cc cc cc cc cc
  cc cc cc cc cc cc cc e8 01 00 00 00 cc 48 89 04 24 <c3> cc cc cc cc 90 66 66 2e
  0f 1f 84 00 00 00 00 00 0f 1f 44 00 00
RSP: 0018:ffffc90002037c60 EFLAGS: 00010202
RAX: 6b6b6b6b6b6b6b6b RBX: ffff88810afc0b00 RCX: 0000000000000018
RDX: 00000000077b668a RSI: 0000000000008000 RDI: ffff88810afc0b00
RBP: 0000000000008000 R08: 0000000000000000 R09: 0000000000008000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000008000 R14: 0000000000000000 R15: ffffffff842f6140
FS:  00007f5457b84740(0000) GS:ffff88842db80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5457d8c710 CR3: 000000011b4b8005 CR4: 00000000003706f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
  <TASK>
  ? die_addr+0x31/0x80
  ? exc_general_protection+0x1b4/0x3c0
  ? asm_exc_general_protection+0x26/0x30
  ? __x86_indirect_thunk_array+0xa/0x20
  ? tcp_tso_segs+0x1c/0x90
  ? tcp_write_xmit+0x74/0x1840
  ? __mod_memcg_state+0x91/0x190
  ? __tcp_push_pending_frames+0x31/0xc0
  ? tcp_sendmsg_locked+0xafc/0xd10
  ? tcp_sendmsg+0x26/0x40
  ? sock_write_iter+0x167/0x1a0
  ? vfs_write+0x35d/0x400
  ? ksys_write+0xc6/0xe0
  ? do_syscall_64+0x3f/0xd0
  ? entry_SYSCALL_64_after_hwframe+0x76/0x7e
  </TASK>

Robert Morris
[email protected]

reviewer
#3 lore raw
Quoted 6 lines · On 12/11/24 3:28 PM, [email protected] wrote:

Martin,

When I build from bpf-next/master with a default .config, I do not get the crash.

When I disable CONFIG_MODULES, I do get a crash from tcpbps12a.c.

During make:

"WARN: resolve_btfids: unresolved symbol module"

Without going into the details, the bpf_try_module_get failed to bump the refcnt because of missing the "struct module" btf_id.

With a quick thought, I see bpf_struct_ops should be able to work around this CONFIG_MODULES=n.

I don't think it should though. The bpf_tcp_ca is using the "struct tcp_congestion_ops" which can be implemented by a kernel module and the kconfig wants nothing other than the built-in tcp-cc. I don't think the bpf_struct_ops should be a way to work around that. I think the right thing to do here is to also disallow attaching bpf_struct_ops when CONFIG_MODULES=n to fix this UAF issue.