Skip to contentSkip to conversation

bpf: allow JIT debugging if CONFIG_BPF_JIT_ALWAYS_ON is set

RFC PATCH In review Ilya Leoshkevich 4 participants 7 messages Nov 6, 2019 – Nov 7, 2019

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 bpf: allow JIT debugging if CONFIG_BPF_JIT_ALWAYS_ON is set

Currently it's not possible to set bpf_jit_enable = 2 when CONFIG_BPF_JIT_ALWAYS_ON is set, which makes debugging certain problems harder.

It looks as if it's safe to allow this, because setting this knob requires root anyway, but I'm not sure about all the security implications, so sending this as an RFC.

---
 net/core/sysctl_net_core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/core/sysctl_net_core.c +1 -1
index eb29e5adc84d..09f1218b5656 100644
@@ -389,7 +389,7 @@ static struct ctl_table net_core_table[] = {
 		.proc_handler	= proc_dointvec_minmax_bpf_enable,
 # ifdef CONFIG_BPF_JIT_ALWAYS_ON
 		.extra1		= SYSCTL_ONE,
-		.extra2		= SYSCTL_ONE,
+		.extra2		= &two,
 # else
 		.extra1		= SYSCTL_ZERO,
 		.extra2		= &two,
Signature · 1 line
2.23.0
#1 lore raw
Quoted 4 lines · Ilya Leoshkevich

Currently it's not possible to set bpf_jit_enable = 2 when CONFIG_BPF_JIT_ALWAYS_ON is set, which makes debugging certain problems harder.

This is obsolete way of debugging.
Please use bpftool dump jited instead.

#2 lore raw
Quoted 10 lines

Am 06.11.2019 um 17:15 schrieb Alexei Starovoitov <[email protected]>:

Quoted 4 lines · Ilya Leoshkevich

Currently it's not possible to set bpf_jit_enable = 2 when CONFIG_BPF_JIT_ALWAYS_ON is set, which makes debugging certain problems harder.

This is obsolete way of debugging.
Please use bpftool dump jited instead.

Is there a way to integrate bpftool nicely with e.g. test_verifier? With bpf_jit_enable = 2, I can see JITed code for each test right away, without pausing it (via gdb or rebuilding with added sleep()) and running bpftool.

#3 lore raw
Quoted 8 lines · Ilya Leoshkevich
Quoted 8 lines

Am 06.11.2019 um 17:15 schrieb Alexei Starovoitov <[email protected]>:

Quoted 4 lines · Ilya Leoshkevich

Currently it's not possible to set bpf_jit_enable = 2 when CONFIG_BPF_JIT_ALWAYS_ON is set, which makes debugging certain problems harder.

Quoted 7 lines
Quoted 2 lines

This is obsolete way of debugging.
Please use bpftool dump jited instead.

Is there a way to integrate bpftool nicely with e.g. test_verifier? With bpf_jit_enable = 2, I can see JITed code for each test right away, without pausing it (via gdb or rebuilding with added sleep()) and running bpftool.

On the library side we can set the log_level causing the verifier logic steps to be printed. I guess adding it to bpftool might be nice. At least I would find it useful. I'll probably get to it sometime if its not already there somewhere and/or someone doesn't beat me to it.

maintainer
#4 lore raw
Quoted 16 lines · John Fastabend
Quoted 7 lines · Ilya Leoshkevich
Quoted 7 lines

Am 06.11.2019 um 17:15 schrieb Alexei Starovoitov <[email protected]>:

Quoted 4 lines · Ilya Leoshkevich

Currently it's not possible to set bpf_jit_enable = 2 when CONFIG_BPF_JIT_ALWAYS_ON is set, which makes debugging certain problems harder.

Quoted 7 lines
Quoted 2 lines

This is obsolete way of debugging.
Please use bpftool dump jited instead.

Is there a way to integrate bpftool nicely with e.g. test_verifier? With bpf_jit_enable = 2, I can see JITed code for each test right away, without pausing it (via gdb or rebuilding with added sleep()) and running bpftool.

Quoted 4 lines

On the library side we can set the log_level causing the verifier logic steps to be printed. I guess adding it to bpftool might be nice. At least I would find it useful. I'll probably get to it sometime if its not already there somewhere and/or someone doesn't beat me to it.

+1

Was wondering whether it may be worth it moving parts of the logic from bpftool into libbpf wrt jit dump as a higher level api, so it could be used directly for checking out the jit disasm + opcodes for specific tests given we have the fd there as well, but that might be too specific perhaps and would bring one more lib dependency to libbpf for a rather narrow use case. Adding sleep before prog fd close and/or shelling out to bpftool etc all is a crude temporary hack as well (currently using something long these lines locally). Would it make sense to dump some meta data and generated opcodes per test case to a file as opt-in e.g. ./test_verifier 711 --dump produces 711.opcodes out of bpf_obj_get_info_by_fd() which then bpftool could dump this artifact through its own disasm?

Thanks,
Daniel

#5 lore raw
Quoted 36 lines

Am 07.11.2019 um 00:07 schrieb Daniel Borkmann <[email protected]>:

Quoted 11 lines · John Fastabend
Quoted 10 lines · Ilya Leoshkevich
Quoted 9 lines

Am 06.11.2019 um 17:15 schrieb Alexei Starovoitov <[email protected]>:

Quoted 4 lines · Ilya Leoshkevich

Currently it's not possible to set bpf_jit_enable = 2 when CONFIG_BPF_JIT_ALWAYS_ON is set, which makes debugging certain problems harder.

This is obsolete way of debugging.
Please use bpftool dump jited instead.

Quoted 8 lines
Quoted 4 lines

Is there a way to integrate bpftool nicely with e.g. test_verifier? With bpf_jit_enable = 2, I can see JITed code for each test right away, without pausing it (via gdb or rebuilding with added sleep()) and running bpftool.

On the library side we can set the log_level causing the verifier logic steps to be printed. I guess adding it to bpftool might be nice. At least I would find it useful. I'll probably get to it sometime if its not already there somewhere and/or someone doesn't beat me to it.

+1

Was wondering whether it may be worth it moving parts of the logic from bpftool into libbpf wrt jit dump as a higher level api, so it could be used directly for checking out the jit disasm + opcodes for specific tests given we have the fd there as well, but that might be too specific perhaps and would bring one more lib dependency to libbpf for a rather narrow use case. Adding sleep before prog fd close and/or shelling out to bpftool etc all is a crude temporary hack as well (currently using something long these lines locally). Would it make sense to dump some meta data and generated opcodes per test case to a file as opt-in e.g. ./test_verifier 711 --dump produces 711.opcodes out of bpf_obj_get_info_by_fd() which then bpftool could dump this artifact through its own disasm?

Quoted 2 lines

Thanks,
Daniel

Yes, this sounds fine - if the test fails or behaves strangely, I won't have to re-run it using a special setup, but rather just disasm the dumped JITted image (maybe even without bpftool, just with objdump).

Another question though: what about seccomp? It looks as if those programs are not shown by bpftool, since they are not created using bpf syscall.

Best regards,
Ilya

maintainer
#6 lore raw
Quoted 36 lines · Ilya Leoshkevich
Quoted 25 lines

Am 07.11.2019 um 00:07 schrieb Daniel Borkmann <[email protected]>:

Quoted 11 lines · John Fastabend
Quoted 10 lines · Ilya Leoshkevich
Quoted 9 lines

Am 06.11.2019 um 17:15 schrieb Alexei Starovoitov <[email protected]>:

Quoted 4 lines · Ilya Leoshkevich

Currently it's not possible to set bpf_jit_enable = 2 when CONFIG_BPF_JIT_ALWAYS_ON is set, which makes debugging certain problems harder.

This is obsolete way of debugging.
Please use bpftool dump jited instead.

Quoted 8 lines
Quoted 4 lines

Is there a way to integrate bpftool nicely with e.g. test_verifier? With bpf_jit_enable = 2, I can see JITed code for each test right away, without pausing it (via gdb or rebuilding with added sleep()) and running bpftool.

On the library side we can set the log_level causing the verifier logic steps to be printed. I guess adding it to bpftool might be nice. At least I would find it useful. I'll probably get to it sometime if its not already there somewhere and/or someone doesn't beat me to it.

+1

Quoted 10 lines

Was wondering whether it may be worth it moving parts of the logic from bpftool into libbpf wrt jit dump as a higher level api, so it could be used directly for checking out the jit disasm + opcodes for specific tests given we have the fd there as well, but that might be too specific perhaps and would bring one more lib dependency to libbpf for a rather narrow use case. Adding sleep before prog fd close and/or shelling out to bpftool etc all is a crude temporary hack as well (currently using something long these lines locally). Would it make sense to dump some meta data and generated opcodes per test case to a file as opt-in e.g. ./test_verifier 711 --dump produces 711.opcodes out of bpf_obj_get_info_by_fd() which then bpftool could dump this artifact through its own disasm?

Quoted 7 lines

Yes, this sounds fine - if the test fails or behaves strangely, I won't have to re-run it using a special setup, but rather just disasm the dumped JITted image (maybe even without bpftool, just with objdump).

Another question though: what about seccomp? It looks as if those programs are not shown by bpftool, since they are not created using bpf syscall.

Correct, they are not shown because they are not native (e)BPF.

The criu folks are using PTRACE_SECCOMP_GET_FILTER as one option to dump (not sure if that helps you much though):

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f8e529ed941ba2bbcbf310b575d968159ce7e895