Skip to contentSkip to conversation

faster uprobes

LSF/MM/BPF TOPIC Jiri Olsa 8 participants 31 messages Feb 29, 2024 – Mar 13, 2024

Subsystems (7)
lore
Share & tools
lore

Jiri Olsa

author
#0 lore raw

One of uprobe pain points is having slow execution that involves two traps in worst case scenario or single trap if the original instruction can be emulated. For return uprobes there's one extra trap on top of that.

My current idea on how to make this faster is to follow the optimized kprobes and replace the normal uprobe trap instruction with jump to user space trampoline that:

- executes syscall to call uprobe consumers callbacks
- executes original instructions
- jumps back to continue with the original code

There are of course corner cases where above will have trouble or won't work completely, like:

- executing original instructions in the trampoline is tricky wrt rip relative addressing

- some instructions we can't move to trampoline at all

- the uprobe address is on page boundary so the jump instruction to trampoline would span across 2 pages, hence the page replace won't be atomic, which might cause issues

- ... ? many others I'm sure

Still with all the limitations I think we could be able to speed up some amount of the uprobes, which seems worth doing.

I'd like to have the discussion on the topic and get some agreement or directions on how this should be done.

#1 lore raw
Quoted 11 lines · Jiri Olsa

One of uprobe pain points is having slow execution that involves two traps in worst case scenario or single trap if the original instruction can be emulated. For return uprobes there's one extra trap on top of that.

My current idea on how to make this faster is to follow the optimized kprobes and replace the normal uprobe trap instruction with jump to user space trampoline that:

- executes syscall to call uprobe consumers callbacks

Did you get a chance to measure relative performance of syscall vs int3 interrupt handling? If not, do you think you'll be able to get some numbers by the time the conference starts? This should inform the decision whether it even makes sense to go through all the trouble.

Quoted 22 lines

- executes original instructions
- jumps back to continue with the original code

There are of course corner cases where above will have trouble or won't work completely, like:

- executing original instructions in the trampoline is tricky wrt rip relative addressing

- some instructions we can't move to trampoline at all

- the uprobe address is on page boundary so the jump instruction to trampoline would span across 2 pages, hence the page replace won't be atomic, which might cause issues

- ... ? many others I'm sure

Still with all the limitations I think we could be able to speed up some amount of the uprobes, which seems worth doing.

I'd like to have the discussion on the topic and get some agreement or directions on how this should be done.

author
#2 lore raw
Quoted 11 lines · Andrii Nakryiko
Quoted 10 lines · Jiri Olsa

One of uprobe pain points is having slow execution that involves two traps in worst case scenario or single trap if the original instruction can be emulated. For return uprobes there's one extra trap on top of that.

My current idea on how to make this faster is to follow the optimized kprobes and replace the normal uprobe trap instruction with jump to user space trampoline that:

Quoted 6 lines
Quoted 1 line

- executes syscall to call uprobe consumers callbacks

Did you get a chance to measure relative performance of syscall vs int3 interrupt handling? If not, do you think you'll be able to get some numbers by the time the conference starts? This should inform the decision whether it even makes sense to go through all the trouble.

right, will do that

jirka

Quoted 23 lines
Quoted 23 lines

- executes original instructions
- jumps back to continue with the original code

There are of course corner cases where above will have trouble or won't work completely, like:

- executing original instructions in the trampoline is tricky wrt rip relative addressing

- some instructions we can't move to trampoline at all

- the uprobe address is on page boundary so the jump instruction to trampoline would span across 2 pages, hence the page replace won't be atomic, which might cause issues

- ... ? many others I'm sure

Still with all the limitations I think we could be able to speed up some amount of the uprobes, which seems worth doing.

I'd like to have the discussion on the topic and get some agreement or directions on how this should be done.

#3 lore raw
Quoted 13 lines · Jiri Olsa
Quoted 11 lines · Andrii Nakryiko
Quoted 10 lines · Jiri Olsa

One of uprobe pain points is having slow execution that involves two traps in worst case scenario or single trap if the original instruction can be emulated. For return uprobes there's one extra trap on top of that.

My current idea on how to make this faster is to follow the optimized kprobes and replace the normal uprobe trap instruction with jump to user space trampoline that:

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

- executes syscall to call uprobe consumers callbacks

Did you get a chance to measure relative performance of syscall vs int3 interrupt handling? If not, do you think you'll be able to get some numbers by the time the conference starts? This should inform the decision whether it even makes sense to go through all the trouble.

right, will do that

I believe Yusheng measured syscall vs uprobe performance difference during LPC. iirc it was something like 3x. Certainly necessary to have a benchmark. selftests/bpf/bench has one for uprobe. Probably should extend with sys_bpf.

Regarding:

Quoted 1 line

replace the normal uprobe trap instruction with jump to

user space trampoline

it should probably be a call to trampoline instead of a jump. Unless you plan to generate a different trampoline for every location ?

Also how would you pick a space for a trampoline in the target process ? Analyze /proc/pid/maps and look for gaps in executable sections?

We can start simple with a USDT that uses nop5 instead of nop1 and explicit single trampoline for all USDT locations that saves all (callee and caller saved) registers and then does sys_bpf with a new cmd.

To replace nop5 with a call to trampoline we can use text_poke_bp
approach: replace 1st byte with int3, replace 2-5 with target addr, replace 1st byte to make an actual call insn.

Once patched there will be no simulation of insns or kernel traps. Just normal user code that calls into trampoline, that calls sys_bpf, and returns back.

#4 lore raw
Quoted 22 lines · Alexei Starovoitov
Quoted 20 lines · Jiri Olsa
Quoted 17 lines · Andrii Nakryiko
Quoted 11 lines · Jiri Olsa

One of uprobe pain points is having slow execution that involves two traps in worst case scenario or single trap if the original instruction can be emulated. For return uprobes there's one extra trap on top of that.

My current idea on how to make this faster is to follow the optimized kprobes and replace the normal uprobe trap instruction with jump to user space trampoline that:

- executes syscall to call uprobe consumers callbacks

Did you get a chance to measure relative performance of syscall vs int3 interrupt handling? If not, do you think you'll be able to get some numbers by the time the conference starts? This should inform the decision whether it even makes sense to go through all the trouble.

Quoted 4 lines
Quoted 1 line

right, will do that

I believe Yusheng measured syscall vs uprobe performance difference during LPC. iirc it was something like 3x.

Do you have a link to slides? Was it actual uprobe vs just some fast syscall (not doing BPF program execution) comparison? Or comparing the performance of int3 handling vs equivalent syscall handling.

I suspect it's the former, and so probably not that representative. I'm curious about the performance of going userspace->kernel->userspace through int3 vs syscall (all other things being equal).

Quoted 5 lines

Certainly necessary to have a benchmark.
selftests/bpf/bench has one for uprobe.
Probably should extend with sys_bpf.

Regarding:

Quoted 8 lines
Quoted 1 line

replace the normal uprobe trap instruction with jump to

user space trampoline

it should probably be a call to trampoline instead of a jump. Unless you plan to generate a different trampoline for every location ?

Also how would you pick a space for a trampoline in the target process ? Analyze /proc/pid/maps and look for gaps in executable sections?

kernel already does that for uretprobes, it adds a new "[uprobes]" memory mapping, so this part is already implemented

Quoted 13 lines

We can start simple with a USDT that uses nop5 instead of nop1 and explicit single trampoline for all USDT locations that saves all (callee and caller saved) registers and then does sys_bpf with a new cmd.

To replace nop5 with a call to trampoline we can use text_poke_bp
approach: replace 1st byte with int3, replace 2-5 with target addr, replace 1st byte to make an actual call insn.

Once patched there will be no simulation of insns or kernel traps. Just normal user code that calls into trampoline, that calls sys_bpf, and returns back.

#5 lore raw

Hi!

I did some basic experiment on bpftime, which combined user space trampoline in bpftime with a bpf_prog_test_run syscall to run eBPF code in kernel. In my laptop, it was about 2-3x faster than original trap based Uprobe.

The experiment code was in
https://github.com/eunomia-bpf/bpftime/blob/71f13ae80e93e8ff45e1b0320c25ff14cb25b4ba/runtime/src/bpftime_prog.cpp#L113

(That's just a poc, not kernel patches)

Quoted 69 lines · Andrii Nakryiko
Quoted 22 lines · Alexei Starovoitov
Quoted 20 lines · Jiri Olsa
Quoted 17 lines · Andrii Nakryiko
Quoted 11 lines · Jiri Olsa

One of uprobe pain points is having slow execution that involves two traps in worst case scenario or single trap if the original instruction can be emulated. For return uprobes there's one extra trap on top of that.

My current idea on how to make this faster is to follow the optimized kprobes and replace the normal uprobe trap instruction with jump to user space trampoline that:

- executes syscall to call uprobe consumers callbacks

Did you get a chance to measure relative performance of syscall vs int3 interrupt handling? If not, do you think you'll be able to get some numbers by the time the conference starts? This should inform the decision whether it even makes sense to go through all the trouble.

Quoted 4 lines
Quoted 1 line

right, will do that

I believe Yusheng measured syscall vs uprobe performance difference during LPC. iirc it was something like 3x.

Do you have a link to slides? Was it actual uprobe vs just some fast syscall (not doing BPF program execution) comparison? Or comparing the performance of int3 handling vs equivalent syscall handling.

I suspect it's the former, and so probably not that representative. I'm curious about the performance of going userspace->kernel->userspace through int3 vs syscall (all other things being equal).

Quoted 5 lines

Certainly necessary to have a benchmark.
selftests/bpf/bench has one for uprobe.
Probably should extend with sys_bpf.

Regarding:

Quoted 8 lines
Quoted 1 line

replace the normal uprobe trap instruction with jump to

user space trampoline

it should probably be a call to trampoline instead of a jump. Unless you plan to generate a different trampoline for every location ?

Also how would you pick a space for a trampoline in the target process ? Analyze /proc/pid/maps and look for gaps in executable sections?

kernel already does that for uretprobes, it adds a new "[uprobes]" memory mapping, so this part is already implemented

Quoted 13 lines

We can start simple with a USDT that uses nop5 instead of nop1 and explicit single trampoline for all USDT locations that saves all (callee and caller saved) registers and then does sys_bpf with a new cmd.

To replace nop5 with a call to trampoline we can use text_poke_bp
approach: replace 1st byte with int3, replace 2-5 with target addr, replace 1st byte to make an actual call insn.

Once patched there will be no simulation of insns or kernel traps. Just normal user code that calls into trampoline, that calls sys_bpf, and returns back.

author
#6 lore raw
In reply to Andrii Nakryiko
Quoted 29 lines · Andrii Nakryiko
Quoted 22 lines · Alexei Starovoitov
Quoted 20 lines · Jiri Olsa
Quoted 17 lines · Andrii Nakryiko
Quoted 11 lines · Jiri Olsa

One of uprobe pain points is having slow execution that involves two traps in worst case scenario or single trap if the original instruction can be emulated. For return uprobes there's one extra trap on top of that.

My current idea on how to make this faster is to follow the optimized kprobes and replace the normal uprobe trap instruction with jump to user space trampoline that:

- executes syscall to call uprobe consumers callbacks

Did you get a chance to measure relative performance of syscall vs int3 interrupt handling? If not, do you think you'll be able to get some numbers by the time the conference starts? This should inform the decision whether it even makes sense to go through all the trouble.

Quoted 4 lines
Quoted 1 line

right, will do that

I believe Yusheng measured syscall vs uprobe performance difference during LPC. iirc it was something like 3x.

Quoted 8 lines

Do you have a link to slides? Was it actual uprobe vs just some fast syscall (not doing BPF program execution) comparison? Or comparing the performance of int3 handling vs equivalent syscall handling.

I suspect it's the former, and so probably not that representative. I'm curious about the performance of going userspace->kernel->userspace through int3 vs syscall (all other things being equal).

I have a simple test [1] comparing:
- uprobe with 2 traps
- uprobe with 1 trap
- syscall executing uprobe

the syscall takes uprobe address as argument, finds the uprobe and executes its consumers, which should be comparable to what the trampoline will do

test does same amount of loops triggering each uprobe type and measures the time it took

  # ./test_progs -t uprobe_syscall_bench -v
  bpf_testmod.ko is already unloaded.
  Loading bpf_testmod.ko...
  Successfully loaded bpf_testmod.ko.
  test_bench_1:PASS:uprobe_bench__open_and_load 0 nsec
  test_bench_1:PASS:uprobe_bench__attach 0 nsec
  test_bench_1:PASS:uprobe1_cnt 0 nsec
  test_bench_1:PASS:syscalls_uprobe1_cnt 0 nsec
  test_bench_1:PASS:uprobe2_cnt 0 nsec
  test_bench_1: uprobes (1 trap) in  36.439s
  test_bench_1: uprobes (2 trap) in  91.960s
  test_bench_1: syscalls         in  17.872s
  #395/1   uprobe_syscall_bench/bench_1:OK
  #395     uprobe_syscall_bench:OK
  Summary: 1/1 PASSED, 0 SKIPPED, 0 FAILED

syscall uprobe execution seems to be ~2x faster than 1 trap uprobe and ~5x faster than 2 traps uprobe

jirka

[1] https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git/log/?h=uprobe_syscall_bench

Quoted 31 lines
Quoted 6 lines

Certainly necessary to have a benchmark.
selftests/bpf/bench has one for uprobe.
Probably should extend with sys_bpf.

Regarding:

Quoted 8 lines
Quoted 1 line

replace the normal uprobe trap instruction with jump to

user space trampoline

it should probably be a call to trampoline instead of a jump. Unless you plan to generate a different trampoline for every location ?

Also how would you pick a space for a trampoline in the target process ? Analyze /proc/pid/maps and look for gaps in executable sections?

kernel already does that for uretprobes, it adds a new "[uprobes]" memory mapping, so this part is already implemented

Quoted 13 lines

We can start simple with a USDT that uses nop5 instead of nop1 and explicit single trampoline for all USDT locations that saves all (callee and caller saved) registers and then does sys_bpf with a new cmd.

To replace nop5 with a call to trampoline we can use text_poke_bp
approach: replace 1st byte with int3, replace 2-5 with target addr, replace 1st byte to make an actual call insn.

Once patched there will be no simulation of insns or kernel traps. Just normal user code that calls into trampoline, that calls sys_bpf, and returns back.

#7 lore raw
Quoted 67 lines · Jiri Olsa
Quoted 29 lines · Andrii Nakryiko
Quoted 22 lines · Alexei Starovoitov
Quoted 20 lines · Jiri Olsa
Quoted 17 lines · Andrii Nakryiko
Quoted 11 lines · Jiri Olsa

One of uprobe pain points is having slow execution that involves two traps in worst case scenario or single trap if the original instruction can be emulated. For return uprobes there's one extra trap on top of that.

My current idea on how to make this faster is to follow the optimized kprobes and replace the normal uprobe trap instruction with jump to user space trampoline that:

- executes syscall to call uprobe consumers callbacks

Did you get a chance to measure relative performance of syscall vs int3 interrupt handling? If not, do you think you'll be able to get some numbers by the time the conference starts? This should inform the decision whether it even makes sense to go through all the trouble.

Quoted 4 lines
Quoted 1 line

right, will do that

I believe Yusheng measured syscall vs uprobe performance difference during LPC. iirc it was something like 3x.

Quoted 8 lines

Do you have a link to slides? Was it actual uprobe vs just some fast syscall (not doing BPF program execution) comparison? Or comparing the performance of int3 handling vs equivalent syscall handling.

I suspect it's the former, and so probably not that representative. I'm curious about the performance of going userspace->kernel->userspace through int3 vs syscall (all other things being equal).

I have a simple test [1] comparing:
- uprobe with 2 traps
- uprobe with 1 trap
- syscall executing uprobe

the syscall takes uprobe address as argument, finds the uprobe and executes its consumers, which should be comparable to what the trampoline will do

test does same amount of loops triggering each uprobe type and measures the time it took

  # ./test_progs -t uprobe_syscall_bench -v
  bpf_testmod.ko is already unloaded.
  Loading bpf_testmod.ko...
  Successfully loaded bpf_testmod.ko.
  test_bench_1:PASS:uprobe_bench__open_and_load 0 nsec
  test_bench_1:PASS:uprobe_bench__attach 0 nsec
  test_bench_1:PASS:uprobe1_cnt 0 nsec
  test_bench_1:PASS:syscalls_uprobe1_cnt 0 nsec
  test_bench_1:PASS:uprobe2_cnt 0 nsec
  test_bench_1: uprobes (1 trap) in  36.439s
  test_bench_1: uprobes (2 trap) in  91.960s
  test_bench_1: syscalls         in  17.872s
  #395/1   uprobe_syscall_bench/bench_1:OK
  #395     uprobe_syscall_bench:OK
  Summary: 1/1 PASSED, 0 SKIPPED, 0 FAILED
Quoted 3 lines

syscall uprobe execution seems to be ~2x faster than 1 trap uprobe and ~5x faster than 2 traps uprobe

Thanks for running benchmarks! I quickly looked at the selftest and noticed this:

+/*
+ * Assuming following prolog:
+ *

+ * 6984ac:       55                      push   %rbp
+ * 6984ad:       48 89 e5                mov    %rsp,%rbp

+ */
+noinline void uprobe2_bench_trigger(void)
+{
+ asm volatile ("");
+}

This actually will be optimized out to just ret in -O2 mode (make RELEASE=1 for selftests):

00000000005a0ce0 <uprobe2_bench_trigger>:
5a0ce0: c3 retq
5a0ce1: 66 66 2e 0f 1f 84 00 00 00 00 00 nopw %cs:(%rax,%rax)
5a0cec: 0f 1f 40 00 nopl (%rax)

So be careful with that.

Also, I just updated our existing set of uprobe benchmarks (see [0]), do you mind adding your syscall-based one as another one there and running all of them and sharing the numbers with us? Very curious to see both absolute and relative numbers from that benchmark. (and please do build with RELEASE=1)

You should be able to just run benchs/run_bench_uprobes.sh (also don't forget to add your syscall-based benchmark to the list of benchmarks in that shell script).

Thank you!

BTW, while I think patching multiple instructions for syscall-based uprobe is going to be extremely tricky, I think at least u*ret*probe's int3 can be pretty easily optimized away with syscall, given that the kernel controls code generation there. If anything, it will get the uretprobe case a bit closer to the performance of uprobe. Give it some thought.

[0] https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]/

Quoted 36 lines

jirka

[1] https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git/log/?h=uprobe_syscall_bench

Quoted 31 lines
Quoted 6 lines

Certainly necessary to have a benchmark.
selftests/bpf/bench has one for uprobe.
Probably should extend with sys_bpf.

Regarding:

Quoted 8 lines
Quoted 1 line

replace the normal uprobe trap instruction with jump to

user space trampoline

it should probably be a call to trampoline instead of a jump. Unless you plan to generate a different trampoline for every location ?

Also how would you pick a space for a trampoline in the target process ? Analyze /proc/pid/maps and look for gaps in executable sections?

kernel already does that for uretprobes, it adds a new "[uprobes]" memory mapping, so this part is already implemented

Quoted 13 lines

We can start simple with a USDT that uses nop5 instead of nop1 and explicit single trampoline for all USDT locations that saves all (callee and caller saved) registers and then does sys_bpf with a new cmd.

To replace nop5 with a call to trampoline we can use text_poke_bp
approach: replace 1st byte with int3, replace 2-5 with target addr, replace 1st byte to make an actual call insn.

Once patched there will be no simulation of insns or kernel traps. Just normal user code that calls into trampoline, that calls sys_bpf, and returns back.

author
#8 lore raw
Quoted 89 lines · Andrii Nakryiko
Quoted 67 lines · Jiri Olsa
Quoted 29 lines · Andrii Nakryiko
Quoted 22 lines · Alexei Starovoitov
Quoted 20 lines · Jiri Olsa
Quoted 17 lines · Andrii Nakryiko
Quoted 11 lines · Jiri Olsa

One of uprobe pain points is having slow execution that involves two traps in worst case scenario or single trap if the original instruction can be emulated. For return uprobes there's one extra trap on top of that.

My current idea on how to make this faster is to follow the optimized kprobes and replace the normal uprobe trap instruction with jump to user space trampoline that:

- executes syscall to call uprobe consumers callbacks

Did you get a chance to measure relative performance of syscall vs int3 interrupt handling? If not, do you think you'll be able to get some numbers by the time the conference starts? This should inform the decision whether it even makes sense to go through all the trouble.

Quoted 4 lines
Quoted 1 line

right, will do that

I believe Yusheng measured syscall vs uprobe performance difference during LPC. iirc it was something like 3x.

Quoted 8 lines

Do you have a link to slides? Was it actual uprobe vs just some fast syscall (not doing BPF program execution) comparison? Or comparing the performance of int3 handling vs equivalent syscall handling.

I suspect it's the former, and so probably not that representative. I'm curious about the performance of going userspace->kernel->userspace through int3 vs syscall (all other things being equal).

I have a simple test [1] comparing:
- uprobe with 2 traps
- uprobe with 1 trap
- syscall executing uprobe

the syscall takes uprobe address as argument, finds the uprobe and executes its consumers, which should be comparable to what the trampoline will do

test does same amount of loops triggering each uprobe type and measures the time it took

  # ./test_progs -t uprobe_syscall_bench -v
  bpf_testmod.ko is already unloaded.
  Loading bpf_testmod.ko...
  Successfully loaded bpf_testmod.ko.
  test_bench_1:PASS:uprobe_bench__open_and_load 0 nsec
  test_bench_1:PASS:uprobe_bench__attach 0 nsec
  test_bench_1:PASS:uprobe1_cnt 0 nsec
  test_bench_1:PASS:syscalls_uprobe1_cnt 0 nsec
  test_bench_1:PASS:uprobe2_cnt 0 nsec
  test_bench_1: uprobes (1 trap) in  36.439s
  test_bench_1: uprobes (2 trap) in  91.960s
  test_bench_1: syscalls         in  17.872s
  #395/1   uprobe_syscall_bench/bench_1:OK
  #395     uprobe_syscall_bench:OK
  Summary: 1/1 PASSED, 0 SKIPPED, 0 FAILED
Quoted 3 lines

syscall uprobe execution seems to be ~2x faster than 1 trap uprobe and ~5x faster than 2 traps uprobe

Thanks for running benchmarks! I quickly looked at the selftest and noticed this:

(quoted diff) +10 -0
+/*
+ * Assuming following prolog:
+ *
+ * 6984ac:       55                      push   %rbp
+ * 6984ad:       48 89 e5                mov    %rsp,%rbp
+ */
+noinline void uprobe2_bench_trigger(void)
+{
+        asm volatile ("");
+}

This actually will be optimized out to just ret in -O2 mode (make RELEASE=1 for selftests):

Quoted 6 lines

00000000005a0ce0 <uprobe2_bench_trigger>:
5a0ce0: c3 retq
5a0ce1: 66 66 2e 0f 1f 84 00 00 00 00 00 nopw %cs:(%rax,%rax)
5a0cec: 0f 1f 40 00 nopl (%rax)

So be careful with that.

right, I did not mean for this to be checked in, just wanted to get the numbers quickly

Quoted 10 lines

Also, I just updated our existing set of uprobe benchmarks (see [0]), do you mind adding your syscall-based one as another one there and running all of them and sharing the numbers with us? Very curious to see both absolute and relative numbers from that benchmark. (and please do build with RELEASE=1)

You should be able to just run benchs/run_bench_uprobes.sh (also don't forget to add your syscall-based benchmark to the list of benchmarks in that shell script).

yes, saw it and was going to run/compare it.. it's good idea to add the syscall one and get all numbers together, will do that

Quoted 10 lines

Thank you!

BTW, while I think patching multiple instructions for syscall-based uprobe is going to be extremely tricky, I think at least u*ret*probe's int3 can be pretty easily optimized away with syscall, given that the kernel controls code generation there. If anything, it will get the uretprobe case a bit closer to the performance of uprobe. Give it some thought.

hm, right.. the trampoline is there already, but at the moment is global and used by all uretprobes.. and int3 code moves userspace (changes rip) to the original return address.. maybe we can do that through syscall as well

or we could add jump back to uretprobe's original return addrress to the trampoline, but then we need special trampoline for each uretprobe, I'll check

thanks,
jirka

Quoted 40 lines

[0] https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]/

Quoted 36 lines

jirka

[1] https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git/log/?h=uprobe_syscall_bench

Quoted 31 lines
Quoted 6 lines

Certainly necessary to have a benchmark.
selftests/bpf/bench has one for uprobe.
Probably should extend with sys_bpf.

Regarding:

Quoted 8 lines
Quoted 1 line

replace the normal uprobe trap instruction with jump to

user space trampoline

it should probably be a call to trampoline instead of a jump. Unless you plan to generate a different trampoline for every location ?

Also how would you pick a space for a trampoline in the target process ? Analyze /proc/pid/maps and look for gaps in executable sections?

kernel already does that for uretprobes, it adds a new "[uprobes]" memory mapping, so this part is already implemented

Quoted 13 lines

We can start simple with a USDT that uses nop5 instead of nop1 and explicit single trampoline for all USDT locations that saves all (callee and caller saved) registers and then does sys_bpf with a new cmd.

To replace nop5 with a call to trampoline we can use text_poke_bp
approach: replace 1st byte with int3, replace 2-5 with target addr, replace 1st byte to make an actual call insn.

Once patched there will be no simulation of insns or kernel traps. Just normal user code that calls into trampoline, that calls sys_bpf, and returns back.

author
#9 lore raw
Quoted 107 lines · Jiri Olsa
Quoted 89 lines · Andrii Nakryiko
Quoted 67 lines · Jiri Olsa
Quoted 29 lines · Andrii Nakryiko
Quoted 22 lines · Alexei Starovoitov
Quoted 20 lines · Jiri Olsa
Quoted 17 lines · Andrii Nakryiko
Quoted 11 lines · Jiri Olsa

One of uprobe pain points is having slow execution that involves two traps in worst case scenario or single trap if the original instruction can be emulated. For return uprobes there's one extra trap on top of that.

My current idea on how to make this faster is to follow the optimized kprobes and replace the normal uprobe trap instruction with jump to user space trampoline that:

- executes syscall to call uprobe consumers callbacks

Did you get a chance to measure relative performance of syscall vs int3 interrupt handling? If not, do you think you'll be able to get some numbers by the time the conference starts? This should inform the decision whether it even makes sense to go through all the trouble.

Quoted 4 lines
Quoted 1 line

right, will do that

I believe Yusheng measured syscall vs uprobe performance difference during LPC. iirc it was something like 3x.

Quoted 8 lines

Do you have a link to slides? Was it actual uprobe vs just some fast syscall (not doing BPF program execution) comparison? Or comparing the performance of int3 handling vs equivalent syscall handling.

I suspect it's the former, and so probably not that representative. I'm curious about the performance of going userspace->kernel->userspace through int3 vs syscall (all other things being equal).

I have a simple test [1] comparing:
- uprobe with 2 traps
- uprobe with 1 trap
- syscall executing uprobe

the syscall takes uprobe address as argument, finds the uprobe and executes its consumers, which should be comparable to what the trampoline will do

test does same amount of loops triggering each uprobe type and measures the time it took

  # ./test_progs -t uprobe_syscall_bench -v
  bpf_testmod.ko is already unloaded.
  Loading bpf_testmod.ko...
  Successfully loaded bpf_testmod.ko.
  test_bench_1:PASS:uprobe_bench__open_and_load 0 nsec
  test_bench_1:PASS:uprobe_bench__attach 0 nsec
  test_bench_1:PASS:uprobe1_cnt 0 nsec
  test_bench_1:PASS:syscalls_uprobe1_cnt 0 nsec
  test_bench_1:PASS:uprobe2_cnt 0 nsec
  test_bench_1: uprobes (1 trap) in  36.439s
  test_bench_1: uprobes (2 trap) in  91.960s
  test_bench_1: syscalls         in  17.872s
  #395/1   uprobe_syscall_bench/bench_1:OK
  #395     uprobe_syscall_bench:OK
  Summary: 1/1 PASSED, 0 SKIPPED, 0 FAILED
Quoted 3 lines

syscall uprobe execution seems to be ~2x faster than 1 trap uprobe and ~5x faster than 2 traps uprobe

Thanks for running benchmarks! I quickly looked at the selftest and noticed this:

(quoted diff) +10 -0
+/*
+ * Assuming following prolog:
+ *
+ * 6984ac:       55                      push   %rbp
+ * 6984ad:       48 89 e5                mov    %rsp,%rbp
+ */
+noinline void uprobe2_bench_trigger(void)
+{
+        asm volatile ("");
+}

This actually will be optimized out to just ret in -O2 mode (make RELEASE=1 for selftests):

Quoted 6 lines

00000000005a0ce0 <uprobe2_bench_trigger>:
5a0ce0: c3 retq
5a0ce1: 66 66 2e 0f 1f 84 00 00 00 00 00 nopw %cs:(%rax,%rax)
5a0cec: 0f 1f 40 00 nopl (%rax)

So be careful with that.

right, I did not mean for this to be checked in, just wanted to get the numbers quickly

Quoted 7 lines

Also, I just updated our existing set of uprobe benchmarks (see [0]), do you mind adding your syscall-based one as another one there and running all of them and sharing the numbers with us? Very curious to see both absolute and relative numbers from that benchmark. (and please do build with RELEASE=1)

Quoted 6 lines
Quoted 3 lines

You should be able to just run benchs/run_bench_uprobes.sh (also don't forget to add your syscall-based benchmark to the list of benchmarks in that shell script).

yes, saw it and was going to run/compare it.. it's good idea to add the syscall one and get all numbers together, will do that

seems to be consistent with my previous test:

base           :   15.854 ± 0.007M/s
uprobe-nop     :    2.859 ± 0.007M/s
uprobe-push    :    2.697 ± 0.002M/s
uprobe-ret     :    1.081 ± 0.000M/s
uprobe-syscall :    5.520 ± 0.006M/s
uretprobe-nop  :    1.422 ± 0.002M/s
uretprobe-push :    1.396 ± 0.002M/s
uretprobe-ret  :    0.787 ± 0.000M/s
uretprobe-syscall:    1.888 ± 0.002M/s

syscall uprobe is ~2x faster than 1 trap uprobe and ~5x faster than 2 traps uprobe

uretprobe is bit more tricky to compare, the speed up is there for the initial uprobe hit, then there's again the trap from the uretprobe trampoline

I have the bench changes in here [1], I'll send it out together with rfc post

jirka

[1] https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git/log/?h=uprobe_syscall_bench_1

Quoted 64 lines
Quoted 11 lines

Thank you!

BTW, while I think patching multiple instructions for syscall-based uprobe is going to be extremely tricky, I think at least u*ret*probe's int3 can be pretty easily optimized away with syscall, given that the kernel controls code generation there. If anything, it will get the uretprobe case a bit closer to the performance of uprobe. Give it some thought.

hm, right.. the trampoline is there already, but at the moment is global and used by all uretprobes.. and int3 code moves userspace (changes rip) to the original return address.. maybe we can do that through syscall as well

or we could add jump back to uretprobe's original return addrress to the trampoline, but then we need special trampoline for each uretprobe, I'll check

thanks,
jirka

Quoted 40 lines

[0] https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]/

Quoted 36 lines

jirka

[1] https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git/log/?h=uprobe_syscall_bench

Quoted 31 lines
Quoted 6 lines

Certainly necessary to have a benchmark.
selftests/bpf/bench has one for uprobe.
Probably should extend with sys_bpf.

Regarding:

Quoted 8 lines
Quoted 1 line

replace the normal uprobe trap instruction with jump to

user space trampoline

it should probably be a call to trampoline instead of a jump. Unless you plan to generate a different trampoline for every location ?

Also how would you pick a space for a trampoline in the target process ? Analyze /proc/pid/maps and look for gaps in executable sections?

kernel already does that for uretprobes, it adds a new "[uprobes]" memory mapping, so this part is already implemented

Quoted 13 lines

We can start simple with a USDT that uses nop5 instead of nop1 and explicit single trampoline for all USDT locations that saves all (callee and caller saved) registers and then does sys_bpf with a new cmd.

To replace nop5 with a call to trampoline we can use text_poke_bp
approach: replace 1st byte with int3, replace 2-5 with target addr, replace 1st byte to make an actual call insn.

Once patched there will be no simulation of insns or kernel traps. Just normal user code that calls into trampoline, that calls sys_bpf, and returns back.

#10 lore raw
Quoted 128 lines · Jiri Olsa
Quoted 107 lines · Jiri Olsa
Quoted 89 lines · Andrii Nakryiko
Quoted 67 lines · Jiri Olsa
Quoted 29 lines · Andrii Nakryiko
Quoted 22 lines · Alexei Starovoitov
Quoted 20 lines · Jiri Olsa
Quoted 17 lines · Andrii Nakryiko
Quoted 11 lines · Jiri Olsa

One of uprobe pain points is having slow execution that involves two traps in worst case scenario or single trap if the original instruction can be emulated. For return uprobes there's one extra trap on top of that.

My current idea on how to make this faster is to follow the optimized kprobes and replace the normal uprobe trap instruction with jump to user space trampoline that:

- executes syscall to call uprobe consumers callbacks

Did you get a chance to measure relative performance of syscall vs int3 interrupt handling? If not, do you think you'll be able to get some numbers by the time the conference starts? This should inform the decision whether it even makes sense to go through all the trouble.

Quoted 4 lines
Quoted 1 line

right, will do that

I believe Yusheng measured syscall vs uprobe performance difference during LPC. iirc it was something like 3x.

Quoted 8 lines

Do you have a link to slides? Was it actual uprobe vs just some fast syscall (not doing BPF program execution) comparison? Or comparing the performance of int3 handling vs equivalent syscall handling.

I suspect it's the former, and so probably not that representative. I'm curious about the performance of going userspace->kernel->userspace through int3 vs syscall (all other things being equal).

I have a simple test [1] comparing:
- uprobe with 2 traps
- uprobe with 1 trap
- syscall executing uprobe

the syscall takes uprobe address as argument, finds the uprobe and executes its consumers, which should be comparable to what the trampoline will do

test does same amount of loops triggering each uprobe type and measures the time it took

  # ./test_progs -t uprobe_syscall_bench -v
  bpf_testmod.ko is already unloaded.
  Loading bpf_testmod.ko...
  Successfully loaded bpf_testmod.ko.
  test_bench_1:PASS:uprobe_bench__open_and_load 0 nsec
  test_bench_1:PASS:uprobe_bench__attach 0 nsec
  test_bench_1:PASS:uprobe1_cnt 0 nsec
  test_bench_1:PASS:syscalls_uprobe1_cnt 0 nsec
  test_bench_1:PASS:uprobe2_cnt 0 nsec
  test_bench_1: uprobes (1 trap) in  36.439s
  test_bench_1: uprobes (2 trap) in  91.960s
  test_bench_1: syscalls         in  17.872s
  #395/1   uprobe_syscall_bench/bench_1:OK
  #395     uprobe_syscall_bench:OK
  Summary: 1/1 PASSED, 0 SKIPPED, 0 FAILED
Quoted 3 lines

syscall uprobe execution seems to be ~2x faster than 1 trap uprobe and ~5x faster than 2 traps uprobe

Thanks for running benchmarks! I quickly looked at the selftest and noticed this:

(quoted diff) +10 -0
+/*
+ * Assuming following prolog:
+ *
+ * 6984ac:       55                      push   %rbp
+ * 6984ad:       48 89 e5                mov    %rsp,%rbp
+ */
+noinline void uprobe2_bench_trigger(void)
+{
+        asm volatile ("");
+}

This actually will be optimized out to just ret in -O2 mode (make RELEASE=1 for selftests):

Quoted 6 lines

00000000005a0ce0 <uprobe2_bench_trigger>:
5a0ce0: c3 retq
5a0ce1: 66 66 2e 0f 1f 84 00 00 00 00 00 nopw %cs:(%rax,%rax)
5a0cec: 0f 1f 40 00 nopl (%rax)

So be careful with that.

right, I did not mean for this to be checked in, just wanted to get the numbers quickly

Quoted 7 lines

Also, I just updated our existing set of uprobe benchmarks (see [0]), do you mind adding your syscall-based one as another one there and running all of them and sharing the numbers with us? Very curious to see both absolute and relative numbers from that benchmark. (and please do build with RELEASE=1)

Quoted 6 lines
Quoted 3 lines

You should be able to just run benchs/run_bench_uprobes.sh (also don't forget to add your syscall-based benchmark to the list of benchmarks in that shell script).

yes, saw it and was going to run/compare it.. it's good idea to add the syscall one and get all numbers together, will do that

seems to be consistent with my previous test:

base           :   15.854 ± 0.007M/s
uprobe-nop     :    2.859 ± 0.007M/s
uprobe-push    :    2.697 ± 0.002M/s
uprobe-ret     :    1.081 ± 0.000M/s
uprobe-syscall :    5.520 ± 0.006M/s
uretprobe-nop  :    1.422 ± 0.002M/s
uretprobe-push :    1.396 ± 0.002M/s
uretprobe-ret  :    0.787 ± 0.000M/s
uretprobe-syscall:    1.888 ± 0.002M/s
Quoted 2 lines

syscall uprobe is ~2x faster than 1 trap uprobe and ~5x faster than 2 traps uprobe

great, thanks a lot for the numbers! It's good that we have comparable benchmark numbers now.

Quoted 2 lines

uretprobe is bit more tricky to compare, the speed up is there for the initial uprobe hit, then there's again the trap from the uretprobe trampoline

yep, makes sense

Quoted 72 lines

I have the bench changes in here [1], I'll send it out together with rfc post

jirka

[1] https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git/log/?h=uprobe_syscall_bench_1

Quoted 64 lines
Quoted 11 lines

Thank you!

BTW, while I think patching multiple instructions for syscall-based uprobe is going to be extremely tricky, I think at least u*ret*probe's int3 can be pretty easily optimized away with syscall, given that the kernel controls code generation there. If anything, it will get the uretprobe case a bit closer to the performance of uprobe. Give it some thought.

hm, right.. the trampoline is there already, but at the moment is global and used by all uretprobes.. and int3 code moves userspace (changes rip) to the original return address.. maybe we can do that through syscall as well

or we could add jump back to uretprobe's original return addrress to the trampoline, but then we need special trampoline for each uretprobe, I'll check

thanks,
jirka

Quoted 40 lines

[0] https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]/

Quoted 36 lines

jirka

[1] https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git/log/?h=uprobe_syscall_bench

Quoted 31 lines
Quoted 6 lines

Certainly necessary to have a benchmark.
selftests/bpf/bench has one for uprobe.
Probably should extend with sys_bpf.

Regarding:

Quoted 8 lines
Quoted 1 line

replace the normal uprobe trap instruction with jump to

user space trampoline

it should probably be a call to trampoline instead of a jump. Unless you plan to generate a different trampoline for every location ?

Also how would you pick a space for a trampoline in the target process ? Analyze /proc/pid/maps and look for gaps in executable sections?

kernel already does that for uretprobes, it adds a new "[uprobes]" memory mapping, so this part is already implemented

Quoted 13 lines

We can start simple with a USDT that uses nop5 instead of nop1 and explicit single trampoline for all USDT locations that saves all (callee and caller saved) registers and then does sys_bpf with a new cmd.

To replace nop5 with a call to trampoline we can use text_poke_bp
approach: replace 1st byte with int3, replace 2-5 with target addr, replace 1st byte to make an actual call insn.

Once patched there will be no simulation of insns or kernel traps. Just normal user code that calls into trampoline, that calls sys_bpf, and returns back.

author
#11 lore raw
In reply to Jiri Olsa
Quoted 125 lines · Jiri Olsa
Quoted 89 lines · Andrii Nakryiko
Quoted 67 lines · Jiri Olsa
Quoted 29 lines · Andrii Nakryiko
Quoted 22 lines · Alexei Starovoitov
Quoted 20 lines · Jiri Olsa
Quoted 17 lines · Andrii Nakryiko
Quoted 11 lines · Jiri Olsa

One of uprobe pain points is having slow execution that involves two traps in worst case scenario or single trap if the original instruction can be emulated. For return uprobes there's one extra trap on top of that.

My current idea on how to make this faster is to follow the optimized kprobes and replace the normal uprobe trap instruction with jump to user space trampoline that:

- executes syscall to call uprobe consumers callbacks

Did you get a chance to measure relative performance of syscall vs int3 interrupt handling? If not, do you think you'll be able to get some numbers by the time the conference starts? This should inform the decision whether it even makes sense to go through all the trouble.

Quoted 4 lines
Quoted 1 line

right, will do that

I believe Yusheng measured syscall vs uprobe performance difference during LPC. iirc it was something like 3x.

Quoted 8 lines

Do you have a link to slides? Was it actual uprobe vs just some fast syscall (not doing BPF program execution) comparison? Or comparing the performance of int3 handling vs equivalent syscall handling.

I suspect it's the former, and so probably not that representative. I'm curious about the performance of going userspace->kernel->userspace through int3 vs syscall (all other things being equal).

I have a simple test [1] comparing:
- uprobe with 2 traps
- uprobe with 1 trap
- syscall executing uprobe

the syscall takes uprobe address as argument, finds the uprobe and executes its consumers, which should be comparable to what the trampoline will do

test does same amount of loops triggering each uprobe type and measures the time it took

  # ./test_progs -t uprobe_syscall_bench -v
  bpf_testmod.ko is already unloaded.
  Loading bpf_testmod.ko...
  Successfully loaded bpf_testmod.ko.
  test_bench_1:PASS:uprobe_bench__open_and_load 0 nsec
  test_bench_1:PASS:uprobe_bench__attach 0 nsec
  test_bench_1:PASS:uprobe1_cnt 0 nsec
  test_bench_1:PASS:syscalls_uprobe1_cnt 0 nsec
  test_bench_1:PASS:uprobe2_cnt 0 nsec
  test_bench_1: uprobes (1 trap) in  36.439s
  test_bench_1: uprobes (2 trap) in  91.960s
  test_bench_1: syscalls         in  17.872s
  #395/1   uprobe_syscall_bench/bench_1:OK
  #395     uprobe_syscall_bench:OK
  Summary: 1/1 PASSED, 0 SKIPPED, 0 FAILED
Quoted 3 lines

syscall uprobe execution seems to be ~2x faster than 1 trap uprobe and ~5x faster than 2 traps uprobe

Thanks for running benchmarks! I quickly looked at the selftest and noticed this:

(quoted diff) +10 -0
+/*
+ * Assuming following prolog:
+ *
+ * 6984ac:       55                      push   %rbp
+ * 6984ad:       48 89 e5                mov    %rsp,%rbp
+ */
+noinline void uprobe2_bench_trigger(void)
+{
+        asm volatile ("");
+}

This actually will be optimized out to just ret in -O2 mode (make RELEASE=1 for selftests):

Quoted 6 lines

00000000005a0ce0 <uprobe2_bench_trigger>:
5a0ce0: c3 retq
5a0ce1: 66 66 2e 0f 1f 84 00 00 00 00 00 nopw %cs:(%rax,%rax)
5a0cec: 0f 1f 40 00 nopl (%rax)

So be careful with that.

right, I did not mean for this to be checked in, just wanted to get the numbers quickly

Quoted 10 lines

Also, I just updated our existing set of uprobe benchmarks (see [0]), do you mind adding your syscall-based one as another one there and running all of them and sharing the numbers with us? Very curious to see both absolute and relative numbers from that benchmark. (and please do build with RELEASE=1)

You should be able to just run benchs/run_bench_uprobes.sh (also don't forget to add your syscall-based benchmark to the list of benchmarks in that shell script).

yes, saw it and was going to run/compare it.. it's good idea to add the syscall one and get all numbers together, will do that

Quoted 10 lines

Thank you!

BTW, while I think patching multiple instructions for syscall-based uprobe is going to be extremely tricky, I think at least u*ret*probe's int3 can be pretty easily optimized away with syscall, given that the kernel controls code generation there. If anything, it will get the uretprobe case a bit closer to the performance of uprobe. Give it some thought.

Quoted 4 lines

hm, right.. the trampoline is there already, but at the moment is global and used by all uretprobes.. and int3 code moves userspace (changes rip) to the original return address.. maybe we can do that through syscall as well

it seems like good idea, I tried change below (use syscall on return trampoline) and got some speedup:

current:

  base           :   15.817 ± 0.009M/s
  uprobe-nop     :    2.901 ± 0.000M/s
  uprobe-push    :    2.743 ± 0.002M/s
  uprobe-ret     :    1.089 ± 0.001M/s
  uretprobe-nop  :    1.448 ± 0.001M/s

uretprobe-push : 1.407 ± 0.001M/s
uretprobe-ret : 0.792 ± 0.001M/s

with syscall:

  base           :   15.831 ± 0.026M/s
  uprobe-nop     :    2.904 ± 0.001M/s
  uprobe-push    :    2.764 ± 0.002M/s
  uprobe-ret     :    1.082 ± 0.001M/s
  uretprobe-nop  :    1.785 ± 0.000M/s

uretprobe-push : 1.733 ± 0.001M/s
uretprobe-ret : 0.885 ± 0.004M/s

~23% for nop/push (emulated) cases, ~11% for ret (sstep) case

jirka

---

arch/x86/entry/syscalls/syscall_64.tbl +1 -0
index 7e8d46f4147f..fa5f8a058bc2 100644
@@ -383,6 +383,7 @@
 459	common	lsm_get_self_attr	sys_lsm_get_self_attr
 460	common	lsm_set_self_attr	sys_lsm_set_self_attr
 461	common	lsm_list_modules	sys_lsm_list_modules
+462	64	uprobe			sys_uprobe
 
 #
 # Due to a historical design error, certain syscalls are numbered differently
arch/x86/kernel/uprobes.c +49 -0

66 lines not on this page · raw message

include/linux/syscalls.h +2 -0
index 77eb9b0e7685..4f7d5b41b718 100644
@@ -987,6 +987,8 @@ asmlinkage long sys_spu_run(int fd, __u32 __user *unpc,
 asmlinkage long sys_spu_create(const char __user *name,
 		unsigned int flags, umode_t mode, int fd);
 
+asmlinkage long sys_uprobe(unsigned long cmd);
+
 
 /*
  * Deprecated system calls which are still defined in
include/linux/uprobes.h +2 -0
index f46e0ca0169c..9ef244c8ff19 100644
@@ -138,6 +138,8 @@ extern bool arch_uretprobe_is_alive(struct return_instance *ret, enum rp_check c
 extern bool arch_uprobe_ignore(struct arch_uprobe *aup, struct pt_regs *regs);
 extern void arch_uprobe_copy_ixol(struct page *page, unsigned long vaddr,
 					 void *src, unsigned long len);
+extern void uprobe_handle_trampoline(struct pt_regs *regs);
+uprobe_opcode_t* arch_uprobe_trampoline(unsigned long *psize);
 #else /* !CONFIG_UPROBES */
 struct uprobes_state {
 };
include/uapi/asm-generic/unistd.h +4 -1
index 75f00965ab15..2702799648e6 100644
@@ -842,8 +842,11 @@ __SYSCALL(__NR_lsm_set_self_attr, sys_lsm_set_self_attr)
 #define __NR_lsm_list_modules 461
 __SYSCALL(__NR_lsm_list_modules, sys_lsm_list_modules)
 
+#define __NR_uprobe 462
+__SYSCALL(__NR_uprobe, sys_uprobe)
+
 #undef __NR_syscalls
-#define __NR_syscalls 462
+#define __NR_syscalls 463
 
 /*
  * 32 bit systems traditionally used different
kernel/events/uprobes.c +14 -4
index 929e98c62965..fefaf4804e1f 100644
@@ -1474,10 +1474,19 @@ static int xol_add_vma(struct mm_struct *mm, struct xol_area *area)
 	return ret;
 }
 
+uprobe_opcode_t* __weak arch_uprobe_trampoline(unsigned long *psize)
+{
+	static uprobe_opcode_t insn = UPROBE_SWBP_INSN;
+
+	*psize = UPROBE_SWBP_INSN_SIZE;
+	return &insn;
+}
+
 static struct xol_area *__create_xol_area(unsigned long vaddr)
 {
 	struct mm_struct *mm = current->mm;
-	uprobe_opcode_t insn = UPROBE_SWBP_INSN;
+	unsigned long insns_size;
+	uprobe_opcode_t *insns;
 	struct xol_area *area;
 
 	area = kmalloc(sizeof(*area), GFP_KERNEL);
@@ -1502,7 +1511,8 @@ static struct xol_area *__create_xol_area(unsigned long vaddr)
 	/* Reserve the 1st slot for get_trampoline_vaddr() */
 	set_bit(0, area->bitmap);
 	atomic_set(&area->slot_count, 1);
-	arch_uprobe_copy_ixol(area->pages[0], 0, &insn, UPROBE_SWBP_INSN_SIZE);
+	insns = arch_uprobe_trampoline(&insns_size);
+	arch_uprobe_copy_ixol(area->pages[0], 0, insns, insns_size);
 
 	if (!xol_add_vma(mm, area))
 		return area;
@@ -2123,7 +2133,7 @@ static struct return_instance *find_next_ret_chain(struct return_instance *ri)
 	return ri;
 }
 
-static void handle_trampoline(struct pt_regs *regs)
+void uprobe_handle_trampoline(struct pt_regs *regs)
 {
 	struct uprobe_task *utask;
 	struct return_instance *ri, *next;
@@ -2188,7 +2198,7 @@ static void handle_swbp(struct pt_regs *regs)
 
 	bp_vaddr = uprobe_get_swbp_addr(regs);
 	if (bp_vaddr == get_trampoline_vaddr())
-		return handle_trampoline(regs);
+		return uprobe_handle_trampoline(regs);
 
 	uprobe = find_active_uprobe(bp_vaddr, &is_swbp);
 	if (!uprobe) {
kernel/sys_ni.c +2 -0
index faad00cce269..ddc954f28317 100644
@@ -391,3 +391,5 @@ COND_SYSCALL(setuid16);
 
 /* restartable sequence */
 COND_SYSCALL(rseq);
+
+COND_SYSCALL(uprobe);
maintainer
#12 lore raw

I forgot everything about the low-level x86_64 code, but...

arch/x86/kernel/uprobes.c static int uprobe_init_insn(struct arch_uprobe *auprobe, struct insn *insn, bool · line 320

Quoted 10 lines · Jiri Olsa

#ifdef CONFIG_X86_64

arch/x86/kernel/uprobes.c +8 -0
+
+asm (
+       ".pushsection .rodata\n"
+       ".global uretprobe_syscall_entry\n"
+       "uretprobe_syscall_entry:\n"
+       "push %rax\n"
+       "mov $462, %rax\n"
+       "syscall\n"

Hmm... I think you need to save/restore more registers clobbered by syscall/entry_SYSCALL_64 ?

arch/x86/kernel/uprobes.c static int uprobe_init_insn(struct arch_uprobe *auprobe, struct insn *insn, bool · line 347

Quoted 13 lines
arch/x86/kernel/uprobes.c +13 -0
+SYSCALL_DEFINE1(uprobe, unsigned long, cmd)
+{
+	struct pt_regs *regs = task_pt_regs(current);
+	unsigned long ax, err;
+
+	/*
+	 * We get invoked from the trampoline that pushed rax
+	 * value on stack, read and restore the value.
+	 */
+	err = copy_from_user((void*) &ax, (void *) regs->sp, sizeof(ax));
+	WARN_ON_ONCE(err);
+
+	regs->ax = ax;

probably not strictly needed, we are going to return ax...

arch/x86/kernel/uprobes.c static int uprobe_init_insn(struct arch_uprobe *auprobe, struct insn *insn, bool · line 348

Quoted 1 line
arch/x86/kernel/uprobes.c +1 -0
+	regs->orig_ax = ax;

This doesn't look right. I think you need

	regs->orig_ax = -1;

Say, to avoid the "Did we come from a system call" checks in arch_do_signal_or_restart() or handle_signal().

Oleg.

author
#13 lore raw
Quoted 4 lines · Oleg Nesterov

I forgot everything about the low-level x86_64 code, but...

Quoted 1 line · Jiri Olsa
Quoted 12 lines

arch/x86/kernel/uprobes.c static int uprobe_init_insn(struct arch_uprobe *auprobe, struct insn *insn, bool · line 320

Quoted 9 lines

#ifdef CONFIG_X86_64

arch/x86/kernel/uprobes.c +8 -0
+
+asm (
+       ".pushsection .rodata\n"
+       ".global uretprobe_syscall_entry\n"
+       "uretprobe_syscall_entry:\n"
+       "push %rax\n"
+       "mov $462, %rax\n"
+       "syscall\n"

Hmm... I think you need to save/restore more registers clobbered by syscall/entry_SYSCALL_64 ?

hum, so the call happens on the function call return, so I thought we should just preserve callee saved registers which seems to be taken care of by the entry_SYSCALL_64 path.. I will double check

Quoted 16 lines
Quoted 4 lines
arch/x86/kernel/uprobes.c +3 -0
+SYSCALL_DEFINE1(uprobe, unsigned long, cmd)
+{
+	struct pt_regs *regs = task_pt_regs(current);

arch/x86/kernel/uprobes.c static int uprobe_init_insn(struct arch_uprobe *auprobe, struct insn *insn, bool · line 347

Quoted 10 lines
arch/x86/kernel/uprobes.c +10 -0
+	unsigned long ax, err;
+
+	/*
+	 * We get invoked from the trampoline that pushed rax
+	 * value on stack, read and restore the value.
+	 */
+	err = copy_from_user((void*) &ax, (void *) regs->sp, sizeof(ax));
+	WARN_ON_ONCE(err);
+
+	regs->ax = ax;

probably not strictly needed, we are going to return ax...

it needs to be there for the bpf program to read proper return value from regs

Quoted 9 lines

arch/x86/kernel/uprobes.c static int uprobe_init_insn(struct arch_uprobe *auprobe, struct insn *insn, bool · line 348

Quoted 2 lines
arch/x86/kernel/uprobes.c +1 -0
+	regs->orig_ax = ax;

This doesn't look right. I think you need

	regs->orig_ax = -1;

Say, to avoid the "Did we come from a system call" checks in arch_do_signal_or_restart() or handle_signal().

ugh right that's probably wrong, I need check on that

thanks,
jirka

maintainer
#14 lore raw
Quoted 19 lines · Jiri Olsa
Quoted 4 lines · Oleg Nesterov

I forgot everything about the low-level x86_64 code, but...

Quoted 1 line · Jiri Olsa
Quoted 12 lines

arch/x86/kernel/uprobes.c static int uprobe_init_insn(struct arch_uprobe *auprobe, struct insn *insn, bool · line 320

Quoted 9 lines

#ifdef CONFIG_X86_64

arch/x86/kernel/uprobes.c +8 -0
+
+asm (
+       ".pushsection .rodata\n"
+       ".global uretprobe_syscall_entry\n"
+       "uretprobe_syscall_entry:\n"
+       "push %rax\n"
+       "mov $462, %rax\n"
+       "syscall\n"

Hmm... I think you need to save/restore more registers clobbered by syscall/entry_SYSCALL_64 ?

Quoted 3 lines

hum, so the call happens on the function call return, so I thought we should just preserve callee saved registers which seems to be taken care of by the entry_SYSCALL_64 path..

Yes, but we do not know if the (ret)probed function obeys the C-calling convention, perhaps it is low level asm code or not a C function.

Quoted 1 line

I will double check

but I won't insist if you think we do not care.

Quoted 7 lines
Quoted 4 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+	regs->ax = ax;

probably not strictly needed, we are going to return ax...

it needs to be there for the bpf program to read proper return value from regs

OK, I see, thanks.

Oleg.

author
#15 lore raw
Quoted 20 lines · Oleg Nesterov
Quoted 19 lines · Jiri Olsa
Quoted 16 lines · Oleg Nesterov

I forgot everything about the low-level x86_64 code, but...

arch/x86/kernel/uprobes.c static int uprobe_init_insn(struct arch_uprobe *auprobe, struct insn *insn, bool · line 320

Quoted 10 lines · Jiri Olsa

#ifdef CONFIG_X86_64

arch/x86/kernel/uprobes.c +8 -0
+
+asm (
+       ".pushsection .rodata\n"
+       ".global uretprobe_syscall_entry\n"
+       "uretprobe_syscall_entry:\n"
+       "push %rax\n"
+       "mov $462, %rax\n"
+       "syscall\n"

Hmm... I think you need to save/restore more registers clobbered by syscall/entry_SYSCALL_64 ?

Quoted 6 lines
Quoted 3 lines

hum, so the call happens on the function call return, so I thought we should just preserve callee saved registers which seems to be taken care of by the entry_SYSCALL_64 path..

Yes, but we do not know if the (ret)probed function obeys the C-calling convention, perhaps it is low level asm code or not a C function.

ah right.. I think we need to make sure all is saved/restored

thanks,
jirka

Quoted 17 lines
Quoted 2 lines

I will double check

but I won't insist if you think we do not care.

Quoted 7 lines
Quoted 4 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+	regs->ax = ax;

probably not strictly needed, we are going to return ax...

it needs to be there for the bpf program to read proper return value from regs

OK, I see, thanks.

Oleg.

#16 lore raw
In reply to Jiri Olsa
Quoted 150 lines · Jiri Olsa
Quoted 125 lines · Jiri Olsa
Quoted 89 lines · Andrii Nakryiko
Quoted 67 lines · Jiri Olsa
Quoted 29 lines · Andrii Nakryiko
Quoted 22 lines · Alexei Starovoitov
Quoted 20 lines · Jiri Olsa
Quoted 17 lines · Andrii Nakryiko
Quoted 11 lines · Jiri Olsa

One of uprobe pain points is having slow execution that involves two traps in worst case scenario or single trap if the original instruction can be emulated. For return uprobes there's one extra trap on top of that.

My current idea on how to make this faster is to follow the optimized kprobes and replace the normal uprobe trap instruction with jump to user space trampoline that:

- executes syscall to call uprobe consumers callbacks

Did you get a chance to measure relative performance of syscall vs int3 interrupt handling? If not, do you think you'll be able to get some numbers by the time the conference starts? This should inform the decision whether it even makes sense to go through all the trouble.

Quoted 4 lines
Quoted 1 line

right, will do that

I believe Yusheng measured syscall vs uprobe performance difference during LPC. iirc it was something like 3x.

Quoted 8 lines

Do you have a link to slides? Was it actual uprobe vs just some fast syscall (not doing BPF program execution) comparison? Or comparing the performance of int3 handling vs equivalent syscall handling.

I suspect it's the former, and so probably not that representative. I'm curious about the performance of going userspace->kernel->userspace through int3 vs syscall (all other things being equal).

I have a simple test [1] comparing:
- uprobe with 2 traps
- uprobe with 1 trap
- syscall executing uprobe

the syscall takes uprobe address as argument, finds the uprobe and executes its consumers, which should be comparable to what the trampoline will do

test does same amount of loops triggering each uprobe type and measures the time it took

  # ./test_progs -t uprobe_syscall_bench -v
  bpf_testmod.ko is already unloaded.
  Loading bpf_testmod.ko...
  Successfully loaded bpf_testmod.ko.
  test_bench_1:PASS:uprobe_bench__open_and_load 0 nsec
  test_bench_1:PASS:uprobe_bench__attach 0 nsec
  test_bench_1:PASS:uprobe1_cnt 0 nsec
  test_bench_1:PASS:syscalls_uprobe1_cnt 0 nsec
  test_bench_1:PASS:uprobe2_cnt 0 nsec
  test_bench_1: uprobes (1 trap) in  36.439s
  test_bench_1: uprobes (2 trap) in  91.960s
  test_bench_1: syscalls         in  17.872s
  #395/1   uprobe_syscall_bench/bench_1:OK
  #395     uprobe_syscall_bench:OK
  Summary: 1/1 PASSED, 0 SKIPPED, 0 FAILED
Quoted 3 lines

syscall uprobe execution seems to be ~2x faster than 1 trap uprobe and ~5x faster than 2 traps uprobe

Thanks for running benchmarks! I quickly looked at the selftest and noticed this:

(quoted diff) +10 -0
+/*
+ * Assuming following prolog:
+ *
+ * 6984ac:       55                      push   %rbp
+ * 6984ad:       48 89 e5                mov    %rsp,%rbp
+ */
+noinline void uprobe2_bench_trigger(void)
+{
+        asm volatile ("");
+}

This actually will be optimized out to just ret in -O2 mode (make RELEASE=1 for selftests):

Quoted 6 lines

00000000005a0ce0 <uprobe2_bench_trigger>:
5a0ce0: c3 retq
5a0ce1: 66 66 2e 0f 1f 84 00 00 00 00 00 nopw %cs:(%rax,%rax)
5a0cec: 0f 1f 40 00 nopl (%rax)

So be careful with that.

right, I did not mean for this to be checked in, just wanted to get the numbers quickly

Quoted 10 lines

Also, I just updated our existing set of uprobe benchmarks (see [0]), do you mind adding your syscall-based one as another one there and running all of them and sharing the numbers with us? Very curious to see both absolute and relative numbers from that benchmark. (and please do build with RELEASE=1)

You should be able to just run benchs/run_bench_uprobes.sh (also don't forget to add your syscall-based benchmark to the list of benchmarks in that shell script).

yes, saw it and was going to run/compare it.. it's good idea to add the syscall one and get all numbers together, will do that

Quoted 10 lines

Thank you!

BTW, while I think patching multiple instructions for syscall-based uprobe is going to be extremely tricky, I think at least u*ret*probe's int3 can be pretty easily optimized away with syscall, given that the kernel controls code generation there. If anything, it will get the uretprobe case a bit closer to the performance of uprobe. Give it some thought.

Quoted 4 lines

hm, right.. the trampoline is there already, but at the moment is global and used by all uretprobes.. and int3 code moves userspace (changes rip) to the original return address.. maybe we can do that through syscall as well

it seems like good idea, I tried change below (use syscall on return trampoline) and got some speedup:

current:

  base           :   15.817 ± 0.009M/s
  uprobe-nop     :    2.901 ± 0.000M/s
  uprobe-push    :    2.743 ± 0.002M/s
  uprobe-ret     :    1.089 ± 0.001M/s
  uretprobe-nop  :    1.448 ± 0.001M/s

uretprobe-push : 1.407 ± 0.001M/s
uretprobe-ret : 0.792 ± 0.001M/s

with syscall:

  base           :   15.831 ± 0.026M/s
  uprobe-nop     :    2.904 ± 0.001M/s
  uprobe-push    :    2.764 ± 0.002M/s
  uprobe-ret     :    1.082 ± 0.001M/s
  uretprobe-nop  :    1.785 ± 0.000M/s
Quoted 6 lines

uretprobe-push : 1.733 ± 0.001M/s
uretprobe-ret : 0.885 ± 0.004M/s

~23% for nop/push (emulated) cases, ~11% for ret (sstep) case

jirka

heh, I tried this as well over weekend, though I cut few more corners (see diff below, I didn't add saving/restoring rax, though that would be required, of course). My test machine is (way) slower, though, so I got a slightly different numbers (up to 15%):

### baseline

uprobe-base    :   79.462 ± 0.058M/s
base           :    2.920 ± 0.004M/s
uprobe-nop     :    1.093 ± 0.001M/s
uprobe-push    :    1.066 ± 0.001M/s
uprobe-ret     :    0.480 ± 0.001M/s
uretprobe-nop  :    0.555 ± 0.000M/s
uretprobe-push :    0.549 ± 0.000M/s
uretprobe-ret  :    0.338 ± 0.000M/s

### uretprobe syscall (vs baseline)

uprobe-base    :   79.488 ± 0.033M/s
base           :    2.917 ± 0.003M/s
uprobe-nop     :    1.095 ± 0.001M/s
uprobe-push    :    1.058 ± 0.000M/s
uprobe-ret     :    0.483 ± 0.000M/s
uretprobe-nop  :    0.638 ± 0.000M/s (+15%)
uretprobe-push :    0.627 ± 0.000M/s (+14.2%)
uretprobe-ret  :    0.366 ± 0.000M/s (+8.3%)

Either way, yes, we should implement this. Are you planning to send an official patch some time soon? I'm working on other small improvements in uprobe/uretprobe, I'll probably send the first patches today/tomorrow, but they shouldn't interfere with this uretprobe code path.

See also a few comments on your code below.

My hacky changes:

commit dcf59baa5ad8ea4edb86ff1558eb1dcc28fcc7c0

Author: Andrii Nakryiko <[email protected]>
Date:   Sat Mar 9 20:16:02 2024 -0800

[WIP] uprobes: implement uretprobe through syscall

(quoted diff) +0 -0

b/arch/x86/entry/syscalls/syscall_32.tbl
index 5f8591ce7f25..0207dfb5018d 100644

arch/x86/entry/syscalls/syscall_32.tbl +1 -0
@@ -466,3 +466,4 @@
 459    i386    lsm_get_self_attr       sys_lsm_get_self_attr
 460    i386    lsm_set_self_attr       sys_lsm_set_self_attr
 461    i386    lsm_list_modules        sys_lsm_list_modules
+462    i386    trace                   sys_trace
(quoted diff) +0 -0

b/arch/x86/entry/syscalls/syscall_64.tbl
index 7e8d46f4147f..c16599e15bbb 100644

arch/x86/entry/syscalls/syscall_64.tbl +1 -0
@@ -383,6 +383,7 @@
 459    common  lsm_get_self_attr       sys_lsm_get_self_attr
 460    common  lsm_set_self_attr       sys_lsm_set_self_attr
 461    common  lsm_list_modules        sys_lsm_list_modules
+462    common  trace                   sys_trace
 
 #
 # Due to a historical design error, certain syscalls are numbered differently
include/linux/syscalls.h +0 -0
index 77eb9b0e7685..f2863c9fab3d 100644
@@ -965,6 +965,8 @@ asmlinkage long sys_lsm_set_self_attr(unsigned int

attr, struct lsm_ctx *ctx,

                                      size_t size, __u32 flags);

asmlinkage long sys_lsm_list_modules(u64 *ids, size_t *size, u32 flags);

+asmlinkage long sys_trace(int cmd);
+
/*
* Architecture-specific system calls
*/

(quoted diff) +0 -0

b/include/uapi/asm-generic/unistd.h
index 75f00965ab15..2f66eb8b068e 100644

include/uapi/asm-generic/unistd.h +4 -1
@@ -842,8 +842,11 @@ __SYSCALL(__NR_lsm_set_self_attr, sys_lsm_set_self_attr)
 #define __NR_lsm_list_modules 461
 __SYSCALL(__NR_lsm_list_modules, sys_lsm_list_modules)
 
+#define __NR_trace 462
+__SYSCALL(__NR_TRACE, sys_trace)
+
 #undef __NR_syscalls
-#define __NR_syscalls 462
+#define __NR_syscalls 463
 
 /*
  * 32 bit systems traditionally used different
kernel/events/uprobes.c +1 -0
index 56a460719628..c6fc15bdbffc 100644
@@ -28,6 +28,7 @@
 #include <linux/khugepaged.h>
 
 #include <linux/uprobes.h>
+#include <linux/syscalls.h>
 
 #define UINSNS_PER_PAGE

(PAGE_SIZE/UPROBE_XOL_SLOT_BYTES)
#define MAX_UPROBE_XOL_SLOTS UINSNS_PER_PAGE
@@ -1488,8 +1489,15 @@ static int xol_add_vma(struct mm_struct *mm, struct xol_area *area)
static struct xol_area *__create_xol_area(unsigned long vaddr)

 {
        struct mm_struct *mm = current->mm;
-       uprobe_opcode_t insn = UPROBE_SWBP_INSN;
+       //uprobe_opcode_t insn = UPROBE_SWBP_INSN;
        struct xol_area *area;
+       char uret_syscall_patch[] = {
+               /* mov rax, __NR_trace */
+               0x48, 0xC7, 0xC0, 0xCE, 0x01, 0x00, 0x00,
+               /* syscall */
+               0x0F, 0x05,

+ };
+ const int URET_SYSCALL_INSNS_SIZE = ARRAY_SIZE(uret_syscall_patch);

        area = kmalloc(sizeof(*area), GFP_KERNEL);
        if (unlikely(!area))
@@ -1513,7 +1521,8 @@ static struct xol_area
*__create_xol_area(unsigned long vaddr)
        /* Reserve the 1st slot for get_trampoline_vaddr() */
        set_bit(0, area->bitmap);
        atomic_set(&area->slot_count, 1);

- arch_uprobe_copy_ixol(area->pages[0], 0, &insn, UPROBE_SWBP_INSN_SIZE);
+ arch_uprobe_copy_ixol(area->pages[0], 0, &uret_syscall_patch,
URET_SYSCALL_INSNS_SIZE);
+ //arch_uprobe_copy_ixol(area->pages[0], 0, &insn,
UPROBE_SWBP_INSN_SIZE);

        if (!xol_add_vma(mm, area))
                return area;

@@ -2366,3 +2375,16 @@ void __init uprobes_init(void)

        BUG_ON(register_die_notifier(&uprobe_exception_nb));

}
+
+SYSCALL_DEFINE0(trace)
+{
+ struct pt_regs *regs = current_pt_regs();
+
+ //printk("BEFORE UTRACE!!!\n");
+
+ handle_trampoline(regs);
+
+ //printk("AFTER UTRACE!!!\n");
+
+ return 0;
+}

kernel/sys_ni.c +1 -0
index faad00cce269..4a3bc957dd43 100644
@@ -174,6 +174,7 @@ COND_SYSCALL_COMPAT(fadvise64_64);
 COND_SYSCALL(lsm_get_self_attr);
 COND_SYSCALL(lsm_set_self_attr);
 COND_SYSCALL(lsm_list_modules);
+COND_SYSCALL(trace);
 
 /* CONFIG_MMU only */
 COND_SYSCALL(swapon);

arch/x86/entry/syscalls/syscall_64.tbl line 387

Quoted 13 lines

---

arch/x86/entry/syscalls/syscall_64.tbl +1 -0
index 7e8d46f4147f..fa5f8a058bc2 100644
@@ -383,6 +383,7 @@
 459    common  lsm_get_self_attr       sys_lsm_get_self_attr
 460    common  lsm_set_self_attr       sys_lsm_set_self_attr
 461    common  lsm_list_modules        sys_lsm_list_modules
+462    64      uprobe                  sys_uprobe
 

we should call it "uretprobe", "uprobe" will be a separate thing with different logic.

I went with generic "trace", but realized that it would be better to have separate more targeted "special/internal" syscalls (where, if necessary, extra arguments would be passed through stack to avoid storing/restoring user-space registers). We have rg_sigreturn precedent which explicitly states that userspace shouldn't use it and shouldn't rely on any specific arguments conventions.

[...]

include/linux/uprobes.h extern bool arch_uretprobe_is_alive(struct return_instance *ret, enum rp_check c · line 142

Quoted 12 lines
arch/x86/entry/syscalls/syscall_64.tbl +0 -0
 /*
  * Deprecated system calls which are still defined in
include/linux/uprobes.h +2 -0
index f46e0ca0169c..9ef244c8ff19 100644
@@ -138,6 +138,8 @@ extern bool arch_uretprobe_is_alive(struct return_instance *ret, enum rp_check c
 extern bool arch_uprobe_ignore(struct arch_uprobe *aup, struct pt_regs *regs);
 extern void arch_uprobe_copy_ixol(struct page *page, unsigned long vaddr,
                                         void *src, unsigned long len);
+extern void uprobe_handle_trampoline(struct pt_regs *regs);
+uprobe_opcode_t* arch_uprobe_trampoline(unsigned long *psize);

just void * here? it can be a sequence of instructions now

Quoted 13 lines
include/linux/uprobes.h +0 -0
 #else /* !CONFIG_UPROBES */
 struct uprobes_state {
 };
include/uapi/asm-generic/unistd.h +2 -0
index 75f00965ab15..2702799648e6 100644
@@ -842,8 +842,11 @@ __SYSCALL(__NR_lsm_set_self_attr, sys_lsm_set_self_attr)
 #define __NR_lsm_list_modules 461
 __SYSCALL(__NR_lsm_list_modules, sys_lsm_list_modules)
 
+#define __NR_uprobe 462
+__SYSCALL(__NR_uprobe, sys_uprobe)

[...]

author
#17 lore raw
Quoted 154 lines · Andrii Nakryiko
Quoted 153 lines · Jiri Olsa
Quoted 125 lines · Jiri Olsa
Quoted 89 lines · Andrii Nakryiko
Quoted 67 lines · Jiri Olsa
Quoted 29 lines · Andrii Nakryiko
Quoted 22 lines · Alexei Starovoitov
Quoted 20 lines · Jiri Olsa
Quoted 17 lines · Andrii Nakryiko
Quoted 11 lines · Jiri Olsa

One of uprobe pain points is having slow execution that involves two traps in worst case scenario or single trap if the original instruction can be emulated. For return uprobes there's one extra trap on top of that.

My current idea on how to make this faster is to follow the optimized kprobes and replace the normal uprobe trap instruction with jump to user space trampoline that:

- executes syscall to call uprobe consumers callbacks

Did you get a chance to measure relative performance of syscall vs int3 interrupt handling? If not, do you think you'll be able to get some numbers by the time the conference starts? This should inform the decision whether it even makes sense to go through all the trouble.

Quoted 4 lines
Quoted 1 line

right, will do that

I believe Yusheng measured syscall vs uprobe performance difference during LPC. iirc it was something like 3x.

Quoted 8 lines

Do you have a link to slides? Was it actual uprobe vs just some fast syscall (not doing BPF program execution) comparison? Or comparing the performance of int3 handling vs equivalent syscall handling.

I suspect it's the former, and so probably not that representative. I'm curious about the performance of going userspace->kernel->userspace through int3 vs syscall (all other things being equal).

I have a simple test [1] comparing:
- uprobe with 2 traps
- uprobe with 1 trap
- syscall executing uprobe

the syscall takes uprobe address as argument, finds the uprobe and executes its consumers, which should be comparable to what the trampoline will do

test does same amount of loops triggering each uprobe type and measures the time it took

  # ./test_progs -t uprobe_syscall_bench -v
  bpf_testmod.ko is already unloaded.
  Loading bpf_testmod.ko...
  Successfully loaded bpf_testmod.ko.
  test_bench_1:PASS:uprobe_bench__open_and_load 0 nsec
  test_bench_1:PASS:uprobe_bench__attach 0 nsec
  test_bench_1:PASS:uprobe1_cnt 0 nsec
  test_bench_1:PASS:syscalls_uprobe1_cnt 0 nsec
  test_bench_1:PASS:uprobe2_cnt 0 nsec
  test_bench_1: uprobes (1 trap) in  36.439s
  test_bench_1: uprobes (2 trap) in  91.960s
  test_bench_1: syscalls         in  17.872s
  #395/1   uprobe_syscall_bench/bench_1:OK
  #395     uprobe_syscall_bench:OK
  Summary: 1/1 PASSED, 0 SKIPPED, 0 FAILED
Quoted 3 lines

syscall uprobe execution seems to be ~2x faster than 1 trap uprobe and ~5x faster than 2 traps uprobe

Thanks for running benchmarks! I quickly looked at the selftest and noticed this:

(quoted diff) +10 -0
+/*
+ * Assuming following prolog:
+ *
+ * 6984ac:       55                      push   %rbp
+ * 6984ad:       48 89 e5                mov    %rsp,%rbp
+ */
+noinline void uprobe2_bench_trigger(void)
+{
+        asm volatile ("");
+}

This actually will be optimized out to just ret in -O2 mode (make RELEASE=1 for selftests):

Quoted 6 lines

00000000005a0ce0 <uprobe2_bench_trigger>:
5a0ce0: c3 retq
5a0ce1: 66 66 2e 0f 1f 84 00 00 00 00 00 nopw %cs:(%rax,%rax)
5a0cec: 0f 1f 40 00 nopl (%rax)

So be careful with that.

right, I did not mean for this to be checked in, just wanted to get the numbers quickly

Quoted 10 lines

Also, I just updated our existing set of uprobe benchmarks (see [0]), do you mind adding your syscall-based one as another one there and running all of them and sharing the numbers with us? Very curious to see both absolute and relative numbers from that benchmark. (and please do build with RELEASE=1)

You should be able to just run benchs/run_bench_uprobes.sh (also don't forget to add your syscall-based benchmark to the list of benchmarks in that shell script).

yes, saw it and was going to run/compare it.. it's good idea to add the syscall one and get all numbers together, will do that

Quoted 10 lines

Thank you!

BTW, while I think patching multiple instructions for syscall-based uprobe is going to be extremely tricky, I think at least u*ret*probe's int3 can be pretty easily optimized away with syscall, given that the kernel controls code generation there. If anything, it will get the uretprobe case a bit closer to the performance of uprobe. Give it some thought.

Quoted 4 lines

hm, right.. the trampoline is there already, but at the moment is global and used by all uretprobes.. and int3 code moves userspace (changes rip) to the original return address.. maybe we can do that through syscall as well

it seems like good idea, I tried change below (use syscall on return trampoline) and got some speedup:

current:

  base           :   15.817 ± 0.009M/s
  uprobe-nop     :    2.901 ± 0.000M/s
  uprobe-push    :    2.743 ± 0.002M/s
  uprobe-ret     :    1.089 ± 0.001M/s
  uretprobe-nop  :    1.448 ± 0.001M/s

uretprobe-push : 1.407 ± 0.001M/s
uretprobe-ret : 0.792 ± 0.001M/s

with syscall:

  base           :   15.831 ± 0.026M/s
  uprobe-nop     :    2.904 ± 0.001M/s
  uprobe-push    :    2.764 ± 0.002M/s
  uprobe-ret     :    1.082 ± 0.001M/s
  uretprobe-nop  :    1.785 ± 0.000M/s

uretprobe-push : 1.733 ± 0.001M/s
uretprobe-ret : 0.885 ± 0.004M/s

Quoted 8 lines
Quoted 3 lines

~23% for nop/push (emulated) cases, ~11% for ret (sstep) case

jirka

heh, I tried this as well over weekend, though I cut few more corners (see diff below, I didn't add saving/restoring rax, though that would be required, of course). My test machine is (way) slower, though, so I got a slightly different numbers (up to 15%):

nice :-) btw I just checked on another slower amd server and it's ~10% in all 3 cases, my previous results are from intel machine.. I guess the hw trap behaviour/speed makes this not proportional across archs

Quoted 22 lines

### baseline

uprobe-base    :   79.462 ± 0.058M/s
base           :    2.920 ± 0.004M/s
uprobe-nop     :    1.093 ± 0.001M/s
uprobe-push    :    1.066 ± 0.001M/s
uprobe-ret     :    0.480 ± 0.001M/s
uretprobe-nop  :    0.555 ± 0.000M/s
uretprobe-push :    0.549 ± 0.000M/s
uretprobe-ret  :    0.338 ± 0.000M/s

### uretprobe syscall (vs baseline)

uprobe-base    :   79.488 ± 0.033M/s
base           :    2.917 ± 0.003M/s
uprobe-nop     :    1.095 ± 0.001M/s
uprobe-push    :    1.058 ± 0.000M/s
uprobe-ret     :    0.483 ± 0.000M/s
uretprobe-nop  :    0.638 ± 0.000M/s (+15%)
uretprobe-push :    0.627 ± 0.000M/s (+14.2%)
uretprobe-ret  :    0.366 ± 0.000M/s (+8.3%)
Quoted 5 lines

Either way, yes, we should implement this. Are you planning to send an official patch some time soon? I'm working on other small improvements in uprobe/uretprobe, I'll probably send the first patches today/tomorrow, but they shouldn't interfere with this uretprobe code path.

yes, wanted to finish/post it this week

SNIP

Quoted 15 lines

arch/x86/entry/syscalls/syscall_64.tbl line 387

Quoted 11 lines

---

arch/x86/entry/syscalls/syscall_64.tbl +1 -0
index 7e8d46f4147f..fa5f8a058bc2 100644
@@ -383,6 +383,7 @@
 459    common  lsm_get_self_attr       sys_lsm_get_self_attr
 460    common  lsm_set_self_attr       sys_lsm_set_self_attr
 461    common  lsm_list_modules        sys_lsm_list_modules
+462    64      uprobe                  sys_uprobe
 

we should call it "uretprobe", "uprobe" will be a separate thing with different logic.

Quoted 6 lines

I went with generic "trace", but realized that it would be better to have separate more targeted "special/internal" syscalls (where, if necessary, extra arguments would be passed through stack to avoid storing/restoring user-space registers). We have rg_sigreturn precedent which explicitly states that userspace shouldn't use it and shouldn't rely on any specific arguments conventions.

somehow I thought of syscalls as of scare resource and wanted to add arguments/commands to the uprobe syscalls.. but having uretprobe dedicated syscall makes things easier

Quoted 17 lines

[...]

include/linux/uprobes.h extern bool arch_uretprobe_is_alive(struct return_instance *ret, enum rp_check c · line 142

Quoted 12 lines

/*
* Deprecated system calls which are still defined in

include/linux/uprobes.h +2 -0
index f46e0ca0169c..9ef244c8ff19 100644
@@ -138,6 +138,8 @@ extern bool arch_uretprobe_is_alive(struct return_instance *ret, enum rp_check c
 extern bool arch_uprobe_ignore(struct arch_uprobe *aup, struct pt_regs *regs);
 extern void arch_uprobe_copy_ixol(struct page *page, unsigned long vaddr,
                                         void *src, unsigned long len);
+extern void uprobe_handle_trampoline(struct pt_regs *regs);
+uprobe_opcode_t* arch_uprobe_trampoline(unsigned long *psize);

just void * here? it can be a sequence of instructions now

hm, it's pointer to u8, which should be fine no? is there benefit to have void* in here instead?

thanks,
jirka

#18 lore raw
Quoted 191 lines · Jiri Olsa
Quoted 154 lines · Andrii Nakryiko
Quoted 153 lines · Jiri Olsa
Quoted 125 lines · Jiri Olsa
Quoted 89 lines · Andrii Nakryiko
Quoted 67 lines · Jiri Olsa
Quoted 29 lines · Andrii Nakryiko
Quoted 22 lines · Alexei Starovoitov
Quoted 20 lines · Jiri Olsa
Quoted 17 lines · Andrii Nakryiko
Quoted 11 lines · Jiri Olsa

One of uprobe pain points is having slow execution that involves two traps in worst case scenario or single trap if the original instruction can be emulated. For return uprobes there's one extra trap on top of that.

My current idea on how to make this faster is to follow the optimized kprobes and replace the normal uprobe trap instruction with jump to user space trampoline that:

- executes syscall to call uprobe consumers callbacks

Did you get a chance to measure relative performance of syscall vs int3 interrupt handling? If not, do you think you'll be able to get some numbers by the time the conference starts? This should inform the decision whether it even makes sense to go through all the trouble.

Quoted 4 lines
Quoted 1 line

right, will do that

I believe Yusheng measured syscall vs uprobe performance difference during LPC. iirc it was something like 3x.

Quoted 8 lines

Do you have a link to slides? Was it actual uprobe vs just some fast syscall (not doing BPF program execution) comparison? Or comparing the performance of int3 handling vs equivalent syscall handling.

I suspect it's the former, and so probably not that representative. I'm curious about the performance of going userspace->kernel->userspace through int3 vs syscall (all other things being equal).

I have a simple test [1] comparing:
- uprobe with 2 traps
- uprobe with 1 trap
- syscall executing uprobe

the syscall takes uprobe address as argument, finds the uprobe and executes its consumers, which should be comparable to what the trampoline will do

test does same amount of loops triggering each uprobe type and measures the time it took

  # ./test_progs -t uprobe_syscall_bench -v
  bpf_testmod.ko is already unloaded.
  Loading bpf_testmod.ko...
  Successfully loaded bpf_testmod.ko.
  test_bench_1:PASS:uprobe_bench__open_and_load 0 nsec
  test_bench_1:PASS:uprobe_bench__attach 0 nsec
  test_bench_1:PASS:uprobe1_cnt 0 nsec
  test_bench_1:PASS:syscalls_uprobe1_cnt 0 nsec
  test_bench_1:PASS:uprobe2_cnt 0 nsec
  test_bench_1: uprobes (1 trap) in  36.439s
  test_bench_1: uprobes (2 trap) in  91.960s
  test_bench_1: syscalls         in  17.872s
  #395/1   uprobe_syscall_bench/bench_1:OK
  #395     uprobe_syscall_bench:OK
  Summary: 1/1 PASSED, 0 SKIPPED, 0 FAILED
Quoted 3 lines

syscall uprobe execution seems to be ~2x faster than 1 trap uprobe and ~5x faster than 2 traps uprobe

Thanks for running benchmarks! I quickly looked at the selftest and noticed this:

(quoted diff) +10 -0
+/*
+ * Assuming following prolog:
+ *
+ * 6984ac:       55                      push   %rbp
+ * 6984ad:       48 89 e5                mov    %rsp,%rbp
+ */
+noinline void uprobe2_bench_trigger(void)
+{
+        asm volatile ("");
+}

This actually will be optimized out to just ret in -O2 mode (make RELEASE=1 for selftests):

Quoted 6 lines

00000000005a0ce0 <uprobe2_bench_trigger>:
5a0ce0: c3 retq
5a0ce1: 66 66 2e 0f 1f 84 00 00 00 00 00 nopw %cs:(%rax,%rax)
5a0cec: 0f 1f 40 00 nopl (%rax)

So be careful with that.

right, I did not mean for this to be checked in, just wanted to get the numbers quickly

Quoted 10 lines

Also, I just updated our existing set of uprobe benchmarks (see [0]), do you mind adding your syscall-based one as another one there and running all of them and sharing the numbers with us? Very curious to see both absolute and relative numbers from that benchmark. (and please do build with RELEASE=1)

You should be able to just run benchs/run_bench_uprobes.sh (also don't forget to add your syscall-based benchmark to the list of benchmarks in that shell script).

yes, saw it and was going to run/compare it.. it's good idea to add the syscall one and get all numbers together, will do that

Quoted 10 lines

Thank you!

BTW, while I think patching multiple instructions for syscall-based uprobe is going to be extremely tricky, I think at least u*ret*probe's int3 can be pretty easily optimized away with syscall, given that the kernel controls code generation there. If anything, it will get the uretprobe case a bit closer to the performance of uprobe. Give it some thought.

Quoted 4 lines

hm, right.. the trampoline is there already, but at the moment is global and used by all uretprobes.. and int3 code moves userspace (changes rip) to the original return address.. maybe we can do that through syscall as well

it seems like good idea, I tried change below (use syscall on return trampoline) and got some speedup:

current:

  base           :   15.817 ± 0.009M/s
  uprobe-nop     :    2.901 ± 0.000M/s
  uprobe-push    :    2.743 ± 0.002M/s
  uprobe-ret     :    1.089 ± 0.001M/s
  uretprobe-nop  :    1.448 ± 0.001M/s

uretprobe-push : 1.407 ± 0.001M/s
uretprobe-ret : 0.792 ± 0.001M/s

with syscall:

  base           :   15.831 ± 0.026M/s
  uprobe-nop     :    2.904 ± 0.001M/s
  uprobe-push    :    2.764 ± 0.002M/s
  uprobe-ret     :    1.082 ± 0.001M/s
  uretprobe-nop  :    1.785 ± 0.000M/s

uretprobe-push : 1.733 ± 0.001M/s
uretprobe-ret : 0.885 ± 0.004M/s

Quoted 8 lines
Quoted 3 lines

~23% for nop/push (emulated) cases, ~11% for ret (sstep) case

jirka

heh, I tried this as well over weekend, though I cut few more corners (see diff below, I didn't add saving/restoring rax, though that would be required, of course). My test machine is (way) slower, though, so I got a slightly different numbers (up to 15%):

nice :-) btw I just checked on another slower amd server and it's ~10% in all 3 cases, my previous results are from intel machine.. I guess the hw trap behaviour/speed makes this not proportional across archs

Quoted 22 lines

### baseline

uprobe-base    :   79.462 ± 0.058M/s
base           :    2.920 ± 0.004M/s
uprobe-nop     :    1.093 ± 0.001M/s
uprobe-push    :    1.066 ± 0.001M/s
uprobe-ret     :    0.480 ± 0.001M/s
uretprobe-nop  :    0.555 ± 0.000M/s
uretprobe-push :    0.549 ± 0.000M/s
uretprobe-ret  :    0.338 ± 0.000M/s

### uretprobe syscall (vs baseline)

uprobe-base    :   79.488 ± 0.033M/s
base           :    2.917 ± 0.003M/s
uprobe-nop     :    1.095 ± 0.001M/s
uprobe-push    :    1.058 ± 0.000M/s
uprobe-ret     :    0.483 ± 0.000M/s
uretprobe-nop  :    0.638 ± 0.000M/s (+15%)
uretprobe-push :    0.627 ± 0.000M/s (+14.2%)
uretprobe-ret  :    0.366 ± 0.000M/s (+8.3%)
Quoted 8 lines
Quoted 5 lines

Either way, yes, we should implement this. Are you planning to send an official patch some time soon? I'm working on other small improvements in uprobe/uretprobe, I'll probably send the first patches today/tomorrow, but they shouldn't interfere with this uretprobe code path.

yes, wanted to finish/post it this week

great, looking forward, we can use these speeds up for uretprobe in our production

Quoted 24 lines

SNIP

Quoted 15 lines

arch/x86/entry/syscalls/syscall_64.tbl line 387

Quoted 11 lines

---

arch/x86/entry/syscalls/syscall_64.tbl +1 -0
index 7e8d46f4147f..fa5f8a058bc2 100644
@@ -383,6 +383,7 @@
 459    common  lsm_get_self_attr       sys_lsm_get_self_attr
 460    common  lsm_set_self_attr       sys_lsm_set_self_attr
 461    common  lsm_list_modules        sys_lsm_list_modules
+462    64      uprobe                  sys_uprobe
 

we should call it "uretprobe", "uprobe" will be a separate thing with different logic.

Quoted 6 lines

I went with generic "trace", but realized that it would be better to have separate more targeted "special/internal" syscalls (where, if necessary, extra arguments would be passed through stack to avoid storing/restoring user-space registers). We have rg_sigreturn precedent which explicitly states that userspace shouldn't use it and shouldn't rely on any specific arguments conventions.

Quoted 3 lines

somehow I thought of syscalls as of scare resource and wanted to add arguments/commands to the uprobe syscalls.. but having uretprobe dedicated syscall makes things easier

given we are at 462 already, not sure I believe it's scarce :) but it's also a performance aspect, not using any arguments means we can avoid saving/restoring user-space registers, so it makes sense to have 2-3 dedicated uprobe/uretprobe syscalls vs 1 more generic on (IMO), at least from performance POV.

Quoted 19 lines
Quoted 4 lines

[...]

Quoted 14 lines

include/linux/uprobes.h extern bool arch_uretprobe_is_alive(struct return_instance *ret, enum rp_check c · line 142

Quoted 12 lines

/*
* Deprecated system calls which are still defined in

include/linux/uprobes.h +2 -0
index f46e0ca0169c..9ef244c8ff19 100644
@@ -138,6 +138,8 @@ extern bool arch_uretprobe_is_alive(struct return_instance *ret, enum rp_check c
 extern bool arch_uprobe_ignore(struct arch_uprobe *aup, struct pt_regs *regs);
 extern void arch_uprobe_copy_ixol(struct page *page, unsigned long vaddr,
                                         void *src, unsigned long len);
+extern void uprobe_handle_trampoline(struct pt_regs *regs);
+uprobe_opcode_t* arch_uprobe_trampoline(unsigned long *psize);

just void * here? it can be a sequence of instructions now

Quoted 3 lines

hm, it's pointer to u8, which should be fine no? is there benefit to have void* in here instead?

Quick grepping initially brought up typedef u32 uprobe_opcode_t;, but that's for non-x86 architectures. I don't think it matters all that much (in my mind it's just a generated code blob, so just void * memory, we don't have to look at its contents).

Quoted 2 lines

thanks,
jirka

author
#19 lore raw
In reply to Alexei Starovoitov
Quoted 21 lines · Alexei Starovoitov
Quoted 20 lines · Jiri Olsa
Quoted 17 lines · Andrii Nakryiko
Quoted 11 lines · Jiri Olsa

One of uprobe pain points is having slow execution that involves two traps in worst case scenario or single trap if the original instruction can be emulated. For return uprobes there's one extra trap on top of that.

My current idea on how to make this faster is to follow the optimized kprobes and replace the normal uprobe trap instruction with jump to user space trampoline that:

- executes syscall to call uprobe consumers callbacks

Did you get a chance to measure relative performance of syscall vs int3 interrupt handling? If not, do you think you'll be able to get some numbers by the time the conference starts? This should inform the decision whether it even makes sense to go through all the trouble.

Quoted 8 lines
Quoted 1 line

right, will do that

I believe Yusheng measured syscall vs uprobe performance difference during LPC. iirc it was something like 3x. Certainly necessary to have a benchmark. selftests/bpf/bench has one for uprobe. Probably should extend with sys_bpf.

ok, did not know there was uprobe benchmark, will check

Quoted 6 lines

Regarding:

Quoted 1 line

replace the normal uprobe trap instruction with jump to

user space trampoline

it should probably be a call to trampoline instead of a jump. Unless you plan to generate a different trampoline for every location ?

I wanted to store the ip of the uprobe as argument for the syscall, but the call instruction will push return address on stack and we can use it to get uprobe's address.. great

Quoted 3 lines

Also how would you pick a space for a trampoline in the target process ? Analyze /proc/pid/maps and look for gaps in executable sections?

As Andrii mentioned in other response there's already one page mapped as '[uprobes]' mapping, it's used as trampoline for return uprobes (contains just int3 instruction) and as buffers to hold the original instruction for the single step execution

I think if we endup with just single trampoline we can just use some of the space from that page, our trampoline should not be big

Quoted 5 lines

We can start simple with a USDT that uses nop5 instead of nop1 and explicit single trampoline for all USDT locations that saves all (callee and caller saved) registers and then does sys_bpf with a new cmd.

ah, I did not realize USDTs are like that, will check, good idea

Quoted 4 lines

To replace nop5 with a call to trampoline we can use text_poke_bp
approach: replace 1st byte with int3, replace 2-5 with target addr, replace 1st byte to make an actual call insn.

I'm bit in the dark in here, but uprobe_write_opcode stores the int3 byte by allocating new page, copying the contents of the old page over and updating it with int3 byte.. then calls __replace_page to put new page in place

should that be enough also for 5 bytes update? the cpu executing that exact page will page fault and get the new updated page? I discussed with Oleg and got this understanding, I might be wrong

hm what if the cpu is just executing the address in the middle of the uprobe's original instructions and the page gets updated.. I need to check more on this ;-)

Quoted 4 lines

Once patched there will be no simulation of insns or kernel traps. Just normal user code that calls into trampoline, that calls sys_bpf, and returns back.

I saw this as generic uprobe enhancement, should it be sys_bpf syscall, not a some generic one? we will call all the uprobe's handlers/consumers

thanks,
jirka

#20 lore raw
Quoted 7 lines · Jiri Olsa

I'm bit in the dark in here, but uprobe_write_opcode stores the int3 byte by allocating new page, copying the contents of the old page over and updating it with int3 byte.. then calls __replace_page to put new page in place

Quoted 7 lines

should that be enough also for 5 bytes update? the cpu executing that exact page will page fault and get the new updated page? I discussed with Oleg and got this understanding, I might be wrong

hm what if the cpu is just executing the address in the middle of the uprobe's original instructions and the page gets updated.. I need to check more on this ;-)

I suspect it's all working fine already. Only x86 is using single byte uprobe.
All other archs are using 2 or 4 byte.
So replacing an insn or two with a call should work.

Quoted 2 lines

I saw this as generic uprobe enhancement, should it be sys_bpf syscall, not a some generic one? we will call all the uprobe's handlers/consumers

yeah. If we can make all uprobes faster without relying on nop5 usdt then it's certainly better.
But if "replace any insn" turns out to be too complex
we can limit it to replacing nop5 or replacing simple insns in the prologue like push, mov.

maintainer
#21 lore raw
Quoted 4 lines · Alexei Starovoitov

I suspect it's all working fine already.
Only x86 is using single byte uprobe.
All other archs are using 2 or 4 byte.

Yes, so we have UPROBE_SWBP_INSN_SIZE

Quoted 1 line

So replacing an insn or two with a call should work.

Please note that __uprobe_register(offset) fails if !IS_ALIGNED(offset, UPROBE_SWBP_INSN_SIZE)

Not to mention that if "call" replaces 2 insns we have another problem:
what if another consumer wants to probe the 2ns insn ?

but perhaps (quite possibly) I misunderstand you.

Oleg.

#22 lore raw
In reply to Jiri Olsa
Quoted 21 lines · Jiri Olsa

One of uprobe pain points is having slow execution that involves two traps in worst case scenario or single trap if the original instruction can be emulated. For return uprobes there's one extra trap on top of that.

My current idea on how to make this faster is to follow the optimized kprobes and replace the normal uprobe trap instruction with jump to user space trampoline that:

- executes syscall to call uprobe consumers callbacks
- executes original instructions
- jumps back to continue with the original code

There are of course corner cases where above will have trouble or won't work completely, like:

- executing original instructions in the trampoline is tricky wrt rip relative addressing

- some instructions we can't move to trampoline at all

Quoted 8 lines

- the uprobe address is on page boundary so the jump instruction to trampoline would span across 2 pages, hence the page replace won't be atomic, which might cause issues

- ... ? many others I'm sure

Still with all the limitations I think we could be able to speed up some amount of the uprobes, which seems worth doing.

Just a random idea related to this.
Could we also run jit code of bpf programs in the user space to collect information instead of going back to the kernel every time? These jit code should not be able to access helpers or kfuncs, but they still can collect and aggregate data, store data in bpf maps, and change behavior of user space programs.

Quoted 4 lines

I'd like to have the discussion on the topic and get some agreement or directions on how this should be done.

author
#23 lore raw
Quoted 29 lines · Kui-Feng Lee
Quoted 25 lines · Jiri Olsa

One of uprobe pain points is having slow execution that involves two traps in worst case scenario or single trap if the original instruction can be emulated. For return uprobes there's one extra trap on top of that.

My current idea on how to make this faster is to follow the optimized kprobes and replace the normal uprobe trap instruction with jump to user space trampoline that:

- executes syscall to call uprobe consumers callbacks
- executes original instructions
- jumps back to continue with the original code

There are of course corner cases where above will have trouble or won't work completely, like:

- executing original instructions in the trampoline is tricky wrt rip relative addressing

- some instructions we can't move to trampoline at all

- the uprobe address is on page boundary so the jump instruction to trampoline would span across 2 pages, hence the page replace won't be atomic, which might cause issues

Quoted 8 lines
Quoted 4 lines

- ... ? many others I'm sure

Still with all the limitations I think we could be able to speed up some amount of the uprobes, which seems worth doing.

Just a random idea related to this.
Could we also run jit code of bpf programs in the user space to collect information instead of going back to the kernel every time?

sorry for late reply, do you mean like ubpf? the scope of this change is to speed up the generic uprobe, ebpf is just one of the consumers

jirka

Quoted 8 lines

These jit code should not be able to access helpers or kfuncs, but they still can collect and aggregate data, store data in bpf maps, and change behavior of user space programs.

Quoted 4 lines

I'd like to have the discussion on the topic and get some agreement or directions on how this should be done.

reviewer
#24 lore raw
Quoted 31 lines · Jiri Olsa
Quoted 29 lines · Kui-Feng Lee
Quoted 25 lines · Jiri Olsa

One of uprobe pain points is having slow execution that involves two traps in worst case scenario or single trap if the original instruction can be emulated. For return uprobes there's one extra trap on top of that.

My current idea on how to make this faster is to follow the optimized kprobes and replace the normal uprobe trap instruction with jump to user space trampoline that:

- executes syscall to call uprobe consumers callbacks
- executes original instructions
- jumps back to continue with the original code

There are of course corner cases where above will have trouble or won't work completely, like:

- executing original instructions in the trampoline is tricky wrt rip relative addressing

- some instructions we can't move to trampoline at all

- the uprobe address is on page boundary so the jump instruction to trampoline would span across 2 pages, hence the page replace won't be atomic, which might cause issues

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

- ... ? many others I'm sure

Still with all the limitations I think we could be able to speed up some amount of the uprobes, which seems worth doing.

Just a random idea related to this.
Could we also run jit code of bpf programs in the user space to collect information instead of going back to the kernel every time?

I was thinking about a similar idea. I guess these user space BPF programs will have limited features that we can probably use them update bpf maps. For this limited scope, we still need bpf_arena. Otherwise, the user space bpf program will need to update the bpf maps with sys_bpf(), which adds the same overhead as triggering the program with a syscall.

Quoted 3 lines

sorry for late reply, do you mean like ubpf? the scope of this change is to speed up the generic uprobe, ebpf is just one of the consumers

I guess this means we need a new syscall?

Thanks,
Song

author
#25 lore raw
Quoted 49 lines · Song Liu
Quoted 31 lines · Jiri Olsa
Quoted 29 lines · Kui-Feng Lee
Quoted 25 lines · Jiri Olsa

One of uprobe pain points is having slow execution that involves two traps in worst case scenario or single trap if the original instruction can be emulated. For return uprobes there's one extra trap on top of that.

My current idea on how to make this faster is to follow the optimized kprobes and replace the normal uprobe trap instruction with jump to user space trampoline that:

- executes syscall to call uprobe consumers callbacks
- executes original instructions
- jumps back to continue with the original code

There are of course corner cases where above will have trouble or won't work completely, like:

- executing original instructions in the trampoline is tricky wrt rip relative addressing

- some instructions we can't move to trampoline at all

- the uprobe address is on page boundary so the jump instruction to trampoline would span across 2 pages, hence the page replace won't be atomic, which might cause issues

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

- ... ? many others I'm sure

Still with all the limitations I think we could be able to speed up some amount of the uprobes, which seems worth doing.

Just a random idea related to this.
Could we also run jit code of bpf programs in the user space to collect information instead of going back to the kernel every time?

I was thinking about a similar idea. I guess these user space BPF programs will have limited features that we can probably use them update bpf maps. For this limited scope, we still need bpf_arena. Otherwise, the user space bpf program will need to update the bpf maps with sys_bpf(), which adds the same overhead as triggering the program with a syscall.

Quoted 1 line
Quoted 4 lines
Quoted 2 lines

sorry for late reply, do you mean like ubpf? the scope of this change is to speed up the generic uprobe, ebpf is just one of the consumers

I guess this means we need a new syscall?

yes that's the idea, to replace the trap with syscall, so far I used light version of that for initial testing [1]

jirka

[1] https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git/log/?h=uprobe_syscall_bench_1

#26 lore raw
In reply to Song Liu
Quoted 41 lines · Song Liu
Quoted 31 lines · Jiri Olsa
Quoted 29 lines · Kui-Feng Lee
Quoted 25 lines · Jiri Olsa

One of uprobe pain points is having slow execution that involves two traps in worst case scenario or single trap if the original instruction can be emulated. For return uprobes there's one extra trap on top of that.

My current idea on how to make this faster is to follow the optimized kprobes and replace the normal uprobe trap instruction with jump to user space trampoline that:

- executes syscall to call uprobe consumers callbacks
- executes original instructions
- jumps back to continue with the original code

There are of course corner cases where above will have trouble or won't work completely, like:

- executing original instructions in the trampoline is tricky wrt rip relative addressing

- some instructions we can't move to trampoline at all

- the uprobe address is on page boundary so the jump instruction to trampoline would span across 2 pages, hence the page replace won't be atomic, which might cause issues

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

- ... ? many others I'm sure

Still with all the limitations I think we could be able to speed up some amount of the uprobes, which seems worth doing.

Just a random idea related to this.
Could we also run jit code of bpf programs in the user space to collect information instead of going back to the kernel every time?

Quoted 5 lines

I was thinking about a similar idea. I guess these user space BPF programs will have limited features that we can probably use them update bpf maps. For this limited scope, we still need bpf_arena. Otherwise, the user space bpf program will need to update the bpf maps with sys_bpf(), which adds the same overhead as triggering

That is true. However, even without bpf_arena, it still works with some workarounds without going through sys_bpf().

Quoted 10 lines

the program with a syscall.

Quoted 3 lines

sorry for late reply, do you mean like ubpf? the scope of this change is to speed up the generic uprobe, ebpf is just one of the consumers

I guess this means we need a new syscall?

Thanks,
Song

#27 lore raw
Quoted 45 lines · Kui-Feng Lee
Quoted 41 lines · Song Liu
Quoted 39 lines · Jiri Olsa
Quoted 37 lines · Kui-Feng Lee
Quoted 21 lines · Jiri Olsa

One of uprobe pain points is having slow execution that involves two traps in worst case scenario or single trap if the original instruction can be emulated. For return uprobes there's one extra trap on top of that.

My current idea on how to make this faster is to follow the optimized kprobes and replace the normal uprobe trap instruction with jump to user space trampoline that:

- executes syscall to call uprobe consumers callbacks
- executes original instructions
- jumps back to continue with the original code

There are of course corner cases where above will have trouble or won't work completely, like:

- executing original instructions in the trampoline is tricky wrt rip relative addressing

- some instructions we can't move to trampoline at all

Quoted 8 lines

- the uprobe address is on page boundary so the jump instruction to trampoline would span across 2 pages, hence the page replace won't be atomic, which might cause issues

- ... ? many others I'm sure

Still with all the limitations I think we could be able to speed up some amount of the uprobes, which seems worth doing.

Just a random idea related to this.
Could we also run jit code of bpf programs in the user space to collect information instead of going back to the kernel every time?

Quoted 8 lines
Quoted 5 lines

I was thinking about a similar idea. I guess these user space BPF programs will have limited features that we can probably use them update bpf maps. For this limited scope, we still need bpf_arena. Otherwise, the user space bpf program will need to update the bpf maps with sys_bpf(), which adds the same overhead as triggering

That is true. However, even without bpf_arena, it still works with some workarounds without going through sys_bpf().

Anything making uprobes faster would be very welcomed for my project. The biggest performance problem for us is the cost of bpf_probe_read_user() relative to raw memory access. Every call to this helper walks the process' page table to check that the access would not cause a fault (I think); this is very slow. I wonder if there's some other option that would keep the safety requirement for the memory access -- I'm imagining an optimistic mode where the raw access is performed (in the target process' memory space) and, in the rare case when a fault happens, the kernel would somehow recover from the fault and fail the bpf_probe_read_user() helper. Would something like that be technically feasible / has there been any prior interest in faster access to user memory?

A more limited option that might be helpful would be a vectorized version of bpf_probe_read_user() that verifies many pointers at once.

Quoted 12 lines
Quoted 11 lines

the program with a syscall.

Quoted 3 lines

sorry for late reply, do you mean like ubpf? the scope of this change is to speed up the generic uprobe, ebpf is just one of the consumers

I guess this means we need a new syscall?

Thanks,
Song

#28 lore raw
Quoted 52 lines · Andrei Matei
Quoted 51 lines · Kui-Feng Lee
Quoted 46 lines · Song Liu
Quoted 31 lines · Jiri Olsa
Quoted 29 lines · Kui-Feng Lee
Quoted 25 lines · Jiri Olsa

One of uprobe pain points is having slow execution that involves two traps in worst case scenario or single trap if the original instruction can be emulated. For return uprobes there's one extra trap on top of that.

My current idea on how to make this faster is to follow the optimized kprobes and replace the normal uprobe trap instruction with jump to user space trampoline that:

- executes syscall to call uprobe consumers callbacks
- executes original instructions
- jumps back to continue with the original code

There are of course corner cases where above will have trouble or won't work completely, like:

- executing original instructions in the trampoline is tricky wrt rip relative addressing

- some instructions we can't move to trampoline at all

- the uprobe address is on page boundary so the jump instruction to trampoline would span across 2 pages, hence the page replace won't be atomic, which might cause issues

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

- ... ? many others I'm sure

Still with all the limitations I think we could be able to speed up some amount of the uprobes, which seems worth doing.

Just a random idea related to this.
Could we also run jit code of bpf programs in the user space to collect information instead of going back to the kernel every time?

I was thinking about a similar idea. I guess these user space BPF programs will have limited features that we can probably use them update bpf maps. For this limited scope, we still need bpf_arena. Otherwise, the user space bpf program will need to update the bpf maps with sys_bpf(), which adds the same overhead as triggering

Quoted 6 lines
Quoted 2 lines

That is true. However, even without bpf_arena, it still works with some workarounds without going through sys_bpf().

Anything making uprobes faster would be very welcomed for my project. The biggest performance problem for us is the cost of bpf_probe_read_user() relative to raw memory access. Every call to this helper walks the process'

"raw memory access"? Do you mean not going through any helper function, reading from a pointer directly?

Quoted 5 lines

page table to check that the access would not cause a fault (I think); this is very slow. I wonder if there's some other option that would keep the safety requirement for the memory access -- I'm imagining an optimistic mode where the raw access is performed (in the target process' memory space) and, in the rare case when a fault happens, the kernel would somehow recover from the fault and

I am not very familiar with this part. I read the implementation of bpf_probe_read_user() a little bit. It does what you mentioned here. It would cause page faults, however, the handler will skip the instruction leaving the counter non-zero. By checking the counter, it knows the instruction is not completed, and returns an error.

I am curious about what your access pattern looks like. Does it access a large number of small chunks of data? Or, does it access a small number of big chunks of data?

Quoted 19 lines

fail the bpf_probe_read_user() helper. Would something like that be technically feasible / has there been any prior interest in faster access to user memory

A more limited option that might be helpful would be a vectorized version of bpf_probe_read_user() that verifies many pointers at once.

Quoted 12 lines
Quoted 11 lines

the program with a syscall.

Quoted 3 lines

sorry for late reply, do you mean like ubpf? the scope of this change is to speed up the generic uprobe, ebpf is just one of the consumers

I guess this means we need a new syscall?

Thanks,
Song

#29 lore raw
Quoted 59 lines · Kui-Feng Lee
Quoted 55 lines · Andrei Matei
Quoted 53 lines · Kui-Feng Lee
Quoted 41 lines · Song Liu
Quoted 31 lines · Jiri Olsa
Quoted 29 lines · Kui-Feng Lee
Quoted 25 lines · Jiri Olsa

One of uprobe pain points is having slow execution that involves two traps in worst case scenario or single trap if the original instruction can be emulated. For return uprobes there's one extra trap on top of that.

My current idea on how to make this faster is to follow the optimized kprobes and replace the normal uprobe trap instruction with jump to user space trampoline that:

- executes syscall to call uprobe consumers callbacks
- executes original instructions
- jumps back to continue with the original code

There are of course corner cases where above will have trouble or won't work completely, like:

- executing original instructions in the trampoline is tricky wrt rip relative addressing

- some instructions we can't move to trampoline at all

- the uprobe address is on page boundary so the jump instruction to trampoline would span across 2 pages, hence the page replace won't be atomic, which might cause issues

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

- ... ? many others I'm sure

Still with all the limitations I think we could be able to speed up some amount of the uprobes, which seems worth doing.

Just a random idea related to this.
Could we also run jit code of bpf programs in the user space to collect information instead of going back to the kernel every time?

Quoted 5 lines

I was thinking about a similar idea. I guess these user space BPF programs will have limited features that we can probably use them update bpf maps. For this limited scope, we still need bpf_arena. Otherwise, the user space bpf program will need to update the bpf maps with sys_bpf(), which adds the same overhead as triggering

That is true. However, even without bpf_arena, it still works with some workarounds without going through sys_bpf().

Quoted 6 lines
Quoted 3 lines

Anything making uprobes faster would be very welcomed for my project. The biggest performance problem for us is the cost of bpf_probe_read_user() relative to raw memory access. Every call to this helper walks the process'

"raw memory access"? Do you mean not going through any helper function, reading from a pointer directly?

Right.
I recognize that, as long as bpf runs "in the kernel", one cannot simply dereference a user-space pointer since the kernel is a different virtual memory space (*). Still, I wish there bpf_probe_read_user() were faster.

(*) Or, is it indeed a different memory space or is the kernel's virtual address space mapped into every process? Did this change through KPTI? I would be curious to read a good resource on what exactly it means to switch from user-space to the kernel and back, if such a thing exists.

Quoted 13 lines
Quoted 6 lines

page table to check that the access would not cause a fault (I think); this is very slow. I wonder if there's some other option that would keep the safety requirement for the memory access -- I'm imagining an optimistic mode where the raw access is performed (in the target process' memory space) and, in the rare case when a fault happens, the kernel would somehow recover from the fault and

I am not very familiar with this part. I read the implementation of bpf_probe_read_user() a little bit. It does what you mentioned here. It would cause page faults, however, the handler will skip the instruction leaving the counter non-zero. By checking the counter, it knows the instruction is not completed, and returns an error.

Quoted 3 lines

I am curious about what your access pattern looks like. Does it access a large number of small chunks of data? Or, does it access a small number of big chunks of data?

My access pattern looks like a lot of small reads. Some of these reads could be done at the same time if we had a vectorized API (i.e. some of the pointers are known in advance); for others there are data dependencies (i.e. we need to dereference a pointer to know what we'll want to read next). Specifically, the use case is a debugger of sorts which uses BPF uprobes for poking around in the target process' memory, rather than the more traditional ptrace-based techniques (ptrace being very slow). This debugger needs to walk a lot of thread stacks by following stack pointers or by using DWARF unwind information, and then it further reads data structures from the target process' stacks and heaps, chasing pointers recursively.

Quoted 20 lines
Quoted 20 lines

fail the bpf_probe_read_user() helper. Would something like that be technically feasible / has there been any prior interest in faster access to user memory

A more limited option that might be helpful would be a vectorized version of bpf_probe_read_user() that verifies many pointers at once.

Quoted 12 lines
Quoted 11 lines

the program with a syscall.

Quoted 3 lines

sorry for late reply, do you mean like ubpf? the scope of this change is to speed up the generic uprobe, ebpf is just one of the consumers

I guess this means we need a new syscall?

Thanks,
Song

#30 lore raw
Quoted 72 lines · Andrei Matei
Quoted 59 lines · Kui-Feng Lee
Quoted 55 lines · Andrei Matei
Quoted 53 lines · Kui-Feng Lee
Quoted 41 lines · Song Liu
Quoted 31 lines · Jiri Olsa
Quoted 29 lines · Kui-Feng Lee
Quoted 25 lines · Jiri Olsa

One of uprobe pain points is having slow execution that involves two traps in worst case scenario or single trap if the original instruction can be emulated. For return uprobes there's one extra trap on top of that.

My current idea on how to make this faster is to follow the optimized kprobes and replace the normal uprobe trap instruction with jump to user space trampoline that:

- executes syscall to call uprobe consumers callbacks
- executes original instructions
- jumps back to continue with the original code

There are of course corner cases where above will have trouble or won't work completely, like:

- executing original instructions in the trampoline is tricky wrt rip relative addressing

- some instructions we can't move to trampoline at all

- the uprobe address is on page boundary so the jump instruction to trampoline would span across 2 pages, hence the page replace won't be atomic, which might cause issues

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

- ... ? many others I'm sure

Still with all the limitations I think we could be able to speed up some amount of the uprobes, which seems worth doing.

Just a random idea related to this.
Could we also run jit code of bpf programs in the user space to collect information instead of going back to the kernel every time?

Quoted 5 lines

I was thinking about a similar idea. I guess these user space BPF programs will have limited features that we can probably use them update bpf maps. For this limited scope, we still need bpf_arena. Otherwise, the user space bpf program will need to update the bpf maps with sys_bpf(), which adds the same overhead as triggering

That is true. However, even without bpf_arena, it still works with some workarounds without going through sys_bpf().

Quoted 6 lines
Quoted 3 lines

Anything making uprobes faster would be very welcomed for my project. The biggest performance problem for us is the cost of bpf_probe_read_user() relative to raw memory access. Every call to this helper walks the process'

"raw memory access"? Do you mean not going through any helper function, reading from a pointer directly?

Right.
I recognize that, as long as bpf runs "in the kernel", one cannot simply dereference a user-space pointer since the kernel is a different virtual memory space (*). Still, I wish there bpf_probe_read_user() were faster.

Quoted 4 lines

(*) Or, is it indeed a different memory space or is the kernel's virtual address space mapped into every process? Did this change through KPTI? I would be curious to read a good resource on what exactly it means to switch from user-space to the kernel and back, if such a thing exists.

FYI! This is architecture dependent. AFAIK, with x86 platforms, kernel can access the memory of the user space directly if it is in a process/task context. But, you should not relies on it.

If you look into bpf_probe_read_user(), it eventually do something like "rep movsb" on x86 platforms. Access user space memory directly with some extra checks. So, the bottleneck here can be the extra checks and memory copying. If you access small chunks like what you said bellow, the overhead of checks could be expensive.

Quoted 18 lines
Quoted 14 lines
Quoted 7 lines

page table to check that the access would not cause a fault (I think); this is very slow. I wonder if there's some other option that would keep the safety requirement for the memory access -- I'm imagining an optimistic mode where the raw access is performed (in the target process' memory space) and, in the rare case when a fault happens, the kernel would somehow recover from the fault and

I am not very familiar with this part. I read the implementation of bpf_probe_read_user() a little bit. It does what you mentioned here. It would cause page faults, however, the handler will skip the instruction leaving the counter non-zero. By checking the counter, it knows the instruction is not completed, and returns an error.

Quoted 3 lines

I am curious about what your access pattern looks like. Does it access a large number of small chunks of data? Or, does it access a small number of big chunks of data?

Quoted 10 lines

My access pattern looks like a lot of small reads. Some of these reads could be done at the same time if we had a vectorized API (i.e. some of the pointers are known in advance); for others there are data dependencies (i.e. we need to dereference a pointer to know what we'll want to read next). Specifically, the use case is a debugger of sorts which uses BPF uprobes for poking around in the target process' memory, rather than the more traditional ptrace-based techniques (ptrace being very slow). This debugger needs to walk a lot of thread stacks by following stack pointers or by using DWARF unwind information, and then it further reads data structures from the target process' stacks and heaps, chasing pointers recursively.

A related information. You may already know that bpf_probe_read_user()
can fail if a page fault happens. A vectorized API probably doesn't
change it. It is a limitation of non-sleepable BPF programs. Sleepable BPF programs should be able to overcome it.

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines

fail the bpf_probe_read_user() helper. Would something like that be technically feasible / has there been any prior interest in faster access to user memory

A more limited option that might be helpful would be a vectorized version of bpf_probe_read_user() that verifies many pointers at once.

Quoted 12 lines
Quoted 11 lines

the program with a syscall.

Quoted 3 lines

sorry for late reply, do you mean like ubpf? the scope of this change is to speed up the generic uprobe, ebpf is just one of the consumers

I guess this means we need a new syscall?

Thanks,
Song