uprobes: Add support to optimize usdt probes on x86_64
Status from this message: The following commit has been merged into the perf/core branch of tip:
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v6 Latest21 of 22 patches 41 repliesApplied
- v522 patches 19 repliesSuperseded
- v422 patches 3 repliesSuperseded
Earlier versions · 3
| Version | Posted | Patches | Replies | Status |
|---|---|---|---|---|
| v3 | 22 patches | 16 replies | Superseded | |
| v2 | 22 patches | 14 replies | Superseded | |
| v1 | 22 patches | 51 replies | Superseded |
Changes in v6 from the cover letter
- reworked and unified int3 update code [Peter]
- used struct to read syscall argument [Peter]
- skip trampoline call instruction when returning via iret
- small changes plus commenct [Masami]
- added acks [Oleg, Andrii, Masami]
21 of 22 patches are in this archive. The cover letter announces 22.
Patches in v6 Applied
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/22 | uprobes: Remove breakpoint in unapply_uprobe under mmap_write_lockkernel/ | +3 | 0 replies | |
| 2/22 | uprobes: Rename arch_uretprobe_trampoline functionarch/ include/ kernel/ | +4 | 0 replies | |
| 3/22 | uprobes: Make copy_from_page globalinclude/ kernel/ | +6 | 0 replies | |
| 4/22 | uprobes: Add uprobe_write functioninclude/ kernel/ | +15 | 0 replies | |
| 5/22 | uprobes: Add nbytes argument to uprobe_writeinclude/ kernel/ | +16 | 0 replies | |
| 6/22 | uprobes: Add is_register argument to uprobe_write and uprobe_write_opcodearch/ include/ kernel/ | +15 | 0 replies | |
| 7/22 | uprobes: Add do_ref_ctr argument to uprobe_write functioninclude/ kernel/ | +5 | 0 replies | |
| 8/22 | uprobes/x86: Add mapping for optimized uprobe trampolinesarch/ include/ kernel/ +1 more | +161 | 2 replies | |
| 9/22 | uprobes/x86: Add uprobe syscall to speed up uprobearch/ arch/ include/ +3 more | +161 | 23 replies | |
| 10/22 | uprobes/x86: Add support to optimize uprobesarch/ arch/ include/ +1 more | +305 | 16 replies | |
| 11/22 | selftests/bpf: Import usdt.h from libbpf/usdt projecttools/ | +545 | 0 replies | |
| 12/22 | selftests/bpf: Reorg the uprobe_syscall test functiontools/ | +12 | 0 replies | |
| 13/22 | selftests/bpf: Rename uprobe_syscall_executed prog to test_uretprobe_multitools/ tools/ | +14 | 0 replies | |
| 14/22 | selftests/bpf: Add uprobe/usdt syscall teststools/ tools/ | +335 | 0 replies | |
| 15/22 | selftests/bpf: Add hit/attach/detach race optimized uprobe testtools/ | +108 | 0 replies | |
| 16/22 | selftests/bpf: Add uprobe syscall sigill signal testtools/ | +36 | 0 replies | |
| 17/22 | selftests/bpf: Add optimized usdt variant for basic usdt testtools/ | +25 | 0 replies | |
| 18/22 | selftests/bpf: Add uprobe_regs_equal testtools/ tools/ | +44 | 0 replies | |
| 19/22 | selftests/bpf: Change test_uretprobe_regs_change for uprobe and uretprobetools/ tools/ | +17 | 0 replies | |
| 20/22 | seccomp: passthrough uprobe systemcall without filteringkernel/ | +25 | 0 replies | |
| 21/22 | selftests/seccomp: validate uprobe syscall passes through seccomptools/ | +86 | 0 replies |
Reviews trailers per patch
All 21 patches have a Reviewed-by or Acked-by of their own; the series has one for all of them.
| Patch | Andrii Nakryiko | Masami Hiramatsu (Google) | Oleg Nesterov | Kees Cook | Alejandro Colomar |
|---|---|---|---|---|---|
| Series | R-b | R-b | |||
| 1/22 | A-b | A-b | A-b | ||
| 2/22 | A-b | A-b | |||
| 3/22 | A-b | A-b | |||
| 4/22 | A-b | A-b | A-b | ||
| 5/22 | A-b | A-b | A-b | ||
| 6/22 | A-b | A-b | A-b | ||
| 7/22 | A-b | R-b | A-b | ||
| 8/22 | A-b | A-b | A-b | ||
| 9/22 | A-b A-b A-b | A-b | A-b | ||
| 10/22 | A-b | A-b | A-b | ||
| 11/22 | A-b | ||||
| 12/22 | A-b | ||||
| 13/22 | A-b | ||||
| 14/22 | A-b | ||||
| 15/22 | A-b | ||||
| 16/22 | A-b | ||||
| 17/22 | A-b | ||||
| 18/22 | A-b | ||||
| 19/22 | A-b | ||||
| 20/22 | R-b | ||||
| 21/22 | R-b |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 5 reviewers.
Cover letter [PATCHv6 perf/core 00/22] uprobes: Add support to optimize usdt probes on x86_64
hi,
this patchset adds support to optimize usdt probes on top of 5-byte nop instruction.
The generic approach (optimize all uprobes) is hard due to emulating possible multiple original instructions and its related issues. The usdt case, which stores 5-byte nop seems much easier, so starting with that.
The basic idea is to replace breakpoint exception with syscall which is faster on x86_64. For more details please see changelog of patch 8.
The run_bench_uprobes.sh benchmark triggers uprobe (on top of different original instructions) in a loop and counts how many of those happened per second (the unit below is million loops).
There's big speed up if you consider current usdt implementation (uprobe-nop) compared to proposed usdt (uprobe-nop5):
current:
usermode-count : 152.501 ± 0.012M/s
syscall-count : 14.463 ± 0.062M/s
--> uprobe-nop : 3.160 ± 0.005M/s
uprobe-push : 3.003 ± 0.003M/s
uprobe-ret : 1.100 ± 0.003M/s
uprobe-nop5 : 3.132 ± 0.012M/s
uretprobe-nop : 2.103 ± 0.002M/s
uretprobe-push : 2.027 ± 0.004M/s
uretprobe-ret : 0.914 ± 0.002M/s
uretprobe-nop5 : 2.115 ± 0.002M/safter the change:
usermode-count : 152.343 ± 0.400M/s
syscall-count : 14.851 ± 0.033M/s
uprobe-nop : 3.204 ± 0.005M/s
uprobe-push : 3.040 ± 0.005M/s
uprobe-ret : 1.098 ± 0.003M/s
--> uprobe-nop5 : 7.286 ± 0.017M/s
uretprobe-nop : 2.144 ± 0.001M/s
uretprobe-push : 2.069 ± 0.002M/s
uretprobe-ret : 0.922 ± 0.000M/s
uretprobe-nop5 : 3.487 ± 0.001M/sI see bit more speed up on Intel (above) compared to AMD. The big nop5 speed up is partly due to emulating nop5 and partly due to optimization.
The key speed up we do this for is the USDT switch from nop to nop5:
uprobe-nop : 3.160 ± 0.005M/s uprobe-nop5 : 7.286 ± 0.017M/s
Changes from v5:
- reworked and unified int3 update code [Peter]
- used struct to read syscall argument [Peter]
- skip trampoline call instruction when returning via iret
- small changes plus commenct [Masami]
- added acks [Oleg, Andrii, Masami]
Changes from v4:
- reworked search for trampoline page, dropped Oleg's ack from that patch because of the change [Masami]
Changes from v3:
- rebased on top of tip/master + bpf-next/master + mm/mm-nonmm-unstable
- reworked patch#8 to lookup trampoline trampoline every 4GB so we don't waste page frames in some cases [Masami]
- several minor fixes [Masami]
- added acks [Oleg, Alejandro, Masami]
Changes from v2:
- rebased on top of tip/master + mm/mm-stable + 1 extra change [1]
- added acks [Oleg,Andrii]
- more details changelog for patch 1 [Masami]
- several tests changes [Andrii]
- add explicit PAGE_SIZE low limit to vm_unmapped_area call [Andrii]
This patchset is adding new syscall, here are notes to check list items in Documentation/process/adding-syscalls.rst:
- System Call Alternatives
New syscall seems like the best way in here, because we need just to quickly enter kernel with no extra arguments processing, which we'd need to do if we decided to use another syscall.
- Designing the API: Planning for Extension
The uprobe syscall is very specific and most likely won't be extended in the future.
- Designing the API: Other Considerations
N/A because uprobe syscall does not return reference to kernel object.
- Proposing the API
Wiring up of the uprobe system call is in separate change, selftests and man page changes are part of the patchset.
- Generic System Call Implementation
There's no CONFIG option for the new functionality because it keeps the same behaviour from the user POV.
- x86 System Call Implementation
It's 64-bit syscall only.
- Compatibility System Calls (Generic)
N/A uprobe syscall has no arguments and is not supported for compat processes.
- Compatibility System Calls (x86)
N/A uprobe syscall is not supported for compat processes.
- System Calls Returning Elsewhere
N/A.
- Other Details
N/A.
- Testing
Adding new bpf selftests.
- Man Page
Attached.
- Do not call System Calls in the Kernel
N/A
pending todo (or follow ups):
- use PROCMAP_QUERY in tests
- alloc 'struct uprobes_state' for mm_struct only when needed [Andrii]
- use mm_cpumask(vma->vm_mm) in text_poke_sync
thanks,
jirka
[1] https://lore.kernel.org/linux-trace-kernel/[email protected]/T/#u
---
Jiri Olsa (21):
uprobes: Remove breakpoint in unapply_uprobe under mmap_write_lock
uprobes: Rename arch_uretprobe_trampoline function
uprobes: Make copy_from_page global
uprobes: Add uprobe_write function
uprobes: Add nbytes argument to uprobe_write
uprobes: Add is_register argument to uprobe_write and uprobe_write_opcode
uprobes: Add do_ref_ctr argument to uprobe_write function
uprobes/x86: Add mapping for optimized uprobe trampolines
uprobes/x86: Add uprobe syscall to speed up uprobe
uprobes/x86: Add support to optimize uprobes
selftests/bpf: Import usdt.h from libbpf/usdt project
selftests/bpf: Reorg the uprobe_syscall test function
selftests/bpf: Rename uprobe_syscall_executed prog to test_uretprobe_multi selftests/bpf: Add uprobe/usdt syscall tests
selftests/bpf: Add hit/attach/detach race optimized uprobe test selftests/bpf: Add uprobe syscall sigill signal test
selftests/bpf: Add optimized usdt variant for basic usdt test selftests/bpf: Add uprobe_regs_equal test
selftests/bpf: Change test_uretprobe_regs_change for uprobe and uretprobe
seccomp: passthrough uprobe systemcall without filtering selftests/seccomp: validate uprobe syscall passes through seccomp
arch/arm/probes/uprobes/core.c | 2 +- arch/x86/entry/syscalls/syscall_64.tbl | 1 + arch/x86/include/asm/uprobes.h | 7 ++ arch/x86/kernel/uprobes.c | 566 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++- include/linux/syscalls.h | 2 + include/linux/uprobes.h | 20 +++- kernel/events/uprobes.c | 100 +++++++++++----- kernel/fork.c | 1 + kernel/seccomp.c | 32 ++++-- kernel/sys_ni.c | 1 + tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c | 518 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------ tools/testing/selftests/bpf/prog_tests/usdt.c | 38 ++++--- tools/testing/selftests/bpf/progs/uprobe_syscall.c | 4 +- tools/testing/selftests/bpf/progs/uprobe_syscall_executed.c | 60 +++++++++- tools/testing/selftests/bpf/test_kmods/bpf_testmod.c | 11 +- tools/testing/selftests/bpf/usdt.h | 545 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ tools/testing/selftests/seccomp/seccomp_bpf.c | 107 ++++++++++++++---- 17 files changed, 1903 insertions(+), 112 deletions(-) create mode 100644 tools/testing/selftests/bpf/usdt.h
Jiri Olsa (1):
man2: Add uprobe syscall page
man/man2/uprobe.2 | 1 + man/man2/uretprobe.2 | 36 ++++++++++++++++++++++++------------ 2 files changed, 25 insertions(+), 12 deletions(-) create mode 100644 man/man2/uprobe.2