uprobes: Add support to optimize usdt probes on x86_64
Subsystems (15)
Jiri Olsa
reviewerv3 cover uprobes: Add support to optimize usdt probes on x86_64
hi,
this patchset adds support to optimize usdt probes on top of 5-byte nop instruction.
The generic approach (optimize all uprobes) is hard due to emulating possible multiple original instructions and its related issues. The usdt case, which stores 5-byte nop seems much easier, so starting with that.
The basic idea is to replace breakpoint exception with syscall which is faster on x86_64. For more details please see changelog of patch 8.
The run_bench_uprobes.sh benchmark triggers uprobe (on top of different original instructions) in a loop and counts how many of those happened per second (the unit below is million loops).
There's big speed up if you consider current usdt implementation (uprobe-nop) compared to proposed usdt (uprobe-nop5):
current:
usermode-count : 152.501 ± 0.012M/s
syscall-count : 14.463 ± 0.062M/s
--> uprobe-nop : 3.160 ± 0.005M/s
uprobe-push : 3.003 ± 0.003M/s
uprobe-ret : 1.100 ± 0.003M/s
uprobe-nop5 : 3.132 ± 0.012M/s
uretprobe-nop : 2.103 ± 0.002M/s
uretprobe-push : 2.027 ± 0.004M/s
uretprobe-ret : 0.914 ± 0.002M/s
uretprobe-nop5 : 2.115 ± 0.002M/safter the change:
usermode-count : 152.343 ± 0.400M/s
syscall-count : 14.851 ± 0.033M/s
uprobe-nop : 3.204 ± 0.005M/s
uprobe-push : 3.040 ± 0.005M/s
uprobe-ret : 1.098 ± 0.003M/s
--> uprobe-nop5 : 7.286 ± 0.017M/s
uretprobe-nop : 2.144 ± 0.001M/s
uretprobe-push : 2.069 ± 0.002M/s
uretprobe-ret : 0.922 ± 0.000M/s
uretprobe-nop5 : 3.487 ± 0.001M/sI see bit more speed up on Intel (above) compared to AMD. The big nop5 speed up is partly due to emulating nop5 and partly due to optimization.
The key speed up we do this for is the USDT switch from nop to nop5:
uprobe-nop : 3.160 ± 0.005M/s uprobe-nop5 : 7.286 ± 0.017M/s
Changes from v2:
- rebased on top of tip/master + mm/mm-stable + 1 extra change [1]
- added acks [Oleg,Andrii]
- more details changelog for patch 1 [Masami]
- several tests changes [Andrii]
- add explicit PAGE_SIZE low limit to vm_unmapped_area call [Andrii]
This patchset is adding new syscall, here are notes to check list items in Documentation/process/adding-syscalls.rst:
- System Call Alternatives
New syscall seems like the best way in here, because we need just to quickly enter kernel with no extra arguments processing, which we'd need to do if we decided to use another syscall.
- Designing the API: Planning for Extension
The uprobe syscall is very specific and most likely won't be extended in the future.
- Designing the API: Other Considerations
N/A because uprobe syscall does not return reference to kernel object.
- Proposing the API
Wiring up of the uprobe system call is in separate change, selftests and man page changes are part of the patchset.
- Generic System Call Implementation
There's no CONFIG option for the new functionality because it keeps the same behaviour from the user POV.
- x86 System Call Implementation
It's 64-bit syscall only.
- Compatibility System Calls (Generic)
N/A uprobe syscall has no arguments and is not supported for compat processes.
- Compatibility System Calls (x86)
N/A uprobe syscall is not supported for compat processes.
- System Calls Returning Elsewhere
N/A.
- Other Details
N/A.
- Testing
Adding new bpf selftests.
- Man Page
Attached.
- Do not call System Calls in the Kernel
N/A
pending todo (or follow ups):
- use PROCMAP_QUERY in tests
- alloc 'struct uprobes_state' for mm_struct only when needed [Andrii]
- use mm_cpumask(vma->vm_mm) in text_poke_sync
thanks,
jirka
[1] https://lore.kernel.org/linux-trace-kernel/[email protected]/T/#u
---
Jiri Olsa (21):
uprobes: Remove breakpoint in unapply_uprobe under mmap_write_lock
uprobes: Rename arch_uretprobe_trampoline function
uprobes: Make copy_from_page global
uprobes: Add uprobe_write function
uprobes: Add nbytes argument to uprobe_write
uprobes: Add is_register argument to uprobe_write and uprobe_write_opcode
uprobes: Add do_ref_ctr argument to uprobe_write function
uprobes/x86: Add mapping for optimized uprobe trampolines
uprobes/x86: Add uprobe syscall to speed up uprobe
uprobes/x86: Add support to optimize uprobes
selftests/bpf: Import usdt.h from libbpf/usdt project
selftests/bpf: Reorg the uprobe_syscall test function
selftests/bpf: Rename uprobe_syscall_executed prog to test_uretprobe_multi selftests/bpf: Add uprobe/usdt syscall tests
selftests/bpf: Add hit/attach/detach race optimized uprobe test selftests/bpf: Add uprobe syscall sigill signal test
selftests/bpf: Add optimized usdt variant for basic usdt test selftests/bpf: Add uprobe_regs_equal test
selftests/bpf: Change test_uretprobe_regs_change for uprobe and uretprobe
seccomp: passthrough uprobe systemcall without filtering selftests/seccomp: validate uprobe syscall passes through seccomp
arch/arm/probes/uprobes/core.c | 2 +- arch/x86/entry/syscalls/syscall_64.tbl | 1 + arch/x86/include/asm/uprobes.h | 7 ++ arch/x86/kernel/uprobes.c | 525 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++- include/linux/syscalls.h | 2 + include/linux/uprobes.h | 20 +++- kernel/events/uprobes.c | 100 ++++++++++++----- kernel/fork.c | 1 + kernel/seccomp.c | 32 ++++-- kernel/sys_ni.c | 1 + tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c | 523 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------ tools/testing/selftests/bpf/prog_tests/usdt.c | 38 ++++--- tools/testing/selftests/bpf/progs/uprobe_syscall.c | 4 +- tools/testing/selftests/bpf/progs/uprobe_syscall_executed.c | 60 +++++++++- tools/testing/selftests/bpf/test_kmods/bpf_testmod.c | 11 +- tools/testing/selftests/bpf/usdt.h | 545 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ tools/testing/selftests/seccomp/seccomp_bpf.c | 107 ++++++++++++++---- 17 files changed, 1867 insertions(+), 112 deletions(-) create mode 100644 tools/testing/selftests/bpf/usdt.h
Jiri Olsa (1):
man2: Add uprobe syscall page
man/man2/uprobe.2 | 1 + man/man2/uretprobe.2 | 36 ++++++++++++++++++++++++------------ 2 files changed, 25 insertions(+), 12 deletions(-) create mode 100644 man/man2/uprobe.2
Jiri Olsa
reviewerPATCH v3 1/22 uprobes: Remove breakpoint in unapply_uprobe under mmap_write_lock
Currently unapply_uprobe takes mmap_read_lock, but it might call remove_breakpoint which eventually changes user pages.
Current code writes either breakpoint or original instruction, so it can go away with read lock as explained in here [1]. But with the upcoming change that writes multiple instructions on the probed address we need to ensure that any update to mm's pages is exclusive.
[1] https://lore.kernel.org/all/[email protected]/
--- kernel/events/uprobes.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
kernel/events/uprobes.c +3 -3
@@ -483,7 +483,7 @@ static int __uprobe_write_opcode(struct vm_area_struct *vma,* @opcode_vaddr: the virtual address to store the opcode. * @opcode: opcode to be written at @opcode_vaddr. * -* Called with mm->mmap_lock held for read or write. +* Called with mm->mmap_lock held for write. * Return 0 (success) or a negative errno. */ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct vm_area_struct *vma,
@@ -1464,7 +1464,7 @@ static int unapply_uprobe(struct uprobe *uprobe, struct mm_struct *mm)struct vm_area_struct *vma;int err =0 ; -mmap_read_lock (mm); +mmap_write_lock (mm);for_each_vma (vmi, vma) {unsigned long vaddr;loff_t offset;
@@ -1481,7 +1481,7 @@ static int unapply_uprobe(struct uprobe *uprobe, struct mm_struct *mm) vaddr =offset_to_vaddr (vma, uprobe->offset); err |=remove_breakpoint (uprobe, vma, vaddr); } -mmap_read_unlock (mm); +mmap_write_unlock (mm);return err; }
Signature · 1 line
2.49.0
Re: [PATCHv3 perf/core 01/22] uprobes: Remove breakpoint in unapply_uprobe under mmap_write_lock
Quoted 10 lines · On Thu, 5 Jun 2025 15:23:28 +0200 Jiri Olsa <[email protected]> wrote:
Currently
unapply_uprobetakesmmap_read_lock, but it might callremove_breakpointwhich eventually changes user pages.Current code writes either breakpoint or original instruction, so it can go away with read lock as explained in here [1]. But with the upcoming change that writes multiple instructions on the probed address we need to ensure that any update to mm's pages is exclusive.
Looks good to me.
Thanks,
Quoted 40 lines
Acked-by Oleg Nesterov <[email protected]>Signed-off-by Jiri Olsa <[email protected]>--- kernel/events/uprobes.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)kernel/events/uprobes.c +3
-3@@ -483,7 +483,7 @@ static int __uprobe_write_opcode(struct vm_area_struct *vma,* @opcode_vaddr: the virtual address to store the opcode. * @opcode: opcode to be written at @opcode_vaddr. * -* Called with mm->mmap_lock held for read or write. +* Called with mm->mmap_lock held for write. * Return 0 (success) or a negative errno. */ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct vm_area_struct *vma,@@ -1464,7 +1464,7 @@ static int unapply_uprobe(struct uprobe *uprobe, struct mm_struct *mm)struct vm_area_struct *vma;int err =0 ; -mmap_read_lock (mm); +mmap_write_lock (mm);for_each_vma (vmi, vma) {unsigned long vaddr;loff_t offset;@@ -1481,7 +1481,7 @@ static int unapply_uprobe(struct uprobe *uprobe, struct mm_struct *mm) vaddr =offset_to_vaddr (vma, uprobe->offset); err |=remove_breakpoint (uprobe, vma, vaddr); } -mmap_read_unlock (mm); +mmap_write_unlock (mm);return err; }Signature · 1 line
2.49.0
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Jiri Olsa
reviewerPATCH v3 2/22 uprobes: Rename arch_uretprobe_trampoline function
In reply to Jiri OlsaWe are about to add uprobe trampoline, so cleaning up the namespace.
--- arch/x86/kernel/uprobes.c | 2 +- include/linux/uprobes.h | 2 +- kernel/events/uprobes.c | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-)
arch/x86/kernel/uprobes.c +1 -1
@@ -338,7 +338,7 @@ extern u8 uretprobe_trampoline_entry[];extern u8 uretprobe_trampoline_end[];extern u8 uretprobe_syscall_check[]; -void *arch_uprobe_trampoline (unsigned long *psize) +void *arch_uretprobe_trampoline (unsigned long *psize) {static uprobe_opcode_t insn =UPROBE_SWBP_INSN ;struct pt_regs *regs =task_pt_regs (current);
include/linux/uprobes.h +1 -1
@@ -224,7 +224,7 @@ extern bool arch_uprobe_ignore(struct arch_uprobe *aup, struct pt_regs *regs);extern void arch_uprobe_copy_ixol (struct page *page,unsigned long vaddr,void *src,unsigned long len);extern void uprobe_handle_trampoline (struct pt_regs *regs); -extern void *arch_uprobe_trampoline (unsigned long *psize); +extern void *arch_uretprobe_trampoline (unsigned long *psize);extern unsigned long uprobe_get_trampoline_vaddr (void );#else /* !CONFIG_UPROBES */ struct uprobes_state {
kernel/events/uprobes.c +2 -2
@@ -1727,7 +1727,7 @@ static int xol_add_vma(struct mm_struct *mm, struct xol_area *area)return ret; } -void *__weak arch_uprobe_trampoline (unsigned long *psize) +void *__weak arch_uretprobe_trampoline (unsigned long *psize) {static uprobe_opcode_t insn =UPROBE_SWBP_INSN ;
@@ -1759,7 +1759,7 @@ static struct xol_area *__create_xol_area(unsigned long vaddr)init_waitqueue_head (&area->wq);/* Reserve the 1st slot for get_trampoline_vaddr() */ set_bit (0 , area->bitmap); - insns =arch_uprobe_trampoline (&insns_size); + insns =arch_uretprobe_trampoline (&insns_size);arch_uprobe_copy_ixol (area->page,0 , insns, insns_size);if (!xol_add_vma (mm, area))
Signature · 1 line
2.49.0
Jiri Olsa
reviewerPATCH v3 3/22 uprobes: Make copy_from_page global
In reply to Jiri OlsaMaking copy_from_page global and adding uprobe prefix. Adding the uprobe prefix to copy_to_page as well for symmetry.
--- include/linux/uprobes.h | 1 + kernel/events/uprobes.c | 10 +++++----- 2 files changed, 6 insertions(+), 5 deletions(-)
include/linux/uprobes.h +1 -0
@@ -226,6 +226,7 @@ extern void arch_uprobe_copy_ixol(struct page *page, unsigned long vaddr,extern void uprobe_handle_trampoline (struct pt_regs *regs);extern void *arch_uretprobe_trampoline (unsigned long *psize);extern unsigned long uprobe_get_trampoline_vaddr (void ); +extern void uprobe_copy_from_page (struct page *page,unsigned long vaddr,void *dst,int len);#else /* !CONFIG_UPROBES */ struct uprobes_state { };
kernel/events/uprobes.c +5 -5
@@ -177,7 +177,7 @@ bool __weak is_trap_insn(uprobe_opcode_t *insn)return is_swbp_insn (insn); } -static void copy_from_page (struct page *page,unsigned long vaddr,void *dst,int len) +void uprobe_copy_from_page (struct page *page,unsigned long vaddr,void *dst,int len) {void *kaddr =kmap_atomic (page);memcpy (dst, kaddr + (vaddr & ~PAGE_MASK ), len);
@@ -205,7 +205,7 @@ static int verify_opcode(struct page *page, unsigned long vaddr, uprobe_opcode_t* is a trap variant; uprobes always wins over any other (gdb) * breakpoint. */ -copy_from_page (page, vaddr, &old_opcode,UPROBE_SWBP_INSN_SIZE ); +uprobe_copy_from_page (page, vaddr, &old_opcode,UPROBE_SWBP_INSN_SIZE ); is_swbp =is_swbp_insn (&old_opcode);if (is_swbp_insn (new_opcode)) {
@@ -1052,7 +1052,7 @@ static int __copy_insn(struct address_space *mapping, struct file *filp,if (IS_ERR (page))return PTR_ERR (page); -copy_from_page (page, offset, insn, nbytes); +uprobe_copy_from_page (page, offset, insn, nbytes);put_page (page);return 0 ;
@@ -1398,7 +1398,7 @@ struct uprobe *uprobe_register(struct inode *inode,return ERR_PTR (-EINVAL );/* -* This ensures that copy_from_page(), copy_to_page() and +* This ensures that uprobe_copy_from_page(), copy_to_page() and * __update_ref_ctr() can't cross page boundary. */ if (!IS_ALIGNED (offset,UPROBE_SWBP_INSN_SIZE ))
@@ -2394,7 +2394,7 @@ static int is_trap_at_addr(struct mm_struct *mm, unsigned long vaddr)if (result <0 )return result; -copy_from_page (page, vaddr, &opcode,UPROBE_SWBP_INSN_SIZE ); +uprobe_copy_from_page (page, vaddr, &opcode,UPROBE_SWBP_INSN_SIZE );put_page (page); out:/* This needs to return true for any variant of the trap insn */
Signature · 1 line
2.49.0
Jiri Olsa
reviewerPATCH v3 4/22 uprobes: Add uprobe_write function
In reply to Jiri OlsaAdding uprobe_write function that does what uprobe_write_opcode did so far, but allows to pass verify callback function that checks the memory location before writing the opcode.
It will be used in following changes to implement specific checking logic for instruction update.
The uprobe_write_opcode now calls uprobe_write with verify_opcode as the verify callback.
--- include/linux/uprobes.h | 5 +++++ kernel/events/uprobes.c | 14 ++++++++++---- 2 files changed, 15 insertions(+), 4 deletions(-)
include/linux/uprobes.h +5 -0
@@ -187,6 +187,9 @@ struct uprobes_state {struct xol_area *xol_area; }; +typedef int (*uprobe_write_verify_t )(struct page *page,unsigned long vaddr, +uprobe_opcode_t *opcode); +extern void __init uprobes_init (void );extern int set_swbp (struct arch_uprobe *aup,struct vm_area_struct *vma,unsigned long vaddr);extern int set_orig_insn (struct arch_uprobe *aup,struct vm_area_struct *vma,unsigned long vaddr);
@@ -195,6 +198,8 @@ extern bool is_trap_insn(uprobe_opcode_t *insn);extern unsigned long uprobe_get_swbp_addr (struct pt_regs *regs);extern unsigned long uprobe_get_trap_addr (struct pt_regs *regs);extern int uprobe_write_opcode (struct arch_uprobe *auprobe,struct vm_area_struct *vma,unsigned long vaddr,uprobe_opcode_t ); +extern int uprobe_write (struct arch_uprobe *auprobe,struct vm_area_struct *vma,const unsigned long opcode_vaddr, +uprobe_opcode_t opcode,uprobe_write_verify_t verify);extern struct uprobe *uprobe_register (struct inode *inode,loff_t offset,loff_t ref_ctr_offset,struct uprobe_consumer *uc);extern int uprobe_apply (struct uprobe *uprobe,struct uprobe_consumer *uc,bool );extern void uprobe_unregister_nosync (struct uprobe *uprobe,struct uprobe_consumer *uc);
kernel/events/uprobes.c +10 -4
@@ -399,7 +399,7 @@ static bool orig_page_is_identical(struct vm_area_struct *vma,return identical; } -static int __uprobe_write_opcode (struct vm_area_struct *vma, +static int __uprobe_write (struct vm_area_struct *vma,struct folio_walk *fw,struct folio *folio,unsigned long opcode_vaddr,uprobe_opcode_t opcode) {
@@ -488,6 +488,12 @@ static int __uprobe_write_opcode(struct vm_area_struct *vma,*/ int uprobe_write_opcode (struct arch_uprobe *auprobe,struct vm_area_struct *vma,const unsigned long opcode_vaddr,uprobe_opcode_t opcode) +{ +return uprobe_write (auprobe, vma, opcode_vaddr, opcode, verify_opcode); +} + +int uprobe_write (struct arch_uprobe *auprobe,struct vm_area_struct *vma, +const unsigned long opcode_vaddr,uprobe_opcode_t opcode,uprobe_write_verify_t verify) {const unsigned long vaddr = opcode_vaddr &PAGE_MASK ;struct mm_struct *mm = vma->vm_mm;
@@ -510,7 +516,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct vm_area_struct *vma,* page that we can safely modify. Use FOLL_WRITE to trigger a write * fault if required. When unregistering, we might be lucky and the * anon page is already gone. So defer write faults until really -* required. Use FOLL_SPLIT_PMD, because __uprobe_write_opcode() +* required. Use FOLL_SPLIT_PMD, because __uprobe_write() * cannot deal with PMDs yet. */ if (is_register)
@@ -522,7 +528,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct vm_area_struct *vma, goto out; folio =page_folio (page); - ret =verify_opcode (page, opcode_vaddr, &opcode); + ret =verify (page, opcode_vaddr, &opcode);if (ret <=0 ) {folio_put (folio); goto out;
@@ -561,7 +567,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct vm_area_struct *vma,/* Walk the page tables again, to perform the actual update. */ if (folio_walk_start (&fw, vma, vaddr,0 )) {if (fw.page == page) - ret =__uprobe_write_opcode (vma, &fw, folio, opcode_vaddr, opcode); + ret =__uprobe_write (vma, &fw, folio, opcode_vaddr, opcode);folio_walk_end (&fw, vma); }
Signature · 1 line
2.49.0
Re: [PATCHv3 perf/core 04/22] uprobes: Add uprobe_write function
Quoted 10 lines · On Thu, 5 Jun 2025 15:23:31 +0200 Jiri Olsa <[email protected]> wrote:
Adding
uprobe_writefunction that does whatuprobe_write_opcodedid so far, but allows to pass verify callback function that checks the memory location before writing the opcode.It will be used in following changes to implement specific checking logic for instruction update.
The
uprobe_write_opcodenow callsuprobe_writewithverify_opcodeas the verify callback.
Looks good to me.
Thanks,
Quoted 86 lines
Acked-by Oleg Nesterov <[email protected]>Signed-off-by Jiri Olsa <[email protected]>--- include/linux/uprobes.h | 5 +++++ kernel/events/uprobes.c | 14 ++++++++++---- 2 files changed, 15 insertions(+), 4 deletions(-)include/linux/uprobes.h +5
-0@@ -187,6 +187,9 @@ struct uprobes_state {struct xol_area *xol_area; }; +typedef int (*uprobe_write_verify_t )(struct page *page,unsigned long vaddr, +uprobe_opcode_t *opcode); +extern void __init uprobes_init (void );extern int set_swbp (struct arch_uprobe *aup,struct vm_area_struct *vma,unsigned long vaddr);extern int set_orig_insn (struct arch_uprobe *aup,struct vm_area_struct *vma,unsigned long vaddr);@@ -195,6 +198,8 @@ extern bool is_trap_insn(uprobe_opcode_t *insn);extern unsigned long uprobe_get_swbp_addr (struct pt_regs *regs);extern unsigned long uprobe_get_trap_addr (struct pt_regs *regs);extern int uprobe_write_opcode (struct arch_uprobe *auprobe,struct vm_area_struct *vma,unsigned long vaddr,uprobe_opcode_t ); +extern int uprobe_write (struct arch_uprobe *auprobe,struct vm_area_struct *vma,const unsigned long opcode_vaddr, +uprobe_opcode_t opcode,uprobe_write_verify_t verify);extern struct uprobe *uprobe_register (struct inode *inode,loff_t offset,loff_t ref_ctr_offset,struct uprobe_consumer *uc);extern int uprobe_apply (struct uprobe *uprobe,struct uprobe_consumer *uc,bool );extern void uprobe_unregister_nosync (struct uprobe *uprobe,struct uprobe_consumer *uc);kernel/events/uprobes.c +10
-4@@ -399,7 +399,7 @@ static bool orig_page_is_identical(struct vm_area_struct *vma,return identical; } -static int __uprobe_write_opcode (struct vm_area_struct *vma, +static int __uprobe_write (struct vm_area_struct *vma,struct folio_walk *fw,struct folio *folio,unsigned long opcode_vaddr,uprobe_opcode_t opcode) {@@ -488,6 +488,12 @@ static int __uprobe_write_opcode(struct vm_area_struct *vma,*/ int uprobe_write_opcode (struct arch_uprobe *auprobe,struct vm_area_struct *vma,const unsigned long opcode_vaddr,uprobe_opcode_t opcode) +{ +return uprobe_write (auprobe, vma, opcode_vaddr, opcode, verify_opcode); +} + +int uprobe_write (struct arch_uprobe *auprobe,struct vm_area_struct *vma, +const unsigned long opcode_vaddr,uprobe_opcode_t opcode,uprobe_write_verify_t verify) {const unsigned long vaddr = opcode_vaddr &PAGE_MASK ;struct mm_struct *mm = vma->vm_mm;@@ -510,7 +516,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct vm_area_struct *vma,* page that we can safely modify. Use FOLL_WRITE to trigger a write * fault if required. When unregistering, we might be lucky and the * anon page is already gone. So defer write faults until really -* required. Use FOLL_SPLIT_PMD, because __uprobe_write_opcode() +* required. Use FOLL_SPLIT_PMD, because __uprobe_write() * cannot deal with PMDs yet. */ if (is_register)@@ -522,7 +528,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct vm_area_struct *vma, goto out; folio =page_folio (page); - ret =verify_opcode (page, opcode_vaddr, &opcode); + ret =verify (page, opcode_vaddr, &opcode);if (ret <=0 ) {folio_put (folio); goto out;@@ -561,7 +567,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct vm_area_struct *vma,/* Walk the page tables again, to perform the actual update. */ if (folio_walk_start (&fw, vma, vaddr,0 )) {if (fw.page == page) - ret =__uprobe_write_opcode (vma, &fw, folio, opcode_vaddr, opcode); + ret =__uprobe_write (vma, &fw, folio, opcode_vaddr, opcode);folio_walk_end (&fw, vma); }Signature · 1 line
2.49.0
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Jiri Olsa
reviewerPATCH v3 5/22 uprobes: Add nbytes argument to uprobe_write
In reply to Jiri OlsaAdding nbytes argument to uprobe_write and related functions as preparation for writing whole instructions in following changes.
Also renaming opcode arguments to insn, which seems to fit better.
--- include/linux/uprobes.h | 4 ++-- kernel/events/uprobes.c | 26 ++++++++++++++------------ 2 files changed, 16 insertions(+), 14 deletions(-)
include/linux/uprobes.h +2 -2
@@ -188,7 +188,7 @@ struct uprobes_state { };typedef int (*uprobe_write_verify_t )(struct page *page,unsigned long vaddr, -uprobe_opcode_t *opcode); +uprobe_opcode_t *insn,int nbytes);extern void __init uprobes_init (void );extern int set_swbp (struct arch_uprobe *aup,struct vm_area_struct *vma,unsigned long vaddr);
@@ -199,7 +199,7 @@ extern unsigned long uprobe_get_swbp_addr(struct pt_regs *regs);extern unsigned long uprobe_get_trap_addr (struct pt_regs *regs);extern int uprobe_write_opcode (struct arch_uprobe *auprobe,struct vm_area_struct *vma,unsigned long vaddr,uprobe_opcode_t );extern int uprobe_write (struct arch_uprobe *auprobe,struct vm_area_struct *vma,const unsigned long opcode_vaddr, -uprobe_opcode_t opcode,uprobe_write_verify_t verify); +uprobe_opcode_t *insn,int nbytes,uprobe_write_verify_t verify);extern struct uprobe *uprobe_register (struct inode *inode,loff_t offset,loff_t ref_ctr_offset,struct uprobe_consumer *uc);extern int uprobe_apply (struct uprobe *uprobe,struct uprobe_consumer *uc,bool );extern void uprobe_unregister_nosync (struct uprobe *uprobe,struct uprobe_consumer *uc);
kernel/events/uprobes.c +14 -12
@@ -191,7 +191,8 @@ static void copy_to_page(struct page *page, unsigned long vaddr, const void *srckunmap_atomic (kaddr); } -static int verify_opcode (struct page *page,unsigned long vaddr,uprobe_opcode_t *new_opcode) +static int verify_opcode (struct page *page,unsigned long vaddr,uprobe_opcode_t *insn, +int nbytes) {uprobe_opcode_t old_opcode;bool is_swbp;
@@ -208,7 +209,7 @@ static int verify_opcode(struct page *page, unsigned long vaddr, uprobe_opcode_tuprobe_copy_from_page (page, vaddr, &old_opcode,UPROBE_SWBP_INSN_SIZE ); is_swbp =is_swbp_insn (&old_opcode); -if (is_swbp_insn (new_opcode)) { +if (is_swbp_insn (insn)) {if (is_swbp)/* register: already installed? */ return 0 ; }else {
@@ -401,10 +402,10 @@ static bool orig_page_is_identical(struct vm_area_struct *vma,static int __uprobe_write (struct vm_area_struct *vma,struct folio_walk *fw,struct folio *folio, -unsigned long opcode_vaddr,uprobe_opcode_t opcode) +unsigned long insn_vaddr,uprobe_opcode_t *insn,int nbytes) { -const unsigned long vaddr = opcode_vaddr &PAGE_MASK ; -const bool is_register = !!is_swbp_insn (&opcode); +const unsigned long vaddr = insn_vaddr &PAGE_MASK ; +const bool is_register = !!is_swbp_insn (insn);bool pmd_mappable;/* For now, we'll only handle PTE-mapped folios. */
@@ -429,7 +430,7 @@ static int __uprobe_write(struct vm_area_struct *vma,*/ flush_cache_page (vma, vaddr,pte_pfn (fw->pte)); fw->pte =ptep_clear_flush (vma, vaddr, fw->ptep); -copy_to_page (fw->page, opcode_vaddr, &opcode,UPROBE_SWBP_INSN_SIZE ); +copy_to_page (fw->page, insn_vaddr, insn, nbytes);/* * When unregistering, we may only zap a PTE if uffd is disabled and
@@ -489,13 +490,14 @@ static int __uprobe_write(struct vm_area_struct *vma,int uprobe_write_opcode (struct arch_uprobe *auprobe,struct vm_area_struct *vma,const unsigned long opcode_vaddr,uprobe_opcode_t opcode) { -return uprobe_write (auprobe, vma, opcode_vaddr, opcode, verify_opcode); +return uprobe_write (auprobe, vma, opcode_vaddr, &opcode,UPROBE_SWBP_INSN_SIZE , verify_opcode); }int uprobe_write (struct arch_uprobe *auprobe,struct vm_area_struct *vma, -const unsigned long opcode_vaddr,uprobe_opcode_t opcode,uprobe_write_verify_t verify) +const unsigned long insn_vaddr,uprobe_opcode_t *insn,int nbytes, +uprobe_write_verify_t verify) { -const unsigned long vaddr = opcode_vaddr &PAGE_MASK ; +const unsigned long vaddr = insn_vaddr &PAGE_MASK ;struct mm_struct *mm = vma->vm_mm;struct uprobe *uprobe;int ret, is_register, ref_ctr_updated =0 ;
@@ -505,7 +507,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma,struct folio *folio;struct page *page; - is_register =is_swbp_insn (&opcode); + is_register =is_swbp_insn (insn); uprobe = container_of(auprobe, struct uprobe, arch);if (WARN_ON_ONCE (!is_cow_mapping (vma->vm_flags)))
@@ -528,7 +530,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma, goto out; folio =page_folio (page); - ret =verify (page, opcode_vaddr, &opcode); + ret =verify (page, insn_vaddr, insn, nbytes);if (ret <=0 ) {folio_put (folio); goto out;
@@ -567,7 +569,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma,/* Walk the page tables again, to perform the actual update. */ if (folio_walk_start (&fw, vma, vaddr,0 )) {if (fw.page == page) - ret =__uprobe_write (vma, &fw, folio, opcode_vaddr, opcode); + ret =__uprobe_write (vma, &fw, folio, insn_vaddr, insn, nbytes);folio_walk_end (&fw, vma); }
Signature · 1 line
2.49.0
Re: [PATCHv3 perf/core 05/22] uprobes: Add nbytes argument to uprobe_write
Quoted 5 lines · On Thu, 5 Jun 2025 15:23:32 +0200 Jiri Olsa <[email protected]> wrote:
Adding nbytes argument to
uprobe_writeand related functions as preparation for writing whole instructions in following changes.Also renaming opcode arguments to insn, which seems to fit better.
Looks good to me.
Thanks,
Quoted 124 lines
Acked-by Andrii Nakryiko <[email protected]>Acked-by Oleg Nesterov <[email protected]>Signed-off-by Jiri Olsa <[email protected]>--- include/linux/uprobes.h | 4 ++-- kernel/events/uprobes.c | 26 ++++++++++++++------------ 2 files changed, 16 insertions(+), 14 deletions(-)include/linux/uprobes.h +2
-2@@ -188,7 +188,7 @@ struct uprobes_state { };typedef int (*uprobe_write_verify_t )(struct page *page,unsigned long vaddr, -uprobe_opcode_t *opcode); +uprobe_opcode_t *insn,int nbytes);extern void __init uprobes_init (void );extern int set_swbp (struct arch_uprobe *aup,struct vm_area_struct *vma,unsigned long vaddr);@@ -199,7 +199,7 @@ extern unsigned long uprobe_get_swbp_addr(struct pt_regs *regs);extern unsigned long uprobe_get_trap_addr (struct pt_regs *regs);extern int uprobe_write_opcode (struct arch_uprobe *auprobe,struct vm_area_struct *vma,unsigned long vaddr,uprobe_opcode_t );extern int uprobe_write (struct arch_uprobe *auprobe,struct vm_area_struct *vma,const unsigned long opcode_vaddr, -uprobe_opcode_t opcode,uprobe_write_verify_t verify); +uprobe_opcode_t *insn,int nbytes,uprobe_write_verify_t verify);extern struct uprobe *uprobe_register (struct inode *inode,loff_t offset,loff_t ref_ctr_offset,struct uprobe_consumer *uc);extern int uprobe_apply (struct uprobe *uprobe,struct uprobe_consumer *uc,bool );extern void uprobe_unregister_nosync (struct uprobe *uprobe,struct uprobe_consumer *uc);kernel/events/uprobes.c +14
-12@@ -191,7 +191,8 @@ static void copy_to_page(struct page *page, unsigned long vaddr, const void *srckunmap_atomic (kaddr); } -static int verify_opcode (struct page *page,unsigned long vaddr,uprobe_opcode_t *new_opcode) +static int verify_opcode (struct page *page,unsigned long vaddr,uprobe_opcode_t *insn, +int nbytes) {uprobe_opcode_t old_opcode;bool is_swbp;@@ -208,7 +209,7 @@ static int verify_opcode(struct page *page, unsigned long vaddr, uprobe_opcode_tuprobe_copy_from_page (page, vaddr, &old_opcode,UPROBE_SWBP_INSN_SIZE ); is_swbp =is_swbp_insn (&old_opcode); -if (is_swbp_insn (new_opcode)) { +if (is_swbp_insn (insn)) {if (is_swbp)/* register: already installed? */ return 0 ; }else {@@ -401,10 +402,10 @@ static bool orig_page_is_identical(struct vm_area_struct *vma,static int __uprobe_write (struct vm_area_struct *vma,struct folio_walk *fw,struct folio *folio, -unsigned long opcode_vaddr,uprobe_opcode_t opcode) +unsigned long insn_vaddr,uprobe_opcode_t *insn,int nbytes) { -const unsigned long vaddr = opcode_vaddr &PAGE_MASK ; -const bool is_register = !!is_swbp_insn (&opcode); +const unsigned long vaddr = insn_vaddr &PAGE_MASK ; +const bool is_register = !!is_swbp_insn (insn);bool pmd_mappable;/* For now, we'll only handle PTE-mapped folios. */ @@ -429,7 +430,7 @@ static int __uprobe_write(struct vm_area_struct *vma,*/ flush_cache_page (vma, vaddr,pte_pfn (fw->pte)); fw->pte =ptep_clear_flush (vma, vaddr, fw->ptep); -copy_to_page (fw->page, opcode_vaddr, &opcode,UPROBE_SWBP_INSN_SIZE ); +copy_to_page (fw->page, insn_vaddr, insn, nbytes);/* * When unregistering, we may only zap a PTE if uffd is disabled and @@ -489,13 +490,14 @@ static int __uprobe_write(struct vm_area_struct *vma,int uprobe_write_opcode (struct arch_uprobe *auprobe,struct vm_area_struct *vma,const unsigned long opcode_vaddr,uprobe_opcode_t opcode) { -return uprobe_write (auprobe, vma, opcode_vaddr, opcode, verify_opcode); +return uprobe_write (auprobe, vma, opcode_vaddr, &opcode,UPROBE_SWBP_INSN_SIZE , verify_opcode); }int uprobe_write (struct arch_uprobe *auprobe,struct vm_area_struct *vma, -const unsigned long opcode_vaddr,uprobe_opcode_t opcode,uprobe_write_verify_t verify) +const unsigned long insn_vaddr,uprobe_opcode_t *insn,int nbytes, +uprobe_write_verify_t verify) { -const unsigned long vaddr = opcode_vaddr &PAGE_MASK ; +const unsigned long vaddr = insn_vaddr &PAGE_MASK ;struct mm_struct *mm = vma->vm_mm;struct uprobe *uprobe;int ret, is_register, ref_ctr_updated =0 ;@@ -505,7 +507,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma,struct folio *folio;struct page *page; - is_register =is_swbp_insn (&opcode); + is_register =is_swbp_insn (insn); uprobe = container_of(auprobe, struct uprobe, arch);if (WARN_ON_ONCE (!is_cow_mapping (vma->vm_flags)))@@ -528,7 +530,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma, goto out; folio =page_folio (page); - ret =verify (page, opcode_vaddr, &opcode); + ret =verify (page, insn_vaddr, insn, nbytes);if (ret <=0 ) {folio_put (folio); goto out;@@ -567,7 +569,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma,/* Walk the page tables again, to perform the actual update. */ if (folio_walk_start (&fw, vma, vaddr,0 )) {if (fw.page == page) - ret =__uprobe_write (vma, &fw, folio, opcode_vaddr, opcode); + ret =__uprobe_write (vma, &fw, folio, insn_vaddr, insn, nbytes);folio_walk_end (&fw, vma); }Signature · 1 line
2.49.0
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Jiri Olsa
reviewerPATCH v3 6/22 uprobes: Add is_register argument to uprobe_write and uprobe_write_opcode
In reply to Jiri OlsaThe uprobe_write has special path to restore the original page when we write original instruction back. This happens when uprobe_write detects that we want to write anything else but breakpoint instruction.
Moving the detection away and passing it to uprobe_write as argument, so it's possible to write different instructions (other than just breakpoint and rest).
--- arch/arm/probes/uprobes/core.c | 2 +- include/linux/uprobes.h | 5 +++-- kernel/events/uprobes.c | 21 +++++++++++---------- 3 files changed, 15 insertions(+), 13 deletions(-)
arch/arm/probes/uprobes/core.c +1 -1
@@ -30,7 +30,7 @@ int set_swbp(struct arch_uprobe *auprobe, struct vm_area_struct *vma, unsigned long vaddr) {return uprobe_write_opcode (auprobe, vma, vaddr, -__opcode_to_mem_arm (auprobe->bpinsn)); +__opcode_to_mem_arm (auprobe->bpinsn), true); }bool arch_uprobe_ignore (struct arch_uprobe *auprobe,struct pt_regs *regs)
include/linux/uprobes.h +3 -2
@@ -197,9 +197,10 @@ extern bool is_swbp_insn(uprobe_opcode_t *insn);extern bool is_trap_insn (uprobe_opcode_t *insn);extern unsigned long uprobe_get_swbp_addr (struct pt_regs *regs);extern unsigned long uprobe_get_trap_addr (struct pt_regs *regs); -extern int uprobe_write_opcode (struct arch_uprobe *auprobe,struct vm_area_struct *vma,unsigned long vaddr,uprobe_opcode_t ); +extern int uprobe_write_opcode (struct arch_uprobe *auprobe,struct vm_area_struct *vma,unsigned long vaddr,uprobe_opcode_t , +bool is_register);extern int uprobe_write (struct arch_uprobe *auprobe,struct vm_area_struct *vma,const unsigned long opcode_vaddr, -uprobe_opcode_t *insn,int nbytes,uprobe_write_verify_t verify); +uprobe_opcode_t *insn,int nbytes,uprobe_write_verify_t verify,bool is_register);extern struct uprobe *uprobe_register (struct inode *inode,loff_t offset,loff_t ref_ctr_offset,struct uprobe_consumer *uc);extern int uprobe_apply (struct uprobe *uprobe,struct uprobe_consumer *uc,bool );extern void uprobe_unregister_nosync (struct uprobe *uprobe,struct uprobe_consumer *uc);
kernel/events/uprobes.c +11 -10
@@ -402,10 +402,10 @@ static bool orig_page_is_identical(struct vm_area_struct *vma,static int __uprobe_write (struct vm_area_struct *vma,struct folio_walk *fw,struct folio *folio, -unsigned long insn_vaddr,uprobe_opcode_t *insn,int nbytes) +unsigned long insn_vaddr,uprobe_opcode_t *insn,int nbytes, +bool is_register) {const unsigned long vaddr = insn_vaddr &PAGE_MASK ; -const bool is_register = !!is_swbp_insn (insn);bool pmd_mappable;/* For now, we'll only handle PTE-mapped folios. */
@@ -488,26 +488,27 @@ static int __uprobe_write(struct vm_area_struct *vma,* Return 0 (success) or a negative errno. */ int uprobe_write_opcode (struct arch_uprobe *auprobe,struct vm_area_struct *vma, -const unsigned long opcode_vaddr,uprobe_opcode_t opcode) +const unsigned long opcode_vaddr,uprobe_opcode_t opcode, +bool is_register) { -return uprobe_write (auprobe, vma, opcode_vaddr, &opcode,UPROBE_SWBP_INSN_SIZE , verify_opcode); +return uprobe_write (auprobe, vma, opcode_vaddr, &opcode,UPROBE_SWBP_INSN_SIZE , + verify_opcode, is_register); }int uprobe_write (struct arch_uprobe *auprobe,struct vm_area_struct *vma,const unsigned long insn_vaddr,uprobe_opcode_t *insn,int nbytes, -uprobe_write_verify_t verify) +uprobe_write_verify_t verify,bool is_register) {const unsigned long vaddr = insn_vaddr &PAGE_MASK ;struct mm_struct *mm = vma->vm_mm;struct uprobe *uprobe; -int ret, is_register, ref_ctr_updated =0 ; +int ret, ref_ctr_updated =0 ;unsigned int gup_flags =FOLL_FORCE ;struct mmu_notifier_range range;struct folio_walk fw;struct folio *folio;struct page *page; - is_register =is_swbp_insn (insn); uprobe = container_of(auprobe, struct uprobe, arch);if (WARN_ON_ONCE (!is_cow_mapping (vma->vm_flags)))
@@ -569,7 +570,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma,/* Walk the page tables again, to perform the actual update. */ if (folio_walk_start (&fw, vma, vaddr,0 )) {if (fw.page == page) - ret =__uprobe_write (vma, &fw, folio, insn_vaddr, insn, nbytes); + ret =__uprobe_write (vma, &fw, folio, insn_vaddr, insn, nbytes, is_register);folio_walk_end (&fw, vma); }
@@ -611,7 +612,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma,int __weak set_swbp (struct arch_uprobe *auprobe,struct vm_area_struct *vma,unsigned long vaddr) { -return uprobe_write_opcode (auprobe, vma, vaddr,UPROBE_SWBP_INSN ); +return uprobe_write_opcode (auprobe, vma, vaddr,UPROBE_SWBP_INSN , true); }/**
@@ -627,7 +628,7 @@ int __weak set_orig_insn(struct arch_uprobe *auprobe, struct vm_area_struct *vma, unsigned long vaddr) {return uprobe_write_opcode (auprobe, vma, vaddr, - *(uprobe_opcode_t *)&auprobe->insn); + *(uprobe_opcode_t *)&auprobe->insn, false); }/* uprobe should have guaranteed positive refcount */
Signature · 1 line
2.49.0
Re: [PATCHv3 perf/core 06/22] uprobes: Add is_register argument to uprobe_write and uprobe_write_opcode
Quoted 7 lines · On Thu, 5 Jun 2025 15:23:33 +0200 Jiri Olsa <[email protected]> wrote:
The
uprobe_writehas special path to restore the original page when we write original instruction back. This happens whenuprobe_writedetects that we want to write anything else but breakpoint instruction.Moving the detection away and passing it to
uprobe_writeas argument, so it's possible to write different instructions (other than just breakpoint and rest).
Looks good to me.
Thanks,
Quoted 120 lines
Acked-by Andrii Nakryiko <[email protected]>Acked-by Oleg Nesterov <[email protected]>Signed-off-by Jiri Olsa <[email protected]>--- arch/arm/probes/uprobes/core.c | 2 +- include/linux/uprobes.h | 5 +++-- kernel/events/uprobes.c | 21 +++++++++++---------- 3 files changed, 15 insertions(+), 13 deletions(-)arch/arm/probes/uprobes/core.c +1
-1@@ -30,7 +30,7 @@ int set_swbp(struct arch_uprobe *auprobe, struct vm_area_struct *vma, unsigned long vaddr) {return uprobe_write_opcode (auprobe, vma, vaddr, -__opcode_to_mem_arm (auprobe->bpinsn)); +__opcode_to_mem_arm (auprobe->bpinsn), true); }bool arch_uprobe_ignore (struct arch_uprobe *auprobe,struct pt_regs *regs)include/linux/uprobes.h +3
-2@@ -197,9 +197,10 @@ extern bool is_swbp_insn(uprobe_opcode_t *insn);extern bool is_trap_insn (uprobe_opcode_t *insn);extern unsigned long uprobe_get_swbp_addr (struct pt_regs *regs);extern unsigned long uprobe_get_trap_addr (struct pt_regs *regs); -extern int uprobe_write_opcode (struct arch_uprobe *auprobe,struct vm_area_struct *vma,unsigned long vaddr,uprobe_opcode_t ); +extern int uprobe_write_opcode (struct arch_uprobe *auprobe,struct vm_area_struct *vma,unsigned long vaddr,uprobe_opcode_t , +bool is_register);extern int uprobe_write (struct arch_uprobe *auprobe,struct vm_area_struct *vma,const unsigned long opcode_vaddr, -uprobe_opcode_t *insn,int nbytes,uprobe_write_verify_t verify); +uprobe_opcode_t *insn,int nbytes,uprobe_write_verify_t verify,bool is_register);extern struct uprobe *uprobe_register (struct inode *inode,loff_t offset,loff_t ref_ctr_offset,struct uprobe_consumer *uc);extern int uprobe_apply (struct uprobe *uprobe,struct uprobe_consumer *uc,bool );extern void uprobe_unregister_nosync (struct uprobe *uprobe,struct uprobe_consumer *uc);kernel/events/uprobes.c +11
-10@@ -402,10 +402,10 @@ static bool orig_page_is_identical(struct vm_area_struct *vma,static int __uprobe_write (struct vm_area_struct *vma,struct folio_walk *fw,struct folio *folio, -unsigned long insn_vaddr,uprobe_opcode_t *insn,int nbytes) +unsigned long insn_vaddr,uprobe_opcode_t *insn,int nbytes, +bool is_register) {const unsigned long vaddr = insn_vaddr &PAGE_MASK ; -const bool is_register = !!is_swbp_insn (insn);bool pmd_mappable;/* For now, we'll only handle PTE-mapped folios. */ @@ -488,26 +488,27 @@ static int __uprobe_write(struct vm_area_struct *vma,* Return 0 (success) or a negative errno. */ int uprobe_write_opcode (struct arch_uprobe *auprobe,struct vm_area_struct *vma, -const unsigned long opcode_vaddr,uprobe_opcode_t opcode) +const unsigned long opcode_vaddr,uprobe_opcode_t opcode, +bool is_register) { -return uprobe_write (auprobe, vma, opcode_vaddr, &opcode,UPROBE_SWBP_INSN_SIZE , verify_opcode); +return uprobe_write (auprobe, vma, opcode_vaddr, &opcode,UPROBE_SWBP_INSN_SIZE , + verify_opcode, is_register); }int uprobe_write (struct arch_uprobe *auprobe,struct vm_area_struct *vma,const unsigned long insn_vaddr,uprobe_opcode_t *insn,int nbytes, -uprobe_write_verify_t verify) +uprobe_write_verify_t verify,bool is_register) {const unsigned long vaddr = insn_vaddr &PAGE_MASK ;struct mm_struct *mm = vma->vm_mm;struct uprobe *uprobe; -int ret, is_register, ref_ctr_updated =0 ; +int ret, ref_ctr_updated =0 ;unsigned int gup_flags =FOLL_FORCE ;struct mmu_notifier_range range;struct folio_walk fw;struct folio *folio;struct page *page; - is_register =is_swbp_insn (insn); uprobe = container_of(auprobe, struct uprobe, arch);if (WARN_ON_ONCE (!is_cow_mapping (vma->vm_flags)))@@ -569,7 +570,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma,/* Walk the page tables again, to perform the actual update. */ if (folio_walk_start (&fw, vma, vaddr,0 )) {if (fw.page == page) - ret =__uprobe_write (vma, &fw, folio, insn_vaddr, insn, nbytes); + ret =__uprobe_write (vma, &fw, folio, insn_vaddr, insn, nbytes, is_register);folio_walk_end (&fw, vma); }@@ -611,7 +612,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma,int __weak set_swbp (struct arch_uprobe *auprobe,struct vm_area_struct *vma,unsigned long vaddr) { -return uprobe_write_opcode (auprobe, vma, vaddr,UPROBE_SWBP_INSN ); +return uprobe_write_opcode (auprobe, vma, vaddr,UPROBE_SWBP_INSN , true); }/** @@ -627,7 +628,7 @@ int __weak set_orig_insn(struct arch_uprobe *auprobe, struct vm_area_struct *vma, unsigned long vaddr) {return uprobe_write_opcode (auprobe, vma, vaddr, - *(uprobe_opcode_t *)&auprobe->insn); + *(uprobe_opcode_t *)&auprobe->insn, false); }/* uprobe should have guaranteed positive refcount */ Signature · 1 line
2.49.0
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Jiri Olsa
reviewerPATCH v3 7/22 uprobes: Add do_ref_ctr argument to uprobe_write function
In reply to Jiri OlsaMaking update_ref_ctr call in uprobe_write conditional based on do_ref_ctr argument. This way we can use uprobe_write for instruction update without doing ref_ctr_offset update.
--- include/linux/uprobes.h | 2 +- kernel/events/uprobes.c | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-)
include/linux/uprobes.h +1 -1
@@ -200,7 +200,7 @@ extern unsigned long uprobe_get_trap_addr(struct pt_regs *regs);extern int uprobe_write_opcode (struct arch_uprobe *auprobe,struct vm_area_struct *vma,unsigned long vaddr,uprobe_opcode_t ,bool is_register);extern int uprobe_write (struct arch_uprobe *auprobe,struct vm_area_struct *vma,const unsigned long opcode_vaddr, -uprobe_opcode_t *insn,int nbytes,uprobe_write_verify_t verify,bool is_register); +uprobe_opcode_t *insn,int nbytes,uprobe_write_verify_t verify,bool is_register,bool do_update_ref_ctr);extern struct uprobe *uprobe_register (struct inode *inode,loff_t offset,loff_t ref_ctr_offset,struct uprobe_consumer *uc);extern int uprobe_apply (struct uprobe *uprobe,struct uprobe_consumer *uc,bool );extern void uprobe_unregister_nosync (struct uprobe *uprobe,struct uprobe_consumer *uc);
kernel/events/uprobes.c +4 -4
@@ -492,12 +492,12 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct vm_area_struct *vma, bool is_register) {return uprobe_write (auprobe, vma, opcode_vaddr, &opcode,UPROBE_SWBP_INSN_SIZE , - verify_opcode, is_register); + verify_opcode, is_register, true/* do_update_ref_ctr */ ); }int uprobe_write (struct arch_uprobe *auprobe,struct vm_area_struct *vma,const unsigned long insn_vaddr,uprobe_opcode_t *insn,int nbytes, -uprobe_write_verify_t verify,bool is_register) +uprobe_write_verify_t verify,bool is_register,bool do_update_ref_ctr) {const unsigned long vaddr = insn_vaddr &PAGE_MASK ;struct mm_struct *mm = vma->vm_mm;
@@ -538,7 +538,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma, }/* We are going to replace instruction, update ref_ctr. */ -if (!ref_ctr_updated && uprobe->ref_ctr_offset) { +if (do_update_ref_ctr && !ref_ctr_updated && uprobe->ref_ctr_offset) { ret =update_ref_ctr (uprobe, mm, is_register ?1 :-1 );if (ret) {folio_put (folio);
@@ -590,7 +590,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma, out:/* Revert back reference counter if instruction update failed. */ -if (ret <0 && ref_ctr_updated) +if (do_update_ref_ctr && ret <0 && ref_ctr_updated)update_ref_ctr (uprobe, mm, is_register ?-1 :1 );/* try collapse pmd for compound page */
Signature · 1 line
2.49.0
Masami Hiramatsu (Google)
maintainerRe: [PATCHv3 perf/core 07/22] uprobes: Add do_ref_ctr argument to uprobe_write function
Quoted 4 lines · On Thu, 5 Jun 2025 15:23:34 +0200 Jiri Olsa <[email protected]> wrote:
Making
update_ref_ctrcall inuprobe_writeconditional based ondo_ref_ctrargument. This way we can useuprobe_writefor instruction update without doingref_ctr_offsetupdate.
Can we just decouple this update from uprobe_write()? If we do this exclusively, I think we can do something like;
lock()update_ref_ctr(uprobe, mm, +1);
...
ret = uprobe_write();
...
if (ret < 0)
update_ref_ctr(uprobe, mm, -1);unlock()
Thank you,
Quoted 60 lines
Acked-by Oleg Nesterov <[email protected]>Signed-off-by Jiri Olsa <[email protected]>--- include/linux/uprobes.h | 2 +- kernel/events/uprobes.c | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-)include/linux/uprobes.h +1
-1@@ -200,7 +200,7 @@ extern unsigned long uprobe_get_trap_addr(struct pt_regs *regs);extern int uprobe_write_opcode (struct arch_uprobe *auprobe,struct vm_area_struct *vma,unsigned long vaddr,uprobe_opcode_t ,bool is_register);extern int uprobe_write (struct arch_uprobe *auprobe,struct vm_area_struct *vma,const unsigned long opcode_vaddr, -uprobe_opcode_t *insn,int nbytes,uprobe_write_verify_t verify,bool is_register); +uprobe_opcode_t *insn,int nbytes,uprobe_write_verify_t verify,bool is_register,bool do_update_ref_ctr);extern struct uprobe *uprobe_register (struct inode *inode,loff_t offset,loff_t ref_ctr_offset,struct uprobe_consumer *uc);extern int uprobe_apply (struct uprobe *uprobe,struct uprobe_consumer *uc,bool );extern void uprobe_unregister_nosync (struct uprobe *uprobe,struct uprobe_consumer *uc);kernel/events/uprobes.c +4
-4@@ -492,12 +492,12 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct vm_area_struct *vma, bool is_register) {return uprobe_write (auprobe, vma, opcode_vaddr, &opcode,UPROBE_SWBP_INSN_SIZE , - verify_opcode, is_register); + verify_opcode, is_register, true/* do_update_ref_ctr */ ); }int uprobe_write (struct arch_uprobe *auprobe,struct vm_area_struct *vma,const unsigned long insn_vaddr,uprobe_opcode_t *insn,int nbytes, -uprobe_write_verify_t verify,bool is_register) +uprobe_write_verify_t verify,bool is_register,bool do_update_ref_ctr) {const unsigned long vaddr = insn_vaddr &PAGE_MASK ;struct mm_struct *mm = vma->vm_mm;@@ -538,7 +538,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma, }/* We are going to replace instruction, update ref_ctr. */ -if (!ref_ctr_updated && uprobe->ref_ctr_offset) { +if (do_update_ref_ctr && !ref_ctr_updated && uprobe->ref_ctr_offset) { ret =update_ref_ctr (uprobe, mm, is_register ?1 :-1 );if (ret) {folio_put (folio);@@ -590,7 +590,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma, out:/* Revert back reference counter if instruction update failed. */ -if (ret <0 && ref_ctr_updated) +if (do_update_ref_ctr && ret <0 && ref_ctr_updated)update_ref_ctr (uprobe, mm, is_register ?-1 :1 );/* try collapse pmd for compound page */ Signature · 1 line
2.49.0
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Re: [PATCHv3 perf/core 07/22] uprobes: Add do_ref_ctr argument to uprobe_write function
Quoted 20 lines · Masami Hiramatsu
Quoted 4 lines · On Thu, 5 Jun 2025 15:23:34 +0200 Jiri Olsa <[email protected]> wrote:
Making
update_ref_ctrcall inuprobe_writeconditional based ondo_ref_ctrargument. This way we can useuprobe_writefor instruction update without doingref_ctr_offsetupdate.Can we just decouple this update from
uprobe_write()? If we do this exclusively, I think we can do something like;
lock()update_ref_ctr(uprobe, mm, +1);
...
ret =uprobe_write();
...
if (ret < 0)
update_ref_ctr(uprobe, mm, -1);unlock()
Quoted 1 line
Thank you,
it was the intention in the v1 but as Oleg pointed out [1] it won't work, because the set_orig_refctr part of this change does not know if the refctr should be really updated or not
while inside uprobe_write it happens right after verify callback and we make sure to update refctr only if the instruction was updated
thanks,
jirka
[1] https://lore.kernel.org/bpf/[email protected]/
Quoted 66 lines
Quoted 62 lines
Acked-by Oleg Nesterov <[email protected]>Signed-off-by Jiri Olsa <[email protected]>--- include/linux/uprobes.h | 2 +- kernel/events/uprobes.c | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-)include/linux/uprobes.h +1
-1@@ -200,7 +200,7 @@ extern unsigned long uprobe_get_trap_addr(struct pt_regs *regs);extern int uprobe_write_opcode (struct arch_uprobe *auprobe,struct vm_area_struct *vma,unsigned long vaddr,uprobe_opcode_t ,bool is_register);extern int uprobe_write (struct arch_uprobe *auprobe,struct vm_area_struct *vma,const unsigned long opcode_vaddr, -uprobe_opcode_t *insn,int nbytes,uprobe_write_verify_t verify,bool is_register); +uprobe_opcode_t *insn,int nbytes,uprobe_write_verify_t verify,bool is_register,bool do_update_ref_ctr);extern struct uprobe *uprobe_register (struct inode *inode,loff_t offset,loff_t ref_ctr_offset,struct uprobe_consumer *uc);extern int uprobe_apply (struct uprobe *uprobe,struct uprobe_consumer *uc,bool );extern void uprobe_unregister_nosync (struct uprobe *uprobe,struct uprobe_consumer *uc);kernel/events/uprobes.c +4
-4@@ -492,12 +492,12 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct vm_area_struct *vma, bool is_register) {return uprobe_write (auprobe, vma, opcode_vaddr, &opcode,UPROBE_SWBP_INSN_SIZE , - verify_opcode, is_register); + verify_opcode, is_register, true/* do_update_ref_ctr */ ); }int uprobe_write (struct arch_uprobe *auprobe,struct vm_area_struct *vma,const unsigned long insn_vaddr,uprobe_opcode_t *insn,int nbytes, -uprobe_write_verify_t verify,bool is_register) +uprobe_write_verify_t verify,bool is_register,bool do_update_ref_ctr) {const unsigned long vaddr = insn_vaddr &PAGE_MASK ;struct mm_struct *mm = vma->vm_mm;@@ -538,7 +538,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma, }/* We are going to replace instruction, update ref_ctr. */ -if (!ref_ctr_updated && uprobe->ref_ctr_offset) { +if (do_update_ref_ctr && !ref_ctr_updated && uprobe->ref_ctr_offset) { ret =update_ref_ctr (uprobe, mm, is_register ?1 :-1 );if (ret) {folio_put (folio);@@ -590,7 +590,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma, out:/* Revert back reference counter if instruction update failed. */ -if (ret <0 && ref_ctr_updated) +if (do_update_ref_ctr && ret <0 && ref_ctr_updated)update_ref_ctr (uprobe, mm, is_register ?-1 :1 );/* try collapse pmd for compound page */ Signature · 1 line
2.49.0Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Re: [PATCHv3 perf/core 07/22] uprobes: Add do_ref_ctr argument to uprobe_write function
Quoted 21 lines · On Wed, 25 Jun 2025 17:11:09 +0200 Jiri Olsa <[email protected]> wrote:
Quoted 20 lines · Masami Hiramatsu
Quoted 4 lines · On Thu, 5 Jun 2025 15:23:34 +0200 Jiri Olsa <[email protected]> wrote:
Making
update_ref_ctrcall inuprobe_writeconditional based ondo_ref_ctrargument. This way we can useuprobe_writefor instruction update without doingref_ctr_offsetupdate.Can we just decouple this update from
uprobe_write()? If we do this exclusively, I think we can do something like;
lock()update_ref_ctr(uprobe, mm, +1);
...
ret =uprobe_write();
...
if (ret < 0)
update_ref_ctr(uprobe, mm, -1);unlock()
Quoted 5 lines
Quoted 1 line
Thank you,
it was the intention in the v1 but as Oleg pointed out [1] it won't work, because the
set_orig_refctrpart of this change does not know if the refctr should be really updated or not
Ah, uprobe_unregister() can not check the given uprobe failed to update the code for each mm....
Quoted 3 lines
while inside
uprobe_writeit happens right after verify callback and we make sure to update refctr only if the instruction was updated
OK. But this is very complicated. Eventually I think uprobe should have a list of mm(or refcnt) to manage whether it is updated the refcounter.
Thanks,
Quoted 73 lines
thanks,
jirka[1] https://lore.kernel.org/bpf/[email protected]/
Quoted 66 lines
Quoted 62 lines
Acked-by Oleg Nesterov <[email protected]>Signed-off-by Jiri Olsa <[email protected]>--- include/linux/uprobes.h | 2 +- kernel/events/uprobes.c | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-)include/linux/uprobes.h +1
-1@@ -200,7 +200,7 @@ extern unsigned long uprobe_get_trap_addr(struct pt_regs *regs);extern int uprobe_write_opcode (struct arch_uprobe *auprobe,struct vm_area_struct *vma,unsigned long vaddr,uprobe_opcode_t ,bool is_register);extern int uprobe_write (struct arch_uprobe *auprobe,struct vm_area_struct *vma,const unsigned long opcode_vaddr, -uprobe_opcode_t *insn,int nbytes,uprobe_write_verify_t verify,bool is_register); +uprobe_opcode_t *insn,int nbytes,uprobe_write_verify_t verify,bool is_register,bool do_update_ref_ctr);extern struct uprobe *uprobe_register (struct inode *inode,loff_t offset,loff_t ref_ctr_offset,struct uprobe_consumer *uc);extern int uprobe_apply (struct uprobe *uprobe,struct uprobe_consumer *uc,bool );extern void uprobe_unregister_nosync (struct uprobe *uprobe,struct uprobe_consumer *uc);kernel/events/uprobes.c +4
-4@@ -492,12 +492,12 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct vm_area_struct *vma, bool is_register) {return uprobe_write (auprobe, vma, opcode_vaddr, &opcode,UPROBE_SWBP_INSN_SIZE , - verify_opcode, is_register); + verify_opcode, is_register, true/* do_update_ref_ctr */ ); }int uprobe_write (struct arch_uprobe *auprobe,struct vm_area_struct *vma,const unsigned long insn_vaddr,uprobe_opcode_t *insn,int nbytes, -uprobe_write_verify_t verify,bool is_register) +uprobe_write_verify_t verify,bool is_register,bool do_update_ref_ctr) {const unsigned long vaddr = insn_vaddr &PAGE_MASK ;struct mm_struct *mm = vma->vm_mm;@@ -538,7 +538,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma, }/* We are going to replace instruction, update ref_ctr. */ -if (!ref_ctr_updated && uprobe->ref_ctr_offset) { +if (do_update_ref_ctr && !ref_ctr_updated && uprobe->ref_ctr_offset) { ret =update_ref_ctr (uprobe, mm, is_register ?1 :-1 );if (ret) {folio_put (folio);@@ -590,7 +590,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma, out:/* Revert back reference counter if instruction update failed. */ -if (ret <0 && ref_ctr_updated) +if (do_update_ref_ctr && ret <0 && ref_ctr_updated)update_ref_ctr (uprobe, mm, is_register ?-1 :1 );/* try collapse pmd for compound page */ Signature · 1 line
2.49.0Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Jiri Olsa
reviewerPATCH v3 8/22 uprobes/x86: Add mapping for optimized uprobe trampolines
In reply to Jiri OlsaAdding support to add special mapping for user space trampoline with following functions:
uprobe_trampoline_get - find or add uprobe_trampoline uprobe_trampoline_put - remove or destroy uprobe_trampoline
The user space trampoline is exported as arch specific user space special mapping through tramp_mapping, which is initialized in following changes with new uprobe syscall.
The uprobe trampoline needs to be callable/reachable from the probed address, so while searching for available address we use is_reachable_by_call function to decide if the uprobe trampoline is callable from the probe address.
All uprobe_trampoline objects are stored in uprobes_state object and are cleaned up when the process mm_struct goes down. Adding new arch hooks for that, because this change is x86_64 specific.
Locking is provided by callers in following changes.
--- arch/x86/kernel/uprobes.c | 115 ++++++++++++++++++++++++++++++++++++++ include/linux/uprobes.h | 6 ++ kernel/events/uprobes.c | 10 ++++ kernel/fork.c | 1 + 4 files changed, 132 insertions(+)
arch/x86/kernel/uprobes.c +115 -0
@@ -608,6 +608,121 @@ static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) *sr = utask->autask.saved_scratch_register; } } + +static int tramp_mremap (const struct vm_special_mapping *sm,struct vm_area_struct *new_vma) +{ +return -EPERM ; +} + +static struct page *tramp_mapping_pages[2 ]__ro_after_init ; + +static struct vm_special_mapping tramp_mapping = { + .name ="[uprobes-trampoline]" , + .mremap = tramp_mremap, + .pages = tramp_mapping_pages, +}; + +struct uprobe_trampoline { +struct hlist_node node; +unsigned long vaddr; +}; + +static bool is_reachable_by_call (unsigned long vtramp,unsigned long vaddr) +{ +long delta = (long )(vaddr +5 - vtramp); + +return delta >=INT_MIN && delta <=INT_MAX ; +} + +static unsigned long find_nearest_page (unsigned long vaddr) +{ +struct vm_unmapped_area_info info = { + .length =PAGE_SIZE , + .align_mask = ~PAGE_MASK , + .flags =VM_UNMAPPED_AREA_TOPDOWN , + .low_limit =PAGE_SIZE , + .high_limit =ULONG_MAX , + }; +unsigned long limit, call_end = vaddr +5 ; + +if (!check_add_overflow (call_end,INT_MIN , &limit)) + info.low_limit = limit; +if (!check_add_overflow (call_end,INT_MAX , &limit)) + info.high_limit = limit; +return vm_unmapped_area (&info); +} + +static struct uprobe_trampoline *create_uprobe_trampoline (unsigned long vaddr) +{ +struct pt_regs *regs =task_pt_regs (current); +struct mm_struct *mm = current->mm; +struct uprobe_trampoline *tramp; +struct vm_area_struct *vma; + +if (!user_64bit_mode (regs)) +return NULL ; + + vaddr =find_nearest_page (vaddr); +if (IS_ERR_VALUE (vaddr)) +return NULL ; + + tramp =kzalloc (sizeof (*tramp),GFP_KERNEL ); +if (unlikely (!tramp)) +return NULL ; + + tramp->vaddr = vaddr; + vma =_install_special_mapping (mm, tramp->vaddr,PAGE_SIZE , +VM_READ |VM_EXEC |VM_MAYEXEC |VM_MAYREAD |VM_DONTCOPY |VM_IO , + &tramp_mapping); +if (IS_ERR (vma)) + goto free_area; +return tramp; + +free_area: +kfree (tramp); +return NULL ; +} + +__maybe_unused +static struct uprobe_trampoline *get_uprobe_trampoline (unsigned long vaddr,bool *new) +{ +struct uprobes_state *state = ¤t->mm->uprobes_state; +struct uprobe_trampoline *tramp =NULL ; + +hlist_for_each_entry (tramp, &state->head_tramps, node) { +if (is_reachable_by_call (tramp->vaddr, vaddr)) +return tramp; + } + + tramp =create_uprobe_trampoline (vaddr); +if (!tramp) +return NULL ; + + *new = true; +hlist_add_head (&tramp->node, &state->head_tramps); +return tramp; +} + +static void destroy_uprobe_trampoline (struct uprobe_trampoline *tramp) +{ +hlist_del (&tramp->node); +kfree (tramp); +} + +void arch_uprobe_init_state (struct mm_struct *mm) +{ +INIT_HLIST_HEAD (&mm->uprobes_state.head_tramps); +} + +void arch_uprobe_clear_state (struct mm_struct *mm) +{ +struct uprobes_state *state = &mm->uprobes_state; +struct uprobe_trampoline *tramp; +struct hlist_node *n; + +hlist_for_each_entry_safe (tramp, n, &state->head_tramps, node) +destroy_uprobe_trampoline (tramp); +}#else /* 32-bit: */ /* * No RIP-relative addressing on 32-bit
include/linux/uprobes.h +6 -0
@@ -17,6 +17,7 @@#include <linux/wait.h> #include <linux/timer.h> #include <linux/seqlock.h> +#include <linux/mutex.h> struct uprobe ;struct vm_area_struct ;
@@ -185,6 +186,9 @@ struct xol_area;struct uprobes_state {struct xol_area *xol_area; +#ifdef CONFIG_X86_64 +struct hlist_head head_tramps; +#endif }; typedef int (*uprobe_write_verify_t)(struct page *page, unsigned long vaddr,
@@ -233,6 +237,8 @@ extern void uprobe_handle_trampoline(struct pt_regs *regs);extern void *arch_uretprobe_trampoline (unsigned long *psize);extern unsigned long uprobe_get_trampoline_vaddr (void );extern void uprobe_copy_from_page (struct page *page,unsigned long vaddr,void *dst,int len); +extern void arch_uprobe_clear_state (struct mm_struct *mm); +extern void arch_uprobe_init_state (struct mm_struct *mm);#else /* !CONFIG_UPROBES */ struct uprobes_state { };
kernel/events/uprobes.c +10 -0
@@ -1802,6 +1802,14 @@ static struct xol_area *get_xol_area(void)return area; } +void __weak arch_uprobe_clear_state (struct mm_struct *mm) +{ +} + +void __weak arch_uprobe_init_state (struct mm_struct *mm) +{ +} +/* * uprobe_clear_state - Free the area allocated for slots. */
@@ -1813,6 +1821,8 @@ void uprobe_clear_state(struct mm_struct *mm)delayed_uprobe_remove (NULL , mm);mutex_unlock (&delayed_uprobe_lock); +arch_uprobe_clear_state (mm); +if (!area)return ;
kernel/fork.c +1 -0
@@ -1010,6 +1010,7 @@ static void mm_init_uprobes_state(struct mm_struct *mm) {#ifdef CONFIG_UPROBES mm->uprobes_state.xol_area =NULL ; +arch_uprobe_init_state (mm);#endif }
Signature · 1 line
2.49.0
Masami Hiramatsu (Google)
maintainerRe: [PATCHv3 perf/core 08/22] uprobes/x86: Add mapping for optimized uprobe trampolines
Quoted 54 lines · On Thu, 5 Jun 2025 15:23:35 +0200 Jiri Olsa <[email protected]> wrote:
Adding support to add special mapping for user space trampoline with following functions:
uprobe_trampoline_get- find or adduprobe_trampolineuprobe_trampoline_put- remove or destroyuprobe_trampolineThe user space trampoline is exported as arch specific user space special mapping through
tramp_mapping, which is initialized in following changes with new uprobe syscall.The uprobe trampoline needs to be callable/reachable from the probed address, so while searching for available address we use
is_reachable_by_callfunction to decide if the uprobe trampoline is callable from the probe address.All
uprobe_trampolineobjects are stored inuprobes_stateobject and are cleaned up when the processmm_structgoes down. Adding new arch hooks for that, because this change isx86_64specific.Locking is provided by callers in following changes.
Acked-by Oleg Nesterov <[email protected]>Signed-off-by Jiri Olsa <[email protected]>--- arch/x86/kernel/uprobes.c | 115 ++++++++++++++++++++++++++++++++++++++ include/linux/uprobes.h | 6 ++ kernel/events/uprobes.c | 10 ++++ kernel/fork.c | 1 + 4 files changed, 132 insertions(+)arch/x86/kernel/uprobes.c +17
-0@@ -608,6 +608,121 @@ static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) *sr = utask->autask.saved_scratch_register; } } + +static int tramp_mremap (const struct vm_special_mapping *sm,struct vm_area_struct *new_vma) +{ +return -EPERM ; +} + +static struct page *tramp_mapping_pages[2 ]__ro_after_init ; + +static struct vm_special_mapping tramp_mapping = { + .name ="[uprobes-trampoline]" , + .mremap = tramp_mremap, + .pages = tramp_mapping_pages, +}; + +struct uprobe_trampoline { +struct hlist_node node; +unsigned long vaddr;
arch/x86/kernel/uprobes.c static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) · line 637
Quoted 10 lines
arch/x86/kernel/uprobes.c +10
-0+}; + +static bool is_reachable_by_call (unsigned long vtramp,unsigned long vaddr) +{ +long delta = (long )(vaddr +5 - vtramp); + +return delta >=INT_MIN && delta <=INT_MAX ; +} + +static unsigned long find_nearest_page (unsigned long vaddr)
nit: this does not return the nearest one, but the highest one?
If you really need the nearest one, we need to call vm_unmapped_area() twice.
[low_limit, call_end] with TOPDOWN flag and
[call_end, high_limit] without TOPDOWN.
and choose the nearest one. But I don't think we need it.
arch/x86/kernel/uprobes.c static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) · line 644
Quoted 7 lines
arch/x86/kernel/uprobes.c +7
-0+{ +struct vm_unmapped_area_info info = { + .length =PAGE_SIZE , + .align_mask = ~PAGE_MASK , + .flags =VM_UNMAPPED_AREA_TOPDOWN , + .low_limit =PAGE_SIZE , + .high_limit =ULONG_MAX ,
Maybe "TASK_SIZE" is better than ULONG_MAX?
Quoted 20 lines
arch/x86/kernel/uprobes.c +20
-0+ }; +unsigned long limit, call_end = vaddr +5 ; + +if (!check_add_overflow (call_end,INT_MIN , &limit)) + info.low_limit = limit; +if (!check_add_overflow (call_end,INT_MAX , &limit)) + info.high_limit = limit; +return vm_unmapped_area (&info); +} + +static struct uprobe_trampoline *create_uprobe_trampoline (unsigned long vaddr) +{ +struct pt_regs *regs =task_pt_regs (current); +struct mm_struct *mm = current->mm; +struct uprobe_trampoline *tramp; +struct vm_area_struct *vma; + +if (!user_64bit_mode (regs)) +return NULL ; +
arch/x86/kernel/uprobes.c static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) · line 674
Quoted 10 lines
arch/x86/kernel/uprobes.c +10
-0+ vaddr =find_nearest_page (vaddr); +if (IS_ERR_VALUE (vaddr)) +return NULL ; + + tramp =kzalloc (sizeof (*tramp),GFP_KERNEL ); +if (unlikely (!tramp)) +return NULL ; + + tramp->vaddr = vaddr; + vma = _install_special_mapping(mm, tramp->vaddr, PAGE_SIZE,
Just make sure, this special mapped page is mapped 1 page for each uprobe? (I think uprobe syscall trampoline size is far smaller than the page size.)
arch/x86/kernel/uprobes.c static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) · line 678
Quoted 4 lines
arch/x86/kernel/uprobes.c +4
-0+VM_READ |VM_EXEC |VM_MAYEXEC |VM_MAYREAD |VM_DONTCOPY |VM_IO , + &tramp_mapping); +if (IS_ERR (vma)) + goto free_area;
nit: To simplify the code, instead of goto,
if (IS_ERR(vma)) {
kfree(tramp);
return NULL;
}Quoted 5 lines
arch/x86/kernel/uprobes.c +5
-0+return tramp; + +free_area: +kfree (tramp); +return NULL ;
arch/x86/kernel/uprobes.c static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) · line 693
Quoted 10 lines
arch/x86/kernel/uprobes.c +10
-0+} + +__maybe_unused +static struct uprobe_trampoline *get_uprobe_trampoline (unsigned long vaddr,bool *new) +{ +struct uprobes_state *state = ¤t->mm->uprobes_state; +struct uprobe_trampoline *tramp =NULL ; + +hlist_for_each_entry (tramp, &state->head_tramps, node) { +if (is_reachable_by_call (tramp->vaddr, vaddr))
This should set '*new = false;' here.
Quoted 6 lines
arch/x86/kernel/uprobes.c +6
-0+return tramp; + } + + tramp =create_uprobe_trampoline (vaddr); +if (!tramp) +return NULL ;
arch/x86/kernel/uprobes.c static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) · line 709
Quoted 10 lines
arch/x86/kernel/uprobes.c +10
-0+ + *new = true; +hlist_add_head (&tramp->node, &state->head_tramps); +return tramp; +} + +static void destroy_uprobe_trampoline (struct uprobe_trampoline *tramp) +{ +hlist_del (&tramp->node); +kfree (tramp);
Don't we need to unmap the tramp->vaddr?
Quoted 32 lines
arch/x86/kernel/uprobes.c +16
-0+} + +void arch_uprobe_init_state (struct mm_struct *mm) +{ +INIT_HLIST_HEAD (&mm->uprobes_state.head_tramps); +} + +void arch_uprobe_clear_state (struct mm_struct *mm) +{ +struct uprobes_state *state = &mm->uprobes_state; +struct uprobe_trampoline *tramp; +struct hlist_node *n; + +hlist_for_each_entry_safe (tramp, n, &state->head_tramps, node) +destroy_uprobe_trampoline (tramp); +}#else /* 32-bit: */ /* * No RIP-relative addressing on 32-bit include/linux/uprobes.h +1
-0@@ -17,6 +17,7 @@#include <linux/wait.h> #include <linux/timer.h> #include <linux/seqlock.h> +#include <linux/mutex.h> struct uprobe ;struct vm_area_struct ;
include/linux/uprobes.h struct xol_area; · line 189
Quoted 4 lines
include/linux/uprobes.h +1
-0@@ -185,6 +186,9 @@ struct xol_area;struct uprobes_state {struct xol_area *xol_area; +#ifdef CONFIG_X86_64
Maybe we can introduce struct arch_uprobe_state{} here?
Quoted 47 lines
include/linux/uprobes.h +4
-0+struct hlist_head head_tramps; +#endif }; typedef int (*uprobe_write_verify_t)(struct page *page, unsigned long vaddr,@@ -233,6 +237,8 @@ extern void uprobe_handle_trampoline(struct pt_regs *regs);extern void *arch_uretprobe_trampoline (unsigned long *psize);extern unsigned long uprobe_get_trampoline_vaddr (void );extern void uprobe_copy_from_page (struct page *page,unsigned long vaddr,void *dst,int len); +extern void arch_uprobe_clear_state (struct mm_struct *mm); +extern void arch_uprobe_init_state (struct mm_struct *mm);#else /* !CONFIG_UPROBES */ struct uprobes_state { };kernel/events/uprobes.c +10
-0@@ -1802,6 +1802,14 @@ static struct xol_area *get_xol_area(void)return area; } +void __weak arch_uprobe_clear_state (struct mm_struct *mm) +{ +} + +void __weak arch_uprobe_init_state (struct mm_struct *mm) +{ +} +/* * uprobe_clear_state - Free the area allocated for slots. */ @@ -1813,6 +1821,8 @@ void uprobe_clear_state(struct mm_struct *mm)delayed_uprobe_remove (NULL , mm);mutex_unlock (&delayed_uprobe_lock); +arch_uprobe_clear_state (mm); +if (!area)return ;kernel/fork.c +0
-0
kernel/fork.c static void mm_init_uprobes_state(struct mm_struct *mm) · line 1014
Quoted 5 lines
kernel/fork.c +1
-0@@ -1010,6 +1010,7 @@ static void mm_init_uprobes_state(struct mm_struct *mm) {#ifdef CONFIG_UPROBES mm->uprobes_state.xol_area =NULL ; +arch_uprobe_init_state (mm);#endif
Can't we make this uprobe_init_state(mm)?
Quoted 5 lines
kernel/fork.c +0
-0}Signature · 1 line
2.49.0
Thank you,
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Re: [PATCHv3 perf/core 08/22] uprobes/x86: Add mapping for optimized uprobe trampolines
Quoted 69 lines · Masami Hiramatsu
Quoted 54 lines · On Thu, 5 Jun 2025 15:23:35 +0200 Jiri Olsa <[email protected]> wrote:
Adding support to add special mapping for user space trampoline with following functions:
uprobe_trampoline_get- find or adduprobe_trampolineuprobe_trampoline_put- remove or destroyuprobe_trampolineThe user space trampoline is exported as arch specific user space special mapping through
tramp_mapping, which is initialized in following changes with new uprobe syscall.The uprobe trampoline needs to be callable/reachable from the probed address, so while searching for available address we use
is_reachable_by_callfunction to decide if the uprobe trampoline is callable from the probe address.All
uprobe_trampolineobjects are stored inuprobes_stateobject and are cleaned up when the processmm_structgoes down. Adding new arch hooks for that, because this change isx86_64specific.Locking is provided by callers in following changes.
Acked-by Oleg Nesterov <[email protected]>Signed-off-by Jiri Olsa <[email protected]>--- arch/x86/kernel/uprobes.c | 115 ++++++++++++++++++++++++++++++++++++++ include/linux/uprobes.h | 6 ++ kernel/events/uprobes.c | 10 ++++ kernel/fork.c | 1 + 4 files changed, 132 insertions(+)arch/x86/kernel/uprobes.c +17
-0@@ -608,6 +608,121 @@ static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) *sr = utask->autask.saved_scratch_register; } } + +static int tramp_mremap (const struct vm_special_mapping *sm,struct vm_area_struct *new_vma) +{ +return -EPERM ; +} + +static struct page *tramp_mapping_pages[2 ]__ro_after_init ; + +static struct vm_special_mapping tramp_mapping = { + .name ="[uprobes-trampoline]" , + .mremap = tramp_mremap, + .pages = tramp_mapping_pages, +}; + +struct uprobe_trampoline { +struct hlist_node node; +unsigned long vaddr;arch/x86/kernel/uprobes.c static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) · line 637
Quoted 10 lines
arch/x86/kernel/uprobes.c +10
-0+}; + +static bool is_reachable_by_call (unsigned long vtramp,unsigned long vaddr) +{ +long delta = (long )(vaddr +5 - vtramp); + +return delta >=INT_MIN && delta <=INT_MAX ; +} + +static unsigned long find_nearest_page (unsigned long vaddr)nit: this does not return the nearest one, but the highest one?
...
Quoted 8 lines
If you really need the nearest one, we need to call
vm_unmapped_area()twice.[
low_limit,call_end] with TOPDOWN flag and
[call_end,high_limit] without TOPDOWN.and choose the nearest one. But I don't think we need it.
ugh you're right, let's rename it.. find_reachable_page ?
Quoted 10 lines
arch/x86/kernel/uprobes.c static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) · line 644
Quoted 8 lines
arch/x86/kernel/uprobes.c +7
-0+{ +struct vm_unmapped_area_info info = { + .length =PAGE_SIZE , + .align_mask = ~PAGE_MASK , + .flags =VM_UNMAPPED_AREA_TOPDOWN , + .low_limit =PAGE_SIZE , + .high_limit =ULONG_MAX ,Maybe "
TASK_SIZE" is better thanULONG_MAX?
ok
Quoted 27 lines
Quoted 27 lines
arch/x86/kernel/uprobes.c +26
-0+ }; +unsigned long limit, call_end = vaddr +5 ; + +if (!check_add_overflow (call_end,INT_MIN , &limit)) + info.low_limit = limit; +if (!check_add_overflow (call_end,INT_MAX , &limit)) + info.high_limit = limit; +return vm_unmapped_area (&info); +} + +static struct uprobe_trampoline *create_uprobe_trampoline (unsigned long vaddr) +{ +struct pt_regs *regs =task_pt_regs (current); +struct mm_struct *mm = current->mm; +struct uprobe_trampoline *tramp; +struct vm_area_struct *vma; + +if (!user_64bit_mode (regs)) +return NULL ; + + vaddr =find_nearest_page (vaddr); +if (IS_ERR_VALUE (vaddr)) +return NULL ; + + tramp =kzalloc (sizeof (*tramp),GFP_KERNEL ); +if (unlikely (!tramp))
Quoted 8 lines
arch/x86/kernel/uprobes.c static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) · line 674
Quoted 4 lines
arch/x86/kernel/uprobes.c +4
-0+return NULL ; + + tramp->vaddr = vaddr; + vma = _install_special_mapping(mm, tramp->vaddr, PAGE_SIZE,Just make sure, this special mapped page is mapped 1 page for each uprobe? (I think uprobe syscall trampoline size is far smaller than the page size.)
so the trampoline is created for first uprobe within 4GB region of the probed address and will be reused by other uprobes in that region
Quoted 8 lines
arch/x86/kernel/uprobes.c static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) · line 678
Quoted 5 lines
arch/x86/kernel/uprobes.c +4
-0+VM_READ |VM_EXEC |VM_MAYEXEC |VM_MAYREAD |VM_DONTCOPY |VM_IO , + &tramp_mapping); +if (IS_ERR (vma)) + goto free_area;nit: To simplify the code, instead of goto,
Quoted 4 lines
if (IS_ERR(vma)) {
kfree(tramp);
return NULL;
}ok
Quoted 10 lines
Quoted 10 lines
arch/x86/kernel/uprobes.c +9
-0+return tramp; + +free_area: +kfree (tramp); +return NULL ; +} + +__maybe_unused +static struct uprobe_trampoline *get_uprobe_trampoline (unsigned long vaddr,bool *new)
Quoted 8 lines
arch/x86/kernel/uprobes.c static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) · line 693
Quoted 6 lines
arch/x86/kernel/uprobes.c +6
-0+{ +struct uprobes_state *state = ¤t->mm->uprobes_state; +struct uprobe_trampoline *tramp =NULL ; + +hlist_for_each_entry (tramp, &state->head_tramps, node) { +if (is_reachable_by_call (tramp->vaddr, vaddr))This should set '*new = false;' here.
right, will fix, thanks
Quoted 11 lines
Quoted 11 lines
arch/x86/kernel/uprobes.c +10
-0+return tramp; + } + + tramp =create_uprobe_trampoline (vaddr); +if (!tramp) +return NULL ; + + *new = true; +hlist_add_head (&tramp->node, &state->head_tramps); +return tramp;
Quoted 8 lines
arch/x86/kernel/uprobes.c static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) · line 709
Quoted 6 lines
arch/x86/kernel/uprobes.c +6
-0+} + +static void destroy_uprobe_trampoline (struct uprobe_trampoline *tramp) +{ +hlist_del (&tramp->node); +kfree (tramp);Don't we need to unmap the
tramp->vaddr?
that's tricky because we have no way to make sure the application is no longer executing the trampoline, it's described in the changelog of following patch:
uprobes/x86: Add support to optimize uprobes
...
We do not unmap and release uprobe trampoline when it's no longer needed, because there's no easy way to make sure none of the threads is still inside the trampoline. But we do not waste memory, because there's just single page for all the uprobe trampoline mappings.
We do waste frame on page mapping for every 4GB by keeping the uprobe trampoline page mapped, but that seems ok.
...
Quoted 34 lines
Quoted 34 lines
arch/x86/kernel/uprobes.c +16
-0+} + +void arch_uprobe_init_state (struct mm_struct *mm) +{ +INIT_HLIST_HEAD (&mm->uprobes_state.head_tramps); +} + +void arch_uprobe_clear_state (struct mm_struct *mm) +{ +struct uprobes_state *state = &mm->uprobes_state; +struct uprobe_trampoline *tramp; +struct hlist_node *n; + +hlist_for_each_entry_safe (tramp, n, &state->head_tramps, node) +destroy_uprobe_trampoline (tramp); +}#else /* 32-bit: */ /* * No RIP-relative addressing on 32-bit include/linux/uprobes.h +1
-0@@ -17,6 +17,7 @@#include <linux/wait.h> #include <linux/timer.h> #include <linux/seqlock.h> +#include <linux/mutex.h> struct uprobe ;struct vm_area_struct ;@@ -185,6 +186,9 @@ struct xol_area;
Quoted 5 lines
include/linux/uprobes.h struct xol_area; · line 189
Quoted 3 lines
include/linux/uprobes.h +1
-0struct uprobes_state {struct xol_area *xol_area; +#ifdef CONFIG_X86_64 Maybe we can introduce struct arch_uprobe_state{} here?
ok, on top of that Andrii also asked for [1]:
- alloc 'struct uprobes_state' for mm_struct only when needed
this could be part of that follow up? I'd rather not complicate this patchset any further
[1] https://lore.kernel.org/bpf/CAEf4BzY2zKPM9JHgn_wa8yCr8q5KntE5w8g=AoT2MnrD2Dx6gA@mail.gmail.com/
Quoted 43 lines
Quoted 43 lines
include/linux/uprobes.h +4
-0+struct hlist_head head_tramps; +#endif }; typedef int (*uprobe_write_verify_t)(struct page *page, unsigned long vaddr,@@ -233,6 +237,8 @@ extern void uprobe_handle_trampoline(struct pt_regs *regs);extern void *arch_uretprobe_trampoline (unsigned long *psize);extern unsigned long uprobe_get_trampoline_vaddr (void );extern void uprobe_copy_from_page (struct page *page,unsigned long vaddr,void *dst,int len); +extern void arch_uprobe_clear_state (struct mm_struct *mm); +extern void arch_uprobe_init_state (struct mm_struct *mm);#else /* !CONFIG_UPROBES */ struct uprobes_state { };kernel/events/uprobes.c +10
-0@@ -1802,6 +1802,14 @@ static struct xol_area *get_xol_area(void)return area; } +void __weak arch_uprobe_clear_state (struct mm_struct *mm) +{ +} + +void __weak arch_uprobe_init_state (struct mm_struct *mm) +{ +} +/* * uprobe_clear_state - Free the area allocated for slots. */ @@ -1813,6 +1821,8 @@ void uprobe_clear_state(struct mm_struct *mm)delayed_uprobe_remove (NULL , mm);mutex_unlock (&delayed_uprobe_lock); +arch_uprobe_clear_state (mm); +if (!area)return ;
Quoted 12 lines
kernel/fork.c static void mm_init_uprobes_state(struct mm_struct *mm) · line 1014
Quoted 10 lines
kernel/fork.c +1
-0@@ -1010,6 +1010,7 @@ static void mm_init_uprobes_state(struct mm_struct *mm) {#ifdef CONFIG_UPROBES mm->uprobes_state.xol_area =NULL ; +arch_uprobe_init_state (mm);#endif Can't we make this
uprobe_init_state(mm)?
hum, there are other mm_init_* functions around, I guess we should keep the same pattern?
unless you mean s/arch_uprobe_init_state/uprobe_init_state/ but that's arch code.. so probably not sure what you mean ;-)
thanks for review,
jirka
Masami Hiramatsu (Google)
maintainerRe: [PATCHv3 perf/core 08/22] uprobes/x86: Add mapping for optimized uprobe trampolines
Quoted 77 lines · On Wed, 25 Jun 2025 17:16:40 +0200 Jiri Olsa <[email protected]> wrote:
Quoted 69 lines · Masami Hiramatsu
Quoted 54 lines · On Thu, 5 Jun 2025 15:23:35 +0200 Jiri Olsa <[email protected]> wrote:
Adding support to add special mapping for user space trampoline with following functions:
uprobe_trampoline_get- find or adduprobe_trampolineuprobe_trampoline_put- remove or destroyuprobe_trampolineThe user space trampoline is exported as arch specific user space special mapping through
tramp_mapping, which is initialized in following changes with new uprobe syscall.The uprobe trampoline needs to be callable/reachable from the probed address, so while searching for available address we use
is_reachable_by_callfunction to decide if the uprobe trampoline is callable from the probe address.All
uprobe_trampolineobjects are stored inuprobes_stateobject and are cleaned up when the processmm_structgoes down. Adding new arch hooks for that, because this change isx86_64specific.Locking is provided by callers in following changes.
Acked-by Oleg Nesterov <[email protected]>Signed-off-by Jiri Olsa <[email protected]>--- arch/x86/kernel/uprobes.c | 115 ++++++++++++++++++++++++++++++++++++++ include/linux/uprobes.h | 6 ++ kernel/events/uprobes.c | 10 ++++ kernel/fork.c | 1 + 4 files changed, 132 insertions(+)arch/x86/kernel/uprobes.c +17
-0@@ -608,6 +608,121 @@ static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) *sr = utask->autask.saved_scratch_register; } } + +static int tramp_mremap (const struct vm_special_mapping *sm,struct vm_area_struct *new_vma) +{ +return -EPERM ; +} + +static struct page *tramp_mapping_pages[2 ]__ro_after_init ; + +static struct vm_special_mapping tramp_mapping = { + .name ="[uprobes-trampoline]" , + .mremap = tramp_mremap, + .pages = tramp_mapping_pages, +}; + +struct uprobe_trampoline { +struct hlist_node node; +unsigned long vaddr;arch/x86/kernel/uprobes.c static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) · line 637
Quoted 10 lines
arch/x86/kernel/uprobes.c +10
-0+}; + +static bool is_reachable_by_call (unsigned long vtramp,unsigned long vaddr) +{ +long delta = (long )(vaddr +5 - vtramp); + +return delta >=INT_MIN && delta <=INT_MAX ; +} + +static unsigned long find_nearest_page (unsigned long vaddr)nit: this does not return the nearest one, but the highest one?
...
Quoted 4 lines
If you really need the nearest one, we need to call
vm_unmapped_area()twice.
Quoted 6 lines
Quoted 4 lines
[
low_limit,call_end] with TOPDOWN flag and
[call_end,high_limit] without TOPDOWN.and choose the nearest one. But I don't think we need it.
ugh you're right, let's rename it..
find_reachable_page?
Hmm, I think it needs to be the nearest one if it is shared among several uprobes in the region. Or, it will choose the farthest address, which means it is easily out of range from the other uprobes.
Quoted 50 lines
Quoted 11 lines
arch/x86/kernel/uprobes.c static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) · line 644
Quoted 9 lines
arch/x86/kernel/uprobes.c +7
-0+{ +struct vm_unmapped_area_info info = { + .length =PAGE_SIZE , + .align_mask = ~PAGE_MASK , + .flags =VM_UNMAPPED_AREA_TOPDOWN , + .low_limit =PAGE_SIZE , + .high_limit =ULONG_MAX ,Maybe "
TASK_SIZE" is better thanULONG_MAX?ok
Quoted 27 lines
Quoted 27 lines
arch/x86/kernel/uprobes.c +26
-0+ }; +unsigned long limit, call_end = vaddr +5 ; + +if (!check_add_overflow (call_end,INT_MIN , &limit)) + info.low_limit = limit; +if (!check_add_overflow (call_end,INT_MAX , &limit)) + info.high_limit = limit; +return vm_unmapped_area (&info); +} + +static struct uprobe_trampoline *create_uprobe_trampoline (unsigned long vaddr) +{ +struct pt_regs *regs =task_pt_regs (current); +struct mm_struct *mm = current->mm; +struct uprobe_trampoline *tramp; +struct vm_area_struct *vma; + +if (!user_64bit_mode (regs)) +return NULL ; + + vaddr =find_nearest_page (vaddr); +if (IS_ERR_VALUE (vaddr)) +return NULL ; + + tramp =kzalloc (sizeof (*tramp),GFP_KERNEL ); +if (unlikely (!tramp))Quoted 8 lines
arch/x86/kernel/uprobes.c static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) · line 674
Quoted 4 lines
arch/x86/kernel/uprobes.c +4
-0+return NULL ; + + tramp->vaddr = vaddr; + vma = _install_special_mapping(mm, tramp->vaddr, PAGE_SIZE,Just make sure, this special mapped page is mapped 1 page for each uprobe? (I think uprobe syscall trampoline size is far smaller than the page size.)
Quoted 2 lines
so the trampoline is created for first uprobe within 4GB region of the probed address and will be reused by other uprobes in that region
Ah, OK.
Quoted 61 lines
Quoted 9 lines
arch/x86/kernel/uprobes.c static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) · line 678
Quoted 6 lines
arch/x86/kernel/uprobes.c +4
-0+VM_READ |VM_EXEC |VM_MAYEXEC |VM_MAYREAD |VM_DONTCOPY |VM_IO , + &tramp_mapping); +if (IS_ERR (vma)) + goto free_area;nit: To simplify the code, instead of goto,
Quoted 4 lines
if (IS_ERR(vma)) { kfree(tramp); return NULL; }ok
Quoted 10 lines
Quoted 10 lines
arch/x86/kernel/uprobes.c +9
-0+return tramp; + +free_area: +kfree (tramp); +return NULL ; +} + +__maybe_unused +static struct uprobe_trampoline *get_uprobe_trampoline (unsigned long vaddr,bool *new)Quoted 8 lines
arch/x86/kernel/uprobes.c static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) · line 693
Quoted 6 lines
arch/x86/kernel/uprobes.c +6
-0+{ +struct uprobes_state *state = ¤t->mm->uprobes_state; +struct uprobe_trampoline *tramp =NULL ; + +hlist_for_each_entry (tramp, &state->head_tramps, node) { +if (is_reachable_by_call (tramp->vaddr, vaddr))This should set '*new = false;' here.
right, will fix, thanks
Quoted 11 lines
Quoted 11 lines
arch/x86/kernel/uprobes.c +10
-0+return tramp; + } + + tramp =create_uprobe_trampoline (vaddr); +if (!tramp) +return NULL ; + + *new = true; +hlist_add_head (&tramp->node, &state->head_tramps); +return tramp;Quoted 8 lines
arch/x86/kernel/uprobes.c static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) · line 709
Quoted 6 lines
arch/x86/kernel/uprobes.c +6
-0+} + +static void destroy_uprobe_trampoline (struct uprobe_trampoline *tramp) +{ +hlist_del (&tramp->node); +kfree (tramp);Don't we need to unmap the
tramp->vaddr?that's tricky because we have no way to make sure the application is no longer executing the trampoline, it's described in the changelog of following patch:
Quoted 9 lines
uprobes/x86: Add support to optimize uprobes
...
We do not unmap and release uprobe trampoline when it's no longer needed, because there's no easy way to make sure none of the threads is still inside the trampoline. But we do not waste memory, because there's just single page for all the uprobe trampoline mappings.
I think we should put this as a code comment.
Quoted 2 lines
We do waste frame on page mapping for every 4GB by keeping the uprobe trampoline page mapped, but that seems ok.
Hmm, this is not right with the current find_nearest_page(), because it always finds a page from the farthest +2GB range until it is full. Thus, in the worst case, if we hits uprobes with the order of uprobe0 -> 1 -> 2 which is put as below;
0x0abc0004 [uprobe2]
...0x0abc2004 [uprobe1]
...0x0abc4004 [uprobe0]
Then the trampoline pages can be allocated as below.
0x8abc0000 [uprobe_tramp2]
[gap]0x8abc2000 [uprobe_tramp1]
[gap]0x8abc4000 [uprobe_tramp0]
Using true "find_nearest_page()", this will be mitigated. But not allocated for "every 4GB". So I think we should drop that part from the comment :)
Quoted 43 lines
...Quoted 34 lines
Quoted 34 lines
arch/x86/kernel/uprobes.c +16
-0+} + +void arch_uprobe_init_state (struct mm_struct *mm) +{ +INIT_HLIST_HEAD (&mm->uprobes_state.head_tramps); +} + +void arch_uprobe_clear_state (struct mm_struct *mm) +{ +struct uprobes_state *state = &mm->uprobes_state; +struct uprobe_trampoline *tramp; +struct hlist_node *n; + +hlist_for_each_entry_safe (tramp, n, &state->head_tramps, node) +destroy_uprobe_trampoline (tramp); +}#else /* 32-bit: */ /* * No RIP-relative addressing on 32-bit include/linux/uprobes.h +1
-0@@ -17,6 +17,7 @@#include <linux/wait.h> #include <linux/timer.h> #include <linux/seqlock.h> +#include <linux/mutex.h> struct uprobe ;struct vm_area_struct ;@@ -185,6 +186,9 @@ struct xol_area;Quoted 5 lines
include/linux/uprobes.h struct xol_area; · line 189
Quoted 3 lines
include/linux/uprobes.h +1
-0struct uprobes_state {struct xol_area *xol_area; +#ifdef CONFIG_X86_64 Maybe we can introduce struct arch_uprobe_state{} here?
Quoted 7 lines
ok, on top of that Andrii also asked for [1]:
- alloc 'structuprobes_state' formm_structonly when neededthis could be part of that follow up? I'd rather not complicate this patchset any further
[1] https://lore.kernel.org/bpf/CAEf4BzY2zKPM9JHgn_wa8yCr8q5KntE5w8g=AoT2MnrD2Dx6gA@mail.gmail.com/
Hmm, OK. But if you need to avoid #ifdef CONFIG_<arch>, you can use include/asm-generic to override macros.
struct uprobes_state {
struct xol_area *xol_area;
uprobe_arch_specific_data
};
--- include/asm-generic/uprobes.h
#define uprobe_arch_specific_data
--- arch/x86/include/asm/uprobes.h
#undef uprobe_arch_specific_data
#define uprobe_arch_specific_data \ struct hlist_head head_tramps;
Quoted 57 lines
Quoted 44 lines
Quoted 44 lines
include/linux/uprobes.h +4
-0+struct hlist_head head_tramps; +#endif }; typedef int (*uprobe_write_verify_t)(struct page *page, unsigned long vaddr,@@ -233,6 +237,8 @@ extern void uprobe_handle_trampoline(struct pt_regs *regs);extern void *arch_uretprobe_trampoline (unsigned long *psize);extern unsigned long uprobe_get_trampoline_vaddr (void );extern void uprobe_copy_from_page (struct page *page,unsigned long vaddr,void *dst,int len); +extern void arch_uprobe_clear_state (struct mm_struct *mm); +extern void arch_uprobe_init_state (struct mm_struct *mm);#else /* !CONFIG_UPROBES */ struct uprobes_state { };kernel/events/uprobes.c +10
-0@@ -1802,6 +1802,14 @@ static struct xol_area *get_xol_area(void)return area; } +void __weak arch_uprobe_clear_state (struct mm_struct *mm) +{ +} + +void __weak arch_uprobe_init_state (struct mm_struct *mm) +{ +} +/* * uprobe_clear_state - Free the area allocated for slots. */ @@ -1813,6 +1821,8 @@ void uprobe_clear_state(struct mm_struct *mm)delayed_uprobe_remove (NULL , mm);mutex_unlock (&delayed_uprobe_lock); +arch_uprobe_clear_state (mm); +if (!area)return ;Quoted 12 lines
kernel/fork.c static void mm_init_uprobes_state(struct mm_struct *mm) · line 1014
Quoted 10 lines
kernel/fork.c +1
-0@@ -1010,6 +1010,7 @@ static void mm_init_uprobes_state(struct mm_struct *mm) {#ifdef CONFIG_UPROBES mm->uprobes_state.xol_area =NULL ; +arch_uprobe_init_state (mm);#endif Can't we make this
uprobe_init_state(mm)?
Quoted 5 lines
hum, there are other mm_init_* functions around, I guess we should keep the same pattern?
unless you mean s/arch_uprobe_init_state/uprobe_init_state/ but that's arch code.. so probably not sure what you mean ;-)
Ah, I misunderstood. Yeah, this part is good to me.
Thank you!
Quoted 3 lines
thanks for review,
jirka
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Re: [PATCHv3 perf/core 08/22] uprobes/x86: Add mapping for optimized uprobe trampolines
On Fri, Jun 27, 2025 at 03:01:45PM +0900, Masami Hiramatsu wrote:
SNIP
Quoted 28 lines
Quoted 28 lines
Quoted 11 lines
Quoted 11 lines
arch/x86/kernel/uprobes.c +10
-0+return tramp; + } + + tramp =create_uprobe_trampoline (vaddr); +if (!tramp) +return NULL ; + + *new = true; +hlist_add_head (&tramp->node, &state->head_tramps); +return tramp;Quoted 8 lines
arch/x86/kernel/uprobes.c static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) · line 709
Quoted 6 lines
arch/x86/kernel/uprobes.c +6
-0+} + +static void destroy_uprobe_trampoline (struct uprobe_trampoline *tramp) +{ +hlist_del (&tramp->node); +kfree (tramp);Don't we need to unmap the
tramp->vaddr?that's tricky because we have no way to make sure the application is no longer executing the trampoline, it's described in the changelog of following patch:
uprobes/x86: Add support to optimize uprobes
...
Quoted 7 lines
Quoted 5 lines
We do not unmap and release uprobe trampoline when it's no longer needed, because there's no easy way to make sure none of the threads is still inside the trampoline. But we do not waste memory, because there's just single page for all the uprobe trampoline mappings.
I think we should put this as a code comment.
ok
Quoted 23 lines
Quoted 3 lines
We do waste frame on page mapping for every 4GB by keeping the uprobe trampoline page mapped, but that seems ok.
Hmm, this is not right with the current
find_nearest_page(), because it always finds a page from the farthest +2GB range until it is full. Thus, in the worst case, if we hits uprobes with the order of uprobe0 -> 1 -> 2 which is put as below;
0x0abc0004[uprobe2]
...0x0abc2004[uprobe1]
...0x0abc4004[uprobe0]Then the trampoline pages can be allocated as below.
0x8abc0000[uprobe_tramp2]
[gap]0x8abc2000[uprobe_tramp1]
[gap]0x8abc4000[uprobe_tramp0]
Quoted 3 lines
Using true "
find_nearest_page()", this will be mitigated. But not allocated for "every 4GB". So I think we should drop that part from the comment :)
I think you're right, it's better to start with nearest page, will change it in new version
SNIP
Quoted 39 lines
Quoted 20 lines
Quoted 14 lines
Quoted 14 lines
include/linux/uprobes.h +1
-0@@ -17,6 +17,7 @@#include <linux/wait.h> #include <linux/timer.h> #include <linux/seqlock.h> +#include <linux/mutex.h> struct uprobe ;struct vm_area_struct ;@@ -185,6 +186,9 @@ struct xol_area;Quoted 5 lines
include/linux/uprobes.h struct xol_area; · line 189
Quoted 3 lines
include/linux/uprobes.h +1
-0struct uprobes_state {struct xol_area *xol_area; +#ifdef CONFIG_X86_64 Maybe we can introduce struct arch_uprobe_state{} here?
Quoted 7 lines
ok, on top of that Andrii also asked for [1]:
- alloc 'structuprobes_state' formm_structonly when neededthis could be part of that follow up? I'd rather not complicate this patchset any further
[1] https://lore.kernel.org/bpf/CAEf4BzY2zKPM9JHgn_wa8yCr8q5KntE5w8g=AoT2MnrD2Dx6gA@mail.gmail.com/
Hmm, OK. But if you need to avoid #ifdef CONFIG_<arch>, you can use
include/asm-genericto override macros.struct
uprobes_state{
structxol_area*xol_area;
uprobe_arch_specific_data
};---
include/asm-generic/uprobes.h
Quoted 7 lines
#define
uprobe_arch_specific_data---
arch/x86/include/asm/uprobes.h#undef
uprobe_arch_specific_data#define uprobe_arch_specific_data \ struct hlist_head head_tramps;
ok
SNIP
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
kernel/fork.c static void mm_init_uprobes_state(struct mm_struct *mm) · line 1014
Quoted 10 lines
kernel/fork.c +1
-0@@ -1010,6 +1010,7 @@ static void mm_init_uprobes_state(struct mm_struct *mm) {#ifdef CONFIG_UPROBES mm->uprobes_state.xol_area =NULL ; +arch_uprobe_init_state (mm);#endif Can't we make this
uprobe_init_state(mm)?
Quoted 7 lines
Quoted 5 lines
hum, there are other mm_init_* functions around, I guess we should keep the same pattern?
unless you mean s/arch_uprobe_init_state/uprobe_init_state/ but that's arch code.. so probably not sure what you mean ;-)
Ah, I misunderstood. Yeah, this part is good to me.
ok, thanks
jirka
Re: [PATCHv3 perf/core 08/22] uprobes/x86: Add mapping for optimized uprobe trampolines
Quoted 65 lines · Jiri Olsa
On Fri, Jun 27, 2025 at 03:01:45PM +0900, Masami Hiramatsu wrote:
SNIP
Quoted 28 lines
Quoted 28 lines
Quoted 11 lines
Quoted 11 lines
arch/x86/kernel/uprobes.c +10
-0+return tramp; + } + + tramp =create_uprobe_trampoline (vaddr); +if (!tramp) +return NULL ; + + *new = true; +hlist_add_head (&tramp->node, &state->head_tramps); +return tramp;Quoted 8 lines
arch/x86/kernel/uprobes.c static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) · line 709
Quoted 6 lines
arch/x86/kernel/uprobes.c +6
-0+} + +static void destroy_uprobe_trampoline (struct uprobe_trampoline *tramp) +{ +hlist_del (&tramp->node); +kfree (tramp);Don't we need to unmap the
tramp->vaddr?that's tricky because we have no way to make sure the application is no longer executing the trampoline, it's described in the changelog of following patch:
uprobes/x86: Add support to optimize uprobes
...
Quoted 7 lines
Quoted 5 lines
We do not unmap and release uprobe trampoline when it's no longer needed, because there's no easy way to make sure none of the threads is still inside the trampoline. But we do not waste memory, because there's just single page for all the uprobe trampoline mappings.
I think we should put this as a code comment.
ok
Quoted 23 lines
Quoted 3 lines
We do waste frame on page mapping for every 4GB by keeping the uprobe trampoline page mapped, but that seems ok.
Hmm, this is not right with the current
find_nearest_page(), because it always finds a page from the farthest +2GB range until it is full. Thus, in the worst case, if we hits uprobes with the order of uprobe0 -> 1 -> 2 which is put as below;
0x0abc0004[uprobe2]
...0x0abc2004[uprobe1]
...0x0abc4004[uprobe0]Then the trampoline pages can be allocated as below.
0x8abc0000[uprobe_tramp2]
[gap]0x8abc2000[uprobe_tramp1]
[gap]0x8abc4000[uprobe_tramp0]
Quoted 6 lines
Quoted 3 lines
Using true "
find_nearest_page()", this will be mitigated. But not allocated for "every 4GB". So I think we should drop that part from the comment :)I think you're right, it's better to start with nearest page, will change it in new version
I wonder we could actualy use page every 4GB (+-) code below seems to work, wdyt?
thanks,
jirka
---
+#define __4GB (1UL << 32)
+#define MASK_4GB ~(__4GB - 1)
+#define PAGE_COUNT(addr) ((addr & ~MASK_4GB) >> PAGE_SHIFT) +
+static unsigned long find_nearest_page(unsigned long vaddr) +{
+ struct vm_unmapped_area_info info = {
+ .length = PAGE_SIZE,
+ .align_mask = ~PAGE_MASK,
+ };
+ unsigned long limit, low_limit = PAGE_SIZE, high_limit = TASK_SIZE;
+ unsigned long cross_4GB, low_4GB, high_4GB;
+ unsigned long low_tramp, high_tramp;
+ unsigned long call_end = vaddr + 5;
+
+ /*
+ * The idea is to create a trampoline every 4GB, so we need to find
+ * free page closest to the 4GB alignment. We find intersecting 4GB
+ * alignment address and search up and down to find the closest free
+ * page.
+ */
+
+ low_4GB = call_end & MASK_4GB;
+ high_4GB = low_4GB + __4GB;
+
+ /* Restrict limits to be within (PAGE_SIZE,TASK_SIZE) boundaries. */
+ if (!check_add_overflow(call_end, INT_MIN, &limit))
+ low_limit = limit;
+ if (low_limit == PAGE_SIZE)
+ low_4GB = low_limit;
+
+ high_limit = call_end + INT_MAX;
+ if (high_limit > TASK_SIZE)
+ high_limit = high_4GB = TASK_SIZE;
+
+ /* Get 4GB alligned address that's within 2GB distance from call_end */
+ if (low_limit <= low_4GB)
+ cross_4GB = low_4GB;
+ else
+ cross_4GB = high_4GB;
+
+ /* Search up from intersecting 4GB alignment address. */
+ info.low_limit = cross_4GB;
+ info.high_limit = high_limit;
+ high_tramp = vm_unmapped_area(&info);
+
+ /* Search down from intersecting 4GB alignment address. */
+ info.low_limit = low_limit;
+ info.high_limit = cross_4GB;
+ info.flags = VM_UNMAPPED_AREA_TOPDOWN;
+ low_tramp = vm_unmapped_area(&info);
+
+ if (IS_ERR_VALUE(high_tramp) && IS_ERR_VALUE(low_tramp))
+ return -ENOMEM;
+ if (IS_ERR_VALUE(high_tramp))
+ return low_tramp;
+ if (IS_ERR_VALUE(low_tramp))
+ return high_tramp;
+
+ /* Return address that's closest to the 4GB alignment address. */
+ if (cross_4GB - low_tramp < high_tramp - cross_4GB)
+ return low_tramp;
+ return high_tramp;
+}
Jiri Olsa
reviewerPATCH v3 9/22 uprobes/x86: Add uprobe syscall to speed up uprobe
In reply to Jiri OlsaAdding new uprobe syscall that calls uprobe handlers for given 'breakpoint' address.
The idea is that the 'breakpoint' address calls the user space trampoline which executes the uprobe syscall.
The syscall handler reads the return address of the initial call to retrieve the original 'breakpoint' address. With this address we find the related uprobe object and call its consumers.
Adding the arch_uprobe_trampoline_mapping function that provides uprobe trampoline mapping. This mapping is backed with one global page initialized at __init time and shared by the all the mapping instances.
We do not allow to execute uprobe syscall if the caller is not from uprobe trampoline mapping.
The uprobe syscall ensures the consumer (bpf program) sees registers values in the state before the trampoline was called.
--- arch/x86/entry/syscalls/syscall_64.tbl | 1 + arch/x86/kernel/uprobes.c | 122 +++++++++++++++++++++++++ include/linux/syscalls.h | 2 + include/linux/uprobes.h | 1 + kernel/events/uprobes.c | 17 ++++ kernel/sys_ni.c | 1 + 6 files changed, 144 insertions(+)
arch/x86/entry/syscalls/syscall_64.tbl +1 -0
@@ -345,6 +345,7 @@ 333 common io_pgetevents sys_io_pgetevents 334 common rseq sys_rseq 335 common uretprobe sys_uretprobe +336 common uprobe sys_uprobe # don't use numbers 387 through 423, add new calls after the last # 'common' entry 424 common pidfd_send_signal sys_pidfd_send_signal
arch/x86/kernel/uprobes.c +122 -0
128 lines not on this page · raw message
include/linux/syscalls.h +2 -0
@@ -998,6 +998,8 @@ asmlinkage long sys_ioperm(unsigned long from, unsigned long num, int on);asmlinkage long sys_uretprobe (void ); +asmlinkage long sys_uprobe (void ); +/* pciconfig: alpha, arm, arm64, ia64, sparc */ asmlinkage long sys_pciconfig_read(unsigned long bus, unsigned long dfn, unsigned long off, unsigned long len,
include/linux/uprobes.h +1 -0
@@ -239,6 +239,7 @@ extern unsigned long uprobe_get_trampoline_vaddr(void);extern void uprobe_copy_from_page (struct page *page,unsigned long vaddr,void *dst,int len);extern void arch_uprobe_clear_state (struct mm_struct *mm);extern void arch_uprobe_init_state (struct mm_struct *mm); +extern void handle_syscall_uprobe (struct pt_regs *regs,unsigned long bp_vaddr);#else /* !CONFIG_UPROBES */ struct uprobes_state { };
kernel/events/uprobes.c +17 -0
@@ -2772,6 +2772,23 @@ static void handle_swbp(struct pt_regs *regs)rcu_read_unlock_trace (); } +void handle_syscall_uprobe (struct pt_regs *regs,unsigned long bp_vaddr) +{ +struct uprobe *uprobe; +int is_swbp; + +guard (rcu_tasks_trace)(); + + uprobe =find_active_uprobe_rcu (bp_vaddr, &is_swbp); +if (!uprobe) +return ; +if (!get_utask ()) +return ; +if (arch_uprobe_ignore (&uprobe->arch, regs)) +return ; +handler_chain (uprobe, regs); +} +/* * Perform required fix-ups and disable singlestep. * Allow pending signals to take effect.
kernel/sys_ni.c +1 -0
@@ -392,3 +392,4 @@ COND_SYSCALL(setuid16);COND_SYSCALL (rseq);COND_SYSCALL (uretprobe); +COND_SYSCALL (uprobe);
Signature · 1 line
2.49.0
Jiri Olsa
reviewerPATCH v3 10/22 uprobes/x86: Add support to optimize uprobes
In reply to Jiri OlsaPutting together all the previously added pieces to support optimized uprobes on top of 5-byte nop instruction.
The current uprobe execution goes through following:
- installs breakpoint instruction over original instruction
- exception handler hit and calls related uprobe consumers
- and either simulates original instruction or does out of line single step execution of it
- returns to user space
The optimized uprobe path does following:
- checks the original instruction is 5-byte nop (plus other checks)
- adds (or uses existing) user space trampoline with uprobe syscall
- overwrites original instruction (5-byte nop) with call to user space trampoline
- the user space trampoline executes uprobe syscall that calls related uprobe consumers
- trampoline returns back to next instruction
This approach won't speed up all uprobes as it's limited to using nop5 as original instruction, but we plan to use nop5 as USDT probe instruction (which currently uses single byte nop) and speed up the USDT probes.
The arch_uprobe_optimize triggers the uprobe optimization and is called after first uprobe hit. I originally had it called on uprobe installation but then it clashed with elf loader, because the user space trampoline was added in a place where loader might need to put elf segments, so I decided to do it after first uprobe hit when loading is done.
The uprobe is un-optimized in arch specific set_orig_insn call.
The instruction overwrite is x86 arch specific and needs to go through 3 updates:
(on top of nop5 instruction)
- write int3 into 1st byte
- write last 4 bytes of the call instruction
- update the call instruction opcode
And cleanup goes though similar reverse stages:
- overwrite call opcode with breakpoint (int3)
- write last 4 bytes of the nop5 instruction
- write the nop5 first instruction byte
We do not unmap and release uprobe trampoline when it's no longer needed, because there's no easy way to make sure none of the threads is still inside the trampoline. But we do not waste memory, because there's just single page for all the uprobe trampoline mappings.
We do waste frame on page mapping for every 4GB by keeping the uprobe trampoline page mapped, but that seems ok.
We take the benefit from the fact that set_swbp and set_orig_insn are called under mmap_write_lock(mm), so we can use the current instruction as the state the uprobe is in - nop5/breakpoint/call trampoline - and decide the needed action (optimize/un-optimize) based on that.
Attaching the speed up from benchs/run_bench_uprobes.sh script:
current:
usermode-count : 152.604 ± 0.044M/s
syscall-count : 13.359 ± 0.042M/s
--> uprobe-nop : 3.229 ± 0.002M/s
uprobe-push : 3.086 ± 0.004M/s
uprobe-ret : 1.114 ± 0.004M/s
uprobe-nop5 : 1.121 ± 0.005M/s
uretprobe-nop : 2.145 ± 0.002M/s
uretprobe-push : 2.070 ± 0.001M/s
uretprobe-ret : 0.931 ± 0.001M/s
uretprobe-nop5 : 0.957 ± 0.001M/safter the change:
usermode-count : 152.448 ± 0.244M/s
syscall-count : 14.321 ± 0.059M/s
uprobe-nop : 3.148 ± 0.007M/s
uprobe-push : 2.976 ± 0.004M/s
uprobe-ret : 1.068 ± 0.003M/s
--> uprobe-nop5 : 7.038 ± 0.007M/s
uretprobe-nop : 2.109 ± 0.004M/s
uretprobe-push : 2.035 ± 0.001M/s
uretprobe-ret : 0.908 ± 0.001M/s
uretprobe-nop5 : 3.377 ± 0.009M/sI see bit more speed up on Intel (above) compared to AMD. The big nop5 speed up is partly due to emulating nop5 and partly due to optimization.
The key speed up we do this for is the USDT switch from nop to nop5:
uprobe-nop : 3.148 ± 0.007M/s
uprobe-nop5 : 7.038 ± 0.007M/s--- arch/x86/include/asm/uprobes.h | 7 + arch/x86/kernel/uprobes.c | 288 ++++++++++++++++++++++++++++++++- include/linux/uprobes.h | 6 +- kernel/events/uprobes.c | 16 +- 4 files changed, 310 insertions(+), 7 deletions(-)
arch/x86/include/asm/uprobes.h +7 -0
@@ -20,6 +20,11 @@ typedef u8 uprobe_opcode_t;#define UPROBE_SWBP_INSN 0xcc#define UPROBE_SWBP_INSN_SIZE 1 +enum { +ARCH_UPROBE_FLAG_CAN_OPTIMIZE =0 , +ARCH_UPROBE_FLAG_OPTIMIZE_FAIL =1 , +}; +struct uprobe_xol_ops ;struct arch_uprobe {
@@ -45,6 +50,8 @@ struct arch_uprobe {u8 ilen; } push; }; + +unsigned long flags; };struct arch_uprobe_task {
arch/x86/kernel/uprobes.c +287 -1
318 lines not on this page · raw message
include/linux/uprobes.h +4 -2
@@ -192,7 +192,7 @@ struct uprobes_state { };typedef int (*uprobe_write_verify_t )(struct page *page,unsigned long vaddr, -uprobe_opcode_t *insn,int nbytes); +uprobe_opcode_t *insn,int nbytes,void *data);extern void __init uprobes_init (void );extern int set_swbp (struct arch_uprobe *aup,struct vm_area_struct *vma,unsigned long vaddr);
@@ -204,7 +204,8 @@ extern unsigned long uprobe_get_trap_addr(struct pt_regs *regs);extern int uprobe_write_opcode (struct arch_uprobe *auprobe,struct vm_area_struct *vma,unsigned long vaddr,uprobe_opcode_t ,bool is_register);extern int uprobe_write (struct arch_uprobe *auprobe,struct vm_area_struct *vma,const unsigned long opcode_vaddr, -uprobe_opcode_t *insn,int nbytes,uprobe_write_verify_t verify,bool is_register,bool do_update_ref_ctr); +uprobe_opcode_t *insn,int nbytes,uprobe_write_verify_t verify,bool is_register,bool do_update_ref_ctr, +void *data);extern struct uprobe *uprobe_register (struct inode *inode,loff_t offset,loff_t ref_ctr_offset,struct uprobe_consumer *uc);extern int uprobe_apply (struct uprobe *uprobe,struct uprobe_consumer *uc,bool );extern void uprobe_unregister_nosync (struct uprobe *uprobe,struct uprobe_consumer *uc);
@@ -240,6 +241,7 @@ extern void uprobe_copy_from_page(struct page *page, unsigned long vaddr, void *extern void arch_uprobe_clear_state (struct mm_struct *mm);extern void arch_uprobe_init_state (struct mm_struct *mm);extern void handle_syscall_uprobe (struct pt_regs *regs,unsigned long bp_vaddr); +extern void arch_uprobe_optimize (struct arch_uprobe *auprobe,unsigned long vaddr);#else /* !CONFIG_UPROBES */ struct uprobes_state { };
kernel/events/uprobes.c +12 -4
@@ -192,7 +192,7 @@ static void copy_to_page(struct page *page, unsigned long vaddr, const void *src }static int verify_opcode (struct page *page,unsigned long vaddr,uprobe_opcode_t *insn, -int nbytes) +int nbytes,void *data) {uprobe_opcode_t old_opcode;bool is_swbp;
@@ -492,12 +492,13 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct vm_area_struct *vma, bool is_register) {return uprobe_write (auprobe, vma, opcode_vaddr, &opcode,UPROBE_SWBP_INSN_SIZE , - verify_opcode, is_register, true/* do_update_ref_ctr */ ); + verify_opcode, is_register, true/* do_update_ref_ctr */ ,NULL ); }int uprobe_write (struct arch_uprobe *auprobe,struct vm_area_struct *vma,const unsigned long insn_vaddr,uprobe_opcode_t *insn,int nbytes, -uprobe_write_verify_t verify,bool is_register,bool do_update_ref_ctr) +uprobe_write_verify_t verify,bool is_register,bool do_update_ref_ctr, +void *data) {const unsigned long vaddr = insn_vaddr &PAGE_MASK ;struct mm_struct *mm = vma->vm_mm;
@@ -531,7 +532,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma, goto out; folio =page_folio (page); - ret =verify (page, insn_vaddr, insn, nbytes); + ret =verify (page, insn_vaddr, insn, nbytes, data);if (ret <=0 ) {folio_put (folio); goto out;
@@ -2697,6 +2698,10 @@ bool __weak arch_uretprobe_is_alive(struct return_instance *ret, enum rp_check creturn true; } +void __weak arch_uprobe_optimize (struct arch_uprobe *auprobe,unsigned long vaddr) +{ +} +/* * Run handler and ask thread to singlestep. * Ensure all non-fatal signals cannot interrupt thread while it singlesteps.
@@ -2761,6 +2766,9 @@ static void handle_swbp(struct pt_regs *regs)handler_chain (uprobe, regs); +/* Try to optimize after first hit. */ +arch_uprobe_optimize (&uprobe->arch, bp_vaddr); +if (arch_uprobe_skip_sstep (&uprobe->arch, regs)) goto out;
Signature · 1 line
2.49.0
Jiri Olsa
reviewerPATCH v3 11/22 selftests/bpf: Import usdt.h from libbpf/usdt project
In reply to Jiri OlsaImporting usdt.h from libbpf/usdt project.
--- tools/testing/selftests/bpf/usdt.h | 545 +++++++++++++++++++++++++++++ 1 file changed, 545 insertions(+) create mode 100644 tools/testing/selftests/bpf/usdt.h
tools/testing/selftests/bpf/usdt.h new file +545 -0
@@ -0,0 +1,545 @@ +// SPDX-License-Identifier: BSD-2-Clause +/* +* This single-header library defines a collection of variadic macros for +* defining and triggering USDTs (User Statically-Defined Tracepoints): +* +* - For USDTs without associated semaphore: +* USDT(group, name, args...) +* +* - For USDTs with implicit (transparent to the user) semaphore: +* USDT_WITH_SEMA(group, name, args...) +* USDT_IS_ACTIVE(group, name) +* +* - For USDTs with explicit (user-defined and provided) semaphore: +* USDT_WITH_EXPLICIT_SEMA(sema, group, name, args...) +* USDT_SEMA_IS_ACTIVE(sema) +* +* all of which emit a NOP instruction into the instruction stream, and so +* have *zero* overhead for the surrounding code. USDTs are identified by +* a combination of `group` and `name` identifiers, which is used by external +* tracing tooling (tracers) for identifying exact USDTs of interest. +* +* USDTs can have an associated (2-byte) activity counter (USDT semaphore), +* automatically maintained by Linux kernel whenever any correctly written +* BPF-based tracer is attached to the USDT. This USDT semaphore can be used +* to check whether there is a need to do any extra data collection and +* processing for a given USDT (if necessary), and otherwise avoid extra work +* for a common case of USDT not being traced ("active"). +* +* See documentation for USDT_WITH_SEMA()/USDT_IS_ACTIVE() or +* USDT_WITH_EXPLICIT_SEMA()/USDT_SEMA_IS_ACTIVE() APIs below for details on +* working with USDTs with implicitly or explicitly associated +* USDT semaphores, respectively. +* +* There is also some additional data recorded into an auxiliary note +* section. The data in the note section describes the operands, in terms of +* size and location, used by tracing tooling to know where to find USDT +* arguments. Each location is encoded as an assembler operand string. +* Tracing tools (bpftrace and BPF-based tracers, systemtap, etc) insert +* breakpoints on top of the nop, and decode the location operand-strings, +* like an assembler, to find the values being passed. +* +* The operand strings are selected by the compiler for each operand. +* They are constrained by inline-assembler codes.The default is: +* +* #define USDT_ARG_CONSTRAINT nor +* +* This is a good default if the operands tend to be integral and +* moderate in number (smaller than number of registers). In other +* cases, the compiler may report "'asm' requires impossible reload" or +* similar. In this case, consider simplifying the macro call (fewer +* and simpler operands), reduce optimization, or override the default +* constraints string via: +* +* #define USDT_ARG_CONSTRAINT g +* #include <usdt.h> +* +* For some historical description of USDT v3 format (the one used by this +* library and generally recognized and assumed by BPF-based tracing tools) +* see [0]. The more formal specification can be found at [1]. Additional +* argument constraints information can be found at [2]. +* +* Original SystemTap's sys/sdt.h implementation ([3]) was used as a base for +* this USDT library implementation. Current implementation differs *a lot* in +* terms of exposed user API and general usability, which was the main goal +* and focus of the reimplementation work. Nevertheless, underlying recorded +* USDT definitions are fully binary compatible and any USDT-based tooling +* should work equally well with USDTs defined by either SystemTap's or this +* library's USDT implementation. +* +* [0] https://ecos.sourceware.org/ml/systemtap/2010-q3/msg00145.html +* [1] https://sourceware.org/systemtap/wiki/UserSpaceProbeImplementation +* [2] https://gcc.gnu.org/onlinedocs/gcc/Constraints.html +* [3] https://sourceware.org/git/?p=systemtap.git;a=blob;f=includes/sys/sdt.h +*/ +#ifndef __USDT_H +#define __USDT_H + +/* +* Changelog: +* +* 0.1.0 +* ----- +* - Initial release +*/ +#define USDT_MAJOR_VERSION 0 +#define USDT_MINOR_VERSION 1 +#define USDT_PATCH_VERSION 0 + +/* C++20 and C23 added __VA_OPT__ as a standard replacement for non-standard `##__VA_ARGS__` extension */ +#if (defined(__STDC_VERSION__) && __STDC_VERSION__ >201710L ) || (defined(__cplusplus) && __cplusplus >201703L ) +#define __usdt_va_opt 1 +#define __usdt_va_args (...) __VA_OPT__(,) __VA_ARGS__ +#else +#define __usdt_va_args (...) , ##__VA_ARGS__ +#endif + +/* +* Trigger USDT with `group`:`name` identifier and pass through `args` as its +* arguments. Zero arguments are acceptable as well. No USDT semaphore is +* associated with this USDT. +* +* Such "semaphoreless" USDTs are commonly used when there is no extra data +* collection or processing needed to collect and prepare USDT arguments and +* they are just available in the surrounding code. USDT() macro will just +* record their locations in CPU registers or in memory for tracing tooling to +* be able to access them, if necessary. +*/ +#ifdef __usdt_va_opt +#define USDT (group, name, ...) \ + __usdt_probe(group, name, __usdt_sema_none, 0 __VA_OPT__(,) __VA_ARGS__) +#else +#define USDT (group, name, ...) \ + __usdt_probe(group, name, __usdt_sema_none, 0, ##__VA_ARGS__) +#endif + +/* +* Trigger USDT with `group`:`name` identifier and pass through `args` as its +* arguments. Zero arguments are acceptable as well. USDT also get an +* implicitly-defined associated USDT semaphore, which will be "activated" by +* tracing tooling and can be used to check whether USDT is being actively +* observed. +* +* USDTs with semaphore are commonly used when there is a need to perform +* additional data collection and processing to prepare USDT arguments, which +* otherwise might not be necessary for the rest of application logic. In such +* case, USDT semaphore can be used to avoid unnecessary extra work. If USDT +* is not traced (which is presumed to be a common situation), the associated +* USDT semaphore is "inactive", and so there is no need to waste resources to +* prepare USDT arguments. Use USDT_IS_ACTIVE(group, name) to check whether +* USDT is "active". +* +* N.B. There is an inherent (albeit short) gap between checking whether USDT +* is active and triggering corresponding USDT, in which external tracer can +* be attached to an USDT and activate USDT semaphore after the activity check. +* If such a race occurs, tracers might miss one USDT execution. Tracers are +* expected to accommodate such possibility and this is expected to not be +* a problem for applications and tracers. +* +* N.B. Implicit USDT semaphore defined by USDT_WITH_SEMA() is contained +* within a single executable or shared library and is not shared outside +* them. I.e., if you use USDT_WITH_SEMA() with the same USDT group and name +* identifier across executable and shared library, it will work and won't +* conflict, per se, but will define independent USDT semaphores, one for each +* shared library/executable in which USDT_WITH_SEMA(group, name) is used. +* That is, if you attach to this USDT in one shared library (or executable), +* then only USDT semaphore within that shared library (or executable) will be +* updated by the kernel, while other libraries (or executable) will not see +* activated USDT semaphore. In short, it's best to use unique USDT group:name +* identifiers across different shared libraries (and, equivalently, between +* executable and shared library). This is advanced consideration and is +* rarely (if ever) seen in practice, but just to avoid surprises this is +* called out here. (Static libraries become a part of final executable, once +* linked by linker, so the above considerations don't apply to them.) +*/ +#ifdef __usdt_va_opt +#define USDT_WITH_SEMA (group, name, ...) \ + __usdt_probe(group, name, \ + __usdt_sema_implicit, __usdt_sema_name(group, name) \ + __VA_OPT__(,) __VA_ARGS__) +#else +#define USDT_WITH_SEMA (group, name, ...) \ + __usdt_probe(group, name, \ + __usdt_sema_implicit, __usdt_sema_name(group, name), \ + ##__VA_ARGS__) +#endif + +struct usdt_sema {volatile unsigned short active; }; + +/* +* Check if USDT with `group`:`name` identifier is "active" (i.e., whether it +* is attached to by external tracing tooling and is actively observed). +* +* This macro can be used to decide whether any additional and potentially +* expensive data collection or processing should be done to pass extra +* information into the given USDT. It is assumed that USDT is triggered with +* USDT_WITH_SEMA() macro which will implicitly define associated USDT +* semaphore. (If one needs more control over USDT semaphore, see +* USDT_DEFINE_SEMA() and USDT_WITH_EXPLICIT_SEMA() macros below.) +* +* N.B. Such checks are necessarily racy and speculative. Between checking +* whether USDT is active and triggering the USDT itself, tracer can be +* detached with no notification. This race should be extremely rare and worst +* case should result in one-time wasted extra data collection and processing. +*/ +#define USDT_IS_ACTIVE (group, name) ({ \ + extern struct usdt_sema __usdt_sema_name(group, name) \ + __usdt_asm_name(__usdt_sema_name(group, name)); \ + __usdt_sema_implicit(__usdt_sema_name(group, name)); \ + __usdt_sema_name(group, name).active > 0; \ +}) + +/* +* APIs for working with user-defined explicit USDT semaphores. +* +* This is a less commonly used advanced API for use cases in which user needs +* an explicit control over (potentially shared across multiple USDTs) USDT +* semaphore instance. This can be used when there is a group of logically +* related USDTs that all need extra data collection and processing whenever +* any of a family of related USDTs are "activated" (i.e., traced). In such +* a case, all such related USDTs will be associated with the same shared USDT +* semaphore defined with USDT_DEFINE_SEMA() and the USDTs themselves will be +* triggered with USDT_WITH_EXPLICIT_SEMA() macros, taking an explicit extra +* USDT semaphore identifier as an extra parameter. +*/ + +/** +* Underlying C global variable name for user-defined USDT semaphore with +* `sema` identifier. Could be useful for debugging, but normally shouldn't be +* used explicitly. +*/ +#define USDT_SEMA (sema) __usdt_sema_##sema + +/* +* Define storage for user-defined USDT semaphore `sema`. +* +* Should be used only once in non-header source file to let compiler allocate +* space for the semaphore variable. Just like with any other global variable. +* +* This macro can be used anywhere where global variable declaration is +* allowed. Just like with global variable definitions, there should be only +* one definition of user-defined USDT semaphore with given `sema` identifier, +* otherwise compiler or linker will complain about duplicate variable +* definition. +* +* For C++, it is allowed to use USDT_DEFINE_SEMA() both in global namespace +* and inside namespaces (including nested namespaces). Just make sure that +* USDT_DECLARE_SEMA() is placed within the namespace where this semaphore is +* referenced, or any of its parent namespaces, so the C++ language-level +* identifier is visible to the code that needs to reference the semaphore. +* At the lowest layer, USDT semaphores have global naming and visibility +* (they have a corresponding `__usdt_sema_<name>` symbol, which can be linked +* against from C or C++ code, if necessary). To keep it simple, putting +* USDT_DECLARE_SEMA() declarations into global namespaces is the simplest +* no-brainer solution. All these aspects are irrelevant for plain C, because +* C doesn't have namespaces and everything is always in the global namespace. +* +* N.B. Due to USDT metadata being recorded in non-allocatable ELF note +* section, it has limitations when it comes to relocations, which, in +* practice, means that it's not possible to correctly share USDT semaphores +* between main executable and shared libraries, or even between multiple +* shared libraries. USDT semaphore has to be contained to individual shared +* library or executable to avoid unpleasant surprises with half-working USDT +* semaphores. We enforce this by marking semaphore ELF symbols as having +* a hidden visibility. This is quite an advanced use case and consideration +* and for most users this should have no consequences whatsoever. +*/ +#define USDT_DEFINE_SEMA (sema) \ + struct usdt_sema __usdt_sema_sec USDT_SEMA(sema) \ + __usdt_asm_name(USDT_SEMA(sema)) \ + __attribute__((visibility("hidden"))) = { 0 } + +/* +* Declare extern reference to user-defined USDT semaphore `sema`. +* +* Refers to a variable defined in another compilation unit by +* USDT_DEFINE_SEMA() and allows to use the same USDT semaphore across +* multiple compilation units (i.e., .c and .cpp files). +* +* See USDT_DEFINE_SEMA() notes above for C++ language usage peculiarities. +*/ +#define USDT_DECLARE_SEMA (sema) \ + extern struct usdt_sema USDT_SEMA(sema) __usdt_asm_name(USDT_SEMA(sema)) + +/* +* Check if user-defined USDT semaphore `sema` is "active" (i.e., whether it +* is attached to by external tracing tooling and is actively observed). +* +* This macro can be used to decide whether any additional and potentially +* expensive data collection or processing should be done to pass extra +* information into USDT(s) associated with USDT semaphore `sema`. +* +* N.B. Such checks are necessarily racy. Between checking the state of USDT +* semaphore and triggering associated USDT(s), the active tracer might attach +* or detach. This race should be extremely rare and worst case should result +* in one-time missed USDT event or wasted extra data collection and +* processing. USDT-using tracers should be written with this in mind and is +* not a concern of the application defining USDTs with associated semaphore. +*/ +#define USDT_SEMA_IS_ACTIVE (sema) (USDT_SEMA(sema).active > 0) + +/* +* Invoke USDT specified by `group` and `name` identifiers and associate +* explicitly user-defined semaphore `sema` with it. Pass through `args` as +* USDT arguments. `args` are optional and zero arguments are acceptable. +* +* Semaphore is defined with the help of USDT_DEFINE_SEMA() macro and can be +* checked whether active with USDT_SEMA_IS_ACTIVE(). +*/ +#ifdef __usdt_va_opt +#define USDT_WITH_EXPLICIT_SEMA (sema, group, name, ...) \ + __usdt_probe(group, name, __usdt_sema_explicit, USDT_SEMA(sema), ##__VA_ARGS__) +#else +#define USDT_WITH_EXPLICIT_SEMA (sema, group, name, ...) \ + __usdt_probe(group, name, __usdt_sema_explicit, USDT_SEMA(sema) __VA_OPT__(,) __VA_ARGS__) +#endif + +/* +* Adjustable implementation aspects +*/ +#ifndef USDT_ARG_CONSTRAINT +#if defined __powerpc__ +#define USDT_ARG_CONSTRAINT nZr +#elif defined __arm__ +#define USDT_ARG_CONSTRAINT g +#elif defined __loongarch__ +#define USDT_ARG_CONSTRAINT nmr +#else +#define USDT_ARG_CONSTRAINT nor +#endif +#endif /* USDT_ARG_CONSTRAINT */ + +#ifndef USDT_NOP +#if defined(__ia64__) || defined(__s390__) || defined(__s390x__) +#define USDT_NOP nop 0 +#else +#define USDT_NOP nop +#endif +#endif /* USDT_NOP */ + +/* +* Implementation details +*/ +/* USDT name for implicitly-defined USDT semaphore, derived from group:name */ +#define __usdt_sema_name (group, name) __usdt_sema_##group##__##name +/* ELF section into which USDT semaphores are put */ +#define __usdt_sema_sec __attribute__((section(".probes"))) + +#define __usdt_concat (a, b) a ## b +#define __usdt_apply (fn, n) __usdt_concat(fn, n) + +#ifndef __usdt_nth +#define __usdt_nth (_, _1, _2, _3, _4, _5, _6, _7, _8, _9, _10, _11, _12,N , ...) N +#endif + +#ifndef __usdt_narg +#ifdef __usdt_va_opt +#define __usdt_narg (...) __usdt_nth(_ __VA_OPT__(,) __VA_ARGS__, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0) +#else +#define __usdt_narg (...) __usdt_nth(_, ##__VA_ARGS__, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0) +#endif +#endif /* __usdt_narg */ + +#define __usdt_hash # +#define __usdt_str_ (x) #x +#define __usdt_str (x) __usdt_str_(x) + +#ifndef __usdt_asm_name +#define __usdt_asm_name (name) __asm__(__usdt_str(name)) +#endif + +#define __usdt_asm0 () "\n" +#define __usdt_asm1 (x) __usdt_str(x) "\n" +#define __usdt_asm2 (x, ...) __usdt_str(x) "," __usdt_asm1(__VA_ARGS__) +#define __usdt_asm3 (x, ...) __usdt_str(x) "," __usdt_asm2(__VA_ARGS__) +#define __usdt_asm4 (x, ...) __usdt_str(x) "," __usdt_asm3(__VA_ARGS__) +#define __usdt_asm5 (x, ...) __usdt_str(x) "," __usdt_asm4(__VA_ARGS__) +#define __usdt_asm6 (x, ...) __usdt_str(x) "," __usdt_asm5(__VA_ARGS__) +#define __usdt_asm7 (x, ...) __usdt_str(x) "," __usdt_asm6(__VA_ARGS__) +#define __usdt_asm8 (x, ...) __usdt_str(x) "," __usdt_asm7(__VA_ARGS__) +#define __usdt_asm9 (x, ...) __usdt_str(x) "," __usdt_asm8(__VA_ARGS__) +#define __usdt_asm10 (x, ...) __usdt_str(x) "," __usdt_asm9(__VA_ARGS__) +#define __usdt_asm11 (x, ...) __usdt_str(x) "," __usdt_asm10(__VA_ARGS__) +#define __usdt_asm12 (x, ...) __usdt_str(x) "," __usdt_asm11(__VA_ARGS__) +#define __usdt_asm (...) __usdt_apply(__usdt_asm, __usdt_narg(__VA_ARGS__))(__VA_ARGS__) + +#ifdef __LP64__ +#define __usdt_asm_addr .8byte +#else +#define __usdt_asm_addr .4byte +#endif + +#define __usdt_asm_strz_ (x) __usdt_asm1(.asciz #x) +#define __usdt_asm_strz (x) __usdt_asm_strz_(x) +#define __usdt_asm_str_ (x) __usdt_asm1(.ascii #x) +#define __usdt_asm_str (x) __usdt_asm_str_(x) + +/* "semaphoreless" USDT case */ +#ifndef __usdt_sema_none +#define __usdt_sema_none (sema) +#endif + +/* implicitly defined __usdt_sema__group__name semaphore (using weak symbols) */ +#ifndef __usdt_sema_implicit +#define __usdt_sema_implicit (sema) \ + __asm__ __volatile__ ( \ + __usdt_asm1(.ifndef sema) \ + __usdt_asm3( .pushsection .probes, "aw", "progbits") \ + __usdt_asm1( .weak sema) \ + __usdt_asm1( .hidden sema) \ + __usdt_asm1( .align 2) \ + __usdt_asm1(sema:) \ + __usdt_asm1( .zero 2) \ + __usdt_asm2( .type sema, @object) \ + __usdt_asm2( .size sema, 2) \ + __usdt_asm1( .popsection) \ + __usdt_asm1(.endif) \ + ); +#endif + +/* externally defined semaphore using USDT_DEFINE_SEMA() and passed explicitly by user */ +#ifndef __usdt_sema_explicit +#define __usdt_sema_explicit (sema) \ + __asm__ __volatile__ ("" :: "m" (sema)); +#endif + +/* main USDT definition (nop and .note.stapsdt metadata) */ +#define __usdt_probe (group, name, sema_def, sema, ...) do { \ + sema_def(sema) \ + __asm__ __volatile__ ( \ + __usdt_asm( 990: USDT_NOP) \ + __usdt_asm3( .pushsection .note.stapsdt, "", "note") \ + __usdt_asm1( .balign 4) \ + __usdt_asm3( .4byte 992f-991f,994f-993f,3) \ + __usdt_asm1(991: .asciz "stapsdt") \ + __usdt_asm1(992: .balign 4) \ + __usdt_asm1(993: __usdt_asm_addr 990b) \ + __usdt_asm1( __usdt_asm_addr _.stapsdt.base) \ + __usdt_asm1( __usdt_asm_addr sema) \ + __usdt_asm_strz(group) \ + __usdt_asm_strz(name) \ + __usdt_asm_args(__VA_ARGS__) \ + __usdt_asm1( .ascii "\0") \ + __usdt_asm1(994: .balign 4) \ + __usdt_asm1( .popsection) \ + __usdt_asm1(.ifndef _.stapsdt.base) \ + __usdt_asm5( .pushsection .stapsdt.base,"aG","progbits",.stapsdt.base,comdat)\ + __usdt_asm1( .weak _.stapsdt.base) \ + __usdt_asm1( .hidden _.stapsdt.base) \ + __usdt_asm1(_.stapsdt.base:) \ + __usdt_asm1( .space 1) \ + __usdt_asm2( .size _.stapsdt.base, 1) \ + __usdt_asm1( .popsection) \ + __usdt_asm1(.endif) \ + :: __usdt_asm_ops(__VA_ARGS__) \ + ); \ +} while (0) + +/* +* NB: gdb PR24541 highlighted an unspecified corner of the sdt.h +* operand note format. +* +* The named register may be a longer or shorter (!) alias for the +* storage where the value in question is found. For example, on +* i386, 64-bit value may be put in register pairs, and a register +* name stored would identify just one of them. Previously, gcc was +* asked to emit the %w[id] (16-bit alias of some registers holding +* operands), even when a wider 32-bit value was used. +* +* Bottom line: the byte-width given before the @ sign governs. If +* there is a mismatch between that width and that of the named +* register, then a sys/sdt.h note consumer may need to employ +* architecture-specific heuristics to figure out where the compiler +* has actually put the complete value. +*/ +#if defined(__powerpc__) || defined(__powerpc64__) +#define __usdt_argref (id) %I[id]%[id] +#elif defined(__i386__) +#define __usdt_argref (id) %k[id]/* gcc.gnu.org/PR80115 sourceware.org/PR24541 */ +#else +#define __usdt_argref (id) %[id] +#endif + +#define __usdt_asm_arg (n) __usdt_asm_str(%c[__usdt_asz##n]) \ + __usdt_asm1(.ascii "@") \ + __usdt_asm_str(__usdt_argref(__usdt_aval##n)) + +#define __usdt_asm_args0/* no arguments */ +#define __usdt_asm_args1 __usdt_asm_arg(1) +#define __usdt_asm_args2 __usdt_asm_args1 __usdt_asm1(.ascii " ") __usdt_asm_arg(2) +#define __usdt_asm_args3 __usdt_asm_args2 __usdt_asm1(.ascii " ") __usdt_asm_arg(3) +#define __usdt_asm_args4 __usdt_asm_args3 __usdt_asm1(.ascii " ") __usdt_asm_arg(4) +#define __usdt_asm_args5 __usdt_asm_args4 __usdt_asm1(.ascii " ") __usdt_asm_arg(5) +#define __usdt_asm_args6 __usdt_asm_args5 __usdt_asm1(.ascii " ") __usdt_asm_arg(6) +#define __usdt_asm_args7 __usdt_asm_args6 __usdt_asm1(.ascii " ") __usdt_asm_arg(7) +#define __usdt_asm_args8 __usdt_asm_args7 __usdt_asm1(.ascii " ") __usdt_asm_arg(8) +#define __usdt_asm_args9 __usdt_asm_args8 __usdt_asm1(.ascii " ") __usdt_asm_arg(9) +#define __usdt_asm_args10 __usdt_asm_args9 __usdt_asm1(.ascii " ") __usdt_asm_arg(10) +#define __usdt_asm_args11 __usdt_asm_args10 __usdt_asm1(.ascii " ") __usdt_asm_arg(11) +#define __usdt_asm_args12 __usdt_asm_args11 __usdt_asm1(.ascii " ") __usdt_asm_arg(12) +#define __usdt_asm_args (...) __usdt_apply(__usdt_asm_args, __usdt_narg(__VA_ARGS__)) + +#define __usdt_is_arr (x) (__builtin_classify_type(x) == 14 || __builtin_classify_type(x) == 5) +#define __usdt_arg_size (x) (__usdt_is_arr(x) ? sizeof(void *) : sizeof(x)) + +/* +* We can't use __builtin_choose_expr() in C++, so fall back to table-based +* signedness determination for known types, utilizing templates magic. +*/ +#ifdef __cplusplus + +#define __usdt_is_signed (x) (!__usdt_is_arr(x) && __usdt_t<__typeof(x)>::is_signed) + +#include <cstddef> + +template<typename T> struct __usdt_t { static const bool is_signed = false; }; +template<typename A> struct __usdt_t<A[]> : public __usdt_t<A *> {}; +template<typename A, size_t N> struct __usdt_t<A[N]> : public __usdt_t<A *> {}; + +#define __usdt_def_signed (T ) \ +template<> struct __usdt_t<T> { static const bool is_signed = true; }; \ +template<> struct __usdt_t<const T> { static const bool is_signed = true; }; \ +template<> struct __usdt_t<volatile T> { static const bool is_signed = true; }; \ +template<> struct __usdt_t<const volatile T> { static const bool is_signed = true; } +#define __usdt_maybe_signed (T ) \ +template<> struct __usdt_t<T> { static const bool is_signed = (T)-1 < (T)1; }; \ +template<> struct __usdt_t<const T> { static const bool is_signed = (T)-1 < (T)1; }; \ +template<> struct __usdt_t<volatile T> { static const bool is_signed = (T)-1 < (T)1; }; \ +template<> struct __usdt_t<const volatile T> { static const bool is_signed = (T)-1 < (T)1; } + +__usdt_def_signed (signed char ); +__usdt_def_signed (short); +__usdt_def_signed (int); +__usdt_def_signed (long); +__usdt_def_signed (long long ); +__usdt_maybe_signed (char); +__usdt_maybe_signed (wchar_t); + +#else /* !__cplusplus */ + +#define __usdt_is_inttype (x) (__builtin_classify_type(x) >= 1 && __builtin_classify_type(x) <= 4) +#define __usdt_inttype (x) __typeof(__builtin_choose_expr(__usdt_is_inttype(x), (x), 0U)) +#define __usdt_is_signed (x) ((__usdt_inttype(x))-1 < (__usdt_inttype(x))1) + +#endif /* __cplusplus */ + +#define __usdt_asm_op (n, x) \ + [__usdt_asz##n] "n" ((__usdt_is_signed(x) ? (int)-1 : 1) * (int)__usdt_arg_size(x)), \ + [__usdt_aval##n] __usdt_str(USDT_ARG_CONSTRAINT)(x) + +#define __usdt_asm_ops0 () [__usdt_dummy] "g" (0) +#define __usdt_asm_ops1 (x) __usdt_asm_op(1, x) +#define __usdt_asm_ops2 (a,x) __usdt_asm_ops1(a), __usdt_asm_op(2, x) +#define __usdt_asm_ops3 (a,b,x) __usdt_asm_ops2(a,b), __usdt_asm_op(3, x) +#define __usdt_asm_ops4 (a,b,c,x) __usdt_asm_ops3(a,b,c), __usdt_asm_op(4, x) +#define __usdt_asm_ops5 (a,b,c,d,x) __usdt_asm_ops4(a,b,c,d), __usdt_asm_op(5, x) +#define __usdt_asm_ops6 (a,b,c,d,e,x) __usdt_asm_ops5(a,b,c,d,e), __usdt_asm_op(6, x) +#define __usdt_asm_ops7 (a,b,c,d,e,f,x) __usdt_asm_ops6(a,b,c,d,e,f), __usdt_asm_op(7, x) +#define __usdt_asm_ops8 (a,b,c,d,e,f,g,x) __usdt_asm_ops7(a,b,c,d,e,f,g), __usdt_asm_op(8, x) +#define __usdt_asm_ops9 (a,b,c,d,e,f,g,h,x) __usdt_asm_ops8(a,b,c,d,e,f,g,h), __usdt_asm_op(9, x) +#define __usdt_asm_ops10 (a,b,c,d,e,f,g,h,i,x) __usdt_asm_ops9(a,b,c,d,e,f,g,h,i), __usdt_asm_op(10, x) +#define __usdt_asm_ops11 (a,b,c,d,e,f,g,h,i,j,x) __usdt_asm_ops10(a,b,c,d,e,f,g,h,i,j), __usdt_asm_op(11, x) +#define __usdt_asm_ops12 (a,b,c,d,e,f,g,h,i,j,k,x) __usdt_asm_ops11(a,b,c,d,e,f,g,h,i,j,k), __usdt_asm_op(12, x) +#define __usdt_asm_ops (...) __usdt_apply(__usdt_asm_ops, __usdt_narg(__VA_ARGS__))(__VA_ARGS__) + +#endif /* __USDT_H */
Signature · 1 line
2.49.0
Jiri Olsa
reviewerPATCH v3 12/22 selftests/bpf: Reorg the uprobe_syscall test function
In reply to Jiri OlsaAdding __test_uprobe_syscall with non x86_64 stub to execute all the tests, so we don't need to keep adding non x86_64 stub functions for new tests.
--- .../selftests/bpf/prog_tests/uprobe_syscall.c | 34 +++++++------------ 1 file changed, 12 insertions(+), 22 deletions(-)
tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c +12 -22
@@ -350,29 +350,8 @@ static void test_uretprobe_shadow_stack(void)ARCH_PRCTL (ARCH_SHSTK_DISABLE ,ARCH_SHSTK_SHSTK ); } -#else -static void test_uretprobe_regs_equal (void ) -{ -test__skip (); -} - -static void test_uretprobe_regs_change (void ) -{ -test__skip (); -} - -static void test_uretprobe_syscall_call (void ) -{ -test__skip (); -} -static void test_uretprobe_shadow_stack (void ) -{ -test__skip (); -} -#endif - -void test_uprobe_syscall (void ) +static void __test_uprobe_syscall (void ) {if (test__start_subtest ("uretprobe_regs_equal" ))test_uretprobe_regs_equal ();
@@ -383,3 +362,14 @@ void test_uprobe_syscall(void)if (test__start_subtest ("uretprobe_shadow_stack" )) test_uretprobe_shadow_stack(); } +#else +static void __test_uprobe_syscall (void ) +{ +test__skip (); +} +#endif + +void test_uprobe_syscall (void ) +{ +__test_uprobe_syscall (); +}
Signature · 1 line
2.49.0
Jiri Olsa
reviewerPATCH v3 13/22 selftests/bpf: Rename uprobe_syscall_executed prog to test_uretprobe_multi
In reply to Jiri OlsaRenaming uprobe_syscall_executed prog to test_uretprobe_multi to fit properly in the following changes that add more programs.
Plus adding pid filter and increasing executed variable.
--- .../selftests/bpf/prog_tests/uprobe_syscall.c | 12 ++++++++---- .../selftests/bpf/progs/uprobe_syscall_executed.c | 8 ++++++-- 2 files changed, 14 insertions(+), 6 deletions(-)
tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c +8 -4
@@ -252,6 +252,7 @@ static void test_uretprobe_syscall_call(void) );struct uprobe_syscall_executed *skel;int pid, status, err, go[2 ], c; +struct bpf_link *link;if (!ASSERT_OK (pipe (go),"pipe" ))return ;
@@ -277,11 +278,14 @@ static void test_uretprobe_syscall_call(void)_exit (0 ); } - skel->links.test =bpf_program__attach_uprobe_multi (skel->progs.test, pid, -"/proc/self/exe" , -"uretprobe_syscall_call" , &opts); -if (!ASSERT_OK_PTR (skel->links.test,"bpf_program__attach_uprobe_multi" )) + skel->bss->pid = pid; + + link =bpf_program__attach_uprobe_multi (skel->progs.test_uretprobe_multi, + pid,"/proc/self/exe" , +"uretprobe_syscall_call" , &opts); +if (!ASSERT_OK_PTR (link,"bpf_program__attach_uprobe_multi" )) goto cleanup; + skel->links.test_uretprobe_multi = link;/* kick the child */ write (go[1 ], &c,1 );
tools/testing/selftests/bpf/progs/uprobe_syscall_executed.c +6 -2
@@ -8,10 +8,14 @@ struct pt_regs regs; char _license[] SEC("license") = "GPL";int executed =0 ; +int pid; SEC("uretprobe.multi") -int test (struct pt_regs *regs) +int test_uretprobe_multi (struct pt_regs *ctx) { - executed =1 ; +if (bpf_get_current_pid_tgid () >>32 != pid) +return 0 ; + + executed++;return 0 ; }
Signature · 1 line
2.49.0
Jiri Olsa
reviewerPATCH v3 14/22 selftests/bpf: Add uprobe/usdt syscall tests
In reply to Jiri OlsaAdding tests for optimized uprobe/usdt probes.
Checking that we get expected trampoline and attached bpf programs get executed properly.
--- .../selftests/bpf/prog_tests/uprobe_syscall.c | 289 +++++++++++++++++- .../bpf/progs/uprobe_syscall_executed.c | 52 ++++ 2 files changed, 340 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c +288 -1
@@ -8,6 +8,7 @@#include <asm/ptrace.h> #include <linux/compiler.h> #include <linux/stringify.h> +#include <linux/kernel.h> #include <sys/wait.h> #include <sys/syscall.h> #include <sys/prctl.h>
@@ -15,6 +16,11 @@#include "uprobe_syscall.skel.h" #include "uprobe_syscall_executed.skel.h" +#define USDT_NOP .byte 0x0f, 0x1f, 0x44, 0x00, 0x00 +#include "usdt.h" + +#pragma GCC diagnostic ignored "-Wattributes" +__naked unsigned long uretprobe_regs_trigger (void ) { asm volatile (
@@ -305,6 +311,270 @@ static void test_uretprobe_syscall_call(void)close (go[0 ]); } +#define TRAMP "[uprobes-trampoline]" + +__attribute__((aligned (16 ))) +__nocf_check __weak __naked void uprobe_test(void) +{ + asmvolatile (" \n" +".byte 0x0f, 0x1f, 0x44, 0x00, 0x00 \n" +"ret \n" + ); +} + +__attribute__((aligned (16 ))) +__nocf_check __weak void usdt_test(void) +{ +USDT (optimized_uprobe, usdt); +} + +static int find_uprobes_trampoline (void **start,void **end) +{ +char line[128 ]; +int ret =-1 ; +FILE *maps; + + maps =fopen ("/proc/self/maps" ,"r" ); +if (!maps) { +fprintf (stderr,"cannot open maps\n" ); +return -1 ; + } + +while (fgets (line,sizeof (line), maps)) { +int m =-1 ; + +/* We care only about private r-x mappings. */ +if (sscanf (line,"%p-%p r-xp %*x %*x:%*x %*u %n" , start, end, &m) !=2 ) +continue ; +if (m <0 ) +continue ; +if (!strncmp (&line[m],TRAMP ,sizeof (TRAMP )-1 )) { + ret =0 ; +break ; + } + } + +fclose (maps); +return ret; +} + +static unsigned char nop5[5 ] = {0x0f ,0x1f ,0x44 ,0x00 ,0x00 }; + +static void *find_nop5 (void *fn) +{ +int i; + +for (i =0 ; i <10 ; i++) { +if (!memcmp (nop5, fn + i,5 )) +return fn + i; + } +return NULL ; +} + +typedef void (__attribute__((nocf_check)) *trigger_t)(void); + +static bool shstk_is_enabled; + +static void check_attach (struct uprobe_syscall_executed *skel,trigger_t trigger, +void *addr,int executed) +{ +void *tramp_start, *tramp_end; +struct __arch_relative_insn { +__u8 op; +__s32 raddr; + } __packed *call; +__s32 delta; +__u8 *bp; + +/* Uprobe gets optimized after first trigger, so let's press twice. */ +trigger (); +trigger (); + +if (!shstk_is_enabled && + !ASSERT_OK (find_uprobes_trampoline (&tramp_start, &tramp_end),"uprobes_trampoline" )) +return ; + +/* Make sure bpf program got executed.. */ +ASSERT_EQ (skel->bss->executed, executed,"executed" ); + +if (shstk_is_enabled) { +/* .. and check optimization is disabled under shadow stack. */ + bp = (__u8 *) addr; +ASSERT_EQ (*bp,0xcc ,"int3" ); + }else { +/* .. and check the trampoline is as expected. */ + call = (struct __arch_relative_insn *) addr; + delta = (unsigned long ) tramp_start - ((unsigned long ) addr +5 ); + +ASSERT_EQ (call->op,0xe8 ,"call" ); +ASSERT_EQ (call->raddr, delta,"delta" ); +ASSERT_EQ (tramp_end - tramp_start,4096 ,"size" ); + } +} + +static void check_detach (struct uprobe_syscall_executed *skel,trigger_t trigger,void *addr) +{ +void *tramp_start, *tramp_end; + +/* [uprobes_trampoline] stays after detach */ +ASSERT_OK (!shstk_is_enabled && +find_uprobes_trampoline (&tramp_start, &tramp_end),"uprobes_trampoline" ); +ASSERT_OK (memcmp (addr, nop5,5 ),"nop5" ); +} + +static void check (struct uprobe_syscall_executed *skel,struct bpf_link *link, +trigger_t trigger,void *addr,int executed) +{ +check_attach (skel, trigger, addr, executed); +bpf_link__destroy (link); +check_detach (skel, trigger, addr); +} + +static void test_uprobe_legacy (void ) +{ +struct uprobe_syscall_executed *skel =NULL ; +LIBBPF_OPTS (bpf_uprobe_opts, opts, + .retprobe = true, + ); +struct bpf_link *link; +unsigned long offset; + + offset =get_uprobe_offset (&uprobe_test); +if (!ASSERT_GE (offset,0 ,"get_uprobe_offset" )) + goto cleanup; + +/* uprobe */ + skel =uprobe_syscall_executed__open_and_load (); +if (!ASSERT_OK_PTR (skel,"uprobe_syscall_executed__open_and_load" )) +return ; + + skel->bss->pid =getpid (); + + link =bpf_program__attach_uprobe_opts (skel->progs.test_uprobe, +0 ,"/proc/self/exe" , offset,NULL ); +if (!ASSERT_OK_PTR (link,"bpf_program__attach_uprobe_opts" )) + goto cleanup; + +check (skel, link, uprobe_test, uprobe_test,2 ); + +/* uretprobe */ + skel->bss->executed =0 ; + + link =bpf_program__attach_uprobe_opts (skel->progs.test_uretprobe, +0 ,"/proc/self/exe" , offset, &opts); +if (!ASSERT_OK_PTR (link,"bpf_program__attach_uprobe_opts" )) + goto cleanup; + +check (skel, link, uprobe_test, uprobe_test,2 ); + +cleanup: +uprobe_syscall_executed__destroy (skel); +} + +static void test_uprobe_multi (void ) +{ +struct uprobe_syscall_executed *skel =NULL ; +LIBBPF_OPTS (bpf_uprobe_multi_opts, opts); +struct bpf_link *link; +unsigned long offset; + + offset =get_uprobe_offset (&uprobe_test); +if (!ASSERT_GE (offset,0 ,"get_uprobe_offset" )) + goto cleanup; + + opts.offsets = &offset; + opts.cnt =1 ; + + skel =uprobe_syscall_executed__open_and_load (); +if (!ASSERT_OK_PTR (skel,"uprobe_syscall_executed__open_and_load" )) +return ; + + skel->bss->pid =getpid (); + +/* uprobe.multi */ + link =bpf_program__attach_uprobe_multi (skel->progs.test_uprobe_multi, +0 ,"/proc/self/exe" ,NULL , &opts); +if (!ASSERT_OK_PTR (link,"bpf_program__attach_uprobe_multi" )) + goto cleanup; + +check (skel, link, uprobe_test, uprobe_test,2 ); + +/* uretprobe.multi */ + skel->bss->executed =0 ; + opts.retprobe = true; + link =bpf_program__attach_uprobe_multi (skel->progs.test_uretprobe_multi, +0 ,"/proc/self/exe" ,NULL , &opts); +if (!ASSERT_OK_PTR (link,"bpf_program__attach_uprobe_multi" )) + goto cleanup; + +check (skel, link, uprobe_test, uprobe_test,2 ); + +cleanup: +uprobe_syscall_executed__destroy (skel); +} + +static void test_uprobe_session (void ) +{ +struct uprobe_syscall_executed *skel =NULL ; +LIBBPF_OPTS (bpf_uprobe_multi_opts, opts, + .session = true, + ); +struct bpf_link *link; +unsigned long offset; + + offset =get_uprobe_offset (&uprobe_test); +if (!ASSERT_GE (offset,0 ,"get_uprobe_offset" )) + goto cleanup; + + opts.offsets = &offset; + opts.cnt =1 ; + + skel =uprobe_syscall_executed__open_and_load (); +if (!ASSERT_OK_PTR (skel,"uprobe_syscall_executed__open_and_load" )) +return ; + + skel->bss->pid =getpid (); + + link =bpf_program__attach_uprobe_multi (skel->progs.test_uprobe_session, +0 ,"/proc/self/exe" ,NULL , &opts); +if (!ASSERT_OK_PTR (link,"bpf_program__attach_uprobe_multi" )) + goto cleanup; + +check (skel, link, uprobe_test, uprobe_test,4 ); + +cleanup: +uprobe_syscall_executed__destroy (skel); +} + +static void test_uprobe_usdt (void ) +{ +struct uprobe_syscall_executed *skel; +struct bpf_link *link; +void *addr; + + errno =0 ; + addr =find_nop5 (usdt_test); +if (!ASSERT_OK_PTR (addr,"find_nop5" )) +return ; + + skel =uprobe_syscall_executed__open_and_load (); +if (!ASSERT_OK_PTR (skel,"uprobe_syscall_executed__open_and_load" )) +return ; + + skel->bss->pid =getpid (); + + link =bpf_program__attach_usdt (skel->progs.test_usdt, +-1 /* all PIDs */ ,"/proc/self/exe" , +"optimized_uprobe" ,"usdt" ,NULL ); +if (!ASSERT_OK_PTR (link,"bpf_program__attach_usdt" )) + goto cleanup; + +check (skel, link, usdt_test, addr,2 ); + +cleanup: +uprobe_syscall_executed__destroy (skel); +} +/* * Borrowed from tools/testing/selftests/x86/test_shadow_stack.c. *
@@ -347,11 +617,20 @@ static void test_uretprobe_shadow_stack(void)return ; } -/* Run all of the uretprobe tests. */ +/* Run all the tests with shadow stack in place. */ + shstk_is_enabled = true; +test_uretprobe_regs_equal ();test_uretprobe_regs_change ();test_uretprobe_syscall_call (); +test_uprobe_legacy (); +test_uprobe_multi (); +test_uprobe_session (); +test_uprobe_usdt (); + + shstk_is_enabled = false; +ARCH_PRCTL (ARCH_SHSTK_DISABLE ,ARCH_SHSTK_SHSTK ); }
@@ -365,6 +644,14 @@ static void __test_uprobe_syscall(void)test_uretprobe_syscall_call ();if (test__start_subtest ("uretprobe_shadow_stack" ))test_uretprobe_shadow_stack (); +if (test__start_subtest ("uprobe_legacy" )) +test_uprobe_legacy (); +if (test__start_subtest ("uprobe_multi" )) +test_uprobe_multi (); +if (test__start_subtest ("uprobe_session" )) +test_uprobe_session (); +if (test__start_subtest ("uprobe_usdt" )) + test_uprobe_usdt(); }#else static void __test_uprobe_syscall (void )
tools/testing/selftests/bpf/progs/uprobe_syscall_executed.c +52 -0
67 lines not on this page · raw message
Signature · 1 line
2.49.0
Jiri Olsa
reviewerPATCH v3 15/22 selftests/bpf: Add hit/attach/detach race optimized uprobe test
In reply to Jiri OlsaAdding test that makes sure parallel execution of the uprobe and attach/detach of optimized uprobe on it works properly.
By default the test runs for 500ms, which is adjustable by using BPF_SELFTESTS_UPROBE_SYSCALL_RACE_MSEC env variable.
--- .../selftests/bpf/prog_tests/uprobe_syscall.c | 108 ++++++++++++++++++ 1 file changed, 108 insertions(+)
tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c +108 -0
@@ -15,6 +15,7 @@#include <asm/prctl.h> #include "uprobe_syscall.skel.h" #include "uprobe_syscall_executed.skel.h" +#include "bpf/libbpf_internal.h" #define USDT_NOP .byte 0x0f, 0x1f, 0x44, 0x00, 0x00#include "usdt.h"
@@ -634,6 +635,111 @@ static void test_uretprobe_shadow_stack(void)ARCH_PRCTL (ARCH_SHSTK_DISABLE ,ARCH_SHSTK_SHSTK ); } +static volatile bool race_stop; + +static USDT_DEFINE_SEMA (race); + +static void *worker_trigger (void *arg) +{ +unsigned long rounds =0 ; + +while (!race_stop) { +uprobe_test (); + rounds++; + } + +printf ("tid %d trigger rounds: %lu\n" ,gettid (), rounds); +return NULL ; +} + +static void *worker_attach (void *arg) +{ +LIBBPF_OPTS (bpf_uprobe_opts, opts); +struct uprobe_syscall_executed *skel; +unsigned long rounds =0 , offset; +const char *sema[2 ] = { +__stringify (USDT_SEMA (race)), +NULL , + }; +unsigned long *ref; +int err; + + offset =get_uprobe_offset (&uprobe_test); +if (!ASSERT_GE (offset,0 ,"get_uprobe_offset" )) +return NULL ; + + err =elf_resolve_syms_offsets ("/proc/self/exe" ,1 , (const char **) &sema, &ref,STT_OBJECT ); +if (!ASSERT_OK (err,"elf_resolve_syms_offsets_sema" )) +return NULL ; + + opts.ref_ctr_offset = *ref; + + skel =uprobe_syscall_executed__open_and_load (); +if (!ASSERT_OK_PTR (skel,"uprobe_syscall_executed__open_and_load" )) +return NULL ; + + skel->bss->pid =getpid (); + +while (!race_stop) { + skel->links.test_uprobe =bpf_program__attach_uprobe_opts (skel->progs.test_uprobe, +0 ,"/proc/self/exe" , offset, &opts); +if (!ASSERT_OK_PTR (skel->links.test_uprobe,"bpf_program__attach_uprobe_opts" )) +break ; + +bpf_link__destroy (skel->links.test_uprobe); + skel->links.test_uprobe =NULL ; + rounds++; + } + +printf ("tid %d attach rounds: %lu hits: %d\n" ,gettid (), rounds, skel->bss->executed); +uprobe_syscall_executed__destroy (skel); +free (ref); +return NULL ; +} + +static useconds_t race_msec (void ) +{ +char *env; + + env =getenv ("BPF_SELFTESTS_UPROBE_SYSCALL_RACE_MSEC" ); +if (env) +return atoi (env); + +/* default duration is 500ms */ +return 500 ; +} + +static void test_uprobe_race (void ) +{ +int err, i, nr_threads; +pthread_t *threads; + + nr_threads =libbpf_num_possible_cpus (); +if (!ASSERT_GT (nr_threads,0 ,"libbpf_num_possible_cpus" )) +return ; + nr_threads =max (2 , nr_threads); + + threads =alloca (sizeof (*threads) * nr_threads); +if (!ASSERT_OK_PTR (threads,"malloc" )) +return ; + +for (i =0 ; i < nr_threads; i++) { + err =pthread_create (&threads[i],NULL , i %2 ? worker_trigger : worker_attach, +NULL ); +if (!ASSERT_OK (err,"pthread_create" )) + goto cleanup; + } + +usleep (race_msec () *1000 ); + +cleanup: + race_stop = true; +for (nr_threads = i, i =0 ; i < nr_threads; i++) +pthread_join (threads[i],NULL ); + +ASSERT_FALSE (USDT_SEMA_IS_ACTIVE (race),"race_semaphore" ); +} +static void __test_uprobe_syscall (void ) {if (test__start_subtest ("uretprobe_regs_equal" ))
@@ -652,6 +758,8 @@ static void __test_uprobe_syscall(void)test_uprobe_session ();if (test__start_subtest ("uprobe_usdt" ))test_uprobe_usdt (); +if (test__start_subtest ("uprobe_race" )) + test_uprobe_race(); }#else static void __test_uprobe_syscall (void )
Signature · 1 line
2.49.0
Jiri Olsa
reviewerPATCH v3 16/22 selftests/bpf: Add uprobe syscall sigill signal test
In reply to Jiri OlsaMake sure that calling uprobe syscall from outside uprobe trampoline results in sigill signal.
--- .../selftests/bpf/prog_tests/uprobe_syscall.c | 36 +++++++++++++++++++ 1 file changed, 36 insertions(+)
tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c +36 -0
@@ -740,6 +740,40 @@ static void test_uprobe_race(void)ASSERT_FALSE (USDT_SEMA_IS_ACTIVE (race),"race_semaphore" ); } +#ifndef __NR_uprobe +#define __NR_uprobe 336 +#endif + +static void test_uprobe_sigill (void ) +{ +int status, err, pid; + + pid =fork (); +if (!ASSERT_GE (pid,0 ,"fork" )) +return ; +/* child */ +if (pid ==0 ) { + asmvolatile ( +"pushq %rax\n" +"pushq %rcx\n" +"pushq %r11\n" + "movq $" __stringify(__NR_uprobe) ", %rax\n" +"syscall\n" +"popq %r11\n" +"popq %rcx\n" +"retq\n" + ); +exit (0 ); + } + + err =waitpid (pid, &status,0 ); +ASSERT_EQ (err, pid,"waitpid" ); + +/* verify the child got killed with SIGILL */ +ASSERT_EQ (WIFSIGNALED (status),1 ,"WIFSIGNALED" ); +ASSERT_EQ (WTERMSIG (status),SIGILL ,"WTERMSIG" ); +} +static void __test_uprobe_syscall (void ) {if (test__start_subtest ("uretprobe_regs_equal" ))
@@ -760,6 +794,8 @@ static void __test_uprobe_syscall(void)test_uprobe_usdt ();if (test__start_subtest ("uprobe_race" ))test_uprobe_race (); +if (test__start_subtest ("uprobe_sigill" )) + test_uprobe_sigill(); }#else static void __test_uprobe_syscall (void )
Signature · 1 line
2.49.0
Jiri Olsa
reviewerPATCH v3 17/22 selftests/bpf: Add optimized usdt variant for basic usdt test
In reply to Jiri OlsaAdding optimized usdt variant for basic usdt test to check that usdt arguments are properly passed in optimized code path.
--- tools/testing/selftests/bpf/prog_tests/usdt.c | 38 ++++++++++++------- 1 file changed, 25 insertions(+), 13 deletions(-)
tools/testing/selftests/bpf/prog_tests/usdt.c +25 -13
@@ -40,12 +40,19 @@ static void __always_inline trigger_func(int x) { } } -static void subtest_basic_usdt (void ) +static void subtest_basic_usdt (bool optimized) {LIBBPF_OPTS (bpf_usdt_opts, opts);struct test_usdt *skel;struct test_usdt__bss *bss; -int err, i; +int err, i, called; + +#define TRIGGER (x) ({ \ + trigger_func(x); \ + if (optimized) \ + trigger_func(x); \ + optimized ? 2 : 1; \ + }) skel =test_usdt__open_and_load ();if (!ASSERT_OK_PTR (skel,"skel_open" ))
@@ -66,11 +73,11 @@ static void subtest_basic_usdt(void)if (!ASSERT_OK_PTR (skel->links.usdt0,"usdt0_link" )) goto cleanup; -trigger_func (1 ); + called =TRIGGER (1 ); -ASSERT_EQ (bss->usdt0_called,1 ,"usdt0_called" ); -ASSERT_EQ (bss->usdt3_called,1 ,"usdt3_called" ); -ASSERT_EQ (bss->usdt12_called,1 ,"usdt12_called" ); +ASSERT_EQ (bss->usdt0_called, called,"usdt0_called" ); +ASSERT_EQ (bss->usdt3_called, called,"usdt3_called" ); +ASSERT_EQ (bss->usdt12_called, called,"usdt12_called" );ASSERT_EQ (bss->usdt0_cookie,0xcafedeadbeeffeed ,"usdt0_cookie" );ASSERT_EQ (bss->usdt0_arg_cnt,0 ,"usdt0_arg_cnt" );
@@ -119,11 +126,11 @@ static void subtest_basic_usdt(void)* bpf_program__attach_usdt() handles this properly and attaches to * all possible places of USDT invocation. */ -trigger_func (2 ); + called +=TRIGGER (2 ); -ASSERT_EQ (bss->usdt0_called,2 ,"usdt0_called" ); -ASSERT_EQ (bss->usdt3_called,2 ,"usdt3_called" ); -ASSERT_EQ (bss->usdt12_called,2 ,"usdt12_called" ); +ASSERT_EQ (bss->usdt0_called, called,"usdt0_called" ); +ASSERT_EQ (bss->usdt3_called, called,"usdt3_called" ); +ASSERT_EQ (bss->usdt12_called, called,"usdt12_called" );/* only check values that depend on trigger_func()'s input value */ ASSERT_EQ (bss->usdt3_args[0 ],2 ,"usdt3_arg1" );
@@ -142,9 +149,9 @@ static void subtest_basic_usdt(void)if (!ASSERT_OK_PTR (skel->links.usdt3,"usdt3_reattach" )) goto cleanup; -trigger_func (3 ); + called +=TRIGGER (3 ); -ASSERT_EQ (bss->usdt3_called,3 ,"usdt3_called" ); +ASSERT_EQ (bss->usdt3_called, called,"usdt3_called" );/* this time usdt3 has custom cookie */ ASSERT_EQ (bss->usdt3_cookie,0xBADC00C51E ,"usdt3_cookie" );ASSERT_EQ (bss->usdt3_arg_cnt,3 ,"usdt3_arg_cnt" );
@@ -158,6 +165,7 @@ static void subtest_basic_usdt(void) cleanup:test_usdt__destroy (skel); +#undef TRIGGER } unsigned short test_usdt_100_semaphore SEC(".probes");
@@ -419,7 +427,11 @@ static void subtest_urandom_usdt(bool auto_attach)void test_usdt (void ) {if (test__start_subtest ("basic" )) -subtest_basic_usdt (); +subtest_basic_usdt (false); +#ifdef __x86_64__ +if (test__start_subtest ("basic_optimized" )) +subtest_basic_usdt (true); +#endif if (test__start_subtest ("multispec" ))subtest_multispec_usdt ();if (test__start_subtest ("urand_auto_attach" ))
Signature · 1 line
2.49.0
Jiri Olsa
reviewerPATCH v3 18/22 selftests/bpf: Add uprobe_regs_equal test
In reply to Jiri OlsaChanging uretprobe_regs_trigger to allow the test for both uprobe and uretprobe and renaming it to uprobe_regs_equal.
We check that both uprobe and uretprobe probes (bpf programs) see expected registers with few exceptions.
--- .../selftests/bpf/prog_tests/uprobe_syscall.c | 56 ++++++++++++++----- .../selftests/bpf/progs/uprobe_syscall.c | 4 +- 2 files changed, 44 insertions(+), 16 deletions(-)
tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c +42 -14
@@ -22,15 +22,17 @@#pragma GCC diagnostic ignored "-Wattributes" -__naked unsigned long uretprobe_regs_trigger (void ) +__attribute__((aligned (16 ))) +__nocf_check __weak __naked unsigned long uprobe_regs_trigger (void ) { asmvolatile ( +".byte 0x0f, 0x1f, 0x44, 0x00, 0x00\n" /* nop5 */ "movq $0xdeadbeef, %rax\n" "ret\n" ); } -__naked void uretprobe_regs (struct pt_regs *before,struct pt_regs *after) +__naked void uprobe_regs (struct pt_regs *before,struct pt_regs *after) { asmvolatile ( "movq %r15, 0(%rdi)\n"
@@ -51,15 +53,17 @@ __naked void uretprobe_regs(struct pt_regs *before, struct pt_regs *after)"movq $0, 120(%rdi)\n" /* orig_rax */ "movq $0, 128(%rdi)\n" /* rip */ "movq $0, 136(%rdi)\n" /* cs */ +"pushq %rax\n" "pushf\n" "pop %rax\n" "movq %rax, 144(%rdi)\n" /* eflags */ +"pop %rax\n" "movq %rsp, 152(%rdi)\n" /* rsp */ "movq $0, 160(%rdi)\n" /* ss */ /* save 2nd argument */ "pushq %rsi\n" -"call uretprobe_regs_trigger\n" +"call uprobe_regs_trigger\n" /* save return value and load 2nd argument pointer to rax */ "pushq %rax\n"
@@ -99,25 +103,37 @@ __naked void uretprobe_regs(struct pt_regs *before, struct pt_regs *after) ); } -static void test_uretprobe_regs_equal (void ) +static void test_uprobe_regs_equal (bool retprobe) { +LIBBPF_OPTS (bpf_uprobe_opts, opts, + .retprobe = retprobe, + );struct uprobe_syscall *skel =NULL ;struct pt_regs before = {}, after = {};unsigned long *pb = (unsigned long *) &before;unsigned long *pa = (unsigned long *) &after;unsigned long *pp; +unsigned long offset;unsigned int i, cnt; -int err; + + offset =get_uprobe_offset (&uprobe_regs_trigger); +if (!ASSERT_GE (offset,0 ,"get_uprobe_offset" )) +return ; skel =uprobe_syscall__open_and_load ();if (!ASSERT_OK_PTR (skel,"uprobe_syscall__open_and_load" )) goto cleanup; - err =uprobe_syscall__attach (skel); -if (!ASSERT_OK (err,"uprobe_syscall__attach" )) + skel->links.probe =bpf_program__attach_uprobe_opts (skel->progs.probe, +0 ,"/proc/self/exe" , offset, &opts); +if (!ASSERT_OK_PTR (skel->links.probe,"bpf_program__attach_uprobe_opts" )) goto cleanup; -uretprobe_regs (&before, &after); +/* make sure uprobe gets optimized */ +if (!retprobe) +uprobe_regs_trigger (); + +uprobe_regs (&before, &after); pp = (unsigned long *) &skel->bss->regs; cnt =sizeof (before)/sizeof (*pb);
@@ -126,7 +142,7 @@ static void test_uretprobe_regs_equal(void)unsigned int offset = i *sizeof (unsigned long );/* -* Check register before and after uretprobe_regs_trigger call +* Check register before and after uprobe_regs_trigger call * that triggers the uretprobe. */ switch (offset) {
@@ -140,7 +156,7 @@ static void test_uretprobe_regs_equal(void)/* * Check register seen from bpf program and register after -* uretprobe_regs_trigger call +* uprobe_regs_trigger call (with rax exception, check below). */ switch (offset) {/*
@@ -153,6 +169,15 @@ static void test_uretprobe_regs_equal(void)case offsetof(struct pt_regs , rsp):case offsetof(struct pt_regs , ss):break ; +/* +* uprobe does not see return value in rax, it needs to see the +* original (before) rax value +*/ +case offsetof(struct pt_regs , rax): +if (!retprobe) { +ASSERT_EQ (pp[i], pb[i],"uprobe rax prog-before value check" ); +break ; + }default :if (!ASSERT_EQ (pp[i], pa[i],"register prog-after value check" ))fprintf (stdout,"failed register offset %u\n" , offset);
@@ -190,13 +215,13 @@ static void test_uretprobe_regs_change(void)unsigned long cnt =sizeof (before)/sizeof (*pb);unsigned int i, err, offset; - offset =get_uprobe_offset (uretprobe_regs_trigger); + offset =get_uprobe_offset (uprobe_regs_trigger); err =write_bpf_testmod_uprobe (offset);if (!ASSERT_OK (err,"register_uprobe" ))return ; -uretprobe_regs (&before, &after); +uprobe_regs (&before, &after); err =write_bpf_testmod_uprobe (0 );if (!ASSERT_OK (err,"unregister_uprobe" ))
@@ -621,7 +646,8 @@ static void test_uretprobe_shadow_stack(void)/* Run all the tests with shadow stack in place. */ shstk_is_enabled = true; -test_uretprobe_regs_equal (); +test_uprobe_regs_equal (false); +test_uprobe_regs_equal (true);test_uretprobe_regs_change ();test_uretprobe_syscall_call ();
@@ -777,7 +803,7 @@ static void test_uprobe_sigill(void)static void __test_uprobe_syscall (void ) {if (test__start_subtest ("uretprobe_regs_equal" )) -test_uretprobe_regs_equal (); +test_uprobe_regs_equal (true);if (test__start_subtest ("uretprobe_regs_change" ))test_uretprobe_regs_change ();if (test__start_subtest ("uretprobe_syscall_call" ))
@@ -796,6 +822,8 @@ static void __test_uprobe_syscall(void)test_uprobe_race ();if (test__start_subtest ("uprobe_sigill" ))test_uprobe_sigill (); +if (test__start_subtest ("uprobe_regs_equal" )) + test_uprobe_regs_equal(false); }#else static void __test_uprobe_syscall (void )
tools/testing/selftests/bpf/progs/uprobe_syscall.c +2 -2
@@ -7,8 +7,8 @@ struct pt_regs regs; char _license[] SEC("license") = "GPL"; -SEC("uretprobe//proc/self/exe:uretprobe_regs_trigger") -int uretprobe (struct pt_regs *ctx) +SEC("uprobe") +int probe (struct pt_regs *ctx) {__builtin_memcpy (®s, ctx,sizeof (regs));return 0 ;
Signature · 1 line
2.49.0
Jiri Olsa
reviewerPATCH v3 19/22 selftests/bpf: Change test_uretprobe_regs_change for uprobe and uretprobe
In reply to Jiri OlsaChanging the test_uretprobe_regs_change test to test both uprobe and uretprobe by adding entry consumer handler to the testmod and making it to change one of the registers.
Making sure that changed values both uprobe and uretprobe handlers propagate to the user space.
--- .../selftests/bpf/prog_tests/uprobe_syscall.c | 12 ++++++++---- tools/testing/selftests/bpf/test_kmods/bpf_testmod.c | 11 +++++++++-- 2 files changed, 17 insertions(+), 6 deletions(-)
tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c +8 -4
@@ -207,7 +207,7 @@ static int write_bpf_testmod_uprobe(unsigned long offset)return ret != n ? (int ) ret :0 ; } -static void test_uretprobe_regs_change (void ) +static void test_regs_change (void ) {struct pt_regs before = {}, after = {};unsigned long *pb = (unsigned long *) &before;
@@ -221,6 +221,9 @@ static void test_uretprobe_regs_change(void)if (!ASSERT_OK (err,"register_uprobe" ))return ; +/* make sure uprobe gets optimized */ +uprobe_regs_trigger (); +uprobe_regs (&before, &after); err =write_bpf_testmod_uprobe (0 );
@@ -648,7 +651,6 @@ static void test_uretprobe_shadow_stack(void)test_uprobe_regs_equal (false);test_uprobe_regs_equal (true); -test_uretprobe_regs_change ();test_uretprobe_syscall_call ();test_uprobe_legacy ();
@@ -656,6 +658,8 @@ static void test_uretprobe_shadow_stack(void)test_uprobe_session ();test_uprobe_usdt (); +test_regs_change (); + shstk_is_enabled = false;ARCH_PRCTL (ARCH_SHSTK_DISABLE ,ARCH_SHSTK_SHSTK );
@@ -804,8 +808,6 @@ static void __test_uprobe_syscall(void) {if (test__start_subtest ("uretprobe_regs_equal" ))test_uprobe_regs_equal (true); -if (test__start_subtest ("uretprobe_regs_change" )) -test_uretprobe_regs_change ();if (test__start_subtest ("uretprobe_syscall_call" ))test_uretprobe_syscall_call ();if (test__start_subtest ("uretprobe_shadow_stack" ))
@@ -824,6 +826,8 @@ static void __test_uprobe_syscall(void)test_uprobe_sigill ();if (test__start_subtest ("uprobe_regs_equal" ))test_uprobe_regs_equal (false); +if (test__start_subtest ("regs_change" )) + test_regs_change(); }#else static void __test_uprobe_syscall (void )
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +9 -2
@@ -500,15 +500,21 @@ static struct bin_attribute bin_attr_bpf_testmod_file __ro_after_init = {*/ #ifdef __x86_64__ +static int +uprobe_handler (struct uprobe_consumer *self,struct pt_regs *regs,__u64 *data) +{ + regs->cx =0x87654321feebdaed ; +return 0 ; +} +static int uprobe_ret_handler (struct uprobe_consumer *self,unsigned long func,struct pt_regs *regs,__u64 *data) { regs->ax =0x12345678deadbeef ; - regs->cx =0x87654321feebdaed ; regs->r11 = (u64) -1 ; -return true; +return 0 ; } struct testmod_uprobe {
@@ -520,6 +526,7 @@ struct testmod_uprobe {static DEFINE_MUTEX (testmod_uprobe_mutex);static struct testmod_uprobe uprobe = { + .consumer.handler = uprobe_handler, .consumer.ret_handler = uprobe_ret_handler, };
Signature · 1 line
2.49.0
Jiri Olsa
reviewerPATCH v3 20/22 seccomp: passthrough uprobe systemcall without filtering
In reply to Jiri OlsaAdding uprobe as another exception to the seccomp filter alongside with the uretprobe syscall.
Same as the uretprobe the uprobe syscall is installed by kernel as replacement for the breakpoint exception and is limited to x86_64 arch and isn't expected to ever be supported in i386.
--- kernel/seccomp.c | 32 +++++++++++++++++++++++++------- 1 file changed, 25 insertions(+), 7 deletions(-)
kernel/seccomp.c +25 -7
@@ -741,6 +741,26 @@ seccomp_prepare_user_filter(const char __user *user_filter) }#ifdef SECCOMP_ARCH_NATIVE +static bool seccomp_uprobe_exception (struct seccomp_data *sd) +{ +#if defined __NR_uretprobe || defined __NR_uprobe +#ifdef SECCOMP_ARCH_COMPAT + if (sd->arch == SECCOMP_ARCH_NATIVE) +#endif + { +#ifdef __NR_uretprobe +if (sd->nr == __NR_uretprobe) +return true; +#endif +#ifdef __NR_uprobe +if (sd->nr == __NR_uprobe) +return true; +#endif + } +#endif +return false; +} +/** * seccomp_is_const_allow - check if filter is constant allow with given data * @fprog: The BPF programs
@@ -758,13 +778,8 @@ static bool seccomp_is_const_allow(struct sock_fprog_kern *fprog,return false;/* Our single exception to filtering. */ -#ifdef __NR_uretprobe -#ifdef SECCOMP_ARCH_COMPAT - if (sd->arch == SECCOMP_ARCH_NATIVE) -#endif -if (sd->nr == __NR_uretprobe) -return true; -#endif +if (seccomp_uprobe_exception (sd)) +return true;for (pc =0 ; pc < fprog->len; pc++) {struct sock_filter *insn = &fprog->filter[pc];
@@ -1042,6 +1057,9 @@ static const int mode1_syscalls[] = { __NR_seccomp_read, __NR_seccomp_write, __NR_seccomp_exit, __NR_seccomp_sigreturn, #ifdef __NR_uretprobe __NR_uretprobe, +#endif +#ifdef __NR_uprobe + __NR_uprobe, #endif -1, /* negative terminated */ };
Signature · 1 line
2.49.0
Jiri Olsa
reviewerPATCH v3 21/22 selftests/seccomp: validate uprobe syscall passes through seccomp
In reply to Jiri OlsaAdding uprobe checks into the current uretprobe tests.
All the related tests are now executed with attached uprobe or uretprobe or without any probe.
Renaming the test fixture to uprobe, because it seems better.
--- tools/testing/selftests/seccomp/seccomp_bpf.c | 107 ++++++++++++++---- 1 file changed, 86 insertions(+), 21 deletions(-)
tools/testing/selftests/seccomp/seccomp_bpf.c +86 -21
@@ -73,6 +73,14 @@#define noinline __attribute__((noinline))#endif +#ifndef __nocf_check +#define __nocf_check __attribute__((nocf_check)) +#endif + +#ifndef __naked +#define __naked __attribute__((__naked__)) +#endif +#ifndef PR_SET_NO_NEW_PRIVS #define PR_SET_NO_NEW_PRIVS 38#define PR_GET_NO_NEW_PRIVS 39
@@ -4896,7 +4904,36 @@ TEST(tsync_vs_dead_thread_leader)EXPECT_EQ (0 , status); } -noinline int probed(void) +#ifdef __x86_64__ + +/* +* We need naked probed_uprobe function. Using __nocf_check +* check to skip possible endbr64 instruction and ignoring +* -Wattributes, otherwise the compilation might fail. +*/ +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wattributes" + +__naked __nocf_check noinline int probed_uprobe(void) +{ +/* +* Optimized uprobe is possible only on top of nop5 instruction. +*/ + asmvolatile (" \n" +".byte 0x0f, 0x1f, 0x44, 0x00, 0x00 \n" +"ret \n" + ); +} +#pragma GCC diagnostic pop + +#else +noinline int probed_uprobe(void) +{ +return 1 ; +} +#endif + +noinline int probed_uretprobe(void) {return 1 ; }
@@ -4949,35 +4986,46 @@ static ssize_t get_uprobe_offset(const void *addr)return found ? (uintptr_t )addr - start + base :-1 ; } -FIXTURE (URETPROBE ) { +FIXTURE (UPROBE ) {int fd; }; -FIXTURE_VARIANT (URETPROBE ) { +FIXTURE_VARIANT (UPROBE ) {/* -* All of the URETPROBE behaviors can be tested with either -* uretprobe attached or not +* All of the U(RET)PROBE behaviors can be tested with either +* u(ret)probe attached or not */ bool attach; +/* +* Test both uprobe and uretprobe. +*/ +bool uretprobe; }; -FIXTURE_VARIANT_ADD(URETPROBE, attached) { +FIXTURE_VARIANT_ADD(UPROBE, not_attached) { + .attach = false, + .uretprobe = false, +}; + +FIXTURE_VARIANT_ADD(UPROBE, uprobe_attached) { .attach = true, + .uretprobe = false, }; -FIXTURE_VARIANT_ADD(URETPROBE, not_attached) { - .attach = false, +FIXTURE_VARIANT_ADD(UPROBE, uretprobe_attached) { + .attach = true, + .uretprobe = true, }; -FIXTURE_SETUP (URETPROBE ) +FIXTURE_SETUP (UPROBE ) {const size_t attr_sz =sizeof (struct perf_event_attr );struct perf_event_attr attr;ssize_t offset;int type, bit; -#ifndef __NR_uretprobe -SKIP (return,"__NR_uretprobe syscall not defined" ); +#if !defined(__NR_uprobe) || !defined(__NR_uretprobe) +SKIP (return,"__NR_uprobe ot __NR_uretprobe syscalls not defined" );#endif if (!variant->attach)
@@ -4987,12 +5035,17 @@ FIXTURE_SETUP(URETPROBE) type =determine_uprobe_perf_type ();ASSERT_GE (type,0 ); - bit =determine_uprobe_retprobe_bit (); -ASSERT_GE (bit,0 ); - offset =get_uprobe_offset (probed); + +if (variant->uretprobe) { + bit =determine_uprobe_retprobe_bit (); +ASSERT_GE (bit,0 ); + } + + offset =get_uprobe_offset (variant->uretprobe ? probed_uretprobe : probed_uprobe);ASSERT_GE (offset,0 ); - attr.config |=1 << bit; +if (variant->uretprobe) + attr.config |=1 << bit; attr.size = attr_sz; attr.type = type; attr.config1 =ptr_to_u64 ("/proc/self/exe" );
@@ -5003,7 +5056,7 @@ FIXTURE_SETUP(URETPROBE) PERF_FLAG_FD_CLOEXEC); } -FIXTURE_TEARDOWN (URETPROBE ) +FIXTURE_TEARDOWN (UPROBE ) {/* we could call close(self->fd), but we'd need extra filter for * that and since we are calling _exit right away..
@@ -5017,11 +5070,17 @@ static int run_probed_with_filter(struct sock_fprog *prog)return -1 ; } -probed (); +/* +* Uprobe is optimized after first hit, so let's hit twice. +*/ +probed_uprobe (); +probed_uprobe (); + +probed_uretprobe ();return 0 ; } -TEST_F(URETPROBE, uretprobe_default_allow) +TEST_F(UPROBE, uprobe_default_allow) {struct sock_filter filter[] = {BPF_STMT (BPF_RET |BPF_K ,SECCOMP_RET_ALLOW ),
@@ -5034,7 +5093,7 @@ TEST_F(URETPROBE, uretprobe_default_allow)ASSERT_EQ (0 ,run_probed_with_filter (&prog)); } -TEST_F(URETPROBE, uretprobe_default_block) +TEST_F(UPROBE, uprobe_default_block) { struct sock_filter filter[] = { BPF_STMT(BPF_LD|BPF_W|BPF_ABS,
@@ -5051,11 +5110,14 @@ TEST_F(URETPROBE, uretprobe_default_block)ASSERT_EQ (0 ,run_probed_with_filter (&prog)); } -TEST_F(URETPROBE, uretprobe_block_uretprobe_syscall) +TEST_F(UPROBE, uprobe_block_syscall) {struct sock_filter filter[] = {BPF_STMT (BPF_LD |BPF_W |BPF_ABS , offsetof(struct seccomp_data , nr)), +#ifdef __NR_uprobe +BPF_JUMP (BPF_JMP |BPF_JEQ |BPF_K , __NR_uprobe,1 ,2 ), +#endif #ifdef __NR_uretprobeBPF_JUMP (BPF_JMP |BPF_JEQ |BPF_K , __NR_uretprobe,0 ,1 ), #endif
@@ -5070,11 +5132,14 @@ TEST_F(URETPROBE, uretprobe_block_uretprobe_syscall)ASSERT_EQ (0 ,run_probed_with_filter (&prog)); } -TEST_F(URETPROBE, uretprobe_default_block_with_uretprobe_syscall) +TEST_F(UPROBE, uprobe_default_block_with_syscall) {struct sock_filter filter[] = {BPF_STMT (BPF_LD |BPF_W |BPF_ABS , offsetof(struct seccomp_data , nr)), +#ifdef __NR_uprobe +BPF_JUMP (BPF_JMP |BPF_JEQ |BPF_K , __NR_uprobe,3 ,0 ), +#endif #ifdef __NR_uretprobeBPF_JUMP (BPF_JMP |BPF_JEQ |BPF_K , __NR_uretprobe,2 ,0 ), #endif
Signature · 1 line
2.49.0
Jiri Olsa
reviewerPATCH v3 22/22 man2: Add uprobe syscall page
In reply to Jiri OlsaChanging uretprobe syscall man page to be shared with new uprobe syscall man page.
--- man/man2/uprobe.2 | 1 + man/man2/uretprobe.2 | 36 ++++++++++++++++++++++++------------ 2 files changed, 25 insertions(+), 12 deletions(-) create mode 100644 man/man2/uprobe.2
man/man2/uprobe.2 new file +1 -0
@@ -0,0 +1 @@ +.so man2/uretprobe.2
man/man2/uretprobe.2 +24 -12
@@ -2,22 +2,28 @@ .\" .\" SPDX-License-Identifier: Linux-man-pages-copyleft .\" -.TH uretprobe 2 (date) "Linux man-pages (unreleased)" +.TH uprobe 2 (date) "Linux man-pages (unreleased)" .SH NAME +uprobe, uretprobe \- -execute pending return uprobes +execute pending entry or return uprobes .SH SYNOPSIS .nf +.B int uprobe(void); .B int uretprobe(void); .fi .SH DESCRIPTION +.BR uprobe () +is an alternative to breakpoint instructions +for triggering entry uprobe consumers. +.P .BR uretprobe () is an alternative to breakpoint instructions for triggering return uprobe consumers. .P Calls to -.BR uretprobe () +these system calls are only made from the user-space trampoline provided by the kernel. Calls from any other place result in a .BR SIGILL .
@@ -26,22 +32,28 @@ The return value is architecture-specific. .SH ERRORS .TP .B SIGILL -.BR uretprobe () -was called by a user-space program. +These system calls +were called by a user-space program. .SH VERSIONS The behavior varies across systems. .SH STANDARDS None. .SH HISTORY +.TP +.BR uprobe () +TBD +.TP +.BR uretprobe () Linux 6.11. .P -.BR uretprobe () -was initially introduced for the x86_64 architecture -where it was shown to be faster than breakpoint traps. -It might be extended to other architectures. +These system calls +were initially introduced for the x86_64 architecture +where they were shown to be faster than breakpoint traps. +They might be extended to other architectures. .SH CAVEATS -.BR uretprobe () -exists only to allow the invocation of return uprobe consumers. -It should +These system calls +exist only to allow the invocation of +entry or return uprobe consumers. +They should .B never be called directly.
Signature · 1 line
2.49.0
Re: [PATCHv3 22/22] man2: Add uprobe syscall page
Quoted 4 lines · Jiri Olsa
Changing uretprobe syscall man page to be shared with new uprobe syscall man page.
Cc Alejandro Colomar <[email protected]>
Quoted 91 lines
Signed-off-by Jiri Olsa <[email protected]>--- man/man2/uprobe.2 | 1 + man/man2/uretprobe.2 | 36 ++++++++++++++++++++++++------------ 2 files changed, 25 insertions(+), 12 deletions(-) create mode 100644 man/man2/uprobe.2man/man2/uprobe.2 new file +1
-0@@ -0,0 +1 @@ +.so man2/uretprobe.2man/man2/uretprobe.2 +24
-12@@ -2,22 +2,28 @@ .\" .\" SPDX-License-Identifier: Linux-man-pages-copyleft .\" -.TH uretprobe 2 (date) "Linux man-pages (unreleased)" +.TH uprobe 2 (date) "Linux man-pages (unreleased)" .SH NAME +uprobe, uretprobe \- -execute pending return uprobes +execute pending entry or return uprobes .SH SYNOPSIS .nf +.B int uprobe(void); .B int uretprobe(void); .fi .SH DESCRIPTION +.BR uprobe () +is an alternative to breakpoint instructions +for triggering entry uprobe consumers. +.P .BR uretprobe () is an alternative to breakpoint instructions for triggering return uprobe consumers. .P Calls to -.BR uretprobe () +these system calls are only made from the user-space trampoline provided by the kernel. Calls from any other place result in a .BR SIGILL .@@ -26,22 +32,28 @@ The return value is architecture-specific. .SH ERRORS .TP .B SIGILL -.BR uretprobe () -was called by a user-space program. +These system calls +were called by a user-space program. .SH VERSIONS The behavior varies across systems. .SH STANDARDS None. .SH HISTORY +.TP +.BR uprobe () +TBD +.TP +.BR uretprobe () Linux 6.11. .P -.BR uretprobe () -was initially introduced for the x86_64 architecture -where it was shown to be faster than breakpoint traps. -It might be extended to other architectures. +These system calls +were initially introduced for the x86_64 architecture +where they were shown to be faster than breakpoint traps. +They might be extended to other architectures. .SH CAVEATS -.BR uretprobe () -exists only to allow the invocation of return uprobe consumers. -It should +These system calls +exist only to allow the invocation of +entry or return uprobe consumers. +They should .B never be called directly.Signature · 1 line
2.49.0
Signature · 1 line
<https://www.alejandro-colomar.es/>
hi, ping
thanks,
jirka
Quoted 174 lines · Jiri Olsa
hi,
this patchset adds support to optimize usdt probes on top of 5-byte nop instruction.The generic approach (optimize all uprobes) is hard due to emulating possible multiple original instructions and its related issues. The usdt case, which stores 5-byte nop seems much easier, so starting with that.
The basic idea is to replace breakpoint exception with syscall which is faster on
x86_64. For more details please see changelog of patch 8.The
run_bench_uprobes.shbenchmark triggers uprobe (on top of different original instructions) in a loop and counts how many of those happened per second (the unit below is million loops).There's big speed up if you consider current usdt implementation (uprobe-nop) compared to proposed usdt (uprobe-nop5):
current:
usermode-count : 152.501 ± 0.012M/s syscall-count : 14.463 ± 0.062M/s --> uprobe-nop : 3.160 ± 0.005M/s uprobe-push : 3.003 ± 0.003M/s uprobe-ret : 1.100 ± 0.003M/s uprobe-nop5 : 3.132 ± 0.012M/s uretprobe-nop : 2.103 ± 0.002M/s uretprobe-push : 2.027 ± 0.004M/s uretprobe-ret : 0.914 ± 0.002M/s uretprobe-nop5 : 2.115 ± 0.002M/safter the change:
usermode-count : 152.343 ± 0.400M/s syscall-count : 14.851 ± 0.033M/s uprobe-nop : 3.204 ± 0.005M/s uprobe-push : 3.040 ± 0.005M/s uprobe-ret : 1.098 ± 0.003M/s --> uprobe-nop5 : 7.286 ± 0.017M/s uretprobe-nop : 2.144 ± 0.001M/s uretprobe-push : 2.069 ± 0.002M/s uretprobe-ret : 0.922 ± 0.000M/s uretprobe-nop5 : 3.487 ± 0.001M/sI see bit more speed up on Intel (above) compared to AMD. The big nop5 speed up is partly due to emulating nop5 and partly due to optimization.
The key speed up we do this for is the USDT switch from nop to nop5:
uprobe-nop : 3.160 ± 0.005M/s uprobe-nop5 : 7.286 ± 0.017M/sChanges from v2:
- rebased on top of tip/master +mm/mm-stable+ 1 extra change [1]
- added acks [Oleg,Andrii]
- more details changelog for patch 1 [Masami]
- several tests changes [Andrii]
- add explicitPAGE_SIZElow limit tovm_unmapped_areacall [Andrii]This patchset is adding new syscall, here are notes to check list items in
Documentation/process/adding-syscalls.rst:- System Call Alternatives
New syscall seems like the best way in here, because we need just to quickly enter kernel with no extra arguments processing, which we'd need to do if we decided to use another syscall.- Designing the API: Planning for Extension
The uprobe syscall is very specific and most likely won't be extended in the future.- Designing the API: Other Considerations
N/A because uprobe syscall does not return reference to kernel object.- Proposing the API
Wiring up of the uprobe system call is in separate change, selftests and man page changes are part of the patchset.- Generic System Call Implementation
There's no CONFIG option for the new functionality because it keeps the same behaviour from the user POV.- x86 System Call Implementation
It's 64-bit syscall only.- Compatibility System Calls (Generic)
N/A uprobe syscall has no arguments and is not supported for compat processes.- Compatibility System Calls (x86)
N/A uprobe syscall is not supported for compat processes.- System Calls Returning Elsewhere
N/A.- Other Details
N/A.- Testing
Adding new bpf selftests.- Man Page
Attached.- Do not call System Calls in the Kernel
N/Apending todo (or follow ups):
- usePROCMAP_QUERYin tests
- alloc 'structuprobes_state' formm_structonly when needed [Andrii]
- usemm_cpumask(vma->vm_mm)intext_poke_syncthanks,
jirka[1] https://lore.kernel.org/linux-trace-kernel/[email protected]/T/#u
---
Jiri Olsa (21):
uprobes: Remove breakpoint inunapply_uprobeundermmap_write_lock
uprobes: Renamearch_uretprobe_trampolinefunction
uprobes: Makecopy_from_pageglobal
uprobes: Adduprobe_writefunction
uprobes: Add nbytes argument touprobe_write
uprobes: Addis_registerargument touprobe_writeanduprobe_write_opcode
uprobes: Adddo_ref_ctrargument touprobe_writefunction
uprobes/x86: Add mapping for optimized uprobe trampolines
uprobes/x86: Add uprobe syscall to speed up uprobe
uprobes/x86: Add support to optimize uprobes
selftests/bpf: Importusdt.hfrom libbpf/usdt project
selftests/bpf: Reorg theuprobe_syscalltest function
selftests/bpf: Renameuprobe_syscall_executedprog totest_uretprobe_multiselftests/bpf: Add uprobe/usdt syscall tests
selftests/bpf: Add hit/attach/detach race optimized uprobe test selftests/bpf: Add uprobe syscall sigill signal test
selftests/bpf: Add optimized usdt variant for basic usdt test selftests/bpf: Adduprobe_regs_equaltest
selftests/bpf: Changetest_uretprobe_regs_changefor uprobe and uretprobe
seccomp: passthrough uprobe systemcall without filtering selftests/seccomp: validate uprobe syscall passes through seccomparch/arm/probes/uprobes/core.c | 2 +- arch/x86/entry/syscalls/syscall_64.tbl | 1 + arch/x86/include/asm/uprobes.h | 7 ++ arch/x86/kernel/uprobes.c | 525 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++- include/linux/syscalls.h | 2 + include/linux/uprobes.h | 20 +++- kernel/events/uprobes.c | 100 ++++++++++++----- kernel/fork.c | 1 + kernel/seccomp.c | 32 ++++-- kernel/sys_ni.c | 1 + tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c | 523 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------ tools/testing/selftests/bpf/prog_tests/usdt.c | 38 ++++--- tools/testing/selftests/bpf/progs/uprobe_syscall.c | 4 +- tools/testing/selftests/bpf/progs/uprobe_syscall_executed.c | 60 +++++++++- tools/testing/selftests/bpf/test_kmods/bpf_testmod.c | 11 +- tools/testing/selftests/bpf/usdt.h | 545 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ tools/testing/selftests/seccomp/seccomp_bpf.c | 107 ++++++++++++++---- 17 files changed, 1867 insertions(+), 112 deletions(-) create mode 100644 tools/testing/selftests/bpf/usdt.hJiri Olsa (1):
man2: Add uprobe syscall pageman/man2/uprobe.2 | 1 + man/man2/uretprobe.2 | 36 ++++++++++++++++++++++++------------ 2 files changed, 25 insertions(+), 12 deletions(-) create mode 100644 man/man2/uprobe.2
ping
Quoted 180 lines · Jiri Olsa
hi, ping
thanks,
jirkaQuoted 174 lines · Jiri Olsa
hi,
this patchset adds support to optimize usdt probes on top of 5-byte nop instruction.The generic approach (optimize all uprobes) is hard due to emulating possible multiple original instructions and its related issues. The usdt case, which stores 5-byte nop seems much easier, so starting with that.
The basic idea is to replace breakpoint exception with syscall which is faster on
x86_64. For more details please see changelog of patch 8.The
run_bench_uprobes.shbenchmark triggers uprobe (on top of different original instructions) in a loop and counts how many of those happened per second (the unit below is million loops).There's big speed up if you consider current usdt implementation (uprobe-nop) compared to proposed usdt (uprobe-nop5):
current:
usermode-count : 152.501 ± 0.012M/s syscall-count : 14.463 ± 0.062M/s --> uprobe-nop : 3.160 ± 0.005M/s uprobe-push : 3.003 ± 0.003M/s uprobe-ret : 1.100 ± 0.003M/s uprobe-nop5 : 3.132 ± 0.012M/s uretprobe-nop : 2.103 ± 0.002M/s uretprobe-push : 2.027 ± 0.004M/s uretprobe-ret : 0.914 ± 0.002M/s uretprobe-nop5 : 2.115 ± 0.002M/safter the change:
usermode-count : 152.343 ± 0.400M/s syscall-count : 14.851 ± 0.033M/s uprobe-nop : 3.204 ± 0.005M/s uprobe-push : 3.040 ± 0.005M/s uprobe-ret : 1.098 ± 0.003M/s --> uprobe-nop5 : 7.286 ± 0.017M/s uretprobe-nop : 2.144 ± 0.001M/s uretprobe-push : 2.069 ± 0.002M/s uretprobe-ret : 0.922 ± 0.000M/s uretprobe-nop5 : 3.487 ± 0.001M/sI see bit more speed up on Intel (above) compared to AMD. The big nop5 speed up is partly due to emulating nop5 and partly due to optimization.
The key speed up we do this for is the USDT switch from nop to nop5:
uprobe-nop : 3.160 ± 0.005M/s uprobe-nop5 : 7.286 ± 0.017M/sChanges from v2:
- rebased on top of tip/master +mm/mm-stable+ 1 extra change [1]
- added acks [Oleg,Andrii]
- more details changelog for patch 1 [Masami]
- several tests changes [Andrii]
- add explicitPAGE_SIZElow limit tovm_unmapped_areacall [Andrii]This patchset is adding new syscall, here are notes to check list items in
Documentation/process/adding-syscalls.rst:- System Call Alternatives
New syscall seems like the best way in here, because we need just to quickly enter kernel with no extra arguments processing, which we'd need to do if we decided to use another syscall.- Designing the API: Planning for Extension
The uprobe syscall is very specific and most likely won't be extended in the future.- Designing the API: Other Considerations
N/A because uprobe syscall does not return reference to kernel object.- Proposing the API
Wiring up of the uprobe system call is in separate change, selftests and man page changes are part of the patchset.- Generic System Call Implementation
There's no CONFIG option for the new functionality because it keeps the same behaviour from the user POV.- x86 System Call Implementation
It's 64-bit syscall only.- Compatibility System Calls (Generic)
N/A uprobe syscall has no arguments and is not supported for compat processes.- Compatibility System Calls (x86)
N/A uprobe syscall is not supported for compat processes.- System Calls Returning Elsewhere
N/A.- Other Details
N/A.- Testing
Adding new bpf selftests.- Man Page
Attached.- Do not call System Calls in the Kernel
N/Apending todo (or follow ups):
- usePROCMAP_QUERYin tests
- alloc 'structuprobes_state' formm_structonly when needed [Andrii]
- usemm_cpumask(vma->vm_mm)intext_poke_syncthanks,
jirka[1] https://lore.kernel.org/linux-trace-kernel/[email protected]/T/#u
---
Jiri Olsa (21):
uprobes: Remove breakpoint inunapply_uprobeundermmap_write_lock
uprobes: Renamearch_uretprobe_trampolinefunction
uprobes: Makecopy_from_pageglobal
uprobes: Adduprobe_writefunction
uprobes: Add nbytes argument touprobe_write
uprobes: Addis_registerargument touprobe_writeanduprobe_write_opcode
uprobes: Adddo_ref_ctrargument touprobe_writefunction
uprobes/x86: Add mapping for optimized uprobe trampolines
uprobes/x86: Add uprobe syscall to speed up uprobe
uprobes/x86: Add support to optimize uprobes
selftests/bpf: Importusdt.hfrom libbpf/usdt project
selftests/bpf: Reorg theuprobe_syscalltest function
selftests/bpf: Renameuprobe_syscall_executedprog totest_uretprobe_multiselftests/bpf: Add uprobe/usdt syscall tests
selftests/bpf: Add hit/attach/detach race optimized uprobe test selftests/bpf: Add uprobe syscall sigill signal test
selftests/bpf: Add optimized usdt variant for basic usdt test selftests/bpf: Adduprobe_regs_equaltest
selftests/bpf: Changetest_uretprobe_regs_changefor uprobe and uretprobe
seccomp: passthrough uprobe systemcall without filtering selftests/seccomp: validate uprobe syscall passes through seccomparch/arm/probes/uprobes/core.c | 2 +- arch/x86/entry/syscalls/syscall_64.tbl | 1 + arch/x86/include/asm/uprobes.h | 7 ++ arch/x86/kernel/uprobes.c | 525 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++- include/linux/syscalls.h | 2 + include/linux/uprobes.h | 20 +++- kernel/events/uprobes.c | 100 ++++++++++++----- kernel/fork.c | 1 + kernel/seccomp.c | 32 ++++-- kernel/sys_ni.c | 1 + tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c | 523 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------ tools/testing/selftests/bpf/prog_tests/usdt.c | 38 ++++--- tools/testing/selftests/bpf/progs/uprobe_syscall.c | 4 +- tools/testing/selftests/bpf/progs/uprobe_syscall_executed.c | 60 +++++++++- tools/testing/selftests/bpf/test_kmods/bpf_testmod.c | 11 +- tools/testing/selftests/bpf/usdt.h | 545 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ tools/testing/selftests/seccomp/seccomp_bpf.c | 107 ++++++++++++++---- 17 files changed, 1867 insertions(+), 112 deletions(-) create mode 100644 tools/testing/selftests/bpf/usdt.hJiri Olsa (1):
man2: Add uprobe syscall pageman/man2/uprobe.2 | 1 + man/man2/uretprobe.2 | 36 ++++++++++++++++++++++++------------ 2 files changed, 25 insertions(+), 12 deletions(-) create mode 100644 man/man2/uprobe.2
Masami Hiramatsu (Google)
maintainerHi Jiri,
Sorry I missed the series. Let me review it.
I think it should go through x86 tree because this is targetting x86 code.
Thanks,
Quoted 183 lines · On Tue, 24 Jun 2025 10:36:29 +0200 Jiri Olsa <[email protected]> wrote:
ping
Quoted 180 lines · Jiri Olsa
hi, ping
thanks,
jirkaQuoted 174 lines · Jiri Olsa
hi,
this patchset adds support to optimize usdt probes on top of 5-byte nop instruction.The generic approach (optimize all uprobes) is hard due to emulating possible multiple original instructions and its related issues. The usdt case, which stores 5-byte nop seems much easier, so starting with that.
The basic idea is to replace breakpoint exception with syscall which is faster on
x86_64. For more details please see changelog of patch 8.The
run_bench_uprobes.shbenchmark triggers uprobe (on top of different original instructions) in a loop and counts how many of those happened per second (the unit below is million loops).There's big speed up if you consider current usdt implementation (uprobe-nop) compared to proposed usdt (uprobe-nop5):
current:
usermode-count : 152.501 ± 0.012M/s syscall-count : 14.463 ± 0.062M/s --> uprobe-nop : 3.160 ± 0.005M/s uprobe-push : 3.003 ± 0.003M/s uprobe-ret : 1.100 ± 0.003M/s uprobe-nop5 : 3.132 ± 0.012M/s uretprobe-nop : 2.103 ± 0.002M/s uretprobe-push : 2.027 ± 0.004M/s uretprobe-ret : 0.914 ± 0.002M/s uretprobe-nop5 : 2.115 ± 0.002M/safter the change:
usermode-count : 152.343 ± 0.400M/s syscall-count : 14.851 ± 0.033M/s uprobe-nop : 3.204 ± 0.005M/s uprobe-push : 3.040 ± 0.005M/s uprobe-ret : 1.098 ± 0.003M/s --> uprobe-nop5 : 7.286 ± 0.017M/s uretprobe-nop : 2.144 ± 0.001M/s uretprobe-push : 2.069 ± 0.002M/s uretprobe-ret : 0.922 ± 0.000M/s uretprobe-nop5 : 3.487 ± 0.001M/sI see bit more speed up on Intel (above) compared to AMD. The big nop5 speed up is partly due to emulating nop5 and partly due to optimization.
The key speed up we do this for is the USDT switch from nop to nop5:
uprobe-nop : 3.160 ± 0.005M/s uprobe-nop5 : 7.286 ± 0.017M/sChanges from v2:
- rebased on top of tip/master +mm/mm-stable+ 1 extra change [1]
- added acks [Oleg,Andrii]
- more details changelog for patch 1 [Masami]
- several tests changes [Andrii]
- add explicitPAGE_SIZElow limit tovm_unmapped_areacall [Andrii]This patchset is adding new syscall, here are notes to check list items in
Documentation/process/adding-syscalls.rst:- System Call Alternatives
New syscall seems like the best way in here, because we need just to quickly enter kernel with no extra arguments processing, which we'd need to do if we decided to use another syscall.- Designing the API: Planning for Extension
The uprobe syscall is very specific and most likely won't be extended in the future.- Designing the API: Other Considerations
N/A because uprobe syscall does not return reference to kernel object.- Proposing the API
Wiring up of the uprobe system call is in separate change, selftests and man page changes are part of the patchset.- Generic System Call Implementation
There's no CONFIG option for the new functionality because it keeps the same behaviour from the user POV.- x86 System Call Implementation
It's 64-bit syscall only.- Compatibility System Calls (Generic)
N/A uprobe syscall has no arguments and is not supported for compat processes.- Compatibility System Calls (x86)
N/A uprobe syscall is not supported for compat processes.- System Calls Returning Elsewhere
N/A.- Other Details
N/A.- Testing
Adding new bpf selftests.- Man Page
Attached.- Do not call System Calls in the Kernel
N/Apending todo (or follow ups):
- usePROCMAP_QUERYin tests
- alloc 'structuprobes_state' formm_structonly when needed [Andrii]
- usemm_cpumask(vma->vm_mm)intext_poke_syncthanks,
jirka[1] https://lore.kernel.org/linux-trace-kernel/[email protected]/T/#u
---
Jiri Olsa (21):
uprobes: Remove breakpoint inunapply_uprobeundermmap_write_lock
uprobes: Renamearch_uretprobe_trampolinefunction
uprobes: Makecopy_from_pageglobal
uprobes: Adduprobe_writefunction
uprobes: Add nbytes argument touprobe_write
uprobes: Addis_registerargument touprobe_writeanduprobe_write_opcode
uprobes: Adddo_ref_ctrargument touprobe_writefunction
uprobes/x86: Add mapping for optimized uprobe trampolines
uprobes/x86: Add uprobe syscall to speed up uprobe
uprobes/x86: Add support to optimize uprobes
selftests/bpf: Importusdt.hfrom libbpf/usdt project
selftests/bpf: Reorg theuprobe_syscalltest function
selftests/bpf: Renameuprobe_syscall_executedprog totest_uretprobe_multiselftests/bpf: Add uprobe/usdt syscall tests
selftests/bpf: Add hit/attach/detach race optimized uprobe test selftests/bpf: Add uprobe syscall sigill signal test
selftests/bpf: Add optimized usdt variant for basic usdt test selftests/bpf: Adduprobe_regs_equaltest
selftests/bpf: Changetest_uretprobe_regs_changefor uprobe and uretprobe
seccomp: passthrough uprobe systemcall without filtering selftests/seccomp: validate uprobe syscall passes through seccomparch/arm/probes/uprobes/core.c | 2 +- arch/x86/entry/syscalls/syscall_64.tbl | 1 + arch/x86/include/asm/uprobes.h | 7 ++ arch/x86/kernel/uprobes.c | 525 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++- include/linux/syscalls.h | 2 + include/linux/uprobes.h | 20 +++- kernel/events/uprobes.c | 100 ++++++++++++----- kernel/fork.c | 1 + kernel/seccomp.c | 32 ++++-- kernel/sys_ni.c | 1 + tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c | 523 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------ tools/testing/selftests/bpf/prog_tests/usdt.c | 38 ++++--- tools/testing/selftests/bpf/progs/uprobe_syscall.c | 4 +- tools/testing/selftests/bpf/progs/uprobe_syscall_executed.c | 60 +++++++++- tools/testing/selftests/bpf/test_kmods/bpf_testmod.c | 11 +- tools/testing/selftests/bpf/usdt.h | 545 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ tools/testing/selftests/seccomp/seccomp_bpf.c | 107 ++++++++++++++---- 17 files changed, 1867 insertions(+), 112 deletions(-) create mode 100644 tools/testing/selftests/bpf/usdt.hJiri Olsa (1):
man2: Add uprobe syscall pageman/man2/uprobe.2 | 1 + man/man2/uretprobe.2 | 36 ++++++++++++++++++++++++------------ 2 files changed, 25 insertions(+), 12 deletions(-) create mode 100644 man/man2/uprobe.2
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>