ptwrite uprobes
Subsystems (12)
uprobes currently always require entering the kernel to log anything. While that works well, it is rather slow.
Modern Intel CPUs have the ptwrite instruction, which can log data to the Processor Trace buffer without entering the kernel.
This patch adds support in uprobes to patch in ptwrites instead of the normal probes. If a user collects Processor Trace with perf the logged data will appear in the PT log, otherwise the instructions will be nops.
The benefit is much faster logging, but it also has a lot of limitations. There is little filtering (other than what perf or PT can do), no EBPF, there are restrictions on what can be logged, and of course it depends on PT being recorded.
Still I find it useful.
For more details and performance numbers see the Documentation patch, but it's multiple orders of magnitude faster than classic uprobes.
This patchkit touches a variety of areas: perf tools, x86 generic, uprobes. It currently relies on a separately posted bug fix ("RCU safety for maple tree iterators"). The patchkit is on the larger side, and maybe it should be split up. But I wanted to keep it together at least for the first post. Some of the earlier patches are generic fixes for uprobes.
Comments appreciated.
-Andi Kleen
RFC 1/19 uprobes: guard trace cleanup against error pointers
Sashiko pointed out the some of the scope cleanups for free_uprobe could get an error pointer. Handle this case in free_uprobe to prevent a crash.
On the other hand the macro doesn't need the guard because free_uprobe itself already does the check.
--- kernel/trace/trace_uprobe.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/trace/trace_uprobe.c +2 -2
@@ -368,7 +368,7 @@ alloc_trace_uprobe(const char *group, const char *event, int nargs, bool is_ret)static void free_trace_uprobe (struct trace_uprobe *tu) { -if (!tu) +if (IS_ERR_OR_NULL (tu))return ;path_put (&tu->path);
@@ -533,7 +533,7 @@ static int register_trace_uprobe(struct trace_uprobe *tu)return ret; } -DEFINE_FREE(free_trace_uprobe, struct trace_uprobe *, if (_T) free_trace_uprobe(_T)) +DEFINE_FREE(free_trace_uprobe, struct trace_uprobe *, free_trace_uprobe(_T))/* * Argument syntax:
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [RFC v1 01/19] uprobes: guard trace cleanup against error pointers
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 9 lines
Sashiko pointed out the some of the scope cleanups for
free_uprobecould get an error pointer. Handle this case infree_uprobeto prevent a crash.On the other hand the macro doesn't need the guard because
free_uprobeitself already does the check.Assisted-by omp:gpt-5.6-luna sashikoSigned-off-by Andi Kleen <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Masami Hiramatsu (Google)
maintainerRe: [RFC v1 01/19] uprobes: guard trace cleanup against error pointers
In reply to Andi KleenQuoted 9 lines · On Mon, 31 Aug 2026 08:04:37 -0700 Andi Kleen <[email protected]> wrote:
Sashiko pointed out the some of the scope cleanups for
free_uprobecould get an error pointer. Handle this case infree_uprobeto prevent a crash.On the other hand the macro doesn't need the guard because
free_uprobeitself already does the check.Assisted-by omp:gpt-5.6-luna sashikoSigned-off-by Andi Kleen <[email protected]>
This looks good to me. Let me pick this to probes/fixes independently.
Thanks!
Quoted 29 lines
--- kernel/trace/trace_uprobe.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)kernel/trace/trace_uprobe.c +2
-2@@ -368,7 +368,7 @@ alloc_trace_uprobe(const char *group, const char *event, int nargs, bool is_ret)static void free_trace_uprobe (struct trace_uprobe *tu) { -if (!tu) +if (IS_ERR_OR_NULL (tu))return ;path_put (&tu->path);@@ -533,7 +533,7 @@ static int register_trace_uprobe(struct trace_uprobe *tu)return ret; } -DEFINE_FREE(free_trace_uprobe, struct trace_uprobe *, if (_T) free_trace_uprobe(_T)) +DEFINE_FREE(free_trace_uprobe, struct trace_uprobe *, free_trace_uprobe(_T))/* * Argument syntax: Signature · 1 line
2.54.0
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
RFC 2/19 uprobes: Correctly reject anonymous VMAs for breakpoint installation
In reply to Andi KleenChecking for anonymous VMAs is supposed to use this special function, not just check vm_file.
--- kernel/events/uprobes.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/events/uprobes.c +2 -1
@@ -139,7 +139,8 @@ static bool valid_vma(struct vm_area_struct *vma, bool is_register)if (is_register) flags |=VM_WRITE ; -return vma->vm_file && (vma->vm_flags & flags) ==VM_MAYEXEC ; +return !vma_is_anonymous (vma) && vma->vm_file && + (vma->vm_flags & flags) ==VM_MAYEXEC ; }static unsigned long offset_to_vaddr (struct vm_area_struct *vma,loff_t offset)
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [RFC v1 02/19] uprobes: Correctly reject anonymous VMAs for breakpoint installation
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Pre-existing issues:
- [High] Page tables are modified under mmap_write_lock without calling vma_start_write(vma), risking a race with concurrent per-VMA locked operations.
- [High] Several functions access vma->vm_file or use it for classification without first checking !vma_is_anonymous(vma), leaving them vulnerable to anonymous VMAs that have a vm_file set. --
commit e85d99904cf4af95b805f715ec281558955b0733
Author: Andi Kleen <[email protected]>
uprobes: Correctly reject anonymous VMAs for breakpoint installation
This patch updates valid_vma() to correctly check for anonymous VMAs using vma_is_anonymous() instead of relying only on vm_file.
kernel/events/uprobes.c static bool valid_vma(struct vm_area_struct *vma, bool is_register) · line 144
Quoted 12 lines
kernel/events/uprobes.c +2
-1@@ -139,7 +139,8 @@ static bool valid_vma(struct vm_area_struct *vma, bool is_register)if (is_register) flags |=VM_WRITE ; -return vma->vm_file && (vma->vm_flags & flags) ==VM_MAYEXEC ; +return !vma_is_anonymous (vma) && vma->vm_file && + (vma->vm_flags & flags) ==VM_MAYEXEC ; }
[Severity: High]
This isn't a bug introduced by this patch, but since you are fixing the anonymous VMA check here in valid_vma(), do we need to apply this same !vma_is_anonymous(vma) check to other functions in this file?
Several other functions, such as valid_ref_ctr_vma(), find_active_uprobe_speculative(), find_active_uprobe_rcu(), and uprobe_mmap(), appear to rely directly on vma->vm_file to determine if a VMA is file-backed.
For example, valid_ref_ctr_vma() checks:
return uprobe->ref_ctr_offset &&
vma->vm_file &&
file_inode(vma->vm_file) == uprobe->inode;If an anonymous VMA happens to have a backing file (e.g., from a private mapping of /dev/zero or a driver mapping using vma_set_anonymous()), could this incorrectly match the inode and update reference counters in anonymous memory, or compute incorrect file offsets in the speculative lookups?
[Severity: High]
This is a pre-existing issue, but while reviewing the VMA handling in uprobes, does this file also need to call vma_start_write(vma) when modifying page tables?
Paths like register_for_each_vma() and unapply_uprobe() take mmap_write_lock() and eventually call uprobe_write() to modify page tables via folio_walk_start() in __uprobe_write().
Because mmap_write_lock() does not drain per-VMA read locks, a concurrent page table operation (like MADV_DONTNEED) under a per-VMA lock could clear a PMD and free the PTE page while uprobe_write() is walking it.
Could this race lead to a use-after-free or page table corruption when tracers manipulate the memory layout concurrently without the protection of vma_start_write() draining concurrent VMA-lock readers?
Quoted 2 lines
kernel/events/uprobes.c +0
-0static unsigned long offset_to_vaddr (struct vm_area_struct *vma,loff_t offset)
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
RFC 3/19 uprobes: Print warning for missing breakpoint install
In reply to Andi KleenWhen installing a uprobe break point fails the error is currently silently ignored. There is no obvious place to return it to, but at least print a rate-limited warning to make it easier to debug.
--- kernel/events/uprobes.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
kernel/events/uprobes.c +7 -1
@@ -1630,7 +1630,13 @@ int uprobe_mmap(struct vm_area_struct *vma)if (!fatal_signal_pending (current) &&filter_chain (uprobe, vma->vm_mm)) {unsigned long vaddr =offset_to_vaddr (vma, uprobe->offset); -install_breakpoint (uprobe, vma, vaddr); +int err =install_breakpoint (uprobe, vma, vaddr); + +if (err) +pr_warn_ratelimited ( +"uprobes: probe %pD+0x%llx failed to install (%d)\n" , + vma->vm_file, + (unsigned long long )uprobe->offset, err); }put_uprobe (uprobe); }
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [RFC v1 03/19] uprobes: Print warning for missing breakpoint install
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Missing VMA write lock during uprobe registration exposes page table modifications to races --
commit c4ab2afca2dc57fb6a2f6789af30c2bec5a5064c
Author: Andi Kleen <[email protected]>
uprobes: Print warning for missing breakpoint install
When a uprobe breakpoint installation fails, the error is currently silently ignored. This commit adds a rate-limited warning to log these failures and make debugging easier.
kernel/events/uprobes.c int uprobe_mmap(struct vm_area_struct *vma) · line 1633
Quoted 10 lines
kernel/events/uprobes.c +1
-1@@ -1630,7 +1630,13 @@ int uprobe_mmap(struct vm_area_struct *vma)if (!fatal_signal_pending (current) &&filter_chain (uprobe, vma->vm_mm)) {unsigned long vaddr =offset_to_vaddr (vma, uprobe->offset); -install_breakpoint (uprobe, vma, vaddr); +int err =install_breakpoint (uprobe, vma, vaddr);
^^^^
[Severity: High]
This isn't a bug introduced by this patch, but modifying install_breakpoint() reminded me of a pre-existing issue where it is called from register_for_each_vma() without isolating the VMA.
In register_for_each_vma(), the code acquires mmap_write_lock(mm) and dispatches install_breakpoint() or remove_breakpoint():
kernel/events/uprobes.c:register_for_each_vma() {
...
mmap_write_lock(mm);
...
if (is_register) {
/* consult only the "caller", new consumer. */
if (consumer_filter(new, mm))
err = install_breakpoint(uprobe, vma, info->vaddr);
} else if (mm_flags_test(MMF_HAS_UPROBES, mm)) {
if (!filter_chain(uprobe, mm))
err |= remove_breakpoint(uprobe, vma, info->vaddr);
}
...
}Both install_breakpoint() and remove_breakpoint() ultimately call uprobe_write() to clear PTEs and collapse PMDs via __uprobe_write() and collapse_pte_mapped_thp().
However, as per VMA lock invariants, mmap_write_lock alone does not exclude concurrent per-VMA read lock holders (like page faults) that acquired the lock prior to the write lock.
Should register_for_each_vma() call vma_start_write(vma) before dispatching these functions? Failing to drain per-VMA lock holders could allow readers to concurrently walk the VMA's page tables while uprobe_write() clears them, potentially leading to use-after-free of PTE pages or stale PMD reads.
Quoted 7 lines
kernel/events/uprobes.c +6
-0+ +if (err) +pr_warn_ratelimited ( +"uprobes: probe %pD+0x%llx failed to install (%d)\n" , + vma->vm_file, + (unsigned long long )uprobe->offset, err); }
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
RFC 4/19 ptwrite uprobes: Add infrastructure for ptwrite uprobes
In reply to Andi Kleenuprobes currently always require entering the kernel to log anything. While that works well, it is rather slow.
Modern Intel CPUs have the ptwrite instruction, which can log data to the Processor Trace buffer. This patch adds support in uprobes to patch in ptwrites instead of the normal probes. If a user collects Processor Trace with perf the logged data will appear in the PT log, otherwise the instructions will be nops. ptwrite is a 5 byte instruction, here it can be only patched into 5 byte nops.
The benefit is much faster logging, but it also has a lot of limitations. There is no filtering, no EBPF, there are restrictions on what can be logged, and of course it depends on PT being recorded.
The instrumentation is similar to normal uprobes. Add a trampoline page. Replace the original instruction (5 byte nop) with a jump to the trampoline. The trampoline does ptwrites and then jumps back.
This is the high level infrastruture without any x86-64 specific parts (except for one data structure). Add register/unregister, basic data structures and high level hooks. Add weak stubs to handle the no uprobes or different architectures case.
--- include/linux/uprobes.h | 81 +++++++++++++++++++- kernel/events/uprobes.c | 166 +++++++++++++++++++++++++++++++++++++++- kernel/fork.c | 1 + mm/mmap.c | 8 +- 4 files changed, 251 insertions(+), 5 deletions(-)
include/linux/uprobes.h +78 -3
@@ -23,9 +23,11 @@ struct uprobe;struct vm_area_struct ;struct mm_struct ;struct inode ; +struct file ;struct notifier_block ;struct page ;struct srcu_ctr ; +struct uprobe_ptwrite_desc ;/* * Allowed return values from uprobe consumer's handler callback
@@ -187,6 +189,37 @@ struct xol_area;struct uprobes_state {struct xol_area *xol_area; +#ifdef CONFIG_X86_64 +struct hlist_head head_ptwrite; +/* Ptwrite pages and metadata use the mm mmap write lock. */ +#endif +}; + +#define UPROBE_PTWRITE_MAX_ARGS 8 + +/* +* Header word: event_id<<48 | nargs<<40 | UPROBE_PTW_HDR_MAGIC (bits 39..0). +*/ +#define UPROBE_PTW_HDR_MAGIC 0x5054525731UL/* "PTRW1" */ + +enum uprobe_ptwrite_src { +UPROBE_PTW_SRC_REG ,/* value = live GPR (index in .reg) */ +UPROBE_PTW_SRC_IMM ,/* value = constant (.val), stored in stub data slot */ +}; + +struct uprobe_ptwrite_arg { +u8 src;/* enum uprobe_ptwrite_src */ +u8 reg;/* x86-64 GPR index (0=rax..15=r15) for SRC_REG */ +u8 size;/* declared type size 1/2/4/8 (decoder hint) */ +u8 reserved; +u64 val;/* SRC_IMM: constant; SRC_REG: unused */ +}; + +struct uprobe_ptwrite_desc { +u16 event_id;/* identifier carried in the header word */ +u8 nargs; +u8 flags; +struct uprobe_ptwrite_arg args[UPROBE_PTWRITE_MAX_ARGS ]; }; typedef int (*uprobe_write_verify_t)(struct page *page, unsigned long vaddr,
@@ -205,6 +238,37 @@ extern int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma, uprobe_opcode_t *insn, int nbytes, uprobe_write_verify_t verify, bool is_register, bool do_update_ref_ctr, void *data);extern struct uprobe *uprobe_register (struct inode *inode,loff_t offset,loff_t ref_ctr_offset,struct uprobe_consumer *uc); +extern struct uprobe *uprobe_register_ptwrite (struct inode *inode, +struct file *file,loff_t offset, +struct uprobe_consumer *uc, +const struct uprobe_ptwrite_desc *desc); +extern bool arch_uprobe_ptwrite_supported (void ); +extern int arch_uprobe_ptwrite_prepare (struct arch_uprobe *auprobe, +const struct uprobe_ptwrite_desc *desc); +extern int arch_uprobe_install_ptwrite (struct arch_uprobe *auprobe, +struct vm_area_struct *vma, +unsigned long vaddr); +extern int arch_uprobe_uninstall_ptwrite (struct arch_uprobe *auprobe, +struct vm_area_struct *vma, +unsigned long vaddr); + +enum uprobe_ptwrite_fetch_kind { +UPROBE_PTW_FETCH_REG ,/* live GPR */ +UPROBE_PTW_FETCH_STACKP ,/* stack pointer value ($stack) */ +UPROBE_PTW_FETCH_STACKN ,/* [SP + imm] ($stackN, imm pre-scaled) */ +UPROBE_PTW_FETCH_MEMREG ,/* [GPR + imm] (imm = disp32) */ +UPROBE_PTW_FETCH_IMM ,/* constant */ +}; + +struct uprobe_ptwrite_fetch { +enum uprobe_ptwrite_fetch_kind kind; +unsigned int reg;/* pt_regs member offset */ +u64 imm;/* IMM value / MEMREG disp / STACKN off */ +}; + +extern int arch_uprobe_ptwrite_fetch (struct uprobe_ptwrite_arg *a, +const struct uprobe_ptwrite_fetch *f); +extern int uprobe_apply (struct uprobe *uprobe,struct uprobe_consumer *uc,bool );extern void uprobe_unregister_nosync (struct uprobe *uprobe,struct uprobe_consumer *uc);extern void uprobe_unregister_sync (void );
@@ -212,7 +276,7 @@ extern int uprobe_mmap(struct vm_area_struct *vma);extern void uprobe_munmap (struct vm_area_struct *vma,unsigned long start,unsigned long end);extern void uprobe_start_dup_mmap (void );extern void uprobe_end_dup_mmap (void ); -extern void uprobe_dup_mmap (struct mm_struct *oldmm,struct mm_struct *newmm); +extern int uprobe_dup_mmap (struct mm_struct *oldmm,struct mm_struct *newmm);extern void uprobe_free_utask (struct task_struct *t);extern void uprobe_copy_process (struct task_struct *t,u64 flags);extern int uprobe_post_sstep_notifier (struct pt_regs *regs);
@@ -236,6 +300,9 @@ extern void uprobe_handle_trampoline(struct pt_regs *regs);extern void *arch_uretprobe_trampoline (unsigned long *psize);extern unsigned long uprobe_get_trampoline_vaddr (void );extern void uprobe_copy_from_page (struct page *page,unsigned long vaddr,void *dst,int len); +extern void arch_uprobe_clear_state (struct mm_struct *mm); +extern void arch_uprobe_init_state (struct mm_struct *mm); +extern int arch_uprobe_dup_ptwrite (struct mm_struct *oldmm,struct mm_struct *newmm);extern void handle_syscall_uprobe (struct pt_regs *regs,unsigned long bp_vaddr);extern void arch_uprobe_optimize (struct arch_uprobe *auprobe,unsigned long vaddr);extern unsigned long arch_uprobe_get_xol_area (void );
@@ -254,6 +321,13 @@ uprobe_register(struct inode *inode, loff_t offset, loff_t ref_ctr_offset, struc {return ERR_PTR (-ENOSYS ); } +static inline struct uprobe * +uprobe_register_ptwrite (struct inode *inode,struct file *file,loff_t offset, +struct uprobe_consumer *uc, +const struct uprobe_ptwrite_desc *desc) +{ +return ERR_PTR (-ENOSYS ); +}static inline int uprobe_apply (struct uprobe * uprobe,struct uprobe_consumer *uc,bool add) {
@@ -280,9 +354,10 @@ static inline void uprobe_start_dup_mmap(void)static inline void uprobe_end_dup_mmap (void ) { } -static inline void -uprobe_dup_mmap (struct mm_struct *oldmm,struct mm_struct *newmm) +static inline int uprobe_dup_mmap (struct mm_struct *oldmm, +struct mm_struct *newmm) { +return 0 ; }static inline void uprobe_notify_resume (struct pt_regs *regs) {
kernel/events/uprobes.c +165 -1
214 lines not on this page · raw message
kernel/fork.c +1 -0
@@ -1076,6 +1076,7 @@ static void mm_init_uprobes_state(struct mm_struct *mm) {#ifdef CONFIG_UPROBES mm->uprobes_state.xol_area =NULL ; +arch_uprobe_init_state (mm);#endif }
mm/mmap.c +7 -1
@@ -1716,7 +1716,6 @@ __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm)if (mmap_write_lock_killable (oldmm))return -EINTR ;flush_cache_dup_mm (oldmm); -uprobe_dup_mmap (oldmm, mm);/* * Not linked in yet - no deadlock potential: */
@@ -1825,6 +1824,13 @@ __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm) }/* a new mm has just been created */ retval =arch_dup_mmap (oldmm, mm); +if (!retval) { +/* +* The arch state follows the fully populated child maple tree. A +* non-fatal allocation failure can leave a child ptwrite hit faulting. +*/ + retval =uprobe_dup_mmap (oldmm, mm); + } loop_out:vma_iter_free (&vmi);if (!retval) {
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [RFC v1 04/19] ptwrite uprobes: Add infrastructure for ptwrite uprobes
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [Critical] handle_swbp() does not check for UPROBE_PTWRITE, leading to a kernel crash (NULL pointer…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Critical] handle_swbp() does not check for UPROBE_PTWRITE, leading to a kernel crash (NULL pointer dereference) if a thread traps at a PTWRITE uprobe address.
- [High] uprobe_register_ptwrite incorrectly reuses dying normal uprobes, leading to race conditions and SIGILL.
- [Low] uprobe_register_ptwrite requires a non-NULL handler despite documenting that it is never invoked.
--
commit 25466f8679470946e2456212a51efd54e55bf02c
Author: Andi Kleen <[email protected]>
ptwrite uprobes: Add infrastructure for ptwrite uprobes
uprobes currently always require entering the kernel to log anything. While that works well, it is rather slow.
Modern Intel CPUs have the ptwrite instruction, which can log data to the Processor Trace buffer. This patch adds support in uprobes to patch in ptwrites instead of the normal probes.
kernel/events/uprobes.c DEFINE_STATIC_SRCU_FAST_UPDOWN(uretprobes_srcu); · line 63
Quoted 10 lines
kernel/events/uprobes.c +2
-0@@ -59,6 +59,8 @@ DEFINE_STATIC_SRCU_FAST_UPDOWN(uretprobes_srcu);/* Have a copy of original instruction */ #define UPROBE_COPY_INSN 0 +#define UPROBE_PTWRITE 1 +
[Severity: Critical]
Since PTWRITE uprobes do not have an initialized uprobe->arch.ops pointer, what happens if an unprivileged user attaches with ptrace and inserts a breakpoint at the exact address of the PTWRITE uprobe?
If the process hits the breakpoint, it enters the kernel's handle_swbp(), looks up the uprobe, and proceeds to call arch_uprobe_skip_sstep(). Since handle_swbp() doesn't check for the UPROBE_PTWRITE flag, wouldn't this unconditionally dereference the uninitialized auprobe->ops->emulate, resulting in a kernel panic (null pointer dereference)?
[ ... ]
kernel/events/uprobes.c struct uprobe *uprobe_register(struct inode *inode, · line 1542
Quoted 9 lines
kernel/events/uprobes.c +9
-0+struct uprobe *uprobe_register_ptwrite (struct inode *inode,struct file *file, +loff_t offset,struct uprobe_consumer *uc, +const struct uprobe_ptwrite_desc *desc) +{ +struct uprobe *uprobe; +int ret; + +if (!file || (!uc->handler && !uc->ret_handler)) +return ERR_PTR (-EINVAL );
[Severity: Low]
Is this validation check correct for a PTWRITE uprobe? The comment above states "(handler is never invoked: no kernel entry at probe hit)", but this check requires the caller to supply a non-null handler anyway.
[ ... ]
kernel/events/uprobes.c struct uprobe *uprobe_register(struct inode *inode, · line 1575
Quoted 10 lines
kernel/events/uprobes.c +10
-0+down_write (&uprobe->register_rwsem); + +/* +* A dying normal uprobe can make reuse temporarily busy; don't overwrite +* it. +*/ +if (!list_empty (&uprobe->consumers)) { + ret = -EBUSY ; + goto out; + }
[Severity: High]
Does this logic safely handle dying normal uprobes?
If a normal uprobe is unregistered, its consumers list becomes empty, but threads might still be hitting the breakpoint and executing handle_swbp(). Concurrently, uprobe_register_ptwrite() called for the same address will reuse the dying uprobe because its consumers list is empty, and then mutate its arch state to PTWRITE, installing a 5-byte jump.
Won't the threads finishing XOL for the normal uprobe return to bp_vaddr + ilen, which now points into the middle of the 5-byte jump, resulting in an invalid opcode exception for the user-space process?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
RFC 5/19 ptwrite uprobes: Add minimal low level support for x86
In reply to Andi KleenAdd more data structures and the x86 machinery to generate the PTWRITE instructions for a ptwrite uprobe. The probe executes PTWRITEs and then jumps back to the original code. In this variant only patching 5 byte nops is supported.
The instructions are pre-generated to templates and then patched when setting up the final user page.
The patching code uses 3 phase patching similar to int3_update.
The ptwrite stub emits a header with a magic value and the number of arguments, and then the actual probed values.
There is no separate config option for ptwrite uprobes, it is just tied to the main uprobes config.
Some limitations in the current implementation:
- The probed 5 byte area cannot cross a page.
- The allocated stubs in the user program are only freed on exit.
--- arch/x86/include/asm/uprobes.h | 24 ++ arch/x86/kernel/uprobes.c | 623 +++++++++++++++++++++++++++++++++ 2 files changed, 647 insertions(+)
arch/x86/include/asm/uprobes.h +24 -0
@@ -23,10 +23,33 @@ typedef u8 uprobe_opcode_t;enum {ARCH_UPROBE_FLAG_CAN_OPTIMIZE =0 ,ARCH_UPROBE_FLAG_OPTIMIZE_FAIL =1 , +ARCH_UPROBE_FLAG_PTWRITE =2 , };struct uprobe_xol_ops ; +/* +* ptwrite probe state. The stub template (code + data slots) is built +* once at registration (mm-independent except the final jmp's rel32, patched +* per-mm at install). Block layout: +* [ptwriteq hdr(%rip)] [arg emissions] [jmp probe+5] [u64 slots: header, imms] +*/ +struct uprobe_ptwrite_arch { +u8 stub[256 ]; +u8 stub_len;/* code + data, whole block */ +u8 jmp_off;/* offset of the final jmp's rel32 field */ +u8 ndata;/* number of u64 data slots */ +u8 orig[MAX_UINSN_BYTES ];/* pristine file bytes, before generic analysis */ +}; + +/* Per-mm page holding generated ptwrite stub blocks (mirrors trampolines). */ +struct uprobe_ptwrite_page { +struct hlist_node node; +struct page *page;/* stub blocks written via kmap */ +unsigned long vaddr;/* mapping base */ +u16 cursor;/* next free block offset */ +}; + struct arch_uprobe { union { u8 insn[MAX_UINSN_BYTES];
@@ -51,6 +74,7 @@ struct arch_uprobe { } push; }; +struct uprobe_ptwrite_arch ptwrite;unsigned long flags; };
arch/x86/kernel/uprobes.c +623 -0
669 lines not on this page · raw message
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [RFC v1 05/19] ptwrite uprobes: Add minimal low level support for x86
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] Calling synchronize_rcu() inside a loop in arch_uprobe_clear_state() causes O(N) grace period…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] Calling synchronize_rcu() inside a loop in arch_uprobe_clear_state() causes O(N) grace period delays while holding mmap_write_lock.
- [High] find_ptwrite_page_area bypasses mmap_min_addr security protections.
- [High] Child processes execute the parent's ptwrite page due to missing VM_WIPEONFORK/VM_DONTCOPY on the special mapping. --
commit eb94d55c50bf768d31e8cac28283dd4258bc8a40
Author: Andi Kleen <[email protected]>
ptwrite uprobes: Add minimal low level support for x86
This commit adds low-level x86 support for generating and executing PTWRITE instructions for uprobes. It pre-generates instructions into templates and patches them when setting up the user page.
Quoted 4 lines
arch/x86/kernel/uprobes.c +0
-0
[ ... ]
Quoted 1 line
arch/x86/kernel/uprobes.c +0
-0@@ -717,6 +721,68 @@ static struct vm_area_struct *get_uprobe_trampoline(struct mm_struct *mm, unsign
[ ... ]
arch/x86/kernel/uprobes.c static struct vm_area_struct *get_uprobe_trampoline(struct mm_struct *mm, unsign · line 741
Quoted 9 lines
arch/x86/kernel/uprobes.c +9
-0+void arch_uprobe_clear_state (struct mm_struct *mm) +{ +struct uprobes_state *state = &mm->uprobes_state; +struct uprobe_ptwrite_page *ptw; +struct hlist_node *n; + +hlist_for_each_entry_safe (ptw, n, &state->head_ptwrite, node) { +hlist_del_rcu (&ptw->node); +synchronize_rcu ();
[Severity: High]
Does calling synchronize_rcu() inside this loop cause an O(N) grace period delay while holding the mmap_write_lock?
As shown in arch_uprobe_dup_ptwrite() below, arch_uprobe_clear_state() is called on the failure path while both the parent and child's mmap_write_lock are held by dup_mmap(). An attacker could potentially fork and intentionally cause memory allocation failures to stall the parent's mmap_write_lock for several RCU grace periods, leading to a Denial of Service.
Quoted 13 lines
arch/x86/kernel/uprobes.c +13
-0+__free_page (ptw->page); +kfree (ptw); + } +} + +int arch_uprobe_dup_ptwrite (struct mm_struct *oldmm,struct mm_struct *newmm) +{ +struct uprobes_state *old_state = &oldmm->uprobes_state; +struct uprobes_state *new_state = &newmm->uprobes_state; +struct uprobe_ptwrite_page *ptw, *new; + +mmap_assert_write_locked (oldmm); +mmap_assert_write_locked (newmm);
[ ... ]
Quoted 4 lines
arch/x86/kernel/uprobes.c +4
-0+fail: +arch_uprobe_clear_state (newmm); +return -ENOMEM ; +}
[ ... ]
Quoted 1 line
arch/x86/kernel/uprobes.c +0
-0@@ -1102,6 +1223,508 @@ static int copy_from_vaddr(struct mm_struct *mm, unsigned long vaddr, void *dst,
[ ... ]
Quoted 7 lines
arch/x86/kernel/uprobes.c +7
-0+static unsigned long find_ptwrite_page_area (struct mm_struct *mm, +unsigned long vaddr) +{ +VMA_ITERATOR (vmi, mm,0 ); +struct vm_area_struct *vma; +unsigned long low, high, prev, call_end; +const unsigned long call_range = (unsigned long )INT_MAX +1 ;
arch/x86/kernel/uprobes.c static int copy_from_vaddr(struct mm_struct *mm, unsigned long vaddr, void *dst, · line 1471
Quoted 10 lines
arch/x86/kernel/uprobes.c +10
-0+ +mmap_assert_write_locked (mm); +if (check_add_overflow (vaddr,5UL , &call_end)) +return -ENOMEM ; +if (call_end < call_range) + low =PAGE_SIZE ; +else + low = call_end - call_range; +if (low <PAGE_SIZE ) + low =PAGE_SIZE ;
[Severity: High]
Does hardcoding the lower bound to PAGE_SIZE bypass mmap_min_addr security protections?
If a traced user-space process installs a probe near the bottom of its address space, find_ptwrite_page_area() will search starting at PAGE_SIZE (0x1000). If a gap is found, _install_special_mapping() could map the executable ptwrite VMA near NULL, which undermines the sysctl protection against kernel NULL-pointer dereference exploits.
Quoted 3 lines
arch/x86/kernel/uprobes.c +3
-0+if (low >ULONG_MAX - (PAGE_SIZE -1 )) +return -ENOMEM ; + low =PAGE_ALIGN (low);
[ ... ]
Quoted 17 lines
arch/x86/kernel/uprobes.c +17
-0+static struct uprobe_ptwrite_page * +create_uprobe_ptwrite_page (struct mm_struct *mm,unsigned long vaddr) +{ +struct uprobe_ptwrite_page *ptw; +struct vm_area_struct *vma; +unsigned long area; + + area =find_ptwrite_page_area (mm, vaddr); +if (IS_ERR_VALUE (area)) +return NULL ; + +mmap_assert_write_locked (mm); + + ptw =kzalloc_obj (*ptw); +if (!ptw) +return NULL ; +
arch/x86/kernel/uprobes.c static int copy_from_vaddr(struct mm_struct *mm, unsigned long vaddr, void *dst, · line 1529
Quoted 10 lines
arch/x86/kernel/uprobes.c +10
-0+ ptw->page =alloc_page (GFP_HIGHUSER | __GFP_ZERO); +if (!ptw->page) { +kfree (ptw); +return NULL ; + } + ptw->vaddr = area; + + vma =_install_special_mapping (mm, area,PAGE_SIZE , +VM_READ |VM_EXEC |VM_MAYEXEC |VM_MAYREAD |VM_IO , + &ptwrite_mapping);
[Severity: High]
Can child processes incorrectly execute the parent's ptwrite page because VM_WIPEONFORK or VM_DONTCOPY are missing here?
When a traced process forks, the child's VMA will receive PTEs pointing to the parent's physical ptwrite page. Later, arch_uprobe_dup_ptwrite() allocates a new page for the child's metadata, but the VMA still executes the parent's page. This can cause the child to execute invalid instructions or zeroes.
Quoted 7 lines
arch/x86/kernel/uprobes.c +7
-0+if (IS_ERR (vma)) { +__free_page (ptw->page); +kfree (ptw); +return NULL ; + } +return ptw; +}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
Re: [RFC v1 05/19] ptwrite uprobes: Add minimal low level support for x86
In reply to Andi KleenNAK.
This is broken as described in [0] and causes use-after-frees.
[0]:https://lore.kernel.org/all/aphJ7olxr-_VhDKt@gremlin/
Quoted 13 lines · Andi Kleen
Add more data structures and the x86 machinery to generate the PTWRITE instructions for a ptwrite uprobe. The probe executes PTWRITEs and then jumps back to the original code. In this variant only patching 5 byte nops is supported.
The instructions are pre-generated to templates and then patched when setting up the final user page.
The patching code uses 3 phase patching similar to
int3_update.The ptwrite stub emits a header with a magic value and the number of arguments, and then the actual probed values.
Quoted 8 lines
There is no separate config option for ptwrite uprobes, it is just tied to the main uprobes config.
Some limitations in the current implementation:
- The probed 5 byte area cannot cross a page.
- The allocated stubs in the user program are only freed on exit.Assisted-by omp:gpt-5.6-luna
I suggest looking into a model more suited to complex kernel development. Googling it, Luna is described thusly:
'GPT-5.6 Luna is OpenAI's fastest and most budget-friendly AI model tier, built specifically for high-volume, latency-sensitive tasks'
Which doesn't strike me as ideal for this kind of work, especially when you are submitting things to the mailing list and asking people to dedicate their own time (and better models) to assessing it.
Quoted 5 lines
Signed-off-by Andi Kleen <[email protected]>--- arch/x86/include/asm/uprobes.h | 24 ++ arch/x86/kernel/uprobes.c | 623 +++++++++++++++++++++++++++++++++ 2 files changed, 647 insertions(+)
(That's a huge diffstat for one patch and your cover letter is missing a full diffstat also...)
Quoted 6 lines
arch/x86/kernel/uprobes.c +0
-0@@ -15,11 +15,15 @@#include <linux/syscalls.h>
...
Quoted 17 lines
arch/x86/kernel/uprobes.c +17
-0+static struct uprobe_ptwrite_page * +create_uprobe_ptwrite_page (struct mm_struct *mm,unsigned long vaddr) +{ +struct uprobe_ptwrite_page *ptw; +struct vm_area_struct *vma; +unsigned long area; + + area =find_ptwrite_page_area (mm, vaddr); +if (IS_ERR_VALUE (area)) +return NULL ; + +mmap_assert_write_locked (mm); + + ptw =kzalloc_obj (*ptw); +if (!ptw) +return NULL ; +
arch/x86/kernel/uprobes.c static int copy_from_vaddr(struct mm_struct *mm, unsigned long vaddr, void *dst, · line 1529
Quoted 10 lines
arch/x86/kernel/uprobes.c +10
-0+ ptw->page =alloc_page (GFP_HIGHUSER | __GFP_ZERO); +if (!ptw->page) { +kfree (ptw); +return NULL ; + } + ptw->vaddr = area; + + vma =_install_special_mapping (mm, area,PAGE_SIZE , +VM_READ |VM_EXEC |VM_MAYEXEC |VM_MAYREAD |VM_IO , + &ptwrite_mapping);
You need to trampoline this (as the existing code does...) to avoid the issue described in [0].
--
Cheers, Lorenzo
RFC 6/19 ptwrite uprobes: Add a sample module to exercise interface
In reply to Andi KleenAdd a basic test module and a test program for uprobes ptwrite.
The module allows to configure and register a ptwrite uprobe in a executable.
This for testing and not intended as a production interface.
--- samples/Kconfig | 13 ++ samples/Makefile | 1 + samples/uprobe-ptwrite/Makefile | 13 ++ samples/uprobe-ptwrite/test_prog.c | 35 ++++ samples/uprobe-ptwrite/uprobe_ptwrite_test.c | 163 +++++++++++++++++++ 5 files changed, 225 insertions(+) create mode 100644 samples/uprobe-ptwrite/Makefile create mode 100644 samples/uprobe-ptwrite/test_prog.c create mode 100644 samples/uprobe-ptwrite/uprobe_ptwrite_test.c
samples/Kconfig +13 -0
@@ -322,6 +322,19 @@ config SAMPLE_HUNG_TASK Reading these files with multiple processes triggers hung task detection by holding locks for a long time (256 seconds). +config SAMPLE_UPROBE_PTWRITE + tristate "Build ptwrite uprobe test module -- loadable module only" + depends on UPROBES && X86_64 && m + help + Builds the minimal ptwrite uprobe prototype driver. It replaces + a 5-byte NOP at a user-specified file offset with a jmp to a per-mm + user-mode stub that emits the requested live registers / immediates + with the PTWRITE instruction into an externally configured Intel PT + stream (no kernel entry, no syscall, no single-step at probe-hit + time). The companion userspace target is test_prog.c in the same + directory; the module params (path/offset/args/event_id) select the + probe site and the emitted values. + source "samples/rust/Kconfig" source "samples/damon/Kconfig"
samples/Makefile +1 -0
@@ -44,4 +44,5 @@ obj-$(CONFIG_SAMPLE_DAMON_WSSE) += damon/ obj-$(CONFIG_SAMPLE_DAMON_PRCL) += damon/ obj-$(CONFIG_SAMPLE_DAMON_MTIER) += damon/ obj-$(CONFIG_SAMPLE_HUNG_TASK) += hung_task/ +obj-$(CONFIG_SAMPLE_UPROBE_PTWRITE) += uprobe-ptwrite/ obj-$(CONFIG_SAMPLE_TSM_MR) += tsm-mr/
samples/uprobe-ptwrite/Makefile new file +13 -0
@@ -0,0 +1,13 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +# ptwrite uprobe prototype sample. +# +# Module build (in-tree or via make M=samples/uprobe-ptwrite): +# obj-$(CONFIG_SAMPLE_UPROBE_PTWRITE) += uprobe_ptwrite_test.o +# +# The userspace test target (test_prog) is built by the out-of-tree test +# harness (see /home/oc/uprobe-ptwrite-test); the harness computes the file +# offset of target()'s 5-byte NOP and passes it to the module's offset +# parameter. + +obj-$(CONFIG_SAMPLE_UPROBE_PTWRITE) += uprobe_ptwrite_test.o
samples/uprobe-ptwrite/test_prog.c new file +35 -0
@@ -0,0 +1,35 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* test_prog - userspace target for the ptwrite uprobe test. +* +* target() carries a 5-byte NOP (nopl 0(%rax,%rax,1) = 0f 1f 44 00 00) at +* its entry. A probe installed at that file offset emits rdi/rsi (a, b) +* via PTWRITE. In the test loop a == i and b == i + 1. +*/ +#ifndef noinline +#define noinline __attribute__((noinline)) +#endif + +#include <stdio.h> + +static noinline unsigned long +target (unsigned long a,unsigned long b) +{ +/* 5-byte NOP: nopl 0(%rax,%rax,1) = 0f 1f 44 00 00. +* Emitted as raw bytes: the assembler would otherwise shrink +* "nopl (%rax,%rax,1)" to the 4-byte form (0f 1f 04 00), which +* the probe site validation rejects (jmp rel32 needs 5 bytes). +*/ + asmvolatile (".byte 0x0f, 0x1f, 0x44, 0x00, 0x00" ); +return a *31 + b; +} + +int main (void ) +{ +unsigned long i, acc =0 ; + +for (i =0 ; i <2000 ; i++) + acc +=target (i, i +1 ); +printf ("acc=%lu\n" , acc); +return 0 ; +}
samples/uprobe-ptwrite/uprobe_ptwrite_test.c new file +163 -0
163 lines not on this page · raw message
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [RFC v1 06/19] ptwrite uprobes: Add a sample module to exercise interface
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] Hardcoded local developer path in Makefile comment. --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Hardcoded local developer path in Makefile comment.
--
commit 6f339cbb02cf7327f16d076027733bac7f4b2900
Author: Andi Kleen <[email protected]>
ptwrite uprobes: Add a sample module to exercise interface
This commit adds a basic test module and a test program for uprobes ptwrite. It provides an interface to configure and register a ptwrite uprobe in an executable for testing purposes.
Quoted 6 lines
samples/uprobe-ptwrite/Makefile new file +0
-0@@ -0,0 +1,13 @@
[ ... ]
samples/uprobe-ptwrite/Makefile line 9
Quoted 3 lines
samples/uprobe-ptwrite/Makefile new file +3
-0+# +# The userspace test target (test_prog) is built by the out-of-tree test +# harness (see /home/oc/uprobe-ptwrite-test); the harness computes the file
[Severity: Low]
This isn't a bug, but is this local path (/home/oc/uprobe-ptwrite-test) intended to be included in the upstream kernel source? It appears to be a local developer directory rather than a path that would be accessible to other developers.
Quoted 4 lines
samples/uprobe-ptwrite/Makefile new file +4
-0+# offset of target()'s 5-byte NOP and passes it to the module's offset +# parameter. + +obj-$(CONFIG_SAMPLE_UPROBE_PTWRITE) += uprobe_ptwrite_test.o
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
RFC 7/19 ptwrite uprobes: Add support to tracing infrastructure
In reply to Andi KleenHook up the low level x86 ptwrite uprobes code to the generic trace uprobes events parser, so that the new probes can be set up. The interface is similar to classic probes, but there is new ptw: syntax and various restrictions.
Add minimal docs.
Architectures without the ptwrite backend are handled by weak stubs.
--- Documentation/trace/uprobetracer.rst | 61 +++++++- arch/x86/Kconfig | 1 + arch/x86/include/asm/uprobes.h | 12 +- include/linux/uprobes.h | 4 + kernel/trace/Kconfig | 6 + kernel/trace/trace_uprobe.c | 221 +++++++++++++++++++++++++-- 6 files changed, 288 insertions(+), 17 deletions(-)
Documentation/trace/uprobetracer.rst +59 -2
@@ -19,8 +19,8 @@ However unlike kprobe-event tracer, the uprobe event interface expects the user to calculate the offset of the probepoint in the object. You can also use /sys/kernel/tracing/dynamic_events instead of -uprobe_events. That interface will provide unified access to other -dynamic events too. +uprobe_events. That interface provides unified access to other event types +too. Synopsis of uprobe_tracer -------------------------
@@ -29,6 +29,7 @@ Synopsis of uprobe_tracer p[:[GRP/][EVENT]] PATH:OFFSET [FETCHARGS] : Set a uprobe r[:[GRP/][EVENT]] PATH:OFFSET [FETCHARGS] : Set a return uprobe (uretprobe) p[:[GRP/][EVENT]] PATH:OFFSET%return [FETCHARGS] : Set a return uprobe (uretprobe) + ptw[:[GRP/][EVENT]] PATH:OFFSET [FETCHARGS] : Set a trap-free ptwrite uprobe -:[GRP/][EVENT] : Clear uprobe or uretprobe event GRP : Group name. If omitted, "uprobes" is the default value.
@@ -76,6 +77,62 @@ offset, and container-size (usually 32). The syntax is:: For $comm, the default type is "string"; any other type is invalid. +ptwrite uprobes (ptw:) +-------------------------- + +See ptwrite-uprobes.rst for more details. + +A ``ptw:`` probe emits values into +an Intel Processor Trace data stream. It is faster +than standard uprobes, but has restrictions. +The Intel Processor Trace recording must be configured +separately. + +Requirements and restrictions: + +- Intel CPU with PTWRITE support (``/sys/devices/intel_pt/format/ptw`` must exist) +- Entry probes only: ``r:``/``%return`` and the SDT reference counter + ``(REF)`` are rejected. +- The probe site must be a 5-byte NOP or a punnable instruction (see + ptwrite-uprobes.rst). For five ``0x90`` bytes from GCC's + ``-fpatchable-function-entry=5``, append ``%multinop`` to the offset. +- FETCHARGS: register names (``%di``, ``%r8``, ...), ``$stack`` (the + stack pointer value) and immediates (``\IMM``), memory sources: + ``$stackN`` (the Nth stack slot, ``[%rsp + 8N]``) and ``+off(FETCHARG)`` + dereferences (e.g. ``+8(%di)`` = ``[%rdi + 8]``). ``u64`` sources use + ``ptwriteq``; ``u32``/``s32``/``x32`` sources use ``ptwritel`` and read + four bytes. Strings, bitfields, and indirect dereferences are not + supported. +- Memory sources execute a user-mode load in the stub. A bad base is + fixed up on the fault path to emit the fault word (0). + The probed task does not receive the fault, and the wire format + (``nargs`` words) is unchanged. uffd-managed pages are resolved by the + app's handler first and read their actual value; only a failed or + interrupted resolution degrades to the fault word. +- The event does not produce ring-buffer records. It provides a type registry + (``events/GRP/EVENT/format``) and the wire ``event_id`` + (``events/GRP/EVENT/id``) for an external decoder. Filters and perf + attach are rejected because filtering would need kernel entry. + +Wire format: each probe hit emits a 64-bit header word ``event_id<<48 | +nargs<<40 | 0x5054525731`` followed by ``nargs`` PTWRITE payloads. Perf +exposes each payload through its existing ``u64`` field. Decode with +``perf script`` and ``tools/perf/scripts/python/uprobe-ptwrite-decode.py``: + +:: + + perf record -e intel_pt/ptw=1,fup_on_ptw=1/u -o perf.data -- ./app + perf script --itrace=qwe -s uprobe-ptwrite-decode.py + +``fup_on_ptw=1`` increases the overhead, but also decoding reliability +because the exact IP of each probe is logged. + +Example:: + + echo 'ptw:t /bin/app:0x1234 %rdi %rsi \0x42' > /sys/kernel/tracing/uprobe_events + echo 1 > /sys/kernel/tracing/events/uprobes/t/enable + + Event Profiling --------------- You can check the total number of probe hits per event via
arch/x86/Kconfig +1 -0
@@ -422,6 +422,7 @@ config HAVE_INTEL_TXT config ARCH_SUPPORTS_UPROBES def_bool y + select UPROBE_EVENTS_PTWRITE if X86_64 && UPROBE_EVENTS config FIX_EARLYCON_MEM def_bool y
arch/x86/include/asm/uprobes.h +10 -2
@@ -28,6 +28,14 @@ enum {struct uprobe_xol_ops ; +/* +* Stub block array size. Worst case = 250 B (8 MEM args, rsp bases, fault +* table); 288 leaves 38 B slack. A file-scope static_assert in +* arch/x86/kernel/uprobes.c re-derives the worst case; prepare() also +* enforces it with -E2BIG at runtime. +*/ +#define UPROBE_PTWRITE_STUB_SIZE 288 +/* * ptwrite probe state. The stub template (code + data slots) is built * once at registration (mm-independent except the final jmp's rel32, patched
@@ -35,8 +43,8 @@ struct uprobe_xol_ops;* [ptwriteq hdr(%rip)] [arg emissions] [jmp probe+5] [u64 slots: header, imms] */ struct uprobe_ptwrite_arch { -u8 stub[256 ]; -u8 stub_len;/* code + data, whole block */ +u8 stub[UPROBE_PTWRITE_STUB_SIZE ]; +u16 stub_len;/* code + data + fault table, whole block */ u8 jmp_off;/* offset of the final jmp's rel32 field */ u8 ndata;/* number of u64 data slots */ u8 orig[MAX_UINSN_BYTES ];/* pristine file bytes, before generic analysis */
include/linux/uprobes.h +4 -0
@@ -205,8 +205,12 @@ struct uprobes_state {enum uprobe_ptwrite_src {UPROBE_PTW_SRC_REG ,/* value = live GPR (index in .reg) */ UPROBE_PTW_SRC_IMM ,/* value = constant (.val), stored in stub data slot */ +UPROBE_PTW_SRC_MEM ,/* value = [.reg + disp32] (v2); requires ALLOW_MEM */ }; +/* uprobe_ptwrite_desc.flags */ +#define UPROBE_PTWRITE_FL_ALLOW_MEM BIT(0)/* SRC_MEM args enabled */ +struct uprobe_ptwrite_arg {u8 src;/* enum uprobe_ptwrite_src */ u8 reg;/* x86-64 GPR index (0=rax..15=r15) for SRC_REG */
kernel/trace/Kconfig +6 -0
@@ -848,6 +848,12 @@ config UPROBE_EVENTS This option is required if you plan to use perf-probe subcommand of perf tools on user space applications. +config UPROBE_EVENTS_PTWRITE + bool + depends on UPROBE_EVENTS + help + ptwrite uprobes ("ptw:" tracefs event type). + config EPROBE_EVENTS bool "Enable event-based dynamic events" depends on TRACING
kernel/trace/trace_uprobe.c +208 -13
342 lines not on this page · raw message
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [RFC v1 07/19] ptwrite uprobes: Add support to tracing infrastructure
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
Hook up the low level x86 ptwrite uprobes code to the generic trace uprobes events parser, so that the new probes can be set up. The interface is similar to classic probes, but there is new ptw: syntax and various restrictions.
Add minimal docs.
Architectures without the ptwrite backend are handled by weak stubs.
Assisted-by omp:gpt-5.6-lunaSigned-off-by Andi Kleen <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
RFC 8/19 ptwrite uprobes / x86: Add a user fault notifier chain
In reply to Andi KleenThe ptwrite uprobes need to catch faults in the user probes, otherwise a bad probe could crash the program. For classic probes that is handled in the kernel, but with these new kinds of probes the crash happens in ring 3 code.
The existing die chain cannot be used for this because it only handles kernel level faults. Add a new user fault notifier chain that is supported for #GP, #PF, #SS. It is only called before a signal would be delivered, so it doesn't slow down any hot paths. The fault handler can then handle the fault and prevent the signal.
Add register code and the hooks for the chain.
Some of the existing fault hardware workarounds could be converted to this in the future (not done yet)
--- arch/x86/include/asm/traps.h | 17 +++++++++++++++++ arch/x86/kernel/traps.c | 32 ++++++++++++++++++++++++++++++++ arch/x86/mm/fault.c | 6 ++++++ 3 files changed, 55 insertions(+)
arch/x86/include/asm/traps.h +17 -0
@@ -4,6 +4,7 @@#include <linux/context_tracking_state.h> #include <linux/kprobes.h> +#include <linux/notifier.h> #include <asm/debugreg.h> #include <asm/idtentry.h>
@@ -59,4 +60,20 @@ static inline void cond_local_irq_disable(struct pt_regs *regs) local_irq_disable(); } +/* +* User-mode fault notifier chain, called before a user exception is +* about to become a signal. NOTIFY_STOP consumes the fault. +*/ +struct x86_user_fault_args { +struct pt_regs *regs; +unsigned long error_code; +unsigned long address;/* #PF: faulting address */ +unsigned int trap; +}; + +extern int register_x86_user_fault_notifier (struct notifier_block *nb); +extern void unregister_x86_user_fault_notifier (struct notifier_block *nb); +extern int notify_x86_user_fault (struct pt_regs *regs,unsigned long error_code, +unsigned long address,unsigned int trap); +#endif /* _ASM_X86_TRAPS_H */
arch/x86/kernel/traps.c +32 -0
@@ -517,6 +517,11 @@ DEFINE_IDTENTRY_ERRORCODE(exc_segment_not_present)DEFINE_IDTENTRY_ERRORCODE (exc_stack_segment) { +if (user_mode (regs) && +notify_x86_user_fault (regs, error_code,0 ,X86_TRAP_SS ) == +NOTIFY_STOP ) +return ; +do_error_trap (regs, error_code,"stack segment" ,X86_TRAP_SS ,SIGBUS ,0 ,NULL ); }
@@ -911,6 +916,30 @@ static void gp_user_force_sig_segv(struct pt_regs *regs, int trapnr,force_sig (SIGSEGV ); } +static ATOMIC_NOTIFIER_HEAD (x86_user_fault_chain); + +int register_x86_user_fault_notifier (struct notifier_block *nb) +{ +return atomic_notifier_chain_register (&x86_user_fault_chain, nb); +} +void unregister_x86_user_fault_notifier (struct notifier_block *nb) +{ +atomic_notifier_chain_unregister (&x86_user_fault_chain, nb); +} + +int notify_x86_user_fault (struct pt_regs *regs,unsigned long error_code, +unsigned long address,unsigned int trap) +{ +struct x86_user_fault_args args = { + .regs = regs, + .error_code = error_code, + .address = address, + .trap = trap, + }; + +return atomic_notifier_call_chain (&x86_user_fault_chain,0 , &args); +} +DEFINE_IDTENTRY_ERRORCODE (exc_general_protection) {char desc[sizeof (GPFSTR ) +50 +2 *sizeof (unsigned long ) +1 ] =GPFSTR ;
@@ -942,6 +971,9 @@ DEFINE_IDTENTRY_ERRORCODE(exc_general_protection)if (emulate_vsyscall_gp (regs)) goto exit; +if (notify_x86_user_fault (regs, error_code,0 ,X86_TRAP_GP ) ==NOTIFY_STOP ) + goto exit; +gp_user_force_sig_segv (regs,X86_TRAP_GP , error_code, desc); goto exit; }
arch/x86/mm/fault.c +6 -0
@@ -821,6 +821,9 @@ __bad_area_nosemaphore(struct pt_regs *regs, unsigned long error_code,if (fixup_vdso_exception (regs,X86_TRAP_PF , error_code, address))return ; +if (notify_x86_user_fault (regs, error_code, address,X86_TRAP_PF ) ==NOTIFY_STOP ) +return ; +if (likely (show_unhandled_signals))show_signal_msg (regs, error_code, address, tsk);
@@ -950,6 +953,9 @@ do_sigbus(struct pt_regs *regs, unsigned long error_code, unsigned long address, return; }#endif +if (notify_x86_user_fault (regs, error_code, address,X86_TRAP_PF ) ==NOTIFY_STOP ) +return ; +force_sig_fault (SIGBUS ,BUS_ADRERR , (void __user *)address); }
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [RFC v1 08/19] ptwrite uprobes / x86: Add a user fault notifier chain
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
The ptwrite uprobes need to catch faults in the user probes, otherwise a bad probe could crash the program. For classic probes that is handled in the kernel, but with these new kinds of probes the crash happens in ring 3 code.
The existing die chain cannot be used for this because it only handles kernel level faults. Add a new user fault notifier chain that is supported for #GP, #PF, #SS. It is only called before a signal would be delivered, so it doesn't slow down any hot paths. The fault handler can then handle the fault and prevent the signal.
Quoted 7 lines
Add register code and the hooks for the chain.
Some of the existing fault hardware workarounds could be converted to this in the future (not done yet)
Assisted-by omp:gpt-5.6-lunaSigned-off-by Andi Kleen <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
RFC 9/19 ptwrite uprobes: Factor file-backed instruction reads
In reply to Andi KleenRefactor copy_insn into a more generic uprobe_copy_from_file. The existing copy_insn is still there, but uses the generic version now. The generic version will be used in later patches.
No semantic change intended.
--- include/linux/uprobes.h | 2 ++ kernel/events/uprobes.c | 56 ++++++++++++++++++++++++++++------------- 2 files changed, 41 insertions(+), 17 deletions(-)
include/linux/uprobes.h +2 -0
@@ -304,6 +304,8 @@ extern void uprobe_handle_trampoline(struct pt_regs *regs);extern void *arch_uretprobe_trampoline (unsigned long *psize);extern unsigned long uprobe_get_trampoline_vaddr (void );extern void uprobe_copy_from_page (struct page *page,unsigned long vaddr,void *dst,int len); +extern int uprobe_copy_from_file (struct inode *inode,struct file *file, +loff_t offset,void *buf,int size);extern void arch_uprobe_clear_state (struct mm_struct *mm);extern void arch_uprobe_init_state (struct mm_struct *mm);extern int arch_uprobe_dup_ptwrite (struct mm_struct *oldmm,struct mm_struct *newmm);
kernel/events/uprobes.c +39 -17
69 lines not on this page · raw message
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [RFC v1 09/19] ptwrite uprobes: Factor file-backed instruction reads
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] copy_insn() unconditionally returns -EIO for partial reads, breaking uprobes placed near the end…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] copy_insn() unconditionally returns -EIO for partial reads, breaking uprobes placed near the end of an executable. --
commit 357913ae591e42b1fd264a2ff741868c174c2724
Author: Andi Kleen <[email protected]>
ptwrite uprobes: Factor file-backed instruction reads
This commit refactors copy_insn() into a more generic uprobe_copy_from_file() function. The existing copy_insn() is updated to use this generic version, which is intended to be used by later patches.
Quoted 4 lines
kernel/events/uprobes.c +0
-0
[ ... ]
Quoted 3 lines
kernel/events/uprobes.c +2
-0+int uprobe_copy_from_file (struct inode *inode,struct file *file, +loff_t offset,void *buf,int size) {
[ ... ]
Quoted 8 lines
kernel/events/uprobes.c +4
-2+while (copied < size) { + file_size =i_size_read (inode); +if (offset >= file_size)break ; - len =min_t (int, size,PAGE_SIZE - (offs & ~PAGE_MASK )); - err =__copy_insn (mapping, filp, insn, len, offs); + len =min_t (loff_t, size - copied, file_size - offset);
[ ... ]
kernel/events/uprobes.c static int __copy_insn(struct address_space *mapping, struct file *filp, · line 1121
Quoted 8 lines
kernel/events/uprobes.c +8
-0+static int copy_insn (struct uprobe *uprobe,struct file *filp) +{ +int ret; + + ret =uprobe_copy_from_file (uprobe->inode, filp, uprobe->offset, + &uprobe->arch.insn, +sizeof (uprobe->arch.insn)); +return ret <0 ? ret : ret ==sizeof (uprobe->arch.insn) ?0 : -EIO ;
[Severity: High]
Does this unconditionally return -EIO for partial reads near the end of the file?
When registering a uprobe on a target file where the instruction offset is close to the end of the file (where file_size - offset is smaller than sizeof(uprobe->arch.insn)), uprobe_copy_from_file() will stop at the file boundary and return a byte count strictly less than the requested size.
The original copy_insn() loop stopped reading at EOF but still returned 0 as long as no explicit error occurred, allowing partial reads to succeed.
By enforcing a strict ret == sizeof(...) check here, will this fail uprobe registration for valid instructions that happen to reside near the end of small binaries or scripts?
Quoted 1 line
}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
RFC 10/19 ptwrite uprobes: Minimal memory references and fault handling
In reply to Andi KleenAdd support for memory references. Currently this is only simple cases, no indirect memory references or strings, that would require saving/restoring registers. Only 8 and 4 byte memory references are supported.
This requires fault handling using the fault notifier hook added earlier. When a fault happens inside a probe return to a specially generated tail that logs zeroes.
It increases the in memory overhead of uprobe_ptwrite_page somewhat (to roughly a KB). I tried to find alternatives, but they all complicated the code, and 1KB should be still acceptable.
--- arch/x86/include/asm/uprobes.h | 14 + arch/x86/kernel/uprobes.c | 308 +++++++++++++++++-- include/linux/uprobes.h | 4 +- samples/uprobe-ptwrite/uprobe_ptwrite_test.c | 41 ++- 4 files changed, 344 insertions(+), 23 deletions(-)
arch/x86/include/asm/uprobes.h +14 -0
@@ -48,6 +48,8 @@ struct uprobe_ptwrite_arch {u8 jmp_off;/* offset of the final jmp's rel32 field */ u8 ndata;/* number of u64 data slots */ u8 orig[MAX_UINSN_BYTES ];/* pristine file bytes, before generic analysis */ +u16 ft_off;/* fault table offset within the block (0 if none) */ +u8 nft;/* number of fault entries */ };/* Per-mm page holding generated ptwrite stub blocks (mirrors trampolines). */
@@ -56,6 +58,18 @@ struct uprobe_ptwrite_page {struct page *page;/* stub blocks written via kmap */ unsigned long vaddr;/* mapping base */ u16 cursor;/* next free block offset */ +u16 nblocks; +struct { +u16 off;/* block offset in the page */ +u16 len;/* generated block length */ +u16 ft_off;/* fault table offset within the block */ +u8 orig0;/* original site byte 0 (pun restore) */ +u8 pun;/* instruction-pun mechanism (single-byte poke) */ +u8 site_len;/* original instruction length (pun identity) */ +s32 site_off;/* probe site - page base (idempotent reinstall) */ +u8 site_insn[MAX_UINSN_BYTES ];/* original bytes (pun identity) */ + } index[PAGE_SIZE /32 ];/* exact: min block = 32 B (nargs >= 1), */ +/* so <= 128 blocks fit a page */ };struct arch_uprobe {
arch/x86/kernel/uprobes.c +288 -20
442 lines not on this page · raw message
include/linux/uprobes.h +2 -2
@@ -213,10 +213,10 @@ enum uprobe_ptwrite_src {struct uprobe_ptwrite_arg {u8 src;/* enum uprobe_ptwrite_src */ -u8 reg;/* x86-64 GPR index (0=rax..15=r15) for SRC_REG */ +u8 reg;/* x86-64 GPR index (0=rax..15=r15) for SRC_REG/SRC_MEM */ u8 size;/* declared type size 1/2/4/8 (decoder hint) */ u8 reserved; -u64 val;/* SRC_IMM: constant; SRC_REG: unused */ +u64 val;/* SRC_IMM: constant, SRC_MEM: disp32 (low 32 bits) */ };struct uprobe_ptwrite_desc {
samples/uprobe-ptwrite/uprobe_ptwrite_test.c +40 -1
55 lines not on this page · raw message
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [RFC v1 10/19] ptwrite uprobes: Minimal memory references and fault handling
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] Premature publication of `ptw->nblocks` without memory barriers exposes uninitialized memory to…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] Premature publication of ptw->nblocks without memory barriers exposes uninitialized memory to concurrent page fault handlers.
- [Medium] Rolling back ptw->nblocks on error allows in-place reuse of array slots without an RCU grace period, causing torn reads in concurrent fault handlers. --
commit 925690411c45029e0f0a323ac69035b5e1cc8c87
Author: Andi Kleen <[email protected]>
ptwrite uprobes: Minimal memory references and fault handling
Add support for memory references. Currently this is only simple cases, no indirect memory references or strings. This requires fault handling using the fault notifier hook added earlier.
Quoted 4 lines
arch/x86/kernel/uprobes.c +0
-0
[ ... ]
Quoted 15 lines
arch/x86/kernel/uprobes.c +4
-2@@ -1689,13 +1852,22 @@ int arch_uprobe_install_ptwrite(struct arch_uprobe *auprobe,return -ENOMEM ; block_off = ptw->cursor; -if (block_off >PAGE_SIZE || - ptw_a->stub_len >PAGE_SIZE - block_off) +if (block_off >PAGE_SIZE || ptw_a->stub_len >PAGE_SIZE - block_off) +return -ENOMEM ; +if (ptw->nblocks >=ARRAY_SIZE (ptw->index))return -ENOMEM ; stub_addr = ptw->vaddr + block_off;if (!ptwrite_rel32 (stub_addr + ptw_a->jmp_off +4 , vaddr +5 , &rel))return -ERANGE ; + site_delta = (s64 )vaddr - (s64 )ptw->vaddr;
arch/x86/kernel/uprobes.c int arch_uprobe_install_ptwrite(struct arch_uprobe *auprobe, · line 1873
Quoted 10 lines
arch/x86/kernel/uprobes.c +7
-0+if (site_delta <INT_MIN || site_delta >INT_MAX ) +return -ERANGE ; + + ptw->index[ptw->nblocks].off = block_off; + ptw->index[ptw->nblocks].len = ptw_a->stub_len; + ptw->index[ptw->nblocks].ft_off = ptw_a->ft_off; + ptw->nblocks++; kaddr =kmap_local_page (ptw->page);memcpy (kaddr + block_off, ptw_a->stub, ptw_a->stub_len);
[Severity: Medium]
Does this code expose uninitialized memory to concurrent page fault handlers?
Since ptw->nblocks is incremented before memcpy() initializes the stub block memory, a concurrent lockless reader in uprobe_ptwrite_handle_fault() might observe the incremented nblocks and read the uninitialized fault table while memcpy() is actively writing to it.
arch/x86/kernel/uprobes.c int arch_uprobe_install_ptwrite(struct arch_uprobe *auprobe, · line 1883
Quoted 12 lines
arch/x86/kernel/uprobes.c +5
-3@@ -1704,9 +1876,13 @@ int arch_uprobe_install_ptwrite(struct arch_uprobe *auprobe,kunmap_local (kaddr); ret =ptwrite_text_poke (auprobe, vma, vaddr, stub_addr); -if (!ret) - ptw->cursor = block_off + ptw_a->stub_len; -return ret; +if (ret) +/* Publish rollback before readers use the reduced block count. */ +smp_store_release (&ptw->nblocks, ptw->nblocks -1 ); +return ret; + }
[Severity: Medium]
Can this rollback of ptw->nblocks cause use-after-free or torn reads for concurrent readers?
If ptwrite_text_poke() fails, ptw->nblocks is rolled back. However, a lockless reader executing uprobe_ptwrite_handle_fault() under rcu_read_lock() might have already read the old ptw->nblocks and be actively iterating over this slot. A subsequent probe installation could then immediately reuse this array slot and block_off memory without waiting for a synchronize_rcu() grace period, causing the lockless reader to observe torn data.
Quoted 3 lines
arch/x86/kernel/uprobes.c +2
-0+ ptw->cursor = block_off + ptw_a->stub_len; +return 0 ; }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
RFC 11/19 ptwrite uprobes: Add multinop support
In reply to Andi KleenGCC's -fpatchable-function-entry=5 may emit five one-byte NOPs. Normally that's not safe to patch because some might jump into a later nop. But for the gcc case it's safe because nobody jumps into the nops. Add a %multinop that allows the user opting into patching these sites. This way patching for the gcc instrumentation works.
--- arch/x86/include/asm/uprobes.h | 1 + arch/x86/kernel/uprobes.c | 4 +- include/linux/uprobes.h | 1 + kernel/trace/trace_uprobe.c | 42 ++++++++++++++------ samples/uprobe-ptwrite/uprobe_ptwrite_test.c | 6 +++ 5 files changed, 40 insertions(+), 14 deletions(-)
arch/x86/include/asm/uprobes.h +1 -0
@@ -50,6 +50,7 @@ struct uprobe_ptwrite_arch {u8 orig[MAX_UINSN_BYTES ];/* pristine file bytes, before generic analysis */ u16 ft_off;/* fault table offset within the block (0 if none) */ u8 nft;/* number of fault entries */ +u8 allow_nop_run;/* accept a five-byte run of 0x90 */ };/* Per-mm page holding generated ptwrite stub blocks (mirrors trampolines). */
arch/x86/kernel/uprobes.c +3 -1
@@ -1401,7 +1401,8 @@ int arch_uprobe_ptwrite_prepare(struct arch_uprobe *auprobe,return -EINVAL ;if (desc->nargs >UPROBE_PTWRITE_MAX_ARGS )return -E2BIG ; -if (desc->flags & ~UPROBE_PTWRITE_FL_ALLOW_MEM ) +if (desc->flags & ~(UPROBE_PTWRITE_FL_ALLOW_MEM | +UPROBE_PTWRITE_FL_ALLOW_NOP_RUN ))return -EINVAL ;/* The generic registration path copied these bytes before this hook. */
@@ -1567,6 +1568,7 @@ int arch_uprobe_ptwrite_prepare(struct arch_uprobe *auprobe, ptw->ndata =1 + n_imm + (n_mem ?1 :0 ); ptw->ft_off = n_mem ? ft_off :0 ; ptw->nft = n_mem; + ptw->allow_nop_run = !!(desc->flags &UPROBE_PTWRITE_FL_ALLOW_NOP_RUN );return 0 ; }#undef PTW_NEED
include/linux/uprobes.h +1 -0
@@ -210,6 +210,7 @@ enum uprobe_ptwrite_src {/* uprobe_ptwrite_desc.flags */ #define UPROBE_PTWRITE_FL_ALLOW_MEM BIT(0)/* SRC_MEM args enabled */ +#define UPROBE_PTWRITE_FL_ALLOW_NOP_RUN BIT(2)/* accept five 1-byte NOPs */ struct uprobe_ptwrite_arg {u8 src;/* enum uprobe_ptwrite_src */
kernel/trace/trace_uprobe.c +29 -13
95 lines not on this page · raw message
samples/uprobe-ptwrite/uprobe_ptwrite_test.c +6 -0
@@ -17,6 +17,7 @@* m<N>[:<disp>][:<size>] = memory arg [reg + disp32], * size 4 (u32 load) or 8 (u64 load, default) * event_id=0x1234 identifier carried in the PTW header word +* allow_nop_run=1 accept five one-byte NOPs at the site */ #include <linux/module.h> #include <linux/uprobes.h>
@@ -35,6 +36,10 @@ static ushort event_id = 0x1234;module_param (event_id, ushort,0444 );MODULE_PARM_DESC (event_id,"event id carried in the PTW header word" ); +static bool allow_nop_run; +module_param (allow_nop_run, bool,0444 ); +MODULE_PARM_DESC (allow_nop_run,"accept five one-byte NOPs at the site" ); +static char *args ="r0" ;module_param (args, charp,0444 );MODULE_PARM_DESC (args,"comma-separated args: r<N> GPR, i<hex> immediate, m<N>[:disp][:4|8] memory" );
@@ -151,6 +156,7 @@ static int __init uprobe_ptwrite_test_init(void)int ret; desc.event_id = event_id; + desc.flags = allow_nop_run ?UPROBE_PTWRITE_FL_ALLOW_NOP_RUN :0 ; ret =parse_probe_args ();if (ret)return ret;
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [RFC v1 11/19] ptwrite uprobes: Add multinop support
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 8 lines
GCC's -fpatchable-function-entry=5 may emit five one-byte NOPs. Normally that's not safe to patch because some might jump into a later nop. But for the gcc case it's safe because nobody jumps into the nops. Add a %multinop that allows the user opting into patching these sites. This way patching for the gcc instrumentation works.
Assisted-by omp:gpt-5.6-lunaSigned-off-by Andi Kleen <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
RFC 12/19 ptwrite uprobes: Add pacing to the probes
In reply to Andi KleenWhen PTWRITEs are too tightly spaced they can lose data. While the decoder makes some effort to recover from this it can be still annoying for the user. This patch adds LFENCEs between the individual instructions to mimimize (mostly avoid) this problem. It can still happen with parallel branch collection or if a high frequency of timing packets are configured. The extend also depends on the core-type.
It can be still disabled with %nopace. This is useful when the user knows the probes are not too tightly spaced. It is usually still needed with many arguments.
This patch generates LFENCEs for the probes unless disabled.
Compared with %nopace, default pacing increases probe cost by roughly 272% with PT off, 227% with full tracing, and 48% in snapshot mode on Alder Lake.
--- arch/x86/include/asm/uprobes.h | 21 ++++++++--- arch/x86/kernel/uprobes.c | 64 +++++++++++++++++++++++++++++----- include/linux/uprobes.h | 1 + kernel/trace/trace_uprobe.c | 22 ++++++++++++ 4 files changed, 94 insertions(+), 14 deletions(-)
arch/x86/include/asm/uprobes.h +16 -5
@@ -29,12 +29,23 @@ enum {struct uprobe_xol_ops ;/* -* Stub block array size. Worst case = 250 B (8 MEM args, rsp bases, fault -* table); 288 leaves 38 B slack. A file-scope static_assert in -* arch/x86/kernel/uprobes.c re-derives the worst case; prepare() also -* enforces it with -E2BIG at runtime. +* Stub block size. Conservative worst case is 298 bytes: 9-byte header, +* one lead fence, eight 21-byte memory forms with one 3-byte fence each, +* a 16-byte original-instruction copy, a 5-byte return jump, alignment, +* and 66 bytes of data/fault metadata. 384 leaves room. A static_assert in +* arch/x86/kernel/uprobes.c checks the bound; prepare() also checks it with +* -E2BIG. */ -#define UPROBE_PTWRITE_STUB_SIZE 288 +#define UPROBE_PTWRITE_STUB_SIZE 384 + +/* the out-of-line original-instruction copy slot (x86 max insn length) */ +#define UPROBE_PTWRITE_COPY_SIZE MAX_UINSN_BYTES + +/* +* Word pacing: insert this many LFENCEs between emitted ptwrite words and +* before the first word, unless UPROBE_PTWRITE_FL_NO_LEAD_PACE is requested. +*/ +#define UPROBE_PTWRITE_SERIALIZE_LFENCES 1/* LFENCEs per word gap */ /* * ptwrite probe state. The stub template (code + data slots) is built
arch/x86/kernel/uprobes.c +55 -9
106 lines not on this page · raw message
include/linux/uprobes.h +1 -0
@@ -210,6 +210,7 @@ enum uprobe_ptwrite_src {/* uprobe_ptwrite_desc.flags */ #define UPROBE_PTWRITE_FL_ALLOW_MEM BIT(0)/* SRC_MEM args enabled */ +#define UPROBE_PTWRITE_FL_NO_LEAD_PACE BIT(1)/* don't slow down probes */ #define UPROBE_PTWRITE_FL_ALLOW_NOP_RUN BIT(2)/* accept five 1-byte NOPs */ struct uprobe_ptwrite_arg {
kernel/trace/trace_uprobe.c +22 -0
@@ -645,6 +645,7 @@ static int __trace_uprobe_create(int argc, const char **argv)enum probe_print_type ptype;bool is_return = false;bool is_ptwrite = false; +bool is_nopace = false;bool is_nop_run = false;int i, ret, arg_start =2 ;
@@ -732,6 +733,11 @@ static int __trace_uprobe_create(int argc, const char **argv)/* Check if there is %return suffix */ tmp =strchr (arg,'%' ); +if (tmp && is_ptwrite && !strcmp (tmp,"%nopace" )) { + *tmp ='\0' ; + is_nopace = true; + tmp =NULL ; + }if (tmp && is_ptwrite) {if (!strcmp (tmp,"%multinop" )) { *tmp ='\0' ;
@@ -756,11 +762,20 @@ static int __trace_uprobe_create(int argc, const char **argv)trace_probe_log_err (arg - filename,BAD_UPROBE_OFFS );return ret; } +if (is_ptwrite && arg_start < argc && + !strcmp (argv[arg_start],"%nopace" )) { + is_nopace = true; + arg_start++; + }if (is_ptwrite) {while (arg_start < argc && !strcmp (argv[arg_start],"%multinop" )) { is_nop_run = true; arg_start++; } +if (arg_start < argc && !strcmp (argv[arg_start],"%nopace" )) { + is_nopace = true; + arg_start++; + } } if (argc - arg_start > MAX_TRACE_ARGS ||
@@ -844,6 +859,8 @@ static int __trace_uprobe_create(int argc, const char **argv) tu->ptwrite_desc.nargs = argc; tu->ptwrite_desc.flags = is_nop_run ?UPROBE_PTWRITE_FL_ALLOW_NOP_RUN :0 ; +if (is_nopace) + tu->ptwrite_desc.flags |=UPROBE_PTWRITE_FL_NO_LEAD_PACE ;for (i =0 ; i < argc; i++) { ret =ptwrite_compile_arg (tu, i);if (ret) {
@@ -902,6 +919,11 @@ static int trace_uprobe_show(struct seq_file *m, struct dyn_event *ev)trace_probe_group_name (&tu->tp),trace_probe_name (&tu->tp), tu->filename, (int)(sizeof(void *) * 2), tu->offset); +if (tu->ptwrite_desc.flags &UPROBE_PTWRITE_FL_NO_LEAD_PACE ) { +seq_puts (m,"%nopace" ); +if (tu->ptwrite_desc.flags &UPROBE_PTWRITE_FL_ALLOW_NOP_RUN ) +seq_putc (m,' ' ); + }if (tu->ptwrite_desc.flags &UPROBE_PTWRITE_FL_ALLOW_NOP_RUN )seq_puts (m,"%multinop" ); }else
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [RFC v1 12/19] ptwrite uprobes: Add pacing to the probes
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
When PTWRITEs are too tightly spaced they can lose data. While the decoder makes some effort to recover from this it can be still annoying for the user. This patch adds LFENCEs between the individual instructions to mimimize (mostly avoid) this problem. It can still happen with parallel branch collection or if a high frequency of timing packets are configured. The extend also depends on the core-type.
It can be still disabled with %nopace. This is useful when the user knows the probes are not too tightly spaced. It is usually still needed with many arguments.
Quoted 7 lines
This patch generates LFENCEs for the probes unless disabled.
Compared with %nopace, default pacing increases probe cost by roughly 272% with PT off, 227% with full tracing, and 48% in snapshot mode on Alder Lake.
Assisted-by omp:gpt-5.6-lunaSigned-off-by Andi Kleen <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
RFC 13/19 ptwrite uprobes: Support instruction puning
In reply to Andi KleenThe previous ptwrite instrumentation only worked on 5 byte+ nops because it needs to patch in a 5 byte branch. But the instruction may be shorter and the kernel cannot prove that nobody jumps to the next instruction.
However that is somewhat limiting because it means most code cannot be probed. This patch implements a simplified variant of the instruction puning technique from Chamith et.al. "Instruction Punning: Lightweight instrumentation for x86-64". The basic idea is to only patch in the one byte opcode for the branch and reuse the existing instruction bytes in the code as the branch target.
If someone branches to the remaining bytes they are still executed in the original way because they didn't change.
This requires placing a target trampoline page at the right address. If the area is not available or points to kernel space it doesn't work.
In general it is somewhat unreliable for non PIE executables because the target is often negative and ends up in kernel space. However on modern Linux distributions near all binaries are PIE and high up in the address space with ample gaps around them, which makes puning have a high success rate.
For example trying to probe every instruction in a PIE linked Debian 13 /bin/bash:
bash has around 210k instructions, of which around 10k were directly patchable 5 byte nops. Running it for 100 times with ASLR 76.4% of the instructions were always pun probeable, with 23% of the instruction never being punnable (and the rest sometimes). So punning is not perfect, but still works most of the time, and is much better than just nops.
The original paper used various fallback techniques to improve success, but I found things work well enough anyways with PIE and PIC.
If the probing fails it's possible for the harness to move the probe around until it finds a better target. Sometimes you're just lucky on rerun with ASLR. Or alternatively just use a classic uprobe.
I made sure to give all instruction error cases unique errnos so that the harness can make informed decisions.
This patch adds the low level machinery for puning. Classify the instruction, poke the target and map trampolines to the right place. One difference to the nop probing is that the previous instruction needs to be copied and fixed up (analogous to classic uprobes)
32bit relative ranches cannot be probed because their target is not available by definition. For 8bit relative branches there are also cases where it might need a second trampoline, or APX JMPABS support, to reach the true target, so these are rejected too for now.
--- arch/x86/include/asm/uprobes.h | 11 +- arch/x86/kernel/uprobes.c | 480 ++++++++++++++++++++++++++++++--- include/linux/uprobes.h | 4 +- kernel/events/uprobes.c | 18 +- 4 files changed, 470 insertions(+), 43 deletions(-)
arch/x86/include/asm/uprobes.h +8 -3
@@ -49,14 +49,19 @@ struct uprobe_xol_ops;/* * ptwrite probe state. The stub template (code + data slots) is built -* once at registration (mm-independent except the final jmp's rel32, patched -* per-mm at install). Block layout: -* [ptwriteq hdr(%rip)] [arg emissions] [jmp probe+5] [u64 slots: header, imms] +* once at registration. Only the final jmp's rel32 and the copy's +* disp/rel fields are patched per-mm at install. Block layout: +* [ptwriteq hdr(%rip)] [arg emissions] [orig-insn copy] +* [jmp probe+len] [u64 slots: header, imms] */ struct uprobe_ptwrite_arch {u8 stub[UPROBE_PTWRITE_STUB_SIZE ];u16 stub_len;/* code + data + fault table, whole block */ u8 jmp_off;/* offset of the final jmp's rel32 field */ +u8 copy_off;/* offset of the out-of-line instruction copy */ +u8 len;/* copy length (0 = drop); back-jmp = vaddr+len */ +u8 disp_off;/* rip-relative disp32 offset in the copy (0 = none) */ +s32 disp;/* original disp32 (delta-patched per-mm) */ u8 ndata;/* number of u64 data slots */ u8 orig[MAX_UINSN_BYTES ];/* pristine file bytes, before generic analysis */ u16 ft_off;/* fault table offset within the block (0 if none) */
arch/x86/kernel/uprobes.c +446 -34
592 lines not on this page · raw message
include/linux/uprobes.h +3 -1
@@ -250,7 +250,9 @@ extern struct uprobe *uprobe_register_ptwrite(struct inode *inode, const struct uprobe_ptwrite_desc *desc);extern bool arch_uprobe_ptwrite_supported (void );extern int arch_uprobe_ptwrite_prepare (struct arch_uprobe *auprobe, -const struct uprobe_ptwrite_desc *desc); +struct inode *inode,struct file *file, +loff_t offset, +const struct uprobe_ptwrite_desc *desc);extern int arch_uprobe_install_ptwrite (struct arch_uprobe *auprobe,struct vm_area_struct *vma,unsigned long vaddr);
kernel/events/uprobes.c +13 -5
@@ -1515,6 +1515,8 @@ bool __weak arch_uprobe_ptwrite_supported(void) }int __weak arch_uprobe_ptwrite_prepare (struct arch_uprobe *auprobe, +struct inode *inode,struct file *file, +loff_t offset,const struct uprobe_ptwrite_desc *desc) {return -EOPNOTSUPP ;
@@ -1595,13 +1597,19 @@ struct uprobe *uprobe_register_ptwrite(struct inode *inode, struct file *file, ret = -EBUSY ; goto out; } - -/* Build the mm-independent stub template once, at registration. */ - ret =arch_uprobe_ptwrite_prepare (&uprobe->arch, desc); +/* +* Prepare the immutable PTWRITE stub before exposing the uprobe. The +* copy-instruction flag also keeps the normal XOL preparation path out. +*/ + ret =copy_insn (uprobe, file);if (ret) goto out; - - + ret =arch_uprobe_ptwrite_prepare (&uprobe->arch, inode, file, offset, + desc); +if (ret) + goto out; +smp_wmb (); +set_bit (UPROBE_COPY_INSN , &uprobe->flags);set_bit (UPROBE_PTWRITE , &uprobe->flags);consumer_add (uprobe, uc); ret =register_for_each_vma (uprobe, uc);
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [RFC v1 13/19] ptwrite uprobes: Support instruction puning
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] Stack out-of-bounds read in `pun_install` when copying the original instruction. - [High]…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Stack out-of-bounds read in pun_install when copying the original instruction.
- [High] Uninstalling a punned uprobe fails and permanently corrupts process memory if the user modifies the instruction's subsequent bytes. --
commit bfefc3cb6738d81a0fc8093e66776a8584e4deed
Author: Andi Kleen <[email protected]>
ptwrite uprobes: Support instruction puning
The previous ptwrite instrumentation only worked on 5 byte+ nops because it needs to patch in a 5 byte branch. But the instruction may be shorter and the kernel cannot prove that nobody jumps to the next instruction.
[ ... ]
Quoted 14 lines
arch/x86/kernel/uprobes.c +6
-0@@ -1866,10 +2001,210 @@ static int ptwrite_text_poke(struct arch_uprobe *auprobe,return err; } +/* +* Replace an aligned five-byte NOP run with a JMP in one eight-byte store. +* The trailing three bytes are read from the existing text. We assume +* nobody else is changing it. This is covered by the Intel/AMD "aligned store" +* cross modifying guarantee. +*/
[ ... ]
Quoted 18 lines
arch/x86/kernel/uprobes.c +18
-0+static int pun_install (struct arch_uprobe *auprobe, +struct vm_area_struct *vma,unsigned long vaddr, +const u8 *orig) +{ +struct mm_struct *mm = vma->vm_mm; +struct uprobe_ptwrite_page *ptw; +struct uprobe_ptwrite_arch *ptw_a = &auprobe->ptwrite; +struct uprobes_state *state = &mm->uprobes_state; +struct write_opcode_ctx ctx = { + .base = vaddr, + .expect =EXPECT_BYTE , + .expect_byte = orig[0 ], + }; +unsigned long t, page_base, block_off, stub_addr; +s64 site_delta; +s32 jump_rel, disp32, orig_rel; +u8 site_len; +bool found = false;
[ ... ]
arch/x86/kernel/uprobes.c static int ptwrite_text_poke(struct arch_uprobe *auprobe, · line 2177
Quoted 13 lines
arch/x86/kernel/uprobes.c +13
-0+/* +* A NOP fallback needs a synthetic rel32 at the site, so it uses +* the full five-byte poke and restore path rather than punning. +*/ + site_len = nop_fallback ?5 : ptw_a->len; + ptw->index[ptw->nblocks].off = block_off; + ptw->index[ptw->nblocks].len = ptw_a->stub_len; + ptw->index[ptw->nblocks].ft_off = ptw_a->ft_off; + ptw->index[ptw->nblocks].pun = !nop_fallback; + ptw->index[ptw->nblocks].orig0 = orig[0 ]; + ptw->index[ptw->nblocks].site_len = site_len; + ptw->index[ptw->nblocks].site_off = (s32 )site_delta; +memcpy (ptw->index[ptw->nblocks].site_insn, orig, site_len);
[Severity: High]
Does this memcpy() read out-of-bounds?
The original instruction is passed down via the orig parameter from arch_uprobe_install_ptwrite(). In arch_uprobe_install_ptwrite(), orig is declared as a 5-byte array on the stack.
However, site_len here can be up to 15 bytes (MAX_UINSN_BYTES) for generic instructions when punning is active. If site_len is greater than 5, this will copy uninitialized kernel stack memory into the permanent site_insn array.
Later, when attempting to reinstall the uprobe, the memcmp() against the true original instruction will fail due to the garbage bytes. This breaks the collision check and could cause a trampoline slot leak on every reinstall.
Quoted 1 line
arch/x86/kernel/uprobes.c +1
-0+smp_store_release (&ptw->nblocks, ptw->nblocks +1 );
[ ... ]
Quoted 21 lines
arch/x86/kernel/uprobes.c +11
-3@@ -1937,15 +2301,63 @@ int arch_uprobe_uninstall_ptwrite(struct arch_uprobe *auprobe, struct vm_area_struct *vma, unsigned long vaddr) {struct mm_struct *mm = vma->vm_mm; +struct uprobe_ptwrite_arch *ptw_a = &auprobe->ptwrite; +struct uprobes_state *state = &mm->uprobes_state;u8 cur[5 ]; +int b; +struct write_opcode_ctx ctx = { + .base = vaddr, + .expect =EXPECT_BYTE , + .expect_byte =0xe9 , + };mmap_assert_write_locked (mm); -if (copy_from_vaddr (mm, vaddr, cur,sizeof (cur)) || - !ptwrite_is_installed (mm, vaddr, cur)) -return ; + { +struct uprobe_ptwrite_page *ptw; +struct uprobe_ptwrite_page *fpw =NULL ;
arch/x86/kernel/uprobes.c int arch_uprobe_uninstall_ptwrite(struct arch_uprobe *auprobe, · line 2327
Quoted 10 lines
arch/x86/kernel/uprobes.c +10
-0+s32 rel; +s64 target; +unsigned long page_base, boff; +int ret; + + ret =copy_from_vaddr (mm, vaddr, cur,sizeof (cur)); +if (ret) +return ret; +if (!ptwrite_is_installed (mm, vaddr, cur)) +return 0 ;
[Severity: High]
Does this validation incorrectly abandon punned uprobes?
If a punned uprobe is installed on an instruction with a relative offset, bytes 1-4 are mutable user code. The user application (e.g., a JIT compiler or dynamic linker) might legitimately update the relative offset in these subsequent bytes.
When the uprobe is uninstalled, ptwrite_is_installed() strictly requires the JMP target to still point to the trampoline. Since the user modified the relative displacement, ptwrite_is_installed() returns false.
This assumes the probe was entirely overwritten and aborts the uninstall prematurely, returning 0. As a result, the injected 0xe9 (JMP) opcode is left permanently in the user's execution stream, which can corrupt the instruction stream and cause crashes or erratic behavior.
Quoted 3 lines
arch/x86/kernel/uprobes.c +3
-0+ +memcpy (&rel, cur +1 ,sizeof (rel)); + target = (s64 )vaddr +5 + (s64 )rel;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
RFC 14/19 ptwrite uprobes: Use atomic patching for multinop sites
In reply to Andi KleenThe earlier multinop patching is not quite safe because the cross modified CPU could be already executing on a later nop when the cross patching occurs. The Intel SDM allows cross modification by larger stores as long as they are aligned. AMD has a similar guarantee.
The motivation for multinop is mainly to support the gcc function entry patch sites and these are always aligned.
So enforce 8 bytes alignment of the multinop and use a safe RMW 8 byte store ot overwrite the 5 byte sequence. This assumes that the code is not changing in parallel, but if that happens cross modification safety is probably the smallest of the issues.
--- arch/x86/kernel/uprobes.c | 32 +++++++++++++++++++++++++------- 1 file changed, 25 insertions(+), 7 deletions(-)
arch/x86/kernel/uprobes.c +25 -7
@@ -1812,8 +1812,9 @@ get_uprobe_ptwrite_page(struct mm_struct *mm, unsigned long vaddr, }/* -* A run of short NOPs is accepted only when requested. This validation does -* not make the three-phase poke safe for threads that already passed byte 0. +* A run of short NOPs is accepted only when requested. It is patched with +* an aligned eight-byte read-modify-write, preserving the following bytes; +* code is not expected to change concurrently. */ static bool pun_site_is_nop (const u8 *orig,bool allow_nop_run) {
@@ -1831,6 +1832,13 @@ static bool pun_site_is_nop(const u8 *orig, bool allow_nop_run) orig[2 ] ==0x90 && orig[3 ] ==0x90 && orig[4 ] ==0x90 ; } +/* Identify the explicitly opted-in run of five one-byte NOPs. */ +static bool ptwrite_site_is_multinop (const u8 *orig,bool allow_nop_run) +{ +return allow_nop_run && orig[0 ] ==0x90 && orig[1 ] ==0x90 && + orig[2 ] ==0x90 && orig[3 ] ==0x90 && orig[4 ] ==0x90 ; +} +/* * Classify the site's single instruction for out-of-line execution. * Returns the length, or 0 when it cannot run safely out of line.
@@ -2224,6 +2232,9 @@ int arch_uprobe_install_ptwrite(struct arch_uprobe *auprobe, ret =copy_from_vaddr (mm, vaddr, orig,sizeof (orig));if (ret)return ret; +if (ptwrite_site_is_multinop (orig, ptw_a->allow_nop_run) && + (vaddr &7 )) +return pun_install (auprobe, vma, vaddr, orig);if (ptwrite_is_installed (mm, vaddr, orig))return 0 ;
@@ -2249,9 +2260,13 @@ int arch_uprobe_install_ptwrite(struct arch_uprobe *auprobe,continue ;if (!__in_uprobe_ptwrite (mm, ptw->vaddr))continue ; - ret =ptwrite_text_poke (auprobe, vma, vaddr, - ptw->vaddr + ptw->index[b].off); - goto out; +if (ptwrite_site_is_multinop (orig, ptw_a->allow_nop_run)) + ret =ptwrite_multinop_text_poke (auprobe, vma, vaddr, + ptw->vaddr + ptw->index[b].off); +else + ret =ptwrite_text_poke (auprobe, vma, vaddr, + ptw->vaddr + ptw->index[b].off); +return ret; } ptw =get_uprobe_ptwrite_page (mm, vaddr, ptw_a->stub_len);if (!ptw)
@@ -2287,8 +2302,11 @@ int arch_uprobe_install_ptwrite(struct arch_uprobe *auprobe,memcpy (kaddr + block_off + ptw_a->jmp_off, &rel,sizeof (rel));kunmap_local (kaddr); - ret =ptwrite_text_poke (auprobe, vma, vaddr, stub_addr); -if (ret) +if (ptwrite_site_is_multinop (orig, ptw_a->allow_nop_run)) + ret =ptwrite_multinop_text_poke (auprobe, vma, vaddr, stub_addr); +else + ret =ptwrite_text_poke (auprobe, vma, vaddr, stub_addr); +if (ret) {/* Publish rollback before readers use the reduced block count. */ smp_store_release (&ptw->nblocks, ptw->nblocks -1 );return ret;
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [RFC v1 14/19] ptwrite uprobes: Use atomic patching for multinop sites
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
New issues:
- [High] Structurally unsafe patching of multinop sequences in user space due to thread preemption.
- [High] Unaligned multinops silently fall back to the unsafe INT3 patching method, violating the patch's safety guarantees.
- [Medium] Lockless Read-Modify-Write (RMW) of user memory resulting in a TOCTOU race and lost updates.
Pre-existing issues:
- [High] O(N) sequential synchronize_rcu() calls inside an RCU teardown loop causing severe process exit delays. --
commit 5a4cb6048cbb1f8b37fe6998acf1c5e1ac8b2670
Author: Andi Kleen <[email protected]>
ptwrite uprobes: Use atomic patching for multinop sites
This commit modifies the patching mechanism for 5-byte NOP sequences (multinop) in ptwrite uprobes. It enforces 8-byte alignment for these sites and replaces the earlier INT3 patching method with an atomic 8-byte read-modify-write store.
Quoted 4 lines
arch/x86/kernel/uprobes.c +0
-0
[ ... ]
arch/x86/kernel/uprobes.c int arch_uprobe_install_ptwrite(struct arch_uprobe *auprobe, · line 2237
Quoted 7 lines
arch/x86/kernel/uprobes.c +3
-0@@ -2224,6 +2232,9 @@ int arch_uprobe_install_ptwrite(struct arch_uprobe *auprobe, ret =copy_from_vaddr (mm, vaddr, orig,sizeof (orig));if (ret)return ret; +if (ptwrite_site_is_multinop (orig, ptw_a->allow_nop_run) && + (vaddr &7 )) +return pun_install (auprobe, vma, vaddr, orig);
[Severity: High]
Does this silently apply the unsafe INT3 patching method to unaligned NOP sequences?
The commit message states that the earlier multinop patching is unsafe and enforces 8-byte alignment. However, when an unaligned 5-byte NOP sequence is requested (vaddr & 7 is true), the code delegates to pun_install().
Looking at pun_install(), this triggers the fallback path:
arch/x86/kernel/uprobes.c:pun_install() {
...
bool nop_fallback = ptwrite_site_is_multinop(orig,
ptw_a->allow_nop_run) &&
(vaddr & 7);
...
if (nop_fallback)
ret = ptwrite_text_poke(auprobe, vma, vaddr, stub_addr);
...
}Will this expose the traced process to the same user-space crash issues that this patch intends to fix?
Quoted 2 lines
arch/x86/kernel/uprobes.c +0
-0if (ptwrite_is_installed (mm, vaddr, orig))return 0 ;
[ ... ]
arch/x86/kernel/uprobes.c int arch_uprobe_install_ptwrite(struct arch_uprobe *auprobe, · line 2265
Quoted 10 lines
arch/x86/kernel/uprobes.c +3
-3@@ -2249,9 +2260,13 @@ int arch_uprobe_install_ptwrite(struct arch_uprobe *auprobe,continue ;if (!__in_uprobe_ptwrite (mm, ptw->vaddr))continue ; - ret =ptwrite_text_poke (auprobe, vma, vaddr, - ptw->vaddr + ptw->index[b].off); - goto out; +if (ptwrite_site_is_multinop (orig, ptw_a->allow_nop_run)) + ret =ptwrite_multinop_text_poke (auprobe, vma, vaddr, + ptw->vaddr + ptw->index[b].off);
[Severity: High]
Is this structural patching safe against thread preemption?
If a user thread executes the first byte of a 5-byte NOP sequence (such as one generated by -fpatchable-function-entry) and is preempted, the kernel will patch the sequence with a 5-byte JMP.
When the thread resumes at vaddr + 1, won't it execute the relative offset bytes of the JMP as garbage instructions and crash with SIGILL or SIGSEGV? While an atomic store prevents fetch anomalies, does it prevent a preempted thread from resuming execution in the middle of the patched instructions?
[Severity: Medium]
Can this lockless Read-Modify-Write result in a TOCTOU race and lost updates?
Looking at ptwrite_multinop_text_poke():
arch/x86/kernel/uprobes.c:ptwrite_multinop_text_poke() {
...
err = copy_from_vaddr(mm, vaddr, patch, sizeof(patch));
if (err)
return err;
patch[0] = 0xe9;
memcpy(&patch[1], &rel, sizeof(rel));
err = uprobe_write(auprobe, vma, vaddr, patch, sizeof(patch),
verify_insn, true, false, &ctx);
...
}If a user process concurrently modifies the trailing bytes (bytes 5-7) of the 8-byte aligned word between the copy_from_vaddr() read and the uprobe_write(), won't those modifications be silently overwritten and lost?
[Severity: High]
This is a pre-existing issue, but does the process cleanup path force an extremely slow O(N) grace-period wait?
Looking at arch_uprobe_clear_state():
arch/x86/kernel/uprobes.c:arch_uprobe_clear_state() {
...
hlist_for_each_entry_safe(ptw, n, &state->head_ptwrite, node) {
hlist_del_rcu(&ptw->node);
synchronize_rcu();
__free_page(ptw->page);
kfree(ptw);
}
...
}If a traced process has many ptwrite uprobes and exits, it will block sequentially for an RCU grace period for every single page. Could this cause severe process exit delays or hung task timeouts?
Quoted 5 lines
arch/x86/kernel/uprobes.c +4
-0+else + ret =ptwrite_text_poke (auprobe, vma, vaddr, + ptw->vaddr + ptw->index[b].off); +return ret; }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=14
RFC 15/19 ptwrite uprobes: Add a tutorial and overview documentation
In reply to Andi KleenNo code changes.
--- Documentation/trace/index.rst | 1 + Documentation/trace/ptwrite-uprobes.rst | 390 ++++++++++++++++++++++++ 2 files changed, 391 insertions(+) create mode 100644 Documentation/trace/ptwrite-uprobes.rst
Documentation/trace/index.rst +1 -0
@@ -90,6 +90,7 @@ interactions. .. toctree:: :maxdepth: 1 + ptwrite-uprobes user_events uprobetracer
Documentation/trace/ptwrite-uprobes.rst new file +390 -0
390 lines not on this page · raw message
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [RFC v1 15/19] ptwrite uprobes: Add a tutorial and overview documentation
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [Low] Missing blank line before a bullet list in reStructuredText. - [Low] Typos in the 'Options in this…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Low] Missing blank line before a bullet list in reStructuredText.
- [Low] Typos in the 'Options in this case' list.
- [Low] Formatting inconsistency with double backticks in indented block. --
commit e96dac3fdefb27a2867c9e268822b99d3e4c7aa7
Author: Andi Kleen <[email protected]>
ptwrite uprobes: Add a tutorial and overview documentation
This patch adds a new documentation file explaining how to use ptwrite uprobes for hardware tracing. It covers performance benefits, requirements, and provides examples for configuring probes via tracefs and perf probe.
Quoted 5 lines
Documentation/trace/ptwrite-uprobes.rst new file +0
-0
[ ... ]
Documentation/trace/ptwrite-uprobes.rst line 121
Quoted 8 lines
Documentation/trace/ptwrite-uprobes.rst new file +8
-0+Punning is a probabilistic method that depends on the existing +instruction bytes and the placement of the executable in memory. +It has a high chance of success on PIE/PIC binaries, but tends +to work poorly on non PIE main executables. + +When punning is not possible the probe is rejected at install +time. Options in this case: +- Move the probe site to a different instruction which may work.
[Severity: Low]
Does this list render correctly in Sphinx without a blank line before it? In reStructuredText, bullet lists generally must be preceded by a blank line, otherwise the list items may get merged into the preceding paragraph or trigger build warnings.
Documentation/trace/ptwrite-uprobes.rst line 126
Quoted 5 lines
+- Rebuild with -fPIE if it's a main problem not using PIE. +- Enable or disable /proc/sys/kernel/randomize_va_space. If theDocumentation/trace/ptwrite-uprobes.rst new file +3
-0+ randomization is enabled it may also just work on a rerun of + the program. +- Fall back to a classic uprobes
[Severity: Low]
There are a couple of minor typos in this section. Could "main problem" be corrected to "main program", and "a classic uprobes" be changed to "a classic uprobe" or "classic uprobes"?
[ ... ]
Documentation/trace/ptwrite-uprobes.rst line 333
Quoted 10 lines
Documentation/trace/ptwrite-uprobes.rst new file +10
-0+If the program itself also executes own ``PTWRITE``, those values mix with the +uprobe output in the stream. The decoder uses the header magic to identify +uprobe records. Other values are printed as ``manual ptwrite:`` lines (with +their IP when ``fup_on_ptw`` is set) and counted in the summary's ``stray`` +field. + +To also print the decoded branch stream alongside the records, add +``b`` to the itrace options and drop the ``q`` + + # ``perf script --itrace=web -s uprobe-ptwrite-decode.py -i perf.data``
[Severity: Low]
Are the double backticks intended here? Because this line is already indented as a literal block, the double backticks will likely be rendered verbatim in the output instead of being parsed as inline literal formatting.
Quoted 3 lines
Documentation/trace/ptwrite-uprobes.rst new file +3
-0+ +Each decoded branch prints as a ``branch:`` line (from => to, with +symbols where resolvable), interleaved with the probe records and any
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=15
RFC 16/19 ptwrite uprobes / perf tools pt: Improve FUP error handling for ptwrite
In reply to Andi KleenThe PT decoder can't see the patched code generated by ptwrite uprobes.
Without ptw_on_fup the ptwrite stubs are invisible to PT branch tracing (other than the ptwrite packet itself) because they don't contain any indirect or conditional branches, so there is no problem with the PT decoder.
However when ptw_on_fup is enabled there is a FUP (Flow Update Packet) reporting the IP of each ptwrite after the PTW packets. The decoder tries to resolve this FUP packet to the code, but it errors out because it can't see the uprobes generated code.
Normally this is not a problem because we just use 'q' mode which doesn't walk instructions, but still reports on the ptwrites and their FUPs.
Also it's possible to disable fup_on_ptw, however that reduces the tolerance to data loss in the uprobes ptwrite decoder.
When full instruction tracing is desired the PTW+FUP errors cause data loss.
Special case this in the decoder instead. When the FUP is associated with a ptwrite don't error out on missing instructions pages. Just report the ptwrite with its IP and continue.
An alternative would be to define a metadata event for the JITed code and let the decoder understand it. That may be desirable in the future so that the PT users sees all the code executed. But for now this simple change is good enough.
--- .../util/intel-pt-decoder/intel-pt-decoder.c | 23 ++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-)
tools/perf/util/intel-pt-decoder/intel-pt-decoder.c +22 -1
@@ -1440,8 +1440,29 @@ static int intel_pt_walk_fup(struct intel_pt_decoder *decoder)return -EAGAIN ; } decoder->set_fup_tx_flags = false; -if (err) +if (err) { +/* +* A ptwrite's FUP can target an address whose +* instruction cannot be resolved (e.g. the +* [uprobes-ptwrite] stub is an anonymous special +* mapping invisible to the machine). The FUP is +* still the ptwrite's IP: report it rather than +* failing the whole walk. +*/ +if (decoder->set_fup_ptw) { + decoder->set_fup_ptw = false; + decoder->pkt_state =INTEL_PT_STATE_IN_SYNC ; + decoder->state.type &= ~INTEL_PT_BRANCH ; + decoder->state.type |=INTEL_PT_PTW ; + decoder->state.flags |=INTEL_PT_FUP_IP ; + decoder->state.from_ip = decoder->ip; + decoder->state.to_ip =0 ; + decoder->state.ptw_payload = + decoder->fup_ptw_payload; +return 0 ; + }return err; + }if (intel_pt_insn.branch ==INTEL_PT_BR_INDIRECT ) { intel_pt_log_at("ERROR: Unexpected indirect branch",
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [RFC v1 16/19] ptwrite uprobes / perf tools pt: Improve FUP error handling for ptwrite
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] Incorrect ptwrite IP reporting and failure to prevent trace abort in full instruction tracing…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Incorrect ptwrite IP reporting and failure to prevent trace abort in full instruction tracing mode.
- [Medium] Bypassing intel_pt_fup_event() causes incomplete manual state reconstruction, leaking instruction metadata and asynchronous FUP flags. --
commit f901af3d0d6f6651e120a5aaaf5502b7aa3f6612
Author: Andi Kleen <[email protected]>
ptwrite uprobes / perf tools pt: Improve FUP error handling for ptwrite
Special case the decoder to handle ptwrite FUPs targeting unmapped code like uprobe stubs. When the decoder can't resolve the instruction page, report the ptwrite with its IP instead of erroring out and failing the walk.
Quoted 13 lines
tools/perf/util/intel-pt-decoder/intel-pt-decoder.c +4
-1@@ -1440,8 +1440,29 @@ static int intel_pt_walk_fup(struct intel_pt_decoder *decoder)return -EAGAIN ; } decoder->set_fup_tx_flags = false; -if (err) +if (err) { +/* +* A ptwrite's FUP can target an address whose +* instruction cannot be resolved (e.g. the
tools/perf/util/intel-pt-decoder/intel-pt-decoder.c static int intel_pt_walk_fup(struct intel_pt_decoder *decoder) · line 1456
Quoted 10 lines
tools/perf/util/intel-pt-decoder/intel-pt-decoder.c +10
-0+* [uprobes-ptwrite] stub is an anonymous special +* mapping invisible to the machine). The FUP is +* still the ptwrite's IP: report it rather than +* failing the whole walk. +*/ +if (decoder->set_fup_ptw) { + decoder->set_fup_ptw = false; + decoder->pkt_state =INTEL_PT_STATE_IN_SYNC ; + decoder->state.type &= ~INTEL_PT_BRANCH ; + decoder->state.type |=INTEL_PT_PTW ;
[Severity: Medium]
Will bypassing intel_pt_fup_event() here leak instruction metadata and asynchronous FUP flags?
Because intel_pt_walk_insn() returned an error early, decoder->state.insn_op and insn_len retain the metadata from the last successfully decoded instruction. By skipping intel_pt_fup_event() which usually clears these fields and handles asynchronous flags, could stale instruction op/len and pending event flags be incorrectly attributed to subsequent events?
tools/perf/util/intel-pt-decoder/intel-pt-decoder.c static int intel_pt_walk_fup(struct intel_pt_decoder *decoder) · line 1458
Quoted 2 lines
tools/perf/util/intel-pt-decoder/intel-pt-decoder.c +2
-0+ decoder->state.flags |=INTEL_PT_FUP_IP ; + decoder->state.from_ip = decoder->ip;
[Severity: High]
Does this code assign the wrong IP for the PTWRITE event and fail to prevent the trace abort it intends to fix?
When intel_pt_walk_insn() encounters an unmapped region like the uprobe stub, it returns an error without advancing decoder->ip. The true PTWRITE IP from the hardware FUP packet is passed as the ip parameter (from decoder->last_ip) to intel_pt_walk_insn().
Assigning decoder->state.from_ip = decoder->ip instead of using the ip parameter attributes the event to the stuck failure address. Additionally, because decoder->ip is never advanced out of the unmapped region, subsequent packets could trigger functions like intel_pt_walk_tip() to call intel_pt_walk_insn(..., 0) starting from the same stuck decoder->ip, failing again and aborting the trace.
Quoted 7 lines
tools/perf/util/intel-pt-decoder/intel-pt-decoder.c +6
-0+ decoder->state.to_ip =0 ; + decoder->state.ptw_payload = + decoder->fup_ptw_payload; +return 0 ; + }return err; + }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=16
RFC 17/19 ptwrite uprobes / perf tools probe: Add support of ptwrite probes
In reply to Andi KleenAdd a --ptwrite option to perf probe to enable ptwrite probes. It is mainly identical to the normal uprobes support, but knows about the new ptw:
probe syntax.
--- tools/perf/Documentation/perf-probe.txt | 6 ++++++ tools/perf/builtin-probe.c | 17 ++++++++++++++++- tools/perf/util/probe-event.c | 18 +++++++++++++----- tools/perf/util/probe-event.h | 2 ++ 4 files changed, 37 insertions(+), 6 deletions(-)
tools/perf/Documentation/perf-probe.txt +6 -0
@@ -136,6 +136,12 @@ OPTIONS --max-probes=NUM:: Set the maximum number of probe points for an event. Default is 128. +--ptwrite:: + Create a ptwrite uprobe (with -x). The probe writes its values + into the Intel PT trace stream instead of entering the kernel, + which is faster. Requires an Intel PT/PTWRITE CPU and a kernel + with ptwrite uprobe support. + --target-ns=PID: Obtain mount namespace information from the target pid. This is used when creating a uprobe for a process that resides in a
tools/perf/builtin-probe.c +16 -1
@@ -40,6 +40,7 @@ static struct {int command;/* Command short_name */ bool list_events;bool uprobes; +bool ptwrite;bool target_used;int nevents;struct perf_probe_event events[MAX_PROBES ];
@@ -62,6 +63,7 @@ static int parse_probe_event(const char *str) } pev->uprobes = params->uprobes; + pev->ptwrite = params->ptwrite;if (params->target) { pev->target =strdup (params->target);if (!pev->target)
@@ -73,9 +75,15 @@ static int parse_probe_event(const char *str)/* Parse a perf-probe command into event */ ret =parse_perf_probe_command (str, pev); +if (ret <0 ) +return ret; +if (pev->ptwrite && pev->point.retprobe) { +pr_err ("Error: ptwrite probes are entry-only (no %%return).\n" ); +return -EINVAL ; + }pr_debug ("%d arguments\n" , pev->nargs); -return ret; +return 0 ; }static int params_add_filter (const char *str)
@@ -532,6 +540,8 @@ __cmd_probe(int argc, const char **argv) "be more verbose (show parsed arguments, etc)"),OPT_BOOLEAN ('q' ,"quiet" , &quiet,"be quiet (do not show any warnings or messages)" ), +OPT_BOOLEAN (0 ,"ptwrite" , ¶ms->ptwrite, +"create trap-free ptwrite uprobes (requires -x)" ),OPT_CALLBACK_DEFAULT ('l' ,"list" ,NULL ,"[GROUP:]EVENT" ,"list up probe events" , opt_set_filter_with_command, DEFAULT_LIST_FILTER),
@@ -733,6 +743,11 @@ __cmd_probe(int argc, const char **argv)parse_options_usage (NULL , options,"x" , true);return -EINVAL ; } +if (params->ptwrite && !params->uprobes) { +pr_err (" Error: --ptwrite requires -x.\n" ); +parse_options_usage (NULL , options,"x" , true); +return -EINVAL ; + } ret =perf_add_probe_events (params->events, params->nevents);if (ret <0 ) {
tools/perf/util/probe-event.c +13 -5
@@ -1930,6 +1930,7 @@ int parse_probe_trace_command(const char *cmd, struct probe_trace_event *tev) ret = -EINVAL ; goto out; } + tev->ptwrite = !strcmp (fmt1_str,"ptw" ); pr = fmt1_str[0 ]; tev->group =strdup (fmt2_str); tev->event =strdup (fmt3_str);
@@ -1937,7 +1938,8 @@ int parse_probe_trace_command(const char *cmd, struct probe_trace_event *tev) ret = -ENOMEM ; goto out; } -pr_debug ("Group:%s Event:%s probe:%c\n" , tev->group, tev->event, pr); +pr_debug ("Group:%s Event:%s probe:%c%s\n" , tev->group, tev->event, pr, + tev->ptwrite ?" (ptwrite)" :"" ); tp->retprobe = (pr =='r' );
@@ -2260,9 +2262,11 @@ char *synthesize_probe_trace_command(struct probe_trace_event *tev)if (strbuf_init (&buf,32 ) <0 )return NULL ; -if (strbuf_addf (&buf,"%c:%s/%s " , tp->retprobe ?'r' :'p' , - tev->group, tev->event) <0 ) - goto error; +if (tev->ptwrite) + err =strbuf_addf (&buf,"ptw:%s/%s " , tev->group, tev->event); +else + err =strbuf_addf (&buf,"%c:%s/%s " , tp->retprobe ?'r' :'p' , + tev->group, tev->event);if (tev->uprobes) err =synthesize_uprobe_trace_def (tp, &buf);
@@ -2274,7 +2278,6 @@ char *synthesize_probe_trace_command(struct probe_trace_event *tev)if (err >=0 ) ret =strbuf_detach (&buf,NULL ); -error:strbuf_release (&buf);return ret; }
@@ -2996,6 +2999,7 @@ static int __add_probe_trace_events(struct perf_probe_event *pev, ret =0 ;for (i =0 ; i < ntevs; i++) { tev = &tevs[i]; + tev->ptwrite = pev->ptwrite; up = tev->uprobes ?1 :0 ;if (fd[up] ==-1 ) {/* Open the kprobe/uprobe_events */ fd[up] = __open_probe_file_and_namelist(up,
@@ -3610,6 +3614,8 @@ int convert_perf_probe_events(struct perf_probe_event *pevs, int npevs)/* Loop 1: convert all events */ for (i =0 ; i < npevs; i++) { +int j; +/* Init kprobe blacklist if needed */ if (!pevs[i].uprobes)kprobe_blacklist__init ();
@@ -3618,6 +3624,8 @@ int convert_perf_probe_events(struct perf_probe_event *pevs, int npevs)if (ret <0 )return ret; pevs[i].ntevs = ret; +for (j =0 ; j < pevs[i].ntevs; j++) + pevs[i].tevs[j].ptwrite = pevs[i].ptwrite; }/* This just release blacklist only if allocated */ kprobe_blacklist__release ();
tools/perf/util/probe-event.h +2 -0
@@ -60,6 +60,7 @@ struct probe_trace_event {int nargs;/* Number of args */ int lang;/* Dwarf language code */ bool uprobes;/* uprobes only */ +bool ptwrite;/* ptwrite uprobe (trap-free) */ struct probe_trace_arg *args;/* Arguments */ };
@@ -99,6 +100,7 @@ struct perf_probe_event {int nargs;/* Number of arguments */ bool sdt;/* SDT/cached event flag */ bool uprobes;/* Uprobe event flag */ +bool ptwrite;/* ptwrite uprobe (trap-free) */ char *target;/* Target binary */ struct perf_probe_arg *args;/* Arguments */ struct probe_trace_event *tevs;
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [RFC v1 17/19] ptwrite uprobes / perf tools probe: Add support of ptwrite probes
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The error code from `strbuf_addf` is silently overwritten, hiding memory allocation or…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The error code from strbuf_addf is silently overwritten, hiding memory allocation or formatting failures. --
commit 1e80616b3ae889174ea132bb4a1470dd0a9429ba
Author: Andi Kleen <[email protected]>
ptwrite uprobes / perf tools probe: Add support of ptwrite probes
This commit adds a --ptwrite option to perf probe to enable trap-free ptwrite uprobes. These probes write their values into the Intel PT trace stream instead of entering the kernel.
Quoted 3 lines
tools/perf/util/probe-event.c +0
-0
[ ... ]
Quoted 5 lines
tools/perf/util/probe-event.c +0
-1@@ -2260,9 +2262,11 @@ char *synthesize_probe_trace_command(struct probe_trace_event *tev)if (strbuf_init (&buf,32 ) <0 )return NULL ; - if (strbuf_addf(&buf, "%c:%s/%s ", tp->retprobe ? 'r' : 'p',
tools/perf/util/probe-event.c char *synthesize_probe_trace_command(struct probe_trace_event *tev) · line 2272
Quoted 10 lines
tools/perf/util/probe-event.c +5
-2- tev->group, tev->event) < 0) - goto error; +if (tev->ptwrite) + err =strbuf_addf (&buf,"ptw:%s/%s " , tev->group, tev->event); +else + err =strbuf_addf (&buf,"%c:%s/%s " , tp->retprobe ?'r' :'p' , + tev->group, tev->event);if (tev->uprobes) err =synthesize_uprobe_trace_def (tp, &buf);
[Severity: Medium]
Does this code silently overwrite the error code from strbuf_addf()?
It appears that if memory allocation fails inside the first strbuf_addf(), the function continues without checking err. The negative error code is then overwritten by the return value of the subsequent synthesize_uprobe_trace_def() or synthesize_kprobe_trace_def() call.
Could this lead to a truncated or malformed trace command being synthesized and sent to the kernel?
Quoted 8 lines
tools/perf/util/probe-event.c +0
-1@@ -2274,7 +2278,6 @@ char *synthesize_probe_trace_command(struct probe_trace_event *tev)if (err >=0 ) ret =strbuf_detach (&buf,NULL ); -error:strbuf_release (&buf);return ret; }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=17
RFC 18/19 ptwrite uprobes / perf tools script: Add ptwrite uprobes decoder
In reply to Andi KleenAdd a decoder for the PTWRITE records generated by ptwrite uprobes. This runs as a python script in perf script.
The main tricky part is how to detect and recover data loss or interleaving with other ptwrites. The decoder uses a magic value in the header to synchronize, and also the IP from the FUP when available (fup_on_ptw=1)
It can display other events (including branches) when they are available.
--- .../scripts/python/uprobe-ptwrite-decode.py | 416 ++++++++++++++++++ 1 file changed, 416 insertions(+) create mode 100755 tools/perf/scripts/python/uprobe-ptwrite-decode.py
tools/perf/scripts/python/uprobe-ptwrite-decode.py new file +416 -0
@@ -0,0 +1,416 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-only +""" +uprobe-ptwrite-decode.py - decode ptwrite-uprobe trace events out of +PT logs. + +Usage (perf script): + perf script --itrace=qwe -s uprobe-ptwrite-decode.py -i perf.data +""" +import os +import re +import struct +import sys + +TRACEFS = "/sys/kernel/tracing" + +# type name -> (size bytes, signed, hex, char) +_TYPES = { + "u8": (1, False, False, False), + "u16": (2, False, False, False), + "u32": (4, False, False, False), + "u64": (8, False, False, False), + "s8": (1, True, False, False), + "s16": (2, True, False, False), + "s32": (4, True, False, False), + "s64": (8, True, False, False), + "x8": (1, False, True, False), + "x16": (2, False, True, False), + "x32": (4, False, True, False), + "x64": (8, False, True, False), + "char": (1, False, False, True), +} + +_FIELD_RE = re.compile(r'^\s*field:(\S+)\s+(arg\d+);') + +HDR_MAGIC = 0x5054525731 # "PTRW1" +HDR_MASK = (1 << 40) - 1 + +def load_events(root=TRACEFS): + """Scan tracefs for (event_id -> (name, [(arg name, type)])).""" + events = {} + try: + groups = os.listdir(root + "/events") + except OSError: + return events + for g in groups: + gdir = root + "/events/" + g + if not os.path.isdir(gdir): + continue + for ev in os.listdir(gdir): + edir = gdir + "/" + ev + if not os.path.isdir(edir): + continue + try: + with open(edir + "/id") as f: + eid = int(f.read().strip()) + with open(edir + "/format") as f: + fields = [] + for line in f: + m = _FIELD_RE.match(line) + if m: + fields.append((m.group(2), m.group(1))) + events[eid] = (g + "/" + ev, fields) + except (OSError, ValueError): + continue + return events + +def type_info(t): + """('u64', size, signed, hex, char) for a format-file type name.""" + if t in _TYPES: + return (t,) + _TYPES[t] + return (t, 8, False, False, False) + +def fmt_value(v, t): + name, size, signed, ishex, ischar = type_info(t) + mask = (1 << (8 * size)) - 1 + v &= mask + if ischar: + return repr(chr(v)) + if signed: + sign = 1 << (8 * size - 1) + if v & sign: + v -= 1 << (8 * size) + if ishex: + return "0x%x" % v + return str(v) + +def is_header(word, events): + if (word & HDR_MASK) != HDR_MAGIC: + return None + eid = (word >> 48) & 0xffff + nargs = (word >> 40) & 0xff + if not (1 <= nargs <= 8): + return None + if events and eid not in events: + return None + return (eid, nargs) + +def decode_words(words, events): + """Walk the ptwrite stream.""" + records = dropped = stray = unknown = 0 + cur = None # (event_id, nargs, [args]) + rec_drop = False + learned = {} + arg_off = {} + lines = [] + for w in words: + if isinstance(w, tuple): + word, ip, key = (w + (0, 0))[:3] if len(w) < 3 else (w[0], w[1], w[2]) + else: + word, ip, key = w, 0, 0 + off = (ip & 0xfff) if ip else 0 + hdr = is_header(word, events) + if hdr is not None: + if cur is not None: + dropped += cur[1] - len(cur[2]) + cur = (hdr[0], hdr[1], []) + rec_drop = False + continue + if cur is None: + # no record open: this is a raw ptwrite from the program + lines.append((key, "manual ptwrite: payload=0x%x ip=0x%x" + % (word, ip))) + stray += 1 + continue + eid, nargs, args = cur + if len(args) >= nargs: + cur = None + lines.append((key, "manual ptwrite: payload=0x%x ip=0x%x" + % (word, ip))) + stray += 1 + continue + if ip and learned.get(eid): + offs = arg_off[eid] + expect = offs[len(args)] + if off != expect: + j = next((k for k in range(len(args) + 1, nargs) + if off == offs[k]), None) + if j is not None: + dropped += j - len(args) + args.extend([None] * (j - len(args))) + rec_drop = True + else: + unknown += 1 + continue + if ip and not learned.get(eid): + offsets = arg_off.setdefault(eid, set()) + if len(offsets) < nargs: + offsets.add(off) + args.append(word) + if len(args) == nargs: + if ip and not learned.get(eid): + offs = sorted(set(arg_off.get(eid, []))) + if len(offs) == nargs: + arg_off[eid] = offs + learned[eid] = True + if not rec_drop: + ev = (events or {}).get(eid) + name = ev[0] if ev else "?" + fields = ev[1] if ev else [] + parts = [] + for i, value in enumerate(args[:nargs]): + field_name = None + field_type = "u64" + if i < len(fields): + field_name, field_type = fields[i] + text = fmt_value(value, field_type) + parts.append("%s=%s" % (field_name, text) + if field_name else text) + fmt = ", ".join(parts) + lines.append((key, "record %d: event=%s id=0x%x args=[%s]" + % (records + 1, name, eid, fmt))) + records += 1 + cur = None + if cur is not None: + dropped += cur[1] - len(cur[2]) + + return records, dropped, stray, unknown, lines + +def _emit_record(eid, nargs, args, events, record_drop): + global _records + if record_drop: + return + ev = (events or {}).get(eid) + name = ev[0] if ev else "?" + fields = ev[1] if ev else [] + parts = [] + for i, value in enumerate(args[:nargs]): + field_name = None + field_type = "u64" + if i < len(fields): + field_name, field_type = fields[i] + text = fmt_value(value, field_type) + parts.append("%s=%s" % (field_name, text) + if field_name else text) + fmt = ", ".join(parts) + print("record %d: event=%s id=0x%x args=[%s]" % + (_records + 1, name, eid, fmt)) + +def _decode_stream_word(word, ip, events, stream_key): + global _records, _dropped, _stray, _unknown + state = _streams.setdefault(stream_key, {"cur": None, "drop": False}) + off = (ip & 0xfff) if ip else 0 + hdr = is_header(word, events) + if hdr is not None: + if state["cur"] is not None: + _dropped += state["cur"][1] - len(state["cur"][2]) + state["cur"] = (hdr[0], hdr[1], []) + state["drop"] = False + return + cur = state["cur"] + if cur is None: + print("manual ptwrite: payload=0x%x ip=0x%x" % (word, ip)) + _stray += 1 + return + eid, nargs, args = cur + if len(args) >= nargs: + state["cur"] = None + print("manual ptwrite: payload=0x%x ip=0x%x" % (word, ip)) + _stray += 1 + return + if ip and _learned.get(eid): + offs = _arg_off[eid] + expect = offs[len(args)] + if off != expect: + j = next((k for k in range(len(args) + 1, nargs) + if off == offs[k]), None) + if j is not None: + _dropped += j - len(args) + args.extend([None] * (j - len(args))) + state["drop"] = True + else: + _unknown += 1 + return + if ip and not _learned.get(eid): + offsets = _arg_off.setdefault(eid, set()) + if len(offsets) < nargs: + offsets.add(off) + args.append(word) + if len(args) == nargs: + if ip and not _learned.get(eid): + offs = sorted(set(_arg_off.get(eid, set()))) + if len(offs) == nargs: + _arg_off[eid] = offs + _learned[eid] = True + _emit_record(eid, nargs, args, events, state["drop"]) + _records += 1 + state["cur"] = None + +def decode_buf(raw_buf, events, ip=0, key=0, stream_key=None): + """Decode one 12-byte perf raw sample without retaining prior samples.""" + if len(raw_buf) < 12: + return + flags = struct.unpack_from("<I", raw_buf, 0)[0] + payload = struct.unpack_from("<Q", raw_buf, 4)[0] + if not flags & 1: + ip = 0 # no FUP: the IP is not recoverable + if stream_key is None: + stream_key = 0 + _decode_stream_word(payload, ip, events, stream_key) + +_events = {} +_streams = {} +_learned = {} +_arg_off = {} +_records = _dropped = _stray = _unknown = 0 +_other_count = 0 +_show_branches = "--no-branches" not in sys.argv + +def _sample_stream_key(sample): + if not sample: + return (None, None, None) + return (sample.get("cpu"), sample.get("pid"), sample.get("tid")) + +def auxtrace_error(*args): + global _errors + if len(args) >= 8: + _errors += 1 + print("ptwrite-decode: error type=%d code=%d cpu=%d pid=%d tid=%d" + " ip=0x%x msg=%s" % (args[0], args[1], args[2], args[3], + args[4], args[5], args[7])) + +def trace_begin(): + global _events, _records, _dropped, _stray, _unknown, _errors, _other_count + if not _events: + _events = load_events() + _streams.clear() + _learned.clear() + _arg_off.clear() + _records = _dropped = _stray = _unknown = 0 + _errors = _other_count = 0 + +def branch_line(param_dict): + sample = param_dict.get("sample") or {} + frm = sample.get("ip") or 0 + to = sample.get("addr") or 0 + frm_sym = param_dict.get("symbol") or "[unknown]" + to_sym = sample.get("addr_symbol") or "[unknown]" + frm_off = param_dict.get("symoff") or 0 + to_off = sample.get("addr_symoff") or 0 + frm_dso = param_dict.get("dso") or "[unknown]" + to_dso = sample.get("addr_dso") or "[unknown]" + fs = "+0x%x" % frm_off if frm_off else "" + ts_ = "+0x%x" % to_off if to_off else "" + return ("branch: 0x%x %s%s (%s) => 0x%x %s%s (%s)" + % (frm, frm_sym, fs, frm_dso, to, to_sym, ts_, to_dso)) + +def other_line(name, sample, comm, sym=None, off=0, dso=None): + """Format a non-ptwrite, non-branch event (classic uprobe, + tracepoint, sample) for interleaved printing.""" + pid = sample.get("pid") + tid = sample.get("tid") + ip = sample.get("ip") or 0 + loc = " %s+0x%x (%s)" % (sym, off, dso) if sym else "" + return ("event: %s comm=%s pid=%s tid=%s ip=0x%x%s" + % (name, comm, pid, tid, ip, loc)) + +def trace_unhandled(handler_name, context, fields, sample=None): + """Print tracepoint-class events immediately in delivery order.""" + global _other_count + fields = fields or {} + name = handler_name.replace("__", ":") + comm = fields.get("common_comm") or "" + pid = fields.get("common_pid") + cpu = fields.get("common_cpu") + print("event: %s comm=%s pid=%s cpu=%s" % + (name, comm, pid, cpu)) + _other_count += 1 + +def process_event(param_dict): + global _other_count + sample = param_dict.get("sample") or {} + name = param_dict.get("ev_name") or "" + if name == "ptwrite": + raw = param_dict.get("raw_buf") + if raw: + decode_buf(raw, _events, sample.get("ip") or 0, + stream_key=_sample_stream_key(sample)) + elif name.startswith("branches"): + if _show_branches: + print(branch_line(param_dict)) + else: + print(other_line(name, sample, + param_dict.get("comm") or "", + param_dict.get("symbol"), + param_dict.get("symoff") or 0, + param_dict.get("dso"))) + _other_count += 1 + +def trace_end(): + global _dropped + for state in _streams.values(): + cur = state["cur"] + if cur is not None: + _dropped += cur[1] - len(cur[2]) + print("summary: records=%d dropped=%d stray=%d unknown=%d errors=%d other=%d" + % (_records, _dropped, _stray, _unknown, _errors, _other_count)) + _streams.clear() + +def main(): + args = sys.argv[1:] + wordfile = None + eid_override = None + types_override = None + i = 0 + while i < len(args): + a = args[i] + if a == "--words": + if i + 1 < len(args) and not args[i + 1].startswith("--"): + wordfile = args[i + 1] + i += 1 + elif a == "--event-id": + if i + 1 >= len(args) or args[i + 1].startswith("--"): + print("--event-id requires a value", file=sys.stderr) + return 2 + try: + eid_override = int(args[i + 1], 0) + except ValueError: + print("--event-id requires an integer", file=sys.stderr) + return 2 + i += 1 + elif a == "--types": + if i + 1 >= len(args) or args[i + 1].startswith("--"): + print("--types requires a value", file=sys.stderr) + return 2 + types_override = args[i + 1].split(",") + i += 1 + i += 1 + + events = dict(_events) + if eid_override is not None and types_override is not None: + events[eid_override] = ("override", + [(None, t) for t in types_override]) + + words = [] + if wordfile: + with open(wordfile) as f: + for line in f: + line = line.strip() + if line.startswith(("0x", "0X")) or line.isdigit(): + words.append(int(line, 16)) + else: + for line in sys.stdin: + line = line.strip() + if line.startswith(("0x", "0X")) or line.isdigit(): + words.append(int(line, 16)) + + records, dropped, stray, unknown, lines = decode_words(words, events) + for _, text in lines: + print(text) + print("summary: records=%d dropped=%d stray=%d unknown=%d" + % (records, dropped, stray, unknown)) + return 0 if (records > 0 and dropped == 0 and unknown == 0) else 1 + +if "--words" in sys.argv: + sys.exit(main())
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [RFC v1 18/19] ptwrite uprobes / perf tools script: Add ptwrite uprobes decoder
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The script queries the local host's `/sys/kernel/tracing` to map dynamic event IDs, which…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The script queries the local host's /sys/kernel/tracing to map dynamic event IDs, which silently breaks offline and cross-platform analysis of perf.data files. --
commit cc31146d83897ea76d8b39eb7c2282185b7e63ac
Author: Andi Kleen <[email protected]>
ptwrite uprobes / perf tools script: Add ptwrite uprobes decoder
Add a decoder for the PTWRITE records generated by ptwrite uprobes. This runs as a python script in perf script.
Quoted 6 lines
tools/perf/scripts/python/uprobe-ptwrite-decode.py new file +0
-0@@ -0,0 +1,416 @@
[ ... ]
Quoted 1 line
+TRACEFS = "
/sys/kernel/tracing"
[ ... ]
tools/perf/scripts/python/uprobe-ptwrite-decode.py line 43
Quoted 5 lines
tools/perf/scripts/python/uprobe-ptwrite-decode.py new file +5
-0+def load_events(root=TRACEFS): + """Scan tracefs for (event_id -> (name, [(arg name, type)])).""" + events = {} + try: + groups = os.listdir(root + "/events")
[Severity: Medium]
Does coupling the event decoding logic to the host machine's live tracefs state break offline cross-platform analysis of perf.data files?
Trace event IDs are dynamically assigned by the kernel and can differ across machines or reboots. When a perf.data file is analyzed offline on a different machine, the embedded ptwrite payload event IDs (which were generated on the recording machine) will be incorrectly mapped to whatever unrelated events happen to share those IDs in the local host's /sys/kernel/tracing. This can lead to garbage decoding output.
Could the script be modified to extract event definitions from the perf.data header instead of silently defaulting to the host's tracefs?
Quoted 6 lines
tools/perf/scripts/python/uprobe-ptwrite-decode.py new file +6
-0+ except OSError: + return events + for g in groups: + gdir = root + "/events/" + g + if not os.path.isdir(gdir): + continue
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=18
RFC 19/19 ptwrite uprobes: Add self tests
In reply to Andi KleenRegression test various cases with ptwrite uprobes. It uses both the main probes and the perf decoder.
--- tools/testing/selftests/Makefile | 1 + .../test.d/kprobe/uprobe_syntax_errors.tc | 44 ++++ tools/testing/selftests/uprobes/Makefile | 15 ++ tools/testing/selftests/uprobes/ptw_probe.c | 156 ++++++++++++ tools/testing/selftests/uprobes/run_decode.sh | 177 ++++++++++++++ tools/testing/selftests/uprobes/run_ptw.sh | 226 ++++++++++++++++++ 6 files changed, 619 insertions(+) create mode 100644 tools/testing/selftests/uprobes/Makefile create mode 100644 tools/testing/selftests/uprobes/ptw_probe.c create mode 100755 tools/testing/selftests/uprobes/run_decode.sh create mode 100755 tools/testing/selftests/uprobes/run_ptw.sh
tools/testing/selftests/Makefile +1 -0
@@ -135,6 +135,7 @@ TARGETS += tpm2 TARGETS += tty TARGETS += ublk TARGETS += uevent +TARGETS += uprobes TARGETS += user_events TARGETS += vDSO TARGETS += mm
tools/testing/selftests/ftrace/test.d/kprobe/uprobe_syntax_errors.tc +44 -0
48 lines not on this page · raw message
tools/testing/selftests/uprobes/Makefile new file +15 -0
@@ -0,0 +1,15 @@ +# SPDX-License-Identifier: GPL-2.0 +# ptwrite uprobe selftests (x86-64). +ARCH ?= $(shell uname -m 2>/dev/null || echo not) +CFLAGS += -O2 -Wall -no-pie + +TEST_GEN_FILES := ptw_probe +TEST_PROGS := run_ptw.sh run_module.sh run_perfprobe.sh run_decode.sh + +ifneq ($(filter x86 x86_64,$(ARCH)),) +TEST_GEN_FILES := ptw_probe +else +TEST_GEN_FILES := +endif + +include ../lib.mk
tools/testing/selftests/uprobes/ptw_probe.c new file +156 -0
156 lines not on this page · raw message
tools/testing/selftests/uprobes/run_decode.sh new file +177 -0
177 lines not on this page · raw message
tools/testing/selftests/uprobes/run_ptw.sh new file +226 -0
226 lines not on this page · raw message
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [RFC v1 19/19] ptwrite uprobes: Add self tests
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Missing test scripts and source files cause selftests build failures and unconditional skips. --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Missing test scripts and source files cause selftests build failures and unconditional skips. --
commit cdbf7a2bf57258522e6b6951a74593c1f59e7dd4
Author: Andi Kleen <[email protected]>
ptwrite uprobes: Add self tests
This commit adds selftests for ptwrite uprobes. It tests various cases using both the main probes and the perf decoder.
tools/testing/selftests/uprobes/Makefile line 7
Quoted 13 lines
tools/testing/selftests/uprobes/Makefile new file +7
-0@@ -0,0 +1,15 @@ +# SPDX-License-Identifier: GPL-2.0 +# ptwrite uprobe selftests (x86-64). +ARCH ?= $(shell uname -m 2>/dev/null || echo not) +CFLAGS += -O2 -Wall -no-pie + +TEST_GEN_FILES := ptw_probe +TEST_PROGS := run_ptw.sh run_module.sh run_perfprobe.sh run_decode.sh
[Severity: Medium]
Will this break the selftests build during a make install? The Makefile lists run_module.sh and run_perfprobe.sh in TEST_PROGS, but these files do not appear to be included in this patch.
[ ... ]
Quoted 6 lines
tools/testing/selftests/uprobes/run_decode.sh new file +0
-0
tools/testing/selftests/uprobes/run_decode.sh line 8
Quoted 8 lines
tools/testing/selftests/uprobes/run_decode.sh new file +8
-0@@ -0,0 +1,177 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# run_decode.sh - ptwrite decoder selftests. +# Exercises the decoder CLI and PT/perf integration. +# Root + tracefs + perf + gcc + a PTWRITE-capable CPU required. +set -u +DIR=$(dirname "$(readlink -f "$0")") +SRC=${1:-"$DIR/manual_ptw.c"}
[Severity: Medium]
Does this cause the test to unconditionally skip? The SRC variable defaults to manual_ptw.c, but this source file seems to be missing from the commit. If the file is added in a subsequent version, should it also be included in TEST_FILES in the Makefile to ensure it is installed correctly during make install?
[ ... ]
tools/testing/selftests/uprobes/run_decode.sh line 58
Quoted 7 lines
tools/testing/selftests/uprobes/run_decode.sh new file +7
-0+if ! gcc -O2 -no-pie -o "$BIN" "$SRC" 2>/dev/null; then + echo "1..0 # SKIP test program build failed" + exit 0 +fi + +TV=$(objdump -d "$BIN" 2>/dev/null | + awk '/^[0-9a-f]+ <target>:/{print $1;exit}' | tr -d ':')
[Severity: Medium]
Is manual_ptw.c missing from the commit? This logic expects the compiled test binary to contain a target function, which further suggests that the omitted C file is strictly required for this test to function.
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=19