Kprobes: User space probes support
cover Kprobes: User space probes support
This patch set provides the basic infrastructure for user-space probes based on IBM's Dprobes. Similar to kprobes, user-space probes uses the breakpoint mechanism to insert probes. User has to write a kernel module to insert probes in the executable/library and specify the handlers in the kernel module. Using this mechanism user can not only log user-space data structure present in the memory when probe was hit, but also log kernel data structures, stack traces, system registers etc.
User-space tools like systemtap can use this interface to probe applications and libraries.
Design and development of this user-space probe mechanism has been discussed in systemtap mailing lists.
a) Features supported:
1. Probes on application executable.
2. Probes on libraries.
3. Probes are visible across fork().
4. Probes can be inserted even on executables/libraries that are not even started running(pages not present in memory).
5. Multiple handlers can be inserted/removed for each probe.
b) Interfaces:
1. register_uprobe(struct uprobe *uprobe) : accepts a pointer to uprobe. User has to allocate the uprobes structure and initialize following elements:
pathname - points to the application's pathname offset - offset of the probe from the file beginning;
[It's still the case that the user has to specify the offset as well
as the address (see TODO list)]
In case of library calls, the offset is the relative offset from the beginning of the of the mapped library.
kp.addr - virtual address within the executable.
kp.pre_handler - handler to be executed when probe is fired.
kp.post_handler - handler to be executed after single stepping
the original instruction.
kp.fault_handler- handler to be executed if fault occurs while
executing the original instruction or the handlers.
As with a kprobe, the user should not modify the uprobe while it is registered. This routine returns zero on successful registeration.
2. unregister_uprobe(struct uprobe *uprobe) : accepts a pointer to uprobe.
c) Objects:
struct uprobe - Allocated per probe by the user.
struct uprobe_module - Allocated per application by the userspace probe
mechanism.
struct uprobe {
/* pointer to the pathname of the application */
char *pathname;
/* kprobe structure with user specified handlers */
struct kprobe kp;
/* hlist of all the userspace probes per application */
struct hlist_node ulist;
/* inode of the probed application */
struct inode *inode;
/* probe offset within the file */
unsigned long offset;};
struct uprobe_module {
/* hlist head of all userspace probes per application */
struct hlist_head ulist_head;
/* list of all uprobe_module for probed application */
struct list_head mlist;
/* to hold path/dentry etc. */
struct nameidata nd;
/* original readpage operations */
struct address_space_operations *ori_a_ops;
/* readpage hooks added operations */
struct address_space_operations user_a_ops;};
d) Usage:
/* Allocate a uprobe structure */
struct uprobe p;
/* Define pre handler */
int handler_pre(struct kprobe *p, struct pt_regs *regs)
{
.............collect useful data..............
return 0;
}
void handler_post(struct kprobe *p, struct pt_regs *regs,
unsigned long flags)
{
.............collect useful data..............
}
int handler_fault(struct kprobe *p, struct pt_regs *regs, int trapnr)
{
........ release allocated resources & try to recover ....
return 0;
}
Before inserting the probe, specify the pathname of the application
on which the probe is to be inserted.
/*pointer to the pathname of the application */
p.pathname = "/home/prasanna/bin/myapp";
p.kp.pre_handler=handler_pre;
p.kp.post_handler=handler_post;
p.kp.fault_handler=handler_fault;
/* Specify the probe address */
/* $nm appln |grep func1 */
p.kp.addr = (kprobe_opcode_t *)0x080484d4;
/* Specify the offset within the application/executable*/
p.offset = (unsigned long)0x4d4;
/* Now register the userspace probe */
if (ret = register_uprobe(&p))
printk("register_uprobe: unsuccessful ret= %d\n", ret);
/* To unregister the registered probed, just call..*/
unregister_uprobe(&p);e) TODO List:
1. Execution of probe handlers are serialized using a uprobe_lock, need to make them scalable.
2. Provide jprobes type mechansim to allow the handlers to run in user mode.
3. Insert probes on copy-on-write pages. Tracks all COW pages for the page containing the specified probe point and inserts/removes all the probe points for that page.
4. Optimize the insertion of probes through readpage hooks. Identify all the probes to be inserted on the read page and insert them at once.
5. A wrapper routine to calculate the offset from the probed file beginning. In case of dynamic shared library, the offset is calculated by subtracting the beginning address of the file mapped from the address of the probe point.
6. Probes are visible if a copy of the probed executable is made, remove probes from the new copied image.
7. Make user-space probes coexist with other debuggers like gdb etc.
8. Support probepoints within MAX_INSN_SIZE bytes of the end of a vm area. (Right now we just copy MAX_INSN_SIZE bytes and assume we won't read of the end of the vm area.) This would be useful for return probes, where we'd like to put the trampoline between mm->end_code and the end of that page, if possible. Really, all we need is 1 byte.
Signature · 5 lines
Prasanna S Panchamukhi Linux Technology Center India Software Labs, IBM Bangalore Email: [email protected] Ph: 91-80-51776329
Re: [1/3 PATCH] Kprobes: User space probes support- base interface
This patch provides two interfaces to insert and remove user space probes. Each probe is uniquely identified by inode and offset within that executable/library file. Insertion of a probe involves getting the code page for a given offset, mapping it into the memory and then insert the breakpoint at the given offset. Also the probe is added to the uprobe_table hash list. A uprobe_module data strcture is allocated for every probed application/library image on disk. Removal of a probe involves getting the code page for a given offset, mapping that page into the memory and then replacing the breakpoint instruction with a the original opcode. This patch also provides aggrigate probe handler feature, where user can define multiple handlers per probe.
arch/i386/kernel/uprobes.c | 71 +++++ fs/namei.c | 13 include/linux/kprobes.h | 51 +++ include/linux/namei.h | 1 kernel/Makefile | 2 kernel/kprobes.c | 3 kernel/uprobes.c | 589 +++++++++++++++++++++++++++++++++++++++++++++ 7 files changed, 725 insertions(+), 5 deletions(-)
include/linux/kprobes.h renamed from include/linux/kprobes.h~kprobes_userspace_probes-base-interface +51 -0
@@ -37,6 +37,10 @@#include <linux/spinlock.h> #include <linux/rcupdate.h> #include <linux/mutex.h> +#include <linux/mm.h> +#include <linux/dcache.h> +#include <linux/namei.h> +#include <linux/pagemap.h> #ifdef CONFIG_KPROBES #include <asm/kprobes.h>
@@ -54,6 +58,7 @@ struct kprobe;struct pt_regs ;struct kretprobe ;struct kretprobe_instance ; +extern struct uprobe *current_uprobe;typedef int (*kprobe_pre_handler_t ) (struct kprobe *,struct pt_regs *);typedef int (*kprobe_break_handler_t ) (struct kprobe *,struct pt_regs *); typedef void (*kprobe_post_handler_t) (struct kprobe *, struct pt_regs *,
@@ -117,6 +122,32 @@ struct jprobe { DECLARE_PER_CPU(struct kprobe *, current_kprobe); DECLARE_PER_CPU(struct kprobe_ctlblk, kprobe_ctlblk); +struct uprobe { +/* pointer to the pathname of the application */ +char *pathname; +/* kprobe structure with user specified handlers */ +struct kprobe kp; +/* hlist of all the userspace probes per application */ +struct hlist_node ulist; +/* inode of the probed application */ +struct inode *inode; +/* probe offset within the file */ +unsigned long offset; +}; + +struct uprobe_module { +/* hlist head of all userspace probes per application */ +struct hlist_head ulist_head; +/* list of all uprobe_module for probed application */ +struct list_head mlist; +/* to hold path/dentry etc. */ +struct nameidata nd; +/* original readpage operations */ +struct address_space_operations *ori_a_ops; +/* readpage hooks added operations */ +struct address_space_operations user_a_ops; +}; +#ifdef ARCH_SUPPORTS_KRETPROBES extern void arch_prepare_kretprobe (struct kretprobe *rp,struct pt_regs *regs);#else /* ARCH_SUPPORTS_KRETPROBES */
@@ -162,9 +193,16 @@ extern void show_registers(struct pt_regextern kprobe_opcode_t *get_insn_slot (void );extern void free_insn_slot (kprobe_opcode_t *slot);extern void kprobes_inc_nmissed_count (struct kprobe *p); +extern void copy_kprobe (struct kprobe *old_p,struct kprobe *p); +extern int arch_copy_uprobe (struct kprobe *p,kprobe_opcode_t *address); +extern void arch_arm_uprobe (kprobe_opcode_t *address); +extern void arch_disarm_uprobe (struct kprobe *p,kprobe_opcode_t *address); +extern void init_uprobes (void );/* Get the kprobe at this addr (if any) - called with preemption disabled */ struct kprobe *get_kprobe (void *addr); +struct kprobe *get_uprobe (void *addr); +extern int arch_alloc_insn (struct kprobe *p);struct hlist_head *kretprobe_inst_table_head (struct task_struct *tsk);/* kprobe_running() will just return the current_kprobe on this CPU */
@@ -183,6 +221,16 @@ static inline struct kprobe_ctlblk *get_return (&__get_cpu_var (kprobe_ctlblk)); } +static inline void set_uprobe_instance (struct kprobe *p) +{ + current_uprobe = container_of(p, struct uprobe, kp); +} + +static inline void reset_uprobe_instance (void ) +{ + current_uprobe =NULL ; +} +int register_kprobe (struct kprobe *p);void unregister_kprobe (struct kprobe *p);int setjmp_pre_handler (struct kprobe *,struct pt_regs *);
@@ -194,6 +242,9 @@ void jprobe_return(void);int register_kretprobe (struct kretprobe *rp);void unregister_kretprobe (struct kretprobe *rp); +int register_uprobe (struct uprobe *uprobe); +void unregister_uprobe (struct uprobe *uprobe); +struct kretprobe_instance *get_free_rp_inst (struct kretprobe *rp);void add_rp_inst (struct kretprobe_instance *ri);void kprobe_flush_task (struct task_struct *tk);
kernel/uprobes.c new file +589 -0
589 lines not on this page · raw message
arch/i386/kernel/uprobes.c new file +71 -0
71 lines not on this page · raw message
kernel/kprobes.c renamed from kernel/kprobes.c~kprobes_userspace_probes-base-interface +2 -1
@@ -356,7 +356,7 @@ static inline void free_rp_inst(struct k/* * Keep all fields in the kprobe consistent */ -static inline void copy_kprobe (struct kprobe *old_p,struct kprobe *p) +void copy_kprobe (struct kprobe *old_p,struct kprobe *p) {memcpy (&p->opcode, &old_p->opcode,sizeof (kprobe_opcode_t));memcpy (&p->ainsn, &old_p->ainsn,sizeof (struct arch_specific_insn ));
@@ -650,6 +650,7 @@ static int __init init_kprobes(void)INIT_HLIST_HEAD (&kretprobe_inst_table[i]); } +init_uprobes (); err =arch_init_kprobes ();if (!err) err =register_die_notifier (&kprobe_exceptions_nb);
include/linux/namei.h renamed from include/linux/namei.h~kprobes_userspace_probes-base-interface +1 -0
@@ -81,6 +81,7 @@ extern int follow_up(struct vfsmount **,extern struct dentry *lock_rename (struct dentry *,struct dentry *);extern void unlock_rename (struct dentry *,struct dentry *); +extern int deny_write_access_to_inode (struct inode *inode);static inline void nd_set_link (struct nameidata *nd,char *path) {
fs/namei.c renamed from fs/namei.c~kprobes_userspace_probes-base-interface +10 -3
@@ -322,10 +322,9 @@ int get_write_access(struct inode * inodreturn 0 ; } -int deny_write_access (struct file * file) +/* This routine increments the writecount for a given inode. */ +int deny_write_access_to_inode (struct inode *inode) { -struct inode *inode = file->f_dentry->d_inode; -spin_lock (&inode->i_lock);if (atomic_read (&inode->i_writecount) >0 ) {spin_unlock (&inode->i_lock);
@@ -337,6 +336,14 @@ int deny_write_access(struct file * filereturn 0 ; } +/* Wrapper routine that increments the writecount for a given file pointer. */ +int deny_write_access (struct file * file) +{ +struct inode *inode = file->f_dentry->d_inode; + +return deny_write_access_to_inode (inode); +} +void path_release (struct nameidata *nd) {dput (nd->dentry);
kernel/Makefile renamed from kernel/Makefile~kprobes_userspace_probes-base-interface +1 -1
@@ -31,7 +31,7 @@ obj-$(CONFIG_IKCONFIG) += configs.o obj-$(CONFIG_STOP_MACHINE) += stop_machine.o obj-$(CONFIG_AUDIT) += audit.o auditfilter.o obj-$(CONFIG_AUDITSYSCALL) += auditsc.o -obj-$(CONFIG_KPROBES) += kprobes.o +obj-$(CONFIG_KPROBES) += kprobes.o uprobes.o obj-$(CONFIG_SYSFS) += ksysfs.o obj-$(CONFIG_DETECT_SOFTLOCKUP) += softlockup.o obj-$(CONFIG_GENERIC_HARDIRQS) += irq/
_
Signature · 5 lines
Prasanna S Panchamukhi Linux Technology Center India Software Labs, IBM Bangalore Email: [email protected] Ph: 91-80-51776329
Re: [2/3 PATCH] Kprobes: User space probes support- readpage hooks
This patch provides the feature of inserting probes on pages that are not present in the memory during registration.
To add readpage and readpages() hooks, two new elements are added to the uprobe_module object:
struct address_space_operations *ori_a_ops; struct address_space_operations user_a_ops;
User-space probes allows probes to be inserted even in pages that are not present in the memory at the time of registration. This is done by adding hooks to the readpage and readpages routines. During registration, the address space operation object is modified by substituting user-space probes's specific readpage and readpages routines. When the pages are read into memory through the readpage and readpages address space operations, any associated probes are automatically inserted into those pages. These user-space probes readpage and readpages routines internally call the original readpage() and readpages() routines, and then check whether probes are to be added to these pages, inserting probes as necessary. The overhead of adding these hooks is limited to the application on which the probes are inserted.
During unregistration, care should be taken to replace the readpage and readpages hooks with the original routines if no probes remain on that application.
kernel/uprobes.c | 121 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 121 insertions(+)
kernel/uprobes.c renamed from kernel/uprobes.c~kprobes_userspace_probes-hook-readpage +121 -0
@@ -302,6 +302,115 @@ static struct uprobe_module __kprobes *greturn NULL ; } +static inline void insert_readpage_uprobe (struct page *page, +struct address_space *mapping,struct uprobe *uprobe) +{ +unsigned long page_start = page->index <<PAGE_CACHE_SHIFT ; +unsigned long page_end = page_start +PAGE_SIZE ; + +if ((uprobe->offset >= page_start) && (uprobe->offset < page_end)) { +map_uprobe_page (page, uprobe, insert_kprobe_user); +flush_vma (mapping, page, uprobe); + } +} + +/** +* This function hooks the readpages() of all modules that have active +* probes on them. The original readpages() is called for the given +* inode/address_space to actually read the pages into the memory. +* Then all probes that are specified on these pages are inserted. +*/ +static int __kprobes uprobe_readpages (struct file *file, +struct address_space *mapping, +struct list_head *pages,unsigned nr_pages) +{ +int retval =0 ; +struct page *page; +struct uprobe_module *umodule; +struct uprobe *uprobe =NULL ; +struct hlist_node *node; + +mutex_lock (&uprobe_mutex); + + umodule =get_module_by_inode (file->f_dentry->d_inode); +if (!umodule) { +/* +* No module associated with this file, call the +* original readpages(). +*/ + retval = mapping->a_ops->readpages (file, mapping, + pages, nr_pages); + goto out; + } + +/* call original readpages() */ + retval = umodule->ori_a_ops->readpages (file, mapping, pages, nr_pages); +if (retval <0 ) + goto out; + +/* +* TODO: Walk through readpages page list and get +* pages with probes instead of find_get_page(). +*/ +hlist_for_each_entry (uprobe, node, &umodule->ulist_head, ulist) { + page =find_get_page (mapping, + uprobe->offset >>PAGE_CACHE_SHIFT ); +if (!page) +continue ; + +if (!uprobe->kp.opcode) +insert_readpage_uprobe (page, mapping, uprobe); +page_cache_release (page); + } + +out: +mutex_unlock (&uprobe_mutex); + +return retval; +} + +/** +* This function hooks the readpage() of all modules that have active +* probes on them. The original readpage() is called for the given +* inode/address_space to actually read the pages into the memory. +* Then all probes that are specified on this page are inserted. +*/ +int __kprobes uprobe_readpage (struct file *file,struct page *page) +{ +int retval =0 ; +struct uprobe_module *umodule; +struct uprobe *uprobe =NULL ; +struct hlist_node *node; +struct address_space *mapping = file->f_dentry->d_inode->i_mapping; + +mutex_lock (&uprobe_mutex); + + umodule =get_module_by_inode (file->f_dentry->d_inode); +if (!umodule) { +/* +* No module associated with this file, call the +* original readpage(). +*/ + retval = mapping->a_ops->readpage (file, page); + goto out; + } + +/* call original readpage() */ + retval = umodule->ori_a_ops->readpage (file, page); +if (retval <0 ) + goto out; + +hlist_for_each_entry (uprobe, node, &umodule->ulist_head, ulist) { +if (!uprobe->kp.opcode) +insert_readpage_uprobe (page, mapping, uprobe); + } + +out: +mutex_unlock (&uprobe_mutex); + +return retval; +} +/** * Gets exclusive write access to the given inode to ensure that the file * on which probes are currently applied does not change. Use the function,
@@ -324,13 +433,23 @@ static inline int ex_write_unlock(struct/** * Add uprobe and uprobe_module to the appropriate hash list. +* Also swithces i_op to hooks into readpage and readpages(). */ static void __kprobes get_inode_ops (struct uprobe *uprobe,struct uprobe_module *umodule) { +struct address_space *as; +INIT_HLIST_HEAD (&umodule->ulist_head);hlist_add_head (&uprobe->ulist, &umodule->ulist_head);list_add (&umodule->mlist, &uprobe_module_list); + as = umodule->nd.dentry->d_inode->i_mapping; + umodule->ori_a_ops = as->a_ops; + umodule->user_a_ops = *as->a_ops; + umodule->user_a_ops.readpage = uprobe_readpage; + umodule->user_a_ops.readpages = uprobe_readpages; + as->a_ops = &umodule->user_a_ops; + }/*
@@ -459,6 +578,8 @@ void __kprobes unregister_uprobe(structhlist_del (&uprobe->ulist);if (hlist_empty (&umodule->ulist_head)) {list_del (&umodule->mlist); + umodule->nd.dentry->d_inode->i_mapping->a_ops = + umodule->ori_a_ops;ex_write_unlock (uprobe->inode);path_release (&umodule->nd);kfree (umodule);
_
Signature · 5 lines
Prasanna S Panchamukhi Linux Technology Center India Software Labs, IBM Bangalore Email: [email protected] Ph: 91-80-51776329
Re: [3/3 PATCH] Kprobes: User space probes support- single stepping out-of-line
This patch provides a mechanism for probe handling and executing the user-specified handlers.
Each userspace probe is uniquely identified by the combination of inode and offset, hence during registeration the inode and offset combination is added to uprobes hash table. Initially when breakpoint instruction is hit, the uprobes hash table is looked up for matching inode and offset. The pre_handlers are called in sequence if multiple probes are registered. Similar to kprobes, uprobes also adopts to single step out-of-line, so that probe miss in SMP environment can be avoided. But for userspace probes, instruction copied into kernel address space cannot be single stepped, hence the instruction must be copied to user address space. The solution is to find free space in the current process address space and then copy the original instruction and single step that instruction.
User processes use stack space to store local variables, agruments and return values. Normally the stack space either below or above the stack pointer indicates the free stack space.
The instruction to be single stepped can modify the stack space, hence before using the free stack space, sufficient stack space should be left. The instruction is copied to the bottom of the page and check is made such that the copied instruction does not cross the page boundry. The copied instruction is then single stepped. Several architectures does not allow the instruction to be executed from the stack location, since no-exec bit is set for the stack pages. In those architectures, the page table entry corresponding to the stack page is identified and the no-exec bit is unset making the instruction on that stack page to be executed.
There are situations where even the free stack space is not enough for the user instruction to be copied and single stepped. In such situations, the virtual memory area(vma) can be expanded beyond the current stack vma. This expaneded stack can be used to copy the original instruction and single step out-of-line.
Even if the vma cannot be extended then the instruction much be executed inline, by replacing the breakpoint instruction with original instruction.
arch/i386/kernel/Makefile | 2 arch/i386/kernel/kprobes.c | 4 arch/i386/kernel/uprobes.c | 537 +++++++++++++++++++++++++++++++++++++++++++++ arch/i386/mm/fault.c | 3 include/asm-i386/kprobes.h | 20 + include/linux/kprobes.h | 8 6 files changed, 569 insertions(+), 5 deletions(-)
include/asm-i386/kprobes.h renamed from include/asm-i386/kprobes.h~kprobes_userspace_probes-ss-out-of-line +20 -0
@@ -26,6 +26,7 @@*/ #include <linux/types.h> #include <linux/ptrace.h> +#include <asm/cacheflush.h> #define __ARCH_WANT_KPROBES_INSN_SLOT
@@ -77,6 +78,19 @@ struct kprobe_ctlblk {struct prev_kprobe prev_kprobe; }; +/* per user probe control block */ +struct uprobe_ctlblk { +unsigned long uprobe_status; +unsigned long uprobe_saved_eflags; +unsigned long uprobe_old_eflags; +unsigned long singlestep_addr; +unsigned long flags; +struct kprobe *curr_p; +pte_t *upte; +struct page *upage; +struct task_struct *tsk; +}; +/* trap3/1 are intr gates for kprobes. So, restore the status of IF, * if necessary, before executing the original int3/1 (trap) handler. */
@@ -88,4 +102,10 @@ static inline void restore_interrupts(stextern int kprobe_exceptions_notify (struct notifier_block *self,unsigned long val,void *data); +extern int uprobe_exceptions_notify (struct notifier_block *self, +unsigned long val,void *data); +extern unsigned long get_segment_eip (struct pt_regs *regs, +unsigned long *eip_limit); +extern int is_IF_modifier (kprobe_opcode_t opcode); +#endif /* _ASM_KPROBES_H */
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +537 -0
546 lines not on this page · raw message
include/linux/kprobes.h renamed from include/linux/kprobes.h~kprobes_userspace_probes-ss-out-of-line +8 -0
@@ -51,6 +51,13 @@#define KPROBE_REENTER 0x00000004#define KPROBE_HIT_SSDONE 0x00000008 +/* uprobe_status settings */ +#define UPROBE_HIT_ACTIVE 0x00000001 +#define UPROBE_HIT_SS 0x00000002 +#define UPROBE_HIT_SSDONE 0x00000004 +#define UPROBE_SS_INLINE 0x00000008 +#define UPROBE_SSDONE_INLINE 0x00000010 +/* Attach to insert probes on any functions which should be ignored*/ #define __kprobes __attribute__((__section__(".kprobes.text")))
@@ -183,6 +190,7 @@ struct kretprobe_instance {struct task_struct *task; }; +extern spinlock_t uprobe_lock;extern spinlock_t kretprobe_lock;extern struct mutex kprobe_mutex;extern int arch_prepare_kprobe (struct kprobe *p);
arch/i386/kernel/kprobes.c renamed from arch/i386/kernel/kprobes.c~kprobes_userspace_probes-ss-out-of-line +2 -2
@@ -88,7 +88,7 @@ static inline int can_boost(kprobe_opcod/* * returns non-zero if opcode modifies the interrupt flag. */ -static inline int is_IF_modifier (kprobe_opcode_t opcode) +int is_IF_modifier (kprobe_opcode_t opcode) {switch (opcode) {case 0xfa :/* cli */
@@ -610,7 +610,7 @@ int __kprobes kprobe_exceptions_notify(sint ret =NOTIFY_DONE ;if (args->regs &&user_mode (args->regs)) -return ret; +return uprobe_exceptions_notify (self, val, data);switch (val) {case DIE_INT3 :
arch/i386/mm/fault.c renamed from arch/i386/mm/fault.c~kprobes_userspace_probes-ss-out-of-line +1 -2
@@ -71,8 +71,7 @@ void bust_spinlocks(int yes)* * This is slow, but is very rarely executed. */ -static inline unsigned long get_segment_eip (struct pt_regs *regs, -unsigned long *eip_limit) +unsigned long get_segment_eip (struct pt_regs *regs,unsigned long *eip_limit) {unsigned long eip = regs->eip;unsigned seg = regs->xcs &0xffff ;
arch/i386/kernel/Makefile renamed from arch/i386/kernel/Makefile~kprobes_userspace_probes-ss-out-of-line +1 -1
@@ -29,7 +29,7 @@ obj-$(CONFIG_KEXEC) += machine_kexec.o obj-$(CONFIG_CRASH_DUMP) += crash_dump.o obj-$(CONFIG_X86_NUMAQ) += numaq.o obj-$(CONFIG_X86_SUMMIT_NUMA) += summit.o -obj-$(CONFIG_KPROBES) += kprobes.o +obj-$(CONFIG_KPROBES) += kprobes.o uprobes.o obj-$(CONFIG_MODULES) += module.o obj-y += sysenter.o vsyscall.o obj-$(CONFIG_ACPI_SRAT) += srat.o
_
Signature · 5 lines
Prasanna S Panchamukhi Linux Technology Center India Software Labs, IBM Bangalore Email: [email protected] Ph: 91-80-51776329
Re: [1/3 PATCH] Kprobes: User space probes support- base interface
In reply to Prasanna S PanchamukhiQuoted 26 lines · Prasanna S Panchamukhi
This patch provides two interfaces to insert and remove user space probes. Each probe is uniquely identified by inode and offset within that executable/library file. Insertion of a probe involves getting the code page for a given offset, mapping it into the memory and then insert the breakpoint at the given offset. Also the probe is added to the
uprobe_tablehash list. Auprobe_moduledata strcture is allocated for every probed application/library image on disk. Removal of a probe involves getting the code page for a given offset, mapping that page into the memory and then replacing the breakpoint instruction with a the original opcode. This patch also provides aggrigate probe handler feature, where user can define multiple handlers per probe.kernel/uprobes.c new file +11
-0+/** +* Finds a uprobe at the specified user-space address in the current task. +* Points current_uprobe at that uprobe and returns the corresponding kprobe. +*/ +struct kprobe __kprobes *get_uprobe (void *addr) +{ +struct mm_struct *mm = current->mm; +struct vm_area_struct *vma; +struct inode *inode; +unsigned long offset; +struct kprobe *p, *kpr;
kernel/uprobes.c line 143
Quoted 10 lines
kernel/uprobes.c new file +10
-0+struct uprobe *uprobe; + + vma =find_vma (mm, (unsigned long )addr); + +BUG_ON (!vma);/* this should not happen, not in our memory map */ + + offset = (unsigned long )addr - (vma->vm_start + + (vma->vm_pgoff <<PAGE_SHIFT )); +if (!vma->vm_file) +return NULL ;
All this appears to be happening without mmap_sem held?
Quoted 9 lines
kernel/uprobes.c new file +9
-0+/** +* Wait for the page to be unlocked if someone else had locked it, +* then map the page and insert or remove the breakpoint. +*/ +static int __kprobes map_uprobe_page (struct page *page,struct uprobe *uprobe, +process_uprobe_func_t process_kprobe_user) +{ +int ret =0 ; +kprobe_opcode_t *uprobe_address;
kernel/uprobes.c line 248
Quoted 10 lines
kernel/uprobes.c new file +10
-0+ +if (!page) +return -EINVAL ;/* TODO: more suitable errno */ + +wait_on_page_locked (page); +/* could probably retry readpage here. */ +if (!PageUptodate (page)) +return -EINVAL ;/* TODO: more suitable errno */ + +lock_page (page);
That's a lot of fuss and might be racy with truncate.
Why not just run lock_page()?
kernel/uprobes.c line 254
Quoted 5 lines
kernel/uprobes.c new file +5
-0+ uprobe_address = (kprobe_opcode_t *)kmap (page); + uprobe_address = (kprobe_opcode_t *)((unsigned long )uprobe_address + + (uprobe->offset & ~PAGE_MASK )); + ret = (*process_kprobe_user)(uprobe, uprobe_address); +kunmap (page);
kmap_atomic() is preferred.
kernel/uprobes.c line 313
Quoted 9 lines
kernel/uprobes.c new file +9
-0+/** +* Gets exclusive write access to the given inode to ensure that the file +* on which probes are currently applied does not change. Use the function, +* deny_write_access_to_inode() we added in fs/namei.c. +*/ +static inline int ex_write_lock (struct inode *inode) +{ +return deny_write_access_to_inode (inode); +}
hm, this code likes to poke around in VFS internals. It would be nice to have an overall description of what it's trying to do, why and how.
Quoted 41 lines
kernel/uprobes.c new file +41
-0+/** +* unregister_uprobe: Disarms the probe, removes the uprobe +* pointers from the hash list and unhooks readpage routines. +*/ +void __kprobes unregister_uprobe (struct uprobe *uprobe) +{ +struct address_space *mapping; +struct uprobe_module *umodule; +struct page *page; +unsigned long flags; +int ret =0 ; + +if (!uprobe->inode) +return ; + + mapping = uprobe->inode->i_mapping; + + page =find_get_page (mapping, uprobe->offset >>PAGE_CACHE_SHIFT ); + +spin_lock_irqsave (&uprobe_lock, flags); + ret =remove_uprobe (uprobe); +spin_unlock_irqrestore (&uprobe_lock, flags); + +mutex_lock (&uprobe_mutex); +if (!(umodule =get_module_by_inode (uprobe->inode))) + goto out; + +hlist_del (&uprobe->ulist); +if (hlist_empty (&umodule->ulist_head)) { +list_del (&umodule->mlist); +ex_write_unlock (uprobe->inode); +path_release (&umodule->nd); +kfree (umodule); + } + +out: +mutex_unlock (&uprobe_mutex); +if (ret) + ret =map_uprobe_page (page, uprobe, remove_kprobe_user); + +if (ret == -EINVAL )
kernel/uprobes.c line 482
Quoted 10 lines
kernel/uprobes.c new file +10
-0+return ; +/* +* TODO: unregister_uprobe should not fail, need to handle +* if it fails. +*/ +flush_vma (mapping, page, uprobe); + +if (page) +page_cache_release (page); +}
That's some pretty awkward coding. Buggy too. It doesn't drop the refcount on the page if map_uprobe_page() returned -EINVAL because it's assuming that EINVAL meant "there was no page". But there are multiple reasons for map_uprobe_page() to return -EINVAL. If that page isn't uptodate, we leak a ref.
This function should be doing the checking for a find_get_page() failure, not map_uprobe_page().
Re: [2/3 PATCH] Kprobes: User space probes support- readpage hooks
In reply to Prasanna S PanchamukhiQuoted 23 lines · Prasanna S Panchamukhi
This patch provides the feature of inserting probes on pages that are not present in the memory during registration.
To add readpage and
readpages()hooks, two new elements are added to theuprobe_moduleobject:struct address_space_operations *ori_a_ops; struct address_space_operations user_a_ops;User-space probes allows probes to be inserted even in pages that are not present in the memory at the time of registration. This is done by adding hooks to the readpage and readpages routines. During registration, the address space operation object is modified by substituting user-space probes's specific readpage and readpages routines. When the pages are read into memory through the readpage and readpages address space operations, any associated probes are automatically inserted into those pages. These user-space probes readpage and readpages routines internally call the original
readpage()andreadpages()routines, and then check whether probes are to be added to these pages, inserting probes as necessary. The overhead of adding these hooks is limited to the application on which the probes are inserted.
Quoted 4 lines
During unregistration, care should be taken to replace the readpage and readpages hooks with the original routines if no probes remain on that application.
So... The code's replacing the address_space's address_space_operations with a copy which has .readpage() and .readpages() modified, because it happens that filemap_nopage() uses those.
This is all rather hacky.
I think we need to step back and discuss what services this feature is trying to provide, and how it is to provide them. Your covering description didn't describe that - it dives straigt into details without even describing what the patches are trying to achieve.
So. What are we trying to achieve here, and how are we trying to achieve it? What problems were encountered in the development of the feature and how were they solved? What alternative solutions were there?
I can mostly work all that out from background knowledge and looking at the code, but I'd rather hear it from the designers please.
Re: [3/3 PATCH] Kprobes: User space probes support- single stepping out-of-line
In reply to Prasanna S PanchamukhiQuoted 62 lines · Prasanna S Panchamukhi
This patch provides a mechanism for probe handling and executing the user-specified handlers.
Each userspace probe is uniquely identified by the combination of inode and offset, hence during registeration the inode and offset combination is added to uprobes hash table. Initially when breakpoint instruction is hit, the uprobes hash table is looked up for matching inode and offset. The
pre_handlersare called in sequence if multiple probes are registered. Similar to kprobes, uprobes also adopts to single step out-of-line, so that probe miss in SMP environment can be avoided. But for userspace probes, instruction copied into kernel address space cannot be single stepped, hence the instruction must be copied to user address space. The solution is to find free space in the current process address space and then copy the original instruction and single step that instruction.User processes use stack space to store local variables, agruments and return values. Normally the stack space either below or above the stack pointer indicates the free stack space.
The instruction to be single stepped can modify the stack space, hence before using the free stack space, sufficient stack space should be left. The instruction is copied to the bottom of the page and check is made such that the copied instruction does not cross the page boundry. The copied instruction is then single stepped. Several architectures does not allow the instruction to be executed from the stack location, since no-exec bit is set for the stack pages. In those architectures, the page table entry corresponding to the stack page is identified and the no-exec bit is unset making the instruction on that stack page to be executed.
There are situations where even the free stack space is not enough for the user instruction to be copied and single stepped. In such situations, the virtual memory area(vma) can be expanded beyond the current stack vma. This expaneded stack can be used to copy the original instruction and single step out-of-line.
Even if the vma cannot be extended then the instruction much be executed inline, by replacing the breakpoint instruction with original instruction.
...
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +18
-0+ +/** +* This routines get the pte of the page containing the specified address. +*/ +static pte_t __kprobes *get_uprobe_pte (unsigned long address) +{ +pgd_t *pgd; +pud_t *pud; +pmd_t *pmd; +pte_t *pte =NULL ; + + pgd =pgd_offset (current->mm, address); +if (!pgd) + goto out; + + pud =pud_offset (pgd, address); +if (!pud) + goto out;
arch/i386/kernel/uprobes.c int __kprobes arch_copy_uprobe(struct kp · line 103
Quoted 10 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +10
-0+ + pmd =pmd_offset (pud, address); +if (!pmd) + goto out; + + pte =pte_alloc_map (current->mm, pmd, address); + +out: +return pte; +}
That's familiar looking code..
I guess this should be given a more generic name then placed in mm/memory.c, which is where we do pagetable walking.
Quoted 23 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +23
-0+/** +* This routine check for space in the current process's stack +* address space. If enough address space is found, copy the original +* instruction on that page for single stepping out-of-line. +*/ +static int __kprobes copy_insn_on_new_page (struct uprobe *uprobe , +struct pt_regs *regs,struct vm_area_struct *vma) +{ +unsigned long addr, stack_addr = regs->esp; +int size =MAX_INSN_SIZE *sizeof (kprobe_opcode_t); + +if (vma->vm_flags &VM_GROWSDOWN ) { +if (((stack_addr -sizeof (long long ))) < + (vma->vm_start + size)) +return -ENOMEM ; + addr = vma->vm_start; + }else if (vma->vm_flags &VM_GROWSUP ) { +if ((vma->vm_end - size) < (stack_addr +sizeof (long long ))) +return -ENOMEM ; + addr = vma->vm_end - size; + }else +return -EFAULT ; +
arch/i386/kernel/uprobes.c int __kprobes arch_copy_uprobe(struct kp · line 137
Quoted 10 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +10
-0+ vma->vm_flags |=VM_LOCKED ; + +if (__copy_to_user_inatomic ((unsigned long *)addr, + (unsigned long *)uprobe->kp.ainsn.insn, size)) +return -EFAULT ; + + regs->eip = addr; + +return 0 ; +}
If we're going to use __copy_to_user_inatomic() then we'll need some nice comments explaining why this is happening.
And we'll need to actually *be* in-atomic. That means we need an open-coded inc_preempt_count() and dec_preempt_count() in there and I don't see them.
Quoted 38 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +38
-0+/** +* This routine expands the stack beyond the present process address +* space and copies the instruction to that location, so that +* processor can single step out-of-line. +*/ +static int __kprobes copy_insn_onexpstack (struct uprobe *uprobe, +struct pt_regs *regs,struct vm_area_struct *vma) +{ +unsigned long addr, vm_addr; +int size =MAX_INSN_SIZE *sizeof (kprobe_opcode_t); +struct vm_area_struct *new_vma; +struct mm_struct *mm = current->mm; + + +if (!down_read_trylock (¤t->mm->mmap_sem)) +return -ENOMEM ; + +if (vma->vm_flags &VM_GROWSDOWN ) + vm_addr = vma->vm_start - size; +else if (vma->vm_flags &VM_GROWSUP ) + vm_addr = vma->vm_end + size; +else { +up_read (¤t->mm->mmap_sem); +return -EFAULT ; + } + + new_vma =find_extend_vma (mm, vm_addr); +if (!new_vma) { +up_read (¤t->mm->mmap_sem); +return -ENOMEM ; + } + +if (new_vma->vm_flags &VM_GROWSDOWN ) + addr = new_vma->vm_start; +else + addr = new_vma->vm_end - size; + + new_vma->vm_flags |=VM_LOCKED ;
arch/i386/kernel/uprobes.c int __kprobes arch_copy_uprobe(struct kp · line 186
Quoted 10 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +10
-0+up_read (¤t->mm->mmap_sem); + +if (__copy_to_user_inatomic ((unsigned long *)addr, + (unsigned long *)uprobe->kp.ainsn.insn, size)) +return -EFAULT ; + + regs->eip = addr; + +return 0 ; +}
Why is VM_LOCKED being set? (It needs a comment).
Where does it get unset?
arch/i386/kernel/uprobes.c int __kprobes arch_copy_uprobe(struct kp · line 225
Quoted 5 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +5
-0+ +if (__copy_to_user_inatomic ((unsigned long *)page_addr, + source, size)) +if (__copy_to_user_inatomic ( + (unsigned long *)(page_addr - size), source, size))
See above.
Quoted 7 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +7
-0+ +/** +* This routines get the page containing the probe, maps it and +* replaced the instruction at the probed address with specified +* opcode. +*/ +void __kprobes replace_original_insn(struct uprobe *uprobe,
arch/i386/kernel/uprobes.c int __kprobes arch_copy_uprobe(struct kp · line 250
Quoted 10 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +10
-0+ struct pt_regs *regs, kprobe_opcode_t opcode) +{ +kprobe_opcode_t *addr; +struct page *page; + + page =find_get_page (uprobe->inode->i_mapping, + uprobe->offset >>PAGE_CACHE_SHIFT ); +BUG_ON (!page); + +__lock_page (page);
Whoa. Why is __lock_page() being used here? It looks like a bug is being covered up.
arch/i386/kernel/uprobes.c int __kprobes arch_copy_uprobe(struct kp · line 256
Quoted 5 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +5
-0+ addr = (kprobe_opcode_t *)kmap_atomic (page,KM_USER1 ); + addr = (kprobe_opcode_t *)((unsigned long )addr + + (unsigned long )(uprobe->offset & ~PAGE_MASK )); + *addr = opcode; +/*TODO: flush vma ? */
flush_dcache_page() would be needed.
But then, what happens if the page is shared by other processes? Do they all start taking debug traps?
Quoted 13 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +13
-0+kunmap_atomic (addr,KM_USER1 ); + +unlock_page (page); + +if (page) +page_cache_release (page); + regs->eip = (unsigned long )uprobe->kp.addr; +} + +/** +* This routine provides the functionality of single stepping +* out-of-line. If single stepping out-of-line cannot be achieved, +* it replaces with the original instruction allowing it to single
arch/i386/kernel/uprobes.c int __kprobes arch_copy_uprobe(struct kp · line 279
Quoted 10 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +10
-0+* step inline. +*/ +static inline int prepare_singlestep_uprobe (struct uprobe *uprobe, +struct uprobe_ctlblk *ucb,struct pt_regs *regs) +{ +unsigned long stack_addr = regs->esp, flags; +struct vm_area_struct *vma =NULL ; +int err =0 ; + + vma =find_vma (current->mm, (stack_addr &PAGE_MASK ));
I don't think mmap_sem is held here?
Quoted 1 line
+static inline int
uprobe_fault_handler(struct pt_regs *regs, int trapnr)
If, for some reason, the compiler decides to not inline this function then you have a hunk of code which isn't marked __kprobes, but it should be.
I'd suggest that you remove all inlining from this code and add the appropriate section markers.
Or I guess you could use __always_inline, but I'm not sure that it's really worth the fuss and obscurity of doing that.
All kprobes-related code should be audited for this problem.
Re: [2/3 PATCH] Kprobes: User space probes support- readpage hooks
In reply to Prasanna S PanchamukhiQuoted 7 lines · Prasanna S Panchamukhi
+/**
+ * This function hooks the readpages() of all modules that have active + * probes on them. The original readpages() is called for the given + * inode/address_space to actually read the pages into the memory. + * Then all probes that are specified on these pages are inserted.+ */
The "/**" thing is designed to introduce a kerneldoc-style comment, but these comments aren't using kerneldoc.
Re: [3/3 PATCH] Kprobes: User space probes support- single stepping out-of-line
In reply to Andrew MortonQuoted 4 lines
And we'll need to actually *be* in-atomic. That means we need an open-coded
inc_preempt_count()anddec_preempt_count()in there and I don't see them.
..
Quoted 3 lines
Why is
VM_LOCKEDbeing set? (It needs a comment).Where does it get unset?
if this is an attempt to make the copy_in_atomic to be atomic, then it is a bug; the user can unset this bit after all via mprotect, even from another thread, and concurrently. UnGood(tm).
Re: [3/3 PATCH] Kprobes: User space probes support- single stepping out-of-line
In reply to Andrew MortonQuoted 34 lines · Andrew Morton
Quoted 18 lines · Prasanna S Panchamukhi
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +18
-0+ +/** +* This routines get the pte of the page containing the specified address. +*/ +static pte_t __kprobes *get_uprobe_pte (unsigned long address) +{ +pgd_t *pgd; +pud_t *pud; +pmd_t *pmd; +pte_t *pte =NULL ; + + pgd =pgd_offset (current->mm, address); +if (!pgd) + goto out; + + pud =pud_offset (pgd, address); +if (!pud) + goto out;arch/i386/kernel/uprobes.c int __kprobes arch_copy_uprobe(struct kp · line 103
Quoted 10 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +10
-0+ + pmd =pmd_offset (pud, address); +if (!pmd) + goto out; + + pte =pte_alloc_map (current->mm, pmd, address); + +out: +return pte; +}That's familiar looking code..
Quoted 3 lines
I guess this should be given a more generic name then placed in
mm/memory.c, which is where we do pagetable walking.
Apart from this, there looks like quite a bit of other mm code that has been crammed into everywhere but mm/ (yes this has happened before, but it shouldn't be encouraged in new code).
For this specific example, I'm not sure that a function returning a pointer to a pte is a good idea to be exporting. I'd like to see some good reasons why things like get_user_pages, find_*_page, and other standard APIs can't be used. Then you can list those reasons in an individual patch to add your required API to mm/. This can be more easily reviewed by people who aren't as good at wading through code as Andrew.
Also, adding your own mm code outside core files really makes things hard to maintain and audit when somebody would like to change anything.
Signature · 2 lines
SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com
Re: [1/3 PATCH] Kprobes: User space probes support- base interface
In reply to Andrew MortonQuoted 31 lines · Andrew Morton
Quoted 30 lines · Prasanna S Panchamukhi
This patch provides two interfaces to insert and remove user space probes. Each probe is uniquely identified by inode and offset within that executable/library file. Insertion of a probe involves getting the code page for a given offset, mapping it into the memory and then insert the breakpoint at the given offset. Also the probe is added to the
uprobe_tablehash list. Auprobe_moduledata strcture is allocated for every probed application/library image on disk. Removal of a probe involves getting the code page for a given offset, mapping that page into the memory and then replacing the breakpoint instruction with a the original opcode. This patch also provides aggrigate probe handler feature, where user can define multiple handlers per probe.kernel/uprobes.c new file +15
-0+/** +* Finds a uprobe at the specified user-space address in the current task. +* Points current_uprobe at that uprobe and returns the corresponding kprobe. +*/ +struct kprobe __kprobes *get_uprobe (void *addr) +{ +struct mm_struct *mm = current->mm; +struct vm_area_struct *vma; +struct inode *inode; +unsigned long offset; +struct kprobe *p, *kpr; +struct uprobe *uprobe; + + vma =find_vma (mm, (unsigned long )addr); +
Quoted 8 lines
kernel/uprobes.c line 143
Quoted 6 lines
kernel/uprobes.c new file +6
-0+BUG_ON (!vma);/* this should not happen, not in our memory map */ + + offset = (unsigned long )addr - (vma->vm_start + + (vma->vm_pgoff <<PAGE_SHIFT )); +if (!vma->vm_file) +return NULL ;All this appears to be happening without
mmap_semheld?
Yes, I will make the changes to hold the mmap_sem.
Quoted 23 lines
Quoted 10 lines
kernel/uprobes.c new file +9
-0+/** +* Wait for the page to be unlocked if someone else had locked it, +* then map the page and insert or remove the breakpoint. +*/ +static int __kprobes map_uprobe_page (struct page *page,struct uprobe *uprobe, +process_uprobe_func_t process_kprobe_user) +{ +int ret =0 ; +kprobe_opcode_t *uprobe_address;kernel/uprobes.c line 248
Quoted 10 lines
kernel/uprobes.c new file +10
-0+ +if (!page) +return -EINVAL ;/* TODO: more suitable errno */ + +wait_on_page_locked (page); +/* could probably retry readpage here. */ +if (!PageUptodate (page)) +return -EINVAL ;/* TODO: more suitable errno */ + +lock_page (page);That's a lot of fuss and might be racy with truncate.
Quoted 1 line
Why not just run
lock_page()?
Yes, I will do that.
Quoted 9 lines
kernel/uprobes.c line 254
Quoted 6 lines
kernel/uprobes.c new file +5
-0+ uprobe_address = (kprobe_opcode_t *)kmap (page); + uprobe_address = (kprobe_opcode_t *)((unsigned long )uprobe_address + + (uprobe->offset & ~PAGE_MASK )); + ret = (*process_kprobe_user)(uprobe, uprobe_address); +kunmap (page);
kmap_atomic()is preferred.
Agreed.
Quoted 4 lines
Quoted 4 lines
+/**
+ * Gets exclusive write access to the given inode to ensure that the file
+ * on which probes are currently applied does not change. Use the function,
+ *deny_write_access_to_inode()we added infs/namei.c.
Quoted 8 lines
kernel/uprobes.c line 313
Quoted 5 lines
+ */
kernel/uprobes.c new file +4
-0+static inline int ex_write_lock (struct inode *inode) +{ +return deny_write_access_to_inode (inode); +}hm, this code likes to poke around in VFS internals. It would be nice to have an overall description of what it's trying to do, why and how.
ok, I should probably separate VFS changes in a different patch with proper comments. However this is required to ensure that the executable associated with this inode on which probes are inserted does not change. deny_write_access_to_inode() just decrements the inode usage count to -1.
Quoted 59 lines
Quoted 42 lines
kernel/uprobes.c new file +41
-0+/** +* unregister_uprobe: Disarms the probe, removes the uprobe +* pointers from the hash list and unhooks readpage routines. +*/ +void __kprobes unregister_uprobe (struct uprobe *uprobe) +{ +struct address_space *mapping; +struct uprobe_module *umodule; +struct page *page; +unsigned long flags; +int ret =0 ; + +if (!uprobe->inode) +return ; + + mapping = uprobe->inode->i_mapping; + + page =find_get_page (mapping, uprobe->offset >>PAGE_CACHE_SHIFT ); + +spin_lock_irqsave (&uprobe_lock, flags); + ret =remove_uprobe (uprobe); +spin_unlock_irqrestore (&uprobe_lock, flags); + +mutex_lock (&uprobe_mutex); +if (!(umodule =get_module_by_inode (uprobe->inode))) + goto out; + +hlist_del (&uprobe->ulist); +if (hlist_empty (&umodule->ulist_head)) { +list_del (&umodule->mlist); +ex_write_unlock (uprobe->inode); +path_release (&umodule->nd); +kfree (umodule); + } + +out: +mutex_unlock (&uprobe_mutex); +if (ret) + ret =map_uprobe_page (page, uprobe, remove_kprobe_user); + +if (ret == -EINVAL )kernel/uprobes.c line 482
Quoted 10 lines
kernel/uprobes.c new file +10
-0+return ; +/* +* TODO: unregister_uprobe should not fail, need to handle +* if it fails. +*/ +flush_vma (mapping, page, uprobe); + +if (page) +page_cache_release (page); +}That's some pretty awkward coding. Buggy too. It doesn't drop the refcount on the page if
map_uprobe_page()returned-EINVALbecause it's assuming that EINVAL meant "there was no page". But there are multiple reasons formap_uprobe_page()to return-EINVAL. If that page isn't uptodate, we leak a ref.
Quoted 2 lines
This function should be doing the checking for a
find_get_page()failure, notmap_uprobe_page().
Yes, that is buggy, will fix.
Thanks
Prasanna
Signature · 5 lines
Prasanna S Panchamukhi Linux Technology Center India Software Labs, IBM Bangalore Email: [email protected] Ph: 91-80-51776329
Re: [2/3 PATCH] Kprobes: User space probes support- readpage hooks
In reply to Andrew MortonQuoted 35 lines · Andrew Morton
Quoted 23 lines · Prasanna S Panchamukhi
This patch provides the feature of inserting probes on pages that are not present in the memory during registration.
To add readpage and
readpages()hooks, two new elements are added to theuprobe_moduleobject:struct address_space_operations *ori_a_ops; struct address_space_operations user_a_ops;User-space probes allows probes to be inserted even in pages that are not present in the memory at the time of registration. This is done by adding hooks to the readpage and readpages routines. During registration, the address space operation object is modified by substituting user-space probes's specific readpage and readpages routines. When the pages are read into memory through the readpage and readpages address space operations, any associated probes are automatically inserted into those pages. These user-space probes readpage and readpages routines internally call the original
readpage()andreadpages()routines, and then check whether probes are to be added to these pages, inserting probes as necessary. The overhead of adding these hooks is limited to the application on which the probes are inserted.Quoted 4 lines
During unregistration, care should be taken to replace the readpage and readpages hooks with the original routines if no probes remain on that application.
So... The code's replacing the address_space's
address_space_operationswith a copy which has .readpage() and .readpages() modified, because it happens thatfilemap_nopage()uses those.This is all rather hacky.
Quoted 9 lines
I think we need to step back and discuss what services this feature is trying to provide, and how it is to provide them. Your covering description didn't describe that - it dives straigt into details without even describing what the patches are trying to achieve.
So. What are we trying to achieve here, and how are we trying to achieve it? What problems were encountered in the development of the feature and how were they solved? What alternative solutions were there?
The basic idea is to insert probes on user applications which may or may not be in memory, at the time of probe insertion.
The base interface patch (1/3) allows the user to insert the probes on the text pages that are already present in the memory. This is done by mapping the page via kmap() and then insert the breakpoint instruction at the given page offset.
Then comes the issue of inserting a probe on pages not currently in memory. This is useful if we'd want to probe an application that hasn't yet started executing. In such situations, we still want to insert the probe, but defer insertion of actual probe until the text page is read into the memory.
Here are a few ways to accomplish this:
1. Add a notifier in the readpage(), readpages() routine, which will notify when the text pages are read into the memory.
2. Reading the page from the executable into memory and then insert probes on that text page. But there are situations where the system can run into low memory problems and those text pages get discarded.
3. Change the readpage() and readpages() routines only for that application where probes will be inserted. This is done by hooking the readpage() and readpages() routines, which is limited to the probed application and will not change anything related to other applications.
The current patchset uses approach 3.
I'd like to know if there are better/cleaner ways to accomplish this?
Thanks
Prasanna
Signature · 5 lines
Prasanna S Panchamukhi Linux Technology Center India Software Labs, IBM Bangalore Email: [email protected] Ph: 91-80-51776329
Re: [3/3 PATCH] Kprobes: User space probes support- single stepping out-of-line
In reply to Andrew MortonQuoted 76 lines · Andrew Morton
Quoted 62 lines · Prasanna S Panchamukhi
This patch provides a mechanism for probe handling and executing the user-specified handlers.
Each userspace probe is uniquely identified by the combination of inode and offset, hence during registeration the inode and offset combination is added to uprobes hash table. Initially when breakpoint instruction is hit, the uprobes hash table is looked up for matching inode and offset. The
pre_handlersare called in sequence if multiple probes are registered. Similar to kprobes, uprobes also adopts to single step out-of-line, so that probe miss in SMP environment can be avoided. But for userspace probes, instruction copied into kernel address space cannot be single stepped, hence the instruction must be copied to user address space. The solution is to find free space in the current process address space and then copy the original instruction and single step that instruction.User processes use stack space to store local variables, agruments and return values. Normally the stack space either below or above the stack pointer indicates the free stack space.
The instruction to be single stepped can modify the stack space, hence before using the free stack space, sufficient stack space should be left. The instruction is copied to the bottom of the page and check is made such that the copied instruction does not cross the page boundry. The copied instruction is then single stepped. Several architectures does not allow the instruction to be executed from the stack location, since no-exec bit is set for the stack pages. In those architectures, the page table entry corresponding to the stack page is identified and the no-exec bit is unset making the instruction on that stack page to be executed.
There are situations where even the free stack space is not enough for the user instruction to be copied and single stepped. In such situations, the virtual memory area(vma) can be expanded beyond the current stack vma. This expaneded stack can be used to copy the original instruction and single step out-of-line.
Even if the vma cannot be extended then the instruction much be executed inline, by replacing the breakpoint instruction with original instruction.
...
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +18
-0+ +/** +* This routines get the pte of the page containing the specified address. +*/ +static pte_t __kprobes *get_uprobe_pte (unsigned long address) +{ +pgd_t *pgd; +pud_t *pud; +pmd_t *pmd; +pte_t *pte =NULL ; + + pgd =pgd_offset (current->mm, address); +if (!pgd) + goto out; + + pud =pud_offset (pgd, address); +if (!pud) + goto out;arch/i386/kernel/uprobes.c int __kprobes arch_copy_uprobe(struct kp · line 103
Quoted 10 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +10
-0+ + pmd =pmd_offset (pud, address); +if (!pmd) + goto out; + + pte =pte_alloc_map (current->mm, pmd, address); + +out: +return pte; +}That's familiar looking code..
Quoted 2 lines
I guess this should be given a more generic name then placed in
mm/memory.c, which is where we do pagetable walking.
Agreed, I will send out a separate patch for the helpers.
Quoted 38 lines
Quoted 24 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +23
-0+/** +* This routine check for space in the current process's stack +* address space. If enough address space is found, copy the original +* instruction on that page for single stepping out-of-line. +*/ +static int __kprobes copy_insn_on_new_page (struct uprobe *uprobe , +struct pt_regs *regs,struct vm_area_struct *vma) +{ +unsigned long addr, stack_addr = regs->esp; +int size =MAX_INSN_SIZE *sizeof (kprobe_opcode_t); + +if (vma->vm_flags &VM_GROWSDOWN ) { +if (((stack_addr -sizeof (long long ))) < + (vma->vm_start + size)) +return -ENOMEM ; + addr = vma->vm_start; + }else if (vma->vm_flags &VM_GROWSUP ) { +if ((vma->vm_end - size) < (stack_addr +sizeof (long long ))) +return -ENOMEM ; + addr = vma->vm_end - size; + }else +return -EFAULT ; +arch/i386/kernel/uprobes.c int __kprobes arch_copy_uprobe(struct kp · line 137
Quoted 10 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +10
-0+ vma->vm_flags |=VM_LOCKED ; + +if (__copy_to_user_inatomic ((unsigned long *)addr, + (unsigned long *)uprobe->kp.ainsn.insn, size)) +return -EFAULT ; + + regs->eip = addr; + +return 0 ; +}If we're going to use
__copy_to_user_inatomic()then we'll need some nice comments explaining why this is happening.
Quoted 4 lines
And we'll need to actually *be* in-atomic. That means we need an open-coded
inc_preempt_count()anddec_preempt_count()in there and I don't see them.
We come here, after probe is hit, through uporbe_handler() with interrupts disabled (since it is a interrupt gate). In uprobe_handler() preemption is disabled and remains disabled until original instruction is single stepped.
I will add proper comments in next iteration.
Quoted 51 lines
Quoted 38 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +38
-0+/** +* This routine expands the stack beyond the present process address +* space and copies the instruction to that location, so that +* processor can single step out-of-line. +*/ +static int __kprobes copy_insn_onexpstack (struct uprobe *uprobe, +struct pt_regs *regs,struct vm_area_struct *vma) +{ +unsigned long addr, vm_addr; +int size =MAX_INSN_SIZE *sizeof (kprobe_opcode_t); +struct vm_area_struct *new_vma; +struct mm_struct *mm = current->mm; + + +if (!down_read_trylock (¤t->mm->mmap_sem)) +return -ENOMEM ; + +if (vma->vm_flags &VM_GROWSDOWN ) + vm_addr = vma->vm_start - size; +else if (vma->vm_flags &VM_GROWSUP ) + vm_addr = vma->vm_end + size; +else { +up_read (¤t->mm->mmap_sem); +return -EFAULT ; + } + + new_vma =find_extend_vma (mm, vm_addr); +if (!new_vma) { +up_read (¤t->mm->mmap_sem); +return -ENOMEM ; + } + +if (new_vma->vm_flags &VM_GROWSDOWN ) + addr = new_vma->vm_start; +else + addr = new_vma->vm_end - size; + + new_vma->vm_flags |=VM_LOCKED ;arch/i386/kernel/uprobes.c int __kprobes arch_copy_uprobe(struct kp · line 186
Quoted 10 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +10
-0+up_read (¤t->mm->mmap_sem); + +if (__copy_to_user_inatomic ((unsigned long *)addr, + (unsigned long *)uprobe->kp.ainsn.insn, size)) +return -EFAULT ; + + regs->eip = addr; + +return 0 ; +}Why is
VM_LOCKEDbeing set? (It needs a comment).
Quoted 1 line
Where does it get unset?
As Arjan says, idea was to make copy_to_user_inatomic() succeed but there is some issue here, I have to look at it again.
Quoted 20 lines
arch/i386/kernel/uprobes.c int __kprobes arch_copy_uprobe(struct kp · line 225
Quoted 5 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +5
-0+ +if (__copy_to_user_inatomic ((unsigned long *)page_addr, + source, size)) +if (__copy_to_user_inatomic ( + (unsigned long *)(page_addr - size), source, size))See above.
Quoted 12 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +12
-0+ +/** +* This routines get the page containing the probe, maps it and +* replaced the instruction at the probed address with specified +* opcode. +*/ +void __kprobes replace_original_insn (struct uprobe *uprobe, +struct pt_regs *regs,kprobe_opcode_t opcode) +{ +kprobe_opcode_t *addr; +struct page *page; +
Quoted 9 lines
arch/i386/kernel/uprobes.c int __kprobes arch_copy_uprobe(struct kp · line 250
Quoted 5 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +5
-0+ page =find_get_page (uprobe->inode->i_mapping, + uprobe->offset >>PAGE_CACHE_SHIFT ); +BUG_ON (!page); + +__lock_page (page);Whoa. Why is
__lock_page()being used here? It looks like a bug is being covered up.
we come here with a spinlock held. I will add the comment.
Quoted 10 lines
arch/i386/kernel/uprobes.c int __kprobes arch_copy_uprobe(struct kp · line 256
Quoted 5 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +5
-0+ addr = (kprobe_opcode_t *)kmap_atomic (page,KM_USER1 ); + addr = (kprobe_opcode_t *)((unsigned long )addr + + (unsigned long )(uprobe->offset & ~PAGE_MASK )); + *addr = opcode; +/*TODO: flush vma ? */
flush_dcache_page()would be needed.But then, what happens if the page is shared by other processes? Do they all start taking debug traps?
Yes, you are right. I think single stepping inline was a bad idea, disarming the probe looks to be a better option
Quoted 17 lines
Quoted 17 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +17
-0+kunmap_atomic (addr,KM_USER1 ); + +unlock_page (page); + +if (page) +page_cache_release (page); + regs->eip = (unsigned long )uprobe->kp.addr; +} + +/** +* This routine provides the functionality of single stepping +* out-of-line. If single stepping out-of-line cannot be achieved, +* it replaces with the original instruction allowing it to single +* step inline. +*/ +static inline int prepare_singlestep_uprobe (struct uprobe *uprobe, +struct uprobe_ctlblk *ucb,struct pt_regs *regs)
Quoted 8 lines
arch/i386/kernel/uprobes.c int __kprobes arch_copy_uprobe(struct kp · line 279
Quoted 6 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +6
-0+{ +unsigned long stack_addr = regs->esp, flags; +struct vm_area_struct *vma =NULL ; +int err =0 ; + + vma =find_vma (current->mm, (stack_addr &PAGE_MASK ));I don't think
mmap_semis held here?
Yes, this will be taken care.
Quoted 6 lines
Quoted 2 lines
+static inline int
uprobe_fault_handler(struct pt_regs *regs, int trapnr)If, for some reason, the compiler decides to not inline this function then you have a hunk of code which isn't marked
__kprobes, but it should be.
Quoted 7 lines
I'd suggest that you remove all inlining from this code and add the appropriate section markers.
Or I guess you could use
__always_inline, but I'm not sure that it's really worth the fuss and obscurity of doing that.All kprobes-related code should be audited for this problem.
Yes, I will audit it and send out a patch if necessary.
Thanks
Prasanna
Signature · 5 lines
Prasanna S Panchamukhi Linux Technology Center India Software Labs, IBM Bangalore Email: [email protected] Ph: 91-80-51776329
Re: [3/3 PATCH] Kprobes: User space probes support- single stepping out-of-line
In reply to Nick PigginQuoted 52 lines · Nick Piggin
Quoted 35 lines · Andrew Morton
Quoted 19 lines · Prasanna S Panchamukhi
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +3
-0+ +/** +* This routines get the pte of the page containing the specified address.
+ */arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +14
-0+static pte_t __kprobes *get_uprobe_pte (unsigned long address) +{ +pgd_t *pgd; +pud_t *pud; +pmd_t *pmd; +pte_t *pte =NULL ; + + pgd =pgd_offset (current->mm, address); +if (!pgd) + goto out; + + pud =pud_offset (pgd, address); +if (!pud) + goto out;arch/i386/kernel/uprobes.c int __kprobes arch_copy_uprobe(struct kp · line 103
Quoted 10 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +10
-0+ + pmd =pmd_offset (pud, address); +if (!pmd) + goto out; + + pte =pte_alloc_map (current->mm, pmd, address); + +out: +return pte; +}That's familiar looking code..
Quoted 3 lines
I guess this should be given a more generic name then placed in
mm/memory.c, which is where we do pagetable walking.Apart from this, there looks like quite a bit of other mm code that has been crammed into everywhere but
mm/(yes this has happened before, but it shouldn't be encouraged in new code).For this specific example, I'm not sure that a function returning a pointer to a pte is a good idea to be exporting. I'd like to see some good reasons why things like
get_user_pages, find_*_page, and other standard APIs can't be used. Then you can list those reasons in an individual patch to add your required API tomm/. This can be more easily reviewed by people who aren't as good at wading through code as Andrew.
Quoted 4 lines
Also, adding your own mm code outside core files really makes things hard to maintain and audit when somebody would like to change anything.
Nick,
I will send out a separate patch to add this piece of code with proper comments.
Thanks
Prasanna
Quoted 3 lines
Signature · 2 lines
SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com
Signature · 5 lines
Prasanna S Panchamukhi Linux Technology Center India Software Labs, IBM Bangalore Email: [email protected] Ph: 91-80-51776329
Re: [2/3 PATCH] Kprobes: User space probes support- readpage hooks
In reply to Andrew MortonQuoted 11 lines · Andrew Morton
Quoted 7 lines · Prasanna S Panchamukhi
+/**
+ * This function hooks the readpages() of all modules that have active + * probes on them. The original readpages() is called for the given + * inode/address_space to actually read the pages into the memory. + * Then all probes that are specified on these pages are inserted.+ */
The "/**" thing is designed to introduce a kerneldoc-style comment, but these comments aren't using kerneldoc.
Yes. I will take care of this issue.
Thanks
Prasanna
Signature · 5 lines
Prasanna S Panchamukhi Linux Technology Center India Software Labs, IBM Bangalore Email: [email protected] Ph: 91-80-51776329
Re: [3/3 PATCH] Kprobes: User space probes support- single stepping out-of-line
In reply to Arjan van de VenQuoted 10 lines · Arjan van de Ven
Lines: 16
Quoted 4 lines
And we'll need to actually *be* in-atomic. That means we need an open-coded
inc_preempt_count()anddec_preempt_count()in there and I don't see them...
Quoted 8 lines
Quoted 3 lines
Why is
VM_LOCKEDbeing set? (It needs a comment).Where does it get unset?
if this is an attempt to make the
copy_in_atomicto be atomic, then it is a bug; the user can unset this bit after all via mprotect, even from another thread, and concurrently. U
You are right, the purpose was to make copy_to_user_inatomic() to suceed. I need to look at this issue again. Any suggestions?
Thanks
Prasanna
Signature · 5 lines
Prasanna S Panchamukhi Linux Technology Center India Software Labs, IBM Bangalore Email: [email protected] Ph: 91-80-51776329
Re: [3/3 PATCH] Kprobes: User space probes support- single stepping out-of-line
Quoted 16 lines · Prasanna S Panchamukhi
Quoted 15 lines · Arjan van de Ven
Lines: 16
Quoted 4 lines
And we'll need to actually *be* in-atomic. That means we need an open-coded
inc_preempt_count()anddec_preempt_count()in there and I don't see them...
Quoted 3 lines
Why is
VM_LOCKEDbeing set? (It needs a comment).Where does it get unset?
Quoted 6 lines
Quoted 3 lines
if this is an attempt to make the
copy_in_atomicto be atomic, then it is a bug; the user can unset this bit after all via mprotect, even from another thread, and concurrently. UYou are right, the purpose was to make
copy_to_user_inatomic()to suceed. I need to look at this issue again. Any suggestions?
get_user_pages seems to be the only viable API for this..
Re: [3/3 PATCH] Kprobes: User space probes support- single stepping out-of-line
In reply to Prasanna S PanchamukhiQuoted 19 lines · Prasanna S Panchamukhi
Quoted 14 lines
arch/i386/kernel/uprobes.c int __kprobes arch_copy_uprobe(struct kp · line 137
Quoted 10 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +9
-0+ +if (__copy_to_user_inatomic ((unsigned long *)addr, + (unsigned long *)uprobe->kp.ainsn.insn, size)) +return -EFAULT ; + + regs->eip = addr; + +return 0 ; +}If we're going to use
__copy_to_user_inatomic()then we'll need some nice comments explaining why this is happening.Quoted 4 lines
And we'll need to actually *be* in-atomic. That means we need an open-coded
inc_preempt_count()anddec_preempt_count()in there and I don't see them.
Quoted 6 lines
We come here, after probe is hit, through
uporbe_handler()with interrupts disabled (since it is a interrupt gate). Inuprobe_handler()preemption is disabled and remains disabled until original instruction is single stepped.I will add proper comments in next iteration.
preempt_disable() is insufficient - it is a no-op on !CONFIG_PREEMPT.
You _must_ run inc_preempt_count(). See how kmap_atomic() and kunmap_atomic() work.
Quoted 17 lines
Quoted 7 lines
Quoted 7 lines
+ */
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +6
-0+void __kprobes replace_original_insn (struct uprobe *uprobe, +struct pt_regs *regs,kprobe_opcode_t opcode) +{ +kprobe_opcode_t *addr; +struct page *page; +Quoted 9 lines
arch/i386/kernel/uprobes.c int __kprobes arch_copy_uprobe(struct kp · line 250
Quoted 5 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +5
-0+ page =find_get_page (uprobe->inode->i_mapping, + uprobe->offset >>PAGE_CACHE_SHIFT ); +BUG_ON (!page); + +__lock_page (page);Whoa. Why is
__lock_page()being used here? It looks like a bug is being covered up.
Quoted 1 line
we come here with a spinlock held. I will add the comment.
Then the code is buggy. __lock_page() can schedule away, causing this CPU to recur onto the same lock and deadlock.
Quoted 8 lines
Quoted 8 lines
arch/i386/kernel/uprobes.c int __kprobes arch_copy_uprobe(struct kp · line 256
Quoted 5 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +5
-0+ addr = (kprobe_opcode_t *)kmap_atomic (page,KM_USER1 ); + addr = (kprobe_opcode_t *)((unsigned long )addr + + (unsigned long )(uprobe->offset & ~PAGE_MASK )); + *addr = opcode; +/*TODO: flush vma ? */
flush_dcache_page()would be needed.
Quoted 6 lines
Quoted 2 lines
But then, what happens if the page is shared by other processes? Do they all start taking debug traps?
Yes, you are right. I think single stepping inline was a bad idea, disarming the probe looks to be a better option
You skipped my second question?
Re: [2/3 PATCH] Kprobes: User space probes support- readpage hooks
In reply to Prasanna S PanchamukhiQuoted 3 lines · Prasanna S Panchamukhi
The basic idea is to insert probes on user applications which may or
may not be in memory, at the time of probe insertion.
umm yes, but what for?
What does this entire feature *do*? Why does Linux need it?
OK, so it allows kernel modules to set breakpoints (via debug traps) into user code. But why do we want to be able to do that? What are the use-cases?
This may sound like boringly obvious stuff to you, but without a complete problem statement from the designers, how are we to evaluate their proposed solution?
Re: [2/3 PATCH] Kprobes: User space probes support- readpage hooks
Andrew, the need is probably better stated as one where system-wide probing or tracing is possible. There are times where one needs a global view. Of course one can use multiple tools to obtain such data e.g. probes in kernel space strace in user space and so on. The advantages of supporting user probes are as follows:
1) A single tool providing the data capture in a consistent manner eases the problems of correlation of events across multiple tools (for kernel and user space)
2) The dynamic aspect allows ad hoc probepoints to be inserted where no existing instrumentation is provided (emergency debug scenario for example).
3) The user probe distinguishes itself from all other externally managed tracing mechanisms in that probepoints are globally applied - i.e. without reference to PID. Compare this with ptrace breakpoints (hence strace and gdb) where tracepoints and breakpoints are localized to a specified set of processes. user-probes achieves this by design without the side effects (privatization of pages) that ptrace has. Again this supports the global view.
4) user-probes also supports the registering of the probepoints before an the probed code is loaded. The clearly has advantages for catching initialization problems.
A real life example of where this capability would have been very useful is with a performance problem I am currently investigating. It involves a GPFS
+ SAMBA + TCPIP + RDAC and some user-space video serving application. We are looking are where the latencies are accumulating in the system for the specific user application. It's a very hard problem. Having multiple tools serve up a partial view and having to coordinate these view from both data analysis and data gathering perspectives is a real nightmare.
- -
Richard J Moore
IBM Advanced Linux Response Team - Linux Technology Centre
MOBEX: 264807; Mobile (+44) (0)7739-875237
Office: (+44) (0)1962-817072
Andrew Morton
<[email protected]>
To
21/03/2006 [email protected]
02:12 cc
[email protected], [email protected], [email protected], Richard J Moore/UK/IBM@IBMGB, [email protected]
bcc
Subject
Re: [2/3 PATCH] Kprobes: User space probes support- readpage hooks
Quoted 3 lines · Prasanna S Panchamukhi
The basic idea is to insert probes on user applications which may or
may not be in memory, at the time of probe insertion.
umm yes, but what for?
What does this entire feature *do*? Why does Linux need it?
OK, so it allows kernel modules to set breakpoints (via debug traps) into user code. But why do we want to be able to do that? What are the use-cases?
This may sound like boringly obvious stuff to you, but without a complete problem statement from the designers, how are we to evaluate their proposed solution?
Re: [3/3 PATCH] Kprobes: User space probes support- single stepping out-of-line
In reply to Andrew MortonQuoted 25 lines · Andrew Morton
Quoted 24 lines · Prasanna S Panchamukhi
Quoted 14 lines
arch/i386/kernel/uprobes.c int __kprobes arch_copy_uprobe(struct kp · line 137
Quoted 10 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +9
-0+ +if (__copy_to_user_inatomic ((unsigned long *)addr, + (unsigned long *)uprobe->kp.ainsn.insn, size)) +return -EFAULT ; + + regs->eip = addr; + +return 0 ; +}If we're going to use
__copy_to_user_inatomic()then we'll need some nice comments explaining why this is happening.Quoted 4 lines
And we'll need to actually *be* in-atomic. That means we need an open-coded
inc_preempt_count()anddec_preempt_count()in there and I don't see them.We come here, after probe is hit, through
uporbe_handler()with interrupts disabled (since it is a interrupt gate). Inuprobe_handler()preemption is disabled and remains disabled until original instruction is single stepped.
Quoted 7 lines
Quoted 1 line
I will add proper comments in next iteration.
preempt_disable()is insufficient - it is a no-op on !CONFIG_PREEMPT.You _must_ run
inc_preempt_count(). See howkmap_atomic()andkunmap_atomic()work.
Yes, I will use inc_preempt_count() instead of preempt_disable().
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
arch/i386/kernel/uprobes.c int __kprobes arch_copy_uprobe(struct kp · line 250
Quoted 12 lines
+ */
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +11
-0+void __kprobes replace_original_insn (struct uprobe *uprobe, +struct pt_regs *regs,kprobe_opcode_t opcode) +{ +kprobe_opcode_t *addr; +struct page *page; + + page =find_get_page (uprobe->inode->i_mapping, + uprobe->offset >>PAGE_CACHE_SHIFT ); +BUG_ON (!page); + +__lock_page (page);Whoa. Why is
__lock_page()being used here? It looks like a bug is being covered up.
Quoted 4 lines
Quoted 1 line
we come here with a spinlock held. I will add the comment.
Then the code is buggy.
__lock_page()can schedule away, causing this CPU to recur onto the same lock and deadlock.
Agreed. I will look at this issue and remove the __lock_page().
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
arch/i386/kernel/uprobes.c int __kprobes arch_copy_uprobe(struct kp · line 256
Quoted 6 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +5
-0+ addr = (kprobe_opcode_t *)kmap_atomic (page,KM_USER1 ); + addr = (kprobe_opcode_t *)((unsigned long )addr + + (unsigned long )(uprobe->offset & ~PAGE_MASK )); + *addr = opcode; +/*TODO: flush vma ? */
flush_dcache_page()would be needed.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
But then, what happens if the page is shared by other processes? Do they all start taking debug traps?
Yes, you are right. I think single stepping inline was a bad idea, disarming the probe looks to be a better option
You skipped my second question?
There is a small window in which other processor will not be able to see
the breakpoint if we are single step inline. But since single stepping inline
is a bad idea, we will disarm the probe forever (replace with original instrcution) if we cannot single step out-of-line. Any suggestions?
Thanks
Prasanna
Signature · 5 lines
Prasanna S Panchamukhi Linux Technology Center India Software Labs, IBM Bangalore Email: [email protected] Ph: 91-80-51776329
Re: [3/3 PATCH] Kprobes: User space probes support- single stepping out-of-line
Quoted 17 lines · Prasanna S Panchamukhi
Quoted 17 lines
Quoted 16 lines
Quoted 10 lines
arch/i386/kernel/uprobes.c int __kprobes arch_copy_uprobe(struct kp · line 256
Quoted 7 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +5
-0+ addr = (kprobe_opcode_t *)kmap_atomic (page,KM_USER1 ); + addr = (kprobe_opcode_t *)((unsigned long )addr + + (unsigned long )(uprobe->offset & ~PAGE_MASK )); + *addr = opcode; +/*TODO: flush vma ? */
flush_dcache_page()would be needed.Quoted 2 lines
But then, what happens if the page is shared by other processes? Do they all start taking debug traps?
Yes, you are right. I think single stepping inline was a bad idea, disarming the probe looks to be a better option
Quoted 6 lines
Quoted 1 line
You skipped my second question?
There is a small window in which other processor will not be able to see
the breakpoint if we are single step inline. But since single stepping inline
is a bad idea, we will disarm the probe forever (replace with original instrcution) if we cannot single step out-of-line. Any suggestions?
This doesn't appear to be working.
Let's go back in time and pretend that these patches were never written, OK? We're standing around the water cooler saying "hey, wouldn't it be cool if someone did X". You guys are way too far into this and you keep on leaving everyone else behind. When I try to drag you up, you resist ;)
So let me rephrase, and thrash around in the dark a little more.
Quoted 1 line
From my reading of the code (and thus far that's my _only_ source of this
information) it appears that a design decision has been made (for reasons which have yet to be disclosed) that the way to implement this (yet to be described) requirement is to set user breakpoints upon particular instructions within executables. System-wide, for all processes and threads.
There are other things that could have been done. For example, you might have chosen to set breakpoints upon a particular virtual address within a heavyweight process. That's a process-oriented viewpoint, rather than a file-oriented one, if you like.
This raises interesting questions, like
- How come that decision was made? Why _is_ this an executable-oriented rather than process-oriented thing? Richard has covered that somewhat.
- What happens if the executable is writeably mapped?
- What happens if someone writes to the executable? (I think both of these were disallowed in the implementation-which-is-not-to-be-named).
- What happens if different processes map the executable at different addresses?
- Various other things which you've thought of and I haven't and which it would be REALLY interesting to hear about.
But this is just one example. I don't think I'm being too picky here - my reaction on seeing all this stuff was, basically, "wtf is all this code for?". References to dprobes won't help, sorry - I've never looked at dprobes and I don't know anyone apart from you guys who has.
What I'm asking you for is a description of what problem we're trying to solve and how this code solves that problem. It is hard, it is inefficient and, worse, it is error-prone for us to try to work all that out from a particular implementation.
Re: [3/3 PATCH] Kprobes: User space probes support- single stepping out-of-line
Andrew Morton <[email protected]> wrote on 21/03/2006 10:05:21:
Quoted 9 lines
Quoted 8 lines · Prasanna S Panchamukhi
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +5
-0+ addr = (kprobe_opcode_t *)kmap_atomic (page,KM_USER1 ); + addr = (kprobe_opcode_t *)((unsigned long )addr + + (unsigned long )(uprobe->offset & ~PAGE_MASK )); + *addr = opcode; +/*TODO: flush vma ? */
Quoted 14 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
flush_dcache_page()would be needed.But then, what happens if the page is shared by other
processes? Do they
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
all start taking debug traps?
Yes, you are right. I think single stepping inline was a bad
idea, disarming
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
the probe looks to be a better option
You skipped my second question?
There is a small window in which other processor will not be able to
see
Quoted 11 lines
Quoted 1 line
the breakpoint if we are single step inline. But since single
stepping inline
Quoted 1 line
is a bad idea, we will disarm the probe forever (replace with
original instrcution) if we cannot single step out-of-line.
Quoted 1 line
Any suggestions?
This doesn't appear to be working.
Let's go back in time and pretend that these patches were never written, OK? We're standing around the water cooler saying "hey, wouldn't it be cool if someone did X". You guys are way too far into this and you keep
on
Quoted 18 lines
leaving everyone else behind. When I try to drag you up, you resist ;)
So let me rephrase, and thrash around in the dark a little more.
From my reading of the code (and thus far that's my _only_ source of this information) it appears that a design decision has been made (for reasons which have yet to be disclosed) that the way to implement this (yet to be described) requirement is to set user breakpoints upon particular instructions within executables. System-wide, for all processes and threads.
There are other things that could have been done. For example, you might have chosen to set breakpoints upon a particular virtual address within a heavyweight process. That's a process-oriented viewpoint, rather than a file-oriented one, if you like.
This raises interesting questions, like
Quoted 7 lines
- How come that decision was made? Why _is_ this an executable-oriented rather than process-oriented thing? Richard has covered that somewhat.
- What happens if the executable is writeably mapped?
- What happens if someone writes to the executable? (I think both of these were disallowed in the
implementation-which-is-not-to-be-named).
Quoted 8 lines
- What happens if different processes map the executable at different addresses?
- Various other things which you've thought of and I haven't and which it would be REALLY interesting to hear about.
But this is just one example. I don't think I'm being too picky here -
my
Quoted 6 lines
reaction on seeing all this stuff was, basically, "wtf is all this code for?". References to dprobes won't help, sorry - I've never looked at dprobes and I don't know anyone apart from you guys who has.
What I'm asking you for is a description of what problem we're trying to solve and how this code solves that problem. It is hard, it is
inefficient
Quoted 2 lines
and, worse, it is error-prone for us to try to work all that out from a particular implementation.
This is completely reasonable. And we don't need to refer to dprobes - that was merely an evolutionary step.
I do have a notes that captured most the early design discussions and decisions, which Prasanna and I should dive into to help answer some of the points you raise.
Let me suggest, we chat among ourselves and pull together the relevant details to answer each of your questions.
Richard
Re: [3/3 PATCH] Kprobes: User space probes support- single stepping out-of-line
In reply to Andrew MortonQuoted 67 lines · Andrew Morton
Quoted 17 lines · Prasanna S Panchamukhi
Quoted 17 lines
Quoted 16 lines
Quoted 10 lines
arch/i386/kernel/uprobes.c int __kprobes arch_copy_uprobe(struct kp · line 256
Quoted 7 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +5
-0+ addr = (kprobe_opcode_t *)kmap_atomic (page,KM_USER1 ); + addr = (kprobe_opcode_t *)((unsigned long )addr + + (unsigned long )(uprobe->offset & ~PAGE_MASK )); + *addr = opcode; +/*TODO: flush vma ? */
flush_dcache_page()would be needed.Quoted 2 lines
But then, what happens if the page is shared by other processes? Do they all start taking debug traps?
Yes, you are right. I think single stepping inline was a bad idea, disarming the probe looks to be a better option
Quoted 6 lines
Quoted 1 line
You skipped my second question?
There is a small window in which other processor will not be able to see
the breakpoint if we are single step inline. But since single stepping inline
is a bad idea, we will disarm the probe forever (replace with original instrcution) if we cannot single step out-of-line. Any suggestions?This doesn't appear to be working.
Let's go back in time and pretend that these patches were never written, OK? We're standing around the water cooler saying "hey, wouldn't it be cool if someone did X". You guys are way too far into this and you keep on leaving everyone else behind. When I try to drag you up, you resist ;)
So let me rephrase, and thrash around in the dark a little more.
Quoted 1 line
From my reading of the code (and thus far that's my _only_ source of this
information) it appears that a design decision has been made (for reasons which have yet to be disclosed) that the way to implement this (yet to be described) requirement is to set user breakpoints upon particular instructions within executables. System-wide, for all processes and threads.
There are other things that could have been done. For example, you might have chosen to set breakpoints upon a particular virtual address within a heavyweight process. That's a process-oriented viewpoint, rather than a file-oriented one, if you like.
This raises interesting questions, like
- How come that decision was made? Why _is_ this an executable-oriented rather than process-oriented thing? Richard has covered that somewhat.
- What happens if the executable is writeably mapped?
- What happens if someone writes to the executable? (I think both of these were disallowed in the implementation-which-is-not-to-be-named).
- What happens if different processes map the executable at different addresses?
- Various other things which you've thought of and I haven't and which it would be REALLY interesting to hear about.
But this is just one example. I don't think I'm being too picky here - my reaction on seeing all this stuff was, basically, "wtf is all this code for?". References to dprobes won't help, sorry - I've never looked at dprobes and I don't know anyone apart from you guys who has.
Quoted 4 lines
What I'm asking you for is a description of what problem we're trying to solve and how this code solves that problem. It is hard, it is inefficient and, worse, it is error-prone for us to try to work all that out from a particular implementation.
Prasanna, I guess putting together a short writeup on the problem description and the thinking behind the design decisions, known flaws etc, in the form of a Documentation patch may help for starters. These questions are likely to come up everytime anyone looks at the code.
The key thinking behind a lot of the design decisions was the
need for a very low overhead probe mechanism that would allow thousands of active probes on the system and could detect any instance which hits the probe, including probes on shared libraries which may be loaded by lots of processes. Thus, (1) no forcing copy-on-write pages, (2) no forcing in executable pages in memory just to place a probe on them (hence zero overhead for probes which are very unlikely to be hit), (3) no restrictions on evicting a page with a probe on it from memory (4) probes being tracked by an (inode, offset) tuple rather than by virtual address so that they can be shared across all processes mapping the executable/library even at different virtual addresses, etc.
Regards
Suparna
Signature · 3 lines
Suparna Bhattacharya ([email protected]) Linux Technology Center IBM Software Lab, India
Re: [2/3 PATCH] Kprobes: User space probes support- readpage hooks
In reply to Richard J MooreQuoted 3 lines
A real life example of where this capability would have been very useful is with a performance problem I am currently investigating. It involves a GPFS
+ SAMBA + TCPIP + RDAC
this pobablt tells more about the crappy code quality of your propritary
code than a real need for this. please argue without reference to huge
blobs of junk.
Re: [2/3 PATCH] Kprobes: User space probes support- readpage hooks
In reply to Andrew MortonQuoted 1 line · Andrew Morton
What does this entire feature *do*? Why does Linux need it?
it's what dtrace does. thus the marketing departments at ibm and redhat decided to copy the features 1:1 instead of thinking what problem they want to solve.
Quoted 5 lines · Prasanna S Panchamukhi
This patch set provides the basic infrastructure for user-space probes based on IBM's Dprobes. Similar to kprobes, user-space probes uses the breakpoint mechanism to insert probes. User has to write a kernel module to insert probes in the executable/library and specify the handlers in the kernel module.
Doing this from kernelspace is wrong. It should be done from userspace, best using a systemcall. Of couse the handlers can't work as-is you'd need to redo that to work similarlt to ptrace - which will hopefully allow some code reuse aswell.
And please - independent of the best api in this case - please don't ever submit large amounts of code again that don't have any in-tree user.
Re: [2/3 PATCH] Kprobes: User space probes support- readpage hooks
In reply to Christoph HellwigChristoph Hellwig <[email protected]> wrote on 21/03/2006 11:14:52:
Quoted 1 line
Quoted 1 line
A real life example of where this capability would have been very
useful is
Quoted 1 line
Quoted 1 line
with a performance problem I am currently investigating. It involves a
GPFS
Quoted 6 lines
Quoted 1 line
+ SAMBA + TCPIP + RDAC
this pobablt tells more about the crappy code quality of your propritary
code than a real need for this. please argue without reference to huge
blobs of junk.
Damn! I knew there was an obvious answer. Thanks Christoph, I'll code a fix over lunch. Your insight is as always most refreshing.
Richard
Re: [2/3 PATCH] Kprobes: User space probes support- readpage hooks
In reply to Christoph HellwigChristoph Hellwig <[email protected]> wrote on 21/03/2006 11:28:07:
Quoted 7 lines · Andrew Morton
Quoted 1 line
What does this entire feature *do*? Why does Linux need it?
it's what dtrace does. thus the marketing departments at ibm and redhat decided to copy the features 1:1 instead of thinking what problem they want to solve.
I think you'll find it happened the other way round. Sun openly references my white papers. They even stole the name of an ancestor to kprobes. But who cares, it not relevant or particularly interesting whether the chicken or the egg came first.
Re: [2/3 PATCH] Kprobes: User space probes support- readpage hooks
Quoted 4 lines
I think you'll find it happened the other way round. Sun openly references my white papers. They even stole the name of an ancestor to kprobes. But who cares, it not relevant or particularly interesting whether the chicken or the egg came first.
I know your papers, too. In fact dprobes' RPN program downloads are a far better design than systemtap's generation of kernel code. it's a pity that you gave up on dprobes instead of applying the required work to it and integrate it with other bits of a tracing framework.
Re: [3/3 PATCH] Kprobes: User space probes support- single stepping out-of-line
In reply to Andrew MortonQuoted 67 lines · Andrew Morton
Quoted 17 lines · Prasanna S Panchamukhi
Quoted 17 lines
Quoted 16 lines
Quoted 10 lines
arch/i386/kernel/uprobes.c int __kprobes arch_copy_uprobe(struct kp · line 256
Quoted 7 lines
arch/i386/kernel/uprobes.c renamed from arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line +5
-0+ addr = (kprobe_opcode_t *)kmap_atomic (page,KM_USER1 ); + addr = (kprobe_opcode_t *)((unsigned long )addr + + (unsigned long )(uprobe->offset & ~PAGE_MASK )); + *addr = opcode; +/*TODO: flush vma ? */
flush_dcache_page()would be needed.Quoted 2 lines
But then, what happens if the page is shared by other processes? Do they all start taking debug traps?
Yes, you are right. I think single stepping inline was a bad idea, disarming the probe looks to be a better option
Quoted 6 lines
Quoted 1 line
You skipped my second question?
There is a small window in which other processor will not be able to see
the breakpoint if we are single step inline. But since single stepping inline
is a bad idea, we will disarm the probe forever (replace with original instrcution) if we cannot single step out-of-line. Any suggestions?This doesn't appear to be working.
Let's go back in time and pretend that these patches were never written, OK? We're standing around the water cooler saying "hey, wouldn't it be cool if someone did X". You guys are way too far into this and you keep on leaving everyone else behind. When I try to drag you up, you resist ;)
So let me rephrase, and thrash around in the dark a little more.
Quoted 1 line
From my reading of the code (and thus far that's my _only_ source of this
information) it appears that a design decision has been made (for reasons which have yet to be disclosed) that the way to implement this (yet to be described) requirement is to set user breakpoints upon particular instructions within executables. System-wide, for all processes and threads.
There are other things that could have been done. For example, you might have chosen to set breakpoints upon a particular virtual address within a heavyweight process. That's a process-oriented viewpoint, rather than a file-oriented one, if you like.
This raises interesting questions, like
- How come that decision was made? Why _is_ this an executable-oriented rather than process-oriented thing? Richard has covered that somewhat.
- What happens if the executable is writeably mapped?
- What happens if someone writes to the executable? (I think both of these were disallowed in the implementation-which-is-not-to-be-named).
- What happens if different processes map the executable at different addresses?
- Various other things which you've thought of and I haven't and which it would be REALLY interesting to hear about.
But this is just one example. I don't think I'm being too picky here - my reaction on seeing all this stuff was, basically, "wtf is all this code for?". References to dprobes won't help, sorry - I've never looked at dprobes and I don't know anyone apart from you guys who has.
Quoted 4 lines
What I'm asking you for is a description of what problem we're trying to solve and how this code solves that problem. It is hard, it is inefficient and, worse, it is error-prone for us to try to work all that out from a particular implementation.
Andrew,
The basic need is to provide infrastructure for user-space dynamic instrumentation. As with kprobes, there is no need to recompile or restart applications for instrumentation, under a debugger for instance.
Possible approaches which were looked up
1. Attaching or loading the application into a tool.
In this method the user application must be loaded into a tool or the tool is attached to already running application. Before the user can instrument an application the user should decide what that instrumentation will consist of. Dynaprof uses such a mechanism.
http://www.dyninst.org/tools.html
2. Using a "jump" instruction to a trampoline and trampoline executing the instrumented code in user-space.
Eg: Paradyn tool. (http://www.paradyn.org/)
Issues with method 1 and 2 are:
* Induces Intel erratum E49 where the other processors see stale data while one processor replaces the jump instruction.
* Instruction can only be replaced atomically if the size of the jump instruction is greater than or equal to the original instruction.
* Other processors need to be stopped if the "jump" instruction size is less than the original instruction.
3. Using breakpoint instruction Using a breakpoint instruction and executing the instrumentation code from within the breakpoint handler in the interrupt context.
The advantages of the approach (3) taken, apart from what Suparna listed earlier in this thread
- User is able to collect user-space data and kernel space data using the same instrumenation code and getting a complete picture
- Probes are visible across fork() syscall.
Richard's mail earlier in this thread details the per process Vs per executable file based probe.
This is just an RFC and we are looking for suggestions (like Christoph's).
Thanks
Prasanna
Signature · 5 lines
Prasanna S Panchamukhi Linux Technology Center India Software Labs, IBM Bangalore Email: [email protected] Ph: 91-80-51776329
Re: [2/3 PATCH] Kprobes: User space probes support- readpage hooks
In reply to Christoph HellwigQuoted 7 lines · Christoph Hellwig
Quoted 3 lines
A real life example of where this capability would have been very useful is with a performance problem I am currently investigating. It involves a GPFS
+ SAMBA + TCPIP + RDACthis pobablt tells more about the crappy code quality of your propritary
code than a real need for this. please argue without reference to huge
blobs of junk.
In real life there are complicated stacks; sometimes they are open source (for example, like JBoss or Tomcat), sometimes they are propietary products, sometimes they are custom applications written by the end-user. Sun has been making big hay about how with dtrace, you can easily figure out what is going on. Systemtap is a tool that will allow us to have have this kind of capability, and user space probes is part of that project.
- Ted
Re: [2/3 PATCH] Kprobes: User space probes support- readpage hooks
In reply to Christoph Hellwig72
Christoph Hellwig <[email protected]> wrote on 21/03/2006 12:15:50:
Quoted 1 line
Quoted 1 line
I think you'll find it happened the other way round. Sun openly
references
Quoted 1 line
Quoted 1 line
my white papers. They even stole the name of an ancestor to kprobes.
But
Quoted 1 line
Quoted 1 line
who cares, it not relevant or particularly interesting whether the
chicken
Quoted 3 lines
Quoted 1 line
or the egg came first.
I know your papers, too. In fact dprobes' RPN program downloads are a
far
Quoted 1 line
better design than systemtap's generation of kernel code. it's a pity
that
Quoted 2 lines
you gave up on dprobes instead of applying the required work to it and integrate it with other bits of a tracing framework.
Fascinating, gave up on dprobes, not really! I thought the kernel community felt it was the wrong implementation. We did remove all the RPN stuff to a loadable kernel module and left behind a minimal API set - krpobes - which comprised the kernel probing mechanism, user-space probes extensions and watchpoint probes extension. The result was identical functionality to the original dprobes but with a minimal patch to the mainline kernel. But in addition it provided a very much more generalized interface that would allow other utilities to exploit the kernel interface, which they have. In this sense dprobes still exists and can be used on top of krpobes. What would you recommend be retained from dprobes? And what further modifications?