ummunot: Userspace support for MMU notifications
Subsystems (5)
As discussed in <http://article.gmane.org/gmane.linux.drivers.openib/61925>
and follow-up messages, libraries using RDMA would like to track precisely when application code changes memory mapping via free(), munmap(), etc. Current pure-userspace solutions using malloc hooks and other tricks are not robust, and the feeling among experts is that the issue is unfixable without kernel help.
We solve this not by implementing the full API proposed in the email linked above but rather with a simpler and more generic interface, which may be useful in other contexts. Specifically, we implement a new character device driver, ummunot, that creates a /dev/ummunot node. A userspace process can open this node read-only and use the fd as follows:
1. ioctl() to register/unregister an address range to watch in the kernel (cf struct ummunot_register_ioctl in <linux/ummunot.h>).
2. read() to retrieve events generated when a mapping in a watched address range is invalidated (cf struct ummunot_event in <linux/ummunot.h>). select()/poll()/epoll() and SIGIO are handled for this IO.
3. mmap() one page at offset 0 to map a kernel page that contains a generation counter that is incremented each time an event is generated. This allows userspace to have a fast path that checks that no events have occurred without a system call.
Thanks to Jason Gunthorpe <[email protected]> for suggestions on the interface design. Also thanks to Jeff Squyres <[email protected]> for prototyping support for this in Open MPI, which helped find several bugs during development.
---
Andrew -- sending this to you since I don't really know who should merge this sort of thing.
Solving the underlying issue here is a pretty big deal to the universe of people that work on MPI implementations (sort of middleware for high-performance computing). I think everyone would be open to better ideas on how to handle this, although I'm pretty happy with how small and self-contained the ummunot implementation ended up being.
Anyway, review, suggestions, etc. all gratefully received.
Thanks,
Roland
drivers/char/Kconfig | 12 ++ drivers/char/Makefile | 1 + drivers/char/ummunot.c | 489 +++++++++++++++++++++++++++++++++++++++++++++++ include/linux/ummunot.h | 130 +++++++++++++ 4 files changed, 632 insertions(+), 0 deletions(-) create mode 100644 drivers/char/ummunot.c create mode 100644 include/linux/ummunot.h
drivers/char/Kconfig +12 -0
@@ -1108,6 +1108,18 @@ config DEVPORT depends on ISA || PCI default y +config UMMUNOT + tristate "Userspace MMU notifications" + select MMU_NOTIFIER + help + The ummunot (userspace MMU notification) driver creates a + character device that can be used by userspace libraries to + get notifications when an application's memory mapping + changed. This is used, for example, by RDMA libraries to + improve the reliability of memory registration caching, since + the kernel's MMU notifications can be used to know precisely + when to shoot down a cached registration. + source "drivers/s390/char/Kconfig" endmenu
drivers/char/Makefile +1 -0
@@ -97,6 +97,7 @@ obj-$(CONFIG_NSC_GPIO) += nsc_gpio.o obj-$(CONFIG_CS5535_GPIO) += cs5535_gpio.o obj-$(CONFIG_GPIO_TB0219) += tb0219.o obj-$(CONFIG_TELCLOCK) += tlclk.o +obj-$(CONFIG_UMMUNOT) += ummunot.o obj-$(CONFIG_MWAVE) += mwave/ obj-$(CONFIG_AGP) += agp/
drivers/char/ummunot.c new file +489 -0
489 lines not on this page · raw message
include/linux/ummunot.h new file +130 -0
130 lines not on this page · raw message
Signature · 1 line
1.6.3.3
Andrew Morton
maintainerQuoted 4 lines · On Wed, 22 Jul 2009 10:47:23 -0700 Roland Dreier <[email protected]> wrote:
As discussed in <http://article.gmane.org/gmane.linux.drivers.openib/61925>
and follow-up messages, libraries using RDMA would like to track precisely when application code changes memory mapping viafree(),munmap(), etc.
It isn't "precisely" - the tracker has a delayed view of the trackee, of course. Only really solvable by blocking the trackee each time it does something.
The time-skew makes the facility considerably weaker. Obviously this is OK for your application but it's a fairly important design point which it would be interesting to hear about.
Quoted 19 lines
Current pure-userspace solutions using malloc hooks
and other tricks are not robust, and the feeling among experts is that the issue is unfixable without kernel help.We solve this not by implementing the full API proposed in the email linked above but rather with a simpler and more generic interface, which may be useful in other contexts. Specifically, we implement a new character device driver, ummunot, that creates a
/dev/ummunotnode. A userspace process can open this node read-only and use the fd as follows:1.
ioctl()to register/unregister an address range to watch in the kernel (cf structummunot_register_ioctlin <linux/ummunot.h>).2.
read()to retrieve events generated when a mapping in a watched address range is invalidated (cf structummunot_eventin <linux/ummunot.h>). select()/poll()/epoll() and SIGIO are handled for this IO.
Quoted 4 lines
3.
mmap()one page at offset 0 to map a kernel page that contains a generation counter that is incremented each time an event is generated. This allows userspace to have a fast path that checks that no events have occurred without a system call.
If you stand back and squint, each of 1, 2 and 3 are things which the kernel already provides for the delivery of ftrace events to userspace.
Did you look at reusing all that stuff?
Quoted 9 lines
Thanks to Jason Gunthorpe <[email protected]> for suggestions on the interface design. Also thanks to Jeff Squyres <[email protected]> for prototyping support for this in Open MPI, which helped find several bugs during development.
Signed-off-by Roland Dreier <[email protected]>---
Andrew -- sending this to you since I don't really know who should merge this sort of thing.
That works.
Quoted 14 lines
Solving the underlying issue here is a pretty big deal to the universe of people that work on MPI implementations (sort of middleware for high-performance computing). I think everyone would be open to better ideas on how to handle this, although I'm pretty happy with how small and self-contained the ummunot implementation ended up being.
...
drivers/char/ummunot.c new file +5
-0+enum { +UMMUNOT_FLAG_DIRTY =1 , +UMMUNOT_FLAG_HINT =2 , +}; +
drivers/char/ummunot.c line 43
Quoted 10 lines
drivers/char/ummunot.c new file +10
-0+struct ummunot_reg { +u64 user_cookie; +unsigned long start; +unsigned long end; +unsigned long hint_start; +unsigned long hint_end; +unsigned long flags; +struct rb_node node; +struct list_head list; +};
Bah.
I looked at the code, asked "wtf is that rbtree being used for", went to the defining data structure and ... nothing.
And without knowing what the data structures do, and the relationship between them it's hard to follow the code. But you need to follow the code to be able to reverse-engineer the data structures.
Ho hum.
What's the rbtree for?
What's a "hint"?
drivers/char/ummunot.c line 56
Quoted 12 lines
drivers/char/ummunot.c new file +12
-0+struct ummunot_file { +struct mmu_notifier mmu_notifier; +struct mm_struct *mm; +struct rb_root reg_tree; +struct list_head dirty_list; +u64 *counter; +spinlock_t lock; +wait_queue_head_t read_wait; +struct fasync_struct *async_queue; +int need_empty; +int used; +};
What's on dirty_list?
etc.
drivers/char/ummunot.c line 64
Quoted 7 lines
drivers/char/ummunot.c new file +7
-0+static struct ummunot_file *to_ummunot_file (struct mmu_notifier *mn) +{ + return container_of(mn, struct ummunot_file, mmu_notifier); +} + +static void ummunot_handle_not (struct mmu_notifier *mn, +unsigned long start,unsigned long end)
I really really want to do s/not/notify/g on this patch ;)
Quoted 37 lines
drivers/char/ummunot.c new file +37
-0+{ +struct ummunot_file *priv =to_ummunot_file (mn); +struct rb_node *n; +struct ummunot_reg *reg; +unsigned long flags; +int hit =0 ; + +spin_lock_irqsave (&priv->lock, flags); + +for (n =rb_first (&priv->reg_tree); n; n =rb_next (n)) { + reg = rb_entry(n, struct ummunot_reg, node); + +if (reg->start >= end) +break ; + +/* +* Ranges overlap if they're not disjoint; and they're +* disjoint if the end of one is before the start of +* the other one. +*/ +if (!(reg->end <= start || end <= reg->start)) { + hit =1 ; + +if (!test_and_set_bit (UMMUNOT_FLAG_DIRTY , ®->flags)) +list_add_tail (®->list, &priv->dirty_list); + +if (test_bit (UMMUNOT_FLAG_HINT , ®->flags)) { +clear_bit (UMMUNOT_FLAG_HINT , ®->flags); + }else { +set_bit (UMMUNOT_FLAG_HINT , ®->flags); + reg->hint_start = start; + reg->hint_end = end; + } + } + } + +if (hit) {
drivers/char/ummunot.c line 111
Quoted 10 lines
drivers/char/ummunot.c new file +10
-0+ ++(*priv->counter); +flush_dcache_page (virt_to_page (priv->counter)); +wake_up_interruptible (&priv->read_wait); +kill_fasync (&priv->async_queue,SIGIO ,POLL_IN ); + } + +spin_unlock_irqrestore (&priv->lock, flags); +} + +static void ummunot_inval_page(struct mmu_notifier *mn,
"invalidate"
Quoted 15 lines
drivers/char/ummunot.c new file +15
-0+struct mm_struct *mm, + unsigned long addr) +{ + ummunot_handle_not(mn, addr, addr + PAGE_SIZE); +} + +static void ummunot_inval_range_start (struct mmu_notifier *mn, +struct mm_struct *mm, +unsigned long start,unsigned long end) +{ +ummunot_handle_not (mn, start, end); +} + +static const struct mmu_notifier_ops ummunot_mmu_notifier_ops = { + .invalidate_page = ummunot_inval_page,
drivers/char/ummunot.c line 136
Quoted 10 lines
drivers/char/ummunot.c new file +10
-0+ .invalidate_range_start = ummunot_inval_range_start, +}; + +static int ummunot_open (struct inode *inode,struct file *filp) +{ +struct ummunot_file *priv; +int ret; + +if (filp->f_mode &FMODE_WRITE ) +return -EINVAL ;
hm, how do you set the permissions on a miscdevice node?
Quoted 73 lines
drivers/char/ummunot.c new file +70
-0+ priv =kmalloc (sizeof *priv,GFP_KERNEL ); +if (!priv) +return -ENOMEM ; + + priv->counter = (void *)get_zeroed_page (GFP_KERNEL ); +if (!priv->counter) { + ret = -ENOMEM ; + goto err; + } + + priv->reg_tree =RB_ROOT ; +INIT_LIST_HEAD (&priv->dirty_list); +spin_lock_init (&priv->lock); +init_waitqueue_head (&priv->read_wait); + priv->async_queue =NULL ; + priv->need_empty =0 ; + priv->used =0 ; + + priv->mmu_notifier.ops = &ummunot_mmu_notifier_ops; +/* +* Register notifier last, since notifications can occur as +* soon as we register.... +*/ + ret =mmu_notifier_register (&priv->mmu_notifier, current->mm); +if (ret) + goto err_page; + + priv->mm = current->mm; +atomic_inc (&priv->mm->mm_count); + + filp->private_data = priv; + +return 0 ; + +err_page: +free_page ((unsigned long ) priv->counter); + +err: +kfree (priv); +return ret; +} + +static ssize_t ummunot_read (struct file *filp,char __user *buf, +size_t count,loff_t *pos) +{ +struct ummunot_file *priv = filp->private_data; +struct ummunot_reg *reg; +ssize_t ret; +struct ummunot_event *events; +int max; +int n; + + priv->used =1 ; + + events = (void *)get_zeroed_page (GFP_KERNEL ); +if (!events) { + ret = -ENOMEM ; + goto out; + } + +spin_lock_irq (&priv->lock); + +while (!ummunot_readable (priv)) { +spin_unlock_irq (&priv->lock); + +if (filp->f_flags &O_NONBLOCK ) { + ret = -EAGAIN ; + goto out; + } +
drivers/char/ummunot.c line 242
Quoted 10 lines
drivers/char/ummunot.c new file +10
-0+if (wait_event_interruptible (priv->read_wait, +ummunot_readable (priv))) { + ret = -ERESTARTSYS ; + goto out; + } + +spin_lock_irq (&priv->lock); + } + + max =min (PAGE_SIZE , count) /sizeof *events;
It used to be the case that architectures disagreed over the type of PAGE_SIZE - unsigned versus unsigned long, iirc.
If still true, this will warn.
Quoted 43 lines
drivers/char/ummunot.c new file +40
-0+for (n =0 ; n < max; ++n) { +if (list_empty (&priv->dirty_list)) { + events[n].type =UMMUNOT_EVENT_TYPE_LAST ; + events[n].user_cookie_counter = *priv->counter; + ++n; + priv->need_empty =0 ; +break ; + } + + reg = list_first_entry(&priv->dirty_list, struct ummunot_reg, + list); + + events[n].type =UMMUNOT_EVENT_TYPE_INVAL ; +if (test_bit (UMMUNOT_FLAG_HINT , ®->flags)) { + events[n].flags =UMMUNOT_EVENT_FLAG_HINT ; + events[n].hint_start =max (reg->start, reg->hint_start); + events[n].hint_end =min (reg->end, reg->hint_end); + }else { + events[n].hint_start = reg->start; + events[n].hint_end = reg->end; + } + events[n].user_cookie_counter = reg->user_cookie; + +list_del (®->list); + reg->flags =0 ; + priv->need_empty =1 ; + } + +spin_unlock_irq (&priv->lock); + +if (copy_to_user (buf, events, n *sizeof *events)) + ret = -EFAULT ; +else + ret = n *sizeof *events; + +out: +free_page ((unsigned long ) events); +return ret; +} +
Quoted 6 lines
Quoted 5 lines
As discussed in <http://article.gmane.org/gmane.linux.drivers.openib/61925>
and follow-up messages, libraries using RDMA would like to track precisely when application code changes memory mapping viafree(),munmap(), etc.
Quoted 7 lines
It isn't "precisely" - the tracker has a delayed view of the trackee, of course. Only really solvable by blocking the trackee each time it does something.
The time-skew makes the facility considerably weaker. Obviously this is OK for your application but it's a fairly important design point which it would be interesting to hear about.
Fair enough. Of course as you say "precise" is impossible if we have multiple threads running on multiple CPUs without synchronization between them. As you alluded to, for the specific use that inspired this, it would be a bug in the userspace code for any changes to occur at the moment that the library cares about things.
Quoted 13 lines
Quoted 8 lines
1.
ioctl()to register/unregister an address range to watch in the kernel (cf structummunot_register_ioctlin <linux/ummunot.h>).2.
read()to retrieve events generated when a mapping in a watched address range is invalidated (cf structummunot_eventin <linux/ummunot.h>). select()/poll()/epoll() and SIGIO are handled for this IO.Quoted 4 lines
3.
mmap()one page at offset 0 to map a kernel page that contains a generation counter that is incremented each time an event is generated. This allows userspace to have a fast path that checks that no events have occurred without a system call.
Quoted 4 lines
If you stand back and squint, each of 1, 2 and 3 are things which the kernel already provides for the delivery of ftrace events to userspace.
Did you look at reusing all that stuff?
No, not really... will investigate a bit further. Any pointers to how the ftrace stuff might work? Specifically how #3 maps to ftrace is a little obscure to me; and also as I understand it, ftrace is controlled through debugfs, which means there's a bit of hassle to make this usable on a default install. And also I'm not sure how the ftrace control path really maps to "here's a 100 address ranges I'd like events for".
So at a first glance after unsquinting a bit I'm not sure how good the fit really is.
Quoted 6 lines
I looked at the code, asked "wtf is that rbtree being used for", went to the defining data structure and ... nothing.
And without knowing what the data structures do, and the relationship between them it's hard to follow the code. But you need to follow the code to be able to reverse-engineer the data structures.
OK, will document data structures a bit more in the next version. The specific questions here:
Quoted 1 line
What's the rbtree for?
To keep track of what ranges userspace is interested in so that it's reasonably efficient to go through them in sort order and reasonably efficient to add/remove ranges too.
Quoted 1 line
What's a "hint"?
If userspace asks us to monitor a range X...Y and we get an event that invalidates some subrange W...Q then we tell userspace that only the subrange is invalid as a hint. If multiple events hit a single range, we give up and don't try to tell userspace what parts got invalidated. So it's a "hint" in the sense of "here's some information that might be helpful that we may give you, but don't count on getting it every time"
Quoted 1 line
What's on
dirty_list?
list of address ranges being watched that have been invalidated.
Quoted 1 line
I really really want to do s/not/notify/g on this patch ;)
The name is of course very provisional... I'd be happy with ummunotify or even perhaps something with _s in it to be more readable. I'll go with ummunotify next time.
Quoted 3 lines
Quoted 1 line
+static void ummunot_inval_page(struct
mmu_notifier*mn,"invalidate"
sure.
Quoted 1 line
hm, how do you set the permissions on a miscdevice node?
On my system I have a udev rule
KERNEL=="ummunot", MODE="0666"
and that seems to work. (Will need to change it to KERNEL=="ummunotify" I guess ;)
Quoted 6 lines
drivers/char/ummunot.c line 242
Quoted 1 line
drivers/char/ummunot.c new file +1
-0+ max =min (PAGE_SIZE , count) /sizeof *events;It used to be the case that architectures disagreed over the type of
PAGE_SIZE- unsigned versus unsigned long, iirc.If still true, this will warn.
Will check it out... anyway it's easy to switch to min_t() to be safe.
Thanks for the review,
Roland
Andrew Morton
maintainerQuoted 13 lines · On Wed, 22 Jul 2009 12:27:42 -0700 Roland Dreier <[email protected]> wrote:
Quoted 13 lines
Quoted 12 lines
1.
ioctl()to register/unregister an address range to watch in the kernel (cf structummunot_register_ioctlin <linux/ummunot.h>).2.
read()to retrieve events generated when a mapping in a watched address range is invalidated (cf structummunot_eventin <linux/ummunot.h>). select()/poll()/epoll() and SIGIO are handled for this IO.3.
mmap()one page at offset 0 to map a kernel page that contains a generation counter that is incremented each time an event is generated. This allows userspace to have a fast path that checks that no events have occurred without a system call.
Quoted 7 lines
Quoted 4 lines
If you stand back and squint, each of 1, 2 and 3 are things which the kernel already provides for the delivery of ftrace events to userspace.
Did you look at reusing all that stuff?
No, not really... will investigate a bit further. Any pointers to how the ftrace stuff might work?
I know who to cc ;)
Quoted 8 lines
Specifically how #3 maps to ftrace is a
little obscure to me; and also as I understand it, ftrace is controlled through debugfs, which means there's a bit of hassle to make this usable on a default install. And also I'm not sure how the ftrace control path really maps to "here's a 100 address ranges I'd like events for".So at a first glance after unsquinting a bit I'm not sure how good the fit really is.
Oh. Here was I hoping that all that code was about to become useful. <runs away>
Steven Rostedt
maintainerQuoted 11 lines · Andrew Morton
Quoted 8 lines · On Wed, 22 Jul 2009 12:27:42 -0700 Roland Dreier <[email protected]> wrote:
Quoted 8 lines
Quoted 8 lines
1.
ioctl()to register/unregister an address range to watch in the kernel (cf structummunot_register_ioctlin <linux/ummunot.h>).2.
read()to retrieve events generated when a mapping in a watched address range is invalidated (cf structummunot_eventin <linux/ummunot.h>). select()/poll()/epoll() and SIGIO are handled for this IO.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
3.
mmap()one page at offset 0 to map a kernel page that contains a generation counter that is incremented each time an event is generated. This allows userspace to have a fast path that checks that no events have occurred without a system call.
Looks like a vsyscall to me.
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines
If you stand back and squint, each of 1, 2 and 3 are things which the kernel already provides for the delivery of ftrace events to userspace.
Did you look at reusing all that stuff?
No, not really... will investigate a bit further. Any pointers to how the ftrace stuff might work?
I know who to cc ;)
You would wouldn't you ;-)
Quoted 5 lines
Quoted 5 lines
Specifically how #3 maps to ftrace is a
little obscure to me; and also as I understand it, ftrace is controlled through debugfs, which means there's a bit of hassle to make this usable on a default install. And also I'm not sure how the ftrace control path
On Fedora 11 (early ftrace kernel)
# mount -t debugfs nodev /sys/kernel/debug # ls /sys/kernel/debug/tracing available_filter_functions set_ftrace_filter available_tracers set_ftrace_notrace buffer_size_kb set_ftrace_pid current_tracer stack_max_size dyn_ftrace_total_info stack_trace failures sysprof_sample_period latency_trace trace process_follow_pid trace_marker process_trace_lifecycle trace_options process_trace_README trace_pipe process_trace_signals tracing_cpumask process_trace_syscalls tracing_enabled process_trace_taskcomm_filter tracing_max_latency process_trace_uid_filter tracing_on README tracing_thresh
Not too hard
Quoted 1 line
Quoted 1 line
really maps to "here's a 100 address ranges I'd like events for".
Now to boot into a more recent kernel:
# cd /sys/kernel/debug/tracing
# echo "ptr > 0xffffffff81100000 && ptr < 0xffffffff8113000" > events/kmem/kmalloc/filter
# echo 1 > events/kmem/kmalloc/enable
# cat events/kmem/kmalloc/filter
ptr > 0xffffffff81100000 && ptr < 0xffffffff81130000
# cat trace
# tracer: nop
#
# TASK-PID CPU# TIMESTAMP FUNCTION
# | | | | |
bash-6652 [002] 80345.390536: kmalloc: call_site=ffffffff810d76e1 ptr=ffff88001e8e7c80 bytes_req=53 bytes_alloc=64 gfp_flags=GFP_KERNEL
bash-6652 [002] 80345.390542: kmalloc: call_site=ffffffff810ba480 ptr=ffff88003c5a2700 bytes_req=32 bytes_alloc=32 gfp_flags=GFP_KERNEL
bash-6652 [002] 80345.390543: kmalloc: call_site=ffffffff810d76e1 ptr=ffff88003c5a2540 bytes_req=4 bytes_alloc=32 gfp_flags=GFP_KERNEL
bash-6652 [002] 80345.390544: kmalloc: call_site=ffffffff810ba210 ptr=ffff8800318c4d00 bytes_req=24 bytes_alloc=32 gfp_flags=GFP_KERNEL
bash-6652 [002] 80345.390550: kmalloc: call_site=ffffffff810ba480 ptr=ffff8800318c43c0 bytes_req=32 bytes_alloc=32 gfp_flags=GFP_KERNEL
bash-6652 [002] 80345.390551: kmalloc: call_site=ffffffff810d76e1 ptr=ffff8800318c4ca0 bytes_req=19 bytes_alloc=32 gfp_flags=GFP_KERNEL
bash-6652 [002] 80345.390552: kmalloc: call_site=ffffffff810ba320 ptr=ffff8800318c4d00 bytes_req=32 bytes_alloc=32 gfp_flags=GFP_KERNEL
bash-6652 [002] 80345.390552: kmalloc: call_site=ffffffff810ba210 ptr=ffff8800318c4e20 bytes_req=24 bytes_alloc=32 gfp_flags=GFP_KERNEL
bash-6652 [002] 80345.390554: kmalloc: call_site=ffffffff810ba480 ptr=ffff8800318c4400 bytes_req=32 bytes_alloc=32 gfp_flags=GFP_KERNEL
bash-6652 [002] 80345.390555: kmalloc: call_site=ffffffff810d76e1 ptr=ffff8800318c4380 bytes_req=4 bytes_alloc=32 gfp_flags=GFP_KERNEL
bash-6652 [002] 80345.390555: kmalloc: call_site=ffffffff810ba210 ptr=ffff8800318c42e0 bytes_req=24 bytes_alloc=32 gfp_flags=GFP_KERNEL
bash-6652 [002] 80345.390561: kmalloc: call_site=ffffffff810ba480 ptr=ffff8800318c4d20 bytes_req=32 bytes_alloc=32 gfp_flags=GFP_KERNEL
bash-6652 [002] 80345.390562: kmalloc: call_site=ffffffff810d76e1 ptr=ffff8800318c4240 bytes_req=19 bytes_alloc=32 gfp_flags=GFP_KERNEL
bash-6652 [002] 80345.390563: kmalloc: call_site=ffffffff810ba320 ptr=ffff8800318c42e0 bytes_req=32 bytes_alloc=32 gfp_flags=GFP_KERNEL
bash-6652 [002] 80345.390563: kmalloc: call_site=ffffffff810ba320 ptr=ffff8800318c4e20 bytes_req=32 bytes_alloc=32 gfp_flags=GFP_KERNEL
bash-6652 [002] 80345.390566: kmalloc: call_site=ffffffff810bb0ee ptr=ffff88003d4e2680 bytes_req=176 bytes_alloc=192 gfp_flags=GFP_KERNEL|GFP_ZERO
bash-6652 [002] 80345.390566: kmalloc: call_site=ffffffff810d76e1 ptr=ffff8800318c42c0 bytes_req=4 bytes_alloc=32 gfp_flags=GFP_KERNEL
bash-6652 [002] 80345.390570: kmalloc: call_site=ffffffff810d76e1 ptr=ffff8800285405c0 bytes_req=4 bytes_alloc=32 gfp_flags=GFP_KERNEL
bash-6652 [002] 80345.390574: kmalloc: call_site=ffffffff810bb320 ptr=ffff88003d4e2680 bytes_req=176 bytes_alloc=192 gfp_flags=GFP_KERNEL|GFP_ZEROQuoted 3 lines
Quoted 3 lines
So at a first glance after unsquinting a bit I'm not sure how good the fit really is.
Well, if you need to add hooks, definitely at least use tracepoints. (see the TRACE_EVENT code in include/trace/events/*.h)
I'm not exactly sure what requirements you have, but it may be something we can work together on. Eliminate some duplicate code, or at least, ftrace can piggy back on it ;-)
Quoted 3 lines
Oh. Here was I hoping that all that code was about to become useful. <runs away>
You better run!
-- Steve
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
1.
ioctl()to register/unregister an address range to watch in the kernel (cf structummunot_register_ioctlin <linux/ummunot.h>).2.
read()to retrieve events generated when a mapping in a watched address range is invalidated (cf structummunot_eventin <linux/ummunot.h>). select()/poll()/epoll() and SIGIO are handled for this IO.3.
mmap()one page at offset 0 to map a kernel page that contains a generation counter that is incremented each time an event is generated. This allows userspace to have a fast path that checks that no events have occurred without a system call.
Quoted 7 lines
Quoted 4 lines
If you stand back and squint, each of 1, 2 and 3 are things which the kernel already provides for the delivery of ftrace events to userspace.
Did you look at reusing all that stuff?
No, not really... will investigate a bit further. Any pointers to how the ftrace stuff might work?
This can be implemented as a tracer or trace events, though I'm not sure if it can fully meet your requirements or not.
To implement it as trace events, we add tracepoints in mmu_inlivadate_xxx(), and define some TRACE_EVENT() macros.
And below shows how to use it:
# mount -t debugfs xxx /mnt
# cd /mnt/tracing/
# echo 1 > events/mmu/enable
# echo '(start >= 10000000 && end <= 10004096) || \
(start >= 20000000 && end <= 20004096)' > events/mmu/filter
# cat trace_pipe
bash-2066 [001] 795.239077: mmu_invalidate_range_start: start=10000000 end=10000100
bash-2066 [001] 795.239091: mmu_invalidate_range_start: start=10000000 end=10000100
bash-2066 [001] 795.239098: mmu_invalidate_range_start: start=10000000 end=10000100
cat-2189 [001] 795.239502: mmu_invalidate_page: start=20000000 end=20003000
cat-2189 [001] 795.239578: mmu_invalidate_page: start=20000000 end=20003000
bash-2066 [001] 795.239626: mmu_invalidate_page: start=20000000 end=20003000The patch is extremely simple:
--- include/linux/mmu_notifier.h | 4 +++ include/trace/events/mmu.h | 55 ++++++++++++++++++++++++++++++++++++++++++ mm/mmu_notifier.c | 3 ++ 3 files changed, 62 insertions(+), 0 deletions(-) create mode 100644 include/trace/events/mmu.h
include/linux/mmu_notifier.h +4 -0
@@ -5,6 +5,8 @@#include <linux/spinlock.h> #include <linux/mm_types.h> +#include <trace/events/mmu.h> +struct mmu_notifier ;struct mmu_notifier_ops ;
@@ -178,6 +180,7 @@ static inline int mmu_notifier_clear_flush_young(struct mm_struct *mm,static inline void mmu_notifier_invalidate_page (struct mm_struct *mm,unsigned long address) { +trace_mmu_invalidate_page (address);if (mm_has_notifiers (mm))__mmu_notifier_invalidate_page (mm, address); }
@@ -185,6 +188,7 @@ static inline void mmu_notifier_invalidate_page(struct mm_struct *mm,static inline void mmu_notifier_invalidate_range_start (struct mm_struct *mm,unsigned long start,unsigned long end) { +trace_mmu_invalidate_range_start (start, end);if (mm_has_notifiers (mm))__mmu_notifier_invalidate_range_start (mm, start, end); }
include/trace/events/mmu.h new file +55 -0
55 lines not on this page · raw message
mm/mmu_notifier.c +3 -0
@@ -17,6 +17,9 @@#include <linux/rcupdate.h> #include <linux/sched.h> +#define CREATE_TRACE_POINTS +#include <trace/events/mmu.h> +/* * This function can't run concurrently against mmu_notifier_register * because mm->mm_users > 0 during mmu_notifier_register and exit_mmap
Signature · 1 line
1.6.3
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
3.
mmap()one page at offset 0 to map a kernel page that contains a generation counter that is incremented each time an event is generated. This allows userspace to have a fast path that checks that no events have occurred without a system call.Looks like a vsyscall to me.
Yes, in a way, although it is quite a bit simpler in the sense that it doesn't require any arch-specific code (or indeed any code mapped from the kernel) and is automatically available in a portable way. Implementing this as a vsyscall seems as if it would add a lot of complexity to the kernel side without much simplification on the userspace side (in fact, hooking up the vsyscall is probably more code than just doing mmap() + dereferencing a pointer).
Quoted 2 lines
# mount -t debugfs nodev /sys/kernel/debug # ls /sys/kernel/debug/tracing
The use case I have in mind is for unprivileged user applications to use this. So requiring debugfs to be mounted hurts (since that isn't done by default), and using the files in tracing really hurts, since they're currently created with mode 0644 and so tracing can't be controlled by unprivileged users.
[ASIDE: why is trace_marker created with the strange permission of 0220 when it is owned by root:root -- is there any reason for the group write permission, or should it just be 0200 permission?]
In fact the whole model of ftrace seems to be a single privileged user controlling a single context; the use case for ummunotify is that a lot of processes running unprivileged (and possibly as multiple different users) each want to get events for parts of their own address space.
So
Quoted 1 line
# echo "ptr > 0xffffffff81100000 && ptr < 0xffffffff8113000" > events/kmem/kmalloc/filter
is very cool; but what I would want is for a given process to be able to say "please give me events for ptr in the following 100 ranges A..B, C..D, ..." and "oh and add this range X..Y" and "oh don't give me events for C..D anymore". And that process should only get events about its own address range; and 10 other (unprivileged) processes should be able to do the same thing simultaneously.
Also is there a raw format for setting the filters that lets userspace swap them atomically (ie change from filter A to filter B with a guarantee that filter A is in effect right up to the time filter B is in effect with no window where eg no filter is in effect).
Quoted 2 lines
Well, if you need to add hooks, definitely at least use tracepoints. (see the
TRACE_EVENTcode ininclude/trace/events/*.h)
I don't think I'm adding hooks -- the mmu notifier infrastructure already suits me perfectly. The only thing I'm doing is forwarding the events delivered by mmu notifiers up to userspace, but not really in a way that's very close to what ftrace does (I don't think).
It seems handling multiple unprivileged contexts accessing different streams of trace events is going to require pretty huge ftrace changes. And ummunotify is currently about 400 lines of code total (+ 300 lines of comments :) so we're not going to simplify that code dramatically. The hope I guess would be that a common interface would make things conceptually simpler, but I don't see how to slot ftrace and ummunotify together very cleanly.
Thanks,
Roland
Quoted 22 lines
This can be implemented as a tracer or trace events, though I'm not sure if it can fully meet your requirements or not.
To implement it as trace events, we add tracepoints in
mmu_inlivadate_xxx(), and define someTRACE_EVENT()macros.And below shows how to use it:
# mount -t debugfs xxx /mnt # cd /mnt/tracing/ # echo 1 > events/mmu/enable # echo '(start >= 10000000 && end <= 10004096) || \ (start >= 20000000 && end <= 20004096)' > events/mmu/filter # cat trace_pipe bash-2066 [001] 795.239077: mmu_invalidate_range_start: start=10000000 end=10000100 bash-2066 [001] 795.239091: mmu_invalidate_range_start: start=10000000 end=10000100 bash-2066 [001] 795.239098: mmu_invalidate_range_start: start=10000000 end=10000100 cat-2189 [001] 795.239502: mmu_invalidate_page: start=20000000 end=20003000 cat-2189 [001] 795.239578: mmu_invalidate_page: start=20000000 end=20003000 bash-2066 [001] 795.239626: mmu_invalidate_page: start=20000000 end=20003000
Quoted 1 line
The patch is extremely simple:
Thanks... unfortunately I don't think this really helps for the use case I'm trying to address. What's desired is a way for an unprivileged userspace process to monitor (parts of) its own address space; I think getting a single stream of all such events is not going to be helpful, because, first, the overhead of filtering out the subset of useful events may be too high to make this useful and, second, exposing events about unrelated processes is probably a security hole (eg "allocation channel" attacks on crypto processes analogous to timing channel attacks)
If there were some way for each process to get a trace of its own events then that would be very close to what I'm trying to implement.
Thanks,
Roland
Steven Rostedt
maintainerQuoted 8 lines · Roland Dreier
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
3.
mmap()one page at offset 0 to map a kernel page that contains a generation counter that is incremented each time an event is generated. This allows userspace to have a fast path that checks that no events have occurred without a system call.Looks like a vsyscall to me.
Quoted 7 lines
Yes, in a way, although it is quite a bit simpler in the sense that it doesn't require any arch-specific code (or indeed any code mapped from the kernel) and is automatically available in a portable way. Implementing this as a vsyscall seems as if it would add a lot of complexity to the kernel side without much simplification on the userspace side (in fact, hooking up the vsyscall is probably more code than just doing
mmap()+ dereferencing a pointer).
Just making an observation, not really suggesting that we convert this to a vsyscall.
Quoted 9 lines
Quoted 3 lines
# mount -t debugfs nodev /sys/kernel/debug # ls /sys/kernel/debug/tracingThe use case I have in mind is for unprivileged user applications to use this. So requiring debugfs to be mounted hurts (since that isn't done by default), and using the files in tracing really hurts, since they're currently created with mode 0644 and so tracing can't be controlled by unprivileged users.
Ah, allowing unprivileged users is a big commitment. That is, everything that you handle must be considered untrusted in all accounts. The thing about tracing inside the kernel, is that enabling it, may affect everyone. Thus we can not simply allow unprivileged users to go messing with the performance of others.
Quoted 4 lines
[ASIDE: why is
trace_markercreated with the strange permission of 0220 when it is owned by root:root -- is there any reason for the group write permission, or should it just be 0200 permission?]
Good question. Probably just an oversite. debugfs is kind of funny with its permissions.
Quoted 22 lines
In fact the whole model of ftrace seems to be a single privileged user controlling a single context; the use case for ummunotify is that a lot of processes running unprivileged (and possibly as multiple different users) each want to get events for parts of their own address space.
So
Quoted 1 line
# echo "ptr > 0xffffffff81100000 && ptr < 0xffffffff8113000" > events/kmem/kmalloc/filteris very cool; but what I would want is for a given process to be able to say "please give me events for ptr in the following 100 ranges A..B, C..D, ..." and "oh and add this range X..Y" and "oh don't give me events for C..D anymore". And that process should only get events about its own address range; and 10 other (unprivileged) processes should be able to do the same thing simultaneously.
Also is there a raw format for setting the filters that lets userspace swap them atomically (ie change from filter A to filter B with a guarantee that filter A is in effect right up to the time filter B is in effect with no window where eg no filter is in effect).
Quoted 7 lines
Quoted 2 lines
Well, if you need to add hooks, definitely at least use tracepoints. (see the
TRACE_EVENTcode ininclude/trace/events/*.h)I don't think I'm adding hooks -- the mmu notifier infrastructure already suits me perfectly. The only thing I'm doing is forwarding the events delivered by mmu notifiers up to userspace, but not really in a way that's very close to what ftrace does (I don't think).
OK, I didn't look at the code enough to know.
Quoted 8 lines
It seems handling multiple unprivileged contexts accessing different streams of trace events is going to require pretty huge ftrace changes. And ummunotify is currently about 400 lines of code total (+ 300 lines of comments :) so we're not going to simplify that code dramatically. The hope I guess would be that a common interface would make things conceptually simpler, but I don't see how to slot ftrace and ummunotify together very cleanly.
I agree, what you are doing is out of the scope of ftrace. But perhaps someday this may change. Currently because of security issues, we keep ftrace a privileged user function. But we've been discussing changing this someday. But any change would need to be scrutinized for security sake. Ftrace gives a user a peak into what is going on inside the kernel. And with that, it theoretically, gives a way to get around kernel security measures.
-- Steve
PATCH v2 1/1 ummunotify: Userspace support for MMU notifications
In reply to Andrew MortonAs discussed in <http://article.gmane.org/gmane.linux.drivers.openib/61925>
and follow-up messages, libraries using RDMA would like to track precisely when application code changes memory mapping via free(), munmap(), etc. Current pure-userspace solutions using malloc hooks and other tricks are not robust, and the feeling among experts is that the issue is unfixable without kernel help.
We solve this not by implementing the full API proposed in the email linked above but rather with a simpler and more generic interface, which may be useful in other contexts. Specifically, we implement a new character device driver, ummunotify, that creates a /dev/ummunotify node. A userspace process can open this node read-only and use the fd as follows:
1. ioctl() to register/unregister an address range to watch in the kernel (cf struct ummunotify_register_ioctl in <linux/ummunotify.h>).
2. read() to retrieve events generated when a mapping in a watched address range is invalidated (cf struct ummunotify_event in <linux/ummunotify.h>). select()/poll()/epoll() and SIGIO are handled for this IO.
3. mmap() one page at offset 0 to map a kernel page that contains a generation counter that is incremented each time an event is generated. This allows userspace to have a fast path that checks that no events have occurred without a system call.
Thanks to Jason Gunthorpe <[email protected]> for suggestions on the interface design. Also thanks to Jeff Squyres <[email protected]> for prototyping support for this in Open MPI, which helped find several bugs during development.
---
Hi Andrew,
since last time I've
- agreed (I think) with Steve that this doesn't really fit in with (current) ftrace code.
- s/ummunot/ummunotify/
- documented the main datastructures
- clarified a few other names (inval -> invalidate, dirty -> invalid)
- used min_t to handle taking min of size_t and PAGE_SIZE
which I think covers everything you mentioned last time.
What else do you think needs to happen to move this upstream?
Thanks,
Roland
drivers/char/Kconfig | 12 + drivers/char/Makefile | 1 + drivers/char/ummunotify.c | 554 ++++++++++++++++++++++++++++++++++++++++++++ include/linux/ummunotify.h | 121 ++++++++++ 4 files changed, 688 insertions(+), 0 deletions(-) create mode 100644 drivers/char/ummunotify.c create mode 100644 include/linux/ummunotify.h
drivers/char/Kconfig +12 -0
@@ -1108,6 +1108,18 @@ config DEVPORT depends on ISA || PCI default y +config UMMUNOTIFY + tristate "Userspace MMU notifications" + select MMU_NOTIFIER + help + The ummunotify (userspace MMU notification) driver creates a + character device that can be used by userspace libraries to + get notifications when an application's memory mapping + changed. This is used, for example, by RDMA libraries to + improve the reliability of memory registration caching, since + the kernel's MMU notifications can be used to know precisely + when to shoot down a cached registration. + source "drivers/s390/char/Kconfig" endmenu
drivers/char/Makefile +1 -0
@@ -97,6 +97,7 @@ obj-$(CONFIG_NSC_GPIO) += nsc_gpio.o obj-$(CONFIG_CS5535_GPIO) += cs5535_gpio.o obj-$(CONFIG_GPIO_TB0219) += tb0219.o obj-$(CONFIG_TELCLOCK) += tlclk.o +obj-$(CONFIG_UMMUNOTIFY) += ummunotify.o obj-$(CONFIG_MWAVE) += mwave/ obj-$(CONFIG_AGP) += agp/
drivers/char/ummunotify.c new file +554 -0
554 lines not on this page · raw message
include/linux/ummunotify.h new file +121 -0
121 lines not on this page · raw message
Signature · 1 line
1.6.3.3
Andrew Morton
maintainerRe: [PATCH v2] ummunotify: Userspace support for MMU notifications
Quoted 35 lines · On Fri, 24 Jul 2009 15:56:17 -0700 Roland Dreier <[email protected]> wrote:
As discussed in <http://article.gmane.org/gmane.linux.drivers.openib/61925>
and follow-up messages, libraries using RDMA would like to track precisely when application code changes memory mapping viafree(),munmap(), etc. Current pure-userspace solutions using malloc hooks and other tricks are not robust, and the feeling among experts is that the issue is unfixable without kernel help.We solve this not by implementing the full API proposed in the email linked above but rather with a simpler and more generic interface, which may be useful in other contexts. Specifically, we implement a new character device driver, ummunotify, that creates a
/dev/ummunotifynode. A userspace process can open this node read-only and use the fd as follows:1.
ioctl()to register/unregister an address range to watch in the kernel (cf structummunotify_register_ioctlin <linux/ummunotify.h>).2.
read()to retrieve events generated when a mapping in a watched address range is invalidated (cf structummunotify_eventin <linux/ummunotify.h>). select()/poll()/epoll() and SIGIO are handled for this IO.3.
mmap()one page at offset 0 to map a kernel page that contains a generation counter that is incremented each time an event is generated. This allows userspace to have a fast path that checks that no events have occurred without a system call.Thanks to Jason Gunthorpe <[email protected]> for suggestions on the interface design. Also thanks to Jeff Squyres <[email protected]> for prototyping support for this in Open MPI, which helped find several bugs during development.
...
drivers/char/Kconfig +1
-0+config UMMUNOTIFY
drivers/char/Kconfig config DEVPORT · line 1121
Quoted 10 lines
drivers/char/Kconfig +10
-0+ tristate "Userspace MMU notifications" + select MMU_NOTIFIER + help + The ummunotify (userspace MMU notification) driver creates a + character device that can be used by userspace libraries to + get notifications when an application's memory mapping + changed. This is used, for example, by RDMA libraries to + improve the reliability of memory registration caching, since + the kernel's MMU notifications can be used to know precisely + when to shoot down a cached registration.
Does `select' dtrt here if UMMUNOTIFY=m? I never trust it...
<searches in vain for ummunotify.txt>
Oh well :(
A little test app would be nice - I assume you have one. We could toss in in the tree as a how-to-use example, and people could perhaps turn it into a regression test - perhaps the LTP people would take it.
drivers/char/ummunotify.c line 158
Quoted 7 lines
...
drivers/char/ummunotify.c new file +4
-0+if (test_bit (UMMUNOTIFY_FLAG_HINT , ®->flags)) { +clear_bit (UMMUNOTIFY_FLAG_HINT , ®->flags); + }else { +set_bit (UMMUNOTIFY_FLAG_HINT , ®->flags);
It's a shame that change_bit() didn't return the old (or new) value.
The overall userspace interface seems a bit klunky, but I can't really suggest anything better. Netlink delivery?
Re: [PATCH v2] ummunotify: Userspace support for MMU notifications
Quoted 6 lines
drivers/char/Kconfig config DEVPORT · line 1113
Quoted 4 lines
drivers/char/Kconfig +3
-0+config UMMUNOTIFY + tristate "Userspace MMU notifications" + select MMU_NOTIFIERDoes `select' dtrt here if UMMUNOTIFY=m? I never trust it...
Yes, it appears to when I test... and all the other users (KVM, SGI_GRU) of MMU_NOTIFIER are tristates that select it, so in practice it seems to work as well.
Quoted 7 lines
<searches in vain for ummunotify.txt>
Oh well :(
A little test app would be nice - I assume you have one. We could toss in in the tree as a how-to-use example, and people could perhaps turn it into a regression test - perhaps the LTP people would take it.
I'll write up a bit of doc and stick it in Documentation/ummunotify.txt, and create samples/ummunotify/ as well for the next iteration.
Quoted 6 lines
drivers/char/ummunotify.c line 158
Quoted 4 lines
drivers/char/ummunotify.c new file +4
-0+if (test_bit (UMMUNOTIFY_FLAG_HINT , ®->flags)) { +clear_bit (UMMUNOTIFY_FLAG_HINT , ®->flags); + }else { +set_bit (UMMUNOTIFY_FLAG_HINT , ®->flags);It's a shame that
change_bit()didn't return the old (or new) value.
Heh, I didn't know about change_bit(). <looks> ... seems like test_and_change_bit() is exactly what would fit here. Will update.
Quoted 2 lines
The overall userspace interface seems a bit klunky, but I can't really suggest anything better. Netlink delivery?
I took a quick look at netlink but discarded the idea when I saw that netlink sockets don't support mmap() ... seems that plumbing in mmap support (to handle the kernel exported generation count) is probably more trouble than it's worth, and I'm not sure that all the netlink encapsulation is that much of a win here anyway. I'd prefer to proceed with the "virtual character device" interface -- another option would be sys_ummunotify_create and sys_ummunotify_ctl but I can't really imagine this being worth another couple syscalls.
- R.
PATCH v3 1/1 ummunotify: Userspace support for MMU notifications
Quoted 1 line
From
8b067027a0fc5d3280fe3f006d936d6d775de297Mon Sep 17 00:00:00 2001
From: Roland Dreier <[email protected]>
Date: Fri, 31 Jul 2009 11:47:53 -0700
Subject: [PATCH] ummunotify: Userspace support for MMU notifications
As discussed in <http://article.gmane.org/gmane.linux.drivers.openib/61925>
and follow-up messages, libraries using RDMA would like to track precisely when application code changes memory mapping via free(), munmap(), etc. Current pure-userspace solutions using malloc hooks and other tricks are not robust, and the feeling among experts is that the issue is unfixable without kernel help.
We solve this not by implementing the full API proposed in the email linked above but rather with a simpler and more generic interface, which may be useful in other contexts. Specifically, we implement a new character device driver, ummunotify, that creates a /dev/ummunotify node. A userspace process can open this node read-only and use the fd as follows:
1. ioctl() to register/unregister an address range to watch in the kernel (cf struct ummunotify_register_ioctl in <linux/ummunotify.h>).
2. read() to retrieve events generated when a mapping in a watched address range is invalidated (cf struct ummunotify_event in <linux/ummunotify.h>). select()/poll()/epoll() and SIGIO are handled for this IO.
3. mmap() one page at offset 0 to map a kernel page that contains a generation counter that is incremented each time an event is generated. This allows userspace to have a fast path that checks that no events have occurred without a system call.
Thanks to Jason Gunthorpe <[email protected]> for suggestions on the interface design. Also thanks to Jeff Squyres <[email protected]> for prototyping support for this in Open MPI, which helped find several bugs during development.
---
Hi Andrew,
This should replace the ummunotify patch you're carrying in -mm (if you'd prefer an incremental patch, let me know).
Changes since v2:
- Added Documentation/ummunotify/ with a text file and a test program (hooked up to CONFIG_BUILD_DOCSRC, fixed things like hooking up ummunotify.h to headers_install)
- Integrated Andrew's checkpatch fixes (no more > 80 char lines in kernel source; userspace test code has some long lines due to not wanting to split printf formats)
- Clean up "if (test_bit) { clear_bit } else { set_bit }" -- code was actually buggy since we don't want to reset the bit after we cleared it (ie 3 events in a row)
Documentation/Makefile | 3 +- Documentation/ummunotify/Makefile | 7 + Documentation/ummunotify/ummunotify.txt | 150 ++++++++ Documentation/ummunotify/umn-test.c | 200 +++++++++++ drivers/char/Kconfig | 12 + drivers/char/Makefile | 1 + drivers/char/ummunotify.c | 566 +++++++++++++++++++++++++++++++ include/linux/ummunotify.h | 121 +++++++ 8 files changed, 1059 insertions(+), 1 deletions(-) create mode 100644 Documentation/ummunotify/Makefile create mode 100644 Documentation/ummunotify/ummunotify.txt create mode 100644 Documentation/ummunotify/umn-test.c create mode 100644 drivers/char/ummunotify.c create mode 100644 include/linux/ummunotify.h
Documentation/Makefile +2 -1
@@ -1,3 +1,4 @@ obj-m := DocBook/ accounting/ auxdisplay/ connector/ \ filesystems/configfs/ ia64/ networking/ \ - pcmcia/ spi/ video4linux/ vm/ watchdog/src/ + pcmcia/ spi/ video4linux/ vm/ ummunotify/ \ + watchdog/src/
Documentation/ummunotify/Makefile new file +7 -0
@@ -0,0 +1,7 @@ +# List of programs to build +hostprogs-y := umn-test + +# Tell kbuild to always build the programs +always := $(hostprogs-y) + +HOSTCFLAGS_umn-test.o += -I$(objtree)/usr/include
Documentation/ummunotify/ummunotify.txt new file +150 -0
150 lines not on this page · raw message
Documentation/ummunotify/umn-test.c new file +200 -0
200 lines not on this page · raw message
drivers/char/Kconfig +12 -0
@@ -1108,6 +1108,18 @@ config DEVPORT depends on ISA || PCI default y +config UMMUNOTIFY + tristate "Userspace MMU notifications" + select MMU_NOTIFIER + help + The ummunotify (userspace MMU notification) driver creates a + character device that can be used by userspace libraries to + get notifications when an application's memory mapping + changed. This is used, for example, by RDMA libraries to + improve the reliability of memory registration caching, since + the kernel's MMU notifications can be used to know precisely + when to shoot down a cached registration. + source "drivers/s390/char/Kconfig" endmenu
drivers/char/Makefile +1 -0
@@ -97,6 +97,7 @@ obj-$(CONFIG_NSC_GPIO) += nsc_gpio.o obj-$(CONFIG_CS5535_GPIO) += cs5535_gpio.o obj-$(CONFIG_GPIO_TB0219) += tb0219.o obj-$(CONFIG_TELCLOCK) += tlclk.o +obj-$(CONFIG_UMMUNOTIFY) += ummunotify.o obj-$(CONFIG_MWAVE) += mwave/ obj-$(CONFIG_AGP) += agp/
drivers/char/ummunotify.c new file +566 -0
566 lines not on this page · raw message
include/linux/ummunotify.h new file +121 -0
121 lines not on this page · raw message
Signature · 1 line
1.6.3.3
Re: [PATCH v3] ummunotify: Userspace support for MMU notifications
Quoted 14 lines · Roland Dreier
As discussed in <http://article.gmane.org/gmane.linux.drivers.openib/61925>
and follow-up messages, libraries using RDMA would like to track precisely when application code changes memory mapping viafree(),munmap(), etc. Current pure-userspace solutions using malloc hooks and other tricks are not robust, and the feeling among experts is that the issue is unfixable without kernel help.We solve this not by implementing the full API proposed in the email linked above but rather with a simpler and more generic interface, which may be useful in other contexts. Specifically, we implement a new character device driver, ummunotify, that creates a
/dev/ummunotifynode. A userspace process can open this node read-only and use the fd as follows:
Quoted 8 lines
1.
ioctl()to register/unregister an address range to watch in the kernel (cf structummunotify_register_ioctlin <linux/ummunotify.h>).2.
read()to retrieve events generated when a mapping in a watched address range is invalidated (cf structummunotify_eventin <linux/ummunotify.h>). select()/poll()/epoll() and SIGIO are handled for this IO.
Hello Roland,
I like the interface but I have a couple questions:
1) Why does userspace have to register these address ranges? I would have just reported all invalidation evens and let user-space check which ones are interesting. My feeling is that the number of invalidation events will usually be lower than the number registered ranges, so you'll report more events through the file descriptor, but userspace will do a lot less ioctls.
2) What happens in case of fork? If father+child keep reading from the previously-open /dev/ummunotify, each event will be delivered only to the first reader, right? Fork is always a mess in HPC, but maybe there's something to do here.
3) What's userspace supposed to do if 2 libraries need such events in the same process? Should each of them open /dev/ummunotify separately? Doesn't matter much for performance, just wondering.
thanks,
Brice
Re: [PATCH v3] ummunotify: Userspace support for MMU notifications
Quoted 2 lines
I like the interface but I have a couple questions:
Thanks.
Quoted 6 lines
1) Why does userspace have to register these address ranges? I would have just reported all invalidation evens and let user-space check which ones are interesting. My feeling is that the number of invalidation events will usually be lower than the number registered ranges, so you'll report more events through the file descriptor, but userspace will do a lot less ioctls.
A couple of reasons. First, MMU notifier events may be delivered (in the kernel) in interrupt context so the amount of allocation we can do in a notifier hook is limited (and any allocation will fail sometimes). So if we just want to report all events to userspace then I don't see any was around having to sometimes deliver an event like "uh, some events got lost" and have userspace have to flush everything.
I suspect that MPI workloads will hit the overflow case in practice, since they probably want to run as close to out-of-memory as possible, and the application may not enter the MPI library often enough to keep the queue of ummunotify events short -- I can imagine some codes that do a lot of memory management, enter MPI infrequently, and end up overflowing the queue and flushing all registrations over and over. Having userspace register ranges means I can preallocate a landing area for each event and make the MMU notifier hook pretty simple.
Second, it turns out that having the filter does cut down quite a bit on the events. From running some Open MPI tests that Jeff provided, I saw that there were often several times as many MMU notifier events delivered in the kernel than ended up being reported to userspace.
Quoted 4 lines
2) What happens in case of fork? If father+child keep reading from the previously-open
/dev/ummunotify, each event will be delivered only to the first reader, right? Fork is always a mess in HPC, but maybe there's something to do here.
It works just like any other file where fork results in two file descriptors in two processes... as you point out the two processes can step on each other. (And in the ummunotify case the file remains associated with the original mm) However this is the case for simpler stuff like sockets etc too, and I think uniformity of interface and least surprise say that ummunotify should follow the same model.
Quoted 3 lines
3) What's userspace supposed to do if 2 libraries need such events in the same process? Should each of them open
/dev/ummunotifyseparately? Doesn't matter much for performance, just wondering.
I guess the libraries could work out some way to share things, but that would require one library to pass events to the other or something like that. It should work fine for 2 libraries to have independent ummunotify files open though (I've not tested but "what could go wrong"?).
Thanks,
Roland
Re: [PATCH v3] ummunotify: Userspace support for MMU notifications
Quoted 9 lines · Roland Dreier
I suspect that MPI workloads will hit the overflow case in practice, since they probably want to run as close to out-of-memory as possible, and the application may not enter the MPI library often enough to keep the queue of ummunotify events short -- I can imagine some codes that do a lot of memory management, enter MPI infrequently, and end up overflowing the queue and flushing all registrations over and over. Having userspace register ranges means I can preallocate a landing area for each event and make the MMU notifier hook pretty simple.
Thanks, I see.
Quoted 5 lines
Second, it turns out that having the filter does cut down quite a bit on the events. From running some Open MPI tests that Jeff provided, I saw that there were often several times as many MMU notifier events delivered in the kernel than ended up being reported to userspace.
So maybe multiple invalidate_page are gathered into the same range event? If so, maybe it'd make sense to cache the last used rb_node in ummunotify_handle_notify()? (and if multiple ranges were invalidated at once, just don't cache anything, it shouldn't happen often anyway)
Quoted 5 lines
Quoted 4 lines
2) What happens in case of fork? If father+child keep reading from the previously-open
/dev/ummunotify, each event will be delivered only to the first reader, right? Fork is always a mess in HPC, but maybe there's something to do here.
Quoted 7 lines
It works just like any other file where fork results in two file descriptors in two processes... as you point out the two processes can step on each other. (And in the ummunotify case the file remains associated with the original mm) However this is the case for simpler stuff like sockets etc too, and I think uniformity of interface and least surprise say that ummunotify should follow the same model.
I was wondering if adding a special event such as "COWED" could help user-space. But maybe fork already invalidates all COW'ed ranges in copy_page_range() anyway?
Brice
Re: [PATCH v3] ummunotify: Userspace support for MMU notifications
Quoted 10 lines
Quoted 5 lines
Second, it turns out that having the filter does cut down quite a bit on the events. From running some Open MPI tests that Jeff provided, I saw that there were often several times as many MMU notifier events delivered in the kernel than ended up being reported to userspace.
So maybe multiple
invalidate_pageare gathered into the same range event? If so, maybe it'd make sense to cache the last usedrb_nodeinummunotify_handle_notify()? (and if multiple ranges were invalidated at once, just don't cache anything, it shouldn't happen often anyway)
Well, I just meant that there were lots of events for parts of the address space that Open MPI wasn't interested in... the fortran runtime or whatever was freeing stuff that wasn't ever used for communication.
Quoted 12 lines
Quoted 11 lines
Quoted 4 lines
2) What happens in case of fork? If father+child keep reading from the previously-open
/dev/ummunotify, each event will be delivered only to the first reader, right? Fork is always a mess in HPC, but maybe there's something to do here.It works just like any other file where fork results in two file descriptors in two processes... as you point out the two processes can step on each other. (And in the ummunotify case the file remains associated with the original mm) However this is the case for simpler stuff like sockets etc too, and I think uniformity of interface and least surprise say that ummunotify should follow the same model.
Quoted 3 lines
I was wondering if adding a special event such as "COWED" could help user-space. But maybe fork already invalidates all COW'ed ranges in
copy_page_range()anyway?
The problem I guess is that there is only one file object (pointed to by two file descriptors of course) after the fork. And it is tracking changes to the parent's mapping. I guess in the parent when touching pages and triggering COW, that might be interesting -- but I don't really know how to distinguish it from any other type of invalidate event (and I don't know how userspace would do anything different anyway).
I haven't actually looked at what fork() does to the MMU notifiers -- but the MMU notifiers hook in at such a low level that it does seem hard to know that what's going on is related to fork or even COW.
- R.