ummunotify: Userspace support for MMU notifications

I am resubmitting this patch because I believe that the discussion has shown this to be an acceptable solution. I have fixed the 32 bit build errors, but other than that change, the code is the same as Roland's V3 patch.

From: Roland Dreier <[email protected]>

As discussed in <http://article.gmane.org/gmane.linux.drivers.openib/61925>
and follow-up messages, libraries using RDMA would like to track precisely when application code changes memory mapping via free(), munmap(), etc. Current pure-userspace solutions using malloc hooks and other tricks are not robust, and the feeling among experts is that the issue is unfixable without kernel help.

Andrew,

I am resubmitting this patch because I believe that the discussion has shown this to be an acceptable solution.

I have fixed the 32 bit
build errors, but other than that change, the code is the same as Roland's V3 patch.

From: Roland Dreier <[email protected]>

As discussed in <http://article.gmane.org/gmane.linux.drivers.openib/61925>
and follow-up messages, libraries using RDMA would like to track precisely when application code changes memory mapping via free(), munmap(), etc. Current pure-userspace solutions using malloc hooks and other tricks are not robust, and the feeling among experts is that the issue is unfixable without kernel help.

We solve this not by implementing the full API proposed in the email linked above but rather with a simpler and more generic interface, which may be useful in other contexts. Specifically, we implement a new character device driver, ummunotify, that creates a /dev/ummunotify node. A userspace process can open this node read-only and use the fd as follows:

1. ioctl() to register/unregister an address range to watch in the kernel (cf struct ummunotify_register_ioctl in <linux/ummunotify.h>).

2. read() to retrieve events generated when a mapping in a watched address range is invalidated (cf struct ummunotify_event in <linux/ummunotify.h>). select()/poll()/epoll() and SIGIO are handled for this IO.

3. mmap() one page at offset 0 to map a kernel page that contains a generation counter that is incremented each time an event is generated. This allows userspace to have a fast path that checks that no events have occurred without a system call.

Thanks to Jason Gunthorpe <jgunthorpe <at> obsidianresearch.com> for suggestions on the interface design. Also thanks to Jeff Squyres <jsquyres <at> cisco.com> for prototyping support for this in Open MPI, which helped find several bugs during development.

Signed-off-by Roland Dreier <[email protected]>

Signed-off-by Eric B Munson <[email protected]>

---

Changes since v3:
- Fixed replaced [get|put] user with copy_[from|to]_user to fix x86 builds

---
 Documentation/Makefile                  |    3 +-
 Documentation/ummunotify/Makefile       |    7 +
 Documentation/ummunotify/ummunotify.txt |  150 ++++++++
 Documentation/ummunotify/umn-test.c     |  200 +++++++++++
 drivers/char/Kconfig                    |   12 +
 drivers/char/Makefile                   |    1 +
 drivers/char/ummunotify.c               |  567 +++++++++++++++++++++++++++++++
 include/linux/ummunotify.h              |  121 +++++++
 8 files changed, 1060 insertions(+), 1 deletions(-)
 create mode 100644 Documentation/ummunotify/Makefile
 create mode 100644 Documentation/ummunotify/ummunotify.txt
 create mode 100644 Documentation/ummunotify/umn-test.c
 create mode 100644 drivers/char/ummunotify.c
 create mode 100644 include/linux/ummunotify.h

Documentation/Makefile +2 -1

index 6fc7ea1..27ba76a 100644
@@ -1,3 +1,4 @@
 obj-m := DocBook/ accounting/ auxdisplay/ connector/ \
 	filesystems/ filesystems/configfs/ ia64/ laptops/ networking/ \
-	pcmcia/ spi/ timers/ video4linux/ vm/ watchdog/src/
+	pcmcia/ spi/ timers/ video4linux/ vm/ ummunotify/ \
+	watchdog/src/

Documentation/ummunotify/Makefile new file +7 -0

new file mode 100644
index 0000000..89f31a0
@@ -0,0 +1,7 @@
+# List of programs to build
+hostprogs-y := umn-test
+
+# Tell kbuild to always build the programs
+always := $(hostprogs-y)
+
+HOSTCFLAGS_umn-test.o += -I$(objtree)/usr/include

Documentation/ummunotify/ummunotify.txt new file +64 -0

new file mode 100644
index 0000000..78a79c2
@@ -0,0 +1,150 @@
+UMMUNOTIFY
+
+  Ummunotify relays MMU notifier events to userspace.  This is useful
+  for libraries that need to track the memory mapping of applications;
+  for example, MPI implementations using RDMA want to cache memory
+  registrations for performance, but tracking all possible crazy cases
+  such as when, say, the FORTRAN runtime frees memory is impossible
+  without kernel help.
+
+Basic Model
+
+  A userspace process uses it by opening /dev/ummunotify, which
+  returns a file descriptor.  Interest in address ranges is registered
+  using ioctl() and MMU notifier events are retrieved using read(), as
+  described in more detail below.  Userspace can register multiple
+  address ranges to watch, and can unregister individual ranges.
+
+  Userspace can also mmap() a single read-only page at offset 0 on
+  this file descriptor.  This page contains (at offest 0) a single
+  64-bit generation counter that the kernel increments each time an
+  MMU notifier event occurs.  Userspace can use this to very quickly
+  check if there are any events to retrieve without needing to do a
+  system call.
+
+Control
+
+  To start using ummunotify, a process opens /dev/ummunotify in
+  read-only mode.  Control from userspace is done via ioctl(); the
+  defined ioctls are:
+
+    UMMUNOTIFY_EXCHANGE_FEATURES: This ioctl takes a single 32-bit
+      word of feature flags as input, and the kernel updates the
+      features flags word to contain only features requested by
+      userspace and also supported by the kernel.
+
+      This ioctl is only included for forward compatibility; no
+      feature flags are currently defined, and the kernel will simply
+      update any requested feature mask to 0.  The kernel will always
+      default to a feature mask of 0 if this ioctl is not used, so
+      current userspace does not need to perform this ioctl.
+
+    UMMUNOTIFY_REGISTER_REGION: Userspace uses this ioctl to tell the
+      kernel to start delivering events for an address range.  The
+      range is described using struct ummunotify_register_ioctl:
+
+	struct ummunotify_register_ioctl {
+		__u64	start;
+		__u64	end;
+		__u64	user_cookie;
+		__u32	flags;
+		__u32	reserved;
+	};
+
+      start and end give the range of userspace virtual addresses;
+      start is included in the range and end is not, so an example of
+      a 4 KB range would be start=0x1000, end=0x2000.
+
+      user_cookie is an opaque 64-bit quantity that is returned by the
+      kernel in events involving the range, and used by userspace to
+      stop watching the range.  Each registered address range must
+      have a distinct user_cookie.
+
+      It is fine with the kernel if userspace registers multiple
+      overlapping or even duplicate address ranges, as long as a

Documentation/ummunotify/ummunotify.txt new file +8 -0

+  Userspace may also
+  set the ummunotify file descriptor to non-blocking mode and use all
+  standard ways of waiting for data to be available on the ummunotify
+  file descriptor, including epoll/poll()/select() and SIGIO.
+
+  The format of event reports is:
+
+	struct ummunotify_event {

Quoted 6 lines

As discussed in <http://article.gmane.org/gmane.linux.drivers.openib/61925>
and follow-up messages, libraries using RDMA would like to track precisely when application code changes memory mapping via free(), munmap(), etc. Current pure-userspace solutions using malloc hooks and other tricks are not robust, and the feeling among experts is that the issue is unfixable without kernel help.

But this info could be reassembled by tracking syscall activity, yes? Perhaps some discussion here explaining why the (possibly enhanced) ptrace, audit, etc interfaces are unsuitable.

Documentation/ummunotify/ummunotify.txt line 150

Quoted 6 lines

Documentation/ummunotify/ummunotify.txt new file +6 -0

+  Userspace can use the generation counter as a quick check to avoid
+  system calls; if the value read from the mapped kernel counter is
+  still equal to the value returned in user_cookie_counter for the
+  most recent LAST event retrieved, then no further events have been
+  queued and there is no need to try a read() on the ummunotify file
+  descriptor.

I _guess_ that works OK on 32-bit, as long as userspace _only_ compares this value with some previous one.

umm, no, there's still a race I think. If the counter increases from 0x00000000ffffffff to 0x0000000100000000 then userspace could see this as two events when using this scheme.

Quoted 6 lines · Andrew Morton

Quoted 6 lines

As discussed in <http://article.gmane.org/gmane.linux.drivers.openib/61925>
and follow-up messages, libraries using RDMA would like to track precisely when application code changes memory mapping via free(), munmap(), etc. Current pure-userspace solutions using malloc hooks and other tricks are not robust, and the feeling among experts is that the issue is unfixable without kernel help.

I am not sure I agree with the premises here. ptMalloc and malloc hooks are not related to the issue in my opinion. User space library calls do not change virtual to physical mapping, system calls do. The following sys calls might change virtual to physical mapping:
munmap(), mremap(), sbrk(), madvice(). What we need is glibc to provide hooks for these 4 sys calls and the general syscall() when its argument is one of the four mentioned syscalls. To me, that is what is needed, and the ummunotify direction seems way too complicated to me.

It is further claimed that "… other tricks are not robust". I wrote the code used in Scali/Platform MPI handling the issue. I do not think its fair to claim that this MPI is not robust in this matter nor that is performance is bad.

Are those system calls the only possible way that virtual to physical mappings can change? Can't page migration or something like that potentially affect things? And even if you did have hooks into every system call that mattered (keep in mind that relying on glibc is not enough, since an MPI application may not use glibc) would decoding them and figuring out what happened really be preferable to a single event type that tells you exactly what address range was affected?

Yeah, virtual<->physical maps can change through swapping, page migration, memory compaction, huge-page aggregation (the latter two not yet being upstream).

Even mlock() doesn't pin virtual<->physical maps.

Quoted 7 lines · Roland Dreier

Are those system calls the only possible way that virtual to physical mappings can change? Can't page migration or something like that potentially affect things? And even if you did have hooks into every system call that mattered (keep in mind that relying on glibc is not enough, since an MPI application may not use glibc) would decoding them and figuring out what happened really be preferable to a single event type that tells you exactly what address range was affected?

Yeah, virtual<->physical maps can change through swapping, page migration, memory compaction, huge-page aggregation (the latter two not yet being upstream).

Even mlock() doesn't pin virtual<->physical maps.

Quoted 13 lines · Peter Zijlstra

Quoted 7 lines · Roland Dreier

Are those system calls the only possible way that virtual to physical mappings can change? Can't page migration or something like that potentially affect things? And even if you did have hooks into every system call that mattered (keep in mind that relying on glibc is not enough, since an MPI application may not use glibc) would decoding them and figuring out what happened really be preferable to a single event type that tells you exactly what address range was affected?

Yeah, virtual<->physical maps can change through swapping, page migration, memory compaction, huge-page aggregation (the latter two not yet being upstream).

Quoted 1 line

Even mlock() doesn't pin virtual<->physical maps.

Pages registered for RDMA are GUPed so no method above should touch them. Fork+cow or unmap/map on the other hand can change virtual<->physical maps. GUPed pages are still GUPed, but they are no longer mapped into process' virtual address space. MPI copes with Fork+cow by marking registered memory as MADV_DONTFORK.

Quoted 4 lines

It is further claimed that "… other tricks are not robust". I wrote the code used in Scali/Platform MPI handling the issue. I do not think its fair to claim that this MPI is not robust in this matter nor that is performance is bad.

The Open MPI developers have spent a lot of effort trying to handle this purely in userspace and still do not believe that a truly robust solution is possible without kernel help. Perhaps they can expand on what the obstacles are.

Quoted 4 lines

The Open MPI developers have spent a lot of effort trying to handle this purely in userspace and still do not believe that a truly robust solution is possible without kernel help. Perhaps they can expand on what the obstacles are.

Quoted 4 lines · On Mon, 12 Apr 2010 07:22:17 +0100 Eric B Munson…

Andrew,

I am resubmitting this patch because I believe that the discussion has shown this to be an acceptable solution.

To whom? Some acked-by's would clarify.

Quoted 4 lines

I have fixed the 32 bit
build errors, but other than that change, the code is the same as Roland's V3 patch.

Quoted 8 lines

From: Roland Dreier <[email protected]>

As discussed in <http://article.gmane.org/gmane.linux.drivers.openib/61925>
and follow-up messages, libraries using RDMA would like to track precisely when application code changes memory mapping via free(), munmap(), etc. Current pure-userspace solutions using malloc hooks and other tricks are not robust, and the feeling among experts is that the issue is unfixable without kernel help.

But this info could be reassembled by tracking syscall activity, yes? Perhaps some discussion here explaining why the (possibly enhanced) ptrace, audit, etc interfaces are unsuitable.

Quoted 15 lines

We solve this not by implementing the full API proposed in the email linked above but rather with a simpler and more generic interface, which may be useful in other contexts. Specifically, we implement a new character device driver, ummunotify, that creates a /dev/ummunotify node. A userspace process can open this node read-only and use the fd as follows:

1. ioctl() to register/unregister an address range to watch in the kernel (cf struct ummunotify_register_ioctl in <linux/ummunotify.h>).

2. read() to retrieve events generated when a mapping in a watched address range is invalidated (cf struct ummunotify_event in <linux/ummunotify.h>). select()/poll()/epoll() and SIGIO are handled for this IO.

Quoted 4 lines

3. mmap() one page at offset 0 to map a kernel page that contains a generation counter that is incremented each time an event is generated. This allows userspace to have a fast path that checks that no events have occurred without a system call.

OK, what's missing from this whole description and from ummunotify.txt
is: how does one specify the target process? Does /dev/ummunotify implicitly attach to current->mm? If so, why, and what are the implications of this?

If instead it is possible to attach to some other process's mmu activity (/proc/<pid>/ummunotity?) then how is that done and what are the security/permissions implications?

Also, the whole thing is obviously racy: by the time userspace finds out that something has happened, it might have changed. This inevitably reduces the applicability/usefulness of the whole thing as compared to some synchronous mechanism which halts the monitored thread until the request has been processed and acked. All this should (IMO) be explored, explained and justified.

Also, what prevents the obvious DoS which occurs when I register for events and just let them queue up until the kernel runs out of memory? presumably events get dropped - what are the reliability implications of this and how is the max queue length managed?

Also, ioctls are unpopular. Were other intefaces considered?

From: Randy Dunlap <[email protected]>

Add ummunotify.h to Kbuild list for export to userspace, fixing 27 build errors in umn-test.c when O=builddir is used.

Signed-off-by Randy Dunlap <[email protected]>