Cross Memory Attach

The basic idea behind cross memory attach is to allow MPI programs doing intra-node communication to do a single copy of the message rather than a double copy of the message via shared memory.

The following patch attempts to achieve this by allowing a destination process, given an address and size from a source process, to copy memory directly from the source process into its own address space via a system call. There is also a symmetrical ability to copy from the current process's address space into a destination process's address space.

Use of vmsplice instead was considered, but has problems. Since you need the reader and writer working co-operatively if the pipe is not drained then you block. Which requires some wrapping to do non blocking on the send side or polling on the receive. In all to all communication it requires ordering otherwise you can deadlock. And in the example of many MPI tasks writing to one MPI task vmsplice serialises the copying.

I've added the use of this capability to OpenMPI and run some MPI benchmarks on a 64-way (with SMT off) Power6 machine which see improvements in the following areas:

HPCC results:
=============

MB/s Num Processes
Naturally Ordered 4 8 16 32
Base 1235 935 622 419
CMA 4741 3769 1977 703

MB/s Num Processes
Randomly Ordered 4 8 16 32
Base 1227 947 638 412
CMA 4666 3682 1978 710

MB/s Num Processes
Max Ping Pong 4 8 16 32
Base 2028 1938 1928 1882
CMA 7424 7510 7598 7708

NPB:
====
BT - 12% improvement
FT - 15% improvement
IS - 30% improvement
SP - 34% improvement

IMB:
===

Ping Pong - ~30% improvement
Ping Ping - ~120% improvement
SendRecv - ~100% improvement
Exchange - ~150% improvement
Gather(v) - ~20% improvement
Scatter(v) - ~20% improvement
AlltoAll(v) - 30-50% improvement

Patch is as below. Any comments?

Quoted 11 lines

NPB:
====
BT - 12% improvement
FT - 15% improvement
IS - 30% improvement
SP - 34% improvement

IMB:
===

Quoted 7 lines

Ping Pong - ~30% improvement
Ping Ping - ~120% improvement
SendRecv - ~100% improvement
Exchange - ~150% improvement
Gather(v) - ~20% improvement
Scatter(v) - ~20% improvement
AlltoAll(v) - 30-50% improvement

btw., how does OpenMPI signal the target tasks that something happened to their address space - is there some pipe/socket side-channel, or perhaps purely based on flags in the modified memory areas, which are polled?

What did those OpenMPI facilities use before your patch - shared memory or sockets?

I have an observation about the interface:

A small detail: 'int flags' should probably be 'unsigned long flags'
- it leaves more space.

Also, note that there is a further performance optimization possible
here: if the other task's ->mm is the same as this task's (they share the MM), then the copy can be done straight in this process context, without GUP. User-space might not necessarily be aware of this so it might make sense to express this special case in the kernel too.

More fundamentally, wouldnt it make sense to create an iovec interface here? If the Gather(v) / Scatter(v) / AlltoAll(v) workloads have any fragmentation on the user-space buffer side then the copy of multiple areas could be done in a single syscall. (the MM lock has to be touched only once, target task only be looked up only once, etc.)

Plus, a small naming detail, shouldnt the naming be more IO like:

sys_process_vm_read()
sys_process_vm_write()

The basic idea behind cross memory attach is to allow MPI programs doing intra-node communication to do a single copy of the message rather than a double copy of the message via shared memory.

The following patch attempts to achieve this by allowing a destination process, given an address and size from a source process, to copy memory directly from the source process into its own address space via a system call. There is also a symmetrical ability to copy from the current process's address space into a destination process's address space.

Quoted 3 lines · Christopher Yeoh

The basic idea behind cross memory attach is to allow MPI programs doing intra-node communication to do a single copy of the message rather than a double copy of the message via shared memory.

If the host has a dma engine (many modern ones do) you can reduce this to zero copies (at least, zero processor copies).

Quoted 6 lines

The following patch attempts to achieve this by allowing a destination process, given an address and size from a source process, to copy memory directly from the source process into its own address space via a system call. There is also a symmetrical ability to copy from the current process's address space into a destination process's address space.

Instead of those two syscalls, how about a vmfd(pid_t pid, ulong start, ulong len) system call which returns an file descriptor that represents a portion of the process address space. You can then use preadv() and pwritev() to copy memory, and io_submit(IO_CMD_PREADV) and io_submit(IO_CMD_PWRITEV) for asynchronous variants (especially useful with a dma engine, since that adds latency).

With some care (and use of mmu_notifiers) you can even mmap() your vmfd and access remote process memory directly.

A nice property of file descriptors is that you can pass them around securely via SCM_RIGHTS. So a process can create a window into its address space and pass it to other processes.

(or you could just use a shared memory object and pass it around)

Quoted 21 lines · Avi Kivity

Quoted 3 lines · Christopher Yeoh

The basic idea behind cross memory attach is to allow MPI programs doing intra-node communication to do a single copy of the message rather than a double copy of the message via shared memory.

If the host has a dma engine (many modern ones do) you can reduce this to zero copies (at least, zero processor copies).

Quoted 6 lines

The following patch attempts to achieve this by allowing a destination process, given an address and size from a source process, to copy memory directly from the source process into its own address space via a system call. There is also a symmetrical ability to copy from the current process's address space into a destination process's address space.

Instead of those two syscalls, how about a vmfd(pid_t pid, ulong start, ulong len) system call which returns an file descriptor that represents a portion of the process address space. You can then use preadv() and pwritev() to copy memory, and io_submit(IO_CMD_PREADV) and io_submit(IO_CMD_PWRITEV) for asynchronous variants (especially useful with a dma engine, since that adds latency).

Quoted 8 lines

With some care (and use of mmu_notifiers) you can even mmap() your vmfd and access remote process memory directly.

A nice property of file descriptors is that you can pass them around securely via SCM_RIGHTS. So a process can create a window into its address space and pass it to other processes.

(or you could just use a shared memory object and pass it around)

Interesting, but how will that work in a scalable way with lots of non-thread tasks?

Say we have 100 processes. We'd have to have 100 fd's - each has to be passed to a new worker process.

In that sense a PID is just as good of a reference as an fd - it can be looked up lockless, etc. - but has the added advantage that it can be passed along just by number.

Quoted 3 lines · Christopher Yeoh

The basic idea behind cross memory attach is to allow MPI programs doing intra-node communication to do a single copy of the message rather than a double copy of the message via shared memory.

If the host has a dma engine (many modern ones do) you can reduce this to zero copies (at least, zero processor copies).

Instead of those two syscalls, how about a vmfd(pid_t pid, ulong start, ulong len) system call which returns an file descriptor that represents a portion of the process address space. You can then use preadv() and pwritev() to copy memory, and io_submit(IO_CMD_PREADV) and io_submit(IO_CMD_PWRITEV) for asynchronous variants (especially useful with a dma engine, since that adds latency).

With some care (and use of mmu_notifiers) you can even mmap() your vmfd and access remote process memory directly.

Quoted 5 lines · On Wed, 15 Sep 2010 12:58:15 +0200 Avi Kivity <[email protected]> wrote:

Quoted 3 lines · Christopher Yeoh

The basic idea behind cross memory attach is to allow MPI programs doing intra-node communication to do a single copy of the message rather than a double copy of the message via shared memory.

Quoted 2 lines

If the host has a dma engine (many modern ones do) you can reduce this to zero copies (at least, zero processor copies).

Yes, this interface doesn't really support that. I've tried to keep things really simple here, but I see potential for increasing level/complexity of support with diminishing returns:

3. ability to map part of another process's address space directly into the current one. Would have setup/tear down overhead, but this would be useful specifically for reduction operations where we don't even need to really copy the data once at all, but use it directly in arithmetic/logical operations on the receiver.

Quoted 5 lines

3. ability to map part of another process's address space directly into the current one. Would have setup/tear down overhead, but this would be useful specifically for reduction operations where we don't even need to really copy the data once at all, but use it directly in arithmetic/logical operations on the receiver.

Don't even think about this. If you want to map another tasks memory, use shared memory. The shared memory code knows about that. The races for anything else are crazy.

Quoted 9 lines · On Wed, 15 Sep 2010 12:58:15 +0200 Avi Kivity <[email protected]> wrote:

Quoted 4 lines · Christopher Yeoh

The basic idea behind cross memory attach is to allow MPI programs doing intra-node communication to do a single copy of the message rather than a double copy of the message via shared memory.

If the host has a dma engine (many modern ones do) you can reduce this to zero copies (at least, zero processor copies).

Yes, this interface doesn't really support that. I've tried to keep things really simple here, but I see potential for increasing level/complexity of support with diminishing returns:

1. single copy (basically what the current implementation does)
2. support for async dma offload (rather arch specific)
3. ability to map part of another process's address space directly into the current one. Would have setup/tear down overhead, but this would be useful specifically for reduction operations where we don't even need to really copy the data once at all, but use it directly in arithmetic/logical operations on the receiver.

For reference, there is also knem http://runtime.bordeaux.inria.fr/knem/
which does implement (2) for I/OAT, though it looks to me the interface and implementation are relatively speaking quite a bit more complex.

I am the guy doing KNEM so I can comment on this. The I/OAT part of KNEM was mostly a research topic, it's mostly useless on current machines since the memcpy performance is much larger than I/OAT DMA Engine. We also have an offload model with a kernel thread, but it wasn't used a lot so far. These features can be ignored for the current discussion.

Quoted 6 lines · Brice Goglin

I am the guy doing KNEM so I can comment on this. The I/OAT part of KNEM was mostly a research topic, it's mostly useless on current machines since the memcpy performance is much larger than I/OAT DMA Engine. We also have an offload model with a kernel thread, but it wasn't used a lot so far. These features can be ignored for the current discussion.

I've just created a knem branch where I removed all the above, and some other stuff that are not necessary for normal users. So it just contains the region management code and two commands to copy between regions or between a region and some local iovecs.

Instead of those two syscalls, how about a vmfd(pid_t pid, ulong start, ulong len) system call which returns an file descriptor that represents a portion of the process address space.  You can then use preadv() and pwritev() to copy memory, and io_submit(IO_CMD_PREADV) and io_submit(IO_CMD_PWRITEV) for asynchronous variants (especially useful with a dma engine, since that adds latency).

With some care (and use of mmu_notifiers) you can even mmap() your vmfd and access remote process memory directly.

Quoted 7 lines · Avi Kivity

Instead of those two syscalls, how about a vmfd(pid_t pid, ulong start, ulong len) system call which returns an file descriptor that represents a portion of the process address space. You can then use preadv() and pwritev() to copy memory, and io_submit(IO_CMD_PREADV) and io_submit(IO_CMD_PWRITEV) for asynchronous variants (especially useful with a dma engine, since that adds latency).

Quoted 2 lines

With some care (and use of mmu_notifiers) you can even mmap() your vmfd and access remote process memory directly.

Rather than introducing a new vmfd() API for this, why not just add implementations for these more efficient operations to the existing /proc/$pid/mem interface?

Quoted 9 lines · Bryan Donlan

Quoted 7 lines · Avi Kivity

Instead of those two syscalls, how about a vmfd(pid_t pid, ulong start, ulong len) system call which returns an file descriptor that represents a portion of the process address space. You can then use preadv() and pwritev() to copy memory, and io_submit(IO_CMD_PREADV) and io_submit(IO_CMD_PWRITEV) for asynchronous variants (especially useful with a dma engine, since that adds latency).

Quoted 5 lines

Quoted 2 lines

With some care (and use of mmu_notifiers) you can even mmap() your vmfd and access remote process memory directly.

Rather than introducing a new vmfd() API for this, why not just add implementations for these more efficient operations to the existing /proc/$pid/mem interface?

Yes, opening that file should be equivalent (and you could certainly implement aio via dma for it).

Quoted 8 lines · Avi Kivity

Instead of those two syscalls, how about a vmfd(pid_t pid, ulong start, ulong len) system call which returns an file descriptor that represents a portion of the process address space.  You can then use preadv() and pwritev() to copy memory, and io_submit(IO_CMD_PREADV) and io_submit(IO_CMD_PWRITEV) for asynchronous variants (especially useful with a dma engine, since that adds latency).

Quoted 2 lines

With some care (and use of mmu_notifiers) you can even mmap() your vmfd and access remote process memory directly.

Rather than introducing a new vmfd() API for this, why not just add implementations for these more efficient operations to the existing /proc/$pid/mem interface?

Quoted 13 lines · On Wed, 15 Sep 2010 23:46:09 +0900 Bryan Donlan<[email protected]> wrote:

Quoted 10 lines · Avi Kivity

Instead of those two syscalls, how about a vmfd(pid_t pid, ulong start, ulong len) system call which returns an file descriptor that represents a portion of the process address space. You can then use preadv() and pwritev() to copy memory, and io_submit(IO_CMD_PREADV) and io_submit(IO_CMD_PWRITEV) for asynchronous variants (especially useful with a dma engine, since that adds latency).

With some care (and use of mmu_notifiers) you can even mmap() your vmfd and access remote process memory directly.

Quoted 3 lines

Rather than introducing a new vmfd() API for this, why not just add implementations for these more efficient operations to the existing /proc/$pid/mem interface?

Perhaps I'm misunderstanding something here, but accessing /proc/$pid/mem requires ptracing the target process. We can't really have all these MPI processes ptraceing each other just to send/receive a message....

Quoted 16 lines · Christopher Yeoh

Quoted 13 lines · On Wed, 15 Sep 2010 23:46:09 +0900 Bryan Donlan<[email protected]> wrote:

Quoted 10 lines · Avi Kivity

Instead of those two syscalls, how about a vmfd(pid_t pid,
ulong start, ulong len) system call which returns an file descriptor that represents a portion of the process address space. You can then use preadv() and pwritev() to copy memory, and io_submit(IO_CMD_PREADV) and io_submit(IO_CMD_PWRITEV) for asynchronous variants (especially useful with a dma engine, since that adds latency).

With some care (and use of mmu_notifiers) you can even mmap()
your vmfd and access remote process memory directly.

Quoted 9 lines

Quoted 3 lines

Rather than introducing a new vmfd() API for this, why not just
add implementations for these more efficient operations to the existing /proc/$pid/mem interface?

Perhaps I'm misunderstanding something here, but accessing /proc/$pid/mem requires ptracing the target process. We can't really have all these MPI processes ptraceing each other just to send/receive a message....

You could have each process open /proc/self/mem and pass the fd using SCM_RIGHTS.

That eliminates a race; with copy_to_process(), by the time the pid is looked up it might designate a different process.

Quoted 6 lines

You could have each process open /proc/self/mem and pass the fd using SCM_RIGHTS.

That eliminates a race; with copy_to_process(), by the time the pid is looked up it might designate a different process.

Just to revive an old thread (I've been on holidays), but this doesn't work either. the ptrace check is done by mem_read (eg on each read) so even if you do pass the fd using SCM_RIGHTS, reads on the fd still fail.

So unless there's good reason to believe that the ptrace permission check is no longer needed, the /proc/pid/mem interface doesn't seem to be an option for what we want to do.

Quoted 8 lines

Instead of those two syscalls, how about a vmfd(pid_t pid, ulong start, ulong len) system call which returns an file descriptor that represents a portion of the process address space.  You can then use preadv() and pwritev() to copy memory, and io_submit(IO_CMD_PREADV) and io_submit(IO_CMD_PWRITEV) for asynchronous variants (especially useful with a dma engine, since that adds latency).

Quoted 2 lines

With some care (and use of mmu_notifiers) you can even mmap() your vmfd and access remote process memory directly.

Rather than introducing a new vmfd() API for this, why not just add implementations for these more efficient operations to the existing /proc/$pid/mem interface?

Quoted 9 lines · Avi Kivity

Quoted 8 lines

Instead of those two syscalls, how about a vmfd(pid_t pid, ulong start, ulong len) system call which returns an file descriptor that represents a portion of the process address space. You can then use preadv() and pwritev() to copy memory, and io_submit(IO_CMD_PREADV) and io_submit(IO_CMD_PWRITEV) for asynchronous variants (especially useful with a dma engine, since that adds latency).

Quoted 6 lines

Quoted 2 lines

With some care (and use of mmu_notifiers) you can even mmap() your vmfd and access remote process memory directly.

Rather than introducing a new vmfd() API for this, why not just add implementations for these more efficient operations to the existing /proc/$pid/mem interface?

As far as I heared from my friend, old HP MPI implementation used /proc/$pid/mem for this purpose. (I don't know current status). However almost implementation doesn't do that because /proc/$pid/mem required the process is ptraced. As far as I understand , very old /proc/$pid/mem doesn't require it. but It changed for security concern. Then, Anybody haven't want to change this interface because they worry break security.

But, I don't know what exactly protected "the process is ptraced" check. If anyone explain the reason and we can remove it. I'm not againt at all.

+
+               if (copy_to) {
+                       ret = copy_from_user(target_kaddr,
+                                            user_buf + *bytes_copied,
+                                            bytes_to_copy);
+                       if (ret) {
+                               kunmap(process_pages[i]);
+                               goto end;
+                       }
+               } else {

+                       ret = copy_to_user(user_buf + *bytes_copied,
+                                          target_kaddr, bytes_to_copy);
+                       if (ret) {
+                               kunmap(process_pages[i]);
+                               goto end;
+                       }
+               }
+               kunmap(process_pages[i]);

Btw, please just do this as

  if (copy_to)
     ret = copy_from_user(..);
  else
     ret = copy_to_user(..);
  kunmap(process_pages[i]);
  if (ret)
     goto out;

In fact, you might consider passing in a "copy_out" function pointer rather than that "copy_to" boolean, and rather than that conditional, just do a

  ret = copy_out(..);

thing. On sane/good architectures, branch target prediction will make it all work out to the same work in the end, and it certainly looks simpler and cleaner.

   if (copy_to)
      ret = copy_from_user(..);
   else
      ret = copy_to_user(..);
   kunmap(process_pages[i]);
   if (ret)
      goto out; 

Quoted 7 lines · Linus Torvalds

   if (copy_to)
      ret = copy_from_user(..);
   else
      ret = copy_to_user(..);
   kunmap(process_pages[i]);
   if (ret)
      goto out;

Shouldn't we be using kmap_atomic()

Quoted 10 lines · Peter Zijlstra

Quoted 7 lines · Linus Torvalds

   if (copy_to)
      ret = copy_from_user(..);
   else
      ret = copy_to_user(..);
   kunmap(process_pages[i]);
   if (ret)
      goto out;

Quoted 1 line

Shouldn't we be using kmap_atomic()

Over a copy_to/from_user? Not bloody likely.

Quoted 2 lines

Over a copy_to/from_user? Not bloody likely.

Gah, indeed. OK, since its not nested kmap() should indeed work. The alternative is using get_user_pages() on both address spaces, but I guess that makes things unnecessarily complex.

Of course, these days I would seriously suggest against trying to optimize the kmap() case. It only matters on crap hardware these days. Anybody running HIGHMEM in 2010 and thinks that it makes sense deserves the pain the get. We should not complicate the kernel further for it, and sane architectures will have a no-op kmap().

Quoted 6 lines · Linus Torvalds

Of course, these days I would seriously suggest against trying to optimize the kmap() case. It only matters on crap hardware these days. Anybody running HIGHMEM in 2010 and thinks that it makes sense deserves the pain the get. We should not complicate the kernel further for it, and sane architectures will have a no-op kmap().

OK, fully agreed. Someone ought to tell ARM though :-)

But there is no way I'm
going to say "oh, HIGHMEM still makes sense in 2010 because the ARM guys are now making all the same mistakes Intel did in 1992".

Quoted 3 lines · Linus Torvalds

But there is no way I'm
going to say "oh, HIGHMEM still makes sense in 2010 because the ARM guys are now making all the same mistakes Intel did in 1992".

Off by a couple of years. Intel did it wit the PPro, in 1995, actually. But it's still going to be 16 years later by the time ARM LPAE actually ships, I guess.

From: [email protected]
Date: Thu, 16 Sep 2010 10:54:29 -0700
Subject: Re: [RFC][PATCH] Cross Memory Attach
To: [email protected]
CC: [email protected]; [email protected]

Quoted 9 lines · Peter Zijlstra

Quoted 6 lines · Linus Torvalds

Of course, these days I would seriously suggest against trying to optimize the kmap() case. It only matters on crap hardware these days. Anybody running HIGHMEM in 2010 and thinks that it makes sense deserves the pain the get. We should not complicate the kernel further for it, and sane architectures will have a no-op kmap().

OK, fully agreed. Someone ought to tell ARM though :-)

You know what? I don't care. If the fact that ARM is messing up means that they will never be able to do well in the micro-server space, that's _their_ problem.

I fought HIGHMEM tooth and nail when it appeared originally. I lost, because we really didn't have any choice. But there is no way I'm going to say "oh, HIGHMEM still makes sense in 2010 because the ARM guys are now making all the same mistakes Intel did in 1992". Because these days we _do_ have a choice.

And all the rumors are that there will be a 64-bit ARM too. So their PAE mess will be out before, but nobody sane should really consider it a primary issue. It will work, but it will work suboptimally. That's what you get when you have bad hardware design.

Linus
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

(Adding Russell King of ARM Linux to CC list)

----------------------------------------

Quoted 40 lines

From: [email protected]
Date: Thu, 16 Sep 2010 10:54:29 -0700
Subject: Re: [RFC][PATCH] Cross Memory Attach
To: [email protected]
CC: [email protected]; [email protected]

Quoted 9 lines · Peter Zijlstra

Quoted 6 lines · Linus Torvalds

Of course, these days I would seriously suggest against trying to optimize the kmap() case. It only matters on crap hardware these days. Anybody running HIGHMEM in 2010 and thinks that it makes sense deserves the pain the get. We should not complicate the kernel further for it, and sane architectures will have a no-op kmap().

OK, fully agreed. Someone ought to tell ARM though :-)

You know what? I don't care. If the fact that ARM is messing up means that they will never be able to do well in the micro-server space, that's _their_ problem.

I fought HIGHMEM tooth and nail when it appeared originally. I lost, because we really didn't have any choice. But there is no way I'm going to say "oh, HIGHMEM still makes sense in 2010 because the ARM guys are now making all the same mistakes Intel did in 1992". Because these days we _do_ have a choice.

And all the rumors are that there will be a 64-bit ARM too. So their PAE mess will be out before, but nobody sane should really consider it a primary issue. It will work, but it will work suboptimally. That's what you get when you have bad hardware design.

Linus
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 2 lines

(Adding Russell King of ARM Linux to CC list)

I'm not sure why you're repeatedly sending me this email (this is the second in less than 24 hours.)

In any case, it's not like I can influence the direction ARM Ltd take their architecture - I only hear about stuff after the decisions have been taken and the direction set. (Sometimes I hear about stuff just before the public announcement.)

So, if you think I can do anything to stop the move towards LPAS (large physical address space) and tell ARM to go to 64-bit directly, you're sadly mistaken.

Of course, these days I would seriously suggest against trying to optimize the kmap() case. It only matters on crap hardware these days. Anybody running HIGHMEM in 2010 and thinks that it makes sense deserves the pain the get. We should not complicate the kernel further for it, and sane architectures will have a no-op kmap().

The basic idea behind cross memory attach is to allow MPI programs doing intra-node communication to do a single copy of the message rather than a double copy of the message via shared memory.

mm/memory.c +7 -0

+	/* Get the pages we're interested in */
+	pages_pinned = get_user_pages(task, task->mm, pa,
+				      nr_pages_to_copy,
+				      copy_to, 0, process_pages, NULL);
+
+	if (pages_pinned != nr_pages_to_copy)
+		goto end;

Interesting, and nice benchmark results. I have a question though:

Quoted 7 lines

mm/memory.c +4 -0

+	/* Get the pages we're interested in */
+	pages_pinned = get_user_pages(task, task->mm, pa,
+				      nr_pages_to_copy,
+				      copy_to, 0, process_pages,

NULL); +

(quoted diff) +2 -0

+	if (pages_pinned != nr_pages_to_copy)
+		goto end;

...

mm/memory.c void print_vma_addr(char *prefix, unsigned long ip) · line 3578

Quoted 6 lines

mm/memory.c +6 -0

+end:
+	for (i = 0; i < pages_pinned; i++) {
+		if (copy_to)
+			set_page_dirty_lock(process_pages[i]);
+		put_page(process_pages[i]);
+	}

It looks to me like if get_user_pages() fails to pin *all* the pages, we treat the target pages as dirty even though we never actually touched them?

Maybe it should be 'if (copy_to && *bytes_copied)'?