Skip to contentSkip to conversation

Inter-process send()/recv() using zero-copy ?

DISCUSSION Xavier Roche 3 participants 4 messages Sep 23, 2009

lore
Share & tools
lore
#0 lore raw

Hi folks,

I was wondering if there was a way to have zero-copy send()/recv(), when the socket is connected to the local machine (to another process on the same machine, for example) ?

Such feature would be only feasible with page-aligned blocks, from an a mmap'ed block to another one, I guess.

Typical case:

Process #1 (uid A)
buff = mmap(0, size, ..) /* anonymous or not */
...
send(s, buff, size, 0)
munmap(buff, size)

Process #2 (uid B)
buff = mmap(0, size, .. | MAP_ANONYMOUS, ..)
recv(s, buff, size, 0)

In an ideal fantasy world, the first process would use send() to transmit the complete page-aligned memory block to the other side, and the second process would use recv() to get the memory block on a similar anonymously mmap'ed block, and the only operation the kernel would do would be to share the memory block between the two processes with copy-on-write.

On the real world, the same operation requires a first read of the whole memory block (possibly partially on disk) and a complete write (possibly partially on disk, too) with two copies of the same memory region at the end.

Two solutions can be used to emulate such feature:

1. use a temporary mmap'ed file
- but requires a temporary file
- permissions for the file ? (not necessarily from the same UID)
- special case for local network block transmissions vs. machine-to-machine

2. use shared memory explicitely
- handling of permissions ? (ditto)
- special case for local network block transmissions vs. machine-to-machine

splice() and friends do not appear to give any help for this case, and I was wondering if there was a chance to do that ?

#1 lore raw
Quoted 28 lines · On Wed, 23 Sep 2009 08:01:27 +0200 Xavier Roche <[email protected]> wrote:

Hi folks,

I was wondering if there was a way to have zero-copy send()/recv(), when the socket is connected to the local machine (to another process on the same machine, for example) ?

Such feature would be only feasible with page-aligned blocks, from an a mmap'ed block to another one, I guess.

Typical case:

Process #1 (uid A)
buff = mmap(0, size, ..) /* anonymous or not */
...
send(s, buff, size, 0)
munmap(buff, size)

Process #2 (uid B)
buff = mmap(0, size, .. | MAP_ANONYMOUS, ..)
recv(s, buff, size, 0)

In an ideal fantasy world, the first process would use send() to transmit the complete page-aligned memory block to the other side, and the second process would use recv() to get the memory block on a similar anonymously mmap'ed block, and the only operation the kernel would do would be to share the memory block between the two processes with copy-on-write.

Quoted 6 lines

On the real world, the same operation requires a first read of the whole memory block (possibly partially on disk) and a complete write (possibly partially on disk, too) with two copies of the same memory region at the end.

Two solutions can be used to

the problem you have is that
1) memory copies are cheap
(say, 3000 cycles/page or less)
2) page table operations (mmap etc) are very expensive.

these two combined tend to not make it a win to substitute simple copies with complex pagetable tricks.

Signature · 3 lines
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#2 lore raw
Quoted 3 lines

the problem you have is that
1) memory copies are cheap
(say, 3000 cycles/page or less)

What about L1 cache pollution? Doesn't it change situation?

Quoted 4 lines

2) page table operations (mmap etc) are very expensive.

these two combined tend to not make it a win to substitute simple copies with complex pagetable tricks.

#3 lore raw
In reply to Arjan van de Ven
Quoted 2 lines · Arjan van de Ven

1) memory copies are cheap
(say, 3000 cycles/page or less)

Yes, but this case would be more than useful for large memory blocks (typically memory_size/N, with N typically 2..10) -- something you generally have when you deal with mmap'ed blocks.