About KVM network zero-copy feature!

Hi,All

I searched from git-log, and found that until now we have vhost TX zero-copy experiment feature, how about RX zero-copy?

For XEN, net-back also only has TX zero-copy, Is there any reason that RX zero-copy still not implemented?

Thanks!

There is no guarantee that packet will ever be read by receiver. This means zero-copy could create memory back pressure stalls.

It would be handy if this could be an optional feature, perhaps not enabled by default due to the problem with stalls you mentioned. I would love to see RX zero-copy implemented natively in KVM, as it might alleviate the need for custom solutions like vPF_RING:

http://www.ntop.org/products/pf_ring/vpf_ring/

Every time a packet is copied, especially from kernel space to user space, there is an opportunity for it to be dropped on its way to the receiving application - which is unacceptable when monitoring high-speed networks for security or bandwidth accounting purposes.

I am attempting to find a highly-efficient way to deploy virtualized network monitoring sensors (Snort, for example). Ideally I want to exploit symmetric hardware-based RSS and SR-IOV functionality for load-balancing and packet distribution completely in ASIC. I've found other existing work in this area (also using custom drivers) indicating significant performance gains in the non-virtualized case:

http://www.ndsl.kaist.edu/~shinae/papers/TR-symRSS.pdf

Is there any interest in exploring native RX zero-copy within the mainline KVM networking code?

Thanks,
Robert Vineyard

(adding Xin Xiaohui to the conversation for comment)

According to the NetworkingTodo page on the KVM wiki, zero-copy RX for macvtap is in fact on the roadmap, assigned to Xin:

http://www.linux-kvm.org/page/NetworkingTodo

The Release Notes for RHEL 6.2 (originally published on 12/06/2011) also specifically mention macvtap/vhost zero-copy capabilities as being included as a Technology Preview:

http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/6.2_Release_Notes/virtualization.html

Since I've been unable to find much information on the current status of this development, can anyone confirm if this functionality is still in the works? If so, is there any planned ETA?

Thanks,
Robert Vineyard

AFAIK Xin left Intel and is not working on it.
Contributions are welcome.

I think this means TX.

MST

Hi, Michael

IIt will be usefull if we can implement rx zero-copy, could you give
me some help if you know some technical details or some
references on the internet?

I am wondering may be I can take a deep look on it first then decide if I can take it over or not.

Thanks a lot.

Last patchset version was this:
[PATCH v16 00/17] Provide a zero-copy method on KVM virtio-net

IIRC the main issue was the need to integrate the patchset with macvtap (as opposed to using a separate device).

That's too bad... do you know of anyone else (at Intel or otherwise) who might be familiar enough with the existing codebase to get me started?

From the code I've looked at, it appears that among other things they're using the splice(2) / vmsplice(2) system calls to effect "copying" without actually copying data. If I understand the semantics correctly, these calls are basically shuffling pointers around to avoid unnecessary memcpy(3) / mmap(2) calls. I've even seen a "zero-copy" version of sendfile(2) that essentially wraps it around a call to splice(2).

I may be able to hack something together based on the current zero-copy TX implementation, but as I'm still wrapping my head around several of the concepts I just described, it may be awhile before I can produce anything useful. I have quite a bit of experience developing for Linux in C, but this would be my first attempt at writing kernel/device-driver code.

It does. I meant to clarify that point in my original email... yes, only TX zero-copy is currently implemented, and it is still marked as "experimental". Outside of the custom solutions like PF_RING that I mentioned, I don't know that I've seen zero-copy for RX.

-- Robert Vineyard

I suppose I should have done my homework... *ahem* time for me to finish RTFM on git...

I'll come back after digging through more kernel code :-)

-- Robert

I suppose I should have done my homework... *ahem* time for me to finish RTFM on git...

I'll come back after digging through more kernel code :-)

-- Robert

It would be handy if this could be an optional feature, perhaps not enabled by default due to the problem with stalls you mentioned. I would love to see RX zero-copy implemented natively in KVM, as it might alleviate the need for custom solutions like vPF_RING:

http://www.ntop.org/products/pf_ring/vpf_ring/

Every time a packet is copied, especially from kernel space to user space, there is an opportunity for it to be dropped on its way to the receiving application - which is unacceptable when monitoring high-speed networks for security or bandwidth accounting purposes.

I am attempting to find a highly-efficient way to deploy virtualized network monitoring sensors (Snort, for example). Ideally I want to exploit symmetric hardware-based RSS and SR-IOV functionality for load-balancing and packet distribution completely in ASIC. I've found other existing work in this area (also using custom drivers) indicating significant performance gains in the non-virtualized case:

http://www.ndsl.kaist.edu/~shinae/papers/TR-symRSS.pdf

Is there any interest in exploring native RX zero-copy within the mainline KVM networking code?

Thanks,
Robert Vineyard