macvtap TX zero copy between guest and host kernel
Subsystems (3)
v2 cover macvtap TX zero copy between guest and host kernel
This patchset add supports for TX zero-copy between guest and host kernel through vhost. It significantly reduces CPU utilization on the local host on which the guest is located (It reduced 30-50% CPU usage for vhost thread for single stream test). The patchset is based on previous submission and comments from the community regarding when/how to handle guest kernel buffers to be released. This is the simplest approach I can think of after comparing with several other solutions.
This patchset includes:
1. Induce a new sock zero-copy flag, SOCK_ZEROCOPY;
2. Induce a new device flag, NETIF_F_ZEROCOPY for device can support zero-copy;
3. Add a new struct skb_ubuf_info in skb_share_info for userspace buffers release callback when device DMA has done for that skb;
4. Add vhost zero-copy callback in vhost when skb last refcnt is gone; add vhost_zerocopy_add_used_and_signal to notify guest to release TX
skb buffers.
5. Add macvtap zero-copy in lower device when sending packet is greater than 128 bytes.
The patchset has passed netperf/netserver test on Chelsio, and continuing test on other 10GbE NICs, like Intel ixgbe, Mellanox mlx4... I will provide guest to host, host to guest performance data next week.
However when running stress test, vhost & virtio_net seems out of sync, and virito_net interrupt was disabled somehow, and it stopped to send any packet. This problem has bothered me for a long long time, I will continue to look at this.
Please review this.
Thanks
Shirley
This patch has built and tested against most recent linus git tree. But I haven't done checkpatch yet. I would like to know whether this approach is acceptable or not first.
Thanks
Shirley
Quoted 34 lines · Shirley Ma
This patchset add supports for TX zero-copy between guest and host kernel through vhost. It significantly reduces CPU utilization on the local host on which the guest is located (It reduced 30-50% CPU usage for vhost thread for single stream test). The patchset is based on previous submission and comments from the community regarding when/how to handle guest kernel buffers to be released. This is the simplest approach I can think of after comparing with several other solutions.
This patchset includes:
1. Induce a new sock zero-copy flag,
SOCK_ZEROCOPY;2. Induce a new device flag,
NETIF_F_ZEROCOPYfor device can support zero-copy;3. Add a new struct
skb_ubuf_infoinskb_share_infofor userspace buffers release callback when device DMA has done for that skb;4. Add vhost zero-copy callback in vhost when skb last refcnt is gone; add
vhost_zerocopy_add_used_and_signalto notify guest to release TX
skb buffers.5. Add macvtap zero-copy in lower device when sending packet is greater than 128 bytes.
The patchset has passed netperf/netserver test on Chelsio, and continuing test on other 10GbE NICs, like Intel ixgbe, Mellanox mlx4... I will provide guest to host, host to guest performance data next week.
However when running stress test, vhost &
virtio_netseems out of sync, andvirito_netinterrupt was disabled somehow, and it stopped to send any packet. This problem has bothered me for a long long time, I will continue to look at this.
Quoted 4 lines
Please review this.
Thanks
Shirley
What's the status here? Since there are core net changes, we'll need to see the final version soon if it's to appear in 2.6.39.
Could the problem be related to the patch
virtio_net: Add schedule check to napi_enable call
?
Also, I expect there should be driver patches for some devices? Where are they?
Thanks,
Signature · 1 line
MST
Quoted 3 lines · Michael S. Tsirkin
What's the status here? Since there are core net changes, we'll need to
see the final version soon if it's to appear in 2.6.39.
I am updating the patch and retesting it for the new kernel. I am trying to understand why zero copy patch caused small message size guest exits doubled.
Quoted 3 lines
Could the problem be related to the patch
virtio_net: Add schedule check tonapi_enablecall
?
Not sure, I will retest w/i this patch.
Quoted 2 lines
Also, I expect there should be driver patches for some devices? Where are they?
Yes, a NET_F_ZEROCOPY... flag is introduced for the device we we want to support zero copy.
Thanks
Shirley