Skip to contentSkip to conversation

netwrok allocator. Sending and receiving zero-copy support. Benchmark.

PATCH New Evgeniy Polyakov 1 participant 1 message Sep 2, 2006

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 netwrok allocator. Sending and receiving zero-copy support. Benchmark.

Hello.

This patch introduce network-based receiving and sending zero-copy support. Receiving side is used for zero-copy sniffer (it is possible to dump all network allocations including netlink, unix socket and so on), sending side zero-copy is used for userspace netchannels network stack implementation [4].

Receiving zero-copy support.
Information about each network allocation is stored in special buffer accessible from userspace through read/write for special char device. This information is added when skb is freed, so data placed there is valid. Such approach does not allow to store transient state of the data, for example encrypted IPsec packets if decryption happens in-place, to store such states it is required to copy information into allocated buffer (which then can be immediately freed, or decryption can happen from source do destination buffers).
Userspace can mmap whole allocation pool (using special commands via ioctl() to get appropriate information) and thus access data without any copy.
Zero-copy sniffer has following overheads:
several atomic operations (in the worst case one atomic_set(), one atomic_inc() and one or two atomic_dec_and_test()),
one lock (bad global lock per sniffer device), which is held when information about new packet is being put into sniffer's queue when skb is freed,
delayed freeing which can lead to increased memory usage, or (like implemented) if introduced maximum amount of "locked" data by sniffer, some packets can be dropped by sniffer.

Sending zero-copy support.
Sending is performaed in two steps. First one is allocation, new object can be accessed without any copy like described above (and actually can be used with things like kevents to implement proposed by Ulrich Drepper "DMA" allocation (allocation of data, which can be used both by kernel and userspace without additional copies)). Second step is data commit, where allocated packet is attached to special skb and pushed into the stack using dst_output().

Above extensions are possible due to design of network allocator. Benchmark test for NTA shows very noticeble performance improvement over existing SLAB one (epoll based web server showed 40% improvement in number of processed requests per second, details can be found on project homepage [1]).

For testing zero-copy sniffer I ran scp over 100Mb lan and dump header into the file. Graph for both tcpdump and zero-copy sniffer sequence numbers is attached. Test shows that there are no dropped packets by sniffer (although for scp of big file over 100Mb lan tcpdump does not drop packets too), in my current environment it is not very exciting test though.

When both sniffers dump the whole traffic into /dev/null, CPU usage for zero-copy sniffer is about 2 times less than with tcpdump.

Interested reader can find all userspace applications (zero-copy sniffer and sending program) on project homepage [1].

Limitations of current version (introduced not due to design problems, but intentionally to test various special usage cases):
- use NTA only for netdev_alloc_skb() and sk_stream_alloc_pskb().
- always compile zero-copy sniffer in, which increases memory usage and adds described above overhead.
- skb_copy() always allocate data from SLAB allocator, although it could check if original skb's data was allocated through NTA, but I think that skb_copy() is completely incompatible with high performance.
- it is possible to eliminate several atomic operations (I'm lazy).
- debug code (poisoning of the tail of the buffer and additional reference counter) is always compiled in.

I would like to ask that should I make some steps for preparing that code for inclusion, and if so it would be good to start to discuss it, or if that code is not very interesting, so I will use it only for my own netchannels [3] developemnt. At this stage I do not copy his message to linux-kernel@ and linux-mm@ mail lists, since practice shows that it is not always the right way to solve technical problems.

Thank you.

1. Network tree allocator.
http://tservice.net.ru/~s0mbre/old/?section=projects&item=nta

2. Sending and receiving zero-copy networking.
http://tservice.net.ru/~s0mbre/blog/devel/networking/zcs/index.html

3. Netchannels implementation.
http://tservice.net.ru/~s0mbre/old/?section=projects&item=netchannel

4. Userspace network stack.
http://tservice.net.ru/~s0mbre/old/?section=projects&item=unetstack

drivers/net/3c59x.c +4 -4
index 80e8ca0..4aba97b 100644
@@ -1680,7 +1680,7 @@ vortex_open(struct net_device *dev)
 			vp->rx_ring[i].next = cpu_to_le32(vp->rx_ring_dma + sizeof(struct boom_rx_desc) * (i+1));
 			vp->rx_ring[i].status = 0;	/* Clear complete bit. */
 			vp->rx_ring[i].length = cpu_to_le32(PKT_BUF_SZ | LAST_FRAG);
-			skb = dev_alloc_skb(PKT_BUF_SZ);
+			skb = netdev_alloc_skb(dev, PKT_BUF_SZ);
 			vp->rx_skbuff[i] = skb;
 			if (skb == NULL)
 				break;			/* Bad news!  */
@@ -2405,7 +2405,7 @@ static int vortex_rx(struct net_device *
 			int pkt_len = rx_status & 0x1fff;
 			struct sk_buff *skb;
 
-			skb = dev_alloc_skb(pkt_len + 5);
+			skb = netdev_alloc_skb(dev, pkt_len + 5);
 			if (vortex_debug > 4)
 				printk(KERN_DEBUG "Receiving packet size %d status %4.4x.\n",
 					   pkt_len, rx_status);
@@ -2486,7 +2486,7 @@ boomerang_rx(struct net_device *dev)
 
 			/* Check if the packet is long enough to just accept without
 			   copying to a properly sized skbuff. */
-			if (pkt_len < rx_copybreak && (skb = dev_alloc_skb(pkt_len + 2)) != 0) {
+			if (pkt_len < rx_copybreak && (skb = netdev_alloc_skb(dev, pkt_len + 2)) != 0) {
 				skb->dev = dev;
 				skb_reserve(skb, 2);	/* Align IP on 16 byte boundaries */
 				pci_dma_sync_single_for_cpu(VORTEX_PCI(vp), dma, PKT_BUF_SZ, PCI_DMA_FROMDEVICE);
@@ -2525,7 +2525,7 @@ boomerang_rx(struct net_device *dev)
 		struct sk_buff *skb;
 		entry = vp->dirty_rx % RX_RING_SIZE;
 		if (vp->rx_skbuff[entry] == NULL) {
-			skb = dev_alloc_skb(PKT_BUF_SZ);
+			skb = netdev_alloc_skb(dev, PKT_BUF_SZ);
 			if (skb == NULL) {
 				static unsigned long last_jif;
 				if (time_after(jiffies, last_jif + 10 * HZ)) {
include/linux/skbuff.h +23 -1
index 19c96d4..bb1018c 100644
@@ -282,7 +282,8 @@ struct sk_buff {
 				nfctinfo:3;
 	__u8			pkt_type:3,
 				fclone:2,
-				ipvs_property:1;
+				ipvs_property:1,
+				nta:1;
 	__be16			protocol;
 
 	void			(*destructor)(struct sk_buff *skb);
@@ -327,22 +328,43 @@ #include <linux/slab.h>
 
 #include <asm/system.h>
 
+extern void *avl_alloc(unsigned int size, gfp_t gfp_mask);
+extern void avl_free(void *ptr, unsigned int size);
+extern int avl_init(void);
+
 extern void kfree_skb(struct sk_buff *skb);
 extern void	       __kfree_skb(struct sk_buff *skb);
 extern struct sk_buff *__alloc_skb(unsigned int size,
 				   gfp_t priority, int fclone);
+extern struct sk_buff *__alloc_skb_emtpy(unsigned int size,
+				   gfp_t priority);
+extern struct sk_buff *__alloc_skb_nta(unsigned int size, gfp_t gfp_mask,
+			    int fclone);
+
 static inline struct sk_buff *alloc_skb(unsigned int size,
 					gfp_t priority)
 {
 	return __alloc_skb(size, priority, 0);
 }
 
+static inline struct sk_buff *alloc_skb_empty(unsigned int size,
+					gfp_t priority)
+{
+	return __alloc_skb_emtpy(size, priority);
+}
+
 static inline struct sk_buff *alloc_skb_fclone(unsigned int size,
 					       gfp_t priority)
 {
 	return __alloc_skb(size, priority, 1);
 }
 
+static inline struct sk_buff *alloc_skb_nta(unsigned int size,
+					gfp_t priority, int fclone)
+{
+	return __alloc_skb_nta(size, priority, fclone);
+}
+
 extern struct sk_buff *alloc_skb_from_cache(kmem_cache_t *cp,
 					    unsigned int size,
 					    gfp_t priority);
include/net/sock.h +1 -1
index 324b3ea..6af3198 100644
@@ -1178,7 +1178,7 @@ static inline struct sk_buff *sk_stream_
 	int hdr_len;
 
 	hdr_len = SKB_DATA_ALIGN(sk->sk_prot->max_header);
-	skb = alloc_skb_fclone(size + hdr_len, gfp);
+	skb = alloc_skb_nta(size + hdr_len, gfp, 1);
 	if (skb) {
 		skb->truesize += mem;
 		if (sk_stream_wmem_schedule(sk, skb->truesize)) {
net/core/Makefile +2 -0
index 2645ba4..d86d468 100644
@@ -10,6 +10,8 @@ obj-$(CONFIG_SYSCTL) += sysctl_net_core.
 obj-y		     += dev.o ethtool.o dev_mcast.o dst.o netevent.o \
 			neighbour.o rtnetlink.o utils.o link_watch.o filter.o
 
+obj-y += alloc/
+
 obj-$(CONFIG_XFRM) += flow.o
 obj-$(CONFIG_SYSFS) += net-sysfs.o
 obj-$(CONFIG_NET_DIVERT) += dv.o
net/core/alloc/Makefile new file +3 -0
new file mode 100644
index 0000000..779eba2
@@ -0,0 +1,3 @@
+obj-y		:= allocator.o
+
+allocator-y	:= avl.o zc.o
net/core/alloc/avl.c new file +767 -0

767 lines not on this page · raw message

net/core/alloc/avl.h new file +224 -0

224 lines not on this page · raw message

net/core/alloc/zc.c new file +483 -0

483 lines not on this page · raw message

net/core/skbuff.c +113 -5

166 lines not on this page · raw message

Signature · 1 line
	Evgeniy Polyakov