Skip to contentSkip to conversation

Intel I/O Acceleration Technology (I/OAT)

PATCH In review Chris Leech 17 participants 49 messages Mar 3, 2006 – Mar 7, 2006

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

cover Intel I/O Acceleration Technology (I/OAT)

This patch series is the first full release of the Intel(R) I/O Acceleration Technology (I/OAT) for Linux. It includes an in kernel API for offloading memory copies to hardware, a driver for the I/OAT DMA memcpy engine, and changes to the TCP stack to offload copies of received networking data to application space.

These changes apply to DaveM's net-2.6.17 tree as of commit 2bd84a93d8bb7192ad8c23ef41008502be1cb603 ([IRDA]: TOIM3232 dongle support)

They are available to pull from

	git://198.78.49.142/~cleech/linux-2.6 ioat-2.6.17

There are 8 patches in the series:
1) The memcpy offload APIs and class code
2) The Intel I/OAT DMA driver (ioatdma)
3) Core networking code to setup networking as a DMA memcpy client
4) Utility functions for sk_buff to iovec offloaded copy
5) Structure changes needed for TCP receive offload
6) Rename cleanup_rbuf to tcp_cleanup_rbuf
7) Add a sysctl to tune the minimum offloaded I/O size for TCP
8) The main TCP receive offload changes

--
Chris Leech <[email protected]>
I/O Acceleration Technology Software Development
LAN Access Division / Digital Enterprise Group

#1 lore raw

PATCH 1/8 DMA memcpy subsystem

Provides an API for offloading memory copies to DMA devices

---

 drivers/Kconfig           |    2 
 drivers/Makefile          |    1 
 drivers/dma/Kconfig       |   13 ++
 drivers/dma/Makefile      |    1 
 drivers/dma/dmaengine.c   |  361 +++++++++++++++++++++++++++++++++++++++++++++
 include/linux/dmaengine.h |  322 ++++++++++++++++++++++++++++++++++++++++
 6 files changed, 700 insertions(+), 0 deletions(-)
drivers/Kconfig +2 -0
index bddf431..ce7ffa7 100644
@@ -70,4 +70,6 @@ source "drivers/sn/Kconfig"
 
 source "drivers/edac/Kconfig"
 
+source "drivers/dma/Kconfig"
+
 endmenu
drivers/Makefile +1 -0
index 5c69b86..516ba5e 100644
@@ -73,3 +73,4 @@ obj-$(CONFIG_SGI_SN)		+= sn/
 obj-y				+= firmware/
 obj-$(CONFIG_CRYPTO)		+= crypto/
 obj-$(CONFIG_SUPERH)		+= sh/
+obj-$(CONFIG_DMA_ENGINE)	+= dma/
drivers/dma/Kconfig new file +13 -0
new file mode 100644
index 0000000..f9ac4bc
@@ -0,0 +1,13 @@
+#
+# DMA engine configuration
+#
+
+menu "DMA Engine support"
+
+config DMA_ENGINE
+	bool "Support for DMA engines"
+	---help---
+	  DMA engines offload copy operations from the CPU to dedicated
+	  hardware, allowing the copies to happen asynchronously.
+
+endmenu
drivers/dma/Makefile new file +1 -0
new file mode 100644
index 0000000..10b7391
@@ -0,0 +1 @@
+obj-y += dmaengine.o
drivers/dma/dmaengine.c new file +361 -0

361 lines not on this page · raw message

include/linux/dmaengine.h new file +322 -0

322 lines not on this page · raw message

#6 lore raw

PATCH 3/8 Setup the networking subsystem as a DMA client

In reply to Chris Leech

Attempts to allocate per-CPU DMA channels

---

 drivers/dma/Kconfig       |   12 +++++
 include/linux/netdevice.h |    6 +++
 include/net/netdma.h      |   36 ++++++++++++++++
 net/core/dev.c            |  102 +++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 156 insertions(+), 0 deletions(-)
drivers/dma/Kconfig +12 -0
index 0f15e76..30d021d 100644
@@ -10,6 +10,18 @@ config DMA_ENGINE
 	  DMA engines offload copy operations from the CPU to dedicated
 	  hardware, allowing the copies to happen asynchronously.
 
+comment "DMA Clients"
+
+config NET_DMA
+	bool "Network: TCP receive copy offload"
+	depends on DMA_ENGINE && NET
+	default y
+	---help---
+	  This enables the use of DMA engines in the network stack to
+	  offload receive copy-to-user operations, freeing CPU cycles.
+	  Since this is the main user of the DMA engine, it should be enabled;
+	  say Y here.
+
 comment "DMA Devices"
 
 config INTEL_IOATDMA
include/linux/netdevice.h +6 -0
index b825be2..ecbde56 100644
@@ -37,6 +37,9 @@
 #include <linux/config.h>
 #include <linux/device.h>
 #include <linux/percpu.h>
+#ifdef CONFIG_NET_DMA
+#include <linux/dmaengine.h>
+#endif
 
 struct divert_blk;
 struct vlan_group;
@@ -592,6 +595,9 @@ struct softnet_data
 	struct sk_buff		*completion_queue;
 
 	struct net_device	backlog_dev;	/* Sorry. 8) */
+#ifdef CONFIG_NET_DMA
+	struct dma_chan		*net_dma;
+#endif
 };
 
 DECLARE_PER_CPU(struct softnet_data,softnet_data);
include/net/netdma.h new file +36 -0
new file mode 100644
index 0000000..3cd9e6d
@@ -0,0 +1,36 @@
+/*****************************************************************************
+Copyright(c) 2004 - 2006 Intel Corporation. All rights reserved.
+
+This program is free software; you can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by the Free
+Software Foundation; either version 2 of the License, or (at your option)
+any later version.
+
+This program is distributed in the hope that it will be useful, but WITHOUT
+ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+more details.
+
+You should have received a copy of the GNU General Public License along with
+this program; if not, write to the Free Software Foundation, Inc., 59
+Temple Place - Suite 330, Boston, MA  02111-1307, USA.
+
+The full GNU General Public License is included in this distribution in the
+file called LICENSE.
+*****************************************************************************/
+#ifndef NETDMA_H
+#define NETDMA_H
+#include <linux/dmaengine.h>
+
+static inline struct dma_chan *get_softnet_dma(void)
+{
+	struct dma_chan *chan;
+	rcu_read_lock();
+	chan = rcu_dereference(__get_cpu_var(softnet_data.net_dma));
+	if (chan)
+		dma_chan_get(chan);
+	rcu_read_unlock();
+	return chan;
+}
+
+#endif /* NETDMA_H */
net/core/dev.c +102 -0

138 lines not on this page · raw message

#11 lore raw

PATCH 6/8 Rename cleanup_rbuf to tcp_cleanup_rbuf and make non-static

In reply to Chris Leech

Needed to be able to call tcp_cleanup_rbuf in tcp_input.c for I/OAT

---

 include/net/tcp.h |    2 ++
 net/ipv4/tcp.c    |   10 +++++-----
 2 files changed, 7 insertions(+), 5 deletions(-)
include/net/tcp.h +2 -0
index fd7c3e4..2fc7f05 100644
@@ -295,6 +295,8 @@ extern int			tcp_rcv_established(struct 
 
 extern void			tcp_rcv_space_adjust(struct sock *sk);
 
+extern void			tcp_cleanup_rbuf(struct sock *sk, int copied);
+
 extern int			tcp_twsk_unique(struct sock *sk,
 						struct sock *sktw, void *twp);
 
net/ipv4/tcp.c +5 -5
index 00aa80e..13abfa2 100644
@@ -936,7 +936,7 @@ static int tcp_recv_urg(struct sock *sk,
  * calculation of whether or not we must ACK for the sake of
  * a window update.
  */
-static void cleanup_rbuf(struct sock *sk, int copied)
+void tcp_cleanup_rbuf(struct sock *sk, int copied)
 {
 	struct tcp_sock *tp = tcp_sk(sk);
 	int time_to_ack = 0;
@@ -1085,7 +1085,7 @@ int tcp_read_sock(struct sock *sk, read_
 
 	/* Clean up data we have read: This will do ACK frames. */
 	if (copied)
-		cleanup_rbuf(sk, copied);
+		tcp_cleanup_rbuf(sk, copied);
 	return copied;
 }
 
@@ -1219,7 +1219,7 @@ int tcp_recvmsg(struct kiocb *iocb, stru
 			}
 		}
 
-		cleanup_rbuf(sk, copied);
+		tcp_cleanup_rbuf(sk, copied);
 
 		if (!sysctl_tcp_low_latency && tp->ucopy.task == user_recv) {
 			/* Install new reader */
@@ -1390,7 +1390,7 @@ skip_copy:
 	 */
 
 	/* Clean up data we have read: This will do ACK frames. */
-	cleanup_rbuf(sk, copied);
+	tcp_cleanup_rbuf(sk, copied);
 
 	TCP_CHECK_TIMER(sk);
 	release_sock(sk);
@@ -1856,7 +1856,7 @@ int tcp_setsockopt(struct sock *sk, int 
 			    (TCPF_ESTABLISHED | TCPF_CLOSE_WAIT) &&
 			    inet_csk_ack_scheduled(sk)) {
 				icsk->icsk_ack.pending |= ICSK_ACK_PUSHED;
-				cleanup_rbuf(sk, 1);
+				tcp_cleanup_rbuf(sk, 1);
 				if (!(val & 1))
 					icsk->icsk_ack.pingpong = 1;
 			}
#12 lore raw

PATCH 7/8 Add a sysctl for tuning the I/OAT offloaded I/O threshold

In reply to Chris Leech

Any socket recv of less than this ammount will not be offloaded

---

 include/linux/sysctl.h     |    1 +
 include/net/tcp.h          |    1 +
 net/core/user_dma.c        |    4 ++++
 net/ipv4/sysctl_net_ipv4.c |   10 ++++++++++
 4 files changed, 16 insertions(+), 0 deletions(-)
include/linux/sysctl.h +1 -0
index dfcf449..f532f1e 100644
@@ -402,6 +402,7 @@ enum
 	NET_IPV4_IPFRAG_MAX_DIST=112,
  	NET_TCP_MTU_PROBING=113,
 	NET_TCP_BASE_MSS=114,
+	NET_TCP_DMA_COPYBREAK=115,
 };
 
 enum {
include/net/tcp.h +1 -0
index 2fc7f05..0740f32 100644
@@ -221,6 +221,7 @@ extern int sysctl_tcp_adv_win_scale;
 extern int sysctl_tcp_tw_reuse;
 extern int sysctl_tcp_frto;
 extern int sysctl_tcp_low_latency;
+extern int sysctl_tcp_dma_copybreak;
 extern int sysctl_tcp_nometrics_save;
 extern int sysctl_tcp_moderate_rcvbuf;
 extern int sysctl_tcp_tso_win_divisor;
net/core/user_dma.c +4 -0
index 1e1aae5..dd259f0 100644
@@ -33,6 +33,10 @@ file called LICENSE.
 
 #ifdef CONFIG_NET_DMA
 
+#define NET_DMA_DEFAULT_COPYBREAK 1024
+
+int sysctl_tcp_dma_copybreak = NET_DMA_DEFAULT_COPYBREAK;
+
 /**
  *	dma_skb_copy_datagram_iovec - Copy a datagram to an iovec.
  *	@skb - buffer to copy
net/ipv4/sysctl_net_ipv4.c +10 -0
index ebf2e0b..f7bd9c2 100644
@@ -680,6 +680,16 @@ ctl_table ipv4_table[] = {
 		.mode		= 0644,
 		.proc_handler	= &proc_dointvec,
 	},
+#ifdef CONFIG_NET_DMA
+	{
+		.ctl_name	= NET_TCP_DMA_COPYBREAK,
+		.procname	= "tcp_dma_copybreak",
+		.data		= &sysctl_tcp_dma_copybreak,
+		.maxlen		= sizeof(int),
+		.mode		= 0644,
+		.proc_handler	= &proc_dointvec
+	},
+#endif
 
 	{ .ctl_name = 0 }
 };
#9 lore raw

PATCH 5/8 Structure changes for TCP recv offload to I/OAT

In reply to Chris Leech

Adds an async_wait_queue and some additional fields to tcp_sock, and a dma_cookie_t to sk_buff.

---

 include/linux/skbuff.h |    6 ++++++
 include/linux/tcp.h    |   10 ++++++++++
 include/net/sock.h     |    2 ++
 include/net/tcp.h      |    9 +++++++++
 net/core/sock.c        |    6 ++++++
 5 files changed, 33 insertions(+), 0 deletions(-)
include/linux/skbuff.h +6 -0
index 75c9631..572b7ae 100644
@@ -29,6 +29,9 @@
 #include <linux/net.h>
 #include <linux/textsearch.h>
 #include <net/checksum.h>
+#ifdef CONFIG_NET_DMA
+#include <linux/dmaengine.h>
+#endif
 
 #define HAVE_ALLOC_SKB		/* For the drivers to know */
 #define HAVE_ALIGNABLE_SKB	/* Ditto 8)		   */
@@ -285,6 +288,9 @@ struct sk_buff {
 	__u16			tc_verd;	/* traffic control verdict */
 #endif
 #endif
+#ifdef CONFIG_NET_DMA
+	dma_cookie_t		dma_cookie;
+#endif
 
 
 	/* These elements must be at the end, see alloc_skb() for details.  */
include/linux/tcp.h +10 -0
index 542d395..6d7dc19 100644
@@ -18,6 +18,9 @@
 #define _LINUX_TCP_H
 
 #include <linux/types.h>
+#ifdef CONFIG_NET_DMA
+#include <linux/dmaengine.h>
+#endif
 #include <asm/byteorder.h>
 
 struct tcphdr {
@@ -233,6 +236,13 @@ struct tcp_sock {
 		struct iovec		*iov;
 		int			memory;
 		int			len;
+#ifdef CONFIG_NET_DMA
+		/* members for async copy */
+		struct dma_chan		*dma_chan;
+		int			wakeup;
+		struct dma_locked_list	*locked_list;
+		dma_cookie_t		dma_cookie;
+#endif
 	} ucopy;
 
 	__u32	snd_wl1;	/* Sequence for window update		*/
include/net/sock.h +2 -0
index 3075803..5d1b895 100644
@@ -132,6 +132,7 @@ struct sock_common {
   *	@sk_receive_queue: incoming packets
   *	@sk_wmem_alloc: transmit queue bytes committed
   *	@sk_write_queue: Packet sending queue
+  *	@sk_async_wait_queue: DMA copied packets
   *	@sk_omem_alloc: "o" is "option" or "other"
   *	@sk_wmem_queued: persistent queue size
   *	@sk_forward_alloc: space allocated forward
@@ -205,6 +206,7 @@ struct sock {
 	atomic_t		sk_omem_alloc;
 	struct sk_buff_head	sk_receive_queue;
 	struct sk_buff_head	sk_write_queue;
+	struct sk_buff_head	sk_async_wait_queue;
 	int			sk_wmem_queued;
 	int			sk_forward_alloc;
 	gfp_t			sk_allocation;
include/net/tcp.h +9 -0
index 16879fa..fd7c3e4 100644
@@ -28,6 +28,9 @@
 #include <linux/cache.h>
 #include <linux/percpu.h>
 #include <linux/skbuff.h>
+#ifdef CONFIG_NET_DMA
+#include <linux/dmaengine.h>
+#endif
 
 #include <net/inet_connection_sock.h>
 #include <net/inet_timewait_sock.h>
@@ -813,6 +816,12 @@ static inline void tcp_prequeue_init(str
 	tp->ucopy.len = 0;
 	tp->ucopy.memory = 0;
 	skb_queue_head_init(&tp->ucopy.prequeue);
+#ifdef CONFIG_NET_DMA
+	tp->ucopy.dma_chan = NULL;
+	tp->ucopy.wakeup = 0;
+	tp->ucopy.locked_list = NULL;
+	tp->ucopy.dma_cookie = 0;
+#endif
 }
 
 /* Packet is added to VJ-style prequeue for processing in process
net/core/sock.c +6 -0
index 6e00811..90275ec 100644
@@ -724,6 +724,9 @@ struct sock *sk_clone(const struct sock 
 		atomic_set(&newsk->sk_omem_alloc, 0);
 		skb_queue_head_init(&newsk->sk_receive_queue);
 		skb_queue_head_init(&newsk->sk_write_queue);
+#ifdef CONFIG_NET_DMA
+		skb_queue_head_init(&newsk->sk_async_wait_queue);
+#endif
 
 		rwlock_init(&newsk->sk_dst_lock);
 		rwlock_init(&newsk->sk_callback_lock);
@@ -1275,6 +1278,9 @@ void sock_init_data(struct socket *sock,
 	skb_queue_head_init(&sk->sk_receive_queue);
 	skb_queue_head_init(&sk->sk_write_queue);
 	skb_queue_head_init(&sk->sk_error_queue);
+#ifdef CONFIG_NET_DMA
+	skb_queue_head_init(&sk->sk_async_wait_queue);
+#endif
 
 	sk->sk_send_head	=	NULL;
 
#7 lore raw

PATCH 4/8 Utility functions for offloading sk_buff to iovec copies

In reply to Chris Leech

Provides for pinning user space pages in memory, copying to iovecs, and copying from sk_buffs including fragmented and chained sk_buffs.

---

 drivers/dma/Makefile      |    1 
 drivers/dma/iovlock.c     |  320 +++++++++++++++++++++++++++++++++++++++++++++
 include/linux/dmaengine.h |   24 +++
 include/net/netdma.h      |    5 +
 net/core/Makefile         |    3 
 net/core/user_dma.c       |  133 +++++++++++++++++++
 6 files changed, 485 insertions(+), 1 deletions(-)
drivers/dma/Makefile +1 -0
index c8a5f56..ea2f110 100644
@@ -1,2 +1,3 @@
 obj-y += dmaengine.o
+obj-$(CONFIG_NET_DMA) += iovlock.o
 obj-$(CONFIG_INTEL_IOATDMA) += ioatdma.o
drivers/dma/iovlock.c new file +320 -0

320 lines not on this page · raw message

include/linux/dmaengine.h +24 -0
index f8a77ab..e198712 100644
@@ -319,4 +319,28 @@ static inline enum dma_status dma_async_
 int dma_async_device_register(struct dma_device *device);
 void dma_async_device_unregister(struct dma_device *device);
 
+/* --- Helper iov-locking functions --- */
+
+struct dma_page_list
+{
+	char *base_address;
+	int nr_pages;
+	struct page **pages;
+};
+
+struct dma_locked_list
+{
+	int nr_iovecs;
+	struct dma_page_list page_list[0];
+};
+
+int dma_lock_iovec_pages(struct iovec *iov, size_t len,
+	struct dma_locked_list	**locked_list);
+void dma_unlock_iovec_pages(struct dma_locked_list* locked_list);
+dma_cookie_t dma_memcpy_toiovec(struct dma_chan *chan, struct iovec *iov,
+	struct dma_locked_list *locked_list, unsigned char *kdata, size_t len);
+dma_cookie_t dma_memcpy_pg_toiovec(struct dma_chan *chan, struct iovec *iov,
+	struct dma_locked_list *locked_list, struct page *page,
+	unsigned int offset, size_t len);
+
 #endif /* DMAENGINE_H */
include/net/netdma.h +5 -0
index 3cd9e6d..415d74c 100644
@@ -21,6 +21,7 @@ file called LICENSE.
 #ifndef NETDMA_H
 #define NETDMA_H
 #include <linux/dmaengine.h>
+#include <linux/skbuff.h>
 
 static inline struct dma_chan *get_softnet_dma(void)
 {
@@ -33,4 +34,8 @@ static inline struct dma_chan *get_softn
 	return chan;
 }
 
+int dma_skb_copy_datagram_iovec(struct dma_chan* chan,
+		const struct sk_buff *skb, int offset, struct iovec *to,
+		size_t len, struct dma_locked_list *locked_list);
+
 #endif /* NETDMA_H */
net/core/Makefile +2 -1
index 630da0f..d02132b 100644
@@ -8,7 +8,8 @@ obj-y := sock.o request_sock.o skbuff.o 
 obj-$(CONFIG_SYSCTL) += sysctl_net_core.o
 
 obj-y		     += dev.o ethtool.o dev_mcast.o dst.o \
-			neighbour.o rtnetlink.o utils.o link_watch.o filter.o
+			neighbour.o rtnetlink.o utils.o link_watch.o filter.o \
+			user_dma.o
 
 obj-$(CONFIG_XFRM) += flow.o
 obj-$(CONFIG_SYSFS) += net-sysfs.o
net/core/user_dma.c new file +133 -0

133 lines not on this page · raw message

#15 lore raw

PATCH 8/8 TCP recv offload to I/OAT

In reply to Chris Leech

Locks down user pages and sets up for DMA in tcp_recvmsg, then calls dma_async_try_early_copy in tcp_v4_do_rcv

---

 include/net/netdma.h |    1 
 net/ipv4/tcp.c       |  111 +++++++++++++++++++++++++++++++++++++++++++++-----
 net/ipv4/tcp_input.c |   78 ++++++++++++++++++++++++++++++++---
 net/ipv4/tcp_ipv4.c  |   20 +++++++++
 net/ipv6/tcp_ipv6.c  |   12 +++++
 5 files changed, 201 insertions(+), 21 deletions(-)
include/net/netdma.h +1 -0
index 415d74c..2d829e1 100644
@@ -37,5 +37,6 @@ static inline struct dma_chan *get_softn
 int dma_skb_copy_datagram_iovec(struct dma_chan* chan,
 		const struct sk_buff *skb, int offset, struct iovec *to,
 		size_t len, struct dma_locked_list *locked_list);
+int dma_async_try_early_copy(struct sock *sk, struct sk_buff *skb, int hlen);
 
 #endif /* NETDMA_H */
net/ipv4/tcp.c +100 -11

164 lines not on this page · raw message

net/ipv4/tcp_input.c +71 -7

111 lines not on this page · raw message

net/ipv4/tcp_ipv4.c +19 -1
index 4eb903d..fecc022 100644
@@ -71,6 +71,9 @@
 #include <net/inet_common.h>
 #include <net/timewait_sock.h>
 #include <net/xfrm.h>
+#ifdef CONFIG_NET_DMA
+#include <net/netdma.h>
+#endif
 
 #include <linux/inet.h>
 #include <linux/ipv6.h>
@@ -1091,8 +1094,18 @@ process:
 	bh_lock_sock(sk);
 	ret = 0;
 	if (!sock_owned_by_user(sk)) {
-		if (!tcp_prequeue(sk, skb))
+#ifdef CONFIG_NET_DMA
+		struct tcp_sock *tp = tcp_sk(sk);
+		if (!tp->ucopy.dma_chan && tp->ucopy.locked_list)
+			tp->ucopy.dma_chan = get_softnet_dma();
+		if (tp->ucopy.dma_chan)
+			ret = tcp_v4_do_rcv(sk, skb);
+		else
+#endif
+		{
+			if (!tcp_prequeue(sk, skb))
 			ret = tcp_v4_do_rcv(sk, skb);
+		}
 	} else
 		sk_add_backlog(sk, skb);
 	bh_unlock_sock(sk);
@@ -1292,6 +1305,11 @@ int tcp_v4_destroy_sock(struct sock *sk)
 	/* Cleans up our, hopefully empty, out_of_order_queue. */
   	__skb_queue_purge(&tp->out_of_order_queue);
 
+#ifdef CONFIG_NET_DMA
+	/* Cleans up our sk_async_wait_queue */
+  	__skb_queue_purge(&sk->sk_async_wait_queue);
+#endif
+
 	/* Clean prequeue, it must be empty really */
 	__skb_queue_purge(&tp->ucopy.prequeue);
 
net/ipv6/tcp_ipv6.c +10 -2
index af6a0c6..acf798c 100644
@@ -1218,8 +1218,16 @@ process:
 	bh_lock_sock(sk);
 	ret = 0;
 	if (!sock_owned_by_user(sk)) {
-		if (!tcp_prequeue(sk, skb))
-			ret = tcp_v6_do_rcv(sk, skb);
+#ifdef CONFIG_NET_DMA
+                struct tcp_sock *tp = tcp_sk(sk);
+                if (tp->ucopy.dma_chan)
+                        ret = tcp_v6_do_rcv(sk, skb);
+                else
+#endif
+		{
+			if (!tcp_prequeue(sk, skb))
+				ret = tcp_v6_do_rcv(sk, skb);
+		}
 	} else
 		sk_add_backlog(sk, skb);
 	bh_unlock_sock(sk);
#23 lore raw
In reply to Chris Leech
Quoted 9 lines · Chris Leech

This patch series is the first full release of the Intel(R) I/O Acceleration Technology (I/OAT) for Linux. It includes an in kernel API for offloading memory copies to hardware, a driver for the I/OAT DMA memcpy engine, and changes to the TCP stack to offload copies of received networking data to application space.

These changes apply to DaveM's net-2.6.17 tree as of commit 2bd84a93d8bb7192ad8c23ef41008502be1cb603 ([IRDA]: TOIM3232 dongle support)

Quoted 6 lines

They are available to pull from

	git://198.78.49.142/~cleech/linux-2.6 ioat-2.6.17

There are 8 patches in the series:
1) The memcpy offload APIs and class code
2) The Intel I/OAT DMA driver (ioatdma)

Patch #2 didn't make it. Too big for the list?

Jeff

#24 lore raw
Quoted 1 line

Patch #2 didn't make it. Too big for the list?

Could be, it's the largest of the series. I've attached the gziped patch. I can try and split this up for the future.

- Chris

#25 lore raw
Quoted 5 lines · Chris Leech
Quoted 1 line

Patch #2 didn't make it. Too big for the list?

Could be, it's the largest of the series. I've attached the gziped patch. I can try and split this up for the future.

Well, for huge hunks of new code, it sometimes gets silly to split it up.

Once its not in a "reply to email" reviewable form, gzip or URL-to-patch work just fine.

Jeff

#31 lore raw
In reply to Chris Leech
Quoted 12 lines · Chris Leech

This patch series is the first full release of the Intel(R) I/O Acceleration Technology (I/OAT) for Linux. It includes an in kernel API for offloading memory copies to hardware, a driver for the I/OAT DMA memcpy engine, and changes to the TCP stack to offload copies of received networking data to application space.

These changes apply to DaveM's net-2.6.17 tree as of commit 2bd84a93d8bb7192ad8c23ef41008502be1cb603 ([IRDA]: TOIM3232 dongle support)

They are available to pull from

	git://198.78.49.142/~cleech/linux-2.6 ioat-2.6.17
Quoted 9 lines

There are 8 patches in the series:
1) The memcpy offload APIs and class code
2) The Intel I/OAT DMA driver (ioatdma)
3) Core networking code to setup networking as a DMA memcpy client
4) Utility functions for sk_buff to iovec offloaded copy
5) Structure changes needed for TCP receive offload
6) Rename cleanup_rbuf to tcp_cleanup_rbuf
7) Add a sysctl to tune the minimum offloaded I/O size for TCP
8) The main TCP receive offload changes

How does this relate to Dan William's ADMA work?

http://marc.theaimsgroup.com/?t=113892936300001&r=1&w=2

- kumar

#32 lore raw
Quoted 2 lines · Kumar Gala

How does this relate to Dan William's ADMA work?

I only became aware of Dan's ADMA work when he posted it last month, and so far have not made any attempts to merge the I/OAT code with it.
Moving forward, combining these interfaces certainly seems like the
right way to go. I particularly like ADMA's handling of operations other than just a copy (memset, compare, XOR, CRC).

Chris

maintainer
#2 lore raw

Re: [PATCH 1/8] [I/OAT] DMA memcpy subsystem

In reply to Chris Leech

From: Chris Leech <[email protected]>
Date: Fri, 03 Mar 2006 13:42:20 -0800

Quoted 1 line

+static spinlock_t dma_list_lock;

Please use DEFINE_SPINLOCK().

Quoted 1 line

+static void dma_chan_free_rcu(struct rcu_head *rcu) {

Newline before the brace please.

Quoted 1 line

+static void dma_async_device_cleanup(struct kref *kref) {

Newline before the brace please.

Review

Quoted 2 lines
(quoted diff) +2 -0
+struct dma_chan_percpu
+{

Left brace on the same line as "struct dma_chan_percpu" please.

Review

Quoted 2 lines
(quoted diff) +2 -0
+struct dma_chan
+{

Similarly.

Otherwise this patch looks mostly ok.

#13 lore raw

Re: [PATCH 7/8] [I/OAT] Add a sysctl for tuning the I/OAT offloaded I/O threshold

In reply to Chris Leech
Quoted 1 line · Chris Leech

Any socket recv of less than this ammount will not be offloaded

There is no documentation update.

#26 lore raw
In reply to Chris Leech
Quoted 4 lines · Chris Leech
Quoted 1 line

Patch #2 didn't make it. Too big for the list?

Could be, it's the largest of the series. I've attached the gziped patch. I can try and split this up for the future.

How can owner of cb_chan->common.device_node be removed?
It looks like that channels are only allocated (without proper error path) and queued into device->common.channels list in
enumerate_dma_channels() in PCI probe callback and no removing at all, only lockless access. PCI remove callback only calls dma_async_device_unregister() where only channel's clients are removed.

Quoted 1 line

- Chris

Signature · 1 line
	Evgeniy Polyakov
#16 lore raw

Re: [PATCH 8/8] [I/OAT] TCP recv offload to I/OAT

In reply to Chris Leech

Hi!

net/ipv4/tcp.c line 268

Quoted 10 lines
net/ipv4/tcp.c +3 -0
@@ -262,6 +262,9 @@
 #include <net/tcp.h>
 #include <net/xfrm.h>
 #include <net/ip.h>
+#ifdef CONFIG_NET_DMA
+#include <net/netdma.h>
+#endif
 

Remove the ifdefs, move them inside .h if needed.

net/ipv4/tcp_input.c line 76

Quoted 11 lines
net/ipv4/tcp_input.c +3 -0
index 7625eaf..9b6290d 100644
@@ -71,6 +71,9 @@
 #include <net/inet_common.h>
 #include <linux/ipsec.h>
 #include <asm/unaligned.h>
+#ifdef CONFIG_NET_DMA
+#include <net/netdma.h>
+#endif

Here, too.

net/ipv4/tcp_input.c int tcp_rcv_established(struct sock *sk, · line 3989

Quoted 8 lines
net/ipv4/tcp_input.c +5 -0
+#ifdef CONFIG_NET_DMA
+			if (copied_early)
+				__skb_queue_tail(&sk->sk_async_wait_queue, skb);
+			else
+#endif
 			if (eaten)
 				__kfree_skb(skb);
 			else

Could you #define copied_early to 0 and avoid ifdefs?

Quoted 8 lines
net/ipv4/tcp_input.c +3 -1
@@ -1091,8 +1094,18 @@ process:
 	bh_lock_sock(sk);
 	ret = 0;
 	if (!sock_owned_by_user(sk)) {
-		if (!tcp_prequeue(sk, skb))
+#ifdef CONFIG_NET_DMA
+		struct tcp_sock *tp = tcp_sk(sk);
+		if (!tp->ucopy.dma_chan && tp->ucopy.locked_list)

net/ipv4/tcp_input.c process: · line 1109

Quoted 10 lines
net/ipv4/tcp_input.c +8 -0
+			tp->ucopy.dma_chan = get_softnet_dma();
+		if (tp->ucopy.dma_chan)
+			ret = tcp_v4_do_rcv(sk, skb);
+		else
+#endif
+		{
+			if (!tcp_prequeue(sk, skb))
 			ret = tcp_v4_do_rcv(sk, skb);
+		}
 	} else

Wrong indentation...
Pavel

Signature · 1 line
Thanks, Sharp!
#33 lore raw
In reply to Chris Leech
Quoted 7 lines

This patch series is the first full release of the Intel(R) I/O Acceleration Technology (I/OAT) for Linux. It includes an in kernel API for offloading memory copies to hardware, a driver for the I/OAT DMA memcpy engine, and changes to the TCP stack to offload copies of received networking data to application space.

Does this buy the normal standard desktop user anything?

Jan Engelhardt

#4 lore raw

Re: [PATCH 1/8] [I/OAT] DMA memcpy subsystem

In reply to Chris Leech
Quoted 2 lines · Chris Leech
(quoted diff) +2 -0
+void dma_async_device_unregister(struct dma_device* device)
+{

...

drivers/dma/dmaengine.c line 341

Quoted 3 lines
drivers/dma/dmaengine.c new file +3 -0
+	kref_put(&device->refcount, dma_async_device_cleanup);
+	wait_for_completion(&device->done);
+}

This looks like a bug: device is dereferenced after it is potentially freed.

-ben

Signature · 2 lines
"Time is of no importance, Mr. President, only life is important."
Don't Email: <[email protected]>.
#35 lore raw
Quoted 6 lines · David S. Miller

From: Jan Engelhardt <[email protected]>
Date: Sat, 4 Mar 2006 19:46:22 +0100 (MET)

Quoted 1 line

Does this buy the normal standard desktop user anything?

Absolutely, it optimizes end-node performance.

Is this quantifiable?, and does it only apply to Intel?

Signature · 7 lines
Cheers, Gene
People having trouble with vz bouncing email to me should add the word
'online' between the 'verizon', and the dot which bypasses vz's
stupid bounce rules.  I do use spamassassin too. :-)
Yahoo.com and AOL/TW attorneys please note, additions to the above
message by Gene Heskett are:
Copyright 2006 by Maurice Eugene Heskett, all rights reserved.
maintainer
#36 lore raw

From: Gene Heskett <[email protected]>
Date: Sat, 04 Mar 2006 17:05:41 -0500

Quoted 9 lines · David S. Miller
Quoted 6 lines

From: Jan Engelhardt <[email protected]>
Date: Sat, 4 Mar 2006 19:46:22 +0100 (MET)

Quoted 1 line

Does this buy the normal standard desktop user anything?

Absolutely, it optimizes end-node performance.

Is this quantifiable?, and does it only apply to Intel?

It applies to whoever has a DMA engine in their computer.

What people need to understand is that this is putting the optimization in the right place, at the end nodes. This is about as old an internet architecting fundamental as you can get, keep the hard work off the routers and intermediate nodes, and put it on the end systems.

Intel has done performance numbers, and I'm sure they will post them at the appropriate time.

#17 lore raw

Re: [PATCH 8/8] [I/OAT] TCP recv offload to I/OAT

In reply to Chris Leech

net/ipv4/tcp.c line 267

Quoted 11 lines · Chris Leech
net/ipv4/tcp.c +3 -0
index 13abfa2..b792048 100644
@@ -262,6 +262,9 @@
 #include <net/tcp.h>
 #include <net/xfrm.h>
 #include <net/ip.h>
+#ifdef CONFIG_NET_DMA
+#include <net/netdma.h>
+#endif

#ifdef is not needed here (try not to put #ifdef in .c files.) I think a few of your other usages of #ifdef in this file can also be removed with judicious use of inline functions in a .h file.

thanks,

greg k-h

#43 lore raw
In reply to David S. Miller
Quoted 6 lines · David S. Miller

From: Jan Engelhardt <[email protected]>
Date: Sat, 4 Mar 2006 19:46:22 +0100 (MET)

Quoted 1 line

Does this buy the normal standard desktop user anything?

Absolutely, it optimizes end-node performance.

It really depends on how it is used.
According to investigation made for kevent based FS AIO reading, get_user_pages() performange graph looks like sqrt() function with plato starting on about 64-80 pages on Xeon 2.4Ghz with 1Gb of ram, while memcopy() is linear, so it can be noticebly slower than copy_to_user() if get_user_pages() is used aggressively, so userspace application must reuse the same, already grabbed buffer for maximum performance, but Intel folks did not provide theirs usage case and any benchmarks as far as I know.

Signature · 1 line
	Evgeniy Polyakov
maintainer
#44 lore raw

From: Evgeniy Polyakov <[email protected]>
Date: Sun, 5 Mar 2006 04:43:25 +0300

Quoted 8 lines

According to investigation made for kevent based FS AIO reading, get_user_pages() performange graph looks like sqrt() function with plato starting on about 64-80 pages on Xeon 2.4Ghz with 1Gb of ram, while memcopy() is linear, so it can be noticebly slower than copy_to_user() if get_user_pages() is used aggressively, so userspace application must reuse the same, already grabbed buffer for maximum performance, but Intel folks did not provide theirs usage case and any benchmarks as far as I know.

Of course, and programming the DMA controller has overhead as well. This is why would would not use I/O AT with small transfer sizes.

#8 lore raw

Re: [PATCH 4/8] [I/OAT] Utility functions for offloading sk_buff to iovec copies

In reply to Chris Leech

drivers/dma/iovlock.c line 38

Quoted 5 lines · Chris Leech
drivers/dma/iovlock.c new file +4 -0
+
+#define NUM_PAGES_SPANNED(start, length) \
+	((PAGE_ALIGN((unsigned long)start + length) - \
+	((unsigned long)start & PAGE_MASK)) >> PAGE_SHIFT)

static inline all-lower-case functions are much nicer.

drivers/dma/iovlock.c line 49

Quoted 10 lines
drivers/dma/iovlock.c new file +10 -0
+/*
+ * Lock down all the iovec pages needed for len bytes.
+ * Return a struct dma_locked_list to keep track of pages locked down.
+ *
+ * We are allocating a single chunk of memory, and then carving it up into
+ * 3 sections, the latter 2 whose size depends on the number of iovecs and the
+ * total number of pages, respectively.
+ */
+int dma_lock_iovec_pages(struct iovec *iov, size_t len, struct dma_locked_list
+	**locked_list)

Please rename this to dma_pin_iovec_pages(). Locking a page is a quite different concept from pinning it, and this function doesn't lock any pages.

drivers/dma/iovlock.c line 58

Quoted 9 lines
drivers/dma/iovlock.c new file +9 -0
+{
+	struct dma_locked_list *local_list;
+	struct page **pages;
+	int i;
+	int ret;
+
+	int nr_iovecs = 0;
+	int iovec_len_used = 0;
+	int iovec_pages_used = 0;

Extraneous blank line there.

drivers/dma/iovlock.c line 64

Quoted 5 lines
drivers/dma/iovlock.c new file +5 -0
+	/* don't lock down non-user-based iovecs */
+	if (segment_eq(get_fs(), KERNEL_DS)) {
+		*locked_list = NULL;
+		return 0;
+	}

hm, haven't seen that before. Makes sense, I guess.

drivers/dma/iovlock.c line 77

Quoted 12 lines
drivers/dma/iovlock.c new file +12 -0
+	/* determine how many iovecs/pages there are, up front */
+	do {
+		iovec_len_used += iov[nr_iovecs].iov_len;
+		iovec_pages_used += NUM_PAGES_SPANNED(iov[nr_iovecs].iov_base,
+		                                      iov[nr_iovecs].iov_len);
+		nr_iovecs++;
+	} while (iovec_len_used < len);
+
+	/* single kmalloc for locked list, page_list[], and the page arrays */
+	local_list = kmalloc(sizeof(*local_list)
+		+ (nr_iovecs * sizeof (struct dma_page_list))
+		+ (iovec_pages_used * sizeof (struct page*)), GFP_KERNEL);

What is the upper bound on the size of this allocation?

drivers/dma/iovlock.c line 85

Quoted 8 lines
drivers/dma/iovlock.c new file +8 -0
+	if (!local_list)
+		return -ENOMEM;
+
+	/* list of pages starts right after the page list array */
+	pages = (struct page **) &local_list->page_list[nr_iovecs];
+
+	/* it's a userspace pointer */
+	might_sleep();

kmalloc(GFP_KERNEL) already did that.

drivers/dma/iovlock.c line 95

Quoted 9 lines
drivers/dma/iovlock.c new file +9 -0
+	for (i = 0; i < nr_iovecs; i++) {
+		struct dma_page_list *page_list = &local_list->page_list[i];
+
+		len -= iov[i].iov_len;
+
+		if (!access_ok(VERIFY_WRITE, iov[i].iov_base, iov[i].iov_len)) {
+			dma_unlock_iovec_pages(local_list);
+			return -EFAULT;
+		}

A return statement buried down in the guts of a largeish function isn't good from a code maintainability POV.

Quoted 8 lines
drivers/dma/iovlock.c new file +8 -0
+		page_list->nr_pages = NUM_PAGES_SPANNED(iov[i].iov_base,
+		                                        iov[i].iov_len);
+		page_list->base_address = iov[i].iov_base;
+
+		page_list->pages = pages;
+		pages += page_list->nr_pages;
+
+		/* lock pages down */

drivers/dma/iovlock.c line 114

Quoted 10 lines
drivers/dma/iovlock.c new file +10 -0
+		down_read(&current->mm->mmap_sem);
+		ret = get_user_pages(
+			current,
+			current->mm,
+			(unsigned long) iov[i].iov_base,
+			page_list->nr_pages,
+			1,
+			0,
+			page_list->pages,
+			NULL);

Yes, it has a lot of args. It's nice to add comments like this:

		ret = get_user_pages(
			current,
			current->mm,
			(unsigned long) iov[i].iov_base,
			page_list->nr_pages,
			1,			/* write */
			0,			/* force */
			page_list->pages,
			NULL);

drivers/dma/iovlock.c line 119

Quoted 5 lines
drivers/dma/iovlock.c new file +5 -0
+		up_read(&current->mm->mmap_sem);
+
+		if (ret != page_list->nr_pages) {
+			goto mem_error;
+		}

Unneded braces.

drivers/dma/iovlock.c line 125

Quoted 5 lines
drivers/dma/iovlock.c new file +5 -0
+		local_list->nr_iovecs = i + 1;
+	}
+
+	*locked_list = local_list;
+	return 0;

Suggest you change this function to return locked_list, or an IS_ERR value on error.

Quoted 7 lines
drivers/dma/iovlock.c new file +7 -0
+void dma_unlock_iovec_pages(struct dma_locked_list *locked_list)
+{
+	int i, j;
+
+	if (!locked_list)
+		return;
+

drivers/dma/iovlock.c line 148

Quoted 10 lines
drivers/dma/iovlock.c new file +10 -0
+	for (i = 0; i < locked_list->nr_iovecs; i++) {
+		struct dma_page_list *page_list = &locked_list->page_list[i];
+		for (j = 0; j < page_list->nr_pages; j++) {
+			SetPageDirty(page_list->pages[j]);
+			page_cache_release(page_list->pages[j]);
+		}
+	}
+
+	kfree(locked_list);
+}

SetPageDirty() is very wrong. It fails to mark pagecache pages as dirty in the radix tree so they won't get written back.

You'll need to use set_page_dirty_lock() here or, if you happen to have protected the inode which backs this potential mmap (really the address_space) from reclaim then set_page_dirty() will work. Probably it'll be set_page_dirty_lock().

If this is called from cant-sleep context then things get ugly. If it's called from interrupt context then moreso. See fs/direct-io.c, bio_set_pages_dirty(), bio_check_pages_dirty(), etc.

I don't see a check for "did we write to user pages" here. Because we don't need to dirty the pages if we were reading them (transmitting from userspace).

But given that dma_lock_iovec_pages() is only set up for writing to userspace I guess this code is implicitly receive-only. It's hard to tell when the description, is, like the code comments, so scant.

drivers/dma/iovlock.c line 151

Quoted 2 lines
drivers/dma/iovlock.c new file +2 -0
+static dma_cookie_t dma_memcpy_tokerneliovec(struct dma_chan *chan, struct
+	iovec *iov, unsigned char *kdata, size_t len)

You owe us two underscores ;)

Quoted 2 lines

+/*
+ * We have already locked down the pages we will be using in the iovecs.

"pinned"

Quoted 19 lines

+ * Each entry in iov array has corresponding entry in locked_list->page_list.
+ * Using array indexing to keep iov[] and page_list[] in sync.
+ * Initial elements in iov array's iov->iov_len will be 0 if already copied into
+ * by another call.
+ * iov array length remaining guaranteed to be bigger than len.
+ */

drivers/dma/iovlock.c new file +13 -0
+dma_cookie_t dma_memcpy_toiovec(struct dma_chan *chan, struct iovec *iov,
+	struct dma_locked_list *locked_list, unsigned char *kdata, size_t len)
+{
+	int iov_byte_offset;
+	int copy;
+	dma_cookie_t dma_cookie = 0;
+	int iovec_idx;
+	int page_idx;
+
+	if (!chan)
+		return memcpy_toiovec(iov, kdata, len);
+
+	/* -> kernel copies (e.g. smbfs) */

drivers/dma/iovlock.c line 204

Quoted 10 lines
drivers/dma/iovlock.c new file +10 -0
+	if (!locked_list)
+		return dma_memcpy_tokerneliovec(chan, iov, kdata, len);
+
+	iovec_idx = 0;
+	while (iovec_idx < locked_list->nr_iovecs) {
+		struct dma_page_list *page_list;
+
+		/* skip already used-up iovecs */
+		while (!iov[iovec_idx].iov_len)
+			iovec_idx++;

Is it assured that this array was zero-terminated?

drivers/dma/iovlock.c line 244

Quoted 4 lines
drivers/dma/iovlock.c new file +4 -0
+
+dma_cookie_t dma_memcpy_pg_toiovec(struct dma_chan *chan, struct iovec *iov,
+	struct dma_locked_list *locked_list, struct page *page,
+	unsigned int offset, size_t len)

pleeeeeze comment your code.

drivers/dma/iovlock.c line 255

Quoted 11 lines
drivers/dma/iovlock.c new file +11 -0
+{
+	int iov_byte_offset;
+	int copy;
+	dma_cookie_t dma_cookie = 0;
+	int iovec_idx;
+	int page_idx;
+	int err;
+
+	/* this needs as-yet-unimplemented buf-to-buff, so punt. */
+	/* TODO: use dma for this */
+	if (!chan || !locked_list) {

Really you should rename locked_list to pinned_list throughout, and dma_locked_list to dma_pinned_list.

drivers/dma/iovlock.c line 204

Quoted 7 lines
drivers/dma/iovlock.c new file +7 -0
+	iovec_idx = 0;
+	while (iovec_idx < locked_list->nr_iovecs) {
+		struct dma_page_list *page_list;
+
+		/* skip already used-up iovecs */
+		while (!iov[iovec_idx].iov_len)
+			iovec_idx++;

Can this also run off the end?

drivers/dma/iovlock.c line 318

Quoted 10 lines
drivers/dma/iovlock.c new file +10 -0
+int dma_lock_iovec_pages(struct iovec *iov, size_t len, struct dma_locked_list
+	**locked_list)
+{
+	*locked_list = NULL;
+
+	return 0;
+}
+
+void dma_unlock_iovec_pages(struct dma_locked_list* locked_list)
+{ }

You might want to make these guys static inlines in a header and not compile this file at all if !CONFIG_DMA_ENGINE.

Review

Quoted 2 lines
(quoted diff) +2 -0
+struct dma_page_list
+{
   struct dma_page_list {

include/linux/dmaengine.h static inline enum dma_status dma_async_ · line 332

Quoted 2 lines
include/linux/dmaengine.h +2 -0
+struct dma_locked_list
+{
   struct dma_pinned_list {

include/linux/dmaengine.h static inline enum dma_status dma_async_ · line 334

Quoted 2 lines
include/linux/dmaengine.h +2 -0
+	int nr_iovecs;
+	struct dma_page_list page_list[0];

We can use [] instead of [0] now that gcc-2.95.x has gone away.

include/linux/dmaengine.h static inline enum dma_status dma_async_ · line 339

Quoted 3 lines
include/linux/dmaengine.h +3 -0
+int dma_lock_iovec_pages(struct iovec *iov, size_t len,
+	struct dma_locked_list	**locked_list);
+void dma_unlock_iovec_pages(struct dma_locked_list* locked_list);

"pin", "unpin".

net/core/user_dma.c line 44

Quoted 11 lines
net/core/user_dma.c new file +11 -0
+#ifdef CONFIG_NET_DMA
+
+/**
+ *	dma_skb_copy_datagram_iovec - Copy a datagram to an iovec.
+ *	@skb - buffer to copy
+ *	@offset - offset in the buffer to start copying from
+ *	@iovec - io vector to copy to
+ *	@len - amount of data to copy from buffer to iovec
+ *	@locked_list - locked iovec buffer data
+ *
+ *	Note: the iovec is modified during the copy.

Modifying the caller's iovec is a bit rude. Hard to avoid, I guess.

Quoted 8 lines

+ */

net/core/user_dma.c new file +7 -0
+int dma_skb_copy_datagram_iovec(struct dma_chan *chan,
+			struct sk_buff *skb, int offset, struct iovec *to,
+			size_t len, struct dma_locked_list *locked_list)
+{
+	int start = skb_headlen(skb);
+	int i, copy = start - offset;
+	dma_cookie_t cookie = 0;

net/core/user_dma.c line 62

Quoted 10 lines
net/core/user_dma.c new file +10 -0
+
+	/* Copy header. */
+	if (copy > 0) {
+		if (copy > len)
+			copy = len;
+		if ((cookie = dma_memcpy_toiovec(chan, to, locked_list,
+		     skb->data + offset, copy)) < 0)
+			goto fault;
+		if ((len -= copy) == 0)
+			goto end;

Please avoid

	if ((lhs = rhs))

constructs. Instead do

	lhs = rhs;
	if (lhs)

(entire patchset - there are quite a lot)

net/core/user_dma.c line 70

Quoted 8 lines
net/core/user_dma.c new file +8 -0
+		offset += copy;
+	}
+
+	/* Copy paged appendix. Hmm... why does this look so complicated? */
+	for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
+		int end;
+
+		BUG_TRAP(start <= offset + len);

<wonders why BUG_TRAP still exists>

net/core/user_dma.c line 106

Quoted 8 lines
net/core/user_dma.c new file +4 -0
+		if ((copy = end - offset) > 0) {
...
+			if (!(len -= copy))
...
+			if ((copy = end - offset) > 0) {
...
+				if ((len -= copy) == 0)

See above.

net/core/user_dma.c line 133

Quoted 10 lines
net/core/user_dma.c new file +10 -0
+#else
+
+int dma_skb_copy_datagram_iovec(struct dma_chan *chan,
+			const struct sk_buff *skb, int offset, struct iovec *to,
+			size_t len, struct dma_locked_list *locked_list)
+{
+	return skb_copy_datagram_iovec(skb, offset, to, len);
+}
+
+#endif

Again, consider putting this in a header as an inline, avoid compiling this file altogether.

#10 lore raw

Re: [PATCH 5/8] [I/OAT] Structure changes for TCP recv offload to I/OAT

In reply to Chris Leech
Quoted 4 lines · Chris Leech

+#ifdef CONFIG_NET_DMA
+#include <linux/dmaengine.h>
+#endif

Please move the ifdefs into the header and include it unconditionally (entire patchset).

#14 lore raw

Re: [PATCH 7/8] [I/OAT] Add a sysctl for tuning the I/OAT offloaded I/O threshold

In reply to Chris Leech
Quoted 6 lines · Chris Leech

Any socket recv of less than this ammount will not be offloaded

...

+int sysctl_tcp_dma_copybreak = NET_DMA_DEFAULT_COPYBREAK;

Is it appropriate that this tunable be kernel-wide, rather than more finely-grained?

#19 lore raw

Re: [PATCH 8/8] [I/OAT] TCP recv offload to I/OAT

In reply to Chris Leech
Quoted 4 lines · Chris Leech

Locks down user pages and sets up for DMA in tcp_recvmsg, then calls dma_async_try_early_copy in tcp_v4_do_rcv

+#ifdef CONFIG_NET_DMA
+#ifdef CONFIG_NET_DMA
+#ifdef CONFIG_NET_DMA
+#ifdef CONFIG_NET_DMA
+#ifdef CONFIG_NET_DMA
+#ifdef CONFIG_NET_DMA
+#ifdef CONFIG_NET_DMA
+#ifdef CONFIG_NET_DMA
+#ifdef CONFIG_NET_DMA
+#ifdef CONFIG_NET_DMA
+#ifdef CONFIG_NET_DMA
+#ifdef CONFIG_NET_DMA
+#ifdef CONFIG_NET_DMA
+#ifdef CONFIG_NET_DMA
+#ifdef CONFIG_NET_DMA
+#ifdef CONFIG_NET_DMA

waaay too many ifdefs. There are various tricks we use to minimise them.

net/ipv4/tcp.c int tcp_recvmsg(struct kiocb *iocb, stru · line 1143

Quoted 5 lines
net/ipv4/tcp.c +5 -0
+#ifdef CONFIG_NET_DMA
+	tp->ucopy.dma_chan = NULL;
+	if ((len > sysctl_tcp_dma_copybreak) && !(flags & MSG_PEEK) && !sysctl_tcp_low_latency && __get_cpu_var(softnet_data.net_dma))
+		dma_lock_iovec_pages(msg->msg_iov, len, &tp->ucopy.locked_list);
+#endif

Please try to fit code into 80 columns.

That's decimal 80 ;)

Quoted 23 lines
net/ipv4/tcp.c +12 -7
@@ -1328,13 +1342,39 @@ do_prequeue:
 		}
 
 		if (!(flags & MSG_TRUNC)) {
-			err = skb_copy_datagram_iovec(skb, offset,
-						      msg->msg_iov, used);
-			if (err) {
-				/* Exception. Bailout! */
-				if (!copied)
-					copied = -EFAULT;
-				break;
+#ifdef CONFIG_NET_DMA
+			if (!tp->ucopy.dma_chan && tp->ucopy.locked_list)
+				tp->ucopy.dma_chan = get_softnet_dma();
+
+			if (tp->ucopy.dma_chan) {
+				tp->ucopy.dma_cookie = dma_skb_copy_datagram_iovec(
+					tp->ucopy.dma_chan, skb, offset,
+					msg->msg_iov, used,
+					tp->ucopy.locked_list);
+
+				if (tp->ucopy.dma_cookie < 0) {
+

net/ipv4/tcp.c do_prequeue: · line 1366

Quoted 10 lines
net/ipv4/tcp.c +10 -0
+					printk(KERN_ALERT "dma_cookie < 0\n");
+
+					/* Exception. Bailout! */
+					if (!copied)
+						copied = -EFAULT;
+					break;
+				}
+				if ((offset + used) == skb->len)
+					copied_early = 1;
+

Consider trimming some of those blank lines. I don't think they add any value?

net/ipv4/tcp.c do_prequeue: · line 1369

Quoted 3 lines
net/ipv4/tcp.c +3 -0
+			} else
+#endif
+			{

These games with ifdefs and else statements aren't at all pleasant. Sometimes they're hard to avoid, but you'll probably find that some code rearrangemnt (in a preceding patch) makes it easier. Like, split this function into several.

Quoted 19 lines
net/ipv4/tcp.c +11 -3
@@ -1354,15 +1394,33 @@ skip_copy:
 
 		if (skb->h.th->fin)
 			goto found_fin_ok;
-		if (!(flags & MSG_PEEK))
-			sk_eat_skb(sk, skb);
+		if (!(flags & MSG_PEEK)) {
+			if (!copied_early)
+				sk_eat_skb(sk, skb);
+#ifdef CONFIG_NET_DMA
+			else {
+				__skb_unlink(skb, &sk->sk_receive_queue);
+				__skb_queue_tail(&sk->sk_async_wait_queue, skb);
+				copied_early = 0;
+			}
+#endif
...
-			sk_eat_skb(sk, skb);
+		if (!(flags & MSG_PEEK)) {

net/ipv4/tcp.c skip_copy: · line 1423

Quoted 10 lines
net/ipv4/tcp.c +10 -0
+			if (!copied_early)
+				sk_eat_skb(sk, skb);
+#ifdef CONFIG_NET_DMA
+			else {
+				__skb_unlink(skb, &sk->sk_receive_queue);
+				__skb_queue_tail(&sk->sk_async_wait_queue, skb);
+				copied_early = 0;
+			}
+#endif
+		}

etc.

net/ipv4/tcp_input.c int tcp_rcv_established(struct sock *sk, · line 3989

Quoted 8 lines
net/ipv4/tcp_input.c +5 -0
+#ifdef CONFIG_NET_DMA
+			if (copied_early)
+				__skb_queue_tail(&sk->sk_async_wait_queue, skb);
+			else
+#endif
 			if (eaten)
 				__kfree_skb(skb);
 			else

etc.

Quoted 4 lines
net/ipv4/tcp_input.c +0 -0
@@ -4049,6 +4067,52 @@ discard:
 	return 0;
 }
 

net/ipv4/tcp_input.c discard: · line 4079

Quoted 10 lines
net/ipv4/tcp_input.c +10 -0
+#ifdef CONFIG_NET_DMA
+int dma_async_try_early_copy(struct sock *sk, struct sk_buff *skb, int hlen)
+{
+	struct tcp_sock *tp = tcp_sk(sk);
+	int chunk = skb->len - hlen;
+	int dma_cookie;
+	int copied_early = 0;
+
+	if (tp->ucopy.wakeup)
+          	goto out;

In this case a simple

		return 0;

would be fine. We haven't done anything yet.

net/ipv4/tcp_ipv4.c process: · line 1109

Quoted 13 lines
net/ipv4/tcp_ipv4.c +11 -0
+#ifdef CONFIG_NET_DMA
+		struct tcp_sock *tp = tcp_sk(sk);
+		if (!tp->ucopy.dma_chan && tp->ucopy.locked_list)
+			tp->ucopy.dma_chan = get_softnet_dma();
+		if (tp->ucopy.dma_chan)
+			ret = tcp_v4_do_rcv(sk, skb);
+		else
+#endif
+		{
+			if (!tcp_prequeue(sk, skb))
 			ret = tcp_v4_do_rcv(sk, skb);
+		}
 	} else

etc.

net/ipv6/tcp_ipv6.c process: · line 1231

Quoted 11 lines
net/ipv6/tcp_ipv6.c +10 -0
+#ifdef CONFIG_NET_DMA
+                struct tcp_sock *tp = tcp_sk(sk);
+                if (tp->ucopy.dma_chan)
+                        ret = tcp_v6_do_rcv(sk, skb);
+                else
+#endif
+		{
+			if (!tcp_prequeue(sk, skb))
+				ret = tcp_v6_do_rcv(sk, skb);
+		}
 	} else

ow, my eyes!

#27 lore raw
In reply to Chris Leech
Quoted 14 lines · Chris Leech
Quoted 2 lines

Patch #2 didn't make it. Too big for the list?

Could be, it's the largest of the series. I've attached the gziped patch. I can try and split this up for the future.

..

[I/OAT] Driver for the Intel(R) I/OAT DMA engine
Adds a new ioatdma driver

...

(quoted diff) +2 -0
+struct cb_pci_pmcap_register {
+	uint32_t	capid:8;	/* RO: 01h */

Review

Quoted 10 lines
(quoted diff) +10 -0
+	uint32_t	nxtcapptr:8;
+	uint32_t	version:3;	/* RO: 010b */
+	uint32_t	pmeclk:1;	/* RO: 0b */
+	uint32_t	reserved:1;	/* RV: 0b */
+	uint32_t	dsi:1;		/* RO: 0b */
+	uint32_t	aux_current:3;	/* RO: 000b */
+	uint32_t	d1_support:1;	/* RO: 0b */
+	uint32_t	d2_support:1;	/* RO: 0b */
+	uint32_t	pme_support:5;	/* RO: 11001b */
+};

This maps onto hardware registers? No big-endian plans in Intel's future? ;)

I have a vague feeling that gcc changed its layout of bitfields many years ago. I guess we're fairly safe against that. Presumably gcc and icc use the same layout?

Still. It's a bit of a concern, but I guess we can worry about that if it happens.

Review

Quoted 5 lines
(quoted diff) +5 -0
+
+static inline u8 read_reg8(struct cb_device *device, unsigned int offset)
+{
+	return readb(device->reg_base + offset);
+}

These are fairly generic-sounding names. In fact the as-yet-unmerged tiacx wireless driver is already using these, privately to drivers/net/wireless/tiacx/pci.c.

Review

Quoted 10 lines
(quoted diff) +10 -0
+static int enumerate_dma_channels(struct cb_device *device)
+{
+	u8 xfercap_scale;
+	u32 xfercap;
+	int i;
+	struct cb_dma_chan *cb_chan;
+
+	device->common.chancnt = read_reg8(device, CB_CHANCNT_OFFSET);
+	xfercap_scale = read_reg8(device, CB_XFERCAP_OFFSET);
+	xfercap = (xfercap_scale == 0 ? ~0UL : (1 << xfercap_scale));

I recommend using just "-1" to represent the all-ones pattern. It simply works, in all situations.

Where you _did_ want the UL was after that "1".

Review

Quoted 4 lines
(quoted diff) +4 -0
+	for (i = 0; i < device->common.chancnt; i++) {
+		cb_chan = kzalloc(sizeof(*cb_chan), GFP_KERNEL);
+		if (!cb_chan)
+			return -ENOMEM;

memory leak?

Review

Quoted 11 lines
(quoted diff) +11 -0
+		cb_chan->device = device;
+		cb_chan->reg_base = device->reg_base + (0x80 * (i + 1));
+		cb_chan->xfercap = xfercap;
+		spin_lock_init(&cb_chan->cleanup_lock);
+		spin_lock_init(&cb_chan->desc_lock);
+		INIT_LIST_HEAD(&cb_chan->free_desc);
+		INIT_LIST_HEAD(&cb_chan->used_desc);
+		/* This should be made common somewhere in dmaengine.c */
+		cb_chan->common.device = &device->common;
+		cb_chan->common.client = NULL;
+		list_add_tail(&cb_chan->common.device_node, &device->common.channels);

No locking needed for that list?

Quoted 1 line

+static struct cb_desc_sw * cb_dma_alloc_descriptor(struct cb_dma_chan *cb_chan)

There's a mix of styles here. I don't think the space after the asterisk does anything useful, and it could be argued that it's incorrect (or misleading) wrt C declaration semantics.

Review

Quoted 2 lines
(quoted diff) +2 -0
+{
+	struct cb_dma_descriptor *desc;

What do all these "cb"'s stand for, anyway?

Review

Quoted 9 lines
(quoted diff) +9 -0
+	struct cb_desc_sw *desc_sw;
+	struct cb_device *cb_device = to_cb_device(cb_chan->common.device);
+	dma_addr_t phys;
+
+	desc = pci_pool_alloc(cb_device->dma_pool, GFP_ATOMIC, &phys);
+	if (!desc)
+		return NULL;
+
+	desc_sw = kzalloc(sizeof(*desc_sw), GFP_ATOMIC);

GFP_ATOMIC is to be avoided if at all possible. It stresses the memory system and can easily fail under load.

Quoted 1 line

From my reading, two of the callers could trivially call this function outside

spin_lock_bh() and the third could perhaps do so with a little work. You could at least fix up two of those callers, and pass in the gfp_flags.

<wonders why the heck dma_pool_alloc() uses SLAB_ATOMIC when the caller's passing in the gfp_flags>

Quoted 5 lines
(quoted diff) +4 -0
+/* returns the actual number of allocated descriptors */
+static int cb_dma_alloc_chan_resources(struct dma_chan *chan)
+{
...
+	/* Allocate descriptors */

Review

Quoted 10 lines
(quoted diff) +10 -0
+	spin_lock_bh(&cb_chan->desc_lock);
+	for (i = 0; i < INITIAL_CB_DESC_COUNT; i++) {
+		desc = cb_dma_alloc_descriptor(cb_chan);
+		if (!desc) {
+			printk(KERN_ERR "CB: Only %d initial descriptors\n", i);
+			break;
+		}
+		list_add_tail(&desc->node, &cb_chan->free_desc);
+	}
+	spin_unlock_bh(&cb_chan->desc_lock);

Here's one such caller.

Quoted 16 lines
(quoted diff) +16 -0
+
+static void cb_dma_free_chan_resources(struct dma_chan *chan)
+{
+	struct cb_dma_chan *cb_chan = to_cb_chan(chan);
+	struct cb_device *cb_device = to_cb_device(chan->device);
+	struct cb_desc_sw *desc, *_desc;
+	u16 chanctrl;
+	int in_use_descs = 0;
+
+	cb_dma_memcpy_cleanup(cb_chan);
+
+	chan_write_reg8(cb_chan, CB_CHANCMD_OFFSET, CB_CHANCMD_RESET);
+
+	spin_lock_bh(&cb_chan->desc_lock);
+	list_for_each_entry_safe(desc, _desc, &cb_chan->used_desc, node) {
+		in_use_descs++;

Review

Quoted 10 lines
(quoted diff) +10 -0
+		list_del(&desc->node);
+		pci_pool_free(cb_device->dma_pool, desc->hw, desc->phys);
+		kfree(desc);
+	}
+	list_for_each_entry_safe(desc, _desc, &cb_chan->free_desc, node) {
+		list_del(&desc->node);
+		pci_pool_free(cb_device->dma_pool, desc->hw, desc->phys);
+		kfree(desc);
+	}
+	spin_unlock_bh(&cb_chan->desc_lock);

Do we actually need the lock there? If we're freeing everything which it protects anwyay?

Review

Quoted 11 lines
(quoted diff) +11 -0
+
+static void cb_dma_memcpy_cleanup(struct cb_dma_chan *chan)
+{
+	unsigned long phys_complete;
+	struct cb_desc_sw *desc, *_desc;
+	dma_cookie_t cookie = 0;
+
+	prefetch(chan->completion_virt);
+
+	if (!spin_trylock(&chan->cleanup_lock))
+		return;

What's going on here? Lock ranking problems? spin_trylock() in non-infrastructural code is a bit of a red flag.

Whatever the reason, it needs a comment in there please. That comment should also explain why simply baling out is acceptable.

Review

Quoted 12 lines
(quoted diff) +12 -0
+
+static irqreturn_t cb_do_interrupt(int irq, void *data, struct pt_regs *regs)
+{
+	struct cb_device *instance = data;
+	unsigned long attnstatus;
+	u8 intrctrl;
+
+	intrctrl = read_reg8(instance, CB_INTRCTRL_OFFSET);
+
+	if (!(intrctrl & CB_INTRCTRL_MASTER_INT_EN)) {
+		return IRQ_NONE;
+	}

braces.

Review

Quoted 1 line
(quoted diff) +1 -0
+	attnstatus = (unsigned long) read_reg32(instance, CB_ATTNSTATUS_OFFSET);

Unneeded cast.

Quoted 21 lines
(quoted diff) +21 -0
+static void cb_start_null_desc(struct cb_dma_chan *cb_chan)
+{
+	struct cb_desc_sw *desc;
+
+	spin_lock_bh(&cb_chan->desc_lock);
+
+	if (!list_empty(&cb_chan->free_desc)) {
+		desc = to_cb_desc(cb_chan->free_desc.next);
+		list_del(&desc->node);
+	} else {
+		/* try to get another desc */
+		desc = cb_dma_alloc_descriptor(cb_chan);
+		/* will this ever happen? */
+		BUG_ON(!desc);
+	}
+
+	desc->hw->ctl = CB_DMA_DESCRIPTOR_NUL;
+	desc->hw->next = 0;
+
+	list_add_tail(&desc->node, &cb_chan->used_desc);
+

Review

Quoted 10 lines
(quoted diff) +10 -0
+#if (BITS_PER_LONG == 64)
+	chan_write_reg64(cb_chan, CB_CHAINADDR_OFFSET, desc->phys);
+#else
+	chan_write_reg32(cb_chan, CB_CHAINADDR_OFFSET_LOW, (u32) desc->phys);
+	chan_write_reg32(cb_chan, CB_CHAINADDR_OFFSET_HIGH, 0);
+#endif
+	chan_write_reg8(cb_chan, CB_CHANCMD_OFFSET, CB_CHANCMD_START);
+
+	spin_unlock_bh(&cb_chan->desc_lock);
+}

Can the chan_write*() calls be moved outside the locked region?

Quoted 3 lines

+/*
+ * Perform a CB transaction to verify the HW works.
+ */

Damn, I wish I knew what CB meant.

Quoted 13 lines
(quoted diff) +13 -0
+#define CB_TEST_SIZE 2000
+
+static int cb_self_test(struct cb_device *device)
+{
+	int i;
+	u8 *src;
+	u8 *dest;
+	struct dma_chan *dma_chan;
+	dma_cookie_t cookie;
+	int err = 0;
+
+	src = kzalloc(sizeof(u8) * CB_TEST_SIZE, SLAB_KERNEL);
+	if (!src)

Review

Quoted 10 lines
(quoted diff) +10 -0
+		return -ENOMEM;
+	dest = kzalloc(sizeof(u8) * CB_TEST_SIZE, SLAB_KERNEL);
+	if (!dest) {
+		kfree(src);
+		return -ENOMEM;
+	}
+
+	/* Fill in src buffer */
+	for (i = 0; i < CB_TEST_SIZE; i++)
+		src[i] = (u8)i;

memset?

Review

Quoted 4 lines
(quoted diff) +4 -0
+	/* Start copy, using first DMA channel */
+	dma_chan = container_of(device->common.channels.next, struct dma_chan, device_node);
+
+	cb_dma_alloc_chan_resources(dma_chan);

cb_dma_alloc_chan_resources() can fail.

Review

Quoted 4 lines
(quoted diff) +4 -0
+	cookie = cb_dma_memcpy_buf_to_buf(dma_chan, dest, src, CB_TEST_SIZE);
+	cb_dma_memcpy_issue_pending(dma_chan);
+
+	udelay(1000);

msleep(1) would be preferred.

Quoted 61 lines
(quoted diff) +61 -0
+static int __devinit cb_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
+{
+	int err;
+	unsigned long mmio_start, mmio_len;
+	void *reg_base;
+	struct cb_device *device;
+
+	err = pci_enable_device(pdev);
+	if (err)
+		goto err_enable_device;
+
+	err = pci_set_dma_mask(pdev, DMA_64BIT_MASK);
+	if (err)
+		err = pci_set_dma_mask(pdev, DMA_32BIT_MASK);
+	if (err)
+		goto err_set_dma_mask;
+
+	err = pci_request_regions(pdev, cb_pci_drv.name);
+	if (err)
+		goto err_request_regions;
+
+	mmio_start = pci_resource_start(pdev, 0);
+	mmio_len = pci_resource_len(pdev, 0);
+
+	reg_base = ioremap(mmio_start, mmio_len);
+	if (!reg_base) {
+		err = -ENOMEM;
+		goto err_ioremap;
+	}
+
+	device = kzalloc(sizeof(*device), GFP_KERNEL);
+	if (!device) {
+		err = -ENOMEM;
+		goto err_kzalloc;
+	}
+
+	/* DMA coherent memory pool for DMA descriptor allocations */
+	device->dma_pool = pci_pool_create("dma_desc_pool", pdev,
+		sizeof(struct cb_dma_descriptor), 64, 0);
+	if (!device->dma_pool) {
+		err = -ENOMEM;
+		goto err_dma_pool;
+	}
+
+	device->completion_pool = pci_pool_create("completion_pool", pdev, sizeof(u64), SMP_CACHE_BYTES, SMP_CACHE_BYTES);
+	if (!device->completion_pool) {
+		err = -ENOMEM;
+		goto err_completion_pool;
+	}
+
+	device->pdev = pdev;
+	pci_set_drvdata(pdev, device);
+#ifdef CONFIG_PCI_MSI
+	if (pci_enable_msi(pdev) == 0) {
+		device->msi = 1;
+	} else {
+		device->msi = 0;
+	}
+#endif
+	err = request_irq(pdev->irq, &cb_do_interrupt, SA_SHIRQ, "ioat",
+		device);

Review

Quoted 10 lines
(quoted diff) +10 -0
+	if (err)
+		goto err_irq;
+
+	device->reg_base = reg_base;
+
+	write_reg8(device, CB_INTRCTRL_OFFSET, CB_INTRCTRL_MASTER_INT_EN);
+	pci_set_master(pdev);
+
+	INIT_LIST_HEAD(&device->common.channels);
+	enumerate_dma_channels(device);

enumerate_dma_channels() can fail.

Quoted 20 lines
(quoted diff) +20 -0
+	device->common.device_alloc_chan_resources = cb_dma_alloc_chan_resources;
+	device->common.device_free_chan_resources = cb_dma_free_chan_resources;
+	device->common.device_memcpy_buf_to_buf = cb_dma_memcpy_buf_to_buf;
+	device->common.device_memcpy_buf_to_pg = cb_dma_memcpy_buf_to_pg;
+	device->common.device_memcpy_pg_to_pg = cb_dma_memcpy_pg_to_pg;
+	device->common.device_memcpy_complete = cb_dma_is_complete;
+	device->common.device_memcpy_issue_pending = cb_dma_memcpy_issue_pending;
+	printk(KERN_INFO "Intel(R) I/OAT DMA Engine found, %d channels\n",
+		device->common.chancnt);
+
+	if ((err = cb_self_test(device)))
+		goto err_self_test;
+
+	dma_async_device_register(&device->common);
+
+	return 0;
+
+err_self_test:
+err_irq:
+	pci_pool_destroy(device->completion_pool);

Review

Quoted 10 lines
(quoted diff) +10 -0
+err_completion_pool:
+	pci_pool_destroy(device->dma_pool);
+err_dma_pool:
+	kfree(device);
+err_kzalloc:
+	iounmap(reg_base);
+err_ioremap:
+	pci_release_regions(pdev);
+err_request_regions:
+err_set_dma_mask:

You might want a pci_disable_device() in here.

Review

Quoted 10 lines
(quoted diff) +10 -0
+err_enable_device:
+	return err;
+}
+
+static void __devexit cb_remove(struct pci_dev *pdev)
+{
+	struct cb_device *device;
+
+	device = pci_get_drvdata(pdev);
+	dma_async_device_unregister(&device->common);

pci_disable_device()?

Quoted 11 lines
(quoted diff) +11 -0
+	free_irq(device->pdev->irq, device);
+#ifdef CONFIG_PCI_MSI
+	if (device->msi)
+		pci_disable_msi(device->pdev);
+#endif
+	pci_pool_destroy(device->dma_pool);
+	pci_pool_destroy(device->completion_pool);
+	iounmap(device->reg_base);
+	pci_release_regions(pdev);
+	kfree(device);
+}

Review

Quoted 10 lines
(quoted diff) +10 -0
+
+/* MODULE API */
+MODULE_VERSION("1.0");
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Intel Corporation");
+
+static int __init cb_init_module(void)
+{
+	/* it's currently unsafe to unload this module */
+	/* if forced, worst case is that rmmod hangs */

How come?

Review

Quoted 13 lines
(quoted diff) +10 -0
+	if (THIS_MODULE != NULL)
+		THIS_MODULE->unsafe = 1;
+
+	return pci_module_init(&cb_pci_drv);
+}
+
 
 
 
+#define CB_LOW_COMPLETION_MASK		0xffffffc0
+
+extern struct list_head dma_device_list;
+extern struct list_head dma_client_list;

It's strange to see extern decls for lists, but no decl for their lock. A comment might help.

Quoted 18 lines
(quoted diff) +18 -0
+struct cb_dma_chan {
+
+	void *reg_base;
+
+	dma_cookie_t completed_cookie;
+	unsigned long last_completion;
+
+	u32 xfercap;	/* XFERCAP register value expanded out */
+
+	spinlock_t cleanup_lock;
+	spinlock_t desc_lock;
+	struct list_head free_desc;
+	struct list_head used_desc;
+
+	int pending;
+
+	struct cb_device *device;
+	struct dma_chan common;

Review

Quoted 10 lines
(quoted diff) +10 -0
+
+	dma_addr_t completion_addr;
+	union {
+		u64 full; /* HW completion writeback */
+		struct {
+			u32 low;
+			u32 high;
+		};
+	} *completion_virt;
+};

Again, is it safe to assume that these parts will never be present in big-endian machines?

#20 lore raw

Re: [PATCH 8/8] [I/OAT] TCP recv offload to I/OAT

In reply to Chris Leech
Quoted 6 lines · Chris Leech

+#ifdef CONFIG_NET_DMA
+ tp->ucopy.dma_chan = NULL;
+ if ((len > sysctl_tcp_dma_copybreak) && !(flags & MSG_PEEK) && !sysctl_tcp_low_latency && __get_cpu_var(softnet_data.net_dma))
+ dma_lock_iovec_pages(msg->msg_iov, len, &tp->ucopy.locked_list);
+#endif

The __get_cpu_var() here will run smp_processor_id() from preemptible context. You'll get a big warning if the correct debug options are set.

The reason for this is that preemption could cause this code to hop between CPUs.

Please always test code with all debug options enabled and with full kernel preemption.

#28 lore raw

Discourage duplicate symbols in the kernel? [Was: Intel I/O Acc...]

In reply to Andrew Morton
Quoted 9 lines · Andrew Morton

Review

Quoted 5 lines
(quoted diff) +5 -0
+
+static inline u8 read_reg8(struct cb_device *device, unsigned int offset)
+{
+	return readb(device->reg_base + offset);
+}

These are fairly generic-sounding names. In fact the as-yet-unmerged tiacx wireless driver is already using these, privately to drivers/net/wireless/tiacx/pci.c.

Do we in general discourage duplicate symbols even if they are static?

[ppc64, allmodconfig]

$> nm vmlinux | fgrep ' t ' | awk '{print $3}' | sort | uniq -dc

      2 .add_bridge
      2 .base_probe
      2 .c_next
      2 .c_start
      2 .c_stop
      3 .cpu_callback
      2 .default_open
      2 .default_read_file
      2 .default_write_file
      2 .dev_ifsioc
      2 .do_open
      4 .dst_output
      2 .dump_seek
      2 .dump_write
      2 .elf_core_dump
      2 .elf_map
      2 .exact_lock
      2 .exact_match
      2 .exit_elf_binfmt
      2 .fill_note
      2 .fill_prstatus
      2 .fillonedir
      2 .fini
      2 .fixup_one_level_bus_range
      5 .init
      8 .init_once
      3 .iommu_bus_setup_null
      3 .iommu_dev_setup_null
      2 .klist_devices_get
      2 .klist_devices_put
      2 .load_elf_binary
      2 .load_elf_interp
      2 .load_elf_library
      3 .m_next
      3 .m_start
      3 .m_stop
      2 .maydump
      3 .modalias_show
      2 .next_device
      3 .notesize
      2 .padzero
      2 .raw_ioctl
      2 .s_next
      2 .s_show
      2 .s_start
      2 .s_stop
      2 .seq_next
      2 .seq_show
      2 .seq_start
      2 .seq_stop
      2 .set_brk
      2 .setkey
      2 .state_show
      2 .state_store
      2 .store_uevent
      2 .u3_ht_cfg_access
      2 .u3_ht_read_config
      2 .u3_ht_write_config
      2 .writenote
      3 __initcall_init
      2 __setup_netdev_boot_setup
      2 __setup_str_netdev_boot_setup

If I did a make allyesconfig the result looks much more scary.

Sam

#29 lore raw

Re: Discourage duplicate symbols in the kernel? [Was: Intel I/O Acc...]

Quoted 12 lines · Sam Ravnborg
Quoted 9 lines · Andrew Morton

Review

Quoted 5 lines
(quoted diff) +5 -0
+
+static inline u8 read_reg8(struct cb_device *device, unsigned int offset)
+{
+	return readb(device->reg_base + offset);
+}

These are fairly generic-sounding names. In fact the as-yet-unmerged tiacx wireless driver is already using these, privately to drivers/net/wireless/tiacx/pci.c.

Quoted 1 line

Do we in general discourage duplicate symbols even if they are static?

Well, it's a bit irritating that it confuses ctags. But in this case, one set is in a header file so the risk of collisions is much-increased.

maintainer
#21 lore raw

Re: [PATCH 8/8] [I/OAT] TCP recv offload to I/OAT

In reply to Andrew Morton

From: Andrew Morton <[email protected]>
Date: Sun, 5 Mar 2006 00:45:34 -0800

Quoted 8 lines

The __get_cpu_var() here will run smp_processor_id() from preemptible context. You'll get a big warning if the correct debug options are set.

The reason for this is that preemption could cause this code to hop between CPUs.

Please always test code with all debug options enabled and with full kernel preemption.

To be fair that warning doesn't trigger on some platforms, such as sparc64 where the __get_cpu_var() implementation simply takes the value from a fixed cpu register and doesn't do the debugging check.

Sparc64 should add the check when debugging options are enabled, for sure, but the point is that it may not entirely be the tester's fault. :-)

#37 lore raw
In reply to David S. Miller
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
Quoted 1 line

Does this buy the normal standard desktop user anything?

Absolutely, it optimizes end-node performance.

Is this quantifiable?, and does it only apply to Intel?

It applies to whoever has a DMA engine in their computer.

How do I find out?

Jan Engelhardt

#38 lore raw
Quoted 6 lines · Jan Engelhardt
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
Quoted 1 line

Does this buy the normal standard desktop user anything?

Absolutely, it optimizes end-node performance.

Is this quantifiable?, and does it only apply to Intel?

It applies to whoever has a DMA engine in their computer.

How do I find out?

if you have an off-the-shelf standard PC, today you don't have one of those.

#39 lore raw
In reply to David S. Miller

Hi,
Le Sat, 04 Mar 2006 14:16:43 -0800, David S. Miller a écrit :

Quoted 7 lines

From: Gene Heskett <[email protected]>
Date: Sat, 04 Mar 2006 17:05:41 -0500

Quoted 4 lines · David S. Miller
Quoted 3 lines

From: Jan Engelhardt <[email protected]>
Date: Sat, 4 Mar 2006 19:46:22 +0100 (MET)

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Does this buy the normal standard desktop user anything?

Absolutely, it optimizes end-node performance.

Is this quantifiable?, and does it only apply to Intel?

It applies to whoever has a DMA engine in their computer.

But we need a special driver ?
The IOAT driver from intel seems to expect a pci device (0x8086 0x1a38) and the common x86 computer have their dma in lpc/isa bridge.

Matthieu

#40 lore raw
Quoted 3 lines · Matthieu CASTET

But we need a special driver ?
The IOAT driver from intel seems to expect a pci device (0x8086 0x1a38) and the common x86 computer have their dma in lpc/isa bridge.

The common x86 computer does not have -asynchronous- DMA.

Jeff

#45 lore raw
In reply to Evgeniy Polyakov
Quoted 11 lines · Evgeniy Polyakov
Quoted 6 lines · David S. Miller

From: Jan Engelhardt <[email protected]>
Date: Sat, 4 Mar 2006 19:46:22 +0100 (MET)

Quoted 1 line

Does this buy the normal standard desktop user anything?

Absolutely, it optimizes end-node performance.

It really depends on how it is used.
According to investigation made for kevent based FS AIO reading, get_user_pages() performange graph looks like sqrt() function

Hmm, so I should resurrect my user page table walker abstraction?

There I would hand each page to a "recording" function, which can drop the page from the collection or coalesce it in the collector if your scatter gather implementation allows it.

Regards

Ingo Oeser

#42 lore raw
In reply to Gene Heskett
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Does this buy the normal standard desktop user anything?

Absolutely, it optimizes end-node performance.

Is this quantifiable?, and does it only apply to Intel?

What we've been developing for is a device integrated into Intel's Enterprise South Bridge 2 (ESB2), so it's a feature of Intel server platforms. But the networking changes are written so that you could drop in a driver if similar functionality existed on other architectures.

I'll look into what performance data I can share, I have to ask the marketing folks.

- Chris

#41 lore raw
In reply to Matthieu CASTET
Quoted 3 lines

But we need a special driver ?
The IOAT driver from intel seems to expect a pci device (0x8086 0x1a38) and the common x86 computer have their dma in lpc/isa bridge.

It's really about bringing the concept of a generic DMA engine up to date with modern system design in order to make it useful for I/O offload. This is a new descriptor programed memory copy engine that shows up as a PCI Express device integrated into the MCH.

- Chris

#18 lore raw

Re: [PATCH 8/8] [I/OAT] TCP recv offload to I/OAT

In reply to Greg KH
Quoted 3 lines

#ifdef is not needed here (try not to put #ifdef in .c files.) I think a few of your other usages of #ifdef in this file can also be removed with judicious use of inline functions in a .h file.

ACK on all the ifdef comments. I may have gone a little ifdef crazy making sure I could get to a zero impact state with these patches applied but CONFIG_NET_DMA turned off. I'll get these cleaned up.

- Chris

#22 lore raw

Re: [PATCH 8/8] [I/OAT] TCP recv offload to I/OAT

In reply to Andrew Morton
Quoted 8 lines · Andrew Morton
Quoted 6 lines · Chris Leech

+#ifdef CONFIG_NET_DMA
+ tp->ucopy.dma_chan = NULL;
+ if ((len > sysctl_tcp_dma_copybreak) && !(flags & MSG_PEEK) && !sysctl_tcp_low_latency && __get_cpu_var(softnet_data.net_dma))
+ dma_lock_iovec_pages(msg->msg_iov, len, &tp->ucopy.locked_list);
+#endif

Quoted 5 lines

The __get_cpu_var() here will run smp_processor_id() from preemptible context. You'll get a big warning if the correct debug options are set.

The reason for this is that preemption could cause this code to hop between CPUs.

I've been playing with different models of where to select which DMA channel to use in order to reduce cache thrash and lock contention in the driver. It's not a clean per-cpu issue because per I/O there are potentially operations happening in both the process syscall and the netrx softirq context.

Right now the code delays selection of a DMA channel until the first offload copy is ready to go, so the __get_cpu_var() you point out is just checking to see if any hardware exists for I/OAT at this point before doing the page pinning. Before anything is done with the channel the per-cpu pointer is re-read safely with preemption disabled and a reference count is incremented.

- Chris

#3 lore raw

Re: [PATCH 1/8] [I/OAT] DMA memcpy subsystem

In reply to David S. Miller
Quoted 17 lines · David S. Miller
Quoted 1 line

+static spinlock_t dma_list_lock;

Please use DEFINE_SPINLOCK().

Quoted 1 line

+static void dma_chan_free_rcu(struct rcu_head *rcu) {

Newline before the brace please.

Quoted 1 line

+static void dma_async_device_cleanup(struct kref *kref) {

Newline before the brace please.

Review

Quoted 2 lines
(quoted diff) +2 -0
+struct dma_chan_percpu
+{

Left brace on the same line as "struct dma_chan_percpu" please.

Quoted 6 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+struct dma_chan
+{

Similarly.

Otherwise this patch looks mostly ok.

Thanks Dave,

I'll apply these and other feedback and get updated patches generated.

- Chris

#5 lore raw

Re: [PATCH 1/8] [I/OAT] DMA memcpy subsystem

In reply to Benjamin LaHaise
Quoted 10 lines · Benjamin LaHaise
Quoted 2 lines · Chris Leech
(quoted diff) +2 -0
+void dma_async_device_unregister(struct dma_device* device)
+{

...

drivers/dma/dmaengine.c line 341

Quoted 3 lines
drivers/dma/dmaengine.c new file +3 -0
+     kref_put(&device->refcount, dma_async_device_cleanup);
+     wait_for_completion(&device->done);
+}

This looks like a bug: device is dereferenced after it is potentially freed.

Actually, this is where the code is waiting to make sure it's safe to free device. The release function for the kref completes device->done. Each of the devices channels holds a reference to the device. When a device is unregistered it's channels are removed from the clients, which hold a reference for each outstanding transaction. When all the outstanding transactions complete, the channels kref goes to 0, and the reference to the device is dropped. When the device kref goes to 0 the completion is set, and it's now safe to free the memory for the device and channel structures.

I have a writeup of the locking and reference counting that I'll finish and add in as a big comment to the code.

-Chris

#30 lore raw

Re: Discourage duplicate symbols in the kernel? [Was: Intel I/O Acc...]

In reply to Andrew Morton
Quoted 9 lines · Andrew Morton
Quoted 8 lines · Sam Ravnborg
Quoted 6 lines · Andrew Morton
Quoted 5 lines
(quoted diff) +5 -0
+
+static inline u8 read_reg8(struct cb_device *device, unsigned int offset)
+{
+        return readb(device->reg_base + offset);
+}
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

These are fairly generic-sounding names. In fact the as-yet-unmerged tiacx wireless driver is already using these, privately to drivers/net/wireless/tiacx/pci.c.

Do we in general discourage duplicate symbols even if they are static?

Well, it's a bit irritating that it confuses ctags. But in this case, one set is in a header file so the risk of collisions is much-increased.

They're in a header file that's specific to a single driver, so I don't see where a conflict would occur. But I didn't think about ctags, and these can easily be prefixed so I'll go ahead and change them.

- Chris

#46 lore raw
In reply to Ingo Oeser
Quoted 13 lines · Ingo Oeser
Quoted 11 lines · Evgeniy Polyakov
Quoted 6 lines · David S. Miller

From: Jan Engelhardt <[email protected]>
Date: Sat, 4 Mar 2006 19:46:22 +0100 (MET)

Quoted 1 line

Does this buy the normal standard desktop user anything?

Absolutely, it optimizes end-node performance.

It really depends on how it is used.
According to investigation made for kevent based FS AIO reading, get_user_pages() performange graph looks like sqrt() function

Quoted 5 lines

Hmm, so I should resurrect my user page table walker abstraction?

There I would hand each page to a "recording" function, which can drop the page from the collection or coalesce it in the collector if your scatter gather implementation allows it.

It depends on where performance growth is stopped.

Quoted 1 line

From the first glance it does not look like find_extend_vma(),

probably follow_page() fault and thus __handle_mm_fault(). I can not say actually, but if it is true and performance growth is stopped due to increased number of faults and it's processing, your approach will hit this problem too, doesn't it?

Quoted 3 lines

Regards

Ingo Oeser

Signature · 1 line
	Evgeniy Polyakov
#47 lore raw
Quoted 7 lines · Evgeniy Polyakov
Quoted 5 lines · Ingo Oeser

Hmm, so I should resurrect my user page table walker abstraction?

There I would hand each page to a "recording" function, which can drop the page from the collection or coalesce it in the collector if your scatter gather implementation allows it.

Quoted 6 lines

It depends on where performance growth is stopped.
From the first glance it does not look like find_extend_vma(), probably follow_page() fault and thus __handle_mm_fault(). I can not say actually, but if it is true and performance growth is stopped due to increased number of faults and it's processing, your approach will hit this problem too, doesn't it?

My approach reduced the number of loops performed and number of memory needed at the expense of doing more work in the main loop of get_user_pages.

This was mitigated for the common case of getting just one page by providing a get_one_user_page() function.

The whole problem, why we need such multiple loops is that we have no common container object for "IO vector + additional data".

So we always do a loop working over the vector returned by get_user_pages() all the time. The bigger that vector, the bigger the impact.

Maybe sth. as simple as providing get_user_pages() with some offset_of and container_of hackery will work these days without the disadvantages my old get_user_pages() work had.

The idea is, that you'll provide a vector (like arguments to calloc) and two
offsets: One for the page to store within the offset and one for the vma to store.

If the offset has a special value (e.g MAX_LONG) you don't store there at all.

But if the performance problem really is get_user_pages() itself (and not its callers), then my approach won't help at all.

Regards

Ingo Oeser

#48 lore raw
Quoted 33 lines · Ingo Oeser
Quoted 7 lines · Evgeniy Polyakov
Quoted 5 lines · Ingo Oeser

Hmm, so I should resurrect my user page table walker abstraction?

There I would hand each page to a "recording" function, which can drop the page from the collection or coalesce it in the collector if your scatter gather implementation allows it.

Quoted 6 lines

It depends on where performance growth is stopped.
From the first glance it does not look like find_extend_vma(), probably follow_page() fault and thus __handle_mm_fault(). I can not say actually, but if it is true and performance growth is stopped due to increased number of faults and it's processing, your approach will hit this problem too, doesn't it?

My approach reduced the number of loops performed and number of memory needed at the expense of doing more work in the main loop of get_user_pages.

This was mitigated for the common case of getting just one page by providing a get_one_user_page() function.

The whole problem, why we need such multiple loops is that we have no common container object for "IO vector + additional data".

So we always do a loop working over the vector returned by get_user_pages() all the time. The bigger that vector, the bigger the impact.

Maybe sth. as simple as providing get_user_pages() with some offset_of and container_of hackery will work these days without the disadvantages my old get_user_pages() work had.

Quoted 5 lines

The idea is, that you'll provide a vector (like arguments to calloc) and two
offsets: One for the page to store within the offset and one for the vma to store.

If the offset has a special value (e.g MAX_LONG) you don't store there at all.

You still need to find VMA in one loop, and run through it's(mm_structu) pages in second loop.

Quoted 2 lines

But if the performance problem really is get_user_pages() itself (and not its callers), then my approach won't help at all.

It looks so.
My test pseudocode is following:
fget_light();
igrab();
kzalloc(number_of_pages * sizeof(void *));
get_user_pages(number_of_pages);
... undo ...

I've attached two graphs of performance with and without get_user_pages(), it is get_user_pages.png and kmalloc.png.

Vertical axis is number of Mbytes per second thrown through above code, horizontal one is number of pages in each run.

Quoted 3 lines

Regards

Ingo Oeser

Signature · 1 line
	Evgeniy Polyakov