Intel I/O Acceleration Technology (I/OAT)
Subsystems (6)
cover Intel I/O Acceleration Technology (I/OAT)
This patch series is the a full release of the Intel(R) I/O Acceleration Technology (I/OAT) for Linux. It includes an in kernel API for offloading memory copies to hardware, a driver for the I/OAT DMA memcpy engine, and changes to the TCP stack to offload copies of received networking data to application space.
Changes from last weeks posting:
fixed return value from sysfs show functions as suggested by Joe Perches code style fixes suggested by Andrew Morton, David Miller, and others renamed anything related to pinning pages from lock/locked to pin/pinned renamed ioatdma register read/write functions with less generic names return a pinned list from dma_pin_iovec_pages instead of passing in a
**dma_pinned_list
replaced all cb/CB symbol prefixes in ioatdma with ioat/IOAT,
CB was an abbreviation of an early code name
use set_page_dirty_lock instead of SetPageDirty pointed out by Andrew Morton rename dma_async_try_early_copy to tcp_dma_try_early_copy and stop exporting
I'll be focusing on reducing ifdefs and adding much needed comments, with another release early next week.
These changes apply to DaveM's net-2.6.17 tree as of commit 32639ad6b7e3da27f233c0516471f0747f1178f5 ([SPARC]: Fixup SO_*SEC values on 32-bit sparc.)
They are available to pull from
git://198.78.49.142/~cleech/linux-2.6 ioat-2.6.17
There are 8 patches in the series:
1) The memcpy offload APIs and class code
2) The Intel I/OAT DMA driver (ioatdma)
3) Core networking code to setup networking as a DMA memcpy client
4) Utility functions for sk_buff to iovec offloaded copy
5) Structure changes needed for TCP receive offload
6) Rename cleanup_rbuf to tcp_cleanup_rbuf
7) Add a sysctl to tune the minimum offloaded I/O size for TCP
8) The main TCP receive offload changes
--
Chris Leech <[email protected]>
I/O Acceleration Technology Software Development
LAN Access Division / Digital Enterprise Group
PATCH 1/8 DMA memcpy subsystem
Provides an API for offloading memory copies to DMA devices
---
drivers/Kconfig | 2 drivers/Makefile | 1 drivers/dma/Kconfig | 13 ++ drivers/dma/Makefile | 1 drivers/dma/dmaengine.c | 360 +++++++++++++++++++++++++++++++++++++++++++++ include/linux/dmaengine.h | 323 ++++++++++++++++++++++++++++++++++++++++ 6 files changed, 700 insertions(+), 0 deletions(-)
drivers/Kconfig +2 -0
@@ -70,4 +70,6 @@ source "drivers/sn/Kconfig" source "drivers/edac/Kconfig" +source "drivers/dma/Kconfig" + endmenu
drivers/Makefile +1 -0
@@ -73,3 +73,4 @@ obj-$(CONFIG_SGI_SN) += sn/ obj-y += firmware/ obj-$(CONFIG_CRYPTO) += crypto/ obj-$(CONFIG_SUPERH) += sh/ +obj-$(CONFIG_DMA_ENGINE) += dma/
drivers/dma/Kconfig new file +13 -0
@@ -0,0 +1,13 @@ +# +# DMA engine configuration +# + +menu "DMA Engine support" + +config DMA_ENGINE + bool "Support for DMA engines" + ---help--- + DMA engines offload copy operations from the CPU to dedicated + hardware, allowing the copies to happen asynchronously. + +endmenu
drivers/dma/Makefile new file +1 -0
@@ -0,0 +1 @@ +obj-y += dmaengine.o
drivers/dma/dmaengine.c new file +360 -0
360 lines not on this page · raw message
include/linux/dmaengine.h new file +323 -0
323 lines not on this page · raw message
PATCH 4/8 Utility functions for offloading sk_buff to iovec copies
In reply to Chris LeechProvides for pinning user space pages in memory, copying to iovecs, and copying from sk_buffs including fragmented and chained sk_buffs.
---
drivers/dma/Makefile | 3 drivers/dma/iovlock.c | 301 +++++++++++++++++++++++++++++++++++++++++++++ include/linux/dmaengine.h | 22 +++ include/net/netdma.h | 6 + net/core/Makefile | 3 net/core/user_dma.c | 141 +++++++++++++++++++++ 6 files changed, 474 insertions(+), 2 deletions(-)
drivers/dma/Makefile +2 -1
@@ -1,2 +1,3 @@ -obj-y += dmaengine.o +obj-$(CONFIG_DMA_ENGINE) += dmaengine.o +obj-$(CONFIG_NET_DMA) += iovlock.o obj-$(CONFIG_INTEL_IOATDMA) += ioatdma.o
drivers/dma/iovlock.c new file +301 -0
301 lines not on this page · raw message
include/linux/dmaengine.h +22 -0
@@ -319,5 +319,27 @@ static inline enum dma_status dma_async_int dma_async_device_register (struct dma_device *device);void dma_async_device_unregister (struct dma_device *device); +/* --- Helper iov-locking functions --- */ + +struct dma_page_list { +char *base_address; +int nr_pages; +struct page **pages; +}; + +struct dma_pinned_list { +int nr_iovecs; +struct dma_page_list page_list[0 ]; +}; + +struct dma_pinned_list *dma_pin_iovec_pages (struct iovec *iov,size_t len); +void dma_unpin_iovec_pages (struct dma_pinned_list * pinned_list); + +dma_cookie_t dma_memcpy_to_iovec (struct dma_chan *chan,struct iovec *iov, +struct dma_pinned_list *pinned_list,unsigned char *kdata,size_t len); +dma_cookie_t dma_memcpy_pg_to_iovec (struct dma_chan *chan,struct iovec *iov, +struct dma_pinned_list *pinned_list,struct page *page, +unsigned int offset,size_t len); +#endif /* CONFIG_DMA_ENGINE */ #endif /* DMAENGINE_H */
include/net/netdma.h +6 -0
@@ -22,6 +22,7 @@ file called LICENSE.#define NETDMA_H #ifdef CONFIG_NET_DMA #include <linux/dmaengine.h> +#include <linux/skbuff.h> static inline struct dma_chan *get_softnet_dma (void ) {
@@ -33,5 +34,10 @@ static inline struct dma_chan *get_softnrcu_read_unlock ();return chan; } + +int dma_skb_copy_datagram_iovec (struct dma_chan * chan, +const struct sk_buff *skb,int offset,struct iovec *to, +size_t len,struct dma_pinned_list *pinned_list); +#endif /* CONFIG_NET_DMA */ #endif /* NETDMA_H */
net/core/Makefile +2 -1
@@ -8,7 +8,8 @@ obj-y := sock.o request_sock.o skbuff.o obj-$(CONFIG_SYSCTL) += sysctl_net_core.o obj-y += dev.o ethtool.o dev_mcast.o dst.o \ - neighbour.o rtnetlink.o utils.o link_watch.o filter.o + neighbour.o rtnetlink.o utils.o link_watch.o filter.o \ + user_dma.o obj-$(CONFIG_XFRM) += flow.o obj-$(CONFIG_SYSFS) += net-sysfs.o
net/core/user_dma.c new file +141 -0
141 lines not on this page · raw message
PATCH 3/8 Setup the networking subsystem as a DMA client
In reply to Chris LeechAttempts to allocate per-CPU DMA channels
---
drivers/dma/Kconfig | 12 +++++ include/linux/netdevice.h | 6 +++ include/net/netdma.h | 37 +++++++++++++++++ net/core/dev.c | 100 +++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 155 insertions(+), 0 deletions(-)
drivers/dma/Kconfig +12 -0
@@ -10,6 +10,18 @@ config DMA_ENGINE DMA engines offload copy operations from the CPU to dedicated hardware, allowing the copies to happen asynchronously. +comment "DMA Clients" + +config NET_DMA + bool "Network: TCP receive copy offload" + depends on DMA_ENGINE && NET + default y + ---help--- + This enables the use of DMA engines in the network stack to + offload receive copy-to-user operations, freeing CPU cycles. + Since this is the main user of the DMA engine, it should be enabled; + say Y here. + comment "DMA Devices" config INTEL_IOATDMA
include/linux/netdevice.h +6 -0
@@ -37,6 +37,9 @@#include <linux/config.h> #include <linux/device.h> #include <linux/percpu.h> +#ifdef CONFIG_NET_DMA +#include <linux/dmaengine.h> +#endif struct divert_blk ;struct vlan_group ;
@@ -592,6 +595,9 @@ struct softnet_datastruct sk_buff *completion_queue;struct net_device backlog_dev;/* Sorry. 8) */ +#ifdef CONFIG_NET_DMA +struct dma_chan *net_dma; +#endif }; DECLARE_PER_CPU(struct softnet_data,softnet_data);
include/net/netdma.h new file +37 -0
@@ -0,0 +1,37 @@ +/***************************************************************************** +Copyright(c) 2004 - 2006 Intel Corporation. All rights reserved. + +This program is free software; you can redistribute it and/or modify it +under the terms of the GNU General Public License as published by the Free +Software Foundation; either version 2 of the License, or (at your option) +any later version. + +This program is distributed in the hope that it will be useful, but WITHOUT +ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +more details. + +You should have received a copy of the GNU General Public License along with +this program; if not, write to the Free Software Foundation, Inc., 59 +Temple Place - Suite 330, Boston, MA 02111-1307, USA. + +The full GNU General Public License is included in this distribution in the +file called LICENSE. +*****************************************************************************/ +#ifndef NETDMA_H +#define NETDMA_H +#ifdef CONFIG_NET_DMA +#include <linux/dmaengine.h> + +static inline struct dma_chan *get_softnet_dma (void ) +{ +struct dma_chan *chan; +rcu_read_lock (); + chan =rcu_dereference (__get_cpu_var (softnet_data.net_dma)); +if (chan) +dma_chan_get (chan); +rcu_read_unlock (); +return chan; +} +#endif /* CONFIG_NET_DMA */ +#endif /* NETDMA_H */
net/core/dev.c +100 -0
130 lines not on this page · raw message
PATCH 6/8 Rename cleanup_rbuf to tcp_cleanup_rbuf and make non-static
In reply to Chris LeechNeeded to be able to call tcp_cleanup_rbuf in tcp_input.c for I/OAT
---
include/net/tcp.h | 2 ++ net/ipv4/tcp.c | 10 +++++----- 2 files changed, 7 insertions(+), 5 deletions(-)
include/net/tcp.h +2 -0
@@ -296,6 +296,8 @@ extern int tcp_rcv_established(structextern void tcp_rcv_space_adjust (struct sock *sk); +extern void tcp_cleanup_rbuf (struct sock *sk,int copied); +extern int tcp_twsk_unique (struct sock *sk,struct sock *sktw,void *twp);
net/ipv4/tcp.c +5 -5
@@ -936,7 +936,7 @@ static int tcp_recv_urg(struct sock *sk,* calculation of whether or not we must ACK for the sake of * a window update. */ -static void cleanup_rbuf (struct sock *sk,int copied) +void tcp_cleanup_rbuf (struct sock *sk,int copied) {struct tcp_sock *tp =tcp_sk (sk);int time_to_ack =0 ;
@@ -1085,7 +1085,7 @@ int tcp_read_sock(struct sock *sk, read_/* Clean up data we have read: This will do ACK frames. */ if (copied) -cleanup_rbuf (sk, copied); +tcp_cleanup_rbuf (sk, copied);return copied; }
@@ -1219,7 +1219,7 @@ int tcp_recvmsg(struct kiocb *iocb, stru } } -cleanup_rbuf (sk, copied); +tcp_cleanup_rbuf (sk, copied);if (!sysctl_tcp_low_latency && tp->ucopy.task == user_recv) {/* Install new reader */
@@ -1390,7 +1390,7 @@ skip_copy:*/ /* Clean up data we have read: This will do ACK frames. */ -cleanup_rbuf (sk, copied); +tcp_cleanup_rbuf (sk, copied);TCP_CHECK_TIMER (sk);release_sock (sk);
@@ -1852,7 +1852,7 @@ static int do_tcp_setsockopt(struct sock (TCPF_ESTABLISHED | TCPF_CLOSE_WAIT) && inet_csk_ack_scheduled(sk)) { icsk->icsk_ack.pending |=ICSK_ACK_PUSHED ; -cleanup_rbuf (sk,1 ); +tcp_cleanup_rbuf (sk,1 );if (!(val &1 )) icsk->icsk_ack.pingpong =1 ; }
PATCH 7/8 Add a sysctl for tuning the I/OAT offloaded I/O threshold
In reply to Chris LeechAny socket recv of less than this ammount will not be offloaded
---
include/linux/sysctl.h | 1 + include/net/tcp.h | 1 + net/core/user_dma.c | 4 ++++ net/ipv4/sysctl_net_ipv4.c | 10 ++++++++++ 4 files changed, 16 insertions(+), 0 deletions(-)
include/linux/sysctl.h +1 -0
@@ -403,6 +403,7 @@ enumNET_TCP_MTU_PROBING =113 ,NET_TCP_BASE_MSS =114 ,NET_IPV4_TCP_WORKAROUND_SIGNED_WINDOWS =115 , + NET_TCP_DMA_COPYBREAK=116, }; enum {
include/net/tcp.h +1 -0
@@ -221,6 +221,7 @@ extern int sysctl_tcp_adv_win_scale;extern int sysctl_tcp_tw_reuse;extern int sysctl_tcp_frto;extern int sysctl_tcp_low_latency; +extern int sysctl_tcp_dma_copybreak;extern int sysctl_tcp_nometrics_save;extern int sysctl_tcp_moderate_rcvbuf;extern int sysctl_tcp_tso_win_divisor;
net/core/user_dma.c +4 -0
@@ -33,6 +33,10 @@ file called LICENSE.#ifdef CONFIG_NET_DMA +#define NET_DMA_DEFAULT_COPYBREAK 1024 + +int sysctl_tcp_dma_copybreak =NET_DMA_DEFAULT_COPYBREAK ; +/** * dma_skb_copy_datagram_iovec - Copy a datagram to an iovec. * @skb - buffer to copy
net/ipv4/sysctl_net_ipv4.c +10 -0
@@ -688,6 +688,16 @@ ctl_table ipv4_table[] = { .mode = 0644, .proc_handler = &proc_dointvec }, +#ifdef CONFIG_NET_DMA + { + .ctl_name = NET_TCP_DMA_COPYBREAK, + .procname = "tcp_dma_copybreak", + .data = &sysctl_tcp_dma_copybreak, + .maxlen = sizeof(int), + .mode = 0644, + .proc_handler = &proc_dointvec + }, +#endif { .ctl_name = 0 } };
PATCH 5/8 Structure changes for TCP recv offload to I/OAT
In reply to Chris LeechAdds an async_wait_queue and some additional fields to tcp_sock, and a dma_cookie_t to sk_buff.
---
include/linux/skbuff.h | 6 ++++++ include/linux/tcp.h | 10 ++++++++++ include/net/sock.h | 2 ++ include/net/tcp.h | 9 +++++++++ net/core/sock.c | 6 ++++++ 5 files changed, 33 insertions(+), 0 deletions(-)
include/linux/skbuff.h +6 -0
@@ -29,6 +29,9 @@#include <linux/net.h> #include <linux/textsearch.h> #include <net/checksum.h> +#ifdef CONFIG_NET_DMA +#include <linux/dmaengine.h> +#endif #define HAVE_ALLOC_SKB /* For the drivers to know */ #define HAVE_ALIGNABLE_SKB /* Ditto 8) */
@@ -285,6 +288,9 @@ struct sk_buff {__u16 tc_verd;/* traffic control verdict */ #endif #endif +#ifdef CONFIG_NET_DMA +dma_cookie_t dma_cookie; +#endif /* These elements must be at the end, see alloc_skb() for details. */
include/linux/tcp.h +10 -0
@@ -18,6 +18,9 @@#define _LINUX_TCP_H#include <linux/types.h> +#ifdef CONFIG_NET_DMA +#include <linux/dmaengine.h> +#endif #include <asm/byteorder.h> struct tcphdr {
@@ -233,6 +236,13 @@ struct tcp_sock {struct iovec *iov;int memory;int len; +#ifdef CONFIG_NET_DMA +/* members for async copy */ +struct dma_chan *dma_chan; +int wakeup; +struct dma_pinned_list *pinned_list; +dma_cookie_t dma_cookie; +#endif } ucopy;__u32 snd_wl1;/* Sequence for window update */
include/net/sock.h +2 -0
@@ -132,6 +132,7 @@ struct sock_common {* @sk_receive_queue: incoming packets * @sk_wmem_alloc: transmit queue bytes committed * @sk_write_queue: Packet sending queue +* @sk_async_wait_queue: DMA copied packets * @sk_omem_alloc: "o" is "option" or "other" * @sk_wmem_queued: persistent queue size * @sk_forward_alloc: space allocated forward
@@ -205,6 +206,7 @@ struct sock {atomic_t sk_omem_alloc;struct sk_buff_head sk_receive_queue;struct sk_buff_head sk_write_queue; +struct sk_buff_head sk_async_wait_queue;int sk_wmem_queued;int sk_forward_alloc;gfp_t sk_allocation;
include/net/tcp.h +9 -0
@@ -28,6 +28,9 @@#include <linux/cache.h> #include <linux/percpu.h> #include <linux/skbuff.h> +#ifdef CONFIG_NET_DMA +#include <linux/dmaengine.h> +#endif #include <net/inet_connection_sock.h> #include <net/inet_timewait_sock.h>
@@ -820,6 +823,12 @@ static inline void tcp_prequeue_init(str tp->ucopy.len =0 ; tp->ucopy.memory =0 ;skb_queue_head_init (&tp->ucopy.prequeue); +#ifdef CONFIG_NET_DMA + tp->ucopy.dma_chan =NULL ; + tp->ucopy.wakeup =0 ; + tp->ucopy.pinned_list =NULL ; + tp->ucopy.dma_cookie =0 ; +#endif }/* Packet is added to VJ-style prequeue for processing in process
net/core/sock.c +6 -0
@@ -735,6 +735,9 @@ struct sock *sk_clone(const struct sockatomic_set (&newsk->sk_omem_alloc,0 );skb_queue_head_init (&newsk->sk_receive_queue);skb_queue_head_init (&newsk->sk_write_queue); +#ifdef CONFIG_NET_DMA +skb_queue_head_init (&newsk->sk_async_wait_queue); +#endif rwlock_init (&newsk->sk_dst_lock);rwlock_init (&newsk->sk_callback_lock);
@@ -1286,6 +1289,9 @@ void sock_init_data(struct socket *sock,skb_queue_head_init (&sk->sk_receive_queue);skb_queue_head_init (&sk->sk_write_queue);skb_queue_head_init (&sk->sk_error_queue); +#ifdef CONFIG_NET_DMA +skb_queue_head_init (&sk->sk_async_wait_queue); +#endif sk->sk_send_head =NULL ;
PATCH 8/8 TCP recv offload to I/OAT
In reply to Chris LeechLocks down user pages and sets up for DMA in tcp_recvmsg, then calls dma_async_try_early_copy in tcp_v4_do_rcv
---
include/net/netdma.h | 1 net/ipv4/tcp.c | 110 +++++++++++++++++++++++++++++++++++++++++++++----- net/ipv4/tcp_input.c | 74 ++++++++++++++++++++++++++++++---- net/ipv4/tcp_ipv4.c | 18 ++++++++ net/ipv6/tcp_ipv6.c | 12 +++++ 5 files changed, 193 insertions(+), 22 deletions(-)
include/net/netdma.h +1 -0
@@ -38,6 +38,7 @@ static inline struct dma_chan *get_softnint dma_skb_copy_datagram_iovec (struct dma_chan * chan,const struct sk_buff *skb,int offset,struct iovec *to,size_t len,struct dma_pinned_list *pinned_list); +int dma_async_try_early_copy (struct sock *sk,struct sk_buff *skb,int hlen);#endif /* CONFIG_NET_DMA */ #endif /* NETDMA_H */
net/ipv4/tcp.c +98 -12
163 lines not on this page · raw message
net/ipv4/tcp_input.c +67 -7
107 lines not on this page · raw message
net/ipv4/tcp_ipv4.c +17 -1
@@ -71,6 +71,7 @@#include <net/inet_common.h> #include <net/timewait_sock.h> #include <net/xfrm.h> +#include <net/netdma.h> #include <linux/inet.h> #include <linux/ipv6.h>
@@ -1091,8 +1092,18 @@ process:bh_lock_sock (sk); ret =0 ;if (!sock_owned_by_user (sk)) { -if (!tcp_prequeue (sk, skb)) +#ifdef CONFIG_NET_DMA +struct tcp_sock *tp =tcp_sk (sk); +if (!tp->ucopy.dma_chan && tp->ucopy.pinned_list) + tp->ucopy.dma_chan =get_softnet_dma (); +if (tp->ucopy.dma_chan) ret =tcp_v4_do_rcv (sk, skb); + else +#endif + { +if (!tcp_prequeue (sk, skb)) + ret =tcp_v4_do_rcv (sk, skb); + } }else sk_add_backlog (sk, skb);bh_unlock_sock (sk);
@@ -1296,6 +1307,11 @@ int tcp_v4_destroy_sock(struct sock *sk)/* Cleans up our, hopefully empty, out_of_order_queue. */ __skb_queue_purge (&tp->out_of_order_queue); +#ifdef CONFIG_NET_DMA +/* Cleans up our sk_async_wait_queue */ +__skb_queue_purge (&sk->sk_async_wait_queue); +#endif +/* Clean prequeue, it must be empty really */ __skb_queue_purge (&tp->ucopy.prequeue);
net/ipv6/tcp_ipv6.c +10 -2
@@ -1218,8 +1218,16 @@ process:bh_lock_sock (sk); ret =0 ;if (!sock_owned_by_user (sk)) { -if (!tcp_prequeue (sk, skb)) - ret =tcp_v6_do_rcv (sk, skb); +#ifdef CONFIG_NET_DMA +struct tcp_sock *tp =tcp_sk (sk); +if (tp->ucopy.dma_chan) + ret =tcp_v6_do_rcv (sk, skb); + else +#endif + { +if (!tcp_prequeue (sk, skb)) + ret =tcp_v6_do_rcv (sk, skb); + } }else sk_add_backlog (sk, skb);bh_unlock_sock (sk);
Re: [PATCH 1/8] [I/OAT] DMA memcpy subsystem
In reply to Chris LeechQuoted 3 lines · Chris Leech
+void
dma_async_device_cleanup(struct kref *kref);
Declarations go in header files, please. Or give it static scope.
Re: [PATCH 3/8] [I/OAT] Setup the networking subsystem as a DMA client
In reply to Chris LeechQuoted 1 line · Chris Leech
There seems to be a small race here.
Quoted 5 lines
+static void
net_dma_rebalance(void)
+{
+ unsigned int cpu, i, n;
+ structdma_chan*chan;
+
Quoted 8 lines
+
lock_cpu_hotplug();
+
+ if (net_dma_count== 0) {
+for_each_online_cpu(cpu)
+rcu_assign_pointer(per_cpu(softnet_data.net_dma, cpu), NULL);
+unlock_cpu_hotplug();
+ return;
+ }
If some other CPU does netdev_dma_event(DMA_RESOURCE_REMOVED) now
Quoted 7 lines
+ i = 0;
+ cpu =first_cpu(cpu_online_map);
+
+rcu_read_lock();
+list_for_each_entry(chan, &net_dma_client->channels, client_node){
+ n = ((num_online_cpus()/net_dma_count)
+ + (i < (num_online_cpus()%net_dma_count) ? 1 : 0));
This will get a divide-by-zero.
Quoted 34 lines
+ while(n) {
+per_cpu(softnet_data.net_dma, cpu)= chan;
+ cpu =next_cpu(cpu, cpu_online_map);
+ n--;
+ }
+ i++;
+ }
+rcu_read_unlock();
+
+unlock_cpu_hotplug();
+}
+
+/**
+ *netdev_dma_event- event callback for thenet_dma_client
+ * @client: should always benet_dma_client
+ * @chan:
+ * @event:
+ */
+static void netdev_dma_event(structdma_client*client, structdma_chan*chan,
+ enumdma_eventevent)
+{
+ switch (event) {
+ caseDMA_RESOURCE_ADDED:
+ net_dma_count++;
+net_dma_rebalance();
+ break;
+ caseDMA_RESOURCE_REMOVED:
+ net_dma_count--;
+net_dma_rebalance();
+ break;
+ default:
+ break;
+ }
+}
Re: [PATCH 3/8] [I/OAT] Setup the networking subsystem as a DMA client
In reply to Chris LeechQuoted 4 lines · Chris Leech
+#ifdef
CONFIG_NET_DMA
+#include <linux/dmaengine.h>
+#endif
There are still a number of instances of this in the patch series. Did you decide to keep the ifdefs in there for some reason?
Re: [PATCH 8/8] [I/OAT] TCP recv offload to I/OAT
In reply to Chris LeechQuoted 3 lines · Chris Leech
Locks down user pages and sets up for DMA in
tcp_recvmsg, then calls
dma_async_try_early_copyintcp_v4_do_rcv
All those ifdefs are still there. They really do put a maintenance burden on, of all places, net/ipv4/tcp.c. Please find a way of abstracting out the conceptual additions which IOAT makes to TCP and to wrap that into inline functions, etc. See how tidily NF_HOOK() has inserted things into the net stack.
Also, for something which is billed as an performance enhancement, benchmark numbers are the key thing which we need to judge the desirability of these changes. What is the plan there?
Thanks.
Re: [PATCH 1/8] [I/OAT] DMA memcpy subsystem
In reply to Chris LeechHi!
Quoted 13 lines
drivers/dma/dmaengine.c new file +10
-0@@ -0,0 +1,360 @@ +/***************************************************************************** +Copyright(c) 2004 - 2006 Intel Corporation. All rights reserved. + +This program is free software; you can redistribute it and/or modify it +under the terms of the GNU General Public License as published by the Free +Software Foundation; either version 2 of the License, or (at your option) +any later version. + +This program is distributed in the hope that it will be useful, but WITHOUT +ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
drivers/dma/dmaengine.c line 20
Quoted 10 lines
drivers/dma/dmaengine.c new file +10
-0+FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +more details. + +You should have received a copy of the GNU General Public License along with +this program; if not, write to the Free Software Foundation, Inc., 59 +Temple Place - Suite 330, Boston, MA 02111-1307, USA. + +The full GNU General Public License is included in this distribution in the +file called LICENSE. +*****************************************************************************/
Could you use
/*
*
*/
comment style, and describe in one or two lines what the source does in the header?
Pavel
Signature · 1 line
209:using System.IO;
Re: [PATCH 1/8] [I/OAT] DMA memcpy subsystem
In reply to Chris Leech[snip]
Quoted 17 lines
+/**
+ *dma_async_client_register- allocate and register a&dma_client
+ * @event_callback: callback for notification of channel addition/ removal
+ */
+structdma_client*dma_async_client_register(dma_event_callbackevent_callback)drivers/dma/dmaengine.c new file +10
-0+{ +struct dma_client *client; + + client =kzalloc (sizeof (*client),GFP_KERNEL ); +if (!client) +return NULL ; + +INIT_LIST_HEAD (&client->channels); +spin_lock_init (&client->lock); +
drivers/dma/dmaengine.c line 217
Quoted 10 lines
drivers/dma/dmaengine.c new file +10
-0+ client->chans_desired =0 ; + client->chan_count =0 ; + client->event_callback = event_callback; + +spin_lock (&dma_list_lock); +list_add_tail (&client->global_node, &dma_client_list); +spin_unlock (&dma_list_lock); + +return client; +}
It would seem that when a client registers (or shortly there after when they call dma_async_client_chan_request()) they would expect to get the number of channels they need by some given time period.
For example, lets say a client registers but no dma device exists. They will never get called to be aware of this condition.
I would think most clients would either spin until they have all the channels they need or fall back to a non-async mechanism.
Also, what do you think about adding an operation type (MEMCPY, XOR, CRYPTO_AES, etc). We can than validate if the operation type expected is supported by the devices that exist.
Quoted 41 lines
drivers/dma/dmaengine.c new file +3
-0+ +/** +* dma_async_client_unregister - unregister a client and free the
&dma_client
+ * @client:
+ *
+ * Force frees any allocated DMA channels, frees the&dma_clientmemory
+ */drivers/dma/dmaengine.c new file +29
-0+void dma_async_client_unregister (struct dma_client *client) +{ +struct dma_chan *chan; + +if (!client) +return ; + +rcu_read_lock (); +list_for_each_entry_rcu (chan, &client->channels, client_node) { +dma_client_chan_free (chan); + } +rcu_read_unlock (); + +spin_lock (&dma_list_lock); +list_del (&client->global_node); +spin_unlock (&dma_list_lock); + +kfree (client); +dma_chans_rebalance (); +} + +/** +* dma_async_client_chan_request - request DMA channels +* @client: &dma_client +* @number: count of DMA channels requested +* +* Clients call dma_async_client_chan_request() to specify how many +* DMA channels they need, 0 to free all currently allocated. +* The resulting allocations/frees are indicated to the client via the
+ * event callback.
+ */
drivers/dma/dmaengine.c line 262
Quoted 7 lines
drivers/dma/dmaengine.c new file +7
-0+void dma_async_client_chan_request (struct dma_client *client, +unsigned int number) +{ + client->chans_desired = number; +dma_chans_rebalance (); +} +
Shouldn't we also have a dma_async_client_chan_free()?
[snip]
Quoted 25 lines
include/linux/dmaengine.h new file +3
-0+/* --- public DMA engine API --- */ + +struct dma_client *dma_async_client_register(dma_event_callback
event_callback);include/linux/dmaengine.h new file +6
-0+void dma_async_client_unregister (struct dma_client *client); +void dma_async_client_chan_request (struct dma_client *client, +unsigned int number); + +/** +* dma_async_memcpy_buf_to_buf - offloaded copy between virtual addresses
+ * @chan: DMA channel to offload copy to
+ * @dest: destination address (virtual)
+ * @src: source address (virtual)
+ * @len: length
+ *
+ * Both @dest and @src must be mappable to a bus address according to the
+ * DMA mapping API rules for streaming mappings.
+ * Both @dest and @src must stay memory resident (kernel memory or locked
+ * user space pages)
+ */
+static inlinedma_cookie_tdma_async_memcpy_buf_to_buf(structdma_chan*chan,
include/linux/dmaengine.h line 211
Quoted 9 lines
include/linux/dmaengine.h new file +9
-0+ void *dest, void *src, size_t len) +{ + int cpu = get_cpu(); +per_cpu_ptr (chan->local, cpu)->bytes_transferred += len; +per_cpu_ptr (chan->local, cpu)->memcpy_count++; +put_cpu (); + +return chan->device->device_memcpy_buf_to_buf (chan, dest, src, len); +}
What about renaming the dma_async_memcpy_* to something like dma_async_op_* and have them take an additional operation argument.
Quoted 131 lines
include/linux/dmaengine.h new file +10
-0+ +/** +* dma_async_memcpy_buf_to_pg - offloaded copy +* @chan: DMA channel to offload copy to +* @page: destination page +* @offset: offset in page to copy to +* @kdata: source address (virtual) +* @len: length +* +* Both @page/@offset and @kdata must be mappable to a bus address according
+ * to the DMA mapping API rules for streaming mappings.
+ * Both @page/@offset and @kdata must stay memory resident (kernel memory or
+ * locked user space pages)
+ */
+static inlinedma_cookie_tdma_async_memcpy_buf_to_pg(structdma_chan*chan,include/linux/dmaengine.h new file +21
-0+ struct page *page, unsigned int offset, void *kdata, size_t len) +{ + int cpu = get_cpu(); +per_cpu_ptr (chan->local, cpu)->bytes_transferred += len; +per_cpu_ptr (chan->local, cpu)->memcpy_count++; +put_cpu (); + +return chan->device->device_memcpy_buf_to_pg (chan, page, offset, + kdata, len); +} + +/** +* dma_async_memcpy_buf_to_pg - offloaded copy +* @chan: DMA channel to offload copy to +* @dest_page: destination page +* @dest_off: offset in page to copy to +* @src_page: source page +* @src_off: offset in page to copy from +* @len: length +* +* Both @dest_page/@dest_off and @src_page/@src_off must be mappable to a bus
+ * address according to the DMA mapping API rules for streaming mappings.
+ * Both @dest_page/@dest_off and @src_page/@src_off must stay memory resident
+ * (kernel memory or locked user space pages)
+ */
+static inlinedma_cookie_tdma_async_memcpy_pg_to_pg(structdma_chan*chan,include/linux/dmaengine.h new file +20
-0+ struct page *dest_pg, unsigned int dest_off, struct page *src_pg, + unsigned int src_off, size_t len) +{ + int cpu = get_cpu(); +per_cpu_ptr (chan->local, cpu)->bytes_transferred += len; +per_cpu_ptr (chan->local, cpu)->memcpy_count++; +put_cpu (); + +return chan->device->device_memcpy_pg_to_pg (chan, dest_pg, dest_off, + src_pg, src_off, len); +} + +/** +* dma_async_memcpy_issue_pending - flush pending copies to HW +* @chan: +* +* This allows drivers to push copies to HW in batches, +* reducing MMIO writes where possible. +*/ +static inline void dma_async_memcpy_issue_pending(struct dma_chan*chan)
include/linux/dmaengine.h new file +12
-0+{ +return chan->device->device_memcpy_issue_pending (chan); +} + +/** +* dma_async_memcpy_complete - poll for transaction completion +* @chan: DMA channel +* @cookie: transaction identifier to check status of +* @last: returns last completed cookie, can be NULL +* @used: returns last issued cookie, can be NULL +* +* If @last and @used are passed in, upon return they reflect the driver
+ * internal state and can be used withdma_async_is_complete()to check
+ * the status of multiple cookies without re-checking hardware state.
+ */
+static inline enumdma_statusdma_async_memcpy_complete(structdma_chan*chan,include/linux/dmaengine.h new file +3
-0+ dma_cookie_t cookie, dma_cookie_t *last, dma_cookie_t *used) +{ + return chan->device->device_memcpy_complete(chan, cookie, last,used);
include/linux/dmaengine.h new file +10
-0+} + +/** +* dma_async_is_complete - test a cookie against chan state +* @cookie: transaction identifier to test status of +* @last_complete: last know completed transaction +* @last_used: last cookie value handed out +* +* dma_async_is_complete() is used in dma_async_memcpy_complete() +* the test logic is seperated for lightweight testing of multiple cookies
+ */
+static inline enumdma_statusdma_async_is_complete(dma_cookie_t cookie,include/linux/dmaengine.h new file +20
-1+ dma_cookie_t last_complete, dma_cookie_t last_used) +{ +if (last_complete <= last_used) { +if ((cookie <= last_complete) || (cookie > last_used)) +return DMA_SUCCESS ; + }else { +if ((cookie <= last_complete) && (cookie > last_used)) +return DMA_SUCCESS ; + } +return DMA_IN_PROGRESS ; +} + + +/* --- DMA device --- */ + +int dma_async_device_register (struct dma_device *device); +void dma_async_device_unregister (struct dma_device *device); + +#endif /* CONFIG_DMA_ENGINE */ +#endif /* DMAENGINE_H */ -To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 1/8] [I/OAT] DMA memcpy subsystem
Quoted 9 lines · Kumar Gala
It would seem that when a client registers (or shortly there after when they call
dma_async_client_chan_request()) they would expect to get the number of channels they need by some given time period.For example, lets say a client registers but no dma device exists. They will never get called to be aware of this condition.
I would think most clients would either spin until they have all the channels they need or fall back to a non-async mechanism.
Clients *are* expected to fall back to non-async if they are not given channels. The reason it was implemented with callbacks for added/removed was that the client may be initializing before the channels are enumerated. For example, the net subsystem will ask for channels and not get them for a while, until the ioatdma PCI device is found and its driver loads. In this scenario, we'd like the net subsystem to be given these channels, instead of them going unused.
Quoted 3 lines
Also, what do you think about adding an operation type (MEMCPY, XOR,
CRYPTO_AES, etc). We can than validate if the operation type expected is supported by the devices that exist.
No objections, but this speculative support doesn't need to be in our initial patchset.
Quoted 1 line
Shouldn't we also have a
dma_async_client_chan_free()?
Well we could just define it to be chan_request(0) but it doesn't seem to be needed. Also, the allocation mechanism we have for channels is different from alloc/free's semantics, so it may be best to not muddy the water in this area.
Regards -- Andy
Re: [PATCH 1/8] [I/OAT] DMA memcpy subsystem
Quoted 11 lines · Andrew Grover
Quoted 9 lines · Kumar Gala
It would seem that when a client registers (or shortly there after when they call
dma_async_client_chan_request()) they would expect to get the number of channels they need by some given time period.For example, lets say a client registers but no dma device exists. They will never get called to be aware of this condition.
I would think most clients would either spin until they have all the channels they need or fall back to a non-async mechanism.
Quoted 7 lines
Clients *are* expected to fall back to non-async if they are not given channels. The reason it was implemented with callbacks for added/removed was that the client may be initializing before the channels are enumerated. For example, the net subsystem will ask for channels and not get them for a while, until the ioatdma PCI device is found and its driver loads. In this scenario, we'd like the net subsystem to be given these channels, instead of them going unused.
Fair, I need to think on that a little more.
Quoted 6 lines
Quoted 3 lines
Also, what do you think about adding an operation type (MEMCPY, XOR,
CRYPTO_AES, etc). We can than validate if the operation type expected is supported by the devices that exist.No objections, but this speculative support doesn't need to be in our initial patchset.
I don't consider it speculative. The patch is for a generic DMA engine interface. That interface should encompass all users. I have a security/crypto DMA engine that I'd like to front with the generic DMA interface today. Also, I believe there is another Intel group with an XOR engine that had a similar concept called ADMA posted a while ago.
http://marc.theaimsgroup.com/?t=112603120100004&r=1&w=2
Quoted 6 lines
Quoted 1 line
Shouldn't we also have a
dma_async_client_chan_free()?Well we could just define it to be
chan_request(0)but it doesn't seem to be needed. Also, the allocation mechanism we have for channels is different from alloc/free's semantics, so it may be best to not muddy the water in this area.
Can you explain what the semantics are.
It's been a little while since I posted so my thoughts on the subject are going to take a little while to come back to me :)
- kumar
Re: [PATCH 1/8] [I/OAT] DMA memcpy subsystem
Quoted 7 lines · Kumar Gala
Quoted 6 lines
Quoted 3 lines
Also, what do you think about adding an operation type (MEMCPY, XOR,
CRYPTO_AES, etc). We can than validate if the operation type expected is supported by the devices that exist.No objections, but this speculative support doesn't need to be in our initial patchset.
Quoted 6 lines
I don't consider it speculative. The patch is for a generic DMA engine interface. That interface should encompass all users. I have a
security/cryptoDMA engine that I'd like to front with the generic DMA interface today. Also, I believe there is another Intel group with an XOR engine that had a similar concept called ADMA posted a while ago.
Please submit patches then. We will be doing another rev of the I/OAT patch very soon, which you will be able to patch against. Or, once the patch gets in mainline then we can enhance it. Code in the Linux kernel is never "done", and the burden of implementing additional functionality falls on those who want it.
Quoted 4 lines
Can you explain what the semantics are.
It's been a little while since I posted so my thoughts on the subject are going to take a little while to come back to me :)
Yeah. Basically you register as a DMA client, and say how many DMA channels you want. Our net_dma patch for example uses multiple channels to help lock contention. Then when channels are available (i.e. a DMA device added or another client gives them up) then you get a callback. If the channel goes away (i.e. DMA device is removed (theoretically possible but practically never happens) or *you* are going away and change your request to 0 channels) then you get a remove callback.
This gets around the problem of DMA clients registering (and therefore not getting) channels simply because they init before the DMA device is discovered.
Regards -- Andy
Re: [PATCH 1/8] [I/OAT] DMA memcpy subsystem
Quoted 16 lines · Andrew Grover
Quoted 7 lines · Kumar Gala
Quoted 6 lines
Quoted 3 lines
Also, what do you think about adding an operation type (MEMCPY, XOR,
CRYPTO_AES, etc). We can than validate if the operation type expected is supported by the devices that exist.No objections, but this speculative support doesn't need to be in our initial patchset.
Quoted 6 lines
I don't consider it speculative. The patch is for a generic DMA engine interface. That interface should encompass all users. I have a
security/cryptoDMA engine that I'd like to front with the generic DMA interface today. Also, I believe there is another Intel group with an XOR engine that had a similar concept called ADMA posted a while ago.
Quoted 5 lines
Please submit patches then. We will be doing another rev of the I/OAT patch very soon, which you will be able to patch against. Or, once the patch gets in mainline then we can enhance it. Code in the Linux kernel is never "done", and the burden of implementing additional functionality falls on those who want it.
I completely understand that. However, I think putting something into mainline that only works or solves the particular problem you have is a bad idea. I'll provide patches for the changes I'd like to see. However, I figured a little discussion on the subject before I went off an spent time on it was worth while.
Quoted 5 lines
Quoted 4 lines
Can you explain what the semantics are.
It's been a little while since I posted so my thoughts on the subject are going to take a little while to come back to me :)
Quoted 8 lines
Yeah. Basically you register as a DMA client, and say how many DMA channels you want. Our
net_dmapatch for example uses multiple channels to help lock contention. Then when channels are available (i.e. a DMA device added or another client gives them up) then you get a callback. If the channel goes away (i.e. DMA device is removed (theoretically possible but practically never happens) or *you* are going away and change your request to 0 channels) then you get a remove callback.
Do you only get callback when a channel is available? How do you decide to do to provide PIO to the client?
A client should only request multiple channel to handle multiple concurrent operations.
Quoted 3 lines
This gets around the problem of DMA clients registering (and therefore not getting) channels simply because they init before the DMA device is discovered.
What do you expect to happen in a system in which the channels are over subscribed?
Do you expect the DMA device driver to handle scheduling of channels between multiple clients?
- kumar
Re: [PATCH 1/8] [I/OAT] DMA memcpy subsystem
Quoted 1 line · Kumar Gala
Do you only get callback when a channel is available?
Yes
Quoted 2 lines
How do you
decide to do to provide PIO to the client?
The client is responsible for using any channels it gets, or falling back to memcpy() if it doesn't get any. (I don't understand how PIO comes into the picture..?)
Quoted 2 lines
A client should only request multiple channel to handle multiple concurrent operations.
Correct, if there aren't any CPU concurrency issues then 1 channel will use the device's full bandwidth (unless some other client has acquired the other channels and is using them, of course.)
Quoted 9 lines
Quoted 3 lines
This gets around the problem of DMA clients registering (and therefore not getting) channels simply because they init before the DMA device is discovered.
What do you expect to happen in a system in which the channels are over subscribed?
Do you expect the DMA device driver to handle scheduling of channels between multiple clients?
It does the simplest thing that could possibly work right now:
channels are allocated first come first serve. When there is a need, it should be straightforward to allow multiple clients to share DMA channels.
Regards -- Andy
Re: [PATCH 1/8] [I/OAT] DMA memcpy subsystem
Quoted 11 lines · Andrew Grover
Quoted 1 line · Kumar Gala
Do you only get callback when a channel is available?
Yes
Quoted 2 lines
How do you
decide to do to provide PIO to the client?The client is responsible for using any channels it gets, or falling back to
memcpy()if it doesn't get any. (I don't understand how PIO comes into the picture..?)
I was under the impression that the DMA engine would provide a "sync" cpu based memcpy (PIO) if a real HW channel wasn't avail, if this is left to the client that's fine. So how does the client know he should use normal memcpy()?
Quoted 14 lines
Quoted 2 lines
A client should only request multiple channel to handle multiple concurrent operations.
Correct, if there aren't any CPU concurrency issues then 1 channel will use the device's full bandwidth (unless some other client has acquired the other channels and is using them, of course.)
Quoted 7 lines
Quoted 3 lines
This gets around the problem of DMA clients registering (and therefore not getting) channels simply because they init before the DMA device is discovered.
What do you expect to happen in a system in which the channels are over subscribed?
Quoted 7 lines
Quoted 2 lines
Do you expect the DMA device driver to handle scheduling of channels between multiple clients?
It does the simplest thing that could possibly work right now:
channels are allocated first come first serve. When there is a need, it should be straightforward to allow multiple clients to share DMA channels.
Sounds good for a start. Have you given any thoughts on handling priorities between clients?
I need to take a look at the latest patches. How would you guys like modifications?
- k
Re: [PATCH 1/8] [I/OAT] DMA memcpy subsystem
Quoted 4 lines · Kumar Gala
I was under the impression that the DMA engine would provide a "sync" cpu based memcpy (PIO) if a real HW channel wasn't avail, if this is left to the client that's fine. So how does the client know he should use normal
memcpy()?
It has to keep track of what DMA channel to use, which it gets when the channel ADDED callback happens. So it's basically
if (some_client_struct->dma_chan)
dma_memcpy()
else
memcpy()
The async memcpy has the added requirement that at some point the client must verify the copies have been completed, so doing async memcopies does require more work on the client's part.
Quoted 5 lines
Sounds good for a start. Have you given any thoughts on handling priorities between clients?
I need to take a look at the latest patches. How would you guys like modifications?
Haven't given any thought to priorities yet -- we've been focusing on getting the 1 client case to perform well. :)
Chris posted a link to this: git://198.78.49.142/~cleech/linux-2.6
branch ioat-2.6.17
So you can post patches against that, or the patches posted here apply against davem's git tree.
Regards -- Andy