I/OAT
Subsystems (7)
cover I/OAT
This patch series is the a full release of the Intel(R) I/O Acceleration Technology (I/OAT) for Linux. It includes an in kernel API for offloading memory copies to hardware, a driver for the I/OAT DMA memcpy engine, and changes to the TCP stack to offload copies of received networking data to application space.
Changes from last posting:
Fixed a page reference leak that happened when offloaded copies were set up but never used for a recv.
Fixed the ioatdma self test to handle failures correctly. Serialized DMA ADD and REMOVE events in the networking core with a lock. Added a long comment in dmaengine.c to describe the locking and reference counting being used.
Disabled preempt around a use of get_cpu_var.
Made tcp_dma_try_early_copy static, it is only used in one file. Made some GFP_ATOMIC allocations GFP_KERNEL where safe to sleep. Made changes to sk_eat_skb, removing some ifdefs in the TCP code.
These changes apply to DaveM's net-2.6.17 tree as of commit 68907dad58cd7ef11536e1db6baeb98b20af91b2 ([DCCP]: Use NULL for pointers, comfort sparse.)
They are available to pull from
git://198.78.49.142/~cleech/linux-2.6 ioat-2.6.17
There are 9 patches in the series:
1) The memcpy offload APIs and class code
2) The Intel I/OAT DMA driver (ioatdma)
3) Core networking code to setup networking as a DMA memcpy client
4) Utility functions for sk_buff to iovec offloaded copy
5) Structure changes needed for TCP receive offload
6) Rename cleanup_rbuf to tcp_cleanup_rbuf
7) Make sk_eat_skb aware of early copied packets
8) Add a sysctl to tune the minimum offloaded I/O size for TCP
9) The main TCP receive offload changes
--
Chris Leech <[email protected]>
I/O Acceleration Technology Software Development
LAN Access Division / Digital Enterprise Group
PATCH 1/9 DMA memcpy subsystem
Provides an API for offloading memory copies to DMA devices
---
drivers/Kconfig | 2 drivers/Makefile | 1 drivers/dma/Kconfig | 13 + drivers/dma/Makefile | 1 drivers/dma/dmaengine.c | 405 +++++++++++++++++++++++++++++++++++++++++++++ include/linux/dmaengine.h | 337 +++++++++++++++++++++++++++++++++++++ 6 files changed, 759 insertions(+), 0 deletions(-)
drivers/Kconfig +2 -0
@@ -72,4 +72,6 @@ source "drivers/edac/Kconfig" source "drivers/rtc/Kconfig" +source "drivers/dma/Kconfig" + endmenu
drivers/Makefile +1 -0
@@ -74,3 +74,4 @@ obj-$(CONFIG_SGI_SN) += sn/ obj-y += firmware/ obj-$(CONFIG_CRYPTO) += crypto/ obj-$(CONFIG_SUPERH) += sh/ +obj-$(CONFIG_DMA_ENGINE) += dma/
drivers/dma/Kconfig new file +13 -0
@@ -0,0 +1,13 @@ +# +# DMA engine configuration +# + +menu "DMA Engine support" + +config DMA_ENGINE + bool "Support for DMA engines" + ---help--- + DMA engines offload copy operations from the CPU to dedicated + hardware, allowing the copies to happen asynchronously. + +endmenu
drivers/dma/Makefile new file +1 -0
@@ -0,0 +1 @@ +obj-y += dmaengine.o
drivers/dma/dmaengine.c new file +405 -0
405 lines not on this page · raw message
include/linux/dmaengine.h new file +337 -0
337 lines not on this page · raw message
Re: [PATCH 1/9] [I/OAT] DMA memcpy subsystem
Quoted 188 lines · Chris Leech
Provides an API for offloading memory copies to DMA devices
Signed-off-by Chris Leech <[email protected]>---
drivers/Kconfig | 2 drivers/Makefile | 1 drivers/dma/Kconfig | 13 + drivers/dma/Makefile | 1 drivers/dma/dmaengine.c | 405 ++++++++++++++++++++++++++++++++++ +++++++++++ include/linux/dmaengine.h | 337 ++++++++++++++++++++++++++++++++++ +++ 6 files changed, 759 insertions(+), 0 deletions(-)drivers/Kconfig +2
-0@@ -72,4 +72,6 @@ source "drivers/edac/Kconfig" source "drivers/rtc/Kconfig" +source "drivers/dma/Kconfig" + endmenudrivers/Makefile +1
-0@@ -74,3 +74,4 @@ obj-$(CONFIG_SGI_SN) += sn/ obj-y += firmware/ obj-$(CONFIG_CRYPTO) += crypto/ obj-$(CONFIG_SUPERH) += sh/ +obj-$(CONFIG_DMA_ENGINE) += dma/drivers/dma/Kconfig new file +13
-0@@ -0,0 +1,13 @@ +# +# DMA engine configuration +# + +menu "DMA Engine support" + +config DMA_ENGINE + bool "Support for DMA engines" + ---help--- + DMA engines offload copy operations from the CPU to dedicated + hardware, allowing the copies to happen asynchronously. + +endmenudrivers/dma/Makefile new file +1
-0@@ -0,0 +1 @@ +obj-y += dmaengine.odrivers/dma/dmaengine.c new file +4
-0@@ -0,0 +1,405 @@ +/* +* Copyright(c) 2004 - 2006 Intel Corporation. All rights reserved. +* +* This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the Free
+ * Software Foundation; either version 2 of the License, or (at your option)
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; if not, write to the Free Software Foundation, Inc., 59
+ * Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ *
+ * The full GNU General Public License is included in this distribution in the
+ * file called COPYING.
+ */drivers/dma/dmaengine.c new file +3
-0+ +/* +* This code implements the DMA subsystem. It provides a HW- neutral interface
+ * for other kernel code to use asynchronous memory copy capabilities,
+ * if present, and allows different HW DMA drivers to register as providing
+ * this capability.
+ *
+ * Due to the fact we are accelerating what is already a relatively fast
+ * operation, the code goes to great lengths to avoid additional overhead,
+ * such as locking.
+ *
+ * LOCKING:
+ *
+ * The subsystem keeps two global lists,dma_device_listanddma_client_list.
+ * Both of these are protected by a spinlock,dma_list_lock.
+ *
+ * Each device has a channels list, which runs unlocked but is never modified
+ * once the device is registered, it's just setup by the driver.
+ *
+ * Each client has a channels list, it's only modified under theclient->lock
+ * and in an RCU callback, so it's safe to read underrcu_read_lock().
+ *
+ * Each device has a kref, which is initialized to 1 when the device is
+ * registered. Akref_putis done for eachclass_deviceregistered. When the
+ *class_deviceis released, the corespondingkref_putis done in the release
+ * method. Every time one of the device's channels is allocated to a client,
+ * akref_getoccurs. When the channel is freed, the corespondingkref_put
+ * happens. The device's release function does a completion, so
+ *unregister_devicedoes a remove event,class_device_unregister, akref_put
+ * for the first reference, then waits on the completion for all other
+ * references to finish.
+ *
+ * Each channel has an open-coded implementation of Rusty Russell's "bigref,"
+ * with a kref and aper_cpulocal_t. A single reference is set when on an
+ * ADDED event, and removed with a REMOVE event. Net DMA client takes an
+ * extra reference per outstanding transaction. The relase function does a
+ *kref_puton the device. -ChrisL
+ */drivers/dma/dmaengine.c new file +19
-0+ +#include <linux/init.h> +#include <linux/module.h> +#include <linux/device.h> +#include <linux/dmaengine.h> +#include <linux/hardirq.h> +#include <linux/spinlock.h> +#include <linux/percpu.h> +#include <linux/rcupdate.h> + +static DEFINE_SPINLOCK (dma_list_lock); +static LIST_HEAD (dma_device_list); +static LIST_HEAD (dma_client_list); + +/* --- sysfs implementation --- */ + +static ssize_t show_memcpy_count (struct class_device *cd,char *buf) +{ + struct dma_chan *chan = container_of(cd, struct dma_chan,
class_dev);drivers/dma/dmaengine.c new file +10
-0+unsigned long count =0 ; +int i; + +for_each_cpu (i) + count +=per_cpu_ptr (chan->local, i)->memcpy_count; + +return sprintf (buf,"%lu\n" , count); +} + +static ssize_t show_bytes_transferred(struct class_device *cd,char *buf)
drivers/dma/dmaengine.c new file +2
-0+{ + struct dma_chan *chan = container_of(cd, struct dma_chan,
class_dev);
drivers/dma/dmaengine.c line 98
Quoted 9 lines
drivers/dma/dmaengine.c new file +9
-0+unsigned long count =0 ; +int i; + +for_each_cpu (i) + count +=per_cpu_ptr (chan->local, i)->bytes_transferred; + +return sprintf (buf,"%lu\n" , count); +} +
What is the utility of exporting memcpy_count, and bytes_transferred to userspace via sysfs? Is this really for debug (and thus should be under debugfs?)
Quoted 5 lines
drivers/dma/dmaengine.c new file +3
-0+static ssize_t show_in_use (struct class_device *cd,char *buf) +{ + struct dma_chan *chan = container_of(cd, struct dma_chan,
class_dev);drivers/dma/dmaengine.c new file +1
-0+
drivers/dma/dmaengine.c line 112
Quoted 10 lines
drivers/dma/dmaengine.c new file +10
-0+return sprintf (buf,"%d\n" , (chan->client ?1 :0 )); +} + +static struct class_device_attribute dma_class_attrs[] = { +__ATTR (memcpy_count,S_IRUGO , show_memcpy_count,NULL ), +__ATTR (bytes_transferred,S_IRUGO , show_bytes_transferred,NULL ), +__ATTR (in_use,S_IRUGO , show_in_use,NULL ), + __ATTR_NULL +}; +
[snip]
- kumar
Re: [PATCH 1/9] [I/OAT] DMA memcpy subsystem
Quoted 3 lines · Kumar Gala
What is the utility of exporting
memcpy_count, andbytes_transferredto userspace via sysfs? Is this really for debug (and thus should be under debugfs?)
Well....it's true they're useful for debugging but I would put them in the category of system statistics that shouldn't go in debugfs. I think they are like /proc/interrupts' interrupt counts or the TX/RX stats reported by ifconfig.
Regards -- Andy
Re: [PATCH 1/9] [I/OAT] DMA memcpy subsystem
Quoted 9 lines · Andrew Grover
Quoted 3 lines · Kumar Gala
What is the utility of exporting
memcpy_count, andbytes_transferredto userspace via sysfs? Is this really for debug (and thus should be under debugfs?)Well....it's true they're useful for debugging but I would put them in the category of system statistics that shouldn't go in debugfs. I think they are like
/proc/interrupts' interrupt counts or the TX/RX stats reported by ifconfig.
Fair, but wouldn't it be better to have the association per client.
Maybe leave the one as a summary and have a dir per client with similar stats that are for each client and add a per channel summary at the top level as well.
- kumar
Re: [PATCH 1/9] [I/OAT] DMA memcpy subsystem
Quoted 11 lines · Kumar Gala
Quoted 4 lines · Andrew Grover
Well....it's true they're useful for debugging but I would put them in the category of system statistics that shouldn't go in debugfs. I think they are like
/proc/interrupts' interrupt counts or the TX/RX stats reported by ifconfig.Fair, but wouldn't it be better to have the association per client.
Maybe leave the one as a summary and have a dir per client with similar stats that are for each client and add a per channel summary at the top level as well.
Such level of detail really belongs to debugging, IMHO.
I think, it would suffer to say, how many channels are in use. So you can answer the question, whether your customers are actually using this experimental technology.
If you want more, let them mount debugfs.
If it becomes really important, we can revisit this later.
Thats the advantage of files under debugfs
not being stable API in any way.
BTW: What is the actual frequency, at which such counters will be incremented?
Regards
Ingo Oeser
Re: [PATCH 1/9] [I/OAT] DMA memcpy subsystem
Quoted 7 lines · Ingo Oeser
Quoted 5 lines · Kumar Gala
Fair, but wouldn't it be better to have the association per client.
Maybe leave the one as a summary and have a dir per client with similar stats that are for each client and add a per channel summary at the top level as well.
Such level of detail really belongs to debugging, IMHO.
[snip]
If we implemented more stats then yes debugfs sounds like it might be the way to go.
Quoted 2 lines
BTW: What is the actual frequency, at which such counters will be incremented?
Currently the code updates these variables (kept per cpu) every time a copy is queued. See include/linux/dmaengine.h.
-- Andy
Re: [PATCH 1/9] [I/OAT] DMA memcpy subsystem
Quoted 10 lines · Andrew Grover
Quoted 7 lines · Ingo Oeser
Quoted 5 lines · Kumar Gala
Fair, but wouldn't it be better to have the association per client.
Maybe leave the one as a summary and have a dir per client with similar stats that are for each client and add a per channel summary at the top level as well.
Such level of detail really belongs to debugging, IMHO.
[snip]
Quoted 8 lines
If we implemented more stats then yes debugfs sounds like it might be the way to go.
Quoted 2 lines
BTW: What is the actual frequency, at which such counters will be incremented?
Currently the code updates these variables (kept per cpu) every time a copy is queued. See
include/linux/dmaengine.h.
Might it be better to update when the transfer is done incase of an error?
- kumar
Re: [PATCH 1/9] [I/OAT] DMA memcpy subsystem
Quoted 5 lines · Kumar Gala
Quoted 2 lines
Currently the code updates these variables (kept per cpu) every time a copy is queued. See
include/linux/dmaengine.h.Might it be better to update when the transfer is done incase of an error?
The queueing function is really in the best position to do this. It knows the size of each request. However in the cleanup/status check routine, all we know is the last request completed -- we don't know the completed requests' sizes.
The other reason is that the DMA engine should never throw an error. If it does then something is very wrong, we print scary warnings, and up-to-date stats are the least of our problems.
Regards -- Andy
PATCH 3/9 Setup the networking subsystem as a DMA client
In reply to Chris LeechAttempts to allocate per-CPU DMA channels
---
drivers/dma/Kconfig | 12 +++++ include/linux/netdevice.h | 4 ++ include/net/netdma.h | 38 ++++++++++++++++ net/core/dev.c | 104 +++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 158 insertions(+), 0 deletions(-)
drivers/dma/Kconfig +12 -0
@@ -10,6 +10,18 @@ config DMA_ENGINE DMA engines offload copy operations from the CPU to dedicated hardware, allowing the copies to happen asynchronously. +comment "DMA Clients" + +config NET_DMA + bool "Network: TCP receive copy offload" + depends on DMA_ENGINE && NET + default y + ---help--- + This enables the use of DMA engines in the network stack to + offload receive copy-to-user operations, freeing CPU cycles. + Since this is the main user of the DMA engine, it should be enabled; + say Y here. + comment "DMA Devices" config INTEL_IOATDMA
include/linux/netdevice.h +4 -0
@@ -37,6 +37,7 @@#include <linux/config.h> #include <linux/device.h> #include <linux/percpu.h> +#include <linux/dmaengine.h> struct divert_blk ;struct vlan_group ;
@@ -592,6 +593,9 @@ struct softnet_datastruct sk_buff *completion_queue;struct net_device backlog_dev;/* Sorry. 8) */ +#ifdef CONFIG_NET_DMA +struct dma_chan *net_dma; +#endif }; DECLARE_PER_CPU(struct softnet_data,softnet_data);
include/net/netdma.h new file +38 -0
@@ -0,0 +1,38 @@ +/* +* Copyright(c) 2004 - 2006 Intel Corporation. All rights reserved. +* +* This program is free software; you can redistribute it and/or modify it +* under the terms of the GNU General Public License as published by the Free +* Software Foundation; either version 2 of the License, or (at your option) +* any later version. +* +* This program is distributed in the hope that it will be useful, but WITHOUT +* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +* more details. +* +* You should have received a copy of the GNU General Public License along with +* this program; if not, write to the Free Software Foundation, Inc., 59 +* Temple Place - Suite 330, Boston, MA 02111-1307, USA. +* +* The full GNU General Public License is included in this distribution in the +* file called COPYING. +*/ +#ifndef NETDMA_H +#define NETDMA_H +#include <linux/config.h> +#ifdef CONFIG_NET_DMA +#include <linux/dmaengine.h> + +static inline struct dma_chan *get_softnet_dma (void ) +{ +struct dma_chan *chan; +rcu_read_lock (); + chan =rcu_dereference (__get_cpu_var (softnet_data.net_dma)); +if (chan) +dma_chan_get (chan); +rcu_read_unlock (); +return chan; +} +#endif /* CONFIG_NET_DMA */ +#endif /* NETDMA_H */
net/core/dev.c +104 -0
134 lines not on this page · raw message
PATCH 4/9 Utility functions for offloading sk_buff to iovec copies
In reply to Chris LeechProvides for pinning user space pages in memory, copying to iovecs, and copying from sk_buffs including fragmented and chained sk_buffs.
---
drivers/dma/Makefile | 3 drivers/dma/iovlock.c | 301 +++++++++++++++++++++++++++++++++++++++++++++ include/linux/dmaengine.h | 22 +++ include/net/netdma.h | 6 + net/core/Makefile | 1 net/core/user_dma.c | 141 +++++++++++++++++++++ 6 files changed, 473 insertions(+), 1 deletions(-)
drivers/dma/Makefile +2 -1
@@ -1,2 +1,3 @@ -obj-y += dmaengine.o +obj-$(CONFIG_DMA_ENGINE) += dmaengine.o +obj-$(CONFIG_NET_DMA) += iovlock.o obj-$(CONFIG_INTEL_IOATDMA) += ioatdma.o
drivers/dma/iovlock.c new file +301 -0
301 lines not on this page · raw message
include/linux/dmaengine.h +22 -0
@@ -333,5 +333,27 @@ static inline enum dma_status dma_async_int dma_async_device_register (struct dma_device *device);void dma_async_device_unregister (struct dma_device *device); +/* --- Helper iov-locking functions --- */ + +struct dma_page_list { +char *base_address; +int nr_pages; +struct page **pages; +}; + +struct dma_pinned_list { +int nr_iovecs; +struct dma_page_list page_list[0 ]; +}; + +struct dma_pinned_list *dma_pin_iovec_pages (struct iovec *iov,size_t len); +void dma_unpin_iovec_pages (struct dma_pinned_list * pinned_list); + +dma_cookie_t dma_memcpy_to_iovec (struct dma_chan *chan,struct iovec *iov, +struct dma_pinned_list *pinned_list,unsigned char *kdata,size_t len); +dma_cookie_t dma_memcpy_pg_to_iovec (struct dma_chan *chan,struct iovec *iov, +struct dma_pinned_list *pinned_list,struct page *page, +unsigned int offset,size_t len); +#endif /* CONFIG_DMA_ENGINE */ #endif /* DMAENGINE_H */
include/net/netdma.h +6 -0
@@ -23,6 +23,7 @@#include <linux/config.h> #ifdef CONFIG_NET_DMA #include <linux/dmaengine.h> +#include <linux/skbuff.h> static inline struct dma_chan *get_softnet_dma (void ) {
@@ -34,5 +35,10 @@ static inline struct dma_chan *get_softnrcu_read_unlock ();return chan; } + +int dma_skb_copy_datagram_iovec (struct dma_chan * chan, +const struct sk_buff *skb,int offset,struct iovec *to, +size_t len,struct dma_pinned_list *pinned_list); +#endif /* CONFIG_NET_DMA */ #endif /* NETDMA_H */
net/core/Makefile +1 -0
@@ -16,3 +16,4 @@ obj-$(CONFIG_NET_DIVERT) += dv.o obj-$(CONFIG_NET_PKTGEN) += pktgen.o obj-$(CONFIG_WIRELESS_EXT) += wireless.o obj-$(CONFIG_NETPOLL) += netpoll.o +obj-$(CONFIG_NET_DMA) += user_dma.o
net/core/user_dma.c new file +141 -0
141 lines not on this page · raw message
PATCH 5/9 Structure changes for TCP recv offload to I/OAT
In reply to Chris LeechAdds an async_wait_queue and some additional fields to tcp_sock, and a dma_cookie_t to sk_buff.
---
include/linux/skbuff.h | 4 ++++ include/linux/tcp.h | 8 ++++++++ include/net/sock.h | 2 ++ include/net/tcp.h | 7 +++++++ net/core/sock.c | 6 ++++++ 5 files changed, 27 insertions(+), 0 deletions(-)
include/linux/skbuff.h +4 -0
@@ -29,6 +29,7 @@#include <linux/net.h> #include <linux/textsearch.h> #include <net/checksum.h> +#include <linux/dmaengine.h> #define HAVE_ALLOC_SKB /* For the drivers to know */ #define HAVE_ALIGNABLE_SKB /* Ditto 8) */
@@ -285,6 +286,9 @@ struct sk_buff {__u16 tc_verd;/* traffic control verdict */ #endif #endif +#ifdef CONFIG_NET_DMA +dma_cookie_t dma_cookie; +#endif /* These elements must be at the end, see alloc_skb() for details. */
include/linux/tcp.h +8 -0
@@ -18,6 +18,7 @@#define _LINUX_TCP_H#include <linux/types.h> +#include <linux/dmaengine.h> #include <asm/byteorder.h> struct tcphdr {
@@ -233,6 +234,13 @@ struct tcp_sock {struct iovec *iov;int memory;int len; +#ifdef CONFIG_NET_DMA +/* members for async copy */ +struct dma_chan *dma_chan; +int wakeup; +struct dma_pinned_list *pinned_list; +dma_cookie_t dma_cookie; +#endif } ucopy;__u32 snd_wl1;/* Sequence for window update */
include/net/sock.h +2 -0
@@ -132,6 +132,7 @@ struct sock_common {* @sk_receive_queue: incoming packets * @sk_wmem_alloc: transmit queue bytes committed * @sk_write_queue: Packet sending queue +* @sk_async_wait_queue: DMA copied packets * @sk_omem_alloc: "o" is "option" or "other" * @sk_wmem_queued: persistent queue size * @sk_forward_alloc: space allocated forward
@@ -205,6 +206,7 @@ struct sock {atomic_t sk_omem_alloc;struct sk_buff_head sk_receive_queue;struct sk_buff_head sk_write_queue; +struct sk_buff_head sk_async_wait_queue;int sk_wmem_queued;int sk_forward_alloc;gfp_t sk_allocation;
include/net/tcp.h +7 -0
@@ -28,6 +28,7 @@#include <linux/cache.h> #include <linux/percpu.h> #include <linux/skbuff.h> +#include <linux/dmaengine.h> #include <net/inet_connection_sock.h> #include <net/inet_timewait_sock.h>
@@ -820,6 +821,12 @@ static inline void tcp_prequeue_init(str tp->ucopy.len =0 ; tp->ucopy.memory =0 ;skb_queue_head_init (&tp->ucopy.prequeue); +#ifdef CONFIG_NET_DMA + tp->ucopy.dma_chan =NULL ; + tp->ucopy.wakeup =0 ; + tp->ucopy.pinned_list =NULL ; + tp->ucopy.dma_cookie =0 ; +#endif }/* Packet is added to VJ-style prequeue for processing in process
net/core/sock.c +6 -0
@@ -818,6 +818,9 @@ struct sock *sk_clone(const struct sockatomic_set (&newsk->sk_omem_alloc,0 );skb_queue_head_init (&newsk->sk_receive_queue);skb_queue_head_init (&newsk->sk_write_queue); +#ifdef CONFIG_NET_DMA +skb_queue_head_init (&newsk->sk_async_wait_queue); +#endif rwlock_init (&newsk->sk_dst_lock);rwlock_init (&newsk->sk_callback_lock);
@@ -1369,6 +1372,9 @@ void sock_init_data(struct socket *sock,skb_queue_head_init (&sk->sk_receive_queue);skb_queue_head_init (&sk->sk_write_queue);skb_queue_head_init (&sk->sk_error_queue); +#ifdef CONFIG_NET_DMA +skb_queue_head_init (&sk->sk_async_wait_queue); +#endif sk->sk_send_head =NULL ;
PATCH 6/9 Rename cleanup_rbuf to tcp_cleanup_rbuf and make non-static
In reply to Chris LeechNeeded to be able to call tcp_cleanup_rbuf in tcp_input.c for I/OAT
---
include/net/tcp.h | 2 ++ net/ipv4/tcp.c | 10 +++++----- 2 files changed, 7 insertions(+), 5 deletions(-)
include/net/tcp.h +2 -0
@@ -294,6 +294,8 @@ extern int tcp_rcv_established(structextern void tcp_rcv_space_adjust (struct sock *sk); +extern void tcp_cleanup_rbuf (struct sock *sk,int copied); +extern int tcp_twsk_unique (struct sock *sk,struct sock *sktw,void *twp);
net/ipv4/tcp.c +5 -5
@@ -937,7 +937,7 @@ static int tcp_recv_urg(struct sock *sk,* calculation of whether or not we must ACK for the sake of * a window update. */ -static void cleanup_rbuf (struct sock *sk,int copied) +void tcp_cleanup_rbuf (struct sock *sk,int copied) {struct tcp_sock *tp =tcp_sk (sk);int time_to_ack =0 ;
@@ -1086,7 +1086,7 @@ int tcp_read_sock(struct sock *sk, read_/* Clean up data we have read: This will do ACK frames. */ if (copied) -cleanup_rbuf (sk, copied); +tcp_cleanup_rbuf (sk, copied);return copied; }
@@ -1220,7 +1220,7 @@ int tcp_recvmsg(struct kiocb *iocb, stru } } -cleanup_rbuf (sk, copied); +tcp_cleanup_rbuf (sk, copied);if (!sysctl_tcp_low_latency && tp->ucopy.task == user_recv) {/* Install new reader */
@@ -1391,7 +1391,7 @@ skip_copy:*/ /* Clean up data we have read: This will do ACK frames. */ -cleanup_rbuf (sk, copied); +tcp_cleanup_rbuf (sk, copied);TCP_CHECK_TIMER (sk);release_sock (sk);
@@ -1853,7 +1853,7 @@ static int do_tcp_setsockopt(struct sock (TCPF_ESTABLISHED | TCPF_CLOSE_WAIT) && inet_csk_ack_scheduled(sk)) { icsk->icsk_ack.pending |=ICSK_ACK_PUSHED ; -cleanup_rbuf (sk,1 ); +tcp_cleanup_rbuf (sk,1 );if (!(val &1 )) icsk->icsk_ack.pingpong =1 ; }
PATCH 7/9 make sk_eat_skb I/OAT aware
In reply to Chris LeechAdd an extra argument to sk_eat_skb, and make it move early copied packets to the async_wait_queue instead of freeing them.
---
include/net/sock.h | 13 ++++++++++++- net/dccp/proto.c | 4 ++-- net/ipv4/tcp.c | 8 ++++---- net/llc/af_llc.c | 2 +- 4 files changed, 19 insertions(+), 8 deletions(-)
include/net/sock.h +12 -1
@@ -1272,11 +1272,22 @@ sock_recv_timestamp(struct msghdr *msg,* This routine must be called with interrupts disabled or with the socket * locked so that the sk_buff queue operation is ok. */ -static inline void sk_eat_skb (struct sock *sk,struct sk_buff *skb) +#ifdef CONFIG_NET_DMA +static inline void sk_eat_skb (struct sock *sk,struct sk_buff *skb,int copied_early) +{ +__skb_unlink (skb, &sk->sk_receive_queue); +if (!copied_early) +__kfree_skb (skb); +else +__skb_queue_tail (&sk->sk_async_wait_queue, skb); +} +#else +static inline void sk_eat_skb (struct sock *sk,struct sk_buff *skb,int copied_early) {__skb_unlink (skb, &sk->sk_receive_queue);__kfree_skb (skb); } +#endif extern void sock_enable_timestamp (struct sock *sk);extern int sock_get_timestamp (struct sock *,struct timeval __user *);
net/dccp/proto.c +2 -2
@@ -719,7 +719,7 @@ int dccp_recvmsg(struct kiocb *iocb, str }dccp_pr_debug ("packet_type=%s\n" ,dccp_packet_name (dh->dccph_type)); -sk_eat_skb (sk, skb); +sk_eat_skb (sk, skb,0 ); verify_sock_status:if (sock_flag (sk,SOCK_DONE )) { len =0 ;
@@ -773,7 +773,7 @@ verify_sock_status: } found_fin_ok:if (!(flags &MSG_PEEK )) -sk_eat_skb (sk, skb); +sk_eat_skb (sk, skb,0 );break ; }while (1 ); out:
net/ipv4/tcp.c +4 -4
@@ -1072,11 +1072,11 @@ int tcp_read_sock(struct sock *sk, read_break ; }if (skb->h.th->fin) { -sk_eat_skb (sk, skb); +sk_eat_skb (sk, skb,0 ); ++seq;break ; } -sk_eat_skb (sk, skb); +sk_eat_skb (sk, skb,0 );if (!desc->count)break ; }
@@ -1356,14 +1356,14 @@ skip_copy:if (skb->h.th->fin) goto found_fin_ok;if (!(flags &MSG_PEEK )) -sk_eat_skb (sk, skb); +sk_eat_skb (sk, skb,0 );continue ; found_fin_ok:/* Process the FIN. */ ++*seq;if (!(flags &MSG_PEEK )) -sk_eat_skb (sk, skb); +sk_eat_skb (sk, skb,0 );break ; }while (len >0 );
net/llc/af_llc.c +1 -1
@@ -789,7 +789,7 @@ static int llc_ui_recvmsg(struct kiocb *continue ;if (!(flags &MSG_PEEK )) { -sk_eat_skb (sk, skb); +sk_eat_skb (sk, skb,0 ); *seq =0 ; } }while (len >0 );
PATCH 8/9 Add a sysctl for tuning the I/OAT offloaded I/O threshold
In reply to Chris LeechAny socket recv of less than this ammount will not be offloaded
---
include/linux/sysctl.h | 1 + include/net/tcp.h | 1 + net/core/user_dma.c | 4 ++++ net/ipv4/sysctl_net_ipv4.c | 10 ++++++++++ 4 files changed, 16 insertions(+), 0 deletions(-)
include/linux/sysctl.h +1 -0
@@ -403,6 +403,7 @@ enumNET_TCP_MTU_PROBING =113 ,NET_TCP_BASE_MSS =114 ,NET_IPV4_TCP_WORKAROUND_SIGNED_WINDOWS =115 , + NET_TCP_DMA_COPYBREAK=116, }; enum {
include/net/tcp.h +1 -0
@@ -219,6 +219,7 @@ extern int sysctl_tcp_adv_win_scale;extern int sysctl_tcp_tw_reuse;extern int sysctl_tcp_frto;extern int sysctl_tcp_low_latency; +extern int sysctl_tcp_dma_copybreak;extern int sysctl_tcp_nometrics_save;extern int sysctl_tcp_moderate_rcvbuf;extern int sysctl_tcp_tso_win_divisor;
net/core/user_dma.c +4 -0
@@ -33,6 +33,10 @@#ifdef CONFIG_NET_DMA +#define NET_DMA_DEFAULT_COPYBREAK 1024 + +int sysctl_tcp_dma_copybreak =NET_DMA_DEFAULT_COPYBREAK ; +/** * dma_skb_copy_datagram_iovec - Copy a datagram to an iovec. * @skb - buffer to copy
net/ipv4/sysctl_net_ipv4.c +10 -0
@@ -688,6 +688,16 @@ ctl_table ipv4_table[] = { .mode = 0644, .proc_handler = &proc_dointvec }, +#ifdef CONFIG_NET_DMA + { + .ctl_name = NET_TCP_DMA_COPYBREAK, + .procname = "tcp_dma_copybreak", + .data = &sysctl_tcp_dma_copybreak, + .maxlen = sizeof(int), + .mode = 0644, + .proc_handler = &proc_dointvec + }, +#endif { .ctl_name = 0 } };
PATCH 9/9 TCP recv offload to I/OAT
In reply to Chris LeechLocks down user pages and sets up for DMA in tcp_recvmsg, then calls dma_async_try_early_copy in tcp_v4_do_rcv
---
net/ipv4/tcp.c | 101 ++++++++++++++++++++++++++++++++++++++++++++------ net/ipv4/tcp_input.c | 74 +++++++++++++++++++++++++++++++++---- net/ipv4/tcp_ipv4.c | 18 ++++++++- net/ipv6/tcp_ipv6.c | 12 +++++- 4 files changed, 183 insertions(+), 22 deletions(-)
net/ipv4/tcp.c +89 -12
@@ -263,7 +263,7 @@#include <net/tcp.h> #include <net/xfrm.h> #include <net/ip.h> - +#include <net/netdma.h> #include <asm/uaccess.h> #include <asm/ioctls.h>
@@ -1110,6 +1110,7 @@ int tcp_recvmsg(struct kiocb *iocb, struint target;/* Read at least this many bytes */ long timeo;struct task_struct *user_recv =NULL ; +int copied_early =0 ;lock_sock (sk);
@@ -1133,6 +1134,15 @@ int tcp_recvmsg(struct kiocb *iocb, stru target =sock_rcvlowat (sk, flags &MSG_WAITALL , len); +#ifdef CONFIG_NET_DMA + tp->ucopy.dma_chan =NULL ; +preempt_disable (); +if ((len > sysctl_tcp_dma_copybreak) && !(flags &MSG_PEEK ) && + !sysctl_tcp_low_latency &&__get_cpu_var (softnet_data.net_dma)) + tp->ucopy.pinned_list =dma_pin_iovec_pages (msg->msg_iov, len); +preempt_enable_no_resched (); +#endif + do { struct sk_buff *skb; u32 offset;
@@ -1274,6 +1284,10 @@ int tcp_recvmsg(struct kiocb *iocb, stru }else sk_wait_data (sk, &timeo); +#ifdef CONFIG_NET_DMA + tp->ucopy.wakeup =0 ; +#endif +if (user_recv) {int chunk;
@@ -1329,13 +1343,39 @@ do_prequeue: }if (!(flags &MSG_TRUNC )) { - err =skb_copy_datagram_iovec (skb, offset, - msg->msg_iov, used); -if (err) { -/* Exception. Bailout! */ -if (!copied) - copied = -EFAULT ; -break ; +#ifdef CONFIG_NET_DMA +if (!tp->ucopy.dma_chan && tp->ucopy.pinned_list) + tp->ucopy.dma_chan =get_softnet_dma (); + +if (tp->ucopy.dma_chan) { + tp->ucopy.dma_cookie =dma_skb_copy_datagram_iovec ( + tp->ucopy.dma_chan, skb, offset, + msg->msg_iov, used, + tp->ucopy.pinned_list); + +if (tp->ucopy.dma_cookie <0 ) { + +printk (KERN_ALERT "dma_cookie < 0\n" ); + +/* Exception. Bailout! */ +if (!copied) + copied = -EFAULT ; +break ; + } +if ((offset + used) == skb->len) + copied_early =1 ; + + } else +#endif + { + err =skb_copy_datagram_iovec (skb, offset, + msg->msg_iov, used); +if (err) { +/* Exception. Bailout! */ +if (!copied) + copied = -EFAULT ; +break ; + } } }
@@ -1355,15 +1395,19 @@ skip_copy:if (skb->h.th->fin) goto found_fin_ok; -if (!(flags &MSG_PEEK )) -sk_eat_skb (sk, skb,0 ); +if (!(flags &MSG_PEEK )) { +sk_eat_skb (sk, skb, copied_early); + copied_early =0 ; + }continue ; found_fin_ok:/* Process the FIN. */ ++*seq; -if (!(flags &MSG_PEEK )) -sk_eat_skb (sk, skb,0 ); +if (!(flags &MSG_PEEK )) { +sk_eat_skb (sk, skb, copied_early); + copied_early =0 ; + }break ; }while (len >0 );
@@ -1386,6 +1430,36 @@ skip_copy: tp->ucopy.len =0 ; } +#ifdef CONFIG_NET_DMA +if (tp->ucopy.dma_chan) { +struct sk_buff *skb; +dma_cookie_t done, used; + +dma_async_memcpy_issue_pending (tp->ucopy.dma_chan); + +while (dma_async_memcpy_complete (tp->ucopy.dma_chan, + tp->ucopy.dma_cookie, &done, + &used) ==DMA_IN_PROGRESS ) { +/* do partial cleanup of sk_async_wait_queue */ +while ((skb =skb_peek (&sk->sk_async_wait_queue)) && + (dma_async_is_complete (skb->dma_cookie, done, + used) ==DMA_SUCCESS )) { +__skb_dequeue (&sk->sk_async_wait_queue); +kfree_skb (skb); + } + } + +/* Safe to free early-copied skbs now */ +__skb_queue_purge (&sk->sk_async_wait_queue); +dma_chan_put (tp->ucopy.dma_chan); + tp->ucopy.dma_chan =NULL ; + } +if (tp->ucopy.pinned_list) { +dma_unpin_iovec_pages (tp->ucopy.pinned_list); + tp->ucopy.pinned_list =NULL ; + } +#endif +/* According to UNIX98, msg_name/msg_namelen are ignored * on connected socket. I was just happy when found this 8) --ANK */
@@ -1653,6 +1727,9 @@ int tcp_disconnect(struct sock *sk, int__skb_queue_purge (&sk->sk_receive_queue);sk_stream_writequeue_purge (sk);__skb_queue_purge (&tp->out_of_order_queue); +#ifdef CONFIG_NET_DMA +__skb_queue_purge (&sk->sk_async_wait_queue); +#endif inet->dport =0 ;
net/ipv4/tcp_input.c +67 -7
107 lines not on this page · raw message
net/ipv4/tcp_ipv4.c +17 -1
@@ -71,6 +71,7 @@#include <net/inet_common.h> #include <net/timewait_sock.h> #include <net/xfrm.h> +#include <net/netdma.h> #include <linux/inet.h> #include <linux/ipv6.h>
@@ -1091,8 +1092,18 @@ process:bh_lock_sock (sk); ret =0 ;if (!sock_owned_by_user (sk)) { -if (!tcp_prequeue (sk, skb)) +#ifdef CONFIG_NET_DMA +struct tcp_sock *tp =tcp_sk (sk); +if (!tp->ucopy.dma_chan && tp->ucopy.pinned_list) + tp->ucopy.dma_chan =get_softnet_dma (); +if (tp->ucopy.dma_chan) ret =tcp_v4_do_rcv (sk, skb); + else +#endif + { +if (!tcp_prequeue (sk, skb)) + ret =tcp_v4_do_rcv (sk, skb); + } }else sk_add_backlog (sk, skb);bh_unlock_sock (sk);
@@ -1296,6 +1307,11 @@ int tcp_v4_destroy_sock(struct sock *sk)/* Cleans up our, hopefully empty, out_of_order_queue. */ __skb_queue_purge (&tp->out_of_order_queue); +#ifdef CONFIG_NET_DMA +/* Cleans up our sk_async_wait_queue */ +__skb_queue_purge (&sk->sk_async_wait_queue); +#endif +/* Clean prequeue, it must be empty really */ __skb_queue_purge (&tp->ucopy.prequeue);
net/ipv6/tcp_ipv6.c +10 -2
@@ -1218,8 +1218,16 @@ process:bh_lock_sock (sk); ret =0 ;if (!sock_owned_by_user (sk)) { -if (!tcp_prequeue (sk, skb)) - ret =tcp_v6_do_rcv (sk, skb); +#ifdef CONFIG_NET_DMA +struct tcp_sock *tp =tcp_sk (sk); +if (tp->ucopy.dma_chan) + ret =tcp_v6_do_rcv (sk, skb); + else +#endif + { +if (!tcp_prequeue (sk, skb)) + ret =tcp_v6_do_rcv (sk, skb); + } }else sk_add_backlog (sk, skb);bh_unlock_sock (sk);