TCP socket splice
Subsystems (5)
cover TCP socket splice
My name is Ashwini Kulkarni and I have been working at Intel Corporation for the past 4 months as an engineering intern. I have been working on the 'TCP socket splice' project with Chris Leech. This is a work-in-progress version of the project with scope for further modifications.
TCP socket splicing:
It allows a TCP socket to be spliced to a file via a pipe buffer. First, to splice data from a socket to a pipe buffer, upto 16 source pages(s) are pulled into the pipe buffer. Then to splice data from the pipe buffer to a file, those pages are migrated into the address space of the target file. It takes place entirely within the kernel and thus results in zero memory copies. It is the receive side complement to sendfile() but unlike sendfile() it is possible to splice from a socket as well and not just to a socket.
Current Method:
+ > Application Buffer +
| |
_________________|_______________________|_____________
| |
Receive or | | Write
I/OAT DMA | |
| |
| V
Network File System
Buffer Buffer
^ |
| |
_________________|_______________________|_____________
DMA | | DMA
| |
Hardware | |
| V
NIC SATA
In the current method, the packet is DMA’d from the NIC into the network buffer. There is a read on socket to the user space and the packet data is copied from the network buffer to the application buffer. A write operation then moves the data from the application buffer to the file system buffer which is then DMA'd to the disk again. Thus, in the current method there will be one full copy of all the data to the user space.
Using TCP socket splice:
Application Control
|
_________________|__________________________________
|
| TCP socket splice
| +---------------------+
| | Direct path |
V | V
Network File System
Buffer Buffer
^ |
| |
_________________|_______________________|__________
DMA | | DMA
| |
Hardware | |
| V
NIC SATA
In this method, the objective is to use TCP socket splicing to create a direct path in the kernel from the network buffer to the file system buffer via a pipe buffer. The pages will migrate from the network buffer (which is associated with the socket) into the pipe buffer for an optimized path. From the pipe buffer, the pages will then be migrated to the output file address space page cache. This will enable to create a LAN to file-system API which will avoid the memcpy operations in user space and thus create a fast path from the network buffer to the storage buffer.
Open Issues (currently being addressed):
There is a performance drop when transferring bigger files (usually larger than 65536 bytes in size). Performance drop increases with the size of the file. Work is in progress to identify the source of this issue.
We encourage the community to review our TCP socket splice project. Feedback would be greatly appreciated.
--
Ashwini Kulkarni
RFC 1/6 Make splice_to_pipe non-static and move structure definitions to a header file
---
fs/splice.c | 18 +----------------- include/linux/pipe_fs_i.h | 18 ++++++++++++++++++ 2 files changed, 19 insertions(+), 17 deletions(-)
fs/splice.c +1 -17
@@ -29,22 +29,6 @@#include <linux/syscalls.h> #include <linux/uio.h> -struct partial_page { -unsigned int offset; -unsigned int len; -}; - -/* -* Passed to splice_to_pipe -*/ -struct splice_pipe_desc { -struct page **pages;/* page map */ -struct partial_page *partial;/* pages[] may not be contig */ -int nr_pages;/* number of pages in map */ -unsigned int flags;/* splice flags */ -struct pipe_buf_operations *ops;/* ops associated with output pipe */ -}; -/* * Attempt to steal a page from a pipe buffer. This should perhaps go into * a vm helper function, it's already simplified quite a bit by the
@@ -173,7 +157,7 @@ static struct pipe_buf_operations user_p* Pipe output worker. This sets up our pipe format with the page cache * pipe buffer operations. Otherwise very similar to the regular pipe_writev(). */ -static ssize_t splice_to_pipe (struct pipe_inode_info *pipe, +ssize_t splice_to_pipe (struct pipe_inode_info *pipe,struct splice_pipe_desc *spd) {int ret, do_wakeup, page_nr;
include/linux/pipe_fs_i.h +18 -0
@@ -100,4 +100,22 @@ extern ssize_t splice_from_pipe(struct p loff_t *, size_t, unsigned int, splice_actor *); +struct partial_page { +unsigned int offset; +unsigned int len; +}; + +/* +* Passed to splice_to_pipe +*/ +struct splice_pipe_desc { +struct page **pages;/* page map */ +struct partial_page *partial;/* pages[] may not be contig */ +int nr_pages;/* number of pages in map */ +unsigned int flags;/* splice flags */ +struct pipe_buf_operations *ops;/* ops associated with output pipe */ +}; + +ssize_t splice_to_pipe (struct pipe_inode_info *,struct splice_pipe_desc *); +#endif
RFC 2/6 Make sock_def_wakeup non-static
In reply to Ashwini Kulkarni---
include/net/sock.h | 1 + net/core/sock.c | 3 ++- 2 files changed, 3 insertions(+), 1 deletions(-)
include/net/sock.h +1 -0
@@ -497,6 +497,7 @@ extern void sk_stream_wait_close(structextern int sk_stream_error (struct sock *sk,int flags,int err);extern void sk_stream_kill_queues (struct sock *sk); +extern void sock_def_wakeup (struct sock *sk);extern int sk_wait_data (struct sock *sk,long *timeo);struct request_sock_ops ;
net/core/sock.c +2 -1
@@ -1400,7 +1400,7 @@ ssize_t sock_no_sendpage(struct socket ** Default Socket Callbacks */ -static void sock_def_wakeup (struct sock *sk) +void sock_def_wakeup (struct sock *sk) {read_lock (&sk->sk_callback_lock);if (sk->sk_sleep &&waitqueue_active (sk->sk_sleep))
@@ -1961,6 +1961,7 @@ EXPORT_SYMBOL(sock_no_poll);EXPORT_SYMBOL (sock_no_recvmsg);EXPORT_SYMBOL (sock_no_sendmsg);EXPORT_SYMBOL (sock_no_sendpage); +EXPORT_SYMBOL (sock_def_wakeup);EXPORT_SYMBOL (sock_no_setsockopt);EXPORT_SYMBOL (sock_no_shutdown);EXPORT_SYMBOL (sock_no_socketpair);
RFC 3/6 Add in TCP related part of splice read to ipv4
In reply to Ashwini Kulkarni---
net/ipv4/af_inet.c | 1 net/ipv4/tcp.c | 135 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 136 insertions(+), 0 deletions(-)
net/ipv4/af_inet.c +1 -0
@@ -807,6 +807,7 @@ const struct proto_ops inet_stream_ops = .recvmsg = sock_common_recvmsg, .mmap = sock_no_mmap, .sendpage = tcp_sendpage, + .splice_read = tcp_splice_read, #ifdef CONFIG_COMPAT .compat_setsockopt = compat_sock_common_setsockopt, .compat_getsockopt = compat_sock_common_getsockopt,
net/ipv4/tcp.c +135 -0
165 lines not on this page · raw message
RFC 4/6 Add TCP socket splicing (tcp_splice_read) support
In reply to Ashwini Kulkarni---
fs/splice.c | 16 ++++++++++++++++ include/linux/net.h | 2 ++ include/linux/pipe_fs_i.h | 1 + include/net/tcp.h | 3 +++ net/socket.c | 13 +++++++++++++ 5 files changed, 35 insertions(+), 0 deletions(-)
fs/splice.c +16 -0
@@ -123,6 +123,12 @@ error:return err; } +void generic_sock_buf_release (struct pipe_inode_info *pipe, +struct pipe_buffer *buf) +{ +put_page (buf->page); +} +static struct pipe_buf_operations page_cache_pipe_buf_ops = { .can_merge =0 , .map = generic_pipe_buf_map,
@@ -133,6 +139,16 @@ static struct pipe_buf_operations page_c .get = generic_pipe_buf_get, }; +static struct pipe_buf_operations sock_buf_ops = { + .can_merge =0 , + .map = generic_pipe_buf_map, + .unmap = generic_pipe_buf_unmap, + .pin = generic_pipe_buf_pin, + .release = generic_sock_buf_release, + .steal = generic_pipe_buf_steal, + .get = generic_pipe_buf_get, +}; +static int user_page_pipe_buf_steal (struct pipe_inode_info *pipe,struct pipe_buffer *buf) {
include/linux/net.h +2 -0
@@ -164,6 +164,8 @@ struct proto_ops { struct vm_area_struct * vma);ssize_t (*sendpage) (struct socket *sock,struct page *page,int offset,size_t size,int flags); +ssize_t (*splice_read)(struct socket *sock,loff_t *ppos, +struct pipe_inode_info *pipe,size_t len,unsigned int flags); };struct net_proto_family {
include/linux/pipe_fs_i.h +1 -0
@@ -72,6 +72,7 @@ void generic_pipe_buf_get(struct pipe_inint generic_pipe_buf_pin (struct pipe_inode_info *,struct pipe_buffer *);int generic_pipe_buf_steal (struct pipe_inode_info *,struct pipe_buffer *); +void generic_sock_buf_release (struct pipe_inode_info *,struct pipe_buffer *);/* * splice is tied to pipes as a transport (at least for now), so we'll just * add the splice flags here.
include/net/tcp.h +3 -0
@@ -300,6 +300,9 @@ extern void tcp_cleanup_rbuf(struct soextern int tcp_twsk_unique (struct sock *sk,struct sock *sktw,void *twp); +extern ssize_t tcp_splice_read (struct socket *sk,loff_t *ppos, +struct pipe_inode_info *pipe,size_t len,unsigned int flags); +static inline void tcp_dec_quickack_mode (struct sock *sk,const unsigned int pkts) {
net/socket.c +13 -0
@@ -117,6 +117,8 @@ static ssize_t sock_writev(struct file * unsigned long count, loff_t *ppos);static ssize_t sock_sendpage (struct file *file,struct page *page,int offset,size_t size,loff_t *ppos,int more); +static ssize_t sock_splice_read (struct file *file,loff_t *ppos, +struct pipe_inode_info *pipe,size_t len,unsigned int flags);/* * Socket files have a set of 'special' operations as well as the generic file ones. These don't appear
@@ -141,6 +143,7 @@ static struct file_operations socket_fil .writev = sock_writev, .sendpage = sock_sendpage, .splice_write = generic_splice_sendpage, + .splice_read = sock_splice_read, };/*
@@ -701,6 +704,16 @@ static ssize_t sock_sendpage(struct filereturn sock->ops->sendpage (sock, page, offset, size, flags); } +static ssize_t sock_splice_read (struct file *file,loff_t *ppos, +struct pipe_inode_info *pipe,size_t len,unsigned int flags) +{ +struct socket *sock; + + sock = file->private_data; + +return sock->ops->splice_read (sock, ppos, pipe, len, flags); +} +static struct sock_iocb *alloc_sock_iocb (struct kiocb *iocb,char __user *ubuf,size_t size,struct sock_iocb *siocb) {
RFC 5/6 Add skb_splice_bits to skbuff.c
In reply to Ashwini Kulkarni---
include/linux/skbuff.h | 2 + net/core/skbuff.c | 137 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 139 insertions(+), 0 deletions(-)
include/linux/skbuff.h +2 -0
@@ -1338,6 +1338,8 @@ extern unsigned int skb_checksum(cons int len, unsigned int csum);extern int skb_copy_bits (const struct sk_buff *skb,int offset,void *to,int len); +extern int skb_splice_bits (const struct sk_buff *skb,int offset, +struct pipe_inode_info *pipe,int len,unsigned int flags);extern int skb_store_bits (const struct sk_buff *skb,int offset,void *from,int len); extern unsigned int skb_copy_and_csum_bits(const struct sk_buff *skb,
net/core/skbuff.c +137 -0
155 lines not on this page · raw message
RFC 6/6 Move i_size_read part from do_splice_to() to __generic_file_splice_read() in splice.c
In reply to Ashwini Kulkarni---
fs/splice.c | 18 ++++++++---------- 1 files changed, 8 insertions(+), 10 deletions(-)
fs/splice.c +8 -10
@@ -271,7 +271,7 @@ __generic_file_splice_read(struct file *struct partial_page partial[PIPE_BUFFERS ];struct page *page;pgoff_t index, end_index; -loff_t isize; +loff_t isize, left;size_t total_len;int error, page_nr;struct splice_pipe_desc spd = {
@@ -421,6 +421,13 @@ __generic_file_splice_read(struct file ** i_size must be checked after ->readpage(). */ isize =i_size_read (mapping->host); +if (unlikely (*ppos >= isize)) +return 0 ; + + left = isize - *ppos; +if (unlikely (left < len)) + len = left; + end_index = (isize -1 ) >>PAGE_CACHE_SHIFT ;if (unlikely (!isize || index > end_index))break ;
@@ -903,7 +910,6 @@ static long do_splice_to(struct file *in struct pipe_inode_info *pipe, size_t len, unsigned int flags) { - loff_t isize, left; int ret;if (unlikely (!in->f_op || !in->f_op->splice_read))
@@ -916,14 +922,6 @@ static long do_splice_to(struct file *inif (unlikely (ret <0 ))return ret; - isize =i_size_read (in->f_mapping->host); -if (unlikely (*ppos >= isize)) -return 0 ; - - left = isize - *ppos; -if (unlikely (left < len)) - len = left; -return in->f_op->splice_read (in, ppos, pipe, len, flags); }
Quoted 30 lines · Ashwini Kulkarni
Using TCP socket splice:
Application Control
|
_________________|__________________________________
|
| TCP socket splice
| +---------------------+
| | Direct path |
V | V
Network File System
Buffer Buffer
^ |
| |
_________________|_______________________|__________
DMA | | DMA
| |
Hardware | |
| V
NIC SATAIn this method, the objective is to use TCP socket splicing to create a direct path in the kernel from the network buffer to the file system buffer via a pipe buffer. The pages will migrate from the network buffer (which is associated with the socket) into the pipe buffer for an optimized path. From the pipe buffer, the pages will then be migrated to the output file address space page cache. This will enable to create a LAN to file-system API which will avoid the memcpy operations in user space and thus create a fast path from the network buffer to the storage buffer.
Quoted 7 lines
Open Issues (currently being addressed):
There is a performance drop when transferring bigger files (usually larger than 65536 bytes in size). Performance drop increases with the size of the file. Work is in progress to identify the source of this issue.We encourage the community to review our TCP socket splice project. Feedback would be greatly appreciated.
First of all it is not zero-copy, most of the time when mtu is not changed skb does not have fragments, which means that you need to copy, and you do it in skb_splice_bits() after skb_headlen() check.
Additionally to copy you add kmap/kunmap overhead, which can be very noticeble. I would not be surprised that exactly that part introduces described above performance drop compared to copy_*_user() approach. Did you checked it with (hacked) drivers, which put data into fragment list? Could you post your benchamrks.
And your coding style is broken noticebly...
Also do not check for every possible error case, negative return value always meant error, otherwise it is ok in your case to proceed.
Quoted 2 lines
--
Ashwini Kulkarni
Signature · 1 line
Evgeniy Polyakov
How is this different than just having the application mmap() the file and recv() into that buffer?
Quoted 81 lines · Ashwini Kulkarni
My name is Ashwini Kulkarni and I have been working at Intel Corporation for the past 4 months as an engineering intern. I have been working on the 'TCP socket splice' project with Chris Leech. This is a work-in-progress version of the project with scope for further modifications.
TCP socket splicing:
It allows a TCP socket to be spliced to a file via a pipe buffer. First, to splice data from a socket to a pipe buffer, upto 16 source pages(s) are pulled into the pipe buffer. Then to splice data from the pipe buffer to a file, those pages are migrated into the address space of the target file. It takes place entirely within the kernel and thus results in zero memory copies. It is the receive side complement tosendfile()but unlikesendfile()it is possible to splice from a socket as well and not just to a socket.Current Method:
+ > Application Buffer +
| |
_________________|_______________________|_____________
| |
Receive or | | Write
I/OAT DMA | |
| |
| V
Network File System
Buffer Buffer
^ |
| |
_________________|_______________________|_____________
DMA | | DMA
| |
Hardware | |
| V
NIC SATAIn the current method, the packet is DMA’d from the NIC into the network buffer. There is a read on socket to the user space and the packet data is copied from the network buffer to the application buffer. A write operation then moves the data from the application buffer to the file system buffer which is then DMA'd to the disk again. Thus, in the current method there will be one full copy of all the data to the user space.
Using TCP socket splice:
Application Control
|
_________________|__________________________________
|
| TCP socket splice
| +---------------------+
| | Direct path |
V | V
Network File System
Buffer Buffer
^ |
| |
_________________|_______________________|__________
DMA | | DMA
| |
Hardware | |
| V
NIC SATAIn this method, the objective is to use TCP socket splicing to create a direct path in the kernel from the network buffer to the file system buffer via a pipe buffer. The pages will migrate from the network buffer (which is associated with the socket) into the pipe buffer for an optimized path. From the pipe buffer, the pages will then be migrated to the output file address space page cache. This will enable to create a LAN to file-system API which will avoid the memcpy operations in user space and thus create a fast path from the network buffer to the storage buffer.
Open Issues (currently being addressed):
There is a performance drop when transferring bigger files (usually larger than 65536 bytes in size). Performance drop increases with the size of the file. Work is in progress to identify the source of this issue.We encourage the community to review our TCP socket splice project. Feedback would be greatly appreciated.
--
Ashwini Kulkarni