Skip to content

RFC TCP socket splice

Ashwini Kulkarni lkml netdev 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest6 patches 2 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/6Make splice_to_pipe non-static and move structure definitions to a header filefs/splice.c include/linux/pipe_fs_i.h+19 -170 replies
2/6Make sock_def_wakeup non-staticinclude/net/sock.h net/core/sock.c+3 -10 replies
3/6Add in TCP related part of splice read to ipv4net/ipv4/af_inet.c net/ipv4/tcp.c+136 -00 replies
4/6Add TCP socket splicing (tcp_splice_read) supportfs/splice.c include/linux/net.h include/linux/pipe_fs_i.h +2 more+35 -00 replies
5/6Add skb_splice_bits to skbuff.cinclude/linux/skbuff.h net/core/skbuff.c+139 -00 replies
6/6Move i_size_read part from do_splice_to() to __generic_file_splice_read() in splice.cfs/splice.c+8 -100 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC 0/6] TCP socket splice

My name is Ashwini Kulkarni and I have been working at Intel Corporation for the past 4 months as an engineering intern. I have been working on the 'TCP socket splice' project with Chris Leech. This is a work-in-progress version of the project with scope for further modifications.

TCP socket splicing:
It allows a TCP socket to be spliced to a file via a pipe buffer. First, to splice data from a socket to a pipe buffer, upto 16 source pages(s) are pulled into the pipe buffer. Then to splice data from the pipe buffer to a file, those pages are migrated into the address space of the target file. It takes place entirely within the kernel and thus results in zero memory copies. It is the receive side complement to sendfile() but unlike sendfile() it is possible to splice from a socket as well and not just to a socket.

Current Method:
+ > Application Buffer +
| |
_________________|_______________________|_____________
| |
Receive or | | Write
I/OAT DMA | |
| |
| V
Network File System
Buffer Buffer
^ |
| |
_________________|_______________________|_____________
DMA | | DMA
| |
Hardware | |
| V
NIC SATA

In the current method, the packet is DMA’d from the NIC into the network buffer. There is a read on socket to the user space and the packet data is copied from the network buffer to the application buffer. A write operation then moves the data from the application buffer to the file system buffer which is then DMA'd to the disk again. Thus, in the current method there will be one full copy of all the data to the user space.

Using TCP socket splice:

Application Control
|
_________________|__________________________________
|
| TCP socket splice
| +---------------------+
| | Direct path |
V | V
Network File System
Buffer Buffer
^ |
| |
_________________|_______________________|__________
DMA | | DMA
| |
Hardware | |
| V
NIC SATA

In this method, the objective is to use TCP socket splicing to create a direct path in the kernel from the network buffer to the file system buffer via a pipe buffer. The pages will migrate from the network buffer (which is associated with the socket) into the pipe buffer for an optimized path. From the pipe buffer, the pages will then be migrated to the output file address space page cache. This will enable to create a LAN to file-system API which will avoid the memcpy operations in user space and thus create a fast path from the network buffer to the storage buffer.

Open Issues (currently being addressed):
There is a performance drop when transferring bigger files (usually larger than 65536 bytes in size). Performance drop increases with the size of the file. Work is in progress to identify the source of this issue.

We encourage the community to review our TCP socket splice project. Feedback would be greatly appreciated.

--
Ashwini Kulkarni