splice, net: Switch over users of sendpage() and remove it
Subsystems (51)
David Howells
maintainercover splice, net: Switch over users of sendpage() and remove it
Here's the final set of patches towards the removal of sendpage. All the drivers that use sendpage() get switched over to using sendmsg() with MSG_SPLICE_PAGES.
skb_splice_from_iter() is given the facility to copy slab data() into fragments - or to coalesce them (in future) with other unspliced buffers in the target skbuff. This means that the caller can work the same way, no matter whether MSG_SPLICE_PAGES is supplied or not and no matter if the protocol just ignores it. If MSG_SPLICE_PAGES is not supplied or if it is just ignored, the data will get copied as normal rather than being spliced.
For the moment, skb_splice_from_iter() is equipped with its own fragment allocator - one that has percpu pages to allocate from to deal with parallel callers but that can also drop the percpu lock around calling the page allocator.
The following changes are made:
(1) Introduce an SMP-safe shared fragment allocator and make skb_splice_from_iter() use it. The allocator is exported so that ocfs2 can use it.
This now doesn't alter the existing page_frag_cache allocator.
(2) Expose information from the allocator in /proc/. This is useful for debugging it, but could be dropped.
(3) Make the protocol drivers behave according to MSG_MORE, not MSG_SENDPAGE_NOTLAST. The latter is restricted to turning on MSG_MORE in the sendpage() wrappers.
(4) Make siw, ceph/rds, skb_send_sock, dlm, nvme, smc, ocfs2, drbd and iscsi use sendmsg(), not sendpage and make them specify MSG_MORE instead of MSG_SENDPAGE_NOTLAST.
ocfs2 now allocates fragments for a couple of cases where it would otherwise pass in a pointer to shared data that doesn't seem to sufficient locking.
(5) Make drbd coalesce its entire message into a single sendmsg().
(6) Kill off sendpage and clean up MSG_SENDPAGE_NOTLAST.
I've pushed the patches here also:
https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git/log/?h=sendpage-3-frag
David
David Howells (17):
net: Copy slab data for sendmsg(MSG_SPLICE_PAGES)
net: Display info about MSG_SPLICE_PAGES memory handling in proc tcp_bpf, smc, tls, espintcp: Reduce MSG_SENDPAGE_NOTLAST usage
siw: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage to transmit
ceph: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage
net: Use sendmsg(MSG_SPLICE_PAGES) not sendpage in skb_send_sock()
ceph: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage()
rds: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage
dlm: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage
nvme: Use sendmsg(MSG_SPLICE_PAGES) rather then sendpage
smc: Drop smc_sendpage() in favour of smc_sendmsg() + MSG_SPLICE_PAGES
ocfs2: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage()
drbd: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage()
drdb: Send an entire bio in a single sendmsg
iscsi: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage
sock: Remove ->sendpage*() in favour of sendmsg(MSG_SPLICE_PAGES)
net: Kill MSG_SENDPAGE_NOTLAST
Documentation/bpf/map_sockmap.rst | 10 +- Documentation/filesystems/locking.rst | 2 - Documentation/filesystems/vfs.rst | 1 - Documentation/networking/scaling.rst | 4 +- crypto/af_alg.c | 28 -- crypto/algif_aead.c | 22 +- crypto/algif_rng.c | 2 - crypto/algif_skcipher.c | 14 - drivers/block/drbd/drbd_main.c | 86 ++---- drivers/infiniband/sw/siw/siw_qp_tx.c | 230 +++------------- .../chelsio/inline_crypto/chtls/chtls.h | 2 - .../chelsio/inline_crypto/chtls/chtls_io.c | 14 - .../chelsio/inline_crypto/chtls/chtls_main.c | 1 - drivers/nvme/host/tcp.c | 44 +-- drivers/nvme/target/tcp.c | 46 ++-- drivers/scsi/iscsi_tcp.c | 26 +- drivers/scsi/iscsi_tcp.h | 2 +- drivers/target/iscsi/iscsi_target_util.c | 14 +- fs/dlm/lowcomms.c | 10 +- fs/nfsd/vfs.c | 2 +- fs/ocfs2/cluster/tcp.c | 107 ++++---- include/crypto/if_alg.h | 2 - include/linux/net.h | 8 - include/linux/skbuff.h | 5 + include/linux/socket.h | 4 +- include/net/inet_common.h | 2 - include/net/sock.h | 6 - include/net/tcp.h | 4 - net/appletalk/ddp.c | 1 - net/atm/pvc.c | 1 - net/atm/svc.c | 1 - net/ax25/af_ax25.c | 1 - net/caif/caif_socket.c | 2 - net/can/bcm.c | 1 - net/can/isotp.c | 1 - net/can/j1939/socket.c | 1 - net/can/raw.c | 1 - net/ceph/messenger_v1.c | 58 ++-- net/ceph/messenger_v2.c | 91 ++----- net/core/skbuff.c | 257 ++++++++++++++++-- net/core/sock.c | 35 +-- net/dccp/ipv4.c | 1 - net/dccp/ipv6.c | 1 - net/ieee802154/socket.c | 2 - net/ipv4/af_inet.c | 21 -- net/ipv4/tcp.c | 43 +-- net/ipv4/tcp_bpf.c | 30 +- net/ipv4/tcp_ipv4.c | 1 - net/ipv4/udp.c | 15 - net/ipv4/udp_impl.h | 2 - net/ipv4/udplite.c | 1 - net/ipv6/af_inet6.c | 3 - net/ipv6/raw.c | 1 - net/ipv6/tcp_ipv6.c | 1 - net/kcm/kcmsock.c | 20 -- net/key/af_key.c | 1 - net/l2tp/l2tp_ip.c | 1 - net/l2tp/l2tp_ip6.c | 1 - net/llc/af_llc.c | 1 - net/mctp/af_mctp.c | 1 - net/mptcp/protocol.c | 2 - net/netlink/af_netlink.c | 1 - net/netrom/af_netrom.c | 1 - net/packet/af_packet.c | 2 - net/phonet/socket.c | 2 - net/qrtr/af_qrtr.c | 1 - net/rds/af_rds.c | 1 - net/rds/tcp_send.c | 74 ++--- net/rose/af_rose.c | 1 - net/rxrpc/af_rxrpc.c | 1 - net/sctp/protocol.c | 1 - net/smc/af_smc.c | 29 -- net/smc/smc_stats.c | 2 +- net/smc/smc_stats.h | 1 - net/smc/smc_tx.c | 20 +- net/smc/smc_tx.h | 2 - net/socket.c | 48 ---- net/tipc/socket.c | 3 - net/tls/tls.h | 6 - net/tls/tls_device.c | 24 +- net/tls/tls_main.c | 9 +- net/tls/tls_sw.c | 37 +-- net/unix/af_unix.c | 19 -- net/vmw_vsock/af_vsock.c | 3 - net/x25/af_x25.c | 1 - net/xdp/xsk.c | 1 - net/xfrm/espintcp.c | 10 +- .../perf/trace/beauty/include/linux/socket.h | 1 - tools/perf/trace/beauty/msg_flags.c | 3 - 89 files changed, 539 insertions(+), 1063 deletions(-)
David Howells
maintainerPATCH 1/17 net: Copy slab data for sendmsg(MSG_SPLICE_PAGES)
If sendmsg() is passed MSG_SPLICE_PAGES and is given a buffer that contains some data that's resident in the slab, copy it rather than returning EIO. This can be made use of by a number of drivers in the kernel, including:
iwarp, ceph/rds, dlm, nvme, ocfs2, drdb. It could also be used by iscsi, rxrpc, sunrpc, cifs and probably others.
skb_splice_from_iter() is given it's own fragment allocator as page_frag_alloc_align() can't be used because it does no locking to prevent parallel callers from racing. alloc_skb_frag() uses a separate folio for each cpu and locks to the cpu whilst allocating, reenabling cpu migration around folio allocation.
This could allocate a whole page instead for each fragment to be copied, as alloc_skb_with_frags() would do instead, but that would waste a lot of space (most of the fragments look like they're going to be small).
This allows an entire message that consists of, say, a protocol header or two, a number of pages of data and a protocol footer to be sent using a single call to sock_sendmsg().
The callers could be made to copy the data into fragments before calling sendmsg(), but that then penalises them if MSG_SPLICE_PAGES gets ignored.
--- include/linux/skbuff.h | 5 ++ net/core/skbuff.c | 172 ++++++++++++++++++++++++++++++++++++++++- 2 files changed, 174 insertions(+), 3 deletions(-)
include/linux/skbuff.h +5 -0
@@ -5037,6 +5037,11 @@ static inline void skb_mark_for_recycle(struct sk_buff *skb)#endif } +void *alloc_skb_frag (size_t fragsz,gfp_t gfp); +void *copy_skb_frag (const void *s,size_t len,gfp_t gfp); +ssize_t skb_splice_from_iter (struct sk_buff *skb,struct iov_iter *iter, +ssize_t maxsize,gfp_t gfp); +ssize_t skb_splice_from_iter (struct sk_buff *skb,struct iov_iter *iter,ssize_t maxsize,gfp_t gfp);
net/core/skbuff.c +169 -3
192 lines not on this page · raw message
Re: [PATCH net-next 01/17] net: Copy slab data for sendmsg(MSG_SPLICE_PAGES)
On Fri, Jun 16, 2023 at 05:12:44PM +0100, David Howells wrote:
...
Quoted 19 lines
net/core/skbuff.c +19
-0+/** +* alloc_skb_frag - Allocate a page fragment for using in a socket +* @fragsz: The size of fragment required +* @gfp: Allocation flags +*/ +void *alloc_skb_frag (size_t fragsz,gfp_t gfp) +{ +struct skb_splice_frag_cache *cache; +struct folio *folio, *spare =NULL ; +size_t offset, fsize; +void *p; + +if (WARN_ON_ONCE (fragsz ==0 )) + fragsz =1 ; + + cache =get_cpu_ptr (&skb_splice_frag_cache); +reload: + folio = cache->folio; + offset = cache->offset;
net/core/skbuff.c nodefer: __kfree_skb(skb); · line 6799
Quoted 10 lines
net/core/skbuff.c +10
-0+try_again: +if (fragsz > offset) + goto insufficient_space; + +/* Make the allocation. */ + cache->pagecnt_bias--; + offset =ALIGN_DOWN (offset - fragsz,SMP_CACHE_BYTES ); + cache->offset = offset; + p = cache->virt + offset; +put_cpu_ptr (skb_splice_frag_cache);
Hi David,
I don't think it makes any difference at run-time.
But to keep Sparse happy, perhaps this ought to be put_cpu_var()
...
David Howells
maintainerRe: [PATCH net-next 01/17] net: Copy slab data for sendmsg(MSG_SPLICE_PAGES)
Quoted 1 line · Simon Horman
Quoted 1 line
(quoted diff) +1
-0+ cache = get_cpu_ptr(&skb_splice_frag_cache);
...
Quoted 6 lines
net/core/skbuff.c nodefer: __kfree_skb(skb); · line 6799
Quoted 1 line
net/core/skbuff.c +1
-0+put_cpu_ptr (skb_splice_frag_cache);Hi David,
I don't think it makes any difference at run-time.
But to keep Sparse happy, perhaps this ought to beput_cpu_var()
Actually, the problem is a missing "&". I think I should use put_cpu_ptr() to match get_cpu_ptr(). It doesn't crash because the argument is ignored.
David
Re: [PATCH net-next 01/17] net: Copy slab data for sendmsg(MSG_SPLICE_PAGES)
Quoted 8 lines · David Howells
Quoted 1 line · Simon Horman
Quoted 1 line
(quoted diff) +1
-0+ cache = get_cpu_ptr(&skb_splice_frag_cache);...
Quoted 4 lines
net/core/skbuff.c nodefer: __kfree_skb(skb); · line 6799
Quoted 1 line
net/core/skbuff.c +1
-0+put_cpu_ptr (skb_splice_frag_cache);Hi David,
Quoted 5 lines
Quoted 2 lines
I don't think it makes any difference at run-time.
But to keep Sparse happy, perhaps this ought to beput_cpu_var()Actually, the problem is a missing "&". I think I should use
put_cpu_ptr()to matchget_cpu_ptr(). It doesn't crash because the argument is ignored.
Thanks David, I agree that is a better idea.
David Howells
maintainerPATCH 2/17 net: Display info about MSG_SPLICE_PAGES memory handling in proc
In reply to David HowellsDisplay information about the memory handling MSG_SPLICE_PAGES does to copy slabbed data into page fragments.
For each CPU that has a cached folio, it displays the folio pfn, the offset pointer within the folio and the size of the folio.
It also displays the number of pages refurbished and the number of pages replaced.
--- net/core/skbuff.c | 42 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 39 insertions(+), 3 deletions(-)
net/core/skbuff.c +39 -3
@@ -83,6 +83,7 @@#include <linux/user_namespace.h> #include <linux/indirect_call_wrapper.h> #include <linux/textsearch.h> +#include <linux/proc_fs.h> #include "dev.h" #include "sock_destructor.h"
@@ -6758,6 +6759,7 @@ nodefer: __kfree_skb(skb);struct skb_splice_frag_cache {struct folio *folio;void *virt; +unsigned int fsize;unsigned int offset;/* we maintain a pagecount bias, so that we dont dirty cache line * containing page->_refcount every time we allocate a fragment.
@@ -6767,6 +6769,26 @@ struct skb_splice_frag_cache { }; static DEFINE_PER_CPU(struct skb_splice_frag_cache, skb_splice_frag_cache); +static atomic_t skb_splice_frag_replaced, skb_splice_frag_refurbished; + +static int skb_splice_show (struct seq_file *m,void *data) +{ +int cpu; + +seq_printf (m,"refurb=%u repl=%u\n" , +atomic_read (&skb_splice_frag_refurbished), +atomic_read (&skb_splice_frag_replaced)); + +for_each_possible_cpu (cpu) { +const struct skb_splice_frag_cache *cache = +per_cpu_ptr (&skb_splice_frag_cache, cpu); + +seq_printf (m,"[%u] %lx %u/%u\n" , + cpu,folio_pfn (cache->folio), + cache->offset, cache->fsize); + } +return 0 ; +}/** * alloc_skb_frag - Allocate a page fragment for using in a socket
@@ -6803,17 +6825,21 @@ void *alloc_skb_frag(size_t fragsz, gfp_t gfp) insufficient_space:/* See if we can refurbish the current folio. */ -if (!folio || !folio_ref_sub_and_test (folio, cache->pagecnt_bias)) +if (!folio) goto get_new_folio; +if (!folio_ref_sub_and_test (folio, cache->pagecnt_bias)) + goto replace_folio;if (unlikely (cache->pfmemalloc)) {__folio_put (folio); - goto get_new_folio; + goto replace_folio; } fsize =folio_size (folio);if (unlikely (fragsz > fsize)) goto frag_too_big; +atomic_inc (&skb_splice_frag_refurbished); +/* OK, page count is 0, we can safely set it */ folio_set_count (folio,PAGE_FRAG_CACHE_MAX_SIZE +1 );
@@ -6822,6 +6848,8 @@ void *alloc_skb_frag(size_t fragsz, gfp_t gfp) offset = fsize; goto try_again; +replace_folio: +atomic_inc (&skb_splice_frag_replaced); get_new_folio:if (!spare) { cache->folio =NULL ;
@@ -6848,6 +6876,7 @@ void *alloc_skb_frag(size_t fragsz, gfp_t gfp) cache->folio = spare; cache->virt =folio_address (spare); + cache->fsize =folio_size (spare); folio = spare; spare =NULL ;
@@ -6858,7 +6887,7 @@ void *alloc_skb_frag(size_t fragsz, gfp_t gfp)/* Reset page count bias and offset to start of new frag */ cache->pagecnt_bias =PAGE_FRAG_CACHE_MAX_SIZE +1 ; - offset =folio_size (folio); + offset = cache->fsize; goto try_again; frag_too_big:
@@ -7008,3 +7037,10 @@ ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter,return spliced ?: ret; }EXPORT_SYMBOL (skb_splice_from_iter); + +static int skb_splice_init (void ) +{ +proc_create_single ("pagefrags" ,S_IFREG |0444 ,NULL , &skb_splice_show); +return 0 ; +} +late_initcall (skb_splice_init);
David Howells
maintainerPATCH 3/17 tcp_bpf, smc, tls, espintcp: Reduce MSG_SENDPAGE_NOTLAST usage
In reply to David HowellsAs MSG_SENDPAGE_NOTLAST is being phased out along with sendpage(), don't use it further in than the sendpage methods, but rather translate it to MSG_MORE and use that instead.
--- net/ipv4/tcp_bpf.c | 3 --- net/smc/smc_tx.c | 6 ++++-- net/tls/tls_device.c | 4 ++-- net/xfrm/espintcp.c | 10 ++++++---- 4 files changed, 12 insertions(+), 11 deletions(-)
net/ipv4/tcp_bpf.c +0 -3
@@ -111,9 +111,6 @@ static int tcp_bpf_push(struct sock *sk, struct sk_msg *msg, u32 apply_bytes,if (has_tx_ulp) msghdr.msg_flags |=MSG_SENDPAGE_NOPOLICY ; -if (flags &MSG_SENDPAGE_NOTLAST ) - msghdr.msg_flags |=MSG_MORE ; -bvec_set_page (&bvec, page, size, off);iov_iter_bvec (&msghdr.msg_iter,ITER_SOURCE , &bvec,1 , size); ret =tcp_sendmsg_locked (sk, &msghdr, size);
net/smc/smc_tx.c +4 -2
@@ -168,8 +168,7 @@ static bool smc_tx_should_cork(struct smc_sock *smc, struct msghdr *msg)* should known how/when to uncork it. */ if ((msg->msg_flags &MSG_MORE || -smc_tx_is_corked (smc) || - msg->msg_flags &MSG_SENDPAGE_NOTLAST ) && +smc_tx_is_corked (smc)) &&atomic_read (&conn->sndbuf_space))return true;
@@ -306,6 +305,9 @@ int smc_tx_sendpage(struct smc_sock *smc, struct page *page, int offset,struct kvec iov;int rc; +if (flags &MSG_SENDPAGE_NOTLAST ) + msg.msg_flags |=MSG_MORE ; + iov.iov_base = kaddr + offset; iov.iov_len = size;iov_iter_kvec (&msg.msg_iter,ITER_SOURCE , &iov,1 , size);
net/tls/tls_device.c +2 -2
@@ -449,7 +449,7 @@ static int tls_push_data(struct sock *sk,return -sk->sk_err; flags |=MSG_SENDPAGE_DECRYPTED ; - tls_push_record_flags = flags |MSG_SENDPAGE_NOTLAST ; + tls_push_record_flags = flags |MSG_MORE ; timeo =sock_sndtimeo (sk, flags &MSG_DONTWAIT );if (tls_is_partially_sent_record (tls_ctx)) {
@@ -532,7 +532,7 @@ static int tls_push_data(struct sock *sk,if (!size) { last_record: tls_push_record_flags = flags; -if (flags & (MSG_SENDPAGE_NOTLAST |MSG_MORE )) { +if (flags &MSG_MORE ) { more = true;break ; }
net/xfrm/espintcp.c +6 -4
@@ -205,13 +205,15 @@ static int espintcp_sendskb_locked(struct sock *sk, struct espintcp_msg *emsg,static int espintcp_sendskmsg_locked (struct sock *sk,struct espintcp_msg *emsg,int flags) { -struct msghdr msghdr = { .msg_flags = flags |MSG_SPLICE_PAGES , }; +struct msghdr msghdr = { + .msg_flags = flags |MSG_SPLICE_PAGES |MSG_MORE , + };struct sk_msg *skmsg = &emsg->skmsg; +bool more = flags &MSG_MORE ;struct scatterlist *sg;int done =0 ;int ret; - msghdr.msg_flags |=MSG_SENDPAGE_NOTLAST ; sg = &skmsg->sg.data[skmsg->sg.start]; do { struct bio_vec bvec;
@@ -221,8 +223,8 @@ static int espintcp_sendskmsg_locked(struct sock *sk, emsg->offset =0 ; -if (sg_is_last (sg)) - msghdr.msg_flags &= ~MSG_SENDPAGE_NOTLAST ; +if (sg_is_last (sg) && !more) + msghdr.msg_flags &= ~MSG_MORE ; p =sg_page (sg); retry:
David Howells
maintainerPATCH 4/17 siw: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage to transmit
In reply to David HowellsWhen transmitting data, call down into TCP using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather than performing several sendmsg and sendpage calls to transmit header, data pages and trailer.
To make this work, the data is assembled in a bio_vec array and attached to a BVEC-type iterator. The header and trailer (if present) are copied into page fragments that can be freed with put_page().
--- drivers/infiniband/sw/siw/siw_qp_tx.c | 230 ++++---------------------- 1 file changed, 35 insertions(+), 195 deletions(-)
drivers/infiniband/sw/siw/siw_qp_tx.c +35 -195
@@ -311,114 +311,8 @@ static int siw_tx_ctrl(struct siw_iwarp_tx *c_tx, struct socket *s,return rv; } -/* -* 0copy TCP transmit interface: Use MSG_SPLICE_PAGES. -* -* Using sendpage to push page by page appears to be less efficient -* than using sendmsg, even if data are copied. -* -* A general performance limitation might be the extra four bytes -* trailer checksum segment to be pushed after user data. -*/ -static int siw_tcp_sendpages (struct socket *s,struct page **page,int offset, -size_t size) -{ -struct bio_vec bvec; -struct msghdr msg = { - .msg_flags = (MSG_MORE |MSG_DONTWAIT |MSG_SENDPAGE_NOTLAST | -MSG_SPLICE_PAGES ), - }; -struct sock *sk = s->sk; -int i =0 , rv =0 , sent =0 ; - -while (size) { -size_t bytes =min_t (size_t,PAGE_SIZE - offset, size); - -if (size + offset <=PAGE_SIZE ) - msg.msg_flags &= ~MSG_SENDPAGE_NOTLAST ; - -tcp_rate_check_app_limited (sk); -bvec_set_page (&bvec, page[i], bytes, offset); -iov_iter_bvec (&msg.msg_iter,ITER_SOURCE , &bvec,1 , size); - -try_page_again: -lock_sock (sk); - rv =tcp_sendmsg_locked (sk, &msg, size); -release_sock (sk); - -if (rv >0 ) { - size -= rv; - sent += rv; -if (rv != bytes) { - offset += rv; - bytes -= rv; - goto try_page_again; - } - offset =0 ; - }else { -if (rv == -EAGAIN || rv ==0 ) -break ; -return rv; - } - i++; - } -return sent; -} - -/* -* siw_0copy_tx() -* -* Pushes list of pages to TCP socket. If pages from multiple -* SGE's, all referenced pages of each SGE are pushed in one -* shot. -*/ -static int siw_0copy_tx (struct socket *s,struct page **page, -struct siw_sge *sge,unsigned int offset, -unsigned int size) -{ -int i =0 , sent =0 , rv; -int sge_bytes =min (sge->length - offset, size); - - offset = (sge->laddr + offset) & ~PAGE_MASK ; - -while (sent != size) { - rv =siw_tcp_sendpages (s, &page[i], offset, sge_bytes); -if (rv >=0 ) { - sent += rv; -if (size == sent || sge_bytes > rv) -break ; - - i +=PAGE_ALIGN (sge_bytes + offset) >>PAGE_SHIFT ; - sge++; - sge_bytes =min (sge->length, size - sent); - offset = sge->laddr & ~PAGE_MASK ; - }else { - sent = rv; -break ; - } - } -return sent; -} -#define MAX_TRAILER (MPA_CRC_SIZE + 4) -static void siw_unmap_pages (struct kvec *iov,unsigned long kmap_mask,int len) -{ -int i; - -/* -* Work backwards through the array to honor the kmap_local_page() -* ordering requirements. -*/ -for (i = (len-1 ); i >=0 ; i--) { -if (kmap_mask &BIT (i)) { -unsigned long addr = (unsigned long )iov[i].iov_base; - -kunmap_local ((void *)(addr &PAGE_MASK )); - } - } -} -/* * siw_tx_hdt() tries to push a complete packet to TCP where all * packet fragments are referenced by the elements of one iovec.
@@ -438,30 +332,21 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) {struct siw_wqe *wqe = &c_tx->wqe_active;struct siw_sge *sge = &wqe->sqe.sge[c_tx->sge_idx]; -struct kvec iov[MAX_ARRAY ]; -struct page *page_array[MAX_ARRAY ]; +struct bio_vec bvec[MAX_ARRAY ];struct msghdr msg = { .msg_flags =MSG_DONTWAIT |MSG_EOR }; +void *trl;int seg =0 , do_crc = c_tx->do_crc, is_kva =0 , rv;unsigned int data_len = c_tx->bytes_unsent, hdr_len =0 , trl_len =0 , sge_off = c_tx->sge_off, sge_idx = c_tx->sge_idx, pbl_idx = c_tx->pbl_idx; -unsigned long kmap_mask =0L ;if (c_tx->state ==SIW_SEND_HDR ) { -if (c_tx->use_sendpage) { - rv =siw_tx_ctrl (c_tx, s,MSG_DONTWAIT |MSG_MORE ); -if (rv) - goto done; +void *hdr = &c_tx->pkt.ctrl + c_tx->ctrl_sent; - c_tx->state =SIW_SEND_DATA ; - }else { - iov[0 ].iov_base = - (char *)&c_tx->pkt.ctrl + c_tx->ctrl_sent; - iov[0 ].iov_len = hdr_len = - c_tx->ctrl_len - c_tx->ctrl_sent; - seg =1 ; - } + hdr_len = c_tx->ctrl_len - c_tx->ctrl_sent; +bvec_set_virt (&bvec[0 ], hdr, hdr_len); + seg =1 ; } wqe->processed += data_len;
@@ -477,28 +362,9 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) }else { is_kva =1 ; } -if (is_kva && !c_tx->use_sendpage) { -/* -* tx from kernel virtual address: either inline data -* or memory region with assigned kernel buffer -*/ - iov[seg].iov_base = -ib_virt_dma_to_ptr (sge->laddr + sge_off); - iov[seg].iov_len = sge_len; - -if (do_crc) -crypto_shash_update (c_tx->mpa_crc_hd, - iov[seg].iov_base, - sge_len); - sge_off += sge_len; - data_len -= sge_len; - seg++; - goto sge_done; - }while (sge_len) {size_t plen =min ((int )PAGE_SIZE - fp_off, sge_len); -void *kaddr;if (!is_kva) {struct page *p;
@@ -511,33 +377,12 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) p =siw_get_upage (mem->umem, sge->laddr + sge_off);if (unlikely (!p)) { -siw_unmap_pages (iov, kmap_mask, seg); wqe->processed -= c_tx->bytes_unsent; rv = -EFAULT ; goto done_crc; } - page_array[seg] = p; - -if (!c_tx->use_sendpage) { -void *kaddr =kmap_local_page (p); - -/* Remember for later kunmap() */ - kmap_mask |=BIT (seg); - iov[seg].iov_base = kaddr + fp_off; - iov[seg].iov_len = plen; - -if (do_crc) -crypto_shash_update ( - c_tx->mpa_crc_hd, - iov[seg].iov_base, - plen); - }else if (do_crc) { - kaddr =kmap_local_page (p); -crypto_shash_update (c_tx->mpa_crc_hd, - kaddr + fp_off, - plen); -kunmap_local (kaddr); - } + +bvec_set_page (&bvec[seg], p, plen, fp_off); }else {/* * Cast to an uintptr_t to preserve all 64 bits
@@ -545,12 +390,16 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s)*/ u64 va = sge->laddr + sge_off; - page_array[seg] =ib_virt_dma_to_page (va); -if (do_crc) -crypto_shash_update ( - c_tx->mpa_crc_hd, -ib_virt_dma_to_ptr (va), - plen); +bvec_set_virt (&bvec[seg],ib_virt_dma_to_ptr (va), plen); + } + +if (do_crc) { +void *kaddr = +kmap_local_page (bvec[seg].bv_page); +crypto_shash_update (c_tx->mpa_crc_hd, + kaddr + bvec[seg].bv_offset, + bvec[seg].bv_len); +kunmap_local (kaddr); } sge_len -= plen;
@@ -560,13 +409,12 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s)if (++seg >= (int )MAX_ARRAY ) {siw_dbg_qp (tx_qp (c_tx),"to many fragments\n" ); -siw_unmap_pages (iov, kmap_mask, seg-1 ); wqe->processed -= c_tx->bytes_unsent; rv = -EMSGSIZE ; goto done_crc; } } -sge_done: +/* Update SGE variables at end of SGE */ if (sge_off == sge->length && (data_len !=0 || wqe->processed < wqe->bytes)) {
@@ -575,15 +423,8 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) sge_off =0 ; } } -/* trailer */ -if (likely (c_tx->state !=SIW_SEND_TRAILER )) { - iov[seg].iov_base = &c_tx->trailer.pad[4 - c_tx->pad]; - iov[seg].iov_len = trl_len =MAX_TRAILER - (4 - c_tx->pad); - }else { - iov[seg].iov_base = &c_tx->trailer.pad[c_tx->ctrl_sent]; - iov[seg].iov_len = trl_len =MAX_TRAILER - c_tx->ctrl_sent; - } +/* Set the CRC in the trailer */ if (c_tx->pad) { *(u32 *)c_tx->trailer.pad =0 ;if (do_crc)
@@ -596,23 +437,23 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s)else if (do_crc)crypto_shash_final (c_tx->mpa_crc_hd, (u8 *)&c_tx->trailer.crc); - data_len = c_tx->bytes_unsent; - -if (c_tx->use_sendpage) { - rv =siw_0copy_tx (s, page_array, &wqe->sqe.sge[c_tx->sge_idx], - c_tx->sge_off, data_len); -if (rv == data_len) { - rv =kernel_sendmsg (s, &msg, &iov[seg],1 , trl_len); -if (rv >0 ) - rv += data_len; -else - rv = data_len; - } +/* Copy the trailer and add it to the output list */ +if (likely (c_tx->state !=SIW_SEND_TRAILER )) { + trl = &c_tx->trailer.pad[4 - c_tx->pad]; + trl_len =MAX_TRAILER - (4 - c_tx->pad); }else { - rv =kernel_sendmsg (s, &msg, iov, seg +1 , - hdr_len + data_len + trl_len); -siw_unmap_pages (iov, kmap_mask, seg); + trl = &c_tx->trailer.pad[c_tx->ctrl_sent]; + trl_len =MAX_TRAILER - c_tx->ctrl_sent; } +bvec_set_virt (&bvec[seg], trl, trl_len); + + data_len = c_tx->bytes_unsent; + +if (c_tx->use_sendpage) + msg.msg_flags |=MSG_SPLICE_PAGES ; +iov_iter_bvec (&msg.msg_iter,ITER_SOURCE , bvec, seg +1 , + hdr_len + data_len + trl_len); + rv =sock_sendmsg (s, &msg);if (rv < (int )hdr_len) {/* Not even complete hdr pushed or negative rv */ wqe->processed -= data_len;
@@ -673,7 +514,6 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) } done_crc: c_tx->do_crc =0 ; -done:return rv; }
David Howells
maintainerPATCH 5/17 ceph: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage
In reply to David HowellsUse sendmsg() and MSG_SPLICE_PAGES rather than sendpage in ceph when transmitting data. For the moment, this can only transmit one page at a time because of the architecture of net/ceph/, but if write_partial_message_data() can be given a bvec[] at a time by the iteration code, this would allow pages to be sent in a batch.
--- net/ceph/messenger_v1.c | 58 ++++++++++++++--------------------------- 1 file changed, 19 insertions(+), 39 deletions(-)
net/ceph/messenger_v1.c +19 -39
@@ -74,37 +74,6 @@ static int ceph_tcp_sendmsg(struct socket *sock, struct kvec *iov,return r; } -/* -* @more: either or both of MSG_MORE and MSG_SENDPAGE_NOTLAST -*/ -static int ceph_tcp_sendpage (struct socket *sock,struct page *page, -int offset,size_t size,int more) -{ -ssize_t (*sendpage)(struct socket *sock,struct page *page, -int offset,size_t size,int flags); -int flags =MSG_DONTWAIT |MSG_NOSIGNAL | more; -int ret; - -/* -* sendpage cannot properly handle pages with page_count == 0, -* we need to fall back to sendmsg if that's the case. -* -* Same goes for slab pages: skb_can_coalesce() allows -* coalescing neighboring slab objects into a single frag which -* triggers one of hardened usercopy checks. -*/ -if (sendpage_ok (page)) - sendpage = sock->ops->sendpage; -else - sendpage = sock_no_sendpage; - - ret =sendpage (sock, page, offset, size, flags); -if (ret == -EAGAIN ) - ret =0 ; - -return ret; -} -static void con_out_kvec_reset (struct ceph_connection *con) {BUG_ON (con->v1.out_skip);
@@ -464,7 +433,6 @@ static int write_partial_message_data(struct ceph_connection *con)struct ceph_msg *msg = con->out_msg;struct ceph_msg_data_cursor *cursor = &msg->cursor;bool do_datacrc = !ceph_test_opt (from_msgr (con->msgr),NOCRC ); -int more =MSG_MORE |MSG_SENDPAGE_NOTLAST ;u32 crc;dout ("%s %p msg %p\n" , __func__, con, msg);
@@ -482,6 +450,10 @@ static int write_partial_message_data(struct ceph_connection *con)*/ crc = do_datacrc ?le32_to_cpu (msg->footer.data_crc) :0 ;while (cursor->total_resid) { +struct bio_vec bvec; +struct msghdr msghdr = { + .msg_flags =MSG_SPLICE_PAGES , + };struct page *page;size_t page_offset;size_t length;
@@ -494,9 +466,12 @@ static int write_partial_message_data(struct ceph_connection *con) page =ceph_msg_data_next (cursor, &page_offset, &length);if (length == cursor->total_resid) - more =MSG_MORE ; - ret =ceph_tcp_sendpage (con->sock, page, page_offset, length, - more); + msghdr.msg_flags |=MSG_MORE ; + +bvec_set_page (&bvec, page, length, page_offset); +iov_iter_bvec (&msghdr.msg_iter,ITER_SOURCE , &bvec,1 , length); + + ret =sock_sendmsg (con->sock, &msghdr);if (ret <=0 ) {if (do_datacrc) msg->footer.data_crc =cpu_to_le32 (crc);
@@ -526,7 +501,10 @@ static int write_partial_message_data(struct ceph_connection *con)*/ static int write_partial_skip (struct ceph_connection *con) { -int more =MSG_MORE |MSG_SENDPAGE_NOTLAST ; +struct bio_vec bvec; +struct msghdr msghdr = { + .msg_flags =MSG_SPLICE_PAGES |MSG_MORE , + };int ret;dout ("%s %p %d left\n" , __func__, con, con->v1.out_skip);
@@ -534,9 +512,11 @@ static int write_partial_skip(struct ceph_connection *con)size_t size =min (con->v1.out_skip, (int )PAGE_SIZE );if (size == con->v1.out_skip) - more =MSG_MORE ; - ret =ceph_tcp_sendpage (con->sock, ceph_zero_page,0 , size, - more); + msghdr.msg_flags &= ~MSG_MORE ; +bvec_set_page (&bvec,ZERO_PAGE (0 ), size,0 ); +iov_iter_bvec (&msghdr.msg_iter,ITER_SOURCE , &bvec,1 , size); + + ret =sock_sendmsg (con->sock, &msghdr);if (ret <=0 ) goto out; con->v1.out_skip -= ret;
David Howells
maintainerPATCH 6/17 net: Use sendmsg(MSG_SPLICE_PAGES) not sendpage in skb_send_sock()
In reply to David HowellsUse sendmsg() with MSG_SPLICE_PAGES rather than sendpage in skb_send_sock(). This causes pages to be spliced from the source iterator if possible.
This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction.
Note that this could perhaps be improved to fill out a bvec array with all the frags and then make a single sendmsg call, possibly sticking the header on the front also.
--- net/core/skbuff.c | 49 ++++++++++++++++++++++++++--------------------- 1 file changed, 27 insertions(+), 22 deletions(-)
net/core/skbuff.c +27 -22
@@ -2990,32 +2990,32 @@ int skb_splice_bits(struct sk_buff *skb, struct sock *sk, unsigned int offset, }EXPORT_SYMBOL_GPL (skb_splice_bits); -static int sendmsg_unlocked (struct sock *sk,struct msghdr *msg, -struct kvec *vec,size_t num,size_t size) +static int sendmsg_locked (struct sock *sk,struct msghdr *msg) {struct socket *sock = sk->sk_socket; +size_t size =msg_data_left (msg);if (!sock)return -EINVAL ; -return kernel_sendmsg (sock, msg, vec, num, size); + +if (!sock->ops->sendmsg_locked) +return sock_no_sendmsg_locked (sk, msg, size); + +return sock->ops->sendmsg_locked (sk, msg, size); } -static int sendpage_unlocked (struct sock *sk,struct page *page,int offset, -size_t size,int flags) +static int sendmsg_unlocked (struct sock *sk,struct msghdr *msg) {struct socket *sock = sk->sk_socket;if (!sock)return -EINVAL ; -return kernel_sendpage (sock, page, offset, size, flags); +return sock_sendmsg (sock, msg); } -typedef int (*sendmsg_func )(struct sock *sk,struct msghdr *msg, -struct kvec *vec,size_t num,size_t size); -typedef int (*sendpage_func )(struct sock *sk,struct page *page,int offset, -size_t size,int flags); +typedef int (*sendmsg_func )(struct sock *sk,struct msghdr *msg);static int __skb_send_sock (struct sock *sk,struct sk_buff *skb,int offset, -int len,sendmsg_func sendmsg,sendpage_func sendpage) +int len,sendmsg_func sendmsg) {unsigned int orig_len = len;struct sk_buff *head = skb;
@@ -3035,8 +3035,9 @@ static int __skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset,memset (&msg,0 ,sizeof (msg)); msg.msg_flags =MSG_DONTWAIT ; - ret =INDIRECT_CALL_2 (sendmsg, kernel_sendmsg_locked, - sendmsg_unlocked, sk, &msg, &kv,1 , slen); +iov_iter_kvec (&msg.msg_iter,ITER_SOURCE , &kv,1 , slen); + ret =INDIRECT_CALL_2 (sendmsg, sendmsg_locked, + sendmsg_unlocked, sk, &msg);if (ret <=0 ) goto error;
@@ -3067,11 +3068,17 @@ static int __skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset, slen =min_t (size_t, len,skb_frag_size (frag) - offset);while (slen) { - ret =INDIRECT_CALL_2 (sendpage, kernel_sendpage_locked, - sendpage_unlocked, sk, -skb_frag_page (frag), -skb_frag_off (frag) + offset, - slen,MSG_DONTWAIT ); +struct bio_vec bvec; +struct msghdr msg = { + .msg_flags =MSG_SPLICE_PAGES |MSG_DONTWAIT , + }; + +bvec_set_page (&bvec,skb_frag_page (frag), slen, +skb_frag_off (frag) + offset); +iov_iter_bvec (&msg.msg_iter,ITER_SOURCE , &bvec,1 , slen); + + ret =INDIRECT_CALL_2 (sendmsg, sendmsg_locked, + sendmsg_unlocked, sk, &msg);if (ret <=0 ) goto error;
@@ -3108,16 +3115,14 @@ static int __skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset,int skb_send_sock_locked (struct sock *sk,struct sk_buff *skb,int offset,int len) { -return __skb_send_sock (sk, skb, offset, len, kernel_sendmsg_locked, - kernel_sendpage_locked); +return __skb_send_sock (sk, skb, offset, len, sendmsg_locked); }EXPORT_SYMBOL_GPL (skb_send_sock_locked);/* Send skb data on a socket. Socket must be unlocked. */ int skb_send_sock (struct sock *sk,struct sk_buff *skb,int offset,int len) { -return __skb_send_sock (sk, skb, offset, len, sendmsg_unlocked, - sendpage_unlocked); +return __skb_send_sock (sk, skb, offset, len, sendmsg_unlocked); }/**
David Howells
maintainerPATCH 7/17 ceph: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage()
In reply to David HowellsUse sendmsg() and MSG_SPLICE_PAGES rather than sendpage in ceph when transmitting data. For the moment, this can only transmit one page at a time because of the architecture of net/ceph/, but if write_partial_message_data() can be given a bvec[] at a time by the iteration code, this would allow pages to be sent in a batch.
--- net/ceph/messenger_v2.c | 91 +++++++++-------------------------------- 1 file changed, 19 insertions(+), 72 deletions(-)
net/ceph/messenger_v2.c +19 -72
@@ -117,91 +117,38 @@ static int ceph_tcp_recv(struct ceph_connection *con)return ret; } -static int do_sendmsg (struct socket *sock,struct iov_iter *it) -{ -struct msghdr msg = { .msg_flags =CEPH_MSG_FLAGS }; -int ret; - - msg.msg_iter = *it; -while (iov_iter_count (it)) { - ret =sock_sendmsg (sock, &msg); -if (ret <=0 ) { -if (ret == -EAGAIN ) - ret =0 ; -return ret; - } - -iov_iter_advance (it, ret); - } - -WARN_ON (msg_data_left (&msg)); -return 1 ; -} - -static int do_try_sendpage (struct socket *sock,struct iov_iter *it) -{ -struct msghdr msg = { .msg_flags =CEPH_MSG_FLAGS }; -struct bio_vec bv; -int ret; - -if (WARN_ON (!iov_iter_is_bvec (it))) -return -EINVAL ; - -while (iov_iter_count (it)) { -/* iov_iter_iovec() for ITER_BVEC */ -bvec_set_page (&bv, it->bvec->bv_page, -min (iov_iter_count (it), - it->bvec->bv_len - it->iov_offset), - it->bvec->bv_offset + it->iov_offset); - -/* -* sendpage cannot properly handle pages with -* page_count == 0, we need to fall back to sendmsg if -* that's the case. -* -* Same goes for slab pages: skb_can_coalesce() allows -* coalescing neighboring slab objects into a single frag -* which triggers one of hardened usercopy checks. -*/ -if (sendpage_ok (bv.bv_page)) { - ret = sock->ops->sendpage (sock, bv.bv_page, - bv.bv_offset, bv.bv_len, -CEPH_MSG_FLAGS ); - }else { -iov_iter_bvec (&msg.msg_iter,ITER_SOURCE , &bv,1 , bv.bv_len); - ret =sock_sendmsg (sock, &msg); - } -if (ret <=0 ) { -if (ret == -EAGAIN ) - ret =0 ; -return ret; - } - -iov_iter_advance (it, ret); - } - -return 1 ; -} -/* * Write as much as possible. The socket is expected to be corked, -* so we don't bother with MSG_MORE/MSG_SENDPAGE_NOTLAST here. +* so we don't bother with MSG_MORE here. * * Return: -* 1 - done, nothing (else) to write +* >0 - done, nothing (else) to write * 0 - socket is full, need to wait * <0 - error */ static int ceph_tcp_send (struct ceph_connection *con) { +struct msghdr msg = { + .msg_iter = con->v2.out_iter, + .msg_flags =CEPH_MSG_FLAGS , + };int ret; +if (WARN_ON (!iov_iter_is_bvec (&con->v2.out_iter))) +return -EINVAL ; + +if (con->v2.out_iter_sendpage) + msg.msg_flags |=MSG_SPLICE_PAGES ; +dout ("%s con %p have %zu try_sendpage %d\n" , __func__, con,iov_iter_count (&con->v2.out_iter), con->v2.out_iter_sendpage); -if (con->v2.out_iter_sendpage) - ret =do_try_sendpage (con->sock, &con->v2.out_iter); -else - ret =do_sendmsg (con->sock, &con->v2.out_iter); + + ret =sock_sendmsg (con->sock, &msg); +if (ret >0 ) +iov_iter_advance (&con->v2.out_iter, ret); +else if (ret == -EAGAIN ) + ret =0 ; +dout ("%s con %p ret %d left %zu\n" , __func__, con, ret,iov_iter_count (&con->v2.out_iter));return ret;
David Howells
maintainerPATCH 8/17 rds: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage
In reply to David HowellsWhen transmitting data, call down into TCP using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather than performing several sendmsg and sendpage calls to transmit header and data pages.
To make this work, the data is assembled in a bio_vec array and attached to a BVEC-type iterator. The header are copied into memory acquired from zcopy_alloc() which just breaks a page up into small pieces that can be freed with put_page().
--- net/rds/tcp_send.c | 74 +++++++++++++++++----------------------------- 1 file changed, 27 insertions(+), 47 deletions(-)
net/rds/tcp_send.c +27 -47
@@ -52,29 +52,23 @@ void rds_tcp_xmit_path_complete(struct rds_conn_path *cp)tcp_sock_set_cork (tc->t_sock->sk, false); } -/* the core send_sem serializes this with other xmit and shutdown */ -static int rds_tcp_sendmsg (struct socket *sock,void *data,unsigned int len) -{ -struct kvec vec = { - .iov_base = data, - .iov_len = len, - }; -struct msghdr msg = { - .msg_flags =MSG_DONTWAIT |MSG_NOSIGNAL , - }; - -return kernel_sendmsg (sock, &msg, &vec,1 , vec.iov_len); -} -/* the core send_sem serializes this with other xmit and shutdown */ int rds_tcp_xmit (struct rds_connection *conn,struct rds_message *rm,unsigned int hdr_off,unsigned int sg,unsigned int off) {struct rds_conn_path *cp = rm->m_inc.i_conn_path;struct rds_tcp_connection *tc = cp->cp_transport_data; +struct msghdr msg = { + .msg_flags =MSG_SPLICE_PAGES |MSG_DONTWAIT |MSG_NOSIGNAL , + }; +struct bio_vec *bvec; +unsigned int i, size =0 , ix =0 ;int done =0 ; -int ret =0 ; -int more; +int ret = -ENOMEM ; + + bvec =kmalloc_array (1 + sg,sizeof (struct bio_vec ),GFP_KERNEL ); +if (!bvec) + goto out;if (hdr_off ==0 ) {/*
@@ -101,41 +95,26 @@ int rds_tcp_xmit(struct rds_connection *conn, struct rds_message *rm,/* see rds_tcp_write_space() */ set_bit (SOCK_NOSPACE , &tc->t_sock->sk->sk_socket->flags); - ret =rds_tcp_sendmsg (tc->t_sock, - (void *)&rm->m_inc.i_hdr + hdr_off, -sizeof (rm->m_inc.i_hdr) - hdr_off); -if (ret <0 ) - goto out; - done += ret; -if (hdr_off + done !=sizeof (struct rds_header )) - goto out; +bvec_set_virt (&bvec[ix], (void *)&rm->m_inc.i_hdr + hdr_off, +sizeof (rm->m_inc.i_hdr) - hdr_off); + size += bvec[ix].bv_len; + ix++; } - more = rm->data.op_nents >1 ? (MSG_MORE |MSG_SENDPAGE_NOTLAST ) :0 ; -while (sg < rm->data.op_nents) { -int flags =MSG_DONTWAIT |MSG_NOSIGNAL | more; - - ret = tc->t_sock->ops->sendpage (tc->t_sock, -sg_page (&rm->data.op_sg[sg]), - rm->data.op_sg[sg].offset + off, - rm->data.op_sg[sg].length - off, - flags); -rdsdebug ("tcp sendpage %p:%u:%u ret %d\n" , (void *)sg_page (&rm->data.op_sg[sg]), - rm->data.op_sg[sg].offset + off, rm->data.op_sg[sg].length - off, - ret); -if (ret <=0 ) -break ; - - off += ret; - done += ret; -if (off == rm->data.op_sg[sg].length) { - off =0 ; - sg++; - } -if (sg == rm->data.op_nents -1 ) - more =0 ; +for (i = sg; i < rm->data.op_nents; i++) { +bvec_set_page (&bvec[ix], +sg_page (&rm->data.op_sg[i]), + rm->data.op_sg[i].length - off, + rm->data.op_sg[i].offset + off); + off =0 ; + size += bvec[ix].bv_len; + ix++; } +iov_iter_bvec (&msg.msg_iter,ITER_SOURCE , bvec, ix, size); + ret =sock_sendmsg (tc->t_sock, &msg); +rdsdebug ("tcp sendmsg-splice %u,%u ret %d\n" , ix, size, ret); + out:if (ret <=0 ) {/* write_space will hit after EAGAIN, all else fatal */
@@ -158,6 +137,7 @@ int rds_tcp_xmit(struct rds_connection *conn, struct rds_message *rm, }if (done ==0 ) done = ret; +kfree (bvec);return done; }
David Howells
maintainerPATCH 9/17 dlm: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage
In reply to David HowellsWhen transmitting data, call down a layer using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather using sendpage. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction.
--- fs/dlm/lowcomms.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
fs/dlm/lowcomms.c +7 -3
@@ -1395,8 +1395,11 @@ int dlm_lowcomms_resend_msg(struct dlm_msg *msg)/* Send a message */ static int send_to_sock (struct connection *con) { -const int msg_flags =MSG_DONTWAIT |MSG_NOSIGNAL ;struct writequeue_entry *e; +struct bio_vec bvec; +struct msghdr msg = { + .msg_flags =MSG_SPLICE_PAGES |MSG_DONTWAIT |MSG_NOSIGNAL , + };int len, offset, ret;spin_lock_bh (&con->writequeue_lock);
@@ -1412,8 +1415,9 @@ static int send_to_sock(struct connection *con)WARN_ON_ONCE (len ==0 && e->users ==0 );spin_unlock_bh (&con->writequeue_lock); - ret =kernel_sendpage (con->sock, e->page, offset, len, - msg_flags); +bvec_set_page (&bvec, e->page, len, offset); +iov_iter_bvec (&msg.msg_iter,ITER_SOURCE , &bvec,1 , len); + ret =sock_sendmsg (con->sock, &msg);trace_dlm_send (con->nodeid, ret);if (ret == -EAGAIN || ret ==0 ) {lock_sock (con->sock->sk);
David Howells
maintainerPATCH 10/17 nvme: Use sendmsg(MSG_SPLICE_PAGES) rather then sendpage
In reply to David HowellsWhen transmitting data, call down into TCP using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather than performing several sendmsg and sendpage calls to transmit header, data pages and trailer.
--- drivers/nvme/host/tcp.c | 44 ++++++++++++++++++------------------- drivers/nvme/target/tcp.c | 46 ++++++++++++++++++++++++--------------- 2 files changed, 51 insertions(+), 39 deletions(-)
drivers/nvme/host/tcp.c +22 -22
@@ -997,25 +997,23 @@ static int nvme_tcp_try_send_data(struct nvme_tcp_request *req)u32 h2cdata_left = req->h2cdata_left;while (true) { +struct bio_vec bvec; +struct msghdr msg = { .msg_flags =MSG_DONTWAIT |MSG_SPLICE_PAGES , };struct page *page =nvme_tcp_req_cur_page (req);size_t offset =nvme_tcp_req_cur_offset (req);size_t len =nvme_tcp_req_cur_length (req);bool last =nvme_tcp_pdu_last_send (req, len);int req_data_sent = req->data_sent; -int ret, flags =MSG_DONTWAIT ; +int ret;if (last && !queue->data_digest && !nvme_tcp_queue_more (queue)) - flags |=MSG_EOR ; + msg.msg_flags |=MSG_EOR ;else - flags |=MSG_MORE |MSG_SENDPAGE_NOTLAST ; + msg.msg_flags |=MSG_MORE ; -if (sendpage_ok (page)) { - ret =kernel_sendpage (queue->sock, page, offset, len, - flags); - }else { - ret =sock_no_sendpage (queue->sock, page, offset, len, - flags); - } +bvec_set_page (&bvec, page, len, offset); +iov_iter_bvec (&msg.msg_iter,ITER_SOURCE , &bvec,1 , len); + ret =sock_sendmsg (queue->sock, &msg);if (ret <=0 )return ret;
@@ -1054,22 +1052,24 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req) {struct nvme_tcp_queue *queue = req->queue;struct nvme_tcp_cmd_pdu *pdu =nvme_tcp_req_cmd_pdu (req); +struct bio_vec bvec; +struct msghdr msg = { .msg_flags =MSG_DONTWAIT |MSG_SPLICE_PAGES , };bool inline_data =nvme_tcp_has_inline_data (req);u8 hdgst =nvme_tcp_hdgst_len (queue);int len =sizeof (*pdu) + hdgst - req->offset; -int flags =MSG_DONTWAIT ;int ret;if (inline_data ||nvme_tcp_queue_more (queue)) - flags |=MSG_MORE |MSG_SENDPAGE_NOTLAST ; + msg.msg_flags |=MSG_MORE ;else - flags |=MSG_EOR ; + msg.msg_flags |=MSG_EOR ;if (queue->hdr_digest && !req->offset)nvme_tcp_hdgst (queue->snd_hash, pdu,sizeof (*pdu)); - ret =kernel_sendpage (queue->sock,virt_to_page (pdu), -offset_in_page (pdu) + req->offset, len, flags); +bvec_set_virt (&bvec, (void *)pdu + req->offset, len); +iov_iter_bvec (&msg.msg_iter,ITER_SOURCE , &bvec,1 , len); + ret =sock_sendmsg (queue->sock, &msg);if (unlikely (ret <=0 ))return ret;
@@ -1093,6 +1093,8 @@ static int nvme_tcp_try_send_data_pdu(struct nvme_tcp_request *req) {struct nvme_tcp_queue *queue = req->queue;struct nvme_tcp_data_pdu *pdu =nvme_tcp_req_data_pdu (req); +struct bio_vec bvec; +struct msghdr msg = { .msg_flags =MSG_DONTWAIT |MSG_MORE , };u8 hdgst =nvme_tcp_hdgst_len (queue);int len =sizeof (*pdu) - req->offset + hdgst;int ret;
@@ -1101,13 +1103,11 @@ static int nvme_tcp_try_send_data_pdu(struct nvme_tcp_request *req)nvme_tcp_hdgst (queue->snd_hash, pdu,sizeof (*pdu));if (!req->h2cdata_left) - ret =kernel_sendpage (queue->sock,virt_to_page (pdu), -offset_in_page (pdu) + req->offset, len, -MSG_DONTWAIT |MSG_MORE |MSG_SENDPAGE_NOTLAST ); -else - ret =sock_no_sendpage (queue->sock,virt_to_page (pdu), -offset_in_page (pdu) + req->offset, len, -MSG_DONTWAIT |MSG_MORE ); + msg.msg_flags |=MSG_SPLICE_PAGES ; + +bvec_set_virt (&bvec, (void *)pdu + req->offset, len); +iov_iter_bvec (&msg.msg_iter,ITER_SOURCE , &bvec,1 , len); + ret =sock_sendmsg (queue->sock, &msg);if (unlikely (ret <=0 ))return ret;
drivers/nvme/target/tcp.c +29 -17
95 lines not on this page · raw message
David Howells
maintainerPATCH 11/17 smc: Drop smc_sendpage() in favour of smc_sendmsg() + MSG_SPLICE_PAGES
In reply to David HowellsDrop the smc_sendpage() code as smc_sendmsg() just passes the call down to the underlying TCP socket and smc_tx_sendpage() is just a wrapper around its sendmsg implementation.
--- net/smc/af_smc.c | 29 ----------------------------- net/smc/smc_stats.c | 2 +- net/smc/smc_stats.h | 1 - net/smc/smc_tx.c | 22 +--------------------- net/smc/smc_tx.h | 2 -- 5 files changed, 2 insertions(+), 54 deletions(-)
net/smc/af_smc.c +0 -29
@@ -3133,34 +3133,6 @@ static int smc_ioctl(struct socket *sock, unsigned int cmd,return put_user (answ, (int __user *)arg); } -static ssize_t smc_sendpage (struct socket *sock,struct page *page, -int offset,size_t size,int flags) -{ -struct sock *sk = sock->sk; -struct smc_sock *smc; -int rc = -EPIPE ; - - smc =smc_sk (sk); -lock_sock (sk); -if (sk->sk_state !=SMC_ACTIVE ) { -release_sock (sk); - goto out; - } -release_sock (sk); -if (smc->use_fallback) { - rc =kernel_sendpage (smc->clcsock, page, offset, - size, flags); - }else { -lock_sock (sk); - rc =smc_tx_sendpage (smc, page, offset, size, flags); -release_sock (sk); -SMC_STAT_INC (smc, sendpage_cnt); - } - -out: -return rc; -} -/* Map the affected portions of the rmbe into an spd, note the number of bytes * to splice in conn->splice_pending, and press 'go'. Delays consumer cursor * updates till whenever a respective page has been fully processed.
@@ -3232,7 +3204,6 @@ static const struct proto_ops smc_sock_ops = { .sendmsg = smc_sendmsg, .recvmsg = smc_recvmsg, .mmap = sock_no_mmap, - .sendpage = smc_sendpage, .splice_read = smc_splice_read, };
net/smc/smc_stats.c +1 -1
@@ -227,7 +227,7 @@ static int smc_nl_fill_stats_tech_data(struct sk_buff *skb, SMC_NLA_STATS_PAD)) goto errattr;if (nla_put_u64_64bit (skb,SMC_NLA_STATS_T_SENDPAGE_CNT , - smc_tech->sendpage_cnt, +0 ,SMC_NLA_STATS_PAD )) goto errattr; if (nla_put_u64_64bit(skb, SMC_NLA_STATS_T_CORK_CNT,
net/smc/smc_stats.h +0 -1
@@ -71,7 +71,6 @@ struct smc_stats_tech {u64 clnt_v2_succ_cnt;u64 srv_v1_succ_cnt;u64 srv_v2_succ_cnt; -u64 sendpage_cnt;u64 urg_data_cnt;u64 splice_cnt;u64 cork_cnt;
net/smc/smc_tx.c +1 -21
@@ -167,8 +167,7 @@ static bool smc_tx_should_cork(struct smc_sock *smc, struct msghdr *msg)* sndbuf_space is still available. The applications * should known how/when to uncork it. */ -if ((msg->msg_flags &MSG_MORE || -smc_tx_is_corked (smc)) && +if ((msg->msg_flags &MSG_MORE ||smc_tx_is_corked (smc)) &&atomic_read (&conn->sndbuf_space))return true;
@@ -297,25 +296,6 @@ int smc_tx_sendmsg(struct smc_sock *smc, struct msghdr *msg, size_t len)return rc; } -int smc_tx_sendpage (struct smc_sock *smc,struct page *page,int offset, -size_t size,int flags) -{ -struct msghdr msg = {.msg_flags = flags}; -char *kaddr =kmap (page); -struct kvec iov; -int rc; - -if (flags &MSG_SENDPAGE_NOTLAST ) - msg.msg_flags |=MSG_MORE ; - - iov.iov_base = kaddr + offset; - iov.iov_len = size; -iov_iter_kvec (&msg.msg_iter,ITER_SOURCE , &iov,1 , size); - rc =smc_tx_sendmsg (smc, &msg, size); -kunmap (page); -return rc; -} -/***************************** sndbuf consumer *******************************/ /* sndbuf consumer: actual data transfer of one target chunk with ISM write */
net/smc/smc_tx.h +0 -2
@@ -31,8 +31,6 @@ void smc_tx_pending(struct smc_connection *conn);void smc_tx_work (struct work_struct *work);void smc_tx_init (struct smc_sock *smc);int smc_tx_sendmsg (struct smc_sock *smc,struct msghdr *msg,size_t len); -int smc_tx_sendpage (struct smc_sock *smc,struct page *page,int offset, -size_t size,int flags);int smc_tx_sndbuf_nonempty (struct smc_connection *conn);void smc_tx_sndbuf_nonfull (struct smc_sock *smc);void smc_tx_consumer_update (struct smc_connection *conn,bool force);
David Howells
maintainerPATCH 12/17 ocfs2: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage()
In reply to David HowellsFix ocfs2 to use the page fragment allocator rather than kzalloc in order to allocate the buffers for the handshake message and keepalive request and reply messages.
Switch from using sendpage() to using sendmsg() + MSG_SPLICE_PAGES so that sendpage can be phased out.
--- fs/ocfs2/cluster/tcp.c | 107 ++++++++++++++++++++++------------------- 1 file changed, 58 insertions(+), 49 deletions(-)
fs/ocfs2/cluster/tcp.c +58 -49
@@ -110,9 +110,6 @@ static struct work_struct o2net_listen_work;static struct o2hb_callback_func o2net_hb_up, o2net_hb_down;#define O2NET_HB_PRI 0x1 -static struct o2net_handshake *o2net_hand; -static struct o2net_msg *o2net_keep_req, *o2net_keep_resp; -static int o2net_sys_err_translations[O2NET_ERR_MAX ] = {[O2NET_ERR_NONE ] =0 , [O2NET_ERR_NO_HNDLR ] = -ENOPROTOOPT ,
@@ -930,19 +927,22 @@ static int o2net_send_tcp_msg(struct socket *sock, struct kvec *vec, }static void o2net_sendpage (struct o2net_sock_container *sc, -void *kmalloced_virt, -size_t size) +void *virt,size_t size) {struct o2net_node *nn =o2net_nn_from_num (sc->sc_node->nd_num); +struct msghdr msg = {}; +struct bio_vec bv;ssize_t ret; +bvec_set_virt (&bv, virt, size); +iov_iter_bvec (&msg.msg_iter,ITER_SOURCE , &bv,1 , size); +while (1 ) { + msg.msg_flags =MSG_DONTWAIT |MSG_SPLICE_PAGES ;mutex_lock (&sc->sc_send_lock); - ret = sc->sc_sock->ops->sendpage (sc->sc_sock, -virt_to_page (kmalloced_virt), -offset_in_page (kmalloced_virt), - size,MSG_DONTWAIT ); + ret =sock_sendmsg (sc->sc_sock, &msg);mutex_unlock (&sc->sc_send_lock); +if (ret == size)break ;if (ret == (ssize_t )-EAGAIN ) {
@@ -1168,6 +1168,7 @@ static int o2net_process_message(struct o2net_sock_container *sc, struct o2net_msg *hdr) {struct o2net_node *nn =o2net_nn_from_num (sc->sc_node->nd_num); +struct o2net_msg *keep_resp;int ret =0 , handler_status;enum o2net_system_error syserr;struct o2net_msg_handler *nmh =NULL ;
@@ -1186,8 +1187,16 @@ static int o2net_process_message(struct o2net_sock_container *sc, be32_to_cpu(hdr->status)); goto out;case O2NET_MSG_KEEP_REQ_MAGIC : -o2net_sendpage (sc, o2net_keep_resp, -sizeof (*o2net_keep_resp)); + keep_resp =alloc_skb_frag (sizeof (*keep_resp), +GFP_KERNEL ); +if (!keep_resp) { + ret = -ENOMEM ; + goto out; + } +memset (keep_resp,0 ,sizeof (*keep_resp)); + keep_resp->magic =cpu_to_be16 (O2NET_MSG_KEEP_RESP_MAGIC ); +o2net_sendpage (sc, keep_resp,sizeof (*keep_resp)); +folio_put (virt_to_folio (keep_resp)); goto out;case O2NET_MSG_KEEP_RESP_MAGIC : goto out;
@@ -1439,15 +1448,22 @@ static void o2net_rx_until_empty(struct work_struct *work)sc_put (sc); } -static void o2net_initialize_handshake (void ) +static struct o2net_handshake *o2net_initialize_handshake (void ) { - o2net_hand->o2hb_heartbeat_timeout_ms =cpu_to_be32 ( -O2HB_MAX_WRITE_TIMEOUT_MS ); - o2net_hand->o2net_idle_timeout_ms =cpu_to_be32 (o2net_idle_timeout ()); - o2net_hand->o2net_keepalive_delay_ms =cpu_to_be32 ( -o2net_keepalive_delay ()); - o2net_hand->o2net_reconnect_delay_ms =cpu_to_be32 ( -o2net_reconnect_delay ()); +struct o2net_handshake *hand; + + hand =alloc_skb_frag (sizeof (*hand),GFP_KERNEL ); +if (!hand) +return NULL ; + +memset (hand,0 ,sizeof (*hand)); + hand->protocol_version =cpu_to_be64 (O2NET_PROTOCOL_VERSION ); + hand->connector_id =cpu_to_be64 (1 ); + hand->o2hb_heartbeat_timeout_ms =cpu_to_be32 (O2HB_MAX_WRITE_TIMEOUT_MS ); + hand->o2net_idle_timeout_ms =cpu_to_be32 (o2net_idle_timeout ()); + hand->o2net_keepalive_delay_ms =cpu_to_be32 (o2net_keepalive_delay ()); + hand->o2net_reconnect_delay_ms =cpu_to_be32 (o2net_reconnect_delay ()); +return hand; }/* ------------------------------------------------------------ */
@@ -1456,16 +1472,22 @@ static void o2net_initialize_handshake(void)* rx path will see the response and mark the sc valid */ static void o2net_sc_connect_completed (struct work_struct *work) { +struct o2net_handshake *hand;struct o2net_sock_container *sc = container_of(work, struct o2net_sock_container, sc_connect_work); + hand =o2net_initialize_handshake (); +if (!hand) + goto out; +mlog (ML_MSG ,"sc sending handshake with ver %llu id %llx\n" , (unsigned long long )O2NET_PROTOCOL_VERSION , - (unsigned long long )be64_to_cpu (o2net_hand->connector_id)); + (unsigned long long )be64_to_cpu (hand->connector_id)); -o2net_initialize_handshake (); -o2net_sendpage (sc, o2net_hand,sizeof (*o2net_hand)); +o2net_sendpage (sc, hand,sizeof (*hand)); +folio_put (virt_to_folio (hand)); +out:sc_put (sc); }
@@ -1475,8 +1497,15 @@ static void o2net_sc_send_keep_req(struct work_struct *work)struct o2net_sock_container *sc = container_of(work, struct o2net_sock_container, sc_keepalive_work.work); +struct o2net_msg *keep_req; -o2net_sendpage (sc, o2net_keep_req,sizeof (*o2net_keep_req)); + keep_req =alloc_skb_frag (sizeof (*keep_req),GFP_KERNEL ); +if (keep_req) { +memset (keep_req,0 ,sizeof (*keep_req)); + keep_req->magic =cpu_to_be16 (O2NET_MSG_KEEP_REQ_MAGIC ); +o2net_sendpage (sc, keep_req,sizeof (*keep_req)); +folio_put (virt_to_folio (keep_req)); + }sc_put (sc); }
@@ -1780,6 +1809,7 @@ static int o2net_accept_one(struct socket *sock, int *more)struct socket *new_sock =NULL ;struct o2nm_node *node =NULL ;struct o2nm_node *local_node =NULL ; +struct o2net_handshake *hand;struct o2net_sock_container *sc =NULL ;struct o2net_node *nn;unsigned int nofs_flag;
@@ -1882,8 +1912,11 @@ static int o2net_accept_one(struct socket *sock, int *more)o2net_register_callbacks (sc->sc_sock->sk, sc);o2net_sc_queue_work (sc, &sc->sc_rx_work); -o2net_initialize_handshake (); -o2net_sendpage (sc, o2net_hand,sizeof (*o2net_hand)); + hand =o2net_initialize_handshake (); +if (hand) { +o2net_sendpage (sc, hand,sizeof (*hand)); +folio_put (virt_to_folio (hand)); + } out:if (new_sock)
@@ -2090,21 +2123,8 @@ int o2net_init(void)unsigned long i;o2quo_init (); -o2net_debugfs_init (); - o2net_hand =kzalloc (sizeof (struct o2net_handshake ),GFP_KERNEL ); - o2net_keep_req =kzalloc (sizeof (struct o2net_msg ),GFP_KERNEL ); - o2net_keep_resp =kzalloc (sizeof (struct o2net_msg ),GFP_KERNEL ); -if (!o2net_hand || !o2net_keep_req || !o2net_keep_resp) - goto out; - - o2net_hand->protocol_version =cpu_to_be64 (O2NET_PROTOCOL_VERSION ); - o2net_hand->connector_id =cpu_to_be64 (1 ); - - o2net_keep_req->magic =cpu_to_be16 (O2NET_MSG_KEEP_REQ_MAGIC ); - o2net_keep_resp->magic =cpu_to_be16 (O2NET_MSG_KEEP_RESP_MAGIC ); -for (i =0 ; i <ARRAY_SIZE (o2net_nodes); i++) {struct o2net_node *nn =o2net_nn_from_num (i);
@@ -2122,21 +2142,10 @@ int o2net_init(void) }return 0 ; - -out: -kfree (o2net_hand); -kfree (o2net_keep_req); -kfree (o2net_keep_resp); -o2net_debugfs_exit (); -o2quo_exit (); -return -ENOMEM ; }void o2net_exit (void ) {o2quo_exit (); -kfree (o2net_hand); -kfree (o2net_keep_req); -kfree (o2net_keep_resp);o2net_debugfs_exit (); }
David Howells
maintainerPATCH 13/17 drbd: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage()
In reply to David HowellsUse sendmsg() conditionally with MSG_SPLICE_PAGES in _drbd_send_page() rather than calling sendpage() or _drbd_no_send_page().
--- drivers/block/drbd/drbd_main.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-)
drivers/block/drbd/drbd_main.c +12 -11
@@ -1540,7 +1540,8 @@ static int _drbd_send_page(struct drbd_peer_device *peer_device, struct page *pa int offset, size_t size, unsigned msg_flags) { struct socket *socket = peer_device->connection->data.socket; -int len = size; +struct bio_vec bvec; +struct msghdr msg = { .msg_flags = msg_flags, };int err = -EIO ;/* e.g. XFS meta- & log-data is in slab pages, which have a
@@ -1549,33 +1550,33 @@ static int _drbd_send_page(struct drbd_peer_device *peer_device, struct page *pa* put_page(); and would cause either a VM_BUG directly, or * __page_cache_release a page that would actually still be referenced * by someone, leading to some obscure delayed Oops somewhere else. */ -if (drbd_disable_sendpage || !sendpage_ok (page)) -return _drbd_no_send_page (peer_device, page, offset, size, msg_flags); +if (!drbd_disable_sendpage &&sendpage_ok (page)) + msg.msg_flags |=MSG_NOSIGNAL |MSG_SPLICE_PAGES ; + +bvec_set_page (&bvec, page, offset, size); +iov_iter_bvec (&msg.msg_iter,ITER_SOURCE , &bvec,1 , size); - msg_flags |=MSG_NOSIGNAL ;drbd_update_congested (peer_device->connection);do {int sent; - sent = socket->ops->sendpage (socket, page, offset, len, msg_flags); + sent =sock_sendmsg (socket, &msg);if (sent <=0 ) {if (sent == -EAGAIN ) {if (we_should_drop_the_connection (peer_device->connection, socket))break ;continue ; } -drbd_warn (peer_device->device,"%s: size=%d len=%d sent=%d\n" , - __func__, (int )size, len, sent); +drbd_warn (peer_device->device,"%s: size=%d len=%zu sent=%d\n" , + __func__, (int )size,msg_data_left (&msg), sent);if (sent <0 ) err = sent;break ; } - len -= sent; - offset += sent; - }while (len >0 /* THINK && device->cstate >= C_CONNECTED*/ ); + }while (msg_data_left (&msg)/* THINK && device->cstate >= C_CONNECTED*/ );clear_bit (NET_CONGESTED , &peer_device->connection->flags); -if (len ==0 ) { +if (!msg_data_left (&msg)) { err =0 ; peer_device->device->send_cnt += size >>9 ; }
David Howells
maintainerPATCH 14/17 drdb: Send an entire bio in a single sendmsg
In reply to David HowellsSince _drdb_sendpage() is now using sendmsg to send the pages rather sendpage, pass the entire bio in one go using a bvec iterator instead of doing it piecemeal.
--- drivers/block/drbd/drbd_main.c | 77 +++++++++++----------------------- 1 file changed, 25 insertions(+), 52 deletions(-)
drivers/block/drbd/drbd_main.c +25 -52
@@ -1520,28 +1520,15 @@ static void drbd_update_congested(struct drbd_connection *connection)* As a workaround, we disable sendpage on pages * with page_count == 0 or PageSlab. */ -static int _drbd_no_send_page (struct drbd_peer_device *peer_device,struct page *page, -int offset,size_t size,unsigned msg_flags) -{ -struct socket *socket; -void *addr; -int err; - - socket = peer_device->connection->data.socket; - addr =kmap (page) + offset; - err =drbd_send_all (peer_device->connection, socket, addr, size, msg_flags); -kunmap (page); -if (!err) - peer_device->device->send_cnt += size >>9 ; -return err; -} - -static int _drbd_send_page (struct drbd_peer_device *peer_device,struct page *page, -int offset,size_t size,unsigned msg_flags) +static int _drbd_send_pages (struct drbd_peer_device *peer_device, +struct iov_iter *iter,unsigned msg_flags) {struct socket *socket = peer_device->connection->data.socket; -struct bio_vec bvec; -struct msghdr msg = { .msg_flags = msg_flags, }; +struct msghdr msg = { + .msg_flags = msg_flags |MSG_NOSIGNAL , + .msg_iter = *iter, + }; +size_t size =iov_iter_count (iter);int err = -EIO ;/* e.g. XFS meta- & log-data is in slab pages, which have a
@@ -1550,11 +1537,8 @@ static int _drbd_send_page(struct drbd_peer_device *peer_device, struct page *pa* put_page(); and would cause either a VM_BUG directly, or * __page_cache_release a page that would actually still be referenced * by someone, leading to some obscure delayed Oops somewhere else. */ -if (!drbd_disable_sendpage &&sendpage_ok (page)) - msg.msg_flags |=MSG_NOSIGNAL |MSG_SPLICE_PAGES ; - -bvec_set_page (&bvec, page, offset, size); -iov_iter_bvec (&msg.msg_iter,ITER_SOURCE , &bvec,1 , size); +if (drbd_disable_sendpage) + msg.msg_flags &= ~(MSG_NOSIGNAL |MSG_SPLICE_PAGES );drbd_update_congested (peer_device->connection); do {
@@ -1585,39 +1569,22 @@ static int _drbd_send_page(struct drbd_peer_device *peer_device, struct page *pastatic int _drbd_send_bio (struct drbd_peer_device *peer_device,struct bio *bio) { -struct bio_vec bvec; -struct bvec_iter iter; +struct iov_iter iter; -/* hint all but last page with MSG_MORE */ -bio_for_each_segment (bvec, bio, iter) { -int err; +iov_iter_bvec (&iter,ITER_SOURCE , bio->bi_io_vec, bio->bi_vcnt, + bio->bi_iter.bi_size); - err =_drbd_no_send_page (peer_device, bvec.bv_page, - bvec.bv_offset, bvec.bv_len, -bio_iter_last (bvec, iter) - ?0 :MSG_MORE ); -if (err) -return err; - } -return 0 ; +return _drbd_send_pages (peer_device, &iter,0 ); }static int _drbd_send_zc_bio (struct drbd_peer_device *peer_device,struct bio *bio) { -struct bio_vec bvec; -struct bvec_iter iter; +struct iov_iter iter; -/* hint all but last page with MSG_MORE */ -bio_for_each_segment (bvec, bio, iter) { -int err; +iov_iter_bvec (&iter,ITER_SOURCE , bio->bi_io_vec, bio->bi_vcnt, + bio->bi_iter.bi_size); - err =_drbd_send_page (peer_device, bvec.bv_page, - bvec.bv_offset, bvec.bv_len, -bio_iter_last (bvec, iter) ?0 :MSG_MORE ); -if (err) -return err; - } -return 0 ; +return _drbd_send_pages (peer_device, &iter,MSG_SPLICE_PAGES ); } static int _drbd_send_zc_ee(struct drbd_peer_device *peer_device,
@@ -1629,10 +1596,16 @@ static int _drbd_send_zc_ee(struct drbd_peer_device *peer_device,/* hint all but last page with MSG_MORE */ page_chain_for_each (page) { +struct iov_iter iter; +struct bio_vec bvec;unsigned l =min_t (unsigned, len,PAGE_SIZE ); - err =_drbd_send_page (peer_device, page,0 , l, -page_chain_next (page) ?MSG_MORE :0 ); +bvec_set_page (&bvec, page,0 , l); +iov_iter_bvec (&iter,ITER_SOURCE , &bvec,1 , l); + + err =_drbd_send_pages (peer_device, &iter, +MSG_SPLICE_PAGES | + (page_chain_next (page) ?MSG_MORE :0 ));if (err)return err; len -= l;
David Howells
maintainerPATCH 15/17 iscsi: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage
In reply to David HowellsUse sendmsg() with MSG_SPLICE_PAGES rather than sendpage. This allows multiple pages and multipage folios to be passed through.
TODO: iscsit_fe_sendpage_sg() should perhaps set up a bio_vec array for the entire set of pages it's going to transfer plus two for the header and trailer and page fragments to hold the header and trailer - and then call sendmsg once for the entire message.
--- drivers/scsi/iscsi_tcp.c | 26 +++++++++--------------- drivers/scsi/iscsi_tcp.h | 2 +- drivers/target/iscsi/iscsi_target_util.c | 14 +++++++------ 3 files changed, 19 insertions(+), 23 deletions(-)
drivers/scsi/iscsi_tcp.c +10 -16
@@ -301,35 +301,32 @@ static int iscsi_sw_tcp_xmit_segment(struct iscsi_tcp_conn *tcp_conn,while (!iscsi_tcp_segment_done (tcp_conn, segment,0 , r)) {struct scatterlist *sg; +struct msghdr msg = {}; +struct bio_vec bv;unsigned int offset, copy; -int flags =0 ; r =0 ; offset = segment->copied; copy = segment->size - offset;if (segment->total_copied + segment->size < segment->total_size) - flags |=MSG_MORE |MSG_SENDPAGE_NOTLAST ; + msg.msg_flags |=MSG_MORE ;if (tcp_sw_conn->queue_recv) - flags |=MSG_DONTWAIT ; + msg.msg_flags |=MSG_DONTWAIT ; -/* Use sendpage if we can; else fall back to sendmsg */ if (!segment->data) { +if (!tcp_conn->iscsi_conn->datadgst_en) + msg.msg_flags |=MSG_SPLICE_PAGES ; sg = segment->sg; offset += segment->sg_offset + sg->offset; - r = tcp_sw_conn->sendpage (sk,sg_page (sg), offset, - copy, flags); +bvec_set_page (&bv,sg_page (sg), copy, offset); }else { -struct msghdr msg = { .msg_flags = flags }; -struct kvec iov = { - .iov_base = segment->data + offset, - .iov_len = copy - }; - - r =kernel_sendmsg (sk, &msg, &iov,1 , copy); +bvec_set_virt (&bv, segment->data + offset, copy); } +iov_iter_bvec (&msg.msg_iter,ITER_SOURCE , &bv,1 , copy); + r =sock_sendmsg (sk, &msg);if (r <0 ) {iscsi_tcp_segment_unmap (segment);return r;
@@ -746,7 +743,6 @@ iscsi_sw_tcp_conn_bind(struct iscsi_cls_session *cls_session,sock_no_linger (sk);iscsi_sw_tcp_conn_set_callbacks (conn); - tcp_sw_conn->sendpage = tcp_sw_conn->sock->ops->sendpage;/* * set receive state machine into initial state */
@@ -777,8 +773,6 @@ static int iscsi_sw_tcp_conn_set_param(struct iscsi_cls_conn *cls_conn,return -ENOTCONN ; }iscsi_set_param (cls_conn, param, buf, buflen); - tcp_sw_conn->sendpage = conn->datadgst_en ? - sock_no_sendpage : tcp_sw_conn->sock->ops->sendpage;mutex_unlock (&tcp_sw_conn->sock_lock);break ;case ISCSI_PARAM_MAX_R2T :
drivers/scsi/iscsi_tcp.h +1 -1
@@ -48,7 +48,7 @@ struct iscsi_sw_tcp_conn {uint32_t sendpage_failures_cnt;uint32_t discontiguous_hdr_cnt; -ssize_t (*sendpage)(struct socket *,struct page *,int ,size_t ,int ); +bool can_splice_to_tcp; };struct iscsi_sw_tcp_host {
drivers/target/iscsi/iscsi_target_util.c +8 -6
@@ -1129,6 +1129,8 @@ int iscsit_fe_sendpage_sg( struct iscsit_conn *conn) {struct scatterlist *sg = cmd->first_data_sg; +struct bio_vec bvec; +struct msghdr msghdr = { .msg_flags =MSG_SPLICE_PAGES , };struct kvec iov;u32 tx_hdr_size, data_len;u32 offset = cmd->first_data_sg_off;
@@ -1172,17 +1174,17 @@ int iscsit_fe_sendpage_sg(u32 space = (sg->length - offset);u32 sub_len =min_t (u32, data_len, space); send_pg: - tx_sent = conn->sock->ops->sendpage (conn->sock, -sg_page (sg), sg->offset + offset, sub_len,0 ); +bvec_set_page (&bvec,sg_page (sg), sub_len, sg->offset + offset); +iov_iter_bvec (&msghdr.msg_iter,ITER_SOURCE , &bvec,1 , sub_len); + + tx_sent = conn->sock->ops->sendmsg (conn->sock, &msghdr, sub_len);if (tx_sent != sub_len) {if (tx_sent == -EAGAIN ) { -pr_err ("tcp_sendpage() returned" -" -EAGAIN\n" ); +pr_err ("sendmsg/splice returned -EAGAIN\n" ); goto send_pg; } -pr_err ("tcp_sendpage() failure: %d\n" , - tx_sent); +pr_err ("sendmsg/splice failure: %d\n" , tx_sent);return -1 ; }
David Howells
maintainerPATCH 16/17 sock: Remove ->sendpage*() in favour of sendmsg(MSG_SPLICE_PAGES)
In reply to David HowellsRemove ->sendpage() and ->sendpage_locked(). sendmsg() with MSG_SPLICE_PAGES should be used instead. This allows multiple pages and multipage folios to be passed through.
--- Documentation/bpf/map_sockmap.rst | 10 ++-- Documentation/filesystems/locking.rst | 2 - Documentation/filesystems/vfs.rst | 1 - Documentation/networking/scaling.rst | 4 +- crypto/af_alg.c | 28 ----------- crypto/algif_aead.c | 22 ++------- crypto/algif_rng.c | 2 - crypto/algif_skcipher.c | 14 ------ .../chelsio/inline_crypto/chtls/chtls.h | 2 - .../chelsio/inline_crypto/chtls/chtls_io.c | 14 ------ .../chelsio/inline_crypto/chtls/chtls_main.c | 1 - fs/nfsd/vfs.c | 2 +- include/crypto/if_alg.h | 2 - include/linux/net.h | 8 ---- include/net/inet_common.h | 2 - include/net/sock.h | 6 --- include/net/tcp.h | 4 -- net/appletalk/ddp.c | 1 - net/atm/pvc.c | 1 - net/atm/svc.c | 1 - net/ax25/af_ax25.c | 1 - net/caif/caif_socket.c | 2 - net/can/bcm.c | 1 - net/can/isotp.c | 1 - net/can/j1939/socket.c | 1 - net/can/raw.c | 1 - net/core/sock.c | 35 +------------- net/dccp/ipv4.c | 1 - net/dccp/ipv6.c | 1 - net/ieee802154/socket.c | 2 - net/ipv4/af_inet.c | 21 -------- net/ipv4/tcp.c | 43 ++--------------- net/ipv4/tcp_bpf.c | 23 +-------- net/ipv4/tcp_ipv4.c | 1 - net/ipv4/udp.c | 15 ------ net/ipv4/udp_impl.h | 2 - net/ipv4/udplite.c | 1 - net/ipv6/af_inet6.c | 3 -- net/ipv6/raw.c | 1 - net/ipv6/tcp_ipv6.c | 1 - net/kcm/kcmsock.c | 20 -------- net/key/af_key.c | 1 - net/l2tp/l2tp_ip.c | 1 - net/l2tp/l2tp_ip6.c | 1 - net/llc/af_llc.c | 1 - net/mctp/af_mctp.c | 1 - net/mptcp/protocol.c | 2 - net/netlink/af_netlink.c | 1 - net/netrom/af_netrom.c | 1 - net/packet/af_packet.c | 2 - net/phonet/socket.c | 2 - net/qrtr/af_qrtr.c | 1 - net/rds/af_rds.c | 1 - net/rose/af_rose.c | 1 - net/rxrpc/af_rxrpc.c | 1 - net/sctp/protocol.c | 1 - net/socket.c | 48 ------------------- net/tipc/socket.c | 3 -- net/tls/tls.h | 6 --- net/tls/tls_device.c | 17 ------- net/tls/tls_main.c | 7 --- net/tls/tls_sw.c | 35 -------------- net/unix/af_unix.c | 19 -------- net/vmw_vsock/af_vsock.c | 3 -- net/x25/af_x25.c | 1 - net/xdp/xsk.c | 1 - 66 files changed, 20 insertions(+), 442 deletions(-)
Documentation/bpf/map_sockmap.rst +5 -5
@@ -240,11 +240,11 @@ offsets into ``msg``, respectively. If a program of type ``BPF_PROG_TYPE_SK_MSG`` is run on a ``msg`` it can only parse data that the (``data``, ``data_end``) pointers have already consumed. For ``sendmsg()`` hooks this is likely the first scatterlist element. But for -calls relying on the ``sendpage`` handler (e.g., ``sendfile()``) this will be -the range (**0**, **0**) because the data is shared with user space and by -default the objective is to avoid allowing user space to modify data while (or -after) BPF verdict is being decided. This helper can be used to pull in data -and to set the start and end pointers to given values. Data will be copied if +calls relying on MSG_SPLICE_PAGES (e.g., ``sendfile()``) this will be the +range (**0**, **0**) because the data is shared with user space and by default +the objective is to avoid allowing user space to modify data while (or after) +BPF verdict is being decided. This helper can be used to pull in data and to +set the start and end pointers to given values. Data will be copied if necessary (i.e., if data was not linear and if start and end pointers do not point to the same chunk).
Documentation/filesystems/locking.rst +0 -2
@@ -521,8 +521,6 @@ prototypes:: int (*fsync) (struct file *, loff_t start, loff_t end, int datasync); int (*fasync) (int, struct file *, int); int (*lock) (struct file *, int, struct file_lock *); - ssize_t (*sendpage) (struct file *, struct page *, int, size_t, - loff_t *, int); unsigned long (*get_unmapped_area)(struct file *, unsigned long, unsigned long, unsigned long, unsigned long); int (*check_flags)(int);
Documentation/filesystems/vfs.rst +0 -1
@@ -1086,7 +1086,6 @@ This describes how the VFS can manipulate an open file. As of kernel int (*fsync) (struct file *, loff_t, loff_t, int datasync); int (*fasync) (int, struct file *, int); int (*lock) (struct file *, int, struct file_lock *); - ssize_t (*sendpage) (struct file *, struct page *, int, size_t, loff_t *, int); unsigned long (*get_unmapped_area)(struct file *, unsigned long, unsigned long, unsigned long, unsigned long); int (*check_flags)(int); int (*flock) (struct file *, int, struct file_lock *);
Documentation/networking/scaling.rst +2 -2
@@ -269,8 +269,8 @@ a single application thread handles flows with many different flow hashes. rps_sock_flow_table is a global flow table that contains the *desired* CPU for flows: the CPU that is currently processing the flow in userspace. Each table value is a CPU index that is updated during calls to recvmsg -and sendmsg (specifically, inet_recvmsg(), inet_sendmsg(), inet_sendpage() -and tcp_splice_read()). +and sendmsg (specifically, inet_recvmsg(), inet_sendmsg() and +tcp_splice_read()). When the scheduler moves a thread to a new CPU while it has outstanding receive packets on the old CPU, packets may arrive out of order. To
crypto/af_alg.c +0 -28
@@ -482,7 +482,6 @@ static const struct proto_ops alg_proto_ops = { .listen = sock_no_listen, .shutdown = sock_no_shutdown, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, .sendmsg = sock_no_sendmsg, .recvmsg = sock_no_recvmsg,
@@ -1106,33 +1105,6 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size, }EXPORT_SYMBOL_GPL (af_alg_sendmsg); -/** -* af_alg_sendpage - sendpage system call handler -* @sock: socket of connection to user space to write to -* @page: data to send -* @offset: offset into page to begin sending -* @size: length of data -* @flags: message send/receive flags -* -* This is a generic implementation of sendpage to fill ctx->tsgl_list. -*/ -ssize_t af_alg_sendpage (struct socket *sock,struct page *page, -int offset,size_t size,int flags) -{ -struct bio_vec bvec; -struct msghdr msg = { - .msg_flags = flags |MSG_SPLICE_PAGES , - }; - -if (flags &MSG_SENDPAGE_NOTLAST ) - msg.msg_flags |=MSG_MORE ; - -bvec_set_page (&bvec, page, size, offset); -iov_iter_bvec (&msg.msg_iter,ITER_SOURCE , &bvec,1 , size); -return sock_sendmsg (sock, &msg); -} -EXPORT_SYMBOL_GPL (af_alg_sendpage); -/** * af_alg_free_resources - release resources required for crypto request * @areq: Request holding the TX and RX SGL
crypto/algif_aead.c +4 -18
@@ -9,10 +9,10 @@* The following concept of the memory management is used: * * The kernel maintains two SGLs, the TX SGL and the RX SGL. The TX SGL is -* filled by user space with the data submitted via sendpage. Filling up -* the TX SGL does not cause a crypto operation -- the data will only be -* tracked by the kernel. Upon receipt of one recvmsg call, the caller must -* provide a buffer which is tracked with the RX SGL. +* filled by user space with the data submitted via sendmsg (maybe with with +* MSG_SPLICE_PAGES). Filling up the TX SGL does not cause a crypto operation +* -- the data will only be tracked by the kernel. Upon receipt of one recvmsg +* call, the caller must provide a buffer which is tracked with the RX SGL. * * During the processing of the recvmsg operation, the cipher request is * allocated and prepared. As part of the recvmsg operation, the processed
@@ -370,7 +370,6 @@ static struct proto_ops algif_aead_ops = { .release = af_alg_release, .sendmsg = aead_sendmsg, - .sendpage = af_alg_sendpage, .recvmsg = aead_recvmsg, .poll = af_alg_poll, };
@@ -422,18 +421,6 @@ static int aead_sendmsg_nokey(struct socket *sock, struct msghdr *msg,return aead_sendmsg (sock, msg, size); } -static ssize_t aead_sendpage_nokey (struct socket *sock,struct page *page, -int offset,size_t size,int flags) -{ -int err; - - err =aead_check_key (sock); -if (err) -return err; - -return af_alg_sendpage (sock, page, offset, size, flags); -} -static int aead_recvmsg_nokey (struct socket *sock,struct msghdr *msg,size_t ignored,int flags) {
@@ -461,7 +448,6 @@ static struct proto_ops algif_aead_ops_nokey = { .release = af_alg_release, .sendmsg = aead_sendmsg_nokey, - .sendpage = aead_sendpage_nokey, .recvmsg = aead_recvmsg_nokey, .poll = af_alg_poll, };
crypto/algif_rng.c +0 -2
@@ -174,7 +174,6 @@ static struct proto_ops algif_rng_ops = { .bind = sock_no_bind, .accept = sock_no_accept, .sendmsg = sock_no_sendmsg, - .sendpage = sock_no_sendpage, .release = af_alg_release, .recvmsg = rng_recvmsg,
@@ -192,7 +191,6 @@ static struct proto_ops __maybe_unused algif_rng_test_ops = { .mmap = sock_no_mmap, .bind = sock_no_bind, .accept = sock_no_accept, - .sendpage = sock_no_sendpage, .release = af_alg_release, .recvmsg = rng_test_recvmsg,
crypto/algif_skcipher.c +0 -14
@@ -194,7 +194,6 @@ static struct proto_ops algif_skcipher_ops = { .release = af_alg_release, .sendmsg = skcipher_sendmsg, - .sendpage = af_alg_sendpage, .recvmsg = skcipher_recvmsg, .poll = af_alg_poll, };
@@ -246,18 +245,6 @@ static int skcipher_sendmsg_nokey(struct socket *sock, struct msghdr *msg,return skcipher_sendmsg (sock, msg, size); } -static ssize_t skcipher_sendpage_nokey (struct socket *sock,struct page *page, -int offset,size_t size,int flags) -{ -int err; - - err =skcipher_check_key (sock); -if (err) -return err; - -return af_alg_sendpage (sock, page, offset, size, flags); -} -static int skcipher_recvmsg_nokey (struct socket *sock,struct msghdr *msg,size_t ignored,int flags) {
@@ -285,7 +272,6 @@ static struct proto_ops algif_skcipher_ops_nokey = { .release = af_alg_release, .sendmsg = skcipher_sendmsg_nokey, - .sendpage = skcipher_sendpage_nokey, .recvmsg = skcipher_recvmsg_nokey, .poll = af_alg_poll, };
drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls.h +0 -2
@@ -569,8 +569,6 @@ int chtls_sendmsg(struct sock *sk, struct msghdr *msg, size_t size);int chtls_recvmsg (struct sock *sk,struct msghdr *msg,size_t len,int flags,int *addr_len);void chtls_splice_eof (struct socket *sock); -int chtls_sendpage (struct sock *sk,struct page *page, -int offset,size_t size,int flags);int send_tx_flowc_wr (struct sock *sk,int compl,u32 snd_nxt,u32 rcv_nxt);void chtls_tcp_push (struct sock *sk,int flags);
drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_io.c +0 -14
@@ -1246,20 +1246,6 @@ void chtls_splice_eof(struct socket *sock)release_sock (sk); } -int chtls_sendpage (struct sock *sk,struct page *page, -int offset,size_t size,int flags) -{ -struct msghdr msg = { .msg_flags = flags |MSG_SPLICE_PAGES , }; -struct bio_vec bvec; - -if (flags &MSG_SENDPAGE_NOTLAST ) - msg.msg_flags |=MSG_MORE ; - -bvec_set_page (&bvec, page, size, offset); -iov_iter_bvec (&msg.msg_iter,ITER_SOURCE , &bvec,1 , size); -return chtls_sendmsg (sk, &msg, size); -} -static void chtls_select_window (struct sock *sk) {struct chtls_sock *csk =rcu_dereference_sk_user_data (sk);
drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_main.c +0 -1
@@ -607,7 +607,6 @@ static void __init chtls_init_ulp_ops(void) chtls_cpl_prot.shutdown = chtls_shutdown; chtls_cpl_prot.sendmsg = chtls_sendmsg; chtls_cpl_prot.splice_eof = chtls_splice_eof; - chtls_cpl_prot.sendpage = chtls_sendpage; chtls_cpl_prot.recvmsg = chtls_recvmsg; chtls_cpl_prot.setsockopt = chtls_setsockopt; chtls_cpl_prot.getsockopt = chtls_getsockopt;
fs/nfsd/vfs.c +1 -1
@@ -936,7 +936,7 @@ nfsd_open_verified(struct svc_rqst *rqstp, struct svc_fh *fhp, int may_flags,/* * Grab and keep cached pages associated with a file in the svc_rqst -* so that they can be passed to the network sendmsg/sendpage routines +* so that they can be passed to the network sendmsg routines * directly. They will be released after the sending has completed. * * Return values: Number of bytes consumed, or -EIO if there are no
include/crypto/if_alg.h +0 -2
@@ -229,8 +229,6 @@ void af_alg_wmem_wakeup(struct sock *sk);int af_alg_wait_for_data (struct sock *sk,unsigned flags,unsigned min);int af_alg_sendmsg (struct socket *sock,struct msghdr *msg,size_t size,unsigned int ivsize); -ssize_t af_alg_sendpage (struct socket *sock,struct page *page, -int offset,size_t size,int flags);void af_alg_free_resources (struct af_alg_async_req *areq);void af_alg_async_cb (void *data,int err); __poll_t af_alg_poll(struct file *file, struct socket *sock,
include/linux/net.h +0 -8
@@ -207,8 +207,6 @@ struct proto_ops { size_t total_len, int flags);int (*mmap) (struct file *file,struct socket *sock,struct vm_area_struct * vma); -ssize_t (*sendpage) (struct socket *sock,struct page *page, -int offset,size_t size,int flags);ssize_t (*splice_read)(struct socket *sock,loff_t *ppos,struct pipe_inode_info *pipe,size_t len,unsigned int flags);void (*splice_eof)(struct socket *sock);
@@ -222,8 +220,6 @@ struct proto_ops { sk_read_actor_t recv_actor);/* This is different from read_sock(), it reads an entire skb at a time. */ int (*read_skb)(struct sock *sk,skb_read_actor_t recv_actor); -int (*sendpage_locked)(struct sock *sk,struct page *page, -int offset,size_t size,int flags);int (*sendmsg_locked)(struct sock *sk,struct msghdr *msg,size_t size);int (*set_rcvlowat)(struct sock *sk,int val);
@@ -341,10 +337,6 @@ int kernel_connect(struct socket *sock, struct sockaddr *addr, int addrlen, int flags);int kernel_getsockname (struct socket *sock,struct sockaddr *addr);int kernel_getpeername (struct socket *sock,struct sockaddr *addr); -int kernel_sendpage (struct socket *sock,struct page *page,int offset, -size_t size,int flags); -int kernel_sendpage_locked (struct sock *sk,struct page *page,int offset, -size_t size,int flags);int kernel_sock_shutdown (struct socket *sock,enum sock_shutdown_cmd how);/* Routine returns the IP overhead imposed by a (caller-protected) socket. */
include/net/inet_common.h +0 -2
@@ -36,8 +36,6 @@ void __inet_accept(struct socket *sock, struct socket *newsock,int inet_send_prepare (struct sock *sk);int inet_sendmsg (struct socket *sock,struct msghdr *msg,size_t size);void inet_splice_eof (struct socket *sock); -ssize_t inet_sendpage (struct socket *sock,struct page *page,int offset, -size_t size,int flags);int inet_recvmsg (struct socket *sock,struct msghdr *msg,size_t size,int flags);int inet_shutdown (struct socket *sock,int how);
include/net/sock.h +0 -6
@@ -1277,8 +1277,6 @@ struct proto { size_t len);int (*recvmsg)(struct sock *sk,struct msghdr *msg,size_t len,int flags,int *addr_len); -int (*sendpage)(struct sock *sk,struct page *page, -int offset,size_t size,int flags);void (*splice_eof)(struct socket *sock);int (*bind)(struct sock *sk,struct sockaddr *addr,int addr_len);
@@ -1919,10 +1917,6 @@ int sock_no_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t len);int sock_no_recvmsg (struct socket *,struct msghdr *,size_t ,int );int sock_no_mmap (struct file *file,struct socket *sock,struct vm_area_struct *vma); -ssize_t sock_no_sendpage (struct socket *sock,struct page *page,int offset, -size_t size,int flags); -ssize_t sock_no_sendpage_locked (struct sock *sk,struct page *page, -int offset,size_t size,int flags);/* * Functions to fill in entries in struct proto_ops when a protocol
include/net/tcp.h +0 -4
@@ -328,10 +328,6 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size);int tcp_sendmsg_fastopen (struct sock *sk,struct msghdr *msg,int *copied,size_t size,struct ubuf_info *uarg);void tcp_splice_eof (struct socket *sock); -int tcp_sendpage (struct sock *sk,struct page *page,int offset,size_t size, -int flags); -int tcp_sendpage_locked (struct sock *sk,struct page *page,int offset, -size_t size,int flags);int tcp_send_mss (struct sock *sk,int *size_goal,int flags);int tcp_wmem_schedule (struct sock *sk,int copy); void tcp_push(struct sock *sk, int flags, int mss_now, int nonagle,
net/appletalk/ddp.c +0 -1
@@ -1929,7 +1929,6 @@ static const struct proto_ops atalk_dgram_ops = { .sendmsg = atalk_sendmsg, .recvmsg = atalk_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, };static struct notifier_block ddp_notifier = {
net/atm/pvc.c +0 -1
@@ -126,7 +126,6 @@ static const struct proto_ops pvc_proto_ops = { .sendmsg = vcc_sendmsg, .recvmsg = vcc_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, };
net/atm/svc.c +0 -1
@@ -654,7 +654,6 @@ static const struct proto_ops svc_proto_ops = { .sendmsg = vcc_sendmsg, .recvmsg = vcc_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, };
net/ax25/af_ax25.c +0 -1
@@ -2022,7 +2022,6 @@ static const struct proto_ops ax25_proto_ops = { .sendmsg = ax25_sendmsg, .recvmsg = ax25_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, };/*
net/caif/caif_socket.c +0 -2
@@ -976,7 +976,6 @@ static const struct proto_ops caif_seqpacket_ops = { .sendmsg = caif_seqpkt_sendmsg, .recvmsg = caif_seqpkt_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, };static const struct proto_ops caif_stream_ops = {
@@ -996,7 +995,6 @@ static const struct proto_ops caif_stream_ops = { .sendmsg = caif_stream_sendmsg, .recvmsg = caif_stream_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, };/* This function is called when a socket is finally destroyed. */
net/can/bcm.c +0 -1
@@ -1703,7 +1703,6 @@ static const struct proto_ops bcm_ops = { .sendmsg = bcm_sendmsg, .recvmsg = bcm_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, };static struct proto bcm_proto__read_mostly = {
net/can/isotp.c +0 -1
@@ -1699,7 +1699,6 @@ static const struct proto_ops isotp_ops = { .sendmsg = isotp_sendmsg, .recvmsg = isotp_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, };static struct proto isotp_proto__read_mostly = {
net/can/j1939/socket.c +0 -1
@@ -1306,7 +1306,6 @@ static const struct proto_ops j1939_ops = { .sendmsg = j1939_sk_sendmsg, .recvmsg = j1939_sk_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, };static struct proto j1939_proto__read_mostly = {
net/can/raw.c +0 -1
@@ -962,7 +962,6 @@ static const struct proto_ops raw_ops = { .sendmsg = raw_sendmsg, .recvmsg = raw_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, };static struct proto raw_proto__read_mostly = {
net/core/sock.c +2 -33
@@ -3267,36 +3267,6 @@ void __receive_sock(struct file *file) } } -ssize_t sock_no_sendpage (struct socket *sock,struct page *page,int offset,size_t size,int flags) -{ -ssize_t res; -struct msghdr msg = {.msg_flags = flags}; -struct kvec iov; -char *kaddr =kmap (page); - iov.iov_base = kaddr + offset; - iov.iov_len = size; - res =kernel_sendmsg (sock, &msg, &iov,1 , size); -kunmap (page); -return res; -} -EXPORT_SYMBOL (sock_no_sendpage); - -ssize_t sock_no_sendpage_locked (struct sock *sk,struct page *page, -int offset,size_t size,int flags) -{ -ssize_t res; -struct msghdr msg = {.msg_flags = flags}; -struct kvec iov; -char *kaddr =kmap (page); - - iov.iov_base = kaddr + offset; - iov.iov_len = size; - res =kernel_sendmsg_locked (sk, &msg, &iov,1 , size); -kunmap (page); -return res; -} -EXPORT_SYMBOL (sock_no_sendpage_locked); -/* * Default Socket Callbacks */
@@ -4052,7 +4022,7 @@ static void proto_seq_printf(struct seq_file *seq, struct proto *proto) { seq_printf(seq, "%-9s %4u %6d %6ld %-3s %6u %-3s %-10s " - "%2c %2c %2c %2c %2c %2c %2c %2c %2c %2c %2c %2c %2c %2c %2c %2c %2c %2c %2c\n", + "%2c %2c %2c %2c %2c %2c %2c %2c %2c %2c %2c %2c %2c %2c %2c %2c %2c %2c\n", proto->name, proto->obj_size, sock_prot_inuse_get(seq_file_net(seq), proto),
@@ -4073,7 +4043,6 @@ static void proto_seq_printf(struct seq_file *seq, struct proto *proto)proto_method_implemented (proto->getsockopt),proto_method_implemented (proto->sendmsg),proto_method_implemented (proto->recvmsg), -proto_method_implemented (proto->sendpage),proto_method_implemented (proto->bind),proto_method_implemented (proto->backlog_rcv), proto_method_implemented(proto->hash),
@@ -4094,7 +4063,7 @@ static int proto_seq_show(struct seq_file *seq, void *v)"maxhdr" ,"slab" ,"module" , - "cl co di ac io in de sh ss gs se re sp bi br ha uh gp em\n"); + "cl co di ac io in de sh ss gs se re bi br ha uh gp em\n"); elseproto_seq_printf (seq ,list_entry (v ,struct proto ,node ));return 0 ;
net/dccp/ipv4.c +0 -1
@@ -1010,7 +1010,6 @@ static const struct proto_ops inet_dccp_ops = { .sendmsg = inet_sendmsg, .recvmsg = sock_common_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, };static struct inet_protosw dccp_v4_protosw = {
net/dccp/ipv6.c +0 -1
@@ -1087,7 +1087,6 @@ static const struct proto_ops inet6_dccp_ops = { .sendmsg = inet_sendmsg, .recvmsg = sock_common_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage,#ifdef CONFIG_COMPAT .compat_ioctl = inet6_compat_ioctl,#endif
net/ieee802154/socket.c +0 -2
@@ -426,7 +426,6 @@ static const struct proto_ops ieee802154_raw_ops = { .sendmsg = ieee802154_sock_sendmsg, .recvmsg = sock_common_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, };/* DGRAM Sockets (802.15.4 dataframes) */
@@ -989,7 +988,6 @@ static const struct proto_ops ieee802154_dgram_ops = { .sendmsg = ieee802154_sock_sendmsg, .recvmsg = sock_common_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, };static void ieee802154_sock_destruct (struct sock *sk)
net/ipv4/af_inet.c +0 -21
@@ -847,23 +847,6 @@ void inet_splice_eof(struct socket *sock) }EXPORT_SYMBOL_GPL (inet_splice_eof); -ssize_t inet_sendpage (struct socket *sock,struct page *page,int offset, -size_t size,int flags) -{ -struct sock *sk = sock->sk; -const struct proto *prot; - -if (unlikely (inet_send_prepare (sk))) -return -EAGAIN ; - -/* IPV6_ADDRFORM can change sk->sk_prot under us. */ - prot =READ_ONCE (sk->sk_prot); -if (prot->sendpage) -return prot->sendpage (sk, page, offset, size, flags); -return sock_no_sendpage (sock, page, offset, size, flags); -} -EXPORT_SYMBOL (inet_sendpage); - INDIRECT_CALLABLE_DECLARE(int udp_recvmsg(struct sock *, struct msghdr *,size_t ,int ,int *)); int inet_recvmsg(struct socket *sock, struct msghdr *msg, size_t size,
@@ -1067,12 +1050,10 @@ const struct proto_ops inet_stream_ops = { .mmap = tcp_mmap, #endif .splice_eof = inet_splice_eof, - .sendpage = inet_sendpage, .splice_read = tcp_splice_read, .read_sock = tcp_read_sock, .read_skb = tcp_read_skb, .sendmsg_locked = tcp_sendmsg_locked, - .sendpage_locked = tcp_sendpage_locked, .peek_len = tcp_peek_len, #ifdef CONFIG_COMPAT .compat_ioctl = inet_compat_ioctl,
@@ -1102,7 +1083,6 @@ const struct proto_ops inet_dgram_ops = { .recvmsg = inet_recvmsg, .mmap = sock_no_mmap, .splice_eof = inet_splice_eof, - .sendpage = inet_sendpage, .set_peek_off = sk_set_peek_off, #ifdef CONFIG_COMPAT .compat_ioctl = inet_compat_ioctl,
@@ -1134,7 +1114,6 @@ static const struct proto_ops inet_sockraw_ops = { .recvmsg = inet_recvmsg, .mmap = sock_no_mmap, .splice_eof = inet_splice_eof, - .sendpage = inet_sendpage,#ifdef CONFIG_COMPAT .compat_ioctl = inet_compat_ioctl,#endif
net/ipv4/tcp.c +4 -39
55 lines not on this page · raw message
net/ipv4/tcp_bpf.c +2 -21
@@ -482,7 +482,7 @@ static int tcp_bpf_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)long timeo;int flags; -/* Don't let internal sendpage flags through */ +/* Don't let internal flags through */ flags = (msg->msg_flags & ~MSG_SENDPAGE_DECRYPTED ); flags |=MSG_NO_SHARED_FRAGS ;
@@ -562,23 +562,6 @@ static int tcp_bpf_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)return copied ? copied : err; } -static int tcp_bpf_sendpage (struct sock *sk,struct page *page,int offset, -size_t size,int flags) -{ -struct bio_vec bvec; -struct msghdr msg = { - .msg_flags = flags |MSG_SPLICE_PAGES , - }; - -bvec_set_page (&bvec, page, size, offset); -iov_iter_bvec (&msg.msg_iter,ITER_SOURCE , &bvec,1 , size); - -if (flags &MSG_SENDPAGE_NOTLAST ) - msg.msg_flags |=MSG_MORE ; - -return tcp_bpf_sendmsg (sk, &msg, size); -} -enum {TCP_BPF_IPV4 ,TCP_BPF_IPV6 ,
@@ -608,7 +591,6 @@ static void tcp_bpf_rebuild_protos(struct proto prot[TCP_BPF_NUM_CFGS], prot[TCP_BPF_TX ] = prot[TCP_BPF_BASE ]; prot[TCP_BPF_TX ].sendmsg = tcp_bpf_sendmsg; - prot[TCP_BPF_TX ].sendpage = tcp_bpf_sendpage; prot[TCP_BPF_RX ] = prot[TCP_BPF_BASE ]; prot[TCP_BPF_RX ].recvmsg = tcp_bpf_recvmsg_parser;
@@ -643,8 +625,7 @@ static int tcp_bpf_assert_proto_ops(struct proto *ops)* indeed valid assumptions. */ return ops->recvmsg = tcp_recvmsg && - ops->sendmsg = tcp_sendmsg && - ops->sendpage = tcp_sendpage ?0 : -ENOTSUPP ; + ops->sendmsg = tcp_sendmsg ?0 : -ENOTSUPP ; }int tcp_bpf_update_proto (struct sock *sk,struct sk_psock *psock,bool restore)
net/ipv4/tcp_ipv4.c +0 -1
@@ -3117,7 +3117,6 @@ struct proto tcp_prot = { .recvmsg = tcp_recvmsg, .sendmsg = tcp_sendmsg, .splice_eof = tcp_splice_eof, - .sendpage = tcp_sendpage, .backlog_rcv = tcp_v4_do_rcv, .release_cb = tcp_release_cb, .hash = inet_hash,
net/ipv4/udp.c +0 -15
@@ -1340,20 +1340,6 @@ void udp_splice_eof(struct socket *sock) }EXPORT_SYMBOL_GPL (udp_splice_eof); -int udp_sendpage (struct sock *sk,struct page *page,int offset, -size_t size,int flags) -{ -struct bio_vec bvec; -struct msghdr msg = { .msg_flags = flags |MSG_SPLICE_PAGES }; - -if (flags &MSG_SENDPAGE_NOTLAST ) - msg.msg_flags |=MSG_MORE ; - -bvec_set_page (&bvec, page, size, offset); -iov_iter_bvec (&msg.msg_iter,ITER_SOURCE , &bvec,1 , size); -return udp_sendmsg (sk, &msg, size); -} -#define UDP_SKB_IS_STATELESS 0x80000000/* all head states (dst, sk, nf conntrack) except skb extensions are
@@ -2933,7 +2919,6 @@ struct proto udp_prot = { .sendmsg = udp_sendmsg, .recvmsg = udp_recvmsg, .splice_eof = udp_splice_eof, - .sendpage = udp_sendpage, .release_cb = ip4_datagram_release_cb, .hash = udp_lib_hash, .unhash = udp_lib_unhash,
net/ipv4/udp_impl.h +0 -2
@@ -19,8 +19,6 @@ int udp_getsockopt(struct sock *sk, int level, int optname,int udp_recvmsg (struct sock *sk,struct msghdr *msg,size_t len,int flags,int *addr_len); -int udp_sendpage (struct sock *sk,struct page *page,int offset,size_t size, -int flags);void udp_destroy_sock (struct sock *sk);#ifdef CONFIG_PROC_FS
net/ipv4/udplite.c +0 -1
@@ -56,7 +56,6 @@ struct proto udplite_prot = { .getsockopt = udp_getsockopt, .sendmsg = udp_sendmsg, .recvmsg = udp_recvmsg, - .sendpage = udp_sendpage, .hash = udp_lib_hash, .unhash = udp_lib_unhash, .rehash = udp_v4_rehash,
net/ipv6/af_inet6.c +0 -3
@@ -696,9 +696,7 @@ const struct proto_ops inet6_stream_ops = { .mmap = tcp_mmap, #endif .splice_eof = inet_splice_eof, - .sendpage = inet_sendpage, .sendmsg_locked = tcp_sendmsg_locked, - .sendpage_locked = tcp_sendpage_locked, .splice_read = tcp_splice_read, .read_sock = tcp_read_sock, .read_skb = tcp_read_skb,
@@ -729,7 +727,6 @@ const struct proto_ops inet6_dgram_ops = { .recvmsg = inet6_recvmsg, /* retpoline's sake */ .read_skb = udp_read_skb, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, .set_peek_off = sk_set_peek_off, #ifdef CONFIG_COMPAT .compat_ioctl = inet6_compat_ioctl,
net/ipv6/raw.c +0 -1
@@ -1296,7 +1296,6 @@ const struct proto_ops inet6_sockraw_ops = { .sendmsg = inet_sendmsg, /* ok */ .recvmsg = sock_common_recvmsg, /* ok */ .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage,#ifdef CONFIG_COMPAT .compat_ioctl = inet6_compat_ioctl,#endif
net/ipv6/tcp_ipv6.c +0 -1
@@ -2151,7 +2151,6 @@ struct proto tcpv6_prot = { .recvmsg = tcp_recvmsg, .sendmsg = tcp_sendmsg, .splice_eof = tcp_splice_eof, - .sendpage = tcp_sendpage, .backlog_rcv = tcp_v6_do_rcv, .release_cb = tcp_release_cb, .hash = inet6_hash,
net/kcm/kcmsock.c +0 -20
@@ -961,24 +961,6 @@ static void kcm_splice_eof(struct socket *sock)release_sock (sk); } -static ssize_t kcm_sendpage (struct socket *sock,struct page *page, -int offset,size_t size,int flags) - -{ -struct bio_vec bvec; -struct msghdr msg = { .msg_flags = flags |MSG_SPLICE_PAGES , }; - -if (flags &MSG_SENDPAGE_NOTLAST ) - msg.msg_flags |=MSG_MORE ; - -if (flags &MSG_OOB ) -return -EOPNOTSUPP ; - -bvec_set_page (&bvec, page, size, offset); -iov_iter_bvec (&msg.msg_iter,ITER_SOURCE , &bvec,1 , size); -return kcm_sendmsg (sock, &msg, size); -} -static int kcm_recvmsg (struct socket *sock,struct msghdr *msg,size_t len,int flags) {
@@ -1767,7 +1749,6 @@ static const struct proto_ops kcm_dgram_ops = { .recvmsg = kcm_recvmsg, .mmap = sock_no_mmap, .splice_eof = kcm_splice_eof, - .sendpage = kcm_sendpage, };static const struct proto_ops kcm_seqpacket_ops = {
@@ -1789,7 +1770,6 @@ static const struct proto_ops kcm_seqpacket_ops = { .recvmsg = kcm_recvmsg, .mmap = sock_no_mmap, .splice_eof = kcm_splice_eof, - .sendpage = kcm_sendpage, .splice_read = kcm_splice_read, };
net/key/af_key.c +0 -1
@@ -3761,7 +3761,6 @@ static const struct proto_ops pfkey_ops = { .listen = sock_no_listen, .shutdown = sock_no_shutdown, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage,/* Now the operations that really occur. */ .release = pfkey_release,
net/l2tp/l2tp_ip.c +0 -1
@@ -624,7 +624,6 @@ static const struct proto_ops l2tp_ip_ops = { .sendmsg = inet_sendmsg, .recvmsg = sock_common_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, };static struct inet_protosw l2tp_ip_protosw = {
net/l2tp/l2tp_ip6.c +0 -1
@@ -751,7 +751,6 @@ static const struct proto_ops l2tp_ip6_ops = { .sendmsg = inet_sendmsg, .recvmsg = sock_common_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage,#ifdef CONFIG_COMPAT .compat_ioctl = inet6_compat_ioctl,#endif
net/llc/af_llc.c +0 -1
@@ -1232,7 +1232,6 @@ static const struct proto_ops llc_ui_ops = { .sendmsg = llc_ui_sendmsg, .recvmsg = llc_ui_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, }; static const char llc_proc_err_msg[] __initconst diff --git a/net/mctp/af_mctp.c b/net/mctp/af_mctp.c
index bb4bd0b6a4f7..f6be58b68c6f 100644
net/mctp/af_mctp.c +0 -1
@@ -485,7 +485,6 @@ static const struct proto_ops mctp_dgram_ops = { .sendmsg = mctp_sendmsg, .recvmsg = mctp_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage,#ifdef CONFIG_COMPAT .compat_ioctl = mctp_compat_ioctl,#endif
net/mptcp/protocol.c +0 -2
@@ -3891,7 +3891,6 @@ static const struct proto_ops mptcp_stream_ops = { .sendmsg = inet_sendmsg, .recvmsg = inet_recvmsg, .mmap = sock_no_mmap, - .sendpage = inet_sendpage, };static struct inet_protosw mptcp_protosw = {
@@ -3986,7 +3985,6 @@ static const struct proto_ops mptcp_v6_stream_ops = { .sendmsg = inet6_sendmsg, .recvmsg = inet6_recvmsg, .mmap = sock_no_mmap, - .sendpage = inet_sendpage,#ifdef CONFIG_COMPAT .compat_ioctl = inet6_compat_ioctl,#endif
net/netlink/af_netlink.c +0 -1
@@ -2815,7 +2815,6 @@ static const struct proto_ops netlink_ops = { .sendmsg = netlink_sendmsg, .recvmsg = netlink_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, };static const struct net_proto_family netlink_family_ops = {
net/netrom/af_netrom.c +0 -1
@@ -1364,7 +1364,6 @@ static const struct proto_ops nr_proto_ops = { .sendmsg = nr_sendmsg, .recvmsg = nr_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, };static struct notifier_block nr_dev_notifier = {
net/packet/af_packet.c +0 -2
@@ -4621,7 +4621,6 @@ static const struct proto_ops packet_ops_spkt = { .sendmsg = packet_sendmsg_spkt, .recvmsg = packet_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, };static const struct proto_ops packet_ops = {
@@ -4643,7 +4642,6 @@ static const struct proto_ops packet_ops = { .sendmsg = packet_sendmsg, .recvmsg = packet_recvmsg, .mmap = packet_mmap, - .sendpage = sock_no_sendpage, };static const struct net_proto_family packet_family_ops = {
net/phonet/socket.c +0 -2
@@ -441,7 +441,6 @@ const struct proto_ops phonet_dgram_ops = { .sendmsg = pn_socket_sendmsg, .recvmsg = sock_common_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, };const struct proto_ops phonet_stream_ops = {
@@ -462,7 +461,6 @@ const struct proto_ops phonet_stream_ops = { .sendmsg = pn_socket_sendmsg, .recvmsg = sock_common_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, };EXPORT_SYMBOL (phonet_stream_ops);
net/qrtr/af_qrtr.c +0 -1
@@ -1244,7 +1244,6 @@ static const struct proto_ops qrtr_proto_ops = { .shutdown = sock_no_shutdown, .release = qrtr_release, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, };static struct proto qrtr_proto = {
net/rds/af_rds.c +0 -1
@@ -653,7 +653,6 @@ static const struct proto_ops rds_proto_ops = { .sendmsg = rds_sendmsg, .recvmsg = rds_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, };static void rds_sock_destruct (struct sock *sk)
net/rose/af_rose.c +0 -1
@@ -1496,7 +1496,6 @@ static const struct proto_ops rose_proto_ops = { .sendmsg = rose_sendmsg, .recvmsg = rose_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, };static struct notifier_block rose_dev_notifier = {
net/rxrpc/af_rxrpc.c +0 -1
@@ -954,7 +954,6 @@ static const struct proto_ops rxrpc_rpc_ops = { .sendmsg = rxrpc_sendmsg, .recvmsg = rxrpc_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, };static struct proto rxrpc_proto = {
net/sctp/protocol.c +0 -1
@@ -1133,7 +1133,6 @@ static const struct proto_ops inet_seqpacket_ops = { .sendmsg = inet_sendmsg, .recvmsg = inet_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, };/* Registration with AF_INET family. */
net/socket.c +0 -48
54 lines not on this page · raw message
net/tipc/socket.c +0 -3
@@ -3375,7 +3375,6 @@ static const struct proto_ops msg_ops = { .sendmsg = tipc_sendmsg, .recvmsg = tipc_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage };static const struct proto_ops packet_ops = {
@@ -3396,7 +3395,6 @@ static const struct proto_ops packet_ops = { .sendmsg = tipc_send_packet, .recvmsg = tipc_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage };static const struct proto_ops stream_ops = {
@@ -3417,7 +3415,6 @@ static const struct proto_ops stream_ops = { .sendmsg = tipc_sendstream, .recvmsg = tipc_recvstream, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage };static const struct net_proto_family tipc_family_ops = {
net/tls/tls.h +0 -6
@@ -98,10 +98,6 @@ void tls_sw_strparser_arm(struct sock *sk, struct tls_context *ctx);void tls_sw_strparser_done (struct tls_context *tls_ctx);int tls_sw_sendmsg (struct sock *sk,struct msghdr *msg,size_t size);void tls_sw_splice_eof (struct socket *sock); -int tls_sw_sendpage_locked (struct sock *sk,struct page *page, -int offset,size_t size,int flags); -int tls_sw_sendpage (struct sock *sk,struct page *page, -int offset,size_t size,int flags);void tls_sw_cancel_work_tx (struct tls_context *tls_ctx);void tls_sw_release_resources_tx (struct sock *sk);void tls_sw_free_ctx_tx (struct tls_context *tls_ctx);
@@ -117,8 +113,6 @@ ssize_t tls_sw_splice_read(struct socket *sock, loff_t *ppos,int tls_device_sendmsg (struct sock *sk,struct msghdr *msg,size_t size);void tls_device_splice_eof (struct socket *sock); -int tls_device_sendpage (struct sock *sk,struct page *page, -int offset,size_t size,int flags);int tls_tx_records (struct sock *sk,int flags);void tls_sw_write_space (struct sock *sk,struct tls_context *ctx);
net/tls/tls_device.c +0 -17
@@ -621,23 +621,6 @@ void tls_device_splice_eof(struct socket *sock)mutex_unlock (&tls_ctx->tx_lock); } -int tls_device_sendpage (struct sock *sk,struct page *page, -int offset,size_t size,int flags) -{ -struct bio_vec bvec; -struct msghdr msg = { .msg_flags = flags |MSG_SPLICE_PAGES , }; - -if (flags &MSG_SENDPAGE_NOTLAST ) - msg.msg_flags |=MSG_MORE ; - -if (flags &MSG_OOB ) -return -EOPNOTSUPP ; - -bvec_set_page (&bvec, page, size, offset); -iov_iter_bvec (&msg.msg_iter,ITER_SOURCE , &bvec,1 , size); -return tls_device_sendmsg (sk, &msg, size); -} -struct tls_record_info *tls_get_record (struct tls_offload_context_tx *context,u32 seq,u64 *p_record_sn) {
net/tls/tls_main.c +0 -7
@@ -958,7 +958,6 @@ static void build_proto_ops(struct proto_ops ops[TLS_NUM_CONFIG][TLS_NUM_CONFIG] ops[TLS_SW ][TLS_BASE ] = ops[TLS_BASE ][TLS_BASE ]; ops[TLS_SW ][TLS_BASE ].splice_eof = tls_sw_splice_eof; - ops[TLS_SW ][TLS_BASE ].sendpage_locked = tls_sw_sendpage_locked; ops[TLS_BASE ][TLS_SW ] = ops[TLS_BASE ][TLS_BASE ]; ops[TLS_BASE ][TLS_SW ].splice_read = tls_sw_splice_read;
@@ -970,17 +969,14 @@ static void build_proto_ops(struct proto_ops ops[TLS_NUM_CONFIG][TLS_NUM_CONFIG]#ifdef CONFIG_TLS_DEVICE ops[TLS_HW ][TLS_BASE ] = ops[TLS_BASE ][TLS_BASE ]; - ops[TLS_HW ][TLS_BASE ].sendpage_locked =NULL ; ops[TLS_HW ][TLS_SW ] = ops[TLS_BASE ][TLS_SW ]; - ops[TLS_HW ][TLS_SW ].sendpage_locked =NULL ; ops[TLS_BASE ][TLS_HW ] = ops[TLS_BASE ][TLS_SW ]; ops[TLS_SW ][TLS_HW ] = ops[TLS_SW ][TLS_SW ]; ops[TLS_HW ][TLS_HW ] = ops[TLS_HW ][TLS_SW ]; - ops[TLS_HW ][TLS_HW ].sendpage_locked =NULL ;#endif #ifdef CONFIG_TLS_TOE ops[TLS_HW_RECORD ][TLS_HW_RECORD ] = *base;
@@ -1029,7 +1025,6 @@ static void build_protos(struct proto prot[TLS_NUM_CONFIG][TLS_NUM_CONFIG], prot[TLS_SW ][TLS_BASE ] = prot[TLS_BASE ][TLS_BASE ]; prot[TLS_SW ][TLS_BASE ].sendmsg = tls_sw_sendmsg; prot[TLS_SW ][TLS_BASE ].splice_eof = tls_sw_splice_eof; - prot[TLS_SW ][TLS_BASE ].sendpage = tls_sw_sendpage; prot[TLS_BASE ][TLS_SW ] = prot[TLS_BASE ][TLS_BASE ]; prot[TLS_BASE ][TLS_SW ].recvmsg = tls_sw_recvmsg;
@@ -1045,12 +1040,10 @@ static void build_protos(struct proto prot[TLS_NUM_CONFIG][TLS_NUM_CONFIG], prot[TLS_HW ][TLS_BASE ] = prot[TLS_BASE ][TLS_BASE ]; prot[TLS_HW ][TLS_BASE ].sendmsg = tls_device_sendmsg; prot[TLS_HW ][TLS_BASE ].splice_eof = tls_device_splice_eof; - prot[TLS_HW ][TLS_BASE ].sendpage = tls_device_sendpage; prot[TLS_HW ][TLS_SW ] = prot[TLS_BASE ][TLS_SW ]; prot[TLS_HW ][TLS_SW ].sendmsg = tls_device_sendmsg; prot[TLS_HW ][TLS_SW ].splice_eof = tls_device_splice_eof; - prot[TLS_HW ][TLS_SW ].sendpage = tls_device_sendpage; prot[TLS_BASE ][TLS_HW ] = prot[TLS_BASE ][TLS_SW ];
net/tls/tls_sw.c +0 -35
@@ -1281,41 +1281,6 @@ void tls_sw_splice_eof(struct socket *sock)mutex_unlock (&tls_ctx->tx_lock); } -int tls_sw_sendpage_locked (struct sock *sk,struct page *page, -int offset,size_t size,int flags) -{ -struct bio_vec bvec; -struct msghdr msg = { .msg_flags = flags |MSG_SPLICE_PAGES , }; - -if (flags & ~(MSG_MORE |MSG_DONTWAIT |MSG_NOSIGNAL | -MSG_SENDPAGE_NOTLAST |MSG_SENDPAGE_NOPOLICY | -MSG_NO_SHARED_FRAGS )) -return -EOPNOTSUPP ; -if (flags &MSG_SENDPAGE_NOTLAST ) - msg.msg_flags |=MSG_MORE ; - -bvec_set_page (&bvec, page, size, offset); -iov_iter_bvec (&msg.msg_iter,ITER_SOURCE , &bvec,1 , size); -return tls_sw_sendmsg_locked (sk, &msg, size); -} - -int tls_sw_sendpage (struct sock *sk,struct page *page, -int offset,size_t size,int flags) -{ -struct bio_vec bvec; -struct msghdr msg = { .msg_flags = flags |MSG_SPLICE_PAGES , }; - -if (flags & ~(MSG_MORE |MSG_DONTWAIT |MSG_NOSIGNAL | -MSG_SENDPAGE_NOTLAST |MSG_SENDPAGE_NOPOLICY )) -return -EOPNOTSUPP ; -if (flags &MSG_SENDPAGE_NOTLAST ) - msg.msg_flags |=MSG_MORE ; - -bvec_set_page (&bvec, page, size, offset); -iov_iter_bvec (&msg.msg_iter,ITER_SOURCE , &bvec,1 , size); -return tls_sw_sendmsg (sk, &msg, size); -} -static int tls_rx_rec_wait (struct sock *sk,struct sk_psock *psock,bool nonblock,bool released)
net/unix/af_unix.c +0 -19
@@ -758,8 +758,6 @@ static int unix_compat_ioctl(struct socket *sock, unsigned int cmd, unsigned lonstatic int unix_shutdown (struct socket *,int );static int unix_stream_sendmsg (struct socket *,struct msghdr *,size_t );static int unix_stream_recvmsg (struct socket *,struct msghdr *,size_t ,int ); -static ssize_t unix_stream_sendpage (struct socket *,struct page *,int offset, -size_t size,int flags);static ssize_t unix_stream_splice_read (struct socket *,loff_t *ppos,struct pipe_inode_info *,size_t size,unsigned int flags);
@@ -852,7 +850,6 @@ static const struct proto_ops unix_stream_ops = { .recvmsg = unix_stream_recvmsg, .read_skb = unix_stream_read_skb, .mmap = sock_no_mmap, - .sendpage = unix_stream_sendpage, .splice_read = unix_stream_splice_read, .set_peek_off = unix_set_peek_off, .show_fdinfo = unix_show_fdinfo,
@@ -878,7 +875,6 @@ static const struct proto_ops unix_dgram_ops = { .read_skb = unix_read_skb, .recvmsg = unix_dgram_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, .set_peek_off = unix_set_peek_off, .show_fdinfo = unix_show_fdinfo, };
@@ -902,7 +898,6 @@ static const struct proto_ops unix_seqpacket_ops = { .sendmsg = unix_seqpacket_sendmsg, .recvmsg = unix_seqpacket_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, .set_peek_off = unix_set_peek_off, .show_fdinfo = unix_show_fdinfo, };
@@ -2294,20 +2289,6 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg,return sent ? : err; } -static ssize_t unix_stream_sendpage (struct socket *socket,struct page *page, -int offset,size_t size,int flags) -{ -struct bio_vec bvec; -struct msghdr msg = { .msg_flags = flags |MSG_SPLICE_PAGES }; - -if (flags &MSG_SENDPAGE_NOTLAST ) - msg.msg_flags |=MSG_MORE ; - -bvec_set_page (&bvec, page, size, offset); -iov_iter_bvec (&msg.msg_iter,ITER_SOURCE , &bvec,1 , size); -return unix_stream_sendmsg (socket, &msg, size); -} -static int unix_seqpacket_sendmsg (struct socket *sock,struct msghdr *msg,size_t len) {
net/vmw_vsock/af_vsock.c +0 -3
@@ -1306,7 +1306,6 @@ static const struct proto_ops vsock_dgram_ops = { .sendmsg = vsock_dgram_sendmsg, .recvmsg = vsock_dgram_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, .read_skb = vsock_read_skb, };
@@ -2234,7 +2233,6 @@ static const struct proto_ops vsock_stream_ops = { .sendmsg = vsock_connectible_sendmsg, .recvmsg = vsock_connectible_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, .set_rcvlowat = vsock_set_rcvlowat, .read_skb = vsock_read_skb, };
@@ -2257,7 +2255,6 @@ static const struct proto_ops vsock_seqpacket_ops = { .sendmsg = vsock_connectible_sendmsg, .recvmsg = vsock_connectible_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, .read_skb = vsock_read_skb, };
net/x25/af_x25.c +0 -1
@@ -1757,7 +1757,6 @@ static const struct proto_ops x25_proto_ops = { .sendmsg = x25_sendmsg, .recvmsg = x25_recvmsg, .mmap = sock_no_mmap, - .sendpage = sock_no_sendpage, };static struct packet_type x25_packet_type__read_mostly = {
net/xdp/xsk.c +0 -1
@@ -1389,7 +1389,6 @@ static const struct proto_ops xsk_proto_ops = { .sendmsg = xsk_sendmsg, .recvmsg = xsk_recvmsg, .mmap = xsk_mmap, - .sendpage = sock_no_sendpage, };static void xsk_destruct (struct sock *sk)
David Howells
maintainerPATCH 17/17 net: Kill MSG_SENDPAGE_NOTLAST
In reply to David Howells--- include/linux/socket.h | 4 +--- net/ipv4/tcp_bpf.c | 4 +++- net/tls/tls_device.c | 3 +-- net/tls/tls_main.c | 2 +- net/tls/tls_sw.c | 2 +- tools/perf/trace/beauty/include/linux/socket.h | 1 - tools/perf/trace/beauty/msg_flags.c | 3 --- 7 files changed, 7 insertions(+), 12 deletions(-)
include/linux/socket.h +1 -3
@@ -319,7 +319,6 @@ struct ucred {#define MSG_MORE 0x8000/* Sender will send more */ #define MSG_WAITFORONE 0x10000/* recvmmsg(): block until 1+ packets avail */ #define MSG_SENDPAGE_NOPOLICY 0x10000/* sendpage() internal : do no apply policy */ -#define MSG_SENDPAGE_NOTLAST 0x20000/* sendpage() internal : not the last page */ #define MSG_BATCH 0x40000/* sendmmsg(): more messages coming */ #define MSG_EOF MSG_FIN#define MSG_NO_SHARED_FRAGS 0x80000/* sendpage() internal : page frags are not shared */
@@ -341,8 +340,7 @@ struct ucred {/* Flags to be cleared on entry by sendmsg and sendmmsg syscalls */ #define MSG_INTERNAL_SENDMSG_FLAGS \ - (MSG_SPLICE_PAGES | MSG_SENDPAGE_NOPOLICY | MSG_SENDPAGE_NOTLAST | \ - MSG_SENDPAGE_DECRYPTED) + (MSG_SPLICE_PAGES | MSG_SENDPAGE_NOPOLICY | MSG_SENDPAGE_DECRYPTED)/* Setsockoptions(2) level. Thanks to BSD these must match IPPROTO_xxx */ #define SOL_IP 0
net/ipv4/tcp_bpf.c +3 -1
@@ -90,7 +90,9 @@ static int tcp_bpf_push(struct sock *sk, struct sk_msg *msg, u32 apply_bytes, {bool apply = apply_bytes;struct scatterlist *sge; -struct msghdr msghdr = { .msg_flags = flags |MSG_SPLICE_PAGES , }; +struct msghdr msghdr = { + .msg_flags = flags |MSG_SPLICE_PAGES |MSG_MORE , + };struct page *page;int size, ret =0 ;u32 off;
net/tls/tls_device.c +1 -2
@@ -441,8 +441,7 @@ static int tls_push_data(struct sock *sk,long timeo;if (flags & - ~(MSG_MORE |MSG_DONTWAIT |MSG_NOSIGNAL |MSG_SENDPAGE_NOTLAST | -MSG_SPLICE_PAGES )) + ~(MSG_MORE |MSG_DONTWAIT |MSG_NOSIGNAL |MSG_SPLICE_PAGES ))return -EOPNOTSUPP ;if (unlikely (sk->sk_err))
net/tls/tls_main.c +1 -1
@@ -127,7 +127,7 @@ int tls_push_sg(struct sock *sk, {struct bio_vec bvec;struct msghdr msg = { - .msg_flags =MSG_SENDPAGE_NOTLAST |MSG_SPLICE_PAGES | flags, + .msg_flags =MSG_SPLICE_PAGES | flags, };int ret =0 ;struct page *p;
net/tls/tls_sw.c +1 -1
@@ -1194,7 +1194,7 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)if (msg->msg_flags & ~(MSG_MORE |MSG_DONTWAIT |MSG_NOSIGNAL |MSG_CMSG_COMPAT |MSG_SPLICE_PAGES | -MSG_SENDPAGE_NOTLAST |MSG_SENDPAGE_NOPOLICY )) +MSG_SENDPAGE_NOPOLICY ))return -EOPNOTSUPP ; ret =mutex_lock_interruptible (&tls_ctx->tx_lock);
tools/perf/trace/beauty/include/linux/socket.h +0 -1
@@ -318,7 +318,6 @@ struct ucred {#define MSG_MORE 0x8000/* Sender will send more */ #define MSG_WAITFORONE 0x10000/* recvmmsg(): block until 1+ packets avail */ #define MSG_SENDPAGE_NOPOLICY 0x10000/* sendpage() internal : do no apply policy */ -#define MSG_SENDPAGE_NOTLAST 0x20000/* sendpage() internal : not the last page */ #define MSG_BATCH 0x40000/* sendmmsg(): more messages coming */ #define MSG_EOF MSG_FIN#define MSG_NO_SHARED_FRAGS 0x80000/* sendpage() internal : page frags are not shared */
tools/perf/trace/beauty/msg_flags.c +0 -3
@@ -8,9 +8,6 @@#ifndef MSG_WAITFORONE #define MSG_WAITFORONE 0x10000#endif -#ifndef MSG_SENDPAGE_NOTLAST -#define MSG_SENDPAGE_NOTLAST 0x20000 -#endif #ifndef MSG_FASTOPEN #define MSG_FASTOPEN 0x20000000#endif