net: datagram: Bypass usercopy checks for kernel iterators
Subsystems (4)
PATCH 1/1 net: datagram: Bypass usercopy checks for kernel iterators
From: Chuck Lever <[email protected]>
Profiling NFSD under an iozone workload showed that hardened usercopy checks consume roughly 1.3% of CPU in the TCP receive path. These checks validate memory regions during copies, but provide no security benefit when both source (skb data) and destination (kernel pages in BVEC/KVEC iterators) reside in kernel address space.
Modify simple_copy_to_iter() and crc32c_and_copy_to_iter() to call _copy_to_iter() directly when the destination is a kernel-only iterator, bypassing the usercopy hardening validation. User-backed iterators (ITER_UBUF, ITER_IOVEC) continue to use copy_to_iter() with full validation.
This benefits kernel consumers of TCP receive such as the NFS client and server and NVMe-TCP, which use ITER_BVEC for their receive buffers.
--- net/core/datagram.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
net/core/datagram.c +12 -2
@@ -490,7 +490,10 @@ static size_t crc32c_and_copy_to_iter(const void *addr, size_t bytes,u32 *crcp = _crcp;size_t copied; - copied =copy_to_iter (addr, bytes, i); +if (user_backed_iter (i)) + copied =copy_to_iter (addr, bytes, i); +else + copied =_copy_to_iter (addr, bytes, i); *crcp =crc32c (*crcp, addr, copied);return copied; }
@@ -515,10 +518,17 @@ int skb_copy_and_crc32c_datagram_iter(const struct sk_buff *skb, int offset,EXPORT_SYMBOL (skb_copy_and_crc32c_datagram_iter);#endif /* CONFIG_NET_CRC32C */ +/* +* Bypass usercopy hardening for kernel-only iterators: no data +* crosses the user/kernel boundary, so the slab whitelist check +* on the source buffer is unnecessary overhead. +*/ static size_t simple_copy_to_iter (const void *addr,size_t bytes,void *data__always_unused ,struct iov_iter *i) { -return copy_to_iter (addr, bytes, i); +if (user_backed_iter (i)) +return copy_to_iter (addr, bytes, i); +return _copy_to_iter (addr, bytes, i); }/**
Signature · 1 line
2.52.0
Jakub Kicinski
maintainerQuoted 14 lines · On Wed, 25 Feb 2026 11:25:32 -0500 Chuck Lever wrote:
From: Chuck Lever <[email protected]>
Profiling NFSD under an iozone workload showed that hardened usercopy checks consume roughly 1.3% of CPU in the TCP receive path. These checks validate memory regions during copies, but provide no security benefit when both source (skb data) and destination (kernel pages in BVEC/KVEC iterators) reside in kernel address space.
Modify
simple_copy_to_iter()andcrc32c_and_copy_to_iter()to call_copy_to_iter()directly when the destination is a kernel-only iterator, bypassing the usercopy hardening validation. User-backed iterators (ITER_UBUF,ITER_IOVEC) continue to usecopy_to_iter()with full validation.
Quoted 3 lines
This benefits kernel consumers of TCP receive such as the NFS client and server and NVMe-TCP, which use
ITER_BVECfor their receive buffers.
If it makes such a difference why not make copy_to_iter() check the iter type? Why force callers to check it?
Quoted 35 lines
net/core/datagram.c +12
-2@@ -490,7 +490,10 @@ static size_t crc32c_and_copy_to_iter(const void *addr, size_t bytes,u32 *crcp = _crcp;size_t copied; - copied =copy_to_iter (addr, bytes, i); +if (user_backed_iter (i)) + copied =copy_to_iter (addr, bytes, i); +else + copied =_copy_to_iter (addr, bytes, i); *crcp =crc32c (*crcp, addr, copied);return copied; }@@ -515,10 +518,17 @@ int skb_copy_and_crc32c_datagram_iter(const struct sk_buff *skb, int offset,EXPORT_SYMBOL (skb_copy_and_crc32c_datagram_iter);#endif /* CONFIG_NET_CRC32C */ +/* +* Bypass usercopy hardening for kernel-only iterators: no data +* crosses the user/kernel boundary, so the slab whitelist check +* on the source buffer is unnecessary overhead. +*/ static size_t simple_copy_to_iter (const void *addr,size_t bytes,void *data__always_unused ,struct iov_iter *i) { -return copy_to_iter (addr, bytes, i); +if (user_backed_iter (i)) +return copy_to_iter (addr, bytes, i); +return _copy_to_iter (addr, bytes, i); }/**