saner calling conventions for csum-and-copy primitives
Subsystems (23)
Al Viro
maintainerRFC PATCH 1/1 saner calling conventions for csum-and-copy primitives
We have 3 per-architecture primitives that copy data and return
the checksum. One is csum_partial_copy_nocheck() (kernel-to-kernel), other two - csum_and_copy_from_user() and csum_and_copy_to_user().
There are default implementations, but for quite a few architectures
these are done in assembler of varying unpleasantness. The calling conventions are due to a large pile of historical accidents; right now they are
__wsum csum_partial_copy_nocheck(src, dst, len, initial_sum):
copy len bytes of data from src to dst, return something that is
comparable mod 65535 with checksum of that data added to initial_sum. As always, __wsum values are defined only modulo 65535 - different kernel configs can yield different 32bit values on the same data, identical blocks of data at different address can yield different 32bit values (on the same kernel), etc. Relatively few call sites.
__wsum csum_and_copy_from_user(src, dst, len, initial_sum, errp)
copy len bytes of data from src (in userspace) to dst. In case
we can't copy the entire thing, set *errp to -EFAULT. Otherwise *errp is left unmodified and we return something that is comparable mod 65535 with the checksum of that data added to initial_sum. Only two call sites (both in lib/iov_iter.c). In case of an error, the copied data is discarded, along with the return value.
__wsum csum_and_copy_to_user(src, dst, len, initial_sum, errp)
copy len bytes of data from src to dst (in userspace). In case
we can't copy the entire thing, set *errp to -EFAULT. Otherwise *errp is left unmodified and we return something that is comparable mod 65535 with the checksum of that data added to initial_sum. Only one call site (in lib/iov_iter.c). In case of an error, the copied data is discarded, along with the return value.
The guts of these primitives are at the very least similar to each other; on architectures with common address space for kernel and userland all three are often implemented via a single asm helper. Unfortunately, the exception handlers are overcomplicated; if nothing else, they need to pass a pointer to store -EFAULT into and some instances are trying to zero the rest of destination (or the entire destination) when we fail to fetch some data.
Note, BTW, that "userspace" in the above is real userspace - we never have csum_and_copy_..._user() called under KERNEL_DS.
It's far too convoluted and that code had been accumulated a lot of cruft over the years - for example, I'm fairly certain that nobody has read amd64 instances through since about 2005.
It's not that hard to untangle, though. First of all, initial_sum part is pointless - there is only one caller that ever passes something other than zero and that one is easy to massage so that it, too, would pass zero (it sums and copies the fragments attached to skb, then does the same to the main part; doing the main part first gets rid of the problem).
Furthermore, using 0xffffffff instead of 0 will yield a non-zero value comparable mod 0xffff with the original one, due to the way csum_add() works; the same goes for its open-coded asm equivalents, as well as various "folding" primitives.
That allows to use a simpler method of reporting an error - simply return zero. I.e.__wsum csum_partial_copy_nocheck(src, dst, len)
copy len bytes of data from src to dst, return something that is
comparable mod 65535 with checksum of that data.
__wsum csum_and_copy_from_user(src, dst, len)
copy len bytes of data from src (in userspace) to dst. In case
we can't copy the entire thing, return 0. Otherwise return something non-zero that is comparable mod 65535 with the checksum of that data.
__wsum csum_and_copy_to_user(src, dst, len)
copy len bytes of data from src to dst (in userspace). In case
we can't copy the entire thing, return 0. Otherwise return something non-zero that is comparable mod 65535 with the checksum of that data.
Exception handlers become trivial that way, of course, and quite a few gross hacks in them go away.
The branch is based at 5.8-rc1 and can be found in git.kernel.org/pub/scm/linux/kernel/git/viro/vfs.git work.csum_and_copy Individual patches will go in followups.
First we get rid of passing the initial sum for csum_partial_copy_nocheck():
skb_copy_and_csum_bits(): don't bother with the last argument icmp_push_reply(): reorder adding the checksum up csum_partial_copy_nocheck(): drop the last argument
Next comes the minimal conversion of csum_and_copy_..._user() to new calling conventions. Asm parts are left unchanged at that point, which makes for
a reasonably small patch. We could split that into per-architecture parts,
but IMO it's better done that way - no need of temporary "has this architecture already switched" config symbols, etc.
csum_and_copy_..._user(): pass 0xffffffff instead of 0 as initial sum
saner calling conventions for csum_and_copy_..._user()
Finally, we propagate the change of calling conventions down into the asm
helpers. That's done on per-architecture basis (and only for the architectures that are not using the default instances, of course).
alpha: propagate the calling convention changes down to csum_partial_copy.c helpers
arm: propagate the calling convention changes down to csum_partial_copy_from_user()
m68k: get rid of zeroing destination on error in csum_and_copy_from_user()
sh: propage the calling conventions change down to csum_partial_copy_generic()
i386: propagate the calling conventions change down to csum_partial_copy_generic()
sparc32: propagate the calling conventions change down to __csum_partial_copy_sparc_generic()
mips: csum_and_copy_{to,from}_user() are never called under KERNEL_DS
mips: __csum_partial_copy_kernel() has no users left
mips: propagate the calling convention change down into __csum_partial_copy_..._user()
xtensa: propagate the calling conventions change down into csum_partial_copy_generic()
sparc64: propagate the calling convention changes down to __csum_partial_copy_...()
amd64: switch csum_partial_copy_generic() to new calling conventions
ppc: propagate the calling conventions change down to csum_partial_copy_generic()
Only lightly tested; a lot of asm shite is killed off, so I would really like the architecture maintainers to look through that thing. Please, review.
Diffstat:
arch/alpha/include/asm/checksum.h | 4 +- arch/alpha/lib/csum_partial_copy.c | 164 ++++++++----------- arch/arm/include/asm/checksum.h | 16 +- arch/arm/lib/csumpartialcopy.S | 4 +- arch/arm/lib/csumpartialcopygeneric.S | 1 + arch/arm/lib/csumpartialcopyuser.S | 26 +-- arch/hexagon/include/asm/checksum.h | 3 +- arch/hexagon/lib/checksum.c | 4 +- arch/ia64/include/asm/checksum.h | 3 +- arch/ia64/lib/csum_partial_copy.c | 4 +- arch/m68k/include/asm/checksum.h | 6 +- arch/m68k/lib/checksum.c | 88 +++------- arch/mips/include/asm/checksum.h | 66 ++------ arch/mips/lib/csum_partial.S | 261 ++++++++++-------------------- arch/nios2/include/asm/checksum.h | 4 +- arch/parisc/include/asm/checksum.h | 22 +-- arch/parisc/lib/checksum.c | 7 +- arch/powerpc/include/asm/checksum.h | 12 +- arch/powerpc/lib/checksum_32.S | 74 ++++----- arch/powerpc/lib/checksum_64.S | 37 ++--- arch/powerpc/lib/checksum_wrappers.c | 74 ++------- arch/s390/include/asm/checksum.h | 4 +- arch/sh/include/asm/checksum_32.h | 35 ++-- arch/sh/lib/checksum.S | 119 ++++---------- arch/sparc/include/asm/checksum.h | 1 + arch/sparc/include/asm/checksum_32.h | 70 ++------ arch/sparc/include/asm/checksum_64.h | 39 +---- arch/sparc/lib/checksum_32.S | 202 +++++------------------ arch/sparc/lib/csum_copy.S | 3 +- arch/sparc/lib/csum_copy_from_user.S | 4 +- arch/sparc/lib/csum_copy_to_user.S | 4 +- arch/sparc/mm/fault_32.c | 6 +- arch/x86/include/asm/checksum_32.h | 40 ++--- arch/x86/include/asm/checksum_64.h | 14 +- arch/x86/lib/checksum_32.S | 117 +++++--------- arch/x86/lib/csum-copy_64.S | 140 +++++++++------- arch/x86/lib/csum-wrappers_64.c | 86 ++-------- arch/x86/um/asm/checksum.h | 5 +- arch/x86/um/asm/checksum_32.h | 23 --- arch/xtensa/include/asm/checksum.h | 33 ++-- arch/xtensa/lib/checksum.S | 67 ++------ drivers/net/ethernet/3com/typhoon.c | 3 +- drivers/net/ethernet/sun/sunvnet_common.c | 2 +- include/asm-generic/checksum.h | 4 +- include/linux/skbuff.h | 2 +- include/net/checksum.h | 15 +- lib/iov_iter.c | 21 ++- net/core/skbuff.c | 13 +- net/ipv4/icmp.c | 10 +- net/ipv4/ip_output.c | 6 +- net/ipv4/raw.c | 2 +- net/ipv6/icmp.c | 4 +- net/ipv6/ip6_output.c | 2 +- net/ipv6/raw.c | 2 +- net/sunrpc/socklib.c | 2 +- 55 files changed, 609 insertions(+), 1371 deletions(-)
Al Viro
maintainerPATCH 1/18 skb_copy_and_csum_bits(): don't bother with the last argument
From: Al Viro <[email protected]>
it's always 0
--- drivers/net/ethernet/sun/sunvnet_common.c | 2 +- include/linux/skbuff.h | 2 +- net/core/skbuff.c | 11 ++++++----- net/ipv4/icmp.c | 2 +- net/ipv4/ip_output.c | 4 ++-- net/ipv6/icmp.c | 4 ++-- net/ipv6/ip6_output.c | 2 +- net/sunrpc/socklib.c | 2 +- 8 files changed, 15 insertions(+), 14 deletions(-)
drivers/net/ethernet/sun/sunvnet_common.c +1 -1
@@ -1168,7 +1168,7 @@ static inline struct sk_buff *vnet_skb_shape(struct sk_buff *skb, int ncookies) *(__sum16 *)(skb->data + offset) =0 ; csum =skb_copy_and_csum_bits (skb, start, nskb->data + start, - skb->len - start,0 ); + skb->len - start);/* add in the header checksums */ if (skb->protocol ==htons (ETH_P_IP )) {
include/linux/skbuff.h +1 -1
@@ -3529,7 +3529,7 @@ int skb_kill_datagram(struct sock *sk, struct sk_buff *skb, unsigned int flags);int skb_copy_bits (const struct sk_buff *skb,int offset,void *to,int len);int skb_store_bits (struct sk_buff *skb,int offset,const void *from,int len);__wsum skb_copy_and_csum_bits (const struct sk_buff *skb,int offset,u8 *to, -int len,__wsum csum); +int len);int skb_splice_bits (struct sk_buff *skb,struct sock *sk,unsigned int offset,struct pipe_inode_info *pipe,unsigned int len,unsigned int flags);
net/core/skbuff.c +6 -5
@@ -2723,19 +2723,20 @@ EXPORT_SYMBOL(skb_checksum);/* Both of above in one bottle. */ __wsum skb_copy_and_csum_bits (const struct sk_buff *skb,int offset, -u8 *to,int len,__wsum csum) +u8 *to,int len) {int start =skb_headlen (skb);int i, copy = start - offset;struct sk_buff *frag_iter;int pos =0 ; +__wsum csum =0 ;/* Copy header. */ if (copy >0 ) {if (copy > len) copy = len; csum =csum_partial_copy_nocheck (skb->data + offset, to, - copy, csum); + copy,0 );if ((len -= copy) ==0 )return csum; offset += copy;
@@ -2791,7 +2792,7 @@ __wsum skb_copy_and_csum_bits(const struct sk_buff *skb, int offset, copy = len; csum2 =skb_copy_and_csum_bits (frag_iter, offset - start, - to, copy,0 ); + to, copy); csum =csum_block_add (csum, csum2, pos);if ((len -= copy) ==0 )return csum;
@@ -3011,7 +3012,7 @@ void skb_copy_and_csum_dev(const struct sk_buff *skb, u8 *to) csum =0 ;if (csstart != skb->len) csum =skb_copy_and_csum_bits (skb, csstart, to + csstart, - skb->len - csstart,0 ); + skb->len - csstart);if (skb->ip_summed ==CHECKSUM_PARTIAL ) {long csstuff = csstart + skb->csum_offset;
@@ -3933,7 +3934,7 @@ struct sk_buff *skb_segment(struct sk_buff *head_skb,skb_copy_and_csum_bits (head_skb, offset,skb_put (nskb, len), - len,0 ); + len);SKB_GSO_CB (nskb)->csum_start =skb_headroom (nskb) + doffset; }else {
net/ipv4/icmp.c +1 -1
@@ -352,7 +352,7 @@ static int icmp_glue_bits(void *from, char *to, int offset, int len, int odd, csum =skb_copy_and_csum_bits (icmp_param->skb, icmp_param->offset + offset, - to, len,0 ); + to, len); skb->csum =csum_block_add (skb->csum, csum, odd);if (icmp_pointers[icmp_param->data.icmph.type].error)
net/ipv4/ip_output.c +2 -2
@@ -1120,7 +1120,7 @@ static int __ip_append_data(struct sock *sk,if (fraggap) { skb->csum =skb_copy_and_csum_bits ( skb_prev, maxfraglen, - data + transhdrlen, fraggap,0 ); + data + transhdrlen, fraggap); skb_prev->csum =csum_sub (skb_prev->csum, skb->csum); data += fraggap;
@@ -1405,7 +1405,7 @@ ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page, skb->csum =skb_copy_and_csum_bits (skb_prev, maxfraglen,skb_transport_header (skb), - fraggap,0 ); + fraggap); skb_prev->csum =csum_sub (skb_prev->csum, skb->csum);pskb_trim_unique (skb_prev, maxfraglen);
net/ipv6/icmp.c +2 -2
@@ -314,10 +314,10 @@ static int icmpv6_getfrag(void *from, char *to, int offset, int len, int odd, st {struct icmpv6_msg *msg = (struct icmpv6_msg *) from;struct sk_buff *org_skb = msg->skb; -__wsum csum =0 ; +__wsum csum; csum =skb_copy_and_csum_bits (org_skb, msg->offset + offset, - to, len, csum); + to, len); skb->csum =csum_block_add (skb->csum, csum, odd);if (!(msg->type &ICMPV6_INFOMSG_MASK ))nf_ct_attach (skb, org_skb);
net/ipv6/ip6_output.c +1 -1
@@ -1613,7 +1613,7 @@ static int __ip6_append_data(struct sock *sk,if (fraggap) { skb->csum =skb_copy_and_csum_bits ( skb_prev, maxfraglen, - data + transhdrlen, fraggap,0 ); + data + transhdrlen, fraggap); skb_prev->csum =csum_sub (skb_prev->csum, skb->csum); data += fraggap;
net/sunrpc/socklib.c +1 -1
@@ -70,7 +70,7 @@ static size_t xdr_skb_read_and_csum_bits(struct xdr_skb_reader *desc, void *to,if (len > desc->count) len = desc->count; pos = desc->offset; - csum2 =skb_copy_and_csum_bits (desc->skb, pos, to, len,0 ); + csum2 =skb_copy_and_csum_bits (desc->skb, pos, to, len); desc->csum =csum_block_add (desc->csum, csum2, pos); desc->count -= len; desc->offset += len;
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH 2/18 icmp_push_reply(): reorder adding the checksum up
From: Al Viro <[email protected]>
do csum_partial_copy_nocheck() on the first fragment, then add the rest to it. Equivalent transformation.
That was the only caller of csum_partial_copy_nocheck() that might pass it non-zero as the last argument.
--- net/ipv4/icmp.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
net/ipv4/icmp.c +4 -4
@@ -376,15 +376,15 @@ static void icmp_push_reply(struct icmp_bxm *icmp_param,ip_flush_pending_frames (sk); }else if ((skb =skb_peek (&sk->sk_write_queue)) !=NULL ) {struct icmphdr *icmph =icmp_hdr (skb); -__wsum csum =0 ; +__wsum csum;struct sk_buff *skb1; + csum =csum_partial_copy_nocheck ((void *)&icmp_param->data, + (char *)icmph, + icmp_param->head_len,0 ); skb_queue_walk(&sk->sk_write_queue, skb1) { csum =csum_add (csum, skb1->csum); } - csum =csum_partial_copy_nocheck ((void *)&icmp_param->data, - (char *)icmph, - icmp_param->head_len, csum); icmph->checksum =csum_fold (csum); skb->ip_summed =CHECKSUM_NONE ;ip_push_pending_frames (sk, fl4);
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH 3/18 csum_partial_copy_nocheck(): drop the last argument
In reply to Al ViroFrom: Al Viro <[email protected]>
It's always 0. Note that we could use ~0U as well - result will be the same modulo 0xffff; later we'll make use of that whenever convenient.
--- arch/alpha/include/asm/checksum.h | 2 +- arch/alpha/lib/csum_partial_copy.c | 4 ++-- arch/arm/include/asm/checksum.h | 2 +- arch/arm/lib/csumpartialcopy.S | 5 +++-- arch/hexagon/include/asm/checksum.h | 3 +-- arch/hexagon/lib/checksum.c | 4 ++-- arch/ia64/include/asm/checksum.h | 3 +-- arch/ia64/lib/csum_partial_copy.c | 4 ++-- arch/m68k/include/asm/checksum.h | 3 +-- arch/m68k/lib/checksum.c | 3 ++- arch/mips/include/asm/checksum.h | 7 +++++-- arch/mips/lib/csum_partial.S | 4 ++-- arch/nios2/include/asm/checksum.h | 4 ++-- arch/parisc/include/asm/checksum.h | 2 +- arch/parisc/lib/checksum.c | 7 ++----- arch/powerpc/include/asm/checksum.h | 4 ++-- arch/s390/include/asm/checksum.h | 4 ++-- arch/sh/include/asm/checksum_32.h | 5 ++--- arch/sparc/include/asm/checksum_32.h | 4 ++-- arch/sparc/include/asm/checksum_64.h | 8 ++++++-- arch/sparc/lib/csum_copy.S | 2 +- arch/x86/include/asm/checksum_32.h | 5 ++--- arch/x86/include/asm/checksum_64.h | 3 +-- arch/x86/lib/csum-wrappers_64.c | 4 ++-- arch/x86/um/asm/checksum.h | 5 ++--- arch/xtensa/include/asm/checksum.h | 5 ++--- drivers/net/ethernet/3com/typhoon.c | 3 +-- include/asm-generic/checksum.h | 4 ++-- lib/iov_iter.c | 2 +- net/core/skbuff.c | 4 ++-- net/ipv4/icmp.c | 2 +- net/ipv4/ip_output.c | 2 +- net/ipv4/raw.c | 2 +- net/ipv6/raw.c | 2 +- 34 files changed, 62 insertions(+), 65 deletions(-)
arch/alpha/include/asm/checksum.h +1 -1
@@ -44,7 +44,7 @@ extern __wsum csum_partial(const void *buff, int len, __wsum sum);#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER__wsum csum_and_copy_from_user (const void __user *src,void *dst,int len,__wsum sum,int *errp); -__wsum csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum); +__wsum csum_partial_copy_nocheck (const void *src,void *dst,int len);/*
arch/alpha/lib/csum_partial_copy.c +2 -2
@@ -372,13 +372,13 @@ csum_and_copy_from_user(const void __user *src, void *dst, int len,EXPORT_SYMBOL (csum_and_copy_from_user);__wsum -csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum) +csum_partial_copy_nocheck (const void *src,void *dst,int len) {__wsum checksum;mm_segment_t oldfs =get_fs ();set_fs (KERNEL_DS ); checksum =csum_and_copy_from_user ((__force const void __user *)src, - dst, len, sum,NULL ); + dst, len,0 ,NULL );set_fs (oldfs);return checksum; }
arch/arm/include/asm/checksum.h +1 -1
@@ -35,7 +35,7 @@ __wsum csum_partial(const void *buff, int len, __wsum sum);*/ __wsum -csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum); +csum_partial_copy_nocheck (const void *src,void *dst,int len);__wsum csum_partial_copy_from_user (const void __user *src,void *dst,int len,__wsum sum,int *err_ptr);
arch/arm/lib/csumpartialcopy.S +3 -2
@@ -9,13 +9,14 @@ .text -/* Function: __u32 csum_partial_copy_nocheck(const char *src, char *dst, int len, __u32 sum) - * Params : r0 = src, r1 = dst, r2 = len, r3 = checksum +/* Function: __u32 csum_partial_copy_nocheck(const char *src, char *dst, int len) + * Params : r0 = src, r1 = dst, r2 = len * Returns : r0 = new checksum */ .macro save_regs stmfd sp!, {r1, r4 - r8, lr} + mov r3, #0 .endm .macro load_regs
arch/hexagon/include/asm/checksum.h +1 -2
@@ -17,8 +17,7 @@ unsigned int do_csum(const void *voidptr, int len);* better 64-bit) boundary */ #define csum_partial_copy_nocheck csum_partial_copy_nocheck -__wsum csum_partial_copy_nocheck (const void *src,void *dst, -int len,__wsum sum); +__wsum csum_partial_copy_nocheck (const void *src,void *dst,int len);/* * computes the checksum of the TCP/UDP pseudo-header
arch/hexagon/lib/checksum.c +2 -2
@@ -181,9 +181,9 @@ unsigned int do_csum(const void *voidptr, int len)* copy from ds while checksumming, otherwise like csum_partial */ __wsum -csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum) +csum_partial_copy_nocheck (const void *src,void *dst,int len) {memcpy (dst, src, len); -return csum_partial (dst, len, sum); +return csum_partial (dst, len,0 ); }EXPORT_SYMBOL (csum_partial_copy_nocheck);
arch/ia64/include/asm/checksum.h +1 -2
@@ -37,8 +37,7 @@ extern __wsum csum_tcpudp_nofold(__be32 saddr, __be32 daddr,*/ extern __wsum csum_partial (const void *buff,int len,__wsum sum); -extern __wsum csum_partial_copy_nocheck (const void *src,void *dst, -int len,__wsum sum); +extern __wsum csum_partial_copy_nocheck (const void *src,void *dst,int len);/* * This routine is used for miscellaneous IP-like checksums, mainly in
arch/ia64/lib/csum_partial_copy.c +2 -2
@@ -104,10 +104,10 @@ unsigned long do_csum_c(const unsigned char * buff, int len, unsigned int psum)* But it's very tricky to get right even in C. */ __wsum -csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum) +csum_partial_copy_nocheck (const void *src,void *dst,int len) {memcpy (dst, src, len); -return csum_partial (dst, len, sum); +return csum_partial (dst, len,0 ); }EXPORT_SYMBOL (csum_partial_copy_nocheck);
arch/m68k/include/asm/checksum.h +1 -2
@@ -37,8 +37,7 @@ extern __wsum csum_and_copy_from_user(const void __user *src, int *csum_err);extern __wsum csum_partial_copy_nocheck (const void *src, -void *dst,int len, -__wsum sum); +void *dst,int len);/* * This is a version of ip_fast_csum() optimized for IP headers,
arch/m68k/lib/checksum.c +2 -1
@@ -324,9 +324,10 @@ EXPORT_SYMBOL(csum_and_copy_from_user);*/ __wsum -csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum) +csum_partial_copy_nocheck (const void *src,void *dst,int len) {unsigned long tmp1, tmp2; +__wsum sum =0 ; __asm__("movel %2,%4\n\t" "btst #1,%4\n\t" /* Check alignment */ "jeq 2f\n\t"
arch/mips/include/asm/checksum.h +5 -2
@@ -101,8 +101,11 @@ __wsum csum_and_copy_to_user(const void *src, void __user *dst, int len,* the same as csum_partial, but copies from user space (but on MIPS * we have just one address space, so this is identical to the above) */ -__wsum csum_partial_copy_nocheck (const void *src,void *dst, -int len,__wsum sum); +__wsum __csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum); +static inline __wsum csum_partial_copy_nocheck (const void *src,void *dst,int len) +{ +return __csum_partial_copy_nocheck (src, dst, len,0 ); +}#define csum_partial_copy_nocheck csum_partial_copy_nocheck/*
arch/mips/lib/csum_partial.S +2 -2
@@ -462,8 +462,8 @@ EXPORT_SYMBOL(csum_partial) lw errptr, 16(sp) #endif .if \__nocheck == 1 - FEXPORT(csum_partial_copy_nocheck) - EXPORT_SYMBOL(csum_partial_copy_nocheck) + FEXPORT(__csum_partial_copy_nocheck) + EXPORT_SYMBOL(__csum_partial_copy_nocheck) .endif move sum, zero move odd, zero
arch/nios2/include/asm/checksum.h +2 -2
@@ -14,8 +14,8 @@extern __wsum csum_partial (const void *buff,int len,__wsum sum);extern __wsum csum_partial_copy (const void *src,void *dst,int len,__wsum sum); -#define csum_partial_copy_nocheck (src, dst, len, sum) \ - csum_partial_copy((src), (dst), (len), (sum)) +#define csum_partial_copy_nocheck (src, dst, len) \ + csum_partial_copy((src), (dst), (len), 0)extern __sum16 ip_fast_csum (const void *iph,unsigned int ihl);extern __sum16 ip_compute_csum (const void *buff,int len);
arch/parisc/include/asm/checksum.h +1 -1
@@ -24,7 +24,7 @@ extern __wsum csum_partial(const void *, int, __wsum);* Here even more important to align src and dst on a 32-bit (or even * better 64-bit) boundary */ -extern __wsum csum_partial_copy_nocheck (const void *,void *,int ,__wsum ); +extern __wsum csum_partial_copy_nocheck (const void *,void *,int );/* * Optimized for IP headers, which always checksum on 4 octet boundaries.
arch/parisc/lib/checksum.c +2 -5
@@ -110,16 +110,13 @@ EXPORT_SYMBOL(csum_partial);/* * copy while checksumming, otherwise like csum_partial */ -__wsum csum_partial_copy_nocheck (const void *src,void *dst, -int len,__wsum sum) +__wsum csum_partial_copy_nocheck (const void *src,void *dst,int len) {/* * It's 2:30 am and I don't feel like doing it real ... * This is lots slower than the real thing (tm) */ - sum =csum_partial (src, len, sum);memcpy (dst, src, len); - -return sum; +return csum_partial (src, len,0 ); }EXPORT_SYMBOL (csum_partial_copy_nocheck);
arch/powerpc/include/asm/checksum.h +2 -2
@@ -29,8 +29,8 @@ extern __wsum csum_and_copy_from_user(const void __user *src, void *dst,extern __wsum csum_and_copy_to_user (const void *src,void __user *dst,int len,__wsum sum,int *err_ptr); -#define csum_partial_copy_nocheck (src, dst, len, sum) \ - csum_partial_copy_generic((src), (dst), (len), (sum), NULL, NULL) +#define csum_partial_copy_nocheck (src, dst, len) \ + csum_partial_copy_generic((src), (dst), (len), 0, NULL, NULL)/*
arch/s390/include/asm/checksum.h +2 -2
@@ -40,10 +40,10 @@ csum_partial(const void *buff, int len, __wsum sum) }static inline __wsum -csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum) +csum_partial_copy_nocheck (const void *src,void *dst,int len) {memcpy (dst,src,len); -return csum_partial (dst, len, sum); +return csum_partial (dst, len,0 ); }/*
arch/sh/include/asm/checksum_32.h +2 -3
@@ -42,10 +42,9 @@ asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst,* access_ok(). */ static inline -__wsum csum_partial_copy_nocheck (const void *src,void *dst, -int len,__wsum sum) +__wsum csum_partial_copy_nocheck (const void *src,void *dst,int len) { -return csum_partial_copy_generic (src, dst, len, sum,NULL ,NULL ); +return csum_partial_copy_generic (src, dst, len,0 ,NULL ,NULL ); }#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER
arch/sparc/include/asm/checksum_32.h +2 -2
@@ -42,7 +42,7 @@ __wsum csum_partial(const void *buff, int len, __wsum sum);unsigned int __csum_partial_copy_sparc_generic (const unsigned char *,unsigned char *);static inline __wsum -csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum) +csum_partial_copy_nocheck (const void *src,void *dst,int len) { register unsigned int ret asm("o0") = (unsigned int)src; register char *d asm("o1") = dst;
@@ -52,7 +52,7 @@ csum_partial_copy_nocheck(const void *src, void *dst, int len, __wsum sum)"call __csum_partial_copy_sparc_generic\n\t" " mov %6, %%g7\n" : "=&r" (ret), "=&r" (d), "=&r" (l) - : "0" (ret), "1" (d), "2" (l), "r" (sum) + : "0" (ret), "1" (d), "2" (l), "r" (0) : "o2", "o3", "o4", "o5", "o7"," g2", " g3", " g4", " g5", " g7", "memory", "cc");
arch/sparc/include/asm/checksum_64.h +6 -2
@@ -38,8 +38,12 @@ __wsum csum_partial(const void * buff, int len, __wsum sum);* here even more important to align src and dst on a 32-bit (or even * better 64-bit) boundary */ -__wsum csum_partial_copy_nocheck (const void *src,void *dst, -int len,__wsum sum); +__wsum __csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum); + +static inline __wsum csum_partial_copy_nocheck (const void *src,void *dst,int len) +{ +return __csum_partial_copy_nocheck (src, dst, len,0 ); +} long __csum_partial_copy_from_user(const void __user *src, void *dst, int len,
arch/sparc/lib/csum_copy.S +1 -1
@@ -33,7 +33,7 @@ #endif #ifndef FUNC_NAME -#define FUNC_NAME csum_partial_copy_nocheck +#define FUNC_NAME __csum_partial_copy_nocheck #endif .register %g2, #scratch
arch/x86/include/asm/checksum_32.h +2 -3
@@ -38,10 +38,9 @@ asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst,* If you use these functions directly please don't forget the * access_ok(). */ -static inline __wsum csum_partial_copy_nocheck (const void *src,void *dst, -int len,__wsum sum) +static inline __wsum csum_partial_copy_nocheck (const void *src,void *dst,int len) { -return csum_partial_copy_generic (src, dst, len, sum,NULL ,NULL ); +return csum_partial_copy_generic (src, dst, len,0 ,NULL ,NULL ); } static inline __wsum csum_and_copy_from_user(const void __user *src,
arch/x86/include/asm/checksum_64.h +1 -2
@@ -139,8 +139,7 @@ extern __wsum csum_and_copy_from_user(const void __user *src, void *dst, int len, __wsum isum, int *errp);extern __wsum csum_and_copy_to_user (const void *src,void __user *dst,int len,__wsum isum,int *errp); -extern __wsum csum_partial_copy_nocheck (const void *src,void *dst, -int len,__wsum sum); +extern __wsum csum_partial_copy_nocheck (const void *src,void *dst,int len);/** * ip_compute_csum - Compute an 16bit IP checksum.
arch/x86/lib/csum-wrappers_64.c +2 -2
@@ -129,9 +129,9 @@ EXPORT_SYMBOL(csum_and_copy_to_user);* Returns an 32bit unfolded checksum of the buffer. */ __wsum -csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum) +csum_partial_copy_nocheck (const void *src,void *dst,int len) { -return csum_partial_copy_generic (src, dst, len, sum,NULL ,NULL ); +return csum_partial_copy_generic (src, dst, len,0 ,NULL ,NULL ); }EXPORT_SYMBOL (csum_partial_copy_nocheck);
arch/x86/um/asm/checksum.h +2 -3
@@ -29,11 +29,10 @@ extern __wsum csum_partial(const void *buff, int len, __wsum sum);*/ static __inline__ -__wsum csum_partial_copy_nocheck (const void *src,void *dst, -int len,__wsum sum) +__wsum csum_partial_copy_nocheck (const void *src,void *dst,int len) {memcpy (dst, src, len); -return csum_partial (dst, len, sum); +return csum_partial (dst, len,0 ); }/**
arch/xtensa/include/asm/checksum.h +2 -3
@@ -46,10 +46,9 @@ asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst,* passed in an incorrect kernel address to one of these functions. */ static inline -__wsum csum_partial_copy_nocheck (const void *src,void *dst, -int len,__wsum sum) +__wsum csum_partial_copy_nocheck (const void *src,void *dst,int len) { -return csum_partial_copy_generic (src, dst, len, sum,NULL ,NULL ); +return csum_partial_copy_generic (src, dst, len,0 ,NULL ,NULL ); }#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER
drivers/net/ethernet/3com/typhoon.c +1 -2
@@ -1419,8 +1419,7 @@ typhoon_download_firmware(struct typhoon *tp)* the checksum, we can do this once, at the end. */ csum =csum_fold (csum_partial_copy_nocheck (image_data, - dpage, len, -0 )); + dpage, len));iowrite32 (len, ioaddr +TYPHOON_REG_BOOT_LENGTH ); iowrite32(le16_to_cpu((__force __le16)csum),
include/asm-generic/checksum.h +2 -2
@@ -26,8 +26,8 @@ extern __wsum csum_partial(const void *buff, int len, __wsum sum);extern __wsum csum_partial_copy (const void *src,void *dst,int len,__wsum sum);#ifndef csum_partial_copy_nocheck -#define csum_partial_copy_nocheck (src, dst, len, sum) \ - csum_partial_copy((src), (dst), (len), (sum)) +#define csum_partial_copy_nocheck (src, dst, len) \ + csum_partial_copy((src), (dst), (len), 0)#endif #ifndef ip_fast_csum
lib/iov_iter.c +1 -1
@@ -580,7 +580,7 @@ static size_t copy_pipe_to_iter(const void *addr, size_t bytes,static __wsum csum_and_memcpy (void *to,const void *from,size_t len,__wsum sum,size_t off) { -__wsum next =csum_partial_copy_nocheck (from, to, len,0 ); +__wsum next =csum_partial_copy_nocheck (from, to, len);return csum_block_add (sum, next, off); }
net/core/skbuff.c +2 -2
@@ -2736,7 +2736,7 @@ __wsum skb_copy_and_csum_bits(const struct sk_buff *skb, int offset,if (copy > len) copy = len; csum =csum_partial_copy_nocheck (skb->data + offset, to, - copy,0 ); + copy);if ((len -= copy) ==0 )return csum; offset += copy;
@@ -2766,7 +2766,7 @@ __wsum skb_copy_and_csum_bits(const struct sk_buff *skb, int offset, vaddr =kmap_atomic (p); csum2 =csum_partial_copy_nocheck (vaddr + p_off, to + copied, - p_len,0 ); + p_len);kunmap_atomic (vaddr); csum =csum_block_add (csum, csum2, pos); pos += p_len;
net/ipv4/icmp.c +1 -1
@@ -381,7 +381,7 @@ static void icmp_push_reply(struct icmp_bxm *icmp_param, csum =csum_partial_copy_nocheck ((void *)&icmp_param->data, (char *)icmph, - icmp_param->head_len,0 ); + icmp_param->head_len); skb_queue_walk(&sk->sk_write_queue, skb1) { csum =csum_add (csum, skb1->csum); }
net/ipv4/ip_output.c +1 -1
@@ -1642,7 +1642,7 @@ static int ip_reply_glue_bits(void *dptr, char *to, int offset, {__wsum csum; - csum =csum_partial_copy_nocheck (dptr+offset, to, len,0 ); + csum =csum_partial_copy_nocheck (dptr+offset, to, len); skb->csum =csum_block_add (skb->csum, csum, odd);return 0 ; }
net/ipv4/raw.c +1 -1
@@ -478,7 +478,7 @@ static int raw_getfrag(void *from, char *to, int offset, int len, int odd, skb->csum =csum_block_add ( skb->csum,csum_partial_copy_nocheck (rfv->hdr.c + offset, - to, copy,0 ), + to, copy), odd); odd =0 ;
net/ipv6/raw.c +1 -1
@@ -746,7 +746,7 @@ static int raw6_getfrag(void *from, char *to, int offset, int len, int odd, skb->csum =csum_block_add ( skb->csum,csum_partial_copy_nocheck (rfv->c + offset, - to, copy,0 ), + to, copy), odd); odd =0 ;
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH 4/18 csum_and_copy_..._user(): pass 0xffffffff instead of 0 as initial sum
In reply to Al ViroFrom: Al Viro <[email protected]>
Preparation for the change of calling conventions; right now all callers pass 0 as initial sum. Passing 0xffffffff instead yields the values comparable mod 0xffff and guarantees that 0 will not be returned on success.
--- lib/iov_iter.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
lib/iov_iter.c +3 -3
@@ -1451,7 +1451,7 @@ size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum,int err =0 ; next =csum_and_copy_from_user (v.iov_base, (to += v.iov_len) - v.iov_len, - v.iov_len,0 , &err); + v.iov_len, ~0U , &err);if (!err) { sum =csum_block_add (sum, next, off); off += v.iov_len;
@@ -1493,7 +1493,7 @@ bool csum_and_copy_from_iter_full(void *addr, size_t bytes, __wsum *csum,int err =0 ; next =csum_and_copy_from_user (v.iov_base, (to += v.iov_len) - v.iov_len, - v.iov_len,0 , &err); + v.iov_len, ~0U , &err);if (err)return false; sum =csum_block_add (sum, next, off);
@@ -1539,7 +1539,7 @@ size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *csump,int err =0 ; next =csum_and_copy_to_user ((from += v.iov_len) - v.iov_len, v.iov_base, - v.iov_len,0 , &err); + v.iov_len, ~0U , &err);if (!err) { sum =csum_block_add (sum, next, off); off += v.iov_len;
Signature · 1 line
2.11.0
Re: [PATCH 04/18] csum_and_copy_..._user(): pass 0xffffffff instead of 0 as initial sum
Quoted 5 lines · Al Viro
Preparation for the change of calling conventions; right now all callers pass 0 as initial sum. Passing
0xffffffffinstead yields the values comparable mod0xffffand guarantees that 0 will not be returned on success.
This seems dangerous to me.
Maybe some implementation depends on the fact that they actually do the csum 16 bits at a time, and never see an overflow in "int", because they keep folding things.
You now break that assumption, and give it an initial value that the csum code itself would never generate, and wouldn't handle right.
But I didn't check. Maybe we don't have anything that stupid in the kernel.
Linus
Re: [PATCH 04/18] csum_and_copy_..._user(): pass 0xffffffff instead of 0 as initial sum
Quoted 11 lines · Linus Torvalds
This seems dangerous to me.
Maybe some implementation depends on the fact that they actually do the csum 16 bits at a time, and never see an overflow in "int", because they keep folding things.
You now break that assumption, and give it an initial value that the csum code itself would never generate, and wouldn't handle right.
But I didn't check. Maybe we don't have anything that stupid in the kernel.
I take it back. The very first place I looked seemed to do exactly that.
See "do_csum()" in the kernel. It doesn't handle carry for any of the usual cases, exactly because it knows it doesn't need to.
Ok, so do_csum() doesn't take that initial value, but it's very much an example of the kind of algorithm I was thinking of: it does do things 32 bits at a time and handles the carry bit in that inner loop, but internally it knows that the val;ues are limited in other places, and doesn't need to handle carry everywhere.
Linus
Al Viro
maintainerRe: [PATCH 04/18] csum_and_copy_..._user(): pass 0xffffffff instead of 0 as initial sum
Quoted 5 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds
This seems dangerous to me.
Quoted 8 lines
Quoted 8 lines
Maybe some implementation depends on the fact that they actually do the csum 16 bits at a time, and never see an overflow in "int", because they keep folding things.
You now break that assumption, and give it an initial value that the csum code itself would never generate, and wouldn't handle right.
But I didn't check. Maybe we don't have anything that stupid in the kernel.
I did.
Quoted 10 lines
I take it back. The very first place I looked seemed to do exactly that.
See "
do_csum()" in the kernel. It doesn't handle carry for any of the usual cases, exactly because it knows it doesn't need to.Ok, so
do_csum()doesn't take that initial value, but it's very much an example of the kind of algorithm I was thinking of: it does do things 32 bits at a time and handles the carry bit in that inner loop, but internally it knows that the val;ues are limited in other places, and doesn't need to handle carry everywhere.
Theoretically - sure. I can post the full analysis of that stuff (starting with the proof that all instances of csum_partial() are OK in that respect, which takes care of the default instances, then instance-by-instance analysis of the rest); will need to collate the pieces, remove the actionable obscenities, etc., but I have done that analysis. Made for rather unpleasant couple of weeks... ;-/
Re: [PATCH 04/18] csum_and_copy_..._user(): pass 0xffffffff instead of 0 as initial sum
Quoted 4 lines · Al Viro
Quoted 2 lines
Quoted 2 lines
But I didn't check. Maybe we don't have anything that stupid in the kernel.
I did.
So then the commit message really should have said so, I feel. That would have avoided the whole worry, and made it clear that it's not an issue.
Linus
Al Viro
maintainerRe: [PATCH 04/18] csum_and_copy_..._user(): pass 0xffffffff instead of 0 as initial sum
In reply to Al ViroQuoted 6 lines · Al Viro
Theoretically - sure. I can post the full analysis of that stuff (starting with the proof that all instances of
csum_partial()are OK in that respect, which takes care of the default instances, then instance-by-instance analysis of the rest); will need to collate the pieces, remove the actionable obscenities, etc., but I have done that analysis. Made for rather unpleasant couple of weeks... ;-/
BTW, "full" in the above refers only to the cases were we do the switchover from initial sum 0 to initial sum ~0U (csum_and_copy_..._user() and those csum_partial_copy_nocheck() instances that share helpers with them).
However, now that I started to write that down I went to look at other csum_partial_copy_nocheck() instances. Turns out that c6x one is, strictly speaking, broken - it works for all initial sum values the kernel ever gives it, but you can come up with a combination of sum and data that will return the wrong value.
The only reason we never hit it is that there's only one caller (gone in this series) that ever passes non-zero, and that one caller (icmp_push_reply()) can't get to the vulnerable values of initial sums since on c6x the values of csum_partial_copy_nocheck() are initial_sum + 16bit unsigned, and initial sum in question is csum_add() of a csums of fragments, possibly rotated 8 bits right. And to get overflow in that sucker you'd need 0xffff as the top halfword, so you'd need at least 257 fragments, including at least one at odd offset. Which, AFAICS, can't happen - all framing headers have even lengths.
RE: [PATCH 04/18] csum_and_copy_..._user(): pass 0xffffffff instead of 0 as initial sum
In reply to Linus TorvaldsFrom: Linus Torvalds
Quoted 10 lines
Sent: 21 July 2020 21:55
Quoted 5 lines · Al Viro
Preparation for the change of calling conventions; right now all callers pass 0 as initial sum. Passing
0xffffffffinstead yields the values comparable mod0xffffand guarantees that 0 will not be returned on success.This seems dangerous to me.
Quoted 8 lines
Maybe some implementation depends on the fact that they actually do the csum 16 bits at a time, and never see an overflow in "int", because they keep folding things.
You now break that assumption, and give it an initial value that the csum code itself would never generate, and wouldn't handle right.
But I didn't check. Maybe we don't have anything that stupid in the kernel.
It isn't necessarily stupid :-)
A 64bit sum can be reduced to 16bits using shifts and adds (as us usually done) of using 'sum % 0xffff'.
Provided the compiler uses 'multiply by reciprocal' the code isn't that bad - it might even be difficult to say which is faster. However that makes the output domain 0..fffe not 1..ffff.
The checksum generation code really needs to know which is used. So it is best never to use the % version.
If the sum is known to be 1..0xffff then after inversion it is 0..fffe but the required domain is 1..ffff.
This can be fixed by adding 1 - provided a compensating 1 is added in before the inversion.
The easy place to do this is to feed 1 (not 0 or ~0) into the first checksum block.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
RE: [PATCH 04/18] csum_and_copy_..._user(): pass 0xffffffff instead of 0 as initial sum
In reply to Al ViroFrom: Al Viro
Quoted 16 lines
Sent: 21 July 2020 21:26
Preparation for the change of calling conventions; right now all callers pass 0 as initial sum. Passing0xffffffffinstead yields the values comparable mod0xffffand guarantees that 0 will not be returned on success.Signed-off-by Al Viro <[email protected]>--- lib/iov_iter.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)lib/iov_iter.c +0
-0
lib/iov_iter.c size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, · line 1457
Quoted 8 lines
lib/iov_iter.c +1
-1@@ -1451,7 +1451,7 @@ size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum,int err =0 ; next =csum_and_copy_from_user (v.iov_base, (to += v.iov_len) - v.iov_len, - v.iov_len,0 , &err); + v.iov_len, ~0U , &err);if (!err) { sum =csum_block_add (sum, next, off); off += v.iov_len;
Can't you remove the csum_block_add() by passing the old 'sum' in instead of the ~0U ?
You'll need to keep track of whether the buffer fragment is odd/even aligned.
After an odd length fragment a bswap32() or 8 bit rotate will fix things (and maybe one right at the end).
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Al Viro
maintainerRe: [PATCH 04/18] csum_and_copy_..._user(): pass 0xffffffff instead of 0 as initial sum
Quoted 24 lines · David Laight
From: Al Viro
Quoted 23 lines
Sent: 21 July 2020 21:26
Preparation for the change of calling conventions; right now all callers pass 0 as initial sum. Passing0xffffffffinstead yields the values comparable mod0xffffand guarantees that 0 will not be returned on success.Signed-off-by Al Viro <[email protected]>--- lib/iov_iter.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)lib/iov_iter.c +1
-1@@ -1451,7 +1451,7 @@ size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum,int err =0 ; next =csum_and_copy_from_user (v.iov_base, (to += v.iov_len) - v.iov_len, - v.iov_len,0 , &err); + v.iov_len, ~0U , &err);if (!err) { sum =csum_block_add (sum, next, off);
Quoted 8 lines
lib/iov_iter.c size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, · line 1457
Quoted 1 line
lib/iov_iter.c +0
-0off += v.iov_len;Can't you remove the
csum_block_add()by passing the old 'sum' in instead of the ~0U ?
You'll need to keep track of whether the buffer fragment is odd/even aligned.
After an odd length fragment abswap32()or 8 bit rotate will fix things (and maybe one right at the end).
And the benefit of that would be...? It wouldn't be any simpler, it almost certainly would not even be a valid microoptimization (nevermind that this is an arch-independent code)...
RE: [PATCH 04/18] csum_and_copy_..._user(): pass 0xffffffff instead of 0 as initial sum
From: Al Viro
Quoted 36 lines
Sent: 22 July 2020 15:42
Quoted 24 lines · David Laight
From: Al Viro
Quoted 23 lines
Sent: 21 July 2020 21:26
Preparation for the change of calling conventions; right now all callers pass 0 as initial sum. Passing0xffffffffinstead yields the values comparable mod0xffffand guarantees that 0 will not be returned on success.Signed-off-by Al Viro <[email protected]>--- lib/iov_iter.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)lib/iov_iter.c +1
-1@@ -1451,7 +1451,7 @@ size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum,int err =0 ; next =csum_and_copy_from_user (v.iov_base, (to += v.iov_len) - v.iov_len, - v.iov_len,0 , &err); + v.iov_len, ~0U , &err);if (!err) { sum =csum_block_add (sum, next, off);Quoted 8 lines
lib/iov_iter.c size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, · line 1457
Quoted 1 line
lib/iov_iter.c +0
-0off += v.iov_len;Can't you remove the
csum_block_add()by passing the old 'sum' in instead of the ~0U ?
You'll need to keep track of whether the buffer fragment is odd/even aligned.
After an odd length fragment abswap32()or 8 bit rotate will fix things (and maybe one right at the end).
Quoted 3 lines
And the benefit of that would be...? It wouldn't be any simpler, it almost certainly would not even be a valid microoptimization (nevermind that this is an arch-independent code)...
It ought to give a minor improvement because it saves the extra csum_fold() when the checksum from a buffer is added to the previous total.
On 64bit systems there are even advantages in passing in a 64bit value - so the caller can add many 32bit values together. If nothing else it lets you use a '<< 8' if the previous fragment had an odd length.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Al Viro
maintainerRe: [PATCH 04/18] csum_and_copy_..._user(): pass 0xffffffff instead of 0 as initial sum
Quoted 8 lines · David Laight
Quoted 3 lines
And the benefit of that would be...? It wouldn't be any simpler, it almost certainly would not even be a valid microoptimization (nevermind that this is an arch-independent code)...
It ought to give a minor improvement because it saves the extra
csum_fold()when the checksum from a buffer is added to the previous total.
Sigh... _WHAT_ csum_fold()?
static inline __wsumcsum_block_add(__wsum csum, __wsum csum2, int offset)
{
u32 sum = (__force u32)csum2;
/* rotate sum to align it with a 16b boundary */
if (offset & 1)
sum = ror32(sum, 8);
return csum_add(csum, (__force __wsum)sum);
}David, do you *ever* bother to RTFS? I mean, competent supercilious twits are annoying, but at least with those you can generally assume that what they say makes sense and has some relation to reality. You, OTOH, keep spewing utter bollocks, without ever lowering yourself to checking if your guesses have anything to do with the reality. With supercilious twit part proudly on the display - you do speak with confidence, and the way you dispense the oh-so-valuable advice to everyone around...
RE: [PATCH 04/18] csum_and_copy_..._user(): pass 0xffffffff instead of 0 as initial sum
From: Al Viro > Sent: 22 July 2020 16:55
Quoted 30 lines
To: David Laight <[email protected]>
Cc: Linus Torvalds <[email protected]>; [email protected]; linux- [email protected]
Subject: Re: [PATCH 04/18] csum_and_copy_..._user(): pass0xffffffffinstead of 0 as initial sumQuoted 8 lines · David Laight
Quoted 3 lines
And the benefit of that would be...? It wouldn't be any simpler, it almost certainly would not even be a valid microoptimization (nevermind that this is an arch-independent code)...
It ought to give a minor improvement because it saves the extra
csum_fold()when the checksum from a buffer is added to the previous total.Sigh... _WHAT_
csum_fold()?static inline
__wsumcsum_block_add(__wsum csum, __wsum csum2, int offset){ u32 sum = (__force u32)csum2; /* rotate sum to align it with a 16b boundary */ if (offset & 1) sum = ror32(sum, 8); return csum_add(csum, (__force __wsum)sum); }
Quoted 7 lines
David, do you *ever* bother to RTFS? I mean, competent supercilious twits are annoying, but at least with those you can generally assume that what they say makes sense and has some relation to reality. You, OTOH, keep spewing utter bollocks, without ever lowering yourself to checking if your guesses have anything to do with the reality. With supercilious twit part proudly on the display - you do speak with confidence, and the way you dispense the oh-so-valuable advice to everyone around...
Yes, I do look at the code.
I've actually spent a lot of time looking at the x86 checksum code. I've posted a patch for a version that is about twice as fast as the current one on a large range of x86 cpus.
Possibly I meant the 32bit reduction inside csum_add() rather than what csum_fold() does.
Having worked on the internals of SYSV, NetBSD and Linux I probably forget the exact names for a few things.
The brain can only hold so much information.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Al Viro
maintainerRe: [PATCH 04/18] csum_and_copy_..._user(): pass 0xffffffff instead of 0 as initial sum
Quoted 8 lines · David Laight
Quoted 7 lines
David, do you *ever* bother to RTFS? I mean, competent supercilious twits are annoying, but at least with those you can generally assume that what they say makes sense and has some relation to reality. You, OTOH, keep spewing utter bollocks, without ever lowering yourself to checking if your guesses have anything to do with the reality. With supercilious twit part proudly on the display - you do speak with confidence, and the way you dispense the oh-so-valuable advice to everyone around...
Quoted 7 lines
Yes, I do look at the code.
I've actually spent a lot of time looking at the x86 checksum code. I've posted a patch for a version that is about twice as fast as the current one on a large range of x86 cpus.Possibly I meant the 32bit reduction inside
csum_add()rather than whatcsum_fold()does.
Really?
static inline unsigned add32_with_carry(unsigned a, unsigned b)
{
asm("addl %2,%0\n\t"
"adcl $0,%0"
: "=r" (a)
: "0" (a), "rm" (b));
return a;
}
static inline __wsum csum_add(__wsum csum, __wsum addend)
{
return (__force __wsum)add32_with_carry((__force unsigned)csum,
(__force unsigned)addend);
}I would love to see your patch, anyway, along with the testcases and performance comparison.
Quoted 2 lines
Having worked on the internals of SYSV, NetBSD and Linux I probably forget the exact names for a few things.
That's usually dealt with by a few minutes with grep and vi...
Quoted 1 line
The brain can only hold so much information.
Bravo. "I can't be arsed to check anything" spun into the claim of one's superior experience.
What it means in practice is that your output is so much garbage that _might_ be untangled into something meaningful if the reader manages to guess the substitutions. Provided that the reconstruction won't not turn out to be a composite of things applying to different versions of different kernels, not being valid for any of those, that is...
RE: [PATCH 04/18] csum_and_copy_..._user(): pass 0xffffffff instead of 0 as initial sum
From: Al Viro
Quoted 20 lines
Sent: 22 July 2020 18:39
Quoted 8 lines · David Laight
Quoted 7 lines
David, do you *ever* bother to RTFS? I mean, competent supercilious twits are annoying, but at least with those you can generally assume that what they say makes sense and has some relation to reality. You, OTOH, keep spewing utter bollocks, without ever lowering yourself to checking if your guesses have anything to do with the reality. With supercilious twit part proudly on the display - you do speak with confidence, and the way you dispense the oh-so-valuable advice to everyone around...
Quoted 7 lines
Yes, I do look at the code.
I've actually spent a lot of time looking at the x86 checksum code. I've posted a patch for a version that is about twice as fast as the current one on a large range of x86 cpus.Possibly I meant the 32bit reduction inside
csum_add()rather than whatcsum_fold()does.Really?
static inline unsignedadd32_with_carry(unsigned a, unsigned b)
Quoted 7 lines
{
asm("addl %2,%0\n\t"
"adcl $0,%0"
: "=r" (a)
: "0" (a), "rm" (b));
return a;
}I agree it isn't much, but both those instructions almost certainly get replicated with the initial value fed into the checksum function.
Everything except x86, sparc/64 and powerpc/64 uses the C code from include/net/checksum.h which is the longer sequences:
csum += addend; csum += csum < addend;
That's three instructions on something like MIPS - not too bad. I'm not sure about ARM - ARM could probably use adc. Some architectures may end up with an actual conditional jump.
Quite how the instructions get scheduled probably makes more difference.
The sequence is a register dependency chain, and the checksum register could easily be limiting the execution speed. On x86 the 'adc' loop runs at two clocks per adc on a wide range of Intel cpus.
Actually there is lot more to be gained in the code that reads the iovec[] from userspace.
The calling sequences for the two nexted functions used are horrid. Fixing that does make a measurable difference to semdmsg().
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
RE: [PATCH 04/18] csum_and_copy_..._user(): pass 0xffffffff instead of 0 as initial sum
In reply to Al ViroFrom: Al Viro
Quoted 3 lines
Sent: 22 July 2020 18:39
I would love to see your patch, anyway, along with the testcases and performance comparison.
See attached program.
Compile and run (as root): csum_iov 1
Unpatched (as shipped) 16 vectors of 1 byte take ~430 clocks on my haswell cpu. With dsl_patch defined they take ~393.
The maximum throughput is ~1.16 clocks/word for 16 vectors of 1k. For longer vectors the data gets lost from the cache between the iterations.
On an older Ivy Bridge cpu it never goes faster than 2 clocks/word. (Due to the implementation of ADC.)
The absolute limit is 1 clock/word - limited by the memory write. I suspect that is achievable on Haswell with much less loop unrolling.
I had to replace the ror32() with __builtin_bswap32(). The kernel object do contain the 'ror' instruction - even though I didn't find the asm for it.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
/* Test program for checksum+copy
*
* Executes csum_and_copy_from_iter() in userspace.
* Uses PERF_COUNT_HW_CPU_CYCLES to see how fast it runs.
* Always copies i6 copies of the same buffer to the target.
* Length of each fragment taken from argv[0].
*
* It needs linking with a copy of csum-copy_64.o (eg from a kernel build). *
* For large buffers the 'adc' loop dominates.
* On anything prior to Haswell this is 2 clocks per adc.
* On Haswell adc is faster and it seems to approach 1.16 clocks/word.
* It ought to be possibly to get to 1 clock/word on Ivy bridge (Sandy?)
* or later.
*/
// define for my version
// #define dsl_patch
#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
#include <errno.h>
#include <unistd.h>
#include <linux/perf_event.h>
#include <sys/mman.h>
#include <sys/syscall.h>
#define likely(x) (x)
#define unlikely(x) (x)
typedef uint32_t __wsum;
struct kvec {
size_t iov_len;
void *iov_base;
};struct iov_iter {
unsigned int count;
unsigned int nr_segs;
const struct kvec *kvec;
size_t iov_offset;
};#define min(a,b) ((a) < (b) ? (a) : (b))
static unsigned short fold(unsigned int csum)
{
csum = (csum & 0xffff) + (csum >> 16);
return csum + (csum >> 16);
}extern __wsum csum_partial_copy_generic(const void *, void *, size_t, __wsum, void *, void *);
__wsumcsum_partial_copy_nocheck(const void *src, void *dst, int len, __wsum sum)
{
return csum_partial_copy_generic(src, dst, len, sum, NULL, NULL);
}static inline unsigned add32_with_carry(unsigned a, unsigned b)
{
asm("addl %2,%0\n\t"
"adcl $0,%0"
: "=r" (a)
: "0" (a), "rm" (b));
return a;
}static inline __wsum csum_add(__wsum csum, __wsum addend)
{
return add32_with_carry(csum, addend);
}static inline __wsumcsum_block_add(__wsum csum, __wsum sum, int offset)
{
/* rotate sum to align it with a 16b boundary */
if (offset & 1)
sum = __builtin_bswap32(sum);
return csum_add(csum, sum);
}//////////////////////////////////////////////////////////////////////
/* Necessary bits from iov_iter.c */
#define iterate_kvec(i, n, __v, __p, skip, STEP) { \
size_t wanted = n; \
__p = i->kvec; \
__v.iov_len = min(n, __p->iov_len - skip); \
if (likely(__v.iov_len)) { \
__v.iov_base = __p->iov_base + skip; \
(void)(STEP); \
skip += __v.iov_len; \
n -= __v.iov_len; \
} \
while (unlikely(n)) { \
__p++; \
__v.iov_len = min(n, __p->iov_len); \
if (unlikely(!__v.iov_len)) \
continue; \
__v.iov_base = __p->iov_base; \
(void)(STEP); \
skip = __v.iov_len; \
n -= __v.iov_len; \
} \
n = wanted; \
}#define iterate_and_advance(i, n, v, I, B, K) { \
if (unlikely(i->count < n)) \
n = i->count; \
if (i->count) { \
size_t skip = i->iov_offset; \
const struct kvec *kvec; \
struct kvec v; \
iterate_kvec(i, n, v, kvec, skip, (K)) \
if (skip == kvec->iov_len) { \
kvec++; \
skip = 0; \
} \
i->nr_segs -= kvec - i->kvec; \
i->kvec = kvec; \
i->count -= n; \
i->iov_offset = skip; \
} \
}static __wsum csum_and_memcpy(void *to, const void *from, size_t len,
__wsum sum, size_t off)
{
#ifdef dsl_patch return csum_partial_copy_nocheck(from, to, len, sum); #else __wsum next = csum_partial_copy_nocheck(from, to, len, 0); return csum_block_add(sum, next, off);
#endif
}
size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum,
struct iov_iter *i)
{
char *to = addr;
__wsum sum, next;
size_t off = 0;
sum = *csum;
iterate_and_advance(i, bytes, v, , ,({
sum = csum_and_memcpy((to += v.iov_len) - v.iov_len,
v.iov_base, v.iov_len,
sum, off);
off += v.iov_len;
#ifdef dsl_patch
if (v.iov_len & 1)
sum = __builtin_bswap32(sum);
#endif
})
)
#ifdef dsl_patch
if (off & 1)
sum = __builtin_bswap32(sum);
#endif
*csum = sum;
return bytes;
}//////////////////////////////////////////////////////////////////////
void ex_handler_uaccess(void) { }
void ex_handler_default(void) { }
static char data[65536] = {
0x46,0x56,0x20,0x04,0x00,0x02,0x00,0x00,0x72,0x4d,0xc6,0x3d,0x31,0x85,0x2d,0xbd, 0xe2,0xe0,0x9d,0x3e,0x3b,0x7a,0x70,0x3d,0xd2,0xfb,0x8c,0xbf,0x95,0x10,0xa9,0xbe, 0xeb,0xfd,0x29,0x40,0xd5,0x7a,0x61,0x40,0xde,0xcd,0x14,0xbf,0x81,0x1b,0xf6,0x3f, 0xbc,0xff,0x17,0x3f,0x67,0x1c,0x6e,0xbe,0xf4,0xc2,0x05,0x40,0x0b,0x13,0x78,0x3f, 0xfe,0x47,0xa7,0xbd,0x59,0xc2,0x15,0x3f,0x07,0xd0,0xea,0xbf,0x97,0xf1,0x3c,0x3f, 0xcc,0xfa,0x6b,0x40,0x72,0x6a,0x4f,0xbe,0x0b,0xe3,0x75,0x3e,0x3c,0x9b,0x0e,0xbf, 0xa9,0xeb,0xb7,0x3f,0xeb,0x4a,0xec,0x3e,0x33,0x8c,0x0c,0x3f,0x6a,0xf2,0xf3,0x3e, 0x2b,0x45,0x86,0x3f,0x83,0xce,0x8a,0x3f,0xf6,0x01,0x16,0x40,0x9c,0x17,0x47,0x3e, 0x44,0x83,0x61,0x40,0x74,0xc7,0x5c,0x3f,0xec,0xe7,0x95,0x3f,0xee,0x19,0xb5,0xbf, 0xb5,0xf0,0x03,0xbf,0xd1,0x02,0x1c,0x3e,0xa3,0x55,0x90,0xbe,0x1e,0x0b,0xa1,0xbf, 0xa4,0xa8,0xb4,0x3f,0xc6,0x68,0x91,0x3f,0xd1,0xc5,0xab,0x3f,0xb9,0x14,0x62,0x3f, 0x7c,0xe0,0xb9,0xbf,0xc0,0xa4,0xb5,0x3d,0x6f,0xd9,0xa7,0x3f,0x8f,0xc4,0xb0,0x3d, 0x48,0x2c,0x7a,0x3e,0x83,0xb2,0x3c,0x40,0x36,0xd3,0x18,0x40,0xb7,0xa9,0x57,0x40, 0xda,0xd3,0x95,0x3f,0x74,0x95,0xc0,0xbe,0xbb,0xce,0x71,0x3e,0x95,0xec,0x18,0xbf, 0x94,0x17,0xdd,0x3f,0x98,0xa5,0x02,0x3f,0xbb,0xfb,0xbb,0x3e,0xd0,0x5a,0x9c,0x3f, 0xd4,0x00,0x9b,0xbf,0x3b,0x9f,0x20,0xc0,0x84,0x5b,0x0f,0x40,0x5e,0x48,0x2c,0xbf,
};
#if 0
struct kvec {
size_t iov_len;
void *iov_base;
};struct iov_iter {
unsigned int count;
unsigned int nr_segs;
const struct kvec *kvec;
size_t iov_offset;
};#endif
static inline unsigned int rdpmc(unsigned int counter)
{
unsigned int low, high;
asm volatile("rdpmc" : "=a" (low), "=d" (high) : "c" (counter));
// return low bits, counter might to 32 or 40 bits wide.
return low;
}unsigned int read_cpu_cycles(void)
{
static struct perf_event_attr perf_attr = {
.type = PERF_TYPE_HARDWARE,
.config = PERF_COUNT_HW_CPU_CYCLES,
// .config = PERF_COUNT_HW_INSTRUCTIONS,
.pinned = 1,
};
static struct perf_event_mmap_page *pc;
unsigned int seq, idx, count;
if (!pc) {
int perf_fd;
perf_fd = syscall(__NR_perf_event_open, &perf_attr, 0, -1, -1, 0);
if (perf_fd < 0) {
fprintf(stderr, "perf_event_open failed: errno %d\n", errno);
exit(1);
}
pc = mmap(NULL, 4096, PROT_READ, MAP_SHARED, perf_fd, 0);
if (pc == MAP_FAILED) {
fprintf(stderr, "perf_event mmap() failed: errno %d\n", errno);
exit(1);
}
}
do {
seq = pc->lock;
asm volatile("":::"memory");
idx = pc->index;
if (!idx) // || !pc->cap_user_rdpmc)
return 0;
count = pc->offset + rdpmc(idx - 1);
asm volatile("":::"memory");
} while (pc->lock != seq);
return count;
}static int target[16 * sizeof data / 4];
#define PASSES 16
int main(int argc, char **argv)
{
struct kvec kvec[16];
struct iov_iter i;
int len;
unsigned int clocks[PASSES];
__wsum csum[PASSES] = {};
unsigned int pass;
unsigned int frag_len;
read_cpu_cycles();
clocks[0] = read_cpu_cycles();
frag_len = argv[1] ? atoi(argv[1]) : 0;
if (!frag_len || frag_len > sizeof data)
frag_len = sizeof data;
for (pass = 1; pass < PASSES; pass++) {
/* Sum the same data 16 times */
i.count = frag_len * 16;
i.nr_segs = 16;
i.kvec = kvec;
i.iov_offset = 0;
for (len = 0; len < 16; len++) {
kvec[len].iov_len = frag_len;
kvec[len].iov_base = data;
}
csum_and_copy_from_iter(target, i.count, csum + pass, &i);
clocks[pass] = read_cpu_cycles();
}
for (pass = 1; pass < PASSES; pass++) {
unsigned int delta = clocks[pass] - clocks[pass - 1];
printf("pass %d: length %d, csum %x, clocks %d, clocks/word %5f\n",
pass, frag_len * 16, fold(csum[pass]), delta, delta / (frag_len * 16/8 + 0.0));
}
return 0;
}RE: [PATCH 04/18] csum_and_copy_..._user(): pass 0xffffffff instead of 0 as initial sum
Quoted 3 lines
I had to replace the
ror32()with__builtin_bswap32(). The kernel object do contain the 'ror' instruction - even though I didn't find the asm for it.
Looking at some instruction timings ror32() and bswap32() seem to need one of the same execution ports. However on Intel cpus bswap64() takes 2 clocks but the ror64() instructions only take 1.
AMD cpus are more symmetric and run all variant in 1 clock.
So ror32() is probably preferable in case anyone copies the code into somewhere with 64bit checksum value.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Al Viro
maintainerRe: [PATCH 04/18] csum_and_copy_..._user(): pass 0xffffffff instead of 0 as initial sum
In reply to David LaightQuoted 17 lines · David Laight
From: Al Viro
Quoted 3 lines
Sent: 22 July 2020 18:39
I would love to see your patch, anyway, along with the testcases and performance comparison.See attached program.
Compile and run (as root):csum_iov1Unpatched (as shipped) 16 vectors of 1 byte take ~430 clocks on my haswell cpu. With
dsl_patchdefined they take ~393.The maximum throughput is ~1.16 clocks/word for 16 vectors of 1k. For longer vectors the data gets lost from the cache between the iterations.
On an older Ivy Bridge cpu it never goes faster than 2 clocks/word. (Due to the implementation of ADC.)
Quoted 6 lines
The absolute limit is 1 clock/word - limited by the memory write. I suspect that is achievable on Haswell with much less loop unrolling.
I had to replace the
ror32()with__builtin_bswap32(). The kernel object do contain the 'ror' instruction - even though I didn't find the asm for it.
First of all,
; git grep -n -w ror32|grep '\.h:'include/linux/bitops.h:109: * ror32 - rotate a 32-bit value right include/linux/bitops.h:113:static inline __u32 ror32(__u32 word, unsigned int shift)include/net/checksum.h:81: sum = ror32(sum, 8);
; grep -A3 ror32 include/linux/bitops.h
* ror32 - rotate a 32-bit value right
* @word: value to rotate
* @shift: bits to roll
*/
static inline __u32 ror32(__u32 word, unsigned int shift)
{
return (word >> (shift & 31)) | (word << ((-shift) & 31));
}
; cat >/tmp/a.c <<'EOF'
unsigned f(unsigned n)
{
return (n >> 8) | (n << 24);
}EOF
; gcc -c -O2 /tmp/a.c -o /tmp/a.o
; objdump /tmp/a.o/tmp/a.o: file format elf64-x86-64
Disassembly of section .text:
0000000000000000 <f>:
0: 89 f8 mov %edi,%eax 2: c1 c8 08 ror $0x8,%eax 5: c3 retq
;
which ought to cover _that_ question. Takes a couple of minutes, but that's a trivial side issue.
Said that, what you've printed for 1-byte segments (and that's going to be seriously affected by the setup costs in csum-copy.S, sensitive to calling convention changes) is time to run the 16-iteration loop divided by 1 * 16 / 8; IOW, your difference for 16 iterations here is 37*2 = 74 cycles. With per-iteration diff being a bit under 5 cycles. Which is not implausible, but
1) extrapolating to other compiler versions, flags, etc. is not obvious
2) the effects of calling convention changes need to be taken into account
3) for copying to/from userland the effects of calling convention changes
are be even larger, and kernel is certainly not going to issue kvec iters of _that_ sort, TYVM.
RE: [PATCH 04/18] csum_and_copy_..._user(): pass 0xffffffff instead of 0 as initial sum
From: Al Viro
Quoted 27 lines
Sent: 23 July 2020 15:54
Quoted 17 lines · David Laight
From: Al Viro
Quoted 3 lines
Sent: 22 July 2020 18:39
I would love to see your patch, anyway, along with the testcases and performance comparison.See attached program.
Compile and run (as root):csum_iov1Unpatched (as shipped) 16 vectors of 1 byte take ~430 clocks on my haswell cpu. With
dsl_patchdefined they take ~393.The maximum throughput is ~1.16 clocks/word for 16 vectors of 1k. For longer vectors the data gets lost from the cache between the iterations.
On an older Ivy Bridge cpu it never goes faster than 2 clocks/word. (Due to the implementation of ADC.)
Quoted 6 lines
The absolute limit is 1 clock/word - limited by the memory write. I suspect that is achievable on Haswell with much less loop unrolling.
I had to replace the
ror32()with__builtin_bswap32(). The kernel object do contain the 'ror' instruction - even though I didn't find the asm for it.First of all,
...
Quoted 5 lines
static inline
__u32ror32(__u32 word, unsigned int shift){ return (word >> (shift & 31)) | (word << ((-shift) & 31)); }; cat >/tmp/a.c <<'EOF'
...
Quoted 2 lines
which ought to cover _that_ question. Takes a couple of minutes, but that's a trivial side issue.
I did find that function. Typing __builtin_bswap32() only took seconds.
Quoted 11 lines
Said that, what you've printed for 1-byte segments (and that's going to be seriously affected by the setup costs in
csum-copy.S, sensitive to calling convention changes) is time to run the 16-iteration loop divided by 1 * 16 / 8; IOW, your difference for 16 iterations here is 37*2 = 74 cycles. With per-iteration diff being a bit under 5 cycles. Which is not implausible, but
1) extrapolating to other compiler versions, flags, etc. is not obvious
2) the effects of calling convention changes need to be taken into account
3) for copying to/from userland the effects of calling convention changes
are be even larger, and kernel is certainly not going to issue kvec iters of _that_ sort, TYVM.
Agreed, I used 1 byte fragments to make changes to that particular code fragment stand out.
Running the program with different sizes shows just how much the code around the inner loop costs.
It isn't as though buffers are a nice multiple of 64 bytes.
For x86_64 the user/kernel calling conventions are much the same. Most modern ones pass a few arguments in registers so passing the old 'csum' in is probably ok.
It may even save a register spill to stack.
The extra two arguments for saving the fail address are horrid. As is passing the csum by address.
For efficiency you do want:
csum = csum_copy(dest, src, length, csum);
And it does make sense to use 0 for 'error'.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Al Viro
maintainerRe: [PATCH 04/18] csum_and_copy_..._user(): pass 0xffffffff instead of 0 as initial sum
In reply to Al ViroQuoted 11 lines · Al Viro
Said that, what you've printed for 1-byte segments (and that's going to be seriously affected by the setup costs in
csum-copy.S, sensitive to calling convention changes) is time to run the 16-iteration loop divided by 1 * 16 / 8; IOW, your difference for 16 iterations here is 37*2 = 74 cycles. With per-iteration diff being a bit under 5 cycles. Which is not implausible, but
1) extrapolating to other compiler versions, flags, etc. is not obvious
2) the effects of calling convention changes need to be taken into account
3) for copying to/from userland the effects of calling convention changes
are be even larger, and kernel is certainly not going to issue kvec iters of _that_ sort, TYVM.
To clarify it a bit: the effects of calling conventions change are mostly due to not passing (and saving) those error pointers, and that could be had with "pass the initial sum in" - just start these iov_iter.c loops with sum = ~0U and we get the same warranties re not getting 0 in absence of faults.
The point is, your "~4.5 cycles per vector" is pretty much noise and the difference between the 3-argument and 4-argument variants could easily be in the same range. It might be a valid microoptimization, it might be not. 3-argument variant is simpler and IMO in absence of strong data we ought to go with that.
RE: [PATCH 04/18] csum_and_copy_..._user(): pass 0xffffffff instead of 0 as initial sum
From: Al Viro
Quoted 1 line
Sent: 23 July 2020 16:21
...
Quoted 5 lines
The point is, your "~4.5 cycles per vector" is pretty much noise and the difference between the 3-argument and 4-argument variants could easily be in the same range. It might be a valid microoptimization, it might be not. 3-argument variant is simpler and IMO in absence of strong data we ought to go with that.
There is definitely more to be gained by rewriting the x86-86 asm.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Al Viro
maintainerPATCH 5/18 saner calling conventions for csum_and_copy_..._user()
In reply to Al ViroFrom: Al Viro <[email protected]>
All callers of these primitives will
* discard anything we might've copied in case of error
* ignore the csum value in case of error
* always pass 0xffffffff as the initial sum, so the
resulting csum value (in case of success, that is) will never be 0.
That suggest the following calling conventions:
* don't pass err_ptr - just return 0 on error.
* don't bother with zeroing destination, etc. in case of error
* don't pass the initial sum - just use 0xffffffff.
This commit does the minimal conversion in the instances of csum_and_copy_...(); the changes of actual asm code behind them are done later in the series. Note that this asm code is often shared with csum_partial_copy_nocheck(); the difference is that csum_partial_copy_nocheck() passes 0 for initial sum while csum_and_copy_..._user() pass 0xffffffff. Fortunately, we are free to pass 0xffffffff in all cases and subsequent patches will use that freedom without any special comments.
A part that could be split off: parisc and uml/i386 claimed to have csum_and_copy_to_user() instances of their own, but those were identical to the generic one, so we simply drop them. Not sure if it's worth a separate commit...
--- arch/alpha/include/asm/checksum.h | 2 +- arch/alpha/lib/csum_partial_copy.c | 25 ++++++------- arch/arm/include/asm/checksum.h | 13 ++++--- arch/m68k/include/asm/checksum.h | 3 +- arch/m68k/lib/checksum.c | 8 ++--- arch/mips/include/asm/checksum.h | 46 ++++++++++++------------ arch/parisc/include/asm/checksum.h | 20 ----------- arch/powerpc/include/asm/checksum.h | 4 +-- arch/powerpc/lib/checksum_wrappers.c | 68 +++++++++++------------------------- arch/sh/include/asm/checksum_32.h | 36 +++++++++---------- arch/sparc/include/asm/checksum_32.h | 65 ++++++++++++++++------------------ arch/sparc/include/asm/checksum_64.h | 14 ++++---- arch/x86/include/asm/checksum_32.h | 35 ++++++++----------- arch/x86/include/asm/checksum_64.h | 6 ++-- arch/x86/lib/csum-wrappers_64.c | 38 +++++++++----------- arch/x86/um/asm/checksum_32.h | 23 ------------ arch/xtensa/include/asm/checksum.h | 30 ++++++++-------- include/net/checksum.h | 15 ++++---- lib/iov_iter.c | 19 +++++----- 19 files changed, 183 insertions(+), 287 deletions(-)
arch/alpha/include/asm/checksum.h +1 -1
@@ -42,7 +42,7 @@ extern __wsum csum_partial(const void *buff, int len, __wsum sum);* better 64-bit) boundary */ #define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER -__wsum csum_and_copy_from_user (const void __user *src,void *dst,int len,__wsum sum,int *errp); +__wsum csum_and_copy_from_user (const void __user *src,void *dst,int len);__wsum csum_partial_copy_nocheck (const void *src,void *dst,int len);
arch/alpha/lib/csum_partial_copy.c +11 -14
@@ -325,30 +325,27 @@ csum_partial_cfu_unaligned(const unsigned long __user * src, }__wsum -csum_and_copy_from_user (const void __user *src,void *dst,int len, -__wsum sum,int *errp) +csum_and_copy_from_user (const void __user *src,void *dst,int len) { -unsigned long checksum = (__force u32 ) sum; +unsigned long checksum = ~0U ;unsigned long soff =7 & (unsigned long ) src;unsigned long doff =7 & (unsigned long ) dst; +int err =0 ;if (len) { -if (!access_ok (src, len)) { -if (errp) *errp = -EFAULT ; -memset (dst,0 , len); -return sum; - } +if (!access_ok (src, len)) +return 0 ;if (!doff) {if (!soff) checksum =csum_partial_cfu_aligned ( (const unsigned long __user *) src, (unsigned long *) dst, - len-8 , checksum, errp); + len-8 , checksum, &err);else checksum =csum_partial_cfu_dest_aligned ( (const unsigned long __user *) src, (unsigned long *) dst, - soff, len-8 , checksum, errp); + soff, len-8 , checksum, &err); }else {unsigned long partial_dest;ldq_u (partial_dest, dst);
@@ -357,15 +354,15 @@ csum_and_copy_from_user(const void __user *src, void *dst, int len, (const unsigned long __user *) src, (unsigned long *) dst, doff, len-8 , checksum, - partial_dest, errp); + partial_dest, &err); else checksum =csum_partial_cfu_unaligned ( (const unsigned long __user *) src, (unsigned long *) dst, soff, doff, len-8 , checksum, - partial_dest, errp); + partial_dest, &err); } - checksum =from64to16 (checksum); + checksum = err ?0 :from64to16 (checksum); }return (__force __wsum )checksum; }
@@ -378,7 +375,7 @@ csum_partial_copy_nocheck(const void *src, void *dst, int len)mm_segment_t oldfs =get_fs ();set_fs (KERNEL_DS ); checksum =csum_and_copy_from_user ((__force const void __user *)src, - dst, len,0 ,NULL ); + dst, len);set_fs (oldfs);return checksum; }
arch/arm/include/asm/checksum.h +6 -7
@@ -42,16 +42,15 @@ csum_partial_copy_from_user(const void __user *src, void *dst, int len, __wsum s#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USERstatic inline -__wsum csum_and_copy_from_user (const void __user *src,void *dst, -int len,__wsum sum,int *err_ptr) +__wsum csum_and_copy_from_user (const void __user *src,void *dst,int len) { -if (access_ok (src, len)) -return csum_partial_copy_from_user (src, dst, len, sum, err_ptr); +int err =0 ; -if (len) - *err_ptr = -EFAULT ; +if (!access_ok (src, len)) +return 0 ; -return sum; + sum =csum_partial_copy_from_user (src, dst, len, ~0U , &err); +return err ?0 : sum; }/*
arch/m68k/include/asm/checksum.h +1 -2
@@ -33,8 +33,7 @@ __wsum csum_partial(const void *buff, int len, __wsum sum);#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USERextern __wsum csum_and_copy_from_user (const void __user *src,void *dst, -int len,__wsum sum, -int *csum_err); +int len);extern __wsum csum_partial_copy_nocheck (const void *src,void *dst,int len);
arch/m68k/lib/checksum.c +3 -5
@@ -129,8 +129,7 @@ EXPORT_SYMBOL(csum_partial);*/ __wsum -csum_and_copy_from_user (const void __user *src,void *dst, -int len,__wsum sum,int *csum_err) +csum_and_copy_from_user (const void __user *src,void *dst,int len) {/* * GCC doesn't like more than 10 operands for the asm
@@ -138,6 +137,7 @@ csum_and_copy_from_user(const void __user *src, void *dst,* code. */ unsigned long tmp1, tmp2; +__wsum sum = ~0U ; __asm__("movel %2,%4\n\t" "btst #1,%4\n\t" /* Check alignment */
@@ -311,9 +311,7 @@ csum_and_copy_from_user(const void __user *src, void *dst, : "0" (sum), "1" (len), "2" (src), "3" (dst) ); - *csum_err = tmp2; - -return (sum); +return tmp2 ?0 : sum; }EXPORT_SYMBOL (csum_and_copy_from_user);
arch/mips/include/asm/checksum.h +22 -24
62 lines not on this page · raw message
arch/parisc/include/asm/checksum.h +0 -20
@@ -181,25 +181,5 @@ static __inline__ __sum16 csum_ipv6_magic(const struct in6_addr *saddr, return csum_fold(sum); } -/* -* Copy and checksum to user -*/ -#define HAVE_CSUM_COPY_USER -static __inline____wsum csum_and_copy_to_user (const void *src, -void __user *dst, -int len,__wsum sum, -int *err_ptr) -{ -/* code stolen from include/asm-mips64 */ - sum =csum_partial (src, len, sum); - -if (copy_to_user (dst, src, len)) { - *err_ptr = -EFAULT ; -return (__force __wsum)-1 ; - } - -return sum; -} -#endif
arch/powerpc/include/asm/checksum.h +2 -2
@@ -24,10 +24,10 @@ extern __wsum csum_partial_copy_generic(const void *src, void *dst,#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USERextern __wsum csum_and_copy_from_user (const void __user *src,void *dst, -int len,__wsum sum,int *err_ptr); +int len);#define HAVE_CSUM_COPY_USER extern __wsum csum_and_copy_to_user (const void *src,void __user *dst, -int len,__wsum sum,int *err_ptr); +int len);#define csum_partial_copy_nocheck (src, dst, len) \ csum_partial_copy_generic((src), (dst), (len), 0, NULL, NULL)
arch/powerpc/lib/checksum_wrappers.c +21 -47
103 lines not on this page · raw message
arch/sh/include/asm/checksum_32.h +18 -18
@@ -49,15 +49,16 @@ __wsum csum_partial_copy_nocheck(const void *src, void *dst, int len)#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USERstatic inline -__wsum csum_and_copy_from_user (const void __user *src,void *dst, -int len,__wsum sum,int *err_ptr) +__wsum csum_and_copy_from_user (const void __user *src,void *dst,int len) { -if (access_ok (src, len)) -return csum_partial_copy_generic ((__force const void *)src, dst, - len, sum, err_ptr,NULL ); -if (len) - *err_ptr = -EFAULT ; -return sum; +int err =0 ; +__wsum sum = ~0U ; + +if (!access_ok (src, len)) +return 0 ; + sum =csum_partial_copy_generic ((__force const void *)src, dst, + len, sum, &err,NULL ); +return err ?0 : sum; }/*
@@ -198,16 +199,15 @@ static inline __sum16 csum_ipv6_magic(const struct in6_addr *saddr,#define HAVE_CSUM_COPY_USER static inline __wsum csum_and_copy_to_user (const void *src,void __user *dst, -int len,__wsum sum, -int *err_ptr) +int len) { -if (access_ok (dst, len)) -return csum_partial_copy_generic ((__force const void *)src, - dst, len, sum,NULL , err_ptr); - -if (len) - *err_ptr = -EFAULT ; - -return (__force __wsum)-1 ;/* invalid checksum */ +int err =0 ; +__wsum sum = ~0U ; + +if (!access_ok (dst, len)) +return 0 ; + sum =csum_partial_copy_generic ((__force const void *)src, + dst, len, sum,NULL , &err); +return err ?0 : sum; }#endif /* __ASM_SH_CHECKSUM_H */
arch/sparc/include/asm/checksum_32.h +30 -35
91 lines not on this page · raw message
arch/sparc/include/asm/checksum_64.h +6 -8
@@ -51,12 +51,11 @@ long __csum_partial_copy_from_user(const void __user *src,static inline __wsum csum_and_copy_from_user (const void __user *src, -void *dst,int len, -__wsum sum,int *err) +void *dst,int len) { -long ret =__csum_partial_copy_from_user (src, dst, len, sum); +long ret =__csum_partial_copy_from_user (src, dst, len, ~0U );if (ret <0 ) - *err = -EFAULT ; +return 0 ;return (__force __wsum ) ret; }
@@ -70,12 +69,11 @@ long __csum_partial_copy_to_user(const void *src,static inline __wsum csum_and_copy_to_user (const void *src, -void __user *dst,int len, -__wsum sum,int *err) +void __user *dst,int len) { -long ret =__csum_partial_copy_to_user (src, dst, len, sum); +long ret =__csum_partial_copy_to_user (src, dst, len, ~0U );if (ret <0 ) - *err = -EFAULT ; +return 0 ;return (__force __wsum ) ret; }
arch/x86/include/asm/checksum_32.h +14 -21
@@ -44,22 +44,19 @@ static inline __wsum csum_partial_copy_nocheck(const void *src, void *dst, int l }static inline __wsum csum_and_copy_from_user (const void __user *src, -void *dst,int len, -__wsum sum,int *err_ptr) +void *dst,int len) {__wsum ret; +int err =0 ;might_sleep (); -if (!user_access_begin (src, len)) { -if (len) - *err_ptr = -EFAULT ; -return sum; - } +if (!user_access_begin (src, len)) +return 0 ; ret =csum_partial_copy_generic ((__force void *)src, dst, - len, sum, err_ptr,NULL ); + len, ~0U , &err,NULL );user_access_end (); -return ret; +return err ?0 : ret; }/*
@@ -177,23 +174,19 @@ static inline __sum16 csum_ipv6_magic(const struct in6_addr *saddr,*/ static inline __wsum csum_and_copy_to_user (const void *src,void __user *dst, -int len,__wsum sum, -int *err_ptr) +int len) {__wsum ret; +int err =0 ;might_sleep (); -if (user_access_begin (dst, len)) { - ret =csum_partial_copy_generic (src, (__force void *)dst, - len, sum,NULL , err_ptr); -user_access_end (); -return ret; - } +if (!user_access_begin (dst, len)) +return 0 ; -if (len) - *err_ptr = -EFAULT ; - -return (__force __wsum)-1 ;/* invalid checksum */ + ret =csum_partial_copy_generic (src, (__force void *)dst, + len, ~0U ,NULL , &err); +user_access_end (); +return err ?0 : ret; }#endif /* _ASM_X86_CHECKSUM_32_H */
arch/x86/include/asm/checksum_64.h +2 -4
@@ -135,10 +135,8 @@ extern __visible __wsum csum_partial_copy_generic(const void *src, const void *d int *src_err_ptr, int *dst_err_ptr); -extern __wsum csum_and_copy_from_user (const void __user *src,void *dst, -int len,__wsum isum,int *errp); -extern __wsum csum_and_copy_to_user (const void *src,void __user *dst, -int len,__wsum isum,int *errp); +extern __wsum csum_and_copy_from_user (const void __user *src,void *dst,int len); +extern __wsum csum_and_copy_to_user (const void *src,void __user *dst,int len);extern __wsum csum_partial_copy_nocheck (const void *src,void *dst,int len);/**
arch/x86/lib/csum-wrappers_64.c +16 -22
@@ -22,13 +22,15 @@*/ __wsum csum_and_copy_from_user (const void __user *src,void *dst, -int len,__wsum isum,int *errp) +int len) { +int err =0 ; +__wsum isum = ~0U ; +might_sleep (); - *errp =0 ;if (!user_access_begin (src, len)) - goto out_err; +return 0 ;/* * Why 6, not 7? To handle odd addresses aligned we
@@ -53,20 +55,15 @@ csum_and_copy_from_user(const void __user *src, void *dst, } } isum =csum_partial_copy_generic ((__force const void *)src, - dst, len, isum, errp,NULL ); + dst, len, isum, &err,NULL );user_access_end (); -if (unlikely (*errp)) - goto out_err; - +if (unlikely (err)) + isum =0 ;return isum; out:user_access_end (); -out_err: - *errp = -EFAULT ; -memset (dst,0 , len); - -return isum; +return 0 ; }EXPORT_SYMBOL (csum_and_copy_from_user);
@@ -83,16 +80,15 @@ EXPORT_SYMBOL(csum_and_copy_from_user);*/ __wsum csum_and_copy_to_user (const void *src,void __user *dst, -int len,__wsum isum,int *errp) +int len) { -__wsum ret; +__wsum ret, isum = ~0U ; +int err =0 ;might_sleep (); -if (!user_access_begin (dst, len)) { - *errp = -EFAULT ; +if (!user_access_begin (dst, len))return 0 ; - }if (unlikely ((unsigned long )dst &6 )) {while (((unsigned long )dst &6 ) && len >=2 ) {
@@ -107,15 +103,13 @@ csum_and_copy_to_user(const void *src, void __user *dst, } } - *errp =0 ; ret =csum_partial_copy_generic (src, (void __force *)dst, - len, isum,NULL , errp); + len, isum,NULL , &err);user_access_end (); -return ret; +return err ?0 : ret; out:user_access_end (); - *errp = -EFAULT ; -return isum; +return 0 ; }EXPORT_SYMBOL (csum_and_copy_to_user);
arch/x86/um/asm/checksum_32.h +0 -23
@@ -35,27 +35,4 @@ static __inline__ __sum16 csum_ipv6_magic(const struct in6_addr *saddr, return csum_fold(sum); } -/* -* Copy and checksum to user -*/ -#define HAVE_CSUM_COPY_USER -static __inline____wsum csum_and_copy_to_user (const void *src, -void __user *dst, -int len,__wsum sum,int *err_ptr) -{ -if (access_ok (dst, len)) { -if (copy_to_user (dst, src, len)) { - *err_ptr = -EFAULT ; -return (__force __wsum)-1 ; - } - -return csum_partial (src, len, sum); - } - -if (len) - *err_ptr = -EFAULT ; - -return (__force __wsum)-1 ;/* invalid checksum */ -} -#endif
arch/xtensa/include/asm/checksum.h +16 -14
@@ -54,14 +54,16 @@ __wsum csum_partial_copy_nocheck(const void *src, void *dst, int len)#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USERstatic inline __wsum csum_and_copy_from_user (const void __user *src,void *dst, -int len,__wsum sum,int *err_ptr) +int len) { -if (access_ok (dst, len)) -return csum_partial_copy_generic ((__force const void *)src, dst, - len, sum, err_ptr,NULL ); -if (len) - *err_ptr = -EFAULT ; -return sum; +int err =0 ; + +if (!access_ok (dst, len)) +return 0 ; + + sum =csum_partial_copy_generic ((__force const void *)src, dst, + len, ~0U , &err,NULL ); +return err ?0 : sum; }/*
@@ -242,15 +244,15 @@ static __inline__ __sum16 csum_ipv6_magic(const struct in6_addr *saddr,*/ #define HAVE_CSUM_COPY_USER static __inline____wsum csum_and_copy_to_user (const void *src, -void __user *dst,int len, -__wsum sum,int *err_ptr) +void __user *dst,int len) { -if (access_ok (dst, len)) -return csum_partial_copy_generic (src,dst,len,sum,NULL ,err_ptr); +int err =0 ; +__wsum sum = ~0U ; -if (len) - *err_ptr = -EFAULT ; +if (!access_ok (dst, len)) +return 0 ; -return (__force __wsum)-1 ;/* invalid checksum */ + sum =csum_partial_copy_generic (src,dst,len,sum,NULL ,&err); +return err ?0 : sum; }#endif
include/net/checksum.h +6 -9
@@ -24,26 +24,23 @@#ifndef _HAVE_ARCH_COPY_AND_CSUM_FROM_USERstatic inline __wsum csum_and_copy_from_user (const void __user *src,void *dst, -int len,__wsum sum,int *err_ptr) +int len) {if (copy_from_user (dst, src, len)) - *err_ptr = -EFAULT ; -return csum_partial (dst, len, sum); +return 0 ; +return csum_partial (dst, len, ~0U ); }#endif #ifndef HAVE_CSUM_COPY_USER static __inline____wsum csum_and_copy_to_user -(const void *src,void __user *dst,int len,__wsum sum,int *err_ptr) +(const void *src,void __user *dst,int len) { - sum =csum_partial (src, len, sum); +__wsum sum =csum_partial (src, len, ~0U );if (copy_to_user (dst, src, len) ==0 )return sum; -if (len) - *err_ptr = -EFAULT ; - -return (__force __wsum)-1 ;/* invalid checksum */ +return 0 ; }#endif
lib/iov_iter.c +8 -11
@@ -1448,15 +1448,14 @@ size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum,return 0 ; } iterate_and_advance(i, bytes, v, ({ - int err = 0; next = csum_and_copy_from_user(v.iov_base, (to += v.iov_len) - v.iov_len, - v.iov_len, ~0U, &err); - if (!err) { + v.iov_len); + if (next) { sum = csum_block_add(sum, next, off); off += v.iov_len; } - err ? v.iov_len : 0; + next ? 0 : v.iov_len; }), ({ char *p = kmap_atomic(v.bv_page); sum = csum_and_memcpy((to += v.bv_len) - v.bv_len,
@@ -1490,11 +1489,10 @@ bool csum_and_copy_from_iter_full(void *addr, size_t bytes, __wsum *csum,if (unlikely (i->count < bytes))return false; iterate_all_kinds(i, bytes, v, ({ - int err = 0; next = csum_and_copy_from_user(v.iov_base, (to += v.iov_len) - v.iov_len, - v.iov_len, ~0U, &err); - if (err) + v.iov_len); + if (!next) return false; sum = csum_block_add(sum, next, off); off += v.iov_len;
@@ -1536,15 +1534,14 @@ size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *csump,return 0 ; } iterate_and_advance(i, bytes, v, ({ - int err = 0; next = csum_and_copy_to_user((from += v.iov_len) - v.iov_len, v.iov_base, - v.iov_len, ~0U, &err); - if (!err) { + v.iov_len); + if (next) { sum = csum_block_add(sum, next, off); off += v.iov_len; } - err ? v.iov_len : 0; + next ? 0 : v.iov_len; }), ({ char *p = kmap_atomic(v.bv_page); sum = csum_and_memcpy(p + v.bv_offset,
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH 6/18 alpha: propagate the calling convention changes down to csum_partial_copy.c helpers
In reply to Al ViroFrom: Al Viro <[email protected]>
get rid of set_fs() in csum_partial_copy_nocheck(), while we are at it - just take the part of csum_and_copy_from_user() sans the access_ok() check into a helper function and have csum_partial_copy_nocheck() call that.
--- arch/alpha/lib/csum_partial_copy.c | 157 ++++++++++++++++--------------------- 1 file changed, 69 insertions(+), 88 deletions(-)
arch/alpha/lib/csum_partial_copy.c +69 -88
@@ -39,12 +39,11 @@ __asm__ __volatile__("insql %1,%2,%0":"=r" (z):"r" (x),"r" (y))#define insqh (x,y,z) \ __asm__ __volatile__("insqh %1,%2,%0":"=r" (z):"r" (x),"r" (y)) - -#define __get_user_u (x,ptr) \ +#define __get_word (insn,x,ptr) \ ({ \ long __guu_err; \ __asm__ __volatile__( \ - "1: ldq_u %0,%2\n" \ + "1: "#insn" %0,%2\n" \ "2:\n" \ EXC(1b,2b,%0,%1) \ : "=r"(x), "=r"(__guu_err) \
@@ -52,19 +51,6 @@ __asm__ __volatile__("insqh %1,%2,%0":"=r" (z):"r" (x),"r" (y)) __guu_err; \ }) -#define __put_user_u (x,ptr) \ -({ \ - long __puu_err; \ - __asm__ __volatile__( \ - "1: stq_u %2,%1\n" \ - "2:\n" \ - EXC(1b,2b,$31,%0) \ - : "=r"(__puu_err) \ - : "m"(__m(addr)), "rJ"(x), "0"(0)); \ - __puu_err; \ -}) - -static inline unsigned short from64to16 (unsigned long x) {/* Using extract instructions is a bit more efficient
@@ -95,15 +81,15 @@ static inline unsigned short from64to16(unsigned long x)*/ static inline unsigned long csum_partial_cfu_aligned (const unsigned long __user *src,unsigned long *dst, -long len,unsigned long checksum, -int *errp) +long len) { +unsigned long checksum = ~0U ;unsigned long carry =0 ; -int err =0 ;while (len >=0 ) {unsigned long word; - err |=__get_user (word, src); +if (__get_word (ldq, word, src)) +return 0 ; checksum += carry; src++; checksum += word;
@@ -116,7 +102,8 @@ csum_partial_cfu_aligned(const unsigned long __user *src, unsigned long *dst, checksum += carry;if (len) {unsigned long word, tmp; - err |=__get_user (word, src); +if (__get_word (ldq, word, src)) +return 0 ; tmp = *dst;mskql (word, len, word); checksum += word;
@@ -125,7 +112,6 @@ csum_partial_cfu_aligned(const unsigned long __user *src, unsigned long *dst, *dst = word | tmp; checksum += carry; } -if (err && errp) *errp = err;return checksum; }
@@ -137,20 +123,21 @@ static inline unsigned long csum_partial_cfu_dest_aligned(const unsigned long __user *src, unsigned long *dst, unsigned long soff, - long len, unsigned long checksum, - int *errp) + long len) { unsigned long first;unsigned long word, carry;unsigned long lastsrc =7 +len+(unsigned long )src; -int err =0 ; +unsigned long checksum = ~0U ; - err |=__get_user_u (first,src); +if (__get_word (ldq_u, first,src)) +return 0 ; carry =0 ;while (len >=0 ) {unsigned long second; - err |=__get_user_u (second, src+1 ); +if (__get_word (ldq_u, second, src+1 )) +return 0 ;extql (first, soff, word); len -=8 ; src++;
@@ -168,7 +155,8 @@ csum_partial_cfu_dest_aligned(const unsigned long __user *src,if (len) {unsigned long tmp;unsigned long second; - err |=__get_user_u (second, lastsrc); +if (__get_word (ldq_u, second, lastsrc)) +return 0 ; tmp = *dst;extql (first, soff, word);extqh (second, soff, first);
@@ -180,7 +168,6 @@ csum_partial_cfu_dest_aligned(const unsigned long __user *src, *dst = word | tmp; checksum += carry; } -if (err && errp) *errp = err;return checksum; }
@@ -191,18 +178,18 @@ static inline unsigned long csum_partial_cfu_src_aligned(const unsigned long __user *src, unsigned long *dst, unsigned long doff, - long len, unsigned long checksum, - unsigned long partial_dest, - int *errp) + long len, + unsigned long partial_dest) { unsigned long carry = 0;unsigned long word;unsigned long second_dest; -int err =0 ; +unsigned long checksum = ~0U ;mskql (partial_dest, doff, partial_dest);while (len >=0 ) { - err |=__get_user (word, src); +if (__get_word (ldq, word, src)) +return 0 ; len -=8 ;insql (word, doff, second_dest); checksum += carry;
@@ -216,7 +203,8 @@ csum_partial_cfu_src_aligned(const unsigned long __user *src, len +=8 ;if (len) { checksum += carry; - err |=__get_user (word, src); +if (__get_word (ldq, word, src)) +return 0 ;mskql (word, len, word); len -=8 ; checksum += word;
@@ -237,7 +225,6 @@ csum_partial_cfu_src_aligned(const unsigned long __user *src,stq_u (partial_dest | second_dest, dst); out: checksum += carry; -if (err && errp) *errp = err;return checksum; }
@@ -249,23 +236,23 @@ static inline unsigned long csum_partial_cfu_unaligned(const unsigned long __user * src, unsigned long * dst, unsigned long soff, unsigned long doff, - long len, unsigned long checksum, - unsigned long partial_dest, - int *errp) + long len, unsigned long partial_dest) { unsigned long carry = 0;unsigned long first;unsigned long lastsrc; -int err =0 ; +unsigned long checksum = ~0U ; - err |=__get_user_u (first, src); +if (__get_word (ldq_u, first, src)) +return 0 ; lastsrc =7 +len+(unsigned long )src;mskql (partial_dest, doff, partial_dest);while (len >=0 ) {unsigned long second, word;unsigned long second_dest; - err |=__get_user_u (second, src+1 ); +if (__get_word (ldq_u, second, src+1 )) +return 0 ;extql (first, soff, word); checksum += carry; len -=8 ;
@@ -286,7 +273,8 @@ csum_partial_cfu_unaligned(const unsigned long __user * src,unsigned long second, word;unsigned long second_dest; - err |=__get_user_u (second, lastsrc); +if (__get_word (ldq_u, second, lastsrc)) +return 0 ;extql (first, soff, word);extqh (second, soff, first); word |= first;
@@ -307,7 +295,8 @@ csum_partial_cfu_unaligned(const unsigned long __user * src,unsigned long second, word;unsigned long second_dest; - err |=__get_user_u (second, lastsrc); +if (__get_word (ldq_u, second, lastsrc)) +return 0 ;extql (first, soff, word);extqh (second, soff, first); word |= first;
@@ -320,63 +309,55 @@ csum_partial_cfu_unaligned(const unsigned long __user * src,stq_u (partial_dest | word | second_dest, dst); checksum += carry; } -if (err && errp) *errp = err;return checksum; } -__wsum -csum_and_copy_from_user (const void __user *src,void *dst,int len) +static __wsum __csum_and_copy (const void __user *src,void *dst,int len) { -unsigned long checksum = ~0U ;unsigned long soff =7 & (unsigned long ) src;unsigned long doff =7 & (unsigned long ) dst; -int err =0 ; - -if (len) { -if (!access_ok (src, len)) -return 0 ; -if (!doff) { -if (!soff) - checksum =csum_partial_cfu_aligned ( - (const unsigned long __user *) src, - (unsigned long *) dst, - len-8 , checksum, &err); -else - checksum =csum_partial_cfu_dest_aligned ( - (const unsigned long __user *) src, - (unsigned long *) dst, - soff, len-8 , checksum, &err); - }else { -unsigned long partial_dest; -ldq_u (partial_dest, dst); -if (!soff) - checksum =csum_partial_cfu_src_aligned ( - (const unsigned long __user *) src, - (unsigned long *) dst, - doff, len-8 , checksum, - partial_dest, &err); -else - checksum =csum_partial_cfu_unaligned ( - (const unsigned long __user *) src, - (unsigned long *) dst, - soff, doff, len-8 , checksum, - partial_dest, &err); - } - checksum = err ?0 :from64to16 (checksum); +unsigned long checksum; + +if (!doff) { +if (!soff) + checksum =csum_partial_cfu_aligned ( + (const unsigned long __user *) src, + (unsigned long *) dst, len-8 ); +else + checksum =csum_partial_cfu_dest_aligned ( + (const unsigned long __user *) src, + (unsigned long *) dst, + soff, len-8 ); + }else { +unsigned long partial_dest; +ldq_u (partial_dest, dst); +if (!soff) + checksum =csum_partial_cfu_src_aligned ( + (const unsigned long __user *) src, + (unsigned long *) dst, + doff, len-8 , partial_dest); +else + checksum =csum_partial_cfu_unaligned ( + (const unsigned long __user *) src, + (unsigned long *) dst, + soff, doff, len-8 , partial_dest); } -return (__force __wsum )checksum; +return (__force __wsum )from64to16 (checksum); +} + +__wsum +csum_and_copy_from_user (const void __user *src,void *dst,int len) +{ +if (!access_ok (src, len)) +return 0 ; +return __csum_and_copy (src, dst, len); }EXPORT_SYMBOL (csum_and_copy_from_user);__wsum csum_partial_copy_nocheck (const void *src,void *dst,int len) { -__wsum checksum; -mm_segment_t oldfs =get_fs (); -set_fs (KERNEL_DS ); - checksum =csum_and_copy_from_user ((__force const void __user *)src, +return __csum_and_copy ((__force const void __user *)src, dst, len); -set_fs (oldfs); -return checksum; }EXPORT_SYMBOL (csum_partial_copy_nocheck);
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH 7/18 arm: propagate the calling convention changes down to csum_partial_copy_from_user()
In reply to Al ViroFrom: Al Viro <[email protected]>
... and get rid of the "clean the destination on error" crap. Simplifies the fault handlers and the function itself...
--- arch/arm/include/asm/checksum.h | 7 ++----- arch/arm/lib/csumpartialcopy.S | 1 - arch/arm/lib/csumpartialcopygeneric.S | 1 + arch/arm/lib/csumpartialcopyuser.S | 26 ++++++-------------------- 4 files changed, 9 insertions(+), 26 deletions(-)
arch/arm/include/asm/checksum.h +2 -5
@@ -38,19 +38,16 @@ __wsum csum_partial_copy_nocheck(const void *src, void *dst, int len);__wsum -csum_partial_copy_from_user (const void __user *src,void *dst,int len,__wsum sum,int *err_ptr); +csum_partial_copy_from_user (const void __user *src,void *dst,int len);#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USERstatic inline __wsum csum_and_copy_from_user (const void __user *src,void *dst,int len) { -int err =0 ; -if (!access_ok (src, len))return 0 ; - sum =csum_partial_copy_from_user (src, dst, len, ~0U , &err); -return err ?0 : sum; +return csum_partial_copy_from_user (src, dst, len); }/*
arch/arm/lib/csumpartialcopy.S +0 -1
@@ -16,7 +16,6 @@ .macro save_regs stmfd sp!, {r1, r4 - r8, lr} - mov r3, #0 .endm .macro load_regs
arch/arm/lib/csumpartialcopygeneric.S +1 -0
@@ -86,6 +86,7 @@ sum .req r3 FN_ENTRY save_regs + mov sum, #-1 cmp len, #8 @ Ensure that we have at least blo .Lless8 @ 8 bytes to copy.
arch/arm/lib/csumpartialcopyuser.S +6 -20
@@ -62,9 +62,9 @@ /* * unsigned int - * csum_partial_copy_from_user(const char *src, char *dst, int len, int sum, int *err_ptr) - * r0 = src, r1 = dst, r2 = len, r3 = sum, [sp] = *err_ptr - * Returns : r0 = checksum, [[sp, #0], #0] = 0 or -EFAULT + * csum_partial_copy_from_user(const char *src, char *dst, int len) + * r0 = src, r1 = dst, r2 = len + * Returns : r0 = checksum or 0 */ #define FN_ENTRY ENTRY(csum_partial_copy_from_user)
@@ -73,25 +73,11 @@ #include "csumpartialcopygeneric.S" /* - * FIXME: minor buglet here - * We don't return the checksum for the data present in the buffer. To do - * so properly, we would have to add in whatever registers were loaded before - * the fault, which, with the current asm above is not predictable. + * We report fault by returning 0 csum - impossible in normal case, since + * we start with 0xffffffff for initial sum. */ .pushsection .text.fixup,"ax" .align 4 -9001: mov r4, #-EFAULT -#ifdef CONFIG_CPU_SW_DOMAIN_PAN - ldr r5, [sp, #9*4] @ *err_ptr -#else - ldr r5, [sp, #8*4] @ *err_ptr -#endif - str r4, [r5] - ldmia sp, {r1, r2} @ retrieve dst, len - add r2, r2, r1 - mov r0, #0 @ zero the buffer -9002: teq r2, r1 - strbne r0, [r1], #1 - bne 9002b +9001: mov r0, #0 load_regs .popsection
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH 8/18 m68k: get rid of zeroing destination on error in csum_and_copy_from_user()
In reply to Al ViroFrom: Al Viro <[email protected]>
--- arch/m68k/lib/checksum.c | 79 +++++++++--------------------------------------- 1 file changed, 15 insertions(+), 64 deletions(-)
arch/m68k/lib/checksum.c +15 -64
@@ -236,82 +236,33 @@ csum_and_copy_from_user(const void __user *src, void *dst, int len) "clrl %5\n\t" "addxl %5,%0\n\t" /* add X bit */ "7:\t" -"clrl %5\n" /* no error - clear return value */ -"8:\n" ".section .fixup,\"ax\"\n" ".even\n" -/* If any exception occurs zero out the rest. -Similarities with the code above are intentional :-) */ + /* If any exception occurs, return 0 */ "90:\t" -"clrw %3@+\n\t" -"movel %1,%4\n\t" -"lsrl #5,%1\n\t" -"jeq 1f\n\t" -"subql #1,%1\n" -"91:\t" -"clrl %3@+\n" -"92:\t" -"clrl %3@+\n" -"93:\t" -"clrl %3@+\n" -"94:\t" -"clrl %3@+\n" -"95:\t" -"clrl %3@+\n" -"96:\t" -"clrl %3@+\n" -"97:\t" -"clrl %3@+\n" -"98:\t" -"clrl %3@+\n\t" -"dbra %1,91b\n\t" -"clrw %1\n\t" -"subql #1,%1\n\t" -"jcc 91b\n" -"1:\t" -"movel %4,%1\n\t" -"andw #0x1c,%4\n\t" -"jeq 1f\n\t" -"lsrw #2,%4\n\t" -"subqw #1,%4\n" -"99:\t" -"clrl %3@+\n\t" -"dbra %4,99b\n\t" -"1:\t" -"andw #3,%1\n\t" -"jeq 9f\n" -"100:\t" -"clrw %3@+\n\t" -"tstw %1\n\t" -"jeq 9f\n" -"101:\t" -"clrb %3@+\n" - "9:\t" -#define STR (X ) STR1(X) -#define STR1 (X ) #X - "moveq #-" STR(EFAULT) ",%5\n\t" - "jra 8b\n" + "clrl %0\n" + "jra 7b\n" ".previous\n" ".section __ex_table,\"a\"\n" ".long 10b,90b\n" - ".long 11b,91b\n" - ".long 12b,92b\n" - ".long 13b,93b\n" - ".long 14b,94b\n" - ".long 15b,95b\n" - ".long 16b,96b\n" - ".long 17b,97b\n" - ".long 18b,98b\n" - ".long 19b,99b\n" - ".long 20b,100b\n" - ".long 21b,101b\n" + ".long 11b,90b\n" + ".long 12b,90b\n" + ".long 13b,90b\n" + ".long 14b,90b\n" + ".long 15b,90b\n" + ".long 16b,90b\n" + ".long 17b,90b\n" + ".long 18b,90b\n" + ".long 19b,90b\n" + ".long 20b,90b\n" + ".long 21b,90b\n" ".previous" : "=d" (sum), "=d" (len), "=a" (src), "=a" (dst), "=&d" (tmp1), "=d" (tmp2) : "0" (sum), "1" (len), "2" (src), "3" (dst) ); -return tmp2 ?0 : sum; +return sum; }EXPORT_SYMBOL (csum_and_copy_from_user);
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH 9/18 sh: propage the calling conventions change down to csum_partial_copy_generic()
In reply to Al ViroFrom: Al Viro <[email protected]>
... and get rid of zeroing destination on error there.
--- arch/sh/include/asm/checksum_32.h | 20 ++----- arch/sh/lib/checksum.S | 119 +++++++++++--------------------------- 2 files changed, 39 insertions(+), 100 deletions(-)
arch/sh/include/asm/checksum_32.h +4 -16
@@ -30,9 +30,7 @@ asmlinkage __wsum csum_partial(const void *buff, int len, __wsum sum);* better 64-bit) boundary */ -asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst, -int len,__wsum sum, -int *src_err_ptr,int *dst_err_ptr); +asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst, int len);/* * Note: when you get a NULL pointer exception here this means someone
@@ -44,21 +42,16 @@ asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst,static inline __wsum csum_partial_copy_nocheck (const void *src,void *dst,int len) { -return csum_partial_copy_generic (src, dst, len,0 ,NULL ,NULL ); +return csum_partial_copy_generic (src, dst, len); }#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USERstatic inline __wsum csum_and_copy_from_user (const void __user *src,void *dst,int len) { -int err =0 ; -__wsum sum = ~0U ; -if (!access_ok (src, len))return 0 ; - sum =csum_partial_copy_generic ((__force const void *)src, dst, - len, sum, &err,NULL ); -return err ?0 : sum; +return csum_partial_copy_generic ((__force const void *)src, dst, len); }/*
@@ -201,13 +194,8 @@ static inline __wsum csum_and_copy_to_user(const void *src, void __user *dst, int len) { - int err = 0; -__wsum sum = ~0U ; -if (!access_ok (dst, len))return 0 ; - sum =csum_partial_copy_generic ((__force const void *)src, - dst, len, sum,NULL , &err); -return err ?0 : sum; +return csum_partial_copy_generic ((__force const void *)src, dst, len); }#endif /* __ASM_SH_CHECKSUM_H */
arch/sh/lib/checksum.S +35 -84
200 lines not on this page · raw message
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH 10/18 i386: propagate the calling conventions change down to csum_partial_copy_generic()
In reply to Al ViroFrom: Al Viro <[email protected]>
... and don't bother zeroing destination on error
--- arch/x86/include/asm/checksum_32.h | 18 ++---- arch/x86/lib/checksum_32.S | 117 +++++++++++++------------------------ 2 files changed, 47 insertions(+), 88 deletions(-)
arch/x86/include/asm/checksum_32.h +6 -12
@@ -27,9 +27,7 @@ asmlinkage __wsum csum_partial(const void *buff, int len, __wsum sum);* better 64-bit) boundary */ -asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst, -int len,__wsum sum, -int *src_err_ptr,int *dst_err_ptr); +asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst, int len);/* * Note: when you get a NULL pointer exception here this means someone
@@ -40,23 +38,21 @@ asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst,*/ static inline __wsum csum_partial_copy_nocheck (const void *src,void *dst,int len) { -return csum_partial_copy_generic (src, dst, len,0 ,NULL ,NULL ); +return csum_partial_copy_generic (src, dst, len); }static inline __wsum csum_and_copy_from_user (const void __user *src,void *dst,int len) {__wsum ret; -int err =0 ;might_sleep ();if (!user_access_begin (src, len))return 0 ; - ret =csum_partial_copy_generic ((__force void *)src, dst, - len, ~0U , &err,NULL ); + ret =csum_partial_copy_generic ((__force void *)src, dst, len);user_access_end (); -return err ?0 : ret; +return ret; }/*
@@ -177,16 +173,14 @@ static inline __wsum csum_and_copy_to_user(const void *src, int len) {__wsum ret; -int err =0 ;might_sleep ();if (!user_access_begin (dst, len))return 0 ; - ret =csum_partial_copy_generic (src, (__force void *)dst, - len, ~0U ,NULL , &err); + ret =csum_partial_copy_generic (src, (__force void *)dst, len);user_access_end (); -return err ?0 : ret; +return ret; }#endif /* _ASM_X86_CHECKSUM_32_H */
arch/x86/lib/checksum_32.S +41 -76
228 lines not on this page · raw message
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH 11/18 sparc32: propagate the calling conventions change down to __csum_partial_copy_sparc_generic()
In reply to Al ViroFrom: Al Viro <[email protected]>
... and get rid of zeroing the target, etc. on fault.
All exception handlers merge into one; moreover, since we are not calling lookup_fault() anymore, we don't need the magic with passing arguments for it from the page fault handler.
--- arch/sparc/include/asm/checksum_32.h | 49 +-------- arch/sparc/lib/checksum_32.S | 202 +++++++---------------------------- arch/sparc/mm/fault_32.c | 6 +- 3 files changed, 44 insertions(+), 213 deletions(-)
arch/sparc/include/asm/checksum_32.h +5 -44
@@ -50,9 +50,9 @@ csum_partial_copy_nocheck(const void *src, void *dst, int len) __asm__ __volatile__ ("call __csum_partial_copy_sparc_generic\n\t" -" mov %6, %%g7\n" +" mov -1, %%g7\n" :"=&r" (ret),"=&r" (d),"=&r" (l) - :"0" (ret),"1" (d),"2" (l),"r" (0 ) + :"0" (ret),"1" (d),"2" (l) :"o2" ,"o3" ,"o4" ,"o5" ,"o7" ,"g2" ,"g3" ,"g4" ,"g5" ,"g7" ,"memory" ,"cc" );
@@ -61,29 +61,10 @@ csum_partial_copy_nocheck(const void *src, void *dst, int len)static inline __wsum csum_and_copy_from_user (const void __user *src,void *dst,int len) - { - register unsigned long ret asm("o0") = (unsigned long)src; - register char *d asm("o1") = dst; - register int l asm("g1") = len; - register __wsum s asm("g7") = ~0U; -int err =0 ; - +{if (unlikely (!access_ok (src, len)))return 0 ; - - __asm__ __volatile__ ( -".section __ex_table,#alloc\n\t" -".align 4\n\t" -".word 1f,2\n\t" -".previous\n" -"1:\n\t" -"call __csum_partial_copy_sparc_generic\n\t" -" st %8, [%%sp + 64]\n" - :"=&r" (ret),"=&r" (d),"=&r" (l),"=&r" (s) - :"0" (ret),"1" (d),"2" (l),"3" (s),"r" (&err) - :"o2" ,"o3" ,"o4" ,"o5" ,"o7" ,"g2" ,"g3" ,"g4" ,"g5" , -"cc" ,"memory" ); -return err ?0 : (__force __wsum )ret; +return csum_partial_copy_nocheck ((__force void *)src, dst, len); }#define HAVE_CSUM_COPY_USER
@@ -91,29 +72,9 @@ csum_and_copy_from_user(const void __user *src, void *dst, int len)static inline __wsum csum_and_copy_to_user (const void *src,void __user *dst,int len) { - register unsigned long ret asm("o0") = (unsigned long)src; - register char __user *d asm("o1") = dst; - register int l asm("g1") = len; - register __wsum s asm("g7") = ~0U; -int err =0 ; -if (!access_ok (dst, len))return 0 ; - - __asm__ __volatile__ ( -".section __ex_table,#alloc\n\t" -".align 4\n\t" -".word 1f,1\n\t" -".previous\n" -"1:\n\t" -"call __csum_partial_copy_sparc_generic\n\t" -" st %8, [%%sp + 64]\n" - :"=&r" (ret),"=&r" (d),"=&r" (l),"=&r" (s) - :"0" (ret),"1" (d),"2" (l),"3" (s),"r" (&err) - :"o2" ,"o3" ,"o4" ,"o5" ,"o7" , -"g2" ,"g3" ,"g4" ,"g5" , -"cc" ,"memory" ); -return err ?0 : (__force __wsum )ret; +return csum_partial_copy_nocheck (src, (__force void *)dst, len); }/* ihl is always 5 or greater, almost always is 5, and iph is word aligned
arch/sparc/lib/checksum_32.S +38 -164
331 lines not on this page · raw message
arch/sparc/mm/fault_32.c +1 -5
@@ -297,8 +297,6 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,if (fixup >10 ) {extern const unsigned int __memset_start[];extern const unsigned int __memset_end[]; -extern const unsigned int __csum_partial_copy_start[]; -extern const unsigned int __csum_partial_copy_end[];#ifdef DEBUG_EXCEPTIONS printk("Exception: PC<%08lx> faddr<%08lx>\n",
@@ -307,9 +305,7 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write, regs->pc, fixup, g2);#endif if ((regs->pc >= (unsigned long )__memset_start && - regs->pc < (unsigned long )__memset_end) || - (regs->pc >= (unsigned long )__csum_partial_copy_start && - regs->pc < (unsigned long )__csum_partial_copy_end)) { + regs->pc < (unsigned long )__memset_end)) { regs->u_regs[UREG_I4 ] = address; regs->u_regs[UREG_I5 ] = regs->pc; }
Signature · 1 line
2.11.0
Re: [PATCH 11/18] sparc32: propagate the calling conventions change down to __csum_partial_copy_sparc_generic()
From: Al Viro <[email protected]>
Date: Tue, 21 Jul 2020 21:25:42 +0100
Quoted 8 lines
From: Al Viro <[email protected]>
... and get rid of zeroing the target, etc. on fault.
All exception handlers merge into one; moreover, since we are not callinglookup_fault()anymore, we don't need the magic with passing arguments for it from the page fault handler.Signed-off-by Al Viro <[email protected]>
Al Viro
maintainerPATCH 12/18 mips: csum_and_copy_{to,from}_user() are never called under KERNEL_DS
In reply to Al ViroFrom: Al Viro <[email protected]>
they are only called for iovec-backed iov_iter and under KERNEL_DS an attempt to create such a beast will yield a kvec-backed one.
--- arch/mips/include/asm/checksum.h | 32 +++++++------------------------- 1 file changed, 7 insertions(+), 25 deletions(-)
arch/mips/include/asm/checksum.h +7 -25
@@ -41,22 +41,6 @@ __wsum __csum_partial_copy_from_user(const void *src, void *dst, int len, __wsum sum, int *err_ptr);__wsum __csum_partial_copy_to_user (const void *src,void *dst,int len,__wsum sum,int *err_ptr); -/* -* this is a new version of the above that records errors it finds in *errp, -* but continues and zeros the rest of the buffer. -*/ -static inline -__wsum csum_partial_copy_from_user (const void __user *src,void *dst,int len, -__wsum sum,int *err_ptr) -{ -might_fault (); -if (uaccess_kernel ()) -return __csum_partial_copy_kernel ((__force void *)src, dst, - len, sum, err_ptr); -else -return __csum_partial_copy_from_user ((__force void *)src, dst, - len, sum, err_ptr); -}#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER static inline
@@ -65,9 +49,12 @@ __wsum csum_and_copy_from_user(const void __user *src, void *dst, int len)__wsum sum = ~0U ;int err =0 ; +might_fault (); +if (!access_ok (src, len))return 0 ; - sum =csum_partial_copy_from_user (src, dst, len, sum, &err); + sum =__csum_partial_copy_from_user ((__force void *)src, dst, + len, sum, &err);return err ?0 : sum; }
@@ -84,14 +71,9 @@ __wsum csum_and_copy_to_user(const void *src, void __user *dst, int len)might_fault ();if (!access_ok (dst, len))return 0 ; -if (uaccess_kernel ()) - sum =__csum_partial_copy_kernel (src, - (__force void *)dst, - len, sum, &err); -else - sum =__csum_partial_copy_to_user (src, - (__force void *)dst, - len, sum, &err); + sum =__csum_partial_copy_to_user (src, + (__force void *)dst, + len, sum, &err);return err ?0 : sum; }
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH 13/18 mips: __csum_partial_copy_kernel() has no users left
In reply to Al ViroFrom: Al Viro <[email protected]>
--- arch/mips/include/asm/checksum.h | 3 --- arch/mips/lib/csum_partial.S | 3 --- 2 files changed, 6 deletions(-)
arch/mips/include/asm/checksum.h +0 -3
@@ -34,9 +34,6 @@*/ __wsum csum_partial (const void *buff,int len,__wsum sum); -__wsum __csum_partial_copy_kernel (const void *src,void *dst, -int len,__wsum sum,int *err_ptr); -__wsum __csum_partial_copy_from_user (const void *src,void *dst,int len,__wsum sum,int *err_ptr); __wsum __csum_partial_copy_to_user(const void *src, void *dst,
arch/mips/lib/csum_partial.S +0 -3
@@ -827,8 +827,6 @@ EXPORT_SYMBOL(csum_partial) .set pop .endm -LEAF(__csum_partial_copy_kernel) -EXPORT_SYMBOL(__csum_partial_copy_kernel) #ifndef CONFIG_EVA FEXPORT(__csum_partial_copy_to_user) EXPORT_SYMBOL(__csum_partial_copy_to_user)
@@ -836,7 +834,6 @@ FEXPORT(__csum_partial_copy_from_user) EXPORT_SYMBOL(__csum_partial_copy_from_user) #endif __BUILD_CSUM_PARTIAL_COPY_USER LEGACY_MODE USEROP USEROP 1 -END(__csum_partial_copy_kernel) #ifdef CONFIG_EVA LEAF(__csum_partial_copy_to_user)
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH 14/18 mips: propagate the calling convention change down into __csum_partial_copy_..._user()
In reply to Al ViroFrom: Al Viro <[email protected]>
and turn the exception handlers into simply returning 0, which simplifies the hell out of things in csum_partial.S
--- arch/mips/include/asm/checksum.h | 26 +--- arch/mips/lib/csum_partial.S | 258 +++++++++++++-------------------------- 2 files changed, 89 insertions(+), 195 deletions(-)
arch/mips/include/asm/checksum.h +6 -20
@@ -34,25 +34,17 @@*/ __wsum csum_partial (const void *buff,int len,__wsum sum); -__wsum __csum_partial_copy_from_user (const void *src,void *dst, -int len,__wsum sum,int *err_ptr); -__wsum __csum_partial_copy_to_user (const void *src,void *dst, -int len,__wsum sum,int *err_ptr); +__wsum __csum_partial_copy_from_user (const void __user *src,void *dst,int len); +__wsum __csum_partial_copy_to_user (const void *src,void __user *dst,int len);#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USERstatic inline __wsum csum_and_copy_from_user (const void __user *src,void *dst,int len) { -__wsum sum = ~0U ; -int err =0 ; -might_fault (); -if (!access_ok (src, len))return 0 ; - sum =__csum_partial_copy_from_user ((__force void *)src, dst, - len, sum, &err); -return err ?0 : sum; +return __csum_partial_copy_from_user (src, dst, len); }/*
@@ -62,26 +54,20 @@ __wsum csum_and_copy_from_user(const void __user *src, void *dst, int len)static inline __wsum csum_and_copy_to_user (const void *src,void __user *dst,int len) { -int err =0 ; -__wsum sum = ~0U ; -might_fault ();if (!access_ok (dst, len))return 0 ; - sum =__csum_partial_copy_to_user (src, - (__force void *)dst, - len, sum, &err); -return err ?0 : sum; +return __csum_partial_copy_to_user (src, dst, len); }/* * the same as csum_partial, but copies from user space (but on MIPS * we have just one address space, so this is identical to the above) */ -__wsum __csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum); +__wsum __csum_partial_copy_nocheck (const void *src,void *dst,int len);static inline __wsum csum_partial_copy_nocheck (const void *src,void *dst,int len) { -return __csum_partial_copy_nocheck (src, dst, len,0 ); +return __csum_partial_copy_nocheck (src, dst, len); }#define csum_partial_copy_nocheck csum_partial_copy_nocheck
arch/mips/lib/csum_partial.S +83 -175
418 lines not on this page · raw message
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH 15/18 xtensa: propagate the calling conventions change down into csum_partial_copy_generic()
In reply to Al ViroFrom: Al Viro <[email protected]>
turn the exception handlers into returning 0.
--- arch/xtensa/include/asm/checksum.h | 20 +++--------- arch/xtensa/lib/checksum.S | 67 +++++++++----------------------------- 2 files changed, 19 insertions(+), 68 deletions(-)
arch/xtensa/include/asm/checksum.h +4 -16
@@ -37,9 +37,7 @@ asmlinkage __wsum csum_partial(const void *buff, int len, __wsum sum);* better 64-bit) boundary */ -asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst, -int len,__wsum sum, -int *src_err_ptr,int *dst_err_ptr); +asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst, int len);/* * Note: when you get a NULL pointer exception here this means someone
@@ -48,7 +46,7 @@ asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst,static inline __wsum csum_partial_copy_nocheck (const void *src,void *dst,int len) { -return csum_partial_copy_generic (src, dst, len,0 ,NULL ,NULL ); +return csum_partial_copy_generic (src, dst, len); }#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER
@@ -56,14 +54,9 @@ static inline__wsum csum_and_copy_from_user (const void __user *src,void *dst,int len) { -int err =0 ; -if (!access_ok (dst, len))return 0 ; - - sum =csum_partial_copy_generic ((__force const void *)src, dst, - len, ~0U , &err,NULL ); -return err ?0 : sum; +return csum_partial_copy_generic ((__force const void *)src, dst, len); }/*
@@ -246,13 +239,8 @@ static __inline__ __sum16 csum_ipv6_magic(const struct in6_addr *saddr,static __inline____wsum csum_and_copy_to_user (const void *src,void __user *dst,int len) { -int err =0 ; -__wsum sum = ~0U ; -if (!access_ok (dst, len))return 0 ; - - sum =csum_partial_copy_generic (src,dst,len,sum,NULL ,&err); -return err ?0 : sum; +return csum_partial_copy_generic (src, (__force void *)dst, len); }#endif
arch/xtensa/lib/checksum.S +15 -52
135 lines not on this page · raw message
Signature · 1 line
2.11.0
Re: [PATCH 15/18] xtensa: propagate the calling conventions change down into csum_partial_copy_generic()
Quoted 10 lines · Al Viro
From: Al Viro <[email protected]>
turn the exception handlers into returning 0.
Signed-off-by Al Viro <[email protected]>--- arch/xtensa/include/asm/checksum.h | 20 +++--------- arch/xtensa/lib/checksum.S | 67 +++++++++----------------------------- 2 files changed, 19 insertions(+), 68 deletions(-)
Signature · 2 lines
Thanks. -- Max
Al Viro
maintainerPATCH 16/18 sparc64: propagate the calling convention changes down to __csum_partial_copy_...()
In reply to Al ViroFrom: Al Viro <[email protected]>
... and rename them into csum_and_copy_...() - the wrappers become pointless.
--- arch/sparc/include/asm/checksum.h | 1 + arch/sparc/include/asm/checksum_32.h | 2 -- arch/sparc/include/asm/checksum_64.h | 41 +++--------------------------------- arch/sparc/lib/csum_copy.S | 5 +++-- arch/sparc/lib/csum_copy_from_user.S | 4 ++-- arch/sparc/lib/csum_copy_to_user.S | 4 ++-- 6 files changed, 11 insertions(+), 46 deletions(-)
arch/sparc/include/asm/checksum.h +1 -0
@@ -2,6 +2,7 @@#ifndef ___ASM_SPARC_CHECKSUM_H#define ___ASM_SPARC_CHECKSUM_H#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER +#define HAVE_CSUM_COPY_USER #if defined(__sparc__) && defined(__arch64__)#include <asm/checksum_64.h> #else
arch/sparc/include/asm/checksum_32.h +0 -2
@@ -67,8 +67,6 @@ csum_and_copy_from_user(const void __user *src, void *dst, int len)return csum_partial_copy_nocheck ((__force void *)src, dst, len); } -#define HAVE_CSUM_COPY_USER -static inline __wsum csum_and_copy_to_user (const void *src,void __user *dst,int len) {
arch/sparc/include/asm/checksum_64.h +3 -38
47 lines not on this page · raw message
arch/sparc/lib/csum_copy.S +3 -2
@@ -33,7 +33,7 @@ #endif #ifndef FUNC_NAME -#define FUNC_NAME __csum_partial_copy_nocheck +#define FUNC_NAME csum_partial_copy_nocheck #endif .register %g2, #scratch
@@ -68,9 +68,10 @@ .globl FUNC_NAME .type FUNC_NAME,#function EXPORT_SYMBOL(FUNC_NAME) -FUNC_NAME: /* %o0=src, %o1=dst, %o2=len, %o3=sum */ +FUNC_NAME: /* %o0=src, %o1=dst, %o2=len */ LOAD(prefetch, %o0 + 0x000, #n_reads) xor %o0, %o1, %g1 + movl %o3, -1 clr %o4 andcc %g1, 0x3, %g0 bne,pn %icc, 95f
arch/sparc/lib/csum_copy_from_user.S +2 -2
@@ -9,14 +9,14 @@ .section .fixup, "ax"; \ .align 4; \ 99: retl; \ - mov -1, %o0; \ + mov 0, %o0; \ .section __ex_table,"a";\ .align 4; \ .word 98b, 99b; \ .text; \ .align 4; -#define FUNC_NAME __csum_partial_copy_from_user +#define FUNC_NAME csum_and_copy_from_user #define LOAD(type,addr,dest) type##a [addr] %asi, dest #include "csum_copy.S"
arch/sparc/lib/csum_copy_to_user.S +2 -2
@@ -9,14 +9,14 @@ .section .fixup,"ax"; \ .align 4; \ 99: retl; \ - mov -1, %o0; \ + mov 0, %o0; \ .section __ex_table,"a";\ .align 4; \ .word 98b, 99b; \ .text; \ .align 4; -#define FUNC_NAME __csum_partial_copy_to_user +#define FUNC_NAME csum_and_copy_to_user #define STORE(type,src,addr) type##a src, [addr] %asi #include "csum_copy.S"
Signature · 1 line
2.11.0
Re: [PATCH 16/18] sparc64: propagate the calling convention changes down to __csum_partial_copy_...()
From: Al Viro <[email protected]>
Date: Tue, 21 Jul 2020 21:25:47 +0100
Quoted 5 lines
From: Al Viro <[email protected]>
... and rename them into csum_and_copy_...() - the wrappers become pointless.
Signed-off-by Al Viro <[email protected]>
Al Viro
maintainerPATCH 17/18 amd64: switch csum_partial_copy_generic() to new calling conventions
In reply to Al ViroFrom: Al Viro <[email protected]>
... and fold handling of misaligned case into it.
Implementation note: we stash the "will we need to rol8 the sum in the end" flag into the MSB of %rcx (the lower 32 bits are used for length); the rest is pretty straightforward.
--- arch/x86/include/asm/checksum_64.h | 5 +- arch/x86/lib/csum-copy_64.S | 140 ++++++++++++++++++++++--------------- arch/x86/lib/csum-wrappers_64.c | 72 +++---------------- 3 files changed, 94 insertions(+), 123 deletions(-)
arch/x86/include/asm/checksum_64.h +1 -4
@@ -130,10 +130,7 @@ static inline __sum16 csum_tcpudp_magic(__be32 saddr, __be32 daddr,extern __wsum csum_partial (const void *buff,int len,__wsum sum);/* Do not call this directly. Use the wrappers below */ -extern __visible __wsum csum_partial_copy_generic(const void *src, const void *dst, -int len,__wsum sum, -int *src_err_ptr,int *dst_err_ptr); - +extern __visible __wsum csum_partial_copy_generic(const void *src, void *dst, int len);extern __wsum csum_and_copy_from_user (const void __user *src,void *dst,int len);extern __wsum csum_and_copy_to_user (const void *src,void __user *dst,int len);
arch/x86/lib/csum-copy_64.S +84 -56
192 lines not on this page · raw message
arch/x86/lib/csum-wrappers_64.c +9 -63
102 lines not on this page · raw message
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH 18/18 ppc: propagate the calling conventions change down to csum_partial_copy_generic()
In reply to Al ViroFrom: Al Viro <[email protected]>
... and get rid of the pointless fallback in the wrappers. On error it used to zero the unwritten area and calculate the csum of the entire thing. Not wanting to do it in assembler part had been very reasonable; doing that in the first place, OTOH... In case of an error the caller discards the data we'd copied, along with whatever checksum it might've had.
--- arch/powerpc/include/asm/checksum.h | 6 +-- arch/powerpc/lib/checksum_32.S | 74 +++++++++++++----------------------- arch/powerpc/lib/checksum_64.S | 37 ++++++------------ arch/powerpc/lib/checksum_wrappers.c | 32 +++------------- 4 files changed, 46 insertions(+), 103 deletions(-)
arch/powerpc/include/asm/checksum.h +2 -4
@@ -18,9 +18,7 @@* Like csum_partial, this must be called with even lengths, * except for the last fragment. */ -extern __wsum csum_partial_copy_generic (const void *src,void *dst, -int len,__wsum sum, -int *src_err,int *dst_err); +extern __wsum csum_partial_copy_generic (const void *src,void *dst,int len);#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER extern __wsum csum_and_copy_from_user(const void __user *src, void *dst,
@@ -30,7 +28,7 @@ extern __wsum csum_and_copy_to_user(const void *src, void __user *dst, int len);#define csum_partial_copy_nocheck (src, dst, len) \ - csum_partial_copy_generic((src), (dst), (len), 0, NULL, NULL) + csum_partial_copy_generic((src), (dst), (len))/*
arch/powerpc/lib/checksum_32.S +27 -47
110 lines not on this page · raw message
arch/powerpc/lib/checksum_64.S +11 -26
@@ -182,34 +182,33 @@ EXPORT_SYMBOL(__csum_partial) .macro srcnr 100: - EX_TABLE(100b,.Lsrc_error_nr) + EX_TABLE(100b,.Lerror_nr) .endm .macro source 150: - EX_TABLE(150b,.Lsrc_error) + EX_TABLE(150b,.Lerror) .endm .macro dstnr 200: - EX_TABLE(200b,.Ldest_error_nr) + EX_TABLE(200b,.Lerror_nr) .endm .macro dest 250: - EX_TABLE(250b,.Ldest_error) + EX_TABLE(250b,.Lerror) .endm /* * Computes the checksum of a memory block at src, length len, - * and adds in "sum" (32-bit), while copying the block to dst. - * If an access exception occurs on src or dst, it stores -EFAULT - * to *src_err or *dst_err respectively. The caller must take any action - * required in this case (zeroing memory, recalculating partial checksum etc). + * and adds in 0xffffffff (32-bit), while copying the block to dst. + * If an access exception occurs, it returns 0. * - * csum_partial_copy_generic(r3=src, r4=dst, r5=len, r6=sum, r7=src_err, r8=dst_err) + * csum_partial_copy_generic(r3=src, r4=dst, r5=len) */ _GLOBAL(csum_partial_copy_generic) + li r6,-1 addic r0,r6,0 /* clear carry */ srdi. r6,r5,3 /* less than 8 bytes? */
@@ -401,29 +400,15 @@ dstnr; stb r6,0(r4) srdi r3,r3,32 blr -.Lsrc_error: +.Lerror: ld r14,STK_REG(R14)(r1) ld r15,STK_REG(R15)(r1) ld r16,STK_REG(R16)(r1) addi r1,r1,STACKFRAMESIZE -.Lsrc_error_nr: - cmpdi 0,r7,0 - beqlr - li r6,-EFAULT - stw r6,0(r7) +.Lerror_nr: + li r3,0 blr -.Ldest_error: - ld r14,STK_REG(R14)(r1) - ld r15,STK_REG(R15)(r1) - ld r16,STK_REG(R16)(r1) - addi r1,r1,STACKFRAMESIZE -.Ldest_error_nr: - cmpdi 0,r8,0 - beqlr - li r6,-EFAULT - stw r6,0(r8) - blr EXPORT_SYMBOL(csum_partial_copy_generic) /*
arch/powerpc/lib/checksum_wrappers.c +6 -26
@@ -14,8 +14,7 @@__wsum csum_and_copy_from_user (const void __user *src,void *dst,int len) { -unsigned int csum; -int err =0 ; +__wsum csum;might_sleep ();
@@ -24,27 +23,16 @@ __wsum csum_and_copy_from_user(const void __user *src, void *dst,allow_read_from_user (src, len); - csum =csum_partial_copy_generic ((void __force *)src, dst, - len, ~0U , &err,NULL ); - -if (unlikely (err)) { -int missing =__copy_from_user (dst, src, len); - -if (missing) - csum =0 ; -else - csum =csum_partial (dst, len, ~0U ); - } + csum =csum_partial_copy_generic ((void __force *)src, dst, len);prevent_read_from_user (src, len); -return (__force __wsum )csum; +return csum; }EXPORT_SYMBOL (csum_and_copy_from_user);__wsum csum_and_copy_to_user (const void *src,void __user *dst,int len) { -unsigned int csum; -int err =0 ; +__wsum csum;might_sleep ();if (unlikely (!access_ok (dst, len)))
@@ -52,17 +40,9 @@ __wsum csum_and_copy_to_user(const void *src, void __user *dst, int len)allow_write_to_user (dst, len); - csum =csum_partial_copy_generic (src, (void __force *)dst, - len, ~0U ,NULL , &err); - -if (unlikely (err)) { - csum =csum_partial (src, len, ~0U ); - -if (copy_to_user (dst, src, len)) - csum =0 ; - } + csum =csum_partial_copy_generic (src, (void __force *)dst, len);prevent_write_to_user (dst, len); -return (__force __wsum )csum; +return csum; }EXPORT_SYMBOL (csum_and_copy_to_user);
Signature · 1 line
2.11.0
Al Viro
maintainerRFC PATCH v2 1/1 saner calling conventions for csum-and-copy primitives
In reply to Al ViroUpdated version pushed. Changes since the first variant:
* xtensa fix from Max Filippov (regression from last cycle, will
send to Linus tonight)
* csum_partial_copy_nocheck() got its default variants consolidated
* sparc64 idiotic typo fixed (along with the broken build script
I'd been using)
* commit messages updated (mostly in "saner calling conventions for
csum_and_copy_..._user()")
The branch is still based at 5.8-rc1 and can be found in git.kernel.org/pub/scm/linux/kernel/git/viro/vfs.git work.csum_and_copy Individual patches will go in followups.
Shortlog:
Al Viro (19):
skb_copy_and_csum_bits(): don't bother with the last argument icmp_push_reply(): reorder adding the checksum up
unify generic instances of csum_partial_copy_nocheck() csum_partial_copy_nocheck(): drop the last argument
csum_and_copy_..._user(): pass 0xffffffff instead of 0 as initial sum
saner calling conventions for csum_and_copy_..._user()
alpha: propagate the calling convention changes down to csum_partial_copy.c helpers
arm: propagate the calling convention changes down to csum_partial_copy_from_user()
m68k: get rid of zeroing destination on error in csum_and_copy_from_user()
sh: propage the calling conventions change down to csum_partial_copy_generic()
i386: propagate the calling conventions change down to csum_partial_copy_generic()
sparc32: propagate the calling conventions change down to __csum_partial_copy_sparc_generic()
mips: csum_and_copy_{to,from}_user() are never called under KERNEL_DS
mips: __csum_partial_copy_kernel() has no users left
mips: propagate the calling convention change down into __csum_partial_copy_..._user()
xtensa: propagate the calling conventions change down into csum_partial_copy_generic()
sparc64: propagate the calling convention changes down to __csum_partial_copy_...()
amd64: switch csum_partial_copy_generic() to new calling conventions
ppc: propagate the calling conventions change down to csum_partial_copy_generic()
Max Filippov (1):
xtensa: fix access check in csum_and_copy_from_user
Diffstat:
arch/alpha/include/asm/checksum.h | 5 +- arch/alpha/lib/csum_partial_copy.c | 164 ++++++++----------- arch/arm/include/asm/checksum.h | 17 +- arch/arm/lib/csumpartialcopy.S | 4 +- arch/arm/lib/csumpartialcopygeneric.S | 1 + arch/arm/lib/csumpartialcopyuser.S | 26 +-- arch/c6x/include/asm/checksum.h | 6 + arch/hexagon/include/asm/checksum.h | 11 -- arch/hexagon/lib/checksum.c | 11 -- arch/ia64/include/asm/checksum.h | 3 - arch/ia64/lib/csum_partial_copy.c | 15 -- arch/m68k/include/asm/checksum.h | 7 +- arch/m68k/lib/checksum.c | 88 +++------- arch/mips/include/asm/checksum.h | 68 ++------ arch/mips/lib/csum_partial.S | 261 ++++++++++-------------------- arch/nios2/include/asm/checksum.h | 5 - arch/parisc/include/asm/checksum.h | 28 ---- arch/parisc/lib/checksum.c | 17 -- arch/powerpc/include/asm/checksum.h | 13 +- arch/powerpc/lib/checksum_32.S | 74 ++++----- arch/powerpc/lib/checksum_64.S | 37 ++--- arch/powerpc/lib/checksum_wrappers.c | 74 ++------- arch/s390/include/asm/checksum.h | 7 - arch/sh/include/asm/checksum_32.h | 36 ++--- arch/sh/lib/checksum.S | 119 ++++---------- arch/sparc/include/asm/checksum.h | 2 + arch/sparc/include/asm/checksum_32.h | 70 ++------ arch/sparc/include/asm/checksum_64.h | 39 +---- arch/sparc/lib/checksum_32.S | 202 +++++------------------ arch/sparc/lib/csum_copy.S | 3 +- arch/sparc/lib/csum_copy_from_user.S | 4 +- arch/sparc/lib/csum_copy_to_user.S | 4 +- arch/sparc/mm/fault_32.c | 6 +- arch/x86/include/asm/checksum.h | 1 + arch/x86/include/asm/checksum_32.h | 40 ++--- arch/x86/include/asm/checksum_64.h | 14 +- arch/x86/lib/checksum_32.S | 117 +++++--------- arch/x86/lib/csum-copy_64.S | 140 +++++++++------- arch/x86/lib/csum-wrappers_64.c | 86 ++-------- arch/x86/um/asm/checksum.h | 16 -- arch/x86/um/asm/checksum_32.h | 23 --- arch/xtensa/include/asm/checksum.h | 34 ++-- arch/xtensa/lib/checksum.S | 67 ++------ drivers/net/ethernet/3com/typhoon.c | 3 +- drivers/net/ethernet/sun/sunvnet_common.c | 2 +- include/asm-generic/checksum.h | 14 -- include/linux/skbuff.h | 2 +- include/net/checksum.h | 22 ++- lib/checksum.c | 11 -- lib/iov_iter.c | 21 ++- net/core/skbuff.c | 13 +- net/ipv4/icmp.c | 10 +- net/ipv4/ip_output.c | 6 +- net/ipv4/raw.c | 2 +- net/ipv6/icmp.c | 4 +- net/ipv6/ip6_output.c | 2 +- net/ipv6/raw.c | 2 +- net/sunrpc/socklib.c | 2 +- 58 files changed, 615 insertions(+), 1466 deletions(-)
Al Viro
maintainerPATCH v2 1/20 xtensa: fix access check in csum_and_copy_from_user
From: Max Filippov <[email protected]>
Commit d341659f470b ("xtensa: switch to providing csum_and_copy_from_user()") introduced access check, but incorrectly tested dst instead of src.
Fix access_ok argument in csum_and_copy_from_user.
--- arch/xtensa/include/asm/checksum.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/xtensa/include/asm/checksum.h +1 -1
@@ -57,7 +57,7 @@ static inline__wsum csum_and_copy_from_user (const void __user *src,void *dst,int len,__wsum sum,int *err_ptr) { -if (access_ok (dst, len)) +if (access_ok (src, len))return csum_partial_copy_generic ((__force const void *)src, dst, len, sum, err_ptr,NULL );if (len)
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH v2 2/20 skb_copy_and_csum_bits(): don't bother with the last argument
From: Al Viro <[email protected]>
it's always 0
--- drivers/net/ethernet/sun/sunvnet_common.c | 2 +- include/linux/skbuff.h | 2 +- net/core/skbuff.c | 11 ++++++----- net/ipv4/icmp.c | 2 +- net/ipv4/ip_output.c | 4 ++-- net/ipv6/icmp.c | 4 ++-- net/ipv6/ip6_output.c | 2 +- net/sunrpc/socklib.c | 2 +- 8 files changed, 15 insertions(+), 14 deletions(-)
drivers/net/ethernet/sun/sunvnet_common.c +1 -1
@@ -1168,7 +1168,7 @@ static inline struct sk_buff *vnet_skb_shape(struct sk_buff *skb, int ncookies) *(__sum16 *)(skb->data + offset) =0 ; csum =skb_copy_and_csum_bits (skb, start, nskb->data + start, - skb->len - start,0 ); + skb->len - start);/* add in the header checksums */ if (skb->protocol ==htons (ETH_P_IP )) {
include/linux/skbuff.h +1 -1
@@ -3529,7 +3529,7 @@ int skb_kill_datagram(struct sock *sk, struct sk_buff *skb, unsigned int flags);int skb_copy_bits (const struct sk_buff *skb,int offset,void *to,int len);int skb_store_bits (struct sk_buff *skb,int offset,const void *from,int len);__wsum skb_copy_and_csum_bits (const struct sk_buff *skb,int offset,u8 *to, -int len,__wsum csum); +int len);int skb_splice_bits (struct sk_buff *skb,struct sock *sk,unsigned int offset,struct pipe_inode_info *pipe,unsigned int len,unsigned int flags);
net/core/skbuff.c +6 -5
@@ -2723,19 +2723,20 @@ EXPORT_SYMBOL(skb_checksum);/* Both of above in one bottle. */ __wsum skb_copy_and_csum_bits (const struct sk_buff *skb,int offset, -u8 *to,int len,__wsum csum) +u8 *to,int len) {int start =skb_headlen (skb);int i, copy = start - offset;struct sk_buff *frag_iter;int pos =0 ; +__wsum csum =0 ;/* Copy header. */ if (copy >0 ) {if (copy > len) copy = len; csum =csum_partial_copy_nocheck (skb->data + offset, to, - copy, csum); + copy,0 );if ((len -= copy) ==0 )return csum; offset += copy;
@@ -2791,7 +2792,7 @@ __wsum skb_copy_and_csum_bits(const struct sk_buff *skb, int offset, copy = len; csum2 =skb_copy_and_csum_bits (frag_iter, offset - start, - to, copy,0 ); + to, copy); csum =csum_block_add (csum, csum2, pos);if ((len -= copy) ==0 )return csum;
@@ -3011,7 +3012,7 @@ void skb_copy_and_csum_dev(const struct sk_buff *skb, u8 *to) csum =0 ;if (csstart != skb->len) csum =skb_copy_and_csum_bits (skb, csstart, to + csstart, - skb->len - csstart,0 ); + skb->len - csstart);if (skb->ip_summed ==CHECKSUM_PARTIAL ) {long csstuff = csstart + skb->csum_offset;
@@ -3933,7 +3934,7 @@ struct sk_buff *skb_segment(struct sk_buff *head_skb,skb_copy_and_csum_bits (head_skb, offset,skb_put (nskb, len), - len,0 ); + len);SKB_GSO_CB (nskb)->csum_start =skb_headroom (nskb) + doffset; }else {
net/ipv4/icmp.c +1 -1
@@ -352,7 +352,7 @@ static int icmp_glue_bits(void *from, char *to, int offset, int len, int odd, csum =skb_copy_and_csum_bits (icmp_param->skb, icmp_param->offset + offset, - to, len,0 ); + to, len); skb->csum =csum_block_add (skb->csum, csum, odd);if (icmp_pointers[icmp_param->data.icmph.type].error)
net/ipv4/ip_output.c +2 -2
@@ -1120,7 +1120,7 @@ static int __ip_append_data(struct sock *sk,if (fraggap) { skb->csum =skb_copy_and_csum_bits ( skb_prev, maxfraglen, - data + transhdrlen, fraggap,0 ); + data + transhdrlen, fraggap); skb_prev->csum =csum_sub (skb_prev->csum, skb->csum); data += fraggap;
@@ -1405,7 +1405,7 @@ ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page, skb->csum =skb_copy_and_csum_bits (skb_prev, maxfraglen,skb_transport_header (skb), - fraggap,0 ); + fraggap); skb_prev->csum =csum_sub (skb_prev->csum, skb->csum);pskb_trim_unique (skb_prev, maxfraglen);
net/ipv6/icmp.c +2 -2
@@ -314,10 +314,10 @@ static int icmpv6_getfrag(void *from, char *to, int offset, int len, int odd, st {struct icmpv6_msg *msg = (struct icmpv6_msg *) from;struct sk_buff *org_skb = msg->skb; -__wsum csum =0 ; +__wsum csum; csum =skb_copy_and_csum_bits (org_skb, msg->offset + offset, - to, len, csum); + to, len); skb->csum =csum_block_add (skb->csum, csum, odd);if (!(msg->type &ICMPV6_INFOMSG_MASK ))nf_ct_attach (skb, org_skb);
net/ipv6/ip6_output.c +1 -1
@@ -1613,7 +1613,7 @@ static int __ip6_append_data(struct sock *sk,if (fraggap) { skb->csum =skb_copy_and_csum_bits ( skb_prev, maxfraglen, - data + transhdrlen, fraggap,0 ); + data + transhdrlen, fraggap); skb_prev->csum =csum_sub (skb_prev->csum, skb->csum); data += fraggap;
net/sunrpc/socklib.c +1 -1
@@ -70,7 +70,7 @@ static size_t xdr_skb_read_and_csum_bits(struct xdr_skb_reader *desc, void *to,if (len > desc->count) len = desc->count; pos = desc->offset; - csum2 =skb_copy_and_csum_bits (desc->skb, pos, to, len,0 ); + csum2 =skb_copy_and_csum_bits (desc->skb, pos, to, len); desc->csum =csum_block_add (desc->csum, csum2, pos); desc->count -= len; desc->offset += len;
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH v2 3/20 icmp_push_reply(): reorder adding the checksum up
In reply to Al ViroFrom: Al Viro <[email protected]>
do csum_partial_copy_nocheck() on the first fragment, then add the rest to it. Equivalent transformation.
That was the only caller of csum_partial_copy_nocheck() that might pass it non-zero as the last argument.
--- net/ipv4/icmp.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
net/ipv4/icmp.c +4 -4
@@ -376,15 +376,15 @@ static void icmp_push_reply(struct icmp_bxm *icmp_param,ip_flush_pending_frames (sk); }else if ((skb =skb_peek (&sk->sk_write_queue)) !=NULL ) {struct icmphdr *icmph =icmp_hdr (skb); -__wsum csum =0 ; +__wsum csum;struct sk_buff *skb1; + csum =csum_partial_copy_nocheck ((void *)&icmp_param->data, + (char *)icmph, + icmp_param->head_len,0 ); skb_queue_walk(&sk->sk_write_queue, skb1) { csum =csum_add (csum, skb1->csum); } - csum =csum_partial_copy_nocheck ((void *)&icmp_param->data, - (char *)icmph, - icmp_param->head_len, csum); icmph->checksum =csum_fold (csum); skb->ip_summed =CHECKSUM_NONE ;ip_push_pending_frames (sk, fl4);
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH v2 4/20 unify generic instances of csum_partial_copy_nocheck()
In reply to Al ViroFrom: Al Viro <[email protected]>
quite a few architectures have the same csum_partial_copy_nocheck() - simply memcpy() the data and then return the csum of the copy.
hexagon, parisc, ia64, s390, um: explicitly spelled out that way.
arc, arm64, csky, h8300, m68k/nommu, microblaze, mips/GENERIC_CSUM, nds32, nios2, openrisc, riscv, unicore32: end up picking the same thing spelled out in lib/checksum.h (with varying amounts of perversions along the way).
everybody else (alpha, arm, c6x, m68k/mmu, mips/!GENERIC_CSUM, powerpc, sh, sparc, x86, xtensa) have non-generic variants. For all except c6x the declaration is in their asm/checksum.h. c6x uses the wrapper from asm-generic/checksum.h that would normally lead to the lib/checksum.h instance, but in case of c6x we end up using an asm function from arch/c6x instead.
Screw that mess - have architectures with private instances define _HAVE_ARCH_CSUM_AND_COPY in their asm/checksum.h and have the default one right in net/checksum.h conditional on _HAVE_ARCH_CSUM_AND_COPY *not* defined.
--- arch/alpha/include/asm/checksum.h | 1 + arch/arm/include/asm/checksum.h | 1 + arch/c6x/include/asm/checksum.h | 6 ++++++ arch/hexagon/include/asm/checksum.h | 11 ----------- arch/hexagon/lib/checksum.c | 11 ----------- arch/ia64/include/asm/checksum.h | 3 --- arch/ia64/lib/csum_partial_copy.c | 15 --------------- arch/m68k/include/asm/checksum.h | 1 + arch/mips/include/asm/checksum.h | 2 +- arch/nios2/include/asm/checksum.h | 5 ----- arch/parisc/include/asm/checksum.h | 8 -------- arch/parisc/lib/checksum.c | 17 ----------------- arch/powerpc/include/asm/checksum.h | 1 + arch/s390/include/asm/checksum.h | 7 ------- arch/sh/include/asm/checksum_32.h | 1 + arch/sparc/include/asm/checksum.h | 1 + arch/x86/include/asm/checksum.h | 1 + arch/x86/um/asm/checksum.h | 16 ---------------- arch/xtensa/include/asm/checksum.h | 1 + include/asm-generic/checksum.h | 14 -------------- include/net/checksum.h | 9 +++++++++ lib/checksum.c | 11 ----------- 22 files changed, 24 insertions(+), 119 deletions(-)
arch/alpha/include/asm/checksum.h +1 -0
@@ -42,6 +42,7 @@ extern __wsum csum_partial(const void *buff, int len, __wsum sum);* better 64-bit) boundary */ #define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER +#define _HAVE_ARCH_CSUM_AND_COPY__wsum csum_and_copy_from_user (const void __user *src,void *dst,int len,__wsum sum,int *errp);__wsum csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum);
arch/arm/include/asm/checksum.h +1 -0
@@ -41,6 +41,7 @@ __wsum csum_partial_copy_from_user(const void __user *src, void *dst, int len, __wsum sum, int *err_ptr);#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER +#define _HAVE_ARCH_CSUM_AND_COPYstatic inline __wsum csum_and_copy_from_user (const void __user *src,void *dst,int len,__wsum sum,int *err_ptr)
arch/c6x/include/asm/checksum.h +6 -0
@@ -26,6 +26,12 @@ csum_tcpudp_nofold(__be32 saddr, __be32 daddr, __u32 len, }#define csum_tcpudp_nofold csum_tcpudp_nofold +extern __wsum csum_partial_copy (const void *src,void *dst,int len,__wsum sum); + +#define _HAVE_ARCH_CSUM_AND_COPY +#define csum_partial_copy_nocheck (src, dst, len, sum) \ + csum_partial_copy((src), (dst), (len), (sum)) +#include <asm-generic/checksum.h> #endif /* _ASM_C6X_CHECKSUM_H */
arch/hexagon/include/asm/checksum.h +0 -11
@@ -10,17 +10,6 @@unsigned int do_csum (const void *voidptr,int len);/* -* the same as csum_partial, but copies from src while it -* checksums -* -* here even more important to align src and dst on a 32-bit (or even -* better 64-bit) boundary -*/ -#define csum_partial_copy_nocheck csum_partial_copy_nocheck -__wsum csum_partial_copy_nocheck (const void *src,void *dst, -int len,__wsum sum); - -/* * computes the checksum of the TCP/UDP pseudo-header * returns a 16-bit checksum, already complemented */
arch/hexagon/lib/checksum.c +0 -11
@@ -176,14 +176,3 @@ unsigned int do_csum(const void *voidptr, int len)return 0xFFFF & sum0; } - -/* -* copy from ds while checksumming, otherwise like csum_partial -*/ -__wsum -csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum) -{ -memcpy (dst, src, len); -return csum_partial (dst, len, sum); -} -EXPORT_SYMBOL (csum_partial_copy_nocheck);
arch/ia64/include/asm/checksum.h +0 -3
@@ -37,9 +37,6 @@ extern __wsum csum_tcpudp_nofold(__be32 saddr, __be32 daddr,*/ extern __wsum csum_partial (const void *buff,int len,__wsum sum); -extern __wsum csum_partial_copy_nocheck (const void *src,void *dst, -int len,__wsum sum); -/* * This routine is used for miscellaneous IP-like checksums, mainly in * icmp.c
arch/ia64/lib/csum_partial_copy.c +0 -15
@@ -96,18 +96,3 @@ unsigned long do_csum_c(const unsigned char * buff, int len, unsigned int psum) out:return result; } - -/* -* XXX Fixme -* -* This is very ugly but temporary. THIS NEEDS SERIOUS ENHANCEMENTS. -* But it's very tricky to get right even in C. -*/ -__wsum -csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum) -{ -memcpy (dst, src, len); -return csum_partial (dst, len, sum); -} - -EXPORT_SYMBOL (csum_partial_copy_nocheck);
arch/m68k/include/asm/checksum.h +1 -0
@@ -31,6 +31,7 @@ __wsum csum_partial(const void *buff, int len, __wsum sum);*/ #define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER +#define _HAVE_ARCH_CSUM_AND_COPY extern __wsum csum_and_copy_from_user(const void __user *src, void *dst, int len, __wsum sum,
arch/mips/include/asm/checksum.h +1 -1
@@ -101,9 +101,9 @@ __wsum csum_and_copy_to_user(const void *src, void __user *dst, int len,* the same as csum_partial, but copies from user space (but on MIPS * we have just one address space, so this is identical to the above) */ +#define _HAVE_ARCH_CSUM_AND_COPY__wsum csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum); -#define csum_partial_copy_nocheck csum_partial_copy_nocheck/* * Fold a partial checksum without adding pseudo headers
arch/nios2/include/asm/checksum.h +0 -5
@@ -12,11 +12,6 @@/* Take these from lib/checksum.c */ extern __wsum csum_partial (const void *buff,int len,__wsum sum); -extern __wsum csum_partial_copy (const void *src,void *dst,int len, -__wsum sum); -#define csum_partial_copy_nocheck (src, dst, len, sum) \ - csum_partial_copy((src), (dst), (len), (sum)) -extern __sum16 ip_fast_csum (const void *iph,unsigned int ihl);extern __sum16 ip_compute_csum (const void *buff,int len);
arch/parisc/include/asm/checksum.h +0 -8
@@ -19,14 +19,6 @@extern __wsum csum_partial (const void *,int ,__wsum );/* -* The same as csum_partial, but copies from src while it checksums. -* -* Here even more important to align src and dst on a 32-bit (or even -* better 64-bit) boundary -*/ -extern __wsum csum_partial_copy_nocheck (const void *,void *,int ,__wsum ); - -/* * Optimized for IP headers, which always checksum on 4 octet boundaries. * * Written by Randolph Chung <[email protected]>, and then mucked with by
arch/parisc/lib/checksum.c +0 -17
@@ -106,20 +106,3 @@ __wsum csum_partial(const void *buff, int len, __wsum sum) }EXPORT_SYMBOL (csum_partial); - -/* -* copy while checksumming, otherwise like csum_partial -*/ -__wsum csum_partial_copy_nocheck (const void *src,void *dst, -int len,__wsum sum) -{ -/* -* It's 2:30 am and I don't feel like doing it real ... -* This is lots slower than the real thing (tm) -*/ - sum =csum_partial (src, len, sum); -memcpy (dst, src, len); - -return sum; -} -EXPORT_SYMBOL (csum_partial_copy_nocheck);
arch/powerpc/include/asm/checksum.h +1 -0
@@ -29,6 +29,7 @@ extern __wsum csum_and_copy_from_user(const void __user *src, void *dst,extern __wsum csum_and_copy_to_user (const void *src,void __user *dst,int len,__wsum sum,int *err_ptr); +#define _HAVE_ARCH_CSUM_AND_COPY#define csum_partial_copy_nocheck (src, dst, len, sum) \ csum_partial_copy_generic((src), (dst), (len), (sum), NULL, NULL)
arch/s390/include/asm/checksum.h +0 -7
@@ -39,13 +39,6 @@ csum_partial(const void *buff, int len, __wsum sum)return sum; } -static inline __wsum -csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum) -{ -memcpy (dst,src,len); -return csum_partial (dst, len, sum); -} -/* * Fold a partial checksum without adding pseudo headers */
arch/sh/include/asm/checksum_32.h +1 -0
@@ -34,6 +34,7 @@ asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst, int len, __wsum sum, int *src_err_ptr, int *dst_err_ptr); +#define _HAVE_ARCH_CSUM_AND_COPY/* * Note: when you get a NULL pointer exception here this means someone * passed in an incorrect kernel address to one of these functions.
arch/sparc/include/asm/checksum.h +1 -0
@@ -1,6 +1,7 @@/* SPDX-License-Identifier: GPL-2.0 */ #ifndef ___ASM_SPARC_CHECKSUM_H#define ___ASM_SPARC_CHECKSUM_H +#define _HAVE_ARCH_CSUM_AND_COPY#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER#if defined(__sparc__) && defined(__arch64__)#include <asm/checksum_64.h>
arch/x86/include/asm/checksum.h +1 -0
@@ -1,6 +1,7 @@/* SPDX-License-Identifier: GPL-2.0 */ #define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER 1#define HAVE_CSUM_COPY_USER +#define _HAVE_ARCH_CSUM_AND_COPY#ifdef CONFIG_X86_32 # include <asm/checksum_32.h> #else
arch/x86/um/asm/checksum.h +0 -16
@@ -20,22 +20,6 @@*/ extern __wsum csum_partial (const void *buff,int len,__wsum sum); -/* -* Note: when you get a NULL pointer exception here this means someone -* passed in an incorrect kernel address to one of these functions. -* -* If you use these functions directly please don't forget the -* access_ok(). -*/ - -static __inline__ -__wsum csum_partial_copy_nocheck (const void *src,void *dst, -int len,__wsum sum) -{ -memcpy (dst, src, len); -return csum_partial (dst, len, sum); -} -/** * csum_fold - Fold and invert a 32bit checksum. * sum: 32bit unfolded sum
arch/xtensa/include/asm/checksum.h +1 -0
@@ -41,6 +41,7 @@ asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst, int len, __wsum sum, int *src_err_ptr, int *dst_err_ptr); +#define _HAVE_ARCH_CSUM_AND_COPY/* * Note: when you get a NULL pointer exception here this means someone * passed in an incorrect kernel address to one of these functions.
include/asm-generic/checksum.h +0 -14
@@ -16,20 +16,6 @@*/ extern __wsum csum_partial (const void *buff,int len,__wsum sum); -/* -* the same as csum_partial, but copies from src while it -* checksums -* -* here even more important to align src and dst on a 32-bit (or even -* better 64-bit) boundary -*/ -extern __wsum csum_partial_copy (const void *src,void *dst,int len,__wsum sum); - -#ifndef csum_partial_copy_nocheck -#define csum_partial_copy_nocheck (src, dst, len, sum) \ - csum_partial_copy((src), (dst), (len), (sum)) -#endif -#ifndef ip_fast_csum/* * This is a version of ip_compute_csum() optimized for IP headers,
include/net/checksum.h +9 -0
@@ -47,6 +47,15 @@ static __inline__ __wsum csum_and_copy_to_user }#endif +#ifndef _HAVE_ARCH_CSUM_AND_COPY +static inline __wsum +csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum) +{ +memcpy (dst, src, len); +return csum_partial (dst, len, sum); +} +#endif +#ifndef HAVE_ARCH_CSUM_ADD static inline __wsum csum_add (__wsum csum,__wsum addend) {
lib/checksum.c +0 -11
@@ -145,17 +145,6 @@ __sum16 ip_compute_csum(const void *buff, int len) }EXPORT_SYMBOL (ip_compute_csum); -/* -* copy from ds while checksumming, otherwise like csum_partial -*/ -__wsum -csum_partial_copy (const void *src,void *dst,int len,__wsum sum) -{ -memcpy (dst, src, len); -return csum_partial (dst, len, sum); -} -EXPORT_SYMBOL (csum_partial_copy); -#ifndef csum_tcpudp_nofoldstatic inline u32 from64to32 (u64 x) {
Signature · 1 line
2.11.0
Re: [PATCH v2 04/20] unify generic instances of csum_partial_copy_nocheck()
Quoted 18 lines · Al Viro
From: Al Viro <[email protected]>
quite a few architectures have the same
csum_partial_copy_nocheck()- simplymemcpy()the data and then return the csum of the copy.hexagon, parisc, ia64, s390, um: explicitly spelled out that way.
arc, arm64, csky, h8300, m68k/nommu, microblaze, mips/GENERIC_CSUM, nds32, nios2, openrisc, riscv, unicore32: end up picking the same thing spelled out in
lib/checksum.h(with varying amounts of perversions along the way).everybody else (alpha, arm, c6x, m68k/mmu, mips/!GENERIC_CSUM, powerpc, sh, sparc, x86, xtensa) have non-generic variants. For all except c6x the declaration is in their
asm/checksum.h. c6x uses the wrapper fromasm-generic/checksum.hthat would normally lead to thelib/checksum.hinstance, but in case of c6x we end up using an asm function fromarch/c6xinstead.
Quoted 4 lines
Screw that mess - have architectures with private instances define
_HAVE_ARCH_CSUM_AND_COPYin theirasm/checksum.hand have the default one right innet/checksum.hconditional on_HAVE_ARCH_CSUM_AND_COPY*not* defined.
net-next has a patch from me killing off csum_and_copy_from_user already:
Al Viro
maintainerRe: [PATCH v2 04/20] unify generic instances of csum_partial_copy_nocheck()
Quoted 24 lines · Christoph Hellwig
Quoted 18 lines · Al Viro
From: Al Viro <[email protected]>
quite a few architectures have the same
csum_partial_copy_nocheck()- simplymemcpy()the data and then return the csum of the copy.hexagon, parisc, ia64, s390, um: explicitly spelled out that way.
arc, arm64, csky, h8300, m68k/nommu, microblaze, mips/GENERIC_CSUM, nds32, nios2, openrisc, riscv, unicore32: end up picking the same thing spelled out in
lib/checksum.h(with varying amounts of perversions along the way).everybody else (alpha, arm, c6x, m68k/mmu, mips/!GENERIC_CSUM, powerpc, sh, sparc, x86, xtensa) have non-generic variants. For all except c6x the declaration is in their
asm/checksum.h. c6x uses the wrapper fromasm-generic/checksum.hthat would normally lead to thelib/checksum.hinstance, but in case of c6x we end up using an asm function fromarch/c6xinstead.Quoted 4 lines
Screw that mess - have architectures with private instances define
_HAVE_ARCH_CSUM_AND_COPYin theirasm/checksum.hand have the default one right innet/checksum.hconditional on_HAVE_ARCH_CSUM_AND_COPY*not* defined.
Quoted 4 lines
net-next has a patch from me killing off
csum_and_copy_from_useralready:
Nothing in that patch of yours touches csum_and_copy_from_user(). what are you talking about?
Re: [PATCH v2 04/20] unify generic instances of csum_partial_copy_nocheck()
Quoted 7 lines · Al Viro
Quoted 4 lines
net-next has a patch from me killing off
csum_and_copy_from_useralready:Nothing in that patch of yours touches
csum_and_copy_from_user(). what are you talking about?
Sorry, I meant csum_and_copy_from_nocheck, just as in this patch.
Merging your branch into the net-next tree thus will conflict in the nios2 and asm-geneeric/checksum.h as well as lib/checksum.c.
Al Viro
maintainerRe: [PATCH v2 04/20] unify generic instances of csum_partial_copy_nocheck()
Quoted 6 lines · Christoph Hellwig
Quoted 5 lines · Al Viro
Quoted 4 lines
net-next has a patch from me killing off
csum_and_copy_from_useralready:
Quoted 7 lines
Quoted 2 lines
Nothing in that patch of yours touches
csum_and_copy_from_user(). what are you talking about?Sorry, I meant
csum_and_copy_from_nocheck, just as in this patch.Merging your branch into the net-next tree thus will conflict in the nios2 and
asm-geneeric/checksum.has well aslib/checksum.c.
Noted, but that asm-generic/checksum.h conflict will be "massage in net-next/outright removal in this branch"; the same goes for lib/checksum.c and nios2. It's c6x that is unpleasant in that respect...
Re: [PATCH v2 04/20] unify generic instances of csum_partial_copy_nocheck()
Quoted 8 lines · Al Viro
Quoted 4 lines
Sorry, I meant
csum_and_copy_from_nocheck, just as in this patch.Merging your branch into the net-next tree thus will conflict in the nios2 and
asm-geneeric/checksum.has well aslib/checksum.c.Noted, but that
asm-generic/checksum.hconflict will be "massage in net-next/outright removal in this branch"; the same goes forlib/checksum.cand nios2. It's c6x that is unpleasant in that respect...
What about just rebasing your branch on the net-next tree?
Al Viro
maintainerRe: [PATCH v2 04/20] unify generic instances of csum_partial_copy_nocheck()
Quoted 11 lines · Christoph Hellwig
Quoted 8 lines · Al Viro
Quoted 4 lines
Sorry, I meant
csum_and_copy_from_nocheck, just as in this patch.Merging your branch into the net-next tree thus will conflict in the nios2 and
asm-geneeric/checksum.has well aslib/checksum.c.Noted, but that
asm-generic/checksum.hconflict will be "massage in net-next/outright removal in this branch"; the same goes forlib/checksum.cand nios2. It's c6x that is unpleasant in that respect...What about just rebasing your branch on the net-next tree?
For now I've just cherry-picked your commit in there. net-next interaction there is minimal; most of the PITA (and potential breakage) is in arch/*...
Al Viro
maintainerPATCH v2 5/20 csum_partial_copy_nocheck(): drop the last argument
In reply to Al ViroFrom: Al Viro <[email protected]>
It's always 0. Note that we could use ~0U as well - result will be the same modulo 0xffff; later we'll make use of that whenever convenient.
--- arch/alpha/include/asm/checksum.h | 2 +- arch/alpha/lib/csum_partial_copy.c | 4 ++-- arch/arm/include/asm/checksum.h | 2 +- arch/arm/lib/csumpartialcopy.S | 5 +++-- arch/m68k/include/asm/checksum.h | 3 +-- arch/m68k/lib/checksum.c | 3 ++- arch/mips/include/asm/checksum.h | 7 +++++-- arch/mips/lib/csum_partial.S | 4 ++-- arch/powerpc/include/asm/checksum.h | 4 ++-- arch/sh/include/asm/checksum_32.h | 5 ++--- arch/sparc/include/asm/checksum_32.h | 4 ++-- arch/sparc/include/asm/checksum_64.h | 8 ++++++-- arch/sparc/lib/csum_copy.S | 2 +- arch/x86/include/asm/checksum_32.h | 5 ++--- arch/x86/include/asm/checksum_64.h | 3 +-- arch/x86/lib/csum-wrappers_64.c | 4 ++-- arch/xtensa/include/asm/checksum.h | 5 ++--- drivers/net/ethernet/3com/typhoon.c | 3 +-- include/net/checksum.h | 4 ++-- lib/iov_iter.c | 2 +- net/core/skbuff.c | 4 ++-- net/ipv4/icmp.c | 2 +- net/ipv4/ip_output.c | 2 +- net/ipv4/raw.c | 2 +- net/ipv6/raw.c | 2 +- 25 files changed, 47 insertions(+), 44 deletions(-)
arch/alpha/include/asm/checksum.h +1 -1
@@ -45,7 +45,7 @@ extern __wsum csum_partial(const void *buff, int len, __wsum sum);#define _HAVE_ARCH_CSUM_AND_COPY__wsum csum_and_copy_from_user (const void __user *src,void *dst,int len,__wsum sum,int *errp); -__wsum csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum); +__wsum csum_partial_copy_nocheck (const void *src,void *dst,int len);/*
arch/alpha/lib/csum_partial_copy.c +2 -2
@@ -372,13 +372,13 @@ csum_and_copy_from_user(const void __user *src, void *dst, int len,EXPORT_SYMBOL (csum_and_copy_from_user);__wsum -csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum) +csum_partial_copy_nocheck (const void *src,void *dst,int len) {__wsum checksum;mm_segment_t oldfs =get_fs ();set_fs (KERNEL_DS ); checksum =csum_and_copy_from_user ((__force const void __user *)src, - dst, len, sum,NULL ); + dst, len,0 ,NULL );set_fs (oldfs);return checksum; }
arch/arm/include/asm/checksum.h +1 -1
@@ -35,7 +35,7 @@ __wsum csum_partial(const void *buff, int len, __wsum sum);*/ __wsum -csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum); +csum_partial_copy_nocheck (const void *src,void *dst,int len);__wsum csum_partial_copy_from_user (const void __user *src,void *dst,int len,__wsum sum,int *err_ptr);
arch/arm/lib/csumpartialcopy.S +3 -2
@@ -9,13 +9,14 @@ .text -/* Function: __u32 csum_partial_copy_nocheck(const char *src, char *dst, int len, __u32 sum) - * Params : r0 = src, r1 = dst, r2 = len, r3 = checksum +/* Function: __u32 csum_partial_copy_nocheck(const char *src, char *dst, int len) + * Params : r0 = src, r1 = dst, r2 = len * Returns : r0 = new checksum */ .macro save_regs stmfd sp!, {r1, r4 - r8, lr} + mov r3, #0 .endm .macro load_regs
arch/m68k/include/asm/checksum.h +1 -2
@@ -38,8 +38,7 @@ extern __wsum csum_and_copy_from_user(const void __user *src, int *csum_err);extern __wsum csum_partial_copy_nocheck (const void *src, -void *dst,int len, -__wsum sum); +void *dst,int len);/* * This is a version of ip_fast_csum() optimized for IP headers,
arch/m68k/lib/checksum.c +2 -1
@@ -324,9 +324,10 @@ EXPORT_SYMBOL(csum_and_copy_from_user);*/ __wsum -csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum) +csum_partial_copy_nocheck (const void *src,void *dst,int len) {unsigned long tmp1, tmp2; +__wsum sum =0 ; __asm__("movel %2,%4\n\t" "btst #1,%4\n\t" /* Check alignment */ "jeq 2f\n\t"
arch/mips/include/asm/checksum.h +5 -2
@@ -102,8 +102,11 @@ __wsum csum_and_copy_to_user(const void *src, void __user *dst, int len,* we have just one address space, so this is identical to the above) */ #define _HAVE_ARCH_CSUM_AND_COPY -__wsum csum_partial_copy_nocheck (const void *src,void *dst, -int len,__wsum sum); +__wsum __csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum); +static inline __wsum csum_partial_copy_nocheck (const void *src,void *dst,int len) +{ +return __csum_partial_copy_nocheck (src, dst, len,0 ); +}/* * Fold a partial checksum without adding pseudo headers
arch/mips/lib/csum_partial.S +2 -2
@@ -462,8 +462,8 @@ EXPORT_SYMBOL(csum_partial) lw errptr, 16(sp) #endif .if \__nocheck == 1 - FEXPORT(csum_partial_copy_nocheck) - EXPORT_SYMBOL(csum_partial_copy_nocheck) + FEXPORT(__csum_partial_copy_nocheck) + EXPORT_SYMBOL(__csum_partial_copy_nocheck) .endif move sum, zero move odd, zero
arch/powerpc/include/asm/checksum.h +2 -2
@@ -30,8 +30,8 @@ extern __wsum csum_and_copy_to_user(const void *src, void __user *dst, int len, __wsum sum, int *err_ptr);#define _HAVE_ARCH_CSUM_AND_COPY -#define csum_partial_copy_nocheck (src, dst, len, sum) \ - csum_partial_copy_generic((src), (dst), (len), (sum), NULL, NULL) +#define csum_partial_copy_nocheck (src, dst, len) \ + csum_partial_copy_generic((src), (dst), (len), 0, NULL, NULL)/*
arch/sh/include/asm/checksum_32.h +2 -3
@@ -43,10 +43,9 @@ asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst,* access_ok(). */ static inline -__wsum csum_partial_copy_nocheck (const void *src,void *dst, -int len,__wsum sum) +__wsum csum_partial_copy_nocheck (const void *src,void *dst,int len) { -return csum_partial_copy_generic (src, dst, len, sum,NULL ,NULL ); +return csum_partial_copy_generic (src, dst, len,0 ,NULL ,NULL ); }#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER
arch/sparc/include/asm/checksum_32.h +2 -2
@@ -42,7 +42,7 @@ __wsum csum_partial(const void *buff, int len, __wsum sum);unsigned int __csum_partial_copy_sparc_generic (const unsigned char *,unsigned char *);static inline __wsum -csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum) +csum_partial_copy_nocheck (const void *src,void *dst,int len) { register unsigned int ret asm("o0") = (unsigned int)src; register char *d asm("o1") = dst;
@@ -52,7 +52,7 @@ csum_partial_copy_nocheck(const void *src, void *dst, int len, __wsum sum)"call __csum_partial_copy_sparc_generic\n\t" " mov %6, %%g7\n" : "=&r" (ret), "=&r" (d), "=&r" (l) - : "0" (ret), "1" (d), "2" (l), "r" (sum) + : "0" (ret), "1" (d), "2" (l), "r" (0) : "o2", "o3", "o4", "o5", "o7"," g2", " g3", " g4", " g5", " g7", "memory", "cc");
arch/sparc/include/asm/checksum_64.h +6 -2
@@ -38,8 +38,12 @@ __wsum csum_partial(const void * buff, int len, __wsum sum);* here even more important to align src and dst on a 32-bit (or even * better 64-bit) boundary */ -__wsum csum_partial_copy_nocheck (const void *src,void *dst, -int len,__wsum sum); +__wsum __csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum); + +static inline __wsum csum_partial_copy_nocheck (const void *src,void *dst,int len) +{ +return __csum_partial_copy_nocheck (src, dst, len,0 ); +} long __csum_partial_copy_from_user(const void __user *src, void *dst, int len,
arch/sparc/lib/csum_copy.S +1 -1
@@ -33,7 +33,7 @@ #endif #ifndef FUNC_NAME -#define FUNC_NAME csum_partial_copy_nocheck +#define FUNC_NAME __csum_partial_copy_nocheck #endif .register %g2, #scratch
arch/x86/include/asm/checksum_32.h +2 -3
@@ -38,10 +38,9 @@ asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst,* If you use these functions directly please don't forget the * access_ok(). */ -static inline __wsum csum_partial_copy_nocheck (const void *src,void *dst, -int len,__wsum sum) +static inline __wsum csum_partial_copy_nocheck (const void *src,void *dst,int len) { -return csum_partial_copy_generic (src, dst, len, sum,NULL ,NULL ); +return csum_partial_copy_generic (src, dst, len,0 ,NULL ,NULL ); } static inline __wsum csum_and_copy_from_user(const void __user *src,
arch/x86/include/asm/checksum_64.h +1 -2
@@ -139,8 +139,7 @@ extern __wsum csum_and_copy_from_user(const void __user *src, void *dst, int len, __wsum isum, int *errp);extern __wsum csum_and_copy_to_user (const void *src,void __user *dst,int len,__wsum isum,int *errp); -extern __wsum csum_partial_copy_nocheck (const void *src,void *dst, -int len,__wsum sum); +extern __wsum csum_partial_copy_nocheck (const void *src,void *dst,int len);/** * ip_compute_csum - Compute an 16bit IP checksum.
arch/x86/lib/csum-wrappers_64.c +2 -2
@@ -129,9 +129,9 @@ EXPORT_SYMBOL(csum_and_copy_to_user);* Returns an 32bit unfolded checksum of the buffer. */ __wsum -csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum) +csum_partial_copy_nocheck (const void *src,void *dst,int len) { -return csum_partial_copy_generic (src, dst, len, sum,NULL ,NULL ); +return csum_partial_copy_generic (src, dst, len,0 ,NULL ,NULL ); }EXPORT_SYMBOL (csum_partial_copy_nocheck);
arch/xtensa/include/asm/checksum.h +2 -3
@@ -47,10 +47,9 @@ asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst,* passed in an incorrect kernel address to one of these functions. */ static inline -__wsum csum_partial_copy_nocheck (const void *src,void *dst, -int len,__wsum sum) +__wsum csum_partial_copy_nocheck (const void *src,void *dst,int len) { -return csum_partial_copy_generic (src, dst, len, sum,NULL ,NULL ); +return csum_partial_copy_generic (src, dst, len,0 ,NULL ,NULL ); }#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER
drivers/net/ethernet/3com/typhoon.c +1 -2
@@ -1419,8 +1419,7 @@ typhoon_download_firmware(struct typhoon *tp)* the checksum, we can do this once, at the end. */ csum =csum_fold (csum_partial_copy_nocheck (image_data, - dpage, len, -0 )); + dpage, len));iowrite32 (len, ioaddr +TYPHOON_REG_BOOT_LENGTH ); iowrite32(le16_to_cpu((__force __le16)csum),
include/net/checksum.h +2 -2
@@ -49,10 +49,10 @@ static __inline__ __wsum csum_and_copy_to_user#ifndef _HAVE_ARCH_CSUM_AND_COPYstatic inline __wsum -csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum) +csum_partial_copy_nocheck (const void *src,void *dst,int len) {memcpy (dst, src, len); -return csum_partial (dst, len, sum); +return csum_partial (dst, len,0 ); }#endif
lib/iov_iter.c +1 -1
@@ -580,7 +580,7 @@ static size_t copy_pipe_to_iter(const void *addr, size_t bytes,static __wsum csum_and_memcpy (void *to,const void *from,size_t len,__wsum sum,size_t off) { -__wsum next =csum_partial_copy_nocheck (from, to, len,0 ); +__wsum next =csum_partial_copy_nocheck (from, to, len);return csum_block_add (sum, next, off); }
net/core/skbuff.c +2 -2
@@ -2736,7 +2736,7 @@ __wsum skb_copy_and_csum_bits(const struct sk_buff *skb, int offset,if (copy > len) copy = len; csum =csum_partial_copy_nocheck (skb->data + offset, to, - copy,0 ); + copy);if ((len -= copy) ==0 )return csum; offset += copy;
@@ -2766,7 +2766,7 @@ __wsum skb_copy_and_csum_bits(const struct sk_buff *skb, int offset, vaddr =kmap_atomic (p); csum2 =csum_partial_copy_nocheck (vaddr + p_off, to + copied, - p_len,0 ); + p_len);kunmap_atomic (vaddr); csum =csum_block_add (csum, csum2, pos); pos += p_len;
net/ipv4/icmp.c +1 -1
@@ -381,7 +381,7 @@ static void icmp_push_reply(struct icmp_bxm *icmp_param, csum =csum_partial_copy_nocheck ((void *)&icmp_param->data, (char *)icmph, - icmp_param->head_len,0 ); + icmp_param->head_len); skb_queue_walk(&sk->sk_write_queue, skb1) { csum =csum_add (csum, skb1->csum); }
net/ipv4/ip_output.c +1 -1
@@ -1642,7 +1642,7 @@ static int ip_reply_glue_bits(void *dptr, char *to, int offset, {__wsum csum; - csum =csum_partial_copy_nocheck (dptr+offset, to, len,0 ); + csum =csum_partial_copy_nocheck (dptr+offset, to, len); skb->csum =csum_block_add (skb->csum, csum, odd);return 0 ; }
net/ipv4/raw.c +1 -1
@@ -478,7 +478,7 @@ static int raw_getfrag(void *from, char *to, int offset, int len, int odd, skb->csum =csum_block_add ( skb->csum,csum_partial_copy_nocheck (rfv->hdr.c + offset, - to, copy,0 ), + to, copy), odd); odd =0 ;
net/ipv6/raw.c +1 -1
@@ -746,7 +746,7 @@ static int raw6_getfrag(void *from, char *to, int offset, int len, int odd, skb->csum =csum_block_add ( skb->csum,csum_partial_copy_nocheck (rfv->c + offset, - to, copy,0 ), + to, copy), odd); odd =0 ;
Signature · 1 line
2.11.0
kernel test robot
build botRe: [PATCH v2 05/20] csum_partial_copy_nocheck(): drop the last argument
Thank you for the patch! Yet something to improve:
Hi Al,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on sparc-next/master]
[also build test ERROR on sparc/master tip/x86/core linus/master v5.8-rc6] [cannot apply to next-20200723]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Al-Viro/xtensa-fix-access-check-in-csum_and_copy_from_user/20200724-103114 base: https://git.kernel.org/pub/scm/linux/kernel/git/davem/sparc-next.git master config: c6x-allyesconfig (attached as .config) compiler: c6x-elf-gcc (GCC) 9.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=c6x
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>):
net/core/skbuff.c: In function 'skb_copy_and_csum_bits':
>> net/core/skbuff.c:2739:12: error: macro "csum_partial_copy_nocheck" requires 4 arguments, but only 3 given
2739 | copy);
| ^
In file included from include/net/checksum.h:22,
from include/linux/skbuff.h:28,
from include/net/net_namespace.h:39,
from include/linux/inet.h:42,
from net/core/skbuff.c:43:
arch/c6x/include/asm/checksum.h:32: note: macro "csum_partial_copy_nocheck" defined here
32 | #define csum_partial_copy_nocheck(src, dst, len, sum) \
|
>> net/core/skbuff.c:2738:10: error: 'csum_partial_copy_nocheck' undeclared (first use in this function); did you mean 'csum_partial_copy'?
2738 | csum = csum_partial_copy_nocheck(skb->data + offset, to,
| ^~~~~~~~~~~~~~~~~~~~~~~~~
| csum_partial_copy
net/core/skbuff.c:2738:10: note: each undeclared identifier is reported only once for each function it appears in
net/core/skbuff.c:2769:16: error: macro "csum_partial_copy_nocheck" requires 4 arguments, but only 3 given
2769 | p_len);
| ^
In file included from include/net/checksum.h:22,
from include/linux/skbuff.h:28,
from include/net/net_namespace.h:39,
from include/linux/inet.h:42,
from net/core/skbuff.c:43:
arch/c6x/include/asm/checksum.h:32: note: macro "csum_partial_copy_nocheck" defined here
32 | #define csum_partial_copy_nocheck(src, dst, len, sum) \
|
--
net/ipv4/ip_output.c: In function 'ip_reply_glue_bits':
>> net/ipv4/ip_output.c:1645:55: error: macro "csum_partial_copy_nocheck" requires 4 arguments, but only 3 given
1645 | csum = csum_partial_copy_nocheck(dptr+offset, to, len);
| ^
In file included from include/net/checksum.h:22,
from include/linux/skbuff.h:28,
from include/net/net_namespace.h:39,
from include/linux/inet.h:42,
from net/ipv4/ip_output.c:59:
arch/c6x/include/asm/checksum.h:32: note: macro "csum_partial_copy_nocheck" defined here
32 | #define csum_partial_copy_nocheck(src, dst, len, sum) \
|
>> net/ipv4/ip_output.c:1645:9: error: 'csum_partial_copy_nocheck' undeclared (first use in this function); did you mean 'csum_partial_copy'?
1645 | csum = csum_partial_copy_nocheck(dptr+offset, to, len);
| ^~~~~~~~~~~~~~~~~~~~~~~~~
| csum_partial_copy
net/ipv4/ip_output.c:1645:9: note: each undeclared identifier is reported only once for each function it appears in
--
net/ipv4/raw.c: In function 'raw_getfrag':
>> net/ipv4/raw.c:481:18: error: macro "csum_partial_copy_nocheck" requires 4 arguments, but only 3 given
481 | to, copy),
| ^
In file included from include/net/checksum.h:22,
from include/linux/skbuff.h:28,
from include/linux/pim.h:5,
from include/linux/mroute.h:6,
from net/ipv4/raw.c:51:
arch/c6x/include/asm/checksum.h:32: note: macro "csum_partial_copy_nocheck" defined here
32 | #define csum_partial_copy_nocheck(src, dst, len, sum) \
|
>> net/ipv4/raw.c:480:5: error: 'csum_partial_copy_nocheck' undeclared (first use in this function); did you mean 'csum_partial_copy'?
480 | csum_partial_copy_nocheck(rfv->hdr.c + offset,
| ^~~~~~~~~~~~~~~~~~~~~~~~~
| csum_partial_copy
net/ipv4/raw.c:480:5: note: each undeclared identifier is reported only once for each function it appears in
--
net/ipv4/icmp.c: In function 'icmp_push_reply':
>> net/ipv4/icmp.c:384:28: error: macro "csum_partial_copy_nocheck" requires 4 arguments, but only 3 given
384 | icmp_param->head_len);
| ^
In file included from include/net/checksum.h:22,
from include/linux/skbuff.h:28,
from include/net/net_namespace.h:39,
from include/linux/inet.h:42,
from net/ipv4/icmp.c:69:
arch/c6x/include/asm/checksum.h:32: note: macro "csum_partial_copy_nocheck" defined here
32 | #define csum_partial_copy_nocheck(src, dst, len, sum) \
|
>> net/ipv4/icmp.c:382:10: error: 'csum_partial_copy_nocheck' undeclared (first use in this function); did you mean 'csum_partial_copy'?
382 | csum = csum_partial_copy_nocheck((void *)&icmp_param->data,
| ^~~~~~~~~~~~~~~~~~~~~~~~~
| csum_partial_copy
net/ipv4/icmp.c:382:10: note: each undeclared identifier is reported only once for each function it appears in
--
net/ipv6/raw.c: In function 'raw6_getfrag':
>> net/ipv6/raw.c:749:18: error: macro "csum_partial_copy_nocheck" requires 4 arguments, but only 3 given
749 | to, copy),
| ^
In file included from include/net/checksum.h:22,
from include/linux/skbuff.h:28,
from include/linux/if_ether.h:19,
from include/uapi/linux/ethtool.h:19,
from include/linux/ethtool.h:18,
from include/linux/netdevice.h:37,
from net/ipv6/raw.c:24:
arch/c6x/include/asm/checksum.h:32: note: macro "csum_partial_copy_nocheck" defined here
32 | #define csum_partial_copy_nocheck(src, dst, len, sum) \
|
>> net/ipv6/raw.c:748:5: error: 'csum_partial_copy_nocheck' undeclared (first use in this function); did you mean 'csum_partial_copy'?
748 | csum_partial_copy_nocheck(rfv->c + offset,
| ^~~~~~~~~~~~~~~~~~~~~~~~~
| csum_partial_copy net/ipv6/raw.c:748:5: note: each undeclared identifier is reported only once for each function it appears in
vim +/csum_partial_copy_nocheck +2739 net/core/skbuff.c
2724
2725 __wsum skb_copy_and_csum_bits(const struct sk_buff *skb, int offset,
2726 u8 *to, int len)
2727 {
2728 int start = skb_headlen(skb);
2729 int i, copy = start - offset;
2730 struct sk_buff *frag_iter;
2731 int pos = 0;
2732 __wsum csum = 0;
2733
2734 /* Copy header. */
2735 if (copy > 0) {
2736 if (copy > len)
2737 copy = len;
> 2738 csum = csum_partial_copy_nocheck(skb->data + offset, to, > 2739 copy);
2740 if ((len -= copy) == 0)
2741 return csum;
2742 offset += copy;
2743 to += copy;
2744 pos = copy;
2745 }
2746
2747 for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
2748 int end;
2749
2750 WARN_ON(start > offset + len);
2751
2752 end = start + skb_frag_size(&skb_shinfo(skb)->frags[i]);
2753 if ((copy = end - offset) > 0) {
2754 skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
2755 u32 p_off, p_len, copied;
2756 struct page *p;
2757 __wsum csum2;
2758 u8 *vaddr;
2759
2760 if (copy > len)
2761 copy = len;
2762
2763 skb_frag_foreach_page(frag,
2764 skb_frag_off(frag) + offset - start,
2765 copy, p, p_off, p_len, copied) {
2766 vaddr = kmap_atomic(p);
2767 csum2 = csum_partial_copy_nocheck(vaddr + p_off,
2768 to + copied,
2769 p_len);
2770 kunmap_atomic(vaddr);
2771 csum = csum_block_add(csum, csum2, pos);
2772 pos += p_len;
2773 }
2774
2775 if (!(len -= copy))
2776 return csum;
2777 offset += copy;
2778 to += copy;
2779 }
2780 start = end;
2781 }
2782
2783 skb_walk_frags(skb, frag_iter) {
2784 __wsum csum2;
2785 int end;
2786
2787 WARN_ON(start > offset + len);
2788
2789 end = start + frag_iter->len;
2790 if ((copy = end - offset) > 0) {
2791 if (copy > len)
2792 copy = len;
2793 csum2 = skb_copy_and_csum_bits(frag_iter,
2794 offset - start,
2795 to, copy);
2796 csum = csum_block_add(csum, csum2, pos);
2797 if ((len -= copy) == 0)
2798 return csum;
2799 offset += copy;
2800 to += copy;
2801 pos += copy;
2802 }
2803 start = end;
2804 }
2805 BUG_ON(len);
2806 return csum;
2807 }
2808 EXPORT_SYMBOL(skb_copy_and_csum_bits);
2809
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
Al Viro
maintainerPATCH v2 6/20 csum_and_copy_..._user(): pass 0xffffffff instead of 0 as initial sum
In reply to Al ViroFrom: Al Viro <[email protected]>
Preparation for the change of calling conventions; right now all callers pass 0 as initial sum. Passing 0xffffffff instead yields the values comparable mod 0xffff and guarantees that 0 will not be returned on success.
Note that this relies upon the correct behaviour with arbitrary initial sum and the above pretty much says "it's been untested with anything other than 0". Analysis is unpleasant, to put it mildly, but the suckers _are_ handling it correctly, surprisingly enough. Perhaps not too surprisingly, since most of the instances share the code with csum_partial_copy_nocheck(), which used to get some testing due to icmp_push_reply()...
--- lib/iov_iter.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
lib/iov_iter.c +3 -3
@@ -1451,7 +1451,7 @@ size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum,int err =0 ; next =csum_and_copy_from_user (v.iov_base, (to += v.iov_len) - v.iov_len, - v.iov_len,0 , &err); + v.iov_len, ~0U , &err);if (!err) { sum =csum_block_add (sum, next, off); off += v.iov_len;
@@ -1493,7 +1493,7 @@ bool csum_and_copy_from_iter_full(void *addr, size_t bytes, __wsum *csum,int err =0 ; next =csum_and_copy_from_user (v.iov_base, (to += v.iov_len) - v.iov_len, - v.iov_len,0 , &err); + v.iov_len, ~0U , &err);if (err)return false; sum =csum_block_add (sum, next, off);
@@ -1539,7 +1539,7 @@ size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *csump,int err =0 ; next =csum_and_copy_to_user ((from += v.iov_len) - v.iov_len, v.iov_base, - v.iov_len,0 , &err); + v.iov_len, ~0U , &err);if (!err) { sum =csum_block_add (sum, next, off); off += v.iov_len;
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH v2 7/20 saner calling conventions for csum_and_copy_..._user()
In reply to Al ViroFrom: Al Viro <[email protected]>
All callers of these primitives will
* discard anything we might've copied in case of error
* ignore the csum value in case of error
* always pass 0xffffffff as the initial sum, so the
resulting csum value (in case of success, that is) will never be 0.
* always pass a positive length.
That suggest the following calling conventions:
* don't pass err_ptr - just return 0 on error.
* don't bother with zeroing destination, etc. in case of error
* don't pass the initial sum - just use 0xffffffff.
This commit does the minimal conversion in the instances of csum_and_copy_...(); the changes of actual asm code behind them are done later in the series. Note that this asm code is often shared with csum_partial_copy_nocheck(); the difference is that csum_partial_copy_nocheck() passes 0 for initial sum while csum_and_copy_..._user() pass 0xffffffff. Fortunately, we are free to pass 0xffffffff in all cases and subsequent patches will use that freedom without any special comments.
A part that could be split off: parisc and uml/i386 claimed to have csum_and_copy_to_user() instances of their own, but those were identical to the generic one, so we simply drop them. Not sure if it's worth a separate commit...
--- arch/alpha/include/asm/checksum.h | 2 +- arch/alpha/lib/csum_partial_copy.c | 25 ++++++------- arch/arm/include/asm/checksum.h | 13 ++++--- arch/m68k/include/asm/checksum.h | 3 +- arch/m68k/lib/checksum.c | 8 ++--- arch/mips/include/asm/checksum.h | 46 ++++++++++++------------ arch/parisc/include/asm/checksum.h | 20 ----------- arch/powerpc/include/asm/checksum.h | 4 +-- arch/powerpc/lib/checksum_wrappers.c | 68 +++++++++++------------------------- arch/sh/include/asm/checksum_32.h | 36 +++++++++---------- arch/sparc/include/asm/checksum_32.h | 65 ++++++++++++++++------------------ arch/sparc/include/asm/checksum_64.h | 14 ++++---- arch/x86/include/asm/checksum_32.h | 35 ++++++++----------- arch/x86/include/asm/checksum_64.h | 6 ++-- arch/x86/lib/csum-wrappers_64.c | 38 +++++++++----------- arch/x86/um/asm/checksum_32.h | 23 ------------ arch/xtensa/include/asm/checksum.h | 30 ++++++++-------- include/net/checksum.h | 15 ++++---- lib/iov_iter.c | 19 +++++----- 19 files changed, 183 insertions(+), 287 deletions(-)
arch/alpha/include/asm/checksum.h +1 -1
@@ -43,7 +43,7 @@ extern __wsum csum_partial(const void *buff, int len, __wsum sum);*/ #define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER#define _HAVE_ARCH_CSUM_AND_COPY -__wsum csum_and_copy_from_user (const void __user *src,void *dst,int len,__wsum sum,int *errp); +__wsum csum_and_copy_from_user (const void __user *src,void *dst,int len);__wsum csum_partial_copy_nocheck (const void *src,void *dst,int len);
arch/alpha/lib/csum_partial_copy.c +11 -14
@@ -325,30 +325,27 @@ csum_partial_cfu_unaligned(const unsigned long __user * src, }__wsum -csum_and_copy_from_user (const void __user *src,void *dst,int len, -__wsum sum,int *errp) +csum_and_copy_from_user (const void __user *src,void *dst,int len) { -unsigned long checksum = (__force u32 ) sum; +unsigned long checksum = ~0U ;unsigned long soff =7 & (unsigned long ) src;unsigned long doff =7 & (unsigned long ) dst; +int err =0 ;if (len) { -if (!access_ok (src, len)) { -if (errp) *errp = -EFAULT ; -memset (dst,0 , len); -return sum; - } +if (!access_ok (src, len)) +return 0 ;if (!doff) {if (!soff) checksum =csum_partial_cfu_aligned ( (const unsigned long __user *) src, (unsigned long *) dst, - len-8 , checksum, errp); + len-8 , checksum, &err);else checksum =csum_partial_cfu_dest_aligned ( (const unsigned long __user *) src, (unsigned long *) dst, - soff, len-8 , checksum, errp); + soff, len-8 , checksum, &err); }else {unsigned long partial_dest;ldq_u (partial_dest, dst);
@@ -357,15 +354,15 @@ csum_and_copy_from_user(const void __user *src, void *dst, int len, (const unsigned long __user *) src, (unsigned long *) dst, doff, len-8 , checksum, - partial_dest, errp); + partial_dest, &err); else checksum =csum_partial_cfu_unaligned ( (const unsigned long __user *) src, (unsigned long *) dst, soff, doff, len-8 , checksum, - partial_dest, errp); + partial_dest, &err); } - checksum =from64to16 (checksum); + checksum = err ?0 :from64to16 (checksum); }return (__force __wsum )checksum; }
@@ -378,7 +375,7 @@ csum_partial_copy_nocheck(const void *src, void *dst, int len)mm_segment_t oldfs =get_fs ();set_fs (KERNEL_DS ); checksum =csum_and_copy_from_user ((__force const void __user *)src, - dst, len,0 ,NULL ); + dst, len);set_fs (oldfs);return checksum; }
arch/arm/include/asm/checksum.h +6 -7
@@ -43,16 +43,15 @@ csum_partial_copy_from_user(const void __user *src, void *dst, int len, __wsum s#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER#define _HAVE_ARCH_CSUM_AND_COPYstatic inline -__wsum csum_and_copy_from_user (const void __user *src,void *dst, -int len,__wsum sum,int *err_ptr) +__wsum csum_and_copy_from_user (const void __user *src,void *dst,int len) { -if (access_ok (src, len)) -return csum_partial_copy_from_user (src, dst, len, sum, err_ptr); +int err =0 ; -if (len) - *err_ptr = -EFAULT ; +if (!access_ok (src, len)) +return 0 ; -return sum; + sum =csum_partial_copy_from_user (src, dst, len, ~0U , &err); +return err ?0 : sum; }/*
arch/m68k/include/asm/checksum.h +1 -2
@@ -34,8 +34,7 @@ __wsum csum_partial(const void *buff, int len, __wsum sum);#define _HAVE_ARCH_CSUM_AND_COPYextern __wsum csum_and_copy_from_user (const void __user *src,void *dst, -int len,__wsum sum, -int *csum_err); +int len);extern __wsum csum_partial_copy_nocheck (const void *src,void *dst,int len);
arch/m68k/lib/checksum.c +3 -5
@@ -129,8 +129,7 @@ EXPORT_SYMBOL(csum_partial);*/ __wsum -csum_and_copy_from_user (const void __user *src,void *dst, -int len,__wsum sum,int *csum_err) +csum_and_copy_from_user (const void __user *src,void *dst,int len) {/* * GCC doesn't like more than 10 operands for the asm
@@ -138,6 +137,7 @@ csum_and_copy_from_user(const void __user *src, void *dst,* code. */ unsigned long tmp1, tmp2; +__wsum sum = ~0U ; __asm__("movel %2,%4\n\t" "btst #1,%4\n\t" /* Check alignment */
@@ -311,9 +311,7 @@ csum_and_copy_from_user(const void __user *src, void *dst, : "0" (sum), "1" (len), "2" (src), "3" (dst) ); - *csum_err = tmp2; - -return (sum); +return tmp2 ?0 : sum; }EXPORT_SYMBOL (csum_and_copy_from_user);
arch/mips/include/asm/checksum.h +22 -24
62 lines not on this page · raw message
arch/parisc/include/asm/checksum.h +0 -20
@@ -173,25 +173,5 @@ static __inline__ __sum16 csum_ipv6_magic(const struct in6_addr *saddr, return csum_fold(sum); } -/* -* Copy and checksum to user -*/ -#define HAVE_CSUM_COPY_USER -static __inline____wsum csum_and_copy_to_user (const void *src, -void __user *dst, -int len,__wsum sum, -int *err_ptr) -{ -/* code stolen from include/asm-mips64 */ - sum =csum_partial (src, len, sum); - -if (copy_to_user (dst, src, len)) { - *err_ptr = -EFAULT ; -return (__force __wsum)-1 ; - } - -return sum; -} -#endif
arch/powerpc/include/asm/checksum.h +2 -2
@@ -24,10 +24,10 @@ extern __wsum csum_partial_copy_generic(const void *src, void *dst,#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USERextern __wsum csum_and_copy_from_user (const void __user *src,void *dst, -int len,__wsum sum,int *err_ptr); +int len);#define HAVE_CSUM_COPY_USER extern __wsum csum_and_copy_to_user (const void *src,void __user *dst, -int len,__wsum sum,int *err_ptr); +int len);#define _HAVE_ARCH_CSUM_AND_COPY#define csum_partial_copy_nocheck (src, dst, len) \
arch/powerpc/lib/checksum_wrappers.c +21 -47
103 lines not on this page · raw message
arch/sh/include/asm/checksum_32.h +18 -18
@@ -50,15 +50,16 @@ __wsum csum_partial_copy_nocheck(const void *src, void *dst, int len)#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USERstatic inline -__wsum csum_and_copy_from_user (const void __user *src,void *dst, -int len,__wsum sum,int *err_ptr) +__wsum csum_and_copy_from_user (const void __user *src,void *dst,int len) { -if (access_ok (src, len)) -return csum_partial_copy_generic ((__force const void *)src, dst, - len, sum, err_ptr,NULL ); -if (len) - *err_ptr = -EFAULT ; -return sum; +int err =0 ; +__wsum sum = ~0U ; + +if (!access_ok (src, len)) +return 0 ; + sum =csum_partial_copy_generic ((__force const void *)src, dst, + len, sum, &err,NULL ); +return err ?0 : sum; }/*
@@ -199,16 +200,15 @@ static inline __sum16 csum_ipv6_magic(const struct in6_addr *saddr,#define HAVE_CSUM_COPY_USER static inline __wsum csum_and_copy_to_user (const void *src,void __user *dst, -int len,__wsum sum, -int *err_ptr) +int len) { -if (access_ok (dst, len)) -return csum_partial_copy_generic ((__force const void *)src, - dst, len, sum,NULL , err_ptr); - -if (len) - *err_ptr = -EFAULT ; - -return (__force __wsum)-1 ;/* invalid checksum */ +int err =0 ; +__wsum sum = ~0U ; + +if (!access_ok (dst, len)) +return 0 ; + sum =csum_partial_copy_generic ((__force const void *)src, + dst, len, sum,NULL , &err); +return err ?0 : sum; }#endif /* __ASM_SH_CHECKSUM_H */
arch/sparc/include/asm/checksum_32.h +30 -35
91 lines not on this page · raw message
arch/sparc/include/asm/checksum_64.h +6 -8
@@ -51,12 +51,11 @@ long __csum_partial_copy_from_user(const void __user *src,static inline __wsum csum_and_copy_from_user (const void __user *src, -void *dst,int len, -__wsum sum,int *err) +void *dst,int len) { -long ret =__csum_partial_copy_from_user (src, dst, len, sum); +long ret =__csum_partial_copy_from_user (src, dst, len, ~0U );if (ret <0 ) - *err = -EFAULT ; +return 0 ;return (__force __wsum ) ret; }
@@ -70,12 +69,11 @@ long __csum_partial_copy_to_user(const void *src,static inline __wsum csum_and_copy_to_user (const void *src, -void __user *dst,int len, -__wsum sum,int *err) +void __user *dst,int len) { -long ret =__csum_partial_copy_to_user (src, dst, len, sum); +long ret =__csum_partial_copy_to_user (src, dst, len, ~0U );if (ret <0 ) - *err = -EFAULT ; +return 0 ;return (__force __wsum ) ret; }
arch/x86/include/asm/checksum_32.h +14 -21
@@ -44,22 +44,19 @@ static inline __wsum csum_partial_copy_nocheck(const void *src, void *dst, int l }static inline __wsum csum_and_copy_from_user (const void __user *src, -void *dst,int len, -__wsum sum,int *err_ptr) +void *dst,int len) {__wsum ret; +int err =0 ;might_sleep (); -if (!user_access_begin (src, len)) { -if (len) - *err_ptr = -EFAULT ; -return sum; - } +if (!user_access_begin (src, len)) +return 0 ; ret =csum_partial_copy_generic ((__force void *)src, dst, - len, sum, err_ptr,NULL ); + len, ~0U , &err,NULL );user_access_end (); -return ret; +return err ?0 : ret; }/*
@@ -177,23 +174,19 @@ static inline __sum16 csum_ipv6_magic(const struct in6_addr *saddr,*/ static inline __wsum csum_and_copy_to_user (const void *src,void __user *dst, -int len,__wsum sum, -int *err_ptr) +int len) {__wsum ret; +int err =0 ;might_sleep (); -if (user_access_begin (dst, len)) { - ret =csum_partial_copy_generic (src, (__force void *)dst, - len, sum,NULL , err_ptr); -user_access_end (); -return ret; - } +if (!user_access_begin (dst, len)) +return 0 ; -if (len) - *err_ptr = -EFAULT ; - -return (__force __wsum)-1 ;/* invalid checksum */ + ret =csum_partial_copy_generic (src, (__force void *)dst, + len, ~0U ,NULL , &err); +user_access_end (); +return err ?0 : ret; }#endif /* _ASM_X86_CHECKSUM_32_H */
arch/x86/include/asm/checksum_64.h +2 -4
@@ -135,10 +135,8 @@ extern __visible __wsum csum_partial_copy_generic(const void *src, const void *d int *src_err_ptr, int *dst_err_ptr); -extern __wsum csum_and_copy_from_user (const void __user *src,void *dst, -int len,__wsum isum,int *errp); -extern __wsum csum_and_copy_to_user (const void *src,void __user *dst, -int len,__wsum isum,int *errp); +extern __wsum csum_and_copy_from_user (const void __user *src,void *dst,int len); +extern __wsum csum_and_copy_to_user (const void *src,void __user *dst,int len);extern __wsum csum_partial_copy_nocheck (const void *src,void *dst,int len);/**
arch/x86/lib/csum-wrappers_64.c +16 -22
@@ -22,13 +22,15 @@*/ __wsum csum_and_copy_from_user (const void __user *src,void *dst, -int len,__wsum isum,int *errp) +int len) { +int err =0 ; +__wsum isum = ~0U ; +might_sleep (); - *errp =0 ;if (!user_access_begin (src, len)) - goto out_err; +return 0 ;/* * Why 6, not 7? To handle odd addresses aligned we
@@ -53,20 +55,15 @@ csum_and_copy_from_user(const void __user *src, void *dst, } } isum =csum_partial_copy_generic ((__force const void *)src, - dst, len, isum, errp,NULL ); + dst, len, isum, &err,NULL );user_access_end (); -if (unlikely (*errp)) - goto out_err; - +if (unlikely (err)) + isum =0 ;return isum; out:user_access_end (); -out_err: - *errp = -EFAULT ; -memset (dst,0 , len); - -return isum; +return 0 ; }EXPORT_SYMBOL (csum_and_copy_from_user);
@@ -83,16 +80,15 @@ EXPORT_SYMBOL(csum_and_copy_from_user);*/ __wsum csum_and_copy_to_user (const void *src,void __user *dst, -int len,__wsum isum,int *errp) +int len) { -__wsum ret; +__wsum ret, isum = ~0U ; +int err =0 ;might_sleep (); -if (!user_access_begin (dst, len)) { - *errp = -EFAULT ; +if (!user_access_begin (dst, len))return 0 ; - }if (unlikely ((unsigned long )dst &6 )) {while (((unsigned long )dst &6 ) && len >=2 ) {
@@ -107,15 +103,13 @@ csum_and_copy_to_user(const void *src, void __user *dst, } } - *errp =0 ; ret =csum_partial_copy_generic (src, (void __force *)dst, - len, isum,NULL , errp); + len, isum,NULL , &err);user_access_end (); -return ret; +return err ?0 : ret; out:user_access_end (); - *errp = -EFAULT ; -return isum; +return 0 ; }EXPORT_SYMBOL (csum_and_copy_to_user);
arch/x86/um/asm/checksum_32.h +0 -23
@@ -35,27 +35,4 @@ static __inline__ __sum16 csum_ipv6_magic(const struct in6_addr *saddr, return csum_fold(sum); } -/* -* Copy and checksum to user -*/ -#define HAVE_CSUM_COPY_USER -static __inline____wsum csum_and_copy_to_user (const void *src, -void __user *dst, -int len,__wsum sum,int *err_ptr) -{ -if (access_ok (dst, len)) { -if (copy_to_user (dst, src, len)) { - *err_ptr = -EFAULT ; -return (__force __wsum)-1 ; - } - -return csum_partial (src, len, sum); - } - -if (len) - *err_ptr = -EFAULT ; - -return (__force __wsum)-1 ;/* invalid checksum */ -} -#endif
arch/xtensa/include/asm/checksum.h +16 -14
@@ -55,14 +55,16 @@ __wsum csum_partial_copy_nocheck(const void *src, void *dst, int len)#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USERstatic inline __wsum csum_and_copy_from_user (const void __user *src,void *dst, -int len,__wsum sum,int *err_ptr) +int len) { -if (access_ok (src, len)) -return csum_partial_copy_generic ((__force const void *)src, dst, - len, sum, err_ptr,NULL ); -if (len) - *err_ptr = -EFAULT ; -return sum; +int err =0 ; + +if (!access_ok (src, len)) +return 0 ; + + sum =csum_partial_copy_generic ((__force const void *)src, dst, + len, ~0U , &err,NULL ); +return err ?0 : sum; }/*
@@ -243,15 +245,15 @@ static __inline__ __sum16 csum_ipv6_magic(const struct in6_addr *saddr,*/ #define HAVE_CSUM_COPY_USER static __inline____wsum csum_and_copy_to_user (const void *src, -void __user *dst,int len, -__wsum sum,int *err_ptr) +void __user *dst,int len) { -if (access_ok (dst, len)) -return csum_partial_copy_generic (src,dst,len,sum,NULL ,err_ptr); +int err =0 ; +__wsum sum = ~0U ; -if (len) - *err_ptr = -EFAULT ; +if (!access_ok (dst, len)) +return 0 ; -return (__force __wsum)-1 ;/* invalid checksum */ + sum =csum_partial_copy_generic (src,dst,len,sum,NULL ,&err); +return err ?0 : sum; }#endif
include/net/checksum.h +6 -9
@@ -24,26 +24,23 @@#ifndef _HAVE_ARCH_COPY_AND_CSUM_FROM_USERstatic inline __wsum csum_and_copy_from_user (const void __user *src,void *dst, -int len,__wsum sum,int *err_ptr) +int len) {if (copy_from_user (dst, src, len)) - *err_ptr = -EFAULT ; -return csum_partial (dst, len, sum); +return 0 ; +return csum_partial (dst, len, ~0U ); }#endif #ifndef HAVE_CSUM_COPY_USER static __inline____wsum csum_and_copy_to_user -(const void *src,void __user *dst,int len,__wsum sum,int *err_ptr) +(const void *src,void __user *dst,int len) { - sum =csum_partial (src, len, sum); +__wsum sum =csum_partial (src, len, ~0U );if (copy_to_user (dst, src, len) ==0 )return sum; -if (len) - *err_ptr = -EFAULT ; - -return (__force __wsum)-1 ;/* invalid checksum */ +return 0 ; }#endif
lib/iov_iter.c +8 -11
@@ -1448,15 +1448,14 @@ size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum,return 0 ; } iterate_and_advance(i, bytes, v, ({ - int err = 0; next = csum_and_copy_from_user(v.iov_base, (to += v.iov_len) - v.iov_len, - v.iov_len, ~0U, &err); - if (!err) { + v.iov_len); + if (next) { sum = csum_block_add(sum, next, off); off += v.iov_len; } - err ? v.iov_len : 0; + next ? 0 : v.iov_len; }), ({ char *p = kmap_atomic(v.bv_page); sum = csum_and_memcpy((to += v.bv_len) - v.bv_len,
@@ -1490,11 +1489,10 @@ bool csum_and_copy_from_iter_full(void *addr, size_t bytes, __wsum *csum,if (unlikely (i->count < bytes))return false; iterate_all_kinds(i, bytes, v, ({ - int err = 0; next = csum_and_copy_from_user(v.iov_base, (to += v.iov_len) - v.iov_len, - v.iov_len, ~0U, &err); - if (err) + v.iov_len); + if (!next) return false; sum = csum_block_add(sum, next, off); off += v.iov_len;
@@ -1536,15 +1534,14 @@ size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *csump,return 0 ; } iterate_and_advance(i, bytes, v, ({ - int err = 0; next = csum_and_copy_to_user((from += v.iov_len) - v.iov_len, v.iov_base, - v.iov_len, ~0U, &err); - if (!err) { + v.iov_len); + if (next) { sum = csum_block_add(sum, next, off); off += v.iov_len; } - err ? v.iov_len : 0; + next ? 0 : v.iov_len; }), ({ char *p = kmap_atomic(v.bv_page); sum = csum_and_memcpy(p + v.bv_offset,
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH v2 8/20 alpha: propagate the calling convention changes down to csum_partial_copy.c helpers
In reply to Al ViroFrom: Al Viro <[email protected]>
get rid of set_fs() in csum_partial_copy_nocheck(), while we are at it - just take the part of csum_and_copy_from_user() sans the access_ok() check into a helper function and have csum_partial_copy_nocheck() call that.
--- arch/alpha/lib/csum_partial_copy.c | 157 ++++++++++++++++--------------------- 1 file changed, 69 insertions(+), 88 deletions(-)
arch/alpha/lib/csum_partial_copy.c +69 -88
@@ -39,12 +39,11 @@ __asm__ __volatile__("insql %1,%2,%0":"=r" (z):"r" (x),"r" (y))#define insqh (x,y,z) \ __asm__ __volatile__("insqh %1,%2,%0":"=r" (z):"r" (x),"r" (y)) - -#define __get_user_u (x,ptr) \ +#define __get_word (insn,x,ptr) \ ({ \ long __guu_err; \ __asm__ __volatile__( \ - "1: ldq_u %0,%2\n" \ + "1: "#insn" %0,%2\n" \ "2:\n" \ EXC(1b,2b,%0,%1) \ : "=r"(x), "=r"(__guu_err) \
@@ -52,19 +51,6 @@ __asm__ __volatile__("insqh %1,%2,%0":"=r" (z):"r" (x),"r" (y)) __guu_err; \ }) -#define __put_user_u (x,ptr) \ -({ \ - long __puu_err; \ - __asm__ __volatile__( \ - "1: stq_u %2,%1\n" \ - "2:\n" \ - EXC(1b,2b,$31,%0) \ - : "=r"(__puu_err) \ - : "m"(__m(addr)), "rJ"(x), "0"(0)); \ - __puu_err; \ -}) - -static inline unsigned short from64to16 (unsigned long x) {/* Using extract instructions is a bit more efficient
@@ -95,15 +81,15 @@ static inline unsigned short from64to16(unsigned long x)*/ static inline unsigned long csum_partial_cfu_aligned (const unsigned long __user *src,unsigned long *dst, -long len,unsigned long checksum, -int *errp) +long len) { +unsigned long checksum = ~0U ;unsigned long carry =0 ; -int err =0 ;while (len >=0 ) {unsigned long word; - err |=__get_user (word, src); +if (__get_word (ldq, word, src)) +return 0 ; checksum += carry; src++; checksum += word;
@@ -116,7 +102,8 @@ csum_partial_cfu_aligned(const unsigned long __user *src, unsigned long *dst, checksum += carry;if (len) {unsigned long word, tmp; - err |=__get_user (word, src); +if (__get_word (ldq, word, src)) +return 0 ; tmp = *dst;mskql (word, len, word); checksum += word;
@@ -125,7 +112,6 @@ csum_partial_cfu_aligned(const unsigned long __user *src, unsigned long *dst, *dst = word | tmp; checksum += carry; } -if (err && errp) *errp = err;return checksum; }
@@ -137,20 +123,21 @@ static inline unsigned long csum_partial_cfu_dest_aligned(const unsigned long __user *src, unsigned long *dst, unsigned long soff, - long len, unsigned long checksum, - int *errp) + long len) { unsigned long first;unsigned long word, carry;unsigned long lastsrc =7 +len+(unsigned long )src; -int err =0 ; +unsigned long checksum = ~0U ; - err |=__get_user_u (first,src); +if (__get_word (ldq_u, first,src)) +return 0 ; carry =0 ;while (len >=0 ) {unsigned long second; - err |=__get_user_u (second, src+1 ); +if (__get_word (ldq_u, second, src+1 )) +return 0 ;extql (first, soff, word); len -=8 ; src++;
@@ -168,7 +155,8 @@ csum_partial_cfu_dest_aligned(const unsigned long __user *src,if (len) {unsigned long tmp;unsigned long second; - err |=__get_user_u (second, lastsrc); +if (__get_word (ldq_u, second, lastsrc)) +return 0 ; tmp = *dst;extql (first, soff, word);extqh (second, soff, first);
@@ -180,7 +168,6 @@ csum_partial_cfu_dest_aligned(const unsigned long __user *src, *dst = word | tmp; checksum += carry; } -if (err && errp) *errp = err;return checksum; }
@@ -191,18 +178,18 @@ static inline unsigned long csum_partial_cfu_src_aligned(const unsigned long __user *src, unsigned long *dst, unsigned long doff, - long len, unsigned long checksum, - unsigned long partial_dest, - int *errp) + long len, + unsigned long partial_dest) { unsigned long carry = 0;unsigned long word;unsigned long second_dest; -int err =0 ; +unsigned long checksum = ~0U ;mskql (partial_dest, doff, partial_dest);while (len >=0 ) { - err |=__get_user (word, src); +if (__get_word (ldq, word, src)) +return 0 ; len -=8 ;insql (word, doff, second_dest); checksum += carry;
@@ -216,7 +203,8 @@ csum_partial_cfu_src_aligned(const unsigned long __user *src, len +=8 ;if (len) { checksum += carry; - err |=__get_user (word, src); +if (__get_word (ldq, word, src)) +return 0 ;mskql (word, len, word); len -=8 ; checksum += word;
@@ -237,7 +225,6 @@ csum_partial_cfu_src_aligned(const unsigned long __user *src,stq_u (partial_dest | second_dest, dst); out: checksum += carry; -if (err && errp) *errp = err;return checksum; }
@@ -249,23 +236,23 @@ static inline unsigned long csum_partial_cfu_unaligned(const unsigned long __user * src, unsigned long * dst, unsigned long soff, unsigned long doff, - long len, unsigned long checksum, - unsigned long partial_dest, - int *errp) + long len, unsigned long partial_dest) { unsigned long carry = 0;unsigned long first;unsigned long lastsrc; -int err =0 ; +unsigned long checksum = ~0U ; - err |=__get_user_u (first, src); +if (__get_word (ldq_u, first, src)) +return 0 ; lastsrc =7 +len+(unsigned long )src;mskql (partial_dest, doff, partial_dest);while (len >=0 ) {unsigned long second, word;unsigned long second_dest; - err |=__get_user_u (second, src+1 ); +if (__get_word (ldq_u, second, src+1 )) +return 0 ;extql (first, soff, word); checksum += carry; len -=8 ;
@@ -286,7 +273,8 @@ csum_partial_cfu_unaligned(const unsigned long __user * src,unsigned long second, word;unsigned long second_dest; - err |=__get_user_u (second, lastsrc); +if (__get_word (ldq_u, second, lastsrc)) +return 0 ;extql (first, soff, word);extqh (second, soff, first); word |= first;
@@ -307,7 +295,8 @@ csum_partial_cfu_unaligned(const unsigned long __user * src,unsigned long second, word;unsigned long second_dest; - err |=__get_user_u (second, lastsrc); +if (__get_word (ldq_u, second, lastsrc)) +return 0 ;extql (first, soff, word);extqh (second, soff, first); word |= first;
@@ -320,63 +309,55 @@ csum_partial_cfu_unaligned(const unsigned long __user * src,stq_u (partial_dest | word | second_dest, dst); checksum += carry; } -if (err && errp) *errp = err;return checksum; } -__wsum -csum_and_copy_from_user (const void __user *src,void *dst,int len) +static __wsum __csum_and_copy (const void __user *src,void *dst,int len) { -unsigned long checksum = ~0U ;unsigned long soff =7 & (unsigned long ) src;unsigned long doff =7 & (unsigned long ) dst; -int err =0 ; - -if (len) { -if (!access_ok (src, len)) -return 0 ; -if (!doff) { -if (!soff) - checksum =csum_partial_cfu_aligned ( - (const unsigned long __user *) src, - (unsigned long *) dst, - len-8 , checksum, &err); -else - checksum =csum_partial_cfu_dest_aligned ( - (const unsigned long __user *) src, - (unsigned long *) dst, - soff, len-8 , checksum, &err); - }else { -unsigned long partial_dest; -ldq_u (partial_dest, dst); -if (!soff) - checksum =csum_partial_cfu_src_aligned ( - (const unsigned long __user *) src, - (unsigned long *) dst, - doff, len-8 , checksum, - partial_dest, &err); -else - checksum =csum_partial_cfu_unaligned ( - (const unsigned long __user *) src, - (unsigned long *) dst, - soff, doff, len-8 , checksum, - partial_dest, &err); - } - checksum = err ?0 :from64to16 (checksum); +unsigned long checksum; + +if (!doff) { +if (!soff) + checksum =csum_partial_cfu_aligned ( + (const unsigned long __user *) src, + (unsigned long *) dst, len-8 ); +else + checksum =csum_partial_cfu_dest_aligned ( + (const unsigned long __user *) src, + (unsigned long *) dst, + soff, len-8 ); + }else { +unsigned long partial_dest; +ldq_u (partial_dest, dst); +if (!soff) + checksum =csum_partial_cfu_src_aligned ( + (const unsigned long __user *) src, + (unsigned long *) dst, + doff, len-8 , partial_dest); +else + checksum =csum_partial_cfu_unaligned ( + (const unsigned long __user *) src, + (unsigned long *) dst, + soff, doff, len-8 , partial_dest); } -return (__force __wsum )checksum; +return (__force __wsum )from64to16 (checksum); +} + +__wsum +csum_and_copy_from_user (const void __user *src,void *dst,int len) +{ +if (!access_ok (src, len)) +return 0 ; +return __csum_and_copy (src, dst, len); }EXPORT_SYMBOL (csum_and_copy_from_user);__wsum csum_partial_copy_nocheck (const void *src,void *dst,int len) { -__wsum checksum; -mm_segment_t oldfs =get_fs (); -set_fs (KERNEL_DS ); - checksum =csum_and_copy_from_user ((__force const void __user *)src, +return __csum_and_copy ((__force const void __user *)src, dst, len); -set_fs (oldfs); -return checksum; }EXPORT_SYMBOL (csum_partial_copy_nocheck);
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH v2 9/20 arm: propagate the calling convention changes down to csum_partial_copy_from_user()
In reply to Al ViroFrom: Al Viro <[email protected]>
... and get rid of the "clean the destination on error" crap. Simplifies the fault handlers and the function itself...
--- arch/arm/include/asm/checksum.h | 7 ++----- arch/arm/lib/csumpartialcopy.S | 1 - arch/arm/lib/csumpartialcopygeneric.S | 1 + arch/arm/lib/csumpartialcopyuser.S | 26 ++++++-------------------- 4 files changed, 9 insertions(+), 26 deletions(-)
arch/arm/include/asm/checksum.h +2 -5
@@ -38,20 +38,17 @@ __wsum csum_partial_copy_nocheck(const void *src, void *dst, int len);__wsum -csum_partial_copy_from_user (const void __user *src,void *dst,int len,__wsum sum,int *err_ptr); +csum_partial_copy_from_user (const void __user *src,void *dst,int len);#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER#define _HAVE_ARCH_CSUM_AND_COPYstatic inline __wsum csum_and_copy_from_user (const void __user *src,void *dst,int len) { -int err =0 ; -if (!access_ok (src, len))return 0 ; - sum =csum_partial_copy_from_user (src, dst, len, ~0U , &err); -return err ?0 : sum; +return csum_partial_copy_from_user (src, dst, len); }/*
arch/arm/lib/csumpartialcopy.S +0 -1
@@ -16,7 +16,6 @@ .macro save_regs stmfd sp!, {r1, r4 - r8, lr} - mov r3, #0 .endm .macro load_regs
arch/arm/lib/csumpartialcopygeneric.S +1 -0
@@ -86,6 +86,7 @@ sum .req r3 FN_ENTRY save_regs + mov sum, #-1 cmp len, #8 @ Ensure that we have at least blo .Lless8 @ 8 bytes to copy.
arch/arm/lib/csumpartialcopyuser.S +6 -20
@@ -62,9 +62,9 @@ /* * unsigned int - * csum_partial_copy_from_user(const char *src, char *dst, int len, int sum, int *err_ptr) - * r0 = src, r1 = dst, r2 = len, r3 = sum, [sp] = *err_ptr - * Returns : r0 = checksum, [[sp, #0], #0] = 0 or -EFAULT + * csum_partial_copy_from_user(const char *src, char *dst, int len) + * r0 = src, r1 = dst, r2 = len + * Returns : r0 = checksum or 0 */ #define FN_ENTRY ENTRY(csum_partial_copy_from_user)
@@ -73,25 +73,11 @@ #include "csumpartialcopygeneric.S" /* - * FIXME: minor buglet here - * We don't return the checksum for the data present in the buffer. To do - * so properly, we would have to add in whatever registers were loaded before - * the fault, which, with the current asm above is not predictable. + * We report fault by returning 0 csum - impossible in normal case, since + * we start with 0xffffffff for initial sum. */ .pushsection .text.fixup,"ax" .align 4 -9001: mov r4, #-EFAULT -#ifdef CONFIG_CPU_SW_DOMAIN_PAN - ldr r5, [sp, #9*4] @ *err_ptr -#else - ldr r5, [sp, #8*4] @ *err_ptr -#endif - str r4, [r5] - ldmia sp, {r1, r2} @ retrieve dst, len - add r2, r2, r1 - mov r0, #0 @ zero the buffer -9002: teq r2, r1 - strbne r0, [r1], #1 - bne 9002b +9001: mov r0, #0 load_regs .popsection
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH v2 10/20 m68k: get rid of zeroing destination on error in csum_and_copy_from_user()
In reply to Al ViroFrom: Al Viro <[email protected]>
--- arch/m68k/lib/checksum.c | 79 +++++++++--------------------------------------- 1 file changed, 15 insertions(+), 64 deletions(-)
arch/m68k/lib/checksum.c +15 -64
@@ -236,82 +236,33 @@ csum_and_copy_from_user(const void __user *src, void *dst, int len) "clrl %5\n\t" "addxl %5,%0\n\t" /* add X bit */ "7:\t" -"clrl %5\n" /* no error - clear return value */ -"8:\n" ".section .fixup,\"ax\"\n" ".even\n" -/* If any exception occurs zero out the rest. -Similarities with the code above are intentional :-) */ + /* If any exception occurs, return 0 */ "90:\t" -"clrw %3@+\n\t" -"movel %1,%4\n\t" -"lsrl #5,%1\n\t" -"jeq 1f\n\t" -"subql #1,%1\n" -"91:\t" -"clrl %3@+\n" -"92:\t" -"clrl %3@+\n" -"93:\t" -"clrl %3@+\n" -"94:\t" -"clrl %3@+\n" -"95:\t" -"clrl %3@+\n" -"96:\t" -"clrl %3@+\n" -"97:\t" -"clrl %3@+\n" -"98:\t" -"clrl %3@+\n\t" -"dbra %1,91b\n\t" -"clrw %1\n\t" -"subql #1,%1\n\t" -"jcc 91b\n" -"1:\t" -"movel %4,%1\n\t" -"andw #0x1c,%4\n\t" -"jeq 1f\n\t" -"lsrw #2,%4\n\t" -"subqw #1,%4\n" -"99:\t" -"clrl %3@+\n\t" -"dbra %4,99b\n\t" -"1:\t" -"andw #3,%1\n\t" -"jeq 9f\n" -"100:\t" -"clrw %3@+\n\t" -"tstw %1\n\t" -"jeq 9f\n" -"101:\t" -"clrb %3@+\n" - "9:\t" -#define STR (X ) STR1(X) -#define STR1 (X ) #X - "moveq #-" STR(EFAULT) ",%5\n\t" - "jra 8b\n" + "clrl %0\n" + "jra 7b\n" ".previous\n" ".section __ex_table,\"a\"\n" ".long 10b,90b\n" - ".long 11b,91b\n" - ".long 12b,92b\n" - ".long 13b,93b\n" - ".long 14b,94b\n" - ".long 15b,95b\n" - ".long 16b,96b\n" - ".long 17b,97b\n" - ".long 18b,98b\n" - ".long 19b,99b\n" - ".long 20b,100b\n" - ".long 21b,101b\n" + ".long 11b,90b\n" + ".long 12b,90b\n" + ".long 13b,90b\n" + ".long 14b,90b\n" + ".long 15b,90b\n" + ".long 16b,90b\n" + ".long 17b,90b\n" + ".long 18b,90b\n" + ".long 19b,90b\n" + ".long 20b,90b\n" + ".long 21b,90b\n" ".previous" : "=d" (sum), "=d" (len), "=a" (src), "=a" (dst), "=&d" (tmp1), "=d" (tmp2) : "0" (sum), "1" (len), "2" (src), "3" (dst) ); -return tmp2 ?0 : sum; +return sum; }EXPORT_SYMBOL (csum_and_copy_from_user);
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH v2 11/20 sh: propage the calling conventions change down to csum_partial_copy_generic()
In reply to Al ViroFrom: Al Viro <[email protected]>
... and get rid of zeroing destination on error there.
--- arch/sh/include/asm/checksum_32.h | 20 ++----- arch/sh/lib/checksum.S | 119 +++++++++++--------------------------- 2 files changed, 39 insertions(+), 100 deletions(-)
arch/sh/include/asm/checksum_32.h +4 -16
@@ -30,9 +30,7 @@ asmlinkage __wsum csum_partial(const void *buff, int len, __wsum sum);* better 64-bit) boundary */ -asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst, -int len,__wsum sum, -int *src_err_ptr,int *dst_err_ptr); +asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst, int len);#define _HAVE_ARCH_CSUM_AND_COPY/*
@@ -45,21 +43,16 @@ asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst,static inline __wsum csum_partial_copy_nocheck (const void *src,void *dst,int len) { -return csum_partial_copy_generic (src, dst, len,0 ,NULL ,NULL ); +return csum_partial_copy_generic (src, dst, len); }#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USERstatic inline __wsum csum_and_copy_from_user (const void __user *src,void *dst,int len) { -int err =0 ; -__wsum sum = ~0U ; -if (!access_ok (src, len))return 0 ; - sum =csum_partial_copy_generic ((__force const void *)src, dst, - len, sum, &err,NULL ); -return err ?0 : sum; +return csum_partial_copy_generic ((__force const void *)src, dst, len); }/*
@@ -202,13 +195,8 @@ static inline __wsum csum_and_copy_to_user(const void *src, void __user *dst, int len) { - int err = 0; -__wsum sum = ~0U ; -if (!access_ok (dst, len))return 0 ; - sum =csum_partial_copy_generic ((__force const void *)src, - dst, len, sum,NULL , &err); -return err ?0 : sum; +return csum_partial_copy_generic ((__force const void *)src, dst, len); }#endif /* __ASM_SH_CHECKSUM_H */
arch/sh/lib/checksum.S +35 -84
200 lines not on this page · raw message
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH v2 12/20 i386: propagate the calling conventions change down to csum_partial_copy_generic()
In reply to Al ViroFrom: Al Viro <[email protected]>
... and don't bother zeroing destination on error
--- arch/x86/include/asm/checksum_32.h | 18 ++---- arch/x86/lib/checksum_32.S | 117 +++++++++++++------------------------ 2 files changed, 47 insertions(+), 88 deletions(-)
arch/x86/include/asm/checksum_32.h +6 -12
@@ -27,9 +27,7 @@ asmlinkage __wsum csum_partial(const void *buff, int len, __wsum sum);* better 64-bit) boundary */ -asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst, -int len,__wsum sum, -int *src_err_ptr,int *dst_err_ptr); +asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst, int len);/* * Note: when you get a NULL pointer exception here this means someone
@@ -40,23 +38,21 @@ asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst,*/ static inline __wsum csum_partial_copy_nocheck (const void *src,void *dst,int len) { -return csum_partial_copy_generic (src, dst, len,0 ,NULL ,NULL ); +return csum_partial_copy_generic (src, dst, len); }static inline __wsum csum_and_copy_from_user (const void __user *src,void *dst,int len) {__wsum ret; -int err =0 ;might_sleep ();if (!user_access_begin (src, len))return 0 ; - ret =csum_partial_copy_generic ((__force void *)src, dst, - len, ~0U , &err,NULL ); + ret =csum_partial_copy_generic ((__force void *)src, dst, len);user_access_end (); -return err ?0 : ret; +return ret; }/*
@@ -177,16 +173,14 @@ static inline __wsum csum_and_copy_to_user(const void *src, int len) {__wsum ret; -int err =0 ;might_sleep ();if (!user_access_begin (dst, len))return 0 ; - ret =csum_partial_copy_generic (src, (__force void *)dst, - len, ~0U ,NULL , &err); + ret =csum_partial_copy_generic (src, (__force void *)dst, len);user_access_end (); -return err ?0 : ret; +return ret; }#endif /* _ASM_X86_CHECKSUM_32_H */
arch/x86/lib/checksum_32.S +41 -76
228 lines not on this page · raw message
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH v2 13/20 sparc32: propagate the calling conventions change down to __csum_partial_copy_sparc_generic()
In reply to Al ViroFrom: Al Viro <[email protected]>
... and get rid of zeroing the target, etc. on fault.
All exception handlers merge into one; moreover, since we are not calling lookup_fault() anymore, we don't need the magic with passing arguments for it from the page fault handler.
--- arch/sparc/include/asm/checksum_32.h | 49 +-------- arch/sparc/lib/checksum_32.S | 202 +++++++---------------------------- arch/sparc/mm/fault_32.c | 6 +- 3 files changed, 44 insertions(+), 213 deletions(-)
arch/sparc/include/asm/checksum_32.h +5 -44
@@ -50,9 +50,9 @@ csum_partial_copy_nocheck(const void *src, void *dst, int len) __asm__ __volatile__ ("call __csum_partial_copy_sparc_generic\n\t" -" mov %6, %%g7\n" +" mov -1, %%g7\n" :"=&r" (ret),"=&r" (d),"=&r" (l) - :"0" (ret),"1" (d),"2" (l),"r" (0 ) + :"0" (ret),"1" (d),"2" (l) :"o2" ,"o3" ,"o4" ,"o5" ,"o7" ,"g2" ,"g3" ,"g4" ,"g5" ,"g7" ,"memory" ,"cc" );
@@ -61,29 +61,10 @@ csum_partial_copy_nocheck(const void *src, void *dst, int len)static inline __wsum csum_and_copy_from_user (const void __user *src,void *dst,int len) - { - register unsigned long ret asm("o0") = (unsigned long)src; - register char *d asm("o1") = dst; - register int l asm("g1") = len; - register __wsum s asm("g7") = ~0U; -int err =0 ; - +{if (unlikely (!access_ok (src, len)))return 0 ; - - __asm__ __volatile__ ( -".section __ex_table,#alloc\n\t" -".align 4\n\t" -".word 1f,2\n\t" -".previous\n" -"1:\n\t" -"call __csum_partial_copy_sparc_generic\n\t" -" st %8, [%%sp + 64]\n" - :"=&r" (ret),"=&r" (d),"=&r" (l),"=&r" (s) - :"0" (ret),"1" (d),"2" (l),"3" (s),"r" (&err) - :"o2" ,"o3" ,"o4" ,"o5" ,"o7" ,"g2" ,"g3" ,"g4" ,"g5" , -"cc" ,"memory" ); -return err ?0 : (__force __wsum )ret; +return csum_partial_copy_nocheck ((__force void *)src, dst, len); }#define HAVE_CSUM_COPY_USER
@@ -91,29 +72,9 @@ csum_and_copy_from_user(const void __user *src, void *dst, int len)static inline __wsum csum_and_copy_to_user (const void *src,void __user *dst,int len) { - register unsigned long ret asm("o0") = (unsigned long)src; - register char __user *d asm("o1") = dst; - register int l asm("g1") = len; - register __wsum s asm("g7") = ~0U; -int err =0 ; -if (!access_ok (dst, len))return 0 ; - - __asm__ __volatile__ ( -".section __ex_table,#alloc\n\t" -".align 4\n\t" -".word 1f,1\n\t" -".previous\n" -"1:\n\t" -"call __csum_partial_copy_sparc_generic\n\t" -" st %8, [%%sp + 64]\n" - :"=&r" (ret),"=&r" (d),"=&r" (l),"=&r" (s) - :"0" (ret),"1" (d),"2" (l),"3" (s),"r" (&err) - :"o2" ,"o3" ,"o4" ,"o5" ,"o7" , -"g2" ,"g3" ,"g4" ,"g5" , -"cc" ,"memory" ); -return err ?0 : (__force __wsum )ret; +return csum_partial_copy_nocheck (src, (__force void *)dst, len); }/* ihl is always 5 or greater, almost always is 5, and iph is word aligned
arch/sparc/lib/checksum_32.S +38 -164
331 lines not on this page · raw message
arch/sparc/mm/fault_32.c +1 -5
@@ -297,8 +297,6 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,if (fixup >10 ) {extern const unsigned int __memset_start[];extern const unsigned int __memset_end[]; -extern const unsigned int __csum_partial_copy_start[]; -extern const unsigned int __csum_partial_copy_end[];#ifdef DEBUG_EXCEPTIONS printk("Exception: PC<%08lx> faddr<%08lx>\n",
@@ -307,9 +305,7 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write, regs->pc, fixup, g2);#endif if ((regs->pc >= (unsigned long )__memset_start && - regs->pc < (unsigned long )__memset_end) || - (regs->pc >= (unsigned long )__csum_partial_copy_start && - regs->pc < (unsigned long )__csum_partial_copy_end)) { + regs->pc < (unsigned long )__memset_end)) { regs->u_regs[UREG_I4 ] = address; regs->u_regs[UREG_I5 ] = regs->pc; }
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH v2 14/20 mips: csum_and_copy_{to,from}_user() are never called under KERNEL_DS
In reply to Al ViroFrom: Al Viro <[email protected]>
they are only called for iovec-backed iov_iter and under KERNEL_DS an attempt to create such a beast will yield a kvec-backed one.
--- arch/mips/include/asm/checksum.h | 32 +++++++------------------------- 1 file changed, 7 insertions(+), 25 deletions(-)
arch/mips/include/asm/checksum.h +7 -25
@@ -41,22 +41,6 @@ __wsum __csum_partial_copy_from_user(const void *src, void *dst, int len, __wsum sum, int *err_ptr);__wsum __csum_partial_copy_to_user (const void *src,void *dst,int len,__wsum sum,int *err_ptr); -/* -* this is a new version of the above that records errors it finds in *errp, -* but continues and zeros the rest of the buffer. -*/ -static inline -__wsum csum_partial_copy_from_user (const void __user *src,void *dst,int len, -__wsum sum,int *err_ptr) -{ -might_fault (); -if (uaccess_kernel ()) -return __csum_partial_copy_kernel ((__force void *)src, dst, - len, sum, err_ptr); -else -return __csum_partial_copy_from_user ((__force void *)src, dst, - len, sum, err_ptr); -}#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER static inline
@@ -65,9 +49,12 @@ __wsum csum_and_copy_from_user(const void __user *src, void *dst, int len)__wsum sum = ~0U ;int err =0 ; +might_fault (); +if (!access_ok (src, len))return 0 ; - sum =csum_partial_copy_from_user (src, dst, len, sum, &err); + sum =__csum_partial_copy_from_user ((__force void *)src, dst, + len, sum, &err);return err ?0 : sum; }
@@ -84,14 +71,9 @@ __wsum csum_and_copy_to_user(const void *src, void __user *dst, int len)might_fault ();if (!access_ok (dst, len))return 0 ; -if (uaccess_kernel ()) - sum =__csum_partial_copy_kernel (src, - (__force void *)dst, - len, sum, &err); -else - sum =__csum_partial_copy_to_user (src, - (__force void *)dst, - len, sum, &err); + sum =__csum_partial_copy_to_user (src, + (__force void *)dst, + len, sum, &err);return err ?0 : sum; }
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH v2 15/20 mips: __csum_partial_copy_kernel() has no users left
In reply to Al ViroFrom: Al Viro <[email protected]>
--- arch/mips/include/asm/checksum.h | 3 --- arch/mips/lib/csum_partial.S | 3 --- 2 files changed, 6 deletions(-)
arch/mips/include/asm/checksum.h +0 -3
@@ -34,9 +34,6 @@*/ __wsum csum_partial (const void *buff,int len,__wsum sum); -__wsum __csum_partial_copy_kernel (const void *src,void *dst, -int len,__wsum sum,int *err_ptr); -__wsum __csum_partial_copy_from_user (const void *src,void *dst,int len,__wsum sum,int *err_ptr); __wsum __csum_partial_copy_to_user(const void *src, void *dst,
arch/mips/lib/csum_partial.S +0 -3
@@ -827,8 +827,6 @@ EXPORT_SYMBOL(csum_partial) .set pop .endm -LEAF(__csum_partial_copy_kernel) -EXPORT_SYMBOL(__csum_partial_copy_kernel) #ifndef CONFIG_EVA FEXPORT(__csum_partial_copy_to_user) EXPORT_SYMBOL(__csum_partial_copy_to_user)
@@ -836,7 +834,6 @@ FEXPORT(__csum_partial_copy_from_user) EXPORT_SYMBOL(__csum_partial_copy_from_user) #endif __BUILD_CSUM_PARTIAL_COPY_USER LEGACY_MODE USEROP USEROP 1 -END(__csum_partial_copy_kernel) #ifdef CONFIG_EVA LEAF(__csum_partial_copy_to_user)
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH v2 16/20 mips: propagate the calling convention change down into __csum_partial_copy_..._user()
In reply to Al ViroFrom: Al Viro <[email protected]>
and turn the exception handlers into simply returning 0, which simplifies the hell out of things in csum_partial.S
--- arch/mips/include/asm/checksum.h | 26 +--- arch/mips/lib/csum_partial.S | 258 +++++++++++++-------------------------- 2 files changed, 89 insertions(+), 195 deletions(-)
arch/mips/include/asm/checksum.h +6 -20
@@ -34,25 +34,17 @@*/ __wsum csum_partial (const void *buff,int len,__wsum sum); -__wsum __csum_partial_copy_from_user (const void *src,void *dst, -int len,__wsum sum,int *err_ptr); -__wsum __csum_partial_copy_to_user (const void *src,void *dst, -int len,__wsum sum,int *err_ptr); +__wsum __csum_partial_copy_from_user (const void __user *src,void *dst,int len); +__wsum __csum_partial_copy_to_user (const void *src,void __user *dst,int len);#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USERstatic inline __wsum csum_and_copy_from_user (const void __user *src,void *dst,int len) { -__wsum sum = ~0U ; -int err =0 ; -might_fault (); -if (!access_ok (src, len))return 0 ; - sum =__csum_partial_copy_from_user ((__force void *)src, dst, - len, sum, &err); -return err ?0 : sum; +return __csum_partial_copy_from_user (src, dst, len); }/*
@@ -62,16 +54,10 @@ __wsum csum_and_copy_from_user(const void __user *src, void *dst, int len)static inline __wsum csum_and_copy_to_user (const void *src,void __user *dst,int len) { -int err =0 ; -__wsum sum = ~0U ; -might_fault ();if (!access_ok (dst, len))return 0 ; - sum =__csum_partial_copy_to_user (src, - (__force void *)dst, - len, sum, &err); -return err ?0 : sum; +return __csum_partial_copy_to_user (src, dst, len); }/*
@@ -79,10 +65,10 @@ __wsum csum_and_copy_to_user(const void *src, void __user *dst, int len)* we have just one address space, so this is identical to the above) */ #define _HAVE_ARCH_CSUM_AND_COPY -__wsum __csum_partial_copy_nocheck (const void *src,void *dst,int len,__wsum sum); +__wsum __csum_partial_copy_nocheck (const void *src,void *dst,int len);static inline __wsum csum_partial_copy_nocheck (const void *src,void *dst,int len) { -return __csum_partial_copy_nocheck (src, dst, len,0 ); +return __csum_partial_copy_nocheck (src, dst, len); }/*
arch/mips/lib/csum_partial.S +83 -175
418 lines not on this page · raw message
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH v2 17/20 xtensa: propagate the calling conventions change down into csum_partial_copy_generic()
In reply to Al ViroFrom: Al Viro <[email protected]>
turn the exception handlers into returning 0.
--- arch/xtensa/include/asm/checksum.h | 20 +++--------- arch/xtensa/lib/checksum.S | 67 +++++++++----------------------------- 2 files changed, 19 insertions(+), 68 deletions(-)
arch/xtensa/include/asm/checksum.h +4 -16
@@ -37,9 +37,7 @@ asmlinkage __wsum csum_partial(const void *buff, int len, __wsum sum);* better 64-bit) boundary */ -asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst, -int len,__wsum sum, -int *src_err_ptr,int *dst_err_ptr); +asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst, int len);#define _HAVE_ARCH_CSUM_AND_COPY/*
@@ -49,7 +47,7 @@ asmlinkage __wsum csum_partial_copy_generic(const void *src, void *dst,static inline __wsum csum_partial_copy_nocheck (const void *src,void *dst,int len) { -return csum_partial_copy_generic (src, dst, len,0 ,NULL ,NULL ); +return csum_partial_copy_generic (src, dst, len); }#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER
@@ -57,14 +55,9 @@ static inline__wsum csum_and_copy_from_user (const void __user *src,void *dst,int len) { -int err =0 ; -if (!access_ok (src, len))return 0 ; - - sum =csum_partial_copy_generic ((__force const void *)src, dst, - len, ~0U , &err,NULL ); -return err ?0 : sum; +return csum_partial_copy_generic ((__force const void *)src, dst, len); }/*
@@ -247,13 +240,8 @@ static __inline__ __sum16 csum_ipv6_magic(const struct in6_addr *saddr,static __inline____wsum csum_and_copy_to_user (const void *src,void __user *dst,int len) { -int err =0 ; -__wsum sum = ~0U ; -if (!access_ok (dst, len))return 0 ; - - sum =csum_partial_copy_generic (src,dst,len,sum,NULL ,&err); -return err ?0 : sum; +return csum_partial_copy_generic (src, (__force void *)dst, len); }#endif
arch/xtensa/lib/checksum.S +15 -52
135 lines not on this page · raw message
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH v2 18/20 sparc64: propagate the calling convention changes down to __csum_partial_copy_...()
In reply to Al ViroFrom: Al Viro <[email protected]>
... and rename them into csum_and_copy_...() - the wrappers become pointless. [braino fixed]
--- arch/sparc/include/asm/checksum.h | 1 + arch/sparc/include/asm/checksum_32.h | 2 -- arch/sparc/include/asm/checksum_64.h | 41 +++--------------------------------- arch/sparc/lib/csum_copy.S | 5 +++-- arch/sparc/lib/csum_copy_from_user.S | 4 ++-- arch/sparc/lib/csum_copy_to_user.S | 4 ++-- 6 files changed, 11 insertions(+), 46 deletions(-)
arch/sparc/include/asm/checksum.h +1 -0
@@ -3,6 +3,7 @@#define ___ASM_SPARC_CHECKSUM_H#define _HAVE_ARCH_CSUM_AND_COPY#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER +#define HAVE_CSUM_COPY_USER #if defined(__sparc__) && defined(__arch64__)#include <asm/checksum_64.h> #else
arch/sparc/include/asm/checksum_32.h +0 -2
@@ -67,8 +67,6 @@ csum_and_copy_from_user(const void __user *src, void *dst, int len)return csum_partial_copy_nocheck ((__force void *)src, dst, len); } -#define HAVE_CSUM_COPY_USER -static inline __wsum csum_and_copy_to_user (const void *src,void __user *dst,int len) {
arch/sparc/include/asm/checksum_64.h +3 -38
47 lines not on this page · raw message
arch/sparc/lib/csum_copy.S +3 -2
@@ -33,7 +33,7 @@ #endif #ifndef FUNC_NAME -#define FUNC_NAME __csum_partial_copy_nocheck +#define FUNC_NAME csum_partial_copy_nocheck #endif .register %g2, #scratch
@@ -68,9 +68,10 @@ .globl FUNC_NAME .type FUNC_NAME,#function EXPORT_SYMBOL(FUNC_NAME) -FUNC_NAME: /* %o0=src, %o1=dst, %o2=len, %o3=sum */ +FUNC_NAME: /* %o0=src, %o1=dst, %o2=len */ LOAD(prefetch, %o0 + 0x000, #n_reads) xor %o0, %o1, %g1 + mov 1, %o3 clr %o4 andcc %g1, 0x3, %g0 bne,pn %icc, 95f
arch/sparc/lib/csum_copy_from_user.S +2 -2
@@ -9,14 +9,14 @@ .section .fixup, "ax"; \ .align 4; \ 99: retl; \ - mov -1, %o0; \ + mov 0, %o0; \ .section __ex_table,"a";\ .align 4; \ .word 98b, 99b; \ .text; \ .align 4; -#define FUNC_NAME __csum_partial_copy_from_user +#define FUNC_NAME csum_and_copy_from_user #define LOAD(type,addr,dest) type##a [addr] %asi, dest #include "csum_copy.S"
arch/sparc/lib/csum_copy_to_user.S +2 -2
@@ -9,14 +9,14 @@ .section .fixup,"ax"; \ .align 4; \ 99: retl; \ - mov -1, %o0; \ + mov 0, %o0; \ .section __ex_table,"a";\ .align 4; \ .word 98b, 99b; \ .text; \ .align 4; -#define FUNC_NAME __csum_partial_copy_to_user +#define FUNC_NAME csum_and_copy_to_user #define STORE(type,src,addr) type##a src, [addr] %asi #include "csum_copy.S"
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH v2 19/20 amd64: switch csum_partial_copy_generic() to new calling conventions
In reply to Al ViroFrom: Al Viro <[email protected]>
... and fold handling of misaligned case into it.
Implementation note: we stash the "will we need to rol8 the sum in the end" flag into the MSB of %rcx (the lower 32 bits are used for length); the rest is pretty straightforward.
--- arch/x86/include/asm/checksum_64.h | 5 +- arch/x86/lib/csum-copy_64.S | 140 ++++++++++++++++++++++--------------- arch/x86/lib/csum-wrappers_64.c | 72 +++---------------- 3 files changed, 94 insertions(+), 123 deletions(-)
arch/x86/include/asm/checksum_64.h +1 -4
@@ -130,10 +130,7 @@ static inline __sum16 csum_tcpudp_magic(__be32 saddr, __be32 daddr,extern __wsum csum_partial (const void *buff,int len,__wsum sum);/* Do not call this directly. Use the wrappers below */ -extern __visible __wsum csum_partial_copy_generic(const void *src, const void *dst, -int len,__wsum sum, -int *src_err_ptr,int *dst_err_ptr); - +extern __visible __wsum csum_partial_copy_generic(const void *src, void *dst, int len);extern __wsum csum_and_copy_from_user (const void __user *src,void *dst,int len);extern __wsum csum_and_copy_to_user (const void *src,void __user *dst,int len);
arch/x86/lib/csum-copy_64.S +84 -56
192 lines not on this page · raw message
arch/x86/lib/csum-wrappers_64.c +9 -63
102 lines not on this page · raw message
Signature · 1 line
2.11.0
Al Viro
maintainerPATCH v2 20/20 ppc: propagate the calling conventions change down to csum_partial_copy_generic()
In reply to Al ViroFrom: Al Viro <[email protected]>
... and get rid of the pointless fallback in the wrappers. On error it used to zero the unwritten area and calculate the csum of the entire thing. Not wanting to do it in assembler part had been very reasonable; doing that in the first place, OTOH... In case of an error the caller discards the data we'd copied, along with whatever checksum it might've had.
--- arch/powerpc/include/asm/checksum.h | 6 +-- arch/powerpc/lib/checksum_32.S | 74 +++++++++++++----------------------- arch/powerpc/lib/checksum_64.S | 37 ++++++------------ arch/powerpc/lib/checksum_wrappers.c | 32 +++------------- 4 files changed, 46 insertions(+), 103 deletions(-)
arch/powerpc/include/asm/checksum.h +2 -4
@@ -18,9 +18,7 @@* Like csum_partial, this must be called with even lengths, * except for the last fragment. */ -extern __wsum csum_partial_copy_generic (const void *src,void *dst, -int len,__wsum sum, -int *src_err,int *dst_err); +extern __wsum csum_partial_copy_generic (const void *src,void *dst,int len);#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER extern __wsum csum_and_copy_from_user(const void __user *src, void *dst,
@@ -31,7 +29,7 @@ extern __wsum csum_and_copy_to_user(const void *src, void __user *dst,#define _HAVE_ARCH_CSUM_AND_COPY#define csum_partial_copy_nocheck (src, dst, len) \ - csum_partial_copy_generic((src), (dst), (len), 0, NULL, NULL) + csum_partial_copy_generic((src), (dst), (len))/*
arch/powerpc/lib/checksum_32.S +27 -47
110 lines not on this page · raw message
arch/powerpc/lib/checksum_64.S +11 -26
@@ -182,34 +182,33 @@ EXPORT_SYMBOL(__csum_partial) .macro srcnr 100: - EX_TABLE(100b,.Lsrc_error_nr) + EX_TABLE(100b,.Lerror_nr) .endm .macro source 150: - EX_TABLE(150b,.Lsrc_error) + EX_TABLE(150b,.Lerror) .endm .macro dstnr 200: - EX_TABLE(200b,.Ldest_error_nr) + EX_TABLE(200b,.Lerror_nr) .endm .macro dest 250: - EX_TABLE(250b,.Ldest_error) + EX_TABLE(250b,.Lerror) .endm /* * Computes the checksum of a memory block at src, length len, - * and adds in "sum" (32-bit), while copying the block to dst. - * If an access exception occurs on src or dst, it stores -EFAULT - * to *src_err or *dst_err respectively. The caller must take any action - * required in this case (zeroing memory, recalculating partial checksum etc). + * and adds in 0xffffffff (32-bit), while copying the block to dst. + * If an access exception occurs, it returns 0. * - * csum_partial_copy_generic(r3=src, r4=dst, r5=len, r6=sum, r7=src_err, r8=dst_err) + * csum_partial_copy_generic(r3=src, r4=dst, r5=len) */ _GLOBAL(csum_partial_copy_generic) + li r6,-1 addic r0,r6,0 /* clear carry */ srdi. r6,r5,3 /* less than 8 bytes? */
@@ -401,29 +400,15 @@ dstnr; stb r6,0(r4) srdi r3,r3,32 blr -.Lsrc_error: +.Lerror: ld r14,STK_REG(R14)(r1) ld r15,STK_REG(R15)(r1) ld r16,STK_REG(R16)(r1) addi r1,r1,STACKFRAMESIZE -.Lsrc_error_nr: - cmpdi 0,r7,0 - beqlr - li r6,-EFAULT - stw r6,0(r7) +.Lerror_nr: + li r3,0 blr -.Ldest_error: - ld r14,STK_REG(R14)(r1) - ld r15,STK_REG(R15)(r1) - ld r16,STK_REG(R16)(r1) - addi r1,r1,STACKFRAMESIZE -.Ldest_error_nr: - cmpdi 0,r8,0 - beqlr - li r6,-EFAULT - stw r6,0(r8) - blr EXPORT_SYMBOL(csum_partial_copy_generic) /*
arch/powerpc/lib/checksum_wrappers.c +6 -26
@@ -14,8 +14,7 @@__wsum csum_and_copy_from_user (const void __user *src,void *dst,int len) { -unsigned int csum; -int err =0 ; +__wsum csum;might_sleep ();
@@ -24,27 +23,16 @@ __wsum csum_and_copy_from_user(const void __user *src, void *dst,allow_read_from_user (src, len); - csum =csum_partial_copy_generic ((void __force *)src, dst, - len, ~0U , &err,NULL ); - -if (unlikely (err)) { -int missing =__copy_from_user (dst, src, len); - -if (missing) - csum =0 ; -else - csum =csum_partial (dst, len, ~0U ); - } + csum =csum_partial_copy_generic ((void __force *)src, dst, len);prevent_read_from_user (src, len); -return (__force __wsum )csum; +return csum; }EXPORT_SYMBOL (csum_and_copy_from_user);__wsum csum_and_copy_to_user (const void *src,void __user *dst,int len) { -unsigned int csum; -int err =0 ; +__wsum csum;might_sleep ();if (unlikely (!access_ok (dst, len)))
@@ -52,17 +40,9 @@ __wsum csum_and_copy_to_user(const void *src, void __user *dst, int len)allow_write_to_user (dst, len); - csum =csum_partial_copy_generic (src, (void __force *)dst, - len, ~0U ,NULL , &err); - -if (unlikely (err)) { - csum =csum_partial (src, len, ~0U ); - -if (copy_to_user (dst, src, len)) - csum =0 ; - } + csum =csum_partial_copy_generic (src, (void __force *)dst, len);prevent_write_to_user (dst, len); -return (__force __wsum )csum; +return csum; }EXPORT_SYMBOL (csum_and_copy_to_user);
Signature · 1 line
2.11.0
Re: [PATCH v2 20/20] ppc: propagate the calling conventions change down to csum_partial_copy_generic()
Hi Al,
Quoted 7 lines · Al Viro
From: Al Viro <[email protected]>
... and get rid of the pointless fallback in the wrappers. On error it used to zero the unwritten area and calculate the csum of the entire thing. Not wanting to do it in assembler part had been very reasonable; doing that in the first place, OTOH... In case of an error the caller discards the data we'd copied, along with whatever checksum it might've had.
This patch is causing crashes in WireGuard's CI over at
https://www.wireguard.com/build-status/ . Apparently sending a simple network packet winds up triggering refcount_t's warn-on-saturate code. I don't know if the new assembly failed to reset some flag or if something else is up. I can start digging into it if you want, but I thought I should let you know first about the issue. The splat follows below.
Thanks,
Jason
$ ping -c 10 -f -W 1 192.168.241.1 PING 192.168.241.1 (192.168.241.1) 56(84) bytes of data. [ 1.432922] ------------[ cut here ]------------ [ 1.433069] refcount_t: saturated; leaking memory. [ 1.433344] WARNING: CPU: 3 PID: 90 at refcount_warn_saturate+0x100/0x1bc [ 1.433646] CPU: 3 PID: 90 Comm: ping Not tainted 5.9.0+ #3 [ 1.433797] NIP: c01a6fa0 LR: c01a6fa0 CTR: c01ccbec [ 1.433964] REGS: cfacfb80 TRAP: 0700 Not tainted (5.9.0+) [ 1.434102] MSR: 00029000 <CE,EE,ME> CR: 28022404 XER: 00000000 [ 1.434345] [ 1.434345] GPR00: c01a6fa0 cfacfc38 cf8eeae0 00000026 3fffefff cfacfa90 cfacfaa0 00021000 [ 1.434345] GPR08: 0f4a1000 00000000 c08b4674 c0918704 42022404 00000000 cfa34180 00000000 [ 1.434345] GPR16: 00000000 cf8ef004 00000000 00000000 00000040 00000000 00000000 cfbac230 [ 1.434345] GPR24: cfacfce8 c02a802c 00000000 cfa34180 cfacfc58 c02aa53c 55c0a4ff 00000000 [ 1.435471] NIP [c01a6fa0] refcount_warn_saturate+0x100/0x1bc [ 1.435615] LR [c01a6fa0] refcount_warn_saturate+0x100/0x1bc [ 1.435825] Call Trace: [ 1.435922] [cfacfc38] [c01a6fa0] refcount_warn_saturate+0x100/0x1bc (unreliable) [ 1.436149] [cfacfc48] [c02a7f14] __ip_append_data.isra.0+0x8a8/0xde0 [ 1.436302] [cfacfce8] [c02a84e0] ip_append_data.part.0+0x94/0xf0 [ 1.436438] [cfacfd18] [c02dffe0] raw_sendmsg+0x298/0xa84 [ 1.436544] [cfacfe48] [c020b9ec] __sys_sendto+0xdc/0x13c [ 1.436641] [cfacff38] [c000f1dc] ret_from_syscall+0x0/0x38 [ 1.436824] --- interrupt: c01 at 0xb7e44f00 [ 1.436824] LR = 0xb7e21ba0 [ 1.437038] Instruction dump: [ 1.437239] 3d20c092 39291bc1 89490001 2c0a0000 4082ff64 3c60c040 7c0802a6 39400001 [ 1.437439] 38633b74 90010014 99490001 4be9b6e1 <0fe00000> 80010014 7c0803a6 4bffff38 [ 1.437753] ---[ end trace aaa4b4788958d0a6 ]--- [ 1.440214] ------------[ cut here ]------------ [ 1.440301] refcount_t: underflow; use-after-free. [ 1.440397] WARNING: CPU: 3 PID: 90 at refcount_warn_saturate+0x1ac/0x1bc [ 1.440587] CPU: 3 PID: 90 Comm: ping Tainted: G W 5.9.0+ #3 [ 1.440741] NIP: c01a704c LR: c01a704c CTR: c01ccbec [ 1.440857] REGS: cfacfaa0 TRAP: 0700 Tainted: G W (5.9.0+) [ 1.441016] MSR: 00029000 <CE,EE,ME> CR: 48022404 XER: 00000000 [ 1.441176] [ 1.441176] GPR00: c01a704c cfacfb58 cf8eeae0 00000026 3fffefff cfacf9b0 cfacf9c0 00021000 [ 1.441176] GPR08: 0f4a1000 00000400 c08b4674 c0918704 42022404 00000000 10020464 00000003 [ 1.441176] GPR16: 7ff00000 10020000 00000080 cfb27000 cfb2704c c0930000 cfacfc54 c092d260 [ 1.441176] GPR24: 0000058c cfa82120 cfa8212c cfa8212c 00000000 cfa82000 cfacfd44 cfacfc58 [ 1.441995] NIP [c01a704c] refcount_warn_saturate+0x1ac/0x1bc [ 1.442125] LR [c01a704c] refcount_warn_saturate+0x1ac/0x1bc [ 1.442252] Call Trace: [ 1.442320] [cfacfb58] [c01a704c] refcount_warn_saturate+0x1ac/0x1bc (unreliable) [ 1.442726] [cfacfb68] [c020e7dc] sock_wfree+0x130/0x134 [ 1.442877] [cfacfb78] [c01f1388] wg_packet_send_staged_packets+0x234/0x6b4 [ 1.443061] [cfacfbb8] [c01eecf8] wg_xmit+0x2a0/0x46c [ 1.443204] [cfacfbf8] [c0232134] dev_hard_start_xmit+0x190/0x1c0 [ 1.443369] [cfacfc38] [c0232f2c] __dev_queue_xmit+0x4d0/0x844 [ 1.443527] [cfacfc88] [c02a7134] ip_finish_output2+0x180/0x6b8 [ 1.443686] [cfacfcb8] [c02aa3e8] ip_output+0xf0/0x1c0 [ 1.443829] [cfacfd08] [c02ab14c] ip_send_skb+0x24/0xe8 [ 1.443975] [cfacfd18] [c02e04bc] raw_sendmsg+0x774/0xa84 [ 1.444124] [cfacfe48] [c020b9ec] __sys_sendto+0xdc/0x13c [ 1.444274] [cfacff38] [c000f1dc] ret_from_syscall+0x0/0x38 [ 1.444437] --- interrupt: c01 at 0xb7e44f00 [ 1.444437] LR = 0xb7e21ba0 [ 1.444644] Instruction dump: [ 1.444736] 4be9b661 0fe00000 80010014 7c0803a6 4bfffeb8 3c60c040 7c0802a6 39400001 [ 1.444989] 38633bd8 90010014 99490003 4be9b635 <0fe00000> 80010014 7c0803a6 4bfffe8c [ 1.445252] ---[ end trace aaa4b4788958d0a7 ]--- [ 1.445583] BUG: Unable to handle kernel instruction fetch (NULL pointer?) [ 1.445767] Faulting instruction address: 0x00000000 [ 1.446051] Oops: Kernel access of bad area, sig: 11 [#1] [ 1.446210] BE PAGE_SIZE=4K PREEMPT SMP NR_CPUS=4 QEMU e500 [ 1.446379] CPU: 3 PID: 90 Comm: ping Tainted: G W 5.9.0+ #3 [ 1.446678] NIP: 00000000 LR: c020e758 CTR: 00000000 [ 1.446812] REGS: cfacfab0 TRAP: 0400 Tainted: G W (5.9.0+) [ 1.446989] MSR: 00029000 <CE,EE,ME> CR: 48022404 XER: 00000000 [ 1.447183] [ 1.447183] GPR00: c020e7dc cfacfb68 cf8eeae0 cfacfc58 3fffefff cfacf9b0 cfacf9c0 00021000 [ 1.447183] GPR08: 0f4a1000 00000000 c08b4674 c0918704 42022404 00000000 10020464 00000003 [ 1.447183] GPR16: 7ff00000 10020000 00000080 cfb27000 cfb2704c c0930000 cfacfc54 c092d260 [ 1.447183] GPR24: 0000058c cfa82120 cfa8212c cfa8212c 00000000 cfa82000 cfacfd44 cfacfc58 [ 1.448144] NIP [00000000] 0x0 [ 1.448236] LR [c020e758] sock_wfree+0xac/0x134 [ 1.448351] Call Trace: [ 1.448425] [cfacfb68] [c020e7dc] sock_wfree+0x130/0x134 (unreliable) [ 1.448603] [cfacfb78] [c01f1388] wg_packet_send_staged_packets+0x234/0x6b4 [ 1.448820] [cfacfbb8] [c01eecf8] wg_xmit+0x2a0/0x46c [ 1.448964] [cfacfbf8] [c0232134] dev_hard_start_xmit+0x190/0x1c0 [ 1.449139] [cfacfc38] [c0232f2c] __dev_queue_xmit+0x4d0/0x844 [ 1.449304] [cfacfc88] [c02a7134] ip_finish_output2+0x180/0x6b8 [ 1.449475] [cfacfcb8] [c02aa3e8] ip_output+0xf0/0x1c0 [ 1.449628] [cfacfd08] [c02ab14c] ip_send_skb+0x24/0xe8 [ 1.449815] [cfacfd18] [c02e04bc] raw_sendmsg+0x774/0xa84 [ 1.449983] [cfacfe48] [c020b9ec] __sys_sendto+0xdc/0x13c [ 1.450150] [cfacff38] [c000f1dc] ret_from_syscall+0x0/0x38 [ 1.450320] --- interrupt: c01 at 0xb7e44f00 [ 1.450320] LR = 0xb7e21ba0 [ 1.450794] Instruction dump: [ 1.450963] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX [ 1.451209] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX [ 1.451637] ---[ end trace aaa4b4788958d0a8 ]--- [ 1.451785] [ 2.555288] Kernel panic - not syncing: Aiee, killing interrupt handler!
Re: [PATCH v2 20/20] ppc: propagate the calling conventions change down to csum_partial_copy_generic()
Quoted 4 lines · Jason A. Donenfeld
This patch is causing crashes in WireGuard's CI over at
https://www.wireguard.com/build-status/ . Apparently sending a simple network packet winds up triggering refcount_t's warn-on-saturate code. I
Ouch.
The C parts look fairly straightforward, and I don't see how they could cause that odd refcount issue.
So I assume it's the low-level asm code conversion that is buggy. And it's apparently the 32-bit conversion, since your ppc64 status looks fine.
I think it's this instruction:
addi r1,r1,16
that should be removed from the function exit, because Al removed the
- stwu r1,-16(r1)
on function entry.
So I think you end up with a corrupt stack pointer and basically random behavior.
Mind trying that? (This is obviously all in arch/powerpc/lib/checksum_32.S, the csum_partial_copy_generic() function).
Linus
Re: [PATCH v2 20/20] ppc: propagate the calling conventions change down to csum_partial_copy_generic()
Quoted 16 lines · Linus Torvalds
Quoted 4 lines · Jason A. Donenfeld
This patch is causing crashes in WireGuard's CI over at
https://www.wireguard.com/build-status/ . Apparently sending a simple network packet winds up triggering refcount_t's warn-on-saturate code. IOuch.
The C parts look fairly straightforward, and I don't see how they could cause that odd refcount issue.
So I assume it's the low-level asm code conversion that is buggy. And it's apparently the 32-bit conversion, since your ppc64 status looks fine.
Quoted 7 lines
I think it's this instruction:
addi r1,r1,16that should be removed from the function exit, because Al removed the
- stwu r1,-16(r1)
I just tried that about a minute ago, and indeed that seems to be what's up. Problem goes away without it. I'll send a patch shortly.
Jason
Re: [PATCH v2 20/20] ppc: propagate the calling conventions change down to csum_partial_copy_generic()
In reply to Linus TorvaldsQuoted 9 lines · Linus Torvalds
I think it's this instruction:
addi r1,r1,16that should be removed from the function exit, because Al removed the
- stwu r1,-16(r1)
Quoted 8 lines
on function entry.
So I think you end up with a corrupt stack pointer and basically random behavior.
Mind trying that? (This is obviously all in
arch/powerpc/lib/checksum_32.S, thecsum_partial_copy_generic()function).
Patch attached to make it easier to test.
NOTE! This is ENTIRELY untested. My ppc asm is so rusty that I might be barking entirely up the wrong tree, and I just made things much worse.
Linus
Re: [PATCH v2 20/20] ppc: propagate the calling conventions change down to csum_partial_copy_generic()
Quoted 21 lines · Linus Torvalds
Quoted 9 lines · Linus Torvalds
I think it's this instruction:
addi r1,r1,16that should be removed from the function exit, because Al removed the
- stwu r1,-16(r1)
Quoted 8 lines
on function entry.
So I think you end up with a corrupt stack pointer and basically random behavior.
Mind trying that? (This is obviously all in
arch/powerpc/lib/checksum_32.S, thecsum_partial_copy_generic()function).
Quoted 5 lines
Patch attached to make it easier to test.
NOTE! This is ENTIRELY untested. My ppc asm is so rusty that I might be barking entirely up the wrong tree, and I just made things much worse.
Indeed - exactly the same thing that's in my tree. Writing a commit message for it now.
Jason
Al Viro
maintainerRe: [PATCH v2 20/20] ppc: propagate the calling conventions change down to csum_partial_copy_generic()
In reply to Linus TorvaldsQuoted 19 lines · Linus Torvalds
Quoted 4 lines · Jason A. Donenfeld
This patch is causing crashes in WireGuard's CI over at
https://www.wireguard.com/build-status/ . Apparently sending a simple network packet winds up triggering refcount_t's warn-on-saturate code. IOuch.
The C parts look fairly straightforward, and I don't see how they could cause that odd refcount issue.
So I assume it's the low-level asm code conversion that is buggy. And it's apparently the 32-bit conversion, since your ppc64 status looks fine.
I think it's this instruction:
addi r1,r1,16
Quoted 8 lines
that should be removed from the function exit, because Al removed the
- stwu r1,-16(r1)
on function entry.
So I think you end up with a corrupt stack pointer and basically random behavior.
Gyahh... ACK, and I really wonder how the hell has it managed to avoid crashing on testing.
Mea culpa, folks.
PATCH 1/1 powerpc32: don't adjust unmoved stack pointer in csum_partial_copy_generic() epilogue
In reply to Jason A. DonenfeldA recent change to the checksum code removed usage of some extra arguments, alongside with storage on the stack for those, and the stack pointer no longer needed to be adjusted in the function prologue. But, a left over subtraction wasn't removed in the function epilogue, causing the function to return with the stack pointer moved 16 bytes away from where it should have. This corrupted local state and lead to weird crashes. This commit simply removes the leftover instruction from the epilogue.
--- arch/powerpc/lib/checksum_32.S | 1 - 1 file changed, 1 deletion(-)
arch/powerpc/lib/checksum_32.S +0 -1
@@ -236,7 +236,6 @@ _GLOBAL(csum_partial_copy_generic) slwi r0,r0,8 adde r12,r12,r0 66: addze r3,r12 - addi r1,r1,16 beqlr+ cr7 rlwinm r3,r3,8,0,31 /* odd destination address: rotate one byte */ blr
Signature · 1 line
2.28.0
Re: [PATCH] powerpc32: don't adjust unmoved stack pointer in csum_partial_copy_generic() epilogue
Thanks - applied and pushed out.
Linus