Skip to contentSkip to conversation

x86/copy_user.S: Remove zero byte check before copy user buffer.

PATCH In review Fenghua Yu 5 participants 12 messages Nov 16, 2013 – Nov 20, 2013 1 message not in the archive x86

lore
Share & tools
lore

Fenghua Yu

author
#0 lore raw

PATCH 1/1 x86/copy_user.S: Remove zero byte check before copy user buffer.

From: Fenghua Yu <[email protected]>

Operation of rep movsb instruction handles zero byte copy. As pointed out by Linus, there is no need to check zero size in kernel. Removing this redundant check saves a few cycles in copy user functions.

---
 arch/x86/lib/copy_user_64.S | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)
arch/x86/lib/copy_user_64.S +2 -6
index a30ca15..ffe4eb9 100644
@@ -236,8 +236,6 @@ ENDPROC(copy_user_generic_unrolled)
 ENTRY(copy_user_generic_string)
 	CFI_STARTPROC
 	ASM_STAC
-	andl %edx,%edx
-	jz 4f
 	cmpl $8,%edx
 	jb 2f		/* less than 8 bytes, go to byte copy loop */
 	ALIGN_DESTINATION
@@ -249,7 +247,7 @@ ENTRY(copy_user_generic_string)
 2:	movl %edx,%ecx
 3:	rep
 	movsb
-4:	xorl %eax,%eax
+	xorl %eax,%eax
 	ASM_CLAC
 	ret
 
@@ -279,12 +277,10 @@ ENDPROC(copy_user_generic_string)
 ENTRY(copy_user_enhanced_fast_string)
 	CFI_STARTPROC
 	ASM_STAC
-	andl %edx,%edx
-	jz 2f
 	movl %edx,%ecx
 1:	rep
 	movsb
-2:	xorl %eax,%eax
+	xorl %eax,%eax
 	ASM_CLAC
 	ret
 
Signature · 1 line
1.8.0.1
#1 lore raw

[tip:x86/asm] x86-64, copy_user: Remove zero byte check before copy user buffer.

x86-64, copy_user: Remove zero byte check before copy user buffer.
Commit-ID:  f4cb1cc18f364d761d5614eb6293cccc6647f259
Gitweb:     http://git.kernel.org/tip/f4cb1cc18f364d761d5614eb6293cccc6647f259
Author:     Fenghua Yu <[email protected]>
AuthorDate: Sat, 16 Nov 2013 12:37:01 -0800
Committer:  H. Peter Anvin <[email protected]>
CommitDate: Sat, 16 Nov 2013 18:00:58 -0800

x86-64, copy_user: Remove zero byte check before copy user buffer.

Operation of rep movsb instruction handles zero byte copy. As pointed out by Linus, there is no need to check zero size in kernel. Removing this redundant check saves a few cycles in copy user functions.

---
 arch/x86/lib/copy_user_64.S | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)
arch/x86/lib/copy_user_64.S +2 -6
index a30ca15..ffe4eb9 100644
@@ -236,8 +236,6 @@ ENDPROC(copy_user_generic_unrolled)
 ENTRY(copy_user_generic_string)
 	CFI_STARTPROC
 	ASM_STAC
-	andl %edx,%edx
-	jz 4f
 	cmpl $8,%edx
 	jb 2f		/* less than 8 bytes, go to byte copy loop */
 	ALIGN_DESTINATION
@@ -249,7 +247,7 @@ ENTRY(copy_user_generic_string)
 2:	movl %edx,%ecx
 3:	rep
 	movsb
-4:	xorl %eax,%eax
+	xorl %eax,%eax
 	ASM_CLAC
 	ret
 
@@ -279,12 +277,10 @@ ENDPROC(copy_user_generic_string)
 ENTRY(copy_user_enhanced_fast_string)
 	CFI_STARTPROC
 	ASM_STAC
-	andl %edx,%edx
-	jz 2f
 	movl %edx,%ecx
 1:	rep
 	movsb
-2:	xorl %eax,%eax
+	xorl %eax,%eax
 	ASM_CLAC
 	ret
 

A message that is not in the archive (sent off-list or not yet received): CA+55aFyFz_6oy8-1jb5Jzk+4VC5MLA5d0KbqhZxki0=+DmggBg@mail.gmail.com

#3 lore raw

Re: [tip:x86/asm] x86-64, copy_user: Remove zero byte check before copy user buffer.

Quoted 4 lines · Linus Torvalds

So this doesn't do the 32-bit truncation in the error path of the generic string copy. Oversight?

Linus

Indeed... although in the kernel it seems to be taken as an invariant that copy lengths over 4G is simply prohibited. There are places all over the kernel which will fail in a massive way if we ever ended up with a copy over 4G in size.

As such, I would argue the code with the patch is actually no more broken than with the truncation in place; if anything it is *more* correct than the modified one, since for a (very small) subset of >=4G copies it will actually do the right thing, albeit slowly.

The truncations do make me twitch a little inside, I have to admit.

-hpa

#4 lore raw

Re: [tip:x86/asm] x86-64, copy_user: Remove zero byte check before copy user buffer.

In reply to a message not in the archive
Quoted 4 lines · Linus Torvalds

So this doesn't do the 32-bit truncation in the error path of the generic string copy. Oversight?

Linus

Hi Linus,

Do you have a preference:

1. Considering the 32-bit truncation incidental (take it or leave it);
2. Require the 32-bit truncation, or
3. Get rid of it completely?

-hpa

#5 lore raw

Re: [tip:x86/asm] x86-64, copy_user: Remove zero byte check before copy user buffer.

Quoted 6 lines · H. Peter Anvin

Do you have a preference:

1. Considering the 32-bit truncation incidental (take it or leave it);
2. Require the 32-bit truncation, or
3. Get rid of it completely?

I don't have a huge preference, but I hate the current situation (with Fenghua's patch) where it's not consistent. One path uses just 32-bits of the count (thanks to the "mov %edx,%ecx") while another path uses 64 bits.

One or the other, but not a mixture of both.

And only tangentially related to this: I do think that we could be stricter about the count. Make it oops if the high bits are set, rather than overwrite a lot of memory. So I would not be adverse to limiting the count to 31 bits (or even less) explicitly, and thus making the while 32-vs-64 bit issue moot.

Linus

#6 lore raw

Re: [tip:x86/asm] x86-64, copy_user: Remove zero byte check before copy user buffer.

Quoted 13 lines · Linus Torvalds
Quoted 6 lines · H. Peter Anvin

Do you have a preference:

1. Considering the 32-bit truncation incidental (take it or leave it);
2. Require the 32-bit truncation, or
3. Get rid of it completely?

I don't have a huge preference, but I hate the current situation (with Fenghua's patch) where it's not consistent. One path uses just 32-bits of the count (thanks to the "mov %edx,%ecx") while another path uses 64 bits.

Quoted 8 lines

One or the other, but not a mixture of both.

And only tangentially related to this: I do think that we could be stricter about the count. Make it oops if the high bits are set, rather than overwrite a lot of memory. So I would not be adverse to limiting the count to 31 bits (or even less) explicitly, and thus making the while 32-vs-64 bit issue moot.

I guess the question is if we want to spend the extra cycles on a test and branch. For now I suggest that we just put back the truncation in the form of a movl instruction.

-hpa

#7 lore raw

Re: [tip:x86/asm] x86-64, copy_user: Remove zero byte check before copy user buffer.

In reply to a message not in the archive
Quoted 4 lines · Linus Torvalds

So this doesn't do the 32-bit truncation in the error path of the generic string copy. Oversight?

Linus

I looked at the code again, and it turns out to be false alarm.

We *do* do 32-bit truncation in every path, still:

Quoted 5 lines

ENTRY(copy_user_generic_string)

	CFI_STARTPROC
	ASM_STAC
	cmpl $8,%edx
	jb 2f		/* less than 8 bytes, go to byte copy loop */

-> If we jump here, we will truncate at 2:

Quoted 2 lines

ALIGN_DESTINATION
movl %edx,%ecx

-> If we don't jb 2f we end up

Quoted 1 line
	shrl $3,%ecx

32-bit truncation here...

Quoted 4 lines
	andl $7,%edx

1: rep
movsq
2: movl %edx,%ecx

32-bit truncation here...

Quoted 9 lines

3: rep
movsb
xorl %eax,%eax
ASM_CLAC
ret

.section .fixup,"ax"
11: lea (%rdx,%rcx,8),%rcx
12: movl %ecx,%edx /* ecx is zerorest also */

-> Even if %rdx+%rcx*8 > 2^32 we end up truncating at 12: -- not that it matters, since both arguments are prototyped as "unsigned" and therefore the C compiler is supposed to guarantee the upper 32 bits are ignored.

So I think Fenghua's patch is fine as-is.

-hpa

#8 lore raw

Re: [tip:x86/asm] x86-64, copy_user: Remove zero byte check before copy user buffer.

Quoted 8 lines · H. Peter Anvin
Quoted 4 lines

.section .fixup,"ax"
11: lea (%rdx,%rcx,8),%rcx
12: movl %ecx,%edx /* ecx is zerorest also */

-> Even if %rdx+%rcx*8 > 2^32 we end up truncating at 12: -- not that it matters, since both arguments are prototyped as "unsigned" and therefore the C compiler is supposed to guarantee the upper 32 bits are ignored.

Ahh. That was the one I thought was broken, but yes, while the upper bits of %rcx are calculated and not zeroed, they end up not actually getting used. So yeah, I'll believe it's correct.

Linus

#9 lore raw

Re: [tip:x86/asm] x86-64, copy_user: Remove zero byte check before copy user buffer.

Quoted 6 lines · Linus Torvalds
Quoted 5 lines · H. Peter Anvin
Quoted 4 lines

.section .fixup,"ax"
11: lea (%rdx,%rcx,8),%rcx
12: movl %ecx,%edx /* ecx is zerorest also */

Quoted 8 lines
Quoted 3 lines

-> Even if %rdx+%rcx*8 > 2^32 we end up truncating at 12: -- not that it matters, since both arguments are prototyped as "unsigned" and therefore the C compiler is supposed to guarantee the upper 32 bits are ignored.

Ahh. That was the one I thought was broken, but yes, while the upper bits of %rcx are calculated and not zeroed, they end up not actually getting used. So yeah, I'll believe it's correct.

That being said, "lea (%rdx,%rcx,8),%ecx" (leal, as opposed to leaq) is a perfectly legitimate instruction and actually one byte shorter. The big question is if some broken version of gas will choke on it.

-hpa

#10 lore raw

Re: [tip:x86/asm] x86-64, copy_user: Remove zero byte check before copy user buffer.

Quoted 4 lines · H. Peter Anvin

That being said, "lea (%rdx,%rcx,8),%ecx" (leal, as opposed to leaq) is a perfectly legitimate instruction and actually one byte shorter. The big question is if some broken version of gas will choke on it.

At least gcc-4.8.2 generates that instruction (try multiplying an integer value by 9), so I guess gas will be happy. Except gcc uses "leal", so to be safe..

Linus

#11 lore raw

[tip:x86/asm] x86-64, copy_user: Use leal to produce 32-bit results

In reply to Fenghua Yu
x86-64, copy_user: Use leal to produce 32-bit results
Commit-ID:  372474e12a858807f03f73cb30e830a76fd1ae07
Gitweb:     http://git.kernel.org/tip/372474e12a858807f03f73cb30e830a76fd1ae07
Author:     H. Peter Anvin <[email protected]>
AuthorDate: Wed, 20 Nov 2013 12:50:51 -0800
Committer:  H. Peter Anvin <[email protected]>
CommitDate: Wed, 20 Nov 2013 12:50:51 -0800

x86-64, copy_user: Use leal to produce 32-bit results

When we are using lea to produce a 32-bit result, we can use the leal form, rather than using leaq and worry about truncation elsewhere.

---
 arch/x86/lib/copy_user_64.S | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/lib/copy_user_64.S +2 -2
index ffe4eb9..f00a87c 100644
@@ -186,7 +186,7 @@ ENTRY(copy_user_generic_unrolled)
 30:	shll $6,%ecx
 	addl %ecx,%edx
 	jmp 60f
-40:	lea (%rdx,%rcx,8),%rdx
+40:	lea (%rdx,%rcx,8),%edx
 	jmp 60f
 50:	movl %ecx,%edx
 60:	jmp copy_user_handle_tail /* ecx is zerorest also */
@@ -252,7 +252,7 @@ ENTRY(copy_user_generic_string)
 	ret
 
 	.section .fixup,"ax"
-11:	lea (%rdx,%rcx,8),%rcx
+11:	lea (%rdx,%rcx,8),%ecx
 12:	movl %ecx,%edx		/* ecx is zerorest also */
 	jmp copy_user_handle_tail
 	.previous
#12 lore raw

[tip:x86/asm] x86-64, copy_user: Use leal to produce 32-bit results

In reply to Fenghua Yu
x86-64, copy_user: Use leal to produce 32-bit results
Commit-ID:  661c80192d21269c7fc566f1d547510b0c867677
Gitweb:     http://git.kernel.org/tip/661c80192d21269c7fc566f1d547510b0c867677
Author:     H. Peter Anvin <[email protected]>
AuthorDate: Wed, 20 Nov 2013 12:50:51 -0800
Committer:  H. Peter Anvin <[email protected]>
CommitDate: Wed, 20 Nov 2013 13:57:07 -0800

x86-64, copy_user: Use leal to produce 32-bit results

When we are using lea to produce a 32-bit result, we can use the leal form, rather than using leaq and worry about truncation elsewhere.

Make the leal explicit, both to be more obvious and since that is what gcc generates and thus is less likely to trigger obscure gas bugs.

---
 arch/x86/lib/copy_user_64.S | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/lib/copy_user_64.S +2 -2
index ffe4eb9..dee945d 100644
@@ -186,7 +186,7 @@ ENTRY(copy_user_generic_unrolled)
 30:	shll $6,%ecx
 	addl %ecx,%edx
 	jmp 60f
-40:	lea (%rdx,%rcx,8),%rdx
+40:	leal (%rdx,%rcx,8),%edx
 	jmp 60f
 50:	movl %ecx,%edx
 60:	jmp copy_user_handle_tail /* ecx is zerorest also */
@@ -252,7 +252,7 @@ ENTRY(copy_user_generic_string)
 	ret
 
 	.section .fixup,"ax"
-11:	lea (%rdx,%rcx,8),%rcx
+11:	leal (%rdx,%rcx,8),%ecx
 12:	movl %ecx,%edx		/* ecx is zerorest also */
 	jmp copy_user_handle_tail
 	.previous