Patch "x86/uaccess/64: Handle the caching of 4-byte nocache copies properly in __copy_user_nocache()" has been added to the 4.4-stable tree

PATCH [email protected] 1 participant 1 message Feb 24, 2016

Subsystems (2)

Stable branch x86

lore

Share & tools

lore

[email protected]

author maintainer Applied

2016-02-24 03:54 UTC #0 lore raw

This is a note to let you know that I've just added the patch titled

x86/uaccess/64: Handle the caching of 4-byte nocache copies properly in __copy_user_nocache()

to the 4.4-stable tree which can be found at:
http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary

The filename of the patch is:
x86-uaccess-64-handle-the-caching-of-4-byte-nocache-copies-properly-in-__copy_user_nocache.patch
and it can be found in the queue-4.4 subdirectory.

If you, or anyone else, feels it should not be added to the stable tree, please let <[email protected]> know about it.

Quoted 1 line

From a82eee7424525e34e98d821dd059ce14560a1e35 Mon Sep 17 00:00:00 2001

From: Toshi Kani <[email protected]>
Date: Thu, 11 Feb 2016 14:24:17 -0700
Subject: x86/uaccess/64: Handle the caching of 4-byte nocache copies properly in __copy_user_nocache()

From: Toshi Kani <[email protected]>

commit a82eee7424525e34e98d821dd059ce14560a1e35 upstream.

Data corruption issues were observed in tests which initiated a system crash/reset while accessing BTT devices. This problem is reproducible.

The BTT driver calls pmem_rw_bytes() to update data in pmem devices. This interface calls __copy_user_nocache(), which uses non-temporal stores so that the stores to pmem are persistent.

__copy_user_nocache() uses non-temporal stores when a request size is 8 bytes or larger (and is aligned by 8 bytes). The BTT driver updates the BTT map table, which entry size is 4 bytes. Therefore, updates to the map table entries remain cached, and are not written to pmem after a crash.

Change __copy_user_nocache() to use non-temporal store when a request size is 4 bytes. The change extends the current byte-copy path for a less-than-8-bytes request, and does not add any overhead to the regular path.

Reported-and-tested-by Micah Parrish <[email protected]>

Reported-and-tested-by Brian Boylston <[email protected]>

Signed-off-by Toshi Kani <[email protected]>

Cc Andrew Morton <[email protected]>

Cc Andy Lutomirski <[email protected]>

Cc Borislav Petkov <[email protected]>

Cc Brian Gerst <[email protected]>

Cc Dan Williams <[email protected]>

Cc Denys Vlasenko <[email protected]>

Cc H. Peter Anvin <[email protected]>

Cc Linus Torvalds <[email protected]>

Cc Luis R. Rodriguez <[email protected]>

Cc Peter Zijlstra <[email protected]>

Cc Ross Zwisler <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Toshi Kani <[email protected]>

Cc Vishal Verma <[email protected]>

Cc [email protected]

Link http://lkml.kernel.org/r/[email protected] open here

[ Small readability edits. ]

Signed-off-by Ingo Molnar <[email protected]>

Signed-off-by Greg Kroah-Hartman <[email protected]>

---
 arch/x86/lib/copy_user_64.S |   36 ++++++++++++++++++++++++++++++++----
 1 file changed, 32 insertions(+), 4 deletions(-)

arch/x86/lib/copy_user_64.S +32 -4

@@ -237,13 +237,14 @@ ENDPROC(copy_user_enhanced_fast_string)
  * Note: Cached memory copy is used when destination or size is not
  * naturally aligned. That is:
  *  - Require 8-byte alignment when size is 8 bytes or larger.
+ *  - Require 4-byte alignment when size is 4 bytes.
  */
 ENTRY(__copy_user_nocache)
 	ASM_STAC
 
-	/* If size is less than 8 bytes, go to byte copy */
+	/* If size is less than 8 bytes, go to 4-byte copy */
 	cmpl $8,%edx
-	jb .L_1b_cache_copy_entry
+	jb .L_4b_nocache_copy_entry
 
 	/* If destination is not 8-byte aligned, "cache" copy to align it */
 	ALIGN_DESTINATION

@@ -282,7 +283,7 @@ ENTRY(__copy_user_nocache)
 	movl %edx,%ecx
 	andl $7,%edx
 	shrl $3,%ecx
-	jz .L_1b_cache_copy_entry	/* jump if count is 0 */
+	jz .L_4b_nocache_copy_entry	/* jump if count is 0 */
 
 	/* Perform 8-byte nocache loop-copy */
 .L_8b_nocache_copy_loop:

@@ -294,11 +295,33 @@ ENTRY(__copy_user_nocache)
 	jnz .L_8b_nocache_copy_loop
 
 	/* If no byte left, we're done */
-.L_1b_cache_copy_entry:
+.L_4b_nocache_copy_entry:
+	andl %edx,%edx
+	jz .L_finish_copy
+
+	/* If destination is not 4-byte aligned, go to byte copy: */
+	movl %edi,%ecx
+	andl $3,%ecx
+	jnz .L_1b_cache_copy_entry
+
+	/* Set 4-byte copy count (1 or 0) and remainder */
+	movl %edx,%ecx
+	andl $3,%edx
+	shrl $2,%ecx
+	jz .L_1b_cache_copy_entry	/* jump if count is 0 */
+
+	/* Perform 4-byte nocache copy: */
+30:	movl (%rsi),%r8d
+31:	movnti %r8d,(%rdi)
+	leaq 4(%rsi),%rsi
+	leaq 4(%rdi),%rdi
+
+	/* If no bytes left, we're done: */
 	andl %edx,%edx
 	jz .L_finish_copy
 
 	/* Perform byte "cache" loop-copy for the remainder */
+.L_1b_cache_copy_entry:
 	movl %edx,%ecx
 .L_1b_cache_copy_loop:
 40:	movb (%rsi),%al

@@ -323,6 +346,9 @@ ENTRY(__copy_user_nocache)
 .L_fixup_8b_copy:
 	lea (%rdx,%rcx,8),%rdx
 	jmp .L_fixup_handle_tail
+.L_fixup_4b_copy:
+	lea (%rdx,%rcx,4),%rdx
+	jmp .L_fixup_handle_tail
 .L_fixup_1b_copy:
 	movl %ecx,%edx
 .L_fixup_handle_tail:

@@ -348,6 +374,8 @@ ENTRY(__copy_user_nocache)
 	_ASM_EXTABLE(16b,.L_fixup_4x8b_copy)
 	_ASM_EXTABLE(20b,.L_fixup_8b_copy)
 	_ASM_EXTABLE(21b,.L_fixup_8b_copy)
+	_ASM_EXTABLE(30b,.L_fixup_4b_copy)
+	_ASM_EXTABLE(31b,.L_fixup_4b_copy)
 	_ASM_EXTABLE(40b,.L_fixup_1b_copy)
 	_ASM_EXTABLE(41b,.L_fixup_1b_copy)
 ENDPROC(__copy_user_nocache)

Patches currently in stable-queue which might be from [email protected] are

queue-4.4/x86-uaccess-64-handle-the-caching-of-4-byte-nocache-copies-properly-in-__copy_user_nocache.patch queue-4.4/devm_memremap_release-fix-memremap-d-addr-handling.patch queue-4.4/x86-mm-fix-vmalloc_fault-to-handle-large-pages-properly.patch queue-4.4/x86-uaccess-64-make-the-__copy_user_nocache-assembly-code-more-readable.patch