uaccess: better might_sleep/might_fault behavior
Subsystems (8)
cover uaccess: better might_sleep/might_fault behavior
I seem to have mis-sent v3. Trying again with same patches after fixing the message id for the cover letter. I hope the duplicates that are thus created don't inconvenience people too much. If they do, I apologize.
I have pared down the Cc list to reduce the noise. sched maintainers are Cc'd on all patches since that's the tree I aim for with these patches.
This improves the might_fault annotations used
by uaccess routines:
1. The only reason uaccess routines might sleep is if they fault. Make this explicit for
all architectures.
2. a voluntary preempt point in uaccess functions means compiler can't inline them efficiently, this breaks assumptions that they are very fast and small that e.g. net code seems to make. remove this preempt point so behaviour matches what callers assume.
3. Accesses (e.g through socket ops) to kernel memory with KERNEL_DS like net/sunrpc does will never sleep. Remove an unconditinal might_sleep in the inline might_fault in kernel.h
(used when PROVE_LOCKING is not set).
4. Accesses with pagefault_disable return EFAULT but won't cause caller to sleep.
Check for that and avoid might_sleep when PROVE_LOCKING is set.
I'd like these changes to go in for 3.11:
besides a general benefit of improved consistency and performance, I would also like them for the vhost driver where we want to call socket ops under a spinlock, and fall back on slower thread handler on error.
If the changes look good, would sched maintainers please consider merging them through sched/core because of the interaction with the scheduler?
Please review, and consider for 3.11.
Note on arch code updates:
I tested x86_64 code.
Other architectures were build-tested.
I don't have cross-build environment for arm64, tile, microblaze and mn10300 architectures. arm64 and tile got acks.
The arch changes look generally safe enough
but would appreciate review/acks from arch maintainers. core changes naturally need acks from sched maintainers.
Version 1 of this change was titled
x86: uaccess s/might_sleep/might_fault/
Changes from v2:
add a patch removing a colunatry preempt point in uaccess functions when PREEMPT_VOLUNATRY is set.
Addresses comments by Arnd Bergmann, and Peter Zijlstra.
comment on future possible simplifications in the git log
for the powerpc patch. Addresses a comment by Arnd Bergmann.
Changes from v1:
add more architectures
fix might_fault() scheduling differently depending on CONFIG_PROVE_LOCKING, as suggested by Ingo
Michael S. Tsirkin (11):
asm-generic: uaccess s/might_sleep/might_fault/
arm64: uaccess s/might_sleep/might_fault/
frv: uaccess s/might_sleep/might_fault/
m32r: uaccess s/might_sleep/might_fault/
microblaze: uaccess s/might_sleep/might_fault/
mn10300: uaccess s/might_sleep/might_fault/
powerpc: uaccess s/might_sleep/might_fault/
tile: uaccess s/might_sleep/might_fault/
x86: uaccess s/might_sleep/might_fault/
kernel: drop voluntary schedule from might_fault
kernel: uaccess in atomic with pagefault_disable
arch/arm64/include/asm/uaccess.h | 4 ++-- arch/frv/include/asm/uaccess.h | 4 ++-- arch/m32r/include/asm/uaccess.h | 12 ++++++------ arch/microblaze/include/asm/uaccess.h | 6 +++--- arch/mn10300/include/asm/uaccess.h | 4 ++-- arch/powerpc/include/asm/uaccess.h | 16 ++++++++-------- arch/tile/include/asm/uaccess.h | 2 +- arch/x86/include/asm/uaccess_64.h | 2 +- include/asm-generic/uaccess.h | 10 +++++----- include/linux/kernel.h | 7 ++----- mm/memory.c | 10 +++++++--- 11 files changed, 39 insertions(+), 38 deletions(-)
Signature · 7 lines
MST -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/11 asm-generic: uaccess s/might_sleep/might_fault/
The only reason uaccess routines might sleep
is if they fault. Make this explicit.
--- include/asm-generic/uaccess.h | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
include/asm-generic/uaccess.h +5 -5
@@ -163,7 +163,7 @@ static inline __must_check long __copy_to_user(void __user *to,#define put_user (x, ptr) \ ({ \ - might_sleep(); \ + might_fault(); \ access_ok(VERIFY_WRITE, ptr, sizeof(*ptr)) ? \ __put_user(x, ptr) : \ -EFAULT; \
@@ -225,7 +225,7 @@ extern int __put_user_bad(void) __attribute__((noreturn));#define get_user (x, ptr) \ ({ \ - might_sleep(); \ + might_fault(); \ access_ok(VERIFY_READ, ptr, sizeof(*ptr)) ? \ __get_user(x, ptr) : \ -EFAULT; \
@@ -255,7 +255,7 @@ extern int __get_user_bad(void) __attribute__((noreturn));static inline long copy_from_user (void *to,const void __user * from,unsigned long n) { -might_sleep (); +might_fault ();if (access_ok (VERIFY_READ , from, n))return __copy_from_user (to, from, n);else
@@ -265,7 +265,7 @@ static inline long copy_from_user(void *to,static inline long copy_to_user (void __user *to,const void *from,unsigned long n) { -might_sleep (); +might_fault ();if (access_ok (VERIFY_WRITE , to, n))return __copy_to_user (to, from, n);else
@@ -336,7 +336,7 @@ __clear_user(void __user *to, unsigned long n)static inline __must_check unsigned long clear_user (void __user *to,unsigned long n) { -might_sleep (); +might_fault ();if (!access_ok (VERIFY_WRITE , to, n))return n;
Signature · 1 line
MST
tip-bot for Michael S. Tsirkin
merge bot[tip:sched/mm] asm-generic: uaccess s/might_sleep/might_fault/
asm-generic: uaccess s/might_sleep/might_fault/
Commit-ID: e0acd0bd0594161be44c054bb6b984972f444beb Gitweb: http://git.kernel.org/tip/e0acd0bd0594161be44c054bb6b984972f444beb Author: Michael S. Tsirkin <[email protected]> AuthorDate: Sun, 26 May 2013 17:30:36 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 28 May 2013 09:41:05 +0200
asm-generic: uaccess s/might_sleep/might_fault/
The only reason uaccess routines might sleep is if they fault. Make this explicit.
--- include/asm-generic/uaccess.h | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
include/asm-generic/uaccess.h +5 -5
@@ -163,7 +163,7 @@ static inline __must_check long __copy_to_user(void __user *to,#define put_user (x, ptr) \ ({ \ - might_sleep(); \ + might_fault(); \ access_ok(VERIFY_WRITE, ptr, sizeof(*ptr)) ? \ __put_user(x, ptr) : \ -EFAULT; \
@@ -225,7 +225,7 @@ extern int __put_user_bad(void) __attribute__((noreturn));#define get_user (x, ptr) \ ({ \ - might_sleep(); \ + might_fault(); \ access_ok(VERIFY_READ, ptr, sizeof(*ptr)) ? \ __get_user(x, ptr) : \ -EFAULT; \
@@ -255,7 +255,7 @@ extern int __get_user_bad(void) __attribute__((noreturn));static inline long copy_from_user (void *to,const void __user * from,unsigned long n) { -might_sleep (); +might_fault ();if (access_ok (VERIFY_READ , from, n))return __copy_from_user (to, from, n);else
@@ -265,7 +265,7 @@ static inline long copy_from_user(void *to,static inline long copy_to_user (void __user *to,const void *from,unsigned long n) { -might_sleep (); +might_fault ();if (access_ok (VERIFY_WRITE , to, n))return __copy_to_user (to, from, n);else
@@ -336,7 +336,7 @@ __clear_user(void __user *to, unsigned long n)static inline __must_check unsigned long clear_user (void __user *to,unsigned long n) { -might_sleep (); +might_fault ();if (!access_ok (VERIFY_WRITE , to, n))return n;
PATCH 2/11 arm64: uaccess s/might_sleep/might_fault/
In reply to Michael S. TsirkinThe only reason uaccess routines might sleep
is if they fault. Make this explicit.
--- arch/arm64/include/asm/uaccess.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/arm64/include/asm/uaccess.h +2 -2
@@ -166,7 +166,7 @@ do { \#define get_user (x, ptr) \ ({ \ - might_sleep(); \ + might_fault(); \ access_ok(VERIFY_READ, (ptr), sizeof(*(ptr))) ? \ __get_user((x), (ptr)) : \ ((x) = 0, -EFAULT); \
@@ -227,7 +227,7 @@ do { \#define put_user (x, ptr) \ ({ \ - might_sleep(); \ + might_fault(); \ access_ok(VERIFY_WRITE, (ptr), sizeof(*(ptr))) ? \ __put_user((x), (ptr)) : \ -EFAULT; \
Signature · 1 line
MST
tip-bot for Michael S. Tsirkin
merge bot[tip:sched/mm] arm64: uaccess s/might_sleep/might_fault/
arm64: uaccess s/might_sleep/might_fault/
Commit-ID: 56d2ef789f7c424918abdf6b95d84a64c1473220 Gitweb: http://git.kernel.org/tip/56d2ef789f7c424918abdf6b95d84a64c1473220 Author: Michael S. Tsirkin <[email protected]> AuthorDate: Sun, 26 May 2013 17:30:42 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 28 May 2013 09:41:06 +0200
arm64: uaccess s/might_sleep/might_fault/
The only reason uaccess routines might sleep is if they fault. Make this explicit.
--- arch/arm64/include/asm/uaccess.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/arm64/include/asm/uaccess.h +2 -2
@@ -166,7 +166,7 @@ do { \#define get_user (x, ptr) \ ({ \ - might_sleep(); \ + might_fault(); \ access_ok(VERIFY_READ, (ptr), sizeof(*(ptr))) ? \ __get_user((x), (ptr)) : \ ((x) = 0, -EFAULT); \
@@ -227,7 +227,7 @@ do { \#define put_user (x, ptr) \ ({ \ - might_sleep(); \ + might_fault(); \ access_ok(VERIFY_WRITE, (ptr), sizeof(*(ptr))) ? \ __put_user((x), (ptr)) : \ -EFAULT; \
PATCH 3/11 frv: uaccess s/might_sleep/might_fault/
In reply to Michael S. TsirkinThe only reason uaccess routines might sleep
is if they fault. Make this explicit.
--- arch/frv/include/asm/uaccess.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/frv/include/asm/uaccess.h +2 -2
@@ -280,14 +280,14 @@ extern long __memcpy_user(void *dst, const void *src, unsigned long count);static inline unsigned long __must_check __copy_to_user (void __user *to,const void *from,unsigned long n) { -might_sleep (); +might_fault ();return __copy_to_user_inatomic (to, from, n); }static inline unsigned long __copy_from_user (void *to,const void __user *from,unsigned long n) { -might_sleep (); +might_fault ();return __copy_from_user_inatomic (to, from, n); }
Signature · 1 line
MST
tip-bot for Michael S. Tsirkin
merge bot[tip:sched/mm] frv: uaccess s/might_sleep/might_fault/
frv: uaccess s/might_sleep/might_fault/
Commit-ID: b607ae78ac8a78f8e5e36817500e7c311519f032 Gitweb: http://git.kernel.org/tip/b607ae78ac8a78f8e5e36817500e7c311519f032 Author: Michael S. Tsirkin <[email protected]> AuthorDate: Sun, 26 May 2013 17:30:47 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 28 May 2013 09:41:07 +0200
frv: uaccess s/might_sleep/might_fault/
The only reason uaccess routines might sleep is if they fault. Make this explicit.
--- arch/frv/include/asm/uaccess.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/frv/include/asm/uaccess.h +2 -2
@@ -280,14 +280,14 @@ extern long __memcpy_user(void *dst, const void *src, unsigned long count);static inline unsigned long __must_check __copy_to_user (void __user *to,const void *from,unsigned long n) { -might_sleep (); +might_fault ();return __copy_to_user_inatomic (to, from, n); }static inline unsigned long __copy_from_user (void *to,const void __user *from,unsigned long n) { -might_sleep (); +might_fault ();return __copy_from_user_inatomic (to, from, n); }
PATCH 4/11 m32r: uaccess s/might_sleep/might_fault/
In reply to Michael S. TsirkinThe only reason uaccess routines might sleep
is if they fault. Make this explicit.
--- arch/m32r/include/asm/uaccess.h | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
arch/m32r/include/asm/uaccess.h +6 -6
@@ -216,7 +216,7 @@ extern int fixup_exception(struct pt_regs *regs); ({ \ long __gu_err = 0; \ unsigned long __gu_val; \ - might_sleep(); \ + might_fault(); \ __get_user_size(__gu_val,(ptr),(size),__gu_err); \ (x) = (__typeof__(*(ptr)))__gu_val; \ __gu_err; \
@@ -227,7 +227,7 @@ extern int fixup_exception(struct pt_regs *regs);long __gu_err = -EFAULT ; \unsigned long __gu_val =0 ; \ const __typeof__(*(ptr)) __user *__gu_addr = (ptr); \ -might_sleep (); \ +might_fault (); \if (access_ok (VERIFY_READ ,__gu_addr,size)) \__get_user_size (__gu_val,__gu_addr,(size),__gu_err); \ (x) = (__typeof__ (*(ptr )))__gu_val; \
@@ -295,7 +295,7 @@ do { \#define __put_user_nocheck (x,ptr,size) \ ({ \ long __pu_err; \ - might_sleep(); \ + might_fault(); \ __put_user_size((x),(ptr),(size),__pu_err); \ __pu_err; \ })
@@ -305,7 +305,7 @@ do { \ ({ \ long __pu_err = -EFAULT; \ __typeof__(*(ptr)) __user *__pu_addr = (ptr); \ - might_sleep(); \ + might_fault(); \ if (access_ok(VERIFY_WRITE,__pu_addr,size)) \ __put_user_size((x),__pu_addr,(size),__pu_err); \ __pu_err; \
@@ -597,7 +597,7 @@ unsigned long __generic_copy_from_user(void *, const void __user *, unsigned lon*/ #define copy_to_user (to,from,n) \ ({ \ - might_sleep(); \ + might_fault(); \ __generic_copy_to_user((to),(from),(n)); \ })
@@ -638,7 +638,7 @@ unsigned long __generic_copy_from_user(void *, const void __user *, unsigned lon*/ #define copy_from_user (to,from,n) \ ({ \ - might_sleep(); \ + might_fault(); \ __generic_copy_from_user((to),(from),(n)); \ })
Signature · 1 line
MST
tip-bot for Michael S. Tsirkin
merge bot[tip:sched/mm] m32r: uaccess s/might_sleep/might_fault/
m32r: uaccess s/might_sleep/might_fault/
Commit-ID: 01682576d5fd1c92b96d79560b17208a6567c331 Gitweb: http://git.kernel.org/tip/01682576d5fd1c92b96d79560b17208a6567c331 Author: Michael S. Tsirkin <[email protected]> AuthorDate: Sun, 26 May 2013 17:30:51 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 28 May 2013 09:41:07 +0200
m32r: uaccess s/might_sleep/might_fault/
The only reason uaccess routines might sleep is if they fault. Make this explicit.
--- arch/m32r/include/asm/uaccess.h | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
arch/m32r/include/asm/uaccess.h +6 -6
@@ -216,7 +216,7 @@ extern int fixup_exception(struct pt_regs *regs); ({ \ long __gu_err = 0; \ unsigned long __gu_val; \ - might_sleep(); \ + might_fault(); \ __get_user_size(__gu_val,(ptr),(size),__gu_err); \ (x) = (__typeof__(*(ptr)))__gu_val; \ __gu_err; \
@@ -227,7 +227,7 @@ extern int fixup_exception(struct pt_regs *regs);long __gu_err = -EFAULT ; \unsigned long __gu_val =0 ; \ const __typeof__(*(ptr)) __user *__gu_addr = (ptr); \ -might_sleep (); \ +might_fault (); \if (access_ok (VERIFY_READ ,__gu_addr,size)) \__get_user_size (__gu_val,__gu_addr,(size),__gu_err); \ (x) = (__typeof__ (*(ptr )))__gu_val; \
@@ -295,7 +295,7 @@ do { \#define __put_user_nocheck (x,ptr,size) \ ({ \ long __pu_err; \ - might_sleep(); \ + might_fault(); \ __put_user_size((x),(ptr),(size),__pu_err); \ __pu_err; \ })
@@ -305,7 +305,7 @@ do { \ ({ \ long __pu_err = -EFAULT; \ __typeof__(*(ptr)) __user *__pu_addr = (ptr); \ - might_sleep(); \ + might_fault(); \ if (access_ok(VERIFY_WRITE,__pu_addr,size)) \ __put_user_size((x),__pu_addr,(size),__pu_err); \ __pu_err; \
@@ -597,7 +597,7 @@ unsigned long __generic_copy_from_user(void *, const void __user *, unsigned lon*/ #define copy_to_user (to,from,n) \ ({ \ - might_sleep(); \ + might_fault(); \ __generic_copy_to_user((to),(from),(n)); \ })
@@ -638,7 +638,7 @@ unsigned long __generic_copy_from_user(void *, const void __user *, unsigned lon*/ #define copy_from_user (to,from,n) \ ({ \ - might_sleep(); \ + might_fault(); \ __generic_copy_from_user((to),(from),(n)); \ })
PATCH 5/11 microblaze: uaccess s/might_sleep/might_fault/
In reply to Michael S. TsirkinThe only reason uaccess routines might sleep
is if they fault. Make this explicit.
--- arch/microblaze/include/asm/uaccess.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/microblaze/include/asm/uaccess.h +3 -3
@@ -145,7 +145,7 @@ static inline unsigned long __must_check __clear_user(void __user *to,static inline unsigned long __must_check clear_user (void __user *to,unsigned long n) { -might_sleep (); +might_fault ();if (unlikely (!access_ok (VERIFY_WRITE , to, n)))return n;
@@ -371,7 +371,7 @@ extern long __user_bad(void);static inline long copy_from_user (void *to,const void __user *from,unsigned long n) { -might_sleep (); +might_fault ();if (access_ok (VERIFY_READ , from, n))return __copy_from_user (to, from, n);return n;
@@ -385,7 +385,7 @@ static inline long copy_from_user(void *to,static inline long copy_to_user (void __user *to,const void *from,unsigned long n) { -might_sleep (); +might_fault ();if (access_ok (VERIFY_WRITE , to, n))return __copy_to_user (to, from, n);return n;
Signature · 1 line
MST
tip-bot for Michael S. Tsirkin
merge bot[tip:sched/mm] microblaze: uaccess s/might_sleep/might_fault/
microblaze: uaccess s/might_sleep/might_fault/
Commit-ID: ac093f8d5e76be1f2654acfd7a59d339ba037654 Gitweb: http://git.kernel.org/tip/ac093f8d5e76be1f2654acfd7a59d339ba037654 Author: Michael S. Tsirkin <[email protected]> AuthorDate: Sun, 26 May 2013 17:30:56 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 28 May 2013 09:41:08 +0200
microblaze: uaccess s/might_sleep/might_fault/
The only reason uaccess routines might sleep is if they fault. Make this explicit.
--- arch/microblaze/include/asm/uaccess.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/microblaze/include/asm/uaccess.h +3 -3
@@ -145,7 +145,7 @@ static inline unsigned long __must_check __clear_user(void __user *to,static inline unsigned long __must_check clear_user (void __user *to,unsigned long n) { -might_sleep (); +might_fault ();if (unlikely (!access_ok (VERIFY_WRITE , to, n)))return n;
@@ -371,7 +371,7 @@ extern long __user_bad(void);static inline long copy_from_user (void *to,const void __user *from,unsigned long n) { -might_sleep (); +might_fault ();if (access_ok (VERIFY_READ , from, n))return __copy_from_user (to, from, n);return n;
@@ -385,7 +385,7 @@ static inline long copy_from_user(void *to,static inline long copy_to_user (void __user *to,const void *from,unsigned long n) { -might_sleep (); +might_fault ();if (access_ok (VERIFY_WRITE , to, n))return __copy_to_user (to, from, n);return n;
PATCH 6/11 mn10300: uaccess s/might_sleep/might_fault/
In reply to Michael S. TsirkinThe only reason uaccess routines might sleep
is if they fault. Make this explicit.
--- arch/mn10300/include/asm/uaccess.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/mn10300/include/asm/uaccess.h +2 -2
@@ -471,13 +471,13 @@ extern unsigned long __generic_copy_from_user(void *, const void __user *,#define __copy_to_user (to, from, n) \ ({ \ - might_sleep(); \ + might_fault(); \ __copy_to_user_inatomic((to), (from), (n)); \ })#define __copy_from_user (to, from, n) \ ({ \ - might_sleep(); \ + might_fault(); \ __copy_from_user_inatomic((to), (from), (n)); \ })
Signature · 1 line
MST
tip-bot for Michael S. Tsirkin
merge bot[tip:sched/mm] mn10300: uaccess s/might_sleep/might_fault/
mn10300: uaccess s/might_sleep/might_fault/
Commit-ID: 3837a3cfe4a27836e0e9f207eb2d4f00b5a8fcba Gitweb: http://git.kernel.org/tip/3837a3cfe4a27836e0e9f207eb2d4f00b5a8fcba Author: Michael S. Tsirkin <[email protected]> AuthorDate: Sun, 26 May 2013 17:31:05 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 28 May 2013 09:41:08 +0200
mn10300: uaccess s/might_sleep/might_fault/
The only reason uaccess routines might sleep is if they fault. Make this explicit.
--- arch/mn10300/include/asm/uaccess.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/mn10300/include/asm/uaccess.h +2 -2
@@ -471,13 +471,13 @@ extern unsigned long __generic_copy_from_user(void *, const void __user *,#define __copy_to_user (to, from, n) \ ({ \ - might_sleep(); \ + might_fault(); \ __copy_to_user_inatomic((to), (from), (n)); \ })#define __copy_from_user (to, from, n) \ ({ \ - might_sleep(); \ + might_fault(); \ __copy_from_user_inatomic((to), (from), (n)); \ })
PATCH 7/11 powerpc: uaccess s/might_sleep/might_fault/
In reply to Michael S. TsirkinThe only reason uaccess routines might sleep
is if they fault. Make this explicit.
Arnd Bergmann suggested that the following code
if (!is_kernel_addr((unsigned long)__pu_addr)) might_fault();
can be further simplified by adding a version of might_fault that includes the kernel addr check.
Will be considered as a further optimization in future.
--- arch/powerpc/include/asm/uaccess.h | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
arch/powerpc/include/asm/uaccess.h +8 -8
@@ -178,7 +178,7 @@ do { \long __pu_err; \ __typeof__(*(ptr)) __user *__pu_addr = (ptr); \if (!is_kernel_addr ((unsigned long )__pu_addr)) \ -might_sleep (); \ +might_fault (); \__chk_user_ptr (ptr); \__put_user_size ((x), __pu_addr, (size), __pu_err); \ __pu_err; \
@@ -188,7 +188,7 @@ do { \ ({ \ long __pu_err = -EFAULT; \ __typeof__(*(ptr)) __user *__pu_addr = (ptr); \ - might_sleep(); \ + might_fault(); \ if (access_ok(VERIFY_WRITE, __pu_addr, size)) \ __put_user_size((x), __pu_addr, (size), __pu_err); \ __pu_err; \
@@ -268,7 +268,7 @@ do { \ const __typeof__(*(ptr)) __user *__gu_addr = (ptr); \__chk_user_ptr (ptr); \if (!is_kernel_addr ((unsigned long )__gu_addr)) \ -might_sleep (); \ +might_fault (); \__get_user_size (__gu_val, __gu_addr, (size), __gu_err); \ (x) = (__typeof__ (*(ptr )))__gu_val; \ __gu_err; \
@@ -282,7 +282,7 @@ do { \ const __typeof__(*(ptr)) __user *__gu_addr = (ptr); \__chk_user_ptr (ptr); \if (!is_kernel_addr ((unsigned long )__gu_addr)) \ -might_sleep (); \ +might_fault (); \__get_user_size (__gu_val, __gu_addr, (size), __gu_err); \ (x) = (__typeof__ (*(ptr )))__gu_val; \ __gu_err; \
@@ -294,7 +294,7 @@ do { \long __gu_err = -EFAULT ; \unsigned long __gu_val =0 ; \ const __typeof__(*(ptr)) __user *__gu_addr = (ptr); \ -might_sleep (); \ +might_fault (); \if (access_ok (VERIFY_READ , __gu_addr, (size))) \__get_user_size (__gu_val, __gu_addr, (size), __gu_err); \ (x) = (__typeof__ (*(ptr )))__gu_val; \
@@ -419,14 +419,14 @@ static inline unsigned long __copy_to_user_inatomic(void __user *to,static inline unsigned long __copy_from_user (void *to,const void __user *from,unsigned long size) { -might_sleep (); +might_fault ();return __copy_from_user_inatomic (to, from, size); }static inline unsigned long __copy_to_user (void __user *to,const void *from,unsigned long size) { -might_sleep (); +might_fault ();return __copy_to_user_inatomic (to, from, size); }
@@ -434,7 +434,7 @@ extern unsigned long __clear_user(void __user *addr, unsigned long size);static inline unsigned long clear_user (void __user *addr,unsigned long size) { -might_sleep (); +might_fault ();if (likely (access_ok (VERIFY_WRITE , addr, size)))return __clear_user (addr, size);if ((unsigned long )addr <TASK_SIZE ) {
Signature · 1 line
MST
Re: [PATCH v3-resend 07/11] powerpc: uaccess s/might_sleep/might_fault/
Quoted 4 lines · Michael S. Tsirkin
The only reason uaccess routines might sleep
is if they fault. Make this explicit.Arnd Bergmann suggested that the following code
Quoted 8 lines
if (!is_kernel_addr((unsigned long)__pu_addr)) might_fault();can be further simplified by adding a version of
might_faultthat includes the kernel addr check.Will be considered as a further optimization in future.
Signed-off-by Michael S. Tsirkin <[email protected]>
As long as Peter is happy with the general semantics of might_fault() and you have at least build-tested it, then I'm happy.
Quoted 79 lines
--- arch/powerpc/include/asm/uaccess.h | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)arch/powerpc/include/asm/uaccess.h +8
-8@@ -178,7 +178,7 @@ do { \long __pu_err; \ __typeof__(*(ptr)) __user *__pu_addr = (ptr); \if (!is_kernel_addr ((unsigned long )__pu_addr)) \ -might_sleep (); \ +might_fault (); \__chk_user_ptr (ptr); \__put_user_size ((x), __pu_addr, (size), __pu_err); \ __pu_err; \@@ -188,7 +188,7 @@ do { \ ({ \ long __pu_err = -EFAULT; \ __typeof__(*(ptr)) __user *__pu_addr = (ptr); \ - might_sleep(); \ + might_fault(); \ if (access_ok(VERIFY_WRITE, __pu_addr, size)) \ __put_user_size((x), __pu_addr, (size), __pu_err); \ __pu_err; \@@ -268,7 +268,7 @@ do { \ const __typeof__(*(ptr)) __user *__gu_addr = (ptr); \__chk_user_ptr (ptr); \if (!is_kernel_addr ((unsigned long )__gu_addr)) \ -might_sleep (); \ +might_fault (); \__get_user_size (__gu_val, __gu_addr, (size), __gu_err); \ (x) = (__typeof__ (*(ptr )))__gu_val; \ __gu_err; \@@ -282,7 +282,7 @@ do { \ const __typeof__(*(ptr)) __user *__gu_addr = (ptr); \__chk_user_ptr (ptr); \if (!is_kernel_addr ((unsigned long )__gu_addr)) \ -might_sleep (); \ +might_fault (); \__get_user_size (__gu_val, __gu_addr, (size), __gu_err); \ (x) = (__typeof__ (*(ptr )))__gu_val; \ __gu_err; \@@ -294,7 +294,7 @@ do { \long __gu_err = -EFAULT ; \unsigned long __gu_val =0 ; \ const __typeof__(*(ptr)) __user *__gu_addr = (ptr); \ -might_sleep (); \ +might_fault (); \if (access_ok (VERIFY_READ , __gu_addr, (size))) \__get_user_size (__gu_val, __gu_addr, (size), __gu_err); \ (x) = (__typeof__ (*(ptr )))__gu_val; \@@ -419,14 +419,14 @@ static inline unsigned long __copy_to_user_inatomic(void __user *to,static inline unsigned long __copy_from_user (void *to,const void __user *from,unsigned long size) { -might_sleep (); +might_fault ();return __copy_from_user_inatomic (to, from, size); }static inline unsigned long __copy_to_user (void __user *to,const void *from,unsigned long size) { -might_sleep (); +might_fault ();return __copy_to_user_inatomic (to, from, size); }@@ -434,7 +434,7 @@ extern unsigned long __clear_user(void __user *addr, unsigned long size);static inline unsigned long clear_user (void __user *addr,unsigned long size) { -might_sleep (); +might_fault ();if (likely (access_ok (VERIFY_WRITE , addr, size)))return __clear_user (addr, size);if ((unsigned long )addr <TASK_SIZE ) {
tip-bot for Michael S. Tsirkin
merge bot[tip:sched/mm] powerpc: uaccess s/might_sleep/might_fault/
In reply to Michael S. Tsirkinpowerpc: uaccess s/might_sleep/might_fault/
Commit-ID: 1af1717dbf96eba8a74a2d6a99e75a7795075a02 Gitweb: http://git.kernel.org/tip/1af1717dbf96eba8a74a2d6a99e75a7795075a02 Author: Michael S. Tsirkin <[email protected]> AuthorDate: Sun, 26 May 2013 17:31:38 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 28 May 2013 09:41:09 +0200
powerpc: uaccess s/might_sleep/might_fault/
The only reason uaccess routines might sleep
is if they fault. Make this explicit.
Arnd Bergmann suggested that the following code
if (!is_kernel_addr((unsigned long)__pu_addr)) might_fault();
can be further simplified by adding a version of might_fault that includes the kernel addr check.
Will be considered as a further optimization in future.
--- arch/powerpc/include/asm/uaccess.h | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
arch/powerpc/include/asm/uaccess.h +8 -8
@@ -178,7 +178,7 @@ do { \long __pu_err; \ __typeof__(*(ptr)) __user *__pu_addr = (ptr); \if (!is_kernel_addr ((unsigned long )__pu_addr)) \ -might_sleep (); \ +might_fault (); \__chk_user_ptr (ptr); \__put_user_size ((x), __pu_addr, (size), __pu_err); \ __pu_err; \
@@ -188,7 +188,7 @@ do { \ ({ \ long __pu_err = -EFAULT; \ __typeof__(*(ptr)) __user *__pu_addr = (ptr); \ - might_sleep(); \ + might_fault(); \ if (access_ok(VERIFY_WRITE, __pu_addr, size)) \ __put_user_size((x), __pu_addr, (size), __pu_err); \ __pu_err; \
@@ -268,7 +268,7 @@ do { \ const __typeof__(*(ptr)) __user *__gu_addr = (ptr); \__chk_user_ptr (ptr); \if (!is_kernel_addr ((unsigned long )__gu_addr)) \ -might_sleep (); \ +might_fault (); \__get_user_size (__gu_val, __gu_addr, (size), __gu_err); \ (x) = (__typeof__ (*(ptr )))__gu_val; \ __gu_err; \
@@ -282,7 +282,7 @@ do { \ const __typeof__(*(ptr)) __user *__gu_addr = (ptr); \__chk_user_ptr (ptr); \if (!is_kernel_addr ((unsigned long )__gu_addr)) \ -might_sleep (); \ +might_fault (); \__get_user_size (__gu_val, __gu_addr, (size), __gu_err); \ (x) = (__typeof__ (*(ptr )))__gu_val; \ __gu_err; \
@@ -294,7 +294,7 @@ do { \long __gu_err = -EFAULT ; \unsigned long __gu_val =0 ; \ const __typeof__(*(ptr)) __user *__gu_addr = (ptr); \ -might_sleep (); \ +might_fault (); \if (access_ok (VERIFY_READ , __gu_addr, (size))) \__get_user_size (__gu_val, __gu_addr, (size), __gu_err); \ (x) = (__typeof__ (*(ptr )))__gu_val; \
@@ -419,14 +419,14 @@ static inline unsigned long __copy_to_user_inatomic(void __user *to,static inline unsigned long __copy_from_user (void *to,const void __user *from,unsigned long size) { -might_sleep (); +might_fault ();return __copy_from_user_inatomic (to, from, size); }static inline unsigned long __copy_to_user (void __user *to,const void *from,unsigned long size) { -might_sleep (); +might_fault ();return __copy_to_user_inatomic (to, from, size); }
@@ -434,7 +434,7 @@ extern unsigned long __clear_user(void __user *addr, unsigned long size);static inline unsigned long clear_user (void __user *addr,unsigned long size) { -might_sleep (); +might_fault ();if (likely (access_ok (VERIFY_WRITE , addr, size)))return __clear_user (addr, size);if ((unsigned long )addr <TASK_SIZE ) {
PATCH 8/11 tile: uaccess s/might_sleep/might_fault/
In reply to Michael S. TsirkinThe only reason uaccess routines might sleep
is if they fault. Make this explicit.
--- arch/tile/include/asm/uaccess.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/tile/include/asm/uaccess.h +1 -1
@@ -442,7 +442,7 @@ extern unsigned long __copy_in_user_inatomic(static inline unsigned long __must_check __copy_in_user (void __user *to,const void __user *from,unsigned long n) { -might_sleep (); +might_fault ();return __copy_in_user_inatomic (to, from, n); }
Signature · 1 line
MST
tip-bot for Michael S. Tsirkin
merge bot[tip:sched/mm] tile: uaccess s/might_sleep/might_fault/
tile: uaccess s/might_sleep/might_fault/
Commit-ID: f8abe86cc4fbd4ba083fd151b88e02fb3ce88b9c Gitweb: http://git.kernel.org/tip/f8abe86cc4fbd4ba083fd151b88e02fb3ce88b9c Author: Michael S. Tsirkin <[email protected]> AuthorDate: Sun, 26 May 2013 17:31:48 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 28 May 2013 09:41:09 +0200
tile: uaccess s/might_sleep/might_fault/
The only reason uaccess routines might sleep is if they fault. Make this explicit.
--- arch/tile/include/asm/uaccess.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/tile/include/asm/uaccess.h +1 -1
@@ -442,7 +442,7 @@ extern unsigned long __copy_in_user_inatomic(static inline unsigned long __must_check __copy_in_user (void __user *to,const void __user *from,unsigned long n) { -might_sleep (); +might_fault ();return __copy_in_user_inatomic (to, from, n); }
PATCH 9/11 x86: uaccess s/might_sleep/might_fault/
In reply to Michael S. TsirkinThe only reason uaccess routines might sleep
is if they fault. Make this explicit.
--- arch/x86/include/asm/uaccess_64.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/uaccess_64.h +1 -1
@@ -235,7 +235,7 @@ extern long __copy_user_nocache(void *dst, const void __user *src,static inline int __copy_from_user_nocache (void *dst,const void __user *src,unsigned size) { -might_sleep (); +might_fault ();return __copy_user_nocache (dst, src, size,1 ); }
Signature · 1 line
MST
tip-bot for Michael S. Tsirkin
merge bot[tip:sched/mm] x86: uaccess s/might_sleep/might_fault/
x86: uaccess s/might_sleep/might_fault/
Commit-ID: 016be2e55d98aee0b97b94b200d6e0e110c8392a Gitweb: http://git.kernel.org/tip/016be2e55d98aee0b97b94b200d6e0e110c8392a Author: Michael S. Tsirkin <[email protected]> AuthorDate: Sun, 26 May 2013 17:31:55 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 28 May 2013 09:41:10 +0200
x86: uaccess s/might_sleep/might_fault/
The only reason uaccess routines might sleep is if they fault. Make this explicit.
--- arch/x86/include/asm/uaccess_64.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/uaccess_64.h +1 -1
@@ -235,7 +235,7 @@ extern long __copy_user_nocache(void *dst, const void __user *src,static inline int __copy_from_user_nocache (void *dst,const void __user *src,unsigned size) { -might_sleep (); +might_fault ();return __copy_user_nocache (dst, src, size,1 ); }
PATCH 10/11 kernel: drop voluntary schedule from might_fault
In reply to Michael S. Tsirkinmight_fault is called from functions like copy_to_user which most callers expect to be very fast, like
a couple of instructions. So functions like memcpy_toiovec call them many times in a loop.
But might_fault calls might_sleep() and with CONFIG_PREEMPT_VOLUNTARY this results in a function call.
Let's not do this - just call __might_sleep that produces a diagnostic for sleep within atomic, but drop might_preempt().
Here's a test sending traffic between the VM and the host, host is built with CONFIG_PREEMPT_VOLUNTARY:
Before:
incoming: 7122.77 Mb/s outgoing: 8480.37 Mb/s after: incoming: 8619.24 Mb/s outgoing: 9455.42 Mb/s
As a side effect, this fixes an issue pointed out by Ingo: might_fault might schedule differently depending on PROVE_LOCKING. Now there's no preemption point in both cases, so it's consistent.
--- include/linux/kernel.h | 2 +- mm/memory.c | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-)
include/linux/kernel.h +1 -1
@@ -198,7 +198,7 @@ void might_fault(void);#else static inline void might_fault (void ) { -might_sleep (); +__might_sleep (__FILE__, __LINE__,0 ); }#endif
mm/memory.c +2 -1
@@ -4222,7 +4222,8 @@ void might_fault(void)if (segment_eq (get_fs (),KERNEL_DS ))return ; -might_sleep (); +__might_sleep (__FILE__, __LINE__,0 ); +/* * it would be nicer only to annotate paths which are not under * pagefault_disable, however that requires a larger audit and
Signature · 7 lines
MST -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Michael S. Tsirkin
merge bot[tip:sched/mm] mm, sched: Drop voluntary schedule from might_fault()
mm, sched: Drop voluntary schedule from might_fault()
Commit-ID: 114276ac0a3beb9c391a410349bd770653e185ce Gitweb: http://git.kernel.org/tip/114276ac0a3beb9c391a410349bd770653e185ce Author: Michael S. Tsirkin <[email protected]> AuthorDate: Sun, 26 May 2013 17:32:13 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 28 May 2013 09:41:11 +0200
mm, sched: Drop voluntary schedule from might_fault()
might_fault() is called from functions like copy_to_user() which most callers expect to be very fast, like a couple of instructions.
So functions like memcpy_toiovec() call them many times in a loop.
But might_fault() calls might_sleep() and with CONFIG_PREEMPT_VOLUNTARY this results in a function call.
Let's not do this - just call __might_sleep() that produces a diagnostic for sleep within atomic, but drop might_preempt().
Here's a test sending traffic between the VM and the host, host is built with CONFIG_PREEMPT_VOLUNTARY:
before:
incoming: 7122.77 Mb/s outgoing: 8480.37 Mb/s
after:
incoming: 8619.24 Mb/s outgoing: 9455.42 Mb/s
As a side effect, this fixes an issue pointed out by Ingo: might_fault might schedule differently depending on PROVE_LOCKING. Now there's no preemption point in both cases, so it's consistent.
--- include/linux/kernel.h | 2 +- mm/memory.c | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-)
include/linux/kernel.h +1 -1
@@ -198,7 +198,7 @@ void might_fault(void);#else static inline void might_fault (void ) { -might_sleep (); +__might_sleep (__FILE__, __LINE__,0 ); }#endif
mm/memory.c +2 -1
@@ -4222,7 +4222,8 @@ void might_fault(void)if (segment_eq (get_fs (),KERNEL_DS ))return ; -might_sleep (); +__might_sleep (__FILE__, __LINE__,0 ); +/* * it would be nicer only to annotate paths which are not under * pagefault_disable, however that requires a larger audit and
PATCH 11/11 kernel: uaccess in atomic with pagefault_disable
In reply to Michael S. TsirkinThis changes might_fault so that it does not trigger a false positive diagnostic for e.g. the following sequence:
spin_lock_irqsave
pagefault_disable
copy_to_user
pagefault_enable
spin_unlock_irqrestore
In particular vhost wants to do this, to call
socket ops from under a lock.
There are 3 cases to consider:CONFIG_PROVE_LOCKING - might_fault is non-inline
so it's easy to move the in_atomic test to fix
up the false positive warning.
CONFIG_DEBUG_ATOMIC_SLEEP - might_fault
is currently inline, but we are calling a
non-inline __might_sleep anyway,
so let's use the non-line version of might_fault
that does the right thing.
!CONFIG_DEBUG_ATOMIC_SLEEP && !CONFIG_PROVE_LOCKING __might_sleep is a nop so might_fault is a nop. Make this explicit.
--- include/linux/kernel.h | 7 ++----- mm/memory.c | 11 +++++++---- 2 files changed, 9 insertions(+), 9 deletions(-)
include/linux/kernel.h +2 -5
@@ -193,13 +193,10 @@ extern int _cond_resched(void); (__x <0 ) ? -__x : __x; \ }) -#ifdef CONFIG_PROVE_LOCKING +#if defined(CONFIG_PROVE_LOCKING ) || defined(CONFIG_DEBUG_ATOMIC_SLEEP )void might_fault (void );#else -static inline void might_fault (void ) -{ -__might_sleep (__FILE__, __LINE__,0 ); -} +static inline void might_fault (void ) { }#endif extern struct atomic_notifier_head panic_notifier_list;
mm/memory.c +7 -4
@@ -4210,7 +4210,7 @@ void print_vma_addr(char *prefix, unsigned long ip)up_read (&mm->mmap_sem); } -#ifdef CONFIG_PROVE_LOCKING +#if defined(CONFIG_PROVE_LOCKING ) || defined(CONFIG_DEBUG_ATOMIC_SLEEP )void might_fault (void ) {/*
@@ -4222,14 +4222,17 @@ void might_fault(void)if (segment_eq (get_fs (),KERNEL_DS ))return ; -__might_sleep (__FILE__, __LINE__,0 ); -/* * it would be nicer only to annotate paths which are not under * pagefault_disable, however that requires a larger audit and * providing helpers like get_user_atomic. */ -if (!in_atomic () && current->mm) +if (in_atomic ()) +return ; + +__might_sleep (__FILE__, __LINE__,0 ); + +if (current->mm)might_lock_read (¤t->mm->mmap_sem); }EXPORT_SYMBOL (might_fault);
Signature · 7 lines
MST -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Michael S. Tsirkin
merge bot[tip:sched/mm] mm, sched: Allow uaccess in atomic with pagefault_disable()
mm, sched: Allow uaccess in atomic with pagefault_disable()
Commit-ID: 662bbcb2747c2422cf98d3d97619509379eee466 Gitweb: http://git.kernel.org/tip/662bbcb2747c2422cf98d3d97619509379eee466 Author: Michael S. Tsirkin <[email protected]> AuthorDate: Sun, 26 May 2013 17:32:23 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 28 May 2013 09:41:11 +0200
mm, sched: Allow uaccess in atomic with pagefault_disable()
This changes might_fault() so that it does not trigger a false positive diagnostic for e.g. the following sequence:
spin_lock_irqsave() pagefault_disable() copy_to_user() pagefault_enable() spin_unlock_irqrestore()
In particular vhost wants to do this, to call socket ops from under a lock.
There are 3 cases to consider:
- CONFIG_PROVE_LOCKING - might_fault is non-inline so it's easy to move the in_atomic test to fix up the false positive warning.
- CONFIG_DEBUG_ATOMIC_SLEEP - might_fault is currently inline, but we are calling a non-inline __might_sleep anyway,
so let's use the non-line version of might_fault that does the right thing.
- !CONFIG_DEBUG_ATOMIC_SLEEP && !CONFIG_PROVE_LOCKING __might_sleep is a nop so might_fault is a nop.
Make this explicit.
--- include/linux/kernel.h | 7 ++----- mm/memory.c | 11 +++++++---- 2 files changed, 9 insertions(+), 9 deletions(-)
include/linux/kernel.h +2 -5
@@ -193,13 +193,10 @@ extern int _cond_resched(void); (__x <0 ) ? -__x : __x; \ }) -#ifdef CONFIG_PROVE_LOCKING +#if defined(CONFIG_PROVE_LOCKING ) || defined(CONFIG_DEBUG_ATOMIC_SLEEP )void might_fault (void );#else -static inline void might_fault (void ) -{ -__might_sleep (__FILE__, __LINE__,0 ); -} +static inline void might_fault (void ) { }#endif extern struct atomic_notifier_head panic_notifier_list;
mm/memory.c +7 -4
@@ -4210,7 +4210,7 @@ void print_vma_addr(char *prefix, unsigned long ip)up_read (&mm->mmap_sem); } -#ifdef CONFIG_PROVE_LOCKING +#if defined(CONFIG_PROVE_LOCKING ) || defined(CONFIG_DEBUG_ATOMIC_SLEEP )void might_fault (void ) {/*
@@ -4222,14 +4222,17 @@ void might_fault(void)if (segment_eq (get_fs (),KERNEL_DS ))return ; -__might_sleep (__FILE__, __LINE__,0 ); -/* * it would be nicer only to annotate paths which are not under * pagefault_disable, however that requires a larger audit and * providing helpers like get_user_atomic. */ -if (!in_atomic () && current->mm) +if (in_atomic ()) +return ; + +__might_sleep (__FILE__, __LINE__,0 ); + +if (current->mm)might_lock_read (¤t->mm->mmap_sem); }EXPORT_SYMBOL (might_fault);
Quoted 5 lines · Michael S. Tsirkin
If the changes look good, would sched maintainers please consider merging them through sched/core because of the interaction with the scheduler?
Please review, and consider for 3.11.
I'll stick them in my queue, we'll see if anything falls over ;-)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>