Skip to contentSkip to conversation

futex: improve user space accesses

PATCH In review Linus Torvalds 4 participants 6 messages Nov 22, 2024 – Dec 9, 2024

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 futex: improve user space accesses

Josh Poimboeuf reports that he got a "will-it-scale.per_process_ops 1.9% improvement" report for his patch that changed __get_user() to use pointer masking instead of the explicit speculation barrier. However, that patch doesn't actually work in the general case, because some (very bad) architecture-specific code actually depends on __get_user() also working on kernel addresses.

A profile showed that the offending __get_user() was the futex code, which really should be fixed up to not use that horrid legacy case. Rewrite futex_get_value_locked() to use the modern user acccess helpers, and inline it so that the compiler not only avoids the function call for a few instructions, but can do CSE on the address masking.

It also turns out the x86 futex functions have unnecessary barriers in other places, so let's fix those up too.

---
 arch/x86/include/asm/futex.h |  8 +++--
 kernel/futex/core.c          | 22 --------------
 kernel/futex/futex.h         | 59 ++++++++++++++++++++++++++++++++++--
 3 files changed, 63 insertions(+), 26 deletions(-)
arch/x86/include/asm/futex.h +6 -2
index 99d345b686fa..6e2458088800 100644
@@ -48,7 +48,9 @@ do {								\
 static __always_inline int arch_futex_atomic_op_inuser(int op, int oparg, int *oval,
 		u32 __user *uaddr)
 {
-	if (!user_access_begin(uaddr, sizeof(u32)))
+	if (can_do_masked_user_access())
+		uaddr = masked_user_access_begin(uaddr);
+	else if (!user_access_begin(uaddr, sizeof(u32)))
 		return -EFAULT;
 
 	switch (op) {
@@ -84,7 +86,9 @@ static inline int futex_atomic_cmpxchg_inatomic(u32 *uval, u32 __user *uaddr,
 {
 	int ret = 0;
 
-	if (!user_access_begin(uaddr, sizeof(u32)))
+	if (can_do_masked_user_access())
+		uaddr = masked_user_access_begin(uaddr);
+	else if (!user_access_begin(uaddr, sizeof(u32)))
 		return -EFAULT;
 	asm volatile("\n"
 		"1:\t" LOCK_PREFIX "cmpxchgl %3, %2\n"
kernel/futex/core.c +0 -22
index 326bfe6549d7..9833fdb63e39 100644
@@ -451,28 +451,6 @@ struct futex_q *futex_top_waiter(struct futex_hash_bucket *hb, union futex_key *
 	return NULL;
 }
 
-int futex_cmpxchg_value_locked(u32 *curval, u32 __user *uaddr, u32 uval, u32 newval)
-{
-	int ret;
-
-	pagefault_disable();
-	ret = futex_atomic_cmpxchg_inatomic(curval, uaddr, uval, newval);
-	pagefault_enable();
-
-	return ret;
-}
-
-int futex_get_value_locked(u32 *dest, u32 __user *from)
-{
-	int ret;
-
-	pagefault_disable();
-	ret = __get_user(*dest, from);
-	pagefault_enable();
-
-	return ret ? -EFAULT : 0;
-}
-
 /**
  * wait_for_owner_exiting - Block until the owner has exited
  * @ret: owner's current futex lock status
kernel/futex/futex.h +57 -2

73 lines not on this page · raw message

Signature · 1 line
2.47.0.293.g502c15559b
Reviewed-by on v1 1/1
#1 lore raw
Quoted 13 lines · Linus Torvalds

Josh Poimboeuf reports that he got a "will-it-scale.per_process_ops 1.9% improvement" report for his patch that changed __get_user() to use pointer masking instead of the explicit speculation barrier. However, that patch doesn't actually work in the general case, because some (very bad) architecture-specific code actually depends on __get_user() also working on kernel addresses.

A profile showed that the offending __get_user() was the futex code, which really should be fixed up to not use that horrid legacy case. Rewrite futex_get_value_locked() to use the modern user acccess helpers, and inline it so that the compiler not only avoids the function call for a few instructions, but can do CSE on the address masking.

Quoted 6 lines

It also turns out the x86 futex functions have unnecessary barriers in other places, so let's fix those up too.

I didn't get a chance to try to recreate the original benchmark, but this looks obviously correct.

Signature · 1 line
Josh
#2 lore raw
In reply to Linus Torvalds
Quoted 7 lines · Linus Torvalds

Josh Poimboeuf reports that he got a "will-it-scale.per_process_ops 1.9% improvement" report for his patch that changed __get_user() to use pointer masking instead of the explicit speculation barrier. However, that patch doesn't actually work in the general case, because some (very bad) architecture-specific code actually depends on __get_user() also working on kernel addresses.

Quoted 5 lines

A profile showed that the offending __get_user() was the futex code, which really should be fixed up to not use that horrid legacy case. Rewrite futex_get_value_locked() to use the modern user acccess helpers, and inline it so that the compiler not only avoids the function call for a few instructions, but can do CSE on the address masking.

This breaks userspace on ppc32. As soon as /init in the initrd is started the kernel hangs (without any messages).

Signature · 3 lines
Andreas Schwab, [email protected]
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."
#3 lore raw
Quoted 3 lines · Andreas Schwab

This breaks userspace on ppc32. As soon as /init in the initrd is started the kernel hangs (without any messages).

Funky, funky. Most of the diff is the code movement (and some small x86-specific stuff), so for ppc, the only part that should be relevant is the futex_get_value_locked().

And since ppc doesn't do the masked user access thing, so it *literally* boils down to just that

        if (!user_read_access_begin(from, sizeof(*from)))
                return -EFAULT;
        unsafe_get_user(val, from, Efault);
        user_access_end();

path.

Ahh... And now that I write that out, the bug is obvious: it should be using

        user_read_access_end();

to match up with the user_read_access_begin().

And yeah, ppc is the only platform that has that "read-vs-write-vs-both" thing, so this bug is not visible anywhere else.

IOW, does this one-liner fix it for you?

--- a/kernel/futex/futex.h
+++ b/kernel/futex/futex.h
@@ -265,7 +265,7 @@

        else if (!user_read_access_begin(from, sizeof(*from)))
                return -EFAULT;
        unsafe_get_user(val, from, Efault);
  -     user_access_end();
  +     user_read_access_end();
        *dest = val;
        return 0;

Efault:

I bet it does, but I'll wait for confirmation before actually committing that fix.

Thanks,

Linus

#4 lore raw

Le 09/12/2024 à 01:32, Linus Torvalds a écrit :

Quoted 17 lines · Andreas Schwab
Quoted 3 lines

This breaks userspace on ppc32. As soon as /init in the initrd is started the kernel hangs (without any messages).

Funky, funky. Most of the diff is the code movement (and some small x86-specific stuff), so for ppc, the only part that should be relevant is the futex_get_value_locked().

And since ppc doesn't do the masked user access thing, so it *literally* boils down to just that

         if (!user_read_access_begin(from, sizeof(*from)))
                 return -EFAULT;
         unsafe_get_user(val, from, Efault);
         user_access_end();
Quoted 7 lines

path.

Ahh... And now that I write that out, the bug is obvious: it should be using

         user_read_access_end();

to match up with the user_read_access_begin().

Yes indeed, especially on book3s/32, which is only able to write-protect user accesses. On that platform user_read_access_...() are no-ops.

user_access_end() and user_write_access_end() are similar, and rely on a thread var stored by user_access_begin(). When calling that user_access_end() without prior call to user_access_begin(), that var has value ~0 instead of the address of the user segment being accessed, and ~0 is a kernel address so user_access_end() applies some user segment flags to a kernel segment which most likely leads to a complete mess allthough I'm not able to trigger the hang with QEMU.

Quoted 19 lines

And yeah, ppc is the only platform that has that "read-vs-write-vs-both" thing, so this bug is not visible anywhere else.

IOW, does this one-liner fix it for you?

   --- a/kernel/futex/futex.h
   +++ b/kernel/futex/futex.h
   @@ -265,7 +265,7 @@
         else if (!user_read_access_begin(from, sizeof(*from)))
                 return -EFAULT;
         unsafe_get_user(val, from, Efault);
   -     user_access_end();
   +     user_read_access_end();
         *dest = val;
         return 0;
    Efault:
Quoted 3 lines

I bet it does, but I'll wait for confirmation before actually committing that fix.

You'll need the same change in the Efault leg.

Christophe

#5 lore raw
In reply to Linus Torvalds
Quoted 5 lines · Linus Torvalds

IOW, does this one-liner fix it for you?

--- a/kernel/futex/futex.h
+++ b/kernel/futex/futex.h
@@ -265,7 +265,7 @@

Quoted 8 lines
        else if (!user_read_access_begin(from, sizeof(*from)))
                return -EFAULT;
        unsafe_get_user(val, from, Efault);
  -     user_access_end();
  +     user_read_access_end();
        *dest = val;
        return 0;

Efault:

Thanks, I can confirm that this fixed the crash (changing both arms as pointed out by Christophe).

Signature · 3 lines
Andreas Schwab, [email protected]
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."