Skip to contentSkip to conversation

rseq: minor optimizations

PATCH Superseded Eric Dumazet 5 participants 26 messages Apr 13, 2021 – Apr 14, 2021 Restartable sequences support

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover rseq: minor optimizations

From: Eric Dumazet <[email protected]>

rseq is a heavy user of copy to/from user data in fast paths. This series tries to reduce the cost.

v2: Addressed Peter and Mathieu feedbacks, thanks !

Eric Dumazet (3):
rseq: optimize rseq_update_cpu_id()
rseq: remove redundant access_ok()
rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()

 kernel/rseq.c | 61 +++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 45 insertions(+), 16 deletions(-)
Signature · 1 line
2.31.1.295.g9ea45b61b8-goog
#1 lore raw

PATCH v2 1/3 rseq: optimize rseq_update_cpu_id()

From: Eric Dumazet <[email protected]>

Two put_user() in rseq_update_cpu_id() are replaced by a pair of unsafe_put_user() with appropriate surroundings.

This removes one stac/clac pair on x86 in fast path.

Cc Mathieu Desnoyers <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc "Paul E. McKenney" <[email protected]>
Cc Boqun Feng <[email protected]>
Cc Arjun Roy <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 kernel/rseq.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)
kernel/rseq.c +11 -4
index a4f86a9d6937cdfa2f13d1dcc9be863c1943d06f..f020f18f512a3f6241c3c9b104ce50e4d2c6188c 100644
@@ -84,13 +84,20 @@
 static int rseq_update_cpu_id(struct task_struct *t)
 {
 	u32 cpu_id = raw_smp_processor_id();
+	struct rseq __user *rseq = t->rseq;
 
-	if (put_user(cpu_id, &t->rseq->cpu_id_start))
-		return -EFAULT;
-	if (put_user(cpu_id, &t->rseq->cpu_id))
-		return -EFAULT;
+	if (!user_write_access_begin(rseq, sizeof(*rseq)))
+		goto efault;
+	unsafe_put_user(cpu_id, &rseq->cpu_id_start, efault_end);
+	unsafe_put_user(cpu_id, &rseq->cpu_id, efault_end);
+	user_write_access_end();
 	trace_rseq_update(t);
 	return 0;
+
+efault_end:
+	user_write_access_end();
+efault:
+	return -EFAULT;
 }
 
 static int rseq_reset_rseq_cpu_id(struct task_struct *t)
Signature · 1 line
2.31.1.295.g9ea45b61b8-goog
#2 lore raw

PATCH v2 2/3 rseq: remove redundant access_ok()

In reply to Eric Dumazet

From: Eric Dumazet <[email protected]>

After commit 8f2817701492 ("rseq: Use get_user/put_user rather than __get_user/__put_user") we no longer need
an access_ok() call from __rseq_handle_notify_resume()

Mathieu pointed out the same cleanup can be done
in rseq_syscall().

Cc Mathieu Desnoyers <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc "Paul E. McKenney" <[email protected]>
Cc Boqun Feng <[email protected]>
Cc Arjun Roy <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 kernel/rseq.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
kernel/rseq.c +1 -4
index f020f18f512a3f6241c3c9b104ce50e4d2c6188c..cfe01ab5253c1c424c0e8b25acbb6a8e1b41a5b6 100644
@@ -273,8 +273,6 @@ void __rseq_handle_notify_resume(struct ksignal *ksig, struct pt_regs *regs)
 
 	if (unlikely(t->flags & PF_EXITING))
 		return;
-	if (unlikely(!access_ok(t->rseq, sizeof(*t->rseq))))
-		goto error;
 	ret = rseq_ip_fixup(regs);
 	if (unlikely(ret < 0))
 		goto error;
@@ -301,8 +299,7 @@ void rseq_syscall(struct pt_regs *regs)
 
 	if (!t->rseq)
 		return;
-	if (!access_ok(t->rseq, sizeof(*t->rseq)) ||
-	    rseq_get_rseq_cs(t, &rseq_cs) || in_rseq_cs(ip, &rseq_cs))
+	if (rseq_get_rseq_cs(t, &rseq_cs) || in_rseq_cs(ip, &rseq_cs))
 		force_sig(SIGSEGV);
 }
 
Signature · 1 line
2.31.1.295.g9ea45b61b8-goog
#3 lore raw

PATCH v2 3/3 rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()

In reply to Eric Dumazet

From: Eric Dumazet <[email protected]>

Commit ec9c82e03a74 ("rseq: uapi: Declare rseq_cs field as union, update includes") added regressions for our servers.

Using copy_from_user() and clear_user() for 64bit values is suboptimal.

We can use faster put_user() and get_user().

32bit arches can be changed to use the ptr32 field, since the padding field must always be zero.

v2: added ideas from Peter and Mathieu about making this
generic, since my initial patch was only dealing with 64bit arches.

Cc Mathieu Desnoyers <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc "Paul E. McKenney" <[email protected]>
Cc Boqun Feng <[email protected]>
Cc Arjun Roy <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 kernel/rseq.c | 41 +++++++++++++++++++++++++++++++++--------
 1 file changed, 33 insertions(+), 8 deletions(-)
kernel/rseq.c +33 -8
index cfe01ab5253c1c424c0e8b25acbb6a8e1b41a5b6..f2eee3f7f5d330688c81cb2e57d47ca6b843873e 100644
@@ -119,23 +119,46 @@ static int rseq_reset_rseq_cpu_id(struct task_struct *t)
 	return 0;
 }
 
+#ifdef CONFIG_64BIT
+static int rseq_get_cs_ptr(struct rseq_cs __user **uptrp,
+			   const struct rseq __user *rseq)
+{
+	u64 ptr;
+
+	if (get_user(ptr, &rseq->rseq_cs.ptr64))
+		return -EFAULT;
+	*uptrp = (struct rseq_cs __user *)ptr;
+	return 0;
+}
+#else
+static int rseq_get_cs_ptr(struct rseq_cs __user **uptrp,
+			   const struct rseq __user *rseq)
+{
+	u32 ptr;
+
+	if (get_user(ptr, &rseq->rseq_cs.ptr.ptr32))
+		return -EFAULT;
+	*uptrp = (struct rseq_cs __user *)ptr;
+	return 0;
+}
+#endif
+
 static int rseq_get_rseq_cs(struct task_struct *t, struct rseq_cs *rseq_cs)
 {
 	struct rseq_cs __user *urseq_cs;
-	u64 ptr;
 	u32 __user *usig;
 	u32 sig;
 	int ret;
 
-	if (copy_from_user(&ptr, &t->rseq->rseq_cs.ptr64, sizeof(ptr)))
+	if (rseq_get_cs_ptr(&urseq_cs, t->rseq))
 		return -EFAULT;
-	if (!ptr) {
+	if (!urseq_cs) {
 		memset(rseq_cs, 0, sizeof(*rseq_cs));
 		return 0;
 	}
-	if (ptr >= TASK_SIZE)
+	if ((unsigned long)urseq_cs >= TASK_SIZE)
 		return -EINVAL;
-	urseq_cs = (struct rseq_cs __user *)(unsigned long)ptr;
+
 	if (copy_from_user(rseq_cs, urseq_cs, sizeof(*rseq_cs)))
 		return -EFAULT;
 
@@ -211,9 +234,11 @@ static int clear_rseq_cs(struct task_struct *t)
 	 *
 	 * Set rseq_cs to NULL.
 	 */
-	if (clear_user(&t->rseq->rseq_cs.ptr64, sizeof(t->rseq->rseq_cs.ptr64)))
-		return -EFAULT;
-	return 0;
+#ifdef CONFIG_64BIT
+	return put_user(0UL, &t->rseq->rseq_cs.ptr64);
+#else
+	return put_user(0UL, &t->rseq->rseq_cs.ptr.ptr32);
+#endif
 }
 
 /*
Signature · 1 line
2.31.1.295.g9ea45b61b8-goog
#4 lore raw

Re: [PATCH v2 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()

Quoted 8 lines · ----- On Apr 13, 2021, at 12:22 PM, Eric Dumazet [email protected] wrote:

From: Eric Dumazet <[email protected]>

Commit ec9c82e03a74 ("rseq: uapi: Declare rseq_cs field as union, update includes") added regressions for our servers.

Using copy_from_user() and clear_user() for 64bit values is suboptimal.

Quoted 8 lines

We can use faster put_user() and get_user().

32bit arches can be changed to use the ptr32 field, since the padding field must always be zero.

v2: added ideas from Peter and Mathieu about making this
generic, since my initial patch was only dealing with 64bit arches.

Ah, now I remember the reason why reading and clearing the entire 64-bit is important: it's because we don't want to allow user-space processes to use this change in behavior to figure out whether they are running on a 32-bit or in a 32-bit compat mode on a 64-bit kernel.

So although I'm fine with making 64-bit kernels faster, we'll want to keep updating the entire 64-bit ptr field on 32-bit kernels as well.

Thanks,

Mathieu

Quoted 87 lines
Cc Mathieu Desnoyers <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc "Paul E. McKenney" <[email protected]>
Cc Boqun Feng <[email protected]>
Cc Arjun Roy <[email protected]>
Cc Ingo Molnar <[email protected]>

---
kernel/rseq.c | 41 +++++++++++++++++++++++++++++++++--------

1 file changed, 33 insertions(+), 8 deletions(-)
kernel/rseq.c +0 -0

index cfe01ab5253c1c424c0e8b25acbb6a8e1b41a5b6..f2eee3f7f5d330688c81cb2e57d47ca6b843873e 100644

kernel/rseq.c +33 -9
@@ -119,23 +119,46 @@ static int rseq_reset_rseq_cpu_id(struct task_struct *t)
 	return 0;
 }
 
+#ifdef CONFIG_64BIT
+static int rseq_get_cs_ptr(struct rseq_cs __user **uptrp,
+			   const struct rseq __user *rseq)
+{
+	u64 ptr;
+
+	if (get_user(ptr, &rseq->rseq_cs.ptr64))
+		return -EFAULT;
+	*uptrp = (struct rseq_cs __user *)ptr;
+	return 0;
+}
+#else
+static int rseq_get_cs_ptr(struct rseq_cs __user **uptrp,
+			   const struct rseq __user *rseq)
+{
+	u32 ptr;
+
+	if (get_user(ptr, &rseq->rseq_cs.ptr.ptr32))
+		return -EFAULT;
+	*uptrp = (struct rseq_cs __user *)ptr;
+	return 0;
+}
+#endif
+
 static int rseq_get_rseq_cs(struct task_struct *t, struct rseq_cs *rseq_cs)
 {
 	struct rseq_cs __user *urseq_cs;
-	u64 ptr;
 	u32 __user *usig;
 	u32 sig;
 	int ret;
 
-	if (copy_from_user(&ptr, &t->rseq->rseq_cs.ptr64, sizeof(ptr)))
+	if (rseq_get_cs_ptr(&urseq_cs, t->rseq))
 		return -EFAULT;
-	if (!ptr) {
+	if (!urseq_cs) {
 		memset(rseq_cs, 0, sizeof(*rseq_cs));
 		return 0;
 	}
-	if (ptr >= TASK_SIZE)
+	if ((unsigned long)urseq_cs >= TASK_SIZE)
 		return -EINVAL;
-	urseq_cs = (struct rseq_cs __user *)(unsigned long)ptr;
+
 	if (copy_from_user(rseq_cs, urseq_cs, sizeof(*rseq_cs)))
 		return -EFAULT;
 
@@ -211,9 +234,11 @@ static int clear_rseq_cs(struct task_struct *t)
 	 *
 	 * Set rseq_cs to NULL.
 	 */
-	if (clear_user(&t->rseq->rseq_cs.ptr64, sizeof(t->rseq->rseq_cs.ptr64)))
-		return -EFAULT;
-	return 0;
+#ifdef CONFIG_64BIT
+	return put_user(0UL, &t->rseq->rseq_cs.ptr64);
+#else
+	return put_user(0UL, &t->rseq->rseq_cs.ptr.ptr32);
+#endif
 }
 
 /*
--

2.31.1.295.g9ea45b61b8-goog

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#5 lore raw

Re: [PATCH v2 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()

Quoted 25 lines · Mathieu Desnoyers
Quoted 8 lines · ----- On Apr 13, 2021, at 12:22 PM, Eric Dumazet [email protected] wrote:

From: Eric Dumazet <[email protected]>

Commit ec9c82e03a74 ("rseq: uapi: Declare rseq_cs field as union, update includes") added regressions for our servers.

Using copy_from_user() and clear_user() for 64bit values is suboptimal.

Quoted 8 lines

We can use faster put_user() and get_user().

32bit arches can be changed to use the ptr32 field, since the padding field must always be zero.

v2: added ideas from Peter and Mathieu about making this
generic, since my initial patch was only dealing with 64bit arches.

Ah, now I remember the reason why reading and clearing the entire 64-bit is important: it's because we don't want to allow user-space processes to use this change in behavior to figure out whether they are running on a 32-bit or in a 32-bit compat mode on a 64-bit kernel.

Quoted 5 lines

So although I'm fine with making 64-bit kernels faster, we'll want to keep updating the entire 64-bit ptr field on 32-bit kernels as well.

Thanks,

So... back to V1 then ?

#6 lore raw

Re: [PATCH v2 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()

Quoted 28 lines · Eric Dumazet
Quoted 25 lines · Mathieu Desnoyers
Quoted 8 lines · ----- On Apr 13, 2021, at 12:22 PM, Eric Dumazet [email protected] wrote:

From: Eric Dumazet <[email protected]>

Commit ec9c82e03a74 ("rseq: uapi: Declare rseq_cs field as union, update includes") added regressions for our servers.

Using copy_from_user() and clear_user() for 64bit values is suboptimal.

Quoted 8 lines

We can use faster put_user() and get_user().

32bit arches can be changed to use the ptr32 field, since the padding field must always be zero.

v2: added ideas from Peter and Mathieu about making this
generic, since my initial patch was only dealing with 64bit arches.

Ah, now I remember the reason why reading and clearing the entire 64-bit is important: it's because we don't want to allow user-space processes to use this change in behavior to figure out whether they are running on a 32-bit or in a 32-bit compat mode on a 64-bit kernel.

Quoted 7 lines
Quoted 5 lines

So although I'm fine with making 64-bit kernels faster, we'll want to keep updating the entire 64-bit ptr field on 32-bit kernels as well.

Thanks,

So... back to V1 then ?

Or add more stuff as in :

#ifdef CONFIG_64BIT
+ return put_user(0UL, &t->rseq->rseq_cs.ptr64);
+#else
+ return put_user(0UL, &t->rseq->rseq_cs.ptr.ptr32) |
put_user(0UL, &t->rseq->rseq_cs.ptr.padding);
+#endif

#7 lore raw

Re: [PATCH v2 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()

Quoted 33 lines · Eric Dumazet
Quoted 31 lines · Eric Dumazet
Quoted 28 lines · Mathieu Desnoyers
Quoted 8 lines · ----- On Apr 13, 2021, at 12:22 PM, Eric Dumazet [email protected] wrote:

From: Eric Dumazet <[email protected]>

Commit ec9c82e03a74 ("rseq: uapi: Declare rseq_cs field as union, update includes") added regressions for our servers.

Using copy_from_user() and clear_user() for 64bit values is suboptimal.

Quoted 8 lines

We can use faster put_user() and get_user().

32bit arches can be changed to use the ptr32 field, since the padding field must always be zero.

v2: added ideas from Peter and Mathieu about making this
generic, since my initial patch was only dealing with 64bit arches.

Ah, now I remember the reason why reading and clearing the entire 64-bit is important: it's because we don't want to allow user-space processes to use this change in behavior to figure out whether they are running on a 32-bit or in a 32-bit compat mode on a 64-bit kernel.

So although I'm fine with making 64-bit kernels faster, we'll want to keep updating the entire 64-bit ptr field on 32-bit kernels as well.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Thanks,

So... back to V1 then ?

Or add more stuff as in :

diff against v2, WDYT ?

kernel/rseq.c +0 -0
index f2eee3f7f5d330688c81cb2e57d47ca6b843873e..537b1f684efa11069990018ffa3642c209993011

100644

kernel/rseq.c +4 -0
@@ -136,6 +136,10 @@ static int rseq_get_cs_ptr(struct rseq_cs __user **uptrp,
 {
        u32 ptr;
 
+       if (get_user(ptr, &rseq->rseq_cs.ptr.padding))
+               return -EFAULT;
+       if (ptr)
+               return -EINVAL;
        if (get_user(ptr, &rseq->rseq_cs.ptr.ptr32))
                return -EFAULT;
        *uptrp = (struct rseq_cs __user *)ptr;
@@ -150,8 +154,9 @@ static int rseq_get_rseq_cs(struct task_struct *t,

struct rseq_cs *rseq_cs)

        u32 sig;
        int ret;

- if (rseq_get_cs_ptr(&urseq_cs, t->rseq))
- return -EFAULT;
+ ret = rseq_get_cs_ptr(&urseq_cs, t->rseq);
+ if (ret)

+               return ret;
        if (!urseq_cs) {
                memset(rseq_cs, 0, sizeof(*rseq_cs));
                return 0;
@@ -237,7 +242,8 @@ static int clear_rseq_cs(struct task_struct *t)
 #ifdef CONFIG_64BIT
        return put_user(0UL, &t->rseq->rseq_cs.ptr64);

#else
- return put_user(0UL, &t->rseq->rseq_cs.ptr.ptr32);
+ return put_user(0UL, &t->rseq->rseq_cs.ptr.ptr32) |
+ put_user(0UL, &t->rseq->rseq_cs.ptr.padding);
#endif
}

#8 lore raw

Re: [PATCH v2 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()

Quoted 34 lines · ----- On Apr 13, 2021, at 1:07 PM, Eric Dumazet [email protected] wrote:
Quoted 33 lines · Eric Dumazet
Quoted 31 lines · Eric Dumazet
Quoted 28 lines · Mathieu Desnoyers
Quoted 8 lines · ----- On Apr 13, 2021, at 12:22 PM, Eric Dumazet [email protected] wrote:

From: Eric Dumazet <[email protected]>

Commit ec9c82e03a74 ("rseq: uapi: Declare rseq_cs field as union, update includes") added regressions for our servers.

Using copy_from_user() and clear_user() for 64bit values is suboptimal.

Quoted 8 lines

We can use faster put_user() and get_user().

32bit arches can be changed to use the ptr32 field, since the padding field must always be zero.

v2: added ideas from Peter and Mathieu about making this
generic, since my initial patch was only dealing with 64bit arches.

Ah, now I remember the reason why reading and clearing the entire 64-bit is important: it's because we don't want to allow user-space processes to use this change in behavior to figure out whether they are running on a 32-bit or in a 32-bit compat mode on a 64-bit kernel.

So although I'm fine with making 64-bit kernels faster, we'll want to keep updating the entire 64-bit ptr field on 32-bit kernels as well.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Thanks,

So... back to V1 then ?

Or add more stuff as in :

diff against v2, WDYT ?

I like this approach slightly better, because it moves the preprocessor ifdefs into rseq_get_rseq_cs and clear_rseq_cs, while keeping the same behavior for a 32-bit process running on native 32-bit kernel and as compat task on a 64-bit kernel.

That being said, I don't expect anyone to care much about performance of 32-bit kernels, so we could use copy_from_user() on 32-bit kernels to remove special-cases in 32-bit specific code. This would eliminate the 32-bit specific "padding" read, and let the TASK_SIZE comparison handle the check for both 32-bit and 64-bit kernels.

As for clear_user(), I wonder whether we could simply keep using it, but change the clear_user() macro to figure out that it can use a faster 8-byte put_user ? I find it odd that performance optimizations which would be relevant elsewhere creep into the rseq code.

Thanks,

Mathieu

Quoted 40 lines
kernel/rseq.c +0 -0

index f2eee3f7f5d330688c81cb2e57d47ca6b843873e..537b1f684efa11069990018ffa3642c209993011 100644

kernel/rseq.c +7 -2
@@ -136,6 +136,10 @@ static int rseq_get_cs_ptr(struct rseq_cs __user **uptrp,
 {
       u32 ptr;
 
+       if (get_user(ptr, &rseq->rseq_cs.ptr.padding))
+               return -EFAULT;
+       if (ptr)
+               return -EINVAL;
       if (get_user(ptr, &rseq->rseq_cs.ptr.ptr32))
               return -EFAULT;
       *uptrp = (struct rseq_cs __user *)ptr;
@@ -150,8 +154,9 @@ static int rseq_get_rseq_cs(struct task_struct *t,
 struct rseq_cs *rseq_cs)
       u32 sig;
       int ret;
 
-       if (rseq_get_cs_ptr(&urseq_cs, t->rseq))
-               return -EFAULT;
+       ret = rseq_get_cs_ptr(&urseq_cs, t->rseq);
+       if (ret)
+               return ret;
       if (!urseq_cs) {
               memset(rseq_cs, 0, sizeof(*rseq_cs));
               return 0;
kernel/rseq.c +2 -1
@@ -237,7 +242,8 @@ static int clear_rseq_cs(struct task_struct *t)
 #ifdef CONFIG_64BIT
       return put_user(0UL, &t->rseq->rseq_cs.ptr64);
 #else
-       return put_user(0UL, &t->rseq->rseq_cs.ptr.ptr32);
+       return put_user(0UL, &t->rseq->rseq_cs.ptr.ptr32) |
+              put_user(0UL, &t->rseq->rseq_cs.ptr.padding);
 #endif
 }
Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#9 lore raw

Re: [PATCH v2 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()

Quoted 55 lines · Mathieu Desnoyers
Quoted 34 lines · ----- On Apr 13, 2021, at 1:07 PM, Eric Dumazet [email protected] wrote:
Quoted 33 lines · Eric Dumazet
Quoted 31 lines · Eric Dumazet
Quoted 28 lines · Mathieu Desnoyers
Quoted 8 lines · ----- On Apr 13, 2021, at 12:22 PM, Eric Dumazet [email protected] wrote:

From: Eric Dumazet <[email protected]>

Commit ec9c82e03a74 ("rseq: uapi: Declare rseq_cs field as union, update includes") added regressions for our servers.

Using copy_from_user() and clear_user() for 64bit values is suboptimal.

Quoted 8 lines

We can use faster put_user() and get_user().

32bit arches can be changed to use the ptr32 field, since the padding field must always be zero.

v2: added ideas from Peter and Mathieu about making this
generic, since my initial patch was only dealing with 64bit arches.

Ah, now I remember the reason why reading and clearing the entire 64-bit is important: it's because we don't want to allow user-space processes to use this change in behavior to figure out whether they are running on a 32-bit or in a 32-bit compat mode on a 64-bit kernel.

So although I'm fine with making 64-bit kernels faster, we'll want to keep updating the entire 64-bit ptr field on 32-bit kernels as well.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Thanks,

So... back to V1 then ?

Or add more stuff as in :

diff against v2, WDYT ?

I like this approach slightly better, because it moves the preprocessor ifdefs into rseq_get_rseq_cs and clear_rseq_cs, while keeping the same behavior for a 32-bit process running on native 32-bit kernel and as compat task on a 64-bit kernel.

That being said, I don't expect anyone to care much about performance of 32-bit kernels, so we could use copy_from_user() on 32-bit kernels to remove special-cases in 32-bit specific code. This would eliminate the 32-bit specific "padding" read, and let the TASK_SIZE comparison handle the check for both 32-bit and 64-bit kernels.

Quoted 4 lines

As for clear_user(), I wonder whether we could simply keep using it, but change the clear_user() macro to figure out that it can use a faster 8-byte put_user ? I find it odd that performance optimizations which would be relevant elsewhere creep into the rseq code.

clear_user() is a maze of arch-dependent macros/functions/assembly

I guess the same could be said from copy_in_user(), but apparently we removed special-casing, like in commit a41e0d754240fe8ca9c4f2070bf67e3b0228aa22

Definitely it seems odd having to carefully choose between multiple methods.

Quoted 50 lines

Thanks,

Mathieu

Quoted 32 lines
kernel/rseq.c +0 -0

index f2eee3f7f5d330688c81cb2e57d47ca6b843873e..537b1f684efa11069990018ffa3642c209993011 100644

kernel/rseq.c +7 -2
@@ -136,6 +136,10 @@ static int rseq_get_cs_ptr(struct rseq_cs __user **uptrp,
 {
       u32 ptr;
 
+       if (get_user(ptr, &rseq->rseq_cs.ptr.padding))
+               return -EFAULT;
+       if (ptr)
+               return -EINVAL;
       if (get_user(ptr, &rseq->rseq_cs.ptr.ptr32))
               return -EFAULT;
       *uptrp = (struct rseq_cs __user *)ptr;
@@ -150,8 +154,9 @@ static int rseq_get_rseq_cs(struct task_struct *t,
 struct rseq_cs *rseq_cs)
       u32 sig;
       int ret;
 
-       if (rseq_get_cs_ptr(&urseq_cs, t->rseq))
-               return -EFAULT;
+       ret = rseq_get_cs_ptr(&urseq_cs, t->rseq);
+       if (ret)
+               return ret;
       if (!urseq_cs) {
               memset(rseq_cs, 0, sizeof(*rseq_cs));
               return 0;

kernel/rseq.c static int clear_rseq_cs(struct task_struct *t) · line 248

Quoted 8 lines
kernel/rseq.c +2 -1
@@ -237,7 +242,8 @@ static int clear_rseq_cs(struct task_struct *t)
 #ifdef CONFIG_64BIT
       return put_user(0UL, &t->rseq->rseq_cs.ptr64);
 #else
-       return put_user(0UL, &t->rseq->rseq_cs.ptr.ptr32);
+       return put_user(0UL, &t->rseq->rseq_cs.ptr.ptr32) |
+              put_user(0UL, &t->rseq->rseq_cs.ptr.padding);
 #endif
 }

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

#10 lore raw

Re: [PATCH v2 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()

Quoted 70 lines · ----- On Apr 13, 2021, at 1:33 PM, Eric Dumazet [email protected] wrote:
Quoted 59 lines · Mathieu Desnoyers
Quoted 34 lines · ----- On Apr 13, 2021, at 1:07 PM, Eric Dumazet [email protected] wrote:
Quoted 33 lines · Eric Dumazet
Quoted 31 lines · Eric Dumazet
Quoted 28 lines · Mathieu Desnoyers
Quoted 8 lines · ----- On Apr 13, 2021, at 12:22 PM, Eric Dumazet [email protected] wrote:

From: Eric Dumazet <[email protected]>

Commit ec9c82e03a74 ("rseq: uapi: Declare rseq_cs field as union, update includes") added regressions for our servers.

Using copy_from_user() and clear_user() for 64bit values is suboptimal.

Quoted 8 lines

We can use faster put_user() and get_user().

32bit arches can be changed to use the ptr32 field, since the padding field must always be zero.

v2: added ideas from Peter and Mathieu about making this
generic, since my initial patch was only dealing with 64bit arches.

Ah, now I remember the reason why reading and clearing the entire 64-bit is important: it's because we don't want to allow user-space processes to use this change in behavior to figure out whether they are running on a 32-bit or in a 32-bit compat mode on a 64-bit kernel.

So although I'm fine with making 64-bit kernels faster, we'll want to keep updating the entire 64-bit ptr field on 32-bit kernels as well.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Thanks,

So... back to V1 then ?

Or add more stuff as in :

diff against v2, WDYT ?

I like this approach slightly better, because it moves the preprocessor ifdefs into
rseq_get_rseq_cs and clear_rseq_cs, while keeping the same behavior for a 32-bit process running on native 32-bit kernel and as compat task on a 64-bit kernel.

That being said, I don't expect anyone to care much about performance of 32-bit kernels, so we could use copy_from_user() on 32-bit kernels to remove special-cases
in 32-bit specific code. This would eliminate the 32-bit specific "padding" read, and
let the TASK_SIZE comparison handle the check for both 32-bit and 64-bit kernels.

Quoted 7 lines

As for clear_user(), I wonder whether we could simply keep using it, but change the
clear_user() macro to figure out that it can use a faster 8-byte put_user ? I find it
odd that performance optimizations which would be relevant elsewhere creep into the
rseq code.

Quoted 6 lines

clear_user() is a maze of arch-dependent macros/functions/assembly

I guess the same could be said from copy_in_user(), but apparently we removed special-casing, like in commit a41e0d754240fe8ca9c4f2070bf67e3b0228aa22

Definitely it seems odd having to carefully choose between multiple methods.

As long as the ifdefs are localized within clearly identified wrappers in the rseq code I don't mind doing the special-casing there.

The point which remains is that I don't think we want to optimize for speed on 32-bit architectures when it adds special-casing and complexity to the 32-bit build. I suspect there is less and less testing performed on 32-bit architectures nowadays, and it's good that as much code as possible is shared between 32-bit and 64-bit builds to share the test coverage.

Thanks,

Mathieu

Quoted 52 lines
Quoted 52 lines

Thanks,

Mathieu

Quoted 32 lines
kernel/rseq.c +0 -0

index f2eee3f7f5d330688c81cb2e57d47ca6b843873e..537b1f684efa11069990018ffa3642c209993011 100644

kernel/rseq.c +7 -2
@@ -136,6 +136,10 @@ static int rseq_get_cs_ptr(struct rseq_cs __user **uptrp,
 {
       u32 ptr;
 
+       if (get_user(ptr, &rseq->rseq_cs.ptr.padding))
+               return -EFAULT;
+       if (ptr)
+               return -EINVAL;
       if (get_user(ptr, &rseq->rseq_cs.ptr.ptr32))
               return -EFAULT;
       *uptrp = (struct rseq_cs __user *)ptr;
@@ -150,8 +154,9 @@ static int rseq_get_rseq_cs(struct task_struct *t,
 struct rseq_cs *rseq_cs)
       u32 sig;
       int ret;
 
-       if (rseq_get_cs_ptr(&urseq_cs, t->rseq))
-               return -EFAULT;
+       ret = rseq_get_cs_ptr(&urseq_cs, t->rseq);
+       if (ret)
+               return ret;
       if (!urseq_cs) {
               memset(rseq_cs, 0, sizeof(*rseq_cs));
               return 0;

kernel/rseq.c static int clear_rseq_cs(struct task_struct *t) · line 248

Quoted 8 lines
kernel/rseq.c +2 -1
@@ -237,7 +242,8 @@ static int clear_rseq_cs(struct task_struct *t)
 #ifdef CONFIG_64BIT
       return put_user(0UL, &t->rseq->rseq_cs.ptr64);
 #else
-       return put_user(0UL, &t->rseq->rseq_cs.ptr.ptr32);
+       return put_user(0UL, &t->rseq->rseq_cs.ptr.ptr32) |
+              put_user(0UL, &t->rseq->rseq_cs.ptr.padding);
 #endif
 }

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#11 lore raw

Re: [PATCH v2 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()

Quoted 11 lines · Mathieu Desnoyers

As long as the ifdefs are localized within clearly identified wrappers in the rseq code I don't mind doing the special-casing there.

The point which remains is that I don't think we want to optimize for speed on 32-bit architectures when it adds special-casing and complexity to the 32-bit build. I suspect there is less and less testing performed on 32-bit architectures nowadays, and it's good that as much code as possible is shared between 32-bit and 64-bit builds to share the test coverage.

Quite frankly V1 was fine, I can't really make it looking better.

Quoted 61 lines

Thanks,

Mathieu

Quoted 44 lines
Quoted 44 lines

Thanks,

Mathieu

Quoted 37 lines
kernel/rseq.c +0 -0

index f2eee3f7f5d330688c81cb2e57d47ca6b843873e..537b1f684efa11069990018ffa3642c209993011 100644

kernel/rseq.c +7 -2
@@ -136,6 +136,10 @@ static int rseq_get_cs_ptr(struct rseq_cs __user **uptrp,
 {
       u32 ptr;
 
+       if (get_user(ptr, &rseq->rseq_cs.ptr.padding))
+               return -EFAULT;
+       if (ptr)
+               return -EINVAL;
       if (get_user(ptr, &rseq->rseq_cs.ptr.ptr32))
               return -EFAULT;
       *uptrp = (struct rseq_cs __user *)ptr;
@@ -150,8 +154,9 @@ static int rseq_get_rseq_cs(struct task_struct *t,
 struct rseq_cs *rseq_cs)
       u32 sig;
       int ret;
 
-       if (rseq_get_cs_ptr(&urseq_cs, t->rseq))
-               return -EFAULT;
+       ret = rseq_get_cs_ptr(&urseq_cs, t->rseq);
+       if (ret)
+               return ret;
       if (!urseq_cs) {
               memset(rseq_cs, 0, sizeof(*rseq_cs));
               return 0;
kernel/rseq.c +1 -1
@@ -237,7 +242,8 @@ static int clear_rseq_cs(struct task_struct *t)
 #ifdef CONFIG_64BIT
       return put_user(0UL, &t->rseq->rseq_cs.ptr64);
 #else
-       return put_user(0UL, &t->rseq->rseq_cs.ptr.ptr32);
+       return put_user(0UL, &t->rseq->rseq_cs.ptr.ptr32) |
Quoted 8 lines
Quoted 8 lines

kernel/rseq.c static int clear_rseq_cs(struct task_struct *t) · line 248

Quoted 3 lines
kernel/rseq.c +1 -0
+              put_user(0UL, &t->rseq->rseq_cs.ptr.padding);
 #endif
 }

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

#12 lore raw

Re: [PATCH v2 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()

Quoted 8 lines · Eric Dumazet
Quoted 5 lines · Mathieu Desnoyers

As long as the ifdefs are localized within clearly identified wrappers in the rseq code I don't mind doing the special-casing there.

Quoted 20 lines
Quoted 6 lines

The point which remains is that I don't think we want to optimize for speed on 32-bit architectures when it adds special-casing and complexity to the 32-bit build. I suspect there is less and less testing performed on 32-bit architectures nowadays, and it's good that as much code as possible is shared between 32-bit and 64-bit builds to share the test coverage.

Quite frankly V1 was fine, I can't really make it looking better.

Quoted 11 lines

Thanks,

Mathieu

Quoted 7 lines
Quoted 7 lines

Thanks,

Mathieu

If we're special-casing 64-bit architectures anyways - unrolling the 32B copy_from_user() for struct rseq_cs appears to be roughly 5-10% savings on x86-64 when I measured it (well, in a microbenchmark, not in rseq_get_rseq_cs() directly). Perhaps that could be an additional avenue for improvement here.

-Arjun

Quoted 50 lines
Quoted 50 lines
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
kernel/rseq.c +0 -0

index f2eee3f7f5d330688c81cb2e57d47ca6b843873e..537b1f684efa11069990018ffa3642c209993011 100644

kernel/rseq.c +7 -2
@@ -136,6 +136,10 @@ static int rseq_get_cs_ptr(struct rseq_cs __user **uptrp,
 {
       u32 ptr;
 
+       if (get_user(ptr, &rseq->rseq_cs.ptr.padding))
+               return -EFAULT;
+       if (ptr)
+               return -EINVAL;
       if (get_user(ptr, &rseq->rseq_cs.ptr.ptr32))
               return -EFAULT;
       *uptrp = (struct rseq_cs __user *)ptr;
@@ -150,8 +154,9 @@ static int rseq_get_rseq_cs(struct task_struct *t,
 struct rseq_cs *rseq_cs)
       u32 sig;
       int ret;
 
-       if (rseq_get_cs_ptr(&urseq_cs, t->rseq))
-               return -EFAULT;
+       ret = rseq_get_cs_ptr(&urseq_cs, t->rseq);
+       if (ret)
+               return ret;
       if (!urseq_cs) {
               memset(rseq_cs, 0, sizeof(*rseq_cs));
               return 0;
kernel/rseq.c +1 -1
@@ -237,7 +242,8 @@ static int clear_rseq_cs(struct task_struct *t)
 #ifdef CONFIG_64BIT
       return put_user(0UL, &t->rseq->rseq_cs.ptr64);
 #else
-       return put_user(0UL, &t->rseq->rseq_cs.ptr.ptr32);
+       return put_user(0UL, &t->rseq->rseq_cs.ptr.ptr32) |
Quoted 8 lines
Quoted 8 lines

kernel/rseq.c static int clear_rseq_cs(struct task_struct *t) · line 248

Quoted 3 lines
kernel/rseq.c +1 -0
+              put_user(0UL, &t->rseq->rseq_cs.ptr.padding);
 #endif
 }

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

#13 lore raw

RE: [PATCH v2 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()

Quoted 5 lines

If we're special-casing 64-bit architectures anyways - unrolling the 32B copy_from_user() for struct rseq_cs appears to be roughly 5-10% savings on x86-64 when I measured it (well, in a microbenchmark, not in rseq_get_rseq_cs() directly). Perhaps that could be an additional avenue for improvement here.

The killer is usually 'user copy hardening'.
It significantly slows down sendmsg() and recvmsg(). I've got measurable performance improvements by using __copy_from_user() when the buffer since has already been checked - but isn't a compile-time constant.

There is also scope for using _get_user() when reading iovec[] (instead of copy_from_user()) and doing all the bound checks (etc) in the loop.
That gives a measurable improvement for writev("/dev/null"). I must sort those patches out again.

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#14 lore raw

Re: [PATCH v2 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()

Quoted 13 lines · David Laight
Quoted 6 lines

If we're special-casing 64-bit architectures anyways - unrolling the 32B copy_from_user() for struct rseq_cs appears to be roughly 5-10% savings on x86-64 when I measured it (well, in a microbenchmark, not in rseq_get_rseq_cs() directly). Perhaps that could be an additional avenue for improvement here.

The killer is usually 'user copy hardening'.
It significantly slows down sendmsg() and recvmsg(). I've got measurable performance improvements by using __copy_from_user() when the buffer since has already been checked - but isn't a compile-time constant.

Quoted 8 lines

There is also scope for using _get_user() when reading iovec[] (instead of copy_from_user()) and doing all the bound checks (etc) in the loop.
That gives a measurable improvement for writev("/dev/null"). I must sort those patches out again.

David

In this case I mean replacing copy_from_user(rseq_cs, urseq_cs, sizeof(*rseq_cs)) with 4 (x8B=32 total) unsafe_get_user() (wrapped in user_read_access_begin/end()) which I think would just bypass user copy hardening (as far as I can tell).

-Arjun

Quoted 3 lines

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#15 lore raw

RE: [PATCH v2 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()

From: Arjun Roy

Quoted 22 lines

Sent: 13 April 2021 23:04

Quoted 19 lines · David Laight
Quoted 6 lines

If we're special-casing 64-bit architectures anyways - unrolling the 32B copy_from_user() for struct rseq_cs appears to be roughly 5-10% savings on x86-64 when I measured it (well, in a microbenchmark, not in rseq_get_rseq_cs() directly). Perhaps that could be an additional avenue for improvement here.

The killer is usually 'user copy hardening'.
It significantly slows down sendmsg() and recvmsg(). I've got measurable performance improvements by using __copy_from_user() when the buffer since has already been checked - but isn't a compile-time constant.

There is also scope for using _get_user() when reading iovec[] (instead of copy_from_user()) and doing all the bound checks (etc) in the loop.
That gives a measurable improvement for writev("/dev/null"). I must sort those patches out again.

Quoted 7 lines
Quoted 2 lines

David

In this case I mean replacing copy_from_user(rseq_cs, urseq_cs, sizeof(*rseq_cs)) with 4 (x8B=32 total) unsafe_get_user() (wrapped in user_read_access_begin/end()) which I think would just bypass user copy hardening (as far as I can tell).

Yes that is one advantage over any of the get_user() calls. You also lose all the 'how shall we optimise this' checks in copy_from_user().

Repeated unsafe_get_user() calls are crying out for an optimisation. You get something like:

	failed = 0;
	copy();
	if (failed) goto error;
	copy();
	if (failed) goto error;

Where 'failed' is set by the fault handler.

This could be optimised to:

	failed = 0;
	copy();
	copy();
	if (failed) goto error;

Even if it faults on every invalid address it probably doesn't matter - no one cares about that path.

I've not really looked at how it could be achieved though.

It might be that the 'asm goto with outputs' variant manages to avoid the compare and jump.

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#16 lore raw

Re: [PATCH v2 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()

Quoted 45 lines · David Laight

From: Arjun Roy

Quoted 22 lines

Sent: 13 April 2021 23:04

Quoted 19 lines · David Laight
Quoted 6 lines

If we're special-casing 64-bit architectures anyways - unrolling the 32B copy_from_user() for struct rseq_cs appears to be roughly 5-10% savings on x86-64 when I measured it (well, in a microbenchmark, not in rseq_get_rseq_cs() directly). Perhaps that could be an additional avenue for improvement here.

The killer is usually 'user copy hardening'.
It significantly slows down sendmsg() and recvmsg(). I've got measurable performance improvements by using __copy_from_user() when the buffer since has already been checked - but isn't a compile-time constant.

There is also scope for using _get_user() when reading iovec[] (instead of copy_from_user()) and doing all the bound checks (etc) in the loop.
That gives a measurable improvement for writev("/dev/null"). I must sort those patches out again.

Quoted 7 lines
Quoted 2 lines

David

In this case I mean replacing copy_from_user(rseq_cs, urseq_cs, sizeof(*rseq_cs)) with 4 (x8B=32 total) unsafe_get_user() (wrapped in user_read_access_begin/end()) which I think would just bypass user copy hardening (as far as I can tell).

Yes that is one advantage over any of the get_user() calls. You also lose all the 'how shall we optimise this' checks in copy_from_user().

Repeated unsafe_get_user() calls are crying out for an optimisation. You get something like:

        failed = 0;
        copy();
        if (failed) goto error;
        copy();
        if (failed) goto error;

Where 'failed' is set by the fault handler.

Quoted 7 lines

This could be optimised to:

        failed = 0;
        copy();
        copy();
        if (failed) goto error;

Even if it faults on every invalid address it probably doesn't matter - no one cares about that path.

On which arch are you looking at ?

On x86_64 at least, code generation is just perfect.
Not even a conditional jmp, it is all handled by exceptions (if any)

stac
copy();
copy();
clac

<out_of_line>
efault_end: do error recovery.

Quoted 11 lines

I've not really looked at how it could be achieved though.

It might be that the 'asm goto with outputs' variant manages to avoid the compare and jump.

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#17 lore raw

RE: [PATCH v2 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()

From: Eric Dumazet

Quoted 1 line

Sent: 14 April 2021 17:00

...

Quoted 23 lines
Quoted 9 lines

Repeated unsafe_get_user() calls are crying out for an optimisation. You get something like:

        failed = 0;
        copy();
        if (failed) goto error;
        copy();
        if (failed) goto error;

Where 'failed' is set by the fault handler.

Quoted 7 lines

This could be optimised to:

        failed = 0;
        copy();
        copy();
        if (failed) goto error;

Even if it faults on every invalid address it probably doesn't matter - no one cares about that path.

On which arch are you looking at ?

On x86_64 at least, code generation is just perfect.
Not even a conditional jmp, it is all handled by exceptions (if any)

Quoted 8 lines

stac
copy();
copy();
clac

<out_of_line>
efault_end: do error recovery.

It will be x86_64.
I'm definitely seeing repeated tests of (IIRC) %rdx.

It may well be because the compiler isn't very new. Will be an Ubuntu build of 9.3.0.
Does that support 'asm goto with outputs' - which may be the difference.

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#18 lore raw

Re: [PATCH v2 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()

Quoted 36 lines · David Laight

From: Eric Dumazet

Quoted 1 line

Sent: 14 April 2021 17:00

...

Quoted 23 lines
Quoted 9 lines

Repeated unsafe_get_user() calls are crying out for an optimisation. You get something like:

        failed = 0;
        copy();
        if (failed) goto error;
        copy();
        if (failed) goto error;

Where 'failed' is set by the fault handler.

Quoted 7 lines

This could be optimised to:

        failed = 0;
        copy();
        copy();
        if (failed) goto error;

Even if it faults on every invalid address it probably doesn't matter - no one cares about that path.

On which arch are you looking at ?

On x86_64 at least, code generation is just perfect.
Not even a conditional jmp, it is all handled by exceptions (if any)

Quoted 8 lines

stac
copy();
copy();
clac

<out_of_line>
efault_end: do error recovery.

Quoted 8 lines

It will be x86_64.
I'm definitely seeing repeated tests of (IIRC) %rdx.

It may well be because the compiler isn't very new. Will be an Ubuntu build of 9.3.0.
Does that support 'asm goto with outputs' - which may be the difference.

Yep, probably. I am using some recent clang version.

Quoted 5 lines

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#19 lore raw

Re: [PATCH v2 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()

Quoted 41 lines · Eric Dumazet
Quoted 39 lines · David Laight

From: Eric Dumazet

Quoted 1 line

Sent: 14 April 2021 17:00

...

Quoted 23 lines
Quoted 9 lines

Repeated unsafe_get_user() calls are crying out for an optimisation. You get something like:

        failed = 0;
        copy();
        if (failed) goto error;
        copy();
        if (failed) goto error;

Where 'failed' is set by the fault handler.

Quoted 7 lines

This could be optimised to:

        failed = 0;
        copy();
        copy();
        if (failed) goto error;

Even if it faults on every invalid address it probably doesn't matter - no one cares about that path.

On which arch are you looking at ?

On x86_64 at least, code generation is just perfect.
Not even a conditional jmp, it is all handled by exceptions (if any)

Quoted 8 lines

stac
copy();
copy();
clac

<out_of_line>
efault_end: do error recovery.

It will be x86_64.
I'm definitely seeing repeated tests of (IIRC) %rdx.

Quoted 8 lines
Quoted 5 lines

It may well be because the compiler isn't very new. Will be an Ubuntu build of 9.3.0.
Does that support 'asm goto with outputs' - which may be the difference.

Yep, probably. I am using some recent clang version.

On x86-64 I can confirm, for me it (4 x unsafe_get_user()) compiles down to stac + lfence + 8 x mov + clac as straight line code. And results in roughly a 5%-10% speedup over copy_from_user().

-Arjun

Quoted 5 lines
Quoted 5 lines

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#20 lore raw

Re: [PATCH v2 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()

Quoted 49 lines · Arjun Roy
Quoted 47 lines · Eric Dumazet
Quoted 44 lines · David Laight

From: Eric Dumazet

Quoted 1 line

Sent: 14 April 2021 17:00

...

Quoted 23 lines
Quoted 9 lines

Repeated unsafe_get_user() calls are crying out for an optimisation. You get something like:

        failed = 0;
        copy();
        if (failed) goto error;
        copy();
        if (failed) goto error;

Where 'failed' is set by the fault handler.

Quoted 7 lines

This could be optimised to:

        failed = 0;
        copy();
        copy();
        if (failed) goto error;

Even if it faults on every invalid address it probably doesn't matter - no one cares about that path.

On which arch are you looking at ?

On x86_64 at least, code generation is just perfect.
Not even a conditional jmp, it is all handled by exceptions (if any)

Quoted 8 lines

stac
copy();
copy();
clac

<out_of_line>
efault_end: do error recovery.

It will be x86_64.
I'm definitely seeing repeated tests of (IIRC) %rdx.

It may well be because the compiler isn't very new. Will be an Ubuntu build of 9.3.0.
Does that support 'asm goto with outputs' - which may be the difference.

Quoted 7 lines
Quoted 2 lines

Yep, probably. I am using some recent clang version.

On x86-64 I can confirm, for me it (4 x unsafe_get_user()) compiles down to stac + lfence + 8 x mov + clac as straight line code. And results in roughly a 5%-10% speedup over copy_from_user().

But rseq_get_rseq_cs() would still need three different copies, with 3 stac+lfence+clac sequences.

Maybe we need to enclose all __rseq_handle_notify_resume() operations in a single section.

Quoted 8 lines

-Arjun

Quoted 5 lines
Quoted 5 lines

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#21 lore raw

Re: [PATCH v2 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()

Quoted 59 lines · Eric Dumazet
Quoted 49 lines · Arjun Roy
Quoted 47 lines · Eric Dumazet
Quoted 44 lines · David Laight

From: Eric Dumazet

Quoted 1 line

Sent: 14 April 2021 17:00

...

Quoted 23 lines
Quoted 9 lines

Repeated unsafe_get_user() calls are crying out for an optimisation. You get something like:

        failed = 0;
        copy();
        if (failed) goto error;
        copy();
        if (failed) goto error;

Where 'failed' is set by the fault handler.

Quoted 7 lines

This could be optimised to:

        failed = 0;
        copy();
        copy();
        if (failed) goto error;

Even if it faults on every invalid address it probably doesn't matter - no one cares about that path.

On which arch are you looking at ?

On x86_64 at least, code generation is just perfect.
Not even a conditional jmp, it is all handled by exceptions (if any)

Quoted 8 lines

stac
copy();
copy();
clac

<out_of_line>
efault_end: do error recovery.

It will be x86_64.
I'm definitely seeing repeated tests of (IIRC) %rdx.

It may well be because the compiler isn't very new. Will be an Ubuntu build of 9.3.0.
Does that support 'asm goto with outputs' - which may be the difference.

Quoted 7 lines
Quoted 2 lines

Yep, probably. I am using some recent clang version.

On x86-64 I can confirm, for me it (4 x unsafe_get_user()) compiles down to stac + lfence + 8 x mov + clac as straight line code. And results in roughly a 5%-10% speedup over copy_from_user().

Quoted 7 lines

But rseq_get_rseq_cs() would still need three different copies, with 3 stac+lfence+clac sequences.

Maybe we need to enclose all __rseq_handle_notify_resume() operations in a single section.

To provide a bit of further exposition on this point, if you do 4x unsafe_get_user() recall I mentioned a 5-10% improvement. On the other hand, 4x normal get_user() I saw something like a 100% (ie. doubling of sys time measured) regression.

I assume that's the fault of multiple stac+clac.

-Arjun

Quoted 13 lines
Quoted 13 lines

-Arjun

Quoted 5 lines
Quoted 5 lines

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#22 lore raw

Re: [PATCH v2 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()

Quoted 69 lines · Arjun Roy
Quoted 59 lines · Eric Dumazet
Quoted 49 lines · Arjun Roy
Quoted 47 lines · Eric Dumazet
Quoted 44 lines · David Laight

From: Eric Dumazet

Quoted 1 line

Sent: 14 April 2021 17:00

...

Quoted 23 lines
Quoted 9 lines

Repeated unsafe_get_user() calls are crying out for an optimisation. You get something like:

        failed = 0;
        copy();
        if (failed) goto error;
        copy();
        if (failed) goto error;

Where 'failed' is set by the fault handler.

Quoted 7 lines

This could be optimised to:

        failed = 0;
        copy();
        copy();
        if (failed) goto error;

Even if it faults on every invalid address it probably doesn't matter - no one cares about that path.

On which arch are you looking at ?

On x86_64 at least, code generation is just perfect.
Not even a conditional jmp, it is all handled by exceptions (if any)

Quoted 8 lines

stac
copy();
copy();
clac

<out_of_line>
efault_end: do error recovery.

It will be x86_64.
I'm definitely seeing repeated tests of (IIRC) %rdx.

It may well be because the compiler isn't very new. Will be an Ubuntu build of 9.3.0.
Does that support 'asm goto with outputs' - which may be the difference.

Quoted 7 lines
Quoted 2 lines

Yep, probably. I am using some recent clang version.

On x86-64 I can confirm, for me it (4 x unsafe_get_user()) compiles down to stac + lfence + 8 x mov + clac as straight line code. And results in roughly a 5%-10% speedup over copy_from_user().

Quoted 7 lines

But rseq_get_rseq_cs() would still need three different copies, with 3 stac+lfence+clac sequences.

Maybe we need to enclose all __rseq_handle_notify_resume() operations in a single section.

Quoted 6 lines

To provide a bit of further exposition on this point, if you do 4x unsafe_get_user() recall I mentioned a 5-10% improvement. On the other hand, 4x normal get_user() I saw something like a 100% (ie. doubling of sys time measured) regression.

I assume that's the fault of multiple stac+clac.

I was suggesting only using unsafe_get_user() and unsafe_put_user(), and one surrounding stac/clac

Basically what we had (partially) in our old Google kernels, before commit 8f2817701492 ("rseq: Use get_user/put_user rather than __get_user/__put_user")
but with all the needed modern stuff.

#23 lore raw

Re: [PATCH v2 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()

Quoted 79 lines · Eric Dumazet
Quoted 69 lines · Arjun Roy
Quoted 59 lines · Eric Dumazet
Quoted 49 lines · Arjun Roy
Quoted 47 lines · Eric Dumazet
Quoted 44 lines · David Laight

From: Eric Dumazet

Quoted 1 line

Sent: 14 April 2021 17:00

...

Quoted 23 lines
Quoted 9 lines

Repeated unsafe_get_user() calls are crying out for an optimisation. You get something like:

        failed = 0;
        copy();
        if (failed) goto error;
        copy();
        if (failed) goto error;

Where 'failed' is set by the fault handler.

Quoted 7 lines

This could be optimised to:

        failed = 0;
        copy();
        copy();
        if (failed) goto error;

Even if it faults on every invalid address it probably doesn't matter - no one cares about that path.

On which arch are you looking at ?

On x86_64 at least, code generation is just perfect.
Not even a conditional jmp, it is all handled by exceptions (if any)

Quoted 8 lines

stac
copy();
copy();
clac

<out_of_line>
efault_end: do error recovery.

It will be x86_64.
I'm definitely seeing repeated tests of (IIRC) %rdx.

It may well be because the compiler isn't very new. Will be an Ubuntu build of 9.3.0.
Does that support 'asm goto with outputs' - which may be the difference.

Quoted 7 lines
Quoted 2 lines

Yep, probably. I am using some recent clang version.

On x86-64 I can confirm, for me it (4 x unsafe_get_user()) compiles down to stac + lfence + 8 x mov + clac as straight line code. And results in roughly a 5%-10% speedup over copy_from_user().

Quoted 7 lines

But rseq_get_rseq_cs() would still need three different copies, with 3 stac+lfence+clac sequences.

Maybe we need to enclose all __rseq_handle_notify_resume() operations in a single section.

Quoted 6 lines

To provide a bit of further exposition on this point, if you do 4x unsafe_get_user() recall I mentioned a 5-10% improvement. On the other hand, 4x normal get_user() I saw something like a 100% (ie. doubling of sys time measured) regression.

I assume that's the fault of multiple stac+clac.

Quoted 7 lines

I was suggesting only using unsafe_get_user() and unsafe_put_user(), and one surrounding stac/clac

Basically what we had (partially) in our old Google kernels, before commit 8f2817701492 ("rseq: Use get_user/put_user rather than __get_user/__put_user")
but with all the needed modern stuff.

Yep - in agreement with that approach.

-Arjun

#24 lore raw

Re: [PATCH v2 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()

In reply to Eric Dumazet
Quoted 16 lines · ----- On Apr 13, 2021, at 2:22 PM, Eric Dumazet [email protected] wrote:
Quoted 5 lines · Mathieu Desnoyers

As long as the ifdefs are localized within clearly identified wrappers in the rseq code I don't mind doing the special-casing there.

Quoted 8 lines

The point which remains is that I don't think we want to optimize for speed on 32-bit architectures when it adds special-casing and complexity to the 32-bit build. I suspect there is less and less testing performed on 32-bit architectures
nowadays, and it's good that as much code as possible is shared between 32-bit and
64-bit builds to share the test coverage.

Quoted 1 line

Quite frankly V1 was fine, I can't really make it looking better.

Yes, I'm OK with V1 of that patch.

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#25 lore raw

Re: [PATCH v2 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()

In reply to Eric Dumazet
Quoted 27 lines · ----- On Apr 13, 2021, at 12:57 PM, Eric Dumazet [email protected] wrote:
Quoted 25 lines · Mathieu Desnoyers
Quoted 8 lines · ----- On Apr 13, 2021, at 12:22 PM, Eric Dumazet [email protected] wrote:

From: Eric Dumazet <[email protected]>

Commit ec9c82e03a74 ("rseq: uapi: Declare rseq_cs field as union, update includes") added regressions for our servers.

Using copy_from_user() and clear_user() for 64bit values is suboptimal.

Quoted 8 lines

We can use faster put_user() and get_user().

32bit arches can be changed to use the ptr32 field, since the padding field must always be zero.

v2: added ideas from Peter and Mathieu about making this
generic, since my initial patch was only dealing with 64bit arches.

Ah, now I remember the reason why reading and clearing the entire 64-bit is important: it's because we don't want to allow user-space processes to use this change in behavior to figure out whether they are running on a 32-bit or in a 32-bit compat mode on a 64-bit kernel.

Quoted 7 lines
Quoted 5 lines

So although I'm fine with making 64-bit kernels faster, we'll want to keep updating the entire 64-bit ptr field on 32-bit kernels as well.

Thanks,

So... back to V1 then ?

In terms of behavior, yes. And it's probably the "easy" fix, but I hate that it adds lots of preprocessor ifdefs into the rseq code.

But this would require auditing get_user()/put_user() for each architecture supported by rseq to ensure they support 8-byte load/store. And it would become an added burden on architecture maintainers wishing to add rseq support for their architecture.

One alternative would be to implement rseq_get_user_u64 and rseq_put_user_u64 wrappers as static functions within rseq.c to hide the preprocessor ifdeffery from the higher-level code. I try very hard to avoid mixing preprocessor ifdefs with C code logic whenever I can.

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com