rseq: minor optimizations
v3 cover rseq: minor optimizations
From: Eric Dumazet <[email protected]>
rseq is a heavy user of copy to/from user data in fast paths. This series tries to reduce the cost.
v3: Third patch going back to v1 (only deal with 64bit arches)
v2: Addressed Peter and Mathieu feedbacks, thanks !
Eric Dumazet (3):
rseq: optimize rseq_update_cpu_id()
rseq: remove redundant access_ok()
rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()
kernel/rseq.c | 29 +++++++++++++++++++++-------- 1 file changed, 21 insertions(+), 8 deletions(-)
Signature · 1 line
2.31.1.295.g9ea45b61b8-goog
PATCH v3 1/3 rseq: optimize rseq_update_cpu_id()
From: Eric Dumazet <[email protected]>
Two put_user() in rseq_update_cpu_id() are replaced by a pair of unsafe_put_user() with appropriate surroundings.
This removes one stac/clac pair on x86 in fast path.
--- kernel/rseq.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
kernel/rseq.c +11 -4
@@ -84,13 +84,20 @@static int rseq_update_cpu_id (struct task_struct *t) {u32 cpu_id =raw_smp_processor_id (); +struct rseq __user *rseq = t->rseq; -if (put_user (cpu_id, &t->rseq->cpu_id_start)) -return -EFAULT ; -if (put_user (cpu_id, &t->rseq->cpu_id)) -return -EFAULT ; +if (!user_write_access_begin (rseq,sizeof (*rseq))) + goto efault; +unsafe_put_user (cpu_id, &rseq->cpu_id_start, efault_end); +unsafe_put_user (cpu_id, &rseq->cpu_id, efault_end); +user_write_access_end ();trace_rseq_update (t);return 0 ; + +efault_end: +user_write_access_end (); +efault: +return -EFAULT ; }static int rseq_reset_rseq_cpu_id (struct task_struct *t)
Signature · 1 line
2.31.1.295.g9ea45b61b8-goog
[tip: sched/core] rseq: Optimize rseq_update_cpu_id()
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 60af388d23889636011488c42763876bcdda3eab Gitweb: https://git.kernel.org/tip/60af388d23889636011488c42763876bcdda3eab Author: Eric Dumazet <[email protected]> AuthorDate: Tue, 13 Apr 2021 13:33:50 -07:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 14 Apr 2021 18:04:09 +02:00
rseq: Optimize rseq_update_cpu_id()
Two put_user() in rseq_update_cpu_id() are replaced by a pair of unsafe_put_user() with appropriate surroundings.
This removes one stac/clac pair on x86 in fast path.
--- kernel/rseq.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
kernel/rseq.c +11 -4
@@ -84,13 +84,20 @@static int rseq_update_cpu_id (struct task_struct *t) {u32 cpu_id =raw_smp_processor_id (); +struct rseq __user *rseq = t->rseq; -if (put_user (cpu_id, &t->rseq->cpu_id_start)) -return -EFAULT ; -if (put_user (cpu_id, &t->rseq->cpu_id)) -return -EFAULT ; +if (!user_write_access_begin (rseq,sizeof (*rseq))) + goto efault; +unsafe_put_user (cpu_id, &rseq->cpu_id_start, efault_end); +unsafe_put_user (cpu_id, &rseq->cpu_id, efault_end); +user_write_access_end ();trace_rseq_update (t);return 0 ; + +efault_end: +user_write_access_end (); +efault: +return -EFAULT ; }static int rseq_reset_rseq_cpu_id (struct task_struct *t)
PATCH v3 2/3 rseq: remove redundant access_ok()
In reply to Eric DumazetFrom: Eric Dumazet <[email protected]>
After commit 8f2817701492 ("rseq: Use get_user/put_user rather than __get_user/__put_user") we no longer need
an access_ok() call from __rseq_handle_notify_resume()
Mathieu pointed out the same cleanup can be done
in rseq_syscall().
--- kernel/rseq.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
kernel/rseq.c +1 -4
@@ -273,8 +273,6 @@ void __rseq_handle_notify_resume(struct ksignal *ksig, struct pt_regs *regs)if (unlikely (t->flags &PF_EXITING ))return ; -if (unlikely (!access_ok (t->rseq,sizeof (*t->rseq)))) - goto error; ret =rseq_ip_fixup (regs);if (unlikely (ret <0 )) goto error;
@@ -301,8 +299,7 @@ void rseq_syscall(struct pt_regs *regs)if (!t->rseq)return ; -if (!access_ok (t->rseq,sizeof (*t->rseq)) || -rseq_get_rseq_cs (t, &rseq_cs) ||in_rseq_cs (ip, &rseq_cs)) +if (rseq_get_rseq_cs (t, &rseq_cs) ||in_rseq_cs (ip, &rseq_cs))force_sig (SIGSEGV ); }
Signature · 1 line
2.31.1.295.g9ea45b61b8-goog
[tip: sched/core] rseq: Remove redundant access_ok()
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 0ed96051531ecc6965f6456d25b19b9b6bdb5c28 Gitweb: https://git.kernel.org/tip/0ed96051531ecc6965f6456d25b19b9b6bdb5c28 Author: Eric Dumazet <[email protected]> AuthorDate: Tue, 13 Apr 2021 13:33:51 -07:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 14 Apr 2021 18:04:09 +02:00
rseq: Remove redundant access_ok()
After commit 8f2817701492 ("rseq: Use get_user/put_user rather than __get_user/__put_user") we no longer need
an access_ok() call from __rseq_handle_notify_resume()
Mathieu pointed out the same cleanup can be done
in rseq_syscall().
--- kernel/rseq.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
kernel/rseq.c +1 -4
@@ -273,8 +273,6 @@ void __rseq_handle_notify_resume(struct ksignal *ksig, struct pt_regs *regs)if (unlikely (t->flags &PF_EXITING ))return ; -if (unlikely (!access_ok (t->rseq,sizeof (*t->rseq)))) - goto error; ret =rseq_ip_fixup (regs);if (unlikely (ret <0 )) goto error;
@@ -301,8 +299,7 @@ void rseq_syscall(struct pt_regs *regs)if (!t->rseq)return ; -if (!access_ok (t->rseq,sizeof (*t->rseq)) || -rseq_get_rseq_cs (t, &rseq_cs) ||in_rseq_cs (ip, &rseq_cs)) +if (rseq_get_rseq_cs (t, &rseq_cs) ||in_rseq_cs (ip, &rseq_cs))force_sig (SIGSEGV ); }
PATCH v3 3/3 rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()
In reply to Eric DumazetFrom: Eric Dumazet <[email protected]>
Commit ec9c82e03a74 ("rseq: uapi: Declare rseq_cs field as union, update includes") added regressions for our servers.
Using copy_from_user() and clear_user() for 64bit values is suboptimal.
We can use faster put_user() and get_user() on 64bit arches.
--- kernel/rseq.c | 9 +++++++++ 1 file changed, 9 insertions(+)
kernel/rseq.c +9 -0
@@ -127,8 +127,13 @@ static int rseq_get_rseq_cs(struct task_struct *t, struct rseq_cs *rseq_cs)u32 sig;int ret; +#ifdef CONFIG_64BIT +if (get_user (ptr, &t->rseq->rseq_cs.ptr64)) +return -EFAULT ; +#else if (copy_from_user (&ptr, &t->rseq->rseq_cs.ptr64,sizeof (ptr)))return -EFAULT ; +#endif if (!ptr) {memset (rseq_cs,0 ,sizeof (*rseq_cs));return 0 ;
@@ -211,9 +216,13 @@ static int clear_rseq_cs(struct task_struct *t)* * Set rseq_cs to NULL. */ +#ifdef CONFIG_64BIT +return put_user (0UL , &t->rseq->rseq_cs.ptr64); +#else if (clear_user (&t->rseq->rseq_cs.ptr64,sizeof (t->rseq->rseq_cs.ptr64)))return -EFAULT ;return 0 ; +#endif }/*
Signature · 1 line
2.31.1.295.g9ea45b61b8-goog
[tip: sched/core] rseq: Optimise rseq_get_rseq_cs() and clear_rseq_cs()
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 5e0ccd4a3b01c5a71732a13186ca110a138516ea Gitweb: https://git.kernel.org/tip/5e0ccd4a3b01c5a71732a13186ca110a138516ea Author: Eric Dumazet <[email protected]> AuthorDate: Tue, 13 Apr 2021 13:33:52 -07:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 14 Apr 2021 18:04:09 +02:00
rseq: Optimise rseq_get_rseq_cs() and clear_rseq_cs()
Commit ec9c82e03a74 ("rseq: uapi: Declare rseq_cs field as union, update includes") added regressions for our servers.
Using copy_from_user() and clear_user() for 64bit values is suboptimal.
We can use faster put_user() and get_user() on 64bit arches.
--- kernel/rseq.c | 9 +++++++++ 1 file changed, 9 insertions(+)
kernel/rseq.c +9 -0
@@ -127,8 +127,13 @@ static int rseq_get_rseq_cs(struct task_struct *t, struct rseq_cs *rseq_cs)u32 sig;int ret; +#ifdef CONFIG_64BIT +if (get_user (ptr, &t->rseq->rseq_cs.ptr64)) +return -EFAULT ; +#else if (copy_from_user (&ptr, &t->rseq->rseq_cs.ptr64,sizeof (ptr)))return -EFAULT ; +#endif if (!ptr) {memset (rseq_cs,0 ,sizeof (*rseq_cs));return 0 ;
@@ -211,9 +216,13 @@ static int clear_rseq_cs(struct task_struct *t)* * Set rseq_cs to NULL. */ +#ifdef CONFIG_64BIT +return put_user (0UL , &t->rseq->rseq_cs.ptr64); +#else if (clear_user (&t->rseq->rseq_cs.ptr64,sizeof (t->rseq->rseq_cs.ptr64)))return -EFAULT ;return 0 ; +#endif }/*
Quoted 7 lines · ----- On Apr 13, 2021, at 4:33 PM, Eric Dumazet [email protected] wrote:
From: Eric Dumazet <[email protected]>
rseq is a heavy user of copy to/from user data in fast paths. This series tries to reduce the cost.
v3: Third patch going back to v1 (only deal with 64bit arches)
v2: Addressed Peter and Mathieu feedbacks, thanks !
For the whole series:
Thanks Eric!
Mathieu
Quoted 11 lines
Eric Dumazet (3):
rseq: optimizerseq_update_cpu_id()
rseq: remove redundantaccess_ok()
rseq: optimiserseq_get_rseq_cs()andclear_rseq_cs()
kernel/rseq.c| 29 +++++++++++++++++++++--------1 file changed, 21 insertions(+), 8 deletions(-)--
2.31.1.295.g9ea45b61b8-goog
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Peter Zijlstra
maintainerQuoted 7 lines · Mathieu Desnoyers
Quoted 5 lines · ----- On Apr 13, 2021, at 4:33 PM, Eric Dumazet [email protected] wrote:
From: Eric Dumazet <[email protected]>
rseq is a heavy user of copy to/from user data in fast paths. This series tries to reduce the cost.
Quoted 6 lines
Quoted 2 lines
v3: Third patch going back to v1 (only deal with 64bit arches)
v2: Addressed Peter and Mathieu feedbacks, thanks !For the whole series:
Reviewed-by Mathieu Desnoyers <[email protected]>
Thanks!