prlimit and set/getpriority tasklist_lock optimizations
v3 cover prlimit and set/getpriority tasklist_lock optimizations
The tasklist_lock popped up as a scalability bottleneck on some testing workloads. The readlocks in do_prlimit and set/getpriority are not necessary in all cases.
Based on a cycles profile, it looked like ~87% of the time was spent in the kernel, ~42% of which was just trying to get *some* spinlock (queued_spin_lock_slowpath, not necessarily the tasklist_lock).
The big offenders (with rough percentages in cycles of the overall trace):
- do_wait 11%
- setpriority 8% (this patchset)
- kill 8%
- do_exit 5%
- clone 3%
- prlimit64 2% (this patchset)
- getrlimit 1% (this patchset)
I can't easily test this patchset on the original workload for various reasons. Instead, I used the microbenchmark below to at least verify there was some improvement. This patchset had a 28% speedup (12% from baseline to set/getprio, then another 14% for prlimit).
One interesting thing is that my libc's getrlimit() was calling prlimit64, so hoisting the read_lock(tasklist_lock) into sys_prlimit64 had no effect - it essentially optimized the older syscalls only. I didn't do that in this patchset, but figured I'd mention it since it was an option from the previous patch's discussion.
v2: https://lore.kernel.org/lkml/[email protected]/
- update_rlimit_cpu on the group_leader instead of for_each_thread.
- update_rlimit_cpu still returns 0 or -ESRCH, even though we don't care about the error here. it felt safer that way in case someone uses that function again.
v1: https://lore.kernel.org/lkml/[email protected]/
#include <sys/resource.h>
#include <sys/time.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main(int argc, char **argv)
{
pid_t child;
struct rlimit rlim[1];
fork(); fork(); fork(); fork(); fork(); fork();
for (int i = 0; i < 5000; i++) {
child = fork();
if (child < 0)
exit(1);
if (child > 0) {
usleep(1000);
kill(child, SIGTERM);
waitpid(child, NULL, 0);
} else {
for (;;) {
setpriority(PRIO_PROCESS, 0,
getpriority(PRIO_PROCESS, 0));
getrlimit(RLIMIT_CPU, rlim);
}
}
}
return 0;
}Barret Rhoden (3):
setpriority: only grab the tasklist_lock for PRIO_PGRP
prlimit: make do_prlimit() static
prlimit: do not grab the tasklist_lock
include/linux/posix-timers.h | 2 +- include/linux/resource.h | 2 - kernel/sys.c | 127 +++++++++++++++++---------------- kernel/time/posix-cpu-timers.c | 12 +++- 4 files changed, 76 insertions(+), 67 deletions(-)
Signature · 1 line
2.34.1.448.ga2b2bfdf31-goog
PATCH v3 1/3 setpriority: only grab the tasklist_lock for PRIO_PGRP
The tasklist_lock is necessary only for PRIO_PGRP for both setpriority() and getpriority().
Unnecessarily grabbing the tasklist_lock can be a scalability bottleneck for workloads that also must grab the tasklist_lock for waiting, killing, and cloning.
This change resulted in a 12% speedup on a microbenchmark where parents kill and wait on their children, and children getpriority, setpriority, and getrlimit.
--- kernel/sys.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
kernel/sys.c +4 -4
@@ -220,7 +220,6 @@ SYSCALL_DEFINE3(setpriority, int, which, int, who, int, niceval) niceval =MAX_NICE ;rcu_read_lock (); -read_lock (&tasklist_lock);switch (which) {case PRIO_PROCESS :if (who)
@@ -231,6 +230,7 @@ SYSCALL_DEFINE3(setpriority, int, which, int, who, int, niceval) error =set_one_prio (p, niceval, error);break ;case PRIO_PGRP : +read_lock (&tasklist_lock);if (who) pgrp =find_vpid (who);else
@@ -238,6 +238,7 @@ SYSCALL_DEFINE3(setpriority, int, which, int, who, int, niceval) do_each_pid_thread(pgrp, PIDTYPE_PGID, p) { error =set_one_prio (p, niceval, error); }while_each_pid_thread (pgrp,PIDTYPE_PGID , p); +read_unlock (&tasklist_lock);break ;case PRIO_USER : uid =make_kuid (cred->user_ns, who);
@@ -258,7 +259,6 @@ SYSCALL_DEFINE3(setpriority, int, which, int, who, int, niceval)break ; } out_unlock: -read_unlock (&tasklist_lock);rcu_read_unlock (); out:return error;
@@ -283,7 +283,6 @@ SYSCALL_DEFINE2(getpriority, int, which, int, who)return -EINVAL ;rcu_read_lock (); -read_lock (&tasklist_lock);switch (which) {case PRIO_PROCESS :if (who)
@@ -297,6 +296,7 @@ SYSCALL_DEFINE2(getpriority, int, which, int, who) }break ;case PRIO_PGRP : +read_lock (&tasklist_lock);if (who) pgrp =find_vpid (who);else
@@ -306,6 +306,7 @@ SYSCALL_DEFINE2(getpriority, int, which, int, who)if (niceval > retval) retval = niceval; }while_each_pid_thread (pgrp,PIDTYPE_PGID , p); +read_unlock (&tasklist_lock);break ;case PRIO_USER : uid =make_kuid (cred->user_ns, who);
@@ -329,7 +330,6 @@ SYSCALL_DEFINE2(getpriority, int, which, int, who)break ; } out_unlock: -read_unlock (&tasklist_lock);rcu_read_unlock ();return retval;
Signature · 1 line
2.34.1.448.ga2b2bfdf31-goog
PATCH v3 3/3 prlimit: do not grab the tasklist_lock
In reply to Barret RhodenUnnecessarily grabbing the tasklist_lock can be a scalability bottleneck for workloads that also must grab the tasklist_lock for waiting, killing, and cloning.
The tasklist_lock was grabbed to protect tsk->sighand from disappearing (becoming NULL). tsk->signal was already protected by holding a reference to tsk.
update_rlimit_cpu() assumed tsk->sighand != NULL. With this commit, it attempts to lock_task_sighand(). However, this means that update_rlimit_cpu() can fail. This only happens when a task is exiting. Note that during exec, sighand may *change*, but it will not be NULL.
Prior to this commit, the do_prlimit() ensured that update_rlimit_cpu() would not fail by read locking the tasklist_lock and checking tsk->sighand != NULL.
If update_rlimit_cpu() fails, there may be other tasks that are not exiting that share tsk->signal. However, the group_leader is the last task to be released, so if we cannot update_rlimit_cpu(group_leader), then the entire process is exiting.
The only other caller of update_rlimit_cpu() is selinux_bprm_committing_creds(). It has tsk == current, so update_rlimit_cpu() cannot fail (current->sighand cannot disappear until current exits).
This change resulted in a 14% speedup on a microbenchmark where parents kill and wait on their children, and children getpriority, setpriority, and getrlimit.
--- include/linux/posix-timers.h | 2 +- kernel/sys.c | 25 ++++++++++++++----------- kernel/time/posix-cpu-timers.c | 12 +++++++++--- 3 files changed, 24 insertions(+), 15 deletions(-)
include/linux/posix-timers.h +1 -1
@@ -253,7 +253,7 @@ void posix_cpu_timers_exit_group(struct task_struct *task);void set_process_cpu_timer (struct task_struct *task,unsigned int clock_idx,u64 *newval,u64 *oldval); -void update_rlimit_cpu (struct task_struct *task,unsigned long rlim_new); +int update_rlimit_cpu (struct task_struct *task,unsigned long rlim_new);void posixtimer_rearm (struct kernel_siginfo *info);#endif
kernel/sys.c +14 -11
@@ -1432,13 +1432,7 @@ static int do_prlimit(struct task_struct *tsk, unsigned int resource,return -EPERM ; } -/* protect tsk->signal and tsk->sighand from disappearing */ -read_lock (&tasklist_lock); -if (!tsk->sighand) { - retval = -ESRCH ; - goto out; - } - +/* Holding a refcount on tsk protects tsk->signal from disappearing. */ rlim = tsk->signal->rlim + resource;task_lock (tsk->group_leader);if (new_rlim) {
@@ -1467,10 +1461,19 @@ static int do_prlimit(struct task_struct *tsk, unsigned int resource,*/ if (!retval && new_rlim && resource ==RLIMIT_CPU && new_rlim->rlim_cur !=RLIM_INFINITY && -IS_ENABLED (CONFIG_POSIX_TIMERS )) -update_rlimit_cpu (tsk, new_rlim->rlim_cur); -out: -read_unlock (&tasklist_lock); +IS_ENABLED (CONFIG_POSIX_TIMERS )) { +/* +* update_rlimit_cpu can fail if the task is exiting, but there +* may be other tasks in the thread group that are not exiting, +* and they need their cpu timers adjusted. +* +* The group_leader is the last task to be released, so if we +* cannot update_rlimit_cpu on it, then the entire process is +* exiting and we do not need to update at all. +*/ +update_rlimit_cpu (tsk->group_leader, new_rlim->rlim_cur); + } +return retval; }
kernel/time/posix-cpu-timers.c +9 -3
@@ -34,14 +34,20 @@ void posix_cputimers_group_init(struct posix_cputimers *pct, u64 cpu_limit)* tsk->signal->posix_cputimers.bases[clock].nextevt expiration cache if * necessary. Needs siglock protection since other code may update the * expiration cache as well. +* +* Returns 0 on success, -ESRCH on failure. Can fail if the task is exiting and +* we cannot lock_task_sighand. Cannot fail if task is current. */ -void update_rlimit_cpu (struct task_struct *task,unsigned long rlim_new) +int update_rlimit_cpu (struct task_struct *task,unsigned long rlim_new) {u64 nsecs = rlim_new *NSEC_PER_SEC ; +unsigned long irq_fl; -spin_lock_irq (&task->sighand->siglock); +if (!lock_task_sighand (task, &irq_fl)) +return -ESRCH ;set_process_cpu_timer (task,CPUCLOCK_PROF , &nsecs,NULL ); -spin_unlock_irq (&task->sighand->siglock); +unlock_task_sighand (task, &irq_fl); +return 0 ; }/*
Signature · 1 line
2.34.1.448.ga2b2bfdf31-goog
PATCH v3 2/3 prlimit: make do_prlimit() static
In reply to Barret RhodenThere are no other callers in the kernel.
Fixed up a comment format and whitespace issue when moving do_prlimit() higher in sys.c.
--- include/linux/resource.h | 2 - kernel/sys.c | 116 ++++++++++++++++++++------------------- 2 files changed, 59 insertions(+), 59 deletions(-)
include/linux/resource.h +0 -2
@@ -8,7 +8,5 @@struct task_struct ;void getrusage (struct task_struct *p,int who,struct rusage *ru); -int do_prlimit (struct task_struct *tsk,unsigned int resource, -struct rlimit *new_rlim,struct rlimit *old_rlim);#endif
kernel/sys.c +59 -57
128 lines not on this page · raw message
Signature · 1 line
2.34.1.448.ga2b2bfdf31-goog
Linus,
Please pull the prlimit-tasklist_lock-for-v5.18 tag from the git tree:
git://git.kernel.org/pub/scm/linux/kernel/git/ebiederm/user-namespace.git prlimit-tasklist_lock-for-v5.18
HEAD: 18c91bb2d87268d23868bf13508f5bc9cf04e89a prlimit: do not grab the tasklist_lock
From: Barret Rhoden <[email protected]>
The tasklist_lock popped up as a scalability bottleneck on some testing workloads. The readlocks in do_prlimit and set/getpriority are not necessary in all cases.
Based on a cycles profile, it looked like ~87% of the time was spent in the kernel, ~42% of which was just trying to get *some* spinlock (queued_spin_lock_slowpath, not necessarily the tasklist_lock).
The big offenders (with rough percentages in cycles of the overall trace):
- do_wait 11%
- setpriority 8% (this patchset)
- kill 8%
- do_exit 5%
- clone 3%
- prlimit64 2% (this patchset)
- getrlimit 1% (this patchset)
I can't easily test this patchset on the original workload for various reasons. Instead, I used the microbenchmark below to at least verify there was some improvement. This patchset had a 28% speedup (12% from baseline to set/getprio, then another 14% for prlimit).
One interesting thing is that my libc's getrlimit() was calling prlimit64, so hoisting the read_lock(tasklist_lock) into sys_prlimit64 had no effect - it essentially optimized the older syscalls only. I didn't do that in this patchset, but figured I'd mention it since it was an option from the previous patch's discussion.
v3: https://lkml.kernel.org/r/[email protected]
v2: https://lore.kernel.org/lkml/[email protected]/
- update_rlimit_cpu on the group_leader instead of for_each_thread.
- update_rlimit_cpu still returns 0 or -ESRCH, even though we don't care about the error here. it felt safer that way in case someone uses that function again.
v1: https://lore.kernel.org/lkml/[email protected]/
int main(int argc, char **argv)
{
pid_t child;
struct rlimit rlim[1];
fork(); fork(); fork(); fork(); fork(); fork();
for (int i = 0; i < 5000; i++) {
child = fork();
if (child < 0)
exit(1);
if (child > 0) {
usleep(1000);
kill(child, SIGTERM);
waitpid(child, NULL, 0);
} else {
for (;;) {
setpriority(PRIO_PROCESS, 0,
getpriority(PRIO_PROCESS, 0));
getrlimit(RLIMIT_CPU, rlim);
}
}
}
return 0;
}Barret Rhoden (3):
setpriority: only grab the tasklist_lock for PRIO_PGRP
prlimit: make do_prlimit() static
prlimit: do not grab the tasklist_lock
include/linux/posix-timers.h | 2 +- include/linux/resource.h | 2 - kernel/sys.c | 127 +++++++++++++++++---------------- kernel/time/posix-cpu-timers.c | 12 +++- 4 files changed, 76 insertions(+), 67 deletions(-)
I have dropped the first change in this series as an almost identical change was merged as commit 7f8ca0edfe07 ("kernel/sys.c: only take tasklist_lock for get/setpriority(PRIO_PGRP)"). -- EWB
The pull request you sent on Wed, 23 Mar 2022 15:14:47 -0500:
The pull request you sent on Wed, 23 Mar 2022 15:14:47 -0500:
Quoted 1 line
git://git.kernel.org/pub/scm/linux/kernel/git/ebiederm/user-namespace.gitprlimit-tasklist_lock-for-v5.18
has been merged into torvalds/linux.git:
https://git.kernel.org/torvalds/c/cd4699c5fd66b00211f4709b9957bfd7b0a02ddc
Thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/prtracker.html