Add support for 32-bit tasks on asymmetric AArch32 systems
Subsystems (8)
Will Deacon
maintainerv11 cover Add support for 32-bit tasks on asymmetric AArch32 systems
Hi everyone,
This is version eleven of the patches previously posted here:
v1: https://lore.kernel.org/r/[email protected]
v2: https://lore.kernel.org/r/[email protected]
v3: https://lore.kernel.org/r/[email protected]
v4: https://lore.kernel.org/r/[email protected]
v5: https://lore.kernel.org/r/[email protected]
v6: https://lore.kernel.org/r/[email protected]
v7: https://lore.kernel.org/r/[email protected]
v8: https://lore.kernel.org/r/[email protected]
v9: https://lore.kernel.org/r/[email protected]
v10: https://lore.kernel.org/r/[email protected]
The main changes since v10 are:
* Now based on v5.14-rc1
* Fixed a lockup found during testing where select_fallback_rq() could return an invalid CPU when trying to prioritise a destination on the same NUMA node
Greg kindly pointed out a potential issue with the new 'aarch32_el0' file in sysfs, where the contents will be truncated if we need to display more than a PAGE_SIZE of data. However, I have elected not to address this for now as (a) I do not think we'll ever have hardware in that configuration (yeah, I know...) and (b) It is useful for scripts if the file behaves the same as the other files in the same directory (e.g. 'online'). Should a solution to this problem emerge for the other files, I will be happy to adopt it here as well.
This series is now mostly scheduler stuff, but the last few patches are arm64 and so I would suggest a shared branch in -tip for merging.
Cheers,
Will
--->8
Will Deacon (16):
sched: Introduce task_cpu_possible_mask() to limit fallback rq
selection
cpuset: Don't use the cpu_possible_mask as a last resort for cgroup v1
cpuset: Honour task_cpu_possible_mask() in guarantee_online_cpus()
cpuset: Cleanup cpuset_cpus_allowed_fallback() use in
select_fallback_rq()
sched: Reject CPU affinity changes based on task_cpu_possible_mask()
sched: Introduce task_struct::user_cpus_ptr to track requested
affinity
sched: Split the guts of sched_setaffinity() into a helper function
sched: Allow task CPU affinity to be restricted on asymmetric systems
sched: Introduce dl_task_check_affinity() to check proposed affinity
arm64: Implement task_cpu_possible_mask()
arm64: exec: Adjust affinity for compat tasks with mismatched 32-bit
EL0
arm64: Prevent offlining first CPU with 32-bit EL0 on mismatched
system
arm64: Advertise CPUs capable of running 32-bit applications in sysfs
arm64: Hook up cmdline parameter to allow mismatched 32-bit EL0
arm64: Remove logic to kill 32-bit tasks on 64-bit-only cores
Documentation: arm64: describe asymmetric 32-bit support
.../ABI/testing/sysfs-devices-system-cpu | 9 + .../admin-guide/kernel-parameters.txt | 11 + Documentation/arm64/asymmetric-32bit.rst | 155 ++++++++ Documentation/arm64/index.rst | 1 + arch/arm64/include/asm/elf.h | 6 +- arch/arm64/include/asm/mmu_context.h | 13 + arch/arm64/kernel/cpufeature.c | 51 ++- arch/arm64/kernel/process.c | 47 ++- arch/arm64/kernel/signal.c | 26 -- include/linux/cpuset.h | 8 +- include/linux/mmu_context.h | 14 + include/linux/sched.h | 21 ++ init/init_task.c | 1 + kernel/cgroup/cpuset.c | 59 ++- kernel/fork.c | 2 + kernel/sched/core.c | 344 ++++++++++++++---- kernel/sched/sched.h | 1 + 17 files changed, 628 insertions(+), 141 deletions(-) create mode 100644 Documentation/arm64/asymmetric-32bit.rst
Signature · 1 line
2.32.0.402.g57bb445576-goog
Will Deacon
maintainerPATCH v11 1/16 sched: Introduce task_cpu_possible_mask() to limit fallback rq selection
Asymmetric systems may not offer the same level of userspace ISA support across all CPUs, meaning that some applications cannot be executed by some CPUs. As a concrete example, upcoming arm64 big.LITTLE designs do not feature support for 32-bit applications on both clusters.
On such a system, we must take care not to migrate a task to an unsupported CPU when forcefully moving tasks in select_fallback_rq() in response to a CPU hot-unplug operation.
Introduce a task_cpu_possible_mask() hook which, given a task argument, allows an architecture to return a cpumask of CPUs that are capable of executing that task. The default implementation returns the cpu_possible_mask, since sane machines do not suffer from per-cpu ISA limitations that affect scheduling. The new mask is used when selecting the fallback runqueue as a last resort before forcing a migration to the first active CPU.
--- include/linux/mmu_context.h | 14 ++++++++++++++ kernel/sched/core.c | 9 +++------ 2 files changed, 17 insertions(+), 6 deletions(-)
include/linux/mmu_context.h +14 -0
@@ -14,4 +14,18 @@static inline void leave_mm (int cpu) { }#endif +/* +* CPUs that are capable of running user task @p. Must contain at least one +* active CPU. It is assumed that the kernel can run on all CPUs, so calling +* this for a kernel thread is pointless. +* +* By default, we assume a sane, homogeneous system. +*/ +#ifndef task_cpu_possible_mask +# define task_cpu_possible_mask (p) cpu_possible_mask +# define task_cpu_possible (cpu, p) true +#else +# define task_cpu_possible (cpu, p) cpumask_test_cpu((cpu), task_cpu_possible_mask(p)) +#endif +#endif
kernel/sched/core.c +3 -6
@@ -2163,7 +2163,7 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)/* Non kernel threads are not allowed during either online or offline. */ if (!(p->flags &PF_KTHREAD )) -return cpu_active (cpu); +return cpu_active (cpu) &&task_cpu_possible (cpu, p);/* KTHREAD_IS_PER_CPU is always allowed. */ if (kthread_is_per_cpu (p))
@@ -3114,9 +3114,7 @@ static int select_fallback_rq(int cpu, struct task_struct *p)/* Look for allowed, online CPU in same node. */ for_each_cpu (dest_cpu, nodemask) { -if (!cpu_active (dest_cpu)) -continue ; -if (cpumask_test_cpu (dest_cpu, p->cpus_ptr)) +if (is_cpu_allowed (p, dest_cpu))return dest_cpu; } }
@@ -3146,10 +3144,9 @@ static int select_fallback_rq(int cpu, struct task_struct *p)* * More yuck to audit. */ -do_set_cpus_allowed (p, cpu_possible_mask); +do_set_cpus_allowed (p,task_cpu_possible_mask (p)); state = fail;break ; -case fail:BUG ();break ;
Signature · 1 line
2.32.0.402.g57bb445576-goog
Will Deacon
maintainerPATCH v11 2/16 cpuset: Don't use the cpu_possible_mask as a last resort for cgroup v1
In reply to Will DeaconIf the scheduler cannot find an allowed CPU for a task, cpuset_cpus_allowed_fallback() will widen the affinity to cpu_possible_mask if cgroup v1 is in use.
In preparation for allowing architectures to provide their own fallback mask, just return early if we're either using cgroup v1 or we're using cgroup v2 with a mask that contains invalid CPUs. This will allow select_fallback_rq() to figure out the mask by itself.
--- include/linux/cpuset.h | 1 + kernel/cgroup/cpuset.c | 8 ++++++-- 2 files changed, 7 insertions(+), 2 deletions(-)
include/linux/cpuset.h +1 -0
@@ -15,6 +15,7 @@#include <linux/cpumask.h> #include <linux/nodemask.h> #include <linux/mm.h> +#include <linux/mmu_context.h> #include <linux/jump_label.h> #ifdef CONFIG_CPUSETS
kernel/cgroup/cpuset.c +6 -2
@@ -3322,9 +3322,13 @@ void cpuset_cpus_allowed(struct task_struct *tsk, struct cpumask *pmask)void cpuset_cpus_allowed_fallback (struct task_struct *tsk) { +const struct cpumask *cs_mask; +const struct cpumask *possible_mask =task_cpu_possible_mask (tsk); +rcu_read_lock (); -do_set_cpus_allowed (tsk,is_in_v2_mode () ? -task_cs (tsk)->cpus_allowed : cpu_possible_mask); + cs_mask =task_cs (tsk)->cpus_allowed; +if (is_in_v2_mode () &&cpumask_subset (cs_mask, possible_mask)) +do_set_cpus_allowed (tsk, cs_mask);rcu_read_unlock ();/*
Signature · 1 line
2.32.0.402.g57bb445576-goog
Will Deacon
maintainerPATCH v11 3/16 cpuset: Honour task_cpu_possible_mask() in guarantee_online_cpus()
In reply to Will DeaconAsymmetric systems may not offer the same level of userspace ISA support across all CPUs, meaning that some applications cannot be executed by some CPUs. As a concrete example, upcoming arm64 big.LITTLE designs do not feature support for 32-bit applications on both clusters.
Modify guarantee_online_cpus() to take task_cpu_possible_mask() into account when trying to find a suitable set of online CPUs for a given task. This will avoid passing an invalid mask to set_cpus_allowed_ptr() during ->attach() and will subsequently allow the cpuset hierarchy to be taken into account when forcefully overriding the affinity mask for a task which requires migration to a compatible CPU.
--- include/linux/cpuset.h | 2 +- kernel/cgroup/cpuset.c | 43 +++++++++++++++++++++++++----------------- 2 files changed, 27 insertions(+), 18 deletions(-)
include/linux/cpuset.h +1 -1
@@ -185,7 +185,7 @@ static inline void cpuset_read_unlock(void) { }static inline void cpuset_cpus_allowed (struct task_struct *p,struct cpumask *mask) { -cpumask_copy (mask, cpu_possible_mask); +cpumask_copy (mask,task_cpu_possible_mask (p)); }static inline void cpuset_cpus_allowed_fallback (struct task_struct *p)
kernel/cgroup/cpuset.c +26 -17
79 lines not on this page · raw message
Signature · 1 line
2.32.0.402.g57bb445576-goog
Will Deacon
maintainerPATCH v11 4/16 cpuset: Cleanup cpuset_cpus_allowed_fallback() use in select_fallback_rq()
In reply to Will Deaconselect_fallback_rq() only needs to recheck for an allowed CPU if the affinity mask of the task has changed since the last check.
Return a 'bool' from cpuset_cpus_allowed_fallback() to indicate whether the affinity mask was updated, and use this to elide the allowed check when the mask has been left alone.
No functional change.
--- include/linux/cpuset.h | 5 +++-- kernel/cgroup/cpuset.c | 10 ++++++++-- kernel/sched/core.c | 3 +-- 3 files changed, 12 insertions(+), 6 deletions(-)
include/linux/cpuset.h +3 -2
@@ -59,7 +59,7 @@ extern void cpuset_wait_for_hotplug(void);extern void cpuset_read_lock (void );extern void cpuset_read_unlock (void );extern void cpuset_cpus_allowed (struct task_struct *p,struct cpumask *mask); -extern void cpuset_cpus_allowed_fallback (struct task_struct *p); +extern bool cpuset_cpus_allowed_fallback (struct task_struct *p);extern nodemask_t cpuset_mems_allowed (struct task_struct *p);#define cpuset_current_mems_allowed (current->mems_allowed)void cpuset_init_current_mems_allowed (void );
@@ -188,8 +188,9 @@ static inline void cpuset_cpus_allowed(struct task_struct *p,cpumask_copy (mask,task_cpu_possible_mask (p)); } -static inline void cpuset_cpus_allowed_fallback (struct task_struct *p) +static inline bool cpuset_cpus_allowed_fallback (struct task_struct *p) { +return false; }static inline nodemask_t cpuset_mems_allowed (struct task_struct *p)
kernel/cgroup/cpuset.c +8 -2
@@ -3327,17 +3327,22 @@ void cpuset_cpus_allowed(struct task_struct *tsk, struct cpumask *pmask)* which will not contain a sane cpumask during cases such as cpu hotplugging. * This is the absolute last resort for the scheduler and it is only used if * _every_ other avenue has been traveled. +* +* Returns true if the affinity of @tsk was changed, false otherwise. **/ -void cpuset_cpus_allowed_fallback (struct task_struct *tsk) +bool cpuset_cpus_allowed_fallback (struct task_struct *tsk) {const struct cpumask *cs_mask; +bool changed = false;const struct cpumask *possible_mask =task_cpu_possible_mask (tsk);rcu_read_lock (); cs_mask =task_cs (tsk)->cpus_allowed; -if (is_in_v2_mode () &&cpumask_subset (cs_mask, possible_mask)) +if (is_in_v2_mode () &&cpumask_subset (cs_mask, possible_mask)) {do_set_cpus_allowed (tsk, cs_mask); + changed = true; + }rcu_read_unlock ();/*
@@ -3357,6 +3362,7 @@ void cpuset_cpus_allowed_fallback(struct task_struct *tsk)* select_fallback_rq() will fix things ups and set cpu_possible_mask * if required. */ +return changed; }void __init cpuset_init_current_mems_allowed (void )
kernel/sched/core.c +1 -2
@@ -3131,8 +3131,7 @@ static int select_fallback_rq(int cpu, struct task_struct *p)/* No more Mr. Nice Guy. */ switch (state) {case cpuset: -if (IS_ENABLED (CONFIG_CPUSETS )) { -cpuset_cpus_allowed_fallback (p); +if (cpuset_cpus_allowed_fallback (p)) { state = possible;break ; }
Signature · 1 line
2.32.0.402.g57bb445576-goog
Will Deacon
maintainerPATCH v11 5/16 sched: Reject CPU affinity changes based on task_cpu_possible_mask()
In reply to Will DeaconReject explicit requests to change the affinity mask of a task via set_cpus_allowed_ptr() if the requested mask is not a subset of the mask returned by task_cpu_possible_mask(). This ensures that the 'cpus_mask' for a given task cannot contain CPUs which are incapable of executing it, except in cases where the affinity is forced.
--- kernel/sched/core.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
kernel/sched/core.c +8 -1
@@ -2700,15 +2700,17 @@ static int __set_cpus_allowed_ptr(struct task_struct *p, u32 flags) {const struct cpumask *cpu_valid_mask = cpu_active_mask; +const struct cpumask *cpu_allowed_mask =task_cpu_possible_mask (p);unsigned int dest_cpu;struct rq_flags rf;struct rq *rq;int ret =0 ; +bool kthread = p->flags &PF_KTHREAD ; rq =task_rq_lock (p, &rf);update_rq_clock (rq); -if (p->flags &PF_KTHREAD ||is_migration_disabled (p)) { +if (kthread ||is_migration_disabled (p)) {/* * Kernel threads are allowed on online && !active CPUs, * however, during cpu-hot-unplug, even these might get pushed
@@ -2722,6 +2724,11 @@ static int __set_cpus_allowed_ptr(struct task_struct *p, cpu_valid_mask = cpu_online_mask; } +if (!kthread && !cpumask_subset (new_mask, cpu_allowed_mask)) { + ret = -EINVAL ; + goto out; + } +/* * Must re-check here, to close a race against __kthread_bind(), * sched_setaffinity() is not guaranteed to observe the flag.
Signature · 1 line
2.32.0.402.g57bb445576-goog
Will Deacon
maintainerPATCH v11 6/16 sched: Introduce task_struct::user_cpus_ptr to track requested affinity
In reply to Will DeaconIn preparation for saving and restoring the user-requested CPU affinity mask of a task, add a new cpumask_t pointer to 'struct task_struct'.
If the pointer is non-NULL, then the mask is copied across fork() and freed on task exit.
--- include/linux/sched.h | 13 +++++++++++++ init/init_task.c | 1 + kernel/fork.c | 2 ++ kernel/sched/core.c | 20 ++++++++++++++++++++ 4 files changed, 36 insertions(+)
include/linux/sched.h +13 -0
@@ -748,6 +748,7 @@ struct task_struct {unsigned int policy;int nr_cpus_allowed;const cpumask_t *cpus_ptr; +cpumask_t *user_cpus_ptr;cpumask_t cpus_mask;void *migration_pending;#ifdef CONFIG_SMP
@@ -1705,6 +1706,8 @@ extern int task_can_attach(struct task_struct *p, const struct cpumask *cs_cpus_#ifdef CONFIG_SMP extern void do_set_cpus_allowed (struct task_struct *p,const struct cpumask *new_mask);extern int set_cpus_allowed_ptr (struct task_struct *p,const struct cpumask *new_mask); +extern int dup_user_cpus_ptr (struct task_struct *dst,struct task_struct *src,int node); +extern void release_user_cpus_ptr (struct task_struct *p);#else static inline void do_set_cpus_allowed (struct task_struct *p,const struct cpumask *new_mask) {
@@ -1715,6 +1718,16 @@ static inline int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumareturn -EINVAL ; return 0; } +static inline int dup_user_cpus_ptr (struct task_struct *dst,struct task_struct *src,int node) +{ +if (src->user_cpus_ptr) +return -EINVAL ; +return 0 ; +} +static inline void release_user_cpus_ptr (struct task_struct *p) +{ +WARN_ON (p->user_cpus_ptr); +}#endif extern int yield_to (struct task_struct *p,bool preempt);
init/init_task.c +1 -0
@@ -80,6 +80,7 @@ struct task_struct init_task .normal_prio = MAX_PRIO - 20, .policy = SCHED_NORMAL, .cpus_ptr = &init_task.cpus_mask, + .user_cpus_ptr = NULL, .cpus_mask = CPU_MASK_ALL, .nr_cpus_allowed= NR_CPUS, .mm = NULL,
kernel/fork.c +2 -0
@@ -446,6 +446,7 @@ void put_task_stack(struct task_struct *tsk)void free_task (struct task_struct *tsk) { +release_user_cpus_ptr (tsk);scs_release (tsk);#ifndef CONFIG_THREAD_INFO_IN_TASK
@@ -924,6 +925,7 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)#endif if (orig->cpus_ptr == &orig->cpus_mask) tsk->cpus_ptr = &tsk->cpus_mask; +dup_user_cpus_ptr (tsk, orig, node);/* * One for the user space visible state that goes away when reaped.
kernel/sched/core.c +20 -0
@@ -2470,6 +2470,26 @@ void do_set_cpus_allowed(struct task_struct *p, const struct cpumask *new_mask)__do_set_cpus_allowed (p, new_mask,0 ); } +int dup_user_cpus_ptr (struct task_struct *dst,struct task_struct *src, +int node) +{ +if (!src->user_cpus_ptr) +return 0 ; + + dst->user_cpus_ptr =kmalloc_node (cpumask_size (),GFP_KERNEL , node); +if (!dst->user_cpus_ptr) +return -ENOMEM ; + +cpumask_copy (dst->user_cpus_ptr, src->user_cpus_ptr); +return 0 ; +} + +void release_user_cpus_ptr (struct task_struct *p) +{ +kfree (p->user_cpus_ptr); + p->user_cpus_ptr =NULL ; +} +/* * This function is wildly self concurrent; here be dragons. *
Signature · 1 line
2.32.0.402.g57bb445576-goog
Will Deacon
maintainerPATCH v11 7/16 sched: Split the guts of sched_setaffinity() into a helper function
In reply to Will DeaconIn preparation for replaying user affinity requests using a saved mask, split sched_setaffinity() up so that the initial task lookup and security checks are only performed when the request is coming directly from userspace.
--- kernel/sched/core.c | 105 ++++++++++++++++++++++++-------------------- 1 file changed, 57 insertions(+), 48 deletions(-)
kernel/sched/core.c +57 -48
@@ -7578,53 +7578,22 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr,return retval; } -long sched_setaffinity (pid_t pid,const struct cpumask *in_mask) +static int +__sched_setaffinity (struct task_struct *p,const struct cpumask *mask) { -cpumask_var_t cpus_allowed, new_mask; -struct task_struct *p;int retval; +cpumask_var_t cpus_allowed, new_mask; -rcu_read_lock (); - - p =find_process_by_pid (pid); -if (!p) { -rcu_read_unlock (); -return -ESRCH ; - } - -/* Prevent p going away */ -get_task_struct (p); -rcu_read_unlock (); +if (!alloc_cpumask_var (&cpus_allowed,GFP_KERNEL )) +return -ENOMEM ; -if (p->flags &PF_NO_SETAFFINITY ) { - retval = -EINVAL ; - goto out_put_task; - } -if (!alloc_cpumask_var (&cpus_allowed,GFP_KERNEL )) { - retval = -ENOMEM ; - goto out_put_task; - }if (!alloc_cpumask_var (&new_mask,GFP_KERNEL )) { retval = -ENOMEM ; goto out_free_cpus_allowed; } - retval = -EPERM ; -if (!check_same_owner (p)) { -rcu_read_lock (); -if (!ns_capable (__task_cred (p)->user_ns,CAP_SYS_NICE )) { -rcu_read_unlock (); - goto out_free_new_mask; - } -rcu_read_unlock (); - } - - retval =security_task_setscheduler (p); -if (retval) - goto out_free_new_mask; -cpuset_cpus_allowed (p, cpus_allowed); -cpumask_and (new_mask, in_mask, cpus_allowed); +cpumask_and (new_mask, mask, cpus_allowed);/* * Since bandwidth control happens on root_domain basis,
@@ -7645,23 +7614,63 @@ long sched_setaffinity(pid_t pid, const struct cpumask *in_mask)#endif again: retval =__set_cpus_allowed_ptr (p, new_mask,SCA_CHECK ); +if (retval) + goto out_free_new_mask; -if (!retval) { -cpuset_cpus_allowed (p, cpus_allowed); -if (!cpumask_subset (new_mask, cpus_allowed)) { -/* -* We must have raced with a concurrent cpuset -* update. Just reset the cpus_allowed to the -* cpuset's cpus_allowed -*/ -cpumask_copy (new_mask, cpus_allowed); - goto again; - } +cpuset_cpus_allowed (p, cpus_allowed); +if (!cpumask_subset (new_mask, cpus_allowed)) { +/* +* We must have raced with a concurrent cpuset update. +* Just reset the cpumask to the cpuset's cpus_allowed. +*/ +cpumask_copy (new_mask, cpus_allowed); + goto again; } + out_free_new_mask:free_cpumask_var (new_mask); out_free_cpus_allowed:free_cpumask_var (cpus_allowed); +return retval; +} + +long sched_setaffinity (pid_t pid,const struct cpumask *in_mask) +{ +struct task_struct *p; +int retval; + +rcu_read_lock (); + + p =find_process_by_pid (pid); +if (!p) { +rcu_read_unlock (); +return -ESRCH ; + } + +/* Prevent p going away */ +get_task_struct (p); +rcu_read_unlock (); + +if (p->flags &PF_NO_SETAFFINITY ) { + retval = -EINVAL ; + goto out_put_task; + } + +if (!check_same_owner (p)) { +rcu_read_lock (); +if (!ns_capable (__task_cred (p)->user_ns,CAP_SYS_NICE )) { +rcu_read_unlock (); + retval = -EPERM ; + goto out_put_task; + } +rcu_read_unlock (); + } + + retval =security_task_setscheduler (p); +if (retval) + goto out_put_task; + + retval =__sched_setaffinity (p, in_mask); out_put_task:put_task_struct (p);return retval;
Signature · 1 line
2.32.0.402.g57bb445576-goog
Will Deacon
maintainerPATCH v11 8/16 sched: Allow task CPU affinity to be restricted on asymmetric systems
In reply to Will DeaconAsymmetric systems may not offer the same level of userspace ISA support across all CPUs, meaning that some applications cannot be executed by some CPUs. As a concrete example, upcoming arm64 big.LITTLE designs do not feature support for 32-bit applications on both clusters.
Although userspace can carefully manage the affinity masks for such tasks, one place where it is particularly problematic is execve() because the CPU on which the execve() is occurring may be incompatible with the new application image. In such a situation, it is desirable to restrict the affinity mask of the task and ensure that the new image is entered on a compatible CPU. From userspace's point of view, this looks the same as if the incompatible CPUs have been hotplugged off in the task's affinity mask. Similarly, if a subsequent execve() reverts to a compatible image, then the old affinity is restored if it is still valid.
In preparation for restricting the affinity mask for compat tasks on arm64 systems without uniform support for 32-bit applications, introduce {force,relax}_compatible_cpus_allowed_ptr(), which respectively restrict and restore the affinity mask for a task based on the compatible CPUs.
--- include/linux/sched.h | 2 + kernel/sched/core.c | 180 ++++++++++++++++++++++++++++++++++++++---- kernel/sched/sched.h | 1 + 3 files changed, 167 insertions(+), 16 deletions(-)
include/linux/sched.h +2 -0
@@ -1708,6 +1708,8 @@ extern void do_set_cpus_allowed(struct task_struct *p, const struct cpumask *newextern int set_cpus_allowed_ptr (struct task_struct *p,const struct cpumask *new_mask);extern int dup_user_cpus_ptr (struct task_struct *dst,struct task_struct *src,int node);extern void release_user_cpus_ptr (struct task_struct *p); +extern void force_compatible_cpus_allowed_ptr (struct task_struct *p); +extern void relax_compatible_cpus_allowed_ptr (struct task_struct *p);#else static inline void do_set_cpus_allowed (struct task_struct *p,const struct cpumask *new_mask) {
kernel/sched/core.c +164 -16
218 lines not on this page · raw message
kernel/sched/sched.h +1 -0
@@ -2234,6 +2234,7 @@ extern struct task_struct *pick_next_task_idle(struct rq *rq);#define SCA_CHECK 0x01#define SCA_MIGRATE_DISABLE 0x02#define SCA_MIGRATE_ENABLE 0x04 +#define SCA_USER 0x08#ifdef CONFIG_SMP
Signature · 1 line
2.32.0.402.g57bb445576-goog
Will Deacon
maintainerPATCH v11 9/16 sched: Introduce dl_task_check_affinity() to check proposed affinity
In reply to Will DeaconIn preparation for restricting the affinity of a task during execve() on arm64, introduce a new dl_task_check_affinity() helper function to give an indication as to whether the restricted mask is admissible for a deadline task.
--- include/linux/sched.h | 6 ++++++ kernel/sched/core.c | 46 +++++++++++++++++++++++++++---------------- 2 files changed, 35 insertions(+), 17 deletions(-)
include/linux/sched.h +6 -0
@@ -1708,6 +1708,7 @@ extern void do_set_cpus_allowed(struct task_struct *p, const struct cpumask *newextern int set_cpus_allowed_ptr (struct task_struct *p,const struct cpumask *new_mask);extern int dup_user_cpus_ptr (struct task_struct *dst,struct task_struct *src,int node);extern void release_user_cpus_ptr (struct task_struct *p); +extern int dl_task_check_affinity (struct task_struct *p,const struct cpumask *mask);extern void force_compatible_cpus_allowed_ptr (struct task_struct *p);extern void relax_compatible_cpus_allowed_ptr (struct task_struct *p);#else
@@ -1730,6 +1731,11 @@ static inline void release_user_cpus_ptr(struct task_struct *p) {WARN_ON (p->user_cpus_ptr); } + +static inline int dl_task_check_affinity (struct task_struct *p,const struct cpumask *mask) +{ +return 0 ; +}#endif extern int yield_to (struct task_struct *p,bool preempt);
kernel/sched/core.c +29 -17
58 lines not on this page · raw message
Signature · 1 line
2.32.0.402.g57bb445576-goog
Will Deacon
maintainerPATCH v11 10/16 arm64: Implement task_cpu_possible_mask()
In reply to Will DeaconProvide an implementation of task_cpu_possible_mask() so that we can prevent 64-bit-only cores being added to the 'cpus_mask' for compat tasks on systems with mismatched 32-bit support at EL0,
--- arch/arm64/include/asm/mmu_context.h | 13 +++++++++++++ 1 file changed, 13 insertions(+)
arch/arm64/include/asm/mmu_context.h +13 -0
@@ -231,6 +231,19 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next,update_saved_ttbr0 (tsk, next); } +static inline const struct cpumask * +task_cpu_possible_mask (struct task_struct *p) +{ +if (!static_branch_unlikely (&arm64_mismatched_32bit_el0)) +return cpu_possible_mask; + +if (!is_compat_thread (task_thread_info (p))) +return cpu_possible_mask; + +return system_32bit_el0_cpumask (); +} +#define task_cpu_possible_mask task_cpu_possible_mask +void verify_cpu_asid_bits (void );void post_ttbr_update_workaround (void );
Signature · 1 line
2.32.0.402.g57bb445576-goog
Will Deacon
maintainerPATCH v11 11/16 arm64: exec: Adjust affinity for compat tasks with mismatched 32-bit EL0
In reply to Will DeaconWhen exec'ing a 32-bit task on a system with mismatched support for 32-bit EL0, try to ensure that it starts life on a CPU that can actually run it.
Similarly, when exec'ing a 64-bit task on such a system, try to restore the old affinity mask if it was previously restricted.
--- arch/arm64/include/asm/elf.h | 6 ++---- arch/arm64/kernel/process.c | 39 +++++++++++++++++++++++++++++++++++- 2 files changed, 40 insertions(+), 5 deletions(-)
arch/arm64/include/asm/elf.h +2 -4
@@ -213,10 +213,8 @@ typedef compat_elf_greg_t compat_elf_gregset_t[COMPAT_ELF_NGREG];/* AArch32 EABI. */ #define EF_ARM_EABI_MASK 0xff000000 -#define compat_elf_check_arch (x) (system_supports_32bit_el0() && \ - ((x)->e_machine == EM_ARM) && \ - ((x)->e_flags & EF_ARM_EABI_MASK)) - +int compat_elf_check_arch (const struct elf32_hdr *); +#define compat_elf_check_arch compat_elf_check_arch#define compat_start_thread compat_start_thread/* * Unlike the native SET_PERSONALITY macro, the compat version maintains
arch/arm64/kernel/process.c +38 -1
@@ -21,6 +21,7 @@#include <linux/mman.h> #include <linux/mm.h> #include <linux/nospec.h> +#include <linux/sched.h> #include <linux/stddef.h> #include <linux/sysctl.h> #include <linux/unistd.h>
@@ -579,6 +580,28 @@ unsigned long arch_align_stack(unsigned long sp)return sp & ~0xf ; } +#ifdef CONFIG_COMPAT +int compat_elf_check_arch (const struct elf32_hdr *hdr) +{ +if (!system_supports_32bit_el0 ()) +return false; + +if ((hdr)->e_machine !=EM_ARM ) +return false; + +if (!((hdr)->e_flags &EF_ARM_EABI_MASK )) +return false; + +/* +* Prevent execve() of a 32-bit program from a deadline task +* if the restricted affinity mask would be inadmissible on an +* asymmetric system. +*/ +return !static_branch_unlikely (&arm64_mismatched_32bit_el0) || + !dl_task_check_affinity (current,system_32bit_el0_cpumask ()); +} +#endif +/* * Called from setup_new_exec() after (COMPAT_)SET_PERSONALITY. */
@@ -588,8 +611,22 @@ void arch_setup_new_exec(void)if (is_compat_task ()) { mmflags =MMCF_AARCH32 ; -if (static_branch_unlikely (&arm64_mismatched_32bit_el0)) + +/* +* Restrict the CPU affinity mask for a 32-bit task so that +* it contains only 32-bit-capable CPUs. +* +* From the perspective of the task, this looks similar to +* what would happen if the 64-bit-only CPUs were hot-unplugged +* at the point of execve(), although we try a bit harder to +* honour the cpuset hierarchy. +*/ +if (static_branch_unlikely (&arm64_mismatched_32bit_el0)) { +force_compatible_cpus_allowed_ptr (current);set_tsk_thread_flag (current,TIF_NOTIFY_RESUME ); + } + }else if (static_branch_unlikely (&arm64_mismatched_32bit_el0)) { +relax_compatible_cpus_allowed_ptr (current); } current->mm->context.flags = mmflags;
Signature · 1 line
2.32.0.402.g57bb445576-goog
Will Deacon
maintainerPATCH v11 12/16 arm64: Prevent offlining first CPU with 32-bit EL0 on mismatched system
In reply to Will DeaconIf we want to support 32-bit applications, then when we identify a CPU with mismatched 32-bit EL0 support we must ensure that we will always have an active 32-bit CPU available to us from then on. This is important for the scheduler, because is_cpu_allowed() will be constrained to 32-bit CPUs for compat tasks and forced migration due to a hotplug event will hang if no 32-bit CPUs are available.
On detecting a mismatch, prevent offlining of either the mismatching CPU if it is 32-bit capable, or find the first active 32-bit capable CPU otherwise.
--- arch/arm64/kernel/cpufeature.c | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-)
arch/arm64/kernel/cpufeature.c +24 -1
@@ -2900,15 +2900,38 @@ void __init setup_cpu_features(void)static int enable_mismatched_32bit_el0 (unsigned int cpu) { +/* +* The first 32-bit-capable CPU we detected and so can no longer +* be offlined by userspace. -1 indicates we haven't yet onlined +* a 32-bit-capable CPU. +*/ +static int lucky_winner =-1 ; +struct cpuinfo_arm64 *info = &per_cpu (cpu_data, cpu);bool cpu_32bit =id_aa64pfr0_32bit_el0 (info->reg_id_aa64pfr0);if (cpu_32bit) {cpumask_set_cpu (cpu, cpu_32bit_el0_mask);static_branch_enable_cpuslocked (&arm64_mismatched_32bit_el0); -setup_elf_hwcaps (compat_elf_hwcaps); } +if (cpumask_test_cpu (0 , cpu_32bit_el0_mask) == cpu_32bit) +return 0 ; + +if (lucky_winner >=0 ) +return 0 ; + +/* +* We've detected a mismatch. We need to keep one of our CPUs with +* 32-bit EL0 online so that is_cpu_allowed() doesn't end up rejecting +* every CPU in the system for a 32-bit task. +*/ + lucky_winner = cpu_32bit ? cpu :cpumask_any_and (cpu_32bit_el0_mask, + cpu_active_mask); +get_cpu_device (lucky_winner)->offline_disabled = true; +setup_elf_hwcaps (compat_elf_hwcaps); +pr_info ("Asymmetric 32-bit EL0 support detected on CPU %u; CPU hot-unplug disabled on CPU %u\n" , + cpu, lucky_winner);return 0 ; }
Signature · 1 line
2.32.0.402.g57bb445576-goog
Will Deacon
maintainerPATCH v11 13/16 arm64: Advertise CPUs capable of running 32-bit applications in sysfs
In reply to Will DeaconSince 32-bit applications will be killed if they are caught trying to execute on a 64-bit-only CPU in a mismatched system, advertise the set of 32-bit capable CPUs to userspace in sysfs.
--- .../ABI/testing/sysfs-devices-system-cpu | 9 +++++++++ arch/arm64/kernel/cpufeature.c | 19 +++++++++++++++++++ 2 files changed, 28 insertions(+)
Documentation/ABI/testing/sysfs-devices-system-cpu +9 -0
@@ -494,6 +494,15 @@ Description: AArch64 CPU registers 'identification' directory exposes the CPU ID registers for identifying model and revision of the CPU. +What: /sys/devices/system/cpu/aarch32_el0 +Date: May 2021 +Contact: Linux ARM Kernel Mailing list <[email protected]> +Description: Identifies the subset of CPUs in the system that can execute + AArch32 (32-bit ARM) applications. If present, the same format as + /sys/devices/system/cpu/{offline,online,possible,present} is used. + If absent, then all or none of the CPUs can execute AArch32 + applications and execve() will behave accordingly. + What: /sys/devices/system/cpu/cpu#/cpu_capacity Date: December 2016 Contact: Linux kernel mailing list <[email protected]>
arch/arm64/kernel/cpufeature.c +19 -0
@@ -67,6 +67,7 @@#include <linux/crash_dump.h> #include <linux/sort.h> #include <linux/stop_machine.h> +#include <linux/sysfs.h> #include <linux/types.h> #include <linux/minmax.h> #include <linux/mm.h>
@@ -1320,6 +1321,24 @@ const struct cpumask *system_32bit_el0_cpumask(void)return cpu_possible_mask; } +static ssize_t aarch32_el0_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +const struct cpumask *mask =system_32bit_el0_cpumask (); + +return sysfs_emit (buf,"%*pbl\n" ,cpumask_pr_args (mask)); +} +static const DEVICE_ATTR_RO (aarch32_el0); + +static int __init aarch32_el0_sysfs_init (void ) +{ +if (!allow_mismatched_32bit_el0) +return 0 ; + +return device_create_file (cpu_subsys.dev_root, &dev_attr_aarch32_el0); +} +device_initcall (aarch32_el0_sysfs_init); +static bool has_32bit_el0 (const struct arm64_cpu_capabilities *entry,int scope) {if (!has_cpuid_feature (entry, scope))
Signature · 1 line
2.32.0.402.g57bb445576-goog
Will Deacon
maintainerPATCH v11 14/16 arm64: Hook up cmdline parameter to allow mismatched 32-bit EL0
In reply to Will DeaconAllow systems with mismatched 32-bit support at EL0 to run 32-bit applications based on a new kernel parameter.
--- Documentation/admin-guide/kernel-parameters.txt | 8 ++++++++ arch/arm64/kernel/cpufeature.c | 7 +++++++ 2 files changed, 15 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +8 -0
@@ -287,6 +287,14 @@ do not want to use tracing_snapshot_alloc() as it needs to be done where GFP_KERNEL allocations are allowed. + allow_mismatched_32bit_el0 [ARM64] + Allow execve() of 32-bit applications and setting of the + PER_LINUX32 personality on systems where only a strict + subset of the CPUs support 32-bit EL0. When this + parameter is present, the set of CPUs supporting 32-bit + EL0 is indicated by /sys/devices/system/cpu/aarch32_el0 + and hot-unplug operations may be restricted. + amd_iommu= [HW,X86-64] Pass parameters to the AMD IOMMU driver in the system. Possible values are:
arch/arm64/kernel/cpufeature.c +7 -0
@@ -1321,6 +1321,13 @@ const struct cpumask *system_32bit_el0_cpumask(void)return cpu_possible_mask; } +static int __init parse_32bit_el0_param (char *str) +{ + allow_mismatched_32bit_el0 = true; +return 0 ; +} +early_param ("allow_mismatched_32bit_el0" , parse_32bit_el0_param); +static ssize_t aarch32_el0_show (struct device *dev,struct device_attribute *attr,char *buf) {
Signature · 1 line
2.32.0.402.g57bb445576-goog
Will Deacon
maintainerPATCH v11 15/16 arm64: Remove logic to kill 32-bit tasks on 64-bit-only cores
In reply to Will DeaconThe scheduler now knows enough about these braindead systems to place 32-bit tasks accordingly, so throw out the safety checks and allow the ret-to-user path to avoid do_notify_resume() if there is nothing to do.
--- arch/arm64/kernel/process.c | 14 +------------- arch/arm64/kernel/signal.c | 26 -------------------------- 2 files changed, 1 insertion(+), 39 deletions(-)
arch/arm64/kernel/process.c +1 -13
@@ -469,15 +469,6 @@ static void erratum_1418040_thread_switch(struct task_struct *prev,write_sysreg (val, cntkctl_el1); } -static void compat_thread_switch (struct task_struct *next) -{ -if (!is_compat_thread (task_thread_info (next))) -return ; - -if (static_branch_unlikely (&arm64_mismatched_32bit_el0)) -set_tsk_thread_flag (next,TIF_NOTIFY_RESUME ); -} -static void update_sctlr_el1 (u64 sctlr) {/*
@@ -519,7 +510,6 @@ __notrace_funcgraph struct task_struct *__switch_to(struct task_struct *prev,ssbs_thread_switch (next);erratum_1418040_thread_switch (prev, next);ptrauth_thread_switch_user (next); -compat_thread_switch (next);/* * Complete any pending TLB or cache maintenance on this CPU in case
@@ -621,10 +611,8 @@ void arch_setup_new_exec(void)* at the point of execve(), although we try a bit harder to * honour the cpuset hierarchy. */ -if (static_branch_unlikely (&arm64_mismatched_32bit_el0)) { +if (static_branch_unlikely (&arm64_mismatched_32bit_el0))force_compatible_cpus_allowed_ptr (current); -set_tsk_thread_flag (current,TIF_NOTIFY_RESUME ); - } }else if (static_branch_unlikely (&arm64_mismatched_32bit_el0)) {relax_compatible_cpus_allowed_ptr (current); }
arch/arm64/kernel/signal.c +0 -26
@@ -911,19 +911,6 @@ static void do_signal(struct pt_regs *regs)restore_saved_sigmask (); } -static bool cpu_affinity_invalid (struct pt_regs *regs) -{ -if (!compat_user_mode (regs)) -return false; - -/* -* We're preemptible, but a reschedule will cause us to check the -* affinity again. -*/ -return !cpumask_test_cpu (raw_smp_processor_id (), -system_32bit_el0_cpumask ()); -} - asmlinkage void do_notify_resume(struct pt_regs *regs,unsigned long thread_flags) {
@@ -951,19 +938,6 @@ asmlinkage void do_notify_resume(struct pt_regs *regs,if (thread_flags & _TIF_NOTIFY_RESUME) {tracehook_notify_resume (regs);rseq_handle_notify_resume (NULL , regs); - -/* -* If we reschedule after checking the affinity -* then we must ensure that TIF_NOTIFY_RESUME -* is set so that we check the affinity again. -* Since tracehook_notify_resume() clears the -* flag, ensure that the compiler doesn't move -* it after the affinity check. -*/ -barrier (); - -if (cpu_affinity_invalid (regs)) -force_sig (SIGKILL ); }if (thread_flags & _TIF_FOREIGN_FPSTATE)
Signature · 1 line
2.32.0.402.g57bb445576-goog
Will Deacon
maintainerPATCH v11 16/16 Documentation: arm64: describe asymmetric 32-bit support
In reply to Will DeaconDocument support for running 32-bit tasks on asymmetric 32-bit systems and its impact on the user ABI when enabled.
--- .../admin-guide/kernel-parameters.txt | 3 + Documentation/arm64/asymmetric-32bit.rst | 155 ++++++++++++++++++ Documentation/arm64/index.rst | 1 + 3 files changed, 159 insertions(+) create mode 100644 Documentation/arm64/asymmetric-32bit.rst
Documentation/admin-guide/kernel-parameters.txt +3 -0
@@ -295,6 +295,9 @@ EL0 is indicated by /sys/devices/system/cpu/aarch32_el0 and hot-unplug operations may be restricted. + See Documentation/arm64/asymmetric-32bit.rst for more + information. + amd_iommu= [HW,X86-64] Pass parameters to the AMD IOMMU driver in the system. Possible values are:
Documentation/arm64/asymmetric-32bit.rst new file +155 -0
155 lines not on this page · raw message
Documentation/arm64/index.rst +1 -0
@@ -10,6 +10,7 @@ ARM64 Architecture acpi_object_usage amu arm-acpi + asymmetric-32bit booting cpu-feature-registers elf_hwcaps
Signature · 1 line
2.32.0.402.g57bb445576-goog
Peter Zijlstra
maintainerRe: [PATCH v11 08/16] sched: Allow task CPU affinity to be restricted on asymmetric systems
In reply to Will DeaconQuoted 24 lines · Will Deacon
(quoted diff) +12
-2@@ -2783,20 +2778,173 @@ static int __set_cpus_allowed_ptr(struct task_struct *p, __do_set_cpus_allowed(p, new_mask, flags); - return affine_move_task(rq, p, &rf, dest_cpu, flags); + if (flags & SCA_USER) + release_user_cpus_ptr(p); + + return affine_move_task(rq, p, rf, dest_cpu, flags); out: - task_rq_unlock(rq, p, &rf); + task_rq_unlock(rq, p, rf); return ret; } +void relax_compatible_cpus_allowed_ptr(struct task_struct *p) +{ + unsigned long flags; + struct cpumask *mask = p->user_cpus_ptr; + + /* + * Try to restore the old affinity mask. If this fails, then
Review static int __set_cpus_allowed_ptr(struct task_struct *p, · line 2808
Quoted 10 lines
(quoted diff) +10
-0+ * we free the mask explicitly to avoid it being inherited across + * a subsequent fork(). + */ + if (!mask || !__sched_setaffinity(p, mask)) + return; + + raw_spin_lock_irqsave(&p->pi_lock, flags); + release_user_cpus_ptr(p); + raw_spin_unlock_irqrestore(&p->pi_lock, flags); +}
Both these are a problem on RT.
The easiest recourse is simply never freeing the CPU mask (except on exit). The alternative is something like the below I suppose..
I'm leaning towards the former option, wdyt?
kernel/sched/core.c +10 -3
@@ -2733,6 +2733,7 @@ static int __set_cpus_allowed_ptr_lockedconst struct cpumask *cpu_allowed_mask =task_cpu_possible_mask (p);const struct cpumask *cpu_valid_mask = cpu_active_mask;bool kthread = p->flags &PF_KTHREAD ; +struct cpumask *user_mask =NULL ;unsigned int dest_cpu;int ret =0 ;
@@ -2792,9 +2793,13 @@ static int __set_cpus_allowed_ptr_locked__do_set_cpus_allowed (p, new_mask, flags);if (flags &SCA_USER ) -release_user_cpus_ptr (p); +swap (user_mask, p->user_cpus_ptr); + + ret =affine_move_task (rq, p, rf, dest_cpu, flags); + +kfree (user_mask); -return affine_move_task (rq, p, rf, dest_cpu, flags); +return ret; out:task_rq_unlock (rq, p, rf);
@@ -2954,8 +2959,10 @@ void relax_compatible_cpus_allowed_ptr(sreturn ;raw_spin_lock_irqsave (&p->pi_lock, flags); -release_user_cpus_ptr (p); + p->user_cpus_ptr =NULL ;raw_spin_unlock_irqrestore (&p->pi_lock, flags); + +kfree (mask); }void set_task_cpu (struct task_struct *p,unsigned int new_cpu)
Peter Zijlstra
maintainerRe: [PATCH v11 07/16] sched: Split the guts of sched_setaffinity() into a helper function
In reply to Will DeaconQuoted 7 lines · Will Deacon
In preparation for replaying user affinity requests using a saved mask, split
sched_setaffinity()up so that the initial task lookup and security checks are only performed when the request is coming directly from userspace.Reviewed-by Valentin Schneider <[email protected]>Signed-off-by Will Deacon <[email protected]>
Should not sched_setaffinity() update user_cpus_ptr when it isn't NULL, such that the upcoming relax_compatible_cpus_allowed_ptr() preserve the full user mask?
Peter Zijlstra
maintainerRe: [PATCH v11 08/16] sched: Allow task CPU affinity to be restricted on asymmetric systems
In reply to Will Deaconkernel/sched/core.c static int __set_cpus_allowed_ptr(struct task_struct *p, · line 2933
Quoted 10 lines · Will Deacon
kernel/sched/core.c +8
-0+struct rq_flags rf; +struct rq *rq; +int err; +struct cpumask *user_mask =NULL ; +cpumask_var_t new_mask; +const struct cpumask *override_mask =task_cpu_possible_mask (p); +unsigned long flags; +struct cpumask *mask = p->user_cpus_ptr;
I've fixed all that up to be proper reverse x-mas trees; similar for other patches.
Will Deacon
maintainerRe: [PATCH v11 08/16] sched: Allow task CPU affinity to be restricted on asymmetric systems
In reply to Peter ZijlstraHi Peter,
Quoted 29 lines · Peter Zijlstra
Quoted 28 lines · Will Deacon
(quoted diff) +16
-2@@ -2783,20 +2778,173 @@ static int __set_cpus_allowed_ptr(struct task_struct *p, __do_set_cpus_allowed(p, new_mask, flags); - return affine_move_task(rq, p, &rf, dest_cpu, flags); + if (flags & SCA_USER) + release_user_cpus_ptr(p); + + return affine_move_task(rq, p, rf, dest_cpu, flags); out: - task_rq_unlock(rq, p, &rf); + task_rq_unlock(rq, p, rf); return ret; } +void relax_compatible_cpus_allowed_ptr(struct task_struct *p) +{ + unsigned long flags; + struct cpumask *mask = p->user_cpus_ptr; + + /* + * Try to restore the old affinity mask. If this fails, then + * we free the mask explicitly to avoid it being inherited across + * a subsequent fork(). + */ + if (!mask || !__sched_setaffinity(p, mask))
Quoted 8 lines
Review static int __set_cpus_allowed_ptr(struct task_struct *p, · line 2808
Quoted 6 lines
(quoted diff) +6
-0+ return; + + raw_spin_lock_irqsave(&p->pi_lock, flags); + release_user_cpus_ptr(p); + raw_spin_unlock_irqrestore(&p->pi_lock, flags); +}Both these are a problem on RT.
Ah, sorry. I didn't realise you couldn't _free_ with a raw lock held in RT. Is there somewhere I can read up on that?
Quoted 4 lines
The easiest recourse is simply never freeing the CPU mask (except on exit). The alternative is something like the below I suppose..
I'm leaning towards the former option, wdyt?
Defering the freeing until exit feels like a little fiddly, as we still want to clear ->user_cpus_ptr on affinity changes when SCA_USER is set so we'd have to keep track of the mask somewhere and reuse it instead of allocating a new one if we need it later on. Do-able, but feels a bit nasty, particular across fork().
As for your other suggestion:
Quoted 27 lines
kernel/sched/core.c +7
-2@@ -2733,6 +2733,7 @@ static int __set_cpus_allowed_ptr_lockedconst struct cpumask *cpu_allowed_mask =task_cpu_possible_mask (p);const struct cpumask *cpu_valid_mask = cpu_active_mask;bool kthread = p->flags &PF_KTHREAD ; +struct cpumask *user_mask =NULL ;unsigned int dest_cpu;int ret =0 ;@@ -2792,9 +2793,13 @@ static int __set_cpus_allowed_ptr_locked__do_set_cpus_allowed (p, new_mask, flags);if (flags &SCA_USER ) -release_user_cpus_ptr (p); +swap (user_mask, p->user_cpus_ptr); + + ret =affine_move_task (rq, p, rf, dest_cpu, flags); + +kfree (user_mask); -return affine_move_task (rq, p, rf, dest_cpu, flags); +return ret; out:task_rq_unlock (rq, p, rf);
kernel/sched/core.c void relax_compatible_cpus_allowed_ptr(s · line 2965
Quoted 8 lines
kernel/sched/core.c +3
-1@@ -2954,8 +2959,10 @@ void relax_compatible_cpus_allowed_ptr(sreturn ;raw_spin_lock_irqsave (&p->pi_lock, flags); -release_user_cpus_ptr (p); + p->user_cpus_ptr =NULL ;raw_spin_unlock_irqrestore (&p->pi_lock, flags); + +kfree (mask);
I think the idea looks good, but perhaps we could wrap things up a bit:
/* Comment about why this is useful with RT */ static cpumask_t *clear_user_cpus_ptr(struct task_struct *p)
{
struct cpumask *user_mask = NULL;
swap(user_mask, p->user_cpus_ptr);
return user_mask;
}void release_user_cpus_ptr(struct task_struct *p)
{
kfree(clear_user_cpus_ptr(p));
}Then just use clear_user_cpus_ptr() in sched/core.c where we know what we're doing (well, at least one of us does!).
Will
Will Deacon
maintainerRe: [PATCH v11 08/16] sched: Allow task CPU affinity to be restricted on asymmetric systems
In reply to Peter ZijlstraQuoted 6 lines · Peter Zijlstra
Quoted 4 lines · Will Deacon
kernel/sched/core.c +4
-0+struct rq_flags rf; +struct rq *rq; +int err; +struct cpumask *user_mask =NULL ;
Quoted 8 lines
kernel/sched/core.c static int __set_cpus_allowed_ptr(struct task_struct *p, · line 2933
Quoted 5 lines
kernel/sched/core.c +4
-0+cpumask_var_t new_mask; +const struct cpumask *override_mask =task_cpu_possible_mask (p); +unsigned long flags; +struct cpumask *mask = p->user_cpus_ptr;I've fixed all that up to be proper reverse x-mas trees; similar for other patches.
Thanks.
Will
Will Deacon
maintainerRe: [PATCH v11 07/16] sched: Split the guts of sched_setaffinity() into a helper function
In reply to Peter ZijlstraQuoted 6 lines · Peter Zijlstra
Quoted 5 lines · Will Deacon
In preparation for replaying user affinity requests using a saved mask, split
sched_setaffinity()up so that the initial task lookup and security checks are only performed when the request is coming directly from userspace.
Quoted 6 lines
Quoted 2 lines
Reviewed-by Valentin Schneider <[email protected]>Signed-off-by Will Deacon <[email protected]>Should not
sched_setaffinity()updateuser_cpus_ptrwhen it isn't NULL, such that the upcomingrelax_compatible_cpus_allowed_ptr()preserve the full user mask?
The idea is that force_compatible_cpus_allowed_ptr() and relax_compatible_cpus_allowed_ptr() are used as a pair, with the former setting ->user_cpus_ptr and the latter restoring it. An intervening call to sched_setaffinity() must _clear_ the saved mask, as we discussed before at:
https://lore.kernel.org/r/[email protected]
Will
Peter Zijlstra
maintainerRe: [PATCH v11 08/16] sched: Allow task CPU affinity to be restricted on asymmetric systems
In reply to Will DeaconQuoted 49 lines · Will Deacon
As for your other suggestion:
Quoted 27 lines
kernel/sched/core.c +7
-2@@ -2733,6 +2733,7 @@ static int __set_cpus_allowed_ptr_lockedconst struct cpumask *cpu_allowed_mask =task_cpu_possible_mask (p);const struct cpumask *cpu_valid_mask = cpu_active_mask;bool kthread = p->flags &PF_KTHREAD ; +struct cpumask *user_mask =NULL ;unsigned int dest_cpu;int ret =0 ;@@ -2792,9 +2793,13 @@ static int __set_cpus_allowed_ptr_locked__do_set_cpus_allowed (p, new_mask, flags);if (flags &SCA_USER ) -release_user_cpus_ptr (p); +swap (user_mask, p->user_cpus_ptr); + + ret =affine_move_task (rq, p, rf, dest_cpu, flags); + +kfree (user_mask); -return affine_move_task (rq, p, rf, dest_cpu, flags); +return ret; out:task_rq_unlock (rq, p, rf);kernel/sched/core.c void relax_compatible_cpus_allowed_ptr(s · line 2965
Quoted 8 lines
kernel/sched/core.c +3
-1@@ -2954,8 +2959,10 @@ void relax_compatible_cpus_allowed_ptr(sreturn ;raw_spin_lock_irqsave (&p->pi_lock, flags); -release_user_cpus_ptr (p); + p->user_cpus_ptr =NULL ;raw_spin_unlock_irqrestore (&p->pi_lock, flags); + +kfree (mask);I think the idea looks good, but perhaps we could wrap things up a bit:
/* Comment about why this is useful with RT */ static
cpumask_t*clear_user_cpus_ptr(structtask_struct*p){ struct cpumask *user_mask = NULL; swap(user_mask, p->user_cpus_ptr); return user_mask; }
Quoted 7 lines
void
release_user_cpus_ptr(struct task_struct *p){ kfree(clear_user_cpus_ptr(p)); }Then just use
clear_user_cpus_ptr()insched/core.cwhere we know what we're doing (well, at least one of us does!).
OK, I'll go make it like that.
Peter Zijlstra
maintainerRe: [PATCH v11 07/16] sched: Split the guts of sched_setaffinity() into a helper function
In reply to Will DeaconQuoted 14 lines · Will Deacon
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Will Deacon
In preparation for replaying user affinity requests using a saved mask, split
sched_setaffinity()up so that the initial task lookup and security checks are only performed when the request is coming directly from userspace.Quoted 6 lines
Quoted 2 lines
Reviewed-by Valentin Schneider <[email protected]>Signed-off-by Will Deacon <[email protected]>Should not
sched_setaffinity()updateuser_cpus_ptrwhen it isn't NULL, such that the upcomingrelax_compatible_cpus_allowed_ptr()preserve the full user mask?
Quoted 7 lines
The idea is that
force_compatible_cpus_allowed_ptr()andrelax_compatible_cpus_allowed_ptr()are used as a pair, with the former setting ->user_cpus_ptr and the latter restoring it. An intervening call tosched_setaffinity()must _clear_ the saved mask, as we discussed before at:
Clearly that deserves a comment somewhere, because I keep trying to make it more consistent than it can be :/ I'll see if I can find a spot.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Peter Zijlstra
maintainerRe: [PATCH v11 08/16] sched: Allow task CPU affinity to be restricted on asymmetric systems
In reply to Will DeaconQuoted 2 lines · Will Deacon
Ah, sorry. I didn't realise you couldn't _free_ with a raw lock held in RT. Is there somewhere I can read up on that?
It's because the allocators use spinlock_t, which cannot nest inside raw_spinlock_t.
Will Deacon
maintainerRe: [PATCH v11 07/16] sched: Split the guts of sched_setaffinity() into a helper function
In reply to Peter ZijlstraQuoted 21 lines · Peter Zijlstra
Quoted 20 lines · Will Deacon
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Will Deacon
In preparation for replaying user affinity requests using a saved mask, split
sched_setaffinity()up so that the initial task lookup and security checks are only performed when the request is coming directly from userspace.Quoted 6 lines
Quoted 2 lines
Reviewed-by Valentin Schneider <[email protected]>Signed-off-by Will Deacon <[email protected]>Should not
sched_setaffinity()updateuser_cpus_ptrwhen it isn't NULL, such that the upcomingrelax_compatible_cpus_allowed_ptr()preserve the full user mask?The idea is that
force_compatible_cpus_allowed_ptr()andrelax_compatible_cpus_allowed_ptr()are used as a pair, with the former setting ->user_cpus_ptr and the latter restoring it. An intervening call tosched_setaffinity()must _clear_ the saved mask, as we discussed before at:
Quoted 4 lines
Quoted 1 line
Clearly that deserves a comment somewhere, because I keep trying to make it more consistent than it can be :/ I'll see if I can find a spot.
Agreed. The relax/force functions are already commented, so maybe alongside SCA_USER?
Will
Peter Zijlstra
maintainerRe: [PATCH v11 08/16] sched: Allow task CPU affinity to be restricted on asymmetric systems
In reply to Peter ZijlstraQuoted 18 lines · Peter Zijlstra
Quoted 16 lines · Will Deacon
I think the idea looks good, but perhaps we could wrap things up a bit:
/* Comment about why this is useful with RT */ static
cpumask_t*clear_user_cpus_ptr(structtask_struct*p){ struct cpumask *user_mask = NULL; swap(user_mask, p->user_cpus_ptr); return user_mask; }void
release_user_cpus_ptr(struct task_struct *p){ kfree(clear_user_cpus_ptr(p)); }
Quoted 4 lines
Quoted 2 lines
Then just use
clear_user_cpus_ptr()insched/core.cwhere we know what we're doing (well, at least one of us does!).OK, I'll go make it like that.
Something like so then?
kernel/sched/core.c +22 -7
@@ -2497,10 +2497,18 @@ int dup_user_cpus_ptr(struct task_structreturn 0 ; } +static inline struct cpumask *clear_user_cpus_ptr (struct task_struct *p) +{ +struct cpumask *user_mask =NULL ; + +swap (p->user_cpus_ptr, user_mask); + +return user_mask; +} +void release_user_cpus_ptr (struct task_struct *p) { -kfree (p->user_cpus_ptr); - p->user_cpus_ptr =NULL ; +kfree (clear_user_cpus_ptr (p)); }/*
@@ -2733,6 +2741,7 @@ static int __set_cpus_allowed_ptr_lockedconst struct cpumask *cpu_allowed_mask =task_cpu_possible_mask (p);const struct cpumask *cpu_valid_mask = cpu_active_mask;bool kthread = p->flags &PF_KTHREAD ; +struct cpumask *user_mask =NULL ;unsigned int dest_cpu;int ret =0 ;
@@ -2792,9 +2801,13 @@ static int __set_cpus_allowed_ptr_locked__do_set_cpus_allowed (p, new_mask, flags);if (flags &SCA_USER ) -release_user_cpus_ptr (p); + user_mask =clear_user_cpus_ptr (p); -return affine_move_task (rq, p, rf, dest_cpu, flags); + ret =affine_move_task (rq, p, rf, dest_cpu, flags); + +kfree (user_mask); + +return ret; out:task_rq_unlock (rq, p, rf);
@@ -2941,20 +2954,22 @@ __sched_setaffinity(struct task_struct **/ void relax_compatible_cpus_allowed_ptr (struct task_struct *p) { +struct cpumask *user_mask = p->user_cpus_ptr;unsigned long flags; -struct cpumask *mask = p->user_cpus_ptr;/* * Try to restore the old affinity mask. If this fails, then * we free the mask explicitly to avoid it being inherited across * a subsequent fork(). */ -if (!mask || !__sched_setaffinity (p, mask)) +if (!user_mask || !__sched_setaffinity (p, user_mask))return ;raw_spin_lock_irqsave (&p->pi_lock, flags); -release_user_cpus_ptr (p); + user_mask =clear_user_cpus_ptr (p);raw_spin_unlock_irqrestore (&p->pi_lock, flags); + +kfree (user_mask); }void set_task_cpu (struct task_struct *p,unsigned int new_cpu)
Will Deacon
maintainerRe: [PATCH v11 08/16] sched: Allow task CPU affinity to be restricted on asymmetric systems
Quoted 19 lines · Peter Zijlstra
Quoted 18 lines · Peter Zijlstra
Quoted 16 lines · Will Deacon
I think the idea looks good, but perhaps we could wrap things up a bit:
/* Comment about why this is useful with RT */ static
cpumask_t*clear_user_cpus_ptr(structtask_struct*p){ struct cpumask *user_mask = NULL; swap(user_mask, p->user_cpus_ptr); return user_mask; }void
release_user_cpus_ptr(struct task_struct *p){ kfree(clear_user_cpus_ptr(p)); }
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Then just use
clear_user_cpus_ptr()insched/core.cwhere we know what we're doing (well, at least one of us does!).OK, I'll go make it like that.
Something like so then?
Looks good to me, thanks!
Will
[tip: sched/core] sched: Introduce dl_task_check_affinity() to check proposed affinity
In reply to Will DeaconThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 234b8ab6476c5edd5262e2ff563de9498d60044a Gitweb: https://git.kernel.org/tip/234b8ab6476c5edd5262e2ff563de9498d60044a Author: Will Deacon <[email protected]> AuthorDate: Fri, 30 Jul 2021 12:24:36 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 20 Aug 2021 12:33:00 +02:00
sched: Introduce dl_task_check_affinity() to check proposed affinity
In preparation for restricting the affinity of a task during execve() on arm64, introduce a new dl_task_check_affinity() helper function to give an indication as to whether the restricted mask is admissible for a deadline task.
--- include/linux/sched.h | 6 +++++- kernel/sched/core.c | 46 ++++++++++++++++++++++++++---------------- 2 files changed, 35 insertions(+), 17 deletions(-)
include/linux/sched.h +6 -0
@@ -1709,6 +1709,7 @@ extern void do_set_cpus_allowed(struct task_struct *p, const struct cpumask *newextern int set_cpus_allowed_ptr (struct task_struct *p,const struct cpumask *new_mask);extern int dup_user_cpus_ptr (struct task_struct *dst,struct task_struct *src,int node);extern void release_user_cpus_ptr (struct task_struct *p); +extern int dl_task_check_affinity (struct task_struct *p,const struct cpumask *mask);extern void force_compatible_cpus_allowed_ptr (struct task_struct *p);extern void relax_compatible_cpus_allowed_ptr (struct task_struct *p);#else
@@ -1731,6 +1732,11 @@ static inline void release_user_cpus_ptr(struct task_struct *p) {WARN_ON (p->user_cpus_ptr); } + +static inline int dl_task_check_affinity (struct task_struct *p,const struct cpumask *mask) +{ +return 0 ; +}#endif extern int yield_to (struct task_struct *p,bool preempt);
kernel/sched/core.c +29 -17
58 lines not on this page · raw message
[tip: sched/core] sched: Allow task CPU affinity to be restricted on asymmetric systems
In reply to Will DeaconThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 07ec77a1d4e82526e1588979fff2f024f8e96df2 Gitweb: https://git.kernel.org/tip/07ec77a1d4e82526e1588979fff2f024f8e96df2 Author: Will Deacon <[email protected]> AuthorDate: Fri, 30 Jul 2021 12:24:35 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 20 Aug 2021 12:33:00 +02:00
sched: Allow task CPU affinity to be restricted on asymmetric systems
Asymmetric systems may not offer the same level of userspace ISA support across all CPUs, meaning that some applications cannot be executed by some CPUs. As a concrete example, upcoming arm64 big.LITTLE designs do not feature support for 32-bit applications on both clusters.
Although userspace can carefully manage the affinity masks for such tasks, one place where it is particularly problematic is execve() because the CPU on which the execve() is occurring may be incompatible with the new application image. In such a situation, it is desirable to restrict the affinity mask of the task and ensure that the new image is entered on a compatible CPU. From userspace's point of view, this looks the same as if the incompatible CPUs have been hotplugged off in the task's affinity mask. Similarly, if a subsequent execve() reverts to a compatible image, then the old affinity is restored if it is still valid.
In preparation for restricting the affinity mask for compat tasks on arm64 systems without uniform support for 32-bit applications, introduce {force,relax}_compatible_cpus_allowed_ptr(), which respectively restrict and restore the affinity mask for a task based on the compatible CPUs.
--- include/linux/sched.h | 2 +- kernel/sched/core.c | 198 +++++++++++++++++++++++++++++++++++++---- kernel/sched/sched.h | 1 +- 3 files changed, 183 insertions(+), 18 deletions(-)
include/linux/sched.h +2 -0
@@ -1709,6 +1709,8 @@ extern void do_set_cpus_allowed(struct task_struct *p, const struct cpumask *newextern int set_cpus_allowed_ptr (struct task_struct *p,const struct cpumask *new_mask);extern int dup_user_cpus_ptr (struct task_struct *dst,struct task_struct *src,int node);extern void release_user_cpus_ptr (struct task_struct *p); +extern void force_compatible_cpus_allowed_ptr (struct task_struct *p); +extern void relax_compatible_cpus_allowed_ptr (struct task_struct *p);#else static inline void do_set_cpus_allowed (struct task_struct *p,const struct cpumask *new_mask) {
kernel/sched/core.c +180 -18
244 lines not on this page · raw message
kernel/sched/sched.h +1 -0
@@ -2244,6 +2244,7 @@ extern struct task_struct *pick_next_task_idle(struct rq *rq);#define SCA_CHECK 0x01#define SCA_MIGRATE_DISABLE 0x02#define SCA_MIGRATE_ENABLE 0x04 +#define SCA_USER 0x08#ifdef CONFIG_SMP
[tip: sched/core] sched: Split the guts of sched_setaffinity() into a helper function
In reply to Will DeaconThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: db3b02ae896e88b6bb7a95c1373602e87e0de84c Gitweb: https://git.kernel.org/tip/db3b02ae896e88b6bb7a95c1373602e87e0de84c Author: Will Deacon <[email protected]> AuthorDate: Fri, 30 Jul 2021 12:24:34 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 20 Aug 2021 12:33:00 +02:00
sched: Split the guts of sched_setaffinity() into a helper function
In preparation for replaying user affinity requests using a saved mask, split sched_setaffinity() up so that the initial task lookup and security checks are only performed when the request is coming directly from userspace.
--- kernel/sched/core.c | 105 +++++++++++++++++++++++-------------------- 1 file changed, 57 insertions(+), 48 deletions(-)
kernel/sched/core.c +57 -48
@@ -7594,53 +7594,22 @@ out_unlock:return retval; } -long sched_setaffinity (pid_t pid,const struct cpumask *in_mask) +static int +__sched_setaffinity (struct task_struct *p,const struct cpumask *mask) { -cpumask_var_t cpus_allowed, new_mask; -struct task_struct *p;int retval; +cpumask_var_t cpus_allowed, new_mask; -rcu_read_lock (); - - p =find_process_by_pid (pid); -if (!p) { -rcu_read_unlock (); -return -ESRCH ; - } - -/* Prevent p going away */ -get_task_struct (p); -rcu_read_unlock (); +if (!alloc_cpumask_var (&cpus_allowed,GFP_KERNEL )) +return -ENOMEM ; -if (p->flags &PF_NO_SETAFFINITY ) { - retval = -EINVAL ; - goto out_put_task; - } -if (!alloc_cpumask_var (&cpus_allowed,GFP_KERNEL )) { - retval = -ENOMEM ; - goto out_put_task; - }if (!alloc_cpumask_var (&new_mask,GFP_KERNEL )) { retval = -ENOMEM ; goto out_free_cpus_allowed; } - retval = -EPERM ; -if (!check_same_owner (p)) { -rcu_read_lock (); -if (!ns_capable (__task_cred (p)->user_ns,CAP_SYS_NICE )) { -rcu_read_unlock (); - goto out_free_new_mask; - } -rcu_read_unlock (); - } - - retval =security_task_setscheduler (p); -if (retval) - goto out_free_new_mask; -cpuset_cpus_allowed (p, cpus_allowed); -cpumask_and (new_mask, in_mask, cpus_allowed); +cpumask_and (new_mask, mask, cpus_allowed);/* * Since bandwidth control happens on root_domain basis,
@@ -7661,23 +7630,63 @@ long sched_setaffinity(pid_t pid, const struct cpumask *in_mask)#endif again: retval =__set_cpus_allowed_ptr (p, new_mask,SCA_CHECK ); +if (retval) + goto out_free_new_mask; -if (!retval) { -cpuset_cpus_allowed (p, cpus_allowed); -if (!cpumask_subset (new_mask, cpus_allowed)) { -/* -* We must have raced with a concurrent cpuset -* update. Just reset the cpus_allowed to the -* cpuset's cpus_allowed -*/ -cpumask_copy (new_mask, cpus_allowed); - goto again; - } +cpuset_cpus_allowed (p, cpus_allowed); +if (!cpumask_subset (new_mask, cpus_allowed)) { +/* +* We must have raced with a concurrent cpuset update. +* Just reset the cpumask to the cpuset's cpus_allowed. +*/ +cpumask_copy (new_mask, cpus_allowed); + goto again; } + out_free_new_mask:free_cpumask_var (new_mask); out_free_cpus_allowed:free_cpumask_var (cpus_allowed); +return retval; +} + +long sched_setaffinity (pid_t pid,const struct cpumask *in_mask) +{ +struct task_struct *p; +int retval; + +rcu_read_lock (); + + p =find_process_by_pid (pid); +if (!p) { +rcu_read_unlock (); +return -ESRCH ; + } + +/* Prevent p going away */ +get_task_struct (p); +rcu_read_unlock (); + +if (p->flags &PF_NO_SETAFFINITY ) { + retval = -EINVAL ; + goto out_put_task; + } + +if (!check_same_owner (p)) { +rcu_read_lock (); +if (!ns_capable (__task_cred (p)->user_ns,CAP_SYS_NICE )) { +rcu_read_unlock (); + retval = -EPERM ; + goto out_put_task; + } +rcu_read_unlock (); + } + + retval =security_task_setscheduler (p); +if (retval) + goto out_put_task; + + retval =__sched_setaffinity (p, in_mask); out_put_task:put_task_struct (p);return retval;
[tip: sched/core] sched: Introduce task_struct::user_cpus_ptr to track requested affinity
In reply to Will DeaconThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: b90ca8badbd11488e5f762346b028666808164e7 Gitweb: https://git.kernel.org/tip/b90ca8badbd11488e5f762346b028666808164e7 Author: Will Deacon <[email protected]> AuthorDate: Fri, 30 Jul 2021 12:24:33 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 20 Aug 2021 12:33:00 +02:00
sched: Introduce task_struct::user_cpus_ptr to track requested affinity
In preparation for saving and restoring the user-requested CPU affinity mask of a task, add a new cpumask_t pointer to 'struct task_struct'.
If the pointer is non-NULL, then the mask is copied across fork() and freed on task exit.
--- include/linux/sched.h | 13 +++++++++++++ init/init_task.c | 1 + kernel/fork.c | 2 ++ kernel/sched/core.c | 20 ++++++++++++++++++++ 4 files changed, 36 insertions(+)
include/linux/sched.h +13 -0
@@ -748,6 +748,7 @@ struct task_struct {unsigned int policy;int nr_cpus_allowed;const cpumask_t *cpus_ptr; +cpumask_t *user_cpus_ptr;cpumask_t cpus_mask;void *migration_pending;#ifdef CONFIG_SMP
@@ -1706,6 +1707,8 @@ extern int task_can_attach(struct task_struct *p, const struct cpumask *cs_cpus_#ifdef CONFIG_SMP extern void do_set_cpus_allowed (struct task_struct *p,const struct cpumask *new_mask);extern int set_cpus_allowed_ptr (struct task_struct *p,const struct cpumask *new_mask); +extern int dup_user_cpus_ptr (struct task_struct *dst,struct task_struct *src,int node); +extern void release_user_cpus_ptr (struct task_struct *p);#else static inline void do_set_cpus_allowed (struct task_struct *p,const struct cpumask *new_mask) {
@@ -1716,6 +1719,16 @@ static inline int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumareturn -EINVAL ; return 0; } +static inline int dup_user_cpus_ptr (struct task_struct *dst,struct task_struct *src,int node) +{ +if (src->user_cpus_ptr) +return -EINVAL ; +return 0 ; +} +static inline void release_user_cpus_ptr (struct task_struct *p) +{ +WARN_ON (p->user_cpus_ptr); +}#endif extern int yield_to (struct task_struct *p,bool preempt);
init/init_task.c +1 -0
@@ -80,6 +80,7 @@ struct task_struct init_task .normal_prio = MAX_PRIO - 20, .policy = SCHED_NORMAL, .cpus_ptr = &init_task.cpus_mask, + .user_cpus_ptr = NULL, .cpus_mask = CPU_MASK_ALL, .nr_cpus_allowed= NR_CPUS, .mm = NULL,
kernel/fork.c +2 -0
@@ -446,6 +446,7 @@ void put_task_stack(struct task_struct *tsk)void free_task (struct task_struct *tsk) { +release_user_cpus_ptr (tsk);scs_release (tsk);#ifndef CONFIG_THREAD_INFO_IN_TASK
@@ -919,6 +920,7 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)#endif if (orig->cpus_ptr == &orig->cpus_mask) tsk->cpus_ptr = &tsk->cpus_mask; +dup_user_cpus_ptr (tsk, orig, node);/* * One for the user space visible state that goes away when reaped.
kernel/sched/core.c +20 -0
@@ -2480,6 +2480,26 @@ void do_set_cpus_allowed(struct task_struct *p, const struct cpumask *new_mask)__do_set_cpus_allowed (p, new_mask,0 ); } +int dup_user_cpus_ptr (struct task_struct *dst,struct task_struct *src, +int node) +{ +if (!src->user_cpus_ptr) +return 0 ; + + dst->user_cpus_ptr =kmalloc_node (cpumask_size (),GFP_KERNEL , node); +if (!dst->user_cpus_ptr) +return -ENOMEM ; + +cpumask_copy (dst->user_cpus_ptr, src->user_cpus_ptr); +return 0 ; +} + +void release_user_cpus_ptr (struct task_struct *p) +{ +kfree (p->user_cpus_ptr); + p->user_cpus_ptr =NULL ; +} +/* * This function is wildly self concurrent; here be dragons. *
[tip: sched/core] cpuset: Cleanup cpuset_cpus_allowed_fallback() use in select_fallback_rq()
In reply to Will DeaconThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 97c0054dbe2c3c59d1156fd233f2d44e91981c8e Gitweb: https://git.kernel.org/tip/97c0054dbe2c3c59d1156fd233f2d44e91981c8e Author: Will Deacon <[email protected]> AuthorDate: Fri, 30 Jul 2021 12:24:31 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 20 Aug 2021 12:32:59 +02:00
cpuset: Cleanup cpuset_cpus_allowed_fallback() use in select_fallback_rq()
select_fallback_rq() only needs to recheck for an allowed CPU if the affinity mask of the task has changed since the last check.
Return a 'bool' from cpuset_cpus_allowed_fallback() to indicate whether the affinity mask was updated, and use this to elide the allowed check when the mask has been left alone.
No functional change.
--- include/linux/cpuset.h | 5 +++-- kernel/cgroup/cpuset.c | 10 ++++++++-- kernel/sched/core.c | 3 +-- 3 files changed, 12 insertions(+), 6 deletions(-)
include/linux/cpuset.h +3 -2
@@ -59,7 +59,7 @@ extern void cpuset_wait_for_hotplug(void);extern void cpuset_read_lock (void );extern void cpuset_read_unlock (void );extern void cpuset_cpus_allowed (struct task_struct *p,struct cpumask *mask); -extern void cpuset_cpus_allowed_fallback (struct task_struct *p); +extern bool cpuset_cpus_allowed_fallback (struct task_struct *p);extern nodemask_t cpuset_mems_allowed (struct task_struct *p);#define cpuset_current_mems_allowed (current->mems_allowed)void cpuset_init_current_mems_allowed (void );
@@ -188,8 +188,9 @@ static inline void cpuset_cpus_allowed(struct task_struct *p,cpumask_copy (mask,task_cpu_possible_mask (p)); } -static inline void cpuset_cpus_allowed_fallback (struct task_struct *p) +static inline bool cpuset_cpus_allowed_fallback (struct task_struct *p) { +return false; }static inline nodemask_t cpuset_mems_allowed (struct task_struct *p)
kernel/cgroup/cpuset.c +8 -2
@@ -3327,17 +3327,22 @@ void cpuset_cpus_allowed(struct task_struct *tsk, struct cpumask *pmask)* which will not contain a sane cpumask during cases such as cpu hotplugging. * This is the absolute last resort for the scheduler and it is only used if * _every_ other avenue has been traveled. +* +* Returns true if the affinity of @tsk was changed, false otherwise. **/ -void cpuset_cpus_allowed_fallback (struct task_struct *tsk) +bool cpuset_cpus_allowed_fallback (struct task_struct *tsk) {const struct cpumask *possible_mask =task_cpu_possible_mask (tsk);const struct cpumask *cs_mask; +bool changed = false;rcu_read_lock (); cs_mask =task_cs (tsk)->cpus_allowed; -if (is_in_v2_mode () &&cpumask_subset (cs_mask, possible_mask)) +if (is_in_v2_mode () &&cpumask_subset (cs_mask, possible_mask)) {do_set_cpus_allowed (tsk, cs_mask); + changed = true; + }rcu_read_unlock ();/*
@@ -3357,6 +3362,7 @@ void cpuset_cpus_allowed_fallback(struct task_struct *tsk)* select_fallback_rq() will fix things ups and set cpu_possible_mask * if required. */ +return changed; }void __init cpuset_init_current_mems_allowed (void )
kernel/sched/core.c +1 -2
@@ -3141,8 +3141,7 @@ static int select_fallback_rq(int cpu, struct task_struct *p)/* No more Mr. Nice Guy. */ switch (state) {case cpuset: -if (IS_ENABLED (CONFIG_CPUSETS )) { -cpuset_cpus_allowed_fallback (p); +if (cpuset_cpus_allowed_fallback (p)) { state = possible;break ; }
[tip: sched/core] sched: Reject CPU affinity changes based on task_cpu_possible_mask()
In reply to Will DeaconThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 234a503e670be01f72841be9fcf68dfb89a1fa8b Gitweb: https://git.kernel.org/tip/234a503e670be01f72841be9fcf68dfb89a1fa8b Author: Will Deacon <[email protected]> AuthorDate: Fri, 30 Jul 2021 12:24:32 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 20 Aug 2021 12:32:59 +02:00
sched: Reject CPU affinity changes based on task_cpu_possible_mask()
Reject explicit requests to change the affinity mask of a task via set_cpus_allowed_ptr() if the requested mask is not a subset of the mask returned by task_cpu_possible_mask(). This ensures that the 'cpus_mask' for a given task cannot contain CPUs which are incapable of executing it, except in cases where the affinity is forced.
--- kernel/sched/core.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
kernel/sched/core.c +8 -1
@@ -2709,7 +2709,9 @@ static int __set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask, u32 flags) { +const struct cpumask *cpu_allowed_mask =task_cpu_possible_mask (p);const struct cpumask *cpu_valid_mask = cpu_active_mask; +bool kthread = p->flags &PF_KTHREAD ;unsigned int dest_cpu;struct rq_flags rf;struct rq *rq;
@@ -2718,7 +2720,7 @@ static int __set_cpus_allowed_ptr(struct task_struct *p, rq =task_rq_lock (p, &rf);update_rq_clock (rq); -if (p->flags &PF_KTHREAD ||is_migration_disabled (p)) { +if (kthread ||is_migration_disabled (p)) {/* * Kernel threads are allowed on online && !active CPUs, * however, during cpu-hot-unplug, even these might get pushed
@@ -2732,6 +2734,11 @@ static int __set_cpus_allowed_ptr(struct task_struct *p, cpu_valid_mask = cpu_online_mask; } +if (!kthread && !cpumask_subset (new_mask, cpu_allowed_mask)) { + ret = -EINVAL ; + goto out; + } +/* * Must re-check here, to close a race against __kthread_bind(), * sched_setaffinity() is not guaranteed to observe the flag.
[tip: sched/core] cpuset: Honour task_cpu_possible_mask() in guarantee_online_cpus()
In reply to Will DeaconThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 431c69fac05baa7477d61a44f2708e069f2bed6c Gitweb: https://git.kernel.org/tip/431c69fac05baa7477d61a44f2708e069f2bed6c Author: Will Deacon <[email protected]> AuthorDate: Fri, 30 Jul 2021 12:24:30 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 20 Aug 2021 12:32:59 +02:00
cpuset: Honour task_cpu_possible_mask() in guarantee_online_cpus()
Asymmetric systems may not offer the same level of userspace ISA support across all CPUs, meaning that some applications cannot be executed by some CPUs. As a concrete example, upcoming arm64 big.LITTLE designs do not feature support for 32-bit applications on both clusters.
Modify guarantee_online_cpus() to take task_cpu_possible_mask() into account when trying to find a suitable set of online CPUs for a given task. This will avoid passing an invalid mask to set_cpus_allowed_ptr() during ->attach() and will subsequently allow the cpuset hierarchy to be taken into account when forcefully overriding the affinity mask for a task which requires migration to a compatible CPU.
--- include/linux/cpuset.h | 2 +- kernel/cgroup/cpuset.c | 43 ++++++++++++++++++++++++----------------- 2 files changed, 27 insertions(+), 18 deletions(-)
include/linux/cpuset.h +1 -1
@@ -185,7 +185,7 @@ static inline void cpuset_read_unlock(void) { }static inline void cpuset_cpus_allowed (struct task_struct *p,struct cpumask *mask) { -cpumask_copy (mask, cpu_possible_mask); +cpumask_copy (mask,task_cpu_possible_mask (p)); }static inline void cpuset_cpus_allowed_fallback (struct task_struct *p)
kernel/cgroup/cpuset.c +26 -17
79 lines not on this page · raw message
[tip: sched/core] cpuset: Don't use the cpu_possible_mask as a last resort for cgroup v1
In reply to Will DeaconThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: d4b96fb92ae7fe7533e11e662504d96161928575 Gitweb: https://git.kernel.org/tip/d4b96fb92ae7fe7533e11e662504d96161928575 Author: Will Deacon <[email protected]> AuthorDate: Fri, 30 Jul 2021 12:24:29 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 20 Aug 2021 12:32:58 +02:00
cpuset: Don't use the cpu_possible_mask as a last resort for cgroup v1
If the scheduler cannot find an allowed CPU for a task, cpuset_cpus_allowed_fallback() will widen the affinity to cpu_possible_mask if cgroup v1 is in use.
In preparation for allowing architectures to provide their own fallback mask, just return early if we're either using cgroup v1 or we're using cgroup v2 with a mask that contains invalid CPUs. This will allow select_fallback_rq() to figure out the mask by itself.
--- include/linux/cpuset.h | 1 + kernel/cgroup/cpuset.c | 8 ++++++-- 2 files changed, 7 insertions(+), 2 deletions(-)
include/linux/cpuset.h +1 -0
@@ -15,6 +15,7 @@#include <linux/cpumask.h> #include <linux/nodemask.h> #include <linux/mm.h> +#include <linux/mmu_context.h> #include <linux/jump_label.h> #ifdef CONFIG_CPUSETS
kernel/cgroup/cpuset.c +6 -2
@@ -3322,9 +3322,13 @@ void cpuset_cpus_allowed(struct task_struct *tsk, struct cpumask *pmask)void cpuset_cpus_allowed_fallback (struct task_struct *tsk) { +const struct cpumask *possible_mask =task_cpu_possible_mask (tsk); +const struct cpumask *cs_mask; +rcu_read_lock (); -do_set_cpus_allowed (tsk,is_in_v2_mode () ? -task_cs (tsk)->cpus_allowed : cpu_possible_mask); + cs_mask =task_cs (tsk)->cpus_allowed; +if (is_in_v2_mode () &&cpumask_subset (cs_mask, possible_mask)) +do_set_cpus_allowed (tsk, cs_mask);rcu_read_unlock ();/*
[tip: sched/core] sched: Introduce task_cpu_possible_mask() to limit fallback rq selection
In reply to Will DeaconThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 9ae606bc74dd0e58d4de894e3c5cbb9d45599267 Gitweb: https://git.kernel.org/tip/9ae606bc74dd0e58d4de894e3c5cbb9d45599267 Author: Will Deacon <[email protected]> AuthorDate: Fri, 30 Jul 2021 12:24:28 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 20 Aug 2021 12:32:58 +02:00
sched: Introduce task_cpu_possible_mask() to limit fallback rq selection
Asymmetric systems may not offer the same level of userspace ISA support across all CPUs, meaning that some applications cannot be executed by some CPUs. As a concrete example, upcoming arm64 big.LITTLE designs do not feature support for 32-bit applications on both clusters.
On such a system, we must take care not to migrate a task to an unsupported CPU when forcefully moving tasks in select_fallback_rq() in response to a CPU hot-unplug operation.
Introduce a task_cpu_possible_mask() hook which, given a task argument, allows an architecture to return a cpumask of CPUs that are capable of executing that task. The default implementation returns the cpu_possible_mask, since sane machines do not suffer from per-cpu ISA limitations that affect scheduling. The new mask is used when selecting the fallback runqueue as a last resort before forcing a migration to the first active CPU.
--- include/linux/mmu_context.h | 14 ++++++++++++++ kernel/sched/core.c | 9 +++------ 2 files changed, 17 insertions(+), 6 deletions(-)
include/linux/mmu_context.h +14 -0
@@ -14,4 +14,18 @@static inline void leave_mm (int cpu) { }#endif +/* +* CPUs that are capable of running user task @p. Must contain at least one +* active CPU. It is assumed that the kernel can run on all CPUs, so calling +* this for a kernel thread is pointless. +* +* By default, we assume a sane, homogeneous system. +*/ +#ifndef task_cpu_possible_mask +# define task_cpu_possible_mask (p) cpu_possible_mask +# define task_cpu_possible (cpu, p) true +#else +# define task_cpu_possible (cpu, p) cpumask_test_cpu((cpu), task_cpu_possible_mask(p)) +#endif +#endif
kernel/sched/core.c +3 -6
@@ -2173,7 +2173,7 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)/* Non kernel threads are not allowed during either online or offline. */ if (!(p->flags &PF_KTHREAD )) -return cpu_active (cpu); +return cpu_active (cpu) &&task_cpu_possible (cpu, p);/* KTHREAD_IS_PER_CPU is always allowed. */ if (kthread_is_per_cpu (p))
@@ -3124,9 +3124,7 @@ static int select_fallback_rq(int cpu, struct task_struct *p)/* Look for allowed, online CPU in same node. */ for_each_cpu (dest_cpu, nodemask) { -if (!cpu_active (dest_cpu)) -continue ; -if (cpumask_test_cpu (dest_cpu, p->cpus_ptr)) +if (is_cpu_allowed (p, dest_cpu))return dest_cpu; } }
@@ -3156,10 +3154,9 @@ static int select_fallback_rq(int cpu, struct task_struct *p)* * More yuck to audit. */ -do_set_cpus_allowed (p, cpu_possible_mask); +do_set_cpus_allowed (p,task_cpu_possible_mask (p)); state = fail;break ; -case fail:BUG ();break ;