sched/mmcid: fixes for execve, mode switches and affinity
Subsystems (7)
Mathieu Desnoyers
maintainercover sched/mmcid: fixes for execve, mode switches and affinity
Hi Thomas, Peter,
This series fixes six problems of the rseq concurrency IDs (mm_cid), and adds selftests for them. I looked for cases the existing tests do not reach:
- a thread count that goes up and down;
- exit, fork and a failed execve();
- affinity changes made by another process.
The fixes come first, and all six are marked for stable:
1 Give a CID to a task that joins its mm while running. A failed
execve() then no longer returns to user space with an mm_cid of 0x80000000.
2 Give no CID to io_uring and vhost workers, so that they do not
take IDs from the threads of the process.
3 Reset the stale per CPU state of an mm when its first user joins.
A machine with two or three CPUs no longer locks up after a failed execve().
4 Keep a running joiner inactive until the fixups are done, so that
it cannot loop in mm_get_cid() while it holds mm_cid.mutex.
5 Add the allowed CPUs of the first user to the mm when it joins.
An affinity change by another task during execve() is then not lost.
6 Fix the use after free of the old mm in set_cpus_allowed_common().
The tests, all in tools/testing/selftests/rseq. Tests 7, 8, 12 and 17 fail without the fix they check:
7 mm_cid_execve_test: the mm_cid after a failed execve(). For patch
1.
8 The last user fails execve() on another CPU. For patch 3, on
guests with 2 and 3 CPUs.
9 mm_cid_transition_test: threads count per mm_cid while the thread
count crosses the mode thresholds.
10 mm_cid_bound_test: the largest mm_cid stays below the possible
CPUs. It prints how far the IDs go above the max_cids target.
11 A transition case: affinity changes made by another process. 12 mm_cid_exec_affinity_test. For patch 5, on guests with 4 to 64
CPUs.
13 Transition cases: threads created by several threads at once, and
threads migrated by another process.
14 A bound case: the IDs after the affinity is widened. It prints
when they are not compact.
15 mm_cid_window_test: two directed cases for mode switch windows.
For patch 4, see the last point below.
16 mm_cid_teardown_test, for patch 6.
17 A second case for patch 3, see the first point below.
Points I would like your opinion on:
- Patch 3. Thomas, this is your cleanup from the thread "[PATCH] sched/mmcid: Bound the CID allocation busy wait", which you preferred to my fix. I wrote its subject and changelog. Change them as you see fit. Without your Signed-off-by, the patch carries my name, with an
Originally-by: tag for you. If you send your Signed-off-by, I will make you its author. Three remarks on the code:
- From patch 5 on, the cleanup runs in sched_mm_cid_add_first_user(), under t->pi_lock and mm_cid.lock. Its loop over the possible CPUs then runs with interrupts off. It could move out of the raw locks:
mm_cid.mutex is held and the mm has no user.
- The cleanup copies the allowed CPUs of the task when the mode is not 0. Patch 5 adds them at every first join. One of the two is enough.
- The stale state is not limited to two or three possible CPUs. mm_cid_calc_pcpu_thrs() also returns 1 when the mm counts one allowed CPU. Test 17 gets a CID leaked that way on guests with 4, 8 and 64 CPUs. It needs an affinity change that the mm missed during an execve(). So it fails only without both patch 3 and patch 5. I reproduced the lockup itself with two and three possible CPUs only.
I did not show that this lockup is the splat of that thread.
- Patch 5 has Fixes: b0c3d51b54f8. Part of its window is older and dates from commit 7e019dcc470f ("sched: Improve cache locality of RSEQ concurrency IDs for intermittent workloads").
- Test 15 passes on a kernel without the two fixes it targets: its windows are a few instructions wide. It fails there only when the kernel makes the chosen thread sleep in the window. I did that with a local debug prctl(), which is not in this series. The test marks the places with "Delay injection" comments. Would you want such delay points upstream, for instance as fault injection?
Testing
=======
All tests pass in x86-64 QEMU guests with lockdep, with 2 to 64 CPUs. Also:
- Without patch 4 and without delay injection, the sequence of test 15 hangs a guest with 2 CPUs when a SCHED_FIFO task preempts the joiner.
- Patch 2 has no test.
- I did not reproduce the use after free that patch 6 fixes. Its exec case needs CONFIG_FUTEX=n. Test 16 passes on a KASAN guest with CONFIG_FUTEX=n and the fix, with no report.
- Tests 11, 13 and 16 are stress tests. They found no bug.
A Spin model of the mode switches found the problem of patch 4. It is not part of the series.
How this was developed
======================
An LLM coding assistant reviewed the mm_cid code under my direction. It wrote all the patches, their messages and the tests, which I reviewed. There are two exceptions. The code of patch 3 is Thomas's. Patch 6 is Weiming Shi's, carried as he posted it. Test 16 follows the shape of his reproducer, and takes no code from it.
A reproducer that stops failing shows that the symptom goes away. It does not show that the fix is the one you would have written, so I ask for a careful reading.
Based on tip sched/urgent, 006991aa8997.
Links:
- Weiming Shi, "[PATCH v3] sched/mmcid: Synchronize MM teardown with affinity updates":
https://lore.kernel.org/all/[email protected]/
- "[PATCH] sched/mmcid: Bound the CID allocation busy wait":
https://lore.kernel.org/all/87mrs8bhp4.ffs@fw13/
- My reply in that thread, with my variant of patch 3 and the reproducer:
https://lore.kernel.org/all/[email protected]/
- Thomas's answer, which prefers the cleanup:
https://lore.kernel.org/all/87eccossye.ffs@fw13/
Mathieu Desnoyers (16):
sched/mmcid: Give a CID to a task that joins its mm while running sched/mmcid: Give no concurrency ID to user workers sched/mmcid: Reset stale state when the first user joins an mm sched/mmcid: Let a running joiner take its CID after the fixups sched/mmcid: Take the allowed CPUs of the first user when it joins selftests/rseq: Test the concurrency ID after a failed execve selftests/rseq: Test the concurrency ID after the last user fails
execve
selftests/rseq: Test the concurrency IDs across ownership mode
switches
selftests/rseq: Test the upper bound of the concurrency IDs selftests/rseq: Test affinity changes made by another process selftests/rseq: Test an affinity change by another process during
execve
selftests/rseq: Test concurrent thread creation and forced migration selftests/rseq: Test the concurrency IDs after the affinity is widened selftests/rseq: Test two race windows of the concurrency ID mode
switches
selftests/rseq: Test affinity changes while a task replaces or drops
its mm
selftests/rseq: Test the last user after an affinity change during
execve
Weiming Shi (1):
sched/mmcid: Synchronize MM teardown with affinity updates
fs/exec.c | 1 + include/linux/sched.h | 2 + kernel/exit.c | 1 + kernel/kthread.c | 1 + kernel/sched/core.c | 151 +++- tools/testing/selftests/rseq/.gitignore | 5 + tools/testing/selftests/rseq/Makefile | 5 +- .../selftests/rseq/mm_cid_bound_test.c | 441 +++++++++++ .../rseq/mm_cid_exec_affinity_test.c | 283 ++++++++ .../selftests/rseq/mm_cid_execve_test.c | 685 ++++++++++++++++++ .../selftests/rseq/mm_cid_teardown_test.c | 396 ++++++++++ .../selftests/rseq/mm_cid_transition_test.c | 511 +++++++++++++ .../selftests/rseq/mm_cid_window_test.c | 485 +++++++++++++ 13 files changed, 2942 insertions(+), 25 deletions(-) create mode 100644 tools/testing/selftests/rseq/mm_cid_bound_test.c create mode 100644 tools/testing/selftests/rseq/mm_cid_exec_affinity_test.c create mode 100644 tools/testing/selftests/rseq/mm_cid_execve_test.c create mode 100644 tools/testing/selftests/rseq/mm_cid_teardown_test.c create mode 100644 tools/testing/selftests/rseq/mm_cid_transition_test.c create mode 100644 tools/testing/selftests/rseq/mm_cid_window_test.c
base-commit: 006991aa89974378ee579416dba87adc22cd60fd
Signature · 1 line
2.47.3
Mathieu Desnoyers
maintainerPATCH 1/17 sched/mmcid: Give a CID to a task that joins its mm while running
sched_mm_cid_fork() makes a task a user of the CIDs of its mm. In per CPU mode the new user gets no CID there. It takes the CID of its CPU when it is scheduled in.
This relies on the new user not running yet. That holds for a fork() child. A successful execve() joins a new mm as its first user, and the first user gets a CID directly.
A failed execve() is missed. bprm_execve() leaves the mm in sched_mm_cid_before_execve(). Its error path rejoins the same mm in sched_mm_cid_after_execve(). If the execve() fails before begin_new_exec() kills the other threads, they are still users of the mm. The mm can then be in per CPU mode. The task is running, so it returns to user space with no schedule in. Its CID is still MM_CID_UNSET, and rseq_force_update() makes the kernel write it to the rseq area. User space reads an mm_cid of 0x80000000 until its next context switch. A program that indexes per-CID data with it would read out of bounds.
An execve() of a file that no binfmt handler recognises fails in that window. In testing, with 20 threads on a machine with 8 CPUs:
mm_cid 2147483648 (0x80000000) after a failed execve, round 0
Fix this by giving a running joiner its CID in sched_mm_cid_fork(). It takes the CID a schedule in would give it, under the runqueue lock.
The switch back to per task mode has the same problem. Since commit 7dbc307b6c72 ("sched/mmcid: Fix affinity-triggered switch back to per-task mode"), a join can do the switch that an affinity change queued. For a running joiner:
1. sched_mm_cid_fork() runs mm_cid_fixup_cpus_to_tasks().
2. The joiner is preempted. It takes a CID at its schedule in.
3. sched_mm_cid_fork() allocates a CID for the new user. That overwrites the CID of step 2.
The bit of the first CID stays set in the bitmap, with no owner. In testing, three threads on two CPUs then read an mm_cid of 3 or more. Leave that allocation to a joiner that is not running. A running joiner takes its CID as above if it has none.
--- kernel/sched/core.c | 43 +++++++++++++++++++++++++++++++++++-------- 1 file changed, 35 insertions(+), 8 deletions(-)
kernel/sched/core.c +35 -8
@@ -10986,10 +10986,19 @@ static bool sched_mm_cid_add_user(struct task_struct *t, struct mm_struct *mm)return mm_update_max_cids (mm); } +/* +* Make @t a user of the CIDs of its mm. Called on fork(2) and on +* execve(2). A failed execve(2) rejoins the mm it left. +* +* In per CPU mode a new user gets no CID here. It takes the CID of its +* CPU when it is scheduled in. A task that joins while it is running +* returns to user space before that, so it takes its CID here. This is +* the case of a failed execve(2) when the mm has other users. +*/ static void sched_mm_cid_fork (struct task_struct *t) {struct mm_struct *mm = t->mm; -bool percpu; +bool percpu = false, flip = false;if (!mm)return ;
@@ -11012,10 +11021,14 @@ static void sched_mm_cid_fork(struct task_struct *t)if (!sched_mm_cid_add_user (t, mm)) {if (!cid_on_cpu (mm->mm_cid.mode)) t->mm_cid.cid =mm_get_cid (mm); -return ; +break ; } -/* Handle the mode change and transfer current's CID */ +/* +* Handle the mode change and transfer current's CID. A +* running joiner has no CID yet. It takes one below. +*/ + flip = true; percpu =cid_on_cpu (mm->mm_cid.mode);if (!percpu)mm_cid_transit_to_task (current, pcp);
@@ -11023,11 +11036,25 @@ static void sched_mm_cid_fork(struct task_struct *t)mm_cid_transit_to_cpu (current, pcp); } -if (percpu) { -mm_cid_fixup_tasks_to_cpus (); - }else { -mm_cid_fixup_cpus_to_tasks (mm); - t->mm_cid.cid =mm_get_cid (mm); +if (flip) { +if (percpu) { +mm_cid_fixup_tasks_to_cpus (); + }else { +mm_cid_fixup_cpus_to_tasks (mm); +/* +* A running joiner may have taken a CID at a +* schedule in. If not, it takes one below. +*/ +if (t != current) + t->mm_cid.cid =mm_get_cid (mm); + } + } + +/* A running joiner with no CID takes one now, see above. */ +if (t == current) { +guard (task_rq_lock)(t); +if (t->mm_cid.cid ==MM_CID_UNSET ) +mm_cid_schedin (t); } }
Signature · 1 line
2.47.3
Mathieu Desnoyers
maintainerPATCH 2/17 sched/mmcid: Give no concurrency ID to user workers
In reply to Mathieu DesnoyersThe workers of io_uring and vhost are PF_USER_WORKER tasks. They are threads of the process they work for, created with CLONE_VM. They only run kernel code: copy_process() gives them a kernel function that ends in do_exit(). They never return to user space and have no rseq area.
sched_mm_cid_fork() still makes them users of the CIDs of the mm, like every task with an mm. A worker takes a CID whenever it runs. It also counts as a user for max_cids and for the switch to per CPU mode.
Nothing reports the CID of a worker. But it is one CID fewer for the threads of the process, which then get larger ones. The added users can also switch the mm to per CPU mode. A process with io_uring workers would then read mm_cid values at or above its number of threads. This was found by reading the code, and has no test.
Fix this by not making a user worker a user of the CIDs. Kernel threads are already left out, as they have no mm.
--- kernel/sched/core.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
kernel/sched/core.c +5 -1
@@ -11000,7 +11000,11 @@ static void sched_mm_cid_fork(struct task_struct *t)struct mm_struct *mm = t->mm;bool percpu = false, flip = false; -if (!mm) +/* +* A user worker (io_uring, vhost) never returns to user space and +* has no rseq area. It needs no CID. +*/ +if (!mm || (t->flags &PF_USER_WORKER ))return ;WARN_ON_ONCE (t->mm_cid.cid !=MM_CID_UNSET );
Signature · 1 line
2.47.3
Mathieu Desnoyers
maintainerPATCH 3/17 sched/mmcid: Reset stale state when the first user joins an mm
In reply to Mathieu Desnoyerssched_mm_cid_exit() ignores the mode change when the last user of an mm leaves. Its comment says there is nothing to do. That holds for an exit:
the mm goes away. It holds for a successful execve() too, which joins a new mm.
A failed execve() is missed. It leaves the mm in sched_mm_cid_before_execve() and rejoins the same mm in sched_mm_cid_after_execve(). If the last user left in per CPU mode, MM_CID_TRANSIT stays set and the other CPUs still own their CIDs.
In the diagram below, the machine has two CPUs. T is the main thread of a process. The four threads it creates only wait. H is another process. It stays on CPU 1.
CPU 0 CPU 1
T: creates four threads
the mm switches to per CPU mode
T: runs on CPU 1
CPU 1 owns a CID
T: runs on CPU 0 again
CPU 0 owns a CID
T: joins the four threads
T is the only user left
T: execve()
sched_mm_cid_before_execve()
T leaves as the last user
drops the CID of CPU 0
MM_CID_TRANSIT is set
CPU 1 still owns its CID
H: sched_setaffinity(T, CPU 1)
T: migrates to CPU 1
execve() fails sched_mm_cid_after_execve()
sched_mm_cid_fork(T)
T is the first user mm_get_cid() succeeds writes that CID to the storage of CPU 1 the CID CPU 1 owned keeps its bit, with no owner
The mm has one CID fewer from then on. Once every remaining CID is taken, mm_get_cid() loops forever with interrupts disabled. On a guest with two CPUs, the next pthread_create() of T does not return:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: RIP: 0010:sched_mm_cid_fork+0x1dd/0x660 copy_process+0x2339/0x2ca0 kernel_clone+0xc3/0x3c0 __do_sys_clone3+0xe2/0x130
Per CPU mode lasts down to one user when mm_cid_calc_pcpu_thrs() returns
1. It does with two or three possible CPUs. It also does on any machine when the mm is allowed a single CPU. The lockup was reproduced in the first case only.
Fix this in sched_mm_cid_fork(), when the first user joins an mm whose mode is not 0. Clear the mode, the CID of every CPU and the CID bitmap, and take the allowed CPUs of the task. A new mm has mode 0, so this does nothing for it.
sched_mm_cid_exit() stays as it is. The last user of an mm that goes away runs no fixup.
Also check the user count again under the lock in mm_update_cpus_allowed(). An affinity change can race with the last user leaving. It would change the allowed CPUs of an mm with no user.
With this patch, in the same test, T keeps CID 0 after the failed execve() and its next pthread_create() returns.
[ mathieu: wrote the changelog from the thread; the code is Thomas's
draft, placed on the current sched_mm_cid_fork() ]
--- kernel/sched/core.c | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+)
kernel/sched/core.c +32 -0
@@ -10847,6 +10847,10 @@ static inline void mm_update_cpus_allowed(struct mm_struct *mm, const struct cpu*/ mc = &mm->mm_cid;guard (raw_spinlock)(&mc->lock); +/* Might have raced with exit() */ +if (!mm->mm_cid.users) +return ; + mm_allowed =mm_cpus_allowed (mm); weight =cpumask_weighted_or (mm_allowed, mm_allowed, affmsk);if (weight == mc->nr_cpus_allowed)
@@ -10986,6 +10990,33 @@ static bool sched_mm_cid_add_user(struct task_struct *t, struct mm_struct *mm)return mm_update_max_cids (mm); } +/* +* sched_mm_cid_exit() does not handle the mode switch for the last user +* because for normal exit() that's a pointless exercise. But if the +* process which issued execve() was multithreaded and the last exiting +* thread switched back from per CPU mode then this leaves stale state +* around. The indicator for that is mm_cid::mode having MM_CID_TRANSIT +* set. +*/ +static void sched_mm_cid_cleanup_execve (struct task_struct *t,struct mm_struct *mm) +{ +if (likely (!mm->mm_cid.mode)) +return ; + +WARN_ON_ONCE (mm->mm_cid.mode &MM_CID_ONCPU ); + mm->mm_cid.mode =0 ; + +unsigned int cpu; + +for_each_possible_cpu (cpu) +per_cpu (mm->mm_cid.pcpu->cid, cpu) =0 ; + +bitmap_zero (mm_cidmask (mm),num_possible_cpus ()); + + mm->mm_cid.nr_cpus_allowed = t->nr_cpus_allowed; +cpumask_copy (mm_cpus_allowed (mm), &t->cpus_mask); +} +/* * Make @t a user of the CIDs of its mm. Called on fork(2) and on * execve(2). A failed execve(2) rejoins the mm it left.
@@ -11015,6 +11046,7 @@ static void sched_mm_cid_fork(struct task_struct *t)/* First user ? */ if (!mm->mm_cid.users) { +sched_mm_cid_cleanup_execve (t, mm);sched_mm_cid_add_user (t, mm); t->mm_cid.cid =mm_get_cid (mm);/* Required for execve() */
Signature · 1 line
2.47.3
Mathieu Desnoyers
maintainerPATCH 4/17 sched/mmcid: Let a running joiner take its CID after the fixups
In reply to Mathieu DesnoyersThe switch to per CPU mode relies on the new user not running yet. The comment above the code says so: the new user is not visible in the system, and the task that forks it runs the fixup.
A failed execve() is missed. It rejoins its mm in sched_mm_cid_after_execve(): the new user is the running task, and it runs the fixup itself. If it is preempted before the fixup is done, mm_get_cid() can loop forever at its schedule in, with the runqueue lock held.
In the diagram below, the machine has two possible CPUs. A and B are threads of one process, which is in per task mode with two CIDs. C is the vfork() child of B. A holds mm_cid.mutex from sched_mm_cid_fork(A) on.
CPU 0 CPU 1
A: execve()
sched_mm_cid_before_execve()
A leaves, users = 1
B: vfork()
sched_mm_cid_fork(C)
users = 2
C takes the last CID
A: execve() fails
sched_mm_cid_after_execve()
sched_mm_cid_fork(A)
users = 3, switch to per CPU mode
A is active and has no CID
A is preempted
A is scheduled in
mm_cid_schedin(A)
mm_get_cid() loops:
B and C hold both CIDs
C: exit()
sched_mm_cid_exit()
waits for mm_cid.mutex, which A holds
Nothing ends the loop. Only the fixup could free a CID, and A is the task that runs it.
A guest with two CPUs hangs on this sequence when a SCHED_FIFO task wakes every 50us on CPU 0 to preempt A. CPU 0 stays in __schedule() and CPU 1 in queued_spin_lock_slowpath(). With this change the same guest completes 20000 attempts.
Fix this by keeping a running joiner inactive until the fixups are done. A schedule in gives it no CID before that. sched_mm_cid_fork() then marks it active, and it takes its CID under the runqueue lock.
--- kernel/sched/core.c | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-)
kernel/sched/core.c +13 -7
@@ -11054,7 +11054,16 @@ static void sched_mm_cid_fork(struct task_struct *t)return ; } -if (!sched_mm_cid_add_user (t, mm)) { + flip =sched_mm_cid_add_user (t, mm); +/* +* A running joiner takes no CID at a schedule in before the +* fixups are done. The users it joins can hold every CID +* until then, and the fixups are its own work. +*/ +if (t == current) + t->mm_cid.active =0 ; + +if (!flip) {if (!cid_on_cpu (mm->mm_cid.mode)) t->mm_cid.cid =mm_get_cid (mm);break ;
@@ -11064,7 +11073,6 @@ static void sched_mm_cid_fork(struct task_struct *t)* Handle the mode change and transfer current's CID. A * running joiner has no CID yet. It takes one below. */ - flip = true; percpu =cid_on_cpu (mm->mm_cid.mode);if (!percpu)mm_cid_transit_to_task (current, pcp);
@@ -11077,18 +11085,16 @@ static void sched_mm_cid_fork(struct task_struct *t)mm_cid_fixup_tasks_to_cpus (); }else {mm_cid_fixup_cpus_to_tasks (mm); -/* -* A running joiner may have taken a CID at a -* schedule in. If not, it takes one below. -*/ +/* A running joiner takes its CID below. */ if (t != current) t->mm_cid.cid =mm_get_cid (mm); } } -/* A running joiner with no CID takes one now, see above. */ +/* The running joiner takes part from here on, see above. */ if (t == current) {guard (task_rq_lock)(t); + t->mm_cid.active =1 ;if (t->mm_cid.cid ==MM_CID_UNSET )mm_cid_schedin (t); }
Signature · 1 line
2.47.3
Mathieu Desnoyers
maintainerPATCH 5/17 sched/mmcid: Take the allowed CPUs of the first user when it joins
In reply to Mathieu DesnoyersThe CIDs of an mm are sized from the CPUs its threads are allowed on. mm_init_cid() copies the allowed CPUs of the task into a new mm. Later changes reach the mm through mm_update_cpus_allowed(). It uses the mm the task points to, and ignores an mm with no user.
An affinity change between mm_init_cid() and the join of the first user is lost. In this example, P is a task allowed on one CPU, and H is another task:
P H
execve()
exec_mmap()
mm_init_cid()
copies the allowed CPUs of P
sched_setaffinity(P, all CPUs)
mm_update_cpus_allowed()
the new mm has no user:
ignored
sched_mm_cid_after_execve()
P joins the new mm, which
counts one allowed CPU
From reading the code, a fork() child has the same window between dup_mm() and sched_mm_cid_fork().
The new mm then stays sized for the CPUs allowed at that time. A process that starts on one CPU and is widened to 64 CPUs during its execve() switches to per CPU mode with its second thread. In testing, two threads of such a process read mm_cid 0 and 63. The IDs stay below the number of possible CPUs. They are no longer compact.
Fix this by adding the allowed CPUs of the first user to the mm when it joins. Hold t->pi_lock for that, as an affinity change holds it. A concurrent change is then in t->cpus_mask already, or it finds the user and updates the mm itself.
--- kernel/sched/core.c | 65 +++++++++++++++++++++++++++++++-------------- 1 file changed, 45 insertions(+), 20 deletions(-)
kernel/sched/core.c +45 -20
@@ -10833,24 +10833,17 @@ static bool mm_update_max_cids(struct mm_struct *mm)return true; } -static inline void mm_update_cpus_allowed (struct mm_struct *mm,const struct cpumask *affmsk) +static void __mm_update_cpus_allowed (struct mm_struct *mm,const struct cpumask *affmsk) { +struct mm_mm_cid *mc = &mm->mm_cid;struct cpumask *mm_allowed; -struct mm_mm_cid *mc;unsigned int weight; -if (!mm || !READ_ONCE (mm->mm_cid.users)) -return ; +lockdep_assert_held (&mc->lock);/* * mm::mm_cid::mm_cpus_allowed is the superset of each threads * allowed CPUs mask which means it can only grow. */ - mc = &mm->mm_cid; -guard (raw_spinlock)(&mc->lock); -/* Might have raced with exit() */ -if (!mm->mm_cid.users) -return ; - mm_allowed =mm_cpus_allowed (mm); weight =cpumask_weighted_or (mm_allowed, mm_allowed, affmsk);if (weight == mc->nr_cpus_allowed)
@@ -10876,6 +10869,20 @@ static inline void mm_update_cpus_allowed(struct mm_struct *mm, const struct cpuirq_work_queue (&mc->irq_work); } +static inline void mm_update_cpus_allowed (struct mm_struct *mm,const struct cpumask *affmsk) +{ +/* The first user takes the allowed CPUs of its task when it joins */ +if (!mm || !READ_ONCE (mm->mm_cid.users)) +return ; + +guard (raw_spinlock)(&mm->mm_cid.lock); +/* Might have raced with exit() */ +if (!mm->mm_cid.users) +return ; + +__mm_update_cpus_allowed (mm, affmsk); +} +static inline void mm_cid_complete_transit (struct mm_struct *mm,unsigned int mode) {/*
@@ -11017,6 +11024,28 @@ static void sched_mm_cid_cleanup_execve(struct task_struct *t, struct mm_structcpumask_copy (mm_cpus_allowed (mm), &t->cpus_mask); } +/* +* Make @t the first user of the CIDs of its mm. +* +* mm_init_cid() copied the allowed CPUs of @t before @t was a user. An +* affinity change since then did not reach the mm. Add the allowed CPUs +* of @t again. Hold t->pi_lock, which every affinity change takes. A +* concurrent change is then in t->cpus_mask already, or it finds the +* user. +*/ +static void sched_mm_cid_add_first_user (struct task_struct *t,struct mm_struct *mm) +{ +guard (raw_spinlock_irq)(&t->pi_lock); +guard (raw_spinlock)(&mm->mm_cid.lock); + +sched_mm_cid_cleanup_execve (t, mm); +sched_mm_cid_add_user (t, mm); +__mm_update_cpus_allowed (mm, &t->cpus_mask); + t->mm_cid.cid =mm_get_cid (mm); +/* Required for execve() */ +__this_cpu_write (mm->mm_cid.pcpu->cid, t->mm_cid.cid); +} +/* * Make @t a user of the CIDs of its mm. Called on fork(2) and on * execve(2). A failed execve(2) rejoins the mm it left.
@@ -11041,19 +11070,15 @@ static void sched_mm_cid_fork(struct task_struct *t)WARN_ON_ONCE (t->mm_cid.cid !=MM_CID_UNSET );guard (mutex)(&mm->mm_cid.mutex); +/* The mutex keeps other users from joining or leaving */ +if (!mm->mm_cid.users) { +sched_mm_cid_add_first_user (t, mm); +return ; + } + scoped_guard(raw_spinlock_irq, &mm->mm_cid.lock) {struct mm_cid_pcpu *pcp =this_cpu_ptr (mm->mm_cid.pcpu); -/* First user ? */ -if (!mm->mm_cid.users) { -sched_mm_cid_cleanup_execve (t, mm); -sched_mm_cid_add_user (t, mm); - t->mm_cid.cid =mm_get_cid (mm); -/* Required for execve() */ - pcp->cid = t->mm_cid.cid; -return ; - } - flip =sched_mm_cid_add_user (t, mm);/* * A running joiner takes no CID at a schedule in before the
Signature · 1 line
2.47.3
Mathieu Desnoyers
maintainerPATCH 6/17 sched/mmcid: Synchronize MM teardown with affinity updates
In reply to Mathieu DesnoyersFrom: Weiming Shi <[email protected]>
set_cpus_allowed_common() passes a remote task's p->mm to mm_update_cpus_allowed() while holding p->pi_lock, but that lock did not stabilize the pointer against MM replacement. A same-owner sched_setaffinity() caller can therefore retain the old pointer while exec replaces and frees it. On architectures which shoot down rather than refcount lazy TLB MMs, exit_mm() and kthread_unuse_mm() have the same lifetime gap.
BUG: KASAN: slab-use-after-free in set_cpus_allowed_common Read of size 4 at addr ffff888123155e4c by task poc/5132 Call Trace: set_cpus_allowed_common (kernel/sched/core.c:10835) set_cpus_allowed_fair (kernel/sched/fair.c:10046) __set_cpus_allowed_ptr_locked (kernel/sched/core.c:3224) __set_cpus_allowed_ptr (kernel/sched/core.c:3258) __sched_setaffinity (kernel/sched/syscalls.c:1161) sched_setaffinity (kernel/sched/syscalls.c:1239) __x64_sys_sched_setaffinity (kernel/sched/syscalls.c:1275) do_syscall_64 (arch/x86/entry/syscall_64.c:84) entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:121) Freed by task 5127: kmem_cache_free (mm/slub.c:6757) __mmput (kernel/fork.c:1204) mmput (kernel/fork.c:1215) setup_new_exec (fs/exec.c:1395) load_elf_binary (fs/binfmt_elf.c:1024) bprm_execve (fs/exec.c:1865) do_execveat_common.isra.0 (fs/exec.c:1963) __x64_sys_execveat (fs/exec.c:2047) do_syscall_64 (arch/x86/entry/syscall_64.c:84) entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:121) Kernel panic - not syncing: KASAN: panic_on_warn set ...
Affinity paths hold p->pi_lock across the p->mm load and its complete use. After publishing a replacement or NULL p->mm, take and release that same lock before dropping the old MM reference. For an updater that is already using the old pointer, the ordering is:
affinity updater MM teardown
---------------- -----------
raw_spin_lock(&p->pi_lock)
mm = p->mm /* old_mm */
p->mm = new_mm or NULL
raw_spin_lock(&p->pi_lock)
/* waits */
mm_update_cpus_allowed(mm, mask)
raw_spin_unlock(&p->pi_lock) /* release */
/* acquire succeeds */
raw_spin_unlock(&p->pi_lock)
...
mmput/mmdrop(old_mm)The teardown-side acquire cannot complete until an updater ordered before it has released p->pi_lock, so old_mm remains alive throughout that updater's access. If teardown is ordered first instead, its unlock is a release that orders the preceding p->mm store before a later updater's lock acquire. The later updater therefore observes new_mm or NULL. An updater that acquired the lock before the store but has not loaded p->mm yet is still ordered before the teardown-side acquisition; it may observe either pointer, but teardown waits for its complete use before dropping old_mm.
All runtime callers of set_cpus_allowed_common() hold p->pi_lock across this access. The runqueue lock is not part of the p->mm lifetime ordering, so the grace period only needs p->pi_lock. Place it after publication and before the old reference drop in exec_mmap(), exit_mm(), and kthread_unuse_mm(). The last two paths matter on architectures which shoot down rather than refcount lazy TLB MMs.
This change was developed with assistance from an LLM.
--- fs/exec.c | 1 + include/linux/sched.h | 2 ++ kernel/exit.c | 1 + kernel/kthread.c | 1 + kernel/sched/core.c | 9 +++++++++ 5 files changed, 14 insertions(+)
fs/exec.c +1 -0
@@ -905,6 +905,7 @@ static int exec_mmap(struct linux_binprm *bprm)BUG_ON (active_mm != old_mm);/* Defer teardown to setup_new_exec(), outside the exec locks. */ bprm->old_mm = old_mm; +sched_mm_cid_mm_barrier (tsk); }else {mmdrop_lazy_tlb (active_mm); }
include/linux/sched.h +2 -0
@@ -2360,6 +2360,7 @@ static __always_inline void alloc_tag_restore(struct alloc_tag *tag, struct allo#ifdef CONFIG_SCHED_MM_CID void sched_mm_cid_before_execve (struct task_struct *t);void sched_mm_cid_after_execve (struct task_struct *t); +void sched_mm_cid_mm_barrier (struct task_struct *t);void sched_mm_cid_exit (struct task_struct *t); static __always_inline int task_mm_cid(struct task_struct *t) {
@@ -2368,6 +2369,7 @@ static __always_inline int task_mm_cid(struct task_struct *t)#else static inline void sched_mm_cid_before_execve (struct task_struct *t) { }static inline void sched_mm_cid_after_execve (struct task_struct *t) { } +static inline void sched_mm_cid_mm_barrier (struct task_struct *t) { }static inline void sched_mm_cid_exit (struct task_struct *t) { } static __always_inline int task_mm_cid(struct task_struct *t) {
kernel/exit.c +1 -0
@@ -582,6 +582,7 @@ static void exit_mm(void)local_irq_enable ();task_unlock (current);mmap_read_unlock (mm); +sched_mm_cid_mm_barrier (current);mm_update_next_owner (mm);mmput (mm);if (test_thread_flag (TIF_MEMDIE ))
kernel/kthread.c +1 -0
@@ -1682,6 +1682,7 @@ void kthread_unuse_mm(struct mm_struct *mm)enter_lazy_tlb (mm, tsk);local_irq_enable ();task_unlock (tsk); +sched_mm_cid_mm_barrier (tsk);mmdrop (mm); }
kernel/sched/core.c +9 -0
@@ -11229,6 +11229,15 @@ void sched_mm_cid_after_execve(struct task_struct *t)sched_mm_cid_fork (t); } +/* +* Pair with set_cpus_allowed_common()'s p->pi_lock critical section. Call +* after publishing p->mm and before dropping the old MM reference. +*/ +void sched_mm_cid_mm_barrier (struct task_struct *t) +{ +guard (raw_spinlock_irq)(&t->pi_lock); +} +static void mm_cid_work_fn (struct work_struct *work) { struct mm_struct *mm = container_of(work, struct mm_struct, mm_cid.work);
Signature · 1 line
2.47.3
Mathieu Desnoyers
maintainerPATCH 7/17 selftests/rseq: Test the concurrency ID after a failed execve
In reply to Mathieu DesnoyersA failed execve() leaves its mm and rejoins it while the task is running. If the other threads are still alive, the mm can be in per CPU mode. There a new user gets no CID at the join, so the task would return to user space with MM_CID_UNSET.
Add mm_cid_execve_test. A process with more threads than CIDs calls execve() on a file that no binfmt handler recognises. That fails with ENOEXEC before the other threads are killed. The process reads its mm_cid after each call. The value must stay below the number of CPUs the process is allowed on.
Per CPU mode needs more threads than the mm has CIDs. The number of CIDs follows the union of the allowed CPUs of the threads, which only grows. The test therefore narrows its affinity before the fork() that creates the process under test. Five threads on two CPUs are then enough on any machine.
Without "sched/mmcid: Give a CID to a task that joins its mm while running", on a kernel with 8 possible CPUs:
mm_cid 2147483648 (0x80000000) after a failed execve, round 0 of 100 not ok 1 mm_cid after a failed execve
With it:
5 threads, 100 failed execve, largest mm_cid 0, below 2 ok 1 mm_cid after a failed execve
--- tools/testing/selftests/rseq/.gitignore | 1 + tools/testing/selftests/rseq/Makefile | 3 +- .../selftests/rseq/mm_cid_execve_test.c | 276 ++++++++++++++++++ 3 files changed, 279 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/rseq/mm_cid_execve_test.c
tools/testing/selftests/rseq/.gitignore +1 -0
@@ -9,6 +9,7 @@ param_test_compare_twice param_test_mm_cid param_test_mm_cid_benchmark param_test_mm_cid_compare_twice +mm_cid_execve_test syscall_errors_test slice_test legacy_check
tools/testing/selftests/rseq/Makefile +2 -1
@@ -19,7 +19,8 @@ LDLIBS += -lpthread -ldl OVERRIDE_TARGETS = 1 TEST_GEN_PROGS = basic_test basic_percpu_ops_test basic_percpu_ops_mm_cid_test \ - param_test_benchmark param_test_mm_cid_benchmark + param_test_benchmark param_test_mm_cid_benchmark \ + mm_cid_execve_test TEST_GEN_PROGS_EXTENDED = librseq.so \ param_test \
tools/testing/selftests/rseq/mm_cid_execve_test.c new file +276 -0
276 lines not on this page · raw message
Signature · 1 line
2.47.3
Mathieu Desnoyers
maintainerPATCH 8/17 selftests/rseq: Test the concurrency ID after the last user fails execve
In reply to Mathieu DesnoyersA failed execve() leaves the mm and rejoins it. When the caller is the only thread left, it leaves as the last user and rejoins as the first one. The kernel must not keep state of the previous users across that.
Add a case to mm_cid_execve_test. Four threads make the mm switch to per CPU mode on two CPUs, then exit. The remaining thread fails execve() in a loop on one CPU while a helper process moves it to the other CPU. It then visits both CPUs and must read mm_cid 0 on each.
The move has to land inside an execve(), so the case runs 50 times with a different delay each time.
With two allowed CPUs, the kernel keeps per CPU mode down to one thread only on a machine with two or three possible CPUs. Elsewhere the single thread reads mm_cid 0 on both CPUs before the execve(), and the case is skipped.
Without "sched/mmcid: Reset stale state when the first user joins an mm", on guests with two and three CPUs (six runs out of six):
single thread, mm_cid 1 on CPU 0 after a failed execve moved from CPU 0 to CPU 1 not ok 2 mm_cid of a single thread after a failed execve on another CPU
With it (six runs out of six):
ok 2 mm_cid of a single thread after a failed execve on another CPU
--- .../selftests/rseq/mm_cid_execve_test.c | 236 +++++++++++++++--- 1 file changed, 208 insertions(+), 28 deletions(-)
tools/testing/selftests/rseq/mm_cid_execve_test.c +208 -28
@@ -8,6 +8,15 @@* running and the other threads are still users. The task returns to user * space before any schedule in, so the join has to give it a CID. * +* A second case covers the last user. A process that is back to a single +* thread fails execve(2) while another process moves it to another CPU. +* It leaves its mm as the last user on one CPU and rejoins it as the first +* user on the other. A single thread must read 0 afterwards. That case +* needs per CPU mode to last down to one thread. With the two allowed CPUs +* of the test, the kernel only does that with at most three possible +* CPUs. The case is skipped when a single thread reads 0 on both CPUs +* before the execve(2). +* * Per CPU mode needs more threads than the mm has CIDs. The number of CIDs * follows the union of the allowed CPUs of the threads, which only grows. * The affinity is therefore narrowed before the fork(2) that creates the
@@ -19,6 +28,7 @@#include <limits.h> #include <pthread.h> #include <sched.h> +#include <signal.h> #include <stdbool.h> #include <stdio.h> #include <stdlib.h>
@@ -33,6 +43,8 @@#define NARROW_CPUS 2#define THREADS 4#define ROUNDS 100 +#define ATTEMPTS 50 +#define MOVE_EXECS 300/* Exit status of the process under test. */ #define CHILD_PASS 0
@@ -204,6 +216,147 @@ static int run_child(void)return ret; } +/* Run on @cpu and return the mm_cid read there. */ +static unsigned int cid_on_cpu (int cpu) +{ +cpu_set_t one; + +CPU_ZERO (&one); +CPU_SET (cpu, &one); +if (sched_setaffinity (0 ,sizeof (one), &one)) +return UINT_MAX ; +return rseq_current_mm_cid (); +} + +/* +* The second process under test. Its threads make the mm switch to per +* CPU mode and exit. The last thread then fails execve(2) in a loop on the +* CPU @from. A helper process moves it to the CPU @to, @delay_us after the +* start of the loop. The thread then visits both CPUs and must read 0 on +* each. +* Return a CHILD_* status. +*/ +static int run_child_moved (int delay_us) +{ +pthread_t thread[THREADS ]; +char path[PATH_MAX ]; +int cpus[NARROW_CPUS ], nr =0 , cpu, i, started =0 , ret =CHILD_PASS ; +int from, to, go[2 ], status; +unsigned int cid0, cid1; +cpu_set_t mask, one; +pid_t helper, self =getpid (); +char c =0 ; + +if (sched_getaffinity (0 ,sizeof (mask), &mask)) +return CHILD_ERROR ; +for (cpu =0 ; cpu <CPU_SETSIZE && nr <NARROW_CPUS ; cpu++) { +if (CPU_ISSET (cpu, &mask)) + cpus[nr++] = cpu; + } +if (nr <NARROW_CPUS ) { +fprintf (stderr,"fewer than %d allowed CPUs\n" ,NARROW_CPUS ); +return CHILD_SKIP ; + } +if (make_unknown_format_file (path,sizeof (path))) { +fprintf (stderr,"no directory to run an unknown format file from\n" ); +return CHILD_SKIP ; + } + +for (i =0 ; i <THREADS ; i++) { +if (pthread_create (&thread[i],NULL , waiter,NULL )) +break ; + started++; + } +/* Run on both CPUs while the threads exist: each CPU takes a CID. */ +for (i =0 ; i <2 *NARROW_CPUS ; i++) +cid_on_cpu (cpus[i %NARROW_CPUS ]); +RSEQ_WRITE_ONCE (stop,1 ); +for (i =0 ; i < started; i++) { +if (join_thread (thread[i])) + ret =CHILD_ERROR ; + } +if (ret !=CHILD_PASS || started <THREADS ) { + ret =CHILD_ERROR ; + goto out; + } + + cid0 =cid_on_cpu (cpus[0 ]); + cid1 =cid_on_cpu (cpus[1 ]); +if (cid0 ==UINT_MAX || cid1 ==UINT_MAX ) { + ret =CHILD_ERROR ; + goto out; + } +/* The CPUs own the CIDs only if one of them reads above 0. */ +if (!cid0 && !cid1) { +fprintf (stderr,"the mm is not in per CPU mode with a single thread\n" ); + ret =CHILD_SKIP ; + goto out; + } +/* Leave from the CPU that does not hold CID 0, rejoin on the other. */ + from = cid0 ? cpus[0 ] : cpus[1 ]; + to = from == cpus[0 ] ? cpus[1 ] : cpus[0 ]; +cid_on_cpu (from); + +if (pipe (go)) { + ret =CHILD_ERROR ; + goto out; + } + helper =fork (); +if (helper <0 ) { +close (go[0 ]); +close (go[1 ]); + ret =CHILD_ERROR ; + goto out; + } +if (!helper) { +/* Run beside the process under test, not on its CPU. */ +CPU_ZERO (&one); +CPU_SET (to, &one); +if (sched_setaffinity (0 ,sizeof (one), &one) ||read (go[0 ], &c,1 ) !=1 ) +_exit (1 ); +usleep (delay_us); +_exit (!!sched_setaffinity (self,sizeof (one), &one)); + } +if (write (go[1 ], &c,1 ) !=1 ) { +/* The helper waits for that byte. */ +kill (helper,SIGKILL ); + ret =CHILD_ERROR ; + } +for (i =0 ; i <MOVE_EXECS ; i++) { +char *const av[] = { path,NULL }; + +if (execve (path, av,NULL ) ==-1 && errno !=ENOEXEC ) { +fprintf (stderr,"execve(%s): %s, expected ENOEXEC\n" , + path,strerror (errno)); + ret =CHILD_ERROR ; +break ; + } + } +if (waitpid (helper, &status,0 ) != helper || !WIFEXITED (status) || +WEXITSTATUS (status)) + ret =CHILD_ERROR ; +close (go[0 ]); +close (go[1 ]); + +/* The first visit lets the kernel settle, the second one is judged. */ +for (i =0 ; i <2 *NARROW_CPUS && ret ==CHILD_PASS ; i++) { +unsigned int cid =cid_on_cpu (cpus[i %NARROW_CPUS ]); + +if (cid ==UINT_MAX ) { + ret =CHILD_ERROR ; + }else if (i >=NARROW_CPUS && cid) { +fprintf (stderr, +"single thread, mm_cid %u on CPU %d after a failed execve moved from CPU %d to CPU %d\n" , + cid, cpus[i %NARROW_CPUS ], from, to); + ret =CHILD_FAIL ; + } + } +out: +if (remove_file (path) && ret ==CHILD_PASS ) + ret =CHILD_ERROR ; +return ret; +} +/* Keep the first @nr CPUs of @src in @dst. */ static int narrow_cpus (cpu_set_t *dst,const cpu_set_t *src,int nr) {
@@ -219,58 +372,85 @@ static int narrow_cpus(cpu_set_t *dst, const cpu_set_t *src, int nr)return taken; } -int main (void ) +/* +* Run @fn(@arg) in a new process and return its exit status. The affinity +* is @narrow when its mm is created. The union of allowed CPUs of an mm +* starts there and only grows. +*/ +static int run_under_test (int (*fn)(int ),int arg,const cpu_set_t *narrow, +const cpu_set_t *allowed) { -cpu_set_t allowed, narrow;int status =0 ;pid_t child; -ksft_print_header (); -ksft_set_plan (1 ); - -if (!rseq_available ()) -ksft_exit_skip ("rseq is unavailable\n" ); -/* glibc before 2.35 registers no rseq area. */ -if (rseq_register_current_thread ()) -ksft_exit_fail_msg ("rseq_register_current_thread: %s\n" ,strerror (errno)); -if (!rseq_mm_cid_available ()) -ksft_exit_skip ("the rseq mm_cid field is unavailable\n" ); -if (sched_getaffinity (0 ,sizeof (allowed), &allowed)) -ksft_exit_fail_msg ("sched_getaffinity: %s\n" ,strerror (errno)); -if (!narrow_cpus (&narrow, &allowed,NARROW_CPUS )) -ksft_exit_skip ("no CPU to run on\n" ); - -/* -* Narrow before the fork(2). The mm of the child starts with these -* allowed CPUs, and a later narrowing does not shrink them. -*/ -if (sched_setaffinity (0 ,sizeof (narrow), &narrow)) +if (sched_setaffinity (0 ,sizeof (*narrow), narrow))ksft_exit_skip ("sched_setaffinity: %s\n" ,strerror (errno)); child =fork ();if (!child) -_exit (run_child ()); +_exit (fn (arg));if (child <0 )ksft_exit_fail_msg ("fork: %s\n" ,strerror (errno)); -if (sched_setaffinity (0 ,sizeof (allowed), &allowed)) +if (sched_setaffinity (0 ,sizeof (*allowed), allowed))ksft_exit_fail_msg ("sched_setaffinity: %s\n" ,strerror (errno));if (waitpid (child, &status,0 ) != child)ksft_exit_fail_msg ("waitpid: %s\n" ,strerror (errno));if (!WIFEXITED (status))ksft_exit_fail_msg ("the process under test was killed by signal %d\n" ,WTERMSIG (status)); +return WEXITSTATUS (status); +} -switch (WEXITSTATUS (status)) { +static void report (int status,const char *name) +{ +switch (status) {case CHILD_PASS : -ksft_test_result_pass ("mm_cid after a failed execve\n" ); +ksft_test_result_pass ("%s\n" , name);break ;case CHILD_SKIP : -ksft_test_result_skip ("mm_cid after a failed execve\n" ); +ksft_test_result_skip ("%s\n" , name);break ;case CHILD_FAIL : -ksft_test_result_fail ("mm_cid after a failed execve\n" ); +ksft_test_result_fail ("%s\n" , name);break ;default :ksft_exit_fail_msg ("the process under test failed to run\n" ); } +} + +static int run_child_threads (int unused) +{ +return run_child (); +} + +int main (void ) +{ +cpu_set_t allowed, narrow; +int status, i; + +ksft_print_header (); +ksft_set_plan (2 ); + +if (!rseq_available ()) +ksft_exit_skip ("rseq is unavailable\n" ); +/* glibc before 2.35 registers no rseq area. */ +if (rseq_register_current_thread ()) +ksft_exit_fail_msg ("rseq_register_current_thread: %s\n" ,strerror (errno)); +if (!rseq_mm_cid_available ()) +ksft_exit_skip ("the rseq mm_cid field is unavailable\n" ); +if (sched_getaffinity (0 ,sizeof (allowed), &allowed)) +ksft_exit_fail_msg ("sched_getaffinity: %s\n" ,strerror (errno)); +if (!narrow_cpus (&narrow, &allowed,NARROW_CPUS )) +ksft_exit_skip ("no CPU to run on\n" ); + + status =run_under_test (run_child_threads,0 , &narrow, &allowed); +report (status,"mm_cid after a failed execve" ); + +/* The helper's move has to land inside an execve: spread its delay. */ +for (i =0 ; i <ATTEMPTS ; i++) { + status =run_under_test (run_child_moved, (i *37 ) %1000 , &narrow, &allowed); +if (status !=CHILD_PASS ) +break ; + } +report (status,"mm_cid of a single thread after a failed execve on another CPU" );ksft_finished (); }
Signature · 1 line
2.47.3
Mathieu Desnoyers
maintainerPATCH 9/17 selftests/rseq: Test the concurrency IDs across ownership mode switches
In reply to Mathieu DesnoyersEither the tasks or the CPUs own the concurrency IDs of an mm. The kernel switches between both when the thread count crosses a threshold. It also does when an affinity change widens the allowed CPUs. No test changes the thread count while the IDs are in use.
Add mm_cid_transition_test. Each round creates 1 to 16 threads and joins them, so the thread count crosses both thresholds. A round has three kinds of threads:
- Most threads count in an array indexed by their mm_cid, in an rseq critical section with no lock. Two threads that run with the same mm_cid at the same time lose counts.
- One thread fails execve() in a loop, so it leaves the mm and rejoins it.
- One thread calls vfork() in a loop. Its child is a user of the mm, and can join while the thread above is out.
A last case starts on two CPUs and widens the affinity of a thread while three others count.
The test fails when counts are lost, or when an mm_cid is not below the number of possible CPUs.
It passes with this series on guests with 2 to 64 CPUs. On a v7.1.8 kernel with 384 CPUs, it reports the ID a failed execve() gets in per CPU mode:
thread count changes: mm_cid 2147483648 (0x80000000), 384 possible CPUs not ok 1 thread count changes on at most 4 CPUs
--- tools/testing/selftests/rseq/.gitignore | 1 + tools/testing/selftests/rseq/Makefile | 2 +- .../selftests/rseq/mm_cid_transition_test.c | 375 ++++++++++++++++++ 3 files changed, 377 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/rseq/mm_cid_transition_test.c
tools/testing/selftests/rseq/.gitignore +1 -0
@@ -10,6 +10,7 @@ param_test_mm_cid param_test_mm_cid_benchmark param_test_mm_cid_compare_twice mm_cid_execve_test +mm_cid_transition_test syscall_errors_test slice_test legacy_check
tools/testing/selftests/rseq/Makefile +1 -1
@@ -20,7 +20,7 @@ OVERRIDE_TARGETS = 1 TEST_GEN_PROGS = basic_test basic_percpu_ops_test basic_percpu_ops_mm_cid_test \ param_test_benchmark param_test_mm_cid_benchmark \ - mm_cid_execve_test + mm_cid_execve_test mm_cid_transition_test TEST_GEN_PROGS_EXTENDED = librseq.so \ param_test \
tools/testing/selftests/rseq/mm_cid_transition_test.c new file +375 -0
375 lines not on this page · raw message
Signature · 1 line
2.47.3
Mathieu Desnoyers
maintainerPATCH 10/17 selftests/rseq: Test the upper bound of the concurrency IDs
In reply to Mathieu DesnoyersNo test checks how large an mm_cid gets when the thread count of a process changes.
The upper bound of an mm_cid is the number of possible CPUs. The kernel also aims at compact IDs. Its target is max_cids:
opt_cids = min(threads, allowed CPUs) max_cids = min(opt_cids + opt_cids / 4, possible CPUs)
The slack above opt_cids gives the switch between per task and per CPU mode some hysteresis. Threads that come and go near the limit then do not cause a costly mode switch each time.
Add mm_cid_bound_test. A process grows from one thread to twice its CPUs plus two, then shrinks back to one. After each step every thread visits every allowed CPU and reads its mm_cid there. The largest value must stay below the number of possible CPUs.
The test also prints the steps where the largest value is above max_cids, or above min(threads, allowed CPUs). These steps do not make it fail.
The test runs on 2 CPUs, on 4 CPUs and on all the allowed CPUs. It passes on every guest tried, from 2 to 64 CPUs.
The printed steps show how far the IDs go above the target. In per CPU mode each CPU keeps the ID it owns. The mode is left only when the thread count drops below mm_cid_calc_pcpu_thrs(): three quarters of the allowed CPUs, capped at half the possible ones. On a 64-CPU guest, on the way down:
35 threads on 64 CPUs: largest mm_cid 63 on CPU 62, max_cids 43
In per task mode the allowance of max_cids shows:
20 threads on 64 CPUs: largest mm_cid 24 on CPU 63, max_cids 25
--- tools/testing/selftests/rseq/.gitignore | 1 + tools/testing/selftests/rseq/Makefile | 3 +- .../selftests/rseq/mm_cid_bound_test.c | 355 ++++++++++++++++++ 3 files changed, 358 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/rseq/mm_cid_bound_test.c
tools/testing/selftests/rseq/.gitignore +1 -0
@@ -9,6 +9,7 @@ param_test_compare_twice param_test_mm_cid param_test_mm_cid_benchmark param_test_mm_cid_compare_twice +mm_cid_bound_test mm_cid_execve_test mm_cid_transition_test syscall_errors_test
tools/testing/selftests/rseq/Makefile +2 -1
@@ -20,7 +20,8 @@ OVERRIDE_TARGETS = 1 TEST_GEN_PROGS = basic_test basic_percpu_ops_test basic_percpu_ops_mm_cid_test \ param_test_benchmark param_test_mm_cid_benchmark \ - mm_cid_execve_test mm_cid_transition_test + mm_cid_execve_test mm_cid_transition_test \ + mm_cid_bound_test TEST_GEN_PROGS_EXTENDED = librseq.so \ param_test \
tools/testing/selftests/rseq/mm_cid_bound_test.c new file +355 -0
355 lines not on this page · raw message
Signature · 1 line
2.47.3
Mathieu Desnoyers
maintainerPATCH 11/17 selftests/rseq: Test affinity changes made by another process
In reply to Mathieu DesnoyersThe mm_cid tests change the affinity of a thread from inside its own process. Another process can change it too, with sched_setaffinity() on a thread ID. The kernel then updates the allowed CPUs of an mm that is not the mm of the caller. The threads of that mm can exit or call vfork() at the same time, and the whole process can exit. No test covers this.
Add a case to mm_cid_transition_test. A process starts on two CPUs with three counting threads. Another process narrows and widens the affinity of each of these threads in a loop. It goes on while the process under test exits. The case runs 100 times, and the changes start after a different delay each time.
The case passes on guests with 2 to 64 CPUs. It adds coverage: it found no bug.
--- .../selftests/rseq/mm_cid_transition_test.c | 69 ++++++++++++++++++- 1 file changed, 68 insertions(+), 1 deletion(-)
tools/testing/selftests/rseq/mm_cid_transition_test.c +68 -1
@@ -20,21 +20,28 @@* A second case starts on two CPUs and widens the affinity of a thread * while the others count. * +* A third case leaves the affinity changes to another process. It narrows +* and widens every thread in a loop, while the threads count, call +* vfork(2) and then exit with their process. +* * The test fails when counts are lost, or when an mm_cid is not below the * number of possible CPUs. A kernel that loops looking for a free ID * shows as a timeout. */ #define _GNU_SOURCE +#include <dirent.h> #include <errno.h> #include <limits.h> #include <pthread.h> #include <sched.h> +#include <signal.h> #include <stdbool.h> #include <stdint.h> #include <stdio.h> #include <stdlib.h> #include <string.h> +#include <sys/prctl.h> #include <sys/stat.h> #include <sys/wait.h> #include <unistd.h>
@@ -266,6 +273,59 @@ static int run_child_widen(int delay_us)return check_counts ("affinity change" ); } +/* +* Narrow and widen the affinity of every thread of @pid in a loop, from +* another process. Start after @delay_us and run until @pid exits. +*/ +static void remote_setter (pid_t pid,int delay_us,const cpu_set_t *narrow) +{ +bool widen = true; +char path[64 ]; + +/* Die with the process under test, not before its exit. */ +prctl (PR_SET_PDEATHSIG ,SIGKILL ); +snprintf (path,sizeof (path),"/proc/%d/task" , pid); +usleep (delay_us); +while (getppid () == pid) { +DIR *dir =opendir (path); +struct dirent *de; + +if (!dir) +break ; +while ((de =readdir (dir))) { +pid_t tid =atoi (de->d_name); + +/* A thread can exit first: ESRCH is expected. */ +if (tid >0 ) +sched_setaffinity (tid,sizeof (all_cpus), + widen ? &all_cpus : narrow); + } +closedir (dir); + widen = !widen; + } +_exit (0 ); +} + +/* Run one round while another process changes the affinity of the threads. */ +static int run_child_remote (int delay_us) +{ +pid_t self =getpid (), setter; +cpu_set_t narrow; + +if (sched_getaffinity (0 ,sizeof (narrow), &narrow)) +return CHILD_ERROR ; +init_workers (); + setter =fork (); +if (setter <0 ) +return CHILD_ERROR ; +if (!setter) +remote_setter (self, delay_us, &narrow); +if (run_round (3 ,-1 )) +return CHILD_ERROR ; +/* The setter keeps running while this process exits. */ +return check_counts ("affinity change by another process" ); +} +static int read_nr_possible (void ) {int first, last, total =0 ;
@@ -343,7 +403,7 @@ int main(void)int status, i;ksft_print_header (); -ksft_set_plan (3 ); +ksft_set_plan (4 );if (!rseq_available ())ksft_exit_skip ("rseq is unavailable\n" );
@@ -371,5 +431,12 @@ int main(void)break ; }ksft_test_result (status ==CHILD_PASS ,"affinity change widens the allowed CPUs\n" ); + +for (i =0 ; i <WIDEN_RUNS ; i++) { + status =run_under_test (run_child_remote, (i *13 ) %ROUND_US , &narrow); +if (status !=CHILD_PASS ) +break ; + } +ksft_test_result (status ==CHILD_PASS ,"affinity changed by another process\n" );ksft_finished (); }
Signature · 1 line
2.47.3
Mathieu Desnoyers
maintainerPATCH 12/17 selftests/rseq: Test an affinity change by another process during execve
In reply to Mathieu DesnoyersThe kernel sizes the concurrency IDs of an mm from the CPUs its threads are allowed on. mm_init_cid() copies the allowed CPUs of the task into a new mm. Later changes reach the mm through mm_update_cpus_allowed().
execve() calls mm_init_cid() in exec_mmap(), when the task switches to its new mm. The task joins that mm later, in sched_mm_cid_after_execve(). Another task can change the affinity in between. No test does.
Add mm_cid_exec_affinity_test. A process starts on one CPU and calls execve() on the test. Its parent widens its affinity to all CPUs after a delay. The delay differs on each of the 60 attempts, so that some changes land inside the execve(). The new program waits until it is allowed on all CPUs. It starts one spinning thread per CPU, lets them spread, then stops all but one. With two threads left, both must read an mm_cid below 2.
A first case runs the same program with all CPUs allowed from the start. It passes.
The second case needs "sched/mmcid: Take the allowed CPUs of the first user when it joins". Without it, the case fails on guests with 4 to 64 CPUs (eight runs out of eight). On the 64-CPU guest:
mm_cid 0 and 63 with 2 threads, largest mm_cid 63 with 64 threads
# attempt 5, affinity widened 185 us after the fork not ok 2 affinity widened by another process during execve
With it, the case passes on the same guests (ten runs out of ten).
--- tools/testing/selftests/rseq/.gitignore | 1 + tools/testing/selftests/rseq/Makefile | 2 +- .../rseq/mm_cid_exec_affinity_test.c | 283 ++++++++++++++++++ 3 files changed, 285 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/rseq/mm_cid_exec_affinity_test.c
tools/testing/selftests/rseq/.gitignore +1 -0
@@ -12,6 +12,7 @@ param_test_mm_cid_compare_twice mm_cid_bound_test mm_cid_execve_test mm_cid_transition_test +mm_cid_exec_affinity_test syscall_errors_test slice_test legacy_check
tools/testing/selftests/rseq/Makefile +1 -1
@@ -21,7 +21,7 @@ OVERRIDE_TARGETS = 1 TEST_GEN_PROGS = basic_test basic_percpu_ops_test basic_percpu_ops_mm_cid_test \ param_test_benchmark param_test_mm_cid_benchmark \ mm_cid_execve_test mm_cid_transition_test \ - mm_cid_bound_test + mm_cid_bound_test mm_cid_exec_affinity_test TEST_GEN_PROGS_EXTENDED = librseq.so \ param_test \
tools/testing/selftests/rseq/mm_cid_exec_affinity_test.c new file +283 -0
283 lines not on this page · raw message
Signature · 1 line
2.47.3
Mathieu Desnoyers
maintainerPATCH 13/17 selftests/rseq: Test concurrent thread creation and forced migration
In reply to Mathieu DesnoyersTwo fixes that followed the rework of the concurrency IDs point at cases no test covers:
- commit b2e48c429ec5 ("sched/mmcid: Prevent CID stalls due to concurrent forks") fixed two tasks that join an mm at the same time;
- commit 4327fb13fa47 ("sched/mmcid: Prevent live lock on task to CPU mode transition") fixed a task that migrates while the mm switches to per CPU mode.
In mm_cid_transition_test only the main thread creates threads, and no thread is moved to another CPU while the thread count changes.
Add two cases. Both repeat the rounds of 1 to 16 threads on at most four CPUs:
- In the first one, most threads create a thread of their own when they start. Several threads then join the mm at the same time.
- In the second one, another process pins every thread to one CPU after another in a loop. The threads migrate while the mode switches in both directions.
Both cases pass on guests with 2 to 64 CPUs, with and without the previous fixes.
--- .../selftests/rseq/mm_cid_transition_test.c | 173 ++++++++++++------ 1 file changed, 121 insertions(+), 52 deletions(-)
tools/testing/selftests/rseq/mm_cid_transition_test.c +121 -52
@@ -24,6 +24,11 @@* and widens every thread in a loop, while the threads count, call * vfork(2) and then exit with their process. * +* Two more cases repeat the rounds. In one, most threads create a thread +* of their own, so that several threads join the mm at the same time. In +* the other, another process moves every thread from CPU to CPU, so that +* threads migrate while the mode switches. +* * The test fails when counts are lost, or when an mm_cid is not below the * number of possible CPUs. A kernel that loops looking for a free ID * shows as a timeout.
@@ -55,6 +60,7 @@#define WIDEN_RUNS 100#define ROUND_US 500#define MAX_TRIES 1000000 +#define MAX_SETS 4#define CHILD_PASS 0#define CHILD_FAIL 1
@@ -66,10 +72,16 @@ struct worker {unsigned int bad_cid;bool execs;bool vforks; +struct worker *buddy; }; +/* run_child_rounds() flags */ +#define NESTED 1 +#define MIGRATED 2 +static intptr_t counts[MAX_CIDS ]; -static struct worker workers[MAX_WORKERS ]; +static struct worker workers[2 *MAX_WORKERS ]; +static bool nested;static int stop;static int nr_possible;static cpu_set_t all_cpus;
@@ -115,14 +127,34 @@ static int remove_file(const char *path)return 0 ; } +static void *worker_fn (void *arg); + +/* Start the thread of @w. Return 0 or an error number. */ +static int create_worker (struct worker *w) +{ +int err, tries =0 ; + +/* clone(CLONE_FS) fails with EAGAIN while a thread is in execve. */ +do { + err =pthread_create (&w->thread,NULL , worker_fn, w); +if (err ==EAGAIN ) +sched_yield (); + }while (err ==EAGAIN && ++tries <MAX_TRIES ); +return err; +} +static void *worker_fn (void *arg) {struct worker *w = arg;char *const av[] = { exec_path,NULL }; +bool has_buddy = false;int i;if (rseq_register_current_thread ())abort (); +/* Other workers do the same now: several threads join at once. */ +if (w->buddy) + has_buddy = !create_worker (w->buddy);while (!RSEQ_READ_ONCE (stop) && w->bad_cid ==UINT_MAX ) {for (i =0 ; i <100 ; i++)count_one (w);
@@ -141,6 +173,8 @@ static void *worker_fn(void *arg) } } } +if (has_buddy) +join_thread (w->buddy->thread);rseq_unregister_current_thread ();return NULL ; }
@@ -152,17 +186,11 @@ static int run_round(int nr, int widen_us)RSEQ_WRITE_ONCE (stop,0 );for (i =0 ; i < nr; i++) { -int err, tries =0 ; - workers[i].execs = i ==0 && exec_path[0 ]; workers[i].vforks = i ==1 ; -/* clone(CLONE_FS) fails with EAGAIN while a thread is in execve. */ -do { - err =pthread_create (&workers[i].thread,NULL , worker_fn, &workers[i]); -if (err ==EAGAIN ) -sched_yield (); - }while (err ==EAGAIN && ++tries <MAX_TRIES ); -if (err) +/* The two workers above keep to their own loop. */ + workers[i].buddy = nested && i >1 ? &workers[MAX_WORKERS + i] :NULL ; +if (create_worker (&workers[i]))break ; started++; }
@@ -187,7 +215,7 @@ static int check_counts(const char *what)intptr_t counted =0 , total =0 ;int i, ret =CHILD_PASS ; -for (i =0 ; i <MAX_WORKERS ; i++) { +for (i =0 ; i <2 *MAX_WORKERS ; i++) { counted += workers[i].counted;if (workers[i].bad_cid !=UINT_MAX ) { fprintf(stderr, "%s: mm_cid %u (0x%x), %d possible CPUs\n",
@@ -209,7 +237,7 @@ static void init_workers(void) {int i; -for (i =0 ; i <MAX_WORKERS ; i++) +for (i =0 ; i <2 *MAX_WORKERS ; i++) workers[i].bad_cid =UINT_MAX ; }
@@ -242,12 +270,76 @@ static int make_unknown_format_file(const char *dir)return -1 ; } -/* Rounds of 1 to MAX_WORKERS threads: the count crosses both thresholds. */ -static int run_child_rounds (int unused) +/* +* Change the affinity of every thread of @pid in a loop, from another +* process. Each change takes the next of the @nr_sets CPU sets in @sets. +* Start after @delay_us and run until @pid exits. +*/ +static void remote_setter (pid_t pid,int delay_us,const cpu_set_t *sets,int nr_sets) { -int round, ret; +unsigned int next =0 ; +char path[64 ]; + +/* Die with the process under test, not before its exit. */ +if (prctl (PR_SET_PDEATHSIG ,SIGKILL )) { +fprintf (stderr,"prctl(PR_SET_PDEATHSIG): %s\n" ,strerror (errno)); +_exit (1 ); + } +snprintf (path,sizeof (path),"/proc/%d/task" , pid); +usleep (delay_us); +while (getppid () == pid) { +DIR *dir =opendir (path); +struct dirent *de; + +if (!dir) +break ; +while ((de =readdir (dir))) { +pid_t tid =atoi (de->d_name); + +/* A thread can exit first: ESRCH is expected. */ +if (tid >0 && +sched_setaffinity (tid,sizeof (sets[0 ]), + &sets[next++ % nr_sets]) && + errno !=ESRCH ) { +fprintf (stderr,"sched_setaffinity: %s\n" ,strerror (errno)); +_exit (1 ); + } + } +closedir (dir); + } +_exit (0 ); +} + +/* +* Rounds of 1 to MAX_WORKERS threads: the count crosses both thresholds. +* With NESTED in @flags, most threads create one more thread. With +* MIGRATED, another process moves the threads between the allowed CPUs. +*/ +static int run_child_rounds (int flags) +{ +static cpu_set_t sets[MAX_SETS ]; +int round, ret, cpu, nr_sets =0 ; +pid_t self =getpid (), setter; +cpu_set_t allowed;init_workers (); + nested = flags &NESTED ; +if (flags &MIGRATED ) { +if (sched_getaffinity (0 ,sizeof (allowed), &allowed)) +return CHILD_ERROR ; +/* One set per allowed CPU: the allowed CPUs of the mm stay. */ +for (cpu =0 ; cpu <CPU_SETSIZE && nr_sets <MAX_SETS ; cpu++) { +if (!CPU_ISSET (cpu, &allowed)) +continue ; +CPU_ZERO (&sets[nr_sets]); +CPU_SET (cpu, &sets[nr_sets++]); + } + setter =fork (); +if (setter <0 ) +return CHILD_ERROR ; +if (!setter) +remote_setter (self,0 , sets, nr_sets); + }/* Without the file the rounds run with no execve. */ if (make_unknown_format_file ("/tmp" ) &&make_unknown_format_file ("." ))fprintf (stderr,"no unknown format file: the rounds run without execve\n" );
@@ -273,53 +365,23 @@ static int run_child_widen(int delay_us)return check_counts ("affinity change" ); } -/* -* Narrow and widen the affinity of every thread of @pid in a loop, from -* another process. Start after @delay_us and run until @pid exits. -*/ -static void remote_setter (pid_t pid,int delay_us,const cpu_set_t *narrow) -{ -bool widen = true; -char path[64 ]; - -/* Die with the process under test, not before its exit. */ -prctl (PR_SET_PDEATHSIG ,SIGKILL ); -snprintf (path,sizeof (path),"/proc/%d/task" , pid); -usleep (delay_us); -while (getppid () == pid) { -DIR *dir =opendir (path); -struct dirent *de; - -if (!dir) -break ; -while ((de =readdir (dir))) { -pid_t tid =atoi (de->d_name); - -/* A thread can exit first: ESRCH is expected. */ -if (tid >0 ) -sched_setaffinity (tid,sizeof (all_cpus), - widen ? &all_cpus : narrow); - } -closedir (dir); - widen = !widen; - } -_exit (0 ); -} -/* Run one round while another process changes the affinity of the threads. */ static int run_child_remote (int delay_us) {pid_t self =getpid (), setter; -cpu_set_t narrow; +cpu_set_t sets[3 ]; -if (sched_getaffinity (0 ,sizeof (narrow), &narrow)) +/* Two wide changes for a narrow one, so that each thread gets both. */ +if (sched_getaffinity (0 ,sizeof (sets[0 ]), &sets[0 ]))return CHILD_ERROR ; + sets[1 ] = all_cpus; + sets[2 ] = all_cpus;init_workers (); setter =fork ();if (setter <0 )return CHILD_ERROR ;if (!setter) -remote_setter (self, delay_us, &narrow); +remote_setter (self, delay_us, sets,3 );if (run_round (3 ,-1 ))return CHILD_ERROR ;/* The setter keeps running while this process exits. */
@@ -403,7 +465,7 @@ int main(void)int status, i;ksft_print_header (); -ksft_set_plan (4 ); +ksft_set_plan (6 );if (!rseq_available ())ksft_exit_skip ("rseq is unavailable\n" );
@@ -438,5 +500,12 @@ int main(void)break ; }ksft_test_result (status ==CHILD_PASS ,"affinity changed by another process\n" ); + +narrow_cpus (&narrow, &all_cpus,MAX_SETS ); + status =run_under_test (run_child_rounds,NESTED , &narrow); +ksft_test_result (status ==CHILD_PASS ,"threads created by several threads at once\n" ); + + status =run_under_test (run_child_rounds,MIGRATED , &narrow); +ksft_test_result (status ==CHILD_PASS ,"threads migrated by another process\n" );ksft_finished (); }
Signature · 1 line
2.47.3
Mathieu Desnoyers
maintainerPATCH 14/17 selftests/rseq: Test the concurrency IDs after the affinity is widened
In reply to Mathieu DesnoyersWhen the allowed CPUs of an mm grow, the kernel can give the concurrency IDs back to the tasks. Commit 7dbc307b6c72 ("sched/mmcid:
Fix affinity-triggered switch back to per-task mode") fixed a threshold that kept the CPUs as owners. No test shows it: the tests that widen the affinity only check that the IDs are unique and in range.
Add a case to mm_cid_bound_test. Three threads start on two CPUs, so the CPUs own the IDs. The threads are then allowed on all CPUs and visit each of them. The kernel aims at an mm_cid below 3 for them. That is a target, not a bound. The case prints when the target is missed. It fails only when an mm_cid is not below the possible CPUs. The case needs 8 CPUs: with fewer, the kernel keeps the CPUs as owners.
The target is met on guests with 8, 16 and 64 CPUs:
3 threads widened to 64 CPUs: largest mm_cid 2 on CPU 63
A v7.1.8 kernel with 384 CPUs, which does not have the commit above, misses it:
3 threads widened to 384 CPUs: largest mm_cid 383 on CPU 383, above the thread count target
--- .../selftests/rseq/mm_cid_bound_test.c | 88 ++++++++++++++++++- 1 file changed, 87 insertions(+), 1 deletion(-)
tools/testing/selftests/rseq/mm_cid_bound_test.c +87 -1
@@ -28,6 +28,11 @@* The number of IDs follows the union of the allowed CPUs of the threads, * which only grows. The affinity is therefore narrowed before the fork(2) * that creates the process under test. +* +* A last case widens the affinity. Three threads start on two CPUs, so the +* CPUs own the IDs. The threads are then allowed on all CPUs. The kernel +* aims at giving the IDs back to the tasks, so that three threads read an +* mm_cid below 3. That is a target too: the case prints when it is missed. */ #define _GNU_SOURCE
@@ -46,6 +51,9 @@#define MAX_THREADS 1024#define MAX_STEPS 64 +#define WIDEN_THREADS 3 +#define WIDEN_MIN_CPUS 8 +#define WIDEN_SETTLE_US 20000/* * Exit status of the process under test: a mask of the bound and of the
@@ -67,6 +75,7 @@ struct worker {static struct worker workers[MAX_THREADS ];static unsigned int epoch;static cpu_set_t allowed; +static cpu_set_t all_cpus;static int nr_allowed;static int nr_possible;
@@ -228,6 +237,49 @@ static int run_child(const int *steps, int nr_steps)return ret; } +/* +* The process under test of the widening case. It starts on two CPUs. +* Return a mask of the OVER_* values. +*/ +static int run_child_widen (void ) +{ +int nr_workers =0 , ret =0 , cpu, err; +unsigned int cid; + +if (sched_getaffinity (0 ,sizeof (allowed), &allowed)) +return CHILD_ERROR ; + nr_allowed =CPU_COUNT (&allowed); + + err =set_workers (&nr_workers,WIDEN_THREADS -1 ); +if (err) +return CHILD_ERROR ; +measure (nr_workers, &cpu); + cid =measure (nr_workers, &cpu); +ksft_print_msg ("%4d threads on %d CPUs: largest mm_cid %u on CPU %d\n" , +WIDEN_THREADS , nr_allowed, cid, cpu); + +/* Each thread widens its own affinity at the end of its next tour. */ + allowed = all_cpus; + nr_allowed =CPU_COUNT (&allowed); +if (sched_setaffinity (0 ,sizeof (allowed), &allowed)) + ret |=CHILD_ERROR ; +measure (nr_workers, &cpu); +/* The kernel changes the owners of the IDs from a work item. */ +usleep (WIDEN_SETTLE_US ); +measure (nr_workers, &cpu); + cid =measure (nr_workers, &cpu); +if (cid >= (unsigned int )nr_possible) + ret |=OVER_POSSIBLE ; +if (cid >=WIDEN_THREADS ) + ret |=OVER_STRICT ; +ksft_print_msg ("%4d threads widened to %d CPUs: largest mm_cid %u on CPU %d%s\n" , +WIDEN_THREADS , nr_allowed, cid, cpu, + ret &OVER_STRICT ?", above the thread count target" :"" ); +if (set_workers (&nr_workers,0 )) + ret |=CHILD_ERROR ; +return ret; +} +/* Keep the first @nr CPUs of @src in @dst. */ static int narrow_cpus (cpu_set_t *dst,const cpu_set_t *src,int nr) {
@@ -326,13 +378,46 @@ static void test_ramp(const cpu_set_t *all, int cpus, const char *name) "%s: below the possible CPUs\n", name); } +static void test_widen (const cpu_set_t *all) +{ +const char *name ="widened from 2 CPUs: below the possible CPUs" ; +cpu_set_t narrow; +int status =0 ; +pid_t child; + +/* With fewer CPUs the kernel keeps the CPUs as owners of the IDs. */ +if (CPU_COUNT (all) <WIDEN_MIN_CPUS || nr_possible <WIDEN_MIN_CPUS ) { +ksft_test_result_skip ("%s: fewer than %d CPUs\n" , name,WIDEN_MIN_CPUS ); +return ; + } +narrow_cpus (&narrow, all,2 ); + all_cpus = *all; +if (sched_setaffinity (0 ,sizeof (narrow), &narrow)) +ksft_exit_fail_msg ("sched_setaffinity: %s\n" ,strerror (errno)); + child =fork (); +if (!child) +_exit (run_child_widen ()); +if (child <0 ) +ksft_exit_fail_msg ("fork: %s\n" ,strerror (errno)); +if (sched_setaffinity (0 ,sizeof (*all), all)) +ksft_exit_fail_msg ("sched_setaffinity: %s\n" ,strerror (errno)); +if (waitpid (child, &status,0 ) != child) +ksft_exit_fail_msg ("waitpid: %s\n" ,strerror (errno)); +if (!WIFEXITED (status) || (WEXITSTATUS (status) &CHILD_ERROR )) +ksft_exit_fail_msg ("the process under test failed to run\n" ); +/* The thread count is a target. Only the bound decides the result. */ +if (WEXITSTATUS (status) &OVER_STRICT ) +ksft_print_msg ("widened from 2 CPUs: above the thread count target\n" ); +ksft_test_result (!(WEXITSTATUS (status) &OVER_POSSIBLE ),"%s\n" , name); +} +int main (void ) {cpu_set_t all;int nr_all;ksft_print_header (); -ksft_set_plan (3 ); +ksft_set_plan (4 );if (!rseq_available ())ksft_exit_skip ("rseq is unavailable\n" );
@@ -351,5 +436,6 @@ int main(void)test_ramp (&all,2 ,"2 CPUs" );test_ramp (&all,4 ,"4 CPUs" );test_ramp (&all, nr_all,"all CPUs" ); +test_widen (&all);ksft_finished (); }
Signature · 1 line
2.47.3
Mathieu Desnoyers
maintainerPATCH 15/17 selftests/rseq: Test two race windows of the concurrency ID mode switches
In reply to Mathieu DesnoyersTwo bugs of the concurrency ID mode switches needed a task to schedule inside a window that is a few instructions wide:
- A failed execve() makes the mm switch to per CPU mode. It then schedules before it has handed the IDs over. Fixed earlier in this series by "sched/mmcid: Let a running joiner take its CID after the fixups".
- A thread migrates after the mm switched to per CPU mode and before its ID is handed over. Fixed by commit 4327fb13fa47 ("sched/mmcid:
Prevent live lock on task to CPU mode transition").
The other tests only reach these windows by chance.
Add mm_cid_window_test, with one case for each sequence, in that order. Each case runs 40 attempts with a different timing. On an unmodified kernel the sequences miss the windows: both cases pass with either fix removed.
To hit a window every time, the kernel has to make one chosen thread sleep in it. The test marks with "Delay injection" comments where that thread would ask for the sleep. No kernel code is added for it. The matching places in kernel/sched/core.c are:
- in sched_mm_cid_fork(), after the block that changes the mode and
before mm_cid_fixup_tasks_to_cpus() (both cases);
- in sched_mm_cid_after_execve(), before sched_mm_cid_fork() (first
case).The results below were measured with a 3ms sleep, fsleep(3000), at these places. Only a task that asked for it took the sleep, through a debug prctl():
- Without "sched/mmcid: Let a running joiner take its CID after the fixups", the first case fails on every guest. A guest with 2 CPUs hangs. Guests with 3, 4 and 8 CPUs report "mm_cid 2 after a failed execve, 2 CPUs allowed".
- Commit 4327fb13fa47 ("sched/mmcid: Prevent live lock on task to CPU mode transition") added the transit mode. With it removed, the second case hangs guests with 3, 4 and 8 CPUs. That case needs 3 CPUs.
- With both fixes, both cases pass on guests with 2 to 8 CPUs.
The process under test dies with the test (PR_SET_PDEATHSIG). A thread which cannot pin itself sets a flag. The case then ends with an error. It does not wait for that thread, and it does not report a pass for a sequence which did not run.
--- tools/testing/selftests/rseq/.gitignore | 1 + tools/testing/selftests/rseq/Makefile | 3 +- .../selftests/rseq/mm_cid_window_test.c | 485 ++++++++++++++++++ 3 files changed, 488 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/rseq/mm_cid_window_test.c
tools/testing/selftests/rseq/.gitignore +1 -0
@@ -13,6 +13,7 @@ mm_cid_bound_test mm_cid_execve_test mm_cid_transition_test mm_cid_exec_affinity_test +mm_cid_window_test syscall_errors_test slice_test legacy_check
tools/testing/selftests/rseq/Makefile +2 -1
@@ -21,7 +21,8 @@ OVERRIDE_TARGETS = 1 TEST_GEN_PROGS = basic_test basic_percpu_ops_test basic_percpu_ops_mm_cid_test \ param_test_benchmark param_test_mm_cid_benchmark \ mm_cid_execve_test mm_cid_transition_test \ - mm_cid_bound_test mm_cid_exec_affinity_test + mm_cid_bound_test mm_cid_exec_affinity_test \ + mm_cid_window_test TEST_GEN_PROGS_EXTENDED = librseq.so \ param_test \
tools/testing/selftests/rseq/mm_cid_window_test.c new file +485 -0
485 lines not on this page · raw message
Signature · 1 line
2.47.3
Mathieu Desnoyers
maintainerPATCH 16/17 selftests/rseq: Test affinity changes while a task replaces or drops its mm
In reply to Mathieu Desnoyerssched_setaffinity() on another task updates the allowed CPUs of the mm of the target. set_cpus_allowed_common() reads p->mm under p->pi_lock for that. The commit "sched/mmcid: Synchronize MM teardown with affinity updates" takes the same lock once p->mm has changed, before the old mm is dropped.
No test runs an affinity change against such a teardown. Weiming Shi reported the use after free with a reproducer: one process calls execve() in a loop while other processes change its affinity.
Add mm_cid_teardown_test. Its first case follows that reproducer. One process calls execve() on the test in a loop on its own CPU. Up to four other processes change its affinity between one and two CPUs.
The second case drops an mm instead of replacing one. Short lived processes exit on one CPU. Another process changes their affinity from a second CPU. No case covers kthread_unuse_mm().
User space cannot see a read of the freed mm. Only KASAN reports it. So both cases check that the two sides keep making progress.
The test starts processes, not threads. glibc needs futex(2) to create a thread. In the exec case, the use after free is only reachable without CONFIG_FUTEX. With CONFIG_FUTEX, futex_exec_done() already takes p->pi_lock after the switch to the new mm.
Both cases pass on guests with 2 to 64 CPUs. The first case needs 3 CPUs and is skipped below that. On the 8-CPU guest:
# 1629 execve calls, 427211 affinity changes ok 1 affinity changes while a task replaces its mm # 2140 processes exited, 86658 affinity changes ok 2 affinity changes while a task drops its mm
A guest with KASAN and without CONFIG_FUTEX ran ten seconds per case on 8 CPUs. The kernel reported nothing. This kernel had that commit. The test was not run on a kernel without it.
--- tools/testing/selftests/rseq/Makefile | 2 +- .../selftests/rseq/mm_cid_teardown_test.c | 396 ++++++++++++++++++ 2 files changed, 397 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/rseq/mm_cid_teardown_test.c
tools/testing/selftests/rseq/Makefile +1 -1
@@ -22,7 +22,7 @@ TEST_GEN_PROGS = basic_test basic_percpu_ops_test basic_percpu_ops_mm_cid_test \ param_test_benchmark param_test_mm_cid_benchmark \ mm_cid_execve_test mm_cid_transition_test \ mm_cid_bound_test mm_cid_exec_affinity_test \ - mm_cid_window_test + mm_cid_window_test mm_cid_teardown_test TEST_GEN_PROGS_EXTENDED = librseq.so \ param_test \
tools/testing/selftests/rseq/mm_cid_teardown_test.c new file +396 -0
396 lines not on this page · raw message
Signature · 1 line
2.47.3
Mathieu Desnoyers
maintainerPATCH 17/17 selftests/rseq: Test the last user after an affinity change during execve
In reply to Mathieu DesnoyersThe last user case of mm_cid_execve_test needs per CPU mode to last down to one thread. With two allowed CPUs, the kernel only does that with two or three possible CPUs. The case is skipped on larger machines.
The kernel also keeps per CPU mode down to one thread when the mm counts one allowed CPU, on any machine. Only one CPU then owns a CID, unless the thread runs on a CPU that the mm does not count. That happens when another process changes its affinity during an execve(): the mm has no user then, and mm_update_cpus_allowed() ignores it.
Add a case that builds this state and then fails execve() as the last user on one CPU, to rejoin on another. The comment above run_child_widened() has the sequence as a diagram. The single thread must read mm_cid 0 on both CPUs at the end.
Without "sched/mmcid: Reset stale state when the first user joins an mm", on guests with 4, 8 and 64 CPUs:
single thread, mm_cid 1 on CPU 0 after a move from CPU 1 to CPU 0 not ok 3 mm_cid of a single thread after its affinity was widened during execve
The thread still reads 1 one second later. With that commit, the case passes on guests with 2 to 64 CPUs.
The failing kernel also lacked "sched/mmcid: Take the allowed CPUs of the first user when it joins". That commit records the affinity change when the task rejoins, so the state is not reached with it. With that commit and without the reset, the case passes on guests with 2 to 64 CPUs.
--- .../selftests/rseq/mm_cid_execve_test.c | 235 +++++++++++++++++- 1 file changed, 232 insertions(+), 3 deletions(-)
tools/testing/selftests/rseq/mm_cid_execve_test.c +232 -3
@@ -17,6 +17,11 @@* CPUs. The case is skipped when a single thread reads 0 on both CPUs * before the execve(2). * +* A third case covers the last user on a machine of any size. The kernel +* also keeps per CPU mode down to one thread when the mm counts one +* allowed CPU. The case makes such a thread run on a second CPU. The +* diagram above run_child_widened() has the sequence. +* * Per CPU mode needs more threads than the mm has CIDs. The number of CIDs * follows the union of the allowed CPUs of the threads, which only grows. * The affinity is therefore narrowed before the fork(2) that creates the
@@ -33,8 +38,10 @@#include <stdio.h> #include <stdlib.h> #include <string.h> +#include <sys/mman.h> #include <sys/stat.h> #include <sys/wait.h> +#include <time.h> #include <unistd.h> #include "rseq.h"
@@ -357,6 +364,209 @@ static int run_child_moved(int delay_us)return ret; } +/* What the process under test and its helper share in run_child_widened(). */ +struct widen_shared { +int phase; +int widened; +}; + +static int widen_cpu[NARROW_CPUS ]; + +static uint64_t now_ms (void ) +{ +struct timespec ts; + +clock_gettime (CLOCK_MONOTONIC , &ts); +return (uint64_t )ts.tv_sec *1000 + ts.tv_nsec /1000000 ; +} + +/* Spin until @sh->phase reaches @phase. Return false after two seconds. */ +static bool widen_wait (struct widen_shared *sh,int phase) +{ +uint64_t end =now_ms () +2000 ; + +while (RSEQ_READ_ONCE (sh->phase) < phase) { +if (now_ms () > end) +return false; + } +return true; +} + +/* +* The helper of run_child_widened(). @pid is the process under test, which +* starts on the CPU @a. +* - Phase 1: widen the affinity of @pid to @a and @b, @delay_us after the +* start of its first execve(2) loop. +* - Phase 2: keep the CPU @a busy, so that the scheduler moves @pid to @b. +* - Phase 3: move @pid back to @a, @delay_us after the start of its second +* execve(2) loop. +* Return the exit status of the helper. +*/ +static int widen_helper (struct widen_shared *sh,pid_t pid,int a,int b,int delay_us) +{ +cpu_set_t set; + +/* Run beside the process under test, not on its CPU. */ +CPU_ZERO (&set); +CPU_SET (b, &set); +if (sched_setaffinity (0 ,sizeof (set), &set) || !widen_wait (sh,1 )) +return 1 ; +usleep (delay_us); +CPU_SET (a, &set); +if (sched_setaffinity (pid,sizeof (set), &set)) +return 1 ; +RSEQ_WRITE_ONCE (sh->widened,1 ); + +if (!widen_wait (sh,2 )) +return 1 ; +CPU_ZERO (&set); +CPU_SET (a, &set); +if (sched_setaffinity (0 ,sizeof (set), &set) || !widen_wait (sh,3 )) +return 1 ; +usleep (delay_us); +return !!sched_setaffinity (pid,sizeof (set), &set); +} + +/* +* Fail execve(2) on @path MOVE_EXECS times. Return 0, or -1 after reporting +* an error other than ENOEXEC. +*/ +static int fail_execs (char *path) +{ +int i; + +for (i =0 ; i <MOVE_EXECS ; i++) { +char *const av[] = { path,NULL }; + +if (execve (path, av,NULL ) ==-1 && errno !=ENOEXEC ) { +fprintf (stderr,"execve(%s): %s, expected ENOEXEC\n" , + path,strerror (errno)); +return -1 ; + } + } +return 0 ; +} + +/* +* The third process under test, P. A and B are two CPUs. P starts with A +* as its only allowed CPU. H is a helper process. It makes the affinity +* changes: a change made by P itself would reach the mm. +* +* P H +* +* execve() fails, in a loop +* leaves the mm: no user left +* sched_setaffinity(P, {A, B}) +* the mm has no user: its count +* of allowed CPUs stays 1 +* rejoins the mm +* creates a thread +* the mm switches to per CPU mode +* joins the thread +* one user left, still per CPU mode +* keeps A busy +* is moved to B by the scheduler +* A and B each own a CID +* execve() fails, in a loop +* leaves the mm on B: no user left +* sched_setaffinity(P, {A}) +* rejoins the mm on A +* visits A and B +* +* P must read 0 on both CPUs at the end. Each change of H has to land +* while P is inside an execve(), so the case runs many times with a +* different delay. +* Return a CHILD_* status. +*/ +static int run_child_widened (int delay_us) +{ +int a = widen_cpu[0 ], b = widen_cpu[1 ], i, status, ret =CHILD_PASS ; +struct widen_shared *sh; +pid_t helper, self =getpid (); +char path[PATH_MAX ]; +pthread_t thread; +uint64_t end; + + sh =mmap (NULL ,sizeof (*sh),PROT_READ |PROT_WRITE , +MAP_SHARED |MAP_ANONYMOUS ,-1 ,0 ); +if (sh ==MAP_FAILED ) +return CHILD_ERROR ; +if (make_unknown_format_file (path,sizeof (path))) { +fprintf (stderr,"no directory to run an unknown format file from\n" ); +return CHILD_SKIP ; + } + helper =fork (); +if (helper <0 ) { +remove_file (path); +return CHILD_ERROR ; + } +if (!helper) +_exit (widen_helper (sh, self, a, b, delay_us)); + +RSEQ_WRITE_ONCE (sh->phase,1 ); +if (fail_execs (path)) { + ret =CHILD_ERROR ; + goto out; + } + end =now_ms () +2000 ; +while (!RSEQ_READ_ONCE (sh->widened) &&now_ms () < end) + ; +if (!RSEQ_READ_ONCE (sh->widened)) { + ret =CHILD_ERROR ; + goto out; + } + +if (pthread_create (&thread,NULL , waiter,NULL )) { + ret =CHILD_ERROR ; + goto out; + } +RSEQ_WRITE_ONCE (stop,1 ); +if (join_thread (thread)) { + ret =CHILD_ERROR ; + goto out; + } + +/* No affinity change by this process: it would reach the mm. */ +RSEQ_WRITE_ONCE (sh->phase,2 ); + end =now_ms () +2000 ; +while (sched_getcpu () != b &&now_ms () < end) + ; +if (sched_getcpu () != b) { +fprintf (stderr,"the scheduler did not move the process to CPU %d\n" , b); + ret =CHILD_SKIP ; + goto out; + } + +RSEQ_WRITE_ONCE (sh->phase,3 ); +if (fail_execs (path)) { + ret =CHILD_ERROR ; + goto out; + } +out: +if (ret !=CHILD_PASS ) +kill (helper,SIGKILL ); +if (waitpid (helper, &status,0 ) != helper || + (ret ==CHILD_PASS && (!WIFEXITED (status) ||WEXITSTATUS (status)))) + ret =CHILD_ERROR ; +if (remove_file (path) && ret ==CHILD_PASS ) + ret =CHILD_ERROR ; + +/* The first visit lets the kernel settle, the second one is judged. */ +for (i =0 ; i <2 *NARROW_CPUS && ret ==CHILD_PASS ; i++) { +unsigned int cid =cid_on_cpu (widen_cpu[i %NARROW_CPUS ]); + +if (cid ==UINT_MAX ) { + ret =CHILD_ERROR ; + }else if (i >=NARROW_CPUS && cid) { +fprintf (stderr, +"single thread, mm_cid %u on CPU %d after a move from CPU %d to CPU %d\n" , + cid, widen_cpu[i %NARROW_CPUS ], b, a); + ret =CHILD_FAIL ; + } + } +return ret; +} +/* Keep the first @nr CPUs of @src in @dst. */ static int narrow_cpus (cpu_set_t *dst,const cpu_set_t *src,int nr) {
@@ -424,11 +634,13 @@ static int run_child_threads(int unused)int main (void ) { -cpu_set_t allowed, narrow; -int status, i; +static const char widened[] = +"mm_cid of a single thread after its affinity was widened during execve" ; +cpu_set_t allowed, narrow, one; +int status, i, cpu, nr =0 ;ksft_print_header (); -ksft_set_plan (2 ); +ksft_set_plan (3 );if (!rseq_available ())ksft_exit_skip ("rseq is unavailable\n" );
@@ -452,5 +664,22 @@ int main(void)break ; }report (status,"mm_cid of a single thread after a failed execve on another CPU" ); + +for (cpu =0 ; cpu <CPU_SETSIZE && nr <NARROW_CPUS ; cpu++) { +if (CPU_ISSET (cpu, &narrow)) + widen_cpu[nr++] = cpu; + } +if (nr <NARROW_CPUS ) { +ksft_test_result_skip ("%s: fewer than %d CPUs\n" , widened,NARROW_CPUS ); +ksft_finished (); + } +narrow_cpus (&one, &allowed,1 ); +/* Both changes of the helper have to land inside an execve. */ +for (i =0 ; i <2 *ATTEMPTS ; i++) { + status =run_under_test (run_child_widened, (i *37 ) %1000 , &one, &allowed); +if (status !=CHILD_PASS ) +break ; + } +report (status, widened);ksft_finished (); }
Signature · 1 line
2.47.3