x86/resctrl: Fix task CLOSID/RMID update race
Subsystems (2)
PATCH v7 1/1 x86/resctrl: Fix task CLOSID/RMID update race
When the user moves a running task to a new rdtgroup using the tasks file interface or by deleting its rdtgroup, the resulting change in CLOSID/RMID must be immediately propagated to the PQR_ASSOC MSR on the task(s) CPUs.
x86 allows reordering loads with prior stores, so if the task starts running between a task_curr() check that the CPU hoisted before the stores in the CLOSID/RMID update then it can start running with the old CLOSID/RMID until it is switched again because __rdtgroup_move_task() failed to determine that it needs to be interrupted to obtain the new CLOSID/RMID.
Refer to the diagram below:
CPU 0 CPU 1
----- -----__rdtgroup_move_task():
curr <- t1->cpu->rq->curr
__schedule():
rq->curr <- t1
resctrl_sched_in():
t1->{closid,rmid} -> {1,1} t1->{closid,rmid} <- {2,2}
if (curr == t1) // false
IPI(t1->cpu)
A similar race impacts rdt_move_group_tasks(), which updates tasks in a deleted rdtgroup.
In both cases, use smp_mb() to order the task_struct::{closid,rmid} stores before the loads in task_curr(). In particular, in the rdt_move_group_tasks() case, simply execute an smp_mb() on every iteration with a matching task.
It is possible to use a single smp_mb() in rdt_move_group_tasks(), but this would require two passes and a means of remembering which task_structs were updated in the first loop. However, benchmarking results below showed too little performance impact in the simple approach to justify implementing the two-pass approach.
Times below were collected using perf stat to measure the time to remove a group containing a 1600-task, parallel workload.
CPU: Intel(R) Xeon(R) Platinum P-8136 CPU @ 2.00GHz (112 threads)
# mkdir /sys/fs/resctrl/test # echo $$ > /sys/fs/resctrl/test/tasks # perf bench sched messaging -g 40 -l 100000
task-clock time ranges collected using:
# perf stat rmdir /sys/fs/resctrl/test
Baseline: 1.54 - 1.60 ms smp_mb() every matching task: 1.57 - 1.67 ms
---
Patch history:
v7:
- Remove exploit case added in v6 and un-CC stable
v6:
- Explain exploit case in changelog for stable
- Add Fixes: lines
v5:
- Just put an smp_mb() between CLOSID/RMID stores and task_curr() calls
- Add a diagram detailing the race to the changelog
v4:
- Reorder the patches so that justification for sending more IPIs can reference the patch fixing __rdtgroup_move_task().
- Correct tense of wording used in changelog and comments
v3:
- Split the handling of multi-task and single-task operations into separate patches, now that they're handled differently.
- Clarify justification in the commit message, including moving some of it out of inline code comment.
v2:
- Following Reinette's suggestion: use task_call_func() for single task, IPI broadcast for group movements.
- Rebased to v6.1-rc4
v1: https://lore.kernel.org/lkml/[email protected]/
v2: https://lore.kernel.org/lkml/[email protected]/
v3: https://lore.kernel.org/lkml/[email protected]/
v4: https://lore.kernel.org/lkml/[email protected]/
v5: https://lore.kernel.org/lkml/[email protected]/
v6: https://lore.kernel.org/lkml/[email protected]/
--- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/resctrl/rdtgroup.c +11 -1
@@ -580,8 +580,10 @@ static int __rdtgroup_move_task(struct task_struct *tsk,/* * Ensure the task's closid and rmid are written before determining if * the task is current that will decide if it will be interrupted. +* This pairs with the full barrier between the rq->curr update and +* resctrl_sched_in() during context switch. */ -barrier (); +smp_mb ();/* * By now, the task's closid and rmid are set. If the task is current
@@ -2401,6 +2403,14 @@ static void rdt_move_group_tasks(struct rdtgroup *from, struct rdtgroup *to,WRITE_ONCE (t->closid, to->closid);WRITE_ONCE (t->rmid, to->mon.rmid); +/* +* Order the closid/rmid stores above before the loads +* in task_curr(). This pairs with the full barrier +* between the rq->curr update and resctrl_sched_in() +* during context switch. +*/ +smp_mb (); +/* * If the task is on a CPU, set the CPU in the mask. * The detection is inaccurate as tasks might move or
base-commit: 830b3c68c1fb1e9176028d02ef86f3cf76aa2476
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
Ran few tests with this patch on AMD systems. They are all looking good.
Signature · 2 lines
Thanks Babu Moger
The following commit has been merged into the x86/urgent branch of tip:
The following commit has been merged into the x86/urgent branch of tip:
Commit-ID: fe1f0714385fbcf76b0cbceb02b7277d842014fc Gitweb: https://git.kernel.org/tip/fe1f0714385fbcf76b0cbceb02b7277d842014fc Author: Peter Newman <[email protected]> AuthorDate: Tue, 20 Dec 2022 17:11:23 +01:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Tue, 10 Jan 2023 19:47:30 +01:00
x86/resctrl: Fix task CLOSID/RMID update race
When the user moves a running task to a new rdtgroup using the task's file interface or by deleting its rdtgroup, the resulting change in CLOSID/RMID must be immediately propagated to the PQR_ASSOC MSR on the task(s) CPUs.
x86 allows reordering loads with prior stores, so if the task starts running between a task_curr() check that the CPU hoisted before the stores in the CLOSID/RMID update then it can start running with the old CLOSID/RMID until it is switched again because __rdtgroup_move_task() failed to determine that it needs to be interrupted to obtain the new CLOSID/RMID.
Refer to the diagram below:
CPU 0 CPU 1
----- -----__rdtgroup_move_task():
curr <- t1->cpu->rq->curr
__schedule():
rq->curr <- t1
resctrl_sched_in():
t1->{closid,rmid} -> {1,1} t1->{closid,rmid} <- {2,2}
if (curr == t1) // false
IPI(t1->cpu)
A similar race impacts rdt_move_group_tasks(), which updates tasks in a deleted rdtgroup.
In both cases, use smp_mb() to order the task_struct::{closid,rmid} stores before the loads in task_curr(). In particular, in the rdt_move_group_tasks() case, simply execute an smp_mb() on every iteration with a matching task.
It is possible to use a single smp_mb() in rdt_move_group_tasks(), but this would require two passes and a means of remembering which task_structs were updated in the first loop. However, benchmarking results below showed too little performance impact in the simple approach to justify implementing the two-pass approach.
Times below were collected using perf stat to measure the time to remove a group containing a 1600-task, parallel workload.
CPU: Intel(R) Xeon(R) Platinum P-8136 CPU @ 2.00GHz (112 threads)
# mkdir /sys/fs/resctrl/test # echo $$ > /sys/fs/resctrl/test/tasks # perf bench sched messaging -g 40 -l 100000
task-clock time ranges collected using:
# perf stat rmdir /sys/fs/resctrl/test
Baseline: 1.54 - 1.60 ms smp_mb() every matching task: 1.57 - 1.67 ms
[ bp: Massage commit message. ]
--- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/resctrl/rdtgroup.c +11 -1
@@ -580,8 +580,10 @@ static int __rdtgroup_move_task(struct task_struct *tsk,/* * Ensure the task's closid and rmid are written before determining if * the task is current that will decide if it will be interrupted. +* This pairs with the full barrier between the rq->curr update and +* resctrl_sched_in() during context switch. */ -barrier (); +smp_mb ();/* * By now, the task's closid and rmid are set. If the task is current
@@ -2402,6 +2404,14 @@ static void rdt_move_group_tasks(struct rdtgroup *from, struct rdtgroup *to,WRITE_ONCE (t->rmid, to->mon.rmid);/* +* Order the closid/rmid stores above before the loads +* in task_curr(). This pairs with the full barrier +* between the rq->curr update and resctrl_sched_in() +* during context switch. +*/ +smp_mb (); + +/* * If the task is on a CPU, set the CPU in the mask. * The detection is inaccurate as tasks might move or * schedule before the smp function call takes place.