resend, cpuset/hotplug fixes
cover resend, cpuset/hotplug fixes
(my apologies to those who see this twice).
The 1st patch is preparation. 2-3 fix different problems, the 3rd one depends on 2nd.
These patches change the code which I don't really understand, please review.
As for the 3rd patch, it replaces
cpu_hotplug-dont-affect-current-tasks-affinity.patch
in -mm tree. Imho the new patch is more simple and clean, but of course this is subjective and I am biased.
Oleg.
Hi Oleg,
Quoted 10 lines · Oleg Nesterov
(my apologies to those who see this twice).
The 1st patch is preparation. 2-3 fix different problems, the 3rd one depends on 2nd.
These patches change the code which I don't really understand, please review.
Quoted 6 lines
As for the 3rd patch, it replaces
cpu_hotplug-dont-affect-current-tasks-affinity.patch
in -mm tree. Imho the new patch is more simple and clean, but of course this is subjective and I am biased.
The patches look good to me FWIW.
Thanks a lot for taking care of this!
Rafael
Quoted 10 lines · Oleg Nesterov
(my apologies to those who see this twice).
The 1st patch is preparation. 2-3 fix different problems, the 3rd one depends on 2nd.
These patches change the code which I don't really understand, please review.
Quoted 6 lines
As for the 3rd patch, it replaces
cpu_hotplug-dont-affect-current-tasks-affinity.patch
in -mm tree. Imho the new patch is more simple and clean, but of course this is subjective and I am biased.
Look good to me.
Ingo, will you stick them in -tip?
Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Oleg Nesterov
(my apologies to those who see this twice).
The 1st patch is preparation. 2-3 fix different problems, the 3rd one depends on 2nd.
These patches change the code which I don't really understand, please review.
As for the 3rd patch, it replaces
cpu_hotplug-dont-affect-current-tasks-affinity.patch
Quoted 10 lines
Quoted 2 lines
in -mm tree. Imho the new patch is more simple and clean, but of course this is subjective and I am biased.
Look good to me.
Acked-by Peter Zijlstra <[email protected]>Ingo, will you stick them in -tip?
Sorry. I was taken ill for weeks and forgot to follow these discussions. Especially I should say sorry to Oleg.
I have different concept. cpuset_cpus_allowed() is not called at atomic context nor non-preemptable context nor other critical context. So it should be allowed to use mutexs. That's what I think.
There is a bug when migration_call() requires a mutex before migration has been finished when cpu offline as Oleg described.
Bug this bug is only happened when cpu offline. cpu offline is rare and is slowpath. I think we should fix cpu offline and ensure it requires the mutex safely.
Oleg's patch moves all dirty things into CPUSET subsystem and makes cpuset_cpus_allowed() does not require any mutex and increases CPUSET's coupling. I don't feel it's good.
Anyway, Oleg's patch works good.
Quoted 5 lines
Quoted 3 lines
cpuset_cpus_allowed()is not only used for CPU offline.Quoted 2 lines
sched_setaffinity()also uses it.Sure. And it must take
get_online_cpus()to avoid the races with hotplug.
Oleg hasn't answered that
"is it safe when pdflush() calls cpuset_cpus_allowed()?".
A patch may be needed to ensure pdflush() calls cpuset_cpus_allowed() safely.
One other minor thing:
Oleg's patch may introduce a trouble in PREEEMPT_RT tree, because spinlock in RT is also mutex. Likely I'm wrong.
- Lai
Quoted 3 lines · Lai Jiangshan
One other minor thing:
Oleg's patch may introduce a trouble inPREEEMPT_RTtree, because spinlock in RT is also mutex. Likely I'm wrong.
But they have priority-inheritance, hence you cannot create a deadlock through preemption. If the kstopmachine thread blocks on a mutex, the owner gets boosted to kstopmachine's prio and runs to completion, after that kstopmachine continues.
Quoted 10 lines · Peter Zijlstra
Quoted 3 lines · Lai Jiangshan
One other minor thing:
Oleg's patch may introduce a trouble inPREEEMPT_RTtree, because spinlock in RT is also mutex. Likely I'm wrong.But they have priority-inheritance, hence you cannot create a deadlock through preemption. If the kstopmachine thread blocks on a mutex, the owner gets boosted to kstopmachine's prio and runs to completion, after that kstopmachine continues.
The deadlock is because the owner is at the dead cpu, It's not because the owner's prio is low. priority-inheritance can't help it.
I think we need to use a raw spinlock.
Quoted 12 lines · Lai Jiangshan
Quoted 10 lines · Peter Zijlstra
Quoted 3 lines · Lai Jiangshan
One other minor thing:
Oleg's patch may introduce a trouble inPREEEMPT_RTtree, because spinlock in RT is also mutex. Likely I'm wrong.But they have priority-inheritance, hence you cannot create a deadlock through preemption. If the kstopmachine thread blocks on a mutex, the owner gets boosted to kstopmachine's prio and runs to completion, after that kstopmachine continues.
Quoted 4 lines
The deadlock is because the owner is at the dead cpu, It's not because the owner's prio is low. priority-inheritance can't help it.
I think we need to use a raw spinlock.
Not in mainline you don't.
Quoted 4 lines · Lai Jiangshan
I have different concept.
cpuset_cpus_allowed()is not called at atomic context nor non-preemptable context nor other critical context. So it should be allowed to use mutexs. That's what I think.
It hardly does any work at all, so why bother with a mutex?
Quoted 4 lines · Lai Jiangshan
I have different concept.
cpuset_cpus_allowed()is not called at atomic context nor non-preemptable context nor other critical context. So it should be allowed to use mutexs. That's what I think.
Well, it is called from non-preemptable context: move_task_off_dead_cpu(). That is why before this patch we had cpuset_cpus_allowed_lock(). And this imho adds unneeded complications.
And I can't understand why sched_setaffinity() path should take the global mutex instead of per-cpuset spinlock.
Quoted 6 lines
There is a bug when
migration_call()requires a mutex before migration has been finished when cpu offline as Oleg described.Bug this bug is only happened when cpu offline. cpu offline is rare and is slowpath. I think we should fix cpu offline and ensure it requires the mutex safely.
This is subjective, but I can't agree. I think we should fix cpusets instead. We should try to avoid the dependencies between different subsystems as much as possible.
Quoted 3 lines
Oleg's patch moves all dirty things into CPUSET subsystem and makes
cpuset_cpus_allowed()does not require any mutex and increases CPUSET's coupling. I don't feel it's good.
Again, subjective... But I can't understand "increases CPUSET's coupling".
Quoted 1 line
From my pov, this patch cleanups and simplifies the code. This was the
main motivation, the bugfix is just the "side effect". I don't understand how this subtle cpuset_lock() can make things better. I can't understand why we need the global lock to calc cpus_allowed.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
cpuset_cpus_allowed()is not only used for CPU offline.Quoted 2 lines
sched_setaffinity()also uses it.Sure. And it must take
get_online_cpus()to avoid the races with hotplug.Oleg hasn't answered that
"is it safe whenpdflush()callscpuset_cpus_allowed()?".
Because I didn't see such a question ;) perhaps I missed it previously...
Quoted 1 line
A patch may be needed to ensure
pdflush()callscpuset_cpus_allowed()safely.
What is wrong with pdflush()->cpuset_cpus_allowed() ? Why this is not safe?
This
cpuset_cpus_allowed(current, cpus_allowed); set_cpus_allowed_ptr(current, cpus_allowed);
looks equally racy, with or without the patch. But this is a bit off-topic, mm/pdflush.c has gone away.
Quoted 3 lines
One other minor thing:
Oleg's patch may introduce a trouble inPREEEMPT_RTtree, because spinlock in RT is also mutex. Likely I'm wrong.
Yes, probably -rt need raw_lock (as you pointed out).
Oleg.