Skip to contentSkip to conversation

resend, cpuset/hotplug fixes

PATCH In review Oleg Nesterov 4 participants 9 messages Sep 10, 2009 – Sep 11, 2009

lore
Share & tools
lore
#0 lore raw

cover resend, cpuset/hotplug fixes

(my apologies to those who see this twice).

The 1st patch is preparation. 2-3 fix different problems, the 3rd one depends on 2nd.

These patches change the code which I don't really understand, please review.

As for the 3rd patch, it replaces

cpu_hotplug-dont-affect-current-tasks-affinity.patch

in -mm tree. Imho the new patch is more simple and clean, but of course this is subjective and I am biased.

Oleg.

#1 lore raw

Hi Oleg,

Quoted 10 lines · Oleg Nesterov

(my apologies to those who see this twice).

The 1st patch is preparation. 2-3 fix different problems, the 3rd one depends on 2nd.

These patches change the code which I don't really understand, please review.

Quoted 6 lines

As for the 3rd patch, it replaces

cpu_hotplug-dont-affect-current-tasks-affinity.patch

in -mm tree. Imho the new patch is more simple and clean, but of course this is subjective and I am biased.

The patches look good to me FWIW.

Thanks a lot for taking care of this!

Rafael

Acked-by on the v1 series
#2 lore raw
In reply to Oleg Nesterov
Quoted 10 lines · Oleg Nesterov

(my apologies to those who see this twice).

The 1st patch is preparation. 2-3 fix different problems, the 3rd one depends on 2nd.

These patches change the code which I don't really understand, please review.

Quoted 6 lines

As for the 3rd patch, it replaces

cpu_hotplug-dont-affect-current-tasks-affinity.patch

in -mm tree. Imho the new patch is more simple and clean, but of course this is subjective and I am biased.

Look good to me.

Ingo, will you stick them in -tip?

#3 lore raw
Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Oleg Nesterov

(my apologies to those who see this twice).

The 1st patch is preparation. 2-3 fix different problems, the 3rd one depends on 2nd.

These patches change the code which I don't really understand, please review.

As for the 3rd patch, it replaces

cpu_hotplug-dont-affect-current-tasks-affinity.patch

Quoted 10 lines
Quoted 2 lines

in -mm tree. Imho the new patch is more simple and clean, but of course this is subjective and I am biased.

Look good to me.

Ingo, will you stick them in -tip?

Sorry. I was taken ill for weeks and forgot to follow these discussions. Especially I should say sorry to Oleg.

I have different concept. cpuset_cpus_allowed() is not called at atomic context nor non-preemptable context nor other critical context. So it should be allowed to use mutexs. That's what I think.

There is a bug when migration_call() requires a mutex before migration has been finished when cpu offline as Oleg described.

Bug this bug is only happened when cpu offline. cpu offline is rare and is slowpath. I think we should fix cpu offline and ensure it requires the mutex safely.

Oleg's patch moves all dirty things into CPUSET subsystem and makes cpuset_cpus_allowed() does not require any mutex and increases CPUSET's coupling. I don't feel it's good.

Anyway, Oleg's patch works good.

Quoted 5 lines
Quoted 3 lines

cpuset_cpus_allowed() is not only used for CPU offline.

Quoted 2 lines

sched_setaffinity() also uses it.

Sure. And it must take get_online_cpus() to avoid the races with hotplug.

Oleg hasn't answered that
"is it safe when pdflush() calls cpuset_cpus_allowed()?".
A patch may be needed to ensure pdflush() calls cpuset_cpus_allowed() safely.

One other minor thing:
Oleg's patch may introduce a trouble in PREEEMPT_RT tree, because spinlock in RT is also mutex. Likely I'm wrong.

- Lai

#4 lore raw
Quoted 3 lines · Lai Jiangshan

One other minor thing:
Oleg's patch may introduce a trouble in PREEEMPT_RT tree, because spinlock in RT is also mutex. Likely I'm wrong.

But they have priority-inheritance, hence you cannot create a deadlock through preemption. If the kstopmachine thread blocks on a mutex, the owner gets boosted to kstopmachine's prio and runs to completion, after that kstopmachine continues.

#5 lore raw
Quoted 10 lines · Peter Zijlstra
Quoted 3 lines · Lai Jiangshan

One other minor thing:
Oleg's patch may introduce a trouble in PREEEMPT_RT tree, because spinlock in RT is also mutex. Likely I'm wrong.

But they have priority-inheritance, hence you cannot create a deadlock through preemption. If the kstopmachine thread blocks on a mutex, the owner gets boosted to kstopmachine's prio and runs to completion, after that kstopmachine continues.

The deadlock is because the owner is at the dead cpu, It's not because the owner's prio is low. priority-inheritance can't help it.

I think we need to use a raw spinlock.

#6 lore raw
Quoted 12 lines · Lai Jiangshan
Quoted 10 lines · Peter Zijlstra
Quoted 3 lines · Lai Jiangshan

One other minor thing:
Oleg's patch may introduce a trouble in PREEEMPT_RT tree, because spinlock in RT is also mutex. Likely I'm wrong.

But they have priority-inheritance, hence you cannot create a deadlock through preemption. If the kstopmachine thread blocks on a mutex, the owner gets boosted to kstopmachine's prio and runs to completion, after that kstopmachine continues.

Quoted 4 lines

The deadlock is because the owner is at the dead cpu, It's not because the owner's prio is low. priority-inheritance can't help it.

I think we need to use a raw spinlock.

Not in mainline you don't.

#7 lore raw
In reply to Lai Jiangshan
Quoted 4 lines · Lai Jiangshan

I have different concept. cpuset_cpus_allowed() is not called at atomic context nor non-preemptable context nor other critical context. So it should be allowed to use mutexs. That's what I think.

It hardly does any work at all, so why bother with a mutex?

#8 lore raw
In reply to Lai Jiangshan
Quoted 4 lines · Lai Jiangshan

I have different concept. cpuset_cpus_allowed() is not called at atomic context nor non-preemptable context nor other critical context. So it should be allowed to use mutexs. That's what I think.

Well, it is called from non-preemptable context: move_task_off_dead_cpu(). That is why before this patch we had cpuset_cpus_allowed_lock(). And this imho adds unneeded complications.

And I can't understand why sched_setaffinity() path should take the global mutex instead of per-cpuset spinlock.

Quoted 6 lines

There is a bug when migration_call() requires a mutex before migration has been finished when cpu offline as Oleg described.

Bug this bug is only happened when cpu offline. cpu offline is rare and is slowpath. I think we should fix cpu offline and ensure it requires the mutex safely.

This is subjective, but I can't agree. I think we should fix cpusets instead. We should try to avoid the dependencies between different subsystems as much as possible.

Quoted 3 lines

Oleg's patch moves all dirty things into CPUSET subsystem and makes cpuset_cpus_allowed() does not require any mutex and increases CPUSET's coupling. I don't feel it's good.

Again, subjective... But I can't understand "increases CPUSET's coupling".

Quoted 1 line

From my pov, this patch cleanups and simplifies the code. This was the

main motivation, the bugfix is just the "side effect". I don't understand how this subtle cpuset_lock() can make things better. I can't understand why we need the global lock to calc cpus_allowed.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

cpuset_cpus_allowed() is not only used for CPU offline.

Quoted 2 lines

sched_setaffinity() also uses it.

Sure. And it must take get_online_cpus() to avoid the races with hotplug.

Oleg hasn't answered that
"is it safe when pdflush() calls cpuset_cpus_allowed()?".

Because I didn't see such a question ;) perhaps I missed it previously...

Quoted 1 line

A patch may be needed to ensure pdflush() calls cpuset_cpus_allowed() safely.

What is wrong with pdflush()->cpuset_cpus_allowed() ? Why this is not safe?

This

	cpuset_cpus_allowed(current, cpus_allowed);
	set_cpus_allowed_ptr(current, cpus_allowed);

looks equally racy, with or without the patch. But this is a bit off-topic, mm/pdflush.c has gone away.

Quoted 3 lines

One other minor thing:
Oleg's patch may introduce a trouble in PREEEMPT_RT tree, because spinlock in RT is also mutex. Likely I'm wrong.

Yes, probably -rt need raw_lock (as you pointed out).

Oleg.