Skip to contentSkip to conversation

CPU hotplug: Slow down hotplug operations

PATCH In review Borislav Petkov 5 participants 9 messages May 7, 2014 – May 12, 2014 Driver core

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 CPU hotplug: Slow down hotplug operations

From: Borislav Petkov <[email protected]>

We have all those eager tester dudes which scratch up a dirty script to pound on CPU hotplug senselessly and then report bugs they've managed to trigger.

Well, first of all, most, if not all, bugs they trigger are CPU hotplug related anyway. But we know hotplug is full of duct tape and brown paper bags. So we end up clearly wasting too much time dealing with a mechanism we know it is b0rked in the first place.

Oh, and I would understand if that pounding were close to some real usage patterns but I've yet to receive a justification for toggling cores on- and offline senselessly.

In any case, before this gets rewritten properly (I'm being told we might get lucky after all) let's slow down hotplugging on purpose and thus make it uninteresting, as a temporary brown paper bag solution until the real thing gets done.

This way we'll save us a lot of time and efforts in chasing the wrong bugs.

Cc H. Peter Anvin <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Linus Torvalds <[email protected]>
---
 drivers/base/cpu.c | 9 +++++++++
 1 file changed, 9 insertions(+)
drivers/base/cpu.c +9 -0
index 006b1bc5297d..615c7af767ed 100644
@@ -40,6 +40,11 @@ static void change_cpu_under_node(struct cpu *cpu,
 	cpu->node_id = to_nid;
 }
 
+static void delay_hotplug(void)
+{
+	schedule_timeout_uninterruptible(msecs_to_jiffies(MSEC_PER_SEC));
+}
+
 static int __ref cpu_subsys_online(struct device *dev)
 {
 	struct cpu *cpu = container_of(dev, struct cpu, dev);
@@ -47,6 +52,8 @@ static int __ref cpu_subsys_online(struct device *dev)
 	int from_nid, to_nid;
 	int ret;
 
+	delay_hotplug();
+
 	from_nid = cpu_to_node(cpuid);
 	if (from_nid == NUMA_NO_NODE)
 		return -ENODEV;
@@ -65,6 +72,8 @@ static int __ref cpu_subsys_online(struct device *dev)
 
 static int cpu_subsys_offline(struct device *dev)
 {
+	delay_hotplug();
+
 	return cpu_down(dev->id);
 }
 
Signature · 1 line
1.9.0
#1 lore raw
Quoted 13 lines · On Wed, 7 May 2014 21:57:41 +0200 Borislav Petkov <[email protected]> wrote:

We have all those eager tester dudes which scratch up a dirty script to pound on CPU hotplug senselessly and then report bugs they've managed to trigger.

Well, first of all, most, if not all, bugs they trigger are CPU hotplug related anyway. But we know hotplug is full of duct tape and brown paper bags. So we end up clearly wasting too much time dealing with a mechanism we know it is b0rked in the first place.

Oh, and I would understand if that pounding were close to some real usage patterns but I've yet to receive a justification for toggling cores on- and offline senselessly.

Quoted 7 lines

In any case, before this gets rewritten properly (I'm being told we might get lucky after all) let's slow down hotplugging on purpose and thus make it uninteresting, as a temporary brown paper bag solution until the real thing gets done.

This way we'll save us a lot of time and efforts in chasing the wrong bugs.

Well, I only yesterday merged Srivatsa's `CPU hotplug, stop-machine:
plug race-window that leads to "IPI-to-offline-CPU"' bugfix. That bug presumably wouldn't have been fixed if this patch was in place.

#2 lore raw
Quoted 20 lines · Andrew Morton
Quoted 18 lines · On Wed, 7 May 2014 21:57:41 +0200 Borislav Petkov <[email protected]> wrote:

We have all those eager tester dudes which scratch up a dirty script to pound on CPU hotplug senselessly and then report bugs they've managed to trigger.

Well, first of all, most, if not all, bugs they trigger are CPU hotplug related anyway. But we know hotplug is full of duct tape and brown paper bags. So we end up clearly wasting too much time dealing with a mechanism we know it is b0rked in the first place.

Oh, and I would understand if that pounding were close to some real usage patterns but I've yet to receive a justification for toggling cores on- and offline senselessly.

In any case, before this gets rewritten properly (I'm being told we might get lucky after all) let's slow down hotplugging on purpose and thus make it uninteresting, as a temporary brown paper bag solution until the real thing gets done.

Quoted 6 lines
Quoted 2 lines

This way we'll save us a lot of time and efforts in chasing the wrong bugs.

Well, I only yesterday merged Srivatsa's `CPU hotplug, stop-machine:
plug race-window that leads to "IPI-to-offline-CPU"' bugfix. That bug presumably wouldn't have been fixed if this patch was in place.

True.

OTOH, if people would have spent the same amount of time to rewrite the hotplug mess, we would have a way bigger benefit. But no, we prefer to add more layers of duct tape and bandaid hackery to it.

I tried a redesign and run out of cycles, but the patches are out there and none of the folks who promised to complete them ever delivered. If nothing fundamental changes, I'm going to spend some serious time on it in the next couple of month.

Thanks,

tglx

#3 lore raw
Quoted 30 lines · Thomas Gleixner
Quoted 20 lines · Andrew Morton
Quoted 18 lines · On Wed, 7 May 2014 21:57:41 +0200 Borislav Petkov <[email protected]> wrote:

We have all those eager tester dudes which scratch up a dirty script to pound on CPU hotplug senselessly and then report bugs they've managed to trigger.

Well, first of all, most, if not all, bugs they trigger are CPU hotplug related anyway. But we know hotplug is full of duct tape and brown paper bags. So we end up clearly wasting too much time dealing with a mechanism we know it is b0rked in the first place.

Oh, and I would understand if that pounding were close to some real usage patterns but I've yet to receive a justification for toggling cores on- and offline senselessly.

In any case, before this gets rewritten properly (I'm being told we might get lucky after all) let's slow down hotplugging on purpose and thus make it uninteresting, as a temporary brown paper bag solution until the real thing gets done.

Quoted 6 lines
Quoted 2 lines

This way we'll save us a lot of time and efforts in chasing the wrong bugs.

Well, I only yesterday merged Srivatsa's `CPU hotplug, stop-machine:
plug race-window that leads to "IPI-to-offline-CPU"' bugfix. That bug presumably wouldn't have been fixed if this patch was in place.

True.

Quoted 8 lines

OTOH, if people would have spent the same amount of time to rewrite the hotplug mess, we would have a way bigger benefit. But no, we prefer to add more layers of duct tape and bandaid hackery to it.

I tried a redesign and run out of cycles, but the patches are out there and none of the folks who promised to complete them ever delivered. If nothing fundamental changes, I'm going to spend some serious time on it in the next couple of month.

... and in the interim, we could slow down the duct tape and bandaid hackery until it gets rewritten properly. The ever increasing in complexity bugfixes say exactly that - it needs a long hard look and a rewrite.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#4 lore raw
Quoted 40 lines · Borislav Petkov
Quoted 30 lines · Thomas Gleixner
Quoted 20 lines · Andrew Morton
Quoted 18 lines · On Wed, 7 May 2014 21:57:41 +0200 Borislav Petkov <[email protected]> wrote:

We have all those eager tester dudes which scratch up a dirty script to pound on CPU hotplug senselessly and then report bugs they've managed to trigger.

Well, first of all, most, if not all, bugs they trigger are CPU hotplug related anyway. But we know hotplug is full of duct tape and brown paper bags. So we end up clearly wasting too much time dealing with a mechanism we know it is b0rked in the first place.

Oh, and I would understand if that pounding were close to some real usage patterns but I've yet to receive a justification for toggling cores on- and offline senselessly.

In any case, before this gets rewritten properly (I'm being told we might get lucky after all) let's slow down hotplugging on purpose and thus make it uninteresting, as a temporary brown paper bag solution until the real thing gets done.

Quoted 6 lines
Quoted 2 lines

This way we'll save us a lot of time and efforts in chasing the wrong bugs.

Well, I only yesterday merged Srivatsa's `CPU hotplug, stop-machine:
plug race-window that leads to "IPI-to-offline-CPU"' bugfix. That bug presumably wouldn't have been fixed if this patch was in place.

True.

Quoted 8 lines

OTOH, if people would have spent the same amount of time to rewrite the hotplug mess, we would have a way bigger benefit. But no, we prefer to add more layers of duct tape and bandaid hackery to it.

I tried a redesign and run out of cycles, but the patches are out there and none of the folks who promised to complete them ever delivered. If nothing fundamental changes, I'm going to spend some serious time on it in the next couple of month.

Quoted 4 lines

... and in the interim, we could slow down the duct tape and bandaid hackery until it gets rewritten properly. The ever increasing in complexity bugfixes say exactly that - it needs a long hard look and a rewrite.

Well. If you add the delay, you'll mask real problems and cause regressions when the delay is removed -- because fix-hotplug will probably take time to get right.

Bad idea, AFAICT.
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#5 lore raw
Quoted 4 lines · Pavel Machek

Well. If you add the delay, you'll mask real problems and cause regressions when the delay is removed -- because fix-hotplug will probably take time to get right.

Bad idea, AFAICT.

Agreed, but Boris is right, that the current duct tape hackery needs to stop. And the delay was the desperate attempt to wake up people to focus on replacing the current mess instead of making it more entangled.

Thanks,

tglx

#6 lore raw
Quoted 10 lines · Thomas Gleixner
Quoted 4 lines · Pavel Machek

Well. If you add the delay, you'll mask real problems and cause regressions when the delay is removed -- because fix-hotplug will probably take time to get right.

Bad idea, AFAICT.

Agreed, but Boris is right, that the current duct tape hackery needs to stop. And the delay was the desperate attempt to wake up people to focus on replacing the current mess instead of making it more entangled.

I think Pavel is missing the point: the delay will be removed with the rewrite of cpu hotplug, after it actually works reliably. Then we won't need delay anyway.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#7 lore raw
Quoted 7 lines · Borislav Petkov
Quoted 6 lines · Thomas Gleixner
Quoted 4 lines · Pavel Machek

Well. If you add the delay, you'll mask real problems and cause regressions when the delay is removed -- because fix-hotplug will probably take time to get right.

Bad idea, AFAICT.

Quoted 8 lines
Quoted 4 lines

Agreed, but Boris is right, that the current duct tape hackery needs to stop. And the delay was the desperate attempt to wake up people to focus on replacing the current mess instead of making it more entangled.

I think Pavel is missing the point: the delay will be removed with the rewrite of cpu hotplug, after it actually works reliably. Then we won't need delay anyway.

Well.. and if you don't get it right at the first try, you'll get regression and bisect pointing at delay removal. Also... with delay in place, you won't notice if it gets less reliable.

printk() or comment should deter people from filling bug reports....

Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#8 lore raw
In reply to Thomas Gleixner
Quoted 30 lines · Thomas Gleixner
Quoted 20 lines · Andrew Morton
Quoted 18 lines · On Wed, 7 May 2014 21:57:41 +0200 Borislav Petkov <[email protected]> wrote:

We have all those eager tester dudes which scratch up a dirty script to pound on CPU hotplug senselessly and then report bugs they've managed to trigger.

Well, first of all, most, if not all, bugs they trigger are CPU hotplug related anyway. But we know hotplug is full of duct tape and brown paper bags. So we end up clearly wasting too much time dealing with a mechanism we know it is b0rked in the first place.

Oh, and I would understand if that pounding were close to some real usage patterns but I've yet to receive a justification for toggling cores on- and offline senselessly.

In any case, before this gets rewritten properly (I'm being told we might get lucky after all) let's slow down hotplugging on purpose and thus make it uninteresting, as a temporary brown paper bag solution until the real thing gets done.

Quoted 6 lines
Quoted 2 lines

This way we'll save us a lot of time and efforts in chasing the wrong bugs.

Well, I only yesterday merged Srivatsa's `CPU hotplug, stop-machine:
plug race-window that leads to "IPI-to-offline-CPU"' bugfix. That bug presumably wouldn't have been fixed if this patch was in place.

True.

Quoted 9 lines

OTOH, if people would have spent the same amount of time to rewrite the hotplug mess, we would have a way bigger benefit. But no, we prefer to add more layers of duct tape and bandaid hackery to it.

I tried a redesign and run out of cycles, but the patches are out there and none of the folks who promised to complete them ever delivered. If nothing fundamental changes, I'm going to spend some serious time on it in the next couple of month.

Yeah, that's quite unfortunate. Even several of my own attempts to try and fix some of the chronic issues of CPU hotplug (such as the removal of CPU hotplug's dependency on stop-machine, consolidation of all the duplicated and buggy CPU hotplug code in various architectures etc.) all met a similar fate. Initially there was some amount of consensus on these patchsets and designs, but eventually they got nowhere due to lack of any further feedback or signs of upstream acceptance.

Stop-machine()-free CPU hotplug, v6:
http://lwn.net/Articles/538819/

With performance improvements:
http://article.gmane.org/gmane.linux.kernel/1435249

Attempt to upstream that patchset in parts, v3:
http://lwn.net/Articles/556727/

Generic SMP boot/cpu-hotplug framework to consolidate arch/ code:
https://lwn.net/Articles/500185/

But, luckily the recent work to fix the notifier deadlock mess actually went upstream, fairly quickly. So we have one less CPU hotplug problem to fix! :-)
https://lkml.org/lkml/2014/3/10/522

Regards,
Srivatsa S. Bhat