x86: fix cpu-hotplug regression

PATCH In review Andreas Herrmann 4 participants 7 messages Nov 7, 2007 – Nov 12, 2007 x86

lore

Share & tools

lore

Andreas Herrmann

author

2007-11-07 01:20 UTC #0 lore raw

PATCH 1/1 x86: fix cpu-hotplug regression

[PATCH] x86: fix cpu-hotplug regression

Commit d435d862baca3e25e5eec236762a43251b1e7ffc ("cpu hotplug: mce: fix cpu hotplug error handling") changed the error handling in mce_cpu_callback.

In cases where not all CPUs are brought up during boot (e.g. using maxcpus and additional_cpus parameters) mce_cpu_callback now returns NOTFIY_BAD because for such CPUs cpu_data is not completely filled when the notifier is called. Thus mce_create_device fails right at its beginning:

        if (!mce_available(&cpu_data[cpu]))
                return -EIO;

As a quick fix I suggest to check boot_cpu_data for MCE.

To reproduce this regression:

(1) boot with maxcpus=2 addtional_cpus=2 on a 4 CPU x86-64 system
(2) # echo 1 >/sys/devices/system/cpu/cpu2/online -bash: echo: write error: Invalid argument

dmesg shows:

_cpu_up: attempt to bring up CPU 2 failed

Signed-off-by Andreas Herrmann <[email protected]>

---
 arch/x86/kernel/cpu/mcheck/mce_64.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

arch/x86/kernel/cpu/mcheck/mce_64.c +1 -1

index b9f802e..5112a70 100644
@@ -808,7 +808,7 @@ static __cpuinit int mce_create_device(unsigned int cpu)
 	int err;
 	int i;
 
-	if (!mce_available(&cpu_data(cpu)))
+	if (!mce_available(&boot_cpu_data))
 		return -EIO;
 
 	memset(&per_cpu(device_mce, cpu).kobj, 0, sizeof(struct kobject));

Signature · 1 line

1.5.3.4

Andi Kleen

2007-11-07 02:39 UTC #1 lore raw

Quoted 11 lines · Andreas Herrmann

In cases where not all CPUs are brought up during boot (e.g. using maxcpus and additional_cpus parameters) mce_cpu_callback now returns NOTFIY_BAD because for such CPUs cpu_data is not completely filled when the notifier is called. Thus mce_create_device fails right at its beginning:
        if (!mce_available(&cpu_data[cpu]))
                return -EIO;
As a quick fix I suggest to check boot_cpu_data for MCE.

I guess it would be better to just move the device creation to after the CPU has booted. AKA call mce_create_dev() on CPU_ONLINE instead.

-Andi

Andreas Herrmann

author

2007-11-07 13:18 UTC #2 lore raw

Quoted 9 lines · Andi Kleen

Quoted 7 lines · Andreas Herrmann
In cases where not all CPUs are brought up during boot (e.g. using maxcpus and additional_cpus parameters) mce_cpu_callback now returns NOTFIY_BAD because for such CPUs cpu_data is not completely filled when the notifier is called. Thus mce_create_device fails right at its beginning:

Quoted 8 lines

Quoted 4 lines
        if (!mce_available(&cpu_data[cpu]))
                return -EIO;
As a quick fix I suggest to check boot_cpu_data for MCE.
I guess it would be better to just move the device creation to after the CPU has booted. AKA call mce_create_dev() on CPU_ONLINE instead.

Yes, and this was the old behaviour. The mentioned patch changed it - ("do mce_create_device in CPU_UP_PREPARE instead of CPU_ONLINE").

Thinking twice about the problem it seems obvious that this part of the patch should just be reverted.

Attached is a new fix (diff against 2.6.24-rc2).

Regards,

Andreas

--
[PATCH] x86: fix cpu hotplug regression (don't call mce_create_device on CPU_UP_PREPARE)

Fix regression introduced with d435d862baca3e25e5eec236762a43251b1e7ffc ("cpu hotplug: mce: fix cpu hotplug error handling").

For CPUs not brought up during boot (using maxcpus and additional_cpus parameters) we don't know whether mce is supported or not at "CPU_UP_PREPARE"-time. Thus mce_cpu_callback should be called after the CPU is online.

Signed-off-by Andreas Herrmann <[email protected]>

---
 arch/x86/kernel/cpu/mcheck/mce_64.c |    6 ++----
 1 files changed, 2 insertions(+), 4 deletions(-)

arch/x86/kernel/cpu/mcheck/mce_64.c +2 -4

index b9f802e..8e83070 100644
@@ -855,12 +855,10 @@ mce_cpu_callback(struct notifier_block *nfb, unsigned long action, void *hcpu)
 	int err = 0;
 
 	switch (action) {
-	case CPU_UP_PREPARE:
-	case CPU_UP_PREPARE_FROZEN:
+	case CPU_ONLINE:
+	case CPU_ONLINE_FROZEN:
 		err = mce_create_device(cpu);
 		break;
-	case CPU_UP_CANCELED:
-	case CPU_UP_CANCELED_FROZEN:
 	case CPU_DEAD:
 	case CPU_DEAD_FROZEN:
 		mce_remove_device(cpu);

Signature · 1 line

1.5.3.4

Akinobu Mita

2007-11-07 14:15 UTC #3 lore raw

Quoted 8 lines

[PATCH] x86: fix cpu hotplug regression (don't call mce_create_device on CPU_UP_PREPARE)
Fix regression introduced with d435d862baca3e25e5eec236762a43251b1e7ffc ("cpu hotplug: mce: fix cpu hotplug error handling").
For CPUs not brought up during boot (using maxcpus and additional_cpus parameters) we don't know whether mce is supported or not at "CPU_UP_PREPARE"-time. Thus mce_cpu_callback should be called after the CPU is online.

Thank you for finding and fixing the problem.
I added two fixes to your patch:

- Avoid mce_remove_device() for the CPU that is not correctly initialized by mce_create_device() failure.

- make CPU_ONLINE callback always return NOTIFY_OK. Because CPU_ONLINE callback return value is always ignored.

Quoted 1 line

Signed-off-by Andreas Herrmann <[email protected]>

[[email protected]: make CPU_ONLINE callback always return NOTIFY_OK] [[email protected]: avoid mce_remove_device() for not initialized device]

Signed-off-by Akinobu Mita <[email protected]>

---
 arch/x86/kernel/cpu/mcheck/mce_64.c |   18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

arch/x86/kernel/cpu/mcheck/mce_64.c +11 -7

Index: 2.6-git/arch/x86/kernel/cpu/mcheck/mce_64.c
===================================================================
@@ -802,6 +802,8 @@ static struct sysdev_attribute *mce_attr
 	NULL
 };
 
+static cpumask_t mce_device_initialized = CPU_MASK_NONE;
+
 /* Per cpu sysdev init.  All of the cpus still share the same ctl bank */
 static __cpuinit int mce_create_device(unsigned int cpu)
 {

@@ -825,6 +827,7 @@ static __cpuinit int mce_create_device(u
 		if (err)
 			goto error;
 	}
+	cpu_set(cpu, mce_device_initialized);
 
 	return 0;
 error:

@@ -841,10 +844,14 @@ static void mce_remove_device(unsigned i
 {
 	int i;
 
+	if (!cpu_isset(cpu, mce_device_initialized))
+		return;
+
 	for (i = 0; mce_attributes[i]; i++)
 		sysdev_remove_file(&per_cpu(device_mce,cpu),
 			mce_attributes[i]);
 	sysdev_unregister(&per_cpu(device_mce,cpu));
+	cpu_clear(cpu, mce_device_initialized);
 }
 
 /* Get notified when a cpu comes on/off. Be hotplug friendly. */

@@ -852,21 +859,18 @@ static int
 mce_cpu_callback(struct notifier_block *nfb, unsigned long action, void *hcpu)
 {
 	unsigned int cpu = (unsigned long)hcpu;
-	int err = 0;
 
 	switch (action) {
-	case CPU_UP_PREPARE:
-	case CPU_UP_PREPARE_FROZEN:
-		err = mce_create_device(cpu);
+	case CPU_ONLINE:
+	case CPU_ONLINE_FROZEN:
+		mce_create_device(cpu);
 		break;
-	case CPU_UP_CANCELED:
-	case CPU_UP_CANCELED_FROZEN:
 	case CPU_DEAD:
 	case CPU_DEAD_FROZEN:
 		mce_remove_device(cpu);
 		break;
 	}
-	return err ? NOTIFY_BAD : NOTIFY_OK;
+	return NOTIFY_OK;
 }
 
 static struct notifier_block mce_cpu_notifier = {

Andrew Morton

2007-11-09 20:17 UTC #4 lore raw

Quoted 18 lines · On Wed, 7 Nov 2007 23:15:38 +0900 "Akinobu Mita" <[email protected]> wrote:

Quoted 8 lines
[PATCH] x86: fix cpu hotplug regression (don't call mce_create_device on CPU_UP_PREPARE)
Fix regression introduced with d435d862baca3e25e5eec236762a43251b1e7ffc ("cpu hotplug: mce: fix cpu hotplug error handling").
For CPUs not brought up during boot (using maxcpus and additional_cpus parameters) we don't know whether mce is supported or not at "CPU_UP_PREPARE"-time. Thus mce_cpu_callback should be called after the CPU is online.
Thank you for finding and fixing the problem.
I added two fixes to your patch:
- Avoid mce_remove_device() for the CPU that is not correctly initialized by mce_create_device() failure.
- make CPU_ONLINE callback always return NOTIFY_OK. Because CPU_ONLINE callback return value is always ignored.

Quoted 6 lines

Quoted 1 line
Signed-off-by Andreas Herrmann <[email protected]>
[[email protected]: make CPU_ONLINE callback always return NOTIFY_OK] [[email protected]: avoid mce_remove_device() for not initialized device]
Signed-off-by Akinobu Mita <[email protected]>

Andreas, could you please review and preferably runtime-test this new version?

Assuming that all goes well could one of you please prepare a final patch with a complete changelog?

Thanks.

Andreas Herrmann

author

2007-11-12 10:08 UTC #5 lore raw

Quoted 23 lines · Andrew Morton

Quoted 20 lines · On Wed, 7 Nov 2007 23:15:38 +0900 "Akinobu Mita" <[email protected]> wrote:
Quoted 8 lines
[PATCH] x86: fix cpu hotplug regression (don't call mce_create_device on CPU_UP_PREPARE)
Fix regression introduced with d435d862baca3e25e5eec236762a43251b1e7ffc ("cpu hotplug: mce: fix cpu hotplug error handling").
For CPUs not brought up during boot (using maxcpus and additional_cpus parameters) we don't know whether mce is supported or not at "CPU_UP_PREPARE"-time. Thus mce_cpu_callback should be called after the CPU is online.
Thank you for finding and fixing the problem.
I added two fixes to your patch:
- Avoid mce_remove_device() for the CPU that is not correctly initialized by mce_create_device() failure.
- make CPU_ONLINE callback always return NOTIFY_OK. Because CPU_ONLINE callback return value is always ignored.
Quoted 1 line
Signed-off-by Andreas Herrmann <[email protected]>

Quoted 7 lines

Quoted 4 lines
[[email protected]: make CPU_ONLINE callback always return NOTIFY_OK] [[email protected]: avoid mce_remove_device() for not initialized device]
Signed-off-by Akinobu Mita <[email protected]>
Andreas, could you please review and preferably runtime-test this new version?

I'll test it today ...

Quoted 2 lines

Assuming that all goes well could one of you please prepare a final patch with a complete changelog?

... and can send a final patch after successful testing.

Regards,

Andreas

Signature · 5 lines

Operating | AMD Saxony Limited Liability Company & Co. KG,
  System  | Wilschdorfer Landstr. 101, 01109 Dresden, Germany
 Research | Register Court Dresden: HRA 4896, General Partner authorized
  Center  | to represent: AMD Saxony LLC (Wilmington, Delaware, US)
  (OSRC)  | General Manager of AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy

Andreas Herrmann

author

2007-11-12 17:38 UTC #6 lore raw

PATCH 1/1 x86: don't call mce_create_device on CPU_UP_PREPARE

In reply to Andrew Morton

Quoted 2 lines · Andrew Morton

Andreas, could you please review and preferably runtime-test this new version?

Reviewed and tested.
Akinobu's changes are necessary as well.
Test was ok.

Quoted 2 lines

Assuming that all goes well could one of you please prepare a final patch with a complete changelog?

Attached is a new patch with a "combined changelog". Patch is against v2.6.24-rc2-247-g6e800af.

Regards,

Andreas
--

x86: don't call mce_create_device on CPU_UP_PREPARE

Fix regression introduced with d435d862baca3e25e5eec236762a43251b1e7ffc ("cpu hotplug: mce: fix cpu hotplug error handling").

A CPU which was not brought up during boot (using maxcpus and additional_cpus parameters) couldn't be onlined anymore. For such a CPU it seemed that MCE was not supported during CPU_UP_PREPARE-time which caused mce_cpu_callback to return NOTIFY_BAD to notifier_call_chain.
To fix this we:

- call mce_create_device for CPU_ONLINE event (instead of CPU_UP_PREPARE),
- avoid mce_remove_device() for the CPU that is not correctly initialized by mce_create_device() failure,
- make mce_cpu_callback always return NOTIFY_OK for CPU_ONLINE event. Because CPU_ONLINE callback return value is always ignored.

[[email protected]: avoid mce_remove_device() for not initialized device]

[[email protected]: make mce_cpu_callback always return NOTIFY_OK]

Signed-off-by Akinobu Mita <[email protected]>

Signed-off-by Andreas Herrmann <[email protected]>

---
 arch/x86/kernel/cpu/mcheck/mce_64.c |   18 +++++++++++-------
 1 files changed, 11 insertions(+), 7 deletions(-)

arch/x86/kernel/cpu/mcheck/mce_64.c +11 -7

index b9f802e..447b351 100644
@@ -802,6 +802,8 @@ static struct sysdev_attribute *mce_attributes[] = {
 	NULL
 };
 
+static cpumask_t mce_device_initialized = CPU_MASK_NONE;
+
 /* Per cpu sysdev init.  All of the cpus still share the same ctl bank */
 static __cpuinit int mce_create_device(unsigned int cpu)
 {

@@ -825,6 +827,7 @@ static __cpuinit int mce_create_device(unsigned int cpu)
 		if (err)
 			goto error;
 	}
+	cpu_set(cpu, mce_device_initialized);
 
 	return 0;
 error:

@@ -841,10 +844,14 @@ static void mce_remove_device(unsigned int cpu)
 {
 	int i;
 
+	if (!cpu_isset(cpu, mce_device_initialized))
+		return;
+
 	for (i = 0; mce_attributes[i]; i++)
 		sysdev_remove_file(&per_cpu(device_mce,cpu),
 			mce_attributes[i]);
 	sysdev_unregister(&per_cpu(device_mce,cpu));
+	cpu_clear(cpu, mce_device_initialized);
 }
 
 /* Get notified when a cpu comes on/off. Be hotplug friendly. */

@@ -852,21 +859,18 @@ static int
 mce_cpu_callback(struct notifier_block *nfb, unsigned long action, void *hcpu)
 {
 	unsigned int cpu = (unsigned long)hcpu;
-	int err = 0;
 
 	switch (action) {
-	case CPU_UP_PREPARE:
-	case CPU_UP_PREPARE_FROZEN:
-		err = mce_create_device(cpu);
+	case CPU_ONLINE:
+	case CPU_ONLINE_FROZEN:
+		mce_create_device(cpu);
 		break;
-	case CPU_UP_CANCELED:
-	case CPU_UP_CANCELED_FROZEN:
 	case CPU_DEAD:
 	case CPU_DEAD_FROZEN:
 		mce_remove_device(cpu);
 		break;
 	}
-	return err ? NOTIFY_BAD : NOTIFY_OK;
+	return NOTIFY_OK;
 }
 
 static struct notifier_block mce_cpu_notifier = {

Signature · 1 line

1.5.3.4