Skip to contentSkip to conversation

cpu hotplug: possible_cpus broken (again?) next-20130607

DISCUSSION Dave Hansen 5 participants 12 messages Jun 11, 2013 – Jun 13, 2013 Driver core

lore
Share & tools
lore
#0 lore raw

possible_cpus looks broken again. I'm booting with:

maxcpus=10 possible_cpus=160

But I only get 0-9 in sysfs:

Quoted 3 lines
# ls /sys/devices/system/cpu/
cpu0  cpu2  cpu4  cpu6  cpu8  cpufreq  kernel_max  offline  possible  probe    uevent
cpu1  cpu3  cpu5  cpu7  cpu9  cpuidle  modalias    online   present   release
#1 lore raw
Quoted 9 lines · Dave Hansen

possible_cpus looks broken again. I'm booting with:

maxcpus=10 possible_cpus=160

But I only get 0-9 in sysfs:

Quoted 3 lines
# ls /sys/devices/system/cpu/
cpu0  cpu2  cpu4  cpu6  cpu8  cpufreq  kernel_max  offline  possible  probe    uevent
cpu1  cpu3  cpu5  cpu7  cpu9  cpuidle  modalias    online   present   release

Can you please test the acpi-hotplug branch of the linux-pm.git tree?

Rafael

Signature · 2 lines
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.
#2 lore raw
Quoted 5 lines · Rafael J. Wysocki
Quoted 4 lines · Dave Hansen

possible_cpus looks broken again. I'm booting with:

maxcpus=10 possible_cpus=160

Quoted 7 lines
Quoted 5 lines

But I only get 0-9 in sysfs:

Quoted 3 lines
# ls /sys/devices/system/cpu/
cpu0  cpu2  cpu4  cpu6  cpu8  cpufreq  kernel_max  offline  possible  probe    uevent
cpu1  cpu3  cpu5  cpu7  cpu9  cpuidle  modalias    online   present   release

Can you please test the acpi-hotplug branch of the linux-pm.git tree?

That branch seems to work happily.

#3 lore raw
Quoted 8 lines · Dave Hansen
Quoted 7 lines · Rafael J. Wysocki
Quoted 6 lines · Dave Hansen

possible_cpus looks broken again. I'm booting with:

maxcpus=10 possible_cpus=160

But I only get 0-9 in sysfs:

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
# ls /sys/devices/system/cpu/
cpu0  cpu2  cpu4  cpu6  cpu8  cpufreq  kernel_max  offline  possible  probe    uevent
cpu1  cpu3  cpu5  cpu7  cpu9  cpuidle  modalias    online   present   release

Can you please test the acpi-hotplug branch of the linux-pm.git tree?

That branch seems to work happily.

In that case the problem may have been reintroduced by a merge conflict fix in linux-next.

Thanks,
Rafael

Signature · 2 lines
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.
#4 lore raw
Quoted 13 lines · Rafael J. Wysocki
Quoted 12 lines · Dave Hansen
Quoted 11 lines · Rafael J. Wysocki
Quoted 9 lines · Dave Hansen

possible_cpus looks broken again. I'm booting with:

maxcpus=10 possible_cpus=160

But I only get 0-9 in sysfs:

Quoted 3 lines
# ls /sys/devices/system/cpu/
cpu0  cpu2  cpu4  cpu6  cpu8  cpufreq  kernel_max  offline  possible  probe    uevent
cpu1  cpu3  cpu5  cpu7  cpu9  cpuidle  modalias    online   present   release
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Can you please test the acpi-hotplug branch of the linux-pm.git tree?

That branch seems to work happily.

In that case the problem may have been reintroduced by a merge conflict fix in linux-next.

I believe the problem was introduced by the following change. From the description, though, this is exactly what this patch was trying to change... Adding Youguan to the list.

commit 3e275a5ba367ab74b3a4e49114307baed989fcac

Author: Youquan Song <[email protected]>
Date:   Fri Jun 7 10:07:08 2013 +1000

drivers/base/cpu.c: fix maxcpus boot option

Thanks,
-Toshi

#5 lore raw
Quoted 25 lines · Toshi Kani
Quoted 13 lines · Rafael J. Wysocki
Quoted 12 lines · Dave Hansen
Quoted 11 lines · Rafael J. Wysocki
Quoted 9 lines · Dave Hansen

possible_cpus looks broken again. I'm booting with:

maxcpus=10 possible_cpus=160

But I only get 0-9 in sysfs:

Quoted 3 lines
# ls /sys/devices/system/cpu/
cpu0  cpu2  cpu4  cpu6  cpu8  cpufreq  kernel_max  offline  possible  probe    uevent
cpu1  cpu3  cpu5  cpu7  cpu9  cpuidle  modalias    online   present   release
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Can you please test the acpi-hotplug branch of the linux-pm.git tree?

That branch seems to work happily.

In that case the problem may have been reintroduced by a merge conflict fix in linux-next.

I believe the problem was introduced by the following change. From the description, though, this is exactly what this patch was trying to change... Adding Youguan to the list.

Quoted 6 lines

commit 3e275a5ba367ab74b3a4e49114307baed989fcac

Author: Youquan Song <[email protected]>
Date:   Fri Jun 7 10:07:08 2013 +1000

drivers/base/cpu.c: fix maxcpus boot option

Hi Toshi,

Thanks Thoshi for the information.
please try the below patch to fix the issue by moving the code to store_online.

drivers/base/cpu.c +2 -0
index 3d48fc8..2378f42 100644
@@ -60,6 +60,13 @@ static ssize_t __ref store_online(struct device *dev,
                        kobject_uevent(&dev->kobj, KOBJ_OFFLINE);
                break;
        case '1':
+#ifdef CONFIG_SMP
+              /* return when cpu number greater than maximum number of

CPUs */
+ if (setup_max_cpus <= num_online_cpus() + 1) {
+ cpu_hotplug_driver_unlock();
+ return -EINVAL;
+ }
+#endif

                from_nid = cpu_to_node(cpuid);
                ret = cpu_up(cpuid);

Thanks
-Youquan

#6 lore raw

Review

Quoted 10 lines · Youquan Song
drivers/base/cpu.c +2 -0
+#ifdef CONFIG_SMP
+              /* return when cpu number greater than maximum number of

CPUs */

(quoted diff) +5 -0
+               if (setup_max_cpus <= num_online_cpus() + 1) {
+                       cpu_hotplug_driver_unlock();
+                       return -EINVAL;
+               }
+#endif
                from_nid = cpu_to_node(cpuid);
                ret = cpu_up(cpuid);

Your patch is line-wrapped.

Also, the #ifdef is unnecessary. If CONFIG_SMP is off:

	static const unsigned int setup_max_cpus = NR_CPUS;
	#define num_online_cpus() 1U

The compiler will take care of optimizing out the the if() without the explicit #ifdef.

Also, the +1 looks goofy to me. Doesn't this do the same thing (and isn't it much easier to read)?

if (num_online_cpus() >= setup_max_cpus)

#7 lore raw
Quoted 19 lines · Youquan Song

Review

Quoted 10 lines
(quoted diff) +2 -0
+#ifdef CONFIG_SMP
+              /* return when cpu number greater than maximum number of

CPUs */

(quoted diff) +5 -0
+               if (setup_max_cpus <= num_online_cpus() + 1) {
+                       cpu_hotplug_driver_unlock();
+                       return -EINVAL;
+               }
+#endif
                from_nid = cpu_to_node(cpuid);
                ret = cpu_up(cpuid);

Your patch is line-wrapped.

Also, the #ifdef is unnecessary. If CONFIG_SMP is off:

	static const unsigned int setup_max_cpus = NR_CPUS;
	#define num_online_cpus() 1U
Quoted 8 lines

The compiler will take care of optimizing out the the if() without the explicit #ifdef.

Also, the +1 looks goofy to me. Doesn't this do the same thing (and isn't it much easier to read)?

if (num_online_cpus() >= setup_max_cpus)

Thanks. Here is a formal patch for it. please review and try.

Subject: [PATCH] core: Fix maxcpus boot option broken

maxcpus boot option to limit maximum number of CPUs on system, but this option is broken at recent kernel. Though we use maxcpus to limit CPUs number, but current kernel will register all of present CPUs in sysfs.
udev will enumerate all registered cpu at sysfs, and it will bring up the CPU if the CPU is offline. So the maxcpus option is broken.

This patch will limit the online cpus number not over limitation of maxcpus option. So it will keep the maxcpus limitation when udev enumeration or other intention of bring up CPUs over the limitation by method like echo 1 > /sys/devices/system/cpu/online

---

drivers/base/cpu.c +7 -0
index 3d48fc8..e32fffa 100644
@@ -60,6 +60,13 @@ static ssize_t __ref store_online(struct device *dev,
 			kobject_uevent(&dev->kobj, KOBJ_OFFLINE);
 		break;
 	case '1':
+	       /* Return when online cpu number equal or greater than
+		*  maximum number of CPUs */
+		if (num_online_cpus() >= setup_max_cpus) {
+			cpu_hotplug_driver_unlock();
+			return -EINVAL;
+		}
+
 		from_nid = cpu_to_node(cpuid);
 		ret = cpu_up(cpuid);
 
#8 lore raw
Quoted 38 lines · Youquan Song
Quoted 19 lines · Youquan Song

drivers/base/cpu.c static ssize_t __ref store_online(struct device *dev, · line 71

Quoted 10 lines
(quoted diff) +2 -0
+#ifdef CONFIG_SMP
+              /* return when cpu number greater than maximum number of

CPUs */

drivers/base/cpu.c +5 -0
+               if (setup_max_cpus <= num_online_cpus() + 1) {
+                       cpu_hotplug_driver_unlock();
+                       return -EINVAL;
+               }
+#endif
                from_nid = cpu_to_node(cpuid);
                ret = cpu_up(cpuid);

Your patch is line-wrapped.

Also, the #ifdef is unnecessary. If CONFIG_SMP is off:

      static const unsigned int setup_max_cpus = NR_CPUS;
      #define num_online_cpus() 1U
Quoted 8 lines

The compiler will take care of optimizing out the the if() without the explicit #ifdef.

Also, the +1 looks goofy to me. Doesn't this do the same thing (and isn't it much easier to read)?

if (num_online_cpus() >= setup_max_cpus)

Thanks. Here is a formal patch for it. please review and try.

Subject: [PATCH] core: Fix maxcpus boot option broken

maxcpus boot option to limit maximum number of CPUs on system, but this option is broken at recent kernel. Though we use maxcpus to limit CPUs number, but current kernel will register all of present CPUs in sysfs.
udev will enumerate all registered cpu at sysfs, and it will bring up the CPU if the CPU is offline. So the maxcpus option is broken.

Quoted 4 lines

This patch will limit the online cpus number not over limitation of maxcpus option. So it will keep the maxcpus limitation when udev enumeration or other intention of bring up CPUs over the limitation by method like echo 1 > /sys/devices/system/cpu/online

Interesting, you are changing long standing meaning of maxcpus=

We always use maxcpus=1 to have one cpu up, and later in user space to online other cpus like
echo 1 > /sys/devices/system/cpuX/online.

aka maxcpus= is a soft limit or initial online nr.

we already have nr_cpus= for hard limit.

So need to drop
commit 3e275a5ba367ab74b3a4e49114307baed989fcac
Author: Youquan Song <[email protected]>
Date: Fri Jun 7 10:07:08 2013 +1000

drivers/base/cpu.c: fix maxcpus boot option

Greg,
Can you drop that 3e275a5ba36 from your drivers/core tree ?

Thanks

Yinghai

#9 lore raw
Quoted 53 lines · Yinghai Lu
Quoted 38 lines · Youquan Song
Quoted 19 lines · Youquan Song

drivers/base/cpu.c static ssize_t __ref store_online(struct device *dev, · line 71

Quoted 10 lines
(quoted diff) +2 -0
+#ifdef CONFIG_SMP
+              /* return when cpu number greater than maximum number of

CPUs */

drivers/base/cpu.c +5 -0
+               if (setup_max_cpus <= num_online_cpus() + 1) {
+                       cpu_hotplug_driver_unlock();
+                       return -EINVAL;
+               }
+#endif
                from_nid = cpu_to_node(cpuid);
                ret = cpu_up(cpuid);

Your patch is line-wrapped.

Also, the #ifdef is unnecessary. If CONFIG_SMP is off:

      static const unsigned int setup_max_cpus = NR_CPUS;
      #define num_online_cpus() 1U
Quoted 8 lines

The compiler will take care of optimizing out the the if() without the explicit #ifdef.

Also, the +1 looks goofy to me. Doesn't this do the same thing (and isn't it much easier to read)?

if (num_online_cpus() >= setup_max_cpus)

Thanks. Here is a formal patch for it. please review and try.

Subject: [PATCH] core: Fix maxcpus boot option broken

maxcpus boot option to limit maximum number of CPUs on system, but this option is broken at recent kernel. Though we use maxcpus to limit CPUs number, but current kernel will register all of present CPUs in sysfs.
udev will enumerate all registered cpu at sysfs, and it will bring up the CPU if the CPU is offline. So the maxcpus option is broken.

Quoted 4 lines

This patch will limit the online cpus number not over limitation of maxcpus option. So it will keep the maxcpus limitation when udev enumeration or other intention of bring up CPUs over the limitation by method like echo 1 > /sys/devices/system/cpu/online

Interesting, you are changing long standing meaning of maxcpus=

We always use maxcpus=1 to have one cpu up, and later in user space to online other cpus like
echo 1 > /sys/devices/system/cpuX/online.

aka maxcpus= is a soft limit or initial online nr.

Quoted 8 lines

we already have nr_cpus= for hard limit.

So need to drop
commit 3e275a5ba367ab74b3a4e49114307baed989fcac
Author: Youquan Song <[email protected]>
Date: Fri Jun 7 10:07:08 2013 +1000

drivers/base/cpu.c: fix maxcpus boot option

Agreed.

Thanks,
Rafael

Quoted 6 lines

Greg,
Can you drop that 3e275a5ba36 from your drivers/core tree ?

Thanks

Yinghai

Signature · 2 lines
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.
#10 lore raw
Quoted 10 lines
Quoted 10 lines

Interesting, you are changing long standing meaning of maxcpus=

We always use maxcpus=1 to have one cpu up, and later in user space to online other cpus like
echo 1 > /sys/devices/system/cpuX/online.

aka maxcpus= is a soft limit or initial online nr.

we already have nr_cpus= for hard limit.

Quoted 8 lines
Quoted 6 lines

So need to drop
commit 3e275a5ba367ab74b3a4e49114307baed989fcac
Author: Youquan Song <[email protected]>
Date: Fri Jun 7 10:07:08 2013 +1000

drivers/base/cpu.c: fix maxcpus boot option

Agreed.

Yes. I also agree to drop it and the fix need more consideration. I try use maxcpus to limit cpu number to debug a well known applition because it fail to run when cpu number is larger to > 69. When I use maxcpus at to limit the boot CPUs number, but udev will enable all of the CPUs at 3.10 kernel automatically.
I also try maxcpus at 3.0 kernel, it does not show the maxcpus issue. I have digged out recently, it is the commit at 3.2 kernel 8a25a2fd126c621f44f3aeaef80d51f00fc11639 "cpu: convert 'cpu' and 'machinecheck' sysdev_class to a regular subsystem" result in udev automatically enable all of CPUs though maxcpus has been provided.

So the next, I need look at udev try to enable all of CPUs though maxcpus provided. Possibly, it can also fix it in udev daemon.

Secondly, I think that the maxcpus= option description is too confused in Documentation/kernel-parameters.txt. The maxcpus and nr_cpus option need switch their name.
Currently:

maxcpus= [SMP] Maximum number of processors that an SMP kernel
should make use of. maxcpus=n : n >= 0 limits the kernel to using 'n' processors. n=0 is a special case, it is equivalent to "nosmp", which also disables the IO APIC.

How about change to

maxcpus= [SMP] Maximum number of processors that an SMP kernel
bring up during booting. maxcpus=n : n >= 0 limits the kernel to using 'n' processors. n=0 is a special case, it is equivalent to "nosmp", which also disables the IO APIC.

Thanks
-Youquan

#11 lore raw
Quoted 15 lines · Youquan Song
Quoted 15 lines
Quoted 15 lines

Interesting, you are changing long standing meaning of maxcpus=

We always use maxcpus=1 to have one cpu up, and later in user space to online other cpus like
echo 1 > /sys/devices/system/cpuX/online.

aka maxcpus= is a soft limit or initial online nr.

we already have nr_cpus= for hard limit.

So need to drop
commit 3e275a5ba367ab74b3a4e49114307baed989fcac
Author: Youquan Song <[email protected]>
Date: Fri Jun 7 10:07:08 2013 +1000

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

drivers/base/cpu.c: fix maxcpus boot option

Agreed.

Yes. I also agree to drop it and the fix need more consideration.

Agreed.

Quoted 5 lines

I try use maxcpus to limit cpu number to debug a well known applition because it fail to run when cpu number is larger to > 69. When I use maxcpus at to limit the boot CPUs number, but udev will enable all of the CPUs at 3.10 kernel automatically.
I also try maxcpus at 3.0 kernel, it does not show the maxcpus issue.

I tested Linus's 3.10-rc5 kernel on Fedora 18, but I did not see such behavior. Could this be related with your environment?

Quoted 11 lines

I have digged out recently, it is the commit at 3.2 kernel 8a25a2fd126c621f44f3aeaef80d51f00fc11639 "cpu: convert 'cpu' and 'machinecheck' sysdev_class to a regular subsystem" result in udev automatically enable all of CPUs though maxcpus has been provided.

So the next, I need look at udev try to enable all of CPUs though maxcpus provided. Possibly, it can also fix it in udev daemon.

Secondly, I think that the maxcpus= option description is too confused in Documentation/kernel-parameters.txt. The maxcpus and nr_cpus option need switch their name.

I am fine with the clarification.

Quoted 8 lines

Currently:

maxcpus= [SMP] Maximum number of processors that an SMP kernel
should make use of. maxcpus=n : n >= 0 limits the kernel to using 'n' processors. n=0 is a special case, it is equivalent to "nosmp", which also disables the IO APIC.

Quoted 4 lines

How about change to

maxcpus= [SMP] Maximum number of processors that an SMP kernel
bring up during booting. maxcpus=n : n >= 0 limits the

bring -> brings

Quoted 3 lines

kernel to using 'n' processors. n=0 is a special case, it is equivalent to "nosmp", which also disables the IO APIC.

Thanks,
-Toshi