smp: Add bootcpus parameter to boot subset of CPUs

In a heterogeneous multiprocessor system, specifying the 'maxcpus' parameter on kernel command line does not provide sufficient control over which CPUs are brought online at kernel boot time, since CPUs may have nonuniform performance characteristics. Thus, add bootcpus kernel parameter to control which CPUs should be brought online during kernel boot. When both maxcpus and bootcpus is set, the more restrictive of the two are booted.

In a heterogeneous multiprocessor system, specifying the 'maxcpus' parameter on kernel command line does not provide sufficient control over which CPUs are brought online at kernel boot time, since CPUs may have nonuniform performance characteristics. Thus, add bootcpus kernel parameter to control which CPUs should be brought online during kernel boot. When both maxcpus and bootcpus is set, the more restrictive of the two are booted.

Quoted 7 lines · Elliot Berman

In a heterogeneous multiprocessor system, specifying the 'maxcpus' parameter on kernel command line does not provide sufficient control over which CPUs are brought online at kernel boot time, since CPUs may have nonuniform performance characteristics. Thus, add bootcpus kernel parameter to control which CPUs should be brought online during kernel boot. When both maxcpus and bootcpus is set, the more restrictive of the two are booted.

What for? 'maxcpus' is a debug hack at best and outright dangerous on certain architectures. Why do we need more of that? Just let the machine boot and offline the CPUs from user space.

Thanks,

tglx

Quoted 10 lines · Thomas Gleixner

Quoted 8 lines · Elliot Berman

In a heterogeneous multiprocessor system, specifying the 'maxcpus' parameter on kernel command line does not provide sufficient control over which CPUs are brought online at kernel boot time, since CPUs may have nonuniform performance characteristics. Thus, add bootcpus kernel parameter to control which CPUs should be brought online during kernel boot. When both maxcpus and bootcpus is set, the more restrictive of the
two are booted.

Quoted 3 lines

What for? 'maxcpus' is a debug hack at best and outright dangerous on certain architectures. Why do we need more of that? Just let the machine boot and offline the CPUs from user space.

Hi Thomas and Peter,

Based on my understanding with maxcpus option provides, maximum no of CPUs are brough up during the device boot up. There is a different case, in which we want to restrict which CPUs to be brough up.
On a system with 8 cpus, if we set maxcpus as 3, cpu0, cpu1, and cpu2 are brough up during the bootup. For example, if we want to bring core0, core3 and core4 current maxcpu(as 3) setting would not help us.
On some platform we want the flexibility on which CPUs to bring up during the device bootup. bootcpus command line is helping to bring specific CPUs and these patches are working downstream.

Quoted 17 lines · On Mon, Oct 26, 2020 at 10:08:47AM -0700, [email protected] wrote:

Quoted 10 lines · Thomas Gleixner

Quoted 8 lines · Elliot Berman

In a heterogeneous multiprocessor system, specifying the 'maxcpus' parameter on kernel command line does not provide sufficient control over which CPUs are brought online at kernel boot time, since CPUs may have nonuniform performance characteristics. Thus, add bootcpus kernel parameter to control which CPUs should be brought online during kernel boot. When both maxcpus and bootcpus is set, the more restrictive of the
two are booted.

Quoted 3 lines

What for? 'maxcpus' is a debug hack at best and outright dangerous on certain architectures. Why do we need more of that? Just let the machine boot and offline the CPUs from user space.

Hi Thomas and Peter,

Quoted 9 lines

Based on my understanding with maxcpus option provides, maximum no of CPUs are brough up during the device boot up. There is a different case, in which we want to restrict which CPUs to be brough up.
On a system with 8 cpus, if we set maxcpus as 3, cpu0, cpu1, and cpu2 are brough up during the bootup. For example, if we want to bring core0, core3 and core4 current maxcpu(as 3) setting would not help us.
On some platform we want the flexibility on which CPUs to bring up during the device bootup. bootcpus command line is helping to bring specific CPUs and these patches are working downstream.

That's a lot of words, but exactly 0 on _WHY_ you would want to do that.

Quoted 28 lines · Peter Zijlstra

Quoted 17 lines · On Mon, Oct 26, 2020 at 10:08:47AM -0700, [email protected] wrote:

Quoted 10 lines · Thomas Gleixner

Quoted 8 lines · Elliot Berman

In a heterogeneous multiprocessor system, specifying the 'maxcpus' parameter on kernel command line does not provide sufficient control over which CPUs are brought online at kernel boot time, since CPUs may have nonuniform performance characteristics. Thus, add bootcpus kernel parameter to control which CPUs should be brought online during kernel boot. When both maxcpus and bootcpus is set, the more restrictive of the
two are booted.

Quoted 3 lines

What for? 'maxcpus' is a debug hack at best and outright dangerous on certain architectures. Why do we need more of that? Just let the machine boot and offline the CPUs from user space.

Hi Thomas and Peter,

Quoted 9 lines

Based on my understanding with maxcpus option provides, maximum no of CPUs are brough up during the device boot up. There is a different case, in which we want to restrict which CPUs to be brough up.
On a system with 8 cpus, if we set maxcpus as 3, cpu0, cpu1, and cpu2 are brough up during the bootup. For example, if we want to bring core0, core3 and core4 current maxcpu(as 3) setting would not help us.
On some platform we want the flexibility on which CPUs to bring up during the device bootup. bootcpus command line is helping to bring specific CPUs and these patches are working downstream.

Quoted 2 lines

That's a lot of words, but exactly 0 on _WHY_ you would want to do that.

We find the ability to limit the number of cpus brought online at bootup useful, and to possibly later enable those cores. One use case is when device is undergoing initial testing is to use bootcpus to limit bootup to only a couple cores and later bring up the other cores for a controlled stress test. A core brought up during boot is also running device initialization. Besides being useful for SoC vendor bringup which typically occurs downstream, this particular use case could be exercised by developer of upstream support for a SoC when initial CPU settings are being determined.

Another use case is if user wishes to limit bootup only to the smaller or bigger cores. maxcpus= is not sufficient here to ensure that only those cores are booted since it limits only to the first N cores, which may not be the desired small or big cores. User may want to bring up only the smaller cores during bootup for thermal reasons. For instance, device may be later sufficiently charged such that boot up of the bigger cores is now permissible. Relying on thermal drivers to later take care of putting core into lower power idle may not occur until much later in boot (for instance, if the governor is a module).

Hi Elliot

+ Sudeep

Quoted 17 lines · Elliot Berman

Quoted 15 lines · Peter Zijlstra

Quoted 14 lines · On Mon, Oct 26, 2020 at 10:08:47AM -0700, [email protected] wrote:

Quoted 13 lines · Thomas Gleixner

Quoted 8 lines · Elliot Berman

In a heterogeneous multiprocessor system, specifying the 'maxcpus' parameter on kernel command line does not provide sufficient control over which CPUs are brought online at kernel boot time, since CPUs may have nonuniform performance characteristics. Thus, add bootcpus kernel parameter to control which CPUs should be brought online during kernel boot. When both maxcpus and bootcpus is set, the more restrictive of the
two are booted.

What for? 'maxcpus' is a debug hack at best and outright dangerous on certain architectures. Why do we need more of that? Just let the machine boot and offline the CPUs from user space.

Quoted 13 lines

Quoted 13 lines

Quoted 12 lines

Hi Thomas and Peter,

Based on my understanding with maxcpus option provides, maximum no of CPUs are brough up during the device boot up. There is a different case, in which we want to restrict which CPUs to be brough up.
On a system with 8 cpus, if we set maxcpus as 3, cpu0, cpu1, and cpu2 are brough up during the bootup. For example, if we want to bring core0, core3 and core4 current maxcpu(as 3) setting would not help us.
On some platform we want the flexibility on which CPUs to bring up during the device bootup. bootcpus command line is helping to bring specific CPUs and these patches are working downstream.

Quoted 13 lines

Quoted 2 lines

That's a lot of words, but exactly 0 on _WHY_ you would want to do that.

We find the ability to limit the number of cpus brought online at bootup useful, and to possibly later enable those cores. One use case is when device is undergoing initial testing is to use bootcpus to limit bootup to only a couple cores and later bring up the other cores for a controlled stress test. A core brought up during boot is also running device initialization. Besides being useful for SoC vendor bringup which typically occurs downstream, this particular use case could be exercised by developer of upstream support for a SoC when initial CPU settings are being determined.

Quoted 9 lines

Another use case is if user wishes to limit bootup only to the smaller or bigger cores. maxcpus= is not sufficient here to ensure that only those cores are booted since it limits only to the first N cores, which may not be the desired small or big cores. User may want to bring up only the smaller cores during bootup for thermal reasons. For instance, device may be later sufficiently charged such that boot up of the bigger cores is now permissible. Relying on thermal drivers to later take care of putting core into lower power idle may not occur until much later in boot (for instance, if the governor is a module).

I would have thought that FW/SCP would have the power to block booting up the CPUs if it deemed that to be unsafe.

Quoted 12 lines

Quoted 12 lines

Quoted 12 lines

Quoted 11 lines

Hi Thomas and Peter,

Based on my understanding with maxcpus option provides, maximum no of CPUs are brough up during the device boot up. There is a different case, in which we want to restrict which CPUs to be brough up.
On a system with 8 cpus, if we set maxcpus as 3, cpu0, cpu1, and cpu2 are brough up during the bootup. For example, if we want to bring core0, core3 and core4 current maxcpu(as 3) setting would not help us.
On some platform we want the flexibility on which CPUs to bring up during the device bootup. bootcpus command line is helping to bring specific CPUs and these patches are working downstream.

Either offline "unwanted" CPUs from user space. If that is not possible for whatever thermal reasons, we need to check if we can disable them in the DT like ACPI does. IIUC, it is not supported for some reasons I need to recall/check, can't remember that now. If that is not possible, make those nodes disappear in the bootloader ?

Quoted 13 lines

Quoted 13 lines

Quoted 2 lines

That's a lot of words, but exactly 0 on _WHY_ you would want to do that.

We find the ability to limit the number of cpus brought online at bootup useful, and to possibly later enable those cores. One use case is when device is undergoing initial testing is to use bootcpus to limit bootup to only a couple cores and later bring up the other cores for a controlled stress test. A core brought up during boot is also running device initialization. Besides being useful for SoC vendor bringup which typically occurs downstream, this particular use case could be exercised by developer of upstream support for a SoC when initial CPU settings are being determined.

Why not try single core instead of couple of core and add the needed ones for the user-space ?

Quoted 10 lines

Quoted 9 lines

Another use case is if user wishes to limit bootup only to the smaller or bigger cores. maxcpus= is not sufficient here to ensure that only those cores are booted since it limits only to the first N cores, which may not be the desired small or big cores. User may want to bring up only the smaller cores during bootup for thermal reasons. For instance, device may be later sufficiently charged such that boot up of the bigger cores is now permissible. Relying on thermal drivers to later take care of putting core into lower power idle may not occur until much later in boot (for instance, if the governor is a module).

Quoted 3 lines

I would have thought that FW/SCP would have the power to block booting up the CPUs if it deemed that to be unsafe.

I think it is more like *desire* to run with whatever battery life is left rather than *unsafe* to bring up the core.

Also not sure if we can put such battery life related policies in the firmware. If there is a thermal constraint, I am sure f/w will and must refuse to boot the core. I doubt if we are talking about that here. It is more a policy to extract max out of the battery life left, at-least the way I see this issue. I may not have full context here, sorry.

Quoted 13 lines · Sudeep Holla

Quoted 12 lines

Quoted 12 lines

Quoted 12 lines

Quoted 11 lines

Hi Thomas and Peter,

Based on my understanding with maxcpus option provides, maximum no of CPUs are brough up during the device boot up. There is a different case, in which we want to restrict which CPUs to be brough up.
On a system with 8 cpus, if we set maxcpus as 3, cpu0, cpu1, and cpu2 are brough up during the bootup. For example, if we want to bring core0, core3 and core4 current maxcpu(as 3) setting would not help us.
On some platform we want the flexibility on which CPUs to bring up during the device bootup. bootcpus command line is helping to bring specific CPUs and these patches are working downstream.

Quoted 6 lines

Either offline "unwanted" CPUs from user space. If that is not possible for whatever thermal reasons, we need to check if we can disable them in the DT like ACPI does. IIUC, it is not supported for some reasons I need to recall/check, can't remember that now. If that is not possible, make those nodes disappear in the bootloader ?

If I disappear the cpu nodes in bootloader, then I can't later online these cpus back up when policy permits.

In our experience, there is a performance hit of ~100ms to modify any devicetree node in bootloader, which is significant on a commercial device wanting to disable bootup of certain cores for thermal.

Quoted 14 lines

Quoted 13 lines

Quoted 13 lines

Quoted 2 lines

That's a lot of words, but exactly 0 on _WHY_ you would want to do that.

We find the ability to limit the number of cpus brought online at bootup useful, and to possibly later enable those cores. One use case is when device is undergoing initial testing is to use bootcpus to limit bootup to only a couple cores and later bring up the other cores for a controlled stress test. A core brought up during boot is also running device initialization. Besides being useful for SoC vendor bringup which typically occurs downstream, this particular use case could be exercised by developer of upstream support for a SoC when initial CPU settings are being determined.

Quoted 2 lines

Why not try single core instead of couple of core and add the needed ones for the user-space ?

In some instances, we have seen that further debugging is needed from firmware or hardware teams. In these instances, we wanted device to still be able to do SMP boot, but with a few cores disabled.

In the case where commercial device is using feature for thermal, device should boot multiple small cores. Booting only one core means we would not be able to use all possible cores to maximum extent possible in this thermal case.

Quoted 17 lines

Quoted 10 lines

Quoted 9 lines

Another use case is if user wishes to limit bootup only to the smaller or bigger cores. maxcpus= is not sufficient here to ensure that only those cores are booted since it limits only to the first N cores, which may not be the desired small or big cores. User may want to bring up only the smaller cores during bootup for thermal reasons. For instance, device may be later sufficiently charged such that boot up of the bigger cores is now permissible. Relying on thermal drivers to later take care of putting core into lower power idle may not occur until much later in boot (for instance, if the governor is a module).

Quoted 3 lines

I would have thought that FW/SCP would have the power to block booting up the CPUs if it deemed that to be unsafe.

I think it is more like *desire* to run with whatever battery life is left rather than *unsafe* to bring up the core.

Quoted 5 lines

Also not sure if we can put such battery life related policies in the firmware. If there is a thermal constraint, I am sure f/w will and must refuse to boot the core. I doubt if we are talking about that here. It is more a policy to extract max out of the battery life left, at-least the way I see this issue. I may not have full context here, sorry.

This is correct, FW here does not actually prevent core from starting since it is not a significant enough thermal issue.

Quoted 4 lines · Elliot Berman

In the case where commercial device is using feature for thermal, device should boot multiple small cores. Booting only one core means we would not be able to use all possible cores to maximum extent possible in this thermal case.

I understood that point. But you haven't responded on my logical vs physical number argument. I am clearly NACKing this patch as is for just usage of logical CPU IDs in the command line while your intention is to control the physical CPUs. So once again, NACK for that reason.

Based on my understanding with maxcpus option provides, maximum no of CPUs are brough up during the device boot up. There is a different case, in which we want to restrict which CPUs to be brough up.

On a system with 8 cpus, if we set maxcpus as 3, cpu0, cpu1, and cpu2 are brough up during the bootup. For example, if we want to bring core0, core3 and core4 current maxcpu(as 3) setting would not help us. On some platform we want the flexibility on which CPUs to bring up during the device bootup. bootcpus command line is helping to bring specific CPUs and these patches are working downstream.

In a heterogeneous multiprocessor system, specifying the 'maxcpus' parameter on kernel command line does not provide sufficient control over which CPUs are brought online at kernel boot time, since CPUs may have nonuniform performance characteristics. Thus, add bootcpus kernel

parameter to control which CPUs should be brought online during kernel boot. When both maxcpus and bootcpus is set, the more restrictive of the two are booted.

In a heterogeneous multiprocessor system, specifying the 'maxcpus' parameter on kernel command line does not provide sufficient control over which CPUs are brought online at kernel boot time, since CPUs may have nonuniform performance characteristics. Thus, add bootcpus kernel parameter to control which CPUs should be brought online during kernel boot. When both maxcpus and bootcpus is set, the more restrictive of the two are booted.

Signed-off-by Elliot Berman <[email protected]>

---
 Documentation/admin-guide/kernel-parameters.txt |  8 +++++++
 include/linux/cpu.h                             |  2 +-
 kernel/cpu.c                                    |  4 ++--
 kernel/smp.c                                    | 28 +++++++++++++++++++++++--
 4 files changed, 37 insertions(+), 5 deletions(-)

Documentation/admin-guide/kernel-parameters.txt +0 -0

index 65d047f..ea31af3 100644
@@ -449,6 +449,14 @@