Skip to contentSkip to conversation

CPU Hotplug - Support for BSP removal

PATCH Ashok Raj 3 participants 3 messages Feb 12, 2005 – Feb 14, 2005

lore
Share & tools
lore

Ashok Raj

author
#0 lore raw

Hi

This patch requires the cpei retarget patch posted earlier.

CAVEAT: tiger4 BIOS does not have the right value for branch reg0 when handed off to OS. I observed i could handoff the BSP to SAL, but due to incorrect jump address in b0, it ends up in some while (1); loop later. So we cannot send another IPI to wakeup the bsp. The system seemed to run fine with rest of the processors.

I had the luxury of using ITP, and manually fixed the b0 reg to be like what other AP's had. Seems like rest of the values are ok. With this one time fix to b0, i was able to bring the BSP up/down. Future cpu_down had correct b0 values and worked fine.

- To test, please set PERMIT_BSP_REMOVAL via xconfig.
- force_cpei=1 boot option when booting. Only if CPEI can be received by any cpu other than one marked in via acpi the platform interrupt source.

You can also set the fore_cpei permanently via the xconfig option FORCE_CPEI_REMOVAL.

I havent done much with the cpep stuff.

Cheers,
ashok

---

Depends on : cpei_override.patch

Support to remove boot cpu, requires a different time master to be choosen if this is time_keeper cpu. Adds 2 config options.

PERMIT_BSP_REMOVE
If your platform supports this, or you need to test, enable this parameter in config. This is turned on automatically if you choose tiger_defconfig

FORCE_CPEI_RETARGET
ACPI 3.0 gives the flexibility to provide a hint that CPEI can be re-targetted. Most tiger systems the ability to re-target CPEI is valid even for today's systems. You can enable this config option to make force this assumption if this is true for your platform. This option is automatically turned on for tiget_defconfig.
---
---

 release_work-araj/arch/ia64/Kconfig          |   23 ++++++
 release_work-araj/arch/ia64/kernel/acpi.c    |    4 -
 release_work-araj/arch/ia64/kernel/iosapic.c |    6 +
 release_work-araj/arch/ia64/kernel/irq.c     |   13 +++
 release_work-araj/arch/ia64/kernel/mca.c     |    8 +-
 release_work-araj/arch/ia64/kernel/perfmon.c |    5 +
 release_work-araj/arch/ia64/kernel/smpboot.c |   92 +++++++++++++++++++++++++--
 release_work-araj/arch/ia64/kernel/time.c    |    9 ++
 release_work-araj/arch/ia64/mm/contig.c      |    4 -
 release_work-araj/arch/ia64/mm/discontig.c   |    4 -
 release_work-araj/include/asm-ia64/mca.h     |    2 
 11 files changed, 155 insertions(+), 15 deletions(-)
arch/ia64/Kconfig renamed from arch/ia64/Kconfig~ia64_bsp_remove +23 -0
diff -puN arch/ia64/Kconfig~ia64_bsp_remove arch/ia64/Kconfig
@@ -260,6 +260,29 @@ config HOTPLUG_CPU
 	  can be controlled through /sys/devices/system/cpu/cpu#.
 	  Say N if you want to disable CPU hotplug.
 
+config PERMIT_BSP_REMOVE
+	bool "Support removal of Bootstrap Processor"
+	depends on HOTPLUG_CPU
+	default n
+	---help---
+	Say Y here if your platform SAL will support removal of BSP with HOTPLUG_CPU
+	support. Many of today's BIOS's still seem to be buggy and dont handle
+	BSP removal correctly when handed off from OS using mechanism's described
+	in SAL specification Section 3.2.5 OS_BOOT_RENDEZ. It appears to work fine
+	for Application processors. On tiger-4, you can remove BSP using this
+	option, but cannot bring it back up like other AP's.
+
+config FORCE_CPEI_RETARGET
+	bool "Force assumption that CPEI can be re-targetted"
+	depends on PERMIT_BSP_REMOVE
+	default n
+	---help---
+	Say Y if you need to force the assumption that CPEI can be re-targetted to
+	any cpu in the system. This hint is available via ACPI 3.0 specifications.
+	Tiger4 systems are capable of re-directing CPEI to any CPU other than BSP.
+	This option it useful to enable this feature on older BIOS's as well.
+	You can also enable this by using boot command line option force_cpei=1.
+
 config PREEMPT
 	bool "Preemptible Kernel"
         help
arch/ia64/kernel/irq.c renamed from arch/ia64/kernel/irq.c~ia64_bsp_remove +12 -1
diff -puN arch/ia64/kernel/irq.c~ia64_bsp_remove arch/ia64/kernel/irq.c
@@ -191,8 +191,19 @@ void fixup_irqs(void)
 {
 	unsigned int irq;
 	extern void ia64_process_pending_intr(void);
+	extern void ia64_disable_timer(void);
+	extern volatile int time_keeper_id;
+
+	ia64_disable_timer();
+
+	/*
+	 * Find a new timesync master
+	 */
+	if (smp_processor_id() = time_keeper_id) {
+		time_keeper_id = first_cpu(cpu_online_map);
+		printk ("CPU %d is now promoted to time-keeper master\n", time_keeper_id);
+	}
 
-	ia64_set_itv(1<<16);
 	/*
 	 * Phase 1: Locate irq's bound to this cpu and
 	 * relocate them for cpu removal.
arch/ia64/kernel/smpboot.c renamed from arch/ia64/kernel/smpboot.c~ia64_bsp_remove +86 -6

141 lines not on this page · raw message

arch/ia64/kernel/time.c renamed from arch/ia64/kernel/time.c~ia64_bsp_remove +7 -2
diff -puN arch/ia64/kernel/time.c~ia64_bsp_remove arch/ia64/kernel/time.c
@@ -36,7 +36,7 @@ u64 jiffies_64 __cacheline_aligned_in_sm
 
 EXPORT_SYMBOL(jiffies_64);
 
-#define TIME_KEEPER_ID	0	/* smp_processor_id() of time-keeper */
+volatile int time_keeper_id = 0; /* smp_processor_id() of time-keeper */
 
 #ifdef CONFIG_IA64_DEBUG_IRQ
 
@@ -75,7 +75,7 @@ timer_interrupt (int irq, void *dev_id, 
 
 		new_itm += local_cpu_data->itm_delta;
 
-		if (smp_processor_id() = TIME_KEEPER_ID) {
+		if (smp_processor_id() = time_keeper_id) {
 			/*
 			 * Here we are in the timer irq handler. We have irqs locally
 			 * disabled, but we don't know if the timer_bh is running on
@@ -240,6 +240,11 @@ static struct irqaction timer_irqaction 
 	.name =		"timer"
 };
 
+void __devinit ia64_disable_timer(void)
+{
+	ia64_set_itv(1 << 16);
+}
+
 void __init
 time_init (void)
 {
arch/ia64/kernel/mca.c renamed from arch/ia64/kernel/mca.c~ia64_bsp_remove +6 -2
diff -puN arch/ia64/kernel/mca.c~ia64_bsp_remove arch/ia64/kernel/mca.c
@@ -271,7 +271,8 @@ ia64_mca_log_sal_error_record(int sal_in
 
 #ifdef CONFIG_ACPI
 
-static int cpe_vector = -1;
+int cpe_vector = -1;
+int ia64_cpe_irq = -1;
 
 static irqreturn_t
 ia64_mca_cpe_int_handler (int cpe_irq, void *arg, struct pt_regs *ptregs)
@@ -1208,11 +1209,13 @@ void __devinit
 ia64_mca_cpu_init(void *cpu_data)
 {
 	void *pal_vaddr;
+	static int first_time=1;
 
-	if (smp_processor_id() = 0) {
+	if (first_time) {
 		void *mca_data;
 		int cpu;
 
+		first_time=0;
 		mca_data = alloc_bootmem(sizeof(struct ia64_mca_cpu)
 					 * NR_CPUS);
 		for (cpu = 0; cpu < NR_CPUS; cpu++) {
@@ -1451,6 +1454,7 @@ ia64_mca_late_init(void)
 					desc = irq_descp(irq);
 					desc->status |= IRQ_PER_CPU;
 					setup_irq(irq, &mca_cpe_irqaction);
+					ia64_cpe_irq = irq;
 				}
 			ia64_mca_register_cpev(cpe_vector);
 			IA64_MCA_DEBUG("%s: CPEI/P setup and enabled.\n", __FUNCTION__);
include/asm-ia64/mca.h renamed from include/asm-ia64/mca.h~ia64_bsp_remove +2 -0
diff -puN include/asm-ia64/mca.h~ia64_bsp_remove include/asm-ia64/mca.h
@@ -117,6 +117,8 @@ struct ia64_mca_cpu {
 /* Array of physical addresses of each CPU's MCA area.  */
 extern unsigned long __per_cpu_mca[NR_CPUS];
 
+extern int cpe_vector;
+extern int ia64_cpe_irq;
 extern void ia64_mca_init(void);
 extern void ia64_mca_cpu_init(void *);
 extern void ia64_os_mca_dispatch(void);
arch/ia64/kernel/iosapic.c renamed from arch/ia64/kernel/iosapic.c~ia64_bsp_remove +6 -0
diff -puN arch/ia64/kernel/iosapic.c~ia64_bsp_remove arch/ia64/kernel/iosapic.c
@@ -493,6 +493,7 @@ get_target_cpu (unsigned int gsi, int ve
 {
 #ifdef CONFIG_SMP
 	static int cpu = -1;
+	extern int cpe_vector;
 
 	/*
 	 * If the platform supports redirection via XTP, let it
@@ -508,6 +509,11 @@ get_target_cpu (unsigned int gsi, int ve
 	if (!cpu_online(smp_processor_id()))
 		return hard_smp_processor_id();
 
+#ifdef CONFIG_ACPI
+		if (cpe_vector > 0 && vector = IA64_CPEP_VECTOR)
+			return get_cpei_target_cpu();
+#endif
+
 #ifdef CONFIG_NUMA
 	{
 		int num_cpus, cpu_index, iosapic_index, numa_cpu, i = 0;
arch/ia64/kernel/acpi.c renamed from arch/ia64/kernel/acpi.c~ia64_bsp_remove +3 -1
diff -puN arch/ia64/kernel/acpi.c~ia64_bsp_remove arch/ia64/kernel/acpi.c
@@ -288,7 +288,9 @@ acpi_parse_plat_int_src (
 
 unsigned int can_cpei_retarget(void)
 {
-	return (1 ? (acpi_cpei_override) : 0);
+	extern unsigned int force_cpei_retarget;
+
+	return (1 ? (acpi_cpei_override || force_cpei_retarget) : 0);
 }
 
 unsigned int is_cpu_cpei_target(unsigned int cpu)
arch/ia64/mm/contig.c renamed from arch/ia64/mm/contig.c~ia64_bsp_remove +3 -1
diff -puN arch/ia64/mm/contig.c~ia64_bsp_remove arch/ia64/mm/contig.c
@@ -180,13 +180,15 @@ per_cpu_init (void)
 {
 	void *cpu_data;
 	int cpu;
+	static int first_time=1;
 
 	/*
 	 * get_free_pages() cannot be used before cpu_init() done.  BSP
 	 * allocates "NR_CPUS" pages for all CPUs to avoid that AP calls
 	 * get_zeroed_page().
 	 */
-	if (smp_processor_id() = 0) {
+	if (first_time) {
+		first_time=0;
 		cpu_data = __alloc_bootmem(PERCPU_PAGE_SIZE * NR_CPUS,
 					   PERCPU_PAGE_SIZE, __pa(MAX_DMA_ADDRESS));
 		for (cpu = 0; cpu < NR_CPUS; cpu++) {
arch/ia64/mm/discontig.c renamed from arch/ia64/mm/discontig.c~ia64_bsp_remove +3 -1
diff -puN arch/ia64/mm/discontig.c~ia64_bsp_remove arch/ia64/mm/discontig.c
@@ -528,8 +528,10 @@ void __init find_memory(void)
 void *per_cpu_init(void)
 {
 	int cpu;
+	static int first_time=1;
 
-	if (smp_processor_id() = 0) {
+	if (first_time) {
+		first_time=0;
 		for (cpu = 0; cpu < NR_CPUS; cpu++) {
 			per_cpu(local_per_cpu_offset, cpu)  				__per_cpu_offset[cpu];
arch/ia64/kernel/perfmon.c renamed from arch/ia64/kernel/perfmon.c~ia64_bsp_remove +4 -1
diff -puN arch/ia64/kernel/perfmon.c~ia64_bsp_remove arch/ia64/kernel/perfmon.c
@@ -6548,6 +6548,7 @@ __initcall(pfm_init);
 void
 pfm_init_percpu (void)
 {
+	static int first_time=1;
 	/*
 	 * make sure no measurement is active
 	 * (may inherit programmed PMCs from EFI).
@@ -6560,8 +6561,10 @@ pfm_init_percpu (void)
 	 */
 	pfm_unfreeze_pmu();
 
-	if (smp_processor_id() = 0)
+	if (first_time) {
 		register_percpu_irq(IA64_PERFMON_VECTOR, &perfmon_irqaction);
+		first_time=0;
+	}
 
 	ia64_setreg(_IA64_REG_CR_PMV, IA64_PERFMON_VECTOR);
 	ia64_srlz_d();

_

#1 lore raw
Quoted 9 lines · Ashok Raj

Hi

This patch requires the cpei retarget patch posted earlier.

CAVEAT: tiger4 BIOS does not have the right value for branch reg0 when handed off to OS. I observed i could handoff the BSP to SAL, but due to incorrect jump address in b0, it ends up in some while (1); loop later. So we cannot send another IPI to wakeup the bsp. The system seemed to run fine with rest of the processors.

In my reading of the spec, I don't see how we can count on the BSP b0
being the appropriate value for return to SAL. If I don't enable the fixup option, we branch off to some bad address and the CPU is lost. With the fixup, hotplugging the BSP on and rx2600 seems to work.

Quoted 1 line

I havent done much with the cpep stuff.

At a minimum, the patches below are required. Thanks,

Alex

Signature · 32 lines
Alex Williamson                             HP Linux & Open Source Lab

--- linux/arch/ia64/Kconfig	2005-02-14 09:36:15.642178325 -0700
+++ linux/arch/ia64/Kconfig	2005-02-14 09:36:29.286709408 -0700
@@ -285,7 +285,7 @@
 
 config BSP_REMOVE_WORKAROUND
 	bool "Enable temporary workaround to fix b0 value for CPU hotplug"
-	depends on PERMIT_BSP_REMOVE && FORCE_CPEI_RETARGET && EXPERIMENTAL
+	depends on PERMIT_BSP_REMOVE && EXPERIMENTAL
 	default y
 	---help---
 	On tiger 4 systems's BSP's branch reg 0 (b0) is not set correctly.
--- linux/arch/ia64/kernel/topology.c	2005-02-14 09:37:20.191005660 -0700
+++ linux/arch/ia64/kernel/topology.c	2005-02-14 09:26:30.178318310 -0700
@@ -31,6 +31,7 @@
 int arch_register_cpu(int num)
 {
 	struct node *parent = NULL;
+	extern int cpe_vector;
 	
 #ifdef CONFIG_NUMA
 	parent = &sysfs_nodes[cpu_to_node(num)];
@@ -40,7 +41,7 @@
 	 * If CPEI cannot be re-targetted, and this is
 	 * CPEI target, then dont create the control file
 	 */
-	if (!can_cpei_retarget() && is_cpu_cpei_target(num))
+	if (cpe_vector > 0 && !can_cpei_retarget() && is_cpu_cpei_target(num))
 		sysfs_cpus[num].cpu.no_control = 1;
 
 	return register_cpu(&sysfs_cpus[num].cpu, num, parent);
#2 lore raw
In reply to Ashok Raj
Quoted 4 lines

In my reading of the spec, I don't see how we can count on the BSP b0
being the appropriate value for return to SAL. If I don't enable the fixup option, we branch off to some bad address and the CPU is lost. With the fixup, hotplugging the BSP on and rx2600 seems to work.

Correct.

Elilo makes no attempt to pass anything to the OS in "b0" (so it will actually contain whatever was left in b0 after the last function call made by elilo[1]). I doubt that this could be fixed in elilo ... the value in b0 when elilo is started presumably just points back to some point in the EFI shell [I think I've seen elilo exit and just go back to EFI, but I may either be misremembering, of misunderstanding the code sequence that occurred to get back to the EFI shell].

-Tony

[1] and since the memory that elilo occupied will have been freed by the time you try to take a cpu offline, this is effectively a garbage pointer.