NUMA-Q breakage 2/7 xquad_portio ioremap deadlock
The cpu_online_map stuff for hotplug cpu created a brand new bootstrap ordering problem for NUMA-Q. The mmapped portio region needs to be ioremapped early but ioremap attempts to do TLB shootdown, and smp_call_function() (called by flush_tlb_all()) deadlocks when cpu_online_map is uninitialized.
Workaround (due to Matt Dobson) below.
arch/i386/kernel/smp.c +1 -1
@@ -569,7 +569,7 @@struct call_data_struct data;int cpus =num_online_cpus ()-1 ; -if (!cpus) +if (cpus <=0 )return 0 ; data.func = func;
Quoted 14 lines
The
cpu_online_mapstuff for hotplug cpu created a brand new bootstrap ordering problem for NUMA-Q. The mmapped portio region needs to be ioremapped early but ioremap attempts to do TLB shootdown, andsmp_call_function()(called byflush_tlb_all()) deadlocks whencpu_online_mapis uninitialized.Workaround (due to Matt Dobson) below.
diff -Nur
linux-2.5.23-vanilla/arch/i386/kernel/smp.clinux-2.5.23-patched/arch/i386/kernel/smp.c
arch/i386/kernel/smp.c line 575
Quoted 8 lines
arch/i386/kernel/smp.c +1
-1@@ -569,7 +569,7 @@struct call_data_struct data;int cpus =num_online_cpus ()-1 ; -if (!cpus) +if (cpus <=0 )return 0 ; data.func = func;
Would it be slightly less of a hack if we just move the ioremap down below set_bit(0, &cpu_online_map); later on in smp_boot_cpus ? Untested patch below. As long as we set up the xquad_portio remap before any other cpus are online, I can't see it matters exactly when we do it ....
Either that, or we just define cpu_online_map to be =1 to start with.
M.
arch/i386/kernel/smpboot.c +10 -10
@@ -1019,16 +1019,6 @@ {int apicid, cpu, bit; -if (clustered_apic_mode && (numnodes >1 )) { -printk ("Remapping cross-quad port I/O for %d quads\n" , - numnodes); -printk ("xquad_portio vaddr 0x%08lx, len %08lx\n" , - (u_long ) xquad_portio, - (u_long ) numnodes *XQUAD_PORTIO_LEN ); - xquad_portio =ioremap (XQUAD_PORTIO_BASE , - numnodes *XQUAD_PORTIO_LEN ); - } -#ifdef CONFIG_MTRR /* Must be done before other processors booted */ mtrr_init_boot_cpu ();
@@ -1126,6 +1116,16 @@if (GET_APIC_ID (apic_read (APIC_ID )) != boot_cpu_physical_apicid)BUG (); + +if (clustered_apic_mode && (numnodes >1 )) { +printk ("Remapping cross-quad port I/O for %d quads\n" , + numnodes); +printk ("xquad_portio vaddr 0x%08lx, len %08lx\n" , + (u_long ) xquad_portio, + (u_long ) numnodes *XQUAD_PORTIO_LEN ); + xquad_portio =ioremap (XQUAD_PORTIO_BASE , + numnodes *XQUAD_PORTIO_LEN ); + }/* * Scan the CPU present map and fire up the other CPUs via do_boot_cpu
Quoted 5 lines
Quoted 5 lines
The
cpu_online_mapstuff for hotplug cpu created a brand new bootstrap ordering problem for NUMA-Q. The mmapped portio region needs to be ioremapped early but ioremap attempts to do TLB shootdown, andsmp_call_function()(called byflush_tlb_all()) deadlocks whencpu_online_mapis uninitialized.
Quoted 6 lines · Martin J. Bligh
Would it be slightly less of a hack if we just move the ioremap down below
set_bit(0, &cpu_online_map); later on insmp_boot_cpus? Untested patch below. As long as we set up thexquad_portioremap before any other cpus are online, I can't see it matters exactly when we do it .... Either that, or we just definecpu_online_mapto be =1 to start with. M.
This looks better than fiddling with smp_call_function().
Cheers,
Bill