Skip to contentSkip to conversation

parisc: Fix TLB related boot crash with PA8000-PA8700 CPUs

PATCH In review Helge Deller 3 participants 8 messages Dec 7, 2016 – Dec 8, 2016 PA-RISC architecture

lore
Share & tools
lore

Helge Deller

author maintainer
#0 lore raw

PATCH 1/1 parisc: Fix TLB related boot crash with PA8000-PA8700 CPUs

Machines with PA8000-PA8700 CPUs crash during startup while we measure and calculate a good threshold for the TLB flush.

Avoid this crash by simply skipping the test until we figure out what really triggers the crash.

Cc <[email protected]> # v3.18+
Signed-off-by Helge Deller <[email protected]>
arch/parisc/kernel/cache.c +10 -0
index c263301..63c10ea 100644
@@ -393,6 +393,14 @@ void __init parisc_setup_cache_timing(void)
 
 	/* calculate TLB flush threshold */
 
+	/* skip TLB measure on PA8000-PA8700 CPUs */
+	if (boot_cpu_data.cpu_type >= pcxu &&
+	    boot_cpu_data.cpu_type <= pcxw2) {
+		threshold = max(cache_info.it_size, cache_info.dt_size);
+		threshold *= PAGE_SIZE;
+		goto set_tlb_threshold;
+	}
+
 	alltime = mfctl(16);
 	flush_tlb_all();
 	alltime = mfctl(16) - alltime;

@@ -411,6 +419,8 @@ void __init parisc_setup_cache_timing(void)
 		alltime, size, rangetime);
 
 	threshold = PAGE_ALIGN(num_online_cpus() * size * alltime / rangetime);
+
+set_tlb_threshold:
 	if (threshold)
 		parisc_tlb_flush_threshold = threshold;
 	printk(KERN_INFO "TLB flush threshold set to %lu KiB\n",
#1 lore raw
Quoted 2 lines · Helge Deller

Machines with PA8000-PA8700 CPUs crash during startup while we measure and calculate a good threshold for the TLB flush.

I haven't seen any crashed on HP c3700 (PA8700)...

A.

#2 lore raw
Quoted 4 lines · Aaro Koskinen
Quoted 2 lines · Helge Deller

Machines with PA8000-PA8700 CPUs crash during startup while we measure and calculate a good threshold for the TLB flush.

I haven't seen any crashed on HP c3700 (PA8700)...

I believe the issue only affects SMP machines. My c3750 also boots okay.

Dave

Signature · 1 line
John David Anglin  [email protected]
author maintainer
#3 lore raw
In reply to Aaro Koskinen

Hi Aaro,

Quoted 5 lines · Aaro Koskinen
Quoted 2 lines · Helge Deller

Machines with PA8000-PA8700 CPUs crash during startup while we measure and calculate a good threshold for the TLB flush.

I haven't seen any crashed on HP c3700 (PA8700)...

Thanks for the feedback!
I did some additional analysis today, and it happens only on SMP machines. I could reproduce the crash on a A500-44 and a J5000, both are 2-way boxes.

Helge

author maintainer
#4 lore raw

Re: [PATCH] parisc: Fix TLB related boot crash on SMP machines

At bootup we run measurements to calculate the best threshold for when we should be using full TLB flushes instead of just flushing a specific amount of TLB entries. This performance test is run over the kernel text segment.

But running this TLB performance test on the kernel text segment turned out to crash some SMP machines when the kernel text pages were mapped as huge pages.

To avoid those crashes this patch simply skips this test on some SMP machines and calculates an optimal threshold based on the maximum number of available TLB entries and number of online CPUs.

On a technical side, this seems to happen:
The TLB measurement code uses flush_tlb_kernel_range() to flush specific TLB entries with a page size of 4k (pdtlb 0(sr1,addr)). On UP systems this purge instruction seems to work without problems even if the pages were mapped as huge pages. But on SMP systems the TLB purge instruction is broadcasted to other CPUs. Those CPUs then crash the machine because the page size is not as expected. C8000 machines with PA8800/PA8900 CPUs were not affected by this problem, because the required cache coherency prohibits to use huge pages at all. Sadly I didn't found any documentation about this behaviour, so this finding is purely based on testing with phyiscal SMP machines (A500-44 and J5000, both were 2-way boxes).

Cc <[email protected]> # v3.18+
Signed-off-by Helge Deller <[email protected]>
arch/parisc/kernel/cache.c +11 -0
index c263301..977f0a4f 100644
@@ -393,6 +393,15 @@ void __init parisc_setup_cache_timing(void)
 
 	/* calculate TLB flush threshold */
 
+	/* On SMP machines, skip the TLB measure of kernel text which
+	 * has been mapped as huge pages. */
+	if (num_online_cpus() > 1 && !parisc_requires_coherency()) {
+		threshold = max(cache_info.it_size, cache_info.dt_size);
+		threshold *= PAGE_SIZE;
+		threshold /= num_online_cpus();
+		goto set_tlb_threshold;
+	}
+
 	alltime = mfctl(16);
 	flush_tlb_all();
 	alltime = mfctl(16) - alltime;

@@ -411,6 +420,8 @@ void __init parisc_setup_cache_timing(void)
 		alltime, size, rangetime);
 
 	threshold = PAGE_ALIGN(num_online_cpus() * size * alltime / rangetime);
+
+set_tlb_threshold:
 	if (threshold)
 		parisc_tlb_flush_threshold = threshold;
 	printk(KERN_INFO "TLB flush threshold set to %lu KiB\n",
#5 lore raw

Re: [PATCH] parisc: Fix TLB related boot crash on SMP machines

Quoted 11 lines · Helge Deller

On a technical side, this seems to happen:
The TLB measurement code uses flush_tlb_kernel_range() to flush specific TLB entries with a page size of 4k (pdtlb 0(sr1,addr)). On UP systems this purge instruction seems to work without problems even if the pages were mapped as huge pages. But on SMP systems the TLB purge instruction is broadcasted to other CPUs. Those CPUs then crash the machine because the page size is not as expected. C8000 machines with PA8800/PA8900 CPUs were not affected by this problem, because the required cache coherency prohibits to use huge pages at all. Sadly I didn't found any documentation about this behaviour, so this finding is purely based on testing with phyiscal SMP machines (A500-44 and J5000, both were 2-way boxes).

I doubt the problem is the 4k iteration using pdtlb 0(sr1,addr). I think the issue is the huge page size for the kernel. Each pdtlb instruction knocks out the same tlb entry including the entry used for tlb interruptions. This likely leads to stack overflow. In any case, it probably doesn't provide accurate timing because each pdtlb knocks out the entry for the interruption handler on systems with combined tlb.

Dave

Signature · 1 line
John David Anglin  [email protected]
author maintainer
#6 lore raw

Re: [PATCH] parisc: Fix TLB related boot crash on SMP machines

Quoted 13 lines · John David Anglin
Quoted 11 lines · Helge Deller

On a technical side, this seems to happen:
The TLB measurement code uses flush_tlb_kernel_range() to flush specific TLB entries with a page size of 4k (pdtlb 0(sr1,addr)). On UP systems this purge instruction seems to work without problems even if the pages were mapped as huge pages. But on SMP systems the TLB purge instruction is broadcasted to other CPUs. Those CPUs then crash the machine because the page size is not as expected. C8000 machines with PA8800/PA8900 CPUs were not affected by this problem, because the required cache coherency prohibits to use huge pages at all. Sadly I didn't found any documentation about this behaviour, so this finding is purely based on testing with phyiscal SMP machines (A500-44 and J5000, both were 2-way boxes).

Quoted 4 lines

I doubt the problem is the 4k iteration using pdtlb 0(sr1,addr). I think the issue is the huge page size for the kernel. Each pdtlb instruction knocks out the same tlb entry including the entry used for tlb interruptions. This likely leads to stack overflow.

Yes, likely.

Quoted 4 lines

In any
case, it probably doesn't provide accurate timing because each pdtlb knocks out the entry for the interruption handler on systems with combined tlb.

True.

So, how to continue?
I see two options:
a) skip the TLB measuring code as my patch does.
b) kmalloc() another region and do measurement there.

I'd like to submit some fix-patch for 4.9, else the machines won't boot 4.9. That's why I'd prefer option a).
Opinions?

Helge

#7 lore raw

Re: [PATCH] parisc: Fix TLB related boot crash on SMP machines

Quoted 25 lines · Helge Deller
Quoted 12 lines · John David Anglin
Quoted 11 lines · Helge Deller

On a technical side, this seems to happen:
The TLB measurement code uses flush_tlb_kernel_range() to flush specific TLB entries with a page size of 4k (pdtlb 0(sr1,addr)). On UP systems this purge instruction seems to work without problems even if the pages were mapped as huge pages. But on SMP systems the TLB purge instruction is broadcasted to other CPUs. Those CPUs then crash the machine because the page size is not as expected. C8000 machines with PA8800/PA8900 CPUs were not affected by this problem, because the required cache coherency prohibits to use huge pages at all. Sadly I didn't found any documentation about this behaviour, so this finding is purely based on testing with phyiscal SMP machines (A500-44 and J5000, both were 2-way boxes).

Quoted 4 lines

I doubt the problem is the 4k iteration using pdtlb 0(sr1,addr). I think the issue is the huge page size for the kernel. Each pdtlb instruction knocks out the same tlb entry including the entry used for tlb interruptions. This likely leads to stack overflow.

Yes, likely.

Quoted 4 lines

In any
case, it probably doesn't provide accurate timing because each pdtlb knocks out the entry for the interruption handler on systems with combined tlb.

True.

Quoted 8 lines

So, how to continue?
I see two options:
a) skip the TLB measuring code as my patch does.
b) kmalloc() another region and do measurement there.

I'd like to submit some fix-patch for 4.9, else the machines won't boot 4.9. That's why I'd prefer option a).
Opinions?

Go with option a) for 4.9

Dave

Signature · 1 line
John David Anglin  [email protected]