sparc64: Duplicate kernel text on every NUMA node

Hi!

There is an idea to duplicate kernel text on every NUMA node. Node access is the fastest, so this should help to execute kernel text a little faster.

This requires the patch which changes init text place in lds file (http://patchwork.ozlabs.org/patch/264713/).

I assume that original kernel text allocates from node 0, and it is not necessary to allocate per node kernel text clone for BSP cpu and for its neighbours. So we should do this only in trampoline_64.S.

I allocate a memory for node copy before page allocator is active. When all kernel patches are applied then it's the time to copy kernel text on every node. So when cpu is starting it receives a kernel text copy which corresponds to its node.

The copy doesn't include init text except the cases when it is in the same 4Mb page with not-init text. In other words (or more correct) tail init text is not copied. It's mapped from the original image.

It looks no problem with everything above: node is already online, everything is initialized, kernel text is locked forever.

Please, tell your comments about this (It's not ready patch, I'm not sure it's completely good yet). Maybe there are any reasons to do anything in another way than I did.

Thanks,
Kirill
---

index e155388..c5273e4 100644
@@ -8,6 +8,9 @@
 #define PAGE_SIZE    (_AC(1,UL) << PAGE_SHIFT)
 #define PAGE_MASK    (~(PAGE_SIZE-1))
 
+#define PAGE4MB_SHIFT		22
+#define PAGE4MB_SIZE		(_AC(1,UL) << PAGE4MB_SHIFT)
+
 /* Flushing for D-cache alias handling is only needed if
  * the page size is smaller than 16K.
  */

index 3676031..cd440bd 100644
@@ -843,6 +843,12 @@ extern pmd_t swapper_low_pmd_dir[2048];
 extern void paging_init(void);
 extern unsigned long find_ecache_flush_span(unsigned long size);
 
+#ifdef CONFIG_NUMA
+extern void numa_copy_kernel_text(void);
+#else
+static inline void numa_copy_kernel_text(void) {}
+#endif
+
 struct seq_file;
 extern void mmu_info(struct seq_file *);
 

index 7e26b2d..a2f0990 100644
@@ -138,6 +138,23 @@ extern struct sun4v_2insn_patch_entry __sun4v_2insn_patch,
 	nop;						\
 	.previous;
 
+#ifdef CONFIG_NUMA
+
+#define __GET_NODEID(REG, TMP)				\
+	__GET_CPUID(REG)				\
+	sethi	%hi(numa_cpu_lookup_table), TMP;	\
+	or	TMP, %lo(numa_cpu_lookup_table), TMP;	\
+	sllx	REG, 2, REG;				\
+	add	TMP, REG, TMP;				\
+	lduw	[TMP], REG;
+
+#else /* !CONFIG_NUMA */
+
+#define __GET_NODEID(REG, TMP)				\
+	clr	REG
+
+#endif /* !CONFIG_NUMA */
+
 #ifdef CONFIG_SMP
 
 #define TRAP_LOAD_TRAP_BLOCK(DEST, TMP)		\

index e142545..f92d31e 100644
@@ -285,7 +285,7 @@ static void ldom_startcpu_cpuid(unsigned int cpu, unsigned long thread_reg,
 				void **descrp)
 {
 	extern unsigned long sparc64_ttable_tl0;
-	extern unsigned long kern_locked_tte_data;
+	extern unsigned long kern_locked_tte_data[MAX_NUMNODES];
 	struct hvtramp_descr *hdesc;
 	unsigned long trampoline_ra;
 	struct trap_per_cpu *tb;

@@ -315,7 +315,7 @@ static void ldom_startcpu_cpuid(unsigned int cpu, unsigned long thread_reg,
 	hdesc->thread_reg = thread_reg;
 
 	tte_vaddr = (unsigned long) KERNBASE;
-	tte_data = kern_locked_tte_data;
+	tte_data = kern_locked_tte_data[0];
 
 	for (i = 0; i < hdesc->num_mappings; i++) {
 		hdesc->maps[i].vaddr = tte_vaddr;

@@ -1214,6 +1214,10 @@ int setup_profiling_timer(unsigned int multiplier)
 
 void __init smp_prepare_cpus(unsigned int max_cpus)
 {
+	/* Dublicate kernel on every node. Do this after
+	 * all kernel patches are applied.
+	 */
+	numa_copy_kernel_text();
 }
 
 void smp_prepare_boot_cpu(void)

index 0661aa6..d3edd6f 100644
@@ -32,7 +32,7 @@ extern struct linux_prom_translation prom_trans[512];
 extern unsigned int prom_trans_ents;
 
 /* Exported for SMP bootup purposes. */
-extern unsigned long kern_locked_tte_data;
+extern unsigned long kern_locked_tte_data[MAX_NUMNODES];
 
 extern void prom_world(int enter);
 

From: Kirill Tkhai <[email protected]>
Date: Thu, 22 Aug 2013 17:39:50 +0400

Interesting change.

What other major architectures do this?

23.08.2013, 09:23, "David Miller" <[email protected]>:

It looks like nobody supports this. I saw an old patch for x86 (https://lkml.org/lkml/2003/6/16/226)
and several others, but they were not integrated. So, sparc64 can be the first.

Other architectures have no as cheap replication as sparc64 has. They must modify page table top every switch_to(), while sparc64 is free from this thing. And it looks like they have to flush all instruction cache every context switch.

I'll work on the patch a little bit more and test it in different boundary cases, before submit it as a ready patch.

Thanks,
Kirill

From: Kirill Tkhai <[email protected]>
Date: Fri, 23 Aug 2013 19:43:56 +0400

Exciting, sparc64 is becomming such a groundbreaking platform :-)

I'd be interested in any performance improvement you can show because that's the reason to do this right?

A parallel kernel build utilizing all nodes ought to show at least something.

23.08.2013, 22:08, "David Miller" <[email protected]>:

I used 16 CPU system with a clock rate of 820 MHz. Test "time make image -j64" gives the following:

Before:

real šššš7m58.466s
user š60m49.660s
sys ššš47m40.030s

After:

real šššš7m55.562s
user š60m20.900s
sys ššš46m36.040s

So, the real profit (system time) is whole 2.2%. I thought, it would be more :) It seems, it's not big enough to really implement this feature. Practice checks a hypothesis.

Thanks for the discussion, David!

Regards,
Kirill.

From: Kirill Tkhai <[email protected]>
Date: Mon, 09 Sep 2013 18:50:00 +0400

You are juding the suitability of your optimization using one statistical sample of a benchmark involving trillions of memory accesses?

Please at least run it 3 or 4 times so you can see how much variation occurs between runs in the same configuration. :-)

09.09.2013, 21:31, "David Miller" <[email protected]>:

Yes, I compiled kernel 3 times for every case. Two above are middle results. Another cases are a little different, but tendency is like above. The low bound of difference is 2.0+%

I did every test after cold start (and even took ethernet cable out). So, the result is honest :)

(For comparison: Plain memory access time ratio between own node and its neighbour is about 1.4 on the tested machine. It looks like instruction cache has the influence on above result. I don't know what is its influence on machines of another type, maybe anywhere it is smaller).

Kirill