Skip to contentSkip to conversation

arch/sparc: support NR_CPUS = 4096

PATCH Superseded Jane Chu 2 participants 5 messages Jun 5, 2017 – Jun 6, 2017 SPARC + UltraSPARC (sparc/sparc64)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Jane Chu

author
#0 lore raw

PATCH v2 1/1 arch/sparc: support NR_CPUS = 4096

Linux SPARC64 limits NR_CPUS to 4064 because init_cpu_send_mondo_info() only allocates a single page for NR_CPUS mondo entries. Thus we cannot use all 4096 CPUs on some SPARC platforms.

To fix, allocate (2^order) pages where order is set according to the size of cpu_list for possible cpus. Since cpu_list_pa and cpu_mondo_block_pa are not used in asm code, there are no imm13 offsets from the base PA that will break because they can only reach one page.

Orabug: 25505750

---
 arch/sparc/Kconfig         |    4 ++--
 arch/sparc/kernel/irq_64.c |   16 ++++++++++++----
 2 files changed, 14 insertions(+), 6 deletions(-)
arch/sparc/Kconfig +2 -2
index 58243b0..4399be7 100644
@@ -192,9 +192,9 @@ config NR_CPUS
 	int "Maximum number of CPUs"
 	depends on SMP
 	range 2 32 if SPARC32
-	range 2 1024 if SPARC64
+	range 2 4096 if SPARC64
 	default 32 if SPARC32
-	default 64 if SPARC64
+	default 4096 if SPARC64
 
 source kernel/Kconfig.hz
 
arch/sparc/kernel/irq_64.c +12 -4
index 4d0248a..cc8f6c3 100644
@@ -1034,17 +1034,25 @@ static void __init init_cpu_send_mondo_info(struct trap_per_cpu *tb)
 {
 #ifdef CONFIG_SMP
 	unsigned long page;
+	void *mondo;
 
-	BUILD_BUG_ON((NR_CPUS * sizeof(u16)) > (PAGE_SIZE - 64));
+	BUILD_BUG_ON((NR_CPUS * sizeof(u16)) > PAGE_SIZE);
+
+	/* Make sure mondo block is 64byte aligned */
+	mondo = kzalloc(64, GFP_KERNEL);
+	if (!mondo) {
+		prom_printf("SUN4V: Error, cannot allocate mondo block.\n");
+		prom_halt();
+	}
+	tb->cpu_mondo_block_pa = __pa(mondo);
 
 	page = get_zeroed_page(GFP_KERNEL);
 	if (!page) {
-		prom_printf("SUN4V: Error, cannot allocate cpu mondo page.\n");
+		prom_printf("SUN4V: Error, cannot allocate cpu list page.\n");
 		prom_halt();
 	}
 
-	tb->cpu_mondo_block_pa = __pa(page);
-	tb->cpu_list_pa = __pa(page + 64);
+	tb->cpu_list_pa = __pa(page);
 #endif
 }
 
Signature · 1 line
1.7.1
maintainer
#1 lore raw

From: Jane Chu <[email protected]>
Date: Mon, 5 Jun 2017 16:48:31 -0600

Quoted 46 lines

Linux SPARC64 limits NR_CPUS to 4064 because init_cpu_send_mondo_info() only allocates a single page for NR_CPUS mondo entries. Thus we cannot use all 4096 CPUs on some SPARC platforms.

To fix, allocate (2^order) pages where order is set according to the size of cpu_list for possible cpus. Since cpu_list_pa and cpu_mondo_block_pa are not used in asm code, there are no imm13 offsets from the base PA that will break because they can only reach one page.

Orabug: 25505750

---
 arch/sparc/Kconfig         |    4 ++--
 arch/sparc/kernel/irq_64.c |   16 ++++++++++++----
 2 files changed, 14 insertions(+), 6 deletions(-)
arch/sparc/Kconfig +2 -2
index 58243b0..4399be7 100644
@@ -192,9 +192,9 @@ config NR_CPUS
 	int "Maximum number of CPUs"
 	depends on SMP
 	range 2 32 if SPARC32
-	range 2 1024 if SPARC64
+	range 2 4096 if SPARC64
 	default 32 if SPARC32
-	default 64 if SPARC64
+	default 4096 if SPARC64
 
 source kernel/Kconfig.hz
 
arch/sparc/kernel/irq_64.c +1 -0
index 4d0248a..cc8f6c3 100644
@@ -1034,17 +1034,25 @@ static void __init init_cpu_send_mondo_info(struct trap_per_cpu *tb)
 {
 #ifdef CONFIG_SMP
 	unsigned long page;
+	void *mondo;
 

arch/sparc/kernel/irq_64.c static void __init init_cpu_send_mondo_info(struct trap_per_cpu *tb) · line 1047

Quoted 10 lines
arch/sparc/kernel/irq_64.c +9 -1
-	BUILD_BUG_ON((NR_CPUS * sizeof(u16)) > (PAGE_SIZE - 64));
+	BUILD_BUG_ON((NR_CPUS * sizeof(u16)) > PAGE_SIZE);
+
+	/* Make sure mondo block is 64byte aligned */
+	mondo = kzalloc(64, GFP_KERNEL);
+	if (!mondo) {
+		prom_printf("SUN4V: Error, cannot allocate mondo block.\n");
+		prom_halt();
+	}
+	tb->cpu_mondo_block_pa = __pa(mondo);

Hmmm, you said that this has to be 64 byte aligned right? We might have to do something in order to insure that, as kmalloc() only guarantees ARCH_KMALLOC_MINALIGN which I think is 8 on sparc.

I suppose this would work:

	mondo = kzalloc(64 + 63, GFP_KERNEL);

and then 64-byte align that pointer.

#2 lore raw
Quoted 49 lines · David Miller

From: Jane Chu <[email protected]>
Date: Mon, 5 Jun 2017 16:48:31 -0600

Quoted 46 lines

Linux SPARC64 limits NR_CPUS to 4064 because init_cpu_send_mondo_info() only allocates a single page for NR_CPUS mondo entries. Thus we cannot use all 4096 CPUs on some SPARC platforms.

To fix, allocate (2^order) pages where order is set according to the size of cpu_list for possible cpus. Since cpu_list_pa and cpu_mondo_block_pa are not used in asm code, there are no imm13 offsets from the base PA that will break because they can only reach one page.

Orabug: 25505750

---
  arch/sparc/Kconfig         |    4 ++--
  arch/sparc/kernel/irq_64.c |   16 ++++++++++++----
  2 files changed, 14 insertions(+), 6 deletions(-)
arch/sparc/Kconfig +2 -2
index 58243b0..4399be7 100644
@@ -192,9 +192,9 @@ config NR_CPUS
  	int "Maximum number of CPUs"
  	depends on SMP
  	range 2 32 if SPARC32
-	range 2 1024 if SPARC64
+	range 2 4096 if SPARC64
  	default 32 if SPARC32
-	default 64 if SPARC64
+	default 4096 if SPARC64
 
  source kernel/Kconfig.hz
 
arch/sparc/kernel/irq_64.c +1 -0
index 4d0248a..cc8f6c3 100644
@@ -1034,17 +1034,25 @@ static void __init init_cpu_send_mondo_info(struct trap_per_cpu *tb)
  {
  #ifdef CONFIG_SMP
  	unsigned long page;
+	void *mondo;
 
Quoted 13 lines

arch/sparc/kernel/irq_64.c static void __init init_cpu_send_mondo_info(struct trap_per_cpu *tb) · line 1047

Quoted 10 lines
arch/sparc/kernel/irq_64.c +9 -1
-	BUILD_BUG_ON((NR_CPUS * sizeof(u16)) > (PAGE_SIZE - 64));
+	BUILD_BUG_ON((NR_CPUS * sizeof(u16)) > PAGE_SIZE);
+
+	/* Make sure mondo block is 64byte aligned */
+	mondo = kzalloc(64, GFP_KERNEL);
+	if (!mondo) {
+		prom_printf("SUN4V: Error, cannot allocate mondo block.\n");
+		prom_halt();
+	}
+	tb->cpu_mondo_block_pa = __pa(mondo);

Hmmm, you said that this has to be 64 byte aligned right? We might have to do something in order to insure that, as kmalloc() only guarantees ARCH_KMALLOC_MINALIGN which I think is 8 on sparc.

Yes.

Quoted 6 lines

I suppose this would work:

	mondo = kzalloc(64 + 63, GFP_KERNEL);

and then 64-byte align that pointer.

On sun4v sparc, it looks like kzalloc(64, GFP_KERNEL) ends up allocating from kmalloc_caches[6] - a 64-byte kmem-cache allocated by kmem_cache_init() with SLAB_HWCACHE_ALIGN flag set, so it's in l3-cache-line-size
alignment, i.e. 64byte. I printed out the 'mondo' pa and verified that.

thanks!
-jane

maintainer
#3 lore raw

From: [email protected]
Date: Mon, 5 Jun 2017 20:03:28 -0700

Quoted 5 lines

On sun4v sparc, it looks like kzalloc(64, GFP_KERNEL) ends up allocating from kmalloc_caches[6] - a 64-byte kmem-cache allocated by kmem_cache_init() with SLAB_HWCACHE_ALIGN flag set, so it's in l3-cache-line-size alignment, i.e. 64byte. I printed out the 'mondo' pa and verified that.

Please check this, because that is an implementation detail. We have three SLAB allocators.

For example, if the user enables SLOB, I bet you don't get that alignment.

The only thing we are universally guaranteed is ARCH_KMALLOC_MINALIGN.

Please implement this properly, rather than relying upon things like the above.

Thank you.

#4 lore raw
Quoted 11 lines · David Miller

From: [email protected]
Date: Mon, 5 Jun 2017 20:03:28 -0700

Quoted 5 lines

On sun4v sparc, it looks like kzalloc(64, GFP_KERNEL) ends up allocating from kmalloc_caches[6] - a 64-byte kmem-cache allocated by kmem_cache_init() with SLAB_HWCACHE_ALIGN flag set, so it's in l3-cache-line-size alignment, i.e. 64byte. I printed out the 'mondo' pa and verified that.

Please check this, because that is an implementation detail. We have three SLAB allocators.

Quoted 7 lines

For example, if the user enables SLOB, I bet you don't get that alignment.

The only thing we are universally guaranteed is ARCH_KMALLOC_MINALIGN.

Please implement this properly, rather than relying upon things like the above.

Good point, agreed. I will update the patch.

thanks!
-jane

Quoted 2 lines

Thank you.