Fixmap mapping region is not enough for system of 14+ CPUs.

ARM Liu hua 2 participants 4 messages Mar 12, 2014 – Mar 13, 2014 1 message not in the archive Arm (32-bit)

lore

Share & tools

lore

Liu hua

author

2014-03-12 11:48 UTC #1 lore raw

In reply to a message not in the archive

Hi Russell, Will or Nicolas,

(In this mail, we only discuss ARM 32-bit linux.)

As we know, the region (0xfff00000-0xfffdffff) is reserved as fixmap mapping region.

The function "kmap_atomic" maps highmem pages to this region referring to CPUID and per-cpu variable "__kmap_atomic_idx" via

  idx = type + KM_TYPE_NR * smp_processor_id();
  vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx);

Size of region used by one cpu is 0x10000 (KM_TYPE_NR << PAGE_SHIFT). And the total size of the fixmap mapping region is 0xe0000. (only support 14 CPUs).

So in a system of more than 14 CPUs, this region is not large enough. should we change the memory layout on ARM Linux to support 14+ cpu system ? Or can we do anything else to support that ?

Thanks,

Liu Hua

Liu hua

author

2014-03-13 07:34 UTC #3 lore raw

Quoted 35 lines · Nicolas Pitre

Quoted 13 lines · Liu hua
Hi Russell, Will or Nicolas,
(In this mail, we only discuss ARM 32-bit linux.)
As we know, the region (0xfff00000-0xfffdffff) is reserved as fixmap mapping region.
The function "kmap_atomic" maps highmem pages to this region referring to CPUID and per-cpu variable "__kmap_atomic_idx" via
  idx = type + KM_TYPE_NR * smp_processor_id();
  vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
Quoted 7 lines
Size of region used by one cpu is 0x10000 (KM_TYPE_NR << PAGE_SHIFT). And the total size of the fixmap mapping region is 0xe0000. (only support 14 CPUs).
So in a system of more than 14 CPUs, this region is not large enough. should we change the memory layout on ARM Linux to support 14+ cpu system ? Or can we do anything else to support that ?
How many CPUs do you have?
What about the following patch? If this doesn't work for you then more intrusive changes will be needed.
arch/arm/include/asm/fixmap.h +0 -0
index 68ea615c2a..254f2df08d 100644
@@ -14,7 +14,17 @@
  */
 

arch/arm/include/asm/fixmap.h line 25

Quoted 10 lines

arch/arm/include/asm/fixmap.h +8 -0

 #define FIXADDR_START		0xfff00000UL
+
+#if !defined(CONFIG_HAVE_TCM) && !defined(CONFIG_CPU_XSCALE)
+/*
+ * If no TCM nor on on a XScale then enlarge the fixmap area to
+ * accommodate up to 30 CPUs.
+ */
+#define FIXADDR_END		0xffff0000UL
+#else
 #define FIXADDR_END		0xfffe0000UL

One cpu need 0x10000-size region; So this patch can only support uo to 15 CPUs

Quoted 9 lines

arch/arm/include/asm/fixmap.h +2 -0

+#endif
+
 #define FIXADDR_TOP		(FIXADDR_END - PAGE_SIZE)
 
 enum fixed_addresses {

Nicolas

We have 16 CPUs in our system. And your patch cannot work if the CPU number exceeds 15.

In addition, we can configure NR_CPUS up to 32. So should we solve this problem completely?

There is an 1M-size hole between DMA mapping region and fixmap mapping region(0xffe00000-0xfff00000).if this region are belonged to fixmap mapping region, it can only support 30 CPUs.

Since we have alloced a second level page table to coverring 0xffe00000 - 0xffffffff(2M). And 0xffff0000 upwards used as vector pages. So if the left region is all used as fixmap mapping region(size : 0x1f0000). Up to 31 CPUs is supported.

The following patch can solve our problem(up to 30 CPUs), But it reduce the DMA region. What do you think about it ?
Is is necessory to solve this problem completely(up to 32 CPUs) ?

Liu Hua

-----------------

arch/arm/include/asm/fixmap.h +1 -1

index 68ea615..8379891 100644
@@ -13,7 +13,7 @@
  * 0xfffe0000 and 0xfffeffff.
  */
 
-#define FIXADDR_START              0xfff00000UL
+#define FIXADDR_START                0xffe00000UL
 #define FIXADDR_END            0xfffe0000UL
 #define FIXADDR_TOP            (FIXADDR_END - PAGE_SIZE)

arch/arm/include/asm/memory.h +1 -1

index 4afb376..0c40674 100644
@@ -83,7 +83,7 @@
  */
 #define IOREMAP_MAX_ORDER      24
 
-#define CONSISTENT_END             (0xffe00000UL)
+#define CONSISTENT_END               (0xffd00000UL)
 
 #else /* CONFIG_MMU */

Nicolas Pitre

2014-03-13 15:01 UTC #4 lore raw

Quoted 36 lines · Liu hua

Quoted 35 lines · Nicolas Pitre
Quoted 13 lines · Liu hua
Hi Russell, Will or Nicolas,
(In this mail, we only discuss ARM 32-bit linux.)
As we know, the region (0xfff00000-0xfffdffff) is reserved as fixmap mapping region.
The function "kmap_atomic" maps highmem pages to this region referring to CPUID and per-cpu variable "__kmap_atomic_idx" via
  idx = type + KM_TYPE_NR * smp_processor_id();
  vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
Quoted 7 lines
Size of region used by one cpu is 0x10000 (KM_TYPE_NR << PAGE_SHIFT). And the total size of the fixmap mapping region is 0xe0000. (only support 14 CPUs).
So in a system of more than 14 CPUs, this region is not large enough. should we change the memory layout on ARM Linux to support 14+ cpu system ? Or can we do anything else to support that ?
How many CPUs do you have?
What about the following patch? If this doesn't work for you then more intrusive changes will be needed.
arch/arm/include/asm/fixmap.h +0 -0
index 68ea615c2a..254f2df08d 100644
@@ -14,7 +14,17 @@
  */
 

Quoted 12 lines

arch/arm/include/asm/fixmap.h line 25
Quoted 10 lines
arch/arm/include/asm/fixmap.h +8 -0
 #define FIXADDR_START		0xfff00000UL
+
+#if !defined(CONFIG_HAVE_TCM) && !defined(CONFIG_CPU_XSCALE)
+/*
+ * If no TCM nor on on a XScale then enlarge the fixmap area to
+ * accommodate up to 30 CPUs.
+ */
+#define FIXADDR_END		0xffff0000UL
+#else
 #define FIXADDR_END		0xfffe0000UL
One cpu need 0x10000-size region; So this patch can only support uo to 15 CPUs

Hmmm... You're right of course.

Quoted 43 lines

We have 16 CPUs in our system. And your patch cannot work if the CPU number exceeds 15.
In addition, we can configure NR_CPUS up to 32. So should we solve this problem completely?
There is an 1M-size hole between DMA mapping region and fixmap mapping region(0xffe00000-0xfff00000).if this region are belonged to fixmap mapping region, it can only support 30 CPUs.
Since we have alloced a second level page table to coverring 0xffe00000 - 0xffffffff(2M). And 0xffff0000 upwards used as vector pages. So if the left region is all used as fixmap mapping region(size : 0x1f0000). Up to 31 CPUs is supported.
The following patch can solve our problem(up to 30 CPUs), But it reduce the DMA region. What do you think about it ?
Is is necessory to solve this problem completely(up to 32 CPUs) ?
Liu Hua
-----------------
arch/arm/include/asm/fixmap.h +1 -1
index 68ea615..8379891 100644
@@ -13,7 +13,7 @@
  * 0xfffe0000 and 0xfffeffff.
  */
 
-#define FIXADDR_START              0xfff00000UL
+#define FIXADDR_START                0xffe00000UL
 #define FIXADDR_END            0xfffe0000UL
 #define FIXADDR_TOP            (FIXADDR_END - PAGE_SIZE)
 
arch/arm/include/asm/memory.h +0 -0
index 4afb376..0c40674 100644

arch/arm/include/asm/memory.h line 86

Quoted 5 lines

arch/arm/include/asm/memory.h +1 -1

@@ -83,7 +83,7 @@
  */
 #define IOREMAP_MAX_ORDER      24
 
-#define CONSISTENT_END             (0xffe00000UL)
+#define CONSISTENT_END               (0xffd00000UL)

Some systems might possibly have issues with a smaller DMA region. I don't recall which though.

you should also update Documentation/arm/memory.txt in your patch. Then I'd suggest you can repost it more formally for comments.

Nicolas