Sparsemem Virtual Memmap V4

Following this email is the current state of the Sparsemem Virtual Memory Map support (description below). This update represents fixes from testing as well as style checks plus changes based on review feedback. The major changes are the generic initialisation has been overhauled (the PMD initialisation was very x86 centric), configuration selection now works properly without SPARSEMEM, and preliminary PPC64 support has been added. Changelog is at the end of this email.

Still to sort out:

- ia64 support needs finalising (there are two implementations here)
- PPC64 needs review
- s390 support
- HOTPLUG review and regression testing

It is worth noting that the ia64 support exposes an essentially private Kconfig option to allow selection of the two implementations. Once the 16Mb support is complete it should become the one and only implementation and that this option would no longer be exposed.

This PPC64 implementation is all new. It slices a chunk (1/16th) off the end of the kernel linear space to hold the vmemmap. Need to make sure that this location makes sense.

I do not have performance data on this round of patches yet, but measurements on the initial PPC64 implementation showed a small but measurable improvement.

This stack is against v2.6.22-rc1-mm1. It has been compile, boot and lightly tested on x86_64, ia64 and PPC64. Sparc64 as been compiled but not booted.

-apw

===
SPARSEMEM is a pretty nice framework that unifies quite a bit of code over all the arches. It would be great if it could be the default so that we can get rid of various forms of DISCONTIG and other variations on memory maps. So far what has hindered this are the additional lookups that SPARSEMEM introduces for virt_to_page and page_address. This goes so far that the code to do this has to be kept in a separate function and cannot be used inline.

This patch introduces a virtual memmap mode for SPARSEMEM, in which the memmap is mapped into a virtually contigious area, only the active sections are physically backed. This allows virt_to_page page_address and cohorts become simple shift/add operations. No page flag fields, no table lookups, nothing involving memory is required.

The two key operations pfn_to_page and page_to_page become:

   #define __pfn_to_page(pfn)      (vmemmap + (pfn))
   #define __page_to_pfn(page)     ((page) - vmemmap)

By having a virtual mapping for the memmap we allow simple access without wasting physical memory. As kernel memory is typically already mapped 1:1 this introduces no additional overhead. The virtual mapping must be big enough to allow a struct page to be allocated and mapped for all valid physical pages. This vill make a virtual memmap difficult to use on 32 bit platforms that support 36 address bits.

However, if there is enough virtual space available and the arch already maps its 1-1 kernel space using TLBs (f.e. true of IA64 and x86_64) then this technique makes SPARSEMEM lookups even more efficient than CONFIG_FLATMEM. FLATMEM needs to read the contents of the mem_map variable to get the start of the memmap and then add the offset to the required entry. vmemmap is a constant to which we can simply add the offset.

This patch has the potential to allow us to make SPARSMEM the default (and even the only) option for most systems. It should be optimal on UP, SMP and NUMA on most platforms. Then we may even be able to remove the other memory models: FLATMEM, DISCONTIG etc.

V3->V4
- SPARC64 support -- from Dave Miller
- PPC64 support -- from Andy Whitcroft
- sparsemem precense/valid split
- rename Kconfig options into SPARSEMEM configuration name space
- redundant vmemmap alignment removed
- split out PMD support to x86_64
- x86_64 Kconfig dependancies
- ia64 Kconfig dependancies
- sparc64 dependancies, cleanup defines
- cleanup function names _pop_ -> _populate_
- markup __meminit
- cleanup style
- whitespace cleanups

V2->V3
- Add IA64 16M vmemmap size support (reduces TLB pressure)
- Add function to test for eventual node/node vmemmap overlaps
- Upper / Lower boundary fix.

V1->V2
- Support for PAGE_SIZE vmemmap which allows the general use of of virtual memmap on any MMU capable platform (enabled IA64 support).
- Fix various issues as suggested by Dave Hansen.
- Add comments and error handling.

---

index cb105a6..caa7e1b 100644
@@ -101,7 +101,7 @@ static inline int sparse_index_init(unsigned long section_nr, int nid)
 
 /*
  * Although written for the SPARSEMEM_EXTREME case, this happens
- * to also work for the flat array case becase
+ * to also work for the flat array case because
  * NR_SECTION_ROOTS==NR_MEM_SECTIONS.
  */
 int __section_nr(struct mem_section* ms)

We have flags to indicate whether a section actually has a valid mem_map associated with it. This is never set and we rely solely on the present bit to indicate a section is valid. By definition a section is not valid if it has no mem_map and there is a window during init where the present bit is set but there is no mem_map, during which pfn_valid() will return true incorrectly.

Use the existing SECTION_HAS_MEM_MAP flag to indicate the presence of a valid mem_map. Switch valid_section{,_nr} and pfn_valid() to this bit. Add a new present_section{,_nr} and pfn_present() interfaces for those users who care to know that a section is going to be valid.

---

index 74b9679..f1f0af8 100644
@@ -239,7 +239,7 @@ store_mem_state(struct sys_device *dev, const char *buf, size_t count)
 	mem = container_of(dev, struct memory_block, sysdev);
 	phys_section_nr = mem->phys_index;
 
-	if (!valid_section_nr(phys_section_nr))
+	if (!present_section_nr(phys_section_nr))
 		goto out;
 
 	if (!strncmp(buf, "online", min((int)count, 6)))

@@ -419,7 +419,7 @@ int register_new_memory(struct mem_section *section)
 
 int unregister_memory_section(struct mem_section *section)
 {
-	if (!valid_section(section))
+	if (!present_section(section))
 		return -EINVAL;
 
 	return remove_memory_block(0, section, 0);

@@ -444,7 +444,7 @@ int __init memory_dev_init(void)
 	 * during boot and have been initialized
 	 */
 	for (i = 0; i < NR_MEM_SECTIONS; i++) {
-		if (!valid_section_nr(i))
+		if (!present_section_nr(i))
 			continue;
 		err = add_memory_block(0, __nr_to_section(i), MEM_ONLINE, 0);
 		if (!ret)

index 6609481..5a3ea4d 100644
@@ -781,12 +781,17 @@ static inline struct page *__section_mem_map_addr(struct mem_section *section)
 	return (struct page *)map;
 }
 
-static inline int valid_section(struct mem_section *section)
+static inline int present_section(struct mem_section *section)
 {
 	return (section && (section->section_mem_map & SECTION_MARKED_PRESENT));
 }
 
-static inline int section_has_mem_map(struct mem_section *section)
+static inline int present_section_nr(unsigned long nr)
+{
+	return present_section(__nr_to_section(nr));
+}
+
+static inline int valid_section(struct mem_section *section)
 {
 	return (section && (section->section_mem_map & SECTION_HAS_MEM_MAP));
 }

@@ -808,6 +813,13 @@ static inline int pfn_valid(unsigned long pfn)
 	return valid_section(__nr_to_section(pfn_to_section_nr(pfn)));
 }
 
+static inline int pfn_present(unsigned long pfn)
+{
+        if (pfn_to_section_nr(pfn) >= NR_MEM_SECTIONS)
+                return 0;
+        return present_section(__nr_to_section(pfn_to_section_nr(pfn)));
+}
+
 /*
  * These are _only_ used during initialisation, therefore they
  * can use __initdata ...  They could have names to indicate

index caa7e1b..d64e628 100644
@@ -170,7 +170,7 @@ unsigned long __init node_memmap_size_bytes(int nid, unsigned long start_pfn,
 		if (nid != early_pfn_to_nid(pfn))
 			continue;
 
-		if (pfn_valid(pfn))
+		if (pfn_present(pfn))
 			nr_pages += PAGES_PER_SECTION;
 	}
 

@@ -201,11 +201,12 @@ static int __meminit sparse_init_one_section(struct mem_section *ms,
 		unsigned long pnum, struct page *mem_map,
 		unsigned long *pageblock_bitmap)
 {
-	if (!valid_section(ms))
+	if (!present_section(ms))
 		return -EINVAL;
 
 	ms->section_mem_map &= ~SECTION_MAP_MASK;
-	ms->section_mem_map |= sparse_encode_mem_map(mem_map, pnum);
+	ms->section_mem_map |= sparse_encode_mem_map(mem_map, pnum) |
+							SECTION_HAS_MEM_MAP;
 	ms->pageblock_flags = pageblock_bitmap;
 
 	return 1;

@@ -308,7 +309,7 @@ void __init sparse_init(void)
 	unsigned long *usemap;
 
 	for (pnum = 0; pnum < NR_MEM_SECTIONS; pnum++) {
-		if (!valid_section_nr(pnum))
+		if (!present_section_nr(pnum))
 			continue;
 
 		map = sparse_early_mem_map_alloc(pnum);

SPARSEMEM is a pretty nice framework that unifies quite a bit of code over all the arches. It would be great if it could be the default so that we can get rid of various forms of DISCONTIG and other variations on memory maps. So far what has hindered this are the additional lookups that SPARSEMEM introduces for virt_to_page and page_address. This goes so far that the code to do this has to be kept in a separate function and cannot be used inline.

This patch introduces a virtual memmap mode for SPARSEMEM, in which the memmap is mapped into a virtually contigious area, only the active sections are physically backed. This allows virt_to_page page_address and cohorts become simple shift/add operations. No page flag fields, no table lookups, nothing involving memory is required.

The two key operations pfn_to_page and page_to_page become:

   #define __pfn_to_page(pfn)      (vmemmap + (pfn))
   #define __page_to_pfn(page)     ((page) - vmemmap)

By having a virtual mapping for the memmap we allow simple access without wasting physical memory. As kernel memory is typically already mapped 1:1 this introduces no additional overhead. The virtual mapping must be big enough to allow a struct page to be allocated and mapped for all valid physical pages. This vill make a virtual memmap difficult to use on 32 bit platforms that support 36 address bits.

However, if there is enough virtual space available and the arch already maps its 1-1 kernel space using TLBs (f.e. true of IA64 and x86_64) then this technique makes SPARSEMEM lookups even more efficient than CONFIG_FLATMEM. FLATMEM needs to read the contents of the mem_map variable to get the start of the memmap and then add the offset to the required entry. vmemmap is a constant to which we can simply add the offset.

This patch has the potential to allow us to make SPARSMEM the default (and even the only) option for most systems. It should be optimal on UP, SMP and NUMA on most platforms. Then we may even be able to remove the other memory models: FLATMEM, DISCONTIG etc.

[[email protected]: config cleanups, resplit code etc]
From: Christoph Lameter <[email protected]>

---

index 30d8d33..52226e1 100644
@@ -46,6 +46,12 @@
 	 __pgdat->node_start_pfn;					\
 })
 
+#elif defined(CONFIG_SPARSEMEM_VMEMMAP)
+
+/* memmap is virtually contigious.  */
+#define __pfn_to_page(pfn)	(vmemmap + (pfn))
+#define __page_to_pfn(page)	((page) - vmemmap)
+
 #elif defined(CONFIG_SPARSEMEM)
 /*
  * Note: section's mem_map is encorded to reflect its start_pfn.

index 8c7b7d3..d440ed5 100644
@@ -1254,5 +1254,10 @@ extern int randomize_va_space;
 
 __attribute__((weak)) const char *arch_vma_name(struct vm_area_struct *vma);
 
+int vmemmap_populate(struct page *start_page, unsigned long pages, int node);
+int vmemmap_populate_pmd(pud_t *, unsigned long, unsigned long, int);
+void *vmemmap_alloc_block(unsigned long size, int node);
+void vmemmap_verify(pte_t *, int, unsigned long, unsigned long);
+
 #endif /* __KERNEL__ */
 #endif /* _LINUX_MM_H */

I get a couple of warnings:

mm/sparse.c:423: warning: '__kmalloc_section_memmap' defined but not used mm/sparse.c:453: warning: '__kfree_section_memmap' defined but not used

when building on IA64

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

This is fixed by my patch "Move three functions that are only needed for CONFIG_MEMORY_HOTPLUG" which I posted yesterday and is in Andrew's tree. Sorry for not cc'ing linux-mm.
--
Cheers,
Stephen Rothwell [email protected]
http://www.canb.auug.org.au/~sfr/

x86_64 uses 2M page table entries to map its 1-1 kernel space. We also implement the virtual memmap using 2M page table entries. So there is no additional runtime overhead over FLATMEM, initialisation is slightly more complex. As FLATMEM still references memory to obtain the mem_map pointer and SPARSEMEM_VMEMMAP uses a compile time constant, SPARSEMEM_VMEMMAP should be superior.

With this SPARSEMEM becomes the most efficient way of handling virt_to_page, pfn_to_page and friends for UP, SMP and NUMA on x86_64.

[[email protected]: code resplit, style fixups]
From: Christoph Lameter <[email protected]>

---

index f42798e..b89b6d2 100644
@@ -9,6 +9,7 @@ ffff800000000000 - ffff80ffffffffff (=40 bits) guard hole
 ffff810000000000 - ffffc0ffffffffff (=46 bits) direct mapping of all phys. memory
 ffffc10000000000 - ffffc1ffffffffff (=40 bits) hole
 ffffc20000000000 - ffffe1ffffffffff (=45 bits) vmalloc/ioremap space
+ffffe20000000000 - ffffe2ffffffffff (=40 bits) virtual memory map (1TB)
 ... unused hole ...
 ffffffff80000000 - ffffffff82800000 (=40 MB)   kernel text mapping, from phys 0
 ... unused hole ...

index 2926670..029815a 100644
@@ -400,6 +400,14 @@ config ARCH_SPARSEMEM_ENABLE
 	def_bool y
 	depends on (NUMA || EXPERIMENTAL)
 
+config SPARSEMEM_VMEMMAP
+	def_bool y
+	depends on SPARSEMEM
+
+config ARCH_POPULATES_SPARSEMEM_VMEMMAP_PMD
+	def_bool y
+	depends on SPARSEMEM_VMEMMAP
+
 config ARCH_MEMORY_PROBE
 	def_bool y
 	depends on MEMORY_HOTPLUG

index 58db7af..a7591d3 100644
@@ -770,3 +770,33 @@ const char *arch_vma_name(struct vm_area_struct *vma)
 		return "[vsyscall]";
 	return NULL;
 }
+
+#ifdef CONFIG_ARCH_POPULATES_SPARSEMEM_VMEMMAP_PMD
+/*
+ * Initialise the sparsemem vmemmap using huge-pages at the PMD level.
+ */
+int __meminit vmemmap_populate_pmd(pud_t *pud, unsigned long addr,
+						unsigned long end, int node)
+{
+	pmd_t *pmd;
+
+	for (pmd = pmd_offset(pud, addr); addr < end;
+						pmd++, addr += PMD_SIZE)
+		if (pmd_none(*pmd)) {
+			pte_t entry;
+			void *p = vmemmap_alloc_block(PMD_SIZE, node);
+			if (!p)
+				return -ENOMEM;
+
+			entry = pfn_pte(__pa(p) >> PAGE_SHIFT, PAGE_KERNEL);
+			mk_pte_huge(entry);
+			set_pmd(pmd, __pmd(pte_val(entry)));
+
+			printk(KERN_DEBUG " [%lx-%lx] PMD ->%p on node %d\n",
+				addr, addr + PMD_SIZE - 1, p, node);
+		} else
+			vmemmap_verify((pte_t *)pmd, node,
+						pmd_addr_end(addr, end), end);
+	return 0;
+}
+#endif

index 88adf1a..c3b52bc 100644
@@ -134,6 +134,7 @@ extern unsigned long __phys_addr(unsigned long);
 	 VM_READ | VM_WRITE | VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC)
 
 #define __HAVE_ARCH_GATE_AREA 1	
+#define vmemmap ((struct page *)VMEMMAP_START)
 
 #include <asm-generic/memory_model.h>
 #include <asm-generic/page.h>

index 44d07bf..15f0003 100644
@@ -137,6 +137,7 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, unsigned long
 #define MAXMEM		 _AC(0x3fffffffffff, UL)
 #define VMALLOC_START    _AC(0xffffc20000000000, UL)
 #define VMALLOC_END      _AC(0xffffe1ffffffffff, UL)
+#define VMEMMAP_START	 _AC(0xffffe20000000000, UL)
 #define MODULES_VADDR    _AC(0xffffffff88000000, UL)
 #define MODULES_END      _AC(0xfffffffffff00000, UL)
 #define MODULES_LEN   (MODULES_END - MODULES_VADDR)

Equip IA64 sparsemem with a virtual memmap. This is similar to the existing CONFIG_VIRTUAL_MEM_MAP functionality for DISCONTIGMEM. It uses a PAGE_SIZE mapping.

This is provided as a minimally intrusive solution. We split the 128TB VMALLOC area into two 64TB areas and use one for the virtual memmap.

This should replace CONFIG_VIRTUAL_MEM_MAP long term.

From: Christoph Lameter <[email protected]>

---

index de1bff6..9d0d101 100644
@@ -355,6 +355,10 @@ config ARCH_SPARSEMEM_ENABLE
 	def_bool y
 	depends on ARCH_DISCONTIGMEM_ENABLE
 
+config SPARSEMEM_VMEMMAP
+	def_bool y
+	depends on SPARSEMEM
+
 config ARCH_DISCONTIGMEM_DEFAULT
 	def_bool y if (IA64_SGI_SN2 || IA64_GENERIC || IA64_HP_ZX1 || IA64_HP_ZX1_SWIOTLB)
 	depends on ARCH_DISCONTIGMEM_ENABLE

index 670b706..366c34b 100644
@@ -236,8 +236,14 @@ ia64_phys_addr_valid (unsigned long addr)
 # define VMALLOC_END		vmalloc_end
   extern unsigned long vmalloc_end;
 #else
+#if defined(CONFIG_SPARSEMEM) && defined(CONFIG_SPARSEMEM_VMEMMAP)
+/* SPARSEMEM_VMEMMAP uses half of vmalloc... */
+# define VMALLOC_END		(RGN_BASE(RGN_GATE) + (1UL << (4*PAGE_SHIFT - 10)))
+# define vmemmap		((struct page *)VMALLOC_END)
+#else
 # define VMALLOC_END		(RGN_BASE(RGN_GATE) + (1UL << (4*PAGE_SHIFT - 9)))
 #endif
+#endif
 
 /* fs/proc/kcore.c */
 #define	kc_vaddr_to_offset(v) ((v) - RGN_BASE(RGN_GATE))

Works fine as is with patch on top of 2.6.22-rc2 here...

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

This implements granule page sized vmemmap support for IA64. This is important because the traditional vmemmap on IA64 uses page size for mapping the TLB. For a typical 8GB node on IA64 we need about (33 - 14 + 6 = 25) = 32 MB of page structs.

Using page size we will end up with (25 - 14 = 11) 2048 page table entries.

This patch will reduce this to two 16MB TLBs. So its a factor of 1000 less TLBs for the virtual memory map.

We modify the alt_dtlb_miss handler to branch to a vmemmap TLB lookup function if bit 60 is set. The vmemmap will start with 0xF000xxx so its going be very distinctive in dumps and can be distinguished easily from 0xE000xxx (kernel 1-1 area) and 0xA000xxx (kernel text, data and vmalloc).

We use a 1 level page table to do lookups for the vmemmap TLBs. Since we need to cover 1 Petabyte we need to reserve 1 megabyte just for the table but we can statically allocate it in the data segment. This simplifies lookups and handling. The fault handler only has to do a single lookup in contrast to 4 for the current vmalloc/vmemmap implementation.

Problems with this patchset are:

1. Large 1M array required to cover all of possible memory (1 Petabyte). Maybe reduce this to actually supported HW sizes? 16TB or 64TB?

2. For systems with small nodes there is a significant chance of large overlaps. We could dynamically determine the TLB size but that would make the code more complex.

[[email protected]: style fixups]
From: Christoph Lameter <[email protected]>

---

index 9d0d101..e8fc8e3 100644
@@ -359,6 +359,18 @@ config SPARSEMEM_VMEMMAP
 	def_bool y
 	depends on SPARSEMEM
 
+config ARCH_POPULATES_SPARSEMEM_VMEMMAP
+	bool "Use 16M TLB for virtual memory map"
+	default y
+	depends on SPARSEMEM_VMEMMAP
+	help
+	  Enables large page virtual memmap support. Each virtual memmap
+	  page will be 16MB in size. That size of vmemmap can cover 4GB
+	  of memory. We only use a single TLB per node. However, if nodes
+	  are small and the distance between the memory of the nodes is
+	  < 4GB then the page struct for some of the early pages in the
+	  node may end up on the prior node.
+
 config ARCH_DISCONTIGMEM_DEFAULT
 	def_bool y if (IA64_SGI_SN2 || IA64_GENERIC || IA64_HP_ZX1 || IA64_HP_ZX1_SWIOTLB)
 	depends on ARCH_DISCONTIGMEM_ENABLE

index 34f44d8..b6deaf7 100644
@@ -391,9 +391,11 @@ ENTRY(alt_dtlb_miss)
 	tbit.z p12,p0=r16,61			// access to region 6?
 	mov r25=PERCPU_PAGE_SHIFT << 2
 	mov r26=PERCPU_PAGE_SIZE
-	nop.m 0
-	nop.b 0
+	tbit.nz p6,p0=r16,60			// Access to VMEMMAP?
+(p6)	br.cond.dptk vmemmap
 	;;
+dtlb_continue:
+	.pred.rel "mutex", p11, p10
 (p10)	mov r19=IA64_KR(PER_CPU_DATA)
 (p11)	and r19=r19,r16				// clear non-ppn fields
 	extr.u r23=r21,IA64_PSR_CPL0_BIT,2	// extract psr.cpl

@@ -416,6 +418,37 @@ ENTRY(alt_dtlb_miss)
 (p7)	itc.d r19		// insert the TLB entry
 	mov pr=r31,-1
 	rfi
+
+vmemmap:
+	//
+	// Granule lookup via vmemmap_table for
+	// the virtual memory map.
+	//
+	tbit.nz p6,p0=r16,59			// more top bits set?
+(p6)	br.cond.spnt dtlb_continue		// then its mmu bootstrap
+	;;
+	rsm psr.dt				// switch to using physical data addressing
+	extr.u r25=r16, IA64_GRANULE_SHIFT, 32
+	;;
+	srlz.d
+	LOAD_PHYSICAL(p0, r26, vmemmap_table)
+	shl r25=r25,2
+	;;
+	add r26=r26,r25				// Index into vmemmap table
+	;;
+	ld4 r25=[r26]				// Get 32 bit descriptor */
+	;;
+	dep.z r19=r25, 0, 31			// Isolate ppn
+	tbit.z p6,p0=r25, 31			// Present bit set?
+(p6)	br.cond.spnt page_fault			// Page not present
+	;;
+	shl r19=r19, IA64_GRANULE_SHIFT		// Shift ppn in place
+	;;
+	or r19=r19,r17		// insert PTE control bits into r19
+	;;
+	itc.d r19		// insert the TLB entry
+	mov pr=r31,-1
+	rfi
 END(alt_dtlb_miss)
 
 	.org ia64_ivt+0x1400

index 366c34b..f4aab5d 100644
@@ -236,7 +236,8 @@ ia64_phys_addr_valid (unsigned long addr)
 # define VMALLOC_END		vmalloc_end
   extern unsigned long vmalloc_end;
 #else
-#if defined(CONFIG_SPARSEMEM) && defined(CONFIG_SPARSEMEM_VMEMMAP)
+#if defined(CONFIG_SPARSEMEM) && defined(CONFIG_SPARSEMEM_VMEMMAP) && \
+			!defined(CONFIG_ARCH_POPULATES_SPARSEMEM_VMEMMAP)
 /* SPARSEMEM_VMEMMAP uses half of vmalloc... */
 # define VMALLOC_END		(RGN_BASE(RGN_GATE) + (1UL << (4*PAGE_SHIFT - 10)))
 # define vmemmap		((struct page *)VMALLOC_END)

@@ -245,6 +246,11 @@ ia64_phys_addr_valid (unsigned long addr)
 #endif
 #endif
 
+#ifdef CONFIG_ARCH_POPULATES_SPARSEMEM_VMEMMAP
+# define VMEMMAP_FLAG (1UL << 60)
+# define vmemmap ((struct page *)(RGN_BASE(RGN_KERNEL) | VMEMMAP_FLAG))
+#endif
+
 /* fs/proc/kcore.c */
 #define	kc_vaddr_to_offset(v) ((v) - RGN_BASE(RGN_GATE))
 #define	kc_offset_to_vaddr(o) ((o) + RGN_BASE(RGN_GATE))

Hey Christoph, here is sparc64 support for this stuff.

After implementing this and seeing more and more how it works, I really like it :-)

Thanks a lot for doing this work Christoph!

[[email protected]: style fixups]
From: David Miller <[email protected]>

---

index 831781c..a81979d 100644
@@ -224,10 +224,17 @@ config ARCH_SPARSEMEM_ENABLE
 
 config ARCH_SPARSEMEM_DEFAULT
 	def_bool y
-	select SPARSEMEM_STATIC
 
 source "mm/Kconfig"
 
+config SPARSEMEM_VMEMMAP
+	def_bool y
+	depends on SPARSEMEM
+
+config ARCH_POPULATES_SPARSEMEM_VMEMMAP
+	def_bool y
+	depends on SPARSEMEM_VMEMMAP
+
 config ISA
 	bool
 	help

index d4024ac..964527d 100644
@@ -226,6 +226,15 @@ kvmap_dtlb_load:
 	ba,pt		%xcc, sun4v_dtlb_load
 	 mov		%g5, %g3
 
+kvmap_vmemmap:
+	sub		%g4, %g5, %g5
+	srlx		%g5, 22, %g5
+	sethi		%hi(vmemmap_table), %g1
+	sllx		%g5, 3, %g5
+	or		%g1, %lo(vmemmap_table), %g1
+	ba,pt		%xcc, kvmap_dtlb_load
+	 ldx		[%g1 + %g5], %g5
+
 kvmap_dtlb_nonlinear:
 	/* Catch kernel NULL pointer derefs.  */
 	sethi		%hi(PAGE_SIZE), %g5

@@ -233,6 +242,13 @@ kvmap_dtlb_nonlinear:
 	bleu,pn		%xcc, kvmap_dtlb_longpath
 	 nop
 
+	/* Do not use the TSB for vmemmap.  */
+	mov		(VMEMMAP_BASE >> 24), %g5
+	sllx		%g5, 24, %g5
+	cmp		%g4,%g5
+	bgeu,pn		%xcc, kvmap_vmemmap
+	 nop
+
 	KERN_TSB_LOOKUP_TL1(%g4, %g6, %g5, %g1, %g2, %g3, kvmap_dtlb_load)
 
 kvmap_dtlb_tsbmiss:

index 9e80ad4..5f5d9fb 100644
@@ -42,6 +42,9 @@
 #define HI_OBP_ADDRESS		_AC(0x0000000100000000,UL)
 #define VMALLOC_START		_AC(0x0000000100000000,UL)
 #define VMALLOC_END		_AC(0x0000000200000000,UL)
+#define VMEMMAP_BASE		_AC(0x0000000200000000,UL)
+
+#define vmemmap			((struct page *)VMEMMAP_BASE)
 
 /* XXX All of this needs to be rethought so we can take advantage
  * XXX cheetah's full 64-bit virtual address space, ie. no more hole

Enable virtual memmap support for SPARSEMEM on PPC64 systems. Slice a 16th off the end of the linear mapping space and use that to hold the vmemmap. Uses the same size mapping as uses in the linear 1:1 kernel mapping.

---

index 56d3c0d..282838c 100644
@@ -523,6 +523,14 @@ config ARCH_POPULATES_NODE_MAP
 
 source "mm/Kconfig"
 
+config SPARSEMEM_VMEMMAP
+	def_bool y
+	depends on SPARSEMEM
+
+config ARCH_POPULATES_SPARSEMEM_VMEMMAP
+	def_bool y
+	depends on SPARSEMEM_VMEMMAP
+
 config ARCH_MEMORY_PROBE
 	def_bool y
 	depends on MEMORY_HOTPLUG

index 704c4e6..5943378 100644
@@ -63,6 +63,14 @@ struct mm_struct;
 #define USER_REGION_ID		(0UL)
 
 /*
+ * Defines the address of the vmemap area, in the top 16th of the
+ * kernel region.
+ */
+#define VMEMMAP_BASE (ASM_CONST(CONFIG_KERNEL_START) + \
+					(0xfUL << (REGION_SHIFT - 4)))
+#define vmemmap ((struct page *)VMEMMAP_BASE)
+
+/*
  * Common bits in a linux-style PTE.  These match the bits in the
  * (hardware-defined) PowerPC PTE as closely as possible. Additional
  * bits may be defined in pgtable-*.h

Right. You can omit 16MB support for the next round. We agreed with the other IA64 people that 16MB is too large and want to shoot for 4MB. The 16k support should be sufficient for this patchset.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Well the performance tests that I did on x86_64 showed a reduction of the performance of virt_to_page from 18us to 9us. So I think we are fine.

Can we get that into mm soon? There are potentially other arches that also may want to run their own vmemmap functions for this and I would like to have an easy way to tinker around with a 4M vmemmap size for IA64.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

We will need this fix to sparsemem on IA64. I hope this will not cause other issues in sparsemem?

IA64: Increase maximum physmem size to cover 8 petabyte

We currently can support these large configurations only with Discontigmem.

Increase sparsemems max physmem bits to also be able to handle 8 petabyte.

Discontigmem supports up to 16 petabyte but I will need to use bit 53 to flag vmemmap addresses for the TLB handler. It seems that the currently used bit 60 for the 16M configuration is not supported by a certain virtualization technique.

Index: linux-2.6.22-rc2/include/asm-ia64/sparsemem.h
=================================--- linux-2.6.22-rc2.orig/include/asm-ia64/sparsemem.h 2007-05-22 17:28:04.000000000 -0700
+++ linux-2.6.22-rc2/include/asm-ia64/sparsemem.h 2007-05-22 17:28:37.000000000 -0700
@@ -8,7 +8,7 @@
*/

#define SECTION_SIZE_BITS (30)
-#define MAX_PHYSMEM_BITS (50)
+#define MAX_PHYSMEM_BITS (53)
#ifdef CONFIG_FORCE_MAX_ZONEORDER
#if ((CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS) #undef SECTION_SIZE_BITS

From: Andy Whitcroft <[email protected]>
Date: Tue, 22 May 2007 23:57:55 +0100

Sparc64 boot tested successfully on Niagara t1000 with 26 cpus.