Skip to contentSkip to conversation

long format VHPT

PATCH In review Darren Williams 6 participants 11 messages Sep 26, 2003 – Oct 2, 2003 Kbuild

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 long format VHPT

Hi
This is a patch that takes work originally performed by Mathew Chapman from UNSW. Matts work was based on the 2.5 series kernel and this patch brings it up to date with 2.6.

What the patch is
1) Allows configuration selection between long and short VHPT formats.
2) Moves some orignal code into macros and attempts to use less/different registers.
3) Pushes some of Matts original work into arch/ia64/asmmacros.h
4) Ethier runs the original code written by David.M. or Matts modifications.

Testing
1) Compiled on HP Simulator, Itanium 1 and 2.
2) Run on HP Sim, Itanium 1 and 2
3) lmbench on Itanium 2 dual processor, results to be presented at www.gelato.unsw.edu.au.
4) Waiting for HP to release Performance monitor tools for 2.6 to complete testing.

Thanks

--------------------------------------------------
Darren Williams <[email protected]>
Gelato@UNSW <www.gelato.unsw.edu.au>
--------------------------------------------------

# This is a BitKeeper generated patch for the following project:
# Project Name: Linux kernel tree
# This patch format is intended for GNU patch command version 2.5 or higher. # This patch includes the following deltas:
# ChangeSet v2.6.0-test5 -> 1.1241
# include/asm-ia64/mmu_context.h 1.15 -> 1.16
# include/asm-ia64/page.h 1.21 -> 1.22
# arch/ia64/kernel/ivt.S 1.23 -> 1.26
# include/asm-ia64/kregs.h 1.6 -> 1.7
# include/asm-ia64/asmmacro.h 1.12 -> 1.13
# include/asm-ia64/pgtable.h 1.28 -> 1.29
# include/asm-ia64/processor.h 1.50 -> 1.52
# include/asm-ia64/tlb.h 1.16 -> 1.17
# arch/ia64/mm/init.c 1.47 -> 1.49
# Makefile 1.427 -> 1.428
# arch/ia64/kernel/patch.c 1.5 -> 1.7
# arch/ia64/mm/tlb.c 1.19 -> 1.20
# include/asm-ia64/tlbflush.h 1.9 -> 1.10
# arch/ia64/Kconfig 1.41 -> 1.42
# arch/ia64/kernel/traps.c 1.37 -> 1.38
# arch/ia64/mm/fault.c 1.15 -> 1.16
#
# The following is the BitKeeper ChangeSet Log
# --------------------------------------------
# 03/09/08 [email protected] 1.1238
# Linux 2.6.0-test5
# --------------------------------------------
# 03/09/25 dsw@quasar.(none) 1.1239
# Test DEBUG vhpt patch
# --------------------------------------------
# 03/09/25 dsw@quasar.(none) 1.1240
# Tidied up MACROS from ivt.S and put them into include/asmmacro.h, # Removed debugging printks'
# --------------------------------------------
# 03/09/25 dsw@quasar.(none) 1.1241

# remove more debug stuff
# --------------------------------------------
#
diff -Nru a/Makefile b/Makefile
--- a/Makefile	Thu Sep 25 11:09:40 2003
+++ b/Makefile	Thu Sep 25 11:09:40 2003
@@ -1,7 +1,7 @@
 VERSION = 2
 PATCHLEVEL = 6
 SUBLEVEL = 0
-EXTRAVERSION = -test5
+EXTRAVERSION = -test5-vhpt

# *DOCUMENTATION*
# To see a list of typical targets execute "make help"

arch/ia64/Kconfig +23 -1
diff -Nru a/arch/ia64/Kconfig b/arch/ia64/Kconfig
@@ -676,7 +676,29 @@
 	  debugging info resulting in a larger kernel image.
 	  Say Y here only if you plan to use gdb to debug the kernel.
 	  If you don't debug the kernel, you can say N.
-	  
+
+config IA64_LONG_FORMAT_VHPT
+	bool "Long format VHPT"
+
+choice
+	prompt "Long format VHPT size"
+	depends on IA64_LONG_FORMAT_VHPT
+	default IA64_LONG_FORMAT_VHPT_1MB
+
+config IA64_LONG_FORMAT_VHPT_64KB
+	bool "64KB"
+
+config IA64_LONG_FORMAT_VHPT_256KB
+	bool "256KB"
+
+config IA64_LONG_FORMAT_VHPT_1MB
+	bool "1MB"
+
+config IA64_LONG_FORMAT_VHPT_4MB
+	bool "4MB"
+
+endchoice
+
 endmenu
 
 source "security/Kconfig"
arch/ia64/kernel/ivt.S +48 -80

287 lines not on this page · raw message

arch/ia64/kernel/patch.c +0 -2
diff -Nru a/arch/ia64/kernel/patch.c b/arch/ia64/kernel/patch.c
@@ -145,7 +145,6 @@
 	extern unsigned long fsyscall_table[NR_syscalls];
 	s32 *offp = (s32 *) start;
 	u64 ip;
-
 	while (offp < (s32 *) end) {
 		ip = (u64) ia64_imva((char *) offp + *offp);
 		ia64_patch_imm64(ip, (u64) fsyscall_table);
@@ -179,7 +178,6 @@
 {
 #	define START(name)	((unsigned long) __start_gate_##name##_patchlist)
 #	define END(name)	((unsigned long)__end_gate_##name##_patchlist)
-
 	patch_fsyscall_table(START(fsyscall), END(fsyscall));
 	patch_brl_fsys_bubble_down(START(brl_fsys_bubble_down), END(brl_fsys_bubble_down));
 	ia64_patch_vtop(START(vtop), END(vtop));
arch/ia64/kernel/traps.c +1 -1
diff -Nru a/arch/ia64/kernel/traps.c b/arch/ia64/kernel/traps.c
@@ -453,7 +453,7 @@
 		"Unknown fault 9", "Unknown fault 10", "Unknown fault 11", "Unknown fault 12",
 		"Unknown fault 13", "Unknown fault 14", "Unknown fault 15"
 	};
-
+	
 	if ((isr & IA64_ISR_NA) && ((isr & IA64_ISR_CODE_MASK) == IA64_ISR_CODE_LFETCH)) {
 		/*
 		 * This fault was due to lfetch.fault, set "ed" bit in the psr to cancel
arch/ia64/mm/fault.c +6 -3
diff -Nru a/arch/ia64/mm/fault.c b/arch/ia64/mm/fault.c
@@ -84,7 +84,7 @@
 	 */
 	if (in_atomic() || !mm)
 		goto no_context;
-
+		
 #ifdef CONFIG_VIRTUAL_MEM_MAP
 	/*
 	 * If fault is in region 5 and we are in the kernel, we may already
@@ -223,9 +223,12 @@
 	 * translation, which fixed the problem. So, we check to see if the translation is
 	 * valid, and return if it is.
 	 */
-	if (REGION_NUMBER(address) == 5 && mapped_kernel_page_is_present(address))
+	if (REGION_NUMBER(address) == 5 && mapped_kernel_page_is_present(address)){
+/*#ifdef CONFIG_IA64_LONG_FORMAT_VHPT
+	printk(KERN_INFO "`ia64_do_page_fault` Got Here `no_context: addr: 0x%x`\n", address);
+#endif*/
 		return;
-
+	}
 	/*
 	 * Oops. The kernel tried to access some bad page. We'll have to terminate things
 	 * with extreme prejudice.
arch/ia64/mm/init.c +44 -4

104 lines not on this page · raw message

arch/ia64/mm/tlb.c +47 -0

65 lines not on this page · raw message

include/asm-ia64/asmmacro.h +131 -0

142 lines not on this page · raw message

include/asm-ia64/kregs.h +1 -0
diff -Nru a/include/asm-ia64/kregs.h b/include/asm-ia64/kregs.h
@@ -30,6 +30,7 @@
 #define IA64_TR_PALCODE		1	/* itr1: maps PALcode as required by EFI */
 #define IA64_TR_PERCPU_DATA	1	/* dtr1: percpu data */
 #define IA64_TR_CURRENT_STACK	2	/* dtr2: maps kernel's memory- & register-stacks */
+#define IA64_TR_LONG_VHPT	3	/* dtr3: maps long format VHPT */
 
 /* Processor status register bits: */
 #define IA64_PSR_BE_BIT		1
include/asm-ia64/mmu_context.h +15 -1
diff -Nru a/include/asm-ia64/mmu_context.h b/include/asm-ia64/mmu_context.h
@@ -17,7 +17,12 @@
 
 #define IA64_REGION_ID_KERNEL	0 /* the kernel's region id (tlb.c depends on this being 0) */
 
-#define ia64_rid(ctx,addr)	(((ctx) << 3) | (addr >> 61))
+#ifdef CONFIG_IA64_LONG_FORMAT_VHPT
+#define redistribute_rid(rid)	(((rid) & ~0xffff) | (((rid) << 8) & 0xff00) | (((rid) >> 8) & 0xff))
+#else
+#define redistribute_rid(rid)	(rid)
+#endif
+#define ia64_rid(ctx,addr)	redistribute_rid(((ctx) << 3) | (addr >> 61))
 
 # ifndef __ASSEMBLY__
 
@@ -141,7 +146,12 @@
 	unsigned long rr0, rr1, rr2, rr3, rr4;
 
 	rid = context << 3;	/* make space for encoding the region number */
+#ifdef CONFIG_IA64_LONG_FORMAT_VHPT
+	rid = redistribute_rid(rid);
+	rid_incr = 1 << 16;
+#else
 	rid_incr = 1 << 8;
+#endif
 
 	/* encode the region id, preferred page size, and VHPT enable bit: */
 	rr0 = (rid << 8) | (PAGE_SHIFT << 2) | 1;
@@ -173,6 +183,10 @@
 		MMU_TRACE('a', smp_processor_id(), mm, context);
 		/* in the unlikely event of a TLB-flush by another thread, redo the load: */
 	} while (unlikely(context != mm->context));
+	
+#if defined(CONFIG_IA64_LONG_FORMAT_VHPT) && defined(CONFIG_SMP)
+	set_bit(smp_processor_id(), &mm->cpu_vm_mask);
+#endif
 }
 
 #define deactivate_mm(tsk,mm)					\
include/asm-ia64/page.h +3 -0
diff -Nru a/include/asm-ia64/page.h b/include/asm-ia64/page.h
@@ -151,6 +151,9 @@
 	return order;
 }
 
+/* Long format VHPT entry */
+typedef struct { unsigned long pte, itir, tag, ig; } long_pte_t;
+
 # endif /* __KERNEL__ */
 #endif /* !__ASSEMBLY__ */
 
include/asm-ia64/pgtable.h +16 -0
diff -Nru a/include/asm-ia64/pgtable.h b/include/asm-ia64/pgtable.h
@@ -493,6 +493,22 @@
 #define KERNEL_TR_PAGE_SHIFT	_PAGE_SIZE_64M
 #define KERNEL_TR_PAGE_SIZE	(1 << KERNEL_TR_PAGE_SHIFT)
 
+ /*
+ * Long format VHPT
+ */
+#if defined(CONFIG_IA64_LONG_FORMAT_VHPT_64KB)
+# define LONG_VHPT_BITS		_PAGE_SIZE_64K
+#elif defined(CONFIG_IA64_LONG_FORMAT_VHPT_256KB)
+# define LONG_VHPT_BITS		_PAGE_SIZE_256K
+#elif defined(CONFIG_IA64_LONG_FORMAT_VHPT_1MB)
+# define LONG_VHPT_BITS		_PAGE_SIZE_1M
+#elif defined(CONFIG_IA64_LONG_FORMAT_VHPT_4MB)
+# define LONG_VHPT_BITS		_PAGE_SIZE_4M
+#endif
+
+#define LONG_VHPT_SIZE		(1 << LONG_VHPT_BITS)
+#define LONG_VHPT_BASE		(0xc000000000000000 - LONG_VHPT_SIZE)
+
 /*
  * No page table caches to initialise
  */
include/asm-ia64/tlb.h +2 -0
diff -Nru a/include/asm-ia64/tlb.h b/include/asm-ia64/tlb.h
@@ -105,8 +105,10 @@
 		vma.vm_mm = tlb->mm;
 		/* flush the address range from the tlb: */
 		flush_tlb_range(&vma, start, end);
+#ifndef CONFIG_IA64_LONG_FORMAT_VHPT
 		/* now flush the virt. page-table area mapping the address range: */
 		flush_tlb_range(&vma, ia64_thash(start), ia64_thash(end));
+#endif
 	}
 
 	/* lastly, release the freed pages */
include/asm-ia64/tlbflush.h +19 -0
diff -Nru a/include/asm-ia64/tlbflush.h b/include/asm-ia64/tlbflush.h
@@ -19,6 +19,21 @@
  * can be very expensive, so try to avoid them whenever possible.
  */
 
+/* Flushing a translation from the long format VHPT */
+#ifdef CONFIG_IA64_LONG_FORMAT_VHPT
+# define INVALID_TAG (1UL << 63)
+
+static inline void
+flush_vhpt_page(unsigned long addr)
+{
+	long_pte_t *hpte;
+	hpte = (long_pte_t *)ia64_thash(addr);
+	hpte->tag = INVALID_TAG;
+}
+#else
+# define flush_vhpt_page(addr)	do { } while (0)
+#endif
+
 /*
  * Flush everything (kernel mapping may also have changed due to
  * vmalloc/vfree).
@@ -53,6 +68,7 @@
 		goto out;
 
 	mm->context = 0;
+	mm->cpu_vm_mask = 0;
 
 	if (atomic_read(&mm->mm_users) == 0)
 		goto out;		/* happens as a result of exit_mmap() */
@@ -78,7 +94,10 @@
 	flush_tlb_range(vma, (addr & PAGE_MASK), (addr & PAGE_MASK) + PAGE_SIZE);
 #else
 	if (vma->vm_mm == current->active_mm)
+	{
+		flush_vhpt_page(addr);
 		ia64_ptcl(addr, (PAGE_SHIFT << 2));
+	}
 	else
 		vma->vm_mm->context = 0;
 #endif
#1 lore raw
Quoted 4 lines · Darren Williams
arch/ia64/mm/init.c +4 -0
+#ifdef CONFIG_IA64_LONG_FORMAT_VHPT
+	int cpu = smp_processor_id();
+	int i;
+	

arch/ia64/mm/init.c line 353

Quoted 10 lines
arch/ia64/mm/init.c +10 -0
+	if (cpu = 0)
+	{
+		/* We allocate all VHPTs here as we may not have enough contiguous memory later */
+		/* Also, we cannot use __get_free_pages on the target processor (until after init_cpu) */
+		for (i = 0; i < NR_CPUS; i++){
+			vhpt_base[i] = (unsigned long)__alloc_bootmem(LONG_VHPT_SIZE, LONG_VHPT_SIZE, __pa(MAX_DMA_ADDRESS));
+			printk(KERN_INFO "CPU %d: Long format VHPT initialised at base address: 0x%lx\n", i, vhpt_base[i]);
+		}
+	}
+#endif

- Can we avoid allocating VHPTs for non-existent CPUs ?
- How about calling alloc_bootmem on cpu0 and __get_free_pages on secondary cpus ?
- Does the VHPT size have to be a config option ? Can we not size it based on the size of the physical memory and some user supplied multiplier ?

I noticed that when we hit ia64_mmu_init() on a secondary processor, cpu0 has already made successful calls to __get_free_pages(). But calls from the secondary cpus fail. Does anyone know why ?

-Arun

PS: patches for some of the above available, but they're based on Matt's earlier patch and need cleaning up.

#2 lore raw
In reply to Darren Williams

Hi Arun!

Quoted 14 lines · Arun Sharma
Quoted 13 lines · Darren Williams
arch/ia64/mm/init.c +7 -0
+#ifdef CONFIG_IA64_LONG_FORMAT_VHPT
+	int cpu = smp_processor_id();
+	int i;
+	
+	if (cpu = 0)
+	{
+		/* We allocate all VHPTs here as we may not have enough 

contiguous memory later */

(quoted diff) +1 -0
+		/* Also, we cannot use __get_free_pages on the target 

processor (until after init_cpu) */

(quoted diff) +2 -0
+		for (i = 0; i < NR_CPUS; i++){
+			vhpt_base[i] = (unsigned 

long)__alloc_bootmem(LONG_VHPT_SIZE, LONG_VHPT_SIZE,

Quoted 8 lines

arch/ia64/mm/init.c line 353

Quoted 6 lines

__pa(MAX_DMA_ADDRESS));

(quoted diff) +1 -0
+			printk(KERN_INFO "CPU %d: Long format VHPT 

initialised at base address: 0x%lx\n", i, vhpt_base[i]);

arch/ia64/mm/init.c +3 -0
+		}
+	}
+#endif

- Can we avoid allocating VHPTs for non-existent CPUs ?

I should of but this as a bug, that I am working on, currently I set the actual cpu No. when configuring the kernel this avoids the default of 2-63 CPUs.

Quoted 2 lines

- How about calling alloc_bootmem on cpu0 and __get_free_pages on secondary cpus ?

Will look into this.

Quoted 3 lines

- Does the VHPT size have to be a config option ? Can we not size it based on the size of the physical memory and some user supplied multiplier ?

Not sure?

Quoted 6 lines

I noticed that when we hit ia64_mmu_init() on a secondary processor, cpu0 has already made successful calls to __get_free_pages(). But calls from the secondary cpus fail. Does anyone know why ?

-Arun

Quoted 7 lines

PS: patches for some of the above available, but they're based on Matt's earlier patch and need cleaning up.

-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--------------------------------------------------
Darren Williams <[email protected]>
Gelato@UNSW <www.gelato.unsw.edu.au>
--------------------------------------------------

#3 lore raw
In reply to Darren Williams
Quoted 8 lines · Arun Sharma

- Can we avoid allocating VHPTs for non-existent CPUs ?
- How about calling alloc_bootmem on cpu0 and __get_free_pages on secondary cpus ?

I noticed that when we hit ia64_mmu_init() on a secondary processor, cpu0 has already made successful calls to __get_free_pages(). But calls from the secondary cpus fail. Does anyone know why ?

I did some hacking over the weekend. VHPT allocation can be done from boot processor in function do_boot_cpu() before waking up secondary CPUs. Allocation can be easily made numa friendly by calling alloc_pages_node (didn't bother to do that for now).

- Ken

arch/ia64/kernel/smpboot.c +16 -0
diff -Nurp 2.6.0-test5.v/arch/ia64/kernel/smpboot.c 2.6.0-test5.ken/arch/ia64/kernel/smpboot.c
@@ -358,11 +358,16 @@ fork_by_hand (void)
 	return copy_process(CLONE_VM|CLONE_IDLETASK, 0, 0, 0, NULL, NULL);
 }
 
+#ifdef CONFIG_IA64_LONG_FORMAT_VHPT
+extern unsigned long vhpt_base[NR_CPUS];
+#endif
+
 static int __init
 do_boot_cpu (int sapicid, int cpu)
 {
 	struct task_struct *idle;
 	int timeout;
+	struct page * vhpt_page;
 
 	/*
 	 * We can't use kernel_thread since we must avoid to reschedule the child.
@@ -382,6 +387,16 @@ do_boot_cpu (int sapicid, int cpu)
 
 	task_for_booting_cpu = idle;
 
+#ifdef CONFIG_IA64_LONG_FORMAT_VHPT
+	vhpt_page = alloc_pages(__GFP_HIGHMEM, LONG_VHPT_BITS - PAGE_SHIFT);
+	if (vhpt_page)
+		vhpt_base[cpu] = (unsigned long) page_address(vhpt_page);
+	else
+		return -EINVAL;
+	printk(KERN_INFO "CPU %d: Long format VHPT initialised at base address:"
+			 " 0x%lx\n", cpu, vhpt_base[cpu]);
+#endif
+
 	Dprintk("Sending wakeup vector %lu to AP 0x%x/0x%x.\n", ap_wakeup_vector, cpu, sapicid);
 
 	platform_send_ipi(cpu, ap_wakeup_vector, IA64_IPI_DM_INT, 0);
@@ -404,6 +419,7 @@ do_boot_cpu (int sapicid, int cpu)
 		printk(KERN_ERR "Processor 0x%x/0x%x is stuck.\n", cpu, sapicid);
 		ia64_cpu_to_sapicid[cpu] = -1;
 		cpu_clear(cpu, cpu_online_map);  /* was set in smp_callin() */
+		__free_pages(vhpt_page, LONG_VHPT_BITS - PAGE_SHIFT);
 		return -EINVAL;
 	}
 	return 0;
arch/ia64/mm/init.c +2 -5
diff -Nurp 2.6.0-test5.v/arch/ia64/mm/init.c 2.6.0-test5.ken/arch/ia64/mm/init.c
@@ -344,11 +344,8 @@ ia64_mmu_init (void *my_cpu_data)
 	if (cpu == 0)
 	{
 		/* We allocate all VHPTs here as we may not have enough contiguous memory later */
-		/* Also, we cannot use __get_free_pages on the target processor (until after init_cpu) */
-		for (i = 0; i < NR_CPUS; i++){
-			vhpt_base[i] = (unsigned long)__alloc_bootmem(LONG_VHPT_SIZE, LONG_VHPT_SIZE, __pa(MAX_DMA_ADDRESS));
-			printk(KERN_INFO "CPU %d: Long format VHPT initialised at base address: 0x%lx\n", i, vhpt_base[i]);
-		}
+		vhpt_base[0] = (unsigned long)__alloc_bootmem(LONG_VHPT_SIZE, LONG_VHPT_SIZE, __pa(MAX_DMA_ADDRESS));
+		printk(KERN_INFO "CPU 0: Long format VHPT initialised at base address: 0x%lx\n", vhpt_base[0]);
 	}
 #endif
 
#4 lore raw
In reply to Darren Williams
Quoted 4 lines · Kenneth W

I did some hacking over the weekend. VHPT allocation can be done from boot processor in function do_boot_cpu() before waking up secondary CPUs. Allocation can be easily made numa friendly by calling alloc_pages_node (didn't bother to do that for now).

I was thinking of the hot-plug CPU case, where the allocation has to be done on the newly added CPU anyway.

-Arun

#5 lore raw
In reply to Darren Williams

Hi Arun

These are useful suggestions and we will need a solution that covers all possible scenarios for the page table walker, I have just begun looking at the Ia64 code and so my responses may be slow until I get my head around it. One question I do have is that if we are initialising vhpt in do_boot_cpu() are we only talking about SMP machines here, what about UNI processor machines?

Darren

Quoted 10 lines · Arun Sharma
Quoted 4 lines · Kenneth W

I did some hacking over the weekend. VHPT allocation can be done from boot processor in function do_boot_cpu() before waking up secondary CPUs. Allocation can be easily made numa friendly by calling alloc_pages_node (didn't bother to do that for now).

I was thinking of the hot-plug CPU case, where the allocation has to be done on the newly added CPU anyway.

-Arun

--------------------------------------------------
Darren Williams <[email protected]>
Gelato@UNSW <www.gelato.unsw.edu.au>
--------------------------------------------------

#6 lore raw
In reply to Darren Williams
Quoted 8 lines · Darren Williams

Hi Arun

These are useful suggestions and we will need a solution that covers all possible scenarios for the page table walker, I have just begun looking at the Ia64 code and so my responses may be slow until I get my head around it. One question I do have is that if we are initialising vhpt in do_boot_cpu() are we only talking about SMP machines here, what about UNI processor machines?

Yes, do_boot_cpu() doesn't work on UP. I'm still looking into allocating the VHPT on secondary CPUs rather than the boot CPU.

-Arun

#7 lore raw
In reply to Darren Williams
Quoted 1 line · Arun Sharma

Yes, do_boot_cpu() doesn't work on UP. I'm still looking into allocating the VHPT on secondary CPUs rather than the boot CPU.

Ken swung by to say that his patch handles the UP case just fine in ia64_mmu_init(). Here's a consolidated patch:

- Doesn't allocate VHPT for non existent CPUs (Ken)
- VHPT size dynamically computed. More intelligent algorithm TBD.
- NUMA aware (but not tested)
- Hopefully hot-plug friendly.

I've tested it on UP and SMP.

-Arun

include/asm-ia64/asmmacro.h +131 -0
Index: linux-2.6/include/asm-ia64/asmmacro.h
===================================================================
@@ -4,6 +4,11 @@
 /*
  * Copyright (C) 2000-2001, 2003 Hewlett-Packard Co
  *	David Mosberger-Tang <[email protected]>
+ *
+ * 25/09/03 Darren Williams <[email protected]> move Long format VHPT MACROS 
+ * 	    from ivt.S to here. I though that this would keep the placement of 
+ *	    code and MACROS correct.
+ *
  */
 
 #include <linux/config.h>
@@ -84,4 +89,130 @@
 # define MCKINLEY_E9_WORKAROUND
 #endif
 
+
+/*
+ * Find a page table entry with the long format VHPT walker
+ */
+#define temp r24
+#define rgn r25
+#define mbz r26
+/* va=r16 ppte=r19 fail=p6 ok=p7 */
+#define FIND_PTE(va, ppte, fail, ok)								\
+	rsm psr.dt;				/* switch to using physical data addressing */	\
+	mov ppte=IA64_KR(PT_BASE);		/* get the page table base address */		\
+	shl mbz=va,3;				/* shift bit 60 into sign bit	*/		\
+	shr.u rgn=va,61;			/* get the region number into 'rgn' */		\
+	;;											\
+	cmp.eq p6,p7=5,rgn;			/* is faulting address in region 5? */		\
+	shr.u temp=va,PGDIR_SHIFT;		/* get bits 33-63 of faulting address */	\
+	;;											\
+(p7)	dep ppte=rgn,ppte,(PAGE_SHIFT-3),3;	/* put region number bits in place */		\
+	srlz.d;											\
+	LOAD_PHYSICAL(p6, ppte, swapper_pg_dir); /* region 5 is rooted at swapper_pg_dir */	\
+												\
+	.pred.rel "mutex", p6, p7;								\
+(p6)	shr.u mbz=mbz,PGDIR_SHIFT+PAGE_SHIFT;							\
+(p7)	shr.u mbz=mbz,PGDIR_SHIFT+PAGE_SHIFT-3;							\
+	;;											\
+(p6)	dep ppte=temp,ppte,3,(PAGE_SHIFT-3);	/* ppte=PTA + IFA(33,42)*8 */			\
+(p7)	dep ppte=temp,ppte,3,(PAGE_SHIFT-6);	/* ppte=PTA + (((IFA(61,63) << 7) | IFA(33,39))*8) */ \
+	cmp.eq ok,fail=0,mbz;			/* unused address bits all zeroes? */		\
+	shr.u temp=va,PMD_SHIFT;			/* shift L2 index into position	*/		\
+	;;											\
+	ld8 ppte=[ppte];			/* fetch the L1 entry (may be 0)*/		\
+	;;											\
+(ok)	cmp.eq fail,ok=ppte,r0;			/* was L1 entry NULL?*/				\
+	dep ppte=temp,ppte,3,(PAGE_SHIFT-3);	/* compute address of L2 page table entry*/	\
+	;;											\
+(ok)	ld8 ppte=[ppte];			/* fetch the L2 entry (may be 0)*/		\
+	shr.u temp=va,PAGE_SHIFT;		/* shift L3 index into position*/		\
+	;;											\
+(ok)	cmp.eq.or.andcm fail,ok=ppte,r0;		/* was L2 entry NULL?*/				\
+	dep ppte=temp,ppte,3,(PAGE_SHIFT-3);	/* compute address of L3 page table entry*/
+
+#ifdef CONFIG_IA64_LONG_FORMAT_VHPT
+
+#define LOAD_PTE_MISS(va, ppte, pte, hpte, failfn)						\
+	;;											\
+	FIND_PTE(va, ppte, p6, p7)								\
+	;;											\
+(p7)	ld8 pte=[ppte];										\
+	;;											\
+(p7)	tbit.z p6,p0=pte,_PAGE_P_BIT;		/* page present bit cleared? */			\
+(p6)	br.cond.spnt failfn; 
+
+/* Since we access the page table physically, we access the long VHPT physically as well
+ * to avoid switching back and forth */
+
+
+#define tag r25
+#define htag r26
+#define LOAD_PTE_FAULT(va, ppte, pte, hpte, failfn)						\
+	thash hpte=va;										\
+	rsm psr.dt;										\
+	;;											\
+	tpa hpte=hpte;				/* make hash address physical */		\
+	ttag tag=va;										\
+	;;											\
+	srlz.d;											\
+	add temp=16,hpte;									\
+	add ppte=24,hpte;									\
+	;;											\
+	ld8 htag=[temp];									\
+	ld8 ppte=[ppte];									\
+	;;											\
+	cmp.ne p6,p7=htag, tag;			/* verify tag */				\
+	;;											\
+(p7)	ld8 pte=[ppte];										\
+(p6)	br.cond.spnt failfn;
+
+#define tir r26
+#define VHPT_INSERT(va, ppte, pte, hpte)							\
+	mov hpte=cr.iha;									\
+	mov tir=cr.itir;									\
+	;;											\
+	tpa hpte=hpte;				/* make hash address physical */		\
+	ttag tag=va;										\
+	;;											\
+	add temp=16,hpte;									\
+	;;											\
+	st8 [hpte]=pte,8;			/* fill out VHPT entry */			\
+	st8 [temp]=tag,8;									\
+	;;											\
+	st8 [hpte]=tir,8;									\
+	st8 [temp]=ppte;
+	
+#define VHPT_UPDATE(cond, pte, hpte)								\
+(cond)	st8 [hpte]=pte,16;
+
+#define VHPT_PURGE(cond, hpte)									\
+(cond)	dep tag=-1,r0,63,1;			/* set tag-invalid bit */			\
+	;;											\
+(cond)	st8 [hpte]=tag;				/* hpte already points to tag (see above) */
+
+#else /* !CONFIG_IA64_LONG_FORMAT_VHPT */
+
+#define LOAD_PTE_MISS(va, ppte, pte, hpte, failfn)						\
+	mov ppte=cr.iha;			/* get virtual address of L3 PTE */		\
+	movl r30=1f;				/* load continuation point */			\
+	;;											\
+1:	ld8 pte=[ppte]; \
+	mov b0=r29;				/* restore possibly destroyed b0 */		\
+	;;											\
+	tbit.z p6,p0=pte,_PAGE_P_BIT;		/* page present bit cleared? */			\
+(p6)	br.cond.spnt failfn;
+
+#define LOAD_PTE_FAULT(va, ppte, pte, hpte, failfn)						\
+	thash ppte=va;				/* get virtual address of L3 PTE */		\
+	movl r30=1f;				/* load continuation point */			\
+	;;											\
+1:	ld8 pte=[ppte];										\
+	mov b0=r29;				/* restore possibly destroyed b0 */
+
+#define VHPT_INSERT(va, ppte, pte, hpte)		/* nothing */
+#define VHPT_UPDATE(cond, pte, hpte)			/* nothing */
+#define VHPT_PURGE(cond, hpte)				/* nothing */
+#endif
+
+
 #endif /* _ASM_IA64_ASMMACRO_H */
include/asm-ia64/tlb.h +2 -0
Index: linux-2.6/include/asm-ia64/tlb.h
===================================================================
@@ -105,8 +105,10 @@
 		vma.vm_mm = tlb->mm;
 		/* flush the address range from the tlb: */
 		flush_tlb_range(&vma, start, end);
+#ifndef CONFIG_IA64_LONG_FORMAT_VHPT
 		/* now flush the virt. page-table area mapping the address range: */
 		flush_tlb_range(&vma, ia64_thash(start), ia64_thash(end));
+#endif
 	}
 
 	/* lastly, release the freed pages */
include/asm-ia64/tlbflush.h +19 -0
Index: linux-2.6/include/asm-ia64/tlbflush.h
===================================================================
@@ -19,6 +19,21 @@
  * can be very expensive, so try to avoid them whenever possible.
  */
 
+/* Flushing a translation from the long format VHPT */
+#ifdef CONFIG_IA64_LONG_FORMAT_VHPT
+# define INVALID_TAG (1UL << 63)
+
+static inline void
+flush_vhpt_page(unsigned long addr)
+{
+	long_pte_t *hpte;
+	hpte = (long_pte_t *)ia64_thash(addr);
+	hpte->tag = INVALID_TAG;
+}
+#else
+# define flush_vhpt_page(addr)	do { } while (0)
+#endif
+
 /*
  * Flush everything (kernel mapping may also have changed due to
  * vmalloc/vfree).
@@ -53,6 +68,7 @@
 		goto out;
 
 	mm->context = 0;
+	mm->cpu_vm_mask = 0;
 
 	if (atomic_read(&mm->mm_users) == 0)
 		goto out;		/* happens as a result of exit_mmap() */
@@ -78,7 +94,10 @@
 	flush_tlb_range(vma, (addr & PAGE_MASK), (addr & PAGE_MASK) + PAGE_SIZE);
 #else
 	if (vma->vm_mm == current->active_mm)
+	{
+		flush_vhpt_page(addr);
 		ia64_ptcl(addr, (PAGE_SHIFT << 2));
+	}
 	else
 		vma->vm_mm->context = 0;
 #endif
include/asm-ia64/mmu_context.h +15 -1
Index: linux-2.6/include/asm-ia64/mmu_context.h
===================================================================
@@ -17,7 +17,12 @@
 
 #define IA64_REGION_ID_KERNEL	0 /* the kernel's region id (tlb.c depends on this being 0) */
 
-#define ia64_rid(ctx,addr)	(((ctx) << 3) | (addr >> 61))
+#ifdef CONFIG_IA64_LONG_FORMAT_VHPT
+#define redistribute_rid(rid)	(((rid) & ~0xffff) | (((rid) << 8) & 0xff00) | (((rid) >> 8) & 0xff))
+#else
+#define redistribute_rid(rid)	(rid)
+#endif
+#define ia64_rid(ctx,addr)	redistribute_rid(((ctx) << 3) | (addr >> 61))
 
 # ifndef __ASSEMBLY__
 
@@ -141,7 +146,12 @@
 	unsigned long rr0, rr1, rr2, rr3, rr4;
 
 	rid = context << 3;	/* make space for encoding the region number */
+#ifdef CONFIG_IA64_LONG_FORMAT_VHPT
+	rid = redistribute_rid(rid);
+	rid_incr = 1 << 16;
+#else
 	rid_incr = 1 << 8;
+#endif
 
 	/* encode the region id, preferred page size, and VHPT enable bit: */
 	rr0 = (rid << 8) | (PAGE_SHIFT << 2) | 1;
@@ -173,6 +183,10 @@
 		MMU_TRACE('a', smp_processor_id(), mm, context);
 		/* in the unlikely event of a TLB-flush by another thread, redo the load: */
 	} while (unlikely(context != mm->context));
+	
+#if defined(CONFIG_IA64_LONG_FORMAT_VHPT) && defined(CONFIG_SMP)
+	set_bit(smp_processor_id(), &mm->cpu_vm_mask);
+#endif
 }
 
 #define deactivate_mm(tsk,mm)					\
include/asm-ia64/pgtable.h +11 -0
Index: linux-2.6/include/asm-ia64/pgtable.h
===================================================================
@@ -474,6 +474,12 @@
     extern void memmap_init (struct page *start, unsigned long size, int nid, unsigned long zone,
 			     unsigned long start_pfn);
 #  endif /* CONFIG_VIRTUAL_MEM_MAP */
+
+#ifdef CONFIG_IA64_LONG_FORMAT_VHPT
+extern unsigned long vhpt_base[NR_CPUS];
+extern unsigned long long_vhpt_bits, long_vhpt_size;
+#endif
+
 # endif /* !__ASSEMBLY__ */
 
 /*
@@ -493,6 +499,11 @@
 #define KERNEL_TR_PAGE_SHIFT	_PAGE_SIZE_64M
 #define KERNEL_TR_PAGE_SIZE	(1 << KERNEL_TR_PAGE_SHIFT)
 
+ /*
+ * Long format VHPT
+ */
+#define LONG_VHPT_BASE		(0xc000000000000000 - long_vhpt_size)
+
 /*
  * No page table caches to initialise
  */
include/asm-ia64/kregs.h +1 -0
Index: linux-2.6/include/asm-ia64/kregs.h
===================================================================
@@ -30,6 +30,7 @@
 #define IA64_TR_PALCODE		1	/* itr1: maps PALcode as required by EFI */
 #define IA64_TR_PERCPU_DATA	1	/* dtr1: percpu data */
 #define IA64_TR_CURRENT_STACK	2	/* dtr2: maps kernel's memory- & register-stacks */
+#define IA64_TR_LONG_VHPT	3	/* dtr3: maps long format VHPT */
 
 /* Processor status register bits: */
 #define IA64_PSR_BE_BIT		1
include/asm-ia64/page.h +3 -0
Index: linux-2.6/include/asm-ia64/page.h
===================================================================
@@ -151,6 +151,9 @@
 	return order;
 }
 
+/* Long format VHPT entry */
+typedef struct { unsigned long pte, itir, tag, ig; } long_pte_t;
+
 # endif /* __KERNEL__ */
 #endif /* !__ASSEMBLY__ */
 
Makefile +1 -1
Index: linux-2.6/Makefile
===================================================================
@@ -1,7 +1,7 @@
 VERSION = 2
 PATCHLEVEL = 6
 SUBLEVEL = 0
-EXTRAVERSION = -test5
+EXTRAVERSION = -test5-vhpt
 
 # *DOCUMENTATION*
 # To see a list of typical targets execute "make help"
arch/ia64/Kconfig +4 -1
Index: linux-2.6/arch/ia64/Kconfig
===================================================================
@@ -677,7 +677,10 @@
 	  debugging info resulting in a larger kernel image.
 	  Say Y here only if you plan to use gdb to debug the kernel.
 	  If you don't debug the kernel, you can say N.
-	  
+
+config IA64_LONG_FORMAT_VHPT
+	bool "Long format VHPT"
+
 endmenu
 
 source "security/Kconfig"
arch/ia64/kernel/setup.c +11 -0
Index: linux-2.6/arch/ia64/kernel/setup.c
===================================================================
@@ -359,6 +359,13 @@
 }
 
 void __init
+compute_vhpt_size()
+{
+	long_vhpt_bits = 22;
+	long_vhpt_size = 1 << long_vhpt_bits;
+}
+
+void __init
 setup_arch (char **cmdline_p)
 {
 	extern unsigned long ia64_iobase;
@@ -374,6 +381,10 @@
 	efi_init();
 	find_memory();
 
+#ifdef CONFIG_IA64_LONG_FORMAT_VHPT
+	compute_vhpt_size();
+#endif
+
 #ifdef CONFIG_ACPI_BOOT
 	/* Initialize the ACPI boot-time table parser */
 	acpi_table_init();
arch/ia64/kernel/ivt.S +48 -80

287 lines not on this page · raw message

arch/ia64/kernel/smpboot.c +7 -1
Index: linux-2.6/arch/ia64/kernel/smpboot.c
===================================================================
@@ -381,7 +381,13 @@
 	unhash_process(idle);
 
 	task_for_booting_cpu = idle;
-
+#ifdef CONFIG_IA64_LONG_FORMAT_VHPT
+	if (alloc_vhpt(cpu)) {
+		panic("Couldn't allocate VHPT on CPU%d, size: %lx\n", cpu, long_vhpt_size);
+	}
+	printk(KERN_INFO "Allocated long format VHPT for CPU%d at: %lx, size: 0x%lx\n", cpu, vhpt_base[cpu], long_vhpt_size);
+#endif /* CONFIG_IA64_LONG_FORMAT_VHPT */
+	
 	Dprintk("Sending wakeup vector %lu to AP 0x%x/0x%x.\n", ap_wakeup_vector, cpu, sapicid);
 
 	platform_send_ipi(cpu, ap_wakeup_vector, IA64_IPI_DM_INT, 0);
arch/ia64/kernel/patch.c +0 -2
Index: linux-2.6/arch/ia64/kernel/patch.c
===================================================================
@@ -145,7 +145,6 @@
 	extern unsigned long fsyscall_table[NR_syscalls];
 	s32 *offp = (s32 *) start;
 	u64 ip;
-
 	while (offp < (s32 *) end) {
 		ip = (u64) ia64_imva((char *) offp + *offp);
 		ia64_patch_imm64(ip, (u64) fsyscall_table);
@@ -179,7 +178,6 @@
 {
 #	define START(name)	((unsigned long) __start_gate_##name##_patchlist)
 #	define END(name)	((unsigned long)__end_gate_##name##_patchlist)
-
 	patch_fsyscall_table(START(fsyscall), END(fsyscall));
 	patch_brl_fsys_bubble_down(START(brl_fsys_bubble_down), END(brl_fsys_bubble_down));
 	ia64_patch_vtop(START(vtop), END(vtop));
arch/ia64/kernel/traps.c +1 -1
Index: linux-2.6/arch/ia64/kernel/traps.c
===================================================================
@@ -453,7 +453,7 @@
 		"Unknown fault 9", "Unknown fault 10", "Unknown fault 11", "Unknown fault 12",
 		"Unknown fault 13", "Unknown fault 14", "Unknown fault 15"
 	};
-
+	
 	if ((isr & IA64_ISR_NA) && ((isr & IA64_ISR_CODE_MASK) == IA64_ISR_CODE_LFETCH)) {
 		/*
 		 * This fault was due to lfetch.fault, set "ed" bit in the psr to cancel
arch/ia64/mm/tlb.c +47 -0

65 lines not on this page · raw message

arch/ia64/mm/init.c +65 -4

131 lines not on this page · raw message

#8 lore raw
In reply to Darren Williams

include/asm-ia64/mmu_context.h line 25

Quoted 9 lines · Darren Williams
include/asm-ia64/mmu_context.h +6 -1
-#define ia64_rid(ctx,addr)	(((ctx) << 3) | (addr >> 61))
+#ifdef CONFIG_IA64_LONG_FORMAT_VHPT
+#define redistribute_rid(rid)	(((rid) & ~0xffff) | (((rid) << 8) & 0xff00) | (((rid) >> 8) & 0xff))
+#else
+#define redistribute_rid(rid)	(rid)
+#endif
+#define ia64_rid(ctx,addr)	redistribute_rid(((ctx) << 3) | (addr >> 61))

Hi Darren, Can you explain what this code is trying to do ?

-Arun

#9 lore raw
In reply to Darren Williams

include/asm-ia64/mmu_context.h line 25

Quoted 9 lines · Aurn> Darren Williams wrote:
include/asm-ia64/mmu_context.h +6 -1
-#define ia64_rid(ctx,addr)	(((ctx) << 3) | (addr >> 61))
+#ifdef CONFIG_IA64_LONG_FORMAT_VHPT
+#define redistribute_rid(rid)	(((rid) & ~0xffff) | (((rid) << 8) & 0xff00) | (((rid) >> 8) & 0xff))
+#else
+#define redistribute_rid(rid)	(rid)
+#endif
+#define ia64_rid(ctx,addr)	redistribute_rid(((ctx) << 3) | (addr >> 61))

Arun> Hi Darren, Can you explain what this code is trying to do ?

If I may comment....

The hash function that the VHPT uses works poorly for consecutive RIDs. The redistribute_rid() macro flips around the low order bits to get something that hashes more evenly -- otherwise in the experiments we did, we saw major hash collision problems.

I've cc'd Matthew, the original author of the code, to clarify if necessary.

--
Dr Peter Chubb http://www.gelato.unsw.edu.au peterc AT gelato.unsw.edu.au
You are lost in a maze of BitKeeper repositories, all slightly different.

#10 lore raw
In reply to Darren Williams
Quoted 14 lines · Peter Chubb

include/asm-ia64/mmu_context.h line 25

Quoted 9 lines · Aurn> Darren Williams wrote:
include/asm-ia64/mmu_context.h +6 -1
-#define ia64_rid(ctx,addr)	(((ctx) << 3) | (addr >> 61))
+#ifdef CONFIG_IA64_LONG_FORMAT_VHPT
+#define redistribute_rid(rid)	(((rid) & ~0xffff) | (((rid) << 8) & 0xff00) | (((rid) >> 8) & 0xff))
+#else
+#define redistribute_rid(rid)	(rid)
+#endif
+#define ia64_rid(ctx,addr)	redistribute_rid(((ctx) << 3) | (addr >> 61))

Arun> Hi Darren, Can you explain what this code is trying to do ?

Quoted 6 lines

If I may comment....

The hash function that the VHPT uses works poorly for consecutive RIDs. The redistribute_rid() macro flips around the low order bits to get something that hashes more evenly -- otherwise in the experiments we did, we saw major hash collision problems.

Yep. The hash function on current processors is basically RID ^ VPN so if you have close together RIDs and similar address space layout, which is common (e.g. fork()) you get significant collision problems. This was a hack to space out the RIDs we present to the hardware without messing with Linux's sequential allocation scheme. Perhaps there is a better place to do this.

Matt