Skip to contentSkip to conversation

fix iounmap and a pageattr memleak (x86 and x86-64)

PATCH Andrea Arcangeli 4 participants 10 messages Oct 28, 2004 – Feb 15, 2005

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

This first patch fixes silent memleak in the pageattr code that I found while searching for the bug Andi fixed in the second patch below (basically reference counting in split page was done on the pmd instead of the pte).

Index: linux-2.5/arch/i386/mm/pageattr.c
===================================================================
RCS file: /home/andrea/crypto/cvs/linux-2.5/arch/i386/mm/pageattr.c,v retrieving revision 1.13

arch/i386/mm/pageattr.c +7 -6
diff -u -p -r1.13 pageattr.c
@@ -117,22 +117,23 @@ __change_page_attr(struct page *page, pg
 	kpte_page = virt_to_page(kpte);
 	if (pgprot_val(prot) != pgprot_val(PAGE_KERNEL)) { 
 		if ((pte_val(*kpte) & _PAGE_PSE) == 0) { 
-			pte_t old = *kpte;
-			pte_t standard = mk_pte(page, PAGE_KERNEL); 
 			set_pte_atomic(kpte, mk_pte(page, prot)); 
-			if (pte_same(old,standard))
-				get_page(kpte_page);
 		} else {
 			struct page *split = split_large_page(address, prot); 
 			if (!split)
 				return -ENOMEM;
-			get_page(kpte_page);
 			set_pmd_pte(kpte,address,mk_pte(split, PAGE_KERNEL));
+			kpte_page = split;
 		}	
+		get_page(kpte_page);
 	} else if ((pte_val(*kpte) & _PAGE_PSE) == 0) { 
 		set_pte_atomic(kpte, mk_pte(page, PAGE_KERNEL));
 		__put_page(kpte_page);
-	}
+	} else
+		BUG();
+
+	/* memleak and potential failed 2M page regeneration */
+	BUG_ON(!page_count(kpte_page));
 
 	if (cpu_has_pse && (page_count(kpte_page) == 1)) {
 		list_add(&kpte_page->lru, &df_list);

Index: linux-2.5/arch/x86_64/mm/pageattr.c
===================================================================
RCS file: /home/andrea/crypto/cvs/linux-2.5/arch/x86_64/mm/pageattr.c,v retrieving revision 1.12

arch/x86_64/mm/pageattr.c +14 -9
diff -u -p -r1.12 pageattr.c
@@ -124,28 +124,33 @@ __change_page_attr(unsigned long address
 	kpte_flags = pte_val(*kpte); 
 	if (pgprot_val(prot) != pgprot_val(ref_prot)) { 
 		if ((kpte_flags & _PAGE_PSE) == 0) { 
-			pte_t old = *kpte;
-			pte_t standard = mk_pte(page, ref_prot); 
-
 			set_pte(kpte, mk_pte(page, prot)); 
-			if (pte_same(old,standard))
-				get_page(kpte_page);
 		} else {
+			/*
+			 * split_large_page will take the reference for this change_page_attr
+			 * on the split page.
+			 */
 			struct page *split = split_large_page(address, prot, ref_prot); 
 			if (!split)
 				return -ENOMEM;
-			get_page(kpte_page);
 			set_pte(kpte,mk_pte(split, ref_prot));
+			kpte_page = split;
 		}	
+		get_page(kpte_page);
 	} else if ((kpte_flags & _PAGE_PSE) == 0) { 
 		set_pte(kpte, mk_pte(page, ref_prot));
 		__put_page(kpte_page);
-	}
+	} else
+		BUG();
 
-	if (page_count(kpte_page) == 1) {
+	switch (page_count(kpte_page)) {
+	case 1:
 		save_page(address, kpte_page); 		     
 		revert_page(address, ref_prot);
-	} 
+		break;
+	case 0:
+		BUG(); /* memleak and failed 2M page regeneration */
+	}
 	return 0;
 } 
 

This below patch from Andi is also needed to make the above work on x86 (otherwise one of my new above BUGS() will trigger signalling the fact a bug was there). The below patch creates a subtle dependency that (_PAGE_PCD << 24) must not be zero. It's not the cleanest thing ever, but since it's an hardware bitflag I doubt it's going to break.

I'm not sure if I'm allowed to add the signedoff for Andi but I think I should since he wrote the x86-64 version, if not please let me know (I only backported it to x86 to test my above changes that otherwise would trigger one of my added BUGs, and I did the other worthless cosmetical fixes).

Index: linux-2.5/arch/i386/mm/ioremap.c
===================================================================
RCS file: /home/andrea/crypto/cvs/linux-2.5/arch/i386/mm/ioremap.c,v retrieving revision 1.23

arch/i386/mm/ioremap.c +2 -2
diff -u -p -r1.23 ioremap.c
@@ -152,7 +152,7 @@ void __iomem * __ioremap(unsigned long p
 	/*
 	 * Ok, go for it..
 	 */
-	area = get_vm_area(size, VM_IOREMAP);
+	area = get_vm_area(size, VM_IOREMAP | (flags << 24));
 	if (!area)
 		return NULL;
 	area->phys_addr = phys_addr;
@@ -232,7 +232,7 @@ void iounmap(volatile void __iomem *addr
 		return;
 	} 
 
-	if (p->flags && p->phys_addr < virt_to_phys(high_memory)) { 
+	if ((p->flags >> 24) && p->phys_addr < virt_to_phys(high_memory)) { 
 		change_page_attr(virt_to_page(__va(p->phys_addr)),
 				 p->size >> PAGE_SHIFT,
 				 PAGE_KERNEL); 				 

Index: linux-2.5/arch/x86_64/mm/ioremap.c
===================================================================
RCS file: /home/andrea/crypto/cvs/linux-2.5/arch/x86_64/mm/ioremap.c,v retrieving revision 1.16

arch/x86_64/mm/ioremap.c +7 -7
diff -u -p -r1.16 ioremap.c
@@ -128,11 +128,11 @@ void __iomem * __ioremap(unsigned long p
 	if (phys_addr >= 0xA0000 && last_addr < 0x100000)
 		return (__force void __iomem *)phys_to_virt(phys_addr);
 
+#ifndef CONFIG_DISCONTIGMEM
 	/*
 	 * Don't allow anybody to remap normal RAM that we're using..
 	 */
 	if (phys_addr < virt_to_phys(high_memory)) {
-#ifndef CONFIG_DISCONTIGMEM
 		char *t_addr, *t_end;
  		struct page *page;
 
@@ -142,8 +142,8 @@ void __iomem * __ioremap(unsigned long p
 		for(page = virt_to_page(t_addr); page <= virt_to_page(t_end); page++)
 			if(!PageReserved(page))
 				return NULL;
-#endif
 	}
+#endif
 
 	/*
 	 * Mappings have to be page-aligned
@@ -155,7 +155,7 @@ void __iomem * __ioremap(unsigned long p
 	/*
 	 * Ok, go for it..
 	 */
-	area = get_vm_area(size, VM_IOREMAP);
+	area = get_vm_area(size, VM_IOREMAP | (flags << 24));
 	if (!area)
 		return NULL;
 	area->phys_addr = phys_addr;
@@ -195,12 +195,12 @@ void __iomem *ioremap_nocache (unsigned 
 	if (!p) 
 		return p; 
 
-	if (phys_addr + size < virt_to_phys(high_memory)) { 
+	if (phys_addr + size - 1 < virt_to_phys(high_memory)) { 
 		struct page *ppage = virt_to_page(__va(phys_addr));		
 		unsigned long npages = (size + PAGE_SIZE - 1) >> PAGE_SHIFT;
 
-		BUG_ON(phys_addr+size > (unsigned long)high_memory);
-		BUG_ON(phys_addr + size < phys_addr);
+		BUG_ON(phys_addr+size >= (unsigned long)high_memory);
+		BUG_ON(phys_addr + size <= phys_addr);
 
 		if (change_page_attr(ppage, npages, PAGE_KERNEL_NOCACHE) < 0) { 
 			iounmap(p); 
@@ -223,7 +223,7 @@ void iounmap(volatile void __iomem *addr
 		return;
 	} 
 
-	if (p->flags && p->phys_addr < virt_to_phys(high_memory)) { 
+	if ((p->flags >> 24) && p->phys_addr + p->size < virt_to_phys(high_memory)) { 
 		change_page_attr(virt_to_page(__va(p->phys_addr)),
 				 p->size >> PAGE_SHIFT,
 				 PAGE_KERNEL); 				 

If you give them some beating on -mm let me know if you've any problem. (running with this stuff on my machines right now, so far so good)

#1 lore raw
Quoted 84 lines · Andrea Arcangeli

This first patch fixes silent memleak in the pageattr code that I found while searching for the bug Andi fixed in the second patch below (basically reference counting in split page was done on the pmd instead of the pte).

Index: linux-2.5/arch/i386/mm/pageattr.c
===================================================================
RCS file: /home/andrea/crypto/cvs/linux-2.5/arch/i386/mm/pageattr.c,v retrieving revision 1.13
diff -u -p -r1.13 pageattr.c

arch/i386/mm/pageattr.c +7 -6
@@ -117,22 +117,23 @@ __change_page_attr(struct page *page, pg
 	kpte_page = virt_to_page(kpte);
 	if (pgprot_val(prot) != pgprot_val(PAGE_KERNEL)) { 
 		if ((pte_val(*kpte) & _PAGE_PSE) == 0) { 
-			pte_t old = *kpte;
-			pte_t standard = mk_pte(page, PAGE_KERNEL); 
 			set_pte_atomic(kpte, mk_pte(page, prot)); 
-			if (pte_same(old,standard))
-				get_page(kpte_page);
 		} else {
 			struct page *split = split_large_page(address, prot); 
 			if (!split)
 				return -ENOMEM;
-			get_page(kpte_page);
 			set_pmd_pte(kpte,address,mk_pte(split, PAGE_KERNEL));
+			kpte_page = split;
 		}	
+		get_page(kpte_page);
 	} else if ((pte_val(*kpte) & _PAGE_PSE) == 0) { 
 		set_pte_atomic(kpte, mk_pte(page, PAGE_KERNEL));
 		__put_page(kpte_page);
-	}
+	} else
+		BUG();
+
+	/* memleak and potential failed 2M page regeneration */
+	BUG_ON(!page_count(kpte_page));
 
 	if (cpu_has_pse && (page_count(kpte_page) == 1)) {
 		list_add(&kpte_page->lru, &df_list);

Index: linux-2.5/arch/x86_64/mm/pageattr.c
===================================================================
RCS file: /home/andrea/crypto/cvs/linux-2.5/arch/x86_64/mm/pageattr.c,v retrieving revision 1.12
diff -u -p -r1.12 pageattr.c

arch/x86_64/mm/pageattr.c +10 -8
@@ -124,28 +124,33 @@ __change_page_attr(unsigned long address
 	kpte_flags = pte_val(*kpte); 
 	if (pgprot_val(prot) != pgprot_val(ref_prot)) { 
 		if ((kpte_flags & _PAGE_PSE) == 0) { 
-			pte_t old = *kpte;
-			pte_t standard = mk_pte(page, ref_prot); 
-
 			set_pte(kpte, mk_pte(page, prot)); 
-			if (pte_same(old,standard))
-				get_page(kpte_page);
 		} else {
+			/*
+			 * split_large_page will take the reference for this change_page_attr
+			 * on the split page.
+			 */
 			struct page *split = split_large_page(address, prot, ref_prot); 
 			if (!split)
 				return -ENOMEM;
-			get_page(kpte_page);
 			set_pte(kpte,mk_pte(split, ref_prot));
+			kpte_page = split;
 		}	
+		get_page(kpte_page);
 	} else if ((kpte_flags & _PAGE_PSE) == 0) { 
 		set_pte(kpte, mk_pte(page, ref_prot));
 		__put_page(kpte_page);
-	}
+	} else
+		BUG();
 
-	if (page_count(kpte_page) == 1) {
+	switch (page_count(kpte_page)) {
+	case 1:

arch/x86_64/mm/pageattr.c __change_page_attr(unsigned long address · line 156

Quoted 10 lines
arch/x86_64/mm/pageattr.c +4 -1
 		save_page(address, kpte_page); 		     
 		revert_page(address, ref_prot);
-	} 
+		break;
+	case 0:
+		BUG(); /* memleak and failed 2M page regeneration */
+	}
 	return 0;
 } 
 

this new patch obsoletes the above one and fixes one issue with the direct mapping near physical address 0 that is apparently still backed by 4k ptes (i.e. reserved ptes with page_count == 0). That generated a bugcheck false positive. But the bugcheck must be retained, since it signaled a true memleak during normal usage (DEBUG_PAGELLOC is the only special usage that end up calling change_page_attr near phys addr 0, maybe agp could do it too in theory, but it's generally very unlikely to trigger without the debugging knob enabled, especially given we start allocating ram from high zones first). anyways this new patch fixes PAGEALLOC_DEBUG. thanks to Dave Hansen for spotting the problem with bootmem memory and for providing all the useful debugging info.

I added a further bugcheck on x86-64, so that if the bugcheck page_count == 0 triggers, we'll also know if it was a reserved page or not (i.e. the reserved bugcheck will trigger first). I wrote the paging part of head.S of x86-64 and I recall I avoided any usage of 4k pages for the initial direct mapping before firing up the paging engine. So I'm confident x86-64 didn't require any modification and that it would never run into bootmem allocated 4k pages like x86 can run into. the additional bugcheck is just in case (mostly for documentation purposes between the differences of the x86 and x86-64 implementation of pageattr).

I was overoptimistic that these fixes (both this and Andi's ioremap symmetry fix combined) would be enough to fix the real life crash. Something is still not right, but I doubt there's any more bug in pageattr right now. More likely ioremap stuff is still buggy (infact an experimental patch from Andi that doesn't touch pageattr.c at all is reported to fix the problem in practice, problem is I don't see why that patch is fixing anything, I suspect some off by one bit). Anyways I'm quite optimistic the below won't require another modification to the file (I mean unless we want to create a true universal API that doesn't require perfect symmetry to the usage).

DEBUG_PAGEALLOC oopses the X server on 2.6.5 based kernel, but not on a 2.6.9rc based kernel, so it could be a true 2.6.5 bug that I've seen with debug pagealloc (and not a false positive), but that's unrelated to this topic.

This works for me on x86 (with debug pagealloc enabled too) and x86-64.

Index: linux-2.5/arch/i386/mm/pageattr.c
===================================================================
RCS file: /home/andrea/crypto/cvs/linux-2.5/arch/i386/mm/pageattr.c,v retrieving revision 1.13

arch/i386/mm/pageattr.c +18 -10
diff -u -p -r1.13 pageattr.c
@@ -117,27 +117,35 @@ __change_page_attr(struct page *page, pg
 	kpte_page = virt_to_page(kpte);
 	if (pgprot_val(prot) != pgprot_val(PAGE_KERNEL)) { 
 		if ((pte_val(*kpte) & _PAGE_PSE) == 0) { 
-			pte_t old = *kpte;
-			pte_t standard = mk_pte(page, PAGE_KERNEL); 
 			set_pte_atomic(kpte, mk_pte(page, prot)); 
-			if (pte_same(old,standard))
-				get_page(kpte_page);
 		} else {
 			struct page *split = split_large_page(address, prot); 
 			if (!split)
 				return -ENOMEM;
-			get_page(kpte_page);
 			set_pmd_pte(kpte,address,mk_pte(split, PAGE_KERNEL));
+			kpte_page = split;
 		}	
+		get_page(kpte_page);
 	} else if ((pte_val(*kpte) & _PAGE_PSE) == 0) { 
 		set_pte_atomic(kpte, mk_pte(page, PAGE_KERNEL));
 		__put_page(kpte_page);
-	}
+	} else
+		BUG();
 
-	if (cpu_has_pse && (page_count(kpte_page) == 1)) {
-		list_add(&kpte_page->lru, &df_list);
-		revert_page(kpte_page, address);
-	} 
+	/*
+	 * If the pte was reserved, it means it was created at boot
+	 * time (not via split_large_page) and in turn we must not
+	 * replace it with a largepage.
+	 */
+	if (!PageReserved(kpte_page)) {
+		/* memleak and potential failed 2M page regeneration */
+		BUG_ON(!page_count(kpte_page));
+
+		if (cpu_has_pse && (page_count(kpte_page) == 1)) {
+			list_add(&kpte_page->lru, &df_list);
+			revert_page(kpte_page, address);
+		}
+	}
 	return 0;
 } 
 

Index: linux-2.5/arch/x86_64/mm/pageattr.c
===================================================================
RCS file: /home/andrea/crypto/cvs/linux-2.5/arch/x86_64/mm/pageattr.c,v retrieving revision 1.12

arch/x86_64/mm/pageattr.c +17 -9
diff -u -p -r1.12 pageattr.c
@@ -124,28 +124,36 @@ __change_page_attr(unsigned long address
 	kpte_flags = pte_val(*kpte); 
 	if (pgprot_val(prot) != pgprot_val(ref_prot)) { 
 		if ((kpte_flags & _PAGE_PSE) == 0) { 
-			pte_t old = *kpte;
-			pte_t standard = mk_pte(page, ref_prot); 
-
 			set_pte(kpte, mk_pte(page, prot)); 
-			if (pte_same(old,standard))
-				get_page(kpte_page);
 		} else {
+			/*
+			 * split_large_page will take the reference for this change_page_attr
+			 * on the split page.
+			 */
 			struct page *split = split_large_page(address, prot, ref_prot); 
 			if (!split)
 				return -ENOMEM;
-			get_page(kpte_page);
 			set_pte(kpte,mk_pte(split, ref_prot));
+			kpte_page = split;
 		}	
+		get_page(kpte_page);
 	} else if ((kpte_flags & _PAGE_PSE) == 0) { 
 		set_pte(kpte, mk_pte(page, ref_prot));
 		__put_page(kpte_page);
-	}
+	} else
+		BUG();
+
+	/* on x86-64 the direct mapping set at boot is not using 4k pages */
+	BUG_ON(PageReserved(kpte_page));
 
-	if (page_count(kpte_page) == 1) {
+	switch (page_count(kpte_page)) {
+	case 1:
 		save_page(address, kpte_page); 		     
 		revert_page(address, ref_prot);
-	} 
+		break;
+	case 0:
+		BUG(); /* memleak and failed 2M page regeneration */
+	}
 	return 0;
 } 
 
#2 lore raw
Quoted 4 lines

I was overoptimistic that these fixes (both this and Andi's ioremap symmetry fix combined) would be enough to fix the real life crash. Something is still not right, but I doubt there's any more bug in pageattr right now. More likely ioremap stuff is still buggy (infact an

At least the NX handling is still broken on i386 (when reverting back it doesn't clear the NX bit for kernel text)

Also currently I can reproduce another problem on x86-64 with my latest patch (appended for your reference). The change_page_attr_addr() addition is not needed in the stock kernel yet, only together with another patch.

I still don't like how you remove the reversal handling completely.

Quoted 2 lines

Review

Quoted 2 lines
(quoted diff) +0 -2
-           if (pte_same(old,standard))
-               get_page(kpte_page);

-Andi

arch/x86_64/mm/ioremap.c +48 -28
Index: linux/arch/x86_64/mm/ioremap.c
===================================================================
@@ -16,7 +16,7 @@
 #include <asm/fixmap.h>
 #include <asm/cacheflush.h>
 #include <asm/tlbflush.h>
-
+#include <asm/proto.h>
 
 static inline void remap_area_pte(pte_t * pte, unsigned long address, unsigned long size,
 	unsigned long phys_addr, unsigned long flags)
@@ -98,8 +98,32 @@
 	return error;
 }
 
+/* 
+ * Fix up the linear direct mapping of the kernel to avoid cache attribute 
+ * conflicts.
+ */  
+static int 
+ioremap_change_attr(unsigned long phys_addr, unsigned long size, 
+					unsigned long flags)
+{
+	int err = 0;
+	if (flags && phys_addr + size - 1 < (end_pfn_map << PAGE_SHIFT)) { 
+		unsigned long npages = (size + PAGE_SIZE - 1) >> PAGE_SHIFT;
+		unsigned long vaddr = (unsigned long) __va(phys_addr);
+		
+		/* 
+ 		 * Must use a address here and not struct page because the phys addr
+		 * can be a in hole between nodes and not have an memmap entry. 
+		 */
+		err = change_page_attr_addr(vaddr, npages, __pgprot(flags));
+		if (!err)
+			global_flush_tlb();
+	} 
+	return err;
+}
+
 /*
- * Generic mapping function (not visible outside):
+ * Generic mapping function
  */
 
 /*
@@ -155,12 +179,17 @@
 	/*
 	 * Ok, go for it..
 	 */
-	area = get_vm_area(size, VM_IOREMAP);
+	area = get_vm_area(size, VM_IOREMAP | (flags << 24));
 	if (!area)
 		return NULL;
 	area->phys_addr = phys_addr;
 	addr = area->addr;
 	if (remap_area_pages((unsigned long) addr, phys_addr, size, flags)) {
+		remove_vm_area((void *)(PAGE_MASK & (unsigned long) addr));
+		return NULL;
+	}
+	if (ioremap_change_attr(phys_addr, size, flags) < 0) { 
+		area->flags &= 0xffffff;
 		vunmap(addr);
 		return NULL;
 	}
@@ -191,43 +220,34 @@
 
 void __iomem *ioremap_nocache (unsigned long phys_addr, unsigned long size)
 {
-	void __iomem *p = __ioremap(phys_addr, size, _PAGE_PCD);
-	if (!p) 
-		return p; 
-
-	if (phys_addr + size < virt_to_phys(high_memory)) { 
-		struct page *ppage = virt_to_page(__va(phys_addr));		
-		unsigned long npages = (size + PAGE_SIZE - 1) >> PAGE_SHIFT;
-
-		BUG_ON(phys_addr+size > (unsigned long)high_memory);
-		BUG_ON(phys_addr + size < phys_addr);
-
-		if (change_page_attr(ppage, npages, PAGE_KERNEL_NOCACHE) < 0) { 
-			iounmap(p); 
-			p = NULL;
-		}
-		global_flush_tlb();
-	} 
-
-	return p;					
+	return __ioremap(phys_addr, size, _PAGE_PCD);
 }
 
 void iounmap(volatile void __iomem *addr)
 {
-	struct vm_struct *p;
+	struct vm_struct *p, **pprev;
+	
 	if (addr <= high_memory) 
 		return; 
-	p = remove_vm_area((void *)(PAGE_MASK & (unsigned long) addr)); 
+
+	write_lock(&vmlist_lock);
+	for (p = vmlist, pprev = &vmlist; p != NULL; pprev = &p->next, p = *pprev)
+		if (p->addr == (void *)(PAGE_MASK & (unsigned long)addr)) 
+			break; 
 	if (!p) { 
 		printk("__iounmap: bad address %p\n", addr);
-		return;
-	} 
-
-	if (p->flags && p->phys_addr < virt_to_phys(high_memory)) { 
+		goto out_unlock;
+	}	
+	*pprev = p->next;
+	unmap_vm_area(p);
+	if ((p->flags >> 24) && 
+		p->phys_addr + p->size - 1 < virt_to_phys(high_memory)) { 
 		change_page_attr(virt_to_page(__va(p->phys_addr)),
 				 p->size >> PAGE_SHIFT,
 				 PAGE_KERNEL); 				 
 		global_flush_tlb();
 	} 
+out_unlock:	
+	write_unlock(&vmlist_lock);
 	kfree(p); 
 }
include/asm-x86_64/cacheflush.h +1 -0
Index: linux/include/asm-x86_64/cacheflush.h
===================================================================
@@ -25,5 +25,6 @@
 
 void global_flush_tlb(void); 
 int change_page_attr(struct page *page, int numpages, pgprot_t prot);
+int change_page_attr_addr(unsigned long addr, int numpages, pgprot_t prot);
 
 #endif /* _X8664_CACHEFLUSH_H */
arch/x86_64/mm/pageattr.c +21 -15
Index: linux/arch/x86_64/mm/pageattr.c
===================================================================
@@ -111,13 +111,12 @@
 }      
 
 static int
-__change_page_attr(unsigned long address, struct page *page, pgprot_t prot, 
-		   pgprot_t ref_prot)
+__change_page_attr(unsigned long address, unsigned long pfn, pgprot_t prot, 
+				   pgprot_t ref_prot)
 { 
 	pte_t *kpte; 
 	struct page *kpte_page;
 	unsigned kpte_flags;
-
 	kpte = lookup_address(address);
 	if (!kpte) return 0;
 	kpte_page = virt_to_page(((unsigned long)kpte) & PAGE_MASK);
@@ -125,20 +124,20 @@
 	if (pgprot_val(prot) != pgprot_val(ref_prot)) { 
 		if ((kpte_flags & _PAGE_PSE) == 0) { 
 			pte_t old = *kpte;
-			pte_t standard = mk_pte(page, ref_prot); 
+			pte_t standard = pfn_pte(pfn, ref_prot); 
 
-			set_pte(kpte, mk_pte(page, prot)); 
+			set_pte(kpte, pfn_pte(pfn, prot)); 
 			if (pte_same(old,standard))
 				get_page(kpte_page);
 		} else {
 			struct page *split = split_large_page(address, prot, ref_prot); 
 			if (!split)
 				return -ENOMEM;
-			get_page(kpte_page);
+			get_page(split);
 			set_pte(kpte,mk_pte(split, ref_prot));
 		}	
 	} else if ((kpte_flags & _PAGE_PSE) == 0) { 
-		set_pte(kpte, mk_pte(page, ref_prot));
+		set_pte(kpte, pfn_pte(pfn, ref_prot));
 		__put_page(kpte_page);
 	}
 
@@ -162,31 +161,38 @@
  * 
  * Caller must call global_flush_tlb() after this.
  */
-int change_page_attr(struct page *page, int numpages, pgprot_t prot)
+int change_page_attr_addr(unsigned long address, int numpages, pgprot_t prot)
 {
 	int err = 0; 
 	int i; 
 
 	down_write(&init_mm.mmap_sem);
-	for (i = 0; i < numpages; !err && i++, page++) { 
-		unsigned long address = (unsigned long)page_address(page); 
-		err = __change_page_attr(address, page, prot, PAGE_KERNEL); 
+	for (i = 0; i < numpages; i++, address += PAGE_SIZE) { 
+		unsigned long pfn = __pa(address) >> PAGE_SHIFT;
+	
+		err = __change_page_attr(address, pfn, prot, PAGE_KERNEL); 
 		if (err) 
 			break; 
 		/* Handle kernel mapping too which aliases part of the
 		 * lowmem */
 		/* Disabled right now. Fixme */ 
-		if (0 && page_to_phys(page) < KERNEL_TEXT_SIZE) {		
+		if (0 && __pa(address) < KERNEL_TEXT_SIZE) {		
 			unsigned long addr2;
-			addr2 = __START_KERNEL_map + page_to_phys(page);
-			err = __change_page_attr(addr2, page, prot, 
-						 PAGE_KERNEL_EXEC);
+			addr2 = __START_KERNEL_map + __pa(address);
+			err = __change_page_attr(addr2, pfn, prot, PAGE_KERNEL_EXEC);
 		} 
 	} 	
 	up_write(&init_mm.mmap_sem); 
 	return err;
 }
 
+/* Don't call this for MMIO areas that may not have a mem_map entry */
+int change_page_attr(struct page *page, int numpages, pgprot_t prot)
+{
+	unsigned long addr = (unsigned long)page_address(page);
+	return change_page_attr_addr(addr, numpages, prot);
+}
+
 void global_flush_tlb(void)
 { 
 	struct deferred_page *df, *next_df;
#3 lore raw
Quoted 2 lines · Andi Kleen

At least the NX handling is still broken on i386 (when reverting back it doesn't clear the NX bit for kernel text)

I'd never use a 32bit kernel on a x86-64 box, so that's sort of low interest bug to me, but acked.

Quoted 1 line

I still don't like how you remove the reversal handling completely.

that's to make it fully symmetric. If you're so attached to the old API, I'm not going to care if you want it back, but if there are 3 symmetric users with the third going to work on a different page, my code will work the previous code will corrupt the mapping due the refcount going down too fast. If you could mention a single case where it would make sense not to be symmetric, I would change my mind. I find so much simpler to remember that as far as I'm always symmetric the refcounting will go right no matter what the other tasks are doing around. The below special case just complicates the API for no good reason. My problem is that it requires somebody understanding pageattr.c and fixing bugs on it daily to remember and to be able to use your below API IMHO (I had no idea myself of this undocumented below subtle detail until I read the code, infact it was the first thing I've removed after I noticed the asymmetry it generated). Bug again no problem, I'll try hard to remember it if we're going to keep it ;).

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(quoted diff) +0 -2
-           if (pte_same(old,standard))
-               get_page(kpte_page);
#4 lore raw

Hello,

the fix for this bug in 2.6.11-rc3 for this bug is wrong, I thought I posted the right one (that I already applied to all SUSE branches except the HEAD branch that probably is in sync with the inferior fix in mainline). Right fix is the below one. And then of course drop those useless -PAGE_SIZE in change_page_attr p->size parameter in the arch code. Exposing vmalloc.c internal knowledge of the guard-page-size into the arch code is an unnecssary breakage of the vmalloc abstraction and I've been very careful to avoid that and I thought I posted it too. It has been even quicker to fix it right for me in a single place than to hand edit the (not single) change_page_attr callers. I'd like the right fix to obsolete the hand editing of arch code in multiple places that breaks the layering. Thanks.

From: Andrea Arcangeli <[email protected]>
Subject: reject zero page vm-area request, align size properly
and hide the guard page from the callers like ioremap - this avoids a kernel crash due one more page being passed to change_page_attr

mm/vmalloc.c renamed from mm/vmalloc.c.~1~ +11 -4
@@ -199,20 +199,22 @@ struct vm_struct *__get_vm_area(unsigned
 		align = 1ul << bit;
 	}
 	addr = ALIGN(start, align);
+	size = PAGE_ALIGN(size);
 
 	area = kmalloc(sizeof(*area), GFP_KERNEL);
 	if (unlikely(!area))
 		return NULL;
 
-	/*
-	 * We always allocate a guard page.
-	 */
-	size += PAGE_SIZE;
 	if (unlikely(!size)) {
 		kfree (area);
 		return NULL;
 	}
 
+	/*
+	 * We always allocate a guard page.
+	 */
+	size += PAGE_SIZE;
+
 	write_lock(&vmlist_lock);
 	for (p = &vmlist; (tmp = *p) != NULL ;p = &tmp->next) {
 		if ((unsigned long)tmp->addr < addr) {
@@ -290,6 +292,11 @@ found:
 	unmap_vm_area(tmp);
 	*p = tmp->next;
 	write_unlock(&vmlist_lock);
+
+	/*
+	 * Remove the guard page.
+	 */
+	tmp->size -= PAGE_SIZE;
 	return tmp;
 }
 
#5 lore raw
Quoted 3 lines · Andrea Arcangeli

Hello,

the fix for this bug in 2.6.11-rc3 for this bug is wrong, I thought I

Can you describe what exactly is wrong?

-Andi

#6 lore raw
Quoted 6 lines · Andi Kleen
Quoted 3 lines · Andrea Arcangeli

Hello,

the fix for this bug in 2.6.11-rc3 for this bug is wrong, I thought I

Can you describe what exactly is wrong?

the wrong thing is that if I change the size of the guard page in vmalloc.c, the arch code will break. the guard page is a debugging knob, people may want to remove it someday to save virtual address space, and they shouldn't be required to look after N architectures.

Plus people will keep forgetting that p->size has the guard page included if you don't fix it the right way, x86-64 and x86 have been corrupting the pageattr of the next physical pages because of that for a long time.

Plus if you applied the best fix in vmalloc.c, you wouldn't even need to touch x86 and x86-64 arch dependent code, you could call change_page_attr on p->size without having to do -PAGE_SIZE.

At least you should define a VMALLOC_GUARD_PAGE_SIZE, or something, the hardcoding of p->size-PAGE_SIZE looks wrong to me.

#7 lore raw
Quoted 4 lines · Andrea Arcangeli
Quoted 3 lines · Andi Kleen
Quoted 2 lines · Andrea Arcangeli

Hello,

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

the fix for this bug in 2.6.11-rc3 for this bug is wrong, I thought I

Can you describe what exactly is wrong?

the wrong thing is that if I change the size of the guard page in vmalloc.c, the arch code will break. the guard page is a debugging knob, people may want to remove it someday to save virtual address space, and they shouldn't be required to look after N architectures.

Ok, so it is merely a cosmetic issue.

I was worried about an actual bug :)

No problem with changing it, but hopefully after 2.6.11.

-Andi

#8 lore raw
Quoted 1 line · Andi Kleen

Ok, so it is merely a cosmetic issue.

In practice it's solved, I didn't mean we had a bug still, I'm just suggesting to fix it in a diffeerent way.

It's not just for cosmetic reasons that I suggest to change this. My point is that the _real_ reason why we had the bug in the first place is that people forgets that p->size includes the guard page (because it shouldn't include the guard page). The fundamental problem of vmalloc exposing the guard page to the callers (which makes it prone for mistakes, and prone for breakage if somebody needs all virtual space and removes the guard page), isn't solved yet.

Quoted 1 line

I was worried about an actual bug :)

No bugs anymore in practice, the -PAGE_SIZE in arch/x86* is clearly equivalent to -PAGE_SIZE in mm/vmalloc.c, in practice it's the same either ways.

Quoted 1 line

No problem with changing it, but hopefully after 2.6.11.

Ok fine with me, take your time it's clearly not urgent because in practice no bug can be triggered anymore ;). thanks!

#9 lore raw
Quoted 9 lines · Andrea Arcangeli

On Tue, Feb 15, 2005 at 11:51:52AM +0100, Andi Kleen wrote:

It's not just for cosmetic reasons that I suggest to change this. My point is that the _real_ reason why we had the bug in the first place is that people forgets that p->size includes the guard page (because it shouldn't include the guard page). The fundamental problem of vmalloc exposing the guard page to the callers (which makes it prone for mistakes, and prone for breakage if somebody needs all virtual space and removes the guard page), isn't solved yet.

Strongly agree with you. I remember this nuisance from large-page-patch
days: the guard page (if any) really should be an implementation detail private to mm/vmalloc.c, never exposed outside.

Quoted 4 lines
Quoted 1 line

No problem with changing it, but hopefully after 2.6.11.

Ok fine with me, take your time it's clearly not urgent because in practice no bug can be triggered anymore ;). thanks!

Yes, not urgent. The instance of exposure I remember is one of the PAGE_SIZEs in fs/proc/kcore.c.

Hugh