RFC: Transparent Hugepage support

Hello,

Lately I've been working to make KVM use hugepages transparently without the usual restrictions of hugetlbfs. Some of the restrictions I'd like to see removed:

1) hugepages have to be swappable or the guest physical memory remains locked in RAM and can't be paged out to swap

2) if a hugepage allocation fails, regular pages should be allocated instead and mixed in the same vma without any failure and without userland noticing

3) if some task quits and more hugepages become available in the buddy, guest physical memory backed by regular pages should be relocated on hugepages automatically in regions under madvise(MADV_HUGEPAGE) (ideally event driven by waking up the kernel deamon if the order=HPAGE_SHIFT-PAGE_SHIFT list becomes not null)

The first (and more tedious) part of this work requires allowing the VM to handle anonymous hugepages mixed with regular pages transparently on regular anonymous vmas. This is what this patch tries to achieve in the least intrusive possible way. We want hugepages and hugetlb to be used in a way so that all applications can benefit without changes (as usual we leverage the KVM virtualization design:
by improving the Linux VM at large, KVM gets the performance boost too).

The most important design choice is: always fallback to 4k allocation if the hugepage allocation fails! This is the _very_ opposite of some large pagecache patches that failed with -EIO back then if a 64k (or similar) allocation failed...

Second important decision (to reduce the impact of the feature on the existing pagetable handling code) is that at any time we can split an hugepage into 512 regular pages and it has to be done with an operation that can't fail. This way the reliability of the swapping isn't decreased (no need to allocate memory when we are short on memory to swap) and it's trivial to plug a split_huge_page* one-liner where needed without polluting the VM. Over time we can teach mprotect, mremap and friends to handle pmd_trans_huge natively without calling split_huge_page*. The fact it can't fail isn't just for swap:
if split_huge_page would return -ENOMEM (instead of the current void) we'd need to rollback the mprotect from the middle of it (ideally including undoing the split_vma) which would be a big change and in the very wrong direction (it'd likely be simpler not to call split_huge_page at all and to teach mprotect and friends to handle hugepages instead of rolling them back from the middle). In short the very value of split_huge_page is that it can't fail.

The collapsing and madvise(MADV_HUGEPAGE) part will remain separated and incremental and it'll just be an "harmless" addition later if this initial part is agreed upon. It also should be noted that locking-wise replacing regular pages with hugepages is going to be very easy if compared to what I'm doing below in split_huge_page, as it will only happen when page_count(page) matches page_mapcount(page) if we can take the PG_lock and mmap_sem in write mode. collapse_huge_page will be a "best effort" that (unlike split_huge_page) can fail at the minimal sign of trouble and we can try again later. collapse_huge_page will be similar to how KSM works and the madvise(MADV_HUGEPAGE) will work similar to madvise(MADV_MERGEABLE).

For now the transparent_hugepage sysctl is for debug only (it'll be moved to sysfs so that the kernel daemon that collapse huge pages will be tuned from the same directory too), and we need more stats (notably the split_huge_page* from smaps has to be removed and the amount of hugepages in each vma should become visible in smaps too). Adam expressed the interest to add hugepage visibility in pagemap too.

The default I like is that transparent hugepages are used at page fault time if they're available in O(1) in the buddy. This can be disabled via sysctl/sysfs setting the value to 0, and if it is disabled they will only be used inside MADV_HUGEPAGE regions. MADV_HUGEPAGE regions will do a lot more effort to shrink caches to create hugepages during the page fault too and not only through the collapse_huge_page kernel daemon. Then a future sysctl/sysfs value of 2 tune can force all page faults to do a lot of efforts to defrag cache and create hugepages whenever possible while still leaving the collapse_huge_page daemon working strictly in MADV_HUGEPAGE regions. Obviously KVM will call madvise(MADV_HUGEPAGE) right after the other madvise it's already running on the guest physical memory host virtual ranges. Ideally the daemon could run system-wide too but I think that would tend to waste some CPU but it remains a possibility and an heuristic would be to timestamp the vma creation and start to call collapse_huge_page from the oldest vmas.

The pmd_trans_frozen/pmd_trans_huge locking is very solid. The put_page (from get_user_page users that can't use mmu notifier like O_DIRECT) that runs against a __split_huge_page_refcount instead was a pain to serialize in a way that would result always in a coherent page count for both tail and head. I think my locking solution with a compound_lock taken only after the page_first is valid and is still a PageHead should be safe but it surely needs review from SMP race point of view. In short there is no current existing way to serialize the O_DIRECT final put_page against split_huge_page_refcount so I had to invent a new one (O_DIRECT loses knowledge on the mapping status by the time gup_fast returns so...). And I didn't want to impact all gup/gup_fast users for now, maybe if we change the gup interface substantially we can avoid this locking, I admit I didn't think too much about it because changing the gup unpinning interface would be invasive.

If we ignored O_DIRECT we could stick to the existing compound refcounting code, by simply adding a get_user_pages_fast_flags(foll_flags) where KVM (and any other mmu notifier user) would call it without FOLL_GET (and if FOLL_GET isn't set we'd just BUG_ON if nobody registered itself in the current task mmu notifier list yet). But O_DIRECT is fundamental for decent performance of virtualized I/O on fast storage so we can't avoid it to solve the race of put_page against split_huge_page_refcount to achieve a complete hugepage feature for KVM.

The KVM patch that enables KVM to run on transparent hugepages will follow later (Marcelo apparently already run KVM with hugepages on top of this ;).

Swap and oom works fine (well just like with regular pages ;). MMU notifier is handled transparently too, with the exception of the young bit on the pmd, that didn't have a range check but I think KVM will be fine because the whole point of hugepages is that EPT/NPT will also use a huge pmd when they notice gup returns pages with PageCompound set, so they won't care of a range and there's just the pmd young bit to check in that case.

There are likely still many missing things, especially in the basic accounting area (overcommit/anon-rss) I didn't pay much attention to and lots of cleanups possible (including perhaps splitting the patch as usual to make merging simpler). This is still a RFC after all...

NOTE: in some cases if the L2 cache is small, this may slowdown and waste memory during COWs because 4M of memory are accessed in a single fault instead of 8k (the payoff is that after COW the program can run faster). So we might want to switch the copy_huge_page (and clear_huge_page too) to not temporal stores. I also extensively researched ways to avoid this cache trashing with a full prefault logic that would cow in 8k/16k/32k/64k up to 1M (I can send those patches that fully implemented prefault) but I concluded they're not worth it and they add an huge additional complexity to save a little bit of memory and some cache during app startup, but they still don't improve substantially the cache-trashing during startup (not as good as only 4k). One reason is that those 4k pte entries copied are still mapped on a perfectly cache-colored hugepage, so the trashing is the worst one can generate in those copies (cow of 4k page copies aren't so well colored so they trashes less, but again this results in software running faster after the page fault). Those prefault patches allowed things like a pte where post-cow pages were local 4k regular anon pages and the not-yet-cowed pte entries were pointing in the middle of some hugepage mapped read-only. If it doesn't payoff substantially with todays hardware it will payoff even less in the future with larger l2 caches, and the prefault logic would blot the VM a lot. If one is emebdded and can't handle the sysctl to be 1 by default because of cache trashing effects during page faults, it is simple enough to just disable transparent hugepage globally and let hugepages be allocated only in the MADV_HUGEPAGE region (both at page fault time, and if enabled with the collapse_huge_page too through the kernel daemon).

This patch supports only hugepages mapped in the pmd, archs that have smaller hugepages will not fit in this patch alone... maybe we can achieve mixed page size of them with a small change, maybe not. I didn't think much about it so far...

Some performance result:

vmx andrea # LD_PRELOAD=/usr/lib64/libhugetlbfs.so HUGETLB_MORECORE=yes HUGETLB_PATH=/mnt/huge/ ./largep ages3
memset page fault 1566023
memset tlb miss 453854
memset second tlb miss 453321
random access tlb miss 41635
random access second tlb miss 41658
vmx andrea # LD_PRELOAD=/usr/lib64/libhugetlbfs.so HUGETLB_MORECORE=yes HUGETLB_PATH=/mnt/huge/ ./largepages3 memset page fault 1566471
memset tlb miss 453375
memset second tlb miss 453320
random access tlb miss 41636
random access second tlb miss 41637
vmx andrea # ./largepages3
memset page fault 1566642
memset tlb miss 453417
memset second tlb miss 453313
random access tlb miss 41630
random access second tlb miss 41647
vmx andrea # ./largepages3
memset page fault 1566872
memset tlb miss 453418
memset second tlb miss 453315
random access tlb miss 41618
random access second tlb miss 41659
vmx andrea # echo 0 > /proc/sys/vm/transparent_hugepage
vmx andrea # ./largepages3
memset page fault 2182476
memset tlb miss 460305
memset second tlb miss 460179
random access tlb miss 44483
random access second tlb miss 44186
vmx andrea # ./largepages3
memset page fault 2182791
memset tlb miss 460742
memset second tlb miss 459962
random access tlb miss 43981
random access second tlb miss 43988

============
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/time.h>

#define SIZE (3UL*1024*1024*1024)

int main()

{
	char *p = malloc(SIZE), *p2;
	struct timeval before, after;

	gettimeofday(&before, NULL);
	memset(p, 0, SIZE);
	gettimeofday(&after, NULL);
	printf("memset page fault %Lu\n",
	       (after.tv_sec-before.tv_sec)*1000000UL +
	       after.tv_usec-before.tv_usec);

	gettimeofday(&before, NULL);
	memset(p, 0, SIZE);
	gettimeofday(&after, NULL);
	printf("memset tlb miss %Lu\n",
	       (after.tv_sec-before.tv_sec)*1000000UL +
	       after.tv_usec-before.tv_usec);

	gettimeofday(&before, NULL);
	memset(p, 0, SIZE);
	gettimeofday(&after, NULL);
	printf("memset second tlb miss %Lu\n",
	       (after.tv_sec-before.tv_sec)*1000000UL +
	       after.tv_usec-before.tv_usec);

	gettimeofday(&before, NULL);
	for (p2 = p; p2 < p+SIZE; p2 += 4096)
		*p2 = 0;
	gettimeofday(&after, NULL);
	printf("random access tlb miss %Lu\n",
	       (after.tv_sec-before.tv_sec)*1000000UL +
	       after.tv_usec-before.tv_usec);

	gettimeofday(&before, NULL);
	for (p2 = p; p2 < p+SIZE; p2 += 4096)
		*p2 = 0;
	gettimeofday(&after, NULL);
	printf("random access second tlb miss %Lu\n",
	       (after.tv_sec-before.tv_sec)*1000000UL +
	       after.tv_usec-before.tv_usec);

	return 0;
}

============

Comments welcome, thanks!

---

@@ -449,6 +449,11 @@ static inline void pte_update(struct mm_
 {
 	PVOP_VCALL3(pv_mmu_ops.pte_update, mm, addr, ptep);
 }
+static inline void pmd_update(struct mm_struct *mm, unsigned long addr,
+			      pmd_t *pmdp)
+{
+	PVOP_VCALL3(pv_mmu_ops.pmd_update, mm, addr, pmdp);
+}
 
 static inline void pte_update_defer(struct mm_struct *mm, unsigned long addr,
 				    pte_t *ptep)

@@ -456,6 +461,12 @@ static inline void pte_update_defer(stru
 	PVOP_VCALL3(pv_mmu_ops.pte_update_defer, mm, addr, ptep);
 }
 
+static inline void pmd_update_defer(struct mm_struct *mm, unsigned long addr,
+				    pmd_t *pmdp)
+{
+	PVOP_VCALL3(pv_mmu_ops.pmd_update_defer, mm, addr, pmdp);
+}
+
 static inline pte_t __pte(pteval_t val)
 {
 	pteval_t ret;

@@ -557,6 +568,16 @@ static inline void set_pte_at(struct mm_
 		PVOP_VCALL4(pv_mmu_ops.set_pte_at, mm, addr, ptep, pte.pte);
 }
 
+static inline void set_pmd_at(struct mm_struct *mm, unsigned long addr,
+			      pmd_t *pmdp, pmd_t pmd)
+{
+	if (sizeof(pmdval_t) > sizeof(long))
+		/* 5 arg words */
+		pv_mmu_ops.set_pmd_at(mm, addr, pmdp, pmd);
+	else
+		PVOP_VCALL4(pv_mmu_ops.set_pmd_at, mm, addr, pmdp, pmd.pmd);
+}
+
 static inline void set_pmd(pmd_t *pmdp, pmd_t pmd)
 {
 	pmdval_t val = native_pmd_val(pmd);

@@ -266,10 +266,16 @@ struct pv_mmu_ops {
 	void (*set_pte_at)(struct mm_struct *mm, unsigned long addr,
 			   pte_t *ptep, pte_t pteval);
 	void (*set_pmd)(pmd_t *pmdp, pmd_t pmdval);
+	void (*set_pmd_at)(struct mm_struct *mm, unsigned long addr,
+			   pmd_t *pmdp, pmd_t pmdval);
 	void (*pte_update)(struct mm_struct *mm, unsigned long addr,
 			   pte_t *ptep);
 	void (*pte_update_defer)(struct mm_struct *mm,
 				 unsigned long addr, pte_t *ptep);
+	void (*pmd_update)(struct mm_struct *mm, unsigned long addr,
+			   pmd_t *pmdp);
+	void (*pmd_update_defer)(struct mm_struct *mm,
+				 unsigned long addr, pmd_t *pmdp);
 
 	pte_t (*ptep_modify_prot_start)(struct mm_struct *mm, unsigned long addr,
 					pte_t *ptep);

@@ -31,6 +31,11 @@ static inline void native_set_pte(pte_t 
 	ptep->pte_low = pte.pte_low;
 }
 
+static inline void native_set_pmd(pmd_t *pmdp, pmd_t pmd)
+{
+	pmdp->pmd = pmd.pmd;
+}
+
 static inline void native_set_pte_atomic(pte_t *ptep, pte_t pte)
 {
 	set_64bit((unsigned long long *)(ptep), native_pte_val(pte));

@@ -71,6 +71,18 @@ static inline pte_t native_ptep_get_and_
 	return ret;
 #endif
 }
+static inline pmd_t native_pmdp_get_and_clear(pmd_t *xp)
+{
+#ifdef CONFIG_SMP
+	return native_make_pmd(xchg(&xp->pmd, 0));
+#else
+	/* native_local_pmdp_get_and_clear,
+	   but duplicated because of cyclic dependency */
+	pmd_t ret = *xp;
+	native_pmd_clear(NULL, 0, xp);
+	return ret;
+#endif
+}
 
 static inline void native_set_pmd(pmd_t *pmdp, pmd_t pmd)
 {

@@ -422,8 +422,11 @@ struct pv_mmu_ops pv_mmu_ops = {
 	.set_pte = native_set_pte,
 	.set_pte_at = native_set_pte_at,
 	.set_pmd = native_set_pmd,
+	.set_pmd_at = native_set_pmd_at,
 	.pte_update = paravirt_nop,
 	.pte_update_defer = paravirt_nop,
+	.pmd_update = paravirt_nop,
+	.pmd_update_defer = paravirt_nop,
 
 	.ptep_modify_prot_start = __ptep_modify_prot_start,
 	.ptep_modify_prot_commit = __ptep_modify_prot_commit,

@@ -179,6 +179,7 @@ static void mark_screen_rdonly(struct mm
 	if (pud_none_or_clear_bad(pud))
 		goto out;
 	pmd = pmd_offset(pud, 0xA0000);
+	split_huge_page_mm(mm, 0xA0000, pmd);
 	if (pmd_none_or_clear_bad(pmd))
 		goto out;
 	pte = pte_offset_map_lock(mm, pmd, 0xA0000, &ptl);

@@ -128,6 +128,10 @@ static noinline int gup_huge_pmd(pmd_t p
 	do {
 		VM_BUG_ON(compound_head(page) != head);
 		pages[*nr] = page;
+		if (PageTail(page)) {
+			VM_BUG_ON(atomic_read(&page->_count) < 0);
+			atomic_inc(&page->_count);
+		}
 		(*nr)++;
 		page++;
 		refs++;

@@ -148,7 +152,7 @@ static int gup_pmd_range(pud_t pud, unsi
 		pmd_t pmd = *pmdp;
 
 		next = pmd_addr_end(addr, end);
-		if (pmd_none(pmd))
+		if (!pmd_present(pmd))
 			return 0;
 		if (unlikely(pmd_large(pmd))) {
 			if (!gup_huge_pmd(pmd, addr, next, write, pages, nr))

@@ -287,6 +287,9 @@ struct mm_struct {
 #ifdef CONFIG_MMU_NOTIFIER
 	struct mmu_notifier_mm *mmu_notifier_mm;
 #endif
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+	pgtable_t pmd_huge_pte; /* protected by page_table_lock */
+#endif
 };
 
 /* Future-safe accessor for struct mm_struct's cpu_vm_mask. */

@@ -243,6 +243,32 @@ static inline void mmu_notifier_mm_destr
 	__pte;								\
 })
 
+#define pmdp_clear_flush_notify(__vma, __address, __pmdp)		\
+({									\
+	pmd_t __pmd;							\
+	struct vm_area_struct *___vma = __vma;				\
+	unsigned long ___address = __address;				\
+	VM_BUG_ON(__address & ~HPAGE_MASK);				\
+	mmu_notifier_invalidate_range_start(___vma->vm_mm, ___address,	\
+					    (__address)+HPAGE_SIZE);	\
+	__pmd = pmdp_clear_flush(___vma, ___address, __pmdp);		\
+	mmu_notifier_invalidate_range_end(___vma->vm_mm, ___address,	\
+					  (__address)+HPAGE_SIZE);	\
+	__pmd;								\
+})
+
+#define pmdp_freeze_flush_notify(__vma, __address, __pmdp)		\
+({									\
+	struct vm_area_struct *___vma = __vma;				\
+	unsigned long ___address = __address;				\
+	VM_BUG_ON(__address & ~HPAGE_MASK);				\
+	mmu_notifier_invalidate_range_start(___vma->vm_mm, ___address,	\
+					    (__address)+HPAGE_SIZE);	\
+	pmdp_freeze_flush(___vma, ___address, __pmdp);			\
+	mmu_notifier_invalidate_range_end(___vma->vm_mm, ___address,	\
+					  (__address)+HPAGE_SIZE);	\
+})
+
 #define ptep_clear_flush_young_notify(__vma, __address, __ptep)		\
 ({									\
 	int __young;							\

@@ -254,6 +280,17 @@ static inline void mmu_notifier_mm_destr
 	__young;							\
 })
 
+#define pmdp_clear_flush_young_notify(__vma, __address, __pmdp)		\
+({									\
+	int __young;							\
+	struct vm_area_struct *___vma = __vma;				\
+	unsigned long ___address = __address;				\
+	__young = pmdp_clear_flush_young(___vma, ___address, __pmdp);	\
+	__young |= mmu_notifier_clear_flush_young(___vma->vm_mm,	\
+						  ___address);		\
+	__young;							\
+})
+
 #define set_pte_at_notify(__mm, __address, __ptep, __pte)		\
 ({									\
 	struct mm_struct *___mm = __mm;					\

@@ -305,7 +342,10 @@ static inline void mmu_notifier_mm_destr
 }
 
 #define ptep_clear_flush_young_notify ptep_clear_flush_young
+#define pmdp_clear_flush_young_notify pmdp_clear_flush_young
 #define ptep_clear_flush_notify ptep_clear_flush
+#define pmdp_clear_flush_notify pmdp_clear_flush
+#define pmdp_freeze_flush_notify pmdp_freeze_flush
 #define set_pte_at_notify set_pte_at
 
 #endif /* CONFIG_MMU_NOTIFIER */

@@ -108,6 +108,7 @@ enum pageflags {
 #ifdef CONFIG_MEMORY_FAILURE
 	PG_hwpoison,		/* hardware poisoned page. Don't touch */
 #endif
+	PG_compound_lock,
 	__NR_PAGEFLAGS,
 
 	/* Filesystems */

@@ -239,6 +240,8 @@ PAGEFLAG(MappedToDisk, mappedtodisk)
 PAGEFLAG(Reclaim, reclaim) TESTCLEARFLAG(Reclaim, reclaim)
 PAGEFLAG(Readahead, reclaim)		/* Reminder to do async read-ahead */
 
+PAGEFLAG(CompoundLock, compound_lock) TESTSETFLAG(CompoundLock, compound_lock)
+
 #ifdef CONFIG_HIGHMEM
 /*
  * Must use a macro here due to header dependency issues. page_zone() is not

@@ -346,7 +349,7 @@ static inline void set_page_writeback(st
  * tests can be used in performance sensitive paths. PageCompound is
  * generally not used in hot code paths.
  */
-__PAGEFLAG(Head, head)
+__PAGEFLAG(Head, head) CLEARPAGEFLAG(Head, head)
 __PAGEFLAG(Tail, tail)
 
 static inline int PageCompound(struct page *page)

@@ -354,6 +357,13 @@ static inline int PageCompound(struct pa
 	return page->flags & ((1L << PG_head) | (1L << PG_tail));
 
 }
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+static inline void ClearPageCompound(struct page *page)
+{
+	BUG_ON(!PageHead(page));
+	ClearPageHead(page);
+}
+#endif
 #else
 /*
  * Reduce page flag use as much as possible by overlapping

@@ -391,6 +401,14 @@ static inline void __ClearPageTail(struc
 	page->flags &= ~PG_head_tail_mask;
 }
 
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+static inline void ClearPageCompound(struct page *page)
+{
+	BUG_ON(page->flags & PG_head_tail_mask != (1L << PG_compound));
+	ClearPageCompound(page);
+}
+#endif
+
 #endif /* !PAGEFLAGS_EXTENDED */
 
 #ifdef CONFIG_HAVE_MLOCKED_PAGE_BIT

@@ -73,6 +73,7 @@ void page_remove_rmap(struct page *);
 
 static inline void page_dup_rmap(struct page *page)
 {
+	VM_BUG_ON(PageTail(page));
 	atomic_inc(&page->_mapcount);
 }
 

@@ -496,6 +496,9 @@ void __mmdrop(struct mm_struct *mm)
 	mm_free_pgd(mm);
 	destroy_context(mm);
 	mmu_notifier_mm_destroy(mm);
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+	VM_BUG_ON(mm->pmd_huge_pte);
+#endif
 	free_mm(mm);
 }
 EXPORT_SYMBOL_GPL(__mmdrop);

@@ -636,6 +639,10 @@ struct mm_struct *dup_mm(struct task_str
 	mm->token_priority = 0;
 	mm->last_interval = 0;
 
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+	mm->pmd_huge_pte = NULL;
+#endif
+
 	if (!mm_init(mm, tsk))
 		goto fail_nomem;
 

@@ -1422,6 +1422,16 @@ static struct ctl_table vm_table[] = {
 		.extra2		= &one,
 	},
 #endif
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname	= "transparent_hugepage",
+		.data		= &sysctl_transparent_hugepage,
+		.maxlen		= sizeof(sysctl_transparent_hugepage),
+		.mode		= 0644,
+		.proc_handler	= &proc_dointvec,
+	},
+#endif
 
 /*
  * NOTE: do not add new entries to this table unless you have read

@@ -290,3 +290,16 @@ config NOMMU_INITIAL_TRIM_EXCESS
 	  of 1 says that all excess pages should be trimmed.
 
 	  See Documentation/nommu-mmap.txt for more information.
+
+config TRANSPARENT_HUGEPAGE
+	bool "Transparent Hugepage support"
+	depends on X86_64
+	help
+	  Transparent Hugepages allows the kernel to use huge pages and
+	  huge tlb transparently to the applications whenever possible.
+	  This feature can improve computing performance to certain
+	  applications by speeding up page faults during memory
+	  allocation, by reducing the number of tlb misses and by speeding
+	  up the pagetable walking.
+
+	  If unsure, say N.

@@ -45,3 +45,4 @@ obj-$(CONFIG_MEMORY_FAILURE) += memory-f
 obj-$(CONFIG_HWPOISON_INJECT) += hwpoison-inject.o
 obj-$(CONFIG_DEBUG_KMEMLEAK) += kmemleak.o
 obj-$(CONFIG_DEBUG_KMEMLEAK_TEST) += kmemleak-test.o
+obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o

@@ -450,6 +450,7 @@ static inline int check_pmd_range(struct
 	pmd = pmd_offset(pud, addr);
 	do {
 		next = pmd_addr_end(addr, end);
+		split_huge_page_vma(vma, pmd);
 		if (pmd_none_or_clear_bad(pmd))
 			continue;
 		if (check_pte_range(vma, pmd, addr, next, nodes,

@@ -102,6 +102,7 @@ static void remove_migration_pte(struct 
                 return;
 
 	pmd = pmd_offset(pud, addr);
+	VM_BUG_ON(pmd_trans_huge(*pmd));
 	if (!pmd_present(*pmd))
 		return;
 

@@ -95,6 +95,7 @@ static long do_mincore(unsigned long add
 	if (pud_none_or_clear_bad(pud))
 		goto none_mapped;
 	pmd = pmd_offset(pud, addr);
+	split_huge_page_vma(vma, pmd);
 	if (pmd_none_or_clear_bad(pmd))
 		goto none_mapped;
 

@@ -89,6 +89,7 @@ static inline void change_pmd_range(stru
 	pmd = pmd_offset(pud, addr);
 	do {
 		next = pmd_addr_end(addr, end);
+		split_huge_page_mm(mm, addr, pmd);
 		if (pmd_none_or_clear_bad(pmd))
 			continue;
 		change_pte_range(mm, pmd, addr, next, newprot, dirty_accountable);

@@ -42,13 +42,15 @@ static pmd_t *get_old_pmd(struct mm_stru
 		return NULL;
 
 	pmd = pmd_offset(pud, addr);
+	split_huge_page_mm(mm, addr, pmd);
 	if (pmd_none_or_clear_bad(pmd))
 		return NULL;
 
 	return pmd;
 }
 
-static pmd_t *alloc_new_pmd(struct mm_struct *mm, unsigned long addr)
+static pmd_t *alloc_new_pmd(struct mm_struct *mm, struct vm_area_struct *vma,
+			    unsigned long addr)
 {
 	pgd_t *pgd;
 	pud_t *pud;

@@ -63,7 +65,7 @@ static pmd_t *alloc_new_pmd(struct mm_st
 	if (!pmd)
 		return NULL;
 
-	if (!pmd_present(*pmd) && __pte_alloc(mm, pmd, addr))
+	if (!pmd_present(*pmd) && __pte_alloc(mm, vma, pmd, addr))
 		return NULL;
 
 	return pmd;

@@ -148,7 +150,7 @@ unsigned long move_page_tables(struct vm
 		old_pmd = get_old_pmd(vma->vm_mm, old_addr);
 		if (!old_pmd)
 			continue;
-		new_pmd = alloc_new_pmd(vma->vm_mm, new_addr);
+		new_pmd = alloc_new_pmd(vma->vm_mm, vma, new_addr);
 		if (!new_pmd)
 			break;
 		next = (new_addr + PMD_SIZE) & PMD_MASK;

@@ -310,6 +310,7 @@ void prep_compound_page(struct page *pag
 	}
 }
 
+/* update __split_huge_page_refcount if you change this function */
 static int destroy_compound_page(struct page *page, unsigned long order)
 {
 	int i;

@@ -587,6 +588,8 @@ static void __free_pages_ok(struct page 
 
 	kmemcheck_free_shadow(page, order);
 
+	if (PageAnon(page))
+		page->mapping = NULL;
 	for (i = 0 ; i < (1 << order) ; ++i)
 		bad += free_pages_check(page + i);
 	if (bad)

@@ -33,6 +33,7 @@ static int walk_pmd_range(pud_t *pud, un
 	pmd = pmd_offset(pud, addr);
 	do {
 		next = pmd_addr_end(addr, end);
+		split_huge_page_mm(walk->mm, addr, pmd);
 		if (pmd_none_or_clear_bad(pmd)) {
 			if (walk->pte_hole)
 				err = walk->pte_hole(addr, next, walk);

@@ -152,6 +152,10 @@ int add_to_swap(struct page *page)
 	VM_BUG_ON(!PageLocked(page));
 	VM_BUG_ON(!PageUptodate(page));
 
+	if (unlikely(PageCompound(page)))
+		if (unlikely(split_huge_page(page)))
+			return 0;
+
 	entry = get_swap_page();
 	if (!entry.val)
 		return 0;

@@ -896,6 +896,8 @@ static inline int unuse_pmd_range(struct
 	pmd = pmd_offset(pud, addr);
 	do {
 		next = pmd_addr_end(addr, end);
+		if (unlikely(pmd_trans_huge(*pmd)))
+			continue;
 		if (pmd_none_or_clear_bad(pmd))
 			continue;
 		ret = unuse_pte_range(vma, pmd, addr, next, entry, page);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I believe your approach is the right one.

It would be interesting to see how much of a performance gain is seen with real applications, though from hugetlbfs experience we already know that some applications can see significant performance gains from using large pages.

As for the code - this patch is a little too big to comment on all the details individually, but most of the code looks good.

It would be nice if some of the code duplication with hugetlbfs could be removed and the patch could be turned into a series of more reasonably sized patches before a merge.

All rights reversed.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Transparent huge page support is something that would be useful in many areas. The larger memories grow the more pressing the issue will become.

Thats not such a big issue IMHO. Paging is not necessary. Swapping is deadly to many performance based loads. You would abort a job anyways that it going to swap. On the other hand I wish we would have migration support (which may be contingent on swap support).

Wont you be running into issues with page dirtying on that level?

Oww. This sounds like a heuristic page promotion demotion scheme.
http://www.cs.rice.edu/~jnavarro/superpages/
We have discussed this a couple of times and there was a strong feeling that the heuristics are bad. But that may no longer be the case since we already have stuff like KSM in the kernel. Memory management may get very complex in the future.

Those also had fall back logic to 4k. Does this scheme also allow I/O with Hugepages through the VFS layer?

I dont get the point of this. What do you mean by "an operation that cannot fail"? Atomic section?

The consequence of this could be a vast waste of memory if you f.e. touch memory only in 1 megabyte increments.

Separate the patch into a patchset for easy review.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Andrea Arcangeli <[email protected]> writes:

In general the best would be to just merge hugetlbfs into the normal VM. It has been growing for far too long as a separate "second VM" by now. This seems like a reasonable first step, but some comments blow.

Haven't looked at the actual code at this point.

The problem is that this will interact badly with 1GB pages -- once you split them up you'll never get them back, because they can't be allocated at runtime.

Even for 2MB pages it can be a problem.

You'll likely need to fix the page table code.

-Andi

[email protected] -- Speaking for myself only.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Yes, swapping is deadly to performance based loads and it should be avoided as much as possible, but it's not nice when in order to get a boost in guest performance when the host isn't low on memory, you lose the ability to swap when the host is low on memory and all VM are locked in memory like in inferior-design virtual machines that won't ever support paging. When system starts swapping the manager can migrate the VM to other hosts with more memory free to restore the full RAM performance as soon as possible. Overcommit can be very useful at maxing out RAM utilization, just like it happens for regular linux tasks (few people runs with overcommit = 2 for this very reason.. besides overcommit = 2 includes swap in its equation so you can still max out ram by adding more free swap).

Agreed, migration is important on numa systems as much as swapping is important on regular hosts, and this patch allows both in the very same way with a few liner addition (that is a noop and doesn't modify the kernel binary when CONFIG_TRANSPARENT_HUGEPAGE=N). The hugepages in this patch should already relocatable just fine with move_pages (I say "should" because I didn't test move_pages yet ;).

Not sure I follow what the problem should be. At the moment when pmd_trans_huge is true, the dirty bit is meaningless (hugepages at the moment are splitted in place into regular pages before they can be converted to swapcache, only after an hugepage becomes swapcache its dirty bit on the pte becomes meaningful to handle the case of an exclusive swapcache mapped writeable into a single pte and marked clean to be able to swap it out at zerocost if memory pressure returns and to avoid a cow if the page is written to before it is paged out again), but the accessed bit is already handled just fine at the pmd level.

The good thing is, all real complexity is in the patch I posted. That solves the locking and the handling of hugepages in regular vmas. The complexity of the collapse_huge_page daemon that will scan the MADV_HUGEPAGE registered mappings and relocate regular pages into hugepages whenever hugepages become available in the buddy, will be _self_contained_. So it'll be additional complex code yes, but it will be self contained in huge_memory.c and it won't make the VM any more complex than this patch already does.

Plus the daemon will be off by default, just like kksmd has to be off by default at boot...

If you run linux purely as hypervisor it's ok to spend some CPU to make sure all 2M pages that become available are immediately going to replace fragmented pages so that the NPT pagetables becomes 3level instead of 4levels and guest immediately runs faster.

Well maybe I remember your patches wrong, or I might have not followed later developments but I was quite sure to remember when we discussed it, the reason of the -EIO failure was the fs had softblocksize bigger than 4k... and in general fs can't handle blocksize bigger than the PAGE_CACHE_SIZE... In effect the core trouble wasnt' the large pagecache but the fact the fs wanted a blocksize larger than PAGE_SIZE, despite not being able to handle it, if the block was splitted in multiple 4k not contiguous areas.

Hugepage right now can only be transparently mapped and swapped/splitted in anon mappings, not in file mappings (not even the MAP_PRIVATE ones that generate anonymous cache with the COW). This is to keep it simple. Also keep in mind this is motivated by KVM needing to run faster like other hypervisors that support hugepages. We already can handle hugepages to get the hardware boost, but we want our guests to run as fast as possible _always_ (not only if hugepages are reserved at boot to avoid memory failure at runtime, or if the user is not ok to swap, and we don't want to lose the other features of regular mappings including migration, plus we want the regular pages to be collapsed in hugepages when they become available). The whole guest physical memory is mapped by anonymous vmas, so it is natural to start from there... It's also orders of magnitude simpler to start from there than to address pagecache ;). Nothing will prevent to extend this logic to pagecache later...

In short I mean it cannot return -ENOMEM (and an additional bonus is that I managed it not to require scheduling or blocking operations). The idea is that you can plug it anywhere with a one liner and your code becomes hugepage compatible (sure it would run faster if you were to teach to your code to handle pmd_trans_huge natively but we can't do it all at once :).

Sure, this is the feature... But if somebody does mmap(2M) supposedly he's not only going to touch 4k, or I'd blame on the app and not on the kernel that tries to make that 2M mapping so much faster both at page fault time (hugely faster ;) and later during random access too.

Now it may very well be the default should be disabled, but I really doubt with any regular workstation anybody wants it off by default. Surely embedded should turn it off, and stick to madvise for their regions (libhugetlbfs will become a bit simpler by only having to run madvise after mmap) to be sure not to waste any precious kbyte.

I'll try yes...

Thanks!
Andrea

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Problem is hugetlbfs as it stands now can't be merged... it deliberately takes its own paths and it tries to be as far away from the VM as possible. But as you said, as people tries to make hugetlbfs vmas "more similar to the regular vmas" hugetlbfs slowly spreads into the VM code defeating the whole reason why hugetlbfs magic exists (i.e. to be out of the way of the VM as much as possible). Trying to make hugetlbfs more similar to regular vmas makes the VM more complex while still not achieving full feature for hugetlbfs (notably overcommit and paging).

1GB pages can't be handled by this code, and clearly it's not practical to hope 1G pages to materialize in the buddy (even if we were to increase the buddy so much slowing it down regular page allocation). Let's forget 1G pages here... we're only focused on sizes that can be allocated dynamically. Main problem are the 64k pages or such that don't fit into a pmd...

In terms of fragmentation split_huge_page itself won't create it.. unless it swaps (but then CPU performance is lost on the mapping anyway). We need to teach mprotect/mremap not to call split_huge_page true, but not to avoid fragmentation. btw, thinking at fragmentation generated by mnmap(last4k) I also think I found a minor bug in munmap if a partial part of the 2M page is unmapped (currently I'm afraid I'm dropping the whole 2M in that case ;), but it's trivial to fix... clearly not many apps are truncating 4k off a 2M mapping.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

It's also needed if something like glibc were to take advantage of it in a generic manner.

thanks,
-chris

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

On Tue, Oct 27, 2009 at 08:30:07PM +0100, Andrea Arcangeli wrote:

Hi Andrea,

I think longer term the standard VM just needs to understand huge pages properly. Originally when huge pages were only considered a "Oracle hack" the separation made sense, but now with more and more use that is really not true anymore.

Also hugetlbfs is gaining more and more functionality all the time.

Maintaining two VMs in parallel forever seems like the wrong thing to do.

Also the fragmentation avoidance heuristics got a lot better in the last years, so it's much more practical than it used to be (at least for 2MB)

That seems short sightened. You do this because 2MB pages give you x% performance advantage, but then it's likely that 1GB pages will give another y% improvement and why should people stop at the smaller improvement?

Ignoring the gigantic pages now would just mean that this would need to be revised later again or that users still need to use hacks like libhugetlbfs.

Given 1GB pages for a time are harder to use on the system administrator level, but at least for applications the interfaces should be similar at least.

What 64k pages? You're talking about soft pages or non x86?

The problem is that the performance will be lost forever. So if you ever do something that only does a little temporary swapping (like a backup run) you would be ready for a reboot. Not good.

I think they just have to be fixed properly.

My suspicion is btw that there's some more code sharing possible in all that VMA handling code of ther different system calls (I remember thinking that when I wrote mbind() :-). Then perhaps variable page support would be easier anyways because less code needs to be changed.

-Andi

[email protected] -- Speaking for myself only.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi Andi,

This is exactly the problem... and these days there's not just Oracle:
KVM, glibc needs hugepages too all the time, so it's not surprising even hugetlbfs is gaining more generic functionality but it's still an awkward model with libhugetlbfs and it still has limitations that prevents generic use.

Agreed.

"more practical at least for 2MB" is why I suggested to ignore sizes like 1G. 2M is already on the largy side, and in this work we should focus on those sizes that can realistically be found on the buddy even by pure luck on a busy system sometime, and that can be reasonably found on the buddy if defrag heuristics shrink the cache in physical order. 1G will never be found by pure luck, at best it could be defragged but with an huge expensive defrag relocation amount of work which might not be justified.

For the reason mentioned above, sizes like 1G will likely remain available only through boot-reservation and splitting an huge page to be swapped would require a very expensive split_huge_page function. Instead of a loop for (i=0; i<512; i++) inside a not preemptive section with all pmd frozen, you will have a loop of 262144... And swapping 1G page natively without splitting it, is even less feasible.

They will still need if they want the extra y%, because 1G pages simply can't be generated by the buddy allocator. I doubt we should increase the MAX_ORDER from 11 to 18, it would slowdown the whole buddy without actually giving us 1G pages in a timely manner (the relocation work over 1G would be very expensive so not suitable for transparent behavior).

I see your point here in wanting to use the generic interface, we could have the page fault in the madvise vmas that fits a 1G naturally aligned region, search into a reserved region first, and if they don't find the 1G page reserved they could search the buddy for 2M pages. But still the problem is there's no way to swap that 1G beast if we go low on memory. It's not transparent behavior, but we could share the same madvise interface, true! I doubt we should map 1G pages in tasks outside of madvised vmas, because of that ram being special and reserved. In some ways hugetlbfs providing for permissions in order to take advantage of the reserved regions is better than unprivileged madvise. Not to tell the problem of clearing and copying of a 1G page during page fault...

For the transparent 2M pages instead if an unprivileged user end up using them the whole system gains because the more people uses 2M pages the less fragmentation there is in the system.

But if we'll get to a point where 1G page is feasible (or we want to obsolete hugetlbfs, which I doubt it will happen until we move transparent hugepages to tmpfs too), we can always add a pud_trans_huge later... Frankly the 1G pages don't worry me at all for the long term. Especially if we'll just manage them with a generic madvise(MADV_HUGEPAGE). I don't plan to nuke hugetlbfs in the very short term. If we get to a point where hugetlbfs has no reason to exist anymore we've just to add pud_trans_huge before nuking it and have the do_huge_anonymous_page search into the reserved 1G regions if the VM_HUGEPAGE is set.

I wasn't talking about soft pages. The whole patch here is tuned for hugetlb. I tried to do prefault and to allocate hugepages and map them partially with ptes (kind of softpages of size a power of 2 between 8k and 1M both included) to avoid zeroing and copying the whole 2M during page faults. But it's not worth it. Whenever we deal with hugepages a huge tlb has always to be involved for this to be worth it. Otherwise it adds even more complexity and there is not enough gain (with the exception of speeding up the initial page fault which is not so important). I think those designs that preallocate hugepages and maps them partially with ptes are inefficient, overcomplex and bloated.

My worry are the archs like powerpc where a hugepage doesn't fit in a pmd_trans_huge. I think x86 will fit the pmd/pud_trans_huge approach in my patch even of 1G pages in the long run, so there is no actual long term limitation with regard to x86. The fact is that the generic pagetable code is tuned for x86 so no problem there.

What I am unsure about and worries me more are those archs that don't use a pmd to map hugepages and to create hugetlb. I am unsure if those archs will be able to take advantage of my patch with minor changes to it given it is wired to pmd_trans_huge availability.

Well until the background daemon calls collapse_huge_page. Also before splitting the page the pmd young bit is checked and it gets huge more priority than the young bit of the pte because the pmd young bit has 512 higher probability of being set than the pte young bit.

Also note, the swapping right now generates fragmentation but later we can add swap entries at the pmd level and we can stop calling split_huge_page even in the swap path, to avoid swap to introduce fragmentation. But we can't do everything at once...

Sure they have in the mid term but just to speedup those syscalls and to avoid them to break the hugetlb speedup, fragmentation is not an issue there.

Somebody worked in that direction with pagewalk.c, but one thing is to do a readonly pagetable walk, another thing is to mangle vmas and pmd/pte etc... So it looks hard to share there, they split vmas as a start and then mangles ptes (and in future pmds) around.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Why glibc?

Yes, there are quite some workloads who benefit.

Even without automatic allocation and the need to prereseve having the same application interface for 1GB pages is still useful. Otherwise people who want to use the 1GB pages have to do the special hacks again.

What I was thinking of was to have a relatively easy to use flag that allows an application to use prereserved GB pages transparently. e.g. could be done with a special command

hugepagehint 1GB app

Yes I realize that this is possible to some extend with libhugetlbfs LD_PRELOAD, but integrating it in the kernel is much saner.

So even if there are some restrictions it would be good to not ignore the 1GB pages completely.

Agreed, prereservation is still the way to go for 1GB.

(although in theory a special allocation could get them without relying on zone alignment or buddy lists by being not O(1))

It would need an administrator hint agreed, but if it's just a single hint per program it would be still "mostly transparent"

Agreed on not doing it unconditionally, ut the advice could be per process or per cgroup.

Even on 2MB pages this problem exists to some point: if you explicitely preallocate 2MB pages to make sure some application can use them with hugetlbfs you don't want random applications to still the "guaranteed" huge pages.

So some policy here would be likely needed anyways and the same could be used for the 1GB pages.

I was just confused by the 64k number.

I see. Some archs (like IA64 or POWER?) require special VA address
ranges for huge pages, for those doing it fully transparent without
a mmap time flag is likely hard.

I'm still uneasy about this, it's a very clear "glass jaw" that might well cause serious problems in practice. Anything that requires regular reboots is bad.

-Andi

[email protected] -- Speaking for myself only.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

PowerPC does not require specific virtual addresses for huge pages, but does require that a consistent page size be used for each slice of the virtual address space. Slices are 256M in size from 0 to 4G and 1TB in size above 1TB while huge pages are 64k, 16M, or 16G. Unless the PPC guys can work some more magic with their mmu, split_huge_page() in its current form just plain won't work on PowerPC. That doesn't even take into account the (already discussed) page table layout differences between x86 and ppc: http://linux-mm.org/PageTableStructure .

Thanks,
Adam

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

it simply won't be able to use Andrea's transparent code until someone fixes the MMU. Doesn't seem a disaster

-Andi

[email protected] -- Speaking for myself only.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Well at least we found a good reason for hugetlbfs that forces hugepages on the whole vma to still exist... Even without split_huge_page (assuming all code would be hugepage aware) the requirement is that if a hugepage allocation fails we _gracefully_ fallback to 4k allocations and we mix those in the same vma with other hugepages (so the daemon can collapse the 4k pages into a hugepage later when they become available).

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

That's what I meant, I said glibc to mean not just KVM (like Chris pointed out before ;)

They will have to do the special hacks for reservation... No many other hacks after that if they accept if they reserve it becomes not swappable. Then it depends how you want to give permissions to use the reserved areas. It's all a reservation logic that you need in order to use 1G pages with this.

I think we should ignore them in the first round of patches, knowing this model can fit them later if we just add a reservation logic and all pud_trans_huge. I don't think we need to provide this immediately as it'd grow the size of the patch, but we can do it soon after. I'm frightened by growing the patch even more, I'd rather try to get optimal on 2M pages and only later worry about 1G pages. I think it's higher priority to remove a couple of split_huge_page than to support transparent gigapages given they won't be really transparent anyway.

To support gigapages, would require to decide a reservation API now. After that, the kernel will map a 1G page if it is available and we add pud_trans_huge all over the place. There are more urgent things like the collapse daemon, removing a couple of split_huge_page, before I can worry about reservation APIs and to bloat further with pud_trans_huge all over the place.

It gets more and more complicated and this "hint" is all about reservation, not something we want to deal with with 2M pages.

This is what the sysctl is about. You can turn it off the transparency, and then the kernel will keep mapping hugepages only inside madvise(MADV_HUGEPAGE). There is no need of reserving anything here.

1GB pages can't use the same logic but again I don't think we will be doing any additional work, if we address 2M pages now transparent, and we lave the reservation required for 1G pages for later.

What I mean with ignore, is not to add a requirement for merging that 1G pages are also supported or we've to add even more logics that are absolutely useless for 2M pages.

Here nothing requires reboot. If you get 2M pages good, otherwise stick to 4k pages transparently, userland can't know. When some task quits and 2M page happens we'll just collapse the 4k pages into the newly generated 2M pages with a background daemon. Over time we can add more logics to try to minimize fragmentation (obviously slab needs a front-allocator that tries 2M page allocation first always, there are many other things we have to do in the defrag front, before we can worry about the effect of swap that calls split_huge_page). The other syscalls that calls split_huge_page as said won't fragment anything physically (with the exception of munmap and madvise_dontneed if used to truncate an hugepage).

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

It's still a big step between just needing reservation and also hacking the application to use new interfaces.

The design at least should not preclude them, even if the code doesn't fully initially. That is why I objected earlier -- the design doesn't seem to support them.

A global sysctl seems like a quite clumpsy way to do that. I hope it would be possible to do better even with relatively simple code.

e.g. a per process flag + prctl wouldn't seem to be particularly complicated.

If there's a per process "use pre-reservation" policy that logic could well be shared for 2MB and 1GB.

I don't think there's much (anything?) in 1GB support that's absolutely useless for 2M. e.g. a flexible reservation policy is certainly not.

When the performance improvement is visible enough people will feel the need to reboot and the practical effect will be that Linux requires reboots for full performance.

We already have this to some extent with the kernel direct mapping breakup over time, but this would make it much worse.

-Andi

[email protected] -- Speaking for myself only.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

The word "transparent" is all about "no need of hacking the application" because "there is no new interface".

I want to keep it as transparent as possible and to defer adding user visible interfaces (with the exception of MADV_HUGEPAGE equivalent to MADV_MERGEABLE for the scan daemon) initially. Even MADV_HUGEPAGE might not be necessary, even the disable/enable global flag may not be necessary but that is the absolute minimum tuning that seems useful and so there's not much risk to obsolete it.

I think it supports them once you solve the reservation, your hinting and you add pud_trans_huge.

btw, the sysctl has to be moved to sysfs. The same sysfs directory will also control the background collapse_huge_page daemon.

You realize we can add those _interfaces_ later _after_ adding pud_trans_huge. I don't even want to add pud_trans_huge right now. Adding them now would force us to be sure to get the interface right. I don't even want to think about it.

Let's defer any not strictly necessary visible user interface for _later_. Anything 1G pages need can be deferred later.

We don't want having to reserve. Yes we could reserve but we don't want to. We want to tell the kernel which regions have to be scanned to recreate 2M pages with the madvise, but that's about it.

Nothing prevents us to add an interface to reserve later, which obviously will be mandatory for 1G pages to ever be allocated. It's not something we need to solve now I think.

I don't see KVM ever using this reservation hint, glibc neither. So yes, you may have a corner case, but for the actual users of transparent hugepages it seems entirely useless to me for the long run. I may be wrong but because this is a new interface, and transparent hugepages is all about _not_ having to modify the app at all, we should better focus on ensuring the MADV_HUGEPAGE fits 1G collapse_huge_page collapsing later (yeah, assuming 1G pages becomes available and that you can hang all apps using that data for as long as copy_page(1g)).

The whole point of ignoring 1G pages is that, we know adding pud_trans_huge later is no problem, and that it'll require userland changes that we want to defer as it's an orthogonal problem, even if it might remotely help some corner case using transparent hugepages.

So you think the collapse_huge_page daemon will not be enough? How can't it be enough? If it's not enough it means the defrag logic isn't smart enough simply. So there's no way anything we do in this patch can make a difference to avoid or not avoid reboot. In short your worry of "need of rebooting" has nothing to do with the code we're discussing but with the ability of the VM to generate hugepages. The collapse_huge_page daemon will do the necessary things if those are made available without need of reboot. yes defrag is another thing to solve but it can be addressed separately and in parallel with this.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I think you need some user visible interfaces to cleanly handle existing reservations on a process base at least, otherwise you'll completely break their semantics.

sysctls that change existing semantics greatly are usually a bad idea because what should the user do if they have existing applications that rely on old semantics, but still want the new functionality?

If you rely on splitting then it all won't work
for 1GB anyways and might need to be redone on the design level. Code that's not complete is ok, but code that is known to need a redesign from the start is not that great.

Also completely ignoring sane reservation semantics in advance also doesn't seem to be a particularly good way. Some way to control this fine grained should be there at least.

It would be set by the administrator.

Can always schedule and check for signals during the copy.

I don't know how well it will hold up in practice. Only data can tell.

The problem I have is that the current "split on demand" approach can fragment even prereserved pages.

-Andi

[email protected] -- Speaking for myself only.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

But we already handle explicit hugepages (with page pools and strict reservations) via hugetlbfs and libhugetlbfs. It seems you're just making an argument for keeping these around (which I certainly agree with).

If you want to reserve some huge pages for a specific, corner-case application, allocate huge pages the way we do today and use libhugetlbfs. Meanwhile, the rest of the system can benefit from this new interface.

Thanks,
Adam

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

That would require not supporting reservations through the transparent mechanism. That wouldn't be very nice semantics, because you end up with "glass jaw" performance always in the transparent case.

-Andi

[email protected] -- Speaking for myself only.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

What is not clear about the word "transparent". This whole effort is about not having to add visible interfaces and userland won't be able to notice (except it runs faster). We don't want new interfaces. We need an madvise to give an hint to the daemon of which regions are critical to have hugepages. It's not so easy for the kernel to find it by itself.

The reason the sysfs enable/disable of the "transparency" is because embedded may want to disable the transparency. Not every hardware out there will have enough memory or enough l2 CPU cache and useful workloads to take advantage of this, so those might (and it's not guaranteed) save a bit of memory by disabling the feature.

In short the fewer new interfaces we add the better, and the only one I think is generic enough and needed enough, is madvise(MADV_HUGEPAGE) (which will tell the kernel to use hugepages even if transparent hugepage is disabled in sysfs and it'll tell the collapse_huge_page daemon the virtual regions to relocate in hugepages). For the time being any additional interface would defeat the objective of not having to modify apps.

memory reservation is the first thing we want to remove as requirement to use hugepages, which is the first reason why 1G won't work anyway as we don't want reservation in this, this is all about not having to reserve anything at boot and not having to modify binaries at all.

1G pages can work but it would need to split 512 pieces and we can do that after my patch will swap natively 2M pages and we won't call split_huge_page anymore. Then split_huge_page can be moved up one level to the pud. Something like that.

Worrying about this right now is too early and not worth it so we better ignore 1G in the transparency area.

It won't need any redesign... besides this is only relevant if you can manage to find 1G page without reservation, otherwise you're better off with with hugetlbfs if you have to do magics visible to userland that _entirely_ depends on reservation for them to have a slight chance to allocate a 1G page.

Eliminating reservation is the first objective of the patch.

same is true for split_huge_page... if copy_page can work on a 1G page then we could even split it at the pte level, but frankly I think it would be a better fit to split the pud at the pmd level only without having to go down to the pte.

1) we eliminate reservation (no preserved pages here) 2) split_huge_page on demand can't generate any fragmentation whatsoever (only swap code can then fragment the hugepage by swapping only part of it but you know the swap code can't swap 2M at once, it's not split_huge_page fault if page is fragmented as it is swapped it, no fragmentation happens when mprotect and mremap calls split_huge_page, however we want to optimize those for performance reasons, and definitely not for fragmentation purposes at all)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Note checking for signals is not enough, and even if it was enough we would need to rollback from the middle and it's an huge complexity... If it was so easy copy_huge_page in hugetlb.c would be doing it too (I mean checking for signals, obviously it doesn't... :). It already results in livelocks, I just recently had bugreports about it and not easy to fix (they had not to cow to avoid the livelock). And they were about 256M pages!!!! Not 1G pages... ;). 256M already livelocks the application if one has to cow 256M and access 512M (not 2G!!) in a fault. Last but not the least when cpus will be fast enough to execute copy_page(1G) as fast as it does now copy_page(2M) we'll be running linux with PAGE_SIZE = 2M in the first place...

Overall I liked to evaluate the feasibility of pud_trans_huge, and discuss about it because one can never know somebody may have petabyte of memory and slow enough cpu to require 4k page, but the notion of providing transparent gigapages at the time being is absurd no matter what design or implementation, hugetlbfs is as good as it can be for it I think.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I've read the patch and have read through this discussion and you are missing the big point that it's best to do such things gradually - one step at a time.

Just like we went from 2 level pagetables to 3 level pagetables, then to 4 level pagetables - and we might go to 5 level pagetables in the future. We didnt go from 2 level pagetables to 5 level page tables in one go, despite predictions clearly pointing out the exponentially increasing need for RAM.

So your obsession with 1GB pages is misguided. If indeed transparent largepages give us real benefits we can extend it to do transparent gbpages as well - should we ever want to. There's nothing 'shortsighted' about being gradual - the change is already ambitious enough as-is, and brings very clear benefits to a difficult, decade-old problem no other person was able to address.

In fact introducing transparent 2MBpages makes 1GB pages support _easier_ to merge: as at that point we'll already have a (finally..) successful hugetlb facility happility used by an increasing range of applications.

Hugetlbfs's big problem was always that it wasnt transparent and hence wasnt gradual for applications. It was an opt-in and constituted an interface/ABI change - that is always a big barrier to app adoption.

So i give Andrea's patch a very big thumbs up - i hope it gets reviewed in fine detail and added to -mm ASAP. Our lack of decent, automatic hugepage support is sticking out like a sore thumb and is hurting us in high-performance setups. If largepage support within Linux has a chance, this might be the way to do it.

A small comment regarding the patch itself: i think it could be simplified further by eliminating CONFIG_TRANSPARENT_HUGEPAGE and by making it a natural feature of hugepage support. If the code is correct i cannot see any scenario under which i wouldnt want a hugepage enabled kernel i'm booting to not have transparent hugepage support as well.

Thanks,

Ingo

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hello Ingo, Andi, everyone,

I totally agree with your assessment.

Agreed.

Thanks a lot for your review!

The two reasons why I added a config option are:

1) because it was easy enough, gcc is smart enough to eliminate the external calls so I didn't need to add ifdefs with the exception of returning 0 from pmd_trans_huge and pmd_trans_frozen. I only had to make the exports of huge_memory.c visible unconditionally so it doesn't warn, after that I don't need to build and link huge_memory.o.

2) to avoid breaking build of archs not implementing pmd_trans_huge and that may never be able to take advantage of it

But we could move CONFIG_TRANSPARENT_HUGEPAGE to an arch define forced to Y on x86-64 and N on power.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Here the incremental fix if somebody is running the patch (this also frees the page after tlb flush, during development I had to call put_page to debug something and I forgot to replace it back to tlb_remove_page ;). In the meantime I'll try to split the patch into more self contained pieces for easier review (including the kvm patch to build 3level NPT/EPT).

Then I will continue working on the daemon that will collapse_huge_page in the madvise(MADV_HUGEPAGE) regions.

This printed 0 before, now it prints 0xff as with transparent hugepage disabled.

#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>

#define SIZE (2*1024*1024)

int main()

{
	char *p = malloc(SIZE*2-1);
	p = (char *)((unsigned long)(p + SIZE - 1) & ~(SIZE-1));
	*p = 0xff;
	munmap(p+SIZE-4096, 4096);
	printf("%x\n", *(unsigned char *)p);

	return 0;
}

---

@@ -850,6 +850,9 @@ extern struct page *follow_trans_huge_pm
 					  unsigned long addr,
 					  pmd_t *pmd,
 					  unsigned int flags);
+extern int zap_pmd_trans_huge(struct mmu_gather *tlb,
+			      struct vm_area_struct *vma,
+			      pmd_t *pmd);
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
 extern int copy_pte_range(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 			  pmd_t *dst_pmd, pmd_t *src_pmd,

@@ -292,8 +292,9 @@ config NOMMU_INITIAL_TRIM_EXCESS
 	  See Documentation/nommu-mmap.txt for more information.
 
 config TRANSPARENT_HUGEPAGE
-	bool "Transparent Hugepage support"
+	bool "Transparent Hugepage support" if EMBEDDED
 	depends on X86_64
+	default y
 	help
 	  Transparent Hugepages allows the kernel to use huge pages and
 	  huge tlb transparently to the applications whenever possible.

@@ -302,4 +303,4 @@ config TRANSPARENT_HUGEPAGE
 	  allocation, by reducing the number of tlb misses and by speeding
 	  up the pagetable walking.
 
-	  If unsure, say N.
+	  If memory constrained on embedded, you may want to say N.

@@ -12,6 +12,7 @@
 #include <linux/mmu_notifier.h>
 #include <linux/rmap.h>
 #include <linux/swap.h>
+#include <asm/tlb.h>
 #include <asm/pgalloc.h>
 #include "internal.h"
 

@@ -374,3 +375,37 @@ struct page *follow_trans_huge_pmd(struc
 out:
 	return page;
 }
+
+int zap_pmd_trans_huge(struct mmu_gather *tlb, struct vm_area_struct *vma,
+		       pmd_t *pmd)
+{
+	int ret = 0;
+
+	spin_lock(&tlb->mm->page_table_lock);
+	if (likely(pmd_trans_huge(*pmd))) {
+		if (unlikely(pmd_trans_frozen(*pmd))) {
+			spin_unlock(&tlb->mm->page_table_lock);
+			wait_split_huge_page(vma->anon_vma,
+					     pmd);
+		} else {
+			struct page *page;
+			pgtable_t pgtable;
+			pgtable = get_pmd_huge_pte(tlb->mm);
+			page = pfn_to_page(pmd_pfn(*pmd));
+			VM_BUG_ON(!PageCompound(page));
+			pmd_clear(pmd);
+			spin_unlock(&tlb->mm->page_table_lock);
+			page_remove_rmap(page);
+			VM_BUG_ON(page_mapcount(page) < 0);
+			add_mm_counter(tlb->mm, anon_rss,
+				       -1<<(HPAGE_SHIFT-
+					    PAGE_SHIFT));
+			tlb_remove_page(tlb, page);
+			pte_free(tlb->mm, pgtable);
+			ret = 1;
+		}
+	} else
+		spin_unlock(&tlb->mm->page_table_lock);
+
+	return ret;
+}

@@ -913,32 +913,12 @@ static inline unsigned long zap_pmd_rang
 	do {
 		next = pmd_addr_end(addr, end);
 		if (pmd_trans_huge(*pmd)) {
-			spin_lock(&tlb->mm->page_table_lock);
-			if (likely(pmd_trans_huge(*pmd))) {
-				if (unlikely(pmd_trans_frozen(*pmd))) {
-					spin_unlock(&tlb->mm->page_table_lock);
-					wait_split_huge_page(vma->anon_vma,
-							     pmd);
-				} else {
-					struct page *page;
-					pgtable_t pgtable;
-					pgtable = get_pmd_huge_pte(tlb->mm);
-					page = pfn_to_page(pmd_pfn(*pmd));
-					VM_BUG_ON(!PageCompound(page));
-					pmd_clear(pmd);
-					spin_unlock(&tlb->mm->page_table_lock);
-					page_remove_rmap(page);
-					VM_BUG_ON(page_mapcount(page) < 0);
-					add_mm_counter(tlb->mm, anon_rss,
-						       -1<<(HPAGE_SHIFT-
-							    PAGE_SHIFT));
-					put_page(page);
-					pte_free(tlb->mm, pgtable);
-					(*zap_work)--;
-					continue;
-				}
-			} else
-				spin_unlock(&tlb->mm->page_table_lock);
+			if (next-addr != HPAGE_SIZE)
+				split_huge_page_vma(vma, pmd);
+			else if (zap_pmd_trans_huge(tlb, vma, pmd)) {
+				(*zap_work)--;
+				continue;
+			}
 			/* fall through */
 		}
 		if (pmd_none_or_clear_bad(pmd)) {

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

How would glibc do that?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Another NUMA issue is how MPOL_INTERLEAVE would work with this. MPOL_INTERLEAVE would cause the spreading of a sequence of pages over a series of nodes. If you coalesce to one huge page then that cannot be done anymore.

May not be a problem as long as you dont allow fs operations with these pages.

The patches modified the page cache logic to determine the page size from the page structs.

We need to know some more detail about the conversion.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

One answer, at least in the beginning, would be to just ignore this detail. Try to make 16MB pages wherever possible, probably even as 16MB pages in the Linux pagetables. But, we can't promote the MMU to use them until get get a 256MB or 1TB chunk. It will definitely mean some ppc-specific bits when we're changing the segment mapping size, but it's not impossible.

That's not going to do any good for the desktop-type users. But, it should be just fine for the HPC or JVM folks. It restricts the users pretty severely, but it gives us *something*.

There will be some benefit to using a 16MB Linux page and pte even if we can't back it with 16MB MMU pages, anyway. Remember, a big chunk of the benefit of using 64k pages can be seen even on systems with no 64k hardware pages.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi Andrea,

I will find some time soon to test out your patch on a (relatively) huge machine and let you know the results.

The memory size on this machine:

	480,700,399,616 bytes of system memory tested OK

This translates to ~240k available 2Mb pages.

Thanks,
Mike

Ah, please keep CONFIG_TRANSPARENT_HUGEPAGE for a while.
Now, memcg don't handle hugetlbfs because it's special and cannot be freed by the kernel, only users can free it. But this new transparent-hugepage seems to be designed as that the kernel can free it for memory reclaiming. So, I'd like to handle this in memcg transparently.

But it seems I need several changes to support this new rule. I'm glad if this new huge page depends on !CONFIG_CGROUP_MEM_RES_CTRL for a while.

Thanks,
-Kame

This isn't possible on all architectures. Some archs have "segment" constraints which mean only one page size per such "segment". Server ppc's for example (segment size being either 256M or 1T depending on the CPU).

Precisely because the approach cannot work on all architectures ?

Cheers,
Ben.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Note: this is server powerpc's. Embedded ones are more flexible but on server we have this limitation and not much we can do about it.

Note also that the "slice" sizes are a SW thing. HW segments are either 256M or 1T (the later being supported only on some processors), and linux maintains that concept of "slices" in order to simplify the tracking of said segments and to use 1T when available.

Ben.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hello Christoph,

So the first important thing is to start the mapping at 2M aligned virtual address. Second important thing is to always do sbrk increments in 2M chunks and mmap extensions in 2M chunks with a mmap on the next 2M (mremap right now calls split_huge_page but later we will teach mremap and mprotect not to call split_huge_page and to handle the pmd_trans_huge natively so they can run a bit faster).

With those two precautions all page faults will be guaranteed to map 2M pages if they're available (and then fragmentation will decrease too as 2M pages will be retained in the mappings).

Even after split_huge_page, the pte will point to the same 2M page as before. And when task is killed all 2M pages will be recombined in the buddy. Page coloring will also be still guaranteed (up to the 512th color of course) even after split_huge_page run.

But if split_huge_page is called because munmap is unmapping just a 4k piece of a 2M page, then split_huge_page will be called to free just that 4k piece so fragmentation will be created. So the last precaution that glibc should use is to munmap (or madvise_dontneed) in 2M chunks naturally aligned to make sure not to create unnecessary fragmentation.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Yeah the accounting (not just memcg) should be checked.. I didn't pay too much attention to stats at this point.

But we want to fix it fast instead of making the two options mutually exclusive.. Where are the pages de-accounted when they are freed? Accounting seems to require just two one liners
calling mem_cgroup_newpage_charge.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hmm 256M is already too large for a transparent allocation. It will require reservation and hugetlbfs to me actually seems a perfect fit for this hardware limitation. The software limits of hugetlbfs matches the hardware limit perfectly and it already provides all necessary permission and reservation features needed to deal with extremely huge page sizes that probabilistically would never be found in the buddy (even if we were to extend it to make it not impossible). That are hugely expensive to defrag dynamically even if we could [and we can't hope to defrag many of those because of slab]. Just in case it's not obvious the probability we can defrag degrades exponentially with the increase of the hugepagesize (which also means 256M is already orders of magnitude more realistic to function than than 1G). Clearly if we increase slab to allocate with a front allocator in 256M chunk then our probability increases substantially, but to make something realistic there's at minimum an order of 10000 times between hugepagesize and total ram size. I.e. if 2M page makes some probabilistic sense with slab front-allocating 2M pages on a 64G system, for 256M pages to make an equivalent sense, system would require minimum 8Terabyte of ram. If pages were 1G sized system would require 32 Terabyte of ram (and the bigger overhead and trouble we would have considering some allocation would still happen in 4k ptes and the fixed overhead of relocating those 4k ranges would be much bigger if the hugepage size is a lot bigger than 2M and the regular page size is still 4k).

I thought the main reason for those patches was to allow a fs blocksize bigger than PAGE_SIZE, a PAGE_CACHE_SIZE of 64k would allow for a 64k fs blocksize without much fs changes. But yes, if the mmu can't fallback, then software can't fallback either and so it impedes the transparent design on those architectures... To me hugetlbfs looks as best as you can get on those mmu.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

POWER is pretty unusual in its mmu. These 256MB (or 1TB) segments are the granularity with which we must make the choice about page size, but they *aren't* the page size itself.

We can fill that 256MB segment with any 16MB pages from all over the physical address space, but we just can't *mix* 4k and 16MB mappings in the same 256MB virtual area.

16*16MB pages are going to be hard to get, but they are much much easier to get than 1 256MB page. But, remember that most ppc64 systems have a 64k page, so the 16MB page is actually only an order-8 allocation. x86-64's huge pages are order-9. So, it sucks, but allocating the pages themselves isn't that big of an issue. It's getting a big enough virtual bunch of them together without any small pages in the segment.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

It's de-accounted at page_remove_rmap() in typical case of Anon. But swap-cache/bacthed-uncarhge related part is complicated, maybe. ...because of me ;(

Okay, I don't request !CONFIG_CGROUP_MEM_RES_CTRL, I'm glad if you CC me.

Yes, maybe so.

Thanks,
-Kame

Right.

Yes. Note that powerpc -embedded- processors don't have that limitation though (in large part because they are mostly SW loaded TLBs and they support a wider collection of page sizes). So it would be possible to implement your transparent scheme on those.

True. 256M might even be worth toying with as an experiment on huge machines with TBs of memory in fact :-)

Well... such systems aren't that far around the corner, so as I said, it might still make sense to toy a bit with it. That would definitely -not- include my G5 workstation though :-)

Right.

I need to look whether your patch would work "better" for us with our embedded processors though.

Cheers,
Ben.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>