Fix do_mlock so page alignment is to hugepage boundries when needed
Subsystems (2)
PATCH 1/1 Fix do_mlock so page alignment is to hugepage boundries when needed
sys_m{,un}lock and do_mlock all align memory references and the length of the mlock given by userspace to page boundaries. If the page being mlocked is a hugepage instead of a normal page the start and finish of the mlock will still only be aligned to normal page boundaries. Ultimately upon the process exiting we will eventually call unmap_vmas which will call unmap_hugepage_range for all of the ranges. unmap_hugepage_range checks to make sure the beginning and the end of the range are actually hugepage aligned and if not will BUG(). Since we only aligned to a normal page boundary the end of the first range and the beginning of the second will likely (unless userspace passed of values already hugepage aligned) not be hugepage aligned and thus we bomb.
To recreate (on any arch with hugepages) just create a hugepage and allow shared memory segments to be at least as large as a hugepage. Attach to that hugepage and then mlock on a non hugepage aligned boundary when you call exit() the kernel will panic with a BUG() in unmap_hugepage_range because the end of the range being unmapped is not aligned properly. Part of a quick reproducer program for i686 where PAGE_SIZE=1024*4 and HPAGE_SIZE=4*1024*1024 is below.
shmflg = SHM_HUGETLB | IPC_CREAT | SHM_R | SHM_W;
shmid = shmget (IPC_PRIVATE, 4*1024*1024, shmflg);
a = shmat(shmid, NULL, shmflg);
ret = mlock(a, 1024*4); // 1024*4 is PAGE_ALIGN'ed but this is a hugepage exit(0)
This will panic because the end of the area to be unmapped will be 4k (the regular pace boundary) not 4M (the hugepage page boundary on this arch)
To fix this I created 2 new macros in mlock.c and a new macro in all of the page.h files that defined HPAGE_SIZE. I also had to move the resource limit checks from sys_mlock to do_mlock. The reason for this is that we can't tell if it is a hugepage unless we have the vma. And so we don't know how long the mlock length will be unless we know how to round the length to the correct page length. Thus resource checking can't be done until after we have the vma in question.
The two macros in mlock.c will, only on systems compiled with CONFIG_HUGETLB_PAGE, check if the page being locked or unlocked is a hugepage and if so ALIGN it to hugepage boundaries rather than normal page boundaries. If the vma is not for a hugepage it will use the same methods as before for alignment. The macro addition to page.h copies the PAGE_ALIGN macro but uses HPAGE_SIZE and HPAGE_MASK. I created this HPAGE_ALIGN macro on the i386, x86_64, ia64, sh, sh64, sparc64, parisc and powerpc archs. Since they appear to be the all arches that implement HPAGE_SIZE. Two points of oddity, x86_64 seems to have HPAGE_SIZE defined irrespective of CONFIG_HUGETLB_PAGE so my new HPAGE_ALIGN macro is always defined as well. It will still only get used if CONFIG_HUGETLB_PAGE is set.
The other point of oddity that I don't know if it is right is the definition on powerpc. include/asm-powerpc/page_64.h does not quite follow the same layout as the other arches so I think it's in the right place but don't have a good place to test. So if a person with power pc is willing to test for me please just let me know if it is either fixed or if it doesn't compile correctly. I think it's right, but I'm not 100%.
include/asm-i386/page.h | 2 + include/asm-ia64/page.h | 2 + include/asm-parisc/page.h | 2 + include/asm-powerpc/page_64.h | 2 + include/asm-sh/page.h | 2 + include/asm-sh64/page.h | 2 + include/asm-sparc64/page.h | 2 + include/asm-x86_64/page.h | 2 + mm/mlock.c | 54 ++++++++++++++++++++++++------------------ 9 files changed, 48 insertions(+), 22 deletions(-)
include/asm-sparc64/page.h renamed from include/asm-sparc64/page.h.paris +2 -0
@@ -104,6 +104,8 @@ typedef unsigned long pgprot_t;#define HUGETLB_PAGE_ORDER (HPAGE_SHIFT - PAGE_SHIFT)#define ARCH_HAS_SETCLEAR_HUGE_PTE #define ARCH_HAS_HUGETLB_PREFAULT_HOOK +/* to align the pointer to the (next) page boundary when dealing with hugepages*/ +#define HPAGE_ALIGN (addr) (((addr)+HPAGE_SIZE-1)&HPAGE_MASK)#endif #define TASK_UNMAPPED_BASE (test_thread_flag(TIF_32BIT) ? \
include/asm-i386/page.h renamed from include/asm-i386/page.h.paris +2 -0
@@ -68,6 +68,8 @@ typedef struct { unsigned long pgprot; }#define HPAGE_MASK (~(HPAGE_SIZE - 1))#define HUGETLB_PAGE_ORDER (HPAGE_SHIFT - PAGE_SHIFT)#define HAVE_ARCH_HUGETLB_UNMAPPED_AREA +/* to align the pointer to the (next) page boundary when dealing with hugepages*/ +#define HPAGE_ALIGN (addr) (((addr)+HPAGE_SIZE-1)&HPAGE_MASK)#endif #define pgd_val (x) ((x).pgd)
include/asm-sh64/page.h renamed from include/asm-sh64/page.h.paris +2 -0
@@ -42,6 +42,8 @@#define HPAGE_MASK (~(HPAGE_SIZE-1))#define HUGETLB_PAGE_ORDER (HPAGE_SHIFT-PAGE_SHIFT)#define ARCH_HAS_SETCLEAR_HUGE_PTE +/* to align the pointer to the (next) page boundary when dealing with hugepages*/ +#define HPAGE_ALIGN (addr) (((addr)+HPAGE_SIZE-1)&HPAGE_MASK)#endif #ifdef __KERNEL__
include/asm-sh/page.h renamed from include/asm-sh/page.h.paris +2 -0
@@ -32,6 +32,8 @@#define HPAGE_MASK (~(HPAGE_SIZE-1))#define HUGETLB_PAGE_ORDER (HPAGE_SHIFT-PAGE_SHIFT)#define ARCH_HAS_SETCLEAR_HUGE_PTE +/* to align the pointer to the (next) page boundary when dealing with hugepages*/ +#define HPAGE_ALIGN (addr) (((addr)+HPAGE_SIZE-1)&HPAGE_MASK)#endif #ifdef __KERNEL__
include/asm-x86_64/page.h renamed from include/asm-x86_64/page.h.paris +2 -0
@@ -40,6 +40,8 @@#define HPAGE_SIZE ((1UL) << HPAGE_SHIFT)#define HPAGE_MASK (~(HPAGE_SIZE - 1))#define HUGETLB_PAGE_ORDER (HPAGE_SHIFT - PAGE_SHIFT) +/* to align the pointer to the (next) page boundary when dealing with hugepages*/ +#define HPAGE_ALIGN (addr) (((addr)+HPAGE_SIZE-1)&HPAGE_MASK)#ifdef __KERNEL__#ifndef __ASSEMBLY__
include/asm-parisc/page.h renamed from include/asm-parisc/page.h.paris +2 -0
@@ -140,6 +140,8 @@ extern int npmem_ranges;#define HPAGE_SIZE ((1UL) << HPAGE_SHIFT)#define HPAGE_MASK (~(HPAGE_SIZE - 1))#define HUGETLB_PAGE_ORDER (HPAGE_SHIFT - PAGE_SHIFT) +/* to align the pointer to the (next) page boundary when dealing with hugepages*/ +#define HPAGE_ALIGN (addr) (((addr)+HPAGE_SIZE-1)&HPAGE_MASK)#endif #define virt_addr_valid (kaddr) pfn_valid(__pa(kaddr) >> PAGE_SHIFT)
include/asm-ia64/page.h renamed from include/asm-ia64/page.h.paris +2 -0
@@ -57,6 +57,8 @@# define HAVE_ARCH_HUGETLB_UNMAPPED_AREA # define ARCH_HAS_HUGEPAGE_ONLY_RANGE +/* to align the pointer to the (next) page boundary when dealing with hugepages*/ +#define HPAGE_ALIGN (addr) (((addr)+HPAGE_SIZE-1)&HPAGE_MASK)#endif /* CONFIG_HUGETLB_PAGE */ #ifdef __ASSEMBLY__
include/asm-powerpc/page_64.h renamed from include/asm-powerpc/page_64.h.paris +2 -0
@@ -85,6 +85,8 @@ extern unsigned int HPAGE_SHIFT;#define HPAGE_SIZE ((1UL) << HPAGE_SHIFT)#define HPAGE_MASK (~(HPAGE_SIZE - 1))#define HUGETLB_PAGE_ORDER (HPAGE_SHIFT - PAGE_SHIFT) +/* to align the pointer to the (next) page boundary when dealing with hugepages*/ +#define HPAGE_ALIGN (addr) (((addr)+HPAGE_SIZE-1)&HPAGE_MASK)#endif /* __ASSEMBLY__ */
mm/mlock.c renamed from mm/mlock.c.paris +32 -22
91 lines not on this page · raw message
include/asm-sparc64/page.h typedef unsigned long pgprot_t; · line 109
Quoted 9 lines · Eric Paris
include/asm-sparc64/page.h renamed from include/asm-sparc64/page.h.paris +2
-0@@ -104,6 +104,8 @@ typedef unsigned long pgprot_t;#define HUGETLB_PAGE_ORDER (HPAGE_SHIFT - PAGE_SHIFT)#define ARCH_HAS_SETCLEAR_HUGE_PTE #define ARCH_HAS_HUGETLB_PREFAULT_HOOK +/* to align the pointer to the (next) page boundary when dealing with hugepages*/ +#define HPAGE_ALIGN (addr) (((addr)+HPAGE_SIZE-1)&HPAGE_MASK)#endif
Could you put the definition of HPAGE_ALIGN into include/linux/hugetlb.h to avoid modifying all the page.h files?
Quoted 14 lines · Christoph Lameter
include/asm-sparc64/page.h typedef unsigned long pgprot_t; · line 109
Quoted 9 lines · Eric Paris
include/asm-sparc64/page.h renamed from include/asm-sparc64/page.h.paris +2
-0@@ -104,6 +104,8 @@ typedef unsigned long pgprot_t;#define HUGETLB_PAGE_ORDER (HPAGE_SHIFT - PAGE_SHIFT)#define ARCH_HAS_SETCLEAR_HUGE_PTE #define ARCH_HAS_HUGETLB_PREFAULT_HOOK +/* to align the pointer to the (next) page boundary when dealing with hugepages*/ +#define HPAGE_ALIGN (addr) (((addr)+HPAGE_SIZE-1)&HPAGE_MASK)#endif Could you put the definition of
HPAGE_ALIGNintoinclude/linux/hugetlb.hto avoid modifying all thepage.hfiles?
I could, I was just following the example of PAGE_ALIGN which was in all the page.h files. But since it really is a nonarch specific hugepage only macro maybe that is a better place for it.
If other results come back positive I'll repost with that change if others think it best.
-Eric
Quoted 4 lines · Eric Paris
sys_m{,un}lock and
do_mlockall align memory references and the length of the mlock given by userspace to page boundaries. If the page being mlocked is a hugepage instead of a normal page the start and finish of the mlock will still only be aligned to normal page boundaries.
Quoted 8 lines
Ultimately upon the process exiting we will eventually call
unmap_vmaswhich will callunmap_hugepage_rangefor all of the ranges.unmap_hugepage_rangechecks to make sure the beginning and the end of the range are actually hugepage aligned and if not willBUG(). Since we only aligned to a normal page boundary the end of the first range and the beginning of the second will likely (unless userspace passed of values already hugepage aligned) not be hugepage aligned and thus we bomb.
When did you test this? It should have been fixed in 2.6.11 onwards by split_vma()'s simple:
if (is_vm_hugetlb_page(vma) && (addr & ~HPAGE_MASK)) return -EINVAL;
Hugh
Quoted 14 lines · Hugh Dickins
Quoted 4 lines · Eric Paris
sys_m{,un}lock and
do_mlockall align memory references and the length of the mlock given by userspace to page boundaries. If the page being mlocked is a hugepage instead of a normal page the start and finish of the mlock will still only be aligned to normal page boundaries.Quoted 8 lines
Ultimately upon the process exiting we will eventually call
unmap_vmaswhich will callunmap_hugepage_rangefor all of the ranges.unmap_hugepage_rangechecks to make sure the beginning and the end of the range are actually hugepage aligned and if not willBUG(). Since we only aligned to a normal page boundary the end of the first range and the beginning of the second will likely (unless userspace passed of values already hugepage aligned) not be hugepage aligned and thus we bomb.
Quoted 7 lines
When did you test this? It should have been fixed in 2.6.11 onwards by split_vma()'s simple:
if (is_vm_hugetlb_page(vma) && (addr & ~HPAGE_MASK)) return -EINVAL;Hugh
You're right. My initial BUG() problem was on pre 2.6.11. I wrote this patch and tested that it worked on 2.6.16 to allow the test program I described to run successfully. Admittedly this is the first time I tested on unpatched 2.6.16 and as you said the mlock will fail with "Invalid Argument" instead of BUG(). So the panic is gone post 2.6.11, but the problem remains. We still are rounding incorrectly for hugepages in the sys_mlock and do_mlock functions.
This patch still solves the problem of the kernel currently being more restrictive on what we accept from userspace for the length of the mlock if it is a hugepage rather than a regular page. With a regular page we will round the value from userspace and happily go about our business of mlocking. For a hugepage it just rejects it if userspace doesn't align it themselves. This allows the kernel to do the same work for hugepages that it does for normal pages.
-Eric
Quoted 8 lines · Eric Paris
This patch still solves the problem of the kernel currently being more restrictive on what we accept from userspace for the length of the mlock if it is a hugepage rather than a regular page. With a regular page we will round the value from userspace and happily go about our business of mlocking. For a hugepage it just rejects it if userspace doesn't align it themselves. This allows the kernel to do the same work for hugepages that it does for normal pages.
You do have a point that there's an inconsistency there. But we could argue a long time what's inconsistent with what - I'd say it's mlock being inconsistent with mprotect, munmap, msync, madvise, etc. I don't see an outright reason to change from the current behaviour.
You do realize that there's almost no point in mlocking a hugepage area anyway? (I wrote that first without the "almost", but now with hugepage faulting, it does provide another way to fault in all the pages at once.)
Hugh