ARM: mm: make UACCESS_WITH_MEMCPY huge page aware
PATCH v2 1/1 ARM: mm: make UACCESS_WITH_MEMCPY huge page aware
The memory pinning code in uaccess_with_memcpy.c does not check for HugeTLB or THP pmds, and will enter an infinite loop should a __copy_to_user or __clear_user occur against a huge page.
This patch adds detection code for huge pages to pin_page_for_write. As this code can be executed in a fast path it refers to the actual pmds rather than the vma. If a HugeTLB or THP is found (they have the same pmd representation on ARM), the page table spinlock is taken to prevent modification whilst the page is pinned.
On ARM, huge pages are only represented as pmds, thus no huge pud checks are performed. (For huge puds one would lock the page table in a similar manner as in the pmd case).
Two helper functions are introduced; pmd_thp_or_huge will check whether or not a page is huge or transparent huge (which have the same pmd layout on ARM), and pmd_hugewillfault will detect whether or not a page fault will occur on write to the page.
Running the following test (with the chunking from read_zero removed):
$ dd if=/dev/zero of=/dev/null bs=10M count=1024
Gave: 2.3 GB/s backed by normal pages,
2.9 GB/s backed by huge pages,
5.1 GB/s backed by huge pages, with page mask=HPAGE_MASK.After some discussion, it was decided not to adopt the HPAGE_MASK, as this would have a significant detrimental effect on the overall system latency due to page_table_lock being held for too long. This could be revisited if split huge page locks are adopted.
--- arch/arm/include/asm/pgtable-3level.h | 3 +++ arch/arm/lib/uaccess_with_memcpy.c | 41 ++++++++++++++++++++++++++++++++--- 2 files changed, 41 insertions(+), 3 deletions(-)
arch/arm/include/asm/pgtable-3level.h +3 -0
@@ -206,6 +206,9 @@ static inline pmd_t *pmd_offset(pud_t *pud, unsigned long addr)#define __HAVE_ARCH_PMD_WRITE#define pmd_write (pmd) (!(pmd_val(pmd) & PMD_SECT_RDONLY)) +#define pmd_hugewillfault (pmd) (!pmd_young(pmd) || !pmd_write(pmd)) +#define pmd_thp_or_huge (pmd) (pmd_huge(pmd) || pmd_trans_huge(pmd)) +#ifdef CONFIG_TRANSPARENT_HUGEPAGE #define pmd_trans_huge (pmd) (pmd_val(pmd) && !(pmd_val(pmd) & PMD_TABLE_BIT))#define pmd_trans_splitting (pmd) (pmd_val(pmd) & PMD_SECT_SPLITTING)
arch/arm/lib/uaccess_with_memcpy.c +38 -3
65 lines not on this page · raw message
Signature · 1 line
1.8.1.4
PATCH v2 1/1 ARM: mm: make UACCESS_WITH_MEMCPY huge page aware
Quoted 3 lines · Steve Capper
The memory pinning code in
uaccess_with_memcpy.cdoes not check for HugeTLB or THP pmds, and will enter an infinite loop should a__copy_to_useror__clear_useroccur against a huge page.
This patch causes a build error with certain configuration options:
arch/arm/lib/uaccess_with_memcpy.c: In function 'pin_page_for_write':arch/arm/lib/uaccess_with_memcpy.c:57:2: error: implicit declaration of function 'pmd_thp_or_huge' arch/arm/lib/uaccess_with_memcpy.c:60:3: error: implicit declaration of function 'pmd_hugewillfault'
found via a randconfig build, configuration:
http://www.arm.linux.org.uk/developer/build/file.php?type=config&idx=6265
Quoted 4 lines
--- arch/arm/include/asm/pgtable-3level.h | 3 +++ arch/arm/lib/uaccess_with_memcpy.c | 41 ++++++++++++++++++++++++++++++++--- 2 files changed, 41 insertions(+), 3 deletions(-)
As the macros are in pgtable-3level.h, this won't work with any configuration using the 2level header file.
PATCH v2 1/1 ARM: mm: make UACCESS_WITH_MEMCPY huge page aware
Quoted 15 lines · Russell King - ARM Linux
Quoted 3 lines · Steve Capper
The memory pinning code in
uaccess_with_memcpy.cdoes not check for HugeTLB or THP pmds, and will enter an infinite loop should a__copy_to_useror__clear_useroccur against a huge page.This patch causes a build error with certain configuration options:
arch/arm/lib/uaccess_with_memcpy.c: In function 'pin_page_for_write':arch/arm/lib/uaccess_with_memcpy.c:57:2: error: implicit declaration of function 'pmd_thp_or_huge'arch/arm/lib/uaccess_with_memcpy.c:60:3: error: implicit declaration of function 'pmd_hugewillfault'found via a randconfig build, configuration:
http://www.arm.linux.org.uk/developer/build/file.php?type=config&idx=6265
Quoted 7 lines
Quoted 4 lines
--- arch/arm/include/asm/pgtable-3level.h | 3 +++ arch/arm/lib/uaccess_with_memcpy.c | 41 ++++++++++++++++++++++++++++++++--- 2 files changed, 41 insertions(+), 3 deletions(-)As the macros are in
pgtable-3level.h, this won't work with any configuration using the 2level header file.
Hi Russell,
Apologies, that was careless rebasing on my part.
Would you like a fix for this, or should I send a new patch?
Thanks,
Signature · 1 line
Steve
PATCH v2 1/1 ARM: mm: make UACCESS_WITH_MEMCPY huge page aware
Quoted 21 lines · Steve Capper
Quoted 20 lines · Russell King - ARM Linux
Quoted 3 lines · Steve Capper
The memory pinning code in
uaccess_with_memcpy.cdoes not check for HugeTLB or THP pmds, and will enter an infinite loop should a__copy_to_useror__clear_useroccur against a huge page.This patch causes a build error with certain configuration options:
arch/arm/lib/uaccess_with_memcpy.c: In function 'pin_page_for_write':arch/arm/lib/uaccess_with_memcpy.c:57:2: error: implicit declaration of function 'pmd_thp_or_huge'arch/arm/lib/uaccess_with_memcpy.c:60:3: error: implicit declaration of function 'pmd_hugewillfault'found via a randconfig build, configuration:
http://www.arm.linux.org.uk/developer/build/file.php?type=config&idx=6265
Quoted 4 lines
--- arch/arm/include/asm/pgtable-3level.h | 3 +++ arch/arm/lib/uaccess_with_memcpy.c | 41 ++++++++++++++++++++++++++++++++--- 2 files changed, 41 insertions(+), 3 deletions(-)
Quoted 7 lines
Quoted 2 lines
As the macros are in
pgtable-3level.h, this won't work with any configuration using the 2level header file.Hi Russell,
Apologies, that was careless rebasing on my part.Would you like a fix for this, or should I send a new patch?
I'd prefer a new patch please.
Thanks.