Skip to content

Avoid IPI while updating page table entries.

Aneesh Kumar K.V linuxppc-dev oe-kbuild-all 4 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v4 Latest22 patches 4 repliesIn review
  2. v322 patches 0 repliesSuperseded
  3. v222 patches 2 repliesSuperseded
Earlier versions · 1
VersionPostedPatchesRepliesStatus
v1 RFC21 patches0 repliesSuperseded

Changes in v4 from the cover letter

* Rebase to latest kernel

Patches in v4 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/22powerpc/pkeys: Avoid using lockless page table walkarch/powerpc/include/asm/mmu.h arch/powerpc/mm/book3s64/hash_utils.c arch/powerpc/mm/fault.c+60 -561 reply
2/22powerpc/pkeys: Check vma before returning key fault error to the userarch/powerpc/mm/fault.c+0 -80 replies
3/22powerpc/mm/hash64: use _PAGE_PTE when checking for pte_presentarch/powerpc/include/asm/book3s/64/pgtable.h arch/powerpc/mm/book3s64/hash_utils.c+19 -70 replies
4/22powerpc/hash64: Restrict page table lookup using init_mm with __flush_hash_table_rangearch/powerpc/include/asm/book3s/64/tlbflush-hash.h arch/powerpc/kernel/pci_64.c arch/powerpc/mm/book3s64/hash_tlb.c+5 -160 replies
5/22powerpc/book3s64/hash: Use the pte_t address from the callerarch/powerpc/mm/book3s64/hash_utils.c+5 -220 replies
6/22powerpc/mce: Don't reload pte val in addr_to_pfnarch/powerpc/kernel/mce_power.c+9 -50 replies
7/22powerpc/perf/callchain: Use __get_user_pages_fast in read_user_stack_slowarch/powerpc/perf/callchain_64.c+14 -320 replies
8/22powerpc/kvm/book3s: switch from raw_spin_*lock to arch_spin_lock.arch/powerpc/kvm/book3s_hv_rm_mmu.c+4 -40 replies
9/22powerpc/kvm/book3s: Add helper to walk partition scoped linux page table.arch/powerpc/include/asm/kvm_book3s_64.h arch/powerpc/kvm/book3s_64_mmu_radix.c arch/powerpc/kvm/book3s_hv_nested.c+20 -73 replies
10/22powerpc/kvm/nested: Add helper to walk nested shadow linux page table.arch/powerpc/kvm/book3s_hv_nested.c+21 -70 replies
11/22powerpc/kvm/book3s: Use kvm helpers to walk shadow or secondary tablearch/powerpc/include/asm/kvm_book3s.h arch/powerpc/include/asm/kvm_book3s_64.h arch/powerpc/kvm/book3s_64_mmu_radix.c +1 more+19 -170 replies
12/22powerpc/kvm/book3s: Add helper for host page table walkarch/powerpc/include/asm/kvm_book3s_64.h+16 -00 replies
13/22powerpc/kvm/book3s: Use find_kvm_host_pte in page fault handlerarch/powerpc/kvm/book3s_64_mmu_hv.c+4 -40 replies
14/22powerpc/kvm/book3s: Use find_kvm_host_pte in h_enterarch/powerpc/kvm/book3s_64_mmu_hv.c arch/powerpc/kvm/book3s_hv_rm_mmu.c+8 -190 replies
15/22powerpc/kvm/book3s: use find_kvm_host_pte in pute_tce functionsarch/powerpc/kvm/book3s_64_vio_hv.c+24 -60 replies
16/22powerpc/kvm/book3s: Avoid using rmap to protect parallel page table update.arch/powerpc/kvm/book3s_64_vio_hv.c+9 -290 replies
17/22powerpc/kvm/book3s: use find_kvm_host_pte in kvmppc_book3s_instantiate_pagearch/powerpc/kvm/book3s_64_mmu_radix.c+4 -40 replies
18/22powerpc/kvm/book3s: Use find_kvm_host_pte in kvmppc_get_hpaarch/powerpc/kvm/book3s_hv_rm_mmu.c+11 -210 replies
19/22powerpc/kvm/book3s: Use pte_present instead of opencoding _PAGE_PRESENT checkarch/powerpc/include/asm/kvm_book3s_64.h+1 -10 replies
20/22powerpc/mm/book3s64: Avoid sending IPI on clearing PMDarch/powerpc/mm/book3s64/hash_pgtable.c arch/powerpc/mm/book3s64/pgtable.c arch/powerpc/mm/book3s64/radix_pgtable.c+7 -310 replies
21/22mm: change pmdp_huge_get_and_clear_full take vm_area_struct as argarch/s390/include/asm/pgtable.h include/asm-generic/pgtable.h mm/huge_memory.c+6 -60 replies
22/22powerpc/mm/book3s64: Fix MADV_DONTNEED and parallel page fault racearch/powerpc/include/asm/book3s/64/pgtable.h arch/powerpc/mm/book3s64/pgtable.c+23 -00 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v4 yet.

Read the v4 thread

Cover letter [PATCH v4 00/22] Avoid IPI while updating page table entries.

Problem Summary:
Slow termination of KVM guest with large guest RAM config due to a large number of IPIs that were caused by clearing level 1 PTE entries (THP) entries. This is shown in the stack trace below.

- qemu-system-ppc [kernel.vmlinux] [k] smp_call_function_many
- smp_call_function_many

      - 36.09% smp_call_function_many
           serialize_against_pte_lookup
           radix__pmdp_huge_get_and_clear
           zap_huge_pmd
           unmap_page_range
           unmap_vmas
           unmap_region
           __do_munmap
           __vm_munmap
           sys_munmap
          system_call
           __munmap
           qemu_ram_munmap
           qemu_anon_ram_free
           reclaim_ramblock
           call_rcu_thread
           qemu_thread_start
           start_thread
           __clone

Why we need to do IPI when clearing PMD entries:
This was added as part of commit: 13bd817bb884 ("powerpc/thp: Serialize pmd clear against a linux page table walk")

serialize_against_pte_lookup makes sure that all parallel lockless page table walk completes before we convert a PMD pte entry to regular pmd entry. We end up doing that conversion in the below scenarios

1) __split_huge_zero_page_pmd
2) do_huge_pmd_wp_page_fallback
3) MADV_DONTNEED running parallel to page faults.

local_irq_disable and lockless page table walk:

The lockless page table walk work with the assumption that we can dereference the page table contents without holding a lock. For this to work, we need to make sure we read the page table contents atomically and page table pages are not going to be freed/released while we are walking the
table pages. We can achieve by using a rcu based freeing for page table pages or if the architecture implements broadcast tlbie, we can block the IPI as we walk the page table pages.

To support both the above framework, lockless page table walk is done with irq disabled instead of rcu_read_lock()

We do have two interface for lockless page table walk, gup fast and __find_linux_pte. This patch series makes __find_linux_pte table walk safe against the conversion of PMD PTE to regular PMD.

gup fast:

gup fast is already safe against THP split because kernel now differentiate between a pmd split and a compound page split. gup fast can run parallel to a pmd split and we prevent a parallel gup fast to a hugepage split, by freezing the page refcount and failing the speculative page ref increment.

Similar to how gup is safe against parallel pmd split, this patch series updates the __find_linux_pte callers to be safe against a parallel pmd split. We do that by enforcing the following rules.

1) Don't reload the pte value, because that can be updated in parallel.
2) Code should be able to work with a stale PTE value and not the recent one. ie, the pte value that we are looking at may not be the latest value in the page table.
3) Before looking at pte value check for _PAGE_PTE bit. We now do this as part of pte_present() check.

Performance:

This speeds up Qemu guest RAM del/unplug time as below 128 core, 496GB guest:

Without patch:
munmap start: timer = 13162 ms, PID=7684
munmap finish: timer = 95312 ms, PID=7684 - delta = 82150 ms

With patch (upto removing IPI)
munmap start: timer = 196449 ms, PID=6681
munmap finish: timer = 196488 ms, PID=6681 - delta = 39ms

With patch (with adding the tlb invalidate in pmdp_huge_get_and_clear_full) munmap start: timer = 196345 ms, PID=6879
munmap finish: timer = 196714 ms, PID=6879 - delta = 369ms

Changes from V3:
* Rebase to latest kernel

Changes from V2:
* Rebase to lastest kernel

Changes from V1:
* Update commit messages
* Qemu Performance numbers

Aneesh Kumar K.V (22):
powerpc/pkeys: Avoid using lockless page table walk powerpc/pkeys: Check vma before returning key fault error to the user powerpc/mm/hash64: use _PAGE_PTE when checking for pte_present powerpc/hash64: Restrict page table lookup using init_mm with
__flush_hash_table_range
powerpc/book3s64/hash: Use the pte_t address from the caller powerpc/mce: Don't reload pte val in addr_to_pfn powerpc/perf/callchain: Use __get_user_pages_fast in
read_user_stack_slow
powerpc/kvm/book3s: switch from raw_spin_*lock to arch_spin_lock. powerpc/kvm/book3s: Add helper to walk partition scoped linux page
table.
powerpc/kvm/nested: Add helper to walk nested shadow linux page table. powerpc/kvm/book3s: Use kvm helpers to walk shadow or secondary table powerpc/kvm/book3s: Add helper for host page table walk powerpc/kvm/book3s: Use find_kvm_host_pte in page fault handler powerpc/kvm/book3s: Use find_kvm_host_pte in h_enter powerpc/kvm/book3s: use find_kvm_host_pte in pute_tce functions powerpc/kvm/book3s: Avoid using rmap to protect parallel page table
update.
powerpc/kvm/book3s: use find_kvm_host_pte in
kvmppc_book3s_instantiate_page
powerpc/kvm/book3s: Use find_kvm_host_pte in kvmppc_get_hpa powerpc/kvm/book3s: Use pte_present instead of opencoding
_PAGE_PRESENT check
powerpc/mm/book3s64: Avoid sending IPI on clearing PMD
mm: change pmdp_huge_get_and_clear_full take vm_area_struct as arg powerpc/mm/book3s64: Fix MADV_DONTNEED and parallel page fault race

 arch/powerpc/include/asm/book3s/64/pgtable.h  | 20 +++--
 .../include/asm/book3s/64/tlbflush-hash.h     |  3 +-
 arch/powerpc/include/asm/kvm_book3s.h         |  2 +-
 arch/powerpc/include/asm/kvm_book3s_64.h      | 34 ++++++++-
 arch/powerpc/include/asm/mmu.h                |  9 ---
 arch/powerpc/kernel/mce_power.c               | 14 ++--
 arch/powerpc/kernel/pci_64.c                  |  2 +-
 arch/powerpc/kvm/book3s_64_mmu_hv.c           | 13 ++--
 arch/powerpc/kvm/book3s_64_mmu_radix.c        | 38 +++++-----
 arch/powerpc/kvm/book3s_64_vio_hv.c           | 64 ++++++++--------
 arch/powerpc/kvm/book3s_hv_nested.c           | 37 ++++++---
 arch/powerpc/kvm/book3s_hv_rm_mmu.c           | 58 +++++---------
 arch/powerpc/mm/book3s64/hash_pgtable.c       | 11 ---
 arch/powerpc/mm/book3s64/hash_tlb.c           | 16 +---
 arch/powerpc/mm/book3s64/hash_utils.c         | 62 ++++-----------
 arch/powerpc/mm/book3s64/pgtable.c            | 24 ++++--
 arch/powerpc/mm/book3s64/radix_pgtable.c      | 19 ++---
 arch/powerpc/mm/fault.c                       | 75 +++++++++++++------
 arch/powerpc/perf/callchain_64.c              | 46 ++++--------
 arch/s390/include/asm/pgtable.h               |  4 +-
 include/asm-generic/pgtable.h                 |  4 +-
 mm/huge_memory.c                              |  4 +-
 22 files changed, 273 insertions(+), 286 deletions(-)
Signature · 1 line
2.26.2