mm/madvise: add cond_resched() in madvise_cold_or_pageout_pte_range()
Subsystems (2)
v2 cover mm/madvise: add cond_resched() in madvise_cold_or_pageout_pte_range()
I conducted real-time testing and observed that madvise_cold_or_pageout_pte_range() causes significant latency under memory pressure, which can be effectively reduced by adding cond_resched() within the loop.
I tested on the LicheePi 4A board using Cylictest for latency testing and Ftrace for latency tracing. The board uses TH1520 processor and has a memory size of 8GB. The kernel version is 6.5.0 with the PREEMPT_RT patch applied.
The script I tested is as follows:
echo wakeup_rt > /sys/kernel/tracing/current_tracer
echo 1 > /sys/kernel/tracing/tracing_on
echo 0 > /sys/kernel/tracing/tracing_max_latency
stress-ng --vm 8 --vm-bytes 2G &
cyclictest --mlockall --smp --priority=99 --distance=0 --duration=30m echo 0 > /sys/kernel/tracing/tracing_on
cat /sys/kernel/tracing/trace
The tracing results before modification are as follows:
# tracer: wakeup_rt
#
# wakeup_rt latency trace v1.1.5 on 6.5.0-rt6-r1208-00003-g999d221864bf # -------------------------------------------------------------------- # latency: 2552 us, #6/6, CPU#3 | (M:preempt_rt VP:0, KP:0, SP:0 HP:0 #P:4) # ----------------- # | task: cyclictest-196 (uid:0 nice:0 policy:1 rt_prio:99) # ----------------- # # _--------=> CPU# # / _-------=> irqs-off/BH-disabled # | / _------=> need-resched # || / _-----=> need-resched-lazy # ||| / _----=> hardirq/softirq # |||| / _---=> preempt-depth # ||||| / _--=> preempt-lazy-depth # |||||| / _-=> migrate-disable # ||||||| / delay # cmd pid |||||||| time | caller # \ / |||||||| \ | / stress-n-206 3dn.h512 2us : 206:120:R + [003] 196: 0:R cyclictest stress-n-206 3dn.h512 7us : <stack trace> => __ftrace_trace_stack => __trace_stack => probe_wakeup => ttwu_do_activate => try_to_wake_up => wake_up_process => hrtimer_wakeup => __hrtimer_run_queues => hrtimer_interrupt => riscv_timer_interrupt => handle_percpu_devid_irq => generic_handle_domain_irq => riscv_intc_irq => handle_riscv_irq => do_irq stress-n-206 3dn.h512 9us#: 0 stress-n-206 3d...3.. 2544us : __schedule stress-n-206 3d...3.. 2545us : 206:120:R ==> [003] 196: 0:R cyclictest stress-n-206 3d...3.. 2551us : <stack trace> => __ftrace_trace_stack => __trace_stack => probe_wakeup_sched_switch => __schedule => preempt_schedule => migrate_enable => rt_spin_unlock => madvise_cold_or_pageout_pte_range => walk_pgd_range => __walk_page_range => walk_page_range => madvise_pageout => madvise_vma_behavior => do_madvise => sys_madvise => do_trap_ecall_u => ret_from_exception
The tracing results after modification are as follows:
# tracer: wakeup_rt
#
# wakeup_rt latency trace v1.1.5 on 6.5.0-rt6-r1208-00004-gca3876fc69a6-dirty # -------------------------------------------------------------------- # latency: 1689 us, #6/6, CPU#0 | (M:preempt_rt VP:0, KP:0, SP:0 HP:0 #P:4) # ----------------- # | task: cyclictest-217 (uid:0 nice:0 policy:1 rt_prio:99) # ----------------- # # _--------=> CPU# # / _-------=> irqs-off/BH-disabled # | / _------=> need-resched # || / _-----=> need-resched-lazy # ||| / _----=> hardirq/softirq # |||| / _---=> preempt-depth # ||||| / _--=> preempt-lazy-depth # |||||| / _-=> migrate-disable # ||||||| / delay # cmd pid |||||||| time | caller # \ / |||||||| \ | / stress-n-232 0dn.h413 1us+: 232:120:R + [000] 217: 0:R cyclictest stress-n-232 0dn.h413 12us : <stack trace> => __ftrace_trace_stack => __trace_stack => probe_wakeup => ttwu_do_activate => try_to_wake_up => wake_up_process => hrtimer_wakeup => __hrtimer_run_queues => hrtimer_interrupt => riscv_timer_interrupt => handle_percpu_devid_irq => generic_handle_domain_irq => riscv_intc_irq => handle_riscv_irq => do_irq stress-n-232 0dn.h413 19us#: 0 stress-n-232 0d...3.. 1671us : __schedule stress-n-232 0d...3.. 1676us+: 232:120:R ==> [000] 217: 0:R cyclictest stress-n-232 0d...3.. 1687us : <stack trace> => __ftrace_trace_stack => __trace_stack => probe_wakeup_sched_switch => __schedule => preempt_schedule => migrate_enable => free_unref_page_list => release_pages => free_pages_and_swap_cache => tlb_batch_pages_flush => tlb_flush_mmu => unmap_page_range => unmap_vmas => unmap_region => do_vmi_align_munmap.constprop.0 => do_vmi_munmap => __vm_munmap => sys_munmap => do_trap_ecall_u => ret_from_exception
After the modification, the cause of maximum latency is no longer madvise_cold_or_pageout_pte_range(), so this modification can reduce the latency caused by madvise_cold_or_pageout_pte_range().
Changes in v2:
- re-enter the loop after the cond_resched() method
Jiexun Wang (1):
mm/madvise: add cond_resched() in madvise_cold_or_pageout_pte_range()
mm/madvise.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
Signature · 1 line
2.34.1
A message that is not in the archive (sent off-list or not yet received): 85363861af65fac66c7a98c251906afc0d9c8098.1695291046.git.wangjiexun@tinylab.org
Andrew Morton
maintainerQuoted 20 lines · On Thu, 21 Sep 2023 20:27:51 +0800 Jiexun Wang <[email protected]> wrote:
Currently the
madvise_cold_or_pageout_pte_range()function exhibits significant latency under memory pressure, which can be effectively reduced by addingcond_resched()within the loop.When the
batch_countreachesSWAP_CLUSTER_MAX, we reschedule the task to ensure fairness and avoid long lock holding times....
mm/madvise.c +1
-0@@ -354,6 +354,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,struct folio *folio =NULL ;LIST_HEAD (folio_list);bool pageout_anon_only_filter; +unsigned int batch_count =0 ;if (fatal_signal_pending (current))return -EINTR ;
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 438
Quoted 5 lines
mm/madvise.c +1
-0@@ -433,6 +434,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, regular_folio:#endif tlb_change_page_size (tlb,PAGE_SIZE ); +restart: start_pte = pte =pte_offset_map_lock (vma->vm_mm, pmd, addr, &ptl);
The handling of start_pte looks OK.
Quoted 8 lines
mm/madvise.c +2
-0if (!start_pte)return 0 ;@@ -441,6 +443,15 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,for (; addr < end; pte++, addr +=PAGE_SIZE ) { ptent =ptep_get (pte); +if (++batch_count ==SWAP_CLUSTER_MAX ) { + batch_count =0 ;
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 457
Quoted 10 lines
mm/madvise.c +7
-0+if (need_resched ()) { +pte_unmap_unlock (start_pte, ptl); +cond_resched (); + goto restart; + } + } +if (pte_none (ptent))continue ;
I think this patch looks OK, but would appreciate careful review from others, please.
Quoted 34 lines · Andrew Morton
Quoted 20 lines
Currently the
madvise_cold_or_pageout_pte_range()function exhibits significant latency under memory pressure, which can be effectively reduced by addingcond_resched()within the loop.When the
batch_countreachesSWAP_CLUSTER_MAX, we reschedule the task to ensure fairness and avoid long lock holding times....
mm/madvise.c +1
-0@@ -354,6 +354,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,struct folio *folio =NULL ;LIST_HEAD (folio_list);bool pageout_anon_only_filter; +unsigned int batch_count =0 ;if (fatal_signal_pending (current))return -EINTR ;mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 438
Quoted 5 lines
mm/madvise.c +1
-0@@ -433,6 +434,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, regular_folio:#endif tlb_change_page_size (tlb,PAGE_SIZE ); +restart: start_pte = pte =pte_offset_map_lock (vma->vm_mm, pmd, addr, &ptl);The handling of
start_ptelooks OK.Quoted 6 lines
mm/madvise.c +0
-0if (!start_pte)return 0 ;@@ -441,6 +443,15 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,for (; addr < end; pte++, addr +=PAGE_SIZE ) { ptent =ptep_get (pte);
Quoted 4 lines
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 449
Quoted 4 lines
mm/madvise.c +4
-0+if (++batch_count ==SWAP_CLUSTER_MAX ) { + batch_count =0 ; +if (need_resched ()) { +pte_unmap_unlock (start_pte, ptl);
Shouldn't it leave lazy MMU mode here?
---
mm/madvise.c +1 -0
@@ -451,6 +451,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,if (++batch_count ==SWAP_CLUSTER_MAX ) { batch_count =0 ;if (need_resched ()) { +arch_leave_lazy_mmu_mode ();pte_unmap_unlock (start_pte, ptl);cond_resched (); goto restart;
PATCH 1/1 mm/madvise: don't forget to leave lazy MMU mode in madvise_cold_or_pageout_pte_range()
In reply to Andrew MortonWe need to leave lazy MMU mode before unlocking.
--- mm/madvise.c | 1 + 1 file changed, 1 insertion(+)
mm/madvise.c +1 -0
@@ -451,6 +451,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,if (++batch_count ==SWAP_CLUSTER_MAX ) { batch_count =0 ;if (need_resched ()) { +arch_leave_lazy_mmu_mode ();pte_unmap_unlock (start_pte, ptl);cond_resched (); goto restart;
Signature · 1 line
2.43.0.429.g432eaa2c6b-goog
Andrew Morton
maintainerRe: [PATCH] mm/madvise: don't forget to leave lazy MMU mode in madvise_cold_or_pageout_pte_range()
Quoted 1 line · On Fri, 26 Jan 2024 12:25:48 +0900 Sergey Senozhatsky…
We need to leave lazy MMU mode before unlocking.
What might be the userspace-visible effects of this?
Quoted 2 lines
Fixes b2f557a21bc8 ("mm/madvise: add cond_resched() in madvise_cold_or_pageout_pte_range()"Signed-off-by Sergey Senozhatsky <[email protected]>
I'll add a cc:stable.
Quoted 10 lines
mm/madvise.c +1
-0@@ -451,6 +451,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,if (++batch_count ==SWAP_CLUSTER_MAX ) { batch_count =0 ;if (need_resched ()) { +arch_leave_lazy_mmu_mode ();pte_unmap_unlock (start_pte, ptl);cond_resched (); goto restart;
Re: [PATCH] mm/madvise: don't forget to leave lazy MMU mode in madvise_cold_or_pageout_pte_range()
Quoted 10 lines · Andrew Morton
Date: Thu, 25 Jan 2024 22:53:36 -0800
From: Andrew Morton <[email protected]>
To: Sergey Senozhatsky <[email protected]>
Cc: Jiexun Wang <[email protected]>, [email protected],
[email protected]
Subject: Re: [PATCH] mm/madvise: don't forget to leave lazy MMU mode in
madvise_cold_or_pageout_pte_range()
Message-Id: <[email protected]>
X-Mailer: Sylpheed 3.8.0beta1 (GTK+ 2.24.33; x86_64-pc-linux-gnu)
Quoted 3 lines
Quoted 1 line · On Fri, 26 Jan 2024 12:25:48 +0900 Sergey Senozhatsky…
We need to leave lazy MMU mode before unlocking.
It depends on the arch, as far as I understand it.
We can enter lazy MMU mode (on each goto restart) more times than leave it, and, for isntance, on powerpc that means that we can preempt_disable() more times than preempt_enable(). That's how enter/leave lazy MMU mode is implemented there:
static inline void arch_enter_lazy_mmu_mode(void)
{
struct ppc64_tlb_batch *batch;
if (radix_enabled())
return;
/*
* apply_to_page_range can call us this preempt enabled when
* operating on kernel page tables.
*/
preempt_disable();
batch = this_cpu_ptr(&ppc64_tlb_batch);
batch->active = 1;
}static inline void arch_leave_lazy_mmu_mode(void)
{
struct ppc64_tlb_batch *batch;
if (radix_enabled())
return;
batch = this_cpu_ptr(&ppc64_tlb_batch);
if (batch->index)
__flush_tlb_pending(batch);
batch->active = 0;
preempt_enable();
}Quoted 6 lines
What might be the userspace-visible effects of this?
Quoted 2 lines
Fixes b2f557a21bc8 ("mm/madvise: add cond_resched() in madvise_cold_or_pageout_pte_range()"Signed-off-by Sergey Senozhatsky <[email protected]>I'll add a cc:stable.
Thanks.