Skip to contentSkip to conversation

c0bff412e6: stress-ng.clone.ops_per_sec -2.9% regression

linus:master kernel test robot 4 participants 22 messages Jul 30, 2024 – Aug 14, 2024

Subsystems (3)
lore
Share & tools
lore

kernel test robot

author build bot
#0 lore raw

Hello,

kernel test robot noticed a -2.9% regression of stress-ng.clone.ops_per_sec on:

commit: c0bff412e67b781d761e330ff9578aa9ed2be79e ("mm: allow anon exclusive check over hugetlb tail pages")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

testcase: stress-ng
test machine: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory parameters:

	nr_threads: 100%
	testtime: 60s
	test: clone
	cpufreq_governor: performance

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-lkp/[email protected]

Details are as below:
-------------------------------------------------------------------------------------------------->

The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20240730/[email protected]

=========================================================================================
compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
gcc-13/performance/x86_64-rhel-8.3/100%/debian-12-x86_64-20240206.cgz/lkp-icl-2sp8/clone/stress-ng/60s

commit:
9cb28da546 ("mm/gup: handle hugetlb in the generic follow_page_mask code") c0bff412e6 ("mm: allow anon exclusive check over hugetlb tail pages")

9cb28da54643ad46 c0bff412e67b781d761e330ff95
---------------- ---------------------------

         %stddev     %change         %stddev
             \          |                \  
     37842            -3.4%      36554        vmstat.system.cs
      0.00 ± 17%     -86.4%       0.00 ±223%  sched_debug.rt_rq:.rt_time.avg
      0.19 ± 17%     -86.4%       0.03 ±223%  sched_debug.rt_rq:.rt_time.max
      0.02 ± 17%     -86.4%       0.00 ±223%  sched_debug.rt_rq:.rt_time.stddev
     24081            -3.7%      23200        proc-vmstat.nr_page_table_pages
    399380            -2.3%     390288        proc-vmstat.nr_slab_reclaimable
   1625589            -2.4%    1585989        proc-vmstat.nr_slab_unreclaimable
 1.019e+08            -3.8%   98035999        proc-vmstat.numa_hit
 1.018e+08            -3.9%   97870705        proc-vmstat.numa_local
 1.092e+08            -3.8%   1.05e+08        proc-vmstat.pgalloc_normal
  1.06e+08            -3.8%  1.019e+08        proc-vmstat.pgfree
   2659199            -2.3%    2597978        proc-vmstat.pgreuse
      2910            +3.4%       3010        stress-ng.clone.microsecs_per_clone
    562874            -2.9%     546587        stress-ng.clone.ops
      9298            -2.9%       9031        stress-ng.clone.ops_per_sec
    686858            -2.8%     667416        stress-ng.time.involuntary_context_switches
   9091031            -3.9%    8734352        stress-ng.time.minor_page_faults
      4200            +2.4%       4299        stress-ng.time.percent_of_cpu_this_job_got
      2543            +2.4%       2603        stress-ng.time.system_time
    342849            -2.8%     333189        stress-ng.time.voluntary_context_switches
      6.67            -6.1%       6.26        perf-stat.i.MPKI
 6.388e+08            -5.4%  6.045e+08        perf-stat.i.cache-misses
 1.558e+09            -4.6%  1.487e+09        perf-stat.i.cache-references
     40791            -3.6%      39330        perf-stat.i.context-switches
    353.55            +5.4%     372.76        perf-stat.i.cycles-between-cache-misses
      7.95 ±  3%      -6.5%       7.43 ±  3%  perf-stat.i.metric.K/sec
    251389 ±  3%      -6.5%     235029 ±  3%  perf-stat.i.minor-faults
    251423 ±  3%      -6.5%     235064 ±  3%  perf-stat.i.page-faults
      6.75            -6.1%       6.33        perf-stat.overall.MPKI
      0.38            -0.0        0.37        perf-stat.overall.branch-miss-rate%
    350.09            +5.8%     370.24        perf-stat.overall.cycles-between-cache-misses
  68503488            -1.2%   67660585        perf-stat.ps.branch-misses
  6.33e+08            -5.4%  5.987e+08        perf-stat.ps.cache-misses
 1.518e+09            -4.6%  1.449e+09        perf-stat.ps.cache-references
     38819            -3.3%      37542        perf-stat.ps.context-switches
      3637            +1.2%       3680        perf-stat.ps.cpu-migrations
    235473 ±  3%      -6.3%     220601 ±  3%  perf-stat.ps.minor-faults
    235504 ±  3%      -6.3%     220632 ±  3%  perf-stat.ps.page-faults
     45.55            -2.5       43.04        perf-profile.calltrace.cycles-pp.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap.__mmput
     44.86            -2.5       42.37        perf-profile.calltrace.cycles-pp.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap
     44.42            -2.1       42.37        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     44.42            -2.1       42.37        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     44.41            -2.1       42.36        perf-profile.calltrace.cycles-pp.__x64_sys_exit.do_syscall_64.entry_SYSCALL_64_after_hwframe
     44.41            -2.1       42.36        perf-profile.calltrace.cycles-pp.do_exit.__x64_sys_exit.do_syscall_64.entry_SYSCALL_64_after_hwframe
     39.08            -1.7       37.34        perf-profile.calltrace.cycles-pp.exit_mm.do_exit.__x64_sys_exit.do_syscall_64.entry_SYSCALL_64_after_hwframe
     38.96            -1.7       37.22        perf-profile.calltrace.cycles-pp.exit_mmap.__mmput.exit_mm.do_exit.__x64_sys_exit
     38.97            -1.7       37.24        perf-profile.calltrace.cycles-pp.__mmput.exit_mm.do_exit.__x64_sys_exit.do_syscall_64
     36.16            -1.6       34.57        perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.__mmput.exit_mm.do_exit
     35.99            -1.6       34.40        perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.__mmput.exit_mm
     32.17            -1.5       30.62        perf-profile.calltrace.cycles-pp.zap_present_ptes.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas
     12.49            -1.0       11.52        perf-profile.calltrace.cycles-pp._compound_head.zap_present_ptes.zap_pte_range.zap_pmd_range.unmap_page_range
      9.66            -0.9        8.74        perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.__mmput.copy_process.kernel_clone
      9.61            -0.9        8.69        perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.__mmput.copy_process
     10.71            -0.9        9.84        perf-profile.calltrace.cycles-pp.__mmput.copy_process.kernel_clone.__do_sys_clone3.do_syscall_64
     10.70            -0.9        9.84        perf-profile.calltrace.cycles-pp.exit_mmap.__mmput.copy_process.kernel_clone.__do_sys_clone3
     10.41            -0.8        9.58        perf-profile.calltrace.cycles-pp.__tlb_batch_free_encoded_pages.tlb_flush_mmu.zap_pte_range.zap_pmd_range.unmap_page_range
     10.42            -0.8        9.59        perf-profile.calltrace.cycles-pp.tlb_flush_mmu.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas
     10.21            -0.8        9.40        perf-profile.calltrace.cycles-pp.free_pages_and_swap_cache.__tlb_batch_free_encoded_pages.tlb_flush_mmu.zap_pte_range.zap_pmd_range
      5.47            -0.4        5.04        perf-profile.calltrace.cycles-pp.folios_put_refs.free_pages_and_swap_cache.__tlb_batch_free_encoded_pages.tlb_flush_mmu.zap_pte_range
      1.11            -0.3        0.79 ± 33%  perf-profile.calltrace.cycles-pp.anon_vma_interval_tree_insert.anon_vma_clone.anon_vma_fork.dup_mmap.dup_mm
     14.18            -0.3       13.87        perf-profile.calltrace.cycles-pp.folio_remove_rmap_ptes.zap_present_ptes.zap_pte_range.zap_pmd_range.unmap_page_range
      5.17            -0.3        4.86        perf-profile.calltrace.cycles-pp.put_files_struct.do_exit.__x64_sys_exit.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.80            -0.3        4.53        perf-profile.calltrace.cycles-pp.filp_close.put_files_struct.do_exit.__x64_sys_exit.do_syscall_64
      4.40            -0.3        4.14        perf-profile.calltrace.cycles-pp.filp_flush.filp_close.put_files_struct.do_exit.__x64_sys_exit
      2.74            -0.2        2.58        perf-profile.calltrace.cycles-pp.anon_vma_fork.dup_mmap.dup_mm.copy_process.kernel_clone
      2.25            -0.1        2.11        perf-profile.calltrace.cycles-pp.anon_vma_clone.anon_vma_fork.dup_mmap.dup_mm.copy_process
      1.47            -0.1        1.34        perf-profile.calltrace.cycles-pp.put_files_struct.copy_process.kernel_clone.__do_sys_clone3.do_syscall_64
      1.87            -0.1        1.76        perf-profile.calltrace.cycles-pp.dnotify_flush.filp_flush.filp_close.put_files_struct.do_exit
      1.98            -0.1        1.88        perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.__mmput.exit_mm.do_exit
      1.28            -0.1        1.18        perf-profile.calltrace.cycles-pp.filp_close.put_files_struct.copy_process.kernel_clone.__do_sys_clone3
      1.19            -0.1        1.09        perf-profile.calltrace.cycles-pp.filp_flush.filp_close.put_files_struct.copy_process.kernel_clone

1.31 ± 2% -0.1 1.25 perf-profile.calltrace.cycles-pp.unlink_anon_vmas.free_pgtables.exit_mmap.__mmput.exit_mm
0.58 -0.0 0.55 perf-profile.calltrace.cycles-pp.vm_normal_page.zap_present_ptes.zap_pte_range.zap_pmd_range.unmap_page_range
33.54 +0.6 34.10 perf-profile.calltrace.cycles-pp.syscall
33.45 +0.6 34.01 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall
33.45 +0.6 34.01 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.syscall
33.35 +0.6 33.90 perf-profile.calltrace.cycles-pp.__do_sys_clone3.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall
33.34 +0.6 33.90 perf-profile.calltrace.cycles-pp.kernel_clone.__do_sys_clone3.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall
33.30 +0.6 33.86 perf-profile.calltrace.cycles-pp.copy_process.kernel_clone.__do_sys_clone3.do_syscall_64.entry_SYSCALL_64_after_hwframe
20.63 +1.6 22.21 perf-profile.calltrace.cycles-pp.dup_mm.copy_process.kernel_clone.__do_sys_clone3.do_syscall_64
20.55 +1.6 22.14 perf-profile.calltrace.cycles-pp.dup_mmap.dup_mm.copy_process.kernel_clone.__do_sys_clone3
19.40 +1.8 21.19 perf-profile.calltrace.cycles-pp.__clone
19.24 +1.8 21.04 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__clone
19.24 +1.8 21.04 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__clone
19.14 +1.8 20.94 perf-profile.calltrace.cycles-pp.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe.__clone
19.14 +1.8 20.94 perf-profile.calltrace.cycles-pp.kernel_clone.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe.__clone
19.05 +1.8 20.85 perf-profile.calltrace.cycles-pp.copy_process.kernel_clone.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe
18.74 +1.8 20.56 perf-profile.calltrace.cycles-pp.dup_mm.copy_process.kernel_clone.__do_sys_clone.do_syscall_64
18.67 +1.8 20.49 perf-profile.calltrace.cycles-pp.dup_mmap.dup_mm.copy_process.kernel_clone.__do_sys_clone
12.24 +3.1 15.35 perf-profile.calltrace.cycles-pp._compound_head.copy_present_ptes.copy_pte_range.copy_p4d_range.copy_page_range
34.37 +3.7 38.02 perf-profile.calltrace.cycles-pp.copy_page_range.dup_mmap.dup_mm.copy_process.kernel_clone
34.34 +3.7 38.00 perf-profile.calltrace.cycles-pp.copy_p4d_range.copy_page_range.dup_mmap.dup_mm.copy_process
30.99 +3.7 34.69 perf-profile.calltrace.cycles-pp.copy_present_ptes.copy_pte_range.copy_p4d_range.copy_page_range.dup_mmap
33.16 +3.7 36.88 perf-profile.calltrace.cycles-pp.copy_pte_range.copy_p4d_range.copy_page_range.dup_mmap.dup_mm
0.00 +3.9 3.90 perf-profile.calltrace.cycles-pp.folio_try_dup_anon_rmap_ptes.copy_present_ptes.copy_pte_range.copy_p4d_range.copy_page_range
49.67 -2.6 47.07 perf-profile.children.cycles-pp.exit_mmap
49.69 -2.6 47.08 perf-profile.children.cycles-pp.__mmput
45.84 -2.5 43.32 perf-profile.children.cycles-pp.unmap_vmas
45.56 -2.5 43.05 perf-profile.children.cycles-pp.zap_pmd_range
45.61 -2.5 43.10 perf-profile.children.cycles-pp.unmap_page_range
44.98 -2.5 42.48 perf-profile.children.cycles-pp.zap_pte_range
44.53 -2.1 42.48 perf-profile.children.cycles-pp.__x64_sys_exit
44.54 -2.1 42.48 perf-profile.children.cycles-pp.do_exit
39.10 -1.7 37.36 perf-profile.children.cycles-pp.exit_mm
32.99 -1.6 31.41 perf-profile.children.cycles-pp.zap_present_ptes
10.53 -0.8 9.71 perf-profile.children.cycles-pp.tlb_flush_mmu
10.91 -0.7 10.19 perf-profile.children.cycles-pp.__tlb_batch_free_encoded_pages
10.88 -0.7 10.16 perf-profile.children.cycles-pp.free_pages_and_swap_cache
6.64 -0.4 6.22 perf-profile.children.cycles-pp.put_files_struct
5.76 -0.4 5.38 perf-profile.children.cycles-pp.folios_put_refs
6.11 -0.4 5.73 perf-profile.children.cycles-pp.filp_close
5.62 -0.4 5.25 perf-profile.children.cycles-pp.filp_flush
14.28 -0.3 13.97 perf-profile.children.cycles-pp.folio_remove_rmap_ptes
2.75 -0.2 2.58 perf-profile.children.cycles-pp.anon_vma_fork
2.38 -0.2 2.22 perf-profile.children.cycles-pp.dnotify_flush
2.50 -0.1 2.36 perf-profile.children.cycles-pp.free_pgtables
2.25 -0.1 2.11 perf-profile.children.cycles-pp.anon_vma_clone
0.20 ± 33% -0.1 0.08 ± 58% perf-profile.children.cycles-pp.ordered_events__queue
0.20 ± 33% -0.1 0.08 ± 58% perf-profile.children.cycles-pp.queue_event
1.24 ± 4% -0.1 1.14 perf-profile.children.cycles-pp.down_write
1.67 ± 2% -0.1 1.58 perf-profile.children.cycles-pp.unlink_anon_vmas
1.59 -0.1 1.50 perf-profile.children.cycles-pp.__alloc_pages_noprof
1.55 -0.1 1.46 perf-profile.children.cycles-pp.alloc_pages_mpol_noprof
1.58 -0.1 1.50 perf-profile.children.cycles-pp.vm_normal_page
1.11 -0.1 1.04 perf-profile.children.cycles-pp.anon_vma_interval_tree_insert
1.33 -0.1 1.26 ± 2% perf-profile.children.cycles-pp.pte_alloc_one
0.47 ± 11% -0.1 0.40 ± 4% perf-profile.children.cycles-pp.rwsem_down_write_slowpath
0.45 ± 11% -0.1 0.38 ± 4% perf-profile.children.cycles-pp.rwsem_optimistic_spin
1.00 -0.1 0.94 ± 2% perf-profile.children.cycles-pp.get_page_from_freelist
1.36 -0.1 1.31 perf-profile.children.cycles-pp.kmem_cache_free
1.08 -0.0 1.04 perf-profile.children.cycles-pp.kmem_cache_alloc_noprof
0.62 -0.0 0.58 ± 2% perf-profile.children.cycles-pp.dup_fd
0.63 -0.0 0.59 ± 3% perf-profile.children.cycles-pp.__pte_alloc
0.73 -0.0 0.69 perf-profile.children.cycles-pp.__tlb_remove_folio_pages_size
0.58 -0.0 0.54 perf-profile.children.cycles-pp.locks_remove_posix
0.90 -0.0 0.86 perf-profile.children.cycles-pp.copy_huge_pmd
0.54 -0.0 0.51 perf-profile.children.cycles-pp.__memcg_kmem_charge_page
0.76 -0.0 0.72 perf-profile.children.cycles-pp.vm_area_dup
0.31 ± 2% -0.0 0.28 ± 3% perf-profile.children.cycles-pp.rwsem_spin_on_owner
0.50 -0.0 0.47 perf-profile.children.cycles-pp.__anon_vma_interval_tree_remove
0.53 -0.0 0.50 perf-profile.children.cycles-pp.clear_page_erms
0.49 -0.0 0.46 perf-profile.children.cycles-pp.free_swap_cache
0.72 -0.0 0.69 perf-profile.children.cycles-pp.__memcg_slab_post_alloc_hook
0.37 ± 2% -0.0 0.34 ± 2% perf-profile.children.cycles-pp.unlink_file_vma
0.62 -0.0 0.60 perf-profile.children.cycles-pp.__memcg_slab_free_hook
0.42 -0.0 0.40 ± 2% perf-profile.children.cycles-pp.rmqueue
0.37 -0.0 0.35 ± 2% perf-profile.children.cycles-pp.__rmqueue_pcplist
0.28 -0.0 0.25 perf-profile.children.cycles-pp.__rb_insert_augmented
0.35 -0.0 0.33 ± 2% perf-profile.children.cycles-pp.rmqueue_bulk
0.56 -0.0 0.54 perf-profile.children.cycles-pp.fput
0.48 -0.0 0.46 perf-profile.children.cycles-pp._raw_spin_lock
0.51 -0.0 0.50 perf-profile.children.cycles-pp.free_unref_page
0.45 -0.0 0.43 perf-profile.children.cycles-pp.__x64_sys_unshare
0.44 -0.0 0.42 perf-profile.children.cycles-pp.free_unref_page_commit
0.45 -0.0 0.43 perf-profile.children.cycles-pp.ksys_unshare
0.31 -0.0 0.30 perf-profile.children.cycles-pp.memcg_account_kmem
0.27 -0.0 0.26 perf-profile.children.cycles-pp.__mod_memcg_state
0.44 -0.0 0.43 perf-profile.children.cycles-pp.__slab_free
0.28 -0.0 0.26 perf-profile.children.cycles-pp.__vm_area_free
0.22 ± 2% -0.0 0.21 perf-profile.children.cycles-pp.___slab_alloc
0.21 -0.0 0.20 ± 2% perf-profile.children.cycles-pp.__tlb_remove_folio_pages
0.13 -0.0 0.12 perf-profile.children.cycles-pp.__rb_erase_color
0.07 -0.0 0.06 perf-profile.children.cycles-pp.find_idlest_cpu
0.09 -0.0 0.08 perf-profile.children.cycles-pp.wake_up_new_task
0.06 -0.0 0.05 perf-profile.children.cycles-pp.kfree
0.06 -0.0 0.05 perf-profile.children.cycles-pp.update_sg_wakeup_stats
0.11 -0.0 0.10 perf-profile.children.cycles-pp.allocate_slab
0.44 ± 2% +0.1 0.53 ± 2% perf-profile.children.cycles-pp.tlb_finish_mmu
98.24 +0.2 98.46 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
98.24 +0.2 98.46 perf-profile.children.cycles-pp.do_syscall_64
33.55 +0.6 34.10 perf-profile.children.cycles-pp.syscall
33.35 +0.6 33.90 perf-profile.children.cycles-pp.__do_sys_clone3
19.41 +1.8 21.20 perf-profile.children.cycles-pp.__clone
19.14 +1.8 20.94 perf-profile.children.cycles-pp.__do_sys_clone
24.94 +2.1 27.07 perf-profile.children.cycles-pp._compound_head
52.48 +2.4 54.84 perf-profile.children.cycles-pp.kernel_clone
52.36 +2.4 54.72 perf-profile.children.cycles-pp.copy_process
39.38 +3.4 42.77 perf-profile.children.cycles-pp.dup_mm
39.24 +3.4 42.64 perf-profile.children.cycles-pp.dup_mmap
34.34 +3.7 38.00 perf-profile.children.cycles-pp.copy_p4d_range
34.37 +3.7 38.03 perf-profile.children.cycles-pp.copy_page_range
33.28 +3.7 36.98 perf-profile.children.cycles-pp.copy_pte_range
31.41 +3.8 35.18 perf-profile.children.cycles-pp.copy_present_ptes
0.00 +4.0 4.01 perf-profile.children.cycles-pp.folio_try_dup_anon_rmap_ptes
18.44 -3.2 15.24 perf-profile.self.cycles-pp.copy_present_ptes
5.66 -0.4 5.28 perf-profile.self.cycles-pp.folios_put_refs
4.78 -0.3 4.46 perf-profile.self.cycles-pp.free_pages_and_swap_cache
14.11 -0.3 13.80 perf-profile.self.cycles-pp.folio_remove_rmap_ptes
4.82 -0.2 4.59 perf-profile.self.cycles-pp.zap_present_ptes
2.66 -0.2 2.49 perf-profile.self.cycles-pp.filp_flush
2.36 -0.2 2.20 perf-profile.self.cycles-pp.dnotify_flush
0.20 ± 32% -0.1 0.08 ± 58% perf-profile.self.cycles-pp.queue_event
1.44 -0.1 1.36 perf-profile.self.cycles-pp.zap_pte_range
1.11 -0.1 1.03 perf-profile.self.cycles-pp.anon_vma_interval_tree_insert
1.26 -0.1 1.20 perf-profile.self.cycles-pp.vm_normal_page
0.56 -0.0 0.52 ± 2% perf-profile.self.cycles-pp.dup_fd
0.56 -0.0 0.53 perf-profile.self.cycles-pp.locks_remove_posix
0.31 -0.0 0.28 perf-profile.self.cycles-pp.put_files_struct
0.58 -0.0 0.55 perf-profile.self.cycles-pp.__tlb_remove_folio_pages_size
0.49 -0.0 0.46 ± 2% perf-profile.self.cycles-pp.__anon_vma_interval_tree_remove
0.30 ± 3% -0.0 0.28 ± 3% perf-profile.self.cycles-pp.rwsem_spin_on_owner
0.52 -0.0 0.49 ± 2% perf-profile.self.cycles-pp.clear_page_erms
0.31 -0.0 0.29 perf-profile.self.cycles-pp.free_swap_cache
0.33 -0.0 0.31 perf-profile.self.cycles-pp.__memcg_slab_free_hook
0.45 -0.0 0.43 perf-profile.self.cycles-pp._raw_spin_lock
0.55 -0.0 0.53 perf-profile.self.cycles-pp.fput
0.38 -0.0 0.36 perf-profile.self.cycles-pp.__memcg_slab_post_alloc_hook
0.47 -0.0 0.45 perf-profile.self.cycles-pp.up_write
0.26 -0.0 0.24 perf-profile.self.cycles-pp.__rb_insert_augmented
0.33 -0.0 0.32 perf-profile.self.cycles-pp.mod_objcg_state
0.31 -0.0 0.30 perf-profile.self.cycles-pp.__free_one_page
0.09 -0.0 0.08 perf-profile.self.cycles-pp.___slab_alloc
24.40 +2.1 26.55 perf-profile.self.cycles-pp._compound_head
0.00 +3.9 3.89 perf-profile.self.cycles-pp.folio_try_dup_anon_rmap_ptes

Disclaimer:
Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance.

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#1 lore raw
Quoted 5 lines · kernel test robot

Hello,

kernel test robot noticed a -2.9% regression of stress-ng.clone.ops_per_sec on:

Is that test even using hugetlb? Anyhow, this pretty much sounds like noise and can be ignored.

Signature · 3 lines
Cheers,

David / dhildenb
#2 lore raw

Hi David,

Quoted 9 lines · David Hildenbrand
Quoted 5 lines · kernel test robot

Hello,

kernel test robot noticed a -2.9% regression of stress-ng.clone.ops_per_sec on:

Is that test even using hugetlb? Anyhow, this pretty much sounds like noise and can be ignored.

It's not about hugetlb. It looks like related with the change:

include/linux/page-flags.h +6 -1
index 888353c209c03..7577fe7debafc 100644
@@ -1095,7 +1095,12 @@ PAGEFLAG(Isolated, isolated, PF_ANY);
 static __always_inline int PageAnonExclusive(const struct page *page)
 {
        VM_BUG_ON_PGFLAGS(!PageAnon(page), page);
-       VM_BUG_ON_PGFLAGS(PageHuge(page) && !PageHead(page), page);
+       /*
+        * HugeTLB stores this information on the head page; THP keeps it per
+        * page
+        */
+       if (PageHuge(page))
+               page = compound_head(page);
        return test_bit(PG_anon_exclusive, &PF_ANY(page, 1)->flags);
 
 

The PageAnonExclusive() function is changed. And the profiling data showed it:

      0.00            +3.9        3.90 

perf-profile.calltrace.cycles-pp.folio_try_dup_anon_rmap_ptes.copy_present_ptes.copy_pte_range.copy_p4d_range.copy_page_range

According https://download.01.org/0day-ci/archive/20240730/[email protected]/config-6.9.0-rc4-00197-gc0bff412e67b:
# CONFIG_DEBUG_VM is not set
So maybe such code change could bring difference?

And yes. 2.9% regression can be in noise range. Thanks.

Regards
Yin, Fengwei

#3 lore raw
Quoted 8 lines · Fengwei

Hi David,

Quoted 5 lines · David Hildenbrand
Quoted 4 lines · kernel test robot

Hello,

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

kernel test robot noticed a -2.9% regression of
stress-ng.clone.ops_per_sec on:

Is that test even using hugetlb? Anyhow, this pretty much sounds like noise and can be ignored.

It's not about hugetlb. It looks like related with the change:

Ah, that makes sense!

Quoted 23 lines
include/linux/page-flags.h +2 -1
index 888353c209c03..7577fe7debafc 100644
@@ -1095,7 +1095,12 @@ PAGEFLAG(Isolated, isolated, PF_ANY);
   static __always_inline int PageAnonExclusive(const struct page *page)
   {
          VM_BUG_ON_PGFLAGS(!PageAnon(page), page);
-       VM_BUG_ON_PGFLAGS(PageHuge(page) && !PageHead(page), page);
+       /*
+        * HugeTLB stores this information on the head page; THP keeps

it per

include/linux/page-flags.h +4 -0
+        * page
+        */
+       if (PageHuge(page))
+               page = compound_head(page);
          return test_bit(PG_anon_exclusive, &PF_ANY(page, 1)->flags);

The PageAnonExclusive() function is changed. And the profiling data showed it:

Quoted 7 lines
        0.00            +3.9        3.90

perf-profile.calltrace.cycles-pp.folio_try_dup_anon_rmap_ptes.copy_present_ptes.copy_pte_range.copy_p4d_range.copy_page_range

According
https://download.01.org/0day-ci/archive/20240730/[email protected]/config-6.9.0-rc4-00197-gc0bff412e67b:
# CONFIG_DEBUG_VM is not set
So maybe such code change could bring difference?

Yes indeed. fork() can be extremely sensitive to each added instruction.

I even pointed out to Peter why I didn't add the PageHuge check in there originally [1].

"Well, and I didn't want to have runtime-hugetlb checks in PageAnonExclusive code called on certainly-not-hugetlb code paths."

We now have to do a page_folio(page) and then test for hugetlb.

	return folio_test_hugetlb(page_folio(page));

Nowadays, folio_test_hugetlb() will be faster than at c0bff412e6 times, so maybe at least part of the overhead is gone.

[1] https://lore.kernel.org/r/all/[email protected]/

Signature · 3 lines
Cheers,

David / dhildenb
#4 lore raw

Hi David,

Quoted 5 lines · David Hildenbrand

We now have to do a page_folio(page) and then test for hugetlb.

    return folio_test_hugetlb(page_folio(page));

Nowadays, folio_test_hugetlb() will be faster than at c0bff412e6 times, so maybe at least part of the overhead is gone.

This is great. We will check the trend to know whether it's recovered in some level.

Regards
Yin, Fengwei

#5 lore raw
Quoted 11 lines · Fengwei

Hi David,

Quoted 6 lines · David Hildenbrand

We now have to do a page_folio(page) and then test for hugetlb.

      return folio_test_hugetlb(page_folio(page));

Nowadays, folio_test_hugetlb() will be faster than at c0bff412e6 times, so maybe at least part of the overhead is gone.

This is great. We will check the trend to know whether it's recovered in some level.

Oh, I think d99e3140a4d33e26066183ff727d8f02f56bec64 went upstream before c0bff412e67b781d761e330ff9578aa9ed2be79e, so at the time of c0bff412e6 we already should have had the faster check!

Signature · 3 lines
Cheers,

David / dhildenb
#6 lore raw
In reply to David Hildenbrand
Quoted 9 lines · David Hildenbrand

Yes indeed. fork() can be extremely sensitive to each added instruction.

I even pointed out to Peter why I didn't add the PageHuge check in there originally [1].

"Well, and I didn't want to have runtime-hugetlb checks in PageAnonExclusive code called on certainly-not-hugetlb code paths."

Quoted 7 lines

We now have to do a page_folio(page) and then test for hugetlb.

	return folio_test_hugetlb(page_folio(page));

Nowadays, folio_test_hugetlb() will be faster than at c0bff412e6 times, so maybe at least part of the overhead is gone.

I'll note page_folio expands to a call to _compound_head.

While _compound_head is declared as an inline, it ends up being big enough that the compiler decides to emit a real function instead and real func calls are not particularly cheap.

I had a brief look with a profiler myself and for single-threaded usage the func is quite high up there, while it manages to get out with the first branch -- that is to say there is definitely performance lost for having a func call instead of an inlined branch.

The routine is deinlined because of a call to page_fixed_fake_head, which itself is annotated with always_inline.

This is of course patchable with minor shoveling.

I did not go for it because stress-ng results were too unstable for me to confidently state win/loss.

But should you want to whack the regression, this is what I would look into.

#7 lore raw
Quoted 32 lines · Mateusz Guzik
Quoted 9 lines · David Hildenbrand

Yes indeed. fork() can be extremely sensitive to each added instruction.

I even pointed out to Peter why I didn't add the PageHuge check in there originally [1].

"Well, and I didn't want to have runtime-hugetlb checks in PageAnonExclusive code called on certainly-not-hugetlb code paths."

Quoted 7 lines

We now have to do a page_folio(page) and then test for hugetlb.

	return folio_test_hugetlb(page_folio(page));

Nowadays, folio_test_hugetlb() will be faster than at c0bff412e6 times, so maybe at least part of the overhead is gone.

I'll note page_folio expands to a call to _compound_head.

While _compound_head is declared as an inline, it ends up being big enough that the compiler decides to emit a real function instead and real func calls are not particularly cheap.

I had a brief look with a profiler myself and for single-threaded usage the func is quite high up there, while it manages to get out with the first branch -- that is to say there is definitely performance lost for having a func call instead of an inlined branch.

The routine is deinlined because of a call to page_fixed_fake_head, which itself is annotated with always_inline.

Quoted 8 lines

This is of course patchable with minor shoveling.

I did not go for it because stress-ng results were too unstable for me to confidently state win/loss.

But should you want to whack the regression, this is what I would look into.

This might improve it, at least for small folios I guess:

include/linux/page-flags.h +1 -1
index 5769fe6e4950..7796ae116018 100644
@@ -1086,7 +1086,7 @@ PAGE_TYPE_OPS(Zsmalloc, zsmalloc, zsmalloc)
  */
 static inline bool PageHuge(const struct page *page)
 {
-       return folio_test_hugetlb(page_folio(page));
+       return PageCompound(page) && folio_test_hugetlb(page_folio(page));
 }
 
 /*

We would avoid the function call for small folios.

Signature · 3 lines
Cheers,

David / dhildenb
#8 lore raw
Quoted 60 lines · David Hildenbrand
Quoted 32 lines · Mateusz Guzik
Quoted 9 lines · David Hildenbrand

Yes indeed. fork() can be extremely sensitive to each added instruction.

I even pointed out to Peter why I didn't add the PageHuge check in there originally [1].

"Well, and I didn't want to have runtime-hugetlb checks in PageAnonExclusive code called on certainly-not-hugetlb code paths."

Quoted 7 lines

We now have to do a page_folio(page) and then test for hugetlb.

     return folio_test_hugetlb(page_folio(page));

Nowadays, folio_test_hugetlb() will be faster than at c0bff412e6 times, so maybe at least part of the overhead is gone.

I'll note page_folio expands to a call to _compound_head.

While _compound_head is declared as an inline, it ends up being big enough that the compiler decides to emit a real function instead and real func calls are not particularly cheap.

I had a brief look with a profiler myself and for single-threaded usage the func is quite high up there, while it manages to get out with the first branch -- that is to say there is definitely performance lost for having a func call instead of an inlined branch.

The routine is deinlined because of a call to page_fixed_fake_head, which itself is annotated with always_inline.

Quoted 8 lines

This is of course patchable with minor shoveling.

I did not go for it because stress-ng results were too unstable for me to confidently state win/loss.

But should you want to whack the regression, this is what I would look into.

This might improve it, at least for small folios I guess:

include/linux/page-flags.h +1 -1
index 5769fe6e4950..7796ae116018 100644
@@ -1086,7 +1086,7 @@ PAGE_TYPE_OPS(Zsmalloc, zsmalloc, zsmalloc)
   */
  static inline bool PageHuge(const struct page *page)
  {
-       return folio_test_hugetlb(page_folio(page));
+       return PageCompound(page) && folio_test_hugetlb(page_folio(page));
  }
 
  /*
Quoted 2 lines

We would avoid the function call for small folios.

why not massage _compound_head back to an inlineable form instead? for all i know you may even register a small win in total

Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
#9 lore raw
Quoted 58 lines · Mateusz Guzik
Quoted 57 lines · David Hildenbrand
Quoted 32 lines · Mateusz Guzik
Quoted 9 lines · David Hildenbrand

Yes indeed. fork() can be extremely sensitive to each added instruction.

I even pointed out to Peter why I didn't add the PageHuge check in there originally [1].

"Well, and I didn't want to have runtime-hugetlb checks in PageAnonExclusive code called on certainly-not-hugetlb code paths."

Quoted 7 lines

We now have to do a page_folio(page) and then test for hugetlb.

      return folio_test_hugetlb(page_folio(page));

Nowadays, folio_test_hugetlb() will be faster than at c0bff412e6 times, so maybe at least part of the overhead is gone.

I'll note page_folio expands to a call to _compound_head.

While _compound_head is declared as an inline, it ends up being big enough that the compiler decides to emit a real function instead and real func calls are not particularly cheap.

I had a brief look with a profiler myself and for single-threaded usage the func is quite high up there, while it manages to get out with the first branch -- that is to say there is definitely performance lost for having a func call instead of an inlined branch.

The routine is deinlined because of a call to page_fixed_fake_head, which itself is annotated with always_inline.

Quoted 8 lines

This is of course patchable with minor shoveling.

I did not go for it because stress-ng results were too unstable for me to confidently state win/loss.

But should you want to whack the regression, this is what I would look into.

This might improve it, at least for small folios I guess:

include/linux/page-flags.h +1 -1
index 5769fe6e4950..7796ae116018 100644
@@ -1086,7 +1086,7 @@ PAGE_TYPE_OPS(Zsmalloc, zsmalloc, zsmalloc)
    */
   static inline bool PageHuge(const struct page *page)
   {
-       return folio_test_hugetlb(page_folio(page));
+       return PageCompound(page) && folio_test_hugetlb(page_folio(page));
   }
 
Quoted 8 lines
Quoted 5 lines
include/linux/page-flags.h +0 -0
   /*

We would avoid the function call for small folios.

why not massage _compound_head back to an inlineable form instead? for all i know you may even register a small win in total

Agreed, likely it will increase code size a bit which is why the compiler decides to not inline. We could force it with __always_inline.

Finding ways to shrink page_fixed_fake_head() might be even better.

Signature · 3 lines
Cheers,

David / dhildenb
#10 lore raw

Hi David,

Quoted 38 lines · David Hildenbrand
Quoted 37 lines · Mateusz Guzik
Quoted 36 lines · David Hildenbrand
Quoted 34 lines · Mateusz Guzik
Quoted 9 lines · David Hildenbrand

Yes indeed. fork() can be extremely sensitive to each added instruction.

I even pointed out to Peter why I didn't add the PageHuge check in there originally [1].

"Well, and I didn't want to have runtime-hugetlb checks in PageAnonExclusive code called on certainly-not-hugetlb code paths."

Quoted 7 lines

We now have to do a page_folio(page) and then test for hugetlb.

      return folio_test_hugetlb(page_folio(page));

Nowadays, folio_test_hugetlb() will be faster than at c0bff412e6 times, so maybe at least part of the overhead is gone.

I'll note page_folio expands to a call to _compound_head.

While _compound_head is declared as an inline, it ends up being big enough that the compiler decides to emit a real function instead and real func calls are not particularly cheap.

I had a brief look with a profiler myself and for single-threaded usage the func is quite high up there, while it manages to get out with the first branch -- that is to say there is definitely performance lost for having a func call instead of an inlined branch.

The routine is deinlined because of a call to page_fixed_fake_head, which itself is annotated with always_inline.

This is of course patchable with minor shoveling.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

I did not go for it because stress-ng results were too unstable for me to confidently state win/loss.

But should you want to whack the regression, this is what I would look into.

This might improve it, at least for small folios I guess:

Do you want us to test this change? Or you have further optimization ongoing? Thanks.

Regards
Yin, Fengwei

Quoted 27 lines
Quoted 14 lines
Quoted 14 lines
include/linux/page-flags.h +1 -1
index 5769fe6e4950..7796ae116018 100644
@@ -1086,7 +1086,7 @@ PAGE_TYPE_OPS(Zsmalloc, zsmalloc, zsmalloc)
     */
    static inline bool PageHuge(const struct page *page)
    {
-       return folio_test_hugetlb(page_folio(page));
+       return PageCompound(page) && 
 folio_test_hugetlb(page_folio(page));
    }
 
Quoted 8 lines
Quoted 5 lines

   /*

We would avoid the function call for small folios.

why not massage _compound_head back to an inlineable form instead? for all i know you may even register a small win in total

Agreed, likely it will increase code size a bit which is why the compiler decides to not inline. We could force it with __always_inline.

Finding ways to shrink page_fixed_fake_head() might be even better.

#11 lore raw
Quoted 48 lines · Yin Fengwei

Hi David,

Quoted 45 lines · David Hildenbrand
Quoted 44 lines · Mateusz Guzik
Quoted 42 lines · David Hildenbrand
Quoted 40 lines · Mateusz Guzik
Quoted 11 lines · David Hildenbrand

Yes indeed. fork() can be extremely sensitive to each added instruction.

I even pointed out to Peter why I didn't add the
PageHuge check in there
originally [1].

"Well, and I didn't want to have runtime-hugetlb checks in PageAnonExclusive code called on certainly-not-hugetlb code paths."

Quoted 8 lines

We now have to do a page_folio(page) and then test for hugetlb.

      return folio_test_hugetlb(page_folio(page));

Nowadays, folio_test_hugetlb() will be faster than at c0bff412e6 times, so
maybe at least part of the overhead is gone.

I'll note page_folio expands to a call to _compound_head.

While _compound_head is declared as an inline, it ends up being big enough that the compiler decides to emit a real function instead and real func calls are not particularly cheap.

I had a brief look with a profiler myself and for single-threaded usage the func is quite high up there, while it manages to get out with the first branch -- that is to say there is definitely performance lost for having a func call instead of an inlined branch.

The routine is deinlined because of a call to page_fixed_fake_head, which itself is annotated with always_inline.

This is of course patchable with minor shoveling.

I did not go for it because stress-ng results were too unstable for me to confidently state win/loss.

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

But should you want to whack the regression, this is what I would look into.

This might improve it, at least for small folios I guess:

Do you want us to test this change? Or you have further optimization ongoing? Thanks.

I verified the thing below boots, I have no idea about performance. If it helps it can be massaged later from style perspective.

include/linux/page-flags.h +4 -25
index 5769fe6e4950..2d5d61ab385b 100644
@@ -194,34 +194,13 @@ enum pageflags {
 #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP
 DECLARE_STATIC_KEY_FALSE(hugetlb_optimize_vmemmap_key);
 
-/*
- * Return the real head page struct iff the @page is a fake head page, otherwise
- * return the @page itself. See Documentation/mm/vmemmap_dedup.rst.
- */
+const struct page *_page_fixed_fake_head(const struct page *page);
+
 static __always_inline const struct page *page_fixed_fake_head(const struct page *page)
 {
 	if (!static_branch_unlikely(&hugetlb_optimize_vmemmap_key))
 		return page;
-
-	/*
-	 * Only addresses aligned with PAGE_SIZE of struct page may be fake head
-	 * struct page. The alignment check aims to avoid access the fields (
-	 * e.g. compound_head) of the @page[1]. It can avoid touch a (possibly)
-	 * cold cacheline in some cases.
-	 */
-	if (IS_ALIGNED((unsigned long)page, PAGE_SIZE) &&
-	    test_bit(PG_head, &page->flags)) {
-		/*
-		 * We can safely access the field of the @page[1] with PG_head
-		 * because the @page is a compound page composed with at least
-		 * two contiguous pages.
-		 */
-		unsigned long head = READ_ONCE(page[1].compound_head);
-
-		if (likely(head & 1))
-			return (const struct page *)(head - 1);
-	}
-	return page;
+	return _page_fixed_fake_head(page);
 }
 #else
 static inline const struct page *page_fixed_fake_head(const struct page *page)
@@ -235,7 +214,7 @@ static __always_inline int page_is_fake_head(const struct page *page)
 	return page_fixed_fake_head(page) != page;
 }
 
-static inline unsigned long _compound_head(const struct page *page)
+static __always_inline unsigned long _compound_head(const struct page *page)
 {
 	unsigned long head = READ_ONCE(page->compound_head);
 
mm/hugetlb_vmemmap.c +27 -0
index 829112b0a914..3fbc00db607a 100644
@@ -19,6 +19,33 @@
 #include <asm/tlbflush.h>
 #include "hugetlb_vmemmap.h"
 
+/*
+ * Return the real head page struct iff the @page is a fake head page, otherwise
+ * return the @page itself. See Documentation/mm/vmemmap_dedup.rst.
+ */
+const struct page *_page_fixed_fake_head(const struct page *page)
+{
+	/*
+	 * Only addresses aligned with PAGE_SIZE of struct page may be fake head
+	 * struct page. The alignment check aims to avoid access the fields (
+	 * e.g. compound_head) of the @page[1]. It can avoid touch a (possibly)
+	 * cold cacheline in some cases.
+	 */
+	if (IS_ALIGNED((unsigned long)page, PAGE_SIZE) &&
+	    test_bit(PG_head, &page->flags)) {
+		/*
+		 * We can safely access the field of the @page[1] with PG_head
+		 * because the @page is a compound page composed with at least
+		 * two contiguous pages.
+		 */
+		unsigned long head = READ_ONCE(page[1].compound_head);
+
+		if (likely(head & 1))
+			return (const struct page *)(head - 1);
+	}
+	return page;
+}
+
 /**
  * struct vmemmap_remap_walk - walk vmemmap page table
  *
#12 lore raw
Quoted 53 lines · Mateusz Guzik
Quoted 52 lines · Yin Fengwei

Hi David,

Quoted 49 lines · David Hildenbrand
Quoted 48 lines · Mateusz Guzik
Quoted 46 lines · David Hildenbrand
Quoted 43 lines · Mateusz Guzik
Quoted 11 lines · David Hildenbrand

Yes indeed. fork() can be extremely sensitive to each added instruction.

I even pointed out to Peter why I didn't add the
PageHuge check in there
originally [1].

"Well, and I didn't want to have runtime-hugetlb checks in PageAnonExclusive code called on certainly-not-hugetlb code paths."

Quoted 8 lines

We now have to do a page_folio(page) and then test for hugetlb.

      return folio_test_hugetlb(page_folio(page));

Nowadays, folio_test_hugetlb() will be faster than at c0bff412e6 times, so
maybe at least part of the overhead is gone.

I'll note page_folio expands to a call to _compound_head.

While _compound_head is declared as an inline, it ends up being big enough that the compiler decides to emit a real function instead and real func calls are not particularly cheap.

I had a brief look with a profiler myself and for single-threaded usage the func is quite high up there, while it manages to get out with the first branch -- that is to say there is definitely performance lost for having a func call instead of an inlined branch.

The routine is deinlined because of a call to page_fixed_fake_head, which itself is annotated with always_inline.

This is of course patchable with minor shoveling.

I did not go for it because stress-ng results were too unstable for me to confidently state win/loss.

But should you want to whack the regression, this is what I would look into.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

This might improve it, at least for small folios I guess:

Do you want us to test this change? Or you have further optimization ongoing? Thanks.

I verified the thing below boots, I have no idea about performance. If it helps it can be massaged later from style perspective.

As quite a lot of setups already run with the vmemmap optimization enabled, I wonder how effective this would be (might need more fine tuning, did not look at the generated code):

include/linux/page-flags.h +1 -1
index 085dd8dcbea2..7ddcdbd712ec 100644
@@ -233,7 +233,7 @@ static __always_inline int page_is_fake_head(const struct page *page)
        return page_fixed_fake_head(page) != page;
 }
 
-static inline unsigned long _compound_head(const struct page *page)
+static __always_inline unsigned long _compound_head(const struct page *page)
 {
        unsigned long head = READ_ONCE(page->compound_head);
 
Signature · 3 lines
Cheers,

David / dhildenb
#13 lore raw
Quoted 73 lines · David Hildenbrand
Quoted 53 lines · Mateusz Guzik
Quoted 52 lines · Yin Fengwei

Hi David,

Quoted 49 lines · David Hildenbrand
Quoted 48 lines · Mateusz Guzik
Quoted 46 lines · David Hildenbrand
Quoted 43 lines · Mateusz Guzik
Quoted 11 lines · David Hildenbrand

Yes indeed. fork() can be extremely sensitive to each added instruction.

I even pointed out to Peter why I didn't add the
PageHuge check in there
originally [1].

"Well, and I didn't want to have runtime-hugetlb checks in PageAnonExclusive code called on certainly-not-hugetlb code paths."

Quoted 8 lines

We now have to do a page_folio(page) and then test for hugetlb.

       return folio_test_hugetlb(page_folio(page));

Nowadays, folio_test_hugetlb() will be faster than at c0bff412e6 times, so
maybe at least part of the overhead is gone.

I'll note page_folio expands to a call to _compound_head.

While _compound_head is declared as an inline, it ends up being big enough that the compiler decides to emit a real function instead and real func calls are not particularly cheap.

I had a brief look with a profiler myself and for single-threaded usage the func is quite high up there, while it manages to get out with the first branch -- that is to say there is definitely performance lost for having a func call instead of an inlined branch.

The routine is deinlined because of a call to page_fixed_fake_head, which itself is annotated with always_inline.

This is of course patchable with minor shoveling.

I did not go for it because stress-ng results were too unstable for me to confidently state win/loss.

But should you want to whack the regression, this is what I would look into.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

This might improve it, at least for small folios I guess:

Do you want us to test this change? Or you have further optimization ongoing? Thanks.

I verified the thing below boots, I have no idea about performance. If it helps it can be massaged later from style perspective.

As quite a lot of setups already run with the vmemmap optimization enabled, I wonder how effective this would be (might need more fine tuning, did not look at the generated code):

include/linux/page-flags.h +0 -0
index 085dd8dcbea2..7ddcdbd712ec 100644

include/linux/page-flags.h static __always_inline int page_is_fake_head(const struct page *page) · line 239

Quoted 8 lines
include/linux/page-flags.h +1 -1
@@ -233,7 +233,7 @@ static __always_inline int page_is_fake_head(const struct page *page)
         return page_fixed_fake_head(page) != page;
  }
 
-static inline unsigned long _compound_head(const struct page *page)
+static __always_inline unsigned long _compound_head(const struct page *page)
  {
         unsigned long head = READ_ONCE(page->compound_head);
 

Well one may need to justify it with bloat-o-meter which is why I did not just straight up inline the entire thing.

But if you are down to fight opposition of the sort I agree this is the patch to benchmark. :)

Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
#14 lore raw
Quoted 86 lines · Mateusz Guzik
Quoted 73 lines · David Hildenbrand
Quoted 53 lines · Mateusz Guzik
Quoted 52 lines · Yin Fengwei

Hi David,

Quoted 49 lines · David Hildenbrand
Quoted 48 lines · Mateusz Guzik
Quoted 46 lines · David Hildenbrand
Quoted 43 lines · Mateusz Guzik
Quoted 11 lines · David Hildenbrand

Yes indeed. fork() can be extremely sensitive to each added instruction.

I even pointed out to Peter why I didn't add the
PageHuge check in there
originally [1].

"Well, and I didn't want to have runtime-hugetlb checks in PageAnonExclusive code called on certainly-not-hugetlb code paths."

Quoted 8 lines

We now have to do a page_folio(page) and then test for hugetlb.

        return folio_test_hugetlb(page_folio(page));

Nowadays, folio_test_hugetlb() will be faster than at c0bff412e6 times, so
maybe at least part of the overhead is gone.

I'll note page_folio expands to a call to _compound_head.

While _compound_head is declared as an inline, it ends up being big enough that the compiler decides to emit a real function instead and real func calls are not particularly cheap.

I had a brief look with a profiler myself and for single-threaded usage the func is quite high up there, while it manages to get out with the first branch -- that is to say there is definitely performance lost for having a func call instead of an inlined branch.

The routine is deinlined because of a call to page_fixed_fake_head, which itself is annotated with always_inline.

This is of course patchable with minor shoveling.

I did not go for it because stress-ng results were too unstable for me to confidently state win/loss.

But should you want to whack the regression, this is what I would look into.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

This might improve it, at least for small folios I guess:

Do you want us to test this change? Or you have further optimization ongoing? Thanks.

I verified the thing below boots, I have no idea about performance. If it helps it can be massaged later from style perspective.

As quite a lot of setups already run with the vmemmap optimization enabled, I wonder how effective this would be (might need more fine tuning, did not look at the generated code):

include/linux/page-flags.h +0 -0
index 085dd8dcbea2..7ddcdbd712ec 100644

include/linux/page-flags.h static __always_inline int page_is_fake_head(const struct page *page) · line 239

Quoted 8 lines
include/linux/page-flags.h +1 -1
@@ -233,7 +233,7 @@ static __always_inline int page_is_fake_head(const struct page *page)
          return page_fixed_fake_head(page) != page;
   }
 
-static inline unsigned long _compound_head(const struct page *page)
+static __always_inline unsigned long _compound_head(const struct page *page)
   {
          unsigned long head = READ_ONCE(page->compound_head);
 

Well one may need to justify it with bloat-o-meter which is why I did not just straight up inline the entire thing.

Quoted 2 lines

But if you are down to fight opposition of the sort I agree this is the patch to benchmark. :)

I spotted that we already to that for PageHead()/PageTail()/page_is_fake_head(). So we effectively force-inline it everywhere except into _compound_head() I think.

But yeah, measuring the bloat would be a necessary exercise.

Signature · 3 lines
Cheers,

David / dhildenb
#15 lore raw
In reply to Mateusz Guzik
Quoted 140 lines · Mateusz Guzik
Quoted 48 lines · Yin Fengwei

Hi David,

Quoted 45 lines · David Hildenbrand
Quoted 44 lines · Mateusz Guzik
Quoted 42 lines · David Hildenbrand
Quoted 40 lines · Mateusz Guzik
Quoted 11 lines · David Hildenbrand

Yes indeed. fork() can be extremely sensitive to each added instruction.

I even pointed out to Peter why I didn't add the
PageHuge check in there
originally [1].

"Well, and I didn't want to have runtime-hugetlb checks in PageAnonExclusive code called on certainly-not-hugetlb code paths."

Quoted 8 lines

We now have to do a page_folio(page) and then test for hugetlb.

      return folio_test_hugetlb(page_folio(page));

Nowadays, folio_test_hugetlb() will be faster than at c0bff412e6 times, so
maybe at least part of the overhead is gone.

I'll note page_folio expands to a call to _compound_head.

While _compound_head is declared as an inline, it ends up being big enough that the compiler decides to emit a real function instead and real func calls are not particularly cheap.

I had a brief look with a profiler myself and for single-threaded usage the func is quite high up there, while it manages to get out with the first branch -- that is to say there is definitely performance lost for having a func call instead of an inlined branch.

The routine is deinlined because of a call to page_fixed_fake_head, which itself is annotated with always_inline.

This is of course patchable with minor shoveling.

I did not go for it because stress-ng results were too unstable for me to confidently state win/loss.

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

But should you want to whack the regression, this is what I would look into.

This might improve it, at least for small folios I guess:

Do you want us to test this change? Or you have further optimization ongoing? Thanks.

I verified the thing below boots, I have no idea about performance. If it helps it can be massaged later from style perspective.

include/linux/page-flags.h +4 -25
index 5769fe6e4950..2d5d61ab385b 100644
@@ -194,34 +194,13 @@ enum pageflags {
  #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP
  DECLARE_STATIC_KEY_FALSE(hugetlb_optimize_vmemmap_key);
 
-/*
- * Return the real head page struct iff the @page is a fake head page, otherwise
- * return the @page itself. See Documentation/mm/vmemmap_dedup.rst.
- */
+const struct page *_page_fixed_fake_head(const struct page *page);
+
  static __always_inline const struct page *page_fixed_fake_head(const struct page *page)
  {
  	if (!static_branch_unlikely(&hugetlb_optimize_vmemmap_key))
  		return page;
-
-	/*
-	 * Only addresses aligned with PAGE_SIZE of struct page may be fake head
-	 * struct page. The alignment check aims to avoid access the fields (
-	 * e.g. compound_head) of the @page[1]. It can avoid touch a (possibly)
-	 * cold cacheline in some cases.
-	 */
-	if (IS_ALIGNED((unsigned long)page, PAGE_SIZE) &&
-	    test_bit(PG_head, &page->flags)) {
-		/*
-		 * We can safely access the field of the @page[1] with PG_head
-		 * because the @page is a compound page composed with at least
-		 * two contiguous pages.
-		 */
-		unsigned long head = READ_ONCE(page[1].compound_head);
-
-		if (likely(head & 1))
-			return (const struct page *)(head - 1);
-	}
-	return page;
+	return _page_fixed_fake_head(page);
  }
  #else
  static inline const struct page *page_fixed_fake_head(const struct page *page)
@@ -235,7 +214,7 @@ static __always_inline int page_is_fake_head(const struct page *page)
  	return page_fixed_fake_head(page) != page;
  }
 
-static inline unsigned long _compound_head(const struct page *page)
+static __always_inline unsigned long _compound_head(const struct page *page)
  {
  	unsigned long head = READ_ONCE(page->compound_head);
 
mm/hugetlb_vmemmap.c +20 -0
index 829112b0a914..3fbc00db607a 100644
@@ -19,6 +19,33 @@
  #include <asm/tlbflush.h>
  #include "hugetlb_vmemmap.h"
 
+/*
+ * Return the real head page struct iff the @page is a fake head page, otherwise
+ * return the @page itself. See Documentation/mm/vmemmap_dedup.rst.
+ */
+const struct page *_page_fixed_fake_head(const struct page *page)
+{
+	/*
+	 * Only addresses aligned with PAGE_SIZE of struct page may be fake head
+	 * struct page. The alignment check aims to avoid access the fields (
+	 * e.g. compound_head) of the @page[1]. It can avoid touch a (possibly)
+	 * cold cacheline in some cases.
+	 */
+	if (IS_ALIGNED((unsigned long)page, PAGE_SIZE) &&
+	    test_bit(PG_head, &page->flags)) {
+		/*
+		 * We can safely access the field of the @page[1] with PG_head
+		 * because the @page is a compound page composed with at least
+		 * two contiguous pages.
+		 */
+		unsigned long head = READ_ONCE(page[1].compound_head);

mm/hugetlb_vmemmap.c line 51

Quoted 10 lines
mm/hugetlb_vmemmap.c +7 -0
+
+		if (likely(head & 1))
+			return (const struct page *)(head - 1);
+	}
+	return page;
+}
+
  /**
   * struct vmemmap_remap_walk - walk vmemmap page table
   *

The change can resolve the regression (from -3% to 0.5%):

Please note:
9cb28da54643ad464c47585cd5866c30b0218e67 is the parent commit 3f16e4b516ef02d9461b7e0b6c50e05ba0811886 is the commit with above
patch
c0bff412e67b781d761e330ff9578aa9ed2be79e is the commit which
introduced regression

=========================================================================================
tbox_group/testcase/rootfs/kconfig/compiler/nr_threads/testtime/test/cpufreq_governor/debug-setup:

lkp-icl-2sp8/stress-ng/debian-12-x86_64-20240206.cgz/x86_64-rhel-8.3/gcc-12/100%/60s/clone/performance/yfw_test2

commit:
9cb28da54643ad464c47585cd5866c30b0218e67
3f16e4b516ef02d9461b7e0b6c50e05ba0811886
c0bff412e67b781d761e330ff9578aa9ed2be79e

9cb28da54643ad46 3f16e4b516ef02d9461b7e0b6c5 c0bff412e67b781d761e330ff95 ---------------- --------------------------- ---------------------------
fail:runs %reproduction fail:runs %reproduction fail:runs
| | | | |
3:3 0% 3:3 0% 3:3
stress-ng.clone.microsecs_per_clone.pass
3:3 0% 3:3 0% 3:3
stress-ng.clone.pass

         %stddev     %change         %stddev     %change         %stddev
             \          |                \          |                \
      2904            -0.6%       2886            +3.7%       3011 
  stress-ng.clone.microsecs_per_clone
    563520            +0.5%     566296            -3.1%     546122 
  stress-ng.clone.ops
      9306            +0.5%       9356            -3.0%       9024 

stress-ng.clone.ops_per_sec

BTW, the change needs to export symbol _page_fixed_fake_head otherwise some modules hit build error.

Regards
Yin, Fengwei

#16 lore raw
Quoted 146 lines · Yin Fengwei
Quoted 144 lines · Mateusz Guzik
Quoted 48 lines · Yin Fengwei

Hi David,

Quoted 45 lines · David Hildenbrand
Quoted 44 lines · Mateusz Guzik
Quoted 42 lines · David Hildenbrand
Quoted 40 lines · Mateusz Guzik
Quoted 11 lines · David Hildenbrand

Yes indeed. fork() can be extremely sensitive to each added instruction.

I even pointed out to Peter why I didn't add the
PageHuge check in there
originally [1].

"Well, and I didn't want to have runtime-hugetlb checks in PageAnonExclusive code called on certainly-not-hugetlb code paths."

Quoted 8 lines

We now have to do a page_folio(page) and then test for hugetlb.

       return folio_test_hugetlb(page_folio(page));

Nowadays, folio_test_hugetlb() will be faster than at c0bff412e6 times, so
maybe at least part of the overhead is gone.

I'll note page_folio expands to a call to _compound_head.

While _compound_head is declared as an inline, it ends up being big enough that the compiler decides to emit a real function instead and real func calls are not particularly cheap.

I had a brief look with a profiler myself and for single-threaded usage the func is quite high up there, while it manages to get out with the first branch -- that is to say there is definitely performance lost for having a func call instead of an inlined branch.

The routine is deinlined because of a call to page_fixed_fake_head, which itself is annotated with always_inline.

This is of course patchable with minor shoveling.

I did not go for it because stress-ng results were too unstable for me to confidently state win/loss.

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

But should you want to whack the regression, this is what I would look into.

This might improve it, at least for small folios I guess:

Do you want us to test this change? Or you have further optimization ongoing? Thanks.

I verified the thing below boots, I have no idea about performance. If it helps it can be massaged later from style perspective.

include/linux/page-flags.h +4 -25
index 5769fe6e4950..2d5d61ab385b 100644
@@ -194,34 +194,13 @@ enum pageflags {
  #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP
  DECLARE_STATIC_KEY_FALSE(hugetlb_optimize_vmemmap_key);
 
-/*
- * Return the real head page struct iff the @page is a fake head page, otherwise
- * return the @page itself. See Documentation/mm/vmemmap_dedup.rst.
- */
+const struct page *_page_fixed_fake_head(const struct page *page);
+
  static __always_inline const struct page *page_fixed_fake_head(const struct page *page)
  {
      if (!static_branch_unlikely(&hugetlb_optimize_vmemmap_key))
              return page;
-
-     /*
-      * Only addresses aligned with PAGE_SIZE of struct page may be fake head
-      * struct page. The alignment check aims to avoid access the fields (
-      * e.g. compound_head) of the @page[1]. It can avoid touch a (possibly)
-      * cold cacheline in some cases.
-      */
-     if (IS_ALIGNED((unsigned long)page, PAGE_SIZE) &&
-         test_bit(PG_head, &page->flags)) {
-             /*
-              * We can safely access the field of the @page[1] with PG_head
-              * because the @page is a compound page composed with at least
-              * two contiguous pages.
-              */
-             unsigned long head = READ_ONCE(page[1].compound_head);
-
-             if (likely(head & 1))
-                     return (const struct page *)(head - 1);
-     }
-     return page;
+     return _page_fixed_fake_head(page);
  }
  #else
  static inline const struct page *page_fixed_fake_head(const struct page *page)
@@ -235,7 +214,7 @@ static __always_inline int page_is_fake_head(const struct page *page)
      return page_fixed_fake_head(page) != page;
  }
 
-static inline unsigned long _compound_head(const struct page *page)
+static __always_inline unsigned long _compound_head(const struct page *page)
  {
      unsigned long head = READ_ONCE(page->compound_head);
 
mm/hugetlb_vmemmap.c +25 -0
index 829112b0a914..3fbc00db607a 100644
@@ -19,6 +19,33 @@
  #include <asm/tlbflush.h>
  #include "hugetlb_vmemmap.h"
 
+/*
+ * Return the real head page struct iff the @page is a fake head page, otherwise
+ * return the @page itself. See Documentation/mm/vmemmap_dedup.rst.
+ */
+const struct page *_page_fixed_fake_head(const struct page *page)
+{
+     /*
+      * Only addresses aligned with PAGE_SIZE of struct page may be fake head
+      * struct page. The alignment check aims to avoid access the fields (
+      * e.g. compound_head) of the @page[1]. It can avoid touch a (possibly)
+      * cold cacheline in some cases.
+      */
+     if (IS_ALIGNED((unsigned long)page, PAGE_SIZE) &&
+         test_bit(PG_head, &page->flags)) {
+             /*
+              * We can safely access the field of the @page[1] with PG_head
+              * because the @page is a compound page composed with at least
+              * two contiguous pages.
+              */
+             unsigned long head = READ_ONCE(page[1].compound_head);
+
+             if (likely(head & 1))
+                     return (const struct page *)(head - 1);
+     }
+     return page;
Quoted 9 lines

mm/hugetlb_vmemmap.c line 51

Quoted 6 lines
mm/hugetlb_vmemmap.c +2 -0
+}
+
  /**
   * struct vmemmap_remap_walk - walk vmemmap page table
   *

The change can resolve the regression (from -3% to 0.5%):

thanks for testing

would you mind benchmarking the change which merely force-inlines _compund_page?

https://lore.kernel.org/linux-mm/[email protected]/

Quoted 33 lines

Please note:

   9cb28da54643ad464c47585cd5866c30b0218e67 is the parent commit
   3f16e4b516ef02d9461b7e0b6c50e05ba0811886 is the commit with above
                                            patch
   c0bff412e67b781d761e330ff9578aa9ed2be79e is the commit which
                                            introduced regression

=========================================================================================
tbox_group/testcase/rootfs/kconfig/compiler/nr_threads/testtime/test/cpufreq_governor/debug-setup:

lkp-icl-2sp8/stress-ng/debian-12-x86_64-20240206.cgz/x86_64-rhel-8.3/gcc-12/100%/60s/clone/performance/yfw_test2

commit:

   9cb28da54643ad464c47585cd5866c30b0218e67
   3f16e4b516ef02d9461b7e0b6c50e05ba0811886
   c0bff412e67b781d761e330ff9578aa9ed2be79e

9cb28da54643ad46 3f16e4b516ef02d9461b7e0b6c5 c0bff412e67b781d761e330ff95 ---------------- --------------------------- ---------------------------
fail:runs %reproduction fail:runs %reproduction fail:runs
| | | | |
3:3 0% 3:3 0% 3:3
stress-ng.clone.microsecs_per_clone.pass
3:3 0% 3:3 0% 3:3
stress-ng.clone.pass

          %stddev     %change         %stddev     %change         %stddev
              \          |                \          |                \
       2904            -0.6%       2886            +3.7%       3011
   stress-ng.clone.microsecs_per_clone
     563520            +0.5%     566296            -3.1%     546122
   stress-ng.clone.ops
       9306            +0.5%       9356            -3.0%       9024
Quoted 6 lines

stress-ng.clone.ops_per_sec

BTW, the change needs to export symbol _page_fixed_fake_head otherwise some modules hit build error.

ok, I'll patch that up if this approach will be the thing to do

Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
#17 lore raw
Quoted 5 lines · Mateusz Guzik

thanks for testing

would you mind benchmarking the change which merely force-inlines _compund_page?

https://lore.kernel.org/linux-mm/[email protected]/

This change can resolve the regression also:
=========================================================================================
tbox_group/testcase/rootfs/kconfig/compiler/nr_threads/testtime/test/cpufreq_governor/debug-setup:

lkp-icl-2sp8/stress-ng/debian-12-x86_64-20240206.cgz/x86_64-rhel-8.3/gcc-12/100%/60s/clone/performance/yfw_test2

commit:

  9cb28da54643ad464c47585cd5866c30b0218e67  parent commit
  c0bff412e67b781d761e330ff9578aa9ed2be79e  commit introduced regression
  450b96d2c4f740152e03c6b79b484a10347b3ea9  the change proposed by David

in above link

9cb28da54643ad46 c0bff412e67b781d761e330ff95 450b96d2c4f740152e03c6b79b4 ---------------- --------------------------- ---------------------------

         %stddev     %change         %stddev     %change         %stddev
             \          |                \          |                \
      2906            +3.5%       3007            +0.4%       2919 
  stress-ng.clone.microsecs_per_clone
    562884            -2.9%     546575            -0.6%     559718 
  stress-ng.clone.ops
      9295            -2.9%       9028            -0.5%       9248 

stress-ng.clone.ops_per_sec

Regards
Yin, Fengwei

#18 lore raw
Quoted 6 lines · Yin Fengwei
Quoted 3 lines · Mateusz Guzik

would you mind benchmarking the change which merely force-inlines _compund_page?

https://lore.kernel.org/linux-mm/[email protected]/

This change can resolve the regression also:

Great, thanks.

David, I guess this means it would be fine to inline the entire thing at least from this bench standpoint. Given that this is your idea I guess you should do the needful(tm)? :)

Quoted 27 lines

=========================================================================================
tbox_group/testcase/rootfs/kconfig/compiler/nr_threads/testtime/test/cpufreq_governor/debug-setup:

lkp-icl-2sp8/stress-ng/debian-12-x86_64-20240206.cgz/x86_64-rhel-8.3/gcc-12/100%/60s/clone/performance/yfw_test2

commit:

   9cb28da54643ad464c47585cd5866c30b0218e67  parent commit
   c0bff412e67b781d761e330ff9578aa9ed2be79e  commit introduced regression
   450b96d2c4f740152e03c6b79b484a10347b3ea9  the change proposed by David

in above link

9cb28da54643ad46 c0bff412e67b781d761e330ff95 450b96d2c4f740152e03c6b79b4 ---------------- --------------------------- ---------------------------

          %stddev     %change         %stddev     %change         %stddev
              \          |                \          |                \
       2906            +3.5%       3007            +0.4%       2919
   stress-ng.clone.microsecs_per_clone
     562884            -2.9%     546575            -0.6%     559718
   stress-ng.clone.ops
       9295            -2.9%       9028            -0.5%       9248

stress-ng.clone.ops_per_sec

Regards
Yin, Fengwei

Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
#19 lore raw
Quoted 5 lines · Mateusz Guzik
Quoted 4 lines · Yin Fengwei
Quoted 2 lines · Mateusz Guzik

would you mind benchmarking the change which merely force-inlines _compund_page?

Quoted 8 lines
Quoted 2 lines
Quoted 1 line

https://lore.kernel.org/linux-mm/[email protected]/

This change can resolve the regression also:

Great, thanks.

David, I guess this means it would be fine to inline the entire thing at least from this bench standpoint. Given that this is your idea I guess you should do the needful(tm)? :)

Testing

include/linux/page-flags.h +1 -1
index 5769fe6e4950..25e25b34f4a0 100644
@@ -235,7 +235,7 @@ static __always_inline int page_is_fake_head(const struct page *page)
        return page_fixed_fake_head(page) != page;
 }
 
-static inline unsigned long _compound_head(const struct page *page)
+static __always_inline unsigned long _compound_head(const struct page *page)
 {
        unsigned long head = READ_ONCE(page->compound_head);
 

With a kernel-config based on something derived from Fedora config-6.8.9-100.fc38.x86_64 for convenience with

CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP=y

add/remove: 15/14 grow/shrink: 79/87 up/down: 12836/-13917 (-1081)
Function old new delta

change_pte_range                               -    2308   +2308
iommu_put_dma_cookie                         454    1276    +822
get_hwpoison_page                           2007    2580    +573
end_bbio_data_read                          1171    1626    +455
end_bbio_meta_read                           492     934    +442
ext4_finish_bio                              773    1208    +435
fq_ring_free_locked                          128     541    +413
end_bbio_meta_write                          493     872    +379
gup_fast_fallback                           4207    4568    +361
v1_free_pgtable                              166     519    +353
iommu_v1_map_pages                          2747    3098    +351
end_bbio_data_write                          609     960    +351
fsverity_verify_bio                          334     656    +322
follow_page_mask                            3399    3719    +320
__read_end_io                                316     635    +319
btrfs_end_super_write                        494     789    +295
iommu_alloc_pages_node.constprop             286     572    +286
free_buffers.part                              -     271    +271
gup_must_unshare                               -     268    +268
smaps_pte_range                             1285    1513    +228
pagemap_pmd_range                           2189    2393    +204
iommu_alloc_pages_node                         -     193    +193
smaps_hugetlb_range                          705     897    +192
follow_page_pte                             1584    1758    +174
__migrate_device_pages                      2435    2595    +160
unpin_user_pages_dirty_lock                  205     362    +157
_compound_head                                 -     150    +150
unpin_user_pages                             143     282    +139
put_ref_page.part                              -     126    +126
iomap_finish_ioend                           866     972    +106
iomap_read_end_io                            673     763     +90
end_bbio_meta_read.cold                       42     131     +89
btrfs_do_readpage                           1759    1845     +86
extent_write_cache_pages                    2133    2212     +79
end_bbio_data_write.cold                      32     108     +76
end_bbio_meta_write.cold                      40     108     +68
__read_end_io.cold                            25      91     +66
btrfs_end_super_write.cold                    25      89     +64
ext4_finish_bio.cold                         118     178     +60
fsverity_verify_bio.cold                      25      84     +59
block_write_begin                            217     274     +57
end_bbio_data_read.cold                      378     426     +48
__pfx__compound_head                           -      48     +48
copy_hugetlb_page_range                     3050    3097     +47
lruvec_stat_mod_folio.constprop              585     630     +45
iomap_finish_ioend.cold                      163     202     +39
md_bitmap_file_unmap                         150     187     +37
free_pgd_range                              1949    1985     +36
prep_move_freepages_block                    319     349     +30
iommu_alloc_pages_node.cold                    -      25     +25
iomap_read_end_io.cold                        65      89     +24
zap_huge_pmd                                 874     897     +23
cont_write_begin.cold                        108     130     +22
skb_splice_from_iter                         822     843     +21
set_pmd_migration_entry                     1037    1058     +21
zerocopy_fill_skb_from_iter                 1321    1340     +19
pagemap_scan_pmd_entry                      3261    3279     +18
try_grab_folio_fast                          452     469     +17
change_huge_pmd                             1174    1191     +17
folio_put                                     48      64     +16
__pfx_set_p4d                                  -      16     +16
__pfx_put_ref_page.part                        -      16     +16
__pfx_lruvec_stat_mod_folio.constprop        208     224     +16
__pfx_iommu_alloc_pages_node.constprop        16      32     +16
__pfx_iommu_alloc_pages_node                   -      16     +16
__pfx_gup_must_unshare                         -      16     +16
__pfx_free_buffers.part                        -      16     +16
__pfx_folio_put                               48      64     +16
__pfx_change_pte_range                         -      16     +16
__pfx___pte                                   32      48     +16
offline_pages                               1962    1975     +13
memfd_pin_folios                            1284    1297     +13
uprobe_write_opcode                         2062    2073     +11
set_p4d                                        -      11     +11
__pte                                         22      33     +11
copy_page_from_iter_atomic                  1714    1724     +10
__migrate_device_pages.cold                   60      70     +10
try_to_unmap_one                            3355    3364      +9
try_to_migrate_one                          3310    3319      +9
stable_page_flags                           1034    1043      +9
io_sqe_buffer_register                      1404    1413      +9
dio_zero_block                               644     652      +8
add_ra_bio_pages.constprop.isra             1542    1550      +8
__add_to_kill                                969     977      +8
btrfs_writepage_fixup_worker                1199    1206      +7
write_protect_page                          1186    1192      +6
iommu_v2_map_pages.cold                      145     151      +6
gup_fast_fallback.cold                       112     117      +5
try_to_merge_one_page                       1857    1860      +3
__apply_to_page_range                       2235    2238      +3
wbc_account_cgroup_owner                     217     219      +2
change_protection.cold                       105     107      +2
can_change_pte_writable                      354     356      +2
vmf_insert_pfn_pud                           699     700      +1
split_huge_page_to_list_to_order.cold        152     151      -1
pte_pfn                                       40      39      -1
move_pages                                  5270    5269      -1
isolate_single_pageblock                    1056    1055      -1
__apply_to_page_range.cold                    92      91      -1
unmap_page_range.cold                         88      86      -2
do_huge_pmd_numa_page                       1175    1173      -2
free_pgd_range.cold                          162     159      -3
copy_page_to_iter                            329     326      -3
copy_page_range.cold                         149     146      -3
copy_page_from_iter                          307     304      -3
can_finish_ordered_extent                    551     548      -3
__replace_page                              1133    1130      -3
__reset_isolation_pfn                        645     641      -4
dio_send_cur_page                           1113    1108      -5
__access_remote_vm                          1010    1005      -5
pagemap_hugetlb_category                     468     459      -9
extent_write_locked_range                   1148    1139      -9
unuse_pte_range                             1834    1821     -13
do_migrate_range                            1935    1922     -13
__get_user_pages                            1952    1938     -14
migrate_vma_collect_pmd                     2817    2802     -15
copy_page_to_iter_nofault                   2373    2358     -15
hugetlb_fault                               4054    4038     -16
__pfx_shake_page                              16       -     -16
__pfx_put_page                                16       -     -16
__pfx_pfn_swap_entry_to_page                  32      16     -16
__pfx_gup_must_unshare.part                   16       -     -16
__pfx_gup_folio_next                          16       -     -16
__pfx_free_buffers                            16       -     -16
__pfx___get_unpoison_page                     16       -     -16
btrfs_cleanup_ordered_extents                622     604     -18
read_rdev                                    694     673     -21
isolate_migratepages_block.cold              222     197     -25
hugetlb_mfill_atomic_pte                    1869    1844     -25
folio_pte_batch.constprop                   1020     995     -25
hugetlb_reserve_pages                       1468    1441     -27
__alloc_fresh_hugetlb_folio                  676     649     -27
intel_pasid_alloc_table.cold                  83      52     -31
__pfx_iommu_put_pages_list                    48      16     -32
__pfx_PageHuge                                32       -     -32
__blockdev_direct_IO.cold                    952     920     -32
io_ctl_prepare_pages                         832     794     -38
__handle_mm_fault                           4237    4195     -42
finish_fault                                1007     962     -45
__pfx_pfn_swap_entry_folio                    64      16     -48
vm_normal_folio_pmd                           84      34     -50
vm_normal_folio                               84      34     -50
set_migratetype_isolate                     1429    1375     -54
do_set_pmd                                   618     561     -57
can_change_pmd_writable                      293     229     -64
__unmap_hugepage_range                      2389    2325     -64
do_fault                                    1187    1121     -66
fault_dirty_shared_page                      425     358     -67
madvise_free_huge_pmd                        863     792     -71
insert_page_into_pte_locked.isra             502     429     -73
restore_exclusive_pte                        539     463     -76
isolate_migratepages_block                  5436    5355     -81
__do_fault                                   366     276     -90
set_pte_range                                593     502     -91
follow_devmap_pmd                            559     468     -91
__pfx_bio_first_folio                        144      48     -96
shake_page                                   105       -    -105
hugetlb_change_protection                   2314    2204    -110
hugetlb_wp                                  2134    2017    -117
__blockdev_direct_IO                        5063    4946    -117
skb_tx_error                                 272     149    -123
put_page                                     123       -    -123
gup_must_unshare.part                        135       -    -135
PageHuge                                     136       -    -136
ksm_scan_thread                             9172    9032    -140
intel_pasid_alloc_table                      596     447    -149
copy_huge_pmd                               1539    1385    -154
skb_split                                   1534    1376    -158
split_huge_pmd_locked                       4024    3865    -159
skb_append_pagefrags                         663     504    -159
memory_failure                              2784    2624    -160
unpoison_memory                             1328    1167    -161
cont_write_begin                             959     793    -166
pfn_swap_entry_to_page                       250      82    -168
skb_pp_cow_data                             1539    1367    -172
gup_folio_next                               180       -    -180
intel_pasid_get_entry.isra                   607     425    -182
v2_alloc_pgtable                             309     126    -183
do_huge_pmd_wp_page                         1173     988    -185
bio_first_folio.cold                         315     105    -210
unmap_page_range                            6091    5873    -218
split_huge_page_to_list_to_order            4141    3905    -236
move_pages_huge_pmd                         2053    1813    -240
free_buffers                                 286       -    -286
iommu_v2_map_pages                          1722    1428    -294
soft_offline_page                           2149    1843    -306
do_wp_page                                  3340    2993    -347
do_swap_page                                4619    4265    -354
md_import_device                            1002     635    -367
copy_page_range                             7436    7040    -396
__get_unpoison_page                          415       -    -415
pfn_swap_entry_folio                         596     149    -447
iommu_put_pages_list                        1071     344    -727
bio_first_folio                             2322     774   -1548
change_protection                           5008    2790   -2218

Total: Before=32786363, After=32785282, chg -0.00%

Signature · 3 lines
Cheers,

David / dhildenb
#20 lore raw
Quoted 38 lines · David Hildenbrand
Quoted 5 lines · Mateusz Guzik
Quoted 4 lines · Yin Fengwei
Quoted 2 lines · Mateusz Guzik

would you mind benchmarking the change which merely force-inlines _compund_page?

Quoted 8 lines
Quoted 2 lines
Quoted 1 line

https://lore.kernel.org/linux-mm/[email protected]/

This change can resolve the regression also:

Great, thanks.

David, I guess this means it would be fine to inline the entire thing at least from this bench standpoint. Given that this is your idea I guess you should do the needful(tm)? :)

Testing

include/linux/page-flags.h +1 -1
index 5769fe6e4950..25e25b34f4a0 100644
@@ -235,7 +235,7 @@ static __always_inline int page_is_fake_head(const struct page *page)
         return page_fixed_fake_head(page) != page;
  }
 
-static inline unsigned long _compound_head(const struct page *page)
+static __always_inline unsigned long _compound_head(const struct page *page)
  {
         unsigned long head = READ_ONCE(page->compound_head);
 

With a kernel-config based on something derived from Fedora config-6.8.9-100.fc38.x86_64 for convenience with

CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP=y

add/remove: 15/14 grow/shrink: 79/87 up/down: 12836/-13917 (-1081)

[snip]

Quoted 1 line

Total: Before=32786363, After=32785282, chg -0.00%

I guess there should be no opposition then?

Given that this is your patch I presume you are going to see this through.

I don't want any mention or cc on the patch, thanks for understanding :)

Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
#21 lore raw
Quoted 38 lines · Mateusz Guzik
Quoted 37 lines · David Hildenbrand
Quoted 5 lines · Mateusz Guzik
Quoted 4 lines · Yin Fengwei
Quoted 2 lines · Mateusz Guzik

would you mind benchmarking the change which merely force-inlines _compund_page?

Quoted 8 lines
Quoted 2 lines
Quoted 1 line

https://lore.kernel.org/linux-mm/[email protected]/

This change can resolve the regression also:

Great, thanks.

David, I guess this means it would be fine to inline the entire thing at least from this bench standpoint. Given that this is your idea I guess you should do the needful(tm)? :)

Testing

include/linux/page-flags.h +1 -1
index 5769fe6e4950..25e25b34f4a0 100644
@@ -235,7 +235,7 @@ static __always_inline int page_is_fake_head(const struct page *page)
          return page_fixed_fake_head(page) != page;
   }
 
-static inline unsigned long _compound_head(const struct page *page)
+static __always_inline unsigned long _compound_head(const struct page *page)
   {
          unsigned long head = READ_ONCE(page->compound_head);
 

With a kernel-config based on something derived from Fedora config-6.8.9-100.fc38.x86_64 for convenience with

CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP=y

Quoted 7 lines
Quoted 1 line

add/remove: 15/14 grow/shrink: 79/87 up/down: 12836/-13917 (-1081)

[snip]

Quoted 1 line

Total: Before=32786363, After=32785282, chg -0.00%

I guess there should be no opposition then?

Given that this is your patch I presume you are going to see this through.

I was hoping that you could send an official patch, after all you did most of the work here.

Quoted 2 lines

I don't want any mention or cc on the patch, thanks for understanding :)

If I have to send it I will respect it.

Signature · 3 lines
Cheers,

David / dhildenb