padata: parallelize deferred page init
Subsystems (10)
Daniel Jordan
maintainerv2 cover padata: parallelize deferred page init
Deferred struct page init is a bottleneck in kernel boot--the biggest for us and probably others. Optimizing it maximizes availability for large-memory systems and allows spinning up short-lived VMs as needed without having to leave them running. It also benefits bare metal machines hosting VMs that are sensitive to downtime. In projects such as VMM Fast Restart[1], where guest state is preserved across kexec reboot, it helps prevent application and network timeouts in the guests.
So, multithread deferred init to take full advantage of system memory bandwidth.
Extend padata, a framework that handles many parallel singlethreaded jobs, to handle multithreaded jobs as well by adding support for splitting up the work evenly, specifying a minimum amount of work that's appropriate for one helper thread to do, load balancing between helpers, and coordinating them. More documentation in patches 4 and 7.
This series is the first step in a project to address other memory proportional bottlenecks in the kernel such as pmem struct page init, vfio page pinning, hugetlb fallocate, and munmap. Deferred page init doesn't require concurrency limits, resource control, or priority adjustments like these other users will because it happens during boot when the system is otherwise idle and waiting for page init to finish.
This has been run on a variety of x86 systems and speeds up kernel boot by 3% to 49%, saving up to 1.6 out of 4 seconds. Patch 5 has more numbers.
Please review and test, and thanks to Alex, Andrew, Josh, and Pavel for their feedback in the last version.
The powerpc and s390 lists are included in case they want to give this a try, they had enabled this feature when it was configured per arch.
Series based on 5.7-rc6 plus these three from mmotm
mm-call-touch_nmi_watchdog-on-max-order-boundaries-in-deferred-init.patch mm-initialize-deferred-pages-with-interrupts-enabled.patch mm-call-cond_resched-from-deferred_init_memmap.patch
and it's available here:
git://oss.oracle.com/git/linux-dmjordan.git padata-mt-definit-v2
https://oss.oracle.com/git/gitweb.cgi?p=linux-dmjordan.git;a=shortlog;h=refs/heads/padata-mt-definit-v2
and the future users and related features are available as work-in-progress:
git://oss.oracle.com/git/linux-dmjordan.git padata-mt-wip-v0.4
https://oss.oracle.com/git/gitweb.cgi?p=linux-dmjordan.git;a=shortlog;h=refs/heads/padata-mt-wip-v0.4
v2:
- Improve the problem statement (Andrew, Josh, Pavel)
- Add T-b's to unchanged patches (Josh)
- Fully initialize max-order blocks to avoid buddy issues (Alex)
- Parallelize on section-aligned boundaries to avoid potential false sharing (Alex)
- Return the maximum thread count from a function that architectures can override, with the generic version returning 1 (current behavior). Override for x86 since that's the only arch this series has been tested on so far. Other archs can test with more threads by dropping patch 6.
- Rebase to v5.7-rc6, rerun tests
RFC v4 [2] -> v1:
- merged with padata (Peter)
- got rid of the 'task' nomenclature (Peter, Jon)
future work branch:
- made lockdep-aware (Jason, Peter)
- adjust workqueue worker priority with renice_or_cancel() (Tejun)
- fixed undo problem in VFIO (Alex)
The remaining feedback, mainly resource control awareness (cgroup etc), is TODO for later series.
[1] https://static.sched.com/hosted_files/kvmforum2019/66/VMM-fast-restart_kvmforum2019.pdf
https://www.youtube.com/watch?v=pBsHnf93tcQ https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
Daniel Jordan (7):
padata: remove exit routine
padata: initialize earlier
padata: allocate work structures for parallel jobs from a pool
padata: add basic support for multithreaded jobs
mm: parallelize deferred_init_memmap()
mm: make deferred init's max threads arch-specific
padata: document multithreaded jobs
Documentation/core-api/padata.rst | 41 +++-- arch/x86/mm/init_64.c | 12 ++ include/linux/memblock.h | 3 + include/linux/padata.h | 43 ++++- init/main.c | 2 + kernel/padata.c | 277 ++++++++++++++++++++++++------ mm/Kconfig | 6 +- mm/page_alloc.c | 67 +++++++- 8 files changed, 373 insertions(+), 78 deletions(-)
base-commit: b9bbe6ed63b2b9f2c9ee5cbd0f2c946a2723f4ce
prerequisite-patch-id: 4ad522141e1119a325a9799dad2bd982fbac8b7c
prerequisite-patch-id: 169273327e56f5461101a71dfbd6b4cfd4570cf0
prerequisite-patch-id: 0f34692c8a9673d4c4f6a3545cf8ec3a2abf8620
Signature · 1 line
2.26.2
Daniel Jordan
maintainerPATCH v2 1/7 padata: remove exit routine
padata_driver_exit() is unnecessary because padata isn't built as a module and doesn't exit.
padata's init routine will soon allocate memory, so getting rid of the exit function now avoids pointless code to free it.
--- kernel/padata.c | 6 ------ 1 file changed, 6 deletions(-)
kernel/padata.c +0 -6
@@ -1072,10 +1072,4 @@ static __init int padata_driver_init(void) }module_init (padata_driver_init); -static __exit void padata_driver_exit(void) -{ -cpuhp_remove_multi_state (CPUHP_PADATA_DEAD ); -cpuhp_remove_multi_state (hp_online); -} -module_exit (padata_driver_exit);#endif
Signature · 1 line
2.26.2
Daniel Jordan
maintainerPATCH v2 2/7 padata: initialize earlier
In reply to Daniel Jordanpadata will soon initialize the system's struct pages in parallel, so it needs to be ready by page_alloc_init_late().
The error return from padata_driver_init() triggers an initcall warning, so add a warning to padata_init() to avoid silent failure.
--- include/linux/padata.h | 6 ++++++ init/main.c | 2 ++ kernel/padata.c | 17 ++++++++--------- 3 files changed, 16 insertions(+), 9 deletions(-)
include/linux/padata.h +6 -0
@@ -164,6 +164,12 @@ struct padata_instance {#define PADATA_INVALID 4 }; +#ifdef CONFIG_PADATA +extern void __init padata_init (void ); +#else +static inline void __init padata_init (void ) {} +#endif +extern struct padata_instance *padata_alloc_possible (const char *name);extern void padata_free (struct padata_instance *pinst);extern struct padata_shell *padata_alloc_shell (struct padata_instance *pinst);
init/main.c +2 -0
@@ -94,6 +94,7 @@#include <linux/rodata_test.h> #include <linux/jump_label.h> #include <linux/mem_encrypt.h> +#include <linux/padata.h> #include <asm/io.h> #include <asm/bugs.h>
@@ -1482,6 +1483,7 @@ static noinline void __init kernel_init_freeable(void)smp_init ();sched_init_smp (); +padata_init ();page_alloc_init_late ();/* Initialize page ext after all struct pages are initialized. */ page_ext_init ();
kernel/padata.c +8 -9
@@ -31,7 +31,6 @@#include <linux/slab.h> #include <linux/sysfs.h> #include <linux/rcupdate.h> -#include <linux/module.h> #define MAX_OBJ_NUM 1000
@@ -1050,26 +1049,26 @@ void padata_free_shell(struct padata_shell *ps) }EXPORT_SYMBOL (padata_free_shell); -#ifdef CONFIG_HOTPLUG_CPU - -static __init int padata_driver_init (void ) +void __init padata_init (void ) { +#ifdef CONFIG_HOTPLUG_CPU int ret; ret =cpuhp_setup_state_multi (CPUHP_AP_ONLINE_DYN ,"padata:online" , padata_cpu_online,NULL );if (ret <0 ) -return ret; + goto err; hp_online = ret; ret =cpuhp_setup_state_multi (CPUHP_PADATA_DEAD ,"padata:dead" ,NULL , padata_cpu_dead);if (ret <0 ) {cpuhp_remove_multi_state (hp_online); -return ret; + goto err; } -return 0 ; -} -module_init (padata_driver_init); +return ; +err: +pr_warn ("padata: initialization failed\n" );#endif +}
Signature · 1 line
2.26.2
Daniel Jordan
maintainerPATCH v2 3/7 padata: allocate work structures for parallel jobs from a pool
In reply to Daniel Jordanpadata allocates per-CPU, per-instance work structs for parallel jobs. A do_parallel call assigns a job to a sequence number and hashes the number to a CPU, where the job will eventually run using the corresponding work.
This approach fit with how padata used to bind a job to each CPU round-robin, makes less sense after commit bfde23ce200e6 ("padata:
unbind parallel jobs from specific CPUs") because a work isn't bound to a particular CPU anymore, and isn't needed at all for multithreaded jobs because they don't have sequence numbers.
Replace the per-CPU works with a preallocated pool, which allows sharing them between existing padata users and the upcoming multithreaded user. The pool will also facilitate setting NUMA-aware concurrency limits with later users.
The pool is sized according to the number of possible CPUs. With this limit, MAX_OBJ_NUM no longer makes sense, so remove it.
If the global pool is exhausted, a parallel job is run in the current task instead to throttle a system trying to do too much in parallel.
--- include/linux/padata.h | 8 +-- kernel/padata.c | 118 +++++++++++++++++++++++++++-------------- 2 files changed, 78 insertions(+), 48 deletions(-)
include/linux/padata.h +1 -7
@@ -24,7 +24,6 @@* @list: List entry, to attach to the padata lists. * @pd: Pointer to the internal control structure. * @cb_cpu: Callback cpu for serializatioon. -* @cpu: Cpu for parallelization. * @seq_nr: Sequence number of the parallelized data object. * @info: Used to pass information from the parallel to the serial function. * @parallel: Parallel execution function.
@@ -34,7 +33,6 @@ struct padata_priv {struct list_head list;struct parallel_data *pd;int cb_cpu; -int cpu;unsigned int seq_nr;int info;void (*parallel)(struct padata_priv *padata);
@@ -68,15 +66,11 @@ struct padata_serial_queue {/** * struct padata_parallel_queue - The percpu padata parallel queue * -* @parallel: List to wait for parallelization. * @reorder: List to wait for reordering after parallel processing. -* @work: work struct for parallelization. * @num_obj: Number of objects that are processed by this cpu. */ struct padata_parallel_queue { -struct padata_list parallel;struct padata_list reorder; -struct work_struct work;atomic_t num_obj; };
@@ -111,7 +105,7 @@ struct parallel_data {struct padata_parallel_queue __percpu *pqueue;struct padata_serial_queue __percpu *squeue;atomic_t refcnt; -atomic_t seq_nr; +unsigned int seq_nr;unsigned int processed;int cpu;struct padata_cpumask cpumask;
kernel/padata.c +77 -41
190 lines not on this page · raw message
Signature · 1 line
2.26.2
Daniel Jordan
maintainerPATCH v2 4/7 padata: add basic support for multithreaded jobs
In reply to Daniel JordanSometimes the kernel doesn't take full advantage of system memory bandwidth, leading to a single CPU spending excessive time in initialization paths where the data scales with memory size.
Multithreading naturally addresses this problem.
Extend padata, a framework that handles many parallel yet singlethreaded jobs, to also handle multithreaded jobs by adding support for splitting up the work evenly, specifying a minimum amount of work that's appropriate for one helper thread to do, load balancing between helpers, and coordinating them.
This is inspired by work from Pavel Tatashin and Steve Sistare.
--- include/linux/padata.h | 29 ++++++++ kernel/padata.c | 152 ++++++++++++++++++++++++++++++++++++++++- 2 files changed, 178 insertions(+), 3 deletions(-)
include/linux/padata.h +29 -0
@@ -4,6 +4,9 @@* * Copyright (C) 2008, 2009 secunet Security Networks AG * Copyright (C) 2008, 2009 Steffen Klassert <[email protected]> +* +* Copyright (c) 2020 Oracle and/or its affiliates. +* Author: Daniel Jordan <[email protected]> */ #ifndef PADATA_H
@@ -130,6 +133,31 @@ struct padata_shell {struct list_head list; }; +/** +* struct padata_mt_job - represents one multithreaded job +* +* @thread_fn: Called for each chunk of work that a padata thread does. +* @fn_arg: The thread function argument. +* @start: The start of the job (units are job-specific). +* @size: size of this node's work (units are job-specific). +* @align: Ranges passed to the thread function fall on this boundary, with the +* possible exceptions of the beginning and end of the job. +* @min_chunk: The minimum chunk size in job-specific units. This allows +* the client to communicate the minimum amount of work that's +* appropriate for one worker thread to do at once. +* @max_threads: Max threads to use for the job, actual number may be less +* depending on task size and minimum chunk size. +*/ +struct padata_mt_job { +void (*thread_fn)(unsigned long start,unsigned long end,void *arg); +void *fn_arg; +unsigned long start; +unsigned long size; +unsigned long align; +unsigned long min_chunk; +int max_threads; +}; +/** * struct padata_instance - The overall control structure. *
@@ -171,6 +199,7 @@ extern void padata_free_shell(struct padata_shell *ps);extern int padata_do_parallel (struct padata_shell *ps,struct padata_priv *padata,int *cb_cpu);extern void padata_do_serial (struct padata_priv *padata); +extern void __init padata_do_multithreaded (struct padata_mt_job *job);extern int padata_set_cpumask (struct padata_instance *pinst,int cpumask_type,cpumask_var_t cpumask);extern int padata_start (struct padata_instance *pinst);
kernel/padata.c +149 -3
205 lines not on this page · raw message
Signature · 1 line
2.26.2
Daniel Jordan
maintainerPATCH v2 5/7 mm: parallelize deferred_init_memmap()
In reply to Daniel JordanDeferred struct page init is a significant bottleneck in kernel boot. Optimizing it maximizes availability for large-memory systems and allows spinning up short-lived VMs as needed without having to leave them running. It also benefits bare metal machines hosting VMs that are sensitive to downtime. In projects such as VMM Fast Restart[1], where guest state is preserved across kexec reboot, it helps prevent application and network timeouts in the guests.
Multithread to take full advantage of system memory bandwidth.
The maximum number of threads is capped at the number of CPUs on the node because speedups always improve with additional threads on every system tested, and at this phase of boot, the system is otherwise idle and waiting on page init to finish.
Helper threads operate on section-aligned ranges to both avoid false sharing when setting the pageblock's migrate type and to avoid accessing uninitialized buddy pages, though max order alignment is enough for the latter.
The minimum chunk size is also a section. There was benefit to using multiple threads even on relatively small memory (1G) systems, and this is the smallest size that the alignment allows.
The time (milliseconds) is the slowest node to initialize since boot blocks until all nodes finish. intel_pstate is loaded in active mode without hwp and with turbo enabled, and intel_idle is active as well.
Intel(R) Xeon(R) Platinum 8167M CPU @ 2.00GHz (Skylake, bare metal)
2 nodes * 26 cores * 2 threads = 104 CPUs
384G/node = 768G memory
kernel boot deferred init
------------------------ ------------------------
node% (thr) speedup time_ms (stdev) speedup time_ms (stdev)
( 0) -- 4078.0 ( 9.0) -- 1779.0 ( 8.7)
2% ( 1) 1.4% 4021.3 ( 2.9) 3.4% 1717.7 ( 7.8)
12% ( 6) 35.1% 2644.7 ( 35.3) 80.8% 341.0 ( 35.5)
25% ( 13) 38.7% 2498.0 ( 34.2) 89.1% 193.3 ( 32.3)
37% ( 19) 39.1% 2482.0 ( 25.2) 90.1% 175.3 ( 31.7)
50% ( 26) 38.8% 2495.0 ( 8.7) 89.1% 193.7 ( 3.5)
75% ( 39) 39.2% 2478.0 ( 21.0) 90.3% 172.7 ( 26.7)
100% ( 52) 40.0% 2448.0 ( 2.0) 91.9% 143.3 ( 1.5)
Intel(R) Xeon(R) CPU E5-2699C v4 @ 2.20GHz (Broadwell, bare metal)
1 node * 16 cores * 2 threads = 32 CPUs
192G/node = 192G memory
kernel boot deferred init
------------------------ ------------------------
node% (thr) speedup time_ms (stdev) speedup time_ms (stdev)
( 0) -- 1996.0 ( 18.0) -- 1104.3 ( 6.7)
3% ( 1) 1.4% 1968.0 ( 3.0) 2.7% 1074.7 ( 9.0)
12% ( 4) 40.1% 1196.0 ( 22.7) 72.4% 305.3 ( 16.8)
25% ( 8) 47.4% 1049.3 ( 17.2) 84.2% 174.0 ( 10.6)
37% ( 12) 48.3% 1032.0 ( 14.9) 86.8% 145.3 ( 2.5)
50% ( 16) 48.9% 1020.3 ( 2.5) 88.0% 133.0 ( 1.7)
75% ( 24) 49.1% 1016.3 ( 8.1) 88.4% 128.0 ( 1.7)
100% ( 32) 49.4% 1009.0 ( 8.5) 88.6% 126.3 ( 0.6)
Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz (Haswell, bare metal)
2 nodes * 18 cores * 2 threads = 72 CPUs
128G/node = 256G memory
kernel boot deferred init
------------------------ ------------------------
node% (thr) speedup time_ms (stdev) speedup time_ms (stdev)
( 0) -- 1682.7 ( 6.7) -- 630.0 ( 4.6)
3% ( 1) 0.4% 1676.0 ( 2.0) 0.7% 625.3 ( 3.2)
12% ( 4) 25.8% 1249.0 ( 1.0) 68.2% 200.3 ( 1.2)
25% ( 9) 30.0% 1178.0 ( 5.2) 79.7% 128.0 ( 3.5)
37% ( 13) 30.6% 1167.7 ( 3.1) 81.3% 117.7 ( 1.2)
50% ( 18) 30.6% 1167.3 ( 2.3) 81.4% 117.0 ( 1.0)
75% ( 27) 31.0% 1161.3 ( 4.6) 82.5% 110.0 ( 6.9)
100% ( 36) 32.1% 1142.0 ( 3.6) 85.7% 90.0 ( 1.0)
AMD EPYC 7551 32-Core Processor (Zen, kvm guest)
1 node * 8 cores * 2 threads = 16 CPUs
64G/node = 64G memory
kernel boot deferred init
------------------------ ------------------------
node% (thr) speedup time_ms (stdev) speedup time_ms (stdev)
( 0) -- 1003.7 ( 16.6) -- 243.3 ( 8.1)
6% ( 1) 1.4% 990.0 ( 4.6) 1.2% 240.3 ( 1.5)
12% ( 2) 11.4% 889.3 ( 16.7) 44.5% 135.0 ( 3.0)
25% ( 4) 16.8% 835.3 ( 9.0) 65.8% 83.3 ( 2.5)
37% ( 6) 18.6% 816.7 ( 17.6) 70.4% 72.0 ( 1.0)
50% ( 8) 18.2% 821.0 ( 5.0) 70.7% 71.3 ( 1.2)
75% ( 12) 19.0% 813.3 ( 5.0) 71.8% 68.7 ( 2.1)
100% ( 16) 19.8% 805.3 ( 10.8) 76.4% 57.3 ( 15.9)Server-oriented distros that enable deferred page init sometimes run in small VMs, and they still benefit even though the fraction of boot time saved is smaller:
AMD EPYC 7551 32-Core Processor (Zen, kvm guest)
1 node * 2 cores * 2 threads = 4 CPUs
16G/node = 16G memory
kernel boot deferred init
------------------------ ------------------------
node% (thr) speedup time_ms (stdev) speedup time_ms (stdev)
( 0) -- 722.3 ( 9.5) -- 50.7 ( 0.6)
25% ( 1) -3.3% 746.3 ( 4.7) -2.0% 51.7 ( 1.2)
50% ( 2) 0.2% 721.0 ( 11.3) 29.6% 35.7 ( 4.9)
75% ( 3) -0.3% 724.3 ( 11.2) 48.7% 26.0 ( 0.0)
100% ( 4) 3.0% 700.3 ( 13.6) 55.9% 22.3 ( 0.6)
Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz (Haswell, kvm guest)
1 node * 2 cores * 2 threads = 4 CPUs
14G/node = 14G memory
kernel boot deferred init
------------------------ ------------------------
node% (thr) speedup time_ms (stdev) speedup time_ms (stdev)
( 0) -- 673.0 ( 6.9) -- 57.0 ( 1.0)
25% ( 1) -0.6% 677.3 ( 19.8) 1.8% 56.0 ( 1.0)
50% ( 2) 3.4% 650.0 ( 3.6) 36.8% 36.0 ( 5.2)
75% ( 3) 4.2% 644.7 ( 7.6) 56.1% 25.0 ( 1.0)
100% ( 4) 5.3% 637.0 ( 5.6) 63.2% 21.0 ( 0.0)On Josh's 96-CPU and 192G memory system:
Without this patch series:
[ 0.487132] node 0 initialised, 23398907 pages in 292ms
[ 0.499132] node 1 initialised, 24189223 pages in 304ms
...
[ 0.629376] Run /sbin/init as init process
With this patch series:
[ 0.227868] node 0 initialised, 23398907 pages in 28ms
[ 0.230019] node 1 initialised, 24189223 pages in 28ms
...
[ 0.361069] Run /sbin/init as init process[1] https://static.sched.com/hosted_files/kvmforum2019/66/VMM-fast-restart_kvmforum2019.pdf
--- mm/Kconfig | 6 ++--- mm/page_alloc.c | 60 ++++++++++++++++++++++++++++++++++++++++++++----- 2 files changed, 58 insertions(+), 8 deletions(-)
mm/Kconfig +3 -3
@@ -750,13 +750,13 @@ config DEFERRED_STRUCT_PAGE_INIT depends on SPARSEMEM depends on !NEED_PER_CPU_KM depends on 64BIT + select PADATA help Ordinarily all struct pages are initialised during early boot in a single thread. On very large machines this can take a considerable amount of time. If this option is set, large machines will bring up - a subset of memmap at boot and then initialise the rest in parallel - by starting one-off "pgdatinitX" kernel thread for each node X. This - has a potential performance impact on processes running early in the + a subset of memmap at boot and then initialise the rest in parallel. + This has a potential performance impact on tasks running early in the lifetime of the system until these kthreads finish the initialisation.
mm/page_alloc.c +55 -5
88 lines not on this page · raw message
Signature · 1 line
2.26.2
Re: [PATCH v2 5/7] mm: parallelize deferred_init_memmap()
Quoted 200 lines · Daniel Jordan
Deferred struct page init is a significant bottleneck in kernel boot. Optimizing it maximizes availability for large-memory systems and allows spinning up short-lived VMs as needed without having to leave them running. It also benefits bare metal machines hosting VMs that are sensitive to downtime. In projects such as VMM Fast Restart[1], where guest state is preserved across kexec reboot, it helps prevent application and network timeouts in the guests.
Multithread to take full advantage of system memory bandwidth.
The maximum number of threads is capped at the number of CPUs on the node because speedups always improve with additional threads on every system tested, and at this phase of boot, the system is otherwise idle and waiting on page init to finish.
Helper threads operate on section-aligned ranges to both avoid false sharing when setting the pageblock's migrate type and to avoid accessing uninitialized buddy pages, though max order alignment is enough for the latter.
The minimum chunk size is also a section. There was benefit to using multiple threads even on relatively small memory (1G) systems, and this is the smallest size that the alignment allows.
The time (milliseconds) is the slowest node to initialize since boot blocks until all nodes finish.
intel_pstateis loaded in active mode without hwp and with turbo enabled, andintel_idleis active as well.Intel(R) Xeon(R) Platinum 8167M CPU @ 2.00GHz (Skylake, bare metal) 2 nodes * 26 cores * 2 threads = 104 CPUs 384G/node = 768G memory kernel boot deferred init ------------------------ ------------------------ node% (thr) speedup time_ms (stdev) speedup time_ms (stdev) ( 0) -- 4078.0 ( 9.0) -- 1779.0 ( 8.7) 2% ( 1) 1.4% 4021.3 ( 2.9) 3.4% 1717.7 ( 7.8) 12% ( 6) 35.1% 2644.7 ( 35.3) 80.8% 341.0 ( 35.5) 25% ( 13) 38.7% 2498.0 ( 34.2) 89.1% 193.3 ( 32.3) 37% ( 19) 39.1% 2482.0 ( 25.2) 90.1% 175.3 ( 31.7) 50% ( 26) 38.8% 2495.0 ( 8.7) 89.1% 193.7 ( 3.5) 75% ( 39) 39.2% 2478.0 ( 21.0) 90.3% 172.7 ( 26.7) 100% ( 52) 40.0% 2448.0 ( 2.0) 91.9% 143.3 ( 1.5) Intel(R) Xeon(R) CPU E5-2699C v4 @ 2.20GHz (Broadwell, bare metal) 1 node * 16 cores * 2 threads = 32 CPUs 192G/node = 192G memory kernel boot deferred init ------------------------ ------------------------ node% (thr) speedup time_ms (stdev) speedup time_ms (stdev) ( 0) -- 1996.0 ( 18.0) -- 1104.3 ( 6.7) 3% ( 1) 1.4% 1968.0 ( 3.0) 2.7% 1074.7 ( 9.0) 12% ( 4) 40.1% 1196.0 ( 22.7) 72.4% 305.3 ( 16.8) 25% ( 8) 47.4% 1049.3 ( 17.2) 84.2% 174.0 ( 10.6) 37% ( 12) 48.3% 1032.0 ( 14.9) 86.8% 145.3 ( 2.5) 50% ( 16) 48.9% 1020.3 ( 2.5) 88.0% 133.0 ( 1.7) 75% ( 24) 49.1% 1016.3 ( 8.1) 88.4% 128.0 ( 1.7) 100% ( 32) 49.4% 1009.0 ( 8.5) 88.6% 126.3 ( 0.6) Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz (Haswell, bare metal) 2 nodes * 18 cores * 2 threads = 72 CPUs 128G/node = 256G memory kernel boot deferred init ------------------------ ------------------------ node% (thr) speedup time_ms (stdev) speedup time_ms (stdev) ( 0) -- 1682.7 ( 6.7) -- 630.0 ( 4.6) 3% ( 1) 0.4% 1676.0 ( 2.0) 0.7% 625.3 ( 3.2) 12% ( 4) 25.8% 1249.0 ( 1.0) 68.2% 200.3 ( 1.2) 25% ( 9) 30.0% 1178.0 ( 5.2) 79.7% 128.0 ( 3.5) 37% ( 13) 30.6% 1167.7 ( 3.1) 81.3% 117.7 ( 1.2) 50% ( 18) 30.6% 1167.3 ( 2.3) 81.4% 117.0 ( 1.0) 75% ( 27) 31.0% 1161.3 ( 4.6) 82.5% 110.0 ( 6.9) 100% ( 36) 32.1% 1142.0 ( 3.6) 85.7% 90.0 ( 1.0) AMD EPYC 7551 32-Core Processor (Zen, kvm guest) 1 node * 8 cores * 2 threads = 16 CPUs 64G/node = 64G memory kernel boot deferred init ------------------------ ------------------------ node% (thr) speedup time_ms (stdev) speedup time_ms (stdev) ( 0) -- 1003.7 ( 16.6) -- 243.3 ( 8.1) 6% ( 1) 1.4% 990.0 ( 4.6) 1.2% 240.3 ( 1.5) 12% ( 2) 11.4% 889.3 ( 16.7) 44.5% 135.0 ( 3.0) 25% ( 4) 16.8% 835.3 ( 9.0) 65.8% 83.3 ( 2.5) 37% ( 6) 18.6% 816.7 ( 17.6) 70.4% 72.0 ( 1.0) 50% ( 8) 18.2% 821.0 ( 5.0) 70.7% 71.3 ( 1.2) 75% ( 12) 19.0% 813.3 ( 5.0) 71.8% 68.7 ( 2.1) 100% ( 16) 19.8% 805.3 ( 10.8) 76.4% 57.3 ( 15.9)Server-oriented distros that enable deferred page init sometimes run in small VMs, and they still benefit even though the fraction of boot time saved is smaller:
AMD EPYC 7551 32-Core Processor (Zen, kvm guest) 1 node * 2 cores * 2 threads = 4 CPUs 16G/node = 16G memory kernel boot deferred init ------------------------ ------------------------ node% (thr) speedup time_ms (stdev) speedup time_ms (stdev) ( 0) -- 722.3 ( 9.5) -- 50.7 ( 0.6) 25% ( 1) -3.3% 746.3 ( 4.7) -2.0% 51.7 ( 1.2) 50% ( 2) 0.2% 721.0 ( 11.3) 29.6% 35.7 ( 4.9) 75% ( 3) -0.3% 724.3 ( 11.2) 48.7% 26.0 ( 0.0) 100% ( 4) 3.0% 700.3 ( 13.6) 55.9% 22.3 ( 0.6) Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz (Haswell, kvm guest) 1 node * 2 cores * 2 threads = 4 CPUs 14G/node = 14G memory kernel boot deferred init ------------------------ ------------------------ node% (thr) speedup time_ms (stdev) speedup time_ms (stdev) ( 0) -- 673.0 ( 6.9) -- 57.0 ( 1.0) 25% ( 1) -0.6% 677.3 ( 19.8) 1.8% 56.0 ( 1.0) 50% ( 2) 3.4% 650.0 ( 3.6) 36.8% 36.0 ( 5.2) 75% ( 3) 4.2% 644.7 ( 7.6) 56.1% 25.0 ( 1.0) 100% ( 4) 5.3% 637.0 ( 5.6) 63.2% 21.0 ( 0.0)On Josh's 96-CPU and 192G memory system:
Without this patch series: [ 0.487132] node 0 initialised, 23398907 pages in 292ms [ 0.499132] node 1 initialised, 24189223 pages in 304ms ... [ 0.629376] Run /sbin/init as init process With this patch series: [ 0.227868] node 0 initialised, 23398907 pages in 28ms [ 0.230019] node 1 initialised, 24189223 pages in 28ms ... [ 0.361069] Run /sbin/init as init process[1] https://static.sched.com/hosted_files/kvmforum2019/66/VMM-fast-restart_kvmforum2019.pdf
Signed-off-by Daniel Jordan <[email protected]>--- mm/Kconfig | 6 ++--- mm/page_alloc.c | 60 ++++++++++++++++++++++++++++++++++++++++++++----- 2 files changed, 58 insertions(+), 8 deletions(-)mm/Kconfig +3
-3@@ -750,13 +750,13 @@ config DEFERRED_STRUCT_PAGE_INIT depends on SPARSEMEM depends on !NEED_PER_CPU_KM depends on 64BIT + select PADATA help Ordinarily all struct pages are initialised during early boot in a single thread. On very large machines this can take a considerable amount of time. If this option is set, large machines will bring up - a subset of memmap at boot and then initialise the rest in parallel - by starting one-off "pgdatinitX" kernel thread for each node X. This - has a potential performance impact on processes running early in the + a subset of memmap at boot and then initialise the rest in parallel. + This has a potential performance impact on tasks running early in the lifetime of the system until these kthreads finish the initialisation.mm/page_alloc.c +19
-0@@ -68,6 +68,7 @@#include <linux/lockdep.h> #include <linux/nmi.h> #include <linux/psi.h> +#include <linux/padata.h> #include <asm/sections.h> #include <asm/tlbflush.h> @@ -1814,16 +1815,44 @@ deferred_init_maxorder(u64 *i, struct zone *zone, unsigned long *start_pfn,return nr_pages; } +struct definit_args { +struct zone *zone; +atomic_long_t nr_pages; +}; + +static void __init +deferred_init_memmap_chunk (unsigned long start_pfn,unsigned long end_pfn, +void *arg) +{ +unsigned long spfn, epfn, nr_pages =0 ; +struct definit_args *args = arg; +struct zone *zone = args->zone; +u64 i; + +deferred_init_mem_pfn_range_in_zone (&i, zone, &spfn, &epfn, start_pfn); + +/* +* Initialize and free pages in MAX_ORDER sized increments so that we
mm/page_alloc.c deferred_init_maxorder(u64 *i, struct zone *zone, unsigned long *start_pfn, · line 1845
Quoted 10 lines
mm/page_alloc.c +10
-0+* can avoid introducing any issues with the buddy allocator. +*/ +while (spfn < end_pfn) { + nr_pages +=deferred_init_maxorder (&i, zone, &spfn, &epfn); +cond_resched (); + } + +atomic_long_add (nr_pages, &args->nr_pages); +} +
Personally I would get rid of nr_pages entirely. It isn't worth the cache thrash to have this atomic variable bouncing around. You could probably just have this function return void since all nr_pages is used for is a pr_info statement at the end of the initialization which will be completely useless now anyway since we really have the threads running in parallel anyway.
We only really need the nr_pages logic in deferred_grow_zone in order to track if we have freed enough pages to allow us to go back to what we were doing.
Quoted 41 lines
mm/page_alloc.c +19
-5/* Initialise remaining memory on a node */ static int __init deferred_init_memmap (void *data) {pg_data_t *pgdat = data;const struct cpumask *cpumask =cpumask_of_node (pgdat->node_id);unsigned long spfn =0 , epfn =0 , nr_pages =0 ; -unsigned long first_init_pfn, flags; +unsigned long first_init_pfn, flags, epfn_align;unsigned long start = jiffies;struct zone *zone; -int zid; +int zid, max_threads;u64 i;/* Bind memory initialisation thread to a local node if possible */ @@ -1863,11 +1892,32 @@ static int __init deferred_init_memmap(void *data) goto zone_empty;/* -* Initialize and free pages in MAX_ORDER sized increments so -* that we can avoid introducing any issues with the buddy -* allocator. +* More CPUs always led to greater speedups on tested systems, up to +* all the nodes' CPUs. Use all since the system is otherwise idle now. */ + max_threads =max (cpumask_weight (cpumask),1u ); +while (spfn < epfn) { + epfn_align =ALIGN_DOWN (epfn,PAGES_PER_SECTION ); + +if (IS_ALIGNED (spfn,PAGES_PER_SECTION ) && + epfn_align - spfn >=PAGES_PER_SECTION ) { +struct definit_args arg = { zone,ATOMIC_LONG_INIT (0 ) }; +struct padata_mt_job job = { + .thread_fn = deferred_init_memmap_chunk, + .fn_arg = &arg, + .start = spfn, + .size = epfn_align - spfn, + .align =PAGES_PER_SECTION , + .min_chunk =PAGES_PER_SECTION , + .max_threads = max_threads,
mm/page_alloc.c static int __init deferred_init_memmap(void *data) · line 1923
Quoted 10 lines
mm/page_alloc.c +7
-0+ }; + +padata_do_multithreaded (&job); + nr_pages +=atomic_long_read (&arg.nr_pages); + spfn = epfn_align; + } + nr_pages +=deferred_init_maxorder (&i, zone, &spfn, &epfn);cond_resched (); }
This doesn't look right. You are basically adding threads in addition to calls to deferred_init_maxorder. In addition you are spawning one job per section instead of per range. Really you should be going for something more along the lines of:
while (spfn < epfn) {
unsigned long epfn_align = ALIGN(epfn,PAGE_PER_SECTION);
struct definit_args arg = { zone, ATOMIC_LONG_INIT(0)
};
struct padata_mt_job job = {
.thread_fn = deferred_init_memmap_chunk,
.fn_arg = &arg,
.start = spfn,
.size = epfn_align - spfn,
.align = PAGES_PER_SECTION,
.min_chunk = PAGES_PER_SECTION,
.max_threads = max_threads,
};
padata_do_multithreaded(&job);
for_each_free_mem_pfn_range_in_zone_from(i, zone,
spfn, epfn) {
if (epfn_align <= spfn)
break;
}
}This should accomplish the same thing, but much more efficiently. The only thing you really lose is the tracking of nr_pages which really doesn't add anything anyway since the value could shift around depending on how many times deferred_grow_zone got called anyway.
Also the spfn should already be sectioned aligned, or at least be in a new section unrelated to the one we just scheduled, so there is no need for the extra checks you had.
Re: [PATCH v2 5/7] mm: parallelize deferred_init_memmap()
Quoted 297 lines · Alexander Duyck
Quoted 200 lines · Daniel Jordan
Deferred struct page init is a significant bottleneck in kernel boot. Optimizing it maximizes availability for large-memory systems and allows spinning up short-lived VMs as needed without having to leave them running. It also benefits bare metal machines hosting VMs that are sensitive to downtime. In projects such as VMM Fast Restart[1], where guest state is preserved across kexec reboot, it helps prevent application and network timeouts in the guests.
Multithread to take full advantage of system memory bandwidth.
The maximum number of threads is capped at the number of CPUs on the node because speedups always improve with additional threads on every system tested, and at this phase of boot, the system is otherwise idle and waiting on page init to finish.
Helper threads operate on section-aligned ranges to both avoid false sharing when setting the pageblock's migrate type and to avoid accessing uninitialized buddy pages, though max order alignment is enough for the latter.
The minimum chunk size is also a section. There was benefit to using multiple threads even on relatively small memory (1G) systems, and this is the smallest size that the alignment allows.
The time (milliseconds) is the slowest node to initialize since boot blocks until all nodes finish.
intel_pstateis loaded in active mode without hwp and with turbo enabled, andintel_idleis active as well.Intel(R) Xeon(R) Platinum 8167M CPU @ 2.00GHz (Skylake, bare metal) 2 nodes * 26 cores * 2 threads = 104 CPUs 384G/node = 768G memory kernel boot deferred init ------------------------ ------------------------ node% (thr) speedup time_ms (stdev) speedup time_ms (stdev) ( 0) -- 4078.0 ( 9.0) -- 1779.0 ( 8.7) 2% ( 1) 1.4% 4021.3 ( 2.9) 3.4% 1717.7 ( 7.8) 12% ( 6) 35.1% 2644.7 ( 35.3) 80.8% 341.0 ( 35.5) 25% ( 13) 38.7% 2498.0 ( 34.2) 89.1% 193.3 ( 32.3) 37% ( 19) 39.1% 2482.0 ( 25.2) 90.1% 175.3 ( 31.7) 50% ( 26) 38.8% 2495.0 ( 8.7) 89.1% 193.7 ( 3.5) 75% ( 39) 39.2% 2478.0 ( 21.0) 90.3% 172.7 ( 26.7) 100% ( 52) 40.0% 2448.0 ( 2.0) 91.9% 143.3 ( 1.5) Intel(R) Xeon(R) CPU E5-2699C v4 @ 2.20GHz (Broadwell, bare metal) 1 node * 16 cores * 2 threads = 32 CPUs 192G/node = 192G memory kernel boot deferred init ------------------------ ------------------------ node% (thr) speedup time_ms (stdev) speedup time_ms (stdev) ( 0) -- 1996.0 ( 18.0) -- 1104.3 ( 6.7) 3% ( 1) 1.4% 1968.0 ( 3.0) 2.7% 1074.7 ( 9.0) 12% ( 4) 40.1% 1196.0 ( 22.7) 72.4% 305.3 ( 16.8) 25% ( 8) 47.4% 1049.3 ( 17.2) 84.2% 174.0 ( 10.6) 37% ( 12) 48.3% 1032.0 ( 14.9) 86.8% 145.3 ( 2.5) 50% ( 16) 48.9% 1020.3 ( 2.5) 88.0% 133.0 ( 1.7) 75% ( 24) 49.1% 1016.3 ( 8.1) 88.4% 128.0 ( 1.7) 100% ( 32) 49.4% 1009.0 ( 8.5) 88.6% 126.3 ( 0.6) Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz (Haswell, bare metal) 2 nodes * 18 cores * 2 threads = 72 CPUs 128G/node = 256G memory kernel boot deferred init ------------------------ ------------------------ node% (thr) speedup time_ms (stdev) speedup time_ms (stdev) ( 0) -- 1682.7 ( 6.7) -- 630.0 ( 4.6) 3% ( 1) 0.4% 1676.0 ( 2.0) 0.7% 625.3 ( 3.2) 12% ( 4) 25.8% 1249.0 ( 1.0) 68.2% 200.3 ( 1.2) 25% ( 9) 30.0% 1178.0 ( 5.2) 79.7% 128.0 ( 3.5) 37% ( 13) 30.6% 1167.7 ( 3.1) 81.3% 117.7 ( 1.2) 50% ( 18) 30.6% 1167.3 ( 2.3) 81.4% 117.0 ( 1.0) 75% ( 27) 31.0% 1161.3 ( 4.6) 82.5% 110.0 ( 6.9) 100% ( 36) 32.1% 1142.0 ( 3.6) 85.7% 90.0 ( 1.0) AMD EPYC 7551 32-Core Processor (Zen, kvm guest) 1 node * 8 cores * 2 threads = 16 CPUs 64G/node = 64G memory kernel boot deferred init ------------------------ ------------------------ node% (thr) speedup time_ms (stdev) speedup time_ms (stdev) ( 0) -- 1003.7 ( 16.6) -- 243.3 ( 8.1) 6% ( 1) 1.4% 990.0 ( 4.6) 1.2% 240.3 ( 1.5) 12% ( 2) 11.4% 889.3 ( 16.7) 44.5% 135.0 ( 3.0) 25% ( 4) 16.8% 835.3 ( 9.0) 65.8% 83.3 ( 2.5) 37% ( 6) 18.6% 816.7 ( 17.6) 70.4% 72.0 ( 1.0) 50% ( 8) 18.2% 821.0 ( 5.0) 70.7% 71.3 ( 1.2) 75% ( 12) 19.0% 813.3 ( 5.0) 71.8% 68.7 ( 2.1) 100% ( 16) 19.8% 805.3 ( 10.8) 76.4% 57.3 ( 15.9)Server-oriented distros that enable deferred page init sometimes run in small VMs, and they still benefit even though the fraction of boot time saved is smaller:
AMD EPYC 7551 32-Core Processor (Zen, kvm guest) 1 node * 2 cores * 2 threads = 4 CPUs 16G/node = 16G memory kernel boot deferred init ------------------------ ------------------------ node% (thr) speedup time_ms (stdev) speedup time_ms (stdev) ( 0) -- 722.3 ( 9.5) -- 50.7 ( 0.6) 25% ( 1) -3.3% 746.3 ( 4.7) -2.0% 51.7 ( 1.2) 50% ( 2) 0.2% 721.0 ( 11.3) 29.6% 35.7 ( 4.9) 75% ( 3) -0.3% 724.3 ( 11.2) 48.7% 26.0 ( 0.0) 100% ( 4) 3.0% 700.3 ( 13.6) 55.9% 22.3 ( 0.6) Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz (Haswell, kvm guest) 1 node * 2 cores * 2 threads = 4 CPUs 14G/node = 14G memory kernel boot deferred init ------------------------ ------------------------ node% (thr) speedup time_ms (stdev) speedup time_ms (stdev) ( 0) -- 673.0 ( 6.9) -- 57.0 ( 1.0) 25% ( 1) -0.6% 677.3 ( 19.8) 1.8% 56.0 ( 1.0) 50% ( 2) 3.4% 650.0 ( 3.6) 36.8% 36.0 ( 5.2) 75% ( 3) 4.2% 644.7 ( 7.6) 56.1% 25.0 ( 1.0) 100% ( 4) 5.3% 637.0 ( 5.6) 63.2% 21.0 ( 0.0)On Josh's 96-CPU and 192G memory system:
Without this patch series: [ 0.487132] node 0 initialised, 23398907 pages in 292ms [ 0.499132] node 1 initialised, 24189223 pages in 304ms ... [ 0.629376] Run /sbin/init as init process With this patch series: [ 0.227868] node 0 initialised, 23398907 pages in 28ms [ 0.230019] node 1 initialised, 24189223 pages in 28ms ... [ 0.361069] Run /sbin/init as init process[1] https://static.sched.com/hosted_files/kvmforum2019/66/VMM-fast-restart_kvmforum2019.pdf
Signed-off-by Daniel Jordan <[email protected]>--- mm/Kconfig | 6 ++--- mm/page_alloc.c | 60 ++++++++++++++++++++++++++++++++++++++++++++----- 2 files changed, 58 insertions(+), 8 deletions(-)mm/Kconfig +3
-3@@ -750,13 +750,13 @@ config DEFERRED_STRUCT_PAGE_INIT depends on SPARSEMEM depends on !NEED_PER_CPU_KM depends on 64BIT + select PADATA help Ordinarily all struct pages are initialised during early boot in a single thread. On very large machines this can take a considerable amount of time. If this option is set, large machines will bring up - a subset of memmap at boot and then initialise the rest in parallel - by starting one-off "pgdatinitX" kernel thread for each node X. This - has a potential performance impact on processes running early in the + a subset of memmap at boot and then initialise the rest in parallel. + This has a potential performance impact on tasks running early in the lifetime of the system until these kthreads finish the initialisation.mm/page_alloc.c +19
-0@@ -68,6 +68,7 @@#include <linux/lockdep.h> #include <linux/nmi.h> #include <linux/psi.h> +#include <linux/padata.h> #include <asm/sections.h> #include <asm/tlbflush.h> @@ -1814,16 +1815,44 @@ deferred_init_maxorder(u64 *i, struct zone *zone, unsigned long *start_pfn,return nr_pages; } +struct definit_args { +struct zone *zone; +atomic_long_t nr_pages; +}; + +static void __init +deferred_init_memmap_chunk (unsigned long start_pfn,unsigned long end_pfn, +void *arg) +{ +unsigned long spfn, epfn, nr_pages =0 ; +struct definit_args *args = arg; +struct zone *zone = args->zone; +u64 i; + +deferred_init_mem_pfn_range_in_zone (&i, zone, &spfn, &epfn, start_pfn); + +/* +* Initialize and free pages in MAX_ORDER sized increments so that we mm/page_alloc.c deferred_init_maxorder(u64 *i, struct zone *zone, unsigned long *start_pfn, · line 1845
Quoted 10 lines
mm/page_alloc.c +10
-0+* can avoid introducing any issues with the buddy allocator. +*/ +while (spfn < end_pfn) { + nr_pages +=deferred_init_maxorder (&i, zone, &spfn, &epfn); +cond_resched (); + } + +atomic_long_add (nr_pages, &args->nr_pages); +} +Personally I would get rid of
nr_pagesentirely. It isn't worth the cache thrash to have this atomic variable bouncing around. You could probably just have this function return void since allnr_pagesis used for is apr_infostatement at the end of the initialization which will be completely useless now anyway since we really have the threads running in parallel anyway.We only really need the
nr_pageslogic indeferred_grow_zonein order to track if we have freed enough pages to allow us to go back to what we were doing.Quoted 41 lines
mm/page_alloc.c +19
-5/* Initialise remaining memory on a node */ static int __init deferred_init_memmap (void *data) {pg_data_t *pgdat = data;const struct cpumask *cpumask =cpumask_of_node (pgdat->node_id);unsigned long spfn =0 , epfn =0 , nr_pages =0 ; -unsigned long first_init_pfn, flags; +unsigned long first_init_pfn, flags, epfn_align;unsigned long start = jiffies;struct zone *zone; -int zid; +int zid, max_threads;u64 i;/* Bind memory initialisation thread to a local node if possible */ @@ -1863,11 +1892,32 @@ static int __init deferred_init_memmap(void *data) goto zone_empty;/* -* Initialize and free pages in MAX_ORDER sized increments so -* that we can avoid introducing any issues with the buddy -* allocator. +* More CPUs always led to greater speedups on tested systems, up to +* all the nodes' CPUs. Use all since the system is otherwise idle now. */ + max_threads =max (cpumask_weight (cpumask),1u ); +while (spfn < epfn) { + epfn_align =ALIGN_DOWN (epfn,PAGES_PER_SECTION ); + +if (IS_ALIGNED (spfn,PAGES_PER_SECTION ) && + epfn_align - spfn >=PAGES_PER_SECTION ) { +struct definit_args arg = { zone,ATOMIC_LONG_INIT (0 ) }; +struct padata_mt_job job = { + .thread_fn = deferred_init_memmap_chunk, + .fn_arg = &arg, + .start = spfn, + .size = epfn_align - spfn, + .align =PAGES_PER_SECTION , + .min_chunk =PAGES_PER_SECTION , + .max_threads = max_threads,mm/page_alloc.c static int __init deferred_init_memmap(void *data) · line 1923
Quoted 10 lines
mm/page_alloc.c +7
-0+ }; + +padata_do_multithreaded (&job); + nr_pages +=atomic_long_read (&arg.nr_pages); + spfn = epfn_align; + } + nr_pages +=deferred_init_maxorder (&i, zone, &spfn, &epfn);cond_resched (); }This doesn't look right. You are basically adding threads in addition to calls to
deferred_init_maxorder. In addition you are spawning one job per section instead of per range. Really you should be going for something more along the lines of:while (spfn < epfn) {
unsigned longepfn_align= ALIGN(epfn,PAGE_PER_SECTION);
structdefinit_argsarg = { zone,ATOMIC_LONG_INIT(0)
};
structpadata_mt_jobjob = {
.thread_fn =deferred_init_memmap_chunk,
.fn_arg = &arg,
.start = spfn,
.size =epfn_align- spfn,
.align =PAGES_PER_SECTION,
.min_chunk =PAGES_PER_SECTION,
.max_threads =max_threads,
};
Quoted 9 lines
padata_do_multithreaded(&job);for_each_free_mem_pfn_range_in_zone_from(i, zone,
spfn, epfn) { if (epfn_align <= spfn) break; } }
So I was thinking about my suggestion further and the loop at the end isn't quite correct as I believe it could lead to gaps. The loop on the end should probably be:
for_each_free_mem_pfn_range_in_zone_from(i, zone, spfn, epfn) {
if (epfn <= epfn_align)
continue;
if (spfn < epfn_align)
spfn = epfn_align;
break;
}That would generate a new range where epfn_align has actually ended and there is a range of new PFNs to process.
Thanks.
- Alex
Daniel Jordan
maintainerRe: [PATCH v2 5/7] mm: parallelize deferred_init_memmap()
Quoted 11 lines · Alexander Duyck
So I was thinking about my suggestion further and the loop at the end isn't quite correct as I believe it could lead to gaps. The loop on the end should probably be:
for_each_free_mem_pfn_range_in_zone_from(i, zone, spfn, epfn) { if (epfn <= epfn_align) continue; if (spfn < epfn_align) spfn = epfn_align; break; }
Quoted 2 lines
That would generate a new range where
epfn_alignhas actually ended and there is a range of new PFNs to process.
Whoops, my email crossed with yours. Agreed, but see the other message.
Daniel Jordan
maintainerRe: [PATCH v2 5/7] mm: parallelize deferred_init_memmap()
In reply to Alexander DuyckQuoted 28 lines · Alexander Duyck
On Wed, May 20, 2020 at 11:27 AM Daniel Jordan
Quoted 27 lines
mm/page_alloc.c +23
-0@@ -1814,16 +1815,44 @@ deferred_init_maxorder(u64 *i, struct zone *zone, unsigned long *start_pfn,return nr_pages; } +struct definit_args { +struct zone *zone; +atomic_long_t nr_pages; +}; + +static void __init +deferred_init_memmap_chunk (unsigned long start_pfn,unsigned long end_pfn, +void *arg) +{ +unsigned long spfn, epfn, nr_pages =0 ; +struct definit_args *args = arg; +struct zone *zone = args->zone; +u64 i; + +deferred_init_mem_pfn_range_in_zone (&i, zone, &spfn, &epfn, start_pfn); + +/* +* Initialize and free pages in MAX_ORDER sized increments so that we +* can avoid introducing any issues with the buddy allocator. +*/ +while (spfn < end_pfn) { + nr_pages +=deferred_init_maxorder (&i, zone, &spfn, &epfn); +cond_resched ();
Quoted 8 lines
mm/page_alloc.c deferred_init_maxorder(u64 *i, struct zone *zone, unsigned long *start_pfn, · line 1845
Quoted 5 lines
mm/page_alloc.c +5
-0+ } + +atomic_long_add (nr_pages, &args->nr_pages); +} +Personally I would get rid of
nr_pagesentirely. It isn't worth the cache thrash to have this atomic variable bouncing around.
One of the things I tried to optimize was the managed_pages atomic adds in __free_pages_core, but performance stayed the same on the biggest machine I tested when it was done once at the end of page init instead of in every thread for every pageblock.
I'm not sure this atomic would matter either, given it's less frequent.
Quoted 5 lines
You could
probably just have this function return void since allnr_pagesis used for is apr_infostatement at the end of the initialization which will be completely useless now anyway since we really have the threads running in parallel anyway.
The timestamp is still useful for observability, page init is a significant part of kernel boot on big machines, over 10% sometimes with these patches.
It's mostly the time that matters though, I agree the number of pages is less important and is probably worth removing just to simplify the code. I'll do it if no one sees a reason to keep it.
Quoted 35 lines
We only really need the
nr_pageslogic indeferred_grow_zonein order to track if we have freed enough pages to allow us to go back to what we were doing.Quoted 31 lines
mm/page_alloc.c +22
-3@@ -1863,11 +1892,32 @@ static int __init deferred_init_memmap(void *data) goto zone_empty;/* -* Initialize and free pages in MAX_ORDER sized increments so -* that we can avoid introducing any issues with the buddy -* allocator. +* More CPUs always led to greater speedups on tested systems, up to +* all the nodes' CPUs. Use all since the system is otherwise idle now. */ + max_threads =max (cpumask_weight (cpumask),1u ); +while (spfn < epfn) { + epfn_align =ALIGN_DOWN (epfn,PAGES_PER_SECTION ); + +if (IS_ALIGNED (spfn,PAGES_PER_SECTION ) && + epfn_align - spfn >=PAGES_PER_SECTION ) { +struct definit_args arg = { zone,ATOMIC_LONG_INIT (0 ) }; +struct padata_mt_job job = { + .thread_fn = deferred_init_memmap_chunk, + .fn_arg = &arg, + .start = spfn, + .size = epfn_align - spfn, + .align =PAGES_PER_SECTION , + .min_chunk =PAGES_PER_SECTION , + .max_threads = max_threads, + }; + +padata_do_multithreaded (&job); + nr_pages +=atomic_long_read (&arg.nr_pages); + spfn = epfn_align;
Quoted 8 lines
mm/page_alloc.c static int __init deferred_init_memmap(void *data) · line 1923
Quoted 5 lines
mm/page_alloc.c +2
-0+ } + nr_pages +=deferred_init_maxorder (&i, zone, &spfn, &epfn);cond_resched (); }This doesn't look right. You are basically adding threads in addition to calls to
deferred_init_maxorder.
The deferred_init_maxorder call is there to do the remaining, non-section aligned part of a range. It doesn't have to be done this way.
Quoted 2 lines
In addition you are spawning one
job per section instead of per range.
That's not what's happening, all the above is doing is aligning the end of the range down to a section. Each thread is working on way more than a section at a time.
Quoted 18 lines
Really you should be going for
something more along the lines of:while (spfn < epfn) {
unsigned longepfn_align= ALIGN(epfn,PAGE_PER_SECTION);
structdefinit_argsarg = { zone,ATOMIC_LONG_INIT(0)
};
structpadata_mt_jobjob = {
.thread_fn =deferred_init_memmap_chunk,
.fn_arg = &arg,
.start = spfn,
.size =epfn_align- spfn,
.align =PAGES_PER_SECTION,
.min_chunk =PAGES_PER_SECTION,
.max_threads =max_threads,
};
Quoted 8 lines
padata_do_multithreaded(&job);for_each_free_mem_pfn_range_in_zone_from(i, zone,
spfn, epfn) { if (epfn_align <= spfn) break; } }
I can see what you're getting at even though I think this can leave ranges uninitialized. Starting with range [a,b), b is aligned up to d and the inner loop skips [c,e).
a b c d e
| | | section boundaries [ ) [ )
We could use deferred_init_mem_pfn_range_in_zone() instead of the for_each loop.
What I was trying to avoid by aligning down is creating a discontiguous pfn range that get passed to padata. We already discussed how those are handled by the zone iterator in the thread function, but job->size can be exaggerated to include parts of the range that are never touched. Thinking more about it though, it's a small fraction of the total work and shouldn't matter.
Quoted 1 line
This should accomplish the same thing, but much more efficiently.
Well, more cleanly. I'll give it a try.
Quoted 8 lines
The
only thing you really lose is the tracking ofnr_pageswhich really doesn't add anything anyway since the value could shift around depending on how many timesdeferred_grow_zonegot called anyway.Also the spfn should already be sectioned aligned, or at least be in a new section unrelated to the one we just scheduled, so there is no need for the extra checks you had.
I was doing it to be robust to future changes. Otherwise epfn_align - spfn could be huge when aligning down, but with aligning up it won't matter and can be removed.
Re: [PATCH v2 5/7] mm: parallelize deferred_init_memmap()
Quoted 39 lines · Daniel Jordan
Quoted 28 lines · Alexander Duyck
On Wed, May 20, 2020 at 11:27 AM Daniel Jordan
Quoted 27 lines
mm/page_alloc.c +23
-0@@ -1814,16 +1815,44 @@ deferred_init_maxorder(u64 *i, struct zone *zone, unsigned long *start_pfn,return nr_pages; } +struct definit_args { +struct zone *zone; +atomic_long_t nr_pages; +}; + +static void __init +deferred_init_memmap_chunk (unsigned long start_pfn,unsigned long end_pfn, +void *arg) +{ +unsigned long spfn, epfn, nr_pages =0 ; +struct definit_args *args = arg; +struct zone *zone = args->zone; +u64 i; + +deferred_init_mem_pfn_range_in_zone (&i, zone, &spfn, &epfn, start_pfn); + +/* +* Initialize and free pages in MAX_ORDER sized increments so that we +* can avoid introducing any issues with the buddy allocator. +*/ +while (spfn < end_pfn) { + nr_pages +=deferred_init_maxorder (&i, zone, &spfn, &epfn); +cond_resched ();Quoted 8 lines
mm/page_alloc.c deferred_init_maxorder(u64 *i, struct zone *zone, unsigned long *start_pfn, · line 1845
Quoted 5 lines
mm/page_alloc.c +5
-0+ } + +atomic_long_add (nr_pages, &args->nr_pages); +} +Personally I would get rid of
nr_pagesentirely. It isn't worth the cache thrash to have this atomic variable bouncing around.
Quoted 6 lines
One of the things I tried to optimize was the
managed_pagesatomic adds in__free_pages_core, but performance stayed the same on the biggest machine I tested when it was done once at the end of page init instead of in every thread for every pageblock.I'm not sure this atomic would matter either, given it's less frequent.
It is more about not bothering with the extra tracking. We don't really need it and having it doesn't really add much in the way of value.
Quoted 8 lines
Quoted 5 lines
You could
probably just have this function return void since allnr_pagesis used for is apr_infostatement at the end of the initialization which will be completely useless now anyway since we really have the threads running in parallel anyway.The timestamp is still useful for observability, page init is a significant part of kernel boot on big machines, over 10% sometimes with these patches.
Agreed.
Quoted 3 lines
It's mostly the time that matters though, I agree the number of pages is less important and is probably worth removing just to simplify the code. I'll do it if no one sees a reason to keep it.
Sounds good.
Quoted 44 lines
Quoted 35 lines
We only really need the
nr_pageslogic indeferred_grow_zonein order to track if we have freed enough pages to allow us to go back to what we were doing.Quoted 31 lines
mm/page_alloc.c +22
-3@@ -1863,11 +1892,32 @@ static int __init deferred_init_memmap(void *data) goto zone_empty;/* -* Initialize and free pages in MAX_ORDER sized increments so -* that we can avoid introducing any issues with the buddy -* allocator. +* More CPUs always led to greater speedups on tested systems, up to +* all the nodes' CPUs. Use all since the system is otherwise idle now. */ + max_threads =max (cpumask_weight (cpumask),1u ); +while (spfn < epfn) { + epfn_align =ALIGN_DOWN (epfn,PAGES_PER_SECTION ); + +if (IS_ALIGNED (spfn,PAGES_PER_SECTION ) && + epfn_align - spfn >=PAGES_PER_SECTION ) { +struct definit_args arg = { zone,ATOMIC_LONG_INIT (0 ) }; +struct padata_mt_job job = { + .thread_fn = deferred_init_memmap_chunk, + .fn_arg = &arg, + .start = spfn, + .size = epfn_align - spfn, + .align =PAGES_PER_SECTION , + .min_chunk =PAGES_PER_SECTION , + .max_threads = max_threads, + }; + +padata_do_multithreaded (&job); + nr_pages +=atomic_long_read (&arg.nr_pages); + spfn = epfn_align;Quoted 8 lines
mm/page_alloc.c static int __init deferred_init_memmap(void *data) · line 1923
Quoted 5 lines
mm/page_alloc.c +2
-0+ } + nr_pages +=deferred_init_maxorder (&i, zone, &spfn, &epfn);cond_resched (); }This doesn't look right. You are basically adding threads in addition to calls to
deferred_init_maxorder.
Quoted 2 lines
The
deferred_init_maxordercall is there to do the remaining, non-section aligned part of a range. It doesn't have to be done this way.
It is also doing the advancing though isn't it?
Quoted 6 lines
Quoted 2 lines
In addition you are spawning one
job per section instead of per range.That's not what's happening, all the above is doing is aligning the end of the range down to a section. Each thread is working on way more than a section at a time.
Yeah, now that I reread it I see that. For some reason I was thinking you were aligning spfn, not epfn.
Quoted 27 lines
Quoted 18 lines
Really you should be going for
something more along the lines of:while (spfn < epfn) {
unsigned longepfn_align= ALIGN(epfn,PAGE_PER_SECTION);
structdefinit_argsarg = { zone,ATOMIC_LONG_INIT(0)
};
structpadata_mt_jobjob = {
.thread_fn =deferred_init_memmap_chunk,
.fn_arg = &arg,
.start = spfn,
.size =epfn_align- spfn,
.align =PAGES_PER_SECTION,
.min_chunk =PAGES_PER_SECTION,
.max_threads =max_threads,
};Quoted 8 lines
padata_do_multithreaded(&job);for_each_free_mem_pfn_range_in_zone_from(i, zone,
spfn, epfn) { if (epfn_align <= spfn) break; } }
Quoted 7 lines
I can see what you're getting at even though I think this can leave ranges uninitialized. Starting with range [a,b), b is aligned up to d and the inner loop skips [c,e).
a b c d e
| | | section boundaries [ ) [ )
I think I resolved this with the fix for it I described in the other email. We just need to swap out spfn for epfn and make sure we align spfn with epfn_align. Then I think that takes care of possible skips.
Quoted 8 lines
We could use
deferred_init_mem_pfn_range_in_zone()instead of thefor_eachloop.What I was trying to avoid by aligning down is creating a discontiguous pfn range that get passed to padata. We already discussed how those are handled by the zone iterator in the thread function, but
job->sizecan be exaggerated to include parts of the range that are never touched. Thinking more about it though, it's a small fraction of the total work and shouldn't matter.
So the problem with aligning down is that you are going to be slowed up as you have to go single threaded to initialize whatever remains. So worst case scenario is that you have a section aligned block and you will process all but 1 section in parallel, and then have to process the remaining section one max order block at a time.
Quoted 3 lines
Quoted 1 line
This should accomplish the same thing, but much more efficiently.
Well, more cleanly. I'll give it a try.
I agree I am not sure if it will make a big difference on x86, however the more ranges you have to process the faster this approach should be as it stays parallel the entire time rather than having to drop out and process the last section one max order block at a time.
Quoted 5 lines
Quoted 5 lines
The
only thing you really lose is the tracking ofnr_pageswhich really doesn't add anything anyway since the value could shift around depending on how many timesdeferred_grow_zonegot called anyway.
Quoted 7 lines
Quoted 3 lines
Also the spfn should already be sectioned aligned, or at least be in a new section unrelated to the one we just scheduled, so there is no need for the extra checks you had.
I was doing it to be robust to future changes. Otherwise
epfn_align- spfn could be huge when aligning down, but with aligning up it won't matter and can be removed.
Right. So that was to catch the case where you could potentially align down below spfn.
Daniel Jordan
maintainerRe: [PATCH v2 5/7] mm: parallelize deferred_init_memmap()
Quoted 3 lines · Alexander Duyck
It is more about not bothering with the extra tracking. We don't really need it and having it doesn't really add much in the way of value.
Yeah, it can probably go.
Quoted 40 lines
Quoted 40 lines
Quoted 39 lines
Quoted 26 lines
mm/page_alloc.c +17
-3@@ -1863,11 +1892,32 @@ static int __init deferred_init_memmap(void *data) goto zone_empty;/* -* Initialize and free pages in MAX_ORDER sized increments so -* that we can avoid introducing any issues with the buddy -* allocator. +* More CPUs always led to greater speedups on tested systems, up to +* all the nodes' CPUs. Use all since the system is otherwise idle now. */ + max_threads =max (cpumask_weight (cpumask),1u ); +while (spfn < epfn) { + epfn_align =ALIGN_DOWN (epfn,PAGES_PER_SECTION ); + +if (IS_ALIGNED (spfn,PAGES_PER_SECTION ) && + epfn_align - spfn >=PAGES_PER_SECTION ) { +struct definit_args arg = { zone,ATOMIC_LONG_INIT (0 ) }; +struct padata_mt_job job = { + .thread_fn = deferred_init_memmap_chunk, + .fn_arg = &arg, + .start = spfn, + .size = epfn_align - spfn, + .align =PAGES_PER_SECTION , + .min_chunk =PAGES_PER_SECTION , + .max_threads = max_threads,mm/page_alloc.c static int __init deferred_init_memmap(void *data) · line 1923
Quoted 10 lines
mm/page_alloc.c +7
-0+ }; + +padata_do_multithreaded (&job); + nr_pages +=atomic_long_read (&arg.nr_pages); + spfn = epfn_align; + } + nr_pages +=deferred_init_maxorder (&i, zone, &spfn, &epfn);cond_resched (); }This doesn't look right. You are basically adding threads in addition to calls to
deferred_init_maxorder.
Quoted 4 lines
Quoted 2 lines
The
deferred_init_maxordercall is there to do the remaining, non-section aligned part of a range. It doesn't have to be done this way.It is also doing the advancing though isn't it?
Yes. Not sure what you're getting at. There's the 'spfn = epfn_align' before so nothing is skipped. It's true that the nonaligned part is done outside of padata when it could be done by a thread that'd otherwise be waiting or idle, which should be addressed in the next version.
Quoted 3 lines
I think I resolved this with the fix for it I described in the other email. We just need to swap out spfn for epfn and make sure we align spfn with
epfn_align. Then I think that takes care of possible skips.
Right, though your fix looks a lot like deferred_init_mem_pfn_range_in_zone(). Seems better to just use that and not repeat ourselves. Lame that it's starting at the beginning of the ranges every time, maybe it could be generalized somehow, but I think it should be fast enough.
Quoted 9 lines
Quoted 8 lines
We could use
deferred_init_mem_pfn_range_in_zone()instead of thefor_eachloop.What I was trying to avoid by aligning down is creating a discontiguous pfn range that get passed to padata. We already discussed how those are handled by the zone iterator in the thread function, but
job->sizecan be exaggerated to include parts of the range that are never touched. Thinking more about it though, it's a small fraction of the total work and shouldn't matter.
Quoted 5 lines
So the problem with aligning down is that you are going to be slowed up as you have to go single threaded to initialize whatever remains. So worst case scenario is that you have a section aligned block and you will process all but 1 section in parallel, and then have to process the remaining section one max order block at a time.
Yes, aligning up is better.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
This should accomplish the same thing, but much more efficiently.
Well, more cleanly. I'll give it a try.
I agree I am not sure if it will make a big difference on x86, however the more ranges you have to process the faster this approach should be as it stays parallel the entire time rather than having to drop out and process the last section one max order block at a time.
Right.
Daniel Jordan
maintainerPATCH v2 6/7 mm: make deferred init's max threads arch-specific
In reply to Daniel JordanUsing padata during deferred init has only been tested on x86, so for now limit it to this architecture.
If another arch wants this, it can find the max thread limit that's best for it and override deferred_page_init_max_threads().
--- arch/x86/mm/init_64.c | 12 ++++++++++++ include/linux/memblock.h | 3 +++ mm/page_alloc.c | 13 ++++++++----- 3 files changed, 23 insertions(+), 5 deletions(-)
arch/x86/mm/init_64.c +12 -0
@@ -1260,6 +1260,18 @@ void __init mem_init(void)mem_init_print_info (NULL ); } +#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT +int __init deferred_page_init_max_threads (const struct cpumask *node_cpumask) +{ +/* +* More CPUs always led to greater speedups on tested systems, up to +* all the nodes' CPUs. Use all since the system is otherwise idle +* now. +*/ +return max_t (int,cpumask_weight (node_cpumask),1 ); +} +#endif +int kernel_set_to_readonly;void mark_rodata_ro (void )
include/linux/memblock.h +3 -0
@@ -275,6 +275,9 @@ void __next_mem_pfn_range_in_zone(u64 *idx, struct zone *zone,#define for_each_free_mem_pfn_range_in_zone_from (i, zone, p_start, p_end) \ for (; i != U64_MAX; \ __next_mem_pfn_range_in_zone(&i, zone, p_start, p_end)) + +int __init deferred_page_init_max_threads (const struct cpumask *node_cpumask); +#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ /**
mm/page_alloc.c +8 -5
@@ -1843,6 +1843,13 @@ deferred_init_memmap_chunk(unsigned long start_pfn, unsigned long end_pfn,atomic_long_add (nr_pages, &args->nr_pages); } +/* An arch may override for more concurrency. */ +__weak int __init +deferred_page_init_max_threads (const struct cpumask *node_cpumask) +{ +return 1 ; +} +/* Initialise remaining memory on a node */ static int __init deferred_init_memmap (void *data) {
@@ -1891,11 +1898,7 @@ static int __init deferred_init_memmap(void *data) first_init_pfn)) goto zone_empty; -/* -* More CPUs always led to greater speedups on tested systems, up to -* all the nodes' CPUs. Use all since the system is otherwise idle now. -*/ - max_threads =max (cpumask_weight (cpumask),1u ); + max_threads =deferred_page_init_max_threads (cpumask);while (spfn < epfn) { epfn_align =ALIGN_DOWN (epfn,PAGES_PER_SECTION );
Signature · 1 line
2.26.2
Daniel Jordan
maintainerPATCH v2 7/7 padata: document multithreaded jobs
In reply to Daniel JordanAdd Documentation for multithreaded jobs.
--- Documentation/core-api/padata.rst | 41 +++++++++++++++++++++++-------- 1 file changed, 31 insertions(+), 10 deletions(-)
Documentation/core-api/padata.rst +31 -10
@@ -4,23 +4,26 @@ The padata parallel execution mechanism ======================================= -:Date: December 2019 +:Date: April 2020 Padata is a mechanism by which the kernel can farm jobs out to be done in -parallel on multiple CPUs while retaining their ordering. It was developed for -use with the IPsec code, which needs to be able to perform encryption and -decryption on large numbers of packets without reordering those packets. The -crypto developers made a point of writing padata in a sufficiently general -fashion that it could be put to other uses as well. +parallel on multiple CPUs while optionally retaining their ordering. -Usage -===== +It was originally developed for IPsec, which needs to perform encryption and +decryption on large numbers of packets without reordering those packets. This +is currently the sole consumer of padata's serialized job support. + +Padata also supports multithreaded jobs, splitting up the job evenly while load +balancing and coordinating between threads. + +Running Serialized Jobs +======================= Initializing ------------ -The first step in using padata is to set up a padata_instance structure for -overall control of how jobs are to be run:: +The first step in using padata to run parallel jobs is to set up a +padata_instance structure for overall control of how jobs are to be run:: #include <linux/padata.h>
@@ -162,6 +165,24 @@ functions that correspond to the allocation in reverse:: It is the user's responsibility to ensure all outstanding jobs are complete before any of the above are called. +Running Multithreaded Jobs +========================== + +A multithreaded job has a main thread and zero or more helper threads, with the +main thread participating in the job and then waiting until all helpers have +finished. padata splits the job into units called chunks, where a chunk is a +piece of the job that one thread completes in one call to the thread function. + +A user has to do three things to run a multithreaded job. First, describe the +job by defining a padata_mt_job structure, which is explained in the Interface +section. This includes a pointer to the thread function, which padata will +call each time it assigns a job chunk to a thread. Then, define the thread +function, which accepts three arguments, ``start``, ``end``, and ``arg``, where +the first two delimit the range that the thread operates on and the last is a +pointer to the job's shared state, if any. Prepare the shared state, which is +typically a stack-allocated structure that wraps the required data. Last, call +padata_do_multithreaded(), which will return once the job is finished. + Interface =========
Signature · 1 line
2.26.2
Quoted 2 lines · Daniel Jordan
Please review and test, and thanks to Alex, Andrew, Josh, and Pavel for their feedback in the last version.
I re-tested v2:
[ 0.231435] node 1 initialised, 24189223 pages in 32ms [ 0.236718] node 0 initialised, 23398907 pages in 36ms