Delay initializing of large sections of memory
Subsystems (5)
cover Delay initializing of large sections of memory
This rfc patch set delays initializing large sections of memory until we have started cpus. This has the effect of reducing startup times on large memory systems. On 16TB it can take over an hour to boot and most of that time is spent initializing memory.
We avoid that bottleneck by delaying initialization until after we have started multiple cpus and can initialize in a multithreaded manner. This allows us to actually reduce boot time rather then just moving around the point of initialization.
Mike and I have worked on this set for a while, with him doing the most of the heavy lifting, and are eager for some feedback.
Mike Travis (2):
x86_64, mm: Delay initializing large portion of memory x86_64, mm: Reinsert the absent memory
Documentation/kernel-parameters.txt | 15 ++ arch/x86/Kconfig | 10 ++ arch/x86/include/asm/e820.h | 16 +- arch/x86/kernel/e820.c | 292 +++++++++++++++++++++++++++++++++++- drivers/base/memory.c | 83 ++++++++++ include/linux/memory.h | 5 + 6 files changed, 413 insertions(+), 8 deletions(-)
Signature · 1 line
1.8.2.1
RFC 1/2 x86_64, mm: Delay initializing large portion of memory
On a 16TB system it can takes upwards of two hours to boot the system with about 60% of the time being spent initializing memory. This patch delays initializing a large portion of memory until after the system is booted. This can significantly reduce the time it takes the boot the system down to the 15 to 30 minute range.
--- Documentation/kernel-parameters.txt | 15 ++++ arch/x86/Kconfig | 10 +++ arch/x86/include/asm/e820.h | 16 +++- arch/x86/kernel/e820.c | 163 ++++++++++++++++++++++++++++++++++-- 4 files changed, 196 insertions(+), 8 deletions(-)
Documentation/kernel-parameters.txt +15 -0
@@ -706,6 +706,21 @@ bytes respectively. Such letter suffixes can also be entirely omitted. Defaults to the default architecture's huge page size if not specified. + delay_mem_init=B:M:n:l:h + This delays the initialization of a large portion of + memory by inserting it into the "absent" memory list. + This allows the system to boot up much faster at the + expense of the time needed to add this absent memory + after the system has booted. That however can be done + in parallel with other operations. + Format: B:M:n:l:h + (1 << B) is the block size (bsize) + ['0' indicates use the default 128M] + (1 << M) is the address space per node + (n * bsize) is minimum sized node memory to slice + (l * bisze) is low memory to leave on node + (h * bisze) is high memory to leave on node + dhash_entries= [KNL] Set number of hash buckets for dentry cache.
arch/x86/Kconfig +10 -0
@@ -1566,6 +1566,16 @@ config EFI_STUB See Documentation/x86/efi-stub.txt for more information. +config DELAY_MEM_INIT + bool "Delay memory initialization" + depends on EFI && MEMORY_HOTPLUG_SPARSE + ---help--- + This option delays initializing a large portion of memory + until after the system is booted. This can significantly + reduce the time it takes the boot the system when there + is a significant amount of memory present. Systems with + 8TB or more of memory benefit the most. + config SECCOMP def_bool y prompt "Enable seccomp to safely compute untrusted bytecode"
arch/x86/include/asm/e820.h +14 -2
@@ -18,8 +18,6 @@ extern int e820_any_mapped(u64 start, u64 end, unsigned type);extern int e820_all_mapped (u64 start,u64 end,unsigned type);extern void e820_add_region (u64 start,u64 size,int type);extern void e820_print_map (char *who); -extern int -sanitize_e820_map (struct e820entry *biosmap,int max_nr_map,u32 *pnr_map);extern u64 e820_update_range (u64 start,u64 size,unsigned old_type,unsigned new_type); extern u64 e820_remove_range(u64 start, u64 size, unsigned old_type,
@@ -31,6 +29,20 @@ extern int e820_search_gap(unsigned long *gapstart, unsigned long *gapsize,struct setup_data ;extern void parse_e820_ext (struct setup_data *data); +extern int +__sanitize_e820_map (struct e820entry *biosmap,int max_nr_map,u32 *pnr_map); + +#ifdef CONFIG_DELAY_MEM_INIT +extern int +sanitize_e820_map (struct e820entry *biosmap,int max_nr_map,u32 *pnr_map); +#else +static inline int sanitize_e820_map (struct e820entry *biosmap,int max_nr_map, +u32 *pnr_map) +{ +return __sanitize_e820_map (biosmap, max_nr_map, pnr_map); +} +#endif /* CONFIG_DELAY_MEM_INIT */ +#if defined(CONFIG_X86_64 ) || \ (defined(CONFIG_X86_32 ) && defined(CONFIG_HIBERNATION ))extern void e820_mark_nosave_regions (unsigned long limit_pfn);
arch/x86/kernel/e820.c +157 -6
202 lines not on this page · raw message
Signature · 1 line
1.8.2.1
Re: [RFC 1/2] x86_64, mm: Delay initializing large portion of memory
Quoted 5 lines · Nathan Zimmer
On a 16TB system it can takes upwards of two hours to boot the system with about 60% of the time being spent initializing memory. This patch delays initializing a large portion of memory until after the system is booted. This can significantly reduce the time it takes the boot the system down to the 15 to 30 minute range.
Why is this conditional? Initialize the minimum amount of memory to bring up each NUMA node, and then have each processor initialize its own memory. I would have thought it was already doing this...
Documentation/kernel-parameters.txt bytes respectively. Such letter suffixes can also be entirely omitted. · line 715
Quoted 12 lines
(quoted diff) +2
-0+ delay_mem_init=B:M:n:l:h + This delays the initialization of a largeportion of
(quoted diff) +1
-0+ memory by inserting it into the "absent" memorylist.
(quoted diff) +1
-0+ This allows the system to boot up much fasterat the
(quoted diff) +1
-0+ expense of the time needed to add this absentmemory
(quoted diff) +1
-0+ after the system has booted. That however canbe done
Documentation/kernel-parameters.txt +1
-0+ in parallel with other operations.
This seems like a giant advertisement primarily aimed at repeating why you think we need to merge the patch, not explaining what it is or how to use it.
I would rephrase:
Defer memory initialization until after SMP
init (so
large memory ranges can be initialized in
parallel) by
moving memory not needed during boot to the
"absent" list.
And I repeat: why do we need to micromanage this? It sounds like all NUMA systems should do something like this. (Single-threaded memory initialization in an SMP system is kind of weird.)
Documentation/kernel-parameters.txt bytes respectively. Such letter suffixes can also be entirely omitted. · line 722
Quoted 9 lines
Documentation/kernel-parameters.txt +3
-0+ Format: B:M:n:l:h + (1 << B) is the block size (bsize) + ['0' indicates use the default128M]
(quoted diff) +2
-0+ (1 << M) is the address space per node + (n * bsize) is minimum sized node memory toslice
Documentation/kernel-parameters.txt +2
-0+ (l * bisze) is low memory to leave on node + (h * bisze) is high memory to leave on node
I don't understand this in the slightest. I understand "low memory to leave on the node", I have no idea why there are four other parameters.
arch/x86/Kconfig config EFI_STUB · line 1577
Quoted 9 lines
arch/x86/Kconfig +9
-0+config DELAY_MEM_INIT + bool "Delay memory initialization" + depends on EFI && MEMORY_HOTPLUG_SPARSE + ---help--- + This option delays initializing a large portion of memory + until after the system is booted. This can significantly + reduce the time it takes the boot the system when there + is a significant amount of memory present. Systems with + 8TB or more of memory benefit the most.
I can see an SMP phone wanting to use this to shave a quarter second off its boot time. Your "large portion of memory" description is a bit myopic.
Rob
RFC 2/2 x86_64, mm: Reinsert the absent memory
In reply to Nathan ZimmerThe memory we set aside in the previous patch needs to be reinserted. We start this process via late_initcall so we will have multiple cpus to do the work.
--- arch/x86/kernel/e820.c | 129 +++++++++++++++++++++++++++++++++++++++++++++++++ drivers/base/memory.c | 83 +++++++++++++++++++++++++++++++ include/linux/memory.h | 5 ++ 3 files changed, 217 insertions(+)
arch/x86/kernel/e820.c +129 -0
@@ -23,6 +23,7 @@#ifdef CONFIG_DELAY_MEM_INIT #include <linux/memory.h> +#include <linux/delay.h> #endif #include <asm/e820.h>
@@ -397,6 +398,22 @@ static u64 min_region_size; /* min size of region to slice from */static u64 pre_region_size;/* multiply bsize for node low memory */ static u64 post_region_size;/* multiply bsize for node high memory */ +static unsigned long add_absent_work_start_time; +static unsigned long add_absent_work_stop_time; +static unsigned int add_absent_job_count; +static atomic_t add_absent_work_count; + +struct absent_work { +struct work_struct work; +struct absent_work *next; +atomic_t busy; +int cpu; +int node; +int index; +}; +static DEFINE_PER_CPU(struct absent_work, absent_work); +static struct absent_work *first_absent_work; +static int __init setup_delay_mem_init (char *str) {int bbits, mpnbits, minmult, premult, postmult;
@@ -527,6 +544,118 @@ int __init sanitize_e820_map(struct e820entry *biosmap, int max_nr_map, }return ret; } + +/* Assign a cpu for this memory chunk and get the per_cpu absent_work struct */ +static struct absent_work *get_absent_work (int node) +{ +int cpu; + +for_each_cpu (cpu,cpumask_of_node (node)) { +struct absent_work *aws = &per_cpu (absent_work, cpu); +if (aws->node) +continue ; + aws->cpu = cpu; + aws->node = node; +return aws; + } + +/* (if this becomes a problem, we can use a cpu on another node) */ +pr_crit ("e820: No CPU on Node %d to schedule absent_work\n" , node); +return NULL ; +} + +/* Count of 'not done' processes */ +static int count_absent_work_notdone (void ) +{ +struct absent_work *aws; +int notdone =0 ; + +for (aws = first_absent_work; aws; aws = aws->next) +if (atomic_read (&aws->busy) <2 ) + notdone++; + +return notdone; +} + +/* The absent_work thread */ +static void add_absent_memory_work (struct work_struct *work) +{ +struct absent_work *aws; +u64 phys_addr, size; +int ret; + + aws = container_of(work, struct absent_work, work); + + phys_addr = e820_absent.map[aws->index].addr; + size = e820_absent.map[aws->index].size; + ret =memory_add_absent (aws->node, phys_addr, size); +if (ret) +pr_crit ("e820: Error %d adding absent memory %llx %llx (%d)\n" , + ret, phys_addr, size, aws->node); + +atomic_set (&aws->busy,2 ); +atomic_dec (&add_absent_work_count); + +/* if no one is waiting, then snap stop time */ +if (!count_absent_work_notdone ()) + add_absent_work_stop_time =get_seconds (); +} + +/* Initialize absent_work threads */ +static int add_absent_memory (void ) +{ +struct absent_work *aws =NULL ; +int cpu, i; + + add_absent_work_start_time =get_seconds (); + add_absent_work_stop_time =0 ; +atomic_set (&add_absent_work_count,0 ); + +for_each_online_cpu (cpu) { +struct absent_work *aws = &per_cpu (absent_work, cpu); + aws->node =0 ; + } + +/* setup each work thread */ +for (i =0 ; i < e820_absent.nr_map; i++) { +u64 phys_addr = e820_absent.map[i].addr; +int node =memory_add_physaddr_to_nid (phys_addr); + +if (!node_online (node)) +continue ; + +if (!aws) { + aws =get_absent_work (node); + first_absent_work = aws; + }else { + aws->next =get_absent_work (node); + aws = aws->next; + } + +if (!aws) +continue ; + +INIT_WORK (&aws->work, add_absent_memory_work); +atomic_set (&aws->busy,0 ); + aws->index = i; + +/* schedule absent_work thread */ +if (!schedule_work_on (aws->cpu, &aws->work)) +BUG (); + } + + +pr_info ("e820: Add absent memory started\n" ); + +return 0 ; +} + +/* Called during bootup to start adding absent_mem early */ +static int absent_memory_init (void ) +{ +return add_absent_memory (); +} +late_initcall (absent_memory_init);#endif /* CONFIG_DELAY_MEM_INIT */ static int __init __append_e820_map (struct e820entry *biosmap,int nr_map)
drivers/base/memory.c +83 -0
89 lines not on this page · raw message
include/linux/memory.h +5 -0
@@ -128,6 +128,11 @@ extern struct memory_block *find_memory_block(struct mem_section *);enum mem_add_context {BOOT ,HOTPLUG };#endif /* CONFIG_MEMORY_HOTPLUG_SPARSE */ +#ifdef CONFIG_DELAY_MEM_INIT +extern int memory_add_absent (int nid,u64 phys_addr,u64 size); +#endif + +#ifdef CONFIG_MEMORY_HOTPLUG #define hotplug_memory_notifier (fn, pri) ({ \ static __meminitdata struct notifier_block fn##_mem_nb =\
Signature · 1 line
1.8.2.1
Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory
Quoted 39 lines · Nathan Zimmer
The memory we set aside in the previous patch needs to be reinserted. We start this process via
late_initcallso we will have multiple cpus to do the work.Signed-off-by Mike Travis <[email protected]>Signed-off-by Nathan Zimmer <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc "H. Peter Anvin" <[email protected]>Cc Greg Kroah-Hartman <[email protected]>Cc Andrew Morton <[email protected]>Cc Yinghai Lu <[email protected]>--- arch/x86/kernel/e820.c | 129 +++++++++++++++++++++++++++++++++++++++++++++++++ drivers/base/memory.c | 83 +++++++++++++++++++++++++++++++ include/linux/memory.h | 5 ++ 3 files changed, 217 insertions(+)arch/x86/kernel/e820.c +6
-0@@ -23,6 +23,7 @@#ifdef CONFIG_DELAY_MEM_INIT #include <linux/memory.h> +#include <linux/delay.h> #endif #include <asm/e820.h> @@ -397,6 +398,22 @@ static u64 min_region_size; /* min size of region to slice from */static u64 pre_region_size;/* multiply bsize for node low memory */ static u64 post_region_size;/* multiply bsize for node high memory */ +static unsigned long add_absent_work_start_time; +static unsigned long add_absent_work_stop_time; +static unsigned int add_absent_job_count; +static atomic_t add_absent_work_count; +
arch/x86/kernel/e820.c static u64 min_region_size; /* min size of region to slice from */ · line 415
Quoted 10 lines
arch/x86/kernel/e820.c +10
-0+struct absent_work { +struct work_struct work; +struct absent_work *next; +atomic_t busy; +int cpu; +int node; +int index; +}; +static DEFINE_PER_CPU(struct absent_work, absent_work); +static struct absent_work *first_absent_work;
That's 4.5 GB/sec initialization speed - that feels a bit slow and the boot time effect should be felt on smaller 'a couple of gigabytes' desktop boxes as well. Do we know exactly where the 2 hours of boot time on a 32 TB system is spent?
While you cannot profile the boot process (yet), you could try your delayed patch and run a "perf record -g" call-graph profiling of the late-time initialization routines. What does 'perf report' show?
Delayed initialization makes sense I guess because 32 TB is a lot of memory - I'm just wondering whether there's some low hanging fruits left in the mem init code, that code is certainly not optimized for performance.
Plus with a struct page size of around 64 bytes (?) 32 TB of RAM has 512 GB of struct page arrays alone. Initializing those will take quite some time as well - and I suspect they are allocated via zeroing them first. If that memset() exists then getting rid of it might be a good move as well.
Yet another thing to consider would be to implement an initialization speedup of 3 orders of magnitude: initialize on the large page (2MB) grandularity and on-demand delay the initialization of the 4K granular struct pages [but still allocating them] - which I suspect are a good chunk of the overhead? That way we could initialize in 2MB steps and speed up the 2 hours bootup of 32 TB of RAM to 14 seconds...
[ The cost would be one more branch in the buddy allocator, to detect
not-yet-initialized 2 MB chunks as we encounter them. Acceptable I think. ]
Thanks,
Ingo
Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory
Quoted 10 lines · Ingo Molnar
Yet another thing to consider would be to implement an initialization speedup of 3 orders of magnitude: initialize on the large page (2MB) grandularity and on-demand delay the initialization of the 4K granular struct pages [but still allocating them] - which I suspect are a good chunk of the overhead? That way we could initialize in 2MB steps and speed up the 2 hours bootup of 32 TB of RAM to 14 seconds...
[ The cost would be one more branch in the buddy allocator, to detect
not-yet-initialized 2 MB chunks as we encounter them. Acceptable I think. ]
One advantage of this scheme would be that we could use it on pretty much any box, it would provide instant boot time speedups everywhere [a couple of hundred msecs on a small 4GB box - significant I think] - and would spread out and parallelize initialization to later stages.
Thanks,
Ingo
H. Peter Anvin
reviewerRe: [RFC 2/2] x86_64, mm: Reinsert the absent memory
Quoted 10 lines · Ingo Molnar
Quoted 7 lines · Ingo Molnar
Yet another thing to consider would be to implement an initialization speedup of 3 orders of magnitude: initialize on the large page (2MB) grandularity and on-demand delay the initialization of the 4K granular struct pages [but still allocating them] - which I suspect are a good chunk of the overhead? That way we could initialize in 2MB steps and speed up the 2 hours bootup of 32 TB of RAM to 14 seconds...
Quoted 9 lines
Quoted 3 lines
[ The cost would be one more branch in the buddy allocator, to detect
not-yet-initialized 2 MB chunks as we encounter them. Acceptable I think. ]One advantage of this scheme would be that we could use it on pretty much any box, it would provide instant boot time speedups everywhere [a couple of hundred msecs on a small 4GB box - significant I think] - and would spread out and parallelize initialization to later stages.
Even better if we could start at the 1 GB level, which most of these really huge machines will have hardware support for.
-hpa
Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory
Quoted 15 lines · H. Peter Anvin
Quoted 14 lines · Ingo Molnar
Quoted 10 lines · Ingo Molnar
Yet another thing to consider would be to implement an initialization speedup of 3 orders of magnitude: initialize on the large page (2MB) grandularity and on-demand delay the initialization of the 4K granular struct pages [but still allocating them] - which I suspect are a good chunk of the overhead? That way we could initialize in 2MB steps and speed up the 2 hours bootup of 32 TB of RAM to 14 seconds...
[ The cost would be one more branch in the buddy allocator, to detect
not-yet-initialized 2 MB chunks as we encounter them. Acceptable I think. ]
Quoted 7 lines
Quoted 4 lines
One advantage of this scheme would be that we could use it on pretty much any box, it would provide instant boot time speedups everywhere [a couple of hundred msecs on a small 4GB box - significant I think] - and would spread out and parallelize initialization to later stages.
Even better if we could start at the 1 GB level, which most of these really huge machines will have hardware support for.
That might be a bit too granular: if we hit such an uninitialized block of memory we'd have to process 262,144 pages - potentially from an IRQ handler that does GFP_ATOMIC... or other latency critical code.
With 2MB we'd have to on-demand initialize 512 pages, which shouldn't show up during normal use.
Thanks,
Ingo
H. Peter Anvin
reviewerRe: [RFC 2/2] x86_64, mm: Reinsert the absent memory
Quoted 8 lines · Ingo Molnar
That might be a bit too granular: if we hit such an uninitialized block of memory we'd have to process 262,144 pages - potentially from an IRQ handler that does
GFP_ATOMIC... or other latency critical code.With 2MB we'd have to on-demand initialize 512 pages, which shouldn't show up during normal use.
No... you split the gigabyte page into 511 2M pages and 512 4K pages.
-hpa
Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory
Quoted 11 lines · H. Peter Anvin
Quoted 8 lines · Ingo Molnar
That might be a bit too granular: if we hit such an uninitialized block of memory we'd have to process 262,144 pages - potentially from an IRQ handler that does
GFP_ATOMIC... or other latency critical code.With 2MB we'd have to on-demand initialize 512 pages, which shouldn't show up during normal use.
No... you split the gigabyte page into 511 2M pages and 512 4K pages.
Oh, double deferred initialization. That should solve the 32 exabyte problem as well I guess ;-)
Thanks,
Ingo
Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory
In reply to Ingo MolnarQuoted 6 lines · Ingo Molnar
That's 4.5 GB/sec initialization speed - that feels a bit slow and the boot time effect should be felt on smaller 'a couple of gigabytes' desktop boxes as well. Do we know exactly where the 2 hours of boot time on a 32 TB system is spent?
There are other several spots that could be improved on a large system but memory initialization is by far the biggest.
Quoted 4 lines
While you cannot profile the boot process (yet), you could try your delayed patch and run a "perf record -g" call-graph profiling of the late-time initialization routines. What does 'perf report' show?
I have some data from earlier runs.memmap_init_zone was the function that was the biggest hitter by far. Parts of it could certianly are low hanging fruit, set_pageblock_migratetype for example.
However it seems for a larger system SetPageReserved will be the largest consumer of cycles. On a 1TB system I just booted it was around 50% of time spent in memmap_init_zone.
perf seems to struggle with 512 cpus, but I did get some data. It seems to indicate similar data to what I found in earlier experiments. Lots of time in memmap_init_zone,
Some are waiting on locks, this guy seems to be representative of that.
- 0.14% kworker/160:1 [kernel.kallsyms] [k] mspin_lock ▒ + mspin_lock ▒ + __mutex_lock_slowpath ▒ - mutex_lock ▒
- 99.69% online_pages
Quoted 24 lines
Delayed initialization makes sense I guess because 32 TB is a lot of memory - I'm just wondering whether there's some low hanging fruits left in the mem init code, that code is certainly not optimized for performance.
Plus with a struct page size of around 64 bytes (?) 32 TB of RAM has 512 GB of struct page arrays alone. Initializing those will take quite some time as well - and I suspect they are allocated via zeroing them first. If that
memset()exists then getting rid of it might be a good move as well.Yet another thing to consider would be to implement an initialization speedup of 3 orders of magnitude: initialize on the large page (2MB) grandularity and on-demand delay the initialization of the 4K granular struct pages [but still allocating them] - which I suspect are a good chunk of the overhead? That way we could initialize in 2MB steps and speed up the 2 hours bootup of 32 TB of RAM to 14 seconds...
[ The cost would be one more branch in the buddy allocator, to detect
not-yet-initialized 2 MB chunks as we encounter them. Acceptable I think. ]Thanks,
Ingo
Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory
Quoted 9 lines · Nathan Zimmer
Quoted 5 lines · Ingo Molnar
That's 4.5 GB/sec initialization speed - that feels a bit slow and the boot time effect should be felt on smaller 'a couple of gigabytes' desktop boxes as well. Do we know exactly where the 2 hours of boot time on a 32 TB system is spent?
There are other several spots that could be improved on a large system but memory initialization is by far the biggest.
My feeling is that deferred/on-demand initialization triggered from the buddy allocator is the better long term solution.
That will also make it much easier to profile/test memory init
performance: boot up a large system and run a simple testprogram that allocates a lot of RAM.
( It will also make people want to optimize the initialization sequence
better, as it will be part of any freshly booted system's memory allocation overhead. )
Thanks,
Ingo
H. Peter Anvin
reviewerRe: [RFC 2/2] x86_64, mm: Reinsert the absent memory
I have to say I really like this concept. It should have some very nice properties including perhaps making THP work better?
Quoted 32 lines · Ingo Molnar
Quoted 3 lines · Nathan Zimmer
Quoted 2 lines · Ingo Molnar
That's 4.5 GB/sec initialization speed - that feels a bit slow and
the
Quoted 6 lines
Quoted 4 lines
boot time effect should be felt on smaller 'a couple of gigabytes' desktop boxes as well. Do we know exactly where the 2 hours of boot
time on a 32 TB system is spent?
There are other several spots that could be improved on a large
system
Quoted 1 line
but memory initialization is by far the biggest.
My feeling is that deferred/on-demand initialization triggered from the
buddy allocator is the better long term solution.
That will also make it much easier to profile/test memory init
performance: boot up a large system and run a simple testprogram that allocates a lot of RAM.( It will also make people want to optimize the initialization sequence
better, as it will be part of any freshly booted system's memory allocation overhead. )
Thanks,
Ingo
Signature · 1 line
Sent from my mobile phone. Please excuse brevity and lack of formatting.
Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory
Thanks. I went through a few iterations until finalizing on the current patches. Originally the memory was inserted using the memory probe interface, but at 2G per addition, it was still very slow. By bypassing the memory driver interface, and going directly to add_memory and memory_online, along with Nathan's changes to unlock the memory mutex on a per node basis, it sped up dramatically.
Quoted 37 lines · H. Peter Anvin
I have to say I really like this concept. It should have some very nice properties including perhaps making THP work better?
Quoted 32 lines · Ingo Molnar
Quoted 3 lines · Nathan Zimmer
Quoted 2 lines · Ingo Molnar
That's 4.5 GB/sec initialization speed - that feels a bit slow and
the
Quoted 6 lines
Quoted 4 lines
boot time effect should be felt on smaller 'a couple of gigabytes' desktop boxes as well. Do we know exactly where the 2 hours of boot
time on a 32 TB system is spent?
There are other several spots that could be improved on a large
system
Quoted 1 line
but memory initialization is by far the biggest.
My feeling is that deferred/on-demand initialization triggered from the
buddy allocator is the better long term solution.
That will also make it much easier to profile/test memory init
performance: boot up a large system and run a simple testprogram that allocates a lot of RAM.( It will also make people want to optimize the initialization sequence
better, as it will be part of any freshly booted system's memory allocation overhead. )
Thanks,
Ingo
Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory
In reply to Ingo MolnarQuoted 10 lines · Ingo Molnar
Quoted 7 lines · Nathan Zimmer
Quoted 5 lines · Ingo Molnar
That's 4.5 GB/sec initialization speed - that feels a bit slow and the boot time effect should be felt on smaller 'a couple of gigabytes' desktop boxes as well. Do we know exactly where the 2 hours of boot time on a 32 TB system is spent?
Quoted 5 lines
Quoted 2 lines
There are other several spots that could be improved on a large system but memory initialization is by far the biggest.
My feeling is that deferred/on-demand initialization triggered from the buddy allocator is the better long term solution.
I haven't caught up with all of Nathan's changes yet (just got back from vacation), but there was an option to either start the memory insertion on boot, or trigger it later using the /sys/.../memory interface. There is also a monitor program that calculates the memory insertion rate. This was extremely useful to determine how changes in the kernel affected the rate.
Quoted 13 lines
That will also make it much easier to profile/test memory init
performance: boot up a large system and run a simple testprogram that allocates a lot of RAM.( It will also make people want to optimize the initialization sequence
better, as it will be part of any freshly booted system's memory allocation overhead. )Thanks,
Ingo
H. Peter Anvin
reviewerRe: [RFC 2/2] x86_64, mm: Reinsert the absent memory
Quoted 18 lines · Mike Travis
Quoted 10 lines · Ingo Molnar
Quoted 7 lines · Nathan Zimmer
Quoted 5 lines · Ingo Molnar
That's 4.5 GB/sec initialization speed - that feels a bit slow and the boot time effect should be felt on smaller 'a couple of gigabytes' desktop boxes as well. Do we know exactly where the 2 hours of boot time on a 32 TB system is spent?
Quoted 5 lines
Quoted 2 lines
There are other several spots that could be improved on a large system but memory initialization is by far the biggest.
My feeling is that deferred/on-demand initialization triggered from the buddy allocator is the better long term solution.
Quoted 8 lines
I haven't caught up with all of Nathan's changes yet (just got back from vacation), but there was an option to either start the memory insertion on boot, or trigger it later using the
/sys/.../memoryinterface. There is also a monitor program that calculates the memory insertion rate. This was extremely useful to determine how changes in the kernel affected the rate.
Sorry, I *totally* did not follow that comment. It seemed like a complete non-sequitur?
-hpa
Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory
Quoted 28 lines · H. Peter Anvin
Quoted 18 lines · Mike Travis
Quoted 10 lines · Ingo Molnar
Quoted 7 lines · Nathan Zimmer
Quoted 5 lines · Ingo Molnar
That's 4.5 GB/sec initialization speed - that feels a bit slow and the boot time effect should be felt on smaller 'a couple of gigabytes' desktop boxes as well. Do we know exactly where the 2 hours of boot time on a 32 TB system is spent?
Quoted 5 lines
Quoted 2 lines
There are other several spots that could be improved on a large system but memory initialization is by far the biggest.
My feeling is that deferred/on-demand initialization triggered from the buddy allocator is the better long term solution.
Quoted 8 lines
I haven't caught up with all of Nathan's changes yet (just got back from vacation), but there was an option to either start the memory insertion on boot, or trigger it later using the
/sys/.../memoryinterface. There is also a monitor program that calculates the memory insertion rate. This was extremely useful to determine how changes in the kernel affected the rate.
Quoted 4 lines
Sorry, I *totally* did not follow that comment. It seemed like a complete non-sequitur?
-hpa
It was I who was not following the question. I'm still reverting back to "work mode".
[There is more code in a separate patch that Nate has not sent yet that instructs the kernel to start adding memory as early as possible, or not. That way you can start the insertion process later and monitor it's progress to determine how changes in the kernel affect that process. It is controlled by a separate CONFIG option.]
Quoted 2 lines
[RFC] Transparent on-demand memory setup initialization embedded in the (GFP) buddy allocator
(Changed the subject, to make it more apparent what we are talking about.)
Quoted 37 lines · Mike Travis
Quoted 28 lines · H. Peter Anvin
Quoted 18 lines · Mike Travis
Quoted 10 lines · Ingo Molnar
Quoted 7 lines · Nathan Zimmer
Quoted 5 lines · Ingo Molnar
That's 4.5 GB/sec initialization speed - that feels a bit slow and the boot time effect should be felt on smaller 'a couple of gigabytes' desktop boxes as well. Do we know exactly where the 2 hours of boot time on a 32 TB system is spent?
Quoted 5 lines
Quoted 2 lines
There are other several spots that could be improved on a large system but memory initialization is by far the biggest.
My feeling is that deferred/on-demand initialization triggered from the buddy allocator is the better long term solution.
Quoted 8 lines
I haven't caught up with all of Nathan's changes yet (just got back from vacation), but there was an option to either start the memory insertion on boot, or trigger it later using the
/sys/.../memoryinterface. There is also a monitor program that calculates the memory insertion rate. This was extremely useful to determine how changes in the kernel affected the rate.Quoted 4 lines
Sorry, I *totally* did not follow that comment. It seemed like a complete non-sequitur?
-hpa
It was I who was not following the question. I'm still reverting back to "work mode".
Quoted 6 lines
[There is more code in a separate patch that Nate has not sent yet that instructs the kernel to start adding memory as early as possible, or not. That way you can start the insertion process later and monitor it's progress to determine how changes in the kernel affect that process. It is controlled by a separate CONFIG option.]
So, just to repeat (and expand upon) the solution hpa and me suggests:
it's not based on /sys, delayed initialization lists or any similar (essentially memory hot plug based) approach.
It's a transparent on-demand initialization scheme based on only initializing the very early memory setup in 1GB (2MB) steps (not in 4K steps like we do it today).
Any subsequent split-up initialization is done on-demand, in alloc_pages() et al, initilizing a batch of 512 (or 1024) struct page head's when an uninitialized portion is first encountered.
This leaves the principle logic of early init largely untouched, we still have the same amount of RAM during and after bootup, except that on 32 TB systems we don't spend ~2 hours initializing 8,589,934,592 page heads.
This scheme could be implemented by introducing a new PG_initialized flag, which is seen by an unlikely() branch in alloc_pages() and which triggers the on-demand initialization of pages.
[ It could probably be made zero-cost for the post-initialization state:
we already check a bunch of rare PG_ flags, one more flag would not introduce any new branch in the page allocation hot path. ]
It's a technically different solution from what was submitted in this thread.
Cons:
- it works after bootup, via GFP. If done in a simple fashion it adds one more branch to the GFP fastpath. [ If done a bit more cleverly it can merge into an existing unlikely() branch and become essentially zero-cost for the fastpath. ]
- it adds an initialization non-determinism to GFP, to the tune of initializing ~512 page heads when RAM is utilized first.
- initialization is done when memory is needed - not during or shortly after bootup. This (slightly) increases first-use overhead. [I don't think this factor is significant - and I think we'll quickly see speedups to initialization, once the overhead becomes more easily measurable.]
Pros:
- it's transparent to the boot process. ('free' shows the same full amount of RAM all the time, there's no weird effects of RAM coming online asynchronously. You see all the RAM you have - etc.)
- it helps the boot time of every single Linux system, not just large RAM ones. On a smallish, 4GB system memory init can take up precious hundreds of milliseconds, so this is a practical issue.
- it spreads initialization overhead to later portions of the system's life time: when there's typically more idle time and more paralellism available.
- initialization overhead, because it's a natural part of first-time memory allocation with this scheme, becomes more measurable (and thus more prominently optimized) than any deferred lists processed in the background.
- as an added bonus it probably speeds up your usecase even more than the patches you are providing: on a 32 TB system the primary initialization would only have to enumerate memory, allocate page heads and buddy bitmaps, and initialize the 1GB granular page heads: there's only 32768 of them.
So unless I overlooked some factor this scheme would be unconditional goodness for everyone.
Thanks,
Ingo
Andrew Morton
maintainerRe: [RFC] Transparent on-demand memory setup initialization embedded in the (GFP) buddy allocator
Quoted 2 lines · On Wed, 26 Jun 2013 11:22:48 +0200 Ingo Molnar <[email protected]> wrote:
except that on 32 TB
systems we don't spend ~2 hours initializing 8,589,934,592 page heads.
That's about a million a second which is crazy slow - even my prehistoric desktop is 100x faster than that.
Where's all this time actually being spent?
Re: [RFC] Transparent on-demand memory setup initialization embedded in the (GFP) buddy allocator
Quoted 9 lines · Andrew Morton
Quoted 2 lines · On Wed, 26 Jun 2013 11:22:48 +0200 Ingo Molnar <[email protected]> wrote:
except that on 32 TB
systems we don't spend ~2 hours initializing 8,589,934,592 page heads.That's about a million a second which is crazy slow - even my prehistoric desktop is 100x faster than that.
Where's all this time actually being spent?
See the earlier part of the thread - apparently it's spent initializing the page heads - remote NUMA node misses from a single boot CPU, going across a zillion cross-connects? I guess there's some other low hanging fruits as well - so making this easier to profile would be nice. The profile posted was not really usable.
Btw., NUMA locality would be another advantage of on-demand
initialization: actual users of RAM tend to allocate node-local (especially on large clusters), so any overhead will be naturally lower.
Thanks,
Ingo
Re: [RFC] Transparent on-demand memory setup initialization embedded in the (GFP) buddy allocator
Quoted 11 lines · Ingo Molnar
Quoted 8 lines · Andrew Morton
Quoted 2 lines · On Wed, 26 Jun 2013 11:22:48 +0200 Ingo Molnar <[email protected]> wrote:
except that on 32 TB
systems we don't spend ~2 hours initializing 8,589,934,592 page heads.That's about a million a second which is crazy slow - even my prehistoric desktop is 100x faster than that.
Quoted 8 lines
Quoted 1 line
Where's all this time actually being spent?
See the earlier part of the thread - apparently it's spent initializing the page heads - remote NUMA node misses from a single boot CPU, going across a zillion cross-connects? I guess there's some other low hanging fruits as well - so making this easier to profile would be nice. The profile posted was not really usable.
That is correct, from what I am seeing, using crude cycle counters, there is far more time spent on the later nodes, i.e. memory near the boot node is initialized a lot faster then remote memory.
I think the other low hanging fruits are currently being drowned out by the lack of locality.
Nate
Quoted 7 lines
Btw., NUMA locality would be another advantage of on-demand
initialization: actual users of RAM tend to allocate node-local (especially on large clusters), so any overhead will be naturally lower.Thanks,
Ingo
Re: [RFC] Transparent on-demand memory setup initialization embedded in the (GFP) buddy allocator
In reply to Ingo MolnarQuoted 11 lines · Ingo Molnar
Quoted 8 lines · Andrew Morton
Quoted 2 lines · On Wed, 26 Jun 2013 11:22:48 +0200 Ingo Molnar <[email protected]> wrote:
except that on 32 TB
systems we don't spend ~2 hours initializing 8,589,934,592 page heads.That's about a million a second which is crazy slow - even my prehistoric desktop is 100x faster than that.
Quoted 7 lines
Quoted 1 line
Where's all this time actually being spent?
See the earlier part of the thread - apparently it's spent initializing the page heads - remote NUMA node misses from a single boot CPU, going across a zillion cross-connects? I guess there's some other low hanging fruits as well - so making this easier to profile would be nice. The profile posted was not really usable.
This is one advantage of delayed memory init. I can do it under the profiler. I will put everything together to accomplish this and then send a perf report.
Quoted 9 lines
Btw., NUMA locality would be another advantage of on-demand
initialization: actual users of RAM tend to allocate node-local (especially on large clusters), so any overhead will be naturally lower.Thanks,
Ingo
Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory
In reply to Nathan ZimmerQuoted 1 line · Nathan Zimmer
perf seems to struggle with 512 cpus, but I did get some data.
Btw., mind outlining in what way it struggles?
Thanks,
Ingo
Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory
Quoted 10 lines · Ingo Molnar
Quoted 1 line · Nathan Zimmer
perf seems to struggle with 512 cpus, but I did get some data.
Btw., mind outlining in what way it struggles?
Thanks,
Ingo
System interactivity take a nose dive, including the ability to halt perf. The commands do get through but they take a while.
Other processes tend to make signifigantly reduced progress.
Also it tends to slow down all running processes.
My guess it is the NMIs overwhelming the system but I have not found a good way to profile perf gone wild so it is only a guess.
The issue mitigate somewhat by restrictring the number of cpus perf is recording or by not using the -g option, but sometimes I do need data on the whole system.
Nate
Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory
Quoted 2 lines · Nathan Zimmer
My guess it is the NMIs overwhelming the system but I have not found a good way to profile perf gone wild so it is only a guess.
I've got an 80-core system and the symptoms sound similar to perf issues I'm seeing. Dropping the sample rate helped for me, and there's a patch in -tip at the moment to do it automatically:
Quoted 1 line
Looks like you've also found that debugging NMIs is fun. :)
Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory
Quoted 9 lines · Dave Hansen
Quoted 2 lines · Nathan Zimmer
My guess it is the NMIs overwhelming the system but I have not found a good way to profile perf gone wild so it is only a guess.
I've got an 80-core system and the symptoms sound similar to perf issues I'm seeing. Dropping the sample rate helped for me, and there's a patch in -tip at the moment to do it automatically:
Quoted 1 line
Looks like you've also found that debugging NMIs is fun. :)
More fun then one person can handle.
I'll have to take this patch for a spin.
Thanks,
Nate
Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory
Quoted 4 lines · Nathan Zimmer
Quoted 3 lines · Dave Hansen
Quoted 2 lines · Nathan Zimmer
My guess it is the NMIs overwhelming the system but I have not found a good way to profile perf gone wild so it is only a guess.
Quoted 8 lines
Quoted 6 lines
I've got an 80-core system and the symptoms sound similar to perf issues I'm seeing. Dropping the sample rate helped for me, and there's a patch in -tip at the moment to do it automatically:
Quoted 1 line
Looks like you've also found that debugging NMIs is fun. :)
More fun then one person can handle.
I'll have to take this patch for a spin.
Please try latest -tip:master, there's a series of patches from Dave that address this problem category.
Does it only happen with -g, or with regular perf record and/or with perf top as well?
Thanks,
Ingo
Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory
Quoted 17 lines · Ingo Molnar
Quoted 4 lines · Nathan Zimmer
Quoted 3 lines · Dave Hansen
Quoted 2 lines · Nathan Zimmer
My guess it is the NMIs overwhelming the system but I have not found a good way to profile perf gone wild so it is only a guess.
Quoted 8 lines
Quoted 6 lines
I've got an 80-core system and the symptoms sound similar to perf issues I'm seeing. Dropping the sample rate helped for me, and there's a patch in -tip at the moment to do it automatically:
Quoted 1 line
Looks like you've also found that debugging NMIs is fun. :)
More fun then one person can handle.
I'll have to take this patch for a spin.Please try latest -tip:master, there's a series of patches from Dave that address this problem category.
Quoted 6 lines
Does it only happen with -g, or with regular perf record and/or with perf top as well?
Thanks,
Ingo
It happens in all those cases, but I notice it most when doing a perf record -g -a I'll try the tip series and see how it compares.
Thanks
Nate
Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory
In reply to Ingo MolnarQuoted 11 lines · Ingo Molnar
Quoted 1 line · Nathan Zimmer
perf seems to struggle with 512 cpus, but I did get some data.
Btw., mind outlining in what way it struggles?
Thanks,
Ingo
I submitted some patches to Jason Wessel that updated the community support for KDB & NMI quite awhile ago that addressed the issues with perf and friends on UV. But I have not heard back from him in a couple of months. Is there a new maintainer for NMI/PERF/KDB etc.?
The primary problem is that the current UV NMI handler is in the primary NMI notifier chain causing excessive reads of a register in the UV hub. When perf is running these approach the millions per second, and the MMIO read is not only expensive but also distrupts the primary HUB activities of directing NUMA traffic. Thus the system slows down considerably, and perf can even lose events.
We've even updated the UV BIOS to make this a faster path but it needs the newer NMI handler to use it. Perhaps I should resubmit them directly to you?
Thanks,
Mike
Greg KH
maintainerQuoted 5 lines · Nathan Zimmer
This rfc patch set delays initializing large sections of memory until we have started cpus. This has the effect of reducing startup times on large memory systems. On 16TB it can take over an hour to boot and most of that time is spent initializing memory.
Quoted 7 lines
We avoid that bottleneck by delaying initialization until after we have started multiple cpus and can initialize in a multithreaded manner. This allows us to actually reduce boot time rather then just moving around the point of initialization.
Mike and I have worked on this set for a while, with him doing the most of the heavy lifting, and are eager for some feedback.
Why make this a config option at all, why not just always do this if the memory size is larger than some specific number (like 8TB?)
Otherwise the distros will always enable this option, and having it be a configuration choice doesn't make any sense.
thanks,
greg k-h
H. Peter Anvin
reviewerQuoted 11 lines · Greg KH
Quoted 10 lines · Nathan Zimmer
This rfc patch set delays initializing large sections of memory until we have started cpus. This has the effect of reducing startup times on large memory systems. On 16TB it can take over an hour to boot and most of that time is spent initializing memory.
We avoid that bottleneck by delaying initialization until after we have started multiple cpus and can initialize in a multithreaded manner. This allows us to actually reduce boot time rather then just moving around the point of initialization.
Quoted 9 lines
Quoted 2 lines
Mike and I have worked on this set for a while, with him doing the most of the heavy lifting, and are eager for some feedback.
Why make this a config option at all, why not just always do this if the memory size is larger than some specific number (like 8TB?)
Otherwise the distros will always enable this option, and having it be a configuration choice doesn't make any sense.
Since you made it a compile time option, it would be good to know how much code it adds, but otherwise I agree with Greg here... this really shouldn't need to be an option. It *especially* shouldn't need to be a hand-set runtime option (which looks quite complex, to boot.)
I suspect the cutoff for this should be a lot lower than 8 TB even, more like 128 GB or so. The only concern is to not set the cutoff so low that we can end up running out of memory or with suboptimal NUMA placement just because of this.
Also, in case it is not bloody obvious: whatever memory the kernel image was loaded into MUST be considered "online", even if it is loaded way high.
-hpa
Quoted 17 lines · H. Peter Anvin
Quoted 16 lines · Greg KH
Quoted 5 lines · Nathan Zimmer
This rfc patch set delays initializing large sections of memory until we have started cpus. This has the effect of reducing startup times on large memory systems. On 16TB it can take over an hour to boot and most of that time is spent initializing memory.
Quoted 7 lines
We avoid that bottleneck by delaying initialization until after we have started multiple cpus and can initialize in a multithreaded manner. This allows us to actually reduce boot time rather then just moving around the point of initialization.
Mike and I have worked on this set for a while, with him doing the most of the heavy lifting, and are eager for some feedback.
Why make this a config option at all, why not just always do this if the memory size is larger than some specific number (like 8TB?)
Quoted 7 lines
Quoted 3 lines
Otherwise the distros will always enable this option, and having it be a configuration choice doesn't make any sense.
Since you made it a compile time option, it would be good to know how much code it adds, but otherwise I agree with Greg here... this really shouldn't need to be an option. It *especially* shouldn't need to be a hand-set runtime option (which looks quite complex, to boot.)
The patchset as a whole is just over 400 lines so it doesn't add alot. If I were to pull the .config option it would probably remove 30 lines.
The command line option is too complex but some of the data I haven't found a way to get at runtime yet.
Quoted 5 lines
I suspect the cutoff for this should be a lot lower than 8 TB even, more like 128 GB or so. The only concern is to not set the cutoff so low that we can end up running out of memory or with suboptimal NUMA placement just because of this.
Even at lower amounts of ram there is an positive impact.I it knocks time off boot even at as small as a 1TB of ram.
Quoted 8 lines
Also, in case it is not bloody obvious: whatever memory the kernel image was loaded into MUST be considered "online", even if it is loaded way high.
-hpa
Ok
H. Peter Anvin
reviewerQuoted 7 lines · Nathan Zimmer
Quoted 5 lines
Quoted 1 line
Since you made it a compile time option, it would be good to know how much code it adds, but otherwise I agree with Greg here... this really shouldn't need to be an option. It *especially* shouldn't need to be a hand-set runtime option (which looks quite complex, to boot.)
The patchset as a whole is just over 400 lines so it doesn't add alot. If I were to pull the .config option it would probably remove 30 lines.
I'm more concerned about bytes of code.
Quoted 2 lines
The command line option is too complex but some of the data I haven't found a way to get at runtime yet.
I think that is probably key.
Quoted 7 lines
Quoted 4 lines
I suspect the cutoff for this should be a lot lower than 8 TB even, more like 128 GB or so. The only concern is to not set the cutoff so low that we can end up running out of memory or with suboptimal NUMA placement just because of this.
Even at lower amounts of ram there is an positive impact.I it knocks time off
boot even at as small as a 1TB of ram.
I am not surprised.
-hpa
Quoted 8 lines · H. Peter Anvin
Quoted 6 lines · Nathan Zimmer
Quoted 4 lines
Since you made it a compile time option, it would be good to know how much code it adds, but otherwise I agree with Greg here... this really shouldn't need to be an option. It *especially* shouldn't need to be a hand-set runtime option (which looks quite complex, to boot.)
The patchset as a whole is just over 400 lines so it doesn't add alot. If I were to pull the .config option it would probably remove 30 lines.
I'm more concerned about bytes of code.
Oh, The difference is just under 32k.
371843425 Jun 21 14:08 vmlinux.o /* DELAY_MEM_INIT is not set */ 371875600 Jun 21 14:36 vmlinux.o /* DELAY_MEM_INIT=y */
Quoted 16 lines
Quoted 3 lines
The command line option is too complex but some of the data I haven't found a way to get at runtime yet.
I think that is probably key.
Quoted 7 lines
Quoted 4 lines
I suspect the cutoff for this should be a lot lower than 8 TB even, more like 128 GB or so. The only concern is to not set the cutoff so low that we can end up running out of memory or with suboptimal NUMA placement just because of this.
Even at lower amounts of ram there is an positive impact.I it knocks time off
boot even at as small as a 1TB of ram.I am not surprised.
-hpa
H. Peter Anvin
reviewerIs this init code? 32K of unconditional runtime addition isn't completely trivial.
Quoted 37 lines · Nathan Zimmer
Quoted 2 lines · H. Peter Anvin
Quoted 1 line · Nathan Zimmer
Quoted 1 line
Since you made it a compile time option, it would be good to know
how
Quoted 1 line
Quoted 1 line
Quoted 1 line
much code it adds, but otherwise I agree with Greg here... this
really
Quoted 1 line
Quoted 1 line
Quoted 1 line
shouldn't need to be an option. It *especially* shouldn't need to
be a
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
hand-set runtime option (which looks quite complex, to boot.)
The patchset as a whole is just over 400 lines so it doesn't add
alot.
Quoted 1 line
Quoted 1 line
If I were to pull the .config option it would probably remove 30
lines.
Quoted 1 line
I'm more concerned about bytes of code.
Oh, The difference is just under 32k.
371843425 Jun 21 14:08 vmlinux.o /*DELAY_MEM_INITis not set */ 371875600 Jun 21 14:36 vmlinux.o /* DELAY_MEM_INIT=y */Quoted 2 lines
Quoted 2 lines
The command line option is too complex but some of the data I
haven't
Quoted 4 lines
Quoted 1 line
found a way to get at runtime yet.
I think that is probably key.
Quoted 1 line
Quoted 1 line
I suspect the cutoff for this should be a lot lower than 8 TB even,
more
Quoted 1 line
Quoted 1 line
Quoted 1 line
like 128 GB or so. The only concern is to not set the cutoff so
low
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
that we can end up running out of memory or with suboptimal NUMA placement just because of this.
Even at lower amounts of ram there is an positive impact.I it knocks time off
boot even at as small as a 1TB of ram.I am not surprised.
-hpa
Signature · 1 line
Sent from my mobile phone. Please excuse brevity and lack of formatting.
Quoted 1 line · H. Peter Anvin
Is this init code? 32K of unconditional runtime addition isn't completely trivial.
Some of it is init code but not all.
I am guessing 24k of that is actually runtime.
Quoted 43 lines
Quoted 37 lines · Nathan Zimmer
Quoted 2 lines · H. Peter Anvin
Quoted 1 line · Nathan Zimmer
Quoted 1 line
Since you made it a compile time option, it would be good to know
how
Quoted 1 line
Quoted 1 line
Quoted 1 line
much code it adds, but otherwise I agree with Greg here... this
really
Quoted 1 line
Quoted 1 line
Quoted 1 line
shouldn't need to be an option. It *especially* shouldn't need to
be a
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
hand-set runtime option (which looks quite complex, to boot.)
The patchset as a whole is just over 400 lines so it doesn't add
alot.
Quoted 1 line
Quoted 1 line
If I were to pull the .config option it would probably remove 30
lines.
Quoted 1 line
I'm more concerned about bytes of code.
Oh, The difference is just under 32k.
371843425 Jun 21 14:08 vmlinux.o /*DELAY_MEM_INITis not set */ 371875600 Jun 21 14:36 vmlinux.o /* DELAY_MEM_INIT=y */Quoted 2 lines
Quoted 2 lines
The command line option is too complex but some of the data I
haven't
Quoted 4 lines
Quoted 1 line
found a way to get at runtime yet.
I think that is probably key.
Quoted 1 line
Quoted 1 line
I suspect the cutoff for this should be a lot lower than 8 TB even,
more
Quoted 1 line
Quoted 1 line
Quoted 1 line
like 128 GB or so. The only concern is to not set the cutoff so
low
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
that we can end up running out of memory or with suboptimal NUMA placement just because of this.
Even at lower amounts of ram there is an positive impact.I it knocks time off
boot even at as small as a 1TB of ram.I am not surprised.
-hpa
Signature · 1 line
Sent from my mobile phone. Please excuse brevity and lack of formatting.
Quoted 1 line · H. Peter Anvin
Is this init code? 32K of unconditional runtime addition isn't completely trivial.
The delay functions that move memory to the absent list are __init but the read back of the list and memory insertion are not. BTW, this option is only available with the memory hotplug option which depends on the sparse memory option. So any usage of these distro configs will normally be on enterprise class machines.
Quoted 41 lines
Quoted 37 lines · Nathan Zimmer
Quoted 2 lines · H. Peter Anvin
Quoted 1 line · Nathan Zimmer
Quoted 1 line
Since you made it a compile time option, it would be good to know
how
Quoted 1 line
Quoted 1 line
Quoted 1 line
much code it adds, but otherwise I agree with Greg here... this
really
Quoted 1 line
Quoted 1 line
Quoted 1 line
shouldn't need to be an option. It *especially* shouldn't need to
be a
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
hand-set runtime option (which looks quite complex, to boot.)
The patchset as a whole is just over 400 lines so it doesn't add
alot.
Quoted 1 line
Quoted 1 line
If I were to pull the .config option it would probably remove 30
lines.
Quoted 1 line
I'm more concerned about bytes of code.
Oh, The difference is just under 32k.
371843425 Jun 21 14:08 vmlinux.o /*DELAY_MEM_INITis not set */ 371875600 Jun 21 14:36 vmlinux.o /* DELAY_MEM_INIT=y */Quoted 2 lines
Quoted 2 lines
The command line option is too complex but some of the data I
haven't
Quoted 4 lines
Quoted 1 line
found a way to get at runtime yet.
I think that is probably key.
Quoted 1 line
Quoted 1 line
I suspect the cutoff for this should be a lot lower than 8 TB even,
more
Quoted 1 line
Quoted 1 line
Quoted 1 line
like 128 GB or so. The only concern is to not set the cutoff so
low
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
that we can end up running out of memory or with suboptimal NUMA placement just because of this.
Even at lower amounts of ram there is an positive impact.I it knocks time off
boot even at as small as a 1TB of ram.I am not surprised.
-hpa
Quoted 10 lines · Nathan Zimmer
Quoted 9 lines · H. Peter Anvin
Quoted 8 lines · Greg KH
Quoted 7 lines · Nathan Zimmer
This rfc patch set delays initializing large sections of memory until we have
started cpus. This has the effect of reducing startup times on large memory
systems. On 16TB it can take over an hour to boot and most of that time
is spent initializing memory.
On 32TB we went from 2:25 to around 20 minutes.
Quoted 19 lines
Quoted 12 lines
Quoted 12 lines
Quoted 9 lines
We avoid that bottleneck by delaying initialization until after we have started multiple cpus and can initialize in a multithreaded manner. This allows us to actually reduce boot time rather then just moving around
the point of initialization.Mike and I have worked on this set for a while, with him doing the most of the
heavy lifting, and are eager for some feedback.Why make this a config option at all, why not just always do this if the memory size is larger than some specific number (like 8TB?)
Quoted 7 lines
Quoted 3 lines
Otherwise the distros will always enable this option, and having it be a configuration choice doesn't make any sense.
Since you made it a compile time option, it would be good to know how much code it adds, but otherwise I agree with Greg here... this really shouldn't need to be an option. It *especially* shouldn't need to be a hand-set runtime option (which looks quite complex, to boot.)
Quoted 5 lines
The patchset as a whole is just over 400 lines so it doesn't add alot. If I were to pull the .config option it would probably remove 30 lines.
The command line option is too complex but some of the data I haven't found a way to get at runtime yet.
Specifically, the physical address space of each node and whether the block size is 128M or 2G is needed. The other params are really there as a fallback as we have not yet verified on the largest possible machine. The parameter is intended to be set by a configurator. There are far too many kernel params to leave them to chance to be set correctly. On UV we use a utility called 'uvconfig'.
What we could do is default the values unless specifically set? Perhaps only set the node address space? Delaying the memory insertion is mostly a debug aid for debugging the insertion functions.
Quoted 5 lines
Quoted 5 lines
I suspect the cutoff for this should be a lot lower than 8 TB even, more like 128 GB or so. The only concern is to not set the cutoff so low that we can end up running out of memory or with suboptimal NUMA placement just because of this.
Exactly.
We test regularly on a machine that has ~4TB and the speedup is negligible. The problem seems to occur as the count of memory blocks is increased over some limit. I think going much lower might start getting in the way of other things, like constructing transparent huge pages, etc.
Also notice that Node 0 and the last Node already have all their memory. There are just too many other types in the memmap that it wasn't worth the hassle. So unless the system has at least 6 or 8 nodes you're not gaining much.
Quoted 7 lines
Even at lower amounts of ram there is an positive impact.I it knocks time off
boot even at as small as a 1TB of ram.Quoted 3 lines
Also, in case it is not bloody obvious: whatever memory the kernel image was loaded into MUST be considered "online", even if it is loaded way high.
Good point, we should add a check since we have that info at boot time. Other checks might be if this is a kdump kernel, or even perhaps a KVM kernel boot (though giving it 16TB is pretty wild.)
Thanks!
Quoted 2 lines
Quoted 2 lines
-hpa
Quoted 6 lines · H. Peter Anvin
On 06/21/2013 09:51 AM, Greg KH wrote:
I suspect the cutoff for this should be a lot lower than 8 TB even, more like 128 GB or so. The only concern is to not set the cutoff so low that we can end up running out of memory or with suboptimal NUMA placement just because of this.
I would suggest another way:
only boot the system with boot node (include cpu, ram and pci root buses). then after boot, could add other nodes.
or something like pxm_mask or node_mask to select boot nodes.
Thanks
Yinghai
Greg KH
maintainerQuoted 11 lines · Yinghai Lu
Quoted 6 lines · H. Peter Anvin
On 06/21/2013 09:51 AM, Greg KH wrote:
I suspect the cutoff for this should be a lot lower than 8 TB even, more like 128 GB or so. The only concern is to not set the cutoff so low that we can end up running out of memory or with suboptimal NUMA placement just because of this.
I would suggest another way:
only boot the system with boot node (include cpu, ram and pci root buses). then after boot, could add other nodes.
What exactly do you mean by "after boot"? Often, the boot process of userspace needs those additional cpus and ram in order to initialize everything (like the pci devices) properly.
thanks,
greg k-h
Quoted 9 lines · Greg KH
Quoted 8 lines · Yinghai Lu
Quoted 6 lines · H. Peter Anvin
On 06/21/2013 09:51 AM, Greg KH wrote:
I suspect the cutoff for this should be a lot lower than 8 TB even, more like 128 GB or so. The only concern is to not set the cutoff so low that we can end up running out of memory or with suboptimal NUMA placement just because of this.
Quoted 7 lines
Quoted 3 lines
I would suggest another way:
only boot the system with boot node (include cpu, ram and pci root buses). then after boot, could add other nodes.What exactly do you mean by "after boot"? Often, the boot process of userspace needs those additional cpus and ram in order to initialize everything (like the pci devices) properly.
I mean for Intel cpu have cpu and memory controller and IIO. every IIO is one peer pci root bus.
So scan root bus that are not with boot node later.
in this way we can keep all numa etc on the place when online ram, cpu, pci...
For example if we have 32 sockets system, most time for boot is with *BIOS* instead of OS. In those kind of system boot is like this way:
only first two sockets get booted from bios to OS.
later use hot add every other two sockets.
that will also make BIOS simpler, and it need to support hot-add for services purpose anyway.
Yinghai
Quoted 26 lines · Yinghai Lu
Quoted 8 lines · Greg KH
Quoted 7 lines · Yinghai Lu
Quoted 6 lines · H. Peter Anvin
On 06/21/2013 09:51 AM, Greg KH wrote:
I suspect the cutoff for this should be a lot lower than 8 TB even, more like 128 GB or so. The only concern is to not set the cutoff so low that we can end up running out of memory or with suboptimal NUMA placement just because of this.
Quoted 6 lines
Quoted 3 lines
I would suggest another way:
only boot the system with boot node (include cpu, ram and pci root buses). then after boot, could add other nodes.What exactly do you mean by "after boot"? Often, the boot process of userspace needs those additional cpus and ram in order to initialize everything (like the pci devices) properly.
I mean for Intel cpu have cpu and memory controller and IIO. every IIO is one peer pci root bus.
So scan root bus that are not with boot node later.in this way we can keep all numa etc on the place when online ram, cpu, pci...
For example if we have 32 sockets system, most time for boot is with *BIOS* instead of OS. In those kind of system boot is like this way:
only first two sockets get booted from bios to OS.
later use hot add every other two sockets.
Quoted 4 lines
that will also make BIOS simpler, and it need to support hot-add for services purpose anyway.
Yinghai
Yes the hot add path was one option we looked at and it did shorten boot times but the goal I had here is to get from power on to having the full machine available as quick as possible. Several clients need significant portions of ram for their key workloads. So that guided my thoughts on this patch.
Quoted 10 lines · Nathan Zimmer
Quoted 9 lines · Yinghai Lu
in this way we can keep all numa etc on the place when online ram, cpu, pci...
For example if we have 32 sockets system, most time for boot is with *BIOS*
instead of OS. In those kind of system boot is like this way:
only first two sockets get booted from bios to OS.
later use hot add every other two sockets.
Quoted 8 lines
Quoted 3 lines
that will also make BIOS simpler, and it need to support hot-add for services purpose anyway.
Yes the hot add path was one option we looked at and it did shorten boot times but the goal I had here is to get from power on to having the full machine available as quick as possible. Several clients need significant portions of ram for their key workloads. So that guided my thoughts on this patch.
you mean boot small number sockets + hot add still take more time? hot add ram still in serialize mode?
Yinghai
Quoted 20 lines · Yinghai Lu
Quoted 10 lines · Nathan Zimmer
Quoted 9 lines · Yinghai Lu
in this way we can keep all numa etc on the place when online ram, cpu, pci...
For example if we have 32 sockets system, most time for boot is with *BIOS*
instead of OS. In those kind of system boot is like this way:
only first two sockets get booted from bios to OS.
later use hot add every other two sockets.Quoted 8 lines
Quoted 3 lines
that will also make BIOS simpler, and it need to support hot-add for services purpose anyway.
Yes the hot add path was one option we looked at and it did shorten boot times but the goal I had here is to get from power on to having the full machine available as quick as possible. Several clients need significant portions of ram for their key workloads. So that guided my thoughts on this patch.
Quoted 4 lines
you mean boot small number sockets + hot add still take more time? hot add ram still in serialize mode?
Yinghai
Yes the hot add path is still fairly serial. lock_memory_hotplug() is still a global lock, which effectivly serializes online pages. Since the hotplug lock could done per node but for that to be effective something would need to be done with lock_system_sleep() since is called by lock_memory_hotplug().
Nate
Quoted 9 lines · Greg KH
Quoted 8 lines · Yinghai Lu
Quoted 6 lines · H. Peter Anvin
On 06/21/2013 09:51 AM, Greg KH wrote:
I suspect the cutoff for this should be a lot lower than 8 TB even, more like 128 GB or so. The only concern is to not set the cutoff so low that we can end up running out of memory or with suboptimal NUMA placement just because of this.
Quoted 7 lines
Quoted 3 lines
I would suggest another way:
only boot the system with boot node (include cpu, ram and pci root buses). then after boot, could add other nodes.What exactly do you mean by "after boot"? Often, the boot process of userspace needs those additional cpus and ram in order to initialize everything (like the pci devices) properly.
Exactly. That's why I left both low and high memory on each node.
Quoted 5 lines
thanks,
greg k-h
Quoted 16 lines · Mike Travis
Quoted 13 lines · Greg KH
Quoted 11 lines · Yinghai Lu
Quoted 6 lines · H. Peter Anvin
On 06/21/2013 09:51 AM, Greg KH wrote:
I suspect the cutoff for this should be a lot lower than 8 TB even, more like 128 GB or so. The only concern is to not set the cutoff so low that we can end up running out of memory or with suboptimal NUMA placement just because of this.
I would suggest another way:
only boot the system with boot node (include cpu, ram and pci root buses). then after boot, could add other nodes.
Quoted 5 lines
Quoted 3 lines
What exactly do you mean by "after boot"? Often, the boot process of userspace needs those additional cpus and ram in order to initialize everything (like the pci devices) properly.
Exactly. That's why I left both low and high memory on each node.
looks like you assume every node have same ram, and before booting you you need to know memory layout to append the boot command line.
We have patchset that moving srat table parse early.git://git.kernel.org/pub/scm/linux/kernel/git/yinghai/linux-yinghai.git
for-x86-mm
https://git.kernel.org/cgit/linux/kernel/git/yinghai/linux-yinghai.git/log/?h=for-x86-mm
on top that, we could make your patch pass more simple command like 1/2^n of every node, and only need to pass n instead.
Thanks
Yinghai
Quoted 26 lines · Yinghai Lu
Quoted 16 lines · Mike Travis
Quoted 13 lines · Greg KH
Quoted 11 lines · Yinghai Lu
Quoted 6 lines · H. Peter Anvin
On 06/21/2013 09:51 AM, Greg KH wrote:
I suspect the cutoff for this should be a lot lower than 8 TB even, more like 128 GB or so. The only concern is to not set the cutoff so low that we can end up running out of memory or with suboptimal NUMA placement just because of this.
I would suggest another way:
only boot the system with boot node (include cpu, ram and pci root buses). then after boot, could add other nodes.Quoted 5 lines
Quoted 3 lines
What exactly do you mean by "after boot"? Often, the boot process of userspace needs those additional cpus and ram in order to initialize everything (like the pci devices) properly.
Exactly. That's why I left both low and high memory on each node.
looks like you assume every node have same ram, and before booting you you need to know memory layout to append the boot command line.
Quoted 7 lines
We have patchset that moving srat table parse early.
git://git.kernel.org/pub/scm/linux/kernel/git/yinghai/linux-yinghai.git
for-x86-mm
https://git.kernel.org/cgit/linux/kernel/git/yinghai/linux-yinghai.git/log/?h=for-x86-mmon top that, we could make your patch pass more simple command like 1/2^n of every node, and only need to pass n instead.
The two params that I couldn't figure out how to provide except via kernel param option was the memory block size (128M or 2G) and the physical address space per node. The other 3 params can be automatically setup by a script when the total system size is known. As soon as we verify on the 32TB system and surmise what will be needed for 64TB, then those 3 params can probably disappear.
Quoted 5 lines
Thanks
Yinghai
H. Peter Anvin
reviewerQuoted 8 lines · Mike Travis
The two params that I couldn't figure out how to provide except via kernel param option was the memory block size (128M or 2G) and the physical address space per node. The other 3 params can be automatically setup by a script when the total system size is known. As soon as we verify on the 32TB system and surmise what will be needed for 64TB, then those 3 params can probably disappear.
"Setup by script" is a no-go. You *have* the total system size already, it is in the e820 tables (anything which isn't in e820 is hotplug, that automagically gets deferred.)
However, please consider Ingo's counterproposal of doing this via the buddy allocator, i.e. hugepages being broken on demand. That is a *very* powerful model, although would require more infrastructure.
-hpa
Quoted 10 lines · H. Peter Anvin
Quoted 8 lines · Mike Travis
The two params that I couldn't figure out how to provide except via kernel param option was the memory block size (128M or 2G) and the physical address space per node. The other 3 params can be automatically setup by a script when the total system size is known. As soon as we verify on the 32TB system and surmise what will be needed for 64TB, then those 3 params can probably disappear.
Quoted 3 lines
"Setup by script" is a no-go. You *have* the total system size already, it is in the e820 tables (anything which isn't in e820 is hotplug, that automagically gets deferred.)
Okay, I'll figure something out. If Yinghai's SRAT patch can help with the node address space, then I might be able to determine if the system is a UV which is the only system I see that uses 2G memory blocks. (Or make get_memory_block_size() a global.)
Then a simple param to start the insertion early or defer it until the system is fully up is still useful, and that's easy to understand.
[I think we still want to keep the actual process of moving memory to the absent list an option, yes? If for no other reason except to rule out this code when a problem crops up. Or at least have a way to disable the process if it's CONFIG'd in.]
Quoted 4 lines
However, please consider Ingo's counterproposal of doing this via the buddy allocator, i.e. hugepages being broken on demand. That is a *very* powerful model, although would require more infrastructure.
We will certainly continue to make improvements as larger system sizes become more commonplace (and customers continue to complain :). But we are cutting it close to including this into the nextgen distro releases, so it would have to be a follow on project. (I've been working on this patch since last November.)
Thanks,
Mike
Quoted 4 lines
-hpa
Quoted 5 lines · H. Peter Anvin
On 06/25/2013 10:35 AM, Mike Travis wrote:
However, please consider Ingo's counterproposal of doing this via the buddy allocator, i.e. hugepages being broken on demand. That is a *very* powerful model, although would require more infrastructure.
Can you or Ingo elaborate more about the buddy allocator proposal?
Thanks
H. Peter Anvin
reviewerQuoted 9 lines · Yinghai Lu
Quoted 5 lines · H. Peter Anvin
On 06/25/2013 10:35 AM, Mike Travis wrote:
However, please consider Ingo's counterproposal of doing this via the buddy allocator, i.e. hugepages being broken on demand. That is a *very* powerful model, although would require more infrastructure.
Can you or Ingo elaborate more about the buddy allocator proposal?
Start by initializing 1G hyperpages only, but mark them so that the allocator knows that if it needs to break them apart it has to initialize the page structures for the 2M subpages.
Same thing with 2M -> 4K.
-hpa
Quoted 11 lines · H. Peter Anvin
Quoted 9 lines · Yinghai Lu
Quoted 5 lines · H. Peter Anvin
On 06/25/2013 10:35 AM, Mike Travis wrote:
However, please consider Ingo's counterproposal of doing this via the buddy allocator, i.e. hugepages being broken on demand. That is a *very* powerful model, although would require more infrastructure.
Can you or Ingo elaborate more about the buddy allocator proposal?
Quoted 9 lines
Start by initializing 1G hyperpages only, but mark them so that the allocator knows that if it needs to break them apart it has to initialize the page structures for the 2M subpages.
Same thing with 2M -> 4K.
-hpa
It is worth experimenting with but the big question would be, if it still avoids the very expensive "memmap_init_zone" and it's sub-functions using huge expanses of memory. I'll do some experimenting as soon as I can. Our 32TB system is being brought back to 16TB (we found a number of problems as we get closer and closer to the 64TB limit), but that's still a significant size.
Quoted 7 lines
It is worth experimenting with but the big question would be, if it still avoids the very expensive "
memmap_init_zone" and it's sub-functions using huge expanses of memory. I'll do some experimenting as soon as I can. Our 32TB system is being brought back to 16TB (we found a number of problems as we get closer and closer to the 64TB limit), but that's still a significant size.
According to Intel SDM, CPU could support 52bits physical addressing.
So how linux kernel will handle it? as we only have 48bits virtual addressing.
Yinghai
H. Peter Anvin
reviewerQuoted 8 lines · Yinghai Lu
Quoted 7 lines
It is worth experimenting with but the big question would be, if it still avoids the very expensive "
memmap_init_zone" and it's sub-functions using huge expanses of memory. I'll do some experimenting as soon as I can. Our 32TB system is being brought back to 16TB (we found a number of problems as we get closer and closer to the 64TB limit), but that's still a significant size.
Quoted 4 lines
According to Intel SDM, CPU could support 52bits physical addressing.
So how linux kernel will handle it? as we only have 48bits virtual addressing.
The Linux kernel will not support more than V-2 bits of physical address space for any number V of virtual address bits.
-hpa
Quoted 7 lines · H. Peter Anvin
Quoted 4 lines
According to Intel SDM, CPU could support 52bits physical addressing.
So how linux kernel will handle it? as we only have 48bits virtual addressing.
The Linux kernel will not support more than V-2 bits of physical address space for any number V of virtual address bits.
ok, then will need cpu support 57bits virtual addressing, and level 5 page table?
Yinghai
Quoted 4 lines · Mike Travis
experimenting as soon as I can. Our 32TB system is being brought back to 16TB (we found a number of problems as we get closer and closer to the 64TB limit), but that's still a significant size.
Hi, Mike,
Can you post e820 memory map on system that have 32TiB or more?
Is there one range size more than 16TiB? like [16TiB, 32TiB)...
Thanks
Yinghai
Quoted 11 lines · Yinghai Lu
Quoted 4 lines · Mike Travis
experimenting as soon as I can. Our 32TB system is being brought back to 16TB (we found a number of problems as we get closer and closer to the 64TB limit), but that's still a significant size.
Hi, Mike,
Can you post e820 memory map on system that have 32TiB or more?
Is there one range size more than 16TiB? like [16TiB, 32TiB)...
[ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: 0000000000000000 - 000000000007f000 (usable) [ 0.000000] BIOS-e820: 000000000007f000 - 0000000000080000 (reserved) [ 0.000000] BIOS-e820: 0000000000080000 - 00000000000a0000 (usable) [ 0.000000] BIOS-e820: 0000000000100000 - 000000007abb1000 (usable) [ 0.000000] BIOS-e820: 000000007abb1000 - 000000007abb2000 (reserved) [ 0.000000] BIOS-e820: 000000007abb2000 - 000000007abc5000 (usable) [ 0.000000] BIOS-e820: 000000007abc5000 - 000000007ae05000 (reserved) [ 0.000000] BIOS-e820: 000000007ae05000 - 000000007bb13000 (usable) [ 0.000000] BIOS-e820: 000000007bb13000 - 000000007bc13000 (reserved) [ 0.000000] BIOS-e820: 000000007bc13000 - 000000007bd13000 (unusable) [ 0.000000] BIOS-e820: 000000007bd13000 - 000000007be13000 (reserved) [ 0.000000] BIOS-e820: 000000007be13000 - 000000007d1f3000 (ACPI NVS) [ 0.000000] BIOS-e820: 000000007d1f3000 - 000000007e000000 (ACPI data) [ 0.000000] BIOS-e820: 000000007e000000 - 000000007e2ee000 (usable) [ 0.000000] BIOS-e820: 000000007e2ee000 - 000000007ef3f000 (ACPI data) [ 0.000000] BIOS-e820: 000000007ef3f000 - 000000007f000000 (usable) [ 0.000000] BIOS-e820: 0000000100000000 - 0000001f80000000 (usable) [ 0.000000] BIOS-e820: 0000004000000000 - 0000005eff000000 (usable) [ 0.000000] BIOS-e820: 0000008000000000 - 0000009eff000000 (usable) [ 0.000000] BIOS-e820: 000000c000000000 - 000000deff000000 (usable) [ 0.000000] BIOS-e820: 0000010000000000 - 0000011eff000000 (usable) [ 0.000000] BIOS-e820: 0000014000000000 - 0000015eff000000 (usable) [ 0.000000] BIOS-e820: 0000018000000000 - 0000019eff000000 (usable) [ 0.000000] BIOS-e820: 000001c000000000 - 000001deff000000 (usable) [ 0.000000] BIOS-e820: 0000020000000000 - 0000021eff000000 (usable) [ 0.000000] BIOS-e820: 0000024000000000 - 0000025eff000000 (usable) [ 0.000000] BIOS-e820: 0000028000000000 - 0000029eff000000 (usable) [ 0.000000] BIOS-e820: 000002c000000000 - 000002deff000000 (usable) [ 0.000000] BIOS-e820: 0000030000000000 - 0000031eff000000 (usable) [ 0.000000] BIOS-e820: 0000034000000000 - 0000035eff000000 (usable) [ 0.000000] BIOS-e820: 0000038000000000 - 0000039eff000000 (usable) [ 0.000000] BIOS-e820: 000003c000000000 - 000003deff000000 (usable) [ 0.000000] BIOS-e820: 0000040000000000 - 0000041eff000000 (usable) [ 0.000000] BIOS-e820: 0000044000000000 - 0000045eff000000 (usable) [ 0.000000] BIOS-e820: 0000048000000000 - 0000049eff000000 (usable) [ 0.000000] BIOS-e820: 000004c000000000 - 000004deff000000 (usable) [ 0.000000] BIOS-e820: 0000050000000000 - 0000051eff000000 (usable) [ 0.000000] BIOS-e820: 0000054000000000 - 0000055eff000000 (usable) [ 0.000000] BIOS-e820: 0000058000000000 - 0000059eff000000 (usable) [ 0.000000] BIOS-e820: 000005c000000000 - 000005deff000000 (usable) [ 0.000000] BIOS-e820: 0000060000000000 - 0000061eff000000 (usable) [ 0.000000] BIOS-e820: 0000064000000000 - 0000065eff000000 (usable) [ 0.000000] BIOS-e820: 0000068000000000 - 0000069eff000000 (usable) [ 0.000000] BIOS-e820: 000006c000000000 - 000006deff000000 (usable) [ 0.000000] BIOS-e820: 0000070000000000 - 0000071eff000000 (usable) [ 0.000000] BIOS-e820: 0000074000000000 - 0000075eff000000 (usable) [ 0.000000] BIOS-e820: 0000078000000000 - 0000079eff000000 (usable) [ 0.000000] BIOS-e820: 000007c000000000 - 000007deff000000 (usable) [ 0.000000] BIOS-e820: 0000080000000000 - 0000081eff000000 (usable) [ 0.000000] BIOS-e820: 0000084000000000 - 0000085eff000000 (usable) [ 0.000000] BIOS-e820: 0000088000000000 - 0000089eff000000 (usable) [ 0.000000] BIOS-e820: 000008c000000000 - 000008deff000000 (usable) [ 0.000000] BIOS-e820: 0000090000000000 - 0000091eff000000 (usable) [ 0.000000] BIOS-e820: 0000094000000000 - 0000095eff000000 (usable) [ 0.000000] BIOS-e820: 0000098000000000 - 0000099eff000000 (usable) [ 0.000000] BIOS-e820: 000009c000000000 - 000009deff000000 (usable) [ 0.000000] BIOS-e820: 00000a0000000000 - 00000a1eff000000 (usable) [ 0.000000] BIOS-e820: 00000a4000000000 - 00000a5eff000000 (usable) [ 0.000000] BIOS-e820: 00000a8000000000 - 00000a9eff000000 (usable) [ 0.000000] BIOS-e820: 00000ac000000000 - 00000adeff000000 (usable) [ 0.000000] BIOS-e820: 00000b0000000000 - 00000b1eff000000 (usable) [ 0.000000] BIOS-e820: 00000b4000000000 - 00000b5eff000000 (usable) [ 0.000000] BIOS-e820: 00000b8000000000 - 00000b9eff000000 (usable) [ 0.000000] BIOS-e820: 00000bc000000000 - 00000bdeff000000 (usable) [ 0.000000] BIOS-e820: 00000c0000000000 - 00000c1eff000000 (usable) [ 0.000000] BIOS-e820: 00000c4000000000 - 00000c5eff000000 (usable) [ 0.000000] BIOS-e820: 00000c8000000000 - 00000c9eff000000 (usable) [ 0.000000] BIOS-e820: 00000cc000000000 - 00000cdeff000000 (usable) [ 0.000000] BIOS-e820: 00000d0000000000 - 00000d1eff000000 (usable) [ 0.000000] BIOS-e820: 00000d4000000000 - 00000d5eff000000 (usable) [ 0.000000] BIOS-e820: 00000d8000000000 - 00000d9eff000000 (usable) [ 0.000000] BIOS-e820: 00000dc000000000 - 00000ddeff000000 (usable) [ 0.000000] BIOS-e820: 00000e0000000000 - 00000e1eff000000 (usable) [ 0.000000] BIOS-e820: 00000e4000000000 - 00000e5eff000000 (usable) [ 0.000000] BIOS-e820: 00000e8000000000 - 00000e9eff000000 (usable) [ 0.000000] BIOS-e820: 00000ec000000000 - 00000edeff000000 (usable) [ 0.000000] BIOS-e820: 00000f0000000000 - 00000f1eff000000 (usable) [ 0.000000] BIOS-e820: 00000f4000000000 - 00000f5eff000000 (usable) [ 0.000000] BIOS-e820: 00000f8000000000 - 00000f9eff000000 (usable) [ 0.000000] BIOS-e820: 00000fc000000000 - 00000fdeff000000 (usable) [ 0.000000] BIOS-e820: 0000100000000000 - 0000101eff000000 (usable) [ 0.000000] BIOS-e820: 0000104000000000 - 0000105eff000000 (usable) [ 0.000000] BIOS-e820: 0000108000000000 - 0000109eff000000 (usable) [ 0.000000] BIOS-e820: 000010c000000000 - 000010deff000000 (usable) [ 0.000000] BIOS-e820: 0000110000000000 - 0000111eff000000 (usable) [ 0.000000] BIOS-e820: 0000114000000000 - 0000115eff000000 (usable) [ 0.000000] BIOS-e820: 0000118000000000 - 0000119eff000000 (usable) [ 0.000000] BIOS-e820: 000011c000000000 - 000011deff000000 (usable) [ 0.000000] BIOS-e820: 0000120000000000 - 0000121eff000000 (usable) [ 0.000000] BIOS-e820: 0000124000000000 - 0000125eff000000 (usable) [ 0.000000] BIOS-e820: 0000128000000000 - 0000129eff000000 (usable) [ 0.000000] BIOS-e820: 000012c000000000 - 000012deff000000 (usable) [ 0.000000] BIOS-e820: 0000130000000000 - 0000131eff000000 (usable) [ 0.000000] BIOS-e820: 0000134000000000 - 0000135eff000000 (usable) [ 0.000000] BIOS-e820: 0000138000000000 - 0000139eff000000 (usable) [ 0.000000] BIOS-e820: 000013c000000000 - 000013deff000000 (usable) [ 0.000000] BIOS-e820: 0000140000000000 - 0000141eff000000 (usable) [ 0.000000] BIOS-e820: 0000144000000000 - 0000145eff000000 (usable) [ 0.000000] BIOS-e820: 0000148000000000 - 0000149eff000000 (usable) [ 0.000000] BIOS-e820: 000014c000000000 - 000014deff000000 (usable) [ 0.000000] BIOS-e820: 0000150000000000 - 0000151eff000000 (usable) [ 0.000000] BIOS-e820: 0000154000000000 - 0000155eff000000 (usable) [ 0.000000] BIOS-e820: 0000158000000000 - 0000159eff000000 (usable) [ 0.000000] BIOS-e820: 000015c000000000 - 000015deff000000 (usable) [ 0.000000] BIOS-e820: 0000160000000000 - 0000161eff000000 (usable) [ 0.000000] BIOS-e820: 0000164000000000 - 0000165eff000000 (usable) [ 0.000000] BIOS-e820: 0000168000000000 - 0000169eff000000 (usable) [ 0.000000] BIOS-e820: 000016c000000000 - 000016deff000000 (usable) [ 0.000000] BIOS-e820: 0000170000000000 - 0000171eff000000 (usable) [ 0.000000] BIOS-e820: 0000174000000000 - 0000175eff000000 (usable) [ 0.000000] BIOS-e820: 0000178000000000 - 0000179eff000000 (usable) [ 0.000000] BIOS-e820: 000017c000000000 - 000017deff000000 (usable) [ 0.000000] BIOS-e820: 0000180000000000 - 0000181eff000000 (usable) [ 0.000000] BIOS-e820: 0000184000000000 - 0000185eff000000 (usable) [ 0.000000] BIOS-e820: 0000188000000000 - 0000189eff000000 (usable) [ 0.000000] BIOS-e820: 000018c000000000 - 000018deff000000 (usable) [ 0.000000] BIOS-e820: 0000190000000000 - 0000191eff000000 (usable) [ 0.000000] BIOS-e820: 0000194000000000 - 0000195eff000000 (usable) [ 0.000000] BIOS-e820: 0000198000000000 - 0000199eff000000 (usable) [ 0.000000] BIOS-e820: 000019c000000000 - 000019deff000000 (usable) [ 0.000000] BIOS-e820: 00001a0000000000 - 00001a1eff000000 (usable) [ 0.000000] BIOS-e820: 00001a4000000000 - 00001a5eff000000 (usable) [ 0.000000] BIOS-e820: 00001a8000000000 - 00001a9eff000000 (usable) [ 0.000000] BIOS-e820: 00001ac000000000 - 00001adeff000000 (usable) [ 0.000000] BIOS-e820: 00001b0000000000 - 00001b1eff000000 (usable) [ 0.000000] BIOS-e820: 00001b4000000000 - 00001b5eff000000 (usable) [ 0.000000] BIOS-e820: 00001b8000000000 - 00001b9eff000000 (usable) [ 0.000000] BIOS-e820: 00001bc000000000 - 00001bdeff000000 (usable) [ 0.000000] bootconsole [earlyser0] enabled [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] EFI v2.30 by EDK II [ 0.000000] ACPI=0x7ef3e000 ACPI 2.0=0x7ef3e014 SMBIOS=0x7be0f000 UVsystab=0x7bbcb000 [ 0.000000] Kernel-defined memdesc doesn't match the one from EFI! [ 0.000000] EFI: mem00: type=3, attr=0xf, range=[0x0000000000000000-0x0000000000001000) (0MB) [ 0.000000] EFI: mem01: type=7, attr=0xf, range=[0x0000000000001000-0x000000000005f000) (0MB) [ 0.000000] EFI: mem02: type=4, attr=0xf, range=[0x000000000005f000-0x0000000000060000) (0MB) [ 0.000000] EFI: mem03: type=3, attr=0xf, range=[0x0000000000060000-0x000000000007f000) (0MB) [ 0.000000] EFI: mem04: type=0, attr=0xf, range=[0x000000000007f000-0x0000000000080000) (0MB) [ 0.000000] EFI: mem05: type=3, attr=0xf, range=[0x0000000000080000-0x00000000000a0000) (0MB) [ 0.000000] EFI: mem06: type=2, attr=0xf, range=[0x0000000000100000-0x0000000000400000) (3MB) [ 0.000000] EFI: mem07: type=3, attr=0xf, range=[0x0000000000400000-0x00000000004ee000) (0MB) [ 0.000000] EFI: mem08: type=7, attr=0xf, range=[0x00000000004ee000-0x0000000000c00000) (7MB) [ 0.000000] EFI: mem09: type=3, attr=0xf, range=[0x0000000000c00000-0x0000000001000000) (4MB) [ 0.000000] EFI: mem10: type=2, attr=0xf, range=[0x0000000001000000-0x0000000001800000) (8MB) [ 0.000000] EFI: mem11: type=7, attr=0xf, range=[0x0000000001800000-0x0000000064821000) (1584MB) [ 0.000000] EFI: mem12: type=2, attr=0xf, range=[0x0000000064821000-0x0000000065051000) (8MB) [ 0.000000] EFI: mem13: type=1, attr=0xf, range=[0x0000000065051000-0x000000006508f000) (0MB) [ 0.000000] EFI: mem14: type=4, attr=0xf, range=[0x000000006508f000-0x000000006514c000) (0MB) [ 0.000000] EFI: mem15: type=1, attr=0xf, range=[0x000000006514c000-0x0000000065209000) (0MB) [ 0.000000] EFI: mem16: type=4, attr=0xf, range=[0x0000000065209000-0x00000000652ab000) (0MB) [ 0.000000] EFI: mem17: type=7, attr=0xf, range=[0x00000000652ab000-0x00000000652ad000) (0MB) [ 0.000000] EFI: mem18: type=4, attr=0xf, range=[0x00000000652ad000-0x00000000652ae000) (0MB) [ 0.000000] EFI: mem19: type=7, attr=0xf, range=[0x00000000652ae000-0x00000000652b6000) (0MB) [ 0.000000] EFI: mem20: type=4, attr=0xf, range=[0x00000000652b6000-0x00000000652bb000) (0MB) [ 0.000000] EFI: mem21: type=7, attr=0xf, range=[0x00000000652bb000-0x00000000652bc000) (0MB) [ 0.000000] EFI: mem22: type=4, attr=0xf, range=[0x00000000652bc000-0x00000000652d3000) (0MB) [ 0.000000] EFI: mem23: type=3, attr=0xf, range=[0x00000000652d3000-0x00000000652d6000) (0MB) [ 0.000000] EFI: mem24: type=4, attr=0xf, range=[0x00000000652d6000-0x00000000652e4000) (0MB) [ 0.000000] EFI: mem25: type=3, attr=0xf, range=[0x00000000652e4000-0x00000000652f8000) (0MB) [ 0.000000] EFI: mem26: type=4, attr=0xf, range=[0x00000000652f8000-0x0000000066a99000) (23MB) [ 0.000000] EFI: mem27: type=3, attr=0xf, range=[0x0000000066a99000-0x0000000066a9f000) (0MB) [ 0.000000] EFI: mem28: type=4, attr=0xf, range=[0x0000000066a9f000-0x000000006814e000) (22MB) [ 0.000000] EFI: mem29: type=3, attr=0xf, range=[0x000000006814e000-0x00000000681b7000) (0MB) [ 0.000000] EFI: mem30: type=4, attr=0xf, range=[0x00000000681b7000-0x0000000068307000) (1MB) [ 0.000000] EFI: mem31: type=3, attr=0xf, range=[0x0000000068307000-0x000000006830a000) (0MB) [ 0.000000] EFI: mem32: type=4, attr=0xf, range=[0x000000006830a000-0x0000000068338000) (0MB) [ 0.000000] EFI: mem33: type=3, attr=0xf, range=[0x0000000068338000-0x0000000068369000) (0MB) [ 0.000000] EFI: mem34: type=4, attr=0xf, range=[0x0000000068369000-0x0000000068390000) (0MB) [ 0.000000] EFI: mem35: type=3, attr=0xf, range=[0x0000000068390000-0x0000000068393000) (0MB) [ 0.000000] EFI: mem36: type=4, attr=0xf, range=[0x0000000068393000-0x000000006839c000) (0MB) [ 0.000000] EFI: mem37: type=3, attr=0xf, range=[0x000000006839c000-0x00000000683b0000) (0MB) [ 0.000000] EFI: mem38: type=4, attr=0xf, range=[0x00000000683b0000-0x00000000683b1000) (0MB) [ 0.000000] EFI: mem39: type=3, attr=0xf, range=[0x00000000683b1000-0x00000000683b4000) (0MB) [ 0.000000] EFI: mem40: type=4, attr=0xf, range=[0x00000000683b4000-0x00000000683b5000) (0MB) [ 0.000000] EFI: mem41: type=3, attr=0xf, range=[0x00000000683b5000-0x00000000683bd000) (0MB) [ 0.000000] EFI: mem42: type=4, attr=0xf, range=[0x00000000683bd000-0x00000000683be000) (0MB) [ 0.000000] EFI: mem43: type=3, attr=0xf, range=[0x00000000683be000-0x00000000683c9000) (0MB) [ 0.000000] EFI: mem44: type=4, attr=0xf, range=[0x00000000683c9000-0x00000000683ca000) (0MB) [ 0.000000] EFI: mem45: type=3, attr=0xf, range=[0x00000000683ca000-0x00000000683cc000) (0MB) [ 0.000000] EFI: mem46: type=4, attr=0xf, range=[0x00000000683cc000-0x00000000683ce000) (0MB) [ 0.000000] EFI: mem47: type=3, attr=0xf, range=[0x00000000683ce000-0x00000000683d4000) (0MB) [ 0.000000] EFI: mem48: type=4, attr=0xf, range=[0x00000000683d4000-0x00000000683d9000) (0MB) [ 0.000000] EFI: mem49: type=3, attr=0xf, range=[0x00000000683d9000-0x00000000683dc000) (0MB) [ 0.000000] EFI: mem50: type=4, attr=0xf, range=[0x00000000683dc000-0x00000000683df000) (0MB) [ 0.000000] EFI: mem51: type=3, attr=0xf, range=[0x00000000683df000-0x00000000683eb000) (0MB) [ 0.000000] EFI: mem52: type=4, attr=0xf, range=[0x00000000683eb000-0x000000006840e000) (0MB) [ 0.000000] EFI: mem53: type=3, attr=0xf, range=[0x000000006840e000-0x0000000068415000) (0MB) [ 0.000000] EFI: mem54: type=4, attr=0xf, range=[0x0000000068415000-0x000000006841b000) (0MB) [ 0.000000] EFI: mem55: type=3, attr=0xf, range=[0x000000006841b000-0x0000000068420000) (0MB) [ 0.000000] EFI: mem56: type=4, attr=0xf, range=[0x0000000068420000-0x0000000068423000) (0MB) [ 0.000000] EFI: mem57: type=3, attr=0xf, range=[0x0000000068423000-0x0000000068436000) (0MB) [ 0.000000] EFI: mem58: type=4, attr=0xf, range=[0x0000000068436000-0x0000000068437000) (0MB) [ 0.000000] EFI: mem59: type=3, attr=0xf, range=[0x0000000068437000-0x000000006843f000) (0MB) [ 0.000000] EFI: mem60: type=4, attr=0xf, range=[0x000000006843f000-0x0000000068440000) (0MB) [ 0.000000] EFI: mem61: type=3, attr=0xf, range=[0x0000000068440000-0x0000000068459000) (0MB) [ 0.000000] EFI: mem62: type=4, attr=0xf, range=[0x0000000068459000-0x000000006845b000) (0MB) [ 0.000000] EFI: mem63: type=3, attr=0xf, range=[0x000000006845b000-0x0000000068475000) (0MB) [ 0.000000] EFI: mem64: type=4, attr=0xf, range=[0x0000000068475000-0x0000000068478000) (0MB) [ 0.000000] EFI: mem65: type=3, attr=0xf, range=[0x0000000068478000-0x0000000068493000) (0MB) [ 0.000000] EFI: mem66: type=4, attr=0xf, range=[0x0000000068493000-0x0000000068495000) (0MB) [ 0.000000] EFI: mem67: type=3, attr=0xf, range=[0x0000000068495000-0x000000006849b000) (0MB) [ 0.000000] EFI: mem68: type=4, attr=0xf, range=[0x000000006849b000-0x000000006849c000) (0MB) [ 0.000000] EFI: mem69: type=3, attr=0xf, range=[0x000000006849c000-0x00000000684a9000) (0MB) [ 0.000000] EFI: mem70: type=4, attr=0xf, range=[0x00000000684a9000-0x00000000684ab000) (0MB) [ 0.000000] EFI: mem71: type=3, attr=0xf, range=[0x00000000684ab000-0x00000000684c2000) (0MB) [ 0.000000] EFI: mem72: type=4, attr=0xf, range=[0x00000000684c2000-0x00000000684c6000) (0MB) [ 0.000000] EFI: mem73: type=3, attr=0xf, range=[0x00000000684c6000-0x00000000684e5000) (0MB) [ 0.000000] EFI: mem74: type=4, attr=0xf, range=[0x00000000684e5000-0x00000000684e6000) (0MB) [ 0.000000] EFI: mem75: type=3, attr=0xf, range=[0x00000000684e6000-0x00000000684e8000) (0MB) [ 0.000000] EFI: mem76: type=4, attr=0xf, range=[0x00000000684e8000-0x00000000684ef000) (0MB) [ 0.000000] EFI: mem77: type=3, attr=0xf, range=[0x00000000684ef000-0x0000000068524000) (0MB) [ 0.000000] EFI: mem78: type=4, attr=0xf, range=[0x0000000068524000-0x0000000068525000) (0MB) [ 0.000000] EFI: mem79: type=3, attr=0xf, range=[0x0000000068525000-0x000000006855f000) (0MB) [ 0.000000] EFI: mem80: type=4, attr=0xf, range=[0x000000006855f000-0x0000000068560000) (0MB) [ 0.000000] EFI: mem81: type=3, attr=0xf, range=[0x0000000068560000-0x000000006856b000) (0MB) [ 0.000000] EFI: mem82: type=4, attr=0xf, range=[0x000000006856b000-0x000000006856d000) (0MB) [ 0.000000] EFI: mem83: type=3, attr=0xf, range=[0x000000006856d000-0x0000000068577000) (0MB) [ 0.000000] EFI: mem84: type=4, attr=0xf, range=[0x0000000068577000-0x0000000068578000) (0MB) [ 0.000000] EFI: mem85: type=3, attr=0xf, range=[0x0000000068578000-0x000000006858a000) (0MB) [ 0.000000] EFI: mem86: type=4, attr=0xf, range=[0x000000006858a000-0x0000000068632000) (0MB) [ 0.000000] EFI: mem87: type=3, attr=0xf, range=[0x0000000068632000-0x0000000068643000) (0MB) [ 0.000000] EFI: mem88: type=4, attr=0xf, range=[0x0000000068643000-0x0000000068805000) (1MB) [ 0.000000] EFI: mem89: type=3, attr=0xf, range=[0x0000000068805000-0x0000000068809000) (0MB) [ 0.000000] EFI: mem90: type=7, attr=0xf, range=[0x0000000068809000-0x000000006880a000) (0MB) [ 0.000000] EFI: mem91: type=4, attr=0xf, range=[0x000000006880a000-0x000000006880c000) (0MB) [ 0.000000] EFI: mem92: type=7, attr=0xf, range=[0x000000006880c000-0x000000006880d000) (0MB) [ 0.000000] EFI: mem93: type=4, attr=0xf, range=[0x000000006880d000-0x000000006880e000) (0MB) [ 0.000000] EFI: mem94: type=7, attr=0xf, range=[0x000000006880e000-0x0000000068838000) (0MB) [ 0.000000] EFI: mem95: type=2, attr=0xf, range=[0x0000000068838000-0x0000000068850000) (0MB) [ 0.000000] EFI: mem96: type=4, attr=0xf, range=[0x0000000068850000-0x00000000691e6000) (9MB) [ 0.000000] EFI: mem97: type=3, attr=0xf, range=[0x00000000691e6000-0x00000000692a4000) (0MB) [ 0.000000] EFI: mem98: type=4, attr=0xf, range=[0x00000000692a4000-0x00000000692ad000) (0MB) [ 0.000000] EFI: mem99: type=3, attr=0xf, range=[0x00000000692ad000-0x00000000692c2000) (0MB) [ 0.000000] EFI: mem100: type=4, attr=0xf, range=[0x00000000692c2000-0x00000000692d5000) (0MB) [ 0.000000] EFI: mem101: type=3, attr=0xf, range=[0x00000000692d5000-0x0000000069305000) (0MB) [ 0.000000] EFI: mem102: type=4, attr=0xf, range=[0x0000000069305000-0x000000006932b000) (0MB) [ 0.000000] EFI: mem103: type=7, attr=0xf, range=[0x000000006932b000-0x000000006932c000) (0MB) [ 0.000000] EFI: mem104: type=4, attr=0xf, range=[0x000000006932c000-0x0000000069339000) (0MB) [ 0.000000] EFI: mem105: type=3, attr=0xf, range=[0x0000000069339000-0x0000000069342000) (0MB) [ 0.000000] EFI: mem106: type=4, attr=0xf, range=[0x0000000069342000-0x0000000069347000) (0MB) [ 0.000000] EFI: mem107: type=3, attr=0xf, range=[0x0000000069347000-0x0000000069348000) (0MB) [ 0.000000] EFI: mem108: type=4, attr=0xf, range=[0x0000000069348000-0x000000006934e000) (0MB) [ 0.000000] EFI: mem109: type=7, attr=0xf, range=[0x000000006934e000-0x0000000069352000) (0MB) [ 0.000000] EFI: mem110: type=2, attr=0xf, range=[0x0000000069352000-0x0000000069357000) (0MB) [ 0.000000] EFI: mem111: type=7, attr=0xf, range=[0x0000000069357000-0x000000006935b000) (0MB) [ 0.000000] EFI: mem112: type=2, attr=0xf, range=[0x000000006935b000-0x000000006936d000) (0MB) [ 0.000000] EFI: mem113: type=4, attr=0xf, range=[0x000000006936d000-0x00000000693b9000) (0MB) [ 0.000000] EFI: mem114: type=7, attr=0xf, range=[0x00000000693b9000-0x00000000693bb000) (0MB) [ 0.000000] EFI: mem115: type=2, attr=0xf, range=[0x00000000693bb000-0x00000000693c1000) (0MB) [ 0.000000] EFI: mem116: type=7, attr=0xf, range=[0x00000000693c1000-0x00000000693c2000) (0MB) [ 0.000000] EFI: mem117: type=4, attr=0xf, range=[0x00000000693c2000-0x00000000693db000) (0MB) [ 0.000000] EFI: mem118: type=2, attr=0xf, range=[0x00000000693db000-0x00000000693dc000) (0MB) [ 0.000000] EFI: mem119: type=4, attr=0xf, range=[0x00000000693dc000-0x00000000693f0000) (0MB) [ 0.000000] EFI: mem120: type=2, attr=0xf, range=[0x00000000693f0000-0x0000000069405000) (0MB) [ 0.000000] EFI: mem121: type=4, attr=0xf, range=[0x0000000069405000-0x0000000069406000) (0MB) [ 0.000000] EFI: mem122: type=2, attr=0xf, range=[0x0000000069406000-0x0000000069409000) (0MB) [ 0.000000] EFI: mem123: type=4, attr=0xf, range=[0x0000000069409000-0x000000006943a000) (0MB) [ 0.000000] EFI: mem124: type=3, attr=0xf, range=[0x000000006943a000-0x0000000069474000) (0MB) [ 0.000000] EFI: mem125: type=4, attr=0xf, range=[0x0000000069474000-0x0000000069475000) (0MB) [ 0.000000] EFI: mem126: type=3, attr=0xf, range=[0x0000000069475000-0x000000006947e000) (0MB) [ 0.000000] EFI: mem127: type=4, attr=0xf, range=[0x000000006947e000-0x0000000069480000) (0MB) [ 0.000000] EFI: mem128: type=3, attr=0xf, range=[0x0000000069480000-0x0000000069495000) (0MB) [ 0.000000] EFI: mem129: type=4, attr=0xf, range=[0x0000000069495000-0x0000000069499000) (0MB) [ 0.000000] EFI: mem130: type=3, attr=0xf, range=[0x0000000069499000-0x00000000694aa000) (0MB) [ 0.000000] EFI: mem131: type=4, attr=0xf, range=[0x00000000694aa000-0x00000000694b7000) (0MB) [ 0.000000] EFI: mem132: type=3, attr=0xf, range=[0x00000000694b7000-0x00000000694ba000) (0MB) [ 0.000000] EFI: mem133: type=4, attr=0xf, range=[0x00000000694ba000-0x00000000694bd000) (0MB) [ 0.000000] EFI: mem134: type=3, attr=0xf, range=[0x00000000694bd000-0x00000000694de000) (0MB) [ 0.000000] EFI: mem135: type=4, attr=0xf, range=[0x00000000694de000-0x00000000698de000) (4MB) [ 0.000000] EFI: mem136: type=3, attr=0xf, range=[0x00000000698de000-0x00000000698e4000) (0MB) [ 0.000000] EFI: mem137: type=4, attr=0xf, range=[0x00000000698e4000-0x00000000698e7000) (0MB) [ 0.000000] EFI: mem138: type=3, attr=0xf, range=[0x00000000698e7000-0x00000000698e9000) (0MB) [ 0.000000] EFI: mem139: type=4, attr=0xf, range=[0x00000000698e9000-0x00000000698eb000) (0MB) [ 0.000000] EFI: mem140: type=3, attr=0xf, range=[0x00000000698eb000-0x00000000698ed000) (0MB) [ 0.000000] EFI: mem141: type=4, attr=0xf, range=[0x00000000698ed000-0x00000000698ee000) (0MB) [ 0.000000] EFI: mem142: type=3, attr=0xf, range=[0x00000000698ee000-0x00000000698f0000) (0MB) [ 0.000000] EFI: mem143: type=4, attr=0xf, range=[0x00000000698f0000-0x00000000698f1000) (0MB) [ 0.000000] EFI: mem144: type=3, attr=0xf, range=[0x00000000698f1000-0x0000000069907000) (0MB) [ 0.000000] EFI: mem145: type=4, attr=0xf, range=[0x0000000069907000-0x000000006994e000) (0MB) [ 0.000000] EFI: mem146: type=3, attr=0xf, range=[0x000000006994e000-0x000000006998b000) (0MB) [ 0.000000] EFI: mem147: type=4, attr=0xf, range=[0x000000006998b000-0x00000000699d8000) (0MB) [ 0.000000] EFI: mem148: type=3, attr=0xf, range=[0x00000000699d8000-0x00000000699da000) (0MB) [ 0.000000] EFI: mem149: type=4, attr=0xf, range=[0x00000000699da000-0x00000000699db000) (0MB) [ 0.000000] EFI: mem150: type=3, attr=0xf, range=[0x00000000699db000-0x00000000699ed000) (0MB) [ 0.000000] EFI: mem151: type=4, attr=0xf, range=[0x00000000699ed000-0x0000000069a25000) (0MB) [ 0.000000] EFI: mem152: type=3, attr=0xf, range=[0x0000000069a25000-0x0000000069a29000) (0MB) [ 0.000000] EFI: mem153: type=4, attr=0xf, range=[0x0000000069a29000-0x000000007abb1000) (273MB) [ 0.000000] EFI: mem154: type=6, attr=0x800000000000000f, range=[0x000000007abb1000-0x000000007abb2000) (0MB) [ 0.000000] EFI: mem155: type=4, attr=0xf, range=[0x000000007abb2000-0x000000007abc5000) (0MB) [ 0.000000] EFI: mem156: type=6, attr=0x800000000000000f, range=[0x000000007abc5000-0x000000007ae05000) (2MB) [ 0.000000] EFI: mem157: type=4, attr=0xf, range=[0x000000007ae05000-0x000000007bb13000) (13MB) [ 0.000000] EFI: mem158: type=6, attr=0x800000000000000f, range=[0x000000007bb13000-0x000000007bc13000) (1MB) [ 0.000000] EFI: mem159: type=5, attr=0x800000000000000f, range=[0x000000007bc13000-0x000000007bd13000) (1MB) [ 0.000000] EFI: mem160: type=0, attr=0xf, range=[0x000000007bd13000-0x000000007be13000) (1MB) [ 0.000000] EFI: mem161: type=10, attr=0xf, range=[0x000000007be13000-0x000000007d1f3000) (19MB) [ 0.000000] EFI: mem162: type=9, attr=0xf, range=[0x000000007d1f3000-0x000000007e000000) (14MB) [ 0.000000] EFI: mem163: type=2, attr=0xf, range=[0x000000007e000000-0x000000007e2ee000) (2MB) [ 0.000000] EFI: mem164: type=9, attr=0xf, range=[0x000000007e2ee000-0x000000007ef3f000) (12MB) [ 0.000000] EFI: mem165: type=4, attr=0xf, range=[0x000000007ef3f000-0x000000007f000000) (0MB) [ 0.000000] EFI: mem166: type=7, attr=0xf, range=[0x0000000100000000-0x0000001f80000000) (124928MB) [ 0.000000] EFI: mem167: type=7, attr=0xf, range=[0x0000004000000000-0x0000005eff000000) (126960MB) [ 0.000000] EFI: mem168: type=7, attr=0xf, range=[0x0000008000000000-0x0000009eff000000) (126960MB) [ 0.000000] EFI: mem169: type=7, attr=0xf, range=[0x000000c000000000-0x000000deff000000) (126960MB) [ 0.000000] EFI: mem170: type=7, attr=0xf, range=[0x0000010000000000-0x0000011eff000000) (126960MB) [ 0.000000] EFI: mem171: type=7, attr=0xf, range=[0x0000014000000000-0x0000015eff000000) (126960MB) [ 0.000000] EFI: mem172: type=7, attr=0xf, range=[0x0000018000000000-0x0000019eff000000) (126960MB) [ 0.000000] EFI: mem173: type=7, attr=0xf, range=[0x000001c000000000-0x000001deff000000) (126960MB) [ 0.000000] EFI: mem174: type=7, attr=0xf, range=[0x0000020000000000-0x0000021eff000000) (126960MB) [ 0.000000] EFI: mem175: type=7, attr=0xf, range=[0x0000024000000000-0x0000025eff000000) (126960MB) [ 0.000000] EFI: mem176: type=7, attr=0xf, range=[0x0000028000000000-0x0000029eff000000) (126960MB) [ 0.000000] EFI: mem177: type=7, attr=0xf, range=[0x000002c000000000-0x000002deff000000) (126960MB) [ 0.000000] EFI: mem178: type=7, attr=0xf, range=[0x0000030000000000-0x0000031eff000000) (126960MB) [ 0.000000] EFI: mem179: type=7, attr=0xf, range=[0x0000034000000000-0x0000035eff000000) (126960MB) [ 0.000000] EFI: mem180: type=7, attr=0xf, range=[0x0000038000000000-0x0000039eff000000) (126960MB) [ 0.000000] EFI: mem181: type=7, attr=0xf, range=[0x000003c000000000-0x000003deff000000) (126960MB) [ 0.000000] EFI: mem182: type=7, attr=0xf, range=[0x0000040000000000-0x0000041eff000000) (126960MB) [ 0.000000] EFI: mem183: type=7, attr=0xf, range=[0x0000044000000000-0x0000045eff000000) (126960MB) [ 0.000000] EFI: mem184: type=7, attr=0xf, range=[0x0000048000000000-0x0000049eff000000) (126960MB) [ 0.000000] EFI: mem185: type=7, attr=0xf, range=[0x000004c000000000-0x000004deff000000) (126960MB) [ 0.000000] EFI: mem186: type=7, attr=0xf, range=[0x0000050000000000-0x0000051eff000000) (126960MB) [ 0.000000] EFI: mem187: type=7, attr=0xf, range=[0x0000054000000000-0x0000055eff000000) (126960MB) [ 0.000000] EFI: mem188: type=7, attr=0xf, range=[0x0000058000000000-0x0000059eff000000) (126960MB) [ 0.000000] EFI: mem189: type=7, attr=0xf, range=[0x000005c000000000-0x000005deff000000) (126960MB) [ 0.000000] EFI: mem190: type=7, attr=0xf, range=[0x0000060000000000-0x0000061eff000000) (126960MB) [ 0.000000] EFI: mem191: type=7, attr=0xf, range=[0x0000064000000000-0x0000065eff000000) (126960MB) [ 0.000000] EFI: mem192: type=7, attr=0xf, range=[0x0000068000000000-0x0000069eff000000) (126960MB) [ 0.000000] EFI: mem193: type=7, attr=0xf, range=[0x000006c000000000-0x000006deff000000) (126960MB) [ 0.000000] EFI: mem194: type=7, attr=0xf, range=[0x0000070000000000-0x0000071eff000000) (126960MB) [ 0.000000] EFI: mem195: type=7, attr=0xf, range=[0x0000074000000000-0x0000075eff000000) (126960MB) [ 0.000000] EFI: mem196: type=7, attr=0xf, range=[0x0000078000000000-0x0000079eff000000) (126960MB) [ 0.000000] EFI: mem197: type=7, attr=0xf, range=[0x000007c000000000-0x000007deff000000) (126960MB) [ 0.000000] EFI: mem198: type=7, attr=0xf, range=[0x0000080000000000-0x0000081eff000000) (126960MB) [ 0.000000] EFI: mem199: type=7, attr=0xf, range=[0x0000084000000000-0x0000085eff000000) (126960MB) [ 0.000000] EFI: mem200: type=7, attr=0xf, range=[0x0000088000000000-0x0000089eff000000) (126960MB) [ 0.000000] EFI: mem201: type=7, attr=0xf, range=[0x000008c000000000-0x000008deff000000) (126960MB) [ 0.000000] EFI: mem202: type=7, attr=0xf, range=[0x0000090000000000-0x0000091eff000000) (126960MB) [ 0.000000] EFI: mem203: type=7, attr=0xf, range=[0x0000094000000000-0x0000095eff000000) (126960MB) [ 0.000000] EFI: mem204: type=7, attr=0xf, range=[0x0000098000000000-0x0000099eff000000) (126960MB) [ 0.000000] EFI: mem205: type=7, attr=0xf, range=[0x000009c000000000-0x000009deff000000) (126960MB) [ 0.000000] EFI: mem206: type=7, attr=0xf, range=[0x00000a0000000000-0x00000a1eff000000) (126960MB) [ 0.000000] EFI: mem207: type=7, attr=0xf, range=[0x00000a4000000000-0x00000a5eff000000) (126960MB) [ 0.000000] EFI: mem208: type=7, attr=0xf, range=[0x00000a8000000000-0x00000a9eff000000) (126960MB) [ 0.000000] EFI: mem209: type=7, attr=0xf, range=[0x00000ac000000000-0x00000adeff000000) (126960MB) [ 0.000000] EFI: mem210: type=7, attr=0xf, range=[0x00000b0000000000-0x00000b1eff000000) (126960MB) [ 0.000000] EFI: mem211: type=7, attr=0xf, range=[0x00000b4000000000-0x00000b5eff000000) (126960MB) [ 0.000000] EFI: mem212: type=7, attr=0xf, range=[0x00000b8000000000-0x00000b9eff000000) (126960MB) [ 0.000000] EFI: mem213: type=7, attr=0xf, range=[0x00000bc000000000-0x00000bdeff000000) (126960MB) [ 0.000000] EFI: mem214: type=7, attr=0xf, range=[0x00000c0000000000-0x00000c1eff000000) (126960MB) [ 0.000000] EFI: mem215: type=7, attr=0xf, range=[0x00000c4000000000-0x00000c5eff000000) (126960MB) [ 0.000000] EFI: mem216: type=7, attr=0xf, range=[0x00000c8000000000-0x00000c9eff000000) (126960MB) [ 0.000000] EFI: mem217: type=7, attr=0xf, range=[0x00000cc000000000-0x00000cdeff000000) (126960MB) [ 0.000000] EFI: mem218: type=7, attr=0xf, range=[0x00000d0000000000-0x00000d1eff000000) (126960MB) [ 0.000000] EFI: mem219: type=7, attr=0xf, range=[0x00000d4000000000-0x00000d5eff000000) (126960MB) [ 0.000000] EFI: mem220: type=7, attr=0xf, range=[0x00000d8000000000-0x00000d9eff000000) (126960MB) [ 0.000000] EFI: mem221: type=7, attr=0xf, range=[0x00000dc000000000-0x00000ddeff000000) (126960MB) [ 0.000000] EFI: mem222: type=7, attr=0xf, range=[0x00000e0000000000-0x00000e1eff000000) (126960MB) [ 0.000000] EFI: mem223: type=7, attr=0xf, range=[0x00000e4000000000-0x00000e5eff000000) (126960MB) [ 0.000000] EFI: mem224: type=7, attr=0xf, range=[0x00000e8000000000-0x00000e9eff000000) (126960MB) [ 0.000000] EFI: mem225: type=7, attr=0xf, range=[0x00000ec000000000-0x00000edeff000000) (126960MB) [ 0.000000] EFI: mem226: type=7, attr=0xf, range=[0x00000f0000000000-0x00000f1eff000000) (126960MB) [ 0.000000] EFI: mem227: type=7, attr=0xf, range=[0x00000f4000000000-0x00000f5eff000000) (126960MB) [ 0.000000] EFI: mem228: type=7, attr=0xf, range=[0x00000f8000000000-0x00000f9eff000000) (126960MB) [ 0.000000] EFI: mem229: type=7, attr=0xf, range=[0x00000fc000000000-0x00000fdeff000000) (126960MB) [ 0.000000] EFI: mem230: type=7, attr=0xf, range=[0x0000100000000000-0x0000101eff000000) (126960MB) [ 0.000000] EFI: mem231: type=7, attr=0xf, range=[0x0000104000000000-0x0000105eff000000) (126960MB) [ 0.000000] EFI: mem232: type=7, attr=0xf, range=[0x0000108000000000-0x0000109eff000000) (126960MB) [ 0.000000] EFI: mem233: type=7, attr=0xf, range=[0x000010c000000000-0x000010deff000000) (126960MB) [ 0.000000] EFI: mem234: type=7, attr=0xf, range=[0x0000110000000000-0x0000111eff000000) (126960MB) [ 0.000000] EFI: mem235: type=7, attr=0xf, range=[0x0000114000000000-0x0000115eff000000) (126960MB) [ 0.000000] EFI: mem236: type=7, attr=0xf, range=[0x0000118000000000-0x0000119eff000000) (126960MB) [ 0.000000] EFI: mem237: type=7, attr=0xf, range=[0x000011c000000000-0x000011deff000000) (126960MB) [ 0.000000] EFI: mem238: type=7, attr=0xf, range=[0x0000120000000000-0x0000121eff000000) (126960MB) [ 0.000000] EFI: mem239: type=7, attr=0xf, range=[0x0000124000000000-0x0000125eff000000) (126960MB) [ 0.000000] EFI: mem240: type=7, attr=0xf, range=[0x0000128000000000-0x0000129eff000000) (126960MB) [ 0.000000] EFI: mem241: type=7, attr=0xf, range=[0x000012c000000000-0x000012deff000000) (126960MB) [ 0.000000] EFI: mem242: type=7, attr=0xf, range=[0x0000130000000000-0x0000131eff000000) (126960MB) [ 0.000000] EFI: mem243: type=7, attr=0xf, range=[0x0000134000000000-0x0000135eff000000) (126960MB) [ 0.000000] EFI: mem244: type=7, attr=0xf, range=[0x0000138000000000-0x0000139eff000000) (126960MB) [ 0.000000] EFI: mem245: type=7, attr=0xf, range=[0x000013c000000000-0x000013deff000000) (126960MB) [ 0.000000] EFI: mem246: type=7, attr=0xf, range=[0x0000140000000000-0x0000141eff000000) (126960MB) [ 0.000000] EFI: mem247: type=7, attr=0xf, range=[0x0000144000000000-0x0000145eff000000) (126960MB) [ 0.000000] EFI: mem248: type=7, attr=0xf, range=[0x0000148000000000-0x0000149eff000000) (126960MB) [ 0.000000] EFI: mem249: type=7, attr=0xf, range=[0x000014c000000000-0x000014deff000000) (126960MB) [ 0.000000] EFI: mem250: type=7, attr=0xf, range=[0x0000150000000000-0x0000151eff000000) (126960MB) [ 0.000000] EFI: mem251: type=7, attr=0xf, range=[0x0000154000000000-0x0000155eff000000) (126960MB) [ 0.000000] EFI: mem252: type=7, attr=0xf, range=[0x0000158000000000-0x0000159eff000000) (126960MB) [ 0.000000] EFI: mem253: type=7, attr=0xf, range=[0x000015c000000000-0x000015deff000000) (126960MB) [ 0.000000] EFI: mem254: type=7, attr=0xf, range=[0x0000160000000000-0x0000161eff000000) (126960MB) [ 0.000000] EFI: mem255: type=7, attr=0xf, range=[0x0000164000000000-0x0000165eff000000) (126960MB) [ 0.000000] EFI: mem256: type=7, attr=0xf, range=[0x0000168000000000-0x0000169eff000000) (126960MB) [ 0.000000] EFI: mem257: type=7, attr=0xf, range=[0x000016c000000000-0x000016deff000000) (126960MB) [ 0.000000] EFI: mem258: type=7, attr=0xf, range=[0x0000170000000000-0x0000171eff000000) (126960MB) [ 0.000000] EFI: mem259: type=7, attr=0xf, range=[0x0000174000000000-0x0000175eff000000) (126960MB) [ 0.000000] EFI: mem260: type=7, attr=0xf, range=[0x0000178000000000-0x0000179eff000000) (126960MB) [ 0.000000] EFI: mem261: type=7, attr=0xf, range=[0x000017c000000000-0x000017deff000000) (126960MB) [ 0.000000] EFI: mem262: type=7, attr=0xf, range=[0x0000180000000000-0x0000181eff000000) (126960MB) [ 0.000000] EFI: mem263: type=7, attr=0xf, range=[0x0000184000000000-0x0000185eff000000) (126960MB) [ 0.000000] EFI: mem264: type=7, attr=0xf, range=[0x0000188000000000-0x0000189eff000000) (126960MB) [ 0.000000] EFI: mem265: type=7, attr=0xf, range=[0x000018c000000000-0x000018deff000000) (126960MB) [ 0.000000] EFI: mem266: type=7, attr=0xf, range=[0x0000190000000000-0x0000191eff000000) (126960MB) [ 0.000000] EFI: mem267: type=7, attr=0xf, range=[0x0000194000000000-0x0000195eff000000) (126960MB) [ 0.000000] EFI: mem268: type=7, attr=0xf, range=[0x0000198000000000-0x0000199eff000000) (126960MB) [ 0.000000] EFI: mem269: type=7, attr=0xf, range=[0x000019c000000000-0x000019deff000000) (126960MB) [ 0.000000] EFI: mem270: type=7, attr=0xf, range=[0x00001a0000000000-0x00001a1eff000000) (126960MB) [ 0.000000] EFI: mem271: type=7, attr=0xf, range=[0x00001a4000000000-0x00001a5eff000000) (126960MB) [ 0.000000] EFI: mem272: type=7, attr=0xf, range=[0x00001a8000000000-0x00001a9eff000000) (126960MB) [ 0.000000] EFI: mem273: type=7, attr=0xf, range=[0x00001ac000000000-0x00001adeff000000) (126960MB) [ 0.000000] EFI: mem274: type=7, attr=0xf, range=[0x00001b0000000000-0x00001b1eff000000) (126960MB) [ 0.000000] EFI: mem275: type=7, attr=0xf, range=[0x00001b4000000000-0x00001b5eff000000) (126960MB) [ 0.000000] EFI: mem276: type=7, attr=0xf, range=[0x00001b8000000000-0x00001b9eff000000) (126960MB) [ 0.000000] EFI: mem277: type=7, attr=0xf, range=[0x00001bc000000000-0x00001bdeff000000) (126960MB) [ 0.000000] EFI: mem278: type=7, attr=0xf, range=[0x00001c0000000000-0x00001c1eff000000) (126960MB) [ 0.000000] EFI: mem279: type=7, attr=0xf, range=[0x00001c4000000000-0x00001c5eff000000) (126960MB) [ 0.000000] EFI: mem280: type=7, attr=0xf, range=[0x00001c8000000000-0x00001c9eff000000) (126960MB) [ 0.000000] EFI: mem281: type=7, attr=0xf, range=[0x00001cc000000000-0x00001cdeff000000) (126960MB) [ 0.000000] EFI: mem282: type=7, attr=0xf, range=[0x00001d0000000000-0x00001d1eff000000) (126960MB) [ 0.000000] EFI: mem283: type=7, attr=0xf, range=[0x00001d4000000000-0x00001d5eff000000) (126960MB) [ 0.000000] EFI: mem284: type=7, attr=0xf, range=[0x00001d8000000000-0x00001d9eff000000) (126960MB) [ 0.000000] EFI: mem285: type=7, attr=0xf, range=[0x00001dc000000000-0x00001ddeff000000) (126960MB) [ 0.000000] EFI: mem286: type=7, attr=0xf, range=[0x00001e0000000000-0x00001e1eff000000) (126960MB) [ 0.000000] EFI: mem287: type=7, attr=0xf, range=[0x00001e4000000000-0x00001e5eff000000) (126960MB) [ 0.000000] EFI: mem288: type=7, attr=0xf, range=[0x00001e8000000000-0x00001e9eff000000) (126960MB) [ 0.000000] EFI: mem289: type=7, attr=0xf, range=[0x00001ec000000000-0x00001edeff000000) (126960MB) [ 0.000000] EFI: mem290: type=7, attr=0xf, range=[0x00001f0000000000-0x00001f1eff000000) (126960MB) [ 0.000000] EFI: mem291: type=7, attr=0xf, range=[0x00001f4000000000-0x00001f5eff000000) (126960MB) [ 0.000000] EFI: mem292: type=7, attr=0xf, range=[0x00001f8000000000-0x00001f9eff000000) (126960MB) [ 0.000000] EFI: mem293: type=7, attr=0xf, range=[0x00001fc000000000-0x00001fdeff000000) (126960MB) [ 0.000000] EFI: mem294: type=7, attr=0xf, range=[0x0000200000000000-0x0000201eff000000) (126960MB) [ 0.000000] EFI: mem295: type=7, attr=0xf, range=[0x0000204000000000-0x0000205eff000000) (126960MB) [ 0.000000] EFI: mem296: type=7, attr=0xf, range=[0x0000208000000000-0x0000209eff000000) (126960MB) [ 0.000000] EFI: mem297: type=7, attr=0xf, range=[0x000020c000000000-0x000020deff000000) (126960MB) [ 0.000000] EFI: mem298: type=7, attr=0xf, range=[0x0000210000000000-0x0000211eff000000) (126960MB) [ 0.000000] EFI: mem299: type=7, attr=0xf, range=[0x0000214000000000-0x0000215eff000000) (126960MB) [ 0.000000] EFI: mem300: type=7, attr=0xf, range=[0x0000218000000000-0x0000219eff000000) (126960MB) [ 0.000000] EFI: mem301: type=7, attr=0xf, range=[0x000021c000000000-0x000021deff000000) (126960MB) [ 0.000000] EFI: mem302: type=7, attr=0xf, range=[0x0000220000000000-0x0000221eff000000) (126960MB) [ 0.000000] EFI: mem303: type=7, attr=0xf, range=[0x0000224000000000-0x0000225eff000000) (126960MB) [ 0.000000] EFI: mem304: type=7, attr=0xf, range=[0x0000228000000000-0x0000229eff000000) (126960MB) [ 0.000000] EFI: mem305: type=7, attr=0xf, range=[0x000022c000000000-0x000022deff000000) (126960MB) [ 0.000000] EFI: mem306: type=7, attr=0xf, range=[0x0000230000000000-0x0000231eff000000) (126960MB) [ 0.000000] EFI: mem307: type=7, attr=0xf, range=[0x0000234000000000-0x0000235eff000000) (126960MB) [ 0.000000] EFI: mem308: type=7, attr=0xf, range=[0x0000238000000000-0x0000239eff000000) (126960MB) [ 0.000000] EFI: mem309: type=7, attr=0xf, range=[0x000023c000000000-0x000023deff000000) (126960MB) [ 0.000000] EFI: mem310: type=7, attr=0xf, range=[0x0000240000000000-0x0000241eff000000) (126960MB) [ 0.000000] EFI: mem311: type=7, attr=0xf, range=[0x0000244000000000-0x0000245eff000000) (126960MB) [ 0.000000] EFI: mem312: type=7, attr=0xf, range=[0x0000248000000000-0x0000249eff000000) (126960MB) [ 0.000000] EFI: mem313: type=7, attr=0xf, range=[0x000024c000000000-0x000024deff000000) (126960MB) [ 0.000000] EFI: mem314: type=7, attr=0xf, range=[0x0000250000000000-0x0000251eff000000) (126960MB) [ 0.000000] EFI: mem315: type=7, attr=0xf, range=[0x0000254000000000-0x0000255eff000000) (126960MB) [ 0.000000] EFI: mem316: type=7, attr=0xf, range=[0x0000258000000000-0x0000259eff000000) (126960MB) [ 0.000000] EFI: mem317: type=7, attr=0xf, range=[0x000025c000000000-0x000025deff000000) (126960MB) [ 0.000000] EFI: mem318: type=7, attr=0xf, range=[0x0000260000000000-0x0000261eff000000) (126960MB) [ 0.000000] EFI: mem319: type=7, attr=0xf, range=[0x0000264000000000-0x0000265eff000000) (126960MB) [ 0.000000] EFI: mem320: type=7, attr=0xf, range=[0x0000268000000000-0x0000269eff000000) (126960MB) [ 0.000000] EFI: mem321: type=7, attr=0xf, range=[0x000026c000000000-0x000026deff000000) (126960MB) [ 0.000000] EFI: mem322: type=7, attr=0xf, range=[0x0000270000000000-0x0000271eff000000) (126960MB) [ 0.000000] EFI: mem323: type=7, attr=0xf, range=[0x0000274000000000-0x0000275eff000000) (126960MB) [ 0.000000] EFI: mem324: type=7, attr=0xf, range=[0x0000278000000000-0x0000279eff000000) (126960MB) [ 0.000000] EFI: mem325: type=7, attr=0xf, range=[0x000027c000000000-0x000027deff000000) (126960MB) [ 0.000000] EFI: mem326: type=7, attr=0xf, range=[0x0000280000000000-0x0000281eff000000) (126960MB) [ 0.000000] EFI: mem327: type=7, attr=0xf, range=[0x0000284000000000-0x0000285eff000000) (126960MB) [ 0.000000] EFI: mem328: type=7, attr=0xf, range=[0x0000288000000000-0x0000289eff000000) (126960MB) [ 0.000000] EFI: mem329: type=7, attr=0xf, range=[0x000028c000000000-0x000028deff000000) (126960MB) [ 0.000000] EFI: mem330: type=7, attr=0xf, range=[0x0000290000000000-0x0000291eff000000) (126960MB) [ 0.000000] EFI: mem331: type=7, attr=0xf, range=[0x0000294000000000-0x0000295eff000000) (126960MB) [ 0.000000] EFI: mem332: type=7, attr=0xf, range=[0x0000298000000000-0x0000299eff000000) (126960MB) [ 0.000000] EFI: mem333: type=7, attr=0xf, range=[0x000029c000000000-0x000029deff000000) (126960MB) [ 0.000000] EFI: mem334: type=7, attr=0xf, range=[0x00002a0000000000-0x00002a1eff000000) (126960MB) [ 0.000000] EFI: mem335: type=7, attr=0xf, range=[0x00002a4000000000-0x00002a5eff000000) (126960MB) [ 0.000000] EFI: mem336: type=7, attr=0xf, range=[0x00002a8000000000-0x00002a9eff000000) (126960MB) [ 0.000000] EFI: mem337: type=7, attr=0xf, range=[0x00002ac000000000-0x00002adeff000000) (126960MB) [ 0.000000] EFI: mem338: type=7, attr=0xf, range=[0x00002b0000000000-0x00002b1eff000000) (126960MB) [ 0.000000] EFI: mem339: type=7, attr=0xf, range=[0x00002b4000000000-0x00002b5eff000000) (126960MB) [ 0.000000] EFI: mem340: type=7, attr=0xf, range=[0x00002b8000000000-0x00002b9eff000000) (126960MB) [ 0.000000] EFI: mem341: type=7, attr=0xf, range=[0x00002bc000000000-0x00002bdeff000000) (126960MB) [ 0.000000] EFI: mem342: type=7, attr=0xf, range=[0x00002c0000000000-0x00002c1eff000000) (126960MB) [ 0.000000] EFI: mem343: type=7, attr=0xf, range=[0x00002c4000000000-0x00002c5eff000000) (126960MB) [ 0.000000] EFI: mem344: type=7, attr=0xf, range=[0x00002c8000000000-0x00002c9eff000000) (126960MB) [ 0.000000] EFI: mem345: type=7, attr=0xf, range=[0x00002cc000000000-0x00002cdeff000000) (126960MB) [ 0.000000] EFI: mem346: type=7, attr=0xf, range=[0x00002d0000000000-0x00002d1eff000000) (126960MB) [ 0.000000] EFI: mem347: type=7, attr=0xf, range=[0x00002d4000000000-0x00002d5eff000000) (126960MB) [ 0.000000] EFI: mem348: type=7, attr=0xf, range=[0x00002d8000000000-0x00002d9eff000000) (126960MB) [ 0.000000] EFI: mem349: type=7, attr=0xf, range=[0x00002dc000000000-0x00002ddeff000000) (126960MB) [ 0.000000] EFI: mem350: type=7, attr=0xf, range=[0x00002e0000000000-0x00002e1eff000000) (126960MB) [ 0.000000] EFI: mem351: type=7, attr=0xf, range=[0x00002e4000000000-0x00002e5eff000000) (126960MB) [ 0.000000] EFI: mem352: type=7, attr=0xf, range=[0x00002e8000000000-0x00002e9eff000000) (126960MB) [ 0.000000] EFI: mem353: type=7, attr=0xf, range=[0x00002ec000000000-0x00002edeff000000) (126960MB) [ 0.000000] EFI: mem354: type=7, attr=0xf, range=[0x00002f0000000000-0x00002f1eff000000) (126960MB) [ 0.000000] EFI: mem355: type=7, attr=0xf, range=[0x00002f4000000000-0x00002f5eff000000) (126960MB) [ 0.000000] EFI: mem356: type=7, attr=0xf, range=[0x00002f8000000000-0x00002f9eff000000) (126960MB) [ 0.000000] EFI: mem357: type=7, attr=0xf, range=[0x00002fc000000000-0x00002fdeff000000) (126960MB) [ 0.000000] EFI: mem358: type=7, attr=0xf, range=[0x0000300000000000-0x0000301eff000000) (126960MB) [ 0.000000] EFI: mem359: type=7, attr=0xf, range=[0x0000304000000000-0x0000305eff000000) (126960MB) [ 0.000000] EFI: mem360: type=7, attr=0xf, range=[0x0000308000000000-0x0000309eff000000) (126960MB) [ 0.000000] EFI: mem361: type=7, attr=0xf, range=[0x000030c000000000-0x000030deff000000) (126960MB) [ 0.000000] EFI: mem362: type=7, attr=0xf, range=[0x0000310000000000-0x0000311eff000000) (126960MB) [ 0.000000] EFI: mem363: type=7, attr=0xf, range=[0x0000314000000000-0x0000315eff000000) (126960MB) [ 0.000000] EFI: mem364: type=7, attr=0xf, range=[0x0000318000000000-0x0000319eff000000) (126960MB) [ 0.000000] EFI: mem365: type=7, attr=0xf, range=[0x000031c000000000-0x000031deff000000) (126960MB) [ 0.000000] EFI: mem366: type=7, attr=0xf, range=[0x0000320000000000-0x0000321eff000000) (126960MB) [ 0.000000] EFI: mem367: type=7, attr=0xf, range=[0x0000324000000000-0x0000325eff000000) (126960MB) [ 0.000000] EFI: mem368: type=7, attr=0xf, range=[0x0000328000000000-0x0000329eff000000) (126960MB) [ 0.000000] EFI: mem369: type=7, attr=0xf, range=[0x000032c000000000-0x000032deff000000) (126960MB) [ 0.000000] EFI: mem370: type=7, attr=0xf, range=[0x0000330000000000-0x0000331eff000000) (126960MB) [ 0.000000] EFI: mem371: type=7, attr=0xf, range=[0x0000334000000000-0x0000335eff000000) (126960MB) [ 0.000000] EFI: mem372: type=7, attr=0xf, range=[0x0000338000000000-0x0000339eff000000) (126960MB) [ 0.000000] EFI: mem373: type=7, attr=0xf, range=[0x000033c000000000-0x000033deff000000) (126960MB) [ 0.000000] EFI: mem374: type=7, attr=0xf, range=[0x0000340000000000-0x0000341eff000000) (126960MB) [ 0.000000] EFI: mem375: type=7, attr=0xf, range=[0x0000344000000000-0x0000345eff000000) (126960MB) [ 0.000000] EFI: mem376: type=7, attr=0xf, range=[0x0000348000000000-0x0000349eff000000) (126960MB) [ 0.000000] EFI: mem377: type=7, attr=0xf, range=[0x000034c000000000-0x000034deff000000) (126960MB) [ 0.000000] EFI: mem378: type=7, attr=0xf, range=[0x0000350000000000-0x0000351eff000000) (126960MB) [ 0.000000] EFI: mem379: type=7, attr=0xf, range=[0x0000354000000000-0x0000355eff000000) (126960MB) [ 0.000000] EFI: mem380: type=7, attr=0xf, range=[0x0000358000000000-0x0000359eff000000) (126960MB) [ 0.000000] EFI: mem381: type=7, attr=0xf, range=[0x000035c000000000-0x000035deff000000) (126960MB) [ 0.000000] EFI: mem382: type=7, attr=0xf, range=[0x0000360000000000-0x0000361eff000000) (126960MB) [ 0.000000] EFI: mem383: type=7, attr=0xf, range=[0x0000364000000000-0x0000365eff000000) (126960MB) [ 0.000000] EFI: mem384: type=7, attr=0xf, range=[0x0000368000000000-0x0000369eff000000) (126960MB) [ 0.000000] EFI: mem385: type=7, attr=0xf, range=[0x0000370000000000-0x0000371eff000000) (126960MB) [ 0.000000] EFI: mem386: type=7, attr=0xf, range=[0x0000378000000000-0x0000379eff000000) (126960MB) [ 0.000000] EFI: mem387: type=7, attr=0xf, range=[0x000037c000000000-0x000037deff000000) (126960MB) [ 0.000000] EFI: mem388: type=7, attr=0xf, range=[0x0000380000000000-0x0000381eff000000) (126960MB) [ 0.000000] EFI: mem389: type=7, attr=0xf, range=[0x0000384000000000-0x0000385eff000000) (126960MB) [ 0.000000] EFI: mem390: type=7, attr=0xf, range=[0x0000388000000000-0x0000389eff000000) (126960MB) [ 0.000000] EFI: mem391: type=7, attr=0xf, range=[0x000038c000000000-0x000038deff000000) (126960MB) [ 0.000000] EFI: mem392: type=7, attr=0xf, range=[0x0000390000000000-0x0000391eff000000) (126960MB) [ 0.000000] EFI: mem393: type=7, attr=0xf, range=[0x0000394000000000-0x0000395eff000000) (126960MB) [ 0.000000] EFI: mem394: type=7, attr=0xf, range=[0x0000398000000000-0x0000399eff000000) (126960MB) [ 0.000000] EFI: mem395: type=7, attr=0xf, range=[0x000039c000000000-0x000039deff000000) (126960MB) [ 0.000000] EFI: mem396: type=7, attr=0xf, range=[0x00003a0000000000-0x00003a1eff000000) (126960MB) [ 0.000000] EFI: mem397: type=7, attr=0xf, range=[0x00003a4000000000-0x00003a5eff000000) (126960MB) [ 0.000000] EFI: mem398: type=7, attr=0xf, range=[0x00003a8000000000-0x00003a9eff000000) (126960MB) [ 0.000000] EFI: mem399: type=7, attr=0xf, range=[0x00003ac000000000-0x00003adeff000000) (126960MB) [ 0.000000] EFI: mem400: type=7, attr=0xf, range=[0x00003b0000000000-0x00003b1eff000000) (126960MB) [ 0.000000] EFI: mem401: type=7, attr=0xf, range=[0x00003b4000000000-0x00003b5eff000000) (126960MB) [ 0.000000] EFI: mem402: type=7, attr=0xf, range=[0x00003b8000000000-0x00003b9eff000000) (126960MB) [ 0.000000] EFI: mem403: type=7, attr=0xf, range=[0x00003bc000000000-0x00003bdeff000000) (126960MB) [ 0.000000] EFI: mem404: type=7, attr=0xf, range=[0x00003c0000000000-0x00003c1eff000000) (126960MB) [ 0.000000] EFI: mem405: type=7, attr=0xf, range=[0x00003c4000000000-0x00003c5eff000000) (126960MB) [ 0.000000] EFI: mem406: type=7, attr=0xf, range=[0x00003c8000000000-0x00003c9eff000000) (126960MB) [ 0.000000] EFI: mem407: type=7, attr=0xf, range=[0x00003cc000000000-0x00003cdeff000000) (126960MB) [ 0.000000] EFI: mem408: type=7, attr=0xf, range=[0x00003d0000000000-0x00003d1eff000000) (126960MB) [ 0.000000] EFI: mem409: type=7, attr=0xf, range=[0x00003d4000000000-0x00003d5eff000000) (126960MB) [ 0.000000] EFI: mem410: type=7, attr=0xf, range=[0x00003d8000000000-0x00003d9eff000000) (126960MB) [ 0.000000] EFI: mem411: type=7, attr=0xf, range=[0x00003dc000000000-0x00003ddeff000000) (126960MB) [ 0.000000] EFI: mem412: type=7, attr=0xf, range=[0x00003e0000000000-0x00003e1eff000000) (126960MB) [ 0.000000] EFI: mem413: type=7, attr=0xf, range=[0x00003e4000000000-0x00003e5eff000000) (126960MB) [ 0.000000] EFI: mem414: type=7, attr=0xf, range=[0x00003e8000000000-0x00003e9eff000000) (126960MB) [ 0.000000] EFI: mem415: type=7, attr=0xf, range=[0x00003ec000000000-0x00003edeff000000) (126960MB) [ 0.000000] EFI: mem416: type=7, attr=0xf, range=[0x00003f0000000000-0x00003f1eff000000) (126960MB) [ 0.000000] EFI: mem417: type=7, attr=0xf, range=[0x00003f4000000000-0x00003f5eff000000) (126960MB) [ 0.000000] EFI: mem418: type=7, attr=0xf, range=[0x00003f8000000000-0x00003f9eff000000) (126960MB) [ 0.000000] EFI: mem419: type=7, attr=0xf, range=[0x00003fc000000000-0x00003fdeff000000) (126960MB) [ 0.000000] EFI: mem420: type=11, attr=0x8000000000000000, range=[0x0000000080000000-0x0000000090000000) (256MB) [ 0.000000] EFI: mem421: type=11, attr=0x8000000000000001, range=[0x00000000f8000000-0x00000000fe000000) (96MB) [ 0.000000] EFI: mem422: type=11, attr=0x8000000000000001, range=[0x00000000fed1c000-0x00000000fed20000) (0MB) [ 0.000000] EFI: mem423: type=11, attr=0x8000000000000001, range=[0x00003ff000000000-0x00003ff003e00000) (62MB) [ 0.000000] EFI: mem424: type=11, attr=0x8000000000000001, range=[0x00003ff003f00000-0x00003ff007e00000) (63MB) [ 0.000000] EFI: mem425: type=11, attr=0x8000000000000001, range=[0x00003ff007f00000-0x00003ff008000000) (1MB) [ 0.000000] EFI: mem426: type=11, attr=0x8000000000000001, range=[0x00003ff00bf00000-0x00003ff00c000000) (1MB) [ 0.000000] EFI: mem427: type=11, attr=0x8000000000000001, range=[0x00003ff00ff00000-0x00003ff010000000) (1MB) [ 0.000000] EFI: mem428: type=11, attr=0x8000000000000001, range=[0x00003ff013f00000-0x00003ff014000000) (1MB) [ 0.000000] EFI: mem429: type=11, attr=0x8000000000000001, range=[0x00003ff017f00000-0x00003ff018000000) (1MB) [ 0.000000] EFI: mem430: type=11, attr=0x8000000000000001, range=[0x00003ff01bf00000-0x00003ff01c000000) (1MB) [ 0.000000] EFI: mem431: type=11, attr=0x8000000000000001, range=[0x00003ff01ff00000-0x00003ff020000000) (1MB) [ 0.000000] EFI: mem432: type=11, attr=0x8000000000000001, range=[0x00003ff023f00000-0x00003ff024000000) (1MB) [ 0.000000] EFI: mem433: type=11, attr=0x8000000000000001, range=[0x00003ff027f00000-0x00003ff028000000) (1MB) [ 0.000000] EFI: mem434: type=11, attr=0x8000000000000001, range=[0x00003ff02bf00000-0x00003ff02c000000) (1MB) [ 0.000000] EFI: mem435: type=11, attr=0x8000000000000001, range=[0x00003ff02ff00000-0x00003ff030000000) (1MB) [ 0.000000] EFI: mem436: type=11, attr=0x8000000000000001, range=[0x00003ff033f00000-0x00003ff034000000) (1MB) [ 0.000000] EFI: mem437: type=11, attr=0x8000000000000001, range=[0x00003ff037f00000-0x00003ff038000000) (1MB) [ 0.000000] EFI: mem438: type=11, attr=0x8000000000000001, range=[0x00003ff03bf00000-0x00003ff03c000000) (1MB) [ 0.000000] EFI: mem439: type=11, attr=0x8000000000000001, range=[0x00003ff03ff00000-0x00003ff040000000) (1MB) [ 0.000000] EFI: mem440: type=11, attr=0x8000000000000001, range=[0x00003ff043f00000-0x00003ff044000000) (1MB) [ 0.000000] EFI: mem441: type=11, attr=0x8000000000000001, range=[0x00003ff047f00000-0x00003ff048000000) (1MB) [ 0.000000] EFI: mem442: type=11, attr=0x8000000000000001, range=[0x00003ff04bf00000-0x00003ff04c000000) (1MB) [ 0.000000] EFI: mem443: type=11, attr=0x8000000000000001, range=[0x00003ff04ff00000-0x00003ff050000000) (1MB) [ 0.000000] EFI: mem444: type=11, attr=0x8000000000000001, range=[0x00003ff053f00000-0x00003ff054000000) (1MB) [ 0.000000] EFI: mem445: type=11, attr=0x8000000000000001, range=[0x00003ff057f00000-0x00003ff058000000) (1MB) [ 0.000000] EFI: mem446: type=11, attr=0x8000000000000001, range=[0x00003ff05bf00000-0x00003ff05c000000) (1MB) [ 0.000000] EFI: mem447: type=11, attr=0x8000000000000001, range=[0x00003ff05ff00000-0x00003ff060000000) (1MB) [ 0.000000] EFI: mem448: type=11, attr=0x8000000000000001, range=[0x00003ff063f00000-0x00003ff064000000) (1MB) [ 0.000000] EFI: mem449: type=11, attr=0x8000000000000001, range=[0x00003ff067f00000-0x00003ff068000000) (1MB) [ 0.000000] EFI: mem450: type=11, attr=0x8000000000000001, range=[0x00003ff06bf00000-0x00003ff06c000000) (1MB) [ 0.000000] EFI: mem451: type=11, attr=0x8000000000000001, range=[0x00003ff06ff00000-0x00003ff070000000) (1MB) [ 0.000000] EFI: mem452: type=11, attr=0x8000000000000001, range=[0x00003ff073f00000-0x00003ff074000000) (1MB) [ 0.000000] EFI: mem453: type=11, attr=0x8000000000000001, range=[0x00003ff077f00000-0x00003ff078000000) (1MB) [ 0.000000] EFI: mem454: type=11, attr=0x8000000000000001, range=[0x00003ff07bf00000-0x00003ff07c000000) (1MB) [ 0.000000] EFI: mem455: type=11, attr=0x8000000000000001, range=[0x00003ff07ff00000-0x00003ff080000000) (1MB) [ 0.000000] EFI: mem456: type=11, attr=0x8000000000000001, range=[0x00003ff083f00000-0x00003ff084000000) (1MB) [ 0.000000] EFI: mem457: type=11, attr=0x8000000000000001, range=[0x00003ff087f00000-0x00003ff088000000) (1MB) [ 0.000000] EFI: mem458: type=11, attr=0x8000000000000001, range=[0x00003ff08bf00000-0x00003ff08c000000) (1MB) [ 0.000000] EFI: mem459: type=11, attr=0x8000000000000001, range=[0x00003ff08ff00000-0x00003ff090000000) (1MB) [ 0.000000] EFI: mem460: type=11, attr=0x8000000000000001, range=[0x00003ff093f00000-0x00003ff094000000) (1MB) [ 0.000000] EFI: mem461: type=11, attr=0x8000000000000001, range=[0x00003ff097f00000-0x00003ff098000000) (1MB) [ 0.000000] EFI: mem462: type=11, attr=0x8000000000000001, range=[0x00003ff09bf00000-0x00003ff09c000000) (1MB) [ 0.000000] EFI: mem463: type=11, attr=0x8000000000000001, range=[0x00003ff09ff00000-0x00003ff0a0000000) (1MB) [ 0.000000] EFI: mem464: type=11, attr=0x8000000000000001, range=[0x00003ff0a3f00000-0x00003ff0a4000000) (1MB) [ 0.000000] EFI: mem465: type=11, attr=0x8000000000000001, range=[0x00003ff0a7f00000-0x00003ff0a8000000) (1MB) [ 0.000000] EFI: mem466: type=11, attr=0x8000000000000001, range=[0x00003ff0abf00000-0x00003ff0ac000000) (1MB) [ 0.000000] EFI: mem467: type=11, attr=0x8000000000000001, range=[0x00003ff0aff00000-0x00003ff0b0000000) (1MB) [ 0.000000] EFI: mem468: type=11, attr=0x8000000000000001, range=[0x00003ff0b3f00000-0x00003ff0b4000000) (1MB) [ 0.000000] EFI: mem469: type=11, attr=0x8000000000000001, range=[0x00003ff0b7f00000-0x00003ff0b8000000) (1MB) [ 0.000000] EFI: mem470: type=11, attr=0x8000000000000001, range=[0x00003ff0bbf00000-0x00003ff0bc000000) (1MB) [ 0.000000] EFI: mem471: type=11, attr=0x8000000000000001, range=[0x00003ff0bff00000-0x00003ff0c0000000) (1MB) [ 0.000000] EFI: mem472: type=11, attr=0x8000000000000001, range=[0x00003ff0c3f00000-0x00003ff0c4000000) (1MB) [ 0.000000] EFI: mem473: type=11, attr=0x8000000000000001, range=[0x00003ff0c7f00000-0x00003ff0c8000000) (1MB) [ 0.000000] EFI: mem474: type=11, attr=0x8000000000000001, range=[0x00003ff0cbf00000-0x00003ff0cc000000) (1MB) [ 0.000000] EFI: mem475: type=11, attr=0x8000000000000001, range=[0x00003ff0cff00000-0x00003ff0d0000000) (1MB) [ 0.000000] EFI: mem476: type=11, attr=0x8000000000000001, range=[0x00003ff0d3f00000-0x00003ff0d4000000) (1MB) [ 0.000000] EFI: mem477: type=11, attr=0x8000000000000001, range=[0x00003ff0d7f00000-0x00003ff0d8000000) (1MB) [ 0.000000] EFI: mem478: type=11, attr=0x8000000000000001, range=[0x00003ff0dbf00000-0x00003ff0dc000000) (1MB) [ 0.000000] EFI: mem479: type=11, attr=0x8000000000000001, range=[0x00003ff0dff00000-0x00003ff0e0000000) (1MB) [ 0.000000] EFI: mem480: type=11, attr=0x8000000000000001, range=[0x00003ff0e3f00000-0x00003ff0e4000000) (1MB) [ 0.000000] EFI: mem481: type=11, attr=0x8000000000000001, range=[0x00003ff0e7f00000-0x00003ff0e8000000) (1MB) [ 0.000000] EFI: mem482: type=11, attr=0x8000000000000001, range=[0x00003ff0ebf00000-0x00003ff0ec000000) (1MB) [ 0.000000] EFI: mem483: type=11, attr=0x8000000000000001, range=[0x00003ff0eff00000-0x00003ff0f0000000) (1MB) [ 0.000000] EFI: mem484: type=11, attr=0x8000000000000001, range=[0x00003ff0f3f00000-0x00003ff0f4000000) (1MB) [ 0.000000] EFI: mem485: type=11, attr=0x8000000000000001, range=[0x00003ff0f7f00000-0x00003ff0f8000000) (1MB) [ 0.000000] EFI: mem486: type=11, attr=0x8000000000000001, range=[0x00003ff0fbf00000-0x00003ff0fc000000) (1MB) [ 0.000000] EFI: mem487: type=11, attr=0x8000000000000001, range=[0x00003ff0fff00000-0x00003ff100000000) (1MB) [ 0.000000] EFI: mem488: type=11, attr=0x8000000000000001, range=[0x00003ff103f00000-0x00003ff104000000) (1MB) [ 0.000000] EFI: mem489: type=11, attr=0x8000000000000001, range=[0x00003ff107f00000-0x00003ff108000000) (1MB) [ 0.000000] EFI: mem490: type=11, attr=0x8000000000000001, range=[0x00003ff10bf00000-0x00003ff10c000000) (1MB) [ 0.000000] EFI: mem491: type=11, attr=0x8000000000000001, range=[0x00003ff10ff00000-0x00003ff110000000) (1MB) [ 0.000000] EFI: mem492: type=11, attr=0x8000000000000001, range=[0x00003ff113f00000-0x00003ff114000000) (1MB) [ 0.000000] EFI: mem493: type=11, attr=0x8000000000000001, range=[0x00003ff117f00000-0x00003ff118000000) (1MB) [ 0.000000] EFI: mem494: type=11, attr=0x8000000000000001, range=[0x00003ff11bf00000-0x00003ff11c000000) (1MB) [ 0.000000] EFI: mem495: type=11, attr=0x8000000000000001, range=[0x00003ff11ff00000-0x00003ff120000000) (1MB) [ 0.000000] EFI: mem496: type=11, attr=0x8000000000000001, range=[0x00003ff123f00000-0x00003ff124000000) (1MB) [ 0.000000] EFI: mem497: type=11, attr=0x8000000000000001, range=[0x00003ff127f00000-0x00003ff128000000) (1MB) [ 0.000000] EFI: mem498: type=11, attr=0x8000000000000001, range=[0x00003ff12bf00000-0x00003ff12c000000) (1MB) [ 0.000000] EFI: mem499: type=11, attr=0x8000000000000001, range=[0x00003ff12ff00000-0x00003ff130000000) (1MB) [ 0.000000] EFI: mem500: type=11, attr=0x8000000000000001, range=[0x00003ff133f00000-0x00003ff134000000) (1MB) [ 0.000000] EFI: mem501: type=11, attr=0x8000000000000001, range=[0x00003ff137f00000-0x00003ff138000000) (1MB) [ 0.000000] EFI: mem502: type=11, attr=0x8000000000000001, range=[0x00003ff13bf00000-0x00003ff13c000000) (1MB) [ 0.000000] EFI: mem503: type=11, attr=0x8000000000000001, range=[0x00003ff13ff00000-0x00003ff140000000) (1MB) [ 0.000000] EFI: mem504: type=11, attr=0x8000000000000001, range=[0x00003ff143f00000-0x00003ff144000000) (1MB) [ 0.000000] EFI: mem505: type=11, attr=0x8000000000000001, range=[0x00003ff147f00000-0x00003ff148000000) (1MB) [ 0.000000] EFI: mem506: type=11, attr=0x8000000000000001, range=[0x00003ff14bf00000-0x00003ff14c000000) (1MB) [ 0.000000] EFI: mem507: type=11, attr=0x8000000000000001, range=[0x00003ff14ff00000-0x00003ff150000000) (1MB) [ 0.000000] EFI: mem508: type=11, attr=0x8000000000000001, range=[0x00003ff153f00000-0x00003ff154000000) (1MB) [ 0.000000] EFI: mem509: type=11, attr=0x8000000000000001, range=[0x00003ff157f00000-0x00003ff158000000) (1MB) [ 0.000000] EFI: mem510: type=11, attr=0x8000000000000001, range=[0x00003ff15bf00000-0x00003ff15c000000) (1MB) [ 0.000000] EFI: mem511: type=11, attr=0x8000000000000001, range=[0x00003ff15ff00000-0x00003ff160000000) (1MB) [ 0.000000] EFI: mem512: type=11, attr=0x8000000000000001, range=[0x00003ff163f00000-0x00003ff164000000) (1MB) [ 0.000000] EFI: mem513: type=11, attr=0x8000000000000001, range=[0x00003ff167f00000-0x00003ff168000000) (1MB) [ 0.000000] EFI: mem514: type=11, attr=0x8000000000000001, range=[0x00003ff16bf00000-0x00003ff16c000000) (1MB) [ 0.000000] EFI: mem515: type=11, attr=0x8000000000000001, range=[0x00003ff16ff00000-0x00003ff170000000) (1MB) [ 0.000000] EFI: mem516: type=11, attr=0x8000000000000001, range=[0x00003ff173f00000-0x00003ff174000000) (1MB) [ 0.000000] EFI: mem517: type=11, attr=0x8000000000000001, range=[0x00003ff177f00000-0x00003ff178000000) (1MB) [ 0.000000] EFI: mem518: type=11, attr=0x8000000000000001, range=[0x00003ff17bf00000-0x00003ff17c000000) (1MB) [ 0.000000] EFI: mem519: type=11, attr=0x8000000000000001, range=[0x00003ff17ff00000-0x00003ff180000000) (1MB) [ 0.000000] EFI: mem520: type=11, attr=0x8000000000000001, range=[0x00003ff183f00000-0x00003ff184000000) (1MB) [ 0.000000] EFI: mem521: type=11, attr=0x8000000000000001, range=[0x00003ff187f00000-0x00003ff188000000) (1MB) [ 0.000000] EFI: mem522: type=11, attr=0x8000000000000001, range=[0x00003ff18bf00000-0x00003ff18c000000) (1MB) [ 0.000000] EFI: mem523: type=11, attr=0x8000000000000001, range=[0x00003ff18ff00000-0x00003ff190000000) (1MB) [ 0.000000] EFI: mem524: type=11, attr=0x8000000000000001, range=[0x00003ff193f00000-0x00003ff194000000) (1MB) [ 0.000000] EFI: mem525: type=11, attr=0x8000000000000001, range=[0x00003ff197f00000-0x00003ff198000000) (1MB) [ 0.000000] EFI: mem526: type=11, attr=0x8000000000000001, range=[0x00003ff19bf00000-0x00003ff19c000000) (1MB) [ 0.000000] EFI: mem527: type=11, attr=0x8000000000000001, range=[0x00003ff19ff00000-0x00003ff1a0000000) (1MB) [ 0.000000] EFI: mem528: type=11, attr=0x8000000000000001, range=[0x00003ff1a3f00000-0x00003ff1a4000000) (1MB) [ 0.000000] EFI: mem529: type=11, attr=0x8000000000000001, range=[0x00003ff1a7f00000-0x00003ff1a8000000) (1MB) [ 0.000000] EFI: mem530: type=11, attr=0x8000000000000001, range=[0x00003ff1abf00000-0x00003ff1ac000000) (1MB) [ 0.000000] EFI: mem531: type=11, attr=0x8000000000000001, range=[0x00003ff1aff00000-0x00003ff1b0000000) (1MB) [ 0.000000] EFI: mem532: type=11, attr=0x8000000000000001, range=[0x00003ff1b3f00000-0x00003ff1b4000000) (1MB) [ 0.000000] EFI: mem533: type=11, attr=0x8000000000000001, range=[0x00003ff1b7f00000-0x00003ff1b8000000) (1MB) [ 0.000000] EFI: mem534: type=11, attr=0x8000000000000001, range=[0x00003ff1bbf00000-0x00003ff1bc000000) (1MB) [ 0.000000] EFI: mem535: type=11, attr=0x8000000000000001, range=[0x00003ff1bff00000-0x00003ff1c0000000) (1MB) [ 0.000000] EFI: mem536: type=11, attr=0x8000000000000001, range=[0x00003ff1c3f00000-0x00003ff1c4000000) (1MB) [ 0.000000] EFI: mem537: type=11, attr=0x8000000000000001, range=[0x00003ff1c7f00000-0x00003ff1c8000000) (1MB) [ 0.000000] EFI: mem538: type=11, attr=0x8000000000000001, range=[0x00003ff1cbf00000-0x00003ff1cc000000) (1MB) [ 0.000000] EFI: mem539: type=11, attr=0x8000000000000001, range=[0x00003ff1cff00000-0x00003ff1d0000000) (1MB) [ 0.000000] EFI: mem540: type=11, attr=0x8000000000000001, range=[0x00003ff1d3f00000-0x00003ff1d4000000) (1MB) [ 0.000000] EFI: mem541: type=11, attr=0x8000000000000001, range=[0x00003ff1d7f00000-0x00003ff1d8000000) (1MB) [ 0.000000] EFI: mem542: type=11, attr=0x8000000000000001, range=[0x00003ff1dbf00000-0x00003ff1dc000000) (1MB) [ 0.000000] EFI: mem543: type=11, attr=0x8000000000000001, range=[0x00003ff1dff00000-0x00003ff1e0000000) (1MB) [ 0.000000] EFI: mem544: type=11, attr=0x8000000000000001, range=[0x00003ff1e3f00000-0x00003ff1e4000000) (1MB) [ 0.000000] EFI: mem545: type=11, attr=0x8000000000000001, range=[0x00003ff1e7f00000-0x00003ff1e8000000) (1MB) [ 0.000000] EFI: mem546: type=11, attr=0x8000000000000001, range=[0x00003ff1ebf00000-0x00003ff1ec000000) (1MB) [ 0.000000] EFI: mem547: type=11, attr=0x8000000000000001, range=[0x00003ff1eff00000-0x00003ff1f0000000) (1MB) [ 0.000000] EFI: mem548: type=11, attr=0x8000000000000001, range=[0x00003ff1f3f00000-0x00003ff1f4000000) (1MB) [ 0.000000] EFI: mem549: type=11, attr=0x8000000000000001, range=[0x00003ff1f7f00000-0x00003ff1f8000000) (1MB) [ 0.000000] EFI: mem550: type=11, attr=0x8000000000000001, range=[0x00003ff1fbf00000-0x00003ff1fc000000) (1MB) [ 0.000000] EFI: mem551: type=11, attr=0x8000000000000001, range=[0x00003ff1fff00000-0x00003ff200000000) (1MB) [ 0.000000] EFI: mem552: type=11, attr=0x8000000000000001, range=[0x00003ff203f00000-0x00003ff204000000) (1MB) [ 0.000000] EFI: mem553: type=11, attr=0x8000000000000001, range=[0x00003ff207f00000-0x00003ff208000000) (1MB) [ 0.000000] EFI: mem554: type=11, attr=0x8000000000000001, range=[0x00003ff20bf00000-0x00003ff20c000000) (1MB) [ 0.000000] EFI: mem555: type=11, attr=0x8000000000000001, range=[0x00003ff20ff00000-0x00003ff210000000) (1MB) [ 0.000000] EFI: mem556: type=11, attr=0x8000000000000001, range=[0x00003ff213f00000-0x00003ff214000000) (1MB) [ 0.000000] EFI: mem557: type=11, attr=0x8000000000000001, range=[0x00003ff217f00000-0x00003ff218000000) (1MB) [ 0.000000] EFI: mem558: type=11, attr=0x8000000000000001, range=[0x00003ff21bf00000-0x00003ff21c000000) (1MB) [ 0.000000] EFI: mem559: type=11, attr=0x8000000000000001, range=[0x00003ff21ff00000-0x00003ff220000000) (1MB) [ 0.000000] EFI: mem560: type=11, attr=0x8000000000000001, range=[0x00003ff223f00000-0x00003ff224000000) (1MB) [ 0.000000] EFI: mem561: type=11, attr=0x8000000000000001, range=[0x00003ff227f00000-0x00003ff228000000) (1MB) [ 0.000000] EFI: mem562: type=11, attr=0x8000000000000001, range=[0x00003ff22bf00000-0x00003ff22c000000) (1MB) [ 0.000000] EFI: mem563: type=11, attr=0x8000000000000001, range=[0x00003ff22ff00000-0x00003ff230000000) (1MB) [ 0.000000] EFI: mem564: type=11, attr=0x8000000000000001, range=[0x00003ff233f00000-0x00003ff234000000) (1MB) [ 0.000000] EFI: mem565: type=11, attr=0x8000000000000001, range=[0x00003ff237f00000-0x00003ff238000000) (1MB) [ 0.000000] EFI: mem566: type=11, attr=0x8000000000000001, range=[0x00003ff23bf00000-0x00003ff23c000000) (1MB) [ 0.000000] EFI: mem567: type=11, attr=0x8000000000000001, range=[0x00003ff23ff00000-0x00003ff240000000) (1MB) [ 0.000000] EFI: mem568: type=11, attr=0x8000000000000001, range=[0x00003ff243f00000-0x00003ff244000000) (1MB) [ 0.000000] EFI: mem569: type=11, attr=0x8000000000000001, range=[0x00003ff247f00000-0x00003ff248000000) (1MB) [ 0.000000] EFI: mem570: type=11, attr=0x8000000000000001, range=[0x00003ff24bf00000-0x00003ff24c000000) (1MB) [ 0.000000] EFI: mem571: type=11, attr=0x8000000000000001, range=[0x00003ff24ff00000-0x00003ff250000000) (1MB) [ 0.000000] EFI: mem572: type=11, attr=0x8000000000000001, range=[0x00003ff253f00000-0x00003ff254000000) (1MB) [ 0.000000] EFI: mem573: type=11, attr=0x8000000000000001, range=[0x00003ff257f00000-0x00003ff258000000) (1MB) [ 0.000000] EFI: mem574: type=11, attr=0x8000000000000001, range=[0x00003ff25bf00000-0x00003ff25c000000) (1MB) [ 0.000000] EFI: mem575: type=11, attr=0x8000000000000001, range=[0x00003ff25ff00000-0x00003ff260000000) (1MB) [ 0.000000] EFI: mem576: type=11, attr=0x8000000000000001, range=[0x00003ff263f00000-0x00003ff264000000) (1MB) [ 0.000000] EFI: mem577: type=11, attr=0x8000000000000001, range=[0x00003ff267f00000-0x00003ff268000000) (1MB) [ 0.000000] EFI: mem578: type=11, attr=0x8000000000000001, range=[0x00003ff26bf00000-0x00003ff26c000000) (1MB) [ 0.000000] EFI: mem579: type=11, attr=0x8000000000000001, range=[0x00003ff26ff00000-0x00003ff270000000) (1MB) [ 0.000000] EFI: mem580: type=11, attr=0x8000000000000001, range=[0x00003ff273f00000-0x00003ff274000000) (1MB) [ 0.000000] EFI: mem581: type=11, attr=0x8000000000000001, range=[0x00003ff277f00000-0x00003ff278000000) (1MB) [ 0.000000] EFI: mem582: type=11, attr=0x8000000000000001, range=[0x00003ff27bf00000-0x00003ff27c000000) (1MB) [ 0.000000] EFI: mem583: type=11, attr=0x8000000000000001, range=[0x00003ff27ff00000-0x00003ff280000000) (1MB) [ 0.000000] EFI: mem584: type=11, attr=0x8000000000000001, range=[0x00003ff283f00000-0x00003ff284000000) (1MB) [ 0.000000] EFI: mem585: type=11, attr=0x8000000000000001, range=[0x00003ff287f00000-0x00003ff288000000) (1MB) [ 0.000000] EFI: mem586: type=11, attr=0x8000000000000001, range=[0x00003ff28bf00000-0x00003ff28c000000) (1MB) [ 0.000000] EFI: mem587: type=11, attr=0x8000000000000001, range=[0x00003ff28ff00000-0x00003ff290000000) (1MB) [ 0.000000] EFI: mem588: type=11, attr=0x8000000000000001, range=[0x00003ff293f00000-0x00003ff294000000) (1MB) [ 0.000000] EFI: mem589: type=11, attr=0x8000000000000001, range=[0x00003ff297f00000-0x00003ff298000000) (1MB) [ 0.000000] EFI: mem590: type=11, attr=0x8000000000000001, range=[0x00003ff29bf00000-0x00003ff29c000000) (1MB) [ 0.000000] EFI: mem591: type=11, attr=0x8000000000000001, range=[0x00003ff29ff00000-0x00003ff2a0000000) (1MB) [ 0.000000] EFI: mem592: type=11, attr=0x8000000000000001, range=[0x00003ff2a3f00000-0x00003ff2a4000000) (1MB) [ 0.000000] EFI: mem593: type=11, attr=0x8000000000000001, range=[0x00003ff2a7f00000-0x00003ff2a8000000) (1MB) [ 0.000000] EFI: mem594: type=11, attr=0x8000000000000001, range=[0x00003ff2abf00000-0x00003ff2ac000000) (1MB) [ 0.000000] EFI: mem595: type=11, attr=0x8000000000000001, range=[0x00003ff2aff00000-0x00003ff2b0000000) (1MB) [ 0.000000] EFI: mem596: type=11, attr=0x8000000000000001, range=[0x00003ff2b3f00000-0x00003ff2b4000000) (1MB) [ 0.000000] EFI: mem597: type=11, attr=0x8000000000000001, range=[0x00003ff2b7f00000-0x00003ff2b8000000) (1MB) [ 0.000000] EFI: mem598: type=11, attr=0x8000000000000001, range=[0x00003ff2bbf00000-0x00003ff2bc000000) (1MB) [ 0.000000] EFI: mem599: type=11, attr=0x8000000000000001, range=[0x00003ff2bff00000-0x00003ff2c0000000) (1MB) [ 0.000000] EFI: mem600: type=11, attr=0x8000000000000001, range=[0x00003ff2c3f00000-0x00003ff2c4000000) (1MB) [ 0.000000] EFI: mem601: type=11, attr=0x8000000000000001, range=[0x00003ff2c7f00000-0x00003ff2c8000000) (1MB) [ 0.000000] EFI: mem602: type=11, attr=0x8000000000000001, range=[0x00003ff2cbf00000-0x00003ff2cc000000) (1MB) [ 0.000000] EFI: mem603: type=11, attr=0x8000000000000001, range=[0x00003ff2cff00000-0x00003ff2d0000000) (1MB) [ 0.000000] EFI: mem604: type=11, attr=0x8000000000000001, range=[0x00003ff2d3f00000-0x00003ff2d4000000) (1MB) [ 0.000000] EFI: mem605: type=11, attr=0x8000000000000001, range=[0x00003ff2d7f00000-0x00003ff2d8000000) (1MB) [ 0.000000] EFI: mem606: type=11, attr=0x8000000000000001, range=[0x00003ff2dbf00000-0x00003ff2dc000000) (1MB) [ 0.000000] EFI: mem607: type=11, attr=0x8000000000000001, range=[0x00003ff2dff00000-0x00003ff2e0000000) (1MB) [ 0.000000] EFI: mem608: type=11, attr=0x8000000000000001, range=[0x00003ff2e3f00000-0x00003ff2e4000000) (1MB) [ 0.000000] EFI: mem609: type=11, attr=0x8000000000000001, range=[0x00003ff2e7f00000-0x00003ff2e8000000) (1MB) [ 0.000000] EFI: mem610: type=11, attr=0x8000000000000001, range=[0x00003ff2ebf00000-0x00003ff2ec000000) (1MB) [ 0.000000] EFI: mem611: type=11, attr=0x8000000000000001, range=[0x00003ff2eff00000-0x00003ff2f0000000) (1MB) [ 0.000000] EFI: mem612: type=11, attr=0x8000000000000001, range=[0x00003ff2f3f00000-0x00003ff2f4000000) (1MB) [ 0.000000] EFI: mem613: type=11, attr=0x8000000000000001, range=[0x00003ff2f7f00000-0x00003ff2f8000000) (1MB) [ 0.000000] EFI: mem614: type=11, attr=0x8000000000000001, range=[0x00003ff2fbf00000-0x00003ff2fc000000) (1MB) [ 0.000000] EFI: mem615: type=11, attr=0x8000000000000001, range=[0x00003ff2fff00000-0x00003ff300000000) (1MB) [ 0.000000] EFI: mem616: type=11, attr=0x8000000000000001, range=[0x00003ff303f00000-0x00003ff304000000) (1MB) [ 0.000000] EFI: mem617: type=11, attr=0x8000000000000001, range=[0x00003ff307f00000-0x00003ff308000000) (1MB) [ 0.000000] EFI: mem618: type=11, attr=0x8000000000000001, range=[0x00003ff30bf00000-0x00003ff30c000000) (1MB) [ 0.000000] EFI: mem619: type=11, attr=0x8000000000000001, range=[0x00003ff30ff00000-0x00003ff310000000) (1MB) [ 0.000000] EFI: mem620: type=11, attr=0x8000000000000001, range=[0x00003ff313f00000-0x00003ff314000000) (1MB) [ 0.000000] EFI: mem621: type=11, attr=0x8000000000000001, range=[0x00003ff317f00000-0x00003ff318000000) (1MB) [ 0.000000] EFI: mem622: type=11, attr=0x8000000000000001, range=[0x00003ff31bf00000-0x00003ff31c000000) (1MB) [ 0.000000] EFI: mem623: type=11, attr=0x8000000000000001, range=[0x00003ff31ff00000-0x00003ff320000000) (1MB) [ 0.000000] EFI: mem624: type=11, attr=0x8000000000000001, range=[0x00003ff323f00000-0x00003ff324000000) (1MB) [ 0.000000] EFI: mem625: type=11, attr=0x8000000000000001, range=[0x00003ff327f00000-0x00003ff328000000) (1MB) [ 0.000000] EFI: mem626: type=11, attr=0x8000000000000001, range=[0x00003ff32bf00000-0x00003ff32c000000) (1MB) [ 0.000000] EFI: mem627: type=11, attr=0x8000000000000001, range=[0x00003ff32ff00000-0x00003ff330000000) (1MB) [ 0.000000] EFI: mem628: type=11, attr=0x8000000000000001, range=[0x00003ff333f00000-0x00003ff334000000) (1MB) [ 0.000000] EFI: mem629: type=11, attr=0x8000000000000001, range=[0x00003ff337f00000-0x00003ff338000000) (1MB) [ 0.000000] EFI: mem630: type=11, attr=0x8000000000000001, range=[0x00003ff33bf00000-0x00003ff33c000000) (1MB) [ 0.000000] EFI: mem631: type=11, attr=0x8000000000000001, range=[0x00003ff33ff00000-0x00003ff340000000) (1MB) [ 0.000000] EFI: mem632: type=11, attr=0x8000000000000001, range=[0x00003ff343f00000-0x00003ff344000000) (1MB) [ 0.000000] EFI: mem633: type=11, attr=0x8000000000000001, range=[0x00003ff347f00000-0x00003ff348000000) (1MB) [ 0.000000] EFI: mem634: type=11, attr=0x8000000000000001, range=[0x00003ff34bf00000-0x00003ff34c000000) (1MB) [ 0.000000] EFI: mem635: type=11, attr=0x8000000000000001, range=[0x00003ff34ff00000-0x00003ff350000000) (1MB) [ 0.000000] EFI: mem636: type=11, attr=0x8000000000000001, range=[0x00003ff353f00000-0x00003ff354000000) (1MB) [ 0.000000] EFI: mem637: type=11, attr=0x8000000000000001, range=[0x00003ff357f00000-0x00003ff358000000) (1MB) [ 0.000000] EFI: mem638: type=11, attr=0x8000000000000001, range=[0x00003ff35bf00000-0x00003ff35c000000) (1MB) [ 0.000000] EFI: mem639: type=11, attr=0x8000000000000001, range=[0x00003ff35ff00000-0x00003ff360000000) (1MB) [ 0.000000] EFI: mem640: type=11, attr=0x8000000000000001, range=[0x00003ff363f00000-0x00003ff364000000) (1MB) [ 0.000000] EFI: mem641: type=11, attr=0x8000000000000001, range=[0x00003ff367f00000-0x00003ff368000000) (1MB) [ 0.000000] EFI: mem642: type=11, attr=0x8000000000000001, range=[0x00003ff36bf00000-0x00003ff36c000000) (1MB) [ 0.000000] EFI: mem643: type=11, attr=0x8000000000000001, range=[0x00003ff36ff00000-0x00003ff370000000) (1MB) [ 0.000000] EFI: mem644: type=11, attr=0x8000000000000001, range=[0x00003ff373f00000-0x00003ff374000000) (1MB) [ 0.000000] EFI: mem645: type=11, attr=0x8000000000000001, range=[0x00003ff377f00000-0x00003ff378000000) (1MB) [ 0.000000] EFI: mem646: type=11, attr=0x8000000000000001, range=[0x00003ff37bf00000-0x00003ff37c000000) (1MB) [ 0.000000] EFI: mem647: type=11, attr=0x8000000000000001, range=[0x00003ff37ff00000-0x00003ff380000000) (1MB) [ 0.000000] EFI: mem648: type=11, attr=0x8000000000000001, range=[0x00003ff383f00000-0x00003ff384000000) (1MB) [ 0.000000] EFI: mem649: type=11, attr=0x8000000000000001, range=[0x00003ff387f00000-0x00003ff388000000) (1MB) [ 0.000000] EFI: mem650: type=11, attr=0x8000000000000001, range=[0x00003ff38bf00000-0x00003ff38c000000) (1MB) [ 0.000000] EFI: mem651: type=11, attr=0x8000000000000001, range=[0x00003ff38ff00000-0x00003ff390000000) (1MB) [ 0.000000] EFI: mem652: type=11, attr=0x8000000000000001, range=[0x00003ff393f00000-0x00003ff394000000) (1MB) [ 0.000000] EFI: mem653: type=11, attr=0x8000000000000001, range=[0x00003ff397f00000-0x00003ff398000000) (1MB) [ 0.000000] EFI: mem654: type=11, attr=0x8000000000000001, range=[0x00003ff39bf00000-0x00003ff39c000000) (1MB) [ 0.000000] EFI: mem655: type=11, attr=0x8000000000000001, range=[0x00003ff39ff00000-0x00003ff3a0000000) (1MB) [ 0.000000] EFI: mem656: type=11, attr=0x8000000000000001, range=[0x00003ff3a3f00000-0x00003ff3a4000000) (1MB) [ 0.000000] EFI: mem657: type=11, attr=0x8000000000000001, range=[0x00003ff3a7f00000-0x00003ff3a8000000) (1MB) [ 0.000000] EFI: mem658: type=11, attr=0x8000000000000001, range=[0x00003ff3abf00000-0x00003ff3ac000000) (1MB) [ 0.000000] EFI: mem659: type=11, attr=0x8000000000000001, range=[0x00003ff3aff00000-0x00003ff3b0000000) (1MB) [ 0.000000] EFI: mem660: type=11, attr=0x8000000000000001, range=[0x00003ff3b3f00000-0x00003ff3b4000000) (1MB) [ 0.000000] EFI: mem661: type=11, attr=0x8000000000000001, range=[0x00003ff3b7f00000-0x00003ff3b8000000) (1MB) [ 0.000000] EFI: mem662: type=11, attr=0x8000000000000001, range=[0x00003ff3bbf00000-0x00003ff3bc000000) (1MB) [ 0.000000] EFI: mem663: type=11, attr=0x8000000000000001, range=[0x00003ff3bff00000-0x00003ff3c0000000) (1MB) [ 0.000000] EFI: mem664: type=11, attr=0x8000000000000001, range=[0x00003ff3c3f00000-0x00003ff3c4000000) (1MB) [ 0.000000] EFI: mem665: type=11, attr=0x8000000000000001, range=[0x00003ff3c7f00000-0x00003ff3c8000000) (1MB) [ 0.000000] EFI: mem666: type=11, attr=0x8000000000000001, range=[0x00003ff3cbf00000-0x00003ff3cc000000) (1MB) [ 0.000000] EFI: mem667: type=11, attr=0x8000000000000001, range=[0x00003ff3cff00000-0x00003ff3d0000000) (1MB) [ 0.000000] EFI: mem668: type=11, attr=0x8000000000000001, range=[0x00003ff3d3f00000-0x00003ff3d4000000) (1MB) [ 0.000000] EFI: mem669: type=11, attr=0x8000000000000001, range=[0x00003ff3d7f00000-0x00003ff3d8000000) (1MB) [ 0.000000] EFI: mem670: type=11, attr=0x8000000000000001, range=[0x00003ff3dbf00000-0x00003ff3dc000000) (1MB) [ 0.000000] EFI: mem671: type=11, attr=0x8000000000000001, range=[0x00003ff3dff00000-0x00003ff3e0000000) (1MB) [ 0.000000] EFI: mem672: type=11, attr=0x8000000000000001, range=[0x00003ff3e3f00000-0x00003ff3e4000000) (1MB) [ 0.000000] EFI: mem673: type=11, attr=0x8000000000000001, range=[0x00003ff3e7f00000-0x00003ff3e8000000) (1MB) [ 0.000000] EFI: mem674: type=11, attr=0x8000000000000001, range=[0x00003ff3ebf00000-0x00003ff3ec000000) (1MB) [ 0.000000] EFI: mem675: type=11, attr=0x8000000000000001, range=[0x00003ff3eff00000-0x00003ff3f0000000) (1MB) [ 0.000000] EFI: mem676: type=11, attr=0x8000000000000001, range=[0x00003ff3f3f00000-0x00003ff3f4000000) (1MB) [ 0.000000] EFI: mem677: type=11, attr=0x8000000000000001, range=[0x00003ff3f7f00000-0x00003ff3f8000000) (1MB) [ 0.000000] EFI: mem678: type=11, attr=0x8000000000000001, range=[0x00003ff3fbf00000-0x00003ff3fc000000) (1MB) [ 0.000000] EFI: mem679: type=11, attr=0x8000000000000001, range=[0x00003ff3fff00000-0x00003ff403000000) (49MB) [ 0.000000] EFI: mem680: type=11, attr=0x8000000000000001, range=[0x00003ff404000000-0x00003ff407000000) (48MB) [ 0.000000] EFI: mem681: type=11, attr=0x8000000000000001, range=[0x00003ff408000000-0x00003ff40b000000) (48MB) [ 0.000000] EFI: mem682: type=11, attr=0x8000000000000001, range=[0x00003ff40c000000-0x00003ff40f000000) (48MB) [ 0.000000] EFI: mem683: type=11, attr=0x8000000000000001, range=[0x00003ff410000000-0x00003ff413000000) (48MB) [ 0.000000] EFI: mem684: type=11, attr=0x8000000000000001, range=[0x00003ff414000000-0x00003ff417000000) (48MB) [ 0.000000] EFI: mem685: type=11, attr=0x8000000000000001, range=[0x00003ff418000000-0x00003ff41b000000) (48MB) [ 0.000000] EFI: mem686: type=11, attr=0x8000000000000001, range=[0x00003ff41c000000-0x00003ff41f000000) (48MB) [ 0.000000] EFI: mem687: type=11, attr=0x8000000000000001, range=[0x00003ff420000000-0x00003ff423000000) (48MB) [ 0.000000] EFI: mem688: type=11, attr=0x8000000000000001, range=[0x00003ff424000000-0x00003ff427000000) (48MB) [ 0.000000] EFI: mem689: type=11, attr=0x8000000000000001, range=[0x00003ff428000000-0x00003ff42b000000) (48MB) [ 0.000000] EFI: mem690: type=11, attr=0x8000000000000001, range=[0x00003ff42c000000-0x00003ff42f000000) (48MB) [ 0.000000] EFI: mem691: type=11, attr=0x8000000000000001, range=[0x00003ff430000000-0x00003ff433000000) (48MB) [ 0.000000] EFI: mem692: type=11, attr=0x8000000000000001, range=[0x00003ff434000000-0x00003ff437000000) (48MB) [ 0.000000] EFI: mem693: type=11, attr=0x8000000000000001, range=[0x00003ff438000000-0x00003ff43b000000) (48MB) [ 0.000000] EFI: mem694: type=11, attr=0x8000000000000001, range=[0x00003ff43c000000-0x00003ff43f000000) (48MB) [ 0.000000] EFI: mem695: type=11, attr=0x8000000000000001, range=[0x00003ff440000000-0x00003ff443000000) (48MB) [ 0.000000] EFI: mem696: type=11, attr=0x8000000000000001, range=[0x00003ff444000000-0x00003ff447000000) (48MB) [ 0.000000] EFI: mem697: type=11, attr=0x8000000000000001, range=[0x00003ff448000000-0x00003ff44b000000) (48MB) [ 0.000000] EFI: mem698: type=11, attr=0x8000000000000001, range=[0x00003ff44c000000-0x00003ff44f000000) (48MB) [ 0.000000] EFI: mem699: type=11, attr=0x8000000000000001, range=[0x00003ff450000000-0x00003ff453000000) (48MB) [ 0.000000] EFI: mem700: type=11, attr=0x8000000000000001, range=[0x00003ff454000000-0x00003ff457000000) (48MB) [ 0.000000] EFI: mem701: type=11, attr=0x8000000000000001, range=[0x00003ff458000000-0x00003ff45b000000) (48MB) [ 0.000000] EFI: mem702: type=11, attr=0x8000000000000001, range=[0x00003ff45c000000-0x00003ff45f000000) (48MB) [ 0.000000] EFI: mem703: type=11, attr=0x8000000000000001, range=[0x00003ff460000000-0x00003ff463000000) (48MB) [ 0.000000] EFI: mem704: type=11, attr=0x8000000000000001, range=[0x00003ff464000000-0x00003ff467000000) (48MB) [ 0.000000] EFI: mem705: type=11, attr=0x8000000000000001, range=[0x00003ff468000000-0x00003ff46b000000) (48MB) [ 0.000000] EFI: mem706: type=11, attr=0x8000000000000001, range=[0x00003ff46c000000-0x00003ff46f000000) (48MB) [ 0.000000] EFI: mem707: type=11, attr=0x8000000000000001, range=[0x00003ff470000000-0x00003ff473000000) (48MB) [ 0.000000] EFI: mem708: type=11, attr=0x8000000000000001, range=[0x00003ff474000000-0x00003ff477000000) (48MB) [ 0.000000] EFI: mem709: type=11, attr=0x8000000000000001, range=[0x00003ff478000000-0x00003ff47b000000) (48MB) [ 0.000000] EFI: mem710: type=11, attr=0x8000000000000001, range=[0x00003ff47c000000-0x00003ff47f000000) (48MB) [ 0.000000] EFI: mem711: type=11, attr=0x8000000000000001, range=[0x00003ff480000000-0x00003ff483000000) (48MB) [ 0.000000] EFI: mem712: type=11, attr=0x8000000000000001, range=[0x00003ff484000000-0x00003ff487000000) (48MB) [ 0.000000] EFI: mem713: type=11, attr=0x8000000000000001, range=[0x00003ff488000000-0x00003ff48b000000) (48MB) [ 0.000000] EFI: mem714: type=11, attr=0x8000000000000001, range=[0x00003ff48c000000-0x00003ff48f000000) (48MB) [ 0.000000] EFI: mem715: type=11, attr=0x8000000000000001, range=[0x00003ff490000000-0x00003ff493000000) (48MB) [ 0.000000] EFI: mem716: type=11, attr=0x8000000000000001, range=[0x00003ff494000000-0x00003ff497000000) (48MB) [ 0.000000] EFI: mem717: type=11, attr=0x8000000000000001, range=[0x00003ff498000000-0x00003ff49b000000) (48MB) [ 0.000000] EFI: mem718: type=11, attr=0x8000000000000001, range=[0x00003ff49c000000-0x00003ff49f000000) (48MB) [ 0.000000] EFI: mem719: type=11, attr=0x8000000000000001, range=[0x00003ff4a0000000-0x00003ff4a3000000) (48MB) [ 0.000000] EFI: mem720: type=11, attr=0x8000000000000001, range=[0x00003ff4a4000000-0x00003ff4a7000000) (48MB) [ 0.000000] EFI: mem721: type=11, attr=0x8000000000000001, range=[0x00003ff4a8000000-0x00003ff4ab000000) (48MB) [ 0.000000] EFI: mem722: type=11, attr=0x8000000000000001, range=[0x00003ff4ac000000-0x00003ff4af000000) (48MB) [ 0.000000] EFI: mem723: type=11, attr=0x8000000000000001, range=[0x00003ff4b0000000-0x00003ff4b3000000) (48MB) [ 0.000000] EFI: mem724: type=11, attr=0x8000000000000001, range=[0x00003ff4b4000000-0x00003ff4b7000000) (48MB) [ 0.000000] EFI: mem725: type=11, attr=0x8000000000000001, range=[0x00003ff4b8000000-0x00003ff4bb000000) (48MB) [ 0.000000] EFI: mem726: type=11, attr=0x8000000000000001, range=[0x00003ff4bc000000-0x00003ff4bf000000) (48MB) [ 0.000000] EFI: mem727: type=11, attr=0x8000000000000001, range=[0x00003ff4c0000000-0x00003ff4c3000000) (48MB) [ 0.000000] EFI: mem728: type=11, attr=0x8000000000000001, range=[0x00003ff4c4000000-0x00003ff4c7000000) (48MB) [ 0.000000] EFI: mem729: type=11, attr=0x8000000000000001, range=[0x00003ff4c8000000-0x00003ff4cb000000) (48MB) [ 0.000000] EFI: mem730: type=11, attr=0x8000000000000001, range=[0x00003ff4cc000000-0x00003ff4cf000000) (48MB) [ 0.000000] EFI: mem731: type=11, attr=0x8000000000000001, range=[0x00003ff4d0000000-0x00003ff4d3000000) (48MB) [ 0.000000] EFI: mem732: type=11, attr=0x8000000000000001, range=[0x00003ff4d4000000-0x00003ff4d7000000) (48MB) [ 0.000000] EFI: mem733: type=11, attr=0x8000000000000001, range=[0x00003ff4d8000000-0x00003ff4db000000) (48MB) [ 0.000000] EFI: mem734: type=11, attr=0x8000000000000001, range=[0x00003ff4dc000000-0x00003ff4df000000) (48MB) [ 0.000000] EFI: mem735: type=11, attr=0x8000000000000001, range=[0x00003ff4e0000000-0x00003ff4e3000000) (48MB) [ 0.000000] EFI: mem736: type=11, attr=0x8000000000000001, range=[0x00003ff4e4000000-0x00003ff4e7000000) (48MB) [ 0.000000] EFI: mem737: type=11, attr=0x8000000000000001, range=[0x00003ff4e8000000-0x00003ff4eb000000) (48MB) [ 0.000000] EFI: mem738: type=11, attr=0x8000000000000001, range=[0x00003ff4ec000000-0x00003ff4ef000000) (48MB) [ 0.000000] EFI: mem739: type=11, attr=0x8000000000000001, range=[0x00003ff4f0000000-0x00003ff4f3000000) (48MB) [ 0.000000] EFI: mem740: type=11, attr=0x8000000000000001, range=[0x00003ff4f4000000-0x00003ff4f7000000) (48MB) [ 0.000000] EFI: mem741: type=11, attr=0x8000000000000001, range=[0x00003ff4f8000000-0x00003ff4fb000000) (48MB) [ 0.000000] EFI: mem742: type=11, attr=0x8000000000000001, range=[0x00003ff4fc000000-0x00003ff4ff000000) (48MB) [ 0.000000] EFI: mem743: type=11, attr=0x8000000000000001, range=[0x00003ff500000000-0x00003ff503000000) (48MB) [ 0.000000] EFI: mem744: type=11, attr=0x8000000000000001, range=[0x00003ff504000000-0x00003ff507000000) (48MB) [ 0.000000] EFI: mem745: type=11, attr=0x8000000000000001, range=[0x00003ff508000000-0x00003ff50b000000) (48MB) [ 0.000000] EFI: mem746: type=11, attr=0x8000000000000001, range=[0x00003ff50c000000-0x00003ff50f000000) (48MB) [ 0.000000] EFI: mem747: type=11, attr=0x8000000000000001, range=[0x00003ff510000000-0x00003ff513000000) (48MB) [ 0.000000] EFI: mem748: type=11, attr=0x8000000000000001, range=[0x00003ff514000000-0x00003ff517000000) (48MB) [ 0.000000] EFI: mem749: type=11, attr=0x8000000000000001, range=[0x00003ff518000000-0x00003ff51b000000) (48MB) [ 0.000000] EFI: mem750: type=11, attr=0x8000000000000001, range=[0x00003ff51c000000-0x00003ff51f000000) (48MB) [ 0.000000] EFI: mem751: type=11, attr=0x8000000000000001, range=[0x00003ff520000000-0x00003ff523000000) (48MB) [ 0.000000] EFI: mem752: type=11, attr=0x8000000000000001, range=[0x00003ff524000000-0x00003ff527000000) (48MB) [ 0.000000] EFI: mem753: type=11, attr=0x8000000000000001, range=[0x00003ff528000000-0x00003ff52b000000) (48MB) [ 0.000000] EFI: mem754: type=11, attr=0x8000000000000001, range=[0x00003ff52c000000-0x00003ff52f000000) (48MB) [ 0.000000] EFI: mem755: type=11, attr=0x8000000000000001, range=[0x00003ff530000000-0x00003ff533000000) (48MB) [ 0.000000] EFI: mem756: type=11, attr=0x8000000000000001, range=[0x00003ff534000000-0x00003ff537000000) (48MB) [ 0.000000] EFI: mem757: type=11, attr=0x8000000000000001, range=[0x00003ff538000000-0x00003ff53b000000) (48MB) [ 0.000000] EFI: mem758: type=11, attr=0x8000000000000001, range=[0x00003ff53c000000-0x00003ff53f000000) (48MB) [ 0.000000] EFI: mem759: type=11, attr=0x8000000000000001, range=[0x00003ff540000000-0x00003ff543000000) (48MB) [ 0.000000] EFI: mem760: type=11, attr=0x8000000000000001, range=[0x00003ff544000000-0x00003ff547000000) (48MB) [ 0.000000] EFI: mem761: type=11, attr=0x8000000000000001, range=[0x00003ff548000000-0x00003ff54b000000) (48MB) [ 0.000000] EFI: mem762: type=11, attr=0x8000000000000001, range=[0x00003ff54c000000-0x00003ff54f000000) (48MB) [ 0.000000] EFI: mem763: type=11, attr=0x8000000000000001, range=[0x00003ff550000000-0x00003ff553000000) (48MB) [ 0.000000] EFI: mem764: type=11, attr=0x8000000000000001, range=[0x00003ff554000000-0x00003ff557000000) (48MB) [ 0.000000] EFI: mem765: type=11, attr=0x8000000000000001, range=[0x00003ff558000000-0x00003ff55b000000) (48MB) [ 0.000000] EFI: mem766: type=11, attr=0x8000000000000001, range=[0x00003ff55c000000-0x00003ff55f000000) (48MB) [ 0.000000] EFI: mem767: type=11, attr=0x8000000000000001, range=[0x00003ff560000000-0x00003ff563000000) (48MB) [ 0.000000] EFI: mem768: type=11, attr=0x8000000000000001, range=[0x00003ff564000000-0x00003ff567000000) (48MB) [ 0.000000] EFI: mem769: type=11, attr=0x8000000000000001, range=[0x00003ff568000000-0x00003ff56b000000) (48MB) [ 0.000000] EFI: mem770: type=11, attr=0x8000000000000001, range=[0x00003ff56c000000-0x00003ff56f000000) (48MB) [ 0.000000] EFI: mem771: type=11, attr=0x8000000000000001, range=[0x00003ff570000000-0x00003ff573000000) (48MB) [ 0.000000] EFI: mem772: type=11, attr=0x8000000000000001, range=[0x00003ff574000000-0x00003ff577000000) (48MB) [ 0.000000] EFI: mem773: type=11, attr=0x8000000000000001, range=[0x00003ff578000000-0x00003ff57b000000) (48MB) [ 0.000000] EFI: mem774: type=11, attr=0x8000000000000001, range=[0x00003ff57c000000-0x00003ff57f000000) (48MB) [ 0.000000] EFI: mem775: type=11, attr=0x8000000000000001, range=[0x00003ff580000000-0x00003ff583000000) (48MB) [ 0.000000] EFI: mem776: type=11, attr=0x8000000000000001, range=[0x00003ff584000000-0x00003ff587000000) (48MB) [ 0.000000] EFI: mem777: type=11, attr=0x8000000000000001, range=[0x00003ff588000000-0x00003ff58b000000) (48MB) [ 0.000000] EFI: mem778: type=11, attr=0x8000000000000001, range=[0x00003ff58c000000-0x00003ff58f000000) (48MB) [ 0.000000] EFI: mem779: type=11, attr=0x8000000000000001, range=[0x00003ff590000000-0x00003ff593000000) (48MB) [ 0.000000] EFI: mem780: type=11, attr=0x8000000000000001, range=[0x00003ff594000000-0x00003ff597000000) (48MB) [ 0.000000] EFI: mem781: type=11, attr=0x8000000000000001, range=[0x00003ff598000000-0x00003ff59b000000) (48MB) [ 0.000000] EFI: mem782: type=11, attr=0x8000000000000001, range=[0x00003ff59c000000-0x00003ff59f000000) (48MB) [ 0.000000] EFI: mem783: type=11, attr=0x8000000000000001, range=[0x00003ff5a0000000-0x00003ff5a3000000) (48MB) [ 0.000000] EFI: mem784: type=11, attr=0x8000000000000001, range=[0x00003ff5a4000000-0x00003ff5a7000000) (48MB) [ 0.000000] EFI: mem785: type=11, attr=0x8000000000000001, range=[0x00003ff5a8000000-0x00003ff5ab000000) (48MB) [ 0.000000] EFI: mem786: type=11, attr=0x8000000000000001, range=[0x00003ff5ac000000-0x00003ff5af000000) (48MB) [ 0.000000] EFI: mem787: type=11, attr=0x8000000000000001, range=[0x00003ff5b0000000-0x00003ff5b3000000) (48MB) [ 0.000000] EFI: mem788: type=11, attr=0x8000000000000001, range=[0x00003ff5b4000000-0x00003ff5b7000000) (48MB) [ 0.000000] EFI: mem789: type=11, attr=0x8000000000000001, range=[0x00003ff5b8000000-0x00003ff5bb000000) (48MB) [ 0.000000] EFI: mem790: type=11, attr=0x8000000000000001, range=[0x00003ff5bc000000-0x00003ff5bf000000) (48MB) [ 0.000000] EFI: mem791: type=11, attr=0x8000000000000001, range=[0x00003ff5c0000000-0x00003ff5c3000000) (48MB) [ 0.000000] EFI: mem792: type=11, attr=0x8000000000000001, range=[0x00003ff5c4000000-0x00003ff5c7000000) (48MB) [ 0.000000] EFI: mem793: type=11, attr=0x8000000000000001, range=[0x00003ff5c8000000-0x00003ff5cb000000) (48MB) [ 0.000000] EFI: mem794: type=11, attr=0x8000000000000001, range=[0x00003ff5cc000000-0x00003ff5cf000000) (48MB) [ 0.000000] EFI: mem795: type=11, attr=0x8000000000000001, range=[0x00003ff5d0000000-0x00003ff5d3000000) (48MB) [ 0.000000] EFI: mem796: type=11, attr=0x8000000000000001, range=[0x00003ff5d4000000-0x00003ff5d7000000) (48MB) [ 0.000000] EFI: mem797: type=11, attr=0x8000000000000001, range=[0x00003ff5d8000000-0x00003ff5db000000) (48MB) [ 0.000000] EFI: mem798: type=11, attr=0x8000000000000001, range=[0x00003ff5dc000000-0x00003ff5df000000) (48MB) [ 0.000000] EFI: mem799: type=11, attr=0x8000000000000001, range=[0x00003ff5e0000000-0x00003ff5e3000000) (48MB) [ 0.000000] EFI: mem800: type=11, attr=0x8000000000000001, range=[0x00003ff5e4000000-0x00003ff5e7000000) (48MB) [ 0.000000] EFI: mem801: type=11, attr=0x8000000000000001, range=[0x00003ff5e8000000-0x00003ff5eb000000) (48MB) [ 0.000000] EFI: mem802: type=11, attr=0x8000000000000001, range=[0x00003ff5ec000000-0x00003ff5ef000000) (48MB) [ 0.000000] EFI: mem803: type=11, attr=0x8000000000000001, range=[0x00003ff5f0000000-0x00003ff5f3000000) (48MB) [ 0.000000] EFI: mem804: type=11, attr=0x8000000000000001, range=[0x00003ff5f4000000-0x00003ff5f7000000) (48MB) [ 0.000000] EFI: mem805: type=11, attr=0x8000000000000001, range=[0x00003ff5f8000000-0x00003ff5fb000000) (48MB) [ 0.000000] EFI: mem806: type=11, attr=0x8000000000000001, range=[0x00003ff5fc000000-0x00003ff5ff000000) (48MB) [ 0.000000] EFI: mem807: type=11, attr=0x8000000000000001, range=[0x00003ff600000000-0x00003ff603000000) (48MB) [ 0.000000] EFI: mem808: type=11, attr=0x8000000000000001, range=[0x00003ff604000000-0x00003ff607000000) (48MB) [ 0.000000] EFI: mem809: type=11, attr=0x8000000000000001, range=[0x00003ff608000000-0x00003ff60b000000) (48MB) [ 0.000000] EFI: mem810: type=11, attr=0x8000000000000001, range=[0x00003ff60c000000-0x00003ff60f000000) (48MB) [ 0.000000] EFI: mem811: type=11, attr=0x8000000000000001, range=[0x00003ff610000000-0x00003ff613000000) (48MB) [ 0.000000] EFI: mem812: type=11, attr=0x8000000000000001, range=[0x00003ff614000000-0x00003ff617000000) (48MB) [ 0.000000] EFI: mem813: type=11, attr=0x8000000000000001, range=[0x00003ff618000000-0x00003ff61b000000) (48MB) [ 0.000000] EFI: mem814: type=11, attr=0x8000000000000001, range=[0x00003ff61c000000-0x00003ff61f000000) (48MB) [ 0.000000] EFI: mem815: type=11, attr=0x8000000000000001, range=[0x00003ff620000000-0x00003ff623000000) (48MB) [ 0.000000] EFI: mem816: type=11, attr=0x8000000000000001, range=[0x00003ff624000000-0x00003ff627000000) (48MB) [ 0.000000] EFI: mem817: type=11, attr=0x8000000000000001, range=[0x00003ff628000000-0x00003ff62b000000) (48MB) [ 0.000000] EFI: mem818: type=11, attr=0x8000000000000001, range=[0x00003ff62c000000-0x00003ff62f000000) (48MB) [ 0.000000] EFI: mem819: type=11, attr=0x8000000000000001, range=[0x00003ff630000000-0x00003ff633000000) (48MB) [ 0.000000] EFI: mem820: type=11, attr=0x8000000000000001, range=[0x00003ff634000000-0x00003ff637000000) (48MB) [ 0.000000] EFI: mem821: type=11, attr=0x8000000000000001, range=[0x00003ff638000000-0x00003ff63b000000) (48MB) [ 0.000000] EFI: mem822: type=11, attr=0x8000000000000001, range=[0x00003ff63c000000-0x00003ff63f000000) (48MB) [ 0.000000] EFI: mem823: type=11, attr=0x8000000000000001, range=[0x00003ff640000000-0x00003ff643000000) (48MB) [ 0.000000] EFI: mem824: type=11, attr=0x8000000000000001, range=[0x00003ff644000000-0x00003ff647000000) (48MB) [ 0.000000] EFI: mem825: type=11, attr=0x8000000000000001, range=[0x00003ff648000000-0x00003ff64b000000) (48MB) [ 0.000000] EFI: mem826: type=11, attr=0x8000000000000001, range=[0x00003ff64c000000-0x00003ff64f000000) (48MB) [ 0.000000] EFI: mem827: type=11, attr=0x8000000000000001, range=[0x00003ff650000000-0x00003ff653000000) (48MB) [ 0.000000] EFI: mem828: type=11, attr=0x8000000000000001, range=[0x00003ff654000000-0x00003ff657000000) (48MB) [ 0.000000] EFI: mem829: type=11, attr=0x8000000000000001, range=[0x00003ff658000000-0x00003ff65b000000) (48MB) [ 0.000000] EFI: mem830: type=11, attr=0x8000000000000001, range=[0x00003ff65c000000-0x00003ff65f000000) (48MB) [ 0.000000] EFI: mem831: type=11, attr=0x8000000000000001, range=[0x00003ff660000000-0x00003ff663000000) (48MB) [ 0.000000] EFI: mem832: type=11, attr=0x8000000000000001, range=[0x00003ff664000000-0x00003ff667000000) (48MB) [ 0.000000] EFI: mem833: type=11, attr=0x8000000000000001, range=[0x00003ff668000000-0x00003ff66b000000) (48MB) [ 0.000000] EFI: mem834: type=11, attr=0x8000000000000001, range=[0x00003ff66c000000-0x00003ff66f000000) (48MB) [ 0.000000] EFI: mem835: type=11, attr=0x8000000000000001, range=[0x00003ff670000000-0x00003ff673000000) (48MB) [ 0.000000] EFI: mem836: type=11, attr=0x8000000000000001, range=[0x00003ff674000000-0x00003ff677000000) (48MB) [ 0.000000] EFI: mem837: type=11, attr=0x8000000000000001, range=[0x00003ff678000000-0x00003ff67b000000) (48MB) [ 0.000000] EFI: mem838: type=11, attr=0x8000000000000001, range=[0x00003ff67c000000-0x00003ff67f000000) (48MB) [ 0.000000] EFI: mem839: type=11, attr=0x8000000000000001, range=[0x00003ff680000000-0x00003ff683000000) (48MB) [ 0.000000] EFI: mem840: type=11, attr=0x8000000000000001, range=[0x00003ff684000000-0x00003ff687000000) (48MB) [ 0.000000] EFI: mem841: type=11, attr=0x8000000000000001, range=[0x00003ff688000000-0x00003ff68b000000) (48MB) [ 0.000000] EFI: mem842: type=11, attr=0x8000000000000001, range=[0x00003ff68c000000-0x00003ff68f000000) (48MB) [ 0.000000] EFI: mem843: type=11, attr=0x8000000000000001, range=[0x00003ff690000000-0x00003ff693000000) (48MB) [ 0.000000] EFI: mem844: type=11, attr=0x8000000000000001, range=[0x00003ff694000000-0x00003ff697000000) (48MB) [ 0.000000] EFI: mem845: type=11, attr=0x8000000000000001, range=[0x00003ff698000000-0x00003ff69b000000) (48MB) [ 0.000000] EFI: mem846: type=11, attr=0x8000000000000001, range=[0x00003ff69c000000-0x00003ff69f000000) (48MB) [ 0.000000] EFI: mem847: type=11, attr=0x8000000000000001, range=[0x00003ff6a0000000-0x00003ff6a3000000) (48MB) [ 0.000000] EFI: mem848: type=11, attr=0x8000000000000001, range=[0x00003ff6a4000000-0x00003ff6a7000000) (48MB) [ 0.000000] EFI: mem849: type=11, attr=0x8000000000000001, range=[0x00003ff6a8000000-0x00003ff6ab000000) (48MB) [ 0.000000] EFI: mem850: type=11, attr=0x8000000000000001, range=[0x00003ff6ac000000-0x00003ff6af000000) (48MB) [ 0.000000] EFI: mem851: type=11, attr=0x8000000000000001, range=[0x00003ff6b0000000-0x00003ff6b3000000) (48MB) [ 0.000000] EFI: mem852: type=11, attr=0x8000000000000001, range=[0x00003ff6b4000000-0x00003ff6b7000000) (48MB) [ 0.000000] EFI: mem853: type=11, attr=0x8000000000000001, range=[0x00003ff6b8000000-0x00003ff6bb000000) (48MB) [ 0.000000] EFI: mem854: type=11, attr=0x8000000000000001, range=[0x00003ff6bc000000-0x00003ff6bf000000) (48MB) [ 0.000000] EFI: mem855: type=11, attr=0x8000000000000001, range=[0x00003ff6c0000000-0x00003ff6c3000000) (48MB) [ 0.000000] EFI: mem856: type=11, attr=0x8000000000000001, range=[0x00003ff6c4000000-0x00003ff6c7000000) (48MB) [ 0.000000] EFI: mem857: type=11, attr=0x8000000000000001, range=[0x00003ff6c8000000-0x00003ff6cb000000) (48MB) [ 0.000000] EFI: mem858: type=11, attr=0x8000000000000001, range=[0x00003ff6cc000000-0x00003ff6cf000000) (48MB) [ 0.000000] EFI: mem859: type=11, attr=0x8000000000000001, range=[0x00003ff6d0000000-0x00003ff6d3000000) (48MB) [ 0.000000] EFI: mem860: type=11, attr=0x8000000000000001, range=[0x00003ff6d4000000-0x00003ff6d7000000) (48MB) [ 0.000000] EFI: mem861: type=11, attr=0x8000000000000001, range=[0x00003ff6d8000000-0x00003ff6db000000) (48MB) [ 0.000000] EFI: mem862: type=11, attr=0x8000000000000001, range=[0x00003ff6dc000000-0x00003ff6df000000) (48MB) [ 0.000000] EFI: mem863: type=11, attr=0x8000000000000001, range=[0x00003ff6e0000000-0x00003ff6e3000000) (48MB) [ 0.000000] EFI: mem864: type=11, attr=0x8000000000000001, range=[0x00003ff6e4000000-0x00003ff6e7000000) (48MB) [ 0.000000] EFI: mem865: type=11, attr=0x8000000000000001, range=[0x00003ff6e8000000-0x00003ff6eb000000) (48MB) [ 0.000000] EFI: mem866: type=11, attr=0x8000000000000001, range=[0x00003ff6ec000000-0x00003ff6ef000000) (48MB) [ 0.000000] EFI: mem867: type=11, attr=0x8000000000000001, range=[0x00003ff6f0000000-0x00003ff6f3000000) (48MB) [ 0.000000] EFI: mem868: type=11, attr=0x8000000000000001, range=[0x00003ff6f4000000-0x00003ff6f7000000) (48MB) [ 0.000000] EFI: mem869: type=11, attr=0x8000000000000001, range=[0x00003ff6f8000000-0x00003ff6fb000000) (48MB) [ 0.000000] EFI: mem870: type=11, attr=0x8000000000000001, range=[0x00003ff6fc000000-0x00003ff6ff000000) (48MB) [ 0.000000] EFI: mem871: type=11, attr=0x8000000000000001, range=[0x00003ff700000000-0x00003ff703000000) (48MB) [ 0.000000] EFI: mem872: type=11, attr=0x8000000000000001, range=[0x00003ff704000000-0x00003ff707000000) (48MB) [ 0.000000] EFI: mem873: type=11, attr=0x8000000000000001, range=[0x00003ff708000000-0x00003ff70b000000) (48MB) [ 0.000000] EFI: mem874: type=11, attr=0x8000000000000001, range=[0x00003ff70c000000-0x00003ff70f000000) (48MB) [ 0.000000] EFI: mem875: type=11, attr=0x8000000000000001, range=[0x00003ff710000000-0x00003ff713000000) (48MB) [ 0.000000] EFI: mem876: type=11, attr=0x8000000000000001, range=[0x00003ff714000000-0x00003ff717000000) (48MB) [ 0.000000] EFI: mem877: type=11, attr=0x8000000000000001, range=[0x00003ff718000000-0x00003ff71b000000) (48MB) [ 0.000000] EFI: mem878: type=11, attr=0x8000000000000001, range=[0x00003ff71c000000-0x00003ff71f000000) (48MB) [ 0.000000] EFI: mem879: type=11, attr=0x8000000000000001, range=[0x00003ff720000000-0x00003ff723000000) (48MB) [ 0.000000] EFI: mem880: type=11, attr=0x8000000000000001, range=[0x00003ff724000000-0x00003ff727000000) (48MB) [ 0.000000] EFI: mem881: type=11, attr=0x8000000000000001, range=[0x00003ff728000000-0x00003ff72b000000) (48MB) [ 0.000000] EFI: mem882: type=11, attr=0x8000000000000001, range=[0x00003ff72c000000-0x00003ff72f000000) (48MB) [ 0.000000] EFI: mem883: type=11, attr=0x8000000000000001, range=[0x00003ff730000000-0x00003ff733000000) (48MB) [ 0.000000] EFI: mem884: type=11, attr=0x8000000000000001, range=[0x00003ff734000000-0x00003ff737000000) (48MB) [ 0.000000] EFI: mem885: type=11, attr=0x8000000000000001, range=[0x00003ff738000000-0x00003ff73b000000) (48MB) [ 0.000000] EFI: mem886: type=11, attr=0x8000000000000001, range=[0x00003ff73c000000-0x00003ff73f000000) (48MB) [ 0.000000] EFI: mem887: type=11, attr=0x8000000000000001, range=[0x00003ff740000000-0x00003ff743000000) (48MB) [ 0.000000] EFI: mem888: type=11, attr=0x8000000000000001, range=[0x00003ff744000000-0x00003ff747000000) (48MB) [ 0.000000] EFI: mem889: type=11, attr=0x8000000000000001, range=[0x00003ff748000000-0x00003ff74b000000) (48MB) [ 0.000000] EFI: mem890: type=11, attr=0x8000000000000001, range=[0x00003ff74c000000-0x00003ff74f000000) (48MB) [ 0.000000] EFI: mem891: type=11, attr=0x8000000000000001, range=[0x00003ff750000000-0x00003ff753000000) (48MB) [ 0.000000] EFI: mem892: type=11, attr=0x8000000000000001, range=[0x00003ff754000000-0x00003ff757000000) (48MB) [ 0.000000] EFI: mem893: type=11, attr=0x8000000000000001, range=[0x00003ff758000000-0x00003ff75b000000) (48MB) [ 0.000000] EFI: mem894: type=11, attr=0x8000000000000001, range=[0x00003ff75c000000-0x00003ff75f000000) (48MB) [ 0.000000] EFI: mem895: type=11, attr=0x8000000000000001, range=[0x00003ff760000000-0x00003ff763000000) (48MB) [ 0.000000] EFI: mem896: type=11, attr=0x8000000000000001, range=[0x00003ff764000000-0x00003ff767000000) (48MB) [ 0.000000] EFI: mem897: type=11, attr=0x8000000000000001, range=[0x00003ff768000000-0x00003ff76b000000) (48MB) [ 0.000000] EFI: mem898: type=11, attr=0x8000000000000001, range=[0x00003ff76c000000-0x00003ff76f000000) (48MB) [ 0.000000] EFI: mem899: type=11, attr=0x8000000000000001, range=[0x00003ff770000000-0x00003ff773000000) (48MB) [ 0.000000] EFI: mem900: type=11, attr=0x8000000000000001, range=[0x00003ff774000000-0x00003ff777000000) (48MB) [ 0.000000] EFI: mem901: type=11, attr=0x8000000000000001, range=[0x00003ff778000000-0x00003ff77b000000) (48MB) [ 0.000000] EFI: mem902: type=11, attr=0x8000000000000001, range=[0x00003ff77c000000-0x00003ff77f000000) (48MB) [ 0.000000] EFI: mem903: type=11, attr=0x8000000000000001, range=[0x00003ff780000000-0x00003ff783000000) (48MB) [ 0.000000] EFI: mem904: type=11, attr=0x8000000000000001, range=[0x00003ff784000000-0x00003ff787000000) (48MB) [ 0.000000] EFI: mem905: type=11, attr=0x8000000000000001, range=[0x00003ff788000000-0x00003ff78b000000) (48MB) [ 0.000000] EFI: mem906: type=11, attr=0x8000000000000001, range=[0x00003ff78c000000-0x00003ff78f000000) (48MB) [ 0.000000] EFI: mem907: type=11, attr=0x8000000000000001, range=[0x00003ff790000000-0x00003ff793000000) (48MB) [ 0.000000] EFI: mem908: type=11, attr=0x8000000000000001, range=[0x00003ff794000000-0x00003ff797000000) (48MB) [ 0.000000] EFI: mem909: type=11, attr=0x8000000000000001, range=[0x00003ff798000000-0x00003ff79b000000) (48MB) [ 0.000000] EFI: mem910: type=11, attr=0x8000000000000001, range=[0x00003ff79c000000-0x00003ff79f000000) (48MB) [ 0.000000] EFI: mem911: type=11, attr=0x8000000000000001, range=[0x00003ff7a0000000-0x00003ff7a3000000) (48MB) [ 0.000000] EFI: mem912: type=11, attr=0x8000000000000001, range=[0x00003ff7a4000000-0x00003ff7a7000000) (48MB) [ 0.000000] EFI: mem913: type=11, attr=0x8000000000000001, range=[0x00003ff7a8000000-0x00003ff7ab000000) (48MB) [ 0.000000] EFI: mem914: type=11, attr=0x8000000000000001, range=[0x00003ff7ac000000-0x00003ff7af000000) (48MB) [ 0.000000] EFI: mem915: type=11, attr=0x8000000000000001, range=[0x00003ff7b0000000-0x00003ff7b3000000) (48MB) [ 0.000000] EFI: mem916: type=11, attr=0x8000000000000001, range=[0x00003ff7b4000000-0x00003ff7b7000000) (48MB) [ 0.000000] EFI: mem917: type=11, attr=0x8000000000000001, range=[0x00003ff7b8000000-0x00003ff7bb000000) (48MB) [ 0.000000] EFI: mem918: type=11, attr=0x8000000000000001, range=[0x00003ff7bc000000-0x00003ff7bf000000) (48MB) [ 0.000000] EFI: mem919: type=11, attr=0x8000000000000001, range=[0x00003ff7c0000000-0x00003ff7c3000000) (48MB) [ 0.000000] EFI: mem920: type=11, attr=0x8000000000000001, range=[0x00003ff7c4000000-0x00003ff7c7000000) (48MB) [ 0.000000] EFI: mem921: type=11, attr=0x8000000000000001, range=[0x00003ff7c8000000-0x00003ff7cb000000) (48MB) [ 0.000000] EFI: mem922: type=11, attr=0x8000000000000001, range=[0x00003ff7cc000000-0x00003ff7cf000000) (48MB) [ 0.000000] EFI: mem923: type=11, attr=0x8000000000000001, range=[0x00003ff7d0000000-0x00003ff7d3000000) (48MB) [ 0.000000] EFI: mem924: type=11, attr=0x8000000000000001, range=[0x00003ff7d4000000-0x00003ff7d7000000) (48MB) [ 0.000000] EFI: mem925: type=11, attr=0x8000000000000001, range=[0x00003ff7d8000000-0x00003ff7db000000) (48MB) [ 0.000000] EFI: mem926: type=11, attr=0x8000000000000001, range=[0x00003ff7dc000000-0x00003ff7df000000) (48MB) [ 0.000000] EFI: mem927: type=11, attr=0x8000000000000001, range=[0x00003ff7e0000000-0x00003ff7e3000000) (48MB) [ 0.000000] EFI: mem928: type=11, attr=0x8000000000000001, range=[0x00003ff7e4000000-0x00003ff7e7000000) (48MB) [ 0.000000] EFI: mem929: type=11, attr=0x8000000000000001, range=[0x00003ff7e8000000-0x00003ff7eb000000) (48MB) [ 0.000000] EFI: mem930: type=11, attr=0x8000000000000001, range=[0x00003ff7ec000000-0x00003ff7ef000000) (48MB) [ 0.000000] EFI: mem931: type=11, attr=0x8000000000000001, range=[0x00003ff7f0000000-0x00003ff7f3000000) (48MB) [ 0.000000] EFI: mem932: type=11, attr=0x8000000000000001, range=[0x00003ff7f4000000-0x00003ff7f7000000) (48MB) [ 0.000000] EFI: mem933: type=11, attr=0x8000000000000001, range=[0x00003ff7f8000000-0x00003ff7fb000000) (48MB) [ 0.000000] EFI: mem934: type=11, attr=0x8000000000000001, range=[0x00003ff7fc000000-0x00003ff7ff000000) (48MB)
Robin
Quoted 8 lines · Yinghai Lu
Quoted 4 lines · Mike Travis
experimenting as soon as I can. Our 32TB system is being brought back to 16TB (we found a number of problems as we get closer and closer to the 64TB limit), but that's still a significant size.
Hi, Mike,
Quoted 8 lines
Can you post e820 memory map on system that have 32TiB or more?
Is there one range size more than 16TiB? like [16TiB, 32TiB)...
Thanks
Yinghai
Here is (was) the 32T system with 2048 cores with HT disabled:
BIOS-e820: 0000000000000000 - 000000000007f000 (usable)
BIOS-e820: 000000000007f000 - 0000000000080000 (reserved)
BIOS-e820: 0000000000080000 - 00000000000a0000 (usable)
BIOS-e820: 0000000000100000 - 000000007ad54000 (usable)
BIOS-e820: 000000007ad54000 - 000000007ad55000 (reserved)
BIOS-e820: 000000007ad55000 - 000000007ad68000 (usable)
BIOS-e820: 000000007ad68000 - 000000007afa8000 (reserved)
BIOS-e820: 000000007afa8000 - 000000007bcb7000 (usable)
BIOS-e820: 000000007bcb7000 - 000000007bdb7000 (reserved)
BIOS-e820: 000000007bdb7000 - 000000007beb7000 (unusable)
BIOS-e820: 000000007beb7000 - 000000007bfb7000 (reserved)
BIOS-e820: 000000007bfb7000 - 000000007d1b7000 (ACPI NVS)
BIOS-e820: 000000007d1b7000 - 000000007e000000 (ACPI data)
BIOS-e820: 000000007e000000 - 000000007e318000 (usable)
BIOS-e820: 000000007e318000 - 000000007ef49000 (ACPI data)
BIOS-e820: 000000007ef49000 - 000000007f000000 (usable)
BIOS-e820: 0000000100000000 - 0000001e00000000 (usable)
BIOS-e820: 0000002000000000 - 0000003dff000000 (usable)
BIOS-e820: 0000004000000000 - 0000005dff000000 (usable)
BIOS-e820: 0000006000000000 - 0000007dff000000 (usable)
....
BIOS-e820: 00000e0000000000 - 00000e1dff000000 (usable)
BIOS-e820: 00000e2000000000 - 00000e3dff000000 (usable)
BIOS-e820: 00000e4000000000 - 00000e5dff000000 (usable)
BIOS-e820: 00000e6000000000 - 00000e7dff000000 (usable)
Note that on UV, there is some memory reserved at the end of each node that is used for the directory RAM by the UV HUB. That is why the ranges are not contiguous. I'd have to double check with the BIOS guys, but I don't believe they would have collapsed ranges across nodes even if the directory RAM did not exist.
-Mike
Quoted 8 lines · Yinghai Lu
Quoted 5 lines · H. Peter Anvin
On 06/25/2013 10:35 AM, Mike Travis wrote:
However, please consider Ingo's counterproposal of doing this via the buddy allocator, i.e. hugepages being broken on demand. That is a *very* powerful model, although would require more infrastructure.
Can you or Ingo elaborate more about the buddy allocator proposal?
Ok, I just wrote a fuller proposal, with cons and pros listed as well.
I think it's a winner but hey.
Thanks,
Ingo
Quoted 17 lines · Mike Travis
Quoted 6 lines · Yinghai Lu
Quoted 1 line · Mike Travis
Exactly. That's why I left both low and high memory on each node.
looks like you assume every node have same ram, and before booting you you need to know memory layout to append the boot command line.
Quoted 7 lines
We have patchset that moving srat table parse early.
git://git.kernel.org/pub/scm/linux/kernel/git/yinghai/linux-yinghai.git
for-x86-mm
https://git.kernel.org/cgit/linux/kernel/git/yinghai/linux-yinghai.git/log/?h=for-x86-mmon top that, we could make your patch pass more simple command like 1/2^n of every node, and only need to pass n instead.
Quoted 6 lines
The two params that I couldn't figure out how to provide except via kernel param option was the memory block size (128M or 2G) and the physical address space per node. The other 3 params can be automatically setup by a script when the total system size is known. As soon as we verify on the 32TB system and surmise what will be needed for 64TB, then those 3 params can probably disappear.
our "numa parsing early" patchset could provide "physical address space per node",
also can calculate memory block size via alignment detection from numa info.
with that, user only can pass "delay_init_mem" only.
Thanks
Yinghai
Quoted 25 lines · Yinghai Lu
Quoted 17 lines · Mike Travis
Quoted 6 lines · Yinghai Lu
Quoted 1 line · Mike Travis
Exactly. That's why I left both low and high memory on each node.
looks like you assume every node have same ram, and before booting you you need to know memory layout to append the boot command line.
Quoted 7 lines
We have patchset that moving srat table parse early.
git://git.kernel.org/pub/scm/linux/kernel/git/yinghai/linux-yinghai.git
for-x86-mm
https://git.kernel.org/cgit/linux/kernel/git/yinghai/linux-yinghai.git/log/?h=for-x86-mmon top that, we could make your patch pass more simple command like 1/2^n of every node, and only need to pass n instead.
Quoted 6 lines
The two params that I couldn't figure out how to provide except via kernel param option was the memory block size (128M or 2G) and the physical address space per node. The other 3 params can be automatically setup by a script when the total system size is known. As soon as we verify on the 32TB system and surmise what will be needed for 64TB, then those 3 params can probably disappear.
Quoted 3 lines
our "numa parsing early" patchset could provide "physical address space per node",
also can calculate memory block size via alignment detection from numa info.
Thanks! I'll try it out.
Quoted 7 lines
with that, user only can pass "
delay_init_mem" only.Thanks
Yinghai
Quoted 4 lines · Nathan Zimmer
This rfc patch set delays initializing large sections of memory until we have started cpus. This has the effect of reducing startup times on large memory systems. On 16TB it can take over an hour to boot and most of that time is spent initializing memory.
One hour on system with 16T ram? BIOS or OS?
I use wall clock to check bootime on one system with 3T and 16 pcie cards, Linus only takes about 3m and 30 seconds from bootloader.
wonder if you boot delay is with so many cpu get onlined in serialized mode.
so can you try boot your system with "maxcpus=128" to get the boot time with wall clock ?
Thanks
Yinghai
Greg Kroah-Hartman
maintainerQuoted 8 lines · Yinghai Lu
Quoted 4 lines · Nathan Zimmer
This rfc patch set delays initializing large sections of memory until we have started cpus. This has the effect of reducing startup times on large memory systems. On 16TB it can take over an hour to boot and most of that time is spent initializing memory.
One hour on system with 16T ram? BIOS or OS?
Quoted 7 lines
I use wall clock to check bootime on one system with 3T and 16 pcie cards, Linus only takes about 3m and 30 seconds from bootloader.
wonder if you boot delay is with so many cpu get onlined in serialized mode.
so can you try boot your system with "maxcpus=128" to get the boot time with wall clock ?
Why use the "wall clock" when we have the wonderful bootchart tools and scripts that do this all for you, and can tell you exactly what part of the kernel is taking what time, to help with fixing issues like this?
thanks,
greg k-h
Quoted 12 lines · Greg Kroah-Hartman
Quoted 11 lines · Yinghai Lu
Quoted 4 lines · Nathan Zimmer
This rfc patch set delays initializing large sections of memory until we have started cpus. This has the effect of reducing startup times on large memory systems. On 16TB it can take over an hour to boot and most of that time is spent initializing memory.
One hour on system with 16T ram? BIOS or OS?
I use wall clock to check bootime on one system with 3T and 16 pcie cards, Linus only takes about 3m and 30 seconds from bootloader.
Quoted 8 lines
Quoted 4 lines
wonder if you boot delay is with so many cpu get onlined in serialized mode.
so can you try boot your system with "maxcpus=128" to get the boot time with wall clock ?
Why use the "wall clock" when we have the wonderful bootchart tools and scripts that do this all for you, and can tell you exactly what part of the kernel is taking what time, to help with fixing issues like this?
bootchart is not completed.
printk timestamp come after mem get initialized.
....
[ 0.004000] tsc: Fast TSC calibration using PIT
before that stamp are all 0.
Yinghai
Quoted 25 lines · Yinghai Lu
Quoted 12 lines · Greg Kroah-Hartman
Quoted 11 lines · Yinghai Lu
Quoted 4 lines · Nathan Zimmer
This rfc patch set delays initializing large sections of memory until we have started cpus. This has the effect of reducing startup times on large memory systems. On 16TB it can take over an hour to boot and most of that time is spent initializing memory.
One hour on system with 16T ram? BIOS or OS?
I use wall clock to check bootime on one system with 3T and 16 pcie cards, Linus only takes about 3m and 30 seconds from bootloader.
Quoted 8 lines
Quoted 4 lines
wonder if you boot delay is with so many cpu get onlined in serialized mode.
so can you try boot your system with "maxcpus=128" to get the boot time with wall clock ?
Why use the "wall clock" when we have the wonderful bootchart tools and scripts that do this all for you, and can tell you exactly what part of the kernel is taking what time, to help with fixing issues like this?
bootchart is not completed.
Quoted 8 lines
printk timestamp come after mem get initialized.
....[ 0.004000] tsc: Fast TSC calibration using PITbefore that stamp are all 0.
Yinghai
On UV the system console function has an option to include timestamps, both sequential in HH:MM:SS and deltas to 100ms. So we get both the BIOS and system times before printk time is active. We also have a custom "script" command that adds timing info.
Quoted 7 lines · Yinghai Lu
Quoted 4 lines · Nathan Zimmer
This rfc patch set delays initializing large sections of memory until we have started cpus. This has the effect of reducing startup times on large memory systems. On 16TB it can take over an hour to boot and most of that time is spent initializing memory.
One hour on system with 16T ram? BIOS or OS?
The BIOS is about 20 minutes *before* the 1+ hour. (When we started this UV project way back when, 8TB took over 4 hours before we threw in the towel.)
Quoted 3 lines
I use wall clock to check bootime on one system with 3T and 16 pcie cards, Linus only takes about 3m and 30 seconds from bootloader.
I can send some stats on where various delays are but most of it was in memory initialization. On average UV nodes carry 128 or 256G per node, so 12 nodes would take about 3 or 4 minutes, perhaps more.
Quoted 2 lines
wonder if you boot delay is with so many cpu get onlined in serialized mode.
Nope. If that was the case, delaying memory but initializing all the cpus would not affect the time.
Quoted 3 lines
so can you try boot your system with "maxcpus=128" to get the boot time with wall clock ?
We could try if you really think it will provide any useful info. Not sure exactly how having memory on nodes with no active cpus will react.
Quoted 5 lines
Thanks
Yinghai