Skip to contentSkip to conversation

mm/page_alloc: fix MEMF_no_dma allocations for single NUMA

PATCH Superseded Sergey Dyasli 2 participants 3 messages Jan 7, 2019 – Jan 8, 2019

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mm/page_alloc: fix MEMF_no_dma allocations for single NUMA

Currently dma_bitsize is zero by default on single NUMA node machines. This makes all alloc_domheap_pages() calls with MEMF_no_dma return NULL.

There is only 1 user of MEMF_no_dma: dom0_memflags, which are used during memory allocation for Dom0. Failing allocation with default dom0_memflags is especially severe for the PV Dom0 case: it makes alloc_chunk() to use suboptimal 2MB allocation algorithm with a search for higher memory addresses.

This can lead to the NMI watchdog timeout during PV Dom0 construction on some machines, which can be worked around by specifying "dma_bits" in Xen's cmdline manually.

Fix the issue by initialising dma_bitsize even on single NUMA machines.

---
CC: Andrew Cooper <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Julien Grall <[email protected]>
CC: Wei Liu <[email protected]>
CC: Boris Ostrovsky <[email protected]>
CC: George Dunlap <[email protected]>
CC: Roger Pau Monné <[email protected]>

---
 xen/common/page_alloc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
xen/common/page_alloc.c +1 -1
index e591601f9c..4515282c27 100644
@@ -1863,7 +1863,7 @@ void __init end_boot_allocator(void)
     nr_bootmem_regions = 0;
     init_heap_pages(virt_to_page(bootmem_region_list), 1);
 
-    if ( !dma_bitsize && (num_online_nodes() > 1) )
+    if ( !dma_bitsize )
         dma_bitsize = arch_get_dma_bitsize();
 
     printk("Domain heap initialised");
Signature · 7 lines
2.17.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw
Quoted 10 lines
Quoted 1 line
Quoted 1 line · On 07.01.19 at 12:27, <[email protected]> wrote:

Currently dma_bitsize is zero by default on single NUMA node machines. This makes all alloc_domheap_pages() calls with MEMF_no_dma return NULL.

There is only 1 user of MEMF_no_dma: dom0_memflags, which are used during memory allocation for Dom0. Failing allocation with default dom0_memflags is especially severe for the PV Dom0 case: it makes alloc_chunk() to use suboptimal 2MB allocation algorithm with a search for higher memory addresses.

Quoted 5 lines

This can lead to the NMI watchdog timeout during PV Dom0 construction on some machines, which can be worked around by specifying "dma_bits" in Xen's cmdline manually.

Fix the issue by initialising dma_bitsize even on single NUMA machines.

I've not yet looked at why exactly this was done for multi-node systems only, but in any event this change renders somewhat stale the comment next to the dma_bitsize definition.

xen/common/page_alloc.c void __init end_boot_allocator(void) · line 1867

Quoted 9 lines
xen/common/page_alloc.c +1 -1
@@ -1863,7 +1863,7 @@ void __init end_boot_allocator(void)
     nr_bootmem_regions = 0;
     init_heap_pages(virt_to_page(bootmem_region_list), 1);
 
-    if ( !dma_bitsize && (num_online_nodes() > 1) )
+    if ( !dma_bitsize )
         dma_bitsize = arch_get_dma_bitsize();

Did you consider the alternative of leaving this code alone and instead doing

if ( !dma_bitsize )
memflags &= ~MEMF_no_dma;
else if ( (dma_zone = bits_to_zone(dma_bitsize)) < zone_hi )
pg = alloc_heap_pages(dma_zone + 1, zone_hi, order, memflags, d);

in alloc_domheap_pages(), which would also address the same issue in the case of arch_get_dma_bitsize() returning zero?

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#2 lore raw
Quoted 33 lines · Jan Beulich
Quoted 10 lines
Quoted 1 line
Quoted 1 line · On 07.01.19 at 12:27, <[email protected]> wrote:

Currently dma_bitsize is zero by default on single NUMA node machines. This makes all alloc_domheap_pages() calls with MEMF_no_dma return NULL.

There is only 1 user of MEMF_no_dma: dom0_memflags, which are used during memory allocation for Dom0. Failing allocation with default dom0_memflags is especially severe for the PV Dom0 case: it makes alloc_chunk() to use suboptimal 2MB allocation algorithm with a search for higher memory addresses.

Quoted 5 lines

This can lead to the NMI watchdog timeout during PV Dom0 construction on some machines, which can be worked around by specifying "dma_bits" in Xen's cmdline manually.

Fix the issue by initialising dma_bitsize even on single NUMA machines.

I've not yet looked at why exactly this was done for multi-node systems only, but in any event this change renders somewhat stale the comment next to the dma_bitsize definition.

xen/common/page_alloc.c void __init end_boot_allocator(void) · line 1867

Quoted 9 lines
xen/common/page_alloc.c +1 -1
@@ -1863,7 +1863,7 @@ void __init end_boot_allocator(void)
     nr_bootmem_regions = 0;
     init_heap_pages(virt_to_page(bootmem_region_list), 1);
 
-    if ( !dma_bitsize && (num_online_nodes() > 1) )
+    if ( !dma_bitsize )
         dma_bitsize = arch_get_dma_bitsize();

Did you consider the alternative of leaving this code alone and instead doing

Quoted 7 lines

if ( !dma_bitsize )
memflags &= ~MEMF_no_dma;
else if ( (dma_zone = bits_to_zone(dma_bitsize)) < zone_hi )
pg = alloc_heap_pages(dma_zone + 1, zone_hi, order, memflags, d);

in alloc_domheap_pages(), which would also address the same issue in the case of arch_get_dma_bitsize() returning zero?

I like this suggestion. There is no point in MEMF_no_dma if dma_zone doesn't exist, so the flag can be ignored. The allocator should first use higher memory addresses for memory allocation requests, and DMA (<4G) memory shouldn't end up allocated for Dom0 RAM (given there is enough mem >4G).

I'll prepare a v2 soon.

Thanks,
Sergey

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel