Can't boot guest with more than 3585MB when using large pages

DISCUSSION Applied Alex Williamson 4 participants 7 messages Mar 24, 2009 – Apr 5, 2009 KVM

Share & tools

Alex Williamson

author

2009-03-24 21:06 UTC #0 lore raw

On a 2.6.29, x86_64 host/guest, what's special about specifying a guest size of -m 3586 when using -mem-path backed by hugetlbfs? 3585 works, 3586 hangs here:

...
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
software IO TLB at phys 0x20000000 - 0x24000000
Memory: 3504832k/4196352k available (2926k kernel code, 524740k absent, 166780k reserved, 1260k data, 496k init)

I can back -mem-path by tmpfs or disk and it works fine. Also works with no -mem-path, but it would obviously be nice to benefit from large pages on big guests. The system has plenty of huge pages to back the request, and booting with -mem-prealloc makes no difference. Tested on latest git as of today. Thanks,

Alex

Ryan Harper

2009-03-24 21:57 UTC #1 lore raw

Quoted 11 lines · Alex Williamson

On a 2.6.29, x86_64 host/guest, what's special about specifying a guest size of -m 3586 when using -mem-path backed by hugetlbfs? 3585 works, 3586 hangs here:
...
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
software IO TLB at phys 0x20000000 - 0x24000000
Memory: 3504832k/4196352k available (2926k kernel code, 524740k absent, 166780k reserved, 1260k data, 496k init)

Quoted 5 lines

I can back -mem-path by tmpfs or disk and it works fine. Also works with no -mem-path, but it would obviously be nice to benefit from large pages on big guests. The system has plenty of huge pages to back the request, and booting with -mem-prealloc makes no difference. Tested on latest git as of today. Thanks,

I've seen this as well, haven't had a chance to dig into the issue yet either. Certainly can test patches if anyone has an idea of what's wrong here.

Signature · 4 lines

Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
[email protected]

Marcelo Tosatti

2009-03-25 16:11 UTC #2 lore raw

Quoted 18 lines · Ryan Harper

Quoted 11 lines · Alex Williamson
On a 2.6.29, x86_64 host/guest, what's special about specifying a guest size of -m 3586 when using -mem-path backed by hugetlbfs? 3585 works, 3586 hangs here:
...
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
software IO TLB at phys 0x20000000 - 0x24000000
Memory: 3504832k/4196352k available (2926k kernel code, 524740k absent, 166780k reserved, 1260k data, 496k init)
Quoted 5 lines
I can back -mem-path by tmpfs or disk and it works fine. Also works with no -mem-path, but it would obviously be nice to benefit from large pages on big guests. The system has plenty of huge pages to back the request, and booting with -mem-prealloc makes no difference. Tested on latest git as of today. Thanks,

Quoted 3 lines

I've seen this as well, haven't had a chance to dig into the issue yet either. Certainly can test patches if anyone has an idea of what's wrong here.

Can you strace and see if the mmap on hugetlbfs is correctly sized?

Alex Williamson

author

2009-03-25 16:26 UTC #3 lore raw

Quoted 13 lines · Marcelo Tosatti

Quoted 12 lines · Ryan Harper
Quoted 10 lines · Alex Williamson
On a 2.6.29, x86_64 host/guest, what's special about specifying a guest size of -m 3586 when using -mem-path backed by hugetlbfs? 3585 works, 3586 hangs here:
...
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
software IO TLB at phys 0x20000000 - 0x24000000
Memory: 3504832k/4196352k available (2926k kernel code, 524740k absent, 166780k reserved, 1260k data, 496k init)

Quoted 5 lines

Quoted 3 lines
I've seen this as well, haven't had a chance to dig into the issue yet either. Certainly can test patches if anyone has an idea of what's wrong here.
Can you strace and see if the mmap on hugetlbfs is correctly sized?

Seems reasonable with some 2MB rounding.

Failing case, -m 3586:

open("/hugepages//kvm.5fuuH5", O_RDWR|O_CREAT|O_EXCL, 0600) = 9
unlink("/hugepages//kvm.5fuuH5") = 0
ftruncate(9, 3783262208) = 0
mmap(NULL, 3783262208, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_POPULATE, 9, 0) = 0x7f37a5e00000

Working case, -m 3585:

open("/hugepages//kvm.Mv6Zgd", O_RDWR|O_CREAT|O_EXCL, 0600) = 9
unlink("/hugepages//kvm.Mv6Zgd") = 0
ftruncate(9, 3781165056) = 0
mmap(NULL, 3781165056, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_POPULATE, 9, 0) = 0x7fd44b800000

Working case using disk backing: -mem-path /tmp -mem-prealloc -m 3586:

open("/tmp/kvm.nPlxl1", O_RDWR|O_CREAT|O_EXCL, 0600) = 9
unlink("/tmp/kvm.nPlxl1") = 0
ftruncate(9, 3783262208) = 0
mmap(NULL, 3783262208, PROT_READ|PROT_WRITE, MAP_PRIVATE, 9, 0) = 0x7f432e055000

Marcelo Tosatti

2009-04-03 23:29 UTC #4 lore raw

In reply to Ryan Harper

Quoted 18 lines · Ryan Harper

Quoted 11 lines · Alex Williamson
On a 2.6.29, x86_64 host/guest, what's special about specifying a guest size of -m 3586 when using -mem-path backed by hugetlbfs? 3585 works, 3586 hangs here:
...
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
software IO TLB at phys 0x20000000 - 0x24000000
Memory: 3504832k/4196352k available (2926k kernel code, 524740k absent, 166780k reserved, 1260k data, 496k init)
Quoted 5 lines
I can back -mem-path by tmpfs or disk and it works fine. Also works with no -mem-path, but it would obviously be nice to benefit from large pages on big guests. The system has plenty of huge pages to back the request, and booting with -mem-prealloc makes no difference. Tested on latest git as of today. Thanks,

Quoted 3 lines

I've seen this as well, haven't had a chance to dig into the issue yet either. Certainly can test patches if anyone has an idea of what's wrong here.

Can you please try the following

------

qemu: kvm: fixup 4GB+ memslot large page alignment

Need to align the 4GB+ memslot after we know its address, not before.

Signed-off-by Marcelo Tosatti <[email protected]>

qemu/hw/pc.c +1 -1

index d4a4320..cc84772 100644
@@ -866,6 +866,7 @@ static void pc_init1(ram_addr_t ram_size, int vga_ram_size,
 
     /* above 4giga memory allocation */
     if (above_4g_mem_size > 0) {
+        ram_addr = qemu_ram_alloc(above_4g_mem_size);
         if (hpagesize) {
             if (ram_addr & (hpagesize-1)) {
                 unsigned long aligned_addr;

@@ -874,7 +875,6 @@ static void pc_init1(ram_addr_t ram_size, int vga_ram_size,
                 ram_addr = aligned_addr;
             }
         }
-        ram_addr = qemu_ram_alloc(above_4g_mem_size);
         cpu_register_physical_memory(0x100000000ULL,
                                      above_4g_mem_size,
                                      ram_addr);

Alex Williamson

author Tested-by

2009-04-04 17:56 UTC #5 lore raw

Quoted 2 lines · Marcelo Tosatti

Can you please try the following

Thanks Marcelo, this seems to fix it. I tested up to a 30G guest with large pages.

Alex

Quoted 7 lines

------
qemu: kvm: fixup 4GB+ memslot large page alignment
Need to align the 4GB+ memslot after we know its address, not before.
Signed-off-by Marcelo Tosatti <[email protected]>

Tested-by Alex Williamson <[email protected]>

Quoted 21 lines

qemu/hw/pc.c +1 -1

index d4a4320..cc84772 100644
@@ -866,6 +866,7 @@ static void pc_init1(ram_addr_t ram_size, int vga_ram_size,
 
     /* above 4giga memory allocation */
     if (above_4g_mem_size > 0) {
+        ram_addr = qemu_ram_alloc(above_4g_mem_size);
         if (hpagesize) {
             if (ram_addr & (hpagesize-1)) {
                 unsigned long aligned_addr;

@@ -874,7 +875,6 @@ static void pc_init1(ram_addr_t ram_size, int vga_ram_size,
                 ram_addr = aligned_addr;
             }
         }
-        ram_addr = qemu_ram_alloc(above_4g_mem_size);
         cpu_register_physical_memory(0x100000000ULL,
                                      above_4g_mem_size,
                                      ram_addr);

Avi Kivity

Applied

2009-04-05 11:54 UTC #6 lore raw

Quoted 8 lines · Alex Williamson

Quoted 2 lines · Marcelo Tosatti
Can you please try the following
Thanks Marcelo, this seems to fix it. I tested up to a 30G guest with large pages.

I've applied the patch, thanks. I keep thinking we need to do additional rounding when we allocate the file.

Signature · 1 line

error compiling committee.c: too many arguments to function