QEMU NUMA and memory allocation problem

Qemu-devel Wanlong Gao 4 participants 8 messages May 16, 2013 – May 20, 2013 Memory management

lore

Share & tools

lore

Wanlong Gao

author

2013-05-16 09:51 UTC #0 lore raw

Hi,

We just met a problem of QEMU memory allocation.
Here is the description:

On my host, I have two nodes,

# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2
node 0 size: 4010 MB
node 0 free: 3021 MB
node 1 cpus: 1 3
node 1 size: 4030 MB
node 1 free: 2881 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10

I created a guest using the following XML:

...
<memory unit='KiB'>1048576</memory> <currentMemory unit='KiB'>1048576</currentMemory> <vcpu placement='static'>2</vcpu> <cputune>

    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='3'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0-1'/>
  </numatune>
  <cpu>
    <topology sockets='2' cores='1' threads='1'/>
    <numa>
      <cell cpus='0' memory='524288'/>
      <cell cpus='1' memory='524288'/>
    </numa>

</cpu>
...

As you can see, I assigned 1G memory to this guest, pined vcpu0 to the host CPU 2, it's in host node0, pined vcpu1 to the host CPU 3 that is in host node1. The guest also has two nodes, each node contains 512M memory.

Now, I started the guest, then printed the host numa state :

# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2
node 0 size: 4010 MB
node 0 free: 2647 MB  <=== freecell of node0
node 1 cpus: 1 3
node 1 size: 4030 MB
node 1 free: 2746 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10

Then I tried to allocate memory from guest node0 using the following code:

Quoted 5 lines

#include <memory.h>
#include <numa.h>
#define MEM (1024*1024*300)

Quoted 8 lines

int main(void)

{
	char *p = numa_alloc_onnode(MEM, 0);
	memset(p, 0, MEM);
	sleep(1000);
	numa_free(p, MEM);
	return 0;
}

And printed the host numa state, it shows that this 300M memory is allocated from host node0,

# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2
node 0 size: 4010 MB
node 0 free: 2345 MB	<===== reduced ~300M
node 1 cpus: 1 3
node 1 size: 4030 MB
node 1 free: 2767 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10

Then, I tried the same method to allocate 300M memory from guest node1, and printed the host numa state:

# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2
node 0 size: 4010 MB
node 0 free: 2059 MB	<=== reduced ~300M
node 1 cpus: 1 3
node 1 size: 4030 MB
node 1 free: 2767 MB	<=== no change
node distances:
node   0   1 
  0:  10  20 
  1:  20  10

To see that this 300M memory is allocated from host node0 again, but not host node1 as I expected.

We think that QEMU can't handled this numa memory allocation well, and it will cause the cross node memory access performance regression.

Any thoughts? Or, am I missing something?

Thanks,
Wanlong Gao

Paolo Bonzini

2013-05-16 12:02 UTC #1 lore raw

Il 16/05/2013 11:50, Wanlong Gao ha scritto:

Quoted 7 lines

To see that this 300M memory is allocated from host node0 again, but not host node1 as I expected.
We think that QEMU can't handled this numa memory allocation well, and it will cause the cross node memory access performance regression.
Any thoughts? Or, am I missing something?

Pinning memory to host NUMA nodes is not implemented. Something like AutoNUMA would be able to balance the memory the right way.

Paolo

Wanlong Gao

author

2013-05-17 07:48 UTC #2 lore raw

Quoted 11 lines · Paolo Bonzini

Il 16/05/2013 11:50, Wanlong Gao ha scritto:
Quoted 7 lines
To see that this 300M memory is allocated from host node0 again, but not host node1 as I expected.
We think that QEMU can't handled this numa memory allocation well, and it will cause the cross node memory access performance regression.
Any thoughts? Or, am I missing something?
Pinning memory to host NUMA nodes is not implemented. Something like AutoNUMA would be able to balance the memory the right way.

Any plan to implement this? Or, handle this by QEMU is not necessary? I just enabled AutoNUMA on the host kernel, but the memory allocation numbers seems no change at all. Am I missing something?

Thanks,
Wanlong Gao

Quoted 4 lines

Paolo

Eduardo Habkost

2013-05-17 13:37 UTC #3 lore raw

Quoted 8 lines · Wanlong Gao

Quoted 7 lines · Paolo Bonzini
Il 16/05/2013 11:50, Wanlong Gao ha scritto:
Quoted 6 lines
To see that this 300M memory is allocated from host node0 again, but not host node1 as I expected.
We think that QEMU can't handled this numa memory allocation well, and it will cause the cross node memory access performance regression.

Quoted 8 lines

Quoted 4 lines
Quoted 1 line
Any thoughts? Or, am I missing something?
Pinning memory to host NUMA nodes is not implemented. Something like AutoNUMA would be able to balance the memory the right way.
Any plan to implement this? Or, handle this by QEMU is not necessary? I just enabled AutoNUMA on the host kernel, but the memory allocation numbers seems no change at all. Am I missing something?

I had plans to implement a mechanism to allow external tools to implement manual pinning, but it is not one of my top priorities. It's the kind of mechanism that may be obsolete since birth, if we have AutoNUMA working and doing the right thing.

Signature · 1 line

Eduardo

Wanlong Gao

author

2013-05-20 02:04 UTC #4 lore raw

In reply to Wanlong Gao

Adding CC AutoNUMA folks:

Paolo said that:

Quoted 4 lines

Pinning memory to host NUMA nodes is not implemented. Something like AutoNUMA would be able to balance the memory the right way.
Paolo

And Eduardo said that:

Quoted 6 lines

I had plans to implement a mechanism to allow external tools to implement manual pinning, but it is not one of my top priorities. It's the kind of mechanism that may be obsolete since birth, if we have AutoNUMA working and doing the right thing.
-- Eduardo

But I didn't see any change when I enabled the AutoNUMA on my host. Can AutoNUMA folks teach me why?
Or any plans to handle this problem in AutoNUMA?

Thanks,
Wanlong Gao

Quoted 112 lines

Hi,
We just met a problem of QEMU memory allocation.
Here is the description:
On my host, I have two nodes,
# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2
node 0 size: 4010 MB
node 0 free: 3021 MB
node 1 cpus: 1 3
node 1 size: 4030 MB
node 1 free: 2881 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 
I created a guest using the following XML:
...
<memory unit='KiB'>1048576</memory> <currentMemory unit='KiB'>1048576</currentMemory> <vcpu placement='static'>2</vcpu> <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='3'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0-1'/>
  </numatune>
  <cpu>
    <topology sockets='2' cores='1' threads='1'/>
    <numa>
      <cell cpus='0' memory='524288'/>
      <cell cpus='1' memory='524288'/>
    </numa>
</cpu>
...
As you can see, I assigned 1G memory to this guest, pined vcpu0 to the host CPU 2, it's in host node0, pined vcpu1 to the host CPU 3 that is in host node1. The guest also has two nodes, each node contains 512M memory.
Now, I started the guest, then printed the host numa state :
# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2
node 0 size: 4010 MB
node 0 free: 2647 MB  <=== freecell of node0
node 1 cpus: 1 3
node 1 size: 4030 MB
node 1 free: 2746 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 
Then I tried to allocate memory from guest node0 using the following code:
Quoted 5 lines
#include <memory.h>
#include <numa.h>
#define MEM (1024*1024*300)
Quoted 8 lines
int main(void)
{
	char *p = numa_alloc_onnode(MEM, 0);
	memset(p, 0, MEM);
	sleep(1000);
	numa_free(p, MEM);
	return 0;
}
And printed the host numa state, it shows that this 300M memory is allocated from host node0,
# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2
node 0 size: 4010 MB
node 0 free: 2345 MB	<===== reduced ~300M
node 1 cpus: 1 3
node 1 size: 4030 MB
node 1 free: 2767 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 
Then, I tried the same method to allocate 300M memory from guest node1, and printed the host numa state:
# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2
node 0 size: 4010 MB
node 0 free: 2059 MB	<=== reduced ~300M
node 1 cpus: 1 3
node 1 size: 4030 MB
node 1 free: 2767 MB	<=== no change
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 
To see that this 300M memory is allocated from host node0 again, but not host node1 as I expected.

Quoted 10 lines

We think that QEMU can't handled this numa memory allocation well, and it will cause the cross node memory access performance regression.
Any thoughts? Or, am I missing something?
Thanks,
Wanlong Gao

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Wanpeng Li

2013-05-20 03:03 UTC #5 lore raw

Quoted 17 lines · Wanlong Gao

Adding CC AutoNUMA folks:
Paolo said that:
Quoted 4 lines
Pinning memory to host NUMA nodes is not implemented. Something like AutoNUMA would be able to balance the memory the right way.
Paolo
And Eduardo said that:
Quoted 6 lines
I had plans to implement a mechanism to allow external tools to implement manual pinning, but it is not one of my top priorities. It's the kind of mechanism that may be obsolete since birth, if we have AutoNUMA working and doing the right thing.
-- Eduardo

Hi Wanlong,

Quoted 4 lines

But I didn't see any change when I enabled the AutoNUMA on my host. Can AutoNUMA folks teach me why?
Or any plans to handle this problem in AutoNUMA?

AutoNUMA is not merged currently, the foundation(automatic NUMA balancing) that either the policy for schednuma or autonuma can be rebased on implemented by Mel has already merged.

Regards,
Wanpeng Li

Quoted 134 lines

Thanks,
Wanlong Gao
Quoted 112 lines
Hi,
We just met a problem of QEMU memory allocation.
Here is the description:
On my host, I have two nodes,
# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2
node 0 size: 4010 MB
node 0 free: 3021 MB
node 1 cpus: 1 3
node 1 size: 4030 MB
node 1 free: 2881 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 
I created a guest using the following XML:
...
<memory unit='KiB'>1048576</memory> <currentMemory unit='KiB'>1048576</currentMemory> <vcpu placement='static'>2</vcpu> <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='3'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0-1'/>
  </numatune>
  <cpu>
    <topology sockets='2' cores='1' threads='1'/>
    <numa>
      <cell cpus='0' memory='524288'/>
      <cell cpus='1' memory='524288'/>
    </numa>
</cpu>
...
As you can see, I assigned 1G memory to this guest, pined vcpu0 to the host CPU 2, it's in host node0, pined vcpu1 to the host CPU 3 that is in host node1. The guest also has two nodes, each node contains 512M memory.
Now, I started the guest, then printed the host numa state :
# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2
node 0 size: 4010 MB
node 0 free: 2647 MB  <=== freecell of node0
node 1 cpus: 1 3
node 1 size: 4030 MB
node 1 free: 2746 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 
Then I tried to allocate memory from guest node0 using the following code:
Quoted 5 lines
#include <memory.h>
#include <numa.h>
#define MEM (1024*1024*300)
Quoted 8 lines
int main(void)
{
	char *p = numa_alloc_onnode(MEM, 0);
	memset(p, 0, MEM);
	sleep(1000);
	numa_free(p, MEM);
	return 0;
}
And printed the host numa state, it shows that this 300M memory is allocated from host node0,
# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2
node 0 size: 4010 MB
node 0 free: 2345 MB	<===== reduced ~300M
node 1 cpus: 1 3
node 1 size: 4030 MB
node 1 free: 2767 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 
Then, I tried the same method to allocate 300M memory from guest node1, and printed the host numa state:
# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2
node 0 size: 4010 MB
node 0 free: 2059 MB	<=== reduced ~300M
node 1 cpus: 1 3
node 1 size: 4030 MB
node 1 free: 2767 MB	<=== no change
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 
To see that this 300M memory is allocated from host node0 again, but not host node1 as I expected.
Quoted 10 lines
We think that QEMU can't handled this numa memory allocation well, and it will cause the cross node memory access performance regression.
Any thoughts? Or, am I missing something?
Thanks,
Wanlong Gao
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Wanpeng Li

2013-05-20 03:03 UTC #6 lore raw

In reply to Wanlong Gao

Quoted 17 lines · Wanlong Gao

Adding CC AutoNUMA folks:
Paolo said that:
Quoted 4 lines
Pinning memory to host NUMA nodes is not implemented. Something like AutoNUMA would be able to balance the memory the right way.
Paolo
And Eduardo said that:
Quoted 6 lines
I had plans to implement a mechanism to allow external tools to implement manual pinning, but it is not one of my top priorities. It's the kind of mechanism that may be obsolete since birth, if we have AutoNUMA working and doing the right thing.
-- Eduardo

Hi Wanlong,

Quoted 4 lines

But I didn't see any change when I enabled the AutoNUMA on my host. Can AutoNUMA folks teach me why?
Or any plans to handle this problem in AutoNUMA?

AutoNUMA is not merged currently, the foundation(automatic NUMA balancing) that either the policy for schednuma or autonuma can be rebased on implemented by Mel has already merged.

Regards,
Wanpeng Li

Quoted 129 lines

Thanks,
Wanlong Gao
Quoted 112 lines
Hi,
We just met a problem of QEMU memory allocation.
Here is the description:
On my host, I have two nodes,
# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2
node 0 size: 4010 MB
node 0 free: 3021 MB
node 1 cpus: 1 3
node 1 size: 4030 MB
node 1 free: 2881 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 
I created a guest using the following XML:
...
<memory unit='KiB'>1048576</memory> <currentMemory unit='KiB'>1048576</currentMemory> <vcpu placement='static'>2</vcpu> <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='3'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0-1'/>
  </numatune>
  <cpu>
    <topology sockets='2' cores='1' threads='1'/>
    <numa>
      <cell cpus='0' memory='524288'/>
      <cell cpus='1' memory='524288'/>
    </numa>
</cpu>
...
As you can see, I assigned 1G memory to this guest, pined vcpu0 to the host CPU 2, it's in host node0, pined vcpu1 to the host CPU 3 that is in host node1. The guest also has two nodes, each node contains 512M memory.
Now, I started the guest, then printed the host numa state :
# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2
node 0 size: 4010 MB
node 0 free: 2647 MB  <=== freecell of node0
node 1 cpus: 1 3
node 1 size: 4030 MB
node 1 free: 2746 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 
Then I tried to allocate memory from guest node0 using the following code:
Quoted 5 lines
#include <memory.h>
#include <numa.h>
#define MEM (1024*1024*300)
Quoted 8 lines
int main(void)
{
	char *p = numa_alloc_onnode(MEM, 0);
	memset(p, 0, MEM);
	sleep(1000);
	numa_free(p, MEM);
	return 0;
}
And printed the host numa state, it shows that this 300M memory is allocated from host node0,
# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2
node 0 size: 4010 MB
node 0 free: 2345 MB	<===== reduced ~300M
node 1 cpus: 1 3
node 1 size: 4030 MB
node 1 free: 2767 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 
Then, I tried the same method to allocate 300M memory from guest node1, and printed the host numa state:
# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2
node 0 size: 4010 MB
node 0 free: 2059 MB	<=== reduced ~300M
node 1 cpus: 1 3
node 1 size: 4030 MB
node 1 free: 2767 MB	<=== no change
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 
To see that this 300M memory is allocated from host node0 again, but not host node1 as I expected.
Quoted 10 lines
We think that QEMU can't handled this numa memory allocation well, and it will cause the cross node memory access performance regression.
Any thoughts? Or, am I missing something?
Thanks,
Wanlong Gao

Quoted 5 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Wanpeng Li

2013-05-20 03:03 UTC #7 lore raw

In reply to Wanlong Gao

Quoted 17 lines · Wanlong Gao

Adding CC AutoNUMA folks:
Paolo said that:
Quoted 4 lines
Pinning memory to host NUMA nodes is not implemented. Something like AutoNUMA would be able to balance the memory the right way.
Paolo
And Eduardo said that:
Quoted 6 lines
I had plans to implement a mechanism to allow external tools to implement manual pinning, but it is not one of my top priorities. It's the kind of mechanism that may be obsolete since birth, if we have AutoNUMA working and doing the right thing.
-- Eduardo

Hi Wanlong,

Quoted 4 lines

But I didn't see any change when I enabled the AutoNUMA on my host. Can AutoNUMA folks teach me why?
Or any plans to handle this problem in AutoNUMA?

AutoNUMA is not merged currently, the foundation(automatic NUMA balancing) that either the policy for schednuma or autonuma can be rebased on implemented by Mel has already merged.

Regards,
Wanpeng Li

Quoted 129 lines

Thanks,
Wanlong Gao
Quoted 112 lines
Hi,
We just met a problem of QEMU memory allocation.
Here is the description:
On my host, I have two nodes,
# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2
node 0 size: 4010 MB
node 0 free: 3021 MB
node 1 cpus: 1 3
node 1 size: 4030 MB
node 1 free: 2881 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 
I created a guest using the following XML:
...
<memory unit='KiB'>1048576</memory> <currentMemory unit='KiB'>1048576</currentMemory> <vcpu placement='static'>2</vcpu> <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='3'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0-1'/>
  </numatune>
  <cpu>
    <topology sockets='2' cores='1' threads='1'/>
    <numa>
      <cell cpus='0' memory='524288'/>
      <cell cpus='1' memory='524288'/>
    </numa>
</cpu>
...
As you can see, I assigned 1G memory to this guest, pined vcpu0 to the host CPU 2, it's in host node0, pined vcpu1 to the host CPU 3 that is in host node1. The guest also has two nodes, each node contains 512M memory.
Now, I started the guest, then printed the host numa state :
# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2
node 0 size: 4010 MB
node 0 free: 2647 MB  <=== freecell of node0
node 1 cpus: 1 3
node 1 size: 4030 MB
node 1 free: 2746 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 
Then I tried to allocate memory from guest node0 using the following code:
Quoted 5 lines
#include <memory.h>
#include <numa.h>
#define MEM (1024*1024*300)
Quoted 8 lines
int main(void)
{
	char *p = numa_alloc_onnode(MEM, 0);
	memset(p, 0, MEM);
	sleep(1000);
	numa_free(p, MEM);
	return 0;
}
And printed the host numa state, it shows that this 300M memory is allocated from host node0,
# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2
node 0 size: 4010 MB
node 0 free: 2345 MB	<===== reduced ~300M
node 1 cpus: 1 3
node 1 size: 4030 MB
node 1 free: 2767 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 
Then, I tried the same method to allocate 300M memory from guest node1, and printed the host numa state:
# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2
node 0 size: 4010 MB
node 0 free: 2059 MB	<=== reduced ~300M
node 1 cpus: 1 3
node 1 size: 4030 MB
node 1 free: 2767 MB	<=== no change
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 
To see that this 300M memory is allocated from host node0 again, but not host node1 as I expected.
Quoted 10 lines
We think that QEMU can't handled this numa memory allocation well, and it will cause the cross node memory access performance regression.
Any thoughts? Or, am I missing something?
Thanks,
Wanlong Gao

Quoted 5 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>