Skip to content

add NUMA support to Xen

Ryan Harper xen-devel 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest6 patches 3 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/6xen: add ACPI SRAT parser and data structures+0 -00 replies
2/6xen: add per-node bucks to page allocator+0 -00 replies
3/6xen: make memory ops NUMA-aware+0 -00 replies
4/6xen: export NUMA topology in physinfo hcall+0 -03 replies
5/6xen, tools: add nr_cpus field to fix nr_cpus calc+0 -00 replies
6/6tools: add numa stat tool and hcalls+0 -00 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH 0/6] add NUMA support to Xen

The following patchset adds NUMA support to the hypervisor. This includes:

- A full SRAT table parser for 32-bit and 64-bit, based on the ACPI NUMA parser from linux 2.6.16.29 with data structures to represent NUMA cpu and memory topology, and NUMA emulation (fake=).
- Changes to the Xen page allocator adding a per-node bucket for each zone. Xen will continue to prioritize using the requested zone at the cost of using non-local memory.
- Modified xen memory_ops to use NUMA-aware page allocator functions to help allocate node-local memory for guests
- export NUMA topology through physinfo hypercall, integreated into Xend
- fix-up nr_cpus calculation now that nr_nodes is not static
- Add a tool to explore NUMA allocations and Xen heap status.

Nothing has functionally changed since the last posting of the patches; I've just been maintaining the patchset against unstable.

Additionally, I've spent the past few weeks collecting some performance benchmarks comparing with and without the NUMA patches on a system configured as a 4-socket server(no NUMA characteristics) and as a multi-node 16-socket NUMA machine.

Machine: x460 4-node, 3Ghz Xeon PIV, 1MB L2, 8MB L3
16 Socket (4 sockets per node) HT, logical 32-way, 64GB RAM, 16G per node
Xen Changeset used: 11536:041be3f6b38e

We ran sdet and dbench, recording aggregate throughput for 4, 8, 12, and 16domUs. This loads the system with 4domUs on each node, one node, two nodes, etc up to 4 nodes pinning the guest VCPUs to the proper physical cpus in their respective nodes.

To look for possible regressions on non-NUMA systems, we used a single node (4 sockets and 16G) for the first 4domU test as a single node does not have any NUMA characteristics.

In summary, there are no regressions for the single node, non-numa case. As we scale the number of domains out the NUMA patches show significant gains.

dbench(tmpfs) 4domU 8domU 12domU 16domU
-------------------------------------------

with           590    1175   1750    2175
without        590    1090   1405    1450

-------------------------------------------
%gain +0.0% +7.8% +24.5% +50.0%

sdet 4domU 8domU 12domU 16domU
---------------------------------------

with       14000  25900  39500   49000
without    13900  24950  34900   38500

---------------------------------------
%gain +0.8% +3.8% +13.2% +27.0%

I've also been working with Raj at Unisys to test out the NUMA patches on their ES7000. Just today Raj confirmed that the previous "dom0 slowdown" that had been reported has been reproduced without the NUMA patches. I believe there are no outstanding machines that have tested the NUMA patches that have any issues.

I would like to see the NUMA patches included in xen-unstable once it is open for wider testing prior to the next release after 3.0.3.

Signature · 5 lines
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
[email protected]