Huge percpu memory usage on multi NUMA node system
Hello,
I use Vmware VM with the following configuration.
- 1 vCPU per 1 NUMA node
- 4 online vCPUs, 128 possible vCPUs. It translates to
- 4 online nodes and 128 possible nodes.
- 192VM memory
Linux 5.15 with CONFIG_NODES_SHIFT=6 complains on node numbers more that maximum supported (1 << 6):
Nov 27 01:59:37 photon-576f8974caf.org kernel: SRAT: PXM 62 -> APIC 0x7c -> Node 62 Nov 27 01:59:37 photon-576f8974caf.org kernel: SRAT: PXM 63 -> APIC 0x7e -> Node 63 Nov 27 01:59:37 photon-576f8974caf.org kernel: SRAT: Too many proximity domains 40 Nov 27 01:59:37 photon-576f8974caf.org kernel: ACPI: SRAT: SRAT not used. Nov 27 01:59:37 photon-576f8974caf.org kernel: No NUMA configuration found
But it boots OK and Percpu memory amount is 53760 kB
If I compile with CONFIG_NODES_SHIFT=10 to support 128 nodes, boot warning disappears, cpu info reports proper numa nodes for existing cpus.
But boot process fails with OOM in pid 1.
Increasing VM RAM from 192 MB to 1024MB fixed OOM. /proc/meminfo reported increase in Percpu to 718048 kB !!
It sounds surprising as number of CPUs are the same in both cases.
Initial analysis showed that each memory cgroup allocates per node structures. Each of them have percpu allocations, doing 128 * 128 * struct size.
See: mem_cgroup_alloc() -> alloc_mem_cgroup_per_node_info()
There is also old comment about it in alloc_mem_cgroup_per_node_info()
/*
* This routine is called against possible nodes.
* But it's BUG to call kmalloc() against offline node.
*
* TODO: this routine can waste much memory for nodes which will
* never be onlined. It's better to use memory hotplug callback
* function.
*/
There are might be other places not efficiently using memory for non existing nodes.
Thanks,
—Alexey
[Cc Johannes, Shakeel and Nico]
Quoted 7 lines · Alexey Makhalov
Hello,
I use Vmware VM with the following configuration.
- 1 vCPU per 1 NUMA node
- 4 online vCPUs, 128 possible vCPUs. It translates to
- 4 online nodes and 128 possible nodes.
- 192VM memory
We have discussed this particular setup in another email thread that is not related to this particular issue but let me just repeat that I conside such a configuration rather surprising and suboptimal. I am not sure what will be the actual topology but a single CPU per NUMA node will have some interesting side effects (e.g. CPU load balancing etc).
Also too many memory&cpu less nodes is not something many kernel subsystems are optimized for. At best we are trying to avoid MAX_NUMNODES scaling and going with the possible nodes. Proper handling of possible nodes without memory requires memory hotplug notifiers and synchronization.
Quoted 23 lines
Linux 5.15 with
CONFIG_NODES_SHIFT=6complains on node numbers more that maximum supported (1 << 6):Nov 27 01:59:37 photon-576f8974caf.org kernel: SRAT: PXM 62 -> APIC 0x7c -> Node 62 Nov 27 01:59:37 photon-576f8974caf.org kernel: SRAT: PXM 63 -> APIC 0x7e -> Node 63 Nov 27 01:59:37 photon-576f8974caf.org kernel: SRAT: Too many proximity domains 40 Nov 27 01:59:37 photon-576f8974caf.org kernel: ACPI: SRAT: SRAT not used. Nov 27 01:59:37 photon-576f8974caf.org kernel: No NUMA configuration foundBut it boots OK and Percpu memory amount is 53760 kB
If I compile with
CONFIG_NODES_SHIFT=10to support 128 nodes, boot warning disappears, cpu info reports proper numa nodes for existing cpus.
But boot process fails with OOM in pid 1.Increasing VM RAM from 192 MB to 1024MB fixed OOM.
/proc/meminforeported increase in Percpu to 718048 kB !!It sounds surprising as number of CPUs are the same in both cases.
Initial analysis showed that each memory cgroup allocates per node structures. Each of them have percpu allocations, doing 128 * 128 * struct size.
See:mem_cgroup_alloc()->alloc_mem_cgroup_per_node_info()
Quoted 10 lines
There is also old comment about it in
alloc_mem_cgroup_per_node_info()
/*
* This routine is called against possible nodes.
* But it's BUG to callkmalloc()against offline node.
*
* TODO: this routine can waste much memory for nodes which will
* never be onlined. It's better to use memory hotplug callback
* function.
*/
There are might be other places not efficiently using memory for non existing nodes.
Yes, another example would be shrinkers: see http://lkml.kernel.org/r/[email protected]
Signature · 2 lines
Michal Hocko SUSE Labs