sched_domains + NUMA issue
Hi,
We are seeing errors in the sched domains debug code when SMT + NUMA is enabled. Nathan pointed out that the recent change to limit the number of nodes in a scheduling group may be causing this - in particular sched_domain_node_span.
It looks like ia64 are the only ones implementing a reasonable node_distance, the others just do:
#define node_distance(from,to) (from != to)
On these architectures I wonder if we should disable the sched_domain_node_span code since we will just get a random grouping of cpus.
Anton
CPU0: online
domain 0: span 00000000,00000000,00000000,00000003
groups: 00000000,00000000,00000000,00000001 00000000,00000000,00000000,00000002 domain 1: span 00000000,00000000,00000000,00000003
groups: 00000000,00000000,00000000,00000003
domain 2: span 00000000,00000000,00000000,000f0003
groups: 00000000,00000000,00000000,00000003 00000000,00000000,00000000,000f0000
CPU1: online
domain 0: span 00000000,00000000,00000000,00000003
groups: 00000000,00000000,00000000,00000002 00000000,00000000,00000000,00000001 domain 1: span 00000000,00000000,00000000,00000003
groups: 00000000,00000000,00000000,00000003
ERROR parent span is not a superset of domain->span
domain 2: span 00000000,00000000,00000000,000f0000
ERROR domain->span does not contain CPU1
groups: 00000000,00000000,00000000,00000003 00000000,00000000,00000000,000f0000
ERROR groups don't span domain->span
CPU16: online
domain 0: span 00000000,00000000,00000000,00030000
groups: 00000000,00000000,00000000,00010000 00000000,00000000,00000000,00020000 domain 1: span 00000000,00000000,00000000,000f0000
groups: 00000000,00000000,00000000,00030000 00000000,00000000,00000000,000c0000 domain 2: span 00000000,00000000,00000000,000f0003
groups: 00000000,00000000,00000000,000f0000 00000000,00000000,00000000,00000003
CPU17: online
domain 0: span 00000000,00000000,00000000,00030000
groups: 00000000,00000000,00000000,00020000 00000000,00000000,00000000,00010000 domain 1: span 00000000,00000000,00000000,000f0000
groups: 00000000,00000000,00000000,00030000 00000000,00000000,00000000,000c0000 domain 2: span 00000000,00000000,00000000,000f0000
groups: 00000000,00000000,00000000,000f0000 00000000,00000000,00000000,00000003
ERROR groups don't span domain->span
CPU18: online
domain 0: span 00000000,00000000,00000000,000c0000
groups: 00000000,00000000,00000000,00040000 00000000,00000000,00000000,00080000 domain 1: span 00000000,00000000,00000000,000f0000
groups: 00000000,00000000,00000000,000c0000 00000000,00000000,00000000,00030000
ERROR parent span is not a superset of domain->span
domain 2: span 00000000,00000000,00000000,00000000
ERROR domain->span does not contain CPU18
groups: 00000000,00000000,00000000,000f0000 00000000,00000000,00000000,00000003
ERROR groups don't span domain->span
CPU19: online
domain 0: span 00000000,00000000,00000000,000c0000
groups: 00000000,00000000,00000000,00080000 00000000,00000000,00000000,00040000 domain 1: span 00000000,00000000,00000000,000f0000
groups: 00000000,00000000,00000000,000c0000 00000000,00000000,00000000,00030000
ERROR parent span is not a superset of domain->span
domain 2: span 00000000,00000000,00000000,00000000
ERROR domain->span does not contain CPU19
groups: 00000000,00000000,00000000,000f0000 00000000,00000000,00000000,00000003
ERROR groups don't span domain->span
Quoted 7 lines · Anton Blanchard
Hi,
We are seeing errors in the sched domains debug code when SMT + NUMA is enabled. Nathan pointed out that the recent change to limit the number of nodes in a scheduling group may be causing this - in particular
sched_domain_node_span.
Quoted 8 lines
It looks like ia64 are the only ones implementing a reasonable
node_distance, the others just do:#define
node_distance(from,to)(from != to)On these architectures I wonder if we should disable the
sched_domain_node_spancode since we will just get a random grouping of cpus.
Hmm... for now that's probably a good idea. There's no CONFIG_NUMA_* value we could key off of to figure out if node_distance is sane, so it's probably our only option.
Jesse
Quoted 10 lines · Anton Blanchard
We are seeing errors in the sched domains debug code when SMT + NUMA is enabled. Nathan pointed out that the recent change to limit the number of nodes in a scheduling group may be causing this - in particular
sched_domain_node_span.
It looks like ia64 are the only ones implementing a reasonablenode_distance, the others just do:
#definenode_distance(from,to)(from != to)
On these architectures I wonder if we should disable thesched_domain_node_spancode since we will just get a random grouping of cpus.
For fsck's sake... macro writers need to exercise more discipline.
include/linux/topology.h +1 -1
@@ -55,7 +55,7 @@ for (node = 0; node < numnodes; node = __next_node_with_cpus(node))#ifndef node_distance -#define node_distance (from,to) (from != to) +#define node_distance (from,to) ((from) != (to))#endif #ifndef PENALTY_FOR_NODE_WITH_CPUS #define PENALTY_FOR_NODE_WITH_CPUS (1)
include/asm-ia64/numa.h +1 -1
@@ -59,7 +59,7 @@*/ extern u8 numa_slit[MAX_NUMNODES *MAX_NUMNODES ]; -#define node_distance (from,to) (numa_slit[from * numnodes + to]) +#define node_distance (from,to) (numa_slit[(from) * numnodes + (to)])extern int paddr_to_nid (unsigned long paddr);
include/asm-i386/topology.h +1 -1
@@ -67,7 +67,7 @@ }/* Node-to-Node distance */ -#define node_distance (from, to) (from != to) +#define node_distance (from, to) ((from) != (to))/* Cross-node load balancing interval. */ #define NODE_BALANCE_RATE 100
NUMA for abstract bus-masters (was Re: sched_domains + NUMA issue)
Hi
I recently raised a discussion on the ARM-kernel ml, so, I just want to check if it would raise any interest here.
I was thinking about architectures, where multiple memory (RAM) pools exist on different buses, and various bus-masters on the system have different distances to various RAMs. Say, distance can be defined as the number of bridges to cross, if that RAM is at all accessible, or infinity otherwise.
The API would on one hand allow to register such RAM pools at different locations, and on the other hand, requests for RAM with a device pointer ([dma|pci]_alloc_*) would try to find the nearest RAM available.
And one would have to optimise those allocated buffers for n (typically 2) bus-masters (e.g., CPU and a device) with various weights...
And the idea would be to re-use (some of) the NUMA framework.
This would be another approach to tackle problems, addressed by the James' dma_declare_coherent_memory patch.
So, does this at all sound reasonable? Anybody finds it useful?
Thanks
Guennadi
---
Guennadi Liakhovetski