Skip to contentSkip to conversation

sched: remove unnecessary sched domains

PATCH In review Siddha, Suresh B 2 participants 4 messages Apr 2, 2005

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 sched: remove unnecessary sched domains

Appended patch removes the unnecessary scheduler domains(containing only one sched group) setup during the sched-domain init.

For example on x86_64, we always have NUMA configured in. On Intel EM64T systems, top most sched domain will be of NUMA and with only one sched_group in it.

With fork/exec balances(recent Nick's fixes in -mm tree), we always endup taking wrong decisions because of this topmost domain (as it contains only one group and find_idlest_group always returns NULL). We will endup loading HT package completely first, letting active load balance kickin and correct it.

In general, this patch also makes sense with out recent Nick's fixes in -mm.

kernel/sched.c +15 -0
@@ -4748,6 +4748,21 @@
 	unsigned long flags;
 	runqueue_t *rq = cpu_rq(cpu);
 	int local = 1;
+	struct sched_domain *tmp = sd, *tmp1;
+
+	/* Remove the sched domains which has only one group
+	 */
+	while (tmp) {
+		tmp1 = tmp->parent;
+		if (!tmp1)
+			break;
+		if (tmp1->groups == tmp1->groups->next)
+			tmp->parent = tmp1->parent;
+		tmp = tmp->parent;
+	}
+
+	if (sd->parent && sd->groups && sd->groups == sd->groups->next)
+		sd = sd->parent;
 
 	sched_domain_debug(sd, cpu);
 
#1 lore raw
Quoted 12 lines · Suresh B

Appended patch removes the unnecessary scheduler domains(containing only one sched group) setup during the sched-domain init.

For example on x86_64, we always have NUMA configured in. On Intel EM64T systems, top most sched domain will be of NUMA and with only one sched_group in it.

With fork/exec balances(recent Nick's fixes in -mm tree), we always endup taking wrong decisions because of this topmost domain (as it contains only one group and find_idlest_group always returns NULL). We will endup loading HT package completely first, letting active load balance kickin and correct it.

In general, this patch also makes sense with out recent Nick's fixes in -mm.

Yeah, this makes sense. We may want to add some other criteria on the removal of a domain as well (because some of the domain flags do things that don't use groups).

I don't like so much that we'd rely on it to fix the above problem. There are a general class of problems with the fork/exec balancing in that it only works on the top most domain, so it may not spread load over lower domains very well.

I was thinking we could fix that by running balance on fork/exec multiple times from top to bottom level domains. I'll have to measure the cost of doing that, because it may be worthwhile.

Thanks

Signature · 1 line
SUSE Labs, Novell Inc.
#2 lore raw
Quoted 21 lines · Nick Piggin
Quoted 7 lines · Suresh B

Appended patch removes the unnecessary scheduler domains(containing only one sched group) setup during the sched-domain init.

For example on x86_64, we always have NUMA configured in. On Intel EM64T systems, top most sched domain will be of NUMA and with only one sched_group in it.

Quoted 8 lines

With fork/exec balances(recent Nick's fixes in -mm tree), we always endup taking wrong decisions because of this topmost domain (as it contains only one group and find_idlest_group always returns NULL). We will endup loading HT package completely first, letting active load balance kickin and correct it.

In general, this patch also makes sense with out recent Nick's fixes in -mm.

Yeah, this makes sense. We may want to add some other criteria on the removal of a domain as well (because some of the domain flags do things that don't use groups).

Quoted 8 lines

I don't like so much that we'd rely on it to fix the above problem. There are a general class of problems with the fork/exec balancing in that it only works on the top most domain, so it may not spread load over lower domains very well.

I was thinking we could fix that by running balance on fork/exec multiple times from top to bottom level domains. I'll have to measure the cost of doing that, because it may be worthwhile.

Agreed.

BTW, why are we setting SD_BALANCE_FORK flag for NUMA domain on i386, ia64. This should be set only on x86_64 and that too not for Intel systems.

thanks,
suresh

#3 lore raw
Quoted 7 lines · Suresh B
Quoted 3 lines · Nick Piggin

I was thinking we could fix that by running balance on fork/exec multiple times from top to bottom level domains. I'll have to measure the cost of doing that, because it may be worthwhile.

Quoted 5 lines

Agreed.

BTW, why are we setting SD_BALANCE_FORK flag for NUMA domain on i386, ia64. This should be set only on x86_64 and that too not for Intel systems.

Mainly to see whether anyone reports regressions, and to exercise the code a bit.

Signature · 1 line
SUSE Labs, Novell Inc.