Skip to contentSkip to conversation

cpuset: add interface to isolated cpus

RFC In review Paul Jackson 4 participants 34 messages Oct 19, 2006 – Oct 25, 2006

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

RFC 1/1 cpuset: add interface to isolated cpus

From: Paul Jackson <[email protected]>

Enable user code to isolate CPUs on a system from the domains that determine scheduler load balancing.

This is already doable using the boot parameter "isolcpus=". The folks running realtime code on production systems, where some nodes are isolated for realtime, and some not, and where it is unacceptable to reboot to adjust this, need to be able to change which CPUs are isolated from the scheduler balancing code on the fly.

This is done by exposing the kernels cpu_isolated_map as a cpumask in the root cpuset directory, in a file called 'isolated_cpus', where it can be read and written by sufficiently privileged user code.

---

I have built and booted this on one system, ia64/sn2, and verified that I can set and query the cpu_isolated_map, and that tasks are not scheduled on such isolated CPUs unless I explicitly place them there.

			-pj
 Documentation/cpusets.txt           |   26 +++++++++++++++++++++++---
 Documentation/kernel-parameters.txt |    4 ++++
 include/linux/sched.h               |    3 +++
 kernel/cpuset.c                     |   31 +++++++++++++++++++++++++++++++
 kernel/sched.c                      |   21 ++++++++++++++++++++-
 5 files changed, 81 insertions(+), 4 deletions(-)
Documentation/cpusets.txt +23 -3
@@ -19,7 +19,8 @@ CONTENTS:
   1.5 What does notify_on_release do ?
   1.6 What is memory_pressure ?
   1.7 What is memory spread ?
-  1.8 How do I use cpusets ?
+  1.8 What is isolated_cpus ?
+  1.9 How do I use cpusets ?
 2. Usage Examples and Syntax
   2.1 Basic Usage
   2.2 Adding/removing cpus
@@ -174,8 +175,9 @@ containing the following files describin
  - notify_on_release flag: run /sbin/cpuset_release_agent on exit?
  - memory_pressure: measure of how much paging pressure in cpuset
 
-In addition, the root cpuset only has the following file:
+In addition, the root cpuset only has the following files:
  - memory_pressure_enabled flag: compute memory_pressure?
+ - isolated_cpus: cpus excluded from scheduler domains
 
 New cpusets are created using the mkdir system call or shell
 command.  The properties of a cpuset, such as its flags, allowed
@@ -378,7 +380,25 @@ data set, the memory allocation across t
 can become very uneven.
 
 
-1.8 How do I use cpusets ?
+1.8 What is isolated_cpus ?
+---------------------------
+
+The cpumask isolated_cpus, visible in the top cpuset, provides direct
+access to the kernel cpu_isolated_map (in kernel/sched.c), which are
+the CPUs which are excluded from scheduler domains.  The scheduler
+will not load balance tasks on isolated cpus.  The 'isolated_cpus'
+file in the root cpuset can be read, to obtain the current setting
+of cpu_isolated_map, and can be written, to write a new cpumask to
+cpu_isolated_map.
+
+The initial value of the cpu_isolated_map can be set by the kernel
+boottime argument "isolcpus=n1,n2,n3,..." where n1, n2, n3, etc
+are specific decimal numbers of CPUs to be isolated.  If not set
+on the kernel boot line, then cpu_isolated_map defaults to the empty
+cpumask.  See further Documentation/kernel-parameters.txt for isolcpus
+documentation.
+
+1.9 How do I use cpusets ?
 --------------------------
 
 In order to minimize the impact of cpusets on critical kernel
Documentation/kernel-parameters.txt +4 -0
@@ -731,6 +731,10 @@ and is between 256 and 4096 characters. 
 			tasks in the system -- can cause problems and
 			suboptimal load balancer performance.
 
+			If CPUSETS is configured, then the set of isolated
+			CPUs can be manippulated after system boot using the
+			"isolated_cpus" cpumask in the top cpuset.
+
 	isp16=		[HW,CD]
 			Format: <io>,<irq>,<dma>,<setup>
 
include/linux/sched.h +3 -0
@@ -715,6 +715,9 @@ struct sched_domain {
 #endif
 };
 
+extern const cpumask_t *sched_get_isolated_cpus(void);
+extern int sched_set_isolated_cpus(const cpumask_t *isolated_cpus);
+
 /*
  * Maximum cache size the migration-costs auto-tuning code will
  * search from:
kernel/cpuset.c +31 -0
@@ -974,6 +974,21 @@ static int update_memory_pressure_enable
 }
 
 /*
+ * Call with manage_mutex held.
+ */
+
+static int update_isolated_cpus(struct cpuset *cs, char *buf)
+{
+	cpumask_t isolated_cpus;
+	int retval;
+
+	retval = cpulist_parse(buf, isolated_cpus);
+	if (retval < 0)
+		return retval;
+	return sched_set_isolated_cpus(&isolated_cpus);
+}
+
+/*
  * update_flag - read a 0 or a 1 in a file and update associated flag
  * bit:	the bit to update (CS_CPU_EXCLUSIVE, CS_MEM_EXCLUSIVE,
  *				CS_NOTIFY_ON_RELEASE, CS_MEMORY_MIGRATE,
@@ -1215,6 +1230,7 @@ typedef enum {
 	FILE_MEMORY_PRESSURE,
 	FILE_SPREAD_PAGE,
 	FILE_SPREAD_SLAB,
+	FILE_ISOLATED_CPUS,
 	FILE_TASKLIST,
 } cpuset_filetype_t;
 
@@ -1282,6 +1298,9 @@ static ssize_t cpuset_common_file_write(
 		retval = update_flag(CS_SPREAD_SLAB, cs, buffer);
 		cs->mems_generation = cpuset_mems_generation++;
 		break;
+	case FILE_ISOLATED_CPUS:
+		retval = update_isolated_cpus(cs, buffer);
+		break;
 	case FILE_TASKLIST:
 		retval = attach_task(cs, buffer, &pathbuf);
 		break;
@@ -1397,6 +1416,10 @@ static ssize_t cpuset_common_file_read(s
 	case FILE_SPREAD_SLAB:
 		*s++ = is_spread_slab(cs) ? '1' : '0';
 		break;
+	case FILE_ISOLATED_CPUS:
+		s += cpulist_scnprintf(page, PAGE_SIZE,
+					*sched_get_isolated_cpus());
+		break;
 	default:
 		retval = -EINVAL;
 		goto out;
@@ -1770,6 +1793,11 @@ static struct cftype cft_spread_slab = {
 	.private = FILE_SPREAD_SLAB,
 };
 
+static struct cftype cft_isolated_cpus = {
+	.name = "isolated_cpus",
+	.private = FILE_ISOLATED_CPUS,
+};
+
 static int cpuset_populate_dir(struct dentry *cs_dentry)
 {
 	int err;
@@ -1960,6 +1988,9 @@ int __init cpuset_init(void)
 	/* memory_pressure_enabled is in root cpuset only */
 	if (err == 0)
 		err = cpuset_add_file(root, &cft_memory_pressure_enabled);
+	/* isolated_cpus is in root cpuset only */
+	if (err == 0)
+		err = cpuset_add_file(root, &cft_isolated_cpus);
 out:
 	return err;
 }
kernel/sched.c +20 -1
@@ -5608,7 +5608,7 @@ static void cpu_attach_domain(struct sch
 }
 
 /* cpus with isolated domains */
-static cpumask_t __cpuinitdata cpu_isolated_map = CPU_MASK_NONE;
+static cpumask_t cpu_isolated_map = CPU_MASK_NONE;
 
 /* Setup the mask of cpus configured for isolated domains */
 static int __init isolated_cpu_setup(char *str)
@@ -6866,6 +6866,25 @@ void __init sched_init_smp(void)
 	if (set_cpus_allowed(current, non_isolated_cpus) < 0)
 		BUG();
 }
+
+const cpumask_t *sched_get_isolated_cpus(void)
+{
+	return &cpu_isolated_map;
+}
+
+int sched_set_isolated_cpus(const cpumask_t *isolated_cpus)
+{
+	cpumask_t newly_isolated_cpus;
+	int err;
+
+	lock_cpu_hotplug();
+	cpus_and(newly_isolated_cpus, cpu_online_map, *isolated_cpus);
+	detach_destroy_domains(&newly_isolated_cpus);
+	cpu_isolated_map = *isolated_cpus;
+	err = arch_init_sched_domains(&cpu_online_map);
+	unlock_cpu_hotplug();
+	return err;
+}
 #else
 void __init sched_init_smp(void)
 {
Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#1 lore raw
Quoted 11 lines · Paul Jackson

From: Paul Jackson <[email protected]>

Enable user code to isolate CPUs on a system from the domains that determine scheduler load balancing.

This is already doable using the boot parameter "isolcpus=". The folks running realtime code on production systems, where some nodes are isolated for realtime, and some not, and where it is unacceptable to reboot to adjust this, need to be able to change which CPUs are isolated from the scheduler balancing code on the fly.

Quoted 3 lines

This is done by exposing the kernels cpu_isolated_map as a cpumask in the root cpuset directory, in a file called 'isolated_cpus', where it can be read and written by sufficiently privileged user code.

This should be done outside cpusets.

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#2 lore raw
Quoted 1 line · Nick

This should be done outside cpusets.

So ... where should it be done?

And what would be better about that other place?

And by the way, I see in another patch you put some other stuff for configuring sched domains in the root cpuset. If the root cpuset was the right place for that, why isn't it the right place for this? ... strange.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#3 lore raw
Quoted 6 lines · Paul Jackson
Quoted 1 line · Nick

This should be done outside cpusets.

So ... where should it be done?

sched.c I suppose.

Quoted 2 lines

And what would be better about that other place?

Because it is not a cpuset specific feature.

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#4 lore raw
Quoted 3 lines
Quoted 1 line

So ... where should it be done?

sched.c I suppose.

Are we discussing where the implementing code should go, or where the isolated cpu map special file should be exposed to user space?

And you didn't answer my other questions, such as:
1) If your other patch to manipulate sched domains has code that belongs in kernel/cpuset.c, and special files that belong in /dev/cpuset, why shouldn't this one naturally go in the same places?
2) Why ... why? What would be better about sched.c and what's wrong with where it is (the code and the exposed file)?

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#5 lore raw
Quoted 8 lines · Paul Jackson
Quoted 3 lines
Quoted 1 line

So ... where should it be done?

sched.c I suppose.

Are we discussing where the implementing code should go, or where the isolated cpu map special file should be exposed to user space?

Both?

Quoted 5 lines

And you didn't answer my other questions, such as:
1) If your other patch to manipulate sched domains has code that belongs in kernel/cpuset.c, and special files that belong in /dev/cpuset, why shouldn't this one naturally go in the same places?

Because they are cpuset specific. This is not.

Quoted 3 lines

2) Why ... why? What would be better about sched.c and what's wrong with where it is (the code and the exposed file)?

Because it is not specific to CONFIG_CPUSETS. People who don't configure CONFIG_CPUSETS may still want to change isolcpus at runtime.

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#6 lore raw
Quoted 6 lines
Quoted 4 lines

1) If your other patch to manipulate sched domains has code that belongs in kernel/cpuset.c, and special files that belong in /dev/cpuset, why shouldn't this one naturally go in the same places?

Because they are cpuset specific. This is not.

Bizarre. That other patch of yours, that had cpu_exclusive cpusets only at the top level defining sched domain partitions, is cpuset specific only because you're hijacking the cpu_exclusive flag to add new semantics to it, and then claiming you're not adding new semantics to it, and then claiming we should be doing this all implicitly without additional kernel-user API apparatus, and then claiming that if your new abuse of cpuset flags intrudes on other usage patterns of cpusets or if they have need of a particular sched domain partitioning that they have to explicitly set up top level cpusets to get a desired sched domain partitioning ...

Stop already ... this is getting insane.

Where are we?

By now, I think everyone (that is still following this thread ;) seems to agree that the current code linking cpu_exclusive and sched domain partitions is borked.

I responded to that realization earlier this week by proposing a patch to add new sched_domain flags to each cpuset.

Suresh responded to this realization by writing:
> ...I don't know much about how job manager interacts with > cpusets but this behavior sounds bad to me.

You responded by proposing a patch to use just the cpu_exclusive flags in the top level cpusets to define sched domain partitions.

The current code linking cpu_exclusive and sched domain partitions is borked, and serving almost no purpose. One of its goals was to provide a way to workaround the scaling issues with the scheduler load balancer code on humongous CPU count systems. This current code hasn't helped us a bit there. We're starting to address these scaling issues in various other ways.

So far as I can tell, the only actual successful application of this current code to manipulate sched domain partitions has been by a real time group, who have managed to isolate CPUs from load balancing with this.

Everyone rejects the current code, and so far, for every replacement suggested, at least one person rejects it. You don't like my patches, and I don't like yours.

Sounds to me like we don't have concensus yet ;). Agreed?

And its not just the code that would try to define sched domains via some implicit or explicit cpuset hooks that is in question here.

The sched domain related code in kernel/sched.c is overly complicated and ifdef'd. I can't personally critique that code, because my brain is too small. After repeated efforts to make sense of it, I still don't understand it. But I have it on good authority from colleagues whose brains are bigger than mine that this code could use an overhaul.

So ... not only do we not have concensus on the specific patch to control the definition of sched domain partitions, this is moreover part of a larger ball of yarn that should be unraveled.

We'd be fools to try to settle on the specifics of the (possibly cpuset related) API to define sched domain partitions in isolation. We should do that as part of such an overhaul of the rest of the sched domain code.

The current implicit hooks between cpu_exclusive cpusets and sched domain partitions should be removed, as proposed in my patch:

[RFC] cpuset: remove sched domain hooks from cpusets

I will submit this patch to Andrew for inclusion in *-mm. It doesn't change any explicit kernel API's - just removes the sched domain side affects of the cpu_exclusive flag, which are currently borked and more dangerous than useful. So long as any rework we do provides some way to isolate CPUs from load balancing sometime in the next several months, before whenever the one real time group doing this needs to see it, no one will miss these current sched domain hooks in cpusets.

I will agree to NAK your (Nick's) patch to overload the cpu_exclusive flags in the top level cpusets with sched domain partitioning side affects, if you agree to NAK my patches to create an 'isolated_cpus' file in the top cpuset.

My colleague Christoph Lameter will be looking into overhauling some of this the sched domain code. I am firmly convinced that any kernel API's, implicit or explicit, cpuset related or not, involving these sched domains and their partitioning should arise naturally out of that effort, and not be band-aided on ahead of time.

Let's agree to give Christoph a clean shot at this, and join him in this effort where we can.

Thanks.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#7 lore raw
Quoted 10 lines · Paul Jackson
Quoted 6 lines
Quoted 4 lines

1) If your other patch to manipulate sched domains has code that belongs in kernel/cpuset.c, and special files that belong in /dev/cpuset, why shouldn't this one naturally go in the same places?

Because they are cpuset specific. This is not.

Bizarre. That other patch of yours, that had cpu_exclusive cpusets only at the top level defining sched domain partitions, is cpuset

I'm talking about isolcpus. What do isolcpus have to do with cpusets.c? You can turn off CONFIG_CPUSETS and still use isolcpusets, can't you?

Quoted 9 lines

specific only because you're hijacking the cpu_exclusive flag to add new semantics to it, and then claiming you're not adding new semantics to it, and then claiming we should be doing this all implicitly without additional kernel-user API apparatus, and then claiming that if your new abuse of cpuset flags intrudes on other usage patterns of cpusets or if they have need of a particular sched domain partitioning that they have to explicitly set up top level cpusets to get a desired sched domain partitioning ...

Quoted 3 lines

Stop already ... this is getting insane.

Where are we?

I'll start again...

set_cpus_allowed is a feature of the scheduler that allows you to restrict one task to a subset of all cpus. Right?

And cpusets uses this interface as the mechanism to implement the semantics which the user has asked for. Yes?

sched-domains partitioning is a feature of the scheduler that allows you to restrict zero or more tasks to the partition, and zero or more tasks to the complement of the partition. OK?

So if you have a particular policy you need to implement, which is one cpus_exclusive cpuset off the root, covering half the cpus in the system (as a simple example)... why is it good to implement that with set_cpus_allowed and bad to implement it with partitions?

Or, another question, how does my patch hijack cpus_allowed? In what way does it change the semantics of cpus_allowed?

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#8 lore raw
Quoted 10 lines · Nick Piggin

set_cpus_allowed is a feature of the scheduler that allows you to restrict one task to a subset of all cpus. Right?

And cpusets uses this interface as the mechanism to implement the semantics which the user has asked for. Yes?

sched-domains partitioning is a feature of the scheduler that allows you to restrict zero or more tasks to the partition, and zero or more tasks to the complement of the partition. OK?

Quoted 7 lines

So if you have a particular policy you need to implement, which is one cpus_exclusive cpuset off the root, covering half the cpus in the system (as a simple example)... why is it good to implement that with set_cpus_allowed and bad to implement it with partitions?

Or, another question, how does my patch hijack cpus_allowed? In what way does it change the semantics of cpus_allowed?

That should be, in what way does it change the semantics of cpusets in any way?

IOW, how could a user possibly notice or care that partitions are being used to implement a given policy? (apart from the fact that the balancing will work better).

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#9 lore raw
Quoted 3 lines · Nick

IOW, how could a user possibly notice or care that partitions are being used to implement a given policy? (apart from the fact that the balancing will work better).

Tasks in higher up cpusets (e.g., the top cpuset) wouldn't load balance across these partitions.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#10 lore raw
In reply to Nick Piggin
Quoted 2 lines · Nick

I'm talking about isolcpus. What do isolcpus have to do with cpusets.c? You can turn off CONFIG_CPUSETS and still use isolcpusets, can't you?

The connection of isolcpus with cpusets is about as strong (or weak) as the connection of sched domain partitioning with cpusets.

Both isolcpus and sched domain partitions are tweaks to the scheduler domain code, to lessen the impact of load balancing, by making sched domains smaller, or by avoiding some cpus entirely.

Cpusets is concerned with the placement of tasks on cpus and nodes.

That's more related to sched domains (and hence to isolated cpus and sched domain partitioning) than it is, say, to the crypto code for generating randomly random numbers from /dev/random.

But I will grant it is not a strong connection.

Apparently you were seeing the potential for a stronger connection between cpusets and sched domain partitioning, because you thought that the cpuset configuration naturally defined some partitions of the systems cpus, that it would behoove the sched domain partitioning code to take advantage of.

Unfortunately, cpusets define no such partitioning ... not system wide.

Quoted 4 lines

So if you have a particular policy you need to implement, which is one cpus_exclusive cpuset off the root, covering half the cpus in the system (as a simple example)... why is it good to implement that with set_cpus_allowed and bad to implement it with partitions?

A cpu_exclusive cpuset does not implement the policy of partitioning a system, such that no one else can use those cpus. It implements a policy of limiting the sharing of cpus, just to ones cpuset ancestors and descendents - no distant cousins. That makes it easier for system admins and batch schedulers to manage the sharing of resources to meet their needs. They have fewer places to worry about that might be intruding.

It would be reasonable, for example, to do the following. One could have a big job, that depended on scheduler load balancing, that ran in the top cpuset, covering the entire system. And one could have smaller jobs, in child cpusets. During the day one pauses the big job and lets the smaller jobs run. During the night, one does the reverse. Granted, this example is a little bit hypothetical. Things like this are done, but usually a couple layers further down the cpuset hierarchy. At the top level, one common configuration that I am aware of puts almost the entire system into one cpuset, to be managed by the batch scheduler, and just a handful of cpus into a separate cpuset, to handle the classic Unix load (init, daemons, cron, admins login.)

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#11 lore raw
In reply to Nick Piggin
Quoted 2 lines · Nick

Or, another question, how does my patch hijack cpus_allowed? In what way does it change the semantics of cpus_allowed?

It limits load balancing for tasks in cpusets containing a superset of that cpusets cpus.

There are always such cpusets - the top cpuset if no other.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#12 lore raw
Quoted 8 lines · Paul Jackson
Quoted 2 lines · Nick

Or, another question, how does my patch hijack cpus_allowed? In what way does it change the semantics of cpus_allowed?

It limits load balancing for tasks in cpusets containing a superset of that cpusets cpus.

There are always such cpusets - the top cpuset if no other.

Its just a corner case issue that Nick didn't consider while doing a quick patch. Nick meant to partition the sched domain at the top exclusive cpuset and he probably missed the case where root cpuset is marked as exclusive.

thanks,
suresh

#13 lore raw
Quoted 4 lines · Suresh

Its just a corner case issue that Nick didn't consider while doing a quick patch. Nick meant to partition the sched domain at the top exclusive cpuset and he probably missed the case where root cpuset is marked as exclusive.

This makes no sense.

If P is a partition of S, that means that P is a set of subsets of S such that the intersection of any two members of P is empty, and the union of the members of P equals S.

If P is a partition of S, then adding S itself to P as another member makes P no longer a partion, for then every element of S is in two elements of P, not one.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#14 lore raw
In reply to Paul Jackson
Quoted 10 lines · Paul Jackson
Quoted 2 lines · Nick

Or, another question, how does my patch hijack cpus_allowed? In what way does it change the semantics of cpus_allowed?

It limits load balancing for tasks in cpusets containing a superset of that cpusets cpus.

There are always such cpusets - the top cpuset if no other.

Ah OK, and there is my misunderstanding with cpusets. From the documentation it appears as though cpu_exclusive cpusets are made in order to do the partitioning thing.

If you always have other domains overlapping them (regardless that it is a parent), then what actual use does cpu_exclusive flag have?

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#15 lore raw
Quoted 3 lines

If you always have other domains overlapping them (regardless that it is a parent), then what actual use does cpu_exclusive flag have?

Good question. To be a tad too honest, I don't consider it to be all that essential.

It does turn out to be a bit useful, in that you can be confident that if you are administering the batch scheduler and it has a cpu_exclusive cpuset covering a big chunk of your system, then no other cpuset other than the top cpuset right above, which you administer pretty tightly, could have anything overlapping it. You don't have to actually look at the cpumasks in all the parallel cpusets and cross check each one against the batch schedulers set of cpus for overlap.

But as you can see by grep'ing in kernel/cpuset.c, it is only used to generate a couple of error returns, for things that you can do just fine by turning off the cpu_exclusive flag.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#16 lore raw
Quoted 16 lines · Paul Jackson
Quoted 3 lines

If you always have other domains overlapping them (regardless that it is a parent), then what actual use does cpu_exclusive flag have?

Good question. To be a tad too honest, I don't consider it to be all that essential.

It does turn out to be a bit useful, in that you can be confident that if you are administering the batch scheduler and it has a cpu_exclusive cpuset covering a big chunk of your system, then no other cpuset other than the top cpuset right above, which you administer pretty tightly, could have anything overlapping it. You don't have to actually look at the cpumasks in all the parallel cpusets and cross check each one against the batch schedulers set of cpus for overlap.

Quoted 3 lines

But as you can see by grep'ing in kernel/cpuset.c, it is only used to generate a couple of error returns, for things that you can do just fine by turning off the cpu_exclusive flag.

Well, it was supposed to be used for sched-domains partitioning, and its uselessness for anything else I guess is what threw me.

But even the way cpu_exclusive semantics are defined makes it not quite compatible with partitioning anyway, unfortunately.

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#17 lore raw
Quoted 2 lines · Nick

Well, it was supposed to be used for sched-domains partitioning, and its uselessness for anything else I guess is what threw me.

The use of cpu_exclusive for sched domain partitioning was added later, by a patch from Dinikar, in April or May of 2005.

In hindsight, I think I made a mistake in agreeing to, and probably encouraging, this particular overloading of cpu_exclusive. I had difficulty adequately understanding what was going on.

Granted, as we've noted elsewhere on this thread, the cpu_exclusive flag is underutilized. It gives cpusets so marked a certain limited exclusivity to its cpus, relative to its siblings, but it doesn't do much else, other than this controversial partitioning of sched domains.

There may well be a useful role for the cpu_exclusive flag in managing sched domains and partitioning. The current role is flawed, in my view.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#18 lore raw
In reply to Paul Jackson
Quoted 11 lines · Paul Jackson

From: Paul Jackson <[email protected]>

Enable user code to isolate CPUs on a system from the domains that determine scheduler load balancing.

This is already doable using the boot parameter "isolcpus=". The folks running realtime code on production systems, where some nodes are isolated for realtime, and some not, and where it is unacceptable to reboot to adjust this, need to be able to change which CPUs are isolated from the scheduler balancing code on the fly.

Quoted 5 lines

This is done by exposing the kernels cpu_isolated_map as a cpumask in the root cpuset directory, in a file called 'isolated_cpus', where it can be read and written by sufficiently privileged user code.

IMO this patch addresses just one of the requirements for partitionable sched domains

1. Partition the system into multiple sched domains such that rebalancing happens within the group of each such partition
2. Ability to create partitioned sched domains that can further be partitioned with exclusive and non exclusive cpusets (exclusive in this context is not tied to sched domains anymore)
3. Ability to partition a system, such that a cpus in a partition dont have any rebalancing at all

Hope this makes sense

-Dinakar

Quoted 208 lines

---

I have built and booted this on one system, ia64/sn2, and verified that I can set and query the cpu_isolated_map, and that tasks are not scheduled on such isolated CPUs unless I explicitly place them there.

			-pj
 Documentation/cpusets.txt           |   26 +++++++++++++++++++++++---
 Documentation/kernel-parameters.txt |    4 ++++
 include/linux/sched.h               |    3 +++
 kernel/cpuset.c                     |   31 +++++++++++++++++++++++++++++++
 kernel/sched.c                      |   21 ++++++++++++++++++++-
 5 files changed, 81 insertions(+), 4 deletions(-)
Documentation/cpusets.txt +23 -3
@@ -19,7 +19,8 @@ CONTENTS:
   1.5 What does notify_on_release do ?
   1.6 What is memory_pressure ?
   1.7 What is memory spread ?
-  1.8 How do I use cpusets ?
+  1.8 What is isolated_cpus ?
+  1.9 How do I use cpusets ?
 2. Usage Examples and Syntax
   2.1 Basic Usage
   2.2 Adding/removing cpus
@@ -174,8 +175,9 @@ containing the following files describin
  - notify_on_release flag: run /sbin/cpuset_release_agent on exit?
  - memory_pressure: measure of how much paging pressure in cpuset
 
-In addition, the root cpuset only has the following file:
+In addition, the root cpuset only has the following files:
  - memory_pressure_enabled flag: compute memory_pressure?
+ - isolated_cpus: cpus excluded from scheduler domains
 
 New cpusets are created using the mkdir system call or shell
 command.  The properties of a cpuset, such as its flags, allowed
@@ -378,7 +380,25 @@ data set, the memory allocation across t
 can become very uneven.
 
 
-1.8 How do I use cpusets ?
+1.8 What is isolated_cpus ?
+---------------------------
+
+The cpumask isolated_cpus, visible in the top cpuset, provides direct
+access to the kernel cpu_isolated_map (in kernel/sched.c), which are
+the CPUs which are excluded from scheduler domains.  The scheduler
+will not load balance tasks on isolated cpus.  The 'isolated_cpus'
+file in the root cpuset can be read, to obtain the current setting
+of cpu_isolated_map, and can be written, to write a new cpumask to
+cpu_isolated_map.
+
+The initial value of the cpu_isolated_map can be set by the kernel
+boottime argument "isolcpus=n1,n2,n3,..." where n1, n2, n3, etc
+are specific decimal numbers of CPUs to be isolated.  If not set
+on the kernel boot line, then cpu_isolated_map defaults to the empty
+cpumask.  See further Documentation/kernel-parameters.txt for isolcpus
+documentation.
+
+1.9 How do I use cpusets ?
 --------------------------
 
 In order to minimize the impact of cpusets on critical kernel
Documentation/kernel-parameters.txt +4 -0
@@ -731,6 +731,10 @@ and is between 256 and 4096 characters. 
 			tasks in the system -- can cause problems and
 			suboptimal load balancer performance.
 
+			If CPUSETS is configured, then the set of isolated
+			CPUs can be manippulated after system boot using the
+			"isolated_cpus" cpumask in the top cpuset.
+
 	isp16=		[HW,CD]
 			Format: <io>,<irq>,<dma>,<setup>
 
include/linux/sched.h +3 -0
@@ -715,6 +715,9 @@ struct sched_domain {
 #endif
 };
 
+extern const cpumask_t *sched_get_isolated_cpus(void);
+extern int sched_set_isolated_cpus(const cpumask_t *isolated_cpus);
+
 /*
  * Maximum cache size the migration-costs auto-tuning code will
  * search from:
kernel/cpuset.c +31 -0
@@ -974,6 +974,21 @@ static int update_memory_pressure_enable
 }
 
 /*
+ * Call with manage_mutex held.
+ */
+
+static int update_isolated_cpus(struct cpuset *cs, char *buf)
+{
+	cpumask_t isolated_cpus;
+	int retval;
+
+	retval = cpulist_parse(buf, isolated_cpus);
+	if (retval < 0)
+		return retval;
+	return sched_set_isolated_cpus(&isolated_cpus);
+}
+
+/*
  * update_flag - read a 0 or a 1 in a file and update associated flag
  * bit:	the bit to update (CS_CPU_EXCLUSIVE, CS_MEM_EXCLUSIVE,
  *				CS_NOTIFY_ON_RELEASE, CS_MEMORY_MIGRATE,
@@ -1215,6 +1230,7 @@ typedef enum {
 	FILE_MEMORY_PRESSURE,
 	FILE_SPREAD_PAGE,
 	FILE_SPREAD_SLAB,
+	FILE_ISOLATED_CPUS,
 	FILE_TASKLIST,
 } cpuset_filetype_t;
 
@@ -1282,6 +1298,9 @@ static ssize_t cpuset_common_file_write(
 		retval = update_flag(CS_SPREAD_SLAB, cs, buffer);
 		cs->mems_generation = cpuset_mems_generation++;
 		break;
+	case FILE_ISOLATED_CPUS:
+		retval = update_isolated_cpus(cs, buffer);
+		break;
 	case FILE_TASKLIST:
 		retval = attach_task(cs, buffer, &pathbuf);
 		break;
@@ -1397,6 +1416,10 @@ static ssize_t cpuset_common_file_read(s
 	case FILE_SPREAD_SLAB:
 		*s++ = is_spread_slab(cs) ? '1' : '0';
 		break;
+	case FILE_ISOLATED_CPUS:
+		s += cpulist_scnprintf(page, PAGE_SIZE,
+					*sched_get_isolated_cpus());
+		break;
 	default:
 		retval = -EINVAL;
 		goto out;
@@ -1770,6 +1793,11 @@ static struct cftype cft_spread_slab = {
 	.private = FILE_SPREAD_SLAB,
 };
 
+static struct cftype cft_isolated_cpus = {
+	.name = "isolated_cpus",
+	.private = FILE_ISOLATED_CPUS,
+};
+
 static int cpuset_populate_dir(struct dentry *cs_dentry)
 {
 	int err;
@@ -1960,6 +1988,9 @@ int __init cpuset_init(void)
 	/* memory_pressure_enabled is in root cpuset only */
 	if (err == 0)
 		err = cpuset_add_file(root, &cft_memory_pressure_enabled);
+	/* isolated_cpus is in root cpuset only */
+	if (err == 0)
+		err = cpuset_add_file(root, &cft_isolated_cpus);
 out:
 	return err;
 }
kernel/sched.c +20 -1
@@ -5608,7 +5608,7 @@ static void cpu_attach_domain(struct sch
 }
 
 /* cpus with isolated domains */
-static cpumask_t __cpuinitdata cpu_isolated_map = CPU_MASK_NONE;
+static cpumask_t cpu_isolated_map = CPU_MASK_NONE;
 
 /* Setup the mask of cpus configured for isolated domains */
 static int __init isolated_cpu_setup(char *str)
@@ -6866,6 +6866,25 @@ void __init sched_init_smp(void)
 	if (set_cpus_allowed(current, non_isolated_cpus) < 0)
 		BUG();
 }
+
+const cpumask_t *sched_get_isolated_cpus(void)
+{
+	return &cpu_isolated_map;
+}
+
+int sched_set_isolated_cpus(const cpumask_t *isolated_cpus)
+{
+	cpumask_t newly_isolated_cpus;
+	int err;
+
+	lock_cpu_hotplug();
+	cpus_and(newly_isolated_cpus, cpu_online_map, *isolated_cpus);
+	detach_destroy_domains(&newly_isolated_cpus);
+	cpu_isolated_map = *isolated_cpus;
+	err = arch_init_sched_domains(&cpu_online_map);
+	unlock_cpu_hotplug();
+	return err;
+}
 #else
 void __init sched_init_smp(void)
 {
Signature · 4 lines
-- 
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#19 lore raw
Quoted 2 lines · Dinakar

IMO this patch addresses just one of the requirements for partitionable sched domains

Correct - this particular patch was just addressing one of these.

Nick raised the reasonable concern that this patch was adding something to cpusets that was not especially related to cpusets.

So I will not be sending this patch to Andrew for *-mm.

There are further opportunities for improvements in some of this code, which my colleague Christoph Lameter may be taking an interest in. Ideally kernel-user API's for isolating and partitioning sched domains would arise from that work, though I don't know if we can wait that long.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#20 lore raw
Quoted 10 lines · Paul Jackson
Quoted 2 lines · Dinakar

IMO this patch addresses just one of the requirements for partitionable sched domains

Correct - this particular patch was just addressing one of these.

Nick raised the reasonable concern that this patch was adding something to cpusets that was not especially related to cpusets.

Did you send resend the patch to remove sched-domain partitioning? After clearing up my confusion, IMO that is needed and could probably go into 2.6.19.

Quoted 7 lines

So I will not be sending this patch to Andrew for *-mm.

There are further opportunities for improvements in some of this code, which my colleague Christoph Lameter may be taking an interest in. Ideally kernel-user API's for isolating and partitioning sched domains would arise from that work, though I don't know if we can wait that long.

The sched-domains code is all there and just ready to be used. IMO using the cpusets API (or a slight extension thereof) would be the best idea if we're going to use any explicit interface at all.

A cool option would be to determine the partitions according to the disjoint set of unions of cpus_allowed masks of all tasks. I see this getting computationally expensive though, probably O(tasks*CPUs)... I guess that isn't too bad.

Might be better than a userspace interface.

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#21 lore raw
Quoted 3 lines · Nick

Did you send resend the patch to remove sched-domain partitioning? After clearing up my confusion, IMO that is needed and could probably go into 2.6.19.

The patch titled
cpuset: remove sched domain hooks from cpusets
went into *-mm on Friday, 20 Oct.

Is that the patch you mean?

It's just the first step - unplugging the old.

Now we need the new:
1) Ability at runtime to isolate cpus for real-time and such.
2) Big systems perform better if we can avoid load balancing across zillions of cpus.

Quoted 3 lines

The sched-domains code is all there and just ready to be used. IMO using the cpusets API (or a slight extension thereof) would be the best idea if we're going to use any explicit interface at all.

Good. Thanks.

Quoted 4 lines

A cool option would be to determine the partitions according to the disjoint set of unions of cpus_allowed masks of all tasks. I see this getting computationally expensive though, probably O(tasks*CPUs)... I guess that isn't too bad.

Yeah - if that would work, from the practical perspective of providing us with a useful partitioning (get those humongous sched domains carved down to a reasonable size) then that would be cool.

I'm guessing that in practice, it would be annoying to use. One would end up with stray tasks that happened to be sitting in one of the bigger cpusets and that did not have their cpus_allowed narrowed, stopping us from getting a useful partitioning. Perhaps anilliary tasks associated with the batch scheduler, or some paused tasks in an inactive job that were it active would need load balancing across a big swath of cpus. These would be tasks that we really didn't need to load balance, but they would appear as if they needed it because of their fat cpus_allowed.

Users (admins) would have to hunt down these tasks that were getting in the way of a nice partitioning and whack their cpus_allowed down to size.

So essentially, one would end up with another userspace API, backdoor again. Like those magic doors in the libraries of wealthy protagonists in mystery novels, where you have to open a particular book and pull the lamp cord to get the door to appear and open.

Automatic chokes and transmissions are great - if they work. If not, give me a knob and a stick.

===

Another idea for a cpuset-based API to this ...

Quoted 1 line

From our internal perspective, it's all about getting the sched domain

partitions cut down to a reasonable size, for performance reasons.

But from the users perspective, the deal we are asking them to consider is to trade in fully automatic, all tasks across all cpus, load balancing, in turn for better performance.

Big system admins would often be quite happy to mark the top cpuset as "no need to load balance tasks in this cpuset." They would take responsibility for moving any non-trivial, unpinned tasks into lower cpusets (or not be upset if something left behind wasn't load balancing.)

And the batch scheduler would be quite happy to mark its top cpuset as "no need to load balance". It could mark any cpusets holding inactive jobs the same way.

This "no need to load balance" flag would be advisory. The kernel might load balance anyway. For example if the batch scheduler were running under a top cpuset that was -not- so marked, we'd still have to load balance everyone. The batch scheduler wouldn't care. It would have done its duty, to mark which of its cpusets didn't need balancing.

All we need from them is the ok to not load balance certain cpusets, and the rest is easy enough. If they give us such ok on enough of the big cpusets, we give back a nice performance improvement.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#22 lore raw
Quoted 19 lines · Paul Jackson
Quoted 4 lines · Nick

A cool option would be to determine the partitions according to the disjoint set of unions of cpus_allowed masks of all tasks. I see this getting computationally expensive though, probably O(tasks*CPUs)... I guess that isn't too bad.

Yeah - if that would work, from the practical perspective of providing us with a useful partitioning (get those humongous sched domains carved down to a reasonable size) then that would be cool.

I'm guessing that in practice, it would be annoying to use. One would end up with stray tasks that happened to be sitting in one of the bigger cpusets and that did not have their cpus_allowed narrowed, stopping us from getting a useful partitioning. Perhaps anilliary tasks associated with the batch scheduler, or some paused tasks in an inactive job that were it active would need load balancing across a big swath of cpus. These would be tasks that we really didn't need to load balance, but they would appear as if they needed it because of their fat cpus_allowed.

Quoted 8 lines

Users (admins) would have to hunt down these tasks that were getting in the way of a nice partitioning and whack their cpus_allowed down to size.

So essentially, one would end up with another userspace API, backdoor again. Like those magic doors in the libraries of wealthy protagonists in mystery novels, where you have to open a particular book and pull the lamp cord to get the door to appear and open.

Also we need to be careful with malicious users partitioning the systems wrongly based on the cpus_allowed for their tasks.

Quoted 10 lines

Automatic chokes and transmissions are great - if they work. If not, give me a knob and a stick.

===

Another idea for a cpuset-based API to this ...

From our internal perspective, it's all about getting the sched domain partitions cut down to a reasonable size, for performance reasons.

we need consider resource partitioning too.. and this is where google interests are probably coming from?

Quoted 25 lines

But from the users perspective, the deal we are asking them to consider is to trade in fully automatic, all tasks across all cpus, load balancing, in turn for better performance.

Big system admins would often be quite happy to mark the top cpuset as "no need to load balance tasks in this cpuset." They would take responsibility for moving any non-trivial, unpinned tasks into lower cpusets (or not be upset if something left behind wasn't load balancing.)

And the batch scheduler would be quite happy to mark its top cpuset as "no need to load balance". It could mark any cpusets holding inactive jobs the same way.

This "no need to load balance" flag would be advisory. The kernel might load balance anyway. For example if the batch scheduler were running under a top cpuset that was -not- so marked, we'd still have to load balance everyone. The batch scheduler wouldn't care. It would have done its duty, to mark which of its cpusets didn't need balancing.

All we need from them is the ok to not load balance certain cpusets, and the rest is easy enough. If they give us such ok on enough of the big cpusets, we give back a nice performance improvement.

#23 lore raw
Quoted 2 lines · Suresh

Also we need to be careful with malicious users partitioning the systems wrongly based on the cpus_allowed for their tasks.

Untrusted users can't do that. On big systems where you have untrusted users, you stick them in smaller cpusets. Their cpus_allowed is not allowed to exceed what's allowed in their cpuset.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#24 lore raw
In reply to Siddha, Suresh B
Quoted 5 lines · Suresh
Quoted 2 lines

From our internal perspective, it's all about getting the sched domain partitions cut down to a reasonable size, for performance reasons.

we need consider resource partitioning too.. and this is where google interests are probably coming from?

I dropped a bit - what's 'resource partitioning' ?

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#25 lore raw
In reply to Paul Jackson
Quoted 7 lines · Paul Jackson
Quoted 3 lines · Nick

Did you send resend the patch to remove sched-domain partitioning? After clearing up my confusion, IMO that is needed and could probably go into 2.6.19.

Quoted 5 lines

The patch titled
cpuset: remove sched domain hooks from cpusets
went into *-mm on Friday, 20 Oct.

Is that the patch you mean?

Yes.

Quoted 6 lines

It's just the first step - unplugging the old.

Now we need the new:
1) Ability at runtime to isolate cpus for real-time and such.
2) Big systems perform better if we can avoid load balancing across zillions of cpus.

These are both part of the same larger solution, which is to partition domains. isolated CPUs are just the case of 1 CPU in its own domain (and that's how they are implemented now).

So we need the interface or some driver to do this partitioning.

Quoted 10 lines
Quoted 4 lines

A cool option would be to determine the partitions according to the disjoint set of unions of cpus_allowed masks of all tasks. I see this getting computationally expensive though, probably O(tasks*CPUs)... I guess that isn't too bad.

Yeah - if that would work, from the practical perspective of providing us with a useful partitioning (get those humongous sched domains carved down to a reasonable size) then that would be cool.

Quoted 8 lines

I'm guessing that in practice, it would be annoying to use. One would end up with stray tasks that happened to be sitting in one of the bigger cpusets and that did not have their cpus_allowed narrowed, stopping us from getting a useful partitioning. Perhaps anilliary tasks associated with the batch scheduler, or some paused tasks in an inactive job that were it active would need load balancing across a big swath of cpus. These would be tasks that we really didn't need to load balance, but they would appear as if they needed it because of their fat cpus_allowed.

But we simply can't make a partition for them because they have asked to use all CPUs. We can't know if this is something that should be partitioned or not, can we?

Quoted 3 lines

Users (admins) would have to hunt down these tasks that were getting in the way of a nice partitioning and whack their cpus_allowed down to size.

In the sense that they get what they ask for, yes. The obvious route for the big SGI systems is to put them in the right cpuset. The job managers (or admins) on those things are surely up to the task.

This leaves unbound and transient kernel threads like pdflush as a remaining problem. Not quite sure what to do about that yet. I see you have a little hack in there...

Quoted 7 lines

So essentially, one would end up with another userspace API, backdoor again. Like those magic doors in the libraries of wealthy protagonists in mystery novels, where you have to open a particular book and pull the lamp cord to get the door to appear and open.

Automatic chokes and transmissions are great - if they work. If not, give me a knob and a stick.

But your knob is just going to be some mechanism to say that you don't care about such and such a task, or you want to put task x into domain y.

Quoted 27 lines

===

Another idea for a cpuset-based API to this ...

Quoted 1 line

From our internal perspective, it's all about getting the sched domain

partitions cut down to a reasonable size, for performance reasons.

But from the users perspective, the deal we are asking them to consider is to trade in fully automatic, all tasks across all cpus, load balancing, in turn for better performance.

Big system admins would often be quite happy to mark the top cpuset as "no need to load balance tasks in this cpuset." They would take responsibility for moving any non-trivial, unpinned tasks into lower cpusets (or not be upset if something left behind wasn't load balancing.)

And the batch scheduler would be quite happy to mark its top cpuset as "no need to load balance". It could mark any cpusets holding inactive jobs the same way.

This "no need to load balance" flag would be advisory. The kernel might load balance anyway. For example if the batch scheduler were running under a top cpuset that was -not- so marked, we'd still have to load balance everyone. The batch scheduler wouldn't care. It would have done its duty, to mark which of its cpusets didn't need balancing.

Quoted 3 lines

All we need from them is the ok to not load balance certain cpusets, and the rest is easy enough. If they give us such ok on enough of the big cpusets, we give back a nice performance improvement.

I think this is much more of an automatic behind your back thing. If they don't want any load balancing to happen, they could pin the tasks in that cpuset to the cpu they're currently on, for example.

It would be trivial to make such a script to parse the root cpuset and do exactly this, wouldn't it?

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#26 lore raw
Quoted 3 lines · Nick

These are both part of the same larger solution, which is to partition domains. isolated CPUs are just the case of 1 CPU in its own domain (and that's how they are implemented now).

Quoted 1 line · he also

I think this is much more of an automatic behind your back thing.

I got confused there.

I agree that if we can do a -good- job of it, then an implicit, automatic solution is better for the problem of reducing sched domain partition sizes on large systems than yet another manual knob.

But I thought that it was good idea, with general agreement, to provide an explicit control of isolated cpus for the real-time folks, even if under the covers it use sched domain partitions of size 1 to implement it.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#27 lore raw
Quoted 9 lines · Paul Jackson
Quoted 3 lines · Nick

These are both part of the same larger solution, which is to partition domains. isolated CPUs are just the case of 1 CPU in its own domain (and that's how they are implemented now).

and later, he also wrote:

Quoted 8 lines
Quoted 1 line

I think this is much more of an automatic behind your back thing.

I got confused there.

I agree that if we can do a -good- job of it, then an implicit, automatic solution is better for the problem of reducing sched domain partition sizes on large systems than yet another manual knob.

OK, good.

Quoted 4 lines

But I thought that it was good idea, with general agreement, to provide an explicit control of isolated cpus for the real-time folks, even if under the covers it use sched domain partitions of size 1 to implement it.

If they isolate it by setting the cpus_allowed masks of processes to reflect the way they'd like balancing to be carried out, then the partition will be made for them.

But an explicit control might be required anyway, and I wouldn't disagree with it. It might be required to do more than just sched partitioning (eg. pdflush and other kernel threads should probably be made to stay off isolated cpus as well, where possible).

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#28 lore raw
In reply to Nick Piggin
Quoted 2 lines

It would be trivial to make such a script to parse the root cpuset and do exactly this, wouldn't it?

Ah - yes - that's doable. A certain company I work for ships pretty much that exact script, to its customers. It works well to remove all the unpinned tasks from the top level cpuset and put them in what we call the 'boot' cpuset, where the classic Unix load (init, cron, daemons, sysadmin login) is confined. This frees up the rest of the system to run "real" work. It works quite well, if I do say so.

Perhaps I'm being overly pessimistic about the potential of driving this partitioning off the cpus_allowed masks of the tasks. As you noted, it would be a cute trick to avoid some combinatorial explosion of the computational costs. But there are enough practical constraints on this problem - that should be quite doable.

Hmmm ...

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#29 lore raw
In reply to Nick Piggin
Quoted 6 lines · Nick

I think this is much more of an automatic behind your back thing. If they don't want any load balancing to happen, they could pin the tasks in that cpuset to the cpu they're currently on, for example.

It would be trivial to make such a script to parse the root cpuset and do exactly this, wouldn't it?

Yeah ... but ...

Such a script is trivial.

But I'm getting mixed messages on this "automatic behind your back" thing.

We can't actually run such a script automatically, without the user asking for it. We only run such a script, to move what tasks we can out of the top cpuset into a smaller cpuset, when the user sets up a configuration asking for it.

And saying "If they don't want any load balancing to happen, they could pin the tasks ..." doesn't sound automatic to me. It sounds like a backdoor API.

So you might say, and continue to hope, that this sched domain partitioning is automatic. But your own words, and my (limited) experience, don't match that expectation.

- We agree that automatic, truly automatic with no user intervention, beats some manually invoked API knob.

- But a straight forward, upfront API knob (such as a flag indicating "no need to load balance across this cpuset") beats a backdoor, indirect, API trying to pretend it's automatic when it not.

Quoted 3 lines

In the sense that they get what they ask for, yes. The obvious route for the big SGI systems is to put them in the right cpuset. The job managers (or admins) on those things are surely up to the task.

This is touching on a key issue.

Sure, if the admins feel motivated, this is within their capacity. For something like the real-time nodes, the admins seem willing to jump through flaming hoops and perform all manner of unnatural acts, to get those nodes running tight, low latency, jitter free.

But admins don't wake up in the morning -wanting- to think about the performance of balancing sched domains. Partitioning sched domains is not a frequently requested feature.

Sched domain partitioning is a workaround (apparently) for some (perhaps inevitable) scaling issues in the schedulers load balancer - right?

So, just how bad is this scaling issue with huge sched domains? And why can't it be addressed at the source? Perhaps say, by ratelimiting the balancing efforts at higher domain sizes?

For instance, I can imagine a dynamically variable ratelimit throttle on how often we try to balance across the higher domain sizes. If such load balancing efforts are not yielding much fruit (seldom yield a worthwhile task movement) then do them less often.

Then if someone is truly running a mixed timeshare load, of varying job sizes, wide open on a big system, with no serious effort at confining the jobs into smaller cpusets, we would continue to spend considerable effort load balancing the system. Not the most efficient way to run a system, but it should work as expected when you do that.

If on the other hand, someone is using lots of small cpusets and pinning to carefully manage thread placement on a large system, we would end up backing off the higher levels of load balancing, as it seldom bore any fruit.

==

To my limited awareness, this sched domain balancing performance problem (due to huge domains) isn't that big a problem.

I've been guessing that this is because the balancer is actually reasonably efficient when presented with a carefully pinned load that has one long living compute bound thread per CPU, even on very high CPU count systems.

I'm afraid we are creating a non-automatic band-aid for a problem, that would only work on systems (running nicely pinned workloads in small cpusets) for which it is not actually a problem, designing the backdoor API under the false assumption that it is really automatic, where instead this API will be the least capable of automatically working for exactly the workloads (poorly placed generic timeshare loads) that it is most needed for.

And if you can parse that last sentence, you're a smarter man than I am ;).

==

I can imagine an algorithm, along the lines Nick suggested yesterday, to extract partitions from the cpus_allowed masks of all the tasks:

It would scan each byte of each cpus_allowed mask (it could do bits, but bytes provide sufficient granularity and are faster.)

It would have an intermediate array of counters, one per cpumask_t byte (that is, NR_CPUS/8 counters.)

For each task, it would note the lowest non-zero byte in that tasks cpus_allowed, and increment the corresponding counter. Then it also note the highest non-zero byte in that cpus_allowed, and decrement the corresponding counter.

Then after this scan of all task cpus_allowed, it would make a single pass over the counter array, summing the counts in a running total. Anytime during that pass that the running total was zero, a partition could be placed.

This resembles the algorithm used to scan parenthesized expressions (Lisp, anyone?) to be sure that the parentheses are balanced (count doesn't go negative) and closed (count ends back at zero.) During such a scan, one increments a count on each open parenthesis, and decrements it on each close.

But I'm going to be surprised if such an algorithm actually finds good sched domain partitions, without at least manual intervention by the sysadmin to remove tasks from all of the large cpusets.

And if it does find good partitions, that will most likely happen on systems that are already administered using detailed job placement, which systems might (?) not have been a problem in the first place.

So we've got an algorithm that is automatic when we least need it, but requires administrative intervention when it was needed (making it not so automatic ;).

And if that's what this leads to, I'd prefer a straight forward flag that tells the system where to partition, or perhaps better, where it can skip load balancing if it wants to.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#30 lore raw
In reply to Nick Piggin
Quoted 22 lines · Nick Piggin
Quoted 8 lines · Paul Jackson
Quoted 2 lines · Dinakar

IMO this patch addresses just one of the requirements for partitionable sched domains

Correct - this particular patch was just addressing one of these.

Nick raised the reasonable concern that this patch was adding something to cpusets that was not especially related to cpusets.

Did you send resend the patch to remove sched-domain partitioning? After clearing up my confusion, IMO that is needed and could probably go into 2.6.19.

Quoted 7 lines

So I will not be sending this patch to Andrew for *-mm.

There are further opportunities for improvements in some of this code, which my colleague Christoph Lameter may be taking an interest in. Ideally kernel-user API's for isolating and partitioning sched domains would arise from that work, though I don't know if we can wait that long.

Quoted 3 lines

The sched-domains code is all there and just ready to be used. IMO using the cpusets API (or a slight extension thereof) would be the best idea if we're going to use any explicit interface at all.

Ok I am getting lost in all of the mails here, so let me try to summaize

The existing cpuset code that partitioned sched domains at the back of a exclusive cpuset has one major problem. Administrators will find that tasks assigned to top level cpusets, that contain child cpusets that are exclusive, can no longer be rebalanced across the entire cpus_allowed mask.
This as far as I can tell is the only problem with the current code. So I dont see why we need a major rewrite that involves a complete change in the approach to the dynamic sched domain implementation.

I really think all we need is to have a new flag (say sched_domain) that can be used by the admin to create a sched domain. Since this is in addition to the cpu_exclusive flag, the admin realizes that the tasks associated with the parent cpuset may need to be moved around to better reflect the cpus that will actually run on. Thats it.

Can somebody tell me why this approach is not good enough ?

I am testing this patch currently and will post it shortly for review

-Dinakar

Quoted 11 lines

A cool option would be to determine the partitions according to the disjoint set of unions of cpus_allowed masks of all tasks. I see this getting computationally expensive though, probably O(tasks*CPUs)... I guess that isn't too bad.

Might be better than a userspace interface.

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#31 lore raw
Quoted 3 lines · Dinakar

This as far as I can tell is the only problem with the current code. So I dont see why we need a major rewrite that involves a complete change in the approach to the dynamic sched domain implementation.

Nick and I agree that if we can get an adequate automatic partition of sched domains, without any such explicit 'sched_domain' API, then that would be better.

Nick keeps hoping we can do this automatically, and I have been fading in and out of agreement. I have doubts we can do an automatic partition that is adequate.

Last night, Nick suggested we could do this by partitioning based on the cpus_allowed masks of the tasks in the system, instead of based on selected cpusets in the system. We could create a partition any place that didn't cut across some tasks cpus_allowed. This would seem to have a better chance than basing it on the cpus masks in cpusets - for example a task in top cpuset that was pinned to a single CPU (many kernel threads fit this description) would no longer impede partitioning.

Right now, I am working on a posting that spells out an algorithm to compute such a partitioning, based on all task cpus_allowed masks in the system. I think that's doable.

But I doubt it works. I afraid it will result in useful partitions only in the cases we seem to need them least, on systems using cpusets to nicely carve up a system.

==

Why the heck are we doing this partitioning in the first place?

Putting aside for a moment the specialized needs of the real-time folks who want to isolate some nodes from any source of jitter, aren't these sched domain partitions just a workaround for performance issues, that arise from trying to load balance across big sched domains?

Granted, there may be no better way to address this performance issue. And granted, it may well be my own employers big honkin NUMA boxes that are most in need of this, and I just don't realize it.

But could someone whack me upside the head with a clear cut instance where we know we need this partitioning?

In particular, if (extreme example) I have 1024 threads on a 1024 CPU system, each compute bound and each pinned to a separate CPU, and nothing else, then do I still need these sched partitions? Or does the scheduler efficiently handle this case, quickly recognizing that it has no useful balancing work worth doing?

==

As it stands right now, if I had to place my "final answer" in Jeopardy on this, I'd vote for something like the patch you describe, which I take it is much like the sched_domain patch with which I started this scrum a week ago, minus the 'sched_domain_enabled' flag that I had in for backwards compatibility. I suspect we agree that we can do without that flag, and that a single clean long term API outweighs perfect backward compatibility, in this case.

==

The only twist to your patch I would like you to consider - instead of a 'sched_domain' flag marking where the partitions go, how about a flag that tells the kernel it is ok not to load balance tasks in a cpuset?

Then lower level cpusets could set such a flag, without immediate and brutal affects on the partitioning of all their parent cpusets. But if the big top level non-overlapping cpusets were so marked, then we could partition all the way down to where we were no longer able to do so, because we hit a cpuset that didn't have this flag set.

I think such a "ok not to load balance tasks in this cpuset" flag better fits what the users see here. They are being asked to let us turn off some automatic load balancing, in return for which they get better performance. I doubt that the phrase "dynamic scheduler domain partitions" is in the vocabulary of most of our users. More of them will understand the concept of load balancing automatically moving tasks to underutilized CPUs, and more of them would be prepared to make the choice between turning off load balancing in some top cpusets, and better kernel scheduler performance.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#32 lore raw

pj wrote to Dinakar:

Quoted 4 lines

The only twist to your patch I would like you to consider - instead of a 'sched_domain' flag marking where the partitions go, how about a flag that tells the kernel it is ok not to load balance tasks in a cpuset?

Dinakar - one possibility that might work well:

Proceed with the 'sched_domain' patch you are working on, as you planned.

(If you like, stop reading here ... <grin>.)

Then I can propose a patch on top of that, to flip the kernel-user API to the "ok not to load balance" style I'm proposing. This patch would:
- leave your internal logic in place, as is,
- remove your 'sched_domain' flag from the visible API (keep it internally),
- add a 'cpu_must_load_balance' (default 1) flag to the API, and
- add a bit of logic to set, top down, your internal sched_domain flag, based on the cpu_must_load_balance and parent sched_domain settings.

Then we can all see these two alternative API styles, your sched_domain style and my cpu_must_load_balance style, and pick one (just by keeping or tossing my extra patch.)

A couple of aspects of my cpu_must_load_balance style that I like:

* The batch scheduler can turn off requiring load balancing on its inactive cpusets, without worrying about whether it has the right (exclusive control) to do that or not.

* I figure that users will relate better to a choice of whether or not they require cpu load balancing, than they will to the question of where to partition scheduler domains.

Of course, if Nick succeeds on his mission to convince us that we can do this automatically, then the above doesn't matter, and we'd need a different patch altogether.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#33 lore raw
In reply to Dinakar Guniguntala
Quoted 4 lines · Dinikar

I really think all we need is to have a new flag (say sched_domain) that can be used by the admin to create a sched domain. ...

I am testing this patch currently and will post it shortly for review

Are you still working on this patch? I'm still figuring that this, or my 'cpu_must_load_balance' variation, will be our best choice.

The other avenue of pursuit, Nick's suggestion to partition based on the individual task cpus_allowed masks, is blowing up on me. Details below, if it matters to anyone ...

===

Quoted 26 lines · A couple days ago, I wrote:

I can imagine an algorithm, along the lines Nick suggested yesterday, to extract partitions from the cpus_allowed masks of all the tasks:

It would scan each byte of each cpus_allowed mask (it could do bits, but bytes provide sufficient granularity and are faster.)

It would have an intermediate array of counters, one per cpumask_t byte (that is, NR_CPUS/8 counters.)

For each task, it would note the lowest non-zero byte in that tasks cpus_allowed, and increment the corresponding counter. Then it also note the highest non-zero byte in that cpus_allowed, and decrement the corresponding counter.

Then after this scan of all task cpus_allowed, it would make a single pass over the counter array, summing the counts in a running total. Anytime during that pass that the running total was zero, a partition could be placed.

This resembles the algorithm used to scan parenthesized expressions (Lisp, anyone?) to be sure that the parentheses are balanced (count doesn't go negative) and closed (count ends back at zero.) During such a scan, one increments a count on each open parenthesis, and decrements it on each close.

Quoted 3 lines

But I'm going to be surprised if such an algorithm actually finds good sched domain partitions, without at least manual intervention by the sysadmin to remove tasks from all of the large cpusets.

This is a terrible algorithm. It does a really poor job of finding optimum (minimum size) sched domain partitions.

For example, if we pinned every task in the system so that it is either running on the even numbered CPUs, or on the odd numbered CPUs, this algorithm would miss the obvious partitioning, into even and odd numbered CPUs.

Or for a more flagrant example, lets say we pinned every single task but one to a single CPU, and then pinned that last task to a pair of CPUs, the lowest numbered and the highest numbered CPUs. Ideally we would only have a single sched domain partition in the entire system that needed balancing - a two CPU partition for that last task. The above algorithm would lump all the CPUs in the system into a partition, because they are numbered in the range of CPUs covered by that last task, into a single sched domain partition covering the entire system.

If people only selected cpus_allowed masks allowing adjacent CPUs, then this algorithm worked just fine. That's not a realistic constraint.

A deterministic algorithm for this might be NP-complete, I don't know. Whether or not a useful approximation exists, I don't know. And even if we found one, whether or not it would adequately meet the needs of high CPU count systems anywhere nearly as well as simply marking cpusets with flag to indicate which ones define sched domains, or which ones don't need to be load balanced ... I don't know, but I'm skeptical.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401