Processor sets (pset) for linux kernel 2.5/2.6?

Hi,

Looking through the archive etc., I cannot see any references to pset functionality more recently than

http://www.ussg.iu.edu/hypermail/linux/kernel/9904.1/0521.html

announcing an updated patch in April 1999. Are there any plans out there to include this, or similar functionality in 2.5/2.6? I'm particularly interested in getting exclusive access to a CPU (plus or minus HT support, for now anyway).

Thanks, Shaheed

sched_getaffinity() and sched_setaffinity()
You'll also need schedutils from http://tech9.net/rml/schedutils

all this (and more) documented at http://www.codemonkey.org.uk/post-halloween-2.5.txt

Dave

Shaheed, I think that task->cpu_allowed covers your need, since it describes a mask of cpu�'s allowed to run the task. CPU migration, for example, is implemented in current scheduler by saving the current cpu_allowed, forcing it to contain only the destination CPU and then restoring it.

I think it's carried over by fork(), but have been unable to find it on the sources.

HT is accounted as a NUMA SMP system with strong memory affinity for his 2 cores, so that when running 2 HT processors (2+2 cores), the tasks are kept preferably on the same HT processor and just bounce cores, since they share the same cache (don't know exactly L1, L2 or both, tough).

Greets, Antonio.

We don't actually do that right now (though it was discussed) - we treat it as flat SMP. Ingo had some patches to share 1 runqueue between each HT pair, which I think is a better plan for this, but we're not seeing much performance improvment from it.

M.

Yep, its inherited over fork().

Robert Love

Hmmm, AFAICS, sched_getaffinity() and sched_setaffinity()
allow the calling process to be bound to the nominated CPU(s), but that is not the same as giving them exclusive access, is it? In other words, other processes which have no particualr affinity needs can presumably still be scheduled to run on the same processor.

I am looking for something more akin to the patch I referred to...or did I miss something in the effect of set_cpus_allowed()?

Thanks for the replies, Shaheed

We strive for simple interfaces here in Linux :)

If you want to give them exclusive access, you need to bind all the other processes to the other processors. One easy way to do this is to have init bind itself elsewhere on boot.

Robert Love

Quoting Robert Love <[email protected]>:

Interesting idea. AFAICS, this involves either changing the code of /sbin/init to set its affinity to a default cpu mask (provided by a kernel boot flag, I presume), or using taskset-like functionality and then hoping that all current shells etc. die before the programs I care about start.

Would it not be better to simply have the kernel use the boot flag directly as the default CPU mask setting?

[ I can see the argument that "if you need such control, you should set up your distro to deal with it", but that is a level of tweaking I would like to avoid for my customers ]

Thanks, Shaheed

Well, you can get away with it easier by having a program (like taskset in schedutils) run in rc.sysinit. Have it bind init to the appropriate processor.

This is not a bad idea either, as it involves minimal code.

Robert Love

Can I propose an addition?

Not-ready features and speedups
===============================

kernel/fork.c
-------------

- Add ability to restrict the the default CPU affinity mask so that sys_setaffinity() can be used to implement exclusive access to a CPU. Patch exists on LKML.

Why is this useful?

Quoting Andrew Morton <[email protected]>:

Because it allows one to dedicate a CPU to a process. For example, lets say you have a quad processor,and want to run joe-random stuff on CPU 0, but a specialised program on CPUs 1, 2, 3 that does not want to compete with joe-random stuff.

With sys_setaffinity(), one can set the affinity of the special program to 0xe...but the default affinity for all the joe-random stuff is still 0xf (from cpu_online_map)! Since its impractical to to modify every single joe-random executable to set its affinity to 0x1, a way is needed to set the default. The logical place is in init(), a.k.a. kernel/fork.c.

I hope that make sense.

Thanks, Shaheed

Is this related or not to processor shielding used by RedHawk Linux? Here is a link to their page:

http://www.ccur.com/realtime/sys_rdhwklnx.html

I saw a presentation by these guys over a year ago. I'm not sure what they're up to now.

Steven

Quoting Steven Cole <[email protected]>:

Yes, if I correctly read the description of this feature, it seems to be the same thing.

Thanks, that is what I suspected.

There seemed to be quite a bit of interest in this from the other customers, although our facility doesn't presently need this functionality. In the spirit of the "squeaky wheel", I'll squeak softly for them.

"In tightly-coupled symmetric multiprocessing systems such as Concurrent’s iHawk real-time systems, RedHawk Linux allows individual CPUs to be shielded from interrupt processing, daemons, bottom halves, and other Linux tasks. Processor shielding provides a highly deterministic execution environment where interrupt response is guaranteed. RedHawk implements shielding via the industry-accepted shield(1) command."

Steven

Quoting Andrew Morton <[email protected]>:

I forgot to add that the result is the rough equivalent of Digital UNIX's psets and Irix's sysmp for my prurposes at least.

And psets and fencing in Solaris too...

You can get exclusive access with mangling the system call, simply by having init bind itself to the non-exclusive processors on boot.

Try it. Every task will then end up on only the non-exclusive processors. Seems a very simple change to me, and one that can be done in user-space.

You do not even have to modify init, if you do not want. Grab
http://tech9.net/rml/schedutils and put a taskset call in your rc.d

Robert Love

Whoops. s/with/without/

Sorry,

Robert Love

Ah. I think I misread your previous note to me on this...that's why my patch modifies init itself (it does not muck with the syscall in any way). I'll try this as soon as I have my 2.5 multiprocessor back. BTW: what are the plans for getting schedutils (and specifically taskset) into a normal 2.6-based distribution? Can I be reasonably sure that this will happen?

I think it is in Debian (unstable at least).

More important to me is getting it into Red Hat and SuSE. I have heard encouring words from Matt Wilson at Red Hat about schedutils possibly going into Rawhide soon. It would not hurt to let Red Hat/SuSE/whoever know that schedutils is something their customers want.

Both Red Hat and SuSE's kernels have the CPU affinity system calls merged, so you do not need to wait until 2.6 is out to use them.

Robert Love

Quoting Robert Love <[email protected]>:

These are the distros I am interested in too. I knew it was in RH AS/ES, but are you saying it is in RH9.0? That would be good news.

On the technical point, I tried out taskset in rc.sysinit, and as you said, it works just fine. On reflection, I feel that editing rc.sysinit is not the right answer given the confidence/competence level of our customers' typical
sysadmins: but I can see that a carefully crafted rc5.d/S00aaaaa script could set the affinity of the executing shell, and its parent(s) upto init to fix all subsequent rcN.d children in the desired manner.

I do suspect that other commercial users will also baulk at editing rc.sysint, and so have to brew the same rcN.d solution. Now, the rcN.d script hackery would be greatly simplified if taskset had a mode of "set the affinity of the identified process, and all its parent processes upto init". Would you accept a patch to taskset along those lines?

I think that would be a very acceptable, easy to deploy, solution.

Thoughts? Shaheed

No, I am saying with luck it will be in the next RH release.

Good :)

It is racey to do this, so its something that should remain a hack and not part of taskset, I think.

If you do it in rc.d, you don't need to set all the parents. rc.d is the first thing run, so if you do it at the top of the script, nothing else is running. Just put:

taskset <mask> 1
taskset <mask> $$

at the top of rc.d.

Another consideration is modifying init (and hopefully having said changes merged back). Init could call sched_setaffinity() when it is first created, based on a setting in /etc/inittab or a command line parameter passed during boot.

My reservation is against doing it in the kernel. I do not particularly care _how_ its done in user-space.

Robert Love

Hmm. I guess you are thinking of daemons?

Perhaps we are talking at cross purposes. As I understand it the calling chain is:

1. kernel bootstrap
2. /sbin/init
3. bash to run /etc/rc.sysinit
4. bash to run individual /etc/rcN.d/whatever

I feel wary of doing it in 3 as you seem to suggest because I am pretty sure this will intimidate my customers. I am happy to do it in 4 - I can avoid the races by only doing it for the distros I care about.

That leaves options 1 and 2 for a community-wide solution. I guess I haven't quite understood the reluctance to do it in 1 given that:

- we know who owns 1 (!!)

- AFAICS, it isn't conceptual bloat because the utility of the implementation of sys_setaffinity() in 1 is greatly limited by not including this feature.

- its hardly physical bloat because the number of bytes required to implement this is absolutely in the noise, and virtually all __init()ed away.

I have no idea with whom to persue this path, and as I say, I feel that solving this once for each distro is crazy IMHO.

I'm sorry to appear foolish, but as explained above, I genuinely don't understand why this does not belong in the kernel. I would be grateful for elaboration. If I really am being thick, then just ignore me and I'll just solve this for myself using route 4.

In any case, thanks for all the patience and kind suggestions so far - it is appreciated.

Regards, Shaheed

It does not have to be done for each distribution. Modify the SysVinit package directly to support this feature. All init needs to do is bind itself to the allowed processors prior to its first fork(). This can be done as part of the core init package, and thus all distributions automatically reap the benefits.

If init is not modified, then it can be done in rc.d or wherever by hand.

Things which can be done in user-space should be done in user-space. There is absolutely zero reason to do this in the kernel. init can do it.

Submit a patch to the init maintainer to have it bind itself on boot to a given command line value. Maybe I will do this if I find the time...

Robert Love

Oh, one other problem with doing it in the kernel via INIT_TASK:

You end up affining any kernel threads, which you absolutely do not want to do _implicitly_. Maybe explicitly, but certainly not implicitly as a blind consequence.

Doing it via init is really the way to go.

Regards,

Robert Love

OK, that does make sense.

I will have a go myself too...

[srhaque@chiswick srhaque]$ rpm -q --whatprovides /sbin/init SysVinit-2.84-2mdk

and rpmfind.net points to ftp://ftp.cistron.nl/pub/people/miquels/software. I'll drop you a line if I make progress.

Thanks, Shaheed

The NUMA scheduler work can trivially be converted into arbitrary scheduler pools - they are not designed for dynamic modification at the moment, but could be without too much effort I think.

Around 2.5.59 or so I posted a patch to rename them to pools, though we had a few other things to sort out at the time. I might revive it at some point - it's pretty simple.

M.

A while ago, I was banging on about kernel patches and whatnot to implement the equivalent of "pset" functionality. After some shoves/pointers from Robert and Miquel, I found that init(8) has the concept of an initscript(5), so that the trivial solution to my problem is to say:

#

# initscript   Executed by init(8) for every program it
#              wants to spawn like this:
#
#              /bin/sh /etc/initscript <id> <level> <action> <process>
#
taskset -p 1 $$
eval exec "$4"

and then modify the relevant programs to set their affinity to the other processors as required. Bingo!

Hope this helps, Shaheed

Hi,

Following up from http://www.ussg.iu.edu/hypermail/linux/kernel/0304.1/1113.html, here is a prototype patch against 2.5.67 to allow CPUs to be reserved to callers of sys_setaffinity(). It seems to work safely on a uniprocessor, and I'll test it on a multi-processor in a few days time. I'm not requesting inclusion of this patch until then.

Any feedback on form as well as content appreciated...I'm particularly wondering if it is worth stealing sys_setaffinity()/sys_getaffinity() for PID 0 to set/return this value since that would allow user-mode code to dynamically choose a non-restricted processor if required.

Thanks, Shaheed

1. The patch is against a clean 2.5.67

2. There is a new boot flag "restricted_cpus". This takes a bit mask of CPUs.

3. The initial task has it's cpus_allowed mask set to ~restricted_cpus in the fork initialisation code (it cannot be done in TASK_INIT since the value is not static). This should mean that any task that does not override cpus_allowed with sys_setaffinity does not get to run on the restricted processors.

4. The parsing of the boot flag does a sanity check to prevent all available CPUs being restricted.

5. Patch follows...

diff -urN -X dontdiff linux-2.5.67/kernel/fork.c newlinux-2.5.67/kernel/fork.c
@@ -206,6 +206,7 @@
 
 	init_task.rlim[RLIMIT_NPROC].rlim_cur = max_threads/2;
 	init_task.rlim[RLIMIT_NPROC].rlim_max = max_threads/2;
+	init_task.cpus_allowed = ~restricted_cpus;
 }
 
 static struct task_struct *dup_task_struct(struct task_struct *orig)

diff -urN -X dontdiff linux-2.5.67/kernel/sched.c newlinux-2.5.67/kernel/sched.c
@@ -40,6 +40,13 @@
 #endif
 
 /*
+ * The boot flag "restricted_cpus" is negated to form the default per-task
+ * cpus_allowed mask. This has the effect of only allowing processes which
+ * call sys_setaffinity() to run on restricted_cpus.
+ */ 
+unsigned long restricted_cpus = 0;
+
+/*
  * Convert user-nice values [ -20 ... 0 ... 19 ]
  * to static priority [ MAX_RT_PRIO..MAX_PRIO-1 ],
  * and back.

@@ -2338,6 +2345,34 @@
 }
 
 /*
+ * set_restricted_cpus - process boot flag to set restricted_cpus.
+ *
+ * NOTE: This function is a __setup and __init function.
+ */
+int __init set_restricted_cpus(char *options)
+{
+	unsigned long temp;
+	char *value;
+
+	if (!options || !*options)
+		return 0;
+	temp = simple_strtoul(value, &value, 0);
+
+	/*
+	 * Make sure that the user has not restricted all available CPUs,
+	 * otherwise a functioning system won't result!
+	 */
+	if (~temp & cpu_online_map) {
+		restricted_cpus = temp;
+	} else {
+		printk(KERN_ERR "cannot restrict all 0x%lx online CPUs!\n", cpu_online_map);
+	}
+	return 0;
+}
+
+__setup("restricted_cpus=", set_restricted_cpus);
+
+/*
  * migration_thread - this is a highprio system thread that performs
  * thread migration by 'pulling' threads into the target runqueue.
  */

diff -urN -X dontdiff linux-2.5.67/Documentation/kernel-parameters.txt newlinux-2.5.67/Documentation/kernel-parameters.txt
@@ -809,6 +809,10 @@
 
 	reserve=	[KNL,BUGS] Force the kernel to ignore some iomem area
 
+	restricted_cpus=
+	                [KNL] Don't use these CPUs for processes by default.
+			Format: <cpu_mask>
+
 	resume=		[SWSUSP] Specify the partition device for software suspension
 
 	riscom8=	[HW,SERIAL]

diff -urN -X dontdiff linux-2.5.67/include/linux/sched.h newlinux-2.5.67/include/linux/sched.h
@@ -88,6 +88,7 @@
 
 extern int nr_threads;
 extern int last_pid;
+extern unsigned long restricted_cpus;
 DECLARE_PER_CPU(unsigned long, process_counts);
 extern int nr_processes(void);
 extern unsigned long nr_running(void);

Hi,

Can any kind soul with a working .config (no modules, boot from AIC7xxx JBOD disk is fine) for a Dell 2650 for 2.5.6anything kindly send me a copy? Off-list is probably best - I'm trying to get this patch going, but am having an unbeliveable amount of difficulty getting this system booted.

Thanks, Shaheed