Skip to contentSkip to conversation

rework shadow CPU affinity

RFC PATCH In review Jan Kiszka 3 participants 11 messages Nov 28, 2006 – Nov 29, 2006 Generic include/asm header files

lore
Share & tools
lore

Jan Kiszka

author
#0 lore raw

RFC PATCH 1/2 rework shadow CPU affinity

Picking up the earlier discussion of CPU affinity of shadow threads, this patch now opens the possibility to assign a CPU mask on xnshadow_map. It cleans up the related nucleus code and updates all callers of xnshadow_map appropriately.

Be warned, it's only tested on (q)emulated SMP. Comments and test reports on real hardware are welcome.

Jan

PS: I would consider these patches for 2.4.

---
 include/asm-generic/system.h |    1 -
 include/nucleus/pod.h        |    3 +++
 include/nucleus/shadow.h     |    4 +++-
 ksrc/nucleus/shadow.c        |   43 ++++++++++++++++++-------------------------
 ksrc/skins/native/syscall.c  |    3 ++-
 ksrc/skins/posix/syscall.c   |    4 ++--
 ksrc/skins/psos+/syscall.c   |    3 ++-
 ksrc/skins/vrtx/syscall.c    |    3 ++-
 ksrc/skins/vxworks/syscall.c |    3 ++-
 9 files changed, 34 insertions(+), 33 deletions(-)
include/asm-generic/system.h +0 -1
Index: xenomai/include/asm-generic/system.h
===================================================================
@@ -38,7 +38,6 @@
 #include <asm/ptrace.h>
 #include <asm/xenomai/hal.h>
 #include <asm/xenomai/atomic.h>
-#include <nucleus/shadow.h>
 
 /* Tracer interface */
 #define xnarch_trace_max_begin(v)		rthal_trace_max_begin(v)
include/nucleus/shadow.h +3 -1
Index: xenomai/include/nucleus/shadow.h
===================================================================
@@ -22,6 +22,7 @@
 
 #include <asm/xenomai/atomic.h>
 #include <asm/xenomai/syscall.h>
+#include <asm/xenomai/system.h>
 #ifdef CONFIG_PROC_FS
 #include <linux/proc_fs.h>
 #endif /* CONFIG_PROC_FS */
@@ -65,7 +66,8 @@ void xnshadow_grab_events(void);
 void xnshadow_release_events(void);
 
 int xnshadow_map(struct xnthread *thread,
-		 xncompletion_t __user *u_completion);
+		 xncompletion_t __user *u_completion,
+		 xnarch_cpumask_t affinity);
 
 void xnshadow_unmap(struct xnthread *thread);
 
ksrc/nucleus/shadow.c +18 -25

87 lines not on this page · raw message

include/nucleus/pod.h +3 -0
Index: xenomai/include/nucleus/pod.h
===================================================================
@@ -33,6 +33,9 @@
 
 #include <nucleus/thread.h>
 #include <nucleus/intr.h>
+#if defined(__KERNEL__) && defined(CONFIG_XENO_OPT_PERVASIVE)
+#include <nucleus/shadow.h>
+#endif /* __KERNEL__  && CONFIG_XENO_OPT_PERVASIVE */
 
 /* Creation flags */
 #define XNREUSE  0x00000001     /* Reuse pod with identical properties */
ksrc/skins/native/syscall.c +2 -1
Index: xenomai/ksrc/skins/native/syscall.c
===================================================================
@@ -173,7 +173,8 @@ static int __rt_task_create(struct task_
 		ph.opaque2 = bulk.a5;	/* hidden pthread_t identifier. */
 		__xn_copy_to_user(curr, (void __user *)bulk.a1, &ph,
 				  sizeof(ph));
-		err = xnshadow_map(&task->thread_base, u_completion);
+		err = xnshadow_map(&task->thread_base, u_completion,
+				   task->affinity);
 	} else {
 		xnfree(task);
 		/* Unblock and pass back error code. */
ksrc/skins/posix/syscall.c +2 -2
Index: xenomai/ksrc/skins/posix/syscall.c
===================================================================
@@ -174,7 +174,7 @@ static int __pthread_create(struct task_
 	if (err)
 		return -err;	/* Conventionally, our error codes are negative. */
 
-	err = xnshadow_map(&k_tid->threadbase, NULL);
+	err = xnshadow_map(&k_tid->threadbase, NULL, XNPOD_ALL_CPUS);
 
 	if (!err && !__pthread_hash(&hkey, k_tid))
 		err = -ENOMEM;
@@ -205,7 +205,7 @@ static pthread_t __pthread_shadow(struct
 	if (err)
 		return ERR_PTR(-err);
 
-	err = xnshadow_map(&k_tid->threadbase, NULL);
+	err = xnshadow_map(&k_tid->threadbase, NULL, XNPOD_ALL_CPUS);
 
 	if (!err && !__pthread_hash(hkey, k_tid))
 		err = -EAGAIN;
ksrc/skins/psos+/syscall.c +2 -1
Index: xenomai/ksrc/skins/psos+/syscall.c
===================================================================
@@ -98,7 +98,8 @@ static int __t_create(struct task_struct
 		tid = xnthread_handle(&task->threadbase);
 		__xn_copy_to_user(curr, (void __user *)__xn_reg_arg4(regs), &tid,
 				  sizeof(tid));
-		err = xnshadow_map(&task->threadbase, u_completion);
+		err = xnshadow_map(&task->threadbase, u_completion,
+				   XNPOD_ALL_CPUS);
 	} else {
 		/* Unblock and pass back error code. */
 
ksrc/skins/vrtx/syscall.c +2 -1
Index: xenomai/ksrc/skins/vrtx/syscall.c
===================================================================
@@ -95,7 +95,8 @@ static int __sc_tecreate(struct task_str
 	} else {
 		__xn_copy_to_user(curr, (void __user *)__xn_reg_arg2(regs),
 				  &tid, sizeof(tid));
-		err = xnshadow_map(&task->threadbase, u_completion);
+		err = xnshadow_map(&task->threadbase, u_completion,
+				   XNPOD_ALL_CPUS);
 	}
 
 	if (err)
ksrc/skins/vxworks/syscall.c +2 -1
Index: xenomai/ksrc/skins/vxworks/syscall.c
===================================================================
@@ -122,7 +122,8 @@ static int __wind_task_init(struct task_
 		ph.handle = xnthread_handle(&task->threadbase);
 		__xn_copy_to_user(curr, (void __user *)__xn_reg_arg2(regs), &ph,
 				  sizeof(ph));
-		err = xnshadow_map(&task->threadbase, u_completion);
+		err = xnshadow_map(&task->threadbase, u_completion,
+				   XNPOD_ALL_CPUS);
 	} else {
 		/* Unblock and pass back error code. */
 
#1 lore raw
Quoted 7 lines · Jan Kiszka

Picking up the earlier discussion of CPU affinity of shadow threads, this patch now opens the possibility to assign a CPU mask on xnshadow_map. It cleans up the related nucleus code and updates all callers of xnshadow_map appropriately.

Be warned, it's only tested on (q)emulated SMP. Comments and test reports on real hardware are welcome.

It is only possible to set the affinity of a user-space thread, why adding another way to do it ?

Signature · 1 line
                                                 Gilles Chanteperdrix
author
#2 lore raw
Quoted 6 lines · Gilles Chanteperdrix
Quoted 5 lines · Jan Kiszka

Picking up the earlier discussion of CPU affinity of shadow threads, this patch now opens the possibility to assign a CPU mask on xnshadow_map. It cleans up the related nucleus code and updates all callers of xnshadow_map appropriately.

Quoted 6 lines
Quoted 2 lines

Be warned, it's only tested on (q)emulated SMP. Comments and test reports on real hardware are welcome.

It is only possible to set the affinity of a user-space thread, why adding another way to do it ?

I'm not yet completely understanding what you mean.

So far the affinity was indirectly assigned to some xnthread by the CPU the caller of xnshadow_map happened to run on. Now this is done explicitly, thus in a more robust way (there was an undefined path in the old code) and without duplicating code (/wrt skins).

Jan

#3 lore raw
Quoted 17 lines · Jan Kiszka
Quoted 7 lines · Gilles Chanteperdrix
Quoted 5 lines · Jan Kiszka

Picking up the earlier discussion of CPU affinity of shadow threads, this patch now opens the possibility to assign a CPU mask on xnshadow_map. It cleans up the related nucleus code and updates all callers of xnshadow_map appropriately.

Quoted 6 lines
Quoted 2 lines

Be warned, it's only tested on (q)emulated SMP. Comments and test reports on real hardware are welcome.

It is only possible to set the affinity of a user-space thread, why adding another way to do it ?

Quoted 6 lines

I'm not yet completely understanding what you mean.

So far the affinity was indirectly assigned to some xnthread by the CPU the caller of xnshadow_map happened to run on. Now this is done explicitly, thus in a more robust way (there was an undefined path in the old code) and without duplicating code (/wrt skins).

I meant "already" instead of "only", it is already possible to set the affinity of a user-space thread with sched_setaffinity.

Signature · 1 line
                                                 Gilles Chanteperdrix
author
#4 lore raw
Quoted 18 lines · Gilles Chanteperdrix
Quoted 17 lines · Jan Kiszka
Quoted 7 lines · Gilles Chanteperdrix
Quoted 5 lines · Jan Kiszka

Picking up the earlier discussion of CPU affinity of shadow threads, this patch now opens the possibility to assign a CPU mask on xnshadow_map. It cleans up the related nucleus code and updates all callers of xnshadow_map appropriately.

Quoted 5 lines
Quoted 2 lines

Be warned, it's only tested on (q)emulated SMP. Comments and test reports on real hardware are welcome.

It is only possible to set the affinity of a user-space thread, why adding another way to do it ?

I'm not yet completely understanding what you mean.

Quoted 8 lines
Quoted 4 lines

So far the affinity was indirectly assigned to some xnthread by the CPU the caller of xnshadow_map happened to run on. Now this is done explicitly, thus in a more robust way (there was an undefined path in the old code) and without duplicating code (/wrt skins).

I meant "already" instead of "only", it is already possible to set the affinity of a user-space thread with sched_setaffinity.

OK. So you are suggesting to read out the affinity mask from task_struct instead of passing it as an additional argument?

#5 lore raw
Quoted 3 lines · Jan Kiszka

OK. So you are suggesting to read out the affinity mask from task_struct instead of passing it as an additional argument?

Unless I am wrong, it is the way it currently works.

Signature · 1 line
                                                 Gilles Chanteperdrix
author
#6 lore raw
Quoted 7 lines · Gilles Chanteperdrix
Quoted 3 lines · Jan Kiszka

OK. So you are suggesting to read out the affinity mask from task_struct instead of passing it as an additional argument?

Unless I am wrong, it is the way it currently works.

Yes, for the posix skin. But native is different due to the spilt-up create/start and its own affinity API. And that caused problems when applying nkaffinity.

Nevertheless, I should try to combine the cleanup of xnshadow_map with passing the mask ALWAYS in task_struct. Let's see how this works out, most likely with less patching.

author
#7 lore raw
Quoted 8 lines · Jan Kiszka
Quoted 6 lines · Gilles Chanteperdrix
Quoted 3 lines · Jan Kiszka

OK. So you are suggesting to read out the affinity mask from task_struct instead of passing it as an additional argument?

Unless I am wrong, it is the way it currently works.

Quoted 7 lines

Yes, for the posix skin. But native is different due to the spilt-up create/start and its own affinity API. And that caused problems when applying nkaffinity.

Nevertheless, I should try to combine the cleanup of xnshadow_map with passing the mask ALWAYS in task_struct. Let's see how this works out, most likely with less patching.

sched_setaffinity is 2.6 only. So we would have to push the mask application into kernel space for the native skin anyway. Hmm.

What are posix skin users supposed to do over 2.4? Upgrade to 2.6? BTW, do you plan to support pthread_attr_setaffinity_np somehow?

Jan

#8 lore raw
Quoted 22 lines · Jan Kiszka
Quoted 11 lines · Jan Kiszka
Quoted 8 lines · Gilles Chanteperdrix
Quoted 3 lines · Jan Kiszka

OK. So you are suggesting to read out the affinity mask from task_struct instead of passing it as an additional argument?

Unless I am wrong, it is the way it currently works.

Quoted 7 lines

Yes, for the posix skin. But native is different due to the spilt-up create/start and its own affinity API. And that caused problems when applying nkaffinity.

Nevertheless, I should try to combine the cleanup of xnshadow_map with passing the mask ALWAYS in task_struct. Let's see how this works out, most likely with less patching.

Quoted 4 lines

sched_setaffinity is 2.6 only. So we would have to push the mask application into kernel space for the native skin anyway. Hmm.

What are posix skin users supposed to do over 2.4?

Right, I did not think about the 2.4 kernel, but I am not sure there are many people using the 2.4 kernel on SMP machines anyway.

As for the native skin, it already allows to pass the affinity to rt_task_create, using T_CPU.

Quoted 1 line

Upgrade to 2.6? BTW, do you plan to support pthread_attr_setaffinity_np somehow?

pthread_attr_setaffinity_np is implemented in kernel space, and the pthread_attr_t is passed unchanged by __wrap_pthread_create to __real_pthread_create. So pthread_attr_setaffinity_np should be already working provided that you use a libc that supports it.

Quoted 3 lines

Jan

Signature · 1 line
                                                 Gilles Chanteperdrix
author
#10 lore raw
In reply to Jan Kiszka

Second try of this patch, now consequently basing the affinity API for xnshadow_map on current->cpus_allowed (affects the native skin). Just like the current code does, this patch restricts a shadow thread to a single (the first) CPU of its affinity mask.

Jan

---
 ksrc/nucleus/shadow.c       |   35 +++++++++++++----------------------
 ksrc/skins/native/syscall.c |    3 +++
 2 files changed, 16 insertions(+), 22 deletions(-)
ksrc/nucleus/shadow.c +13 -22
Index: xenomai/ksrc/nucleus/shadow.c
===================================================================
@@ -409,18 +409,6 @@ static void lostage_handler(void *cookie
 		xnltt_log_event(xeno_ev_lohandler, reqnum, p->comm, p->pid);
 
 		switch (rq->req[reqnum].type) {
-		case LO_START_REQ:
-
-#ifdef CONFIG_SMP
-			if (xnshadow_thread(p))
-				/* Set up the initial task affinity using the
-				   information passed to xnpod_start_thread(). */
-				set_cpus_allowed(p,
-						 xnshadow_thread(p)->affinity);
-#endif /* CONFIG_SMP */
-
-			goto do_wakeup;
-
 		case LO_WAKEUP_REQ:
 
 #ifdef CONFIG_SMP
@@ -443,7 +431,9 @@ static void lostage_handler(void *cookie
 			if (!xnshadow_thread(current))
 				xnpod_renice_root(XNPOD_ROOT_PRIO_BASE);
 #endif /* CONFIG_XENO_OPT_RPIDISABLE */
-		      do_wakeup:
+
+			/* fall through */
+		case LO_START_REQ:
 
 #ifdef CONFIG_XENO_OPT_ISHIELD
 			if (xnshadow_thread(p) &&
@@ -853,22 +843,23 @@ int xnshadow_map(xnthread_t *thread, xnc
 	xnshadow_thrptd(current) = thread;
 	xnpod_suspend_thread(thread, XNRELAX, XN_INFINITE, NULL);
 
+	/* Restrict affinity to a single CPU of the current set */
+	affinity = xnarch_cpumask_of_cpu(
+		xnarch_first_cpu(current->cpus_allowed));
+	set_cpus_allowed(current, affinity);
+
 	if (u_completion) {
 		xnshadow_signal_completion(u_completion, 0);
 		return 0;
 	}
 
-	/* Nobody waits for us, so we may start the shadow immediately
-	   after having forced the CPU affinity to the current
-	   processor. Note that we don't use smp_processor_id() to prevent
-	   kernel debug stuff to yell at us for calling it in a preemptible
-	   section of code. */
-
-	affinity = xnarch_cpumask_of_cpu(rthal_processor_id());
-	set_cpus_allowed(current, affinity);
+	/* Nobody waits for us, so we may start the shadow immediately. */
 
 	mode = thread->rrperiod != XN_INFINITE ? XNRRB : 0;
-	xnpod_start_thread(thread, mode, 0, affinity, NULL, NULL);
+	err = xnpod_start_thread(thread, mode, 0, affinity, NULL, NULL);
+
+	if (err)
+		return err;
 
 	err = xnshadow_harden();
 
ksrc/skins/native/syscall.c +3 -0
Index: xenomai/ksrc/skins/native/syscall.c
===================================================================
@@ -168,6 +168,9 @@ static int __rt_task_create(struct task_
 	err = rt_task_create(task, name, 0, prio, XNFPU | XNSHADOW | mode);
 
 	if (err == 0) {
+		/* Apply CPU affinity */
+		set_cpus_allowed(current, task->affinity);
+
 		/* Copy back the registry handle to the ph struct. */
 		ph.opaque = xnthread_handle(&task->thread_base);
 		ph.opaque2 = bulk.a5;	/* hidden pthread_t identifier. */
#9 lore raw
In reply to Gilles Chanteperdrix
Quoted 23 lines · Gilles Chanteperdrix
Quoted 22 lines · Jan Kiszka
Quoted 18 lines · Jan Kiszka
Quoted 8 lines · Gilles Chanteperdrix
Quoted 3 lines · Jan Kiszka

OK. So you are suggesting to read out the affinity mask from task_struct instead of passing it as an additional argument?

Unless I am wrong, it is the way it currently works.

Yes, for the posix skin. But native is different due to the spilt-up create/start and its own affinity API. And that caused problems when applying nkaffinity.

Nevertheless, I should try to combine the cleanup of xnshadow_map with passing the mask ALWAYS in task_struct. Let's see how this works out, most likely with less patching.

Quoted 8 lines
Quoted 4 lines

sched_setaffinity is 2.6 only. So we would have to push the mask application into kernel space for the native skin anyway. Hmm.

What are posix skin users supposed to do over 2.4?

Right, I did not think about the 2.4 kernel, but I am not sure there are many people using the 2.4 kernel on SMP machines anyway.

Ack. 2.4 backports are provided to help people who do not want to double jump from <some-rtos> to Xenomai _and_ from 2.4 to 2.6 in the same move for upgrading some embedded setup; they can move to Xenomai keeping their good old system software if they wish, then move to 2.6 if they wish so later, once the dust has settled. People interested by multi-cores and/or big irons of some sort should obviously go for 2.6.

Quoted 17 lines

As for the native skin, it already allows to pass the affinity to rt_task_create, using T_CPU.

Quoted 1 line

Upgrade to 2.6? BTW, do you plan to support pthread_attr_setaffinity_np somehow?

pthread_attr_setaffinity_np is implemented in kernel space, and the pthread_attr_t is passed unchanged by __wrap_pthread_create to __real_pthread_create. So pthread_attr_setaffinity_np should be already working provided that you use a libc that supports it.

Quoted 3 lines

Jan

Signature · 1 line
Philippe.