SGI Altix cross partition functionality

#include <asm/page.h>
#include <asm/sal.h>
#include <asm/pal.h>

spinlock_t sal_lock __cacheline_aligned = SPIN_LOCK_UNLOCKED;
+EXPORT_SYMBOL(sal_lock);

Quoted 2 lines

cpumask_t node_to_cpu_mask[MAX_NUMNODES] __cacheline_aligned;
+EXPORT_SYMBOL(node_to_cpu_mask);

Isn't there a higher level interface for this?

Quoted 4 lines · Christoph Hellwig

Quoted 2 lines

cpumask_t node_to_cpu_mask[MAX_NUMNODES] __cacheline_aligned;
+EXPORT_SYMBOL(node_to_cpu_mask);

Isn't there a higher level interface for this?

Yeah, but node_to_cpumask() is just a wrapper around this array.

Review end_wait4: · line 1187

Quoted 8 lines

Index: linux/kernel/exit.c
=================================> > --- linux.orig/kernel/exit.c Thu Jun 10 15:04:09 2004
+++ linux/kernel/exit.c Thu Jun 10 16:31:55 2004

(quoted diff) +1 -0

@@ -1184,6 +1184,7 @@ end_wait4:
 	remove_wait_queue(&current->wait_chldexit,&wait);
 	return retval;
 }
+EXPORT_SYMBOL(sys_wait4);

No way. Calling syscalls from kernelspace is always a bad idea. (And this is not linux-ia64 material anyway)

Review asmlinkage long sys_sched_setscheduler(p · line 2817

Quoted 9 lines

Index: linux/kernel/sched.c
=================================> > --- linux.orig/kernel/sched.c Thu Jun 10 16:30:39 2004
+++ linux/kernel/sched.c Thu Jun 10 16:31:55 2004

(quoted diff) +1 -0

@@ -2814,6 +2814,7 @@ asmlinkage long sys_sched_setscheduler(p
 {
 	return setscheduler(pid, policy, param);
 }
+EXPORT_SYMBOL(sys_sched_setscheduler);

Again, don't mess with scheduler paramters from your modules.

Quoted 11 lines

Review end_wait4: · line 1187

Quoted 8 lines

Index: linux/kernel/exit.c
=================================> > > --- linux.orig/kernel/exit.c Thu Jun 10 15:04:09 2004
+++ linux/kernel/exit.c Thu Jun 10 16:31:55 2004

(quoted diff) +1 -0

@@ -1184,6 +1184,7 @@ end_wait4:
 	remove_wait_queue(&current->wait_chldexit,&wait);
 	return retval;
 }
+EXPORT_SYMBOL(sys_wait4);

No way. Calling syscalls from kernelspace is always a bad idea. (And this is not linux-ia64 material anyway)

How should a parent thread be reaping children?

Quoted 11 lines

Review asmlinkage long sys_sched_setscheduler(p · line 2817

Quoted 9 lines

Index: linux/kernel/sched.c
=================================> > > --- linux.orig/kernel/sched.c Thu Jun 10 16:30:39 2004
+++ linux/kernel/sched.c Thu Jun 10 16:31:55 2004

(quoted diff) +1 -0

@@ -2814,6 +2814,7 @@ asmlinkage long sys_sched_setscheduler(p
 {
 	return setscheduler(pid, policy, param);
 }
+EXPORT_SYMBOL(sys_sched_setscheduler);

Again, don't mess with scheduler paramters from your modules.

How should a kernel thread raise itself to real-time priority?

Quoted 26 lines · Robin Holt

Quoted 11 lines

Review end_wait4: · line 1187

Quoted 8 lines

Index: linux/kernel/exit.c
=================================> > > > --- linux.orig/kernel/exit.c Thu Jun 10 15:04:09 2004
+++ linux/kernel/exit.c Thu Jun 10 16:31:55 2004

(quoted diff) +1 -0

@@ -1184,6 +1184,7 @@ end_wait4:
 	remove_wait_queue(&current->wait_chldexit,&wait);
 	return retval;
 }
+EXPORT_SYMBOL(sys_wait4);

No way. Calling syscalls from kernelspace is always a bad idea. (And this is not linux-ia64 material anyway)

How should a parent thread be reaping children?

Quoted 11 lines

Review asmlinkage long sys_sched_setscheduler(p · line 2817

Quoted 9 lines

Index: linux/kernel/sched.c
=================================> > > > --- linux.orig/kernel/sched.c Thu Jun 10 16:30:39 2004
+++ linux/kernel/sched.c Thu Jun 10 16:31:55 2004

(quoted diff) +1 -0

@@ -2814,6 +2814,7 @@ asmlinkage long sys_sched_setscheduler(p
 {
 	return setscheduler(pid, policy, param);
 }
+EXPORT_SYMBOL(sys_sched_setscheduler);

Again, don't mess with scheduler paramters from your modules.

Quoted 1 line

How should a kernel thread raise itself to real-time priority?

Answer to both: it shouldn't

Quoted 6 lines · Dean Nelson

#include <asm/page.h>
#include <asm/sal.h>
#include <asm/pal.h>

spinlock_t sal_lock __cacheline_aligned = SPIN_LOCK_UNLOCKED;
+EXPORT_SYMBOL(sal_lock);

No. Please don't make random sal calls from modules.

Review default_handler (void) · line 44

Quoted 9 lines

(quoted diff) +2 -0

+
 unsigned long sal_platform_features;
 
 unsigned short sal_revision;

@@ -38,6 +41,8 @@ default_handler (void)
 }
 
 ia64_sal_handler ia64_sal = (ia64_sal_handler) default_handler;
+EXPORT_SYMBOL(ia64_sal);

Dito. I think you need to find a better abstraction.

mm/page_alloc.c unsigned long totalram_pages; · line 43

Quoted 7 lines · Dean Nelson

mm/page_alloc.c +1 -0

@@ -40,6 +40,7 @@ unsigned long totalram_pages;
 unsigned long totalhigh_pages;
 int nr_swap_pages;
 int numnodes = 1;
+EXPORT_SYMBOL(numnodes);

bad idea. I have patches to completely kill numnodes, don't try to mess with it. So far it's also completly an architecture-thing.

Quoted 5 lines · Christoph Hellwig

Quoted 4 lines · Dean Nelson

#include <asm/page.h>
#include <asm/sal.h>
#include <asm/pal.h>

Quoted 4 lines

Quoted 2 lines

spinlock_t sal_lock __cacheline_aligned = SPIN_LOCK_UNLOCKED;
+EXPORT_SYMBOL(sal_lock);

No. Please don't make random sal calls from modules.

We're only making one sal call (i.e., SN_SAL_GET_PARTITION_ADDR) to get the address of a partition's reserved page.

Quoted 1 line

No. Please don't make random sal calls from modules.

We're only making one sal call (i.e., SN_SAL_GET_PARTITION_ADDR) to get the address of a partition's reserved page.

In include/asm-ia64/sn/sn_sal.h, there are inline functions that are wrappers for the SAL_CALL() macro, (for example, ia64_sn_get_console_nasid()). Is this what you are looking for? (But even if we did create such a function, we'd still need to export sal_lock and ia64_sal because both are referenced by the SAL_CALL() macro.) I'd hate to put such an abstraction into sal.c, since the functionality of my sal call is ia64/sn specific, and sal.c is generic ia64.

Quoted 2 lines

bad idea. I have patches to completely kill numnodes, don't try to mess with it. So far it's also completly an architecture-thing.

What are you planning to replace numnodes with?

XPC has two functions (xpc_allow_IPI_ops() and xpc_restrict_IPI_ops()) which need to change the protections governing IPI operations and AMO (memory) operations for each node in a partition. What mechanism are you proposing to allow one to iterate through all of the nodes in a partition?

Quoted 4 lines · Dean Nelson

Quoted 2 lines

bad idea. I have patches to completely kill numnodes, don't try to mess with it. So far it's also completly an architecture-thing.

What are you planning to replace numnodes with?

A bitmap of online nodes once pj's nodemask_t is in. That way the silly distinctinon of nasid_t and cnodeid_t values in IP27 and SN2 code can go away.

Quoted 5 lines

XPC has two functions (xpc_allow_IPI_ops() and xpc_restrict_IPI_ops()) which need to change the protections governing IPI operations and AMO (memory) operations for each node in a partition. What mechanism are you proposing to allow one to iterate through all of the nodes in a partition?

I'm not exactly sure. I'll look through once I get a little time.

-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 4 lines · Dean Nelson

Quoted 1 line

No. Please don't make random sal calls from modules.

We're only making one sal call (i.e., SN_SAL_GET_PARTITION_ADDR) to get the address of a partition's reserved page.

Then please provide some accessor from core SN2 code. If that partition addr is constant after bootup a global variable should do it, else a small wrapper function.

Quoted 7 lines

In include/asm-ia64/sn/sn_sal.h, there are inline functions that are wrappers for the SAL_CALL() macro, (for example, ia64_sn_get_console_nasid()). Is this what you are looking for? (But even if we did create such a function, we'd still need to export sal_lock and ia64_sal because both are referenced by the SAL_CALL() macro.) I'd hate to put such an abstraction into sal.c, since the functionality of my sal call is ia64/sn specific, and sal.c is generic ia64.

similar to that, just not inline.

Dean> The question is whether we export ia64_sal and sal_lock? Or change these Dean> inline functions to no longer be inline, move them to a new file (sn_sal.c), Dean> and export the individual function names?

Dean> From my point of view the first solution is better from the standpoint of Dean> the diagnostics folks who write tests that need to make SAL calls. It doesn't Dean> seem appropriate that the linux kernel be the repository of a bunch of ad hoc Dean> and miscellaneous SAL wrapper functions that no one but the diagnostics folks Dean> use.

Dean> I would like to know your opinions on the direction to go with this.

This looks like a lesser of two evils decision. On balance I think that I'd prefer not to open up the floodgates for modules to make arbitrary SAL calls ... so I'd prefer to see you un-inline and export the functions that you need. "sn_sal.c" sounds a plausible name for a file to house these functions.

Quoted 10 lines · Tony

Dean> The question is whether we export ia64_sal and sal_lock? Or change these Dean> inline functions to no longer be inline, move them to a new file (sn_sal.c), Dean> and export the individual function names?

Dean> From my point of view the first solution is better from the standpoint of Dean> the diagnostics folks who write tests that need to make SAL calls. It doesn't Dean> seem appropriate that the linux kernel be the repository of a bunch of ad hoc Dean> and miscellaneous SAL wrapper functions that no one but the diagnostics folks Dean> use.

Quoted 7 lines

Dean> I would like to know your opinions on the direction to go with this.

This looks like a lesser of two evils decision. On balance I think that I'd prefer not to open up the floodgates for modules to make arbitrary SAL calls ... so I'd prefer to see you un-inline and export the functions that you need. "sn_sal.c" sounds a plausible name for a file to house these functions.

This leaves our diags people with a fairly difficult position. They have some online diags which need to excercise parts of the shub. Those activities are currently coded in the PROM (make online and offline diags a lot more consistent) executed via SAL calls.

Would you be alright with changing the inlines into real functions, but still exporting the ia64_sal symbol as well?

Will you accept the online diags people sending sal call additions that no kernel component will ever use? Will that policy remain consistent into the future? This makes our job much more difficult as we move toward not having our own kernel at all, but using standard distribution kernels as they will always lag the community in merging back changes made to the ia64 tree. Again, that argues for having the ia64_sal exported. Please consider those needs when making your decision.

Tony,

I'm interested in your response to what Robin wrote. Care to comment?

And just another data point: we no longer need the sal_lock exported, just ia64_sal, does this affect your thoughts on this matter?

Quoted 5 lines · Dean Nelson

arch/ia64/sn/kernel/setup.c +0 -0

@@ -50,6 +50,7 @@
 #include <asm/sn/sn2/shub.h>
 

arch/ia64/sn/kernel/setup.c line 53

Quoted 2 lines

arch/ia64/sn/kernel/setup.c +1 -0

 DEFINE_PER_CPU(struct pda_s, pda_percpu);
+EXPORT_SYMBOL(per_cpu__pda_percpu);

Umm, no. Poking into kernel per-cpu data like the pda from modules sounds like a bad idea. And if you absolutely must use the proper helpers for exporting it.

Quoted 4 lines · Dean Nelson

Quoted 2 lines

bad idea. I have patches to completely kill numnodes, don't try to mess with it. So far it's also completly an architecture-thing.

What are you planning to replace numnodes with?

A bitmap of online nodes once pj's nodemask_t is in. That way the silly distinctinon of nasid_t and cnodeid_t values in IP27 and SN2 code can go away.

Quoted 4 lines

XPC has two functions (xpc_allow_IPI_ops() and xpc_restrict_IPI_ops()) which need to change the protections governing IPI operations and AMO (memory) operations for each node in a partition. What mechanism are you proposing to allow one to iterate through all of the nodes in a partition?

I'm not exactly sure. I'll look through once I get a little time.

Just an FYI, XPC needs to use the enable_shub_wars_1_1() macro which accesses pda->shub_1_1_found. So we do need to export per_cpu__pda_percpu. I've switched EXPORT_SYMBOL() to EXPORT_PER_CPU_SYMBOL() instead.

Although I agree that the nodemask_t patch is the proper solution to the numnodes issue, I don't feel that waiting for its arrival should prevent my continuing on with my XP[C|NET] patch. So I'm going to resubmit my proposed patch with numnodes exported. This is with the understanding that once the nodemask_t patch is accepted by the community (resulting in the 'EXPORT_SYMBOL(numnodes);' line being deleted), I'll be glade to rework the two XPC functions to use the 'for_each_node()' macro instead.

Quoted 7 lines · Dean Nelson

Although I agree that the nodemask_t patch is the proper solution to the numnodes issue, I don't feel that waiting for its arrival should prevent my continuing on with my XP[C|NET] patch. So I'm going to resubmit my proposed patch with numnodes exported. This is with the understanding that once the nodemask_t patch is accepted by the community (resulting in the 'EXPORT_SYMBOL(numnodes);' line being deleted), I'll be glade to rework the two XPC functions to use the 'for_each_node()' macro instead.

Can you post a current snapshot of your work? As in the last mail numnodes is not the kind if 'interface' we'd like to export, especially with things like node hotplug in the pipe.

Review line 2817

Quoted 6 lines

+++ bk-linux-2.6/kernel/sched.c 2004-08-24 07:29:11.000000000 -0500

(quoted diff) +1 -0

@@ -2814,6 +2814,7 @@
 {
 	return setscheduler(pid, policy, param);
 }
+EXPORT_SYMBOL(sys_sched_setscheduler);

As said previously you're not supposed to mess with this one.

Quoted 5 lines · Christoph Hellwig

Quoted 4 lines · Robin Holt

Quoted 3 lines

Quoted 1 line

+EXPORT_SYMBOL(sys_sched_setscheduler);

Again, don't mess with scheduler paramters from your modules.

Quoted 3 lines

Quoted 1 line

How should a kernel thread raise itself to real-time priority?

Answer to both: it shouldn't

To the second, we found that contention would result in very high latency without raising the priority to real-time levels. What is the proper way to handle having a user thread at the same priority as a kernel thread causing this holdoff?

Quoted 1 line

As said previously you're not supposed to mess with this one.

Yeah, I know, but how is one suppose to deal with the following issue raised by Robin? (Never did get a response from you.)

Quoted 6 lines · Robin Holt

Quoted 5 lines · Christoph Hellwig

Quoted 4 lines · Robin Holt

Quoted 3 lines

Quoted 1 line

+EXPORT_SYMBOL(sys_sched_setscheduler);

Again, don't mess with scheduler paramters from your modules.

Quoted 8 lines

Quoted 3 lines

Quoted 1 line

How should a kernel thread raise itself to real-time priority?

Answer to both: it shouldn't

To the second, we found that contention would result in very high latency without raising the priority to real-time levels. What is the proper way to handle having a user thread at the same priority as a kernel thread causing this holdoff?

The problem arises when enough user processes, which have the same priority as XPC's kthreads, are spinning doing a bit of work mixed with sleeping. Because of the sleep, these processes get a bonus which gives them a higher effective priority than the XPC kthreads. As a result, when cross partition interrupts come in, the XPC kthreads do not get scheduled immediately, but are held off until the end of the user processes' time slice.

This problem was encountered running a legitimate user land work load. I concocted the following program to reproduce the behavior we were seeing. I did this to see if the problem still exists on the 2.6 kernel (we originally saw the problem running on 2.4).

Quoted 29 lines · Dean Nelson

Quoted 1 line

As said previously you're not supposed to mess with this one.

Yeah, I know, but how is one suppose to deal with the following issue raised by Robin? (Never did get a response from you.)

Quoted 6 lines · Robin Holt

Quoted 5 lines · Christoph Hellwig

Quoted 4 lines · Robin Holt

Quoted 3 lines

Quoted 1 line

+EXPORT_SYMBOL(sys_sched_setscheduler);

Again, don't mess with scheduler paramters from your modules.

Quoted 8 lines

Quoted 3 lines

Quoted 1 line

How should a kernel thread raise itself to real-time priority?

Answer to both: it shouldn't

To the second, we found that contention would result in very high latency without raising the priority to real-time levels. What is the proper way to handle having a user thread at the same priority as a kernel thread causing this holdoff?

The problem arises when enough user processes, which have the same priority as XPC's kthreads, are spinning doing a bit of work mixed with sleeping. Because of the sleep, these processes get a bonus which gives them a higher effective priority than the XPC kthreads. As a result, when cross partition interrupts come in, the XPC kthreads do not get scheduled immediately, but are held off until the end of the user processes' time slice.

Quoted 4 lines

This problem was encountered running a legitimate user land work load. I concocted the following program to reproduce the behavior we were seeing. I did this to see if the problem still exists on the 2.6 kernel (we originally saw the problem running on 2.4).

Any reason you're doing that work from thread and not tasklets that get scheduled ASAP?