Miscellaneous RCU fixes for 3.5

init/Kconfig config RCU_BOOST_PRIO

Quoted 4 lines · Paul E. McKenney

init/Kconfig +4 -0

+         This option specifies the real-time priority to which long-term
+         preempted RCU readers are to be boosted.  If you are working
+         with a real-time application that has one or more CPU-bound
+         threads running at a real-time priority level,

Then your application is broken ;-) the kernel is known to mis-behave under these circumstances since it doesn't get to run house-keeping tasks. RCU is just one of these and elevating it doesn't make it work.

init/Kconfig config RCU_BOOST_PRIO · line 525

Quoted 5 lines

you should set

init/Kconfig +4 -0

+         RCU_BOOST_PRIO to a priority higher then the highest-priority
+         real-time CPU-bound thread.  The default RCU_BOOST_PRIO value
+         of 1 is appropriate in the common case, which is real-time
+         applications that do not have any CPU-bound threads.

Alternatively, 1 is the worst possible choice forcing people to consider the issue.

init/Kconfig config RCU_BOOST_PRIO · line 536

Quoted 10 lines

init/Kconfig +10 -0

+         Some real-time applications might not have a single real-time
+         thread that saturates a given CPU, but instead might have
+         multiple real-time threads that, taken together, fully utilize
+         that CPU.  In this case, you should set RCU_BOOST_PRIO to
+         a priority higher than the lowest-priority thread that is
+         conspiring to prevent the CPU from running any non-real-time
+         tasks.  For example, if one thread at priority 10 and another
+         thread at priority 5 are between themselves fully consuming
+         the CPU time on a given CPU, then RCU_BOOST_PRIO should be
+         set to priority 6 or higher. 

I'd call this misleading, who's to say that preempting the 5 would yield enough time to complete the RCU work?

This all gets us back to the fun question of RCU delayed bandwidth budgeting.. ideally every 'task' that does call_rcu() should donate some of its budget towards the thread running the callback.

Anyway, I'd argue both the old and new description are bonkers.

Currently, __kfree_rcu() is implemented as an inline function, and contains a BUILD_BUG_ON() that malfunctions if __kfree_rcu() is compiled as an out-of-line function. Unfortunately, there are compiler settings (e.g., -O0) that can result in __kfree_rcu() being compiled out of line, resulting in annoying build breakage. This commit therefore converts both __kfree_rcu() and __is_kfree_rcu_offset() from inline functions to macros to prevent such misbehavior on the part of the compiler.

Quoted 7 lines · Paul E. McKenney

Currently, __kfree_rcu() is implemented as an inline function, and contains a BUILD_BUG_ON() that malfunctions if __kfree_rcu() is compiled as an out-of-line function. Unfortunately, there are compiler settings (e.g., -O0) that can result in __kfree_rcu() being compiled out of line, resulting in annoying build breakage. This commit therefore converts both __kfree_rcu() and __is_kfree_rcu_offset() from inline functions to macros to prevent such misbehavior on the part of the compiler.

The kernel very explicitly doesn't support being compiled with -O0, so this is a non-issue, I think you can make it work if you add -finline-functions.

I'd drop this, either make the entire kernel compile or don't bother.

Quoted 9 lines · Peter Zijlstra

Quoted 7 lines · Paul E. McKenney

Currently, __kfree_rcu() is implemented as an inline function, and contains a BUILD_BUG_ON() that malfunctions if __kfree_rcu() is compiled as an out-of-line function. Unfortunately, there are compiler settings (e.g., -O0) that can result in __kfree_rcu() being compiled out of line, resulting in annoying build breakage. This commit therefore converts both __kfree_rcu() and __is_kfree_rcu_offset() from inline functions to macros to prevent such misbehavior on the part of the compiler.

Quoted 5 lines

The kernel very explicitly doesn't support being compiled with -O0, so this is a non-issue, I think you can make it work if you add -finline-functions.

I'd drop this, either make the entire kernel compile or don't bother.

It was not originally meant to make the entire kernel compile with -O0, but only select modules/.c files.

That, or __always_inline just does not always inline.

From: "Paul E. McKenney" <[email protected]>

Commit #0209f649 (rcu: limit rcu_node leaf-level fanout) set an upper limit of 16 on the leaf-level fanout for the rcu_node tree. This was needed to reduce lock contention that was induced by the synchronization of scheduling-clock interrupts, which was in turn needed to improve energy efficiency for moderate-sized lightly loaded servers.

However, reducing the leaf-level fanout means that there are more leaf-level rcu_node structures in the tree, which in turn means that RCU's grace-period initialization incurs more cache misses. This is not a problem on moderate-sized servers with only a few tens of CPUs, but becomes a major source of real-time latency spikes on systems with many hundreds of CPUs. In addition, the workloads running on these large systems tend to be CPU-bound, which eliminates the energy-efficiency advantages of synchronizing scheduling-clock interrupts. Therefore, these systems need maximal values for the rcu_node leaf-level fanout.

This commit addresses this problem by introducing a new kernel parameter named RCU_FANOUT_LEAF that directly controls the leaf-level fanout. This parameter defaults to 16 to handle the common case of a moderate sized lightly loaded servers, but may be set higher on larger systems.

Quoted 18 lines · Paul E. McKenney

From: "Paul E. McKenney" <[email protected]>

Commit #0209f649 (rcu: limit rcu_node leaf-level fanout) set an upper limit of 16 on the leaf-level fanout for the rcu_node tree. This was needed to reduce lock contention that was induced by the synchronization of scheduling-clock interrupts, which was in turn needed to improve energy efficiency for moderate-sized lightly loaded servers.

However, reducing the leaf-level fanout means that there are more leaf-level rcu_node structures in the tree, which in turn means that RCU's grace-period initialization incurs more cache misses. This is not a problem on moderate-sized servers with only a few tens of CPUs, but becomes a major source of real-time latency spikes on systems with many hundreds of CPUs. In addition, the workloads running on these large systems tend to be CPU-bound, which eliminates the energy-efficiency advantages of synchronizing scheduling-clock interrupts. Therefore, these systems need maximal values for the rcu_node leaf-level fanout.

Quoted 4 lines

This commit addresses this problem by introducing a new kernel parameter named RCU_FANOUT_LEAF that directly controls the leaf-level fanout. This parameter defaults to 16 to handle the common case of a moderate sized lightly loaded servers, but may be set higher on larger systems.

Wouldn't it be much better to match the rcu fanout tree to the physical topology of the machine?

From what I am hearing, doing so requires me to morph the rcu_node tree

Quoted 2 lines · Peter Zijlstra

Wouldn't it be much better to match the rcu fanout tree to the physical topology of the machine?

From what I am hearing, doing so requires me to morph the rcu_node tree at run time. I might eventually become courageous/inspired/senile enough to try this, but not yet. ;-)

Actually, some of this topology shifting seems to me like a firmware bug. Why not arrange the Linux-visible numbering in a way to promote locality for code sequencing through the CPUs?

Quoted 8 lines · Paul E. McKenney

Quoted 2 lines · Peter Zijlstra

Wouldn't it be much better to match the rcu fanout tree to the physical topology of the machine?

From what I am hearing, doing so requires me to morph the rcu_node tree at run time. I might eventually become courageous/inspired/senile enough to try this, but not yet. ;-)

Yes, boot time with possibly some hotplug hooks.

Quoted 3 lines

Actually, some of this topology shifting seems to me like a firmware bug. Why not arrange the Linux-visible numbering in a way to promote locality for code sequencing through the CPUs?

I'm not sure.. but it seems well established on x86 to first enumerate the cores (thread 0) and then the sibling threads (thread 1) -- one 'advantage' is that if you boot with max_cpus=$half you get all cores instead of half the cores.

OTOH it does make linear iteration of the cpus 'funny' :-)

Also, a fanout of 16 is nice when your machine doesn't have HT and has a 2^n core count, but some popular machines these days have 6/10 cores per socket, resulting in your fanout splitting caches.

Quoted 6 lines · Peter Zijlstra

Quoted 5 lines · Paul E. McKenney

Quoted 2 lines · Peter Zijlstra

Wouldn't it be much better to match the rcu fanout tree to the physical topology of the machine?

Quoted 5 lines

Quoted 3 lines

From what I am hearing, doing so requires me to morph the rcu_node tree at run time. I might eventually become courageous/inspired/senile enough to try this, but not yet. ;-)

Yes, boot time with possibly some hotplug hooks.

Has anyone actually measured any slowdown due to the rcu_node structure not matching the topology? (But see also below.)

Quoted 9 lines

Quoted 3 lines

Actually, some of this topology shifting seems to me like a firmware bug. Why not arrange the Linux-visible numbering in a way to promote locality for code sequencing through the CPUs?

I'm not sure.. but it seems well established on x86 to first enumerate the cores (thread 0) and then the sibling threads (thread 1) -- one 'advantage' is that if you boot with max_cpus=$half you get all cores instead of half the cores.

Quoted 1 line

OTOH it does make linear iteration of the cpus 'funny' :-)

Like I said, firmware bug. Seems like the fix should be there as well. Perhaps there needs to be two CPU numberings, one for people wanting whole cores and another for people who want cache locality. Yes, this could be confusing, but keep in mind that you are asking every kernel subsystem to keep its own version of the cache-locality numbering, and that will be even more confusing.

Quoted 3 lines

Also, a fanout of 16 is nice when your machine doesn't have HT and has a 2^n core count, but some popular machines these days have 6/10 cores per socket, resulting in your fanout splitting caches.

That is easy. Such systems can set CONFIG_RCU_FANOUT to 6, 12, 10, or 20, depending on preference. With a patch intended for 3.6, they could set the smallest reasonable value at build time and adjust to the hardware using the boot parameter.

http://www.gossamer-threads.com/lists/linux/kernel/1524864

I expect to make other similar changes over time, but will be proceeding cautiously.

I can very easily give you the size (nr cpus in) a node, still as long as you iterate the cpu space linearly that's not going to be much help.

Quoted 5 lines · Peter Zijlstra

I can very easily give you the size (nr cpus in) a node, still as long as you iterate the cpu space linearly that's not going to be much help.

Oh, I forgot, the numa masks etc are available, depending on CONFIG_NUMA, as cpumask_of_node(n).

Quoted 6 lines · Peter Zijlstra

Quoted 5 lines · Peter Zijlstra

I can very easily give you the size (nr cpus in) a node, still as long as you iterate the cpu space linearly that's not going to be much help.

Quoted 2 lines

Oh, I forgot, the numa masks etc are available, depending on CONFIG_NUMA, as cpumask_of_node(n).

These change with each CPU-hotplug operation? Or is a given CPU hotplug operation guaranteed to change only those node masks that the CPU was (or will be, in the case of online) a member?

Quoted 11 lines · Paul E. McKenney

Quoted 10 lines · Peter Zijlstra

Quoted 8 lines · Paul E. McKenney

Quoted 2 lines · Peter Zijlstra

Wouldn't it be much better to match the rcu fanout tree to the physical topology of the machine?

From what I am hearing, doing so requires me to morph the rcu_node tree at run time. I might eventually become courageous/inspired/senile enough to try this, but not yet. ;-)

Quoted 4 lines

Quoted 1 line

Yes, boot time with possibly some hotplug hooks.

Has anyone actually measured any slowdown due to the rcu_node structure not matching the topology? (But see also below.)

Nope, I'm just whinging ;-)

Quoted 9 lines

Quoted 9 lines

Quoted 3 lines

Actually, some of this topology shifting seems to me like a firmware bug. Why not arrange the Linux-visible numbering in a way to promote locality for code sequencing through the CPUs?

I'm not sure.. but it seems well established on x86 to first enumerate the cores (thread 0) and then the sibling threads (thread 1) -- one 'advantage' is that if you boot with max_cpus=$half you get all cores instead of half the cores.

Quoted 8 lines

Quoted 1 line

OTOH it does make linear iteration of the cpus 'funny' :-)

Like I said, firmware bug. Seems like the fix should be there as well. Perhaps there needs to be two CPU numberings, one for people wanting whole cores and another for people who want cache locality. Yes, this could be confusing, but keep in mind that you are asking every kernel subsystem to keep its own version of the cache-locality numbering, and that will be even more confusing.

I really don't see why it would matter, as far I care they're completely randomized on boot, its all done using cpu-bitmasks anyway.

Suppose the linear thing would have threads/cores/cache continuity like you want, that still leaves the node interconnects, and we all know people love to be creative with those, no way you're going to fold that into a linear scheme :-)

Anyway, as it currently stands I can offer you: cpus_share_cache(), which will return true/false depending on if the two cpus do indeed share a cache.

On top of that there's node_distance(), which will (hopefully) reflect the interconnect topology between nodes.

Using these you can construct enough of the topology layout to be useful. NOTE: node topologies don't need to be symmetric!

Quoted 9 lines

Quoted 3 lines

Also, a fanout of 16 is nice when your machine doesn't have HT and has a 2^n core count, but some popular machines these days have 6/10 cores per socket, resulting in your fanout splitting caches.

That is easy. Such systems can set CONFIG_RCU_FANOUT to 6, 12, 10, or 20, depending on preference. With a patch intended for 3.6, they could set the smallest reasonable value at build time and adjust to the hardware using the boot parameter.

Quoted 4 lines

http://www.gossamer-threads.com/lists/linux/kernel/1524864

I expect to make other similar changes over time, but will be proceeding cautiously.

I can very easily give you the size (nr cpus in) a node, still as long as you iterate the cpu space linearly that's not going to be much help.

I can also offer you access to the scheduler topology if you want.. I've got the below patch pending which should (hopefully) improve the scheduler's node topology -- it implements that detection based on node_distance().

I've tested it using some fake-numa hacks to feed it the distance table of an AMD quad-socket Magny-Cours:

"numa=fake=8:10,16,16,22,16,22,16,22,

              16,10,22,16,22,16,22,16,
              16,22,10,16,16,22,16,22,
              22,16,16,10,22,16,22,16,
              16,22,16,22,10,16,16,22,
              22,16,22,16,16,10,22,16,
              16,22,16,22,16,22,10,16,
              22,16,22,16,22,16,16,10"

---
Subject: sched: Rewrite the CONFIG_NUMA sched domain support.
From: Peter Zijlstra <[email protected]>
Date: Tue Apr 17 15:49:36 CEST 2012

The current code groups up to 16 nodes in a level and then puts an ALLNODES domain spanning the entire tree on top of that. This doesn't reflect the numa topology and esp for the smaller not-fully-connected machines out there today this might make a difference.

Therefore, build a proper numa topology based on node_distance().

TODO: figure out a way to set SD_flags based on distance such that
we disable various expensive load-balancing features at some point and increase the balance interval prop. to the distance.

Cc Anton Blanchard <[email protected]>

Cc Benjamin Herrenschmidt <[email protected]>

Cc Chris Metcalf <[email protected]>

Cc David Howells <[email protected]>

Cc "David S. Miller" <[email protected]>

Cc Fenghua Yu <[email protected]>

Cc "H. Peter Anvin" <[email protected]>

Cc Ingo Molnar <[email protected]>

Cc Ivan Kokshaysky <[email protected]>

Cc [email protected]

Cc [email protected]

Cc [email protected]

Cc [email protected]

Cc [email protected]

Cc [email protected]

Cc Matt Turner <[email protected]>

Cc Paul Mackerras <[email protected]>

Cc Paul Mundt <[email protected]>

Cc Ralf Baechle <[email protected]>

Cc Richard Henderson <[email protected]>

Cc [email protected]

Cc Thomas Gleixner <[email protected]>

Cc Tony Luck <[email protected]>

Cc [email protected]

Signed-off-by Peter Zijlstra <[email protected]>

Link http://lkml.kernel.org/n/[email protected] open here

---
 arch/powerpc/include/asm/topology.h |    6 
 arch/x86/include/asm/topology.h     |   38 -----
 include/linux/topology.h            |   36 -----
 kernel/sched/core.c                 |  253 ++++++++++++++++++++++--------------
 4 files changed, 158 insertions(+), 175 deletions(-)

Index: linux-2.6/include/linux/topology.h
===================================================================

include/linux/topology.h +0 -36

@@ -176,48 +176,12 @@ int arch_update_cpu_topology(void);
 }
 #endif
 
-/* sched_domains SD_ALLNODES_INIT for NUMA machines */
-#define SD_ALLNODES_INIT (struct sched_domain) {			\
-	.min_interval		= 64,					\
-	.max_interval		= 64*num_online_cpus(),			\
-	.busy_factor		= 128,					\
-	.imbalance_pct		= 133,					\
-	.cache_nice_tries	= 1,					\
-	.busy_idx		= 3,					\
-	.idle_idx		= 3,					\
-	.flags			= 1*SD_LOAD_BALANCE			\
-				| 1*SD_BALANCE_NEWIDLE			\
-				| 0*SD_BALANCE_EXEC			\
-				| 0*SD_BALANCE_FORK			\
-				| 0*SD_BALANCE_WAKE			\
-				| 0*SD_WAKE_AFFINE			\
-				| 0*SD_SHARE_CPUPOWER			\
-				| 0*SD_POWERSAVINGS_BALANCE		\
-				| 0*SD_SHARE_PKG_RESOURCES		\
-				| 1*SD_SERIALIZE			\
-				| 0*SD_PREFER_SIBLING			\
-				,					\
-	.last_balance		= jiffies,				\
-	.balance_interval	= 64,					\
-}
-
-#ifndef SD_NODES_PER_DOMAIN
-#define SD_NODES_PER_DOMAIN 16
-#endif
-
 #ifdef CONFIG_SCHED_BOOK
 #ifndef SD_BOOK_INIT
 #error Please define an appropriate SD_BOOK_INIT in include/asm/topology.h!!!
 #endif
 #endif /* CONFIG_SCHED_BOOK */
 
-#ifdef CONFIG_NUMA
-#ifndef SD_NODE_INIT
-#error Please define an appropriate SD_NODE_INIT in include/asm/topology.h!!!
-#endif
-
-#endif /* CONFIG_NUMA */
-
 #ifdef CONFIG_USE_PERCPU_NUMA_NODE_ID
 DECLARE_PER_CPU(int, numa_node);
 

Index: linux-2.6/kernel/sched/core.c
===================================================================

kernel/sched/core.c +86 -95

@@ -5560,7 +5560,8 @@ static int sched_domain_debug_one(struct
 			break;
 		}
 
-		if (cpumask_intersects(groupmask, sched_group_cpus(group))) {
+		if (!(sd->flags & SD_OVERLAP) &&
+		    cpumask_intersects(groupmask, sched_group_cpus(group))) {
 			printk(KERN_CONT "\n");
 			printk(KERN_ERR "ERROR: repeated CPUs\n");
 			break;

@@ -5898,92 +5899,6 @@ static int __init isolated_cpu_setup(cha
 
 __setup("isolcpus=", isolated_cpu_setup);
 
-#ifdef CONFIG_NUMA
-
-/**
- * find_next_best_node - find the next node to include in a sched_domain
- * @node: node whose sched_domain we're building
- * @used_nodes: nodes already in the sched_domain
- *
- * Find the next node to include in a given scheduling domain. Simply
- * finds the closest node not already in the @used_nodes map.
- *
- * Should use nodemask_t.
- */
-static int find_next_best_node(int node, nodemask_t *used_nodes)
-{
-	int i, n, val, min_val, best_node = -1;
-
-	min_val = INT_MAX;
-
-	for (i = 0; i < nr_node_ids; i++) {
-		/* Start at @node */
-		n = (node + i) % nr_node_ids;
-
-		if (!nr_cpus_node(n))
-			continue;
-
-		/* Skip already used nodes */
-		if (node_isset(n, *used_nodes))
-			continue;
-
-		/* Simple min distance search */
-		val = node_distance(node, n);
-
-		if (val < min_val) {
-			min_val = val;
-			best_node = n;
-		}
-	}
-
-	if (best_node != -1)
-		node_set(best_node, *used_nodes);
-	return best_node;
-}
-
-/**
- * sched_domain_node_span - get a cpumask for a node's sched_domain
- * @node: node whose cpumask we're constructing
- * @span: resulting cpumask
- *
- * Given a node, construct a good cpumask for its sched_domain to span. It
- * should be one that prevents unnecessary balancing, but also spreads tasks
- * out optimally.
- */
-static void sched_domain_node_span(int node, struct cpumask *span)
-{
-	nodemask_t used_nodes;
-	int i;
-
-	cpumask_clear(span);
-	nodes_clear(used_nodes);
-
-	cpumask_or(span, span, cpumask_of_node(node));
-	node_set(node, used_nodes);
-
-	for (i = 1; i < SD_NODES_PER_DOMAIN; i++) {
-		int next_node = find_next_best_node(node, &used_nodes);
-		if (next_node < 0)
-			break;
-		cpumask_or(span, span, cpumask_of_node(next_node));
-	}
-}
-
-static const struct cpumask *cpu_node_mask(int cpu)
-{
-	lockdep_assert_held(&sched_domains_mutex);
-
-	sched_domain_node_span(cpu_to_node(cpu), sched_domains_tmpmask);
-
-	return sched_domains_tmpmask;
-}
-
-static const struct cpumask *cpu_allnodes_mask(int cpu)
-{
-	return cpu_possible_mask;
-}
-#endif /* CONFIG_NUMA */
-
 static const struct cpumask *cpu_cpu_mask(int cpu)
 {
 	return cpumask_of_node(cpu_to_node(cpu));

@@ -6020,6 +5935,7 @@ struct sched_domain_topology_level {
 	sched_domain_init_f init;
 	sched_domain_mask_f mask;
 	int		    flags;
+	int		    numa_level;
 	struct sd_data      data;
 };
 

@@ -6211,10 +6127,6 @@ sd_init_##type(struct sched_domain_topol
 }
 
 SD_INIT_FUNC(CPU)
-#ifdef CONFIG_NUMA
- SD_INIT_FUNC(ALLNODES)
- SD_INIT_FUNC(NODE)
-#endif
 #ifdef CONFIG_SCHED_SMT
  SD_INIT_FUNC(SIBLING)
 #endif

@@ -6336,15 +6248,164 @@ static struct sched_domain_topology_leve
 	{ sd_init_BOOK, cpu_book_mask, },
 #endif
 	{ sd_init_CPU, cpu_cpu_mask, },
-#ifdef CONFIG_NUMA
-	{ sd_init_NODE, cpu_node_mask, SDTL_OVERLAP, },
-	{ sd_init_ALLNODES, cpu_allnodes_mask, },
-#endif
 	{ NULL, },
 };
 
 static struct sched_domain_topology_level *sched_domain_topology = default_topology;
 
+#ifdef CONFIG_NUMA
+
+static int sched_domains_numa_levels;
+static int sched_domains_numa_scale;
+static int *sched_domains_numa_distance;
+static struct cpumask ** __percpu sched_domains_numa_masks;
+static int sched_domains_curr_level;
+
+#define NUMA_SCALE(x, level)	\
+
+static inline unsigned long numa_scale(unsigned long x, int level)
+{
+	return x * sched_domains_numa_distance[level] / sched_domains_numa_scale;
+}
+
+static inline int sd_local_flags(int level)
+{
+	if (sched_domains_numa_distance[level] > REMOTE_DISTANCE)
+		return 0;
+
+	return SD_BALANCE_EXEC | SD_BALANCE_FORK | SD_WAKE_AFFINE;
+}
+
+static struct sched_domain *
+sd_init_NUMA(struct sched_domain_topology_level *tl, int cpu)
+{
+	struct sched_domain *sd = *per_cpu_ptr(tl->data.sd, cpu);
+	int level = tl->numa_level;
+	int sd_weight = cpumask_weight(per_cpu_ptr(
+				sched_domains_numa_masks[level],
+				cpu));
+
+	*sd = (struct sched_domain){
+		.min_interval		= sd_weight,
+		.max_interval		= 2*sd_weight,
+		.busy_factor		= 32,
+		.imbalance_pct		= 100 + numa_scale(25, level),
+		.cache_nice_tries	= 2,
+		.busy_idx		= 3,
+		.idle_idx		= 2,
+		.newidle_idx		= 0,
+		.wake_idx		= 0,
+		.forkexec_idx		= 0,
+
+		.flags			= 1*SD_LOAD_BALANCE
+					| 1*SD_BALANCE_NEWIDLE
+					| 0*SD_BALANCE_EXEC
+					| 0*SD_BALANCE_FORK
+					| 0*SD_BALANCE_WAKE
+					| 0*SD_WAKE_AFFINE
+					| 0*SD_PREFER_LOCAL
+					| 0*SD_SHARE_CPUPOWER
+					| 0*SD_POWERSAVINGS_BALANCE
+					| 0*SD_SHARE_PKG_RESOURCES
+					| 1*SD_SERIALIZE
+					| 0*SD_PREFER_SIBLING
+					| sd_local_flags(level)
+					,
+		.last_balance		= jiffies,
+		.balance_interval	= sd_weight,
+	};
+	SD_INIT_NAME(sd, NUMA);
+	sd->private = &tl->data;
+
+	/*
+	 * Ugly hack to pass state to sd_numa_mask()...
+	 */
+	sched_domains_curr_level = tl->numa_level;
+
+	return sd;
+}
+
+static const struct cpumask *sd_numa_mask(int cpu)
+{
+	return per_cpu_ptr(sched_domains_numa_masks[sched_domains_curr_level], cpu);
+}
+
+static void sched_init_numa(void)
+{
+	int next_distance, curr_distance = node_distance(0, 0);
+	struct sched_domain_topology_level *tl;
+	int level = 0;
+	int i, j, k;

kernel/sched/core.c +13 -0

+
+	sched_domains_numa_masks = kzalloc(sizeof(void *) * level, GFP_KERNEL);
+	if (!sched_domains_numa_masks)
+		return;
+
+	for (i = 0; i < level; i++) {
+		sched_domains_numa_masks[i] = alloc_percpu(cpumask_t);
+		if (!sched_domains_numa_masks[i])
+			return;
+
+		for_each_possible_cpu(j) {
+			struct cpumask *mask =
+				per_cpu_ptr(sched_domains_numa_masks[i], j);

kernel/sched/core.c +17 -0

+			.init = sd_init_NUMA,
+			.mask = sd_numa_mask,
+			.flags = SDTL_OVERLAP,
+			.numa_level = j,
+		};
+	}
+
+	sched_domain_topology = tl;
+}
+#else
+static inline void sched_init_numa(void)
+{
+}
+#endif /* CONFIG_NUMA */
+
 static int __sdt_alloc(const struct cpumask *cpu_map)
 {
 	struct sched_domain_topology_level *tl;

@@ -6828,6 +6889,8 @@ void __init sched_init_smp(void)
 	alloc_cpumask_var(&non_isolated_cpus, GFP_KERNEL);
 	alloc_cpumask_var(&fallback_doms, GFP_KERNEL);
 
+	sched_init_numa();
+
 	get_online_cpus();
 	mutex_lock(&sched_domains_mutex);
 	init_sched_domains(cpu_active_mask);

Index: linux-2.6/arch/powerpc/include/asm/topology.h
===================================================================

arch/powerpc/include/asm/topology.h +0 -6

@@ -18,12 +18,6 @@ struct device_node;
  */
 #define RECLAIM_DISTANCE 10
 
-/*
- * Avoid creating an extra level of balancing (SD_ALLNODES) on the largest
- * POWER7 boxes which have a maximum of 32 nodes.
- */
-#define SD_NODES_PER_DOMAIN 32
-
 #include <asm/mmzone.h>
 
 static inline int cpu_to_node(int cpu)

Index: linux-2.6/arch/x86/include/asm/topology.h
===================================================================

arch/x86/include/asm/topology.h +0 -39

@@ -92,44 +92,6 @@ extern void setup_node_to_cpumask_map(vo
 
 #define pcibus_to_node(bus) __pcibus_to_node(bus)
 
-#ifdef CONFIG_X86_32
-# define SD_CACHE_NICE_TRIES	1
-# define SD_IDLE_IDX		1
-#else
-# define SD_CACHE_NICE_TRIES	2
-# define SD_IDLE_IDX		2
-#endif
-
-/* sched_domains SD_NODE_INIT for NUMA machines */
-#define SD_NODE_INIT (struct sched_domain) {				\
-	.min_interval		= 8,					\
-	.max_interval		= 32,					\
-	.busy_factor		= 32,					\
-	.imbalance_pct		= 125,					\
-	.cache_nice_tries	= SD_CACHE_NICE_TRIES,			\
-	.busy_idx		= 3,					\
-	.idle_idx		= SD_IDLE_IDX,				\
-	.newidle_idx		= 0,					\
-	.wake_idx		= 0,					\
-	.forkexec_idx		= 0,					\
-									\
-	.flags			= 1*SD_LOAD_BALANCE			\
-				| 1*SD_BALANCE_NEWIDLE			\
-				| 1*SD_BALANCE_EXEC			\
-				| 1*SD_BALANCE_FORK			\
-				| 0*SD_BALANCE_WAKE			\
-				| 1*SD_WAKE_AFFINE			\
-				| 0*SD_PREFER_LOCAL			\
-				| 0*SD_SHARE_CPUPOWER			\
-				| 0*SD_POWERSAVINGS_BALANCE		\
-				| 0*SD_SHARE_PKG_RESOURCES		\
-				| 1*SD_SERIALIZE			\
-				| 0*SD_PREFER_SIBLING			\
-				,					\
-	.last_balance		= jiffies,				\
-	.balance_interval	= 1,					\
-}
-
 extern int __node_distance(int, int);
 #define node_distance(a, b) __node_distance(a, b)
 
 
 
--

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Interesting! A few probably clueless questions and comments below.

Quoted 6 lines

kernel/sched/core.c +6 -0

+static void sched_init_numa(void)
+{
+     int next_distance, curr_distance = node_distance(0, 0);
+     struct sched_domain_topology_level *tl;
+     int level = 0;
+     int i, j, k;

kernel/sched/core.c static struct sched_domain_topology_leve · line 6348

Quoted 10 lines

kernel/sched/core.c +10 -0

+
+     sched_domains_numa_scale = curr_distance;
+     sched_domains_numa_distance = kzalloc(sizeof(int) * nr_node_ids, GFP_KERNEL);
+     if (!sched_domains_numa_distance)
+             return;
+
+     next_distance = curr_distance;
+     for (i = 0; i < nr_node_ids; i++) {
+             for (j = 0; j < nr_node_ids; j++) {
+                     int distance = node_distance(0, j);

Shouldn't this be "node_distance(i, j)"?

Actually, I don't see any use of "i" in this loop anywhere, which seems strange.

kernel/sched/core.c static struct sched_domain_topology_leve · line 6356

Quoted 8 lines

kernel/sched/core.c +8 -0

+                     if (distance > curr_distance &&
+                                     (distance < next_distance ||
+                                      next_distance == curr_distance))
+                             next_distance = distance;
+             }
+             if (next_distance != curr_distance) {
+                     sched_domains_numa_distance[level++] = next_distance;
+                     sched_domains_numa_levels = level;

I don't understand the intent here. One approach would be to have one sched_domains_numa_distance[] array per node, with distances in each array sorted by increasing distance from the array's node.

As it is, I believe you get N copies of the sorted distances from node 0 in an array that is only guaranteed to be big enough for 1 copy. Unless you know something about the node_distance() matrix that limits the number of distinct values.

kernel/sched/core.c static struct sched_domain_topology_leve · line 6359

Quoted 3 lines

kernel/sched/core.c +3 -0

+                     curr_distance = next_distance;
+             } else break;
+     }

The above loop seems to be doing a selection sort eliminating duplicates.

Would it make sense to put a real sort implementation into lib/?

Quoted 18 lines

kernel/sched/core.c +18 -0

+
+     sched_domains_numa_masks = kzalloc(sizeof(void *) * level, GFP_KERNEL);
+     if (!sched_domains_numa_masks)
+             return;
+
+     for (i = 0; i < level; i++) {
+             sched_domains_numa_masks[i] = alloc_percpu(cpumask_t);
+             if (!sched_domains_numa_masks[i])
+                     return;
+
+             for_each_possible_cpu(j) {
+                     struct cpumask *mask =
+                             per_cpu_ptr(sched_domains_numa_masks[i], j);
+
+                     for (k = 0; k < nr_node_ids; k++) {
+                             if (node_distance(cpu_to_node(j), k) >
+                                             sched_domains_numa_distance[i])
+                                     continue;

kernel/sched/core.c static struct sched_domain_topology_leve · line 6382

Quoted 5 lines

kernel/sched/core.c +5 -0

+
+                             cpumask_or(mask, mask, cpumask_of_node(k));
+                     }
+             }
+     }

OK, if we can assume that the node distances are symmetric, so that node_distance(i, j) == node_distance(j, i),

I think I see what you
are getting at, though I am having a hard time seeing how to pack it into a linear array.

The idea seems to be to compute a per-CPU list of CPU masks, with the first entry having bits set for the CPUs closest to the CPU corresponding to the list, and subsequent entries adding more-distant CPUs. The last CPU mask would presumably have bits set for all CPUs.

I take it that there is no data structure listing per-node CPU masks, indicating which CPUs are members of a given node? Or is something else going on here?

Quoted 5 lines

kernel/sched/core.c +5 -0

+
+     tl = kzalloc((ARRAY_SIZE(default_topology) + level) *
+                     sizeof(struct sched_domain_topology_level), GFP_KERNEL);
+     if (!tl)
+             return;

kernel/sched/core.c static struct sched_domain_topology_leve · line 6393

Quoted 6 lines

kernel/sched/core.c +6 -0

+
+     for (i = 0; default_topology[i].init; i++)
+             tl[i] = default_topology[i];
+
+     for (j = 0; j < level; i++, j++) {
+             tl[i] = (struct sched_domain_topology_level){

tl[j]?

Quoted 9 lines

kernel/sched/core.c +9 -0

+                     .init = sd_init_NUMA,
+                     .mask = sd_numa_mask,
+                     .flags = SDTL_OVERLAP,
+                     .numa_level = j,
+             };
+     }
+
+     sched_domain_topology = tl;
+} 

On Thu, 2012-04-26 at 13:28 -0700, Paul E. McKenney wrote:

Quoted 3 lines

I think I see what you
are getting at, though I am having a hard time seeing how to pack it into a linear array.

Yeah, I'm not sure you can either. Hence me building a tree ;-) But you too have a tree, its tree-rcu after all.

Quoted 4 lines

The idea seems to be to compute a per-CPU list of CPU masks, with the first entry having bits set for the CPUs closest to the CPU corresponding to the list, and subsequent entries adding more-distant CPUs. The last CPU mask would presumably have bits set for all CPUs.

Indeed. So the scheduler already knows about nodes (included in the default_topology thing), here we're constructing masks spanning nodes based on distance.

So the first level is all nodes that are directly connected, the second level are all nodes that have one intermediate hop, etc.. with indeed the last level being the entire machine.

Quoted 3 lines

I take it that there is no data structure listing per-node CPU masks, indicating which CPUs are members of a given node? Or is something else going on here?

There is, its cpumask_of_node(), you'll find it used in the above code :-) We do the for_each_cpu loop because we need the mask per-node and there's no such thing as per-node memory so we fudge it using per-cpu memory.

This could be optimized to reduce overhead if this all turns out to work well.

So in short: for every 'i < level', for every cpu, we build a mask of which cpus are '<= i' hops away from our current node.

Quoted 5 lines

Quoted 5 lines

kernel/sched/core.c +5 -0

+
+     tl = kzalloc((ARRAY_SIZE(default_topology) + level) *
+                     sizeof(struct sched_domain_topology_level), GFP_KERNEL);
+     if (!tl)
+             return;

Quoted 8 lines

kernel/sched/core.c static struct sched_domain_topology_leve · line 6393

Quoted 6 lines

kernel/sched/core.c +6 -0

+
+     for (i = 0; default_topology[i].init; i++)
+             tl[i] = default_topology[i];
+
+     for (j = 0; j < level; i++, j++) {
+             tl[i] = (struct sched_domain_topology_level){

tl[j]?

No, [i]. See how we allocate an array of ARRAY_SIZE(default_topology) + level, then copy the default topology array then continue i by j additional levels.

Quoted 9 lines

Quoted 9 lines

kernel/sched/core.c +9 -0

+                     .init = sd_init_NUMA,
+                     .mask = sd_numa_mask,
+                     .flags = SDTL_OVERLAP,
+                     .numa_level = j,
+             };
+     }
+
+     sched_domain_topology = tl;
+} 

From: "Paul E. McKenney" <[email protected]>

Commit #0209f649 (rcu: limit rcu_node leaf-level fanout) set an upper limit of 16 on the leaf-level fanout for the rcu_node tree. This was needed to reduce lock contention that was induced by the synchronization of scheduling-clock interrupts, which was in turn needed to improve energy efficiency for moderate-sized lightly loaded servers.

However, reducing the leaf-level fanout means that there are more leaf-level rcu_node structures in the tree, which in turn means that RCU's grace-period initialization incurs more cache misses. This is not a problem on moderate-sized servers with only a few tens of CPUs,

Quoted 8 lines · Paul E. McKenney

From: "Paul E. McKenney" <[email protected]>

Commit #0209f649 (rcu: limit rcu_node leaf-level fanout) set an upper limit of 16 on the leaf-level fanout for the rcu_node tree. This was needed to reduce lock contention that was induced by the synchronization of scheduling-clock interrupts, which was in turn needed to improve energy efficiency for moderate-sized lightly loaded servers.

Quoted 4 lines

However, reducing the leaf-level fanout means that there are more leaf-level rcu_node structures in the tree, which in turn means that RCU's grace-period initialization incurs more cache misses. This is not a problem on moderate-sized servers with only a few tens of CPUs,

With a distro config (4096 CPUs) interrupt latency is bad even on a quad. Traversing empty nodes taking locks and cache misses hurts.

Quoted 14 lines · Mike Galbraith

Quoted 12 lines · Paul E. McKenney

From: "Paul E. McKenney" <[email protected]>

Commit #0209f649 (rcu: limit rcu_node leaf-level fanout) set an upper limit of 16 on the leaf-level fanout for the rcu_node tree. This was needed to reduce lock contention that was induced by the synchronization of scheduling-clock interrupts, which was in turn needed to improve energy efficiency for moderate-sized lightly loaded servers.

However, reducing the leaf-level fanout means that there are more leaf-level rcu_node structures in the tree, which in turn means that RCU's grace-period initialization incurs more cache misses. This is not a problem on moderate-sized servers with only a few tens of CPUs,

Quoted 2 lines

With a distro config (4096 CPUs) interrupt latency is bad even on a quad. Traversing empty nodes taking locks and cache misses hurts.

Agreed -- and I will be working on an additional patch that makes RCU avoid initializing its data structures for CPUs that don't exist.

That said, increasing the leaf-level fanout from 16 to 64 should reduce the latency pain by a factor of four. In addition, I would expect that real-time builds of the kernel would set NR_CPUS to some value much smaller than 4096. ;-)

Quoted 15 lines · Paul E. McKenney

Quoted 14 lines · Mike Galbraith

Quoted 12 lines · Paul E. McKenney

From: "Paul E. McKenney" <[email protected]>

Commit #0209f649 (rcu: limit rcu_node leaf-level fanout) set an upper limit of 16 on the leaf-level fanout for the rcu_node tree. This was needed to reduce lock contention that was induced by the synchronization of scheduling-clock interrupts, which was in turn needed to improve energy efficiency for moderate-sized lightly loaded servers.

However, reducing the leaf-level fanout means that there are more leaf-level rcu_node structures in the tree, which in turn means that RCU's grace-period initialization incurs more cache misses. This is not a problem on moderate-sized servers with only a few tens of CPUs,

Quoted 5 lines

Quoted 2 lines

With a distro config (4096 CPUs) interrupt latency is bad even on a quad. Traversing empty nodes taking locks and cache misses hurts.

Agreed -- and I will be working on an additional patch that makes RCU avoid initializing its data structures for CPUs that don't exist.

That's still on my todo list too, your initial patch (and my butchery thereof to skip taking lock) showed this helps a heap.

Quoted 4 lines

That said, increasing the leaf-level fanout from 16 to 64 should reduce the latency pain by a factor of four. In addition, I would expect that real-time builds of the kernel would set NR_CPUS to some value much smaller than 4096. ;-)

Yup, else you would have heard whimpering months ago ;-)

Quoted 22 lines · Mike Galbraith

Quoted 20 lines · Paul E. McKenney

Quoted 9 lines · Mike Galbraith

Quoted 8 lines · Paul E. McKenney

From: "Paul E. McKenney" <[email protected]>

Commit #0209f649 (rcu: limit rcu_node leaf-level fanout) set an upper limit of 16 on the leaf-level fanout for the rcu_node tree. This was needed to reduce lock contention that was induced by the synchronization of scheduling-clock interrupts, which was in turn needed to improve energy efficiency for moderate-sized lightly loaded servers.

Quoted 7 lines

Quoted 4 lines

However, reducing the leaf-level fanout means that there are more leaf-level rcu_node structures in the tree, which in turn means that RCU's grace-period initialization incurs more cache misses. This is not a problem on moderate-sized servers with only a few tens of CPUs,

With a distro config (4096 CPUs) interrupt latency is bad even on a quad. Traversing empty nodes taking locks and cache misses hurts.

Agreed -- and I will be working on an additional patch that makes RCU avoid initializing its data structures for CPUs that don't exist.

Quoted 2 lines

That's still on my todo list too, your initial patch (and my butchery thereof to skip taking lock) showed this helps a heap.

Indeed, I am a bit worried about the safety of skipping the lock in that case. Either way, if you would rather keep working on this, I have no problem pushing it further down my todo list.

From: "Paul E. McKenney" <[email protected]>

The rcu_blocking_is_gp() function tests to see if there is only one online CPU, and if so, synchronize_sched() and friends become no-ops. However, for larger systems, num_online_cpus() scans a large vector,

and might be preempted while doing so. While preempted, any number of CPUs might come online and go offline, potentially resulting in num_online_cpus() returning 1 when there never had only been one CPU online. This would result in a too-short RCU grace period, which could in turn result in total failure.

This commit therefore protects the num_online_cpus() with preempt_disable().

Signed-off-by Paul E. McKenney <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

---
 include/linux/rcutree.h |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

include/linux/rcutree.h +0 -0

index e8ee5dd..997179f 100644

Quoted 6 lines

From: "Paul E. McKenney" <[email protected]>

The rcu_blocking_is_gp() function tests to see if there is only one online CPU, and if so, synchronize_sched() and friends become no-ops. However, for larger systems, num_online_cpus() scans a large vector,

Venki had posted a patch to optimize that by using a variable, so that we don't calculate the value each and every time.
http://thread.gmane.org/gmane.linux.kernel/1240569/focus=1246659

However, unfortunately there was some confusion around that patch and even though it made it to akpm's tree and stayed there briefly, it didn't go upstream. Venki had attempted to resolve the confusion here:
http://thread.gmane.org/gmane.linux.kernel/1240569/focus=1260702

Quoted 6 lines

and might be preempted while doing so. While preempted, any number of CPUs might come online and go offline, potentially resulting in num_online_cpus() returning 1 when there never had only been one CPU online. This would result in a too-short RCU grace period, which could in turn result in total failure.

So, if I understand correctly, the "too-short RCU grace period being troublesome" case occurs when we move from UP to SMP (ie., RCU thought that the system is in UP judging by the value returned by num_online_cpus(), but underneath, some CPUs got onlined in the meantime and hence the system now became SMP).

Quoted 8 lines

This commit therefore protects the num_online_cpus() with preempt_disable().

Signed-off-by Paul E. McKenney <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

---
 include/linux/rcutree.h |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

include/linux/rcutree.h extern void rcu_sched_force_quiescent_state(void); · line 105

Quoted 10 lines

include/linux/rcutree.h +2 -0

index e8ee5dd..997179f 100644
@@ -101,8 +101,13 @@ extern void rcu_sched_force_quiescent_state(void);
 /* A context switch is a grace period for RCU-sched and RCU-bh. */
 static inline int rcu_blocking_is_gp(void)
 {
+	int ret;
+

Wouldn't it be more appropriate to return a bool? (and of course change the function prototype as well..)

include/linux/rcutree.h extern void rcu_sched_force_quiescent_state(void); · line 107

Quoted 3 lines

include/linux/rcutree.h +1 -1

 	might_sleep();  /* Check for RCU read-side critical section. */
-	return num_online_cpus() == 1;
+	preempt_disable();  /* Prevent CPU hotplug from confusing us. */

Preempt disable only prevents CPU offline from occurring concurrently. So even if we use preempt_disable/enable(), new CPUs can always come online! And going by the above concern about RCU grace periods being too-short, I guess we should also protect against CPUs from coming online concurrently right?

[By the way, we don't really need to protect against CPU offlining, because an SMP -> UP switch is harmless; we'll just miss a fast-path in the worst case, but the code correctness is still guaranteed.]

include/linux/rcutree.h extern void rcu_sched_force_quiescent_state(void); · line 111

Quoted 4 lines

include/linux/rcutree.h +3 -0

+	ret = num_online_cpus() == 1;
+	preempt_enable();
+	return ret;
 }

Maybe I am missing something, but how is this code safe? Isn't it still racy? Even if we use:

{
	might_sleep();
	get_online_cpus(); /* Prevent both CPU offline and CPU online */
	ret = num_online_cpus() == 1;
	put_online_cpus();
			<===============
	return ret;
}

What prevents a CPU Hotplug from happening right before we return from this function, thereby rendering our num_online_cpus() calculation out-dated and hence useless?

Quoted 8 lines · Srivatsa S. Bhat

Quoted 6 lines

From: "Paul E. McKenney" <[email protected]>

The rcu_blocking_is_gp() function tests to see if there is only one online CPU, and if so, synchronize_sched() and friends become no-ops. However, for larger systems, num_online_cpus() scans a large vector,

Quoted 8 lines

Venki had posted a patch to optimize that by using a variable, so that we don't calculate the value each and every time.
http://thread.gmane.org/gmane.linux.kernel/1240569/focus=1246659

However, unfortunately there was some confusion around that patch and even though it made it to akpm's tree and stayed there briefly, it didn't go upstream. Venki had attempted to resolve the confusion here:
http://thread.gmane.org/gmane.linux.kernel/1240569/focus=1260702

Having a single variable tracking the online state would be good, but as you say it isn't there yet.

Quoted 7 lines

Quoted 5 lines

and might be preempted while doing so. While preempted, any number of CPUs might come online and go offline, potentially resulting in num_online_cpus() returning 1 when there never had only been one CPU online. This would result in a too-short RCU grace period, which could in turn result in total failure.

The problematic case is instead the one where we were SMP throughout, but rcu_blocking_is_gp() mistakenly decides that we were UP. For example, consider the following sequence of events, based on the commit log's sentence "While preempted, any number of CPUs might come online and go offline, potentially resulting in num_online_cpus() returning 1 when there never had only been one CPU online":

o CPUs 100 and 150 are initially online, with a long-running RCU
read-side critical section on CPU 100 and rcu_blocking_is_gp() initially running on CPU 150.

o The rcu_blocking_is_gp() function checks the bits for CPUs
0-63, and counts zero online CPUs.

o CPU 1 comes online.

o The rcu_blocking_is_gp() function checks the bits for CPUs
64-127, and counts one online CPUs, for a total thus far of one CPU online..

o CPU 150 goes offline. Ah, but it cannot do this, because
this is non-preemptible RCU, which means that the RCU read-side critical section has disabled preemption on CPU 100, which prevents CPU 150 from going offline, which prevents this scenario from occurring.

So, yes, rcu_blocking_is_gp() can be fooled into incorrectly stating that the system has only one CPU (or even that it has only zero CPUs), but not while there is actually a non-preemptible RCU read-side critical section running. Yow!

I clearly had not thought this change through far enough, thank you for calling it to my attention!

So I could replace this patch with a patch that adds a comment explaining why this works.

Though this patch might be simpler and
easier to understand. ;-)

But not so good for real-time response
on large systems, I suppose.

And rcu_blocking_is_gp() is called only from synchronize_sched() and synchronize_rcu_bh(), so it is safe for all current callers. But it is defined publicly, so I should move it to somewhere like kernel/rcutree.c to keep new users from being fatally confused and disappointed.

I can also change the comparison from "num_online_cpus() == 1" to "num_online_cpus() <= 1". That should at least serve as a caution to people who might attempt to use it where it shouldn't be used. ;-)

On Tue, Apr 24, 2012 at 09:05:20PM +0530, Srivatsa S. Bhat wrote:

Thank you for looking this over!

Quoted 13 lines · Paul E. McKenney

Quoted 6 lines

From: "Paul E. McKenney" <[email protected]>

The rcu_blocking_is_gp() function tests to see if there is only one online CPU, and if so, synchronize_sched() and friends become no-ops. However, for larger systems, num_online_cpus() scans a large vector,

Venki had posted a patch to optimize that by using a variable, so that we don't calculate the value each and every time.
http://thread.gmane.org/gmane.linux.kernel/1240569/focus=1246659

Quoted 4 lines

However, unfortunately there was some confusion around that patch and even though it made it to akpm's tree and stayed there briefly, it didn't go upstream. Venki had attempted to resolve the confusion here:
http://thread.gmane.org/gmane.linux.kernel/1240569/focus=1260702

Having a single variable tracking the online state would be good, but as you say it isn't there yet.