Skip to contentSkip to conversation

cpuset - rework cpuset_zone_allowed api

PATCH In review Paul Jackson 3 participants 4 messages Dec 8, 2006 – Dec 9, 2006

Subsystems (7)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 cpuset - rework cpuset_zone_allowed api

From: Paul Jackson <[email protected]>

Elaborate the API for calling cpuset_zone_allowed(), so that users have to explicitly choose between the two variants:

cpuset_zone_allowed_hardwall()
cpuset_zone_allowed_softwall()

Until now, whether or not you got the hardwall flavor depended solely on whether or not you or'd in the __GFP_HARDWALL gfp flag to the gfp_mask argument.

If you didn't specify __GFP_HARDWALL, you implicitly got the softwall version.

Unfortunately, this meant that users would end up with the softwall version without thinking about it. Since only the softwall version might sleep, this led to bugs with possible sleeping in interrupt context on more than one occassion.

The hardwall version requires that the current tasks mems_allowed allows the node of the specified zone (or that you're in interrupt or that __GFP_THISNODE is set or that you're on a one cpuset system.)

The softwall version, depending on the gfp_mask, might allow a node if it was allowed in the nearest enclusing cpuset marked mem_exclusive (which requires taking the cpuset lock 'callback_mutex' to evaluate.)

This patch removes the cpuset_zone_allowed() call, and forces the caller to explicitly choose between the hardwall and the softwall case.

If the caller wants the gfp_mask to determine this choice, they should (1) be sure they can sleep or that __GFP_HARDWALL is set, and (2) invoke the cpuset_zone_allowed_softwall() routine.

This adds another 100 or 200 bytes to the kernel text space, due to the few lines of nearly duplicate code at the top of both cpuset_zone_allowed_* routines. It should save a few instructions executed for the calls that turned into calls of cpuset_zone_allowed_hardwall, thanks to not having to set (before the call) then check (within the call) the __GFP_HARDWALL flag.

For the most critical call, from get_page_from_freelist(), the same instructions are executed as before -- the old cpuset_zone_allowed() routine it used to call is the same code as the cpuset_zone_allowed_softwall() routine that it calls now.

Not a perfect win, but seems worth it, to reduce this chance of hitting a sleeping with irq off complaint again.

---
 include/linux/cpuset.h |   22 ++++++++++---
 kernel/cpuset.c        |   82 +++++++++++++++++++++++++++++++++++++++----------
 mm/hugetlb.c           |    2 -
 mm/oom_kill.c          |    2 -
 mm/page_alloc.c        |    2 -
 mm/slab.c              |    3 -
 mm/vmscan.c            |    8 ++--
 7 files changed, 92 insertions(+), 29 deletions(-)
include/linux/cpuset.h +18 -4
@@ -30,10 +30,19 @@ void cpuset_update_task_memory_state(voi
 		nodes_subset((nodes), current->mems_allowed)
 int cpuset_zonelist_valid_mems_allowed(struct zonelist *zl);
 
-extern int __cpuset_zone_allowed(struct zone *z, gfp_t gfp_mask);
-static int inline cpuset_zone_allowed(struct zone *z, gfp_t gfp_mask)
+extern int __cpuset_zone_allowed_softwall(struct zone *z, gfp_t gfp_mask);
+extern int __cpuset_zone_allowed_hardwall(struct zone *z, gfp_t gfp_mask);
+
+static int inline cpuset_zone_allowed_softwall(struct zone *z, gfp_t gfp_mask)
+{
+	return number_of_cpusets <= 1 ||
+		__cpuset_zone_allowed_softwall(z, gfp_mask);
+}
+
+static int inline cpuset_zone_allowed_hardwall(struct zone *z, gfp_t gfp_mask)
 {
-	return number_of_cpusets <= 1 || __cpuset_zone_allowed(z, gfp_mask);
+	return number_of_cpusets <= 1 ||
+		__cpuset_zone_allowed_hardwall(z, gfp_mask);
 }
 
 extern int cpuset_excl_nodes_overlap(const struct task_struct *p);
@@ -94,7 +103,12 @@ static inline int cpuset_zonelist_valid_
 	return 1;
 }
 
-static inline int cpuset_zone_allowed(struct zone *z, gfp_t gfp_mask)
+static inline int cpuset_zone_allowed_softwall(struct zone *z, gfp_t gfp_mask)
+{
+	return 1;
+}
+
+static inline int cpuset_zone_allowed_hardwall(struct zone *z, gfp_t gfp_mask)
 {
 	return 1;
 }
kernel/cpuset.c +66 -16

116 lines not on this page · raw message

mm/hugetlb.c +1 -1
@@ -73,7 +73,7 @@ static struct page *dequeue_huge_page(st
 
 	for (z = zonelist->zones; *z; z++) {
 		nid = zone_to_nid(*z);
-		if (cpuset_zone_allowed(*z, GFP_HIGHUSER) &&
+		if (cpuset_zone_allowed_softwall(*z, GFP_HIGHUSER) &&
 		    !list_empty(&hugepage_freelists[nid]))
 			break;
 	}
mm/oom_kill.c +1 -1
@@ -177,7 +177,7 @@ static inline int constrained_alloc(stru
 	nodemask_t nodes = node_online_map;
 
 	for (z = zonelist->zones; *z; z++)
-		if (cpuset_zone_allowed(*z, gfp_mask))
+		if (cpuset_zone_allowed_softwall(*z, gfp_mask))
 			node_clear(zone_to_nid(*z), nodes);
 		else
 			return CONSTRAINT_CPUSET;
mm/page_alloc.c +1 -1
@@ -1179,7 +1179,7 @@ zonelist_scan:
 			zone->zone_pgdat != zonelist->zones[0]->zone_pgdat))
 				break;
 		if ((alloc_flags & ALLOC_CPUSET) &&
-			!cpuset_zone_allowed(zone, gfp_mask))
+			!cpuset_zone_allowed_softwall(zone, gfp_mask))
 				goto try_next_zone;
 
 		if (!(alloc_flags & ALLOC_NO_WATERMARKS)) {
mm/slab.c +1 -2
@@ -3261,8 +3261,7 @@ void *fallback_alloc(struct kmem_cache *
 		int nid = zone_to_nid(*z);
 
 		if (zone_idx(*z) <= ZONE_NORMAL &&
-				cpuset_zone_allowed(*z,
-					flags | __GFP_HARDWALL) &&
+				cpuset_zone_allowed_hardwall(*z, flags) &&
 				cache->nodelists[nid])
 			obj = ____cache_alloc_node(cache,
 					flags | __GFP_THISNODE, nid);
mm/vmscan.c +4 -4
@@ -986,7 +986,7 @@ static unsigned long shrink_zones(int pr
 		if (!populated_zone(zone))
 			continue;
 
-		if (!cpuset_zone_allowed(zone, __GFP_HARDWALL))
+		if (!cpuset_zone_allowed_hardwall(zone, GFP_KERNEL))
 			continue;
 
 		note_zone_scanning_priority(zone, priority);
@@ -1051,7 +1051,7 @@ unsigned long try_to_free_pages(struct z
 	for (i = 0; zones[i] != NULL; i++) {
 		struct zone *zone = zones[i];
 
-		if (!cpuset_zone_allowed(zone, __GFP_HARDWALL))
+		if (!cpuset_zone_allowed_hardwall(zone, GFP_KERNEL))
 			continue;
 
 		lru_pages += zone->nr_active + zone->nr_inactive;
@@ -1106,7 +1106,7 @@ out:
 	for (i = 0; zones[i] != 0; i++) {
 		struct zone *zone = zones[i];
 
-		if (!cpuset_zone_allowed(zone, __GFP_HARDWALL))
+		if (!cpuset_zone_allowed_hardwall(zone, GFP_KERNEL))
 			continue;
 
 		zone->prev_priority = priority;
@@ -1372,7 +1372,7 @@ void wakeup_kswapd(struct zone *zone, in
 		return;
 	if (pgdat->kswapd_max_order < order)
 		pgdat->kswapd_max_order = order;
-	if (!cpuset_zone_allowed(zone, __GFP_HARDWALL))
+	if (!cpuset_zone_allowed_hardwall(zone, GFP_KERNEL))
 		return;
 	if (!waitqueue_active(&pgdat->kswapd_wait))
 		return;
Signature · 3 lines
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <[email protected]> 1.650.933.1373
#1 lore raw

kernel/cpuset.c int __cpuset_zone_allowed(struct zone *z · line 2352

Quoted 9 lines · Paul Jackson
kernel/cpuset.c +1 -1
-int __cpuset_zone_allowed(struct zone *z, gfp_t gfp_mask)
+int __cpuset_zone_allowed_softwall(struct zone *z, gfp_t gfp_mask)
 {
        int node;                       /* node that zone z is on */
        const struct cpuset *cs;        /* current cpuset ancestors */
@@ -2335,6 +2351,40 @@ int __cpuset_zone_allowed(struct zone *z
        return allowed;
 }

While you're changing this, is there a good reason not to check is_mem_exclusive() *before* taking callback_mutex and calling nearest_exclusive_ancestor()?

something like:

rcu_read_lock();
exc = is_mem_exclusive(rcu_dereference(current->cs)); rcu_read_unlock();
if (exc)
return 1;

... take callback_mutex, etc ...

Paul

#2 lore raw
Quoted 11 lines · Paul M

While you're changing this, is there a good reason not to check is_mem_exclusive() *before* taking callback_mutex and calling nearest_exclusive_ancestor()?

something like:

rcu_read_lock();
exc = is_mem_exclusive(rcu_dereference(current->cs)); rcu_read_unlock();
if (exc)
return 1;

Hmmm ... Interesting suggestion, but I'm not sure this is a good idea.

For one thing, shouldn't that be "return 0", not "return 1". If the current tasks cpuset is mem_exclusive, and if we've already determined that it doesn't allow the node in question, and if we've also just determined that it is itself our nearest mem_exclusive ancestor, then can't we conclude that the node in question is -not- allowed in our nearest mem_exclusive ancestor?

And for another thing, it extends the RCU locking use just a teeny bit. Until now, we just RCU to let us check whether the cpuset mems_generation was changed or not, without risking an invalid memory reference. The above proposal makes stronger demands, as follows.

Say for instance, another task changed our tasks cpuset just as we were running this cpuset_zone_allowed() check, from a cpuset whose -parent- would have allowed the node in question and which parent was the nearest enclosing mem_exclusive cpuset, to a different cpuset which would itself have allowed the node in question and which was marked mem_exclusive. Either the old or the new cpuset would have allowed the node, but if we flip at just the wrong instant, after we realize the node isn't allowed in the current cpuset, before we check to see if that cpuset is mem_exclusive, we would conclude that the node is not allowed.

I can't imagine even a micro benchmark test case that would detect the above race failing, not to mention a real world noticable impact. But it is a lost race. Better not to code races, than to learn two years later why they might matter.

For a third thing, this adds a little more kernel text, in order to optimize the case that the current cpuset is mem_exclusive, at the expense of a slightly longer code path for the case that it is some ancestor that is the nearest enclosing mem_exclusive cpuset.

I guess it is workload and cpuset config dependent whether or not such a tradeoff is an improvement, or a step backward. Lacking a persuasive argument that the case for which this optimizes is enough more frequent than the other case to matter, I'm inclined to pick the solution with the least code -- what's there now.

What am I missing?

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#3 lore raw
In reply to Paul Jackson
Quoted 11 lines · On Fri, 08 Dec 2006 03:21:52 -0800 Paul Jackson <[email protected]> wrote:

From: Paul Jackson <[email protected]>

Elaborate the API for calling cpuset_zone_allowed(), so that users have to explicitly choose between the two variants:

cpuset_zone_allowed_hardwall()
cpuset_zone_allowed_softwall()

Until now, whether or not you got the hardwall flavor depended solely on whether or not you or'd in the __GFP_HARDWALL gfp flag to the gfp_mask argument.

yeah, nice change that.