Skip to contentSkip to conversation

earlier allocation of order 0 pages from pcp in __alloc_pages

PATCH In review Seth, Rohit 5 participants 11 messages Sep 29, 2005 – Sep 30, 2005

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 earlier allocation of order 0 pages from pcp in __alloc_pages

Try to service a order 0 page request from pcp list. This will allow us to not check and possibly start the reclaim activity when there are free pages present on the pcp. This early allocation does not try to replenish an empty pcp.

mm/page_alloc.c +39 -0
@@ -716,6 +716,39 @@
 		clear_highpage(page + i);
 }
 
+/* This routine allocates a order 0 page from cpu's pcp list when one is present.
+ * It does not try to remove the pages from zone_free_list as the zone low
+ * water mark has not yet been checked.
+ */
+
+static struct page *
+remove_from_pcp(struct zone *zone, unsigned int __nocast gfp_flags)
+{
+	unsigned long flags;
+	struct per_cpu_pages *pcp;
+	struct page *page = NULL;
+	int cold = !!(gfp_flags & __GFP_COLD);
+
+	pcp = &zone_pcp(zone, get_cpu())->pcp[cold];
+	local_irq_save(flags);
+	if (pcp->count > pcp->low) {
+		page = list_entry(pcp->list.next, struct page, lru);
+		list_del(&page->lru);
+		pcp->count--;
+	}
+	local_irq_restore(flags);
+	put_cpu();
+
+	if (page != NULL) {
+		mod_page_state_zone(zone, pgalloc, 1 );
+		prep_new_page(page, 0);
+
+		if (gfp_flags & __GFP_ZERO)
+			prep_zero_page(page, 0, gfp_flags);
+	}
+	return page;
+}
+
 /*
  * Really, prep_compound_page() should be called from __rmqueue_bulk().  But
  * we cheat by calling it from here, in the order > 0 path.  Saves a branch
@@ -905,6 +938,12 @@
 		if (!cpuset_zone_allowed(z, __GFP_HARDWALL))
 			continue;
 
+		if (order == 0) {
+			page = remove_from_pcp(z, gfp_mask);
+			if (page)
+				goto got_pg;
+		}
+
 		/*
 		 * If the zone is to attempt early page reclaim then this loop
 		 * will try to reclaim pages and check the watermark a second

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#1 lore raw
Quoted 4 lines · Rohit

[PATCH]: Try to service a order 0 page request in __alloc_pages from the pcp list before checking the aone_watermarks.

Try to service a order 0 page request from pcp list. This will allow us to not check and possibly start the reclaim activity when there are free pages present on the pcp. This early allocation does not try to replenish an empty pcp.

(Please avoid the 240-column emails!)

Why is this a desirable change to make?

author
#2 lore raw
Quoted 8 lines · Andrew Morton
Quoted 4 lines · Rohit

[PATCH]: Try to service a order 0 page request in __alloc_pages from the pcp list before checking the aone_watermarks.

Try to service a order 0 page request from pcp list. This will allow us to not check and possibly start the reclaim activity when there are free pages present on the pcp. This early allocation does not try to replenish an empty pcp.

(Please avoid the 240-column emails!)

This is bad. Sorry.

Quoted 1 line

Why is this a desirable change to make?

This change avoids any checks for watermarks and starting potential reclaims for the cases where we already know we don't need to.

-rohit

#3 lore raw
In reply to Seth, Rohit
Quoted 3 lines

Try to service a order 0 page request from pcp list. This will allow us to not check and possibly start the reclaim activity when there are free pages present on the pcp. This early allocation does not try to replenish an empty pcp.

It seems a bit odd to copy more code to do this, which I think we already have in buffered_rmqueue? Can we clean up the flow a bit here ... it is already looking messy in __alloc_pages with various gotos and crud there. I'm not saying what you're trying to do is bad, but the flow in there is getting more and move convoluted, and we perhaps need to straighten it.

It looks like we're now dropping into direct reclaim as the first thing in __alloc_pages before even trying to kick off kswapd. When the hell did that start? Or is that only meant to trigger if we're already below the low watermark level?

What do we want to do at a higher level?

if (order 0) and (have stuff in the local lists)
take from local lists
else if (we're under a little pressure)
do kswapd reclaim
else if (we're under a lot of pressure)
do direct reclaim?

That whole code area seems to have been turned into spagetti, without any clear comments.

M.

Quoted 57 lines
mm/page_alloc.c +39 -1
@@ -716,6 +716,39 @@
 		clear_highpage(page + i);
 }
 
+/* This routine allocates a order 0 page from cpu's pcp list when one is present.
+ * It does not try to remove the pages from zone_free_list as the zone low
+ * water mark has not yet been checked.
+ */
+
+static struct page *
+remove_from_pcp(struct zone *zone, unsigned int __nocast gfp_flags)
+{
+	unsigned long flags;
+	struct per_cpu_pages *pcp;
+	struct page *page = NULL;
+	int cold = !!(gfp_flags & __GFP_COLD);
+
+	pcp = &zone_pcp(zone, get_cpu())->pcp[cold];
+	local_irq_save(flags);
+	if (pcp->count > pcp->low) {
+		page = list_entry(pcp->list.next, struct page, lru);
+		list_del(&page->lru);
+		pcp->count--;
+	}
+	local_irq_restore(flags);
+	put_cpu();
+
+	if (page != NULL) {
+		mod_page_state_zone(zone, pgalloc, 1 );
+		prep_new_page(page, 0);
+
+		if (gfp_flags & __GFP_ZERO)
+			prep_zero_page(page, 0, gfp_flags);
+	}
+	return page;
+}
+
 /*
  * Really, prep_compound_page() should be called from __rmqueue_bulk().  But
  * we cheat by calling it from here, in the order > 0 path.  Saves a branch
@@ -905,6 +938,12 @@
 		if (!cpuset_zone_allowed(z, __GFP_HARDWALL))
 			continue;
 
+		if (order == 0) {
+			page = remove_from_pcp(z, gfp_mask);
+			if (page)
+				goto got_pg;
+		}
+
 		/*
 		 * If the zone is to attempt early page reclaim then this loop
 		 * will try to reclaim pages and check the watermark a second
 
--
Quoted 6 lines

To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#4 lore raw
Quoted 5 lines · Martin J. Bligh

It looks like we're now dropping into direct reclaim as the first thing in __alloc_pages before even trying to kick off kswapd. When the hell did that start? Or is that only meant to trigger if we're already below the low watermark level?

That's all the numa goop which Martin Hicks added. It's all disabled if z->reclaim_pages is zero (it is). However we could be testing that flag a bit earlier, I think.

And yeah, some de-spaghettification would be nice. Certainly before adding more logic.

Martin, should we take out the early zone reclaim logic? It's all unreachable at present anyway.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#5 lore raw
Quoted 11 lines · Andrew Morton
Quoted 5 lines · Martin J. Bligh

It looks like we're now dropping into direct reclaim as the first thing in __alloc_pages before even trying to kick off kswapd. When the hell did that start? Or is that only meant to trigger if we're already below the low watermark level?

That's all the numa goop which Martin Hicks added. It's all disabled if z->reclaim_pages is zero (it is). However we could be testing that flag a bit earlier, I think.

Quoted 6 lines

And yeah, some de-spaghettification would be nice. Certainly before adding more logic.

Martin, should we take out the early zone reclaim logic? It's all unreachable at present anyway.

...yeah just like sys_set_zone_reclaim. was it intended to be added as a system call?

-rohit

#6 lore raw
In reply to Andrew Morton
Quoted 11 lines · Andrew Morton
Quoted 5 lines · Martin J. Bligh

It looks like we're now dropping into direct reclaim as the first thing in __alloc_pages before even trying to kick off kswapd. When the hell did that start? Or is that only meant to trigger if we're already below the low watermark level?

That's all the numa goop which Martin Hicks added. It's all disabled if z->reclaim_pages is zero (it is). However we could be testing that flag a bit earlier, I think.

Quoted 6 lines

And yeah, some de-spaghettification would be nice. Certainly before adding more logic.

Martin, should we take out the early zone reclaim logic? It's all unreachable at present anyway.

Yes, Please do.

mh

Signature · 7 lines
Martin Hicks   ||   Silicon Graphics Inc.   ||   [email protected]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#7 lore raw
In reply to Martin J. Bligh
Quoted 11 lines · Martin J. Bligh
Quoted 3 lines

Try to service a order 0 page request from pcp list. This will allow us to not check and possibly start the reclaim activity when there are free pages present on the pcp. This early allocation does not try to replenish an empty pcp.

It seems a bit odd to copy more code to do this, which I think we already have in buffered_rmqueue? Can we clean up the flow a bit here ... it is already looking messy in __alloc_pages with various gotos and crud there. I'm not saying what you're trying to do is bad, but the flow in there is getting more and move convoluted, and we perhaps need to straighten it.

I will update/streamline __alloc_pages code and send the patch.

Quoted 5 lines

It looks like we're now dropping into direct reclaim as the first thing in __alloc_pages before even trying to kick off kswapd. When the hell did that start? Or is that only meant to trigger if we're already below the low watermark level?

As Andrew said in the other mail that do_reclaim is never true so the first reclaim never happens. That also means that we don't look at pcp for the scenarios when zone has run below the low water mark before waking kswapd.

Quoted 11 lines

What do we want to do at a higher level?

if (order 0) and (have stuff in the local lists)
take from local lists
else if (we're under a little pressure)
do kswapd reclaim
else if (we're under a lot of pressure)
do direct reclaim?

That whole code area seems to have been turned into spagetti, without any clear comments.

Agreed.

#8 lore raw
Quoted 22 lines

I will update/streamline __alloc_pages code and send the patch.

Quoted 5 lines

It looks like we're now dropping into direct reclaim as the first thing in __alloc_pages before even trying to kick off kswapd. When the hell did that start? Or is that only meant to trigger if we're already below the low watermark level?

As Andrew said in the other mail that do_reclaim is never true so the first reclaim never happens. That also means that we don't look at pcp for the scenarios when zone has run below the low water mark before waking kswapd.

Quoted 9 lines

What do we want to do at a higher level?

if (order 0) and (have stuff in the local lists)
take from local lists
else if (we're under a little pressure)
do kswapd reclaim
else if (we're under a lot of pressure)
do direct reclaim?

Quoted 4 lines
Quoted 2 lines

That whole code area seems to have been turned into spagetti, without any clear comments.

Agreed.

Thanks. I didn't mean you'd done so, BTW ... just many iterations of people stacking more and more stuff on top without it ever getting cleaned up, and it's got a bit silly ;-)

M.

#9 lore raw
In reply to Seth, Rohit
Quoted 23 lines · Rohit
mm/page_alloc.c +23 -0
+/* This routine allocates a order 0 page from cpu's pcp list when one is present.
+ * It does not try to remove the pages from zone_free_list as the zone low
+ * water mark has not yet been checked.
+ */
+
+static struct page *
+remove_from_pcp(struct zone *zone, unsigned int __nocast gfp_flags)
+{
+	unsigned long flags;
+	struct per_cpu_pages *pcp;
+	struct page *page = NULL;
+	int cold = !!(gfp_flags & __GFP_COLD);
+
+	pcp = &zone_pcp(zone, get_cpu())->pcp[cold];
+	local_irq_save(flags);
+	if (pcp->count > pcp->low) {
+		page = list_entry(pcp->list.next, struct page, lru);
+		list_del(&page->lru);
+		pcp->count--;
+	}
+	local_irq_restore(flags);
+	put_cpu();
+

mm/page_alloc.c line 751

Quoted 10 lines
mm/page_alloc.c +10 -0
+	if (page != NULL) {
+		mod_page_state_zone(zone, pgalloc, 1 );
+		prep_new_page(page, 0);
+
+		if (gfp_flags & __GFP_ZERO)
+			prep_zero_page(page, 0, gfp_flags);
+	}
+	return page;
+}
+

That looks to share a decent amount of logic with the pcp code in buffered_rmqueue. Any chance it could be consolidated instead of copy/pasting?

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#10 lore raw
Quoted 5 lines · Dave Hansen

That looks to share a decent amount of logic with the pcp code in buffered_rmqueue. Any chance it could be consolidated instead of copy/pasting?

It indeed does share most of the code with buffered_rmqueue. And it is definitely possible to streamline this control flow. But that would require more changes in the existing code (didn't want to make that as part of this patch to start with).

-rohit