Skip to contentSkip to conversation

Performance regression in scsi sequential throughput (iozone) due to "e084b - page-allocator: preserve PFN ordering when __GFP_COLD is set"

PATCH Christian Ehrhardt 3 participants 41 messages Dec 7, 2009 – Mar 3, 2010

Subsystems (8)
lore
Share & tools
lore
#0 lore raw

Hi,
I tracked a huge performance regression for a while and got it bisected down to commit "e084b2d95e48b31aa45f9c49ffc6cdae8bdb21d4 - page-allocator: preserve PFN ordering when __GFP_COLD is set".

The scenario I'm running is a low memory system (256M total), that does sequential I/O with parallel iozone processes. One process per disk, each process reading a 2Gb file. The disks I use are fcp scsi disks attached to a s390 host. File system is ext2.

The regression appears as up to 76% loss in throughput at 16 processes (processes are scaled from 1 to 64, performance is bad everywhere - 16 is just the peak - avg loss is about 40% throughput). I already know that giving the system just a bit (~64m+) more memory solves the issue almost completely, probably because there is almost no "memory pressure" left in that cases. I also know that using direct-I/O instead of I/O through page cache doesn't have the problem at all.
Comparing sysstat data taken while running with the kernels just with & without the bisected patch shows nothing obvious except that I/O seems to take much longer (lower interrupt ratio etc).

The patch alone looks very reasonable, so I'd prefer understanding and fixing the real issue instead of getting it eventually reverted due to this regression being larger than the one it was intended to fix. In the patch it is clear that hot pages (cold==0) freed in rmqueue_bulk should behave like before. So maybe the question is "are our pages cold while they shouldn't be"? Well I don't really have a clue yet to explain how patch e084b exactly causes that big regression, ideas welcome :-)

Kind regards,
Christian

P.S: The original patch is concise enough to attach it here to act as reference without bloating the mail too much:
commit e084b2d95e48b31aa45f9c49ffc6cdae8bdb21d4

Author: Mel Gorman <[email protected]>
Date:   Wed Jul 29 15:02:04 2009 -0700

page-allocator: preserve PFN ordering when __GFP_COLD is set

Fix a post-2.6.24 performace regression caused by 3dfa5721f12c3d5a441448086bee156887daa961 ("page-allocator: preserve PFN ordering when __GFP_COLD is set").

Narayanan reports "The regression is around 15%. There is no disk
controller
as our setup is based on Samsung OneNAND used as a memory mapped
device on a
OMAP2430 based board."

The page allocator tries to preserve contiguous PFN ordering when
returning
pages such that repeated callers to the allocator have a strong
chance of
getting physically contiguous pages, particularly when external
fragmentation
is low. However, of the bulk of the allocations have __GFP_COLD set
as they
are due to aio_read() for example, then the PFNs are in reverse PFN
order.
This can cause performance degration when used with IO controllers
that could
have merged the requests.

This patch attempts to preserve the contiguous ordering of PFNs for
users of
__GFP_COLD.

mm/page_alloc.c +0 -0
index caa9268..ae28c22 100644
@@ -882,7 +882,7 @@ retry_reserve:
 */

static int rmqueue_bulk(struct zone *zone, unsigned int order,
unsigned long count, struct list_head *list,
- int migratetype)
+ int migratetype, int cold)

{
    int i;

@@ -901,7 +901,10 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order,
* merge IO requests if the physical pages are ordered
* properly.
*/
- list_add(&page->lru, list);
+ if (likely(cold == 0))
+ list_add(&page->lru, list);
+ else

+            list_add_tail(&page->lru, list);
        set_page_private(page, migratetype);
        list = &page->lru;
    }
@@ -1119,7 +1122,8 @@ again:
        local_irq_save(flags);
        if (!pcp->count) {
            pcp->count = rmqueue_bulk(zone, 0,
-                    pcp->batch, &pcp->list, migratetype);
+                    pcp->batch, &pcp->list,
+                    migratetype, cold);
            if (unlikely(!pcp->count))
                goto failed;
        }
@@ -1138,7 +1142,8 @@ again:
        /* Allocate more to the pcp list if necessary */
        if (unlikely(&page->lru == &pcp->list)) {
            pcp->count += rmqueue_bulk(zone, 0,
-                    pcp->batch, &pcp->list, migratetype);
+                    pcp->batch, &pcp->list,
+                    migratetype, cold);
            page = list_entry(pcp->list.next, struct page, lru);
        }
Signature · 2 lines
Grüsse / regards, Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization
#1 lore raw
Quoted 5 lines · Christian Ehrhardt

Hi,
I tracked a huge performance regression for a while and got it bisected down to commit "e084b2d95e48b31aa45f9c49ffc6cdae8bdb21d4 -
page-allocator: preserve PFN ordering when __GFP_COLD is set".

Darn. That is related to IO controllers being able to automatically merge requests. The problem it was fixing was that pages were arriving in reverse PFN order, the controller was unable to merge and performance was impaired. Any controller that can merge should be faster as a result of the patch.

Quoted 5 lines

The scenario I'm running is a low memory system (256M total), that does sequential I/O with parallel iozone processes.
One process per disk, each process reading a 2Gb file. The disks I use are fcp scsi disks attached to a s390 host. File system is ext2.

I don't know what controller is in use there but does it opportunistically merge requests if they are on physically contiguous pages? If so, can it be disabled?

Quoted 8 lines

The regression appears as up to 76% loss in throughput at 16 processes (processes are scaled from 1 to 64, performance is bad everywhere - 16 is just the peak - avg loss is about 40% throughput).
I already know that giving the system just a bit (~64m+) more memory solves the issue almost completely, probably because there is almost no "memory pressure" left in that cases.
I also know that using direct-I/O instead of I/O through page cache doesn't have the problem at all.

This makes sense because it's a sequentual read load, so readahead is a factor and that is why __GFP_COLD is used - the data is not for immediate use so doesn't need to be cache hot.

Quoted 4 lines

Comparing sysstat data taken while running with the kernels just with & without the bisected patch shows nothing obvious except that I/O seems to take much longer (lower interrupt ratio etc).

Maybe the controller is spending an age trying to merge requests because it should be able to but takes a long time figuring it out?

Quoted 9 lines

The patch alone looks very reasonable, so I'd prefer understanding and fixing the real issue instead of getting it eventually reverted due to this regression being larger than the one it was intended to fix. In the patch it is clear that hot pages (cold==0) freed in rmqueue_bulk should behave like before. So maybe the question is "are our pages cold while they shouldn't be"?
Well I don't really have a clue yet to explain how patch e084b exactly causes that big regression, ideas welcome :-)

Only theory I have at the moment is that the controller notices it can merge requests and either spends a long time figuring out how to do the merging or performs worse with merged IO requests.

If the problem is in the driver, oprofile might show where the problem lies.

Signature · 3 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
#2 lore raw
Quoted 8 lines · Mel Gorman

On Mon, Dec 07, 2009 at 03:39:49PM +0100, Christian Ehrhardt wrote:

[...]

I don't know what controller is in use there but does it opportunistically merge requests if they are on physically contiguous pages? If so, can it be disabled?

As far as i could clarify it our controllers don't support such a opportunistic merging.

Quoted 9 lines
Quoted 4 lines

The regression appears as up to 76% loss in throughput at 16 processes (processes are scaled from 1 to 64, performance is bad everywhere - 16 is just the peak - avg loss is about 40% throughput). I already know that giving the system just a bit (~64m+) more memory solves the issue almost completely, probably because there is almost no "memory pressure" left in that cases. I also know that using direct-I/O instead of I/O through page cache doesn't have the problem at all.

This makes sense because it's a sequentual read load, so readahead is a factor and that is why __GFP_COLD is used - the data is not for immediate use so doesn't need to be cache hot.

In the meanwhile I was able to verify that this also applies to random reads which are still reads but have zero read ahead requests. I attached more regression data in the post scriptum at the end of the mail.

Quoted 8 lines
Quoted 4 lines

Comparing sysstat data taken while running with the kernels just with & without the bisected patch shows nothing obvious except that I/O seems to take much longer (lower interrupt ratio etc).

Maybe the controller is spending an age trying to merge requests because it should be able to but takes a long time figuring it out?

I thought that too, but now comes the funny part.
I gathered HW statistics from our I/O controllers and latency statistics clearly state that your patch is working as intended - the latency from entering the controller until the interrupt to linux device driver is ~30% lower!. Remember as I stated above that they don't support that opportunistic merging so I will have some fun finding out why it is faster in HW now :-)

Quoted 11 lines
Quoted 5 lines

The patch alone looks very reasonable, so I'd prefer understanding and fixing the real issue instead of getting it eventually reverted due to this regression being larger than the one it was intended to fix. In the patch it is clear that hot pages (cold==0) freed in rmqueue_bulk should behave like before. So maybe the question is "are our pages cold while they shouldn't be"? Well I don't really have a clue yet to explain how patch e084b exactly causes that big regression, ideas welcome :-)

Only theory I have at the moment is that the controller notices it can merge requests and either spends a long time figuring out how to do the merging or performs worse with merged IO requests.

If the problem is in the driver, oprofile might show where the problem lies

With the effective throughput dropping by such a large amount while hardware latency improves by 30% I agree and suggest the issue is in the driver. I'll do some research in breaking down where in our drivers time is lost and reply here for advises and comments in regard to what general memory management could/should/might do.

Kind regards,
Christian

p.s.
FYI a bit more regression data, now I had the patch identified I made a full regression test scaling from 1 to 64 processes. Comparing just without / with the commit e084b
I wondered, but obviously random read is also suffering from that patch.

Sequential Read
Procs Deviation in %
1 -4.9
2 5.2

4             -82.6
8             -65.6
16            -44.2
32            -30.0
64            -37.7

Random Read
Procs Deviation in %

1             -48.3
2             -45.7
4             -50.5
8             -59.0
16            -61.8
32            -48.3
64            -21.0
Signature · 2 lines
Grüsse / regards, Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization 
#3 lore raw

Keeping the old discussion in the mail tail, adding the new information up here were everyone finds them :-)

Things I was able to confirm so far summarized:
- The controller doesn't care about pfn ordering in any way (proved by HW statistics)
- regression appears in sequential AND random workloads -> also without readahead
- oprofile & co are no option atm.
The effective consumed cpu cycles per transferred kb are almost the
same so I would not expect sampling would give us huge insights.
Therefore I expect that it is more a matter of lost time (latency)
than more expensive tasks (cpu consumption)
I don't want to preclude it completely, but sampling has to wait as
long as we have better tracks to follow

So the question is where time is lost in Linux. I used blktrace to create latency summaries. I only list the random case for discussion as the effects are more clear int hat data. Abbreviations are (like the blkparse man page explains) - sorted in order it would appear per request:
A -- remap For stacked devices, incoming i/o is remapped to
device below it in the i/o stack. The remap action details what exactly is being remapped to what.
G -- get request To send any type of request to a block device, a
struct request container must be allocated first.
I -- inserted A request is being sent to the i/o scheduler for
addition to the internal queue and later service by the driver. The request is fully formed at this time.
D -- issued A request that previously resided on the block layer
queue or in the i/o scheduler has been sent to the driver.
C -- complete A previously issued request has been completed.
The output will detail the sector and size of that request, as well as the success or failure of it.

The following table shows the average latencies from A to G, G to I and so on.
C2A is special and tries to summarize how long it takes after completing an I/O until the next one arrives in the block device layer.

avg-A2G avg-G2I avg-I2D avg-D2C
avg-C2A-in-avg+-stddev %C2A-in-avg+-stddev
deviation good->bad -3.48% -0.56% -1.57% -1.31% 128.69% 97.26%

It clearly shows that all latencies once block device layer and device driver are involved are almost equal. Remember that the throughput of good vs. bad case is more than x3. But we can also see that the value of C2A increases by a huge amount. That huge C2A increase let me assume that the time is actually lost "above" the block device layer.

I don't expect the execution speed of iozone as user process itself is affected by commit e084b, so the question is where the time is lost between the "read" issued by iozone and entering the block device layer. Actually I expect it somewhere in the area of getting a page cache page for the I/O. On one hand page handling is what commit e084b changes and on the other hand pages are under pressure (systat vm effectiveness ~100%, >40% scanned directly in both cases).

I'll continue hunting down the lost time - maybe with ftrace if it is not concealing the effect by its invasiveness -, any further ideas/comments welcome.

kind regards,
Christian

Quoted 74 lines · Christian Ehrhardt
Quoted 7 lines · Mel Gorman

On Mon, Dec 07, 2009 at 03:39:49PM +0100, Christian Ehrhardt wrote:
[...]

I don't know what controller is in use there but does it opportunistically merge requests if they are on physically contiguous pages? If so, can it be disabled?

As far as i could clarify it our controllers don't support such a opportunistic merging.

Quoted 9 lines
Quoted 4 lines

The regression appears as up to 76% loss in throughput at 16 processes (processes are scaled from 1 to 64, performance is bad everywhere - 16 is just the peak - avg loss is about 40% throughput). I already know that giving the system just a bit (~64m+) more memory solves the issue almost completely, probably because there is almost no "memory pressure" left in that cases. I also know that using direct-I/O instead of I/O through page cache doesn't have the problem at all.

This makes sense because it's a sequentual read load, so readahead is a factor and that is why __GFP_COLD is used - the data is not for immediate use so doesn't need to be cache hot.

In the meanwhile I was able to verify that this also applies to random reads which are still reads but have zero read ahead requests. I attached more regression data in the post scriptum at the end of the mail.

Quoted 8 lines
Quoted 4 lines

Comparing sysstat data taken while running with the kernels just with & without the bisected patch shows nothing obvious except that I/O seems to take much longer (lower interrupt ratio etc).

Maybe the controller is spending an age trying to merge requests because it should be able to but takes a long time figuring it out?

I thought that too, but now comes the funny part.
I gathered HW statistics from our I/O controllers and latency statistics clearly state that your patch is working as intended - the latency from entering the controller until the interrupt to linux device driver is ~30% lower!. Remember as I stated above that they don't support that opportunistic merging so I will have some fun finding out why it is faster in HW now :-)

Quoted 11 lines
Quoted 5 lines

The patch alone looks very reasonable, so I'd prefer understanding and fixing the real issue instead of getting it eventually reverted due to this regression being larger than the one it was intended to fix. In the patch it is clear that hot pages (cold==0) freed in rmqueue_bulk should behave like before. So maybe the question is "are our pages cold while they shouldn't be"? Well I don't really have a clue yet to explain how patch e084b exactly causes that big regression, ideas welcome :-)

Only theory I have at the moment is that the controller notices it can merge requests and either spends a long time figuring out how to do the merging or performs worse with merged IO requests.

If the problem is in the driver, oprofile might show where the problem lies

With the effective throughput dropping by such a large amount while hardware latency improves by 30% I agree and suggest the issue is in the driver. I'll do some research in breaking down where in our drivers time is lost and reply here for advises and comments in regard to what general memory management could/should/might do.

Kind regards,
Christian

p.s.
FYI a bit more regression data, now I had the patch identified I made a full regression test scaling from 1 to 64 processes. Comparing just without / with the commit e084b
I wondered, but obviously random read is also suffering from that patch.

Sequential Read
Procs Deviation in %
1 -4.9
2 5.2

4             -82.6
8             -65.6
16            -44.2
32            -30.0
64            -37.7

Random Read
Procs Deviation in %

1             -48.3
2             -45.7
4             -50.5
8             -59.0
16            -61.8
32            -48.3
64            -21.0
Signature · 2 lines
Grüsse / regards, Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization 
#4 lore raw
Quoted 13 lines · Christian Ehrhardt

Keeping the old discussion in the mail tail, adding the new information up here were everyone finds them :-)

Things I was able to confirm so far summarized:
- The controller doesn't care about pfn ordering in any way (proved by HW statistics)
- regression appears in sequential AND random workloads -> also without readahead
- oprofile & co are no option atm.
The effective consumed cpu cycles per transferred kb are almost the
same so I would not expect sampling would give us huge insights.
Therefore I expect that it is more a matter of lost time (latency) than
more expensive tasks (cpu consumption)

But earlier, you said that latency was lower - "latency statistics clearly state that your patch is working as intended - the latency from entering the controller until the interrupt to linux device driver is ~30% lower!."

Also, if the controller is doing no merging of IO requests, why is the interrupt rate lower?

Quoted 31 lines

I don't want to preclude it completely, but sampling has to wait as
long as we have better tracks to follow

So the question is where time is lost in Linux. I used blktrace to create latency summaries.
I only list the random case for discussion as the effects are more clear int hat data.
Abbreviations are (like the blkparse man page explains) - sorted in order it would appear per request:
A -- remap For stacked devices, incoming i/o is remapped to device
below it in the i/o stack. The remap action details what exactly is being remapped to what.
G -- get request To send any type of request to a block device, a
struct request container must be allocated first.
I -- inserted A request is being sent to the i/o scheduler for
addition to the internal queue and later service by the driver. The request is fully formed at this time.
D -- issued A request that previously resided on the block layer
queue or in the i/o scheduler has been sent to the driver.
C -- complete A previously issued request has been completed. The
output will detail the sector and size of that request, as well as the success or failure of it.

The following table shows the average latencies from A to G, G to I and so on.
C2A is special and tries to summarize how long it takes after completing an I/O until the next one arrives in the block device layer.

avg-A2G avg-G2I avg-I2D avg-D2C avg-C2A-in-avg+-stddev %C2A-in-avg+-stddev
deviation good->bad -3.48% -0.56% -1.57% -1.31% 128.69% 97.26%

Quoted 7 lines

It clearly shows that all latencies once block device layer and device driver are involved are almost equal. Remember that the throughput of good vs. bad case is more than x3.
But we can also see that the value of C2A increases by a huge amount. That huge C2A increase let me assume that the time is actually lost "above" the block device layer.

To be clear. As C is "completion" and "A" is remapping new IO, it implies that time is being lost between when one IO completes and another starts, right?

Quoted 2 lines

I don't expect the execution speed of iozone as user process itself is affected by commit e084b,

Not by this much anyway. Lets say cache hotness is a problem, I would expect some performance loss but not this much.

Quoted 11 lines

so the question is where the time is lost
between the "read" issued by iozone and entering the block device layer. Actually I expect it somewhere in the area of getting a page cache page for the I/O. On one hand page handling is what commit e084b changes and on the other hand pages are under pressure (systat vm effectiveness ~100%, >40% scanned directly in both cases).

I'll continue hunting down the lost time - maybe with ftrace if it is not concealing the effect by its invasiveness -, any further ideas/comments welcome.

One way of finding out if cache hottness was the problem would be to profile for cache misses and see if there are massive differences with and without the patch. Is that an option?

Signature · 3 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
#5 lore raw
Quoted 16 lines · Mel Gorman
Quoted 13 lines · Christian Ehrhardt

Keeping the old discussion in the mail tail, adding the new information up here were everyone finds them :-)

Things I was able to confirm so far summarized:
- The controller doesn't care about pfn ordering in any way (proved by HW statistics)
- regression appears in sequential AND random workloads -> also without readahead
- oprofile & co are no option atm.
The effective consumed cpu cycles per transferred kb are almost the
same so I would not expect sampling would give us huge insights.
Therefore I expect that it is more a matter of lost time (latency) than
more expensive tasks (cpu consumption)

Quoted 4 lines

But earlier, you said that latency was lower - "latency statistics clearly state that your patch is working as intended - the latency from entering the controller until the interrupt to linux device driver is ~30% lower!."

Thats right, but the pure Hardware time is only lower due to the case of less I/O in flight. As it has less concurrency and contention that way a single I/O in HW is faster.
But in HW it's so fast (both cases) that as verified in the linux layers it doesn't matter. Both cases take more or less the same time from an I/O entering the block device layer until completion.

Quoted 3 lines

Also, if the controller is doing no merging of IO requests, why is the interrupt rate lower?

I was wondering about that when I started to work on this too, but the answer is simply that there are less requests coming in per second - and that implies a lower interrupt rate too.

Quoted 23 lines
Quoted 23 lines

I don't want to preclude it completely, but sampling has to wait as
long as we have better tracks to follow

So the question is where time is lost in Linux. I used blktrace to create latency summaries. I only list the random case for discussion as the effects are more clear int hat data. Abbreviations are (like the blkparse man page explains) - sorted in order it would appear per request:
A -- remap For stacked devices, incoming i/o is remapped to device
below it in the i/o stack. The remap action details what exactly is being remapped to what.
G -- get request To send any type of request to a block device, a
struct request container must be allocated first.
I -- inserted A request is being sent to the i/o scheduler for
addition to the internal queue and later service by the driver. The request is fully formed at this time.
D -- issued A request that previously resided on the block layer
queue or in the i/o scheduler has been sent to the driver.
C -- complete A previously issued request has been completed. The
output will detail the sector and size of that request, as well as the success or failure of it.

The following table shows the average latencies from A to G, G to I and so on.
C2A is special and tries to summarize how long it takes after completing an I/O until the next one arrives in the block device layer.

avg-A2G avg-G2I avg-I2D avg-D2C avg-C2A-in-avg+-stddev %C2A-in-avg+-stddev
deviation good->bad -3.48% -0.56% -1.57% -1.31% 128.69% 97.26%

Quoted 10 lines
Quoted 4 lines

It clearly shows that all latencies once block device layer and device driver are involved are almost equal. Remember that the throughput of good vs. bad case is more than x3. But we can also see that the value of C2A increases by a huge amount. That huge C2A increase let me assume that the time is actually lost "above" the block device layer.

To be clear. As C is "completion" and "A" is remapping new IO, it implies that time is being lost between when one IO completes and another starts, right?

Absolutely correct

Quoted 5 lines
Quoted 2 lines

I don't expect the execution speed of iozone as user process itself is affected by commit e084b,

Not by this much anyway. Lets say cache hotness is a problem, I would expect some performance loss but not this much.

I agree, even if it is cache hotness it wouldn't be that much.
And cold caches would appear as "longer" instructions because they would need more cycles due to e.g. dcache misses. But as mentioned before the cycles per transferred amount of data are the same so I don't expect it is due to cache hot/cold.

Quoted 11 lines
Quoted 6 lines

so the question is where the time is lost between the "read" issued by iozone and entering the block device layer.
Actually I expect it somewhere in the area of getting a page cache page for the I/O. On one hand page handling is what commit e084b changes and on the other hand pages are under pressure (systat vm effectiveness ~100%, >40% scanned directly in both cases).

I'll continue hunting down the lost time - maybe with ftrace if it is not concealing the effect by its invasiveness -, any further ideas/comments welcome.

One way of finding out if cache hottness was the problem would be to profile for cache misses and see if there are massive differences with and without the patch. Is that an option?

It is an option to verify things, but as mentioned above I would expect an increase amounts of consumed cycles per kb which I don't see. I'll track caches anyway to be sure.

My personal current assumption is that either there is some time lost from the read syscall until the A event blocktrace tracks or I'm wrong with my assumption about user processes and iozone runs "slower".

Signature · 2 lines
Grüsse / regards, Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization 
#6 lore raw
Quoted 4 lines · Christian Ehrhardt

Mel Gorman wrote:
[...]

Quoted 2 lines
Quoted 1 line
Quoted 9 lines
Quoted 4 lines

One way of finding out if cache hottness was the problem would be to profile for cache misses and see if there are massive differences with and without the patch. Is that an option?

It is an option to verify things, but as mentioned above I would expect an increase amounts of consumed cycles per kb which I don't see. I'll track caches anyway to be sure.

My personal current assumption is that either there is some time lost from the read syscall until the A event blocktrace tracks or I'm wrong with my assumption about user processes and iozone runs "slower".

After I was able to verify that the time is lost above the block device layer I ran further tests to check out where the additional latency due to commit e084b occurs. With "strace -frT -e trace=open,close,read,write" I could isolate the continuous stream of read calls done by the iozone child process.
In those data two things were revealed:
a) The time is lost between read and Block device enter (not e.g. due to slow userspace execution or scheduling effects)
Time spent in userspace until the next read call is almost the
same, but the average time spent "inside" the read call increases from 788µs to 3278µs
b) The time is lost in a few big peaks
The majority of read calls are ~500-750µs in both cases, but there
is a slight shift of approximately 8% that occur now as durations between 15000 and 60000µs

As mentioned before there is no significant increase of spent cpu cycles - therefore I watched out how the read call could end up in sleeps (latency but not used cycles).
I found that there is a chance to run into "congestion_wait" in case __alloc_pages_direct_reclaim() was able to free some pages with try_to_free_pages() but doesn't get a page in the subsequent get_page_from_freelist(). To get details about that I patched in some simple counters and ktime_get based timings. The results are impressive and point exactly towards the congestion_wait theory

The following data is again taken with iozone 1 thread random read in a very low memory environment, the counters were zeroed before starting iozone Bad = the kernel just with commit e084b, good = kernel one commit before e084b. The throughput in good case while taking this data was approximately ~168% of that of the bad case.

                                        GOOD     BAD
perf_count_congestion_wait                245    1388
perf_count_pages_direct_reclaim         10993    5365
perf_count_failed_pages_direct_reclaim      0    1090

Note - the data also shows that the average time spent in the f_op->aio_read call issued by do_sync_read increases by 68.5% which is directly inversely proportional to the average loss in throughput.

This lets me conclude that patch e084b causes the system to run into congestion_wait after the semi-failed call to __alloc_pages_direct_reclaim.
Unfortunately I don't understand memory management enough to find the relation between e084b actually changing the order and position of freed pages in the LRU list to __alloc_pages_direct_reclaim not getting pages as effectively as before. Ideas and comments welcome!

Kind regards,
Christian

P.S: attached are the three patches I used to get those new numbers

Signature · 2 lines
Grüsse / regards, Christian Ehrhardt
IBM Linux Technology Center, Linux Performance on System z 
fs/read_write.c +13 -1
Index: linux-2.6-git-2.6.31-bisect2/fs/read_write.c
===================================================================
@@ -249,25 +249,37 @@ static void wait_on_retry_sync_kiocb(str
 	__set_current_state(TASK_RUNNING);
 }
 
+unsigned long perf_count_do_sync_read_calls_aio_read = 0;
+unsigned long perf_tsum_do_sync_read_calls_aio_read = 0;
+unsigned long perf_count_do_sync_read_calls_wait_on_sync_kiocb = 0;
+unsigned long perf_tsum_do_sync_read_calls_wait_on_sync_kiocb = 0;
 ssize_t do_sync_read(struct file *filp, char __user *buf, size_t len, loff_t *ppos)
 {
 	struct iovec iov = { .iov_base = buf, .iov_len = len };
 	struct kiocb kiocb;
 	ssize_t ret;
+	ktime_t tv64_pre;
 
 	init_sync_kiocb(&kiocb, filp);
 	kiocb.ki_pos = *ppos;
 	kiocb.ki_left = len;
 
 	for (;;) {
+		perf_count_do_sync_read_calls_aio_read++;
+		tv64_pre = ktime_get();
 		ret = filp->f_op->aio_read(&kiocb, &iov, 1, kiocb.ki_pos);
+		perf_tsum_do_sync_read_calls_aio_read += ktime_sub(ktime_get(),tv64_pre).tv64;
 		if (ret != -EIOCBRETRY)
 			break;
 		wait_on_retry_sync_kiocb(&kiocb);
 	}
 
-	if (-EIOCBQUEUED == ret)
+	if (-EIOCBQUEUED == ret) {
+		perf_count_do_sync_read_calls_wait_on_sync_kiocb++;
+		tv64_pre = ktime_get();
 		ret = wait_on_sync_kiocb(&kiocb);
+		perf_tsum_do_sync_read_calls_wait_on_sync_kiocb += ktime_sub(ktime_get(),tv64_pre).tv64;
+	}
 	*ppos = kiocb.ki_pos;
 	return ret;
 }
kernel/sysctl.c +36 -0
Index: linux-2.6-git-2.6.31-bisect2/kernel/sysctl.c
===================================================================
@@ -257,6 +257,10 @@ extern unsigned long perf_count_pages_di
 extern unsigned long perf_count_failed_pages_direct_reclaim;
 extern unsigned long perf_count_do_generic_file_read_calls_page_cache_alloc_cold;
 extern unsigned long perf_tsum_do_generic_file_read_calls_page_cache_alloc_cold;
+extern unsigned long perf_count_do_sync_read_calls_aio_read;
+extern unsigned long perf_tsum_do_sync_read_calls_aio_read;
+extern unsigned long perf_count_do_sync_read_calls_wait_on_sync_kiocb;
+extern unsigned long perf_tsum_do_sync_read_calls_wait_on_sync_kiocb;
 static struct ctl_table perf_table[] = {
 	{
 		.ctl_name	= CTL_UNNUMBERED,
@@ -297,6 +301,38 @@ static struct ctl_table perf_table[] = {
 		.maxlen		= sizeof(unsigned long),
 		.mode           = 0666,
 		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_do_sync_read_calls_aio_read",
+		.data           = &perf_count_do_sync_read_calls_aio_read,
+		.maxlen		= sizeof(unsigned long),
+		.mode           = 0666,
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_tsum_do_sync_read_calls_aio_read",
+		.data           = &perf_tsum_do_sync_read_calls_aio_read,
+		.maxlen		= sizeof(unsigned long),
+		.mode           = 0666,
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_do_sync_read_calls_wait_on_sync_kiocb",
+		.data           = &perf_count_do_sync_read_calls_wait_on_sync_kiocb,
+		.maxlen		= sizeof(unsigned long),
+		.mode           = 0666,
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_tsum_do_sync_read_calls_wait_on_sync_kiocb",
+		.data           = &perf_tsum_do_sync_read_calls_wait_on_sync_kiocb,
+		.maxlen		= sizeof(unsigned long),
+		.mode           = 0666,
+		.proc_handler   = &proc_doulongvec_minmax,
 	},
 	{ .ctl_name = 0 }
 };
include/linux/sysctl.h +1 -0
Index: linux-2.6-git-2.6.31-bisect2/include/linux/sysctl.h
===================================================================
@@ -69,6 +69,7 @@ enum
 	CTL_BUS=8,		/* Busses */
 	CTL_ABI=9,		/* Binary emulation */
 	CTL_CPU=10,		/* CPU stuff (speed scaling, etc) */
+	CTL_PERF=11,		/* Performance counters and timer sums for debugging */
 	CTL_ARLAN=254,		/* arlan wireless driver */
 	CTL_S390DBF=5677,	/* s390 debug */
 	CTL_SUNRPC=7249,	/* sunrpc debug */
kernel/sysctl.c +21 -0
Index: linux-2.6-git-2.6.31-bisect2/kernel/sysctl.c
===================================================================
@@ -177,6 +177,7 @@ static struct ctl_table_root sysctl_tabl
 	.default_set.list = LIST_HEAD_INIT(root_table_header.ctl_entry),
 };
 
+static struct ctl_table perf_table[];
 static struct ctl_table kern_table[];
 static struct ctl_table vm_table[];
 static struct ctl_table fs_table[];
@@ -230,6 +231,13 @@ static struct ctl_table root_table[] = {
 		.mode		= 0555,
 		.child		= dev_table,
 	},
+	{
+		.ctl_name	= CTL_PERF,
+		.procname	= "perf",
+		.mode		= 0555,
+		.child		= perf_table,
+	},
+
 /*
  * NOTE: do not add new entries to this table unless you have read
  * Documentation/sysctl/ctl_unnumbered.txt
@@ -244,6 +252,19 @@ static int min_wakeup_granularity_ns;
 static int max_wakeup_granularity_ns = NSEC_PER_SEC;	/* 1 second */
 #endif
 
+extern unsigned long perf_count_congestion_wait;
+static struct ctl_table perf_table[] = {
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_congestion_wait",
+		.data           = &perf_count_congestion_wait,
+		.mode           = 0666,
+		.maxlen		= sizeof(unsigned long),
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{ .ctl_name = 0 }
+};
+
 static struct ctl_table kern_table[] = {
 #ifdef CONFIG_SCHED_DEBUG
 	{
mm/backing-dev.c +2 -0
Index: linux-2.6-git-2.6.31-bisect2/mm/backing-dev.c
===================================================================
@@ -305,6 +305,7 @@ void set_bdi_congested(struct backing_de
 }
 EXPORT_SYMBOL(set_bdi_congested);
 
+unsigned long perf_count_congestion_wait = 0;
 /**
  * congestion_wait - wait for a backing_dev to become uncongested
  * @sync: SYNC or ASYNC IO
@@ -320,6 +321,7 @@ long congestion_wait(int sync, long time
 	DEFINE_WAIT(wait);
 	wait_queue_head_t *wqh = &congestion_wqh[sync];
 
+	perf_count_congestion_wait++;
 	prepare_to_wait(wqh, &wait, TASK_UNINTERRUPTIBLE);
 	ret = io_schedule_timeout(timeout);
 	finish_wait(wqh, &wait);
kernel/sysctl.c +18 -0
Index: linux-2.6-git-2.6.31-bisect2/kernel/sysctl.c
===================================================================
@@ -253,6 +253,8 @@ static int max_wakeup_granularity_ns = N
 #endif
 
 extern unsigned long perf_count_congestion_wait;
+extern unsigned long perf_count_pages_direct_reclaim;
+extern unsigned long perf_count_failed_pages_direct_reclaim;
 static struct ctl_table perf_table[] = {
 	{
 		.ctl_name	= CTL_UNNUMBERED,
@@ -262,6 +264,22 @@ static struct ctl_table perf_table[] = {
 		.maxlen		= sizeof(unsigned long),
 		.proc_handler   = &proc_doulongvec_minmax,
 	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_pages_direct_reclaim",
+		.data           = &perf_count_pages_direct_reclaim,
+		.maxlen		= sizeof(unsigned long),
+		.mode           = 0666,
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_failed_pages_direct_reclaim",
+		.data           = &perf_count_failed_pages_direct_reclaim,
+		.maxlen		= sizeof(unsigned long),
+		.mode           = 0666,
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
 	{ .ctl_name = 0 }
 };
 
mm/page_alloc.c +8 -0
Index: linux-2.6-git-2.6.31-bisect2/mm/page_alloc.c
===================================================================
@@ -1605,6 +1605,9 @@ out:
 	return page;
 }
 
+unsigned long perf_count_pages_direct_reclaim = 0;
+unsigned long perf_count_failed_pages_direct_reclaim = 0;
+
 /* The really slow allocator path where we enter direct reclaim */
 static inline struct page *
 __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order,
@@ -1645,6 +1648,11 @@ __alloc_pages_direct_reclaim(gfp_t gfp_m
 					zonelist, high_zoneidx,
 					alloc_flags, preferred_zone,
 					migratetype);
+
+	perf_count_pages_direct_reclaim++;
+	if (!page)
+		perf_count_failed_pages_direct_reclaim++;
+
 	return page;
 }
 
#7 lore raw
Quoted 36 lines · Christian Ehrhardt
Quoted 10 lines · Christian Ehrhardt

Mel Gorman wrote:
[...]

Quoted 8 lines
Quoted 1 line

One way of finding out if cache hottness was the problem would be to profile
for cache misses and see if there are massive differences with and without
the patch. Is that an option?

Quoted 8 lines

It is an option to verify things, but as mentioned above I would expect an increase amounts of consumed cycles per kb which I don't see. I'll track caches anyway to be sure.

My personal current assumption is that either there is some time lost from the read syscall until the A event blocktrace tracks or I'm wrong with my assumption about user processes and iozone runs "slower".

After I was able to verify that the time is lost above the block device layer I ran further tests to check out where the additional latency due to commit e084b occurs.
With "strace -frT -e trace=open,close,read,write" I could isolate the continuous stream of read calls done by the iozone child process. In those data two things were revealed:
a) The time is lost between read and Block device enter (not e.g. due to slow userspace execution or scheduling effects)
Time spent in userspace until the next read call is almost the same,
but the average time spent "inside" the read call increases from 788µs to 3278µs
b) The time is lost in a few big peaks
The majority of read calls are ~500-750µs in both cases, but there is
a slight shift of approximately 8% that occur now as durations between 15000 and 60000µs

Quoted 7 lines

As mentioned before there is no significant increase of spent cpu cycles
- therefore I watched out how the read call could end up in sleeps (latency but not used cycles).
I found that there is a chance to run into "congestion_wait" in case __alloc_pages_direct_reclaim() was able to free some pages with try_to_free_pages() but doesn't get a page in the subsequent get_page_from_freelist().

This is true although it was also the case before the patch.

Quoted 15 lines

To get details about that I patched in some simple counters and ktime_get based timings. The results are impressive and point exactly towards the congestion_wait theory

The following data is again taken with iozone 1 thread random read in a very low memory environment, the counters were zeroed before starting iozone
Bad = the kernel just with commit e084b, good = kernel one commit before e084b. The throughput in good case while taking this data was approximately ~168% of that of the bad case.

                                        GOOD     BAD
perf_count_congestion_wait                245    1388
perf_count_pages_direct_reclaim         10993    5365
perf_count_failed_pages_direct_reclaim      0    1090
Quoted 7 lines

Note - the data also shows that the average time spent in the f_op->aio_read call issued by do_sync_read increases by 68.5% which is directly inversely proportional to the average loss in throughput.

This lets me conclude that patch e084b causes the system to run into congestion_wait after the semi-failed call to __alloc_pages_direct_reclaim.

You mention that the "GOOD" kernel is one commit before e084b but can you test with just that patch reverted please? As the patch is only about list manipulation, I'm failing to see how it can affect congestion_wait().

Quoted 4 lines

Unfortunately I don't understand memory management enough to find the relation between e084b actually changing the order and position of freed pages in the LRU list to __alloc_pages_direct_reclaim not getting pages as effectively as before.

Off-hand, neither can I. I regret that I'm going off-line for the Christmas very soon and I'll be in the middle of nowhere with no internet connection or test machines in the interim. It's going to be the new year before I get to look at this properly.

Even if this patch is not the real problem, your instrumentation patches will help pin down the real problem. Thanks and sorry I'll be delayed in getting back to you properly.

Quoted 253 lines

Ideas and comments welcome!

Kind regards,
Christian

P.S: attached are the three patches I used to get those new numbers

Signature · 6 lines
Grüsse / regards, Christian Ehrhardt
IBM Linux Technology Center, Linux Performance on System z 


Index: linux-2.6-git-2.6.31-bisect2/fs/read_write.c
===================================================================
fs/read_write.c +13 -1
@@ -249,25 +249,37 @@ static void wait_on_retry_sync_kiocb(str
 	__set_current_state(TASK_RUNNING);
 }
 
+unsigned long perf_count_do_sync_read_calls_aio_read = 0;
+unsigned long perf_tsum_do_sync_read_calls_aio_read = 0;
+unsigned long perf_count_do_sync_read_calls_wait_on_sync_kiocb = 0;
+unsigned long perf_tsum_do_sync_read_calls_wait_on_sync_kiocb = 0;
 ssize_t do_sync_read(struct file *filp, char __user *buf, size_t len, loff_t *ppos)
 {
 	struct iovec iov = { .iov_base = buf, .iov_len = len };
 	struct kiocb kiocb;
 	ssize_t ret;
+	ktime_t tv64_pre;
 
 	init_sync_kiocb(&kiocb, filp);
 	kiocb.ki_pos = *ppos;
 	kiocb.ki_left = len;
 
 	for (;;) {
+		perf_count_do_sync_read_calls_aio_read++;
+		tv64_pre = ktime_get();
 		ret = filp->f_op->aio_read(&kiocb, &iov, 1, kiocb.ki_pos);
+		perf_tsum_do_sync_read_calls_aio_read += ktime_sub(ktime_get(),tv64_pre).tv64;
 		if (ret != -EIOCBRETRY)
 			break;
 		wait_on_retry_sync_kiocb(&kiocb);
 	}
 
-	if (-EIOCBQUEUED == ret)
+	if (-EIOCBQUEUED == ret) {
+		perf_count_do_sync_read_calls_wait_on_sync_kiocb++;
+		tv64_pre = ktime_get();
 		ret = wait_on_sync_kiocb(&kiocb);
+		perf_tsum_do_sync_read_calls_wait_on_sync_kiocb += ktime_sub(ktime_get(),tv64_pre).tv64;
+	}
 	*ppos = kiocb.ki_pos;
 	return ret;
 }

Index: linux-2.6-git-2.6.31-bisect2/kernel/sysctl.c
===================================================================

kernel/sysctl.c +36 -0
@@ -257,6 +257,10 @@ extern unsigned long perf_count_pages_di
 extern unsigned long perf_count_failed_pages_direct_reclaim;
 extern unsigned long perf_count_do_generic_file_read_calls_page_cache_alloc_cold;
 extern unsigned long perf_tsum_do_generic_file_read_calls_page_cache_alloc_cold;
+extern unsigned long perf_count_do_sync_read_calls_aio_read;
+extern unsigned long perf_tsum_do_sync_read_calls_aio_read;
+extern unsigned long perf_count_do_sync_read_calls_wait_on_sync_kiocb;
+extern unsigned long perf_tsum_do_sync_read_calls_wait_on_sync_kiocb;
 static struct ctl_table perf_table[] = {
 	{
 		.ctl_name	= CTL_UNNUMBERED,
@@ -297,6 +301,38 @@ static struct ctl_table perf_table[] = {
 		.maxlen		= sizeof(unsigned long),
 		.mode           = 0666,
 		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_do_sync_read_calls_aio_read",
+		.data           = &perf_count_do_sync_read_calls_aio_read,
+		.maxlen		= sizeof(unsigned long),
+		.mode           = 0666,
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_tsum_do_sync_read_calls_aio_read",
+		.data           = &perf_tsum_do_sync_read_calls_aio_read,
+		.maxlen		= sizeof(unsigned long),
+		.mode           = 0666,
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_do_sync_read_calls_wait_on_sync_kiocb",
+		.data           = &perf_count_do_sync_read_calls_wait_on_sync_kiocb,
+		.maxlen		= sizeof(unsigned long),
+		.mode           = 0666,
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_tsum_do_sync_read_calls_wait_on_sync_kiocb",
+		.data           = &perf_tsum_do_sync_read_calls_wait_on_sync_kiocb,
+		.maxlen		= sizeof(unsigned long),
+		.mode           = 0666,
+		.proc_handler   = &proc_doulongvec_minmax,
 	},
 	{ .ctl_name = 0 }
 };

Index: linux-2.6-git-2.6.31-bisect2/include/linux/sysctl.h
===================================================================

include/linux/sysctl.h +1 -0
@@ -69,6 +69,7 @@ enum
 	CTL_BUS=8,		/* Busses */
 	CTL_ABI=9,		/* Binary emulation */
 	CTL_CPU=10,		/* CPU stuff (speed scaling, etc) */
+	CTL_PERF=11,		/* Performance counters and timer sums for debugging */
 	CTL_ARLAN=254,		/* arlan wireless driver */
 	CTL_S390DBF=5677,	/* s390 debug */
 	CTL_SUNRPC=7249,	/* sunrpc debug */

Index: linux-2.6-git-2.6.31-bisect2/kernel/sysctl.c
===================================================================

kernel/sysctl.c +21 -0
@@ -177,6 +177,7 @@ static struct ctl_table_root sysctl_tabl
 	.default_set.list = LIST_HEAD_INIT(root_table_header.ctl_entry),
 };
 
+static struct ctl_table perf_table[];
 static struct ctl_table kern_table[];
 static struct ctl_table vm_table[];
 static struct ctl_table fs_table[];
@@ -230,6 +231,13 @@ static struct ctl_table root_table[] = {
 		.mode		= 0555,
 		.child		= dev_table,
 	},
+	{
+		.ctl_name	= CTL_PERF,
+		.procname	= "perf",
+		.mode		= 0555,
+		.child		= perf_table,
+	},
+
 /*
  * NOTE: do not add new entries to this table unless you have read
  * Documentation/sysctl/ctl_unnumbered.txt
@@ -244,6 +252,19 @@ static int min_wakeup_granularity_ns;
 static int max_wakeup_granularity_ns = NSEC_PER_SEC;	/* 1 second */
 #endif
 
+extern unsigned long perf_count_congestion_wait;
+static struct ctl_table perf_table[] = {
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_congestion_wait",
+		.data           = &perf_count_congestion_wait,
+		.mode           = 0666,
+		.maxlen		= sizeof(unsigned long),
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{ .ctl_name = 0 }
+};
+
 static struct ctl_table kern_table[] = {
 #ifdef CONFIG_SCHED_DEBUG
 	{

Index: linux-2.6-git-2.6.31-bisect2/mm/backing-dev.c
===================================================================

mm/backing-dev.c +2 -0
@@ -305,6 +305,7 @@ void set_bdi_congested(struct backing_de
 }
 EXPORT_SYMBOL(set_bdi_congested);
 
+unsigned long perf_count_congestion_wait = 0;
 /**
  * congestion_wait - wait for a backing_dev to become uncongested
  * @sync: SYNC or ASYNC IO
@@ -320,6 +321,7 @@ long congestion_wait(int sync, long time
 	DEFINE_WAIT(wait);
 	wait_queue_head_t *wqh = &congestion_wqh[sync];
 
+	perf_count_congestion_wait++;
 	prepare_to_wait(wqh, &wait, TASK_UNINTERRUPTIBLE);
 	ret = io_schedule_timeout(timeout);
 	finish_wait(wqh, &wait);

Index: linux-2.6-git-2.6.31-bisect2/kernel/sysctl.c
===================================================================

kernel/sysctl.c +18 -0
@@ -253,6 +253,8 @@ static int max_wakeup_granularity_ns = N
 #endif
 
 extern unsigned long perf_count_congestion_wait;
+extern unsigned long perf_count_pages_direct_reclaim;
+extern unsigned long perf_count_failed_pages_direct_reclaim;
 static struct ctl_table perf_table[] = {
 	{
 		.ctl_name	= CTL_UNNUMBERED,
@@ -262,6 +264,22 @@ static struct ctl_table perf_table[] = {
 		.maxlen		= sizeof(unsigned long),
 		.proc_handler   = &proc_doulongvec_minmax,
 	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_pages_direct_reclaim",
+		.data           = &perf_count_pages_direct_reclaim,
+		.maxlen		= sizeof(unsigned long),
+		.mode           = 0666,
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_failed_pages_direct_reclaim",
+		.data           = &perf_count_failed_pages_direct_reclaim,
+		.maxlen		= sizeof(unsigned long),
+		.mode           = 0666,
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
 	{ .ctl_name = 0 }
 };
 

Index: linux-2.6-git-2.6.31-bisect2/mm/page_alloc.c
===================================================================

mm/page_alloc.c +8 -0
@@ -1605,6 +1605,9 @@ out:
 	return page;
 }
 
+unsigned long perf_count_pages_direct_reclaim = 0;
+unsigned long perf_count_failed_pages_direct_reclaim = 0;
+
 /* The really slow allocator path where we enter direct reclaim */
 static inline struct page *
 __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order,
@@ -1645,6 +1648,11 @@ __alloc_pages_direct_reclaim(gfp_t gfp_m
 					zonelist, high_zoneidx,
 					alloc_flags, preferred_zone,
 					migratetype);
+
+	perf_count_pages_direct_reclaim++;
+	if (!page)
+		perf_count_failed_pages_direct_reclaim++;
+
 	return page;
 }
 
Signature · 3 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
#8 lore raw
Quoted 14 lines · Mel Gorman
Quoted 3 lines · Christian Ehrhardt

Christian Ehrhardt wrote:

[...]

You mention that the "GOOD" kernel is one commit before e084b but can you test with just that patch reverted please? As the patch is only about list manipulation, I'm failing to see how it can affect congestion_wait().

Quoted 2 lines

Unfortunately I don't understand memory management enough to find the relation between e084b actually changing the order and position of freed pages in the LRU list to __alloc_pages_direct_reclaim not getting pages as effectively as before.

Quoted 9 lines

Off-hand, neither can I. I regret that I'm going off-line for the Christmas very soon and I'll be in the middle of nowhere with no internet connection or test machines in the interim. It's going to be the new year before I get to look at this properly.

Even if this patch is not the real problem, your instrumentation patches will help pin down the real problem. Thanks and sorry I'll be delayed in getting back to you properly.

Back to business :-)
First - yes it is reproducible in 2.6.32 final and fixable by unapplying e084b.
But I don't think un-applying it is really a fix as no one really understands how e084b cause this regression (it might just work around whatever goes on in the background and someone still hits the hidden issue).

Lets better look what we know summarized:
- the patch e084a causes this regression
- it causes it only in very rare if not theoretically high memory constraints
- time is lost between read system call and the block device layer enter
- via debugging and discussion we found that time lost is spent in congestion_wait

I did not yet check if it is possible, but we might have a spot tho might fix the issue.
Congestion_wait states it would "Waits for up to @timeout jiffies for a backing_dev (any backing_dev) to exit write congestion. If no backing_devs are congested then just wait for the next write to be completed.". Now in some case (and also my test) there is absolutely no write to do or in flight. So maybe it would be possible to either detect this before calling congestion_wait from page_alloc.c or let congestion_wait return an error code or something similar.

I mean as far as I can see the kernel currently does a loop like that (pseudo code and simplified):
1. get a page <- fail
2. try_to_free page <- returns >=1 pages freed
3. get_page_from_freelist <- fails
4. should_alloc_retry <- true
5. congestion_wait <- FUTILE - just loosing time, nothing will be freed by writes
6. goto 1
-> So maybe we should detect the futility of that congestion_wait call and not do it at all repeating instantly, go to oom, .., whatever (thats out for discussion).

If we can't fix it that way I would propose we keep the code as is, hoping that it is a theoretical case that never hits a customer system. But in that case we should definitely add a userspace readable counter to congestion_wait similar to my debug patches. This would allow everyone that ever assumes an issue might be related to this to verify it by checking the congestion_wait counter. This should not be too hard in a technical way, but also not performance critical as congestion_wait is going to wait anyway. On the other hand everyone hates introducing new kernel interfaces that need to be kept compatible until nirvana - especially for bug tracking its not the best idea :-) So it should be our very last resort.

Comments/Opinions to those ideas ?

Signature · 2 lines
Grüsse / regards, Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization 
#9 lore raw
Quoted 27 lines · Christian Ehrhardt
Quoted 16 lines · Mel Gorman
Quoted 2 lines · Christian Ehrhardt

Christian Ehrhardt wrote:

    [...]

You mention that the "GOOD" kernel is one commit before e084b but can you test with just that patch reverted please? As the patch is only about list manipulation, I'm failing to see how it can affect congestion_wait().

Quoted 5 lines

Unfortunately I don't understand memory management enough to find the
relation between e084b actually changing the order and position of
freed pages in the LRU list to __alloc_pages_direct_reclaim not getting pages as effectively as before.

Quoted 9 lines

Off-hand, neither can I. I regret that I'm going off-line for the Christmas very soon and I'll be in the middle of nowhere with no internet connection or test machines in the interim. It's going to be the new year before I get to look at this properly.

Even if this patch is not the real problem, your instrumentation patches will help pin down the real problem. Thanks and sorry I'll be delayed in getting back to you properly.

Quoted 8 lines

Back to business :-)
First - yes it is reproducible in 2.6.32 final and fixable by unapplying e084b.
But I don't think un-applying it is really a fix as no one really understands how e084b cause this regression (it might just work around whatever goes on in the background and someone still hits the hidden issue).

Agreed on reverting is not an option. While it is not understood why it causes your regression, it's known to fix a regression for hardware that does do merging.

Quoted 8 lines

Lets better look what we know summarized:
- the patch e084a causes this regression
- it causes it only in very rare if not theoretically high memory constraints
- time is lost between read system call and the block device layer enter
- via debugging and discussion we found that time lost is spent in congestion_wait

Thanks for the summary.

There is an outside chance it's due to a difference in free page availablity and the associated counters. Particularly if some subsystem is optimistally trying high-order allocations depending on availability. If there are many allocation failures, the counters gets skewed due to a bug, watermarks are not met and direct reclaim is used more. A patch is on its way upstream to fix this is http://www.spinics.net/lists/mm-commits/msg75671.html . Can you try it please?

There is a report on swap-based workloads being impacted on 2.6.32 and a fix exists but it wouldn't have affected 2.6.31. If you are testing on 2.6.32 though, this patch should be applied http://lkml.org/lkml/2009/12/21/245

Quoted 11 lines

I did not yet check if it is possible, but we might have a spot tho might fix the issue.
Congestion_wait states it would "Waits for up to @timeout jiffies for a backing_dev (any backing_dev) to exit write congestion. If no backing_devs are congested then just wait for the next write to be completed.".
Now in some case (and also my test) there is absolutely no write to do or in flight. So maybe it would be possible to either detect this before calling congestion_wait from page_alloc.c or let congestion_wait return an error code or something similar.

While I think that idea has merit, it's fixing the symptons and not the underlying problem. There is a growing opinion that congestion_wait() being used in the VM at all is a mistake. If the VM really needs to wait on pages being written out, it should have been on a waitqueue for a number of pages rather than a time-based method.

In other words, I'm more interesting in figuring out *why* we enter congestion_wait() at all with the patch and it's avoided without rather than what congestion_wait() does when it gets called.

Also, are you sure about thw "waiting for the next write to complete"? In 2.6.31 at least, it's waiting on the async write queue. If it's a case that it waits the full timeout if there is no async traffic then that's obviously bad as well but still doesn't explain why the problem patch makes a difference.

Quoted 13 lines

I mean as far as I can see the kernel currently does a loop like that (pseudo code and simplified):
1. get a page <- fail
2. try_to_free page <- returns >=1 pages freed
3. get_page_from_freelist <- fails
4. should_alloc_retry <- true
5. congestion_wait <- FUTILE - just loosing time, nothing will be freed by writes
6. goto 1
-> So maybe we should detect the futility of that congestion_wait call and not do it at all repeating instantly, go to oom, .., whatever (thats out for discussion).

Quoted 1 line

If we can't fix it that way I would propose we keep the code as is,

It's not my preferred way to fix this although I'm willing to explore it. I'd much prefer an explanation as to why your testcase is sensitive to the PFN ordering of the pages it gets back.

One possibility is that your testcase requires a number of high-order allocations and the patch is preventing them being merged. If that was the case, the following patch would also help
http://www.spinics.net/lists/mm-commits/msg75672.html but it's less than ideal.

Quoted 5 lines

hoping that it is a theoretical case that never hits a customer system. But in that case we should definitely add a userspace readable counter to congestion_wait similar to my debug patches. This would allow everyone that ever assumes an issue might be related to this to verify it by checking the congestion_wait counter.

Would the accounting features available with CONFIG_TASK_DELAY_ACCT be sufficient? There is a description in Documentation/accounting/delay-accounting.txt on how to use it. An abnormal amount of time spent on IO might explain the problem.

Quoted 6 lines

This should not be too hard in a technical way, but also not performance critical as congestion_wait is going to wait anyway. On the other hand everyone hates introducing new kernel interfaces that need to be kept compatible until nirvana - especially for bug tracking its not the best idea :-) So it should be our very last resort.

Agreed.

Signature · 8 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
#10 lore raw

I'll keep the old thread below as reference.

After taking a round of ensuring reproducibility and a pile of new measurements I now can come back with several new insights.

FYI - I'm now running iozone triplets (4, then 8, then 16 parallel threads) with sequential read load and all that 4 times to find potential noise. But since I changed to that load instead of random read wit hone thread and ensuring the most memory is cleared (sync + echo 3 > /proc/sys/vm/drop_caches + a few sleeps) . The noise is now down <2%. For detailed questions about the setup feel free to ask me directly as I won't flood this thread too much with such details.

So in the past I identified git id e084b for bringing a huge degradation, that is still true, but I need to revert my old statement that unapplying e084b on v2.6.32 helps - it simply doesn't.

Another bisect (keepign e084b reverted) brought up git id 5f8dcc21 which came in later. Both patches unapplied individually don't improve anything. But both patches reverted at the same time on git v2.6.32 bring us back to our old values (remember that is more than 2Gb/s improvement in throughput)!

Unfortunately 5f8dcc21 is as unobvious as e84b in explaining how this can cause so much trouble.

In the past I identified congestion_wait as the point where the "time is lost" when comparing good and bad case. Fortunately at least this is still true when comparing 2.6.32 vs 2.6.32 with both patches reverted. So I upgraded my old counter patches a bit and can now report the following:

BAD CASE
4 THREAD
READ 8 THREAD READ 16 THREAD READ 16THR % portions
perf_count_congestion_wait 305 1970 8980
perf_count_call_congestion_wait_from_alloc_pages_high_priority 0 0 0

perf_count_call_congestion_wait_from_alloc_pages_slowpath         305                        1970                        8979                100.00%
perf_count_pages_direct_reclaim                                  1153                        6818                        3221
perf_count_failed_pages_direct_reclaim                            305                        1556                        8979
perf_count_failed_pages_direct_reclaim_but_progress               305                        1478                        8979                27.87%

GOOD CASE WITH REVERTS 4 THREAD READ 8 THREAD READ 16 THREAD READ 16THR % portions
perf_count_congestion_wait 25 76 1114
perf_count_call_congestion_wait_from_alloc_pages_high_priority 0 0 0
perf_count_call_congestion_wait_from_alloc_pages_slowpath 25 76 1114 99.98%
perf_count_pages_direct_reclaim 1054 9150 62297

perf_count_failed_pages_direct_reclaim                             25                          64                         1114
perf_count_failed_pages_direct_reclaim_but_progress                25                          57                         1114                1.79%

I hope the format is kept, it should be good with every monospace viewer.
You can all clearly see that the patches somehow affect the ratio at which __alloc_pages_direct_reclaim runs into a race between try_to_free pages that could actually free something, but a few lines below can't get a page from the free list. Outside in the function alloc_pages_slowpath this leads to a call to congestion_wait which is absolutely futile as there are absolutely no writes in flight to wait for.

Now this kills effectively up to 80% of our throughput - Any idea of better understanding the link between the patches and the effect is welcome and might lead to a solution.

FYI - I tried all patches you suggested - none of them affects this.

Updated perf counter patches are attached.
As for some questions in the old thread:

Quoted 10 lines
Quoted 5 lines

Back to business :-)
First - yes it is reproducible in 2.6.32 final and fixable by unapplying e084b.
But I don't think un-applying it is really a fix as no one really understands how e084b cause this regression (it might just work around whatever goes on in the background and someone still hits the hidden issue).

Agreed on reverting is not an option. While it is not understood why it causes your regression, it's known to fix a regression for hardware that does do merging.

I'm not taking this point but fairly compared it is +15% vs -80% - so it is not "only" good :-)

Quoted 20 lines
Quoted 9 lines

Lets better look what we know summarized:
- the patch e084a causes this regression
- it causes it only in very rare if not theoretically high memory constraints
- time is lost between read system call and the block device layer enter
- via debugging and discussion we found that time lost is spent in congestion_wait

Thanks for the summary.

There is an outside chance it's due to a difference in free page availablity and the associated counters. Particularly if some subsystem is optimistally trying high-order allocations depending on availability. If there are many allocation failures, the counters gets skewed due to a bug, watermarks are not met and direct reclaim is used more. A patch is on its way upstream to fix this is http://www.spinics.net/lists/mm-commits/msg75671.html . Can you try it please?

Quoted 4 lines

There is a report on swap-based workloads being impacted on 2.6.32 and a fix exists but it wouldn't have affected 2.6.31. If you are testing on 2.6.32 though, this patch should be applied http://lkml.org/lkml/2009/12/21/245

both are not affecting my scenario at all

Quoted 13 lines
Quoted 6 lines

I did not yet check if it is possible, but we might have a spot tho might fix the issue.
Congestion_wait states it would "Waits for up to @timeout jiffies for a backing_dev (any backing_dev) to exit write congestion. If no backing_devs are congested then just wait for the next write to be completed.". Now in some case (and also my test) there is absolutely no write to do or in flight. So maybe it would be possible to either detect this before calling congestion_wait from page_alloc.c or let congestion_wait return an error code or something similar.

While I think that idea has merit, it's fixing the symptons and not the underlying problem. There is a growing opinion that congestion_wait() being used in the VM at all is a mistake. If the VM really needs to wait on pages being written out, it should have been on a waitqueue for a number of pages rather than a time-based method.

Quoted 9 lines

In other words, I'm more interesting in figuring out *why* we enter congestion_wait() at all with the patch and it's avoided without rather than what congestion_wait() does when it gets called.

Also, are you sure about thw "waiting for the next write to complete"? In 2.6.31 at least, it's waiting on the async write queue. If it's a case that it waits the full timeout if there is no async traffic then that's obviously bad as well but still doesn't explain why the problem patch makes a difference.

I agree that this part of the discussion is about symptoms and for now we should try to focus on the cause that avtually brings up the race in __alloc_pages_direct_reclaim.

Quoted 13 lines
Quoted 10 lines

I mean as far as I can see the kernel currently does a loop like that (pseudo code and simplified):
1. get a page <- fail
2. try_to_free page <- returns >=1 pages freed
3. get_page_from_freelist <- fails
4. should_alloc_retry <- true
5. congestion_wait <- FUTILE - just loosing time, nothing will be freed by writes
6. goto 1
-> So maybe we should detect the futility of that congestion_wait call
and not do it at all repeating instantly, go to oom, .., whatever (thats out for discussion).

Quoted 2 lines

If we can't fix it that way I would propose we keep the code as is,

Quoted 10 lines

It's not my preferred way to fix this although I'm willing to explore it. I'd much prefer an explanation as to why your testcase is sensitive to the PFN ordering of the pages it gets back.

One possibility is that your testcase requires a number of high-order allocations and the patch is preventing them being merged. If that was the case, the following patch would also help
http://www.spinics.net/lists/mm-commits/msg75672.html but it's less than ideal.

again not affecting my workload at all

Quoted 9 lines
Quoted 2 lines

hoping that it is a theoretical case that never hits a customer system. But in that case we should definitely add a userspace readable counter to congestion_wait similar to my debug patches. This would allow everyone that ever assumes an issue might be related to this to verify it by checking the congestion_wait counter.

Would the accounting features available with CONFIG_TASK_DELAY_ACCT be sufficient? There is a description in Documentation/accounting/delay-accounting.txt on how to use it. An abnormal amount of time spent on IO might explain the problem.

Eventually it will appear as a huge increase in I/O-wait for no obvious reason - DELAY_ACCT is fine, actually even plain iostat is be enough. The congestion_wait counter would just help to differ this issue, from others also increasing I/O-wait - but that is not too important. I think we can all live without a counter and better fix the cause :-)

Quoted 7 lines
Quoted 3 lines

This should not be too hard in a technical way, but also not performance critical as congestion_wait is going to wait anyway. On the other hand everyone hates introducing new kernel interfaces that need to be kept compatible until nirvana - especially for bug tracking its not the best idea :-) So it should be our very last resort.

Agreed.

Signature · 2 lines
Grüsse / regards, Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization 
include/linux/sysctl.h +1 -0
Index: linux-2.6-git/include/linux/sysctl.h
===================================================================
@@ -68,6 +68,7 @@
 	CTL_BUS=8,		/* Busses */
 	CTL_ABI=9,		/* Binary emulation */
 	CTL_CPU=10,		/* CPU stuff (speed scaling, etc) */
+  	CTL_PERF=11,		/* Performance counters and timer sums for debugging */
 	CTL_ARLAN=254,		/* arlan wireless driver */
 	CTL_S390DBF=5677,	/* s390 debug */
 	CTL_SUNRPC=7249,	/* sunrpc debug */
kernel/sysctl.c +21 -0
Index: linux-2.6-git/kernel/sysctl.c
===================================================================
@@ -183,6 +183,7 @@
 	.default_set.list = LIST_HEAD_INIT(root_table_header.ctl_entry),
 };
 
+static struct ctl_table perf_table[];
 static struct ctl_table kern_table[];
 static struct ctl_table vm_table[];
 static struct ctl_table fs_table[];
@@ -236,6 +237,13 @@
 		.mode		= 0555,
 		.child		= dev_table,
 	},
+	{
+		.ctl_name	= CTL_PERF,
+		.procname	= "perf",
+		.mode		= 0555,
+		.child		= perf_table,
+	},
+
 /*
  * NOTE: do not add new entries to this table unless you have read
  * Documentation/sysctl/ctl_unnumbered.txt
@@ -250,6 +258,19 @@
 static int max_wakeup_granularity_ns = NSEC_PER_SEC;	/* 1 second */
 #endif
 
+extern unsigned long perf_count_congestion_wait;
+static struct ctl_table perf_table[] = {
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_congestion_wait",
+		.data           = &perf_count_congestion_wait,
+		.mode           = 0666,
+		.maxlen		= sizeof(unsigned long),
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{ .ctl_name = 0 }
+};
+
 static struct ctl_table kern_table[] = {
 	{
 		.ctl_name	= CTL_UNNUMBERED,
mm/backing-dev.c +2 -0
Index: linux-2.6-git/mm/backing-dev.c
===================================================================
@@ -739,6 +739,7 @@
 }
 EXPORT_SYMBOL(set_bdi_congested);
 
+unsigned long perf_count_congestion_wait = 0;
 /**
  * congestion_wait - wait for a backing_dev to become uncongested
  * @sync: SYNC or ASYNC IO
@@ -754,6 +755,7 @@
 	DEFINE_WAIT(wait);
 	wait_queue_head_t *wqh = &congestion_wqh[sync];
 
+	perf_count_congestion_wait++;
 	prepare_to_wait(wqh, &wait, TASK_UNINTERRUPTIBLE);
 	ret = io_schedule_timeout(timeout);
 	finish_wait(wqh, &wait);
kernel/sysctl.c +18 -0
Index: linux-2.6-git/kernel/sysctl.c
===================================================================
@@ -262,6 +262,8 @@
 extern unsigned long perf_count_pages_direct_reclaim;
 extern unsigned long perf_count_failed_pages_direct_reclaim;
 extern unsigned long perf_count_failed_pages_direct_reclaim_but_progress;
+extern unsigned long perf_count_call_congestion_wait_from_alloc_pages_high_priority;
+extern unsigned long perf_count_call_congestion_wait_from_alloc_pages_slowpath;
 static struct ctl_table perf_table[] = {
 	{
 		.ctl_name	= CTL_UNNUMBERED,
@@ -270,6 +272,22 @@
 		.mode           = 0666,
 		.maxlen		= sizeof(unsigned long),
 		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_call_congestion_wait_from_alloc_pages_high_priority",
+		.data           = &perf_count_call_congestion_wait_from_alloc_pages_high_priority,
+		.mode           = 0666,
+		.maxlen		= sizeof(unsigned long),
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_call_congestion_wait_from_alloc_pages_slowpath",
+		.data           = &perf_count_call_congestion_wait_from_alloc_pages_slowpath,
+		.mode           = 0666,
+		.maxlen		= sizeof(unsigned long),
+		.proc_handler   = &proc_doulongvec_minmax,
 	},
 	{
 		.ctl_name	= CTL_UNNUMBERED,
mm/page_alloc.c +9 -3
Index: linux-2.6-git/mm/page_alloc.c
===================================================================
@@ -1663,6 +1663,7 @@
 
 unsigned long perf_count_pages_direct_reclaim = 0;
 unsigned long perf_count_failed_pages_direct_reclaim = 0;
+unsigned long perf_count_failed_pages_direct_reclaim_but_progress = 0;
 
 /* The really slow allocator path where we enter direct reclaim */
 static inline struct page *
@@ -1704,12 +1705,13 @@
 	perf_count_pages_direct_reclaim++;
 	if (!page)
 		perf_count_failed_pages_direct_reclaim++;
-	if (!page && !(*did_some_progress))
-		perf_count_failed_pages_direct_reclaim_but_progress++
+	if (!page && (*did_some_progress))
+		perf_count_failed_pages_direct_reclaim_but_progress++;
 
 	return page;
 }
 
+unsigned long perf_count_call_congestion_wait_from_alloc_pages_high_priority = 0;
 /*
  * This is called in the allocator slow-path if the allocation request is of
  * sufficient urgency to ignore watermarks and take other desperate measures
@@ -1727,8 +1729,10 @@
 			zonelist, high_zoneidx, ALLOC_NO_WATERMARKS,
 			preferred_zone, migratetype);
 
-		if (!page && gfp_mask & __GFP_NOFAIL)
+		if (!page && gfp_mask & __GFP_NOFAIL) {
+			perf_count_call_congestion_wait_from_alloc_pages_high_priority++;
 			congestion_wait(BLK_RW_ASYNC, HZ/50);
+		}
 	} while (!page && (gfp_mask & __GFP_NOFAIL));
 
 	return page;
@@ -1783,6 +1787,7 @@
 	return alloc_flags;
 }
 
+unsigned long perf_count_call_congestion_wait_from_alloc_pages_slowpath = 0;
 static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	struct zonelist *zonelist, enum zone_type high_zoneidx,
@@ -1899,6 +1904,7 @@
 	pages_reclaimed += did_some_progress;
 	if (should_alloc_retry(gfp_mask, order, pages_reclaimed)) {
 		/* Wait for some write requests to complete then retry */
+		perf_count_call_congestion_wait_from_alloc_pages_slowpath++;
 		congestion_wait(BLK_RW_ASYNC, HZ/50);
 		goto rebalance;
 	}
kernel/sysctl.c +27 -0
Index: linux-2.6-git/kernel/sysctl.c
===================================================================
@@ -259,6 +259,9 @@
 #endif
 
 extern unsigned long perf_count_congestion_wait;
+extern unsigned long perf_count_pages_direct_reclaim;
+extern unsigned long perf_count_failed_pages_direct_reclaim;
+extern unsigned long perf_count_failed_pages_direct_reclaim_but_progress;
 static struct ctl_table perf_table[] = {
 	{
 		.ctl_name	= CTL_UNNUMBERED,
@@ -268,6 +271,30 @@
 		.maxlen		= sizeof(unsigned long),
 		.proc_handler   = &proc_doulongvec_minmax,
 	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_pages_direct_reclaim",
+		.data           = &perf_count_pages_direct_reclaim,
+		.maxlen		= sizeof(unsigned long),
+		.mode           = 0666,
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_failed_pages_direct_reclaim",
+		.data           = &perf_count_failed_pages_direct_reclaim,
+		.maxlen		= sizeof(unsigned long),
+		.mode           = 0666,
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_failed_pages_direct_reclaim_but_progress",
+		.data           = &perf_count_failed_pages_direct_reclaim_but_progress,
+		.maxlen		= sizeof(unsigned long),
+		.mode           = 0666,
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
 	{ .ctl_name = 0 }
 };
 
mm/page_alloc.c +10 -0
Index: linux-2.6-git/mm/page_alloc.c
===================================================================
@@ -1661,6 +1661,9 @@
 	return page;
 }
 
+unsigned long perf_count_pages_direct_reclaim = 0;
+unsigned long perf_count_failed_pages_direct_reclaim = 0;
+
 /* The really slow allocator path where we enter direct reclaim */
 static inline struct page *
 __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order,
@@ -1697,6 +1700,13 @@
 					zonelist, high_zoneidx,
 					alloc_flags, preferred_zone,
 					migratetype);
+
+	perf_count_pages_direct_reclaim++;
+	if (!page)
+		perf_count_failed_pages_direct_reclaim++;
+	if (!page && !(*did_some_progress))
+		perf_count_failed_pages_direct_reclaim_but_progress++
+
 	return page;
 }
 
#11 lore raw
Quoted 5 lines · Christian Ehrhardt

I'll keep the old thread below as reference.

After taking a round of ensuring reproducibility and a pile of new measurements I now can come back with several new insights.

Quoted 8 lines

FYI - I'm now running iozone triplets (4, then 8, then 16 parallel threads) with sequential read load and all that 4 times to find potential noise. But since I changed to that load instead of random read wit hone thread and ensuring the most memory is cleared (sync + echo 3 > /proc/sys/vm/drop_caches + a few sleeps) . The noise is now down <2%. For detailed questions about the setup feel free to ask me directly as I won't flood this thread too much with such details.

Is there any chance you have a driver script for the test that you could send me? I'll then try reproducing based on that script and see what happens. I'm not optimistic I'll be able to reproduce the problem because I think it's specific to your setup but you never know.

Quoted 4 lines

So in the past I identified git id e084b for bringing a huge degradation, that is still true, but I need to revert my old statement that unapplying e084b on v2.6.32 helps - it simply doesn't.

Hmm, ok. Odd that it used to work but now doesn't.

How reproducible are these results with patch e084b reverted? i.e. I know it does not work now, but did reverting on the old kernel always fix it or were there occasions where the figures would be still bad?

Quoted 9 lines

Another bisect (keepign e084b reverted) brought up git id 5f8dcc21 which came in later. Both patches unapplied individually don't improve anything. But both patches reverted at the same time on git v2.6.32 bring us back to our old values (remember that is more than 2Gb/s improvement in throughput)!

Unfortunately 5f8dcc21 is as unobvious as e84b in explaining how this can cause so much trouble.

There is a bug in commit 5f8dcc21. One consequence of it is that swap-based workloads can suffer. A second is that users of high-order allocations can enter direct reclaim a lot more than previously. This was fixed in commit a7016235a61d520e6806f38129001d935c4b6661 but you say that's not the fix in your case.

The only other interesting thing in commit 5f8dcc21 is that it increases the memory overhead of a per-cpu structure. If your memory usage is really borderline, it might have been just enough to push it over the edge.

Quoted 5 lines

In the past I identified congestion_wait as the point where the "time is lost" when comparing good and bad case. Fortunately at least this is still true when comparing 2.6.32 vs 2.6.32 with both patches reverted. So I upgraded my old counter patches a bit and can now report the following:

Quoted 7 lines

BAD CASE
4 THREAD READ 8 THREAD READ 16 THREAD READ 16THR % portions
perf_count_congestion_wait 305 1970 8980
perf_count_call_congestion_wait_from_alloc_pages_high_priority 0 0 0

perf_count_call_congestion_wait_from_alloc_pages_slowpath         305                        1970                        8979                   100.00%
perf_count_pages_direct_reclaim                                  1153                        6818                        3221
perf_count_failed_pages_direct_reclaim                            305                        1556                        8979

Something is wrong with the counters there. For 16 threads, it says direct reclaim was entered 3221 but it failed 8979 times. How does that work? The patch you supplied below looks like a patch on top of another debugging patch. Can you send the full debugging patch please?

Quoted 7 lines

perf_count_failed_pages_direct_reclaim_but_progress 305 1478 8979 27.87%

GOOD CASE WITH REVERTS 4 THREAD READ 8 THREAD READ 16 THREAD READ 16THR % portions
perf_count_congestion_wait 25 76 1114
perf_count_call_congestion_wait_from_alloc_pages_high_priority 0 0 0
perf_count_call_congestion_wait_from_alloc_pages_slowpath 25 76 1114 99.98%
perf_count_pages_direct_reclaim 1054 9150 62297

Quoted 5 lines
perf_count_failed_pages_direct_reclaim                             25                          64                         1114
perf_count_failed_pages_direct_reclaim_but_progress                25                          57                         1114                1.79%

I hope the format is kept, it should be good with every monospace viewer.

It got manged but I think I've it fixed above. The biggest thing I can see is that direct reclaim is a lot more successful with the patches reverted but that in itself doesn't make sense. Neither patch affects how many pages should be free or reclaimable - just what order they are allocated and freed in.

With botgh patches reverted, is the performance 100% reliable or does it sometimes vary?

If reverting 5f8dcc21 is required, I'm leaning towards believing that the additional memory overhead with that patch is enough to push this workload over the edge where entering direct reclaim is necessary a lot more to keep the zones over the min watermark. You mentioned early on that adding 64MB to the machine makes the problem go away. Do you know what the cut-off point is? For example, is adding 4MB enough?

If a small amount of memory does help, I'd also test with min_free_kbytes at a lower value? If reducing it restores the performance, it again implies that memory usage is the real problem.

Thing is, even if adding small amounts of memory or reducing min_free_kbytes helps, it does not explain why e084b ever made a difference because all that patch does is alter the ordering of pages on the free list. That might have some cache consequences but it would not impact direct reclaim like this.

Quoted 4 lines

You can all clearly see that the patches somehow affect the ratio at which __alloc_pages_direct_reclaim runs into a race between try_to_free pages that could actually free something, but a few lines below can't get a page from the free list.

There actually is potentially a significant delay from when direct reclaim returns and a new allocation attempt happens. However, it's been like this for a long time so I don't think it's the issue.

I have a prototype patch that removes usage of congestion_wait() altogether and puts processes to sleep on a waitqueue waiting for watermarks to restore or a timeout. However, I think it would be just treating the symptoms here rather than the real issue.

Quoted 10 lines

Outside in the function alloc_pages_slowpath this leads to a call to congestion_wait which is absolutely futile as there are absolutely no writes in flight to wait for.

Now this kills effectively up to 80% of our throughput - Any idea of better understanding the link between the patches and the effect is welcome and might lead to a solution.

FYI - I tried all patches you suggested - none of them affects this.

I'm still at a total loss to explain this. Memory overhead of the second patch is a vague possibility and worth checking out by slightly increasing available memory on the partition or reducing min_free_kbytes. It does not explain why the first patch makes a difference.

Quoted 1 line

<SNIP>

Signature · 3 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
#12 lore raw
Quoted 6 lines · Mel Gorman
Quoted 4 lines · Christian Ehrhardt

I'll keep the old thread below as reference.

After taking a round of ensuring reproducibility and a pile of new measurements I now can come back with several new insights.

Quoted 9 lines
Quoted 3 lines

FYI - I'm now running iozone triplets (4, then 8, then 16 parallel threads) with sequential read load and all that 4 times to find potential noise. But since I changed to that load instead of random read wit hone thread and ensuring the most memory is cleared (sync + echo 3 > /proc/sys/vm/drop_caches + a few sleeps) . The noise is now down <2%. For detailed questions about the setup feel free to ask me directly as I won't flood this thread too much with such details.

Is there any chance you have a driver script for the test that you could send me? I'll then try reproducing based on that script and see what happens. I'm not optimistic I'll be able to reproduce the problem because I think it's specific to your setup but you never know.

I don't have one as it runs in a bigger automated test environment, but it is easy enough to write down something comparable.

Quoted 10 lines
Quoted 3 lines

So in the past I identified git id e084b for bringing a huge degradation, that is still true, but I need to revert my old statement that unapplying e084b on v2.6.32 helps - it simply doesn't.

Hmm, ok. Odd that it used to work but now doesn't.

How reproducible are these results with patch e084b reverted? i.e. I know it does not work now, but did reverting on the old kernel always fix it or were there occasions where the figures would be still bad?

Reverting e084b in the past showed something like +40%, so I thought it helps.
Afterwards I found out it wasn't a good testcase for reproducibility and when looking at a series it had +5%,-7%,+20%,... So by far too noisy to be useful for bisect, discussion or fix testing.
Thats why I made my big round reinventing the testcases in a more reproducible way like described above.
In those the original issue is still very visible - which means 4 runs comparing 2.6.32 vs 2.6.32-Reve084b being each max +/-1%. While gitid-e084b vs. the one just before (51fbb) gives ~-60% all the time.

Quoted 12 lines
Quoted 5 lines

Another bisect (keepign e084b reverted) brought up git id 5f8dcc21 which came in later. Both patches unapplied individually don't improve anything. But both patches reverted at the same time on git v2.6.32 bring us back to our old values (remember that is more than 2Gb/s improvement in throughput)!

Unfortunately 5f8dcc21 is as unobvious as e84b in explaining how this can cause so much trouble.

There is a bug in commit 5f8dcc21. One consequence of it is that swap-based workloads can suffer. A second is that users of high-order allocations can enter direct reclaim a lot more than previously. This was fixed in commit a7016235a61d520e6806f38129001d935c4b6661 but you say that's not the fix in your case.

Quoted 4 lines

The only other interesting thing in commit 5f8dcc21 is that it increases the memory overhead of a per-cpu structure. If your memory usage is really borderline, it might have been just enough to push it over the edge.

I had this thoughts too, but as it only shows an effect with 5f8dcc21 and e084b reverted at the same time I'm wondering if it can be that. But I agree that this should be considered too until a verification can be done.

Quoted 13 lines
Quoted 4 lines

In the past I identified congestion_wait as the point where the "time is lost" when comparing good and bad case. Fortunately at least this is still true when comparing 2.6.32 vs 2.6.32 with both patches reverted. So I upgraded my old counter patches a bit and can now report the following:

Quoted 8 lines

BAD CASE
4 THREAD READ 8 THREAD READ 16 THREAD READ 16THR % portions
perf_count_congestion_wait 305 1970 8980
perf_count_call_congestion_wait_from_alloc_pages_high_priority 0 0 0

perf_count_call_congestion_wait_from_alloc_pages_slowpath         305                        1970                        8979                   100.00%
perf_count_pages_direct_reclaim                                  1153                        6818                        3221
perf_count_failed_pages_direct_reclaim                            305                        1556                        8979
Quoted 5 lines

Something is wrong with the counters there. For 16 threads, it says direct reclaim was entered 3221 but it failed 8979 times. How does that work? The patch you supplied below looks like a patch on top of another debugging patch. Can you send the full debugging patch please?

This is just a single 7 deleted accidentally when bringing it to a plain text format for this mail. The percentage is fine and properly it would look like:

4 THREAD READ 8 THREAD READ 16 THREAD READ 16THR % portions

perf_count_congestion_wait 305 1970 8980
perf_count_call_congestion_wait_from_alloc_pages_high_priority 0 0 0

perf_count_call_congestion_wait_from_alloc_pages_slowpath         305                        1970                         8979               100.00%
perf_count_pages_direct_reclaim                                  1153                        6818                        32217
perf_count_failed_pages_direct_reclaim                            305                        1556                         8979
perf_count_failed_pages_direct_reclaim_but_progress               305                        1478                         8979               27.87%

There were three patches attached in the last mail which should work in this order:

~/kernels/linux-2.6$ git checkout -f v2.6.32
~/kernels/linux-2.6$ patch -p1 < ~/Desktop/patches-git/perf-count-congestion-wait.diff patching file include/linux/sysctl.h
patching file kernel/sysctl.c
patching file mm/backing-dev.c
~/kernels/linux-2.6$ patch -p1 < ~/Desktop/patches-git/perf-count-failed-direct-recalims.diff patching file kernel/sysctl.c
patching file mm/page_alloc.c
Hunk #1 succeeded at 1670 (offset 9 lines).
Hunk #2 succeeded at 1709 (offset 9 lines).
~/kernels/linux-2.6$ patch -p1 < ~/Desktop/patches-git/perf-count-congestion-wait-calls.diff patching file kernel/sysctl.c
patching file mm/page_alloc.c
Hunk #1 succeeded at 1672 (offset 9 lines).
Hunk #2 succeeded at 1714 (offset 9 lines).
Hunk #3 succeeded at 1738 (offset 9 lines).
Hunk #4 succeeded at 1796 (offset 9 lines).
Hunk #5 succeeded at 1913 (offset 9 lines).

Offsets are due to e084b/5f8dcc21 being reverted or not, it works in both cases. Let me know if they work out for you that way.

Quoted 15 lines
Quoted 8 lines

perf_count_failed_pages_direct_reclaim_but_progress 305 1478 8979 27.87%

GOOD CASE WITH REVERTS 4 THREAD READ 8 THREAD READ 16 THREAD READ 16THR % portions
perf_count_congestion_wait 25 76 1114
perf_count_call_congestion_wait_from_alloc_pages_high_priority 0 0 0
perf_count_call_congestion_wait_from_alloc_pages_slowpath 25 76 1114 99.98%
perf_count_pages_direct_reclaim 1054 9150 62297

Quoted 6 lines
perf_count_failed_pages_direct_reclaim                             25                          64                         1114
perf_count_failed_pages_direct_reclaim_but_progress                25                          57                         1114                1.79%

I hope the format is kept, it should be good with every monospace viewer.

Quoted 8 lines

It got manged but I think I've it fixed above. The biggest thing I can see is that direct reclaim is a lot more successful with the patches reverted but that in itself doesn't make sense. Neither patch affects how many pages should be free or reclaimable - just what order they are allocated and freed in.

With botgh patches reverted, is the performance 100% reliable or does it sometimes vary?

It is 100% percent reliable now - reliably bad with plain 2.6.32 as well as reliably much better (e.g. +40% @ 16threads) with both reverted.

Quoted 7 lines

If reverting 5f8dcc21 is required, I'm leaning towards believing that the additional memory overhead with that patch is enough to push this workload over the edge where entering direct reclaim is necessary a lot more to keep the zones over the min watermark. You mentioned early on that adding 64MB to the machine makes the problem go away. Do you know what the cut-off point is? For example, is adding 4MB enough?

That was another one probably only seen due to the lack of good reproducibility in my old setup.
I made bigger memory scaling tests with the new setup. Therefore I ran the workload with 160 to 356 megabytes in 32mb steps (256 is the default in all other runs). The result is that more than 256m memory only brings a slight benefit (which is ok as the load doesn't reread the data read into page cache and it just doesn't need much more).

Here some data about scaling memory normalized to the 256m memory values:
- deviation to 256m case - 160m 192m 224m 256m 288m 320m 356m

plain 2.6.32                              +9.12% -55.11% -17.15%   =100%  +1.46%  +1.46%  +1.46%
2.6.32 - 5f8dcc21 and e084b reverted      +6.95%  -6.95%  -0.80%   =100%  +2.14%  +2.67%  +2.67%

------------------------------------------------------------------------------------------------
deviation between each other (all +) 60.64% 182.93% 63.44% 36.50% 37.41% 38.13% 38.13%
What can be seen:
a) more memory brings up to +2.67%/+1.46%, but not more when further increasing memory (reasonable as we don't reread cached files)
b) decreasing memory drops performance by up to -6.95% @ -64mb with both reverted, but down to -55% @ -64mb (compared to the already much lower 256m throughput) -> plain 2.6.32 is much more sensible to lower available memory AND
always a level below
c) there is a weird but very reproducible improvement with even lower memory - very probably another issue or better another solution and not related here - but who knows :-) -> still both reverted is always much better than plain 2.6.32

Quoted 4 lines

If a small amount of memory does help, I'd also test with min_free_kbytes at a lower value? If reducing it restores the performance, it again implies that memory usage is the real problem.

I'll run a few tests with bigger/smaller numbers of min_free_kbytes just to be sure.

Quoted 7 lines

Thing is, even if adding small amounts of memory or reducing min_free_kbytes helps, it does not explain why e084b ever made a difference because all that patch does is alter the ordering of pages on the free list. That might have some cache consequences but it would not impact direct reclaim like this.

Quoted 7 lines
Quoted 2 lines

You can all clearly see that the patches somehow affect the ratio at which __alloc_pages_direct_reclaim runs into a race between try_to_free pages that could actually free something, but a few lines below can't get a page from the free list.

There actually is potentially a significant delay from when direct reclaim returns and a new allocation attempt happens. However, it's been like this for a long time so I don't think it's the issue.

Due to the increased percentage of progress, but !page in direct_reclaim I still think it is related.
I agree that "having" that potential race between try_to_free and get_page is not the issue, but I assume it is very likely that the patches change the timing there so that it happens much more often.

Quoted 5 lines

I have a prototype patch that removes usage of congestion_wait() altogether and puts processes to sleep on a waitqueue waiting for watermarks to restore or a timeout. However, I think it would be just treating the symptoms here rather than the real issue.

Anyway I'd be happy to test this patch if you could sent it to me, because as long as we don't have a "real" solution I like such symptom-fixes as a start :-) At least it would fix going into congestion_wait even without dirty pages or I/O's in flight - waiting for watermarks sounds much better to me independent to the current issue.

Quoted 4 lines
Quoted 4 lines

Outside in the function alloc_pages_slowpath this leads to a call to congestion_wait which is absolutely futile as there are absolutely no writes in flight to wait for.

Now this kills effectively up to 80% of our throughput - Any idea of better understanding the link between the patches and the effect is welcome and might lead to a solution.

Quoted 9 lines
Quoted 3 lines

FYI - I tried all patches you suggested - none of them affects this.

I'm still at a total loss to explain this. Memory overhead of the second patch is a vague possibility and worth checking out by slightly increasing available memory on the partition or reducing min_free_kbytes. It does not explain why the first patch makes a difference.

In a discussion with Hannes Reinecke ([email protected]) he brought up that in a low memory scenario the ordering of the pages might twist.
Today we have the single linst of pages - add hot ones to the head and cold to the tail. Both patches affect that list management - e084b changes the order of some, and 5f8dcc is splitting it per migrate type. What now could happen is that you free pages and get a list like:
H1 - H2 - H3 - C3 - C2 - C1 (H is hot and C is cold)
In low mem scenarios it could now happen that a allocation for hot pages falls back to cold pages (as usual fallback), but couldnt it be that it then also gets the cold pages in reverse order again ? Without e084b it would be like:
H1 - H2 - H3 - C1 - C2 - C3 and therefore all at least in order (with the issue that cold allocations from the right are reverse -> e084b)
5f8dcc is now lowering the size of those lists by splitting it into different migration types, so maybe both together are increasing the chance to get such an issue.

I hope I explained that idea right, Hannes please feel free to correct and extend if needed. I also added Nick on his request.

Quoted 4 lines
Quoted 2 lines

<SNIP>

Signature · 2 lines
Grüsse / regards, Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization 
#40 lore raw
In reply to Mel Gorman

*resend with fixed line wrap in the tables*

Quoted 17 lines · Mel Gorman
Quoted 5 lines · Christian Ehrhardt

I'll keep the old thread below as reference.

After taking a round of ensuring reproducibility and a pile of new measurements I now can come back with several new insights.

Quoted 9 lines

FYI - I'm now running iozone triplets (4, then 8, then 16 parallel threads) with sequential read load and all that 4 times to find potential noise. But since I changed to that load instead of random read wit hone thread and ensuring the most memory is cleared (sync + echo 3 > /proc/sys/vm/drop_caches + a few sleeps) . The noise is now down <2%. For detailed questions about the setup feel free to ask me directly as I won't flood this thread too much with such details.

Quoted 5 lines

Is there any chance you have a driver script for the test that you could send me? I'll then try reproducing based on that script and see what happens. I'm not optimistic I'll be able to reproduce the problem because I think it's specific to your setup but you never know.

I don't have one as it runs in a bigger automated test environment, but it is easy enough to write down something comparable.

Quoted 6 lines
Quoted 5 lines

So in the past I identified git id e084b for bringing a huge degradation, that is still true, but I need to revert my old statement that unapplying e084b on v2.6.32 helps - it simply doesn't.

Quoted 6 lines

Hmm, ok. Odd that it used to work but now doesn't.

How reproducible are these results with patch e084b reverted? i.e. I know it does not work now, but did reverting on the old kernel always fix it or were there occasions where the figures would be still bad?

Reverting e084b in the past showed something like +40%, so I thought it helps.
Afterwards I found out it wasn't a good testcase for reproducibility and when looking at a series it had +5%,-7%,+20%,...
So by far too noisy to be useful for bisect, discussion or fix testing. Thats why I made my big round reinventing the testcases in a more reproducible way like described above.
In those the original issue is still very visible - which means 4 runs comparing 2.6.32 vs 2.6.32-Reve084b being each max +/-1%.
While gitid-e084b vs. the one just before (51fbb) gives ~-60% all the time.

Quoted 17 lines
Quoted 10 lines

Another bisect (keepign e084b reverted) brought up git id 5f8dcc21 which came in later. Both patches unapplied individually don't improve anything. But both patches reverted at the same time on git v2.6.32 bring us back to our old values (remember that is more than 2Gb/s improvement in throughput)!

Unfortunately 5f8dcc21 is as unobvious as e84b in explaining how this can cause so much trouble.

There is a bug in commit 5f8dcc21. One consequence of it is that swap-based workloads can suffer. A second is that users of high-order allocations can enter direct reclaim a lot more than previously. This was fixed in commit a7016235a61d520e6806f38129001d935c4b6661 but you say that's not the fix in your case.

Quoted 4 lines

The only other interesting thing in commit 5f8dcc21 is that it increases the memory overhead of a per-cpu structure. If your memory usage is really borderline, it might have been just enough to push it over the edge.

I had this thoughts too, but as it only shows an effect with 5f8dcc21 and e084b reverted at the same time I'm wondering if it can be that. But I agree that this should be considered too until a verification can be done.

Quoted 15 lines
Quoted 6 lines

In the past I identified congestion_wait as the point where the "time is lost" when comparing good and bad case. Fortunately at least this is still true when comparing 2.6.32 vs 2.6.32 with both patches reverted. So I upgraded my old counter patches a bit and can now report the following:

Quoted 8 lines

BAD CASE
4 THREAD READ 8 THREAD READ 16 THREAD READ 16THR % portions
perf_count_congestion_wait 305 1970 8980
perf_count_call_congestion_wait_from_alloc_pages_high_priority 0 0 0

perf_count_call_congestion_wait_from_alloc_pages_slowpath         305                        1970                        8979                   100.00%
perf_count_pages_direct_reclaim                                  1153                        6818                        3221
perf_count_failed_pages_direct_reclaim                            305                        1556                        8979
Quoted 5 lines

Something is wrong with the counters there. For 16 threads, it says direct reclaim was entered 3221 but it failed 8979 times. How does that work? The patch you supplied below looks like a patch on top of another debugging patch. Can you send the full debugging patch please?

This is just a single 7 deleted accidentally when bringing it to a plain text format for this mail. The percentage is fine and properly it would look like:

4 THREAD READ 8 THREAD READ 16 THREAD READ 16THR % portions

perf_count_congestion_wait 305 1970 8980
perf_count_call_congestion_wait_from_alloc_pages_high_priority 0 0 0

perf_count_call_congestion_wait_from_alloc_pages_slowpath         305                        1970                         8979               100.00%
perf_count_pages_direct_reclaim                                  1153                        6818                        32217
perf_count_failed_pages_direct_reclaim                            305                        1556                         8979
perf_count_failed_pages_direct_reclaim_but_progress               305                        1478                         8979               27.87%

There were three patches attached in the last mail which should work in this order:

~/kernels/linux-2.6$ git checkout -f v2.6.32 ~/kernels/linux-2.6$ patch -p1 < ~/Desktop/patches-git/perf-count-congestion-wait.diff patching file include/linux/sysctl.h
patching file kernel/sysctl.c
patching file mm/backing-dev.c ~/kernels/linux-2.6$ patch -p1 < ~/Desktop/patches-git/perf-count-failed-direct-recalims.diff patching file kernel/sysctl.c
patching file mm/page_alloc.c
Hunk #1 succeeded at 1670 (offset 9 lines).
Hunk #2 succeeded at 1709 (offset 9 lines). ~/kernels/linux-2.6$ patch -p1 < ~/Desktop/patches-git/perf-count-congestion-wait-calls.diff patching file kernel/sysctl.c
patching file mm/page_alloc.c
Hunk #1 succeeded at 1672 (offset 9 lines).
Hunk #2 succeeded at 1714 (offset 9 lines).
Hunk #3 succeeded at 1738 (offset 9 lines).
Hunk #4 succeeded at 1796 (offset 9 lines).
Hunk #5 succeeded at 1913 (offset 9 lines).

Offsets are due to e084b/5f8dcc21 being reverted or not, it works in both cases.
Let me know if they work out for you that way.

Quoted 15 lines
Quoted 8 lines

perf_count_failed_pages_direct_reclaim_but_progress 305 1478 8979 27.87%

GOOD CASE WITH REVERTS 4 THREAD READ 8 THREAD READ 16 THREAD READ 16THR % portions
perf_count_congestion_wait 25 76 1114
perf_count_call_congestion_wait_from_alloc_pages_high_priority 0 0 0
perf_count_call_congestion_wait_from_alloc_pages_slowpath 25 76 1114 99.98%
perf_count_pages_direct_reclaim 1054 9150 62297

Quoted 6 lines
perf_count_failed_pages_direct_reclaim                             25                          64                         1114
perf_count_failed_pages_direct_reclaim_but_progress                25                          57                         1114                1.79%

I hope the format is kept, it should be good with every monospace viewer.

Quoted 8 lines

It got manged but I think I've it fixed above. The biggest thing I can see is that direct reclaim is a lot more successful with the patches reverted but that in itself doesn't make sense. Neither patch affects how many pages should be free or reclaimable - just what order they are allocated and freed in.

With botgh patches reverted, is the performance 100% reliable or does it sometimes vary?

It is 100% percent reliable now - reliably bad with plain 2.6.32 as well as reliably much better (e.g. +40% @ 16threads) with both reverted.

Quoted 7 lines

If reverting 5f8dcc21 is required, I'm leaning towards believing that the additional memory overhead with that patch is enough to push this workload over the edge where entering direct reclaim is necessary a lot more to keep the zones over the min watermark. You mentioned early on that adding 64MB to the machine makes the problem go away. Do you know what the cut-off point is? For example, is adding 4MB enough?

That was another one probably only seen due to the lack of good reproducibility in my old setup.
I made bigger memory scaling tests with the new setup. Therefore I ran the workload with 160 to 356 megabytes in 32mb steps (256 is the default in all other runs).
The result is that more than 256m memory only brings a slight benefit (which is ok as the load doesn't reread the data read into page cache and it just doesn't need much more).

Here some data about scaling memory normalized to the 256m memory values:
- deviation to 256m case - 160m 192m 224m 256m 288m 320m 356m

plain 2.6.32                              +9.12% -55.11% -17.15%   =100%  +1.46%  +1.46%  +1.46%
2.6.32 - 5f8dcc21 and e084b reverted      +6.95%  -6.95%  -0.80%   =100%  +2.14%  +2.67%  +2.67%

------------------------------------------------------------------------------------------------
deviation between each other (all +) 60.64% 182.93% 63.44% 36.50% 37.41% 38.13% 38.13%
What can be seen:
a) more memory brings up to +2.67%/+1.46%, but not more when further increasing memory (reasonable as we don't reread cached files)
b) decreasing memory drops performance by up to -6.95% @ -64mb with both reverted, but down to -55% @ -64mb (compared to the already much lower 256m throughput)
-> plain 2.6.32 is much more sensible to lower available memory AND
always a level below
c) there is a weird but very reproducible improvement with even lower memory - very probably another issue or better another solution and not related here - but who knows :-)
-> still both reverted is always much better than plain 2.6.32

Quoted 4 lines

If a small amount of memory does help, I'd also test with min_free_kbytes at a lower value? If reducing it restores the performance, it again implies that memory usage is the real problem.

I'll run a few tests with bigger/smaller numbers of min_free_kbytes just to be sure.

Quoted 13 lines

Thing is, even if adding small amounts of memory or reducing min_free_kbytes helps, it does not explain why e084b ever made a difference because all that patch does is alter the ordering of pages on the free list. That might have some cache consequences but it would not impact direct reclaim like this.

Quoted 5 lines

You can all clearly see that the patches somehow affect the ratio at which __alloc_pages_direct_reclaim runs into a race between try_to_free pages that could actually free something, but a few lines below can't get a page from the free list.

Quoted 4 lines

There actually is potentially a significant delay from when direct reclaim returns and a new allocation attempt happens. However, it's been like this for a long time so I don't think it's the issue.

Due to the increased percentage of progress, but !page in direct_reclaim I still think it is related.
I agree that "having" that potential race between try_to_free and get_page is not the issue, but I assume it is very likely that the patches change the timing there so that it happens much more often.

Quoted 5 lines

I have a prototype patch that removes usage of congestion_wait() altogether and puts processes to sleep on a waitqueue waiting for watermarks to restore or a timeout. However, I think it would be just treating the symptoms here rather than the real issue.

Anyway I'd be happy to test this patch if you could sent it to me, because as long as we don't have a "real" solution I like such symptom-fixes as a start :-)
At least it would fix going into congestion_wait even without dirty pages or I/O's in flight - waiting for watermarks sounds much better to me independent to the current issue.

Quoted 8 lines
Quoted 8 lines

Outside in the function alloc_pages_slowpath this leads to a call to congestion_wait which is absolutely futile as there are absolutely no writes in flight to wait for.

Now this kills effectively up to 80% of our throughput - Any idea of better understanding the link between the patches and the effect is welcome and might lead to a solution.

Quoted 9 lines
Quoted 3 lines

FYI - I tried all patches you suggested - none of them affects this.

I'm still at a total loss to explain this. Memory overhead of the second patch is a vague possibility and worth checking out by slightly increasing available memory on the partition or reducing min_free_kbytes. It does not explain why the first patch makes a difference.

In a discussion with Hannes Reinecke ([email protected]) he brought up that in a low memory scenario the ordering of the pages might twist. Today we have the single linst of pages - add hot ones to the head and cold to the tail. Both patches affect that list management - e084b changes the order of some, and 5f8dcc is splitting it per migrate type. What now could happen is that you free pages and get a list like:
H1 - H2 - H3 - C3 - C2 - C1 (H is hot and C is cold)
In low mem scenarios it could now happen that a allocation for hot pages falls back to cold pages (as usual fallback), but couldnt it be that it then also gets the cold pages in reverse order again ? Without e084b it would be like:
H1 - H2 - H3 - C1 - C2 - C3 and therefore all at least in order (with the issue that cold allocations from the right are reverse -> e084b) 5f8dcc is now lowering the size of those lists by splitting it into different migration types, so maybe both together are increasing the chance to get such an issue.

I hope I explained that idea right, Hannes please feel free to correct and extend if needed.
I also added Nick on his request.

Quoted 4 lines
Quoted 2 lines

<SNIP>

Signature · 2 lines
Grüsse / regards, Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization 
#13 lore raw
In reply to Christian Ehrhardt
Quoted 20 lines · Christian Ehrhardt
Quoted 17 lines · Mel Gorman
Quoted 14 lines · Christian Ehrhardt

I'll keep the old thread below as reference.

After taking a round of ensuring reproducibility and a pile of new measurements I now can come back with several new insights.

FYI - I'm now running iozone triplets (4, then 8, then 16 parallel threads) with sequential read load and all that 4 times to find potential noise. But since I changed to that load instead of random read wit hone thread and ensuring the most memory is cleared (sync + echo 3 > /proc/sys/vm/drop_caches + a few sleeps) . The noise is now down <2%. For detailed questions about the setup feel free to ask me directly as I won't flood this thread too much with such details.

Quoted 8 lines
Quoted 5 lines

Is there any chance you have a driver script for the test that you could send me? I'll then try reproducing based on that script and see what happens. I'm not optimistic I'll be able to reproduce the problem because I think it's specific to your setup but you never know.

I don't have one as it runs in a bigger automated test environment, but it is easy enough to write down something comparable.

I'd appreciate it, thanks.

Quoted 14 lines
Quoted 7 lines
Quoted 6 lines

So in the past I identified git id e084b for bringing a huge degradation, that is still true, but I need to revert my old statement that unapplying e084b on v2.6.32 helps - it simply doesn't.

Quoted 6 lines

Hmm, ok. Odd that it used to work but now doesn't.

How reproducible are these results with patch e084b reverted? i.e. I know it does not work now, but did reverting on the old kernel always fix it or were there occasions where the figures would be still bad?

Quoted 10 lines

Reverting e084b in the past showed something like +40%, so I thought it helps.
Afterwards I found out it wasn't a good testcase for reproducibility and when looking at a series it had +5%,-7%,+20%,...
So by far too noisy to be useful for bisect, discussion or fix testing. Thats why I made my big round reinventing the testcases in a more reproducible way like described above.
In those the original issue is still very visible - which means 4 runs comparing 2.6.32 vs 2.6.32-Reve084b being each max +/-1%.
While gitid-e084b vs. the one just before (51fbb) gives ~-60% all the time.

About all e084b can be affecting is cache hotness when fallback is occuring but 60% regression still seems way too high for just that.

Quoted 21 lines
Quoted 16 lines
Quoted 9 lines

Another bisect (keepign e084b reverted) brought up git id 5f8dcc21 which came in later. Both patches unapplied individually don't improve anything. But both patches reverted at the same time on git v2.6.32 bring us back to our old values (remember that is more than 2Gb/s improvement in throughput)!

Unfortunately 5f8dcc21 is as unobvious as e84b in explaining how this
can cause so much trouble.

There is a bug in commit 5f8dcc21. One consequence of it is that swap-based workloads can suffer. A second is that users of high-order allocations can enter direct reclaim a lot more than previously. This was fixed in commit a7016235a61d520e6806f38129001d935c4b6661 but you say that's not the fix in your case.

Quoted 4 lines

The only other interesting thing in commit 5f8dcc21 is that it increases the memory overhead of a per-cpu structure. If your memory usage is really borderline, it might have been just enough to push it over the edge.

Quoted 5 lines

I had this thoughts too, but as it only shows an effect with 5f8dcc21 and e084b reverted at the same time I'm wondering if it can be that. But I agree that this should be considered too until a verification can be done.

Quoted 1 line

Another consequence of 5f8dcc is that pages are on the per-cpu lists that it is possible for pages to be on the per-cpu lists when the page allocator is called. There is potential that page reclaim is being entered as a result even though pages were free on the per-cpu lists.

It would not explain why reverting e084b makes such a difference but I can prototype a patch that falls back to other per-cpu lists before calling the page allocator as it used to do but without linear searching the per-cpu lists.

Quoted 29 lines
Quoted 15 lines
Quoted 6 lines

In the past I identified congestion_wait as the point where the "time is lost" when comparing good and bad case. Fortunately at least this is still true when comparing 2.6.32 vs 2.6.32 with both patches reverted.
So I upgraded my old counter patches a bit and can now report the following:

Quoted 8 lines

BAD CASE
4 THREAD READ 8 THREAD READ 16 THREAD READ 16THR % portions
perf_count_congestion_wait 305 1970 8980
perf_count_call_congestion_wait_from_alloc_pages_high_priority 0 0 0

perf_count_call_congestion_wait_from_alloc_pages_slowpath         305                        1970                        8979                   100.00%
perf_count_pages_direct_reclaim                                  1153                        6818                        3221
perf_count_failed_pages_direct_reclaim                            305                        1556                        8979
Quoted 5 lines

Something is wrong with the counters there. For 16 threads, it says direct reclaim was entered 3221 but it failed 8979 times. How does that work? The patch you supplied below looks like a patch on top of another debugging patch. Can you send the full debugging patch please?

This is just a single 7 deleted accidentally when bringing it to a plain text format for this mail. The percentage is fine and properly it would look like:

4 THREAD READ 8 THREAD READ 16 THREAD READ 16THR % portions

perf_count_congestion_wait 305 1970 8980
perf_count_call_congestion_wait_from_alloc_pages_high_priority 0 0 0

Quoted 9 lines
perf_count_call_congestion_wait_from_alloc_pages_slowpath         305                               1970                         8979               100.00%
perf_count_pages_direct_reclaim                                  1153                               6818                        32217
perf_count_failed_pages_direct_reclaim                            305                               1556                         8979
perf_count_failed_pages_direct_reclaim_but_progress               305                               1478                         8979               27.87%

There were three patches attached in the last mail which should work in this order:

Bah. I was reading just an automatically-inlined version and didn't notice the other two attachments. Sorry.

Quoted 45 lines

~/kernels/linux-2.6$ git checkout -f v2.6.32 ~/kernels/linux-2.6$ patch -p1 < ~/Desktop/patches-git/perf-count-congestion-wait.diff patching file include/linux/sysctl.h
patching file kernel/sysctl.c
patching file mm/backing-dev.c ~/kernels/linux-2.6$ patch -p1 < ~/Desktop/patches-git/perf-count-failed-direct-recalims.diff patching file kernel/sysctl.c
patching file mm/page_alloc.c
Hunk #1 succeeded at 1670 (offset 9 lines).
Hunk #2 succeeded at 1709 (offset 9 lines). ~/kernels/linux-2.6$ patch -p1 < ~/Desktop/patches-git/perf-count-congestion-wait-calls.diff patching file kernel/sysctl.c
patching file mm/page_alloc.c
Hunk #1 succeeded at 1672 (offset 9 lines).
Hunk #2 succeeded at 1714 (offset 9 lines).
Hunk #3 succeeded at 1738 (offset 9 lines).
Hunk #4 succeeded at 1796 (offset 9 lines).
Hunk #5 succeeded at 1913 (offset 9 lines).

Offsets are due to e084b/5f8dcc21 being reverted or not, it works in both cases.
Let me know if they work out for you that way.

Quoted 20 lines
Quoted 8 lines

perf_count_failed_pages_direct_reclaim_but_progress 305 1478 8979 27.87%

GOOD CASE WITH REVERTS 4 THREAD READ 8 THREAD READ 16 THREAD READ 16THR % portions
perf_count_congestion_wait 25 76 1114
perf_count_call_congestion_wait_from_alloc_pages_high_priority 0 0 0
perf_count_call_congestion_wait_from_alloc_pages_slowpath 25 76 1114 99.98%
perf_count_pages_direct_reclaim 1054 9150 62297

Quoted 6 lines
perf_count_failed_pages_direct_reclaim                             25                          64                         1114
perf_count_failed_pages_direct_reclaim_but_progress                25                          57                         1114                1.79%

I hope the format is kept, it should be good with every monospace viewer.

It got manged but I think I've it fixed above. The biggest thing I can see is that direct reclaim is a lot more successful with the patches reverted but that in itself doesn't make sense. Neither patch affects how many pages should be free or reclaimable - just what order they are allocated and freed in.

Quoted 7 lines
Quoted 3 lines

With botgh patches reverted, is the performance 100% reliable or does it sometimes vary?

It is 100% percent reliable now - reliably bad with plain 2.6.32 as well as reliably much better (e.g. +40% @ 16threads) with both reverted.

Ok.

Quoted 9 lines
Quoted 7 lines

If reverting 5f8dcc21 is required, I'm leaning towards believing that the additional memory overhead with that patch is enough to push this workload over the edge where entering direct reclaim is necessary a lot more to keep the zones over the min watermark. You mentioned early on that adding 64MB to the machine makes the problem go away. Do you know what the cut-off point is? For example, is adding 4MB enough?

That was another one probably only seen due to the lack of good reproducibility in my old setup.

Ok.

Quoted 11 lines

I made bigger memory scaling tests with the new setup. Therefore I ran the workload with 160 to 356 megabytes in 32mb steps (256 is the default in all other runs).
The result is that more than 256m memory only brings a slight benefit (which is ok as the load doesn't reread the data read into page cache and it just doesn't need much more).

Here some data about scaling memory normalized to the 256m memory values:
- deviation to 256m case - 160m 192m 224m 256m 288m 320m 356m

plain 2.6.32                              +9.12% -55.11% -17.15%   =100%   +1.46%  +1.46%  +1.46%
2.6.32 - 5f8dcc21 and e084b reverted      +6.95%  -6.95%  -0.80%   =100%   +2.14%  +2.67%  +2.67%
Quoted 15 lines

------------------------------------------------------------------------------------------------
deviation between each other (all +) 60.64% 182.93% 63.44% 36.50% 37.41% 38.13% 38.13%
What can be seen:
a) more memory brings up to +2.67%/+1.46%, but not more when further increasing memory (reasonable as we don't reread cached files)
b) decreasing memory drops performance by up to -6.95% @ -64mb with both reverted, but down to -55% @ -64mb (compared to the already much lower 256m throughput)
-> plain 2.6.32 is much more sensible to lower available memory AND
always a level below
c) there is a weird but very reproducible improvement with even lower memory - very probably another issue or better another solution and not related here - but who knows :-)
-> still both reverted is always much better than plain 2.6.32

Ok. Two avenues of attack then although both are focused on 5f8dcc21. The first is the prototype below that should avoid congestion_wait. The second is to reintroduce a fallback to other per-cpu lists and see if that was a big factor.

Quoted 5 lines
Quoted 4 lines

If a small amount of memory does help, I'd also test with min_free_kbytes at a lower value? If reducing it restores the performance, it again implies that memory usage is the real problem.

Quoted 19 lines

I'll run a few tests with bigger/smaller numbers of min_free_kbytes just to be sure.

Quoted 17 lines

Thing is, even if adding small amounts of memory or reducing min_free_kbytes helps, it does not explain why e084b ever made a difference because all that patch does is alter the ordering of pages on the free list. That might have some cache consequences but it would not impact direct reclaim like this.

Quoted 5 lines

You can all clearly see that the patches somehow affect the ratio at which __alloc_pages_direct_reclaim runs into a race between try_to_free pages that could actually free something, but a few lines below can't get a page from the free list.

There actually is potentially a significant delay from when direct reclaim returns and a new allocation attempt happens. However, it's been like this for a long time so I don't think it's the issue.

<

Quoted 2 lines

Due to the increased percentage of progress, but !page in direct_reclaim I still think it is related.

It probably yes, but unfortunately the e084b has very little to do with that logic although 5f8dcc21 might indirectly be affecting it.

Quoted 4 lines

I agree that "having" that potential race between try_to_free and get_page is not the issue, but I assume it is very likely that the patches change the timing there so that it happens much more often.

Patch e084b would make very little difference to the timing of events though except from a cache perspective but the slowdown is too severe for that.

Quoted 6 lines
Quoted 5 lines

I have a prototype patch that removes usage of congestion_wait() altogether and puts processes to sleep on a waitqueue waiting for watermarks to restore or a timeout. However, I think it would be just treating the symptoms here rather than the real issue.

Quoted 6 lines

Anyway I'd be happy to test this patch if you could sent it to me, because as long as we don't have a "real" solution I like such symptom-fixes as a start :-)
At least it would fix going into congestion_wait even without dirty pages or I/O's in flight - waiting for watermarks sounds much better to me independent to the current issue.

Hmm, the patch as it currently stands is below. However, I warn you that it has only been boot-tested on qemu with no proper testing doing, either functional or performance testing. It may just go mad, drink your beer and chew on your dog.

Quoted 17 lines
Quoted 8 lines
Quoted 8 lines

Outside in the function alloc_pages_slowpath this leads to a call to congestion_wait which is absolutely futile as there are absolutely no
writes in flight to wait for.

Now this kills effectively up to 80% of our throughput - Any idea of better understanding the link between the patches and the effect is welcome and might lead to a solution.

Quoted 9 lines
Quoted 3 lines

FYI - I tried all patches you suggested - none of them affects this.

I'm still at a total loss to explain this. Memory overhead of the second patch is a vague possibility and worth checking out by slightly increasing available memory on the partition or reducing min_free_kbytes. It does not explain why the first patch makes a difference.

Quoted 5 lines

In a discussion with Hannes Reinecke ([email protected]) he brought up that in a low memory scenario the ordering of the pages might twist. Today we have the single linst of pages - add hot ones to the head and cold to the tail. Both patches affect that list management - e084b changes the order of some, and 5f8dcc is splitting it per migrate type.

Correct on both counts.

Quoted 5 lines

What now could happen is that you free pages and get a list like:
H1 - H2 - H3 - C3 - C2 - C1 (H is hot and C is cold)
In low mem scenarios it could now happen that a allocation for hot pages falls back to cold pages (as usual fallback), but couldnt it be that it then also gets the cold pages in reverse order again ?

Yes, this is true but that would only matter from a cache perspective as you say your hardware and drivers are doing no automatic merging of requests.

Quoted 7 lines

Without e084b it would be like:
H1 - H2 - H3 - C1 - C2 - C3 and therefore all at least in order (with the issue that cold allocations from the right are reverse -> e084b) 5f8dcc is now lowering the size of those lists by splitting it into different migration types, so maybe both together are increasing the chance to get such an issue.

There is a good chance that 5f8dcc is causing direct reclaim to be entered when it could have been avoided as pages were on the per-cpu lists but I don't think the difference in cache hotness is enough to explain a 60% loss on your machine.

Quoted 3 lines

I hope I explained that idea right, Hannes please feel free to correct and extend if needed.
I also added Nick on his request.

The prototype patch for avoiding congestion_wait is below. I'll start work on a fallback-to-other-percpu-lists patch.

include/linux/mmzone.h +3 -0
index 30fe668..72465c1 100644
@@ -398,6 +398,9 @@ struct zone {
 	unsigned long		wait_table_hash_nr_entries;
 	unsigned long		wait_table_bits;
 
+	/* queue for processes waiting for pressure to relieve */
+	wait_queue_head_t	*pressure_wq;
+
 	/*
 	 * Discontig memory support fields.
 	 */
mm/internal.h +3 -0
index 6a697bb..d447d57 100644
@@ -251,6 +251,9 @@ int __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
 #define ZONE_RECLAIM_SUCCESS	1
 #endif
 
+extern void check_zone_pressure(struct zone *zone);
+extern long zonepressure_wait(struct zone *zone, long timeout);
+
 extern int hwpoison_filter(struct page *p);
 
 extern u32 hwpoison_filter_dev_major;
mm/mmzone.c +40 -0
index f5b7d17..c43d068 100644
@@ -9,6 +9,7 @@
 #include <linux/mm.h>
 #include <linux/mmzone.h>
 #include <linux/module.h>
+#include <linux/sched.h>
 
 struct pglist_data *first_online_pgdat(void)
 {
@@ -87,3 +88,42 @@ int memmap_valid_within(unsigned long pfn,
 	return 1;
 }
 #endif /* CONFIG_ARCH_HAS_HOLES_MEMORYMODEL */
+
+void check_zone_pressure(struct zone *zone)
+{
+	/* If no process is waiting, nothing to do */
+	if (!waitqueue_active(zone->pressure_wq))
+		return;
+
+	/* Check if the high watermark is ok for order 0 */
+	if (zone_watermark_ok(zone, 0, low_wmark_pages(zone), 0, 0))
+		wake_up_interruptible(zone->pressure_wq);
+}
+
+/**
+ * zonepressure_wait - Wait for pressure on a zone to ease off
+ * @zone: The zone that is expected to be under pressure
+ * @timeout: Wait until pressure is relieved or this timeout is reached
+ *
+ * Waits for up to @timeout jiffies for pressure on a zone to be relieved.
+ * It's considered to be relieved if any direct reclaimer or kswapd brings
+ * the zone above the high watermark
+ */
+long zonepressure_wait(struct zone *zone, long timeout)
+{
+	long ret;
+	DEFINE_WAIT(wait);
+
+	prepare_to_wait(zone->pressure_wq, &wait, TASK_INTERRUPTIBLE);
+
+	/*
+	 * XXX: Is io_schedule_timeout() really appropriate here? Maybe not
+	 * because it'll get accounted for as IO waiting where that is not
+	 * really happening any more. Revisit what sort of wait this should
+	 * be
+	 */
+	ret = io_schedule_timeout(timeout);
+	finish_wait(zone->pressure_wq, &wait);
+
+	return ret;
+}
mm/page_alloc.c +45 -5

76 lines not on this page · raw message

mm/vmscan.c +11 -5
index c26986c..7a9aaae 100644
@@ -1709,6 +1709,7 @@ static void shrink_zones(int priority, struct zonelist *zonelist,
 		}
 
 		shrink_zone(priority, zone, sc);
+		check_zone_pressure(zone);
 	}
 }
 
@@ -1954,7 +1955,7 @@ static int sleeping_prematurely(pg_data_t *pgdat, int order, long remaining)
  * interoperates with the page allocator fallback scheme to ensure that aging
  * of pages is balanced across the zones.
  */
-static unsigned long balance_pgdat(pg_data_t *pgdat, int order)
+static unsigned long balance_pgdat(pg_data_t *pgdat, wait_queue_t *wait, int order)
 {
 	int all_zones_ok;
 	int priority;
@@ -2080,8 +2081,10 @@ loop_again:
 			 * zone has way too many pages free already.
 			 */
 			if (!zone_watermark_ok(zone, order,
-					8*high_wmark_pages(zone), end_zone, 0))
+					8*high_wmark_pages(zone), end_zone, 0)) {
 				shrink_zone(priority, zone, &sc);
+			}
+			check_zone_pressure(zone);
 			reclaim_state->reclaimed_slab = 0;
 			nr_slab = shrink_slab(sc.nr_scanned, GFP_KERNEL,
 						lru_pages);
@@ -2120,8 +2123,11 @@ loop_again:
 		if (total_scanned && (priority < DEF_PRIORITY - 2)) {
 			if (has_under_min_watermark_zone)
 				count_vm_event(KSWAPD_SKIP_CONGESTION_WAIT);
-			else
-				congestion_wait(BLK_RW_ASYNC, HZ/10);
+			else {
+				prepare_to_wait(&pgdat->kswapd_wait, wait, TASK_INTERRUPTIBLE);
+				schedule_timeout(HZ/10);
+				finish_wait(&pgdat->kswapd_wait, wait);
+			}
 		}
 
 		/*
@@ -2270,7 +2276,7 @@ static int kswapd(void *p)
 		 * after returning from the refrigerator
 		 */
 		if (!ret)
-			balance_pgdat(pgdat, order);
+			balance_pgdat(pgdat, &wait, order);
 	}
 	return 0;
 }
#14 lore raw
Quoted 4 lines

<SNIP>
The prototype patch for avoiding congestion_wait is below. I'll start work on a fallback-to-other-percpu-lists patch.

And here is the prototype of the fallback-to-other-percpu-lists patch. I'm afraid I've only managed to test it on qemu. My three test machines are still occupied :(

==== CUT HERE ====
page allocator: Fallback to other per-cpu lists when the target list is empty and memory is low

When a per-cpu list of pages for a given migratetype is empty, the page allocator is called to refill the PCP list. It's possible when memory is low that this results in the process entering direct reclaim even if it wasn't strictly necessary because there were pages free for other migratetypes. Unconditionally falling back to other PCP lists hurts the fragmentation-avoidance strategy which is also undesirable.

When the desired PCP list is empty, this patch checks how many free pages there are on the PCP lists and if refilling the list could result in direct reclaim. If direct reclaim is unlikely, the PCP list is refilled to maintain fragmentation-avoidance. Otherwise, a page from an alternative PCP list is chosen to maintain performance and avoid direct reclaim.

---
 mm/page_alloc.c |   37 ++++++++++++++++++++++++++++++++++---
 1 files changed, 34 insertions(+), 3 deletions(-)
mm/page_alloc.c +34 -3
index 8deb9d0..009d683 100644
@@ -1168,6 +1168,39 @@ void split_page(struct page *page, unsigned int order)
 		set_page_refcounted(page + i);
 }
 
+/* Decide whether to find an alternative PCP list or refill */
+static struct list_head *pcp_fallback(struct zone *zone,
+			struct per_cpu_pages *pcp, 
+			int start_migratetype, int cold)
+{
+	int i;
+	int migratetype;
+	struct list_head *list;
+	long free_pages = zone_page_state(zone, NR_FREE_PAGES) - pcp->batch;
+
+	/* 
+	 * Find a PCPU list with free pages in the same order as
+	 * fragmentation-avoidance fallback in the event that refilling
+	 * the PCP list may result in direct reclaim
+	 */
+	if (pcp->count && free_pages <= low_wmark_pages(zone)) {
+		for (i = 0; i < MIGRATE_PCPTYPES - 1; i++) {
+			migratetype = fallbacks[start_migratetype][i];
+			list = &pcp->lists[migratetype];
+
+			if (!list_empty(list))
+				return list;
+		}
+	}
+
+	/* Alternatively, we need to allocate more memory to the PCP lists */
+	list = &pcp->lists[start_migratetype];
+	pcp->count += rmqueue_bulk(zone, 0, pcp->batch, list,
+					migratetype, cold);
+
+	return list;
+}
+
 /*
  * Really, prep_compound_page() should be called from __rmqueue_bulk().  But
  * we cheat by calling it from here, in the order > 0 path.  Saves a branch
@@ -1193,9 +1226,7 @@ again:
 		list = &pcp->lists[migratetype];
 		local_irq_save(flags);
 		if (list_empty(list)) {
-			pcp->count += rmqueue_bulk(zone, 0,
-					pcp->batch, list,
-					migratetype, cold);
+			list = pcp_fallback(zone, pcp, migratetype, cold);
 			if (unlikely(list_empty(list)))
 				goto failed;
 		}
#15 lore raw
In reply to Mel Gorman
Quoted 24 lines · Mel Gorman
Quoted 23 lines · Christian Ehrhardt
Quoted 21 lines · Mel Gorman
Quoted 5 lines · Christian Ehrhardt

I'll keep the old thread below as reference.

After taking a round of ensuring reproducibility and a pile of new measurements I now can come back with several new insights.

Quoted 9 lines

FYI - I'm now running iozone triplets (4, then 8, then 16 parallel threads) with sequential read load and all that 4 times to find potential noise. But since I changed to that load instead of random read wit hone thread and ensuring the most memory is cleared (sync + echo 3 > /proc/sys/vm/drop_caches + a few sleeps) . The noise is now down <2%. For detailed questions about the setup feel free to ask me directly as I won't flood this thread too much with such details.

Is there any chance you have a driver script for the test that you could send me? I'll then try reproducing based on that script and see what happens. I'm not optimistic I'll be able to reproduce the problem because I think it's specific to your setup but you never know.

Quoted 5 lines
Quoted 2 lines

I don't have one as it runs in a bigger automated test environment, but it is easy enough to write down something comparable.

I'd appreciate it, thanks.

Testing of your two patches starts in a few minutes, thanks in advance.

Here the info how to execute the core of the test - I cross fingers that anyone else can reproduce it that way :-)

I use it in a huge automation framework which takes care of setting up the system, disks, gathering statistics and so on, but it essentially comes down to something simple like that:

#!/bin/bash

# reboot your system with 256m
# attach 16 disks (usually up to 64, but 16 should be enough to show the issue)
# mount your disks at /mnt/subw0, /mnt/subw1, ...
for i in 4 8 16 4 8 16 4 8 16 4 8 16
do
        sync; sleep 10s; echo 3 > /proc/sys/vm/drop_caches; sleep 2s;
        iozone -s 2000m -r 64k -t $i  -e -w -R -C  -i 0 -F /mnt/subw0 /mnt/subw1 /mnt/subw2 /mnt/subw3 /mnt/subw4 /mnt/subw5 /mnt/subw6 /mnt/subw7 /mnt/subw8 /mnt/subw9 /mnt/subw10 /mnt/subw11 /mnt/subw12 /mnt/subw13 /mnt/subw14 /mnt/subw15
        sync; sleep 10s; echo 3 > /proc/sys/vm/drop_caches; sleep 2s;
        iozone -s 2000m -r 64k -t $i  -e -w -R -C  -i 1 -F /mnt/subw0 /mnt/subw1 /mnt/subw2 /mnt/subw3 /mnt/subw4 /mnt/subw5 /mnt/subw6 /mnt/subw7 /mnt/subw8 /mnt/subw9 /mnt/subw10 /mnt/subw11 /mnt/subw12 /mnt/subw13 /mnt/subw14 /mnt/subw15
done
# while we could reduce the number of writes to one 16 thread write I use it that way as it is more similar to our original load (makes no difference anyway)

[...]

Signature · 2 lines
Grüsse / regards, Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization 
#16 lore raw
In reply to Mel Gorman
Quoted 5 lines · Mel Gorman
Quoted 4 lines · Christian Ehrhardt
Quoted 2 lines · Mel Gorman

On Fri, Feb 05, 2010 at 04:51:10PM +0100, Christian Ehrhardt wrote:

[...]

Quoted 15 lines
Quoted 4 lines
Quoted 4 lines

How reproducible are these results with patch e084b reverted? i.e. I know it does not work now, but did reverting on the old kernel always fix it or were there occasions where the figures would be still bad?

Quoted 10 lines

Reverting e084b in the past showed something like +40%, so I thought it helps.
Afterwards I found out it wasn't a good testcase for reproducibility and when looking at a series it had +5%,-7%,+20%,...
So by far too noisy to be useful for bisect, discussion or fix testing. Thats why I made my big round reinventing the testcases in a more reproducible way like described above.
In those the original issue is still very visible - which means 4 runs comparing 2.6.32 vs 2.6.32-Reve084b being each max +/-1%.
While gitid-e084b vs. the one just before (51fbb) gives ~-60% all the time.

Quoted 2 lines

About all e084b can be affecting is cache hotness when fallback is occuring but 60% regression still seems way too high for just that.

Yes, most the ideas you, I an a few others so far had went for cache hotness - but always the summary was "couldn't be that much".

On the other hand we have to consider that it could be a small timing due to cache or whatever else which leads into congestion_wait and then it waits HZ/50. That way a 1µs cause can lead to 20001µs lost which then cause 60%. Another important thing is that "cold page" afaik means the page content, and not the management struct etc. As the time lost is in management only (=> before accessing the actual 4k page content) it should not be important if a page is cold/hot for the mm code itself. That much for direct_reclaim that either runs into progress, but !page (BAD case) or not (GOOD case) - there only cache effects of struct page, pcp lists etc should have an effect.

Quoted 30 lines
Quoted 15 lines
Quoted 15 lines
Quoted 9 lines

Another bisect (keepign e084b reverted) brought up git id 5f8dcc21 which came in later. Both patches unapplied individually don't improve anything. But both patches reverted at the same time on git v2.6.32 bring us back to our old values (remember that is more than 2Gb/s improvement in throughput)!

Unfortunately 5f8dcc21 is as unobvious as e84b in explaining how this
can cause so much trouble.

There is a bug in commit 5f8dcc21. One consequence of it is that swap-based workloads can suffer. A second is that users of high-order allocations can enter direct reclaim a lot more than previously. This was fixed in commit a7016235a61d520e6806f38129001d935c4b6661 but you say that's not the fix in your case.

Quoted 9 lines
Quoted 4 lines

The only other interesting thing in commit 5f8dcc21 is that it increases the memory overhead of a per-cpu structure. If your memory usage is really borderline, it might have been just enough to push it over the edge.

I had this thoughts too, but as it only shows an effect with 5f8dcc21 and e084b reverted at the same time I'm wondering if it can be that. But I agree that this should be considered too until a verification can be done.

Quoted 1 line

Another consequence of 5f8dcc is that pages are on the per-cpu lists that it is possible for pages to be on the per-cpu lists when the page allocator is called. There is potential that page reclaim is being entered as a result even though pages were free on the per-cpu lists.

Quoted 4 lines

It would not explain why reverting e084b makes such a difference but I can prototype a patch that falls back to other per-cpu lists before calling the page allocator as it used to do but without linear searching the per-cpu lists.

I tested these patch you submitted in another reply.
Short - it only gives it ~+4% by probably speeding up things a bit but not touching the issue in any way. Some more detail below at the other test result.

[...]

Quoted 10 lines
Quoted 10 lines

4 THREAD READ 8 THREAD READ 16 THREAD READ 16THR % portions

perf_count_congestion_wait 305 1970 8980
perf_count_call_congestion_wait_from_alloc_pages_high_priority 0 0 0

perf_count_call_congestion_wait_from_alloc_pages_slowpath         305                               1970                         8979               100.00%
perf_count_pages_direct_reclaim                                  1153                               6818                        32217
perf_count_failed_pages_direct_reclaim                            305                               1556                         8979
perf_count_failed_pages_direct_reclaim_but_progress               305                               1478                         8979               27.87%

[...]

Quoted 67 lines
Quoted 19 lines
Quoted 19 lines
Quoted 8 lines

perf_count_failed_pages_direct_reclaim_but_progress 305 1478 8979 27.87%

GOOD CASE WITH REVERTS 4 THREAD READ 8 THREAD READ 16 THREAD READ 16THR % portions
perf_count_congestion_wait 25 76 1114
perf_count_call_congestion_wait_from_alloc_pages_high_priority 0 0 0
perf_count_call_congestion_wait_from_alloc_pages_slowpath 25 76 1114 99.98%
perf_count_pages_direct_reclaim 1054 9150 62297

Quoted 6 lines
perf_count_failed_pages_direct_reclaim                             25                          64                         1114
perf_count_failed_pages_direct_reclaim_but_progress                25                          57                         1114                1.79%

I hope the format is kept, it should be good with every monospace viewer.

It got manged but I think I've it fixed above. The biggest thing I can see is that direct reclaim is a lot more successful with the patches reverted but that in itself doesn't make sense. Neither patch affects how many pages should be free or reclaimable - just what order they are allocated and freed in.

Quoted 6 lines
Quoted 3 lines

With botgh patches reverted, is the performance 100% reliable or does it sometimes vary?

It is 100% percent reliable now - reliably bad with plain 2.6.32 as well as reliably much better (e.g. +40% @ 16threads) with both reverted.

Ok.

Quoted 9 lines
Quoted 7 lines

If reverting 5f8dcc21 is required, I'm leaning towards believing that the additional memory overhead with that patch is enough to push this workload over the edge where entering direct reclaim is necessary a lot more to keep the zones over the min watermark. You mentioned early on that adding 64MB to the machine makes the problem go away. Do you know what the cut-off point is? For example, is adding 4MB enough?

That was another one probably only seen due to the lack of good reproducibility in my old setup.

Ok.

Quoted 11 lines

I made bigger memory scaling tests with the new setup. Therefore I ran the workload with 160 to 356 megabytes in 32mb steps (256 is the default in all other runs).
The result is that more than 256m memory only brings a slight benefit (which is ok as the load doesn't reread the data read into page cache and it just doesn't need much more).

Here some data about scaling memory normalized to the 256m memory values:
- deviation to 256m case - 160m 192m 224m 256m 288m 320m 356m

plain 2.6.32                              +9.12% -55.11% -17.15%   =100%   +1.46%  +1.46%  +1.46%
2.6.32 - 5f8dcc21 and e084b reverted      +6.95%  -6.95%  -0.80%   =100%   +2.14%  +2.67%  +2.67%
Quoted 15 lines

------------------------------------------------------------------------------------------------
deviation between each other (all +) 60.64% 182.93% 63.44% 36.50% 37.41% 38.13% 38.13%
What can be seen:
a) more memory brings up to +2.67%/+1.46%, but not more when further increasing memory (reasonable as we don't reread cached files)
b) decreasing memory drops performance by up to -6.95% @ -64mb with both reverted, but down to -55% @ -64mb (compared to the already much lower 256m throughput)
-> plain 2.6.32 is much more sensible to lower available memory AND
always a level below
c) there is a weird but very reproducible improvement with even lower memory - very probably another issue or better another solution and not related here - but who knows :-)
-> still both reverted is always much better than plain 2.6.32

Quoted 4 lines

Ok. Two avenues of attack then although both are focused on 5f8dcc21. The first is the prototype below that should avoid congestion_wait. The second is to reintroduce a fallback to other per-cpu lists and see if that was a big factor.

I tested both of your patches, btw thanks for the quick work!

[...]

Quoted 5 lines

Hmm, the patch as it currently stands is below. However, I warn you that it has only been boot-tested on qemu with no proper testing doing, either functional or performance testing. It may just go mad, drink your beer and chew on your dog.

In opposition to all your warnings it worked pretty well.
The patch replacing congestion_wait calls - which we know re futile in my scenario anyway - by zone_wait calls helps a lot. But so far as you suggested only by alleviating the symptoms.

Quoted 1 line

From a throughput perspective alone it looks almost fixed:

                                                    4 thread      8 thread    16 thread
plain 2.6.32                                        100%          100%        100%
2.6.32 avoid direct reclaim                         103.85%       103.60%     104.04%
2.6.32 zone wait                                    125.46%       140.87%     131.89%
2.6.32 zone wait + 5f8dcc21 and e084b reverted      124.46%       143.35%     144.21%

Looking at the counters this theory (only fixing symptoms) seems to be confirmed. Before the ratio of perf_count_pages_direct_reclaim calls that ended in perf_count_failed_pages_direct_reclaim_but_progress was ~2% in the good cases and ~30% in bad cases. As congestion_wait in my case almost always waits the full HZ/50 this is a lot. With the zone wait patch I see a huge throughput win, most probably by all the (still ~30%!) waits that don't have to wait for the full timeout.
But still reverting 5f8dcc21 and e084b fix the issue of so much direct reclaims running in into that condition and that way giving throughput another last push.

% ran into try_to_free did progress but get_page got !page

plain 2.6.32                                          30.16%
2.6.32 zone wait                                      29.05%
2.6.32 zone wait + 5f8dcc21 and e084b reverted        2.93%

Note - with the zone wait patch there are also almost twice as much direct_reclaim calls. Probably by several waiters coming of the list when the watermark is restored and allocating too much.

Another good side note - 5f8dcc21 and e084b reverted plus your zone wait patch is better than anything I have seen so far :-)
Maybe its worth to measure both of your patches plus 5f8dcc21 and e084b reverted with 64 threads (the effect grows with # threads) just to rejoice in huge throughput values for a short time.

Quoted 42 lines
Quoted 16 lines
Quoted 8 lines
Quoted 8 lines

Outside in the function alloc_pages_slowpath this leads to a call to congestion_wait which is absolutely futile as there are absolutely no
writes in flight to wait for.

Now this kills effectively up to 80% of our throughput - Any idea of better understanding the link between the patches and the effect is welcome and might lead to a solution.

Quoted 8 lines
Quoted 3 lines

FYI - I tried all patches you suggested - none of them affects this.

I'm still at a total loss to explain this. Memory overhead of the second patch is a vague possibility and worth checking out by slightly increasing available memory on the partition or reducing min_free_kbytes. It does not explain why the first patch makes a difference.

Quoted 5 lines

In a discussion with Hannes Reinecke ([email protected]) he brought up that in a low memory scenario the ordering of the pages might twist. Today we have the single linst of pages - add hot ones to the head and cold to the tail. Both patches affect that list management - e084b changes the order of some, and 5f8dcc is splitting it per migrate type.

Correct on both counts.

Quoted 5 lines

What now could happen is that you free pages and get a list like:
H1 - H2 - H3 - C3 - C2 - C1 (H is hot and C is cold)
In low mem scenarios it could now happen that a allocation for hot pages falls back to cold pages (as usual fallback), but couldnt it be that it then also gets the cold pages in reverse order again ?

Yes, this is true but that would only matter from a cache perspective as you say your hardware and drivers are doing no automatic merging of requests.

Quoted 7 lines

Without e084b it would be like:
H1 - H2 - H3 - C1 - C2 - C3 and therefore all at least in order (with the issue that cold allocations from the right are reverse -> e084b) 5f8dcc is now lowering the size of those lists by splitting it into different migration types, so maybe both together are increasing the chance to get such an issue.

Quoted 5 lines

There is a good chance that 5f8dcc is causing direct reclaim to be entered when it could have been avoided as pages were on the per-cpu lists but I don't think the difference in cache hotness is enough to explain a 60% loss on your machine.

As far as I get it thats what your patch #2 fixed (page allocator: Fallback to other per-cpu lists when the target list is empty and memory is low), but as mentioned below that gave me +4%. Which is actually a fine win, but not for the issue currently in discussion.
As bonus I can give you now a little tested-by for both patches :-)

[...]

Signature · 2 lines
Grüsse / regards, Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization 
#17 lore raw
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 4 lines

How reproducible are these results with patch e084b reverted? i.e. I know it does not work now, but did reverting on the old kernel always fix it or were there occasions where the figures would be still bad?

Reverting e084b in the past showed something like +40%, so I thought it helps.
Afterwards I found out it wasn't a good testcase for reproducibility and when looking at a series it had +5%,-7%,+20%,...
So by far too noisy to be useful for bisect, discussion or fix testing. Thats why I made my big round reinventing the testcases in a more reproducible way like described above.
In those the original issue is still very visible - which means 4 runs comparing 2.6.32 vs 2.6.32-Reve084b being each max +/-1%.
While gitid-e084b vs. the one just before (51fbb) gives ~-60% all the time.

Quoted 5 lines
Quoted 2 lines

About all e084b can be affecting is cache hotness when fallback is occuring but 60% regression still seems way too high for just that.

Yes, most the ideas you, I an a few others so far had went for cache hotness - but always the summary was "couldn't be that much".

Well, it's cache hotness or the driver or hardware doing automatic merging of requests (which you say is not happening). There are two other possibilties but their effect should be very low as well.

Quoted 3 lines

On the other hand we have to consider that it could be a small timing due to cache or whatever else which leads into congestion_wait and then it waits HZ/50. That way a 1µs cause can lead to 20001µs lost which then cause 60%.

Maybe although it would imply the window where it can go wrong is very small. It seems unlikely that it would be hit every time.

Quoted 4 lines

Another important thing is that "cold page" afaik means the page content, and not the management struct etc. As the time lost is in management only (=> before accessing the actual 4k page content) it should not be important if a page is cold/hot for the mm code itself.

True. Even the zeroing should be a non-temporal store so the timing should not be affected.

Quoted 3 lines

That much for direct_reclaim that either runs into progress, but !page (BAD case) or not (GOOD case) - there only cache effects of struct page, pcp lists etc should have an effect.

There are only two other possibilities I can think of.

1. The extra parameter to rmqueue_bulk() means that your platform generates particularly bad code to pass a 6th parameter.
2. The branch inside the for loop is causing serious problems somehow

Causing a 60% slowdown from either of these seems unlikely but there isn't much choice than to eliminate them as possibilities. The attached patch is for "option 1" above. I'll think more on how to best handle "option 2"

Quoted 34 lines
Quoted 29 lines
Quoted 15 lines
Quoted 15 lines
Quoted 9 lines

Another bisect (keepign e084b reverted) brought up git id 5f8dcc21 which came in later. Both patches unapplied individually don't improve anything. But both patches reverted at the same time on git v2.6.32 bring us back to our old values (remember that is more than 2Gb/s improvement in throughput)!

Unfortunately 5f8dcc21 is as unobvious as e84b in explaining how this
can cause so much trouble.

There is a bug in commit 5f8dcc21. One consequence of it is that swap-based workloads can suffer. A second is that users of high-order allocations can enter direct reclaim a lot more than previously. This was fixed in commit a7016235a61d520e6806f38129001d935c4b6661 but you say that's not the fix in your case.

Quoted 8 lines
Quoted 4 lines

The only other interesting thing in commit 5f8dcc21 is that it increases the memory overhead of a per-cpu structure. If your memory usage is really borderline, it might have been just enough to push it over the edge.

I had this thoughts too, but as it only shows an effect with 5f8dcc21 and e084b reverted at the same time I'm wondering if it can be that. But I agree that this should be considered too until a verification can be done.

Another consequence of 5f8dcc is that pages are on the per-cpu lists that it is possible for pages to be on the per-cpu lists when the page allocator is called. There is potential that page reclaim is being entered as a result even though pages were free on the per-cpu lists.

Quoted 4 lines

It would not explain why reverting e084b makes such a difference but I can prototype a patch that falls back to other per-cpu lists before calling the page allocator as it used to do but without linear searching the per-cpu lists.

Quoted 5 lines

I tested these patch you submitted in another reply.
Short - it only gives it ~+4% by probably speeding up things a bit but not touching the issue in any way.
Some more detail below at the other test result.

You mention it gains 4% which is not great, but what impact, if any, had it on the number of calls to direct reclaim?

Quoted 8 lines

<SNIP>

Quoted 4 lines

Ok. Two avenues of attack then although both are focused on 5f8dcc21. The first is the prototype below that should avoid congestion_wait. The second is to reintroduce a fallback to other per-cpu lists and see if that was a big factor.

I tested both of your patches, btw thanks for the quick work!

Thanks for persisting with this! I know it's dragging on a lot longer than anyone would like.

Quoted 6 lines
Quoted 4 lines

Hmm, the patch as it currently stands is below. However, I warn you that it has only been boot-tested on qemu with no proper testing doing, either functional or performance testing. It may just go mad, drink your beer and chew on your dog.

In opposition to all your warnings it worked pretty well.

Nice one.

Quoted 10 lines

The patch replacing congestion_wait calls - which we know re futile in my scenario anyway - by zone_wait calls helps a lot. But so far as you suggested only by alleviating the symptoms.

From a throughput perspective alone it looks almost fixed:

                                                    4 thread      8 thread    16 thread
plain 2.6.32                                        100%          100%        100%
2.6.32 avoid direct reclaim                         103.85%       103.60%     104.04%
2.6.32 zone wait                                    125.46%       140.87%     131.89%
2.6.32 zone wait + 5f8dcc21 and e084b reverted      124.46%       143.35%     144.21%

Ok, the fact that the congestion_wait-avoidance patch makes such a large difference is very interesting although if the direct reclaim calls were lower, it would be less pronounced. I will be having those patches on the back-burner for another while unless we cannot resolve this problem a better way but it's nice to know they may have some potential.

Quoted 8 lines

Looking at the counters this theory (only fixing symptoms) seems to be confirmed. Before the ratio of perf_count_pages_direct_reclaim calls that ended in perf_count_failed_pages_direct_reclaim_but_progress was ~2% in the good cases and ~30% in bad cases. As congestion_wait in my case almost always waits the full HZ/50 this is a lot.

With the zone wait patch I see a huge throughput win, most probably by all the (still ~30%!) waits that don't have to wait for the full timeout.

That would be my expectation. The underlying problem of too-many-direct-reclaims is not solved.

Quoted 8 lines

But still reverting 5f8dcc21 and e084b fix the issue of so much direct
reclaims running in into that condition and that way giving throughput
another last push.
% ran into try_to_free did progress but get_page got !page

plain 2.6.32                                          30.16%
2.6.32 zone wait                                      29.05%
2.6.32 zone wait + 5f8dcc21 and e084b reverted        2.93%

And what impact did "Fallback to other per-cpu lists when the target list is empty and memory is low" have on the number of calls to direct reclaim?

Quoted 3 lines

Note - with the zone wait patch there are also almost twice as much direct_reclaim calls. Probably by several waiters coming of the list when the watermark is restored and allocating too much.

It could be either a thundering herd problem or just the fact that the processes are not asleep.

Quoted 2 lines

Another good side note - 5f8dcc21 and e084b reverted plus your zone wait patch is better than anything I have seen so far :-)

Well, that's something positive out of all this. The congestion_wait() patch was developed in response to this thread. Even though I knew it was treating symptons and not problems, the congestion_wait() was identified as a bad idea some time ago because the lack of pages may have nothing to do with IO.

Quoted 3 lines

Maybe its worth to measure both of your patches plus 5f8dcc21 and e084b reverted with 64 threads (the effect grows with # threads) just to rejoice in huge throughput values for a short time.

:)

Quoted 11 lines
Quoted 7 lines

<SNIP>

There is a good chance that 5f8dcc is causing direct reclaim to be entered when it could have been avoided as pages were on the per-cpu lists but I don't think the difference in cache hotness is enough to explain a 60% loss on your machine.

As far as I get it thats what your patch #2 fixed (page allocator: Fallback to other per-cpu lists when the target list is empty and memory is low), but as mentioned below that gave me +4%.

I need to know what effect, if any, it had on the direct reclaim figures. If it does drop them, but not by enough, then fallback can be allowed to occur in more cirsumstances - for example, by changing

+ if (pcp->count && free_pages <= low_wmark_pages(zone)) {

with

+ if (pcp->count && free_pages <= high_wmark_pages(zone)) {

Quoted 3 lines

Which is actually a fine win, but not for the issue currently in discussion. As bonus I can give you now a little tested-by for both patches :-)

Thanks :)

Still, the grind continues to try resolve this :(

1. If the "Fallback to other per-cpu lists" does reduce the number of calls to direct reclaim, but not by enough, please replace the comparison of low_wmark_pages with high_wmark_pages and let me know a) how much it affects throughput and b) how much it affects the number of calls to direct reclaim

This is addressing potential problems in 5f8dcc

2. Test with the patch below rmqueue_bulk-fewer-parameters to see if the number of parameters being passed is making some unexpected large difference

This is trying to figure out more what the problem with e084b is

==== CUT HERE ====
rmqueue_bulk-fewer-parameters

mm/page_alloc.c +4 -4
index 7021c68..180bab2 100644
@@ -936,7 +936,7 @@ retry_reserve:
  * a single hold of the lock, for efficiency.  Add them to the supplied list.
  * Returns the number of new pages which were placed at *list.
  */
-static int rmqueue_bulk(struct zone *zone, unsigned int order, 
+static int rmqueue_bulk(struct zone *zone,
 			unsigned long count, struct list_head *list,
 			int migratetype, int cold)
 {
@@ -944,7 +944,7 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order,
 	
 	spin_lock(&zone->lock);
 	for (i = 0; i < count; ++i) {
-		struct page *page = __rmqueue(zone, order, migratetype);
+		struct page *page = __rmqueue(zone, 0, migratetype);
 		if (unlikely(page == NULL))
 			break;
 
@@ -964,7 +964,7 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order,
 		set_page_private(page, migratetype);
 		list = &page->lru;
 	}
-	__mod_zone_page_state(zone, NR_FREE_PAGES, -(i << order));
+	__mod_zone_page_state(zone, NR_FREE_PAGES, -1);
 	spin_unlock(&zone->lock);
 	return i;
 }
@@ -1231,7 +1231,7 @@ again:
 		list = &pcp->lists[migratetype];
 		local_irq_save(flags);
 		if (list_empty(list)) {
-			pcp->count += rmqueue_bulk(zone, 0,
+			pcp->count += rmqueue_bulk(zone,
 					pcp->batch, list,
 					migratetype, cold);
 			if (unlikely(list_empty(list)))
#18 lore raw
Quoted 38 lines · Mel Gorman
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 4 lines

How reproducible are these results with patch e084b reverted? i.e. I know it does not work now, but did reverting on the old kernel always fix it or were there occasions where the figures would be still bad?

Reverting e084b in the past showed something like +40%, so I thought it helps.
Afterwards I found out it wasn't a good testcase for reproducibility and when looking at a series it had +5%,-7%,+20%,...
So by far too noisy to be useful for bisect, discussion or fix testing. Thats why I made my big round reinventing the testcases in a more reproducible way like described above.
In those the original issue is still very visible - which means 4 runs comparing 2.6.32 vs 2.6.32-Reve084b being each max +/-1%.
While gitid-e084b vs. the one just before (51fbb) gives ~-60% all the time.

Quoted 4 lines
Quoted 2 lines

About all e084b can be affecting is cache hotness when fallback is occuring but 60% regression still seems way too high for just that.

Yes, most the ideas you, I an a few others so far had went for cache hotness - but always the summary was "couldn't be that much".

Well, it's cache hotness or the driver or hardware doing automatic merging of requests (which you say is not happening). There are two other possibilties but their effect should be very low as well.

Quoted 3 lines

On the other hand we have to consider that it could be a small timing due to cache or whatever else which leads into congestion_wait and then it waits HZ/50. That way a 1µs cause can lead to 20001µs lost which then cause 60%.

Maybe although it would imply the window where it can go wrong is very small. It seems unlikely that it would be hit every time.

Quoted 4 lines

Another important thing is that "cold page" afaik means the page content, and not the management struct etc. As the time lost is in management only (=> before accessing the actual 4k page content) it should not be important if a page is cold/hot for the mm code itself.

True. Even the zeroing should be a non-temporal store so the timing should not be affected.

Quoted 8 lines
Quoted 3 lines

That much for direct_reclaim that either runs into progress, but !page (BAD case) or not (GOOD case) - there only cache effects of struct page, pcp lists etc should have an effect.

There are only two other possibilities I can think of.

1. The extra parameter to rmqueue_bulk() means that your platform generates particularly bad code to pass a 6th parameter.

I never checked that number, but out of the box I can tell you that 5 parameters can be passed per register and starting with the 6th they need to go via stack. So much about hope, some numbers below to keep results together.

Quoted 1 line

2. The branch inside the for loop is causing serious problems somehow

I also some branches can be tricky on s390, so I look forward to see your idea how we could verify that by not changing too much behavior.

Quoted 37 lines

Causing a 60% slowdown from either of these seems unlikely but there isn't much choice than to eliminate them as possibilities. The attached patch is for "option 1" above. I'll think more on how to best handle "option 2"

Quoted 32 lines
Quoted 32 lines
Quoted 15 lines
Quoted 15 lines
Quoted 9 lines

Another bisect (keepign e084b reverted) brought up git id 5f8dcc21 which came in later. Both patches unapplied individually don't improve anything. But both patches reverted at the same time on git v2.6.32 bring us back to our old values (remember that is more than 2Gb/s improvement in throughput)!

Unfortunately 5f8dcc21 is as unobvious as e84b in explaining how this
can cause so much trouble.

There is a bug in commit 5f8dcc21. One consequence of it is that swap-based workloads can suffer. A second is that users of high-order allocations can enter direct reclaim a lot more than previously. This was fixed in commit a7016235a61d520e6806f38129001d935c4b6661 but you say that's not the fix in your case.

Quoted 8 lines
Quoted 4 lines

The only other interesting thing in commit 5f8dcc21 is that it increases the memory overhead of a per-cpu structure. If your memory usage is really borderline, it might have been just enough to push it over the edge.

I had this thoughts too, but as it only shows an effect with 5f8dcc21 and e084b reverted at the same time I'm wondering if it can be that. But I agree that this should be considered too until a verification can be done.

Another consequence of 5f8dcc is that pages are on the per-cpu lists that it is possible for pages to be on the per-cpu lists when the page allocator is called. There is potential that page reclaim is being entered as a result even though pages were free on the per-cpu lists.

It would not explain why reverting e084b makes such a difference but I can prototype a patch that falls back to other per-cpu lists before calling the page allocator as it used to do but without linear searching the per-cpu lists.

Quoted 8 lines
Quoted 5 lines

I tested these patch you submitted in another reply.
Short - it only gives it ~+4% by probably speeding up things a bit but not touching the issue in any way.
Some more detail below at the other test result.

You mention it gains 4% which is not great, but what impact, if any, had it on the number of calls to direct reclaim?

sorry I missed to add that numbers yesterday.
Unfortunately it had no effect. vomparing 2.6.32 to 2.6.32 with avoid direct reclaim patch it has these for e.g. 16 threads:

with avoid direct reclaim (pcp list fallback)
perf_count_pages_direct_reclaim 32909
perf_count_failed_pages_direct_reclaim_but_progress 9802 =29.79%

remmember original 2.6.32 had:
perf_count_pages_direct_reclaim 33662
perf_count_failed_pages_direct_reclaim_but_progress 10152 =30.16%

So no big progress due to that (~4% TP as mentioned before).

Quoted 57 lines
Quoted 7 lines

<SNIP>

Quoted 4 lines

Ok. Two avenues of attack then although both are focused on 5f8dcc21. The first is the prototype below that should avoid congestion_wait. The second is to reintroduce a fallback to other per-cpu lists and see if that was a big factor.

I tested both of your patches, btw thanks for the quick work!

Thanks for persisting with this! I know it's dragging on a lot longer than anyone would like.

Quoted 5 lines
Quoted 4 lines

Hmm, the patch as it currently stands is below. However, I warn you that it has only been boot-tested on qemu with no proper testing doing, either functional or performance testing. It may just go mad, drink your beer and chew on your dog.

In opposition to all your warnings it worked pretty well.

Nice one.

Quoted 10 lines

The patch replacing congestion_wait calls - which we know re futile in my scenario anyway - by zone_wait calls helps a lot. But so far as you suggested only by alleviating the symptoms.

From a throughput perspective alone it looks almost fixed:

                                                    4 thread      8 thread    16 thread
plain 2.6.32                                        100%          100%        100%
2.6.32 avoid direct reclaim                         103.85%       103.60%     104.04%
2.6.32 zone wait                                    125.46%       140.87%     131.89%
2.6.32 zone wait + 5f8dcc21 and e084b reverted      124.46%       143.35%     144.21%

Ok, the fact that the congestion_wait-avoidance patch makes such a large difference is very interesting although if the direct reclaim calls were lower, it would be less pronounced. I will be having those patches on the back-burner for another while unless we cannot resolve this problem a better way but it's nice to know they may have some potential.

Quoted 8 lines

Looking at the counters this theory (only fixing symptoms) seems to be confirmed. Before the ratio of perf_count_pages_direct_reclaim calls that ended in perf_count_failed_pages_direct_reclaim_but_progress was ~2% in the good cases and ~30% in bad cases. As congestion_wait in my case almost always waits the full HZ/50 this is a lot.

With the zone wait patch I see a huge throughput win, most probably by all the (still ~30%!) waits that don't have to wait for the full timeout.

That would be my expectation. The underlying problem of too-many-direct-reclaims is not solved.

Quoted 8 lines

But still reverting 5f8dcc21 and e084b fix the issue of so much direct
reclaims running in into that condition and that way giving throughput
another last push.
% ran into try_to_free did progress but get_page got !page

plain 2.6.32                                          30.16%
2.6.32 zone wait                                      29.05%
2.6.32 zone wait + 5f8dcc21 and e084b reverted        2.93%
Quoted 2 lines

And what impact did "Fallback to other per-cpu lists when the target list is empty and memory is low" have on the number of calls to direct reclaim?

see above

Quoted 7 lines
Quoted 4 lines

Note - with the zone wait patch there are also almost twice as much direct_reclaim calls. Probably by several waiters coming of the list when the watermark is restored and allocating too much.

It could be either a thundering herd problem or just the fact that the processes are not asleep.

Thats sounds right, but as other things we had that is just related to the symptoms so I guess we can neglect it for now.

Quoted 41 lines
Quoted 2 lines

Another good side note - 5f8dcc21 and e084b reverted plus your zone wait patch is better than anything I have seen so far :-)

Well, that's something positive out of all this. The congestion_wait() patch was developed in response to this thread. Even though I knew it was treating symptons and not problems, the congestion_wait() was identified as a bad idea some time ago because the lack of pages may have nothing to do with IO.

Quoted 3 lines

Maybe its worth to measure both of your patches plus 5f8dcc21 and e084b reverted with 64 threads (the effect grows with # threads) just to rejoice in huge throughput values for a short time.

:)

Quoted 10 lines
Quoted 7 lines

<SNIP>

There is a good chance that 5f8dcc is causing direct reclaim to be entered when it could have been avoided as pages were on the per-cpu lists but I don't think the difference in cache hotness is enough to explain a 60% loss on your machine.

As far as I get it thats what your patch #2 fixed (page allocator: Fallback to other per-cpu lists when the target list is empty and memory is low), but as mentioned below that gave me +4%.

I need to know what effect, if any, it had on the direct reclaim figures. If it does drop them, but not by enough, then fallback can be allowed to occur in more cirsumstances - for example, by changing

(quoted diff) +1 -0
+	if (pcp->count && free_pages <= low_wmark_pages(zone)) {

with

(quoted diff) +1 -0
+	if (pcp->count && free_pages <= high_wmark_pages(zone)) {
Quoted 4 lines

Which is actually a fine win, but not for the issue currently in discussion. As bonus I can give you now a little tested-by for both patches :-)

Thanks :)

Quoted 7 lines

Still, the grind continues to try resolve this :(

1. If the "Fallback to other per-cpu lists" does reduce the number of calls to direct reclaim, but not by enough, please replace the comparison of low_wmark_pages with high_wmark_pages and let me know a) how much it affects throughput and b) how much it affects the number of calls to direct reclaim

The difference between low or high watermark is minimal.

here with avoid direct reclaims with high watermark
perf_count_pages_direct_reclaim 32839
perf_count_failed_pages_direct_reclaim_but_progress 488 =30.16%

as reference from above the one with low watermark
perf_count_pages_direct_reclaim 32909
perf_count_failed_pages_direct_reclaim_but_progress 9802 =29.79%

=> this is inside the small amount of noise that is left.

Quoted 6 lines

This is addressing potential problems in 5f8dcc

2. Test with the patch below rmqueue_bulk-fewer-parameters to see if the number of parameters being passed is making some unexpected large difference

BINGO - this definitely hit something.
This experimental patch does two things - on one hand it closes the race we had:

4 THREAD READ 8 THREAD READ 16 THREAD READ %ofcalls
perf_count_congestion_wait 13 27 52
perf_count_call_congestion_wait_from_alloc_pages_high_priority 0 0 0
perf_count_call_congestion_wait_from_alloc_pages_slowpath 13 27 52 99.52%
perf_count_pages_direct_reclaim 30867 56811 131552

perf_count_failed_pages_direct_reclaim                             14              24                52	
perf_count_failed_pages_direct_reclaim_but_progress                14              21                52              0.04% !!!

On the other hand we see that the number of direct_reclaim calls increased by ~x4.

I assume (might be totally wrong) that the x4 increase of direct_reclaim calls could be caused by the fact that before we used higher orders which worked on x4 number of pages at once.

This leaves me with two ideas what the real issue could be:
1. either really the 6th parameter as this is the first one that needs to go on stack and that way might open a race and rob gcc a big pile of optimization chances
2. maybe just the fact that we now only do "order 0" operations changes the behaviour

I think I try to cover that tomorrow by keeping the old call with 6 parms but rewriting it internally to 0 again (and hope I don't need to much tricks to stop gcc from optimizing it out). Maybe I could also extend statistics to track order in the failing cases vs. all cases or something like this.

I don't have full access to the machines tomorrow, so it might get monday for a proper response. Thanks a lot Mel, this brought us much closer to a final fix.

Quoted 4 lines

This is trying to figure out more what the problem with e084b is

==== CUT HERE ====
rmqueue_bulk-fewer-parameters

[...]

Signature · 2 lines
Grüsse / regards, Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization 
#19 lore raw
Quoted 12 lines · Christian Ehrhardt
Quoted 3 lines

2. Test with the patch below rmqueue_bulk-fewer-parameters to see if the number of parameters being passed is making some unexpected large difference

BINGO - this definitely hit something.
This experimental patch does two things - on one hand it closes the race we had:

4 THREAD READ 8 THREAD READ 16 THREAD READ %ofcalls
perf_count_congestion_wait 13 27 52
perf_count_call_congestion_wait_from_alloc_pages_high_priority 0 0 0
perf_count_call_congestion_wait_from_alloc_pages_slowpath 13 27 52 99.52%
perf_count_pages_direct_reclaim 30867 56811 131552

Quoted 6 lines
perf_count_failed_pages_direct_reclaim                             14              24                52	
perf_count_failed_pages_direct_reclaim_but_progress                14              21                52              0.04% !!!

On the other hand we see that the number of direct_reclaim calls increased by ~x4.

I assume (might be totally wrong) that the x4 increase of direct_reclaim calls could be caused by the fact that before we used higher orders which worked on x4 number of pages at once.

But the order parameter was always passed as constant 0 by the caller?

Quoted 2 lines

This leaves me with two ideas what the real issue could be:
1. either really the 6th parameter as this is the first one that needs to go on stack and that way might open a race and rob gcc a big pile of optimization chances

It must be something to do wih code generation AFAIKS. I'm surprised the function isn't inlined, giving exactly the same result regardless of the patch.

Unlikely to be a correctness issue with code generation, but I'm really surprised that a small difference in performance could have such a big (and apparently repeatable) effect on behaviour like this.

What's the assembly look like?

#20 lore raw
Quoted 7 lines · Nick Piggin
Quoted 2 lines

This leaves me with two ideas what the real issue could be:
1. either really the 6th parameter as this is the first one that needs to go on stack and that way might open a race and rob gcc a big pile of optimization chances

It must be something to do wih code generation AFAIKS. I'm surprised the function isn't inlined, giving exactly the same result regardless of the patch.

Quoted 5 lines

Unlikely to be a correctness issue with code generation, but I'm really surprised that a small difference in performance could have such a big (and apparently repeatable) effect on behaviour like this.

What's the assembly look like?

Oh, and what happens if you always_inline it? I would have thought gcc should be able to generate the same code in both cases -- unless I'm really missing something and there is a functional change in there?

#21 lore raw
In reply to Nick Piggin
Quoted 15 lines · Nick Piggin
Quoted 14 lines · Christian Ehrhardt
Quoted 3 lines

2. Test with the patch below rmqueue_bulk-fewer-parameters to see if the number of parameters being passed is making some unexpected large difference

BINGO - this definitely hit something.
This experimental patch does two things - on one hand it closes the race we had:

4 THREAD READ 8 THREAD READ 16 THREAD READ %ofcalls
perf_count_congestion_wait 13 27 52
perf_count_call_congestion_wait_from_alloc_pages_high_priority 0 0 0
perf_count_call_congestion_wait_from_alloc_pages_slowpath 13 27 52 99.52%
perf_count_pages_direct_reclaim 30867 56811 131552

perf_count_failed_pages_direct_reclaim                             14              24                52	
perf_count_failed_pages_direct_reclaim_but_progress                14              21                52              0.04% !!!
Quoted 5 lines
Quoted 3 lines

On the other hand we see that the number of direct_reclaim calls increased by ~x4.

I assume (might be totally wrong) that the x4 increase of direct_reclaim calls could be caused by the fact that before we used higher orders which worked on x4 number of pages at once.

But the order parameter was always passed as constant 0 by the caller?

Uh - I didn't see that in the first look - you're right.
So the x4 in direct_reclaim calls are not caused by e.g. missing order information. Thanks for spotting this.

Quoted 7 lines
Quoted 3 lines

This leaves me with two ideas what the real issue could be:
1. either really the 6th parameter as this is the first one that needs to go on stack and that way might open a race and rob gcc a big pile of optimization chances

It must be something to do with code generation AFAIKS. I'm surprised the function isn't inlined, giving exactly the same result regardless of the patch.

after checking asm I can tell that rmqueue_bulk is inlined.
Therefore the test to inline it explicitly as requested in another mail can be considered negligible. It actually boils down to something different in Mels patch - see below.

Quoted 6 lines

Unlikely to be a correctness issue with code generation, but I'm really surprised that a small difference in performance could have such a big (and apparently repeatable) effect on behaviour like this.

What's the assembly look like?

Line 214763
This is the preparation of the __mod_zone_page_state call from rmqueue_bulk which is inlined into get_page_from_freelist.

!ORDER CHANGED FOR READABILITY!

FAST                               SLOW
lgr  %r2,%r11  #parm 1 from r11    lgr  %r2, %r11  #parm 1 from r11 - same
lghi %r3,0     #parm 2 = 0 const   lghi %r3,0      #parm 2 = 0 - same
lghi %r4,-1    #parm 3 = -1 const  lcr  %r4,%r12   #parm 3 as complement of r12?

lgfr %r4,%r4 #parm 3 - clear upper 32 bit?
brasl #call brasl #call

=> This is most probably this part of Mel's patch:
23 @@ -965,7 +965,7 @@
24 set_page_private(page, migratetype);
25 list = &page->lru;

 26         }
 27 -       __mod_zone_page_state(zone, NR_FREE_PAGES, -(i << order));
 28 +       __mod_zone_page_state(zone, NR_FREE_PAGES, -1);
 29         spin_unlock(&zone->lock);
 30         return i;

31 }

-> doesn't look too important, but to be sure we can build an executable with just this change, but still passing order to rmqueue_bulk - see below.

Line 214800
Differend end of the function. In Slow path there is a check to %r4 and dependent two different jump targets inside get_page_from_freelist, while in the fast version there is only an unconditional jump (both after a _raw_spin_lock_wait).

FAST SLOW
brasl # call _raw_spin_lock_wait brasl # _raw_spin_lock_wait
j 1e6294 get_page_from_freelist lg %r4,288(%r15) # from stack
ltgr %r4,%r4 # compare
jne 1e62a2 get_page_from_freelist lhi %r12,0 # new call gets r12 @ 0 (GOT?)
j 1e6340 get_page_from_freelist
The rest is the same. So what is left is that the slow variant has a new branch back in to get_page_from_freelist with r12 set to zero. This jump wents directly after the also new lcr call seen in the first difference and might therfore be related to that change.

Now I first thought it might not be rmqueue_bulk that misses optimization, but __mod_zone_page_state - but that one looks the same in both cases.

Therefore I took a shot for 2.6.32 with just that snippet above applied (__mod_zone_page_state call without order which should be fine as we know it is 0 in all cases). And it is interesting to see that this snippet alone is enough to fix throughput and the direct_reclaim -> progress&!page ratio.

The differences in asm are pretty much the same, as before rmqueue_bulk was already inlined the actually intended change to its parameters was negligible. I wondered if it would be important if that is a constant value (-1) or if the reason was caused by that shift. So I tried:

23 @@ -965,7 +965,7 @@
24 set_page_private(page, migratetype);
25 list = &page->lru;

 26         }
 27 -       __mod_zone_page_state(zone, NR_FREE_PAGES, -(i << order));
 28 +       __mod_zone_page_state(zone, NR_FREE_PAGES, -i);
 29         spin_unlock(&zone->lock);
 30         return i;

31 }

Now that one has still the issue of the very huge throughput loss and the bad ratio of driect_reclaims leading into congestion_wait. Now as far as I can see that - oh so important - "-i" or "-1" ends up in zone_page_state_add as variable x:
static inline void zone_page_state_add(long x, struct zone *zone,
enum zone_stat_item item)

{
        atomic_long_add(x, &zone->vm_stat[item]);
        atomic_long_add(x, &vm_stat[item]);
}

I guess the intention there is to update the zone with the number of pages freed - and I also guess that -1 as constant might be wrong there. That would also explain some weird output I saw like this after boot:
h37lp01:~ # cat /proc/meminfo

MemTotal:         251216 kB
MemFree:          483460 kB bigger than total

BUT, and that is now again open for discussion - "__mod_zone_page_state(zone, NR_FREE_PAGES, -1)" even being wrong - fixed the issue in terms of throughput and the congestion_waits as good as reverting e084b+5f8dcc21!

Signature · 2 lines
Grüsse / regards, Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization 
#22 lore raw
Quoted 31 lines · Christian Ehrhardt
Quoted 15 lines · Nick Piggin
Quoted 14 lines · Christian Ehrhardt
Quoted 3 lines

2. Test with the patch below rmqueue_bulk-fewer-parameters to see if the number of parameters being passed is making some unexpected large difference

BINGO - this definitely hit something.
This experimental patch does two things - on one hand it closes the race we had:

4 THREAD READ 8 THREAD READ 16 THREAD READ %ofcalls
perf_count_congestion_wait 13 27 52
perf_count_call_congestion_wait_from_alloc_pages_high_priority 0 0 0
perf_count_call_congestion_wait_from_alloc_pages_slowpath 13 27 52 99.52%
perf_count_pages_direct_reclaim 30867 56811 131552

perf_count_failed_pages_direct_reclaim                             14              24                52	
perf_count_failed_pages_direct_reclaim_but_progress                14              21                52              0.04% !!!
Quoted 5 lines
Quoted 3 lines

On the other hand we see that the number of direct_reclaim calls increased by ~x4.

I assume (might be totally wrong) that the x4 increase of direct_reclaim calls could be caused by the fact that before we used higher orders which worked on x4 number of pages at once.

But the order parameter was always passed as constant 0 by the caller?

Uh - I didn't see that in the first look - you're right.
So the x4 in direct_reclaim calls are not caused by e.g. missing order information. Thanks for spotting this.

Quoted 4 lines
Quoted 3 lines

This leaves me with two ideas what the real issue could be:
1. either really the 6th parameter as this is the first one that needs to go on stack and that way might open a race and rob gcc a big pile of optimization chances

Quoted 6 lines
Quoted 3 lines

It must be something to do with code generation AFAIKS. I'm surprised the function isn't inlined, giving exactly the same result regardless of the patch.

after checking asm I can tell that rmqueue_bulk is inlined.
Therefore the test to inline it explicitly as requested in another mail can be considered negligible.

Well, this is good at least. The expectation was that it was getting automatically inlined but I'd stopped expecting anything as this problem is weird.

Quoted 13 lines

It actually boils down to something different in Mels patch - see below.

Quoted 6 lines

Unlikely to be a correctness issue with code generation, but I'm really surprised that a small difference in performance could have such a big (and apparently repeatable) effect on behaviour like this.

What's the assembly look like?

Line 214763
This is the preparation of the __mod_zone_page_state call from rmqueue_bulk which is inlined into get_page_from_freelist.

Quoted 6 lines

!ORDER CHANGED FOR READABILITY!

FAST                               SLOW
lgr  %r2,%r11  #parm 1 from r11    lgr  %r2, %r11  #parm 1 from r11 - same
lghi %r3,0     #parm 2 = 0 const   lghi %r3,0      #parm 2 = 0 - same
lghi %r4,-1    #parm 3 = -1 const  lcr  %r4,%r12   #parm 3 as complement of r12?

lgfr %r4,%r4 #parm 3 - clear upper 32 bit?

I'm not an s390 assembler expert but from the (only) reference I found.

LGHI - loads a half word (16 bits) to a 64 bit
to -1 into register 4

LCR - load the comlpement of count (i.e. -count)
LGFR - load a 64 register with a 32 value, no idea if this clears the
upper bits or not.

So, they are doing very different things here.... err, crap, because I broke it. The patch to eliminate the parameter is wrong

- __mod_zone_page_state(zone, NR_FREE_PAGES, -(i << order));
+ __mod_zone_page_state(zone, NR_FREE_PAGES, -1);

should have been

- __mod_zone_page_state(zone, NR_FREE_PAGES, -(i << order));
+ __mod_zone_page_state(zone, NR_FREE_PAGES, -i);

Quoted 11 lines

brasl #call brasl #call

=> This is most probably this part of Mel's patch:
23 @@ -965,7 +965,7 @@
24 set_page_private(page, migratetype);
25 list = &page->lru;

 26         }
 27 -       __mod_zone_page_state(zone, NR_FREE_PAGES, -(i << order));
 28 +       __mod_zone_page_state(zone, NR_FREE_PAGES, -1);
 29         spin_unlock(&zone->lock);
 30         return i;
Quoted 5 lines

31 }

-> doesn't look too important, but to be sure we can build an executable with just this change, but still passing order to rmqueue_bulk - see below.

It's important, it's broken. That -1 should be -i. Can you test with that fixed? I'm very sorry.

Quoted 19 lines

Line 214800
Differend end of the function. In Slow path there is a check to %r4 and dependent two different jump targets inside get_page_from_freelist, while in the fast version there is only an unconditional jump (both after a _raw_spin_lock_wait).

FAST SLOW
brasl # call _raw_spin_lock_wait brasl # _raw_spin_lock_wait
j 1e6294 get_page_from_freelist lg %r4,288(%r15) # from stack
ltgr %r4,%r4 # compare
jne 1e62a2 get_page_from_freelist lhi %r12,0 # new call gets r12 @ 0 (GOT?)
j 1e6340 get_page_from_freelist

The rest is the same. So what is left is that the slow variant has a new branch back in to get_page_from_freelist with r12 set to zero. This jump wents directly after the also new lcr call seen in the first difference and might therfore be related to that change.

Quoted 8 lines

Now I first thought it might not be rmqueue_bulk that misses optimization, but __mod_zone_page_state - but that one looks the same in both cases.

Therefore I took a shot for 2.6.32 with just that snippet above applied (__mod_zone_page_state call without order which should be fine as we know it is 0 in all cases). > And it is interesting to see that this snippet alone is enough to fix throughput and the direct_reclaim -> progress&!page ratio.

Unfortunately, it's because I broke the free counters in that patch so we are probably breaking watermarks. I imagine the counters in /proc/vmstat are very funny looking.

Quoted 18 lines

The differences in asm are pretty much the same, as before rmqueue_bulk was already inlined the actually intended change to its parameters was negligible. I wondered if it would be important if that is a constant value (-1) or if the reason was caused by that shift. So I tried:

23 @@ -965,7 +965,7 @@
24 set_page_private(page, migratetype);
25 list = &page->lru;

 26         }
 27 -       __mod_zone_page_state(zone, NR_FREE_PAGES, -(i << order));
 28 +       __mod_zone_page_state(zone, NR_FREE_PAGES, -i);
 29         spin_unlock(&zone->lock);
 30         return i;

31 }

Now that one has still the issue of the very huge throughput loss and the bad ratio of driect_reclaims leading into congestion_wait. Now as far as I can see that - oh so important - "-i" or "-1" ends up in zone_page_state_add as variable x:
static inline void zone_page_state_add(long x, struct zone *zone,
enum zone_stat_item item)

Quoted 6 lines
{
        atomic_long_add(x, &zone->vm_stat[item]);
        atomic_long_add(x, &vm_stat[item]);
}

I guess the intention there is to update the zone with the number of pages freed - and I also guess that -1 as constant might be wrong there.

It is.

Quoted 6 lines

That would also explain some weird output I saw like this after boot:
h37lp01:~ # cat /proc/meminfo

MemTotal:         251216 kB
MemFree:          483460 kB bigger than total

BUT, and that is now again open for discussion - "__mod_zone_page_state(zone, NR_FREE_PAGES, -1)" even being wrong - fixed the issue in terms of throughput and the congestion_waits as good as reverting e084b+5f8dcc21!

It "fixes" it only by not calling direct reclaim when it should :(

Signature · 3 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
#23 lore raw
Quoted 1 line · Mel Gorman

On Mon, Feb 15, 2010 at 04:46:53PM +0100, Christian Ehrhardt wrote:

[...]

Quoted 13 lines
Quoted 13 lines

The differences in asm are pretty much the same, as before rmqueue_bulk was already inlined the actually intended change to its parameters was negligible. I wondered if it would be important if that is a constant value (-1) or if the reason was caused by that shift. So I tried:

23 @@ -965,7 +965,7 @@
24 set_page_private(page, migratetype);
25 list = &page->lru;

 26         }
 27 -       __mod_zone_page_state(zone, NR_FREE_PAGES, -(i << order));
 28 +       __mod_zone_page_state(zone, NR_FREE_PAGES, -i);
 29         spin_unlock(&zone->lock);
 30         return i;

31 }

[...]

Quoted 1 line

It "fixes" it only by not calling direct reclaim when it should :(

yeah as we both realized -1 was not right so it was more a crazy workaround :-)

Anyway after that being a dead end again I dug even deeper into the details of direct_reclaim - I think we can agree that out of the counters we already know the race between try_to_free making progress and get_page not getting a page causing the congestion_wait is source of the issue.

So what I tried to bring some more light into all that was extending my perf counters to track a few more details in direct_reclaim. Two patches are attached and apply after the other three already available in that thread.
The intention is
a) to track the time
a1) spent in try_to_free_pages
a2) consumed after try_to_free_pages until get_page_from_freelist
a3) spent in get_pages_from_freelist
b1) after seeing that order!=0 -> drain_all_pages I wondered if that might differ even all calls look like they have zero b2) tracking the average amount of pages freed by try_to_free_pages for fast path and slow path (progres&!page)

Naming convention (otherwise it might get confusing here)
Good case - the scenario e.g. with e084b and 5f8dcc21 reverted resulting in high throughput and a low ratio of direct_reclaim running into progress&!page Bad case - the scenario e.g. on a clean 2.6.32
Fast path - direct reclaim calls that did not run into progress&!page
Slow path - direct reclaim calls that ran into progress&!page ending up in a long congestion_wait and therefore called "slow" path

Mini summary of what we had before in huge tables:
fast path slow path
GOOD CASE ~98% ~1-3%
BAD CASE ~70% ~30%
-> leading to throughput impact of e.g. 600 mb/s with 16 iozone threads (worse with even more threads)

Out of the numbers I got the following things might help us to create a new approach to a solution. The timings showed that that the so called slow case is actually much faster passing though direct_reclaim in bad case.

GOOD CASE duration
a1 Fast-avg-duration_pre_ttf_2_post_ttf 164099
a2 Fast-avg-duration_post_ttf_2_pre_get_page 459
a3 Fast-avg-duration_pre_get_page_2_post_get_page 346
a1 Slow-avg-duration_pre_ttf_2_post_ttf 127621
a2 Slow-avg-duration_post_ttf_2_pre_get_page 1957
a3 Slow-avg-duration_pre_get_page_2_post_get_page 256

BAD CASE                                         duration   deviation to good case in %
a1 Fast-avg-duration_pre_ttf_2_post_ttf            122921   -25.09%
a2 Fast-avg-duration_post_ttf_2_pre_get_page          521   13.53%
a3 Fast-avg-duration_pre_get_page_2_post_get_page     244   -29.55%
a1 Slow-avg-duration_pre_ttf_2_post_ttf            109740   -14.01%
a2 Slow-avg-duration_post_ttf_2_pre_get_page          250   -87.18%
a3 Slow-avg-duration_pre_get_page_2_post_get_page     117   -54.16%

That means that in the bad case the execution is much faster. Especially in the case that eventually runs into the slow path try_to_free is 14% faster, more important the time between try_to_free and get_pages is 87%! faster => less than a fifth and finally get_page is 54% faster, but that is probably just failing in an almost empty list which is fast^^.

As I checked order which always was zero the time is not spent in drain_all_pages and the only other thing left might be cond_resched ?!
Everything else are a few assignments so it can't be much else.
But why would e.g. not running into schedule in cond_resched cause get_pages to not find anything - I don't know and I would expect it should be the other way around - the faster you get from free to get the more pages should be left.

I thought the progress try_to_free_pages is doing might be important as well so I took numbers for that too.

Quoted 1 line

From those I see that the good case as well as the bad case has an average of 62 pages freed in fast path.

But in slow path they start to differ - while the good case that is running only seldom in that path anyway frees an average of 1.46 pages (that might be the thing causing it not getting a page eventually) in the bad case it makes a progress of avg 37 pages even in slow path.

PAGES-FREED  fast path   slow path
GOOD CASE      ~62       ~1.46
BAD CASE       ~62       ~37

Thinking of it as asking "how few pages do I have to free until I fall from fast to slow path" the kernels behave different it looks wrong but interesting. The good case only drops to slow path (!page) in case of ~1.46 pages freed while the bad case seems to enter that much earlier with even 37 pages freed.

As order is always 0 and get_page afaik about getting just "one" page I wonder where these 37 pages disappeared especially as in bad case it is much faster getting to get_pages after freeing those ~37 pages.

Comments and ideas welcome!

Signature · 2 lines
Grüsse / regards, Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization 
kernel/sysctl.c +55 -0
Index: linux-2.6-git/kernel/sysctl.c
===================================================================
@@ -264,9 +264,64 @@
 extern unsigned long perf_count_failed_pages_direct_reclaim_but_progress;
 extern unsigned long perf_count_call_congestion_wait_from_alloc_pages_high_priority;
 extern unsigned long perf_count_call_congestion_wait_from_alloc_pages_slowpath;
+
+extern unsigned long long perf_count_direct_reclaim_slow_dursum_pre_ttf_2_post_ttf;
+extern unsigned long long perf_count_direct_reclaim_slow_dursum_post_ttf_2_pre_get_page;
+extern unsigned long long perf_count_direct_reclaim_slow_dursum_pre_get_page_2_post_ttf;
+extern unsigned long long perf_count_direct_reclaim_fast_dursum_pre_ttf_2_post_ttf;
+extern unsigned long long perf_count_direct_reclaim_fast_dursum_post_ttf_2_pre_get_page;
+extern unsigned long long perf_count_direct_reclaim_fast_dursum_pre_get_page_2_post_ttf;
 static struct ctl_table perf_table[] = {
 	{
 		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_direct_reclaim_slow_dursum_pre_ttf_2_post_ttf",
+		.data           = &perf_count_direct_reclaim_slow_dursum_pre_ttf_2_post_ttf,
+		.mode           = 0666,
+		.maxlen		= sizeof(unsigned long long),
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_direct_reclaim_slow_dursum_post_ttf_2_pre_get_page",
+		.data           = &perf_count_direct_reclaim_slow_dursum_post_ttf_2_pre_get_page,
+		.mode           = 0666,
+		.maxlen		= sizeof(unsigned long long),
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_direct_reclaim_slow_dursum_pre_get_page_2_post_get_page",
+		.data           = &perf_count_direct_reclaim_slow_dursum_pre_get_page_2_post_get_page,
+		.mode           = 0666,
+		.maxlen		= sizeof(unsigned long long),
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_direct_reclaim_fast_dursum_pre_ttf_2_post_ttf",
+		.data           = &perf_count_direct_reclaim_fast_dursum_pre_ttf_2_post_ttf,
+		.mode           = 0666,
+		.maxlen		= sizeof(unsigned long long),
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_direct_reclaim_fast_dursum_post_ttf_2_pre_get_page",
+		.data           = &perf_count_direct_reclaim_fast_dursum_post_ttf_2_pre_get_page,
+		.mode           = 0666,
+		.maxlen		= sizeof(unsigned long long),
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_direct_reclaim_fast_dursum_pre_get_page_2_post_get_page",
+		.data           = &perf_count_direct_reclaim_fast_dursum_pre_get_page_2_post_get_page,
+		.mode           = 0666,
+		.maxlen		= sizeof(unsigned long long),
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
 		.procname       = "perf_count_congestion_wait",
 		.data           = &perf_count_congestion_wait,
 		.mode           = 0666,
mm/page_alloc.c +29 -1
Index: linux-2.6-git/mm/page_alloc.c
===================================================================
@@ -51,6 +51,7 @@
 #include <trace/events/kmem.h>
 
 #include <asm/tlbflush.h>
+#include <asm/timex.h>
 #include <asm/div64.h>
 #include "internal.h"
 
@@ -1655,6 +1656,17 @@
 unsigned long perf_count_failed_pages_direct_reclaim = 0;
 unsigned long perf_count_failed_pages_direct_reclaim_but_progress = 0;
 
+unsigned long long perf_count_direct_reclaim_slow_dursum_pre_ttf_2_post_ttf = 0;
+unsigned long long perf_count_direct_reclaim_slow_dursum_post_ttf_2_pre_get_page = 0;
+unsigned long long perf_count_direct_reclaim_slow_dursum_pre_get_page_2_post_get_page = 0;
+unsigned long long perf_count_direct_reclaim_fast_dursum_pre_ttf_2_post_ttf = 0;
+unsigned long long perf_count_direct_reclaim_fast_dursum_post_ttf_2_pre_get_page = 0;
+unsigned long long perf_count_direct_reclaim_fast_dursum_pre_get_page_2_post_get_page = 0;
+
+static inline unsigned long long tod_duration_ns(unsigned long long from, unsigned long long to) {
+	return ((to*125)>>9)-((from*125)>>9);
+}
+
 /* The really slow allocator path where we enter direct reclaim */
 static inline struct page *
 __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order,
@@ -1665,6 +1677,7 @@
 	struct page *page = NULL;
 	struct reclaim_state reclaim_state;
 	struct task_struct *p = current;
+	unsigned long long t1,t2,t3,t4;
 
 	cond_resched();
 
@@ -1675,8 +1688,12 @@
 	reclaim_state.reclaimed_slab = 0;
 	p->reclaim_state = &reclaim_state;
 
+	t1 = get_clock();
+
 	*did_some_progress = try_to_free_pages(zonelist, order, gfp_mask, nodemask);
 
+	t2 = get_clock();
+
 	p->reclaim_state = NULL;
 	lockdep_clear_current_reclaim_state();
 	p->flags &= ~PF_MEMALLOC;
@@ -1686,17 +1703,28 @@
 	if (order != 0)
 		drain_all_pages();
 
+	t3 = get_clock();
 	if (likely(*did_some_progress))
 		page = get_page_from_freelist(gfp_mask, nodemask, order,
 					zonelist, high_zoneidx,
 					alloc_flags, preferred_zone,
 					migratetype);
+	t4 = get_clock();
 
 	perf_count_pages_direct_reclaim++;
 	if (!page)
 		perf_count_failed_pages_direct_reclaim++;
-	if (!page && (*did_some_progress))
+	if (!page && (*did_some_progress)) {
 		perf_count_failed_pages_direct_reclaim_but_progress++;
+		perf_count_direct_reclaim_slow_dursum_pre_ttf_2_post_ttf += tod_duration_ns(t1,t2);
+		perf_count_direct_reclaim_slow_dursum_post_ttf_2_pre_get_page += tod_duration_ns(t2,t3);
+		perf_count_direct_reclaim_slow_dursum_pre_get_page_2_post_get_page += tod_duration_ns(t3,t4);
+	}
+	else {
+		perf_count_direct_reclaim_fast_dursum_pre_ttf_2_post_ttf += tod_duration_ns(t1,t2);
+		perf_count_direct_reclaim_fast_dursum_post_ttf_2_pre_get_page += tod_duration_ns(t2,t3);
+		perf_count_direct_reclaim_fast_dursum_pre_get_page_2_post_get_page += tod_duration_ns(t3,t4);
+	}
 
 	return page;
 }
kernel/sysctl.c +38 -2
Index: linux-2.6-git/kernel/sysctl.c
===================================================================
@@ -267,13 +267,49 @@
 
 extern unsigned long long perf_count_direct_reclaim_slow_dursum_pre_ttf_2_post_ttf;
 extern unsigned long long perf_count_direct_reclaim_slow_dursum_post_ttf_2_pre_get_page;
-extern unsigned long long perf_count_direct_reclaim_slow_dursum_pre_get_page_2_post_ttf;
+extern unsigned long long perf_count_direct_reclaim_slow_dursum_pre_get_page_2_post_get_page;
 extern unsigned long long perf_count_direct_reclaim_fast_dursum_pre_ttf_2_post_ttf;
 extern unsigned long long perf_count_direct_reclaim_fast_dursum_post_ttf_2_pre_get_page;
-extern unsigned long long perf_count_direct_reclaim_fast_dursum_pre_get_page_2_post_ttf;
+extern unsigned long long perf_count_direct_reclaim_fast_dursum_pre_get_page_2_post_get_page;
+extern unsigned long perf_count_direct_reclaim_fast_sum_progress;
+extern unsigned long perf_count_direct_reclaim_slow_sum_progress;
+extern unsigned long perf_count_direct_reclaim_fast_sum_order;
+extern unsigned long perf_count_direct_reclaim_slow_sum_order;
 static struct ctl_table perf_table[] = {
 	{
 		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_direct_reclaim_slow_sum_progress",
+		.data           = &perf_count_direct_reclaim_slow_sum_progress,
+		.mode           = 0666,
+		.maxlen		= sizeof(unsigned long long),
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_direct_reclaim_slow_sum_order",
+		.data           = &perf_count_direct_reclaim_slow_sum_order,
+		.mode           = 0666,
+		.maxlen		= sizeof(unsigned long long),
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_direct_reclaim_fast_sum_progress",
+		.data           = &perf_count_direct_reclaim_fast_sum_progress,
+		.mode           = 0666,
+		.maxlen		= sizeof(unsigned long long),
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname       = "perf_count_direct_reclaim_fast_sum_order",
+		.data           = &perf_count_direct_reclaim_fast_sum_order,
+		.mode           = 0666,
+		.maxlen		= sizeof(unsigned long long),
+		.proc_handler   = &proc_doulongvec_minmax,
+	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
 		.procname       = "perf_count_direct_reclaim_slow_dursum_pre_ttf_2_post_ttf",
 		.data           = &perf_count_direct_reclaim_slow_dursum_pre_ttf_2_post_ttf,
 		.mode           = 0666,
mm/page_alloc.c +11 -0
Index: linux-2.6-git/mm/page_alloc.c
===================================================================
@@ -1663,6 +1663,11 @@
 unsigned long long perf_count_direct_reclaim_fast_dursum_post_ttf_2_pre_get_page = 0;
 unsigned long long perf_count_direct_reclaim_fast_dursum_pre_get_page_2_post_get_page = 0;
 
+unsigned long perf_count_direct_reclaim_fast_sum_progress = 0;
+unsigned long perf_count_direct_reclaim_slow_sum_progress = 0;
+unsigned long perf_count_direct_reclaim_fast_sum_order = 0;
+unsigned long perf_count_direct_reclaim_slow_sum_order = 0;
+
 static inline unsigned long long tod_duration_ns(unsigned long long from, unsigned long long to) {
 	return ((to*125)>>9)-((from*125)>>9);
 }
@@ -1719,11 +1724,17 @@
 		perf_count_direct_reclaim_slow_dursum_pre_ttf_2_post_ttf += tod_duration_ns(t1,t2);
 		perf_count_direct_reclaim_slow_dursum_post_ttf_2_pre_get_page += tod_duration_ns(t2,t3);
 		perf_count_direct_reclaim_slow_dursum_pre_get_page_2_post_get_page += tod_duration_ns(t3,t4);
+
+		perf_count_direct_reclaim_slow_sum_progress += *did_some_progress;
+		perf_count_direct_reclaim_slow_sum_order += order;
 	}
 	else {
 		perf_count_direct_reclaim_fast_dursum_pre_ttf_2_post_ttf += tod_duration_ns(t1,t2);
 		perf_count_direct_reclaim_fast_dursum_post_ttf_2_pre_get_page += tod_duration_ns(t2,t3);
 		perf_count_direct_reclaim_fast_dursum_pre_get_page_2_post_get_page += tod_duration_ns(t3,t4);
+
+		perf_count_direct_reclaim_fast_sum_progress += *did_some_progress;
+		perf_count_direct_reclaim_fast_sum_order += order;
 	}
 
 	return page;
#24 lore raw
Quoted 63 lines · Christian Ehrhardt
Quoted 1 line · Mel Gorman

On Mon, Feb 15, 2010 at 04:46:53PM +0100, Christian Ehrhardt wrote:

[...]

Quoted 13 lines
Quoted 13 lines

The differences in asm are pretty much the same, as before rmqueue_bulk was already inlined the actually intended change to its parameters was negligible. I wondered if it would be important if that is a constant value (-1) or if the reason was caused by that shift. So I tried:

23 @@ -965,7 +965,7 @@
24 set_page_private(page, migratetype);
25 list = &page->lru;

 26         }
 27 -       __mod_zone_page_state(zone, NR_FREE_PAGES, -(i << order));
 28 +       __mod_zone_page_state(zone, NR_FREE_PAGES, -i);
 29         spin_unlock(&zone->lock);
 30         return i;

31 }

[...]

Quoted 1 line

It "fixes" it only by not calling direct reclaim when it should :(

yeah as we both realized -1 was not right so it was more a crazy workaround :-)

Anyway after that being a dead end again I dug even deeper into the details of direct_reclaim - I think we can agree that out of the counters we already know the race between try_to_free making progress and get_page not getting a page causing the congestion_wait is source of the issue.

So what I tried to bring some more light into all that was extending my perf counters to track a few more details in direct_reclaim. Two patches are attached and apply after the other three already available in that thread.
The intention is
a) to track the time
a1) spent in try_to_free_pages
a2) consumed after try_to_free_pages until get_page_from_freelist
a3) spent in get_pages_from_freelist
b1) after seeing that order!=0 -> drain_all_pages I wondered if that might differ even all calls look like they have zero b2) tracking the average amount of pages freed by try_to_free_pages for fast path and slow path (progres&!page)

Naming convention (otherwise it might get confusing here)
Good case - the scenario e.g. with e084b and 5f8dcc21 reverted resulting in high throughput and a low ratio of direct_reclaim running into progress&!page Bad case - the scenario e.g. on a clean 2.6.32
Fast path - direct reclaim calls that did not run into progress&!page
Slow path - direct reclaim calls that ran into progress&!page ending up in a long congestion_wait and therefore called "slow" path

Mini summary of what we had before in huge tables:
fast path slow path
GOOD CASE ~98% ~1-3%
BAD CASE ~70% ~30%
-> leading to throughput impact of e.g. 600 mb/s with 16 iozone threads (worse with even more threads)

Out of the numbers I got the following things might help us to create a new approach to a solution. The timings showed that that the so called slow case is actually much faster passing though direct_reclaim in bad case.

GOOD CASE duration
a1 Fast-avg-duration_pre_ttf_2_post_ttf 164099
a2 Fast-avg-duration_post_ttf_2_pre_get_page 459
a3 Fast-avg-duration_pre_get_page_2_post_get_page 346
a1 Slow-avg-duration_pre_ttf_2_post_ttf 127621
a2 Slow-avg-duration_post_ttf_2_pre_get_page 1957
a3 Slow-avg-duration_pre_get_page_2_post_get_page 256

BAD CASE                                         duration   deviation to good case in %
a1 Fast-avg-duration_pre_ttf_2_post_ttf            122921   -25.09%
a2 Fast-avg-duration_post_ttf_2_pre_get_page          521   13.53%
a3 Fast-avg-duration_pre_get_page_2_post_get_page     244   -29.55%
a1 Slow-avg-duration_pre_ttf_2_post_ttf            109740   -14.01%
a2 Slow-avg-duration_post_ttf_2_pre_get_page          250   -87.18%
a3 Slow-avg-duration_pre_get_page_2_post_get_page     117   -54.16%
Quoted 5 lines

That means that in the bad case the execution is much faster. Especially in the case that eventually runs into the slow path try_to_free is 14% faster, more important the time between try_to_free and get_pages is 87%! faster => less than a fifth and finally get_page is 54% faster, but that is probably just failing in an almost empty list which is fast^^.

As I checked order which always was zero the time is not spent in drain_all_pages and the only other thing left might be cond_resched ?!
Everything else are a few assignments so it can't be much else.
But why would e.g. not running into schedule in cond_resched cause get_pages to not find anything - I don't know and I would expect it should be the other way around - the faster you get from free to get the more pages should be left.

THe reason here is probably the the fact that in the bad case a lot of processes are waiting on congestion_wait and are therefore not runnnable and that way not scheduled via cond_resched.

I'll test this theory today or tomorrow with cond_resched in direct_reclaim commented out and expect almost no difference.

Quoted 7 lines

I thought the progress try_to_free_pages is doing might be important as well so I took numbers for that too.
From those I see that the good case as well as the bad case has an average of 62 pages freed in fast path.
But in slow path they start to differ - while the good case that is running only seldom in that path anyway frees an average of 1.46 pages (that might be the thing causing it not getting a page eventually) in the bad case it makes a progress of avg 37 pages even in slow path.

PAGES-FREED  fast path   slow path
GOOD CASE      ~62       ~1.46
BAD CASE       ~62       ~37

5f8dcc21 introduced per migrate type pcp lists, is it possible that we run in a scenario where try_to_free frees a lot of pages via, but of the wrong migrate type? And afterwards get_page
At least try_to_free_page and it's called shrink_ functions is not migrate type aware while get_page and its subsequent buffered_rmqueue and rmqueue_bulk are - btw here comes patch e084b.

I only see buffered_rmqueue chose a specific pcp list based on migrate type, and a fallback to migrate_reserve - is that enough fallback, what if the reserve is empty too but a few other types would not and those other types are the ones filled by try_to_free?

I'll try to get a per migrate type #pages statistic after direct_reclaim reaches !page - maybe that can confirm some parts of my theory.

Quoted 8 lines

Thinking of it as asking "how few pages do I have to free until I fall from fast to slow path" the kernels behave different it looks wrong but interesting. The good case only drops to slow path (!page) in case of ~1.46 pages freed while the bad case seems to enter that much earlier with even 37 pages freed.

As order is always 0 and get_page afaik about getting just "one" page I wonder where these 37 pages disappeared especially as in bad case it is much faster getting to get_pages after freeing those ~37 pages.

Comments and ideas welcome!

Signature · 2 lines
Grüsse / regards, Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization
#25 lore raw
Quoted 83 lines · Christian Ehrhardt
Quoted 64 lines · Christian Ehrhardt
Quoted 1 line · Mel Gorman

On Mon, Feb 15, 2010 at 04:46:53PM +0100, Christian Ehrhardt wrote:

[...]

Quoted 13 lines
Quoted 13 lines

The differences in asm are pretty much the same, as before rmqueue_bulk was already inlined the actually intended change to its parameters was negligible. I wondered if it would be important if that is a constant value (-1) or if the reason was caused by that shift. So I tried:

23 @@ -965,7 +965,7 @@
24 set_page_private(page, migratetype);
25 list = &page->lru;

 26         }
 27 -       __mod_zone_page_state(zone, NR_FREE_PAGES, -(i << order));
 28 +       __mod_zone_page_state(zone, NR_FREE_PAGES, -i);
 29         spin_unlock(&zone->lock);
 30         return i;

31 }

[...]

Quoted 1 line

It "fixes" it only by not calling direct reclaim when it should :(

yeah as we both realized -1 was not right so it was more a crazy workaround :-)

Anyway after that being a dead end again I dug even deeper into the details of direct_reclaim - I think we can agree that out of the counters we already know the race between try_to_free making progress and get_page not getting a page causing the congestion_wait is source of the issue.

So what I tried to bring some more light into all that was extending my perf counters to track a few more details in direct_reclaim. Two patches are attached and apply after the other three already available in that thread.
The intention is
a) to track the time
a1) spent in try_to_free_pages
a2) consumed after try_to_free_pages until get_page_from_freelist
a3) spent in get_pages_from_freelist
b1) after seeing that order!=0 -> drain_all_pages I wondered if that might differ even all calls look like they have zero b2) tracking the average amount of pages freed by try_to_free_pages for fast path and slow path (progres&!page)

Naming convention (otherwise it might get confusing here)
Good case - the scenario e.g. with e084b and 5f8dcc21 reverted resulting in high throughput and a low ratio of direct_reclaim running into progress&!page Bad case - the scenario e.g. on a clean 2.6.32
Fast path - direct reclaim calls that did not run into progress&!page
Slow path - direct reclaim calls that ran into progress&!page ending up in a long congestion_wait and therefore called "slow" path

Mini summary of what we had before in huge tables:
fast path slow path
GOOD CASE ~98% ~1-3%
BAD CASE ~70% ~30%
-> leading to throughput impact of e.g. 600 mb/s with 16 iozone threads (worse with even more threads)

Out of the numbers I got the following things might help us to create a new approach to a solution. The timings showed that that the so called slow case is actually much faster passing though direct_reclaim in bad case.

GOOD CASE duration
a1 Fast-avg-duration_pre_ttf_2_post_ttf 164099
a2 Fast-avg-duration_post_ttf_2_pre_get_page 459
a3 Fast-avg-duration_pre_get_page_2_post_get_page 346
a1 Slow-avg-duration_pre_ttf_2_post_ttf 127621
a2 Slow-avg-duration_post_ttf_2_pre_get_page 1957
a3 Slow-avg-duration_pre_get_page_2_post_get_page 256

BAD CASE                                         duration   deviation to good case in %
a1 Fast-avg-duration_pre_ttf_2_post_ttf            122921   -25.09%
a2 Fast-avg-duration_post_ttf_2_pre_get_page          521   13.53%
a3 Fast-avg-duration_pre_get_page_2_post_get_page     244   -29.55%
a1 Slow-avg-duration_pre_ttf_2_post_ttf            109740   -14.01%
a2 Slow-avg-duration_post_ttf_2_pre_get_page          250   -87.18%
a3 Slow-avg-duration_pre_get_page_2_post_get_page     117   -54.16%
Quoted 5 lines

That means that in the bad case the execution is much faster. Especially in the case that eventually runs into the slow path try_to_free is 14% faster, more important the time between try_to_free and get_pages is 87%! faster => less than a fifth and finally get_page is 54% faster, but that is probably just failing in an almost empty list which is fast^^.

As I checked order which always was zero the time is not spent in drain_all_pages and the only other thing left might be cond_resched ?!
Everything else are a few assignments so it can't be much else.
But why would e.g. not running into schedule in cond_resched cause get_pages to not find anything - I don't know and I would expect it should be the other way around - the faster you get from free to get the more pages should be left.

THe reason here is probably the the fact that in the bad case a lot of processes are waiting on congestion_wait and are therefore not runnnable and that way not scheduled via cond_resched.

I'll test this theory today or tomorrow with cond_resched in direct_reclaim commented out and expect almost no difference.

Quoted 7 lines

I thought the progress try_to_free_pages is doing might be important as well so I took numbers for that too.
From those I see that the good case as well as the bad case has an average of 62 pages freed in fast path.
But in slow path they start to differ - while the good case that is running only seldom in that path anyway frees an average of 1.46 pages (that might be the thing causing it not getting a page eventually) in the bad case it makes a progress of avg 37 pages even in slow path.

PAGES-FREED  fast path   slow path
GOOD CASE      ~62       ~1.46
BAD CASE       ~62       ~37
Quoted 5 lines

5f8dcc21 introduced per migrate type pcp lists, is it possible that we run in a scenario where try_to_free frees a lot of pages via, but of the wrong migrate type? And afterwards get_page
At least try_to_free_page and it's called shrink_ functions is not migrate type aware while get_page and its subsequent buffered_rmqueue and rmqueue_bulk are - btw here comes patch e084b.

I only see buffered_rmqueue chose a specific pcp list based on migrate type, and a fallback to migrate_reserve - is that enough fallback, what if the reserve is empty too but a few other types would not and those other types are the ones filled by try_to_free?

I just saw the full type iteration in __rmqueue_fallback, but still somewhere the average of 37 freed pages need to go so that get_page doesn't get one.

Quoted 1 line

I'll try to get a per migrate type #pages statistic after direct_reclaim reaches !page - maybe that can confirm some parts of my theory.

This might be still worth a try.

Quoted 9 lines
Quoted 8 lines

Thinking of it as asking "how few pages do I have to free until I fall from fast to slow path" the kernels behave different it looks wrong but interesting. The good case only drops to slow path (!page) in case of ~1.46 pages freed while the bad case seems to enter that much earlier with even 37 pages freed.

As order is always 0 and get_page afaik about getting just "one" page I wonder where these 37 pages disappeared especially as in bad case it is much faster getting to get_pages after freeing those ~37 pages.

Comments and ideas welcome!

Signature · 2 lines
Grüsse / regards, Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization
#26 lore raw
In reply to Christian Ehrhardt
Quoted 81 lines · Christian Ehrhardt
Quoted 63 lines · Christian Ehrhardt
Quoted 1 line · Mel Gorman

On Mon, Feb 15, 2010 at 04:46:53PM +0100, Christian Ehrhardt wrote:

[...]

Quoted 13 lines
Quoted 13 lines

The differences in asm are pretty much the same, as before rmqueue_bulk was already inlined the actually intended change to its parameters was negligible. I wondered if it would be important if that is a constant value (-1) or if the reason was caused by that shift. So I tried:

23 @@ -965,7 +965,7 @@
24 set_page_private(page, migratetype);
25 list = &page->lru;

 26         }
 27 -       __mod_zone_page_state(zone, NR_FREE_PAGES, -(i << order));
 28 +       __mod_zone_page_state(zone, NR_FREE_PAGES, -i);
 29         spin_unlock(&zone->lock);
 30         return i;

31 }

[...]

Quoted 1 line

It "fixes" it only by not calling direct reclaim when it should :(

yeah as we both realized -1 was not right so it was more a crazy workaround :-)

Anyway after that being a dead end again I dug even deeper into the details of direct_reclaim - I think we can agree that out of the counters we already know the race between try_to_free making progress and get_page not getting a page causing the congestion_wait is source of the issue.

So what I tried to bring some more light into all that was extending my perf counters to track a few more details in direct_reclaim. Two patches are attached and apply after the other three already available in that thread.
The intention is
a) to track the time
a1) spent in try_to_free_pages
a2) consumed after try_to_free_pages until get_page_from_freelist
a3) spent in get_pages_from_freelist
b1) after seeing that order!=0 -> drain_all_pages I wondered if that might differ even all calls look like they have zero b2) tracking the average amount of pages freed by try_to_free_pages for fast path and slow path (progres&!page)

Naming convention (otherwise it might get confusing here)
Good case - the scenario e.g. with e084b and 5f8dcc21 reverted resulting in high throughput and a low ratio of direct_reclaim running into progress&!page Bad case - the scenario e.g. on a clean 2.6.32
Fast path - direct reclaim calls that did not run into progress&!page
Slow path - direct reclaim calls that ran into progress&!page ending up in a long congestion_wait and therefore called "slow" path

Mini summary of what we had before in huge tables:
fast path slow path
GOOD CASE ~98% ~1-3%
BAD CASE ~70% ~30%
-> leading to throughput impact of e.g. 600 mb/s with 16 iozone threads (worse with even more threads)

Out of the numbers I got the following things might help us to create a new approach to a solution. The timings showed that that the so called slow case is actually much faster passing though direct_reclaim in bad case.

GOOD CASE duration
a1 Fast-avg-duration_pre_ttf_2_post_ttf 164099
a2 Fast-avg-duration_post_ttf_2_pre_get_page 459
a3 Fast-avg-duration_pre_get_page_2_post_get_page 346
a1 Slow-avg-duration_pre_ttf_2_post_ttf 127621
a2 Slow-avg-duration_post_ttf_2_pre_get_page 1957
a3 Slow-avg-duration_pre_get_page_2_post_get_page 256

BAD CASE                                         duration   deviation to good case in %
a1 Fast-avg-duration_pre_ttf_2_post_ttf            122921   -25.09%
a2 Fast-avg-duration_post_ttf_2_pre_get_page          521   13.53%
a3 Fast-avg-duration_pre_get_page_2_post_get_page     244   -29.55%
a1 Slow-avg-duration_pre_ttf_2_post_ttf            109740   -14.01%
a2 Slow-avg-duration_post_ttf_2_pre_get_page          250   -87.18%
a3 Slow-avg-duration_pre_get_page_2_post_get_page     117   -54.16%
Quoted 5 lines

That means that in the bad case the execution is much faster. Especially in the case that eventually runs into the slow path try_to_free is 14% faster, more important the time between try_to_free and get_pages is 87%! faster => less than a fifth and finally get_page is 54% faster, but that is probably just failing in an almost empty list which is fast^^.

As I checked order which always was zero the time is not spent in drain_all_pages and the only other thing left might be cond_resched ?!
Everything else are a few assignments so it can't be much else.
But why would e.g. not running into schedule in cond_resched cause get_pages to not find anything - I don't know and I would expect it should be the other way around - the faster you get from free to get the more pages should be left.

THe reason here is probably the the fact that in the bad case a lot of processes are waiting on congestion_wait and are therefore not runnnable and that way not scheduled via cond_resched.

I'll test this theory today or tomorrow with cond_resched in direct_reclaim commented out and expect almost no difference.

Quoted 4 lines

I thought the progress try_to_free_pages is doing might be important as well so I took numbers for that too.
From those I see that the good case as well as the bad case has an average of 62 pages freed in fast path.
But in slow path they start to differ - while the good case that is running only seldom in that path anyway frees an average of 1.46 pages (that might be the thing causing it not getting a page eventually) in the bad case it makes a progress of avg 37 pages even in slow path.

Quoted 7 lines
Quoted 3 lines
PAGES-FREED  fast path   slow path
GOOD CASE      ~62       ~1.46
BAD CASE       ~62       ~37

5f8dcc21 introduced per migrate type pcp lists, is it possible that we run in a scenario where try_to_free frees a lot of pages via, but of the wrong migrate type?

It's possible but the window is small. When a threshold is reached on the PCP lists, they get drained to the buddy lists and later picked up again by __rmqueue_fallback(). I had considered the possibility of pages of the wrong type being on the PCP lists which led to the patch "page-allocator:
Fallback to other per-cpu lists when the target list is empty and memory is low" but you reported it made no difference even when fallback was allowed with high watermarks.

Quoted 4 lines

And afterwards get_page
At least try_to_free_page and it's called shrink_ functions is not migrate type aware while get_page and its subsequent buffered_rmqueue and rmqueue_bulk are -

Unless a large number of pages are pinned on those PCP lists (which doesn't appear to be the case or the fallback patch would have helped), the pages of other migratetypes are found by __rmqueue_fallback().

Quoted 2 lines

btw here comes patch e084b.

Which still has zero explanation.

Quoted 5 lines

I only see buffered_rmqueue chose a specific pcp list based on migrate type, and a fallback to migrate_reserve - is that enough fallback, what if the reserve is empty too but a few other types would not and those other types are the ones filled by try_to_free?

They get found by __rmqueue_fallback().

Quoted 9 lines

I'll try to get a per migrate type #pages statistic after direct_reclaim reaches !page - maybe that can confirm some parts of my theory.

Quoted 6 lines

Thinking of it as asking "how few pages do I have to free until I fall from fast to slow path" the kernels behave different it looks wrong but interesting. The good case only drops to slow path (!page) in case of ~1.46 pages freed while the bad case seems to enter that much earlier with even 37 pages freed.

As order is always 0 and get_page afaik about getting just "one" page I wonder where these 37 pages disappeared especially as in bad case it is much faster getting to get_pages after freeing those ~37 pages.

Comments and ideas welcome!

Signature · 3 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
#27 lore raw
Quoted 82 lines · Mel Gorman
Quoted 72 lines · Christian Ehrhardt
Quoted 62 lines · Christian Ehrhardt
Quoted 1 line · Mel Gorman

On Mon, Feb 15, 2010 at 04:46:53PM +0100, Christian Ehrhardt wrote:

[...]

Quoted 13 lines
Quoted 13 lines

The differences in asm are pretty much the same, as before rmqueue_bulk was already inlined the actually intended change to its parameters was negligible. I wondered if it would be important if that is a constant value (-1) or if the reason was caused by that shift. So I tried:

23 @@ -965,7 +965,7 @@
24 set_page_private(page, migratetype);
25 list = &page->lru;

 26         }
 27 -       __mod_zone_page_state(zone, NR_FREE_PAGES, -(i << order));
 28 +       __mod_zone_page_state(zone, NR_FREE_PAGES, -i);
 29         spin_unlock(&zone->lock);
 30         return i;

31 }

[...]

Quoted 1 line

It "fixes" it only by not calling direct reclaim when it should :(

yeah as we both realized -1 was not right so it was more a crazy workaround :-)

Anyway after that being a dead end again I dug even deeper into the details of direct_reclaim - I think we can agree that out of the counters we already know the race between try_to_free making progress and get_page not getting a page causing the congestion_wait is source of the issue.

So what I tried to bring some more light into all that was extending my perf counters to track a few more details in direct_reclaim. Two patches are attached and apply after the other three already available in that thread.
The intention is
a) to track the time
a1) spent in try_to_free_pages
a2) consumed after try_to_free_pages until get_page_from_freelist
a3) spent in get_pages_from_freelist
b1) after seeing that order!=0 -> drain_all_pages I wondered if that might differ even all calls look like they have zero b2) tracking the average amount of pages freed by try_to_free_pages for fast path and slow path (progres&!page)

Naming convention (otherwise it might get confusing here)
Good case - the scenario e.g. with e084b and 5f8dcc21 reverted resulting in high throughput and a low ratio of direct_reclaim running into progress&!page Bad case - the scenario e.g. on a clean 2.6.32
Fast path - direct reclaim calls that did not run into progress&!page
Slow path - direct reclaim calls that ran into progress&!page ending up in a long congestion_wait and therefore called "slow" path

Mini summary of what we had before in huge tables:
fast path slow path
GOOD CASE ~98% ~1-3%
BAD CASE ~70% ~30%
-> leading to throughput impact of e.g. 600 mb/s with 16 iozone threads (worse with even more threads)

Out of the numbers I got the following things might help us to create a new approach to a solution. The timings showed that that the so called slow case is actually much faster passing though direct_reclaim in bad case.

GOOD CASE duration
a1 Fast-avg-duration_pre_ttf_2_post_ttf 164099
a2 Fast-avg-duration_post_ttf_2_pre_get_page 459
a3 Fast-avg-duration_pre_get_page_2_post_get_page 346
a1 Slow-avg-duration_pre_ttf_2_post_ttf 127621
a2 Slow-avg-duration_post_ttf_2_pre_get_page 1957
a3 Slow-avg-duration_pre_get_page_2_post_get_page 256

BAD CASE                                         duration   deviation to good case in %
a1 Fast-avg-duration_pre_ttf_2_post_ttf            122921   -25.09%
a2 Fast-avg-duration_post_ttf_2_pre_get_page          521   13.53%
a3 Fast-avg-duration_pre_get_page_2_post_get_page     244   -29.55%
a1 Slow-avg-duration_pre_ttf_2_post_ttf            109740   -14.01%
a2 Slow-avg-duration_post_ttf_2_pre_get_page          250   -87.18%
a3 Slow-avg-duration_pre_get_page_2_post_get_page     117   -54.16%
Quoted 5 lines

That means that in the bad case the execution is much faster. Especially in the case that eventually runs into the slow path try_to_free is 14% faster, more important the time between try_to_free and get_pages is 87%! faster => less than a fifth and finally get_page is 54% faster, but that is probably just failing in an almost empty list which is fast^^.

As I checked order which always was zero the time is not spent in drain_all_pages and the only other thing left might be cond_resched ?!
Everything else are a few assignments so it can't be much else.
But why would e.g. not running into schedule in cond_resched cause get_pages to not find anything - I don't know and I would expect it should be the other way around - the faster you get from free to get the more pages should be left.

THe reason here is probably the the fact that in the bad case a lot of processes are waiting on congestion_wait and are therefore not runnnable and that way not scheduled via cond_resched.

I'll test this theory today or tomorrow with cond_resched in direct_reclaim commented out and expect almost no difference.

Quoted 8 lines
Quoted 7 lines

I thought the progress try_to_free_pages is doing might be important as well so I took numbers for that too.
From those I see that the good case as well as the bad case has an average of 62 pages freed in fast path.
But in slow path they start to differ - while the good case that is running only seldom in that path anyway frees an average of 1.46 pages (that might be the thing causing it not getting a page eventually) in the bad case it makes a progress of avg 37 pages even in slow path.

PAGES-FREED  fast path   slow path
GOOD CASE      ~62       ~1.46
BAD CASE       ~62       ~37

5f8dcc21 introduced per migrate type pcp lists, is it possible that we run in a scenario where try_to_free frees a lot of pages via, but of the wrong migrate type?

Quoted 8 lines

It's possible but the window is small. When a threshold is reached on the PCP lists, they get drained to the buddy lists and later picked up again by __rmqueue_fallback(). I had considered the possibility of pages of the wrong type being on the PCP lists which led to the patch "page-allocator:
Fallback to other per-cpu lists when the target list is empty and memory is low" but you reported it made no difference even when fallback was allowed with high watermarks.
[...]

Today I created rather huge debug logs - I'll spare everyone with too much detail.
Eventually it comes down to some kind of cat /proc/zoneinfo like output extended to list all things per migrate type too.

From that I still think there should be plenty of pages to get the allocation through, as it was suggested by the high amount of pages freed by try_to_free.

Due to that I tried calling a second get_page_from_freelist after the first failing one and after the statistic output which showed me there wer enough pages available. Even that second one failed - but this is good, that allows me to run that second call with a kind of "debugthis" parameter which I'll run tomorrow.

If anyone wants the full 6.2M of that data let me know, but I think the step by step debug of the second try will give us much smaller and better insights to discuss about.

More about that tomorrow,
Christian

P.S. Just one output of it here from a 16 thread read case:
### extended zoneinfo after failed direct reclaim ###
Progress just made in try_to_free 32
Current allocation order 0
Current allocation gfp_flags 201da GFP_COLD '1'
Current allocation migrate type '2'
Current cpu id 0
zone DMA
zni: pages_free 503

        min                       508
        low                       635
        high                      762
        scanned                    96
        spanned                 65536
        present                 64640
    nr_free_pages                 503
    nr_inactive_anon             1536
    nr_active_anon               1480
    nr_inactive_file            33197
    nr_active_file               9492
    nr_unevictable               3016
    nr_mlock                     3016
    nr_anon_pages                5188
    nr_mapped                    2295
    nr_file_pages               43714
    nr_dirty                      206
    nr_writeback                    0
    nr_slab_reclaimable          2762
    nr_slab_unreclaimable        5368
    nr_page_table_pages           151
    nr_kernel_stack               215
    nr_unstable                     0
    nr_bounce                       0
    nr_vmscan_write               476
    nr_writeback_temp               0
    nr_isolated_anon                0
    nr_isolated_file               32
    nr_shmem                       62
        protection: (0,0,0)

Free areas:
Order 0 - nr_free 198 (matching 198 order 0 pages)
MIGRATE_UNMOVABLE 198
MIGRATE_RECLAIMABLE 0
d get_page_from_freelist fail

   MIGRATE_MOVABLE              0*
   MIGRATE_RESERVE              0
   MIGRATE_ISOLATE              0
Order  1 - nr_free          2 (matching          4 order 0 pages)
   MIGRATE_UNMOVABLE            1
   MIGRATE_RECLAIMABLE          0
   MIGRATE_MOVABLE              1*
   MIGRATE_RESERVE              0
   MIGRATE_ISOLATE              0
Order  2 - nr_free          0 (matching          0 order 0 pages)
   MIGRATE_UNMOVABLE            0
   MIGRATE_RECLAIMABLE          0
   MIGRATE_MOVABLE              0*
   MIGRATE_RESERVE              0
   MIGRATE_ISOLATE              0
Order  3 - nr_free          0 (matching          0 order 0 pages)
   MIGRATE_UNMOVABLE            0
   MIGRATE_RECLAIMABLE          0
   MIGRATE_MOVABLE              0*
   MIGRATE_RESERVE              0
   MIGRATE_ISOLATE              0
Order  4 - nr_free          0 (matching          0 order 0 pages)
   MIGRATE_UNMOVABLE            0
   MIGRATE_RECLAIMABLE          0
   MIGRATE_MOVABLE              0*
   MIGRATE_RESERVE              0
   MIGRATE_ISOLATE              0
Order  5 - nr_free          0 (matching          0 order 0 pages)
   MIGRATE_UNMOVABLE            0
   MIGRATE_RECLAIMABLE          0
   MIGRATE_MOVABLE              0*
   MIGRATE_RESERVE              0
   MIGRATE_ISOLATE              0
Order  6 - nr_free          0 (matching          0 order 0 pages)
   MIGRATE_UNMOVABLE            0
   MIGRATE_RECLAIMABLE          0
   MIGRATE_MOVABLE              0*
   MIGRATE_RESERVE              0
   MIGRATE_ISOLATE              0
Order  7 - nr_free          0 (matching          0 order 0 pages)
   MIGRATE_UNMOVABLE            0
   MIGRATE_RECLAIMABLE          0
   MIGRATE_MOVABLE              0*
   MIGRATE_RESERVE              0
   MIGRATE_ISOLATE              0
Order  8 - nr_free          1 (matching        256 order 0 pages)
   MIGRATE_UNMOVABLE            0
   MIGRATE_RECLAIMABLE          0
   MIGRATE_MOVABLE              0*

d get_page_from_freelist fail
MIGRATE_RESERVE 1
MIGRATE_ISOLATE 0
pagesets

    cpu:  0 (current cpu)
          count:         46
          high:          90
          batch:         15
          MIGRATE_UNMOVABLE            0
          MIGRATE_RECLAIMABLE          0
          MIGRATE_MOVABLE             46*
          vm_stats_threshold:         16
    cpu:  1
          count:         77
          high:          90
          batch:         15
          MIGRATE_UNMOVABLE            0
          MIGRATE_RECLAIMABLE          0
          MIGRATE_MOVABLE             77*
          vm_stats_threshold:         16
    cpu:  2
          count:         10
          high:          90
          batch:         15
          MIGRATE_UNMOVABLE            0
          MIGRATE_RECLAIMABLE          0
          MIGRATE_MOVABLE             10*
          vm_stats_threshold:         16
    cpu:  3
          count:          8
          high:          90
          batch:         15
          MIGRATE_UNMOVABLE            0
          MIGRATE_RECLAIMABLE          0
          MIGRATE_MOVABLE              8*
          vm_stats_threshold:         16
    cpu:  4
          count:         11
          high:          90
          batch:         15
          MIGRATE_UNMOVABLE            0
          MIGRATE_RECLAIMABLE          0
          MIGRATE_MOVABLE             11*
          vm_stats_threshold:         16
    cpu:  5
          count:         74
          high:          90
          batch:         15
          MIGRATE_UNMOVABLE            0
          MIGRATE_RECLAIMABLE          0
          MIGRATE_MOVABLE             74*
          vm_stats_threshold:         16
    cpu:  6
          count:         65
          high:          90
          batch:         15
          MIGRATE_UNMOVABLE            0
          MIGRATE_RECLAIMABLE          0
          MIGRATE_MOVABLE             65*
          vm_stats_threshold:         16
    cpu:  7
          count:          0
          high:          90
          batch:         15
          MIGRATE_UNMOVABLE            0
          MIGRATE_RECLAIMABLE          0
          MIGRATE_MOVABLE              0*
          vm_stats_threshold:         16

all_unreclaimable: 0
prev_priority: 10
start_pfn: 0
inactive_ratio: 1

Signature · 2 lines
Grüsse / regards, Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization
#28 lore raw
Quoted 7 lines · Christian Ehrhardt
Quoted 4 lines · Mel Gorman
Quoted 3 lines · Christian Ehrhardt
Quoted 2 lines · Christian Ehrhardt
Quoted 1 line · Mel Gorman

On Mon, Feb 15, 2010 at 04:46:53PM +0100, Christian Ehrhardt wrote:

[...]

Quoted 28 lines
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
PAGES-FREED  fast path   slow path
GOOD CASE      ~62       ~1.46
BAD CASE       ~62       ~37

5f8dcc21 introduced per migrate type pcp lists, is it possible that we run in a scenario where try_to_free frees a lot of pages via, but of the wrong migrate type?

Quoted 10 lines

It's possible but the window is small. When a threshold is reached on the PCP lists, they get drained to the buddy lists and later picked up again by __rmqueue_fallback(). I had considered the possibility of pages of the wrong type being on the PCP lists which led to the patch "page-allocator:
Fallback to other per-cpu lists when the target list is empty and memory is
low" but you reported it made no difference even when fallback was allowed
with high watermarks.
[...]

Today I created rather huge debug logs - I'll spare everyone with too much detail.
Eventually it comes down to some kind of cat /proc/zoneinfo like output extended to list all things per migrate type too.

From that I still think there should be plenty of pages to get the
allocation through, as it was suggested by the high amount of pages freed by try_to_free.

Quoted 1 line

[...]

Quoted 1 line

More about that tomorrow,

Well tomorrow is now, and I think I got some important new insights.

As mentioned before I realized that a second call still fails most of the time (>99%). Therefore I added a "debugme" parameter to get_page_from_freelist and buffered_rmqueue to see where the allocations exactly fails (patch attached).

Now with debugme active in a second call after it had progress&&!page in direct_reclaim I saw the following repeating pattern in most of the cases:
get_page_from_freelist - zone loop - current zone 'DMA'
get_page_from_freelist - watermark check due to !(alloc_flags & ALLOC_NO_WATERMARKS)
get_page_from_freelist - goto zone_full due to zone_reclaim_mode==0
get_page_from_freelist - return page '(null)'

Ok - now we at least exactly know why it gets no page.
Remember there are plenty of pages like it was in my zoneinfo like report in the last mail. I didn't expect that, but actually watermarks are what stops the allocations here. The zone_watermark_ok check reports that there are not enough pages for the current watermark and finally it ends with zone_reclaim_mode which is always zero on s390 as we are not CONFIG_NUMA.

Ok remember my scenario - several parallel iozone sequential read processes - theres not much allocation going on except for the page cache read ahead related to that read workload.
The allocations for page cache seem to have no special watermarks selected via their GFP flags and therefore get stalled by congestion_wait - which in consequence of no available writes in flight consumes its full timeout.

As I see significant impacts to the iozone throughput and not only e.g. bad counters in direct_reclaim the congestion_wait stalling seems to be so often/long to stall the application I/O itself. That means from the time VFS starting a read ahead it seems to stall that page allocation long enough that the data is not ready when the application tries to read it, while it would be if the allocation would be fast enough.

A simple test for this theory was to allow those failing allocations a second chance without watermark restrictions before putting them to sleep for such a long time.

mm/page_alloc.c 2010-02-19 09:56:26.000000000 +0100 renamed from mm/page_alloc.c 2010-02-19 09:53:14.000000000 +0100 +16 -1
Index: linux-2.6-git/mm/page_alloc.c
===================================================================
@@ -1954,7 +1954,22 @@
 
        if (should_alloc_retry(gfp_mask, order, pages_reclaimed)) {
                /* Wait for some write requests to complete then retry */
-               congestion_wait(BLK_RW_ASYNC, HZ/50);
+               /*
+                * If it gets here try it without watermarks before going
+                * to sleep.
+                *
+                * This will end up in alloc_high_priority and if that fails
+                * once more direct_reclaim but this time without watermark
+                * checks.
+                *
+                * FIXME: that is just for verification - a real fix needs to
+                * ensure e.g. page cache allocations don't drain all pages
+                * under watermark
+                */
+               if (!(alloc_flags & ALLOC_NO_WATERMARKS))
+                       alloc_flags &= ALLOC_NO_WATERMARKS;
+               else
+                       congestion_wait(BLK_RW_ASYNC, HZ/50);
                goto rebalance;
        }
 

This fixes all issues I have, but as stated in the FIXME it is unfortunately no fix for the real world.

Unfortunately even now knowing the place of the issue so well I don't see the connection to the commits e084b+5f8dcc21 - I couldn't find something but did they change the accounting somewhere or e.g. changed the timing/order of watermark updates and allocations?

Eventually it might come down to a discussion of allocation priorities and we might even keep them as is and accept this issue - I still would prefer a good second chance implementation, other page cache allocation flags or something else that explicitly solves this issue. Mel's patch that replaces congestion_wait with a wait for the zone watermarks becoming available again is definitely a step in the right direction and should go into upstream and the long term support branches.

Signature · 2 lines
Grüsse / regards, Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization 
mm/page_alloc.c +43 -17

176 lines not on this page · raw message

#29 lore raw
Quoted 45 lines · Christian Ehrhardt
Quoted 28 lines
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
PAGES-FREED  fast path   slow path
GOOD CASE      ~62       ~1.46
BAD CASE       ~62       ~37

5f8dcc21 introduced per migrate type pcp lists, is it possible that we run in a scenario where try_to_free frees a lot of pages via, but of the wrong migrate type?

Quoted 10 lines

It's possible but the window is small. When a threshold is reached on the PCP lists, they get drained to the buddy lists and later picked up again by __rmqueue_fallback(). I had considered the possibility of pages of the wrong type being on the PCP lists which led to the patch "page-allocator:
Fallback to other per-cpu lists when the target list is empty and memory is
low" but you reported it made no difference even when fallback was allowed
with high watermarks.
[...]

Today I created rather huge debug logs - I'll spare everyone with too much detail.
Eventually it comes down to some kind of cat /proc/zoneinfo like output extended to list all things per migrate type too.

From that I still think there should be plenty of pages to get the
allocation through, as it was suggested by the high amount of pages freed by try_to_free.

Quoted 1 line

[...]

Quoted 1 line

More about that tomorrow,

Well tomorrow is now, and I think I got some important new insights.

As mentioned before I realized that a second call still fails most of the time (>99%). Therefore I added a "debugme" parameter to get_page_from_freelist and buffered_rmqueue to see where the allocations exactly fails (patch attached).

Now with debugme active in a second call after it had progress&&!page in direct_reclaim I saw the following repeating pattern in most of the cases:
get_page_from_freelist - zone loop - current zone 'DMA'
get_page_from_freelist - watermark check due to !(alloc_flags & ALLOC_NO_WATERMARKS)
get_page_from_freelist - goto zone_full due to zone_reclaim_mode==0
get_page_from_freelist - return page '(null)'

Quoted 3 lines

Ok - now we at least exactly know why it gets no page.
Remember there are plenty of pages like it was in my zoneinfo like report in the last mail. I didn't expect that, but actually watermarks are what stops the allocations here.

That is somewhat expected. We also don't want to go underneath them beause that can lead to system deadlock.

Quoted 10 lines

The zone_watermark_ok check reports that there are not enough pages for the current watermark and finally it ends with zone_reclaim_mode which is always zero on s390 as we are not CONFIG_NUMA.

Ok remember my scenario - several parallel iozone sequential read processes
- theres not much allocation going on except for the page cache read ahead related to that read workload.
The allocations for page cache seem to have no special watermarks selected via their GFP flags and therefore get stalled by congestion_wait - which in consequence of no available writes in flight consumes its full timeout.

Which I'd expect to some extent, but it still stuns me that e084b makes a difference to any of this. The one-list-per-migratetype patch would make some sense except restoring something similar to the old behaviour didn't help either.

Quoted 41 lines

As I see significant impacts to the iozone throughput and not only e.g. bad counters in direct_reclaim the congestion_wait stalling seems to be so often/long to stall the application I/O itself.

That means from the time VFS starting a read ahead it seems to stall that page allocation long enough that the data is not ready when the application tries to read it, while it would be if the allocation would be fast enough.

A simple test for this theory was to allow those failing allocations a second chance without watermark restrictions before putting them to sleep for such a long time.

Index: linux-2.6-git/mm/page_alloc.c
===================================================================

mm/page_alloc.c 2010-02-19 09:56:26.000000000 +0100 renamed from mm/page_alloc.c 2010-02-19 09:53:14.000000000 +0100 +16 -1
@@ -1954,7 +1954,22 @@
 
        if (should_alloc_retry(gfp_mask, order, pages_reclaimed)) {
                /* Wait for some write requests to complete then retry */
-               congestion_wait(BLK_RW_ASYNC, HZ/50);
+               /*
+                * If it gets here try it without watermarks before going
+                * to sleep.
+                *
+                * This will end up in alloc_high_priority and if that fails
+                * once more direct_reclaim but this time without watermark
+                * checks.
+                *
+                * FIXME: that is just for verification - a real fix needs to
+                * ensure e.g. page cache allocations don't drain all pages
+                * under watermark
+                */
+               if (!(alloc_flags & ALLOC_NO_WATERMARKS))
+                       alloc_flags &= ALLOC_NO_WATERMARKS;
+               else
+                       congestion_wait(BLK_RW_ASYNC, HZ/50);
                goto rebalance;
        }
 
Quoted 2 lines

This fixes all issues I have, but as stated in the FIXME it is unfortunately no fix for the real world.

It's possible to deadlock a system with this patch. It's also not acting as you intended. I think you meant either |= or = there but anyway.

Quoted 2 lines

Unfortunately even now knowing the place of the issue so well I don't see the connection to the commits e084b+5f8dcc21

Still a mystery.

Quoted 4 lines

- I couldn't find something but
did they change the accounting somewhere or e.g. changed the timing/order of watermark updates and allocations?

Not that I can think of.

Quoted 5 lines

Eventually it might come down to a discussion of allocation priorities and we might even keep them as is and accept this issue - I still would prefer a good second chance implementation, other page cache allocation flags or something else that explicitly solves this issue.

In that line, the patch that replaced congestion_wait() with a waitqueue makes some sense.

Quoted 3 lines

Mel's patch that replaces congestion_wait with a wait for the zone watermarks becoming available again is definitely a step in the right direction and should go into upstream and the long term support branches.

I'll need to do a number of tests before I can move that upstream but I don't think it's a merge candidate. Unfortunately, I'll be offline for a week starting tomorrow so I won't be able to do the testing.

When I get back, I'll revisit those patches with the view to pushing them upstream. I hate to treat symptoms here without knowing the underlying problem but this has been spinning in circles for ages with little forward progress :(

Signature · 3 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
#30 lore raw

Mel Gorman wrote:
[...]

Quoted 21 lines
Quoted 3 lines

Unfortunately even now knowing the place of the issue so well I don't see the connection to the commits e084b+5f8dcc21

Still a mystery.

Quoted 4 lines

- I couldn't find something but
did they change the accounting somewhere or e.g. changed the timing/order of watermark updates and allocations?

Not that I can think of.

Quoted 5 lines

Eventually it might come down to a discussion of allocation priorities and we might even keep them as is and accept this issue - I still would prefer a good second chance implementation, other page cache allocation flags or something else that explicitly solves this issue.

In that line, the patch that replaced congestion_wait() with a waitqueue makes some sense.

[...]

Quoted 8 lines

I'll need to do a number of tests before I can move that upstream but I don't think it's a merge candidate. Unfortunately, I'll be offline for a week starting tomorrow so I won't be able to do the testing.

When I get back, I'll revisit those patches with the view to pushing them upstream. I hate to treat symptoms here without knowing the underlying problem but this has been spinning in circles for ages with little forward progress :(

I'll continue with some debugging in search for the real reasons, but if I can't find a new way to look at it I think we have to drop it for now.

While I hate fixing symptoms too, I completely agree that it is time to bring this fix upstream and in fact into the stable kernel too, to have at least a ~98% workaround out there.

I'm looking forward for your revised patch after you are back and I'm eager to test this one again.

Signature · 2 lines
Grüsse / regards, Christian Ehrhardt
IBM Linux Technology Center, System z Linux Performance
#31 lore raw
Quoted 15 lines · Christian Ehrhardt

Mel Gorman wrote:
[...]

Quoted 8 lines

I'll need to do a number of tests before I can move that upstream but I don't think it's a merge candidate. Unfortunately, I'll be offline for a week starting tomorrow so I won't be able to do the testing.

When I get back, I'll revisit those patches with the view to pushing them upstream. I hate to treat symptoms here without knowing the underlying problem but this has been spinning in circles for ages with little forward progress :(

I'll continue with some debugging in search for the real reasons, but if I can't find a new way to look at it I think we have to drop it for now.

[...]

Quoted 1 line

As a last try I partially rewrote my debug patches which now report what I call "extended zone info" (like proc/zoneinfo plus free area and per migrate type counters) once every second at a random direct_reclaim call.

Naming: As before I call the plain 2.6.32 orig or "bad case" and 2.6.32 with e084b and 5f8dcc21 reverted Rev or "good case". Depending on the fact if a allocation failed or not before the statistics were reported I call them "failed" or "worked" Therefore I splitted the resulting data into the four cases orig-failed, orig-worked, Rev-failed, Rev-worked.

This could again end up in a report that most people expected (like the stopped by watermark last time), but still I think it is worth to report and have everyone take a look at it.

PRE)
First and probably most important to keep it in mind later on - the good case seems to have more pages free, living usually above the watermark and is therefore not running into that failed direct_reclaim allocations. The question for all facts I describe below is "can this affect the number of free pages either directly or be indirectly a pointer what else is going on affecting the # free pages"

As notice for all data below the page cache allocations that occur when running the read workload are GFCP_COLD=1 and preferred migration type MOVABLE.

1) Free page distribution per order in free areas lists
These numbers cover migrate type distribution across free areas per order, similar to what /proc/pagetypeinfo reports.

There is a major difference between the plain 2.6.32 and the one with
e084b and 5f8dcc21 reverted. While the good case shows at least some distribution having a few elements in order 2-7 the bad case looks quite different. Bad case has a huge peak in order 0, is about even on order 1 and then much fewer in order 2-7. Eventually both cases have one order 8 page as reserve at all times.

Pages per Order      0      1      2      3      4     5      6       7    8
Bad Case        272.85  22.10   2.43   0.51   0.14  0.01   0.01    0.06    1
Good Case        97.55  15.29   3.08   1.39   0.77  0.11   0.29    0.90    1

This might not look much but factorized down to 4k order 0 pages this numbers would look like:

4kPages per Order 0 1 2 3 4 5 6 7 8

Bad Case        272.85  44.21   9.73   4.08   2.23  0.23   0.45    7.62  256
Good Case        97.55  30.58  12.30  11.12  12.24  3.64  18.67  114.91  256

So something seems to allow grouping into higher orders much better in the good case.

I wonder if there might be code doing things like this somewhere:
if (couldIcollapsethisintohigherorder)
free
else
donothing
-> leaving less free and less higher order pages in the bad case.

Remember my introduction - what we eventually search is why the bad case has fewer free pages.

3) Migrate types on free areas
Looking at the numbers above more in detail, meaning "splitted into the different migrate types" shows another difference.
The bad case has most of the pages as unmovable, interestingly almost exactly that amount of pages that are shifted from higher orders to order 0 when comparing good/bad case.
So this might be related to that different order distribution we see above. (BTW - on s390 all memory 0-2Gb is one zone, as we have 256m in these tests all is in one zone)

BAD CASE

Free pgs per migrate type @ order     0     1     2     3     4     5     6     7     8
MIGRATE_UNMOVABLE                178.17  0.38  0.00  0.00  0.00  0.00  0.00  0.00     0
MIGRATE_RECLAIMABLE               12.95  0.58  0.01  0.00  0.00  0.00  0.00  0.00     0
MIGRATE_MOVABLE                   81.74 21.14  2.29  0.50  0.13  0.00  0.00  0.00     0
MIGRATE_RESERVE                    0.00  0.00  0.13  0.01  0.01  0.01  0.01  0.06     1

GOOD CASE
Free pgs per migrate type @ order 0 1 2 3 4 5 6 7 8

Normal	MIGRATE_UNMOVABLE         21.70  0.14  0.00  0.00  0.00  0.00  0.00  0.00  0.00
Normal	MIGRATE_RECLAIMABLE        4.15  0.22  0.00  0.00  0.00  0.00  0.00  0.00  0.00
Normal	MIGRATE_MOVABLE           68.71 12.38  0.88  0.63  0.06  0.00  0.00  0.00  0.00
Normal	MIGRATE_RESERVE            2.99  2.56  2.19  0.77  0.71  0.11  0.29  0.90  1.00
Normal	MIGRATE_ISOLATE            0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00

Maybe this gives someone a good hint why we see that different grouping or even why we have less free pages in bad case.

4) PCP list fill ratio
Finally the last major difference I see is the fill ratio of the pcp lists. The good case has an average of ~TODO pages on the pcp lists while the bad case has only ~TODO pages.

AVG count on pcp lists
bad case 35.33
good case 62.46

When looking at the migrate types at the pcp lists (which is only possible without 5f8dcc21 reverted) it looks like "there the movable ones have gone", which they might have not before migrate differentiation type on pcp list support.

AVG count per migrate type in bad case
MIGRATE_UNMOVABLE 12.57
MIGRATE_RECLAIMABLE 2.03
MIGRATE_MOVABLE 31.89

Is it possible that with 5f8dcc21 the MIGRATE_MOVABLE pages are drained from free areas to the pcp list more agressive and leave MIGRATE_UNMOVABLE which then might e.g. not groupable or something like that - and eventually that way somehow leave fewer free pages left ?

FIN)
So, thats it from my side.
I look forward to the finalized congestion_wait->zone wait patch however this turns out (zone wait is resonable if fixing this symptom or not in my opinion).
But still I have a small amount of hope left that all the data I found here might give someone the kick to see whats going on in mm's backstage due to that patches.

Signature · 2 lines
Grüsse / regards, Christian Ehrhardt
IBM Linux Technology Center, System z Linux Performance
#32 lore raw
Quoted 9 lines · Christian Ehrhardt

Christian Ehrhardt wrote:
FIN)
So, thats it from my side.
I look forward to the finalized congestion_wait->zone wait patch however this turns out (zone wait is resonable if fixing this symptom or not in my opinion).
But still I have a small amount of hope left that all the data I found here might give someone the kick to see whats going on in mm's backstage due to that patches.

Hi Christian,

Thanks for doing all this work. I've been looking at a couple of other regressions while it seemed like you guys were making progress. I can't seem to reproduce it on my x86-64 system (was anybody able to reproduce it on x86?)

Anyway I have asked our mainframe guy whether we can set up an environment.

I agree it is a real problem that really needs to be properly explained.

Thanks,
Nick

#33 lore raw
In reply to Mel Gorman
Quoted 14 lines · Mel Gorman
Quoted 5 lines · Christian Ehrhardt

Eventually it might come down to a discussion of allocation priorities and we might even keep them as is and accept this issue - I still would prefer a good second chance implementation, other page cache allocation flags or something else that explicitly solves this issue.

In that line, the patch that replaced congestion_wait() with a waitqueue makes some sense.

Quoted 3 lines

Mel's patch that replaces congestion_wait with a wait for the zone watermarks becoming available again is definitely a step in the right direction and should go into upstream and the long term support branches.

Quoted 8 lines

I'll need to do a number of tests before I can move that upstream but I don't think it's a merge candidate. Unfortunately, I'll be offline for a week starting tomorrow so I won't be able to do the testing.

When I get back, I'll revisit those patches with the view to pushing them upstream. I hate to treat symptoms here without knowing the underlying problem but this has been spinning in circles for ages with little forward progress :(

The zone pressure waitqueue patch makes sense. We may even want to make it more strictly FIFO (eg. check upfront if there are waiters on the queue before allocating a page, and if yes then add ourself to the back of the waitqueue). And also possibly even look at doing the wakeups in the page-freeing path. Although that might start adding too much overhead, so it's quite possible your sloppy-but-lighter timeout approach is preferable.

#34 lore raw
Quoted 19 lines · Nick Piggin
Quoted 18 lines · Mel Gorman
Quoted 5 lines · Christian Ehrhardt

Eventually it might come down to a discussion of allocation priorities and we might even keep them as is and accept this issue - I still would prefer a good second chance implementation, other page cache allocation flags or something else that explicitly solves this issue.

In that line, the patch that replaced congestion_wait() with a waitqueue makes some sense.

Quoted 3 lines

Mel's patch that replaces congestion_wait with a wait for the zone watermarks becoming available again is definitely a step in the right direction and should go into upstream and the long term support branches.

I'll need to do a number of tests before I can move that upstream but I don't think it's a merge candidate. Unfortunately, I'll be offline for a week starting tomorrow so I won't be able to do the testing.

Quoted 6 lines
Quoted 4 lines

When I get back, I'll revisit those patches with the view to pushing them upstream. I hate to treat symptoms here without knowing the underlying problem but this has been spinning in circles for ages with little forward progress :(

The zone pressure waitqueue patch makes sense.

I've just started the rebase and considering what sort of test is best for it.

Quoted 4 lines

We may even want to make
it more strictly FIFO (eg. check upfront if there are waiters on the queue before allocating a page, and if yes then add ourself to the back of the waitqueue).

To be really strict about this, we'd have to check in the hot-path of the per-cpu allocator which would be undesirable. We could check further in the slow-path but I bet it'd be very rare that the logic would be triggered. For a process to enter the FIFO due to waiters that were not yet woken up, the system would have to be a) under heavy memory pressure b) reclaim taking such a long time that check_zone_pressure() is not being called in time and c) a process exiting or otherwise freeing memory such that the watermarks are cleared without reclaim being involved.

This seems overkill but maybe you have a simplier case in mind?

Quoted 5 lines

And also possibly even look at doing the wakeups in the page-freeing path. Although that might start adding too much overhead, so it's quite possible your sloppy-but-lighter timeout approach is preferable.

That's how I felt about it. I was going to put another check_zone_pressure() check after a pcp drain but thought it was too expensive.

Signature · 3 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
#35 lore raw
Quoted 6 lines · Mel Gorman
Quoted 1 line · Nick Piggin

The zone pressure waitqueue patch makes sense.

I've just started the rebase and considering what sort of test is best for it.

Quoted 7 lines
Quoted 4 lines

We may even want to make
it more strictly FIFO (eg. check upfront if there are waiters on the queue before allocating a page, and if yes then add ourself to the back of the waitqueue).

To be really strict about this, we'd have to check in the hot-path of the per-cpu allocator which would be undesirable.

Yes, but it would also be desirable for other reasons (eliminating all unnecessary latency here). Which is why I say it is a tradeoff but it might be worthwhile.

We obviously wouldn't load the waitqueue itself in the fastpath, but probably some field in a cacheline we already touch.

Quoted 7 lines

We could check further in the
slow-path but I bet it'd be very rare that the logic would be triggered. For a process to enter the FIFO due to waiters that were not yet woken up, the system would have to be a) under heavy memory pressure b) reclaim taking such a long time that check_zone_pressure() is not being called in time and c) a process exiting or otherwise freeing memory such that the watermarks are cleared without reclaim being involved.

I don't think it would be too rare. Things can get freed up and other allocations come in while reclaim is happening. But anyway the nasty thing about the "rare" events is that they do add a rare source of unexpected latency or starvation.

I'm not saying necessarily that it will make a noticable improvement to throughput for this test case, but that now that we have the waitqueue there, we should think about exactly how far we want to go with it.

#36 lore raw
Quoted 15 lines · Nick Piggin
Quoted 6 lines · Mel Gorman
Quoted 1 line · Nick Piggin

The zone pressure waitqueue patch makes sense.

I've just started the rebase and considering what sort of test is best for it.

Quoted 7 lines
Quoted 4 lines

We may even want to make
it more strictly FIFO (eg. check upfront if there are waiters on the queue before allocating a page, and if yes then add ourself to the back of the waitqueue).

To be really strict about this, we'd have to check in the hot-path of the per-cpu allocator which would be undesirable.

Quoted 7 lines

Yes, but it would also be desirable for other reasons (eliminating all unnecessary latency here). Which is why I say it is a tradeoff but it might be worthwhile.

We obviously wouldn't load the waitqueue itself in the fastpath, but probably some field in a cacheline we already touch.

I'm struggling to justify it in my head. Granted, right now the pressure_wq is located beside the wait_table as a "rarely-used" field. It could be located before the free_area[] so it would be pulled in which would have some impact for the high-order lists but probably not too bad. This would make accessing pressure_wq a little cheaper.

The impact of *not* checking is unfairness. A late process makes forward progress while others sleep on a queue waiting for a reclaim pass to finish and the queue to be woken up. At worst, processes continue sleeping for the entire timeout because late processes kept the zone below the watermark.

Is a little unfairness in a path concerned with heavy memory pressure enough to justify checking on every page allocation?

Quoted 8 lines
Quoted 7 lines

We could check further in the
slow-path but I bet it'd be very rare that the logic would be triggered. For a process to enter the FIFO due to waiters that were not yet woken up, the system would have to be a) under heavy memory pressure b) reclaim taking such a long time that check_zone_pressure() is not being called in time and c) a process exiting or otherwise freeing memory such that the watermarks are cleared without reclaim being involved.

Quoted 5 lines

I don't think it would be too rare. Things can get freed up and other allocations come in while reclaim is happening. But anyway the nasty thing about the "rare" events is that they do add a rare source of unexpected latency or starvation.

If processes are asleep on the waitqueue, reclaim must be active (by kswapd if nothing else). If pages are getting freed above the necessary watermark, then the processes will be woken up when the current shrink_zone() finished unless unfair processes are keeping the zone below watermarks. But unless reclaim is taking an extraordinary long length of time, there would be little difference between waking the queue in the free path and waking it in the reclaim path.

Again, is a little unfairness under heavy memory pressure enough to alter the main paths?

Quoted 4 lines

I'm not saying necessarily that it will make a noticable improvement to throughput for this test case, but that now that we have the waitqueue there, we should think about exactly how far we want to go with it.

Fair point but I'd sooner look at the other places the VM waits on timeouts and see can they be converted to waiting on events. Just in case, after these tests complete (and assuming they generate usable figures), I'll prototype some of the suggestions and see what the impact is.

Signature · 3 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
#37 lore raw
Quoted 24 lines · Mel Gorman
Quoted 15 lines · Nick Piggin
Quoted 6 lines · Mel Gorman
Quoted 1 line · Nick Piggin

The zone pressure waitqueue patch makes sense.

I've just started the rebase and considering what sort of test is best for it.

Quoted 7 lines
Quoted 4 lines

We may even want to make
it more strictly FIFO (eg. check upfront if there are waiters on the queue before allocating a page, and if yes then add ourself to the back of the waitqueue).

To be really strict about this, we'd have to check in the hot-path of the per-cpu allocator which would be undesirable.

Quoted 7 lines

Yes, but it would also be desirable for other reasons (eliminating all unnecessary latency here). Which is why I say it is a tradeoff but it might be worthwhile.

We obviously wouldn't load the waitqueue itself in the fastpath, but probably some field in a cacheline we already touch.

Quoted 5 lines

I'm struggling to justify it in my head. Granted, right now the pressure_wq is located beside the wait_table as a "rarely-used" field. It could be located before the free_area[] so it would be pulled in which would have some impact for the high-order lists but probably not too bad. This would make accessing pressure_wq a little cheaper.

No you would mark it in another field in the slowpath.

It would be justified if the aggregate cost of the work is outweighed by the reduction in unnecessary wait time in the congestion code, plus some unquantifiable goodness from the extra fairness.

Obviously that depends on the workload, but so does every change we do like this.

Quoted 7 lines

The impact of *not* checking is unfairness. A late process makes forward progress while others sleep on a queue waiting for a reclaim pass to finish and the queue to be woken up. At worst, processes continue sleeping for the entire timeout because late processes kept the zone below the watermark.

Is a little unfairness in a path concerned with heavy memory pressure enough to justify checking on every page allocation?

Well, we've had patches that do a lot more than that in the allocator that most people don't really use :) (and they've had relatively small impact).

But no, it adds at least another branch, icache cost, and at least one more memory bit in the fastpath. Obviously that's a drawback.

Quoted 14 lines
Quoted 8 lines
Quoted 7 lines

We could check further in the
slow-path but I bet it'd be very rare that the logic would be triggered. For a process to enter the FIFO due to waiters that were not yet woken up, the system would have to be a) under heavy memory pressure b) reclaim taking such a long time that check_zone_pressure() is not being called in time and c) a process exiting or otherwise freeing memory such that the watermarks are cleared without reclaim being involved.

Quoted 5 lines

I don't think it would be too rare. Things can get freed up and other allocations come in while reclaim is happening. But anyway the nasty thing about the "rare" events is that they do add a rare source of unexpected latency or starvation.

Quoted 7 lines

If processes are asleep on the waitqueue, reclaim must be active (by kswapd if nothing else). If pages are getting freed above the necessary watermark, then the processes will be woken up when the current shrink_zone() finished unless unfair processes are keeping the zone below watermarks. But unless reclaim is taking an extraordinary long length of time, there would be little difference between waking the queue in the free path and waking it in the reclaim path.

Reclaim can take quite a while, yes.

Quoted 2 lines

Again, is a little unfairness under heavy memory pressure enough to alter the main paths?

Well you didn't let me answer yet :) See above.

Quoted 9 lines
Quoted 4 lines

I'm not saying necessarily that it will make a noticable improvement to throughput for this test case, but that now that we have the waitqueue there, we should think about exactly how far we want to go with it.

Fair point but I'd sooner look at the other places the VM waits on timeouts and see can they be converted to waiting on events. Just in case, after these tests complete (and assuming they generate usable figures), I'll prototype some of the suggestions and see what the impact is.

#38 lore raw
Quoted 25 lines · Nick Piggin
Quoted 24 lines · Mel Gorman
Quoted 22 lines · Nick Piggin
Quoted 6 lines · Mel Gorman
Quoted 1 line · Nick Piggin

The zone pressure waitqueue patch makes sense.

I've just started the rebase and considering what sort of test is best for it.

Quoted 7 lines
Quoted 4 lines

We may even want to make
it more strictly FIFO (eg. check upfront if there are waiters on the queue before allocating a page, and if yes then add ourself to the back of the waitqueue).

To be really strict about this, we'd have to check in the hot-path of the per-cpu allocator which would be undesirable.

Yes, but it would also be desirable for other reasons (eliminating all unnecessary latency here). Which is why I say it is a tradeoff but it might be worthwhile.

We obviously wouldn't load the waitqueue itself in the fastpath, but probably some field in a cacheline we already touch.

Quoted 8 lines
Quoted 5 lines

I'm struggling to justify it in my head. Granted, right now the pressure_wq is located beside the wait_table as a "rarely-used" field. It could be located before the free_area[] so it would be pulled in which would have some impact for the high-order lists but probably not too bad. This would make accessing pressure_wq a little cheaper.

No you would mark it in another field in the slowpath.

i.e. a flag in zone->flags that is set if something is waiting on the queue?

Quoted 7 lines

It would be justified if the aggregate cost of the work is outweighed by the reduction in unnecessary wait time in the congestion code, plus some unquantifiable goodness from the extra fairness.

Obviously that depends on the workload, but so does every change we do like this.

True. I'll sketch out what needs to happen along these lines and see what it looks like.

Quoted 8 lines
Quoted 7 lines

The impact of *not* checking is unfairness. A late process makes forward progress while others sleep on a queue waiting for a reclaim pass to finish and the queue to be woken up. At worst, processes continue sleeping for the entire timeout because late processes kept the zone below the watermark.

Is a little unfairness in a path concerned with heavy memory pressure enough to justify checking on every page allocation?

Quoted 7 lines

Well, we've had patches that do a lot more than that in the allocator that most people don't really use :) (and they've had relatively small impact).

But no, it adds at least another branch, icache cost, and at least one more memory bit in the fastpath. Obviously that's a drawback.

Of which the branch would be the greatest concern but it's also a very unlikely branch.

Quoted 41 lines
Quoted 14 lines
Quoted 8 lines
Quoted 7 lines

We could check further in the
slow-path but I bet it'd be very rare that the logic would be triggered. For a process to enter the FIFO due to waiters that were not yet woken up, the system would have to be a) under heavy memory pressure b) reclaim taking such a long time that check_zone_pressure() is not being called in time and c) a process exiting or otherwise freeing memory such that the watermarks are cleared without reclaim being involved.

Quoted 5 lines

I don't think it would be too rare. Things can get freed up and other allocations come in while reclaim is happening. But anyway the nasty thing about the "rare" events is that they do add a rare source of unexpected latency or starvation.

Quoted 7 lines

If processes are asleep on the waitqueue, reclaim must be active (by kswapd if nothing else). If pages are getting freed above the necessary watermark, then the processes will be woken up when the current shrink_zone() finished unless unfair processes are keeping the zone below watermarks. But unless reclaim is taking an extraordinary long length of time, there would be little difference between waking the queue in the free path and waking it in the reclaim path.

Reclaim can take quite a while, yes.

Quoted 2 lines

Again, is a little unfairness under heavy memory pressure enough to alter the main paths?

Well you didn't let me answer yet :) See above.

Quoted 9 lines
Quoted 4 lines

I'm not saying necessarily that it will make a noticable improvement to throughput for this test case, but that now that we have the waitqueue there, we should think about exactly how far we want to go with it.

Fair point but I'd sooner look at the other places the VM waits on timeouts and see can they be converted to waiting on events. Just in case, after these tests complete (and assuming they generate usable figures), I'll prototype some of the suggestions and see what the impact is.

Signature · 3 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
#39 lore raw
Quoted 5 lines · Mel Gorman
Quoted 4 lines · Nick Piggin
Quoted 3 lines · Mel Gorman
Quoted 2 lines · Nick Piggin
Quoted 1 line · Mel Gorman

On Tue, Mar 02, 2010 at 05:52:25PM +1100, Nick Piggin wrote:

[...]

Quoted 20 lines
Quoted 20 lines
Quoted 19 lines
Quoted 7 lines
Quoted 7 lines

We could check further in the
slow-path but I bet it'd be very rare that the logic would be triggered. For a process to enter the FIFO due to waiters that were not yet woken up, the system would have to be a) under heavy memory pressure b) reclaim taking such a long time that check_zone_pressure() is not being called in time and c) a process exiting or otherwise freeing memory such that the watermarks are cleared without reclaim being involved.

Quoted 5 lines

I don't think it would be too rare. Things can get freed up and other allocations come in while reclaim is happening. But anyway the nasty thing about the "rare" events is that they do add a rare source of unexpected latency or starvation.

If processes are asleep on the waitqueue, reclaim must be active (by kswapd if nothing else). If pages are getting freed above the necessary watermark, then the processes will be woken up when the current shrink_zone() finished unless unfair processes are keeping the zone below watermarks. But unless reclaim is taking an extraordinary long length of time, there would be little difference between waking the queue in the free path and waking it in the reclaim path.

Quoted 2 lines
Quoted 2 lines

Reclaim can take quite a while, yes.

On one Hand the question if "waiter A is not yet awoken after shrink_zone(), but greedy B just drained pages under the water mark again" is good to make these new waitqueue approach as good as it can be.
On the other Hand you can see it this way - it is now at least waiting for the right thing "the related watermark being restored", which will in any way be better than waiting for writes who might or might not free enough pages or as in my case might not even be there :-) And additionally its timing even if it could be a bit racy as you described will be much better than it is at the moment.

Signature · 2 lines
Grüsse / regards, Christian Ehrhardt
IBM Linux Technology Center, System z Linux Performance