vmscan: limit concurrent reclaimers in shrink_zone
Subsystems (12)
PATCH v2 1/1 vmscan: limit concurrent reclaimers in shrink_zone
Under very heavy multi-process workloads, like AIM7, the VM can get into trouble in a variety of ways. The trouble start when there are hundreds, or even thousands of processes active in the page reclaim code.
Not only can the system suffer enormous slowdowns because of lock contention (and conditional reschedules) between thousands of processes in the page reclaim code, but each process will try to free up to SWAP_CLUSTER_MAX pages, even when the system already has lots of memory free.
It should be possible to avoid both of those issues at once, by simply limiting how many processes are active in the page reclaim code simultaneously.
If too many processes are active doing page reclaim in one zone, simply go to sleep in shrink_zone().
On wakeup, check whether enough memory has been freed already before jumping into the page reclaim code ourselves. We want to use the same threshold here that is used in the page allocator for deciding whether or not to call the page reclaim code in the first place, otherwise some unlucky processes could end up freeing memory for the rest of the system.
---
v2:
- fix typos in sysctl.c and vm.txt
- move the code in sysctl.c out from under the ifdef
- only __GFP_FS|__GFP_IO tasks can wait
Documentation/sysctl/vm.txt | 18 ++++++++++++++ include/linux/mmzone.h | 4 +++ include/linux/swap.h | 1 + kernel/sysctl.c | 7 +++++ mm/page_alloc.c | 3 ++ mm/vmscan.c | 40 +++++++++++++++++++++++++++++++++ 6 files changed, 73 insertions(+), 0 deletions(-)
Documentation/sysctl/vm.txt +18 -0
@@ -32,6 +32,7 @@ Currently, these files are in /proc/sys/vm: - legacy_va_layout - lowmem_reserve_ratio - max_map_count +- max_zone_concurrent_reclaimers - memory_failure_early_kill - memory_failure_recovery - min_free_kbytes
@@ -278,6 +279,23 @@ The default value is 65536. ============================================================= +max_zone_concurrent_reclaimers: + +The number of processes that are allowed to simultaneously reclaim +memory from a particular memory zone. + +With certain workloads, hundreds of processes end up in the page +reclaim code simultaneously. This can cause large slowdowns due +to lock contention, freeing of way too much memory and occasionally +false OOM kills. + +To avoid these problems, only allow a smaller number of processes +to reclaim pages from each memory zone simultaneously. + +The default value is 8. + +============================================================= + memory_failure_early_kill: Control how to kill processes when uncorrected memory error (typically
include/linux/mmzone.h +4 -0
@@ -345,6 +345,10 @@ struct zone {/* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS ]; +/* Number of processes running page reclaim code on this zone. */ +atomic_t concurrent_reclaimers; +wait_queue_head_t reclaim_wait; +/* * prev_priority holds the scanning priority for this zone. It is * defined as the scanning priority at which we achieved our reclaim
include/linux/swap.h +1 -0
@@ -254,6 +254,7 @@ extern unsigned long shrink_all_memory(unsigned long nr_pages);extern int vm_swappiness;extern int remove_mapping (struct address_space *mapping,struct page *page);extern long vm_total_pages; +extern int max_zone_concurrent_reclaimers;#ifdef CONFIG_NUMA extern int zone_reclaim_mode;
kernel/sysctl.c +7 -0
@@ -1271,6 +1271,13 @@ static struct ctl_table vm_table[] = { .extra2 = &one, },#endif + { + .procname = "max_zone_concurrent_reclaimers", + .data = &max_zone_concurrent_reclaimers, + .maxlen = sizeof(max_zone_concurrent_reclaimers), + .mode = 0644, + .proc_handler = &proc_dointvec, + },/* * NOTE: do not add new entries to this table unless you have read
mm/page_alloc.c +3 -0
@@ -3852,6 +3852,9 @@ static void __paginginit free_area_init_core(struct pglist_data *pgdat, zone->prev_priority =DEF_PRIORITY ; +atomic_set (&zone->concurrent_reclaimers,0 ); +init_waitqueue_head (&zone->reclaim_wait); +zone_pcp_init (zone);for_each_lru (l) {INIT_LIST_HEAD (&zone->lru[l].list);
mm/vmscan.c +40 -0
@@ -40,6 +40,7 @@#include <linux/memcontrol.h> #include <linux/delayacct.h> #include <linux/sysctl.h> +#include <linux/wait.h> #include <asm/tlbflush.h> #include <asm/div64.h>
@@ -129,6 +130,17 @@ struct scan_control {int vm_swappiness =60 ;long vm_total_pages;/* The total number of pages which the VM controls */ +/* +* Maximum number of processes concurrently running the page +* reclaim code in a memory zone. Having too many processes +* just results in them burning CPU time waiting for locks, +* so we're better off limiting page reclaim to a sane number +* of processes at a time. We do this per zone so local node +* reclaim on one NUMA node will not block other nodes from +* making progress. +*/ +int max_zone_concurrent_reclaimers =8 ; +static LIST_HEAD (shrinker_list);static DECLARE_RWSEM (shrinker_rwsem);
@@ -1600,6 +1612,31 @@ static void shrink_zone(int priority, struct zone *zone,struct zone_reclaim_stat *reclaim_stat =get_reclaim_stat (zone, sc);int noswap =0 ; +if (!current_is_kswapd () &&atomic_read (&zone->concurrent_reclaimers) > + max_zone_concurrent_reclaimers && + (sc->gfp_mask & (__GFP_IO|__GFP_FS)) == + (__GFP_IO|__GFP_FS)) { +/* +* Do not add to the lock contention if this zone has +* enough processes doing page reclaim already, since +* we would just make things slower. +*/ +sleep_on (&zone->reclaim_wait); + +/* +* If other processes freed enough memory while we waited, +* break out of the loop and go back to the allocator. +*/ +if (zone_watermark_ok (zone, sc->order,low_wmark_pages (zone), +0 ,0 )) { +wake_up (&zone->reclaim_wait); + sc->nr_reclaimed += nr_to_reclaim; +return ; + } + } + +atomic_inc (&zone->concurrent_reclaimers); +/* If we have no swap space, do not bother scanning anon pages. */ if (!sc->may_swap || (nr_swap_pages <=0 )) { noswap =1 ;
@@ -1655,6 +1692,9 @@ static void shrink_zone(int priority, struct zone *zone,shrink_active_list (SWAP_CLUSTER_MAX , zone, sc, priority,0 );throttle_vm_writeout (sc->gfp_mask); + +atomic_dec (&zone->concurrent_reclaimers); +wake_up (&zone->reclaim_wait); }/*
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Hi, Rik.
Quoted 18 lines · Rik van Riel
Under very heavy multi-process workloads, like AIM7, the VM can get into trouble in a variety of ways. The trouble start when there are hundreds, or even thousands of processes active in the page reclaim code.
Not only can the system suffer enormous slowdowns because of lock contention (and conditional reschedules) between thousands of processes in the page reclaim code, but each process will try to free up to
SWAP_CLUSTER_MAXpages, even when the system already has lots of memory free.It should be possible to avoid both of those issues at once, by simply limiting how many processes are active in the page reclaim code simultaneously.
If too many processes are active doing page reclaim in one zone, simply go to sleep in
shrink_zone().
Quoted 6 lines
On wakeup, check whether enough memory has been freed already before jumping into the page reclaim code ourselves. We want to use the same threshold here that is used in the page allocator for deciding whether or not to call the page reclaim code in the first place, otherwise some unlucky processes could end up freeing memory for the rest of the system.
I am worried about one.
Now, we can put too many processes reclaim_wait with NR_UNINTERRUBTIBLE state. If OOM happens, OOM will kill many innocent processes since uninterruptible task
can't handle kill signal until the processes free from reclaim_wait list.
I think reclaim_wait list staying time might be long if VM pressure is heavy. Is this a exaggeration?
If it is serious problem, how about this?
We add new PF_RECLAIM_BLOCK flag and don't pick the process in select_bad_process.
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 12 lines · Minchan Kim
Quoted 2 lines · Rik van Riel
If too many processes are active doing page reclaim in one zone, simply go to sleep in
shrink_zone().I am worried about one.
Now, we can put too many processes
reclaim_waitwithNR_UNINTERRUBTIBLEstate. If OOM happens, OOM will kill many innocent processes since uninterruptible task
can't handle kill signal until the processes free fromreclaim_waitlist.
Quoted 7 lines
I think
reclaim_waitlist staying time might be long if VM pressure is heavy. Is this a exaggeration?If it is serious problem, how about this?
We add new
PF_RECLAIM_BLOCKflag and don't pick the process inselect_bad_process.
A simpler solution may be to use sleep_on_interruptible, and simply have the process continue into shrink_zone() if it gets a signal.
Signature · 7 lines
All rights reversed. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 19 lines · Rik van Riel
Quoted 17 lines · Minchan Kim
Quoted 2 lines · Rik van Riel
If too many processes are active doing page reclaim in one zone, simply go to sleep in
shrink_zone().I am worried about one.
Now, we can put too many processes
reclaim_waitwithNR_UNINTERRUBTIBLEstate.
If OOM happens, OOM will kill many innocent processes since uninterruptible task
can't handle kill signal until the processes free fromreclaim_waitlist.I think
reclaim_waitlist staying time might be long if VM pressure is heavy.
Is this a exaggeration?
Quoted 8 lines
Quoted 4 lines
If it is serious problem, how about this?
We add new
PF_RECLAIM_BLOCKflag and don't pick the process inselect_bad_process.A simpler solution may be to use
sleep_on_interruptible, and simply have the process continue intoshrink_zone()if it gets a signal.
I thought it but I was not sure.
Okay. If it is possible, It' more simple.
Could you repost patch with that?
Sorry but I have one requesting.
===
+The default value is 8.
+
+=============================================================
I like this. but why do you select default value as constant 8? Do you have any reason?
I think it would be better to select the number proportional to NR_CPU. ex) NR_CPU * 2 or something.
Otherwise looks good to me.
Pessimistically, I assume that the pageout code spends maybe 10% of its time on locking (we have seen far, far worse than this with thousands of processes in the pageout code). That means if we have more than 10 threads in the pageout code, we could end up spending more time on locking and less doing real work - slowing everybody down.
I rounded it down to the closest power of 2 to come up with an arbitrary number that looked safe :)
===
We discussed above.
I want to add your desciption into changelog.
That's because after long time, We don't know why we select '8' as default value.
Your desciption in changelog will explain it to follow-up people. :)
Sorry for bothering you.
Quoted 3 lines
--
All rights reversed.
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 9 lines · Minchan Kim
Quoted 3 lines · Rik van Riel
A simpler solution may be to use
sleep_on_interruptible, and simply have the process continue intoshrink_zone()if it gets a signal.I thought it but I was not sure.
Okay. If it is possible, It' more simple.
Could you repost patch with that?
Sure, not a problem.
Quoted 21 lines
+The default value is 8.
+
+=============================================================I like this. but why do you select default value as constant 8? Do you have any reason?
I think it would be better to select the number proportional to
NR_CPU. ex)NR_CPU* 2 or something.Otherwise looks good to me.
Pessimistically, I assume that the pageout code spends maybe 10% of its time on locking (we have seen far, far worse than this with thousands of processes in the pageout code). That means if we have more than 10 threads in the pageout code, we could end up spending more time on locking and less doing real work - slowing everybody down.
Quoted 6 lines
I rounded it down to the closest power of 2 to come up with an arbitrary number that looked safe :)
===We discussed above.
I want to add your desciption into changelog.
The thing is, I don't know if 8 is the best value for the default, which is a reason I made it tunable in the first place.
There are a lot of assumptions in my reasoning, and they may be wrong. I suspect that documenting something wrong is probably worse than letting people wonder out the default (and maybe finding a better one).
Signature · 7 lines
All rights reversed. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 43 lines · Rik van Riel
Quoted 10 lines · Minchan Kim
Quoted 3 lines · Rik van Riel
A simpler solution may be to use
sleep_on_interruptible, and simply have the process continue intoshrink_zone()if it gets a signal.I thought it but I was not sure.
Okay. If it is possible, It' more simple.
Could you repost patch with that?Sure, not a problem.
Quoted 22 lines
+The default value is 8.
+
+=============================================================I like this. but why do you select default value as constant 8? Do you have any reason?
I think it would be better to select the number proportional to
NR_CPU.
ex)NR_CPU* 2 or something.Otherwise looks good to me.
Pessimistically, I assume that the pageout code spends maybe 10% of its time on locking (we have seen far, far worse than this with thousands of processes in the pageout code). That means if we have more than 10 threads in the pageout code, we could end up spending more time on locking and less doing real work - slowing everybody down.
Quoted 6 lines
I rounded it down to the closest power of 2 to come up with an arbitrary number that looked safe :)
===We discussed above.
I want to add your desciption into changelog.
Quoted 8 lines
The thing is, I don't know if 8 is the best value for the default, which is a reason I made it tunable in the first place.
There are a lot of assumptions in my reasoning, and they may be wrong. I suspect that documenting something wrong is probably worse than letting people wonder out the default (and maybe finding a better one).
Indeed. But whenever I see magic values in kernel, I have a question about that.
Actually I used to doubt the value because I guess
"that value was determined by server or desktop experiments. so probably it don't proper small system."
At least if we put the logical why which might be wrong, later people can think that value is not proper any more now or his system(ex, small or super computer and so on) by based on our description.
so they can improve it.
I think it isn't important your reasoning is right or wrong. Most important thing is which logical reason determines that value.
I want to not bother you if you mind my suggestion. Pz, think it was just nitpick. :)
Quoted 3 lines
--
All rights reversed.
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 18 lines
Under very heavy multi-process workloads, like AIM7, the VM can get into trouble in a variety of ways. The trouble start when there are hundreds, or even thousands of processes active in the page reclaim code.
Not only can the system suffer enormous slowdowns because of lock contention (and conditional reschedules) between thousands of processes in the page reclaim code, but each process will try to free up to
SWAP_CLUSTER_MAXpages, even when the system already has lots of memory free.It should be possible to avoid both of those issues at once, by simply limiting how many processes are active in the page reclaim code simultaneously.
If too many processes are active doing page reclaim in one zone, simply go to sleep in
shrink_zone().
Quoted 6 lines
On wakeup, check whether enough memory has been freed already before jumping into the page reclaim code ourselves. We want to use the same threshold here that is used in the page allocator for deciding whether or not to call the page reclaim code in the first place, otherwise some unlucky processes could end up freeing memory for the rest of the system.
This patch seems very similar to my old effort. afaik, there are another two benefit.
1. Improve resource gurantee
if thousands tasks start to vmscan at the same time, they eat all memory for PF_MEMALLOC. it might cause another dangerous problem. some filesystem and io device don't handle allocation failure properly.
2. Improve OOM contidion behavior
Currently, vmscan don't handle SIGKILL at all. then if the system have hevy memory pressure, OOM killer can't kill the target process soon. it might cause OOM killer kill next innocent process. This patch can fix it.
Quoted 4 lines
mm/vmscan.c +0
-0@@ -1600,6 +1612,31 @@ static void shrink_zone(int priority, struct zone *zone,struct zone_reclaim_stat *reclaim_stat =get_reclaim_stat (zone, sc);int noswap =0 ;
mm/vmscan.c static void shrink_zone(int priority, struct zone *zone, · line 1624
Quoted 10 lines
mm/vmscan.c +10
-0+if (!current_is_kswapd () &&atomic_read (&zone->concurrent_reclaimers) > + max_zone_concurrent_reclaimers && + (sc->gfp_mask & (__GFP_IO|__GFP_FS)) == + (__GFP_IO|__GFP_FS)) { +/* +* Do not add to the lock contention if this zone has +* enough processes doing page reclaim already, since +* we would just make things slower. +*/ +sleep_on (&zone->reclaim_wait);
Oops. this is bug. sleep_on() is not SMP safe.
I made few fixing patch today. I'll post it soon.
btw, following is mesurement result by hackbench.
================
unit: sec
parameter old new
130 (5200 processes) 5.463 4.442
140 (5600 processes) 479.357 7.792
150 (6000 processes) 729.640 20.529
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/8 vmscan: Make shrink_zone_begin/end helper function
concurrent_reclaimers limitation related code made mess to shrink_zone. To introduce helper function increase readability.
--- mm/vmscan.c | 58 +++++++++++++++++++++++++++++++++++----------------------- 1 files changed, 35 insertions(+), 23 deletions(-)
mm/vmscan.c +35 -23
@@ -1597,25 +1597,11 @@ static unsigned long nr_scan_try_batch(unsigned long nr_to_scan,return nr; } -/* -* This is a basic per-zone page freer. Used by both kswapd and direct reclaim. -*/ -static void shrink_zone (int priority,struct zone *zone, -struct scan_control *sc) +static int shrink_zone_begin (struct zone *zone,struct scan_control *sc) { -unsigned long nr[NR_LRU_LISTS ]; -unsigned long nr_to_scan; -unsigned long percent[2 ];/* anon @ 0; file @ 1 */ -enum lru_list l; -unsigned long nr_reclaimed = sc->nr_reclaimed; -unsigned long nr_to_reclaim = sc->nr_to_reclaim; -struct zone_reclaim_stat *reclaim_stat =get_reclaim_stat (zone, sc); -int noswap =0 ; - -if (!current_is_kswapd () &&atomic_read (&zone->concurrent_reclaimers) > - max_zone_concurrent_reclaimers && - (sc->gfp_mask & (__GFP_IO|__GFP_FS)) == - (__GFP_IO|__GFP_FS)) { +if (!current_is_kswapd () && +atomic_read (&zone->concurrent_reclaimers) > max_zone_concurrent_reclaimers && + (sc->gfp_mask & (__GFP_IO|__GFP_FS)) == (__GFP_IO|__GFP_FS)) {/* * Do not add to the lock contention if this zone has * enough processes doing page reclaim already, since
@@ -1630,12 +1616,40 @@ static void shrink_zone(int priority, struct zone *zone,if (zone_watermark_ok (zone, sc->order,low_wmark_pages (zone),0 ,0 )) {wake_up (&zone->reclaim_wait); - sc->nr_reclaimed += nr_to_reclaim; -return ; + sc->nr_reclaimed += sc->nr_to_reclaim; +return -ERESTARTSYS ; } }atomic_inc (&zone->concurrent_reclaimers); +return 0 ; +} + +static void shrink_zone_end (struct zone *zone,struct scan_control *sc) +{ +atomic_dec (&zone->concurrent_reclaimers); +wake_up (&zone->reclaim_wait); +} + +/* +* This is a basic per-zone page freer. Used by both kswapd and direct reclaim. +*/ +static void shrink_zone (int priority,struct zone *zone, +struct scan_control *sc) +{ +unsigned long nr[NR_LRU_LISTS ]; +unsigned long nr_to_scan; +unsigned long percent[2 ];/* anon @ 0; file @ 1 */ +enum lru_list l; +unsigned long nr_reclaimed = sc->nr_reclaimed; +unsigned long nr_to_reclaim = sc->nr_to_reclaim; +struct zone_reclaim_stat *reclaim_stat =get_reclaim_stat (zone, sc); +int noswap =0 ; +int ret; + + ret =shrink_zone_begin (zone, sc); +if (ret) +return ;/* If we have no swap space, do not bother scanning anon pages. */ if (!sc->may_swap || (nr_swap_pages <=0 )) {
@@ -1692,9 +1706,7 @@ static void shrink_zone(int priority, struct zone *zone,shrink_active_list (SWAP_CLUSTER_MAX , zone, sc, priority,0 );throttle_vm_writeout (sc->gfp_mask); - -atomic_dec (&zone->concurrent_reclaimers); -wake_up (&zone->reclaim_wait); +shrink_zone_end (zone, sc); }/*
Signature · 9 lines
1.6.5.2 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [cleanup][PATCH 1/8] vmscan: Make shrink_zone_begin/end helper function
Quoted 4 lines · KOSAKI Motohiro
concurrent_reclaimerslimitation related code made mess toshrink_zone. To introduce helper function increase readability.Signed-off-by KOSAKI Motohiro<[email protected]>
Signature · 7 lines
All rights reversed. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [cleanup][PATCH 1/8] vmscan: Make shrink_zone_begin/end helper function
In reply to KOSAKI MotohiroQuoted 4 lines · On Mon, 14 Dec 2009 21:23:46 +0900 (JST) KOSAKI Motohiro…
concurrent_reclaimerslimitation related code made mess toshrink_zone. To introduce helper function increase readability.Signed-off-by KOSAKI Motohiro <[email protected]>
Through looking over patch series, it's nice clean up.
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 2/8 Mark sleep_on as deprecated
In reply to KOSAKI Motohirosleep_on() function is SMP and/or kernel preemption unsafe. we shouldn't use it on new code.
--- include/linux/wait.h | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-)
include/linux/wait.h +4 -4
@@ -427,11 +427,11 @@ static inline void remove_wait_queue_locked(wait_queue_head_t *q,* They are racy. DO NOT use them, use the wait_event* interfaces above. * We plan to remove these interfaces. */ -extern void sleep_on (wait_queue_head_t *q); -extern long sleep_on_timeout (wait_queue_head_t *q, +extern void __deprecated sleep_on (wait_queue_head_t *q); +extern long __deprecated sleep_on_timeout (wait_queue_head_t *q,signed long timeout); -extern void interruptible_sleep_on (wait_queue_head_t *q); -extern long interruptible_sleep_on_timeout (wait_queue_head_t *q, +extern void __deprecated interruptible_sleep_on (wait_queue_head_t *q); +extern long __deprecated interruptible_sleep_on_timeout (wait_queue_head_t *q,signed long timeout);/*
Signature · 9 lines
1.6.5.2 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/8] Mark sleep_on as deprecated
Quoted 4 lines · KOSAKI Motohiro
sleep_on()function is SMP and/or kernel preemption unsafe. we shouldn't use it on new code.
And the best way to archive this is to remove the function.
In Linus' current tree I find:
- 5 instances of sleep_on(), all in old and obscure block drivers
- 2 instances of sleep_on_timeout(), both in old and obscure drivers
- 28 instances of interruptible_sleep_on_timeout(), mostly in obscure drivers with a high concentration in the old oss core which should be killed anyway. And unfortunately a few relatively recent additions like the SGI xpc driver or usbvision driver
- tons of instances of interruptible_sleep_on all over the drivers code
So I think you're be better off just killing the first two ASAP instead of just deprecating them.
for the other two deprecating and removing some of the horrible drivers still using them might be best.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/8] Mark sleep_on as deprecated
Quoted 6 lines · On Mon, 14 Dec 2009 08:03:02 -0500 Christoph Hellwig <[email protected]> wrote:
Quoted 4 lines · KOSAKI Motohiro
sleep_on()function is SMP and/or kernel preemption unsafe. we shouldn't use it on new code.
Quoted 6 lines
And the best way to archive this is to remove the function.
In Linus' current tree I find:
- 5 instances of
sleep_on(), all in old and obscure block drivers
- 2 instances ofsleep_on_timeout(), both in old and obscure drivers
these should both die; the sleep_on() ones using BROKEN in Kconfig.. .. sleep_on() has not worked in the 2.6 series ever.... ;)
Quoted 5 lines
- 28 instances of
interruptible_sleep_on_timeout(), mostly in obscure drivers with a high concentration in the old oss core which should
be killed anyway. And unfortunately a few relatively recent additions
like the SGI xpc driver or usbvision driver
can we also make sure that checkpatch.pl catches any new addition? (not saying checkpatch.pl is the end-all, but the people who do run it at least have now have a chance ;-)
Signature · 9 lines
Arjan van de Ven Intel Open Source Technology Centre For development, discussion and tips for power savings, visit http://www.lesswatts.org -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/8] Mark sleep_on as deprecated
In reply to KOSAKI MotohiroQuoted 6 lines · KOSAKI Motohiro
sleep_on()function is SMP and/or kernel preemption unsafe. we shouldn't use it on new code.Signed-off-by KOSAKI Motohiro<[email protected]>
Signature · 7 lines
All rights reversed. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/8] Mark sleep_on as deprecated
In reply to KOSAKI MotohiroQuoted 6 lines · On Mon, 14 Dec 2009 21:24:40 +0900 (JST) KOSAKI Motohiro…
sleep_on()function is SMP and/or kernel preemption unsafe. we shouldn't use it on new code.Signed-off-by KOSAKI Motohiro <[email protected]>
We would be better to remove this function. But it's enough to that in this patch series. We have to remove sleep_on with another patch series.
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 3/8 Don't use sleep_on()
In reply to KOSAKI Motohirosleep_on() is SMP and/or kernel preemption unsafe. This patch replace it with safe code.
--- mm/vmscan.c | 40 ++++++++++++++++++++++++++++++---------- 1 files changed, 30 insertions(+), 10 deletions(-)
mm/vmscan.c +30 -10
@@ -1599,15 +1599,32 @@ static unsigned long nr_scan_try_batch(unsigned long nr_to_scan,static int shrink_zone_begin (struct zone *zone,struct scan_control *sc) { -if (!current_is_kswapd () && -atomic_read (&zone->concurrent_reclaimers) > max_zone_concurrent_reclaimers && - (sc->gfp_mask & (__GFP_IO|__GFP_FS)) == (__GFP_IO|__GFP_FS)) { -/* -* Do not add to the lock contention if this zone has -* enough processes doing page reclaim already, since -* we would just make things slower. -*/ -sleep_on (&zone->reclaim_wait); +DEFINE_WAIT (wait); +wait_queue_head_t *wq = &zone->reclaim_wait; + +if (current_is_kswapd ()) + goto out; + +/* +* GFP_NOIO and GFP_NOFS mean caller may have some lock implicitly. +* Thus, we can't wait here. otherwise it might cause deadlock. +*/ +if ((sc->gfp_mask & (__GFP_IO|__GFP_FS)) != (__GFP_IO|__GFP_FS)) + goto out; + +/* +* Do not add to the lock contention if this zone has +* enough processes doing page reclaim already, since +* we would just make things slower. +*/ +for (;;) { +prepare_to_wait (wq, &wait,TASK_UNINTERRUPTIBLE ); + +if (atomic_read (&zone->concurrent_reclaimers) <= + max_zone_concurrent_reclaimers) +break ; + +schedule ();/* * If other processes freed enough memory while we waited,
@@ -1615,12 +1632,15 @@ static int shrink_zone_begin(struct zone *zone, struct scan_control *sc)*/ if (zone_watermark_ok (zone, sc->order,low_wmark_pages (zone),0 ,0 )) { -wake_up (&zone->reclaim_wait); +wake_up (wq); +finish_wait (wq, &wait); sc->nr_reclaimed += sc->nr_to_reclaim;return -ERESTARTSYS ; } } +finish_wait (wq, &wait); + out:atomic_inc (&zone->concurrent_reclaimers);return 0 ; }
Signature · 9 lines
1.6.5.2 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 3/8] Don't use sleep_on()
Quoted 4 lines · KOSAKI Motohiro
sleep_on()is SMP and/or kernel preemption unsafe. This patch replace it with safe code.Signed-off-by KOSAKI Motohiro<[email protected]>
Signature · 7 lines
All rights reversed. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 3/8] Don't use sleep_on()
In reply to KOSAKI MotohiroQuoted 4 lines · On Mon, 14 Dec 2009 21:29:28 +0900 (JST) KOSAKI Motohiro…
sleep_on()is SMP and/or kernel preemption unsafe. This patch replace it with safe code.Signed-off-by KOSAKI Motohiro <[email protected]>
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 4/8 Use prepare_to_wait_exclusive() instead prepare_to_wait()
In reply to KOSAKI Motohiroif we don't use exclusive queue, wake_up() function wake _all_ waited task. This is simply cpu wasting.
--- mm/vmscan.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
mm/vmscan.c +2 -2
@@ -1618,7 +1618,7 @@ static int shrink_zone_begin(struct zone *zone, struct scan_control *sc)* we would just make things slower. */ for (;;) { -prepare_to_wait (wq, &wait,TASK_UNINTERRUPTIBLE ); +prepare_to_wait_exclusive (wq, &wait,TASK_UNINTERRUPTIBLE );if (atomic_read (&zone->concurrent_reclaimers) <= max_zone_concurrent_reclaimers)
@@ -1632,7 +1632,7 @@ static int shrink_zone_begin(struct zone *zone, struct scan_control *sc)*/ if (zone_watermark_ok (zone, sc->order,low_wmark_pages (zone),0 ,0 )) { -wake_up (wq); +wake_up_all (wq);finish_wait (wq, &wait); sc->nr_reclaimed += sc->nr_to_reclaim;return -ERESTARTSYS ;
Signature · 1 line
1.6.5.2
Re: [PATCH 4/8] Use prepare_to_wait_exclusive() instead prepare_to_wait()
mm/vmscan.c static int shrink_zone_begin(struct zone *zone, struct scan_control *sc) · line 1638
Quoted 12 lines · KOSAKI Motohiro
if we don't use exclusive queue,
wake_up()function wake _all_ waited task. This is simply cpu wasting.Signed-off-by KOSAKI Motohiro<[email protected]>if (zone_watermark_ok(zone,
sc->order,low_wmark_pages(zone),
0, 0)) {mm/vmscan.c +1
-1-wake_up (wq); +wake_up_all (wq);finish_wait (wq,&wait); sc->nr_reclaimed += sc->nr_to_reclaim;return -ERESTARTSYS ;
I believe we want to wake the processes up one at a time here. If the queue of waiting processes is very large and the amount of excess free memory is fairly low, the first processes that wake up can take the amount of free memory back down below the threshold. The rest of the waiters should stay asleep when this happens.
Signature · 7 lines
All rights reversed. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 4/8] Use prepare_to_wait_exclusive() instead prepare_to_wait()
Quoted 6 lines · KOSAKI Motohiro
Quoted 4 lines
if we don't use exclusive queue,
wake_up()function wake _all_ waited task. This is simply cpu wasting.Signed-off-by KOSAKI Motohiro<[email protected]>
Quoted 14 lines
Review
Quoted 7 lines
if (zone_watermark_ok(zone,
sc->order,low_wmark_pages(zone),
0, 0)) {(quoted diff) +1
-1- wake_up(wq); + wake_up_all(wq); finish_wait(wq,&wait); sc->nr_reclaimed += sc->nr_to_reclaim; return -ERESTARTSYS;I believe we want to wake the processes up one at a time here. If the queue of waiting processes is very large and the amount of excess free memory is fairly low, the first processes that wake up can take the amount of free memory back down below the threshold. The rest of the waiters should stay asleep when this happens.
OK.
Actually, wake_up() and wake_up_all() aren't different so much. Although we use wake_up(), the task wake up next task before try to alloate memory. then, it's similar to wake_up_all().
However, there are few difference. recent scheduler latency improvement effort reduce default scheduler latency target. it mean, if we have lots tasks of running state, the task have very few time slice. too frequently context switch decrease VM efficiency.
Thank you, Rik. I didn't notice wake_up() makes better performance than wake_up_all() on current kernel.
Subject: [PATCH 9/8] replace wake_up_all with wake_up
Fix typo.
--- mm/vmscan.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
mm/vmscan.c +1 -1
@@ -1644,7 +1644,7 @@ static int shrink_zone_begin(struct zone *zone, struct scan_control *sc)return 0 ; found_lots_memory: -wake_up_all (wq); +wake_up (wq); stop_reclaim:finish_wait (wq, &wait); sc->nr_reclaimed += sc->nr_to_reclaim;
Signature · 9 lines
1.6.5.2 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 4/8] Use prepare_to_wait_exclusive() instead prepare_to_wait()
Quoted 21 lines · KOSAKI Motohiro
Quoted 6 lines · KOSAKI Motohiro
Quoted 4 lines
if we don't use exclusive queue,
wake_up()function wake _all_ waited task. This is simply cpu wasting.Signed-off-by KOSAKI Motohiro<[email protected]>Quoted 14 lines
Review
Quoted 7 lines
if (zone_watermark_ok(zone,
sc->order,low_wmark_pages(zone),
0, 0)) {(quoted diff) +1
-1- wake_up(wq); + wake_up_all(wq); finish_wait(wq,&wait); sc->nr_reclaimed += sc->nr_to_reclaim; return -ERESTARTSYS;I believe we want to wake the processes up one at a time here. If the queue of waiting processes is very large and the amount of excess free memory is fairly low, the first processes that wake up can take the amount of free memory back down below the threshold. The rest of the waiters should stay asleep when this happens.
Quoted 5 lines
OK.
Actually,
wake_up()andwake_up_all()aren't different so much. Although we usewake_up(), the task wake up next task before try to alloate memory. then, it's similar towake_up_all().
What happens to waiters should running tasks not allocate for a while?
Quoted 6 lines
However, there are few difference. recent scheduler latency improvement effort reduce default scheduler latency target. it mean, if we have lots tasks of running state, the task have very few time slice. too frequently context switch decrease VM efficiency.
Thank you, Rik. I didn't noticewake_up()makes better performance thanwake_up_all()on current kernel.
Perhaps this is a spot where an explicit wake_up_all_nopreempt() would be handy. Excessive wakeup preemption from wake_up_all() has long been annoying when there are many waiters, but converting it to only have the first wakeup be preemptive proved harmful to performance. Recent tweaks will have aggravated the problem somewhat, but it's not new.
-Mike
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 4/8] Use prepare_to_wait_exclusive() instead prepare_to_wait()
Quoted 37 lines · Mike Galbraith
Quoted 21 lines · KOSAKI Motohiro
Quoted 6 lines · KOSAKI Motohiro
Quoted 4 lines
if we don't use exclusive queue,
wake_up()function wake _all_ waited task. This is simply cpu wasting.Signed-off-by KOSAKI Motohiro<[email protected]>Quoted 14 lines
Review
Quoted 7 lines
if (zone_watermark_ok(zone,
sc->order,low_wmark_pages(zone),
0, 0)) {(quoted diff) +1
-1- wake_up(wq); + wake_up_all(wq); finish_wait(wq,&wait); sc->nr_reclaimed += sc->nr_to_reclaim; return -ERESTARTSYS;I believe we want to wake the processes up one at a time here. If the queue of waiting processes is very large and the amount of excess free memory is fairly low, the first processes that wake up can take the amount of free memory back down below the threshold. The rest of the waiters should stay asleep when this happens.
Quoted 5 lines
OK.
Actually,
wake_up()andwake_up_all()aren't different so much. Although we usewake_up(), the task wake up next task before try to alloate memory. then, it's similar towake_up_all().What happens to waiters should running tasks not allocate for a while?
Quoted 6 lines
However, there are few difference. recent scheduler latency improvement effort reduce default scheduler latency target. it mean, if we have lots tasks of running state, the task have very few time slice. too frequently context switch decrease VM efficiency.
Thank you, Rik. I didn't noticewake_up()makes better performance thanwake_up_all()on current kernel.
Quoted 2 lines
Perhaps this is a spot where an explicit
wake_up_all_nopreempt()would be handy....
Maybe something like below. I can also imagine that under _heavy_ vm pressure, it'd likely be good for throughput to not provide for sleeper fairness for these wakeups as well, as that increases vruntime spread, and thus increases preemption with no benefit in sight.
--- include/linux/sched.h | 1 + include/linux/wait.h | 3 +++ kernel/sched.c | 21 +++++++++++++++++++++ kernel/sched_fair.c | 2 +- 4 files changed, 26 insertions(+), 1 deletion(-)
include/linux/sched.h +1 -0
@@ -1065,6 +1065,7 @@ struct sched_domain;*/ #define WF_SYNC 0x01/* waker goes to sleep after wakup */ #define WF_FORK 0x02/* child wakeup after fork */ +#define WF_NOPREEMPT 0x04/* wakeup is not preemptive */ struct sched_class {const struct sched_class *next;
include/linux/wait.h +3 -0
@@ -140,6 +140,7 @@ static inline void __remove_wait_queue(w }void __wake_up (wait_queue_head_t *q,unsigned int mode,int nr,void *key); +void __wake_up_nopreempt (wait_queue_head_t *q,unsigned int mode,int nr,void *key);void __wake_up_locked_key (wait_queue_head_t *q,unsigned int mode,void *key);void __wake_up_sync_key (wait_queue_head_t *q,unsigned int mode,int nr,void *key);
@@ -154,8 +155,10 @@ int out_of_line_wait_on_bit_lock(void *,wait_queue_head_t *bit_waitqueue (void *,int );#define wake_up (x) __wake_up(x, TASK_NORMAL, 1, NULL) +#define wake_up_nopreempt (x) __wake_up_nopreempt(x, TASK_NORMAL, 1, NULL)#define wake_up_nr (x, nr) __wake_up(x, TASK_NORMAL, nr, NULL)#define wake_up_all (x) __wake_up(x, TASK_NORMAL, 0, NULL) +#define wake_up_all_nopreempt (x) __wake_up_nopreempt(x, TASK_NORMAL, 0, NULL)#define wake_up_locked (x) __wake_up_locked((x), TASK_NORMAL)#define wake_up_interruptible (x) __wake_up(x, TASK_INTERRUPTIBLE, 1, NULL)
kernel/sched.c +21 -0
@@ -5682,6 +5682,27 @@ void __wake_up(wait_queue_head_t *q, uns }EXPORT_SYMBOL (__wake_up); +/** +* __wake_up_nopreempt - wake up threads blocked on a waitqueue. +* @q: the waitqueue +* @mode: which threads +* @nr_exclusive: how many wake-one or wake-many threads to wake up +* @key: is directly passed to the wakeup function +* +* It may be assumed that this function implies a write memory barrier before +* changing the task state if and only if any tasks are woken up. +*/ +void __wake_up_nopreempt (wait_queue_head_t *q,unsigned int mode, +int nr_exclusive,void *key) +{ +unsigned long flags; + +spin_lock_irqsave (&q->lock, flags); +__wake_up_common (q, mode, nr_exclusive,WF_NOPREEMPT , key); +spin_unlock_irqrestore (&q->lock, flags); +} +EXPORT_SYMBOL (__wake_up_nopreempt); +/* * Same as __wake_up but called with the spinlock in wait_queue_head_t held. */
kernel/sched_fair.c +1 -1
@@ -1709,7 +1709,7 @@ static void check_preempt_wakeup(struct pse->avg_overlap < sysctl_sched_migration_cost) goto preempt; -if (!sched_feat (WAKEUP_PREEMPT )) +if (!sched_feat (WAKEUP_PREEMPT ) || (wake_flags &WF_NOPREEMPT ))return ;update_curr (cfs_rq);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 4/8] Use prepare_to_wait_exclusive() instead prepare_to_wait()
Quoted 38 lines · Mike Galbraith
Quoted 37 lines · Mike Galbraith
Quoted 21 lines · KOSAKI Motohiro
Quoted 6 lines · KOSAKI Motohiro
Quoted 4 lines
if we don't use exclusive queue,
wake_up()function wake _all_ waited task. This is simply cpu wasting.Signed-off-by KOSAKI Motohiro<[email protected]>Quoted 14 lines
Review
Quoted 7 lines
if (zone_watermark_ok(zone,
sc->order,low_wmark_pages(zone),
0, 0)) {(quoted diff) +1
-1- wake_up(wq); + wake_up_all(wq); finish_wait(wq,&wait); sc->nr_reclaimed += sc->nr_to_reclaim; return -ERESTARTSYS;I believe we want to wake the processes up one at a time here. If the queue of waiting processes is very large and the amount of excess free memory is fairly low, the first processes that wake up can take the amount of free memory back down below the threshold. The rest of the waiters should stay asleep when this happens.
Quoted 5 lines
OK.
Actually,
wake_up()andwake_up_all()aren't different so much. Although we usewake_up(), the task wake up next task before try to alloate memory. then, it's similar towake_up_all().What happens to waiters should running tasks not allocate for a while?
Quoted 6 lines
However, there are few difference. recent scheduler latency improvement effort reduce default scheduler latency target. it mean, if we have lots tasks of running state, the task have very few time slice. too frequently context switch decrease VM efficiency.
Thank you, Rik. I didn't noticewake_up()makes better performance thanwake_up_all()on current kernel.
Quoted 7 lines
Quoted 2 lines
Perhaps this is a spot where an explicit
wake_up_all_nopreempt()would be handy....Maybe something like below. I can also imagine that under _heavy_ vm pressure, it'd likely be good for throughput to not provide for sleeper fairness for these wakeups as well, as that increases vruntime spread, and thus increases preemption with no benefit in sight.
Copy/pasting some methods, and hardcoding futexes, where I know vmark loads to the point of ridiculous on my little box, it's good for ~17% throughput boost. Used prudently, something along these lines could save some thrashing when core code knows it's handling a surge. It would have a very negative effect at low to modest load though.
Hohum.
--- include/linux/completion.h | 2 include/linux/sched.h | 10 ++- include/linux/wait.h | 3 kernel/sched.c | 140 ++++++++++++++++++++++++++++++++++++--------- kernel/sched_fair.c | 32 +++++----- kernel/sched_idletask.c | 2 kernel/sched_rt.c | 6 - 7 files changed, 146 insertions(+), 49 deletions(-)
include/linux/sched.h +8 -2
@@ -1065,12 +1065,16 @@ struct sched_domain;*/ #define WF_SYNC 0x01/* waker goes to sleep after wakup */ #define WF_FORK 0x02/* child wakeup after fork */ +#define WF_BATCH 0x04/* batch wakeup, not preemptive */ +#define WF_REQUEUE 0x00/* task requeue */ +#define WF_WAKE 0x10/* task waking */ +#define WF_SLEEP 0x20/* task going to sleep */ struct sched_class {const struct sched_class *next; -void (*enqueue_task) (struct rq *rq,struct task_struct *p,int wakeup); -void (*dequeue_task) (struct rq *rq,struct task_struct *p,int sleep); +void (*enqueue_task) (struct rq *rq,struct task_struct *p,int flags); +void (*dequeue_task) (struct rq *rq,struct task_struct *p,int flags);void (*yield_task) (struct rq *rq);void (*check_preempt_curr) (struct rq *rq,struct task_struct *p,int flags);
@@ -2028,6 +2032,8 @@ extern void do_timer(unsigned long ticksextern int wake_up_state (struct task_struct *tsk,unsigned int state);extern int wake_up_process (struct task_struct *tsk); +extern int wake_up_state_batch (struct task_struct *tsk,unsigned int state); +extern int wake_up_process_batch (struct task_struct *tsk);extern void wake_up_new_task (struct task_struct *tsk,unsigned long clone_flags);#ifdef CONFIG_SMP
include/linux/wait.h +3 -0
@@ -140,6 +140,7 @@ static inline void __remove_wait_queue(w }void __wake_up (wait_queue_head_t *q,unsigned int mode,int nr,void *key); +void __wake_up_batch (wait_queue_head_t *q,unsigned int mode,int nr,void *key);void __wake_up_locked_key (wait_queue_head_t *q,unsigned int mode,void *key);void __wake_up_sync_key (wait_queue_head_t *q,unsigned int mode,int nr,void *key);
@@ -154,8 +155,10 @@ int out_of_line_wait_on_bit_lock(void *,wait_queue_head_t *bit_waitqueue (void *,int );#define wake_up (x) __wake_up(x, TASK_NORMAL, 1, NULL) +#define wake_up_batch (x) __wake_up_batch(x, TASK_NORMAL, 1, NULL)#define wake_up_nr (x, nr) __wake_up(x, TASK_NORMAL, nr, NULL)#define wake_up_all (x) __wake_up(x, TASK_NORMAL, 0, NULL) +#define wake_up_all_batch (x) __wake_up_batch(x, TASK_NORMAL, 0, NULL)#define wake_up_locked (x) __wake_up_locked((x), TASK_NORMAL)#define wake_up_interruptible (x) __wake_up(x, TASK_INTERRUPTIBLE, 1, NULL)
kernel/sched.c +113 -27
295 lines not on this page · raw message
kernel/sched_fair.c +16 -16
@@ -722,7 +722,7 @@ static void check_spread(struct cfs_rq * }static void -place_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,int initial) +place_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,int flags) {u64 vruntime = cfs_rq->min_vruntime;
@@ -732,11 +732,11 @@ place_entity(struct cfs_rq *cfs_rq, stru* little, place the new task so that it fits in the slot that * stays open at the end. */ -if (initial &&sched_feat (START_DEBIT )) +if (flags &WF_FORK &&sched_feat (START_DEBIT )) vruntime +=sched_vslice (cfs_rq, se);/* sleeps up to a single latency don't count. */ -if (!initial &&sched_feat (FAIR_SLEEPERS )) { +if (!(flags & (WF_FORK |WF_BATCH )) &&sched_feat (FAIR_SLEEPERS )) {unsigned long thresh = sysctl_sched_latency;/*
@@ -766,7 +766,7 @@ place_entity(struct cfs_rq *cfs_rq, stru }static void -enqueue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,int wakeup) +enqueue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,int flags) {/* * Update run-time statistics of the 'current'.
@@ -774,8 +774,8 @@ enqueue_entity(struct cfs_rq *cfs_rq, stupdate_curr (cfs_rq);account_entity_enqueue (cfs_rq, se); -if (wakeup) { -place_entity (cfs_rq, se,0 ); +if (flags &WF_WAKE ) { +place_entity (cfs_rq, se, flags);enqueue_sleeper (cfs_rq, se); }
@@ -801,7 +801,7 @@ static void clear_buddies(struct cfs_rq }static void -dequeue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,int sleep) +dequeue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,int flags) {/* * Update run-time statistics of the 'current'.
@@ -809,7 +809,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, stupdate_curr (cfs_rq);update_stats_dequeue (cfs_rq, se); -if (sleep) { +if (flags &WF_SLEEP ) {#ifdef CONFIG_SCHEDSTATS if (entity_is_task (se)) {struct task_struct *tsk =task_of (se);
@@ -1034,7 +1034,7 @@ static inline void hrtick_update(struct* increased. Here we update the fair scheduling stats and * then put the task into the rbtree: */ -static void enqueue_task_fair (struct rq *rq,struct task_struct *p,int wakeup) +static void enqueue_task_fair (struct rq *rq,struct task_struct *p,int flags) {struct cfs_rq *cfs_rq;struct sched_entity *se = &p->se;
@@ -1043,8 +1043,8 @@ static void enqueue_task_fair(struct rqif (se->on_rq)break ; cfs_rq =cfs_rq_of (se); -enqueue_entity (cfs_rq, se, wakeup); - wakeup =1 ; +enqueue_entity (cfs_rq, se, flags); + flags |=WF_WAKE ; }hrtick_update (rq);
@@ -1055,18 +1055,18 @@ static void enqueue_task_fair(struct rq* decreased. We remove the task from the rbtree and * update the fair scheduling stats: */ -static void dequeue_task_fair (struct rq *rq,struct task_struct *p,int sleep) +static void dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) {struct cfs_rq *cfs_rq;struct sched_entity *se = &p->se;for_each_sched_entity (se) { cfs_rq =cfs_rq_of (se); -dequeue_entity (cfs_rq, se, sleep); +dequeue_entity (cfs_rq, se, flags);/* Don't dequeue parent if it has other entities besides us */ if (cfs_rq->load.weight)break ; - sleep =1 ; + flags |=WF_SLEEP ; }hrtick_update (rq);
@@ -1709,7 +1709,7 @@ static void check_preempt_wakeup(struct pse->avg_overlap < sysctl_sched_migration_cost) goto preempt; -if (!sched_feat (WAKEUP_PREEMPT )) +if (!sched_feat (WAKEUP_PREEMPT ) || (wake_flags &WF_BATCH ))return ;update_curr (cfs_rq);
@@ -1964,7 +1964,7 @@ static void task_fork_fair(struct task_sif (curr) se->vruntime = curr->vruntime; -place_entity (cfs_rq, se,1 ); +place_entity (cfs_rq, se,WF_FORK );if (sysctl_sched_child_runs_first && curr &&entity_before (curr, se)) {/*
include/linux/completion.h +2 -0
@@ -88,6 +88,8 @@ extern bool completion_done(struct complextern void complete (struct completion *);extern void complete_all (struct completion *); +extern void complete_batch (struct completion *); +extern void complete_all_batch (struct completion *);/** * INIT_COMPLETION: - reinitialize a completion structure
kernel/sched_rt.c +3 -3
@@ -878,11 +878,11 @@ static void dequeue_rt_entity(struct sch/* * Adding/removing a task to/from a priority array: */ -static void enqueue_task_rt (struct rq *rq,struct task_struct *p,int wakeup) +static void enqueue_task_rt (struct rq *rq,struct task_struct *p,int flags) {struct sched_rt_entity *rt_se = &p->rt; -if (wakeup) +if (flags &WF_WAKE ) rt_se->timeout =0 ;enqueue_rt_entity (rt_se);
@@ -891,7 +891,7 @@ static void enqueue_task_rt(struct rq *renqueue_pushable_task (rq, p); } -static void dequeue_task_rt (struct rq *rq,struct task_struct *p,int sleep) +static void dequeue_task_rt (struct rq *rq,struct task_struct *p,int flags) {struct sched_rt_entity *rt_se = &p->rt;
kernel/sched_idletask.c +1 -1
@@ -32,7 +32,7 @@ static struct task_struct *pick_next_tas* message if some code attempts to do it: */ static void -dequeue_task_idle (struct rq *rq,struct task_struct *p,int sleep) +dequeue_task_idle (struct rq *rq,struct task_struct *p,int flags) {raw_spin_unlock_irq (&rq->lock);pr_err ("bad: scheduling from the idle thread!\n" );
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 4/8] Use prepare_to_wait_exclusive() instead prepare_to_wait()
In reply to Mike GalbraithQuoted 18 lines · Mike Galbraith
Quoted 16 lines · KOSAKI Motohiro
Quoted 16 lines · KOSAKI Motohiro
Review
Quoted 12 lines
if we don't use exclusive queue,
wake_up()function wake _all_ waited task. This is simply cpu wasting.Signed-off-by KOSAKI Motohiro<[email protected]>if (zone_watermark_ok(zone, sc->order, low_wmark_pages(zone), 0, 0)) {(quoted diff) +1
-1- wake_up(wq); + wake_up_all(wq); finish_wait(wq,&wait); sc->nr_reclaimed += sc->nr_to_reclaim; return -ERESTARTSYS;I believe we want to wake the processes up one at a time here.
Quoted 3 lines
Quoted 3 lines
Actually,
wake_up()andwake_up_all()aren't different so much. Although we usewake_up(), the task wake up next task before try to alloate memory. then, it's similar towake_up_all().
That is a good point. Maybe processes need to wait a little in this if() condition, before the wake_up(). That would give the previous process a chance to allocate memory and we can avoid waking up too many processes.
Quoted 1 line
What happens to waiters should running tasks not allocate for a while?
When a waiter is woken up, it will either:
1) see that there is enough free memory and wake up the next guy, or
2) run shrink_zone and wake up the next guy
Either way, the processes that just got woken up will ensure that the sleepers behind them in the queue will get woken up.
Signature · 7 lines
All rights reversed. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 4/8] Use prepare_to_wait_exclusive() instead prepare_to_wait()
Quoted 28 lines · Rik van Riel
Quoted 18 lines · Mike Galbraith
Quoted 16 lines · KOSAKI Motohiro
Quoted 16 lines · KOSAKI Motohiro
Review
Quoted 12 lines
if we don't use exclusive queue,
wake_up()function wake _all_ waited task. This is simply cpu wasting.Signed-off-by KOSAKI Motohiro<[email protected]>if (zone_watermark_ok(zone, sc->order, low_wmark_pages(zone), 0, 0)) {(quoted diff) +1
-1- wake_up(wq); + wake_up_all(wq); finish_wait(wq,&wait); sc->nr_reclaimed += sc->nr_to_reclaim; return -ERESTARTSYS;I believe we want to wake the processes up one at a time here.
Quoted 3 lines
Quoted 3 lines
Actually,
wake_up()andwake_up_all()aren't different so much. Although we usewake_up(), the task wake up next task before try to alloate memory. then, it's similar towake_up_all().That is a good point. Maybe processes need to wait a little in this
if()condition, before thewake_up(). That would give the previous process a chance to allocate memory and we can avoid waking up too many processes.
Quoted 8 lines
Quoted 1 line
What happens to waiters should running tasks not allocate for a while?
When a waiter is woken up, it will either:
1) see that there is enough free memory and wake up the next guy, or
2) runshrink_zoneand wake up the next guyEither way, the processes that just got woken up will ensure that the sleepers behind them in the queue will get woken up.
OK, that more or less covers my worry. From the scheduler standpoint though, you're better off turning them all loose and letting them race, _with_ the caveat than thundering herds do indeed make thunder (reason for patch). Turning them loose piecemeal spreads things out over time, which prolongs surge operations, possibly much longer than necessary. We had the same long ago with everyone waiting for kswapd to do all the work. Sticky problem, this roll-down to inevitable wait.
-Mike
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 4/8] Use prepare_to_wait_exclusive() instead prepare_to_wait()
In reply to Rik van RielQuoted 19 lines · Rik van Riel
Quoted 17 lines · Mike Galbraith
Quoted 16 lines · KOSAKI Motohiro
Quoted 16 lines · KOSAKI Motohiro
Review
Quoted 12 lines
if we don't use exclusive queue,
wake_up()function wake _all_ waited task. This is simply cpu wasting.Signed-off-by KOSAKI Motohiro<[email protected]>if (zone_watermark_ok(zone, sc->order, low_wmark_pages(zone), 0, 0)) {(quoted diff) +1
-1- wake_up(wq); + wake_up_all(wq); finish_wait(wq,&wait); sc->nr_reclaimed += sc->nr_to_reclaim; return -ERESTARTSYS;I believe we want to wake the processes up one at a time here.
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Actually,
wake_up()andwake_up_all()aren't different so much. Although we usewake_up(), the task wake up next task before try to alloate memory. then, it's similar towake_up_all().That is a good point. Maybe processes need to wait a little in this
if()condition, before thewake_up(). That would give the previous process a chance to allocate memory and we can avoid waking up too many processes.
Pondering, I think I'd at least wake NR_CPUS. If there's not enough to go round, oh darn, but if there is, you have full utilization quicker.
$.02.
-Mike
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 4/8] Use prepare_to_wait_exclusive() instead prepare_to_wait()
Quoted 24 lines · Mike Galbraith
Quoted 23 lines · Rik van Riel
Quoted 21 lines · Mike Galbraith
Quoted 20 lines · KOSAKI Motohiro
Quoted 6 lines · KOSAKI Motohiro
Quoted 4 lines
if we don't use exclusive queue,
wake_up()function wake _all_ waited task. This is simply cpu wasting.Signed-off-by KOSAKI Motohiro<[email protected]>Quoted 10 lines
Review
Quoted 7 lines
if (zone_watermark_ok(zone, sc->order, low_wmark_pages(zone), 0, 0)) {(quoted diff) +1
-1- wake_up(wq); + wake_up_all(wq); finish_wait(wq,&wait); sc->nr_reclaimed += sc->nr_to_reclaim; return -ERESTARTSYS;I believe we want to wake the processes up one at a time here.
Actually,
wake_up()andwake_up_all()aren't different so much. Although we usewake_up(), the task wake up next task before try to alloate memory. then, it's similar towake_up_all().
Quoted 7 lines
Quoted 4 lines
That is a good point. Maybe processes need to wait a little in this
if()condition, before thewake_up(). That would give the previous process a chance to allocate memory and we can avoid waking up too many processes.Pondering, I think I'd at least wake
NR_CPUS. If there's not enough to go round, oh darn, but if there is, you have full utilization quicker.
That depends on what the other CPUs in the system are doing.
If they were doing work, you've just wasted some resources.
Signature · 7 lines
All rights reversed. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 4/8] Use prepare_to_wait_exclusive() instead prepare_to_wait()
In reply to Rik van RielQuoted 19 lines · Mike Galbraith
Quoted 17 lines · KOSAKI Motohiro
Quoted 16 lines
Quoted 16 lines · KOSAKI Motohiro
Review
Quoted 12 lines
if we don't use exclusive queue,
wake_up()function wake _all_ waited task. This is simply cpu wasting.Signed-off-by KOSAKI Motohiro<[email protected]>if (zone_watermark_ok(zone, sc->order, low_wmark_pages(zone), 0, 0)) {(quoted diff) +1
-1- wake_up(wq); + wake_up_all(wq); finish_wait(wq,&wait); sc->nr_reclaimed += sc->nr_to_reclaim; return -ERESTARTSYS;I believe we want to wake the processes up one at a time here.
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Actually,
wake_up()andwake_up_all()aren't different so much. Although we usewake_up(), the task wake up next task before try to alloate memory. then, it's similar towake_up_all().That is a good point. Maybe processes need to wait a little in this
if()condition, before thewake_up(). That would give the previous process a chance to allocate memory and we can avoid waking up too many processes.
if we really need wait a bit, Mike's wake_up_batch is best, I think. It mean
- if another CPU is idle, wake up one process soon. iow, it don't make meaningless idle.
- if another CPU is busy, woken process don't start to run awhile. then, zone_watermark_ok() can calculate correct value.
Quoted 11 lines
Quoted 1 line
What happens to waiters should running tasks not allocate for a while?
When a waiter is woken up, it will either:
1) see that there is enough free memory and wake up the next guy, or
2) runshrink_zoneand wake up the next guyEither way, the processes that just got woken up will ensure that the sleepers behind them in the queue will get woken up.
Signature · 1 line
All rights reversed.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 4/8] Use prepare_to_wait_exclusive() instead prepare_to_wait()
Quoted 6 lines · KOSAKI Motohiro
if we really need wait a bit, Mike's
wake_up_batchis best, I think. It mean
- if another CPU is idle, wake up one process soon. iow, it don't make meaningless idle.
- if another CPU is busy, woken process don't start to run awhile. then,zone_watermark_ok()can calculate correct value.
Agreed, that should work great.
Signature · 7 lines
All rights reversed. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 4/8] Use prepare_to_wait_exclusive() instead prepare_to_wait()
In reply to KOSAKI MotohiroQuoted 28 lines · KOSAKI Motohiro
Quoted 19 lines · Mike Galbraith
Quoted 17 lines · KOSAKI Motohiro
Quoted 16 lines
Quoted 16 lines · KOSAKI Motohiro
Review
Quoted 12 lines
if we don't use exclusive queue,
wake_up()function wake _all_ waited task. This is simply cpu wasting.Signed-off-by KOSAKI Motohiro<[email protected]>if (zone_watermark_ok(zone, sc->order, low_wmark_pages(zone), 0, 0)) {(quoted diff) +1
-1- wake_up(wq); + wake_up_all(wq); finish_wait(wq,&wait); sc->nr_reclaimed += sc->nr_to_reclaim; return -ERESTARTSYS;I believe we want to wake the processes up one at a time here.
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Actually,
wake_up()andwake_up_all()aren't different so much. Although we usewake_up(), the task wake up next task before try to alloate memory. then, it's similar towake_up_all().That is a good point. Maybe processes need to wait a little in this
if()condition, before thewake_up(). That would give the previous process a chance to allocate memory and we can avoid waking up too many processes.
Quoted 4 lines
if we really need wait a bit, Mike's
wake_up_batchis best, I think. It mean
- if another CPU is idle, wake up one process soon. iow, it don't make meaningless idle.
Along those lines, there's also NEWIDLE balancing considerations. That idle may result in a task being pulled, which may or may not hurt a bit.
'course, if you're jamming up on memory allocation, that's the least of your worries, but every idle avoided is potentially a pull avoided.
Just a thought.
-Mike
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 4/8] Use prepare_to_wait_exclusive() instead prepare_to_wait()
In reply to KOSAKI MotohiroQuoted 22 lines · On Mon, 14 Dec 2009 21:30:19 +0900 (JST) KOSAKI Motohiro…
if we don't use exclusive queue,
wake_up()function wake _all_ waited task. This is simply cpu wasting.Signed-off-by KOSAKI Motohiro <[email protected]>--- mm/vmscan.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)mm/vmscan.c +1
-1@@ -1618,7 +1618,7 @@ static int shrink_zone_begin(struct zone *zone, struct scan_control *sc)* we would just make things slower. */ for (;;) { -prepare_to_wait (wq, &wait,TASK_UNINTERRUPTIBLE ); +prepare_to_wait_exclusive (wq, &wait,TASK_UNINTERRUPTIBLE );if (atomic_read (&zone->concurrent_reclaimers) <= max_zone_concurrent_reclaimers)
mm/vmscan.c static int shrink_zone_begin(struct zone *zone, struct scan_control *sc) · line 1635
Quoted 5 lines
mm/vmscan.c +1
-1@@ -1632,7 +1632,7 @@ static int shrink_zone_begin(struct zone *zone, struct scan_control *sc)*/ if (zone_watermark_ok (zone, sc->order,low_wmark_pages (zone),0 ,0 )) { -wake_up (wq); +wake_up_all (wq);
I think it's typo. The description in changelog says we want "wake_up". Otherwise, looks good to me.
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 5/8 Use io_schedule() instead schedule()
In reply to KOSAKI MotohiroAll task sleeping point in vmscan (e.g. congestion_wait) use io_schedule. then shrink_zone_begin use it too.
--- mm/vmscan.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
mm/vmscan.c +1 -1
@@ -1624,7 +1624,7 @@ static int shrink_zone_begin(struct zone *zone, struct scan_control *sc) max_zone_concurrent_reclaimers)break ; -schedule (); +io_schedule ();/* * If other processes freed enough memory while we waited,
Signature · 1 line
1.6.5.2
Re: [PATCH 5/8] Use io_schedule() instead schedule()
Quoted 2 lines · KOSAKI Motohiro
All task sleeping point in vmscan (e.g.
congestion_wait) useio_schedule. thenshrink_zone_beginuse it too.
I'm not sure we really are in io wait when waiting on this queue, but there's a fair chance we may be so this is a reasonable change.
Quoted 1 line
Signed-off-by KOSAKI Motohiro<[email protected]>
Signature · 7 lines
All rights reversed. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 5/8] Use io_schedule() instead schedule()
In reply to KOSAKI MotohiroQuoted 13 lines · On Mon, 14 Dec 2009 21:30:54 +0900 (JST) KOSAKI Motohiro…
All task sleeping point in vmscan (e.g.
congestion_wait) useio_schedule. thenshrink_zone_beginuse it too.Signed-off-by KOSAKI Motohiro <[email protected]>--- mm/vmscan.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)mm/vmscan.c +0
-0
mm/vmscan.c static int shrink_zone_begin(struct zone *zone, struct scan_control *sc) · line 1627
Quoted 5 lines
mm/vmscan.c +1
-1@@ -1624,7 +1624,7 @@ static int shrink_zone_begin(struct zone *zone, struct scan_control *sc) max_zone_concurrent_reclaimers)break ; -schedule (); +io_schedule ();
Hmm. We have many cond_resched which is not io_schedule in vmscan.c. In addition, if system doesn't have swap device space and out of page cache due to heavy memory pressue, VM might scan & drop pages until priority is zero or zone is unreclaimable.
I think it would be not a IO wait.
Quoted 8 lines
mm/vmscan.c +0
-0/* * If other processes freed enough memory while we waited, Signature · 1 line
1.6.5.2
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 5/8] Use io_schedule() instead schedule()
Quoted 19 lines
Quoted 16 lines · On Mon, 14 Dec 2009 21:30:54 +0900 (JST) KOSAKI Motohiro…
All task sleeping point in vmscan (e.g.
congestion_wait) useio_schedule. thenshrink_zone_beginuse it too.Signed-off-by KOSAKI Motohiro <[email protected]>--- mm/vmscan.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)mm/vmscan.c +0
-0@@ -1624,7 +1624,7 @@ static int shrink_zone_begin(struct zone *zone, struct scan_control *sc) max_zone_concurrent_reclaimers)break ;
Quoted 4 lines
mm/vmscan.c static int shrink_zone_begin(struct zone *zone, struct scan_control *sc) · line 1627
Quoted 2 lines
mm/vmscan.c +1
-1-schedule (); +io_schedule ();Hmm. We have many
cond_reschedwhich is notio_scheduleinvmscan.c.
cond_resched don't mean sleep on wait queue. it's similar to yield.
Quoted 5 lines
In addition, if system doesn't have swap device space and out of page cache due to heavy memory pressue, VM might scan & drop pages until priority is zero or zone is unreclaimable.
I think it would be not a IO wait.
two point.
1. For long time, Administrator usually watch iowait% at heavy memory pressure. I don't hope change this without reasonable reason. 2. iowait makes scheduler bonus a bit, vmscan task should have many time slice than memory consume task. it improve VM stabilization.
but I agree the benefit isn't so big. if we have reasonable reason, I don't oppose use schedule().
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 5/8] Use io_schedule() instead schedule()
Quoted 19 lines · KOSAKI Motohiro
Quoted 19 lines
Quoted 16 lines · On Mon, 14 Dec 2009 21:30:54 +0900 (JST) KOSAKI Motohiro…
All task sleeping point in vmscan (e.g.
congestion_wait) useio_schedule. thenshrink_zone_beginuse it too.Signed-off-by KOSAKI Motohiro <[email protected]>---
mm/vmscan.c | 2 +-1 files changed, 1 insertions(+), 1 deletions(-)mm/vmscan.c +0
-0@@ -1624,7 +1624,7 @@ static int shrink_zone_begin(struct zone *zone, struct scan_control *sc) max_zone_concurrent_reclaimers) break;
Quoted 6 lines
Quoted 4 lines
mm/vmscan.c static int shrink_zone_begin(struct zone *zone, struct scan_control *sc) · line 1627
Quoted 2 lines
mm/vmscan.c +1
-1- schedule(); + io_schedule();Hmm. We have many
cond_reschedwhich is notio_scheduleinvmscan.c.
cond_rescheddon't mean sleep on wait queue. it's similar to yield.
I confused it.
Thanks for correcting me. :)
Quoted 5 lines
Quoted 5 lines
In addition, if system doesn't have swap device space and out of page cache due to heavy memory pressue, VM might scan & drop pages until priority is zero or zone is unreclaimable.
Quoted 7 lines
Quoted 1 line
I think it would be not a IO wait.
two point.
1. For long time, Administrator usually watch iowait% at heavy memory pressure. I don't hope change this without reasonable reason. 2. iowait makes scheduler bonus a bit, vmscan task should have many time slice than memory consume task. it improve VM stabilization.
AFAIK, CFS scheduler doesn't give the bonus to I/O wait task any more.
Quoted 7 lines
but I agree the benefit isn't so big. if we have reasonable reason, I don't oppose use
schedule().
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 6/8 Stop reclaim quickly when the task reclaimed enough lots pages
In reply to KOSAKI MotohiroFrom latency view, There isn't any reason shrink_zones() continue to reclaim another zone's page if the task reclaimed enough lots pages.
--- mm/vmscan.c | 16 ++++++++++++---- 1 files changed, 12 insertions(+), 4 deletions(-)
mm/vmscan.c +12 -4
@@ -1654,7 +1654,7 @@ static void shrink_zone_end(struct zone *zone, struct scan_control *sc)/* * This is a basic per-zone page freer. Used by both kswapd and direct reclaim. */ -static void shrink_zone (int priority,struct zone *zone, +static int shrink_zone (int priority,struct zone *zone,struct scan_control *sc) {unsigned long nr[NR_LRU_LISTS ];
@@ -1669,7 +1669,7 @@ static void shrink_zone(int priority, struct zone *zone, ret =shrink_zone_begin (zone, sc);if (ret) -return ; +return ret;/* If we have no swap space, do not bother scanning anon pages. */ if (!sc->may_swap || (nr_swap_pages <=0 )) {
@@ -1692,6 +1692,7 @@ static void shrink_zone(int priority, struct zone *zone, &reclaim_stat->nr_saved_scan[l]); } + ret =0 ;while (nr[LRU_INACTIVE_ANON ] || nr[LRU_ACTIVE_FILE ] || nr[LRU_INACTIVE_FILE ]) {for_each_evictable_lru (l) {
@@ -1712,8 +1713,10 @@ static void shrink_zone(int priority, struct zone *zone,* with multiple processes reclaiming pages, the total * freeing target can get unreasonably large. */ -if (nr_reclaimed >= nr_to_reclaim && priority <DEF_PRIORITY ) +if (nr_reclaimed >= nr_to_reclaim && priority <DEF_PRIORITY ) { + ret = -ERESTARTSYS ;break ; + } } sc->nr_reclaimed = nr_reclaimed;
@@ -1727,6 +1730,8 @@ static void shrink_zone(int priority, struct zone *zone,throttle_vm_writeout (sc->gfp_mask);shrink_zone_end (zone, sc); + +return ret; }/*
@@ -1751,6 +1756,7 @@ static void shrink_zones(int priority, struct zonelist *zonelist,enum zone_type high_zoneidx =gfp_zone (sc->gfp_mask);struct zoneref *z;struct zone *zone; +int ret; sc->all_unreclaimable =1 ; for_each_zone_zonelist_nodemask(zone, z, zonelist, high_zoneidx,
@@ -1780,7 +1786,9 @@ static void shrink_zones(int priority, struct zonelist *zonelist, priority); } -shrink_zone (priority, zone, sc); + ret =shrink_zone (priority, zone, sc); +if (ret) +return ; } }
Signature · 9 lines
1.6.5.2 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 6/8] Stop reclaim quickly when the task reclaimed enough lots pages
Quoted 3 lines · KOSAKI Motohiro
From latency view, There isn't any reason
shrink_zones()continue to
reclaim another zone's page if the task reclaimed enough lots pages.
IIRC there is one reason - keeping equal pageout pressure between zones.
However, it may be enough if just kswapd keeps evening out the pressure, now that we limit the number of concurrent direct reclaimers in the system.
Since kswapd does not use shrink_zones ...
Quoted 1 line
Signed-off-by KOSAKI Motohiro<[email protected]>
Signature · 7 lines
All rights reversed. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 6/8] Stop reclaim quickly when the task reclaimed enough lots pages
Quoted 5 lines · KOSAKI Motohiro
Quoted 3 lines
From latency view, There isn't any reason
shrink_zones()continue to
reclaim another zone's page if the task reclaimed enough lots pages.
Quoted 8 lines
IIRC there is one reason - keeping equal pageout pressure between zones.
However, it may be enough if just kswapd keeps evening out the pressure, now that we limit the number of concurrent direct reclaimers in the system.
Since kswapd does not use
shrink_zones...
Sure. That's exactly my point.
plus, balance_pgdat() scan only one node. then zone balancing is meaingfull. but shrink_zones() scan all zone in all node. we don't need inter node balancing. it's vmscan's buisiness.
Quoted 3 lines
Quoted 1 line
Signed-off-by KOSAKI Motohiro<[email protected]>Reviewed-by Rik van Riel <[email protected]>
Thanks.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 6/8] Stop reclaim quickly when the task reclaimed enough lots pages
In reply to KOSAKI MotohiroQuoted 40 lines · On Mon, 14 Dec 2009 21:31:36 +0900 (JST) KOSAKI Motohiro…
From latency view, There isn't any reason
shrink_zones()continue to reclaim another zone's page if the task reclaimed enough lots pages.Signed-off-by KOSAKI Motohiro <[email protected]>--- mm/vmscan.c | 16 ++++++++++++---- 1 files changed, 12 insertions(+), 4 deletions(-)mm/vmscan.c +3
-2@@ -1654,7 +1654,7 @@ static void shrink_zone_end(struct zone *zone, struct scan_control *sc)/* * This is a basic per-zone page freer. Used by both kswapd and direct reclaim. */ -static void shrink_zone (int priority,struct zone *zone, +static int shrink_zone (int priority,struct zone *zone,struct scan_control *sc) {unsigned long nr[NR_LRU_LISTS ];@@ -1669,7 +1669,7 @@ static void shrink_zone(int priority, struct zone *zone, ret =shrink_zone_begin (zone, sc);if (ret) -return ; +return ret;/* If we have no swap space, do not bother scanning anon pages. */ if (!sc->may_swap || (nr_swap_pages <=0 )) {@@ -1692,6 +1692,7 @@ static void shrink_zone(int priority, struct zone *zone, &reclaim_stat->nr_saved_scan[l]); } + ret =0 ;while (nr[LRU_INACTIVE_ANON ] || nr[LRU_ACTIVE_FILE ] || nr[LRU_INACTIVE_FILE ]) {for_each_evictable_lru (l) {
mm/vmscan.c static void shrink_zone(int priority, struct zone *zone, · line 1717
Quoted 6 lines
mm/vmscan.c +2
-1@@ -1712,8 +1713,10 @@ static void shrink_zone(int priority, struct zone *zone,* with multiple processes reclaiming pages, the total * freeing target can get unreasonably large. */ -if (nr_reclaimed >= nr_to_reclaim && priority <DEF_PRIORITY ) +if (nr_reclaimed >= nr_to_reclaim && priority <DEF_PRIORITY ) { + ret = -ERESTARTSYS ;
Just nitpick.
shrink_zone's return value is matter?shrink_zones never handle that.
Quoted 39 lines
mm/vmscan.c +7
-1break ; + } } sc->nr_reclaimed = nr_reclaimed;@@ -1727,6 +1730,8 @@ static void shrink_zone(int priority, struct zone *zone,throttle_vm_writeout (sc->gfp_mask);shrink_zone_end (zone, sc); + +return ret; }/* @@ -1751,6 +1756,7 @@ static void shrink_zones(int priority, struct zonelist *zonelist,enum zone_type high_zoneidx =gfp_zone (sc->gfp_mask);struct zoneref *z;struct zone *zone; +int ret; sc->all_unreclaimable =1 ; for_each_zone_zonelist_nodemask(zone, z, zonelist, high_zoneidx,@@ -1780,7 +1786,9 @@ static void shrink_zones(int priority, struct zonelist *zonelist, priority); } -shrink_zone (priority, zone, sc); + ret =shrink_zone (priority, zone, sc); +if (ret) +return ; } }Signature · 2 lines
-- 1.6.5.2
As a matter of fact, I am worried about this patch.
My opinion is we put aside this patch until we can solve Larry's problem. We could apply this patch in future.
I don't want to see the side effect while we focus Larry's problem. But If you mind my suggestion, I also will not bother you by this nitpick.
Thanks for great cleanup and improving VM, Kosaki. :)
Signature · 2 lines
Kind regards, Minchan Kim
Re: [PATCH 6/8] Stop reclaim quickly when the task reclaimed enough lots pages
Quoted 52 lines
Quoted 40 lines · On Mon, 14 Dec 2009 21:31:36 +0900 (JST) KOSAKI Motohiro…
From latency view, There isn't any reason
shrink_zones()continue to reclaim another zone's page if the task reclaimed enough lots pages.Signed-off-by KOSAKI Motohiro <[email protected]>--- mm/vmscan.c | 16 ++++++++++++---- 1 files changed, 12 insertions(+), 4 deletions(-)mm/vmscan.c +3
-2@@ -1654,7 +1654,7 @@ static void shrink_zone_end(struct zone *zone, struct scan_control *sc)/* * This is a basic per-zone page freer. Used by both kswapd and direct reclaim. */ -static void shrink_zone (int priority,struct zone *zone, +static int shrink_zone (int priority,struct zone *zone,struct scan_control *sc) {unsigned long nr[NR_LRU_LISTS ];@@ -1669,7 +1669,7 @@ static void shrink_zone(int priority, struct zone *zone, ret =shrink_zone_begin (zone, sc);if (ret) -return ; +return ret;/* If we have no swap space, do not bother scanning anon pages. */ if (!sc->may_swap || (nr_swap_pages <=0 )) {@@ -1692,6 +1692,7 @@ static void shrink_zone(int priority, struct zone *zone, &reclaim_stat->nr_saved_scan[l]); } + ret =0 ;while (nr[LRU_INACTIVE_ANON ] || nr[LRU_ACTIVE_FILE ] || nr[LRU_INACTIVE_FILE ]) {for_each_evictable_lru (l) {mm/vmscan.c static void shrink_zone(int priority, struct zone *zone, · line 1717
Quoted 6 lines
mm/vmscan.c +2
-1@@ -1712,8 +1713,10 @@ static void shrink_zone(int priority, struct zone *zone,* with multiple processes reclaiming pages, the total * freeing target can get unreasonably large. */ -if (nr_reclaimed >= nr_to_reclaim && priority <DEF_PRIORITY ) +if (nr_reclaimed >= nr_to_reclaim && priority <DEF_PRIORITY ) { + ret = -ERESTARTSYS ;Just nitpick.
Quoted 2 lines
shrink_zone's return value is matter?
shrink_zonesnever handle that.
shrink_zones() stop vmscan quickly if ret isn't !0.
if we already scanned rather than nr_to_reclaim, we can stop vmscan.
Quoted 9 lines
As a matter of fact, I am worried about this patch.
My opinion is we put aside this patch until we can solve Larry's problem. We could apply this patch in future.
I don't want to see the side effect while we focus Larry's problem. But If you mind my suggestion, I also will not bother you by this nitpick.
Thanks for great cleanup and improving VM, Kosaki. :)
I agree with Larry's issue is highest priority.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 7/8 Use TASK_KILLABLE instead TASK_UNINTERRUPTIBLE
In reply to KOSAKI MotohiroWhen fork bomb invoke OOM Killer, almost task might start to reclaim and sleep on shrink_zone_begin(). if we use TASK_UNINTERRUPTIBLE, OOM killer can't kill such task. it mean we never recover from fork bomb.
This patch fixes it.
--- mm/vmscan.c | 20 +++++++++++++------- 1 files changed, 13 insertions(+), 7 deletions(-)
mm/vmscan.c +13 -7
@@ -1618,7 +1618,10 @@ static int shrink_zone_begin(struct zone *zone, struct scan_control *sc)* we would just make things slower. */ for (;;) { -prepare_to_wait_exclusive (wq, &wait,TASK_UNINTERRUPTIBLE ); +prepare_to_wait_exclusive (wq, &wait,TASK_KILLABLE ); + +if (fatal_signal_pending (current)) + goto stop_reclaim;if (atomic_read (&zone->concurrent_reclaimers) <= max_zone_concurrent_reclaimers)
@@ -1631,18 +1634,21 @@ static int shrink_zone_begin(struct zone *zone, struct scan_control *sc)* break out of the loop and go back to the allocator. */ if (zone_watermark_ok (zone, sc->order,low_wmark_pages (zone), -0 ,0 )) { -wake_up_all (wq); -finish_wait (wq, &wait); - sc->nr_reclaimed += sc->nr_to_reclaim; -return -ERESTARTSYS ; - } +0 ,0 )) + goto found_lots_memory; }finish_wait (wq, &wait); out:atomic_inc (&zone->concurrent_reclaimers);return 0 ; + + found_lots_memory: +wake_up_all (wq); + stop_reclaim: +finish_wait (wq, &wait); + sc->nr_reclaimed += sc->nr_to_reclaim; +return -ERESTARTSYS ; }static void shrink_zone_end (struct zone *zone,struct scan_control *sc)
Signature · 9 lines
1.6.5.2 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 7/8] Use TASK_KILLABLE instead TASK_UNINTERRUPTIBLE
Quoted 7 lines · KOSAKI Motohiro
When fork bomb invoke OOM Killer, almost task might start to reclaim and sleep on
shrink_zone_begin(). if we useTASK_UNINTERRUPTIBLE, OOM killer can't kill such task. it mean we never recover from fork bomb.This patch fixes it.
Signed-off-by KOSAKI Motohiro<[email protected]>
As with patch 4/8 I am not convinced that wake_up_all is the correct thing to do.
Other than that:
Signature · 7 lines
All rights reversed. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 7/8] Use TASK_KILLABLE instead TASK_UNINTERRUPTIBLE
In reply to KOSAKI MotohiroQuoted 7 lines · On Mon, 14 Dec 2009 21:32:18 +0900 (JST) KOSAKI Motohiro…
When fork bomb invoke OOM Killer, almost task might start to reclaim and sleep on
shrink_zone_begin(). if we useTASK_UNINTERRUPTIBLE, OOM killer can't kill such task. it mean we never recover from fork bomb.This patch fixes it.
Signed-off-by KOSAKI Motohiro <[email protected]>
I doubt wake_up_all, too. I think it's typo.
Otherwise, Looks good to me.
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 8/8 mm: Give up allocation if the task have fatal signal
In reply to KOSAKI MotohiroIn OOM case, almost processes may be in vmscan. There isn't any reason the killed process continue allocation. process exiting free lots pages rather than greedy vmscan.
--- mm/page_alloc.c | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-)
mm/page_alloc.c +8 -0
@@ -1878,6 +1878,14 @@ rebalance: goto got_pg;/* +* If the allocation is for userland page and we have fatal signal, +* there isn't any reason to continue allocation. instead, the task +* should exit soon. +*/ +if (fatal_signal_pending (current) && (gfp_mask & __GFP_HIGHMEM)) + goto nopage; + +/* * If we failed to make any progress reclaiming, then we are * running out of options and have to consider going OOM */
Signature · 9 lines
1.6.5.2 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 8/8] mm: Give up allocation if the task have fatal signal
Quoted 5 lines · KOSAKI Motohiro
In OOM case, almost processes may be in vmscan. There isn't any reason the killed process continue allocation. process exiting free lots pages rather than greedy vmscan.
Signed-off-by KOSAKI Motohiro<[email protected]>
Signature · 7 lines
All rights reversed. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 8/8] mm: Give up allocation if the task have fatal signal
In reply to KOSAKI MotohiroQuoted 14 lines · On Mon, 14 Dec 2009 21:32:58 +0900 (JST) KOSAKI Motohiro…
In OOM case, almost processes may be in vmscan. There isn't any reason the killed process continue allocation. process exiting free lots pages rather than greedy vmscan.
Signed-off-by KOSAKI Motohiro <[email protected]>--- mm/page_alloc.c | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-)mm/page_alloc.c +0
-0
mm/page_alloc.c rebalance: · line 1886
Quoted 9 lines
mm/page_alloc.c +6
-0@@ -1878,6 +1878,14 @@ rebalance: goto got_pg;/* +* If the allocation is for userland page and we have fatal signal, +* there isn't any reason to continue allocation. instead, the task +* should exit soon. +*/ +if (fatal_signal_pending (current) && (gfp_mask & __GFP_HIGHMEM)) + goto nopage;
If we jump nopage, we meets dump_stack and show_mem. Even, we can meet OOM which might kill innocent process.
Quoted 10 lines
mm/page_alloc.c +2
-0+ +/* * If we failed to make any progress reclaiming, then we are * running out of options and have to consider going OOM */ Signature · 1 line
1.6.5.2
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 8/8] mm: Give up allocation if the task have fatal signal
Quoted 10 lines
mm/page_alloc.c rebalance: · line 1886
Quoted 7 lines
mm/page_alloc.c +6
-0/* +* If the allocation is for userland page and we have fatal signal, +* there isn't any reason to continue allocation. instead, the task +* should exit soon. +*/ +if (fatal_signal_pending (current) && (gfp_mask & __GFP_HIGHMEM)) + goto nopage;If we jump nopage, we meets
dump_stackandshow_mem. Even, we can meet OOM which might kill innocent process.
Which point you oppose? noprint is better?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 8/8] mm: Give up allocation if the task have fatal signal
Quoted 11 lines · On Tue, 15 Dec 2009 09:50:47 +0900 (JST) KOSAKI Motohiro…
Quoted 10 lines
mm/page_alloc.c rebalance: · line 1886
Quoted 7 lines
mm/page_alloc.c +6
-0/* +* If the allocation is for userland page and we have fatal signal, +* there isn't any reason to continue allocation. instead, the task +* should exit soon. +*/ +if (fatal_signal_pending (current) && (gfp_mask & __GFP_HIGHMEM)) + goto nopage;If we jump nopage, we meets
dump_stackandshow_mem. Even, we can meet OOM which might kill innocent process.
Quoted 3 lines
Which point you oppose? noprint is better?
Sorry fot not clarity.
My point was following as.
First,
I don't want to print.
Why do we print stack and mem when the process receives the SIGKILL?
Second,
1) A process try to allocate anon page in do_anonymous_page.
2) A process receives SIGKILL.
3) kernel doesn't allocate page to A process by your patch.
4) do_anonymous_page returns VF_FAULT_OOM.
5) call mm_fault_error
6) call out_of_memory
7) It migth kill innocent task.
If I missed something, Pz, corret me. :)
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 8/8] mm: Give up allocation if the task have fatal signal
Quoted 34 lines
Quoted 11 lines · On Tue, 15 Dec 2009 09:50:47 +0900 (JST) KOSAKI Motohiro…
Quoted 10 lines
mm/page_alloc.c rebalance: · line 1886
Quoted 7 lines
mm/page_alloc.c +6
-0/* +* If the allocation is for userland page and we have fatal signal, +* there isn't any reason to continue allocation. instead, the task +* should exit soon. +*/ +if (fatal_signal_pending (current) && (gfp_mask & __GFP_HIGHMEM)) + goto nopage;If we jump nopage, we meets
dump_stackandshow_mem. Even, we can meet OOM which might kill innocent process.Quoted 3 lines
Which point you oppose? noprint is better?
Sorry fot not clarity.
My point was following as.First,
I don't want to print.
Why do we print stack and mem when the process receives the SIGKILL?Second,
1) A process try to allocate anon page indo_anonymous_page.
2) A process receives SIGKILL.
3) kernel doesn't allocate page to A process by your patch.
4)do_anonymous_pagereturnsVF_FAULT_OOM.
5) callmm_fault_error
6) callout_of_memory
7) It migth kill innocent task.
Quoted 1 line
If I missed something, Pz, corret me. :)
Doh, you are complely right. I had forgot recent meaning change of VM_FAULT_OOM. yes, this patch is crap. I need to remake it.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 9 lines
btw, following is mesurement result by hackbench.
================unit: sec
parameter old new
130 (5200 processes) 5.463 4.442
140 (5600 processes) 479.357 7.792
150 (6000 processes) 729.640 20.529
old mean mmotm1208
new mean mmotm1208 + Rik patch + my patch
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
On Fri, 2009-12-11 at 16:46 -0500, Rik van Riel wrote:
Rik, the latest patch appears to have a problem although I dont know what the problem is yet. When the system ran out of memory we see thousands of runnable processes and 100% system time:
9420 2 29824 79856 62676 19564 0 0 0 0 8054 379 0
100 0 0 0
9420 2 29824 79368 62292 19564 0 0 0 0 8691 413 0
100 0 0 0
9421 1 29824 79780 61780 19820 0 0 0 0 8928 408 0
100 0 0 0
The system would not respond so I dont know whats going on yet. I'll add debug code to figure out why its in that state as soon as I get access to the hardware.
Larry
Quoted 211 lines
Under very heavy multi-process workloads, like AIM7, the VM can get into trouble in a variety of ways. The trouble start when there are hundreds, or even thousands of processes active in the page reclaim code.
Not only can the system suffer enormous slowdowns because of lock contention (and conditional reschedules) between thousands of processes in the page reclaim code, but each process will try to free up to
SWAP_CLUSTER_MAXpages, even when the system already has lots of memory free.It should be possible to avoid both of those issues at once, by simply limiting how many processes are active in the page reclaim code simultaneously.
If too many processes are active doing page reclaim in one zone, simply go to sleep in
shrink_zone().On wakeup, check whether enough memory has been freed already before jumping into the page reclaim code ourselves. We want to use the same threshold here that is used in the page allocator for deciding whether or not to call the page reclaim code in the first place, otherwise some unlucky processes could end up freeing memory for the rest of the system.
Reported-by Larry Woodman <[email protected]>Signed-off-by Rik van Riel <[email protected]>---
v2:
- fix typos insysctl.cand vm.txt
- move the code insysctl.cout from under the ifdef
- only __GFP_FS|__GFP_IO tasks can waitDocumentation/sysctl/vm.txt | 18 ++++++++++++++ include/linux/mmzone.h | 4 +++ include/linux/swap.h | 1 + kernel/sysctl.c | 7 +++++ mm/page_alloc.c | 3 ++ mm/vmscan.c | 40 +++++++++++++++++++++++++++++++++ 6 files changed, 73 insertions(+), 0 deletions(-)Documentation/sysctl/vm.txt +18
-0@@ -32,6 +32,7 @@ Currently, these files are in /proc/sys/vm: - legacy_va_layout - lowmem_reserve_ratio - max_map_count +- max_zone_concurrent_reclaimers - memory_failure_early_kill - memory_failure_recovery - min_free_kbytes@@ -278,6 +279,23 @@ The default value is 65536. ============================================================= +max_zone_concurrent_reclaimers: + +The number of processes that are allowed to simultaneously reclaim +memory from a particular memory zone. + +With certain workloads, hundreds of processes end up in the page +reclaim code simultaneously. This can cause large slowdowns due +to lock contention, freeing of way too much memory and occasionally +false OOM kills. + +To avoid these problems, only allow a smaller number of processes +to reclaim pages from each memory zone simultaneously. + +The default value is 8. + +============================================================= + memory_failure_early_kill: Control how to kill processes when uncorrected memory error (typicallyinclude/linux/mmzone.h +4
-0@@ -345,6 +345,10 @@ struct zone {/* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS ]; +/* Number of processes running page reclaim code on this zone. */ +atomic_t concurrent_reclaimers; +wait_queue_head_t reclaim_wait; +/* * prev_priority holds the scanning priority for this zone. It is * defined as the scanning priority at which we achieved our reclaim include/linux/swap.h +1
-0@@ -254,6 +254,7 @@ extern unsigned long shrink_all_memory(unsigned long nr_pages);extern int vm_swappiness;extern int remove_mapping (struct address_space *mapping,struct page *page);extern long vm_total_pages; +extern int max_zone_concurrent_reclaimers;#ifdef CONFIG_NUMA extern int zone_reclaim_mode;kernel/sysctl.c +7
-0@@ -1271,6 +1271,13 @@ static struct ctl_table vm_table[] = { .extra2 = &one, },#endif + { + .procname = "max_zone_concurrent_reclaimers", + .data = &max_zone_concurrent_reclaimers, + .maxlen = sizeof(max_zone_concurrent_reclaimers), + .mode = 0644, + .proc_handler = &proc_dointvec, + },/* * NOTE: do not add new entries to this table unless you have read mm/page_alloc.c +3
-0@@ -3852,6 +3852,9 @@ static void __paginginit free_area_init_core(struct pglist_data *pgdat, zone->prev_priority =DEF_PRIORITY ; +atomic_set (&zone->concurrent_reclaimers,0 ); +init_waitqueue_head (&zone->reclaim_wait); +zone_pcp_init (zone);for_each_lru (l) {INIT_LIST_HEAD (&zone->lru[l].list);mm/vmscan.c +40
-1@@ -40,6 +40,7 @@#include <linux/memcontrol.h> #include <linux/delayacct.h> #include <linux/sysctl.h> +#include <linux/wait.h> #include <asm/tlbflush.h> #include <asm/div64.h> @@ -129,6 +130,17 @@ struct scan_control {int vm_swappiness =60 ;long vm_total_pages;/* The total number of pages which the VM controls */ +/* +* Maximum number of processes concurrently running the page +* reclaim code in a memory zone. Having too many processes +* just results in them burning CPU time waiting for locks, +* so we're better off limiting page reclaim to a sane number +* of processes at a time. We do this per zone so local node +* reclaim on one NUMA node will not block other nodes from +* making progress. +*/ +int max_zone_concurrent_reclaimers =8 ; +static LIST_HEAD (shrinker_list);static DECLARE_RWSEM (shrinker_rwsem);@@ -1600,6 +1612,31 @@ static void shrink_zone(int priority, struct zone *zone,struct zone_reclaim_stat *reclaim_stat =get_reclaim_stat (zone, sc);int noswap =0 ; +if (!current_is_kswapd () &&atomic_read (&zone->concurrent_reclaimers) > + max_zone_concurrent_reclaimers && + (sc->gfp_mask & (__GFP_IO|__GFP_FS)) == + (__GFP_IO|__GFP_FS)) { +/* +* Do not add to the lock contention if this zone has +* enough processes doing page reclaim already, since +* we would just make things slower. +*/ +sleep_on (&zone->reclaim_wait); + +/* +* If other processes freed enough memory while we waited, +* break out of the loop and go back to the allocator. +*/ +if (zone_watermark_ok (zone, sc->order,low_wmark_pages (zone), +0 ,0 )) { +wake_up (&zone->reclaim_wait); + sc->nr_reclaimed += nr_to_reclaim; +return ; + } + } + +atomic_inc (&zone->concurrent_reclaimers); +/* If we have no swap space, do not bother scanning anon pages. */ if (!sc->may_swap || (nr_swap_pages <=0 )) { noswap =1 ;@@ -1655,6 +1692,9 @@ static void shrink_zone(int priority, struct zone *zone,shrink_active_list (SWAP_CLUSTER_MAX , zone, sc, priority,0 );throttle_vm_writeout (sc->gfp_mask); + +atomic_dec (&zone->concurrent_reclaimers); +wake_up (&zone->reclaim_wait); }/* --
Quoted 4 lines
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 14 lines · Rik van Riel
Rik, the latest patch appears to have a problem although I dont know what the problem is yet. When the system ran out of memory we see thousands of runnable processes and 100% system time:
9420 2 29824 79856 62676 19564 0 0 0 0 8054 379 0
100 0 0 0
9420 2 29824 79368 62292 19564 0 0 0 0 8691 413 0
100 0 0 0
9421 1 29824 79780 61780 19820 0 0 0 0 8928 408 0
100 0 0 0
Quoted 5 lines
The system would not respond so I dont know whats going on yet. I'll add debug code to figure out why its in that state as soon as I get access to the hardware.
Larry
There are 9421 running processces. it mean concurrent task limitation don't works well. hmm?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
A message that is not in the archive (sent off-list or not yet received): [email protected]
PATCH v2 1/1 vmscan: limit concurrent reclaimers in shrink_zone
Quoted 5 lines · KOSAKI Motohiro
(offlist)
Larry, May I ask current status of following your issue?
I don't reproduce it. and I don't hope to keep lots patch are up in the air.
Yes, sorry for the delay but I dont have direct or exclusive access to these large systems and workloads. As far as I can tell this patch series does help prevent total system hangs running AIM7. I did have trouble with the early postings mostly due to using sleep_on() and wakeup() but those appear to be fixed.
However, I did add more debug code and see ~10000 processes blocked in shrink_zone_begin().
This is expected but bothersome, practically all of the processes remain runnable for the entire
duration of these AIM runs. Collectively all these runnable processes overwhelm the VM system. There are many more runnable processes now than were ever seen before, ~10000 now versus ~100 on RHEL5(2.6.18 based). So, we have also been experimenting around with some of the
CFS scheduler tunables to see of this is responsible...
Quoted 4 lines
plus, I integrated
page_referenced()improvement patch series and limit concurrent reclaimers patch series privately. I plan to post it to lkml at this week end. comments are welcome.
The only problem I noticed with the page_referenced patch was an increase in the try_to_unmap() failures which causes more re-activations. This is very obvious with the using tracepoints I have posted over the past few months but they were never included. I didnt get a chance to figure out the exact cause due to access to the hardware and workload. This patch series also seems to help the overall stalls in the VM system.
Quoted 33 lines
changelog from last post:
- remake limit concurrent reclaimers series and sort out its patch order
- change default max concurrent reclaimers from 8 tonum_online_cpu(). it mean, Andi only talked negative feeling comment in last post. he dislike constant default value. plus, overnum_online_cpu()is really silly. iow, it is really low risk.
(probably we might change default value. as far as I mesure, small
value makes better benchmark result. but I'm not sure small value don't make regression)
- Improve OOM and SIGKILL behavior.
(because RHEL5's vmscan hasTIF_MEMDIErecovering logic, but
current mainline doesn't. I don't hope RHEL6 has regression)Quoted 16 lines · Rik van Riel
Rik, the latest patch appears to have a problem although I dont know what the problem is yet. When the system ran out of memory we see thousands of runnable processes and 100% system time:
9420 2 29824 79856 62676 19564 0 0 0 0 8054 379 0
100 0 0 09420 2 29824 79368 62292 19564 0 0 0 0 8691 413 0 100 0 0 0 9421 1 29824 79780 61780 19820 0 0 0 0 8928 408 0 100 0 0 0The system would not respond so I dont know whats going on yet. I'll add debug code to figure out why its in that state as soon as I get access to the hardware.
This was in response to Rik's first patch and seems to be fixed by the latest path set.
Finally, having said all that, the system still struggles reclaiming memory with ~10000 processes trying at the same time, you fix one bottleneck and it moves somewhere else. The latest run showed all but one running process spinning in page_lock_anon_vma() trying for the anon_vma_lock. I noticed that there are ~5000 vma's linked to one anon_vma, this seems excessive!!!
I changed the anon_vma->lock to a rwlock_t and page_lock_anon_vma() to use read_lock() so multiple callers could execute the page_reference_anon code. This seems to help quite a bit.
Quoted 1 line
Quoted 1 line
Larry
include/linux/rmap.h +3 -3
@@ -25,7 +25,7 @@* pointing to this anon_vma once its vma list is empty. */ struct anon_vma { -spinlock_t lock;/* Serialize access to vma list */ +rwlock_t lock;/* Serialize access to vma list */ /* * NOTE: the LSB of the head.next is set by * mm_take_all_locks() _after_ taking the above lock. So the
@@ -43,14 +43,14 @@ static inline void anon_vma_lock(struct vm_area_struct *vma) {struct anon_vma *anon_vma = vma->anon_vma;if (anon_vma) -spin_lock (&anon_vma->lock); +write_lock (&anon_vma->lock); }static inline void anon_vma_unlock (struct vm_area_struct *vma) {struct anon_vma *anon_vma = vma->anon_vma;if (anon_vma) -spin_unlock (&anon_vma->lock); +write_unlock (&anon_vma->lock); }/*
mm/migrate.c +2 -2
@@ -183,12 +183,12 @@ static void remove_anon_migration_ptes(struct page *old, struct page *new)* We hold the mmap_sem lock. So no need to call page_lock_anon_vma. */ anon_vma = (struct anon_vma *) (mapping -PAGE_MAPPING_ANON ); -spin_lock (&anon_vma->lock); +write_lock (&anon_vma->lock);list_for_each_entry (vma, &anon_vma->head, anon_vma_node)remove_migration_pte (vma, old, new); -spin_unlock (&anon_vma->lock); +write_unlock (&anon_vma->lock); }/*
mm/mmap.c +4 -4
@@ -592,7 +592,7 @@ again: remove_next = 1 + (end > next->vm_end);if (vma->anon_vma && (insert || importer || start != vma->vm_start)) anon_vma = vma->anon_vma;if (anon_vma) { -spin_lock (&anon_vma->lock); +write_lock (&anon_vma->lock);/* * Easily overlooked: when mprotect shifts the boundary, * make sure the expanding vma has anon_vma set if the
@@ -646,7 +646,7 @@ again: remove_next = 1 + (end > next->vm_end); }if (anon_vma) -spin_unlock (&anon_vma->lock); +write_unlock (&anon_vma->lock);if (mapping)spin_unlock (&mapping->i_mmap_lock);
@@ -2442,7 +2442,7 @@ static void vm_lock_anon_vma(struct mm_struct *mm, struct anon_vma *anon_vma)* The LSB of head.next can't change from under us * because we hold the mm_all_locks_mutex. */ -spin_lock_nest_lock (&anon_vma->lock, &mm->mmap_sem); +write_lock (&anon_vma->lock);/* * We can safely modify head.next after taking the * anon_vma->lock. If some other vma in this mm shares
@@ -2558,7 +2558,7 @@ static void vm_unlock_anon_vma(struct anon_vma *anon_vma)if (!__test_and_clear_bit (0 , (unsigned long *) &anon_vma->head.next))BUG (); -spin_unlock (&anon_vma->lock); +write_unlock (&anon_vma->lock); } }
mm/rmap.c +9 -9
@@ -116,7 +116,7 @@ int anon_vma_prepare(struct vm_area_struct *vma)return -ENOMEM ; allocated = anon_vma; } -spin_lock (&anon_vma->lock); +write_lock (&anon_vma->lock);/* page_table_lock to protect against threads */ spin_lock (&mm->page_table_lock);
@@ -127,7 +127,7 @@ int anon_vma_prepare(struct vm_area_struct *vma) }spin_unlock (&mm->page_table_lock); -spin_unlock (&anon_vma->lock); +write_unlock (&anon_vma->lock);if (unlikely (allocated))anon_vma_free (allocated); }
@@ -153,9 +153,9 @@ void anon_vma_link(struct vm_area_struct *vma)struct anon_vma *anon_vma = vma->anon_vma;if (anon_vma) { -spin_lock (&anon_vma->lock); +write_lock (&anon_vma->lock);list_add_tail (&vma->anon_vma_node, &anon_vma->head); -spin_unlock (&anon_vma->lock); +write_unlock (&anon_vma->lock); } }
@@ -167,12 +167,12 @@ void anon_vma_unlink(struct vm_area_struct *vma)if (!anon_vma)return ; -spin_lock (&anon_vma->lock); +write_lock (&anon_vma->lock);list_del (&vma->anon_vma_node);/* We must garbage collect the anon_vma if it's empty */ empty =list_empty (&anon_vma->head); -spin_unlock (&anon_vma->lock); +write_unlock (&anon_vma->lock);if (empty)anon_vma_free (anon_vma);
@@ -182,7 +182,7 @@ static void anon_vma_ctor(void *data) {struct anon_vma *anon_vma = data; -spin_lock_init (&anon_vma->lock); +rwlock_init (&anon_vma->lock);INIT_LIST_HEAD (&anon_vma->head); }
@@ -209,7 +209,7 @@ struct anon_vma *page_lock_anon_vma(struct page *page) goto out; anon_vma = (struct anon_vma *) (anon_mapping -PAGE_MAPPING_ANON ); -spin_lock (&anon_vma->lock); +read_lock (&anon_vma->lock);return anon_vma; out:rcu_read_unlock ();
@@ -218,7 +218,7 @@ out:void page_unlock_anon_vma (struct anon_vma *anon_vma) { -spin_unlock (&anon_vma->lock); +read_unlock (&anon_vma->lock);rcu_read_unlock (); }
Quoted 7 lines · Larry Woodman
Quoted 3 lines
Quoted 3 lines
The system would not respond so I dont know whats going on yet. I'll add debug code to figure out why its in that state as soon as I get access to the hardware.
This was in response to Rik's first patch and seems to be fixed by the latest path set.
Quoted 8 lines
Finally, having said all that, the system still struggles reclaiming memory with
~10000 processes trying at the same time, you fix one bottleneck and it moves
somewhere else. The latest run showed all but one running process spinning inpage_lock_anon_vma()trying for theanon_vma_lock. I noticed that there are ~5000 vma's linked to oneanon_vma, this seems excessive!!!
I have some ideas on how to keep processes waiting better on the per zone reclaim_wait waitqueue.
For one, we should probably only do the lots-free wakeup if we have more than zone->pages_high free pages in the zone - having each of the waiters free some memory one after another should not be a problem as long as we do not have too much free memory in the zone.
Currently it's a hair trigger, with the threshold for processes going into the page reclaim path and processes exiting it "plenty free" being exactly the same.
Some hysteresis there could help.
Signature · 1 line
All rights reversed.
After removing some more immediate bottlenecks with the patches by Kosaki and me, Larry ran into a really big one:
Quoted 9 lines · Larry Woodman
Finally, having said all that, the system still struggles reclaiming memory with ~10000 processes trying at the same time, you fix one bottleneck and it moves somewhere else. The latest run showed all but one running process spinning in
page_lock_anon_vma()trying for theanon_vma_lock. I noticed that there are ~5000 vma's linked to oneanon_vma, this seems excessive!!!I changed the
anon_vma->lockto arwlock_tandpage_lock_anon_vma()to useread_lock()so multiple callers could execute thepage_reference_anoncode. This seems to help quite a bit.
The system has 10000 processes, all of which are child processes of the same parent.
Pretty much all memory is anonymous memory.
This means that pretty much every anonymous page in the system:
1) belongs to just one process, but
2) belongs to an anon_vma which is attached to 10,000 VMAs!
This results in page_referenced scanning 10,000 VMAs for every page, despite the fact that each page is typically only mapped into one process.
This seems to be our real scalability issue.
The only way out I can think is to have a new anon_vma when we start a child process and to have COW place new pages in the new anon_vma.
However, this is a bit of a paradigm shift in our object rmap system and I am wondering if somebody else has a better idea :)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 35 lines · Rik van Riel
After removing some more immediate bottlenecks with the patches by Kosaki and me, Larry ran into a really big one:
Quoted 9 lines · Larry Woodman
Finally, having said all that, the system still struggles reclaiming memory with
~10000 processes trying at the same time, you fix one bottleneck and it moves
somewhere else. The latest run showed all but one running process spinning inpage_lock_anon_vma()trying for theanon_vma_lock. I noticed that there are ~5000 vma's linked to oneanon_vma, this seems excessive!!!Quoted 3 lines
I changed the
anon_vma->lockto arwlock_tandpage_lock_anon_vma()to useread_lock()so multiple callers could execute thepage_reference_anoncode. This seems to help quite a bit.The system has 10000 processes, all of which are child processes of the same parent.
Pretty much all memory is anonymous memory.
This means that pretty much every anonymous page in the system:
1) belongs to just one process, but
2) belongs to ananon_vmawhich is attached to 10,000 VMAs!This results in
page_referencedscanning 10,000 VMAs for every page, despite the fact that each page is typically only mapped into one process.This seems to be our real scalability issue.
Quoted 7 lines
The only way out I can think is to have a new
anon_vmawhen we start a child process and to have COW place new pages in the newanon_vma.However, this is a bit of a paradigm shift in our object rmap system and I am wondering if somebody else has a better idea :)
Please first clarify whether what Larry is running is actually a workload that people need to behave well in real life.
Quoted 1 line
From time to time such cases have been constructed, but we've
usually found better things to do than solve them, because they've been no more than academic problems.
I'm not asserting that this one is purely academic, but I do think we need more than an artificial case to worry much about it.
An rwlock there has been proposed on several occasions, but we resist because that change benefits this case but performs worse on more common cases (I believe: no numbers to back that up).
Substitute a MAP_SHARED file underneath those 10000 vmas, and don't you have an equal problem with the prio_tree, which would be harder to solve than the anon_vma case?
Hugh
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 39 lines · Hugh Dickins
Quoted 37 lines · Rik van Riel
After removing some more immediate bottlenecks with the patches by Kosaki and me, Larry ran into a really big one:
Quoted 11 lines · Larry Woodman
Finally, having said all that, the system still struggles reclaiming memory with
~10000 processes trying at the same time, you fix one bottleneck and it moves somewhere else. The latest run showed all but one running process spinning inpage_lock_anon_vma()trying for theanon_vma_lock. I noticed that there are ~5000 vma's linked to oneanon_vma, this seems excessive!!!I changed the
anon_vma->lockto arwlock_tandpage_lock_anon_vma()to useread_lock()so multiple callers could execute thepage_reference_anoncode. This seems to help quite a bit.The system has 10000 processes, all of which are child processes of the same parent.
Pretty much all memory is anonymous memory.
This means that pretty much every anonymous page in the system:
1) belongs to just one process, but
2) belongs to ananon_vmawhich is attached to 10,000 VMAs!This results in
page_referencedscanning 10,000 VMAs for every page, despite the fact that each page is typically only mapped into one process.This seems to be our real scalability issue.
The only way out I can think is to have a new
anon_vmawhen we start a child process and to have COW place new pages in the newanon_vma.
Quoted 6 lines
Quoted 3 lines
However, this is a bit of a paradigm shift in our object rmap system and I am wondering if somebody else has a better idea :)
Please first clarify whether what Larry is running is actually a workload that people need to behave well in real life.
AIM7 is fairly artificial, but real life workloads like Oracle, PostgreSQL and Apache can also fork off large numbers of child processes, which also cause the system to end up with lots of VMAs attached to the anon_vmas which all the anonymous pages belong to.
10,000 is fairly extreme, but very large Oracle
workloads can get up to 1,000 or 2,000 today.
This number is bound to grow in the future.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 2 lines · Hugh Dickins
Please first clarify whether what Larry is running is actually a workload that people need to behave well in real life.
Anything with 10000 connections using a connection-per-thread/process model, should use threads if good performance are expected, processes not. Most things that are using multi-process design will never use one-connection-per-process design (yes there are exceptions and no we can't expect to fix those as they're proprietary ;). So I'm not particularly worried.
Also make sure this also happens on older kernels, newer kernels uses rmap chains and mangle over ptes even when there's no VM pressure for no good reason. Older kernels would only hit on the anon_vma chain on any anon page, only after this anon page was converted to swapcache and swap was hit, so it makes a whole lot of difference. Anon_vma chains should only be touched after we are I/O bound if anybody is to expect decent performance out of the kernel.
Quoted 2 lines
I'm not asserting that this one is purely academic, but I do think we need more than an artificial case to worry much about it.
Tend to agree.
Quoted 3 lines
An rwlock there has been proposed on several occasions, but we resist because that change benefits this case but performs worse on more common cases (I believe: no numbers to back that up).
I think rwlock for anon_vma is a must. Whatever higher overhead of the fast path with no contention is practically zero, and in large smp it allows rmap on long chains to run in parallel, so very much worth it because downside is practically zero and upside may be measurable instead in certain corner cases. I don't think it'll be enough, but I definitely like it.
Quoted 3 lines
Substitute a
MAP_SHAREDfile underneath those 10000 vmas, and don't you have an equal problem with theprio_tree, which would be harder to solve than theanon_vmacase?
That is a very good point.
Rik suggested to me to have a cowed newly allocated page to use its own anon_vma. Conceptually Rik's idea is fine one, but the only complication then is how to chain the same vma into multiple anon_vma (in practice insert/removal will be slower and more metadata will be needed for additional anon_vmas and vams queued in more than anon_vma). But this only will help if the mapcount of the page is 1, if the mapcount is 10000 no change to anon_vma or prio_tree will solve this, and we've to start breaking the rmap loop after 64 test_and_clear_young instead to mitigate the inefficiency on pages that are used and never will go into swap and so where wasting 10000 cachelines just because this used page eventually is in the tail position of the lru uis entirely wasted.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 6 lines · Andrea Arcangeli
Quoted 3 lines · Hugh Dickins
An rwlock there has been proposed on several occasions, but we resist because that change benefits this case but performs worse on more common cases (I believe: no numbers to back that up).
Quoted 6 lines
I think rwlock for
anon_vmais a must. Whatever higher overhead of the fast path with no contention is practically zero, and in large smp it allows rmap on long chains to run in parallel, so very much worth it because downside is practically zero and upside may be measurable instead in certain corner cases. I don't think it'll be enough, but I definitely like it.
I agree, changing the anon_vma lock to an rwlock should work a lot better than what we have today. The tradeoff is a tiny slowdown in medium contention cases, at the benefit of avoiding catastrophic slowdown in some cases.
With Nick Piggin's fair rwlocks, there should be no issue at all.
Quoted 8 lines
Rik suggested to me to have a cowed newly allocated page to use its own
anon_vma. Conceptually Rik's idea is fine one, but the only complication then is how to chain the same vma into multipleanon_vma(in practice insert/removal will be slower and more metadata will be needed for additionalanon_vmasand vams queued in more thananon_vma). But this only will help if the mapcount of the page is 1, if the mapcount is 10000 no change toanon_vmaorprio_treewill solve this,
It's even more complex than this for anonymous pages.
Anonymous pages get COW copied in child (and parent) processes, potentially resulting in one page, at each offset into the anon_vma, for every process attached to the anon_vma.
As a result, with 10000 child processes, page_referenced can end up searching through 10000 VMAs even for pages with a mapcount of 1!
Signature · 7 lines
All rights reversed. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 15 lines
Quoted 5 lines · KOSAKI Motohiro
(offlist)
Larry, May I ask current status of following your issue?
I don't reproduce it. and I don't hope to keep lots patch are up in the air.Yes, sorry for the delay but I dont have direct or exclusive access to these large systems
and workloads. As far as I can tell this patch series does help prevent total system
hangs running AIM7. I did have trouble with the early postings mostly due to usingsleep_on()
andwakeup()but those appear to be fixed.
Quoted 11 lines
However, I did add more debug code and see ~10000 processes blocked in
shrink_zone_begin().
This is expected but bothersome, practically all of the processes remain runnable for the entire
duration of these AIM runs. Collectively all these runnable processes overwhelm the VM system.
There are many more runnable processes now than were ever seen before, ~10000 now versus
~100 on RHEL5(2.6.18 based). So, we have also been experimenting around with some of the
CFS scheduler tunables to see of this is responsible...
What point you bother? throughput, latency or somethingelse? Actually, unfairness itself is right thing from VM view. because perfectly fairness VM easily makes livelock. (e.g. process-A swap out process-B's page, parocess-B swap out process-A's page). swap token solve above simplest case. but run many process easily makes similar circulation dependency. recovering from heavy memory pressure need lots unfairness.
Of cource, if the unfairness makes performance regression, it's bug. it should be fixed.
Quoted 10 lines
The only problem I noticed with the
page_referencedpatch was an increase in thetry_to_unmap()failures which causes more re-activations. This is very obvious with
the using tracepoints I have posted over the past few months but they were never
included. I didnt get a chance to figure out the exact cause due to access to the hardware
and workload. This patch series also seems to help the overall stalls in the VM system.
I (and many VM developer) don't forget your tracepoint effort. we only hope to solve the regression at first.
Quoted 30 lines
Quoted 12 lines
Quoted 12 lines
Rik, the latest patch appears to have a problem although I dont know what the problem is yet. When the system ran out of memory we see thousands of runnable processes and 100% system time:
9420 2 29824 79856 62676 19564 0 0 0 0 8054 379 0
100 0 0 0
9420 2 29824 79368 62292 19564 0 0 0 0 8691 413 0
100 0 0 0
9421 1 29824 79780 61780 19820 0 0 0 0 8928 408 0
100 0 0 0Quoted 4 lines
Quoted 4 lines
The system would not respond so I dont know whats going on yet. I'll add debug code to figure out why its in that state as soon as I get access to the hardware.
This was in response to Rik's first patch and seems to be fixed by the latest path set.
Finally, having said all that, the system still struggles reclaiming memory with
~10000 processes trying at the same time, you fix one bottleneck and it moves
somewhere else. The latest run showed all but one running process spinning inpage_lock_anon_vma()trying for theanon_vma_lock. I noticed that there are
~5000 vma's linked to oneanon_vma, this seems excessive!!!
Quoted 3 lines
I changed the
anon_vma->lockto arwlock_tandpage_lock_anon_vma()to useread_lock()so multiple callers could execute thepage_reference_anoncode. This seems to help quite a bit.
Ug. no. rw-spinlock is evil. please don't use it. rw-spinlock has bad performance characteristics, plenty read_lock block write_lock for very long time.
and I would like to confirm one thing. anon_vma design didn't change for long year. Is this really performance regression? Do we strike right regression point?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 16 lines · KOSAKI Motohiro
Quoted 12 lines
KOSAKI Motohiro wrote:
Finally, having said all that, the system still struggles reclaiming memory with
~10000 processes trying at the same time, you fix one bottleneck and it moves
somewhere else. The latest run showed all but one running process spinning inpage_lock_anon_vma()trying for theanon_vma_lock. I noticed that there are
~5000 vma's linked to oneanon_vma, this seems excessive!!!Quoted 3 lines
I changed the
anon_vma->lockto arwlock_tandpage_lock_anon_vma()to useread_lock()so multiple callers could execute thepage_reference_anoncode. This seems to help quite a bit.
Quoted 7 lines
Ug. no. rw-spinlock is evil. please don't use it. rw-spinlock has bad performance characteristics, plenty
read_lockblockwrite_lockfor very long time.and I would like to confirm one thing.
anon_vmadesign didn't change for long year. Is this really performance regression? Do we strike right regression point?
In 2.6.9 and 2.6.18 the system would hit different contention points before getting to the anon_vma lock. Now that we've gotten the other contention points out of the way, this one has finally been exposed.
Signature · 7 lines
All rights reversed. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 33 lines · Rik van Riel
Under very heavy multi-process workloads, like AIM7, the VM can get into trouble in a variety of ways. The trouble start when there are hundreds, or even thousands of processes active in the page reclaim code.
Not only can the system suffer enormous slowdowns because of lock contention (and conditional reschedules) between thousands of processes in the page reclaim code, but each process will try to free up to
SWAP_CLUSTER_MAXpages, even when the system already has lots of memory free.It should be possible to avoid both of those issues at once, by simply limiting how many processes are active in the page reclaim code simultaneously.
If too many processes are active doing page reclaim in one zone, simply go to sleep in
shrink_zone().On wakeup, check whether enough memory has been freed already before jumping into the page reclaim code ourselves. We want to use the same threshold here that is used in the page allocator for deciding whether or not to call the page reclaim code in the first place, otherwise some unlucky processes could end up freeing memory for the rest of the system.
Control how to kill processes when uncorrected memory error (typically
include/linux/mmzone.h +0
-0
include/linux/mmzone.h struct zone { · line 352
Quoted 8 lines
include/linux/mmzone.h +4
-0@@ -345,6 +345,10 @@ struct zone {/* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS ]; +/* Number of processes running page reclaim code on this zone. */ +atomic_t concurrent_reclaimers; +wait_queue_head_t reclaim_wait; +
Counting semaphore?
Signature · 8 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 6 lines · Avi Kivity
Quoted 4 lines
+ /* Number of processes running page reclaim code on this zone. */
+atomic_tconcurrent_reclaimers;
+wait_queue_head_treclaim_wait;
+Counting semaphore?
I don't see a safe way to adjust a counting semaphore through /proc/sys.
Signature · 7 lines
All rights reversed. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 11 lines · Rik van Riel
Quoted 6 lines · Avi Kivity
Quoted 4 lines
+ /* Number of processes running page reclaim code on this zone. */
+atomic_tconcurrent_reclaimers;
+wait_queue_head_treclaim_wait;
+Counting semaphore?
I don't see a safe way to adjust a counting semaphore through
/proc/sys.
True. Maybe it's worthwhile to add such a way if this pattern has more uses.
Signature · 8 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>