mm: swap out anonymous page regardless of laptop_mode
PATCH 1/1 mm: swap out anonymous page regardless of laptop_mode
Hi Luigi,
Sorry for really really late response.
Today I have a time to look at this problem and it seems to found the problem. By your help, I can reprocude this problem easily on my KVM machine and this patch solves the problem.
Could you test below patch? Although this patch is based on recent mmotm, I guess you can apply it easily to 3.4.
No problem at all---as I mentioned, we stopped using laptop_mode, so this is no longer an issue for us.
I should be able to test the patch for you in the next 2-3 days. I will let you know if I run into problems.
Thanks!
Luigi
Quoted 175 lines · Minchan Kim
Hi Luigi,
Sorry for really really late response.
Today I have a time to look at this problem and it seems to found the problem. By your help, I can reprocude this problem easily on my KVM machine and this patch solves the problem.Could you test below patch? Although this patch is based on recent mmotm, I guess you can apply it easily to 3.4.
From
f74fdf644bec3e7875d245154db953b47b6c9594Mon Sep 17 00:00:00 2001
From: Minchan Kim <[email protected]>
Date: Tue, 8 Jan 2013 16:23:31 +0900
Subject: [PATCH] mm: swap out anonymous page regardless oflaptop_modeRecently, Luigi reported there are lots of free swap space when OOM happens. It's easily reproduced on zram-over-swap, where many instance of memory hogs are running and
laptop_modeis enabled.Luigi reported there was no problem when he disabled
laptop_mode. The problem when I investigate problem is following as.
try_to_free_pagesdisablemay_writepageiflaptop_modeis enabled.shrink_page_listadds lots of anon pages in swap cache byadd_to_swap, which makes pages Dirty and rotate them to head of inactive LRU without pageout. If it is repeated, inactive anon LRU is full of Dirty and SwapCache pages.In case of that,
isolate_lru_pagesfails because it try to isolate clean page due tomay_writepage== 0.
may_writepagecould be 1 only iftotal_scannedis higher thanwriteback_thresholdindo_try_to_free_pagesbut unfortunately, VM can't isolate anon pages from inactive anon lru list by above reason and we already reclaimed all file-backed pages. So it ends up OOM killing.This patch makes
may_writepagecould be set whenshrink_inactive_listencounters SwapCachePage from tail of inactive anon LRU. What it means that anon LRU list is short and memory pressure is severe so it would be better to swap out that pages by sacrificing the power rather than OOM killing.Reported-by Luigi Semenzato <[email protected]>Signed-off-by Minchan Kim <[email protected]>--- mm/vmscan.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)mm/vmscan.c +12
-2@@ -1102,7 +1102,7 @@ static unsigned long isolate_lru_pages(unsigned long nr_to_scan,prefetchw_prev_lru_page (page, src, flags);VM_BUG_ON (!PageLRU (page)); - +retry:switch (__isolate_lru_page (page, mode)) {case 0 : nr_pages =hpage_nr_pages (page);@@ -1112,6 +1112,17 @@ static unsigned long isolate_lru_pages(unsigned long nr_to_scan,break ;case -EBUSY : +/* +* If VM encounters PageSwapCache from inactive LRU, +* it means we havd to swap out those pages regardless +* of laptop_mode for preventing OOM kill. +*/ +if ((mode &ISOLATE_CLEAN ) &&PageSwapCache (page) && + !PageActive (page)) { + mode &= ~ISOLATE_CLEAN ; + sc->may_writepage =1 ; + goto retry; + }/* else it is being freed elsewhere */ list_move (&page->lru, src);continue ; --1.7.9.5
Quoted 81 lines · Luigi Semenzato
Oh well, I found the problem, it's
laptop_mode. We keep it on by default. When I turn it off, I can allocate as fast as I can, and no OOMs happen until swap is exhausted.I don't think this is a desirable behavior even for
laptop_mode, so if anybody wants to help me debug it (or wants my help in debugging it) do let me know.Thanks!
LuigiQuoted 69 lines · Luigi Semenzato
Minchan:
I tried your suggestion to move the call to
wake_all_kswapdfrom after "restart:" to after "rebalance:". The behavior is still similar, but slightly improved. Here's what I see.Allocating as fast as I can: 1.5 GB of the 3 GB of zram swap are used, then OOM kills happen, and the system ends up with 1 GB swap used, 2 unused.
Allocating 10 MB/s: some kills happen when only 1 to 1.5 GB are used, and continue happening while swap fills up. Eventually swap fills up completely. This is better than before (could not go past about 1 GB of swap used), but there are too many kills too early. I would like to see no OOM kills until swap is full or almost full.
Allocating 20 MB/s: almost as good as with 10 MB/s, but more kills happen earlier, and not all swap space is used (400 MB free at the end).
This is with 200 processes using 20 MB each, and 2:1 compression ratio.
So it looks like kswapd is still not aggressive enough in pushing pages out. What's the best way of changing that? Play around with the watermarks?
Incidentally, I also tried removing the
min_filelist_kbyteshacky patch, but, as usual, the system thrashes so badly that it's impossible to complete any experiment. I set it to a lower minimum amount of free file pages, 10 MB instead of the 50 MB which we use normally, and I could run with some thrashing, but I got the same results.Thanks!
LuigiQuoted 31 lines · Luigi Semenzato
I am beginning to understand why zram appears to work fine on our x86 systems but not on our ARM systems. The bottom line is that swapping doesn't work as I would expect when allocation is "too fast".
In one of my tests, opening 50 tabs simultaneously in a Chrome browser on devices with 2 GB of RAM and a zram-disk of 3 GB (uncompressed), I was observing that on the x86 device all of the zram swap space was used before OOM kills happened, but on the ARM device I would see OOM kills when only about 1 GB (out of 3) was swapped out.
I wrote a simple program to understand this behavior. The program (called "hog") allocates memory and fills it with a mix of incompressible data (from
/dev/urandom) and highly compressible data (1's, just to avoid zero pages) in a given ratio. The memory is never touched again.It turns out that if I don't limit the allocation speed, I see premature OOM kills also on the x86 device. If I limit the allocation to 10 MB/s, the premature OOM kills stop happening on the x86 device, but still happen on the ARM device. If I further limit the allocation speed to 5 Mb/s, the premature OOM kills disappear also from the ARM device.
I have noticed a few time constants in the MM whose value is not well explained, and I am wondering if the code is tuned for some ideal system that doesn't behave like ours (considering, for instance, that zram is much faster than swapping to a disk device, but it also uses more CPU). If this is plausible, I am wondering if anybody has suggestions for changes that I could try out to obtain a better behavior with a higher allocation speed.
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Thanks!
Luigi--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 3 lines
--
Kind regards,
Minchan Kim
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Hi Luigi,
Quoted 5 lines · Luigi Semenzato
No problem at all---as I mentioned, we stopped using
laptop_mode, so this is no longer an issue for us.I should be able to test the patch for you in the next 2-3 days. I will let you know if I run into problems.
Right now, I sent new version. I think it's better than this patch. Could you test new version instead of this?
Thanks!
Quoted 189 lines
Thanks!
LuigiQuoted 175 lines · Minchan Kim
Hi Luigi,
Sorry for really really late response.
Today I have a time to look at this problem and it seems to found the problem. By your help, I can reprocude this problem easily on my KVM machine and this patch solves the problem.Could you test below patch? Although this patch is based on recent mmotm, I guess you can apply it easily to 3.4.
From
f74fdf644bec3e7875d245154db953b47b6c9594Mon Sep 17 00:00:00 2001
From: Minchan Kim <[email protected]>
Date: Tue, 8 Jan 2013 16:23:31 +0900
Subject: [PATCH] mm: swap out anonymous page regardless oflaptop_modeRecently, Luigi reported there are lots of free swap space when OOM happens. It's easily reproduced on zram-over-swap, where many instance of memory hogs are running and
laptop_modeis enabled.Luigi reported there was no problem when he disabled
laptop_mode. The problem when I investigate problem is following as.
try_to_free_pagesdisablemay_writepageiflaptop_modeis enabled.shrink_page_listadds lots of anon pages in swap cache byadd_to_swap, which makes pages Dirty and rotate them to head of inactive LRU without pageout. If it is repeated, inactive anon LRU is full of Dirty and SwapCache pages.In case of that,
isolate_lru_pagesfails because it try to isolate clean page due tomay_writepage== 0.
may_writepagecould be 1 only iftotal_scannedis higher thanwriteback_thresholdindo_try_to_free_pagesbut unfortunately, VM can't isolate anon pages from inactive anon lru list by above reason and we already reclaimed all file-backed pages. So it ends up OOM killing.This patch makes
may_writepagecould be set whenshrink_inactive_listencounters SwapCachePage from tail of inactive anon LRU. What it means that anon LRU list is short and memory pressure is severe so it would be better to swap out that pages by sacrificing the power rather than OOM killing.Reported-by Luigi Semenzato <[email protected]>Signed-off-by Minchan Kim <[email protected]>--- mm/vmscan.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)mm/vmscan.c +12
-2@@ -1102,7 +1102,7 @@ static unsigned long isolate_lru_pages(unsigned long nr_to_scan,prefetchw_prev_lru_page (page, src, flags);VM_BUG_ON (!PageLRU (page)); - +retry:switch (__isolate_lru_page (page, mode)) {case 0 : nr_pages =hpage_nr_pages (page);@@ -1112,6 +1112,17 @@ static unsigned long isolate_lru_pages(unsigned long nr_to_scan,break ;case -EBUSY : +/* +* If VM encounters PageSwapCache from inactive LRU, +* it means we havd to swap out those pages regardless +* of laptop_mode for preventing OOM kill. +*/ +if ((mode &ISOLATE_CLEAN ) &&PageSwapCache (page) && + !PageActive (page)) { + mode &= ~ISOLATE_CLEAN ; + sc->may_writepage =1 ; + goto retry; + }/* else it is being freed elsewhere */ list_move (&page->lru, src);continue ; --1.7.9.5
Quoted 81 lines · Luigi Semenzato
Oh well, I found the problem, it's
laptop_mode. We keep it on by default. When I turn it off, I can allocate as fast as I can, and no OOMs happen until swap is exhausted.I don't think this is a desirable behavior even for
laptop_mode, so if anybody wants to help me debug it (or wants my help in debugging it) do let me know.Thanks!
LuigiQuoted 69 lines · Luigi Semenzato
Minchan:
I tried your suggestion to move the call to
wake_all_kswapdfrom after "restart:" to after "rebalance:". The behavior is still similar, but slightly improved. Here's what I see.Allocating as fast as I can: 1.5 GB of the 3 GB of zram swap are used, then OOM kills happen, and the system ends up with 1 GB swap used, 2 unused.
Allocating 10 MB/s: some kills happen when only 1 to 1.5 GB are used, and continue happening while swap fills up. Eventually swap fills up completely. This is better than before (could not go past about 1 GB of swap used), but there are too many kills too early. I would like to see no OOM kills until swap is full or almost full.
Allocating 20 MB/s: almost as good as with 10 MB/s, but more kills happen earlier, and not all swap space is used (400 MB free at the end).
This is with 200 processes using 20 MB each, and 2:1 compression ratio.
So it looks like kswapd is still not aggressive enough in pushing pages out. What's the best way of changing that? Play around with the watermarks?
Incidentally, I also tried removing the
min_filelist_kbyteshacky patch, but, as usual, the system thrashes so badly that it's impossible to complete any experiment. I set it to a lower minimum amount of free file pages, 10 MB instead of the 50 MB which we use normally, and I could run with some thrashing, but I got the same results.Thanks!
LuigiQuoted 31 lines · Luigi Semenzato
I am beginning to understand why zram appears to work fine on our x86 systems but not on our ARM systems. The bottom line is that swapping doesn't work as I would expect when allocation is "too fast".
In one of my tests, opening 50 tabs simultaneously in a Chrome browser on devices with 2 GB of RAM and a zram-disk of 3 GB (uncompressed), I was observing that on the x86 device all of the zram swap space was used before OOM kills happened, but on the ARM device I would see OOM kills when only about 1 GB (out of 3) was swapped out.
I wrote a simple program to understand this behavior. The program (called "hog") allocates memory and fills it with a mix of incompressible data (from
/dev/urandom) and highly compressible data (1's, just to avoid zero pages) in a given ratio. The memory is never touched again.It turns out that if I don't limit the allocation speed, I see premature OOM kills also on the x86 device. If I limit the allocation to 10 MB/s, the premature OOM kills stop happening on the x86 device, but still happen on the ARM device. If I further limit the allocation speed to 5 Mb/s, the premature OOM kills disappear also from the ARM device.
I have noticed a few time constants in the MM whose value is not well explained, and I am wondering if the code is tuned for some ideal system that doesn't behave like ours (considering, for instance, that zram is much faster than swapping to a disk device, but it also uses more CPU). If this is plausible, I am wondering if anybody has suggestions for changes that I could try out to obtain a better behavior with a higher allocation speed.
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Thanks!
Luigi--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>Quoted 3 lines
--
Kind regards,
Minchan Kim--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
I'll make sure I use the new version!
Quoted 239 lines · Minchan Kim
Hi Luigi,
Quoted 5 lines · Luigi Semenzato
No problem at all---as I mentioned, we stopped using
laptop_mode, so this is no longer an issue for us.I should be able to test the patch for you in the next 2-3 days. I will let you know if I run into problems.
Right now, I sent new version. I think it's better than this patch. Could you test new version instead of this?
Thanks!
Quoted 9 lines
Thanks!
LuigiQuoted 4 lines · Minchan Kim
Hi Luigi,
Sorry for really really late response.
Today I have a time to look at this problem and it seems to found theproblem.
Quoted 1 line
Quoted 1 line
By your help, I can reprocude this problem easily on my KVM machine
and this
Quoted 3 lines
Quoted 3 lines
patch solves the problem.
Could you test below patch? Although this patch is based on recent
mmotm,
Quoted 41 lines
Quoted 41 lines
I guess you can apply it easily to 3.4.
From
f74fdf644bec3e7875d245154db953b47b6c9594Mon Sep 17 00:00:00 2001
From: Minchan Kim <[email protected]>
Date: Tue, 8 Jan 2013 16:23:31 +0900
Subject: [PATCH] mm: swap out anonymous page regardless oflaptop_modeRecently, Luigi reported there are lots of free swap space when OOM happens. It's easily reproduced on zram-over-swap, where many instance of memory hogs are running and
laptop_modeis enabled.Luigi reported there was no problem when he disabled
laptop_mode. The problem when I investigate problem is following as.
try_to_free_pagesdisablemay_writepageiflaptop_modeis enabled.shrink_page_listadds lots of anon pages in swap cache byadd_to_swap, which makes pages Dirty and rotate them to head of inactive LRU without pageout. If it is repeated, inactive anon LRU is full of Dirty and SwapCache pages.In case of that,
isolate_lru_pagesfails because it try to isolate clean page due tomay_writepage== 0.
may_writepagecould be 1 only iftotal_scannedis higher thanwriteback_thresholdindo_try_to_free_pagesbut unfortunately, VM can't isolate anon pages from inactive anon lru list by above reason and we already reclaimed all file-backed pages. So it ends up OOM killing.This patch makes
may_writepagecould be set whenshrink_inactive_listencounters SwapCachePage from tail of inactive anon LRU. What it means that anon LRU list is short and memory pressure is severe so it would be better to swap out that pages by sacrificing the power rather than OOM killing.Reported-by Luigi Semenzato <[email protected]>Signed-off-by Minchan Kim <[email protected]>--- mm/vmscan.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)Quoted 5 lines
mm/vmscan.c static unsigned long isolate_lru_pages(unsigned
Quoted 5 lines
mm/vmscan.c +0
-0@@ -1102,7 +1102,7 @@ static unsigned long isolate_lru_pages(unsignedlong
nr_to_scan,Quoted 9 lines
mm/vmscan.c static unsigned long isolate_lru_pages(unsigned
Quoted 9 lines
mm/vmscan.c +1
-1prefetchw_prev_lru_page (page, src, flags);VM_BUG_ON (!PageLRU (page)); - +retry:switch (__isolate_lru_page (page, mode)) {case 0 : nr_pages =hpage_nr_pages (page);@@ -1112,6 +1112,17 @@ static unsigned long isolate_lru_pages(unsignedlong
nr_to_scan,Quoted 5 lines
mm/vmscan.c static unsigned long isolate_lru_pages(unsigned · line 1116
Quoted 5 lines
mm/vmscan.c +2
-0break ;case -EBUSY : +/* +* If VM encounters PageSwapCache from inactive LRU,
Quoted 1 line
mm/vmscan.c static unsigned long isolate_lru_pages(unsigned · line 1117
Quoted 1 line
mm/vmscan.c +1
-0+* it means we havd to swap out those pages regardless
Quoted 3 lines
mm/vmscan.c static unsigned long isolate_lru_pages(unsigned · line 1120
Quoted 3 lines
mm/vmscan.c +3
-0+* of laptop_mode for preventing OOM kill. +*/ + if ((mode & ISOLATE_CLEAN) &&PageSwapCache(page) &&
Quoted 25 lines
Quoted 25 lines
mm/vmscan.c +5
-1+ !PageActive(page)) { + mode &= ~ISOLATE_CLEAN ; + sc->may_writepage =1 ; + goto retry; + }/* else it is being freed elsewhere */ list_move (&page->lru, src);continue ; --1.7.9.5
Quoted 12 lines · Luigi Semenzato
Oh well, I found the problem, it's
laptop_mode. We keep it on by default. When I turn it off, I can allocate as fast as I can, and no OOMs happen until swap is exhausted.I don't think this is a desirable behavior even for
laptop_mode, so if anybody wants to help me debug it (or wants my help in debugging it) do let me know.Thanks!
LuigiOn Thu, Nov 29, 2012 at 10:46 AM, Luigi Semenzato <
Quoted 3 lines · [email protected]> wrote:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Minchan:
I tried your suggestion to move the call to
wake_all_kswapdfromafter
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"restart:" to after "rebalance:". The behavior is still similar,
but
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
slightly improved. Here's what I see.
Allocating as fast as I can: 1.5 GB of the 3 GB of zram swap are
used,
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
then OOM kills happen, and the system ends up with 1 GB swap used, 2 unused.
Allocating 10 MB/s: some kills happen when only 1 to 1.5 GB are
used,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
and continue happening while swap fills up. Eventually swap fills
up
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
completely. This is better than before (could not go past about 1
GB
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
of swap used), but there are too many kills too early. I would like to see no OOM kills until swap is full or almost full.
Allocating 20 MB/s: almost as good as with 10 MB/s, but more kills happen earlier, and not all swap space is used (400 MB free at the end).
This is with 200 processes using 20 MB each, and 2:1 compression
ratio.
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
So it looks like kswapd is still not aggressive enough in pushing pages out. What's the best way of changing that? Play around with the watermarks?
Incidentally, I also tried removing the
min_filelist_kbyteshacky patch, but, as usual, the system thrashes so badly that it's impossible to complete any experiment. I set it to a lower minimum amount of free file pages, 10 MB instead of the 50 MB which we use normally, and I could run with some thrashing, but I got the same results.Thanks!
LuigiOn Wed, Nov 28, 2012 at 4:31 PM, Luigi Semenzato <
Quoted 1 line · [email protected]> wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
I am beginning to understand why zram appears to work fine on our
x86
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
systems but not on our ARM systems. The bottom line is that
swapping
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
doesn't work as I would expect when allocation is "too fast".
In one of my tests, opening 50 tabs simultaneously in a Chrome
browser
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
on devices with 2 GB of RAM and a zram-disk of 3 GB
(uncompressed), I
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
was observing that on the x86 device all of the zram swap space was used before OOM kills happened, but on the ARM device I would see
OOM
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
kills when only about 1 GB (out of 3) was swapped out.
I wrote a simple program to understand this behavior. The program (called "hog") allocates memory and fills it with a mix of incompressible data (from
/dev/urandom) and highly compressibledata
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(1's, just to avoid zero pages) in a given ratio. The memory is
never
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
touched again.
It turns out that if I don't limit the allocation speed, I see premature OOM kills also on the x86 device. If I limit the
allocation
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to 10 MB/s, the premature OOM kills stop happening on the x86
device,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
but still happen on the ARM device. If I further limit the
allocation
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
speed to 5 Mb/s, the premature OOM kills disappear also from the
ARM
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
device.
I have noticed a few time constants in the MM whose value is not
well
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
explained, and I am wondering if the code is tuned for some ideal system that doesn't behave like ours (considering, for instance,
that
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
zram is much faster than swapping to a disk device, but it also
uses
Quoted 17 lines
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
more CPU). If this is plausible, I am wondering if anybody has suggestions for changes that I could try out to obtain a better behavior with a higher allocation speed.
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Thanks!
Luigi--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>Quoted 3 lines
--
Kind regards,
Minchan KimQuoted 5 lines
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>--
Kind regards,
Minchan Kim