zram/zsmalloc issues in very low memory conditions
Subsystems (4)
I am trying to use zram in very low memory conditions and I am having some issues. zram is in the reclaim path. So if the system is very low on memory the system is trying to reclaim pages by swapping out (in this case to zram). However, since we are very low on memory zram fails to get a page from zsmalloc and thus zram fails to store the page. We get into a cycle where the system is low on memory so it tries to swap out to get more memory but swap out fails because there is not enough memory in the system! The major problem I am seeing is that there does not seem to be a way for zram to tell the upper layers to stop swapping out because the swap device is essentially "full" (since there is no more memory available for zram pages). Has anyone thought about this issue already and have ideas how to solve this or am I missing something and I should not be seeing this issue?
I am also seeing a couple other issues that I was wondering whether folks have already thought about:
1) The size of a swap device is statically computed when the swap device is turned on (nr_swap_pages). The size of zram swap device is dynamic since we are compressing the pages and thus the swap subsystem thinks that the zram swap device is full when it is not really full. Any plans/thoughts about the possibility of being able to update the size and/or the # of available pages in a swap device on the fly?
2) zsmalloc fails when the page allocated is at physical address 0 (pfn = 0) since the handle returned from zsmalloc is encoded as (<PFN>, <obj_idx>) and thus the resulting handle will be 0 (since obj_idx starts at 0). zs_malloc returns the handle but does not distinguish between a valid handle of 0 and a failure to allocate. A possible solution to this would be to start the obj_idx at 1. Is this feasible?
Thanks,
Olav Haugan
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
Quoted 5 lines · Olav Haugan
I am trying to use zram in very low memory conditions and I am having some issues. zram is in the reclaim path. So if the system is very low on memory the system is trying to reclaim pages by swapping out (in this case to zram). However, since we are very low on memory zram fails to get a page from zsmalloc and thus zram fails to store the page. We get
Quoted 8 lines
into a cycle where the system is low on memory so it tries to swap out to get more memory but swap out fails because there is not enough memory in the system! The major problem I am seeing is that there does not seem to be a way for zram to tell the upper layers to stop swapping out because the swap device is essentially "full" (since there is no more memory available for zram pages). Has anyone thought about this issue already and have ideas how to solve this or am I missing something and I should not be seeing this issue?
What do you want the system to do at this point? OOM kill? Also, if you are that low on memory, how are you preventing thrashing on the code pages?
I am asking because we also use zram but haven't run into this problem---however we had to deal with other problems that motivate these questions.
Quoted 11 lines
I am also seeing a couple other issues that I was wondering whether folks have already thought about:
1) The size of a swap device is statically computed when the swap device is turned on (
nr_swap_pages). The size of zram swap device is dynamic since we are compressing the pages and thus the swap subsystem thinks that the zram swap device is full when it is not really full. Any plans/thoughts about the possibility of being able to update the size and/or the # of available pages in a swap device on the fly?
That is a known limitation of zram. If can predict your compression ratio and your working set size, it's not a big problem: allocate a swap device which, based on the expected compression ratio, will use up RAM until what's left is just enough for the working set.
Quoted 8 lines
2) zsmalloc fails when the page allocated is at physical address 0 (pfn = 0) since the handle returned from zsmalloc is encoded as (<PFN>, <
obj_idx>) and thus the resulting handle will be 0 (sinceobj_idxstarts at 0).zs_mallocreturns the handle but does not distinguish between a valid handle of 0 and a failure to allocate. A possible solution to this would be to start theobj_idxat 1. Is this feasible?
Sorry, no idea on this. Probably Minchan can reply.
Quoted 15 lines
Thanks,
Olav Haugan
--
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
(sorry about the HTML in the previous message)
Quoted 5 lines · Olav Haugan
I am trying to use zram in very low memory conditions and I am having some issues. zram is in the reclaim path. So if the system is very low on memory the system is trying to reclaim pages by swapping out (in this case to zram). However, since we are very low on memory zram fails to get a page from zsmalloc and thus zram fails to store the page. We get
Quoted 8 lines
into a cycle where the system is low on memory so it tries to swap out to get more memory but swap out fails because there is not enough memory in the system! The major problem I am seeing is that there does not seem to be a way for zram to tell the upper layers to stop swapping out because the swap device is essentially "full" (since there is no more memory available for zram pages). Has anyone thought about this issue already and have ideas how to solve this or am I missing something and I should not be seeing this issue?
What do you want the system to do at this point? OOM kill? Also, if you are that low on memory, how are you preventing thrashing on the code pages?
I am asking because we also use zram but haven't run into this problem---however we had to deal with other problems that motivate these questions.
Quoted 10 lines
I am also seeing a couple other issues that I was wondering whether folks have already thought about:
1) The size of a swap device is statically computed when the swap device is turned on (
nr_swap_pages). The size of zram swap device is dynamic since we are compressing the pages and thus the swap subsystem thinks that the zram swap device is full when it is not really full. Any plans/thoughts about the possibility of being able to update the size and/or the # of available pages in a swap device on the fly?
That is a known limitation of zram. If you can predict your compression ratio and your working set size, it's not a big problem:
allocate a swap device which, based on the expected compression ratio, will use up RAM until what's left is just enough for the working set.
Quoted 15 lines
2) zsmalloc fails when the page allocated is at physical address 0 (pfn = 0) since the handle returned from zsmalloc is encoded as (<PFN>, <
obj_idx>) and thus the resulting handle will be 0 (sinceobj_idxstarts at 0).zs_mallocreturns the handle but does not distinguish between a valid handle of 0 and a failure to allocate. A possible solution to this would be to start theobj_idxat 1. Is this feasible?Thanks,
Olav Haugan
--
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
Quoted 5 lines
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 5 lines · Olav Haugan
I am trying to use zram in very low memory conditions and I am having some issues. zram is in the reclaim path. So if the system is very low on memory the system is trying to reclaim pages by swapping out (in this case to zram). However, since we are very low on memory zram fails to get a page from zsmalloc and thus zram fails to store the page. We get
Quoted 9 lines
into a cycle where the system is low on memory so it tries to swap out to get more memory but swap out fails because there is not enough memory in the system! The major problem I am seeing is that there does not seem to be a way for zram to tell the upper layers to stop swapping out because the swap device is essentially "full" (since there is no more memory available for zram pages). Has anyone thought about this issue already and have ideas how to solve this or am I missing something and I should not be seeing this issue?
The same question as Luigi "What do you want the system to do at this point?"
If swap fails then OOM killer will be triggered, I don't think this will be a issue.
By the way, could you take a try with zswap? Which can write pages to real swap device if compressed pool is full.
Quoted 11 lines
I am also seeing a couple other issues that I was wondering whether folks have already thought about:
1) The size of a swap device is statically computed when the swap device is turned on (
nr_swap_pages). The size of zram swap device is dynamic since we are compressing the pages and thus the swap subsystem thinks that the zram swap device is full when it is not really full. Any plans/thoughts about the possibility of being able to update the size and/or the # of available pages in a swap device on the fly?2) zsmalloc fails when the page allocated is at physical address 0 (pfn
AFAIK, this will never happen.
Quoted 10 lines
= 0) since the handle returned from zsmalloc is encoded as (<PFN>, <
obj_idx>) and thus the resulting handle will be 0 (sinceobj_idxstarts at 0).zs_mallocreturns the handle but does not distinguish between a valid handle of 0 and a failure to allocate. A possible solution to this would be to start theobj_idxat 1. Is this feasible?Thanks,
Olav Haugan
Signature · 8 lines
Regards, -Bob -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 5 lines · Olav Haugan
I am trying to use zram in very low memory conditions and I am having some issues. zram is in the reclaim path. So if the system is very low on memory the system is trying to reclaim pages by swapping out (in this case to zram). However, since we are very low on memory zram fails to get a page from zsmalloc and thus zram fails to store the page. We get
Quoted 8 lines
into a cycle where the system is low on memory so it tries to swap out to get more memory but swap out fails because there is not enough memory in the system! The major problem I am seeing is that there does not seem to be a way for zram to tell the upper layers to stop swapping out because the swap device is essentially "full" (since there is no more memory available for zram pages). Has anyone thought about this issue already and have ideas how to solve this or am I missing something and I should not be seeing this issue?
I agree with Luigi and Bob.
zram's size is based on how many free memory you expect to use for zram. In my test, the compression ratio is about 1:4, of course the working sets may be
different with yours.
Further more, may be you can modify vm_swap_full() to let kernel free swap_entry aggressively.
Quoted 29 lines
I am also seeing a couple other issues that I was wondering whether folks have already thought about:
1) The size of a swap device is statically computed when the swap device is turned on (
nr_swap_pages). The size of zram swap device is dynamic since we are compressing the pages and thus the swap subsystem thinks that the zram swap device is full when it is not really full. Any plans/thoughts about the possibility of being able to update the size and/or the # of available pages in a swap device on the fly?2) zsmalloc fails when the page allocated is at physical address 0 (pfn = 0) since the handle returned from zsmalloc is encoded as (<PFN>, <
obj_idx>) and thus the resulting handle will be 0 (sinceobj_idxstarts at 0).zs_mallocreturns the handle but does not distinguish between a valid handle of 0 and a failure to allocate. A possible solution to this would be to start theobj_idxat 1. Is this feasible?Thanks,
Olav Haugan
--
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Hi Bob, Luigi,
Quoted 17 lines · Bob Liu
Quoted 5 lines · Olav Haugan
I am trying to use zram in very low memory conditions and I am having some issues. zram is in the reclaim path. So if the system is very low on memory the system is trying to reclaim pages by swapping out (in this case to zram). However, since we are very low on memory zram fails to get a page from zsmalloc and thus zram fails to store the page. We get
Quoted 9 lines
into a cycle where the system is low on memory so it tries to swap out to get more memory but swap out fails because there is not enough memory in the system! The major problem I am seeing is that there does not seem to be a way for zram to tell the upper layers to stop swapping out because the swap device is essentially "full" (since there is no more memory available for zram pages). Has anyone thought about this issue already and have ideas how to solve this or am I missing something and I should not be seeing this issue?
Quoted 5 lines
The same question as Luigi "What do you want the system to do at this point?"
If swap fails then OOM killer will be triggered, I don't think this will be a issue.
I definitely don't want OOM killer to run since OOM killer can kill critical processes (this is on Android so we have Android LMK to handle the killing in a more "safe" way). However, what I am seeing is that when I run low on memory zram fails to swap out and returns error but the swap subsystem just continues to try to swap out even when this error occurs (it tries over and over again very rapidly causing the kernel to be filled with error messages [at least two error messages per failure btw]).
What I expected to happen is for the swap subsystem to stop trying to swap out until memory is available to swap out. I guess this could be handled several ways. Either 1) the swap subsystem, upon encountering an error to swap out, backs off from trying to swap out for some time or 2) zram informs the swap subsystem that the swap device is full.
Could this be handled by congestion control? However, I found the following comment in the code in vmscan.c:
* If the page is swapcache, write it back even if that would
* block, for some throttling. This happens by accident, because
* swap_backing_dev_info is bust: it doesn't reflect the
* congestion state of the swapdevs. Easy to fix, if needed.
However, how would one update the congested state of zram when it becomes un-congested?
Quoted 2 lines
By the way, could you take a try with zswap? Which can write pages to real swap device if compressed pool is full.
zswap might not be feasible in all cases if you only have flash as backing storage.
Quoted 6 lines
Quoted 4 lines
I am also seeing a couple other issues that I was wondering whether folks have already thought about:
2) zsmalloc fails when the page allocated is at physical address 0 (pfn
AFAIK, this will never happen.
I can easily get this to happen since I have memory starting at physical address 0.
Thanks,
Olav Haugan
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
Quoted 35 lines · Olav Haugan
Hi Bob, Luigi,
Quoted 17 lines · Bob Liu
Quoted 5 lines · Olav Haugan
I am trying to use zram in very low memory conditions and I am having some issues. zram is in the reclaim path. So if the system is very low on memory the system is trying to reclaim pages by swapping out (in this case to zram). However, since we are very low on memory zram fails to get a page from zsmalloc and thus zram fails to store the page. We get
Quoted 9 lines
into a cycle where the system is low on memory so it tries to swap out to get more memory but swap out fails because there is not enough memory in the system! The major problem I am seeing is that there does not seem to be a way for zram to tell the upper layers to stop swapping out because the swap device is essentially "full" (since there is no more memory available for zram pages). Has anyone thought about this issue already and have ideas how to solve this or am I missing something and I should not be seeing this issue?
Quoted 5 lines
The same question as Luigi "What do you want the system to do at this point?"
If swap fails then OOM killer will be triggered, I don't think this will be a issue.
I definitely don't want OOM killer to run since OOM killer can kill critical processes (this is on Android so we have Android LMK to handle the killing in a more "safe" way). However, what I am seeing is that when I run low on memory zram fails to swap out and returns error but the swap subsystem just continues to try to swap out even when this error occurs (it tries over and over again very rapidly causing the kernel to be filled with error messages [at least two error messages per failure btw]).
Quoted 5 lines
What I expected to happen is for the swap subsystem to stop trying to swap out until memory is available to swap out. I guess this could be handled several ways. Either 1) the swap subsystem, upon encountering an error to swap out, backs off from trying to swap out for some time or 2) zram informs the swap subsystem that the swap device is full.
There is a lot I don't know, both about the specifics of your case and the MM subsystem in general, so I'll make some guesses. Don't trust anything I say here (as if you would anyway :-).
As the system gets low on memory, the MM tries to reclaim it in various ways. The biggest (I think) sources of reclaim come from swapping out anonymous pages (process data), and discarding file-backed pages (code, for instance). The "swappiness" parameter decides how much attention to give to each of these types of memory.
It's possible that you get in a situation in which attempts to swap out anonymous pages with zram fail because you're out of memory at that point, but then some memory is reclaimed by discarding file pages, and that's why you don't see OOM kills or kernel panic. Either way you should be really close to that moment, unless something funny is going on.
You could try to snapshot the memory usage when those message are produced. You can, for instance, use SysRQ-M to dump a bunch of data in the syslog. You may also want to monitor the zram device utlization from the sysfs.
It's possible that by the time you see those messages you're already thrashing badly and that things slowed down so much that you aren't quite getting to the OOM killer. You could try to reduce the size of your swap device, and/or change the swappiness.
By the way, I am under the impression that Android uses the OOM killer as part of their memory management strategy.
Quoted 8 lines
Could this be handled by congestion control? However, I found the following comment in the code in
vmscan.c:* If the page is swapcache, write it back even if that would
* block, for some throttling. This happens by accident, because
*swap_backing_dev_infois bust: it doesn't reflect the
* congestion state of the swapdevs. Easy to fix, if needed.
Quoted 8 lines
However, how would one update the congested state of zram when it becomes un-congested?
Quoted 2 lines
By the way, could you take a try with zswap? Which can write pages to real swap device if compressed pool is full.
zswap might not be feasible in all cases if you only have flash as backing storage.
Zswap can be configured to run without a backing storage.
Quoted 18 lines
Quoted 7 lines
Quoted 5 lines
I am also seeing a couple other issues that I was wondering whether folks have already thought about:
2) zsmalloc fails when the page allocated is at physical address 0 (pfn
AFAIK, this will never happen.
I can easily get this to happen since I have memory starting at physical address 0.
Thanks,
Olav Haugan
--
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
Quoted 26 lines · Olav Haugan
Hi Bob, Luigi,
Quoted 17 lines · Bob Liu
Quoted 5 lines · Olav Haugan
I am trying to use zram in very low memory conditions and I am having some issues. zram is in the reclaim path. So if the system is very low on memory the system is trying to reclaim pages by swapping out (in this case to zram). However, since we are very low on memory zram fails to get a page from zsmalloc and thus zram fails to store the page. We get
Quoted 9 lines
into a cycle where the system is low on memory so it tries to swap out to get more memory but swap out fails because there is not enough memory in the system! The major problem I am seeing is that there does not seem to be a way for zram to tell the upper layers to stop swapping out because the swap device is essentially "full" (since there is no more memory available for zram pages). Has anyone thought about this issue already and have ideas how to solve this or am I missing something and I should not be seeing this issue?
Quoted 5 lines
The same question as Luigi "What do you want the system to do at this point?"
If swap fails then OOM killer will be triggered, I don't think this will be a issue.
Quoted 9 lines
I definitely don't want OOM killer to run since OOM killer can kill critical processes (this is on Android so we have Android LMK to handle the killing in a more "safe" way). However, what I am seeing is that when I run low on memory zram fails to swap out and returns error but the swap subsystem just continues to try to swap out even when this error occurs (it tries over and over again very rapidly causing the kernel to be filled with error messages [at least two error messages per failure btw]).
A simple way to disable the error messages is delete the printk line in zram source code.
Quoted 2 lines
What I expected to happen is for the swap subsystem to stop trying to swap out until memory is available to swap out. I guess this could be
In my opinion, the system already entered heavy memory pressure state when zram fails to allocate a page.
In this case, the OOM killer or Low Memory Killer should be waked up and kill some processes in order to free some memory pages.
After this happen, the system free memory may above water mark and no swap will happen. Even swap happens again, it's unlikely that zram will fail to alloc page. If it fails again, OOM killer or LMK should be triggered once more.
Quoted 12 lines
handled several ways. Either 1) the swap subsystem, upon encountering an error to swap out, backs off from trying to swap out for some time or 2) zram informs the swap subsystem that the swap device is full.
Could this be handled by congestion control? However, I found the following comment in the code in
vmscan.c:* If the page is swapcache, write it back even if that would
* block, for some throttling. This happens by accident, because
*swap_backing_dev_infois bust: it doesn't reflect the
* congestion state of the swapdevs. Easy to fix, if needed.
Quoted 9 lines
However, how would one update the congested state of zram when it becomes un-congested?
Quoted 2 lines
By the way, could you take a try with zswap? Which can write pages to real swap device if compressed pool is full.
zswap might not be feasible in all cases if you only have flash as backing storage.
Yes, that's still a problem of zswap. Perhaps you can create a swap file on the backing storage.
I'll try to add a fake swap device for zswap, so that users can have one more choice besides zram.
Quoted 10 lines
Quoted 6 lines
Quoted 4 lines
I am also seeing a couple other issues that I was wondering whether folks have already thought about:
2) zsmalloc fails when the page allocated is at physical address 0 (pfn
AFAIK, this will never happen.
I can easily get this to happen since I have memory starting at physical address 0.
Could you confirm this? AFAIR, physical memory start from 0 usually reserved for some special usage.
I used 'cat /proc/zoneinfo' on x86 and arm, neither of the 'start_pfn' was 0.
Signature · 8 lines
Regards, -Bob -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Hello,
I had no enough time to think over your great questions since I should enjoy in Edinburgh so if I miss something, Sorry!
Quoted 9 lines · Olav Haugan
I am trying to use zram in very low memory conditions and I am having some issues. zram is in the reclaim path. So if the system is very low on memory the system is trying to reclaim pages by swapping out (in this case to zram). However, since we are very low on memory zram fails to get a page from zsmalloc and thus zram fails to store the page. We get into a cycle where the system is low on memory so it tries to swap out to get more memory but swap out fails because there is not enough memory in the system! The major problem I am seeing is that there does not seem to be a way for zram to tell the upper layers to stop swapping out
True. The zram is block device so at a moment, I don't want to make zram swap-specific if it's possible.
Quoted 4 lines
because the swap device is essentially "full" (since there is no more memory available for zram pages). Has anyone thought about this issue already and have ideas how to solve this or am I missing something and I should not be seeing this issue?
It's true. We might need feedback loop and it shoudn't be specific for zram-swap. One think I can imagine is that we could move failed victim pages into LRU active list when the swapout failed so VM will have more weight for file pages than anon ones. For detail, you could see AOP_WRITEPAGE_ACTIVATE and get_scan_count for detail.
The problem is it's on fs layer while zram is on block layer so what I can think at a moment is follwing as
mm/vmscan.c +2 -0
@@ -502,6 +502,8 @@ static pageout_t pageout(struct page *page, struct address_space *mapping,if (!PageWriteback (page)) {/* synchronous write or broken a_ops? */ ClearPageReclaim (page); +if (PageError (page)) +return PAGE_ACTIVATE ; }trace_mm_vmscan_writepage (page,trace_reclaim_flags (page));inc_zone_page_state (page,NR_VMSCAN_WRITE );
It doesn't prevent swapout at all but it should throttle pick up anonymous pages for reclaiming so file-backed pages will be preferred by VM so sometime, zsmalloc succeed to allocate a free page and swapout will resume again.
Quoted 10 lines
I am also seeing a couple other issues that I was wondering whether folks have already thought about:
1) The size of a swap device is statically computed when the swap device is turned on (
nr_swap_pages). The size of zram swap device is dynamic since we are compressing the pages and thus the swap subsystem thinks that the zram swap device is full when it is not really full. Any plans/thoughts about the possibility of being able to update the size and/or the # of available pages in a swap device on the fly?
It's really good question. We could make zram's size bigger to prevent such problem when you set zram's disksize from the beginning but in this case, zram's meta(ie, struct table) size will be increased a bit. Is such memory overhead is critical for you?
Quoted 7 lines
2) zsmalloc fails when the page allocated is at physical address 0 (pfn = 0) since the handle returned from zsmalloc is encoded as (<PFN>, <
obj_idx>) and thus the resulting handle will be 0 (sinceobj_idxstarts at 0).zs_mallocreturns the handle but does not distinguish between a valid handle of 0 and a failure to allocate. A possible solution to this would be to start theobj_idxat 1. Is this feasible?
I think it's doable.
Quoted 8 lines
Thanks,
Olav Haugan
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
Signature · 2 lines
Kind regards, Minchan Kim
Hi Luigi,
Quoted 7 lines · Luigi Semenzato
Quoted 6 lines · Olav Haugan
Hi Bob, Luigi,
Quoted 2 lines · Bob Liu
On 10/24/2013 05:51 AM, Olav Haugan wrote:
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
By the way, could you take a try with zswap? Which can write pages to real swap device if compressed pool is full.
zswap might not be feasible in all cases if you only have flash as backing storage.
Zswap can be configured to run without a backing storage.
I was under the impression that zswap requires a backing storage. Can you elaborate on how to configure zswap to not need a backing storage?
Olav Haugan
Signature · 8 lines
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Hi Olav,
I haven't personally done it. Seth outlines the configuration in this thread:
http://thread.gmane.org/gmane.linux.kernel.mm/105378/focus=105543
Stephen, can you add more detail from your experience?
Thanks!
Luigi
Quoted 30 lines · Olav Haugan
Hi Luigi,
Quoted 1 line · Luigi Semenzato
On Thu, Oct 24, 2013 at 5:35 PM, Olav Haugan <[email protected]>
Quoted 6 lines · wrote:
Quoted 6 lines
Hi Bob, Luigi,
Quoted 2 lines · Bob Liu
On 10/24/2013 05:51 AM, Olav Haugan wrote:
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
By the way, could you take a try with zswap? Which can write pages to real swap device if compressed pool is full.
zswap might not be feasible in all cases if you only have flash as backing storage.
Zswap can be configured to run without a backing storage.
I was under the impression that zswap requires a backing storage. Can you elaborate on how to configure zswap to not need a backing storage?
Olav Haugan
--
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
[apologies for the previous HTML email]
Hi Olav,
I haven't personally done it. Seth outlines the configuration in this thread:
http://thread.gmane.org/gmane.linux.kernel.mm/105378/focus=105543
Stephen, can you add more detail from your experience?
Thanks!
Quoted 28 lines · Olav Haugan
Hi Luigi,
Quoted 7 lines · Luigi Semenzato
Quoted 6 lines · Olav Haugan
Hi Bob, Luigi,
Quoted 2 lines · Bob Liu
On 10/24/2013 05:51 AM, Olav Haugan wrote:
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
By the way, could you take a try with zswap? Which can write pages to real swap device if compressed pool is full.
zswap might not be feasible in all cases if you only have flash as backing storage.
Zswap can be configured to run without a backing storage.
I was under the impression that zswap requires a backing storage. Can you elaborate on how to configure zswap to not need a backing storage?
Olav Haugan
--
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
Hi Olav,
Quoted 13 lines · Olav Haugan
Hi Luigi,
Quoted 10 lines · Luigi Semenzato
Quoted 9 lines · Olav Haugan
Hi Bob, Luigi,
Quoted 5 lines · Bob Liu
On 10/24/2013 05:51 AM, Olav Haugan wrote:
By the way, could you take a try with zswap? Which can write pages to real swap device if compressed pool is full.
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
zswap might not be feasible in all cases if you only have flash as backing storage.
Zswap can be configured to run without a backing storage.
I was under the impression that zswap requires a backing storage. Can you elaborate on how to configure zswap to not need a backing storage?
AFAIK, currently zswap can't be used without a backing storage. Perhaps you can take a try by creating a swap device backed by a file on storage.
Signature · 8 lines
Regards, -Bob -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 48 lines · Minchan Kim
Hello,
I had no enough time to think over your great questions since I should enjoy in Edinburgh so if I miss something, Sorry!
Quoted 9 lines · Olav Haugan
I am trying to use zram in very low memory conditions and I am having some issues. zram is in the reclaim path. So if the system is very low on memory the system is trying to reclaim pages by swapping out (in this case to zram). However, since we are very low on memory zram fails to get a page from zsmalloc and thus zram fails to store the page. We get into a cycle where the system is low on memory so it tries to swap out to get more memory but swap out fails because there is not enough memory in the system! The major problem I am seeing is that there does not seem to be a way for zram to tell the upper layers to stop swapping out
True. The zram is block device so at a moment, I don't want to make zram swap-specific if it's possible.
Quoted 4 lines
because the swap device is essentially "full" (since there is no more memory available for zram pages). Has anyone thought about this issue already and have ideas how to solve this or am I missing something and I should not be seeing this issue?
It's true. We might need feedback loop and it shoudn't be specific for zram-swap. One think I can imagine is that we could move failed victim pages into LRU active list when the swapout failed so VM will have more weight for file pages than anon ones. For detail, you could see
AOP_WRITEPAGE_ACTIVATEandget_scan_countfor detail.The problem is it's on fs layer while zram is on block layer so what I can think at a moment is follwing as
mm/vmscan.c +2
-0@@ -502,6 +502,8 @@ static pageout_t pageout(struct page *page, struct address_space *mapping,if (!PageWriteback (page)) {/* synchronous write or broken a_ops? */ ClearPageReclaim (page); +if (PageError (page)) +return PAGE_ACTIVATE ; }trace_mm_vmscan_writepage (page,trace_reclaim_flags (page));inc_zone_page_state (page,NR_VMSCAN_WRITE );
Quoted 3 lines
It doesn't prevent swapout at all but it should throttle pick up anonymous pages for reclaiming so file-backed pages will be preferred by VM so sometime, zsmalloc succeed to allocate a free page and swapout will resume again.
I tried the above suggestion but it does not seem to have any noticeable impact. The system is still trying to swap out at a very high rate after zram reported failure to swap out. The error logging is actually so much that my system crashed due to excessive logging (we have a watchdog that is not getting pet because the kernel is busy logging kernel messages).
There isn't anything that can be set to tell the fs layer to back off completely for a while (congestion control)?
Olav Haugan
Signature · 8 lines
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Hi Olav,
Quoted 7 lines · Olav Haugan
I tried the above suggestion but it does not seem to have any noticeable impact. The system is still trying to swap out at a very high rate after zram reported failure to swap out. The error logging is actually so much that my system crashed due to excessive logging (we have a watchdog that is not getting pet because the kernel is busy logging kernel messages).
I have a question that why the low memory killer didn't get triggered in this situation?
Is it possible to set the LMK a bit more aggressive?
Quoted 3 lines
There isn't anything that can be set to tell the fs layer to back off completely for a while (congestion control)?
The other way I think might fix your issue is the same as your mentioned in your previous email.
Set the congested bit for swap device also.
Like:
(quoted diff) +0 -0
b/drivers/staging/zram/zram_drv.c
index 91d94b5..c4fc63e 100644
drivers/staging/zram/zram_drv.c +0 -0
@@ -474,6 +474,7 @@ static int zram_bvec_write(struct zram *zram, struct
bio_vec *bvec, u32 index,
if (!handle) {
pr_info("Error allocating memory for compressed page:
%u, size=%zu\n",
index, clen);
+ blk_set_queue_congested(zram->disk->queue, BLK_RW_ASYNC);
ret = -ENOMEM;
goto out;
}mm/vmscan.c +0 -0
@@ -394,8 +394,6 @@ static inline int is_page_cache_freeable(struct page
*page)
static int may_write_to_queue(struct backing_dev_info *bdi,
struct scan_control *sc)
{
- if (current->flags & PF_SWAPWRITE)
- return 1;
--------------------------------------------------------------
For the update of the congested state of zram, I think you can clear it from use space eg. after LMK triggered and reclaimed some memory.
Of course this depends on zram driver to export a sysfs node like "/sys/block/zram0/clear_congested".
Signature · 8 lines
Regards, -Bob -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
I did get zswap working without a backing store, using Bob's patches from http://thread.gmane.org/gmane.linux.kernel.mm/105627/focus=105642
applied to 3.11-rc6, The major issue we had with it was that i915 graphics buffers were getting corrupted somehow, which wasn't a problem with zram on the same kernel version. I'm sure with a little more investigation that problem could be fixed, though, and I don't think there were any other major problems with it in our (admittedly not very exhaustive) testing of those patches.
Hope that helps,
Steve Barber
Quoted 6 lines · Luigi Semenzato
[apologies for the previous HTML email]
Hi Olav,
I haven't personally done it. Seth outlines the configuration in this thread:
Quoted 36 lines
http://thread.gmane.org/gmane.linux.kernel.mm/105378/focus=105543
Stephen, can you add more detail from your experience?
Thanks!
Quoted 28 lines · Olav Haugan
Hi Luigi,
Quoted 7 lines · Luigi Semenzato
Quoted 6 lines · Olav Haugan
Hi Bob, Luigi,
Quoted 2 lines · Bob Liu
On 10/24/2013 05:51 AM, Olav Haugan wrote:
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
By the way, could you take a try with zswap? Which can write pages to real swap device if compressed pool is full.
zswap might not be feasible in all cases if you only have flash as backing storage.
Zswap can be configured to run without a backing storage.
I was under the impression that zswap requires a backing storage. Can you elaborate on how to configure zswap to not need a backing storage?
Olav Haugan
--
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>