fadvise DONTNEED implementation (or lack thereof)

DISCUSSION Ben Gamari 7 participants 24 messages Nov 4, 2010 – Nov 18, 2010 Memory management

Share & tools

Ben Gamari

author

2010-11-04 05:59 UTC #0 lore raw

I've recently been trying to track down the root cause of my server's persistent issue of thrashing horribly after being left inactive. It seems that the issue is likely my nightly backup schedule (using rsync) which traverses my entire 50GB home directory. I was surprised to find that rsync does not use fadvise to notify the kernel of its use-once data usage pattern.

It looks like a patch[1] was written (although never merged, it seems) incorporating fadvise support, but I found its implementation rather odd, using mincore() and FADV_DONTNEED to kick out only regions brought in by rsync. It seemed to me the simpler and more appropriate solution would be to simply flag every touched file with FADV_NOREUSE and let the kernel manage automatically expelling used pages.

After looking deeper into the kernel implementation[2] of fadvise() the reason for using DONTNEED became more apparant. It seems that the kernel implements NOREUSE as a noop. A little googling revealed[3] that I not the first person to encounter this limitation. It looks like a few folks[4] have discussed addressing the issue in the past, but nothing has happened as of 2.6.36. Are there plans to implement this functionality in the near future? It seems like the utility of fadvise is severely limited by lacking support for NOREUSE.

Cheers,

- Ben

[1] http://insights.oetiker.ch/linux/fadvise.html
[2] http://lxr.free-electrons.com/source/mm/fadvise.c?a=avr32
[3] https://issues.apache.org/jira/browse/CASSANDRA-1470

    http://chbits.blogspot.com/2010/06/lucene-and-fadvisemadvise.html
[4] http://www.mail-archive.com/[email protected]/msg179576.html
    http://lkml.indiana.edu/hypermail/linux/kernel/0807.2/0442.html

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Wayne Davison

2010-11-06 16:23 UTC #1 lore raw

Quoted 2 lines · Ben Gamari

It looks like a few folks have discussed addressing the issue in the past, but nothing has happened as of 2.6.36.

Yeah, the linux code for this has long been buggy and near useless. What is really needed is a way for some file access to be marked as generating low-priority cache data into the filesystem cache. Such a flag should only apply to newly cached data, so that copying a file that was cached by some other program would not lower its cache priority (nor kick it out of the cache). If some other process comes along and reads from the low-priority cache with a normal-priority read, it should get upgraded to normal priority. Something like that seems pretty simple and useful.

As for rsync, all current implementations of cache dropping are way too klugey to go into rsync. I'd personally suggest that someone create a linux-specific pre-load library that overrides read() and write() calls and use that when running rsync (or whatever else) to implement the extreme weirdness that is currently needed for cache twiddling.

..wayne..

KOSAKI Motohiro

2010-11-09 07:28 UTC #2 lore raw

In reply to Ben Gamari

Quoted 14 lines

I've recently been trying to track down the root cause of my server's persistent issue of thrashing horribly after being left inactive. It seems that the issue is likely my nightly backup schedule (using rsync) which traverses my entire 50GB home directory. I was surprised to find that rsync does not use fadvise to notify the kernel of its use-once data usage pattern.
It looks like a patch[1] was written (although never merged, it seems) incorporating fadvise support, but I found its implementation rather odd, using mincore() and FADV_DONTNEED to kick out only regions brought in by rsync. It seemed to me the simpler and more appropriate solution would be to simply flag every touched file with FADV_NOREUSE and let the kernel manage automatically expelling used pages.

Quoted 8 lines

After looking deeper into the kernel implementation[2] of fadvise() the reason for using DONTNEED became more apparant. It seems that the kernel implements NOREUSE as a noop. A little googling revealed[3] that I not the first person to encounter this limitation. It looks like a few folks[4] have discussed addressing the issue in the past, but nothing has happened as of 2.6.36. Are there plans to implement this functionality in the near future? It seems like the utility of fadvise is severely limited by lacking support for NOREUSE.

btw, Other OSs seems to also don't implement it.
example,

http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/lib/libc/port/gen/posix_fadvise.c

     35 /*
     36  * SUSv3 - file advisory information
     37  *
     38  * This function does nothing, but that's OK because the
     39  * Posix specification doesn't require it to do anything
     40  * other than return appropriate error numbers.
     41  *
     42  * In the future, a file system dependent fadvise() or fcntl()
     43  * interface, similar to madvise(), should be developed to enable
     44  * the kernel to optimize I/O operations based on the given advice.
     45  */
     46 
     47 /* ARGSUSED1 */
     48 int
     49 posix_fadvise(int fd, off_t offset, off_t len, int advice)
     50 {
     51 	struct stat64 statb;
     52 
     53 	switch (advice) {
     54 	case POSIX_FADV_NORMAL:
     55 	case POSIX_FADV_RANDOM:
     56 	case POSIX_FADV_SEQUENTIAL:
     57 	case POSIX_FADV_WILLNEED:
     58 	case POSIX_FADV_DONTNEED:
     59 	case POSIX_FADV_NOREUSE:
     60 		break;
     61 	default:
     62 		return (EINVAL);
     63 	}
     64 	if (len < 0)
     65 		return (EINVAL);
     66 	if (fstat64(fd, &statb) != 0)
     67 		return (EBADF);
     68 	if (S_ISFIFO(statb.st_mode))
     69 		return (ESPIPE);
     70 	return (0);
     71 }

So, I don't think application developers will use fadvise() aggressively because we don't have a cross platform agreement of a fadvice behavior.

KOSAKI Motohiro

2010-11-09 08:03 UTC #3 lore raw

Quoted 23 lines

Quoted 14 lines
I've recently been trying to track down the root cause of my server's persistent issue of thrashing horribly after being left inactive. It seems that the issue is likely my nightly backup schedule (using rsync) which traverses my entire 50GB home directory. I was surprised to find that rsync does not use fadvise to notify the kernel of its use-once data usage pattern.
It looks like a patch[1] was written (although never merged, it seems) incorporating fadvise support, but I found its implementation rather odd, using mincore() and FADV_DONTNEED to kick out only regions brought in by rsync. It seemed to me the simpler and more appropriate solution would be to simply flag every touched file with FADV_NOREUSE and let the kernel manage automatically expelling used pages.
Quoted 8 lines
After looking deeper into the kernel implementation[2] of fadvise() the reason for using DONTNEED became more apparant. It seems that the kernel implements NOREUSE as a noop. A little googling revealed[3] that I not the first person to encounter this limitation. It looks like a few folks[4] have discussed addressing the issue in the past, but nothing has happened as of 2.6.36. Are there plans to implement this functionality in the near future? It seems like the utility of fadvise is severely limited by lacking support for NOREUSE.

Quoted 2 lines

btw, Other OSs seems to also don't implement it.
example,

I've heared other OSs status of fadvise() from private mail.

NetBSD: no-op (as linux) FreeBSD/DragonflyBSD/OpenBSD: don't exist posix_fadvise(2)

Ben Gamari

author

2010-11-09 12:54 UTC #4 lore raw

In reply to KOSAKI Motohiro

Quoted 3 lines · KOSAKI Motohiro

So, I don't think application developers will use fadvise() aggressively because we don't have a cross platform agreement of a fadvice behavior.

I strongly disagree. For a long time I have been trying to resolve interactivity issues caused by my rsync-based backup script. Many kernel developers have said that there is nothing the kernel can do without more information from user-space (e.g. cgroups, madvise). While cgroups help, the fix is round-about at best and requires configuration where really none should be necessary. The easiest solution for everyone involved would be for rsync to use FADV_DONTNEED. The behavior doesn't need to be perfectly consistent between platforms for the flag to be useful so long as each implementation does something sane to help use-once access patterns.

People seem to mention frequently that there are no users of FADV_DONTNEED and therefore we don't need to implement it. It seems like this is ignoring an obvious catch-22. Currently rsync has no fadvise support at all, since using[1] the implemented hints to get the desired effect is far too complicated^M^M^M^Mhacky to be considered merge-worthy. Considering the number of Google hits returned for fadvise, I wouldn't be surprised if there were countless other projects with this same difficulty. We want to be able to tell the kernel about our useage patterns, but the kernel won't listen.

Cheers,

- Ben

[1] http://insights.oetiker.ch/linux/fadvise.html

KOSAKI Motohiro

2010-11-14 05:09 UTC #5 lore raw

Quoted 15 lines · KOSAKI Motohiro

Quoted 3 lines
So, I don't think application developers will use fadvise() aggressively because we don't have a cross platform agreement of a fadvice behavior.
I strongly disagree. For a long time I have been trying to resolve interactivity issues caused by my rsync-based backup script. Many kernel developers have said that there is nothing the kernel can do without more information from user-space (e.g. cgroups, madvise). While cgroups help, the fix is round-about at best and requires configuration where really none should be necessary. The easiest solution for everyone involved would be for rsync to use FADV_DONTNEED. The behavior doesn't need to be perfectly consistent between platforms for the flag to be useful so long as each implementation does something sane to help use-once access patterns.

Quoted 9 lines

People seem to mention frequently that there are no users of FADV_DONTNEED and therefore we don't need to implement it. It seems like this is ignoring an obvious catch-22. Currently rsync has no fadvise support at all, since using[1] the implemented hints to get the desired effect is far too complicated^M^M^M^Mhacky to be considered merge-worthy. Considering the number of Google hits returned for fadvise, I wouldn't be surprised if there were countless other projects with this same difficulty. We want to be able to tell the kernel about our useage patterns, but the kernel won't listen.

Because we have an alternative solution already. please try memcgroup :)

Ben Gamari

author

2010-11-14 05:21 UTC #6 lore raw

Quoted 2 lines · KOSAKI Motohiro

Because we have an alternative solution already. please try memcgroup :)

Alright, fair enough. It still seems like there are many cases where fadvise seems more appropriate, but memcg should at least satisfy my personal needs so I'll shut up now. Thanks!

- Ben

Brian K. White

2010-11-14 21:33 UTC #7 lore raw

Quoted 8 lines · Ben Gamari

Quoted 2 lines · KOSAKI Motohiro
Because we have an alternative solution already. please try memcgroup :)
Alright, fair enough. It still seems like there are many cases where fadvise seems more appropriate, but memcg should at least satisfy my personal needs so I'll shut up now. Thanks!
- Ben

Could someone expand on this a little?

The "there are no users of this feature" argument is indeed a silly one. I've only wanted the ability to perform i/o without poisoning the cache since oh, 10 or more years ago at least. It really hurts my users since they are all direct login interactive db app users. No load balancing web interface can hide the fact when a box goes to a crawl.

How would one use memcgroup to prevent a backup or other large file operation from wiping out the cache with used-once garbage?

(note for rsync in particular, how does this help rsync on other platforms?)

Signature · 8 lines

bkw

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Minchan Kim

2010-11-15 06:07 UTC #8 lore raw

In reply to KOSAKI Motohiro

Quoted 3 lines · KOSAKI Motohiro

Quoted 3 lines · KOSAKI Motohiro
Quoted 2 lines
So, I don't think application developers will use fadvise() aggressively because we don't have a cross platform agreement of a fadvice behavior

KOSAKI Motohiro

2010-11-15 07:09 UTC #9 lore raw

Quoted 10 lines

Quoted 1 line
Because we have an alternative solution already. please try memcgroup :)
I think memcg could be a solution of them but fundamental solution is that we have to cure it in VM itself.
I feel it's absolutely absurd to enable and use memcg for amending it.
I wonder what's the problem in Peter's patch 'drop behind'.
http://www.mail-archive.com/[email protected]/msg179576.html
Could anyone tell me why it can't accept upstream?

I don't know the reason. And this one looks reasonable to me. I'm curious the above patch solve rsync issue or not.
Minchan, have you tested it yourself?

Minchan Kim

2010-11-15 07:19 UTC #10 lore raw

Quoted 6 lines · KOSAKI Motohiro

Quoted 6 lines
Quoted 1 line
Because we have an alternative solution already. please try memcgroup :)
I think memcg could be a solution of them but fundamental solution is that we have to cure it in VM itself.
I feel it's absolutely absurd to enable and use memcg for amending it.

Quoted 8 lines

Quoted 4 lines
I wonder what's the problem in Peter's patch 'drop behind'.
http://www.mail-archive.com/[email protected]/msg179576.html
Could anyone tell me why it can't accept upstream?
I don't know the reason. And this one looks reasonable to me. I'm curious the above patch solve rsync issue or not.
Minchan, have you tested it yourself?

Still yet. :)
If we all think it's reasonable, it would be valuable to adjust it with current mmotm and see the effect.

Quoted 3 lines

Signature · 2 lines

Kind regards,
Minchan Kim

KOSAKI Motohiro

2010-11-15 07:28 UTC #11 lore raw

Quoted 13 lines

Quoted 11 lines · KOSAKI Motohiro
Quoted 10 lines
Quoted 1 line
Because we have an alternative solution already. please try memcgroup :)
I think memcg could be a solution of them but fundamental solution is that we have to cure it in VM itself.
I feel it's absolutely absurd to enable and use memcg for amending it.
I wonder what's the problem in Peter's patch 'drop behind'.
http://www.mail-archive.com/[email protected]/msg179576.html
Could anyone tell me why it can't accept upstream?

Quoted 7 lines

Quoted 3 lines
I don't know the reason. And this one looks reasonable to me. I'm curious the above patch solve rsync issue or not.
Minchan, have you tested it yourself?
Still yet. :)
If we all think it's reasonable, it would be valuable to adjust it with current mmotm and see the effect.

Who can make rsync like io pattern test suite? a code change is easy. but to comfirm justification is more harder work.

Minchan Kim

2010-11-15 07:46 UTC #12 lore raw

Quoted 17 lines · KOSAKI Motohiro

Quoted 17 lines
Quoted 14 lines · KOSAKI Motohiro
Quoted 10 lines
Quoted 1 line
Because we have an alternative solution already. please try memcgroup :)
I think memcg could be a solution of them but fundamental solution is that we have to cure it in VM itself.
I feel it's absolutely absurd to enable and use memcg for amending it.
I wonder what's the problem in Peter's patch 'drop behind'.
http://www.mail-archive.com/[email protected]/msg179576.html
Could anyone tell me why it can't accept upstream?
I don't know the reason. And this one looks reasonable to me. I'm curious the above patch solve rsync issue or not.
Minchan, have you tested it yourself?

Quoted 6 lines

Quoted 3 lines
Still yet. :)
If we all think it's reasonable, it would be valuable to adjust it with current mmotm and see the effect.
Who can make rsync like io pattern test suite? a code change is easy. but to comfirm justification is more harder work.

Maybe Ben, Brian those reports the problem. :)

Signature · 9 lines

Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Peter Zijlstra

2010-11-15 08:46 UTC #14 lore raw

In reply to Minchan Kim

Quoted 27 lines · Minchan Kim

Quoted 25 lines · KOSAKI Motohiro
Quoted 24 lines · KOSAKI Motohiro
Quoted 3 lines
So, I don't think application developers will use fadvise() aggressively because we don't have a cross platform agreement of a fadvice behavior.
I strongly disagree. For a long time I have been trying to resolve interactivity issues caused by my rsync-based backup script. Many kernel developers have said that there is nothing the kernel can do without more information from user-space (e.g. cgroups, madvise). While cgroups help, the fix is round-about at best and requires configuration where really none should be necessary. The easiest solution for everyone involved would be for rsync to use FADV_DONTNEED. The behavior doesn't need to be perfectly consistent between platforms for the flag to be useful so long as each implementation does something sane to help use-once access patterns.
People seem to mention frequently that there are no users of FADV_DONTNEED and therefore we don't need to implement it. It seems like this is ignoring an obvious catch-22. Currently rsync has no fadvise support at all, since using[1] the implemented hints to get the desired effect is far too complicated^M^M^M^Mhacky to be considered merge-worthy. Considering the number of Google hits returned for fadvise, I wouldn't be surprised if there were countless other projects with this same difficulty. We want to be able to tell the kernel about our useage patterns, but the kernel won't listen.

Quoted 1 line

Quoted 1 line
Because we have an alternative solution already. please try memcgroup :)

Using memcgroup for this is utter crap, it just contains the trainwreck, it doesn't solve it in any way.

Quoted 3 lines

I think memcg could be a solution of them but fundamental solution is that we have to cure it in VM itself.
I feel it's absolutely absurd to enable and use memcg for amending it.

Agreed..

Quoted 4 lines

I wonder what's the problem in Peter's patch 'drop behind'.
http://www.mail-archive.com/[email protected]/msg179576.html
Could anyone tell me why it can't accept upstream?

Read the thread, its quite clear nobody got convinced it was a good idea and wanted to fix the use-once policy, then Rik rewrote all of page-reclaim.

Minchan Kim

2010-11-15 09:05 UTC #15 lore raw

Quoted 39 lines · Peter Zijlstra

Quoted 27 lines · Minchan Kim
Quoted 25 lines · KOSAKI Motohiro
Quoted 24 lines · KOSAKI Motohiro
Quoted 3 lines
So, I don't think application developers will use fadvise() aggressively because we don't have a cross platform agreement of a fadvice behavior.
I strongly disagree. For a long time I have been trying to resolve interactivity issues caused by my rsync-based backup script. Many kernel developers have said that there is nothing the kernel can do without more information from user-space (e.g. cgroups, madvise). While cgroups help, the fix is round-about at best and requires configuration where really none should be necessary. The easiest solution for everyone involved would be for rsync to use FADV_DONTNEED. The behavior doesn't need to be perfectly consistent between platforms for the flag to be useful so long as each implementation does something sane to help use-once access patterns.
People seem to mention frequently that there are no users of FADV_DONTNEED and therefore we don't need to implement it. It seems like this is ignoring an obvious catch-22. Currently rsync has no fadvise support at all, since using[1] the implemented hints to get the desired effect is far too complicated^M^M^M^Mhacky to be considered merge-worthy. Considering the number of Google hits returned for fadvise, I wouldn't be surprised if there were countless other projects with this same difficulty. We want to be able to tell the kernel about our useage patterns, but the kernel won't listen.
Quoted 1 line
Quoted 1 line
Because we have an alternative solution already. please try memcgroup :)
Using memcgroup for this is utter crap, it just contains the trainwreck, it doesn't solve it in any way.
Quoted 3 lines
I think memcg could be a solution of them but fundamental solution is that we have to cure it in VM itself.
I feel it's absolutely absurd to enable and use memcg for amending it.
Agreed..

Quoted 9 lines

Quoted 4 lines
I wonder what's the problem in Peter's patch 'drop behind'.
http://www.mail-archive.com/[email protected]/msg179576.html
Could anyone tell me why it can't accept upstream?
Read the thread, its quite clear nobody got convinced it was a good idea and wanted to fix the use-once policy, then Rik rewrote all of page-reclaim.

Thanks for the information.
I hope this is a chance to rethink about it. Rik, Could you give us to any comment about this idea?

Signature · 9 lines

Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

KOSAKI Motohiro

2010-11-15 09:10 UTC #23 lore raw

In reply to Peter Zijlstra

Quoted 8 lines

Quoted 4 lines
I wonder what's the problem in Peter's patch 'drop behind'.
http://www.mail-archive.com/[email protected]/msg179576.html
Could anyone tell me why it can't accept upstream?
Read the thread, its quite clear nobody got convinced it was a good idea and wanted to fix the use-once policy, then Rik rewrote all of page-reclaim.

If my understand is correct, rsync touch data twice (for a hash calculation and for a copy). then, currect used-once-heuristics seems still doesn't work.

Ben Gamari

author

2010-11-15 12:46 UTC #13 lore raw

In reply to KOSAKI Motohiro

Quoted 3 lines · KOSAKI Motohiro

Who can make rsync like io pattern test suite? a code change is easy. but to comfirm justification is more harder work.

I'm afraid I don't have time to work up any code. I would be happy to try the patch with my backup use-case though. I'll just have to think of an objective way of measuring the result.

- Ben

Rik van Riel

2010-11-15 14:48 UTC #16 lore raw

In reply to Minchan Kim

Quoted 8 lines · Minchan Kim

Quoted 7 lines · Peter Zijlstra
Quoted 4 lines · Minchan Kim
I wonder what's the problem in Peter's patch 'drop behind'.
http://www.mail-archive.com/[email protected]/msg179576.html
Could anyone tell me why it can't accept upstream?

Quoted 8 lines

Quoted 4 lines
Read the thread, its quite clear nobody got convinced it was a good idea and wanted to fix the use-once policy, then Rik rewrote all of page-reclaim.
Thanks for the information.
I hope this is a chance to rethink about it. Rik, Could you give us to any comment about this idea?

At the time, there were all kinds of general problems in page reclaim that all needed to be fixed. Peter's patch was mostly a band-aid for streaming IO.

However, now that most of the other page reclaim problems seem to have been resolved, it would be worthwhile to test whether Peter's drop-behind approach gives an additional improvement.

I could see it help by getting rid of already-read pages earlier, leaving more space for read-ahead data.

I suspect it would do fairly little to protect the working set, because we do not scan the active file list at all unless it grows to be larger than the inactive file list.

Signature · 8 lines

All rights reversed

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Minchan Kim

2010-11-17 10:16 UTC #17 lore raw

Quoted 12 lines · Rik van Riel

Quoted 11 lines · Minchan Kim
Quoted 8 lines · Peter Zijlstra
Quoted 4 lines · Minchan Kim
I wonder what's the problem in Peter's patch 'drop behind'.
http://www.mail-archive.com/[email protected]/msg179576.html
Could anyone tell me why it can't accept upstream?

Quoted 8 lines

Quoted 8 lines
Quoted 4 lines
Read the thread, its quite clear nobody got convinced it was a good idea and wanted to fix the use-once policy, then Rik rewrote all of page-reclaim.
Thanks for the information.
I hope this is a chance to rethink about it. Rik, Could you give us to any comment about this idea?

Sorry for late reply, Rik.

Quoted 8 lines

At the time, there were all kinds of general problems in page reclaim that all needed to be fixed. Peter's patch was mostly a band-aid for streaming IO.
However, now that most of the other page reclaim problems seem to have been resolved, it would be worthwhile to test whether Peter's drop-behind approach gives an additional improvement.

Okay. I will have a time to make the workload for testing.

Quoted 3 lines

I could see it help by getting rid of already-read pages earlier, leaving more space for read-ahead data.

Yes. Peter's logic breaks demotion if the page is in active list. But I think if it's just active page like rsync's two touch, we have to move tail of inactive although it's in active list.
I will look into this, too.

Quoted 4 lines

I suspect it would do fairly little to protect the working set, because we do not scan the active file list at all unless it grows to be larger than the inactive file list.

Absolutely. But how about rsync's two touch?
It can evict working set.

I need the time for investigation.
Thanks for the comment.

Quoted 4 lines

--
All rights reversed

Signature · 9 lines

Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Minchan Kim

2010-11-17 11:16 UTC #18 lore raw

Quoted 37 lines · Minchan Kim

Quoted 12 lines · Rik van Riel
Quoted 11 lines · Minchan Kim
Quoted 8 lines · Peter Zijlstra
Quoted 4 lines · Minchan Kim
I wonder what's the problem in Peter's patch 'drop behind'.
http://www.mail-archive.com/[email protected]/msg179576.html
Could anyone tell me why it can't accept upstream?
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Read the thread, its quite clear nobody got convinced it was a good idea and wanted to fix the use-once policy, then Rik rewrote all of page-reclaim.
Thanks for the information.
I hope this is a chance to rethink about it. Rik, Could you give us to any comment about this idea?
Sorry for late reply, Rik.
Quoted 8 lines
At the time, there were all kinds of general problems in page reclaim that all needed to be fixed. Peter's patch was mostly a band-aid for streaming IO.
However, now that most of the other page reclaim problems seem to have been resolved, it would be worthwhile to test whether Peter's drop-behind approach gives an additional improvement.
Okay. I will have a time to make the workload for testing.
Quoted 1 line

Quoted 7 lines

Quoted 2 lines
I could see it help by getting rid of already-read pages earlier, leaving more space for read-ahead data.
Yes. Peter's logic breaks demotion if the page is in active list. But I think if it's just active page like rsync's two touch, we have to move tail of inactive although it's in active list.
I will look into this, too.

Most important thing is how to know it's real working set or just trick by two touch.
If it's very hard, recent Mandeep's patch can be a another solution.
http://thread.gmane.org/gmane.linux.kernel.mm/54572
I will try it, too.

Signature · 9 lines

Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Rik van Riel

2010-11-17 16:22 UTC #19 lore raw

In reply to Minchan Kim

Quoted 5 lines · Minchan Kim

Absolutely. But how about rsync's two touch?
It can evict working set.
I need the time for investigation.
Thanks for the comment.

Maybe we could exempt MADV_SEQUENTIAL and FADV_SEQUENTIAL touches from promoting the page to the active list?

Then we just need to make sure rsync uses fadvise properly to keep the working set protected from rsync.

Signature · 8 lines

All rights reversed

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Minchan Kim

2010-11-18 02:47 UTC #20 lore raw

Quoted 11 lines · Rik van Riel

Quoted 5 lines · Minchan Kim
Absolutely. But how about rsync's two touch?
It can evict working set.
I need the time for investigation.
Thanks for the comment.
Maybe we could exempt MADV_SEQUENTIAL and FADV_SEQUENTIAL touches from promoting the page to the active list?

The problem is non-mapped file page.
non-mapped file page promotion happens by only mark_page_accessed. But it doesn't enough information to prevent promotion(ex, vma or file) Hmm.. Do other guys have any idea?

Here is another idea.
Current problem is following as.
User can use fadivse with FADV_DONTNEED.
But problem is that it can't affect when it meet dirty pages. So user have to sync dirty page before calling fadvise with FADV_DONTNEED. It would lose performance.

Let's add some semantic of FADV_DONTNEED.
It invalidates only pages which are not dirty.
If it meets dirty page, let's move the page into inactive's tail or head. If we move the page into tail, shrinker can move it into head again for deferred write if it isn't written the backed device.

Quoted 6 lines

Then we just need to make sure rsync uses fadvise properly to keep the working set protected from rsync.
--
All rights reversed

Signature · 9 lines

Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Rik van Riel

2010-11-18 03:25 UTC #21 lore raw

Quoted 9 lines · Minchan Kim

Quoted 8 lines · Rik van Riel
Quoted 5 lines · Minchan Kim
Absolutely. But how about rsync's two touch?
It can evict working set.
I need the time for investigation.
Thanks for the comment.

Quoted 7 lines

Quoted 3 lines
Maybe we could exempt MADV_SEQUENTIAL and FADV_SEQUENTIAL touches from promoting the page to the active list?
The problem is non-mapped file page.
non-mapped file page promotion happens by only mark_page_accessed. But it doesn't enough information to prevent promotion(ex, vma or file)

I believe we have enough information in filemap.c and can just pass that as a parameter to mark_page_accessed.

Quoted 7 lines

Here is another idea.
Current problem is following as.
User can use fadivse with FADV_DONTNEED.
But problem is that it can't affect when it meet dirty pages. So user have to sync dirty page before calling fadvise with FADV_DONTNEED. It would lose performance.

Quoted 5 lines

Let's add some semantic of FADV_DONTNEED.
It invalidates only pages which are not dirty.
If it meets dirty page, let's move the page into inactive's tail or head. If we move the page into tail, shrinker can move it into head again for deferred write if it isn't written the backed device.

That sounds like a good idea.

Signature · 8 lines

All rights reversed

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Minchan Kim

2010-11-18 03:46 UTC #22 lore raw

Quoted 16 lines · Rik van Riel

Quoted 15 lines · Minchan Kim
Quoted 12 lines · Rik van Riel
Quoted 5 lines · Minchan Kim
Absolutely. But how about rsync's two touch?
It can evict working set.
I need the time for investigation.
Thanks for the comment.
Maybe we could exempt MADV_SEQUENTIAL and FADV_SEQUENTIAL touches from promoting the page to the active list?

Quoted 6 lines

Quoted 3 lines
The problem is non-mapped file page.
non-mapped file page promotion happens by only mark_page_accessed. But it doesn't enough information to prevent promotion(ex, vma or file)
I believe we have enough information in filemap.c and can just pass that as a parameter to mark_page_accessed.

FADV_SEQUENTIAL is per file/vma semantic and It is used by many place. I think changing all those places isn't simple and I don't want to add new structure to propagate the information to mark_page_accessed.

Quoted 11 lines

Quoted 11 lines
Here is another idea.
Current problem is following as.
User can use fadivse with FADV_DONTNEED.
But problem is that it can't affect when it meet dirty pages. So user have to sync dirty page before calling fadvise with FADV_DONTNEED. It would lose performance.
Let's add some semantic of FADV_DONTNEED.
It invalidates only pages which are not dirty.
If it meets dirty page, let's move the page into inactive's tail or head