mm: don't flush TLB when propagate PTE access bit to struct page.
Subsystems (3)
PATCH 1/1 mm: don't flush TLB when propagate PTE access bit to struct page.
kswapd's use case of hardware PTE accessed bit is to approximate page LRU. The ActiveLRU demotion to InactiveLRU are not base on accessed bit, while it is only used to promote when a page is on inactive LRU list. All of the state transitions are triggered by memory pressure and thus has weak relationship with respect to time. In addition, hardware already transparently flush tlb whenever CPU context switch processes and given limited hardware TLB resource, the time period in which a page is accessed but not yet propagated to struct page is very small in practice. With the nature of approximation, kernel really don't need to flush TLB for changing PTE's access bit. This commit removes the flush operation from it.
--- include/linux/mmu_notifier.h | 12 ++++++++++++ mm/rmap.c | 2 +- 2 files changed, 13 insertions(+), 1 deletions(-)
include/linux/mmu_notifier.h +12 -0
@@ -254,6 +254,17 @@ static inline void mmu_notifier_mm_destroy(struct mm_struct *mm) __young; \ }) +#define ptep_clear_young_notify (__vma, __address, __ptep) \ +({ \ + int __young; \ + struct vm_area_struct *___vma = __vma; \ + unsigned long ___address = __address; \ + __young = ptep_test_and_clear_young(___vma, ___address, __ptep);\ + __young |= mmu_notifier_clear_flush_young(___vma->vm_mm, \ + ___address); \ + __young; \ +}) +#define set_pte_at_notify (__mm, __address, __ptep, __pte) \ ({ \ struct mm_struct *___mm = __mm; \
@@ -304,6 +315,7 @@ static inline void mmu_notifier_mm_destroy(struct mm_struct *mm) { } +#define ptep_clear_young_notify ptep_test_and_clear_young#define ptep_clear_flush_young_notify ptep_clear_flush_young#define ptep_clear_flush_notify ptep_clear_flush#define set_pte_at_notify set_pte_at
mm/rmap.c +1 -1
@@ -506,7 +506,7 @@ int page_referenced_one(struct page *page, struct vm_area_struct *vma, goto out_unmap; } -if (ptep_clear_flush_young_notify (vma, address, pte)) { +if (ptep_clear_young_notify (vma, address, pte)) {/* * Don't treat a reference through a sequentially read * mapping as such. If the page has been used in
Signature · 7 lines
1.7.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 10 lines · Ying Han
kswapd's use case of hardware PTE accessed bit is to approximate page LRU. The ActiveLRU demotion to InactiveLRU are not base on accessed bit, while it is only used to promote when a page is on inactive LRU list. All of the state transitions are triggered by memory pressure and thus has weak relationship with respect to time. In addition, hardware already transparently flush tlb whenever CPU context switch processes and given limited hardware TLB resource, the time period in which a page is accessed but not yet propagated to struct page is very small in practice. With the nature of approximation, kernel really don't need to flush TLB for changing PTE's access bit. This commit removes the flush operation from it.
Quoted 2 lines
Signed-off-by Ying Han<[email protected]>Singed-off-by Ken Chen<[email protected]>
The reasoning behind the patch makes sense.
However, have you measured any improvements in run time with this patch? The VM is already tweaked to minimize the number of pages that get aged, so it would be interesting to know where you saw issues.
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 24 lines · Rik van Riel
Quoted 20 lines · Ying Han
kswapd's use case of hardware PTE accessed bit is to approximate page LRU. The
ActiveLRU demotion to InactiveLRU are not base on accessed bit, while it is only
used to promote when a page is on inactive LRU list. All of the state transitions
are triggered by memory pressure and thus has weak relationship with respect to
time. In addition, hardware already transparently flush tlb whenever CPU context
switch processes and given limited hardware TLB resource, the time period in
which a page is accessed but not yet propagated to struct page is very small
in practice. With the nature of approximation, kernel really don't need to flush TLB
for changing PTE's access bit. This commit removes the flush operation from it.Quoted 2 lines
Signed-off-by Ying Han<[email protected]>Singed-off-by Ken Chen<[email protected]>
Quoted 6 lines
The reasoning behind the patch makes sense.
However, have you measured any improvements in run time with this patch? The VM is already tweaked to minimize the number of pages that get aged, so it would be interesting to know where you saw issues.
Firstly, not all CPUs do flush the TLB on VM switch, and secondly, it would be theoretically possible to spin and never be able to flush free pages even if none are ever being touched.
It doesn't have to be an absurdly tiny machine, either. You could cover a good few megs with TLBs (and a small embedded system could easily have less than that of mapped memory on its LRU).
I agree the theory is fine because if the CPU thinks it is worth to keep a TLB entry around, then it probably knows better than our stupid LRU :) And TLB flushing can get nasty when we start swapping a lot with threaded apps.
However, to handle corner cases it should either:
flush all TLBs once per *something* [eg. every scan priority level above N, or every N pages scanned, etc]
start doing the flush versions of the ptep manipulation when memory pressure is getting high.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 45 lines · Nick Piggin
Quoted 24 lines · Rik van Riel
Quoted 20 lines · Ying Han
kswapd's use case of hardware PTE accessed bit is to approximate page LRU. The
ActiveLRU demotion to InactiveLRU are not base on accessed bit, while it is only
used to promote when a page is on inactive LRU list. All of the state transitions
are triggered by memory pressure and thus has weak relationship with respect to
time. In addition, hardware already transparently flush tlb whenever CPU context
switch processes and given limited hardware TLB resource, the time period in
which a page is accessed but not yet propagated to struct page is very small
in practice. With the nature of approximation, kernel really don't need to flush TLB
for changing PTE's access bit. This commit removes the flush operation from it.Quoted 2 lines
Signed-off-by Ying Han<[email protected]>Singed-off-by Ken Chen<[email protected]>Quoted 6 lines
The reasoning behind the patch makes sense.
However, have you measured any improvements in run time with this patch? The VM is already tweaked to minimize the number of pages that get aged, so it would be interesting to know where you saw issues.
Firstly, not all CPUs do flush the TLB on VM switch, and secondly, it would be theoretically possible to spin and never be able to flush free pages even if none are ever being touched.
It doesn't have to be an absurdly tiny machine, either. You could cover a good few megs with TLBs (and a small embedded system could easily have less than that of mapped memory on its LRU).
I agree the theory is fine because if the CPU thinks it is worth to keep a TLB entry around, then it probably knows better than our stupid LRU :) And TLB flushing can get nasty when we start swapping a lot with threaded apps.
Quoted 8 lines
However, to handle corner cases it should either:
flush all TLBs once per *something* [eg. every scan priority level above N, or every N pages scanned, etc]
start doing the flush versions of the ptep manipulation when memory pressure is getting high.
I'm sorry, that's absurd, ignore that :)
However, it's a scary change -- higher chance of reclaiming a TLB covered page.
I had a vague memory of this problem biting someone when this flush wasn't actually done properly... maybe powerpc.
But anyway, same solution could be possible, by flushing every N pages scanned.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 14 lines · Nick Piggin
Quoted 13 lines · Nick Piggin
Quoted 12 lines · Rik van Riel
Quoted 11 lines · Ying Han
kswapd's use case of hardware PTE accessed bit is to approximate page LRU. The
ActiveLRU demotion to InactiveLRU are not base on accessed bit, while it is only
used to promote when a page is on inactive LRU list. All of the state transitions
are triggered by memory pressure and thus has weak relationship with respect to
time. In addition, hardware already transparently flush tlb whenever CPU context
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
switch processes and given limited hardware TLB resource, the time period in
which a page is accessed but not yet propagated to struct page is very small
in practice. With the nature of approximation, kernel really don't need to flush TLB
for changing PTE's access bit. This commit removes the flush operation from it.
It should at least add a comment there in page_referenced_one(), that a TLB flush ought to be done, but is now judged not worth the effort.
(I'd expect architectures to differ on whether it's worth the effort.)
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Signed-off-by Ying Han<[email protected]>Singed-off-by Ken Chen<[email protected]>
Hey, Ken, switch off those curling tongs :)
Quoted 1 line
However, it's a scary change -- higher chance of reclaiming a TLB covered page.
Yes, I was often tempted to make such a change in the past; but ran away when it appeared to be in danger of losing the pte referenced bit of precisely the most intensively referenced pages.
Ying's point (about what the pte referenced bit is being used for in our current implementation) is interesting, and might have tipped the balance; but that's not clear to me - and the flush is only done when mm is on CPU.
Quoted 5 lines
I had a vague memory of this problem biting someone when this flush wasn't actually done properly... maybe powerpc.
But anyway, same solution could be possible, by flushing every N pages scanned.
Yes, batching seems safer.
Hugh
Quoted 27 lines · Hugh Dickins
Quoted 2 lines · Nick Piggin
Quoted 1 line · Nick Piggin
On Wed, Oct 27, 2010 at 12:05 PM, Rik van Riel <[email protected]>
Quoted 3 lines · wrote:
Quoted 3 lines
Quoted 3 lines · Ying Han
Quoted 2 lines
kswapd's use case of hardware PTE accessed bit is to approximate page
LRU.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The
ActiveLRU demotion to InactiveLRU are not base on accessed bit, whileit
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
is only
used to promote when a page is on inactive LRU list. All of thestate
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
transitions
are triggered by memory pressure and thus has weak relationship with respect to
time. In addition, hardware already transparently flush tlb wheneverCPU
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
context
switch processes and given limited hardware TLB resource, the timeperiod
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
in
which a page is accessed but not yet propagated to struct page isvery
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
small
in practice. With the nature of approximation, kernel really don't
Quoted 9 lines
need to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
flush TLB
for changing PTE's access bit. This commit removes the flushoperation
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
from it.
It should at least add a comment there in
page_referenced_one(), that a TLB flush ought to be done, but is now judged not worth the effort.
I will make the change here.
Quoted 3 lines
(I'd expect architectures to differ on whether it's worth the effort.)
Right :) I would like hear from upstream if the problem is general enough to solve, and thus
we can plan put further effort into it.
Quoted 9 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Signed-off-by Ying Han<[email protected]>Singed-off-by Ken Chen<[email protected]>Hey, Ken, switch off those curling tongs :)
Quoted 1 line
However, it's a scary change -- higher chance of reclaiming a TLB covered
page.
Quoted 8 lines
Yes, I was often tempted to make such a change in the past; but ran away when it appeared to be in danger of losing the pte referenced bit of precisely the most intensively referenced pages.
Ying's point (about what the pte referenced bit is being used for in our current implementation) is interesting, and might have tipped the balance; but that's not clear to me - and the flush is only done when mm is on CPU.
The initial patch is from Ken, and I am helping out here to get feedback from
upstream and further improvement. :)
Quoted 10 lines
Quoted 2 lines
I had a vague memory of this problem biting someone when this flush
wasn't
Quoted 3 lines
actually done properly... maybe powerpc.
But anyway, same solution could be possible, by flushing every N pages
scanned.
Yes, batching seems safer.
I might be able to take a look at it.
--Ying
Quoted 2 lines
Hugh
Quoted 5 lines · On Wed, 27 Oct 2010 13:35:02 -0700 Ying Han <[email protected]> wrote:
Quoted 5 lines
Quoted 2 lines
I had a vague memory of this problem biting someone when this flush
wasn't
Quoted 2 lines
actually done properly... maybe powerpc.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
But anyway, same solution could be possible, by flushing every N pages
scanned.
Yes, batching seems safer.
I might be able to take a look at it.
I'd like to vote for batching.
Thanks,
-Kame
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 1 line · KAMEZAWA Hiroyuki
I'd like to vote for batching.
Batch mode isn't going to add much value because the effect of accessed bit is already deferred. There are two outcome: (1) the tlb mapping is already flushed due to capacity conflict or (2) process context'ed out. You would want to transfer accessed bit from pte to page table, but flushing TLB on a already deferred operation seems not that useful.
- Ken
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 11 lines · On Thu, 28 Oct 2010 18:30:23 -0700 Ken Chen <[email protected]> wrote:
Quoted 1 line · KAMEZAWA Hiroyuki
I'd like to vote for batching.
Batch mode isn't going to add much value because the effect of accessed bit is already deferred. There are two outcome: (1) the tlb mapping is already flushed due to capacity conflict or (2) process context'ed out. You would want to transfer accessed bit from pte to page table, but flushing TLB on a already deferred operation seems not that useful.
Hmm. Without flushing anywhere in memory reclaim path, a process which cause page fault and enter vmscan will not see his own recent access bit on pages in LRU ?
I think it should be flushed at least once..
Thanks,
-Kame
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 3 lines · KAMEZAWA Hiroyuki
Hmm. Without flushing anywhere in memory reclaim path, a process which cause page fault and enter vmscan will not see his own recent access bit on pages in LRU ?
Worse still, because kernel threads do a lazy mmu switch, even page faulting in the process will not cause the TLB entries to be flushed.
Quoted 1 line
I think it should be flushed at least once..
A periodic flush may make sense.
Maybe something along the lines of if the TLB has not been flushed for over a second (we can see that in timer or scheduler code), flush it?
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 11 lines · Rik van Riel
Quoted 4 lines · KAMEZAWA Hiroyuki
Hmm. Without flushing anywhere in memory reclaim path, a process which cause page fault and enter vmscan will not see his own recent access bit on
pages in LRU ?Worse still, because kernel threads do a lazy mmu switch, even page faulting in the process will not cause the TLB entries to be flushed.
Quoted 7 lines
Quoted 1 line
I think it should be flushed at least once..
A periodic flush may make sense.
Maybe something along the lines of if the TLB has not been flushed for over a second (we can see that in timer or scheduler code), flush it?
What happens if we don't flush TLB?
It will make for old page to pretend young page.
If it is, how does it affect reclaim?
It makes for old page to promote into active list by page_check_references. Of couse, It's not good. But for it, we have to keep wrong TLB until moving head to tail in inactive list. It's very unlikely. That's because TLB is very smalll and the process will be switching out.
If lumpy happens(ie, not waiting from head to tail in inactive list to hold a victim page), that's all right since we ignore young bit in lumpy case.
I think it's no problem unless inactive list is very short. Remained one is kernel thread's lazy TLB flush.
So how about flushing TLB in kswapd scheduled in?
Quoted 3 lines
--
All rights reversed
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 3 lines · Minchan Kim
What happens if we don't flush TLB?
It will make for old page to pretend young page.
If it is, how does it affect reclaim?
Other way around - it will make a young page pretend to be an old page, because the TLB won't know it needs to flush the Accessed bit into the page tables (where the bit was recently cleared).
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 10 lines · Rik van Riel
Quoted 3 lines · Minchan Kim
What happens if we don't flush TLB?
It will make for old page to pretend young page.
If it is, how does it affect reclaim?Other way around - it will make a young page pretend to be an old page, because the TLB won't know it needs to flush the Accessed bit into the page tables (where the bit was recently cleared).
Ying's patch just removes TLB flush when page access bit is changed from young to old.
We still flush TLB flush when from old to young change by ptep_set_access_flags. Do I miss something?
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 12 lines · Minchan Kim
Quoted 10 lines · Rik van Riel
Quoted 3 lines · Minchan Kim
What happens if we don't flush TLB?
It will make for old page to pretend young page.
If it is, how does it affect reclaim?Other way around - it will make a young page pretend to be an old page, because the TLB won't know it needs to flush the Accessed bit into the page tables (where the bit was recently cleared).
Quoted 4 lines
Ying's patch just removes TLB flush when page access bit is changed from young to old.
We still flush TLB flush when from old to young change byptep_set_access_flags. Do I miss something?
The TLB is write-through for the accessed and dirty bits.
If the TLB has a page translation without the accessed bit (and is accessing it), the accessed bit will be set in the page table entry.
If the TLB has a page translation that already has the accessed bit set, nothing will be written to the page table entry.
With Ying's change, we will clear the accessed bit in the page table, without invalidating the corresponding TLB entry.
This can cause accesses to pages to not lead to the accessed bit getting set in the corresponding page table entry.
Making sure the TLB is flushed periodically could fix that issue.
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 35 lines · Rik van Riel
Quoted 14 lines · Minchan Kim
Quoted 11 lines · Rik van Riel
Quoted 3 lines · Minchan Kim
What happens if we don't flush TLB?
It will make for old page to pretend young page.
If it is, how does it affect reclaim?Other way around - it will make a young page pretend to be an old page, because the TLB won't know it needs to flush the Accessed bit into the page tables (where the bit was recently cleared).
Quoted 4 lines
Ying's patch just removes TLB flush when page access bit is changed from young to old.
We still flush TLB flush when from old to young change byptep_set_access_flags. Do I miss something?The TLB is write-through for the accessed and dirty bits.
If the TLB has a page translation without the accessed bit (and is accessing it), the accessed bit will be set in the page table entry.
If the TLB has a page translation that already has the accessed bit set, nothing will be written to the page table entry.
With Ying's change, we will clear the accessed bit in the page table, without invalidating the corresponding TLB entry.
Quoted 6 lines
This can cause accesses to pages to not lead to the accessed bit getting set in the corresponding page table entry.
Making sure the TLB is flushed periodically could fix that issue.
Thanks for the kind explanation, Rik.
I got your point.
While we lost access bit during short time, inactive list size is enough long to promote the page.
So for the page lost access bit is to be working set, it have to be accessed again until moving the tail of inactive list
I think inactive list's size could be point for starting TLB flush.
Quoted 4 lines
--
All rights reversed
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 25 lines · Rik van Riel
Quoted 19 lines · Ying Han
kswapd's use case of hardware PTE accessed bit is to approximate page LRU.
The
ActiveLRU demotion to InactiveLRU are not base on accessed bit, while it is only
used to promote when a page is on inactive LRU list. All of the state transitions
are triggered by memory pressure and thus has weak relationship with respect to
time. In addition, hardware already transparently flush tlb whenever CPU context
switch processes and given limited hardware TLB resource, the time period in
which a page is accessed but not yet propagated to struct page is very small
in practice. With the nature of approximation, kernel really don't need to flush TLB
for changing PTE's access bit. This commit removes the flush operation from it.Quoted 3 lines
Signed-off-by Ying Han<[email protected]>Singed-off-by Ken Chen<[email protected]>
Quoted 7 lines
The reasoning behind the patch makes sense.
However, have you measured any improvements in run time with this patch? The VM is already tweaked to minimize the number of pages that get aged, so it would be interesting to know where you saw issues.
Rik, the workload we were running are some MapReduce jobs.
--Ying
Quoted 4 lines
--
All rights reversed
Quoted 34 lines · Ying Han
On Wed, Oct 27, 2010 at 11:05 AM, Rik van Riel <[email protected] <mailto:[email protected]>> wrote:
On 10/27/2010 01:21 PM, Ying Han wrote:
kswapd's use case of hardware PTE accessed bit is to approximate page LRU. The
ActiveLRU demotion to InactiveLRU are not base on accessed bit, while it is only
used to promote when a page is on inactive LRU list. All of the state transitions
are triggered by memory pressure and thus has weak relationship with respect to
time. In addition, hardware already transparently flush tlb whenever CPU context
switch processes and given limited hardware TLB resource, the time period in
which a page is accessed but not yet propagated to struct page is very small
in practice. With the nature of approximation, kernel really don't need to flush TLB
for changing PTE's access bit. This commit removes the flush operation from it.Signed-off-by Ying Han<[email protected]<mailto:[email protected]>>
Singed-off-by Ken Chen<[email protected]<mailto:[email protected]>>
The reasoning behind the patch makes sense.
Quoted 7 lines
However, have you measured any improvements in run time with this patch? The VM is already tweaked to minimize the number of pages that get aged, so it would be interesting to know where you saw issues.
Rik, the workload we were running are some MapReduce jobs.
Well, what kind of performance improvement did you measure?
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>