optimise copy page range
Subsystems (6)
PATCH 1/2 optimise copy page range
Some of you have seen this before. Just resending because I based my next patch on top of this one.
Suggested by Linus: optimise a condition in the clear_p?d_range functions. Results in one less conditional branch on i386 with gcc-3.4.4
---
linux-2.6-npiggin/mm/memory.c | 9 ++++++--- 1 files changed, 6 insertions(+), 3 deletions(-)
mm/memory.c renamed from mm/memory.c~mm-opt-cpr +6 -3
@@ -98,7 +98,8 @@ static inline void clear_pmd_range(strucpmd_clear (pmd);return ; } -if (!(start & ~PMD_MASK ) && !(end & ~PMD_MASK )) { +if (!((start | end) & ~PMD_MASK )) { +/* Only clear full, aligned ranges */ page =pmd_page (*pmd);pmd_clear (pmd);dec_page_state (nr_page_table_pages);
@@ -131,7 +132,8 @@ static inline void clear_pud_range(struc addr = next; }while (addr && (addr < end)); -if (!(start & ~PUD_MASK ) && !(end & ~PUD_MASK )) { +if (!((start | end) & ~PUD_MASK )) { +/* Only clear full, aligned ranges */ pud_clear (pud);pmd_free_tlb (tlb, __pmd); }
@@ -162,7 +164,8 @@ static inline void clear_pgd_range(struc addr = next; }while (addr && (addr < end)); -if (!(start & ~PGDIR_MASK ) && !(end & ~PGDIR_MASK )) { +if (!((start | end) & ~PGDIR_MASK )) { +/* Only clear full, aligned ranges */ pgd_clear (pgd);pud_free_tlb (tlb, __pud); }
_
PATCH 2/2 page table iterators
I am pretty surprised myself that I was able to consolidate all "page table range" functions into a single type of iterator (well, there are a couple of variations, but it's not too bad).
I thought at least the functions which allocate new page tables would have to be seperate from those which don't.... it turns out that if you slightly change the implementation of the allocating functions, they start walking, quacking, etc. like the other type.
This may also opens the way for things like:
#define for_each_pud(pgd, start, end, pud, pud_start, pud_end) \
if (pud = (pud_t *)pgd, pud_start = start, pud_end = end, 1)
for 2 and 3 level page tables (don't laugh if I've messed up the above completely - you get the idea).
Then the last bit of the performance puzzle I think will just come from inlining things. Now Andi doesn't want to do that (probably rightly so)... what about compiling mm/ with -funit-at-a-time? It doesn't do much harm to the stack, and it shaves off quite a few KB...
Anyway, this iterator patch shaves off a bit itself. I haven't looked at generated code, but I hope it is alright.
npiggin@didi:~/usr/src/linux-2.6$ size mm/memory.o.before text data bss dec hex filename 11349 4 120 11473 2cd1 mm/memory.o npiggin@didi:~/usr/src/linux-2.6$ size mm/memory.o.after text data bss dec hex filename 11221 4 120 11345 2c51 mm/memory.o
Suggestions, help, etc. welcome.
Nick
---
drivers/char/mem.c | 0 linux-2.6-npiggin/arch/i386/mm/ioremap.c | 78 +-- linux-2.6-npiggin/include/asm-generic/pgtable.h | 128 +++++ linux-2.6-npiggin/mm/memory.c | 591 ++++++++++-------------- linux-2.6-npiggin/mm/mprotect.c | 118 +--- linux-2.6-npiggin/mm/msync.c | 159 ++---- linux-2.6-npiggin/mm/vmalloc.c | 222 +++------ 7 files changed, 600 insertions(+), 696 deletions(-)
include/asm-generic/pgtable.h renamed from include/asm-generic/pgtable.h~vm-pgt-walkers +128 -0
@@ -134,4 +134,132 @@ static inline void ptep_mkdirty(pte_t *p#define pgd_offset_gate (mm, addr) pgd_offset(mm, addr)#endif +/* +* for_each_pgd - iterates through pgd entries in a given mm structa +* +* @mm: the mm to use +* @start: the first address, inclusive +* @end: the last address, exclusive +* @pgd: the pgd iterator +* @pgd_start: the first address within the current 'pgd' +* @pgd_end: the last address within the current 'pgd' +* +* mm, start, end are all unchanged +* pgd, pgd_start, pgd_end may all be changed +*/ +#define for_each_pgd (mm, start, end, pgd, pgd_start, pgd_end) \ + for ( pgd = pgd_offset(mm, start), \ + pgd_start = start; \ + pgd_end = (pgd_start + PGDIR_SIZE) & PGDIR_MASK, \ + pgd_end = ((pgd_end && pgd_end <= end) ? pgd_end : end), \ + pgd <= pgd_offset(mm, end-1); \ + pgd_start = pgd_end, \ + pgd++ ) + +/* +* for_each_pgd_k - iterates through pgd entries in the kernel mapping +* +* see for_each_pgd +*/ +#define for_each_pgd_k (start, end, pgd, pgd_start, pgd_end) \ + for ( pgd = pgd_offset_k(start), \ + pgd_start = start; \ + pgd_end = (pgd_start + PGDIR_SIZE) & PGDIR_MASK, \ + pgd_end = ((pgd_end && pgd_end <= end) ? pgd_end : end), \ + pgd <= pgd_offset_k(end-1); \ + pgd_start = pgd_end, \ + pgd++ ) + +/* +* for_each_pud - iterate through pud entries in a given pgd +* +* see for_each_pgd +*/ +#define for_each_pud (pgd, start, end, pud, pud_start, pud_end) \ + for ( pud = pud_offset(pgd, start), \ + pud_start = start; \ + pud_end = (pud_start + PUD_SIZE) & PUD_MASK, \ + pud_end = ((pud_end && pud_end <= end) ? pud_end : end), \ + pud <= pud_offset(pgd, end-1); \ + pud_start = pud_end, \ + pud++ ) + +/* +* for_each_pmd - iterate through pmd entries in a given pud +* +* see for_each_pgd +*/ +#define for_each_pmd (pud, start, end, pmd, pmd_start, pmd_end) \ + for ( pmd = pmd_offset(pud, start), \ + pmd_start = start; \ + pmd_end = (pmd_start + PMD_SIZE) & PMD_MASK, \ + pmd_end = ((pmd_end && pmd_end <= end) ? pmd_end : end), \ + pmd <= pmd_offset(pud, end-1); \ + pmd_start = pmd_end, \ + pmd++ ) + +/* +* for_each_pte_map - iterate through pte entries in a given pmd +* +* @pmd: the pmd to use +* @start: the first address, inclusive +* @end: the last address, exclusive +* @pte: the pte iterator +* @addr: the address of the current 'pte' +* +* for_each_pte_map maps the ptes which it iterates over. +* +* Usage: +* for_each_pte_map(pmd, start, end, pte, addr) { +* // do something with pte and/or addr +* } for_each_pte_map_end; +*/ +#define for_each_pte_map (pmd, start, end, pte, addr) \ +do { \ + int ___i = (end - start) >> PAGE_SHIFT; \ + pte_t *___p = pte_offset_map(pmd, start); \ + pte = ___p; \ + for ( addr = start; \ + ___i--; \ + addr += PAGE_SIZE, pte++) + +#define for_each_pte_map_end \ + pte_unmap(___p); \ +} while (0) + +/* +* for_each_pte_map_nested +* +* See for_each_pte_map. Does a nested mapping of the pte. +*/ +#define for_each_pte_map_nested (pmd, start, end, pte, addr) \ +do { \ + int ___i = (end - start) >> PAGE_SHIFT; \ + pte_t *___p = pte_offset_map_nested(pmd, start); \ + pte = ___p; \ + for ( addr = start; \ + ___i--; \ + addr += PAGE_SIZE, pte++) + +#define for_each_pte_map_nested_end \ + pte_unmap_nested(___p); \ +} while (0) + +/* +* for_each_pte_kernel +* +* See for_each_pte_map. Iterates over kernel ptes. +*/ +#define for_each_pte_kernel (pmd, start, end, pte, addr) \ +do { \ + int ___i = (end - start) >> PAGE_SHIFT; \ + pte_t *___p = pte_offset_kernel(pmd, start); \ + pte = ___p; \ + for ( addr = start; \ + ___i--; \ + addr += PAGE_SIZE, pte++) + +#define for_each_pte_kernel_end \ +} while (0) +#endif /* _ASM_GENERIC_PGTABLE_H */
mm/memory.c renamed from mm/memory.c~vm-pgt-walkers +249 -342
844 lines not on this page · raw message
mm/msync.c renamed from mm/msync.c~vm-pgt-walkers +62 -97
233 lines not on this page · raw message
mm/mprotect.c renamed from mm/mprotect.c~vm-pgt-walkers +46 -72
156 lines not on this page · raw message
mm/vmalloc.c renamed from mm/vmalloc.c~vm-pgt-walkers +83 -139
295 lines not on this page · raw message
arch/i386/mm/ioremap.c renamed from arch/i386/mm/ioremap.c~vm-pgt-walkers +32 -46
118 lines not on this page · raw message
diff -puN include/linux/mm.h~vm-pgt-walkers include/linux/mm.h diff -puN drivers/char/mem.c~vm-pgt-walkers drivers/char/mem.c
_
Re: [PATCH 2/2] page table iterators
Quoted 4 lines · Nick Piggin
I am pretty surprised myself that I was able to consolidate all "page table range" functions into a single type of iterator (well, there are a couple of variations, but it's not too bad).
Ok, this is post-2.6.11 material, so please remind me.
Linus
Re: [PATCH 2/2] page table iterators
Quoted 10 lines · Linus Torvalds
Quoted 3 lines · Nick Piggin
I am pretty surprised myself that I was able to consolidate all "page table range" functions into a single type of iterator (well, there are a couple of variations, but it's not too bad).
Ok, this is post-2.6.11 material, so please remind me.
Sure... it will probably be best to go through -mm, but either way I'll package the patches up nicely and rediff them against 2.6.11 when it comes out.
Re: [PATCH 2/2] page table iterators
In reply to Nick PigginQuoted 3 lines · Nick Piggin
I am pretty surprised myself that I was able to consolidate all "page table range" functions into a single type of iterator (well, there are a couple of variations, but it's not too bad).
I started a similar project - but it uses the existing loops, just using {pte,pmd,pud,pgd}_next. The idea is to optimize page table walking by keeping some state in the struct page of the page table page that says whether an entry is set or not. To make this work I switched everything to indexes instead of pointers.
Main problem are some nasty include loops.
-Andi
Re: [PATCH 2/2] page table iterators
Quoted 5 lines · Andi Kleen
Quoted 3 lines · Nick Piggin
I am pretty surprised myself that I was able to consolidate all "page table range" functions into a single type of iterator (well, there are a couple of variations, but it's not too bad).
Quoted 8 lines
I started a similar project - but it uses the existing loops, just using {pte,pmd,pud,pgd}_next. The idea is to optimize page table walking by keeping some state in the struct page of the page table page that says whether an entry is set or not. To make this work I switched everything to indexes instead of pointers.
Main problem are some nasty include loops.
I though about both ways yesterday, and in the end, I prefer Nick stuff, at least for now. It gives us also more flexibility to change gory implementation details in the future. I still have to run it through a bit of torture testing though.
Ben.
Re: [PATCH 2/2] page table iterators
Quoted 4 lines
I though about both ways yesterday, and in the end, I prefer Nick stuff, at least for now. It gives us also more flexibility to change gory implementation details in the future. I still have to run it through a bit of torture testing though.
They're really solving different problems. My code is just aimed at getting x86-64 fork/exec/etc. as fast as before 4level again (currently they are significantly slower because they have to walk a lot more page tables)
The problem is that the index based approach (I think you have to use indexes for this, pointers get very messy) probably does not fit very well into Nick's complex macros.
Nick's macros are essentially just code transformations with some micro optimizations.
That's not bad, but it won't give you the big speedups the lazy walking approach will give.
And to be honest we only have about 6 or 7 of these walkers in the whole kernel. And 90% of them are in memory.c While doing 4level I think I changed all of them around several times and it wasn't that big an issue. So it's not that we have a big pressing problem here...
-Andi
Re: [PATCH 2/2] page table iterators
Quoted 5 lines · Andi Kleen
And to be honest we only have about 6 or 7 of these walkers in the whole kernel. And 90% of them are in
memory.cWhile doing 4level I think I changed all of them around several times and it wasn't that big an issue. So it's not that we have a big pressing problem here...
We have about 50% of them in memory.c :) But my main problem is more that every single of them is implemented slightly differently.
Going Nick's way is a good start. If they are all consolidated to use the same macro, they will be easier for you to change later on anyway.
Ben.
Re: [PATCH 2/2] page table iterators
In reply to Andi KleenQuoted 5 lines · On Fri, 18 Feb 2005 00:03:42 +0100 Andi Kleen <[email protected]> wrote:
And to be honest we only have about 6 or 7 of these walkers in the whole kernel. And 90% of them are in
memory.cWhile doing 4level I think I changed all of them around several times and it wasn't that big an issue. So it's not that we have a big pressing problem here...
It's super error prone. A regression added by your edit of these walkers for the 4level changes was only discovered and fixed yesterday by the ppc folks.
I absolutely support any change which consolidates these things.
Re: [PATCH 2/2] page table iterators
In reply to Benjamin HerrenschmidtQuoted 10 lines · Benjamin Herrenschmidt
Quoted 5 lines · Andi Kleen
And to be honest we only have about 6 or 7 of these walkers in the whole kernel. And 90% of them are in
memory.cWhile doing 4level I think I changed all of them around several times and it wasn't that big an issue. So it's not that we have a big pressing problem here...We have about 50% of them in
memory.c:) But my main problem is more that every single of them is implemented slightly differently.
No much more. But I only count real walkers, not stuff like vmalloc.
The ioremap duplication over architectures is a bit annoying, but the fix for that would be to factor the code out completely, not only improve walking.
-Andi
Re: [PATCH 2/2] page table iterators
In reply to David S. MillerQuoted 10 lines · David S. Miller
Quoted 5 lines · On Fri, 18 Feb 2005 00:03:42 +0100 Andi Kleen <[email protected]> wrote:
And to be honest we only have about 6 or 7 of these walkers in the whole kernel. And 90% of them are in
memory.cWhile doing 4level I think I changed all of them around several times and it wasn't that big an issue. So it's not that we have a big pressing problem here...It's super error prone. A regression added by your edit of these
Actually it was in Nick's code (PUD layer ;-). But I won't argue that my code didn't have bugs too...
Quoted 4 lines
walkers for the 4level changes was only discovered and fixed yesterday by the ppc folks.
I absolutely support any change which consolidates these things.
The problem is just that these walker macros when they do all the lazy walking stuff will be quite complicated. And I don't really want another uaccess.h-like macro mess.
Yes currently they look simple, but that will change.
Open coding is probably the smaller evil.
And they're really not changed that often.
-Andi
Re: [PATCH 2/2] page table iterators
Quoted 12 lines · Andi Kleen
Quoted 10 lines · David S. Miller
Quoted 5 lines · On Fri, 18 Feb 2005 00:03:42 +0100 Andi Kleen <[email protected]> wrote:
And to be honest we only have about 6 or 7 of these walkers in the whole kernel. And 90% of them are in
memory.cWhile doing 4level I think I changed all of them around several times and it wasn't that big an issue. So it's not that we have a big pressing problem here...
Quoted 7 lines
Quoted 1 line
It's super error prone. A regression added by your edit of these
Actually it was in Nick's code (PUD layer ;-). But I won't argue that my code didn't have bugs too...
I won't look back to see where the error came from :) But yeah it is equally (if not more) likely to have come from me. And it probably did happen because all the code is slightly different and hard to understand.
Quoted 12 lines
Quoted 4 lines
walkers for the 4level changes was only discovered and fixed yesterday by the ppc folks.
I absolutely support any change which consolidates these things.
The problem is just that these walker macros when they do all the lazy walking stuff will be quite complicated. And I don't really want another uaccess.h-like macro mess.
Yes currently they look simple, but that will change.
But even in that case, it will still be better to have the extra complexity once in the macro rather than throughout mm/
Quoted 4 lines
Open coding is probably the smaller evil.
And they're really not changed that often.
It is not so much a matter of changing, so much as having 10 slightly different implementations.
I think it should be easier to go from the iterators patch to perhaps more complex iterators, or some open coding, etc etc. rather than try to put a big complex pt walker on top of these 10 different open coded implementations.
But perhaps I'm missing something you're not - I'd need to see the lazy walking code I guess.
Nick
Re: [PATCH 2/2] page table iterators
Quoted 13 lines · Nick Piggin
Quoted 6 lines · Andi Kleen
The problem is just that these walker macros when they do all the lazy walking stuff will be quite complicated. And I don't really want another uaccess.h-like macro mess.
Yes currently they look simple, but that will change.
But even in that case, it will still be better to have the extra complexity once in the macro rather than throughout
mm/Quoted 2 lines
Open coding is probably the smaller evil.
And they're really not changed that often.
My opinion FWIW: I'm all for regularizing the pagetable loops to work the same way, changing their variables to use the same names, improving their efficiency; but I do like to see what a loop is up to.
list_for_each and friends are very widely used, they're fine, and I'm quite glad to have their prefetching hidden away from me; but usually I groan, grin and bear it, each time someone devises a clever new for_each macro concealing half the details of some specialist loop.
In a minority?
Hugh
Re: [PATCH 2/2] page table iterators
Quoted 11 lines · Hugh Dickins
My opinion FWIW: I'm all for regularizing the pagetable loops to
work the same way, changing their variables to use the same names, improving their efficiency; but I do like to see what a loop is up to.
list_for_eachand friends are very widely used, they're fine, and I'm quite glad to have their prefetching hidden away from me; but usually I groan, grin and bear it, each time someone devises a clever newfor_eachmacro concealing half the details of some specialist loop.In a minority?
Of two.
Let's see what they look like. They'd need to be very good, IMO.
Re: [PATCH 2/2] page table iterators
Quoted 6 lines · Andrew Morton
Quoted 5 lines · Hugh Dickins
My opinion FWIW: I'm all for regularizing the pagetable loops to
work the same way, changing their variables to use the same names, improving their efficiency; but I do like to see what a loop is up to.
Quoted 8 lines
Quoted 6 lines
list_for_eachand friends are very widely used, they're fine, and I'm quite glad to have their prefetching hidden away from me; but usually I groan, grin and bear it, each time someone devises a clever newfor_eachmacro concealing half the details of some specialist loop.In a minority?
Of two.
Well, we basically have that bunch of loops that all do the same thing to iterate the page tables. Only the inner part is different (that is what is done on each PTE).
All of them are slightly differently implemented, some check overflow, some don't, some have redudant checking, some aren't even consistent between all 3/4 loops of a given walk routine set, and we have seen the tendency to introduce subtle bugs in one of them when they all have to be changed for some reason.
I'm all for turning them into something more consistent, and I like the for_each_* idea...
It also allows to completely remove the code of the unused levels on 2 and 3 level page tables easily, regaining some of the perfs lost by the move to 4 levels.
Now, we also need, in the long run, to improve perfs of walking the page tables, especially PTEs, for things like tearing down processes or fork, for example via a bitmap of used PGD entries etc...
With proper iterators, such a thing could be implemented just by modifying the iterator, and all loops would benefit from it.
I think that is enough to justify the move.
Ben.
Re: [PATCH 2/2] page table iterators
Quoted 6 lines · Benjamin Herrenschmidt
All of them are slightly differently implemented, some check overflow, some don't, some have redudant checking, some aren't even consistent between all 3/4 loops of a given walk routine set, and we have seen the tendency to introduce subtle bugs in one of them when they all have to be changed for some reason.
Quoted 7 lines
I'm all for turning them into something more consistent, and I like the for_each_* idea...
It also allows to completely remove the code of the unused levels on 2 and 3 level page tables easily, regaining some of the perfs lost by the move to 4 levels.
It appears to do even better on 2-levels (i386, !PAE) than the old 3-level code, not surprisingly. lmbench fork+exit overhead is under 100us on a 3.4GHz xeon now, which is the lowest I've seen.
Haven't yet pulled out a pre-4-level kernel to see how 3-level compares I guess I'll do that now.
Quoted 7 lines
Now, we also need, in the long run, to improve perfs of walking the page tables, especially PTEs, for things like tearing down processes or fork, for example via a bitmap of used PGD entries etc...
With proper iterators, such a thing could be implemented just by modifying the iterator, and all loops would benefit from it.
After looking at David's bitmap walking code, I'm starting to think that my current macros only _just_ scrape by because of the uniform nature of the walkers, and their relative simplicity. Anything much more complex will start to get ugly.
I'd like to look at a slightly more involved reworking in order to nicely support optimisations like bitmap walking, without blowing out the complexity of the macros and without hiding too much of the workings.
However, my main aim for these macros was mainly to fix the performance regressions on 2 and 3 level architectures. Ben's complaints about these loops just served to hurry it along. I think that these reasons (performance, code consistency) make it a good idea.
Nick
Re: [PATCH 2/2] page table iterators
Quoted 3 lines · Nick Piggin
Haven't yet pulled out a pre-4-level kernel to see how 3-level compares I guess I'll do that now.
Close.
Before 4level: 119.5us, after folded walkers: 132.8us
I think most of this is now coming from clear_page_range, rather than the actual traversing of the page tables (because they should be completely folded by now):
before:
4089 total 0.0017 753 kmap_atomic 4.7358 682 do_wp_page 0.6713 680 do_page_fault 0.4561 261 zap_pte_range 0.3625 176 copy_page_range 0.2133 159 pte_alloc_one 1.5743 145 clear_page_tables 0.4866
after:
4307 total 0.0018 676 kmap_atomic 4.2516 665 do_page_fault 0.4472 615 do_wp_page 0.6225 550 clear_page_range 0.9982 262 zap_pte_range 0.4870
I think the additional work done by clear_page_range (versus clear_page_tables) justifies the extra cost, even for 3-level architectures.
Re: [PATCH 2/2] page table iterators
In reply to Hugh DickinsQuoted 11 lines · Hugh Dickins
Quoted 3 lines · Nick Piggin
Quoted 3 lines
Open coding is probably the smaller evil.
And they're really not changed that often.My opinion FWIW: I'm all for regularizing the pagetable loops to work the same way, changing their variables to use the same names, improving their efficiency; but I do like to see what a loop is up to.
Quoted 6 lines
list_for_eachand friends are very widely used, they're fine, and I'm quite glad to have their prefetching hidden away from me; but usually I groan, grin and bear it, each time someone devises a clever newfor_eachmacro concealing half the details of some specialist loop.In a minority?
OK, I think Andrew is now sitting on the fence after seeing the code. So you (and Andi?) are the ones with remaining reservations about this.
I don't disagree with your stance entirely, Hugh. I think these macros are close to being too complicated... But I don't think they is hiding too much detail: we all know that conceptually, walking a page table page is reasonably simple. There are just a few tricky bits like wrapping and termination that caused such a divergent range of implementations - I would argue that hiding these details is OK, because they are basically inconsequencial to the job at hand. I think that actually makes the high level intention of the code clearer, if anything.
If you are reading just the patch, that doesn't quite do it justice IMO - in that case, have a look at the code after the patch is applied (I can send you one which applies to current kernels if you'd like).
Also, the implementation of the macros is not insanely difficult to understand, so the details are still accessible.
Lastly, they fold to 2 and 3 levels easily, which is something that couldn't sanely be done with the open-coded implementation. I think with an infinitely smart compiler, there shouldn't need to be any folding here. But in practice I see quite a large speedup, which is something we shouldn't ignore.
I do think that they are probably not ideal candidates for a more general abstraction that would allow for example the transparent drop in of Dave's bitmap walking functions (it would be possible, but would not be pretty AFAIKS). I have some other ideas to work towards those goals, but before that I think these macros do help with the deficiencies of the current situation.
Nick
Re: [PATCH 2/2] page table iterators
Quoted 3 lines · Nick Piggin
Also, the implementation of the macros is not insanely difficult to understand, so the details are still accessible.
True, they're okay; but I do think they've got one argument too many, and I'd prefer to avoid that extra evaluation of the limit, which comes from your move from "do while ()" to "for (;;)".
Quoted 5 lines
Lastly, they fold to 2 and 3 levels easily, which is something that couldn't sanely be done with the open-coded implementation. I think with an infinitely smart compiler, there shouldn't need to be any folding here. But in practice I see quite a large speedup, which is something we shouldn't ignore.
Ben made the same point, yes, that is the real value of it. But I rather think the same can be done without for_each_* macros:
compiling with gcc 3.3.4 suggests my *_limit macros may result in slightly bigger code, but shorter codepath (no extra eval).
Quoted 7 lines
I do think that they are probably not ideal candidates for a more general abstraction that would allow for example the transparent drop in of Dave's bitmap walking functions (it would be possible, but would not be pretty AFAIKS). I have some other ideas to work towards those goals, but before that I think these macros do help with the deficiencies of the current situation.
As I said before, I am keen on regularizing the implementations and speeding them up; but not so keen on forcing into the straitjacket of for_each_* macros instead of opencoded loops. I've not seen Dave's bitmap walking functions (for clearing?), would they fit in better with my way?
I'm off to bed, but since your appetite for looking at patches is greater than mine, I'll throw what I'm currently testing over the wall to you now. Against 2.6.11-rc4-bk9, but my starting point was obviously your patches. Not yet split up, but clearly should be. Includes mm/swapfile.c which you missed. I'm inlining pmd and pud levels, but not pte and pgd levels. No description yet, sorry. One point worth making, I do believe throughout that whatever the address layout, "end" cannot be 0 - BUG_ON(addr >= end) assures.
Hugh
arch/i386/mm/ioremap.c +54 -55
@@ -17,86 +17,85 @@#include <asm/tlbflush.h> #include <asm/pgtable.h> -static inline void remap_area_pte (pte_t * pte,unsigned long address,unsigned long size, -unsigned long phys_addr,unsigned long flags) +static inline void remap_area_pte (pmd_t *pmd,unsigned long addr, +unsigned long end,unsigned long phys_addr,unsigned long flags) { -unsigned long end;unsigned long pfn; +pte_t *pte; - address &= ~PMD_MASK ; - end = address + size; -if (end >PMD_SIZE ) - end =PMD_SIZE ; -if (address >= end) -BUG (); pfn = phys_addr >>PAGE_SHIFT ; + pte =pte_offset_kernel (pmd, addr);do { -if (!pte_none (*pte)) { -printk ("remap_area_pte: page already exists\n" ); -BUG (); - } +BUG_ON (!pte_none (*pte));set_pte (pte,pfn_pte (pfn,__pgprot (_PAGE_PRESENT | _PAGE_RW | _PAGE_DIRTY | _PAGE_ACCESSED | flags))); - address +=PAGE_SIZE ; pfn++; - pte++; - }while (address && (address < end)); + }while (pte++, addr +=PAGE_SIZE , addr < end); } -static inline int remap_area_pmd (pmd_t * pmd,unsigned long address,unsigned long size, -unsigned long phys_addr,unsigned long flags) +static inline int remap_area_pmd (pud_t *pud,unsigned long addr, +unsigned long end,unsigned long phys_addr,unsigned long flags) { -unsigned long end; +unsigned long next; +pmd_t *pmd; - address &= ~PGDIR_MASK ; - end = address + size; -if (end >PGDIR_SIZE ) - end =PGDIR_SIZE ; - phys_addr -= address; -if (address >= end) -BUG (); + phys_addr -= addr; + pmd =pmd_offset (pud, addr);do { -pte_t * pte =pte_alloc_kernel (&init_mm, pmd, address); -if (!pte) + next =pmd_limit (addr, end); +if (!pte_alloc_kernel (&init_mm, pmd, addr))return -ENOMEM ; -remap_area_pte (pte, address, end - address, address + phys_addr, flags); - address = (address +PMD_SIZE ) &PMD_MASK ; - pmd++; - }while (address && (address < end)); +remap_area_pte (pmd, addr, next, phys_addr + addr, flags); + }while (pmd++, addr = next, addr < end);return 0 ; } -static int remap_area_pages (unsigned long address,unsigned long phys_addr, +static inline int remap_area_pud (pgd_t *pgd,unsigned long addr, +unsigned long end,unsigned long phys_addr,unsigned long flags) +{ +unsigned long next; +pud_t *pud; +int error; + + phys_addr -= addr; + pud =pud_offset (pgd, addr); +do { + next =pud_limit (addr, end); +if (pmd_alloc (&init_mm, pud, addr)) + error =remap_area_pmd (pud, addr, next, + phys_addr + addr, flags); +else + error = -ENOMEM ; +if (error) +break ; + }while (pud++, addr = next, addr < end); +return error; +} + +static int remap_area_pages (unsigned long addr,unsigned long phys_addr,unsigned long size,unsigned long flags) { +unsigned long end = addr + size; +unsigned long next; +pgd_t *pgd;int error; -pgd_t * dir; -unsigned long end = address + size; - phys_addr -= address; - dir =pgd_offset (&init_mm, address); +BUG_ON (addr >= end); +flush_cache_all (); -if (address >= end) -BUG (); + phys_addr -= addr; + pgd =pgd_offset_k (addr);spin_lock (&init_mm.page_table_lock);do { -pud_t *pud; -pmd_t *pmd; - - error = -ENOMEM ; - pud =pud_alloc (&init_mm, dir, address); -if (!pud) -break ; - pmd =pmd_alloc (&init_mm, pud, address); -if (!pmd) -break ; -if (remap_area_pmd (pmd, address, end - address, - phys_addr + address, flags)) + next =pgd_limit (addr, end); +if (pud_alloc (&init_mm, pgd, addr)) + error =remap_area_pud (pgd, addr, next, + phys_addr + addr, flags); +else + error = -ENOMEM ; +if (error)break ; - error =0 ; - address = (address +PGDIR_SIZE ) &PGDIR_MASK ; - dir++; - }while (address && (address < end)); + }while (pgd++, addr = next, addr < end);spin_unlock (&init_mm.page_table_lock);flush_tlb_all ();return error;
include/asm-generic/pgtable-nopmd.h +5 -0
@@ -5,6 +5,8 @@#include <asm-generic/pgtable-nopud.h> +#define __PAGETABLE_PMD_FOLDED +/* * Having the pmd type consist of a pud gets the size right, and allows * us to conceptually access the pud entry that this pmd is folded into
@@ -55,6 +57,9 @@ static inline pmd_t * pmd_offset(pud_t *#define pmd_free (x) do { } while (0)#define __pmd_free_tlb (tlb, x) do { } while (0) +#undef pmd_limit +#define pmd_limit (addr, end) (end) +#endif /* __ASSEMBLY__ */ #endif /* _PGTABLE_NOPMD_H */
include/asm-generic/pgtable-nopud.h +5 -0
@@ -3,6 +3,8 @@#ifndef __ASSEMBLY__ +#define __PAGETABLE_PUD_FOLDED +/* * Having the pud type consist of a pgd gets the size right, and allows * us to conceptually access the pgd entry that this pud is folded into
@@ -52,5 +54,8 @@ static inline pud_t * pud_offset(pgd_t *#define pud_free (x) do { } while (0)#define __pud_free_tlb (tlb, x) do { } while (0) +#undef pud_limit +#define pud_limit (addr, end) (end) +#endif /* __ASSEMBLY__ */ #endif /* _PGTABLE_NOPUD_H */
include/asm-generic/pgtable.h +19 -0
@@ -134,4 +134,23 @@ static inline void ptep_mkdirty(pte_t *p#define pgd_offset_gate (mm, addr) pgd_offset(mm, addr)#endif +#ifndef pmd_limit +#define pmd_limit (addr, end) \ +({ unsigned long __limit = ((addr) + PMD_SIZE) & PMD_MASK; \ + (__limit <= (end) && __limit)? __limit: (end); \ +}) +#endif + +#ifndef pud_limit +#define pud_limit (addr, end) \ +({ unsigned long __limit = ((addr) + PUD_SIZE) & PUD_MASK; \ + (__limit <= (end) && __limit)? __limit: (end); \ +}) +#endif + +#define pgd_limit (addr, end) \ +({ unsigned long __limit = ((addr) + PGDIR_SIZE) & PGDIR_MASK; \ + (__limit <= (end) && __limit)? __limit: (end); \ +}) +#endif /* _ASM_GENERIC_PGTABLE_H */
mm/memory.c +326 -410
1,030 lines not on this page · raw message
mm/mprotect.c +53 -79
164 lines not on this page · raw message
mm/msync.c +82 -127
261 lines not on this page · raw message
mm/swapfile.c +76 -111
268 lines not on this page · raw message
mm/vmalloc.c +100 -154
312 lines not on this page · raw message
Re: [PATCH 2/2] page table iterators
Quoted 2 lines · On Wed, 23 Feb 2005 02:06:28 +0000 (GMT) Hugh Dickins <[email protected]> wrote:
I've not seen Dave's bitmap walking functions (for clearing?), would they fit in better with my way?
This is what Nick is referring to:
--------------------
I hacked up something slightly different today. I only have it being used by clear_page_range() but it is extremely effective.
Things like fork+exit latencies on my 750Mhz sparc64 box went from ~490 microseconds to ~367 microseconds. fork+execve latency went down from ~1595 microseconds to ~1351 microseconds.
Two issues:
1) I'm not terribly satisfied with the interface. I think with some improvements it can be applies to the two other routines this thing really makes sense for, namely copy_page_range and unmap_page_range
2) I don't think it will collapse well for 2-level page tables, someone take a look?
It's easy to toy with the sparc64 optimization on other platforms, just add the necessary hacks to pmd_set and pgd_set, allocation of pmd and pgd tables, use "PAGE_SHIFT - 5" instead of "PAGE_SHIFT - 6" on 32-bit platforms, and then copy the asm-sparc64/pgwalk.h bits over into your platforms asm-${ARCH}/pgwalk.h
I just got also reminded that we walk these damn pagetables completely twice every exit, once to unmap the VMAs pte mappings, once again to zap the page tables. It might be fruitful to explore combining those two steps, perhaps not.
Anyways, comments and improvment suggestions welcome. Particularly interesting would be if this thing helps a lot on other platforms too, such as x86_64, ia64, alpha and ppc64.
# This is a BitKeeper generated diff -Nru style patch. #
# ChangeSet
# 2004/08/10 23:44:24-07:00 [email protected]
# [MM]: Add arch-overridable page table walking machinery.
#
# Currently very rudimentary but is used fully for # clear_page_range(). An optimized implementation # is there for sparc64 and it is extremely effective # particularly for 64-bit processes. # # For things like lat_fork and friends clear_page_tables() # use to be 2nd or 3rd in the kernel profile, now it has # dropped to the 20th or so entry. # # Signed-off-by: David S. Miller <[email protected]> # # mm/memory.c # 2004/08/10 23:42:42-07:00 [email protected] +10 -26 # [MM]: Add arch-overridable page table walking machinery. # # include/asm-sparc64/pgtable.h # 2004/08/10 23:42:42-07:00 [email protected] +28 -4 # [MM]: Add arch-overridable page table walking machinery. # # include/asm-sparc64/pgalloc.h # 2004/08/10 23:42:42-07:00 [email protected] +10 -2 # [MM]: Add arch-overridable page table walking machinery. # # arch/sparc64/mm/init.c # 2004/08/10 23:42:42-07:00 [email protected] +2 -2 # [MM]: Add arch-overridable page table walking machinery. # # include/asm-x86_64/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +6 -0 # [MM]: Add arch-overridable page table walking machinery. # # include/asm-v850/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +6 -0 # [MM]: Add arch-overridable page table walking machinery. # # include/asm-um/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +6 -0 # [MM]: Add arch-overridable page table walking machinery. # # include/asm-sparc64/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +114 -0 # [MM]: Add arch-overridable page table walking machinery. # # include/asm-sparc/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +6 -0 # [MM]: Add arch-overridable page table walking machinery. # # include/asm-sh64/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +6 -0 # [MM]: Add arch-overridable page table walking machinery. # # include/asm-sh/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +6 -0 # [MM]: Add arch-overridable page table walking machinery. # # include/asm-s390/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +6 -0 # [MM]: Add arch-overridable page table walking machinery. # # include/asm-ppc64/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +6 -0 # [MM]: Add arch-overridable page table walking machinery. # # include/asm-ppc/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +6 -0 # [MM]: Add arch-overridable page table walking machinery. # # include/asm-parisc/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +6 -0 # [MM]: Add arch-overridable page table walking machinery. # # include/asm-mips/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +6 -0 # [MM]: Add arch-overridable page table walking machinery. # # include/asm-m68knommu/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +6 -0 # [MM]: Add arch-overridable page table walking machinery. # # include/asm-m68k/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +6 -0 # [MM]: Add arch-overridable page table walking machinery. # # include/asm-ia64/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +6 -0 # [MM]: Add arch-overridable page table walking machinery. # # include/asm-i386/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +6 -0 # [MM]: Add arch-overridable page table walking machinery. # # include/asm-h8300/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +6 -0 # [MM]: Add arch-overridable page table walking machinery. # # include/asm-generic/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +96 -0 # [MM]: Add arch-overridable page table walking machinery. # # include/asm-cris/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +6 -0 # [MM]: Add arch-overridable page table walking machinery. # # include/asm-arm26/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +6 -0 # [MM]: Add arch-overridable page table walking machinery. # # include/asm-arm/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +6 -0 # [MM]: Add arch-overridable page table walking machinery. # # include/asm-x86_64/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +0 -0 # BitKeeper file /disk1/BK/sparc-2.6/include/asm-x86_64/pgwalk.h # # include/asm-v850/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +0 -0 # BitKeeper file /disk1/BK/sparc-2.6/include/asm-v850/pgwalk.h # # include/asm-um/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +0 -0 # BitKeeper file /disk1/BK/sparc-2.6/include/asm-um/pgwalk.h # # include/asm-sparc64/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +0 -0 # BitKeeper file /disk1/BK/sparc-2.6/include/asm-sparc64/pgwalk.h # # include/asm-sparc/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +0 -0 # BitKeeper file /disk1/BK/sparc-2.6/include/asm-sparc/pgwalk.h # # include/asm-sh64/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +0 -0 # BitKeeper file /disk1/BK/sparc-2.6/include/asm-sh64/pgwalk.h # # include/asm-sh/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +0 -0 # BitKeeper file /disk1/BK/sparc-2.6/include/asm-sh/pgwalk.h # # include/asm-s390/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +0 -0 # BitKeeper file /disk1/BK/sparc-2.6/include/asm-s390/pgwalk.h # # include/asm-ppc64/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +0 -0 # BitKeeper file /disk1/BK/sparc-2.6/include/asm-ppc64/pgwalk.h # # include/asm-ppc/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +0 -0 # BitKeeper file /disk1/BK/sparc-2.6/include/asm-ppc/pgwalk.h # # include/asm-parisc/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +0 -0 # BitKeeper file /disk1/BK/sparc-2.6/include/asm-parisc/pgwalk.h # # include/asm-mips/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +0 -0 # BitKeeper file /disk1/BK/sparc-2.6/include/asm-mips/pgwalk.h # # include/asm-m68knommu/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +0 -0 # BitKeeper file /disk1/BK/sparc-2.6/include/asm-m68knommu/pgwalk.h # # include/asm-m68k/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +0 -0 # BitKeeper file /disk1/BK/sparc-2.6/include/asm-m68k/pgwalk.h # # include/asm-ia64/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +0 -0 # BitKeeper file /disk1/BK/sparc-2.6/include/asm-ia64/pgwalk.h # # include/asm-i386/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +0 -0 # BitKeeper file /disk1/BK/sparc-2.6/include/asm-i386/pgwalk.h # # include/asm-h8300/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +0 -0 # BitKeeper file /disk1/BK/sparc-2.6/include/asm-h8300/pgwalk.h # # include/asm-generic/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +0 -0 # BitKeeper file /disk1/BK/sparc-2.6/include/asm-generic/pgwalk.h # # include/asm-cris/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +0 -0 # BitKeeper file /disk1/BK/sparc-2.6/include/asm-cris/pgwalk.h # # include/asm-arm26/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +0 -0 # BitKeeper file /disk1/BK/sparc-2.6/include/asm-arm26/pgwalk.h # # include/asm-arm/pgwalk.h # 2004/08/10 23:42:14-07:00 [email protected] +0 -0 # BitKeeper file /disk1/BK/sparc-2.6/include/asm-arm/pgwalk.h # # include/asm-alpha/pgwalk.h # 2004/08/10 23:42:13-07:00 [email protected] +6 -0 # [MM]: Add arch-overridable page table walking machinery. # # include/asm-alpha/pgwalk.h # 2004/08/10 23:42:13-07:00 [email protected] +0 -0 # BitKeeper file /disk1/BK/sparc-2.6/include/asm-alpha/pgwalk.h # diff -Nru a/arch/sparc64/mm/init.c b/arch/sparc64/mm/init.c --- a/arch/sparc64/mm/init.c 2004-08-10 23:44:47 -07:00 +++ b/arch/sparc64/mm/init.c 2004-08-10 23:44:47 -07:00 @@ -419,7 +419,7 @@ if (ptep == NULL) early_pgtable_allocfail("pte"); memset(ptep, 0, BASE_PAGE_SIZE); - pmd_set(pmdp, ptep); + pmd_set_k(pmdp, ptep); } ptep = (pte_t *)__pmd_page(*pmdp) + ((vaddr >> 13) & 0x3ff); @@ -1455,7 +1455,7 @@ memset(swapper_pmd_dir, 0, sizeof(swapper_pmd_dir));
/* Now can init the kernel/bad page tables. */
- pgd_set(&swapper_pg_dir[0], swapper_pmd_dir + (shift / sizeof(pgd_t)));
+ pgd_set_k(&swapper_pg_dir[0], swapper_pmd_dir + (shift / sizeof(pgd_t)));
sparc64_vpte_patchme1[0] |=
(((unsigned long)pgd_val(init_mm.pgd[0])) >> 10);
include/asm-alpha/pgwalk.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef _ALPHA_PGWALK_H +#define _ALPHA_PGWALK_H + +#include <asm-generic/pgwalk.h> + +#endif /* _ALPHA_PGWALK_H */
include/asm-arm/pgwalk.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef _ARM_PGWALK_H +#define _ARM_PGWALK_H + +#include <asm-generic/pgwalk.h> + +#endif /* _ARM_PGWALK_H */
include/asm-arm26/pgwalk.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef _ARM26_PGWALK_H +#define _ARM26_PGWALK_H + +#include <asm-generic/pgwalk.h> + +#endif /* _ARM26_PGWALK_H */
include/asm-cris/pgwalk.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef _CRIS_PGWALK_H +#define _CRIS_PGWALK_H + +#include <asm-generic/pgwalk.h> + +#endif /* _CRIS_PGWALK_H */
include/asm-generic/pgwalk.h new file +96 -0
96 lines not on this page · raw message
include/asm-h8300/pgwalk.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef _H8300_PGWALK_H +#define _H8300_PGWALK_H + +#include <asm-generic/pgwalk.h> + +#endif /* _H8300_PGWALK_H */
include/asm-i386/pgwalk.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef _I386_PGWALK_H +#define _I386_PGWALK_H + +#include <asm-generic/pgwalk.h> + +#endif /* _I386_PGWALK_H */
include/asm-ia64/pgwalk.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef _IA64_PGWALK_H +#define _IA64_PGWALK_H + +#include <asm-generic/pgwalk.h> + +#endif /* _IA64_PGWALK_H */
include/asm-m68k/pgwalk.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef _M68K_PGWALK_H +#define _M68K_PGWALK_H + +#include <asm-generic/pgwalk.h> + +#endif /* _M68K_PGWALK_H */
include/asm-m68knommu/pgwalk.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef _M68KNOMMU_PGWALK_H +#define _M68KNOMMU_PGWALK_H + +#include <asm-generic/pgwalk.h> + +#endif /* _M68KNOMMU_PGWALK_H */
include/asm-mips/pgwalk.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef _ALPHA_PGWALK_H +#define _ALPHA_PGWALK_H + +#include <asm-generic/pgwalk.h> + +#endif /* _ALPHA_PGWALK_H */
include/asm-parisc/pgwalk.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef _PARISC_PGWALK_H +#define _PARISC_PGWALK_H + +#include <asm-generic/pgwalk.h> + +#endif /* _PARISC_PGWALK_H */
include/asm-ppc/pgwalk.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef _PPC_PGWALK_H +#define _PPC_PGWALK_H + +#include <asm-generic/pgwalk.h> + +#endif /* _PPC_PGWALK_H */
include/asm-ppc64/pgwalk.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef _PPC64_PGWALK_H +#define _PPC64_PGWALK_H + +#include <asm-generic/pgwalk.h> + +#endif /* _PPC64_PGWALK_H */
include/asm-s390/pgwalk.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef _S390_PGWALK_H +#define _S390_PGWALK_H + +#include <asm-generic/pgwalk.h> + +#endif /* _S390_PGWALK_H */
include/asm-sh/pgwalk.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef _SH_PGWALK_H +#define _SH_PGWALK_H + +#include <asm-generic/pgwalk.h> + +#endif /* _SH_PGWALK_H */
include/asm-sh64/pgwalk.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef _SH64_PGWALK_H +#define _SH64_PGWALK_H + +#include <asm-generic/pgwalk.h> + +#endif /* _SH64_PGWALK_H */
include/asm-sparc/pgwalk.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef _SPARC_PGWALK_H +#define _SPARC_PGWALK_H + +#include <asm-generic/pgwalk.h> + +#endif /* _SPARC_PGWALK_H */
include/asm-sparc64/pgalloc.h +10 -2
@@ -93,6 +93,8 @@static __inline__void free_pgd_fast (pgd_t *pgd) { +virt_to_page (pgd)->index =0UL ; +preempt_disable (); *(unsigned long *)pgd = (unsigned long ) pgd_quicklist; pgd_quicklist = (unsigned long *) pgd;
@@ -113,8 +115,10 @@ }else {preempt_enable (); ret = (unsigned long *)__get_free_page (GFP_KERNEL |__GFP_REPEAT); -if (ret) +if (ret) {memset (ret,0 ,PAGE_SIZE ); +virt_to_page (ret)->index =0UL ; + } }return (pgd_t *)ret; }
@@ -162,8 +166,10 @@ pmd =pmd_alloc_one_fast (mm, address);if (!pmd) { pmd = (pmd_t *)__get_free_page (GFP_KERNEL |__GFP_REPEAT); -if (pmd) +if (pmd) {memset (pmd,0 ,PAGE_SIZE ); +virt_to_page (pmd)->index =0UL ; + } }return pmd; }
@@ -171,6 +177,8 @@static __inline__void free_pmd_fast (pmd_t *pmd) {unsigned long color =DCACHE_COLOR ((unsigned long )pmd); + +virt_to_page (pmd)->index =0UL ;preempt_disable (); *(unsigned long *)pmd = (unsigned long ) pte_quicklist[color];
include/asm-sparc64/pgtable.h +28 -4
@@ -259,10 +259,34 @@return __pte; } -#define pmd_set (pmdp, ptep) \ - (pmd_val(*(pmdp)) = (__pa((unsigned long) (ptep)) >> 11UL)) -#define pgd_set (pgdp, pmdp) \ - (pgd_val(*(pgdp)) = (__pa((unsigned long) (pmdp)) >> 11UL)) + +#define PGTABLE_BIT_SHIFT (PAGE_SHIFT - 6) +#define PGTABLE_BIT_MASK ((1UL << PGTABLE_BIT_SHIFT) - 1) +#define PGTABLE_BIT_REGION (1UL << PGTABLE_BIT_SHIFT) +#define PGTABLE_BIT (ptr) \ + (1UL << (((unsigned long)(ptr) & ~PAGE_MASK) >> PGTABLE_BIT_SHIFT)) +#define __PGTABLE_REGION_NEXT (ptr,type) \ + ((type *)(((unsigned long)(ptr) + PGTABLE_BIT_REGION) & \ + ~PGTABLE_BIT_MASK)) +#define PMD_REGION_NEXT (pmdp) __PGTABLE_REGION_NEXT(pmdp,pmd_t) +#define PGD_REGION_NEXT (pgdp) __PGTABLE_REGION_NEXT(pgdp,pgd_t) + +#define pmd_set (pmdp, ptep) \ +do { \ + virt_to_page(pmdp)->index |= PGTABLE_BIT(pmdp); \ + pmd_val(*pmdp) = __pa((unsigned long) (ptep)) >> 11UL; \ +} while (0) +#define pmd_set_k (pmdp, ptep) \ + (pmd_val(*pmdp) = __pa((unsigned long) (ptep)) >> 11UL) + +#define pgd_set (pgdp, pmdp) \ +do { \ + virt_to_page(pgdp)->index |= PGTABLE_BIT(pgdp); \ + pgd_val(*pgdp) = __pa((unsigned long) (pmdp)) >> 11UL; \ +} while (0) +#define pgd_set_k (pgdp, pmdp) \ + (pgd_val(*pgdp) = __pa((unsigned long) (pmdp)) >> 11UL) +#define __pmd_page (pmd) \ ((unsigned long) __va((((unsigned long)pmd_val(pmd))<<11UL)))#define pmd_page (pmd) virt_to_page((void *)__pmd_page(pmd))
include/asm-sparc64/pgwalk.h new file +114 -0
114 lines not on this page · raw message
include/asm-um/pgwalk.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef _UM_PGWALK_H +#define _UM_PGWALK_H + +#include <asm-generic/pgwalk.h> + +#endif /* _UM_PGWALK_H */
include/asm-v850/pgwalk.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef _V850_PGWALK_H +#define _V850_PGWALK_H + +#include <asm-generic/pgwalk.h> + +#endif /* _V850_PGWALK_H */
include/asm-x86_64/pgwalk.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef _X86_64_PGWALK_H +#define _X86_64_PGWALK_H + +#include <asm-generic/pgwalk.h> + +#endif /* _X86_64_PGWALK_H */
mm/memory.c +10 -26
@@ -52,6 +52,7 @@#include <asm/tlb.h> #include <asm/tlbflush.h> #include <asm/pgtable.h> +#include <asm/pgwalk.h> #include <linux/swapops.h> #include <linux/elf.h>
@@ -100,40 +101,25 @@* Note: this doesn't free the actual pages themselves. That * has been handled earlier when unmapping all the memory regions. */ -static inline void free_one_pmd (struct mmu_gather *tlb,pmd_t * dir) +static void free_one_pmd (struct pte_walk_state *walk,pmd_t *dir) {struct page *page; -if (pmd_none (*dir)) -return ; -if (unlikely (pmd_bad (*dir))) { -pmd_ERROR (*dir); -pmd_clear (dir); -return ; - } page =pmd_page (*dir);pmd_clear (dir);dec_page_state (nr_page_table_pages); -pte_free_tlb (tlb, page); +pte_free_tlb (pte_walk_client_state (walk), page); } -static inline void free_one_pgd (struct mmu_gather *tlb,pgd_t * dir) +static void free_one_pgd (struct pte_walk_state *walk,pgd_t *dir) { -int j;pmd_t * pmd; -if (pgd_none (*dir)) -return ; -if (unlikely (pgd_bad (*dir))) { -pgd_ERROR (*dir); -pgd_clear (dir); -return ; - } pmd =pmd_offset (dir,0 );pgd_clear (dir); -for (j =0 ; j <PTRS_PER_PMD ; j++) -free_one_pmd (tlb, pmd+j); -pmd_free_tlb (tlb, pmd); +pmd_walk_init (walk, pmd, pmd +PTRS_PER_PMD ); +pmd_walk (walk, free_one_pmd); +pmd_free_tlb (pte_walk_client_state (tlb), pmd); }/*
@@ -144,13 +130,11 @@*/ void clear_page_tables (struct mmu_gather *tlb,unsigned long first,int nr) { +struct pte_walk_state walk;pgd_t * page_dir = tlb->mm->pgd; - page_dir += first; -do { -free_one_pgd (tlb, page_dir); - page_dir++; - }while (--nr); +pgd_walk_init (&walk, tlb, page_dir, page_dir + nr); +pgd_walk (&walk, free_one_pgd); } pte_t fastcall * pte_alloc_map(struct mm_struct *mm, pmd_t *pmd, unsigned long address)
Re: [PATCH 2/2] page table iterators
Quoted 6 lines · David S. Miller
Quoted 2 lines · On Wed, 23 Feb 2005 02:06:28 +0000 (GMT) Hugh Dickins <[email protected]> wrote:
I've not seen Dave's bitmap walking functions (for clearing?), would they fit in better with my way?
Hugh: I'll have more of a look through your patch when I get some time... to be honest I'm not too worried either way, so long as one or the other gets in.
Very trivial point, but I'm not sure that I like the name p?d_limit... maybe p?d_span or _span_end... hmm, they're not really pleasing either.
You _are_ repeating a bit of mindless loop accounting in every page table walk, and it isn't completely clear to me that it is giving you much more flexibility (than for_each_*). But my loops _are_ a bit contorted.
Quoted 2 lines
This is what Nick is referring to:
[snip]
Quoted 3 lines
It's easy to toy with the sparc64 optimization on other platforms, just add the necessary hacks to
pmd_setandpgd_set, allocation of pmd and pgd tables
David: just an implementation detail that I had meant to bring up earlier - would it feel like less of a hack to put these in pmd_populate and pgd_populate?
Nick
Re: [PATCH 2/2] page table iterators
Quoted 7 lines · On Wed, 23 Feb 2005 15:49:30 +1100 Nick Piggin <[email protected]> wrote:
Quoted 3 lines
It's easy to toy with the sparc64 optimization on other platforms, just add the necessary hacks to
pmd_setandpgd_set, allocation of pmd and pgd tablesDavid: just an implementation detail that I had meant to bring up earlier - would it feel like less of a hack to put these in
pmd_populateandpgd_populate?
Sure, no problem. They get defined to pmd_set/pgd_set calls anyways. But wouldn't that miss pgd_clear() and pmd_clear()? Someone may find it worthwhile to, on a *_clear(), to see if a set bit can now be clear because all the neighboring entries are empty as well.
That might have been the reason I put it there, but I may be giving myself too much credit :-)
Re: [PATCH 2/2] page table iterators
In reply to David S. MillerQuoted 5 lines · David S. Miller
I just got also reminded that we walk these damn pagetables completely twice every exit, once to unmap the VMAs pte mappings, once again to zap the page tables. It might be fruitful to explore combining those two steps, perhaps not.
I'm going to have a look at refcounting page table pages, which will hopefully allow us to get back (and more) the clear_page_range overhead introduced by the aggressive page table freeing.
It may also allow nice things like dropping file backed page table mappings if they get reclaimed, and also a single walk to do the freeing. I haven't looked into details yet though, these are just vague hopes.
Quoted 4 lines
Anyways, comments and improvment suggestions welcome. Particularly interesting would be if this thing helps a lot on other platforms too, such as
x86_64, ia64, alpha and ppc64.
I have a feeling it should provide nice benefits to all archs if we get it into all the walkers. Downsides are few - the bitmap walk probably only becomes more expensive when all but a handful of cachelines are present in a page table page.
I'd like to look at ways to make this patch happen with you soon... First, for 2.6.12 my main concern is to get pt walking consistent, and try to claw back some of the clear_page_range regression.
Thanks,
Nick
Find local movie times and trailers on Yahoo! Movies.
http://au.movies.yahoo.com
Re: [PATCH 2/2] page table iterators
In reply to Hugh DickinsQuoted 4 lines · Hugh Dickins
I'm off to bed, but since your appetite for looking at patches is greater than mine, I'll throw what I'm currently testing over the wall to you now. Against 2.6.11-rc4-bk9, but my starting point was obviously your patches. Not yet split up, but clearly should be.
Yeah you've snuck a few other clever things in there ;)
Quoted 1 line
Includes
mm/swapfile.cwhich you missed. I'm inlining pmd and pud
Thanks.
Quoted 1 line
levels, but not pte and pgd levels. No description yet, sorry.
OK - that's probably sufficient for debugging. There is only so much that can go wrong in the middle levels... how does it look performance wise? (I can give it a test when it gets split out)
Quoted 3 lines
One point worth making, I do believe throughout that whatever the address layout, "end" cannot be 0 -
BUG_ON(addr >= end)assures.
OK after sleeping on it, I'm warming to your way.
I don't think it makes something like David's modifications any easier, but mine didn't go a long way to that end either. And being a more incremental approach gives us more room to move in future (for example, maybe toward something that really *will* accommodate the bitmap walking code nicely).
So I'd be pretty happy for you to queue this up with Andrew for 2.6.12. Anyone else?
Nick
Re: [PATCH 2/2] page table iterators
Quoted 2 lines · On Thu, 24 Feb 2005 10:52:23 +1100 Nick Piggin <[email protected]> wrote:
So I'd be pretty happy for you to queue this up with Andrew for 2.6.12. Anyone else?
No objections from me.
Re: [PATCH 2/2] page table iterators
In reply to Nick PigginQuoted 6 lines · Nick Piggin
Quoted 1 line · Hugh Dickins
I'm inlining pmd and pud levels, but not pte and pgd levels.
OK - that's probably sufficient for debugging. There is only so much that can go wrong in the middle levels...
Yes, that was my thinking.
Quoted 2 lines
how does it look
performance wise? (I can give it a test when it gets split out)
Yesterday shattered in various directions, I hope to try today.
Quoted 2 lines
Quoted 2 lines
One point worth making, I do believe throughout that whatever the address layout, "end" cannot be 0 -
BUG_ON(addr >= end)assures.
Of course, that does allow some simplifications in your for_each macros; but it still looked like my p??_limits were better for shortest codepath, and close to yours for codesize.
Quoted 7 lines
OK after sleeping on it, I'm warming to your way.
I don't think it makes something like David's modifications any easier, but mine didn't go a long way to that end either. And being a more incremental approach gives us more room to move in future (for example, maybe toward something that really *will* accommodate the bitmap walking code nicely).
I'll take a quick look at David's today.
Just so long as we don't make them harder.
Quoted 2 lines
So I'd be pretty happy for you to queue this up with Andrew for 2.6.12. Anyone else?
Oh, okay, thanks. You weren't very happy with p??_limit(addr, end), and good naming is important to me. I didn't care for your tentative p??_span or p??_span_end. Would p??_end be better? p??_enda would be fun for one of them...
Hugh
Re: [PATCH 2/2] page table iterators
Quoted 4 lines · Hugh Dickins
Quoted 2 lines · Nick Piggin
OK after sleeping on it, I'm warming to your way.
Quoted 9 lines
Quoted 5 lines
I don't think it makes something like David's modifications any easier, but mine didn't go a long way to that end either. And being a more incremental approach gives us more room to move in future (for example, maybe toward something that really *will* accommodate the bitmap walking code nicely).
I'll take a quick look at David's today.
Just so long as we don't make them harder.
No, I think we may want to move to something better abstracted:
it makes things sufficiently complex that you wouldn't want to have it open coded everywhere.
But no, you're not making it harder than the present situation.
Quoted 8 lines
Quoted 2 lines
So I'd be pretty happy for you to queue this up with Andrew for 2.6.12. Anyone else?
Oh, okay, thanks. You weren't very happy with p??_limit(addr, end), and good naming is important to me. I didn't care for your tentative p??_span or p??_span_end. Would p??_end be better? p??_enda would be fun for one of them...
pud_addr_end?
http://mobile.yahoo.com.au - Yahoo! Mobile
- Check & compose your email via SMS on your Telstra or Vodafone mobile.
Re: [PATCH 2/2] page table iterators
Quoted 2 lines · Nick Piggin
pud_addr_end?
next = pud_addr_end(addr, end);
Hmm, yes, I'll go with that, thanks (unless a better idea follows).
Something I do intend on top of what I sent before, is another set of three macros, like
if (pud_none_or_clear_bad(pud)) continue;
to replace all the p??_none, p??_bad clauses: not to save space, but just for clarity, those loops now seeming dominated by the unlikeliest of cases.
Has anyone _ever_ seen a p??_ERROR message? I'm inclined to just put three functions into mm/memory.c to do the p??_ERROR and p??_clear, but that way the __FILE__ and __LINE__ will always come out the same. I think if it ever proves a problem, we'd just add in a dump_stack.
Hugh
Re: [PATCH 2/2] page table iterators
Quoted 1 line · On Thu, 24 Feb 2005 11:58:42 +0000 (GMT) Hugh Dickins <[email protected]> wrote:
Has anyone _ever_ seen a p??_ERROR message?
It triggers when you're writing new platform pagetable support or making drastric changes in same. But on sparc64 I've set them all to nops to make the code output smaller. :-)
Re: [PATCH 2/2] page table iterators
In reply to Hugh DickinsQuoted 15 lines · Hugh Dickins
Quoted 1 line · Nick Piggin
pud_addr_end?next = pud_addr_end(addr, end);Hmm, yes, I'll go with that, thanks (unless a better idea follows).
Something I do intend on top of what I sent before, is another set of three macros, like
if (pud_none_or_clear_bad(pud)) continue;
Quoted 9 lines
to replace all the p??_none, p??_bad clauses: not to save space, but just for clarity, those loops now seeming dominated by the unlikeliest of cases.
Has anyone _ever_ seen a p??_ERROR message? I'm inclined to just put three functions into
mm/memory.cto do the p??_ERROR and p??_clear, but that way the__FILE__and__LINE__will always come out the same. I think if it ever proves a problem, we'd just add in adump_stack.
I think a function is the most sensible. And a good idea, it should reduce the icache pressure in the loops (although gcc does seem to do a pretty good job of moving unlikely()s away from the fastpath).
I think at the point these things get detected, there is little use for having a dump_stack. But we may as well add one anyway if it is an out of line function?
Nick
Re: [PATCH 2/2] page table iterators
Quoted 10 lines · Nick Piggin
Quoted 5 lines · Hugh Dickins
Has anyone _ever_ seen a p??_ERROR message? I'm inclined to just put three functions into
mm/memory.cto do the p??_ERROR and p??_clear, but that way the__FILE__and__LINE__will always come out the same. I think if it ever proves a problem, we'd just add in adump_stack.I think a function is the most sensible. And a good idea, it should reduce the icache pressure in the loops (although gcc does seem to do a pretty good job of moving unlikely()s away from the fastpath).
At one stage I was adding unlikelies to all the p??_bads, then it seemed more sensible to hide that in a new macro (which of course must do the none and bad tests inline, before going off to the function).
David's response confirms that __FILE__,__LINE__ shouldn't be an issue.
Quoted 3 lines
I think at the point these things get detected, there is little use for having a
dump_stack. But we may as well add one anyway if it is an out of line function?
We could at little cost. But I think if these messages come up at all, they're likely to come up in clumps, where the backtrace won't actually be giving any interesting info, and the quantity of them be a nuisance itself. I'd rather leave it to the next person who gets the error and wants the backtrace to add it.
Hugh
Re: [PATCH 2/2] page table iterators
Quoted 5 lines · Hugh Dickins
At one stage I was adding unlikelies to all the p??_bads, then it seemed more sensible to hide that in a new macro (which of course must do the none and bad tests inline, before going off to the function).
Yeah that sounds OK. I think (un)likely can propagate through inline functions too, if that's any help to you.
Quoted 7 lines
We could at little cost. But I think if these messages come up at all, they're likely to come up in clumps, where the backtrace won't actually be giving any interesting info, and the quantity of them be a nuisance itself. I'd rather leave it to the next person who gets the error and wants the backtrace to add it.
You're probably right - I know when I see them (from my hacking up the code) they usually come in clumps :P
Nick
Re: [PATCH 2/2] page table iterators
In reply to David S. MillerQuoted 8 lines · David S. Miller
Quoted 1 line · On Thu, 24 Feb 2005 11:58:42 +0000 (GMT) Hugh Dickins <[email protected]> wrote:
Has anyone _ever_ seen a p??_ERROR message?
It triggers when you're writing new platform pagetable support or making drastric changes in same. But on sparc64 I've set them all to nops to make the code output smaller. :-)
I don't think it's useful except for early debugging.
Also at least on i386/x86-64 the CPU sets a bit in the page fault handler when it encounters a corrupted page table. On x86-64 it is handled (not on i386)
-Andi