10.31 second kernel compile
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 3 lines · [email protected] writes:
is there a 64 bit machine with hardware search of pagetables? Even ibm only has a hardware search of hash tables - which we agree are simply a means of making your hardware TLB larger and slower.
x86-64 aka AMD Hammer does hardware (or more likely microcode) search of page tables.
It has a 4 level page table with 4K pages. Generic Linux MM code only sees the first slot in 4th level page limit user space to 512GB with 3 levels. Direct mappings and kernel mappings are handled specially by architecture specific code outside that first slot.
The CPU itself has I/D TLBs split into L1 and L2.
-Andi
Quoted 4 lines · Andi Kleen
x86-64 aka AMD Hammer does hardware (or more likely microcode) search of page tables.
It has a 4 level page table with 4K pages. Generic Linux MM code only sees the first slot in 4th level page limit user space to 512GB with 3 levels.
What about 2M pages?
Quoted 4 lines
Direct mappings and kernel mappings are handled specially by architecture specific code outside that first slot.
The CPU itself has I/D TLBs split into L1 and L2.
There was something in some AMD doc about preventing tlbflush on process switch - through a context like thing perhaps? Any idea?
Quoted 2 lines
-Andi
Signature · 4 lines
--------------------------------------------------------- Victor Yodaiken Finite State Machine Labs: The RTLinux Company. www.fsmlabs.com www.rtlinux.com
Quoted 2 lines · On Sat, 16 Mar 2002 [email protected] wrote:
What about 2M pages?
Not useful for generic loads right now, and the latencies for clearing or copying them them etc (ie single page faults - nopage or COW) are still big enough that it would likely be a performance problem at that level. And while doing IO in 2MB chunks sounds like fun, since most files are still just a few kB, your page cache memory overhead would be prohibitive (ie even if you had 64GB of memory, you might want to cache more than a few thousand files at the same time).
So then you'd need to do page caching at a finer granularity than you do mmap, which imples absolutely horrible things from a coherency standpoint (mmap/read/write are supposed to be coherent in a nice UNIX - even if there are some of non-nice unixes still around).
We may get there some day, but right now 2M pages are not usable for use access.
64kB would be fine, though.
Oh, and in the specific case of hammer, one of the main advantages of the thing is of course running old binaries unchanged. And old binaries certainly do mmap's at smaller granularity than 2M (and have to, because a 3G user address space won't fit all that many 2M chunks).
Give up on large pages - it's just not happening. Even when a 64kB page would make sense from a technology standpoint these days, backwards compatibility makes people stay at 4kB.
Instead of large pages, you should be asking for larger and wider TLB's (for example, nothign says that a TLB entry has to be a single page:
people already do the kind of "super-entries", where one TLB entry actually contains data for 4 or 8 aligned pages, so you get the _effect_ of a 32kB page that really is 8 consecutive 4kB pages).
Such a "wide" TLB entry has all the advantages of small pages (no memory fragmentation, backwards compatibility etc), while still being able to load 64kB worth of translations in one go.
(One of the advantages of a page table tree over a hashed setup becomes clear in this kind of situation: you cannot usefully load multiple entries from the same cacheline into one TLB entry in a hashed table, while in a tree it's truly trivial)
Linus
Quoted 9 lines · Linus Torvalds
Instead of large pages, you should be asking for larger and wider TLB's (for example, nothign says that a TLB entry has to be a single page:
people already do the kind of "super-entries", where one TLB entry actually contains data for 4 or 8 aligned pages, so you get the _effect_ of a 32kB page that really is 8 consecutive 4kB pages).Such a "wide" TLB entry has all the advantages of small pages (no memory fragmentation, backwards compatibility etc), while still being able to load 64kB worth of translations in one go.
These are contiguous physical pages, or just logical (virtual) pages?
Regards, Richard.... Permanent: [email protected] Current: [email protected]
Quoted 2 lines · Richard Gooch
These are contiguous physical pages, or just logical (virtual) pages?
Contiguous virtual pages, but discontiguous physical pages.
The advantage being that you only need one set of virtual tags per "wide" entry, and you just fill the whole wide entry directly from the cacheline (ie the TLB entry is not really 32 bits any more, it's a full cacheline).
The _real_ advantage being that it should be totally invisible to software. I think Intel does something like this, but the point is, I don't even have to know, and it still works.
Linus
Quoted 6 lines · Linus Torvalds
Quoted 2 lines · Richard Gooch
These are contiguous physical pages, or just logical (virtual) pages?
Contiguous virtual pages, but discontiguous physical pages.
That's what I was hoping. Having both contiguous would be of some benefit, but of course at the cost of having to unfragment physical pages. Even if Andi can cleanly do that with rmap, it's still going to cost (page copies, Dcache footprint, locking and more). I like the "wide" TLB approach much more.
Quoted 8 lines
The advantage being that you only need one set of virtual tags per "wide" entry, and you just fill the whole wide entry directly from the cacheline (ie the TLB entry is not really 32 bits any more, it's a full cacheline).
The _real_ advantage being that it should be totally invisible to software. I think Intel does something like this, but the point is, I don't even have to know, and it still works.
Completely behind the kernel's, back? Even so, is there some hint we can give to the CPU to help? Or perhaps a hint an application can give to the kernel to specify better alignment of mappings? The latter would require a way for the kernel to find out the preferred alignment from the CPU. Is this information available?
Anyone know if AMD does this as well?
Regards, Richard.... Permanent: [email protected] Current: [email protected]
Quoted 9 lines · Linus Torvalds
Quoted 2 lines · On Sat, 16 Mar 2002 [email protected] wrote:
What about 2M pages?
Not useful for generic loads right now, and the latencies for clearing or copying them them etc (ie single page faults - nopage or COW) are still big enough that it would likely be a performance problem at that level. And while doing IO in 2MB chunks sounds like fun, since most files are still just a few kB,
In other words, large pages should be a "special hack" for special applications, like Oracle and maybe some scientific calculations ?
Grabbing some bitflags in generic datastructures shouldn't be an issue since free bits are available.
regards,
Rik
Signature · 3 lines
<insert bitkeeper endorsement here> http://www.surriel.com/ http://distro.conectiva.com/
In article <[email protected]>,
Quoted 4 lines · Rik van Riel
In other words, large pages should be a "special hack" for special applications, like Oracle and maybe some scientific calculations ?
Yes, I think so.
That said, a 64kB page would be useful for generic use.
Quoted 2 lines
Grabbing some bitflags in generic datastructures shouldn't be an issue since free bits are available.
I had large-page-support working in the VM a long time ago, back when I did the original VM portability rewrite. I actually exposed the kernel large pages to the VM, and it worked fine - I didn't even need a new bit, since the code just used the "large page" bit in the page table directly.
But it wasn't ever exposed to user space, and in the end I just made the kenel mapping just not visible to the VM and simplified the x86 pmd_xxx() macros. The approach definitely worked, though.
Linus
Quoted 20 lines · Linus Torvalds
In article <[email protected]>,
Quoted 4 lines · Rik van Riel
In other words, large pages should be a "special hack" for special applications, like Oracle and maybe some scientific calculations ?
Yes, I think so.
That said, a 64kB page would be useful for generic use.
Quoted 2 lines
Grabbing some bitflags in generic datastructures shouldn't be an issue since free bits are available.
I had large-page-support working in the VM a long time ago, back when I did the original VM portability rewrite. I actually exposed the kernel large pages to the VM, and it worked fine - I didn't even need a new bit, since the code just used the "large page" bit in the page table directly.
Quoted 3 lines
But it wasn't ever exposed to user space, and in the end I just made the kenel mapping just not visible to the VM and simplified the x86
pmd_xxx()macros. The approach definitely worked, though.
Couldn't we choose the page size depending on the map size ?
If we start mixing page sizes, what about kernel code that assumes PAGE_SIZE ?
- Davide
Quoted 5 lines · Davide Libenzi
Quoted 3 lines · Linus Torvalds
In article <[email protected]>,
Quoted 1 line · Rik van Riel
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
In other words, large pages should be a "special hack" for special applications, like Oracle and maybe some scientific calculations ?
Yes, I think so.
Couldn't we choose the page size depending on the map size ?
For on-disk files I guess this is better an mmap flag, but for shared memory segments we could try to do this automagically.
Quoted 2 lines
If we start mixing page sizes, what about kernel code that assumes
PAGE_SIZE?
We fix it.
Rik
Signature · 3 lines
<insert bitkeeper endorsement here> http://www.surriel.com/ http://distro.conectiva.com/
Quoted 10 lines · Rik van Riel
Quoted 9 lines · Davide Libenzi
Quoted 7 lines · Linus Torvalds
In article <[email protected]>,
Quoted 4 lines · Rik van Riel
In other words, large pages should be a "special hack" for special applications, like Oracle and maybe some scientific calculations ?
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Yes, I think so.
Couldn't we choose the page size depending on the map size ?
For on-disk files I guess this is better an mmap flag, but for shared memory segments we could try to do this automagically.
What's the reason that would make more convenient for us, upon receiving a request to map a NNN MB file, to map it using 4Kb pages instead of 4MB ones ?
- Davide
Quoted 3 lines · Davide Libenzi
What's the reason that would make more convenient for us, upon receiving a request to map a NNN MB file, to map it using 4Kb pages instead of 4MB ones ?
Ehh.. Let me count the ways:
- reliably allocation of 4MB of contiguous data
- graceful fallback when you need to start paging
- sane coherency with somebody who mapped the same file/segment in a much smaller chunk
Guyes, 4MB pages are always going to be a special case. There's no sane way to make them automatic, for the simple reason that they are USELESS for "normal" work, and they have tons of problems that are quite fundamental and just aren't going away and cannot be worked around.
The only sane way to use 4MB segments is:
- the application does a special system call (or special flag to mmap) saying that it wants a big page and doesn't care about coherency with anybody else that didn't set the flag (and realize that that probably includes things like read/write)
- the machine has sufficiently enough memory that the user can be allowed to _lock_ the area down, so that you don't have to worry about swapping out that thing in 4M pieces. (This of course implies that per-user memory counters have to work too, or we have to limit it by default with a rlimit or something to zero).
In short, very much a special case.
(There are two reasons you don't want to handle paging on 4M chunks: (a) they may be wonderful for IO throughput, but they are horrible for latency for other people and (b) you now have basically just a few bits of usage information for 4M worth of memory, as opposed to a finer granularity view of which parts are actually _used_).
Once you can count on having memory sizes in the hundreds of Gigs, and disk throughput speeds in the hundreds of megs a second, and ther are enough of these machines to _matter_ (and reliably 64-bit address spaces so that virtual fragmentation doesn't matter), we might make 4MB the regular mapping entity.
That's probably at least a decade away.
Linus
Quoted 37 lines · Linus Torvalds
Quoted 3 lines · Davide Libenzi
What's the reason that would make more convenient for us, upon receiving a request to map a NNN MB file, to map it using 4Kb pages instead of 4MB ones ?
Ehh.. Let me count the ways:
- reliably allocation of 4MB of contiguous data
- graceful fallback when you need to start paging
- sane coherency with somebody who mapped the same file/segment in a much smaller chunkGuyes, 4MB pages are always going to be a special case. There's no sane way to make them automatic, for the simple reason that they are USELESS for "normal" work, and they have tons of problems that are quite fundamental and just aren't going away and cannot be worked around.
The only sane way to use 4MB segments is:
- the application does a special system call (or special flag to mmap) saying that it wants a big page and doesn't care about coherency with anybody else that didn't set the flag (and realize that that probably includes things like read/write)
- the machine has sufficiently enough memory that the user can be allowed to _lock_ the area down, so that you don't have to worry about swapping out that thing in 4M pieces. (This of course implies that per-user memory counters have to work too, or we have to limit it by default with a rlimit or something to zero).
In short, very much a special case.
(There are two reasons you don't want to handle paging on 4M chunks: (a) they may be wonderful for IO throughput, but they are horrible for latency for other people and (b) you now have basically just a few bits of usage information for 4M worth of memory, as opposed to a finer granularity view of which parts are actually _used_).
Quoted 7 lines
Once you can count on having memory sizes in the hundreds of Gigs, and disk throughput speeds in the hundreds of megs a second, and ther are enough of these machines to _matter_ (and reliably 64-bit address spaces so that virtual fragmentation doesn't matter), we might make 4MB the regular mapping entity.
That's probably at least a decade away.
Plenty of reason thanks Linus :) ... even if workstations with Gigs of RAM and no swap are not so uncommon nowadays. Anyway i agree about the flag driven activation ...
- Davide
Quoted 14 lines · Davide Libenzi
Quoted 12 lines · Rik van Riel
Quoted 10 lines · Davide Libenzi
Quoted 8 lines · Linus Torvalds
In article <[email protected]>,
Quoted 4 lines · Rik van Riel
In other words, large pages should be a "special hack" for special applications, like Oracle and maybe some scientific calculations ?
Yes, I think so.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Couldn't we choose the page size depending on the map size ?
For on-disk files I guess this is better an mmap flag, but for shared memory segments we could try to do this automagically.
What's the reason that would make more convenient for us, upon receiving a request to map a NNN MB file, to map it using 4Kb pages instead of 4MB ones ?
... the VM chooses to unmap a mmaped page, it chooses a 4mb page, later it needs just a few bytes from that unmaped page and free 4mb instead of 4kb (worst case) to map that page again...
Quoted 8 lines · Mike Fedyk
Quoted 2 lines · Davide Libenzi
What's the reason that would make more convenient for us, upon receiving a request to map a NNN MB file, to map it using 4Kb pages instead of 4MB ones ?
... the VM chooses to unmap a mmaped page, it chooses a 4mb page, later it needs just a few bytes from that unmaped page and free 4mb instead of 4kb (worst case) to map that page again...
The big-page property should be vma related and should be obviously handled correctly ...
- Davide
Quoted 4 lines · Linus Torvalds
We may get there some day, but right now 2M pages are not usable for use access.
64kB would be fine, though.
[...]
Quoted 3 lines
Give up on large pages - it's just not happening. Even when a 64kB page would make sense from a technology standpoint these days, backwards compatibility makes people stay at 4kB.
I would think that "large page support" that the processors give you is indeed unusable, but what do you think about "software large(r) pages"?
What I mean is that instead of doing the 4k that the ia32 hardware gives us, we pretend that pages are (e.g.) 8k. Thus we always load a pair of page table entries. memmap ends up having 8k granularity, IO is done on page-sized (i.e. 8k in this case) chunks etc etc. (%)
So we have a "PAGE_SIZE" define all around the kernel. Keep that the same (for compatibility), but make a "REAL_PAGE_SIZE" that governs the loop that actually sets the page table (or tlb) entries.... Note that a first implementation may actually effectivly reduce the size of the TLB on machines with a software loaded TLB....
Why would I want this? Well, suppose I have a machine that unavoidably has to swap on some of its workload. In practise you will almost double the disk troughput by increasing the page size by a factor of two. If the hit rate on the "extra page" that you swap in by pretending pages are 8k and not 4k is over a couple of percents (*), then this is advantageous: A seek plus transfer of 4k costs say 10ms + 0.16us while a seek plus transfer of 8k costs 10ms + .33 us (#). Thus the "penalty" of the extra 4k transfer is very, very small.
Now, for all the reasons you mention, keeping 4k as the "default" on Intel is good. But a config option:
on on
intel alpha
page size: 4k
8k 8k 16k 16k 32k 32k 64k 64k 128k 128k
would also be good for some people.
Roger.
(*) Actually it has to be a "couple of percents better than the 4k page that we had to evict to be able to accomodate the current extra 4k...
(#) A 10k RPM disk rotates in 6ms, so average rotational latency is about 3ms, and with an average seek time of 7ms, that comes to 10ms.
(%) And we get ext2 8k block size support on ia32!
Signature · 4 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2137555 ** *-- BitWizard writes Linux device drivers for any device you may have! --* * There are old pilots, and there are bold pilots. * There are also old, bald pilots.
Quoted 8 lines · Rogier Wolff
...
So we have a "PAGE_SIZE" define all around the kernel. Keep that the same (for compatibility), but make a "REAL_PAGE_SIZE" that governs the loop that actually sets the page table (or tlb) entries.... Note that a first implementation may actually effectivly reduce the size of the TLB on machines with a software loaded TLB....
Quoted 4 lines
Why would I want this? Well, suppose I have a machine that unavoidably has to swap on some of its workload. In practise you will almost double the disk troughput by increasing the page size by a factor of two.
swapin and swapout already perform multipage clustering - you'd get the same benefits from increasing SWAP_CLUSTER_MAX and page_cluster.
Which is a three-line patch.
Frankly, all the discussion I've seen about altering page sizes threatens to add considerable complexity for very dubious gains. The only place where I've seen a solid justification is for scientific applications which have a huge working set, and need large pages to save on TLB thrashing.
For everything else, I believe we can get the efficiencies which we need by writing efficient code; no need to go playing with page sizes.
If someone can point at a real-world workload and say "we suck", and we can't fix that suckage without altering the page size then would that person please come forth.
-
Quoted 2 lines
Frankly, all the discussion I've seen about altering page sizes threatens to add considerable complexity for very dubious gains.
If we don't mix page sizes, but just increase the default from 4k, does this still add a lot of complexity in your eyes? I can't see why it would ... ?
Quoted 3 lines
If someone can point at a real-world workload and say "we suck", and we can't fix that suckage without altering the page size then would that person please come forth.
I believe one of the traditional problems stated for this case is the amount of virtual address space taken up by all the struct pages for a machine with large amounts of memory (32-64Gb). At the moment, the obvious choice of architecture is still 32 bit, but maybe AMD Hammer will fix this ... Unless someone has a plan to move all those up into highmem as well ....
M.
Quoted 5 lines · Andrew Morton
Frankly, all the discussion I've seen about altering page sizes threatens to add considerable complexity for very dubious gains. The only place where I've seen a solid justification is for scientific applications which have a huge working set, and need large pages to save on TLB thrashing.
A widely used example is mprime - the mersenne prime finding program (
http://www.mersenne.org/ ). This typically uses 8 or more MBytes of RAM which it completely thrashes.
The program is written in very efficient assembler code and has been designed not to thrash the TLB as much as possible, but with a working set of > 8 MBs (which is iterated through many times a second at maximum memory bandwith) large pages would make a real improvement to it. Since each run takes weeks any improvement would be eagerly snatched at by the 1000s of people running this program ;-)
If there was some hack where 4MB pages could be allocated for applications like this then I'd be very happy!
Signature · 2 lines
Nick Craig-Wood [email protected]
Quoted 2 lines
If there was some hack where 4MB pages could be allocated for applications like this then I'd be very happy!
You could always run it as a kernel module.
Just need to add schedule points or use a preemptive kernel. When you allocate data using get_free_pages() it'll return a pointer in the 2 or 4MB mapped direct mapping of the kernel. It'll only work when your memory is not fragmented.
-Andi
Quoted 1 line
Quoted 4 lines · Linus Torvalds
Oh, and in the specific case of hammer, one of the main advantages of the thing is of course running old binaries unchanged. And old binaries certainly do mmap's at smaller granularity than 2M (and have to, because a 3G user address space won't fit all that many 2M chunks).
The idea was to only map selected mappings using large pages, e.g. shared memory mappings to help all the databases or use a special mmap flag for the Beowulf people.
Quoted 3 lines
Give up on large pages - it's just not happening. Even when a 64kB page would make sense from a technology standpoint these days, backwards compatibility makes people stay at 4kB.
Yes the 4KB page has to be kept at least for now.
-ANdi
Quoted 6 lines · On Sat, 16 Mar 2002 21:22:29 +0100 Andi Kleen <[email protected]> wrote:
Quoted 3 lines · Linus Torvalds
Give up on large pages - it's just not happening. Even when a 64kB page would make sense from a technology standpoint these days, backwards compatibility makes people stay at 4kB.
Yes the 4KB page has to be kept at least for now.
We have sysconf(_SC_PAGESIZE). I say, introduce an experimental CONFIG for 64k pagesize in 2.5, so we can start to weed out the problem apps NOW.
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
Quoted 7 lines · On Sat, Mar 16, 2002 at 12:57:11PM -0700, [email protected] wrote:
Quoted 4 lines · Andi Kleen
x86-64 aka AMD Hammer does hardware (or more likely microcode) search of page tables.
It has a 4 level page table with 4K pages. Generic Linux MM code only sees the first slot in 4th level page limit user space to 512GB with 3 levels.What about 2M pages?
They are not supported for user space, but used in private mappings for kernel text and direct memory mappings. Generic code never sees them.
That will hopefully change eventually because 2M pages are a bit help for a lot of applications that are limited by TLB thrashing, but needs some thinking on how to avoid the fragmentation trap (e.g. I'm considering to add a highmem zone again just for that and use rmap with targetted physical freeing to allocating them)
Quoted 8 lines
Quoted 5 lines
Direct mappings and kernel mappings are handled specially by architecture specific code outside that first slot.
The CPU itself has I/D TLBs split into L1 and L2.
There was something in some AMD doc about preventing tlbflush on process switch - through a context like thing perhaps? Any idea?
There are global pages which are normally not flushed over context switch. That is used for all kernel mappings.
There is also some optimization in the CPU that tries to do a selective flush only when you reload CR3, but as far as I can see doesn't help for the Linux context switch. It only works around broken TLB flushing algorithms in some Windows version.
-Andi
Quoted 9 lines · Andi Kleen
Quoted 7 lines · On Sat, Mar 16, 2002 at 12:57:11PM -0700, [email protected] wrote:
Quoted 4 lines · Andi Kleen
x86-64 aka AMD Hammer does hardware (or more likely microcode) search of page tables.
It has a 4 level page table with 4K pages. Generic Linux MM code only sees the first slot in 4th level page limit user space to 512GB with 3 levels.What about 2M pages?
Quoted 8 lines
They are not supported for user space, but used in private mappings for kernel text and direct memory mappings. Generic code never sees them.
That will hopefully change eventually because 2M pages are a bit help for a lot of applications that are limited by TLB thrashing, but needs some thinking on how to avoid the fragmentation trap (e.g. I'm considering to add a highmem zone again just for that and use rmap with targetted physical freeing to allocating them)
To me, once you have a G of memory, wasting a few meg on unused process memory seems no big deal.
Quoted 10 lines
Quoted 2 lines
There was something in some AMD doc about preventing tlbflush on process switch - through a context like thing perhaps? Any idea?
There are global pages which are normally not flushed over context switch. That is used for all kernel mappings.
There is also some optimization in the CPU that tries to do a selective flush only when you reload CR3, but as far as I can see doesn't help for the Linux context switch. It only works around broken TLB flushing algorithms in some Windows version.
They say:
Hammer microarchitecture features a flush filter allowing multiple processes to share TLB without SW intervention.
Not a lot of technical detail in that.
Quoted 2 lines
-Andi
Signature · 4 lines
--------------------------------------------------------- Victor Yodaiken Finite State Machine Labs: The RTLinux Company. www.fsmlabs.com www.rtlinux.com
Quoted 9 lines · [email protected] writes:
Quoted 5 lines · Andi Kleen
That will hopefully change eventually because 2M pages are a bit help for a lot of applications that are limited by TLB thrashing, but needs some thinking on how to avoid the fragmentation trap (e.g. I'm considering to add a highmem zone again just for that and use rmap with targetted physical freeing to allocating them)
To me, once you have a G of memory, wasting a few meg on unused process memory seems no big deal.
I'm not happy to throw away 2 MiB per process. My workstation has 1 GiB of RAM, and 65 processes (and that's fairly low compared to your average desktop these days, because I just use olwm and don't have a fancy desktop or lots of windows). You want me to throw over 1/8th of my RAM away?!?
And in fact, isn't it going to be more than 2 MiB wasted per process? For each shared object loaded, only partial pages are going to be used. *My* libc is less than 700 KiB, so I'd be wasting most of a page to map it in.
I want that 1 GiB of RAM to be used to cache most of my data. Those NASA 1km/pixel satellite mosaics of the world are pretty big, you know (21600x21600x3 per hemisphere:-).
Regards, Richard.... Permanent: [email protected] Current: [email protected]
Quoted 3 lines · On Sat, 16 Mar 2002 [email protected] wrote:
To me, once you have a G of memory, wasting a few meg on unused process memory seems no big deal.
It's not the process memory, and it is a whole lot than a "few meg" if your page size is 2M.
Look at "free" output one day, and notice that "cached" line? On my 2G machine, I usually have about a gig cached or so. Guess what the most common thing in that case is? Yeah, the kernel.
And my kernel tree (with bk overhead etc) is right not about 25.000 files. That's without object files etc. At 2M a pop in the page cache, that's a whole lot more memory for caching than I have in my machine.
Ok, so assume you compress that, and you only actually use the full 2M when mapping into user space, you now added a lot of complexity, but at least you make the ridiculous memory use go down.
But even in the process space, I've got about 150 processes quite normally, and while most of them are idle, if we had 2M pages most of them would waste at least 2M of memory (probably more - the stack doesn't even need half a page, and the data section would probably waste half a page on average).
That's 300M just wasted.
Tell me that's peanuts even if you've got a few gigs of ram on your machine.
Admit it, you're just wrong. 2M page sizes are _not_ useful for the common case, and won't be for years to come.
In short, youäre
Quoted 5 lines
They say:
Hammer microarchitecture features a flush filter allowing multiple processes to share TLB without SW intervention.Not a lot of technical detail in that.
I suspect it's some special case for windows with a special MSR that enables something illegal that just works well for whatever patterns windows does.
Linus
Quoted 8 lines · Linus Torvalds
Quoted 3 lines · On Sat, 16 Mar 2002 [email protected] wrote:
To me, once you have a G of memory, wasting a few meg on unused process memory seems no big deal.
It's not the process memory, and it is a whole lot than a "few meg" if your page size is 2M.
I forget what an extremist you are. My claim is that
some processes benefit from big pages, some do not.
A 16G process needs 2^25 bytes of PTE at 4kbytes/page if I did the numbers right. Just populating 4 million odd page tables is a pain. I might be wrong about it, but I wonder if just scaling up from a working 32 bit strategy gets you anywhere.
If you want to optimize for gnome, you get a very different layout. But Hammer and ia64 are supposedly designed for huge databases, routing tables, and images. Our good friends at Intel claim "carrier grade" Linux needs to run threaded apps with 10,000 threads to depose Solaris in telecom - all sharing the same monster address space.
Quoted 2 lines
Admit it, you're just wrong. 2M page sizes are _not_ useful for the common case, and won't be for years to come.
What's the "common case" for 64 bit ? Do you really think it will be on desktop soon?
Quoted 2 lines
In short, youäre
Don't use umlauts unless you are ready to back it up.
Quoted 10 lines
Quoted 6 lines
They say:
Hammer microarchitecture features a flush filter allowing multiple processes to share TLB without SW intervention.Not a lot of technical detail in that.
I suspect it's some special case for windows with a special MSR that enables something illegal that just works well for whatever patterns windows does.
sounds like it from what Andi wrote. disappointing.
Quoted 2 lines
Linus
Signature · 4 lines
--------------------------------------------------------- Victor Yodaiken Finite State Machine Labs: The RTLinux Company. www.fsmlabs.com www.rtlinux.com
Quoted 4 lines
databases, routing tables, and images. Our good friends at Intel claim "carrier grade" Linux needs to run threaded apps with 10,000 threads to depose Solaris in telecom - all sharing the same monster address space.=20
Thats intel though. The same people who seem to think that hyperthreading in the CPU is required for carrier grade work 8)
Quoted 3 lines · Alan Cox
Thats intel though. The same people who seem to think that hyperthreading in the CPU is required for carrier grade work 8)
Well, to be fair, they do seem to be making inroads against Sun.
Maybe that factor-of-five performance edge has something to do with that part, though ;)
Linus
Quoted 8 lines · Kai Henningsen
[email protected] wrote on 16.03.02 in <[email protected]>:
Quoted 6 lines · Alan Cox
Quoted 5 lines
Quoted 4 lines
databases, routing tables, and images. Our good friends at Intel claim "carrier grade" Linux needs to run threaded apps with 10,000 threads to depose Solaris in telecom - all sharing the same monster address space.=20
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Thats intel though. The same people who seem to think that hyperthreading in the CPU is required for carrier grade work 8)
I love the whole sound of "carrier grade" though: Do you use "carrier grade" Linux or just the "recreational boating" version?
Wrong carrier, though. It's not US Navy carriers (those people use NT, after all, and this was "depose Solaris"), it's carriers like AT&T - phone
Really? So why are they always talking about that ship SS7 and the sonar network - SONET ? I think Alan may have something there about the carrier pigeon angle, though. Needs more study.
Actually, it makes me think of "as big, manoeuverable, and low cost as an aircraft carrier" although that is certainly unfair.
Quoted 7 lines
companies. And I suspect many of those 10,000 threads are handling one phone conversation each. Or maybe one half of one.
In fact, that's a problem space I find much more interesting than the military. *These* people need to be robust in peacetime. They can't afford a big showy piece of hardware that breaks down when it's finally needed, because "finally" is a very short-term goal.
But in my, as always ever so humble, opinion, 10,000 threads is a programming error based on the incorrect Solaris theory that threads were a good substitute for thinking about scheduling operations. So making Linux handle 10K threads is not necessarily an appealing idea unless you can think of some very clever way to do it.
On the other hand, if I just wanted to sell chips, I might think differently.
Quoted 7 lines
MfG Kai
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 4 lines
--------------------------------------------------------- Victor Yodaiken Finite State Machine Labs: The RTLinux Company. www.fsmlabs.com www.rtlinux.com
Quoted 4 lines · On Sat, 16 Mar 2002 [email protected] wrote:
If you want to optimize for gnome, you get a very different layout. But Hammer and ia64 are supposedly designed for huge databases, routing tables, and images.
Yeah, and I'm claiming that databases etc count for a whole lot less than most other apps.
Quoted 2 lines
What's the "common case" for 64 bit ? Do you really think it will be on desktop soon?
Oh, not Itanium for sure. But if AMD succeeds even reasonably with Hammer, Intel will certainly see the error of its ways (except they won't admit it) and make their version of a 64-bit P4 core available. They're reportedly working on it already, just in case the Itanic sinks (and judging by current market behaviour it certainly isn't flying).
Quoted 3 lines
Quoted 1 line
In short, youäre
Don't use umlauts unless you are ready to back it up.
That's not an umlaut, that's an "ä", which is a real letter in Finnish and Swedish (it just _looks_ like an a with an umlaut to you uncultured people), and it happens to be a letter that is just left of the ' mark on a Finnish keyboard.
Linus
[email protected] (Linus Torvalds) wrote on 16.03.02 in <[email protected]>:
Quoted 10 lines · On Sat, 16 Mar 2002 [email protected] wrote:
Quoted 4 lines
Quoted 2 lines
In short, youaere
Don't use umlauts unless you are ready to back it up.
That's not an umlaut, that's an "ae", which is a real letter in Finnish and Swedish (it just _looks_ like an a with an umlaut to you uncultured people), and it happens to be a letter that is just left of the ' mark on a Finnish keyboard.
Hey, careful there! Those English speakers stole that name from German, and in German those umlauts are real letters, too. Incidentally, my ae is next to the '# key ...
MfG Kai
Quoted 8 lines
Quoted 4 lines
That's not an umlaut, that's an "ae", which is a real letter in Finnish and Swedish (it just _looks_ like an a with an umlaut to you uncultured people), and it happens to be a letter that is just left of the ' mark on a Finnish keyboard.
Hey, careful there! Those English speakers stole that name from German, and in German those umlauts are real letters, too. Incidentally, my ae is next to the '# key ...
There are still a couple of places you can legitimaely use an ae symbol in English. It's not quite dead yet 8)
2.4.19-pre3-ac1 - Quotactl patch
You might want to upgrade to the newer one, the patch has undergone a lot of changes and -ac against the XFS merge of the quotactl patch seriously breaks ;-(
Shawn.
Re: 2.4.19-pre3-ac1 - Quotactl patch
Quoted 3 lines
You might want to upgrade to the newer one, the patch has undergone a lot of changes and -ac against the XFS merge of the quotactl patch seriously breaks ;-(
The -ac patch is the current stable 2.4 one for 32bit uid quota. XFS is a 2.5 problem so for the moment I'm not remotely bothered by it
Re: 2.4.19-pre3-ac1 - Quotactl patch
oh ok.
Thanks.
Quoted 9 lines · Alan Cox
Quoted 3 lines
You might want to upgrade to the newer one, the patch has undergone a lot of changes and -ac against the XFS merge of the quotactl patch seriously breaks ;-(
The -ac patch is the current stable 2.4 one for 32bit uid quota. XFS is a 2.5 problem so for the moment I'm not remotely bothered by it
OT: "real" letters [Was: 10.31 second kernel compile]
In reply to Alan CoxQuoted 3 lines
Quoted 3 lines
Hey, careful there! Those English speakers stole that name from German, and in German those umlauts are real letters, too. Incidentally, my ae is next to the '# key ...
There are many ways to count "real" letters. Eg., Hebrew (my native lang.) counts just 22 but there are 27 glyphs and many combinations with points (Nikkud) and accents. The Spanish count 27 letters in their alphabet (ñ is a real letter) while the Brazilians count only 23 (the 26 English letters minus K,W and Y). Some east-Asian languages haven't finished the count yet (or never have started)...
Quoted 2 lines
There are still a couple of places you can legitimaely use an ae symbol in English. It's not quite dead yet 8)
The only example that I've seen in English texts is use of ï as in "naïve".
I know that ä is used in German and in other languages but if I see a text that contains a double-ä or 3 ä in one word that's almost certainly Finnish...
-- Itai
Re: OT: "real" letters [Was: 10.31 second kernel compile]
In reply to Alan CoxQuoted 6 lines · Itai Nahshon
Quoted 3 lines
There are still a couple of places you can legitimaely use an ae symbol in English. It's not quite dead yet 8)
The only example that I've seen in English texts is use of ï as in "naïve".
The two I can recall off the top of my head are "dæmon" and "færie".
DF
Re: OT: "real" letters [Was: 10.31 second kernel compile]
In reply to Alan CoxQuoted 9 lines · Derek Fawcus
Quoted 6 lines · Itai Nahshon
Quoted 3 lines
There are still a couple of places you can legitimaely use an ae symbol in English. It's not quite dead yet 8)
The only example that I've seen in English texts is use of ï as in "naïve".
The two I can recall off the top of my head are "dæmon" and "færie".
And, of course, encyclopædia. Or pædiatrician. Perhaps Lænux?
Signature · 4 lines
Tim Coleman <[email protected]> [43.28 N 80.31 W] BMath, Honours Combinatorics and Optimization, University of Waterloo "They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." -- Benjamin Franklin
Quoted 7 lines · Alan Cox
Quoted 4 lines
databases, routing tables, and images. Our good friends at Intel claim "carrier grade" Linux needs to run threaded apps with 10,000 threads to depose Solaris in telecom - all sharing the same monster address space.=20
Thats intel though. The same people who seem to think that hyperthreading in the CPU is required for carrier grade work 8)
I love the whole sound of "carrier grade" though: Do you use "carrier grade" Linux or just the "recreational boating" version?
Signature · 4 lines
--------------------------------------------------------- Victor Yodaiken Finite State Machine Labs: The RTLinux Company. www.fsmlabs.com www.rtlinux.com
On Sat, 16 Mar 2002 [email protected] wrote:
| On Sat, Mar 16, 2002 at 10:00:07PM +0000, Alan Cox wrote: | > > databases, routing tables, and images. Our good friends at Intel | > > claim "carrier grade" Linux needs to run threaded apps | > > with 10,000 threads to depose Solaris in telecom - all sharing the | > > same monster address space.=20 | > | > Thats intel though. The same people who seem to think that hyperthreading | > in the CPU is required for carrier grade work 8) | | I love the whole sound of "carrier grade" though: Do you use "carrier grade" | Linux or just the "recreational boating" version?
Don't know what Intel is claiming, but OSDL is sponsoring a "Carrier Grade Linux Working Group". See info at www.osdl.org and/or at cglinux.sf.net .
Signature · 1 line
~Randy
Quoted 2 lines
I love the whole sound of "carrier grade" though: Do you use "carrier grade" Linux or just the "recreational boating" version?
Linux is alread carrier (pigeon) grade
-> www.blug.linux.no/rfc1149/
[email protected] wrote on 16.03.02 in <[email protected]>:
Quoted 11 lines · Alan Cox
Quoted 7 lines
Quoted 4 lines
databases, routing tables, and images. Our good friends at Intel claim "carrier grade" Linux needs to run threaded apps with 10,000 threads to depose Solaris in telecom - all sharing the same monster address space.=20
Thats intel though. The same people who seem to think that hyperthreading in the CPU is required for carrier grade work 8)
I love the whole sound of "carrier grade" though: Do you use "carrier grade" Linux or just the "recreational boating" version?
Wrong carrier, though. It's not US Navy carriers (those people use NT, after all, and this was "depose Solaris"), it's carriers like AT&T - phone companies. And I suspect many of those 10,000 threads are handling one phone conversation each. Or maybe one half of one.
In fact, that's a problem space I find much more interesting than the military. *These* people need to be robust in peacetime. They can't afford a big showy piece of hardware that breaks down when it's finally needed, because "finally" is a very short-term goal.
MfG Kai
Hi!
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
To me, once you have a G of memory, wasting a few meg on unused process memory seems no big deal.
It's not the process memory, and it is a whole lot than a "few meg" if your page size is 2M.
Quoted 9 lines
I forget what an extremist you are. My claim is that
some processes benefit from big pages, some do not.
A 16G process needs 2^25 bytes of PTE at 4kbytes/page if I did the numbers right. Just populating 4 million odd page tables is a pain. I might be wrong about it, but I wonder if just scaling up from a working 32 bit strategy gets you anywhere.
If you want to optimize for gnome, you get a very different layout. But Hammer and ia64 are supposedly designed for huge databases, routing tables, and images. Our good friends at Intel
Hammer is designed for desktop, AFAICT. [Its slightly modified athlon, you see?]
Pavel
Signature · 2 lines
(about SSSCA) "I don't say this lightly. However, I really think that the U.S. no longer is classifiable as a democracy, but rather as a plutocracy." --hpa
Quoted 2 lines · Pavel Machek
Hammer is designed for desktop, AFAICT. [Its slightly modified athlon, you see?]
Thanks for the insight. Only by reading LKM could I have found out that AMD doesn't care about server space.
Quoted 6 lines · On Tue, 19 Mar 2002 [email protected] wrote:
Quoted 2 lines · Pavel Machek
Hammer is designed for desktop, AFAICT. [Its slightly modified athlon, you see?]
Thanks for the insight. Only by reading LKM could I have found out that AMD doesn't care about server space.
You can get BS from Intel lovers in the Intel group as well. If you believe that 8-way SMP doesn't indicate an interest in server space you see a larger market for games and DSI than I do.
Signature · 3 lines
bill davidsen <[email protected]> CTO, TMR Associates, Inc Doing interesting things with little computers since 1979.
Quoted 7 lines · [email protected] wrote:
Quoted 2 lines · Pavel Machek
Hammer is designed for desktop, AFAICT. [Its slightly modified athlon, you see?]
Thanks for the insight. Only by reading LKM could I have found out that AMD doesn't care about server space.
The sledgehammer is a bit more than a slightly modified athlon...
up to 8 way multiprocessing
64-bit addressing
hypertransport
integrated DDR memory controller
Signature · 4 lines
Chris Friesen | MailStop: 043/33/F10 Nortel Networks | work: (613) 765-0557 3500 Carling Avenue | fax: (613) 765-2986 Nepean, ON K2H 8E9 Canada | email: [email protected]
Quoted 9 lines · Chris Friesen
Quoted 7 lines · [email protected] wrote:
Quoted 2 lines · Pavel Machek
Hammer is designed for desktop, AFAICT. [Its slightly modified athlon, you see?]
Thanks for the insight. Only by reading LKM could I have found out that AMD doesn't care about server space.
Quoted 6 lines
The sledgehammer is a bit more than a slightly modified athlon...
up to 8 way multiprocessing
64-bit addressing
hypertransport
integrated DDR memory controller
Look at www.amd.com front page for some subtle hints on where the margin is.
Quoted 12 lines
Signature · 10 lines
-- Chris Friesen | MailStop: 043/33/F10 Nortel Networks | work: (613) 765-0557 3500 Carling Avenue | fax: (613) 765-2986 Nepean, ON K2H 8E9 Canada | email: [email protected] - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Signature · 4 lines
--------------------------------------------------------- Victor Yodaiken Finite State Machine Labs: The RTLinux Company. www.fsmlabs.com www.rtlinux.com
Quoted 8 lines · Richard Gooch
Quoted 7 lines · [email protected] writes:
Quoted 5 lines · Andi Kleen
That will hopefully change eventually because 2M pages are a bit help for a lot of applications that are limited by TLB thrashing, but needs some thinking on how to avoid the fragmentation trap (e.g. I'm considering to add a highmem zone again just for that and use rmap with targetted physical freeing to allocating them)
Quoted 8 lines
Quoted 2 lines
To me, once you have a G of memory, wasting a few meg on unused process memory seems no big deal.
I'm not happy to throw away 2 MiB per process. My workstation has 1 GiB of RAM, and 65 processes (and that's fairly low compared to your average desktop these days, because I just use olwm and don't have a fancy desktop or lots of windows). You want me to throw over 1/8th of my RAM away?!?
Why not? If you just ran vim on console you'd be more productive and not need all those worthless processes.
At 4KB/page and 8bytes/pte a
1G process will need at least 2MB of pte alone ! Add in the 4 layers, the software VM struct, ...
Quoted 5 lines
And in fact, isn't it going to be more than 2 MiB wasted per process? For each shared object loaded, only partial pages are going to be used. *My* libc is less than 700 KiB, so I'd be wasting most of a page to map it in.
You're using a politically incorrect libc.
But sure, big pages are not always good.
Quoted 11 lines
I want that 1 GiB of RAM to be used to cache most of my data. Those NASA 1km/pixel satellite mosaics of the world are pretty big, you know (21600x21600x3 per hemisphere:-).
Regards, Richard.... Permanent: [email protected] Current: [email protected]
Signature · 4 lines
--------------------------------------------------------- Victor Yodaiken Finite State Machine Labs: The RTLinux Company. www.fsmlabs.com www.rtlinux.com
Quoted 12 lines · [email protected] writes:
Quoted 11 lines · Richard Gooch
Quoted 9 lines · [email protected] writes:
Quoted 5 lines · Andi Kleen
That will hopefully change eventually because 2M pages are a bit help for a lot of applications that are limited by TLB thrashing, but needs some thinking on how to avoid the fragmentation trap (e.g. I'm considering to add a highmem zone again just for that and use rmap with targetted physical freeing to allocating them)
To me, once you have a G of memory, wasting a few meg on unused process memory seems no big deal.
Quoted 8 lines
Quoted 5 lines
I'm not happy to throw away 2 MiB per process. My workstation has 1 GiB of RAM, and 65 processes (and that's fairly low compared to your average desktop these days, because I just use olwm and don't have a fancy desktop or lots of windows). You want me to throw over 1/8th of my RAM away?!?
Why not? If you just ran vim on console you'd be more productive and not need all those worthless processes.
Yeah, right.
Quoted 3 lines
At 4KB/page and 8bytes/pte a
1G process will need at least 2MB of pte alone ! Add in the 4 layers, the software VM struct, ...
This isn't a dedicated bigass-image display box. It's a workstation. It's where I read email, hack kernels, write visualisation tools and stuff like that.
And I can afford a few MiB of RAM for PTE's and such for *the one process which is mapping my huge data files*! That's effectively a small, one-time cost. Every other process doesn't have a significant PTE cost.
I'm not using my kernel as a device driver for an image display programme. I'm using it run a box that's generally useful to me.
Quoted 6 lines
Quoted 4 lines
And in fact, isn't it going to be more than 2 MiB wasted per process? For each shared object loaded, only partial pages are going to be used. *My* libc is less than 700 KiB, so I'd be wasting most of a page to map it in.
You're using a politically incorrect libc.
Yeah :-) Man it feels good.
Quoted 1 line
But sure, big pages are not always good.
Hm. With wide TLB's, what are the benefits to big pages? One pathological case that hit me a few years back was a workload which bounced around in VM in a pattern that really thrash the cache due to aliasing. It wouldn't have been a problem if we had truly fully set-associative caches, rather than N-way (where N is 2, 4 or 8 usually). But big pages won't help that much here (it's just a way of reducing TLB thrash, but doesn't help with cache thrashing).
Regards, Richard.... Permanent: [email protected] Current: [email protected]
Quoted 2 lines · Richard Gooch
And I can afford a few MiB of RAM for PTE's and such for *the one process which is mapping my huge data files*!
Once you have this, you might as well make that granularity per VMA.
This gives you the advantage of being able to share the mapping for libc.so ;)
Quoted 1 line
From what I can see, you'll basically want large pages
for:
1) Oracle and maybe other large shared memory situations where the page table overhead would otherwise be prohibitively high
2) scientific calculations and other programs with a huge dataset where TLB misses would be prohibitively slow
regards,
Rik
Signature · 3 lines
<insert bitkeeper endorsement here> http://www.surriel.com/ http://distro.conectiva.com/
Quoted 4 lines · Richard Gooch
Quoted 2 lines
Why not? If you just ran vim on console you'd be more productive and not need all those worthless processes.
Yeah, right.
I was just trying to be nice.
Quoted 4 lines
Quoted 3 lines
At 4KB/page and 8bytes/pte a
1G process will need at least 2MB of pte alone ! Add in the 4 layers, the software VM struct, ...
Quoted 8 lines
This isn't a dedicated bigass-image display box. It's a workstation. It's where I read email, hack kernels, write visualisation tools and stuff like that.
And I can afford a few MiB of RAM for PTE's and such for *the one process which is mapping my huge data files*! That's effectively a small, one-time cost. Every other process doesn't have a significant PTE cost.
Well, it's a matter of target. I'm thinking about our customers who do high grade image processing on a stream of gig+ bitmaps. They need 64 (some are already painfully stranded on Alphas) and they don't use these boxes for email.
Quoted 3 lines
Quoted 1 line
But sure, big pages are not always good.
Hm. With wide TLB's, what are the benefits to big pages? One
tlb miss rates, mm structure overhead and setup/teardown, swap speed,
---------------------------------------------------------
Victor Yodaiken
Finite State Machine Labs: The RTLinux Company.
www.fsmlabs.com www.rtlinux.com
On Sat, Mar 16, 2002 at 09:05:04PM +0100, Andi Kleen wrote:
On Sat, Mar 16, 2002 at 12:57:11PM -0700, [email protected] wrote:
>
> What about 2M pages?
They are not supported for user space, but used in private mappings for kernel text and direct memory mappings. Generic code never sees them.
Is there any reason we couldn't use them for mapping large frame-buffers and similar?
--cw
Quoted 6 lines
They are not supported for user space, but used in private mappings for kernel text and direct memory mappings. Generic code never sees them.
Is there any reason we couldn't use them for mapping large frame-buffers and similar?
You are labouring under the belief that processors touch the frame buffer nowdays. For a current accelerated frame buffer that isnt very true.
On Sun, Mar 17, 2002 at 03:43:29AM +0000, Alan Cox wrote:
You are labouring under the belief that processors touch the frame buffer nowdays. For a current accelerated frame buffer that isnt very true.
/s/frame-buffer/hunk-of-memory/
Either way, we have tens of MB of ram where we either put textures, options or whatever --- the CPU has to meddle with it one way or another.
--cw
Quoted 3 lines
Either way, we have tens of MB of ram where we either put textures, options or whatever --- the CPU has to meddle with it one way or another.
The disk DMA's it to RAM, the graphics card fetches it via the GART mappings. We shouldn't be touching a lot of it.