Skip to contentSkip to conversation

Very large memory configurations: > 16 TB

DISCUSSION Jack Steiner 4 participants 6 messages Jan 6, 2011 – Jan 7, 2011 Memory management

lore
Share & tools
lore
#0 lore raw

SGI is currently developing an x86_64 system with more than 16TB of memory per SSI. As far as I can tell, this should be supported. The relevant definitions such as MAX_PHYSMEM_BITS appear ok.

One area of concern is page counts. Exceeding 16TB will also exceed MAX_INT page frames. The kernel (at least in all places I've found) keep pagecounts in longs.

Have I missed anything? Should this > 16TB work? Are there any kernel problems or problems with user tools that anyone knows of.

Any help or pointers to potential problem areas would be appreciated...

---
Jack Steiner ([email protected])
SGI - Silicon Graphics, Inc.

#1 lore raw
Quoted 4 lines · Jack Steiner

SGI is currently developing an x86_64 system with more than 16TB of memory per SSI. As far as I can tell, this should be supported. The relevant definitions such as MAX_PHYSMEM_BITS appear ok.

Quoted 8 lines

One area of concern is page counts. Exceeding 16TB will also exceed MAX_INT page frames. The kernel (at least in all places I've found) keep pagecounts in longs.

Have I missed anything? Should this > 16TB work? Are there any kernel problems or problems with user tools that anyone knows of.

Any help or pointers to potential problem areas would be appreciated...

I don't know of any place that uses ints to count physical pages. However, the page_referenced functions in mm/rmap.c return reference counts as an integer. I believe a wraparound would only mislead the LRU algorithms, but I haven't thought about it much. (Not sure why we return a count anyway, since I believe callers only want to compare it against zero ???)

Signature · 2 lines
Michel "Walken" Lespinasse
A program is never fully debugged until the last user dies.
#2 lore raw
In reply to Jack Steiner
Quoted 5 lines · Jack Steiner

SGI is currently developing an x86_64 system with more than 16TB of memory per SSI. As far as I can tell, this should be supported. The relevant definitions such as MAX_PHYSMEM_BITS appear ok.

Quoted 7 lines

One area of concern is page counts. Exceeding 16TB will also exceed MAX_INT page frames. The kernel (at least in all places I've found) keep pagecounts in longs.

Have I missed anything? Should this > 16TB work? Are there any kernel problems or problems with user tools that anyone knows of.

Any help or pointers to potential problem areas would be appreciated...

See this older 2008 mail i wrote about our current x86 64-bit limits:

http://lkml.indiana.edu/hypermail/linux/kernel/0812.2/00292.html

In that mail i outlined the various limits and the methods that it would take to increase those limits, in order of difficulty. It appears we can probably go up to 32 TB relatively easily and up to 64 TB realistically - 128 TB theoretically.

Note that obviously there can be a number of unknown problems rise up, so you should try to simulate a ton of RAM ASAP, before building the hardware ;-) (We could even try to add a "memory size debug" feature to the kernel which would inject huge 'fake' blocks of RAM that the kernel will pretend to have but will skip in the buddy allocator or so.

Beyond 64 TB it probably gets painful, very painful - a hardware extension to the pagetable and canonical virtual memory space is the pragmatic solution there.

Thanks,

Ingo

#3 lore raw

Andi put a description of the memory layout in Documentation/x86/x86_64/mm.txt. Seems to indicate that 64 TB was considered as a maximum when the memory layout for x86_64 was set up:

------

Virtual memory map with 4 level page tables:

0000000000000000 - 00007fffffffffff (G bits) user space, different per mm hole caused by [48:63] sign extension
ffff800000000000 - ffff80ffffffffff (@ bits) guard hole ffff880000000000 - ffffc7ffffffffff (d TB) direct mapping of all phys. memory ffffc80000000000 - ffffc8ffffffffff (@ bits) hole ffffc90000000000 - ffffe8ffffffffff (E bits) vmalloc/ioremap space ffffe90000000000 - ffffe9ffffffffff (@ bits) hole ffffea0000000000 - ffffeaffffffffff (@ bits) virtual memory map (1TB) ... unused hole ...
ffffffff80000000 - ffffffffa0000000 (Q2 MB) kernel text mapping, from phys 0 ffffffffa0000000 - fffffffffff00000 (36 MB) module mapping space

The direct mapping covers all memory in the system up to the highest memory address (this means in some cases it can also include PCI memory holes).

vmalloc space is lazily synchronized into the different PML4 pages of the processes using the page fault handler, with init_level4_pgt as reference.

Current X86-64 implementations only support 40 bits of address space, but we support up to 46 bits. This expands into MBZ space in the page tables.

-Andi Kleen, Jul 2004

#4 lore raw
Quoted 3 lines · Christoph Lameter

Andi put a description of the memory layout in Documentation/x86/x86_64/mm.txt. Seems to indicate that 64 TB was considered as a maximum when the memory layout for x86_64 was set up:

Yes, that document was rather incomplete and does not really answer Jack's questions, that's why i sent this more complete description originally:

http://lkml.indiana.edu/hypermail/linux/kernel/0812.2/00292.html

a few years ago, answering a similar question.

Thanks,

Ingo