sparc64 page table
cover sparc64 page table
From: bob picco <[email protected]>
We encountered an issue with the current sparc64 page table implementation. The T5 memory controller is different than T4. Its coarse grain interleave factor causes the unsigned types of pgd_t and pmd_t to be insufficient. We managed to partially resolve the issue by taking the physical address and storing exactly the physical page frame within the pgd_t and pmd_t. This would be deficient for a large memory configuration with all sockets populated. This memblockÞbug snippet of T5-2 is for a 256Gb configured machine:
MEMBLOCK configuration:
memory size = 0x3fcf214000
memory.cnt = 0x5
memory[0x0] [0x00000024400000-0x00001ff36e1fff], 0x1fcf2e2000 bytes memory[0x1] [0x00001ff36e8000-0x00001ff36e9fff], 0x2000 bytes memory[0x2] [0x00001ff36ee000-0x00001ff36effff], 0x2000 bytes memory[0x3] [0x00080000000000-0x00081ffff23fff], 0x1ffff24000 bytes memory[0x4] [0x00081ffff34000-0x00081ffff3dfff], 0xa000 bytes
reserved.cnt = 0x1
reserved[0x0] [0x00080000000000-0x00080000cd6980], 0xcd6981 bytes
. This is only a two socket machine and minimally memory configured.
The second platform to encounter this issue is Fujitsu's Athena. Athena physical memory is mapped at the most significant bits. You can observe this from this snippet of boot up with memblockÞbug
MEMBLOCK configuration:
memory size = 0xff8f1fe000
memory.cnt = 0xb
memory[0x0] [0x00780070400000-0x0078007f69dfff], 0xf29e000 bytes memory[0x1] [0x0078007fffe000-0x00781fffffffff], 0x1f80002000 bytes memory[0x2] [0x00790000000000-0x00791fffffffff], 0x2000000000 bytes memory[0x3] [0x007a0000000000-0x007a1fffffffff], 0x2000000000 bytes memory[0x4] [0x007b0000000000-0x007b1fffffffff], 0x2000000000 bytes memory[0x5] [0x007c0000000000-0x007c1fffffffff], 0x2000000000 bytes memory[0x6] [0x007d0000000000-0x007d1fffffffff], 0x2000000000 bytes memory[0x7] [0x007e0000000000-0x007e1fffffffff], 0x2000000000 bytes memory[0x8] [0x007f0000000000-0x007f1ffff41fff], 0x1ffff42000 bytes memory[0x9] [0x007f1ffff5a000-0x007f1ffff69fff], 0x10000 bytes memory[0xa] [0x007f1ffffd6000-0x007f1ffffe1fff], 0xc000 bytes
reserved.cnt = 0x2
reserved[0x0] [0x007f0000000000-0x007f00010ca900], 0x10ca901 bytes reserved[0x1] [0x007f0004000000-0x007f0008d4e462], 0x4d4e463 bytes
.
There are several ways to approach a solution for this. We chose to introduce a four level page table scheme. This scheme promotes pgd_t and pmd_t to unsigned long. The pud folding isn't used and the pud_t is also unsigned long.
There are penalties for a four level scheme. A small memory increase because of the addition of a pud. TSB misses will cost another memory load because of the new level. We've yet to observe a negative ramification from the fourth level.
There is a configuration choice between the four level page table scheme and the current three level page table scheme. It is our hope that the exisiting three level scheme has been left unchanged functionally. This can be viewed like sparc64 THP which impacted the existing three level page table scheme and is configuration selectable.
There could potentially be further code consolidation by keying off of pud folding or not. I feel it is better to leave the two levels as independent as possible for now.
My skill level for naming leaves much to be desired at times. I'm certainly not strongly attached to PGTABLE_LEVEL4.
thanx,
bob
bob picco (4):
sparc64 expand linear mapping region
sparc64 move three level page table scheme
sparc64 four level page table support
sparc64 kconfig four level page table
arch/sparc/Kconfig | 9 ++ arch/sparc/include/asm/page_64.h | 20 +++- arch/sparc/include/asm/page_64_lvl3.h | 15 +++ arch/sparc/include/asm/page_64_lvl4.h | 21 ++++ arch/sparc/include/asm/pgalloc_64.h | 16 +++ arch/sparc/include/asm/pgtable_64.h | 135 +++------------------ arch/sparc/include/asm/pgtable_64_lvl3.h | 118 ++++++++++++++++++ arch/sparc/include/asm/pgtable_64_lvl4.h | 191 ++++++++++++++++++++++++++++++ arch/sparc/include/asm/sparsemem.h | 5 + arch/sparc/include/asm/tsb.h | 55 +++++++++ arch/sparc/kernel/ktlb.S | 12 +- arch/sparc/kernel/smp_64.c | 7 ++ arch/sparc/mm/init_64.c | 36 ++++-- arch/sparc/mm/init_64.h | 4 + 14 files changed, 508 insertions(+), 136 deletions(-) create mode 100644 arch/sparc/include/asm/page_64_lvl3.h create mode 100644 arch/sparc/include/asm/page_64_lvl4.h create mode 100644 arch/sparc/include/asm/pgtable_64_lvl3.h create mode 100644 arch/sparc/include/asm/pgtable_64_lvl4.h
Signature · 1 line
1.7.10.4
Every distribution is going to turn on the most capable configuration.
So having a choice between 3-level and 4-level page tables isn't going to buy anything.
I'd rather find a solution that allows us to stay at 3-level page tables and avoid all of the extra cost, as any change here is going to make every single TLB miss more expensive.
From: Bob Picco <[email protected]>
Date: Mon, 16 Sep 2013 09:46:35 -0400
Quoted 5 lines
There is a configuration choice between the four level page table scheme and the current three level page table scheme. It is our hope that the exisiting three level scheme has been left unchanged functionally. This can be viewed like sparc64 THP which impacted the existing three level page table scheme and is configuration selectable.
So again this is unacceptable.
Even worse, the 4-level page table setting creates a kernel that absolutely does not work on older chips.
The problem is the PAGE_OFFSET selections. You can't use a PAGE_OFFSET value that tries to use parts of the virtual address space which is not supported on UltraSPARC-I et al.
So, in particular, PAGE_OFFSET will need to be determined dynamically at run time. Yes, this means looking at the appropriate cpu properties to see how much virtual address space is supported, and then code patching the appropriate assembler instructions. We already have good support for the latter.
Hi,
David Miller wrote: [Wed Sep 18 2013, 12:28:09PM EDT]
Quoted 10 lines
From: Bob Picco <[email protected]>
Date: Mon, 16 Sep 2013 09:46:35 -0400Quoted 5 lines
There is a configuration choice between the four level page table scheme and the current three level page table scheme. It is our hope that the exisiting three level scheme has been left unchanged functionally. This can be viewed like sparc64 THP which impacted the existing three level page table scheme and is configuration selectable.
So again this is unacceptable.
I understand.
Quoted 7 lines
Even worse, the 4-level page table setting creates a kernel that absolutely does not work on older chips.
The problem is the
PAGE_OFFSETselections. You can't use aPAGE_OFFSETvalue that tries to use parts of the virtual address space which is not supported on UltraSPARC-I et al.
This I'm aware of. Otherwise why would it remain where it is at for S3 core.
Quoted 6 lines
So, in particular,
PAGE_OFFSETwill need to be determined dynamically at run time. Yes, this means looking at the appropriate cpu properties to see how much virtual address space is supported, and then code patching the appropriate assembler instructions. We already have good support for the latter.
Above I didn't consider because of the config option.
To Summarize:
1) Dynamically patch PAGE_OFFSET.
2) Remain at three level page table.
. So effectively remain at a single binary for sparc64 without the aid of a config option. Do I have this correct?
thanx,
bob
From: Bob Picco <[email protected]>
Date: Wed, 18 Sep 2013 17:59:59 -0400
Quoted 5 lines
To Summarize:
1) Dynamically patchPAGE_OFFSET.
2) Remain at three level page table.
. So effectively remain at a single binary for sparc64 without the aid of a config option. Do I have this correct?
I don't think remaining at a three level page table for 48 bit physical addresses is feasible.
Let's start one thing at a time, namely dynamic PAGE_OFFSET. I'm working on that right now.
Hi,
David Miller wrote: [Wed Sep 18 2013, 06:32:08PM EDT]
Quoted 9 lines
From: Bob Picco <[email protected]>
Date: Wed, 18 Sep 2013 17:59:59 -0400Quoted 5 lines
To Summarize:
1) Dynamically patchPAGE_OFFSET.
2) Remain at three level page table.
. So effectively remain at a single binary for sparc64 without the aid of a config option. Do I have this correct?
Quoted 5 lines
I don't think remaining at a three level page table for 48 bit physical addresses is feasible.
Let's start one thing at a time, namely dynamic
PAGE_OFFSET. I'm working on that right now.
This UltraSPARC - older sparc64 - issue will arise in other work that is engaged. Would you be willing to suggest an UltraSPARC System which would cover most issues? My desire is to avoid burdening you with my future efforts.
I have this recommendation: http://www.berkcom.com/ for refurbished equipment.
This would be for home office computer rack.
thanx,
bob
From: Bob Picco <[email protected]>
Date: Fri, 20 Sep 2013 11:32:18 -0400
Quoted 4 lines
This UltraSPARC - older sparc64 - issue will arise in other work that is engaged. Would you be willing to suggest an UltraSPARC System which would cover most issues? My desire is to avoid burdening you with my future efforts.
I even don't have all UltraSPARC variants available to me at one time, you're going to have to cover UltraSPARC-I/II, III, T1/T2, T3, T4, and T5.
It's not a burdon for me to take care of this, I know what all of the limitations and parameters are of all of these chips, and I should be the one doing this kind of core infrastructure work anyways.
David Miller wrote: [Fri Sep 20 2013, 12:01:42PM EDT]
Quoted 11 lines
From: Bob Picco <[email protected]>
Date: Fri, 20 Sep 2013 11:32:18 -0400Quoted 4 lines
This UltraSPARC - older sparc64 - issue will arise in other work that is engaged. Would you be willing to suggest an UltraSPARC System which would cover most issues? My desire is to avoid burdening you with my future efforts.
I even don't have all UltraSPARC variants available to me at one time, you're going to have to cover UltraSPARC-I/II, III, T1/T2, T3, T4, and T5.
A ticket has been filed for this hardware. Most Niagara we have with Linux installed. It is the UltraSPARC I/II, III where we are deficient.
Quoted 8 lines
It's not a burdon for me to take care of this, I know what all of the limitations and parameters are of all of these chips, and I should be the one doing this kind of core infrastructure work anyways. --
To unsubscribe from this list: send the line "unsubscribe sparclinux" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html