Skip to contentSkip to conversation

sparc64 page table

PATCH In review Bob Picco 2 participants 8 messages Sep 16, 2013 – Sep 22, 2013

lore
Share & tools
lore

Bob Picco

author
#0 lore raw

cover sparc64 page table

From: bob picco <[email protected]>

We encountered an issue with the current sparc64 page table implementation. The T5 memory controller is different than T4. Its coarse grain interleave factor causes the unsigned types of pgd_t and pmd_t to be insufficient. We managed to partially resolve the issue by taking the physical address and storing exactly the physical page frame within the pgd_t and pmd_t. This would be deficient for a large memory configuration with all sockets populated. This memblockÞbug snippet of T5-2 is for a 256Gb configured machine:
MEMBLOCK configuration:
memory size = 0x3fcf214000
memory.cnt = 0x5

 memory[0x0]    [0x00000024400000-0x00001ff36e1fff], 0x1fcf2e2000 bytes
 memory[0x1]    [0x00001ff36e8000-0x00001ff36e9fff], 0x2000 bytes
 memory[0x2]    [0x00001ff36ee000-0x00001ff36effff], 0x2000 bytes
 memory[0x3]    [0x00080000000000-0x00081ffff23fff], 0x1ffff24000 bytes
 memory[0x4]    [0x00081ffff34000-0x00081ffff3dfff], 0xa000 bytes

reserved.cnt = 0x1
reserved[0x0] [0x00080000000000-0x00080000cd6980], 0xcd6981 bytes
. This is only a two socket machine and minimally memory configured.

The second platform to encounter this issue is Fujitsu's Athena. Athena physical memory is mapped at the most significant bits. You can observe this from this snippet of boot up with memblockÞbug
MEMBLOCK configuration:
memory size = 0xff8f1fe000
memory.cnt = 0xb

 memory[0x0]    [0x00780070400000-0x0078007f69dfff], 0xf29e000 bytes
 memory[0x1]    [0x0078007fffe000-0x00781fffffffff], 0x1f80002000 bytes
 memory[0x2]    [0x00790000000000-0x00791fffffffff], 0x2000000000 bytes
 memory[0x3]    [0x007a0000000000-0x007a1fffffffff], 0x2000000000 bytes
 memory[0x4]    [0x007b0000000000-0x007b1fffffffff], 0x2000000000 bytes
 memory[0x5]    [0x007c0000000000-0x007c1fffffffff], 0x2000000000 bytes
 memory[0x6]    [0x007d0000000000-0x007d1fffffffff], 0x2000000000 bytes
 memory[0x7]    [0x007e0000000000-0x007e1fffffffff], 0x2000000000 bytes
 memory[0x8]    [0x007f0000000000-0x007f1ffff41fff], 0x1ffff42000 bytes
 memory[0x9]    [0x007f1ffff5a000-0x007f1ffff69fff], 0x10000 bytes
 memory[0xa]    [0x007f1ffffd6000-0x007f1ffffe1fff], 0xc000 bytes

reserved.cnt = 0x2
reserved[0x0] [0x007f0000000000-0x007f00010ca900], 0x10ca901 bytes reserved[0x1] [0x007f0004000000-0x007f0008d4e462], 0x4d4e463 bytes
.

There are several ways to approach a solution for this. We chose to introduce a four level page table scheme. This scheme promotes pgd_t and pmd_t to unsigned long. The pud folding isn't used and the pud_t is also unsigned long.

There are penalties for a four level scheme. A small memory increase because of the addition of a pud. TSB misses will cost another memory load because of the new level. We've yet to observe a negative ramification from the fourth level.

There is a configuration choice between the four level page table scheme and the current three level page table scheme. It is our hope that the exisiting three level scheme has been left unchanged functionally. This can be viewed like sparc64 THP which impacted the existing three level page table scheme and is configuration selectable.

There could potentially be further code consolidation by keying off of pud folding or not. I feel it is better to leave the two levels as independent as possible for now.

My skill level for naming leaves much to be desired at times. I'm certainly not strongly attached to PGTABLE_LEVEL4.

thanx,

bob

bob picco (4):
sparc64 expand linear mapping region
sparc64 move three level page table scheme
sparc64 four level page table support
sparc64 kconfig four level page table

 arch/sparc/Kconfig                       |    9 ++
 arch/sparc/include/asm/page_64.h         |   20 +++-
 arch/sparc/include/asm/page_64_lvl3.h    |   15 +++
 arch/sparc/include/asm/page_64_lvl4.h    |   21 ++++
 arch/sparc/include/asm/pgalloc_64.h      |   16 +++
 arch/sparc/include/asm/pgtable_64.h      |  135 +++------------------
 arch/sparc/include/asm/pgtable_64_lvl3.h |  118 ++++++++++++++++++
 arch/sparc/include/asm/pgtable_64_lvl4.h |  191 ++++++++++++++++++++++++++++++
 arch/sparc/include/asm/sparsemem.h       |    5 +
 arch/sparc/include/asm/tsb.h             |   55 +++++++++
 arch/sparc/kernel/ktlb.S                 |   12 +-
 arch/sparc/kernel/smp_64.c               |    7 ++
 arch/sparc/mm/init_64.c                  |   36 ++++--
 arch/sparc/mm/init_64.h                  |    4 +
 14 files changed, 508 insertions(+), 136 deletions(-)
 create mode 100644 arch/sparc/include/asm/page_64_lvl3.h
 create mode 100644 arch/sparc/include/asm/page_64_lvl4.h
 create mode 100644 arch/sparc/include/asm/pgtable_64_lvl3.h
 create mode 100644 arch/sparc/include/asm/pgtable_64_lvl4.h
Signature · 1 line
1.7.10.4
#1 lore raw

Every distribution is going to turn on the most capable configuration.

So having a choice between 3-level and 4-level page tables isn't going to buy anything.

I'd rather find a solution that allows us to stay at 3-level page tables and avoid all of the extra cost, as any change here is going to make every single TLB miss more expensive.

#2 lore raw
In reply to Bob Picco

From: Bob Picco <[email protected]>
Date: Mon, 16 Sep 2013 09:46:35 -0400

Quoted 5 lines

There is a configuration choice between the four level page table scheme and the current three level page table scheme. It is our hope that the exisiting three level scheme has been left unchanged functionally. This can be viewed like sparc64 THP which impacted the existing three level page table scheme and is configuration selectable.

So again this is unacceptable.

Even worse, the 4-level page table setting creates a kernel that absolutely does not work on older chips.

The problem is the PAGE_OFFSET selections. You can't use a PAGE_OFFSET value that tries to use parts of the virtual address space which is not supported on UltraSPARC-I et al.

So, in particular, PAGE_OFFSET will need to be determined dynamically at run time. Yes, this means looking at the appropriate cpu properties to see how much virtual address space is supported, and then code patching the appropriate assembler instructions. We already have good support for the latter.

author
#3 lore raw
In reply to Bob Picco

Hi,
David Miller wrote: [Wed Sep 18 2013, 12:28:09PM EDT]

Quoted 10 lines

From: Bob Picco <[email protected]>
Date: Mon, 16 Sep 2013 09:46:35 -0400

Quoted 5 lines

There is a configuration choice between the four level page table scheme and the current three level page table scheme. It is our hope that the exisiting three level scheme has been left unchanged functionally. This can be viewed like sparc64 THP which impacted the existing three level page table scheme and is configuration selectable.

So again this is unacceptable.

I understand.

Quoted 7 lines

Even worse, the 4-level page table setting creates a kernel that absolutely does not work on older chips.

The problem is the PAGE_OFFSET selections. You can't use a PAGE_OFFSET value that tries to use parts of the virtual address space which is not supported on UltraSPARC-I et al.

This I'm aware of. Otherwise why would it remain where it is at for S3 core.

Quoted 6 lines

So, in particular, PAGE_OFFSET will need to be determined dynamically at run time. Yes, this means looking at the appropriate cpu properties to see how much virtual address space is supported, and then code patching the appropriate assembler instructions. We already have good support for the latter.

Above I didn't consider because of the config option.

To Summarize:
1) Dynamically patch PAGE_OFFSET.
2) Remain at three level page table.
. So effectively remain at a single binary for sparc64 without the aid of a config option. Do I have this correct?

thanx,

bob

#4 lore raw
In reply to Bob Picco

From: Bob Picco <[email protected]>
Date: Wed, 18 Sep 2013 17:59:59 -0400

Quoted 5 lines

To Summarize:
1) Dynamically patch PAGE_OFFSET.
2) Remain at three level page table.
. So effectively remain at a single binary for sparc64 without the aid of a config option. Do I have this correct?

I don't think remaining at a three level page table for 48 bit physical addresses is feasible.

Let's start one thing at a time, namely dynamic PAGE_OFFSET. I'm working on that right now.

author
#5 lore raw
In reply to Bob Picco

Hi,
David Miller wrote: [Wed Sep 18 2013, 06:32:08PM EDT]

Quoted 9 lines

From: Bob Picco <[email protected]>
Date: Wed, 18 Sep 2013 17:59:59 -0400

Quoted 5 lines

To Summarize:
1) Dynamically patch PAGE_OFFSET.
2) Remain at three level page table.
. So effectively remain at a single binary for sparc64 without the aid of a config option. Do I have this correct?

Quoted 5 lines

I don't think remaining at a three level page table for 48 bit physical addresses is feasible.

Let's start one thing at a time, namely dynamic PAGE_OFFSET. I'm working on that right now.

This UltraSPARC - older sparc64 - issue will arise in other work that is engaged. Would you be willing to suggest an UltraSPARC System which would cover most issues? My desire is to avoid burdening you with my future efforts.

I have this recommendation: http://www.berkcom.com/ for refurbished equipment.

This would be for home office computer rack.

thanx,

bob

#6 lore raw
In reply to Bob Picco

From: Bob Picco <[email protected]>
Date: Fri, 20 Sep 2013 11:32:18 -0400

Quoted 4 lines

This UltraSPARC - older sparc64 - issue will arise in other work that is engaged. Would you be willing to suggest an UltraSPARC System which would cover most issues? My desire is to avoid burdening you with my future efforts.

I even don't have all UltraSPARC variants available to me at one time, you're going to have to cover UltraSPARC-I/II, III, T1/T2, T3, T4, and T5.

It's not a burdon for me to take care of this, I know what all of the limitations and parameters are of all of these chips, and I should be the one doing this kind of core infrastructure work anyways.

author
#7 lore raw
In reply to Bob Picco

David Miller wrote: [Fri Sep 20 2013, 12:01:42PM EDT]

Quoted 11 lines

From: Bob Picco <[email protected]>
Date: Fri, 20 Sep 2013 11:32:18 -0400

Quoted 4 lines

This UltraSPARC - older sparc64 - issue will arise in other work that is engaged. Would you be willing to suggest an UltraSPARC System which would cover most issues? My desire is to avoid burdening you with my future efforts.

I even don't have all UltraSPARC variants available to me at one time, you're going to have to cover UltraSPARC-I/II, III, T1/T2, T3, T4, and T5.

A ticket has been filed for this hardware. Most Niagara we have with Linux installed. It is the UltraSPARC I/II, III where we are deficient.

Quoted 8 lines

It's not a burdon for me to take care of this, I know what all of the limitations and parameters are of all of these chips, and I should be the one doing this kind of core infrastructure work anyways. --
To unsubscribe from this list: send the line "unsubscribe sparclinux" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html