Skip to contentSkip to conversation

Deprecate SPARSEMEM and have only SPARSEMEM_VMEMMAP

LSF/MM/BFP TOPIC Oscar Salvador 3 participants 6 messages May 10, 2024 – May 13, 2024 Memory management

lore
Share & tools
lore
#0 lore raw

Hi all,

I would like to discuss the following topic in the LSFMM.

We have SPARSEMEM memory model and SPARSEMEM_VMEMMAP, where the only difference between the two of them is that the latter allocates a virtual chunk to represent the memmap array, which speeds operations like page_to_pfn/pfn_to_page/nth_page/folio_page_idx.

page->flags layout would also be changed.

Currently we have (extracted from include/linux/page-flags-layout.h):

/*
* page->flags layout:
* No sparsemem or sparsemem vmemmap: | NODE | ZONE |
* ... | FLAGS |
* " plus space for last_cpupid: | NODE | ZONE |
* LAST_CPUPID ... | FLAGS |
* classic sparse with space for node:| SECTION | NODE | ZONE |
* ... | FLAGS |
* " plus space for last_cpupid: | SECTION | NODE | ZONE |
* LAST_CPUPID ... | FLAGS |
* classic sparse no space for node: | SECTION | ZONE | ...
* | FLAGS |
*/

The last three could be gone.

Also, by getting rid of SPARSEMEM we would also get rid of a non-trivial amount of code.

I did some research on which arches use CONFIG_SPARSE_MEMMAP/VMEMMAP or none(using flatmem?).

SPARSE_MEMMAP SPARSE_VMEMMAP
arc
arm X
arm64 X X
csky
hexagon
loongarch X X
m68k
microblaze
mips X
nios2
openrisc
parisc

 powerpc          X                     X
 riscv            X                     X
 s390             X                     X
 sh               X
 sparc            X                     X

um
x86 X X
xtensa

arm, mips, parisc and sh operate with SPARSE_MEMMAP but are lacking code for SPARSE_VMEMMAP.
I think these archs should be the first thing to focus on, to see if we can make them work on SPARSE_VMEMMAP.
If we can, and when all arches can run on SPARSE_VMEMMAP, we can think of killing SPARSE_MEMMAP.

This is not for free, of course.
There is a certain memory overhead because we have to allocate the virtual memmap for each section.

Taking MIPS as an example:

- SECTION_SIZE_BITS 28 (256MB)
PAGE_SIZE_4KB: 4MB per section
PAGE_SIZE_16KB: 1MB per section
PAGE_SIZE_64KB: 256KB per section
- SECTION_SIZE_BITS 29 (512MB)
PAGE_SIZE_64KB: 512KB per section

e.g: When the section size is 256MB and we operate on 4KB page size, we spend
4MB for the virtual memmap array per section.
(65536 pages per section * sizeof(struct page)) = 4MB

Ideally, we can discuss:

1) whether it makes sense to definitely switch over to SPARSE_VMEMMAP
2) if yes, how can this be arranged. Implementing all the code and giving a grace period?
3) if not, why? and what can be done so we can proceed.

Thanks

Signature · 2 lines
Oscar Salvador
SUSE Labs
#1 lore raw
Quoted 25 lines · Oscar Salvador

I did some research on which arches use CONFIG_SPARSE_MEMMAP/VMEMMAP or none(using flatmem?).

SPARSE_MEMMAP SPARSE_VMEMMAP
arc
arm X
arm64 X X
csky
hexagon
loongarch X X
m68k
microblaze
mips X
nios2
openrisc
parisc

 powerpc          X                     X
 riscv            X                     X
 s390             X                     X
 sh               X
 sparc            X                     X

um
x86 X X
xtensa

Quoted 6 lines

arm, mips, parisc and sh operate with SPARSE_MEMMAP but are lacking code for SPARSE_VMEMMAP.
I think these archs should be the first thing to focus on, to see if we can make them work on SPARSE_VMEMMAP.
If we can, and when all arches can run on SPARSE_VMEMMAP, we can think of killing SPARSE_MEMMAP.

I'm a little concerned about having this conversation without the affected architecture maintainers in the room. However, I can speak to PA-RISC.

Early models have a dense memory layout and we need not be concerned with them. I'm not quite sure about the PA-7200 to PA-8500 ccio based machines, would need to do some research. For the PA-8500+ astro based machines, the 256MB that would be in the range 3.75GB to 4GB is relocated to 67.75-68GB to leave space for PCI mmio. So if you have a machine with 8GB of memory (fairly typical for a J6000 machine), you'd have three ranges of memory:

0-3.75GB
4-8GB
67.75-68GB

and I'd like to see an analysis of how laying out memmap would differ for those machines.

The PA-8800+ pluto chipset does something similar, except it supports more memory and more PCI mmio, so a 32GB rp3440 would have a memmap:

0-3GB
4-32GB
259-260GB

I think this would actually work better as three zones rather than three sections. It might match quite well with the TAO proposal.

#2 lore raw
Quoted 3 lines · Matthew Wilcox

I'm a little concerned about having this conversation without the affected architecture maintainers in the room. However, I can speak to PA-RISC.

Yes, having the architecture maintainers would be great.

Quoted 8 lines

Early models have a dense memory layout and we need not be concerned with them. I'm not quite sure about the PA-7200 to PA-8500 ccio based machines, would need to do some research. For the PA-8500+ astro based machines, the 256MB that would be in the range 3.75GB to 4GB is relocated to 67.75-68GB to leave space for PCI mmio. So if you have a machine with 8GB of memory (fairly typical for a J6000 machine), you'd have three ranges of memory:

Quoted 6 lines

0-3.75GB
4-8GB
67.75-68GB

and I'd like to see an analysis of how laying out memmap would differ for those machines.

Maybe Mike can prove me wrong, but I assume that memblock will report the above ranges as memory, and the 3.75GB to 4GB as somewhat reserved. Then, we only mark those sections falling within the ranges reported as having memory by memblock as present, and we only populate the memmap for present sections.

So, those ranges from above will be represented by present sections and hence with the vmemmap populated, and anything that falls off will not.

I am not sure if I got your concern right though.

Quoted 10 lines

The PA-8800+ pluto chipset does something similar, except it supports more memory and more PCI mmio, so a 32GB rp3440 would have a memmap:

0-3GB
4-32GB
259-260GB

I think this would actually work better as three zones rather than three sections. It might match quite well with the TAO proposal.

But those are not really sections, are they? But rather memory ranges. Checking the code from parisc, sections are at 128MB granularity. So you will have

0-3GB ->     0    .. 23   section (memmap populated)
4-32GB ->    31   .. 255  section (memmap populated)

259-260GB -> 2071 .. 2079 section (memmap populated)

Signature · 2 lines
Oscar Salvador
SUSE Labs
#3 lore raw
Quoted 3 lines · Oscar Salvador
0-3GB ->     0    .. 23   section (memmap populated)
4-32GB ->    31   .. 255  section (memmap populated)

259-260GB -> 2071 .. 2079 section (memmap populated)

And just to add more info, considering a 128MB section granularity (which seems to be what paric has) a PAGE_SIZE of 4KB (not really sure which page size parisc can deal with), PAGES_PER_SECTION then being 32768 and the vmemmap range starting at 0xffffea0000000000, we would have the following vmemmap layout:

0-3GB     ->  0xffffea0000000000 - 0xffffea0003000000
4-32GB    ->  0xffffea0004000000 - 0xffffea0007800000
259-260GB ->  0xffffea0023e00000 - 0xffffea0024e00000

I am not sure though whether we can hit any limitation when it comes to the vmemmap space we have available.

Signature · 2 lines
Oscar Salvador
SUSE Labs
#4 lore raw
In reply to Oscar Salvador
Quoted 19 lines · Oscar Salvador
Quoted 3 lines · Matthew Wilcox

I'm a little concerned about having this conversation without the affected architecture maintainers in the room. However, I can speak to PA-RISC.

Yes, having the architecture maintainers would be great.

Quoted 12 lines

Early models have a dense memory layout and we need not be concerned with them. I'm not quite sure about the PA-7200 to PA-8500 ccio based machines, would need to do some research. For the PA-8500+ astro based machines, the 256MB that would be in the range 3.75GB to 4GB is relocated to 67.75-68GB to leave space for PCI mmio. So if you have a machine with 8GB of memory (fairly typical for a J6000 machine), you'd have three ranges of memory:

0-3.75GB
4-8GB
67.75-68GB

Quoted 5 lines
Quoted 2 lines

and I'd like to see an analysis of how laying out memmap would differ for those machines.

Maybe Mike can prove me wrong, but I assume that memblock will report the above ranges as memory, and the 3.75GB to 4GB as somewhat reserved.

The populated ranges will be reported as memory and it seems that there just will be a hole at the 3.75GB-4GB range. Not that it's important from the sections layout perspective.

Quoted 4 lines
Signature · 2 lines
Oscar Salvador
SUSE Labs
Signature · 2 lines
Sincerely yours,
Mike.
#5 lore raw
In reply to Oscar Salvador
Quoted 8 lines · Oscar Salvador

Hi all,

I would like to discuss the following topic in the LSFMM.

We have SPARSEMEM memory model and SPARSEMEM_VMEMMAP, where the only difference between the two of them is that the latter allocates a virtual chunk to represent the memmap array, which speeds operations like page_to_pfn/pfn_to_page/nth_page/folio_page_idx.

...

Quoted 2 lines

I did some research on which arches use CONFIG_SPARSE_MEMMAP/VMEMMAP or none(using flatmem?).

FLATMEM it is.

Quoted 18 lines

SPARSE_MEMMAP SPARSE_VMEMMAP
arc
arm X
arm64 X X
csky
hexagon
loongarch X X
m68k
microblaze
mips X
nios2
openrisc
parisc

 powerpc          X                     X
 riscv            X                     X
 s390             X                     X
 sh               X
 sparc            X                     X
Quoted 6 lines

um
x86 X X
xtensa

arm, mips, parisc and sh operate with SPARSE_MEMMAP but are lacking code for SPARSE_VMEMMAP.

arm can use "sparse flatmem" in the sense that it can free unused parts of the memory map so FLATMEM does not have memory overhead with sparsely located memory banks. So possible we can just drop SPARSEMEM on arm.

But I agree with Matthew that for this discussion we'd need arch maintainers to participate.

Quoted 6 lines

Thanks

Signature · 2 lines
Oscar Salvador
SUSE Labs
Signature · 2 lines
Sincerely yours,
Mike.