Skip to contentSkip to conversation

V4 ia64 SPARSEMEM

PATCH In review Bob Picco 5 participants 13 messages Sep 22, 2005 – Sep 27, 2005

lore
Share & tools
lore

Bob Picco

author
#0 lore raw

cover V4 ia64 SPARSEMEM

Tony,

This patchset differs very little from V3. It requires SPARSEMEM EXTREME patches which are in 2.6.14-rcX.

ChangeLog V2:
Jesse's review input has been applied to patches 3 and 4. These were all cosmetic changes.
ChangeLog V3:
This patchset has changes for building ia64 with ARCH_SPARSEMEM_EXTREME config options. The configuration options SECTION_BITS and PHYSICAL_MEMORY_BITS have been removed. I've chosen SECTION_SIZE_BITS to be 30. I'm sure this will be debated but seems adequate for the near term.

This patchset depends on SPARSEMEM EXTREME patch submitted to -mm. The -mm patch makes mem_section a one dimensional array of pointers to mem_sections. The two level layout reduces the memory requirements of SPARSEMEM for very sparse memory but at the expense of an additional shift and load instruction.

The SPARSEMEM function memory_present, which marks sections with memory, is called from paging_init instead of find_memory. This is done because the bootmem allocator in used by SPARSEMEM to allocate the SPARSEMEM section arrays. This is the only code change within this patchset.
ChanageLog V4:
SPARSEMEM EXTREME is now the default SPARSEMEM configuration. Thus the arch doesn't need to request EXTREME. A couple of preprocessor #elif were cleaned up. There was a merge conflict in discontig.c's paging_init function and some minor code rearranging was required.

Thanks to Kame and Yasunori for review and corrections.

Memory models SPARSEMEM+EXTREME, DISCONTIG+VIRTUAL_MEM_MAP, FLATMEM and FLATMEM+VIRTUAL_MEM_MAP were boot tested on rx2600 by me; against -rcX. Yasunori did the same configuration testing on his Tiger4 box with Node emulation for NUMA; against -rc1.

thanks,

bob

#1 lore raw
Quoted 2 lines

This patchset differs very little from V3. It requires SPARSEMEM EXTREME patches which are in 2.6.14-rcX.

Bob,

Can you provide a bit more of a write up on what this change means? Which config options have gone, what the new ones, and what are the rules that determine what's legal. The help options in Kconfig don't seem very helpful here.

I think it would be a good idea to bundle a set of changes to the default config files along with this change ... which means we should have a discussion on what are the sensible defaults out of the new array of options, and which of the non-default options are still popular enough that I should make sure to test build them in my regular scripts. Right now this whole change is a close to being a no-op as none of the default configs enables SPARSEMEM.

So can you throw together a straw-man proposal for each of the default configs (maybe SGI can provide the sn2 version)? I'm assuming here that SPARSEMEM is going to be the preferred choice for some system(s) ... I'm just not sure which.

-Tony

#2 lore raw
In reply to Bob Picco
Quoted 2 lines · Tony

I think it would be a good idea to bundle a set of changes to the default config files along with this change

Especially as this patch breaks the build for bigsur_defconfig.

I think this is because this config is using:

CONFIG_DISCONTIGMEM=y
# CONFIG_NUMA is not set
# CONFIG_VIRTUAL_MEM_MAP is not set

but it might be more convoluted. The failure is early, while building asm-offsets.s

-Tony

author
#3 lore raw
In reply to Bob Picco

luck wrote: [Mon Sep 26 2005, 04:40:36PM EDT]

Quoted 9 lines
Quoted 2 lines

This patchset differs very little from V3. It requires SPARSEMEM EXTREME patches which are in 2.6.14-rcX.

Bob,

Can you provide a bit more of a write up on what this change means? Which config options have gone, what the new ones, and what are the rules that determine what's legal. The help options in Kconfig don't seem very helpful here.

I'm assuming you refer to the config changes in arch/ia64/Kconfig. Well the new ones are documented in mm/Kconfig. If that documentation is inadequate, then I can attempt to supply more.

Quoted 8 lines

I think it would be a good idea to bundle a set of changes to the default config files along with this change ... which means we should have a discussion on what are the sensible defaults out of the new array of options, and which of the non-default options are still popular enough that I should make sure to test build them in my regular scripts. Right now this whole change is a close to being a no-op as none of the default configs enables SPARSEMEM.

This is true and just about universally true for other arches with SPARSEMEM enabled. I took a quick look and only i386 selects SPARSEMEM for NUMA. I couldn't find any default config files with SPARSEMEM. I think the objective was for more soak time before defaulting to SPARSEMEM. Also just a few arches have SPARSEMEM support. Dave Hansen can possibly fill in my gaps.

Quoted 5 lines

So can you throw together a straw-man proposal for each of the default configs (maybe SGI can provide the sn2 version)? I'm assuming here that SPARSEMEM is going to be the preferred choice for some system(s) ... I'm just not sure which.

We could be more aggressive than other arches but still follow the cautious path. I'm reluctant to suggest this but adding a couple of default configuration files with SPARSEMEM might be useful. Perhaps sn2_defconfig_sparsemem and defconfig_sparsemem.

Certainly a more aggressive strategy would be any config with DISCONTIG, VIRTUAL_MEM_MAP and NUMA be change to SPARSEMEM and NUMA. We've only have performance numbers done for a large Altix configuration and rx2600. Tiger, rx2600, Altix and HP Architecture Simulator for NUMA are the only known machines tested with SPARSEMEM. It's doubtful there would be issues but doesn't seem like a good strategy.

Should you think we won't get sufficient test converage without some default configuration files, then I'd suggest the two mentioned about be introduced. Long term should SPARSEMEM become the default and DISCONTIG+VIRTUAL_MEM_MAP be obsoleted then we could remove the config files.

Quoted 2 lines

-Tony

bob

author
#4 lore raw
In reply to Bob Picco

luck wrote: [Mon Sep 26 2005, 06:23:12PM EDT]

Quoted 8 lines · Tony
Quoted 2 lines

I think it would be a good idea to bundle a set of changes to the default config files along with this change

Especially as this patch breaks the build for bigsur_defconfig.

I think this is because this config is using:

Quoted 8 lines

CONFIG_DISCONTIGMEM=y
# CONFIG_NUMA is not set
# CONFIG_VIRTUAL_MEM_MAP is not set

but it might be more convoluted. The failure is early, while building asm-offsets.s

-Tony

Okay. I obviously should have checked all default configs but didn't. Let me look into this. I see partially why this is happening but not the full picture yet. I suspect my conditional change in nodedata.h.

bob

#5 lore raw
In reply to Bob Picco
Quoted 3 lines

I'm assuming you refer to the config changes in arch/ia64/Kconfig. Well the new ones are documented in mm/Kconfig. If that documentation is inadequate, then I can attempt to supply more.

There aren't any words in there about why I might want to use sparsemem ... just a standard version of if you don't know what to pick, go with the old discontig stuff.

Quoted 2 lines

Should you think we won't get sufficient test converage without some default configuration files, then I'd suggest the two mentioned about be introduced.

This is definitely the prime issue ... config space on ia64 is already very fragmented, so testting of some combination of options is rare to non-existant. If SPARSEMEM isn't turned on, then it won't be used by anyone that isn't already reading the lhms list. Coupled with the lack of sales pitch in the Kconfig help files, it looks like this isn't going to be used by anyone!

Quoted 2 lines

Long term should SPARSEMEM become the default and DISCONTIG+VIRTUAL_MEM_MAP be obsoleted then we could remove the config files.

If benchmarks show no difference, then I'll consolidate the configuration options. I still think that VIRTUAL_MEM_MAP has a great deal of elegance to it ... auto-sizing to just about any degree of sparseness, but I think we need to simplify.

-Tony

#6 lore raw
In reply to Bob Picco
Quoted 8 lines · Tony
Quoted 2 lines

Long term should SPARSEMEM become the default and DISCONTIG+VIRTUAL_MEM_MAP be obsoleted then we could remove the config files.

If benchmarks show no difference, then I'll consolidate the configuration options. I still think that VIRTUAL_MEM_MAP has a great deal of elegance to it ... auto-sizing to just about any degree of sparseness, but I think we need to simplify.

Benchmarks cannot prove the absence of a performance difference in *general*, they can only do that for specific tests and workloads. So, unless SPARSEMEM actually performs *better* on some benchmarks (and no worse on others), the proper course seems to be to stick with VIRTUAL_MEM_MAP where SPARSEMEM isn't needed.

I have voice my objection in the past to Bob's suggestion that SPARSEMEM should replace VIRTUAL_MEM_MAP and since then I haven't seen any convincing argument why that should be the case, so I do not understand why Bob keeps bringing that up.

--david
--
Mosberger Consulting LLC, voice/fax: 510-744-9372,
http://www.mosberger-consulting.com/
35706 Runckel Lane, Fremont, CA 94536

#7 lore raw
In reply to Bob Picco

Hi David-san.

Quoted 15 lines
Quoted 8 lines
Quoted 2 lines

Long term should SPARSEMEM become the default and DISCONTIG+VIRTUAL_MEM_MAP be obsoleted then we could remove the config files.

If benchmarks show no difference, then I'll consolidate the configuration options. I still think that VIRTUAL_MEM_MAP has a great deal of elegance to it ... auto-sizing to just about any degree of sparseness, but I think we need to simplify.

Benchmarks cannot prove the absence of a performance difference in *general*, they can only do that for specific tests and workloads. So, unless SPARSEMEM actually performs *better* on some benchmarks (and no worse on others), the proper course seems to be to stick with VIRTUAL_MEM_MAP where SPARSEMEM isn't needed.

Quoted 4 lines

I have voice my objection in the past to Bob's suggestion that SPARSEMEM should replace VIRTUAL_MEM_MAP and since then I haven't seen any convincing argument why that should be the case, so I do not understand why Bob keeps bringing that up.

SPARSEMEM is essential implementation for "memory hotplug" feature now. It gives basis that system can manage to disable/enable a part of memory. And it is not only for ia64 but also for ppc64 and x86_64.

Bye.

Signature · 1 line
Yasunori Goto 
#8 lore raw
In reply to Bob Picco
Quoted 5 lines

We could be more aggressive than other arches but still follow the cautious path. I'm reluctant to suggest this but adding a couple of default configuration files with SPARSEMEM might be useful. Perhaps sn2_defconfig_sparsemem and defconfig_sparsemem.

Although many developers at SGI use SN-specific configuration files, customers run SuSE & RH and use kernels built with the GENERIC config since the kernel binary runs on all hardware vendors platforms. It is important to make the default config file run efficiently on all platforms.

I haven't looked in detail at SPARSEMEM recently - is one config file for ALL hardware platforms easy to do without significantly compromising performance of any individual platform?

Signature · 1 line
Jack
author
#9 lore raw
In reply to Bob Picco

luck wrote: [Mon Sep 26 2005, 08:03:09PM EDT]

Quoted 7 lines
Quoted 3 lines

I'm assuming you refer to the config changes in arch/ia64/Kconfig. Well the new ones are documented in mm/Kconfig. If that documentation is inadequate, then I can attempt to supply more.

There aren't any words in there about why I might want to use sparsemem ... just a standard version of if you don't know what to pick, go with the old discontig stuff.

I can't say there is disagreement with this point. So you'd obviously like more help text in mm/Kconfig?

Quoted 5 lines
Quoted 4 lines

Should you think we won't get sufficient test converage without some default configuration files, then I'd suggest the two mentioned about be introduced.

Quoted 7 lines

This is definitely the prime issue ... config space on ia64 is already very fragmented, so testting of some combination of options is rare to non-existant. If SPARSEMEM isn't turned on, then it won't be used by anyone that isn't already reading the lhms list. Coupled with the lack of sales pitch in the Kconfig help files, it looks like this isn't going to be used by anyone!

Yep. I thought this is what you are after.

Quoted 9 lines
Quoted 3 lines

Long term should SPARSEMEM become the default and DISCONTIG+VIRTUAL_MEM_MAP be obsoleted then we could remove the config files.

If benchmarks show no difference, then I'll consolidate the configuration options. I still think that VIRTUAL_MEM_MAP has a great deal of elegance to it ... auto-sizing to just about any degree of sparseness, but I think we need to simplify.

Well I'm not attempting to be critical of VIRTUAL_MEM_MAP. Obviously it satisfies a solution which was previously unanswered. I'm just hoping to reduce our configuration options. We aren't even remotely close to possibly considering this.

Quoted 1 line

The bigsur problem isn't nodedata.h related. It's in arch/ia64/Kconfig which figures. I should have this resolved some time today after checking the default configs.

Review

Quoted 2 lines
(quoted diff) +0 -2
-Tony
-

bob

author
#10 lore raw
In reply to Bob Picco

Jack Steiner wrote: [Tue Sep 27 2005, 08:45:00AM EDT]

Quoted 11 lines
Quoted 5 lines

We could be more aggressive than other arches but still follow the cautious path. I'm reluctant to suggest this but adding a couple of default configuration files with SPARSEMEM might be useful. Perhaps sn2_defconfig_sparsemem and defconfig_sparsemem.

Although many developers at SGI use SN-specific configuration files, customers run SuSE & RH and use kernels built with the GENERIC config since the kernel binary runs on all hardware vendors platforms. It is important to make the default config file run efficiently on all platforms.

okay.

Quoted 4 lines

I haven't looked in detail at SPARSEMEM recently - is one config file for ALL hardware platforms easy to do without significantly compromising performance of any individual platform?

I don't know whether that can be achieved. We've really done limited performance comparisons. It would have to be investigated.

Quoted 4 lines
Signature · 2 lines
-- 
Jack

bob

author
#11 lore raw
In reply to Bob Picco

Bob Picco wrote: [Tue Sep 27 2005, 10:09:20AM EDT]

Quoted 37 lines

luck wrote: [Mon Sep 26 2005, 08:03:09PM EDT]

Quoted 7 lines
Quoted 3 lines

I'm assuming you refer to the config changes in arch/ia64/Kconfig. Well the new ones are documented in mm/Kconfig. If that documentation is inadequate, then I can attempt to supply more.

There aren't any words in there about why I might want to use sparsemem ... just a standard version of if you don't know what to pick, go with the old discontig stuff.

I can't say there is disagreement with this point. So you'd obviously like more help text in mm/Kconfig?

Quoted 5 lines
Quoted 4 lines

Should you think we won't get sufficient test converage without some default configuration files, then I'd suggest the two mentioned about be introduced.

Quoted 7 lines

This is definitely the prime issue ... config space on ia64 is already very fragmented, so testting of some combination of options is rare to non-existant. If SPARSEMEM isn't turned on, then it won't be used by anyone that isn't already reading the lhms list. Coupled with the lack of sales pitch in the Kconfig help files, it looks like this isn't going to be used by anyone!

Yep. I thought this is what you are after.

Quoted 9 lines
Quoted 3 lines

Long term should SPARSEMEM become the default and DISCONTIG+VIRTUAL_MEM_MAP be obsoleted then we could remove the config files.

If benchmarks show no difference, then I'll consolidate the configuration options. I still think that VIRTUAL_MEM_MAP has a great deal of elegance to it ... auto-sizing to just about any degree of sparseness, but I think we need to simplify.

Well I'm not attempting to be critical of VIRTUAL_MEM_MAP. Obviously it satisfies a solution which was previously unanswered. I'm just hoping to reduce our configuration options. We aren't even remotely close to possibly considering this.

Quoted 1 line
Quoted 6 lines

The bigsur problem isn't nodedata.h related. It's in arch/ia64/Kconfig which figures. I should have this resolved some time today after checking the default configs.

Review

Quoted 2 lines
(quoted diff) +0 -2
-Tony
-

bob

Okay. There was a bug in my Kconfig. All defconfigs and my own memory model configs build correctly and produce the correct config options. Below is a replacement for patch 1/4 which I've included for your config testing. I'll post V5 patchset when we agree on where we want to proceed with config(s).

I'm thinking all we need is sparsemem_defconfig which is defconfig but with the SPARSEMEM memory model.

thanks,

bob

 arch/ia64/Kconfig     |   94 +++++++++++++++++++++++++++++---------------------
 arch/ia64/mm/Makefile |    5 +-
 2 files changed, 58 insertions(+), 41 deletions(-)

Index: linux-2.6.14-rc2/arch/ia64/Kconfig
=================================--- linux-2.6.14-rc2.orig/arch/ia64/Kconfig 2005-09-26 19:16:57.000000000 -0400
+++ linux-2.6.14-rc2/arch/ia64/Kconfig 2005-09-27 14:31:21.000000000 -0400
@@ -63,8 +63,6 @@ config IA64_GENERIC
select ACPI
select NUMA
select ACPI_NUMA
- select VIRTUAL_MEM_MAP
- select DISCONTIGMEM
help
This selects the system type of your hardware. A "generic" kernel will run on any supported IA-64 system. However, if you configure
@@ -176,40 +174,6 @@ config IA64_L1_CACHE_SHIFT
default "6" if ITANIUM

 # align cache-sensitive data to 64 bytes
-config NUMA
-	bool "NUMA support"
-	depends on !IA64_HP_SIM
-	default y if IA64_SGI_SN2
-	select ACPI_NUMA
-	help
-	  Say Y to compile the kernel to support NUMA (Non-Uniform Memory
-	  Access).  This option is for configuring high-end multiprocessor
-	  server systems.  If in doubt, say N.
-
-config VIRTUAL_MEM_MAP
-	bool "Virtual mem map"
-	default y if !IA64_HP_SIM
-	help
-	  Say Y to compile the kernel with support for a virtual mem map.
-	  This code also only takes effect if a memory hole of greater than
-	  1 Gb is found during boot.  You must turn this option on if you
-	  require the DISCONTIGMEM option for your machine. If you are
-	  unsure, say Y.
-
-config HOLES_IN_ZONE
-	bool
-	default y if VIRTUAL_MEM_MAP
-
-config ARCH_DISCONTIGMEM_ENABLE
-	bool "Discontiguous memory support"
-	depends on (IA64_DIG || IA64_SGI_SN2 || IA64_GENERIC || IA64_HP_ZX1 || IA64_HP_ZX1_SWIOTLB) && NUMA && VIRTUAL_MEM_MAP
-	default y if (IA64_SGI_SN2 || IA64_GENERIC) && NUMA
-	help
-	  Say Y to support efficient handling of discontiguous physical memory,
-	  for architectures which are either NUMA (Non-Uniform Memory Access)
-	  or have huge holes in the physical address space for other reasons.
-	  See <file:Documentation/vm/numa> for more.
-
 config IA64_CYCLONE
 	bool "Cyclone (EXA) Time Source support"
 	help
@@ -232,8 +196,10 @@ config IA64_SGI_SN_XP
 	  based on a network adapter and DMA messaging.

config FORCE_MAX_ZONEORDER
- int
- default "18"
+ int "MAX_ORDER (11 - 17)" if !HUGETLB_PAGE
+ range 11 17 if !HUGETLB_PAGE
+ default "17" if HUGETLB_PAGE
+ default "11"

config SMP
bool "Symmetric multi-processing support"
@@ -298,6 +264,58 @@ config PREEMPT

source "mm/Kconfig"

+config ARCH_SELECT_MEMORY_MODEL
+ def_bool y
+
+config ARCH_DISCONTIGMEM_ENABLE
+ def_bool y
+ help

+	  Say Y to support efficient handling of discontiguous physical memory,
+	  for architectures which are either NUMA (Non-Uniform Memory Access)
+	  or have huge holes in the physical address space for other reasons.

+ See <file:Documentation/vm/numa> for more.
+
+config ARCH_FLATMEM_ENABLE
+ def_bool y
+
+config ARCH_SPARSEMEM_ENABLE
+ def_bool y
+ depends on ARCH_DISCONTIGMEM_ENABLE
+
+config ARCH_DISCONTIGMEM_DEFAULT
+ def_bool y if (IA64_SGI_SN2 || IA64_GENERIC || IA64_HP_ZX1 || IA64_HP_ZX1_SWIOTLB)
+ depends on ARCH_DISCONTIGMEM_ENABLE
+
+config NUMA
+ bool "NUMA support"
+ depends on !IA64_HP_SIM && !FLATMEM
+ default y if IA64_SGI_SN2
+ help

+	  Say Y to compile the kernel to support NUMA (Non-Uniform Memory
+	  Access).  This option is for configuring high-end multiprocessor
+	  server systems.  If in doubt, say N.

+
+# VIRTUAL_MEM_MAP and FLAT_NODE_MEM_MAP are functionally equivalent. +# VIRTUAL_MEM_MAP has been retained for historical reasons. +config VIRTUAL_MEM_MAP
+ bool "Virtual mem map"
+ depends on !SPARSEMEM
+ default y if !IA64_HP_SIM
+ help

+	  Say Y to compile the kernel with support for a virtual mem map.
+	  This code also only takes effect if a memory hole of greater than
+	  1 Gb is found during boot.  You must turn this option on if you
+	  require the DISCONTIGMEM option for your machine. If you are
+	  unsure, say Y.

+
+config HOLES_IN_ZONE
+ bool
+ default y if VIRTUAL_MEM_MAP
+
+config HAVE_ARCH_EARLY_PFN_TO_NID
+ def_bool y
+ depends on NEED_MULTIPLE_NODES
+
config IA32_SUPPORT
bool "Support for Linux/x86 binaries" help
Index: linux-2.6.14-rc2/arch/ia64/mm/Makefile
=================================--- linux-2.6.14-rc2.orig/arch/ia64/mm/Makefile 2005-09-26 19:16:57.000000000 -0400
+++ linux-2.6.14-rc2/arch/ia64/mm/Makefile 2005-09-26 19:49:27.000000000 -0400
@@ -7,6 +7,5 @@ obj-y := init.o fault.o tlb.o extable.o
obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o
obj-$(CONFIG_NUMA) += numa.o
obj-$(CONFIG_DISCONTIGMEM) += discontig.o
-ifndef CONFIG_DISCONTIGMEM
-obj-y += contig.o
-endif
+obj-$(CONFIG_SPARSEMEM) += discontig.o
+obj-$(CONFIG_FLATMEM) += contig.o

#12 lore raw
In reply to Bob Picco
Quoted 4 lines

Okay. There was a bug in my Kconfig. All defconfigs and my own memory model configs build correctly and produce the correct config options. Below is a replacement for patch 1/4 which I've included for your config testing. I'll post V5 patchset when we agree on where we want to proceed with config(s).

Yup, all my configs now build too (includes some of the distributed ones with CONFIG_SMP turned off). I booted three sample kernels (zx1, generic and generic with s/discontig/sparse/) on rx2620 with nothing odd happening.

Quoted 2 lines

I'm thinking all we need is sparsemem_defconfig which is defconfig but with the SPARSEMEM memory model.

That would do as a starting point ... it gives me an easy hook into my automatic build script to make sure that SPARSEMEM is being test compiled.

Do you have, or can you generate, the full list of all the combinations of memory model related options that Kconfig will accept?

-Tony