sparcemem or discontig?

Hi,

I'm working on a 32bit mips 1 cpu platform that has 2 large banks of memory that are discontiguous in the physical address space. That is to say there is a large hole in between them. We are using the 2.6.24 kernel. In order to save memory on page tables, we'd like to employ the use of either the CONFIG_SPARCEMEM, or CONFIG_DISCONTIG configuration options, but I'm a bit unsure which I should use.

My understanding is that SPARCEMEM is the way of the future, and since I don't really have a NUMA machine, maybe sparcemem is more appropriate, yes? On the other hand I can't find much info about how it works or how to add support for it on an architecture that has here-to-fore not supported that option.

Is there anywhere that there is a paper or rfp that describes how the spacemem (or discontig) features work (and/or the differences between then)? Has anyone out there done this for MIPS32? has anyone had experience with adding support for either sparcemem or discontig on an arch before , that could give me a their thoughts the process or "gotchas"?

thanks
Mike

- - - - - Cisco - - - - -
This e-mail and any attachments may contain information which is confidential, proprietary, privileged or otherwise protected by law. The information is solely intended for the named addressee (or a person responsible for delivering it to the addressee). If you are not the intended recipient of this message, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete it from your computer.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I think you're talking about sparsemem. :)

My opinion is that NUMA and DISCONTIG are too intertwined to be useful apart from the other. I use sparsemem on my non-NUMA 2 CPU laptop since it has a 1GB hole. It is *possible* to use DISCONTIG without NUMA, and I'm sure people use it this way, but I just personally think it is a bit of a pain.

Basically, to add sparsemem support for an architecture, you need a header like these:

dave@nimitz:~/lse/linux/2.5/linux-2.6.git$ find | grep sparse | xargs
grep -c '^.*$'
./include/asm-powerpc/sparsemem.h:32
./include/asm-x86/sparsemem.h:34
./include/asm-sh/sparsemem.h:16
./include/asm-mips/sparsemem.h:14
./include/asm-ia64/sparsemem.h:20
./include/asm-s390/sparsemem.h:18
./include/asm-arm/sparsemem.h:10

These are generally pretty darn small (the largest is 34 lines). You also need to tweak some things in your per-arch Kconfig. ARM looks like a pretty simple use of sparsemem. You might want to start with what they've done. We tried really, really hard to make it easy to add to new architectures.

Feel free to cc me and Andy (cc'd) on the patches that you come up with. I'd be happy to sanity check them for you. If *you* want to document the process for the next guy, I'm sure we'd be able to find some spot in Documentation/ so the next guy has an easier time. :)

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Always interested in new users of sparsemem. Cc me :).

-apw

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Pardon my ignorance, but is sparcemem independent of the bootmem allocator?

We also use highmem. I noticed that all of our kmap and kmap_atomic code is located in the arch/mips directory. Is the sparcemem also independent of that? should I expect that I will have to make some changes in that...

I'm happy to write a how to for sparcemem.

- - - - - Cisco - - - - -
This e-mail and any attachments may contain information which is confidential, proprietary, privileged or otherwise protected by law. The information is solely intended for the named addressee (or a person responsible for delivering it to the addressee). If you are not the intended recipient of this message, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete it from your computer.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Yes, it really don't have much to do with bootmem.

No, I don't think you'll need any changes to those.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi Andy and Dave,

I turned on sparsemem as you described before. I am crashing in the mem_init() function when I try a call to pfn_to_page().

I've noticed that that macro uses the sparsemem macro __pfn_to_section(pfn) and that intern calls __nr_to_section(nr). That finally looks at the mem_section[] variable. well.. this returns NULL since it seems as though my mem_section[] array looks to be not initialized correctly.

QUESTION: where does this array get initialized. I've looked through the code and can't seem to see how that is initialized.

recall I'm using mips32 processor, but I've looked in all the processors. it seems as though sparse_init() and memory present() both use __nr_to_section() and thus would require mem_section[] to be set up already.

thanks for your help
Mike

- - - - - Cisco - - - - -
This e-mail and any attachments may contain information which is confidential, proprietary, privileged or otherwise protected by law. The information is solely intended for the named addressee (or a person responsible for delivering it to the addressee). If you are not the intended recipient of this message, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete it from your computer.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

My first guess is that you're missing a call to sparse_init() in your architecture-specific code. On x86_32, we do that in paging_init(), just before zone_sizes_init() (arch/x86/mm/init_32.c).

Before you call this, you'll also have to call memory_present(...) on the memory that you do have. But, you should probably already have done that.

__nr_to_section() is special. It takes a section number and just gives you back the 'struct mem_section'. It doesn't actually look into that mem_section and make sure it is valid, it just locates the data structure.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

yes I call sparse_init()..... My code might be not calling memory_present for some of the memory I'm looking at that right now.

still that code is strange to me:

-------------code
static inline struct mem_section *__nr_to_section(unsigned long nr)

{
       if (!mem_section[SECTION_NR_TO_ROOT(nr)])
               return NULL;
       return &mem_section[SECTION_NR_TO_ROOT(nr)][nr & SECTION_ROOT_MASK];
}

--------------

on the first line of the function above, what does it mean "if not <struct>"? seems that returns true if the contents of that struct is "0"... but either way, doesn't that have to be initialized to something before it is called from memory_present()?

mike

- - - - - Cisco - - - - -
This e-mail and any attachments may contain information which is confidential, proprietary, privileged or otherwise protected by law. The information is solely intended for the named addressee (or a person responsible for delivering it to the addressee). If you are not the intended recipient of this message, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete it from your computer.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Yeah, this is confusing code. The goal here was to not have any #ifdefs for the normal vs. extreme cases. In the !EXTREME case, the mem_section[] array is statically allocated and that first check gets optimized out completely. We check for it, but the compiler kills that check.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>