Skip to contentSkip to conversation

DISCUSSION Andrea Arcangeli 2 participants 5 messages Oct 4, 2012 – Oct 5, 2012 Memory management

lore
Share & tools
lore
#0 lore raw

Subject: Re: [PATCH 29/33] autonuma: page_autonuma
Reply-To:
In-Reply-To: <0000013a2c223da2-632aa43e-21f8-4abd-a0ba-2e1b49881e3a-000000@email.amazonses.com>

Hi Christoph,

Quoted 11 lines · Christoph Lameter
Quoted 3 lines · Andrea Arcangeli

Move the autonuma_last_nid from the "struct page" to a separate page_autonuma data structure allocated in the memsection (with sparsemem) or in the pgdat (with flatmem).

Note that there is a available word in struct page before the autonuma patches on x86_64 with CONFIG_HAVE_ALIGNED_STRUCT_PAGE.

In fact the page_autonuma fills up the structure to nicely fit in one 64 byte cacheline.

Good point indeed.

So we could drop page_autonuma by creating a CONFIG_SLUB=y dependency (AUTONUMA wouldn't be available in the kernel config if SLAB=y, and it also wouldn't be available on 32bit archs but the latter isn't a problem).

I think it's a reasonable alternative to page_autonuma. Certainly it looks more appealing than taking over 16 precious bits from page->flags. There are still pros and cons. I'm neutral on it so more comments would be welcome ;).

Andrea

PS. randomly moved some in Cc over to Bcc as I overflowed the max header allowed on linux-kernel oops!

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#1 lore raw

Re: your mail

Quoted 4 lines · Andrea Arcangeli

So we could drop page_autonuma by creating a CONFIG_SLUB=y dependency (AUTONUMA wouldn't be available in the kernel config if SLAB=y, and it also wouldn't be available on 32bit archs but the latter isn't a problem).

Nope it should depend on page struct alignment. Other kernel subsystems may be depeding on page struct alignment in the future (and some other arches may already have that requirement)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#2 lore raw

Re: [PATCH 29/33] autonuma: page_autonuma

Hi Christoph,

Quoted 10 lines · Christoph Lameter
Quoted 4 lines · Andrea Arcangeli

So we could drop page_autonuma by creating a CONFIG_SLUB=y dependency (AUTONUMA wouldn't be available in the kernel config if SLAB=y, and it also wouldn't be available on 32bit archs but the latter isn't a problem).

Nope it should depend on page struct alignment. Other kernel subsystems may be depeding on page struct alignment in the future (and some other arches may already have that requirement)

But currently only SLUB x86 64bit selects
CONFIG_HAVE_ALIGNED_STRUCT_PAGE:

arch/Kconfig:config HAVE_ALIGNED_STRUCT_PAGE

arch/x86/Kconfig:       select HAVE_ALIGNED_STRUCT_PAGE if SLUB && !M386
include/linux/mm_types.h:       defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
include/linux/mm_types.h:#ifdef CONFIG_HAVE_ALIGNED_STRUCT_PAGE
mm/slub.c:    defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
mm/slub.c:    defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
mm/slub.c:    defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)

So in practice a dependency on CONFIG_HAVE_ALIGNED_STRUCT_PAGE would still mean the same: only available when SLUB enables it, and only on x86 64bit (ppc64?).

If you mean CONFIG_AUTONUMA=y should select (not depend) on CONFIG_HAVE_ALIGNED_STRUCT_PAGE, that would allow to enable it in all .configs but it would have a worse cons: losing 8bytes per page unconditionally (even when booting on non-NUMA hardware).

The current page_autonuma solution is substantially memory-cheaper than selecting CONFIG_HAVE_ALIGNED_STRUCT_PAGE: it allocates 2bytes per page at boot time but only if booting on real NUMA hardware (without altering the page structure). So to me it looks still quite a decent tradeoff.

Thanks,
Andrea

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#3 lore raw

Re: [PATCH 29/33] autonuma: page_autonuma

Quoted 4 lines · Andrea Arcangeli

If you mean CONFIG_AUTONUMA=y should select (not depend) on CONFIG_HAVE_ALIGNED_STRUCT_PAGE, that would allow to enable it in all .configs but it would have a worse cons: losing 8bytes per page unconditionally (even when booting on non-NUMA hardware).

I did not say anything like that. Still not convinced that autonuma is worth doing and that it is beneficial given the complexity it adds to the kernel. Just wanted to point out that there is a case to be made for adding another word to the page struct.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#4 lore raw

Re: [PATCH 29/33] autonuma: page_autonuma

Hi Christoph,

Quoted 4 lines · Christoph Lameter

I did not say anything like that. Still not convinced that autonuma is worth doing and that it is beneficial given the complexity it adds to the kernel. Just wanted to point out that there is a case to be made for adding another word to the page struct.

You've seen the benchmarks, no other solution that exists today solves all those cases and never showed a regression compared to upstream. Running that much faster is very beneficial in my view.

Expecting the admin of a 2 socket system to use hard bindings manually is unrealistic, even for a 4 socket is unrealistic.

If you've 512 node system well then you can afford to setup everything manually and boot with noautonuma, no argument about that.

About the complexity, well there's no simple solution to an hard problem. The proof comes from the schednuma crowd that is currently copying the AutoNUMA scheduler cpu-follow-memory design at full force as we speak.

Thanks,
Andrea

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>