Subject: Re: [PATCH 29/33] autonuma: page_autonuma
Reply-To:
In-Reply-To: <0000013a2c223da2-632aa43e-21f8-4abd-a0ba-2e1b49881e3a-000000@email.amazonses.com>
Hi Christoph,
Quoted 11 lines · Christoph Lameter
Quoted 3 lines · Andrea Arcangeli
Move the
autonuma_last_nidfrom the "struct page" to a separatepage_autonumadata structure allocated in the memsection (with sparsemem) or in the pgdat (with flatmem).Note that there is a available word in struct page before the autonuma patches on
x86_64withCONFIG_HAVE_ALIGNED_STRUCT_PAGE.In fact the
page_autonumafills up the structure to nicely fit in one 64 byte cacheline.
Good point indeed.
So we could drop page_autonuma by creating a CONFIG_SLUB=y dependency (AUTONUMA wouldn't be available in the kernel config if SLAB=y, and it also wouldn't be available on 32bit archs but the latter isn't a problem).
I think it's a reasonable alternative to page_autonuma. Certainly it looks more appealing than taking over 16 precious bits from page->flags. There are still pros and cons. I'm neutral on it so more comments would be welcome ;).
Andrea
PS. randomly moved some in Cc over to Bcc as I overflowed the max header allowed on linux-kernel oops!
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: your mail
Quoted 4 lines · Andrea Arcangeli
So we could drop
page_autonumaby creating aCONFIG_SLUB=ydependency (AUTONUMA wouldn't be available in the kernel config if SLAB=y, and it also wouldn't be available on 32bit archs but the latter isn't a problem).
Nope it should depend on page struct alignment. Other kernel subsystems may be depeding on page struct alignment in the future (and some other arches may already have that requirement)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 29/33] autonuma: page_autonuma
Hi Christoph,
Quoted 10 lines · Christoph Lameter
Quoted 4 lines · Andrea Arcangeli
So we could drop
page_autonumaby creating aCONFIG_SLUB=ydependency (AUTONUMA wouldn't be available in the kernel config if SLAB=y, and it also wouldn't be available on 32bit archs but the latter isn't a problem).Nope it should depend on page struct alignment. Other kernel subsystems may be depeding on page struct alignment in the future (and some other arches may already have that requirement)
But currently only SLUB x86 64bit selectsCONFIG_HAVE_ALIGNED_STRUCT_PAGE:
arch/Kconfig:config HAVE_ALIGNED_STRUCT_PAGE
arch/x86/Kconfig: select HAVE_ALIGNED_STRUCT_PAGE if SLUB && !M386 include/linux/mm_types.h: defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE) include/linux/mm_types.h:#ifdef CONFIG_HAVE_ALIGNED_STRUCT_PAGE mm/slub.c: defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE) mm/slub.c: defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE) mm/slub.c: defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
So in practice a dependency on CONFIG_HAVE_ALIGNED_STRUCT_PAGE would still mean the same: only available when SLUB enables it, and only on x86 64bit (ppc64?).
If you mean CONFIG_AUTONUMA=y should select (not depend) on CONFIG_HAVE_ALIGNED_STRUCT_PAGE, that would allow to enable it in all .configs but it would have a worse cons: losing 8bytes per page unconditionally (even when booting on non-NUMA hardware).
The current page_autonuma solution is substantially memory-cheaper than selecting CONFIG_HAVE_ALIGNED_STRUCT_PAGE: it allocates 2bytes per page at boot time but only if booting on real NUMA hardware (without altering the page structure). So to me it looks still quite a decent tradeoff.
Thanks,
Andrea
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 29/33] autonuma: page_autonuma
Quoted 4 lines · Andrea Arcangeli
If you mean
CONFIG_AUTONUMA=yshould select (not depend) onCONFIG_HAVE_ALIGNED_STRUCT_PAGE, that would allow to enable it in all .configs but it would have a worse cons: losing 8bytes per page unconditionally (even when booting on non-NUMA hardware).
I did not say anything like that. Still not convinced that autonuma is worth doing and that it is beneficial given the complexity it adds to the kernel. Just wanted to point out that there is a case to be made for adding another word to the page struct.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 29/33] autonuma: page_autonuma
Hi Christoph,
Quoted 4 lines · Christoph Lameter
I did not say anything like that. Still not convinced that autonuma is worth doing and that it is beneficial given the complexity it adds to the kernel. Just wanted to point out that there is a case to be made for adding another word to the page struct.
You've seen the benchmarks, no other solution that exists today solves all those cases and never showed a regression compared to upstream. Running that much faster is very beneficial in my view.
Expecting the admin of a 2 socket system to use hard bindings manually is unrealistic, even for a 4 socket is unrealistic.
If you've 512 node system well then you can afford to setup everything manually and boot with noautonuma, no argument about that.
About the complexity, well there's no simple solution to an hard problem. The proof comes from the schednuma crowd that is currently copying the AutoNUMA scheduler cpu-follow-memory design at full force as we speak.
Thanks,
Andrea
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>