Skip to contentSkip to conversation

Transparent on-demand memory setup initialization embedded in the (GFP) buddy allocator

RFC Daniel J Blueman 3 participants 4 messages Jun 27, 2013 – Jun 29, 2013

lore
Share & tools
lore
#0 lore raw
Quoted 10 lines · Andrew Morton
Quoted 2 lines · On Wed, 26 Jun 2013 11:22:48 +0200 Ingo Molnar <[email protected]> wrote:

except that on 32 TB
systems we don't spend ~2 hours initializing 8,589,934,592 page heads.

That's about a million a second which is crazy slow - even my prehistoric desktop is 100x faster than that.

Where's all this time actually being spent?

The complexity of a directory-lookup architecture to make the (intrinsically unscalable) cache-coherency protocol scalable gives you a ~1us roundtrip to remote NUMA nodes.

Probably a lot of time is spent in some memsets, and RMW cycles which are setting page bits, which are intrinsically synchronous, so the initialising core can't get to 12 or so outstanding memory transactions.

Since EFI memory ranges have a flag to state if they are zerod (which may be a fair assumption for memory on non-bootstrap processor NUMA nodes), we can probably collapse the RMWs to just writes.

A normal write will require a coherency cycle, then a fetch and a writeback when it's evicted from the cache. For this purpose, non-temporal writes would eliminate the cache line fetch and give a massive increase in bandwidth. We wouldn't even need a store-fence as the initialising core is the only one online.

Daniel

Signature · 2 lines
Daniel J Blueman
Principal Software Engineer, Numascale Asia
#1 lore raw
Quoted 17 lines · Daniel J Blueman
Quoted 2 lines · Andrew Morton

On Wed, 26 Jun 2013 11:22:48 +0200 Ingo Molnar <[email protected]>

Quoted 3 lines · wrote:
Quoted 3 lines

except that on 32 TB
systems we don't spend ~2 hours initializing 8,589,934,592 page

heads.

Quoted 2 lines

That's about a million a second which is crazy slow - even my

prehistoric desktop

Quoted 3 lines

is 100x faster than that.

Where's all this time actually being spent?

The complexity of a directory-lookup architecture to make the (intrinsically unscalable) cache-coherency protocol scalable gives you a ~1us roundtrip to remote NUMA nodes.

Quoted 7 lines

Probably a lot of time is spent in some memsets, and RMW cycles which are setting page bits, which are intrinsically synchronous, so the initialising core can't get to 12 or so outstanding memory transactions.

Since EFI memory ranges have a flag to state if they are zerod (which may be a fair assumption for memory on non-bootstrap processor NUMA nodes), we can probably collapse the RMWs to just writes.

A normal write will require a coherency cycle, then a fetch and a writeback when it's evicted from the cache. For this purpose, non-temporal writes would eliminate the cache line fetch and give a massive increase in bandwidth. We wouldn't even need a store-fence as the initialising core is the only one online.

Daniel

Could you elaborate a bit more? or suggest a specific area to look at?

After some experiments with trying to just set some fields in the struct page directly I haven't been able to produce any improvements. Of course there is lots about the area which I don't have much experience with.

Nate

#2 lore raw
Quoted 35 lines · Nathan Zimmer
Quoted 28 lines · Daniel J Blueman
Quoted 2 lines · Andrew Morton

On Wed, 26 Jun 2013 11:22:48 +0200 Ingo Molnar

Quoted 3 lines · <[email protected]> wrote:
Quoted 3 lines

except that on 32 TB
systems we don't spend ~2 hours initializing 8,589,934,592

page heads.

Quoted 2 lines

That's about a million a second which is crazy slow - even my

prehistoric desktop

Quoted 3 lines

is 100x faster than that.

Where's all this time actually being spent?

The complexity of a directory-lookup architecture to make the (intrinsically unscalable) cache-coherency protocol scalable gives you a ~1us roundtrip to remote NUMA nodes.

Probably a lot of time is spent in some memsets, and RMW cycles which are setting page bits, which are intrinsically synchronous, so the initialising core can't get to 12 or so outstanding memory transactions.

Since EFI memory ranges have a flag to state if they are zerod (which may be a fair assumption for memory on non-bootstrap processor NUMA nodes), we can probably collapse the RMWs to just writes.

Quoted 5 lines

A normal write will require a coherency cycle, then a fetch and a writeback when it's evicted from the cache. For this purpose, non-temporal writes would eliminate the cache line fetch and give a massive increase in bandwidth. We wouldn't even need a store-fence as the initialising core is the only one online.

Quoted 6 lines

Could you elaborate a bit more? or suggest a specific area to look at?

After some experiments with trying to just set some fields in the struct page directly I haven't been able to produce any improvements. Of course there is lots about the area which I don't have much experience with.

Any such improvement will at most be in the 10-20% range.

I'd suggest first concentrating on the 1000-fold boot time initialization speedup that the buddy allocator delayed initialization can offer, and speeding up whatever remains after that stage - in a much more development-friendly environment. (You'll be able to run 'perf record ./calloc-1TB' after bootup and get meaningful results, etc.)

Thanks,

Ingo

#3 lore raw
Quoted 53 lines · Ingo Molnar
Quoted 35 lines · Nathan Zimmer
Quoted 28 lines · Daniel J Blueman
Quoted 2 lines · Andrew Morton

On Wed, 26 Jun 2013 11:22:48 +0200 Ingo Molnar

Quoted 3 lines · <[email protected]> wrote:
Quoted 3 lines

except that on 32 TB
systems we don't spend ~2 hours initializing 8,589,934,592

page heads.

Quoted 2 lines

That's about a million a second which is crazy slow - even my

prehistoric desktop

Quoted 3 lines

is 100x faster than that.

Where's all this time actually being spent?

The complexity of a directory-lookup architecture to make the (intrinsically unscalable) cache-coherency protocol scalable gives you a ~1us roundtrip to remote NUMA nodes.

Probably a lot of time is spent in some memsets, and RMW cycles which are setting page bits, which are intrinsically synchronous, so the initialising core can't get to 12 or so outstanding memory transactions.

Since EFI memory ranges have a flag to state if they are zerod (which may be a fair assumption for memory on non-bootstrap processor NUMA nodes), we can probably collapse the RMWs to just writes.

Quoted 5 lines

A normal write will require a coherency cycle, then a fetch and a writeback when it's evicted from the cache. For this purpose, non-temporal writes would eliminate the cache line fetch and give a massive increase in bandwidth. We wouldn't even need a store-fence as the initialising core is the only one online.

Quoted 6 lines

Could you elaborate a bit more? or suggest a specific area to look at?

After some experiments with trying to just set some fields in the struct page directly I haven't been able to produce any improvements. Of course there is lots about the area which I don't have much experience with.

Any such improvement will at most be in the 10-20% range.

I'd suggest first concentrating on the 1000-fold boot time initialization speedup that the buddy allocator delayed initialization can offer, and speeding up whatever remains after that stage - in a much more development-friendly environment. (You'll be able to run 'perf record ./calloc-1TB' after bootup and get meaningful results, etc.)

Quoted 3 lines

Thanks,

Ingo

I had been focusing on the bigger gains but my attention had been diverted by hope of an easy, alibiet smaller, win.

I have been experimenting with the patch proper, I am just doing 2MB pages for the moment. The improvement is vast, I'll worry about proper numbers once I think I have a fully working patch.

Some progress is being made on the real patch. I think the memory is being set up correctly, On aligned pages setting the up the page as normal plus setting new PG_ flag.

Right now I am trying to sort out free_pages_prepare and free_pages_check.

Thanks,
Nate