NUMA? bisected performance regression 3.11->3.12

https://github.com/antonblanchard/will-it-scale/blob/master/tests/open1.c

    17.65%   +3.47%  [kernel.kallsyms]  [k] _raw_spin_lock_irqsave           
    13.80%   -0.31%  [kernel.kallsyms]  [k] _raw_spin_lock                   
     7.21%   -0.51%  [unknown]          [.] 0x00007f7849058640               
     3.43%   +0.15%  [kernel.kallsyms]  [k] setup_object                     
     2.99%   -0.31%  [kernel.kallsyms]  [k] file_free_rcu                    
     2.71%   -0.13%  [kernel.kallsyms]  [k] rcu_process_callbacks            
     2.26%   -0.09%  [kernel.kallsyms]  [k] get_empty_filp                   

     2.06%   -0.09%  [kernel.kallsyms]  [k] kmem_cache_alloc                 
     1.65%   -0.08%  [kernel.kallsyms]  [k] link_path_walk                   
     1.53%   -0.08%  [kernel.kallsyms]  [k] memset                           
     1.46%   -0.09%  [kernel.kallsyms]  [k] do_dentry_open                   
     1.44%   -0.04%  [kernel.kallsyms]  [k] __d_lookup_rcu                   
     1.27%   -0.04%  [kernel.kallsyms]  [k] do_last                          
     1.18%   -0.04%  [kernel.kallsyms]  [k] ext4_release_file                
     1.16%   -0.04%  [kernel.kallsyms]  [k] __call_rcu.constprop.11          

Hey Johannes,

I'm running an open/close microbenchmark from the will-it-scale set:

Quoted 1 line

https://github.com/antonblanchard/will-it-scale/blob/master/tests/open1.c

I was seeing some weird symptoms on 3.12 vs 3.11. The throughput in that test was going from down from 50 million to 35 million.

The profiles show an increase in cpu time in _raw_spin_lock_irq. The profiles pointed to slub code that hasn't been touched in quite a while.
I bisected it down to:

81c0a2bb515fd4daae8cab64352877480792b515 is the first bad commit commit 81c0a2bb515fd4daae8cab64352877480792b515

Author: Johannes Weiner <[email protected]>
Date:   Wed Sep 11 14:20:47 2013 -0700

Which also seems a bit weird, but I've tested with this and its preceding commit enough times to be fairly sure that I did it right.

__slab_free() and free_one_page() both seem to be spending more time spinning on their respective spinlocks, even though the throughput went down and we should have been doing fewer actual allocations/frees. The best explanation for this would be if CPUs are tending to go after and contending for remote cachelines more often once this patch is applied.

Any ideas?

It's a 8-socket/160-thread (one NUMA node per socket) system that is not under memory pressure during the test. The latencies are also such that vm.zone_reclaim_mode=0.

Raw perf profiles and .config are in here:
http://www.sr71.net/~dave/intel/201311-wisregress0/

Here's a chunk of the 'perf diff':

Quoted 15 lines

    17.65%   +3.47%  [kernel.kallsyms]  [k] _raw_spin_lock_irqsave           
    13.80%   -0.31%  [kernel.kallsyms]  [k] _raw_spin_lock                   
     7.21%   -0.51%  [unknown]          [.] 0x00007f7849058640               
     3.43%   +0.15%  [kernel.kallsyms]  [k] setup_object                     
     2.99%   -0.31%  [kernel.kallsyms]  [k] file_free_rcu                    
     2.71%   -0.13%  [kernel.kallsyms]  [k] rcu_process_callbacks            
     2.26%   -0.09%  [kernel.kallsyms]  [k] get_empty_filp                   
     2.06%   -0.09%  [kernel.kallsyms]  [k] kmem_cache_alloc                 
     1.65%   -0.08%  [kernel.kallsyms]  [k] link_path_walk                   
     1.53%   -0.08%  [kernel.kallsyms]  [k] memset                           
     1.46%   -0.09%  [kernel.kallsyms]  [k] do_dentry_open                   
     1.44%   -0.04%  [kernel.kallsyms]  [k] __d_lookup_rcu                   
     1.27%   -0.04%  [kernel.kallsyms]  [k] do_last                          
     1.18%   -0.04%  [kernel.kallsyms]  [k] ext4_release_file                
     1.16%   -0.04%  [kernel.kallsyms]  [k] __call_rcu.constprop.11          

Quoted 3 lines

Quoted 3 lines

It's a 8-socket/160-thread (one NUMA node per socket) system that is not under memory pressure during the test. The latencies are also such that vm.zone_reclaim_mode=0.

The change will definitely spread allocations out to all nodes then and it's plausible that the remote references will hurt kernel object allocations in a tight loop. Just to confirm, could you rerun the test with zone_reclaim_mode enabled to make the allocator stay in the local zones?

Quoted 8 lines · Johannes Weiner

Quoted 3 lines

Quoted 3 lines

It's a 8-socket/160-thread (one NUMA node per socket) system that is not under memory pressure during the test. The latencies are also such that vm.zone_reclaim_mode=0.

The change will definitely spread allocations out to all nodes then and it's plausible that the remote references will hurt kernel object allocations in a tight loop. Just to confirm, could you rerun the test with zone_reclaim_mode enabled to make the allocator stay in the local zones?

Yeah, setting vm.zone_reclaim_mode=1 fixes it pretty instantaneously.

For what it's worth, I'm pretty convinced that the numbers folks put in the SLIT tables are, at best, horribly inconsistent from system to system. At worst, they're utter fabrications not linked at all to the reality of the actual latencies.

Quoted 4 lines · Dave Hansen

For what it's worth, I'm pretty convinced that the numbers folks put in the SLIT tables are, at best, horribly inconsistent from system to system. At worst, they're utter fabrications not linked at all to the reality of the actual latencies.

You mean the reported distances should probably be bigger on this particular machine?

But even when correct, zone_reclaim_mode might not be the best predictor. Just because it's not worth yet to invest direct reclaim efforts to stay local does not mean that remote references are free.

I'm currently running some tests with the below draft to see if this would still leave us with enough fairness. Does the patch restore performance even with zone_reclaim_mode disabled?

Hey Johannes,

I'm running an open/close microbenchmark from the will-it-scale set:

Quoted 1 line

https://github.com/antonblanchard/will-it-scale/blob/master/tests/open1.c

I was seeing some weird symptoms on 3.12 vs 3.11. The throughput in that test was going from down from 50 million to 35 million.

The profiles show an increase in cpu time in _raw_spin_lock_irq. The profiles pointed to slub code that hasn't been touched in quite a while.
I bisected it down to:

Quoted 6 lines · Dave Hansen

I'm running an open/close microbenchmark from the will-it-scale set:

Quoted 1 line

https://github.com/antonblanchard/will-it-scale/blob/master/tests/open1.c

I was seeing some weird symptoms on 3.12 vs 3.11. The throughput in that test was going from down from 50 million to 35 million.

Quoted 3 lines

The profiles show an increase in cpu time in _raw_spin_lock_irq. The profiles pointed to slub code that hasn't been touched in quite a while.
I bisected it down to:

Dave, do you mind retesting this against "[RFC PATCH 0/5] Memory compaction efficiency improvements" please? I have not finished reviewing the series yet but patch 3 mentions lower allocation success rates with Johannes' patch and notes that it is unlikely to be a bug with the patch itself.