Skip to contentSkip to conversation

f7b5d647946: -3.0% dbench.throughput-MB/sec

mm Fengguang Wu 2 participants 6 messages Aug 19, 2014

lore
Share & tools
lore
#0 lore raw

Hi Mel,

We noticed a minor dbench throughput regression on commit f7b5d647946aae1647bf5cd26c16b3a793c1ac49 ("mm: page_alloc: abort fair zone allocation policy when remotes nodes are encountered").

testcase: ivb44/dbench/100%

bb0b6dffa2ccfbd f7b5d647946aae1647bf5cd26
--------------- -------------------------

     25692 ± 0%      -3.0%      24913 ± 0%  dbench.throughput-MB/sec
   6974259 ± 6%     -12.1%    6127616 ± 0%  meminfo.DirectMap2M
     18.43 ± 0%      -4.6%      17.59 ± 0%  turbostat.RAM_W
      9302 ± 0%      -3.6%       8965 ± 1%  time.user_time
   1425791 ± 1%      -2.0%    1396598 ± 0%  time.involuntary_context_switches

Disclaimer:
Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance.

Thanks,
Fengguang

echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu10/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu11/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu12/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu13/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu14/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu15/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu16/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu17/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu18/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu19/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu20/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu21/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu22/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu23/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu24/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu25/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu26/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu27/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu28/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu29/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu3/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu30/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu31/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu32/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu33/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu34/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu35/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu36/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu37/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu38/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu39/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu4/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu40/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu41/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu42/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu43/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu44/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu45/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu46/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu47/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu5/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu6/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu7/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu8/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu9/cpufreq/scaling_governor dbench 48 -c /usr/share/dbench/client.txt

#1 lore raw
Quoted 16 lines · Fengguang Wu

Hi Mel,

We noticed a minor dbench throughput regression on commit f7b5d647946aae1647bf5cd26c16b3a793c1ac49 ("mm: page_alloc: abort fair zone allocation policy when remotes nodes are encountered").

testcase: ivb44/dbench/100%

bb0b6dffa2ccfbd f7b5d647946aae1647bf5cd26
--------------- -------------------------

     25692 ± 0%      -3.0%      24913 ± 0%  dbench.throughput-MB/sec
   6974259 ± 6%     -12.1%    6127616 ± 0%  meminfo.DirectMap2M
     18.43 ± 0%      -4.6%      17.59 ± 0%  turbostat.RAM_W
      9302 ± 0%      -3.6%       8965 ± 1%  time.user_time
   1425791 ± 1%      -2.0%    1396598 ± 0%  time.involuntary_context_switches
Quoted 5 lines

Disclaimer:
Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance.

DirectMap2M changing is a major surprise and doesn't make sense for this machine. Did the amount of memory in the machine change between two tests?

Signature · 2 lines
Mel Gorman
SUSE Labs
#2 lore raw
Quoted 17 lines · Mel Gorman
Quoted 16 lines · Fengguang Wu

Hi Mel,

We noticed a minor dbench throughput regression on commit f7b5d647946aae1647bf5cd26c16b3a793c1ac49 ("mm: page_alloc: abort fair zone allocation policy when remotes nodes are encountered").

testcase: ivb44/dbench/100%

bb0b6dffa2ccfbd f7b5d647946aae1647bf5cd26
--------------- -------------------------

     25692 ± 0%      -3.0%      24913 ± 0%  dbench.throughput-MB/sec
   6974259 ± 6%     -12.1%    6127616 ± 0%  meminfo.DirectMap2M
     18.43 ± 0%      -4.6%      17.59 ± 0%  turbostat.RAM_W
      9302 ± 0%      -3.6%       8965 ± 1%  time.user_time
   1425791 ± 1%      -2.0%    1396598 ± 0%  time.involuntary_context_switches
Quoted 8 lines
Quoted 5 lines

Disclaimer:
Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance.

DirectMap2M changing is a major surprise and doesn't make sense for this machine.

The ivb44's hardware configuration is

        model: Ivytown Ivy Bridge-EP
        nr_cpu: 48
        memory: 64G

And note that this is an in-memory dbench run, which is why dbench.throughput-MB/sec is so high.

Quoted 1 line

Did the amount of memory in the machine change between two tests?

Nope. They are back-to-back test runs, so the environment pretty much remains the same.

Thanks,
Fengguang

#3 lore raw
Quoted 33 lines · Fengguang Wu
Quoted 17 lines · Mel Gorman
Quoted 16 lines · Fengguang Wu

Hi Mel,

We noticed a minor dbench throughput regression on commit f7b5d647946aae1647bf5cd26c16b3a793c1ac49 ("mm: page_alloc: abort fair zone allocation policy when remotes nodes are encountered").

testcase: ivb44/dbench/100%

bb0b6dffa2ccfbd f7b5d647946aae1647bf5cd26
--------------- -------------------------

     25692 ± 0%      -3.0%      24913 ± 0%  dbench.throughput-MB/sec
   6974259 ± 6%     -12.1%    6127616 ± 0%  meminfo.DirectMap2M
     18.43 ± 0%      -4.6%      17.59 ± 0%  turbostat.RAM_W
      9302 ± 0%      -3.6%       8965 ± 1%  time.user_time
   1425791 ± 1%      -2.0%    1396598 ± 0%  time.involuntary_context_switches
Quoted 8 lines
Quoted 5 lines

Disclaimer:
Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance.

DirectMap2M changing is a major surprise and doesn't make sense for this machine.

The ivb44's hardware configuration is

        model: Ivytown Ivy Bridge-EP
        nr_cpu: 48
        memory: 64G
Quoted 7 lines

And note that this is an in-memory dbench run, which is why dbench.throughput-MB/sec is so high.

Quoted 1 line

Did the amount of memory in the machine change between two tests?

Nope. They are back-to-back test runs, so the environment pretty much remains the same.

Then how did directmap2m change? The sum of the direct maps should correspond to the amount of physical memory and this patch has nothing to do with any memory initialisation paths that might affect this.

Signature · 2 lines
Mel Gorman
SUSE Labs
#4 lore raw
Quoted 37 lines · Mel Gorman
Quoted 36 lines · Fengguang Wu
Quoted 17 lines · Mel Gorman
Quoted 16 lines · Fengguang Wu

Hi Mel,

We noticed a minor dbench throughput regression on commit f7b5d647946aae1647bf5cd26c16b3a793c1ac49 ("mm: page_alloc: abort fair zone allocation policy when remotes nodes are encountered").

testcase: ivb44/dbench/100%

bb0b6dffa2ccfbd f7b5d647946aae1647bf5cd26
--------------- -------------------------

     25692 ± 0%      -3.0%      24913 ± 0%  dbench.throughput-MB/sec
   6974259 ± 6%     -12.1%    6127616 ± 0%  meminfo.DirectMap2M
     18.43 ± 0%      -4.6%      17.59 ± 0%  turbostat.RAM_W
      9302 ± 0%      -3.6%       8965 ± 1%  time.user_time
   1425791 ± 1%      -2.0%    1396598 ± 0%  time.involuntary_context_switches
Quoted 8 lines
Quoted 5 lines

Disclaimer:
Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance.

DirectMap2M changing is a major surprise and doesn't make sense for this machine.

The ivb44's hardware configuration is

        model: Ivytown Ivy Bridge-EP
        nr_cpu: 48
        memory: 64G

And note that this is an in-memory dbench run, which is why dbench.throughput-MB/sec is so high.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Did the amount of memory in the machine change between two tests?

Nope. They are back-to-back test runs, so the environment pretty much remains the same.

Then how did directmap2m change? The sum of the direct maps should correspond to the amount of physical memory and this patch has nothing to do with any memory initialisation paths that might affect this.

Good question. Not sure for the moment, but it looks that even the multiple boots for the same kernel bb0b6dffa2ccfbd have different DirectMap2M. And it only happen for kernel bb0b6dffa2ccfbd. f7b5d64794 remains stable for all boots.

bb0b6dffa2ccfbd f7b5d647946aae1647bf5cd26
--------------- -------------------------
%stddev %change %stddev
\ | /
6974259 ± 6% -12.1% 6127616 ± 0% meminfo.DirectMap2M

Looking at the concrete numbers for each boot, the DirectMap2M changes each time.

"meminfo.DirectMap2M": [

    7182336,
    7190528,
    7178240,
    6131712,
    7188480

],

The MemTotal does remain stable for this kernel:

"meminfo.MemTotal": [

    65869268,
    65869268,
    65869268,
    65869268,
    65869268

],

Attached are the full stats for the 2 kernels.

Thanks,
Fengguang

#5 lore raw
In reply to Fengguang Wu
Quoted 27 lines · Fengguang Wu
Quoted 17 lines · Mel Gorman
Quoted 16 lines · Fengguang Wu

Hi Mel,

We noticed a minor dbench throughput regression on commit f7b5d647946aae1647bf5cd26c16b3a793c1ac49 ("mm: page_alloc: abort fair zone allocation policy when remotes nodes are encountered").

testcase: ivb44/dbench/100%

bb0b6dffa2ccfbd f7b5d647946aae1647bf5cd26
--------------- -------------------------

     25692 ± 0%      -3.0%      24913 ± 0%  dbench.throughput-MB/sec
   6974259 ± 6%     -12.1%    6127616 ± 0%  meminfo.DirectMap2M
     18.43 ± 0%      -4.6%      17.59 ± 0%  turbostat.RAM_W
      9302 ± 0%      -3.6%       8965 ± 1%  time.user_time
   1425791 ± 1%      -2.0%    1396598 ± 0%  time.involuntary_context_switches
Quoted 8 lines
Quoted 5 lines

Disclaimer:
Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance.

DirectMap2M changing is a major surprise and doesn't make sense for this machine.

Quoted 9 lines

The ivb44's hardware configuration is

        model: Ivytown Ivy Bridge-EP
        nr_cpu: 48
        memory: 64G

And note that this is an in-memory dbench run, which is why dbench.throughput-MB/sec is so high.

Ok, it's a NUMA machine. I expect in this case that prior to the patch more local memory would have been used on node 0 due to the fair zone allocation policy skipping remote nodes. The patch corrects the behaviour of zonelist but the downside is more remote accesses for processes running on node 0. The behaviour is correct although not necessarily desirable from a performance point of view. Users should boot with numa_zonelist_order=node if it's a problem.

Signature · 2 lines
Mel Gorman
SUSE Labs