c4a7ca77494: +6.0% fsmark.time.involuntary_context_switches, no primary result change
FYI, we noticed the below changes on
commit c4a7ca774949960064dac11b326908f28407e8c3 ("SUNRPC: Allow waiting on memory allocation")
testbox/testcase/testparams: nhm4/fsmark/performance-1x-32t-1HDD-f2fs-nfsv4-8K-400M-fsyncBeforeClose-16d-256fpd
127b21b89f9d8ba0 c4a7ca774949960064dac11b32
---------------- --------------------------
%stddev %change %stddev
\ | \
52524 ± 0% +6.0% 55672 ± 0% fsmark.time.involuntary_context_switches
436 ± 14% +54.9% 676 ± 20% sched_debug.cfs_rq[0]:/.tg_load_contrib
433 ± 15% +54.7% 670 ± 21% sched_debug.cfs_rq[0]:/.blocked_load_avg
8348 ± 7% +27.0% 10602 ± 9% sched_debug.cfs_rq[0]:/.min_vruntime
190081 ± 13% +32.7% 252269 ± 13% sched_debug.cpu#0.sched_goidle
205783 ± 12% +30.2% 267903 ± 13% sched_debug.cpu#0.ttwu_local
464065 ± 11% +26.6% 587524 ± 12% sched_debug.cpu#0.nr_switches
464278 ± 11% +26.6% 587734 ± 12% sched_debug.cpu#0.sched_count
15807 ± 11% +19.6% 18910 ± 12% sched_debug.cpu#4.nr_load_updates
300041 ± 8% +20.3% 360969 ± 10% sched_debug.cpu#0.ttwu_count
1863 ± 9% +18.1% 2201 ± 10% sched_debug.cfs_rq[4]:/.exec_clocktestbox/testcase/testparams: nhm4/fsmark/performance-1x-32t-1HDD-btrfs-nfsv4-8K-400M-fsyncBeforeClose-16d-256fpd
127b21b89f9d8ba0 c4a7ca774949960064dac11b32
---------------- --------------------------
fail:runs %reproduction fail:runs
| | |
52184 ± 0% +5.6% 55122 ± 0% fsmark.time.involuntary_context_switches
557 ± 19% +21.5% 677 ± 9% sched_debug.cfs_rq[5]:/.blocked_load_avg
217 ± 19% -42.9% 124 ± 21% sched_debug.cfs_rq[2]:/.load
45852 ± 14% -39.4% 27773 ± 24% sched_debug.cpu#7.ttwu_local
457 ± 18% +50.1% 686 ± 20% sched_debug.cfs_rq[0]:/.tg_load_contrib
455 ± 18% +46.7% 668 ± 19% sched_debug.cfs_rq[0]:/.blocked_load_avg
66605 ± 10% -26.7% 48826 ± 14% sched_debug.cpu#7.sched_goidle
78249 ± 9% -22.5% 60678 ± 11% sched_debug.cpu#7.ttwu_count
153506 ± 9% -22.7% 118649 ± 12% sched_debug.cpu#7.nr_switches
153613 ± 9% -22.7% 118755 ± 12% sched_debug.cpu#7.sched_count
15806 ± 6% +19.2% 18833 ± 18% sched_debug.cpu#4.nr_load_updates
2171 ± 5% +15.6% 2510 ± 13% sched_debug.cfs_rq[4]:/.exec_clock
9924 ± 11% -27.0% 7244 ± 25% sched_debug.cfs_rq[3]:/.min_vruntime
3156 ± 4% -13.4% 2734 ± 8% sched_debug.cfs_rq[7]:/.min_vruntimetestbox/testcase/testparams: nhm4/fsmark/performance-1x-32t-1HDD-ext4-nfsv4-9B-400M-fsyncBeforeClose-16d-256fpd
127b21b89f9d8ba0 c4a7ca774949960064dac11b32
---------------- --------------------------
104802 ± 0% +7.7% 112883 ± 0% fsmark.time.involuntary_context_switches
471755 ± 0% -1.3% 465592 ± 0% fsmark.time.voluntary_context_switches
1977 ± 36% +90.8% 3771 ± 8% sched_debug.cpu#4.curr->pid
2 ± 34% +80.0% 4 ± 24% sched_debug.cpu#6.cpu_load[1]
4 ± 33% +83.3% 8 ± 31% sched_debug.cpu#6.cpu_load[0]
193 ± 17% +48.0% 286 ± 19% sched_debug.cfs_rq[2]:/.blocked_load_avg
196 ± 17% +47.5% 290 ± 19% sched_debug.cfs_rq[2]:/.tg_load_contrib
96 ± 18% +40.6% 135 ± 11% sched_debug.cfs_rq[7]:/.load
97 ± 18% +38.5% 135 ± 11% sched_debug.cpu#7.load
2274 ± 7% -16.5% 1898 ± 3% proc-vmstat.pgalloc_dma
319 ± 6% -29.7% 224 ± 24% sched_debug.cfs_rq[1]:/.tg_load_contrib
314 ± 5% -29.4% 222 ± 25% sched_debug.cfs_rq[1]:/.blocked_load_avg
621 ± 10% +41.9% 881 ± 37% sched_debug.cfs_rq[4]:/.avg->runnable_avg_sumnhm4: Nehalem
Memory: 4G
fsmark.time.involuntary_context_switches
114000 ++-----------------------------------------------------------------+
113000 O+ O O O O O O O O O O O O O O O O O |
| O O O O O
112000 ++ |
111000 ++ |
| |
110000 ++ |
109000 ++ |
108000 ++ |
| |
107000 ++ |
106000 ++ |
| |
105000 *+.*..*..*..*..*..*..*..*..*..*...*..*..*..*..*..*..*..* |104000 ++-----------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad sample
To reproduce:
apt-get install ruby
git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git
cd lkp-tests
bin/setup-local job.yaml # the job file attached in this email
bin/run-local job.yaml
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance.
Thanks,
Fengguang
_______________________________________________
LKP mailing list
LKP(a)linux.intel.com
---
testcase: fsmark
default-monitors:
wait: pre-test
uptime:
iostat:
vmstat:
numa-numastat:
numa-vmstat:
numa-meminfo:
proc-vmstat:
proc-stat:
meminfo:
slabinfo:
interrupts:
lock_stat:
latency_stats:
softirqs:
bdi_dev_mapping:
diskstats:
nfsstat:
cpuidle:
cpufreq-stats:
turbostat:
pmeter:
sched_debug:
interval: 10default_watchdogs:
watch-oom:
watchdog:cpufreq_governor: performance
commit: 5721f7f0f14b682d2e86e9a4aa9025acaf69399d
model: Nehalemnr_cpu: 8
memory: 4Ghdd_partitions: "/dev/disk/by-id/ata-WDC_WD1003FBYZ-010FB0_WD-WCAW36812041-part1"swap_partitions: "/dev/disk/by-id/ata-WDC_WD1003FBYZ-010FB0_WD-WCAW36812041-part2"rootfs_partition: "/dev/disk/by-id/ata-WDC_WD1003FBYZ-010FB0_WD-WCAW36812041-part3"netconsole_port: 6649
iterations: 1xnr_threads: 32t
disk: 1HDD
fs: ext4
fs2: nfsv4
fsmark:
filesize: 9B
test_size: 400M
sync_method: fsyncBeforeClose
nr_directories: 16d
nr_files_per_directory: 256fpd
testbox: nhm4tbox_group: nhm4
kconfig: x86_64-rhelenqueue_time: 2015-02-10 23:57:36.647136389 +08:00head_commit: 5721f7f0f14b682d2e86e9a4aa9025acaf69399dbase_commit: bfa76d49576599a4b9f9b7a71f23d73d6dcff735
branch: next/master
kernel: "/kernel/x86_64-rhel/5721f7f0f14b682d2e86e9a4aa9025acaf69399d/vmlinuz-3.19.0-next-20150211-g5721f7f"
user: lkp
queue: cyclic
rootfs: debian-x86_64-2015-02-07.cgzresult_root: "/result/nhm4/fsmark/performance-1x-32t-1HDD-ext4-nfsv4-9B-400M-fsyncBeforeClose-16d-256fpd/debian-x86_64-2015-02-07.cgz/x86_64-rhel/5721f7f0f14b682d2e86e9a4aa9025acaf69399d/0"job_file: "/lkp/scheduled/nhm4/cyclic_fsmark-performance-1x-32t-1HDD-ext4-nfsv4-9B-400M-fsyncBeforeClose-16d-256fpd-x86_64-rhel-HEAD-5721f7f0f14b682d2e86e9a4aa9025acaf69399d-0-20150210-12345-1obo8ps.yaml"dequeue_time: 2015-02-11 16:52:17.667759731 +08:00job_state: finished
loadavg: 29.23 38.04 33.63 1/155 27817start_time: '1423644766'end_time: '1423646499'
version: "/lkp/lkp/.src-20150211-114913"
mkfs -t ext4 -q -F /dev/sda1
mount -t ext4 /dev/sda1 /fs/sda1/etc/init.d/rpcbind start/etc/init.d/nfs-common start/etc/init.d/nfs-kernel-server start
mount -t nfs -o vers=4 localhost:/fs/sda1 /nfs/sda1
./fs_mark -d /nfs/sda1/1 -d /nfs/sda1/2 -d /nfs/sda1/3 -d /nfs/sda1/4 -d /nfs/sda1/5 -d /nfs/sda1/6 -d /nfs/sda1/7 -d /nfs/sda1/8 -d /nfs/sda1/9 -d /nfs/sda1/10 -d /nfs/sda1/11 -d /nfs/sda1/12 -d /nfs/sda1/13 -d /nfs/sda1/14 -d /nfs/sda1/15 -d /nfs/sda1/16 -d /nfs/sda1/17 -d /nfs/sda1/18 -d /nfs/sda1/19 -d /nfs/sda1/20 -d /nfs/sda1/21 -d /nfs/sda1/22 -d /nfs/sda1/23 -d /nfs/sda1/24 -d /nfs/sda1/25 -d /nfs/sda1/26 -d /nfs/sda1/27 -d /nfs/sda1/28 -d /nfs/sda1/29 -d /nfs/sda1/30 -d /nfs/sda1/31 -d /nfs/sda1/32 -D 16 -N 256 -n 3200 -L 1 -S 1 -s 9
Hi guys,
Quoted 90 lines · Huang Ying
FYI, we noticed the below changes on
commit
c4a7ca774949960064dac11b326908f28407e8c3("SUNRPC: Allow waiting on memory allocation")testbox/testcase/testparams: nhm4/fsmark/performance-1x-32t-1HDD-f2fs-nfsv4-8K-400M-fsyncBeforeClose-16d-256fpd
127b21b89f9d8ba0c4a7ca774949960064dac11b32
---------------- --------------------------%stddev %change %stddev \ | \ 52524 ± 0% +6.0% 55672 ± 0% fsmark.time.involuntary_context_switches 436 ± 14% +54.9% 676 ± 20% sched_debug.cfs_rq[0]:/.tg_load_contrib 433 ± 15% +54.7% 670 ± 21% sched_debug.cfs_rq[0]:/.blocked_load_avg 8348 ± 7% +27.0% 10602 ± 9% sched_debug.cfs_rq[0]:/.min_vruntime 190081 ± 13% +32.7% 252269 ± 13% sched_debug.cpu#0.sched_goidle 205783 ± 12% +30.2% 267903 ± 13% sched_debug.cpu#0.ttwu_local 464065 ± 11% +26.6% 587524 ± 12% sched_debug.cpu#0.nr_switches 464278 ± 11% +26.6% 587734 ± 12% sched_debug.cpu#0.sched_count 15807 ± 11% +19.6% 18910 ± 12% sched_debug.cpu#4.nr_load_updates 300041 ± 8% +20.3% 360969 ± 10% sched_debug.cpu#0.ttwu_count 1863 ± 9% +18.1% 2201 ± 10% sched_debug.cfs_rq[4]:/.exec_clocktestbox/testcase/testparams: nhm4/fsmark/performance-1x-32t-1HDD-btrfs-nfsv4-8K-400M-fsyncBeforeClose-16d-256fpd
127b21b89f9d8ba0 c4a7ca774949960064dac11b32 ---------------- -------------------------- fail:runs %reproduction fail:runs | | | 52184 ± 0% +5.6% 55122 ± 0% fsmark.time.involuntary_context_switches 557 ± 19% +21.5% 677 ± 9% sched_debug.cfs_rq[5]:/.blocked_load_avg 217 ± 19% -42.9% 124 ± 21% sched_debug.cfs_rq[2]:/.load 45852 ± 14% -39.4% 27773 ± 24% sched_debug.cpu#7.ttwu_local 457 ± 18% +50.1% 686 ± 20% sched_debug.cfs_rq[0]:/.tg_load_contrib 455 ± 18% +46.7% 668 ± 19% sched_debug.cfs_rq[0]:/.blocked_load_avg 66605 ± 10% -26.7% 48826 ± 14% sched_debug.cpu#7.sched_goidle 78249 ± 9% -22.5% 60678 ± 11% sched_debug.cpu#7.ttwu_count 153506 ± 9% -22.7% 118649 ± 12% sched_debug.cpu#7.nr_switches 153613 ± 9% -22.7% 118755 ± 12% sched_debug.cpu#7.sched_count 15806 ± 6% +19.2% 18833 ± 18% sched_debug.cpu#4.nr_load_updates 2171 ± 5% +15.6% 2510 ± 13% sched_debug.cfs_rq[4]:/.exec_clock 9924 ± 11% -27.0% 7244 ± 25% sched_debug.cfs_rq[3]:/.min_vruntime 3156 ± 4% -13.4% 2734 ± 8% sched_debug.cfs_rq[7]:/.min_vruntimetestbox/testcase/testparams: nhm4/fsmark/performance-1x-32t-1HDD-ext4-nfsv4-9B-400M-fsyncBeforeClose-16d-256fpd
127b21b89f9d8ba0c4a7ca774949960064dac11b32
---------------- --------------------------104802 ± 0% +7.7% 112883 ± 0% fsmark.time.involuntary_context_switches 471755 ± 0% -1.3% 465592 ± 0% fsmark.time.voluntary_context_switches 1977 ± 36% +90.8% 3771 ± 8% sched_debug.cpu#4.curr->pid 2 ± 34% +80.0% 4 ± 24% sched_debug.cpu#6.cpu_load[1] 4 ± 33% +83.3% 8 ± 31% sched_debug.cpu#6.cpu_load[0] 193 ± 17% +48.0% 286 ± 19% sched_debug.cfs_rq[2]:/.blocked_load_avg 196 ± 17% +47.5% 290 ± 19% sched_debug.cfs_rq[2]:/.tg_load_contrib 96 ± 18% +40.6% 135 ± 11% sched_debug.cfs_rq[7]:/.load 97 ± 18% +38.5% 135 ± 11% sched_debug.cpu#7.load 2274 ± 7% -16.5% 1898 ± 3% proc-vmstat.pgalloc_dma 319 ± 6% -29.7% 224 ± 24% sched_debug.cfs_rq[1]:/.tg_load_contrib 314 ± 5% -29.4% 222 ± 25% sched_debug.cfs_rq[1]:/.blocked_load_avg 621 ± 10% +41.9% 881 ± 37% sched_debug.cfs_rq[4]:/.avg->runnable_avg_sumnhm4: Nehalem
Memory: 4Gfsmark.time.involuntary_context_switches
114000 ++-----------------------------------------------------------------+
113000 O+ O O O O O O O O O O O O O O O O O | | O O O O O 112000 ++ | 111000 ++ | | | 110000 ++ | 109000 ++ | 108000 ++ | | | 107000 ++ | 106000 ++ | | | 105000 *+.*..*..*..*..*..*..*..*..*..*...*..*..*..*..*..*..*..* |104000 ++-----------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad sample
Quoted 7 lines
To reproduce:
apt-get install ruby
git clonegit://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git
cd lkp-tests
bin/setup-localjob.yaml# the job file attached in this email
bin/run-localjob.yaml
So this is on a loopback NFS setup (i.e. the server resides on the same node as the client, which just mounts from the loopback IP address 127.0.0.1)?
That's a fairly quirky setup as far as memory management goes. In low memory situations, you have very a nasty feedback mechanism whereby the NFS server ends up pushing the client to write back more data, increasing the memory pressure on the NFS server, etc.
It is quite possible that allowing the NFS client to block more aggressively in low memory situations could worsen that feedback mechanism, however that's not our main target platform; we actively discourage people from using loopback NFS in production systems.
Is there any way you could confirm this performance change using a remote NFS server instead of the loopback NFS?
Cheers
Trond
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, PrimaryData trond.myklebust(a)primarydata.com
Quoted 101 lines · Trond Myklebust
Hi guys,
Quoted 90 lines · Huang Ying
FYI, we noticed the below changes on
commit
c4a7ca774949960064dac11b326908f28407e8c3("SUNRPC: Allow waiting on memory allocation")testbox/testcase/testparams: nhm4/fsmark/performance-1x-32t-1HDD-f2fs-nfsv4-8K-400M-fsyncBeforeClose-16d-256fpd
127b21b89f9d8ba0c4a7ca774949960064dac11b32
---------------- --------------------------%stddev %change %stddev \ | \ 52524 ± 0% +6.0% 55672 ± 0% fsmark.time.involuntary_context_switches 436 ± 14% +54.9% 676 ± 20% sched_debug.cfs_rq[0]:/.tg_load_contrib 433 ± 15% +54.7% 670 ± 21% sched_debug.cfs_rq[0]:/.blocked_load_avg 8348 ± 7% +27.0% 10602 ± 9% sched_debug.cfs_rq[0]:/.min_vruntime 190081 ± 13% +32.7% 252269 ± 13% sched_debug.cpu#0.sched_goidle 205783 ± 12% +30.2% 267903 ± 13% sched_debug.cpu#0.ttwu_local 464065 ± 11% +26.6% 587524 ± 12% sched_debug.cpu#0.nr_switches 464278 ± 11% +26.6% 587734 ± 12% sched_debug.cpu#0.sched_count 15807 ± 11% +19.6% 18910 ± 12% sched_debug.cpu#4.nr_load_updates 300041 ± 8% +20.3% 360969 ± 10% sched_debug.cpu#0.ttwu_count 1863 ± 9% +18.1% 2201 ± 10% sched_debug.cfs_rq[4]:/.exec_clocktestbox/testcase/testparams: nhm4/fsmark/performance-1x-32t-1HDD-btrfs-nfsv4-8K-400M-fsyncBeforeClose-16d-256fpd
127b21b89f9d8ba0 c4a7ca774949960064dac11b32 ---------------- -------------------------- fail:runs %reproduction fail:runs | | | 52184 ± 0% +5.6% 55122 ± 0% fsmark.time.involuntary_context_switches 557 ± 19% +21.5% 677 ± 9% sched_debug.cfs_rq[5]:/.blocked_load_avg 217 ± 19% -42.9% 124 ± 21% sched_debug.cfs_rq[2]:/.load 45852 ± 14% -39.4% 27773 ± 24% sched_debug.cpu#7.ttwu_local 457 ± 18% +50.1% 686 ± 20% sched_debug.cfs_rq[0]:/.tg_load_contrib 455 ± 18% +46.7% 668 ± 19% sched_debug.cfs_rq[0]:/.blocked_load_avg 66605 ± 10% -26.7% 48826 ± 14% sched_debug.cpu#7.sched_goidle 78249 ± 9% -22.5% 60678 ± 11% sched_debug.cpu#7.ttwu_count 153506 ± 9% -22.7% 118649 ± 12% sched_debug.cpu#7.nr_switches 153613 ± 9% -22.7% 118755 ± 12% sched_debug.cpu#7.sched_count 15806 ± 6% +19.2% 18833 ± 18% sched_debug.cpu#4.nr_load_updates 2171 ± 5% +15.6% 2510 ± 13% sched_debug.cfs_rq[4]:/.exec_clock 9924 ± 11% -27.0% 7244 ± 25% sched_debug.cfs_rq[3]:/.min_vruntime 3156 ± 4% -13.4% 2734 ± 8% sched_debug.cfs_rq[7]:/.min_vruntimetestbox/testcase/testparams: nhm4/fsmark/performance-1x-32t-1HDD-ext4-nfsv4-9B-400M-fsyncBeforeClose-16d-256fpd
127b21b89f9d8ba0c4a7ca774949960064dac11b32
---------------- --------------------------104802 ± 0% +7.7% 112883 ± 0% fsmark.time.involuntary_context_switches 471755 ± 0% -1.3% 465592 ± 0% fsmark.time.voluntary_context_switches 1977 ± 36% +90.8% 3771 ± 8% sched_debug.cpu#4.curr->pid 2 ± 34% +80.0% 4 ± 24% sched_debug.cpu#6.cpu_load[1] 4 ± 33% +83.3% 8 ± 31% sched_debug.cpu#6.cpu_load[0] 193 ± 17% +48.0% 286 ± 19% sched_debug.cfs_rq[2]:/.blocked_load_avg 196 ± 17% +47.5% 290 ± 19% sched_debug.cfs_rq[2]:/.tg_load_contrib 96 ± 18% +40.6% 135 ± 11% sched_debug.cfs_rq[7]:/.load 97 ± 18% +38.5% 135 ± 11% sched_debug.cpu#7.load 2274 ± 7% -16.5% 1898 ± 3% proc-vmstat.pgalloc_dma 319 ± 6% -29.7% 224 ± 24% sched_debug.cfs_rq[1]:/.tg_load_contrib 314 ± 5% -29.4% 222 ± 25% sched_debug.cfs_rq[1]:/.blocked_load_avg 621 ± 10% +41.9% 881 ± 37% sched_debug.cfs_rq[4]:/.avg->runnable_avg_sumnhm4: Nehalem
Memory: 4Gfsmark.time.involuntary_context_switches
114000 ++-----------------------------------------------------------------+
113000 O+ O O O O O O O O O O O O O O O O O | | O O O O O 112000 ++ | 111000 ++ | | | 110000 ++ | 109000 ++ | 108000 ++ | | | 107000 ++ | 106000 ++ | | | 105000 *+.*..*..*..*..*..*..*..*..*..*...*..*..*..*..*..*..*..* |104000 ++-----------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad sampleQuoted 7 lines
To reproduce:
apt-get install ruby
git clonegit://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git
cd lkp-tests
bin/setup-localjob.yaml# the job file attached in this email
bin/run-localjob.yaml
Quoted 3 lines
So this is on a loopback NFS setup (i.e. the server resides on the same node as the client, which just mounts from the loopback IP address 127.0.0.1)?
Yes. This is on a loopback NFS setup.
Quoted 11 lines
That's a fairly quirky setup as far as memory management goes. In low memory situations, you have very a nasty feedback mechanism whereby the NFS server ends up pushing the client to write back more data, increasing the memory pressure on the NFS server, etc.
It is quite possible that allowing the NFS client to block more aggressively in low memory situations could worsen that feedback mechanism, however that's not our main target platform; we actively discourage people from using loopback NFS in production systems.Is there any way you could confirm this performance change using a remote NFS server instead of the loopback NFS?
We are working on a remote NFS setup now :)
Best Regards,
Huang, Ying