a0f9ec1f181: -4.3% will-it-scale.per_thread_ops
Hi Tejun,
FYI, we noticed the below changes on
git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git review-kill-tree_mutex
commit a0f9ec1f181534694cb5bf40b7b56515b8cabef9 ("cgroup: use cgroup_kn_lock_live() in other cgroup kernfs methods")
Test case : lkp-nex05/will-it-scale/writeseek
2074b6e38668e62 a0f9ec1f181534694cb5bf40b
--------------- -------------------------
1027273 ~ 0% -4.3% 982732 ~ 0% TOTAL will-it-scale.per_thread_ops
136 ~ 3% -43.1% 77 ~43% TOTAL proc-vmstat.nr_dirtied
0.51 ~ 3% +98.0% 1.01 ~ 4% TOTAL perf-profile.cpu-cycles.shmem_write_end.generic_perform_write.__generic_file_aio_write.generic_file_aio_write.do_sync_write
1078 ~ 9% -16.3% 903 ~11% TOTAL numa-meminfo.node0.Unevictable
269 ~ 9% -16.2% 225 ~11% TOTAL numa-vmstat.node0.nr_unevictable
1.64 ~ 1% -14.3% 1.41 ~ 4% TOTAL perf-profile.cpu-cycles.find_lock_entry.shmem_getpage_gfp.shmem_write_begin.generic_perform_write.__generic_file_aio_write
1.62 ~ 2% +14.1% 1.84 ~ 1% TOTAL perf-profile.cpu-cycles.lseek64Legend:
~XX% - stddev percent
[+-]XX% - change percent
Thanks,
Jet
echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu10/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu11/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu12/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu13/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu14/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu15/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu16/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu17/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu18/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu19/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu20/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu21/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu22/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu23/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu24/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu25/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu26/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu27/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu28/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu29/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu3/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu30/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu31/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu32/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu33/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu34/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu35/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu36/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu37/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu38/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu39/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu4/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu40/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu41/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu42/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu43/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu44/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu45/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu46/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu47/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu48/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu49/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu5/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu50/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu51/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu52/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu53/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu54/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu55/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu56/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu57/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu58/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu59/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu6/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu60/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu61/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu62/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu63/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu7/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu8/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu9/cpufreq/scaling_governor ./runtest.py writeseek 20 1 10 20 30 48 64
Hello,
Quoted 9 lines · Jet Chen
FYI, we noticed the below changes on
git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.gitreview-kill-tree_mutex
commita0f9ec1f181534694cb5bf40b7b56515b8cabef9("cgroup: usecgroup_kn_lock_live()in other cgroup kernfs methods")Test case : lkp-nex05/will-it-scale/writeseek
2074b6e38668e62a0f9ec1f181534694cb5bf40b
--------------- -------------------------
Quoted 7 lines
1027273 ~ 0% -4.3% 982732 ~ 0% TOTAL will-it-scale.per_thread_ops
136 ~ 3% -43.1% 77 ~43% TOTAL proc-vmstat.nr_dirtied
0.51 ~ 3% +98.0% 1.01 ~ 4% TOTAL perf-profile.cpu-cycles.shmem_write_end.generic_perform_write.__generic_file_aio_write.generic_file_aio_write.do_sync_write
1078 ~ 9% -16.3% 903 ~11% TOTAL numa-meminfo.node0.Unevictable
269 ~ 9% -16.2% 225 ~11% TOTAL numa-vmstat.node0.nr_unevictable
1.64 ~ 1% -14.3% 1.41 ~ 4% TOTAL perf-profile.cpu-cycles.find_lock_entry.shmem_getpage_gfp.shmem_write_begin.generic_perform_write.__generic_file_aio_write
1.62 ~ 2% +14.1% 1.84 ~ 1% TOTAL perf-profile.cpu-cycles.lseek64I have no idea how to read the above. Which direction is plus and which is minus? Are they counting cpu cycles? Which files is the test seeking?
Thanks.
Signature · 1 line
tejun
Hi Tejun,
Quoted 12 lines · Tejun Heo
Hello,
Quoted 9 lines · Jet Chen
FYI, we noticed the below changes on
git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.gitreview-kill-tree_mutex
commita0f9ec1f181534694cb5bf40b7b56515b8cabef9("cgroup: usecgroup_kn_lock_live()in other cgroup kernfs methods")Test case : lkp-nex05/will-it-scale/writeseek
2074b6e38668e62a0f9ec1f181534694cb5bf40b
--------------- -------------------------
2074b6e38668e62 is the base of comparison. So "-4.3% will-it-scale.per_thread_ops" in the below line means a0f9ec1f18 has lower will-it-scale throughput.
Quoted 7 lines
Quoted 7 lines
1027273 ~ 0% -4.3% 982732 ~ 0% TOTAL will-it-scale.per_thread_ops 136 ~ 3% -43.1% 77 ~43% TOTAL proc-vmstat.nr_dirtied 0.51 ~ 3% +98.0% 1.01 ~ 4% TOTAL perf-profile.cpu-cycles.shmem_write_end.generic_perform_write.__generic_file_aio_write.generic_file_aio_write.do_sync_write 1078 ~ 9% -16.3% 903 ~11% TOTAL numa-meminfo.node0.Unevictable 269 ~ 9% -16.2% 225 ~11% TOTAL numa-vmstat.node0.nr_unevictable 1.64 ~ 1% -14.3% 1.41 ~ 4% TOTAL perf-profile.cpu-cycles.find_lock_entry.shmem_getpage_gfp.shmem_write_begin.generic_perform_write.__generic_file_aio_write 1.62 ~ 2% +14.1% 1.84 ~ 1% TOTAL perf-profile.cpu-cycles.lseek64
The perf-profile.cpu-cycles.* lines are from "perf record/report".
The last line shows that lseek64() takes 1.62% CPU cycles for commit 2074b6e38668e62 and that percent increased by +14.1% on a0f9ec1f181. One of the raw perf record output is
1.84% writeseek_proce libc-2.17.so [.] lseek64
|
--- lseek64There are 5 runs and 1.62% is the average value.
Quoted 3 lines
I have no idea how to read the above. Which direction is plus and which is minus? Are they counting cpu cycles? Which files is the test seeking?
It's tmpfs files. Because the will-it-scale test case is mean to measure scalability of syscalls. We do not use HDD/SSD etc. storage devices when running it.
The will-it-scale/writeseek test code is
char *testcase_description = "Separate file seek+write";
void testcase(unsigned long long *iterations)
{
char buf[BUFLEN];
char tmpfile[] = "/tmp/willitscale.XXXXXX";
int fd = mkstemp(tmpfile);
memset(buf, 0, sizeof(buf));
assert(fd >= 0);
unlink(tmpfile);
while (1) {
lseek(fd, 0, SEEK_SET);
assert(write(fd, buf, BUFLEN) == BUFLEN);
(*iterations)++;
}
}Thanks,
Fengguang
Hello, Fengguang.
Quoted 26 lines · Fengguang Wu
Quoted 2 lines
Quoted 2 lines
2074b6e38668e62a0f9ec1f181534694cb5bf40b
--------------- -------------------------
2074b6e38668e62is the base of comparison. So "-4.3% will-it-scale.per_thread_ops" in the below line meansa0f9ec1f18has lower will-it-scale throughput.Quoted 7 lines
Quoted 7 lines
1027273 ~ 0% -4.3% 982732 ~ 0% TOTAL will-it-scale.per_thread_ops 136 ~ 3% -43.1% 77 ~43% TOTAL proc-vmstat.nr_dirtied 0.51 ~ 3% +98.0% 1.01 ~ 4% TOTAL perf-profile.cpu-cycles.shmem_write_end.generic_perform_write.__generic_file_aio_write.generic_file_aio_write.do_sync_write 1078 ~ 9% -16.3% 903 ~11% TOTAL numa-meminfo.node0.Unevictable 269 ~ 9% -16.2% 225 ~11% TOTAL numa-vmstat.node0.nr_unevictable 1.64 ~ 1% -14.3% 1.41 ~ 4% TOTAL perf-profile.cpu-cycles.find_lock_entry.shmem_getpage_gfp.shmem_write_begin.generic_perform_write.__generic_file_aio_write 1.62 ~ 2% +14.1% 1.84 ~ 1% TOTAL perf-profile.cpu-cycles.lseek64The perf-profile.cpu-cycles.* lines are from "perf record/report".
The last line shows that
lseek64()takes 1.62% CPU cycles for commit2074b6e38668e62and that percent increased by +14.1% ona0f9ec1f181. One of the raw perf record output is1.84% writeseek_proce libc-2.17.so [.] lseek64 | --- lseek64There are 5 runs and 1.62% is the average value.
Quoted 7 lines
Quoted 3 lines
I have no idea how to read the above. Which direction is plus and which is minus? Are they counting cpu cycles? Which files is the test seeking?
It's tmpfs files. Because the will-it-scale test case is mean to measure scalability of syscalls. We do not use HDD/SSD etc. storage devices when running it.
Hmmm... I'm completely stumped. The commit in question has nothing to do with tmpfs. It only affects three cgroup files - "tasks", "cgroup.procs" and "release_agent". It can't possibly have any effect on tmpfs operation. Maybe random effect through code alignment? Even that is highly unlikely. I'll look into it tomorrow but can you please try to repeat the test? It really doesn't make any sense to me.
Thanks.
Signature · 1 line
tejun
Quoted 5 lines · Tejun Heo
Hmmm... I'm completely stumped. The commit in question has nothing to do with tmpfs. It only affects three cgroup files - "tasks", "cgroup.procs" and "
release_agent". It can't possibly have any effect on tmpfs operation. Maybe random effect through code alignment? Even that is highly unlikely.
Unlikely, but it happens. I bisected a tbench regression down to a completely unrelated driver change, and watched it repeatably drop throughput 5% despite skeptical "No way, show me that trick again".
-Mike
Quoted 37 lines · Tejun Heo
Hello, Fengguang.
Quoted 26 lines · Fengguang Wu
Quoted 2 lines
Quoted 2 lines
2074b6e38668e62a0f9ec1f181534694cb5bf40b
--------------- -------------------------
2074b6e38668e62is the base of comparison. So "-4.3% will-it-scale.per_thread_ops" in the below line meansa0f9ec1f18has lower will-it-scale throughput.Quoted 7 lines
Quoted 7 lines
1027273 ~ 0% -4.3% 982732 ~ 0% TOTAL will-it-scale.per_thread_ops 136 ~ 3% -43.1% 77 ~43% TOTAL proc-vmstat.nr_dirtied 0.51 ~ 3% +98.0% 1.01 ~ 4% TOTAL perf-profile.cpu-cycles.shmem_write_end.generic_perform_write.__generic_file_aio_write.generic_file_aio_write.do_sync_write 1078 ~ 9% -16.3% 903 ~11% TOTAL numa-meminfo.node0.Unevictable 269 ~ 9% -16.2% 225 ~11% TOTAL numa-vmstat.node0.nr_unevictable 1.64 ~ 1% -14.3% 1.41 ~ 4% TOTAL perf-profile.cpu-cycles.find_lock_entry.shmem_getpage_gfp.shmem_write_begin.generic_perform_write.__generic_file_aio_write 1.62 ~ 2% +14.1% 1.84 ~ 1% TOTAL perf-profile.cpu-cycles.lseek64The perf-profile.cpu-cycles.* lines are from "perf record/report".
The last line shows that
lseek64()takes 1.62% CPU cycles for commit2074b6e38668e62and that percent increased by +14.1% ona0f9ec1f181. One of the raw perf record output is1.84% writeseek_proce libc-2.17.so [.] lseek64 | --- lseek64There are 5 runs and 1.62% is the average value.
Quoted 7 lines
Quoted 3 lines
I have no idea how to read the above. Which direction is plus and which is minus? Are they counting cpu cycles? Which files is the test seeking?
It's tmpfs files. Because the will-it-scale test case is mean to measure scalability of syscalls. We do not use HDD/SSD etc. storage devices when running it.
Quoted 7 lines
Hmmm... I'm completely stumped. The commit in question has nothing to do with tmpfs. It only affects three cgroup files - "tasks", "cgroup.procs" and "
release_agent". It can't possibly have any effect on tmpfs operation. Maybe random effect through code alignment? Even that is highly unlikely. I'll look into it tomorrow but can you please try to repeat the test? It really doesn't make any sense to me.
Yes, sorry! Even though the "first bad" commit a0f9ec1f1 and its parent commit 2074b6e38 has clear and stable performance changes:
5 runs of a0f9ec1f1:
"will-it-scale.per_thread_ops": [
983098,
985112,
982690,
976157,
986606],
5 runs of 2074b6e38:
"will-it-scale.per_thread_ops": [
1027667,
1029414,
1026736,
1025678,
1026871],
Comparing the bisect-good and bisect-bad *kernels*, you'll find the performance changes are not as stable:
will-it-scale.per_thread_ops
1.14e+06 ++---------------------------------------------------------------+
1.12e+06 ++ *.. |
| : * |
1.1e+06 ++ : : |
1.08e+06 ++ : : |
| : : |
1.06e+06 ++ : : |
1.04e+06 *+.*...*..*..*..*...*..*.. : : ..*..*.. |
1.02e+06 ++ O *..* *..*. *..*...*..*..*
| O O |
1e+06 O+ O O O O O |
980000 ++ O O O O O O O |
| |
960000 ++ O O |940000 ++---------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad sample
So it might be some subtle data padding/alignment issue.
Thanks,
Fengguang