Application performance: regressions, controlling preemption
Hi everyone,
We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.
I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.
The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.
The KS dates should put us right at the end of our regression hunt, I can talk through the main problems we hit, how (if) we fixed them and hopefully offer tests to keep them from coming back.
We've split the tiers up between Josef Bacik, Jens Axboe, myself and a few others. I'd nominate Josef and Jens to come share their findings as well.
Another topic here is controlling preemption from userland without needing to go full real time. CPU intensive in-memory databases are leaving a ton of performance on the floor in context switches. I'm hoping to experiment with better preemption controls and the userland RCU project to improve things.
-chris
Quoted 11 lines · Chris Mason
Hi everyone,
We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.
I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.
The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.
Please consider me for this discussion. At HP we've been working on different performance issues in the kernel, including a lot of locking improvements and optimizations, which have boosted performance for a wide variety of workloads. There are plenty of really interesting results and I'd be happy to share some of them and other findings.
Furthermore, I'm always on the lookout for new performance issues so I'd be interested in knowing some of the seen regressions and if they are still present today (lots of changes since 3.10).
Quoted 7 lines
The KS dates should put us right at the end of our regression hunt, I can talk through the main problems we hit, how (if) we fixed them and hopefully offer tests to keep them from coming back.
We've split the tiers up between Josef Bacik, Jens Axboe, myself and a few others. I'd nominate Josef and Jens to come share their findings as well.
I'd also like to include PeterZ and Ingo.
Quoted 5 lines
Another topic here is controlling preemption from userland without needing to go full real time. CPU intensive in-memory databases are leaving a ton of performance on the floor in context switches. I'm hoping to experiment with better preemption controls and the userland RCU project to improve things.
Hmm specifically for this, it would also be good to have Khalid Aziz in the discussion. He's been trying to add userspace controlled preemption for a while now.
Thanks,
Davidlohr
Hello,
Quoted 10 lines · Chris Mason
We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.
I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.
The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.
I'd be interested in this discussion as well. We are in a similar
situation in SUSE now when moving our enterprise kernel from 3.0 to 3.12 base. So me, Jiri Kosina, or Mel Gorman can fill in things we found and you didn't ;) Plus we are slowly working on setting up some more systematic and continuous way of tracking performance in kernels so sharing experiences with what is useful to run would be interesting as well.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 15 lines · Chris Mason
Hi everyone,
We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.
I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.
The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.
The KS dates should put us right at the end of our regression hunt, I can talk through the main problems we hit, how (if) we fixed them and hopefully offer tests to keep them from coming back.
Quoted 8 lines
We've split the tiers up between Josef Bacik, Jens Axboe, myself and a few others. I'd nominate Josef and Jens to come share their findings as well.
Another topic here is controlling preemption from userland without needing to go full real time. CPU intensive in-memory databases are leaving a ton of performance on the floor in context switches. I'm hoping to experiment with better preemption controls and the userland RCU project to improve things.
I wonder how much of this overhead is the actual interrupt + context switch and how much is overhead due to caching effects. If the former, there may be value in having a simple benchmark for how long preemption takes and running it periodically.
--Andy
Quoted 11 lines · Chris Mason
Hi everyone,
We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.
I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.
The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.
How long have we been having this discussion? 8 years? It's not like people don't know that performance testing needs to be constantly happening, we've been saying that for a long time. It's just that no one seems to listen to us :(
And that is the larger problem, what can we do about that issue. Honestly, I don't think much, as it takes money from companies to commit to do this work, which no one seems to ever want to do. What make this year the year that something different happens?
greg k-h
Quoted 10 lines · Greg KH
Quoted 9 lines · Chris Mason
Hi everyone,
We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.
I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.
Quoted 8 lines
Quoted 2 lines
The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.
How long have we been having this discussion? 8 years? It's not like people don't know that performance testing needs to be constantly happening, we've been saying that for a long time. It's just that no one seems to listen to us :(
Yes and no. Intel listened, and I think they have had a huge positive impact here. Others have as well, maybe not as consistently, but still.
I do find it really interesting that even with huge improvements all over the place, we have a very hard time upgrading production workloads without hitting big regressions.
Sometimes it is just a few .config entries, and sometimes we paper over it with improvements in other areas, but it's almost always worse.
Quoted 4 lines
And that is the larger problem, what can we do about that issue. Honestly, I don't think much, as it takes money from companies to commit to do this work, which no one seems to ever want to do. What make this year the year that something different happens?
I can't promise different. At best we'll tease out some new benchmarks and try to get them into a format that get run often.
-chris
Quoted 20 lines · Chris Mason
Quoted 10 lines · Greg KH
Quoted 9 lines · Chris Mason
Hi everyone,
We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.
I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.
Quoted 8 lines
Quoted 2 lines
The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.
How long have we been having this discussion? 8 years? It's not like people don't know that performance testing needs to be constantly happening, we've been saying that for a long time. It's just that no one seems to listen to us :(
Quoted 8 lines
Yes and no. Intel listened, and I think they have had a huge positive impact here. Others have as well, maybe not as consistently, but still.
I do find it really interesting that even with huge improvements all over the place, we have a very hard time upgrading production workloads without hitting big regressions.
Sometimes it is just a few .config entries,
Do you mean some regressions are introduced by new configs i.e. new features? If so, I don't think that can be strictly called regressions.
Quoted 3 lines
and sometimes we paper over
it with improvements in other areas, but it's almost always worse.
Quoted 24 lines · Li Zefan
Quoted 23 lines · Chris Mason
Quoted 10 lines · Greg KH
Quoted 9 lines · Chris Mason
Hi everyone,
We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.
I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.
Quoted 8 lines
Quoted 2 lines
The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.
How long have we been having this discussion? 8 years? It's not like people don't know that performance testing needs to be constantly happening, we've been saying that for a long time. It's just that no one seems to listen to us :(
Yes and no. Intel listened, and I think they have had a huge positive impact here. Others have as well, maybe not as consistently, but still.
Quoted 9 lines
Quoted 5 lines
I do find it really interesting that even with huge improvements all over the place, we have a very hard time upgrading production workloads without hitting big regressions.
Sometimes it is just a few .config entries,
Do you mean some regressions are introduced by new configs i.e. new features? If so, I don't think that can be strictly called regressions.
Agreed, at least unless the .config choice that makes it slow is the new default. +/- slub, I haven't hit this case.
-chris
Quoted 10 lines · Greg KH
Quoted 9 lines · Chris Mason
Hi everyone,
We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.
I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.
Quoted 7 lines
Quoted 2 lines
The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.
How long have we been having this discussion? 8 years? It's not like people don't know that performance testing needs to be constantly happening, we've been saying that for a long time. It's just that no one seems to listen to us :(
I agree with you that this is coming over and over again for quite a few
years.
Quoted 4 lines
And that is the larger problem, what can we do about that issue. Honestly, I don't think much, as it takes money from companies to commit to do this work, which no one seems to ever want to do. What make this year the year that something different happens?
So what I found interesting about the topic is:
1) What has regressed this time in particular? This might be interesting for a broader audience I believe. With my SUSE hat on, I can learn which problems should I expect our customers to see that we didn't find during our testing. With my community hat on, I can possibly learn some things which are easy enough to test for...
2) We are looking into some more continuous testing in SUSE so learning what others have tested would be interesting for me.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 10 lines · Chris Mason
Hi everyone,
We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.
I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.
The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.
How automated are your benchmark workloads, how long do they take, and how consistent are they from run to run (on a system running nothing else)? What about getting them into Fengguang Wu's automated patch checker, or a similar system that checks every patch or pull rather than just full releases? If we had feedback at the time of patch submission that a specific patch made the kernel x% slower for a specific well-defined workload, that would prove much easier to act on than just a comparison of 3.x and 3.y.
- Josh Triplett
Quoted 12 lines · Josh Triplett
Quoted 10 lines · Chris Mason
Hi everyone,
We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.
I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.
The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.
Quoted 8 lines
How automated are your benchmark workloads, how long do they take, and how consistent are they from run to run (on a system running nothing else)? What about getting them into Fengguang Wu's automated patch checker, or a similar system that checks every patch or pull rather than just full releases? If we had feedback at the time of patch submission that a specific patch made the kernel x% slower for a specific well-defined workload, that would prove much easier to act on than just a comparison of 3.x and 3.y.
This sounds ideal, but reality is very very different.
Fengguang's scripts are quite nice and work for a number of scenarios, but cannot possibly cover everything. And the regressions Chris mentions are quite common, depending what and where you're looking at. Just consider proprietary tools and benchmarks (ie: Oracle -- and no, I'm not talking about pgbench only). Or just about anything that's not synthetic and easy to setup (ie: Hadoop). Subtle architecture specific changes (ie: non x86) are also beyond this scope and can trigger major performance regressions. And even common benchmarks and systems such as aim7 (which I know Fengguang runs) and x86 can bypass the automated checks, just look at https://lkml.org/lkml/2014/3/17/587.
There are just too many variables to control.
That said, I do agree that we could do better, and yeah, adding more workloads to Fengguang's scripts are always a good thing -- perhaps even adding stuff from perf-bench.
Thanks,
Davidlohr
Quoted 22 lines · Davidlohr Bueso
Quoted 12 lines · Josh Triplett
Quoted 10 lines · Chris Mason
Hi everyone,
We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.
I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.
The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.
Quoted 8 lines
How automated are your benchmark workloads, how long do they take, and how consistent are they from run to run (on a system running nothing else)? What about getting them into Fengguang Wu's automated patch checker, or a similar system that checks every patch or pull rather than just full releases? If we had feedback at the time of patch submission that a specific patch made the kernel x% slower for a specific well-defined workload, that would prove much easier to act on than just a comparison of 3.x and 3.y.
Quoted 4 lines
This sounds ideal, but reality is very very different.
Fengguang's scripts are quite nice and work for a number of scenarios, but cannot possibly cover everything.
Sorry for being late.. Yup, test coverage is a huge challenge and I believe collaborations are the key to make substantial progresses.
Intel OTC has been running a LKP (Linux Kernel Performance) project which does boot, functional, performance and power tests over the community kernel git trees. Some diligent hackers (Hi Paul!) can occasionally trigger our regression reports. We believe it could potentially be a tool for more developers to evaluate performance/power of their wip patches, in a more direct and manageable way.
So we are excited to share LKP test cases with the community in GPLv2:
git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests
It's still missing the documents part -- when ready, I'll make a public announce in LKML. Basically it enables a kernel developer to run LKP tests in his own test box and generate/compare test results like this:
https://lists.01.org/pipermail/lkp/2014-July/000324.html
The proc-vmstat, perf-profile, cpuidle, turbostat etc. "monitors" are inspired by Mel Gorman's mmtests suite and they are really helpful in catching&analyzing the subtle impacts a patch might bring to the system.
Quoted 10 lines
And the regressions Chris mentions
are quite common, depending what and where you're looking at. Just consider proprietary tools and benchmarks (ie: Oracle -- and no, I'm not talking about pgbench only). Or just about anything that's not synthetic and easy to setup (ie: Hadoop). Subtle architecture specific changes (ie: non x86) are also beyond this scope and can trigger major performance regressions. And even common benchmarks and systems such as aim7 (which I know Fengguang runs) and x86 can bypass the automated checks, just look at https://lkml.org/lkml/2014/3/17/587.
There are just too many variables to control.
Yes, there are often the need to test combinations of parameters. In LKP, we make it convenient to define "matrix" test jobs like:
fio:
rw:
- randwrite
- randrw
ioengine:
- sync
- mmap
bs:
- 4k
- 64k
Which will be split into 2*2*2 unit jobs for execution. For example, the first unit job is:
fio:
rw: randwrite
ioengine: sync
bs: 4k
Quoted 3 lines
That said, I do agree that we could do better, and yeah, adding more workloads to Fengguang's scripts are always a good thing -- perhaps even adding stuff from perf-bench.
You are very welcome to add new cases, monitors or setup scripts! Depending on their nature and resource requirement, we may choose the adequate policy to run them in our LKP test infrastructure -- which works 7x24 on the fresh new code in 400+ kernel git trees. By feeding it more test cases, we may reasonably safeguard more kernel code and use scenarios from _silent_ regressions in future.
Thanks,
Fengguang
Quoted 3 lines · Fengguang Wu
The proc-vmstat, perf-profile, cpuidle, turbostat etc. "monitors" are inspired by Mel Gorman's mmtests suite and they are really helpful in catching&analyzing the subtle impacts a patch might bring to the system.
Mel also has some interesting tests in the suite like the page fault test and aim9 among numerous others. Could those also be added?
Quoted 8 lines · Christoph Lameter
Quoted 3 lines · Fengguang Wu
The proc-vmstat, perf-profile, cpuidle, turbostat etc. "monitors" are inspired by Mel Gorman's mmtests suite and they are really helpful in catching&analyzing the subtle impacts a patch might bring to the system.
Mel also has some interesting tests in the suite like the page fault test and aim9 among numerous others. Could those also be added?
Sure. I just added aim9 to the TODO list.
pft is already included in lkp-tests:
% ls */pftjobs/pft.yaml pack/pft pack/pft.patch stats/pft tests/pft
Thanks,
Fengguang
Quoted 10 lines · Fengguang Wu
Quoted 8 lines · Christoph Lameter
Quoted 3 lines · Fengguang Wu
The proc-vmstat, perf-profile, cpuidle, turbostat etc. "monitors" are inspired by Mel Gorman's mmtests suite and they are really helpful in catching&analyzing the subtle impacts a patch might bring to the system.
Mel also has some interesting tests in the suite like the page fault test and aim9 among numerous others. Could those also be added?
Quoted 6 lines
Sure. I just added aim9 to the TODO list.
pft is already included in lkp-tests:
% ls */pft
jobs/pft.yamlpack/pftpack/pft.patchstats/pft tests/pft
Are the test results somewhere publicly available?
[Ksummit-discuss] 0day kernel performance/power test service
Quoted 12 lines · Christoph Lameter
Quoted 10 lines · Fengguang Wu
Quoted 8 lines · Christoph Lameter
Quoted 3 lines · Fengguang Wu
The proc-vmstat, perf-profile, cpuidle, turbostat etc. "monitors" are inspired by Mel Gorman's mmtests suite and they are really helpful in catching&analyzing the subtle impacts a patch might bring to the system.
Mel also has some interesting tests in the suite like the page fault test and aim9 among numerous others. Could those also be added?
Quoted 8 lines
Quoted 6 lines
Sure. I just added aim9 to the TODO list.
pft is already included in lkp-tests:
% ls */pft
jobs/pft.yamlpack/pftpack/pft.patchstats/pft tests/pftAre the test results somewhere publicly available?
Not available yet.. Sorry! For now I'm afraid I can only send tar balls of raw results on your request.
btw, we'll offer an experimental 0day performance/power testing service for the core maintainer trees. Just send me a lkp-tests patch (attached an example for ext4:dev) to selectively run a set of test jobs on a set of machines for your tree:branch. Future git push will then trigger the test set and you'll receive the report that compares the branch BASE:HEAD performance in 24 hours.
The patch is basically selecting a combination of test jobs and machines. All the available test jobs and machines are included in lkp-tests. You are also welcome to add new jobs to the pool.
wfg /c/lkp-tests% ls hosts
bay ivb43 lkp-nex04 lkp-sb03 lkp-t410 loslunas vm-kbuild-2G vm-kbuild-yocto-x86_64 vm-vp-quantal-x86_64 xps2 bens ivb44 lkp-nex05 lkp-sbx04 lkp-ws02 nhm4 vm-kbuild-4G vm-vp-1G wsm brickland1 lkp-ne02 lkp-nex06 lkp-snb01 lkp-wsx01 nhm-white vm-kbuild-yocto-i386 vm-vp-2G xbm brickland3 lkp-ne04 lkp-sb02 lkp-st02 lkp-wsx02 snb-drag vm-kbuild-yocto-ia32 vm-vp-quantal-i386 xps
wfg /c/lkp-tests% ls jobs
blogbench.yaml dd-write.yaml hackbench-8.yaml netperf.yaml tbench.yaml boot.yaml debug.yaml hackbench-cpuset.yaml nfs-iozone.yaml tcrypt.yaml borrow-1d.yaml DEFAULTS hackbench-perf.yaml nuttcp.yaml thrulay.yaml borrow-1h.yaml ebizzy.yaml hackbench.yaml oltp.yaml tlbflush.yaml dbench.yaml fileio.yaml idle.yaml packetdrill.yaml unixbench.yaml dd-write-11hdd.yaml fio-jbod-12hdd-randwrite-sync-4k.yaml iozone.yaml perf-bench-numa-mem.yaml vm-scalability-numa.yaml dd-write-1hdd-fuse.yaml fio-jbod-12hdd.yaml iperf.yaml perf-bench-sched-pipe.yaml vm-scalability.yaml dd-write-1hdd.yaml fsmark.yaml kbuild.yaml pft.yaml will-it-scale.yaml dd-write-1ssd.yaml ftq.yaml kernel_selftests.yaml piglit.yaml xfstests-btrfs.yaml dd-write-4hdd-fuse.yaml fwq.yaml linpack.yaml pigz.yaml xfstests-ext4.yaml dd-write-4hdd.yaml glbenchmark.yaml ltp.yaml qperf.yaml xfstests-generic.yaml dd-write-hdd-usb.yaml hackbench-6.yaml nepim.yaml sockperf.yaml xfstests-xfs.yaml
Thanks,
Fengguang
Quoted 1 line
From
8cd384c2f34f593d829457e7043ee6cca204bda3Mon Sep 17 00:00:00 2001
From: Fengguang Wu <[email protected]>
Date: Sat, 16 Aug 2014 09:12:35 +0800
Subject: [PATCH] define 0day performance test set for ext4/dev
This set of test jobs will be auto triggered by git push and email report should be sent to the committer in 1 day.
--- allot/ext4:dev/lkp-ne04/fsmark-ext4.yaml | 27 ++++++++++++++++++++++ allot/ext4:dev/lkp-st02/dd-write-11hdd-ext4.yaml | 27 ++++++++++++++++++++++ .../vm-kbuild-2G/xfstests-generic-ext4.yaml | 16 +++++++++++++ allot/ext4:dev/vm-vp-1G/xfstests-ext4.yaml | 1 + 4 files changed, 71 insertions(+) create mode 100644 allot/ext4:dev/lkp-ne04/fsmark-ext4.yaml create mode 100644 allot/ext4:dev/lkp-st02/dd-write-11hdd-ext4.yaml create mode 100644 allot/ext4:dev/vm-kbuild-2G/xfstests-generic-ext4.yaml create mode 120000 allot/ext4:dev/vm-vp-1G/xfstests-ext4.yaml
allot/ext4:dev/lkp-ne04/fsmark-ext4.yaml new file +27 -0
@@ -0,0 +1,27 @@ +testcase: fsmark + +iterations: 1x +nr_threads: 32t + +disk: 1HDD +fs: +- ext4 + +fsmark: + filesize: 16MB + test_size: 60G + sync_method: + - NoSync + - fsyncBeforeClose + nr_directories: 16d + nr_files_per_directory: 256f + +--- +fsmark: + filesize: + - 8K + - 9B + test_size: 400M + sync_method: fsyncBeforeClose + nr_directories: 16d + nr_files_per_directory: 256f
allot/ext4:dev/lkp-st02/dd-write-11hdd-ext4.yaml new file +27 -0
@@ -0,0 +1,27 @@ +testcase: dd-write + +runtime: 5m + +disk: 11HDD +md: +- JBOD +- RAID5 +iosched: +- cfq +fs: +- ext4 + +monitors: + perf-stat: + perf-profile: + ftrace: + - events: + balance_dirty_pages + bdi_dirty_ratelimit + global_dirty_state + writeback_single_inode + +nr_threads: +- 100dd + +dd:
allot/ext4:dev/vm-kbuild-2G/xfstests-generic-ext4.yaml new file +16 -0
@@ -0,0 +1,16 @@ +testcase: xfstests + +default_monitors: + +disk: 4HDD + +fs: +- ext4 + +xfstests: + test: + - generic-113 + - generic-slow1 + - generic-slow2 + - generic-mid + - generic-quick
allot/ext4:dev/vm-vp-1G/xfstests-ext4.yaml new file +1 -0
@@ -0,0 +1 @@ +../../../jobs/xfstests-ext4.yaml
Signature · 1 line
2.1.0.rc1
Quoted 12 lines · Josh Triplett
Quoted 10 lines · Chris Mason
Hi everyone,
We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.
I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.
The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.
Quoted 8 lines
How automated are your benchmark workloads, how long do they take, and how consistent are they from run to run (on a system running nothing else)? What about getting them into Fengguang Wu's automated patch checker, or a similar system that checks every patch or pull rather than just full releases? If we had feedback at the time of patch submission that a specific patch made the kernel x% slower for a specific well-defined workload, that would prove much easier to act on than just a comparison of 3.x and 3.y.
Good point on Fengguang's checker! I do sometimes get emails from him noting changes in performance, conveniently bisected down to the commit.
Thanx, Paul
Hi Chris,
Quoted 3 lines
The KS dates should put us right at the end of our regression hunt, I can talk through the main problems we hit, how (if) we fixed them and hopefully offer tests to keep them from coming back.
It'd be sweet if your tests (and others') can be added to
git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests
Which runs in Intel actively to keep the relevant regressions from coming back.
Currently the repository has test cases
wfg /c/lkp-tests% ls tests
aim7 debug-test fio-jbod glbenchmark kbuild nepim packetdrill piglit sockperf tlbflush wrapper blogbench ebizzy fsmark hackbench kernel_selftests netperf perf-bench-numa-mem pigz tbench unixbench xfstests dbench fileio ftq iozone linpack nuttcp perf-bench-sched-pipe qperf tcrypt vm-scalability dd fio fwq iperf ltp oltp pft sleep thrulay will-it-scale
Which is not only too few in number, but also rather limited in test parameters and system setups. For example, if run them in different cgroup/numa/governor/network/... setups, or even with different kernel configs, the results may become quite different.
Thanks,
Fengguang