Skip to contentSkip to conversation

Application performance: regressions, controlling preemption

Ksummit-discuss Chris Mason 10 participants 18 messages May 12, 2014 – Aug 18, 2014

lore
Share & tools
lore
#0 lore raw

Hi everyone,

We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.

I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.

The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.

The KS dates should put us right at the end of our regression hunt, I can talk through the main problems we hit, how (if) we fixed them and hopefully offer tests to keep them from coming back.

We've split the tiers up between Josef Bacik, Jens Axboe, myself and a few others. I'd nominate Josef and Jens to come share their findings as well.

Another topic here is controlling preemption from userland without needing to go full real time. CPU intensive in-memory databases are leaving a ton of performance on the floor in context switches. I'm hoping to experiment with better preemption controls and the userland RCU project to improve things.

-chris

#1 lore raw
Quoted 11 lines · Chris Mason

Hi everyone,

We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.

I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.

The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.

Please consider me for this discussion. At HP we've been working on different performance issues in the kernel, including a lot of locking improvements and optimizations, which have boosted performance for a wide variety of workloads. There are plenty of really interesting results and I'd be happy to share some of them and other findings.

Furthermore, I'm always on the lookout for new performance issues so I'd be interested in knowing some of the seen regressions and if they are still present today (lots of changes since 3.10).

Quoted 7 lines

The KS dates should put us right at the end of our regression hunt, I can talk through the main problems we hit, how (if) we fixed them and hopefully offer tests to keep them from coming back.

We've split the tiers up between Josef Bacik, Jens Axboe, myself and a few others. I'd nominate Josef and Jens to come share their findings as well.

I'd also like to include PeterZ and Ingo.

Quoted 5 lines

Another topic here is controlling preemption from userland without needing to go full real time. CPU intensive in-memory databases are leaving a ton of performance on the floor in context switches. I'm hoping to experiment with better preemption controls and the userland RCU project to improve things.

Hmm specifically for this, it would also be good to have Khalid Aziz in the discussion. He's been trying to add userspace controlled preemption for a while now.

Thanks,
Davidlohr

#2 lore raw
In reply to Chris Mason

Hello,

Quoted 10 lines · Chris Mason

We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.

I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.

The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.

I'd be interested in this discussion as well. We are in a similar
situation in SUSE now when moving our enterprise kernel from 3.0 to 3.12 base. So me, Jiri Kosina, or Mel Gorman can fill in things we found and you didn't ;) Plus we are slowly working on setting up some more systematic and continuous way of tracking performance in kernels so sharing experiences with what is useful to run would be interesting as well.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#3 lore raw
In reply to Chris Mason
Quoted 15 lines · Chris Mason

Hi everyone,

We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.

I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.

The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.

The KS dates should put us right at the end of our regression hunt, I can talk through the main problems we hit, how (if) we fixed them and hopefully offer tests to keep them from coming back.

Quoted 8 lines

We've split the tiers up between Josef Bacik, Jens Axboe, myself and a few others. I'd nominate Josef and Jens to come share their findings as well.

Another topic here is controlling preemption from userland without needing to go full real time. CPU intensive in-memory databases are leaving a ton of performance on the floor in context switches. I'm hoping to experiment with better preemption controls and the userland RCU project to improve things.

I wonder how much of this overhead is the actual interrupt + context switch and how much is overhead due to caching effects. If the former, there may be value in having a simple benchmark for how long preemption takes and running it periodically.

--Andy

#4 lore raw
In reply to Chris Mason
Quoted 11 lines · Chris Mason

Hi everyone,

We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.

I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.

The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.

How long have we been having this discussion? 8 years? It's not like people don't know that performance testing needs to be constantly happening, we've been saying that for a long time. It's just that no one seems to listen to us :(

And that is the larger problem, what can we do about that issue. Honestly, I don't think much, as it takes money from companies to commit to do this work, which no one seems to ever want to do. What make this year the year that something different happens?

greg k-h

#5 lore raw
Quoted 10 lines · Greg KH
Quoted 9 lines · Chris Mason

Hi everyone,

We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.

I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.

Quoted 8 lines
Quoted 2 lines

The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.

How long have we been having this discussion? 8 years? It's not like people don't know that performance testing needs to be constantly happening, we've been saying that for a long time. It's just that no one seems to listen to us :(

Yes and no. Intel listened, and I think they have had a huge positive impact here. Others have as well, maybe not as consistently, but still.

I do find it really interesting that even with huge improvements all over the place, we have a very hard time upgrading production workloads without hitting big regressions.

Sometimes it is just a few .config entries, and sometimes we paper over it with improvements in other areas, but it's almost always worse.

Quoted 4 lines

And that is the larger problem, what can we do about that issue. Honestly, I don't think much, as it takes money from companies to commit to do this work, which no one seems to ever want to do. What make this year the year that something different happens?

I can't promise different. At best we'll tease out some new benchmarks and try to get them into a format that get run often.

-chris

#6 lore raw
Quoted 20 lines · Chris Mason
Quoted 10 lines · Greg KH
Quoted 9 lines · Chris Mason

Hi everyone,

We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.

I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.

Quoted 8 lines
Quoted 2 lines

The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.

How long have we been having this discussion? 8 years? It's not like people don't know that performance testing needs to be constantly happening, we've been saying that for a long time. It's just that no one seems to listen to us :(

Quoted 8 lines

Yes and no. Intel listened, and I think they have had a huge positive impact here. Others have as well, maybe not as consistently, but still.

I do find it really interesting that even with huge improvements all over the place, we have a very hard time upgrading production workloads without hitting big regressions.

Sometimes it is just a few .config entries,

Do you mean some regressions are introduced by new configs i.e. new features? If so, I don't think that can be strictly called regressions.

Quoted 3 lines

and sometimes we paper over
it with improvements in other areas, but it's almost always worse.

#7 lore raw
Quoted 24 lines · Li Zefan
Quoted 23 lines · Chris Mason
Quoted 10 lines · Greg KH
Quoted 9 lines · Chris Mason

Hi everyone,

We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.

I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.

Quoted 8 lines
Quoted 2 lines

The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.

How long have we been having this discussion? 8 years? It's not like people don't know that performance testing needs to be constantly happening, we've been saying that for a long time. It's just that no one seems to listen to us :(

Yes and no. Intel listened, and I think they have had a huge positive impact here. Others have as well, maybe not as consistently, but still.

Quoted 9 lines
Quoted 5 lines

I do find it really interesting that even with huge improvements all over the place, we have a very hard time upgrading production workloads without hitting big regressions.

Sometimes it is just a few .config entries,

Do you mean some regressions are introduced by new configs i.e. new features? If so, I don't think that can be strictly called regressions.

Agreed, at least unless the .config choice that makes it slow is the new default. +/- slub, I haven't hit this case.

-chris

#8 lore raw
In reply to Greg KH
Quoted 10 lines · Greg KH
Quoted 9 lines · Chris Mason

Hi everyone,

We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.

I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.

Quoted 7 lines
Quoted 2 lines

The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.

How long have we been having this discussion? 8 years? It's not like people don't know that performance testing needs to be constantly happening, we've been saying that for a long time. It's just that no one seems to listen to us :(

I agree with you that this is coming over and over again for quite a few
years.

Quoted 4 lines

And that is the larger problem, what can we do about that issue. Honestly, I don't think much, as it takes money from companies to commit to do this work, which no one seems to ever want to do. What make this year the year that something different happens?

So what I found interesting about the topic is:
1) What has regressed this time in particular? This might be interesting for a broader audience I believe. With my SUSE hat on, I can learn which problems should I expect our customers to see that we didn't find during our testing. With my community hat on, I can possibly learn some things which are easy enough to test for...

2) We are looking into some more continuous testing in SUSE so learning what others have tested would be interesting for me.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#9 lore raw
In reply to Chris Mason
Quoted 10 lines · Chris Mason

Hi everyone,

We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.

I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.

The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.

How automated are your benchmark workloads, how long do they take, and how consistent are they from run to run (on a system running nothing else)? What about getting them into Fengguang Wu's automated patch checker, or a similar system that checks every patch or pull rather than just full releases? If we had feedback at the time of patch submission that a specific patch made the kernel x% slower for a specific well-defined workload, that would prove much easier to act on than just a comparison of 3.x and 3.y.

- Josh Triplett

#10 lore raw
Quoted 12 lines · Josh Triplett
Quoted 10 lines · Chris Mason

Hi everyone,

We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.

I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.

The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.

Quoted 8 lines

How automated are your benchmark workloads, how long do they take, and how consistent are they from run to run (on a system running nothing else)? What about getting them into Fengguang Wu's automated patch checker, or a similar system that checks every patch or pull rather than just full releases? If we had feedback at the time of patch submission that a specific patch made the kernel x% slower for a specific well-defined workload, that would prove much easier to act on than just a comparison of 3.x and 3.y.

This sounds ideal, but reality is very very different.

Fengguang's scripts are quite nice and work for a number of scenarios, but cannot possibly cover everything. And the regressions Chris mentions are quite common, depending what and where you're looking at. Just consider proprietary tools and benchmarks (ie: Oracle -- and no, I'm not talking about pgbench only). Or just about anything that's not synthetic and easy to setup (ie: Hadoop). Subtle architecture specific changes (ie: non x86) are also beyond this scope and can trigger major performance regressions. And even common benchmarks and systems such as aim7 (which I know Fengguang runs) and x86 can bypass the automated checks, just look at https://lkml.org/lkml/2014/3/17/587.
There are just too many variables to control.

That said, I do agree that we could do better, and yeah, adding more workloads to Fengguang's scripts are always a good thing -- perhaps even adding stuff from perf-bench.

Thanks,
Davidlohr

#11 lore raw
Quoted 22 lines · Davidlohr Bueso
Quoted 12 lines · Josh Triplett
Quoted 10 lines · Chris Mason

Hi everyone,

We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.

I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.

The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.

Quoted 8 lines

How automated are your benchmark workloads, how long do they take, and how consistent are they from run to run (on a system running nothing else)? What about getting them into Fengguang Wu's automated patch checker, or a similar system that checks every patch or pull rather than just full releases? If we had feedback at the time of patch submission that a specific patch made the kernel x% slower for a specific well-defined workload, that would prove much easier to act on than just a comparison of 3.x and 3.y.

Quoted 4 lines

This sounds ideal, but reality is very very different.

Fengguang's scripts are quite nice and work for a number of scenarios, but cannot possibly cover everything.

Sorry for being late.. Yup, test coverage is a huge challenge and I believe collaborations are the key to make substantial progresses.

Intel OTC has been running a LKP (Linux Kernel Performance) project which does boot, functional, performance and power tests over the community kernel git trees. Some diligent hackers (Hi Paul!) can occasionally trigger our regression reports. We believe it could potentially be a tool for more developers to evaluate performance/power of their wip patches, in a more direct and manageable way.

So we are excited to share LKP test cases with the community in GPLv2:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests

It's still missing the documents part -- when ready, I'll make a public announce in LKML. Basically it enables a kernel developer to run LKP tests in his own test box and generate/compare test results like this:

        https://lists.01.org/pipermail/lkp/2014-July/000324.html

The proc-vmstat, perf-profile, cpuidle, turbostat etc. "monitors" are inspired by Mel Gorman's mmtests suite and they are really helpful in catching&analyzing the subtle impacts a patch might bring to the system.

Quoted 10 lines

And the regressions Chris mentions
are quite common, depending what and where you're looking at. Just consider proprietary tools and benchmarks (ie: Oracle -- and no, I'm not talking about pgbench only). Or just about anything that's not synthetic and easy to setup (ie: Hadoop). Subtle architecture specific changes (ie: non x86) are also beyond this scope and can trigger major performance regressions. And even common benchmarks and systems such as aim7 (which I know Fengguang runs) and x86 can bypass the automated checks, just look at https://lkml.org/lkml/2014/3/17/587.
There are just too many variables to control.

Yes, there are often the need to test combinations of parameters. In LKP, we make it convenient to define "matrix" test jobs like:

fio:
rw:
- randwrite
- randrw
ioengine:
- sync
- mmap
bs:
- 4k
- 64k

Which will be split into 2*2*2 unit jobs for execution. For example, the first unit job is:

fio:
rw: randwrite
ioengine: sync
bs: 4k

Quoted 3 lines

That said, I do agree that we could do better, and yeah, adding more workloads to Fengguang's scripts are always a good thing -- perhaps even adding stuff from perf-bench.

You are very welcome to add new cases, monitors or setup scripts! Depending on their nature and resource requirement, we may choose the adequate policy to run them in our LKP test infrastructure -- which works 7x24 on the fresh new code in 400+ kernel git trees. By feeding it more test cases, we may reasonably safeguard more kernel code and use scenarios from _silent_ regressions in future.

Thanks,
Fengguang

#12 lore raw
Quoted 3 lines · Fengguang Wu

The proc-vmstat, perf-profile, cpuidle, turbostat etc. "monitors" are inspired by Mel Gorman's mmtests suite and they are really helpful in catching&analyzing the subtle impacts a patch might bring to the system.

Mel also has some interesting tests in the suite like the page fault test and aim9 among numerous others. Could those also be added?

#13 lore raw
Quoted 8 lines · Christoph Lameter
Quoted 3 lines · Fengguang Wu

The proc-vmstat, perf-profile, cpuidle, turbostat etc. "monitors" are inspired by Mel Gorman's mmtests suite and they are really helpful in catching&analyzing the subtle impacts a patch might bring to the system.

Mel also has some interesting tests in the suite like the page fault test and aim9 among numerous others. Could those also be added?

Sure. I just added aim9 to the TODO list.

pft is already included in lkp-tests:

% ls */pft
jobs/pft.yaml pack/pft pack/pft.patch stats/pft tests/pft

Thanks,
Fengguang

#14 lore raw
Quoted 10 lines · Fengguang Wu
Quoted 8 lines · Christoph Lameter
Quoted 3 lines · Fengguang Wu

The proc-vmstat, perf-profile, cpuidle, turbostat etc. "monitors" are inspired by Mel Gorman's mmtests suite and they are really helpful in catching&analyzing the subtle impacts a patch might bring to the system.

Mel also has some interesting tests in the suite like the page fault test and aim9 among numerous others. Could those also be added?

Quoted 6 lines

Sure. I just added aim9 to the TODO list.

pft is already included in lkp-tests:

% ls */pft
jobs/pft.yaml pack/pft pack/pft.patch stats/pft tests/pft

Are the test results somewhere publicly available?

#15 lore raw

[Ksummit-discuss] 0day kernel performance/power test service

Quoted 12 lines · Christoph Lameter
Quoted 10 lines · Fengguang Wu
Quoted 8 lines · Christoph Lameter
Quoted 3 lines · Fengguang Wu

The proc-vmstat, perf-profile, cpuidle, turbostat etc. "monitors" are inspired by Mel Gorman's mmtests suite and they are really helpful in catching&analyzing the subtle impacts a patch might bring to the system.

Mel also has some interesting tests in the suite like the page fault test and aim9 among numerous others. Could those also be added?

Quoted 8 lines
Quoted 6 lines

Sure. I just added aim9 to the TODO list.

pft is already included in lkp-tests:

% ls */pft
jobs/pft.yaml pack/pft pack/pft.patch stats/pft tests/pft

Are the test results somewhere publicly available?

Not available yet.. Sorry! For now I'm afraid I can only send tar balls of raw results on your request.

btw, we'll offer an experimental 0day performance/power testing service for the core maintainer trees. Just send me a lkp-tests patch (attached an example for ext4:dev) to selectively run a set of test jobs on a set of machines for your tree:branch. Future git push will then trigger the test set and you'll receive the report that compares the branch BASE:HEAD performance in 24 hours.

The patch is basically selecting a combination of test jobs and machines. All the available test jobs and machines are included in lkp-tests. You are also welcome to add new jobs to the pool.

wfg /c/lkp-tests% ls hosts

bay         ivb43     lkp-nex04  lkp-sb03   lkp-t410   loslunas   vm-kbuild-2G          vm-kbuild-yocto-x86_64  vm-vp-quantal-x86_64  xps2
bens        ivb44     lkp-nex05  lkp-sbx04  lkp-ws02   nhm4       vm-kbuild-4G          vm-vp-1G                wsm
brickland1  lkp-ne02  lkp-nex06  lkp-snb01  lkp-wsx01  nhm-white  vm-kbuild-yocto-i386  vm-vp-2G                xbm
brickland3  lkp-ne04  lkp-sb02   lkp-st02   lkp-wsx02  snb-drag   vm-kbuild-yocto-ia32  vm-vp-quantal-i386      xps

wfg /c/lkp-tests% ls jobs

blogbench.yaml           dd-write.yaml                          hackbench-8.yaml       netperf.yaml                tbench.yaml
boot.yaml                debug.yaml                             hackbench-cpuset.yaml  nfs-iozone.yaml             tcrypt.yaml
borrow-1d.yaml           DEFAULTS                               hackbench-perf.yaml    nuttcp.yaml                 thrulay.yaml
borrow-1h.yaml           ebizzy.yaml                            hackbench.yaml         oltp.yaml                   tlbflush.yaml
dbench.yaml              fileio.yaml                            idle.yaml              packetdrill.yaml            unixbench.yaml
dd-write-11hdd.yaml      fio-jbod-12hdd-randwrite-sync-4k.yaml  iozone.yaml            perf-bench-numa-mem.yaml    vm-scalability-numa.yaml
dd-write-1hdd-fuse.yaml  fio-jbod-12hdd.yaml                    iperf.yaml             perf-bench-sched-pipe.yaml  vm-scalability.yaml
dd-write-1hdd.yaml       fsmark.yaml                            kbuild.yaml            pft.yaml                    will-it-scale.yaml
dd-write-1ssd.yaml       ftq.yaml                               kernel_selftests.yaml  piglit.yaml                 xfstests-btrfs.yaml
dd-write-4hdd-fuse.yaml  fwq.yaml                               linpack.yaml           pigz.yaml                   xfstests-ext4.yaml
dd-write-4hdd.yaml       glbenchmark.yaml                       ltp.yaml               qperf.yaml                  xfstests-generic.yaml
dd-write-hdd-usb.yaml    hackbench-6.yaml                       nepim.yaml             sockperf.yaml               xfstests-xfs.yaml

Thanks,
Fengguang

Quoted 1 line

From 8cd384c2f34f593d829457e7043ee6cca204bda3 Mon Sep 17 00:00:00 2001

From: Fengguang Wu <[email protected]>
Date: Sat, 16 Aug 2014 09:12:35 +0800
Subject: [PATCH] define 0day performance test set for ext4/dev

This set of test jobs will be auto triggered by git push and email report should be sent to the committer in 1 day.

---
 allot/ext4:dev/lkp-ne04/fsmark-ext4.yaml           | 27 ++++++++++++++++++++++
 allot/ext4:dev/lkp-st02/dd-write-11hdd-ext4.yaml   | 27 ++++++++++++++++++++++
 .../vm-kbuild-2G/xfstests-generic-ext4.yaml        | 16 +++++++++++++
 allot/ext4:dev/vm-vp-1G/xfstests-ext4.yaml         |  1 +
 4 files changed, 71 insertions(+)
 create mode 100644 allot/ext4:dev/lkp-ne04/fsmark-ext4.yaml
 create mode 100644 allot/ext4:dev/lkp-st02/dd-write-11hdd-ext4.yaml
 create mode 100644 allot/ext4:dev/vm-kbuild-2G/xfstests-generic-ext4.yaml
 create mode 120000 allot/ext4:dev/vm-vp-1G/xfstests-ext4.yaml
allot/ext4:dev/lkp-ne04/fsmark-ext4.yaml new file +27 -0
new file mode 100644
index 0000000..ca2e3ea
@@ -0,0 +1,27 @@
+testcase: fsmark
+
+iterations: 1x
+nr_threads: 32t
+
+disk: 1HDD
+fs:
+- ext4
+
+fsmark:
+  filesize: 16MB
+  test_size: 60G
+  sync_method:
+  - NoSync
+  - fsyncBeforeClose
+  nr_directories: 16d
+  nr_files_per_directory: 256f
+
+---
+fsmark:
+  filesize:
+  - 8K
+  - 9B
+  test_size: 400M
+  sync_method: fsyncBeforeClose
+  nr_directories: 16d
+  nr_files_per_directory: 256f
allot/ext4:dev/lkp-st02/dd-write-11hdd-ext4.yaml new file +27 -0
new file mode 100644
index 0000000..d59f3dd
@@ -0,0 +1,27 @@
+testcase: dd-write
+
+runtime: 5m
+
+disk: 11HDD
+md:
+- JBOD
+- RAID5
+iosched:
+- cfq
+fs:
+- ext4
+
+monitors:
+  perf-stat:
+  perf-profile:
+  ftrace:
+  - events:
+      balance_dirty_pages
+      bdi_dirty_ratelimit
+      global_dirty_state
+      writeback_single_inode
+
+nr_threads:
+- 100dd
+
+dd:
allot/ext4:dev/vm-kbuild-2G/xfstests-generic-ext4.yaml new file +16 -0
new file mode 100644
index 0000000..2561003
@@ -0,0 +1,16 @@
+testcase: xfstests
+
+default_monitors:
+
+disk: 4HDD
+
+fs:
+- ext4
+
+xfstests:
+  test:
+  - generic-113
+  - generic-slow1
+  - generic-slow2
+  - generic-mid
+  - generic-quick
allot/ext4:dev/vm-vp-1G/xfstests-ext4.yaml new file +1 -0
new file mode 120000
index 0000000..e3ea5c2
@@ -0,0 +1 @@
+../../../jobs/xfstests-ext4.yaml
\ No newline at end of file
Signature · 1 line
2.1.0.rc1
#16 lore raw
In reply to Josh Triplett
Quoted 12 lines · Josh Triplett
Quoted 10 lines · Chris Mason

Hi everyone,

We're in the middle of upgrading the tiers here from older kernels (2.6.38, 3.2) into 3.10 and higher.

I've been doing this upgrade game for a number of years now, with different business cards taped to my forehead and with different target workloads.

The result is always the same...if I'm really lucky the system isn't slower, but usually I'm left with a steaming pile of 10-30% regressions.

Quoted 8 lines

How automated are your benchmark workloads, how long do they take, and how consistent are they from run to run (on a system running nothing else)? What about getting them into Fengguang Wu's automated patch checker, or a similar system that checks every patch or pull rather than just full releases? If we had feedback at the time of patch submission that a specific patch made the kernel x% slower for a specific well-defined workload, that would prove much easier to act on than just a comparison of 3.x and 3.y.

Good point on Fengguang's checker! I do sometimes get emails from him noting changes in performance, conveniently bisected down to the commit.

Thanx, Paul

#17 lore raw
In reply to Chris Mason

Hi Chris,

Quoted 3 lines

The KS dates should put us right at the end of our regression hunt, I can talk through the main problems we hit, how (if) we fixed them and hopefully offer tests to keep them from coming back.

It'd be sweet if your tests (and others') can be added to

git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests

Which runs in Intel actively to keep the relevant regressions from coming back.

Currently the repository has test cases

wfg /c/lkp-tests% ls tests

aim7       debug-test  fio-jbod  glbenchmark  kbuild            nepim    packetdrill            piglit  sockperf  tlbflush        wrapper
blogbench  ebizzy      fsmark    hackbench    kernel_selftests  netperf  perf-bench-numa-mem    pigz    tbench    unixbench       xfstests
dbench     fileio      ftq       iozone       linpack           nuttcp   perf-bench-sched-pipe  qperf   tcrypt    vm-scalability
dd         fio         fwq       iperf        ltp               oltp     pft                    sleep   thrulay   will-it-scale

Which is not only too few in number, but also rather limited in test parameters and system setups. For example, if run them in different cgroup/numa/governor/network/... setups, or even with different kernel configs, the results may become quite different.

Thanks,
Fengguang