New readahead - ups and downs
Many have noticed positive sides of the new readahead system. I too see that, bootup is quicker and starting a big app like firefox is also noticeable faster.
I made my own little io-intensive test, that shows a case where performance drops.
I boot the machine, and starts "debsums", a debian utility that checksums every file managed by debian package management. As soon as the machine starts swapping, I also start start a process that applies an mm-patch to the kernel tree, and times this.
This patching took 1m28s with cold cache, without debsums running. With the 2.6.15 kernel (old readahead), and debsums running, this took 2m20s to complete, and 360kB in swap at the worst.
With the new readahead in 2.6.17-mm3 I get 6m22s for patching, and 22MB in swap at the most. Runs with mm1 and mm2 were similiar, 5-6 minutes patching and 22MB swap.
My patching clearly takes more times this way. I don't know if debsums improved though, it could be as simple as a fairness issue. Memory pressure definitely went up.
Helge Hafting
Hi Helge,
Thanks for testing it out.
I'll check it when I have time(I'll be out for a week...).
Wu
Quoted 32 lines · Helge Hafting
Many have noticed positive sides of the new readahead system. I too see that, bootup is quicker and starting a big app like firefox is also noticeable faster.
I made my own little io-intensive test, that shows a case where performance drops.
I boot the machine, and starts "debsums", a debian utility that checksums every file managed by debian package management. As soon as the machine starts swapping, I also start start a process that applies an mm-patch to the kernel tree, and times this.
This patching took 1m28s with cold cache, without debsums running. With the 2.6.15 kernel (old readahead), and debsums running, this took 2m20s to complete, and 360kB in swap at the worst.
With the new readahead in 2.6.17-mm3 I get 6m22s for patching, and 22MB in swap at the most. Runs with mm1 and mm2 were similiar, 5-6 minutes patching and 22MB swap.
My patching clearly takes more times this way. I don't know if debsums improved though, it could be as simple as a fairness issue. Memory pressure definitely went up.
Helge Hafting
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Hi Helge,
Quoted 13 lines · Helge Hafting
I made my own little io-intensive test, that shows a case where performance drops.
I boot the machine, and starts "debsums", a debian utility that checksums every file managed by debian package management. As soon as the machine starts swapping, I also start start a process that applies an mm-patch to the kernel tree, and times this.
This patching took 1m28s with cold cache, without debsums running. With the 2.6.15 kernel (old readahead), and debsums running, this took 2m20s to complete, and 360kB in swap at the worst.
Quoted 7 lines
With the new readahead in 2.6.17-mm3 I get 6m22s for patching, and 22MB in swap at the most. Runs with mm1 and mm2 were similiar, 5-6 minutes patching and 22MB swap.
My patching clearly takes more times this way. I don't know if debsums improved though, it could be as simple as a fairness issue. Memory pressure definitely went up.
There are a lot changes between 2.6.15 and 2.6.17-mmX. Would you use the single 2.6.17-mm5 kernel for benchmarking? It's easy:
- select old readahead:
echo 1 > /proc/sys/vm/readahead_ratio
- select new readahead:
echo 50 > /proc/sys/vm/readahead_ratio
Thanks,
Wu
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: New readahead - ups and downs new test
Quoted 24 lines · Fengguang Wu
Hi Helge,
Quoted 13 lines · Helge Hafting
I made my own little io-intensive test, that shows a case where performance drops.
I boot the machine, and starts "debsums", a debian utility that checksums every file managed by debian package management. As soon as the machine starts swapping, I also start start a process that applies an mm-patch to the kernel tree, and times this.
This patching took 1m28s with cold cache, without debsums running. With the 2.6.15 kernel (old readahead), and debsums running, this took 2m20s to complete, and 360kB in swap at the worst.
Quoted 7 lines
With the new readahead in 2.6.17-mm3 I get 6m22s for patching, and 22MB in swap at the most. Runs with mm1 and mm2 were similiar, 5-6 minutes patching and 22MB swap.
My patching clearly takes more times this way. I don't know if debsums improved though, it could be as simple as a fairness issue. Memory pressure definitely went up.
Quoted 10 lines
There are a lot changes between 2.6.15 and 2.6.17-mmX. Would you use the single 2.6.17-mm5 kernel for benchmarking? It's easy:
- select old readahead:
echo 1 >/proc/sys/vm/readahead_ratio- select new readahead:
echo 50 >/proc/sys/vm/readahead_ratio
I just tried this with 2.5.17-mm5. I did in on a faster machine (opteron cpu, but still 512MB) so don't compare with my previous test which ran on a pentium-IV.
Single cpu in both cases.
Test procdure:
1. Reboot, log in through xdm
2. run vmstat 10 for swap monitoring
3. time debsums -s
4. As soon as the machine touches swap, launch
time bzcat 2.6.15-mm5.bz2 | patch -p1
In either case, testing starts with 320MB free memory after boot, which debsums caching eats in about a minute and swapping starts. Then I start the patching, which finished before debsums.
Old readahed:
Max swap was 700kB, but it dropped back to 244kB after 10s and stayed there.
Patch timing:
real 0m37.662s user 0m5.002s sys 0m2.023s debsums timing: real 5m50.333s user 0m21.127s sys 0m14.506s
New readahead:
Max swap: 244kB. (On another try it jumped to 816kB and then fell back to 244kB).
patch timing:
real 0m40.951s user 0m5.043s sys 0m2.061s debsums timing: real 5m46.555s user 0m21.195s sys 0m13.918s
Timing and memory load seems to be almost identical this time, perhaps this is a load where the type of readahead doesn't matter.
Helge Hafting
Re: New readahead - ups and downs new test
Quoted 75 lines · Helge Hafting
Quoted 24 lines · Fengguang Wu
Hi Helge,
Quoted 13 lines · Helge Hafting
I made my own little io-intensive test, that shows a case where performance drops.
I boot the machine, and starts "debsums", a debian utility that checksums every file managed by debian package management. As soon as the machine starts swapping, I also start start a process that applies an mm-patch to the kernel tree, and times this.
This patching took 1m28s with cold cache, without debsums running. With the 2.6.15 kernel (old readahead), and debsums running, this took 2m20s to complete, and 360kB in swap at the worst.
Quoted 7 lines
With the new readahead in 2.6.17-mm3 I get 6m22s for patching, and 22MB in swap at the most. Runs with mm1 and mm2 were similiar, 5-6 minutes patching and 22MB swap.
My patching clearly takes more times this way. I don't know if debsums improved though, it could be as simple as a fairness issue. Memory pressure definitely went up.
Quoted 10 lines
There are a lot changes between 2.6.15 and 2.6.17-mmX. Would you use the single 2.6.17-mm5 kernel for benchmarking? It's easy:
- select old readahead:
echo 1 >/proc/sys/vm/readahead_ratio- select new readahead:
echo 50 >/proc/sys/vm/readahead_ratioI just tried this with 2.5.17-mm5. I did in on a faster machine (opteron cpu, but still 512MB) so don't compare with my previous test which ran on a pentium-IV.
Single cpu in both cases.Test procdure:
1. Reboot, log in through xdm
2. run vmstat 10 for swap monitoring
3. time debsums -s
4. As soon as the machine touches swap, launch
time bzcat 2.6.15-mm5.bz2 | patch -p1In either case, testing starts with 320MB free memory after boot, which debsums caching eats in about a minute and swapping starts. Then I start the patching, which finished before debsums.
Old readahed:
Max swap was 700kB, but it dropped back to 244kB after 10s and stayed there.
Patch timing:real 0m37.662s user 0m5.002s sys 0m2.023s debsums timing: real 5m50.333s user 0m21.127s sys 0m14.506sNew readahead:
Max swap: 244kB. (On another try it jumped to 816kB and then fell back to 244kB).
patch timing:real 0m40.951s user 0m5.043s sys 0m2.061s debsums timing: real 5m46.555s user 0m21.195s sys 0m13.918s
Quoted 3 lines
Timing and memory load seems to be almost identical this time, perhaps this is a load where the type of readahead doesn't matter.
Thanks. You are right, the readahead logic won't affect the swap cache. Nor will the readahead size, I guess. But to be sure, you can do one more test on it with the following command, using the same 2.5.17-mm5:
blockdev --setra /dev/hda1 256
Please replace /dev/hda1 with the root device on your system, thanks.
Wu
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: New readahead - ups and downs new test
Quoted 76 lines · Fengguang Wu
Quoted 75 lines · Helge Hafting
Quoted 24 lines · Fengguang Wu
Hi Helge,
Quoted 13 lines · Helge Hafting
I made my own little io-intensive test, that shows a case where performance drops.
I boot the machine, and starts "debsums", a debian utility that checksums every file managed by debian package management. As soon as the machine starts swapping, I also start start a process that applies an mm-patch to the kernel tree, and times this.
This patching took 1m28s with cold cache, without debsums running. With the 2.6.15 kernel (old readahead), and debsums running, this took 2m20s to complete, and 360kB in swap at the worst.
Quoted 7 lines
With the new readahead in 2.6.17-mm3 I get 6m22s for patching, and 22MB in swap at the most. Runs with mm1 and mm2 were similiar, 5-6 minutes patching and 22MB swap.
My patching clearly takes more times this way. I don't know if debsums improved though, it could be as simple as a fairness issue. Memory pressure definitely went up.
Quoted 10 lines
There are a lot changes between 2.6.15 and 2.6.17-mmX. Would you use the single 2.6.17-mm5 kernel for benchmarking? It's easy:
- select old readahead:
echo 1 >/proc/sys/vm/readahead_ratio- select new readahead:
echo 50 >/proc/sys/vm/readahead_ratioI just tried this with 2.5.17-mm5. I did in on a faster machine (opteron cpu, but still 512MB) so don't compare with my previous test which ran on a pentium-IV.
Single cpu in both cases.Test procdure:
1. Reboot, log in through xdm
2. run vmstat 10 for swap monitoring
3. time debsums -s
4. As soon as the machine touches swap, launch
time bzcat 2.6.15-mm5.bz2 | patch -p1In either case, testing starts with 320MB free memory after boot, which debsums caching eats in about a minute and swapping starts. Then I start the patching, which finished before debsums.
Old readahed:
Max swap was 700kB, but it dropped back to 244kB after 10s and stayed there.
Patch timing:real 0m37.662s user 0m5.002s sys 0m2.023s debsums timing: real 5m50.333s user 0m21.127s sys 0m14.506sNew readahead:
Max swap: 244kB. (On another try it jumped to 816kB and then fell back to 244kB).
patch timing:real 0m40.951s user 0m5.043s sys 0m2.061s debsums timing: real 5m46.555s user 0m21.195s sys 0m13.918s
Quoted 7 lines
Quoted 3 lines
Timing and memory load seems to be almost identical this time, perhaps this is a load where the type of readahead doesn't matter.
Thanks. You are right, the readahead logic won't affect the swap cache. Nor will the readahead size, I guess. But to be sure, you can do one more test on it with the following command, using the same 2.5.17-mm5:
Well, I did not expect readahead to directly affect swap, but there was this very noticeable difference on the pentium-IV machine. Different io patterns & disk head movement patterns may alter timing and make the memory pressure situation seem different (and more/less data coming in as readahead might affect memory pressure also.) 369k vs 22M swap is a lot.
I have found an important difference between the two machines, the one with the big differences with/without new readahead has /usr and /usr/src on the same physical disk, although separate partitions. That makes for _lots_ of head movement, when bzcat & patch is operating on /usr/src and debsums is reading /usr.
That machine is not available for testing right now, but I'll re-do my test with/without new readahed with a kernel source tree on the same device as /usr.
Quoted 3 lines
blockdev --setra /dev/hda1 256Using blockdev --getra on the two disks that holds /usr and /usr/src gives me 2048. So, now we get a test with 1/8 of the normal readahead?
Results: Swap went up to 500k and was down at the usual 244k 10s later.
patch timing:
real 0m38.265s user 0m5.010s sys 0m2.097s
debsums timing:
real 5m48.367s user 0m21.015s sys 0m13.950s
Seems --setra made no difference.
I'll copy the kernel tree to /usr, to see if anything interesting happes when the two processes actually compete for the same device. That's what got so different last time, although with differing kernel versions.
Helge Hafting
Re: New readahead - ups and downs new test. Vm oddities.
In reply to a message not in the archiveI have now re-run my tests (parallel debsums and bzcat+patch) this time with everything on the same device so as to get competition for io.
New and old readahead didn't make much difference this time either, so it seems that my idea about readahead problems were wrong. Which is good, as the new readahead improves so many other things.
Results with new readahead using one disk device:
Swap went up to 32M, dropped to 244k when testing ended. patch timing:
real 6m8.451s user 0m5.183s sys 0m2.897s debsums timing: real 7m42.851s user 0m21.172s sys 0m13.642s
Results with old readahead, one disk device:
Swap went to 32M, dropped to 244k when testing ended. timings:
patch:
real 6m18.191s user 0m5.226s sys 0m2.724s debsums: real 7m49.860s user 0m21.243s sys 0m14.268s
A tiny bit slower, but very little.
No surprise that everyting is slower when using a single disk instead of two.
The swap difference from using two disks is striking though. Nothing to do with readahead, but
why 32M swap when using one disk, and 244k swap when using two?
The amount of data processed is the same either way, is the VM very timing-sensitive?
Helge Hafting
Re: New readahead - ups and downs new test. Vm oddities.
Hi Helge,
Quoted 31 lines · Helge Hafting
I have now re-run my tests (parallel debsums and bzcat+patch) this time with everything on the same device so as to get competition for io.
New and old readahead didn't make much difference this time either, so it seems that my idea about readahead problems were wrong. Which is good, as the new readahead improves so many other things.
Results with new readahead using one disk device:
Swap went up to 32M, dropped to 244k when testing ended. patch timing:real 6m8.451s user 0m5.183s sys 0m2.897s debsums timing: real 7m42.851s user 0m21.172s sys 0m13.642sResults with old readahead, one disk device:
Swap went to 32M, dropped to 244k when testing ended. timings:
patch:real 6m18.191s user 0m5.226s sys 0m2.724s debsums: real 7m49.860s user 0m21.243s sys 0m14.268s
Quoted 5 lines
A tiny bit slower, but very little.
No surprise that everyting is slower when using a single disk instead of two.
Thanks for all the efforts!
Quoted 6 lines
The swap difference from using two disks is striking though. Nothing to do with readahead, but
why 32M swap when using one disk, and 244k swap when using two?The amount of data processed is the same either way, is the VM very timing-sensitive?
Because read/write request go for the same elevator queue I guess.
When there are concurrent read/writes, writes will be hold back, giving priority to reads. So there will be more dirtied pages taking up your memory during the test.
Thanks,
Wu