Skip to contentSkip to conversation

New readahead - ups and downs

DISCUSSION Helge Hafting 3 participants 8 messages Jun 27, 2006 – Jul 4, 2006 2 messages not in the archive

lore
Share & tools
lore
#0 lore raw

Many have noticed positive sides of the new readahead system. I too see that, bootup is quicker and starting a big app like firefox is also noticeable faster.

I made my own little io-intensive test, that shows a case where performance drops.

I boot the machine, and starts "debsums", a debian utility that checksums every file managed by debian package management. As soon as the machine starts swapping, I also start start a process that applies an mm-patch to the kernel tree, and times this.

This patching took 1m28s with cold cache, without debsums running. With the 2.6.15 kernel (old readahead), and debsums running, this took 2m20s to complete, and 360kB in swap at the worst.

With the new readahead in 2.6.17-mm3 I get 6m22s for patching, and 22MB in swap at the most. Runs with mm1 and mm2 were similiar, 5-6 minutes patching and 22MB swap.

My patching clearly takes more times this way. I don't know if debsums improved though, it could be as simple as a fairness issue. Memory pressure definitely went up.

Helge Hafting

#1 lore raw

Hi Helge,

Thanks for testing it out.
I'll check it when I have time(I'll be out for a week...).

Wu

Quoted 32 lines · Helge Hafting

Many have noticed positive sides of the new readahead system. I too see that, bootup is quicker and starting a big app like firefox is also noticeable faster.

I made my own little io-intensive test, that shows a case where performance drops.

I boot the machine, and starts "debsums", a debian utility that checksums every file managed by debian package management. As soon as the machine starts swapping, I also start start a process that applies an mm-patch to the kernel tree, and times this.

This patching took 1m28s with cold cache, without debsums running. With the 2.6.15 kernel (old readahead), and debsums running, this took 2m20s to complete, and 360kB in swap at the worst.

With the new readahead in 2.6.17-mm3 I get 6m22s for patching, and 22MB in swap at the most. Runs with mm1 and mm2 were similiar, 5-6 minutes patching and 22MB swap.

My patching clearly takes more times this way. I don't know if debsums improved though, it could be as simple as a fairness issue. Memory pressure definitely went up.

Helge Hafting
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#2 lore raw
In reply to Helge Hafting

Hi Helge,

Quoted 13 lines · Helge Hafting

I made my own little io-intensive test, that shows a case where performance drops.

I boot the machine, and starts "debsums", a debian utility that checksums every file managed by debian package management. As soon as the machine starts swapping, I also start start a process that applies an mm-patch to the kernel tree, and times this.

This patching took 1m28s with cold cache, without debsums running. With the 2.6.15 kernel (old readahead), and debsums running, this took 2m20s to complete, and 360kB in swap at the worst.

Quoted 7 lines

With the new readahead in 2.6.17-mm3 I get 6m22s for patching, and 22MB in swap at the most. Runs with mm1 and mm2 were similiar, 5-6 minutes patching and 22MB swap.

My patching clearly takes more times this way. I don't know if debsums improved though, it could be as simple as a fairness issue. Memory pressure definitely went up.

There are a lot changes between 2.6.15 and 2.6.17-mmX. Would you use the single 2.6.17-mm5 kernel for benchmarking? It's easy:

- select old readahead:
echo 1 > /proc/sys/vm/readahead_ratio

- select new readahead:
echo 50 > /proc/sys/vm/readahead_ratio

Thanks,
Wu

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw

Re: New readahead - ups and downs new test

Quoted 24 lines · Fengguang Wu

Hi Helge,

Quoted 13 lines · Helge Hafting

I made my own little io-intensive test, that shows a case where performance drops.

I boot the machine, and starts "debsums", a debian utility that checksums every file managed by debian package management. As soon as the machine starts swapping, I also start start a process that applies an mm-patch to the kernel tree, and times this.

This patching took 1m28s with cold cache, without debsums running. With the 2.6.15 kernel (old readahead), and debsums running, this took 2m20s to complete, and 360kB in swap at the worst.

Quoted 7 lines

With the new readahead in 2.6.17-mm3 I get 6m22s for patching, and 22MB in swap at the most. Runs with mm1 and mm2 were similiar, 5-6 minutes patching and 22MB swap.

My patching clearly takes more times this way. I don't know if debsums improved though, it could be as simple as a fairness issue. Memory pressure definitely went up.

Quoted 10 lines

There are a lot changes between 2.6.15 and 2.6.17-mmX. Would you use the single 2.6.17-mm5 kernel for benchmarking? It's easy:

- select old readahead:
echo 1 > /proc/sys/vm/readahead_ratio

- select new readahead:
echo 50 > /proc/sys/vm/readahead_ratio

I just tried this with 2.5.17-mm5. I did in on a faster machine (opteron cpu, but still 512MB) so don't compare with my previous test which ran on a pentium-IV.
Single cpu in both cases.

Test procdure:
1. Reboot, log in through xdm
2. run vmstat 10 for swap monitoring
3. time debsums -s
4. As soon as the machine touches swap, launch
time bzcat 2.6.15-mm5.bz2 | patch -p1

In either case, testing starts with 320MB free memory after boot, which debsums caching eats in about a minute and swapping starts. Then I start the patching, which finished before debsums.

Old readahed:
Max swap was 700kB, but it dropped back to 244kB after 10s and stayed there.
Patch timing:

real    0m37.662s
user    0m5.002s
sys     0m2.023s
debsums timing:
real    5m50.333s
user    0m21.127s
sys     0m14.506s

New readahead:
Max swap: 244kB. (On another try it jumped to 816kB and then fell back to 244kB).
patch timing:

real    0m40.951s
user    0m5.043s
sys     0m2.061s
debsums timing:
real    5m46.555s
user    0m21.195s
sys     0m13.918s

Timing and memory load seems to be almost identical this time, perhaps this is a load where the type of readahead doesn't matter.

Helge Hafting

#5 lore raw

Re: New readahead - ups and downs new test

Quoted 75 lines · Helge Hafting
Quoted 24 lines · Fengguang Wu

Hi Helge,

Quoted 13 lines · Helge Hafting

I made my own little io-intensive test, that shows a case where performance drops.

I boot the machine, and starts "debsums", a debian utility that checksums every file managed by debian package management. As soon as the machine starts swapping, I also start start a process that applies an mm-patch to the kernel tree, and times this.

This patching took 1m28s with cold cache, without debsums running. With the 2.6.15 kernel (old readahead), and debsums running, this took 2m20s to complete, and 360kB in swap at the worst.

Quoted 7 lines

With the new readahead in 2.6.17-mm3 I get 6m22s for patching, and 22MB in swap at the most. Runs with mm1 and mm2 were similiar, 5-6 minutes patching and 22MB swap.

My patching clearly takes more times this way. I don't know if debsums improved though, it could be as simple as a fairness issue. Memory pressure definitely went up.

Quoted 10 lines

There are a lot changes between 2.6.15 and 2.6.17-mmX. Would you use the single 2.6.17-mm5 kernel for benchmarking? It's easy:

- select old readahead:
echo 1 > /proc/sys/vm/readahead_ratio

- select new readahead:
echo 50 > /proc/sys/vm/readahead_ratio

I just tried this with 2.5.17-mm5. I did in on a faster machine (opteron cpu, but still 512MB) so don't compare with my previous test which ran on a pentium-IV.
Single cpu in both cases.

Test procdure:
1. Reboot, log in through xdm
2. run vmstat 10 for swap monitoring
3. time debsums -s
4. As soon as the machine touches swap, launch
time bzcat 2.6.15-mm5.bz2 | patch -p1

In either case, testing starts with 320MB free memory after boot, which debsums caching eats in about a minute and swapping starts. Then I start the patching, which finished before debsums.

Old readahed:
Max swap was 700kB, but it dropped back to 244kB after 10s and stayed there.
Patch timing:

real    0m37.662s
user    0m5.002s
sys     0m2.023s
debsums timing:
real    5m50.333s
user    0m21.127s
sys     0m14.506s

New readahead:
Max swap: 244kB. (On another try it jumped to 816kB and then fell back to 244kB).
patch timing:

real    0m40.951s
user    0m5.043s
sys     0m2.061s
debsums timing:
real    5m46.555s
user    0m21.195s
sys     0m13.918s
Quoted 3 lines

Timing and memory load seems to be almost identical this time, perhaps this is a load where the type of readahead doesn't matter.

Thanks. You are right, the readahead logic won't affect the swap cache. Nor will the readahead size, I guess. But to be sure, you can do one more test on it with the following command, using the same 2.5.17-mm5:

        blockdev --setra /dev/hda1 256

Please replace /dev/hda1 with the root device on your system, thanks.

Wu

A message that is not in the archive (sent off-list or not yet received): [email protected]

#7 lore raw

Re: New readahead - ups and downs new test

Quoted 76 lines · Fengguang Wu
Quoted 75 lines · Helge Hafting
Quoted 24 lines · Fengguang Wu

Hi Helge,

Quoted 13 lines · Helge Hafting

I made my own little io-intensive test, that shows a case where performance drops.

I boot the machine, and starts "debsums", a debian utility that checksums every file managed by debian package management. As soon as the machine starts swapping, I also start start a process that applies an mm-patch to the kernel tree, and times this.

This patching took 1m28s with cold cache, without debsums running. With the 2.6.15 kernel (old readahead), and debsums running, this took 2m20s to complete, and 360kB in swap at the worst.

Quoted 7 lines

With the new readahead in 2.6.17-mm3 I get 6m22s for patching, and 22MB in swap at the most. Runs with mm1 and mm2 were similiar, 5-6 minutes patching and 22MB swap.

My patching clearly takes more times this way. I don't know if debsums improved though, it could be as simple as a fairness issue. Memory pressure definitely went up.

Quoted 10 lines

There are a lot changes between 2.6.15 and 2.6.17-mmX. Would you use the single 2.6.17-mm5 kernel for benchmarking? It's easy:

- select old readahead:
echo 1 > /proc/sys/vm/readahead_ratio

- select new readahead:
echo 50 > /proc/sys/vm/readahead_ratio

I just tried this with 2.5.17-mm5. I did in on a faster machine (opteron cpu, but still 512MB) so don't compare with my previous test which ran on a pentium-IV.
Single cpu in both cases.

Test procdure:
1. Reboot, log in through xdm
2. run vmstat 10 for swap monitoring
3. time debsums -s
4. As soon as the machine touches swap, launch
time bzcat 2.6.15-mm5.bz2 | patch -p1

In either case, testing starts with 320MB free memory after boot, which debsums caching eats in about a minute and swapping starts. Then I start the patching, which finished before debsums.

Old readahed:
Max swap was 700kB, but it dropped back to 244kB after 10s and stayed there.
Patch timing:

real    0m37.662s
user    0m5.002s
sys     0m2.023s
debsums timing:
real    5m50.333s
user    0m21.127s
sys     0m14.506s

New readahead:
Max swap: 244kB. (On another try it jumped to 816kB and then fell back to 244kB).
patch timing:

real    0m40.951s
user    0m5.043s
sys     0m2.061s
debsums timing:
real    5m46.555s
user    0m21.195s
sys     0m13.918s
Quoted 7 lines
Quoted 3 lines

Timing and memory load seems to be almost identical this time, perhaps this is a load where the type of readahead doesn't matter.

Thanks. You are right, the readahead logic won't affect the swap cache. Nor will the readahead size, I guess. But to be sure, you can do one more test on it with the following command, using the same 2.5.17-mm5:

Well, I did not expect readahead to directly affect swap, but there was this very noticeable difference on the pentium-IV machine. Different io patterns & disk head movement patterns may alter timing and make the memory pressure situation seem different (and more/less data coming in as readahead might affect memory pressure also.) 369k vs 22M swap is a lot.

I have found an important difference between the two machines, the one with the big differences with/without new readahead has /usr and /usr/src on the same physical disk, although separate partitions. That makes for _lots_ of head movement, when bzcat & patch is operating on /usr/src and debsums is reading /usr.

That machine is not available for testing right now, but I'll re-do my test with/without new readahed with a kernel source tree on the same device as /usr.

Quoted 3 lines
        blockdev --setra /dev/hda1 256

Using blockdev --getra on the two disks that holds /usr and /usr/src gives me 2048. So, now we get a test with 1/8 of the normal readahead?

Results: Swap went up to 500k and was down at the usual 244k 10s later.

patch timing:

real    0m38.265s
user    0m5.010s
sys     0m2.097s

debsums timing:

real    5m48.367s
user    0m21.015s
sys     0m13.950s

Seems --setra made no difference.

I'll copy the kernel tree to /usr, to see if anything interesting happes when the two processes actually compete for the same device. That's what got so different last time, although with differing kernel versions.

Helge Hafting

#8 lore raw

Re: New readahead - ups and downs new test. Vm oddities.

In reply to a message not in the archive

I have now re-run my tests (parallel debsums and bzcat+patch) this time with everything on the same device so as to get competition for io.

New and old readahead didn't make much difference this time either, so it seems that my idea about readahead problems were wrong. Which is good, as the new readahead improves so many other things.

Results with new readahead using one disk device:
Swap went up to 32M, dropped to 244k when testing ended. patch timing:

real    6m8.451s
user    0m5.183s
sys     0m2.897s
debsums timing:
real    7m42.851s
user    0m21.172s
sys     0m13.642s

Results with old readahead, one disk device:
Swap went to 32M, dropped to 244k when testing ended. timings:
patch:

real    6m18.191s
user    0m5.226s
sys     0m2.724s
debsums:
real    7m49.860s
user    0m21.243s
sys     0m14.268s

A tiny bit slower, but very little.

No surprise that everyting is slower when using a single disk instead of two.

The swap difference from using two disks is striking though. Nothing to do with readahead, but
why 32M swap when using one disk, and 244k swap when using two?

The amount of data processed is the same either way, is the VM very timing-sensitive?

Helge Hafting

#9 lore raw

Re: New readahead - ups and downs new test. Vm oddities.

Hi Helge,

Quoted 31 lines · Helge Hafting

I have now re-run my tests (parallel debsums and bzcat+patch) this time with everything on the same device so as to get competition for io.

New and old readahead didn't make much difference this time either, so it seems that my idea about readahead problems were wrong. Which is good, as the new readahead improves so many other things.

Results with new readahead using one disk device:
Swap went up to 32M, dropped to 244k when testing ended. patch timing:

real    6m8.451s
user    0m5.183s
sys     0m2.897s
debsums timing:
real    7m42.851s
user    0m21.172s
sys     0m13.642s

Results with old readahead, one disk device:
Swap went to 32M, dropped to 244k when testing ended. timings:
patch:

real    6m18.191s
user    0m5.226s
sys     0m2.724s
debsums:
real    7m49.860s
user    0m21.243s
sys     0m14.268s
Quoted 5 lines

A tiny bit slower, but very little.

No surprise that everyting is slower when using a single disk instead of two.

Thanks for all the efforts!

Quoted 6 lines

The swap difference from using two disks is striking though. Nothing to do with readahead, but
why 32M swap when using one disk, and 244k swap when using two?

The amount of data processed is the same either way, is the VM very timing-sensitive?

Because read/write request go for the same elevator queue I guess.

When there are concurrent read/writes, writes will be hold back, giving priority to reads. So there will be more dirtied pages taking up your memory during the test.

Thanks,
Wu