Skip to contentSkip to conversation

Benchmarks for performance patches (-ck) for 2.4.19

DISCUSSION Con Kolivas 8 participants 10 messages Sep 2, 2002 – Sep 3, 2002

lore
Share & tools
lore
#0 lore raw

My merged patchset (http://kernel.kolivas.net) was designed to improve system responsiveness. I have yet to find a good benchmark that measures such a thing. However, in response to criticism about not providing benchmarks I have made available some standard benchmarks at the excellent resources of the open source development laboratory scalable test platform. They are available here:

http://www.osdl.org/stp

my patchsets are the following:
-ck5 patch is patch #781
-ck5-rmap is #782
-ck5-ll is #783

I have conducted some basic tests on #781 and the numbers show it is at least equivalent to stock 2.4.19 (#747), although as I said none of these benchmarks are designed to test desktop system responsiveness.

Please feel free to conduct any tests you like on these patches. I would be interested to hear if anyone can suggest the most suitable benchmark. Please cc me to ensure I receive any comments.

Con Kolivas

#1 lore raw

Wouldn't the majority (to an undeniable extent) of the "responsiveness" of desktop usage be on X's code? if we are talking about that. The problem with finding a benchmark is that first you have to have a definition of what you're benchmarking. The "system responsiveness" term is far too vague. When there's a definition to the term there can be a benchmark made to measure it. I mean, besides making the kernel with as low latency as possible, what is bad about the responsiveness in the kernel? If there's any lag in responsiveness that i see it's always something X related, particularly Xfree86.

You can't really benchmark something where everyone's issues are "things go slow when i use "some" benchmark and try using my computer at the same time" or "it feels better or worse." I'd be quite interested in making a benchmark and/or using it to help responsiveness in linux, who wouldn't want that? But I just think we have a fairly good kernel and the bottleneck is not it in the case of the majority of users and what they report as "system responsiveness." As computers get faster and faster we expect to see that reflected in the software we use, perhaps it's just making some design issues with X more apparent ...perhaps not.
Just some food for thought.

Quoted 28 lines · Con Kolivas

My merged patchset (http://kernel.kolivas.net) was designed to improve system responsiveness. I have yet to find a good benchmark that measures such a thing. However, in response to criticism about not providing benchmarks I have made available some standard benchmarks at the excellent resources of the open source development laboratory scalable test platform. They are available here:

http://www.osdl.org/stp

my patchsets are the following:

(quoted diff) +0 -3
-ck5 patch is patch #781
-ck5-rmap is #782
-ck5-ll is #783

I have conducted some basic tests on #781 and the numbers show it is at least equivalent to stock 2.4.19 (#747), although as I said none of these benchmarks are designed to test desktop system responsiveness.

Please feel free to conduct any tests you like on these patches. I would be interested to hear if anyone can suggest the most suitable benchmark. Please cc me to ensure I receive any comments.

Con Kolivas
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#2 lore raw
Quoted 6 lines · Ed Sweetman

Wouldn't the majority (to an undeniable extent) of the "responsiveness" of desktop usage be on X's code? if we are talking about that. The problem with finding a benchmark is that first you have to have a definition of what you're benchmarking. The "system responsiveness" term is far too vague. When there's a definition to the term there can be a benchmark made to measure it.

Agreed. Things like system responsiveness are fairly complex though and in many cases the average numbers measured by benchmarks don't mean anything to users.

I wish we had a way to measure this stuff, but I'll happily philosophise with you guys until we come up with a useful definition of something we could measure...

Quoted 4 lines

I mean, besides making the kernel with as low latency as possible, what is bad about the responsiveness in the kernel? If there's any lag in responsiveness that i see it's always something X related, particularly Xfree86.

"low latency" != responsiveness

Any latency which is below the point the user can notice is effectively zero, so whether the 10000 wakeups/minute that the user doesn't notice are 2ms or 5ms don't really matter.

What does matter are the wakeups that make the user's mp3 skip, even if these don't influence the statistics at all because there's only 1 every few minutes, or none, if the VM is balanced right ;)

Another responsiveness thing is how fast you can swap in Mozilla when the user comes in in the morning. More of a throughput than a latency thing in this case ... but you still have to make sure the mp3 doesn't skip while mozilla is being loaded.

regards,

Rik

Signature · 3 lines
Bravely reimplemented by the knights who say "NIH".

http://www.surriel.com/		http://distro.conectiva.com/
#3 lore raw
Quoted 26 lines · Rik van Riel
Quoted 6 lines · Ed Sweetman

Wouldn't the majority (to an undeniable extent) of the "responsiveness" of desktop usage be on X's code? if we are talking about that. The problem with finding a benchmark is that first you have to have a definition of what you're benchmarking. The "system responsiveness" term is far too vague. When there's a definition to the term there can be a benchmark made to measure it.

Agreed. Things like system responsiveness are fairly complex though and in many cases the average numbers measured by benchmarks don't mean anything to users.

I wish we had a way to measure this stuff, but I'll happily philosophise with you guys until we come up with a useful definition of something we could measure...

Quoted 4 lines

I mean, besides making the kernel with as low latency as possible, what is bad about the responsiveness in the kernel? If there's any lag in responsiveness that i see it's always something X related, particularly Xfree86.

Quoted 6 lines

"low latency" != responsiveness

Any latency which is below the point the user can notice is effectively zero, so whether the 10000 wakeups/minute that the user doesn't notice are 2ms or 5ms don't really matter.

true. But the latency that's N low multiplied by many processes can equal latencies that are in the range users can notice in a single app that maybe getting bullied by the others. Lowering latencies cant be a bad thing unless they create unelegent code or introduce bugs. That's all i meant by mentioning latency.

Quoted 10 lines

What does matter are the wakeups that make the user's mp3 skip, even if these don't influence the statistics at all because there's only 1 every few minutes, or none, if the VM is balanced right ;)

Another responsiveness thing is how fast you can swap in Mozilla when the user comes in in the morning. More of a throughput than a latency thing in this case ... but you still have to make sure the mp3 doesn't skip while mozilla is being loaded.

Maybe i'm weird then because I've never had that happen. The only time i've had my mp3s skip is when i purposely did things that would do that (saturate via bonnie or dbench). I can see this occuring during swapping when dma is not used, but this just adds to the fact that it's no easy matter to define the problem.

Quoted 3 lines

regards,

Rik

I dont experience audio cutouts with anything i do, even the really abusive things to the computer. I've only had it cut out when using bonnie or dbench and that's something you should expect. So what i see as responsiveness is switching windows on the same desktop and switching between virtual desktops. I see responsiveness as the time between i do something and the time it takes to redraw it. Using a G450, I expect a certain level of hardware performance and half a second or so to redraw a screen is not what i call responsive for a Tbird system. This is of course all X related because i dont see much or any problem with the console and with the kernel. At least nothing compared to X latencies.

the common user wants to see desktop performance equal to mac osX and windows and they come to linux which uses X and dont see that. The problem is Xfree86 is a crossplatform network oriented windowing system and you get the same argument with it that people have with gcc. The purpose of xfree86 and gcc is kind of the same and not the same as these other propriatary systems / compilers such as OSX or icc. It's tough luck for us all but this is something that the xpert mailing list could better handle since what we're looking for is a tool that measures the performance of X under real workloads and where problem areas are.

Both the kernel and the programs used are involved when talking "system responsiveness" but in this case i think the kernel has been fine tuned much more strictly and thoroughly than xfree86 has. I'd like to know if that's due to lack of man/woman power or if they're being restricted by compliance to aspects of X that maybe need to change as use has changed.

#4 lore raw

Hi Ed,
Hi Rik,
Hi lkml,

Quoted 10 lines · Ed Sweetman

I dont experience audio cutouts with anything i do, even the really abusive things to the computer. I've only had it cut out when using bonnie or dbench and that's something you should expect. So what i see as responsiveness is switching windows on the same desktop and switching between virtual desktops. I see responsiveness as the time between i do something and the time it takes to redraw it. Using a G450, I expect a certain level of hardware performance and half a second or so to redraw a screen is not what i call responsive for a Tbird system. This is of course all X related because i dont see much or any problem with the console and with the kernel. At least nothing compared to X latencies.

This is not only X. The problem is the software layering bloat. Thats a design mistake we copied from W*ndows in our Linux applications.

Take your favorite - and I mean your favorite, not the easiest - application, find the handler for the problem event, try to do a really complete call graph trough all software layers involved (say from read(inputdevice_fd) to get the event up to write(outputdevice_fd) to write to the framebuffer) and count cycles. Don't forget the cache misses due to following pointers, which lead to random, non-local memory access patterns (read: a VM and cache nightmare).

Once you really finish this boring task completely you know our real performance problem.

Felix von Leitner did a nice presentation on our Chemnitzer Linux Tag (but in German) and many experienced programmers and software enginieers were really stunned.

So the problem is not really the kernel or the X layer. The NUMBER of abstraction layers is the issue to attack.

People tend to import big bloated libraries or even completely overkill operating systems just because it has ONE SIMPLE property the actually need[1]. This must change.

Regards

Ingo Oeser

[1] Buying/Reselling Windows NT licenses + some Windows FTP server license just because we need FTP. Buying WinCE licenses, just because the embedded device must be able to be accessible via TCP/IP. More examples available per private mail. Stupidity has no limit here.

Signature · 1 line
Science is what we can tell a computer. Art is everything else. --- D.E.Knuth
#5 lore raw
Quoted 3 lines · Ingo Oeser

This is not only X. The problem is the software layering bloat. Thats a design mistake we copied from W*ndows in our Linux applications.

Agreed, it is not just in one layer of the system.

However, changing the kernel _has_ resulted in a large improvement in interactivity and does allow me to run bonnie++ or even a 'bk clone' while playing mp3s and without losing interactivity or having the mp3s skip...

On a machine with 192 MB RAM.

regards,

Rik

Signature · 3 lines
Bravely reimplemented by the knights who say "NIH".

http://www.surriel.com/		http://distro.conectiva.com/
#6 lore raw
In reply to Ingo Oeser
Quoted 36 lines · Ingo Oeser

Hi Ed,
Hi Rik,
Hi lkml,

Quoted 10 lines · Ed Sweetman

I dont experience audio cutouts with anything i do, even the really abusive things to the computer. I've only had it cut out when using bonnie or dbench and that's something you should expect. So what i see as responsiveness is switching windows on the same desktop and switching between virtual desktops. I see responsiveness as the time between i do something and the time it takes to redraw it. Using a G450, I expect a certain level of hardware performance and half a second or so to redraw a screen is not what i call responsive for a Tbird system. This is of course all X related because i dont see much or any problem with the console and with the kernel. At least nothing compared to X latencies.

This is not only X. The problem is the software layering bloat. Thats a design mistake we copied from W*ndows in our Linux applications.

Take your favorite - and I mean your favorite, not the easiest - application, find the handler for the problem event, try to do a really complete call graph trough all software layers involved (say from read(inputdevice_fd) to get the event up to write(outputdevice_fd) to write to the framebuffer) and count cycles. Don't forget the cache misses due to following pointers, which lead to random, non-local memory access patterns (read: a VM and cache nightmare).

Once you really finish this boring task completely you know our real performance problem.

Felix von Leitner did a nice presentation on our Chemnitzer Linux Tag (but in German) and many experienced programmers and software enginieers were really stunned.

Quoted 6 lines

So the problem is not really the kernel or the X layer. The NUMBER of abstraction layers is the issue to attack.

People tend to import big bloated libraries or even completely overkill operating systems just because it has ONE SIMPLE property the actually need[1]. This must change.

I dont' believe it is really X11. The real killer of perceived performance seems to be the pager. Great strides have been made here but i find that file I/O seems to be replacing often used pages so that when i change desktops or touch something that i haven't used for a while it stalls while my disk drive sounds like a nest of hornets. I shouldn't see THAT much difference in behavior between a 10 minute and a 2 minute interval when i have plenty of memory to hold my working set. Backups and updatedb are guaranteed to effect this.

It is of course most noticable with the large multi-layered GUI apps because a single event faults far too many pages as it works its way sparsely through the multiple libraries and their respecitive data. Readahead would help more if there was better locality.

Unfortunately there isn't much we are going to be able to do about this at the application level. Perhaps the libraries could be optimized by reducing the layering but that is a matter for other lists.

The one thing the kernel can do is to not reclaim my often used pages when the pagecache is full of stale data that is unlikely to be reused. Yes, i know this is HARD to do but it needs to be better. I appreciate all those with headaches from trying to fix this.

There was a patch that went by here recently that looked promising to me. If i recall correctly it put newly read pages onto a special list so that if they didn't get read again before reaching the bottom the were reused. Otherwise the wound up on the LRU.

Signature · 5 lines
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		[email protected]

		Remember Cernan and Schmitt
#7 lore raw
In reply to Rik van Riel
Quoted 24 lines · Rik van Riel <[email protected]>, on Sun Sep 01, 2002 [11:32:42 PM] said:
Quoted 4 lines · Ed Sweetman

I mean, besides making the kernel with as low latency as possible, what is bad about the responsiveness in the kernel? If there's any lag in responsiveness that i see it's always something X related, particularly Xfree86.

"low latency" != responsiveness

Any latency which is below the point the user can notice is effectively zero, so whether the 10000 wakeups/minute that the user doesn't notice are 2ms or 5ms don't really matter.

What does matter are the wakeups that make the user's mp3 skip, even if these don't influence the statistics at all because there's only 1 every few minutes, or none, if the VM is balanced right ;)

Another responsiveness thing is how fast you can swap in Mozilla when the user comes in in the morning. More of a throughput than a latency thing in this case ... but you still have to make sure the mp3 doesn't skip while mozilla is being loaded.

Quoted 3 lines

regards,

Rik

	Hi;

	I just thought I would toss this in: For some classes

of users, latency is extremely critical. For example, people making music live, doing realtime audio effects processing, or recording audio. These people would probably like it below 2ms.
mp3 playing latency requirements are very
loose-- the buffer takes up the slack, unless the latency is insane.
The good news is that for me, using the ck4 patchset,
my smp system is stable, and has a max latency of 3.5ms under latencytest (during a make -j4 kernel compile no less) _and_ outperformed (slightly) virgin 2.4.19 on generic benchmarks. (like building a kernel, bonnie,...)
Latency may not be the holy grail, but it remains
a valid requirement for a class of user. It may not merit degredation for the common case, but perhaps it need not?

Paul
[email protected]

ps. ck4 uses the aa vm, but if there was a version with rmap, I would test it.

#8 lore raw

The ck5 kernel comes in more than one flavor

rmap, aa and low latency are offered -

I'm running 2.4.19-ck5-ll and I'm happy to report that it also plays well with freeswan ipsec - I'm running freeswan 1.98b and it's all good - not to mention the nice smooth feel in multimedia and 3D fps shooters ;-)

Thanks to conman for the tedious patching work!

Joe

Quoted 3 lines · Paul

ps. ck4 uses the aa vm, but if there was a version with rmap, I would test it

#9 lore raw
In reply to Paul
Quoted 4 lines · Paul

...
I just thought I would toss this in: For some classes
of users, latency is extremely critical.

There are several types of latency: interrupt, scheduling, IO, probably more.

The ones we're most interested in are scheduling latency and IO latency.

These are two utterly, completely, wildly different things! They differ by two or three orders of magnitude and their underlying causes and effects are quite different.

Let's please be careful in making the distinction between the two.

scheduling latency is pretty specialised. You have to push the system pretty hard to experience a scheduling stall which is longer than a monitor refresh interval. I don't believe that scheduling latency is perceptible in desktop use. Audio, yes. Games, maybe.

IO latency is a much larger problem in Linux. Once you start pushing the system it pops up all over the place. One simple test I use to quantify this is:

- boot with mem=512m
- perform a continuous appending write to an ext2 file on a scratch disk.
- Now see how long it takes to build a kernel on a different disk.

This is not a completely silly scenario, and we have problems. I have one (dopey, hate it) patch which reduces that kernel build time by 30%.

The problem is that `gcc' is being forced to write back data to that scratch disk within the page allocator. It will take some algorithmic rework in the VM to fix this properly.

Another prominent source of latency is at the disk request layer, where heavy writers starve readers. There's a hack^Wfix in Alan's kernels for this, but nobody has discussed in in a while and I'm not sure how useful it is being.