Skip to contentSkip to conversation

Serious performance regression in Wine applications and Linux 2.6.24.*

DISCUSSION Kai 4 participants 11 messages Mar 23, 2008 – Aug 7, 2008 1 message not in the archive

lore
Share & tools
lore

Kai

author
#0 lore raw

Originally, it was just about everything -- Wine itself, and most any app I tried to run with it.

In my most recent tests, however, it only seems to affect Age of Wonders II and Age of Wonders Shadow Magic, which use very similar engines.

Quoted 20 lines · David
Quoted 9 lines · Kai

Performance degrades by about 60% when I run Wine under any of the 2.6.24.* kernels. Attached are the output of lspci -vv and the two config files of each kernel.

I upgraded to 2.6.24, back to 2.6.23.(not sure), then tried 2.6.24.1 and 2.6.24.3 and the issue is present in both; it's severe enough that Wine apps are virtually unusable for me with this version of the kernel; I'm having to use 2.6.23 until this somehow is resolved.

Quoted 5 lines

I'd like some help figuring out why this performance regression exists, and what can be done to mitigate it.

Please CC replies; I'm not on the list.

This is strange, as my (two) Windows apps ran (and run) just fine under 2.6.24 (and now under 2.6.25-rc6). Could you specify the applications you're having problems with?

David

#1 lore raw

(Please don't top post. If we figure out who to CC: who knows more about this than I do -- which wouldn't be hard -- then they're going to want to see all this in order. Also, please always do a reply-to-all on this list, it's preferred.)

Quoted 19 lines · Kai
Quoted 10 lines
Quoted 9 lines · Kai

Performance degrades by about 60% when I run Wine under any of the 2.6.24.* kernels. Attached are the output of lspci -vv and the two config files of each kernel.

I upgraded to 2.6.24, back to 2.6.23.(not sure), then tried 2.6.24.1 and 2.6.24.3 and the issue is present in both; it's severe enough that Wine apps are virtually unusable for me with this version of the kernel; I'm having to use 2.6.23 until this somehow is resolved.

Quoted 8 lines
Quoted 5 lines

I'd like some help figuring out why this performance regression exists, and what can be done to mitigate it.

Please CC replies; I'm not on the list.

This is strange, as my (two) Windows apps ran (and run) just fine under 2.6.24 (and now under 2.6.25-rc6). Could you specify the applications you're having problems with?

Quoted 5 lines

Originally, it was just about everything -- Wine itself, and most any
app I tried to run with it.

In my most recent tests, however, it only seems to affect Age of Wonders II and Age of Wonders Shadow Magic, which use very similar engines.

What version of X? What version of WINE? Which video driver are you using? Does the speed issue show up with glxgears, or something else that would be much easier for someone else to reproduce (tuxracer, etc.)? Is your system using software or hardware rendering?

author
#2 lore raw
Quoted 23 lines · Kai
Quoted 22 lines
Quoted 10 lines
Quoted 9 lines · Kai

Performance degrades by about 60% when I run Wine under any of the 2.6.24.* kernels. Attached are the output of lspci -vv and the two config files of each kernel.

I upgraded to 2.6.24, back to 2.6.23.(not sure), then tried 2.6.24.1 and 2.6.24.3 and the issue is present in both; it's severe enough that Wine apps are virtually unusable for me with this version of the kernel; I'm having to use 2.6.23 until this somehow is resolved.

Quoted 8 lines
Quoted 5 lines

I'd like some help figuring out why this performance regression exists, and what can be done to mitigate it.

Please CC replies; I'm not on the list.

This is strange, as my (two) Windows apps ran (and run) just fine under 2.6.24 (and now under 2.6.25-rc6). Could you specify the applications you're having problems with?

Originally, it was just about everything -- Wine itself, and most any
app I tried to run with it.

Quoted 7 lines
Quoted 2 lines

In my most recent tests, however, it only seems to affect Age of Wonders II and Age of Wonders Shadow Magic, which use very similar engines.

What version of X? What version of WINE? Which video driver are you using? Does the speed issue show up with glxgears, or something else that would be much easier for someone else to reproduce (tuxracer, etc.)? Is your system using software or hardware rendering?

The issue only shows with Age of Wonders II and AoW Shadow Magic.

Every version of wine from 0.9.55 to 0.9.58, including several intermediate git builds.

Xorg version 1.4.0.90, and a couple other versions (the problems has persisted across a couple updates of Xorg)

Hardware rendering; using the prop. nV driver; updating that also doesn't affect the issue, nor does retaining the same version.

#3 lore raw
Quoted 10 lines · Kai

The issue only shows with Age of Wonders II and AoW Shadow Magic.

Every version of wine from 0.9.55 to 0.9.58, including several intermediate git builds.

Xorg version 1.4.0.90, and a couple other versions (the problems has persisted across a couple updates of Xorg)

Hardware rendering; using the prop. nV driver; updating that also doesn't affect the issue, nor does retaining the same version.

could you run this script while such a slowdown is really prominent:

http://people.redhat.com/mingo/cfs-scheduler/tools/cfs-debug-info.sh

and send me the output it generates? The output is the most useful if you do this on a kernel that has CONFIG_SCHED_DEBUG=y and CONFIG_SCHEDSTATS=y enabled.

on the off chance that this issue has been fixed in the soon-to-be 2.6.25 kernel, you might also want to try x86.git/latest, which is based on the latest Linus tree and has all relevant x86 fixes and improvements added as well:

http://people.redhat.com/mingo/x86.git/README

several of the changes can affect performance.

a third (and most comprehensive) way to debug this would be to send me a scheduler trace of such a slowdown, you can generate a scheduler trace the following way:

http://people.redhat.com/mingo/sched-devel.git/readme-tracer.txt

but we can probably give a first estimation based on the cfs-debug-info output already. Btw., you can combine the scheduler and the x86 git tree into a temporary unified tree by doing these two commands:

git-checkout -b tmp x86/latest
git-merge sched-devel/latest

(run "make oldconfig" to pick up the new config options.)

Ingo

author
#4 lore raw
Quoted 37 lines · Ingo Molnar
Quoted 10 lines · Kai

The issue only shows with Age of Wonders II and AoW Shadow Magic.

Every version of wine from 0.9.55 to 0.9.58, including several intermediate git builds.

Xorg version 1.4.0.90, and a couple other versions (the problems has persisted across a couple updates of Xorg)

Hardware rendering; using the prop. nV driver; updating that also doesn't affect the issue, nor does retaining the same version.

could you run this script while such a slowdown is really prominent:

http://people.redhat.com/mingo/cfs-scheduler/tools/cfs-debug-info.sh

and send me the output it generates? The output is the most useful if you do this on a kernel that has CONFIG_SCHED_DEBUG=y and CONFIG_SCHEDSTATS=y enabled.

on the off chance that this issue has been fixed in the soon-to-be 2.6.25 kernel, you might also want to try x86.git/latest, which is based on the latest Linus tree and has all relevant x86 fixes and improvements added as well:

http://people.redhat.com/mingo/x86.git/README

several of the changes can affect performance.

a third (and most comprehensive) way to debug this would be to send me a scheduler trace of such a slowdown, you can generate a scheduler trace the following way:

http://people.redhat.com/mingo/sched-devel.git/readme-tracer.txt

Quoted 8 lines

but we can probably give a first estimation based on the cfs-debug-info output already. Btw., you can combine the scheduler and the x86 git tree into a temporary unified tree by doing these two commands:

git-checkout -b tmp x86/latest
git-merge sched-devel/latest

(run "make oldconfig" to pick up the new config options.)

Okay, so I don't know a lot about git; I tried following the instructions in the readme at
http://people.redhat.com/mingo/x86.git/README to get a latest x86 tree, and merge it with sched-devel/latest using the commands git-checkout -b tmp x86/latest and git-merge sched-devel/latest:

mkdir linux.trees.git || exit -1
cd linux.trees.git
git-init-db
git-remote add linus
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
git-remote add x86
git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86.git
git-remote update
git-checkout -b tmp x86/latest
git-merge sched-devel/latest

Then I copied over a config file and updated it, but building failed shortly into the process, with some errors about missing a } before ;, and excess elements in scalar initializer, in file arch/x86/kernel/init_task.cc.. anyway, I tried to update again, using git-remote update, but it kept giving me this output:

Updating linus
Updating sched-devel
fatal: The remote end hung up unexpectedly
fetch sched-devel: command returned error: 128

So I'm just going to test using 2.6.25-rc8-git2; however, can someone also give me the proper sequence of commands to get a working tree as Ingo Molnar requested, so I can test that way, too? Sorry to be a pain.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw
Quoted 2 lines · Kai

I did, however, test it on 2.6.25-rc8, and attached is the output of cfs-debug-info.sh

your vmstat:

-- vmstat:
--procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 3  0      0 870540    340 746184    0    0   952   137  198 4752 21 12 60  7
 1  0      0 870952    340 746320    0    0   128     0  356 17723 28 23 50  0
 1  0      0 872224    340 746320    0    0     0    21  394 18066 30 20 50  0
 1  0      0 872244    340 746320    0    0     0     0  353 17987 29 22 50  0
 1  0      0 871416    340 746320    0    0     0     0  392 17971 28 22 50  0

suggests that there's a constant 50% idle time. Probably one core running X and the client, the other core staying idle?

Ingo

author
#7 lore raw

That is possible; I will see if I'm only using one core in 2.6.23 as well; if not, that might suggest what our problem is.

Quoted 28 lines · Ingo Molnar
Quoted 2 lines · Kai

I did, however, test it on 2.6.25-rc8, and attached is the output of cfs-debug-info.sh

your vmstat:

-- vmstat:
--procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
3 0 0 870540 340 746184 0 0 952 137 198 4752 21 12
60 7
1 0 0 870952 340 746320 0 0 128 0 356 17723 28 23
50 0
1 0 0 872224 340 746320 0 0 0 21 394 18066 30 20
50 0
1 0 0 872244 340 746320 0 0 0 0 353 17987 29 22
50 0
1 0 0 871416 340 746320 0 0 0 0 392 17971 28 22
50 0

suggests that there's a constant 50% idle time. Probably one core running X and the client, the other core staying idle?

Ingo

author
#8 lore raw
In reply to Ingo Molnar

Some stuff came up and I was unable to focus on this bug for a while, but it's gone as of 2.6.26. Performance is comparable to 2.6.23 once again, maybe a bit better in places.

Quoted 28 lines · Ingo Molnar
Quoted 2 lines · Kai

I did, however, test it on 2.6.25-rc8, and attached is the output of cfs-debug-info.sh

your vmstat:

-- vmstat:
--procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
3 0 0 870540 340 746184 0 0 952 137 198 4752 21 12
60 7
1 0 0 870952 340 746320 0 0 128 0 356 17723 28 23
50 0
1 0 0 872224 340 746320 0 0 0 21 394 18066 30 20
50 0
1 0 0 872244 340 746320 0 0 0 0 353 17987 29 22
50 0
1 0 0 871416 340 746320 0 0 0 0 392 17971 28 22
50 0

suggests that there's a constant 50% idle time. Probably one core running X and the client, the other core staying idle?

Ingo

#9 lore raw
In reply to Kai
Quoted 5 lines · Kai

Originally, it was just about everything -- Wine itself, and most any app I tried to run with it.

In my most recent tests, however, it only seems to affect Age of Wonders II and Age of Wonders Shadow Magic, which use very similar engines.

A simple test if the scheduler is likely to blame would be to compare context switch rates between the different kernel versions who show differing performance. You can do that with running "vmstat 1" in parallel and checking the "cs" column.

Another powerful but quite slow method if you can easily reproduce it would be a full git bisect. That would identify exactly which change introduced your problem.

-Andi

author
#10 lore raw
Quoted 13 lines · Andi Kleen
Quoted 5 lines · Kai

Originally, it was just about everything -- Wine itself, and most any app I tried to run with it.

In my most recent tests, however, it only seems to affect Age of Wonders II and Age of Wonders Shadow Magic, which use very similar engines.

A simple test if the scheduler is likely to blame would be to compare context switch rates between the different kernel versions who show differing performance. You can do that with running "vmstat 1" in parallel and checking the "cs" column.

Quoted 5 lines

Another powerful but quite slow method if you can easily reproduce it would be a full git bisect. That would identify exactly which change introduced your problem.

-Andi

Can you explain more clearly what you mean with vmstat?

Also, I'd like to find a way to feasibly do the git bisect (I assume you basically mean testing against every change in sequence since .23), but I'd like some advice on how to do this, and how to do it reasonably expeditiously.

#11 lore raw
Quoted 1 line

Can you explain more clearly what you mean with vmstat?

Open shell window. Enter "vmstat 1". Run Wine application. Watch "cs" column over time.

An possible improvement would be also to use a larger number than 1 to get values over long time.

Compare that value to the old kernel.

Quoted 3 lines

Also, I'd like to find a way to feasibly do the git bisect (I assume you basically mean testing against every change in sequence since .23), but

Yes, although it is just every 1+log2 changes

Quoted 2 lines

I'd like some advice on how to do this, and how to do it reasonably expeditiously.

Type git bisect into google, there are lots of tutorials.

-Andi

Quoted 1 line