Benchmarking on CentOS 5
Hi,
I wanted to some benchmarking on a CentOS 5 host. However, the warning in dmesg about preempt notifiers indicates I shouldn't benchmark. Is there a workaround (aside from updating the kernel) that handles the issue?
Thanks,
Cam
Quoted 6 lines · Cam Macdonell
Hi,
I wanted to some benchmarking on a CentOS 5 host. However, the warning in dmesg about preempt notifiers indicates I shouldn't benchmark. Is there a workaround (aside from updating the kernel) that handles the issue?
The problem is that the preempt notifier emulation through hardware breakpoints is worse at optimizing lightweight vm exists than the real preempt notifiers supported by recent kernels. If you benchmark without upgrading the kernel, it has to be clear it is running slower than it would on a recent optimized host kernel. If you decide to upgrade and compile the host kernel yourself, you need to configure it with CONFIG_KVM=m to be sure the preempt notifiers are enabled in the host kernel and in turn the printk will go away.
Thanks!
Quoted 5 lines · Andrea Arcangeli
Quoted 3 lines · Cam Macdonell
Hi,
I wanted to some benchmarking on a CentOS 5 host. However, the warning in dmesg about preempt notifiers indicates I shouldn't benchmark. Is there a workaround (aside from updating the kernel) that handles the issue?
Quoted 8 lines
The problem is that the preempt notifier emulation through hardware breakpoints is worse at optimizing lightweight vm exists than the real preempt notifiers supported by recent kernels. If you benchmark without upgrading the kernel, it has to be clear it is running slower than it would on a recent optimized host kernel. If you decide to upgrade and compile the host kernel yourself, you need to configure it with
CONFIG_KVM=mto be sure the preempt notifiers are enabled in the host kernel and in turn the printk will go away.
ok. so what is qumranet recommendation?
the just released latest rhel 5.2 kernel is kernel-2.6.18-92.el5 is it good enough for kvm host os? or its' better to change some other distro eg: fedora 9?
until now we try to use the latest rhel/centos on all of our servers while we use fedora (currently 8 but may be upgrade in a few weeks if 9 become stable) on desktops. but now it seems probably then for a kvm host rhel/centos is not enough:-(
Signature · 1 line
Levente "Si vis pacem para bellum!"
Quoted 1 line · Farkas Levente
ok. so what is qumranet recommendation?
I'm not aware of a Qumranet reccomandation for this but I can give you my opinion.
Quoted 7 lines
the just released latest rhel 5.2 kernel is kernel-2.6.18-92.el5 is it good enough for kvm host os? or its' better to change some other distro eg:
fedora 9?
until now we try to use the latest rhel/centos on all of our servers while we use fedora (currently 8 but may be upgrade in a few weeks if 9 become stable) on desktops. but now it seems probably then for a kvm host rhel/centos is not enough:-(
This is a almost the opposite question of the previous one from Cam ;), here you're asking about a production kernel, Cam was asking for a benchmarking setup.
I think any distro enterprise kernel is better for production systems than a latest mainline, but that's just me. You'll find others preferring to run 2.6.25 in production a few days after it is released.
The slowdown we're talking about here for the preempt notifiers isn't going to make a big difference in a production system, more important that you're sure your host kernel is rock solid and well tested IMHO. But if it was pure benchmarking what you were doing, then using latest mainline was better to get the best possible score, that is why such printk is there.
Quoted 18 lines · Andrea Arcangeli
Quoted 1 line · Farkas Levente
ok. so what is qumranet recommendation?
I'm not aware of a Qumranet reccomandation for this but I can give you my opinion.
Quoted 2 lines
the just released latest rhel 5.2 kernel is kernel-2.6.18-92.el5 is it good enough for kvm host os? or its' better to change some other distro eg: fedora 9?
until now we try to use the latest rhel/centos on all of our servers while we use fedora (currently 8 but may be upgrade in a few weeks if 9 become stable) on desktops. but now it seems probably then for a kvm host rhel/centos is not enough:-(This is a almost the opposite question of the previous one from Cam ;), here you're asking about a production kernel, Cam was asking for a benchmarking setup.
I think any distro enterprise kernel is better for production systems than a latest mainline, but that's just me. You'll find others preferring to run 2.6.25 in production a few days after it is released.
Quoted 6 lines
The slowdown we're talking about here for the preempt notifiers isn't going to make a big difference in a production system, more important that you're sure your host kernel is rock solid and well tested IMHO. But if it was pure benchmarking what you were doing, then using latest mainline was better to get the best possible score, that is why such printk is there.
i want rock solid, but smp guest and both 32 and 64bit guest centos too. 5-10% performance lost is not important but 50-200% is important. so my real question is that kernel-2.6.18-92.el5 has "enough" features for an good working kvm if i recompile kvm and kvm-kmod? ehough here means no more than 5-10% performance lost.
Signature · 1 line
Levente "Si vis pacem para bellum!"
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
this is out production server at the development department (10-15) people using it so actually if i tell them that i'll stop the host and all guests for max an hour it's acceptable, but more not really. it's run it type programs. from my experience in the last 6-12 months is that kvm is not production ready. as you can read from this list there are far too many change day-by-day which are very core. and this comes from the current state of kvm. which indicate that rh can't include in there stable server distro such a kvm version which is feature rich and stable. i hope rhel 6 will be based on at least kernel-2.6.25 or later. and in that case it'll production ready, but even then may be at rhel 6.1 (since imho rhel 6.0 will be based on a kernel which is released nowadays). imho the biggest problem with the current development of kvm that there is not a stable releases which is somewhat related to the current release number. eg kvm-0.5.x kvm-0.6.x would be better. but currently kvm development is so fast that keep 2-3 parallel branch where there is a development and stable release seems to too much work.
so to answer to your question i don't know:-(
i'd like to see a kvm version which is working on rhel/centos 5 host on intel and amd, with smp and stable (ie. boot, reboot, save state, a run without crash) and the performance is similar (ie 80-90%) to the development version. unfortunately it seems currently there is no such version:-( may be around 62 one of the version is the best choice.
next week i'll spend a half day to try to find a stable version...
On Fri, May 30, 2008 at 3:38 PM, David S. Ahern <[email protected] <mailto:[email protected]>> wrote:
Do you run in a production type environment or lab environment? Do the guests run standard IT type programs -- web server, database, etc?
I ask because something I am working on here might force me to drop back to RHEL5, but I still want to use kvm for the virtualization layer. I'm trying to get an idea of what to expect. It sounds like it has worked ok for you overall, you just have to be picky about specific kvm releases you use.
thanks for the information,
david
Quoted 1 line · Farkas Levente
yes as i wrote i always recompile create new rpms for kvm and
kvm-kmod
Quoted 2 lines
on centos. the host hasn't any problem (with the last few dozens of kvm), but the guest's are more problematic. we need both 32 and
64 bit
Quoted 2 lines
smp guests and it cause a lots of problem. first is the time source, second (as i wrote earlier guest see only 3 of 4 vcpu) and with
kvm-69
Quoted 1 line
we've got a lots of performance lost compared to kvm-62. and see
random
Quoted 3 lines
high load on the guests.
On Fri, May 30, 2008 at 5:34 AM, David S. Ahern
<[email protected] <mailto:[email protected]>
Quoted 3 lines
<mailto:[email protected] <mailto:[email protected]>>> wrote:
I take it from your posts you are running newer kvm releases
(e.g.,
Quoted 1 line
kvm-69) on Centos 5 for the host. How's that working out for
you? Any
Quoted 8 lines
major hiccups -- stability issues or runtime issues?
david
Farkas Levente wrote:
> Andrea Arcangeli wrote:
>> On Wed, May 28, 2008 at 10:27:08PM +0200, Farkas Levente
Quoted 7 lines · wrote:
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
ok. so what is qumranet recommendation?
I'm not aware of a Qumranet reccomandation for this but I can
give you
>> my opinion.
>>
>>> the just released latest rhel 5.2 kernel is
kernel-2.6.18-92.el5 is
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
it good enough for kvm host os? or its' better to change
some other
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
distro eg: fedora 9?
until now we try to use the latest rhel/centos on all of
our servers
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
while we use fedora (currently 8 but may be upgrade in a few
weeks if
>>> 9 become stable) on desktops. but now it seems probably
then for a
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
kvm host rhel/centos is not enough:-(
This is a almost the opposite question of the previous one
from Cam
Quoted 1 line
Quoted 1 line
Quoted 1 line
;), here you're asking about a production kernel, Cam was
asking
Quoted 4 lines
for a
>> benchmarking setup.
>>
>> I think any distro enterprise kernel is better for
production systems
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
than a latest mainline, but that's just me. You'll find others preferring to run 2.6.25 in production a few days after it is released.
The slowdown we're talking about here for the preempt
notifiers isn't
Quoted 1 line
Quoted 1 line
Quoted 1 line
going to make a big difference in a production system,
more important
Quoted 1 line
Quoted 1 line
Quoted 1 line
that you're sure your host kernel is rock solid and well
tested
Quoted 1 line
Quoted 1 line
Quoted 1 line
IMHO. But if it was pure benchmarking what you were doing,
then using
Quoted 1 line
Quoted 1 line
Quoted 1 line
latest mainline was better to get the best possible score,
that
Quoted 6 lines
is why
>> such printk is there.
>
> i want rock solid, but smp guest and both 32 and 64bit guest centos too.
> 5-10% performance lost is not important but 50-200% is
important.
Quoted 2 lines
so my
> real question is that kernel-2.6.18-92.el5 has "enough"features
Quoted 2 lines
for an
> good working kvm if i recompile kvm and kvm-kmod? ehough
here means no
Quoted 8 lines
Quoted 2 lines
more than 5-10% performance lost.
--
Levente "Si vis pacem para bellum!"
Signature · 4 lines
Levente "Si vis pacem para bellum!" -- Levente "Si vis pacem para bellum!"
Quoted 2 lines · Farkas Levente
without crash) and the performance is similar (ie 80-90%) to the development version. unfortunately it seems currently there is no such
I hope you didn't reach this conclusion about different performance between enterprise kernel host and mainline kernel host because I didn't answer promptly to your previous email, sorry!
With regard to your performance question, preempt notifier emulation really shouldn't be noticeable in real life. It worth to run the fastest code only if you're doing pure benchmarking, in which case you want to avoid any unnecessary exception and heavyweight exit (that should reduces the time to schedule of a couple thousand cycles I guess, but re-schedule events aren't so frequent and this is a per-task per-cpu breakpoint with my latest emulation logic taking advantage of the __switch_to internals).
I can hardly measure any difference here between SLES10 SP2 host and mainline host with a simple dd from pagecache to /dev/null. The change between enterprise kernel and current mainline kernel may be more significant for the final performance than the impact of the preempt notifier emulation. In any case the difference should be much less than 5% as far as preempt notifier emulation is concerned. It should be couple usec lost every couple msec or so.
Quoted 7 lines · On Friday 30 May 2008 23:00:41 Farkas Levente wrote:
this is out production server at the development department (10-15) people using it so actually if i tell them that i'll stop the host and all guests for max an hour it's acceptable, but more not really. it's run it type programs. from my experience in the last 6-12 months is that kvm is not production ready. as you can read from this list there are far too many change day-by-day which are very core. and this comes from the current state of kvm. which indicate that rh can't include in there
You'll find the most stable version of kvm in the kernel that your distribution ships. Linux-2.6.x (where x > 20) should also be stable. The development on kvm will continue to proceed at a fast pace, so you'll see several kvm releases and this, as a result, is bound to bring in a few new bugs in each iteration.
Quoted 3 lines
imho the biggest problem with the current development of kvm that there is not a stable releases which is somewhat related to the current release number. eg kvm-0.5.x kvm-0.6.x would be better. but currently
So the short answer is: if you're looking for a stable version of kvm, look at a kernel.org kernel or the kvm version provided to you by your distribution.
Quoted 3 lines
kvm development is so fast that keep 2-3 parallel branch where there is a development and stable release seems to too much work.
so to answer to your question i don't know:-(
The "stable" branch of kvm is the one in the most-recently available Linux kernel from kernel.org. kvm.git is the development version.
Quoted 22 lines · Amit Shah
Quoted 7 lines · On Friday 30 May 2008 23:00:41 Farkas Levente wrote:
this is out production server at the development department (10-15) people using it so actually if i tell them that i'll stop the host and all guests for max an hour it's acceptable, but more not really. it's run it type programs. from my experience in the last 6-12 months is that kvm is not production ready. as you can read from this list there are far too many change day-by-day which are very core. and this comes from the current state of kvm. which indicate that rh can't include in there
You'll find the most stable version of kvm in the kernel that your distribution ships. Linux-2.6.x (where x > 20) should also be stable. The development on kvm will continue to proceed at a fast pace, so you'll see several kvm releases and this, as a result, is bound to bring in a few new bugs in each iteration.
Quoted 3 lines
imho the biggest problem with the current development of kvm that there is not a stable releases which is somewhat related to the current release number. eg kvm-0.5.x kvm-0.6.x would be better. but currently
So the short answer is: if you're looking for a stable version of kvm, look at a kernel.org kernel or the kvm version provided to you by your distribution.
Quoted 6 lines
Quoted 3 lines
kvm development is so fast that keep 2-3 parallel branch where there is a development and stable release seems to too much work.
so to answer to your question i don't know:-(The "stable" branch of kvm is the one in the most-recently available Linux kernel from kernel.org. kvm.git is the development version.
In the near future we'll publish a stable branch. There are actually 2
repositories: kernel repo, based on the latest kernel - 2.6.26 and a userspace repository that will be based on kvm-68.
The idea is to maintain the above repos together and only apply bug fixes. New features will come with every next kernel release.
We're in the process of creating an automatic test framework for kvm. It will be open source framework based on autotest and similar to Anthony's kvmtest. It will help stabilizing both the 'stable' branch and the master.
Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html