kvm-autotest -- introducing kvm_runtest_2

Hello,

KVM-autotest is a test framework for kvm, based on autotest
(http://autotest.kernel.org).

Its purpose is to keep kvm stable.
For developers, we want to find regressions early.
For users, we want users to feel confident kvm runs well on their own machine.
Also we would like to present results nicely, similar to
http://test.kernel.org.

Today we have quite a few guests (including linux, windows, unix) and some tests (guest installation, boot, reboot and more)

For guest installation, we need the guest iso images. One needs to get the iso images before running guest installation tests (which is by default needed for all other tests to run).

Source code is located at
git://git.kernel.org/pub/scm/virt/kvm/kvm-autotest.git

Interesting directories are
client/tests/kvm_runtest_2 (newer)
client/tests/kvm_runtest (older, almost obsolete, not as interesting)

* Uri Lublin <[email protected]> [2009-03-01 13:10]:

Quoted 1 line

KVM-autotest is a test framework for kvm, based on autotest (http://autotest.kernel.org).

I've been digging into kvm_runtest_2 and have some feedback, but first to say, runtest_2 is huge cleanup and simplification from runtest, so thanks. Now for some comments:

- kvm_tests.cfg has a decent learning curve to wrap your head around. It would have been useful to have some debugging that would dump out what rules were filtering out guests... the dependencies aren't always easy to find. My first experience with the dep hunt is just changing 'only qcow2' in fc8_quick to raw and getting no output from kvm_config.py. Turns out, that 'raw' has a smp2 requirement ... that sort of filtering could be displayed with debugging output making configuration changes easier.

- documentation of keywords and structure would be nice, explaining what -variant , only and @ are doing for you, etc.

- it seems like the definition and rules ought to be separate from the last section which defines which tests to run (the fc8_quick area), so adding something as simple as include support to kvm_config.py would be sufficient to support a common definition file but different testing rules.

- kvm_runtest_2 as mentioned doesn't mess with your host networking and relies on -net user and redir, would be good to plumb through -net tap support that can be configured instead of always using -net user

- make -vnc parameter config/optional

I noticed the references to the setup isos for windows that presumbly install cygwin telnetd/sshd, are those available? if the isos themselves aren't, if the build instructions are, that would be very useful.

- guest install wizard using md5sum region matching ... ouch. This is quite fickle. I've seen different kvms generate different md5sum for the same region a couple of times. I know distributing screenshots of certain OSes is a grey area, but it would be nice to plumb through screenshot comparison and make that configurable. FWIW, I'll probably look at pulling the screenshot comparison bits from kvmtest and getting that integrated in kvm_runtest_2.

- kvm_runtest_2 looks a lot more like a regular autotest test, which is a Good Thing(TM). There are still some things that would prevent it going upstream autotest (which I assume is the long term goal)

- a lot of the ssh and scp work to copy autotest client into a guest is already handled by autoserv

- vm.py has a lot of infrastructure that should be integrated into autotest/server/kvm.py or possibly client-side common code to support the next comment

- kvm_tests.py defines new tests as functions, each of these tests should be a separate client tests which sounds like a pain, but should allow for easier test composition and hopefully make it easier to add new tests that look like any other client side test with just the implementation.py and control file
- this model moves toward eliminating kvm_runtest_2 and having a server-side generate a set of tests to run and spawning those on a target system.

I do still like the idea of having a client-side test that can just run on a developer/user's system to produce results without having to configure all of the autotest server-side bits.

Quoted 6 lines

- it seems like the definition and rules ought to be separate from the last section which defines which tests to run (the fc8_quick area), so adding something as simple as include support to kvm_config.py would be sufficient to support a common definition file but different testing rules.

An include support is one way to do it. We thought of a different way, which is
to add rules from the control file. So the control file would pick the test-list
it wants to run. Your suggestion is simpler but you need both a config file and
a control file to change the test-list. We need to change only the control file.

Quoted 5 lines

- kvm_runtest_2 as mentioned doesn't mess with your host networking and relies on -net user and redir, would be good to plumb through -net tap support that can be configured instead of always using -net user

We want to add -net tap support, as that is what users usually use. kvm_runtest does exactly that (a part of kvm_host.cfg). The drawbacks of tap are
(among others):
- One must run tests as root, or play with sudo/chmod (True for /dev/kvm, but
simpler)
- You have to a have a dhcpd around. kvm_runtest by default runs a local dhcpd
to serve kvm guests (part of setup/cleanup tests).
- A bit more difficult to configure.

Quoted 5 lines

I noticed the references to the setup isos for windows that presumbly install cygwin telnetd/sshd, are those available? if the isos themselves aren't, if the build instructions are, that would be very useful.

You are right. We do have an installation iso images for telnetd/sshd. I did not want to commit iso images. Also, I am not sure about licensing, and I prefer that we would generate them on the user machine. We'll add the build instructions to the wiki.

Quoted 7 lines

- guest install wizard using md5sum region matching ... ouch. This is quite fickle. I've seen different kvms generate different md5sum for the same region a couple of times. I know distributing screenshots of certain OSes is a grey area, but it would be nice to plumb through screenshot comparison and make that configurable. FWIW, I'll probably look at pulling the screenshot comparison bits from kvmtest and getting that integrated in kvm_runtest_2.

Creating a step file is not as easy as it seems, exactly for that reason. One has to pick a part of the screenshot that only available when input is expected and that would be consistent. We were thinking of replacing the md5sum with a tiny compressed image of the part of the image that was picked.

We had two other implementation for guest-wizard, one which only compares two/three consecutive screendumps (problems with e.g. clock ticking), and

one similar to kvmtest. The kvmtest way is to let the user create his/her own screendumps to be used later. I did not want to add so many screendumps images to the repository. Step-Maker keeps the images it uses, so we can compare them upon failure. Step-Editor lets the user to change a single barrier_2 step (or more) by looking at the original image and picking a different area.

Quoted 2 lines

- a lot of the ssh and scp work to copy autotest client into a guest is already handled by autoserv

That is true, but we want to be able to run it as client test too. That way a user does not have to install the server to run kvm tests on his/her machine.

Quoted 3 lines

- vm.py has a lot of infrastructure that should be integrated into autotest/server/kvm.py or possibly client-side common code to support the next comment

In the long term, there should be a client-server shared directory that deals with kvm guests (letting the host-client be the server for kvm-guests clients)

Quoted 9 lines

- kvm_tests.py defines new tests as functions, each of these tests should be a separate client tests which sounds like a pain, but should allow for easier test composition and hopefully make it easier to add new tests that look like any other client side test with just the implementation.py and control file
- this model moves toward eliminating kvm_runtest_2 and having a server-side generate a set of tests to run and spawning those on a target system.

I am not sure that I follow. Why implementing a test as a function is a pain ?

The plan is to keep kvm_runtest_2 in the client side, but move the configuration file + parser to the server (if one wants to dispatch test from the server). The server can dispatch the client test (using kvm_runtest_2 "test" + dictionary + tag). We have dependencies and we can spread unrelated kvm tests among similar hosts (e.g install guest OS and run tests for Fedora-10 64 bit on one machine, and install guest OS and run tests for Windows-XP 32 bit on another).

We may have to add hosts into the configuration file, or add it while running (from the control file ?). We sure do not want to go back to the way kvm_runtest works (playing with symbolic links so that autotest would find and run the test), and we do not want too many kvm_test_NAME directories under client/tests/ .

Quoted 3 lines

I do still like the idea of having a client-side test that can just run on a developer/user's system to produce results without having to configure all of the autotest server-side bits.

Me too :-)

Thanks for all the comments and suggestions,

Quoted 6 lines · Uri Lublin

Quoted 6 lines

 - it seems like the definition and rules ought to be separate from the last section which defines which tests to run (the fc8_quick area), so adding something as simple as include support to kvm_config.py would be sufficient to support a common definition file but different testing rules.

Quoted 8 lines

An include support is one way to do it. We thought of a different way, which is
to add rules from the control file. So the control file would pick the test-list
it wants to run. Your suggestion is simpler but you need both a config file and
a control file to change the test-list. We need to change only the control file.

OK, well, I viewed almost all of the test config file as static.  The rules about guests, features, config variants, can change over time, but not that often which is what I was wanting an include of that mostly static data and then something else that picked which guests and tests to run.  It does make sense for the control file to pick the set of tests, so I think we're in agreement, though I do think adding include support is still a good idea, but much lower on the priority list.

Quoted 5 lines

Quoted 5 lines

- kvm_runtest_2 as mentioned doesn't mess with your host networking and relies on -net user and redir, would be good to plumb through -net tap support that can be configured instead of always using -net user

Quoted 12 lines

We want to add -net tap support, as that is what users usually use. kvm_runtest does exactly that (a part of kvm_host.cfg). The drawbacks of tap are
(among others):
 - One must run tests as root, or play with sudo/chmod (True for /dev/kvm, but
simpler)
 - You have to a have a dhcpd around. kvm_runtest by default runs a local dhcpd
to serve kvm guests (part of setup/cleanup tests).
 - A bit more difficult to configure.

I don't want to have the test *setup* tap and all of the infrastructure like dhcp, or dnsmasq... I just want to be able to configure whether a guest is launched with -net tap or -net user.  kvm_runtest_2 punts on building and install kvm as well as the networking, which is a great idea, I just want to be flexible enough to run kvm_runtest_2 with -net tap as well as -net user.

Quoted 5 lines

Quoted 5 lines

I noticed the references to the setup isos for windows that presumbly install cygwin telnetd/sshd, are those available?  if the isos themselves aren't, if the build instructions are, that would be very useful.

Quoted 4 lines

You are right. We do have an installation iso images for telnetd/sshd. I did not want to commit iso images. Also, I am not sure about licensing, and I prefer that we would generate them on the user machine. We'll add the build instructions to the wiki.

Agreed.  Sounds like a plan.

Quoted 8 lines

Quoted 7 lines

- guest install wizard using md5sum region matching ... ouch.  This is quite fickle.  I've seen different kvms generate different md5sum for the same region a couple of times.  I know distributing screenshots of certain OSes is a grey area, but it would be nice to plumb through screenshot comparison and make that configurable.  FWIW, I'll probably look at pulling the screenshot comparison bits from kvmtest and getting that integrated in kvm_runtest_2.

Creating a step file is not as easy as it seems, exactly for that reason.

Quoted 4 lines

One has to pick a part of the screenshot that only available when input is expected and that would be consistent. We were thinking of replacing the md5sum with a tiny compressed image of the part of the image that was picked.

It isn't just that step file creation isn't easy is that even with a good stepfile with smart region boxes, md5sum can *still* fail because KVM renders one pixel in the region differently than the system where the original was created; this creates false positives failures.

Quoted 4 lines

We had two other implementation for guest-wizard, one which only compares two/three consecutive screendumps (problems with e.g. clock ticking), and

Right, I like the idea of the region selection in stepmaker, it lets us avoid areas which have VM specific info, like the device path or clock. I'd like to keep the current stepmaker and region code, what I'd like to see instead of just md5sum compare of the cropped region, to be able to plug in different comparisons.  If a user does have screens from stepmaker available, guest_wizard could attempt to do screen compare like we do in kvmtests with match percentages.  If no screens are available, fallback on md5 or some other comparison algorithm.

Quoted 6 lines

one similar to kvmtest. The kvmtest way is to let the user create his/her own screendumps to be used later. I did not want to add so many screendumps images to the repository. Step-Maker keeps the images it uses, so we can compare them upon failure. Step-Editor lets the user to change a single barrier_2 step (or more) by looking at the original image and picking a different area.

Agreed, I don't want to commit screens to the repo either, I just want to be able to use screens if a user has them available.

Quoted 5 lines

Quoted 2 lines

 - a lot of the ssh and scp work to copy autotest client into a guest is already handled by autoserv

That is true, but we want to be able to run it as client test too. That way a user does not have to install the server to run kvm tests on his/her machine.

While true, we should try to use the existing server code for autotest install.

Quoted 6 lines

Quoted 3 lines

 - vm.py has a lot of infrastructure that should be integrated into autotest/server/kvm.py  or possibly client-side common code to support the next comment

In the long term, there should be a client-server shared directory that deals with kvm guests (letting the host-client be the server for kvm-guests clients)

I believe client/common_lib is imported into server-side as common code, so moving kvm infrastructure bits there will allow server-side and any other client test to manipulate VM/KVM objects.

Quoted 11 lines

Quoted 9 lines

 - kvm_tests.py defines new tests as functions, each of these tests should be a separate client tests  which sounds like a pain, but should allow for easier test composition and hopefully make it easier to add new tests that look like any other client side test with just the implementation.py and control file
   - this model moves toward eliminating kvm_runtest_2 and having a server-side generate a set of tests to run and spawning those on a target system.

I am not sure that I follow. Why implementing a test as a function is a pain ?

Test as a function of course isn't a pain.  What I meant was that if I've already have to guests and I just want to do a migration test, it would be nice to just be able to:

% cd client/tests/kvm/migration
% $AUTOTEST_BIN ./control

I'd like to move away from tests eval-ing and exec-ing other tests; it just feels a little hacky and stands out versus other autotest client tests.

We can probably table the discussion until we push patches at autotest and see what that community thinks of kvm_runtest_2 at that time.

Quoted 7 lines

The plan is to keep kvm_runtest_2 in the client side, but move the configuration file + parser to the server (if one wants to dispatch test from the server). The server can dispatch the client test (using kvm_runtest_2 "test" + dictionary + tag). We have dependencies and we can spread unrelated kvm tests among similar hosts (e.g install guest OS and run tests for Fedora-10 64 bit on one machine, and install guest OS and run tests for Windows-XP 32 bit on another).

Yeah, that sounds reasonable.

Quoted 5 lines

We may have to add hosts into the configuration file, or add it while running (from the control file ?). We sure do not want to go back to the way kvm_runtest works (playing with symbolic links so that autotest would find and run the test), and we do not want too many kvm_test_NAME directories under client/tests/ .

Agree with no symbolic links.  If we move common kvm utils and objects to client/common_lib that avoids any of that hackery.

On the dir structure, I agree we don't have to pollute client/tests with a ton of kvm_* tests.  I'll look around upstream autotest and see if there is an other client side tests to use an example.

Quoted 6 lines

Quoted 3 lines

 I do still like the idea of having a client-side test that can just run on a developer/user's system to produce results without having to configure all of the autotest server-side bits.

Me too :-)

Thanks for all the comments and suggestions,

Sure

--
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 57 lines · Ryan Harper

Quoted 6 lines · Uri Lublin

Quoted 6 lines

- it seems like the definition and rules ought to be separate from the last section which defines which tests to run (the fc8_quick area), so adding something as simple as include support to kvm_config.py would be sufficient to support a common definition file but different testing rules.

Quoted 9 lines

An include support is one way to do it. We thought of a different way, which is
to add rules from the control file. So the control file would pick the test-list
it wants to run. Your suggestion is simpler but you need both a config file and
a control file to change the test-list. We need to change only the control file.

OK, well, I viewed almost all of the test config file as static. The rules about guests, features, config variants, can change over time, but not that often which is what I was wanting an include of that mostly static data and then something else that picked which guests and tests to run. It does make sense for the control file to pick the set of tests, so I think we're in agreement, though I do think adding include support is still a good idea, but much lower on the priority list.

Quoted 4 lines

Quoted 4 lines

- kvm_runtest_2 as mentioned doesn't mess with your host networking and relies on -net user and redir, would be good to plumb through -net tap support that can be configured instead of always using -net user

Quoted 13 lines

We want to add -net tap support, as that is what users usually use. kvm_runtest does exactly that (a part of kvm_host.cfg). The drawbacks of tap are
(among others):
- One must run tests as root, or play with sudo/chmod (True for /dev/kvm, but
simpler)
- You have to a have a dhcpd around. kvm_runtest by default runs a local dhcpd
to serve kvm guests (part of setup/cleanup tests).
- A bit more difficult to configure.

I don't want to have the test *setup* tap and all of the infrastructure like dhcp, or dnsmasq... I just want to be able to configure whether a guest is launched with -net tap or -net user. kvm_runtest_2 punts on building and install kvm as well as the networking, which is a great idea, I just want to be flexible enough to run kvm_runtest_2 with -net tap as well as -net user.

Quoted 5 lines

Quoted 5 lines

I noticed the references to the setup isos for windows that presumbly install cygwin telnetd/sshd, are those available? if the isos themselves aren't, if the build instructions are, that would be very useful.

Quoted 18 lines

Quoted 5 lines

You are right. We do have an installation iso images for telnetd/sshd. I did not want to commit iso images. Also, I am not sure about licensing, and I prefer that we would generate them on the user machine. We'll add the build instructions to the wiki.

Agreed. Sounds like a plan.

Quoted 10 lines

Quoted 8 lines

- guest install wizard using md5sum region matching ... ouch. This is quite fickle. I've seen different kvms generate different md5sum for the same region a couple of times. I know distributing screenshots of certain OSes is a grey area, but it would be nice to plumb through screenshot comparison and make that configurable. FWIW, I'll probably look at pulling the screenshot comparison bits from kvmtest and getting that integrated in kvm_runtest_2.

Creating a step file is not as easy as it seems, exactly for that reason.

I agree here 100% as per my experience.

Quoted 5 lines

Quoted 5 lines

One has to pick a part of the screenshot that only available when input is expected and that would be consistent. We were thinking of replacing the md5sum with a tiny compressed image of the part of the image that was picked.

Quoted 5 lines

It isn't just that step file creation isn't easy is that even with a good stepfile with smart region boxes, md5sum can *still* fail because KVM renders one pixel in the region differently than the system where the original was created; this creates false positives failures.

I also have faced this issue. Even on the same system the step file may fail. I saw few(though not frequent) situations where the same md5sum passed in one time and failed in the other attempt.

Quoted 4 lines

Quoted 4 lines

We had two other implementation for guest-wizard, one which only compares two/three consecutive screendumps (problems with e.g. clock ticking), and

Quoted 9 lines

Right, I like the idea of the region selection in stepmaker, it lets us avoid areas which have VM specific info, like the device path or clock. I'd like to keep the current stepmaker and region code, what I'd like to see instead of just md5sum compare of the cropped region, to be able to plug in different comparisons. If a user does have screens from stepmaker available, guest_wizard could attempt to do screen compare like we do in kvmtests with match percentages. If no screens are available, fallback on md5 or some other comparison algorithm.

That sounds better. Yet none of my step files worked in one go. I remember running my test and stepmaker parallelly and continue taking screenshots untill one passes and putting that md5sum in the step file. So at the end I was fully meshed up with respect to the cropped images and I had no idea which cropped image corresponded to which md5sum.

Though the RHEL5.3 step files that I generated in the text mode installation were quite strong.

Quoted 11 lines

Quoted 7 lines

one similar to kvmtest. The kvmtest way is to let the user create his/her own screendumps to be used later. I did not want to add so many screendumps images to the repository. Step-Maker keeps the images it uses, so we can compare them upon failure. Step-Editor lets the user to change a single barrier_2 step (or more) by looking at the original image and picking a different area.

Agreed, I don't want to commit screens to the repo either, I just want to be able to use screens if a user has them available.

I have two questions with respect to stepfiles.
1. The timeouts: Timeouts may fall short if a step file generated on a high end machine is to be used on a very low config machine or installing N virtual machines (say N ~ 50,100 etc) in parallel.
2. If there are changes in KVM display in future the md5sum will fail. So are we prepared for that?

Quoted 99 lines

Quoted 7 lines

Quoted 3 lines

- a lot of the ssh and scp work to copy autotest client into a guest is already handled by autoserv

That is true, but we want to be able to run it as client test too. That way a user does not have to install the server to run kvm tests on his/her machine.

While true, we should try to use the existing server code for autotest install.

Quoted 8 lines

Quoted 4 lines

- vm.py has a lot of infrastructure that should be integrated into autotest/server/kvm.py or possibly client-side common code to support the next comment

In the long term, there should be a client-server shared directory that deals with kvm guests (letting the host-client be the server for kvm-guests clients)

I believe client/common_lib is imported into server-side as common code, so moving kvm infrastructure bits there will allow server-side and any other client test to manipulate VM/KVM objects.

Quoted 9 lines

Quoted 9 lines

- kvm_tests.py defines new tests as functions, each of these tests should be a separate client tests which sounds like a pain, but should allow for easier test composition and hopefully make it easier to add new tests that look like any other client side test with just the implementation.py and control file
- this model moves toward eliminating kvm_runtest_2 and having a server-side generate a set of tests to run and spawning those on a target system.

Quoted 3 lines

I am not sure that I follow. Why implementing a test as a function is a pain ?

Test as a function of course isn't a pain. What I meant was that if I've already have to guests and I just want to do a migration test, it would be nice to just be able to:

% cd client/tests/kvm/migration
% $AUTOTEST_BIN ./control

I'd like to move away from tests eval-ing and exec-ing other tests; it just feels a little hacky and stands out versus other autotest client tests.

We can probably table the discussion until we push patches at autotest and see what that community thinks of kvm_runtest_2 at that time.

Quoted 8 lines

The plan is to keep kvm_runtest_2 in the client side, but move the configuration file + parser to the server (if one wants to dispatch test from the server). The server can dispatch the client test (using kvm_runtest_2 "test" + dictionary + tag). We have dependencies and we can spread unrelated kvm tests among similar hosts (e.g install guest OS and run tests for Fedora-10 64 bit on one machine, and install guest OS and run tests for Windows-XP 32 bit on another).

Yeah, that sounds reasonable.

Quoted 6 lines

We may have to add hosts into the configuration file, or add it while running (from the control file ?). We sure do not want to go back to the way kvm_runtest works (playing with symbolic links so that autotest would find and run the test), and we do not want too many kvm_test_NAME directories under client/tests/ .

Agree with no symbolic links. If we move common kvm utils and objects to client/common_lib that avoids any of that hackery.

On the dir structure, I agree we don't have to pollute client/tests with a ton of kvm_* tests. I'll look around upstream autotest and see if there is an other client side tests to use an example.

Quoted 8 lines

Quoted 4 lines

I do still like the idea of having a client-side test that can just run on a developer/user's system to produce results without having to configure all of the autotest server-side bits.

Me too :-)

Thanks for all the comments and suggestions,

Sure

--
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 5 lines

I noticed the references to the setup isos for windows that presumbly install cygwin telnetd/sshd, are those available? if the isos themselves aren't, if the build instructions are, that would be very useful.

You are right. We do have an installation iso images for telnetd/sshd. I did not want to commit iso images. Also, I am not sure about licensing, and I prefer that we would generate them on the user machine. We'll add the build instructions to the wiki.

Quoted 6 lines · Uri Lublin

Quoted 4 lines

I noticed the references to the setup isos for windows that presumbly install cygwin telnetd/sshd, are those available? if the isos themselves aren't, if the build instructions are, that would be very useful.

You are right. We do have an installation iso images for telnetd/sshd.
I did not want to commit iso images. Also, I am not sure about licensing, and I prefer that we would generate them on the user machine. We'll add the build instructions to the wiki.

Any ETA on the build instructions?