kvm-autotest
Hello,
We are happy to announce the availability of kvm-autotest, a test framework
for KVM based on autotest. Naturally, the purpose is to make KVM stable, find/fix bugs faster and prevent regressions. It is to serve developers, to test if their new code breaks anything, as well as users, to verify that the KVM they are using is stable in their own environment.
There are many things to improve. Currently supported are only a few tests
(mainly reboot/migration) and a single guest (Fedora 8 i386). This will change soon as we add tests and guests to the test matrix.
We've tried to be very flexible with configuration of tests, so there are
quite a few options to choose from (check the documentation). For example one may choose to download and install the latest (or a specific) KVM release, the latest (or a specific) daily snapshot, use git, or use other source. Almost all configuration is located in kvm.cfg (under client directory).
By default kvm-autotest creates a machine-local subnet with a
local dhcp server. The dhcp server is configured to ignore unknown clients and only serves KVM guests. One may disable it by changing kvm.cfg (See documentation)
The test framework is based on autotest ( http://test.kernel.org/autotest ).
Currently we only using client tests, later we may want to use server tests for more complicated tests.
To download the source:
cd /usr/local ( or another root-writeable directory )
git clone git://git.kernel.org/pub/scm/virt/kvm/kvm-autotest.git
cd kvm-autotest/client
To read documentation look at http://kvm.qumranet.com/kvmwiki/KVM_RegressionTest
The simplest way to use it is:
0. login as root (not necessary but simpler, as tests must be run as root)
1. Download kvm-autotest (see a few lines above)
2. Change the following in kvm.cfg according to your preference (or leave the default settings):
local_kvm_dir -- "base" directory
local_guest_dir -- "images" directroy (may take a lot of disk space)
local_iso_dir -- "isos" directroy (may take a lot of disk space)
bridge -- change if your local network conflicts with
192.168.1/24 subnet
change bridge name if conflicts with a currently used
bridge.
dhcp_net -- change accordingly (e.g. use 10.0.1.2)
3. Make sure you have enough disk space for isos and images (df -h /var/local)
4. The default guest-installation test requires Fedora-8-i386-DVD.iso.
It is configured to fetch it automatically from the web unless it already
exists in the local_iso_dir (default path is /var/local/kvm_regression/isos/linux/Fedora-8-i386-DVD.iso).
If you already have that iso image locally, create a link or change
local_iso_dir.
Alternatively, in tests/kvm_guest_install/Fedora-8-i386/Fedora-8-i386.ini
set iso=/path/to/Fedora-8-i386-DVD.iso.
5. Default machine-local subnet is 192.168.1/24. If it conflicts with your network, you need change network configuration in kvm.cfg
6. Run the tests as root (./bin/autotest tests/kvm_runtest/control)
7. Checkout the results at results/default/
Caveats/Issues:
- Upon failure check results/defualt/debug/error-*
- After first download of KVM sources (and building + loading the modules), one needs to either manually remove /var/local/kvm_regression/{src,build} or set kvm_install=no in kvm.cfg. Otherwise the test will fail. The reason for it is that we do not want to remove directories automatically.
- The message "Guest installation completed but guest is not responsive", usually means that the guest installation failed.
- The tests must be run as root.
- When a guest is installing, do not move its mouse or type its keyboard. It confuses the guest installation algorithm (when watching its progress, e.g. with vncviewer).
- Guest installation is currently very sensitive to timing. If installation fails, sometimes it's helpfull to enlarge sleep time before the reboot and between steps. Do it by modifying tests/kvm_guest_install/Fedora-8-i386/Fedora-8-i386.steps
- After the guest is installed and you want to rerun the other tests set guest_install=no in kvm.cfg
Also if you want to test an existing image, create a new vm-class (guest configuration) and set guest_install=no in kvm.cfg
We hope many will test KVM on their systems and report results to the
community. We plan to have a web page with results. We hope many will add tests and improve KVM regression tests.
Thanks,
Uri.
Quoted 18 lines · Uri Lublin
Hello,
We are happy to announce the availability of kvm-autotest, a test
framework for KVM based on autotest. Naturally, the purpose is to make KVM stable, find/fix bugs faster and prevent regressions. It is to serve developers, to test if their new code breaks anything, as well as users, to verify that the KVM they are using is stable in their own environment.There are many things to improve. Currently supported are only a few tests
(mainly reboot/migration) and a single guest (Fedora 8 i386). This will change soon as we add tests and guests to the test matrix.We've tried to be very flexible with configuration of tests, so there are
quite a few options to choose from (check the documentation). For example one may choose to download and install the latest (or a specific) KVM release, the latest (or a specific) daily snapshot, use git, or use other source. Almost all configuration is located in kvm.cfg (under client directory).
Quoted 7 lines
By default kvm-autotest creates a machine-local subnet with a
local dhcp server. The dhcp server is configured to ignore unknown clients and only serves KVM guests. One may disable it by changing kvm.cfg (See documentation)The test framework is based on autotest ( http://test.kernel.org/autotest ).
Currently we only using client tests, later we may want to use server tests for more complicated tests.
This is looking great. Easy to use and fast. A few comments:
- As you mention, it should reuse the server/client model for running tests inside guests. I hacked up a "kvm_autotest" test that basically does:
tests = ["linus_stress", "bash_shared_mapping", "rmaptest", "tsc", "scrashme", "isic", "sleeptest", "libhugetlbfs", "..."]
vm.ssh.scp_to_remote(autotest_tarball, '/root')
(s,o) = vm.ssh.ssh('tar zvxf kvm-autotest.tar.gz')
for i in range(0, len(tests)):
(s,o) = vm.ssh.ssh('kvm-autotest/client/bin/autotest ' +
'kvm-autotest/client/tests/' + tests[i] + '/control')
print(o)
Which poorly replicates what the client/server infrastructure already provides. IMO its a waste of time to write specialized client tests (other than virt specific ones).
- On top of the client tests, we want to be able to force "special stress conditions", such as:
- host memory pressure (to test mmu shrink and mmu notifiers)
- CPU starvation (lower the priority of the guest process with higher prio CPU hungry tasks running on the host).
- synthetic cpu migration stress. ChrisW wrote this script sometime ago to test clock stability:
PID=$1
while :
do
for x in 0x01 0x02 0x04 0x08 0x10 0x20 0x40 0x80
do
taskset -p $x $PID > /dev/null 2>&1
usleep 5
done
done
- migration (as it stands now there is no transparent way to schedule client tests to run _while_ migration is in progress, one needs to manually change the migration test for that).
- Currently its difficult to debug client test failure inside guests, since the VM and its image are destroyed. Perhaps the client/server model handles error handling/reporting much more nicely.
Great work!
Quoted 23 lines · Marcelo Tosatti
Quoted 4 lines · Uri Lublin
The test framework is based on autotest ( http://test.kernel.org/autotest ).
Currently we only using client tests, later we may want to use server tests for more complicated tests.This is looking great. Easy to use and fast. A few comments:
- As you mention, it should reuse the server/client model for running tests inside guests. I hacked up a "
kvm_autotest" test that basically does:tests = ["
linus_stress", "bash_shared_mapping", "rmaptest", "tsc", "scrashme", "isic", "sleeptest", "libhugetlbfs", "..."]
vm.ssh.scp_to_remote(autotest_tarball, '/root')
(s,o) =vm.ssh.ssh('tar zvxf kvm-autotest.tar.gz')
for i inrange(0, len(tests)):
(s,o) = vm.ssh.ssh('kvm-autotest/client/bin/autotest ' +
'kvm-autotest/client/tests/' + tests[i] + '/control')
print(o)
Quoted 4 lines
Which poorly replicates what the client/server infrastructure already provides. IMO its a waste of time to write specialized client tests (other than virt specific ones).
You see guests as clients and the host as the server.
We were thinking of the host as a client and multi-host operations to be done by a server. guest-operations would be done using ssh (for linux guests) as your example above. You make a good point that we can use server/client infrastructure for guest operations. As it is simpler to write autotest client tests, and we thought most of the tests would be run as client tests, we want to postpone the server tests and focus on adding tests and guests to the matrix.
Quoted 3 lines
- On top of the client tests, we want to be able to force "special stress conditions", such as:
- host memory pressure (to test mmu shrink and mmu notifiers)
Similar is how many guests (or what to memory over-subscription percentage) we can start while all of them are responsive.
Quoted 5 lines
- CPU starvation (lower the priority of the guest process with higher prio CPU hungry tasks running on the host).
- synthetic cpu migration stress. ChrisW wrote this script sometime ago to test clock stability:
Quoted 12 lines
PID=$1
while :
do
for x in0x010x020x040x080x100x200x400x80
do
taskset -p $x $PID >/dev/null2>&1
usleep 5
done
done
- migration (as it stands now there is no transparent way to schedule client tests to run _while_ migration is in progress, one needs to manually change the migration test for that).
You probably do not need to change the migration test. I think we need to run some load on the guest (we'd probably have many load options, and the user will choose/configure which one to use) before migration starts and keep it running during the migration process.
Quoted 4 lines
- Currently its difficult to debug client test failure inside guests, since the VM and its image are destroyed. Perhaps the client/server model handles error handling/reporting much more nicely.
Agreed, we thought of that, but it's not cooked yet. Currently we always cleanup. We should not remove the (temporary) image if the test fails. Should we keep the guest running upon failure ? Currently we continue to test the next guest. We should probably have a configuration flag for that too.
Thanks,
Uri.
Quoted 31 lines · Uri Lublin
Quoted 25 lines · Marcelo Tosatti
Quoted 6 lines · Uri Lublin
The test framework is based on autotest ( http://test.kernel.org/autotest ).Currently we only using client tests, later we may want to use server tests
for more complicated tests.This is looking great. Easy to use and fast. A few comments:
- As you mention, it should reuse the server/client model for running tests inside guests. I hacked up a "
kvm_autotest" test that basically does:tests = ["
linus_stress", "bash_shared_mapping", "rmaptest", "tsc", "scrashme", "isic", "sleeptest", "libhugetlbfs", "..."]
vm.ssh.scp_to_remote(autotest_tarball, '/root')
(s,o) =vm.ssh.ssh('tar zvxf kvm-autotest.tar.gz')for i inrange(0, len(tests)):
(s,o) = vm.ssh.ssh('kvm-autotest/client/bin/autotest ' +
'kvm-autotest/client/tests/' + tests[i] + '/control')
print(o)Quoted 4 lines
Which poorly replicates what the client/server infrastructure already provides. IMO its a waste of time to write specialized client tests (other than virt specific ones).
Quoted 8 lines
You see guests as clients and the host as the server.
We were thinking of the host as a client and multi-host operations to be done by a server. guest-operations would be done using ssh (for linux guests) as your example above. You make a good point that we can use server/client infrastructure for guest operations. As it is simpler to write autotest client tests, and we thought most of the tests would be run as client tests, we want to postpone the server tests and focus on adding tests and guests to the matrix.
It's definitely worth looking at the autotest server code/samples. There exists code in-tree already to build an deploy kvm via autotest server mode which a single machine can drive the building, installing, creation of guests on N number of clients, directing each guest image to run various autotest client tests, collecting all of the results.
See autotest/server/samples/*kvm*
A proper server setup is a little involved[1] but much more streamlined these days.
Quoted 10 lines
You probably do not need to change the migration test. I think we need to run some load on the guest (we'd probably have many load options, and the user will choose/configure which one to use) before migration starts and keep it running during the migration process.
Quoted 4 lines
- Currently its difficult to debug client test failure inside guests, since the VM and its image are destroyed. Perhaps the client/server model handles error handling/reporting much more nicely.
Quoted 5 lines
Agreed, we thought of that, but it's not cooked yet. Currently we always cleanup. We should not remove the (temporary) image if the test fails. Should we keep the guest running upon failure ? Currently we continue to test the next guest. We should probably have a configuration flag for that too.
I suppose it depends on the failure, if we're capturing the boot log then for boot failures, I don't see any reason to keep running. If we succeed in booting but fail at some further point, then I think it makes sense to keep them running.
Signature · 5 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 [email protected]
Quoted 33 lines · Ryan Harper
Quoted 18 lines · Uri Lublin
Quoted 17 lines · Marcelo Tosatti
On Sun, Jul 06, 2008 at 01:16:13PM +0300, Uri Lublin wrote:
- As you mention, it should reuse the server/client model for running tests inside guests. I hacked up a "
kvm_autotest" test that basically does:tests = ["
linus_stress", "bash_shared_mapping", "rmaptest", "tsc", "scrashme", "isic", "sleeptest", "libhugetlbfs", "..."]
vm.ssh.scp_to_remote(autotest_tarball, '/root')
(s,o) =vm.ssh.ssh('tar zvxf kvm-autotest.tar.gz')for i inrange(0, len(tests)):
(s,o) = vm.ssh.ssh('kvm-autotest/client/bin/autotest ' +
'kvm-autotest/client/tests/' + tests[i] + '/control')
print(o)Quoted 6 lines
Quoted 4 lines
Which poorly replicates what the client/server infrastructure already provides. IMO its a waste of time to write specialized client tests (other than virt specific ones).
You see guests as clients and the host as the server.
We were thinking of the host as a client and multi-host operations to be done by a server. guest-operations would be done using ssh (for linux guests) as your example above. You make a good point that we can use server/client infrastructure for guest operations. As it is simpler to write autotest client tests, and we thought most of the tests would be run as client tests, we want to postpone the server tests and focus on adding tests and guests to the matrix.It's definitely worth looking at the autotest server code/samples. There exists code in-tree already to build an deploy kvm via autotest server mode which a single machine can drive the building, installing, creation of guests on N number of clients, directing each guest image to run various autotest client tests, collecting all of the results.
Quoted 5 lines
See autotest/server/samples/*kvm*
A proper server setup is a little involved[1] but much more streamlined these days.
Let's think of a guest-installation test. Would you implement it on the server or on the client ? What do you plan for non-linux guests ?
We'll try this little exercise of writing a kvm-test on the server side and on the client side and compare complexity.
Thanks,
Uri.
Quoted 15 lines
Quoted 9 lines
It's definitely worth looking at the autotest server code/samples. There exists code in-tree already to build an deploy kvm via autotest server mode which a single machine can drive the building, installing, creation of guests on N number of clients, directing each guest image to run various autotest client tests, collecting all of the results.
Quoted 5 lines
See autotest/server/samples/*kvm*
A proper server setup is a little involved[1] but much more streamlined these days.
Quoted 7 lines
Let's think of a guest-installation test. Would you implement it on the server or on the client ? What do you plan for non-linux guests ?
We'll try this little exercise of writing a kvm-test on the server side and on the client side and compare complexity.
Thanks,
Uri.
IMHO we need a mixture:
- kvm/environment setup
autoserve tests/deploy
- Internal guest tests
Implemented as client test, executed from the server. Composed of
benchmarks, standard functionality, applications, unit tests, etc.
- guest installation, guest boot
client test that execute on the kvm host
Regards,
Dor
Quoted 42 lines · Uri Lublin
Quoted 41 lines · Ryan Harper
Quoted 22 lines · Uri Lublin
Quoted 17 lines · Marcelo Tosatti
On Sun, Jul 06, 2008 at 01:16:13PM +0300, Uri Lublin wrote:
- As you mention, it should reuse the server/client model for running tests inside guests. I hacked up a "
kvm_autotest" test that basically does:tests = ["
linus_stress", "bash_shared_mapping", "rmaptest", "tsc", "scrashme", "isic", "sleeptest", "libhugetlbfs", "..."]
vm.ssh.scp_to_remote(autotest_tarball, '/root')(s,o) =vm.ssh.ssh('tar zvxf kvm-autotest.tar.gz')for i inrange(0, len(tests)):
(s,o) = vm.ssh.ssh('kvm-autotest/client/bin/autotest ' +
'kvm-autotest/client/tests/' + tests[i] + '/control')
print(o)Quoted 4 lines
Which poorly replicates what the client/server infrastructure already provides. IMO its a waste of time to write specialized client tests (other than virt specific ones).
Quoted 8 lines
You see guests as clients and the host as the server.
We were thinking of the host as a client and multi-host operations to be done by a server. guest-operations would be done using ssh (for linux guests) as your example above. You make a good point that we can use server/client infrastructure for guest operations. As it is simpler to write autotest client tests, and we thought most of the tests would be run as client tests, we want to postpone the server tests and focus on adding tests and guests to the matrix.It's definitely worth looking at the autotest server code/samples. There exists code in-tree already to build an deploy kvm via autotest server mode which a single machine can drive the building, installing, creation of guests on N number of clients, directing each guest image to run various autotest client tests, collecting all of the results.
See autotest/server/samples/*kvm*
Quoted 7 lines
Quoted 3 lines
A proper server setup is a little involved[1] but much more streamlined these days.
Let's think of a guest-installation test. Would you implement it on the server or on the client ?
What do you plan for non-linux guests ?
Client side, for installation, we already have a solution that works for all types of guests:
http://kvm.qumranet.com/kvmwiki/KVMTest
which is already integrated as a client test in autotest. Once you record your installation via kvmtest, then it is just matter for keeping the iso and an empty disk image around and replaying the installation with -snapshot.
Now, I'm actually more interested in doing the following:
use kvmtest to replay an installation of a guest and instead of throwing the guest away once (running with -snapshot) it has passes the install, it is now ready to be used to execute autotest client tests or something else.
Quoted 3 lines
We'll try this little exercise of writing a kvm-test on the server side and on the client side and compare complexity.
That's a bit vague, what sort of test are you talking about? If you mean installation, i'm not interested since that's been handled by KVMTest. As to complexity, I urge you to look at the existing kvm examples[2] in the autotest server dir, those look pretty darn simple to me and already include all of the infrastructure for capturing console logs, results and errors.
Oh, I forgot my pointer to the server setup last time:
1. http://test.kernel.org/autotest/AutotestServerInstall
2. autotest/server/samples/kvm.srv
Signature · 5 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 [email protected]
Quoted 45 lines · Ryan Harper
Quoted 33 lines · Uri Lublin
Quoted 32 lines · Ryan Harper
Quoted 18 lines · Uri Lublin
Quoted 17 lines · Marcelo Tosatti
On Sun, Jul 06, 2008 at 01:16:13PM +0300, Uri Lublin wrote:
- As you mention, it should reuse the server/client model for running tests inside guests. I hacked up a "
kvm_autotest" test that basically does:tests = ["
linus_stress", "bash_shared_mapping", "rmaptest", "tsc", "scrashme", "isic", "sleeptest", "libhugetlbfs", "..."]
vm.ssh.scp_to_remote(autotest_tarball, '/root')(s,o) =vm.ssh.ssh('tar zvxf kvm-autotest.tar.gz')for i inrange(0, len(tests)):
(s,o) = vm.ssh.ssh('kvm-autotest/client/bin/autotest ' +
'kvm-autotest/client/tests/' + tests[i] + '/control')
print(o)Quoted 6 lines
Quoted 4 lines
Which poorly replicates what the client/server infrastructure already provides. IMO its a waste of time to write specialized client tests (other than virt specific ones).
You see guests as clients and the host as the server.
We were thinking of the host as a client and multi-host operations to be done by a server. guest-operations would be done using ssh (for linux guests) as your example above. You make a good point that we can use server/client infrastructure for guest operations. As it is simpler to write autotest client tests, and we thought most of the tests would be run as client tests, we want to postpone the server tests and focus on adding tests and guests to the matrix.It's definitely worth looking at the autotest server code/samples. There exists code in-tree already to build an deploy kvm via autotest server mode which a single machine can drive the building, installing, creation of guests on N number of clients, directing each guest image to run various autotest client tests, collecting all of the results.
Quoted 7 lines
Quoted 5 lines
See autotest/server/samples/*kvm*
A proper server setup is a little involved[1] but much more streamlined these days.
Let's think of a guest-installation test. Would you implement it on the server or on the client ? What do you plan for non-linux guests ?
Client side, for installation, we already have a solution that works for all types of guests:
Quoted 8 lines
http://kvm.qumranet.com/kvmwiki/KVMTest
which is already integrated as a client test in autotest. Once you record your installation via kvmtest, then it is just matter for keeping the iso and an empty disk image around and replaying the installation with -snapshot.
So guest installation is a client test. KVMtest has its own way of managing/booting/communicating-with guests, and naturally does not use KVM/KVMGuest classes of autotest server.
How about the test, suggested by Marcelo/Chris, of changing physical cpu of a VM using taskset. Would that be a client test or a server test ?
What about stop/cont, save/restore ?
I think most kvm-tests will be client tests.
More complex tests, which involve multiple hosts, such as migration between two hosts, should be server tests.
Managing all those tests, can be done by autoserv.
Quoted 8 lines
Now, I'm actually more interested in doing the following:
use kvmtest to replay an installation of a guest and instead of throwing the guest away once (running with -snapshot) it has passes the install, it is now ready to be used to execute autotest client tests or something else.
autotest client tests can be consider as "guest load" which usually does is orthogonal to the "real" kvm-test that is running (e.g. migration-test while watching a movie, or migration-test while building the kernel). Also what would you do for non-linux guest ?
Quoted 6 lines
Quoted 2 lines
We'll try this little exercise of writing a kvm-test on the server side and on the client side and compare complexity.
That's a bit vague, what sort of test are you talking about? If you mean installation, i'm not interested since that's been handled by KVMTest.
Are you actually running autotest tests with KVMTest installed guests? Do you have to manually exchange ssh-keys.
Quoted 4 lines
As to complexity, I urge you to look at the existing kvm examples[2] in the autotest server dir, those look pretty darn simple to me and already include all of the infrastructure for capturing console logs, results and errors.
It does look simple. It was written for a different purpose though, which is to run autotest tests on guests, not to run kvm tests.
Quoted 8 lines
Oh, I forgot my pointer to the server setup last time:
1. http://test.kernel.org/autotest/AutotestServerInstall
2. autotest/server/samples/kvm.srv
So what do you propose ?
We've were thinking today how we can move things to the server and want to get your (or anyone's) opinion. Does the server always starts (boots) guests ? Do most tests run on the server-side (similar to [2]) or may client tests start guests (similar to KVMTest and migration)? If both how would you synchronize between them. If some tests run on the client side and only the server starts guests, how do those tests access the guest. Do you assume the guest images always exist before running the tests or do you create them automatically if needed/requested.
Thanks,
Uri
Quoted 12 lines · Uri Lublin
Quoted 11 lines
Client side, for installation, we already have a solution that works for all types of guests:
http://kvm.qumranet.com/kvmwiki/KVMTest
which is already integrated as a client test in autotest. Once you record your installation via kvmtest, then it is just matter for keeping the iso and an empty disk image around and replaying the installation with -snapshot.
Quoted 3 lines
So guest installation is a client test. KVMtest has its own way of managing/booting/communicating-with guests, and naturally does not use KVM/KVMGuest classes of autotest server.
It does not use the KVM/KVMGuest classes in autotest server precisely because it is a client test, there is no need to do anything server-side w.r.t KVMTest. We ensure that we've built and installed whatever version of kvm on the target machine and that the target machine has access to require inputs (iso,disk image) and invoke KVMTest.
Quoted 7 lines
How about the test, suggested by Marcelo/Chris, of changing physical cpu of a VM
using taskset. Would that be a client test or a server test ? What about stop/cont, save/restore ?I think most kvm-tests will be client tests.
Agreed, the above examples will be client tests. Autotest client can do parallel execution, or even step-wise. Between those, we should be able to ensure we get proper coverage for the envisioned scenarios.
Quoted 5 lines
More complex tests, which involve multiple hosts, such as migration between two hosts, should be server tests.
Managing all those tests, can be done by autoserv.
Agreed.
Quoted 5 lines
Quoted 5 lines
Now, I'm actually more interested in doing the following:
Quoted 7 lines
Quoted 4 lines
use kvmtest to replay an installation of a guest and instead of throwing the guest away once (running with -snapshot) it has passes the install, it is now ready to be used to execute autotest client tests or something else.
autotest client tests can be consider as "guest load" which usually does is orthogonal to the "real" kvm-test that is running (e.g. migration-test
Definitely. I just want to utilize autotest to drive both guest creation and test orchestration which includes installation as well as testing known working guests in various scenarios and yes autotest simplifies generating "guest load".
Quoted 2 lines
while watching a movie, or migration-test while building the kernel). Also what would you do for non-linux guest ?
I'm not quite sure what to do here since non-linux guests isn't really in my scope beyond simple tests (installation, shutdown, reboot, configuration variance).
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines
We'll try this little exercise of writing a kvm-test on the server side and on the client side and compare complexity.
That's a bit vague, what sort of test are you talking about? If you mean installation, i'm not interested since that's been handled by KVMTest.
Are you actually running autotest tests with KVMTest installed guests? Do
Yes, but the transition between using KVMtest to install aguest and running autotest inside isn't completely automated -- yet.
Quoted 1 line
you have to manually exchange ssh-keys.
Yeah, we'll need a solution, but it should be pretty simple to automate, Mostly likely pregenerate a key and serve it up to the guest via cdrom image and then install that into the guest.
Quoted 1 line
Quoted 1 line
As to complexity, I urge you to look at the existing kvm
u
Quoted 6 lines
Quoted 3 lines
examples[2] in the autotest server dir, those look pretty darn simple to me and already include all of the infrastructure for capturing console logs, results and errors.
It does look simple. It was written for a different purpose though, which is to run autotest tests on guests, not to run kvm tests.
Sure, but that doesn't mean it isn't a good infrastructure on top of which we can build kvm testing.
Quoted 11 lines
Quoted 9 lines
Oh, I forgot my pointer to the server setup last time:
1. http://test.kernel.org/autotest/AutotestServerInstall
2. autotest/server/samples/kvm.srv
Quoted 4 lines
So what do you propose ?
We've were thinking today how we can move things to the server and want to get your (or anyone's) opinion. Does the server always starts (boots) guests ? Do most tests run on the server-side (similar to [2]) or may
Yeah, let me explain a bit more about how the server and clients work:
You'll have one master server which runs the job monitor, it will look for autotest server files (.srv) and that file will run on the server, but you will write your srv file to execute on a set of machines that your master server manages. Autotest server maintains a db of machines.
Looking at autotest/server/samples/sleeptest.srv:
def run(machine):
host = hosts.SSHHost(machine)
at = autotest.Autotest(host)
at.run_test('sleeptest')
job.parallel_simple(run, machines)
We're defining a run function, and then running that across all the machines in the grid.
The other interesting one to look at is netperf-guest-to-host-far.srv.
That file demonstrates installing kvm to differnt host machines (not the server where the .srv file is running); on each host machine, create a KVM guest, and then on each host, creating a KVM guest, in each guest, running a netperf test between each guest.
Quoted 2 lines
client tests start guests (similar to KVMTest and migration)? If both how would you synchronize between them. If some tests run on the client side
Synchronization is done via barriers, see autotest/client/tests/netperf2/netperf2.py and autotest/client/common_lib/barrier.py for more details.
Quoted 3 lines
and only the server starts guests, how do those tests access the guest. Do you assume the guest images always exist before running the tests or do you create them automatically if needed/requested.
For multi-machine setup, either we'd need to create the image as needed, or you'll want some sort of shared storage mounted on each host that you want to run kvm tests upon.
And obviously, shared storage is need as soon as we put migration into the mix.
Signature · 5 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 [email protected]
Quoted 8 lines · Ryan Harper
Quoted 2 lines · Uri Lublin
I think most kvm-tests will be client tests.
Agreed, the above examples will be client tests. Autotest client can do parallel execution, or even step-wise. Between those, we should be able to ensure we get proper coverage for the envisioned scenarios.
Quoted 7 lines
Quoted 4 lines
More complex tests, which involve multiple hosts, such as migration between two hosts, should be server tests.
Managing all those tests, can be done by autoserv.
Agreed.
We all agree that most kvm tests are client-side tests. The server manages multiple hosts and complex tests (such as multi-host migration). The server also may run autotest tests on linux guests. We consider kvm-install, kvm-boot, kvm-guest-install as client tests.
Following your comments our plan is to move the control to the server size:
The server executes a .srv file (or a server side control). It calls client-tests (including simple building blocks such as kvm-installation, boot/destroy a KVM-VM, etc), providing guest id and some specific tests parameters. Guests are always created on the client (similar to current kvm-autotest behaviour) and a client-side class that knows how to map guest-id with a running guest (including guest-ip) will be used by all kvm client tests.
Accessing a linux guest (e.g. running autotest tests) can be done from the server simply by using hosts.SSHHost just like any other linux host. This will only work for global dhcp (no local host-only bridge) or if server and host are the same.
Client tests can also be started from the client side using client-side control files.
We do not know yet what we'll do with non-linux guests, but we sure want to test them too.
Thanks,
Uri