Skip to contentSkip to conversation

Patchset of network related subtests

RFC PATCH Superseded Amos Kong 4 participants 44 messages Jul 20, 2010 – Aug 11, 2010 KVM

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Amos Kong

author
#0 lore raw

cover Patchset of network related subtests

The following series contain 11 network related subtests, welcome to give me some suggestions about correctness, design, enhancement.

Thank you so much!

---

Amos Kong (14):
KVM-test: Add a new macaddress pool algorithm
KVM Test: Add a function get_interface_name() to kvm_net_utils.py KVM Test: Add a common ping module for network related tests
KVM-test: Add a new subtest ping
KVM-test: Add a subtest jumbo
KVM-test: Add basic file transfer test
KVM-test: Add a subtest of load/unload nic driver
KVM-test: Add a subtest of nic promisc
KVM-test: Add a subtest of multicast
KVM-test: Add a subtest of pxe
KVM-test: Add a subtest of changing mac address
KVM-test: Add a subtest of netperf
KVM-test: Improve vlan subtest
KVM-test: Add subtest of testing offload by ethtool

 0 files changed, 0 insertions(+), 0 deletions(-)
Signature · 1 line
Amos Kong
author
#1 lore raw

RFC PATCH 1/14 KVM-test: Add a new macaddress pool algorithm

Old method uses the mac address in the configuration files which could lead serious problem when multiple tests running in different hosts.

This patch adds a new macaddress pool algorithm, it generates the mac prefix based on mac address of the host which could eliminate the duplicated mac addresses between machines.

When user have set the mac_prefix in the configuration file, we should use it in stead of the dynamic generated mac prefix.

Other change:
. Fix randomly generating mac address so that it correspond to IEEE802. . Update clone function to decide clone mac address or not. . Update get_macaddr function.
. Add set_mac_address function.

New auto mac address pool algorithm:
If address_index is defined, VM will get mac from config file then record mac in to address_pool. If address_index is not defined, VM will call get_mac_from_pool to auto create mac then recored mac to address_pool in following format:
{'macpool': {'AE:9D:94:6A:9b:f9': ['20100310-165222-Wt7l:0']}}

AE:9D:94:6A:9b:f9 : mac address
20100310-165222-Wt7l : instance attribute of VM
0 : index of NIC

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/kvm_utils.py +99 -0
index fb2d1c2..7c0946e 100644
@@ -5,6 +5,7 @@ KVM test utility functions.
 """
 
 import time, string, random, socket, os, signal, re, logging, commands, cPickle
+import fcntl, shelve
 from autotest_lib.client.bin import utils
 from autotest_lib.client.common_lib import error, logging_config
 import kvm_subprocess
@@ -82,6 +83,104 @@ def get_sub_dict_names(dict, keyword):
 
 # Functions related to MAC/IP addresses
 
+def get_mac_from_pool(root_dir, vm, nic_index, prefix='00:11:22:33:'):
+    """
+    random generated mac address.
+
+    1) First try to generate macaddress based on the mac address prefix.
+    2) And then try to use total random generated mac address.
+
+    @param root_dir: Root dir for kvm
+    @param vm: Here we use instance of vm
+    @param nic_index: The index of nic.
+    @param prefix: Prefix of mac address.
+    @Return: Return mac address.
+    """
+
+    lock_filename = os.path.join(root_dir, "mac_lock")
+    lock_file = open(lock_filename, 'w')
+    fcntl.lockf(lock_file.fileno() ,fcntl.LOCK_EX)
+    mac_filename = os.path.join(root_dir, "address_pool")
+    mac_shelve = shelve.open(mac_filename, writeback=False)
+
+    mac_pool = mac_shelve.get("macpool")
+
+    if not mac_pool:
+        mac_pool = {}
+    found = False
+
+    val = "%s:%s" % (vm, nic_index)
+    for key in mac_pool.keys():
+        if val in mac_pool[key]:
+            mac_pool[key].append(val)
+            found = True
+            mac = key
+
+    while not found:
+        postfix = "%02x:%02x" % (random.randint(0x00,0xfe),
+                                random.randint(0x00,0xfe))
+        mac = prefix + postfix
+        mac_list = mac.split(":")
+        # Clear multicast bit
+        mac_list[0] = int(mac_list[0],16) & 0xfe
+        # Set local assignment bit (IEEE802)
+        mac_list[0] = mac_list[0] | 0x02
+        mac_list[0] = "%02x" % mac_list[0]
+        mac = ":".join(mac_list)
+        if mac not in mac_pool.keys() or 0 == len(mac_pool[mac]):
+            mac_pool[mac] = ["%s:%s" % (vm,nic_index)]
+            found = True
+    mac_shelve["macpool"] = mac_pool
+    logging.debug("generating mac addr %s " % mac)
+
+    mac_shelve.close()
+    fcntl.lockf(lock_file.fileno(), fcntl.LOCK_UN)
+    lock_file.close()
+    return mac
+
+
+def put_mac_to_pool(root_dir, mac, vm):
+    """
+    Put the macaddress back to address pool
+
+    @param root_dir: Root dir for kvm
+    @param vm: Here we use instance attribute of vm
+    @param mac: mac address will be put.
+    @Return: mac address.
+    """
+
+    lock_filename = os.path.join(root_dir, "mac_lock")
+    lock_file = open(lock_filename, 'w')
+    fcntl.lockf(lock_file.fileno() ,fcntl.LOCK_EX)
+    mac_filename = os.path.join(root_dir, "address_pool")
+    mac_shelve = shelve.open(mac_filename, writeback=False)
+
+    mac_pool = mac_shelve.get("macpool")
+
+    if not mac_pool or (not mac in mac_pool):
+        logging.debug("Try to free a macaddress does no in pool")
+        logging.debug("macaddress is %s" % mac)
+        logging.debug("pool is %s" % mac_pool)
+    else:
+        if len(mac_pool[mac]) <= 1:
+            mac_pool.pop(mac)
+        else:
+            for value in mac_pool[mac]:
+                if vm in value:
+                    mac_pool[mac].remove(value)
+                    break
+            if len(mac_pool[mac]) == 0:
+                mac_pool.pop(mac)
+
+    mac_shelve["macpool"] = mac_pool
+    logging.debug("freeing mac addr %s " % mac)
+
+    mac_shelve.close()
+    fcntl.lockf(lock_file.fileno(), fcntl.LOCK_UN)
+    lock_file.close()
+    return mac
+
+
 def mac_str_to_int(addr):
     """
     Convert MAC address string to integer.
client/tests/kvm/kvm_vm.py +111 -7

206 lines not on this page · raw message

client/tests/kvm/tests_base.cfg.sample +1 -1
index fd9e72f..6710c00 100644
@@ -51,7 +51,7 @@ guest_port_remote_shell = 22
 nic_mode = user
 #nic_mode = tap
 nic_script = scripts/qemu-ifup
-address_index = 0
+#address_index = 0
 run_tcpdump = yes
 
 # Misc
#2 lore raw

Re: [Qemu-devel] [RFC PATCH 01/14] KVM-test: Add a new macaddress pool algorithm

Quoted 23 lines · Amos Kong

Old method uses the mac address in the configuration files which could lead serious problem when multiple tests running in different hosts.

This patch adds a new macaddress pool algorithm, it generates the mac prefix based on mac address of the host which could eliminate the duplicated mac addresses between machines.

When user have set the mac_prefix in the configuration file, we should use it in stead of the dynamic generated mac prefix.

Other change:
. Fix randomly generating mac address so that it correspond to IEEE802. . Update clone function to decide clone mac address or not. . Update get_macaddr function.
. Add set_mac_address function.

New auto mac address pool algorithm:
If address_index is defined, VM will get mac from config file then record mac in to address_pool. If address_index is not defined, VM will call get_mac_from_pool to auto create mac then recored mac to address_pool in following format:
{'macpool': {'AE:9D:94:6A:9b:f9': ['20100310-165222-Wt7l:0']}}

Quoted 3 lines

AE:9D:94:6A:9b:f9 : mac address
20100310-165222-Wt7l : instance attribute of VM
0 : index of NIC

Why do you use the mac address as a key, instead of the instance string
+ nic index? When the mac address is used as a key, each key has a list of values instead of just one value. This order seems unnatural. If it were the other way around (i.e. key = VM instance + nic index, value = mac address), then each key would have exactly one value, and I think this patch would be shorter and simpler.

Quoted 19 lines

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/kvm_utils.py +1 -0
index fb2d1c2..7c0946e 100644
@@ -5,6 +5,7 @@ KVM test utility functions.
 """
 
 import time, string, random, socket, os, signal, re, logging, commands, cPickle
+import fcntl, shelve
 from autotest_lib.client.bin import utils
 from autotest_lib.client.common_lib import error, logging_config
 import kvm_subprocess

client/tests/kvm/kvm_utils.py def get_sub_dict_names(dict, keyword): · line 86

Quoted 4 lines
client/tests/kvm/kvm_utils.py +1 -0
@@ -82,6 +83,104 @@ def get_sub_dict_names(dict, keyword):
 
 # Functions related to MAC/IP addresses
 
+def get_mac_from_pool(root_dir, vm, nic_index, prefix='00:11:22:33:'):

The name of this function is confusing because it does the exact
opposite: it puts a mac address in address_pool. Maybe the pool you're referring to in the name isn't address_pool, but still a less confusing name should probably be used.

Quoted 7 lines
client/tests/kvm/kvm_utils.py +7 -0
+    """
+    random generated mac address.
+
+    1) First try to generate macaddress based on the mac address prefix.
+    2) And then try to use total random generated mac address.
+
+    @param root_dir: Root dir for kvm

client/tests/kvm/kvm_utils.py def get_sub_dict_names(dict, keyword): · line 103

Quoted 10 lines
client/tests/kvm/kvm_utils.py +10 -0
+    @param vm: Here we use instance of vm
+    @param nic_index: The index of nic.
+    @param prefix: Prefix of mac address.
+    @Return: Return mac address.
+    """
+
+    lock_filename = os.path.join(root_dir, "mac_lock")
+    lock_file = open(lock_filename, 'w')
+    fcntl.lockf(lock_file.fileno() ,fcntl.LOCK_EX)
+    mac_filename = os.path.join(root_dir, "address_pool")

Maybe it makes sense to put address_pool and the lock file in /tmp, where they can be shared by more than a single autotest instance running on the same host (unlikely, but theoretically possible).

client/tests/kvm/kvm_utils.py def get_sub_dict_names(dict, keyword): · line 106

Quoted 3 lines
client/tests/kvm/kvm_utils.py +3 -0
+    mac_shelve = shelve.open(mac_filename, writeback=False)
+
+    mac_pool = mac_shelve.get("macpool")

Why is this 'macpool' needed? Why not put the keys directly in the shelve object?

client/tests/kvm/kvm_utils.py def get_sub_dict_names(dict, keyword): · line 115

Quoted 9 lines
client/tests/kvm/kvm_utils.py +9 -0
+
+    if not mac_pool:
+        mac_pool = {}
+    found = False
+
+    val = "%s:%s" % (vm, nic_index)
+    for key in mac_pool.keys():
+        if val in mac_pool[key]:
+            mac_pool[key].append(val)

Why append val to mac_pool[key] if val is already in mac_pool[key]?

client/tests/kvm/kvm_utils.py def get_sub_dict_names(dict, keyword): · line 128

Quoted 13 lines
client/tests/kvm/kvm_utils.py +13 -0
+            found = True
+            mac = key
+
+    while not found:
+        postfix = "%02x:%02x" % (random.randint(0x00,0xfe),
+                                random.randint(0x00,0xfe))
+        mac = prefix + postfix
+        mac_list = mac.split(":")
+        # Clear multicast bit
+        mac_list[0] = int(mac_list[0],16) & 0xfe
+        # Set local assignment bit (IEEE802)
+        mac_list[0] = mac_list[0] | 0x02
+        mac_list[0] = "%02x" % mac_list[0]

Why is this needed? Most mac addresses begin with 00. If the mac address is generated from the address of eth0 (using the method in this patch) it begins with 00, which is fine. If the prefix is set by the user using mac_prefix, I don't think we should modify it.

Quoted 4 lines
client/tests/kvm/kvm_utils.py +4 -0
+        mac = ":".join(mac_list)
+        if mac not in mac_pool.keys() or 0 == len(mac_pool[mac]):
+            mac_pool[mac] = ["%s:%s" % (vm,nic_index)]
+            found = True

client/tests/kvm/kvm_utils.py def get_sub_dict_names(dict, keyword): · line 142

Quoted 10 lines
client/tests/kvm/kvm_utils.py +10 -0
+    mac_shelve["macpool"] = mac_pool
+    logging.debug("generating mac addr %s " % mac)
+
+    mac_shelve.close()
+    fcntl.lockf(lock_file.fileno(), fcntl.LOCK_UN)
+    lock_file.close()
+    return mac
+
+
+def put_mac_to_pool(root_dir, mac, vm):

This function removes a mac address from address_pool. I wonder why you chose this name.

Quoted 91 lines
client/tests/kvm/kvm_utils.py +41 -0
+    """
+    Put the macaddress back to address pool
+
+    @param root_dir: Root dir for kvm
+    @param vm: Here we use instance attribute of vm
+    @param mac: mac address will be put.
+    @Return: mac address.
+    """
+
+    lock_filename = os.path.join(root_dir, "mac_lock")
+    lock_file = open(lock_filename, 'w')
+    fcntl.lockf(lock_file.fileno() ,fcntl.LOCK_EX)
+    mac_filename = os.path.join(root_dir, "address_pool")
+    mac_shelve = shelve.open(mac_filename, writeback=False)
+
+    mac_pool = mac_shelve.get("macpool")
+
+    if not mac_pool or (not mac in mac_pool):
+        logging.debug("Try to free a macaddress does no in pool")
+        logging.debug("macaddress is %s" % mac)
+        logging.debug("pool is %s" % mac_pool)
+    else:
+        if len(mac_pool[mac]) <= 1:
+            mac_pool.pop(mac)
+        else:
+            for value in mac_pool[mac]:
+                if vm in value:
+                    mac_pool[mac].remove(value)
+                    break
+            if len(mac_pool[mac]) == 0:
+                mac_pool.pop(mac)
+
+    mac_shelve["macpool"] = mac_pool
+    logging.debug("freeing mac addr %s " % mac)
+
+    mac_shelve.close()
+    fcntl.lockf(lock_file.fileno(), fcntl.LOCK_UN)
+    lock_file.close()
+    return mac
+
+
 def mac_str_to_int(addr):
     """
     Convert MAC address string to integer.
client/tests/kvm/kvm_vm.py +11 -2
index 6cd0688..a9ba6e7 100755
@@ -5,7 +5,7 @@ Utility classes and functions to handle Virtual Machine creation using qemu.
 @copyright: 2008-2009 Red Hat Inc.
 """
 
-import time, socket, os, logging, fcntl, re, commands, glob
+import time, socket, os, logging, fcntl, re, commands, shelve, glob
 import kvm_utils, kvm_subprocess, kvm_monitor, rss_file_transfer
 from autotest_lib.client.common_lib import error
 from autotest_lib.client.bin import utils
@@ -117,6 +117,7 @@ class VM:
         self.params = params
         self.root_dir = root_dir
         self.address_cache = address_cache
+        self.mac_prefix = params.get('mac_prefix')
         self.netdev_id = []
 
         # Find a unique identifier for this VM
@@ -126,8 +127,15 @@ class VM:
             if not glob.glob("/tmp/*%s" % self.instance):
                 break
 
+        if self.mac_prefix is None:
+            s, o = commands.getstatusoutput("ifconfig eth0")
+            if s == 0:
+                mac = re.findall("HWaddr (\S*)", o)[0]
+                self.mac_prefix = '00' + mac[8:] + ':'
+
 
-    def clone(self, name=None, params=None, root_dir=None, address_cache=None):
+    def clone(self, name=None, params=None, root_dir=None,
+                    address_cache=None, mac_clone=True):
         """
         Return a clone of the VM object with optionally modified parameters.
         The clone is initially not alive and needs to be started using create().
@@ -138,6 +146,7 @@ class VM:
         @param params: Optional new VM creation parameters
         @param root_dir: Optional new base directory for relative filenames
         @param address_cache: A dict that maps MAC addresses to IP addresses
+        @param mac_clone: Clone mac address or not.
         """
         if name is None:
             name = self.name

client/tests/kvm/kvm_vm.py class VM: · line 162

Quoted 8 lines
client/tests/kvm/kvm_vm.py +4 -1
@@ -147,9 +156,19 @@ class VM:
             root_dir = self.root_dir
         if address_cache is None:
             address_cache = self.address_cache
-        return VM(name, params, root_dir, address_cache)
+        vm = VM(name, params, root_dir, address_cache)
+        if mac_clone:
+            # Clone mac address by coping 'self.instance' to the new vm.
+            vm.instance = self.instance

Copying self.instance isn't a good idea because the monitor, serial console and testlog filenames use self.instance. self.instance should be unique. If we still want to use the same mac address for 2 VMs, for example in migration, a different solution should be found.

Quoted 27 lines
client/tests/kvm/kvm_vm.py +14 -0
+        return vm
 
 
+    def free_mac_address(self):
+        nic_num = len(kvm_utils.get_sub_dict_names(self.params, "nics"))
+        for nic in range(nic_num):
+            mac = self.get_macaddr(nic_index=nic)
+            kvm_utils.put_mac_to_pool(self.root_dir, mac, vm=self.instance)
+
     def make_qemu_command(self, name=None, params=None, root_dir=None):
         """
         Generate a qemu command line. All parameters are optional. If a
@@ -383,6 +402,13 @@ class VM:
             mac = None
             if "address_index" in nic_params:
                 mac = kvm_utils.get_mac_ip_pair_from_dict(nic_params)[0]
+                self.set_mac_address(mac=mac, nic_index=vlan)
+            else:
+                mac = kvm_utils.get_mac_from_pool(self.root_dir,
+                                                  vm=self.instance,
+                                                  nic_index=vlan,
+                                                  prefix=self.mac_prefix)
+
             qemu_cmd += add_nic(help, vlan, nic_params.get("nic_model"), mac,
                                 self.netdev_id[vlan])
             # Handle the '-net tap' or '-net user' part

client/tests/kvm/kvm_vm.py class VM: · line 425

Quoted 5 lines
client/tests/kvm/kvm_vm.py +1 -1
@@ -396,7 +422,7 @@ class VM:
             if tftp:
                 tftp = kvm_utils.get_path(root_dir, tftp)
             qemu_cmd += add_net(help, vlan, nic_params.get("nic_mode", "user"),
-                                nic_params.get("nic_ifname"),
+                                self.get_ifname(vlan),

Why is get_ifname() useful here?

Quoted 73 lines
client/tests/kvm/kvm_vm.py +21 -3
                                 script, downscript, tftp,
                                 nic_params.get("bootp"), redirs,
                                 self.netdev_id[vlan])
@@ -720,7 +746,7 @@ class VM:
             lockfile.close()
 
 
-    def destroy(self, gracefully=True):
+    def destroy(self, gracefully=True, free_macaddr=True):
         """
         Destroy the VM.
 
@@ -731,6 +757,7 @@ class VM:
         @param gracefully: Whether an attempt will be made to end the VM
                 using a shell command before trying to end the qemu process
                 with a 'quit' or a kill signal.
+        @param free_macaddr: Whether free macaddresses when destory a vm.
         """
         try:
             # Is it already dead?
@@ -751,11 +778,18 @@ class VM:
                         logging.debug("Shutdown command sent; waiting for VM "
                                       "to go down...")
                         if kvm_utils.wait_for(self.is_dead, 60, 1, 1):
-                            logging.debug("VM is down")
+                            logging.debug("VM is down, free mac address.")
+                            # free mac address
+                            if free_macaddr:
+                                self.free_mac_address()
                             return
                     finally:
                         session.close()
 
+            # free mac address
+            if free_macaddr:
+                self.free_mac_address()
+
             if self.monitor:
                 # Try to destroy with a monitor command
                 logging.debug("Trying to kill VM with monitor command...")
@@ -881,10 +915,14 @@ class VM:
         nic_name = nics[index]
         nic_params = kvm_utils.get_sub_dict(self.params, nic_name)
         if nic_params.get("nic_mode") == "tap":
-            mac, ip = kvm_utils.get_mac_ip_pair_from_dict(nic_params)
+            mac = self.get_macaddr(index)
             if not mac:
                 logging.debug("MAC address unavailable")
                 return None
+            else:
+                mac = mac.lower()
+            ip = None
+
             if not ip or nic_params.get("always_use_tcpdump") == "yes":
                 # Get the IP address from the cache
                 ip = self.address_cache.get(mac)
@@ -897,6 +935,7 @@ class VM:
                              for nic in nics]
                 macs = [kvm_utils.get_mac_ip_pair_from_dict(dict)[0]
                         for dict in nic_dicts]
+                macs.append(mac)
                 if not kvm_utils.verify_ip_address_ownership(ip, macs):
                     logging.debug("Could not verify MAC-IP address mapping: "
                                   "%s ---> %s" % (mac, ip))
@@ -925,6 +964,71 @@ class VM:
                              "redirected" % port)
             return self.redirs.get(port)
 
+    def get_ifname(self, nic_index=0):
+        """
+        Return the ifname of tap device for the guest nic.
+
+        @param nic_index: Index of the NIC

client/tests/kvm/kvm_vm.py class VM: · line 981

Quoted 10 lines
client/tests/kvm/kvm_vm.py +10 -0
+        """
+
+        nics = kvm_utils.get_sub_dict_names(self.params, "nics")
+        nic_name = nics[nic_index]
+        nic_params = kvm_utils.get_sub_dict(self.params, nic_name)
+        if nic_params.get("nic_ifname"):
+            return nic_params.get("nic_ifname")
+        else:
+            return "%s_%s_%s" % (nic_params.get("nic_model"),
+                                 nic_index, self.vnc_port)

What's the purpose of this string?

Quoted 67 lines
client/tests/kvm/kvm_vm.py +49 -0
+    def get_macaddr(self, nic_index=0):
+        """
+        Return the macaddr of guest nic.
+
+        @param nic_index: Index of the NIC
+        """
+        mac_filename = os.path.join(self.root_dir, "address_pool")
+        mac_shelve = shelve.open(mac_filename, writeback=False)
+        mac_pool = mac_shelve.get("macpool")
+        val = "%s:%s" % (self.instance, nic_index)
+        for key in mac_pool.keys():
+            if val in mac_pool[key]:
+                return key
+        return None
+
+    def set_mac_address(self, mac, nic_index=0, shareable=False):
+        """
+        Set mac address for guest. Note: It just update address pool.
+
+        @param mac: address will set to guest
+        @param nic_index: Index of the NIC
+        @param shareable: Where VM can share mac with other VM or not.
+        """
+        lock_filename = os.path.join(self.root_dir, "mac_lock")
+        lock_file = open(lock_filename, 'w')
+        fcntl.lockf(lock_file.fileno() ,fcntl.LOCK_EX)
+        mac_filename = os.path.join(self.root_dir, "address_pool")
+        mac_shelve = shelve.open(mac_filename, writeback=False)
+        mac_pool = mac_shelve.get("macpool")
+        if not mac_pool:
+            mac_pool = {}
+        value = "%s:%s" % (self.instance, nic_index)
+        if mac not in mac_pool.keys():
+            for key in mac_pool.keys():
+                if value in mac_pool[key]:
+                    mac_pool[key].remove(value)
+                if len(mac_pool[key]) == 0:
+                    mac_pool.pop(key)
+            mac_pool[mac] = [value]
+        else:
+            if shareable:
+                mac_pool[mac].append(value)
+            else:
+                logging.error("Mac address already be used!")
+                return False
+        mac_shelve["macpool"] = mac_pool
+        mac_shelve.close()
+        fcntl.lockf(lock_file.fileno(), fcntl.LOCK_UN)
+        lock_file.close()
 
     def get_pid(self):
         """
client/tests/kvm/tests_base.cfg.sample +1 -1
index fd9e72f..6710c00 100644
@@ -51,7 +51,7 @@ guest_port_remote_shell = 22
 nic_mode = user
 #nic_mode = tap
 nic_script = scripts/qemu-ifup
-address_index = 0
+#address_index = 0
 run_tcpdump = yes
 
 # Misc
author
#3 lore raw

Re: [Qemu-devel] [RFC PATCH 01/14] KVM-test: Add a new macaddress pool algorithm

Quoted 1 line · Michael Goldish

Michael,

Thanks for your comments. Let's simplify this method together.

Quoted 28 lines · Amos Kong
Quoted 23 lines

Old method uses the mac address in the configuration files which could lead serious problem when multiple tests running in different hosts.

This patch adds a new macaddress pool algorithm, it generates the mac prefix based on mac address of the host which could eliminate the duplicated mac addresses between machines.

When user have set the mac_prefix in the configuration file, we should use it in stead of the dynamic generated mac prefix.

Other change:
. Fix randomly generating mac address so that it correspond to IEEE802. . Update clone function to decide clone mac address or not. . Update get_macaddr function.
. Add set_mac_address function.

New auto mac address pool algorithm:
If address_index is defined, VM will get mac from config file then record mac in to address_pool. If address_index is not defined, VM will call get_mac_from_pool to auto create mac then recored mac to address_pool in following format:
{'macpool': {'AE:9D:94:6A:9b:f9': ['20100310-165222-Wt7l:0']}}

Quoted 3 lines

AE:9D:94:6A:9b:f9 : mac address
20100310-165222-Wt7l : instance attribute of VM
0 : index of NIC

Quoted 6 lines

Why do you use the mac address as a key, instead of the instance string
+ nic index? When the mac address is used as a key, each key has a list of values instead of just one value. This order seems unnatural. If it were the other way around (i.e. key = VM instance + nic index, value = mac address), then each key would have exactly one value, and I think this patch would be shorter and simpler.

One mac address may be used by two VMs, eg. migration.

Quoted 20 lines
Quoted 20 lines

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/kvm_utils.py +1 -0
index fb2d1c2..7c0946e 100644
@@ -5,6 +5,7 @@ KVM test utility functions.
 """
 
 import time, string, random, socket, os, signal, re, logging, commands, cPickle
+import fcntl, shelve
 from autotest_lib.client.bin import utils
 from autotest_lib.client.common_lib import error, logging_config
 import kvm_subprocess
@@ -82,6 +83,104 @@ def get_sub_dict_names(dict, keyword):
 
Quoted 8 lines

client/tests/kvm/kvm_utils.py def get_sub_dict_names(dict, keyword): · line 86

Quoted 3 lines
client/tests/kvm/kvm_utils.py +1 -0
 # Functions related to MAC/IP addresses
 
+def get_mac_from_pool(root_dir, vm, nic_index, prefix='00:11:22:33:'):

The name of this function is confusing because it does the exact
opposite: it puts a mac address in address_pool. Maybe the pool you're referring to in the name isn't address_pool, but still a less confusing name should probably be used.

How about allocate_mac(...) ?
address_pool -> address_container

Allocate mac address and record into address_container.

Quoted 13 lines
Quoted 13 lines
client/tests/kvm/kvm_utils.py +13 -0
+    """
+    random generated mac address.
+
+    1) First try to generate macaddress based on the mac address prefix.
+    2) And then try to use total random generated mac address.
+
+    @param root_dir: Root dir for kvm
+    @param vm: Here we use instance of vm
+    @param nic_index: The index of nic.
+    @param prefix: Prefix of mac address.
+    @Return: Return mac address.
+    """
+
Quoted 8 lines

client/tests/kvm/kvm_utils.py def get_sub_dict_names(dict, keyword): · line 103

Quoted 4 lines
client/tests/kvm/kvm_utils.py +4 -0
+    lock_filename = os.path.join(root_dir, "mac_lock")
+    lock_file = open(lock_filename, 'w')
+    fcntl.lockf(lock_file.fileno() ,fcntl.LOCK_EX)
+    mac_filename = os.path.join(root_dir, "address_pool")

Maybe it makes sense to put address_pool and the lock file in /tmp, where they can be shared by more than a single autotest instance running on the same host (unlikely, but theoretically possible).

good idea.

Quoted 6 lines

client/tests/kvm/kvm_utils.py def get_sub_dict_names(dict, keyword): · line 106

Quoted 3 lines
client/tests/kvm/kvm_utils.py +3 -0
+    mac_shelve = shelve.open(mac_filename, writeback=False)
+
+    mac_pool = mac_shelve.get("macpool")

Why is this 'macpool' needed? Why not put the keys directly in the shelve object?

yes, put keys directly in the shelve object is better.

Quoted 10 lines

client/tests/kvm/kvm_utils.py def get_sub_dict_names(dict, keyword): · line 115

Quoted 8 lines
client/tests/kvm/kvm_utils.py +8 -0
+    if not mac_pool:
+        mac_pool = {}
+    found = False
+
+    val = "%s:%s" % (vm, nic_index)
+    for key in mac_pool.keys():
+        if val in mac_pool[key]:
+            mac_pool[key].append(val)

Why append val to mac_pool[key] if val is already in mac_pool[key]?

need drop it.

Quoted 14 lines

client/tests/kvm/kvm_utils.py def get_sub_dict_names(dict, keyword): · line 128

Quoted 13 lines
client/tests/kvm/kvm_utils.py +13 -0
+            found = True
+            mac = key
+
+    while not found:
+        postfix = "%02x:%02x" % (random.randint(0x00,0xfe),
+                                random.randint(0x00,0xfe))
+        mac = prefix + postfix
+        mac_list = mac.split(":")
+        # Clear multicast bit
+        mac_list[0] = int(mac_list[0],16) & 0xfe
+        # Set local assignment bit (IEEE802)
+        mac_list[0] = mac_list[0] | 0x02
+        mac_list[0] = "%02x" % mac_list[0]
Quoted 4 lines

Why is this needed? Most mac addresses begin with 00. If the mac address is generated from the address of eth0 (using the method in this patch) it begins with 00, which is fine. If the prefix is set by the user using mac_prefix, I don't think we should modify it.

linux-2.6/include/linux/etherdevice.h

/**
* random_ether_addr - Generate software assigned random Ethernet address
* @addr: Pointer to a six-byte array containing the Ethernet address *
* Generate a random Ethernet address (MAC) that is not multicast
* and has the local assigned bit set.
*/
static inline void random_ether_addr(u8 *addr)

{
        get_random_bytes (addr, ETH_ALEN);
        addr [0] &= 0xfe;       /* clear multicast bit */
        addr [0] |= 0x02;       /* set local assignment bit (IEEE802) */
}
Quoted 9 lines
Quoted 9 lines
client/tests/kvm/kvm_utils.py +9 -0
+        mac = ":".join(mac_list)
+        if mac not in mac_pool.keys() or 0 == len(mac_pool[mac]):
+            mac_pool[mac] = ["%s:%s" % (vm,nic_index)]
+            found = True
+    mac_shelve["macpool"] = mac_pool
+    logging.debug("generating mac addr %s " % mac)
+
+    mac_shelve.close()
+    fcntl.lockf(lock_file.fileno(), fcntl.LOCK_UN)
Quoted 8 lines

client/tests/kvm/kvm_utils.py def get_sub_dict_names(dict, keyword): · line 142

Quoted 5 lines
client/tests/kvm/kvm_utils.py +5 -0
+    lock_file.close()
+    return mac
+
+
+def put_mac_to_pool(root_dir, mac, vm):

This function removes a mac address from address_pool. I wonder why you chose this name.

address_pool file is a container to record macaddress. The 'pool' we get/put mac is an abstract resource pool. This is really confused ;)

Quoted 96 lines
Quoted 96 lines
client/tests/kvm/kvm_utils.py +41 -0
+    """
+    Put the macaddress back to address pool
+
+    @param root_dir: Root dir for kvm
+    @param vm: Here we use instance attribute of vm
+    @param mac: mac address will be put.
+    @Return: mac address.
+    """
+
+    lock_filename = os.path.join(root_dir, "mac_lock")
+    lock_file = open(lock_filename, 'w')
+    fcntl.lockf(lock_file.fileno() ,fcntl.LOCK_EX)
+    mac_filename = os.path.join(root_dir, "address_pool")
+    mac_shelve = shelve.open(mac_filename, writeback=False)
+
+    mac_pool = mac_shelve.get("macpool")
+
+    if not mac_pool or (not mac in mac_pool):
+        logging.debug("Try to free a macaddress does no in pool")
+        logging.debug("macaddress is %s" % mac)
+        logging.debug("pool is %s" % mac_pool)
+    else:
+        if len(mac_pool[mac]) <= 1:
+            mac_pool.pop(mac)
+        else:
+            for value in mac_pool[mac]:
+                if vm in value:
+                    mac_pool[mac].remove(value)
+                    break
+            if len(mac_pool[mac]) == 0:
+                mac_pool.pop(mac)
+
+    mac_shelve["macpool"] = mac_pool
+    logging.debug("freeing mac addr %s " % mac)
+
+    mac_shelve.close()
+    fcntl.lockf(lock_file.fileno(), fcntl.LOCK_UN)
+    lock_file.close()
+    return mac
+
+
 def mac_str_to_int(addr):
     """
     Convert MAC address string to integer.
client/tests/kvm/kvm_vm.py +12 -3
index 6cd0688..a9ba6e7 100755
@@ -5,7 +5,7 @@ Utility classes and functions to handle Virtual Machine creation using qemu.
 @copyright: 2008-2009 Red Hat Inc.
 """
 
-import time, socket, os, logging, fcntl, re, commands, glob
+import time, socket, os, logging, fcntl, re, commands, shelve, glob
 import kvm_utils, kvm_subprocess, kvm_monitor, rss_file_transfer
 from autotest_lib.client.common_lib import error
 from autotest_lib.client.bin import utils
@@ -117,6 +117,7 @@ class VM:
         self.params = params
         self.root_dir = root_dir
         self.address_cache = address_cache
+        self.mac_prefix = params.get('mac_prefix')
         self.netdev_id = []
 
         # Find a unique identifier for this VM
@@ -126,8 +127,15 @@ class VM:
             if not glob.glob("/tmp/*%s" % self.instance):
                 break
 
+        if self.mac_prefix is None:
+            s, o = commands.getstatusoutput("ifconfig eth0")
+            if s == 0:
+                mac = re.findall("HWaddr (\S*)", o)[0]
+                self.mac_prefix = '00' + mac[8:] + ':'
+
 
-    def clone(self, name=None, params=None, root_dir=None, address_cache=None):
+    def clone(self, name=None, params=None, root_dir=None,
+                    address_cache=None, mac_clone=True):
         """
         Return a clone of the VM object with optionally modified parameters.
         The clone is initially not alive and needs to be started using create().
@@ -138,6 +146,7 @@ class VM:
         @param params: Optional new VM creation parameters
         @param root_dir: Optional new base directory for relative filenames
         @param address_cache: A dict that maps MAC addresses to IP addresses
+        @param mac_clone: Clone mac address or not.
         """
         if name is None:
             name = self.name
@@ -147,9 +156,19 @@ class VM:
             root_dir = self.root_dir
         if address_cache is None:
             address_cache = self.address_cache
-        return VM(name, params, root_dir, address_cache)
+        vm = VM(name, params, root_dir, address_cache)
Quoted 8 lines

client/tests/kvm/kvm_vm.py class VM: · line 162

Quoted 3 lines
client/tests/kvm/kvm_vm.py +3 -0
+        if mac_clone:
+            # Clone mac address by coping 'self.instance' to the new vm.
+            vm.instance = self.instance

Copying self.instance isn't a good idea because the monitor, serial console and testlog filenames use self.instance. self.instance should be unique. If we still want to use the same mac address for 2 VMs, for example in migration, a different solution should be found.

We almost only clone mac in migration test, the src guest would be killed. If the instance of dest guest is same as src guest, the serial connection would not break, it would continually write log to original log file.

Quoted 26 lines
Quoted 26 lines
client/tests/kvm/kvm_vm.py +14 -0
+        return vm
 
 
+    def free_mac_address(self):
+        nic_num = len(kvm_utils.get_sub_dict_names(self.params, "nics"))
+        for nic in range(nic_num):
+            mac = self.get_macaddr(nic_index=nic)
+            kvm_utils.put_mac_to_pool(self.root_dir, mac, vm=self.instance)
+
     def make_qemu_command(self, name=None, params=None, root_dir=None):
         """
         Generate a qemu command line. All parameters are optional. If a
@@ -383,6 +402,13 @@ class VM:
             mac = None
             if "address_index" in nic_params:
                 mac = kvm_utils.get_mac_ip_pair_from_dict(nic_params)[0]
+                self.set_mac_address(mac=mac, nic_index=vlan)
+            else:
+                mac = kvm_utils.get_mac_from_pool(self.root_dir,
+                                                  vm=self.instance,
+                                                  nic_index=vlan,
+                                                  prefix=self.mac_prefix)
+
             qemu_cmd += add_nic(help, vlan, nic_params.get("nic_model"), mac,
                                 self.netdev_id[vlan])
             # Handle the '-net tap' or '-net user' part
Quoted 8 lines

client/tests/kvm/kvm_vm.py class VM: · line 425

Quoted 6 lines
client/tests/kvm/kvm_vm.py +1 -1
@@ -396,7 +422,7 @@ class VM:
             if tftp:
                 tftp = kvm_utils.get_path(root_dir, tftp)
             qemu_cmd += add_net(help, vlan, nic_params.get("nic_mode", "user"),
-                                nic_params.get("nic_ifname"),
+                                self.get_ifname(vlan),

Why is get_ifname() useful here?

dynamically generate the ifname is better.

Quoted 77 lines
Quoted 77 lines
client/tests/kvm/kvm_vm.py +25 -3
                                 script, downscript, tftp,
                                 nic_params.get("bootp"), redirs,
                                 self.netdev_id[vlan])
@@ -720,7 +746,7 @@ class VM:
             lockfile.close()
 
 
-    def destroy(self, gracefully=True):
+    def destroy(self, gracefully=True, free_macaddr=True):
         """
         Destroy the VM.
 
@@ -731,6 +757,7 @@ class VM:
         @param gracefully: Whether an attempt will be made to end the VM
                 using a shell command before trying to end the qemu process
                 with a 'quit' or a kill signal.
+        @param free_macaddr: Whether free macaddresses when destory a vm.
         """
         try:
             # Is it already dead?
@@ -751,11 +778,18 @@ class VM:
                         logging.debug("Shutdown command sent; waiting for VM "
                                       "to go down...")
                         if kvm_utils.wait_for(self.is_dead, 60, 1, 1):
-                            logging.debug("VM is down")
+                            logging.debug("VM is down, free mac address.")
+                            # free mac address
+                            if free_macaddr:
+                                self.free_mac_address()
                             return
                     finally:
                         session.close()
 
+            # free mac address
+            if free_macaddr:
+                self.free_mac_address()
+
             if self.monitor:
                 # Try to destroy with a monitor command
                 logging.debug("Trying to kill VM with monitor command...")
@@ -881,10 +915,14 @@ class VM:
         nic_name = nics[index]
         nic_params = kvm_utils.get_sub_dict(self.params, nic_name)
         if nic_params.get("nic_mode") == "tap":
-            mac, ip = kvm_utils.get_mac_ip_pair_from_dict(nic_params)
+            mac = self.get_macaddr(index)
             if not mac:
                 logging.debug("MAC address unavailable")
                 return None
+            else:
+                mac = mac.lower()
+            ip = None
+
             if not ip or nic_params.get("always_use_tcpdump") == "yes":
                 # Get the IP address from the cache
                 ip = self.address_cache.get(mac)
@@ -897,6 +935,7 @@ class VM:
                              for nic in nics]
                 macs = [kvm_utils.get_mac_ip_pair_from_dict(dict)[0]
                         for dict in nic_dicts]
+                macs.append(mac)
                 if not kvm_utils.verify_ip_address_ownership(ip, macs):
                     logging.debug("Could not verify MAC-IP address mapping: "
                                   "%s ---> %s" % (mac, ip))
@@ -925,6 +964,71 @@ class VM:
                              "redirected" % port)
             return self.redirs.get(port)
 
+    def get_ifname(self, nic_index=0):
+        """
+        Return the ifname of tap device for the guest nic.
+
+        @param nic_index: Index of the NIC
+        """
+
+        nics = kvm_utils.get_sub_dict_names(self.params, "nics")
+        nic_name = nics[nic_index]
Quoted 8 lines

client/tests/kvm/kvm_vm.py class VM: · line 981

Quoted 6 lines
client/tests/kvm/kvm_vm.py +6 -0
+        nic_params = kvm_utils.get_sub_dict(self.params, nic_name)
+        if nic_params.get("nic_ifname"):
+            return nic_params.get("nic_ifname")
+        else:
+            return "%s_%s_%s" % (nic_params.get("nic_model"),
+                                 nic_index, self.vnc_port)

What's the purpose of this string?

Just avoid repeated ifname. The vnc_port is unique for each VM, nic_index is unique for each nic of one VM.

Quoted 68 lines
Quoted 67 lines
client/tests/kvm/kvm_vm.py +49 -0
+    def get_macaddr(self, nic_index=0):
+        """
+        Return the macaddr of guest nic.
+
+        @param nic_index: Index of the NIC
+        """
+        mac_filename = os.path.join(self.root_dir, "address_pool")
+        mac_shelve = shelve.open(mac_filename, writeback=False)
+        mac_pool = mac_shelve.get("macpool")
+        val = "%s:%s" % (self.instance, nic_index)
+        for key in mac_pool.keys():
+            if val in mac_pool[key]:
+                return key
+        return None
+
+    def set_mac_address(self, mac, nic_index=0, shareable=False):
+        """
+        Set mac address for guest. Note: It just update address pool.
+
+        @param mac: address will set to guest
+        @param nic_index: Index of the NIC
+        @param shareable: Where VM can share mac with other VM or not.
+        """
+        lock_filename = os.path.join(self.root_dir, "mac_lock")
+        lock_file = open(lock_filename, 'w')
+        fcntl.lockf(lock_file.fileno() ,fcntl.LOCK_EX)
+        mac_filename = os.path.join(self.root_dir, "address_pool")
+        mac_shelve = shelve.open(mac_filename, writeback=False)
+        mac_pool = mac_shelve.get("macpool")
+        if not mac_pool:
+            mac_pool = {}
+        value = "%s:%s" % (self.instance, nic_index)
+        if mac not in mac_pool.keys():
+            for key in mac_pool.keys():
+                if value in mac_pool[key]:
+                    mac_pool[key].remove(value)
+                if len(mac_pool[key]) == 0:
+                    mac_pool.pop(key)
+            mac_pool[mac] = [value]
+        else:
+            if shareable:
+                mac_pool[mac].append(value)
+            else:
+                logging.error("Mac address already be used!")
+                return False
+        mac_shelve["macpool"] = mac_pool
+        mac_shelve.close()
+        fcntl.lockf(lock_file.fileno(), fcntl.LOCK_UN)
+        lock_file.close()
 
     def get_pid(self):
         """
client/tests/kvm/tests_base.cfg.sample +1 -1
index fd9e72f..6710c00 100644
@@ -51,7 +51,7 @@ guest_port_remote_shell = 22
 nic_mode = user
 #nic_mode = tap
 nic_script = scripts/qemu-ifup
-address_index = 0
+#address_index = 0
 run_tcpdump = yes
 
 # Misc
#4 lore raw

Re: [Qemu-devel] [RFC PATCH 01/14] KVM-test: Add a new macaddress pool algorithm

Quoted 35 lines · Amos Kong
Quoted 1 line · Michael Goldish

Michael,

Thanks for your comments. Let's simplify this method together.

Quoted 28 lines · Amos Kong
Quoted 26 lines

Old method uses the mac address in the configuration files which could lead serious problem when multiple tests running in different hosts.

This patch adds a new macaddress pool algorithm, it generates the mac prefix based on mac address of the host which could eliminate the duplicated mac addresses between machines.

When user have set the mac_prefix in the configuration file, we should use it in stead of the dynamic generated mac prefix.

Other change:
. Fix randomly generating mac address so that it correspond to IEEE802. . Update clone function to decide clone mac address or not. . Update get_macaddr function.
. Add set_mac_address function.

New auto mac address pool algorithm:
If address_index is defined, VM will get mac from config file then record mac in to address_pool. If address_index is not defined, VM will call get_mac_from_pool to auto create mac then recored mac to address_pool in following format:
{'macpool': {'AE:9D:94:6A:9b:f9': ['20100310-165222-Wt7l:0']}}

AE:9D:94:6A:9b:f9 : mac address
20100310-165222-Wt7l : instance attribute of VM
0 : index of NIC

Quoted 8 lines
Quoted 6 lines

Why do you use the mac address as a key, instead of the instance string
+ nic index? When the mac address is used as a key, each key has a list of values instead of just one value. This order seems unnatural. If it were the other way around (i.e. key = VM instance + nic index, value = mac address), then each key would have exactly one value, and I think this patch would be shorter and simpler.

One mac address may be used by two VMs, eg. migration.

Sure, that's why I thought the opposite direction would be better: keys = VMs (nics), values = mac addresses. That way we have one value per key, instead of a list of values per key.

To clarify, instead of using:

{'AE:9D:94:6A:9b:f9': ['20100310-165222-Wt7l:0', '20100310-165222-Wt7l:1', '20100310-165222-Wt7l:2']}

I suggest:

{'20100310-165222-Wt7l:0': 'AE:9D:94:6A:9b:f9',
'20100310-165222-Wt7l:1': 'AE:9D:94:6A:9b:f9',
'20100310-165222-Wt7l:2': 'AE:9D:94:6A:9b:f9'}

Quoted 29 lines
Quoted 20 lines
Quoted 20 lines

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/kvm_utils.py +1 -0
index fb2d1c2..7c0946e 100644
@@ -5,6 +5,7 @@ KVM test utility functions.
 """
 
 import time, string, random, socket, os, signal, re, logging, commands, cPickle
+import fcntl, shelve
 from autotest_lib.client.bin import utils
 from autotest_lib.client.common_lib import error, logging_config
 import kvm_subprocess
@@ -82,6 +83,104 @@ def get_sub_dict_names(dict, keyword):
 
Quoted 8 lines

client/tests/kvm/kvm_utils.py def get_sub_dict_names(dict, keyword): · line 86

Quoted 3 lines
client/tests/kvm/kvm_utils.py +1 -0
 # Functions related to MAC/IP addresses
 
+def get_mac_from_pool(root_dir, vm, nic_index, prefix='00:11:22:33:'):

The name of this function is confusing because it does the exact
opposite: it puts a mac address in address_pool. Maybe the pool you're referring to in the name isn't address_pool, but still a less confusing name should probably be used.

Quoted 4 lines

How about allocate_mac(...) ?
address_pool -> address_container

Allocate mac address and record into address_container.

Yes, something like that, sounds less confusing.

Quoted 76 lines
Quoted 13 lines
Quoted 13 lines
client/tests/kvm/kvm_utils.py +13 -0
+    """
+    random generated mac address.
+
+    1) First try to generate macaddress based on the mac address prefix.
+    2) And then try to use total random generated mac address.
+
+    @param root_dir: Root dir for kvm
+    @param vm: Here we use instance of vm
+    @param nic_index: The index of nic.
+    @param prefix: Prefix of mac address.
+    @Return: Return mac address.
+    """
+
Quoted 8 lines

client/tests/kvm/kvm_utils.py def get_sub_dict_names(dict, keyword): · line 103

Quoted 4 lines
client/tests/kvm/kvm_utils.py +4 -0
+    lock_filename = os.path.join(root_dir, "mac_lock")
+    lock_file = open(lock_filename, 'w')
+    fcntl.lockf(lock_file.fileno() ,fcntl.LOCK_EX)
+    mac_filename = os.path.join(root_dir, "address_pool")

Maybe it makes sense to put address_pool and the lock file in /tmp, where they can be shared by more than a single autotest instance running on the same host (unlikely, but theoretically possible).

good idea.

Quoted 6 lines

client/tests/kvm/kvm_utils.py def get_sub_dict_names(dict, keyword): · line 106

Quoted 3 lines
client/tests/kvm/kvm_utils.py +3 -0
+    mac_shelve = shelve.open(mac_filename, writeback=False)
+
+    mac_pool = mac_shelve.get("macpool")

Why is this 'macpool' needed? Why not put the keys directly in the shelve object?

yes, put keys directly in the shelve object is better.

Quoted 10 lines

client/tests/kvm/kvm_utils.py def get_sub_dict_names(dict, keyword): · line 115

Quoted 8 lines
client/tests/kvm/kvm_utils.py +8 -0
+    if not mac_pool:
+        mac_pool = {}
+    found = False
+
+    val = "%s:%s" % (vm, nic_index)
+    for key in mac_pool.keys():
+        if val in mac_pool[key]:
+            mac_pool[key].append(val)

Why append val to mac_pool[key] if val is already in mac_pool[key]?

need drop it.

Quoted 14 lines

client/tests/kvm/kvm_utils.py def get_sub_dict_names(dict, keyword): · line 128

Quoted 13 lines
client/tests/kvm/kvm_utils.py +13 -0
+            found = True
+            mac = key
+
+    while not found:
+        postfix = "%02x:%02x" % (random.randint(0x00,0xfe),
+                                random.randint(0x00,0xfe))
+        mac = prefix + postfix
+        mac_list = mac.split(":")
+        # Clear multicast bit
+        mac_list[0] = int(mac_list[0],16) & 0xfe
+        # Set local assignment bit (IEEE802)
+        mac_list[0] = mac_list[0] | 0x02
+        mac_list[0] = "%02x" % mac_list[0]
Quoted 4 lines

Why is this needed? Most mac addresses begin with 00. If the mac address is generated from the address of eth0 (using the method in this patch) it begins with 00, which is fine. If the prefix is set by the user using mac_prefix, I don't think we should modify it.

linux-2.6/include/linux/etherdevice.h

/**
* random_ether_addr - Generate software assigned random Ethernet address
* @addr: Pointer to a six-byte array containing the Ethernet address *
* Generate a random Ethernet address (MAC) that is not multicast
* and has the local assigned bit set.
*/
static inline void random_ether_addr(u8 *addr)

Quoted 5 lines
{
        get_random_bytes (addr, ETH_ALEN);
        addr [0] &= 0xfe;       /* clear multicast bit */
        addr [0] |= 0x02;       /* set local assignment bit (IEEE802) */
}

I saw this googling, but I still don't understand why it's necessary when real devices start with 00.

Quoted 18 lines
Quoted 9 lines
Quoted 9 lines
client/tests/kvm/kvm_utils.py +9 -0
+        mac = ":".join(mac_list)
+        if mac not in mac_pool.keys() or 0 == len(mac_pool[mac]):
+            mac_pool[mac] = ["%s:%s" % (vm,nic_index)]
+            found = True
+    mac_shelve["macpool"] = mac_pool
+    logging.debug("generating mac addr %s " % mac)
+
+    mac_shelve.close()
+    fcntl.lockf(lock_file.fileno(), fcntl.LOCK_UN)
Quoted 8 lines

client/tests/kvm/kvm_utils.py def get_sub_dict_names(dict, keyword): · line 142

Quoted 5 lines
client/tests/kvm/kvm_utils.py +5 -0
+    lock_file.close()
+    return mac
+
+
+def put_mac_to_pool(root_dir, mac, vm):

This function removes a mac address from address_pool. I wonder why you chose this name.

Quoted 3 lines

address_pool file is a container to record macaddress. The 'pool' we get/put mac is an abstract resource pool. This is really confused ;)

Yes, I guessed it would be something like that. Still a bit confusing, I agree, but it's not a big deal to change function names.

Quoted 105 lines
Quoted 96 lines
Quoted 96 lines
client/tests/kvm/kvm_utils.py +41 -0
+    """
+    Put the macaddress back to address pool
+
+    @param root_dir: Root dir for kvm
+    @param vm: Here we use instance attribute of vm
+    @param mac: mac address will be put.
+    @Return: mac address.
+    """
+
+    lock_filename = os.path.join(root_dir, "mac_lock")
+    lock_file = open(lock_filename, 'w')
+    fcntl.lockf(lock_file.fileno() ,fcntl.LOCK_EX)
+    mac_filename = os.path.join(root_dir, "address_pool")
+    mac_shelve = shelve.open(mac_filename, writeback=False)
+
+    mac_pool = mac_shelve.get("macpool")
+
+    if not mac_pool or (not mac in mac_pool):
+        logging.debug("Try to free a macaddress does no in pool")
+        logging.debug("macaddress is %s" % mac)
+        logging.debug("pool is %s" % mac_pool)
+    else:
+        if len(mac_pool[mac]) <= 1:
+            mac_pool.pop(mac)
+        else:
+            for value in mac_pool[mac]:
+                if vm in value:
+                    mac_pool[mac].remove(value)
+                    break
+            if len(mac_pool[mac]) == 0:
+                mac_pool.pop(mac)
+
+    mac_shelve["macpool"] = mac_pool
+    logging.debug("freeing mac addr %s " % mac)
+
+    mac_shelve.close()
+    fcntl.lockf(lock_file.fileno(), fcntl.LOCK_UN)
+    lock_file.close()
+    return mac
+
+
 def mac_str_to_int(addr):
     """
     Convert MAC address string to integer.
client/tests/kvm/kvm_vm.py +12 -3
index 6cd0688..a9ba6e7 100755
@@ -5,7 +5,7 @@ Utility classes and functions to handle Virtual Machine creation using qemu.
 @copyright: 2008-2009 Red Hat Inc.
 """
 
-import time, socket, os, logging, fcntl, re, commands, glob
+import time, socket, os, logging, fcntl, re, commands, shelve, glob
 import kvm_utils, kvm_subprocess, kvm_monitor, rss_file_transfer
 from autotest_lib.client.common_lib import error
 from autotest_lib.client.bin import utils
@@ -117,6 +117,7 @@ class VM:
         self.params = params
         self.root_dir = root_dir
         self.address_cache = address_cache
+        self.mac_prefix = params.get('mac_prefix')
         self.netdev_id = []
 
         # Find a unique identifier for this VM
@@ -126,8 +127,15 @@ class VM:
             if not glob.glob("/tmp/*%s" % self.instance):
                 break
 
+        if self.mac_prefix is None:
+            s, o = commands.getstatusoutput("ifconfig eth0")
+            if s == 0:
+                mac = re.findall("HWaddr (\S*)", o)[0]
+                self.mac_prefix = '00' + mac[8:] + ':'
+
 
-    def clone(self, name=None, params=None, root_dir=None, address_cache=None):
+    def clone(self, name=None, params=None, root_dir=None,
+                    address_cache=None, mac_clone=True):
         """
         Return a clone of the VM object with optionally modified parameters.
         The clone is initially not alive and needs to be started using create().
@@ -138,6 +146,7 @@ class VM:
         @param params: Optional new VM creation parameters
         @param root_dir: Optional new base directory for relative filenames
         @param address_cache: A dict that maps MAC addresses to IP addresses
+        @param mac_clone: Clone mac address or not.
         """
         if name is None:
             name = self.name
@@ -147,9 +156,19 @@ class VM:
             root_dir = self.root_dir
         if address_cache is None:
             address_cache = self.address_cache
-        return VM(name, params, root_dir, address_cache)
+        vm = VM(name, params, root_dir, address_cache)
Quoted 8 lines

client/tests/kvm/kvm_vm.py class VM: · line 162

Quoted 3 lines
client/tests/kvm/kvm_vm.py +3 -0
+        if mac_clone:
+            # Clone mac address by coping 'self.instance' to the new vm.
+            vm.instance = self.instance

Copying self.instance isn't a good idea because the monitor, serial console and testlog filenames use self.instance. self.instance should be unique. If we still want to use the same mac address for 2 VMs, for example in migration, a different solution should be found.

Quoted 3 lines

We almost only clone mac in migration test, the src guest would be killed. If the instance of dest guest is same as src guest, the serial connection would not break, it would continually write log to original log file.

What about the monitor(s)? The fact that the serial console connection doesn't break doesn't necessarily mean everything's OK.

Quoted 35 lines
Quoted 26 lines
Quoted 26 lines
client/tests/kvm/kvm_vm.py +14 -0
+        return vm
 
 
+    def free_mac_address(self):
+        nic_num = len(kvm_utils.get_sub_dict_names(self.params, "nics"))
+        for nic in range(nic_num):
+            mac = self.get_macaddr(nic_index=nic)
+            kvm_utils.put_mac_to_pool(self.root_dir, mac, vm=self.instance)
+
     def make_qemu_command(self, name=None, params=None, root_dir=None):
         """
         Generate a qemu command line. All parameters are optional. If a
@@ -383,6 +402,13 @@ class VM:
             mac = None
             if "address_index" in nic_params:
                 mac = kvm_utils.get_mac_ip_pair_from_dict(nic_params)[0]
+                self.set_mac_address(mac=mac, nic_index=vlan)
+            else:
+                mac = kvm_utils.get_mac_from_pool(self.root_dir,
+                                                  vm=self.instance,
+                                                  nic_index=vlan,
+                                                  prefix=self.mac_prefix)
+
             qemu_cmd += add_nic(help, vlan, nic_params.get("nic_model"), mac,
                                 self.netdev_id[vlan])
             # Handle the '-net tap' or '-net user' part
Quoted 8 lines

client/tests/kvm/kvm_vm.py class VM: · line 425

Quoted 6 lines
client/tests/kvm/kvm_vm.py +1 -1
@@ -396,7 +422,7 @@ class VM:
             if tftp:
                 tftp = kvm_utils.get_path(root_dir, tftp)
             qemu_cmd += add_net(help, vlan, nic_params.get("nic_mode", "user"),
-                                nic_params.get("nic_ifname"),
+                                self.get_ifname(vlan),

Why is get_ifname() useful here?

Quoted 1 line

dynamically generate the ifname is better.

But why? Is this ifname useful for any test?

Quoted 86 lines
Quoted 77 lines
Quoted 77 lines
client/tests/kvm/kvm_vm.py +25 -3
                                 script, downscript, tftp,
                                 nic_params.get("bootp"), redirs,
                                 self.netdev_id[vlan])
@@ -720,7 +746,7 @@ class VM:
             lockfile.close()
 
 
-    def destroy(self, gracefully=True):
+    def destroy(self, gracefully=True, free_macaddr=True):
         """
         Destroy the VM.
 
@@ -731,6 +757,7 @@ class VM:
         @param gracefully: Whether an attempt will be made to end the VM
                 using a shell command before trying to end the qemu process
                 with a 'quit' or a kill signal.
+        @param free_macaddr: Whether free macaddresses when destory a vm.
         """
         try:
             # Is it already dead?
@@ -751,11 +778,18 @@ class VM:
                         logging.debug("Shutdown command sent; waiting for VM "
                                       "to go down...")
                         if kvm_utils.wait_for(self.is_dead, 60, 1, 1):
-                            logging.debug("VM is down")
+                            logging.debug("VM is down, free mac address.")
+                            # free mac address
+                            if free_macaddr:
+                                self.free_mac_address()
                             return
                     finally:
                         session.close()
 
+            # free mac address
+            if free_macaddr:
+                self.free_mac_address()
+
             if self.monitor:
                 # Try to destroy with a monitor command
                 logging.debug("Trying to kill VM with monitor command...")
@@ -881,10 +915,14 @@ class VM:
         nic_name = nics[index]
         nic_params = kvm_utils.get_sub_dict(self.params, nic_name)
         if nic_params.get("nic_mode") == "tap":
-            mac, ip = kvm_utils.get_mac_ip_pair_from_dict(nic_params)
+            mac = self.get_macaddr(index)
             if not mac:
                 logging.debug("MAC address unavailable")
                 return None
+            else:
+                mac = mac.lower()
+            ip = None
+
             if not ip or nic_params.get("always_use_tcpdump") == "yes":
                 # Get the IP address from the cache
                 ip = self.address_cache.get(mac)
@@ -897,6 +935,7 @@ class VM:
                              for nic in nics]
                 macs = [kvm_utils.get_mac_ip_pair_from_dict(dict)[0]
                         for dict in nic_dicts]
+                macs.append(mac)
                 if not kvm_utils.verify_ip_address_ownership(ip, macs):
                     logging.debug("Could not verify MAC-IP address mapping: "
                                   "%s ---> %s" % (mac, ip))
@@ -925,6 +964,71 @@ class VM:
                              "redirected" % port)
             return self.redirs.get(port)
 
+    def get_ifname(self, nic_index=0):
+        """
+        Return the ifname of tap device for the guest nic.
+
+        @param nic_index: Index of the NIC
+        """
+
+        nics = kvm_utils.get_sub_dict_names(self.params, "nics")
+        nic_name = nics[nic_index]
Quoted 8 lines

client/tests/kvm/kvm_vm.py class VM: · line 981

Quoted 6 lines
client/tests/kvm/kvm_vm.py +6 -0
+        nic_params = kvm_utils.get_sub_dict(self.params, nic_name)
+        if nic_params.get("nic_ifname"):
+            return nic_params.get("nic_ifname")
+        else:
+            return "%s_%s_%s" % (nic_params.get("nic_model"),
+                                 nic_index, self.vnc_port)

What's the purpose of this string?

Quoted 1 line

Just avoid repeated ifname. The vnc_port is unique for each VM, nic_index is unique for each nic of one VM.

self.instance should also be unique, though it's quite long.

Quoted 69 lines
Quoted 68 lines
Quoted 67 lines
client/tests/kvm/kvm_vm.py +49 -0
+    def get_macaddr(self, nic_index=0):
+        """
+        Return the macaddr of guest nic.
+
+        @param nic_index: Index of the NIC
+        """
+        mac_filename = os.path.join(self.root_dir, "address_pool")
+        mac_shelve = shelve.open(mac_filename, writeback=False)
+        mac_pool = mac_shelve.get("macpool")
+        val = "%s:%s" % (self.instance, nic_index)
+        for key in mac_pool.keys():
+            if val in mac_pool[key]:
+                return key
+        return None
+
+    def set_mac_address(self, mac, nic_index=0, shareable=False):
+        """
+        Set mac address for guest. Note: It just update address pool.
+
+        @param mac: address will set to guest
+        @param nic_index: Index of the NIC
+        @param shareable: Where VM can share mac with other VM or not.
+        """
+        lock_filename = os.path.join(self.root_dir, "mac_lock")
+        lock_file = open(lock_filename, 'w')
+        fcntl.lockf(lock_file.fileno() ,fcntl.LOCK_EX)
+        mac_filename = os.path.join(self.root_dir, "address_pool")
+        mac_shelve = shelve.open(mac_filename, writeback=False)
+        mac_pool = mac_shelve.get("macpool")
+        if not mac_pool:
+            mac_pool = {}
+        value = "%s:%s" % (self.instance, nic_index)
+        if mac not in mac_pool.keys():
+            for key in mac_pool.keys():
+                if value in mac_pool[key]:
+                    mac_pool[key].remove(value)
+                if len(mac_pool[key]) == 0:
+                    mac_pool.pop(key)
+            mac_pool[mac] = [value]
+        else:
+            if shareable:
+                mac_pool[mac].append(value)
+            else:
+                logging.error("Mac address already be used!")
+                return False
+        mac_shelve["macpool"] = mac_pool
+        mac_shelve.close()
+        fcntl.lockf(lock_file.fileno(), fcntl.LOCK_UN)
+        lock_file.close()
 
     def get_pid(self):
         """
client/tests/kvm/tests_base.cfg.sample +1 -1
index fd9e72f..6710c00 100644
@@ -51,7 +51,7 @@ guest_port_remote_shell = 22
 nic_mode = user
 #nic_mode = tap
 nic_script = scripts/qemu-ifup
-address_index = 0
+#address_index = 0
 run_tcpdump = yes
 
 # Misc
author
#5 lore raw

Re: [Qemu-devel] [RFC PATCH 01/14] KVM-test: Add a new macaddress pool algorithm

Quoted 7 lines · Michael Goldish
Quoted 6 lines · Amos Kong
Quoted 1 line · Michael Goldish

Michael,

Thanks for your comments. Let's simplify this method together.

I would produce a v2 later basing on our discussion.

Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 5 lines
client/tests/kvm/kvm_vm.py +5 -0
+    def get_ifname(self, nic_index=0):
+        """
+        Return the ifname of tap device for the guest nic.
+
+        @param nic_index: Index of the NIC

client/tests/kvm/kvm_vm.py class VM: · line 981

Quoted 10 lines
client/tests/kvm/kvm_vm.py +10 -0
+        """
+
+        nics = kvm_utils.get_sub_dict_names(self.params, "nics")
+        nic_name = nics[nic_index]
+        nic_params = kvm_utils.get_sub_dict(self.params, nic_name)
+        if nic_params.get("nic_ifname"):
+            return nic_params.get("nic_ifname")
+        else:
+            return "%s_%s_%s" % (nic_params.get("nic_model"),
+                                 nic_index, self.vnc_port)

What's the purpose of this string?

Quoted 3 lines
Quoted 1 line

Just avoid repeated ifname. The vnc_port is unique for each VM, nic_index is unique for each nic of one VM.

self.instance should also be unique, though it's quite long.

qemu can only receive 15 chars as ifname, instance is too long. if the prefix of two instance are same, the actual ifnames will not be unique.

linux-user/syscall.c:
#define IFNAMSIZ 16

net/tap-linux.c:
int tap_open(...
...

    if (ifname[0] != '\0')
        pstrcpy(ifr.ifr_name, IFNAMSIZ, ifname);
    else
        pstrcpy(ifr.ifr_name, IFNAMSIZ, "tap%d");
#6 lore raw

Re: [RFC PATCH 01/14] KVM-test: Add a new macaddress pool algorithm

In reply to Amos Kong
Quoted 9 lines · Amos Kong

Old method uses the mac address in the configuration files which could lead serious problem when multiple tests running in different hosts.

This patch adds a new macaddress pool algorithm, it generates the mac prefix based on mac address of the host which could eliminate the duplicated mac addresses between machines.

When user have set the mac_prefix in the configuration file, we should use it in stead of the dynamic generated mac prefix.

I have some additional comments besides the good points made by Michael on his review.

Quoted 37 lines

Other change:
. Fix randomly generating mac address so that it correspond to IEEE802. . Update clone function to decide clone mac address or not. . Update get_macaddr function.
. Add set_mac_address function.

New auto mac address pool algorithm:
If address_index is defined, VM will get mac from config file then record mac in to address_pool. If address_index is not defined, VM will call get_mac_from_pool to auto create mac then recored mac to address_pool in following format:
{'macpool': {'AE:9D:94:6A:9b:f9': ['20100310-165222-Wt7l:0']}}

AE:9D:94:6A:9b:f9 : mac address
20100310-165222-Wt7l : instance attribute of VM
0 : index of NIC

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/kvm_utils.py +1 -0
index fb2d1c2..7c0946e 100644
@@ -5,6 +5,7 @@ KVM test utility functions.
 """
 
 import time, string, random, socket, os, signal, re, logging, commands, cPickle
+import fcntl, shelve
 from autotest_lib.client.bin import utils
 from autotest_lib.client.common_lib import error, logging_config
 import kvm_subprocess

client/tests/kvm/kvm_utils.py def get_sub_dict_names(dict, keyword): · line 88

Quoted 6 lines
client/tests/kvm/kvm_utils.py +3 -0
@@ -82,6 +83,104 @@ def get_sub_dict_names(dict, keyword):
 
 # Functions related to MAC/IP addresses
 
+def get_mac_from_pool(root_dir, vm, nic_index, prefix='00:11:22:33:'):
+    """
+    random generated mac address.

I think I got your original idea of naming this function get_mac_from_pool (the prefix assessed is the base of the 'pool'), but as Michael pointed out, a new MAC is being generated here and added to the mac pool, so a more appropriate name should be used. A straightforward suggestion would be generate_mac_address. Also, the description of the function would be more on the lines of "Generate a MAC address and add it to the MAC pool".

Quoted 5 lines
client/tests/kvm/kvm_utils.py +5 -0
+    1) First try to generate macaddress based on the mac address prefix.
+    2) And then try to use total random generated mac address.
+
+    @param root_dir: Root dir for kvm
+    @param vm: Here we use instance of vm

client/tests/kvm/kvm_utils.py def get_sub_dict_names(dict, keyword): · line 104

Quoted 10 lines
client/tests/kvm/kvm_utils.py +10 -0
+    @param nic_index: The index of nic.
+    @param prefix: Prefix of mac address.
+    @Return: Return mac address.
+    """
+
+    lock_filename = os.path.join(root_dir, "mac_lock")
+    lock_file = open(lock_filename, 'w')
+    fcntl.lockf(lock_file.fileno() ,fcntl.LOCK_EX)
+    mac_filename = os.path.join(root_dir, "address_pool")
+    mac_shelve = shelve.open(mac_filename, writeback=False)

^ Indeed we should keep the mac pool under /tmp as Michael already pointed out.

Quoted 7 lines
client/tests/kvm/kvm_utils.py +7 -0
+    mac_pool = mac_shelve.get("macpool")
+
+    if not mac_pool:
+        mac_pool = {}
+    found = False
+
+    val = "%s:%s" % (vm, nic_index)

client/tests/kvm/kvm_utils.py def get_sub_dict_names(dict, keyword): · line 122

Quoted 10 lines
client/tests/kvm/kvm_utils.py +10 -0
+    for key in mac_pool.keys():
+        if val in mac_pool[key]:
+            mac_pool[key].append(val)
+            found = True
+            mac = key
+
+    while not found:
+        postfix = "%02x:%02x" % (random.randint(0x00,0xfe),
+                                random.randint(0x00,0xfe))
+        mac = prefix + postfix

^ It's more natural to use 'sufix' instead of 'postfix' here.

client/tests/kvm/kvm_utils.py def get_sub_dict_names(dict, keyword): · line 134

Quoted 12 lines
client/tests/kvm/kvm_utils.py +12 -0
+        mac_list = mac.split(":")
+        # Clear multicast bit
+        mac_list[0] = int(mac_list[0],16) & 0xfe
+        # Set local assignment bit (IEEE802)
+        mac_list[0] = mac_list[0] | 0x02
+        mac_list[0] = "%02x" % mac_list[0]
+        mac = ":".join(mac_list)
+        if mac not in mac_pool.keys() or 0 == len(mac_pool[mac]):
+            mac_pool[mac] = ["%s:%s" % (vm,nic_index)]
+            found = True
+    mac_shelve["macpool"] = mac_pool
+    logging.debug("generating mac addr %s " % mac)

^ Here the MAC was already generated, so here you could use something like "Generated MAC address %s"

client/tests/kvm/kvm_utils.py def get_sub_dict_names(dict, keyword): · line 147

Quoted 12 lines
client/tests/kvm/kvm_utils.py +12 -0
+    mac_shelve.close()
+    fcntl.lockf(lock_file.fileno(), fcntl.LOCK_UN)
+    lock_file.close()
+    return mac
+
+
+def put_mac_to_pool(root_dir, mac, vm):
+    """
+    Put the macaddress back to address pool
+
+    @param root_dir: Root dir for kvm
+    @param vm: Here we use instance attribute of vm

^ @param vm: Instance attribute of a vm. Also, we might want to rename this parameter to vm_instance, to make things more clear and avoid confusions (vm is referred in several places as a vm instance).

Quoted 4 lines
client/tests/kvm/kvm_utils.py +4 -0
+    @param mac: mac address will be put.
+    @Return: mac address.
+    """
+

client/tests/kvm/kvm_utils.py def get_sub_dict_names(dict, keyword): · line 161

Quoted 10 lines
client/tests/kvm/kvm_utils.py +10 -0
+    lock_filename = os.path.join(root_dir, "mac_lock")
+    lock_file = open(lock_filename, 'w')
+    fcntl.lockf(lock_file.fileno() ,fcntl.LOCK_EX)
+    mac_filename = os.path.join(root_dir, "address_pool")
+    mac_shelve = shelve.open(mac_filename, writeback=False)
+
+    mac_pool = mac_shelve.get("macpool")
+
+    if not mac_pool or (not mac in mac_pool):
+        logging.debug("Try to free a macaddress does no in pool")

^ logging.debug("MAC not present in the MAC pool, not modifying pool")

Quoted 5 lines
client/tests/kvm/kvm_utils.py +5 -0
+        logging.debug("macaddress is %s" % mac)
+        logging.debug("pool is %s" % mac_pool)
+    else:
+        if len(mac_pool[mac]) <= 1:
+            mac_pool.pop(mac)

client/tests/kvm/kvm_utils.py def get_sub_dict_names(dict, keyword): · line 176

Quoted 10 lines
client/tests/kvm/kvm_utils.py +10 -0
+        else:
+            for value in mac_pool[mac]:
+                if vm in value:
+                    mac_pool[mac].remove(value)
+                    break
+            if len(mac_pool[mac]) == 0:
+                mac_pool.pop(mac)
+
+    mac_shelve["macpool"] = mac_pool
+    logging.debug("freeing mac addr %s " % mac)

^ logging.debug("Freeing mac addr %s " % mac). Also, this should go to the beginning of the function, before the "MAC not present in the..." message.

Quoted 49 lines
client/tests/kvm/kvm_utils.py +7 -0
+
+    mac_shelve.close()
+    fcntl.lockf(lock_file.fileno(), fcntl.LOCK_UN)
+    lock_file.close()
+    return mac
+
+
 def mac_str_to_int(addr):
     """
     Convert MAC address string to integer.
client/tests/kvm/kvm_vm.py +10 -2
index 6cd0688..a9ba6e7 100755
@@ -5,7 +5,7 @@ Utility classes and functions to handle Virtual Machine creation using qemu.
 @copyright: 2008-2009 Red Hat Inc.
 """
 
-import time, socket, os, logging, fcntl, re, commands, glob
+import time, socket, os, logging, fcntl, re, commands, shelve, glob
 import kvm_utils, kvm_subprocess, kvm_monitor, rss_file_transfer
 from autotest_lib.client.common_lib import error
 from autotest_lib.client.bin import utils
@@ -117,6 +117,7 @@ class VM:
         self.params = params
         self.root_dir = root_dir
         self.address_cache = address_cache
+        self.mac_prefix = params.get('mac_prefix')
         self.netdev_id = []
 
         # Find a unique identifier for this VM
@@ -126,8 +127,15 @@ class VM:
             if not glob.glob("/tmp/*%s" % self.instance):
                 break
 
+        if self.mac_prefix is None:
+            s, o = commands.getstatusoutput("ifconfig eth0")
+            if s == 0:
+                mac = re.findall("HWaddr (\S*)", o)[0]
+                self.mac_prefix = '00' + mac[8:] + ':'
+
 
-    def clone(self, name=None, params=None, root_dir=None, address_cache=None):
+    def clone(self, name=None, params=None, root_dir=None,
+                    address_cache=None, mac_clone=True):
         """
         Return a clone of the VM object with optionally modified parameters.
         The clone is initially not alive and needs to be started using create().

client/tests/kvm/kvm_vm.py class VM: · line 150

Quoted 5 lines
client/tests/kvm/kvm_vm.py +1 -0
@@ -138,6 +146,7 @@ class VM:
         @param params: Optional new VM creation parameters
         @param root_dir: Optional new base directory for relative filenames
         @param address_cache: A dict that maps MAC addresses to IP addresses
+        @param mac_clone: Clone mac address or not.
         """

Here if you redesign the patch as suggested by Michael, making the unique key the string 'instance' + 'nic_index', you could add a parameter like 'preserve_mac', to preserve the original mac address, for things like migration.

In such cases, first the entry of the nics present on the source vm would be removed from the MAC pool, then the new entries would be added.

Quoted 9 lines
client/tests/kvm/kvm_vm.py +2 -1
         if name is None:
             name = self.name
@@ -147,9 +156,19 @@ class VM:
             root_dir = self.root_dir
         if address_cache is None:
             address_cache = self.address_cache
-        return VM(name, params, root_dir, address_cache)
+        vm = VM(name, params, root_dir, address_cache)
+        if mac_clone:

client/tests/kvm/kvm_vm.py class VM: · line 170

Quoted 10 lines
client/tests/kvm/kvm_vm.py +8 -0
+            # Clone mac address by coping 'self.instance' to the new vm.
+            vm.instance = self.instance
+        return vm
 
 
+    def free_mac_address(self):
+        nic_num = len(kvm_utils.get_sub_dict_names(self.params, "nics"))
+        for nic in range(nic_num):
+            mac = self.get_macaddr(nic_index=nic)
+            kvm_utils.put_mac_to_pool(self.root_dir, mac, vm=self.instance)

^ Here we are removing an association between a 'instance'+'nic_index' and a MAC, so maybe this should be called 'clear_macs_from_nics'

Quoted 36 lines
client/tests/kvm/kvm_vm.py +9 -2
     def make_qemu_command(self, name=None, params=None, root_dir=None):
         """
         Generate a qemu command line. All parameters are optional. If a
@@ -383,6 +402,13 @@ class VM:
             mac = None
             if "address_index" in nic_params:
                 mac = kvm_utils.get_mac_ip_pair_from_dict(nic_params)[0]
+                self.set_mac_address(mac=mac, nic_index=vlan)
+            else:
+                mac = kvm_utils.get_mac_from_pool(self.root_dir,
+                                                  vm=self.instance,
+                                                  nic_index=vlan,
+                                                  prefix=self.mac_prefix)
+
             qemu_cmd += add_nic(help, vlan, nic_params.get("nic_model"), mac,
                                 self.netdev_id[vlan])
             # Handle the '-net tap' or '-net user' part
@@ -396,7 +422,7 @@ class VM:
             if tftp:
                 tftp = kvm_utils.get_path(root_dir, tftp)
             qemu_cmd += add_net(help, vlan, nic_params.get("nic_mode", "user"),
-                                nic_params.get("nic_ifname"),
+                                self.get_ifname(vlan),
                                 script, downscript, tftp,
                                 nic_params.get("bootp"), redirs,
                                 self.netdev_id[vlan])
@@ -720,7 +746,7 @@ class VM:
             lockfile.close()
 
 
-    def destroy(self, gracefully=True):
+    def destroy(self, gracefully=True, free_macaddr=True):
         """
         Destroy the VM.
 

client/tests/kvm/kvm_vm.py class VM: · line 760

Quoted 4 lines
client/tests/kvm/kvm_vm.py +1 -0
@@ -731,6 +757,7 @@ class VM:
         @param gracefully: Whether an attempt will be made to end the VM
                 using a shell command before trying to end the qemu process
                 with a 'quit' or a kill signal.
+        @param free_macaddr: Whether free macaddresses when destory a vm.

^ Why would we want to conditionally remove the associations NIC-MAC? Don't we want to do it on all cases?

Quoted 133 lines
client/tests/kvm/kvm_vm.py +79 -2
         """
         try:
             # Is it already dead?
@@ -751,11 +778,18 @@ class VM:
                         logging.debug("Shutdown command sent; waiting for VM "
                                       "to go down...")
                         if kvm_utils.wait_for(self.is_dead, 60, 1, 1):
-                            logging.debug("VM is down")
+                            logging.debug("VM is down, free mac address.")
+                            # free mac address
+                            if free_macaddr:
+                                self.free_mac_address()
                             return
                     finally:
                         session.close()
 
+            # free mac address
+            if free_macaddr:
+                self.free_mac_address()
+
             if self.monitor:
                 # Try to destroy with a monitor command
                 logging.debug("Trying to kill VM with monitor command...")
@@ -881,10 +915,14 @@ class VM:
         nic_name = nics[index]
         nic_params = kvm_utils.get_sub_dict(self.params, nic_name)
         if nic_params.get("nic_mode") == "tap":
-            mac, ip = kvm_utils.get_mac_ip_pair_from_dict(nic_params)
+            mac = self.get_macaddr(index)
             if not mac:
                 logging.debug("MAC address unavailable")
                 return None
+            else:
+                mac = mac.lower()
+            ip = None
+
             if not ip or nic_params.get("always_use_tcpdump") == "yes":
                 # Get the IP address from the cache
                 ip = self.address_cache.get(mac)
@@ -897,6 +935,7 @@ class VM:
                              for nic in nics]
                 macs = [kvm_utils.get_mac_ip_pair_from_dict(dict)[0]
                         for dict in nic_dicts]
+                macs.append(mac)
                 if not kvm_utils.verify_ip_address_ownership(ip, macs):
                     logging.debug("Could not verify MAC-IP address mapping: "
                                   "%s ---> %s" % (mac, ip))
@@ -925,6 +964,71 @@ class VM:
                              "redirected" % port)
             return self.redirs.get(port)
 
+    def get_ifname(self, nic_index=0):
+        """
+        Return the ifname of tap device for the guest nic.
+
+        @param nic_index: Index of the NIC
+        """
+
+        nics = kvm_utils.get_sub_dict_names(self.params, "nics")
+        nic_name = nics[nic_index]
+        nic_params = kvm_utils.get_sub_dict(self.params, nic_name)
+        if nic_params.get("nic_ifname"):
+            return nic_params.get("nic_ifname")
+        else:
+            return "%s_%s_%s" % (nic_params.get("nic_model"),
+                                 nic_index, self.vnc_port)
+
+    def get_macaddr(self, nic_index=0):
+        """
+        Return the macaddr of guest nic.
+
+        @param nic_index: Index of the NIC
+        """
+        mac_filename = os.path.join(self.root_dir, "address_pool")
+        mac_shelve = shelve.open(mac_filename, writeback=False)
+        mac_pool = mac_shelve.get("macpool")
+        val = "%s:%s" % (self.instance, nic_index)
+        for key in mac_pool.keys():
+            if val in mac_pool[key]:
+                return key
+        return None
+
+    def set_mac_address(self, mac, nic_index=0, shareable=False):
+        """
+        Set mac address for guest. Note: It just update address pool.
+
+        @param mac: address will set to guest
+        @param nic_index: Index of the NIC
+        @param shareable: Where VM can share mac with other VM or not.
+        """
+        lock_filename = os.path.join(self.root_dir, "mac_lock")
+        lock_file = open(lock_filename, 'w')
+        fcntl.lockf(lock_file.fileno() ,fcntl.LOCK_EX)
+        mac_filename = os.path.join(self.root_dir, "address_pool")
+        mac_shelve = shelve.open(mac_filename, writeback=False)
+        mac_pool = mac_shelve.get("macpool")
+        if not mac_pool:
+            mac_pool = {}
+        value = "%s:%s" % (self.instance, nic_index)
+        if mac not in mac_pool.keys():
+            for key in mac_pool.keys():
+                if value in mac_pool[key]:
+                    mac_pool[key].remove(value)
+                if len(mac_pool[key]) == 0:
+                    mac_pool.pop(key)
+            mac_pool[mac] = [value]
+        else:
+            if shareable:
+                mac_pool[mac].append(value)
+            else:
+                logging.error("Mac address already be used!")
+                return False
+        mac_shelve["macpool"] = mac_pool
+        mac_shelve.close()
+        fcntl.lockf(lock_file.fileno(), fcntl.LOCK_UN)
+        lock_file.close()
 
     def get_pid(self):
         """
client/tests/kvm/tests_base.cfg.sample +1 -1
index fd9e72f..6710c00 100644
@@ -51,7 +51,7 @@ guest_port_remote_shell = 22
 nic_mode = user
 #nic_mode = tap
 nic_script = scripts/qemu-ifup
-address_index = 0
+#address_index = 0
 run_tcpdump = yes
 
 # Misc
author
#7 lore raw

RFC PATCH 2/14 KVM Test: Add a function get_interface_name() to kvm_net_utils.py

In reply to Amos Kong

The function get_interface_name is used to get the interface name of linux guest through the macaddress of specified macaddress.

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/kvm_net_utils.py new file +18 -0
new file mode 100644
index 0000000..ede4965
@@ -0,0 +1,18 @@
+import re
+
+def get_linux_ifname(session, mac_address):
+    """
+    Get the interface name through the mac address.
+
+    @param session: session to the virtual machine
+    @mac_address: the macaddress of nic
+    """
+
+    output = session.get_command_output("ifconfig -a")
+
+    try:
+        ethname = re.findall("(\w+)\s+Link.*%s" % mac_address, output,
+                             re.IGNORECASE)[0]
+        return ethname
+    except:
+        return None
#8 lore raw

Re: [Qemu-devel] [RFC PATCH 02/14] KVM Test: Add a function get_interface_name() to kvm_net_utils.py

Quoted 2 lines · Amos Kong

The function get_interface_name is used to get the interface name of linux guest through the macaddress of specified macaddress.

I wonder if it wouldn't be overkill to have separate utility libraries on the kvm test instead of a single kvm_utils and kvm_test_utils like you are proposing. Any thoughts Michael?

Quoted 31 lines

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/kvm_net_utils.py new file +18 -0
new file mode 100644
index 0000000..ede4965
@@ -0,0 +1,18 @@
+import re
+
+def get_linux_ifname(session, mac_address):
+    """
+    Get the interface name through the mac address.
+
+    @param session: session to the virtual machine
+    @mac_address: the macaddress of nic
+    """
+
+    output = session.get_command_output("ifconfig -a")
+
+    try:
+        ethname = re.findall("(\w+)\s+Link.*%s" % mac_address, output,
+                             re.IGNORECASE)[0]
+        return ethname
+    except:
+        return None
#9 lore raw

Re: [Qemu-devel] [RFC PATCH 02/14] KVM Test: Add a function get_interface_name() to kvm_net_utils.py

Quoted 7 lines · Lucas Meneghel Rodrigues
Quoted 2 lines · Amos Kong

The function get_interface_name is used to get the interface name of linux guest through the macaddress of specified macaddress.

I wonder if it wouldn't be overkill to have separate utility libraries on the kvm test instead of a single kvm_utils and kvm_test_utils like you are proposing. Any thoughts Michael?

Yes, looks like this could be in kvm_test_utils.py, especially if there's only a small number of functions here.

Quoted 37 lines
Quoted 21 lines

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/kvm_net_utils.py new file +8 -0
new file mode 100644
index 0000000..ede4965
@@ -0,0 +1,18 @@
+import re
+
+def get_linux_ifname(session, mac_address):
+    """
+    Get the interface name through the mac address.
+
+    @param session: session to the virtual machine
+    @mac_address: the macaddress of nic

client/tests/kvm/kvm_net_utils.py line 18

Quoted 10 lines
client/tests/kvm/kvm_net_utils.py new file +10 -0
+    """
+
+    output = session.get_command_output("ifconfig -a")
+
+    try:
+        ethname = re.findall("(\w+)\s+Link.*%s" % mac_address, output,
+                             re.IGNORECASE)[0]
+        return ethname
+    except:
+        return None

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#10 lore raw

Re: [Qemu-devel] [RFC PATCH 02/14] KVM Test: Add a function get_interface_name() to kvm_net_utils.py

Quoted 11 lines · Michael Goldish
Quoted 7 lines · Lucas Meneghel Rodrigues
Quoted 2 lines · Amos Kong

The function get_interface_name is used to get the interface name of linux guest through the macaddress of specified macaddress.

I wonder if it wouldn't be overkill to have separate utility libraries on the kvm test instead of a single kvm_utils and kvm_test_utils like you are proposing. Any thoughts Michael?

Yes, looks like this could be in kvm_test_utils.py, especially if there's only a small number of functions here.

Current functions of network are not too much, but we may add a lot of new utility functions in the future. It's more clear to use another file.

Quoted 38 lines
Quoted 37 lines
Quoted 21 lines

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/kvm_net_utils.py new file +8 -0
new file mode 100644
index 0000000..ede4965
@@ -0,0 +1,18 @@
+import re
+
+def get_linux_ifname(session, mac_address):
+    """
+    Get the interface name through the mac address.
+
+    @param session: session to the virtual machine
+    @mac_address: the macaddress of nic

client/tests/kvm/kvm_net_utils.py line 18

Quoted 10 lines
client/tests/kvm/kvm_net_utils.py new file +10 -0
+    """
+
+    output = session.get_command_output("ifconfig -a")
+
+    try:
+        ethname = re.findall("(\w+)\s+Link.*%s" % mac_address, output,
+                             re.IGNORECASE)[0]
+        return ethname
+    except:
+        return None

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#11 lore raw

RFC PATCH 3/14 KVM Test: Add a common ping module for network related tests

In reply to Amos Kong

The kvm_net_utils.py is a just a place that wraps common network related commands which is used to do the network-related tests. Use -1 as the packet ratio for loss analysis.
Use quiet mode when doing the flood ping.

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/kvm_net_utils.py +111 -1
index ede4965..8a71858 100644
@@ -1,4 +1,114 @@
-import re
+import logging, re, signal
+from autotest_lib.client.common_lib import error
+import kvm_subprocess, kvm_utils
+
+def get_loss_ratio(output):
+    """
+    Get the packet loss ratio from the output of ping
+
+    @param output
+    """
+    try:
+        return int(re.findall('(\d+)% packet loss', output)[0])
+    except IndexError:
+        logging.debug(output)
+        return -1
+
+def raw_ping(command, timeout, session, output_func):
+    """
+    Low-level ping command execution.
+
+    @param command: ping command
+    @param timeout: timeout of the ping command
+    @param session: local executon hint or session to execute the ping command
+    """
+    if session == "localhost":
+        process = kvm_subprocess.run_bg(command, output_func=output_func,
+                                        timeout=timeout)
+
+        # Send SIGINT singal to notify the timeout of running ping process,
+        # Because ping have the ability to catch the SIGINT signal so we can
+        # always get the packet loss ratio even if timeout.
+        if process.is_alive():
+            kvm_utils.kill_process_tree(process.get_pid(), signal.SIGINT)
+
+        status = process.get_status()
+        output = process.get_output()
+
+        process.close()
+        return status, output
+    else:
+        session.sendline(command)
+        status, output = session.read_up_to_prompt(timeout=timeout,
+                                                   print_func=output_func)
+        if status is False:
+            # Send ctrl+c (SIGINT) through ssh session
+            session.sendline("\003")
+            status, output2 = session.read_up_to_prompt(print_func=output_func)
+            output += output2
+            if status is False:
+                # We also need to use this session to query the return value
+                session.sendline("\003")
+
+        session.sendline(session.status_test_command)
+        s2, o2 = session.read_up_to_prompt()
+        if s2 is False:
+            status = -1
+        else:
+            try:
+                status = int(re.findall("\d+", o2)[0])
+            except:
+                status = -1
+
+        return status, output
+
+def ping(dest = "localhost", count = None, interval = None, interface = None,
+         packetsize = None, ttl = None, hint = None, adaptive = False,
+         broadcast = False, flood = False, timeout = 0,
+         output_func = logging.debug, session = "localhost"):
+    """
+    Wrapper of ping.
+
+    @param dest: destination address
+    @param count: count of icmp packet
+    @param interval: interval of two icmp echo request
+    @param interface: specified interface of the source address
+    @param packetsize: packet size of icmp
+    @param ttl: ip time to live
+    @param hint: path mtu discovery hint
+    @param adaptive: adaptive ping flag
+    @param broadcast: broadcast ping flag
+    @param flood: flood ping flag
+    @param timeout: timeout for the ping command
+    @param output_func: function used to log the result of ping
+    @param session: local executon hint or session to execute the ping command
+    """
+
+    command = "ping %s " % dest
+
+    if count is not None:
+        command += " -c %s" % count
+    if interval is not None:
+        command += " -i %s" % interval
+    if interface is not None:
+        command += " -I %s" % interface
+    if packetsize is not None:
+        command += " -s %s" % packetsize
+    if ttl is not None:
+        command += " -t %s" % ttl
+    if hint is not None:
+        command += " -M %s" % hint
+    if adaptive is True:
+        command += " -A"
+    if broadcast is True:
+        command += " -b"
+    if flood is True:
+        # temporary workaround as the kvm_subprocess may not properly handle
+        # the timeout for the output of flood ping
+        command += " -f -q"
+        output_func = None
+
+    return raw_ping(command, timeout, session, output_func)
 
 def get_linux_ifname(session, mac_address):
     """
#12 lore raw

Re: [Qemu-devel] [RFC PATCH 03/14] KVM Test: Add a common ping module for network related tests

Quoted 33 lines · Amos Kong

The kvm_net_utils.py is a just a place that wraps common network related commands which is used to do the network-related tests. Use -1 as the packet ratio for loss analysis.
Use quiet mode when doing the flood ping.

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/kvm_net_utils.py +16 -1
index ede4965..8a71858 100644
@@ -1,4 +1,114 @@
-import re
+import logging, re, signal
+from autotest_lib.client.common_lib import error
+import kvm_subprocess, kvm_utils
+
+def get_loss_ratio(output):
+    """
+    Get the packet loss ratio from the output of ping
+
+    @param output
+    """
+    try:
+        return int(re.findall('(\d+)% packet loss', output)[0])
+    except IndexError:
+        logging.debug(output)
+        return -1
 
+def raw_ping(command, timeout, session, output_func):

client/tests/kvm/kvm_net_utils.py line 27

Quoted 10 lines
client/tests/kvm/kvm_net_utils.py +10 -0
+    """
+    Low-level ping command execution.
+
+    @param command: ping command
+    @param timeout: timeout of the ping command
+    @param session: local executon hint or session to execute the ping command
+    """
+    if session == "localhost":
+        process = kvm_subprocess.run_bg(command, output_func=output_func,
+                                        timeout=timeout)

Do we really have to resort to kvm_subprocess here? We use subprocess on special occasions, usually when the command in question is required to live throughout the tests, which doesn't seem to be the case. kvm_subprocess is a good library, but it is a more heavyweight alternative (spawns its own shell process, for example). Maybe you are using it because you can directly send input to the process at any time, such as the ctrl+c later on this same code.

client/tests/kvm/kvm_net_utils.py line 29

Quoted 2 lines
client/tests/kvm/kvm_net_utils.py +2 -0
+
+        # Send SIGINT singal to notify the timeout of running ping process,

^ typo, signal

Quoted 5 lines
client/tests/kvm/kvm_net_utils.py +5 -0
+        # Because ping have the ability to catch the SIGINT signal so we can
+        # always get the packet loss ratio even if timeout.
+        if process.is_alive():
+            kvm_utils.kill_process_tree(process.get_pid(), signal.SIGINT)
+

client/tests/kvm/kvm_net_utils.py line 44

Quoted 10 lines
client/tests/kvm/kvm_net_utils.py +10 -0
+        status = process.get_status()
+        output = process.get_output()
+
+        process.close()
+        return status, output
+    else:
+        session.sendline(command)
+        status, output = session.read_up_to_prompt(timeout=timeout,
+                                                   print_func=output_func)
+        if status is False:

^ For None objects, it is better to explicitly test for is None. However the above construct seems more natural if put as

if not status:

Any particular reason you tested explicitely for False?

client/tests/kvm/kvm_net_utils.py line 55

Quoted 11 lines
client/tests/kvm/kvm_net_utils.py +11 -0
+            # Send ctrl+c (SIGINT) through ssh session
+            session.sendline("\003")
+            status, output2 = session.read_up_to_prompt(print_func=output_func)
+            output += output2
+            if status is False:
+                # We also need to use this session to query the return value
+                session.sendline("\003")
+
+        session.sendline(session.status_test_command)
+        s2, o2 = session.read_up_to_prompt()
+        if s2 is False:

^ See comment above

client/tests/kvm/kvm_net_utils.py line 68

Quoted 13 lines
client/tests/kvm/kvm_net_utils.py +13 -0
+            status = -1
+        else:
+            try:
+                status = int(re.findall("\d+", o2)[0])
+            except:
+                status = -1
+
+        return status, output
+
+def ping(dest = "localhost", count = None, interval = None, interface = None,
+         packetsize = None, ttl = None, hint = None, adaptive = False,
+         broadcast = False, flood = False, timeout = 0,
+         output_func = logging.debug, session = "localhost"):

^ On function headers, we pretty much follow PEP 8 and don't use spacing between argument keys, the equal sign and the default value, so this header should be rewritten

def ping(dest="localhost",...)

Quoted 31 lines
client/tests/kvm/kvm_net_utils.py +31 -0
+    """
+    Wrapper of ping.
+
+    @param dest: destination address
+    @param count: count of icmp packet
+    @param interval: interval of two icmp echo request
+    @param interface: specified interface of the source address
+    @param packetsize: packet size of icmp
+    @param ttl: ip time to live
+    @param hint: path mtu discovery hint
+    @param adaptive: adaptive ping flag
+    @param broadcast: broadcast ping flag
+    @param flood: flood ping flag
+    @param timeout: timeout for the ping command
+    @param output_func: function used to log the result of ping
+    @param session: local executon hint or session to execute the ping command
+    """
+
+    command = "ping %s " % dest
+
+    if count is not None:
+        command += " -c %s" % count
+    if interval is not None:
+        command += " -i %s" % interval
+    if interface is not None:
+        command += " -I %s" % interface
+    if packetsize is not None:
+        command += " -s %s" % packetsize
+    if ttl is not None:
+        command += " -t %s" % ttl
+    if hint is not None:

client/tests/kvm/kvm_net_utils.py line 109

Quoted 10 lines
client/tests/kvm/kvm_net_utils.py +10 -0
+        command += " -M %s" % hint
+    if adaptive is True:
+        command += " -A"
+    if broadcast is True:
+        command += " -b"
+    if flood is True:
+        # temporary workaround as the kvm_subprocess may not properly handle
+        # the timeout for the output of flood ping
+        command += " -f -q"
+        output_func = None

^ the last 3 if clauses could be simpler if put as:

if adaptive:

if broadcast:

if flood:

Quoted 6 lines
client/tests/kvm/kvm_net_utils.py +1 -0
+    return raw_ping(command, timeout, session, output_func)
 
 def get_linux_ifname(session, mac_address):
     """
#13 lore raw

Re: [Qemu-devel] [RFC PATCH 03/14] KVM Test: Add a common ping module for network related tests

Quoted 33 lines · Lucas Meneghel Rodrigues
Quoted 31 lines · Amos Kong

The kvm_net_utils.py is a just a place that wraps common network related commands which is used to do the network-related tests. Use -1 as the packet ratio for loss analysis.
Use quiet mode when doing the flood ping.

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/kvm_net_utils.py +15 -1
index ede4965..8a71858 100644
@@ -1,4 +1,114 @@
-import re
+import logging, re, signal
+from autotest_lib.client.common_lib import error
+import kvm_subprocess, kvm_utils
+
+def get_loss_ratio(output):
+    """
+    Get the packet loss ratio from the output of ping
+
+    @param output
+    """
+    try:
+        return int(re.findall('(\d+)% packet loss', output)[0])
+    except IndexError:
+        logging.debug(output)
+        return -1
Quoted 9 lines

client/tests/kvm/kvm_net_utils.py line 25

Quoted 9 lines
client/tests/kvm/kvm_net_utils.py +9 -0
+def raw_ping(command, timeout, session, output_func):
+    """
+    Low-level ping command execution.
+
+    @param command: ping command
+    @param timeout: timeout of the ping command
+    @param session: local executon hint or session to execute the ping command
+    """
+    if session == "localhost":

I have no problem with this, but wouldn't it be nicer to use None to indicate that the session should be local?

Quoted 35 lines

client/tests/kvm/kvm_net_utils.py line 27

Quoted 2 lines
client/tests/kvm/kvm_net_utils.py +2 -0
+        process = kvm_subprocess.run_bg(command, output_func=output_func,
+                                        timeout=timeout)

Do we really have to resort to kvm_subprocess here? We use subprocess on special occasions, usually when the command in question is required to live throughout the tests, which doesn't seem to be the case. kvm_subprocess is a good library, but it is a more heavyweight alternative (spawns its own shell process, for example). Maybe you are using it because you can directly send input to the process at any time, such as the ctrl+c later on this same code.

client/tests/kvm/kvm_net_utils.py line 28

Quoted 2 lines
client/tests/kvm/kvm_net_utils.py +2 -0
+
+        # Send SIGINT singal to notify the timeout of running ping process,

^ typo, signal

Quoted 5 lines
client/tests/kvm/kvm_net_utils.py +5 -0
+        # Because ping have the ability to catch the SIGINT signal so we can
+        # always get the packet loss ratio even if timeout.
+        if process.is_alive():
+            kvm_utils.kill_process_tree(process.get_pid(), signal.SIGINT)
+

client/tests/kvm/kvm_net_utils.py line 44

Quoted 10 lines
client/tests/kvm/kvm_net_utils.py +10 -0
+        status = process.get_status()
+        output = process.get_output()
+
+        process.close()
+        return status, output
+    else:
+        session.sendline(command)
+        status, output = session.read_up_to_prompt(timeout=timeout,
+                                                   print_func=output_func)
+        if status is False:

^ For None objects, it is better to explicitly test for is None. However the above construct seems more natural if put as

Quoted 3 lines

if not status:

Any particular reason you tested explicitely for False?

read_up_to_prompt() returns True/False as the first member of the tuple.

Quoted 2 lines

client/tests/kvm/kvm_net_utils.py line 45

Quoted 2 lines
client/tests/kvm/kvm_net_utils.py +2 -0
+            # Send ctrl+c (SIGINT) through ssh session
+            session.sendline("\003")

I think you can use send() here. sendline() calls send() with a newline added to the string.

Quoted 5 lines

client/tests/kvm/kvm_net_utils.py line 51

Quoted 5 lines
client/tests/kvm/kvm_net_utils.py +5 -0
+            status, output2 = session.read_up_to_prompt(print_func=output_func)
+            output += output2
+            if status is False:
+                # We also need to use this session to query the return value
+                session.sendline("\003")

Same here.

Quoted 25 lines

client/tests/kvm/kvm_net_utils.py line 55

Quoted 4 lines
client/tests/kvm/kvm_net_utils.py +4 -0
+
+        session.sendline(session.status_test_command)
+        s2, o2 = session.read_up_to_prompt()
+        if s2 is False:

^ See comment above

client/tests/kvm/kvm_net_utils.py line 68

Quoted 13 lines
client/tests/kvm/kvm_net_utils.py +13 -0
+            status = -1
+        else:
+            try:
+                status = int(re.findall("\d+", o2)[0])
+            except:
+                status = -1
+
+        return status, output
+
+def ping(dest = "localhost", count = None, interval = None, interface = None,
+         packetsize = None, ttl = None, hint = None, adaptive = False,
+         broadcast = False, flood = False, timeout = 0,
+         output_func = logging.debug, session = "localhost"):

^ On function headers, we pretty much follow PEP 8 and don't use spacing between argument keys, the equal sign and the default value, so this header should be rewritten

Quoted 18 lines

def ping(dest="localhost",...)

Quoted 16 lines
client/tests/kvm/kvm_net_utils.py +16 -0
+    """
+    Wrapper of ping.
+
+    @param dest: destination address
+    @param count: count of icmp packet
+    @param interval: interval of two icmp echo request
+    @param interface: specified interface of the source address
+    @param packetsize: packet size of icmp
+    @param ttl: ip time to live
+    @param hint: path mtu discovery hint
+    @param adaptive: adaptive ping flag
+    @param broadcast: broadcast ping flag
+    @param flood: flood ping flag
+    @param timeout: timeout for the ping command
+    @param output_func: function used to log the result of ping
+    @param session: local executon hint or session to execute the ping command

Don't we need this for Windows as well? If we do, wouldn't it be easier to use a parameter (e.g. 'ping_options = -c 3') to directly control the ping options from the config file, instead of using an OS-specific ping wrapper?

Quoted 15 lines
Quoted 15 lines
client/tests/kvm/kvm_net_utils.py +15 -0
+    """
+
+    command = "ping %s " % dest
+
+    if count is not None:
+        command += " -c %s" % count
+    if interval is not None:
+        command += " -i %s" % interval
+    if interface is not None:
+        command += " -I %s" % interface
+    if packetsize is not None:
+        command += " -s %s" % packetsize
+    if ttl is not None:
+        command += " -t %s" % ttl
+    if hint is not None:
Quoted 8 lines

client/tests/kvm/kvm_net_utils.py line 107

Quoted 8 lines
client/tests/kvm/kvm_net_utils.py +8 -0
+        command += " -M %s" % hint
+    if adaptive is True:
+        command += " -A"
+    if broadcast is True:
+        command += " -b"
+    if flood is True:
+        # temporary workaround as the kvm_subprocess may not properly handle
+        # the timeout for the output of flood ping

What do you mean by this? If there's a problem with kvm_subprocess it should be fixed. Have you tried passing internal_timeout=0 to the kvm_subprocess function you're using?

Quoted 1 line

client/tests/kvm/kvm_net_utils.py line 108

Quoted 1 line
client/tests/kvm/kvm_net_utils.py +1 -0
+        command += " -f -q"

If a lot of output is generated, it may not be a bad idea to use -q here regardless of kvm_subprocess limitations.

Quoted 14 lines

client/tests/kvm/kvm_net_utils.py line 109

Quoted 1 line
client/tests/kvm/kvm_net_utils.py +1 -0
+        output_func = None

^ the last 3 if clauses could be simpler if put as:

if adaptive:

if broadcast:

if flood:

Quoted 4 lines
client/tests/kvm/kvm_net_utils.py +1 -0
+    return raw_ping(command, timeout, session, output_func)
 
 def get_linux_ifname(session, mac_address):
     """
#14 lore raw

Re: [Qemu-devel] [RFC PATCH 03/14] KVM Test: Add a common ping module for network related tests

Quoted 146 lines · Michael Goldish
Quoted 33 lines · Lucas Meneghel Rodrigues
Quoted 31 lines · Amos Kong

The kvm_net_utils.py is a just a place that wraps common network related commands which is used to do the network-related tests. Use -1 as the packet ratio for loss analysis.
Use quiet mode when doing the flood ping.

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/kvm_net_utils.py +15 -1
index ede4965..8a71858 100644
@@ -1,4 +1,114 @@
-import re
+import logging, re, signal
+from autotest_lib.client.common_lib import error
+import kvm_subprocess, kvm_utils
+
+def get_loss_ratio(output):
+    """
+    Get the packet loss ratio from the output of ping
+
+    @param output
+    """
+    try:
+        return int(re.findall('(\d+)% packet loss', output)[0])
+    except IndexError:
+        logging.debug(output)
+        return -1
Quoted 9 lines

client/tests/kvm/kvm_net_utils.py line 25

Quoted 9 lines
client/tests/kvm/kvm_net_utils.py +9 -0
+def raw_ping(command, timeout, session, output_func):
+    """
+    Low-level ping command execution.
+
+    @param command: ping command
+    @param timeout: timeout of the ping command
+    @param session: local executon hint or session to execute the ping command
+    """
+    if session == "localhost":

I have no problem with this, but wouldn't it be nicer to use None to indicate that the session should be local?

Quoted 35 lines

client/tests/kvm/kvm_net_utils.py line 27

Quoted 2 lines
client/tests/kvm/kvm_net_utils.py +2 -0
+        process = kvm_subprocess.run_bg(command, output_func=output_func,
+                                        timeout=timeout)

Do we really have to resort to kvm_subprocess here? We use subprocess on special occasions, usually when the command in question is required to live throughout the tests, which doesn't seem to be the case. kvm_subprocess is a good library, but it is a more heavyweight alternative (spawns its own shell process, for example). Maybe you are using it because you can directly send input to the process at any time, such as the ctrl+c later on this same code.

client/tests/kvm/kvm_net_utils.py line 28

Quoted 2 lines
client/tests/kvm/kvm_net_utils.py +2 -0
+
+        # Send SIGINT singal to notify the timeout of running ping process,

^ typo, signal

Quoted 5 lines
client/tests/kvm/kvm_net_utils.py +5 -0
+        # Because ping have the ability to catch the SIGINT signal so we can
+        # always get the packet loss ratio even if timeout.
+        if process.is_alive():
+            kvm_utils.kill_process_tree(process.get_pid(), signal.SIGINT)
+

client/tests/kvm/kvm_net_utils.py line 44

Quoted 10 lines
client/tests/kvm/kvm_net_utils.py +10 -0
+        status = process.get_status()
+        output = process.get_output()
+
+        process.close()
+        return status, output
+    else:
+        session.sendline(command)
+        status, output = session.read_up_to_prompt(timeout=timeout,
+                                                   print_func=output_func)
+        if status is False:

^ For None objects, it is better to explicitly test for is None. However the above construct seems more natural if put as

Quoted 3 lines

if not status:

Any particular reason you tested explicitely for False?

read_up_to_prompt() returns True/False as the first member of the tuple.

Quoted 2 lines

client/tests/kvm/kvm_net_utils.py line 45

Quoted 2 lines
client/tests/kvm/kvm_net_utils.py +2 -0
+            # Send ctrl+c (SIGINT) through ssh session
+            session.sendline("\003")

I think you can use send() here. sendline() calls send() with a newline added to the string.

Quoted 5 lines

client/tests/kvm/kvm_net_utils.py line 51

Quoted 5 lines
client/tests/kvm/kvm_net_utils.py +5 -0
+            status, output2 = session.read_up_to_prompt(print_func=output_func)
+            output += output2
+            if status is False:
+                # We also need to use this session to query the return value
+                session.sendline("\003")

Same here.

Quoted 25 lines

client/tests/kvm/kvm_net_utils.py line 55

Quoted 4 lines
client/tests/kvm/kvm_net_utils.py +4 -0
+
+        session.sendline(session.status_test_command)
+        s2, o2 = session.read_up_to_prompt()
+        if s2 is False:

^ See comment above

client/tests/kvm/kvm_net_utils.py line 68

Quoted 13 lines
client/tests/kvm/kvm_net_utils.py +13 -0
+            status = -1
+        else:
+            try:
+                status = int(re.findall("\d+", o2)[0])
+            except:
+                status = -1
+
+        return status, output
+
+def ping(dest = "localhost", count = None, interval = None, interface = None,
+         packetsize = None, ttl = None, hint = None, adaptive = False,
+         broadcast = False, flood = False, timeout = 0,
+         output_func = logging.debug, session = "localhost"):

^ On function headers, we pretty much follow PEP 8 and don't use spacing between argument keys, the equal sign and the default value, so this header should be rewritten

Quoted 18 lines

def ping(dest="localhost",...)

Quoted 16 lines
client/tests/kvm/kvm_net_utils.py +16 -0
+    """
+    Wrapper of ping.
+
+    @param dest: destination address
+    @param count: count of icmp packet
+    @param interval: interval of two icmp echo request
+    @param interface: specified interface of the source address
+    @param packetsize: packet size of icmp
+    @param ttl: ip time to live
+    @param hint: path mtu discovery hint
+    @param adaptive: adaptive ping flag
+    @param broadcast: broadcast ping flag
+    @param flood: flood ping flag
+    @param timeout: timeout for the ping command
+    @param output_func: function used to log the result of ping
+    @param session: local executon hint or session to execute the ping command
Quoted 4 lines

Don't we need this for Windows as well? If we do, wouldn't it be easier to use a parameter (e.g. 'ping_options = -c 3') to directly control the ping options from the config file, instead of using an OS-specific ping wrapper?

Please disregard this comment. After looking at some of the other patches in the set I realized it wasn't relevant.

Quoted 47 lines
Quoted 15 lines
Quoted 15 lines
client/tests/kvm/kvm_net_utils.py +15 -0
+    """
+
+    command = "ping %s " % dest
+
+    if count is not None:
+        command += " -c %s" % count
+    if interval is not None:
+        command += " -i %s" % interval
+    if interface is not None:
+        command += " -I %s" % interface
+    if packetsize is not None:
+        command += " -s %s" % packetsize
+    if ttl is not None:
+        command += " -t %s" % ttl
+    if hint is not None:
Quoted 8 lines

client/tests/kvm/kvm_net_utils.py line 107

Quoted 8 lines
client/tests/kvm/kvm_net_utils.py +8 -0
+        command += " -M %s" % hint
+    if adaptive is True:
+        command += " -A"
+    if broadcast is True:
+        command += " -b"
+    if flood is True:
+        # temporary workaround as the kvm_subprocess may not properly handle
+        # the timeout for the output of flood ping

What do you mean by this? If there's a problem with kvm_subprocess it should be fixed. Have you tried passing internal_timeout=0 to the kvm_subprocess function you're using?

Quoted 1 line

client/tests/kvm/kvm_net_utils.py line 108

Quoted 1 line
client/tests/kvm/kvm_net_utils.py +1 -0
+        command += " -f -q"

If a lot of output is generated, it may not be a bad idea to use -q here regardless of kvm_subprocess limitations.

Quoted 14 lines

client/tests/kvm/kvm_net_utils.py line 109

Quoted 1 line
client/tests/kvm/kvm_net_utils.py +1 -0
+        output_func = None

^ the last 3 if clauses could be simpler if put as:

if adaptive:

if broadcast:

if flood:

Quoted 4 lines
client/tests/kvm/kvm_net_utils.py +1 -0
+    return raw_ping(command, timeout, session, output_func)
 
 def get_linux_ifname(session, mac_address):
     """
author
#15 lore raw

RFC PATCH 4/14 KVM-test: Add a new subtest ping

In reply to Amos Kong

This test use ping to check the virtual nics, it contains two kinds of test:
1. Packet loss ratio test, ping the guest with different size of packets.
2. Stress test, flood ping guest then use ordinary ping to test the network.

The interval and packet size could be configurated through tests_base.cfg

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/tests/ping.py new file +71 -0
new file mode 100644
index 0000000..cfccda4
@@ -0,0 +1,71 @@
+import logging, time, re, commands
+from autotest_lib.client.common_lib import error
+import kvm_subprocess, kvm_test_utils, kvm_utils, kvm_net_utils
+
+
+def run_ping(test, params, env):
+    """
+    Ping the guest with different size of packets.
+
+    Packet Loss Test:
+    1) Ping the guest with different size/interval of packets.
+    Stress Test:
+    1) Flood ping the guest.
+    2) Check if the network is still usable.
+
+    @param test: Kvm test object
+    @param params: Dictionary with the test parameters
+    @param env: Dictionary with test environment.
+    """
+
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    session = kvm_test_utils.wait_for_login(vm)
+
+    counts = params.get("ping_counts", 100)
+    flood_minutes = float(params.get("flood_minutes", 10))
+    nics = params.get("nics").split()
+    strict_check = params.get("strict_check", "no") == "yes"
+
+    packet_size = [0, 1, 4, 48, 512, 1440, 1500, 1505, 4054, 4055, 4096, 4192,
+                   8878, 9000, 32767, 65507]
+
+    try:
+        for i, nic in enumerate(nics):
+            ip = vm.get_address(i)
+            if not ip:
+                logging.error("Could not get the ip of nic index %d" % i)
+                continue
+
+            for size in packet_size:
+                logging.info("Ping with packet size %s" % size)
+                status, output = kvm_net_utils.ping(ip, 10,
+                                                    packetsize = size,
+                                                    timeout = 20)
+                if strict_check:
+                    ratio = kvm_net_utils.get_loss_ratio(output)
+                    if ratio != 0:
+                        raise error.TestFail(" Loss ratio is %s for packet size"
+                                             " %s" % (ratio, size))
+                else:
+                    if status != 0:
+                        raise error.TestFail(" Ping returns non-zero value %s" %
+                                             output)
+
+            logging.info("Flood ping test")
+            kvm_net_utils.ping(ip, None, flood = True, output_func= None,
+                               timeout = flood_minutes * 60)
+
+            logging.info("Final ping test")
+            status, output = kvm_net_utils.ping(ip, counts,
+                                                timeout = float(counts) * 1.5)
+            if strict_check:
+                ratio = kvm_net_utils.get_loss_ratio(output)
+                if ratio != 0:
+                    raise error.TestFail("Packet loss ratio is %s after flood"
+                                         % ratio)
+            else:
+                if status != 0:
+                    raise error.TestFail(" Ping returns non-zero value %s" %
+                                         output)
+    finally:
+        session.close()
client/tests/kvm/tests_base.cfg.sample +5 -0
index 6710c00..4f58dc0 100644
@@ -349,6 +349,11 @@ variants:
         kill_vm_gracefully_vm2 = no
         address_index_vm2 = 1
 
+    - ping: install setup unattended_install.cdrom
+        type = ping
+        counts = 100
+        flood_minutes = 10
+
     - physical_resources_check: install setup unattended_install.cdrom
         type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'
#16 lore raw

Re: [RFC PATCH 04/14] KVM-test: Add a new subtest ping

Quoted 59 lines · Amos Kong

This test use ping to check the virtual nics, it contains two kinds of test:
1. Packet loss ratio test, ping the guest with different size of packets.
2. Stress test, flood ping guest then use ordinary ping to test the network.

The interval and packet size could be configurated through tests_base.cfg

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/tests/ping.py new file +42 -0
new file mode 100644
index 0000000..cfccda4
@@ -0,0 +1,71 @@
+import logging, time, re, commands
+from autotest_lib.client.common_lib import error
+import kvm_subprocess, kvm_test_utils, kvm_utils, kvm_net_utils
+
+
+def run_ping(test, params, env):
+    """
+    Ping the guest with different size of packets.
+
+    Packet Loss Test:
+    1) Ping the guest with different size/interval of packets.
+    Stress Test:
+    1) Flood ping the guest.
+    2) Check if the network is still usable.
+
+    @param test: Kvm test object
+    @param params: Dictionary with the test parameters
+    @param env: Dictionary with test environment.
+    """
+
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    session = kvm_test_utils.wait_for_login(vm)
+
+    counts = params.get("ping_counts", 100)
+    flood_minutes = float(params.get("flood_minutes", 10))
+    nics = params.get("nics").split()
+    strict_check = params.get("strict_check", "no") == "yes"
+
+    packet_size = [0, 1, 4, 48, 512, 1440, 1500, 1505, 4054, 4055, 4096, 4192,
+                   8878, 9000, 32767, 65507]
+
+    try:
+        for i, nic in enumerate(nics):
+            ip = vm.get_address(i)
+            if not ip:
+                logging.error("Could not get the ip of nic index %d" % i)
+                continue
+
+            for size in packet_size:
+                logging.info("Ping with packet size %s" % size)
+                status, output = kvm_net_utils.ping(ip, 10,
+                                                    packetsize = size,

client/tests/kvm/tests/ping.py line 52

Quoted 10 lines
client/tests/kvm/tests/ping.py new file +10 -0
+                                                    timeout = 20)
+                if strict_check:
+                    ratio = kvm_net_utils.get_loss_ratio(output)
+                    if ratio != 0:
+                        raise error.TestFail(" Loss ratio is %s for packet size"
+                                             " %s" % (ratio, size))
+                else:
+                    if status != 0:
+                        raise error.TestFail(" Ping returns non-zero value %s" %
+                                             output)

^ "Ping failed, status: %s, output: %s" would be a better exception message.

client/tests/kvm/tests/ping.py line 56

Quoted 4 lines
client/tests/kvm/tests/ping.py new file +4 -0
+
+            logging.info("Flood ping test")
+            kvm_net_utils.ping(ip, None, flood = True, output_func= None,
+                               timeout = flood_minutes * 60)

^ Please get rid of the spaces here

client/tests/kvm/tests/ping.py line 60

Quoted 3 lines
client/tests/kvm/tests/ping.py new file +3 -0
+            logging.info("Final ping test")
+            status, output = kvm_net_utils.ping(ip, counts,
+                                                timeout = float(counts) * 1.5)

^ What is this last test for again?

Quoted 32 lines
client/tests/kvm/tests/ping.py new file +11 -0
+            if strict_check:
+                ratio = kvm_net_utils.get_loss_ratio(output)
+                if ratio != 0:
+                    raise error.TestFail("Packet loss ratio is %s after flood"
+                                         % ratio)
+            else:
+                if status != 0:
+                    raise error.TestFail(" Ping returns non-zero value %s" %
+                                         output)
+    finally:
+        session.close()
client/tests/kvm/tests_base.cfg.sample +5 -1
index 6710c00..4f58dc0 100644
@@ -349,6 +349,11 @@ variants:
         kill_vm_gracefully_vm2 = no
         address_index_vm2 = 1
 
+    - ping: install setup unattended_install.cdrom
+        type = ping
+        counts = 100
+        flood_minutes = 10
+
     - physical_resources_check: install setup unattended_install.cdrom
         type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'
 
--

To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#17 lore raw

Re: [RFC PATCH 04/14] KVM-test: Add a new subtest ping

Quoted 79 lines · Lucas Meneghel Rodrigues
Quoted 59 lines · Amos Kong

This test use ping to check the virtual nics, it contains two kinds of test:
1. Packet loss ratio test, ping the guest with different size of packets.
2. Stress test, flood ping guest then use ordinary ping to test the network.

The interval and packet size could be configurated through tests_base.cfg

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/tests/ping.py new file +42 -0
new file mode 100644
index 0000000..cfccda4
@@ -0,0 +1,71 @@
+import logging, time, re, commands
+from autotest_lib.client.common_lib import error
+import kvm_subprocess, kvm_test_utils, kvm_utils, kvm_net_utils
+
+
+def run_ping(test, params, env):
+    """
+    Ping the guest with different size of packets.
+
+    Packet Loss Test:
+    1) Ping the guest with different size/interval of packets.
+    Stress Test:
+    1) Flood ping the guest.
+    2) Check if the network is still usable.
+
+    @param test: Kvm test object
+    @param params: Dictionary with the test parameters
+    @param env: Dictionary with test environment.
+    """
+
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    session = kvm_test_utils.wait_for_login(vm)
+
+    counts = params.get("ping_counts", 100)
+    flood_minutes = float(params.get("flood_minutes", 10))
+    nics = params.get("nics").split()
+    strict_check = params.get("strict_check", "no") == "yes"
+
+    packet_size = [0, 1, 4, 48, 512, 1440, 1500, 1505, 4054, 4055, 4096, 4192,
+                   8878, 9000, 32767, 65507]
+
+    try:
+        for i, nic in enumerate(nics):
+            ip = vm.get_address(i)
+            if not ip:
+                logging.error("Could not get the ip of nic index %d" % i)
+                continue
+
+            for size in packet_size:
+                logging.info("Ping with packet size %s" % size)
+                status, output = kvm_net_utils.ping(ip, 10,
+                                                    packetsize = size,

client/tests/kvm/tests/ping.py line 52

Quoted 10 lines
client/tests/kvm/tests/ping.py new file +10 -0
+                                                    timeout = 20)
+                if strict_check:
+                    ratio = kvm_net_utils.get_loss_ratio(output)
+                    if ratio != 0:
+                        raise error.TestFail(" Loss ratio is %s for packet size"
+                                             " %s" % (ratio, size))
+                else:
+                    if status != 0:
+                        raise error.TestFail(" Ping returns non-zero value %s" %
+                                             output)

^ "Ping failed, status: %s, output: %s" would be a better exception message.

client/tests/kvm/tests/ping.py line 56

Quoted 4 lines
client/tests/kvm/tests/ping.py new file +4 -0
+
+            logging.info("Flood ping test")
+            kvm_net_utils.ping(ip, None, flood = True, output_func= None,
+                               timeout = flood_minutes * 60)
Quoted 7 lines

^ Please get rid of the spaces here

client/tests/kvm/tests/ping.py line 60

Quoted 3 lines
client/tests/kvm/tests/ping.py new file +3 -0
+            logging.info("Final ping test")
+            status, output = kvm_net_utils.ping(ip, counts,
+                                                timeout = float(counts) * 1.5)

^ What is this last test for again?

We could not raise an error when flood ping failed(loss packet/ return non-zero), it's too strict. But we must check the ping result before/after floop-ping.

Quoted 35 lines
Quoted 33 lines
client/tests/kvm/tests/ping.py new file +11 -0
+            if strict_check:
+                ratio = kvm_net_utils.get_loss_ratio(output)
+                if ratio != 0:
+                    raise error.TestFail("Packet loss ratio is %s after flood"
+                                         % ratio)
+            else:
+                if status != 0:
+                    raise error.TestFail(" Ping returns non-zero value %s" %
+                                         output)
+    finally:
+        session.close()
client/tests/kvm/tests_base.cfg.sample +5 -1
index 6710c00..4f58dc0 100644
@@ -349,6 +349,11 @@ variants:
         kill_vm_gracefully_vm2 = no
         address_index_vm2 = 1
 
+    - ping: install setup unattended_install.cdrom
+        type = ping
+        counts = 100
+        flood_minutes = 10
+
     - physical_resources_check: install setup unattended_install.cdrom
         type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'
 
--

To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#18 lore raw

RFC PATCH 5/14 KVM-test: Add a subtest jumbo

In reply to Amos Kong

According to different nic model set different MTU for it. And ping from guest to host, to see whether tested size can be received by host.

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/tests/jumbo.py new file +133 -0
new file mode 100644
index 0000000..9f56a87
@@ -0,0 +1,133 @@
+import os, re, logging, commands, time, random
+from autotest_lib.client.common_lib import error
+import kvm_subprocess, kvm_test_utils, kvm_utils, kvm_net_utils
+
+def run_jumbo(test, params, env):
+    """
+    Test the RX jumbo frame function of vnics:
+    1) boot the vm
+    2) change the MTU of guest nics and host taps depends on the nic model
+    3) add the static arp entry for guest nic
+    4) wait for the MTU ok
+    5) verify the patch mtu using ping
+    6) ping the guest with large frames
+    7) increament size ping
+    8) flood ping the guest with large frames
+    9) verify the path mtu
+    10) revocer the mtu
+
+    @param test: kvm test object
+    @param params: Dictionary with the test parameters
+    @param env: Dictionary with test environment.
+    """
+
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    session = kvm_test_utils.wait_for_login(vm)
+    mtu = params.get("mtu", "1500")
+    flood_time = params.get("flood_time", "300")
+    max_icmp_pkt_size = int(mtu) - 28
+
+    ifname = vm.get_ifname(0)
+    ip = vm.get_address(0)
+    if ip is None:
+        raise error.TestError("Could not get the ip address")
+
+    try:
+        # Environment preparartion
+        ethname = kvm_net_utils.get_linux_ifname(session, vm.get_macaddr(0))
+
+        logging.info("Changing the mtu of guest ...")
+        guest_mtu_cmd = "ifconfig %s mtu %s" % (ethname , mtu)
+        s, o = session.get_command_status_output(guest_mtu_cmd)
+        if s != 0:
+            logging.error(o)
+            raise error.TestError("Fail to set the mtu of guest nic: %s"
+                                  % ethname)
+
+        logging.info("Chaning the mtu of host tap ...")
+        host_mtu_cmd = "ifconfig %s mtu %s" % (ifname, mtu)
+        s, o = commands.getstatusoutput(host_mtu_cmd)
+        if s != 0:
+            raise error.TestError("Fail to set the mtu of %s" % ifname)
+
+        logging.info("Add a temporary static arp entry ...")
+        arp_add_cmd = "arp -s %s %s -i %s" % (ip, vm.get_macaddr(0), ifname)
+        s, o = commands.getstatusoutput(arp_add_cmd)
+        if s != 0 :
+            raise error.TestError("Fail to add temporary arp entry")
+
+        def is_mtu_ok():
+            s, o = kvm_net_utils.ping(ip, 1, interface = ifname,
+                                      packetsize = max_icmp_pkt_size,
+                                      hint = "do", timeout = 2)
+            if s != 0:
+                return False
+            else:
+                return True
+
+        def verify_mtu():
+            logging.info("Verify the path mtu")
+            s, o = kvm_net_utils.ping(ip, 10, interface = ifname,
+                                      packetsize = max_icmp_pkt_size,
+                                      hint = "do", timeout = 15)
+            if s != 0 :
+                logging.error(o)
+                raise error.TestFail("Path MTU is not as expected")
+            if kvm_net_utils.get_loss_ratio(o) != 0:
+                logging.error(o)
+                raise error.TestFail("Packet loss ratio during mtu verification"
+                                     " is not zero")
+
+        def flood_ping():
+            logging.info("Flood with large frames")
+            kvm_net_utils.ping(ip, interface = ifname,
+                               packetsize = max_icmp_pkt_size,
+                               flood = True, timeout = float(flood_time))
+
+        def large_frame_ping(count = 100):
+            logging.info("Large frame ping")
+            s, o = kvm_net_utils.ping(ip, count, interface = ifname,
+                                      packetsize = max_icmp_pkt_size,
+                                      timeout = float(count) * 2)
+            ratio = kvm_net_utils.get_loss_ratio(o)
+            if ratio != 0:
+                raise error.TestFail("Loss ratio of large frame ping is %s" \
+                                     % ratio)
+
+        def size_increase_ping(step = random.randrange(90, 110)):
+            logging.info("Size increase ping")
+            for size in range(0, max_icmp_pkt_size + 1, step):
+                logging.info("Ping %s with size %s" % (ip, size))
+                s, o = kvm_net_utils.ping(ip, 1, interface = ifname,
+                                          packetsize = size,
+                                          hint = "do", timeout = 1)
+                if s != 0:
+                    s, o = kvm_net_utils.ping(ip, 10, interface = ifname,
+                                              packetsize = size,
+                                              adaptive = True, hint = "do",
+                                              timeout = 20)
+                    if kvm_net_utils.get_loss_ratio(o) > 50:
+                        raise error.TestFail("ping loss ratio is greater "
+                                             "than 50% for size %s" % size)
+
+        logging.info("Waiting for the MTU to be OK")
+        # Wait for the MTU is OK
+        if not kvm_utils.wait_for(is_mtu_ok, 10, 0, 1):
+            logging.debug(commands.getoutput("ifconfig -a"))
+            raise error.TestError("10 seconds elapsed while the mtu is not"
+                                  " as expected")
+
+        # Functional Test
+        verify_mtu()
+        large_frame_ping()
+        size_increase_ping()
+
+        # Stress test
+        flood_ping()
+        verify_mtu()
+
+    finally:
+        # Environment clean
+        session.close()
+        logging.info("Removing the temporary arp entry")
+        commands.getstatusoutput("arp -d %s -i %s" % (ip, ifname))
client/tests/kvm/tests_base.cfg.sample +10 -1
index 4f58dc0..7f7b56a 100644
@@ -354,6 +354,9 @@ variants:
         counts = 100
         flood_minutes = 10
 
+    - jumbo: install setup unattended_install.cdrom
+        type = jumbo
+
     - physical_resources_check: install setup unattended_install.cdrom
         type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'
@@ -489,10 +492,16 @@ variants:
 variants:
     - @rtl8139:
         nic_model = rtl8139
+        jumbo:
+            mtu = 1500
     - e1000:
         nic_model = e1000
+        jumbo:
+            mtu = 16110
     - virtio:
         nic_model = virtio
+        jumbo:
+            mtu = 65520
 
 
 # Guests
@@ -1024,7 +1033,7 @@ variants:
 
     # Windows section
     - @Windows:
-        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks)
+        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo
         shutdown_command = shutdown /s /f /t 0
         reboot_command = shutdown /r /f /t 0
         status_test_command = echo %errorlevel%
#19 lore raw

Re: [RFC PATCH 05/14] KVM-test: Add a subtest jumbo

Quoted 14 lines · Amos Kong

According to different nic model set different MTU for it. And ping from guest to host, to see whether tested size can be received by host.

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/tests/jumbo.py new file +0 -0
new file mode 100644
index 0000000..9f56a87

client/tests/kvm/tests/jumbo.py line 9

Quoted 9 lines
client/tests/kvm/tests/jumbo.py new file +9 -0
@@ -0,0 +1,133 @@
+import os, re, logging, commands, time, random
+from autotest_lib.client.common_lib import error
+import kvm_subprocess, kvm_test_utils, kvm_utils, kvm_net_utils
+
+def run_jumbo(test, params, env):
+    """
+    Test the RX jumbo frame function of vnics:
+    1) boot the vm
+    2) change the MTU of guest nics and host taps depends on the nic model

^ I think you meant 'depending on the nic model'

client/tests/kvm/tests/jumbo.py line 14

Quoted 5 lines
client/tests/kvm/tests/jumbo.py new file +5 -0
+    3) add the static arp entry for guest nic
+    4) wait for the MTU ok
+    5) verify the patch mtu using ping
+    6) ping the guest with large frames
+    7) increament size ping

^ typo, increment

client/tests/kvm/tests/jumbo.py line 17

Quoted 3 lines
client/tests/kvm/tests/jumbo.py new file +3 -0
+    8) flood ping the guest with large frames
+    9) verify the path mtu
+    10) revocer the mtu

^ maybe a typo, restore?

Quoted 9 lines
client/tests/kvm/tests/jumbo.py new file +9 -0
+
+    @param test: kvm test object
+    @param params: Dictionary with the test parameters
+    @param env: Dictionary with test environment.
+    """
+
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    session = kvm_test_utils.wait_for_login(vm)
+    mtu = params.get("mtu", "1500")

client/tests/kvm/tests/jumbo.py line 36

Quoted 10 lines
client/tests/kvm/tests/jumbo.py new file +10 -0
+    flood_time = params.get("flood_time", "300")
+    max_icmp_pkt_size = int(mtu) - 28
+
+    ifname = vm.get_ifname(0)
+    ip = vm.get_address(0)
+    if ip is None:
+        raise error.TestError("Could not get the ip address")
+
+    try:
+        # Environment preparartion

^ typo, preparation

Quoted 16 lines
client/tests/kvm/tests/jumbo.py new file +16 -0
+        ethname = kvm_net_utils.get_linux_ifname(session, vm.get_macaddr(0))
+
+        logging.info("Changing the mtu of guest ...")
+        guest_mtu_cmd = "ifconfig %s mtu %s" % (ethname , mtu)
+        s, o = session.get_command_status_output(guest_mtu_cmd)
+        if s != 0:
+            logging.error(o)
+            raise error.TestError("Fail to set the mtu of guest nic: %s"
+                                  % ethname)
+
+        logging.info("Chaning the mtu of host tap ...")
+        host_mtu_cmd = "ifconfig %s mtu %s" % (ifname, mtu)
+        s, o = commands.getstatusoutput(host_mtu_cmd)
+        if s != 0:
+            raise error.TestError("Fail to set the mtu of %s" % ifname)
+

client/tests/kvm/tests/jumbo.py line 62

Quoted 10 lines
client/tests/kvm/tests/jumbo.py new file +10 -0
+        logging.info("Add a temporary static arp entry ...")
+        arp_add_cmd = "arp -s %s %s -i %s" % (ip, vm.get_macaddr(0), ifname)
+        s, o = commands.getstatusoutput(arp_add_cmd)
+        if s != 0 :
+            raise error.TestError("Fail to add temporary arp entry")
+
+        def is_mtu_ok():
+            s, o = kvm_net_utils.ping(ip, 1, interface = ifname,
+                                      packetsize = max_icmp_pkt_size,
+                                      hint = "do", timeout = 2)

^ please fix the default argument attribution, here and all subsequent calls

client/tests/kvm/tests/jumbo.py line 66

Quoted 4 lines
client/tests/kvm/tests/jumbo.py new file +4 -0
+            if s != 0:
+                return False
+            else:
+                return True

^ Here you could simple use return s != 0

Quoted 19 lines
client/tests/kvm/tests/jumbo.py new file +19 -0
+
+        def verify_mtu():
+            logging.info("Verify the path mtu")
+            s, o = kvm_net_utils.ping(ip, 10, interface = ifname,
+                                      packetsize = max_icmp_pkt_size,
+                                      hint = "do", timeout = 15)
+            if s != 0 :
+                logging.error(o)
+                raise error.TestFail("Path MTU is not as expected")
+            if kvm_net_utils.get_loss_ratio(o) != 0:
+                logging.error(o)
+                raise error.TestFail("Packet loss ratio during mtu verification"
+                                     " is not zero")
+
+        def flood_ping():
+            logging.info("Flood with large frames")
+            kvm_net_utils.ping(ip, interface = ifname,
+                               packetsize = max_icmp_pkt_size,
+                               flood = True, timeout = float(flood_time))

client/tests/kvm/tests/jumbo.py line 95

Quoted 10 lines
client/tests/kvm/tests/jumbo.py new file +10 -0
+
+        def large_frame_ping(count = 100):
+            logging.info("Large frame ping")
+            s, o = kvm_net_utils.ping(ip, count, interface = ifname,
+                                      packetsize = max_icmp_pkt_size,
+                                      timeout = float(count) * 2)
+            ratio = kvm_net_utils.get_loss_ratio(o)
+            if ratio != 0:
+                raise error.TestFail("Loss ratio of large frame ping is %s" \
+                                     % ratio)

^ No need to use the backslash to end line here

Quoted 5 lines
client/tests/kvm/tests/jumbo.py new file +5 -0
+        def size_increase_ping(step = random.randrange(90, 110)):
+            logging.info("Size increase ping")
+            for size in range(0, max_icmp_pkt_size + 1, step):
+                logging.info("Ping %s with size %s" % (ip, size))
+                s, o = kvm_net_utils.ping(ip, 1, interface = ifname,

client/tests/kvm/tests/jumbo.py line 111

Quoted 10 lines
client/tests/kvm/tests/jumbo.py new file +10 -0
+                                          packetsize = size,
+                                          hint = "do", timeout = 1)
+                if s != 0:
+                    s, o = kvm_net_utils.ping(ip, 10, interface = ifname,
+                                              packetsize = size,
+                                              adaptive = True, hint = "do",
+                                              timeout = 20)
+                    if kvm_net_utils.get_loss_ratio(o) > 50:
+                        raise error.TestFail("ping loss ratio is greater "
+                                             "than 50% for size %s" % size)

^ How do we stablish 50% as an acceptable packet loss ratio for this test? Seems high to me, but I might be missing something

client/tests/kvm/tests/jumbo.py line 114

Quoted 3 lines
client/tests/kvm/tests/jumbo.py new file +3 -0
+
+        logging.info("Waiting for the MTU to be OK")
+        # Wait for the MTU is OK

^ Unneeded comment, as we have an info message stating the same thing

Quoted 64 lines
client/tests/kvm/tests/jumbo.py new file +19 -0
+        if not kvm_utils.wait_for(is_mtu_ok, 10, 0, 1):
+            logging.debug(commands.getoutput("ifconfig -a"))
+            raise error.TestError("10 seconds elapsed while the mtu is not"
+                                  " as expected")
+
+        # Functional Test
+        verify_mtu()
+        large_frame_ping()
+        size_increase_ping()
+
+        # Stress test
+        flood_ping()
+        verify_mtu()
+
+    finally:
+        # Environment clean
+        session.close()
+        logging.info("Removing the temporary arp entry")
+        commands.getstatusoutput("arp -d %s -i %s" % (ip, ifname))
client/tests/kvm/tests_base.cfg.sample +10 -2
index 4f58dc0..7f7b56a 100644
@@ -354,6 +354,9 @@ variants:
         counts = 100
         flood_minutes = 10
 
+    - jumbo: install setup unattended_install.cdrom
+        type = jumbo
+
     - physical_resources_check: install setup unattended_install.cdrom
         type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'
@@ -489,10 +492,16 @@ variants:
 variants:
     - @rtl8139:
         nic_model = rtl8139
+        jumbo:
+            mtu = 1500
     - e1000:
         nic_model = e1000
+        jumbo:
+            mtu = 16110
     - virtio:
         nic_model = virtio
+        jumbo:
+            mtu = 65520
 
 
 # Guests
@@ -1024,7 +1033,7 @@ variants:
 
     # Windows section
     - @Windows:
-        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks)
+        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo
         shutdown_command = shutdown /s /f /t 0
         reboot_command = shutdown /r /f /t 0
         status_test_command = echo %errorlevel%
 
--

To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#20 lore raw

Re: [RFC PATCH 05/14] KVM-test: Add a subtest jumbo

Quoted 148 lines · Lucas Meneghel Rodrigues
Quoted 14 lines · Amos Kong

According to different nic model set different MTU for it. And ping from guest to host, to see whether tested size can be received by host.

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/tests/jumbo.py new file +0 -0
new file mode 100644
index 0000000..9f56a87

client/tests/kvm/tests/jumbo.py line 9

Quoted 9 lines
client/tests/kvm/tests/jumbo.py new file +9 -0
@@ -0,0 +1,133 @@
+import os, re, logging, commands, time, random
+from autotest_lib.client.common_lib import error
+import kvm_subprocess, kvm_test_utils, kvm_utils, kvm_net_utils
+
+def run_jumbo(test, params, env):
+    """
+    Test the RX jumbo frame function of vnics:
+    1) boot the vm
+    2) change the MTU of guest nics and host taps depends on the nic model

^ I think you meant 'depending on the nic model'

client/tests/kvm/tests/jumbo.py line 14

Quoted 5 lines
client/tests/kvm/tests/jumbo.py new file +5 -0
+    3) add the static arp entry for guest nic
+    4) wait for the MTU ok
+    5) verify the patch mtu using ping
+    6) ping the guest with large frames
+    7) increament size ping

^ typo, increment

client/tests/kvm/tests/jumbo.py line 17

Quoted 3 lines
client/tests/kvm/tests/jumbo.py new file +3 -0
+    8) flood ping the guest with large frames
+    9) verify the path mtu
+    10) revocer the mtu

^ maybe a typo, restore?

Quoted 9 lines
client/tests/kvm/tests/jumbo.py new file +9 -0
+
+    @param test: kvm test object
+    @param params: Dictionary with the test parameters
+    @param env: Dictionary with test environment.
+    """
+
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    session = kvm_test_utils.wait_for_login(vm)
+    mtu = params.get("mtu", "1500")

client/tests/kvm/tests/jumbo.py line 36

Quoted 10 lines
client/tests/kvm/tests/jumbo.py new file +10 -0
+    flood_time = params.get("flood_time", "300")
+    max_icmp_pkt_size = int(mtu) - 28
+
+    ifname = vm.get_ifname(0)
+    ip = vm.get_address(0)
+    if ip is None:
+        raise error.TestError("Could not get the ip address")
+
+    try:
+        # Environment preparartion

^ typo, preparation

Quoted 16 lines
client/tests/kvm/tests/jumbo.py new file +16 -0
+        ethname = kvm_net_utils.get_linux_ifname(session, vm.get_macaddr(0))
+
+        logging.info("Changing the mtu of guest ...")
+        guest_mtu_cmd = "ifconfig %s mtu %s" % (ethname , mtu)
+        s, o = session.get_command_status_output(guest_mtu_cmd)
+        if s != 0:
+            logging.error(o)
+            raise error.TestError("Fail to set the mtu of guest nic: %s"
+                                  % ethname)
+
+        logging.info("Chaning the mtu of host tap ...")
+        host_mtu_cmd = "ifconfig %s mtu %s" % (ifname, mtu)
+        s, o = commands.getstatusoutput(host_mtu_cmd)
+        if s != 0:
+            raise error.TestError("Fail to set the mtu of %s" % ifname)
+

client/tests/kvm/tests/jumbo.py line 62

Quoted 10 lines
client/tests/kvm/tests/jumbo.py new file +10 -0
+        logging.info("Add a temporary static arp entry ...")
+        arp_add_cmd = "arp -s %s %s -i %s" % (ip, vm.get_macaddr(0), ifname)
+        s, o = commands.getstatusoutput(arp_add_cmd)
+        if s != 0 :
+            raise error.TestError("Fail to add temporary arp entry")
+
+        def is_mtu_ok():
+            s, o = kvm_net_utils.ping(ip, 1, interface = ifname,
+                                      packetsize = max_icmp_pkt_size,
+                                      hint = "do", timeout = 2)

^ please fix the default argument attribution, here and all subsequent calls

client/tests/kvm/tests/jumbo.py line 66

Quoted 4 lines
client/tests/kvm/tests/jumbo.py new file +4 -0
+            if s != 0:
+                return False
+            else:
+                return True

^ Here you could simple use return s != 0

Quoted 19 lines
client/tests/kvm/tests/jumbo.py new file +19 -0
+
+        def verify_mtu():
+            logging.info("Verify the path mtu")
+            s, o = kvm_net_utils.ping(ip, 10, interface = ifname,
+                                      packetsize = max_icmp_pkt_size,
+                                      hint = "do", timeout = 15)
+            if s != 0 :
+                logging.error(o)
+                raise error.TestFail("Path MTU is not as expected")
+            if kvm_net_utils.get_loss_ratio(o) != 0:
+                logging.error(o)
+                raise error.TestFail("Packet loss ratio during mtu verification"
+                                     " is not zero")
+
+        def flood_ping():
+            logging.info("Flood with large frames")
+            kvm_net_utils.ping(ip, interface = ifname,
+                               packetsize = max_icmp_pkt_size,
+                               flood = True, timeout = float(flood_time))

client/tests/kvm/tests/jumbo.py line 95

Quoted 10 lines
client/tests/kvm/tests/jumbo.py new file +10 -0
+
+        def large_frame_ping(count = 100):
+            logging.info("Large frame ping")
+            s, o = kvm_net_utils.ping(ip, count, interface = ifname,
+                                      packetsize = max_icmp_pkt_size,
+                                      timeout = float(count) * 2)
+            ratio = kvm_net_utils.get_loss_ratio(o)
+            if ratio != 0:
+                raise error.TestFail("Loss ratio of large frame ping is %s" \
+                                     % ratio)

^ No need to use the backslash to end line here

Quoted 5 lines
client/tests/kvm/tests/jumbo.py new file +5 -0
+        def size_increase_ping(step = random.randrange(90, 110)):
+            logging.info("Size increase ping")
+            for size in range(0, max_icmp_pkt_size + 1, step):
+                logging.info("Ping %s with size %s" % (ip, size))
+                s, o = kvm_net_utils.ping(ip, 1, interface = ifname,

client/tests/kvm/tests/jumbo.py line 111

Quoted 10 lines
client/tests/kvm/tests/jumbo.py new file +10 -0
+                                          packetsize = size,
+                                          hint = "do", timeout = 1)
+                if s != 0:
+                    s, o = kvm_net_utils.ping(ip, 10, interface = ifname,
+                                              packetsize = size,
+                                              adaptive = True, hint = "do",
+                                              timeout = 20)
+                    if kvm_net_utils.get_loss_ratio(o) > 50:
+                        raise error.TestFail("ping loss ratio is greater "
+                                             "than 50% for size %s" % size)
Quoted 2 lines

^ How do we stablish 50% as an acceptable packet loss ratio for this test? Seems high to me, but I might be missing something

It's just a 'experimental' standard, we can also make it can be setup in config file.

(missing some email, reply by personal Email :)

author
#21 lore raw

RFC PATCH 6/14 KVM-test: Add basic file transfer test

In reply to Amos Kong

This test is the basic test of transfering file between host and guest. Try to transfer a large file from host to guest, and transfer it back to host, then compare the files by diff command.
The default file size is 4000M, scp timeout is 1000s. It means if the average speed is less than 4M/s, this test will be fail.
We can extend this test by using another disk later, then we can transfer larger files without the limit of first disk size.

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/tests/file_transfer.py new file +54 -0
new file mode 100644
index 0000000..a20e62e
@@ -0,0 +1,54 @@
+import logging, commands
+from autotest_lib.client.common_lib import error
+import kvm_utils, kvm_test_utils
+
+def run_file_transfer(test, params, env):
+    """
+    Test ethrnet device function by ethtool
+
+    1) Boot up a virtual machine
+    2) Create a large file by dd on host
+    3) Copy this file from host to guest
+    4) Copy this file from guest to host
+    5) Check if file transfers good
+
+    @param test: Kvm test object
+    @param params: Dictionary with the test parameters.
+    @param env: Dictionary with test environment.
+    """
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    timeout=int(params.get("login_timeout", 360))
+    logging.info("Trying to log into guest '%s' by serial", vm.name)
+    session = kvm_utils.wait_for(lambda: vm.serial_login(),
+                                 timeout, 0, step=2)
+    if not session:
+        raise error.TestFail("Could not log into guest '%s'" % vm.name)
+
+    dir = test.tmpdir
+    scp_timeout = int(params.get("scp_timeout"))
+    cmd = "dd if=/dev/urandom of=%s/a.out bs=1M count=%d" %  (dir, int(
+                                         params.get("filesize", 4000)))
+    try:
+        logging.info("Create file by dd command on host, cmd: %s" % cmd)
+        s, o = commands.getstatusoutput(cmd)
+        if s != 0:
+            raise error.TestError("Fail to create file, output:%s" % o)
+
+        logging.info("Transfer file from host to guest")
+        if not vm.copy_files_to("%s/a.out" % dir, "/tmp/b.out",
+                                                        timeout=scp_timeout):
+            raise error.TestFail("Fail to transfer file from host to guest")
+
+        logging.info("Transfer file from guest to host")
+        if not vm.copy_files_from("/tmp/b.out", "%s/c.out" % dir,
+                                                        timeout=scp_timeout):
+            raise error.TestFail("Fail to transfer file from guest to host")
+
+        logging.debug(commands.getoutput("ls -l %s/[ac].out" % dir))
+        s, o = commands.getstatusoutput("diff %s/a.out %s/c.out" % (dir, dir))
+        if s != 0:
+            raise error.TestFail("File changed after transfer. Output:%s" % o)
+    finally:
+        session.get_command_status("rm -f /tmp/b.out")
+        commands.getoutput("rm -f %s/[ac].out" % dir)
+        session.close()
client/tests/kvm/tests_base.cfg.sample +6 -1
index 7f7b56a..872674e 100644
@@ -357,6 +357,11 @@ variants:
     - jumbo: install setup unattended_install.cdrom
         type = jumbo
 
+    - file_transfer: install setup unattended_install.cdrom
+        type = file_transfer
+        filesize = 4000
+        scp_timeout = 1000
+
     - physical_resources_check: install setup unattended_install.cdrom
         type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'
@@ -1033,7 +1038,7 @@ variants:
 
     # Windows section
     - @Windows:
-        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo
+        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer
         shutdown_command = shutdown /s /f /t 0
         reboot_command = shutdown /r /f /t 0
         status_test_command = echo %errorlevel%
#22 lore raw

Re: [Qemu-devel] [RFC PATCH 06/14] KVM-test: Add basic file transfer test

Quoted 5 lines · Amos Kong

This test is the basic test of transfering file between host and guest. Try to transfer a large file from host to guest, and transfer it back to host, then compare the files by diff command.
The default file size is 4000M, scp timeout is 1000s. It means if the average speed is less than 4M/s, this test will be fail.

^ About this average 4MB/s throughput, is there some sort of agreement/standard I'm not aware of?

Quoted 26 lines

We can extend this test by using another disk later, then we can transfer larger files without the limit of first disk size.

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/tests/file_transfer.py new file +13 -0
new file mode 100644
index 0000000..a20e62e
@@ -0,0 +1,54 @@
+import logging, commands
+from autotest_lib.client.common_lib import error
+import kvm_utils, kvm_test_utils
+
+def run_file_transfer(test, params, env):
+    """
+    Test ethrnet device function by ethtool
+
+    1) Boot up a virtual machine
+    2) Create a large file by dd on host
+    3) Copy this file from host to guest
+    4) Copy this file from guest to host
+    5) Check if file transfers good

client/tests/kvm/tests/file_transfer.py line 23

Quoted 10 lines
client/tests/kvm/tests/file_transfer.py new file +10 -0
+
+    @param test: Kvm test object
+    @param params: Dictionary with the test parameters.
+    @param env: Dictionary with test environment.
+    """
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    timeout=int(params.get("login_timeout", 360))
+    logging.info("Trying to log into guest '%s' by serial", vm.name)
+    session = kvm_utils.wait_for(lambda: vm.serial_login(),
+                                 timeout, 0, step=2)

^ I assume the serial session is being used here to avoid interfering with the bandwidth required for the test to complete as little as possible, right?

client/tests/kvm/tests/file_transfer.py line 35

Quoted 12 lines
client/tests/kvm/tests/file_transfer.py new file +12 -0
+    if not session:
+        raise error.TestFail("Could not log into guest '%s'" % vm.name)
+
+    dir = test.tmpdir
+    scp_timeout = int(params.get("scp_timeout"))
+    cmd = "dd if=/dev/urandom of=%s/a.out bs=1M count=%d" %  (dir, int(
+                                         params.get("filesize", 4000)))
+    try:
+        logging.info("Create file by dd command on host, cmd: %s" % cmd)
+        s, o = commands.getstatusoutput(cmd)
+        if s != 0:
+            raise error.TestError("Fail to create file, output:%s" % o)

^ I've seen this throughout the patchseries, the use of functions present on commands rather than utils.system, or utils.run. The former are preferable on autotest tests, that's important to have in mind. Would you consider changing those calls to the autotest API functions?

Quoted 4 lines
client/tests/kvm/tests/file_transfer.py new file +4 -0
+        logging.info("Transfer file from host to guest")
+        if not vm.copy_files_to("%s/a.out" % dir, "/tmp/b.out",
+                                                        timeout=scp_timeout):
+            raise error.TestFail("Fail to transfer file from host to guest")

client/tests/kvm/tests/file_transfer.py line 50

Quoted 10 lines
client/tests/kvm/tests/file_transfer.py new file +10 -0
+
+        logging.info("Transfer file from guest to host")
+        if not vm.copy_files_from("/tmp/b.out", "%s/c.out" % dir,
+                                                        timeout=scp_timeout):
+            raise error.TestFail("Fail to transfer file from guest to host")
+
+        logging.debug(commands.getoutput("ls -l %s/[ac].out" % dir))
+        s, o = commands.getstatusoutput("diff %s/a.out %s/c.out" % (dir, dir))
+        if s != 0:
+            raise error.TestFail("File changed after transfer. Output:%s" % o)

^ It seems faster to use md5 or sha1 to verify the integrity of the files rather than diff. Unless there's a good reason to use diff, I'd like you to change it to one of the programs mentioned above.

Quoted 31 lines
client/tests/kvm/tests/file_transfer.py new file +4 -0
+    finally:
+        session.get_command_status("rm -f /tmp/b.out")
+        commands.getoutput("rm -f %s/[ac].out" % dir)
+        session.close()
client/tests/kvm/tests_base.cfg.sample +6 -1
index 7f7b56a..872674e 100644
@@ -357,6 +357,11 @@ variants:
     - jumbo: install setup unattended_install.cdrom
         type = jumbo
 
+    - file_transfer: install setup unattended_install.cdrom
+        type = file_transfer
+        filesize = 4000
+        scp_timeout = 1000
+
     - physical_resources_check: install setup unattended_install.cdrom
         type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'
@@ -1033,7 +1038,7 @@ variants:
 
     # Windows section
     - @Windows:
-        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo
+        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer
         shutdown_command = shutdown /s /f /t 0
         reboot_command = shutdown /r /f /t 0
         status_test_command = echo %errorlevel%
#23 lore raw

Re: [Autotest] [Qemu-devel] [RFC PATCH 06/14] KVM-test: Add basic file transfer test

Quoted 9 lines · Lucas Meneghel Rodrigues
Quoted 5 lines · Amos Kong

This test is the basic test of transfering file between host and guest. Try to transfer a large file from host to guest, and transfer it back to host, then compare the files by diff command.
The default file size is 4000M, scp timeout is 1000s. It means if the average speed is less than 4M/s, this test will be fail.

^ About this average 4MB/s throughput, is there some sort of agreement/standard I'm not aware of?

Also an 'experimental' standard.
The most important purpose is not check the transfer result, but catch the bug during file transfer.
If case fail of transfer timeout, we need try to augment the timeout, and compare the test result.

Quoted 32 lines
Quoted 32 lines

We can extend this test by using another disk later, then we can transfer larger files without the limit of first disk size.

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/tests/file_transfer.py new file +19 -0
new file mode 100644
index 0000000..a20e62e
@@ -0,0 +1,54 @@
+import logging, commands
+from autotest_lib.client.common_lib import error
+import kvm_utils, kvm_test_utils
+
+def run_file_transfer(test, params, env):
+    """
+    Test ethrnet device function by ethtool
+
+    1) Boot up a virtual machine
+    2) Create a large file by dd on host
+    3) Copy this file from host to guest
+    4) Copy this file from guest to host
+    5) Check if file transfers good
+
+    @param test: Kvm test object
+    @param params: Dictionary with the test parameters.
+    @param env: Dictionary with test environment.
+    """
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
Quoted 8 lines

client/tests/kvm/tests/file_transfer.py line 23

Quoted 4 lines
client/tests/kvm/tests/file_transfer.py new file +4 -0
+    timeout=int(params.get("login_timeout", 360))
+    logging.info("Trying to log into guest '%s' by serial", vm.name)
+    session = kvm_utils.wait_for(lambda: vm.serial_login(),
+                                 timeout, 0, step=2)

^ I assume the serial session is being used here to avoid interfering with the bandwidth required for the test to complete as little as possible, right?

Just want to split the control connection and data connection, if network is down, some clean work would not be done.
But there is only one serial console session per VM, when we use serial_login, serial console log would be disabled.

Quoted 13 lines

client/tests/kvm/tests/file_transfer.py line 35

Quoted 12 lines
client/tests/kvm/tests/file_transfer.py new file +12 -0
+    if not session:
+        raise error.TestFail("Could not log into guest '%s'" % vm.name)
+
+    dir = test.tmpdir
+    scp_timeout = int(params.get("scp_timeout"))
+    cmd = "dd if=/dev/urandom of=%s/a.out bs=1M count=%d" %  (dir, int(
+                                         params.get("filesize", 4000)))
+    try:
+        logging.info("Create file by dd command on host, cmd: %s" % cmd)
+        s, o = commands.getstatusoutput(cmd)
+        if s != 0:
+            raise error.TestError("Fail to create file, output:%s" % o)
Quoted 4 lines

^ I've seen this throughout the patchseries, the use of functions present on commands rather than utils.system, or utils.run. The former are preferable on autotest tests, that's important to have in mind. Would you consider changing those calls to the autotest API functions?

ok, I will try to use autotest API.

Quoted 10 lines
Quoted 10 lines
client/tests/kvm/tests/file_transfer.py new file +10 -0
+        logging.info("Transfer file from host to guest")
+        if not vm.copy_files_to("%s/a.out" % dir, "/tmp/b.out",
+                                                        timeout=scp_timeout):
+            raise error.TestFail("Fail to transfer file from host to guest")
+
+        logging.info("Transfer file from guest to host")
+        if not vm.copy_files_from("/tmp/b.out", "%s/c.out" % dir,
+                                                        timeout=scp_timeout):
+            raise error.TestFail("Fail to transfer file from guest to host")
+
Quoted 8 lines

client/tests/kvm/tests/file_transfer.py line 50

Quoted 4 lines
client/tests/kvm/tests/file_transfer.py new file +4 -0
+        logging.debug(commands.getoutput("ls -l %s/[ac].out" % dir))
+        s, o = commands.getstatusoutput("diff %s/a.out %s/c.out" % (dir, dir))
+        if s != 0:
+            raise error.TestFail("File changed after transfer. Output:%s" % o)

^ It seems faster to use md5 or sha1 to verify the integrity of the files rather than diff. Unless there's a good reason to use diff, I'd like you to change it to one of the programs mentioned above.

I would change it to md5.

Quoted 38 lines
Quoted 24 lines
client/tests/kvm/tests/file_transfer.py new file +4 -0
+    finally:
+        session.get_command_status("rm -f /tmp/b.out")
+        commands.getoutput("rm -f %s/[ac].out" % dir)
+        session.close()
client/tests/kvm/tests_base.cfg.sample +0 -0
index 7f7b56a..872674e 100644
@@ -357,6 +357,11 @@ variants:

     - jumbo: install setup unattended_install.cdrom type = jumbo

+    - file_transfer: install setup unattended_install.cdrom
+        type = file_transfer
+        filesize = 4000
+        scp_timeout = 1000
+
     - physical_resources_check: install setup unattended_install.cdrom type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'

client/tests/kvm/tests_base.cfg.sample +0 -0
@@ -1033,7 +1038,7 @@ variants:
 
      # Windows section

     - @Windows:

Quoted 7 lines
-        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo
+        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer

         shutdown_command = shutdown /s /f /t 0
         reboot_command = shutdown /r /f /t 0
         status_test_command = echo %errorlevel%

_______________________________________________
Autotest mailing list
[email protected]
http://test.kernel.org/cgi-bin/mailman/listinfo/autotest

author
#24 lore raw

RFC PATCH 7/14 KVM-test: Add a subtest of load/unload nic driver

In reply to Amos Kong

Repeatedly load/unload nic driver, try to transfer file between guest and host by threads at the same time, and check the md5sum.

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/tests/nicdriver_unload.py new file +128 -0
new file mode 100644
index 0000000..22f9f44
@@ -0,0 +1,128 @@
+import logging, commands, threading, re, os
+from autotest_lib.client.common_lib import error
+import kvm_utils, kvm_test_utils, kvm_net_utils
+
+def run_nicdriver_unload(test, params, env):
+    """
+    Test nic driver
+
+    1) Boot a vm
+    2) Get the nic driver name
+    3) Repeatedly unload/load nic driver
+    4) Multi-session TCP transfer on test interface
+    5) Check the test interface should still work
+
+    @param test: KVM test object
+    @param params: Dictionary with the test parameters
+    @param env: Dictionary with test environment.
+    """
+    timeout = int(params.get("login_timeout", 360))
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    session = kvm_test_utils.wait_for_login(vm, timeout=timeout)
+    logging.info("Trying to log into guest '%s' by serial", vm.name)
+    session2 = kvm_utils.wait_for(lambda: vm.serial_login(),
+                                  timeout, 0, step=2)
+    if not session2:
+        raise error.TestFail("Could not log into guest '%s'" % vm.name)
+
+    ethname = kvm_net_utils.get_linux_ifname(session, vm.get_macaddr(0))
+    try:
+        # FIXME: Try three waies to get nic driver name, because the
+        # modprobe.conf is dropped in latest system, and ethtool method is not
+        # supported by virtio_nic.
+
+        output = session.get_command_output("cat /etc/modprobe.conf")
+        driver = re.findall(r'%s (\w+)' % ethname,output)
+        if not driver:
+            output = session.get_command_output("ethtool -i %s" % ethname)
+            driver = re.findall(r'driver: (\w+)', output)
+        if not driver:
+            output = session.get_command_output("lspci -k")
+            driver = re.findall("Ethernet controller.*\n.*\n.*Kernel driver"
+                                " in use: (\w+)", output)
+        driver = driver[0]
+    except IndexError:
+        raise error.TestError("Could not find driver name")
+
+    logging.info("driver is %s" % driver)
+
+    class ThreadScp(threading.Thread):
+        def run(self):
+            remote_file = '/tmp/' + self.getName()
+            file_list.append(remote_file)
+            ret = vm.copy_files_to(file_name, remote_file, timeout=scp_timeout)
+            logging.debug("Copy result of %s: %s" % (remote_file, ret))
+
+    def compare(origin_file, receive_file):
+        cmd = "md5sum %s"
+        output1 = commands.getstatusoutput(cmd % origin_file)[1].strip()
+        check_sum1 = output1.split()[0]
+        s, output2 = session.get_command_status_output(cmd % receive_file)
+        if s != 0:
+            logging.error("Could not get md5sum of receive_file")
+            return False
+        check_sum2 = output2.strip().split()[0]
+        logging.debug("origin: %s, receive: %s" % (check_sum1, check_sum2))
+        if check_sum1 != check_sum2:
+            logging.error("md5sum doesn't match")
+            return False
+        return True
+
+    #produce sized file in host
+    file_size = params.get("file_size")
+    file_name = "/tmp/nicdriver_unload_file"
+    cmd = "dd if=/dev/urandom of=%s bs=%sM count=1"
+    s, o = commands.getstatusoutput(cmd % (file_name, file_size))
+    if s != 0:
+        raise error.TestFail("Fail to create file by dd")
+
+    connect_time = params.get("connect_time")
+    scp_timeout = int(params.get("scp_timeout"))
+    thread_num = int(params.get("thread_num"))
+    file_list = []
+
+    unload_load_cmd = "sleep %s && ifconfig %s down && modprobe -r %s && "
+    unload_load_cmd += "sleep 1 && modprobe %s && ifconfig %s up"
+    unload_load_cmd = unload_load_cmd % (connect_time, ethname, driver,
+                                         driver, ethname)
+    pid = os.fork()
+    if pid != 0:
+        logging.info("unload/load nic driver repeatedly in guest...")
+        while True:
+            logging.debug("Try to unload/load nic drive once")
+            if session2.get_command_status(unload_load_cmd, timeout=120) != 0:
+                session.get_command_output("rm -rf /tmp/Thread-*")
+                raise error.TestFail("Unload/load nic driver failed")
+            pid, s = os.waitpid(pid, os.WNOHANG)
+            status = os.WEXITSTATUS(s)
+            if (pid, status) != (0, 0):
+                logging.debug("Child process ending")
+                break
+    else:
+        logging.info("Multi-session tcp data transfer")
+        threads = []
+        for i in range(thread_num):
+            t = ThreadScp()
+            t.start()
+            threads.append(t)
+        for t in threads:
+            t.join(timeout = scp_timeout)
+        os._exit(0)
+
+    session2.close()
+
+    try:
+        logging.info("Check md5sum for received files in multi-session")
+        for f in file_list:
+            if not compare(file_name, f):
+                raise error.TestFail("Fail to compare (guest)file %s" % f)
+
+        logging.info("Test nic function after load/unload")
+        if not vm.copy_files_to(file_name, file_name):
+            raise error.TestFail("Fail to copy file from host to guest")
+        if not compare(file_name, file_name):
+            raise error.TestFail("Test nic function after load/unload fail")
+
+    finally:
+        session.get_command_output("rm -rf /tmp/Thread-*")
+        session.close()
client/tests/kvm/tests_base.cfg.sample +9 -1
index 872674e..03d15c0 100644
@@ -362,6 +362,14 @@ variants:
         filesize = 4000
         scp_timeout = 1000
 
+    - nicdriver_unload:  install setup unattended_install.cdrom
+        type = nicdriver_unload
+        nic_mode = tap
+        file_size = 100
+        connect_time = 4
+        scp_timeout = 300
+        thread_num = 10
+
     - physical_resources_check: install setup unattended_install.cdrom
         type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'
@@ -1038,7 +1046,7 @@ variants:
 
     # Windows section
     - @Windows:
-        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer
+        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer nicdriver_unload
         shutdown_command = shutdown /s /f /t 0
         reboot_command = shutdown /r /f /t 0
         status_test_command = echo %errorlevel%
#25 lore raw

Re: [Qemu-devel] [RFC PATCH 07/14] KVM-test: Add a subtest of load/unload nic driver

Quoted 48 lines · Amos Kong

Repeatedly load/unload nic driver, try to transfer file between guest and host by threads at the same time, and check the md5sum.

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/tests/nicdriver_unload.py new file +35 -0
new file mode 100644
index 0000000..22f9f44
@@ -0,0 +1,128 @@
+import logging, commands, threading, re, os
+from autotest_lib.client.common_lib import error
+import kvm_utils, kvm_test_utils, kvm_net_utils
+
+def run_nicdriver_unload(test, params, env):
+    """
+    Test nic driver
+
+    1) Boot a vm
+    2) Get the nic driver name
+    3) Repeatedly unload/load nic driver
+    4) Multi-session TCP transfer on test interface
+    5) Check the test interface should still work
+
+    @param test: KVM test object
+    @param params: Dictionary with the test parameters
+    @param env: Dictionary with test environment.
+    """
+    timeout = int(params.get("login_timeout", 360))
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    session = kvm_test_utils.wait_for_login(vm, timeout=timeout)
+    logging.info("Trying to log into guest '%s' by serial", vm.name)
+    session2 = kvm_utils.wait_for(lambda: vm.serial_login(),
+                                  timeout, 0, step=2)
+    if not session2:
+        raise error.TestFail("Could not log into guest '%s'" % vm.name)
+
+    ethname = kvm_net_utils.get_linux_ifname(session, vm.get_macaddr(0))
+    try:
+        # FIXME: Try three waies to get nic driver name, because the
+        # modprobe.conf is dropped in latest system, and ethtool method is not
+        # supported by virtio_nic.
+
+        output = session.get_command_output("cat /etc/modprobe.conf")
+        driver = re.findall(r'%s (\w+)' % ethname,output)

client/tests/kvm/tests/nicdriver_unload.py line 45

Quoted 10 lines
client/tests/kvm/tests/nicdriver_unload.py new file +10 -0
+        if not driver:
+            output = session.get_command_output("ethtool -i %s" % ethname)
+            driver = re.findall(r'driver: (\w+)', output)
+        if not driver:
+            output = session.get_command_output("lspci -k")
+            driver = re.findall("Ethernet controller.*\n.*\n.*Kernel driver"
+                                " in use: (\w+)", output)
+        driver = driver[0]
+    except IndexError:
+        raise error.TestError("Could not find driver name")

^ About this whole block where there's an attempt to discover what the driver name is. The methods listed there are not reliable (depends on ethtool being installed, lspci -k not allways will list the kernel driver in use, /etc/modprobe.conf will not be present on more recent distributions). Instead of these methods, why don't we have a variant for this test on each linux distro definition block, with the appropriate driver names? It'd be something along the lines:

                    - 13.64:
                        image_name = f13-64
                        cdrom_cd1 = linux/Fedora-13-x86_64-DVD.iso
                        md5sum = 6fbae6379cf27f36e1f2c7827ba7dc35
                        md5sum_1m = 68821b9de4d3b5975d6634334e7f47a6
                        unattended_install:
                            unattended_file = unattended/Fedora-13.ks
                            tftp = images/f13-64/tftpboot
                            floppy = images/f13-64/floppy.img
                        nicdriver_unload:
                            e1000:
                                driver = e1000c
                            virtio:
                                driver = virtio_net
                            rtl8139:
                                driver = rtl8939

I believe it's safer than having to rely on the above methods.

client/tests/kvm/tests/nicdriver_unload.py line 54

Quoted 9 lines
client/tests/kvm/tests/nicdriver_unload.py new file +9 -0
+
+    logging.info("driver is %s" % driver)
+
+    class ThreadScp(threading.Thread):
+        def run(self):
+            remote_file = '/tmp/' + self.getName()
+            file_list.append(remote_file)
+            ret = vm.copy_files_to(file_name, remote_file, timeout=scp_timeout)
+            logging.debug("Copy result of %s: %s" % (remote_file, ret))

^ Here it'd be worth to have 2 debug messages,

if ret:
logging.debug("File %s was transfered successfuly", remote_file)
else:
logging.debug("Failed to transfer file %s", remote_file)

client/tests/kvm/tests/nicdriver_unload.py line 58

Quoted 3 lines
client/tests/kvm/tests/nicdriver_unload.py new file +3 -0
+    def compare(origin_file, receive_file):
+        cmd = "md5sum %s"
+        output1 = commands.getstatusoutput(cmd % origin_file)[1].strip()

^ If we are only interested on the output, getoutput() could be used. But in this case we care whether md5 succeeded or not, so better to do appropriate return code treatment, as you do below. Also, you could use utils.hash() instead of using directly md5, and that *must* yield that exact same result, being faster than resorting to subprocesses.

Also, remembering that I don't really love seeing things from the commands API instead of utils.system, utils.run and others.

client/tests/kvm/tests/nicdriver_unload.py line 65

Quoted 7 lines
client/tests/kvm/tests/nicdriver_unload.py new file +7 -0
+        check_sum1 = output1.split()[0]
+        s, output2 = session.get_command_status_output(cmd % receive_file)
+        if s != 0:
+            logging.error("Could not get md5sum of receive_file")
+            return False
+        check_sum2 = output2.strip().split()[0]
+        logging.debug("origin: %s, receive: %s" % (check_sum1, check_sum2))

^ "original file md5: %s, received file md5: %s"

client/tests/kvm/tests/nicdriver_unload.py line 77

Quoted 12 lines
client/tests/kvm/tests/nicdriver_unload.py new file +12 -0
+        if check_sum1 != check_sum2:
+            logging.error("md5sum doesn't match")
+            return False
+        return True
+
+    #produce sized file in host
+    file_size = params.get("file_size")
+    file_name = "/tmp/nicdriver_unload_file"
+    cmd = "dd if=/dev/urandom of=%s bs=%sM count=1"
+    s, o = commands.getstatusoutput(cmd % (file_name, file_size))
+    if s != 0:
+        raise error.TestFail("Fail to create file by dd")

Here utils.system would already raise a CmdError exception, so you could just use it and not do exception handling

client/tests/kvm/tests/nicdriver_unload.py line 87

Quoted 9 lines
client/tests/kvm/tests/nicdriver_unload.py new file +9 -0
+    connect_time = params.get("connect_time")
+    scp_timeout = int(params.get("scp_timeout"))
+    thread_num = int(params.get("thread_num"))
+    file_list = []
+
+    unload_load_cmd = "sleep %s && ifconfig %s down && modprobe -r %s && "
+    unload_load_cmd += "sleep 1 && modprobe %s && ifconfig %s up"
+    unload_load_cmd = unload_load_cmd % (connect_time, ethname, driver,
+                                         driver, ethname)

^ Here it'd be better to construct the whole command on a single assignment:

    unload_load_cmd = ("sleep %s && ifconfig %s down && modprobe -r %s && "
                       "sleep 1 && modprobe %s && ifconfig %s up" %
                       (connect_time, ethname, driver, driver, ethname))
Quoted 71 lines
client/tests/kvm/tests/nicdriver_unload.py new file +41 -0
+    pid = os.fork()
+    if pid != 0:
+        logging.info("unload/load nic driver repeatedly in guest...")
+        while True:
+            logging.debug("Try to unload/load nic drive once")
+            if session2.get_command_status(unload_load_cmd, timeout=120) != 0:
+                session.get_command_output("rm -rf /tmp/Thread-*")
+                raise error.TestFail("Unload/load nic driver failed")
+            pid, s = os.waitpid(pid, os.WNOHANG)
+            status = os.WEXITSTATUS(s)
+            if (pid, status) != (0, 0):
+                logging.debug("Child process ending")
+                break
+    else:
+        logging.info("Multi-session tcp data transfer")
+        threads = []
+        for i in range(thread_num):
+            t = ThreadScp()
+            t.start()
+            threads.append(t)
+        for t in threads:
+            t.join(timeout = scp_timeout)
+        os._exit(0)
+
+    session2.close()
+
+    try:
+        logging.info("Check md5sum for received files in multi-session")
+        for f in file_list:
+            if not compare(file_name, f):
+                raise error.TestFail("Fail to compare (guest)file %s" % f)
+
+        logging.info("Test nic function after load/unload")
+        if not vm.copy_files_to(file_name, file_name):
+            raise error.TestFail("Fail to copy file from host to guest")
+        if not compare(file_name, file_name):
+            raise error.TestFail("Test nic function after load/unload fail")
+
+    finally:
+        session.get_command_output("rm -rf /tmp/Thread-*")
+        session.close()
client/tests/kvm/tests_base.cfg.sample +9 -1
index 872674e..03d15c0 100644
@@ -362,6 +362,14 @@ variants:
         filesize = 4000
         scp_timeout = 1000
 
+    - nicdriver_unload:  install setup unattended_install.cdrom
+        type = nicdriver_unload
+        nic_mode = tap
+        file_size = 100
+        connect_time = 4
+        scp_timeout = 300
+        thread_num = 10
+
     - physical_resources_check: install setup unattended_install.cdrom
         type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'
@@ -1038,7 +1046,7 @@ variants:
 
     # Windows section
     - @Windows:
-        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer
+        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer nicdriver_unload
         shutdown_command = shutdown /s /f /t 0
         reboot_command = shutdown /r /f /t 0
         status_test_command = echo %errorlevel%
author
#26 lore raw

RFC PATCH 8/14 KVM-test: Add a subtest of nic promisc

In reply to Amos Kong

This test mainly covers TCP sent from host to guest and from guest to host with repeatedly turn on/off NIC promiscuous mode.

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/tests/nic_promisc.py new file +87 -0
new file mode 100644
index 0000000..9a0c979
@@ -0,0 +1,87 @@
+import logging, commands
+from autotest_lib.client.common_lib import error
+import kvm_utils, kvm_test_utils, kvm_net_utils
+
+def run_nic_promisc(test, params, env):
+    """
+    Test nic driver in promisc mode:
+
+    1) Boot up a guest
+    2) Repeatedly enable/disable promiscuous mode in guest
+    3) TCP data transmission from host to guest, and from guest to host,
+       with 1/1460/65000/100000000 bytes payloads
+    4) Clean temporary files
+    5) Stop enable/disable promiscuous mode change
+
+    @param test: kvm test object
+    @param params: Dictionary with the test parameters
+    @param env: Dictionary with test environment
+    """
+    timeout = int(params.get("login_timeout", 360))
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    session = kvm_test_utils.wait_for_login(vm, timeout=timeout)
+    logging.info("Trying to log into guest '%s' by serial", vm.name)
+    session2 = kvm_utils.wait_for(lambda: vm.serial_login(),
+                                  timeout, 0, step=2)
+    if not session2:
+        raise error.TestFail("Could not log into guest '%s'" % vm.name)
+
+    def compare(filename):
+        cmd = "md5sum %s" % filename
+        s1, ret_host = commands.getstatusoutput(cmd)
+        s2, ret_guest = session.get_command_status_output(cmd)
+        if s1 != 0 or s2 != 0:
+            logging.debug("ret_host:%s, ret_guest:%s" % (ret_host, ret_guest))
+            logging.error("Could not get md5, cmd:%s" % cmd)
+            return False
+        if ret_host.strip() != ret_guest.strip():
+            logging.debug("ret_host :%s, ret_guest:%s" % (ret_host, ret_guest))
+            logging.error("Files' md5sum mismatch" % (receiver))
+            return False
+        return True
+
+    ethname = kvm_net_utils.get_linux_ifname(session, vm.get_macaddr(0))
+    set_promisc_cmd = "ip link set %s promisc on; sleep 0.01;" % ethname
+    set_promisc_cmd += "ip link set %s promisc off; sleep 0.01" % ethname
+    logging.info("Set promisc change repeatedly in guest")
+    session2.sendline("while true; do %s; done" % set_promisc_cmd)
+
+    dd_cmd = "dd if=/dev/urandom of=%s bs=%d count=1"
+    filename = "/tmp/nic_promisc_file"
+    file_size = params.get("file_size", "1, 1460, 65000, 100000000").split(",")
+    try:
+        for size in file_size:
+            logging.info("Create %s bytes file on host" % size)
+            s, o = commands.getstatusoutput(dd_cmd % (filename, int(size)))
+            if s != 0:
+                logging.debug("Output: %s"% o)
+                raise error.TestFail("Create file on host failed")
+
+            logging.info("Transfer file from host to guest")
+            if not vm.copy_files_to(filename, filename):
+                raise error.TestFail("File transfer failed")
+            if not compare(filename):
+                raise error.TestFail("Compare file failed")
+
+            logging.info("Create %s bytes file on guest" % size)
+            if session.get_command_status(dd_cmd % (filename, int(size)),
+                                                    timeout=100) != 0:
+                raise error.TestFail("Create file on guest failed")
+
+            logging.info("Transfer file from guest to host")
+            if not vm.copy_files_from(filename, filename):
+                raise error.TestFail("File transfer failed")
+            if not compare(filename):
+                raise error.TestFail("Compare file failed")
+
+            logging.info("Clean temporal files")
+            cmd = "rm -f %s" % filename
+            s1, o = commands.getstatusoutput(cmd)
+            s2 = session.get_command_status(cmd)
+            if s1 != 0 or s2 != 0:
+                raise error.TestError("Fail to clean temporal files")
+    finally:
+        logging.info("Restore the %s to the nonpromisc mode" % ethname)
+        session2.close()
+        session.get_command_status("ip link set %s promisc off" % ethname)
+        session.close()
client/tests/kvm/tests_base.cfg.sample +5 -1
index 03d15c0..9e2b9a0 100644
@@ -370,6 +370,10 @@ variants:
         scp_timeout = 300
         thread_num = 10
 
+    - nic_promisc:  install setup unattended_install.cdrom
+        type = nic_promisc
+        file_size = 1, 1460, 65000, 100000000
+
     - physical_resources_check: install setup unattended_install.cdrom
         type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'
@@ -1046,7 +1050,7 @@ variants:
 
     # Windows section
     - @Windows:
-        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer nicdriver_unload
+        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer nicdriver_unload nic_promisc
         shutdown_command = shutdown /s /f /t 0
         reboot_command = shutdown /r /f /t 0
         status_test_command = echo %errorlevel%
#27 lore raw

Re: [RFC PATCH 08/14] KVM-test: Add a subtest of nic promisc

Quoted 43 lines · Amos Kong

This test mainly covers TCP sent from host to guest and from guest to host with repeatedly turn on/off NIC promiscuous mode.

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/tests/nic_promisc.py new file +30 -0
new file mode 100644
index 0000000..9a0c979
@@ -0,0 +1,87 @@
+import logging, commands
+from autotest_lib.client.common_lib import error
+import kvm_utils, kvm_test_utils, kvm_net_utils
+
+def run_nic_promisc(test, params, env):
+    """
+    Test nic driver in promisc mode:
+
+    1) Boot up a guest
+    2) Repeatedly enable/disable promiscuous mode in guest
+    3) TCP data transmission from host to guest, and from guest to host,
+       with 1/1460/65000/100000000 bytes payloads
+    4) Clean temporary files
+    5) Stop enable/disable promiscuous mode change
+
+    @param test: kvm test object
+    @param params: Dictionary with the test parameters
+    @param env: Dictionary with test environment
+    """
+    timeout = int(params.get("login_timeout", 360))
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    session = kvm_test_utils.wait_for_login(vm, timeout=timeout)
+    logging.info("Trying to log into guest '%s' by serial", vm.name)
+    session2 = kvm_utils.wait_for(lambda: vm.serial_login(),
+                                  timeout, 0, step=2)
+    if not session2:
+        raise error.TestFail("Could not log into guest '%s'" % vm.name)
+
+    def compare(filename):
+        cmd = "md5sum %s" % filename

client/tests/kvm/tests/nic_promisc.py line 40

Quoted 10 lines
client/tests/kvm/tests/nic_promisc.py new file +10 -0
+        s1, ret_host = commands.getstatusoutput(cmd)
+        s2, ret_guest = session.get_command_status_output(cmd)
+        if s1 != 0 or s2 != 0:
+            logging.debug("ret_host:%s, ret_guest:%s" % (ret_host, ret_guest))
+            logging.error("Could not get md5, cmd:%s" % cmd)
+            return False
+        if ret_host.strip() != ret_guest.strip():
+            logging.debug("ret_host :%s, ret_guest:%s" % (ret_host, ret_guest))
+            logging.error("Files' md5sum mismatch" % (receiver))
+            return False

^ The above debug messages will be confusing when looked by someone who is not familiar with the test code, so we should make their lives easier:

def compare(filename):
cmd = "md5sum %s" % filename
rc_host, md5_host = commands.getstatusoutput(cmd)
rc_guest, md5_guest = session.get_command_status_output(cmd)
if rc_host:
logging.debug('Could not get MD5 hash for file %s on host, output: %s', filename, md5_host) return False
if rc_guest:
logging.debug('Could not get MD5 hash for file %s on guest, output: %s', filename, md5_guest) return False
md5host = md5host.strip()
md5guest = md5guest.strip()
if md5host != md5guest:
logging.error('MD5 hash mismatch between file %s present on guest and on host', filename) logging.error('MD5 hash for file on guest: %s, MD5 hash for file on host: %s', md5_host, md5_guest) return False
return True

client/tests/kvm/tests/nic_promisc.py line 45

Quoted 5 lines
client/tests/kvm/tests/nic_promisc.py new file +5 -0
+        return True
+
+    ethname = kvm_net_utils.get_linux_ifname(session, vm.get_macaddr(0))
+    set_promisc_cmd = "ip link set %s promisc on; sleep 0.01;" % ethname
+    set_promisc_cmd += "ip link set %s promisc off; sleep 0.01" % ethname

^ You could do the above on a single attribution, see comment on patch 7 of the patchseries.

Quoted 9 lines
client/tests/kvm/tests/nic_promisc.py new file +9 -0
+    logging.info("Set promisc change repeatedly in guest")
+    session2.sendline("while true; do %s; done" % set_promisc_cmd)
+
+    dd_cmd = "dd if=/dev/urandom of=%s bs=%d count=1"
+    filename = "/tmp/nic_promisc_file"
+    file_size = params.get("file_size", "1, 1460, 65000, 100000000").split(",")
+    try:
+        for size in file_size:
+            logging.info("Create %s bytes file on host" % size)

client/tests/kvm/tests/nic_promisc.py line 64

Quoted 10 lines
client/tests/kvm/tests/nic_promisc.py new file +10 -0
+            s, o = commands.getstatusoutput(dd_cmd % (filename, int(size)))
+            if s != 0:
+                logging.debug("Output: %s"% o)
+                raise error.TestFail("Create file on host failed")
+
+            logging.info("Transfer file from host to guest")
+            if not vm.copy_files_to(filename, filename):
+                raise error.TestFail("File transfer failed")
+            if not compare(filename):
+                raise error.TestFail("Compare file failed")

^ It'd be better if we don't abruptly fail the whole test if we get a failure for a single size, what about having a global failure counter, and increment it if we have failures, making sure we log errors appropriately?

client/tests/kvm/tests/nic_promisc.py line 75

Quoted 10 lines
client/tests/kvm/tests/nic_promisc.py new file +10 -0
+            logging.info("Create %s bytes file on guest" % size)
+            if session.get_command_status(dd_cmd % (filename, int(size)),
+                                                    timeout=100) != 0:
+                raise error.TestFail("Create file on guest failed")
+
+            logging.info("Transfer file from guest to host")
+            if not vm.copy_files_from(filename, filename):
+                raise error.TestFail("File transfer failed")
+            if not compare(filename):
+                raise error.TestFail("Compare file failed")

^ Same comment as above.

client/tests/kvm/tests/nic_promisc.py line 77

Quoted 1 line
client/tests/kvm/tests/nic_promisc.py new file +1 -0
+            logging.info("Clean temporal files")

^ Typo, temporary

client/tests/kvm/tests/nic_promisc.py line 82

Quoted 5 lines
client/tests/kvm/tests/nic_promisc.py new file +5 -0
+            cmd = "rm -f %s" % filename
+            s1, o = commands.getstatusoutput(cmd)
+            s2 = session.get_command_status(cmd)
+            if s1 != 0 or s2 != 0:
+                raise error.TestError("Fail to clean temporal files")

^ Same as above, with the exception that maybe a failure during cleanup is not something so bad we'd need to fail the entire test because of it.

Quoted 34 lines
client/tests/kvm/tests/nic_promisc.py new file +5 -0
+    finally:
+        logging.info("Restore the %s to the nonpromisc mode" % ethname)
+        session2.close()
+        session.get_command_status("ip link set %s promisc off" % ethname)
+        session.close()
client/tests/kvm/tests_base.cfg.sample +5 -2
index 03d15c0..9e2b9a0 100644
@@ -370,6 +370,10 @@ variants:
         scp_timeout = 300
         thread_num = 10
 
+    - nic_promisc:  install setup unattended_install.cdrom
+        type = nic_promisc
+        file_size = 1, 1460, 65000, 100000000
+
     - physical_resources_check: install setup unattended_install.cdrom
         type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'
@@ -1046,7 +1050,7 @@ variants:
 
     # Windows section
     - @Windows:
-        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer nicdriver_unload
+        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer nicdriver_unload nic_promisc
         shutdown_command = shutdown /s /f /t 0
         reboot_command = shutdown /r /f /t 0
         status_test_command = echo %errorlevel%
 
--

To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#28 lore raw

Re: [Autotest] [RFC PATCH 08/14] KVM-test: Add a subtest of nic promisc

Quoted 103 lines · Lucas Meneghel Rodrigues
Quoted 43 lines · Amos Kong

This test mainly covers TCP sent from host to guest and from guest to host with repeatedly turn on/off NIC promiscuous mode.

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/tests/nic_promisc.py new file +30 -0
new file mode 100644
index 0000000..9a0c979
@@ -0,0 +1,87 @@
+import logging, commands
+from autotest_lib.client.common_lib import error
+import kvm_utils, kvm_test_utils, kvm_net_utils
+
+def run_nic_promisc(test, params, env):
+    """
+    Test nic driver in promisc mode:
+
+    1) Boot up a guest
+    2) Repeatedly enable/disable promiscuous mode in guest
+    3) TCP data transmission from host to guest, and from guest to host,
+       with 1/1460/65000/100000000 bytes payloads
+    4) Clean temporary files
+    5) Stop enable/disable promiscuous mode change
+
+    @param test: kvm test object
+    @param params: Dictionary with the test parameters
+    @param env: Dictionary with test environment
+    """
+    timeout = int(params.get("login_timeout", 360))
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    session = kvm_test_utils.wait_for_login(vm, timeout=timeout)
+    logging.info("Trying to log into guest '%s' by serial", vm.name)
+    session2 = kvm_utils.wait_for(lambda: vm.serial_login(),
+                                  timeout, 0, step=2)
+    if not session2:
+        raise error.TestFail("Could not log into guest '%s'" % vm.name)
+
+    def compare(filename):
+        cmd = "md5sum %s" % filename

client/tests/kvm/tests/nic_promisc.py line 40

Quoted 10 lines
client/tests/kvm/tests/nic_promisc.py new file +10 -0
+        s1, ret_host = commands.getstatusoutput(cmd)
+        s2, ret_guest = session.get_command_status_output(cmd)
+        if s1 != 0 or s2 != 0:
+            logging.debug("ret_host:%s, ret_guest:%s" % (ret_host, ret_guest))
+            logging.error("Could not get md5, cmd:%s" % cmd)
+            return False
+        if ret_host.strip() != ret_guest.strip():
+            logging.debug("ret_host :%s, ret_guest:%s" % (ret_host, ret_guest))
+            logging.error("Files' md5sum mismatch" % (receiver))
+            return False

^ The above debug messages will be confusing when looked by someone who is not familiar with the test code, so we should make their lives easier:

   def compare(filename):
       cmd = "md5sum %s" % filename
       rc_host, md5_host = commands.getstatusoutput(cmd)
       rc_guest, md5_guest = session.get_command_status_output(cmd)
       if rc_host:
           logging.debug('Could not get MD5 hash for file %s on host, output: %s', filename, md5_host) return False
       if rc_guest:
           logging.debug('Could not get MD5 hash for file %s on guest, output: %s', filename, md5_guest) return False
       md5host = md5host.strip()
       md5guest = md5guest.strip()
       if md5host != md5guest:
           logging.error('MD5 hash mismatch between file %s present on guest and on host', filename) logging.error('MD5 hash for file on guest: %s, MD5 hash for file on host: %s', md5_host, md5_guest) return False
       return True

client/tests/kvm/tests/nic_promisc.py line 45

Quoted 5 lines
client/tests/kvm/tests/nic_promisc.py new file +5 -0
+        return True
+
+    ethname = kvm_net_utils.get_linux_ifname(session, vm.get_macaddr(0))
+    set_promisc_cmd = "ip link set %s promisc on; sleep 0.01;" % ethname
+    set_promisc_cmd += "ip link set %s promisc off; sleep 0.01" % ethname

^ You could do the above on a single attribution, see comment on patch 7 of the patchseries.

Quoted 16 lines
client/tests/kvm/tests/nic_promisc.py new file +16 -0
+    logging.info("Set promisc change repeatedly in guest")
+    session2.sendline("while true; do %s; done" % set_promisc_cmd)
+
+    dd_cmd = "dd if=/dev/urandom of=%s bs=%d count=1"
+    filename = "/tmp/nic_promisc_file"
+    file_size = params.get("file_size", "1, 1460, 65000, 100000000").split(",")
+    try:
+        for size in file_size:
+            logging.info("Create %s bytes file on host" % size)
+            s, o = commands.getstatusoutput(dd_cmd % (filename, int(size)))
+            if s != 0:
+                logging.debug("Output: %s"% o)
+                raise error.TestFail("Create file on host failed")
+
+            logging.info("Transfer file from host to guest")
+            if not vm.copy_files_to(filename, filename):
Quoted 8 lines

client/tests/kvm/tests/nic_promisc.py line 64

Quoted 3 lines
client/tests/kvm/tests/nic_promisc.py new file +3 -0
+                raise error.TestFail("File transfer failed")
+            if not compare(filename):
+                raise error.TestFail("Compare file failed")

^ It'd be better if we don't abruptly fail the whole test if we get a failure for a single size, what about having a global failure counter, and increment it if we have failures, making sure we log errors appropriately?

How about add a success counter ?
only success_counter+1 after successfully compare.
If the end success_counter doesn't equals to len(file_size), raise and fail exception.

Quoted 67 lines

client/tests/kvm/tests/nic_promisc.py line 75

Quoted 10 lines
client/tests/kvm/tests/nic_promisc.py new file +10 -0
+            logging.info("Create %s bytes file on guest" % size)
+            if session.get_command_status(dd_cmd % (filename, int(size)),
+                                                    timeout=100) != 0:
+                raise error.TestFail("Create file on guest failed")
+
+            logging.info("Transfer file from guest to host")
+            if not vm.copy_files_from(filename, filename):
+                raise error.TestFail("File transfer failed")
+            if not compare(filename):
+                raise error.TestFail("Compare file failed")

^ Same comment as above.

client/tests/kvm/tests/nic_promisc.py line 77

Quoted 1 line
client/tests/kvm/tests/nic_promisc.py new file +1 -0
+            logging.info("Clean temporal files")

^ Typo, temporary

client/tests/kvm/tests/nic_promisc.py line 82

Quoted 5 lines
client/tests/kvm/tests/nic_promisc.py new file +5 -0
+            cmd = "rm -f %s" % filename
+            s1, o = commands.getstatusoutput(cmd)
+            s2 = session.get_command_status(cmd)
+            if s1 != 0 or s2 != 0:
+                raise error.TestError("Fail to clean temporal files")

^ Same as above, with the exception that maybe a failure during cleanup is not something so bad we'd need to fail the entire test because of it.

Quoted 26 lines
client/tests/kvm/tests/nic_promisc.py new file +5 -0
+    finally:
+        logging.info("Restore the %s to the nonpromisc mode" % ethname)
+        session2.close()
+        session.get_command_status("ip link set %s promisc off" % ethname)
+        session.close()
client/tests/kvm/tests_base.cfg.sample +0 -0
index 03d15c0..9e2b9a0 100644
@@ -370,6 +370,10 @@ variants:

         scp_timeout = 300
         thread_num = 10

+    - nic_promisc:  install setup unattended_install.cdrom
+        type = nic_promisc
+        file_size = 1, 1460, 65000, 100000000
+
     - physical_resources_check: install setup unattended_install.cdrom type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'

client/tests/kvm/tests_base.cfg.sample +0 -0
@@ -1046,7 +1050,7 @@ variants:
 
      # Windows section

     - @Windows:

-        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer nicdriver_unload
+        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer nicdriver_unload nic_promisc
Quoted 8 lines

         shutdown_command = shutdown /s /f /t 0
         reboot_command = shutdown /r /f /t 0
         status_test_command = echo %errorlevel%

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

_______________________________________________
Autotest mailing list
[email protected]
http://test.kernel.org/cgi-bin/mailman/listinfo/autotest

author
#29 lore raw

RFC PATCH 9/14 KVM-test: Add a subtest of multicast

In reply to Amos Kong

Use 'ping' to test send/recive multicat packets. Flood ping test is also added. Limit guest network as 'bridge' mode, because multicast packets could not be transmitted to guest when using 'user' network.
Add join_mcast.py for joining machine into multicast groups.

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/scripts/join_mcast.py new file +29 -0
new file mode 100755
index 0000000..0d90e5c
@@ -0,0 +1,29 @@
+import socket, struct, os, signal, sys
+# this script is used to join machine into multicast groups
+# author: Amos Kong <[email protected]>
+
+if len(sys.argv) < 4:
+    print """%s [mgroup_count] [prefix] [suffix]
+    mgroup_count: count of multicast addresses
+    prefix: multicast address prefix
+    suffix: multicast address suffix""" % sys.argv[0]
+    sys.exit()
+
+mgroup_count = int(sys.argv[1])
+prefix = sys.argv[2]
+suffix = int(sys.argv[3])
+
+s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+for i in range(mgroup_count):
+    mcast = prefix + "." + str(suffix + i)
+    try:
+        mreq = struct.pack("4sl", socket.inet_aton(mcast), socket.INADDR_ANY)
+        s.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, mreq)
+    except:
+        s.close()
+        print "Could not join multicast: %s" % mcast
+        raise
+
+print "join_mcast_pid:%s" % os.getpid()
+os.kill(os.getpid(), signal.SIGSTOP)
+s.close()
client/tests/kvm/tests/multicast.py new file +67 -0

67 lines not on this page · raw message

client/tests/kvm/tests_base.cfg.sample +8 -1
index 9e2b9a0..9594a38 100644
@@ -374,6 +374,13 @@ variants:
         type = nic_promisc
         file_size = 1, 1460, 65000, 100000000
 
+    - multicast: install setup unattended_install.cdrom
+        type = multicast
+        nic_mode = tap
+        mcast = 225.0.0.1
+        mgroup_count = 20
+        flood_minutes = 1
+
     - physical_resources_check: install setup unattended_install.cdrom
         type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'
@@ -1050,7 +1057,7 @@ variants:
 
     # Windows section
     - @Windows:
-        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer nicdriver_unload nic_promisc
+        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer nicdriver_unload nic_promisc multicast
         shutdown_command = shutdown /s /f /t 0
         reboot_command = shutdown /r /f /t 0
         status_test_command = echo %errorlevel%
#30 lore raw

Re: [Qemu-devel] [RFC PATCH 09/14] KVM-test: Add a subtest of multicast

Quoted 65 lines · Amos Kong

Use 'ping' to test send/recive multicat packets. Flood ping test is also added. Limit guest network as 'bridge' mode, because multicast packets could not be transmitted to guest when using 'user' network.
Add join_mcast.py for joining machine into multicast groups.

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/scripts/join_mcast.py new file +29 -0
new file mode 100755
index 0000000..0d90e5c
@@ -0,0 +1,29 @@
+import socket, struct, os, signal, sys
+# this script is used to join machine into multicast groups
+# author: Amos Kong <[email protected]>
+
+if len(sys.argv) < 4:
+    print """%s [mgroup_count] [prefix] [suffix]
+    mgroup_count: count of multicast addresses
+    prefix: multicast address prefix
+    suffix: multicast address suffix""" % sys.argv[0]
+    sys.exit()
+
+mgroup_count = int(sys.argv[1])
+prefix = sys.argv[2]
+suffix = int(sys.argv[3])
+
+s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+for i in range(mgroup_count):
+    mcast = prefix + "." + str(suffix + i)
+    try:
+        mreq = struct.pack("4sl", socket.inet_aton(mcast), socket.INADDR_ANY)
+        s.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, mreq)
+    except:
+        s.close()
+        print "Could not join multicast: %s" % mcast
+        raise
+
+print "join_mcast_pid:%s" % os.getpid()
+os.kill(os.getpid(), signal.SIGSTOP)
+s.close()
client/tests/kvm/tests/multicast.py new file +15 -0
new file mode 100644
index 0000000..6b0e106
@@ -0,0 +1,67 @@
+import logging, commands, os, re
+from autotest_lib.client.common_lib import error
+import kvm_test_utils, kvm_net_utils
+
+
+def run_multicast(test, params, env):
+    """
+    Test multicast function of nic (rtl8139/e1000/virtio)
+
+    1) Create a VM
+    2) Join guest into multicast groups
+    3) Ping multicast addresses on host
+    4) Flood ping test with different size of packets
+    5) Final ping test and check if lose packet
+

client/tests/kvm/tests/multicast.py line 25

Quoted 10 lines
client/tests/kvm/tests/multicast.py new file +10 -0
+    @param test: Kvm test object
+    @param params: Dictionary with the test parameters.
+    @param env: Dictionary with test environment.
+    """
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    session = kvm_test_utils.wait_for_login(vm,
+                  timeout=int(params.get("login_timeout", 360)))
+
+    # stop iptable/selinux on guest/host
+    cmd = "/etc/init.d/iptables stop && echo 0 > /selinux/enforce"

^ Different linux distros handle init scripts differently, so it's better to just flush the firewall rules with iptables -F instead of resorting to the init script.

Also, not all distros use selinux, so it'd be better to test for the presence of selinux on the system (for example, by checking if there's a /selinux/enforce file before trying to write to it). Also, need to do proper return code checking/treatment.

Quoted 7 lines
client/tests/kvm/tests/multicast.py new file +7 -0
+    session.get_command_status(cmd)
+    commands.getoutput(cmd)
+    # make sure guest replies to broadcasts
+    session.get_command_status("echo 0 > /proc/sys/net/ipv4/icmp_echo_ignore"
+            "_broadcasts && echo 0 > /proc/sys/net/ipv4/icmp_echo_ignore_all")
+
+    # base multicast address

client/tests/kvm/tests/multicast.py line 42

Quoted 10 lines
client/tests/kvm/tests/multicast.py new file +10 -0
+    mcast = params.get("mcast", "225.0.0.1")
+    # count of multicast addresses, less than 20
+    mgroup_count = int(params.get("mgroup_count", 5))
+    flood_minutes = float(params.get("flood_minutes", 10))
+    ifname = vm.get_ifname()
+    prefix = re.findall("\d+.\d+.\d+", mcast)[0]
+    suffix = int(re.findall("\d+", mcast)[-1])
+    # copy python script to guest for joining guest to multicast groups
+    mcast_path = os.path.join(test.bindir, "scripts/join_mcast.py")
+    vm.copy_files_to(mcast_path, "/tmp")

^ What if copy_files_to fails? Need to do proper return handling here

Quoted 12 lines
client/tests/kvm/tests/multicast.py new file +12 -0
+    output = session.get_command_output("python /tmp/join_mcast.py %d %s %d" %
+                                        (mgroup_count, prefix, suffix))
+    # if success to join multicast the process will be paused, and return pid.
+    if not re.findall("join_mcast_pid:(\d+)", output):
+        raise error.TestFail("Can't join multicast groups,output:%s" % output)
+    pid = output.split()[0]
+
+    try:
+        for i in range(mgroup_count):
+            new_suffix = suffix + i
+            mcast = "%s.%d" % (prefix, new_suffix)
+            logging.info("Initial ping test, mcast: %s", mcast)

client/tests/kvm/tests/multicast.py line 64

Quoted 10 lines
client/tests/kvm/tests/multicast.py new file +10 -0
+            s, o = kvm_net_utils.ping(mcast, 10, interface=ifname, timeout=20)
+            if s != 0:
+                raise error.TestFail(" Ping return non-zero value %s" % o)
+            logging.info("Flood ping test, mcast: %s", mcast)
+            kvm_net_utils.ping(mcast, None, interface=ifname, flood=True,
+                               output_func=None, timeout=flood_minutes*60)
+            logging.info("Final ping test, mcast: %s", mcast)
+            s, o = kvm_net_utils.ping(mcast, 10, interface=ifname, timeout=20)
+            if s != 0:
+                raise error.TestFail(" Ping return non-zero value %s" % o)

^ "Ping failed, status: %s, output: %s" % (s, o) would be a better exception

Quoted 36 lines
client/tests/kvm/tests/multicast.py new file +3 -0
+    finally:
+        session.get_command_output("kill -s SIGCONT %s" % pid)
+        session.close()
client/tests/kvm/tests_base.cfg.sample +8 -2
index 9e2b9a0..9594a38 100644
@@ -374,6 +374,13 @@ variants:
         type = nic_promisc
         file_size = 1, 1460, 65000, 100000000
 
+    - multicast: install setup unattended_install.cdrom
+        type = multicast
+        nic_mode = tap
+        mcast = 225.0.0.1
+        mgroup_count = 20
+        flood_minutes = 1
+
     - physical_resources_check: install setup unattended_install.cdrom
         type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'
@@ -1050,7 +1057,7 @@ variants:
 
     # Windows section
     - @Windows:
-        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer nicdriver_unload nic_promisc
+        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer nicdriver_unload nic_promisc multicast
         shutdown_command = shutdown /s /f /t 0
         reboot_command = shutdown /r /f /t 0
         status_test_command = echo %errorlevel%
 
--

To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#31 lore raw

RFC PATCH 10/14 KVM-test: Add a subtest of pxe

In reply to Amos Kong

This case just snoop tftp packet through tcpdump, it depends on public dhcp server, better to test it through dnsmasq.

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/tests/pxe.py new file +30 -0
new file mode 100644
index 0000000..8859aaa
@@ -0,0 +1,30 @@
+import logging
+from autotest_lib.client.common_lib import error
+import kvm_subprocess, kvm_test_utils, kvm_utils
+
+
+def run_pxe(test, params, env):
+    """
+    PXE test:
+
+    1) Snoop the tftp packet in the tap device
+    2) Wait for some seconds
+    3) Check whether capture tftp packets
+
+    @param test: kvm test object
+    @param params: Dictionary with the test parameters
+    @param env: Dictionary with test environment.
+    """
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    timeout = int(params.get("pxe_timeout", 60))
+
+    logging.info("Try to boot from pxe")
+    status, output = kvm_subprocess.run_fg("tcpdump -nli %s" % vm.get_ifname(),
+                                           logging.debug,
+                                           "(pxe) ",
+                                           timeout)
+
+    logging.info("Analysing the tcpdump result...")
+    if not "tftp" in output:
+        raise error.TestFail("Couldn't find tftp packet in %s seconds" % timeout)
+    logging.info("Found tftp packet")
client/tests/kvm/tests_base.cfg.sample +13 -0
index 9594a38..5515601 100644
@@ -381,6 +381,19 @@ variants:
         mgroup_count = 20
         flood_minutes = 1
 
+    - pxe:
+        type = pxe
+        images = pxe
+        image_name_pxe = pxe-test
+        image_size_pxe = 1G
+        force_create_image_pxe = yes
+        remove_image_pxe = yes
+        extra_params += ' -boot n'
+        kill_vm_on_error = yes
+        network = bridge
+        restart_vm = yes
+        pxe_timeout = 60
+
     - physical_resources_check: install setup unattended_install.cdrom
         type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'
#32 lore raw

Re: [Qemu-devel] [RFC PATCH 10/14] KVM-test: Add a subtest of pxe

Quoted 2 lines · Amos Kong

This case just snoop tftp packet through tcpdump, it depends on public dhcp server, better to test it through dnsmasq.

It would be a good idea to have an alternate implementation using dnsmasq, but not urgent.

Quoted 26 lines

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/tests/pxe.py new file +15 -0
new file mode 100644
index 0000000..8859aaa
@@ -0,0 +1,30 @@
+import logging
+from autotest_lib.client.common_lib import error
+import kvm_subprocess, kvm_test_utils, kvm_utils
+
+
+def run_pxe(test, params, env):
+    """
+    PXE test:
+
+    1) Snoop the tftp packet in the tap device
+    2) Wait for some seconds
+    3) Check whether capture tftp packets
+
+    @param test: kvm test object
+    @param params: Dictionary with the test parameters

client/tests/kvm/tests/pxe.py line 25

Quoted 10 lines
client/tests/kvm/tests/pxe.py new file +10 -0
+    @param env: Dictionary with test environment.
+    """
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    timeout = int(params.get("pxe_timeout", 60))
+
+    logging.info("Try to boot from pxe")
+    status, output = kvm_subprocess.run_fg("tcpdump -nli %s" % vm.get_ifname(),
+                                           logging.debug,
+                                           "(pxe) ",
+                                           timeout)

^ The only complaint I could make here is that since this command doesn't need to live throughout tests, utils.run would do just fine. Other than that, looks fine to me.

client/tests/kvm/tests/pxe.py line 27

Quoted 1 line
client/tests/kvm/tests/pxe.py new file +1 -0
+    logging.info("Analysing the tcpdump result...")

^ typo, analyzing

Quoted 29 lines
client/tests/kvm/tests/pxe.py new file +3 -0
+    if not "tftp" in output:
+        raise error.TestFail("Couldn't find tftp packet in %s seconds" % timeout)
+    logging.info("Found tftp packet")
client/tests/kvm/tests_base.cfg.sample +13 -0
index 9594a38..5515601 100644
@@ -381,6 +381,19 @@ variants:
         mgroup_count = 20
         flood_minutes = 1
 
+    - pxe:
+        type = pxe
+        images = pxe
+        image_name_pxe = pxe-test
+        image_size_pxe = 1G
+        force_create_image_pxe = yes
+        remove_image_pxe = yes
+        extra_params += ' -boot n'
+        kill_vm_on_error = yes
+        network = bridge
+        restart_vm = yes
+        pxe_timeout = 60
+
     - physical_resources_check: install setup unattended_install.cdrom
         type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'
author
#33 lore raw

Re: [Qemu-devel] [RFC PATCH 10/14] KVM-test: Add a subtest of pxe

Quoted 6 lines · Lucas Meneghel Rodrigues
Quoted 2 lines · Amos Kong

This case just snoop tftp packet through tcpdump, it depends on public dhcp server, better to test it through dnsmasq.

It would be a good idea to have an alternate implementation using dnsmasq, but not urgent.

I changed unattended_install to use dnsmasq, but it was blocked by a pxe bug. Also need completed test, we can add this later.

Quoted 32 lines
Quoted 32 lines

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/tests/pxe.py new file +21 -0
new file mode 100644
index 0000000..8859aaa
@@ -0,0 +1,30 @@
+import logging
+from autotest_lib.client.common_lib import error
+import kvm_subprocess, kvm_test_utils, kvm_utils
+
+
+def run_pxe(test, params, env):
+    """
+    PXE test:
+
+    1) Snoop the tftp packet in the tap device
+    2) Wait for some seconds
+    3) Check whether capture tftp packets
+
+    @param test: kvm test object
+    @param params: Dictionary with the test parameters
+    @param env: Dictionary with test environment.
+    """
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    timeout = int(params.get("pxe_timeout", 60))
+
+    logging.info("Try to boot from pxe")
Quoted 8 lines

client/tests/kvm/tests/pxe.py line 25

Quoted 4 lines
client/tests/kvm/tests/pxe.py new file +4 -0
+    status, output = kvm_subprocess.run_fg("tcpdump -nli %s" % vm.get_ifname(),
+                                           logging.debug,
+                                           "(pxe) ",
+                                           timeout)

^ The only complaint I could make here is that since this command doesn't need to live throughout tests, utils.run would do just fine. Other than that, looks fine to me.

utils.run() desen't support timeout, tcpdump would not stop by itself. also could not add output prefix "(pxe) " ?

Quoted 35 lines

client/tests/kvm/tests/pxe.py line 27

Quoted 1 line
client/tests/kvm/tests/pxe.py new file +1 -0
+    logging.info("Analysing the tcpdump result...")

^ typo, analyzing

Quoted 29 lines
client/tests/kvm/tests/pxe.py new file +3 -0
+    if not "tftp" in output:
+        raise error.TestFail("Couldn't find tftp packet in %s seconds" % timeout)
+    logging.info("Found tftp packet")
client/tests/kvm/tests_base.cfg.sample +13 -0
index 9594a38..5515601 100644
@@ -381,6 +381,19 @@ variants:
         mgroup_count = 20
         flood_minutes = 1
 
+    - pxe:
+        type = pxe
+        images = pxe
+        image_name_pxe = pxe-test
+        image_size_pxe = 1G
+        force_create_image_pxe = yes
+        remove_image_pxe = yes
+        extra_params += ' -boot n'
+        kill_vm_on_error = yes
+        network = bridge
+        restart_vm = yes
+        pxe_timeout = 60
+
     - physical_resources_check: install setup unattended_install.cdrom
         type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'
author
#34 lore raw

RFC PATCH 11/14 KVM-test: Add a subtest of changing mac address

In reply to Amos Kong

Mainly test steps:
1. get a new mac from pool, and the old mac addr of guest.
2. execute the mac_change.sh in guest.
3. relogin to guest and query the interfaces info by ifconfig

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/tests/mac_change.py new file +66 -0
new file mode 100644
index 0000000..dc93377
@@ -0,0 +1,66 @@
+import logging
+from autotest_lib.client.common_lib import error
+import kvm_utils, kvm_test_utils, kvm_net_utils
+
+
+def run_mac_change(test, params, env):
+    """
+    Change MAC Address of Guest.
+
+    1. get a new mac from pool, and the old mac addr of guest.
+    2. set new mac in guest and regain new IP.
+    3. re-log into guest with new mac
+
+    @param test: kvm test object
+    @param params: Dictionary with the test parameters
+    @param env: Dictionary with test environment.
+    """
+    timeout = int(params.get("login_timeout", 360))
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    logging.info("Trying to log into guest '%s' by serial", vm.name)
+    session = kvm_utils.wait_for(lambda: vm.serial_login(),
+                                  timeout, 0, step=2)
+    if not session:
+        raise error.TestFail("Could not log into guest '%s'" % vm.name)
+
+    old_mac = vm.get_macaddr(0)
+    kvm_utils.put_mac_to_pool(vm.root_dir, old_mac, vm.instance)
+    new_mac = kvm_utils.get_mac_from_pool(vm.root_dir,
+                                          vm=vm.instance,
+                                          nic_index=0,
+                                          prefix=vm.mac_prefix)
+    logging.info("The initial MAC address is %s" % old_mac)
+    interface = kvm_net_utils.get_linux_ifname(session, old_mac)
+
+    # Start change mac address
+    logging.info("Changing mac address to %s" % new_mac)
+    change_cmd = "ifconfig %s down && ifconfig %s hw ether %s && ifconfig %s up"\
+                 % (interface, interface, new_mac, interface)
+    if session.get_command_status(change_cmd) != 0:
+        raise error.TestFail("Fail to send mac_change command")
+
+    # Verify whether mac address is changed to new one
+    logging.info("Verifying the new mac address")
+    if session.get_command_status("ifconfig | grep -i %s" % new_mac) != 0:
+        raise error.TestFail("Fail to change mac address")
+
+    # Restart `dhclient' to regain IP for new mac address
+    logging.info("Re-start the network to gain new ip")
+    dhclient_cmd = "dhclient -r && dhclient %s" % interface
+    session.sendline(dhclient_cmd)
+
+    # Re-log into the guest after changing mac address
+    if kvm_utils.wait_for(session.is_responsive, 120, 20, 3):
+        # Just warning when failed to see the session become dead,
+        # because there is a little chance the ip does not change.
+        logging.warn("The session is still responsive, settings may fail.")
+    session.close()
+
+    # Re-log into guest and check if session is responsive
+    logging.info("Re-log into the guest")
+    session = kvm_test_utils.wait_for_login(vm,
+              timeout=int(params.get("login_timeout", 360)))
+    if not session.is_responsive():
+        raise error.TestFail("The new session is not responsive.")
+
+    session.close()
client/tests/kvm/tests_base.cfg.sample +5 -1
index 5515601..7716d48 100644
@@ -394,6 +394,10 @@ variants:
         restart_vm = yes
         pxe_timeout = 60
 
+    - mac_change: install setup unattended_install.cdrom
+        type = mac_change
+        kill_vm = yes
+
     - physical_resources_check: install setup unattended_install.cdrom
         type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'
@@ -1070,7 +1074,7 @@ variants:
 
     # Windows section
     - @Windows:
-        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer nicdriver_unload nic_promisc multicast
+        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer nicdriver_unload nic_promisc multicast mac_change
         shutdown_command = shutdown /s /f /t 0
         reboot_command = shutdown /r /f /t 0
         status_test_command = echo %errorlevel%
#35 lore raw

Re: [Qemu-devel] [RFC PATCH 11/14] KVM-test: Add a subtest of changing mac address

Quoted 28 lines · Amos Kong

Mainly test steps:
1. get a new mac from pool, and the old mac addr of guest.
2. execute the mac_change.sh in guest.
3. relogin to guest and query the interfaces info by ifconfig

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/tests/mac_change.py new file +12 -0
new file mode 100644
index 0000000..dc93377
@@ -0,0 +1,66 @@
+import logging
+from autotest_lib.client.common_lib import error
+import kvm_utils, kvm_test_utils, kvm_net_utils
+
+
+def run_mac_change(test, params, env):
+    """
+    Change MAC Address of Guest.
+
+    1. get a new mac from pool, and the old mac addr of guest.
+    2. set new mac in guest and regain new IP.
+    3. re-log into guest with new mac

client/tests/kvm/tests/mac_change.py line 22

Quoted 10 lines
client/tests/kvm/tests/mac_change.py new file +10 -0
+
+    @param test: kvm test object
+    @param params: Dictionary with the test parameters
+    @param env: Dictionary with test environment.
+    """
+    timeout = int(params.get("login_timeout", 360))
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    logging.info("Trying to log into guest '%s' by serial", vm.name)
+    session = kvm_utils.wait_for(lambda: vm.serial_login(),
+                                  timeout, 0, step=2)

^ One thing that I forgot to comment on previous patches: For more clarity, it'd be good to name the session variable in a way that lets people refer easily that is a serial session

session_serial = ...

Quoted 24 lines
client/tests/kvm/tests/mac_change.py new file +24 -0
+    if not session:
+        raise error.TestFail("Could not log into guest '%s'" % vm.name)
+
+    old_mac = vm.get_macaddr(0)
+    kvm_utils.put_mac_to_pool(vm.root_dir, old_mac, vm.instance)
+    new_mac = kvm_utils.get_mac_from_pool(vm.root_dir,
+                                          vm=vm.instance,
+                                          nic_index=0,
+                                          prefix=vm.mac_prefix)
+    logging.info("The initial MAC address is %s" % old_mac)
+    interface = kvm_net_utils.get_linux_ifname(session, old_mac)
+
+    # Start change mac address
+    logging.info("Changing mac address to %s" % new_mac)
+    change_cmd = "ifconfig %s down && ifconfig %s hw ether %s && ifconfig %s up"\
+                 % (interface, interface, new_mac, interface)
+    if session.get_command_status(change_cmd) != 0:
+        raise error.TestFail("Fail to send mac_change command")
+
+    # Verify whether mac address is changed to new one
+    logging.info("Verifying the new mac address")
+    if session.get_command_status("ifconfig | grep -i %s" % new_mac) != 0:
+        raise error.TestFail("Fail to change mac address")
+

client/tests/kvm/tests/mac_change.py line 56

Quoted 10 lines
client/tests/kvm/tests/mac_change.py new file +10 -0
+    # Restart `dhclient' to regain IP for new mac address
+    logging.info("Re-start the network to gain new ip")
+    dhclient_cmd = "dhclient -r && dhclient %s" % interface
+    session.sendline(dhclient_cmd)
+
+    # Re-log into the guest after changing mac address
+    if kvm_utils.wait_for(session.is_responsive, 120, 20, 3):
+        # Just warning when failed to see the session become dead,
+        # because there is a little chance the ip does not change.
+        logging.warn("The session is still responsive, settings may fail.")

^ Isn't this a serial session? Then why the IP of guest changing would make this session un-responsive? I think the best idea here is to:

1) Release the IP through dhclient -r [interface]
2) Make sure we can't stablish a ssh based session to the guest by making a try/except block with kvm_test_utils.wait_for_login() with the appropriate timeouts and other parameters, if succeeds, fail the test, if it doesn't, proceed with the test.
3) Get a new IP with dhclient [interface]
4) Try to stablish a new, ssh based session to the guest and see if that works.

client/tests/kvm/tests/mac_change.py line 64

Quoted 8 lines
client/tests/kvm/tests/mac_change.py new file +8 -0
+    session.close()
+
+    # Re-log into guest and check if session is responsive
+    logging.info("Re-log into the guest")
+    session = kvm_test_utils.wait_for_login(vm,
+              timeout=int(params.get("login_timeout", 360)))
+    if not session.is_responsive():
+        raise error.TestFail("The new session is not responsive.")

^ Is it possible that right after you stablish the session it becomes non-responsive? It seems like a redundant verification step to me.

Quoted 31 lines
client/tests/kvm/tests/mac_change.py new file +1 -0
+    session.close()
client/tests/kvm/tests_base.cfg.sample +5 -2
index 5515601..7716d48 100644
@@ -394,6 +394,10 @@ variants:
         restart_vm = yes
         pxe_timeout = 60
 
+    - mac_change: install setup unattended_install.cdrom
+        type = mac_change
+        kill_vm = yes
+
     - physical_resources_check: install setup unattended_install.cdrom
         type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'
@@ -1070,7 +1074,7 @@ variants:
 
     # Windows section
     - @Windows:
-        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer nicdriver_unload nic_promisc multicast
+        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer nicdriver_unload nic_promisc multicast mac_change
         shutdown_command = shutdown /s /f /t 0
         reboot_command = shutdown /r /f /t 0
         status_test_command = echo %errorlevel%
 
--

To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#36 lore raw

RFC PATCH 12/14 KVM-test: Add a subtest of netperf

In reply to Amos Kong

Add network load by netperf, server is launched on guest, execute netperf client with different protocols on host. if all clients execute successfully, case will be pass. Test result will be record into result.txt. Now this case only tests with "TCP_RR TCP_CRR UDP_RR TCP_STREAM TCP_MAERTS TCP_SENDFILE UDP_STREAM". DLPI only supported by Unix, unix domain test is not necessary, so drop test of DLPI and unix domain.

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/tests/netperf.py new file +56 -0
new file mode 100644
index 0000000..00a91f0
@@ -0,0 +1,56 @@
+import logging, commands, os
+from autotest_lib.client.common_lib import error
+import kvm_subprocess, kvm_test_utils, kvm_utils
+
+def run_netperf(test, params, env):
+    """
+    Network stress test with netperf
+
+    1) Boot up a virtual machine
+    2) Launch netserver on guest
+    3) Execute netperf client on host with different protocols
+    4) Outout the test result
+
+    @param test: Kvm test object
+    @param params: Dictionary with the test parameters.
+    @param env: Dictionary with test environment.
+    """
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    session = kvm_test_utils.wait_for_login(vm,
+                   timeout=int(params.get("login_timeout", 360)))
+    netperf_dir = os.path.join(os.environ['AUTODIR'], "tests/netperf2")
+    setup_cmd = params.get("setup_cmd")
+    guest_ip = vm.get_address()
+    result_file = os.path.join(test.debugdir, "result.txt")
+
+    session.get_command_output("service iptables stop")
+    for i in params.get("netperf_files").split():
+        if not vm.copy_files_to(os.path.join(netperf_dir, i), "/tmp"):
+            raise error.TestError("Could not copy files to guest")
+    if session.get_command_status(setup_cmd % "/tmp", timeout=100) != 0:
+        raise error.TestFail("Fail to setup netperf on guest")
+    if session.get_command_status(params.get("netserver_cmd") % "/tmp") != 0:
+        raise error.TestFail("Fail to start netperf server on guest")
+
+    try:
+        logging.info("Setup and run netperf client on host")
+        s, o = commands.getstatusoutput(setup_cmd % netperf_dir)
+        if s != 0:
+            raise error.TestFail("Fail to setup netperf on host, o: %s" % o)
+        success = True
+        file(result_file, "w").write("Netperf Test Result\n")
+        for i in params.get("protocols").split():
+            cmd = params.get("netperf_cmd") % (netperf_dir, i, guest_ip)
+            logging.debug("Execute netperf client test: %s" % cmd)
+            s, o = commands.getstatusoutput(cmd)
+            if s != 0:
+                logging.error("Fail to execute netperf test, protocol:%s" % i)
+                success = False
+            else:
+                logging.info(o)
+                file(result_file, "a+").write("%s\n" % o)
+        if not success:
+            raise error.TestFail("Not all the test passed")
+    finally:
+        session.get_command_output("killall netserver")
+        session.close()
client/tests/kvm/tests_base.cfg.sample +10 -0
index 7716d48..dec988e 100644
@@ -398,6 +398,16 @@ variants:
         type = mac_change
         kill_vm = yes
 
+    - netperf: install setup unattended_install.cdrom
+        type = netperf
+        nic_mode = tap
+        netperf_files = netperf-2.4.5.tar.bz2 wait_before_data.patch
+        setup_cmd = "cd %s && tar xvfj netperf-2.4.5.tar.bz2 && cd netperf-2.4.5 && patch -p0 < ../wait_before_data.patch && ./configure && make"
+        netserver_cmd =  %s/netperf-2.4.5/src/netserver
+        # test time is 60 seconds, set the buffer size to 1 for more hardware interrupt
+        netperf_cmd = %s/netperf-2.4.5/src/netperf -t %s -H %s -l 60 -- -m 1
+        protocols = "TCP_STREAM TCP_MAERTS TCP_RR TCP_CRR UDP_RR TCP_SENDFILE UDP_STREAM"
+
     - physical_resources_check: install setup unattended_install.cdrom
         type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'
#37 lore raw

Re: [Qemu-devel] [RFC PATCH 12/14] KVM-test: Add a subtest of netperf

Quoted 6 lines · Amos Kong

Add network load by netperf, server is launched on guest, execute netperf client with different protocols on host. if all clients execute successfully, case will be pass. Test result will be record into result.txt. Now this case only tests with "TCP_RR TCP_CRR UDP_RR TCP_STREAM TCP_MAERTS TCP_SENDFILE UDP_STREAM". DLPI only supported by Unix, unix domain test is not necessary, so drop test of DLPI and unix domain.

Some general thoughts about this test:
* netperf2 has a fairly good wrapper code, that can do postprocessing
of the results. It'd be good to make use of it. What I was thinking:
- Import the netperf2 wrapper class on host and call run_once with
appropriate parameters on host
- Create a client control file and run it on guest using
kvm_test_utils.run_autotest()
- As this approach wouldn't work on windows, we could make a
netperf_windows test that would be very much like this test.

Of course, we could do that on a later time, I don't think it is something urgent anyway.

Quoted 24 lines

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/tests/netperf.py new file +14 -0
new file mode 100644
index 0000000..00a91f0
@@ -0,0 +1,56 @@
+import logging, commands, os
+from autotest_lib.client.common_lib import error
+import kvm_subprocess, kvm_test_utils, kvm_utils
+
+def run_netperf(test, params, env):
+    """
+    Network stress test with netperf
+
+    1) Boot up a virtual machine
+    2) Launch netserver on guest
+    3) Execute netperf client on host with different protocols
+    4) Outout the test result
+
+    @param test: Kvm test object

client/tests/kvm/tests/netperf.py line 24

Quoted 10 lines
client/tests/kvm/tests/netperf.py new file +10 -0
+    @param params: Dictionary with the test parameters.
+    @param env: Dictionary with test environment.
+    """
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    session = kvm_test_utils.wait_for_login(vm,
+                   timeout=int(params.get("login_timeout", 360)))
+    netperf_dir = os.path.join(os.environ['AUTODIR'], "tests/netperf2")
+    setup_cmd = params.get("setup_cmd")
+    guest_ip = vm.get_address()
+    result_file = os.path.join(test.debugdir, "result.txt")

^ test.resultsdir would be better. Also, it's better if we take into account the fact that we can have more than one iteration of the test, so we'd have something like:

self.results_path = os.path.join(self.resultsdir, 'raw_output_%s' %self.iteration)

Which is what we are doing currently for other benchmarks.

client/tests/kvm/tests/netperf.py line 26

Quoted 1 line
client/tests/kvm/tests/netperf.py new file +1 -0
+    session.get_command_output("service iptables stop")

^ Again I think it'd be better to simply flush the firewall rules with iptables -F because that's more likely to work on different linux guests.

Also, this linux specific command makes the entire test linux specific, so we have to put a rule telling the kvm autotest to not run it on windows guests.

Quoted 17 lines
client/tests/kvm/tests/netperf.py new file +17 -0
+    for i in params.get("netperf_files").split():
+        if not vm.copy_files_to(os.path.join(netperf_dir, i), "/tmp"):
+            raise error.TestError("Could not copy files to guest")
+    if session.get_command_status(setup_cmd % "/tmp", timeout=100) != 0:
+        raise error.TestFail("Fail to setup netperf on guest")
+    if session.get_command_status(params.get("netserver_cmd") % "/tmp") != 0:
+        raise error.TestFail("Fail to start netperf server on guest")
+
+    try:
+        logging.info("Setup and run netperf client on host")
+        s, o = commands.getstatusoutput(setup_cmd % netperf_dir)
+        if s != 0:
+            raise error.TestFail("Fail to setup netperf on host, o: %s" % o)
+        success = True
+        file(result_file, "w").write("Netperf Test Result\n")
+        for i in params.get("protocols").split():
+            cmd = params.get("netperf_cmd") % (netperf_dir, i, guest_ip)

client/tests/kvm/tests/netperf.py line 53

Quoted 10 lines
client/tests/kvm/tests/netperf.py new file +10 -0
+            logging.debug("Execute netperf client test: %s" % cmd)
+            s, o = commands.getstatusoutput(cmd)
+            if s != 0:
+                logging.error("Fail to execute netperf test, protocol:%s" % i)
+                success = False
+            else:
+                logging.info(o)
+                file(result_file, "a+").write("%s\n" % o)
+        if not success:
+            raise error.TestFail("Not all the test passed")

^ "Some of the netperf tests failed"

Quoted 26 lines
client/tests/kvm/tests/netperf.py new file +3 -0
+    finally:
+        session.get_command_output("killall netserver")
+        session.close()
client/tests/kvm/tests_base.cfg.sample +10 -0
index 7716d48..dec988e 100644
@@ -398,6 +398,16 @@ variants:
         type = mac_change
         kill_vm = yes
 
+    - netperf: install setup unattended_install.cdrom
+        type = netperf
+        nic_mode = tap
+        netperf_files = netperf-2.4.5.tar.bz2 wait_before_data.patch
+        setup_cmd = "cd %s && tar xvfj netperf-2.4.5.tar.bz2 && cd netperf-2.4.5 && patch -p0 < ../wait_before_data.patch && ./configure && make"
+        netserver_cmd =  %s/netperf-2.4.5/src/netserver
+        # test time is 60 seconds, set the buffer size to 1 for more hardware interrupt
+        netperf_cmd = %s/netperf-2.4.5/src/netperf -t %s -H %s -l 60 -- -m 1
+        protocols = "TCP_STREAM TCP_MAERTS TCP_RR TCP_CRR UDP_RR TCP_SENDFILE UDP_STREAM"
+
     - physical_resources_check: install setup unattended_install.cdrom
         type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'
author
#38 lore raw

RFC PATCH 13/14 KVM-test: Improve vlan subtest

In reply to Amos Kong

This is an enhancement of existed vlan test. Rename the vlan_tag.py to vlan.py, it is more reasonable.
. Setup arp from "/proc/sys/net/ipv4/conf/all/arp_ignore"
. Multiple vlans exist simultaneously
. Test ping between same and different vlans
. Test by TCP data transfer, floop ping between same vlan
. Maximal plumb/unplumb vlans

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/tests/vlan.py new file +181 -0
new file mode 100644
index 0000000..dc7611b
@@ -0,0 +1,181 @@
+import logging, time, re
+from autotest_lib.client.common_lib import error
+import kvm_subprocess, kvm_test_utils, kvm_utils, kvm_net_utils
+
+def run_vlan(test, params, env):
+    """
+    Test 802.1Q vlan of NIC, config it by vconfig command.
+
+    1) Create two VMs
+    2) Setup guests in 10 different vlans by vconfig and using hard-coded
+       ip address
+    3) Test by ping between same and different vlans of two VMs
+    4) Test by TCP data transfer, floop ping between same vlan of two VMs
+    5) Test maximal plumb/unplumb vlans
+    6) Recover the vlan config
+
+    @param test: KVM test object.
+    @param params: Dictionary with the test parameters.
+    @param env: Dictionary with test environment.
+    """
+
+    vm = []
+    session = []
+    vm_ip = []
+    digest_origin = []
+    vlan_ip = ['', '']
+    ip_unit = ['1', '2']
+    subnet = params.get("subnet")
+    vlan_num = int(params.get("vlan_num"))
+    maximal = int(params.get("maximal"))
+    file_size = params.get("file_size")
+
+    vm.append(kvm_test_utils.get_living_vm(env, params.get("main_vm")))
+    vm.append(kvm_test_utils.get_living_vm(env, "vm2"))
+
+    def add_vlan(session, id, iface="eth0"):
+        if session.get_command_status("vconfig add %s %s" % (iface, id)) != 0:
+            raise error.TestError("Fail to add %s.%s" % (iface, id))
+
+    def set_ip_vlan(session, id, ip, iface="eth0"):
+        iface = "%s.%s" % (iface, id)
+        if session.get_command_status("ifconfig %s %s" % (iface, ip)) != 0:
+            raise error.TestError("Fail to configure ip for %s" % iface)
+
+    def set_arp_ignore(session, iface="eth0"):
+        ignore_cmd = "echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore"
+        if session.get_command_status(ignore_cmd) != 0:
+            raise error.TestError("Fail to set arp_ignore of %s" % session)
+
+    def rem_vlan(session, id, iface="eth0"):
+        rem_vlan_cmd = "if [[ -e /proc/net/vlan/%s ]];then vconfig rem %s;fi"
+        iface = "%s.%s" % (iface, id)
+        s = session.get_command_status(rem_vlan_cmd % (iface, iface))
+        return s
+
+    def nc_transfer(src, dst):
+        nc_port = kvm_utils.find_free_port(1025, 5334, vm_ip[dst])
+        listen_cmd = params.get("listen_cmd")
+        send_cmd = params.get("send_cmd")
+
+        #listen in dst
+        listen_cmd = listen_cmd % (nc_port, "receive")
+        session[dst].sendline(listen_cmd)
+        time.sleep(2)
+        #send file from src to dst
+        send_cmd = send_cmd % (vlan_ip[dst], str(nc_port), "file")
+        if session[src].get_command_status(send_cmd, timeout = 60) != 0:
+            raise error.TestFail ("Fail to send file"
+                                    " from vm%s to vm%s" % (src+1, dst+1))
+        s, o = session[dst].read_up_to_prompt(timeout=60)
+        if s != True:
+            raise error.TestFail ("Fail to receive file"
+                                    " from vm%s to vm%s" % (src+1, dst+1))
+        #check MD5 message digest of receive file in dst
+        output = session[dst].get_command_output("md5sum receive").strip()
+        digest_receive = re.findall(r'(\w+)', output)[0]
+        if digest_receive == digest_origin[src]:
+            logging.info("file succeed received in vm %s" % vlan_ip[dst])
+        else:
+            logging.info("digest_origin is  %s" % digest_origin[src])
+            logging.info("digest_receive is %s" % digest_receive)
+            raise error.TestFail("File transfered differ from origin")
+        session[dst].get_command_status("rm -f receive")
+
+    for i in range(2):
+        session.append(kvm_test_utils.wait_for_login(vm[i],
+                       timeout=int(params.get("login_timeout", 360))))
+        if not session[i] :
+            raise error.TestError("Could not log into guest(vm%d)" % i)
+        logging.info("Logged in")
+
+        #get guest ip
+        vm_ip.append(vm[i].get_address())
+
+        #produce sized file in vm
+        dd_cmd = "dd if=/dev/urandom of=file bs=1024k count=%s"
+        if session[i].get_command_status(dd_cmd % file_size) != 0:
+            raise error.TestFail("File producing failed")
+        #record MD5 message digest of file
+        s, output =session[i].get_command_status_output("md5sum file",
+                                                        timeout=60)
+        if s != 0:
+            raise error.TestFail("File MD5_checking failed" )
+        digest_origin.append(re.findall(r'(\w+)', output)[0])
+
+        #stop firewall in vm
+        session[i].get_command_status("/etc/init.d/iptables stop")
+
+        #load 8021q module for vconfig
+        load_8021q_cmd = "modprobe 8021q"
+        if session[i].get_command_status(load_8021q_cmd) != 0:
+            raise error.TestError("Fail to load 8021q module on VM%s" % i)
+
+    try:
+        for i in range(2):
+            for vlan_i in range(1, vlan_num+1):
+                add_vlan(session[i], vlan_i)
+                set_ip_vlan(session[i], vlan_i, "%s.%s.%s" % \
+                                          (subnet, vlan_i, ip_unit[i]))
+            set_arp_ignore(session[i])
+
+        for vlan in range(1, vlan_num+1):
+            logging.info("Test for vlan %s" % vlan)
+
+            logging.info("Ping between vlans")
+            interface = 'eth0.' + str(vlan)
+            for vlan2 in range(1, vlan_num+1):
+                for i in range(2):
+                    dest = subnet +'.'+ str(vlan2)+ '.' + ip_unit[(i+1)%2]
+                    s, o = kvm_net_utils.ping(dest, count=2,
+                                              interface=interface,
+                                              session=session[i], timeout=30)
+                    if ((vlan == vlan2) ^ (s == 0)):
+                        raise error.TestFail ("%s ping %s unexpected" %
+                                                    (interface, dest))
+
+            vlan_ip[0] = subnet + '.' + str(vlan) + '.' + ip_unit[0]
+            vlan_ip[1] = subnet + '.' + str(vlan) + '.' + ip_unit[1]
+
+            logging.info("Flood ping")
+            def flood_ping(src, dst):
+                # we must use a dedicated session becuase the kvm_subprocess
+                # does not have the other method to interrupt the process in
+                # the guest rather than close the session.
+                session_flood = kvm_test_utils.wait_for_login(vm[src],
+                                                              timeout = 60)
+                kvm_net_utils.ping(vlan_ip[dst], flood=True,
+                                   interface=interface,
+                                   session=session_flood, timeout=10)
+                session_flood.close()
+
+            flood_ping(0,1)
+            flood_ping(1,0)
+
+            logging.info("Transfering data through nc")
+            nc_transfer(0, 1)
+            nc_transfer(1, 0)
+
+    finally:
+        for vlan in range(1, vlan_num+1):
+            rem_vlan(session[0], vlan)
+            rem_vlan(session[1], vlan)
+            logging.info("rem eth0.%s" % vlan)
+
+    #Plumb/unplumb maximal unber of vlan interfaces
+    i = 1
+    s = 0
+    try:
+        logging.info("Testing the plumb of vlan interface")
+        for i in range (1, maximal+1):
+            add_vlan(session[0], i)
+    finally:
+        for j in range (1, i+1):
+            s = s or rem_vlan(session[0], j)
+        if s == 0:
+            logging.info("maximal interface plumb test done")
+        else:
+            logging.error("maximal interface plumb test failed")
+
+    session[0].close()
+    session[1].close()
client/tests/kvm/tests/vlan_tag.py deleted +0 -68

68 lines not on this page · raw message

client/tests/kvm/tests_base.cfg.sample +10 -6
index dec988e..b25980e 100644
@@ -337,17 +337,21 @@ variants:
             - fmt_raw:
                 image_format_stg = raw
 
-    - vlan_tag:  install setup unattended_install.cdrom
-        type = vlan_tag
+    - vlan:  install setup unattended_install.cdrom
+        type = vlan
         # subnet should not be used by host
-        subnet = 192.168.123
-        vlans = "10 20"
+        subnet = "192.168"
+        vlan_num = 5
+        file_size = 10
+        maximal = 4094
+        listen_cmd = "nc -l %s > %s"
+        send_cmd = "nc %s %s < %s"
         nic_mode = tap
         vms += " vm2"
         extra_params_vm1 += " -snapshot"
         extra_params_vm2 += " -snapshot"
+        kill_vm_vm2 = yes
         kill_vm_gracefully_vm2 = no
-        address_index_vm2 = 1
 
     - ping: install setup unattended_install.cdrom
         type = ping
@@ -1084,7 +1088,7 @@ variants:
 
     # Windows section
     - @Windows:
-        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer nicdriver_unload nic_promisc multicast mac_change
+        no autotest linux_s3 vlan ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer nicdriver_unload nic_promisc multicast mac_change
         shutdown_command = shutdown /s /f /t 0
         reboot_command = shutdown /r /f /t 0
         status_test_command = echo %errorlevel%
author
#39 lore raw

RFC PATCH 14/14 KVM-test: Add subtest of testing offload by ethtool

In reply to Amos Kong

The latest case contains TX/RX/SG/TSO/GSO/GRO/LRO test. RTL8139 NIC doesn't support TSO, LRO, it's too old, so drop offload test from rtl8139. LRO, GRO are only supported by latest kernel, virtio nic doesn't support receive offloading function.
Initialize the callbacks first and execute all the sub tests one by one, all the result will be check at the end.
When execute this test, vhost should be enabled, then most of new feature can be used. Vhost doestn't support VIRTIO_NET_F_MRG_RXBUF, so do not check large packets in received offload test.
Transfer files by scp between host and guest, match new opened TCP port by netstat. Capture the packages info by tcpdump, it contains package length.

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/tests/ethtool.py new file +205 -0
new file mode 100644
index 0000000..7274eae
@@ -0,0 +1,205 @@
+import time, os, logging, commands, re
+from autotest_lib.client.common_lib import error
+from autotest_lib.client.bin import utils
+import kvm_test_utils, kvm_utils, kvm_net_utils
+
+def run_ethtool(test, params, env):
+    """
+    Test offload functions of ethernet device by ethtool
+
+    1) Log into a guest
+    2) Initialize the callback of sub functions
+    3) Enable/disable sub function of NIC
+    4) Execute callback function
+    5) Check the return value
+    6) Restore original configuration
+
+    @param test: Kvm test object
+    @param params: Dictionary with the test parameters.
+    @param env: Dictionary with test environment.
+    """
+    def ethtool_get(type):
+        feature_pattern = {
+            'tx':  'tx.*checksumming',
+            'rx':  'rx.*checksumming',
+            'sg':  'scatter.*gather',
+            'tso': 'tcp.*segmentation.*offload',
+            'gso': 'generic.*segmentation.*offload',
+            'gro': 'generic.*receive.*offload',
+            'lro': 'large.*receive.*offload',
+            }
+        s, o = session.get_command_status_output("ethtool -k %s" % ethname)
+        try:
+            return re.findall("%s: (.*)" % feature_pattern.get(type), o)[0]
+        except IndexError:
+            logging.debug("Could not get %s status" % type)
+
+    def ethtool_set(type, status):
+        """
+        Set ethernet device offload status
+
+        @param type: Offload type name
+        @param status: New status will be changed to
+        """
+        logging.info("Try to set %s %s" % (type, status))
+        if status not in ["off", "on"]:
+            return False
+        cmd = "ethtool -K %s %s %s" % (ethname, type, status)
+        if ethtool_get(type) != status:
+            return session.get_command_status(cmd) == 0
+        if ethtool_get(type) != status:
+            logging.error("Fail to set %s %s" % (type, status))
+            return False
+        return True
+
+    def ethtool_save_params():
+        logging.info("Save ethtool configuration")
+        for i in supported_features:
+            feature_status[i] = ethtool_get(i)
+
+    def ethtool_restore_params():
+        logging.info("Restore ethtool configuration")
+        for i in supported_features:
+            ethtool_set(i, feature_status[i])
+
+    def compare_md5sum(name):
+        logging.info("Compare md5sum of the files on guest and host")
+        host_result = utils.hash_file(name, method="md5")
+        try:
+            o = session.get_command_output("md5sum %s" % name)
+            guest_result = re.findall("\w+", o)[0]
+        except IndexError:
+            logging.error("Could not get file md5sum in guest")
+            return False
+        logging.debug("md5sum: guest(%s), host(%s)" % (guest_result,
+                                                   host_result))
+        return guest_result == host_result
+
+    def transfer_file(src="guest"):
+        """
+        Transfer file by scp, use tcpdump to capture packets, then check the
+        return string.
+
+        @param src: Source host of transfer file
+        @return: Tuple (status, error msg/tcpdump result)
+        """
+        session2.get_command_status("rm -rf %s" % filename)
+        dd_cmd = "dd if=/dev/urandom of=%s bs=1M count=%s" % (filename,
+                                                   params.get("filesize"))
+        logging.info("Creat file in source host, cmd: %s" % dd_cmd)
+        tcpdump_cmd = "tcpdump -lep -s 0 tcp -vv port ssh"
+        if src == "guest":
+            s = session.get_command_status(dd_cmd, timeout=360)
+            tcpdump_cmd += " and src %s" % guest_ip
+            copy_files_fun = vm.copy_files_from
+        else:
+            s, o = commands.getstatusoutput(dd_cmd)
+            tcpdump_cmd += " and dst %s" % guest_ip
+            copy_files_fun = vm.copy_files_to
+        if s != 0:
+            return (False, "Fail to create file by dd, cmd: %s" % dd_cmd)
+
+        # only capture the new tcp port after offload setup
+        original_tcp_ports = re.findall("tcp.*:(\d+).*%s" % guest_ip,
+                                       commands.getoutput("/bin/netstat -nap"))
+        for i in original_tcp_ports:
+            tcpdump_cmd += " and not port %s" % i
+        logging.debug("Listen by command: %s" % tcpdump_cmd)
+        session2.sendline(tcpdump_cmd)
+        if not kvm_utils.wait_for(lambda: session.get_command_status(
+                                           "pgrep tcpdump") == 0, 30):
+            return (False, "Tcpdump process wasn't launched")
+
+        logging.info("Start to transfer file")
+        if not copy_files_fun(filename, filename):
+            return (False, "Child process transfer file failed")
+        logging.info("Transfer file completed")
+        if session.get_command_status("killall tcpdump") != 0:
+            return (False, "Could not kill all tcpdump process")
+        s, tcpdump_string = session2.read_up_to_prompt(timeout=60)
+        if not s:
+            return (False, "Fail to read tcpdump's output")
+
+        if not compare_md5sum(filename):
+            return (False, "Files' md5sum mismatched")
+        return (True, tcpdump_string)
+
+    def tx_callback(status="on"):
+        s, o = transfer_file(src="guest")
+        if not s:
+            logging.error(o)
+            return False
+        return True
+
+    def rx_callback(status="on"):
+        s, o = transfer_file(src="host")
+        if not s:
+            logging.error(o)
+            return False
+        return True
+
+    def so_callback(status="on"):
+        s, o = transfer_file(src="guest")
+        if not s:
+            logging.error(o)
+            return False
+        logging.info("Check if contained large frame")
+        # mtu: default IPv4 MTU is 1500 Bytes, ethernet header is 14 Bytes
+        return (status == "on") ^ (len([i for i in re.findall(
+                                   "length (\d*):", o) if int(i) > mtu]) == 0)
+
+    def ro_callback(status="on"):
+        s, o = transfer_file(src="host")
+        if not s:
+            logging.error(o)
+            return False
+        return True
+
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    session = kvm_test_utils.wait_for_login(vm,
+                  timeout=int(params.get("login_timeout", 360)))
+    session2 = kvm_test_utils.wait_for_login(vm,
+                  timeout=int(params.get("login_timeout", 360)))
+    mtu = 1514
+    feature_status = {}
+    filename = "/tmp/ethtool.dd"
+    guest_ip = vm.get_address()
+    ethname = kvm_net_utils.get_linux_ifname(session, vm.get_macaddr(0))
+    supported_features = params.get("supported_features").split()
+    test_matrix = {
+        # type:(callback,    (dependence), (exclude)
+        "tx":  (tx_callback, (), ()),
+        "rx":  (rx_callback, (), ()),
+        "sg":  (tx_callback, ("tx",), ()),
+        "tso": (so_callback, ("tx", "sg",), ("gso",)),
+        "gso": (so_callback, (), ("tso",)),
+        "gro": (ro_callback, ("rx",), ("lro",)),
+        "lro": (rx_callback, (), ("gro",)),
+        }
+    ethtool_save_params()
+    success = True
+    try:
+        for type in supported_features:
+            callback = test_matrix[type][0]
+            for i in test_matrix[type][2]:
+                if not ethtool_set(i, "off"):
+                    logging.error("Fail to disable %s" % i)
+                    success = False
+            for i in [f for f in test_matrix[type][1]] + [type]:
+                if not ethtool_set(i, "on"):
+                    logging.error("Fail to enable %s" % i)
+                    success = False
+            if not callback():
+                raise error.TestFail("Test failed, %s: on" % type)
+
+            if not ethtool_set(type, "off"):
+                logging.error("Fail to disable %s" % type)
+                success = False
+            if not callback(status="off"):
+                raise error.TestFail("Test failed, %s: off" % type)
+        if not success:
+            raise error.TestError("Enable/disable offload function fail")
+    finally:
+        ethtool_restore_params()
+        session.close()
+        session2.close()
client/tests/kvm/tests_base.cfg.sample +14 -1
index b25980e..6f0e295 100644
@@ -412,6 +412,19 @@ variants:
         netperf_cmd = %s/netperf-2.4.5/src/netperf -t %s -H %s -l 60 -- -m 1
         protocols = "TCP_STREAM TCP_MAERTS TCP_RR TCP_CRR UDP_RR TCP_SENDFILE UDP_STREAM"
 
+    - ethtool: install setup unattended_install.cdrom
+        type = ethtool
+        filesize = 512
+        nic_mode = tap
+        variants:
+            # gso gro lro is only supported by latest kernel
+            - nic_virtio:
+                pci_model = virtio
+                supported_features = "tx sg tso gso"
+            - nic_e1000:
+                pci_model = e1000
+                supported_features = "tx rx sg tso gso gro lro"
+
     - physical_resources_check: install setup unattended_install.cdrom
         type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'
@@ -1088,7 +1101,7 @@ variants:
 
     # Windows section
     - @Windows:
-        no autotest linux_s3 vlan ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer nicdriver_unload nic_promisc multicast mac_change
+        no autotest linux_s3 vlan ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer nicdriver_unload nic_promisc multicast mac_change ethtool
         shutdown_command = shutdown /s /f /t 0
         reboot_command = shutdown /r /f /t 0
         status_test_command = echo %errorlevel%
#40 lore raw

Re: [RFC PATCH 14/14] KVM-test: Add subtest of testing offload by ethtool

Quoted 11 lines · Amos Kong

The latest case contains TX/RX/SG/TSO/GSO/GRO/LRO test. RTL8139 NIC doesn't support TSO, LRO, it's too old, so drop offload test from rtl8139. LRO, GRO are only supported by latest kernel, virtio nic doesn't support receive offloading function.
Initialize the callbacks first and execute all the sub tests one by one, all the result will be check at the end.
When execute this test, vhost should be enabled, then most of new feature can be used. Vhost doestn't support VIRTIO_NET_F_MRG_RXBUF, so do not check large packets in received offload test.
Transfer files by scp between host and guest, match new opened TCP port by netstat. Capture the packages info by tcpdump, it contains package length.

This test is heavily dependent on ethtool, so we need to make sure the package is going to be installed on linux guests. The default package selection for Fedora 13 does not include it for example. So, we need to modify linux guest kickstarts/XMLs to add ethtool to the default package selection.

Quoted 168 lines

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/tests/ethtool.py new file +158 -0
new file mode 100644
index 0000000..7274eae
@@ -0,0 +1,205 @@
+import time, os, logging, commands, re
+from autotest_lib.client.common_lib import error
+from autotest_lib.client.bin import utils
+import kvm_test_utils, kvm_utils, kvm_net_utils
+
+def run_ethtool(test, params, env):
+    """
+    Test offload functions of ethernet device by ethtool
+
+    1) Log into a guest
+    2) Initialize the callback of sub functions
+    3) Enable/disable sub function of NIC
+    4) Execute callback function
+    5) Check the return value
+    6) Restore original configuration
+
+    @param test: Kvm test object
+    @param params: Dictionary with the test parameters.
+    @param env: Dictionary with test environment.
+    """
+    def ethtool_get(type):
+        feature_pattern = {
+            'tx':  'tx.*checksumming',
+            'rx':  'rx.*checksumming',
+            'sg':  'scatter.*gather',
+            'tso': 'tcp.*segmentation.*offload',
+            'gso': 'generic.*segmentation.*offload',
+            'gro': 'generic.*receive.*offload',
+            'lro': 'large.*receive.*offload',
+            }
+        s, o = session.get_command_status_output("ethtool -k %s" % ethname)
+        try:
+            return re.findall("%s: (.*)" % feature_pattern.get(type), o)[0]
+        except IndexError:
+            logging.debug("Could not get %s status" % type)
+
+    def ethtool_set(type, status):
+        """
+        Set ethernet device offload status
+
+        @param type: Offload type name
+        @param status: New status will be changed to
+        """
+        logging.info("Try to set %s %s" % (type, status))
+        if status not in ["off", "on"]:
+            return False
+        cmd = "ethtool -K %s %s %s" % (ethname, type, status)
+        if ethtool_get(type) != status:
+            return session.get_command_status(cmd) == 0
+        if ethtool_get(type) != status:
+            logging.error("Fail to set %s %s" % (type, status))
+            return False
+        return True
+
+    def ethtool_save_params():
+        logging.info("Save ethtool configuration")
+        for i in supported_features:
+            feature_status[i] = ethtool_get(i)
+
+    def ethtool_restore_params():
+        logging.info("Restore ethtool configuration")
+        for i in supported_features:
+            ethtool_set(i, feature_status[i])
+
+    def compare_md5sum(name):
+        logging.info("Compare md5sum of the files on guest and host")
+        host_result = utils.hash_file(name, method="md5")
+        try:
+            o = session.get_command_output("md5sum %s" % name)
+            guest_result = re.findall("\w+", o)[0]
+        except IndexError:
+            logging.error("Could not get file md5sum in guest")
+            return False
+        logging.debug("md5sum: guest(%s), host(%s)" % (guest_result,
+                                                   host_result))
+        return guest_result == host_result
+
+    def transfer_file(src="guest"):
+        """
+        Transfer file by scp, use tcpdump to capture packets, then check the
+        return string.
+
+        @param src: Source host of transfer file
+        @return: Tuple (status, error msg/tcpdump result)
+        """
+        session2.get_command_status("rm -rf %s" % filename)
+        dd_cmd = "dd if=/dev/urandom of=%s bs=1M count=%s" % (filename,
+                                                   params.get("filesize"))
+        logging.info("Creat file in source host, cmd: %s" % dd_cmd)
+        tcpdump_cmd = "tcpdump -lep -s 0 tcp -vv port ssh"
+        if src == "guest":
+            s = session.get_command_status(dd_cmd, timeout=360)
+            tcpdump_cmd += " and src %s" % guest_ip
+            copy_files_fun = vm.copy_files_from
+        else:
+            s, o = commands.getstatusoutput(dd_cmd)
+            tcpdump_cmd += " and dst %s" % guest_ip
+            copy_files_fun = vm.copy_files_to
+        if s != 0:
+            return (False, "Fail to create file by dd, cmd: %s" % dd_cmd)
+
+        # only capture the new tcp port after offload setup
+        original_tcp_ports = re.findall("tcp.*:(\d+).*%s" % guest_ip,
+                                       commands.getoutput("/bin/netstat -nap"))
+        for i in original_tcp_ports:
+            tcpdump_cmd += " and not port %s" % i
+        logging.debug("Listen by command: %s" % tcpdump_cmd)
+        session2.sendline(tcpdump_cmd)
+        if not kvm_utils.wait_for(lambda: session.get_command_status(
+                                           "pgrep tcpdump") == 0, 30):
+            return (False, "Tcpdump process wasn't launched")
+
+        logging.info("Start to transfer file")
+        if not copy_files_fun(filename, filename):
+            return (False, "Child process transfer file failed")
+        logging.info("Transfer file completed")
+        if session.get_command_status("killall tcpdump") != 0:
+            return (False, "Could not kill all tcpdump process")
+        s, tcpdump_string = session2.read_up_to_prompt(timeout=60)
+        if not s:
+            return (False, "Fail to read tcpdump's output")
+
+        if not compare_md5sum(filename):
+            return (False, "Files' md5sum mismatched")
+        return (True, tcpdump_string)
+
+    def tx_callback(status="on"):
+        s, o = transfer_file(src="guest")
+        if not s:
+            logging.error(o)
+            return False
+        return True
+
+    def rx_callback(status="on"):
+        s, o = transfer_file(src="host")
+        if not s:
+            logging.error(o)
+            return False
+        return True
+
+    def so_callback(status="on"):
+        s, o = transfer_file(src="guest")
+        if not s:
+            logging.error(o)
+            return False
+        logging.info("Check if contained large frame")
+        # mtu: default IPv4 MTU is 1500 Bytes, ethernet header is 14 Bytes
+        return (status == "on") ^ (len([i for i in re.findall(
+                                   "length (\d*):", o) if int(i) > mtu]) == 0)
+
+    def ro_callback(status="on"):
+        s, o = transfer_file(src="host")
+        if not s:
+            logging.error(o)
+            return False
+        return True
+
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))

client/tests/kvm/tests/ethtool.py line 168

Quoted 10 lines
client/tests/kvm/tests/ethtool.py new file +10 -0
+    session = kvm_test_utils.wait_for_login(vm,
+                  timeout=int(params.get("login_timeout", 360)))
+    session2 = kvm_test_utils.wait_for_login(vm,
+                  timeout=int(params.get("login_timeout", 360)))
+    mtu = 1514
+    feature_status = {}
+    filename = "/tmp/ethtool.dd"
+    guest_ip = vm.get_address()
+    ethname = kvm_net_utils.get_linux_ifname(session, vm.get_macaddr(0))
+    supported_features = params.get("supported_features").split()

^ We could use ethtool to query the machine for the supported features, and if it's not possible to do that (virtio_net), we resort to supported_feature being set on the config file, what do you think?

Quoted 71 lines
client/tests/kvm/tests/ethtool.py new file +37 -0
+    test_matrix = {
+        # type:(callback,    (dependence), (exclude)
+        "tx":  (tx_callback, (), ()),
+        "rx":  (rx_callback, (), ()),
+        "sg":  (tx_callback, ("tx",), ()),
+        "tso": (so_callback, ("tx", "sg",), ("gso",)),
+        "gso": (so_callback, (), ("tso",)),
+        "gro": (ro_callback, ("rx",), ("lro",)),
+        "lro": (rx_callback, (), ("gro",)),
+        }
+    ethtool_save_params()
+    success = True
+    try:
+        for type in supported_features:
+            callback = test_matrix[type][0]
+            for i in test_matrix[type][2]:
+                if not ethtool_set(i, "off"):
+                    logging.error("Fail to disable %s" % i)
+                    success = False
+            for i in [f for f in test_matrix[type][1]] + [type]:
+                if not ethtool_set(i, "on"):
+                    logging.error("Fail to enable %s" % i)
+                    success = False
+            if not callback():
+                raise error.TestFail("Test failed, %s: on" % type)
+
+            if not ethtool_set(type, "off"):
+                logging.error("Fail to disable %s" % type)
+                success = False
+            if not callback(status="off"):
+                raise error.TestFail("Test failed, %s: off" % type)
+        if not success:
+            raise error.TestError("Enable/disable offload function fail")
+    finally:
+        ethtool_restore_params()
+        session.close()
+        session2.close()
client/tests/kvm/tests_base.cfg.sample +14 -1
index b25980e..6f0e295 100644
@@ -412,6 +412,19 @@ variants:
         netperf_cmd = %s/netperf-2.4.5/src/netperf -t %s -H %s -l 60 -- -m 1
         protocols = "TCP_STREAM TCP_MAERTS TCP_RR TCP_CRR UDP_RR TCP_SENDFILE UDP_STREAM"
 
+    - ethtool: install setup unattended_install.cdrom
+        type = ethtool
+        filesize = 512
+        nic_mode = tap
+        variants:
+            # gso gro lro is only supported by latest kernel
+            - nic_virtio:
+                pci_model = virtio
+                supported_features = "tx sg tso gso"
+            - nic_e1000:
+                pci_model = e1000
+                supported_features = "tx rx sg tso gso gro lro"
+
     - physical_resources_check: install setup unattended_install.cdrom
         type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'
@@ -1088,7 +1101,7 @@ variants:
 
     # Windows section
     - @Windows:
-        no autotest linux_s3 vlan ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer nicdriver_unload nic_promisc multicast mac_change
+        no autotest linux_s3 vlan ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer nicdriver_unload nic_promisc multicast mac_change ethtool
         shutdown_command = shutdown /s /f /t 0
         reboot_command = shutdown /r /f /t 0
         status_test_command = echo %errorlevel%
#41 lore raw

Re: [RFC PATCH 14/14] KVM-test: Add subtest of testing offload by ethtool

Quoted 13 lines · Lucas Meneghel Rodrigues
Quoted 11 lines · Amos Kong

The latest case contains TX/RX/SG/TSO/GSO/GRO/LRO test. RTL8139 NIC doesn't support TSO, LRO, it's too old, so drop offload test from rtl8139. LRO, GRO are only supported by latest kernel, virtio nic doesn't support receive offloading function.
Initialize the callbacks first and execute all the sub tests one by one, all the result will be check at the end.
When execute this test, vhost should be enabled, then most of new feature can be used. Vhost doestn't support VIRTIO_NET_F_MRG_RXBUF, so do not check large packets in received offload test.
Transfer files by scp between host and guest, match new opened TCP port by netstat. Capture the packages info by tcpdump, it contains package length.

Quoted 5 lines

This test is heavily dependent on ethtool, so we need to make sure the package is going to be installed on linux guests. The default package selection for Fedora 13 does not include it for example. So, we need to modify linux guest kickstarts/XMLs to add ethtool to the default package selection.

OK.

Quoted 174 lines
Quoted 174 lines

---

 0 files changed, 0 insertions(+), 0 deletions(-)
client/tests/kvm/tests/ethtool.py new file +164 -0
new file mode 100644
index 0000000..7274eae
@@ -0,0 +1,205 @@
+import time, os, logging, commands, re
+from autotest_lib.client.common_lib import error
+from autotest_lib.client.bin import utils
+import kvm_test_utils, kvm_utils, kvm_net_utils
+
+def run_ethtool(test, params, env):
+    """
+    Test offload functions of ethernet device by ethtool
+
+    1) Log into a guest
+    2) Initialize the callback of sub functions
+    3) Enable/disable sub function of NIC
+    4) Execute callback function
+    5) Check the return value
+    6) Restore original configuration
+
+    @param test: Kvm test object
+    @param params: Dictionary with the test parameters.
+    @param env: Dictionary with test environment.
+    """
+    def ethtool_get(type):
+        feature_pattern = {
+            'tx':  'tx.*checksumming',
+            'rx':  'rx.*checksumming',
+            'sg':  'scatter.*gather',
+            'tso': 'tcp.*segmentation.*offload',
+            'gso': 'generic.*segmentation.*offload',
+            'gro': 'generic.*receive.*offload',
+            'lro': 'large.*receive.*offload',
+            }
+        s, o = session.get_command_status_output("ethtool -k %s" % ethname)
+        try:
+            return re.findall("%s: (.*)" % feature_pattern.get(type), o)[0]
+        except IndexError:
+            logging.debug("Could not get %s status" % type)
+
+    def ethtool_set(type, status):
+        """
+        Set ethernet device offload status
+
+        @param type: Offload type name
+        @param status: New status will be changed to
+        """
+        logging.info("Try to set %s %s" % (type, status))
+        if status not in ["off", "on"]:
+            return False
+        cmd = "ethtool -K %s %s %s" % (ethname, type, status)
+        if ethtool_get(type) != status:
+            return session.get_command_status(cmd) == 0
+        if ethtool_get(type) != status:
+            logging.error("Fail to set %s %s" % (type, status))
+            return False
+        return True
+
+    def ethtool_save_params():
+        logging.info("Save ethtool configuration")
+        for i in supported_features:
+            feature_status[i] = ethtool_get(i)
+
+    def ethtool_restore_params():
+        logging.info("Restore ethtool configuration")
+        for i in supported_features:
+            ethtool_set(i, feature_status[i])
+
+    def compare_md5sum(name):
+        logging.info("Compare md5sum of the files on guest and host")
+        host_result = utils.hash_file(name, method="md5")
+        try:
+            o = session.get_command_output("md5sum %s" % name)
+            guest_result = re.findall("\w+", o)[0]
+        except IndexError:
+            logging.error("Could not get file md5sum in guest")
+            return False
+        logging.debug("md5sum: guest(%s), host(%s)" % (guest_result,
+                                                   host_result))
+        return guest_result == host_result
+
+    def transfer_file(src="guest"):
+        """
+        Transfer file by scp, use tcpdump to capture packets, then check the
+        return string.
+
+        @param src: Source host of transfer file
+        @return: Tuple (status, error msg/tcpdump result)
+        """
+        session2.get_command_status("rm -rf %s" % filename)
+        dd_cmd = "dd if=/dev/urandom of=%s bs=1M count=%s" % (filename,
+                                                   params.get("filesize"))
+        logging.info("Creat file in source host, cmd: %s" % dd_cmd)
+        tcpdump_cmd = "tcpdump -lep -s 0 tcp -vv port ssh"
+        if src == "guest":
+            s = session.get_command_status(dd_cmd, timeout=360)
+            tcpdump_cmd += " and src %s" % guest_ip
+            copy_files_fun = vm.copy_files_from
+        else:
+            s, o = commands.getstatusoutput(dd_cmd)
+            tcpdump_cmd += " and dst %s" % guest_ip
+            copy_files_fun = vm.copy_files_to
+        if s != 0:
+            return (False, "Fail to create file by dd, cmd: %s" % dd_cmd)
+
+        # only capture the new tcp port after offload setup
+        original_tcp_ports = re.findall("tcp.*:(\d+).*%s" % guest_ip,
+                                       commands.getoutput("/bin/netstat -nap"))
+        for i in original_tcp_ports:
+            tcpdump_cmd += " and not port %s" % i
+        logging.debug("Listen by command: %s" % tcpdump_cmd)
+        session2.sendline(tcpdump_cmd)
+        if not kvm_utils.wait_for(lambda: session.get_command_status(
+                                           "pgrep tcpdump") == 0, 30):
+            return (False, "Tcpdump process wasn't launched")
+
+        logging.info("Start to transfer file")
+        if not copy_files_fun(filename, filename):
+            return (False, "Child process transfer file failed")
+        logging.info("Transfer file completed")
+        if session.get_command_status("killall tcpdump") != 0:
+            return (False, "Could not kill all tcpdump process")
+        s, tcpdump_string = session2.read_up_to_prompt(timeout=60)
+        if not s:
+            return (False, "Fail to read tcpdump's output")
+
+        if not compare_md5sum(filename):
+            return (False, "Files' md5sum mismatched")
+        return (True, tcpdump_string)
+
+    def tx_callback(status="on"):
+        s, o = transfer_file(src="guest")
+        if not s:
+            logging.error(o)
+            return False
+        return True
+
+    def rx_callback(status="on"):
+        s, o = transfer_file(src="host")
+        if not s:
+            logging.error(o)
+            return False
+        return True
+
+    def so_callback(status="on"):
+        s, o = transfer_file(src="guest")
+        if not s:
+            logging.error(o)
+            return False
+        logging.info("Check if contained large frame")
+        # mtu: default IPv4 MTU is 1500 Bytes, ethernet header is 14 Bytes
+        return (status == "on") ^ (len([i for i in re.findall(
+                                   "length (\d*):", o) if int(i) > mtu]) == 0)
+
+    def ro_callback(status="on"):
+        s, o = transfer_file(src="host")
+        if not s:
+            logging.error(o)
+            return False
+        return True
+
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    session = kvm_test_utils.wait_for_login(vm,
+                  timeout=int(params.get("login_timeout", 360)))
+    session2 = kvm_test_utils.wait_for_login(vm,
+                  timeout=int(params.get("login_timeout", 360)))
+    mtu = 1514
+    feature_status = {}
Quoted 8 lines

client/tests/kvm/tests/ethtool.py line 168

Quoted 4 lines
client/tests/kvm/tests/ethtool.py new file +4 -0
+    filename = "/tmp/ethtool.dd"
+    guest_ip = vm.get_address()
+    ethname = kvm_net_utils.get_linux_ifname(session, vm.get_macaddr(0))
+    supported_features = params.get("supported_features").split()

^ We could use ethtool to query the machine for the supported features, and if it's not possible to do that (virtio_net), we resort to supported_feature being set on the config file, what do you think?

How to query which offload features are supported by nic dev by ethtool ? '# ethtool -k eth0' can only list the function status.
I try to enable all the function, by this order 'tx rx gs tso ufo gso gro lro', if some of them failed, it should not be supported ? This method is not dependable in test environment.

Quoted 70 lines
Quoted 70 lines
client/tests/kvm/tests/ethtool.py new file +37 -0
+    test_matrix = {
+        # type:(callback,    (dependence), (exclude)
+        "tx":  (tx_callback, (), ()),
+        "rx":  (rx_callback, (), ()),
+        "sg":  (tx_callback, ("tx",), ()),
+        "tso": (so_callback, ("tx", "sg",), ("gso",)),
+        "gso": (so_callback, (), ("tso",)),
+        "gro": (ro_callback, ("rx",), ("lro",)),
+        "lro": (rx_callback, (), ("gro",)),
+        }
+    ethtool_save_params()
+    success = True
+    try:
+        for type in supported_features:
+            callback = test_matrix[type][0]
+            for i in test_matrix[type][2]:
+                if not ethtool_set(i, "off"):
+                    logging.error("Fail to disable %s" % i)
+                    success = False
+            for i in [f for f in test_matrix[type][1]] + [type]:
+                if not ethtool_set(i, "on"):
+                    logging.error("Fail to enable %s" % i)
+                    success = False
+            if not callback():
+                raise error.TestFail("Test failed, %s: on" % type)
+
+            if not ethtool_set(type, "off"):
+                logging.error("Fail to disable %s" % type)
+                success = False
+            if not callback(status="off"):
+                raise error.TestFail("Test failed, %s: off" % type)
+        if not success:
+            raise error.TestError("Enable/disable offload function fail")
+    finally:
+        ethtool_restore_params()
+        session.close()
+        session2.close()
client/tests/kvm/tests_base.cfg.sample +0 -0
index b25980e..6f0e295 100644
@@ -412,6 +412,19 @@ variants:

         netperf_cmd = %s/netperf-2.4.5/src/netperf -t %s -H %s -l 60 -- -m 1 protocols = "TCP_STREAM TCP_MAERTS TCP_RR TCP_CRR UDP_RR TCP_SENDFILE UDP_STREAM"

+    - ethtool: install setup unattended_install.cdrom
+        type = ethtool
+        filesize = 512
+        nic_mode = tap
+        variants:
+            # gso gro lro is only supported by latest kernel
+            - nic_virtio:
+                pci_model = virtio
+                supported_features = "tx sg tso gso"
+            - nic_e1000:
+                pci_model = e1000
+                supported_features = "tx rx sg tso gso gro lro" +
     - physical_resources_check: install setup unattended_install.cdrom type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'

client/tests/kvm/tests_base.cfg.sample +0 -0
@@ -1088,7 +1101,7 @@ variants:
 
      # Windows section

     - @Windows:

-        no autotest linux_s3 vlan ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer nicdriver_unload nic_promisc multicast mac_change
+        no autotest linux_s3 vlan ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer nicdriver_unload nic_promisc multicast mac_change ethtool

         shutdown_command = shutdown /s /f /t 0
         reboot_command = shutdown /r /f /t 0
         status_test_command = echo %errorlevel%

#42 lore raw
In reply to Amos Kong
Quoted 2 lines · Amos Kong

The following series contain 11 network related subtests, welcome to give me some suggestions about correctness, design, enhancement.

Awesome work, will start to review them today. Thanks!

Quoted 23 lines

Thank you so much!

---

Amos Kong (14):
KVM-test: Add a new macaddress pool algorithm
KVM Test: Add a function get_interface_name() to kvm_net_utils.py KVM Test: Add a common ping module for network related tests
KVM-test: Add a new subtest ping
KVM-test: Add a subtest jumbo
KVM-test: Add basic file transfer test
KVM-test: Add a subtest of load/unload nic driver
KVM-test: Add a subtest of nic promisc
KVM-test: Add a subtest of multicast
KVM-test: Add a subtest of pxe
KVM-test: Add a subtest of changing mac address
KVM-test: Add a subtest of netperf
KVM-test: Improve vlan subtest
KVM-test: Add subtest of testing offload by ethtool

 0 files changed, 0 insertions(+), 0 deletions(-)
#43 lore raw
In reply to Amos Kong
Quoted 4 lines · Amos Kong

The following series contain 11 network related subtests, welcome to give me some suggestions about correctness, design, enhancement.

Thank you so much!

Ok Amos, now that I made the first review of this patchset, I'll wait for a v2 and supersed v1.

Thanks!

Quoted 21 lines

---

Amos Kong (14):
KVM-test: Add a new macaddress pool algorithm
KVM Test: Add a function get_interface_name() to kvm_net_utils.py KVM Test: Add a common ping module for network related tests
KVM-test: Add a new subtest ping
KVM-test: Add a subtest jumbo
KVM-test: Add basic file transfer test
KVM-test: Add a subtest of load/unload nic driver
KVM-test: Add a subtest of nic promisc
KVM-test: Add a subtest of multicast
KVM-test: Add a subtest of pxe
KVM-test: Add a subtest of changing mac address
KVM-test: Add a subtest of netperf
KVM-test: Improve vlan subtest
KVM-test: Add subtest of testing offload by ethtool

 0 files changed, 0 insertions(+), 0 deletions(-)