Skip to contentSkip to conversation

iotests: Selfish patches

PATCH Superseded Max Reitz 4 participants 9 messages May 17, 2019

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Max Reitz

author
#0 lore raw

cover iotests: Selfish patches

Hi,

These are some rather selfish iotests patches. The first patch helps me personally because I tend to run the tests over SSH and forget to set $DISPLAY. That makes test 139 skip the tests annotated with skip_if_unsupprted(), because iotests.py can no longer determine the list of whitelisted formats.

Patch 2 through 4 are specifically for RHEL. We have not whitelisted null-aio, so it would be nice if tests didn’t require it. Sorry, I don’t have a better reason to give.
In all seriousness, null-co is used widely in many tests, it basically is our standard null driver. Tests should prefer it over null-aio, just for consistency alone. It is not completely unreasonable to treat null-aio as optional. I guess.

Final note: The best thing would probably to skip the null-aio tests in 093/136 if there is no null-aio support. However, I didn’t get anything to work: Annotating with @iotests.skip_if_unsupported() didn’t work because the format name is a class instance attribute; and just iotests.skipTest() didn’t work because that would print 's' characters instead of '.' in the output (and emit the number of skipped tests), so the comparison against the reference output fails... Which is why I decided to just run the test with null-co then. That means that some tests run twice with null-co (if there is no null-aio support), but that’s not too bad. Just ugly.

Max Reitz (4):
iotests: Add -display none to the qemu options
iotests: Prefer null-co over null-aio
iotests: Test driver whitelisting in 093
iotests: Test driver whitelisting in 136

 tests/qemu-iotests/093   | 22 +++++++++++++++-------
 tests/qemu-iotests/136   | 17 +++++++++++++----
 tests/qemu-iotests/245   |  2 +-
 tests/qemu-iotests/check |  2 +-
 4 files changed, 30 insertions(+), 13 deletions(-)
Signature · 1 line
2.21.0
author
#1 lore raw

PATCH 1/4 iotests: Add -display none to the qemu options

Without this argument, qemu will print an angry message about not being able to connect to a display server if $DISPLAY is not set. For me, that breaks iotests.supported_formats() because it thus only sees ["Could", "not", "connect"] as the supported formats.

---
 tests/qemu-iotests/check | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/check +1 -1
index f9c24b6753..d6fec4bf3c 100755
@@ -130,7 +130,7 @@ export CACHEMODE="writeback"
 export QEMU_IO_OPTIONS=""
 export QEMU_IO_OPTIONS_NO_FMT=""
 export CACHEMODE_IS_DEFAULT=true
-export QEMU_OPTIONS="-nodefaults -machine accel=qtest"
+export QEMU_OPTIONS="-nodefaults -machine accel=qtest -display none"
 export VALGRIND_QEMU=
 export IMGKEYSECRET=
 export IMGOPTSSYNTAX=false
Signature · 1 line
2.21.0
Reviewed-by on v1 1/4
#2 lore raw

Re: [Qemu-devel] [PATCH 1/4] iotests: Add -display none to the qemu options

Quoted 16 lines · Max Reitz

Without this argument, qemu will print an angry message about not being able to connect to a display server if $DISPLAY is not set. For me, that breaks iotests.supported_formats() because it thus only sees ["Could", "not", "connect"] as the supported formats.

---
 tests/qemu-iotests/check | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/check +0 -0
index f9c24b6753..d6fec4bf3c 100755

tests/qemu-iotests/check export CACHEMODE="writeback" · line 136

Quoted 8 lines
tests/qemu-iotests/check +1 -1
@@ -130,7 +130,7 @@ export CACHEMODE="writeback"
 export QEMU_IO_OPTIONS=""
 export QEMU_IO_OPTIONS_NO_FMT=""
 export CACHEMODE_IS_DEFAULT=true
-export QEMU_OPTIONS="-nodefaults -machine accel=qtest"
+export QEMU_OPTIONS="-nodefaults -machine accel=qtest -display none"
 export VALGRIND_QEMU=
 export IMGKEYSECRET=
 export IMGOPTSSYNTAX=false
Reviewed-by on v1 1/4
#3 lore raw

Re: [Qemu-devel] [PATCH 1/4] iotests: Add -display none to the qemu options

In reply to Max Reitz
Quoted 6 lines · Max Reitz

Without this argument, qemu will print an angry message about not being able to connect to a display server if $DISPLAY is not set. For me, that breaks iotests.supported_formats() because it thus only sees ["Could", "not", "connect"] as the supported formats.

Quoted 9 lines
---
 tests/qemu-iotests/check | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/check +0 -0
index f9c24b6753..d6fec4bf3c 100755

tests/qemu-iotests/check export CACHEMODE="writeback" · line 136

Quoted 8 lines
tests/qemu-iotests/check +1 -1
@@ -130,7 +130,7 @@ export CACHEMODE="writeback"
 export QEMU_IO_OPTIONS=""
 export QEMU_IO_OPTIONS_NO_FMT=""
 export CACHEMODE_IS_DEFAULT=true
-export QEMU_OPTIONS="-nodefaults -machine accel=qtest"
+export QEMU_OPTIONS="-nodefaults -machine accel=qtest -display none"
 export VALGRIND_QEMU=
 export IMGKEYSECRET=
 export IMGOPTSSYNTAX=false

--
Alex Bennée

author
#4 lore raw

PATCH 2/4 iotests: Prefer null-co over null-aio

In reply to Max Reitz

We use null-co basically everywhere in the iotests. Unless we want to test null-aio specifically, we should use it instead (for consistency).

---
 tests/qemu-iotests/093 | 7 +++----
 tests/qemu-iotests/245 | 2 +-
 2 files changed, 4 insertions(+), 5 deletions(-)
tests/qemu-iotests/093 +3 -4
index d88fbc182e..bd56c94708 100755
@@ -267,13 +267,12 @@ class ThrottleTestCoroutine(ThrottleTestCase):
     test_img = "null-co://"
 
 class ThrottleTestGroupNames(iotests.QMPTestCase):
-    test_img = "null-aio://"
     max_drives = 3
 
     def setUp(self):
         self.vm = iotests.VM()
         for i in range(0, self.max_drives):
-            self.vm.add_drive(self.test_img, "throttling.iops-total=100")
+            self.vm.add_drive("null-co://", "throttling.iops-total=100")
         self.vm.launch()
 
     def tearDown(self):
@@ -377,10 +376,10 @@ class ThrottleTestRemovableMedia(iotests.QMPTestCase):
 
     def test_removable_media(self):
         # Add a couple of dummy nodes named cd0 and cd1
-        result = self.vm.qmp("blockdev-add", driver="null-aio",
+        result = self.vm.qmp("blockdev-add", driver="null-co",
                              node_name="cd0")
         self.assert_qmp(result, 'return', {})
-        result = self.vm.qmp("blockdev-add", driver="null-aio",
+        result = self.vm.qmp("blockdev-add", driver="null-co",
                              node_name="cd1")
         self.assert_qmp(result, 'return', {})
 
tests/qemu-iotests/245 +1 -1
index a04c6235c1..d888a1354c 100644
@@ -598,7 +598,7 @@ class TestBlockdevReopen(iotests.QMPTestCase):
         ##################
         ###### null ######
         ##################
-        opts = {'driver': 'null-aio', 'node-name': 'root', 'size': 1024}
+        opts = {'driver': 'null-co', 'node-name': 'root', 'size': 1024}
 
         result = self.vm.qmp('blockdev-add', conv_keys = False, **opts)
         self.assert_qmp(result, 'return', {})
Signature · 1 line
2.21.0
author
#5 lore raw

PATCH 3/4 iotests: Test driver whitelisting in 093

In reply to Max Reitz

null-aio may not be whitelisted. If it is not, fall back to null-co. This may run tests twice in the same configuration, but this is the simplest way to effectively skip the tests in setUp() (without changing the output, and while having the respective driver in a class attribute).

---
 tests/qemu-iotests/093 | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)
tests/qemu-iotests/093 +12 -3
index bd56c94708..d6f285001a 100755
@@ -22,9 +22,11 @@
 import iotests
 
 nsec_per_sec = 1000000000
+supported_null_drivers = list(filter(lambda f: f.startswith('null-'),
+                                     iotests.supported_formats()))
 
 class ThrottleTestCase(iotests.QMPTestCase):
-    test_img = "null-aio://"
+    test_driver = "null-aio"
     max_drives = 3
 
     def blockstats(self, device):
@@ -36,9 +38,14 @@ class ThrottleTestCase(iotests.QMPTestCase):
         raise Exception("Device not found for blockstats: %s" % device)
 
     def setUp(self):
+        global supported_null_drivers
+        if self.test_driver not in supported_null_drivers:
+            # Silently fall back to supported driver
+            self.test_driver = supported_null_drivers[0]
+
         self.vm = iotests.VM()
         for i in range(0, self.max_drives):
-            self.vm.add_drive(self.test_img)
+            self.vm.add_drive(self.test_driver + "://")
         self.vm.launch()
 
     def tearDown(self):
@@ -264,7 +271,7 @@ class ThrottleTestCase(iotests.QMPTestCase):
         self.assertEqual(self.blockstats('drive1')[0], 4096)
 
 class ThrottleTestCoroutine(ThrottleTestCase):
-    test_img = "null-co://"
+    test_driver = "null-co"
 
 class ThrottleTestGroupNames(iotests.QMPTestCase):
     max_drives = 3
@@ -426,4 +433,6 @@ class ThrottleTestRemovableMedia(iotests.QMPTestCase):
 
 
 if __name__ == '__main__':
+    if 'null-co' not in supported_null_drivers:
+        iotests.notrun('null-co driver support missing')
     iotests.main(supported_fmts=["raw"])
Signature · 1 line
2.21.0
#6 lore raw

Re: [Qemu-devel] [PATCH 3/4] iotests: Test driver whitelisting in 093

Am 17.05.2019 um 11:56 hat Max Reitz geschrieben:

Quoted 16 lines

null-aio may not be whitelisted. If it is not, fall back to null-co. This may run tests twice in the same configuration, but this is the simplest way to effectively skip the tests in setUp() (without changing the output, and while having the respective driver in a class attribute).

---
 tests/qemu-iotests/093 | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)
tests/qemu-iotests/093 +0 -0
index bd56c94708..d6f285001a 100755

tests/qemu-iotests/093 line 26

Quoted 5 lines
tests/qemu-iotests/093 +2 -0
@@ -22,9 +22,11 @@
 import iotests
 
 nsec_per_sec = 1000000000
+supported_null_drivers = list(filter(lambda f: f.startswith('null-'),
+                                     iotests.supported_formats()))

Is this just a convoluted way of writing the following?

supported_null_drivers = [ f for f in iotests.supported_formats()
if f.startswith('null-') ]

Quoted 7 lines
tests/qemu-iotests/093 +1 -1
 class ThrottleTestCase(iotests.QMPTestCase):
-    test_img = "null-aio://"
+    test_driver = "null-aio"
     max_drives = 3
 
     def blockstats(self, device):

tests/qemu-iotests/093 class ThrottleTestCase(iotests.QMPTestCase): · line 44

Quoted 7 lines
tests/qemu-iotests/093 +4 -0
@@ -36,9 +38,14 @@ class ThrottleTestCase(iotests.QMPTestCase):
         raise Exception("Device not found for blockstats: %s" % device)
 
     def setUp(self):
+        global supported_null_drivers
+        if self.test_driver not in supported_null_drivers:
+            # Silently fall back to supported driver
+            self.test_driver = supported_null_drivers[0]

I think this is what you mentioned in the cover letter:

Quoted 7 lines

Final note: The best thing would probably to skip the null-aio tests in 093/136 if there is no null-aio support. However, I didn’t get anything to work: Annotating with @iotests.skip_if_unsupported() didn’t work because the format name is a class instance attribute; and just iotests.skipTest() didn’t work because that would print 's' characters
instead of '.' in the output (and emit the number of skipped tests), so
the comparison against the reference output fails...

With a little modification to the @skip_if_unsupported() decorator it can be done. I think I'd prefer this (hacked up on top of this series):

tests/qemu-iotests/iotests.py +5 -2
index f811f69135..f83d56b156 100644
@@ -789,8 +789,11 @@ def skip_if_unsupported(required_formats=[], read_only=False):
        Runs the test if all the required formats are whitelisted'''
     def skip_test_decorator(func):
         def func_wrapper(*args, **kwargs):
-            usf_list = list(set(required_formats) -
-                            set(supported_formats(read_only)))
+            if callable(required_formats):
+                fmts = required_formats(args[0])
+            else:
+                fmts = required_formats
+            usf_list = list(set(fmts) - set(supported_formats(read_only)))
             if usf_list:
                 case_notrun('{}: formats {} are not whitelisted'.format(
                     args[0], usf_list))
tests/qemu-iotests/093 +13 -7
index d6f285001a..e23a8189bc 100755
@@ -22,8 +22,6 @@
 import iotests
 
 nsec_per_sec = 1000000000
-supported_null_drivers = list(filter(lambda f: f.startswith('null-'),
-                                     iotests.supported_formats()))
 
 class ThrottleTestCase(iotests.QMPTestCase):
     test_driver = "null-aio"
@@ -37,11 +35,12 @@ class ThrottleTestCase(iotests.QMPTestCase):
                 return stat['rd_bytes'], stat['rd_operations'], stat['wr_bytes'], stat['wr_operations']
         raise Exception("Device not found for blockstats: %s" % device)
 
+    def required_driver(self):
+        return self.test_driver
+
     def setUp(self):
-        global supported_null_drivers
-        if self.test_driver not in supported_null_drivers:
-            # Silently fall back to supported driver
-            self.test_driver = supported_null_drivers[0]
+        if not self.required_driver() in iotests.supported_formats():
+            return
 
         self.vm = iotests.VM()
         for i in range(0, self.max_drives):
@@ -49,6 +48,9 @@ class ThrottleTestCase(iotests.QMPTestCase):
         self.vm.launch()
 
     def tearDown(self):
+        if not self.required_driver() in iotests.supported_formats():
+            return
+
         self.vm.shutdown()
 
     def configure_throttle(self, ndrives, params):
@@ -145,6 +147,7 @@ class ThrottleTestCase(iotests.QMPTestCase):
         self.vm.qtest("clock_step %d" % ns)
 
     # Connect N drives to a VM and test I/O in all of them
+    @iotests.skip_if_unsupported(required_driver)
     def test_all(self):
         params = {"bps": 4096,
                   "bps_rd": 4096,
@@ -163,6 +166,7 @@ class ThrottleTestCase(iotests.QMPTestCase):
                 self.do_test_throttle(ndrives, 5, limits)
 
     # Connect N drives to a VM and test I/O in just one of them a time
+    @iotests.skip_if_unsupported(required_driver)
     def test_one(self):
         params = {"bps": 4096,
                   "bps_rd": 4096,
@@ -180,6 +184,7 @@ class ThrottleTestCase(iotests.QMPTestCase):
                 self.configure_throttle(self.max_drives, limits)
                 self.do_test_throttle(1, 5, limits, drive)
 
+    @iotests.skip_if_unsupported(required_driver)
     def test_burst(self):
         params = {"bps": 4096,
                   "bps_rd": 4096,
@@ -218,6 +223,7 @@ class ThrottleTestCase(iotests.QMPTestCase):
     # Test that removing a drive from a throttle group should not
     # affect the remaining members of the group.
     # https://bugzilla.redhat.com/show_bug.cgi?id=1535914
+    @iotests.skip_if_unsupported(required_driver)
     def test_remove_group_member(self):
         # Create a throttle group with two drives
         # and set a 4 KB/s read limit.
@@ -433,6 +439,6 @@ class ThrottleTestRemovableMedia(iotests.QMPTestCase):
 
 
 if __name__ == '__main__':
-    if 'null-co' not in supported_null_drivers:
+    if 'null-co' not in iotests.supported_formats():
         iotests.notrun('null-co driver support missing')
     iotests.main(supported_fmts=["raw"])
author
#7 lore raw

Re: [Qemu-devel] [PATCH 3/4] iotests: Test driver whitelisting in 093

Quoted 19 lines · Kevin Wolf

Am 17.05.2019 um 11:56 hat Max Reitz geschrieben:

Quoted 18 lines

null-aio may not be whitelisted. If it is not, fall back to null-co. This may run tests twice in the same configuration, but this is the simplest way to effectively skip the tests in setUp() (without changing the output, and while having the respective driver in a class attribute).

---
 tests/qemu-iotests/093 | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)
tests/qemu-iotests/093 +0 -0
index bd56c94708..d6f285001a 100755
@@ -22,9 +22,11 @@
 import iotests
 
Quoted 8 lines

tests/qemu-iotests/093 line 26

Quoted 3 lines
tests/qemu-iotests/093 +2 -0
 nsec_per_sec = 1000000000
+supported_null_drivers = list(filter(lambda f: f.startswith('null-'),
+                                     iotests.supported_formats()))

Is this just a convoluted way of writing the following?

supported_null_drivers = [ f for f in iotests.supported_formats()
if f.startswith('null-') ]

Well, it’s a different way, yes. I would call it the Ruby way, but that would have been "iotests.supported_formats() & ['null-co', 'null-aio']".
(And with list(set() & set()) it suddenly isn’t short anymore.)

Sorry, I’m just not suited for Python.

Quoted 25 lines
Quoted 7 lines
tests/qemu-iotests/093 +1 -1
 class ThrottleTestCase(iotests.QMPTestCase):
-    test_img = "null-aio://"
+    test_driver = "null-aio"
     max_drives = 3
 
     def blockstats(self, device):

tests/qemu-iotests/093 class ThrottleTestCase(iotests.QMPTestCase): · line 44

Quoted 7 lines
tests/qemu-iotests/093 +4 -0
@@ -36,9 +38,14 @@ class ThrottleTestCase(iotests.QMPTestCase):
         raise Exception("Device not found for blockstats: %s" % device)
 
     def setUp(self):
+        global supported_null_drivers
+        if self.test_driver not in supported_null_drivers:
+            # Silently fall back to supported driver
+            self.test_driver = supported_null_drivers[0]

I think this is what you mentioned in the cover letter:

Quoted 7 lines

Final note: The best thing would probably to skip the null-aio tests in 093/136 if there is no null-aio support. However, I didn’t get anything to work: Annotating with @iotests.skip_if_unsupported() didn’t work because the format name is a class instance attribute; and just iotests.skipTest() didn’t work because that would print 's' characters
instead of '.' in the output (and emit the number of skipped tests), so
the comparison against the reference output fails...

Quoted 2 lines

With a little modification to the @skip_if_unsupported() decorator it can be done. I think I'd prefer this (hacked up on top of this series):

Hm. I really don’t like having to put it above every single test method, and putting it above setUp() has the problem of not actually skipping the test.

I guess I’ll look back into filtering the test output to remove the “skipped” stuff.

Max

author
#8 lore raw

PATCH 4/4 iotests: Test driver whitelisting in 136

In reply to Max Reitz

null-aio may not be whitelisted. If it is not, fall back to null-co. This may run tests twice in the same configuration, but this is the simplest way to effectively skip the tests in setUp() (without changing the output, and while having the respective driver in a class attribute).

---
 tests/qemu-iotests/136 | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)
tests/qemu-iotests/136 +13 -4
index af7ffa4540..44e3bc1575 100755
@@ -28,9 +28,11 @@ op_latency = nsec_per_sec // 1000 # See qtest_latency_ns in accounting.c
 bad_sector = 8192
 bad_offset = bad_sector * 512
 blkdebug_file = os.path.join(iotests.test_dir, 'blkdebug.conf')
+supported_null_drivers = list(filter(lambda f: f.startswith('null-'),
+                                     iotests.supported_formats()))
 
 class BlockDeviceStatsTestCase(iotests.QMPTestCase):
-    test_img = "null-aio://"
+    test_driver = "null-aio"
     total_rd_bytes = 0
     total_rd_ops = 0
     total_wr_bytes = 0
@@ -68,6 +70,11 @@ sector = "%d"
         file.close()
 
     def setUp(self):
+        global supported_null_drivers
+        if self.test_driver not in supported_null_drivers:
+            # Silently fall back to supported driver
+            self.test_driver = supported_null_drivers[0]
+
         drive_args = []
         drive_args.append("stats-intervals.0=%d" % interval_length)
         drive_args.append("stats-account-invalid=%s" %
@@ -75,8 +82,8 @@ sector = "%d"
         drive_args.append("stats-account-failed=%s" %
                           (self.account_failed and "on" or "off"))
         self.create_blkdebug_file()
-        self.vm = iotests.VM().add_drive('blkdebug:%s:%s' %
-                                         (blkdebug_file, self.test_img),
+        self.vm = iotests.VM().add_drive('blkdebug:%s:%s://' %
+                                         (blkdebug_file, self.test_driver),
                                          ','.join(drive_args))
         self.vm.launch()
         # Set an initial value for the clock
@@ -336,7 +343,9 @@ class BlockDeviceStatsTestAccountBoth(BlockDeviceStatsTestCase):
     account_failed = True
 
 class BlockDeviceStatsTestCoroutine(BlockDeviceStatsTestCase):
-    test_img = "null-co://"
+    test_driver = "null-co"
 
 if __name__ == '__main__':
+    if 'null-co' not in supported_null_drivers:
+        iotests.notrun('null-co driver support missing')
     iotests.main(supported_fmts=["raw"])
Signature · 1 line
2.21.0