Skip to contentSkip to conversation

iotests/129: Fix it

PATCH In review Max Reitz 4 participants 32 messages Jan 13, 2021

lore
Share & tools
lore

Max Reitz

author
#0 lore raw

cover iotests/129: Fix it

Hi,

There are some problems with iotests 129 (perhaps more than these, but these are the ones I know of):

1. It checks @busy to see whether a block job is still running; however, block jobs tend to unset @busy all the time (when they yield). [Fixed by patch 3]

2. It uses blockdev throttling, which quite some time ago has been moved to the BB level; since then, such throttling will no longer affect block jobs. We can get throttling to work by using a throttle filter node.
[Fixed by patch 4]

3. The mirror job has a large buffer size by default. A simple drain may lead to it making significant process, which is kind of dangerous, because we don’t want the job to complete.
To get around this, we can simply limit its buffer size. (And we should make the commit job an actual commit job instead of an active commit (which is just mirror), because the commit interface does not allow setting a buffer size.)
[Fixed by patches 5 and 6]

This series fixes those things, and now 129 seems to reliably pass for me.

Apart from the major issues above, there are also minor flaws:

- It doesn’t remove the test images.
[Fixed by patches 1 and 2]

- pylint and mypy complain.
(Running mypy with the options given in 297.) [Patch 4 removes one pylint complaint; patch 7 the rest.]

Max Reitz (7):
iotests: Move try_remove to iotests.py iotests/129: Remove test images in tearDown() iotests/129: Do not check @busy iotests/129: Use throttle node iotests/129: Actually test a commit job iotests/129: Limit mirror job's buffer size iotests/129: Clean up pylint and mypy complaints

 tests/qemu-iotests/124        |  8 +---
 tests/qemu-iotests/129        | 76 ++++++++++++++++++++++-------------
 tests/qemu-iotests/iotests.py | 11 +++--
 3 files changed, 55 insertions(+), 40 deletions(-)
Signature · 1 line
2.29.2
author
#1 lore raw

PATCH 1/7 iotests: Move try_remove to iotests.py

---
 tests/qemu-iotests/124        |  8 +-------
 tests/qemu-iotests/iotests.py | 11 +++++++----
 2 files changed, 8 insertions(+), 11 deletions(-)
tests/qemu-iotests/124 +1 -7
index 3705cbb6b3..e40eeb50b9 100755
@@ -22,6 +22,7 @@
 
 import os
 import iotests
+from iotests import try_remove
 
 
 def io_write_patterns(img, patterns):
@@ -29,13 +30,6 @@ def io_write_patterns(img, patterns):
         iotests.qemu_io('-c', 'write -P%s %s %s' % pattern, img)
 
 
-def try_remove(img):
-    try:
-        os.remove(img)
-    except OSError:
-        pass
-
-
 def transaction_action(action, **kwargs):
     return {
         'type': action,
tests/qemu-iotests/iotests.py +7 -4
index dcdcd0387f..15704ca466 100644
@@ -515,12 +515,15 @@ class FilePath:
         return False
 
 
+def try_remove(img):
+    try:
+        os.remove(img)
+    except OSError:
+        pass
+
 def file_path_remover():
     for path in reversed(file_path_remover.paths):
-        try:
-            os.remove(path)
-        except OSError:
-            pass
+        try_remove(path)
 
 
 def file_path(*names, base_dir=test_dir):
Signature · 1 line
2.29.2
Reviewed-by on v1 1/7
#2 lore raw

Re: [PATCH 1/7] iotests: Move try_remove to iotests.py

Quoted 6 lines · Max Reitz
---
 tests/qemu-iotests/124        |  8 +-------
 tests/qemu-iotests/iotests.py | 11 +++++++----
 2 files changed, 8 insertions(+), 11 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
author
#3 lore raw

PATCH 2/7 iotests/129: Remove test images in tearDown()

In reply to Max Reitz
---
 tests/qemu-iotests/129 | 2 ++
 1 file changed, 2 insertions(+)
tests/qemu-iotests/129 +2 -0
index 0e13244d85..2fc65ada6a 100755
@@ -47,6 +47,8 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
         result = self.vm.qmp("block_set_io_throttle", conv_keys=False,
                              **params)
         self.vm.shutdown()
+        for img in (self.test_img, self.target_img, self.base_img):
+            iotests.try_remove(img)
 
     def do_test_stop(self, cmd, **args):
         """Test 'stop' while block job is running on a throttled drive.
Signature · 1 line
2.29.2
Reviewed-by on v1 2/7
#4 lore raw

Re: [PATCH 2/7] iotests/129: Remove test images in tearDown()

Quoted 1 line · Max Reitz
Quoted 18 lines
---
  tests/qemu-iotests/129 | 2 ++
  1 file changed, 2 insertions(+)
tests/qemu-iotests/129 +2 -0
index 0e13244d85..2fc65ada6a 100755
@@ -47,6 +47,8 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
          result = self.vm.qmp("block_set_io_throttle", conv_keys=False,
                               **params)
          self.vm.shutdown()
+        for img in (self.test_img, self.target_img, self.base_img):
+            iotests.try_remove(img)
 
      def do_test_stop(self, cmd, **args):
          """Test 'stop' while block job is running on a throttled drive.
Signature · 2 lines
Best regards,
Vladimir
Reviewed-by on v1 2/7
#5 lore raw

Re: [PATCH 2/7] iotests/129: Remove test images in tearDown()

In reply to Max Reitz
Quoted 4 lines · Max Reitz
---
 tests/qemu-iotests/129 | 2 ++
 1 file changed, 2 insertions(+)
Quoted 15 lines
tests/qemu-iotests/129 +2 -0
index 0e13244d85..2fc65ada6a 100755
@@ -47,6 +47,8 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
         result = self.vm.qmp("block_set_io_throttle", conv_keys=False,
                              **params)
         self.vm.shutdown()
+        for img in (self.test_img, self.target_img, self.base_img):
+            iotests.try_remove(img)
 
     def do_test_stop(self, cmd, **args):
         """Test 'stop' while block job is running on a throttled drive.
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
author
#6 lore raw

PATCH 3/7 iotests/129: Do not check @busy

In reply to Max Reitz

@busy is false when the job is paused, which happens all the time because that is how jobs yield (e.g. for mirror at least since commit 565ac01f8d3).

Back when 129 was added (2015), perhaps there was no better way of checking whether the job was still actually running. Now we have the @status field (as of 58b295ba52c, i.e. 2018), which can give us exactly that information.

---
 tests/qemu-iotests/129 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/129 +1 -1
index 2fc65ada6a..dd23bb2e5a 100755
@@ -69,7 +69,7 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
         result = self.vm.qmp("stop")
         self.assert_qmp(result, 'return', {})
         result = self.vm.qmp("query-block-jobs")
-        self.assert_qmp(result, 'return[0]/busy', True)
+        self.assert_qmp(result, 'return[0]/status', 'running')
         self.assert_qmp(result, 'return[0]/ready', False)
 
     def test_drive_mirror(self):
Signature · 1 line
2.29.2
Reviewed-by on v1 3/7
#7 lore raw

Re: [PATCH 3/7] iotests/129: Do not check @busy

Quoted 9 lines · Max Reitz

@busy is false when the job is paused, which happens all the time because that is how jobs yield (e.g. for mirror at least since commit 565ac01f8d3).

Back when 129 was added (2015), perhaps there was no better way of checking whether the job was still actually running. Now we have the @status field (as of 58b295ba52c, i.e. 2018), which can give us exactly that information.

Quoted 4 lines
---
 tests/qemu-iotests/129 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 15 lines
tests/qemu-iotests/129 +1 -1
index 2fc65ada6a..dd23bb2e5a 100755
@@ -69,7 +69,7 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
         result = self.vm.qmp("stop")
         self.assert_qmp(result, 'return', {})
         result = self.vm.qmp("query-block-jobs")
-        self.assert_qmp(result, 'return[0]/busy', True)
+        self.assert_qmp(result, 'return[0]/status', 'running')
         self.assert_qmp(result, 'return[0]/ready', False)
 
     def test_drive_mirror(self):
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
Reviewed-by on v1 3/7
#8 lore raw

Re: [PATCH 3/7] iotests/129: Do not check @busy

In reply to Max Reitz
Quoted 10 lines · Max Reitz

@busy is false when the job is paused, which happens all the time because that is how jobs yield (e.g. for mirror at least since commit 565ac01f8d3).

Back when 129 was added (2015), perhaps there was no better way of checking whether the job was still actually running. Now we have the @status field (as of 58b295ba52c, i.e. 2018), which can give us exactly that information.

Quoted 18 lines
---
  tests/qemu-iotests/129 | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/129 +1 -1
index 2fc65ada6a..dd23bb2e5a 100755
@@ -69,7 +69,7 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
          result = self.vm.qmp("stop")
          self.assert_qmp(result, 'return', {})
          result = self.vm.qmp("query-block-jobs")
-        self.assert_qmp(result, 'return[0]/busy', True)
+        self.assert_qmp(result, 'return[0]/status', 'running')
          self.assert_qmp(result, 'return[0]/ready', False)
 
      def test_drive_mirror(self):
Signature · 2 lines
Best regards,
Vladimir
author
#9 lore raw

PATCH 4/7 iotests/129: Use throttle node

In reply to Max Reitz

Throttling on the BB has not affected block jobs in a while, so it is possible that one of the jobs in 129 finishes before the VM is stopped. We can fix that by running the job from a throttle node.

---
 tests/qemu-iotests/129 | 39 +++++++++++++++------------------------
 1 file changed, 15 insertions(+), 24 deletions(-)
tests/qemu-iotests/129 +15 -24
index dd23bb2e5a..febc806398 100755
@@ -32,20 +32,18 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
         iotests.qemu_img('create', '-f', iotests.imgfmt, self.test_img,
                          "-b", self.base_img, '-F', iotests.imgfmt)
         iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 1M 128M', self.test_img)
-        self.vm = iotests.VM().add_drive(self.test_img)
+        self.vm = iotests.VM()
+        self.vm.add_object('throttle-group,id=tg0,x-bps-total=1024')
+
+        source_drive = 'driver=throttle,' + \
+                       'throttle-group=tg0,' + \
+                      f'file.driver={iotests.imgfmt},' + \
+                      f'file.file.filename={self.test_img}'
+
+        self.vm.add_drive(None, source_drive)
         self.vm.launch()
 
     def tearDown(self):
-        params = {"device": "drive0",
-                  "bps": 0,
-                  "bps_rd": 0,
-                  "bps_wr": 0,
-                  "iops": 0,
-                  "iops_rd": 0,
-                  "iops_wr": 0,
-                 }
-        result = self.vm.qmp("block_set_io_throttle", conv_keys=False,
-                             **params)
         self.vm.shutdown()
         for img in (self.test_img, self.target_img, self.base_img):
             iotests.try_remove(img)
@@ -53,33 +51,26 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
     def do_test_stop(self, cmd, **args):
         """Test 'stop' while block job is running on a throttled drive.
         The 'stop' command shouldn't drain the job"""
-        params = {"device": "drive0",
-                  "bps": 1024,
-                  "bps_rd": 0,
-                  "bps_wr": 0,
-                  "iops": 0,
-                  "iops_rd": 0,
-                  "iops_wr": 0,
-                 }
-        result = self.vm.qmp("block_set_io_throttle", conv_keys=False,
-                             **params)
-        self.assert_qmp(result, 'return', {})
         result = self.vm.qmp(cmd, **args)
         self.assert_qmp(result, 'return', {})
+
         result = self.vm.qmp("stop")
         self.assert_qmp(result, 'return', {})
         result = self.vm.qmp("query-block-jobs")
+
         self.assert_qmp(result, 'return[0]/status', 'running')
         self.assert_qmp(result, 'return[0]/ready', False)
 
+        self.vm.qmp("block-job-cancel", device="drive0", force=True)
+
     def test_drive_mirror(self):
         self.do_test_stop("drive-mirror", device="drive0",
-                          target=self.target_img,
+                          target=self.target_img, format=iotests.imgfmt,
                           sync="full")
 
     def test_drive_backup(self):
         self.do_test_stop("drive-backup", device="drive0",
-                          target=self.target_img,
+                          target=self.target_img, format=iotests.imgfmt,
                           sync="full")
 
     def test_block_commit(self):
Signature · 1 line
2.29.2
author
#10 lore raw

Re: [PATCH 4/7] iotests/129: Use throttle node

Quoted 13 lines · Max Reitz

Throttling on the BB has not affected block jobs in a while, so it is possible that one of the jobs in 129 finishes before the VM is stopped. We can fix that by running the job from a throttle node.

---
  tests/qemu-iotests/129 | 39 +++++++++++++++------------------------
  1 file changed, 15 insertions(+), 24 deletions(-)
tests/qemu-iotests/129 +0 -0
index dd23bb2e5a..febc806398 100755

[...]

Quoted 17 lines
tests/qemu-iotests/129 +0 -11
@@ -53,33 +51,26 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
      def do_test_stop(self, cmd, **args):
          """Test 'stop' while block job is running on a throttled drive.
          The 'stop' command shouldn't drain the job"""
-        params = {"device": "drive0",
-                  "bps": 1024,
-                  "bps_rd": 0,
-                  "bps_wr": 0,
-                  "iops": 0,
-                  "iops_rd": 0,
-                  "iops_wr": 0,
-                 }
-        result = self.vm.qmp("block_set_io_throttle", conv_keys=False,
-                             **params)
-        self.assert_qmp(result, 'return', {})
          result = self.vm.qmp(cmd, **args)
          self.assert_qmp(result, 'return', {})

tests/qemu-iotests/129 class TestStopWithBlockJob(iotests.QMPTestCase): · line 65

Quoted 10 lines
tests/qemu-iotests/129 +4 -0
+
          result = self.vm.qmp("stop")
          self.assert_qmp(result, 'return', {})
          result = self.vm.qmp("query-block-jobs")
+
          self.assert_qmp(result, 'return[0]/status', 'running')
          self.assert_qmp(result, 'return[0]/ready', False)
 
+        self.vm.qmp("block-job-cancel", device="drive0", force=True)
+

Sorry, somewhere along the way I forgot to remove this block-job-cancel.
It’s still from
https://lists.nongnu.org/archive/html/qemu-block/2019-06/msg00499.html and I don’t know why I added it there. It should probably go.

('stop' should have drained, that’s the point, so there shouldn’t be any further delay if we quit the VM without cancelling the job.)

Max

Reviewed-by
#11 lore raw

Re: [PATCH 4/7] iotests/129: Use throttle node

Quoted 25 lines · Max Reitz
Quoted 10 lines · Max Reitz

Throttling on the BB has not affected block jobs in a while, so it is possible that one of the jobs in 129 finishes before the VM is stopped. We can fix that by running the job from a throttle node.

---
  tests/qemu-iotests/129 | 39 +++++++++++++++------------------------

  1 file changed, 15 insertions(+), 24 deletions(-)
Quoted 8 lines

          result = self.vm.qmp("stop") self.assert_qmp(result, 'return', {})
          result = self.vm.qmp("query-block-jobs")
+
          self.assert_qmp(result, 'return[0]/status', 'running') self.assert_qmp(result, 'return[0]/ready', False) +        self.vm.qmp("block-job-cancel", device="drive0", force=True) +

Sorry, somewhere along the way I forgot to remove this block-job-cancel. It’s still from
https://lists.nongnu.org/archive/html/qemu-block/2019-06/msg00499.html
and I don’t know why I added it there.  It should probably go.

Quoted 2 lines

('stop' should have drained, that’s the point, so there shouldn’t be any further delay if we quit the VM without cancelling the job.)

Since the test still passes without the added block-job-cancel, I'm fine with removing that line then adding:

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#12 lore raw

Re: [PATCH 4/7] iotests/129: Use throttle node

In reply to Max Reitz
Quoted 47 lines · Max Reitz
Quoted 13 lines · Max Reitz

Throttling on the BB has not affected block jobs in a while, so it is possible that one of the jobs in 129 finishes before the VM is stopped. We can fix that by running the job from a throttle node.

---
  tests/qemu-iotests/129 | 39 +++++++++++++++------------------------

  1 file changed, 15 insertions(+), 24 deletions(-)
tests/qemu-iotests/129 +0 -0
index dd23bb2e5a..febc806398 100755

[...]

Quoted 18 lines
tests/qemu-iotests/129 +0 -0
@@ -53,33 +51,26 @@ class TestStopWithBlockJob(iotests.QMPTestCase):

      def do_test_stop(self, cmd, **args):
          """Test 'stop' while block job is running on a throttled drive. The 'stop' command shouldn't drain the job"""
-        params = {"device": "drive0",
-                  "bps": 1024,
-                  "bps_rd": 0,
-                  "bps_wr": 0,
-                  "iops": 0,
-                  "iops_rd": 0,
-                  "iops_wr": 0,
-                 }
-        result = self.vm.qmp("block_set_io_throttle", conv_keys=False, -                             **params)
-        self.assert_qmp(result, 'return', {})
          result = self.vm.qmp(cmd, **args) self.assert_qmp(result, 'return', {})
+

Quoted 8 lines

          result = self.vm.qmp("stop") self.assert_qmp(result, 'return', {})
          result = self.vm.qmp("query-block-jobs")
+
          self.assert_qmp(result, 'return[0]/status', 'running') self.assert_qmp(result, 'return[0]/ready', False) +        self.vm.qmp("block-job-cancel", device="drive0", force=True) +

Sorry, somewhere along the way I forgot to remove this block-job-cancel.  It’s still from
https://lists.nongnu.org/archive/html/qemu-block/2019-06/msg00499.html and I don’t know why I added it there.  It should probably go.

Quoted 2 lines

('stop' should have drained, that’s the point, so there shouldn’t be any further delay if we quit the VM without cancelling the job.)

what do you mean by 'stop' should have drained? As I understand, after "stop" jobs should continue and it's OK.. So, immediately after "stop" command job may generate new requests, why not?

Signature · 2 lines
Best regards,
Vladimir
author
#13 lore raw

Re: [PATCH 4/7] iotests/129: Use throttle node

Quoted 48 lines · Vladimir Sementsov-Ogievskiy
Quoted 47 lines · Max Reitz
Quoted 13 lines · Max Reitz

Throttling on the BB has not affected block jobs in a while, so it is possible that one of the jobs in 129 finishes before the VM is stopped. We can fix that by running the job from a throttle node.

---
  tests/qemu-iotests/129 | 39 +++++++++++++++------------------------

  1 file changed, 15 insertions(+), 24 deletions(-)
tests/qemu-iotests/129 +0 -0
index dd23bb2e5a..febc806398 100755

[...]

Quoted 18 lines
tests/qemu-iotests/129 +0 -0
@@ -53,33 +51,26 @@ class TestStopWithBlockJob(iotests.QMPTestCase):

      def do_test_stop(self, cmd, **args):
          """Test 'stop' while block job is running on a throttled drive. The 'stop' command shouldn't drain the job"""
-        params = {"device": "drive0",
-                  "bps": 1024,
-                  "bps_rd": 0,
-                  "bps_wr": 0,
-                  "iops": 0,
-                  "iops_rd": 0,
-                  "iops_wr": 0,
-                 }
-        result = self.vm.qmp("block_set_io_throttle", conv_keys=False, -                             **params)
-        self.assert_qmp(result, 'return', {})
          result = self.vm.qmp(cmd, **args) self.assert_qmp(result, 'return', {})
+

Quoted 8 lines

          result = self.vm.qmp("stop") self.assert_qmp(result, 'return', {})
          result = self.vm.qmp("query-block-jobs")
+
          self.assert_qmp(result, 'return[0]/status', 'running') self.assert_qmp(result, 'return[0]/ready', False) +        self.vm.qmp("block-job-cancel", device="drive0", force=True) +

Sorry, somewhere along the way I forgot to remove this block-job-cancel.  It’s still from
https://lists.nongnu.org/archive/html/qemu-block/2019-06/msg00499.html and I don’t know why I added it there.  It should probably go.

Quoted 4 lines
Quoted 2 lines

('stop' should have drained, that’s the point, so there shouldn’t be any further delay if we quit the VM without cancelling the job.)

what do you mean by 'stop' should have drained? As I understand, after "stop" jobs should continue and it's OK.. So, immediately after "stop" command job may generate new requests, why not?

Hm, right.

Well, either way, the cancel shouldn’t make a difference. If quitting the VM were to stall because of the job, then cancelling it beforehand only means that then cancel will stall.

Max

#14 lore raw

Re: [PATCH 4/7] iotests/129: Use throttle node

In reply to Max Reitz
Quoted 15 lines · Max Reitz

Throttling on the BB has not affected block jobs in a while, so it is possible that one of the jobs in 129 finishes before the VM is stopped. We can fix that by running the job from a throttle node.

---
  tests/qemu-iotests/129 | 39 +++++++++++++++------------------------
  1 file changed, 15 insertions(+), 24 deletions(-)
tests/qemu-iotests/129 +0 -0
index dd23bb2e5a..febc806398 100755
@@ -32,20 +32,18 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
          iotests.qemu_img('create', '-f', iotests.imgfmt, self.test_img,

tests/qemu-iotests/129 class TestStopWithBlockJob(iotests.QMPTestCase): · line 41

Quoted 10 lines
tests/qemu-iotests/129 +7 -1
                           "-b", self.base_img, '-F', iotests.imgfmt)
          iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 1M 128M', self.test_img)
-        self.vm = iotests.VM().add_drive(self.test_img)
+        self.vm = iotests.VM()
+        self.vm.add_object('throttle-group,id=tg0,x-bps-total=1024')
+
+        source_drive = 'driver=throttle,' + \
+                       'throttle-group=tg0,' + \
+                      f'file.driver={iotests.imgfmt},' + \
+                      f'file.file.filename={self.test_img}'

python has a "c-like" string "concatenation", i.e., you may omit "+" operators (I don't know is it a good practice, but I do so:)

Quoted 34 lines
tests/qemu-iotests/129 +2 -21
+
+        self.vm.add_drive(None, source_drive)
          self.vm.launch()
 
      def tearDown(self):
-        params = {"device": "drive0",
-                  "bps": 0,
-                  "bps_rd": 0,
-                  "bps_wr": 0,
-                  "iops": 0,
-                  "iops_rd": 0,
-                  "iops_wr": 0,
-                 }
-        result = self.vm.qmp("block_set_io_throttle", conv_keys=False,
-                             **params)
          self.vm.shutdown()
          for img in (self.test_img, self.target_img, self.base_img):
              iotests.try_remove(img)
@@ -53,33 +51,26 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
      def do_test_stop(self, cmd, **args):
          """Test 'stop' while block job is running on a throttled drive.
          The 'stop' command shouldn't drain the job"""
-        params = {"device": "drive0",
-                  "bps": 1024,
-                  "bps_rd": 0,
-                  "bps_wr": 0,
-                  "iops": 0,
-                  "iops_rd": 0,
-                  "iops_wr": 0,
-                 }
-        result = self.vm.qmp("block_set_io_throttle", conv_keys=False,
-                             **params)
-        self.assert_qmp(result, 'return', {})
          result = self.vm.qmp(cmd, **args)

tests/qemu-iotests/129 class TestStopWithBlockJob(iotests.QMPTestCase): · line 64

Quoted 10 lines
tests/qemu-iotests/129 +3 -0
          self.assert_qmp(result, 'return', {})
+
          result = self.vm.qmp("stop")
          self.assert_qmp(result, 'return', {})
          result = self.vm.qmp("query-block-jobs")
+
          self.assert_qmp(result, 'return[0]/status', 'running')
          self.assert_qmp(result, 'return[0]/ready', False)
 
+        self.vm.qmp("block-job-cancel", device="drive0", force=True)

won't hurt, but I'd drop it as unrelated to commit

tests/qemu-iotests/129 class TestStopWithBlockJob(iotests.QMPTestCase): · line 69

Quoted 6 lines
tests/qemu-iotests/129 +2 -1
+
      def test_drive_mirror(self):
          self.do_test_stop("drive-mirror", device="drive0",
-                          target=self.target_img,
+                          target=self.target_img, format=iotests.imgfmt,
                            sync="full")

this doesn't seem need/related too..

Quoted 9 lines
tests/qemu-iotests/129 +1 -1
      def test_drive_backup(self):
          self.do_test_stop("drive-backup", device="drive0",
-                          target=self.target_img,
+                          target=self.target_img, format=iotests.imgfmt,
                            sync="full")
 
      def test_block_commit(self):
 
Signature · 2 lines
Best regards,
Vladimir
author
#15 lore raw

Re: [PATCH 4/7] iotests/129: Use throttle node

Quoted 20 lines · Vladimir Sementsov-Ogievskiy
Quoted 19 lines · Max Reitz

Throttling on the BB has not affected block jobs in a while, so it is possible that one of the jobs in 129 finishes before the VM is stopped. We can fix that by running the job from a throttle node.

---
  tests/qemu-iotests/129 | 39 +++++++++++++++------------------------

  1 file changed, 15 insertions(+), 24 deletions(-)
tests/qemu-iotests/129 +1 -1
index dd23bb2e5a..febc806398 100755
@@ -32,20 +32,18 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
           iotests.qemu_img('create', '-f', iotests.imgfmt, self.test_img,
                            "-b", self.base_img, '-F', iotests.imgfmt)
           iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 1M 128M', self.test_img)
-        self.vm = iotests.VM().add_drive(self.test_img)
+        self.vm = iotests.VM()
Quoted 8 lines

tests/qemu-iotests/129 class TestStopWithBlockJob(iotests.QMPTestCase): · line 41

Quoted 6 lines
tests/qemu-iotests/129 +6 -0
+        self.vm.add_object('throttle-group,id=tg0,x-bps-total=1024')
+
+        source_drive = 'driver=throttle,' + \
+                       'throttle-group=tg0,' + \
+                      f'file.driver={iotests.imgfmt},' + \
+                      f'file.file.filename={self.test_img}'

python has a "c-like" string "concatenation", i.e., you may omit "+" operators (I don't know is it a good practice, but I do so:)

OK, why not.

Quoted 45 lines
Quoted 34 lines
tests/qemu-iotests/129 +2 -0
+
+        self.vm.add_drive(None, source_drive)
           self.vm.launch()

      def tearDown(self):
-        params = {"device": "drive0",
-                  "bps": 0,
-                  "bps_rd": 0,
-                  "bps_wr": 0,
-                  "iops": 0,
-                  "iops_rd": 0,
-                  "iops_wr": 0,
-                 }
-        result = self.vm.qmp("block_set_io_throttle", conv_keys=False, -                             **params)
          self.vm.shutdown()
          for img in (self.test_img, self.target_img, self.base_img):
              iotests.try_remove(img)

tests/qemu-iotests/129 +0 -0
@@ -53,33 +51,26 @@ class TestStopWithBlockJob(iotests.QMPTestCase):

      def do_test_stop(self, cmd, **args):
          """Test 'stop' while block job is running on a throttled drive. The 'stop' command shouldn't drain the job"""
-        params = {"device": "drive0",
-                  "bps": 1024,
-                  "bps_rd": 0,
-                  "bps_wr": 0,
-                  "iops": 0,
-                  "iops_rd": 0,
-                  "iops_wr": 0,
-                 }
-        result = self.vm.qmp("block_set_io_throttle", conv_keys=False, -                             **params)
-        self.assert_qmp(result, 'return', {})
          result = self.vm.qmp(cmd, **args) self.assert_qmp(result, 'return', {})

Quoted 8 lines

+
          result = self.vm.qmp("stop") self.assert_qmp(result, 'return', {})
          result = self.vm.qmp("query-block-jobs")
+
          self.assert_qmp(result, 'return[0]/status', 'running') self.assert_qmp(result, 'return[0]/ready', False) +        self.vm.qmp("block-job-cancel", device="drive0", force=True)

won't hurt, but I'd drop it as unrelated to commit

Quoted 8 lines
Quoted 6 lines

+
      def test_drive_mirror(self):
          self.do_test_stop("drive-mirror", device="drive0", -                          target=self.target_img,
+                          target=self.target_img, format=iotests.imgfmt, sync="full")

this doesn't seem need/related too..

It is, because without a @format parameter, the job uses the driver of @device as the output format. That driver is now 'throttle', which doesn’t work as a format.

Max

#16 lore raw

Re: [PATCH 4/7] iotests/129: Use throttle node

Quoted 84 lines · Max Reitz
Quoted 20 lines · Vladimir Sementsov-Ogievskiy
Quoted 19 lines · Max Reitz

Throttling on the BB has not affected block jobs in a while, so it is possible that one of the jobs in 129 finishes before the VM is stopped. We can fix that by running the job from a throttle node.

---
  tests/qemu-iotests/129 | 39 +++++++++++++++------------------------

  1 file changed, 15 insertions(+), 24 deletions(-)
tests/qemu-iotests/129 +1 -1
index dd23bb2e5a..febc806398 100755
@@ -32,20 +32,18 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
           iotests.qemu_img('create', '-f', iotests.imgfmt, self.test_img,
                            "-b", self.base_img, '-F', iotests.imgfmt)
           iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 1M 128M', self.test_img)
-        self.vm = iotests.VM().add_drive(self.test_img)
+        self.vm = iotests.VM()
Quoted 8 lines

tests/qemu-iotests/129 class TestStopWithBlockJob(iotests.QMPTestCase): · line 41

Quoted 6 lines
tests/qemu-iotests/129 +6 -0
+        self.vm.add_object('throttle-group,id=tg0,x-bps-total=1024')
+
+        source_drive = 'driver=throttle,' + \
+                       'throttle-group=tg0,' + \
+                      f'file.driver={iotests.imgfmt},' + \
+                      f'file.file.filename={self.test_img}'

python has a "c-like" string "concatenation", i.e., you may omit "+" operators (I don't know is it a good practice, but I do so:)

OK, why not.

Quoted 52 lines
Quoted 34 lines
tests/qemu-iotests/129 +2 -0
+
+        self.vm.add_drive(None, source_drive)
           self.vm.launch()

      def tearDown(self):
-        params = {"device": "drive0",
-                  "bps": 0,
-                  "bps_rd": 0,
-                  "bps_wr": 0,
-                  "iops": 0,
-                  "iops_rd": 0,
-                  "iops_wr": 0,
-                 }
-        result = self.vm.qmp("block_set_io_throttle", conv_keys=False, -                             **params)
          self.vm.shutdown()
          for img in (self.test_img, self.target_img, self.base_img):
              iotests.try_remove(img)

tests/qemu-iotests/129 +0 -0
@@ -53,33 +51,26 @@ class TestStopWithBlockJob(iotests.QMPTestCase):

      def do_test_stop(self, cmd, **args):
          """Test 'stop' while block job is running on a throttled drive. The 'stop' command shouldn't drain the job"""
-        params = {"device": "drive0",
-                  "bps": 1024,
-                  "bps_rd": 0,
-                  "bps_wr": 0,
-                  "iops": 0,
-                  "iops_rd": 0,
-                  "iops_wr": 0,
-                 }
-        result = self.vm.qmp("block_set_io_throttle", conv_keys=False, -                             **params)
-        self.assert_qmp(result, 'return', {})
          result = self.vm.qmp(cmd, **args) self.assert_qmp(result, 'return', {})

Quoted 8 lines

+
          result = self.vm.qmp("stop") self.assert_qmp(result, 'return', {})
          result = self.vm.qmp("query-block-jobs")
+
          self.assert_qmp(result, 'return[0]/status', 'running') self.assert_qmp(result, 'return[0]/ready', False) +        self.vm.qmp("block-job-cancel", device="drive0", force=True)

won't hurt, but I'd drop it as unrelated to commit

Quoted 6 lines

+
      def test_drive_mirror(self):
          self.do_test_stop("drive-mirror", device="drive0", -                          target=self.target_img,
+                          target=self.target_img, format=iotests.imgfmt, sync="full")

Quoted 4 lines
Quoted 1 line

this doesn't seem need/related too..

It is, because without a @format parameter, the job uses the driver of @device as the output format.  That driver is now 'throttle', which doesn’t work as a format.

Ah, understand now, OK.

Signature · 2 lines
Best regards,
Vladimir
author
#17 lore raw

PATCH 5/7 iotests/129: Actually test a commit job

In reply to Max Reitz

Before this patch, test_block_commit() performs an active commit, which under the hood is a mirror job. If we want to test various different block jobs, we should perhaps run an actual commit job instead.

Doing so requires adding an overlay above the source node before the commit is done (and then specifying the source node as the top node for the commit job).

---
 tests/qemu-iotests/129 | 27 +++++++++++++++++++++++++--
 1 file changed, 25 insertions(+), 2 deletions(-)
tests/qemu-iotests/129 +25 -2
index febc806398..e6753c90ef 100755
@@ -26,6 +26,7 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
     test_img = os.path.join(iotests.test_dir, 'test.img')
     target_img = os.path.join(iotests.test_dir, 'target.img')
     base_img = os.path.join(iotests.test_dir, 'base.img')
+    overlay_img = os.path.join(iotests.test_dir, 'overlay.img')
 
     def setUp(self):
         iotests.qemu_img('create', '-f', iotests.imgfmt, self.base_img, "1G")
@@ -36,6 +37,7 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
         self.vm.add_object('throttle-group,id=tg0,x-bps-total=1024')
 
         source_drive = 'driver=throttle,' + \
+                       'node-name=source,' + \
                        'throttle-group=tg0,' + \
                       f'file.driver={iotests.imgfmt},' + \
                       f'file.file.filename={self.test_img}'
@@ -45,7 +47,8 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
 
     def tearDown(self):
         self.vm.shutdown()
-        for img in (self.test_img, self.target_img, self.base_img):
+        for img in (self.test_img, self.target_img, self.base_img,
+                    self.overlay_img):
             iotests.try_remove(img)
 
     def do_test_stop(self, cmd, **args):
@@ -74,7 +77,27 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
                           sync="full")
 
     def test_block_commit(self):
-        self.do_test_stop("block-commit", device="drive0")
+        # Add overlay above the source node so that we actually use a
+        # commit job instead of a mirror job
+
+        iotests.qemu_img('create', '-f', iotests.imgfmt, self.overlay_img,
+                         '1G')
+
+        result = self.vm.qmp('blockdev-add', **{
+                                 'node-name': 'overlay',
+                                 'driver': iotests.imgfmt,
+                                 'file': {
+                                     'driver': 'file',
+                                     'filename': self.overlay_img
+                                 }
+                             })
+        self.assert_qmp(result, 'return', {})
+
+        result = self.vm.qmp('blockdev-snapshot',
+                             node='source', overlay='overlay')
+        self.assert_qmp(result, 'return', {})
+
+        self.do_test_stop('block-commit', device='drive0', top_node='source')
 
 if __name__ == '__main__':
     iotests.main(supported_fmts=["qcow2"],
Signature · 1 line
2.29.2
Reviewed-by on v1 5/7
#18 lore raw

Re: [PATCH 5/7] iotests/129: Actually test a commit job

tests/qemu-iotests/129

Quoted 14 lines · Max Reitz

Before this patch, test_block_commit() performs an active commit, which under the hood is a mirror job. If we want to test various different block jobs, we should perhaps run an actual commit job instead.

Doing so requires adding an overlay above the source node before the commit is done (and then specifying the source node as the top node for the commit job).

---
 tests/qemu-iotests/129 | 27 +++++++++++++++++++++++++--
 1 file changed, 25 insertions(+), 2 deletions(-)
tests/qemu-iotests/129 +0 -0
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
Reviewed-by on v1 5/7
#19 lore raw

Re: [PATCH 5/7] iotests/129: Actually test a commit job

In reply to Max Reitz
Quoted 9 lines · Max Reitz

Before this patch, test_block_commit() performs an active commit, which under the hood is a mirror job. If we want to test various different block jobs, we should perhaps run an actual commit job instead.

Doing so requires adding an overlay above the source node before the commit is done (and then specifying the source node as the top node for the commit job).

Signature · 2 lines
Best regards,
Vladimir
author
#20 lore raw

PATCH 6/7 iotests/129: Limit mirror job's buffer size

In reply to Max Reitz

Issuing 'stop' on the VM drains all nodes. If the mirror job has many large requests in flight, this may lead to significant I/O that looks a bit like the job is being drained.

We can limit the I/O in flight by limiting the buffer size, so mirror will make very little progress during the 'stop' drain.

(We do not need to do anything about commit, which has a buffer size of 512 kB by default; or backup, which goes cluster by cluster. Once we have asynchronous requests for backup, that will change, but then we can fine-tune the backup job to only perform a single request on a very small chunk, too.)

---
 tests/qemu-iotests/129 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/129 +1 -1
index e6753c90ef..c3ad584ba2 100755
@@ -69,7 +69,7 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
     def test_drive_mirror(self):
         self.do_test_stop("drive-mirror", device="drive0",
                           target=self.target_img, format=iotests.imgfmt,
-                          sync="full")
+                          sync="full", buf_size=65536)
 
     def test_drive_backup(self):
         self.do_test_stop("drive-backup", device="drive0",
Signature · 1 line
2.29.2
Reviewed-by on v1 6/7
#21 lore raw

Re: [PATCH 6/7] iotests/129: Limit mirror job's buffer size

Quoted 13 lines · Max Reitz

Issuing 'stop' on the VM drains all nodes. If the mirror job has many large requests in flight, this may lead to significant I/O that looks a bit like the job is being drained.

We can limit the I/O in flight by limiting the buffer size, so mirror will make very little progress during the 'stop' drain.

(We do not need to do anything about commit, which has a buffer size of 512 kB by default; or backup, which goes cluster by cluster. Once we have asynchronous requests for backup, that will change, but then we can fine-tune the backup job to only perform a single request on a very small chunk, too.)

Quoted 4 lines
---
 tests/qemu-iotests/129 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 15 lines
tests/qemu-iotests/129 +1 -1
index e6753c90ef..c3ad584ba2 100755
@@ -69,7 +69,7 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
     def test_drive_mirror(self):
         self.do_test_stop("drive-mirror", device="drive0",
                           target=self.target_img, format=iotests.imgfmt,
-                          sync="full")
+                          sync="full", buf_size=65536)
 
     def test_drive_backup(self):
         self.do_test_stop("drive-backup", device="drive0",
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
Reviewed-by on v1 6/7
#22 lore raw

Re: [PATCH 6/7] iotests/129: Limit mirror job's buffer size

In reply to Max Reitz
Quoted 3 lines · Max Reitz

Issuing 'stop' on the VM drains all nodes. If the mirror job has many large requests in flight, this may lead to significant I/O that looks a bit like the job is being drained.

what do you mean by "looks like the job is being drained"? If "drain" in Qemu means "flush all in-flight requests, and don't create more during the drain", then job is exactly "drained" during the drain..

do you mean, that it looks like "job trying to finish copying the whole drive"? Or "job hangs", or "stop hangs"...

Quoted 9 lines

We can limit the I/O in flight by limiting the buffer size, so mirror will make very little progress during the 'stop' drain.

(We do not need to do anything about commit, which has a buffer size of 512 kB by default; or backup, which goes cluster by cluster. Once we have asynchronous requests for backup, that will change, but then we can fine-tune the backup job to only perform a single request on a very small chunk, too.)

OK

Quoted 2 lines
Quoted 18 lines
---
  tests/qemu-iotests/129 | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/129 +1 -1
index e6753c90ef..c3ad584ba2 100755
@@ -69,7 +69,7 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
      def test_drive_mirror(self):
          self.do_test_stop("drive-mirror", device="drive0",
                            target=self.target_img, format=iotests.imgfmt,
-                          sync="full")
+                          sync="full", buf_size=65536)
 
      def test_drive_backup(self):
          self.do_test_stop("drive-backup", device="drive0",
Signature · 2 lines
Best regards,
Vladimir
author
#23 lore raw

PATCH 7/7 iotests/129: Clean up pylint and mypy complaints

In reply to Max Reitz
---
 tests/qemu-iotests/129 | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
tests/qemu-iotests/129 +4 -2
index c3ad584ba2..ec303069e9 100755
@@ -20,9 +20,10 @@
 
 import os
 import iotests
-import time
 
 class TestStopWithBlockJob(iotests.QMPTestCase):
+    assert iotests.test_dir is not None
+
     test_img = os.path.join(iotests.test_dir, 'test.img')
     target_img = os.path.join(iotests.test_dir, 'target.img')
     base_img = os.path.join(iotests.test_dir, 'base.img')
@@ -32,7 +33,8 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
         iotests.qemu_img('create', '-f', iotests.imgfmt, self.base_img, "1G")
         iotests.qemu_img('create', '-f', iotests.imgfmt, self.test_img,
                          "-b", self.base_img, '-F', iotests.imgfmt)
-        iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 1M 128M', self.test_img)
+        iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 1M 128M',
+                        self.test_img)
         self.vm = iotests.VM()
         self.vm.add_object('throttle-group,id=tg0,x-bps-total=1024')
 
Signature · 1 line
2.29.2
Reviewed-by on v1 7/7
#24 lore raw

Re: [PATCH 7/7] iotests/129: Clean up pylint and mypy complaints

Quoted 5 lines · Max Reitz
---
 tests/qemu-iotests/129 | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#25 lore raw
In reply to Max Reitz
Quoted 15 lines · Max Reitz

Hi,

There are some problems with iotests 129 (perhaps more than these, but these are the ones I know of):

1. It checks @busy to see whether a block job is still running; however, block jobs tend to unset @busy all the time (when they yield). [Fixed by patch 3]

2. It uses blockdev throttling, which quite some time ago has been moved to the BB level; since then, such throttling will no longer affect block jobs. We can get throttling to work by using a throttle filter node.
[Fixed by patch 4]

Quoted 3 lines

3. The mirror job has a large buffer size by default. A simple drain may lead to it making significant process, which is kind of dangerous, because we don’t want the job to complete.

Not quite clear to me. iotest 129 wants to mirror 128M of data. Mirror by default will have 1M chunk size and maximum of 16 parallel requests. So with throttling (even if throttling can't correctly handle parallel requests) we will not exceed 16M of progress.. Why we need limiting buffer size?

Quoted 34 lines

To get around this, we can simply limit its buffer size. (And we should make the commit job an actual commit job instead of an active commit (which is just mirror), because the commit interface does not allow setting a buffer size.)
[Fixed by patches 5 and 6]

This series fixes those things, and now 129 seems to reliably pass for me.

Apart from the major issues above, there are also minor flaws:

- It doesn’t remove the test images.
[Fixed by patches 1 and 2]

- pylint and mypy complain.
(Running mypy with the options given in 297.) [Patch 4 removes one pylint complaint; patch 7 the rest.]

Max Reitz (7):
iotests: Move try_remove to iotests.py iotests/129: Remove test images in tearDown() iotests/129: Do not check @busy iotests/129: Use throttle node iotests/129: Actually test a commit job iotests/129: Limit mirror job's buffer size iotests/129: Clean up pylint and mypy complaints

  tests/qemu-iotests/124        |  8 +---
  tests/qemu-iotests/129        | 76 ++++++++++++++++++++++-------------
  tests/qemu-iotests/iotests.py | 11 +++--
  3 files changed, 55 insertions(+), 40 deletions(-)
Signature · 2 lines
Best regards,
Vladimir
author
#26 lore raw
Quoted 16 lines · Vladimir Sementsov-Ogievskiy
Quoted 15 lines · Max Reitz

Hi,

There are some problems with iotests 129 (perhaps more than these, but these are the ones I know of):

1. It checks @busy to see whether a block job is still running; however, block jobs tend to unset @busy all the time (when they yield). [Fixed by patch 3]

2. It uses blockdev throttling, which quite some time ago has been moved to the BB level; since then, such throttling will no longer affect block jobs.  We can get throttling to work by using a throttle filter node.
    [Fixed by patch 4]

Quoted 8 lines
Quoted 3 lines

3. The mirror job has a large buffer size by default.  A simple drain may lead to it making significant process, which is kind of dangerous, because we don’t want the job to complete.

Not quite clear to me. iotest 129 wants to mirror 128M of data. Mirror by default will have 1M chunk size and maximum of 16 parallel requests. So with throttling (even if throttling can't correctly handle parallel requests) we will not exceed 16M of progress.. Why we need limiting buffer size?

It does exceed 16M of progress; without the limit, I generally see something between 16M and 32M.

Now, that still is below 128M, but it’s kind of in the same magnitude. I don’t feel comfortable with that, especially given it’s so easy to limit it to much less (buf_size=64k makes the job proceed to 128k).

Also, maybe the default is increased in the future. Increasing the chunk size by 4 would mean that it might be possible to reach 128M.

I find not relying on the default better.

Max

#27 lore raw
Quoted 26 lines · Max Reitz
Quoted 16 lines · Vladimir Sementsov-Ogievskiy
Quoted 15 lines · Max Reitz

Hi,

There are some problems with iotests 129 (perhaps more than these, but these are the ones I know of):

1. It checks @busy to see whether a block job is still running; however, block jobs tend to unset @busy all the time (when they yield). [Fixed by patch 3]

2. It uses blockdev throttling, which quite some time ago has been moved to the BB level; since then, such throttling will no longer affect block jobs.  We can get throttling to work by using a throttle filter node.
    [Fixed by patch 4]

Quoted 8 lines
Quoted 3 lines

3. The mirror job has a large buffer size by default.  A simple drain may lead to it making significant process, which is kind of dangerous, because we don’t want the job to complete.

Not quite clear to me. iotest 129 wants to mirror 128M of data. Mirror by default will have 1M chunk size and maximum of 16 parallel requests. So with throttling (even if throttling can't correctly handle parallel requests) we will not exceed 16M of progress.. Why we need limiting buffer size?

Quoted 7 lines

It does exceed 16M of progress; without the limit, I generally see something between 16M and 32M.

Now, that still is below 128M, but it’s kind of in the same magnitude. I don’t feel comfortable with that, especially given it’s so easy to limit it to much less (buf_size=64k makes the job proceed to 128k).

Also, maybe the default is increased in the future.  Increasing the chunk size by 4 would mean that it might be possible to reach 128M.

I find not relying on the default better.

Hmm, OK, agreed

Signature · 2 lines
Best regards,
Vladimir
#28 lore raw
In reply to Max Reitz

Am 13.01.2021 um 15:06 hat Max Reitz geschrieben:

Quoted 3 lines

- pylint and mypy complain.
(Running mypy with the options given in 297.) [Patch 4 removes one pylint complaint; patch 7 the rest.]

Should we add it to 297 then to make sure we won't regress?

At some point, I guess we'll want to cover all Python tests, but I assume it was too much to change in the same series as for the iotests.py cleanup.

Kevin

#29 lore raw
Quoted 11 lines · Kevin Wolf

Am 13.01.2021 um 15:06 hat Max Reitz geschrieben:

Quoted 3 lines

- pylint and mypy complain.
(Running mypy with the options given in 297.) [Patch 4 removes one pylint complaint; patch 7 the rest.]

Should we add it to 297 then to make sure we won't regress?

At some point, I guess we'll want to cover all Python tests, but I assume it was too much to change in the same series as for the iotests.py cleanup.

Related:

anyone knows, what's the difference between pylint and pylint-3?

I know I can do pip3 install pylint, which brings pylint binary.. Also there is pylint fedora package which provides pylint binary and python3-pylint fedora package which provides pylint-3 binary, both package names looks so that they are... the same?

Also. Interesting, but pylint don't check PEP8 code style (or at least not everything of it). For example, pycodestyle has a lot of complains about iotests.py (most of them "E302 expected 2 blank lines, found 1").. So, what about adding pycodestyle (or may be better flake8 which is pycodestyle + some additional things) to 127 iotest?

Signature · 2 lines
Best regards,
Vladimir
author
#30 lore raw
In reply to Kevin Wolf
Quoted 6 lines · Kevin Wolf

Am 13.01.2021 um 15:06 hat Max Reitz geschrieben:

Quoted 3 lines

- pylint and mypy complain.
(Running mypy with the options given in 297.) [Patch 4 removes one pylint complaint; patch 7 the rest.]

Should we add it to 297 then to make sure we won't regress?

Sounds like a good precedent to set indeed.

Quoted 3 lines

At some point, I guess we'll want to cover all Python tests, but I assume it was too much to change in the same series as for the iotests.py cleanup.

Originally, I hadn’t intended to write patch 7 at all; I just wanted to see what pylint/mypy said to my changes in this series, but then they made me aware of pre-existing litter. I thought adding patch 7 would be the right thing to do, so I did.

(That’s to say so far I had no intentions of cleaning up all Python tests. It’s just coincidence that I did so for 129.)

Max

#31 lore raw

Am 13.01.2021 um 16:26 hat Max Reitz geschrieben:

Quoted 13 lines · Kevin Wolf
Quoted 6 lines

Am 13.01.2021 um 15:06 hat Max Reitz geschrieben:

Quoted 3 lines

- pylint and mypy complain.
(Running mypy with the options given in 297.) [Patch 4 removes one pylint complaint; patch 7 the rest.]

Should we add it to 297 then to make sure we won't regress?

Sounds like a good precedent to set indeed.

Quoted 3 lines

At some point, I guess we'll want to cover all Python tests, but I assume it was too much to change in the same series as for the iotests.py cleanup.

Quoted 7 lines

Originally, I hadn’t intended to write patch 7 at all; I just wanted to see what pylint/mypy said to my changes in this series, but then they made me aware of pre-existing litter. I thought adding patch 7 would be the right thing to do, so I did.

(That’s to say so far I had no intentions of cleaning up all Python tests. It’s just coincidence that I did so for 129.)

Indeed, and I wasn't trying to imply that you should, but just making a guess why we cover iotests.py and nothing else so far.

Cleaning up 129 is a nice side effect of this series. If more such side effects accumulate and the rest becomes small enough, I might eventually just convert the rest. Or I might not, who knows.

Kevin