Skip to contentSkip to conversation

Rework iotests/check

PATCH Superseded Vladimir Sementsov-Ogievskiy 4 participants 46 messages Jan 16, 2021 – Jan 25, 2021

Subsystems (2)

v7 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v7 cover Rework iotests/check

Hi all!

These series has 3 goals:

- get rid of group file (to forget about rebase and in-list conflicts)
- introduce human-readable names for tests
- rewrite check into python

v7:
- fix wording and grammar
- satisfy python linters
- move argv interfaces all into one in new check script
- support '-n' == '--dry-run' option
- update check-block to run check with correct PYTHON

findtests: - stop parsing test file after first '# group: ' line
- use parse_test_name in add_group_file()
- make new list instead of modifying parameter exclude_groups

testenv: - fix machine_map
- fix self.python

testrunner: use env.python to run python tests

Vladimir Sementsov-Ogievskiy (11):
iotests/277: use dot slash for nbd-fault-injector.py running iotests/303: use dot slash for qcow2.py running
iotests: fix some whitespaces in test output files
iotests: make tests executable
iotests/294: add shebang line
iotests: define group in each iotest
iotests: add findtests.py
iotests: add testenv.py
iotests: add testrunner.py
iotests: rewrite check into python
iotests: rename and move 169 and 199 tests

 docs/devel/testing.rst                        |   50 +-
 Makefile                                      |    1 -
 tests/check-block.sh                          |    2 +-
 tests/qemu-iotests/001                        |    1 +
 tests/qemu-iotests/002                        |    1 +
 tests/qemu-iotests/003                        |    1 +
 tests/qemu-iotests/004                        |    1 +
 tests/qemu-iotests/005                        |    1 +
 tests/qemu-iotests/007                        |    1 +
 tests/qemu-iotests/008                        |    1 +
 tests/qemu-iotests/009                        |    1 +
 tests/qemu-iotests/010                        |    1 +
 tests/qemu-iotests/011                        |    1 +
 tests/qemu-iotests/012                        |    1 +
 tests/qemu-iotests/013                        |    1 +
 tests/qemu-iotests/014                        |    1 +
 tests/qemu-iotests/015                        |    1 +
 tests/qemu-iotests/017                        |    1 +
 tests/qemu-iotests/018                        |    1 +
 tests/qemu-iotests/019                        |    1 +
 tests/qemu-iotests/020                        |    1 +
 tests/qemu-iotests/021                        |    1 +
 tests/qemu-iotests/022                        |    1 +
 tests/qemu-iotests/023                        |    1 +
 tests/qemu-iotests/024                        |    1 +
 tests/qemu-iotests/025                        |    1 +
 tests/qemu-iotests/026                        |    1 +
 tests/qemu-iotests/027                        |    1 +
 tests/qemu-iotests/028                        |    1 +
 tests/qemu-iotests/029                        |    1 +
 tests/qemu-iotests/030                        |    1 +
 tests/qemu-iotests/031                        |    1 +
 tests/qemu-iotests/032                        |    1 +
 tests/qemu-iotests/033                        |    1 +
 tests/qemu-iotests/034                        |    1 +
 tests/qemu-iotests/035                        |    1 +
 tests/qemu-iotests/036                        |    1 +
 tests/qemu-iotests/037                        |    1 +
 tests/qemu-iotests/038                        |    1 +
 tests/qemu-iotests/039                        |    1 +
 tests/qemu-iotests/040                        |    1 +
 tests/qemu-iotests/041                        |    1 +
 tests/qemu-iotests/042                        |    1 +
 tests/qemu-iotests/043                        |    1 +
 tests/qemu-iotests/044                        |    1 +
 tests/qemu-iotests/045                        |    1 +
 tests/qemu-iotests/046                        |    1 +
 tests/qemu-iotests/047                        |    1 +
 tests/qemu-iotests/048                        |    1 +
 tests/qemu-iotests/049                        |    1 +
 tests/qemu-iotests/050                        |    1 +
 tests/qemu-iotests/051                        |    1 +
 tests/qemu-iotests/052                        |    1 +
 tests/qemu-iotests/053                        |    1 +
 tests/qemu-iotests/054                        |    1 +
 tests/qemu-iotests/055                        |    1 +
 tests/qemu-iotests/056                        |    1 +
 tests/qemu-iotests/057                        |    1 +
 tests/qemu-iotests/058                        |    1 +
 tests/qemu-iotests/059                        |    1 +
 tests/qemu-iotests/060                        |    1 +
 tests/qemu-iotests/061                        |    1 +
 tests/qemu-iotests/062                        |    1 +
 tests/qemu-iotests/063                        |    1 +
 tests/qemu-iotests/064                        |    1 +
 tests/qemu-iotests/065                        |    1 +
 tests/qemu-iotests/066                        |    1 +
 tests/qemu-iotests/068                        |    1 +
 tests/qemu-iotests/069                        |    1 +
 tests/qemu-iotests/070                        |    1 +
 tests/qemu-iotests/071                        |    1 +
 tests/qemu-iotests/072                        |    1 +
 tests/qemu-iotests/073                        |    1 +
 tests/qemu-iotests/074                        |    1 +
 tests/qemu-iotests/075                        |    1 +
 tests/qemu-iotests/076                        |    1 +
 tests/qemu-iotests/077                        |    1 +
 tests/qemu-iotests/078                        |    1 +
 tests/qemu-iotests/079                        |    1 +
 tests/qemu-iotests/080                        |    1 +
 tests/qemu-iotests/081                        |    1 +
 tests/qemu-iotests/082                        |    1 +
 tests/qemu-iotests/083                        |    1 +
 tests/qemu-iotests/084                        |    1 +
 tests/qemu-iotests/085                        |    1 +
 tests/qemu-iotests/086                        |    1 +
 tests/qemu-iotests/087                        |    1 +
 tests/qemu-iotests/088                        |    1 +
 tests/qemu-iotests/089                        |    1 +
 tests/qemu-iotests/090                        |    1 +
 tests/qemu-iotests/091                        |    1 +
 tests/qemu-iotests/092                        |    1 +
 tests/qemu-iotests/093                        |    1 +
 tests/qemu-iotests/094                        |    1 +
 tests/qemu-iotests/095                        |    1 +
 tests/qemu-iotests/096                        |    1 +
 tests/qemu-iotests/097                        |    1 +
 tests/qemu-iotests/098                        |    1 +
 tests/qemu-iotests/099                        |    1 +
 tests/qemu-iotests/101                        |    1 +
 tests/qemu-iotests/102                        |    1 +
 tests/qemu-iotests/103                        |    1 +
 tests/qemu-iotests/104                        |    1 +
 tests/qemu-iotests/105                        |    1 +
 tests/qemu-iotests/106                        |    1 +
 tests/qemu-iotests/107                        |    1 +
 tests/qemu-iotests/108                        |    1 +
 tests/qemu-iotests/109                        |    1 +
 tests/qemu-iotests/110                        |    1 +
 tests/qemu-iotests/111                        |    1 +
 tests/qemu-iotests/112                        |    1 +
 tests/qemu-iotests/113                        |    1 +
 tests/qemu-iotests/114                        |    1 +
 tests/qemu-iotests/115                        |    1 +
 tests/qemu-iotests/116                        |    1 +
 tests/qemu-iotests/117                        |    1 +
 tests/qemu-iotests/118                        |    1 +
 tests/qemu-iotests/119                        |    1 +
 tests/qemu-iotests/120                        |    1 +
 tests/qemu-iotests/121                        |    1 +
 tests/qemu-iotests/122                        |    1 +
 tests/qemu-iotests/123                        |    1 +
 tests/qemu-iotests/124                        |    1 +
 tests/qemu-iotests/125                        |    1 +
 tests/qemu-iotests/126                        |    1 +
 tests/qemu-iotests/127                        |    1 +
 tests/qemu-iotests/128                        |    1 +
 tests/qemu-iotests/129                        |    1 +
 tests/qemu-iotests/130                        |    1 +
 tests/qemu-iotests/131                        |    1 +
 tests/qemu-iotests/132                        |    1 +
 tests/qemu-iotests/133                        |    1 +
 tests/qemu-iotests/134                        |    1 +
 tests/qemu-iotests/135                        |    1 +
 tests/qemu-iotests/136                        |    1 +
 tests/qemu-iotests/137                        |    1 +
 tests/qemu-iotests/138                        |    1 +
 tests/qemu-iotests/139                        |    1 +
 tests/qemu-iotests/140                        |    1 +
 tests/qemu-iotests/141                        |    1 +
 tests/qemu-iotests/143                        |    1 +
 tests/qemu-iotests/144                        |    1 +
 tests/qemu-iotests/145                        |    1 +
 tests/qemu-iotests/146                        |    1 +
 tests/qemu-iotests/147                        |    1 +
 tests/qemu-iotests/148                        |    1 +
 tests/qemu-iotests/149                        |    1 +
 tests/qemu-iotests/150                        |    1 +
 tests/qemu-iotests/151                        |    1 +
 tests/qemu-iotests/152                        |    1 +
 tests/qemu-iotests/153                        |    1 +
 tests/qemu-iotests/154                        |    1 +
 tests/qemu-iotests/155                        |    1 +
 tests/qemu-iotests/156                        |    1 +
 tests/qemu-iotests/157                        |    1 +
 tests/qemu-iotests/158                        |    1 +
 tests/qemu-iotests/159                        |    1 +
 tests/qemu-iotests/160                        |    1 +
 tests/qemu-iotests/161                        |    1 +
 tests/qemu-iotests/162                        |    1 +
 tests/qemu-iotests/163                        |    1 +
 tests/qemu-iotests/165                        |    1 +
 tests/qemu-iotests/170                        |    1 +
 tests/qemu-iotests/171                        |    1 +
 tests/qemu-iotests/172                        |    1 +
 tests/qemu-iotests/173                        |    1 +
 tests/qemu-iotests/174                        |    1 +
 tests/qemu-iotests/175                        |    1 +
 tests/qemu-iotests/175.out                    |    2 +-
 tests/qemu-iotests/176                        |    1 +
 tests/qemu-iotests/177                        |    1 +
 tests/qemu-iotests/178                        |    1 +
 tests/qemu-iotests/179                        |    1 +
 tests/qemu-iotests/181                        |    1 +
 tests/qemu-iotests/182                        |    1 +
 tests/qemu-iotests/183                        |    1 +
 tests/qemu-iotests/184                        |    1 +
 tests/qemu-iotests/185                        |    1 +
 tests/qemu-iotests/186                        |    1 +
 tests/qemu-iotests/187                        |    1 +
 tests/qemu-iotests/188                        |    1 +
 tests/qemu-iotests/189                        |    1 +
 tests/qemu-iotests/190                        |    1 +
 tests/qemu-iotests/191                        |    1 +
 tests/qemu-iotests/192                        |    1 +
 tests/qemu-iotests/194                        |    1 +
 tests/qemu-iotests/195                        |    1 +
 tests/qemu-iotests/196                        |    1 +
 tests/qemu-iotests/197                        |    1 +
 tests/qemu-iotests/198                        |    1 +
 tests/qemu-iotests/200                        |    1 +
 tests/qemu-iotests/201                        |    1 +
 tests/qemu-iotests/202                        |    1 +
 tests/qemu-iotests/203                        |    1 +
 tests/qemu-iotests/204                        |    1 +
 tests/qemu-iotests/205                        |    1 +
 tests/qemu-iotests/206                        |    1 +
 tests/qemu-iotests/207                        |    1 +
 tests/qemu-iotests/208                        |    1 +
 tests/qemu-iotests/209                        |    1 +
 tests/qemu-iotests/210                        |    1 +
 tests/qemu-iotests/211                        |    1 +
 tests/qemu-iotests/212                        |    1 +
 tests/qemu-iotests/213                        |    1 +
 tests/qemu-iotests/214                        |    1 +
 tests/qemu-iotests/215                        |    1 +
 tests/qemu-iotests/216                        |    1 +
 tests/qemu-iotests/217                        |    1 +
 tests/qemu-iotests/218                        |    1 +
 tests/qemu-iotests/219                        |    1 +
 tests/qemu-iotests/220                        |    1 +
 tests/qemu-iotests/221                        |    1 +
 tests/qemu-iotests/222                        |    1 +
 tests/qemu-iotests/223                        |    1 +
 tests/qemu-iotests/224                        |    1 +
 tests/qemu-iotests/225                        |    1 +
 tests/qemu-iotests/226                        |    1 +
 tests/qemu-iotests/227                        |    1 +
 tests/qemu-iotests/228                        |    1 +
 tests/qemu-iotests/229                        |    1 +
 tests/qemu-iotests/231                        |    1 +
 tests/qemu-iotests/232                        |    1 +
 tests/qemu-iotests/233                        |    1 +
 tests/qemu-iotests/234                        |    1 +
 tests/qemu-iotests/235                        |    1 +
 tests/qemu-iotests/236                        |    1 +
 tests/qemu-iotests/237                        |    1 +
 tests/qemu-iotests/238                        |    1 +
 tests/qemu-iotests/239                        |    1 +
 tests/qemu-iotests/240                        |    1 +
 tests/qemu-iotests/241                        |    1 +
 tests/qemu-iotests/242                        |    1 +
 tests/qemu-iotests/243                        |    1 +
 tests/qemu-iotests/244                        |    1 +
 tests/qemu-iotests/245                        |    1 +
 tests/qemu-iotests/246                        |    1 +
 tests/qemu-iotests/247                        |    1 +
 tests/qemu-iotests/248                        |    1 +
 tests/qemu-iotests/249                        |    1 +
 tests/qemu-iotests/250                        |    1 +
 tests/qemu-iotests/251                        |    1 +
 tests/qemu-iotests/252                        |    1 +
 tests/qemu-iotests/253                        |    1 +
 tests/qemu-iotests/254                        |    1 +
 tests/qemu-iotests/255                        |    1 +
 tests/qemu-iotests/256                        |    1 +
 tests/qemu-iotests/257                        |    1 +
 tests/qemu-iotests/258                        |    1 +
 tests/qemu-iotests/259                        |    1 +
 tests/qemu-iotests/260                        |    1 +
 tests/qemu-iotests/261                        |    1 +
 tests/qemu-iotests/262                        |    1 +
 tests/qemu-iotests/263                        |    1 +
 tests/qemu-iotests/264                        |    1 +
 tests/qemu-iotests/265                        |    1 +
 tests/qemu-iotests/266                        |    1 +
 tests/qemu-iotests/267                        |    1 +
 tests/qemu-iotests/268                        |    1 +
 tests/qemu-iotests/270                        |    1 +
 tests/qemu-iotests/271                        |    1 +
 tests/qemu-iotests/271.out                    |   12 +-
 tests/qemu-iotests/272                        |    1 +
 tests/qemu-iotests/273                        |    1 +
 tests/qemu-iotests/274                        |    1 +
 tests/qemu-iotests/277                        |    3 +-
 tests/qemu-iotests/279                        |    1 +
 tests/qemu-iotests/280                        |    1 +
 tests/qemu-iotests/281                        |    1 +
 tests/qemu-iotests/282                        |    1 +
 tests/qemu-iotests/283                        |    1 +
 tests/qemu-iotests/284                        |    1 +
 tests/qemu-iotests/286                        |    1 +
 tests/qemu-iotests/287                        |    1 +
 tests/qemu-iotests/287.out                    |   10 +-
 tests/qemu-iotests/288                        |    1 +
 tests/qemu-iotests/289                        |    1 +
 tests/qemu-iotests/290                        |    1 +
 tests/qemu-iotests/291                        |    1 +
 tests/qemu-iotests/292                        |    1 +
 tests/qemu-iotests/293                        |    1 +
 tests/qemu-iotests/294                        |    2 +
 tests/qemu-iotests/295                        |    1 +
 tests/qemu-iotests/296                        |    1 +
 tests/qemu-iotests/297                        |    1 +
 tests/qemu-iotests/298                        |    0
 tests/qemu-iotests/299                        |    1 +
 tests/qemu-iotests/300                        |    1 +
 tests/qemu-iotests/301                        |    1 +
 tests/qemu-iotests/302                        |    1 +
 tests/qemu-iotests/303                        |    3 +-
 tests/qemu-iotests/304                        |    1 +
 tests/qemu-iotests/305                        |    1 +
 tests/qemu-iotests/307                        |    1 +
 tests/qemu-iotests/308                        |    1 +
 tests/qemu-iotests/309                        |    1 +
 tests/qemu-iotests/312                        |    1 +
 tests/qemu-iotests/check                      | 1085 ++---------------
 tests/qemu-iotests/common.env.in              |    3 -
 tests/qemu-iotests/findtests.py               |  159 +++
 tests/qemu-iotests/group                      |  321 -----
 tests/qemu-iotests/meson.build                |    3 -
 tests/qemu-iotests/testenv.py                 |  255 ++++
 tests/qemu-iotests/testrunner.py              |  344 ++++++
 .../migrate-bitmaps-postcopy-test}            |    1 +
 .../migrate-bitmaps-postcopy-test.out}        |    0
 .../{169 => tests/migrate-bitmaps-test}       |    1 +
 .../migrate-bitmaps-test.out}                 |    0
 307 files changed, 1233 insertions(+), 1310 deletions(-)
 mode change 100644 => 100755 tests/qemu-iotests/283
 mode change 100644 => 100755 tests/qemu-iotests/298
 mode change 100644 => 100755 tests/qemu-iotests/299
 delete mode 100644 tests/qemu-iotests/common.env.in
 create mode 100644 tests/qemu-iotests/findtests.py
 delete mode 100644 tests/qemu-iotests/group
 create mode 100644 tests/qemu-iotests/testenv.py
 create mode 100644 tests/qemu-iotests/testrunner.py
 rename tests/qemu-iotests/{199 => tests/migrate-bitmaps-postcopy-test} (99%)
 rename tests/qemu-iotests/{199.out => tests/migrate-bitmaps-postcopy-test.out} (100%)
 rename tests/qemu-iotests/{169 => tests/migrate-bitmaps-test} (99%)
 rename tests/qemu-iotests/{169.out => tests/migrate-bitmaps-test.out} (100%)
Signature · 1 line
2.29.2
#1 lore raw

PATCH v7 1/11 iotests/277: use dot slash for nbd-fault-injector.py running

If you run './check 277', check includes common.config which adjusts $PATH to include '.' first, and therefore finds nbd-fault-injector.py on PATH. But if you run './277' directly, there is nothing to adjust PATH, and if '.' is not already on your PATH by other means, the test fails because the executable is not found. Adjust how we invoke the helper executable to avoid needing a PATH search in the first place.

---
 tests/qemu-iotests/277 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/277 +1 -1
index d34f87021f..a39ce2d873 100755
@@ -42,7 +42,7 @@ def make_conf_file(event):
 def start_server_NBD(event):
     make_conf_file(event)
 
-    srv = subprocess.Popen(['nbd-fault-injector.py', '--classic-negotiation',
+    srv = subprocess.Popen(['./nbd-fault-injector.py', '--classic-negotiation',
                            nbd_sock, conf_file], stdout=subprocess.PIPE,
                            stderr=subprocess.STDOUT, universal_newlines=True)
     line = srv.stdout.readline()
Signature · 1 line
2.29.2
#2 lore raw

PATCH v7 2/11 iotests/303: use dot slash for qcow2.py running

In reply to Vladimir Sementsov-Ogievskiy

If you run './check 303', check includes common.config which adjusts $PATH to include '.' first, and therefore finds qcow2.py on PATH. But if you run './303' directly, there is nothing to adjust PATH, and if '.' is not already on your PATH by other means, the test fails because the executable is not found. Adjust how we invoke the helper executable to avoid needing a PATH search in the first place.

---
 tests/qemu-iotests/303 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/303 +1 -1
index 6c21774483..11cd9eeb26 100755
@@ -56,7 +56,7 @@ qemu_img_create('-f', iotests.imgfmt, disk, '10M')
 
 add_bitmap(1, 0, 6, False)
 add_bitmap(2, 6, 8, True)
-dump = ['qcow2.py', disk, 'dump-header']
+dump = ['./qcow2.py', disk, 'dump-header']
 subprocess.run(dump)
 # Dump the metadata in JSON format
 dump.append('-j')
Signature · 1 line
2.29.2
#3 lore raw

PATCH v7 3/11 iotests: fix some whitespaces in test output files

In reply to Vladimir Sementsov-Ogievskiy

We are going to be stricter about comparing test result with .out files. So, fix some whitespaces now.

---
 tests/qemu-iotests/175.out |  2 +-
 tests/qemu-iotests/271.out | 12 ++++++------
 tests/qemu-iotests/287.out | 10 +++++-----
 3 files changed, 12 insertions(+), 12 deletions(-)
tests/qemu-iotests/175.out +1 -1
index 39c2ee0f62..40a5bd1ce6 100644
@@ -23,4 +23,4 @@ size=4096, min allocation
 == resize empty image with block_resize ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=0
 size=1048576, min allocation
- *** done
+*** done
tests/qemu-iotests/271.out +6 -6
index 92deb7ebb0..81043ba4d7 100644
@@ -500,7 +500,7 @@ L2 entry #0: 0x8000000000050001 0000000000000001
 
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576
 L2 entry #1: 0x8000000000060000 00000001ffffffff
-qcow2: Marking image as corrupt: Invalid cluster entry found  (L2 offset: 0x40000, L2 index: 0x1); further corruption events will be suppressed
+qcow2: Marking image as corrupt: Invalid cluster entry found (L2 offset: 0x40000, L2 index: 0x1); further corruption events will be suppressed
 write failed: Input/output error
 
 ### Corrupted L2 entries - write test (unallocated) ###
@@ -515,14 +515,14 @@ L2 entry #0: 0x8000000000060000 0000000000000001
 
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576
 L2 entry #0: 0x0000000000000000 0000000000000001
-qcow2: Marking image as corrupt: Invalid cluster entry found  (L2 offset: 0x40000, L2 index: 0); further corruption events will be suppressed
+qcow2: Marking image as corrupt: Invalid cluster entry found (L2 offset: 0x40000, L2 index: 0); further corruption events will be suppressed
 write failed: Input/output error
 
 # Both 'subcluster is zero' and 'subcluster is allocated' bits set
 
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576
 L2 entry #1: 0x0000000000000000 0000000100000001
-qcow2: Marking image as corrupt: Invalid cluster entry found  (L2 offset: 0x40000, L2 index: 0x1); further corruption events will be suppressed
+qcow2: Marking image as corrupt: Invalid cluster entry found (L2 offset: 0x40000, L2 index: 0x1); further corruption events will be suppressed
 write failed: Input/output error
 
 ### Compressed cluster with subcluster bitmap != 0 - write test ###
@@ -583,7 +583,7 @@ read 524288/524288 bytes at offset 0
 read 524288/524288 bytes at offset 524288
 512 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 Offset          Length          Mapped to       File
-0               0x80000        0               TEST_DIR/t.qcow2.base
+0               0x80000         0               TEST_DIR/t.qcow2.base
 # backing file and preallocation=falloc
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=524288 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=raw preallocation=falloc
 Image resized.
@@ -592,7 +592,7 @@ read 524288/524288 bytes at offset 0
 read 524288/524288 bytes at offset 524288
 512 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 Offset          Length          Mapped to       File
-0               0x80000        0               TEST_DIR/t.qcow2.base
+0               0x80000         0               TEST_DIR/t.qcow2.base
 # backing file and preallocation=full
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=524288 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=raw preallocation=full
 Image resized.
@@ -601,7 +601,7 @@ read 524288/524288 bytes at offset 0
 read 524288/524288 bytes at offset 524288
 512 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 Offset          Length          Mapped to       File
-0               0x80000        0               TEST_DIR/t.qcow2.base
+0               0x80000         0               TEST_DIR/t.qcow2.base
 
 ### Image resizing with preallocation and backing files ###
 
tests/qemu-iotests/287.out +5 -5
index 6b9dfb4af0..49ab6a27d5 100644
@@ -10,22 +10,22 @@ incompatible_features     []
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 incompatible_features     [3]
 
-=== Testing zlib with incompatible bit set  ===
+=== Testing zlib with incompatible bit set ===
 
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 incompatible_features     [3]
 
-=== Testing zstd with incompatible bit unset  ===
+=== Testing zstd with incompatible bit unset ===
 
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 incompatible_features     []
 
-=== Testing compression type values  ===
+=== Testing compression type values ===
 
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
-   0
+0
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
-   1
+1
 
 === Testing simple reading and writing with zstd ===
 
Signature · 1 line
2.29.2
#4 lore raw

PATCH v7 4/11 iotests: make tests executable

In reply to Vladimir Sementsov-Ogievskiy

All other test files are executable. Fix these.

---
 tests/qemu-iotests/283 | 0
 tests/qemu-iotests/298 | 0
 tests/qemu-iotests/299 | 0
 3 files changed, 0 insertions(+), 0 deletions(-)
 mode change 100644 => 100755 tests/qemu-iotests/283
 mode change 100644 => 100755 tests/qemu-iotests/298
 mode change 100644 => 100755 tests/qemu-iotests/299
tests/qemu-iotests/283 +0 -0
old mode 100644
new mode 100755
tests/qemu-iotests/298 +0 -0
old mode 100644
new mode 100755
tests/qemu-iotests/299 +0 -0
old mode 100644
new mode 100755
Signature · 1 line
2.29.2
#6 lore raw

PATCH v7 6/11 iotests: define group in each iotest

In reply to Vladimir Sementsov-Ogievskiy

We are going to drop group file. Define group in tests as a preparatory step.

The patch is generated by

    cd tests/qemu-iotests

    grep '^[0-9]\{3\} ' group | while read line; do
        file=$(awk '{print $1}' <<< "$line");
        groups=$(sed -e 's/^... //' <<< "$line");
        awk "NR==2{print \"# group: $groups\"}1" $file > tmp;
        cat tmp > $file;
    done
---
 tests/qemu-iotests/001 | 1 +
 tests/qemu-iotests/002 | 1 +
 tests/qemu-iotests/003 | 1 +
 tests/qemu-iotests/004 | 1 +
 tests/qemu-iotests/005 | 1 +
 tests/qemu-iotests/007 | 1 +
 tests/qemu-iotests/008 | 1 +
 tests/qemu-iotests/009 | 1 +
 tests/qemu-iotests/010 | 1 +
 tests/qemu-iotests/011 | 1 +
 tests/qemu-iotests/012 | 1 +
 tests/qemu-iotests/013 | 1 +
 tests/qemu-iotests/014 | 1 +
 tests/qemu-iotests/015 | 1 +
 tests/qemu-iotests/017 | 1 +
 tests/qemu-iotests/018 | 1 +
 tests/qemu-iotests/019 | 1 +
 tests/qemu-iotests/020 | 1 +
 tests/qemu-iotests/021 | 1 +
 tests/qemu-iotests/022 | 1 +
 tests/qemu-iotests/023 | 1 +
 tests/qemu-iotests/024 | 1 +
 tests/qemu-iotests/025 | 1 +
 tests/qemu-iotests/026 | 1 +
 tests/qemu-iotests/027 | 1 +
 tests/qemu-iotests/028 | 1 +
 tests/qemu-iotests/029 | 1 +
 tests/qemu-iotests/030 | 1 +
 tests/qemu-iotests/031 | 1 +
 tests/qemu-iotests/032 | 1 +
 tests/qemu-iotests/033 | 1 +
 tests/qemu-iotests/034 | 1 +
 tests/qemu-iotests/035 | 1 +
 tests/qemu-iotests/036 | 1 +
 tests/qemu-iotests/037 | 1 +
 tests/qemu-iotests/038 | 1 +
 tests/qemu-iotests/039 | 1 +
 tests/qemu-iotests/040 | 1 +
 tests/qemu-iotests/041 | 1 +
 tests/qemu-iotests/042 | 1 +
 tests/qemu-iotests/043 | 1 +
 tests/qemu-iotests/044 | 1 +
 tests/qemu-iotests/045 | 1 +
 tests/qemu-iotests/046 | 1 +
 tests/qemu-iotests/047 | 1 +
 tests/qemu-iotests/048 | 1 +
 tests/qemu-iotests/049 | 1 +
 tests/qemu-iotests/050 | 1 +
 tests/qemu-iotests/051 | 1 +
 tests/qemu-iotests/052 | 1 +
 tests/qemu-iotests/053 | 1 +
 tests/qemu-iotests/054 | 1 +
 tests/qemu-iotests/055 | 1 +
 tests/qemu-iotests/056 | 1 +
 tests/qemu-iotests/057 | 1 +
 tests/qemu-iotests/058 | 1 +
 tests/qemu-iotests/059 | 1 +
 tests/qemu-iotests/060 | 1 +
 tests/qemu-iotests/061 | 1 +
 tests/qemu-iotests/062 | 1 +
 tests/qemu-iotests/063 | 1 +
 tests/qemu-iotests/064 | 1 +
 tests/qemu-iotests/065 | 1 +
 tests/qemu-iotests/066 | 1 +
 tests/qemu-iotests/068 | 1 +
 tests/qemu-iotests/069 | 1 +
 tests/qemu-iotests/070 | 1 +
 tests/qemu-iotests/071 | 1 +
 tests/qemu-iotests/072 | 1 +
 tests/qemu-iotests/073 | 1 +
 tests/qemu-iotests/074 | 1 +
 tests/qemu-iotests/075 | 1 +
 tests/qemu-iotests/076 | 1 +
 tests/qemu-iotests/077 | 1 +
 tests/qemu-iotests/078 | 1 +
 tests/qemu-iotests/079 | 1 +
 tests/qemu-iotests/080 | 1 +
 tests/qemu-iotests/081 | 1 +
 tests/qemu-iotests/082 | 1 +
 tests/qemu-iotests/083 | 1 +
 tests/qemu-iotests/084 | 1 +
 tests/qemu-iotests/085 | 1 +
 tests/qemu-iotests/086 | 1 +
 tests/qemu-iotests/087 | 1 +
 tests/qemu-iotests/088 | 1 +
 tests/qemu-iotests/089 | 1 +
 tests/qemu-iotests/090 | 1 +
 tests/qemu-iotests/091 | 1 +
 tests/qemu-iotests/092 | 1 +
 tests/qemu-iotests/093 | 1 +
 tests/qemu-iotests/094 | 1 +
 tests/qemu-iotests/095 | 1 +
 tests/qemu-iotests/096 | 1 +
 tests/qemu-iotests/097 | 1 +
 tests/qemu-iotests/098 | 1 +
 tests/qemu-iotests/099 | 1 +
 tests/qemu-iotests/101 | 1 +
 tests/qemu-iotests/102 | 1 +
 tests/qemu-iotests/103 | 1 +
 tests/qemu-iotests/104 | 1 +
 tests/qemu-iotests/105 | 1 +
 tests/qemu-iotests/106 | 1 +
 tests/qemu-iotests/107 | 1 +
 tests/qemu-iotests/108 | 1 +
 tests/qemu-iotests/109 | 1 +
 tests/qemu-iotests/110 | 1 +
 tests/qemu-iotests/111 | 1 +
 tests/qemu-iotests/112 | 1 +
 tests/qemu-iotests/113 | 1 +
 tests/qemu-iotests/114 | 1 +
 tests/qemu-iotests/115 | 1 +
 tests/qemu-iotests/116 | 1 +
 tests/qemu-iotests/117 | 1 +
 tests/qemu-iotests/118 | 1 +
 tests/qemu-iotests/119 | 1 +
 tests/qemu-iotests/120 | 1 +
 tests/qemu-iotests/121 | 1 +
 tests/qemu-iotests/122 | 1 +
 tests/qemu-iotests/123 | 1 +
 tests/qemu-iotests/124 | 1 +
 tests/qemu-iotests/125 | 1 +
 tests/qemu-iotests/126 | 1 +
 tests/qemu-iotests/127 | 1 +
 tests/qemu-iotests/128 | 1 +
 tests/qemu-iotests/129 | 1 +
 tests/qemu-iotests/130 | 1 +
 tests/qemu-iotests/131 | 1 +
 tests/qemu-iotests/132 | 1 +
 tests/qemu-iotests/133 | 1 +
 tests/qemu-iotests/134 | 1 +
 tests/qemu-iotests/135 | 1 +
 tests/qemu-iotests/136 | 1 +
 tests/qemu-iotests/137 | 1 +
 tests/qemu-iotests/138 | 1 +
 tests/qemu-iotests/139 | 1 +
 tests/qemu-iotests/140 | 1 +
 tests/qemu-iotests/141 | 1 +
 tests/qemu-iotests/143 | 1 +
 tests/qemu-iotests/144 | 1 +
 tests/qemu-iotests/145 | 1 +
 tests/qemu-iotests/146 | 1 +
 tests/qemu-iotests/147 | 1 +
 tests/qemu-iotests/148 | 1 +
 tests/qemu-iotests/149 | 1 +
 tests/qemu-iotests/150 | 1 +
 tests/qemu-iotests/151 | 1 +
 tests/qemu-iotests/152 | 1 +
 tests/qemu-iotests/153 | 1 +
 tests/qemu-iotests/154 | 1 +
 tests/qemu-iotests/155 | 1 +
 tests/qemu-iotests/156 | 1 +
 tests/qemu-iotests/157 | 1 +
 tests/qemu-iotests/158 | 1 +
 tests/qemu-iotests/159 | 1 +
 tests/qemu-iotests/160 | 1 +
 tests/qemu-iotests/161 | 1 +
 tests/qemu-iotests/162 | 1 +
 tests/qemu-iotests/163 | 1 +
 tests/qemu-iotests/165 | 1 +
 tests/qemu-iotests/169 | 1 +
 tests/qemu-iotests/170 | 1 +
 tests/qemu-iotests/171 | 1 +
 tests/qemu-iotests/172 | 1 +
 tests/qemu-iotests/173 | 1 +
 tests/qemu-iotests/174 | 1 +
 tests/qemu-iotests/175 | 1 +
 tests/qemu-iotests/176 | 1 +
 tests/qemu-iotests/177 | 1 +
 tests/qemu-iotests/178 | 1 +
 tests/qemu-iotests/179 | 1 +
 tests/qemu-iotests/181 | 1 +
 tests/qemu-iotests/182 | 1 +
 tests/qemu-iotests/183 | 1 +
 tests/qemu-iotests/184 | 1 +
 tests/qemu-iotests/185 | 1 +
 tests/qemu-iotests/186 | 1 +
 tests/qemu-iotests/187 | 1 +
 tests/qemu-iotests/188 | 1 +
 tests/qemu-iotests/189 | 1 +
 tests/qemu-iotests/190 | 1 +
 tests/qemu-iotests/191 | 1 +
 tests/qemu-iotests/192 | 1 +
 tests/qemu-iotests/194 | 1 +
 tests/qemu-iotests/195 | 1 +
 tests/qemu-iotests/196 | 1 +
 tests/qemu-iotests/197 | 1 +
 tests/qemu-iotests/198 | 1 +
 tests/qemu-iotests/199 | 1 +
 tests/qemu-iotests/200 | 1 +
 tests/qemu-iotests/201 | 1 +
 tests/qemu-iotests/202 | 1 +
 tests/qemu-iotests/203 | 1 +
 tests/qemu-iotests/204 | 1 +
 tests/qemu-iotests/205 | 1 +
 tests/qemu-iotests/206 | 1 +
 tests/qemu-iotests/207 | 1 +
 tests/qemu-iotests/208 | 1 +
 tests/qemu-iotests/209 | 1 +
 tests/qemu-iotests/210 | 1 +
 tests/qemu-iotests/211 | 1 +
 tests/qemu-iotests/212 | 1 +
 tests/qemu-iotests/213 | 1 +
 tests/qemu-iotests/214 | 1 +
 tests/qemu-iotests/215 | 1 +
 tests/qemu-iotests/216 | 1 +
 tests/qemu-iotests/217 | 1 +
 tests/qemu-iotests/218 | 1 +
 tests/qemu-iotests/219 | 1 +
 tests/qemu-iotests/220 | 1 +
 tests/qemu-iotests/221 | 1 +
 tests/qemu-iotests/222 | 1 +
 tests/qemu-iotests/223 | 1 +
 tests/qemu-iotests/224 | 1 +
 tests/qemu-iotests/225 | 1 +
 tests/qemu-iotests/226 | 1 +
 tests/qemu-iotests/227 | 1 +
 tests/qemu-iotests/228 | 1 +
 tests/qemu-iotests/229 | 1 +
 tests/qemu-iotests/231 | 1 +
 tests/qemu-iotests/232 | 1 +
 tests/qemu-iotests/233 | 1 +
 tests/qemu-iotests/234 | 1 +
 tests/qemu-iotests/235 | 1 +
 tests/qemu-iotests/236 | 1 +
 tests/qemu-iotests/237 | 1 +
 tests/qemu-iotests/238 | 1 +
 tests/qemu-iotests/239 | 1 +
 tests/qemu-iotests/240 | 1 +
 tests/qemu-iotests/241 | 1 +
 tests/qemu-iotests/242 | 1 +
 tests/qemu-iotests/243 | 1 +
 tests/qemu-iotests/244 | 1 +
 tests/qemu-iotests/245 | 1 +
 tests/qemu-iotests/246 | 1 +
 tests/qemu-iotests/247 | 1 +
 tests/qemu-iotests/248 | 1 +
 tests/qemu-iotests/249 | 1 +
 tests/qemu-iotests/250 | 1 +
 tests/qemu-iotests/251 | 1 +
 tests/qemu-iotests/252 | 1 +
 tests/qemu-iotests/253 | 1 +
 tests/qemu-iotests/254 | 1 +
 tests/qemu-iotests/255 | 1 +
 tests/qemu-iotests/256 | 1 +
 tests/qemu-iotests/257 | 1 +
 tests/qemu-iotests/258 | 1 +
 tests/qemu-iotests/259 | 1 +
 tests/qemu-iotests/260 | 1 +
 tests/qemu-iotests/261 | 1 +
 tests/qemu-iotests/262 | 1 +
 tests/qemu-iotests/263 | 1 +
 tests/qemu-iotests/264 | 1 +
 tests/qemu-iotests/265 | 1 +
 tests/qemu-iotests/266 | 1 +
 tests/qemu-iotests/267 | 1 +
 tests/qemu-iotests/268 | 1 +
 tests/qemu-iotests/270 | 1 +
 tests/qemu-iotests/271 | 1 +
 tests/qemu-iotests/272 | 1 +
 tests/qemu-iotests/273 | 1 +
 tests/qemu-iotests/274 | 1 +
 tests/qemu-iotests/277 | 1 +
 tests/qemu-iotests/279 | 1 +
 tests/qemu-iotests/280 | 1 +
 tests/qemu-iotests/281 | 1 +
 tests/qemu-iotests/282 | 1 +
 tests/qemu-iotests/283 | 1 +
 tests/qemu-iotests/284 | 1 +
 tests/qemu-iotests/286 | 1 +
 tests/qemu-iotests/287 | 1 +
 tests/qemu-iotests/288 | 1 +
 tests/qemu-iotests/289 | 1 +
 tests/qemu-iotests/290 | 1 +
 tests/qemu-iotests/291 | 1 +
 tests/qemu-iotests/292 | 1 +
 tests/qemu-iotests/293 | 1 +
 tests/qemu-iotests/294 | 1 +
 tests/qemu-iotests/295 | 1 +
 tests/qemu-iotests/296 | 1 +
 tests/qemu-iotests/297 | 1 +
 tests/qemu-iotests/299 | 1 +
 tests/qemu-iotests/300 | 1 +
 tests/qemu-iotests/301 | 1 +
 tests/qemu-iotests/302 | 1 +
 tests/qemu-iotests/303 | 1 +
 tests/qemu-iotests/304 | 1 +
 tests/qemu-iotests/305 | 1 +
 tests/qemu-iotests/307 | 1 +
 tests/qemu-iotests/308 | 1 +
 tests/qemu-iotests/309 | 1 +
 tests/qemu-iotests/312 | 1 +
 291 files changed, 291 insertions(+)
tests/qemu-iotests/001 +1 -0
index 696726e45f..6f980fd34d 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test simple read/write using plain bdrv_pread/bdrv_pwrite
 #
tests/qemu-iotests/002 +1 -0
index 1a0d411df5..5ce1647531 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test simple read/write using plain bdrv_pread/bdrv_pwrite
 #
tests/qemu-iotests/003 +1 -0
index 33eeade0de..03f902a83c 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto
 #
 # Test simple read/write using bdrv_aio_readv/bdrv_aio_writev
 #
tests/qemu-iotests/004 +1 -0
index d308dc4b49..e955579a67 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Make sure we can't read and write outside of the image size.
 #
tests/qemu-iotests/005 +1 -0
index b6d03ac37d..40e64a9a8f 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: img auto quick
 #
 # Make sure qemu-img can create 5TB images
 #
tests/qemu-iotests/007 +1 -0
index 160683adf8..936d3f14fb 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: snapshot auto
 #
 # Check for one possible case of qcow2 refcount corruption.
 #
tests/qemu-iotests/008 +1 -0
index 2b81b119bf..fa4990b513 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test simple asynchronous read/write operations.
 #
tests/qemu-iotests/009 +1 -0
index 4dc7d210f9..efa852bad3 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Nolan I qcow2 corruption - incorrectly reports free clusters
 #
tests/qemu-iotests/010 +1 -0
index df809b3088..4ae9027b47 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Nolan II qcow2 corruption - wrong used cluster
 #
tests/qemu-iotests/011 +1 -0
index 57b99ae4a9..5c99ac987f 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test for AIO allocation on the same cluster
 #
tests/qemu-iotests/012 +1 -0
index 12957285b3..3a24d2ca8d 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: auto quick
 #
 # Make sure we can open read-only images
 #
tests/qemu-iotests/013 +1 -0
index 5cb9032f16..d39d0cd88b 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto
 #
 # qcow2 pattern test, empty and compressed image - 4k cluster patterns
 #
tests/qemu-iotests/014 +1 -0
index e1221c0fff..2d23469332 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw
 #
 # qcow2 pattern test, complex patterns including compression and snapshots
 # Using patterns for 4k cluster size.
tests/qemu-iotests/015 +1 -0
index 4d8effd0ae..40c23235a6 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw snapshot
 #
 # Combined test to grow the refcount table and test snapshots.
 #
tests/qemu-iotests/017 +1 -0
index 3413e34f27..2024b85e79 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw backing auto quick
 #
 # Simple backing file reads
 #
tests/qemu-iotests/018 +1 -0
index 191b461a4d..6fcebbb40e 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw backing auto quick
 #
 # Merge backing file into test image when converting the image
 #
tests/qemu-iotests/019 +1 -0
index d3c11256dc..fa4458fd27 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw backing auto quick
 #
 # When using a backing file for the output image in qemu-img convert,
 # the backing file clusters must not copied. The data must still be
tests/qemu-iotests/020 +1 -0
index 596505be2d..60c672e17b 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw backing auto quick
 #
 # Commit changes to backing file
 #
tests/qemu-iotests/021 +1 -0
index f888269fd4..0fc89df2fe 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: io auto quick
 #
 # Test handling of invalid patterns arguments to qemu-io
 #
tests/qemu-iotests/022 +1 -0
index 99eb08f57f..a116cfe255 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw snapshot auto
 #
 # Test bdrv_load/save_vmstate using the usual patterns
 #
tests/qemu-iotests/023 +1 -0
index 02ed047820..d19d13ff5d 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw
 #
 # qcow2 pattern test with various cluster sizes
 #
tests/qemu-iotests/024 +1 -0
index 12aceb2d41..25a564a150 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw backing auto quick
 #
 # Rebasing COW images
 #
tests/qemu-iotests/025 +1 -0
index 1569d912f4..da77ed3154 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Resizing images
 #
tests/qemu-iotests/026 +1 -0
index 9ecc5880b1..d37e266dad 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw blkdbg
 #
 # qcow2 error path testing
 #
tests/qemu-iotests/027 +1 -0
index 494be0921f..b279c88f33 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test that sub-cluster allocating writes zero the rest of the cluster
 #
tests/qemu-iotests/028 +1 -0
index 57d34aae99..8c391f2adc 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw backing quick
 #
 # Test that backing files can be smaller than the image
 #
tests/qemu-iotests/029 +1 -0
index 61d78c00a4..bd71dd2f22 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # qcow2 internal snapshots/VM state tests
 #
tests/qemu-iotests/030 +1 -0
index dcb4b5d6a6..890784b116 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw auto backing
 #
 # Tests for image streaming.
 #
tests/qemu-iotests/031 +1 -0
index 2bcbc5886e..58b57a0ef2 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test that all qcow2 header extensions survive a header rewrite
 #
tests/qemu-iotests/032 +1 -0
index 8337a4d825..ebbe7cb0ba 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test that AIO requests are drained before an image is closed. This used
 # to segfault because the request coroutine kept running even after the
tests/qemu-iotests/033 +1 -0
index 8b40991d55..da9133c44b 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test aligned and misaligned write zeroes operations.
 #
tests/qemu-iotests/034 +1 -0
index 08f7aea6d5..ac1af8f646 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto backing quick
 #
 # Test bdrv_pwrite_zeroes with backing files (see also 154)
 #
tests/qemu-iotests/035 +1 -0
index d950a0dd1e..0c0c4fdd42 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Let a few AIO requests run in parallel and have them access different L2
 # tables so that the cache has a chance to get used up.
tests/qemu-iotests/036 +1 -0
index 6b82638080..5e567012a8 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test qcow2 feature bits
 #
tests/qemu-iotests/037 +1 -0
index bb893c43dc..85b1015056 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto backing quick
 #
 # Test COW from backing files
 #
tests/qemu-iotests/038 +1 -0
index 30f1f73c25..65bf7a753e 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto backing quick
 #
 # Test COW from backing files with AIO
 #
tests/qemu-iotests/039 +1 -0
index ad3867c3fc..12b2c7fa7b 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test qcow2 lazy refcounts
 #
tests/qemu-iotests/040 +1 -0
index dc6069edc0..7ebc9ed825 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw auto
 #
 # Tests for image block commit.
 #
tests/qemu-iotests/041 +1 -0
index a7780853cd..5cc02b24fc 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw auto backing
 #
 # Tests for image mirroring.
 #
tests/qemu-iotests/042 +1 -0
index e8f23a174c..411e54ae04 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test qemu-img operation on zero size images
 #
tests/qemu-iotests/043 +1 -0
index 3271737f69..f8ce3288db 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto backing
 #
 # Test that qemu-img info --backing-chain detects infinite loops
 #
tests/qemu-iotests/044 +1 -0
index 7e99ea7c68..64b18eb7c8 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw
 #
 # Tests growing a large refcount table.
 #
tests/qemu-iotests/045 +1 -0
index 5acc89099c..45eb239baa 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Tests for fdsets and getfd.
 #
tests/qemu-iotests/046 +1 -0
index ed6fae3529..50b0678f60 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto aio quick
 #
 # Test concurrent cluster allocations
 #
tests/qemu-iotests/047 +1 -0
index 4528465fb0..8dd21e0a81 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Regression test for commit b7ab0fea (which was a corruption fix,
 # despite the commit message claiming otherwise)
tests/qemu-iotests/048 +1 -0
index 2af6b74b41..bf8e4bf528 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: img auto quick
 ##
 ## qemu-img compare test
 ##
tests/qemu-iotests/049 +1 -0
index 82b1e6c202..ed12fa49d7 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto
 #
 # Check qemu-img option parsing
 #
tests/qemu-iotests/050 +1 -0
index 741bdb610e..1de01c312e 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto backing quick
 #
 # Test qemu-img rebase with zero clusters
 #
tests/qemu-iotests/051 +1 -0
index bee26075b2..7cbd1415ce 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw
 #
 # Test command line configuration of block devices and driver-specific options
 #
tests/qemu-iotests/052 +1 -0
index 8d5c10601f..2f23ac9b65 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto backing quick
 #
 # Test bdrv_pread/bdrv_pwrite using BDRV_O_SNAPSHOT
 #
tests/qemu-iotests/053 +1 -0
index 71d299c4f9..9a2958d42d 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test qemu-img convert when image length is not a multiple of cluster size
 #
tests/qemu-iotests/054 +1 -0
index 40922db2b1..ea147012c3 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test huge qcow2 images
 #
tests/qemu-iotests/055 +1 -0
index 4d3744b0d3..5d6b607051 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw
 #
 # Tests for drive-backup and blockdev-backup
 #
tests/qemu-iotests/056 +1 -0
index 052456aa00..0e6b8591e7 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw backing
 #
 # Tests for drive-backup
 #
tests/qemu-iotests/057 +1 -0
index a8b4bb60e0..b0d431999e 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw
 #
 # Tests for internal snapshot.
 #
tests/qemu-iotests/058 +1 -0
index d84740ed9f..ce35ff4ee0 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test export internal snapshot by qemu-nbd, convert it by qemu-img.
 #
tests/qemu-iotests/059 +1 -0
index dcc442be9f..65c0c32b26 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test case for vmdk
 #
tests/qemu-iotests/060 +1 -0
index 4b81d1aa51..db26c6b246 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test case for image corruption (overlapping data structures) in qcow2
 #
tests/qemu-iotests/061 +1 -0
index 5747beb7ed..e26d94a0df 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto
 #
 # Test case for image option amendment in qcow2.
 #
tests/qemu-iotests/062 +1 -0
index f26b88df9d..321252298d 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test case for snapshotting images with unallocated zero clusters in
 # qcow2
tests/qemu-iotests/063 +1 -0
index c750b3806e..3a44758053 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # test of qemu-img convert -n - convert without creation
 #
tests/qemu-iotests/064 +1 -0
index 90673186ec..71fc575b21 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test VHDX read/write from a sample image created with Hyper-V
 #
tests/qemu-iotests/065 +1 -0
index 29a7f7ad60..3c2ca27627 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Test for additional information emitted by qemu-img info on qcow2
 # images
tests/qemu-iotests/066 +1 -0
index a4ac613f8e..a780ed7ab5 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test case for preallocated zero clusters in qcow2
 #
tests/qemu-iotests/068 +1 -0
index ccd1a9f1db..03e03508a6 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test case for loading a saved VM state from a qcow2 image
 #
tests/qemu-iotests/069 +1 -0
index a4da83b2d9..222dcba741 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test case for deleting a backing file
 #
tests/qemu-iotests/070 +1 -0
index cb0f927c16..b181e00f9b 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test VHDX log replay from an image with a journal that needs to be
 # replayed
tests/qemu-iotests/071 +1 -0
index 49faae6684..d99cef5a42 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test case for the QMP blkdebug and blkverify interfaces
 #
tests/qemu-iotests/072 +1 -0
index f0b73e7e65..c492ab8a78 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test case for nested image formats
 #
tests/qemu-iotests/073 +1 -0
index 68517821e8..90afd420bd 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test count_contiguous_clusters in qcow2
 #
tests/qemu-iotests/074 +1 -0
index db03edf0b0..c32c94b50d 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 ##
 ## qemu-img compare test (qcow2 only ones)
 ##
tests/qemu-iotests/075 +1 -0
index 389d5675fa..ca2ed2a05c 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # cloop format input validation tests
 #
tests/qemu-iotests/076 +1 -0
index 0d405ef3f2..1a8927d765 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: io
 #
 # parallels format input validation tests
 #
tests/qemu-iotests/077 +1 -0
index c284952082..fbb90d8036 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test concurrent pread/pwrite
 #
tests/qemu-iotests/078 +1 -0
index 54fc654d8e..0b48b7f137 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # bochs format input validation tests
 #
tests/qemu-iotests/079 +1 -0
index 0f0d94a2ac..793e1f9d08 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto
 #
 # Test qcow2 preallocation with different cluster_sizes
 #
tests/qemu-iotests/080 +1 -0
index bda8617c38..3306500683 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto
 #
 # qcow2 format input validation tests
 #
tests/qemu-iotests/081 +1 -0
index 4e19972931..1ac66f197e 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test Quorum block driver
 #
tests/qemu-iotests/082 +1 -0
index 2135581867..021b9bef06 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test qemu-img command line parsing
 #
tests/qemu-iotests/083 +1 -0
index 10fdfc8ebb..bc32b537b2 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw
 #
 # Test NBD client unexpected disconnect
 #
tests/qemu-iotests/084 +1 -0
index c29d7395e9..e51e91a7c8 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: img quick
 #
 # Test case for VDI header corruption; image too large, and too many blocks.
 # Also simple test for creating dynamic and static VDI images.
tests/qemu-iotests/085 +1 -0
index e99eb44581..d557522943 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw
 #
 # Live snapshot tests
 #
tests/qemu-iotests/086 +1 -0
index fea1a7bd8a..c055e7bfe1 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test qemu-img progress output
 #
tests/qemu-iotests/087 +1 -0
index 678e748c58..edd43f1a28 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test unsupported blockdev-add cases
 #
tests/qemu-iotests/088 +1 -0
index ef1163346c..e3102fe888 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # vpc (VHD) format input validation tests
 #
tests/qemu-iotests/089 +1 -0
index f0929b64c0..48bdc42e42 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test case for support of JSON filenames
 #
tests/qemu-iotests/090 +1 -0
index 87e872ebf4..2044c09e9b 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test for discarding compressed clusters on qcow2 images
 #
tests/qemu-iotests/091 +1 -0
index 8dee168bf6..9d144b9439 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw migration quick
 #
 # Live migration test
 #
tests/qemu-iotests/092 +1 -0
index 40ec62b6f1..bfa116d191 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # qcow1 format input validation tests
 #
tests/qemu-iotests/093 +1 -0
index 32ded11430..7745cb04b6 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: throttle
 #
 # Tests for IO throttling
 #
tests/qemu-iotests/094 +1 -0
index 2d3e1004d3..a295fb20ef 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test case for drive-mirror to NBD
 #
tests/qemu-iotests/095 +1 -0
index 7604ae6966..20b5f9bf61 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test for commit of larger active layer
 #
tests/qemu-iotests/096 +1 -0
index 5915f92786..b5d7636bdc 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Test that snapshots move the throttling configuration to the active
 # layer
tests/qemu-iotests/097 +1 -0
index 1837d4e8e0..30313f8867 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto backing
 #
 # Commit changes into backing chains and empty the top image if the
 # backing image is not explicitly specified
tests/qemu-iotests/098 +1 -0
index a35ce7205e..4c37eb0cf5 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto backing quick
 #
 # Test qcow2's bdrv_make_empty for images without internal snapshots
 #
tests/qemu-iotests/099 +1 -0
index 65e8e92572..2f1199ce04 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test valid filenames for blkdebug and blkverify representatively for
 # other protocols (such as NBD) when queried
tests/qemu-iotests/101 +1 -0
index a4c1b6366a..4c4a8cea11 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test short file I/O
 #
tests/qemu-iotests/102 +1 -0
index 9d747c7bbf..8b4c4c905f 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test case for qemu-io -c map and qemu-img map
 #
tests/qemu-iotests/103 +1 -0
index 220481db4c..726f8313ef 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test case for qcow2 metadata cache size specification
 #
tests/qemu-iotests/104 +1 -0
index c70f28a9a1..3ebb74cf6e 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto
 #
 # Test image creation with aligned and unaligned sizes
 #
tests/qemu-iotests/105 +1 -0
index 4d55a2d3ef..d804685110 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Create, read, write big image
 #
tests/qemu-iotests/106 +1 -0
index 20ad7bd5a2..333144502c 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test preallocated resize of raw images
 #
tests/qemu-iotests/107 +1 -0
index d24829ccf9..e68f1e07c7 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Tests updates of the qcow2 L1 table
 #
tests/qemu-iotests/108 +1 -0
index ba67748bdf..8eaef0b8bf 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test case for repairing qcow2 images which cannot be repaired using
 # the on-disk refcount structures
tests/qemu-iotests/109 +1 -0
index 3ffeaf3c55..e207a555f3 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw
 #
 # Test writing image headers of other formats into raw images
 #
tests/qemu-iotests/110 +1 -0
index f1813d0dfb..1fa36ccdb7 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto backing quick
 #
 # Test case for relative backing file names in complex BDS trees
 #
tests/qemu-iotests/111 +1 -0
index bd839a39f4..3ba25f6161 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test case for non-existing backing file when creating a qcow2 image
 # and not specifying the size
tests/qemu-iotests/112 +1 -0
index 6e413f5651..07ac74fb2c 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw
 #
 # Test cases for different refcount_bits values
 #
tests/qemu-iotests/113 +1 -0
index 71a65de2e7..ee59b9a4b8 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test case for accessing creation options on image formats and
 # protocols not supporting image creation
tests/qemu-iotests/114 +1 -0
index 80e5e5e591..43cb0bc6c3 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test invalid backing file format in qcow2 images
 #
tests/qemu-iotests/115 +1 -0
index 7f53987d1b..26dd37dd6d 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw
 #
 # Test case for non-self-referential qcow2 refcount blocks
 #
tests/qemu-iotests/116 +1 -0
index 941b07a1a9..4f40fcb3d2 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test error code paths for invalid QED images
 #
tests/qemu-iotests/117 +1 -0
index 9039555ac4..48ebc012b1 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto
 #
 # Test case for shared BDS between backend trees
 #
tests/qemu-iotests/118 +1 -0
index 2350929fd8..1a2e219057 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw
 #
 # Test case for the QMP 'change' command and all other associated
 # commands
tests/qemu-iotests/119 +1 -0
index ea6770a484..5770b50045 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # NBD test case for overriding BDRV_O_PROTOCOL by explicitly specifying
 # a driver
tests/qemu-iotests/120 +1 -0
index 45c55c1c01..7187731253 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Non-NBD test cases for overriding BDRV_O_PROTOCOL by explicitly
 # specifying a driver
tests/qemu-iotests/121 +1 -0
index 8357ce089a..ba3d8d9377 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw
 #
 # Test cases for qcow2 refcount table growth
 #
tests/qemu-iotests/122 +1 -0
index 0f3d4ca851..5d550ed13e 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw
 #
 # Test some qemu-img convert cases
 #
tests/qemu-iotests/123 +1 -0
index 01b771c76e..e19111f70d 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test case for qemu-img convert to NBD
 #
tests/qemu-iotests/124 +1 -0
index 3705cbb6b3..3b21bc497f 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw backing
 #
 # Tests for incremental drive-backup
 #
tests/qemu-iotests/125 +1 -0
index 5720e86dce..bd390b3a99 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw
 #
 # Test preallocated growth of qcow2 images
 #
tests/qemu-iotests/126 +1 -0
index dd5a211227..92c0547746 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto backing
 #
 # Tests handling of colons in filenames (which may be confused with protocol
 # prefixes)
tests/qemu-iotests/127 +1 -0
index 77fdfd0205..98e8e82a82 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto backing quick
 #
 # Test case for mirroring with dataplane
 #
tests/qemu-iotests/128 +1 -0
index 3606c41760..d0e00d24b1 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test that opening O_DIRECT succeeds when image file I/O produces EIO
 #
tests/qemu-iotests/129 +1 -0
index 0e13244d85..f57a2e19f6 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Tests that "bdrv_drain_all" doesn't drain block jobs
 #
tests/qemu-iotests/130 +1 -0
index a7b365701c..7257f09677 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test that temporary backing file overrides (on the command line or in
 # blockdev-add) don't replace the original path stored in the image during
tests/qemu-iotests/131 +1 -0
index 27870231cf..d7456cae5b 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # parallels format validation tests (created by QEMU)
 #
tests/qemu-iotests/132 +1 -0
index 39ea43067e..367ea08036 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Test mirror with unmap
 #
tests/qemu-iotests/133 +1 -0
index bc82d8ebd7..d997db1685 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: auto quick
 #
 # Test for reopen
 #
tests/qemu-iotests/134 +1 -0
index 17fe1d6ed4..ded153c0b9 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test encrypted read/write using plain bdrv_pread/bdrv_pwrite
 #
tests/qemu-iotests/135 +1 -0
index 3b3d1dc2a5..299075b4c9 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw
 #
 # Test VPC open of image with large Max Table Entries value.
 #
tests/qemu-iotests/136 +1 -0
index d59400c9fc..8fce88bd67 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw
 #
 # Tests for block device statistics
 #
tests/qemu-iotests/137 +1 -0
index de555a91c9..4680d5df3d 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto
 #
 # Test qcow2 reopen
 #
tests/qemu-iotests/138 +1 -0
index e87a64eb89..951cfa67d4 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # General test case for qcow2's image check
 #
tests/qemu-iotests/139 +1 -0
index 1452fd24b3..e79b3c21fd 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Test cases for the QMP 'blockdev-del' command
 #
tests/qemu-iotests/140 +1 -0
index ff6b904fa0..91e08c30d4 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test case for ejecting a BlockBackend with an NBD server attached to it
 #
tests/qemu-iotests/141 +1 -0
index 21aa0b42d8..115cc1691e 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test case for ejecting BDSs with block jobs still running on them
 #
tests/qemu-iotests/143 +1 -0
index d2349903b1..72151acf27 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: auto quick
 #
 # Test case for connecting to a non-existing NBD export name
 #
tests/qemu-iotests/144 +1 -0
index 4569ac0b4b..60e9ddd75f 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 # Check live snapshot, followed by active commit, and another snapshot.
 #
 # This test is to catch the error case of BZ #1300209:
tests/qemu-iotests/145 +1 -0
index 9427549651..a2ce92516d 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: quick
 #
 # Test the combination of -incoming and snapshot=on
 #
tests/qemu-iotests/146 +1 -0
index ddc3c1fd80..98aca96732 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: quick
 #
 # Test VHD image format creator detection and override
 #
tests/qemu-iotests/147 +1 -0
index d7a9f31089..47dfa62e6b 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: img
 #
 # Test case for NBD's blockdev-add interface
 #
tests/qemu-iotests/148 +1 -0
index 5e14a455b1..7ccbde4633 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Test the rate limit of QMP events
 #
tests/qemu-iotests/149 +1 -0
index 852768f80a..328fd05a4c 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw sudo
 #
 # Copyright (C) 2016 Red Hat, Inc.
 #
tests/qemu-iotests/150 +1 -0
index 3b1f32197a..ac6930ae20 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test that qemu-img convert -S 0 fully allocates the target image
 #
tests/qemu-iotests/151 +1 -0
index f2df72c29c..182f6b5321 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw
 #
 # Tests for active mirroring
 #
tests/qemu-iotests/152 +1 -0
index cc2ea09654..4e179c340f 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Tests for drive-mirror with source size unaligned to granularity
 #
tests/qemu-iotests/153 +1 -0
index 34045ea3cf..607af59091 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test image locking
 #
tests/qemu-iotests/154 +1 -0
index 34a1c051b6..24e29ae2ff 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto backing quick
 #
 # qcow2 specific bdrv_pwrite_zeroes tests with backing files (complements 034)
 #
tests/qemu-iotests/155 +1 -0
index 988f986144..bafef9dd9a 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw
 #
 # Test whether the backing BDSs are correct after completion of a
 # mirror block job; in "existing" modes (drive-mirror with
tests/qemu-iotests/156 +1 -0
index 9c7878dd2d..65dcedd493 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Tests oVirt-like storage migration:
 #  - Create snapshot
tests/qemu-iotests/157 +1 -0
index 7cbac38099..0dc9ba68d2 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: quick
 #
 # Test command line configuration of block devices with qdev
 #
tests/qemu-iotests/158 +1 -0
index cf23742c59..a95878e4ce 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test encrypted read/write using backing files
 #
tests/qemu-iotests/159 +1 -0
index f9690053a2..4eb476d3a8 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # qemu-img dd test with different block sizes
 #
tests/qemu-iotests/160 +1 -0
index 0572b5ae9a..7984a9c6f7 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # qemu-img dd test for the skip option
 #
tests/qemu-iotests/161 +1 -0
index 4fb7d0cbf0..f25effab93 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test reopening a backing image after block-stream and block-commit
 #
tests/qemu-iotests/162 +1 -0
index c0053ed975..cf17f494d8 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: quick
 #
 # Test case for specifying runtime options of the wrong type to some
 # block drivers
tests/qemu-iotests/163 +1 -0
index 5a3cc840a5..dedce8ef43 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw
 #
 # Tests for shrinking images
 #
tests/qemu-iotests/165 +1 -0
index fb56a769b4..abc4ffadd5 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Tests for persistent dirty bitmaps.
 #
tests/qemu-iotests/169 +1 -0
index 40afb15299..a5c7bc83e0 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw migration
 #
 # Tests for dirty bitmaps migration.
 #
tests/qemu-iotests/170 +1 -0
index 6c8f0e8085..41387e4d66 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # qemu-img dd test
 #
tests/qemu-iotests/171 +1 -0
index f3582edb10..d1d77f7013 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test 'offset' and 'size' options of the raw driver. Make sure we can't
 # (or can) read and write outside of the image size.
tests/qemu-iotests/172 +1 -0
index b45782e8db..0ac942a6c5 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: auto
 #
 # Test floppy configuration
 #
tests/qemu-iotests/173 +1 -0
index ec6d1705e5..9594f3c5ea 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw
 #
 # Test QAPI commands looking up protocol based images with relative
 # filename backing strings
tests/qemu-iotests/174 +1 -0
index 1b0dd2e8b7..d4cecb5756 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: auto
 #
 # Test that qemu-io fail with non-zero exit code
 #
tests/qemu-iotests/175 +1 -0
index 21a77a2bf5..f74f053b71 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: quick
 #
 # Test creating raw image preallocation mode
 #
tests/qemu-iotests/176 +1 -0
index 5ce3b27069..27ac25467f 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto backing
 #
 # Commit changes into backing chains and empty the top image if the
 # backing image is not explicitly specified.
tests/qemu-iotests/177 +1 -0
index 595bfd4236..8d8745b29a 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test corner cases with unusual block geometries
 #
tests/qemu-iotests/178 +1 -0
index f09b27caac..3b1a7adce4 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: img
 #
 # qemu-img measure sub-command tests
 #
tests/qemu-iotests/179 +1 -0
index 7ada04c641..09447b5610 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test case for write zeroes with unmap
 #
tests/qemu-iotests/181 +1 -0
index 438c2dcd80..820c53ef35 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto migration quick
 #
 # Test postcopy live migration with shared storage
 #
tests/qemu-iotests/182 +1 -0
index 56a2dd58e6..55a0384c08 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test image locking for POSIX locks
 #
tests/qemu-iotests/183 +1 -0
index d889a3b19c..ee62939e72 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw migration quick
 #
 # Test old-style block migration (migrate -b)
 #
tests/qemu-iotests/184 +1 -0
index eebb53faed..513d167098 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test I/O throttle block filter driver interface
 #
tests/qemu-iotests/185 +1 -0
index fd5e6ebe11..7bc8fe5767 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw
 #
 # Test exiting qemu while jobs are still running
 #
tests/qemu-iotests/186 +1 -0
index 0db25b0e68..072e54e62b 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto
 #
 # Test 'info block' with all kinds of configurations
 #
tests/qemu-iotests/187 +1 -0
index f262d83e3a..70b74b033c 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto
 #
 # Test switching between read-only and read-write
 #
tests/qemu-iotests/188 +1 -0
index 13b225fded..ce087d1873 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test encrypted read/write using plain bdrv_pread/bdrv_pwrite
 #
tests/qemu-iotests/189 +1 -0
index 3e5ded14c6..4e463385b2 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw
 #
 # Test encrypted read/write using backing files
 #
tests/qemu-iotests/190 +1 -0
index c22d8d64f9..7fb8447354 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # qemu-img measure sub-command tests on huge qcow2 files
 #
tests/qemu-iotests/191 +1 -0
index 95a891350d..ce695b95c2 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto
 #
 # Test commit block job where top has two parents
 #
tests/qemu-iotests/192 +1 -0
index d2ba55dd90..d809187fca 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test NBD export with -incoming (non-shared storage migration use case from
 # libvirt)
tests/qemu-iotests/194 +1 -0
index 7a4863ab18..3889266afa 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw migration quick
 #
 # Copyright (C) 2017 Red Hat, Inc.
 #
tests/qemu-iotests/195 +1 -0
index 967af5b7b5..f1df69079f 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test change-backing-file command
 #
tests/qemu-iotests/196 +1 -0
index e8fcf37273..2451515094 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick migration
 #
 # Test clearing unknown autoclear_features flag by qcow2 after
 # migration. This test mimics migration to older qemu.
tests/qemu-iotests/197 +1 -0
index a161c89816..a2547bc280 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test case for copy-on-read into qcow2
 #
tests/qemu-iotests/198 +1 -0
index 46f0c54537..b333a8f281 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw
 #
 # Test commit of encrypted qcow2 files
 #
tests/qemu-iotests/199 +1 -0
index 58fad872a1..dbf10e58d3 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw migration
 #
 # Tests for dirty bitmaps postcopy migration.
 #
tests/qemu-iotests/200 +1 -0
index 046539154f..f80517e342 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw
 #
 # Block job co-routine race condition test.
 #
tests/qemu-iotests/201 +1 -0
index 483eb189c5..1b8eb51d8f 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw migration quick
 #
 # Test savevm and loadvm after live migration with postcopy flag
 #
tests/qemu-iotests/202 +1 -0
index e3900a44d1..8eb5f32d15 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Copyright (C) 2017 Red Hat, Inc.
 #
tests/qemu-iotests/203 +1 -0
index 4b4bd3307d..ea30e50497 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw auto migration quick
 #
 # Copyright (C) 2017 Red Hat, Inc.
 #
tests/qemu-iotests/204 +1 -0
index 536bb8b534..ab68b6d75c 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test corner cases with unusual block geometries
 #
tests/qemu-iotests/205 +1 -0
index 43432cb599..c0e107328f 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Tests for qmp command nbd-server-remove.
 #
tests/qemu-iotests/206 +1 -0
index d12d7cb566..c3cdad4ce4 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw
 #
 # Test qcow2 and file image creation
 #
tests/qemu-iotests/207 +1 -0
index a6621410da..f9f3fd7131 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw
 #
 # Test ssh image creation
 #
tests/qemu-iotests/208 +1 -0
index 54aa4be273..6117f165fa 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Copyright (C) 2018 Red Hat, Inc.
 #
tests/qemu-iotests/209 +1 -0
index 8c804f4a30..ff7efea11b 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Tests for NBD BLOCK_STATUS extension
 #
tests/qemu-iotests/210 +1 -0
index 7bf591f313..5a62ed4dd1 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw
 #
 # Test luks and file image creation
 #
tests/qemu-iotests/211 +1 -0
index 4969edb00c..f52cadade1 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Test VDI and file image creation
 #
tests/qemu-iotests/212 +1 -0
index 45d08842bb..d4af0c4ac8 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Test parallels and file image creation
 #
tests/qemu-iotests/213 +1 -0
index cf638eb927..78d839ab64 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Test vhdx and file image creation
 #
tests/qemu-iotests/214 +1 -0
index 75ae7a14b5..0889089d81 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto
 #
 # Test qcow2 image compression
 #
tests/qemu-iotests/215 +1 -0
index f99bae78c7..d464596f14 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test case for copy-on-read into qcow2, using the COR filter driver
 #
tests/qemu-iotests/216 +1 -0
index f93c61aad6..c02f8d2880 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Copy-on-read tests using a COR filter node
 #
tests/qemu-iotests/217 +1 -0
index 7385342498..e693f326a3 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # I/O errors when working with internal qcow2 snapshots, and repairing
 # the result
tests/qemu-iotests/218 +1 -0
index 5586870456..ae7c4fb187 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # This test covers what happens when a mirror block job is cancelled
 # in various phases of its existence.
tests/qemu-iotests/219 +1 -0
index db272c5249..16c3ca7fff 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw
 #
 # Copyright (C) 2018 Red Hat, Inc.
 #
tests/qemu-iotests/220 +1 -0
index 9ba3b3fdcb..7d08b9b040 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto
 #
 # max limits on compression in huge qcow2 files
 #
tests/qemu-iotests/221 +1 -0
index 7e6086b205..c463fd4b11 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test qemu-img vs. unaligned images
 # (See also 253, which is the O_DIRECT version)
tests/qemu-iotests/222 +1 -0
index 14d67c875b..b48afe623e 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # This test covers the basic fleecing workflow, which provides a
 # point-in-time snapshot of a node that can be queried over NBD.
tests/qemu-iotests/223 +1 -0
index d68bc3cb6f..da87f2f4a2 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test reading dirty bitmap over NBD
 #
tests/qemu-iotests/224 +1 -0
index 017b0685ba..38dd153625 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Test json:{} filenames with qemu-internal BDSs
 # (the one of commit, to be precise)
tests/qemu-iotests/225 +1 -0
index 0186ec8156..c0053790db 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test vmdk backing file correlation
 #
tests/qemu-iotests/226 +1 -0
index c1e1fb2b1c..6a9adb4a0b 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: auto quick
 #
 # This test covers expected filetypes for the file, host_cdrom and
 # host_device drivers.
tests/qemu-iotests/227 +1 -0
index 637d7c3726..7e45a47ac6 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: quick
 #
 # Test query-blockstats with different ways to create a BB
 #
tests/qemu-iotests/228 +1 -0
index 266fce6cda..a5eda2e149 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Test for when a backing file is considered overridden (thus, a
 # json:{} filename is generated for the overlay) and when it is not
tests/qemu-iotests/229 +1 -0
index 273ac2472d..4bc99390b5 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: auto quick
 #
 # Test for force canceling a running blockjob that is paused in
 # an error state.
tests/qemu-iotests/231 +1 -0
index c0b053ac30..0f66d0ca36 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: quick
 #
 # Test legacy and modern option parsing for rbd/ceph.  This will not
 # actually connect to a ceph server, but rather looks for the appropriate
tests/qemu-iotests/232 +1 -0
index 685356ac3b..b30faaa218 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: quick
 #
 # Test for auto-read-only
 #
tests/qemu-iotests/233 +1 -0
index a5c17c3963..7ce5764903 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: quick
 #
 # Test NBD TLS certificate / authorization integration
 #
tests/qemu-iotests/234 +1 -0
index 73c899ddae..cb5f1753e0 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: quick migration
 #
 # Copyright (C) 2018 Red Hat, Inc.
 #
tests/qemu-iotests/235 +1 -0
index d1b10ac36b..20d16dbf38 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: quick
 #
 # Simple mirror test
 #
tests/qemu-iotests/236 +1 -0
index 6f5cee2444..f6c44517d6 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: quick
 #
 # Test bitmap merges.
 #
tests/qemu-iotests/237 +1 -0
index 5b21ad3509..43dfd3bd40 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Test vmdk and file image creation
 #
tests/qemu-iotests/238 +1 -0
index b8fcf15a1f..8a10af57f1 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: quick
 #
 # Regression test for throttle group member unregister segfault with iothread
 #
tests/qemu-iotests/239 +1 -0
index b0991ffe59..4f0037148d 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test case for dmg
 #
tests/qemu-iotests/240 +1 -0
index c0f71f0461..ab077f4ceb 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: quick
 
 # Test hot plugging and unplugging with iothreads
 #
tests/qemu-iotests/241 +1 -0
index 8dae8d39e4..c962c8b607 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test qemu-nbd vs. unaligned images
 #
tests/qemu-iotests/242 +1 -0
index a16de3085f..a9b27668c2 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Test for qcow2 bitmap printed information
 #
tests/qemu-iotests/243 +1 -0
index 17388a4644..8bbb510120 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test qcow2 preallocation
 #
tests/qemu-iotests/244 +1 -0
index f2b2dddf1c..a46b441627 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test qcow2 with external data files
 #
tests/qemu-iotests/245 +1 -0
index e60c8326d3..86f00f290f 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw
 #
 # Test cases for the QMP 'x-blockdev-reopen' command
 #
tests/qemu-iotests/246 +1 -0
index 58a479cc1f..fa3102c546 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Test persistent bitmap resizing.
 #
tests/qemu-iotests/247 +1 -0
index 6cf2679750..ace6dba052 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test for auto-read-only with commit block job
 #
tests/qemu-iotests/248 +1 -0
index 18ba03467e..4daaed1530 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Test resume mirror after auto pause on ENOSPC
 #
tests/qemu-iotests/249 +1 -0
index 29453b8c90..28bffd4d57 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test that a backing image is put back in read-only mode after
 # block-commit (both when it fails and when it succeeds).
tests/qemu-iotests/250 +1 -0
index 3df275c76b..f069ca1759 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test big discard in qcow2 shrink
 #
tests/qemu-iotests/251 +1 -0
index 294773bdc1..8bdec37d32 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test qemu-img convert --salvage
 #
tests/qemu-iotests/252 +1 -0
index 1d74afff99..2134b9993a 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto backing quick
 #
 # Tests for rebasing COW images that require zero cluster support
 #
tests/qemu-iotests/253 +1 -0
index d88d5afa45..35039d20a8 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test qemu-img vs. unaligned images; O_DIRECT version
 # (Originates from 221)
tests/qemu-iotests/254 +1 -0
index 150e58be8e..49da948407 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw backing quick
 #
 # Test external snapshot with bitmap copying and moving.
 #
tests/qemu-iotests/255 +1 -0
index 8f08f741da..c43aa9c67a 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Test commit job graph modifications while requests are active
 #
tests/qemu-iotests/256 +1 -0
index db8d6f31cf..8d82a1dd86 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw auto quick
 #
 # Test incremental/backup across iothread contexts
 #
tests/qemu-iotests/257 +1 -0
index c80e06ae28..a2f4b5afe6 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw
 #
 # Test bitmap-sync backups (incremental, differential, and partials)
 #
tests/qemu-iotests/258 +1 -0
index e305a1502f..9a2d33ae5e 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Very specific tests for adjacent commit/stream block jobs
 #
tests/qemu-iotests/259 +1 -0
index 62e29af05f..76cde429c4 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test generic image creation fallback (by using NBD)
 #
tests/qemu-iotests/260 +1 -0
index 804a7addb9..a35cb7b61f 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Tests for temporary external snapshot when we have bitmaps.
 #
tests/qemu-iotests/261 +1 -0
index 847b4c6a37..d1c8037ab1 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw
 #
 # Test case for qcow2's handling of extra data in snapshot table entries
 # (and more generally, how certain cases of broken snapshot tables are
tests/qemu-iotests/262 +1 -0
index 03af061f94..32d69988ef 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick migration
 #
 # Copyright (C) 2019 Red Hat, Inc.
 #
tests/qemu-iotests/263 +1 -0
index f598a12899..ec09b41405 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test encrypted write that crosses cluster boundary of two unallocated clusters
 # Based on 188
tests/qemu-iotests/264 +1 -0
index 666f164ed8..960f0449a4 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw
 #
 # Test nbd reconnect
 #
tests/qemu-iotests/265 +1 -0
index 0e800fb524..4b3b52c6e6 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test reverse-ordered qcow2 writes on a sub-cluster level
 #
tests/qemu-iotests/266 +1 -0
index 91bdf8729e..71ce81d0df 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Test VPC and file image creation
 #
tests/qemu-iotests/267 +1 -0
index e44be49c77..2e2afdad9c 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick snapshot
 #
 # Test which nodes are involved in internal snapshots
 #
tests/qemu-iotests/268 +1 -0
index ddf4312284..9a8a563760 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test write request with required alignment larger than the cluster size
 #
tests/qemu-iotests/270 +1 -0
index 00339c0f78..74352342db 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw backing quick
 #
 # Test large write to a qcow2 image
 #
tests/qemu-iotests/271 +1 -0
index e242b28b58..599b849cc6 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto
 #
 # Test qcow2 images with extended L2 entries
 #
tests/qemu-iotests/272 +1 -0
index de475bf6f0..4bcf410d81 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw
 #
 # Test compressed write to a qcow2 image at an offset above 4 GB
 #
tests/qemu-iotests/273 +1 -0
index 79b4ab4b05..6c33305b4d 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: backing quick
 #
 # Test multiple blockdev-snapshot calls with 'backing': null
 #
tests/qemu-iotests/274 +1 -0
index 76b1ba6a52..caab008e07 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw backing
 #
 # Copyright (C) 2019 Red Hat, Inc.
 #
tests/qemu-iotests/277 +1 -0
index a39ce2d873..24833e7eb6 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Test NBD client reconnection
 #
tests/qemu-iotests/279 +1 -0
index 2a6315cf17..6afef78bc8 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw backing quick
 #
 # Test qemu-img --backing-chain --image-opts
 #
tests/qemu-iotests/280 +1 -0
index f594bb9ed2..628f3c33ca 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw migration quick
 #
 # Copyright (C) 2019 Red Hat, Inc.
 #
tests/qemu-iotests/281 +1 -0
index 0bf973bca6..956698083f 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Test cases for blockdev + IOThread interactions
 #
tests/qemu-iotests/282 +1 -0
index 27da2a0023..3140445989 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw img quick
 #
 # Test qemu-img file cleanup for LUKS when using a non-UTF8 secret
 #
tests/qemu-iotests/283 +1 -0
index 383797ed62..79643e375b 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: auto quick
 #
 # Test for backup-top filter permission activation failure
 #
tests/qemu-iotests/284 +1 -0
index 9f6c29a79c..5a82639e7f 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw
 #
 # Test ref count checks on encrypted images
 #
tests/qemu-iotests/286 +1 -0
index f64e0eccea..120a8375b7 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test qemu-img snapshot -l
 #
tests/qemu-iotests/287 +1 -0
index 3bb383fd4b..22ce9ff0e4 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: auto quick
 #
 # Test case for an image using zstd compression
 #
tests/qemu-iotests/288 +1 -0
index 6c62065aef..47aca6592a 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: quick
 #
 # qemu-img measure tests for LUKS images
 #
tests/qemu-iotests/289 +1 -0
index fe69bde1eb..5dd6ec62db 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # qcow2 v3-exclusive error path testing
 # (026 tests paths common to v2 and v3)
tests/qemu-iotests/290 +1 -0
index 35c38d4f80..ed80da2685 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test how 'qemu-io -c discard' behaves on v2 and v3 qcow2 images
 #
tests/qemu-iotests/291 +1 -0
index bc70d5e389..20efb080a6 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test qemu-img bitmap handling
 #
tests/qemu-iotests/292 +1 -0
index 3ae2772e3b..73cbb9364a 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test resizing a qcow2 image with a backing file
 #
tests/qemu-iotests/293 +1 -0
index 3363bf07f0..37294487b1 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw
 #
 # Test encryption key management with luks
 # Based on 134
tests/qemu-iotests/294 +1 -0
index 4c375ed609..9059eb26b3 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Copyright (C) 2019 Red Hat, Inc.
 #
tests/qemu-iotests/295 +1 -0
index 59e674fa85..01a6c0b31f 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw
 #
 # Test case QMP's encrypted key management
 #
tests/qemu-iotests/296 +1 -0
index fb7dec88aa..0bc3c6c7d7 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw
 #
 # Test case for encryption key management versus image sharing
 #
tests/qemu-iotests/297 +1 -0
index 5c5420712b..85bc1c0c85 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: meta
 #
 # Copyright (C) 2020 Red Hat, Inc.
 #
tests/qemu-iotests/299 +1 -0
index e129c7f7cb..a7122941fd 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: auto quick
 #
 # Test shutdown when bitmap is exported through NBD server
 #
tests/qemu-iotests/300 +1 -0
index 5b75121b84..23aca59d9c 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: migration
 #
 # Copyright (C) 2020 Red Hat, Inc.
 #
tests/qemu-iotests/301 +1 -0
index 3823e95617..9f943cadbe 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: backing quick
 #
 # Test qcow backing file warnings
 #
tests/qemu-iotests/302 +1 -0
index a8506bda15..5695af4914 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: quick
 #
 # Tests converting qcow2 compressed to NBD
 #
tests/qemu-iotests/303 +1 -0
index 11cd9eeb26..425544c064 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Test for dumping of qcow2 image metadata
 #
tests/qemu-iotests/304 +1 -0
index aaf9e14617..198f282087 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick
 #
 # Tests dirty-bitmap backup with unaligned bitmap granularity
 #
tests/qemu-iotests/305 +1 -0
index 5a415eb2a4..8b26156923 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test the handling of errors in write requests with multiple allocations
 #
tests/qemu-iotests/307 +1 -0
index de7c25fcfc..9008974346 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# group: rw quick export
 #
 # Copyright (C) 2020 Red Hat, Inc.
 #
tests/qemu-iotests/308 +1 -0
index b30f4400f6..f122065d0f 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw
 #
 # Test FUSE exports (in ways that are not captured by the generic
 # tests)
tests/qemu-iotests/309 +1 -0
index fb61157c2e..b90b279994 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw auto quick
 #
 # Test qemu-nbd -A
 #
tests/qemu-iotests/312 +1 -0
index 41340494b0..4139745f0e 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# group: rw quick
 #
 # Test drive-mirror with quorum
 #
Signature · 1 line
2.29.2
#7 lore raw

PATCH v7 7/11 iotests: add findtests.py

In reply to Vladimir Sementsov-Ogievskiy

Add python script with new logic of searching for tests:

Current ./check behavior:
- tests are named [0-9][0-9][0-9]
- tests must be registered in group file (even if test doesn't belong to any group, like 142)

Behavior of findtests.py:
- group file is dropped
- tests are all files in tests/ subdirectory (except for .out files), so it's not needed more to "register the test", just create it with appropriate name in tests/ subdirectory. Old names like [0-9][0-9][0-9] (in root iotests directory) are supported too, but not recommended for new tests
- groups are parsed from '# group: ' line inside test files
- optional file group.local may be used to define some additional groups for downstreams
- 'disabled' group is used to temporary disable tests. So instead of commenting tests in old 'group' file you now can add them to disabled group with help of 'group.local' file
- selecting test ranges like 5-15 are not supported more (to support restarting failed ./check command from the middle of the process, new argument is added: --start-from)

Benefits:
- no rebase conflicts in group file on patch porting from branch to branch
- no conflicts in upstream, when different series want to occupy same test number
- meaningful names for test files
For example, with digital number, when some person wants to add some test about block-stream, he most probably will just create a new test. But if there would be test-block-stream test already, he will at first look at it and may be just add a test-case into it. And anyway meaningful names are better.

This commit don't update check behavior (which will be done in further commit), still, the documentation changed like new behavior is already here. Let's live with this small inconsistency for the following few commits, until final change.

The file findtests.py is self-executable and may be used for debugging purposes.

---
 docs/devel/testing.rst          |  50 +++++++++-
 tests/qemu-iotests/findtests.py | 159 ++++++++++++++++++++++++++++++++
 2 files changed, 208 insertions(+), 1 deletion(-)
 create mode 100644 tests/qemu-iotests/findtests.py
docs/devel/testing.rst +49 -1
index 0aa7a13bba..454b706141 100644
@@ -111,7 +111,7 @@ check-block
 -----------
 
 ``make check-block`` runs a subset of the block layer iotests (the tests that
-are in the "auto" group in ``tests/qemu-iotests/group``).
+are in the "auto" group).
 See the "QEMU iotests" section below for more information.
 
 GCC gcov support
@@ -224,6 +224,54 @@ another application on the host may have locked the file, possibly leading to a
 test failure.  If using such devices are explicitly desired, consider adding
 ``locking=off`` option to disable image locking.
 
+Test case groups
+----------------
+
+"Tests may belong to one or more test groups, which are defined in the form
+of a comment in the test source file. By convention, test groups are listed
+in the second line of the test file, after the "#!/..." line, like this:
+
+.. code::
+
+  #!/usr/bin/env python3
+  # group: auto quick
+  #
+  ...
+
+Another way of defining groups is creating the tests/qemu-iotests/group.local
+file. This should be used only for downstream (this file should never appear
+in upstream). This file may be used for defining some downstream test groups
+or for temporarily disabling tests, like this:
+
+.. code::
+
+  # groups for some company downstream process
+  #
+  # ci - tests to run on build
+  # down - our downstream tests, not for upstream
+  #
+  # Format of each line is:
+  # TEST_NAME TEST_GROUP [TEST_GROUP ]...
+
+  013 ci
+  210 disabled
+  215 disabled
+  our-ugly-workaround-test down ci
+
+Note that the following group names have a special meaning:
+
+- quick: Tests in this group should finish within a few seconds.
+
+- auto: Tests in this group are used during "make check" and should be
+  runnable in any case. That means they should run with every QEMU binary
+  (also non-x86), with every QEMU configuration (i.e. must not fail if
+  an optional feature is not compiled in - but reporting a "skip" is ok),
+  work at least with the qcow2 file format, work with all kind of host
+  filesystems and users (e.g. "nobody" or "root") and must not take too
+  much memory and disk space (since CI pipelines tend to fail otherwise).
+
+- disabled: Tests in this group are disabled and ignored by check.
+
 .. _docker-ref:
 
 Docker based tests
tests/qemu-iotests/findtests.py new file +159 -0

159 lines not on this page · raw message

Signature · 1 line
2.29.2
Reviewed-by on v7 7/11
#8 lore raw

Re: [PATCH v7 07/11] iotests: add findtests.py

Quoted 36 lines · Vladimir Sementsov-Ogievskiy

Add python script with new logic of searching for tests:

Current ./check behavior:
- tests are named [0-9][0-9][0-9]
- tests must be registered in group file (even if test doesn't belong to any group, like 142)

Behavior of findtests.py:
- group file is dropped
- tests are all files in tests/ subdirectory (except for .out files), so it's not needed more to "register the test", just create it with appropriate name in tests/ subdirectory. Old names like [0-9][0-9][0-9] (in root iotests directory) are supported too, but not recommended for new tests
- groups are parsed from '# group: ' line inside test files
- optional file group.local may be used to define some additional groups for downstreams
- 'disabled' group is used to temporary disable tests. So instead of commenting tests in old 'group' file you now can add them to disabled group with help of 'group.local' file
- selecting test ranges like 5-15 are not supported more (to support restarting failed ./check command from the middle of the process, new argument is added: --start-from)

Benefits:
- no rebase conflicts in group file on patch porting from branch to branch
- no conflicts in upstream, when different series want to occupy same test number
- meaningful names for test files
For example, with digital number, when some person wants to add some test about block-stream, he most probably will just create a new test. But if there would be test-block-stream test already, he will at first look at it and may be just add a test-case into it. And anyway meaningful names are better.

Quoted 1 line

This commit don't update check behavior (which will be done in further

doesn't

Quoted 17 lines

commit), still, the documentation changed like new behavior is already here. Let's live with this small inconsistency for the following few commits, until final change.

The file findtests.py is self-executable and may be used for debugging purposes.

---
 docs/devel/testing.rst          |  50 +++++++++-
 tests/qemu-iotests/findtests.py | 159 ++++++++++++++++++++++++++++++++
 2 files changed, 208 insertions(+), 1 deletion(-)
 create mode 100644 tests/qemu-iotests/findtests.py

+++ b/tests/qemu-iotests/findtests.py

tests/qemu-iotests/findtests.py line 49

Quoted 9 lines
tests/qemu-iotests/findtests.py new file +9 -0
+class TestFinder:
+    def __init__(self, test_dir: Optional[str] = None) -> None:
+        self.groups = defaultdict(set)
+
+        with chdir(test_dir):
+            self.all_tests = glob.glob('[0-9][0-9][0-9]')
+            self.all_tests += [f for f in glob.iglob('tests/*')
+                               if not f.endswith('.out') and
+                               os.path.isfile(f + '.out')]

Interesting that 'NNN' is a test even if 'NNN.out' is not present, but 'tests/NNN' is not. Not sure if it is worth tweaking, though.

tests/qemu-iotests/findtests.py line 76

Quoted 3 lines
tests/qemu-iotests/findtests.py new file +3 -0
+    def parse_test_name(self, name: str) -> str:
+        if '/' in name:
+            raise ValueError('Paths are unsupported for test selecting, '

selection

tests/qemu-iotests/findtests.py line 80

Quoted 4 lines
tests/qemu-iotests/findtests.py new file +4 -0
+                             f'requiring "{name}" is wrong')
+
+        if re.fullmatch(r'\d+', name):
+            # Numbered tests are old naming convetion. We should convert them

convention

Quoted 5 lines
tests/qemu-iotests/findtests.py new file +5 -0
+            # to three-digit-length, like 1 --> 001.
+            name = f'{int(name):03}'
+        else:
+            # Named tests all should be in tests/ subdirectory
+            name = os.path.join('tests', name)

tests/qemu-iotests/findtests.py line 95

Quoted 10 lines
tests/qemu-iotests/findtests.py new file +10 -0
+
+        if name not in self.all_tests:
+            raise ValueError(f'Test "{name}" is not found')
+
+        return name
+
+    def find_tests(self, groups: Optional[List[str]] = None,
+                   exclude_groups: Optional[List[str]] = None,
+                   tests: Optional[List[str]] = None,
+                   start_from: Optional[str] = None) -> List[str]:
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#9 lore raw

Re: [PATCH v7 07/11] iotests: add findtests.py

In reply to Vladimir Sementsov-Ogievskiy
Quoted 15 lines · Vladimir Sementsov-Ogievskiy

Add python script with new logic of searching for tests:

The file findtests.py is self-executable and may be used for debugging purposes.

---

tests/qemu-iotests/findtests.py new file +0 -0
new file mode 100644
index 0000000000..065ec13125

tests/qemu-iotests/findtests.py line 1

Quoted 1 line
tests/qemu-iotests/findtests.py new file +1 -0
@@ -0,0 +1,159 @@
+# TestFinder class, define set of tests to run.

Per the commit message, the file is self-executable; doesn't that mean it should have 755 permissions instead of 644, and have a #! python line?

tests/qemu-iotests/findtests.py line 159

Quoted 2 lines
tests/qemu-iotests/findtests.py new file +1 -0
+        return sequence

I guess it would also need a __main__ blurb at the end? Or maybe I'm misunderstanding the intent of the commit message line.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#10 lore raw

Re: [PATCH v7 07/11] iotests: add findtests.py

Quoted 21 lines · Eric Blake
Quoted 15 lines · Vladimir Sementsov-Ogievskiy

Add python script with new logic of searching for tests:

The file findtests.py is self-executable and may be used for debugging purposes.

---

tests/qemu-iotests/findtests.py new file +0 -0
new file mode 100644
index 0000000000..065ec13125

tests/qemu-iotests/findtests.py line 1

Quoted 1 line
tests/qemu-iotests/findtests.py new file +1 -0
@@ -0,0 +1,159 @@
+# TestFinder class, define set of tests to run.

Per the commit message, the file is self-executable; doesn't that mean it should have 755 permissions instead of 644, and have a #! python line?

Quoted 6 lines

tests/qemu-iotests/findtests.py line 159

Quoted 2 lines
tests/qemu-iotests/findtests.py new file +1 -0
+        return sequence

I guess it would also need a __main__ blurb at the end? Or maybe I'm misunderstanding the intent of the commit message line.

These things are dropped recently and I forget to updated commit-msg.

So, what should be done is just drop this sentence from commit message.

Signature · 2 lines
Best regards,
Vladimir
#11 lore raw

Re: [PATCH v7 07/11] iotests: add findtests.py

In reply to Vladimir Sementsov-Ogievskiy

Am 16.01.2021 um 14:44 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 36 lines

Add python script with new logic of searching for tests:

Current ./check behavior:
- tests are named [0-9][0-9][0-9]
- tests must be registered in group file (even if test doesn't belong to any group, like 142)

Behavior of findtests.py:
- group file is dropped
- tests are all files in tests/ subdirectory (except for .out files), so it's not needed more to "register the test", just create it with appropriate name in tests/ subdirectory. Old names like [0-9][0-9][0-9] (in root iotests directory) are supported too, but not recommended for new tests
- groups are parsed from '# group: ' line inside test files
- optional file group.local may be used to define some additional groups for downstreams
- 'disabled' group is used to temporary disable tests. So instead of commenting tests in old 'group' file you now can add them to disabled group with help of 'group.local' file
- selecting test ranges like 5-15 are not supported more (to support restarting failed ./check command from the middle of the process, new argument is added: --start-from)

Benefits:
- no rebase conflicts in group file on patch porting from branch to branch
- no conflicts in upstream, when different series want to occupy same test number
- meaningful names for test files
For example, with digital number, when some person wants to add some test about block-stream, he most probably will just create a new test. But if there would be test-block-stream test already, he will at first look at it and may be just add a test-case into it. And anyway meaningful names are better.

Quoted 7 lines

This commit don't update check behavior (which will be done in further commit), still, the documentation changed like new behavior is already here. Let's live with this small inconsistency for the following few commits, until final change.

The file findtests.py is self-executable and may be used for debugging purposes.

As Eric mentioned, this isn't accurate any more.

You mentioned using it as a way to debug things. I assume this is now covered by the dry run option?

Quoted 8 lines
---
 docs/devel/testing.rst          |  50 +++++++++-
 tests/qemu-iotests/findtests.py | 159 ++++++++++++++++++++++++++++++++
 2 files changed, 208 insertions(+), 1 deletion(-)
 create mode 100644 tests/qemu-iotests/findtests.py
tests/qemu-iotests/findtests.py new file +1 -0
+    def add_group_file(self, fname: str) -> None:

tests/qemu-iotests/findtests.py line 69

Quoted 10 lines
tests/qemu-iotests/findtests.py new file +10 -0
+        with open(fname) as f:
+            for line in f:
+                line = line.strip()
+
+                if (not line) or line[0] == '#':
+                    continue
+
+                words = line.split()
+                test_file = self.parse_test_name(words[0])
+                groups = words[1:]

The previous version still had this:

+ if test_file not in self.all_tests:
+ print(f'Warning: {fname}: "{test_file}" test is not found.'
+ ' Skip.')
+ continue

Why did you remove it? I found this useful when I had a wrong test name in my group.local. Now it's silently ignored.

tests/qemu-iotests/findtests.py line 72

Quoted 2 lines
tests/qemu-iotests/findtests.py new file +2 -0
+                for g in groups:
+                    self.groups[g].add(test_file)

Kevin

#12 lore raw

Re: [PATCH v7 07/11] iotests: add findtests.py

Quoted 42 lines · Kevin Wolf

Am 16.01.2021 um 14:44 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 41 lines

Add python script with new logic of searching for tests:

Current ./check behavior:
- tests are named [0-9][0-9][0-9]
- tests must be registered in group file (even if test doesn't belong to any group, like 142)

Behavior of findtests.py:
- group file is dropped
- tests are all files in tests/ subdirectory (except for .out files), so it's not needed more to "register the test", just create it with appropriate name in tests/ subdirectory. Old names like [0-9][0-9][0-9] (in root iotests directory) are supported too, but not recommended for new tests
- groups are parsed from '# group: ' line inside test files
- optional file group.local may be used to define some additional groups for downstreams
- 'disabled' group is used to temporary disable tests. So instead of commenting tests in old 'group' file you now can add them to disabled group with help of 'group.local' file
- selecting test ranges like 5-15 are not supported more (to support restarting failed ./check command from the middle of the process, new argument is added: --start-from)

Benefits:
- no rebase conflicts in group file on patch porting from branch to branch
- no conflicts in upstream, when different series want to occupy same test number
- meaningful names for test files
For example, with digital number, when some person wants to add some test about block-stream, he most probably will just create a new test. But if there would be test-block-stream test already, he will at first look at it and may be just add a test-case into it. And anyway meaningful names are better.

This commit don't update check behavior (which will be done in further commit), still, the documentation changed like new behavior is already here. Let's live with this small inconsistency for the following few commits, until final change.

Quoted 7 lines
Quoted 2 lines

The file findtests.py is self-executable and may be used for debugging purposes.

As Eric mentioned, this isn't accurate any more.

You mentioned using it as a way to debug things. I assume this is now covered by the dry run option?

yes

Quoted 27 lines
Quoted 9 lines
---
  docs/devel/testing.rst          |  50 +++++++++-
  tests/qemu-iotests/findtests.py | 159 ++++++++++++++++++++++++++++++++
  2 files changed, 208 insertions(+), 1 deletion(-)

create mode 100644 tests/qemu-iotests/findtests.py

tests/qemu-iotests/findtests.py new file +1 -0
+    def add_group_file(self, fname: str) -> None:

tests/qemu-iotests/findtests.py line 69

Quoted 10 lines
tests/qemu-iotests/findtests.py new file +10 -0
+        with open(fname) as f:
+            for line in f:
+                line = line.strip()
+
+                if (not line) or line[0] == '#':
+                    continue
+
+                words = line.split()
+                test_file = self.parse_test_name(words[0])
+                groups = words[1:]

The previous version still had this:

(quoted diff) +4 -0
+                if test_file not in self.all_tests:
+                    print(f'Warning: {fname}: "{test_file}" test is not found.'
+                          ' Skip.')
+                    continue
Quoted 2 lines

Why did you remove it? I found this useful when I had a wrong test name in my group.local. Now it's silently ignored.

Because now we use parse_test_name which will raise ValueError, so we will not go to this if anyway.

So, wrong name will not be silently ignored, check will fail, and you'll have to fix group file.. It is suitable?

Quoted 6 lines

tests/qemu-iotests/findtests.py line 72

Quoted 3 lines
tests/qemu-iotests/findtests.py new file +2 -0
+                for g in groups:
+                    self.groups[g].add(test_file)

Kevin

Signature · 2 lines
Best regards,
Vladimir
#13 lore raw

Re: [PATCH v7 07/11] iotests: add findtests.py

Am 22.01.2021 um 12:57 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 25 lines
Quoted 20 lines · Kevin Wolf

Am 16.01.2021 um 14:44 hat Vladimir Sementsov-Ogievskiy geschrieben:

tests/qemu-iotests/findtests.py line 69

Quoted 11 lines
tests/qemu-iotests/findtests.py new file +11 -0
+    def add_group_file(self, fname: str) -> None:
+        with open(fname) as f:
+            for line in f:
+                line = line.strip()
+
+                if (not line) or line[0] == '#':
+                    continue
+
+                words = line.split()
+                test_file = self.parse_test_name(words[0])
+                groups = words[1:]

The previous version still had this:

(quoted diff) +4 -0
+                if test_file not in self.all_tests:
+                    print(f'Warning: {fname}: "{test_file}" test is not found.'
+                          ' Skip.')
+                    continue
Quoted 2 lines

Why did you remove it? I found this useful when I had a wrong test name in my group.local. Now it's silently ignored.

Quoted 5 lines

Because now we use parse_test_name which will raise ValueError, so we will not go to this if anyway.

So, wrong name will not be silently ignored, check will fail, and you'll have to fix group file.. It is suitable?

It doesn't, though.

Oh, wait... Is it possible that you lost support for group.local altogether? grep for "group.local" comes up empty, and add_group_file() is only defined, but never called.

So the reason for the behaviour seems to be that it doesn't even try to parse the group file.

Kevin

#14 lore raw

Re: [PATCH v7 07/11] iotests: add findtests.py

Quoted 32 lines · Kevin Wolf

Am 22.01.2021 um 12:57 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 25 lines
Quoted 20 lines · Kevin Wolf

Am 16.01.2021 um 14:44 hat Vladimir Sementsov-Ogievskiy geschrieben:

tests/qemu-iotests/findtests.py line 69

Quoted 11 lines
tests/qemu-iotests/findtests.py new file +11 -0
+    def add_group_file(self, fname: str) -> None:
+        with open(fname) as f:
+            for line in f:
+                line = line.strip()
+
+                if (not line) or line[0] == '#':
+                    continue
+
+                words = line.split()
+                test_file = self.parse_test_name(words[0])
+                groups = words[1:]

The previous version still had this:

(quoted diff) +4 -0
+                if test_file not in self.all_tests:
+                    print(f'Warning: {fname}: "{test_file}" test is not found.'
+                          ' Skip.')
+                    continue
Quoted 2 lines

Why did you remove it? I found this useful when I had a wrong test name in my group.local. Now it's silently ignored.

Quoted 5 lines

Because now we use parse_test_name which will raise ValueError, so we will not go to this if anyway.

So, wrong name will not be silently ignored, check will fail, and you'll have to fix group file.. It is suitable?

Quoted 9 lines

It doesn't, though.

Oh, wait... Is it possible that you lost support for group.local altogether? grep for "group.local" comes up empty, and add_group_file() is only defined, but never called.

So the reason for the behaviour seems to be that it doesn't even try to parse the group file.

Ooops, you are right :( I've dropped an extra layer of indirection to make things simpler and group.local was lost. It's the reason to send v8, I'll do it now.

In a mean time, reverting 06 for now is OK for me.

Signature · 2 lines
Best regards,
Vladimir
#15 lore raw

Re: [PATCH v7 07/11] iotests: add findtests.py

Am 22.01.2021 um 14:16 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 39 lines
Quoted 38 lines · Kevin Wolf

Am 22.01.2021 um 12:57 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 25 lines
Quoted 20 lines · Kevin Wolf

Am 16.01.2021 um 14:44 hat Vladimir Sementsov-Ogievskiy geschrieben:

tests/qemu-iotests/findtests.py line 69

Quoted 11 lines
tests/qemu-iotests/findtests.py new file +11 -0
+    def add_group_file(self, fname: str) -> None:
+        with open(fname) as f:
+            for line in f:
+                line = line.strip()
+
+                if (not line) or line[0] == '#':
+                    continue
+
+                words = line.split()
+                test_file = self.parse_test_name(words[0])
+                groups = words[1:]

The previous version still had this:

(quoted diff) +4 -0
+                if test_file not in self.all_tests:
+                    print(f'Warning: {fname}: "{test_file}" test is not found.'
+                          ' Skip.')
+                    continue
Quoted 2 lines

Why did you remove it? I found this useful when I had a wrong test name in my group.local. Now it's silently ignored.

Quoted 5 lines

Because now we use parse_test_name which will raise ValueError, so we will not go to this if anyway.

So, wrong name will not be silently ignored, check will fail, and you'll have to fix group file.. It is suitable?

It doesn't, though.

Oh, wait... Is it possible that you lost support for group.local altogether? grep for "group.local" comes up empty, and add_group_file() is only defined, but never called.

Quoted 6 lines
Quoted 2 lines

So the reason for the behaviour seems to be that it doesn't even try to parse the group file.

Ooops, you are right :( I've dropped an extra layer of indirection to make things simpler and group.local was lost. It's the reason to send v8, I'll do it now.

You can wait with sending v8 until I've completed review in case something else comes up. So far I'm done with the changes to the part that I reviewed last time and apart from this bug they look good to me. Now it's the remaining patches.

Quoted 1 line

In a mean time, reverting 06 for now is OK for me.

Not a big deal if we get it fixed soon, it only becomes a problem if the rest of this series gets shelved for a longer time. Maybe we can complete it today, maybe on Monday, and then I'll send a pull request right away.

Kevin

#16 lore raw

Re: [PATCH v7 07/11] iotests: add findtests.py

Quoted 43 lines · Kevin Wolf

Am 22.01.2021 um 14:16 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 42 lines
Quoted 40 lines · Kevin Wolf

Am 22.01.2021 um 12:57 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 25 lines
Quoted 20 lines · Kevin Wolf

Am 16.01.2021 um 14:44 hat Vladimir Sementsov-Ogievskiy geschrieben:

tests/qemu-iotests/findtests.py line 69

Quoted 11 lines
tests/qemu-iotests/findtests.py new file +11 -0
+    def add_group_file(self, fname: str) -> None:
+        with open(fname) as f:
+            for line in f:
+                line = line.strip()
+
+                if (not line) or line[0] == '#':
+                    continue
+
+                words = line.split()
+                test_file = self.parse_test_name(words[0])
+                groups = words[1:]

The previous version still had this:

(quoted diff) +4 -0
+                if test_file not in self.all_tests:
+                    print(f'Warning: {fname}: "{test_file}" test is not found.'
+                          ' Skip.')
+                    continue
Quoted 2 lines

Why did you remove it? I found this useful when I had a wrong test name in my group.local. Now it's silently ignored.

Quoted 5 lines

Because now we use parse_test_name which will raise ValueError, so we will not go to this if anyway.

So, wrong name will not be silently ignored, check will fail, and you'll have to fix group file.. It is suitable?

It doesn't, though.

Oh, wait... Is it possible that you lost support for group.local altogether? grep for "group.local" comes up empty, and add_group_file() is only defined, but never called.

So the reason for the behaviour seems to be that it doesn't even try to parse the group file.

Quoted 8 lines
Quoted 3 lines

Ooops, you are right :( I've dropped an extra layer of indirection to make things simpler and group.local was lost. It's the reason to send v8, I'll do it now.

You can wait with sending v8 until I've completed review in case something else comes up. So far I'm done with the changes to the part that I reviewed last time and apart from this bug they look good to me. Now it's the remaining patches.

OK. This thing is to be fixed in [10], not here, I'll send a squash-in

Quoted 10 lines
Quoted 2 lines

In a mean time, reverting 06 for now is OK for me.

Not a big deal if we get it fixed soon, it only becomes a problem if the rest of this series gets shelved for a longer time. Maybe we can complete it today, maybe on Monday, and then I'll send a pull request right away.

Kevin

Signature · 2 lines
Best regards,
Vladimir
#17 lore raw

Re: [PATCH v7 07/11] iotests: add findtests.py

In reply to Vladimir Sementsov-Ogievskiy

Am 16.01.2021 um 14:44 hat Vladimir Sementsov-Ogievskiy geschrieben:

tests/qemu-iotests/findtests.py line 144

Quoted 3 lines
tests/qemu-iotests/findtests.py new file +3 -0
+        if 'disabled' not in groups and 'disabled' not in exclude_groups:
+            # Don't want to modify function argument, so create new list.
+            exclude_groups = exclude_groups + ['disabled']

Oops, forgot the other comment I wanted to make:

This would only have been needed if you had turned exclude_groups into a Sequence. Now that it's still a list, copying the list isn't strictly necessary.

Kevin

#18 lore raw

Re: [PATCH v7 07/11] iotests: add findtests.py

Quoted 11 lines · Kevin Wolf

Am 16.01.2021 um 14:44 hat Vladimir Sementsov-Ogievskiy geschrieben:

tests/qemu-iotests/findtests.py line 144

Quoted 3 lines
tests/qemu-iotests/findtests.py new file +3 -0
+        if 'disabled' not in groups and 'disabled' not in exclude_groups:
+            # Don't want to modify function argument, so create new list.
+            exclude_groups = exclude_groups + ['disabled']

Oops, forgot the other comment I wanted to make:

This would only have been needed if you had turned exclude_groups into a Sequence. Now that it's still a list, copying the list isn't strictly necessary.

I just think that such side effects (changing function arguments when it is not part of function intention) are better to avoid.

Signature · 2 lines
Best regards,
Vladimir
#19 lore raw

PATCH v7 8/11 iotests: add testenv.py

In reply to Vladimir Sementsov-Ogievskiy

Add TestEnv class, which will handle test environment in a new python iotests running framework.

Difference with current ./check interface:
- -v (verbose) option dropped, as it is unused

- -xdiff option is dropped, until somebody complains that it is needed
- same for -n option
- same for looking for binaries in $build_iotests directory.

---
 tests/qemu-iotests/testenv.py | 255 ++++++++++++++++++++++++++++++++++
 1 file changed, 255 insertions(+)
 create mode 100644 tests/qemu-iotests/testenv.py
tests/qemu-iotests/testenv.py new file +255 -0
new file mode 100644
index 0000000000..0fe5076088
@@ -0,0 +1,255 @@
+# TestEnv class to manage test environment variables.
+#
+# Copyright (c) 2020-2021 Virtuozzo International GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import os
+import sys
+import tempfile
+from pathlib import Path
+import shutil
+import collections
+import subprocess
+from contextlib import AbstractContextManager
+from typing import Dict, Any, Optional
+
+
+def get_default_machine(qemu_prog: str) -> str:
+    outp = subprocess.run([qemu_prog, '-machine', 'help'], check=True,
+                          text=True, stdout=subprocess.PIPE).stdout
+
+    machines = outp.split('\n')
+    default_machine = next(m for m in machines if m.endswith(' (default)'))
+    default_machine = default_machine.split(' ', 1)[0]
+
+    alias_suf = ' (alias of {})'.format(default_machine)
+    alias = next((m for m in machines if m.endswith(alias_suf)), None)
+    if alias is not None:
+        default_machine = alias.split(' ', 1)[0]
+
+    return default_machine
+
+
+class TestEnv(AbstractContextManager['TestEnv']):
+    """
+    Manage system environment for running tests
+
+    The following variables are supported/provided. They are represented by
+    lower-cased TestEnv attributes.
+    """
+
+    # We store environment variables as instance attributes, and there are a
+    # lot of them. Silence pylint:
+    # pylint: disable=too-many-instance-attributes
+
+    env_variables = ['PYTHONPATH', 'TEST_DIR', 'SOCK_DIR', 'SAMPLE_IMG_DIR',
+                     'OUTPUT_DIR', 'PYTHON', 'QEMU_PROG', 'QEMU_IMG_PROG',
+                     'QEMU_IO_PROG', 'QEMU_NBD_PROG', 'QSD_PROG',
+                     'SOCKET_SCM_HELPER', 'QEMU_OPTIONS', 'QEMU_IMG_OPTIONS',
+                     'QEMU_IO_OPTIONS', 'QEMU_NBD_OPTIONS', 'IMGOPTS',
+                     'IMGFMT', 'IMGPROTO', 'AIOMODE', 'CACHEMODE',
+                     'VALGRIND_QEMU', 'CACHEMODE_IS_DEFAULT', 'IMGFMT_GENERIC',
+                     'IMGOPTSSYNTAX', 'IMGKEYSECRET', 'QEMU_DEFAULT_MACHINE']
+
+    def get_env(self) -> Dict[str, str]:
+        env = {}
+        for v in self.env_variables:
+            val = getattr(self, v.lower(), None)
+            if val is not None:
+                env[v] = val
+
+        return env
+
+    def init_directories(self):
+        """Init directory variables:
+             PYTHONPATH
+             TEST_DIR
+             SOCK_DIR
+             SAMPLE_IMG_DIR
+             OUTPUT_DIR
+        """
+        self.pythonpath = os.getenv('PYTHONPATH')
+        if self.pythonpath:
+            self.pythonpath = self.source_iotests + os.pathsep + \
+                self.pythonpath
+        else:
+            self.pythonpath = self.source_iotests
+
+        self.test_dir = os.getenv('TEST_DIR',
+                                  os.path.join(os.getcwd(), 'scratch'))
+        Path(self.test_dir).mkdir(parents=True, exist_ok=True)
+
+        self.sock_dir = os.getenv('SOCK_DIR')
+        self.tmp_sock_dir = False
+        if self.sock_dir:
+            Path(self.test_dir).mkdir(parents=True, exist_ok=True)
+        else:
+            self.sock_dir = tempfile.mkdtemp()
+            self.tmp_sock_dir = True
+
+        self.sample_img_dir = os.getenv('SAMPLE_IMG_DIR',
+                                        os.path.join(self.source_iotests,
+                                                     'sample_images'))
+
+        self.output_dir = os.getcwd()  # OUTPUT_DIR
+
+    def init_binaries(self):
+        """Init binary path variables:
+             PYTHON (for bash tests)
+             QEMU_PROG, QEMU_IMG_PROG, QEMU_IO_PROG, QEMU_NBD_PROG, QSD_PROG
+             SOCKET_SCM_HELPER
+        """
+        self.python = sys.executable
+
+        def root(*names):
+            return os.path.join(self.build_root, *names)
+
+        arch = os.uname().machine
+        if 'ppc64' in arch:
+            arch = 'ppc64'
+
+        self.qemu_prog = os.getenv('QEMU_PROG', root(f'qemu-system-{arch}'))
+        self.qemu_img_prog = os.getenv('QEMU_IMG_PROG', root('qemu-img'))
+        self.qemu_io_prog = os.getenv('QEMU_IO_PROG', root('qemu-io'))
+        self.qemu_nbd_prog = os.getenv('QEMU_NBD_PROG', root('qemu-nbd'))
+        self.qsd_prog = os.getenv('QSD_PROG', root('storage-daemon',
+                                                   'qemu-storage-daemon'))
+
+        for b in [self.qemu_img_prog, self.qemu_io_prog, self.qemu_nbd_prog,
+                  self.qemu_prog, self.qsd_prog]:
+            if not os.path.exists(b):
+                sys.exit('Not such file: ' + b)
+            if not os.access(b, os.X_OK):
+                sys.exit('Not executable: ' + b)
+
+        helper_path = os.path.join(self.build_iotests, 'socket_scm_helper')
+        if os.access(helper_path, os.X_OK):
+            self.socket_scm_helper = helper_path  # SOCKET_SCM_HELPER
+
+    def __init__(self, imgfmt: str, imgproto: str, aiomode: str,
+                 cachemode: Optional[str] = None,
+                 imgopts: Optional[str] = None,
+                 misalign: bool = False,
+                 debug: bool = False) -> None:
+        self.imgfmt = imgfmt
+        self.imgproto = imgproto
+        self.aiomode = aiomode
+        self.imgopts = imgopts
+        self.misalign = misalign
+        self.debug = debug
+
+        if cachemode is None:
+            self.cachemode_is_default = 'true'
+            self.cachemode = 'writeback'
+        else:
+            self.cachemode_is_default = 'false'
+            self.cachemode = cachemode
+
+        # Initialize generic paths: build_root, build_iotests, source_iotests,
+        # which are needed to initialize some environment variables. They are
+        # used by init_*() functions as well.
+
+        if os.path.islink(sys.argv[0]):
+            # called from the build tree
+            self.source_iotests = os.path.dirname(os.readlink(sys.argv[0]))
+            self.build_iotests = os.path.dirname(os.path.abspath(sys.argv[0]))
+        else:
+            # called from the source tree
+            self.source_iotests = os.getcwd()
+            self.build_iotests = self.source_iotests
+
+        self.build_root = os.path.join(self.build_iotests, '..', '..')
+
+        self.init_directories()
+        self.init_binaries()
+
+        # QEMU_OPTIONS
+        self.qemu_options = '-nodefaults -display none -accel qtest'
+        machine_map = (
+            ('arm', 'virt'),
+            ('aarch64', 'virt'),
+            ('avr', 'mega2560'),
+            ('rx', 'gdbsim-r5f562n8'),
+            ('tricore', 'tricore_testboard')
+        )
+        for suffix, machine in machine_map:
+            if self.qemu_prog.endswith(f'qemu-system-{suffix}'):
+                self.qemu_options += f' -machine {machine}'
+
+        # QEMU_DEFAULT_MACHINE
+        self.qemu_default_machine = get_default_machine(self.qemu_prog)
+
+        self.qemu_img_options = os.getenv('QEMU_IMG_OPTIONS')
+        self.qemu_nbd_options = os.getenv('QEMU_NBD_OPTIONS')
+
+        is_generic = self.imgfmt not in ['bochs', 'cloop', 'dmg']
+        self.imgfmt_generic = 'true' if is_generic else 'false'
+
+        self.qemu_io_options = f'--cache {self.cachemode} --aio {self.aiomode}'
+        if self.misalign:
+            self.qemu_io_options += ' --misalign'
+
+        self.qemu_io_options_no_fmt = self.qemu_io_options
+
+        if self.imgfmt == 'luks':
+            self.imgoptssyntax = 'true'
+            self.imgkeysecret = '123456'
+            if not self.imgopts:
+                self.imgopts = 'iter-time=10'
+            elif 'iter-time=' not in self.imgopts:
+                self.imgopts += ',iter-time=10'
+        else:
+            self.imgoptssyntax = 'false'
+            self.qemu_io_options += ' -f ' + self.imgfmt
+
+        if self.imgfmt == 'vmkd':
+            if not self.imgopts:
+                self.imgopts = 'zeroed_grain=on'
+            elif 'zeroed_grain=' not in self.imgopts:
+                self.imgopts += ',zeroed_grain=on'
+
+    def close(self) -> None:
+        if self.tmp_sock_dir:
+            shutil.rmtree(self.sock_dir)
+
+    def __enter__(self) -> 'TestEnv':
+        return self
+
+    def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None:
+        self.close()
+
+    def print_env(self) -> None:
+        template = """\
+QEMU          -- "{QEMU_PROG}" {QEMU_OPTIONS}
+QEMU_IMG      -- "{QEMU_IMG_PROG}" {QEMU_IMG_OPTIONS}
+QEMU_IO       -- "{QEMU_IO_PROG}" {QEMU_IO_OPTIONS}
+QEMU_NBD      -- "{QEMU_NBD_PROG}" {QEMU_NBD_OPTIONS}
+IMGFMT        -- {IMGFMT}{imgopts}
+IMGPROTO      -- {IMGPROTO}
+PLATFORM      -- {platform}
+TEST_DIR      -- {TEST_DIR}
+SOCK_DIR      -- {SOCK_DIR}
+SOCKET_SCM_HELPER -- {SOCKET_SCM_HELPER}"""
+
+        args = collections.defaultdict(str, self.get_env())
+
+        if 'IMGOPTS' in args:
+            args['imgopts'] = f" ({args['IMGOPTS']})"
+
+        u = os.uname()
+        args['platform'] = f'{u.sysname}/{u.machine} {u.nodename} {u.release}'
+
+        print(template.format_map(args))
Signature · 1 line
2.29.2
Tested-by on v7 8/11
#20 lore raw

Re: [PATCH v7 08/11] iotests: add testenv.py

Quoted 22 lines · Vladimir Sementsov-Ogievskiy

Add TestEnv class, which will handle test environment in a new python iotests running framework.

Difference with current ./check interface:
- -v (verbose) option dropped, as it is unused

- -xdiff option is dropped, until somebody complains that it is needed
- same for -n option
- same for looking for binaries in $build_iotests directory.

---
 tests/qemu-iotests/testenv.py | 255 ++++++++++++++++++++++++++++++++++
 1 file changed, 255 insertions(+)
 create mode 100644 tests/qemu-iotests/testenv.py
tests/qemu-iotests/testenv.py new file +0 -0
new file mode 100644
index 0000000000..0fe5076088

tests/qemu-iotests/testenv.py line 1

Quoted 1 line
tests/qemu-iotests/testenv.py new file +1 -0
@@ -0,0 +1,255 @@
+# TestEnv class to manage test environment variables.

Should this have permissions 755 and a #! python line?

Quoted 4 lines
tests/qemu-iotests/testenv.py new file +4 -0
+def get_default_machine(qemu_prog: str) -> str:
+    outp = subprocess.run([qemu_prog, '-machine', 'help'], check=True,
+                          text=True, stdout=subprocess.PIPE).stdout
+

tests/qemu-iotests/testenv.py line 43

Quoted 10 lines
tests/qemu-iotests/testenv.py new file +10 -0
+    machines = outp.split('\n')
+    default_machine = next(m for m in machines if m.endswith(' (default)'))
+    default_machine = default_machine.split(' ', 1)[0]
+
+    alias_suf = ' (alias of {})'.format(default_machine)
+    alias = next((m for m in machines if m.endswith(alias_suf)), None)
+    if alias is not None:
+        default_machine = alias.split(' ', 1)[0]
+
+    return default_machine

I have no idea if this is the most efficient and idiomatic way for python to extract the default machine out of the output list, but it seems to work.

tests/qemu-iotests/testenv.py line 134

Quoted 12 lines
tests/qemu-iotests/testenv.py new file +10 -0
+
+
+class TestEnv(AbstractContextManager['TestEnv']):
 
 
+        self.qsd_prog = os.getenv('QSD_PROG', root('storage-daemon',
+                                                   'qemu-storage-daemon'))
+
+        for b in [self.qemu_img_prog, self.qemu_io_prog, self.qemu_nbd_prog,
+                  self.qemu_prog, self.qsd_prog]:
+            if not os.path.exists(b):
+                sys.exit('Not such file: ' + b)

No such file

Quoted 1 line

+ if self.imgfmt == 'vmkd':

vmdk (which means we need to test that './check -vmdk' still works...)

I can see a rough correspondence to the existing shell code in ./check, and with the entire series applied, things still work (other than vmdk, which should work with the typo fix). I'm not comfortable enough to offer my full review of the python code, but I can give:

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#21 lore raw

Re: [PATCH v7 08/11] iotests: add testenv.py

Quoted 17 lines · Eric Blake
Quoted 16 lines · Vladimir Sementsov-Ogievskiy

Add TestEnv class, which will handle test environment in a new python iotests running framework.

Difference with current ./check interface:
- -v (verbose) option dropped, as it is unused

- -xdiff option is dropped, until somebody complains that it is needed
- same for -n option
- same for looking for binaries in $build_iotests directory.

---
  tests/qemu-iotests/testenv.py | 255 ++++++++++++++++++++++++++++++++++
  1 file changed, 255 insertions(+)

create mode 100644 tests/qemu-iotests/testenv.py

Quoted 9 lines

tests/qemu-iotests/testenv.py line 1

Quoted 7 lines
tests/qemu-iotests/testenv.py new file +1 -0
new file mode 100644
index 0000000000..0fe5076088
@@ -0,0 +1,255 @@
+# TestEnv class to manage test environment variables.

Should this have permissions 755 and a #! python line?

I understood that keeping cmdline interface inside these three classes doesn't help but only makes things more difficult. So it's the change of v7: the whole cmdline interface is now in check script, and these three .py files just defines normal classes. So, I dropped __main__ things as well together with executable permissions.

Quoted 47 lines
Quoted 5 lines
tests/qemu-iotests/testenv.py new file +4 -0
+def get_default_machine(qemu_prog: str) -> str:
+    outp = subprocess.run([qemu_prog, '-machine', 'help'], check=True,
+                          text=True, stdout=subprocess.PIPE).stdout
+

tests/qemu-iotests/testenv.py line 43

Quoted 10 lines
tests/qemu-iotests/testenv.py new file +10 -0
+    machines = outp.split('\n')
+    default_machine = next(m for m in machines if m.endswith(' (default)'))
+    default_machine = default_machine.split(' ', 1)[0]
+
+    alias_suf = ' (alias of {})'.format(default_machine)
+    alias = next((m for m in machines if m.endswith(alias_suf)), None)
+    if alias is not None:
+        default_machine = alias.split(' ', 1)[0]
+
+    return default_machine

I have no idea if this is the most efficient and idiomatic way for python to extract the default machine out of the output list, but it seems to work.

tests/qemu-iotests/testenv.py line 134

Quoted 12 lines
tests/qemu-iotests/testenv.py new file +10 -0
+
+
+class TestEnv(AbstractContextManager['TestEnv']):
 
 
+        self.qsd_prog = os.getenv('QSD_PROG', root('storage-daemon',
+                                                   'qemu-storage-daemon'))
+
+        for b in [self.qemu_img_prog, self.qemu_io_prog, self.qemu_nbd_prog,
+                  self.qemu_prog, self.qsd_prog]:
+            if not os.path.exists(b):
+                sys.exit('Not such file: ' + b)

No such file

Quoted 1 line

+ if self.imgfmt == 'vmkd':

vmdk (which means we need to test that './check -vmdk' still works...)

I can see a rough correspondence to the existing shell code in ./check, and with the entire series applied, things still work (other than vmdk, which should work with the typo fix). I'm not comfortable enough to offer my full review of the python code, but I can give:

Signature · 2 lines
Best regards,
Vladimir
#22 lore raw

Re: [PATCH v7 08/11] iotests: add testenv.py

In reply to Vladimir Sementsov-Ogievskiy

Am 16.01.2021 um 14:44 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 14 lines

Add TestEnv class, which will handle test environment in a new python iotests running framework.

Difference with current ./check interface:
- -v (verbose) option dropped, as it is unused

- -xdiff option is dropped, until somebody complains that it is needed
- same for -n option
- same for looking for binaries in $build_iotests directory.

tests/qemu-iotests/testenv.py new file +2 -0
+        if self.imgfmt == 'luks':
+            self.imgoptssyntax = 'true'

tests/qemu-iotests/testenv.py line 218

Quoted 10 lines
tests/qemu-iotests/testenv.py new file +10 -0
+            self.imgkeysecret = '123456'
+            if not self.imgopts:
+                self.imgopts = 'iter-time=10'
+            elif 'iter-time=' not in self.imgopts:
+                self.imgopts += ',iter-time=10'
+        else:
+            self.imgoptssyntax = 'false'
+            self.qemu_io_options += ' -f ' + self.imgfmt
+
+        if self.imgfmt == 'vmkd':

This should be 'vmdk', of course.

tests/qemu-iotests/testenv.py line 222

Quoted 4 lines
tests/qemu-iotests/testenv.py new file +4 -0
+            if not self.imgopts:
+                self.imgopts = 'zeroed_grain=on'
+            elif 'zeroed_grain=' not in self.imgopts:
+                self.imgopts += ',zeroed_grain=on'

Kevin

#23 lore raw

PATCH v7 9/11 iotests: add testrunner.py

In reply to Vladimir Sementsov-Ogievskiy

Add TestRunner class, which will run tests in a new python iotests running framework.

There are some differences with current ./check behavior, most significant are:
- Consider all tests self-executable, just run them, don't run python by hand.
- Elapsed time is cached in json file
- Elapsed time precision increased a bit
- use python difflib instead of "diff -w", to ignore spaces at line ends strip lines by hand. Do not ignore other spaces.

---
 tests/qemu-iotests/testrunner.py | 344 +++++++++++++++++++++++++++++++
 1 file changed, 344 insertions(+)
 create mode 100644 tests/qemu-iotests/testrunner.py
tests/qemu-iotests/testrunner.py new file +344 -0
new file mode 100644
index 0000000000..92722cc68b
@@ -0,0 +1,344 @@
+# Class for actual tests running.
+#
+# Copyright (c) 2020-2021 Virtuozzo International GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import os
+import random
+from pathlib import Path
+import datetime
+import time
+import difflib
+import subprocess
+import collections
+import contextlib
+import json
+import termios
+import sys
+from contextlib import contextmanager
+from contextlib import AbstractContextManager
+from typing import List, Optional, Iterator, Any
+
+from testenv import TestEnv
+
+
+def silent_unlink(path: Path) -> None:
+    try:
+        path.unlink()
+    except OSError:
+        pass
+
+
+def file_diff(file1: str, file2: str) -> List[str]:
+    with open(file1) as f1, open(file2) as f2:
+        # We want to ignore spaces at line ends. There are a lot of mess about
+        # it in iotests.
+        # TODO: fix all tests to not produce extra spaces, fix all .out files
+        # and use strict diff here!
+        seq1 = [line.rstrip() for line in f1]
+        seq2 = [line.rstrip() for line in f2]
+        return list(difflib.unified_diff(seq1, seq2, file1, file2))
+
+
+# We want to save current tty settings during test run,
+# since an aborting qemu call may leave things screwed up.
+@contextmanager
+def savetty() -> Iterator[None]:
+    isterm = sys.stdin.isatty()
+    if isterm:
+        fd = sys.stdin.fileno()
+        attr = termios.tcgetattr(0)
+
+    try:
+        yield
+    finally:
+        if isterm:
+            termios.tcsetattr(fd, termios.TCSADRAIN, attr)
+
+
+class LastElapsedTime(AbstractContextManager['LastElapsedTime']):
+    """ Cache for elapsed time for tests, to show it during new test run
+
+    Use get() in any time. But, if use update you should then call save(),
+    or use update() inside with-block.
+    """
+    def __init__(self, cache_file: str, env: TestEnv) -> None:
+        self.env = env
+        self.cache_file = cache_file
+
+        try:
+            with open(cache_file) as f:
+                self.cache = json.load(f)
+        except (OSError, ValueError):
+            self.cache = {}
+
+    def get(self, test: str,
+            default: Optional[float] = None) -> Optional[float]:
+        if test not in self.cache:
+            return default
+
+        if self.env.imgproto not in self.cache[test]:
+            return default
+
+        return self.cache[test][self.env.imgproto].get(self.env.imgfmt,
+                                                       default)
+
+    def update(self, test: str, elapsed: float) -> None:
+        d = self.cache.setdefault(test, {})
+        d = d.setdefault(self.env.imgproto, {})
+        d[self.env.imgfmt] = elapsed
+
+    def save(self) -> None:
+        with open(self.cache_file, 'w') as f:
+            json.dump(self.cache, f)
+
+    def __enter__(self) -> 'LastElapsedTime':
+        return self
+
+    def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None:
+        self.save()
+
+
+TestResult = collections.namedtuple(
+    'TestResult',
+    ['status', 'description', 'elapsed', 'diff', 'casenotrun'],
+    defaults=('', '', '', ''))
+
+
+class TestRunner(AbstractContextManager['TestRunner']):
+    def __init__(self, env: TestEnv, makecheck: bool = False) -> None:
+        self.env = env
+        self.test_run_env = self.env.get_env()
+        if 'MALLOC_PERTURB_' not in os.environ and \
+                'MALLOC_PERTURB_' not in self.test_run_env:
+            x = random.randrange(1, 255)
+            self.test_run_env['MALLOC_PERTURB_'] = str(x)
+
+        self.makecheck = makecheck
+
+        self.last_elapsed = LastElapsedTime('.last-elapsed-cache', env)
+
+    def __enter__(self) -> 'TestRunner':
+        # pylint: disable=attribute-defined-outside-init
+        self._stack = contextlib.ExitStack()
+        self._stack.enter_context(self.env)
+        self._stack.enter_context(self.last_elapsed)
+        self._stack.enter_context(savetty())
+        return self
+
+    def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None:
+        self._stack.close()
+
+    def test_print_one_line(self, test: str, starttime: str,
+                            endtime: Optional[str] = None, status: str = '...',
+                            lasttime: Optional[float] = None,
+                            thistime: Optional[str] = None,
+                            description: str = '',
+                            test_field_width: Optional[int] = None,
+                            end: str = '\n') -> None:
+        """ Print short test info before/after test run """
+        test = os.path.basename(test)
+
+        if test_field_width is None:
+            test_field_width = 8
+
+        if self.makecheck and status != '...':
+            if status and status != 'pass':
+                status = f' [{status}]'
+            else:
+                status = ''
+
+            print(f'  TEST   iotest-{self.env.imgfmt}: {test}{status}')
+            return
+
+        if lasttime:
+            lasttime_s = f' (last: {lasttime}s)'
+        else:
+            lasttime_s = ''
+        if thistime:
+            thistime = f'{thistime:.1f}s'
+        else:
+            thistime = '...'
+
+        if endtime:
+            endtime = f'[{endtime}]'
+        else:
+            endtime = ''
+
+        if status == 'pass':
+            col = '\033[32m'
+        elif status == 'fail':
+            col = '\033[1m\033[31m'
+        elif status == 'not run':
+            col = '\033[33m'
+        else:
+            col = ''
+
+        col_end = '\033[0m'
+
+        print(f'{test:{test_field_width}} {col}{status:10}{col_end} '
+              f'[{starttime}] {endtime:13}{thistime:5} {lasttime_s:14} '
+              f'{description}', end=end)
+
+    def find_reference(self, test: str) -> str:
+        if self.env.cachemode == 'none':
+            ref = f'{test}.out.nocache'
+            if os.path.isfile(ref):
+                return ref
+
+        ref = f'{test}.out.{self.env.imgfmt}'
+        if os.path.isfile(ref):
+            return ref
+
+        ref = f'{test}.{self.env.qemu_default_machine}.out'
+        if os.path.isfile(ref):
+            return ref
+
+        return f'{test}.out'
+
+    def do_run_test(self, test: str) -> TestResult:
+        f_test = Path(test)
+        f_bad = Path(f_test.name + '.out.bad')
+        f_notrun = Path(f_test.name + '.notrun')
+        f_casenotrun = Path(f_test.name + '.casenotrun')
+        f_reference = Path(self.find_reference(test))
+
+        if not f_test.exists():
+            return TestResult(status='fail',
+                              description=f'No such test file: {f_test}')
+
+        if not os.access(str(f_test), os.X_OK):
+            sys.exit(f'Not executable: {f_test}')
+
+        if not f_reference.exists():
+            return TestResult(status='fail',
+                              description='No qualified output '
+                                          f'(expected {f_reference})')
+
+        for p in (f_bad, f_notrun, f_casenotrun):
+            silent_unlink(p)
+
+        args = [str(f_test.resolve())]
+        if self.env.debug:
+            args.append('-d')
+
+        with f_test.open() as f:
+            try:
+                if f.readline() == '#!/usr/bin/env python3':
+                    args.insert(0, self.env.python)
+            except UnicodeDecodeError:  # binary test? for future.
+                pass
+
+        env = os.environ.copy()
+        env.update(self.test_run_env)
+
+        t0 = time.time()
+        with f_bad.open('w') as f:
+            ret = subprocess.run(args, cwd=str(f_test.parent), env=env,
+                                 stdout=f, stderr=subprocess.STDOUT,
+                                 check=False).returncode
+        elapsed = round(time.time() - t0, 1)
+
+        if ret != 0:
+            return TestResult(status='fail', elapsed=elapsed,
+                              description=f'failed, exit status {ret}',
+                              diff=file_diff(str(f_reference), str(f_bad)))
+
+        if f_notrun.exists():
+            return TestResult(status='not run', elapsed=elapsed,
+                              description=f_notrun.read_text().strip())
+
+        casenotrun = None
+        if f_casenotrun.exists():
+            casenotrun = f_casenotrun.read_text()
+
+        diff = file_diff(str(f_reference), str(f_bad))
+        if diff:
+            return TestResult(status='fail', elapsed=elapsed,
+                              description='output mismatch (see {f_bad})',
+                              diff=diff, casenotrun=casenotrun)
+        else:
+            f_bad.unlink()
+            self.last_elapsed.update(test, elapsed)
+            return TestResult(status='pass', elapsed=elapsed,
+                              casenotrun=casenotrun)
+
+    def run_test(self, test: str,
+                 test_field_width: Optional[int] = None) -> TestResult:
+        last_el = self.last_elapsed.get(test)
+        start = datetime.datetime.now().strftime('%H:%M:%S')
+
+        self.test_print_one_line(test=test, starttime=start, lasttime=last_el,
+                                 end='\r', test_field_width=test_field_width)
+
+        res = self.do_run_test(test)
+
+        end = datetime.datetime.now().strftime('%H:%M:%S')
+        self.test_print_one_line(test=test, status=res.status,
+                                 starttime=start, endtime=end,
+                                 lasttime=last_el, thistime=res.elapsed,
+                                 description=res.description,
+                                 test_field_width=test_field_width)
+
+        if res.casenotrun:
+            print(res.casenotrun)
+
+        return res
+
+    def run_tests(self, tests: List[str]) -> None:
+        n_run = 0
+        failed = []
+        notrun = []
+        casenotrun = []
+
+        if not self.makecheck:
+            self.env.print_env()
+            print()
+
+        test_field_width = max(len(os.path.basename(t)) for t in tests) + 2
+
+        for t in tests:
+            name = os.path.basename(t)
+            res = self.run_test(t, test_field_width=test_field_width)
+
+            assert res.status in ('pass', 'fail', 'not run')
+
+            if res.casenotrun:
+                casenotrun.append(t)
+
+            if res.status != 'not run':
+                n_run += 1
+
+            if res.status == 'fail':
+                failed.append(name)
+                if self.makecheck:
+                    self.env.print_env()
+                if res.diff:
+                    print('\n'.join(res.diff))
+            elif res.status == 'not run':
+                notrun.append(name)
+
+        if notrun:
+            print('Not run:', ' '.join(notrun))
+
+        if casenotrun:
+            print('Some cases not run in:', ' '.join(casenotrun))
+
+        if failed:
+            print('Failures:', ' '.join(failed))
+            print(f'Failed {len(failed)} of {n_run} iotests')
+        else:
+            print(f'Passed all {n_run} iotests')
Signature · 1 line
2.29.2
Tested-by on v7 9/11
#24 lore raw

Re: [PATCH v7 09/11] iotests: add testrunner.py

Quoted 11 lines · Vladimir Sementsov-Ogievskiy

Add TestRunner class, which will run tests in a new python iotests running framework.

There are some differences with current ./check behavior, most significant are:
- Consider all tests self-executable, just run them, don't run python by hand.
- Elapsed time is cached in json file
- Elapsed time precision increased a bit
- use python difflib instead of "diff -w", to ignore spaces at line ends strip lines by hand. Do not ignore other spaces.

Awkward wording. Maybe:

Instead of using "diff -w" which ignores all whitespace differences, manually strip whitespace at line end then use python difflib, which no longer ignores spacing mid-line

Quoted 13 lines
---
 tests/qemu-iotests/testrunner.py | 344 +++++++++++++++++++++++++++++++
 1 file changed, 344 insertions(+)
 create mode 100644 tests/qemu-iotests/testrunner.py
tests/qemu-iotests/testrunner.py new file +0 -0
new file mode 100644
index 0000000000..92722cc68b

tests/qemu-iotests/testrunner.py line 1

Quoted 1 line
tests/qemu-iotests/testrunner.py new file +1 -0
@@ -0,0 +1,344 @@
+# Class for actual tests running.

for actually running tests

again, should this file be 755 with #! python?

tests/qemu-iotests/testrunner.py line 53

Quoted 10 lines
tests/qemu-iotests/testrunner.py new file +10 -0
+
+def file_diff(file1: str, file2: str) -> List[str]:
+    with open(file1) as f1, open(file2) as f2:
+        # We want to ignore spaces at line ends. There are a lot of mess about
+        # it in iotests.
+        # TODO: fix all tests to not produce extra spaces, fix all .out files
+        # and use strict diff here!
+        seq1 = [line.rstrip() for line in f1]
+        seq2 = [line.rstrip() for line in f2]
+        return list(difflib.unified_diff(seq1, seq2, file1, file2))

Offhand, do you have the list of tests where (actual/expected) output has trailing whitespace and would fail without the .rstrip()?

tests/qemu-iotests/testrunner.py line 63

Quoted 10 lines
tests/qemu-iotests/testrunner.py new file +10 -0
+
+
+# We want to save current tty settings during test run,
+# since an aborting qemu call may leave things screwed up.
+@contextmanager
+def savetty() -> Iterator[None]:
+    isterm = sys.stdin.isatty()
+    if isterm:
+        fd = sys.stdin.fileno()
+        attr = termios.tcgetattr(0)

Isn't fd always going to be 0? It looks odd to hard-code zero in the very next line; either we should s/0/fd/ here, or...

tests/qemu-iotests/testrunner.py line 69

Quoted 6 lines
tests/qemu-iotests/testrunner.py new file +6 -0
+
+    try:
+        yield
+    finally:
+        if isterm:
+            termios.tcsetattr(fd, termios.TCSADRAIN, attr)

... s/fd/0/ here and drop fd altogether.

tests/qemu-iotests/testrunner.py line 76

Quoted 7 lines
tests/qemu-iotests/testrunner.py new file +7 -0
+
+
+class LastElapsedTime(AbstractContextManager['LastElapsedTime']):
+    """ Cache for elapsed time for tests, to show it during new test run
+
+    Use get() in any time. But, if use update you should then call save(),
+    or use update() inside with-block.

Grammar is hard, maybe:

It is safe to use get() at any time. To use update(), you must either use a with-block or first use save().

tests/qemu-iotests/testrunner.py line 186

Quoted 9 lines
tests/qemu-iotests/testrunner.py new file +8 -0
+    def test_print_one_line(self, test: str, starttime: str,
 
+
+        if status == 'pass':
+            col = '\033[32m'
+        elif status == 'fail':
+            col = '\033[1m\033[31m'
+        elif status == 'not run':
+            col = '\033[33m'

This hard-codes the use of ANSI escape sequences without first checking that we are writing to a terminal. Is that wise? Should we have this be tunable by a tri-state command-line option, similar to ls --color? (--color=auto is default, and bases decision on istty(), --color=off turns color off even for a terminal, --color=on uses color even when outputting to a pipe, which can be useful depending on the other end of the pipeline...)

tests/qemu-iotests/testrunner.py line 243

Quoted 6 lines
tests/qemu-iotests/testrunner.py new file +6 -0
+        with f_test.open() as f:
+            try:
+                if f.readline() == '#!/usr/bin/env python3':
+                    args.insert(0, self.env.python)
+            except UnicodeDecodeError:  # binary test? for future.
+                pass

Is pass the right action here? Or will it silently skip a test file with encoding errors?

Again, I'm not comfortable enough to give a full review of the python, but it looks fairly similar to the existing shell code, and with the series applied, things still work. So I can offer

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#25 lore raw

Re: [PATCH v7 09/11] iotests: add testrunner.py

Quoted 36 lines · Eric Blake
Quoted 11 lines · Vladimir Sementsov-Ogievskiy

Add TestRunner class, which will run tests in a new python iotests running framework.

There are some differences with current ./check behavior, most significant are:
- Consider all tests self-executable, just run them, don't run python by hand.
- Elapsed time is cached in json file
- Elapsed time precision increased a bit
- use python difflib instead of "diff -w", to ignore spaces at line ends strip lines by hand. Do not ignore other spaces.

Awkward wording. Maybe:

Instead of using "diff -w" which ignores all whitespace differences, manually strip whitespace at line end then use python difflib, which no longer ignores spacing mid-line

Quoted 13 lines
---
  tests/qemu-iotests/testrunner.py | 344 +++++++++++++++++++++++++++++++
  1 file changed, 344 insertions(+)

create mode 100644 tests/qemu-iotests/testrunner.py

tests/qemu-iotests/testrunner.py new file +0 -0
new file mode 100644
index 0000000000..92722cc68b

tests/qemu-iotests/testrunner.py line 1

Quoted 1 line
tests/qemu-iotests/testrunner.py new file +1 -0
@@ -0,0 +1,344 @@
+# Class for actual tests running.

for actually running tests

Quoted 1 line

again, should this file be 755 with #! python?

In my latest considerations - no it shouldn't.. We still make implement __main__ things later if needed.

Quoted 14 lines

tests/qemu-iotests/testrunner.py line 53

Quoted 11 lines
tests/qemu-iotests/testrunner.py new file +10 -0
+
+def file_diff(file1: str, file2: str) -> List[str]:
+    with open(file1) as f1, open(file2) as f2:
+        # We want to ignore spaces at line ends. There are a lot of mess about
+        # it in iotests.
+        # TODO: fix all tests to not produce extra spaces, fix all .out files
+        # and use strict diff here!
+        seq1 = [line.rstrip() for line in f1]
+        seq2 = [line.rstrip() for line in f2]
+        return list(difflib.unified_diff(seq1, seq2, file1, file2))

Offhand, do you have the list of tests where (actual/expected) output has trailing whitespace and would fail without the .rstrip()?

No.. But it's simple to make it

Quoted 14 lines

tests/qemu-iotests/testrunner.py line 63

Quoted 11 lines
tests/qemu-iotests/testrunner.py new file +10 -0
+
+
+# We want to save current tty settings during test run,
+# since an aborting qemu call may leave things screwed up.
+@contextmanager
+def savetty() -> Iterator[None]:
+    isterm = sys.stdin.isatty()
+    if isterm:
+        fd = sys.stdin.fileno()
+        attr = termios.tcgetattr(0)

Isn't fd always going to be 0? It looks odd to hard-code zero in the very next line; either we should s/0/fd/ here, or...

agree that's strange.

Quoted 9 lines

tests/qemu-iotests/testrunner.py line 69

Quoted 7 lines
tests/qemu-iotests/testrunner.py new file +6 -0
+
+    try:
+        yield
+    finally:
+        if isterm:
+            termios.tcsetattr(fd, termios.TCSADRAIN, attr)

... s/fd/0/ here and drop fd altogether.

Either way is OK for me, I think, I'll do s/0/fd/

Quoted 11 lines

tests/qemu-iotests/testrunner.py line 76

Quoted 8 lines
tests/qemu-iotests/testrunner.py new file +7 -0
+
+
+class LastElapsedTime(AbstractContextManager['LastElapsedTime']):
+    """ Cache for elapsed time for tests, to show it during new test run
+
+    Use get() in any time. But, if use update you should then call save(),
+    or use update() inside with-block.

Grammar is hard, maybe:

Quoted 2 lines

It is safe to use get() at any time. To use update(), you must either use a with-block or first use save().

OK, thanks

Quoted 12 lines

tests/qemu-iotests/testrunner.py line 186

Quoted 11 lines
tests/qemu-iotests/testrunner.py new file +8 -0
+    def test_print_one_line(self, test: str, starttime: str,
 
+
+        if status == 'pass':
+            col = '\033[32m'
+        elif status == 'fail':
+            col = '\033[1m\033[31m'
+        elif status == 'not run':
+            col = '\033[33m'
Quoted 7 lines

This hard-codes the use of ANSI escape sequences without first checking that we are writing to a terminal. Is that wise? Should we have this be tunable by a tri-state command-line option, similar to ls --color? (--color=auto is default, and bases decision on istty(), --color=off turns color off even for a terminal, --color=on uses color even when outputting to a pipe, which can be useful depending on the other end of the pipeline...)

Hmm, yes. It's preexisting in old bash check script I think. I can add a separate patch for it

Quoted 11 lines

tests/qemu-iotests/testrunner.py line 243

Quoted 8 lines
tests/qemu-iotests/testrunner.py new file +6 -0
+        with f_test.open() as f:
+            try:
+                if f.readline() == '#!/usr/bin/env python3':
+                    args.insert(0, self.env.python)
+            except UnicodeDecodeError:  # binary test? for future.
+                pass

Is pass the right action here? Or will it silently skip a test file with encoding errors?

No, we'll not skip it. Here we just failed to recognize python test, so, it will be executed as self-executable. So, if there are real problems, we'll see them when try to execute the file.

Quoted 7 lines

Again, I'm not comfortable enough to give a full review of the python, but it looks fairly similar to the existing shell code, and with the series applied, things still work. So I can offer

Thanks a lot for testing and reviewing so many my patches!!

Signature · 2 lines
Best regards,
Vladimir
#26 lore raw

Re: [PATCH v7 09/11] iotests: add testrunner.py

In reply to Vladimir Sementsov-Ogievskiy

Am 16.01.2021 um 14:44 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 18 lines

Add TestRunner class, which will run tests in a new python iotests running framework.

There are some differences with current ./check behavior, most significant are:
- Consider all tests self-executable, just run them, don't run python by hand.
- Elapsed time is cached in json file
- Elapsed time precision increased a bit
- use python difflib instead of "diff -w", to ignore spaces at line ends strip lines by hand. Do not ignore other spaces.

---
 tests/qemu-iotests/testrunner.py | 344 +++++++++++++++++++++++++++++++
 1 file changed, 344 insertions(+)
 create mode 100644 tests/qemu-iotests/testrunner.py

tests/qemu-iotests/testrunner.py line 118

Quoted 4 lines
tests/qemu-iotests/testrunner.py new file +4 -0
+TestResult = collections.namedtuple(
+    'TestResult',
+    ['status', 'description', 'elapsed', 'diff', 'casenotrun'],
+    defaults=('', '', '', ''))

defaults was only introduced in Python 3.7, it seems.

Why not use a normal class? I don't think we need the elements to be iterable or indexable?

tests/qemu-iotests/testrunner.py line 126

Quoted 7 lines
tests/qemu-iotests/testrunner.py new file +7 -0
+
+class TestRunner(AbstractContextManager['TestRunner']):
+    def __init__(self, env: TestEnv, makecheck: bool = False) -> None:
+        self.env = env
+        self.test_run_env = self.env.get_env()
+        if 'MALLOC_PERTURB_' not in os.environ and \
+                'MALLOC_PERTURB_' not in self.test_run_env:

'MALLOC_PERTURB_' is not in TestEnv.env_variables, so it will never be in self.test_run_env here.

tests/qemu-iotests/testrunner.py line 135

Quoted 9 lines
tests/qemu-iotests/testrunner.py new file +9 -0
+            x = random.randrange(1, 255)
+            self.test_run_env['MALLOC_PERTURB_'] = str(x)
+
+        self.makecheck = makecheck
+
+        self.last_elapsed = LastElapsedTime('.last-elapsed-cache', env)
+
+    def __enter__(self) -> 'TestRunner':
+        # pylint: disable=attribute-defined-outside-init

You can avoid this by declaring the attribute in __init__ without initialising it yet:

self._stack: contextlib.ExitStack

tests/qemu-iotests/testrunner.py line 140

Quoted 5 lines
tests/qemu-iotests/testrunner.py new file +5 -0
+        self._stack = contextlib.ExitStack()
+        self._stack.enter_context(self.env)
+        self._stack.enter_context(self.last_elapsed)
+        self._stack.enter_context(savetty())
+        return self

Kevin

#27 lore raw

Re: [PATCH v7 09/11] iotests: add testrunner.py

Quoted 19 lines · Kevin Wolf

Am 16.01.2021 um 14:44 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 17 lines

Add TestRunner class, which will run tests in a new python iotests running framework.

There are some differences with current ./check behavior, most significant are:
- Consider all tests self-executable, just run them, don't run python by hand.
- Elapsed time is cached in json file
- Elapsed time precision increased a bit
- use python difflib instead of "diff -w", to ignore spaces at line ends strip lines by hand. Do not ignore other spaces.

---
  tests/qemu-iotests/testrunner.py | 344 +++++++++++++++++++++++++++++++
  1 file changed, 344 insertions(+)

create mode 100644 tests/qemu-iotests/testrunner.py

Quoted 6 lines

tests/qemu-iotests/testrunner.py line 118

Quoted 4 lines
tests/qemu-iotests/testrunner.py new file +4 -0
+TestResult = collections.namedtuple(
+    'TestResult',
+    ['status', 'description', 'elapsed', 'diff', 'casenotrun'],
+    defaults=('', '', '', ''))

defaults was only introduced in Python 3.7, it seems.

hmm, yes

Quoted 2 lines

Why not use a normal class

OK

Quoted 11 lines

tests/qemu-iotests/testrunner.py line 126

Quoted 8 lines
tests/qemu-iotests/testrunner.py new file +7 -0
+
+class TestRunner(AbstractContextManager['TestRunner']):
+    def __init__(self, env: TestEnv, makecheck: bool = False) -> None:
+        self.env = env
+        self.test_run_env = self.env.get_env()
+        if 'MALLOC_PERTURB_' not in os.environ and \
+                'MALLOC_PERTURB_' not in self.test_run_env:

'MALLOC_PERTURB_' is not in TestEnv.env_variables, so it will never be in self.test_run_env here.

Hmm, right..

Quoted 3 lines

tests/qemu-iotests/testrunner.py line 128

Quoted 3 lines
tests/qemu-iotests/testrunner.py new file +2 -0
+            x = random.randrange(1, 255)
+            self.test_run_env['MALLOC_PERTURB_'] = str(x)

I don't remember why this is in TestRunner and not in TestEnv. May be better to move this logic to TestEnv.

Quoted 11 lines

tests/qemu-iotests/testrunner.py line 135

Quoted 7 lines
tests/qemu-iotests/testrunner.py new file +7 -0
+
+        self.makecheck = makecheck
+
+        self.last_elapsed = LastElapsedTime('.last-elapsed-cache', env)
+
+    def __enter__(self) -> 'TestRunner':
+        # pylint: disable=attribute-defined-outside-init

You can avoid this by declaring the attribute in __init__ without initialising it yet:

Quoted 1 line

self._stack: contextlib.ExitStack

OK

Quoted 9 lines

tests/qemu-iotests/testrunner.py line 140

Quoted 6 lines
tests/qemu-iotests/testrunner.py new file +5 -0
+        self._stack = contextlib.ExitStack()
+        self._stack.enter_context(self.env)
+        self._stack.enter_context(self.last_elapsed)
+        self._stack.enter_context(savetty())
+        return self

Kevin

Signature · 2 lines
Best regards,
Vladimir
#28 lore raw

Re: [PATCH v7 09/11] iotests: add testrunner.py

In reply to Vladimir Sementsov-Ogievskiy

Am 16.01.2021 um 14:44 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 12 lines

Add TestRunner class, which will run tests in a new python iotests running framework.

There are some differences with current ./check behavior, most significant are:
- Consider all tests self-executable, just run them, don't run python by hand.
- Elapsed time is cached in json file
- Elapsed time precision increased a bit
- use python difflib instead of "diff -w", to ignore spaces at line ends strip lines by hand. Do not ignore other spaces.

Quoted 1 line

Sorry for sending so many individual mails, but I'm running into things as I actually run the code during the review of the final 'check' rewrite.

I get this output for a failing test case:

    001   fail       [15:44:41] [15:44:42]   0.5s   (last: 13.2s) output mismatch (see {f_bad})
    --- /home/kwolf/source/qemu/tests/qemu-iotests/001.out

    +++ 001.out.bad

    @@ -1,4 +1,4 @@

    -QA output created by 001
    +Broken QA output created by 001
     Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728

     == reading whole image ==
    Failures: 001

Somehow we get additional empty lines in the diff header that shouldn't be there. Not sure where it comes from. Especially the empty line after @@ even corrupts the patch if you try to apply it.

The other problem is obvious enough:

tests/qemu-iotests/testrunner.py line 271

Quoted 4 lines
tests/qemu-iotests/testrunner.py new file +4 -0
+        diff = file_diff(str(f_reference), str(f_bad))
+        if diff:
+            return TestResult(status='fail', elapsed=elapsed,
+                              description='output mismatch (see {f_bad})',

This needs to be an f-string.

Kevin

#29 lore raw

Re: [PATCH v7 09/11] iotests: add testrunner.py

Quoted 13 lines · Kevin Wolf

Am 16.01.2021 um 14:44 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 12 lines

Add TestRunner class, which will run tests in a new python iotests running framework.

There are some differences with current ./check behavior, most significant are:
- Consider all tests self-executable, just run them, don't run python by hand.
- Elapsed time is cached in json file
- Elapsed time precision increased a bit
- use python difflib instead of "diff -w", to ignore spaces at line ends strip lines by hand. Do not ignore other spaces.

Quoted 3 lines
Quoted 1 line

Sorry for sending so many individual mails,

Not a problem

Quoted 20 lines

but I'm running into things
as I actually run the code during the review of the final 'check' rewrite.

I get this output for a failing test case:

     001   fail       [15:44:41] [15:44:42]   0.5s   (last: 13.2s) output mismatch (see {f_bad})
     --- /home/kwolf/source/qemu/tests/qemu-iotests/001.out

     +++ 001.out.bad

     @@ -1,4 +1,4 @@

     -QA output created by 001
     +Broken QA output created by 001
      Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728

      == reading whole image ==
     Failures: 001
Quoted 3 lines

Somehow we get additional empty lines in the diff header that shouldn't be there. Not sure where it comes from. Especially the empty line after @@ even corrupts the patch if you try to apply it.

Looking at it.

Seems the problem is that difflib adds '\n' for lines that it generates, but not for lines passed to it as lines list. And then I work with final lines list like there no embedded '\n' at all.

this helps:

tests/qemu-iotests/testrunner.py +3 -1
@@ -50,7 +50,9 @@ def file_diff(file1: str, file2: str) -> List[str]:
         # and use strict diff here!
         seq1 = [line.rstrip() for line in f1]
         seq2 = [line.rstrip() for line in f2]
-        return list(difflib.unified_diff(seq1, seq2, file1, file2))
+        res = [line.rstrip()
+               for line in difflib.unified_diff(seq1, seq2, file1, file2)]
+        return res
 
 
 
Quoted 12 lines

The other problem is obvious enough:

Review

Quoted 4 lines
(quoted diff) +4 -0
+        diff = file_diff(str(f_reference), str(f_bad))
+        if diff:
+            return TestResult(status='fail', elapsed=elapsed,
+                              description='output mismatch (see {f_bad})',

This needs to be an f-string.

Kevin

Signature · 2 lines
Best regards,
Vladimir
#30 lore raw

PATCH v7 10/11 iotests: rewrite check into python

In reply to Vladimir Sementsov-Ogievskiy

Just use classes introduced in previous three commits. Behavior difference is described in these three commits.

Drop group file, as it becomes unused.

Drop common.env: now check is in python, and for tests we use same python interpreter that runs the check itself. Use build environment PYTHON in check-block instead, to keep "make check" use the same python.

---
 Makefile                         |    1 -
 tests/check-block.sh             |    2 +-
 tests/qemu-iotests/check         | 1085 ++++--------------------------
 tests/qemu-iotests/common.env.in |    3 -
 tests/qemu-iotests/group         |  321 ---------
 tests/qemu-iotests/meson.build   |    3 -
 6 files changed, 120 insertions(+), 1295 deletions(-)
 delete mode 100644 tests/qemu-iotests/common.env.in
 delete mode 100644 tests/qemu-iotests/group
Makefile +0 -1
index fb9923ff22..2134b5147c 100644
@@ -236,7 +236,6 @@ distclean: clean
 	rm -f config-host.mak config-host.h*
 	rm -f tests/tcg/config-*.mak
 	rm -f config-all-disas.mak config.status
-	rm -f tests/qemu-iotests/common.env
 	rm -f roms/seabios/config.mak roms/vgabios/config.mak
 	rm -f qemu-plugins-ld.symbols qemu-plugins-ld64.symbols
 	rm -f *-config-target.h *-config-devices.mak *-config-devices.h
tests/check-block.sh +1 -1
index fb4c1baae9..26eb1c0a9b 100755
@@ -69,7 +69,7 @@ export QEMU_CHECK_BLOCK_AUTO=1
 
 ret=0
 for fmt in $format_list ; do
-    ./check -makecheck -$fmt $group || ret=1
+    ${PYTHON} ./check -makecheck -$fmt $group || ret=1
 done
 
 exit $ret
tests/qemu-iotests/check +119 -966

1,093 lines not on this page · raw message

tests/qemu-iotests/common.env.in deleted +0 -3
deleted file mode 100644
index e565cdf40c..0000000000
@@ -1,3 +0,0 @@
-# Automatically generated by configure - do not modify
-
-export PYTHON='@PYTHON@'
tests/qemu-iotests/group deleted +0 -321

321 lines not on this page · raw message

tests/qemu-iotests/meson.build +0 -3
index 26658ce25c..67aed1e492 100644
@@ -3,6 +3,3 @@ if 'CONFIG_LINUX' in config_host
 else
     socket_scm_helper = []
 endif
-configure_file(output: 'common.env',
-               input: files('common.env.in'),
-               configuration: {'PYTHON': python.full_path()})
Signature · 1 line
2.29.2
Tested-by on v7 10/11
#31 lore raw

Re: [PATCH v7 10/11] iotests: rewrite check into python

Quoted 15 lines · Vladimir Sementsov-Ogievskiy

Just use classes introduced in previous three commits. Behavior difference is described in these three commits.

Drop group file, as it becomes unused.

Drop common.env: now check is in python, and for tests we use same python interpreter that runs the check itself. Use build environment PYTHON in check-block instead, to keep "make check" use the same python.

---

+++ b/tests/qemu-iotests/check

tests/qemu-iotests/check line 5

Quoted 8 lines
tests/qemu-iotests/check +4 -3
@@ -1,7 +1,8 @@
-#!/usr/bin/env bash
+#!/usr/bin/env python3
 #
-# Copyright (C) 2009 Red Hat, Inc.
-# Copyright (c) 2000-2002,2006 Silicon Graphics, Inc.  All Rights Reserved.
+# Configure environment and run group of tests in it.
+#
+# Copyright (c) 2020-2021 Virtuozzo International GmbH

Normally dropping old copyrights is suspicious, but as this is a complete rewrite, it makes sense here.

Quoted 14 lines
tests/qemu-iotests/check +13 -1
-exit
+import os
+import sys
+import argparse
+from findtests import TestFinder
+from testenv import TestEnv
+from testrunner import TestRunner
+
+
+def make_argparser() -> argparse.ArgumentParser:
+    p = argparse.ArgumentParser(description="Test run options")
+
+    p.add_argument('-n', '--dry-run', action='store_true',
+                   help='show me, do not run tests')

tests/qemu-iotests/check line 41

Quoted 10 lines
tests/qemu-iotests/check +10 -0
+    p.add_argument('-makecheck', action='store_true',
+                   help='pretty print output for make check')
+
+    p.add_argument('-d', dest='debug', action='store_true', help='debug')
+    p.add_argument('-misalign', action='store_true',
+                   help='misalign memory allocations')
+
+    g_env = p.add_argument_group('test environment options')
+    mg = g_env.add_mutually_exclusive_group()
+    # We don't set default for cachemode, as we need to distinguish dafult

default

Quoted 14 lines
tests/qemu-iotests/check +14 -0
+    # from user input later.
+    mg.add_argument('-nocache', dest='cachemode', action='store_const',
+                    const='none', help='set cache mode "none" (O_DIRECT), '
+                    'sets CACHEMODE environment variable')
+    mg.add_argument('-c', dest='cachemode',
+                    help='sets CACHEMODE environment variable')
+
+    g_env.add_argument('-i', dest='aiomode', default='threads',
+                       help='sets AIOMODE environment variable')
+
+    p.set_defaults(imgfmt='raw', imgproto='file')
+
+    format_list = ['raw', 'bochs', 'cloop', 'parallels', 'qcow', 'qcow2',
+                   'qed', 'vdi', 'vpc', 'vhdx', 'vmdk', 'luks', 'dmg']

tests/qemu-iotests/check line 65

Quoted 10 lines
tests/qemu-iotests/check +10 -0
+    g_fmt = p.add_argument_group(
+        '  image format options',
+        'The following options set the IMGFMT environment variable. '
+        'At most one choice is allowed, default is "raw"')
+    mg = g_fmt.add_mutually_exclusive_group()
+    for fmt in format_list:
+        mg.add_argument('-' + fmt, dest='imgfmt', action='store_const',
+                        const=fmt, help=f'test {fmt}')
+
+    protocol_list = ['file', 'rbd', 'sheepdoc', 'nbd', 'ssh', 'nfs',

sheepdog

Quoted 29 lines
tests/qemu-iotests/check +29 -0
+                     'fuse']
+    g_prt = p.add_argument_group(
+        '  image protocol options',
+        'The following options set the IMGPROTO environment variable. '
+        'At most one choice is allowed, default is "file"')
+    mg = g_prt.add_mutually_exclusive_group()
+    for prt in protocol_list:
+        mg.add_argument('-' + prt, dest='imgproto', action='store_const',
+                        const=prt, help=f'test {prt}')
+
+    g_bash = p.add_argument_group('bash tests options',
+                                  'The following options are ignored by '
+                                  'python tests.')
+    # TODO: make support for the following options in iotests.py
+    g_bash.add_argument('-o', dest='imgopts',
+                        help='options to pass to qemu-img create/convert, '
+                        'sets IMGOPTS environment variable')
+    g_bash.add_argument('-valgrind', dest='VALGRIND_QEMU',
+                        action='store_const', const='y',
+                        help='use valgrind, sets VALGRIND_QEMU environment '
+                        'variable')
+
+    g_sel = p.add_argument_group('test selecting options',
+                                 'The following options specify test set '
+                                 'to run.')
+    g_sel.add_argument('-g', '--groups', metavar='group1,...',
+                       help='include tests from these groups')
+    g_sel.add_argument('-x', '--exclude-groups', metavar='group1,...',
+                       help='exclude tests from these groups')

tests/qemu-iotests/check line 104

Quoted 10 lines
tests/qemu-iotests/check +10 -0
+    g_sel.add_argument('--start-from', metavar='TEST',
+                       help='Start from specified test: make sorted sequence '
+                       'of tests as usual and then drop tests from the first '
+                       'one to TEST (not inclusive). This may be used to '
+                       'rerun failed ./check command, starting from the '
+                       'middle of the process.')
+    g_sel.add_argument('tests', metavar='TEST_FILES', nargs='*',
+                       help='tests to run')
+
+    return p

The change to drop ranges and instead use --start-from makes sense (you can't do ranges of named files); it will take a minor adjustment to my keyboarding habits to get used to, but I don't see it as a show-stopper as iotests are for developers and not promised to be a stable interface.

Quoted 19 lines
tests/qemu-iotests/check +19 -0
+
+
+if __name__ == '__main__':
+    args = make_argparser().parse_args()
+
+    env = TestEnv(imgfmt=args.imgfmt, imgproto=args.imgproto,
+                  aiomode=args.aiomode, cachemode=args.cachemode,
+                  imgopts=args.imgopts, misalign=args.misalign,
+                  debug=args.debug)
+
+    testfinder = TestFinder(test_dir=env.source_iotests)
+
+    groups = args.groups.split(',') if args.groups else None
+    x_groups = args.exlude_groups.split(',') if args.exclude_groups else None
+
+    try:
+        tests = testfinder.find_tests(groups=groups, exclude_groups=x_groups,
+                                      tests=args.tests,
+                                      start_from=args.start_from)

tests/qemu-iotests/check line 133

Quoted 10 lines
tests/qemu-iotests/check +10 -0
+        if not tests:
+            raise ValueError('No tests selected')
+    except ValueError as e:
+        sys.exit(e)
+
+    if args.dry_run:
+        print('\n'.join(tests))
+    else:
+        with TestRunner(env, args.makecheck) as tr:
+            tr.run_tests([os.path.join(env.source_iotests, t) for t in tests])

Other than the typo, this looks like a fairly straightforward conversion of the old shell code, but at present, I'm only comfortable giving:

(of course assuming you fix the typo that broke ./check -sheepdog)

The longer we sit on this series, the more merge conflicts it will introduce (new tests need to add their own #group header line); the changes to fix those conflicts should be obvious, but I hope we get another reviewer soon. I'm wondering if I should take the second half of this series (since I already sent the pull request for the first half since it touched NBD tests), or let Max take it.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#32 lore raw

Re: [PATCH v7 10/11] iotests: rewrite check into python

In reply to Vladimir Sementsov-Ogievskiy
Quoted 11 lines · Vladimir Sementsov-Ogievskiy

Just use classes introduced in previous three commits. Behavior difference is described in these three commits.

Drop group file, as it becomes unused.

Drop common.env: now check is in python, and for tests we use same python interpreter that runs the check itself. Use build environment PYTHON in check-block instead, to keep "make check" use the same python.

squash-in to support group.local:

@@ -117,6 +117,13 @@ if __name__ == '__main__':
groups = args.groups.split(',') if args.groups else None x_groups = args.exlude_groups.split(',') if args.exclude_groups else None

+ group_local = os.path.join(env.source_iotests, 'group.local')
+ if os.path.isfile(group_local):
+ try:
+ testfinder.add_group_file(group_local)
+ except ValueError as e:
+ sys.exit(f"Filed to parse group file '{group_local}': {e}")
+
try:
tests = testfinder.find_tests(groups=groups, exclude_groups=x_groups,
tests=args.tests,

Signature · 2 lines
Best regards,
Vladimir
#33 lore raw

Re: [PATCH v7 10/11] iotests: rewrite check into python

In reply to Vladimir Sementsov-Ogievskiy

Am 16.01.2021 um 14:44 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 17 lines

Just use classes introduced in previous three commits. Behavior difference is described in these three commits.

Drop group file, as it becomes unused.

Drop common.env: now check is in python, and for tests we use same python interpreter that runs the check itself. Use build environment PYTHON in check-block instead, to keep "make check" use the same python.

tests/check-block.sh +0 -0
index fb4c1baae9..26eb1c0a9b 100755

tests/check-block.sh export QEMU_CHECK_BLOCK_AUTO=1 · line 73

Quoted 6 lines
tests/check-block.sh +1 -1
@@ -69,7 +69,7 @@ export QEMU_CHECK_BLOCK_AUTO=1
 
 ret=0
 for fmt in $format_list ; do
-    ./check -makecheck -$fmt $group || ret=1
+    ${PYTHON} ./check -makecheck -$fmt $group || ret=1
 done

When I add an echo to print that command line, it seems that ${PYTHON} is empty for me. Is this expected?

Quoted 5 lines
tests/check-block.sh +0 -0
 exit $ret
tests/qemu-iotests/check +0 -0
index 952762d5ed..914321806a 100755

[...]

tests/qemu-iotests/check line 537

Quoted 7 lines
tests/qemu-iotests/check +0 -7
-
-# Set qemu-io cache mode with $CACHEMODE we have
-QEMU_IO_OPTIONS="$QEMU_IO_OPTIONS --cache $CACHEMODE"
-# Set qemu-io aio mode with $AIOMODE we have
-QEMU_IO_OPTIONS="$QEMU_IO_OPTIONS --aio $AIOMODE"
-
-QEMU_IO_OPTIONS_NO_FMT="$QEMU_IO_OPTIONS"

TestEnv sets the attribute, but forgets to include QEMU_IO_OPTIONS_NO_FMT in env_variables, so I think it's not actually exported.

tests/qemu-iotests/check line 545

Quoted 8 lines
tests/qemu-iotests/check +0 -8
-if [ "$IMGOPTSSYNTAX" != "true" ]; then
-    QEMU_IO_OPTIONS="$QEMU_IO_OPTIONS -f $IMGFMT"
-fi
-
-# Set default options for qemu-img create -o if they were not specified
-if [ "$IMGFMT" == "qcow2" ] && ! (echo "$IMGOPTS" | grep "compat=" > /dev/null); then
-    IMGOPTS=$(_optstr_add "$IMGOPTS" "compat=1.1")
-fi

You dropped this one, which makes sense because it's already the default in 'qemu-img create'. Maybe the commit message could mention it as a difference.

Quoted 45 lines
tests/qemu-iotests/check +0 -45
-if [ "$IMGFMT" == "luks" ] && ! (echo "$IMGOPTS" | grep "iter-time=" > /dev/null); then
-    IMGOPTS=$(_optstr_add "$IMGOPTS" "iter-time=10")
-fi
-if [ "$IMGFMT" == "vmdk" ] && ! (echo "$IMGOPTS" | grep "zeroed_grain=" > /dev/null); then
-    IMGOPTS=$(_optstr_add "$IMGOPTS" "zeroed_grain=on")
-fi
-
-if [ -z "$SAMPLE_IMG_DIR" ]; then
-        SAMPLE_IMG_DIR="$source_iotests/sample_images"
-fi
-
-export TEST_DIR
-export SOCK_DIR
-export SAMPLE_IMG_DIR
-
-if [ -s $tmp.list ]
-then
-    # found some valid test numbers ... this is good
-    :
-else
-    if $have_test_arg
-    then
-        # had test numbers, but none in group file ... do nothing
-        touch $tmp.list
-    else
-        # no test numbers, do everything from group file
-        sed -n -e '/^[0-9][0-9][0-9]*/s/^\([0-9]*\).*/\1/p' <"$source_iotests/group" >$tmp.list
-    fi
-fi
-
-# should be sort -n, but this did not work for Linux when this
-# was ported from IRIX
-#
-list=$(sort $tmp.list)
-rm -f $tmp.list $tmp.tmp $tmp.sed
-
-if [ -z "$QEMU_PROG" ]
-then
-    if [ -x "$build_iotests/qemu" ]; then
-        export QEMU_PROG="$build_iotests/qemu"
-    elif [ -x "$build_root/qemu-system-${qemu_arch}" ]; then
-        export QEMU_PROG="$build_root/qemu-system-${qemu_arch}"
-    else
-        pushd "$build_root" > /dev/null
-        for binary in qemu-system-*

tests/qemu-iotests/check line 600

Quoted 10 lines
tests/qemu-iotests/check +0 -10
-        do
-            if [ -x "$binary" ]
-            then
-                export QEMU_PROG="$build_root/$binary"
-                break
-            fi
-        done
-        popd > /dev/null
-        [ "$QEMU_PROG" = "" ] && _init_error "qemu not found"
-    fi

I think this else branch is kind of important (if there is no system emulator binary for the host architecture, find _any_ system emulator binary that was built). I can't find its equivalent in the new code.

Quoted 60 lines
tests/qemu-iotests/check +0 -60
-fi
-export QEMU_PROG="$(type -p "$QEMU_PROG")"
-
-export QEMU_OPTIONS="-nodefaults -display none -accel qtest"
-case "$QEMU_PROG" in
-    *qemu-system-arm|*qemu-system-aarch64)
-        export QEMU_OPTIONS="$QEMU_OPTIONS -machine virt"
-        ;;
-    *qemu-system-avr)
-        export QEMU_OPTIONS="$QEMU_OPTIONS -machine mega2560"
-        ;;
-    *qemu-system-rx)
-        export QEMU_OPTIONS="$QEMU_OPTIONS -machine gdbsim-r5f562n8"
-        ;;
-    *qemu-system-tricore)
-        export QEMU_OPTIONS="-$QEMU_OPTIONS -machine tricore_testboard"
-        ;;
-esac
-
-if [ -z "$QEMU_IMG_PROG" ]; then
-    if [ -x "$build_iotests/qemu-img" ]; then
-        export QEMU_IMG_PROG="$build_iotests/qemu-img"
-    elif [ -x "$build_root/qemu-img" ]; then
-        export QEMU_IMG_PROG="$build_root/qemu-img"
-    else
-        _init_error "qemu-img not found"
-    fi
-fi
-export QEMU_IMG_PROG="$(type -p "$QEMU_IMG_PROG")"
-
-if [ -z "$QEMU_IO_PROG" ]; then
-    if [ -x "$build_iotests/qemu-io" ]; then
-        export QEMU_IO_PROG="$build_iotests/qemu-io"
-    elif [ -x "$build_root/qemu-io" ]; then
-        export QEMU_IO_PROG="$build_root/qemu-io"
-    else
-        _init_error "qemu-io not found"
-    fi
-fi
-export QEMU_IO_PROG="$(type -p "$QEMU_IO_PROG")"
-
-if [ -z $QEMU_NBD_PROG ]; then
-    if [ -x "$build_iotests/qemu-nbd" ]; then
-        export QEMU_NBD_PROG="$build_iotests/qemu-nbd"
-    elif [ -x "$build_root/qemu-nbd" ]; then
-        export QEMU_NBD_PROG="$build_root/qemu-nbd"
-    else
-        _init_error "qemu-nbd not found"
-    fi
-fi
-export QEMU_NBD_PROG="$(type -p "$QEMU_NBD_PROG")"
-
-if [ -z "$QSD_PROG" ]; then
-    if [ -x "$build_iotests/qemu-storage-daemon" ]; then
-        export QSD_PROG="$build_iotests/qemu-storage-daemon"
-    elif [ -x "$build_root/storage-daemon/qemu-storage-daemon" ]; then
-        export QSD_PROG="$build_root/storage-daemon/qemu-storage-daemon"
-    else
-        _init_error "qemu-storage-daemon not found"
-    fi

tests/qemu-iotests/check line 670

Quoted 10 lines
tests/qemu-iotests/check +0 -10
-fi
-export QSD_PROG="$(type -p "$QSD_PROG")"
-
-if [ -x "$build_iotests/socket_scm_helper" ]
-then
-    export SOCKET_SCM_HELPER="$build_iotests/socket_scm_helper"
-fi
-
-python_usable=false
-if $PYTHON -c 'import sys; sys.exit(0 if sys.version_info >= (3,6) else 1)'

This is not checked any more. Probably okay because building QEMU already requires 3.6 now.

tests/qemu-iotests/check line 673

Quoted 3 lines
tests/qemu-iotests/check +0 -3
-then
-    # Our python framework also requires virtio-blk
-    if "$QEMU_PROG" -M none -device help | grep -q virtio-blk >/dev/null 2>&1

It looks like we lost this check, too? This will probably cause failures on more exotic architectures.

Quoted 8 lines
tests/qemu-iotests/check +0 -8
-    then
-        python_usable=true
-    else
-        python_unusable_because="Missing virtio-blk in QEMU binary"
-    fi
-else
-    python_unusable_because="Unsupported Python version"
-fi

[...]

Quoted 58 lines
tests/qemu-iotests/check +58 -0
+import os
+import sys
+import argparse
+from findtests import TestFinder
+from testenv import TestEnv
+from testrunner import TestRunner
+
+
+def make_argparser() -> argparse.ArgumentParser:
+    p = argparse.ArgumentParser(description="Test run options")
+
+    p.add_argument('-n', '--dry-run', action='store_true',
+                   help='show me, do not run tests')
+    p.add_argument('-makecheck', action='store_true',
+                   help='pretty print output for make check')
+
+    p.add_argument('-d', dest='debug', action='store_true', help='debug')
+    p.add_argument('-misalign', action='store_true',
+                   help='misalign memory allocations')
+
+    g_env = p.add_argument_group('test environment options')
+    mg = g_env.add_mutually_exclusive_group()
+    # We don't set default for cachemode, as we need to distinguish dafult
+    # from user input later.
+    mg.add_argument('-nocache', dest='cachemode', action='store_const',
+                    const='none', help='set cache mode "none" (O_DIRECT), '
+                    'sets CACHEMODE environment variable')
+    mg.add_argument('-c', dest='cachemode',
+                    help='sets CACHEMODE environment variable')
+
+    g_env.add_argument('-i', dest='aiomode', default='threads',
+                       help='sets AIOMODE environment variable')
+
+    p.set_defaults(imgfmt='raw', imgproto='file')
+
+    format_list = ['raw', 'bochs', 'cloop', 'parallels', 'qcow', 'qcow2',
+                   'qed', 'vdi', 'vpc', 'vhdx', 'vmdk', 'luks', 'dmg']
+    g_fmt = p.add_argument_group(
+        '  image format options',
+        'The following options set the IMGFMT environment variable. '
+        'At most one choice is allowed, default is "raw"')
+    mg = g_fmt.add_mutually_exclusive_group()
+    for fmt in format_list:
+        mg.add_argument('-' + fmt, dest='imgfmt', action='store_const',
+                        const=fmt, help=f'test {fmt}')
+
+    protocol_list = ['file', 'rbd', 'sheepdoc', 'nbd', 'ssh', 'nfs',
+                     'fuse']
+    g_prt = p.add_argument_group(
+        '  image protocol options',
+        'The following options set the IMGPROTO environment variable. '
+        'At most one choice is allowed, default is "file"')
+    mg = g_prt.add_mutually_exclusive_group()
+    for prt in protocol_list:
+        mg.add_argument('-' + prt, dest='imgproto', action='store_const',
+                        const=prt, help=f'test {prt}')
+
+    g_bash = p.add_argument_group('bash tests options',

tests/qemu-iotests/check line 86

Quoted 10 lines
tests/qemu-iotests/check +10 -0
+                                  'The following options are ignored by '
+                                  'python tests.')
+    # TODO: make support for the following options in iotests.py
+    g_bash.add_argument('-o', dest='imgopts',
+                        help='options to pass to qemu-img create/convert, '
+                        'sets IMGOPTS environment variable')
+    g_bash.add_argument('-valgrind', dest='VALGRIND_QEMU',
+                        action='store_const', const='y',
+                        help='use valgrind, sets VALGRIND_QEMU environment '
+                        'variable')

-valgrind is parsed, but the result is never used.

Quoted 21 lines
tests/qemu-iotests/check +21 -0
+    g_sel = p.add_argument_group('test selecting options',
+                                 'The following options specify test set '
+                                 'to run.')
+    g_sel.add_argument('-g', '--groups', metavar='group1,...',
+                       help='include tests from these groups')
+    g_sel.add_argument('-x', '--exclude-groups', metavar='group1,...',
+                       help='exclude tests from these groups')
+    g_sel.add_argument('--start-from', metavar='TEST',
+                       help='Start from specified test: make sorted sequence '
+                       'of tests as usual and then drop tests from the first '
+                       'one to TEST (not inclusive). This may be used to '
+                       'rerun failed ./check command, starting from the '
+                       'middle of the process.')
+    g_sel.add_argument('tests', metavar='TEST_FILES', nargs='*',
+                       help='tests to run')
+
+    return p
+
+
+if __name__ == '__main__':
+    args = make_argparser().parse_args()

tests/qemu-iotests/check line 118

Quoted 10 lines
tests/qemu-iotests/check +10 -0
+
+    env = TestEnv(imgfmt=args.imgfmt, imgproto=args.imgproto,
+                  aiomode=args.aiomode, cachemode=args.cachemode,
+                  imgopts=args.imgopts, misalign=args.misalign,
+                  debug=args.debug)
+
+    testfinder = TestFinder(test_dir=env.source_iotests)
+
+    groups = args.groups.split(',') if args.groups else None
+    x_groups = args.exlude_groups.split(',') if args.exclude_groups else None

s/exlude_groups/exclude_groups/

Here the squashed in part:

Quoted 6 lines

group_local = os.path.join(env.source_iotests, 'group.local') if os.path.isfile(group_local):
try:
testfinder.add_group_file(group_local)
except ValueError as e:
sys.exit(f"Filed to parse group file '{group_local}': {e}")

s/Filed/Failed/

Quoted 5 lines
tests/qemu-iotests/check +5 -0
+
+    try:
+        tests = testfinder.find_tests(groups=groups, exclude_groups=x_groups,
+                                      tests=args.tests,
+                                      start_from=args.start_from)

tests/qemu-iotests/check line 133

Quoted 10 lines
tests/qemu-iotests/check +10 -0
+        if not tests:
+            raise ValueError('No tests selected')
+    except ValueError as e:
+        sys.exit(e)
+
+    if args.dry_run:
+        print('\n'.join(tests))
+    else:
+        with TestRunner(env, args.makecheck) as tr:
+            tr.run_tests([os.path.join(env.source_iotests, t) for t in tests])

Maybe it would be worth catching KeyboardInterrupt somewhere? I always get frightened when I press Ctrl-C and get a stack trace. :-)

Could even be done in the TestRunner, so that the result summary can still be printed for those tests that were already completed.

Kevin

#34 lore raw

Re: [PATCH v7 10/11] iotests: rewrite check into python

Quoted 19 lines · Kevin Wolf

Am 16.01.2021 um 14:44 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 18 lines

Just use classes introduced in previous three commits. Behavior difference is described in these three commits.

Drop group file, as it becomes unused.

Drop common.env: now check is in python, and for tests we use same python interpreter that runs the check itself. Use build environment PYTHON in check-block instead, to keep "make check" use the same python.

tests/check-block.sh +0 -0
index fb4c1baae9..26eb1c0a9b 100755
@@ -69,7 +69,7 @@ export QEMU_CHECK_BLOCK_AUTO=1
 
Quoted 8 lines

tests/check-block.sh export QEMU_CHECK_BLOCK_AUTO=1 · line 73

Quoted 5 lines
tests/check-block.sh +1 -1
  ret=0
  for fmt in $format_list ; do
-    ./check -makecheck -$fmt $group || ret=1
+    ${PYTHON} ./check -makecheck -$fmt $group || ret=1
  done

When I add an echo to print that command line, it seems that ${PYTHON} is empty for me. Is this expected?

It seems to be defined defined when called from make check. Did you just call check-block directly?

It's not intentional, but I think it's OK: if PYTHON is not defined let's just execute check as self-executable. And for make-check PYTHON is defined and correct python is used.

Quoted 6 lines
Quoted 6 lines
tests/check-block.sh +0 -0
  exit $ret
tests/qemu-iotests/check +0 -0
index 952762d5ed..914321806a 100755

[..]

Quoted 10 lines

tests/qemu-iotests/check line 600

Quoted 9 lines
tests/qemu-iotests/check +0 -9
-            if [ -x "$binary" ]
-            then
-                export QEMU_PROG="$build_root/$binary"
-                break
-            fi
-        done
-        popd > /dev/null
-        [ "$QEMU_PROG" = "" ] && _init_error "qemu not found"
-    fi
Quoted 3 lines

I think this else branch is kind of important (if there is no system emulator binary for the host architecture, find _any_ system emulator binary that was built). I can't find its equivalent in the new code.

Hmm, I decided testing "first found" emulator is strange.. It seems like we have several emulators and user don't care which would be tested?

Probably we should instead used qemu-system-* binary only if there is only one matching binary. And fail if there are many.

Signature · 2 lines
Best regards,
Vladimir
#35 lore raw

Re: [PATCH v7 10/11] iotests: rewrite check into python

Am 23.01.2021 um 16:08 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 28 lines
Quoted 13 lines · Kevin Wolf

Am 16.01.2021 um 14:44 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 11 lines

Just use classes introduced in previous three commits. Behavior difference is described in these three commits.

Drop group file, as it becomes unused.

Drop common.env: now check is in python, and for tests we use same python interpreter that runs the check itself. Use build environment PYTHON in check-block instead, to keep "make check" use the same python.

Quoted 13 lines

tests/check-block.sh export QEMU_CHECK_BLOCK_AUTO=1 · line 72

Quoted 10 lines
tests/check-block.sh +1 -1
index fb4c1baae9..26eb1c0a9b 100755
@@ -69,7 +69,7 @@ export QEMU_CHECK_BLOCK_AUTO=1
  ret=0
  for fmt in $format_list ; do
-    ./check -makecheck -$fmt $group || ret=1
+    ${PYTHON} ./check -makecheck -$fmt $group || ret=1
  done

When I add an echo to print that command line, it seems that ${PYTHON} is empty for me. Is this expected?

Quoted 2 lines

It seems to be defined defined when called from make check. Did you just call check-block directly?

D>

Quoted 3 lines

It's not intentional, but I think it's OK: if PYTHON is not defined let's just execute check as self-executable. And for make-check PYTHON is defined and correct python is used.

Hm, where does that happen in 'make check'? It seems the old makefiles were quite readable in comparison to what we have now...

Anyway, I think 'make check-block' should run just the block-specific subset of 'make check', without changing the behaviour of the remaining tests. Anything that can be started through make should respect the configured Python interpreter.

Quoted 18 lines
Quoted 5 lines
Quoted 5 lines
tests/check-block.sh +0 -0
  exit $ret
tests/qemu-iotests/check +0 -0
index 952762d5ed..914321806a 100755

[..]

Quoted 10 lines
Quoted 9 lines
tests/qemu-iotests/check +0 -9
-            if [ -x "$binary" ]
-            then
-                export QEMU_PROG="$build_root/$binary"
-                break
-            fi
-        done
-        popd > /dev/null
-        [ "$QEMU_PROG" = "" ] && _init_error "qemu not found"
-    fi
Quoted 7 lines
Quoted 3 lines

I think this else branch is kind of important (if there is no system emulator binary for the host architecture, find _any_ system emulator binary that was built). I can't find its equivalent in the new code.

Hmm, I decided testing "first found" emulator is strange.. It seems like we have several emulators and user don't care which would be tested?

Remember that we're mainly testing the block layer, which is the same in all qemu-system-* binaries anyway. So yes, any system emulator binary is good enough for many test cases, and certainly better than having no system emulator. Differences are only in the supported guest devices, which may cause some tests to be skipped.

If there are multiple binaries that we could use, we could change the way to select one instead of just the first one, e.g. by trying x86_64 first because this is what enables the largest set of tests.

But anything is better than failing with "qemu not found".

Quoted 2 lines

Probably we should instead used qemu-system-* binary only if there is only one matching binary. And fail if there are many.

No, 'make check' shouldn't fail because I built arm and ppc emulators on my x86_64 machine without also building a x86_64 emulator. (And I think this is a case that fails both with the actual patch under review and with your suggested change.)

Kevin

#36 lore raw

Re: [PATCH v7 10/11] iotests: rewrite check into python

Quoted 82 lines · Kevin Wolf

Am 23.01.2021 um 16:08 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 28 lines
Quoted 13 lines · Kevin Wolf

Am 16.01.2021 um 14:44 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 11 lines

Just use classes introduced in previous three commits. Behavior difference is described in these three commits.

Drop group file, as it becomes unused.

Drop common.env: now check is in python, and for tests we use same python interpreter that runs the check itself. Use build environment PYTHON in check-block instead, to keep "make check" use the same python.

Quoted 13 lines

tests/check-block.sh export QEMU_CHECK_BLOCK_AUTO=1 · line 72

Quoted 10 lines
tests/check-block.sh +1 -1
index fb4c1baae9..26eb1c0a9b 100755
@@ -69,7 +69,7 @@ export QEMU_CHECK_BLOCK_AUTO=1
   ret=0
   for fmt in $format_list ; do
-    ./check -makecheck -$fmt $group || ret=1
+    ${PYTHON} ./check -makecheck -$fmt $group || ret=1
   done

When I add an echo to print that command line, it seems that ${PYTHON} is empty for me. Is this expected?

Quoted 2 lines

It seems to be defined defined when called from make check. Did you just call check-block directly?

D>

Quoted 3 lines

It's not intentional, but I think it's OK: if PYTHON is not defined let's just execute check as self-executable. And for make-check PYTHON is defined and correct python is used.

Hm, where does that happen in 'make check'? It seems the old makefiles were quite readable in comparison to what we have now...

Anyway, I think 'make check-block' should run just the block-specific subset of 'make check', without changing the behaviour of the remaining tests. Anything that can be started through make should respect the configured Python interpreter.

Quoted 18 lines
Quoted 5 lines
Quoted 5 lines
tests/check-block.sh +0 -0
   exit $ret
tests/qemu-iotests/check +0 -0
index 952762d5ed..914321806a 100755

[..]

Quoted 10 lines
Quoted 9 lines
tests/qemu-iotests/check +0 -9
-            if [ -x "$binary" ]
-            then
-                export QEMU_PROG="$build_root/$binary"
-                break
-            fi
-        done
-        popd > /dev/null
-        [ "$QEMU_PROG" = "" ] && _init_error "qemu not found"
-    fi
Quoted 7 lines
Quoted 3 lines

I think this else branch is kind of important (if there is no system emulator binary for the host architecture, find _any_ system emulator binary that was built). I can't find its equivalent in the new code.

Hmm, I decided testing "first found" emulator is strange.. It seems like we have several emulators and user don't care which would be tested?

Remember that we're mainly testing the block layer, which is the same in all qemu-system-* binaries anyway. So yes, any system emulator binary is good enough for many test cases, and certainly better than having no system emulator. Differences are only in the supported guest devices, which may cause some tests to be skipped.

If there are multiple binaries that we could use, we could change the way to select one instead of just the first one, e.g. by trying x86_64 first because this is what enables the largest set of tests.

But anything is better than failing with "qemu not found".

Quoted 8 lines
Quoted 2 lines

Probably we should instead used qemu-system-* binary only if there is only one matching binary. And fail if there are many.

No, 'make check' shouldn't fail because I built arm and ppc emulators on my x86_64 machine without also building a x86_64 emulator. (And I think this is a case that fails both with the actual patch under review and with your suggested change.)

OK, I'll send a squash-in

Signature · 2 lines
Best regards,
Vladimir
#37 lore raw

PATCH v7 11/11 iotests: rename and move 169 and 199 tests

In reply to Vladimir Sementsov-Ogievskiy

Rename bitmaps migration tests and move them to tests subdirectory to demonstrate new human-friendly test naming.

---
 tests/qemu-iotests/{199 => tests/migrate-bitmaps-postcopy-test}   | 0
 .../{199.out => tests/migrate-bitmaps-postcopy-test.out}          | 0
 tests/qemu-iotests/{169 => tests/migrate-bitmaps-test}            | 0
 tests/qemu-iotests/{169.out => tests/migrate-bitmaps-test.out}    | 0
 4 files changed, 0 insertions(+), 0 deletions(-)
 rename tests/qemu-iotests/{199 => tests/migrate-bitmaps-postcopy-test} (100%)
 rename tests/qemu-iotests/{199.out => tests/migrate-bitmaps-postcopy-test.out} (100%)
 rename tests/qemu-iotests/{169 => tests/migrate-bitmaps-test} (100%)
 rename tests/qemu-iotests/{169.out => tests/migrate-bitmaps-test.out} (100%)
tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test renamed from tests/qemu-iotests/199 +0 -0
similarity index 100%
rename from tests/qemu-iotests/199
rename to tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test
tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test.out renamed from tests/qemu-iotests/199.out +0 -0
similarity index 100%
rename from tests/qemu-iotests/199.out
rename to tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test.out
tests/qemu-iotests/tests/migrate-bitmaps-test renamed from tests/qemu-iotests/169 +0 -0
similarity index 100%
rename from tests/qemu-iotests/169
rename to tests/qemu-iotests/tests/migrate-bitmaps-test
tests/qemu-iotests/tests/migrate-bitmaps-test.out renamed from tests/qemu-iotests/169.out +0 -0
similarity index 100%
rename from tests/qemu-iotests/169.out
rename to tests/qemu-iotests/tests/migrate-bitmaps-test.out
Signature · 1 line
2.29.2
#38 lore raw
In reply to Vladimir Sementsov-Ogievskiy
Quoted 8 lines · Vladimir Sementsov-Ogievskiy

Hi all!

These series has 3 goals:

- get rid of group file (to forget about rebase and in-list conflicts)
- introduce human-readable names for tests
- rewrite check into python

Quoted 6 lines

v7:
- fix wording and grammar
- satisfy python linters
- move argv interfaces all into one in new check script
- support '-n' == '--dry-run' option
- update check-block to run check with correct PYTHON

I'd really like a second reviewer on 7-11, but I'm queueing 1-6 on my NBD tree now.

Quoted 23 lines

findtests: - stop parsing test file after first '# group: ' line
- use parse_test_name in add_group_file()
- make new list instead of modifying parameter exclude_groups

testenv: - fix machine_map
- fix self.python

testrunner: use env.python to run python tests

Vladimir Sementsov-Ogievskiy (11):
iotests/277: use dot slash for nbd-fault-injector.py running iotests/303: use dot slash for qcow2.py running
iotests: fix some whitespaces in test output files
iotests: make tests executable
iotests/294: add shebang line
iotests: define group in each iotest
iotests: add findtests.py
iotests: add testenv.py
iotests: add testrunner.py
iotests: rewrite check into python
iotests: rename and move 169 and 199 tests

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#39 lore raw

Am 20.01.2021 um 21:52 hat Eric Blake geschrieben:

Quoted 16 lines · Vladimir Sementsov-Ogievskiy
Quoted 8 lines

Hi all!

These series has 3 goals:

- get rid of group file (to forget about rebase and in-list conflicts)
- introduce human-readable names for tests
- rewrite check into python

Quoted 6 lines

v7:
- fix wording and grammar
- satisfy python linters
- move argv interfaces all into one in new check script
- support '-n' == '--dry-run' option
- update check-block to run check with correct PYTHON

Quoted 2 lines

I'd really like a second reviewer on 7-11, but I'm queueing 1-6 on my NBD tree now.

Oh, you already sent a pull request? Having 6 in without the rest is not a good state. We now have the group info duplicated and one of them is ignored, but will become the meaningful copy later. We need to be careful to not let them diverge now.

I hope the rest is fine so we can switch over quickly, otherwise I'd prefer to revert 6 and redo it from the then current state when we merge the whole series.

Kevin

#40 lore raw
Quoted 20 lines · Kevin Wolf

Am 20.01.2021 um 21:52 hat Eric Blake geschrieben:

Quoted 16 lines · Vladimir Sementsov-Ogievskiy
Quoted 8 lines

Hi all!

These series has 3 goals:

- get rid of group file (to forget about rebase and in-list conflicts)
- introduce human-readable names for tests
- rewrite check into python

Quoted 6 lines

v7:
- fix wording and grammar
- satisfy python linters
- move argv interfaces all into one in new check script
- support '-n' == '--dry-run' option
- update check-block to run check with correct PYTHON

Quoted 2 lines

I'd really like a second reviewer on 7-11, but I'm queueing 1-6 on my NBD tree now.

Quoted 9 lines

Oh, you already sent a pull request? Having 6 in without the rest is not a good state. We now have the group info duplicated and one of them is ignored, but will become the meaningful copy later. We need to be careful to not let them diverge now.

I hope the rest is fine so we can switch over quickly, otherwise I'd prefer to revert 6 and redo it from the then current state when we merge the whole series.

I can make a new version now with tiny fixes suggested by Eric if it is convenient. (and keep --color suggestion for a separate follow-up)

Signature · 2 lines
Best regards,
Vladimir
#41 lore raw
In reply to Kevin Wolf
Quoted 20 lines · Kevin Wolf

Am 20.01.2021 um 21:52 hat Eric Blake geschrieben:

Quoted 16 lines · Vladimir Sementsov-Ogievskiy
Quoted 8 lines

Hi all!

These series has 3 goals:

- get rid of group file (to forget about rebase and in-list conflicts)
- introduce human-readable names for tests
- rewrite check into python

Quoted 6 lines

v7:
- fix wording and grammar
- satisfy python linters
- move argv interfaces all into one in new check script
- support '-n' == '--dry-run' option
- update check-block to run check with correct PYTHON

Quoted 2 lines

I'd really like a second reviewer on 7-11, but I'm queueing 1-6 on my NBD tree now.

Quoted 8 lines

Oh, you already sent a pull request? Having 6 in without the rest is not a good state. We now have the group info duplicated and one of them is ignored, but will become the meaningful copy later. We need to be careful to not let them diverge now.

I hope the rest is fine so we can switch over quickly, otherwise I'd prefer to revert 6 and redo it from the then current state when we merge the whole series.

Yeah, I probably jumped the gun there. If we don't have v8 in good working shape soon, I'm happy to send a temporary revert pull request for patch 6 - let's shoot for late Monday as the cut-off point where I will act if the revert is needed.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#42 lore raw

Am 22.01.2021 um 17:08 hat Eric Blake geschrieben:

Quoted 26 lines · Kevin Wolf
Quoted 25 lines

Am 20.01.2021 um 21:52 hat Eric Blake geschrieben:

Quoted 16 lines · Vladimir Sementsov-Ogievskiy
Quoted 8 lines

Hi all!

These series has 3 goals:

- get rid of group file (to forget about rebase and in-list conflicts)
- introduce human-readable names for tests
- rewrite check into python

Quoted 6 lines

v7:
- fix wording and grammar
- satisfy python linters
- move argv interfaces all into one in new check script
- support '-n' == '--dry-run' option
- update check-block to run check with correct PYTHON

Quoted 2 lines

I'd really like a second reviewer on 7-11, but I'm queueing 1-6 on my NBD tree now.

Oh, you already sent a pull request? Having 6 in without the rest is not a good state. We now have the group info duplicated and one of them is ignored, but will become the meaningful copy later. We need to be careful to not let them diverge now.

Quoted 8 lines
Quoted 3 lines

I hope the rest is fine so we can switch over quickly, otherwise I'd prefer to revert 6 and redo it from the then current state when we merge the whole series.

Yeah, I probably jumped the gun there. If we don't have v8 in good working shape soon, I'm happy to send a temporary revert pull request for patch 6 - let's shoot for late Monday as the cut-off point where I will act if the revert is needed.

I think I'm going to send a pull request anyway, so I could include it there if necessary.

Kevin

#43 lore raw
In reply to Vladimir Sementsov-Ogievskiy
Quoted 7 lines · Vladimir Sementsov-Ogievskiy

Hi all!

These series has 3 goals:

- get rid of group file (to forget about rebase and in-list conflicts)
- introduce human-readable names for tests
- rewrite check into python

Very nice!

I have wondered for some time if we should rewrite check as a wrapper around avocado. Your effort certainly goes in the right direction, since it at least uses the same programming language and introduces abstractions that can be easily turned into avocado wrappers.

Paolo

Quoted 350 lines

v7:
- fix wording and grammar
- satisfy python linters
- move argv interfaces all into one in new check script
- support '-n' == '--dry-run' option
- update check-block to run check with correct PYTHON

findtests: - stop parsing test file after first '# group: ' line
- use parse_test_name in add_group_file()
- make new list instead of modifying parameter exclude_groups

testenv: - fix machine_map
- fix self.python

testrunner: use env.python to run python tests

Vladimir Sementsov-Ogievskiy (11):
iotests/277: use dot slash for nbd-fault-injector.py running iotests/303: use dot slash for qcow2.py running
iotests: fix some whitespaces in test output files
iotests: make tests executable
iotests/294: add shebang line
iotests: define group in each iotest
iotests: add findtests.py
iotests: add testenv.py
iotests: add testrunner.py
iotests: rewrite check into python
iotests: rename and move 169 and 199 tests

  docs/devel/testing.rst                        |   50 +-
  Makefile                                      |    1 -
  tests/check-block.sh                          |    2 +-
  tests/qemu-iotests/001                        |    1 +
  tests/qemu-iotests/002                        |    1 +
  tests/qemu-iotests/003                        |    1 +
  tests/qemu-iotests/004                        |    1 +
  tests/qemu-iotests/005                        |    1 +
  tests/qemu-iotests/007                        |    1 +
  tests/qemu-iotests/008                        |    1 +
  tests/qemu-iotests/009                        |    1 +
  tests/qemu-iotests/010                        |    1 +
  tests/qemu-iotests/011                        |    1 +
  tests/qemu-iotests/012                        |    1 +
  tests/qemu-iotests/013                        |    1 +
  tests/qemu-iotests/014                        |    1 +
  tests/qemu-iotests/015                        |    1 +
  tests/qemu-iotests/017                        |    1 +
  tests/qemu-iotests/018                        |    1 +
  tests/qemu-iotests/019                        |    1 +
  tests/qemu-iotests/020                        |    1 +
  tests/qemu-iotests/021                        |    1 +
  tests/qemu-iotests/022                        |    1 +
  tests/qemu-iotests/023                        |    1 +
  tests/qemu-iotests/024                        |    1 +
  tests/qemu-iotests/025                        |    1 +
  tests/qemu-iotests/026                        |    1 +
  tests/qemu-iotests/027                        |    1 +
  tests/qemu-iotests/028                        |    1 +
  tests/qemu-iotests/029                        |    1 +
  tests/qemu-iotests/030                        |    1 +
  tests/qemu-iotests/031                        |    1 +
  tests/qemu-iotests/032                        |    1 +
  tests/qemu-iotests/033                        |    1 +
  tests/qemu-iotests/034                        |    1 +
  tests/qemu-iotests/035                        |    1 +
  tests/qemu-iotests/036                        |    1 +
  tests/qemu-iotests/037                        |    1 +
  tests/qemu-iotests/038                        |    1 +
  tests/qemu-iotests/039                        |    1 +
  tests/qemu-iotests/040                        |    1 +
  tests/qemu-iotests/041                        |    1 +
  tests/qemu-iotests/042                        |    1 +
  tests/qemu-iotests/043                        |    1 +
  tests/qemu-iotests/044                        |    1 +
  tests/qemu-iotests/045                        |    1 +
  tests/qemu-iotests/046                        |    1 +
  tests/qemu-iotests/047                        |    1 +
  tests/qemu-iotests/048                        |    1 +
  tests/qemu-iotests/049                        |    1 +
  tests/qemu-iotests/050                        |    1 +
  tests/qemu-iotests/051                        |    1 +
  tests/qemu-iotests/052                        |    1 +
  tests/qemu-iotests/053                        |    1 +
  tests/qemu-iotests/054                        |    1 +
  tests/qemu-iotests/055                        |    1 +
  tests/qemu-iotests/056                        |    1 +
  tests/qemu-iotests/057                        |    1 +
  tests/qemu-iotests/058                        |    1 +
  tests/qemu-iotests/059                        |    1 +
  tests/qemu-iotests/060                        |    1 +
  tests/qemu-iotests/061                        |    1 +
  tests/qemu-iotests/062                        |    1 +
  tests/qemu-iotests/063                        |    1 +
  tests/qemu-iotests/064                        |    1 +
  tests/qemu-iotests/065                        |    1 +
  tests/qemu-iotests/066                        |    1 +
  tests/qemu-iotests/068                        |    1 +
  tests/qemu-iotests/069                        |    1 +
  tests/qemu-iotests/070                        |    1 +
  tests/qemu-iotests/071                        |    1 +
  tests/qemu-iotests/072                        |    1 +
  tests/qemu-iotests/073                        |    1 +
  tests/qemu-iotests/074                        |    1 +
  tests/qemu-iotests/075                        |    1 +
  tests/qemu-iotests/076                        |    1 +
  tests/qemu-iotests/077                        |    1 +
  tests/qemu-iotests/078                        |    1 +
  tests/qemu-iotests/079                        |    1 +
  tests/qemu-iotests/080                        |    1 +
  tests/qemu-iotests/081                        |    1 +
  tests/qemu-iotests/082                        |    1 +
  tests/qemu-iotests/083                        |    1 +
  tests/qemu-iotests/084                        |    1 +
  tests/qemu-iotests/085                        |    1 +
  tests/qemu-iotests/086                        |    1 +
  tests/qemu-iotests/087                        |    1 +
  tests/qemu-iotests/088                        |    1 +
  tests/qemu-iotests/089                        |    1 +
  tests/qemu-iotests/090                        |    1 +
  tests/qemu-iotests/091                        |    1 +
  tests/qemu-iotests/092                        |    1 +
  tests/qemu-iotests/093                        |    1 +
  tests/qemu-iotests/094                        |    1 +
  tests/qemu-iotests/095                        |    1 +
  tests/qemu-iotests/096                        |    1 +
  tests/qemu-iotests/097                        |    1 +
  tests/qemu-iotests/098                        |    1 +
  tests/qemu-iotests/099                        |    1 +
  tests/qemu-iotests/101                        |    1 +
  tests/qemu-iotests/102                        |    1 +
  tests/qemu-iotests/103                        |    1 +
  tests/qemu-iotests/104                        |    1 +
  tests/qemu-iotests/105                        |    1 +
  tests/qemu-iotests/106                        |    1 +
  tests/qemu-iotests/107                        |    1 +
  tests/qemu-iotests/108                        |    1 +
  tests/qemu-iotests/109                        |    1 +
  tests/qemu-iotests/110                        |    1 +
  tests/qemu-iotests/111                        |    1 +
  tests/qemu-iotests/112                        |    1 +
  tests/qemu-iotests/113                        |    1 +
  tests/qemu-iotests/114                        |    1 +
  tests/qemu-iotests/115                        |    1 +
  tests/qemu-iotests/116                        |    1 +
  tests/qemu-iotests/117                        |    1 +
  tests/qemu-iotests/118                        |    1 +
  tests/qemu-iotests/119                        |    1 +
  tests/qemu-iotests/120                        |    1 +
  tests/qemu-iotests/121                        |    1 +
  tests/qemu-iotests/122                        |    1 +
  tests/qemu-iotests/123                        |    1 +
  tests/qemu-iotests/124                        |    1 +
  tests/qemu-iotests/125                        |    1 +
  tests/qemu-iotests/126                        |    1 +
  tests/qemu-iotests/127                        |    1 +
  tests/qemu-iotests/128                        |    1 +
  tests/qemu-iotests/129                        |    1 +
  tests/qemu-iotests/130                        |    1 +
  tests/qemu-iotests/131                        |    1 +
  tests/qemu-iotests/132                        |    1 +
  tests/qemu-iotests/133                        |    1 +
  tests/qemu-iotests/134                        |    1 +
  tests/qemu-iotests/135                        |    1 +
  tests/qemu-iotests/136                        |    1 +
  tests/qemu-iotests/137                        |    1 +
  tests/qemu-iotests/138                        |    1 +
  tests/qemu-iotests/139                        |    1 +
  tests/qemu-iotests/140                        |    1 +
  tests/qemu-iotests/141                        |    1 +
  tests/qemu-iotests/143                        |    1 +
  tests/qemu-iotests/144                        |    1 +
  tests/qemu-iotests/145                        |    1 +
  tests/qemu-iotests/146                        |    1 +
  tests/qemu-iotests/147                        |    1 +
  tests/qemu-iotests/148                        |    1 +
  tests/qemu-iotests/149                        |    1 +
  tests/qemu-iotests/150                        |    1 +
  tests/qemu-iotests/151                        |    1 +
  tests/qemu-iotests/152                        |    1 +
  tests/qemu-iotests/153                        |    1 +
  tests/qemu-iotests/154                        |    1 +
  tests/qemu-iotests/155                        |    1 +
  tests/qemu-iotests/156                        |    1 +
  tests/qemu-iotests/157                        |    1 +
  tests/qemu-iotests/158                        |    1 +
  tests/qemu-iotests/159                        |    1 +
  tests/qemu-iotests/160                        |    1 +
  tests/qemu-iotests/161                        |    1 +
  tests/qemu-iotests/162                        |    1 +
  tests/qemu-iotests/163                        |    1 +
  tests/qemu-iotests/165                        |    1 +
  tests/qemu-iotests/170                        |    1 +
  tests/qemu-iotests/171                        |    1 +
  tests/qemu-iotests/172                        |    1 +
  tests/qemu-iotests/173                        |    1 +
  tests/qemu-iotests/174                        |    1 +
  tests/qemu-iotests/175                        |    1 +
  tests/qemu-iotests/175.out                    |    2 +-
  tests/qemu-iotests/176                        |    1 +
  tests/qemu-iotests/177                        |    1 +
  tests/qemu-iotests/178                        |    1 +
  tests/qemu-iotests/179                        |    1 +
  tests/qemu-iotests/181                        |    1 +
  tests/qemu-iotests/182                        |    1 +
  tests/qemu-iotests/183                        |    1 +
  tests/qemu-iotests/184                        |    1 +
  tests/qemu-iotests/185                        |    1 +
  tests/qemu-iotests/186                        |    1 +
  tests/qemu-iotests/187                        |    1 +
  tests/qemu-iotests/188                        |    1 +
  tests/qemu-iotests/189                        |    1 +
  tests/qemu-iotests/190                        |    1 +
  tests/qemu-iotests/191                        |    1 +
  tests/qemu-iotests/192                        |    1 +
  tests/qemu-iotests/194                        |    1 +
  tests/qemu-iotests/195                        |    1 +
  tests/qemu-iotests/196                        |    1 +
  tests/qemu-iotests/197                        |    1 +
  tests/qemu-iotests/198                        |    1 +
  tests/qemu-iotests/200                        |    1 +
  tests/qemu-iotests/201                        |    1 +
  tests/qemu-iotests/202                        |    1 +
  tests/qemu-iotests/203                        |    1 +
  tests/qemu-iotests/204                        |    1 +
  tests/qemu-iotests/205                        |    1 +
  tests/qemu-iotests/206                        |    1 +
  tests/qemu-iotests/207                        |    1 +
  tests/qemu-iotests/208                        |    1 +
  tests/qemu-iotests/209                        |    1 +
  tests/qemu-iotests/210                        |    1 +
  tests/qemu-iotests/211                        |    1 +
  tests/qemu-iotests/212                        |    1 +
  tests/qemu-iotests/213                        |    1 +
  tests/qemu-iotests/214                        |    1 +
  tests/qemu-iotests/215                        |    1 +
  tests/qemu-iotests/216                        |    1 +
  tests/qemu-iotests/217                        |    1 +
  tests/qemu-iotests/218                        |    1 +
  tests/qemu-iotests/219                        |    1 +
  tests/qemu-iotests/220                        |    1 +
  tests/qemu-iotests/221                        |    1 +
  tests/qemu-iotests/222                        |    1 +
  tests/qemu-iotests/223                        |    1 +
  tests/qemu-iotests/224                        |    1 +
  tests/qemu-iotests/225                        |    1 +
  tests/qemu-iotests/226                        |    1 +
  tests/qemu-iotests/227                        |    1 +
  tests/qemu-iotests/228                        |    1 +
  tests/qemu-iotests/229                        |    1 +
  tests/qemu-iotests/231                        |    1 +
  tests/qemu-iotests/232                        |    1 +
  tests/qemu-iotests/233                        |    1 +
  tests/qemu-iotests/234                        |    1 +
  tests/qemu-iotests/235                        |    1 +
  tests/qemu-iotests/236                        |    1 +
  tests/qemu-iotests/237                        |    1 +
  tests/qemu-iotests/238                        |    1 +
  tests/qemu-iotests/239                        |    1 +
  tests/qemu-iotests/240                        |    1 +
  tests/qemu-iotests/241                        |    1 +
  tests/qemu-iotests/242                        |    1 +
  tests/qemu-iotests/243                        |    1 +
  tests/qemu-iotests/244                        |    1 +
  tests/qemu-iotests/245                        |    1 +
  tests/qemu-iotests/246                        |    1 +
  tests/qemu-iotests/247                        |    1 +
  tests/qemu-iotests/248                        |    1 +
  tests/qemu-iotests/249                        |    1 +
  tests/qemu-iotests/250                        |    1 +
  tests/qemu-iotests/251                        |    1 +
  tests/qemu-iotests/252                        |    1 +
  tests/qemu-iotests/253                        |    1 +
  tests/qemu-iotests/254                        |    1 +
  tests/qemu-iotests/255                        |    1 +
  tests/qemu-iotests/256                        |    1 +
  tests/qemu-iotests/257                        |    1 +
  tests/qemu-iotests/258                        |    1 +
  tests/qemu-iotests/259                        |    1 +
  tests/qemu-iotests/260                        |    1 +
  tests/qemu-iotests/261                        |    1 +
  tests/qemu-iotests/262                        |    1 +
  tests/qemu-iotests/263                        |    1 +
  tests/qemu-iotests/264                        |    1 +
  tests/qemu-iotests/265                        |    1 +
  tests/qemu-iotests/266                        |    1 +
  tests/qemu-iotests/267                        |    1 +
  tests/qemu-iotests/268                        |    1 +
  tests/qemu-iotests/270                        |    1 +
  tests/qemu-iotests/271                        |    1 +
  tests/qemu-iotests/271.out                    |   12 +-
  tests/qemu-iotests/272                        |    1 +
  tests/qemu-iotests/273                        |    1 +
  tests/qemu-iotests/274                        |    1 +
  tests/qemu-iotests/277                        |    3 +-
  tests/qemu-iotests/279                        |    1 +
  tests/qemu-iotests/280                        |    1 +
  tests/qemu-iotests/281                        |    1 +
  tests/qemu-iotests/282                        |    1 +
  tests/qemu-iotests/283                        |    1 +
  tests/qemu-iotests/284                        |    1 +
  tests/qemu-iotests/286                        |    1 +
  tests/qemu-iotests/287                        |    1 +
  tests/qemu-iotests/287.out                    |   10 +-
  tests/qemu-iotests/288                        |    1 +
  tests/qemu-iotests/289                        |    1 +
  tests/qemu-iotests/290                        |    1 +
  tests/qemu-iotests/291                        |    1 +
  tests/qemu-iotests/292                        |    1 +
  tests/qemu-iotests/293                        |    1 +
  tests/qemu-iotests/294                        |    2 +
  tests/qemu-iotests/295                        |    1 +
  tests/qemu-iotests/296                        |    1 +
  tests/qemu-iotests/297                        |    1 +
  tests/qemu-iotests/298                        |    0
  tests/qemu-iotests/299                        |    1 +
  tests/qemu-iotests/300                        |    1 +
  tests/qemu-iotests/301                        |    1 +
  tests/qemu-iotests/302                        |    1 +
  tests/qemu-iotests/303                        |    3 +-
  tests/qemu-iotests/304                        |    1 +
  tests/qemu-iotests/305                        |    1 +
  tests/qemu-iotests/307                        |    1 +
  tests/qemu-iotests/308                        |    1 +
  tests/qemu-iotests/309                        |    1 +
  tests/qemu-iotests/312                        |    1 +
  tests/qemu-iotests/check                      | 1085 ++---------------
  tests/qemu-iotests/common.env.in              |    3 -
  tests/qemu-iotests/findtests.py               |  159 +++
  tests/qemu-iotests/group                      |  321 -----
  tests/qemu-iotests/meson.build                |    3 -
  tests/qemu-iotests/testenv.py                 |  255 ++++
  tests/qemu-iotests/testrunner.py              |  344 ++++++
  .../migrate-bitmaps-postcopy-test}            |    1 +
  .../migrate-bitmaps-postcopy-test.out}        |    0
  .../{169 => tests/migrate-bitmaps-test}       |    1 +
  .../migrate-bitmaps-test.out}                 |    0
  307 files changed, 1233 insertions(+), 1310 deletions(-)

mode change 100644 => 100755 tests/qemu-iotests/283
mode change 100644 => 100755 tests/qemu-iotests/298
mode change 100644 => 100755 tests/qemu-iotests/299
delete mode 100644 tests/qemu-iotests/common.env.in
create mode 100644 tests/qemu-iotests/findtests.py
delete mode 100644 tests/qemu-iotests/group
create mode 100644 tests/qemu-iotests/testenv.py
create mode 100644 tests/qemu-iotests/testrunner.py
rename tests/qemu-iotests/{199 => tests/migrate-bitmaps-postcopy-test} (99%) rename tests/qemu-iotests/{199.out => tests/migrate-bitmaps-postcopy-test.out} (100%) rename tests/qemu-iotests/{169 => tests/migrate-bitmaps-test} (99%) rename tests/qemu-iotests/{169.out => tests/migrate-bitmaps-test.out} (100%)

#44 lore raw
In reply to Vladimir Sementsov-Ogievskiy

Am 16.01.2021 um 14:44 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 8 lines

Hi all!

These series has 3 goals:

- get rid of group file (to forget about rebase and in-list conflicts)
- introduce human-readable names for tests
- rewrite check into python

Quoted 6 lines

v7:
- fix wording and grammar
- satisfy python linters
- move argv interfaces all into one in new check script
- support '-n' == '--dry-run' option
- update check-block to run check with correct PYTHON

Okay, I think I'm finished with the review for this version.

I also tried pylint/mypy again and it's mostly clean now (pylint complains about the TODO comments, I think we should just disable that warning).

Feel free to include the following as patch 12 in v8.

Kevin

tests/qemu-iotests/297 +4 -2
index 85bc1c0c85..8aaa3e455c 100755
@@ -31,13 +31,15 @@ if ! type -p "mypy" > /dev/null; then
     _notrun "mypy not found"
 fi
 
-pylint-3 --score=n iotests.py
+FILES="findtests.py iotests.py testenv.py testrunner.py check"
+
+pylint-3 --score=n $FILES
 
 MYPYPATH=../../python/ mypy --warn-unused-configs --disallow-subclassing-any \
     --disallow-any-generics --disallow-incomplete-defs \
     --disallow-untyped-decorators --no-implicit-optional \
     --warn-redundant-casts --warn-unused-ignores \
-    --no-implicit-reexport iotests.py
+    --no-implicit-reexport $FILES
 
 # success, all done
 echo "*** done"
tests/qemu-iotests/297.out +1 -1
index 6acc843649..85213ef96e 100644
@@ -1,3 +1,3 @@
 QA output created by 297
-Success: no issues found in 1 source file
+Success: no issues found in 5 source files
 *** done
tests/qemu-iotests/pylintrc +2 -1
index cd3702e23c..980bfa7b20 100644
@@ -9,7 +9,8 @@
 # --enable=similarities". If you want to run only the classes checker, but have
 # no Warning level messages displayed, use "--disable=all --enable=classes
 # --disable=W".
-disable=invalid-name,
+disable=fixme,
+        invalid-name,
         no-else-return,
         too-few-public-methods,
         too-many-arguments,
#45 lore raw
Quoted 18 lines · Kevin Wolf

Am 16.01.2021 um 14:44 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 8 lines

Hi all!

These series has 3 goals:

- get rid of group file (to forget about rebase and in-list conflicts)
- introduce human-readable names for tests
- rewrite check into python

Quoted 6 lines

v7:
- fix wording and grammar
- satisfy python linters
- move argv interfaces all into one in new check script
- support '-n' == '--dry-run' option
- update check-block to run check with correct PYTHON

Okay, I think I'm finished with the review for this version.

Quoted 7 lines

I also tried pylint/mypy again and it's mostly clean now (pylint complains about the TODO comments, I think we should just disable that warning).

Feel free to include the following as patch 12 in v8.

Kevin

I remember Max already queued good changes for 297 in his block branch

Quoted 38 lines
tests/qemu-iotests/297 +4 -2
index 85bc1c0c85..8aaa3e455c 100755
@@ -31,13 +31,15 @@ if ! type -p "mypy" > /dev/null; then
      _notrun "mypy not found"
  fi
 
-pylint-3 --score=n iotests.py
+FILES="findtests.py iotests.py testenv.py testrunner.py check"
+
+pylint-3 --score=n $FILES
 
  MYPYPATH=../../python/ mypy --warn-unused-configs --disallow-subclassing-any \
      --disallow-any-generics --disallow-incomplete-defs \
      --disallow-untyped-decorators --no-implicit-optional \
      --warn-redundant-casts --warn-unused-ignores \
-    --no-implicit-reexport iotests.py
+    --no-implicit-reexport $FILES
 
  # success, all done
  echo "*** done"
tests/qemu-iotests/297.out +1 -1
index 6acc843649..85213ef96e 100644
@@ -1,3 +1,3 @@
  QA output created by 297
-Success: no issues found in 1 source file
+Success: no issues found in 5 source files
  *** done
tests/qemu-iotests/pylintrc +0 -0
index cd3702e23c..980bfa7b20 100644

tests/qemu-iotests/pylintrc line 13

Quoted 6 lines
tests/qemu-iotests/pylintrc +2 -1
@@ -9,7 +9,8 @@
  # --enable=similarities". If you want to run only the classes checker, but have
  # no Warning level messages displayed, use "--disable=all --enable=classes
  # --disable=W".
-disable=invalid-name,
+disable=fixme,
+        invalid-name,

Max just disable it in 297. I think we should not disable it globally, as additional hint doesn't hurt (except for test output).

Quoted 4 lines
tests/qemu-iotests/pylintrc +0 -0
          no-else-return,
          too-few-public-methods,
          too-many-arguments,
Signature · 2 lines
Best regards,
Vladimir