Skip to contentSkip to conversation

image-fuzzer: Port to Python 3

PATCH Applied Eduardo Habkost 4 participants 21 messages Oct 16, 2019 – Nov 5, 2019

lore
Share & tools
lore
#0 lore raw

cover image-fuzzer: Port to Python 3

This series ports image-fuzzer to Python 3.

Eduardo Habkost (10):
image-fuzzer: Open image files in binary mode
image-fuzzer: Write bytes instead of string to image file
image-fuzzer: Explicitly use integer division operator
image-fuzzer: Use io.StringIO
image-fuzzer: Use %r for all fiels at Field.__repr__()
image-fuzzer: Return bytes objects on string fuzzing functions
image-fuzzer: Use bytes constant for field values
image-fuzzer: Encode file name and file format to bytes
image-fuzzer: Run using python3
image-fuzzer: Use errors parameter of subprocess.Popen()

 tests/image-fuzzer/qcow2/__init__.py |  1 -
 tests/image-fuzzer/qcow2/fuzz.py     | 54 +++++++++++++-------------
 tests/image-fuzzer/qcow2/layout.py   | 57 ++++++++++++++--------------
 tests/image-fuzzer/runner.py         | 12 +++---
 4 files changed, 61 insertions(+), 63 deletions(-)
Signature · 1 line
2.21.0
#1 lore raw

PATCH 1/10 image-fuzzer: Open image files in binary mode

This probably never caused problems because on Linux there's no actual newline conversion happening, but on Python 3 the binary/text distinction is stronger and we must explicitly open the image file in binary mode.

---
 tests/image-fuzzer/qcow2/layout.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/image-fuzzer/qcow2/layout.py +1 -1
index 675877da96..c57418fa15 100644
@@ -503,7 +503,7 @@ class Image(object):
 
     def write(self, filename):
         """Write an entire image to the file."""
-        image_file = open(filename, 'w')
+        image_file = open(filename, 'wb')
         for field in self:
             image_file.seek(field.offset)
             image_file.write(struct.pack(field.fmt, field.value))
Signature · 1 line
2.21.0
Reviewed-by on v1 1/10
#2 lore raw

Re: [PATCH 01/10] image-fuzzer: Open image files in binary mode

Quoted 16 lines · Eduardo Habkost

This probably never caused problems because on Linux there's no actual newline conversion happening, but on Python 3 the binary/text distinction is stronger and we must explicitly open the image file in binary mode.

---
  tests/image-fuzzer/qcow2/layout.py | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
tests/image-fuzzer/qcow2/layout.py +0 -0
index 675877da96..c57418fa15 100644

tests/image-fuzzer/qcow2/layout.py class Image(object): · line 509

Quoted 8 lines
tests/image-fuzzer/qcow2/layout.py +1 -1
@@ -503,7 +503,7 @@ class Image(object):
 
      def write(self, filename):
          """Write an entire image to the file."""
-        image_file = open(filename, 'w')
+        image_file = open(filename, 'wb')
          for field in self:
              image_file.seek(field.offset)
              image_file.write(struct.pack(field.fmt, field.value))
#3 lore raw

PATCH 2/10 image-fuzzer: Write bytes instead of string to image file

In reply to Eduardo Habkost

This is necessary for Python 3 compatibility.

---
 tests/image-fuzzer/qcow2/layout.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/image-fuzzer/qcow2/layout.py +1 -1
index c57418fa15..fe273d4143 100644
@@ -518,7 +518,7 @@ class Image(object):
         rounded = (size + self.cluster_size - 1) & ~(self.cluster_size - 1)
         if rounded > size:
             image_file.seek(rounded - 1)
-            image_file.write("\0")
+            image_file.write(b'\x00')
         image_file.close()
 
     @staticmethod
Signature · 1 line
2.21.0
Reviewed-by on v1 2/10
#4 lore raw

Re: [PATCH 02/10] image-fuzzer: Write bytes instead of string to image file

Quoted 13 lines · Eduardo Habkost

This is necessary for Python 3 compatibility.

---
  tests/image-fuzzer/qcow2/layout.py | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
tests/image-fuzzer/qcow2/layout.py +0 -0
index c57418fa15..fe273d4143 100644

tests/image-fuzzer/qcow2/layout.py class Image(object): · line 524

Quoted 8 lines
tests/image-fuzzer/qcow2/layout.py +1 -1
@@ -518,7 +518,7 @@ class Image(object):
          rounded = (size + self.cluster_size - 1) & ~(self.cluster_size - 1)
          if rounded > size:
              image_file.seek(rounded - 1)
-            image_file.write("\0")
+            image_file.write(b'\x00')
          image_file.close()
 
      @staticmethod
#5 lore raw

PATCH 3/10 image-fuzzer: Explicitly use integer division operator

In reply to Eduardo Habkost

Most of the division expressions in image-fuzzer assume integer division. Use the // operator to keep the same behavior when we move to Python 3.

---
 tests/image-fuzzer/qcow2/fuzz.py   | 12 ++++-----
 tests/image-fuzzer/qcow2/layout.py | 40 +++++++++++++++---------------
 2 files changed, 26 insertions(+), 26 deletions(-)
tests/image-fuzzer/qcow2/fuzz.py +6 -6
index abc4f0635d..154dc06cc0 100644
@@ -27,14 +27,14 @@ UINT64 = 0xffffffffffffffff
 UINT32_M = 31
 UINT64_M = 63
 # Fuzz vectors
-UINT8_V = [0, 0x10, UINT8/4, UINT8/2 - 1, UINT8/2, UINT8/2 + 1, UINT8 - 1,
+UINT8_V = [0, 0x10, UINT8//4, UINT8//2 - 1, UINT8//2, UINT8//2 + 1, UINT8 - 1,
            UINT8]
-UINT16_V = [0, 0x100, 0x1000, UINT16/4, UINT16/2 - 1, UINT16/2, UINT16/2 + 1,
+UINT16_V = [0, 0x100, 0x1000, UINT16//4, UINT16//2 - 1, UINT16//2, UINT16//2 + 1,
             UINT16 - 1, UINT16]
-UINT32_V = [0, 0x100, 0x1000, 0x10000, 0x100000, UINT32/4, UINT32/2 - 1,
-            UINT32/2, UINT32/2 + 1, UINT32 - 1, UINT32]
-UINT64_V = UINT32_V + [0x1000000, 0x10000000, 0x100000000, UINT64/4,
-                       UINT64/2 - 1, UINT64/2, UINT64/2 + 1, UINT64 - 1,
+UINT32_V = [0, 0x100, 0x1000, 0x10000, 0x100000, UINT32//4, UINT32//2 - 1,
+            UINT32//2, UINT32//2 + 1, UINT32 - 1, UINT32]
+UINT64_V = UINT32_V + [0x1000000, 0x10000000, 0x100000000, UINT64//4,
+                       UINT64//2 - 1, UINT64//2, UINT64//2 + 1, UINT64 - 1,
                        UINT64]
 STRING_V = ['%s%p%x%d', '.1024d', '%.2049d', '%p%p%p%p', '%x%x%x%x',
             '%d%d%d%d', '%s%s%s%s', '%99999999999s', '%08x', '%%20d', '%%20n',
tests/image-fuzzer/qcow2/layout.py +20 -20
index fe273d4143..6501c9fd4b 100644
@@ -253,7 +253,7 @@ class Image(object):
                 ['>I', self.ext_offset, 0x6803f857, 'ext_magic'],
                 # One feature table contains 3 fields and takes 48 bytes
                 ['>I', self.ext_offset + UINT32_S,
-                 len(feature_tables) / 3 * 48, 'ext_length']
+                 len(feature_tables) // 3 * 48, 'ext_length']
             ] + feature_tables)
             self.ext_offset = inner_offset
 
@@ -271,7 +271,7 @@ class Image(object):
         def create_l2_entry(host, guest, l2_cluster):
             """Generate one L2 entry."""
             offset = l2_cluster * self.cluster_size
-            l2_size = self.cluster_size / UINT64_S
+            l2_size = self.cluster_size // UINT64_S
             entry_offset = offset + UINT64_S * (guest % l2_size)
             cluster_descriptor = host * self.cluster_size
             if not self.header['version'][0].value == 2:
@@ -283,8 +283,8 @@ class Image(object):
 
         def create_l1_entry(l2_cluster, l1_offset, guest):
             """Generate one L1 entry."""
-            l2_size = self.cluster_size / UINT64_S
-            entry_offset = l1_offset + UINT64_S * (guest / l2_size)
+            l2_size = self.cluster_size // UINT64_S
+            entry_offset = l1_offset + UINT64_S * (guest // l2_size)
             # While snapshots are not supported bit #63 = 1
             entry_val = (1 << 63) + l2_cluster * self.cluster_size
             return ['>Q', entry_offset, entry_val, 'l1_entry']
@@ -298,11 +298,11 @@ class Image(object):
             l2 = []
         else:
             meta_data = self._get_metadata()
-            guest_clusters = random.sample(range(self.image_size /
+            guest_clusters = random.sample(range(self.image_size //
                                                  self.cluster_size),
                                            len(self.data_clusters))
             # Number of entries in a L1/L2 table
-            l_size = self.cluster_size / UINT64_S
+            l_size = self.cluster_size // UINT64_S
             # Number of clusters necessary for L1 table
             l1_size = int(ceil((max(guest_clusters) + 1) / float(l_size**2)))
             l1_start = self._get_adjacent_clusters(self.data_clusters |
@@ -318,7 +318,7 @@ class Image(object):
             # L2 entries
             l2 = []
             for host, guest in zip(self.data_clusters, guest_clusters):
-                l2_id = guest / l_size
+                l2_id = guest // l_size
                 if l2_id not in l2_ids:
                     l2_ids.append(l2_id)
                     l2_clusters.append(self._get_adjacent_clusters(
@@ -339,14 +339,14 @@ class Image(object):
         def allocate_rfc_blocks(data, size):
             """Return indices of clusters allocated for refcount blocks."""
             cluster_ids = set()
-            diff = block_ids = set([x / size for x in data])
+            diff = block_ids = set([x // size for x in data])
             while len(diff) != 0:
                 # Allocate all yet not allocated clusters
                 new = self._get_available_clusters(data | cluster_ids,
                                                    len(diff))
                 # Indices of new refcount blocks necessary to cover clusters
                 # in 'new'
-                diff = set([x / size for x in new]) - block_ids
+                diff = set([x // size for x in new]) - block_ids
                 cluster_ids |= new
                 block_ids |= diff
             return cluster_ids, block_ids
@@ -359,7 +359,7 @@ class Image(object):
             blocks = set(init_blocks)
             clusters = set()
             # Number of entries in one cluster of the refcount table
-            size = self.cluster_size / UINT64_S
+            size = self.cluster_size // UINT64_S
             # Number of clusters necessary for the refcount table based on
             # the current number of refcount blocks
             table_size = int(ceil((max(blocks) + 1) / float(size)))
@@ -373,7 +373,7 @@ class Image(object):
                                                  table_size + 1))
             # New refcount blocks necessary for clusters occupied by the
             # refcount table
-            diff = set([c / block_size for c in table_clusters]) - blocks
+            diff = set([c // block_size for c in table_clusters]) - blocks
             blocks |= diff
             while len(diff) != 0:
                 # Allocate clusters for new refcount blocks
@@ -382,12 +382,12 @@ class Image(object):
                                                    len(diff))
                 # Indices of new refcount blocks necessary to cover
                 # clusters in 'new'
-                diff = set([x / block_size for x in new]) - blocks
+                diff = set([x // block_size for x in new]) - blocks
                 clusters |= new
                 blocks |= diff
                 # Check if the refcount table needs one more cluster
                 if int(ceil((max(blocks) + 1) / float(size))) > table_size:
-                    new_block_id = (table_start + table_size) / block_size
+                    new_block_id = (table_start + table_size) // block_size
                     # Check if the additional table cluster needs
                     # one more refcount block
                     if new_block_id not in blocks:
@@ -399,13 +399,13 @@ class Image(object):
         def create_table_entry(table_offset, block_cluster, block_size,
                                cluster):
             """Generate a refcount table entry."""
-            offset = table_offset + UINT64_S * (cluster / block_size)
+            offset = table_offset + UINT64_S * (cluster // block_size)
             return ['>Q', offset, block_cluster * self.cluster_size,
                     'refcount_table_entry']
 
         def create_block_entry(block_cluster, block_size, cluster):
             """Generate a list of entries for the current block."""
-            entry_size = self.cluster_size / block_size
+            entry_size = self.cluster_size // block_size
             offset = block_cluster * self.cluster_size
             entry_offset = offset + entry_size * (cluster % block_size)
             # While snapshots are not supported all refcounts are set to 1
@@ -415,7 +415,7 @@ class Image(object):
         # Number of refcount entries per refcount block
         # Convert self.cluster_size from bytes to bits to have the same
         # base for the numerator and denominator
-        block_size = self.cluster_size * 8 / refcount_bits
+        block_size = self.cluster_size * 8 // refcount_bits
         meta_data = self._get_metadata()
         if len(self.data_clusters) == 0:
             # All metadata for an empty guest image needs 4 clusters:
@@ -452,8 +452,8 @@ class Image(object):
         rfc_blocks = []
 
         for cluster in sorted(self.data_clusters | meta_data):
-            if cluster / block_size != block_id:
-                block_id = cluster / block_size
+            if cluster // block_size != block_id:
+                block_id = cluster // block_size
                 block_cluster = block_clusters[block_ids.index(block_id)]
                 rfc_table.append(create_table_entry(table_offset,
                                                     block_cluster,
@@ -587,7 +587,7 @@ class Image(object):
     def _alloc_data(img_size, cluster_size):
         """Return a set of random indices of clusters allocated for guest data.
         """
-        num_of_cls = img_size/cluster_size
+        num_of_cls = img_size // cluster_size
         return set(random.sample(range(1, num_of_cls + 1),
                                  random.randint(0, num_of_cls)))
 
@@ -595,7 +595,7 @@ class Image(object):
         """Return indices of clusters allocated for image metadata."""
         ids = set()
         for x in self:
-            ids.add(x.offset/self.cluster_size)
+            ids.add(x.offset // self.cluster_size)
         return ids
 
 
Signature · 1 line
2.21.0
#6 lore raw

PATCH 4/10 image-fuzzer: Use io.StringIO

In reply to Eduardo Habkost

StringIO.StringIO is not available on Python 3, but io.StringIO is available on both Python 2 and 3. io.StringIO is slightly different from the Python 2 StringIO module, though, so we need bytes coming from subprocess.Popen() to be explicitly decoded.

---
 tests/image-fuzzer/runner.py | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)
tests/image-fuzzer/runner.py +9 -5
index 95d84f38f3..94cab5bd93 100755
@@ -28,7 +28,7 @@ import shutil
 from itertools import count
 import time
 import getopt
-import StringIO
+import io
 import resource
 
 try:
@@ -84,8 +84,12 @@ def run_app(fd, q_args):
     try:
         out, err = process.communicate()
         signal.alarm(0)
-        fd.write(out)
-        fd.write(err)
+        # fd is a text file, so we need to decode the process output before
+        # writing to it.
+        # We could be simply using the `errors` parameter of subprocess.Popen(),
+        # but this will be possible only after migrating to Python 3
+        fd.write(out.decode(errors='replace'))
+        fd.write(err.decode(errors='replace'))
         fd.flush()
         return process.returncode
 
@@ -183,7 +187,7 @@ class TestEnv(object):
                                            MAX_BACKING_FILE_SIZE) * (1 << 20)
         cmd = self.qemu_img + ['create', '-f', backing_file_fmt,
                                backing_file_name, str(backing_file_size)]
-        temp_log = StringIO.StringIO()
+        temp_log = io.StringIO()
         retcode = run_app(temp_log, cmd)
         if retcode == 0:
             temp_log.close()
@@ -240,7 +244,7 @@ class TestEnv(object):
                            "Backing file: %s\n" \
                            % (self.seed, " ".join(current_cmd),
                               self.current_dir, backing_file_name)
-            temp_log = StringIO.StringIO()
+            temp_log = io.StringIO()
             try:
                 retcode = run_app(temp_log, current_cmd)
             except OSError as e:
Signature · 1 line
2.21.0
#7 lore raw

PATCH 5/10 image-fuzzer: Use %r for all fiels at Field.__repr__()

In reply to Eduardo Habkost

This makes the formatting code simpler, and safer if we change the type of self.value from str to bytes.

---
 tests/image-fuzzer/qcow2/layout.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tests/image-fuzzer/qcow2/layout.py +2 -2
index 6501c9fd4b..0adcbd448d 100644
@@ -53,8 +53,8 @@ class Field(object):
         return iter([self.fmt, self.offset, self.value, self.name])
 
     def __repr__(self):
-        return "Field(fmt='%s', offset=%d, value=%s, name=%s)" % \
-            (self.fmt, self.offset, str(self.value), self.name)
+        return "Field(fmt=%r, offset=%r, value=%r, name=%r)" % \
+            (self.fmt, self.offset, self.value, self.name)
 
 
 class FieldsList(object):
Signature · 1 line
2.21.0
#8 lore raw

Re: [PATCH 05/10] image-fuzzer: Use %r for all fiels at Field.__repr__()

"fields" in the commit message.

Quoted 24 lines · Eduardo Habkost

This makes the formatting code simpler, and safer if we change the type of self.value from str to bytes.

---
 tests/image-fuzzer/qcow2/layout.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tests/image-fuzzer/qcow2/layout.py +2 -2
index 6501c9fd4b..0adcbd448d 100644
@@ -53,8 +53,8 @@ class Field(object):
         return iter([self.fmt, self.offset, self.value, self.name])
 
     def __repr__(self):
-        return "Field(fmt='%s', offset=%d, value=%s, name=%s)" % \
-            (self.fmt, self.offset, str(self.value), self.name)
+        return "Field(fmt=%r, offset=%r, value=%r, name=%r)" % \
+            (self.fmt, self.offset, self.value, self.name)
 
 
 class FieldsList(object):
#9 lore raw

PATCH 6/10 image-fuzzer: Return bytes objects on string fuzzing functions

In reply to Eduardo Habkost

No caller of fuzzer functions is interested in unicode string values, so replace them with bytes sequences.

---
 tests/image-fuzzer/qcow2/fuzz.py | 42 ++++++++++++++++----------------
 1 file changed, 21 insertions(+), 21 deletions(-)
tests/image-fuzzer/qcow2/fuzz.py +21 -21
index 154dc06cc0..c58bf11005 100644
@@ -36,11 +36,11 @@ UINT32_V = [0, 0x100, 0x1000, 0x10000, 0x100000, UINT32//4, UINT32//2 - 1,
 UINT64_V = UINT32_V + [0x1000000, 0x10000000, 0x100000000, UINT64//4,
                        UINT64//2 - 1, UINT64//2, UINT64//2 + 1, UINT64 - 1,
                        UINT64]
-STRING_V = ['%s%p%x%d', '.1024d', '%.2049d', '%p%p%p%p', '%x%x%x%x',
-            '%d%d%d%d', '%s%s%s%s', '%99999999999s', '%08x', '%%20d', '%%20n',
-            '%%20x', '%%20s', '%s%s%s%s%s%s%s%s%s%s', '%p%p%p%p%p%p%p%p%p%p',
-            '%#0123456x%08x%x%s%p%d%n%o%u%c%h%l%q%j%z%Z%t%i%e%g%f%a%C%S%08x%%',
-            '%s x 129', '%x x 257']
+BYTES_V = [b'%s%p%x%d', b'.1024d', b'%.2049d', b'%p%p%p%p', b'%x%x%x%x',
+           b'%d%d%d%d', b'%s%s%s%s', b'%99999999999s', b'%08x', b'%%20d', b'%%20n',
+           b'%%20x', b'%%20s', b'%s%s%s%s%s%s%s%s%s%s', b'%p%p%p%p%p%p%p%p%p%p',
+           b'%#0123456x%08x%x%s%p%d%n%o%u%c%h%l%q%j%z%Z%t%i%e%g%f%a%C%S%08x%%',
+           b'%s x 129', b'%x x 257']
 
 
 def random_from_intervals(intervals):
@@ -76,12 +76,12 @@ def random_bits(bit_ranges):
     return val
 
 
-def truncate_string(strings, length):
-    """Return strings truncated to specified length."""
-    if type(strings) == list:
-        return [s[:length] for s in strings]
+def truncate_bytes(sequences, length):
+    """Return sequences truncated to specified length."""
+    if type(sequences) == list:
+        return [s[:length] for s in sequences]
     else:
-        return strings[:length]
+        return sequences[:length]
 
 
 def validator(current, pick, choices):
@@ -110,12 +110,12 @@ def bit_validator(current, bit_ranges):
     return validator(current, random_bits, bit_ranges)
 
 
-def string_validator(current, strings):
-    """Return a random string value from the list not equal to the current.
+def bytes_validator(current, sequences):
+    """Return a random bytes value from the list not equal to the current.
 
     This function is useful for selection from valid values except current one.
     """
-    return validator(current, random.choice, strings)
+    return validator(current, random.choice, sequences)
 
 
 def selector(current, constraints, validate=int_validator):
@@ -283,9 +283,9 @@ def header_length(current):
 def bf_name(current):
     """Fuzz the backing file name."""
     constraints = [
-        truncate_string(STRING_V, len(current))
+        truncate_bytes(BYTES_V, len(current))
     ]
-    return selector(current, constraints, string_validator)
+    return selector(current, constraints, bytes_validator)
 
 
 def ext_magic(current):
@@ -303,10 +303,10 @@ def ext_length(current):
 def bf_format(current):
     """Fuzz backing file format in the corresponding header extension."""
     constraints = [
-        truncate_string(STRING_V, len(current)),
-        truncate_string(STRING_V, (len(current) + 7) & ~7)  # Fuzz padding
+        truncate_bytes(BYTES_V, len(current)),
+        truncate_bytes(BYTES_V, (len(current) + 7) & ~7)  # Fuzz padding
     ]
-    return selector(current, constraints, string_validator)
+    return selector(current, constraints, bytes_validator)
 
 
 def feature_type(current):
@@ -324,10 +324,10 @@ def feature_bit_number(current):
 def feature_name(current):
     """Fuzz feature name field of a feature name table header extension."""
     constraints = [
-        truncate_string(STRING_V, len(current)),
-        truncate_string(STRING_V, 46)  # Fuzz padding (field length = 46)
+        truncate_bytes(BYTES_V, len(current)),
+        truncate_bytes(BYTES_V, 46)  # Fuzz padding (field length = 46)
     ]
-    return selector(current, constraints, string_validator)
+    return selector(current, constraints, bytes_validator)
 
 
 def l1_entry(current):
Signature · 1 line
2.21.0
#10 lore raw

PATCH 7/10 image-fuzzer: Use bytes constant for field values

In reply to Eduardo Habkost

Field values are supposed to be bytes objects, not unicode strings. Change two constants that were declared as strings.

---
 tests/image-fuzzer/qcow2/layout.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tests/image-fuzzer/qcow2/layout.py +2 -2
index 0adcbd448d..a0fd53c7ad 100644
@@ -122,7 +122,7 @@ class Image(object):
     def create_header(self, cluster_bits, backing_file_name=None):
         """Generate a random valid header."""
         meta_header = [
-            ['>4s', 0, "QFI\xfb", 'magic'],
+            ['>4s', 0, b"QFI\xfb", 'magic'],
             ['>I', 4, random.randint(2, 3), 'version'],
             ['>Q', 8, 0, 'backing_file_offset'],
             ['>I', 16, 0, 'backing_file_size'],
@@ -231,7 +231,7 @@ class Image(object):
             feature_tables = []
             feature_ids = []
             inner_offset = self.ext_offset + ext_header_len
-            feat_name = 'some cool feature'
+            feat_name = b'some cool feature'
             while len(feature_tables) < num_fnt_entries * 3:
                 feat_type, feat_bit = gen_feat_ids()
                 # Remove duplicates
Signature · 1 line
2.21.0
Reviewed-by on v1 7/10
#11 lore raw

Re: [PATCH 07/10] image-fuzzer: Use bytes constant for field values

Quoted 23 lines · Eduardo Habkost

Field values are supposed to be bytes objects, not unicode strings. Change two constants that were declared as strings.

---
  tests/image-fuzzer/qcow2/layout.py | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
tests/image-fuzzer/qcow2/layout.py +1 -1
index 0adcbd448d..a0fd53c7ad 100644
@@ -122,7 +122,7 @@ class Image(object):
      def create_header(self, cluster_bits, backing_file_name=None):
          """Generate a random valid header."""
          meta_header = [
-            ['>4s', 0, "QFI\xfb", 'magic'],
+            ['>4s', 0, b"QFI\xfb", 'magic'],
              ['>I', 4, random.randint(2, 3), 'version'],
              ['>Q', 8, 0, 'backing_file_offset'],
              ['>I', 16, 0, 'backing_file_size'],

tests/image-fuzzer/qcow2/layout.py class Image(object): · line 237

Quoted 8 lines
tests/image-fuzzer/qcow2/layout.py +1 -1
@@ -231,7 +231,7 @@ class Image(object):
              feature_tables = []
              feature_ids = []
              inner_offset = self.ext_offset + ext_header_len
-            feat_name = 'some cool feature'
+            feat_name = b'some cool feature'
              while len(feature_tables) < num_fnt_entries * 3:
                  feat_type, feat_bit = gen_feat_ids()
                  # Remove duplicates
#12 lore raw

PATCH 8/10 image-fuzzer: Encode file name and file format to bytes

In reply to Eduardo Habkost

Callers of create_image() will pass strings as arguments, but the Image class will expect bytes objects to be provided. Encode them inside create_image().

---
 tests/image-fuzzer/qcow2/layout.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tests/image-fuzzer/qcow2/layout.py +2 -2
index a0fd53c7ad..01bff4d05e 100644
@@ -602,8 +602,8 @@ class Image(object):
 def create_image(test_img_path, backing_file_name=None, backing_file_fmt=None,
                  fields_to_fuzz=None):
     """Create a fuzzed image and write it to the specified file."""
-    image = Image(backing_file_name)
-    image.set_backing_file_format(backing_file_fmt)
+    image = Image(backing_file_name.encode())
+    image.set_backing_file_format(backing_file_fmt.encode())
     image.create_feature_name_table()
     image.set_end_of_extension_area()
     image.create_l_structures()
Signature · 1 line
2.21.0
#13 lore raw

PATCH 9/10 image-fuzzer: Run using python3

In reply to Eduardo Habkost

image-fuzzer is now supposed to be ready to run using Python 3. Remove the __future__ imports and change the interpreter line to "#!/usr/bin/env python3".

---
 tests/image-fuzzer/qcow2/__init__.py | 1 -
 tests/image-fuzzer/qcow2/layout.py   | 1 -
 tests/image-fuzzer/runner.py         | 3 +--
 3 files changed, 1 insertion(+), 4 deletions(-)
tests/image-fuzzer/qcow2/__init__.py +0 -1
index 09ef59821b..ed3af5da86 100644
@@ -1,2 +1 @@
-from __future__ import absolute_import
 from .layout import create_image
tests/image-fuzzer/qcow2/layout.py +0 -1
index 01bff4d05e..57ebe86e9a 100644
@@ -16,7 +16,6 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-from __future__ import absolute_import
 import random
 import struct
 from . import fuzz
tests/image-fuzzer/runner.py +1 -2
index 94cab5bd93..0793234815 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # Tool for running fuzz tests
 #
@@ -18,7 +18,6 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-from __future__ import print_function
 import sys
 import os
 import signal
Signature · 1 line
2.21.0
#14 lore raw

PATCH 10/10 image-fuzzer: Use errors parameter of subprocess.Popen()

In reply to Eduardo Habkost

Instead of manually encoding stderr and stdout output, use errors parameter of subprocess.Popen(). This will make process.communicate() return unicode strings instead of bytes objects.

---
 tests/image-fuzzer/runner.py | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)
tests/image-fuzzer/runner.py +4 -7
index 0793234815..4ba5c79e13 100755
@@ -79,16 +79,13 @@ def run_app(fd, q_args):
     devnull = open('/dev/null', 'r+')
     process = subprocess.Popen(q_args, stdin=devnull,
                                stdout=subprocess.PIPE,
-                               stderr=subprocess.PIPE)
+                               stderr=subprocess.PIPE,
+                               errors='replace')
     try:
         out, err = process.communicate()
         signal.alarm(0)
-        # fd is a text file, so we need to decode the process output before
-        # writing to it.
-        # We could be simply using the `errors` parameter of subprocess.Popen(),
-        # but this will be possible only after migrating to Python 3
-        fd.write(out.decode(errors='replace'))
-        fd.write(err.decode(errors='replace'))
+        fd.write(out)
+        fd.write(err)
         fd.flush()
         return process.returncode
 
Signature · 1 line
2.21.0
#15 lore raw
In reply to Eduardo Habkost
Quoted 14 lines · Eduardo Habkost

This series ports image-fuzzer to Python 3.

Eduardo Habkost (10):
image-fuzzer: Open image files in binary mode
image-fuzzer: Write bytes instead of string to image file
image-fuzzer: Explicitly use integer division operator
image-fuzzer: Use io.StringIO
image-fuzzer: Use %r for all fiels at Field.__repr__()
image-fuzzer: Return bytes objects on string fuzzing functions
image-fuzzer: Use bytes constant for field values
image-fuzzer: Encode file name and file format to bytes
image-fuzzer: Run using python3
image-fuzzer: Use errors parameter of subprocess.Popen()

Quoted 6 lines
 tests/image-fuzzer/qcow2/__init__.py |  1 -
 tests/image-fuzzer/qcow2/fuzz.py     | 54 +++++++++++++-------------
 tests/image-fuzzer/qcow2/layout.py   | 57 ++++++++++++++--------------
 tests/image-fuzzer/runner.py         | 12 +++---
 4 files changed, 61 insertions(+), 63 deletions(-)

When I gave my try at converting this to python3 I noticed that the "except OSError as e" segments used e[1] in a way that was not seemingly supported.

Did you fix that in this series or did I miss it?

--js

#16 lore raw
Quoted 24 lines · John Snow
Quoted 14 lines · Eduardo Habkost

This series ports image-fuzzer to Python 3.

Eduardo Habkost (10):
image-fuzzer: Open image files in binary mode
image-fuzzer: Write bytes instead of string to image file
image-fuzzer: Explicitly use integer division operator
image-fuzzer: Use io.StringIO
image-fuzzer: Use %r for all fiels at Field.__repr__()
image-fuzzer: Return bytes objects on string fuzzing functions
image-fuzzer: Use bytes constant for field values
image-fuzzer: Encode file name and file format to bytes
image-fuzzer: Run using python3
image-fuzzer: Use errors parameter of subprocess.Popen()

Quoted 6 lines
 tests/image-fuzzer/qcow2/__init__.py |  1 -
 tests/image-fuzzer/qcow2/fuzz.py     | 54 +++++++++++++-------------
 tests/image-fuzzer/qcow2/layout.py   | 57 ++++++++++++++--------------
 tests/image-fuzzer/runner.py         | 12 +++---
 4 files changed, 61 insertions(+), 63 deletions(-)
Quoted 5 lines

When I gave my try at converting this to python3 I noticed that the "except OSError as e" segments used e[1] in a way that was not seemingly supported.

Did you fix that in this series or did I miss it?

Good catch, I hadn't noticed that. I didn't fix it.

Signature · 1 line
Eduardo
#17 lore raw
Quoted 25 lines · Eduardo Habkost
Quoted 24 lines · John Snow
Quoted 20 lines · Eduardo Habkost

This series ports image-fuzzer to Python 3.

Eduardo Habkost (10):
image-fuzzer: Open image files in binary mode
image-fuzzer: Write bytes instead of string to image file
image-fuzzer: Explicitly use integer division operator
image-fuzzer: Use io.StringIO
image-fuzzer: Use %r for all fiels at Field.__repr__()
image-fuzzer: Return bytes objects on string fuzzing functions
image-fuzzer: Use bytes constant for field values
image-fuzzer: Encode file name and file format to bytes
image-fuzzer: Run using python3
image-fuzzer: Use errors parameter of subprocess.Popen()

 tests/image-fuzzer/qcow2/__init__.py |  1 -
 tests/image-fuzzer/qcow2/fuzz.py     | 54 +++++++++++++-------------
 tests/image-fuzzer/qcow2/layout.py   | 57 ++++++++++++++--------------
 tests/image-fuzzer/runner.py         | 12 +++---
 4 files changed, 61 insertions(+), 63 deletions(-)
Quoted 8 lines
Quoted 5 lines

When I gave my try at converting this to python3 I noticed that the "except OSError as e" segments used e[1] in a way that was not seemingly supported.

Did you fix that in this series or did I miss it?

Good catch, I hadn't noticed that. I didn't fix it.

I recommend using pylint(3) with a bunch of the style issues turned off, e.g.;

--disable=missing-docstring --disable=invalid-name

it will still whine about a lot of reasonably harmless stuff, but sometimes it has a few errors to show.

--js

#18 lore raw
In reply to Eduardo Habkost
Quoted 25 lines · Eduardo Habkost
Quoted 24 lines · John Snow
Quoted 20 lines · Eduardo Habkost

This series ports image-fuzzer to Python 3.

Eduardo Habkost (10):
image-fuzzer: Open image files in binary mode
image-fuzzer: Write bytes instead of string to image file
image-fuzzer: Explicitly use integer division operator
image-fuzzer: Use io.StringIO
image-fuzzer: Use %r for all fiels at Field.__repr__()
image-fuzzer: Return bytes objects on string fuzzing functions
image-fuzzer: Use bytes constant for field values
image-fuzzer: Encode file name and file format to bytes
image-fuzzer: Run using python3
image-fuzzer: Use errors parameter of subprocess.Popen()

 tests/image-fuzzer/qcow2/__init__.py |  1 -
 tests/image-fuzzer/qcow2/fuzz.py     | 54 +++++++++++++-------------
 tests/image-fuzzer/qcow2/layout.py   | 57 ++++++++++++++--------------
 tests/image-fuzzer/runner.py         | 12 +++---
 4 files changed, 61 insertions(+), 63 deletions(-)
Quoted 7 lines
Quoted 5 lines

When I gave my try at converting this to python3 I noticed that the "except OSError as e" segments used e[1] in a way that was not seemingly supported.

Did you fix that in this series or did I miss it?

Good catch, I hadn't noticed that. I didn't fix it.

Separate patch sent for that issue:
https://lore.kernel.org/qemu-devel/[email protected]/

Signature · 1 line
Eduardo
Reviewed-by
#19 lore raw
Quoted 30 lines · Eduardo Habkost
Quoted 29 lines · Eduardo Habkost
Quoted 28 lines · John Snow
Quoted 14 lines · Eduardo Habkost

This series ports image-fuzzer to Python 3.

Eduardo Habkost (10):
image-fuzzer: Open image files in binary mode
image-fuzzer: Write bytes instead of string to image file
image-fuzzer: Explicitly use integer division operator
image-fuzzer: Use io.StringIO
image-fuzzer: Use %r for all fiels at Field.__repr__()
image-fuzzer: Return bytes objects on string fuzzing functions
image-fuzzer: Use bytes constant for field values
image-fuzzer: Encode file name and file format to bytes
image-fuzzer: Run using python3
image-fuzzer: Use errors parameter of subprocess.Popen()

Quoted 6 lines
 tests/image-fuzzer/qcow2/__init__.py |  1 -
 tests/image-fuzzer/qcow2/fuzz.py     | 54 +++++++++++++-------------
 tests/image-fuzzer/qcow2/layout.py   | 57 ++++++++++++++--------------
 tests/image-fuzzer/runner.py         | 12 +++---
 4 files changed, 61 insertions(+), 63 deletions(-)

When I gave my try at converting this to python3 I noticed that the "except OSError as e" segments used e[1] in a way that was not seemingly supported.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Did you fix that in this series or did I miss it?

Good catch, I hadn't noticed that. I didn't fix it.

Separate patch sent for that issue:
https://lore.kernel.org/qemu-devel/[email protected]/

With that squished in somewhere into this series:

#20 lore raw
In reply to Eduardo Habkost
Quoted 14 lines · Eduardo Habkost

This series ports image-fuzzer to Python 3.

Eduardo Habkost (10):
image-fuzzer: Open image files in binary mode
image-fuzzer: Write bytes instead of string to image file
image-fuzzer: Explicitly use integer division operator
image-fuzzer: Use io.StringIO
image-fuzzer: Use %r for all fiels at Field.__repr__()
image-fuzzer: Return bytes objects on string fuzzing functions
image-fuzzer: Use bytes constant for field values
image-fuzzer: Encode file name and file format to bytes
image-fuzzer: Run using python3
image-fuzzer: Use errors parameter of subprocess.Popen()

Quoted 10 lines
 tests/image-fuzzer/qcow2/__init__.py |  1 -
 tests/image-fuzzer/qcow2/fuzz.py     | 54 +++++++++++++-------------
 tests/image-fuzzer/qcow2/layout.py   | 57 ++++++++++++++--------------
 tests/image-fuzzer/runner.py         | 12 +++---
 4 files changed, 61 insertions(+), 63 deletions(-)
Signature · 1 line
2.21.0

Thanks, applied to my block tree:
https://github.com/stefanha/qemu/commits/block

Stefan