Skip to contentSkip to conversation

support/testing/tests/fs/test_ext: specify 16 MB as ext filesystem size

PATCH New Thomas Petazzoni 2 participants 8 messages Jun 21, 2021 – Jun 26, 2021

lore
Share & tools
lore
#0 lore raw

PATCH 1/4 support/testing/tests/fs/test_ext: specify 16 MB as ext filesystem size

Since Qemu 5.1, the SD card size must be a power of two, so the default size for ext2/3/4 filesystem images of 60 MB is not suitable. Since 16 MB is used for the Ext4 test, let's use the same value for the other tests as well (ext2, ext2r1, ext3). Without this change, the ext2, ext2r1 and ext3 simply fail to run under Qemu >= 5.1.

---
 support/testing/tests/fs/test_ext.py | 3 +++
 1 file changed, 3 insertions(+)
support/testing/tests/fs/test_ext.py +3 -0
index 16b308cf6d..0fe8af7e6d 100644
@@ -43,6 +43,7 @@ class TestExt2(infra.basetest.BRTest):
         BR2_TARGET_ROOTFS_EXT2=y
         BR2_TARGET_ROOTFS_EXT2_2r0=y
         BR2_TARGET_ROOTFS_EXT2_LABEL="foobaz"
+        BR2_TARGET_ROOTFS_EXT2_SIZE="16384"
         # BR2_TARGET_ROOTFS_TAR is not set
         """
 
@@ -62,6 +63,7 @@ class TestExt2r1(infra.basetest.BRTest):
         BR2_TARGET_ROOTFS_EXT2=y
         BR2_TARGET_ROOTFS_EXT2_2r1=y
         BR2_TARGET_ROOTFS_EXT2_LABEL="foobar"
+        BR2_TARGET_ROOTFS_EXT2_SIZE="16384"
         # BR2_TARGET_ROOTFS_TAR is not set
         """
 
@@ -81,6 +83,7 @@ class TestExt3(infra.basetest.BRTest):
         """
         BR2_TARGET_ROOTFS_EXT2=y
         BR2_TARGET_ROOTFS_EXT2_3=y
+        BR2_TARGET_ROOTFS_EXT2_SIZE="16384"
         # BR2_TARGET_ROOTFS_TAR is not set
         """
 
Signature · 1 line
2.31.1
#1 lore raw

PATCH 2/4 support/testing/infra: add img_round_power2() function

Since Qemu 5.1, SD card images must have a size that are a power of two. While some filesystem (such as ext2/3/4) allow to specify the expected size of the filesystem, others such as SquashFS do not have this capability.

We were already extending the size of such images to the next 1 MB boundary using "truncate -s %1M", but that is no longer sufficient. So instead, we introduce a helper function that extends the size of an image to the next power of two.

---
 support/testing/infra/__init__.py | 11 +++++++++++
 1 file changed, 11 insertions(+)
support/testing/infra/__init__.py +11 -0
index 6522a265f3..c540fd1940 100644
@@ -112,3 +112,14 @@ def get_elf_prog_interpreter(builddir, prefix, fpath):
             continue
         return m.group(1)
     return None
+
+
+def img_round_power2(img):
+    """
+    Rounds up the size of an image file to the next power of 2
+    """
+    sz = os.stat(img).st_size
+    pow2 = 1
+    while pow2 < sz:
+        pow2 = pow2 << 1
+    subprocess.call(["truncate", "-s", str(pow2), img])
Signature · 1 line
2.31.1
#2 lore raw

PATCH 2/4 support/testing/infra: add img_round_power2() function

Thomas, All,

On 2021-06-21 22:25 +0200, Thomas Petazzoni spake thusly:

Quoted 24 lines

Since Qemu 5.1, SD card images must have a size that are a power of two. While some filesystem (such as ext2/3/4) allow to specify the expected size of the filesystem, others such as SquashFS do not have this capability.

We were already extending the size of such images to the next 1 MB boundary using "truncate -s %1M", but that is no longer sufficient. So instead, we introduce a helper function that extends the size of an image to the next power of two.

---
 support/testing/infra/__init__.py | 11 +++++++++++
 1 file changed, 11 insertions(+)
support/testing/infra/__init__.py +1 -0
index 6522a265f3..c540fd1940 100644
@@ -112,3 +112,14 @@ def get_elf_prog_interpreter(builddir, prefix, fpath):
             continue
         return m.group(1)
     return None
+

support/testing/infra/__init__.py def get_elf_prog_interpreter(builddir, prefix, fpath): · line 125

Quoted 10 lines
support/testing/infra/__init__.py +10 -0
+
+def img_round_power2(img):
+    """
+    Rounds up the size of an image file to the next power of 2
+    """
+    sz = os.stat(img).st_size
+    pow2 = 1
+    while pow2 < sz:
+        pow2 = pow2 << 1
+    subprocess.call(["truncate", "-s", str(pow2), img])

This subprocess call is not very pythonic, so I've changed that to:

with open(img, 'ab') as f:
f.truncate(pow2)

Applied to master, thanks.

Regards,
Yann E. MORIN.

Quoted 7 lines
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#3 lore raw

PATCH 3/4 support/testing/tests: fix tests to use infra.img_round_power2()

In reply to Thomas Petazzoni

All the tests that are using if=sd as a Qemu options are changed to use infra.img_round_power2() instead of simply extending the size of the image to the next MB boundary, which is not longer sufficient with Qemu >= 5.1.

---
 support/testing/tests/core/test_file_capabilities.py | 2 +-
 support/testing/tests/fs/test_squashfs.py            | 2 +-
 support/testing/tests/init/base.py                   | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
support/testing/tests/core/test_file_capabilities.py +1 -1
index b9ece18d7b..bd46f5138d 100644
@@ -28,7 +28,7 @@ class TestFileCapabilities(infra.basetest.BRTest):
 
     def test_run(self):
         img = os.path.join(self.builddir, "images", "rootfs.squashfs")
-        subprocess.call(["truncate", "-s", "%1M", img])
+        infra.img_round_power2(img)
 
         self.emulator.boot(arch="armv7",
                            kernel=os.path.join(self.builddir, "images", "zImage"),
support/testing/tests/fs/test_squashfs.py +1 -1
index 234f4944be..7d6297ec5e 100644
@@ -22,7 +22,7 @@ class TestSquashfs(infra.basetest.BRTest):
         self.assertEqual(out[3], "Compression lz4")
 
         img = os.path.join(self.builddir, "images", "rootfs.squashfs")
-        subprocess.call(["truncate", "-s", "%1M", img])
+        infra.img_round_power2(img)
 
         self.emulator.boot(arch="armv7",
                            kernel="builtin",
support/testing/tests/init/base.py +1 -1
index 75cfbe9c59..7ed035263b 100644
@@ -7,7 +7,7 @@ class InitSystemBase(infra.basetest.BRTest):
 
     def start_emulator(self, fs_type, kernel=None, dtb=None, init=None):
         img = os.path.join(self.builddir, "images", "rootfs.{}".format(fs_type))
-        subprocess.call(["truncate", "-s", "%1M", img])
+        infra.img_round_power2(img)
 
         options = ["-drive",
                    "file={},if=sd,format=raw".format(img),
Signature · 1 line
2.31.1
#4 lore raw

PATCH 3/4 support/testing/tests: fix tests to use infra.img_round_power2()

Thomas, All,

On 2021-06-21 22:25 +0200, Thomas Petazzoni spake thusly:

Quoted 30 lines

All the tests that are using if=sd as a Qemu options are changed to use infra.img_round_power2() instead of simply extending the size of the image to the next MB boundary, which is not longer sufficient with Qemu >= 5.1.

---
 support/testing/tests/core/test_file_capabilities.py | 2 +-
 support/testing/tests/fs/test_squashfs.py            | 2 +-
 support/testing/tests/init/base.py                   | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
support/testing/tests/core/test_file_capabilities.py +1 -1
index b9ece18d7b..bd46f5138d 100644
@@ -28,7 +28,7 @@ class TestFileCapabilities(infra.basetest.BRTest):
 
     def test_run(self):
         img = os.path.join(self.builddir, "images", "rootfs.squashfs")
-        subprocess.call(["truncate", "-s", "%1M", img])
+        infra.img_round_power2(img)
 
         self.emulator.boot(arch="armv7",
                            kernel=os.path.join(self.builddir, "images", "zImage"),
support/testing/tests/fs/test_squashfs.py +0 -0
index 234f4944be..7d6297ec5e 100644

support/testing/tests/fs/test_squashfs.py class TestSquashfs(infra.basetest.BRTest): · line 25

Quoted 5 lines
support/testing/tests/fs/test_squashfs.py +1 -1
@@ -22,7 +22,7 @@ class TestSquashfs(infra.basetest.BRTest):
         self.assertEqual(out[3], "Compression lz4")
 
         img = os.path.join(self.builddir, "images", "rootfs.squashfs")
-        subprocess.call(["truncate", "-s", "%1M", img])
+        infra.img_round_power2(img)

Not the fault of this patch, but still your fault: this squashfs test no longer works after the kernel update in 3cf2782906d5...

I temporarily switched it back to using gxip to test it (but did not commit that change).

Applied to master, thanks.

Regards,
Yann E. MORIN.

Quoted 22 lines
support/testing/tests/fs/test_squashfs.py +0 -0
         self.emulator.boot(arch="armv7",
                            kernel="builtin",
support/testing/tests/init/base.py +1 -1
index 75cfbe9c59..7ed035263b 100644
@@ -7,7 +7,7 @@ class InitSystemBase(infra.basetest.BRTest):
 
     def start_emulator(self, fs_type, kernel=None, dtb=None, init=None):
         img = os.path.join(self.builddir, "images", "rootfs.{}".format(fs_type))
-        subprocess.call(["truncate", "-s", "%1M", img])
+        infra.img_round_power2(img)
 
         options = ["-drive",
                    "file={},if=sd,format=raw".format(img),
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#5 lore raw

PATCH 4/4 support/testing/tests/fs/test_ext: add missing "format" option for Qemu

In reply to Thomas Petazzoni

Will avoid the following warning:

WARNING: Image format was not specified for '/home/thomas/projets/outputs/TestExt3/images/rootfs.ext3' and probing guessed raw. Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted. Specify the 'raw' format explicitly to remove the restrictions.

---
 support/testing/tests/fs/test_ext.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
support/testing/tests/fs/test_ext.py +1 -1
index 0fe8af7e6d..2a6e0c4274 100644
@@ -31,7 +31,7 @@ def boot_img_and_check_fs_type(emulator, builddir, fs_type):
                   kernel="builtin",
                   kernel_cmdline=["root=/dev/mmcblk0",
                                   "rootfstype={}".format(fs_type)],
-                  options=["-drive", "file={},if=sd".format(img)])
+                  options=["-drive", "file={},if=sd,format=raw".format(img)])
     emulator.login()
     _, exit_code = emulator.run(CHECK_FS_TYPE_CMD.format(fs_type))
     return exit_code
Signature · 1 line
2.31.1
#6 lore raw

PATCH 4/4 support/testing/tests/fs/test_ext: add missing "format" option for Qemu

Thomas, All,

On 2021-06-21 22:25 +0200, Thomas Petazzoni spake thusly:

Quoted 10 lines

Will avoid the following warning:

WARNING: Image format was not specified for '/home/thomas/projets/outputs/TestExt3/images/rootfs.ext3' and probing guessed raw. Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted. Specify the 'raw' format explicitly to remove the restrictions.

Applied to master, thanks.

Regards,
Yann E. MORIN.

Quoted 24 lines
---
 support/testing/tests/fs/test_ext.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
support/testing/tests/fs/test_ext.py +1 -1
index 0fe8af7e6d..2a6e0c4274 100644
@@ -31,7 +31,7 @@ def boot_img_and_check_fs_type(emulator, builddir, fs_type):
                   kernel="builtin",
                   kernel_cmdline=["root=/dev/mmcblk0",
                                   "rootfstype={}".format(fs_type)],
-                  options=["-drive", "file={},if=sd".format(img)])
+                  options=["-drive", "file={},if=sd,format=raw".format(img)])
     emulator.login()
     _, exit_code = emulator.run(CHECK_FS_TYPE_CMD.format(fs_type))
     return exit_code
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#7 lore raw

PATCH 1/4 support/testing/tests/fs/test_ext: specify 16 MB as ext filesystem size

In reply to Thomas Petazzoni

Thomas, All,

On 2021-06-21 22:25 +0200, Thomas Petazzoni spake thusly:

Quoted 8 lines

Since Qemu 5.1, the SD card size must be a power of two, so the default size for ext2/3/4 filesystem images of 60 MB is not suitable. Since 16 MB is used for the Ext4 test, let's use the same value for the other tests as well (ext2, ext2r1, ext3). Without this change, the ext2, ext2r1 and ext3 simply fail to run under Qemu >= 5.1.

Applied to master, thanks.

Regards,
Yann E. MORIN.

Quoted 39 lines
---
 support/testing/tests/fs/test_ext.py | 3 +++
 1 file changed, 3 insertions(+)
support/testing/tests/fs/test_ext.py +3 -0
index 16b308cf6d..0fe8af7e6d 100644
@@ -43,6 +43,7 @@ class TestExt2(infra.basetest.BRTest):
         BR2_TARGET_ROOTFS_EXT2=y
         BR2_TARGET_ROOTFS_EXT2_2r0=y
         BR2_TARGET_ROOTFS_EXT2_LABEL="foobaz"
+        BR2_TARGET_ROOTFS_EXT2_SIZE="16384"
         # BR2_TARGET_ROOTFS_TAR is not set
         """
 
@@ -62,6 +63,7 @@ class TestExt2r1(infra.basetest.BRTest):
         BR2_TARGET_ROOTFS_EXT2=y
         BR2_TARGET_ROOTFS_EXT2_2r1=y
         BR2_TARGET_ROOTFS_EXT2_LABEL="foobar"
+        BR2_TARGET_ROOTFS_EXT2_SIZE="16384"
         # BR2_TARGET_ROOTFS_TAR is not set
         """
 
@@ -81,6 +83,7 @@ class TestExt3(infra.basetest.BRTest):
         """
         BR2_TARGET_ROOTFS_EXT2=y
         BR2_TARGET_ROOTFS_EXT2_3=y
+        BR2_TARGET_ROOTFS_EXT2_SIZE="16384"
         # BR2_TARGET_ROOTFS_TAR is not set
         """
 
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'