Fix 3 Python test cases
cover Fix 3 Python test cases
Hello,
This small patch series fixes 3 Python test cases. They fail because we use an initramfs in all Python tests (and in many other tests). The issue of initramfs is that if the initramfs filesystem grows too large, the kernel fails to decompress it, and simply continues booting with a partially uncompressed initramfs: some files are simply missing.
In this patch series, we fix just the 3 failing test cases, for the sake of having a reasonably minimal solution to apply in master. We switch those test cases to use an initramfs to using an ext2 image that is exposed by Qemu as a regular hard-drive.
Long term, we probably want to switch all tests to using this strategy, so that we don't face a weird limit caused by initramfs. That being said, there is no silver bullet: even if we switch to ext2, we will have to define, for each test, the size of the ext2 filesystem image. So there will still be some sort of arbitrary limit that will have to be tweaked from time to time as a given test grows in size (more dependencies, dependencies growing, etc.).
Best regards,
Thomas Petazzoni
Thomas Petazzoni (3):
support/testing/tests/package/test_python_boto3: use ext2 instead of
cpio
support/testing/tests/package/test_python_botocore: use ext2 instead
of cpio
support/testing/tests/package/test_python_s3transfer: use ext2 instead
of cpio
support/testing/tests/package/test_python_boto3.py | 12 ++++++++++++ .../testing/tests/package/test_python_botocore.py | 12 ++++++++++++ .../testing/tests/package/test_python_s3transfer.py | 12 ++++++++++++ 3 files changed, 36 insertions(+)
Signature · 6 lines
2.37.2 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH 1/3 support/testing/tests/package/test_python_boto3: use ext2 instead of cpio
The CPIO filesystem generated by the test_python_boto3 test is too large, and doesn't fit as an initramfs in the 256MB of RAM available in the versatilepb machine. This causes a "Initramfs unpacking failed:
write error" when booting, and many files being missing from the root filesystem, ultimately causing the test to fail.
It would make sense to switch all test cases to use ext2 + a hard-drive, but for now, let's fix the few test cases that are causing problems.
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/2884635041
--- support/testing/tests/package/test_python_boto3.py | 12 ++++++++++++ 1 file changed, 12 insertions(+)
support/testing/tests/package/test_python_boto3.py +12 -0
@@ -1,4 +1,5 @@ from tests.package.test_python import TestPythonPackageBase +import os class TestPythonPy3Boto3(TestPythonPackageBase):
@@ -7,6 +8,17 @@ class TestPythonPy3Boto3(TestPythonPackageBase): """ BR2_PACKAGE_PYTHON3=y BR2_PACKAGE_PYTHON_BOTO3=y + BR2_TARGET_ROOTFS_EXT2=y + BR2_TARGET_ROOTFS_EXT2_SIZE="120M" + # BR2_TARGET_ROOTFS_TAR is not set """ sample_scripts = ["tests/package/sample_python_boto3.py"] timeout = 10 + + def login(self): + ext2_file = os.path.join(self.builddir, "images", "rootfs.ext2") + self.emulator.boot(arch="armv5", + kernel="builtin", + options=["-drive", "file=%s,if=scsi,format=raw" % ext2_file], + kernel_cmdline=["rootwait", "root=/dev/sda"]) + self.emulator.login()
Signature · 6 lines
2.37.2 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 1/3] support/testing/tests/package/test_python_boto3: use ext2 instead of cpio
Thomas, All,
On 2022-08-15 23:48 +0200, Thomas Petazzoni via buildroot spake thusly:
Quoted 30 lines
The CPIO filesystem generated by the
test_python_boto3test is too large, and doesn't fit as an initramfs in the 256MB of RAM available in the versatilepb machine. This causes a "Initramfs unpacking failed:
write error" when booting, and many files being missing from the root filesystem, ultimately causing the test to fail.It would make sense to switch all test cases to use ext2 + a hard-drive, but for now, let's fix the few test cases that are causing problems.
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/2884635041
Signed-off-by Thomas Petazzoni <[email protected]>--- support/testing/tests/package/test_python_boto3.py | 12 ++++++++++++ 1 file changed, 12 insertions(+)support/testing/tests/package/test_python_boto3.py +1
-0@@ -1,4 +1,5 @@ from tests.package.test_python import TestPythonPackageBase +import os class TestPythonPy3Boto3(TestPythonPackageBase):
support/testing/tests/package/test_python_boto3.py class TestPythonPy3Boto3(TestPythonPackageBase): · line 13
Quoted 6 lines
support/testing/tests/package/test_python_boto3.py +3
-0@@ -7,6 +8,17 @@ class TestPythonPy3Boto3(TestPythonPackageBase): """ BR2_PACKAGE_PYTHON3=y BR2_PACKAGE_PYTHON_BOTO3=y + BR2_TARGET_ROOTFS_EXT2=y + BR2_TARGET_ROOTFS_EXT2_SIZE="120M" + # BR2_TARGET_ROOTFS_TAR is not set
BR2_TARGET_ROOTFS_TAR is already unset in TestPythonPackageBase.config, so disabling it here is superfluous.
But this will however still build the cpio archive, because it is enabled in TestPythonPackageBase.config.
We can disable it with:
class TestPythonPy3Boto3(TestPythonPackageBase):
config = TestPythonPackageBase.config.replace('BR2_TARGET_ROOTFS_CPIO=y\n', '') + \
"""
blablabla
"""If you believe that it is acceptable, I can fix when applying. Or we can continue to uselessly build the cpio archive, which does not take very long to assemble...
Regards,
Yann E. MORIN.
Quoted 18 lines
support/testing/tests/package/test_python_boto3.py +8
-0""" sample_scripts = ["tests/package/sample_python_boto3.py"] timeout = 10 + + def login(self): + ext2_file = os.path.join(self.builddir, "images", "rootfs.ext2") + self.emulator.boot(arch="armv5", + kernel="builtin", + options=["-drive", "file=%s,if=scsi,format=raw" % ext2_file], + kernel_cmdline=["rootwait", "root=/dev/sda"]) + self.emulator.login()Signature · 6 lines
2.37.2 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 1/3] support/testing/tests/package/test_python_boto3: use ext2 instead of cpio
Hello,
Quoted 11 lines · On Tue, 16 Aug 2022 18:26:26 +0200 "Yann E. MORIN" <[email protected]>…
support/testing/tests/package/test_python_boto3.py class TestPythonPy3Boto3(TestPythonPackageBase): · line 13
Quoted 8 lines
class TestPythonPy3Boto3(TestPythonPackageBase):
support/testing/tests/package/test_python_boto3.py +3
-0@@ -7,6 +8,17 @@ class TestPythonPy3Boto3(TestPythonPackageBase): """ BR2_PACKAGE_PYTHON3=y BR2_PACKAGE_PYTHON_BOTO3=y + BR2_TARGET_ROOTFS_EXT2=y + BR2_TARGET_ROOTFS_EXT2_SIZE="120M" + # BR2_TARGET_ROOTFS_TAR is not set
BR2_TARGET_ROOTFS_TARis already unset in TestPythonPackageBase.config, so disabling it here is superfluous.
Ah indeed.
Quoted 11 lines
But this will however still build the cpio archive, because it is enabled in TestPythonPackageBase.config.
We can disable it with:
class TestPythonPy3Boto3(TestPythonPackageBase): config = TestPythonPackageBase.config.replace('BR2_TARGET_ROOTFS_CPIO=y\n', '') + \ """ blablabla """
Quoted 3 lines
If you believe that it is acceptable, I can fix when applying. Or we can continue to uselessly build the cpio archive, which does not take very long to assemble...
I think we can continue to uselessly build the cpio archive. As I said, my long term proposal would rather be to switch all tests to use ext2 + a hard drive. We still need to define an image size, but at least we can a hard and clear error if the filesystem no longer fits rather than these weird errors caused by initramfs being partially uncompressed.
Thanks for the review!
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH 2/3 support/testing/tests/package/test_python_botocore: use ext2 instead of cpio
In reply to Thomas Petazzoni via buildrootThe CPIO filesystem generated by the test_python_botocore test is too large, and doesn't fit as an initramfs in the 256MB of RAM available in the versatilepb machine. This causes a "Initramfs unpacking failed:
write error" when booting, and many files being missing from the root filesystem, ultimately causing the test to fail.
It would make sense to switch all test cases to use ext2 + a hard-drive, but for now, let's fix the few test cases that are causing problems.
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/2884635042
--- .../testing/tests/package/test_python_botocore.py | 12 ++++++++++++ 1 file changed, 12 insertions(+)
support/testing/tests/package/test_python_botocore.py +12 -0
@@ -1,4 +1,5 @@ from tests.package.test_python import TestPythonPackageBase +import os class TestPythonPy3Botocore(TestPythonPackageBase):
@@ -7,6 +8,17 @@ class TestPythonPy3Botocore(TestPythonPackageBase): """ BR2_PACKAGE_PYTHON3=y BR2_PACKAGE_PYTHON_BOTOCORE=y + BR2_TARGET_ROOTFS_EXT2=y + BR2_TARGET_ROOTFS_EXT2_SIZE="120M" + # BR2_TARGET_ROOTFS_TAR is not set """ sample_scripts = ["tests/package/sample_python_botocore.py"] timeout = 10 + + def login(self): + ext2_file = os.path.join(self.builddir, "images", "rootfs.ext2") + self.emulator.boot(arch="armv5", + kernel="builtin", + options=["-drive", "file=%s,if=scsi,format=raw" % ext2_file], + kernel_cmdline=["rootwait", "root=/dev/sda"]) + self.emulator.login()
Signature · 6 lines
2.37.2 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH 3/3 support/testing/tests/package/test_python_s3transfer: use ext2 instead of cpio
In reply to Thomas Petazzoni via buildrootThe CPIO filesystem generated by the test_python_s3transfer test is too large, and doesn't fit as an initramfs in the 256MB of RAM available in the versatilepb machine. This causes a "Initramfs unpacking failed: write error" when booting, and many files being missing from the root filesystem, ultimately causing the test to fail.
It would make sense to switch all test cases to use ext2 + a hard-drive, but for now, let's fix the few test cases that are causing problems.
Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/2884635126
--- .../testing/tests/package/test_python_s3transfer.py | 12 ++++++++++++ 1 file changed, 12 insertions(+)
support/testing/tests/package/test_python_s3transfer.py +12 -0
@@ -1,4 +1,5 @@ from tests.package.test_python import TestPythonPackageBase +import os class TestPythonPy3S3transfer(TestPythonPackageBase):
@@ -7,6 +8,17 @@ class TestPythonPy3S3transfer(TestPythonPackageBase): """ BR2_PACKAGE_PYTHON3=y BR2_PACKAGE_PYTHON_S3TRANSFER=y + BR2_TARGET_ROOTFS_EXT2=y + BR2_TARGET_ROOTFS_EXT2_SIZE="120M" + # BR2_TARGET_ROOTFS_TAR is not set """ sample_scripts = ["tests/package/sample_python_s3transfer.py"] timeout = 10 + + def login(self): + ext2_file = os.path.join(self.builddir, "images", "rootfs.ext2") + self.emulator.boot(arch="armv5", + kernel="builtin", + options=["-drive", "file=%s,if=scsi,format=raw" % ext2_file], + kernel_cmdline=["rootwait", "root=/dev/sda"]) + self.emulator.login()
Signature · 6 lines
2.37.2 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Thomas, All,
On 2022-08-15 23:48 +0200, Thomas Petazzoni via buildroot spake thusly:
[--SNIP--]
Quoted 7 lines
Thomas Petazzoni (3):
support/testing/tests/package/test_python_boto3: use ext2 instead of
cpio
support/testing/tests/package/test_python_botocore: use ext2 instead
of cpio
support/testing/tests/package/test_python_s3transfer: use ext2 instead
of cpio
All threeour applied to master, after removing the superfluous line disabling rootfs-tar.
Regards,
Yann E. MORIN.
Quoted 12 lines
support/testing/tests/package/test_python_boto3.py | 12 ++++++++++++ .../testing/tests/package/test_python_botocore.py | 12 ++++++++++++ .../testing/tests/package/test_python_s3transfer.py | 12 ++++++++++++ 3 files changed, 36 insertions(+)Signature · 6 lines
2.37.2 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot