Skip to contentSkip to conversation

Acceptance Tests: Test the Raspberry Pi 2 board

PATCH In review Philippe Mathieu-Daudé 3 participants 13 messages Mar 12, 2019 – Jun 5, 2019

lore
Share & tools
lore
#0 lore raw

v2 cover Acceptance Tests: Test the Raspberry Pi 2 board

Add the raspi2 to the list of boards tested by the boot_linux_console Avocado test.

Based on "Acceptance Tests: target architecture support" v5:
https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg04111.html
Based-on: <[email protected]>

Philippe Mathieu-Daudé (2):
BootLinuxConsoleTest: Let extract_from_deb handle various compressions Boot Linux Console Test: add a test for the Raspberry Pi 2

 tests/acceptance/boot_linux_console.py | 34 ++++++++++++++++++++++++--
 1 file changed, 32 insertions(+), 2 deletions(-)
Signature · 1 line
2.20.1
#1 lore raw

PATCH v2 1/2 BootLinuxConsoleTest: Let extract_from_deb handle various compressions

Debian binary package format supports various compressions.

Per man deb(5):

NAME
deb - Debian binary package format

FORMAT
...
The third, last required member is named data.tar. It contains the filesystem as a tar archive, either not compressed (supported since dpkg 1.10.24), or compressed with gzip (with .gz extension), xz (with .xz extension, supported since dpkg 1.15.6), bzip2 (with .bz2 extension, supported since dpkg 1.10.24) or lzma (with .lzma extension, supported since dpkg 1.13.25).

List the archive files to have the 3rd name with the correct extension.

The function avocado.utils.archive.extract() will handle the different compression format for us.

---
v2: Use process.run() (Cleber)

---
 tests/acceptance/boot_linux_console.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
tests/acceptance/boot_linux_console.py +3 -2
index 23fca39422..0936589fd8 100644
@@ -55,8 +55,9 @@ class BootLinuxConsole(Test):
         """
         cwd = os.getcwd()
         os.chdir(self.workdir)
-        process.run("ar x %s data.tar.gz" % deb)
-        archive.extract("data.tar.gz", self.workdir)
+        file_path = process.run("ar t %s" % deb).stdout_text.split()[2]
+        process.run("ar x %s %s" % (deb, file_path))
+        archive.extract(file_path, self.workdir)
         os.chdir(cwd)
         return self.workdir + path
 
Signature · 1 line
2.20.1
Reviewed-by on v2 1/2
#2 lore raw

Re: [Qemu-devel] [PATCH v2 1/2] BootLinuxConsoleTest: Let extract_from_deb handle various compressions

Quoted 16 lines · Philippe Mathieu-Daudé

Debian binary package format supports various compressions.

Per man deb(5):

NAME
deb - Debian binary package format

FORMAT
...
The third, last required member is named data.tar. It contains the filesystem as a tar archive, either not compressed (supported since dpkg 1.10.24), or compressed with gzip (with .gz extension), xz (with .xz extension, supported since dpkg 1.15.6), bzip2 (with .bz2 extension, supported since dpkg 1.10.24) or lzma (with .lzma extension, supported since dpkg 1.13.25).

Quoted 6 lines

List the archive files to have the 3rd name with the correct extension.

The function avocado.utils.archive.extract() will handle the different compression format for us.

#3 lore raw

PATCH v2 2/2 Boot Linux Console Test: add a test for the Raspberry Pi 2

In reply to Philippe Mathieu-Daudé

From: Philippe Mathieu-Daudé <[email protected]>

Similar to the x86_64/pc test, it boots a Linux kernel on a raspi2 board and verify the serial is working.

If ARM is a target being built, "make check-acceptance" will automatically include this test by the use of the "arch:arm" tags.

Alternatively, this test can be run using:

    $ avocado run -t arch:arm tests/acceptance
    $ avocado run -t machine:raspi2 tests/acceptance
---
 tests/acceptance/boot_linux_console.py | 29 ++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)
tests/acceptance/boot_linux_console.py +29 -0
index 0936589fd8..dd2733b55f 100644
@@ -197,6 +197,35 @@ class BootLinuxConsole(Test):
         console_pattern = 'Kernel command line: %s' % kernel_command_line
         self.wait_for_console_pattern(console_pattern)
 
+    def test_arm_raspi2(self):
+        """
+        The kernel can be rebuilt using the kernel source referenced
+        and following the instructions on the on:
+        https://www.raspberrypi.org/documentation/linux/kernel/building.md
+
+        :avocado: tags=arch:arm
+        :avocado: tags=machine:raspi2
+        """
+        deb_url = ('http://archive.raspberrypi.org/debian/'
+                   'pool/main/r/raspberrypi-firmware/'
+                   'raspberrypi-kernel_1.20190215-1_armhf.deb')
+        deb_hash = 'cd284220b32128c5084037553db3c482426f3972'
+        deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
+        print("deb_path: ", deb_path)
+        kernel_path = self.extract_from_deb(deb_path, '/boot/kernel7.img')
+        dtb_path = self.extract_from_deb(deb_path, '/boot/bcm2709-rpi-2-b.dtb')
+
+        self.vm.set_machine('raspi2')
+        self.vm.set_console()
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
+                               'console=ttyAMA0')
+        self.vm.add_args('-kernel', kernel_path,
+                         '-dtb', dtb_path,
+                         '-append', kernel_command_line)
+        self.vm.launch()
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+        self.wait_for_console_pattern(console_pattern)
+
     def test_s390x_s390_ccw_virtio(self):
         """
         :avocado: tags=arch:s390x
Signature · 1 line
2.20.1
#4 lore raw

Re: [Qemu-devel] [PATCH v2 2/2] Boot Linux Console Test: add a test for the Raspberry Pi 2

Quoted 31 lines · Philippe Mathieu-Daudé

From: Philippe Mathieu-Daudé <[email protected]>

Similar to the x86_64/pc test, it boots a Linux kernel on a raspi2 board and verify the serial is working.

If ARM is a target being built, "make check-acceptance" will automatically include this test by the use of the "arch:arm" tags.

Alternatively, this test can be run using:

    $ avocado run -t arch:arm tests/acceptance
    $ avocado run -t machine:raspi2 tests/acceptance
---
 tests/acceptance/boot_linux_console.py | 29 ++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)
tests/acceptance/boot_linux_console.py +5 -0
index 0936589fd8..dd2733b55f 100644
@@ -197,6 +197,35 @@ class BootLinuxConsole(Test):
         console_pattern = 'Kernel command line: %s' % kernel_command_line
         self.wait_for_console_pattern(console_pattern)
 
+    def test_arm_raspi2(self):
+        """
+        The kernel can be rebuilt using the kernel source referenced
+        and following the instructions on the on:
+        https://www.raspberrypi.org/documentation/linux/kernel/building.md

tests/acceptance/boot_linux_console.py class BootLinuxConsole(Test): · line 214

Quoted 10 lines
tests/acceptance/boot_linux_console.py +10 -0
+
+        :avocado: tags=arch:arm
+        :avocado: tags=machine:raspi2
+        """
+        deb_url = ('http://archive.raspberrypi.org/debian/'
+                   'pool/main/r/raspberrypi-firmware/'
+                   'raspberrypi-kernel_1.20190215-1_armhf.deb')
+        deb_hash = 'cd284220b32128c5084037553db3c482426f3972'
+        deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
+        print("deb_path: ", deb_path)

This looks like a left over you forgot to remove.

Quoted 20 lines
tests/acceptance/boot_linux_console.py +14 -0
+        kernel_path = self.extract_from_deb(deb_path, '/boot/kernel7.img')
+        dtb_path = self.extract_from_deb(deb_path, '/boot/bcm2709-rpi-2-b.dtb')
+
+        self.vm.set_machine('raspi2')
+        self.vm.set_console()
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
+                               'console=ttyAMA0')
+        self.vm.add_args('-kernel', kernel_path,
+                         '-dtb', dtb_path,
+                         '-append', kernel_command_line)
+        self.vm.launch()
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+        self.wait_for_console_pattern(console_pattern)
+
     def test_s390x_s390_ccw_virtio(self):
         """
         :avocado: tags=arch:s390x
Signature · 1 line
2.20.1

Other than that, it looks good.

- Cleber.

#5 lore raw

Re: [Qemu-arm] [Qemu-devel] [PATCH v2 2/2] Boot Linux Console Test: add a test for the Raspberry Pi 2

In reply to Philippe Mathieu-Daudé
Quoted 8 lines · Philippe Mathieu-Daudé

From: Philippe Mathieu-Daudé <[email protected]>

Similar to the x86_64/pc test, it boots a Linux kernel on a raspi2 board and verify the serial is working.

If ARM is a target being built, "make check-acceptance" will automatically include this test by the use of the "arch:arm" tags.

Quoted 6 lines

Alternatively, this test can be run using:

    $ avocado run -t arch:arm tests/acceptance
    $ avocado run -t machine:raspi2 tests/acceptance

We're getting timeouts on travis-ci:
https://travis-ci.org/ehabkost/qemu/jobs/535468057#L3289

I don't know yet if the guest is hanging, or if we just need to increase the timeout. I could reproduce the timeout locally, too.

Signature · 1 line
Eduardo
#6 lore raw

Re: [Qemu-devel] [PATCH v2 2/2] Boot Linux Console Test: add a test for the Raspberry Pi 2

----- Original Message -----

Quoted 26 lines

From: "Eduardo Habkost" <[email protected]>
To: "Philippe Mathieu-Daudé" <[email protected]>
Cc: [email protected], "Cleber Rosa" <[email protected]>, "Peter Maydell" <[email protected]>, [email protected], "Philippe Mathieu-Daudé" <[email protected]>, "Andrew Baumann" <[email protected]>
Sent: Wednesday, May 22, 2019 4:52:34 PM
Subject: Re: [Qemu-devel] [PATCH v2 2/2] Boot Linux Console Test: add a test for the Raspberry Pi 2

Quoted 8 lines · Philippe Mathieu-Daudé

From: Philippe Mathieu-Daudé <[email protected]>

Similar to the x86_64/pc test, it boots a Linux kernel on a raspi2 board and verify the serial is working.

If ARM is a target being built, "make check-acceptance" will automatically include this test by the use of the "arch:arm" tags.

Quoted 6 lines

Alternatively, this test can be run using:

    $ avocado run -t arch:arm tests/acceptance
    $ avocado run -t machine:raspi2 tests/acceptance

We're getting timeouts on travis-ci:
https://travis-ci.org/ehabkost/qemu/jobs/535468057#L3289

Quoted 7 lines

I don't know yet if the guest is hanging, or if we just need to increase the timeout. I could reproduce the timeout locally, too.

--
Eduardo

It may be related to:

https://bugs.launchpad.net/qemu/+bug/1829779

And this is a proof that we urgently need to have a better way of presenting/storing test artifacts. The brief output is nice when everything goes well, but makes the test results close to useless once a failure happens.

Philippe showed us how GitLab allows CI jobs to preserve artifacts, so maybe the solution is to move the loads there.

- Cleber.

#7 lore raw

Re: [Qemu-arm] [Qemu-devel] [PATCH v2 2/2] Boot Linux Console Test: add a test for the Raspberry Pi 2

Quoted 24 lines · Cleber Rosa

----- Original Message -----

Quoted 23 lines

From: "Eduardo Habkost" <[email protected]>

Quoted 8 lines · Philippe Mathieu-Daudé

From: Philippe Mathieu-Daudé <[email protected]>

Similar to the x86_64/pc test, it boots a Linux kernel on a raspi2 board and verify the serial is working.

If ARM is a target being built, "make check-acceptance" will automatically include this test by the use of the "arch:arm" tags.

Quoted 6 lines

Alternatively, this test can be run using:

    $ avocado run -t arch:arm tests/acceptance
    $ avocado run -t machine:raspi2 tests/acceptance

We're getting timeouts on travis-ci:
https://travis-ci.org/ehabkost/qemu/jobs/535468057#L3289

I don't know yet if the guest is hanging, or if we just need to increase the timeout. I could reproduce the timeout locally, too.

That's odd, I can't reproduce (this test is quicker than the following test_arm_emcraft_sf2 which start u-boot then timeouts and start the kernel).

My guess is network issues, since this test use a different mirror:
archive.raspberrypi.org

Gerd already raised this problem (timeout reached while fetching artifacts) to Cleber.
Cleber, can we add test_setup() methods that use different timeouts?

Regards,

Phil.

Quoted 14 lines

It may be related to:

https://bugs.launchpad.net/qemu/+bug/1829779

And this is a proof that we urgently need to have a better way of presenting/storing test artifacts. The brief output is nice when everything goes well, but makes the test results close to useless once a failure happens.

Philippe showed us how GitLab allows CI jobs to preserve artifacts, so maybe the solution is to move the loads there.

- Cleber.

#8 lore raw

Re: [Qemu-arm] [Qemu-devel] [PATCH v2 2/2] Boot Linux Console Test: add a test for the Raspberry Pi 2

----- Original Message -----

Quoted 33 lines

From: "Philippe Mathieu-Daudé" <[email protected]>
To: "Cleber Rosa" <[email protected]>, "Eduardo Habkost" <[email protected]>
Cc: [email protected], "Peter Maydell" <[email protected]>, [email protected], "Philippe Mathieu-Daudé" <[email protected]>, "Andrew Baumann" <[email protected]>, "Gerd Hoffmann" <[email protected]>
Sent: Thursday, May 23, 2019 5:29:22 AM
Subject: Re: [Qemu-devel] [PATCH v2 2/2] Boot Linux Console Test: add a test for the Raspberry Pi 2

Quoted 24 lines · Cleber Rosa

----- Original Message -----

Quoted 23 lines

From: "Eduardo Habkost" <[email protected]>

Quoted 8 lines · Philippe Mathieu-Daudé

From: Philippe Mathieu-Daudé <[email protected]>

Similar to the x86_64/pc test, it boots a Linux kernel on a raspi2 board and verify the serial is working.

If ARM is a target being built, "make check-acceptance" will automatically include this test by the use of the "arch:arm" tags.

Quoted 6 lines

Alternatively, this test can be run using:

    $ avocado run -t arch:arm tests/acceptance
    $ avocado run -t machine:raspi2 tests/acceptance

We're getting timeouts on travis-ci:
https://travis-ci.org/ehabkost/qemu/jobs/535468057#L3289

I don't know yet if the guest is hanging, or if we just need to increase the timeout. I could reproduce the timeout locally, too.

Quoted 6 lines

That's odd, I can't reproduce (this test is quicker than the following test_arm_emcraft_sf2 which start u-boot then timeouts and start the kernel).

My guess is network issues, since this test use a different mirror:
archive.raspberrypi.org

It could be a network issue, it could be something else. I think the very first step, and I'd urge us to get that on master ASAP, is to show the entire logs in CI, I mean:

---

.travis.yml +1 -1
index 6fd89b3d91..fd8f6ca2d2 100644
@@ -225,7 +225,7 @@ matrix:
     # Acceptance (Functional) tests
     - env:
         - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu"
-        - TEST_CMD="make check-acceptance"
+        - TEST_CMD="make AVOCADO_SHOW=test check-acceptance"
       addons:
         apt:
           packages:

---

That way we can know for sure what's going on.

Quoted 4 lines

Gerd already raised this problem (timeout reached while fetching artifacts) to Cleber.
Cleber, can we add test_setup() methods that use different timeouts?

Not in a released Avocado version. Interestingly enough, I wrote a PoC that adds different timeouts to tearDown()[1], that can be used the same way for setUp(), and it looks like Intel DAOS is already using those patches[2].

So, I guess I can work on a non-PoC version for that.

- Cleber.

[1] - https://github.com/avocado-framework/avocado/pull/3076
[2] - https://github.com/daos-stack/daos/commit/084ec23461e7bd9b997d4b6f5e8095a4eaffc685

Quoted 19 lines

Regards,

Phil.

Quoted 14 lines

It may be related to:

https://bugs.launchpad.net/qemu/+bug/1829779

And this is a proof that we urgently need to have a better way of presenting/storing test artifacts. The brief output is nice when everything goes well, but makes the test results close to useless once a failure happens.

Philippe showed us how GitLab allows CI jobs to preserve artifacts, so maybe the solution is to move the loads there.

- Cleber.

#9 lore raw

Re: [Qemu-arm] [Qemu-devel] [PATCH v2 2/2] Boot Linux Console Test: add a test for the Raspberry Pi 2

Quoted 52 lines · Cleber Rosa

----- Original Message -----

Quoted 33 lines

From: "Philippe Mathieu-Daudé" <[email protected]>
To: "Cleber Rosa" <[email protected]>, "Eduardo Habkost" <[email protected]>
Cc: [email protected], "Peter Maydell" <[email protected]>, [email protected], "Philippe Mathieu-Daudé" <[email protected]>, "Andrew Baumann" <[email protected]>, "Gerd Hoffmann" <[email protected]>
Sent: Thursday, May 23, 2019 5:29:22 AM
Subject: Re: [Qemu-devel] [PATCH v2 2/2] Boot Linux Console Test: add a test for the Raspberry Pi 2

Quoted 24 lines · Cleber Rosa

----- Original Message -----

Quoted 23 lines

From: "Eduardo Habkost" <[email protected]>

Quoted 8 lines · Philippe Mathieu-Daudé

From: Philippe Mathieu-Daudé <[email protected]>

Similar to the x86_64/pc test, it boots a Linux kernel on a raspi2 board and verify the serial is working.

If ARM is a target being built, "make check-acceptance" will automatically include this test by the use of the "arch:arm" tags.

Quoted 6 lines

Alternatively, this test can be run using:

    $ avocado run -t arch:arm tests/acceptance
    $ avocado run -t machine:raspi2 tests/acceptance

We're getting timeouts on travis-ci:
https://travis-ci.org/ehabkost/qemu/jobs/535468057#L3289

I don't know yet if the guest is hanging, or if we just need to increase the timeout. I could reproduce the timeout locally, too.

Quoted 6 lines

That's odd, I can't reproduce (this test is quicker than the following test_arm_emcraft_sf2 which start u-boot then timeouts and start the kernel).

My guess is network issues, since this test use a different mirror:
archive.raspberrypi.org

It could be a network issue, it could be something else. I think the very first step, and I'd urge us to get that on master ASAP, is to show the entire logs in CI, I mean:

---

.travis.yml +0 -0
index 6fd89b3d91..fd8f6ca2d2 100644

.travis.yml matrix: · line 231

Quoted 8 lines
.travis.yml +1 -1
@@ -225,7 +225,7 @@ matrix:
     # Acceptance (Functional) tests
     - env:
         - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu"
-        - TEST_CMD="make check-acceptance"
+        - TEST_CMD="make AVOCADO_SHOW=test check-acceptance"
       addons:
         apt:
           packages:

I have applied this change on python-next. The failure can be seen here:
https://travis-ci.org/ehabkost/qemu/jobs/541758418#L4033

While we don't fix this, I'm removing this test case from the queue, so I can send a pull request. Sorry for taking so long to act on this.

Signature · 1 line
Eduardo
#10 lore raw
In reply to Philippe Mathieu-Daudé
Quoted 7 lines · Philippe Mathieu-Daudé

Add the raspi2 to the list of boards tested by the boot_linux_console Avocado test.

Based on "Acceptance Tests: target architecture support" v5:
https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg04111.html
Based-on: <[email protected]>

Quoted 6 lines

Philippe Mathieu-Daudé (2):
BootLinuxConsoleTest: Let extract_from_deb handle various compressions Boot Linux Console Test: add a test for the Raspberry Pi 2

 tests/acceptance/boot_linux_console.py | 34 ++++++++++++++++++++++++--
 1 file changed, 32 insertions(+), 2 deletions(-)

I was hoping this series would be included in "Acceptance Tests: target architecture support" by Eduardo (removing the 'print' debug line) but I just realized I forgot to Cc him :S

#11 lore raw
Quoted 12 lines · Philippe Mathieu-Daudé
Quoted 11 lines · Philippe Mathieu-Daudé

Add the raspi2 to the list of boards tested by the boot_linux_console Avocado test.

Based on "Acceptance Tests: target architecture support" v5:
https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg04111.html
Based-on: <[email protected]>

Philippe Mathieu-Daudé (2):
BootLinuxConsoleTest: Let extract_from_deb handle various compressions Boot Linux Console Test: add a test for the Raspberry Pi 2

Quoted 6 lines
Quoted 2 lines
 tests/acceptance/boot_linux_console.py | 34 ++++++++++++++++++++++++--
 1 file changed, 32 insertions(+), 2 deletions(-)

I was hoping this series would be included in "Acceptance Tests: target architecture support" by Eduardo (removing the 'print' debug line) but I just realized I forgot to Cc him :S

Problem solved: queued on python-next (after removing the print statement from patch 2/2). :)

Signature · 1 line
Eduardo
#12 lore raw
Quoted 16 lines · Eduardo Habkost
Quoted 15 lines · Philippe Mathieu-Daudé
Quoted 13 lines · Philippe Mathieu-Daudé

Add the raspi2 to the list of boards tested by the boot_linux_console Avocado test.

Based on "Acceptance Tests: target architecture support" v5:
https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg04111.html
Based-on: <[email protected]>

Philippe Mathieu-Daudé (2):
BootLinuxConsoleTest: Let extract_from_deb handle various compressions Boot Linux Console Test: add a test for the Raspberry Pi 2

 tests/acceptance/boot_linux_console.py | 34 ++++++++++++++++++++++++--
 1 file changed, 32 insertions(+), 2 deletions(-)
Quoted 6 lines
Quoted 3 lines

I was hoping this series would be included in "Acceptance Tests: target architecture support" by Eduardo (removing the 'print' debug line) but I just realized I forgot to Cc him :S

Problem solved: queued on python-next (after removing the print statement from patch 2/2). :)

Thanks a lot!