Miscellaneous acceptance test and Travis CI improvements
cover Miscellaneous acceptance test and Travis CI improvements
This is a collection of small improvements to some of the acceptance tests, and the Travis CI experience.
The main goal was to make tests a bit more robust when run in parallel (an Avocado feature pending review), and Travis CI diagnostics better by printing out the full Avocado job log when any error or test failure occurs.
Cleber Rosa (8):
Travis: print acceptance tests logs in case of job failure tests/requirements.txt: pin paramiko version requirement Acceptance tests: drop left over usage of ":avocado: enable" Boot Linux Console Test: add a test for ppc64 + pseries VNC Acceptance test: use UNIX domain sockets to avoid port collisions VNC Acceptance test: simplify test names
VNC Acceptance test: check protocol version
Migration acceptance test: reduce the possibility of port collisions
.travis.yml | 4 ++- tests/acceptance/boot_linux_console.py | 19 ++++++++++++++ tests/acceptance/migration.py | 5 +--- tests/acceptance/vnc.py | 36 +++++++++++++++++++++++--- tests/requirements.txt | 2 +- 5 files changed, 56 insertions(+), 10 deletions(-)
Signature · 1 line
2.21.0
PATCH 1/8 Travis: print acceptance tests logs in case of job failure
Because Travis doesn't allow us to keep files produced during tests (such as log files), let's print the complete job log to the "console" in case of job failures.
This is a debugging aid, and given that there's been some timeouts happening on some tests, we absolutely needs the logs to have a proper action.
--- .travis.yml | 2 ++ 1 file changed, 2 insertions(+)
.travis.yml +2 -0
@@ -226,6 +226,8 @@ matrix: - env: - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu" - TEST_CMD="make check-acceptance" + after_failure: + - cat tests/results/latest/job.log addons: apt: packages:
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH 1/8] Travis: print acceptance tests logs in case of job failure
.travis.yml
Quoted 17 lines · Cleber Rosa
Because Travis doesn't allow us to keep files produced during tests (such as log files), let's print the complete job log to the "console" in case of job failures.
This is a debugging aid, and given that there's been some timeouts happening on some tests, we absolutely needs the logs to have a proper action.
Signed-off-by Cleber Rosa <[email protected]>--- .travis.yml | 2 ++ 1 file changed, 2 insertions(+).travis.yml +0
-0
It's handy. Unfortunately you cannot archive the log files in Travis, otherwise that would be a nice option.
Quoted 9 lines
.travis.yml +2
-0@@ -226,6 +226,8 @@ matrix: - env: - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu" - TEST_CMD="make check-acceptance" + after_failure: + - cat tests/results/latest/job.log addons: apt: packages:
Re: [Qemu-devel] [PATCH 1/8] Travis: print acceptance tests logs in case of job failure
In reply to Cleber RosaQuoted 9 lines · Cleber Rosa
Because Travis doesn't allow us to keep files produced during tests (such as log files), let's print the complete job log to the "console" in case of job failures.
This is a debugging aid, and given that there's been some timeouts happening on some tests, we absolutely needs the logs to have a proper action.
Signed-off-by Cleber Rosa <[email protected]>
Quoted 9 lines
--- .travis.yml | 2 ++ 1 file changed, 2 insertions(+).travis.yml +0
-0
.travis.yml matrix: · line 233
Quoted 8 lines
.travis.yml +2
-0@@ -226,6 +226,8 @@ matrix: - env: - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu" - TEST_CMD="make check-acceptance" + after_failure: + - cat tests/results/latest/job.log addons: apt: packages:
--
Alex Bennée
PATCH 2/8 tests/requirements.txt: pin paramiko version requirement
In reply to Cleber RosaIt's a good practice (I'd really say a must) to pin as much as possible of the software versions used during test, so let's apply that to paramiko.
According to https://pypi.org/project/paramiko/, 2.4.2 is the latest released version. It's also easily obtainable on systems such as Fedora 30.
--- tests/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/requirements.txt +1 -1
@@ -2,4 +2,4 @@ # in the tests/venv Python virtual environment. For more info, # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 avocado-framework==68.0 -paramiko +paramiko==2.4.2
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH 2/8] tests/requirements.txt: pin paramiko version requirement
Quoted 4 lines · Cleber Rosa
It's a good practice (I'd really say a must) to pin as much as possible of the software versions used during test, so let's apply that to paramiko.
Quoted 8 lines
According to https://pypi.org/project/paramiko/, 2.4.2 is the latest released version. It's also easily obtainable on systems such as Fedora 30.
Signed-off-by Cleber Rosa <[email protected]>--- tests/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 11 lines
tests/requirements.txt +1
-1@@ -2,4 +2,4 @@ # in the tests/venv Python virtual environment. For more info, # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 avocado-framework==68.0 -paramiko +paramiko==2.4.2
Re: [Qemu-devel] [PATCH 2/8] tests/requirements.txt: pin paramiko version requirement
In reply to Cleber RosaQuoted 19 lines · Cleber Rosa
It's a good practice (I'd really say a must) to pin as much as possible of the software versions used during test, so let's apply that to paramiko.
According to https://pypi.org/project/paramiko/, 2.4.2 is the latest released version. It's also easily obtainable on systems such as Fedora 30.
Signed-off-by Cleber Rosa <[email protected]>--- tests/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/requirements.txt +0
-0
tests/requirements.txt line 5
Quoted 5 lines
tests/requirements.txt +1
-1@@ -2,4 +2,4 @@ # in the tests/venv Python virtual environment. For more info, # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 avocado-framework==68.0 -paramiko +paramiko==2.4.2
PATCH 3/8 Acceptance tests: drop left over usage of ":avocado: enable"
In reply to Cleber RosaCommit 9531d26c10 removed all of ":avocado: enable" tags, but then a new entry was added with the introduction of migration.py.
Let's remove it for consistency.
--- tests/acceptance/migration.py | 3 --- 1 file changed, 3 deletions(-)
tests/acceptance/migration.py +0 -3
@@ -17,9 +17,6 @@ from avocado.utils import wait class Migration(Test): - """ - :avocado: enable - """ timeout = 10
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH 3/8] Acceptance tests: drop left over usage of ":avocado: enable"
Quoted 9 lines · Cleber Rosa
Commit
9531d26c10removed all of ":avocado: enable" tags, but then a new entry was added with the introduction ofmigration.py.Let's remove it for consistency.
Signed-off-by Cleber Rosa <[email protected]>--- tests/acceptance/migration.py | 3 --- 1 file changed, 3 deletions(-)
I was about to send a patch to remove it as well. :)
Quoted 15 lines
tests/acceptance/migration.py +0
-3@@ -17,9 +17,6 @@ from avocado.utils import wait class Migration(Test): - """ - :avocado: enable - """ timeout = 10
Re: [Qemu-devel] [PATCH 3/8] Acceptance tests: drop left over usage of ":avocado: enable"
In reply to Cleber RosaQuoted 16 lines · Cleber Rosa
Commit
9531d26c10removed all of ":avocado: enable" tags, but then a new entry was added with the introduction ofmigration.py.Let's remove it for consistency.
Signed-off-by Cleber Rosa <[email protected]>--- tests/acceptance/migration.py | 3 --- 1 file changed, 3 deletions(-)tests/acceptance/migration.py +0
-0
tests/acceptance/migration.py from avocado.utils import wait · line 22
Quoted 9 lines
tests/acceptance/migration.py +0
-3@@ -17,9 +17,6 @@ from avocado.utils import wait class Migration(Test): - """ - :avocado: enable - """ timeout = 10
PATCH 4/8 Boot Linux Console Test: add a test for ppc64 + pseries
In reply to Cleber RosaJust like the previous tests, boots a Linux kernel on a ppc64 target using the pseries machine.
--- .travis.yml | 2 +- tests/acceptance/boot_linux_console.py | 19 +++++++++++++++++++ 2 files changed, 20 insertions(+), 1 deletion(-)
.travis.yml +1 -1
@@ -224,7 +224,7 @@ matrix: # Acceptance (Functional) tests - env: - - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu" + - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc64-softmmu" - TEST_CMD="make check-acceptance" after_failure: - cat tests/results/latest/job.log
tests/acceptance/boot_linux_console.py +19 -0
@@ -217,3 +217,22 @@ class BootLinuxConsole(Test): self.vm.launch() console_pattern = 'Kernel command line: %s' % kernel_command_line self.wait_for_console_pattern(console_pattern) + + def test_ppc64_pseries(self): + """ + :avocado: tags=arch:ppc64 + :avocado: tags=machine:pseries + """ + kernel_url = ('https://download.fedoraproject.org/pub/fedora-secondary/' + 'releases/29/Everything/ppc64le/os/ppc/ppc64/vmlinuz') + kernel_hash = '3fe04abfc852b66653b8c3c897a59a689270bc77' + kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash) + + self.vm.set_machine('pseries') + self.vm.set_console() + kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=hvc0' + self.vm.add_args('-kernel', kernel_path, + '-append', kernel_command_line) + self.vm.launch() + console_pattern = 'Kernel command line: %s' % kernel_command_line + self.wait_for_console_pattern(console_pattern)
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH 4/8] Boot Linux Console Test: add a test for ppc64 + pseries
Hi Cleber,
Quoted 41 lines · Cleber Rosa
Just like the previous tests, boots a Linux kernel on a ppc64 target using the pseries machine.
Signed-off-by Cleber Rosa <[email protected]>Reviewed-by Caio Carrara <[email protected]>CC Daniel P. Berrangé <[email protected]>--- .travis.yml | 2 +- tests/acceptance/boot_linux_console.py | 19 +++++++++++++++++++ 2 files changed, 20 insertions(+), 1 deletion(-).travis.yml +1
-1@@ -224,7 +224,7 @@ matrix: # Acceptance (Functional) tests - env: - - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu" + - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc64-softmmu" - TEST_CMD="make check-acceptance" after_failure: - cat tests/results/latest/job.logtests/acceptance/boot_linux_console.py +9
-0@@ -217,3 +217,22 @@ class BootLinuxConsole(Test): self.vm.launch() console_pattern = 'Kernel command line: %s' % kernel_command_line self.wait_for_console_pattern(console_pattern) + + def test_ppc64_pseries(self): + """ + :avocado: tags=arch:ppc64 + :avocado: tags=machine:pseries + """ + kernel_url = ('https://download.fedoraproject.org/pub/fedora-secondary/' + 'releases/29/Everything/ppc64le/os/ppc/ppc64/vmlinuz') + kernel_hash = '3fe04abfc852b66653b8c3c897a59a689270bc77'
tests/acceptance/boot_linux_console.py class BootLinuxConsole(Test): · line 238
Quoted 10 lines
tests/acceptance/boot_linux_console.py +10
-0+ kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash) + + self.vm.set_machine('pseries') + self.vm.set_console() + kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=hvc0' + self.vm.add_args('-kernel', kernel_path, + '-append', kernel_command_line) + self.vm.launch() + console_pattern = 'Kernel command line: %s' % kernel_command_line + self.wait_for_console_pattern(console_pattern)
I'm refactoring the acceptance tests which boot a Linux Kernel so that they can share common functions. My latest implementation [1] was re-based on this patch, and it would help manage my contribution if we could merge this very soon.
Anyway, tested this patch on my Fedora 30 x86_64 machine. Passed.
[1] https://github.com/wainersm/qemu/tree/acceptance_boot_linux
PATCH 5/8 VNC Acceptance test: use UNIX domain sockets to avoid port collisions
In reply to Cleber RosaWhile running in parallel, the VNC tests that use a TCP port easily collide. There's a number of possibilities to reduce the probability of collisions, but none that completely prevents it from happening.
So, to avoid those collisions, and given that the scope of the tests are really not related to nature of the socket type, let's switch to UNIX domain sockets created in temporary directories.
Note: the amount of boiler plate code is far from the ideal, but it's related to the fact that a test "workdir"[1] attribute can not be used here, because of the 108 bytes limitation of the UNIX socket path (see ad9579aaa16). There's a fair assumption here that the temporary directory returned by Python's tempfile.mkdtemp() won't be anywhere close to 100 bytes.
[1] https://avocado-framework.readthedocs.io/en/68.0/api/test/avocado.html#avocado.Test.workdir
--- tests/acceptance/vnc.py | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-)
tests/acceptance/vnc.py +18 -2
@@ -8,6 +8,10 @@ # This work is licensed under the terms of the GNU GPL, version 2 or # later. See the COPYING file in the top-level directory. +import os +import tempfile +import shutil + from avocado_qemu import Test
@@ -34,8 +38,16 @@ class Vnc(Test): self.assertEqual(set_password_response['error']['desc'], 'Could not set password') +class VncUnixSocket(Test): + + def setUp(self): + super(VncUnixSocket, self).setUp() + self.socket_dir = tempfile.mkdtemp() + self.socket_path = os.path.join(self.socket_dir, 'vnc-socket') + def test_vnc_change_password_requires_a_password(self): - self.vm.add_args('-nodefaults', '-S', '-vnc', ':0') + self.vm.add_args('-nodefaults', '-S', + '-vnc', 'unix:%s' % self.socket_path) self.vm.launch() self.assertTrue(self.vm.qmp('query-vnc')['return']['enabled']) set_password_response = self.vm.qmp('change',
@@ -49,7 +61,8 @@ class Vnc(Test): 'Could not set password') def test_vnc_change_password(self): - self.vm.add_args('-nodefaults', '-S', '-vnc', ':0,password') + self.vm.add_args('-nodefaults', '-S', + '-vnc', 'unix:%s,password' % self.socket_path) self.vm.launch() self.assertTrue(self.vm.qmp('query-vnc')['return']['enabled']) set_password_response = self.vm.qmp('change',
@@ -57,3 +70,6 @@ class Vnc(Test): target='password', arg='new_password') self.assertEqual(set_password_response['return'], {}) + + def tearDown(self): + shutil.rmtree(self.socket_dir)
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH 5/8] VNC Acceptance test: use UNIX domain sockets to avoid port collisions
Quoted 66 lines · Cleber Rosa
While running in parallel, the VNC tests that use a TCP port easily collide. There's a number of possibilities to reduce the probability of collisions, but none that completely prevents it from happening.
So, to avoid those collisions, and given that the scope of the tests are really not related to nature of the socket type, let's switch to UNIX domain sockets created in temporary directories.
Note: the amount of boiler plate code is far from the ideal, but it's related to the fact that a test "workdir"[1] attribute can not be used here, because of the 108 bytes limitation of the UNIX socket path (see
ad9579aaa16). There's a fair assumption here that the temporary directory returned by Python'stempfile.mkdtemp()won't be anywhere close to 100 bytes.[1] https://avocado-framework.readthedocs.io/en/68.0/api/test/avocado.html#avocado.Test.workdir
Signed-off-by Cleber Rosa <[email protected]>--- tests/acceptance/vnc.py | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-)tests/acceptance/vnc.py +15
-2@@ -8,6 +8,10 @@ # This work is licensed under the terms of the GNU GPL, version 2 or # later. See the COPYING file in the top-level directory. +import os +import tempfile +import shutil + from avocado_qemu import Test@@ -34,8 +38,16 @@ class Vnc(Test): self.assertEqual(set_password_response['error']['desc'], 'Could not set password') +class VncUnixSocket(Test): + + def setUp(self): + super(VncUnixSocket, self).setUp() + self.socket_dir = tempfile.mkdtemp() + self.socket_path = os.path.join(self.socket_dir, 'vnc-socket') + def test_vnc_change_password_requires_a_password(self): - self.vm.add_args('-nodefaults', '-S', '-vnc', ':0') + self.vm.add_args('-nodefaults', '-S', + '-vnc', 'unix:%s' % self.socket_path) self.vm.launch() self.assertTrue(self.vm.qmp('query-vnc')['return']['enabled']) set_password_response = self.vm.qmp('change',@@ -49,7 +61,8 @@ class Vnc(Test): 'Could not set password') def test_vnc_change_password(self): - self.vm.add_args('-nodefaults', '-S', '-vnc', ':0,password') + self.vm.add_args('-nodefaults', '-S', + '-vnc', 'unix:%s,password' % self.socket_path) self.vm.launch() self.assertTrue(self.vm.qmp('query-vnc')['return']['enabled']) set_password_response = self.vm.qmp('change',
tests/acceptance/vnc.py class Vnc(Test): · line 75
Quoted 6 lines
tests/acceptance/vnc.py +3
-0@@ -57,3 +70,6 @@ class Vnc(Test): target='password', arg='new_password') self.assertEqual(set_password_response['return'], {}) + + def tearDown(self): + shutil.rmtree(self.socket_dir)
You missed to call super's tearDown in order to gently shutdown all VM created in by the tests. Other than that, it looks good to me.
- Wainer
PATCH 6/8 VNC Acceptance test: simplify test names
In reply to Cleber RosaThe test name is composed of the class name and method name, so it looks like there's some redundancy here that we can eliminate.
--- tests/acceptance/vnc.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tests/acceptance/vnc.py +2 -2
@@ -45,7 +45,7 @@ class VncUnixSocket(Test): self.socket_dir = tempfile.mkdtemp() self.socket_path = os.path.join(self.socket_dir, 'vnc-socket') - def test_vnc_change_password_requires_a_password(self): + def test_change_password_requires_a_password(self): self.vm.add_args('-nodefaults', '-S', '-vnc', 'unix:%s' % self.socket_path) self.vm.launch()
@@ -60,7 +60,7 @@ class VncUnixSocket(Test): self.assertEqual(set_password_response['error']['desc'], 'Could not set password') - def test_vnc_change_password(self): + def test_change_password(self): self.vm.add_args('-nodefaults', '-S', '-vnc', 'unix:%s,password' % self.socket_path) self.vm.launch()
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH 6/8] VNC Acceptance test: simplify test names
Quoted 7 lines · Cleber Rosa
The test name is composed of the class name and method name, so it looks like there's some redundancy here that we can eliminate.
Signed-off-by Cleber Rosa <[email protected]>--- tests/acceptance/vnc.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Quoted 23 lines
tests/acceptance/vnc.py +2
-2@@ -45,7 +45,7 @@ class VncUnixSocket(Test): self.socket_dir = tempfile.mkdtemp() self.socket_path = os.path.join(self.socket_dir, 'vnc-socket') - def test_vnc_change_password_requires_a_password(self): + def test_change_password_requires_a_password(self): self.vm.add_args('-nodefaults', '-S', '-vnc', 'unix:%s' % self.socket_path) self.vm.launch()@@ -60,7 +60,7 @@ class VncUnixSocket(Test): self.assertEqual(set_password_response['error']['desc'], 'Could not set password') - def test_vnc_change_password(self): + def test_change_password(self): self.vm.add_args('-nodefaults', '-S', '-vnc', 'unix:%s,password' % self.socket_path) self.vm.launch()
Re: [Qemu-devel] [PATCH 6/8] VNC Acceptance test: simplify test names
In reply to Cleber RosaQuoted 23 lines · Cleber Rosa
The test name is composed of the class name and method name, so it looks like there's some redundancy here that we can eliminate.
Signed-off-by Cleber Rosa <[email protected]>--- tests/acceptance/vnc.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)tests/acceptance/vnc.py +1
-1@@ -45,7 +45,7 @@ class VncUnixSocket(Test): self.socket_dir = tempfile.mkdtemp() self.socket_path = os.path.join(self.socket_dir, 'vnc-socket') - def test_vnc_change_password_requires_a_password(self): + def test_change_password_requires_a_password(self): self.vm.add_args('-nodefaults', '-S', '-vnc', 'unix:%s' % self.socket_path) self.vm.launch()
tests/acceptance/vnc.py class VncUnixSocket(Test): · line 66
Quoted 8 lines
tests/acceptance/vnc.py +1
-1@@ -60,7 +60,7 @@ class VncUnixSocket(Test): self.assertEqual(set_password_response['error']['desc'], 'Could not set password') - def test_vnc_change_password(self): + def test_change_password(self): self.vm.add_args('-nodefaults', '-S', '-vnc', 'unix:%s,password' % self.socket_path) self.vm.launch()
PATCH 7/8 VNC Acceptance test: check protocol version
In reply to Cleber RosaThis goes a bit further than the other tests, and does a basic (read only) interaction with the VNC protocol.
This is not a enough to perform a handshake, but enough to make sure that the socket is somewhat operational and that the expected initial step of the handshake is performed by the server and that the version matches.
--- tests/acceptance/vnc.py | 12 ++++++++++++ 1 file changed, 12 insertions(+)
tests/acceptance/vnc.py +12 -0
@@ -11,6 +11,7 @@ import os import tempfile import shutil +import socket from avocado_qemu import Test
@@ -71,5 +72,16 @@ class VncUnixSocket(Test): arg='new_password') self.assertEqual(set_password_response['return'], {}) + def test_protocol_version(self): + self.vm.add_args('-nodefaults', '-S', + '-vnc', 'unix:%s' % self.socket_path) + self.vm.launch() + self.assertTrue(self.vm.qmp('query-vnc')['return']['enabled']) + client = socket.socket(socket.AF_UNIX) + client.connect(self.socket_path) + expected = b'RFB 003.008' + actual = client.recv(len(expected)) + self.assertEqual(expected, actual, "Wrong protocol version") + def tearDown(self): shutil.rmtree(self.socket_dir)
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH 7/8] VNC Acceptance test: check protocol version
Quoted 7 lines · Cleber Rosa
This goes a bit further than the other tests, and does a basic (read only) interaction with the VNC protocol.
This is not a enough to perform a handshake, but enough to make sure that the socket is somewhat operational and that the expected initial step of the handshake is performed by the server and that the version matches.
The GTK-VNC project provides a low level library libgvnc that can be used to talk the RFB protocol in a fairly fine grained manner. This is built using GObject, so is accessible from Python thanks to GObject Introspection.
We could use libgvnc for exercising the VNC server instead of writing custom RFB code. For your simple test just sending/receiving the version it won't save much, but if we ever want to test TLS or SASL integration, it would save alot of work dealing wth the auth handshake and subsequent encryption needs.
The main limitation it would have is that it would only work well for sending "well formed" RFB protocol messages. If we ever want to send intentionally evil/bad RFB data to check QEMU's VNC server security hardening it would be harder.
As the maintainer of GTK-VNC though, I would be open to adding more APIs to the low level gvnc library to facilitate QEMU's testing needs if we want.
Quoted 39 lines
Signed-off-by Cleber Rosa <[email protected]>--- tests/acceptance/vnc.py | 12 ++++++++++++ 1 file changed, 12 insertions(+)tests/acceptance/vnc.py +12
-0@@ -11,6 +11,7 @@ import os import tempfile import shutil +import socket from avocado_qemu import Test@@ -71,5 +72,16 @@ class VncUnixSocket(Test): arg='new_password') self.assertEqual(set_password_response['return'], {}) + def test_protocol_version(self): + self.vm.add_args('-nodefaults', '-S', + '-vnc', 'unix:%s' % self.socket_path) + self.vm.launch() + self.assertTrue(self.vm.qmp('query-vnc')['return']['enabled']) + client = socket.socket(socket.AF_UNIX) + client.connect(self.socket_path) + expected = b'RFB 003.008' + actual = client.recv(len(expected)) + self.assertEqual(expected, actual, "Wrong protocol version") + def tearDown(self): shutil.rmtree(self.socket_dir)Signature · 1 line
2.21.0
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-devel] [PATCH 7/8] VNC Acceptance test: check protocol version
Quoted 9 lines · Daniel P. Berrangé
Quoted 7 lines · Cleber Rosa
This goes a bit further than the other tests, and does a basic (read only) interaction with the VNC protocol.
This is not a enough to perform a handshake, but enough to make sure that the socket is somewhat operational and that the expected initial step of the handshake is performed by the server and that the version matches.
Quoted 5 lines
The GTK-VNC project provides a low level library libgvnc that can be used to talk the RFB protocol in a fairly fine grained manner. This is built using GObject, so is accessible from Python thanks to GObject Introspection.
Interesting.
Quoted 6 lines
We could use libgvnc for exercising the VNC server instead of writing custom RFB code. For your simple test just sending/receiving the version it won't save much, but if we ever want to test TLS or SASL integration, it would save alot of work dealing wth the auth handshake and subsequent encryption needs.
Absolutely.
Quoted 5 lines
The main limitation it would have is that it would only work well for sending "well formed" RFB protocol messages. If we ever want to send intentionally evil/bad RFB data to check QEMU's VNC server security hardening it would be harder.
Right. Still, there's a lot that can be done until we eventually exaust all possibilities and look into sending bad messages.
Quoted 4 lines
As the maintainer of GTK-VNC though, I would be open to adding more APIs to the low level gvnc library to facilitate QEMU's testing needs if we want.
I personally need to get acquainted with the currently available APIs first, but it looks like you alread have ideas for extensions that would come in handy.
Also, the one concern I have is how to deploy the library and Python bindings so that we can host those more advanced tests and still allow for a "make check-acceptance"-like experience. What I mean is, I expect the Python bindings to be easily installed by pip, by I'd be (positively) surprised if the libgvnc would also have such an easy bootstrap.
Any ideas on this?
Thanks!
- Cleber.
Quoted 46 lines
Quoted 39 lines
Signed-off-by Cleber Rosa <[email protected]>--- tests/acceptance/vnc.py | 12 ++++++++++++ 1 file changed, 12 insertions(+)tests/acceptance/vnc.py +12
-0@@ -11,6 +11,7 @@ import os import tempfile import shutil +import socket from avocado_qemu import Test@@ -71,5 +72,16 @@ class VncUnixSocket(Test): arg='new_password') self.assertEqual(set_password_response['return'], {}) + def test_protocol_version(self): + self.vm.add_args('-nodefaults', '-S', + '-vnc', 'unix:%s' % self.socket_path) + self.vm.launch() + self.assertTrue(self.vm.qmp('query-vnc')['return']['enabled']) + client = socket.socket(socket.AF_UNIX) + client.connect(self.socket_path) + expected = b'RFB 003.008' + actual = client.recv(len(expected)) + self.assertEqual(expected, actual, "Wrong protocol version") + def tearDown(self): shutil.rmtree(self.socket_dir)Signature · 1 line
2.21.0Regards,
DanielSignature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-devel] [PATCH 7/8] VNC Acceptance test: check protocol version
Quoted 45 lines · Cleber Rosa
Quoted 9 lines · Daniel P. Berrangé
Quoted 7 lines · Cleber Rosa
This goes a bit further than the other tests, and does a basic (read only) interaction with the VNC protocol.
This is not a enough to perform a handshake, but enough to make sure that the socket is somewhat operational and that the expected initial step of the handshake is performed by the server and that the version matches.
Quoted 5 lines
The GTK-VNC project provides a low level library libgvnc that can be used to talk the RFB protocol in a fairly fine grained manner. This is built using GObject, so is accessible from Python thanks to GObject Introspection.
Interesting.
Quoted 6 lines
We could use libgvnc for exercising the VNC server instead of writing custom RFB code. For your simple test just sending/receiving the version it won't save much, but if we ever want to test TLS or SASL integration, it would save alot of work dealing wth the auth handshake and subsequent encryption needs.
Absolutely.
Quoted 5 lines
The main limitation it would have is that it would only work well for sending "well formed" RFB protocol messages. If we ever want to send intentionally evil/bad RFB data to check QEMU's VNC server security hardening it would be harder.
Right. Still, there's a lot that can be done until we eventually exaust all possibilities and look into sending bad messages.
Quoted 4 lines
As the maintainer of GTK-VNC though, I would be open to adding more APIs to the low level gvnc library to facilitate QEMU's testing needs if we want.
I personally need to get acquainted with the currently available APIs first, but it looks like you alread have ideas for extensions that would come in handy.
Quoted 8 lines
Also, the one concern I have is how to deploy the library and Python bindings so that we can host those more advanced tests and still allow for a "make check-acceptance"-like experience. What I mean is, I expect the Python bindings to be easily installed by pip, by I'd be (positively) surprised if the libgvnc would also have such an easy bootstrap.
Any ideas on this?
IMHO we shouldn't try to install anything from pip. It should all be available from distro repos already. In fact if you have "virt-install" or "virt-manager" installed, you should already have the required python bits.
On RHEL/Fedora you need python3-gobject to get the GObject introspection support and then all you need is the native gvnc library. There is no python library for gvnc because GObject introspection makes this redundant - the python API is dynamically created at runtime using FFI to access the native library APIs.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-devel] [PATCH 7/8] VNC Acceptance test: check protocol version
In reply to Cleber RosaQuoted 26 lines · Cleber Rosa
This goes a bit further than the other tests, and does a basic (read only) interaction with the VNC protocol.
This is not a enough to perform a handshake, but enough to make sure that the socket is somewhat operational and that the expected initial step of the handshake is performed by the server and that the version matches.
Signed-off-by Cleber Rosa <[email protected]>--- tests/acceptance/vnc.py | 12 ++++++++++++ 1 file changed, 12 insertions(+)tests/acceptance/vnc.py +1
-0@@ -11,6 +11,7 @@ import os import tempfile import shutil +import socket from avocado_qemu import Test
tests/acceptance/vnc.py class VncUnixSocket(Test): · line 79
Quoted 8 lines
tests/acceptance/vnc.py +5
-0@@ -71,5 +72,16 @@ class VncUnixSocket(Test): arg='new_password') self.assertEqual(set_password_response['return'], {}) + def test_protocol_version(self): + self.vm.add_args('-nodefaults', '-S', + '-vnc', 'unix:%s' % self.socket_path) + self.vm.launch() + self.assertTrue(self.vm.qmp('query-vnc')['return']['enabled'])
I will advocate for the use QEMUMachine.command() instead of qmp(). The former do some checks on the qmp response and raises a more gently exception if something went wrong. This patch looks good to me though.
Quoted 8 lines
tests/acceptance/vnc.py +6
-0+ client = socket.socket(socket.AF_UNIX) + client.connect(self.socket_path) + expected = b'RFB 003.008' + actual = client.recv(len(expected)) + self.assertEqual(expected, actual, "Wrong protocol version") + def tearDown(self): shutil.rmtree(self.socket_dir)
PATCH 8/8 Migration acceptance test: reduce the possibility of port collisions
In reply to Cleber RosaThe avocado.utils.network.find_free_port() will by default attempt to return the first port given in the range, if it seems to be available.
Let's reduce the probability of collisions by picking a random port by default.
As a side note, this behavior has changed in a recent Avocado version, and the default use will then be equivalent to this proposed change.
Reference: https://avocado-framework.readthedocs.io/en/68.0/api/utils/avocado.utils.html#avocado.utils.network.find_free_port
Reference: https://github.com/avocado-framework/avocado/commit/f232e4505f7cfefc513449e0b97790b517275da7
--- tests/acceptance/migration.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/acceptance/migration.py +1 -1
@@ -25,7 +25,7 @@ class Migration(Test): return vm.command('query-migrate')['status'] in ('completed', 'failed') def _get_free_port(self): - port = network.find_free_port() + port = network.find_free_port(sequent=False) if port is None: self.cancel('Failed to find a free port') return port
Signature · 1 line
2.21.0
Quoted 7 lines · Cleber Rosa
This is a collection of small improvements to some of the acceptance tests, and the Travis CI experience.
The main goal was to make tests a bit more robust when run in parallel (an Avocado feature pending review), and Travis CI diagnostics better by printing out the full Avocado job log when any error or test failure occurs.
Forgot to add these hopefully useful pointers:
Git Info:
- URI: https://github.com/clebergnu/qemu/tree/misc_tests_improvements
- Remote: https://github.com/clebergnu/qemu
- Branch: misc_tests_improvements
Travis CI Info:
- Build: https://travis-ci.org/clebergnu/qemu/builds/542815049
- Job: https://travis-ci.org/clebergnu/qemu/jobs/542815074
Regards,
- Cleber.
Quoted 21 lines
Cleber Rosa (8):
Travis: print acceptance tests logs in case of job failuretests/requirements.txt: pin paramiko version requirement Acceptance tests: drop left over usage of ":avocado: enable" Boot Linux Console Test: add a test for ppc64 + pseries VNC Acceptance test: use UNIX domain sockets to avoid port collisions VNC Acceptance test: simplify test names
VNC Acceptance test: check protocol version
Migration acceptance test: reduce the possibility of port collisions.travis.yml | 4 ++- tests/acceptance/boot_linux_console.py | 19 ++++++++++++++ tests/acceptance/migration.py | 5 +--- tests/acceptance/vnc.py | 36 +++++++++++++++++++++++--- tests/requirements.txt | 2 +- 5 files changed, 56 insertions(+), 10 deletions(-)Signature · 1 line
2.21.0