Received: from eggs.gnu.org ([209.51.188.92]:45891) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hJHMb-0008OP-2X for qemu-devel@nongnu.org; Wed, 24 Apr 2019 08:51:02 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hJHMZ-0006NT-OT for qemu-devel@nongnu.org; Wed, 24 Apr 2019 08:51:01 -0400 Date: Wed, 24 Apr 2019 13:50:46 +0100 From: Daniel =?utf-8?B?UC4gQmVycmFuZ8Op?= Message-ID: <20190424125046.GJ31388@redhat.com> Reply-To: Daniel =?utf-8?B?UC4gQmVycmFuZ8Op?= References: <20190424103747.10173-1-thuth@redhat.com> <20190424103747.10173-3-thuth@redhat.com> <20190424112537.GF31388@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] [PATCH 2/6] tests/qemu-iotests/group: Introduce a new "ci" group for CI pipelines List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Thomas Huth Cc: qemu-devel@nongnu.org, Fam Zheng , Kevin Wolf , Ed Maste , qemu-block@nongnu.org, Alex =?utf-8?Q?Benn=C3=A9e?= , Max Reitz , Philippe =?utf-8?Q?Mathieu-Daud=C3=A9?= , Li-Wen Hsu On Wed, Apr 24, 2019 at 02:37:02PM +0200, Thomas Huth wrote: > On 24/04/2019 13.25, Daniel P. Berrang=C3=A9 wrote: > > On Wed, Apr 24, 2019 at 12:37:43PM +0200, Thomas Huth wrote: > >> Tests in this group are supposed to run in every possible QEMU confi= guration. > >> That means they should run with every QEMU binary (also non-x86), wi= thout > >> dependencies on an optional features, they must work at least with t= he qcow2 > >> image format and be able to run on all kind of host filesystems and = users > >> (i.e. also as "nobody" or "root"). > >> > >> The initial list has been created as a subset of the "quick" group, = where > >> I've disabled all tests that are failing with qemu-system-aarch64 or > >> qemu-system-tricore or in one of our CI pipelines. > >> > >> Signed-off-by: Thomas Huth > >> --- > >> tests/qemu-iotests/group | 194 ++++++++++++++++++++----------------= --- > >> 1 file changed, 102 insertions(+), 92 deletions(-) > >> > >> diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group > >> index bae7718380..2ed42dcc14 100644 > >> --- a/tests/qemu-iotests/group > >> +++ b/tests/qemu-iotests/group > >> @@ -4,63 +4,73 @@ > >> # - do not start group names with a digit > >> # > >> =20 > >> +# > >> +# Some notes about the groups: > >> +# - Tests in the "quick" group should finish within some few second= s > >> +# - Tests in the "ci" group are suitable for running in CI systems.= That > >> +# means they should run with every QEMU binary (also non-x86), wi= th > >> +# every QEMU configuration (i.e. no dependency on an optional fea= ture), > >> +# at least with the qcow2 image format and all kind of host files= ystems > >> +# and users (i.e. also as "nobody" or "root"). > >=20 > > No dependancy on optional features feels a bit restrictive from my PO= V. > >=20 > > We have quite alot of testing coverage of stuff that depends on the > > crypto libraries that is important for us to run in "CI". This includ= es > > LUKS block drivers, qcow2 with LUKS, NBD with TLS. Personally I expe= ct > > all of those to be tested by CI systems. > >=20 > > IOW for a "CI" group, the bar should be higher than "no optional feat= ures" >=20 > Ok, I think I just did not write it properly. What I meant was: the tes= t > should not *fail* because of a missing feature (what some tests are > unfortunately doing). It's fine if the test detects the missing feature > and then marks the test as *skip*. Ok, this brings back the topic I don't think we ever resolved around how to best "detect" features in the iotests. eg if we want to know whether we can use TLS features, how do we decide this ? One option is to do a dummy launch of QEMU using TLS & check if it fails, but that's kind of fragile for CI. A genuine bug in QEMU might cause this dummy launch to fail, and thus mistakenly skip all the tests instead of failing. We can't look at config-host.{h,mak} either as iotests are supposed to be runnable outside of a build tree. We could look at whether gnutls is linked to QEMU but again that is going to be fragile if a bug in configure causes us to mistakenly stop enabling gnutls. For developers, dynamically probing features & skipping is reasonable. For automated CI, I think we should set clear desired featureset upfront based on what we know we are supposde to have available and not silently skip tests Regards, Daniel --=20 |: https://berrange.com -o- https://www.flickr.com/photos/dberran= ge :| |: https://libvirt.org -o- https://fstop138.berrange.c= om :| |: https://entangle-photo.org -o- https://www.instagram.com/dberran= ge :|