Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.2 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77B8CC07E95 for ; Sun, 4 Jul 2021 21:15:14 +0000 (UTC) Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8AADE6135B for ; Sun, 4 Jul 2021 21:15:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8AADE6135B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=konsulko.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id D334682B20; Sun, 4 Jul 2021 23:15:10 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=none (p=none dis=none) header.from=konsulko.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=konsulko.com header.i=@konsulko.com header.b="HUOq7jZW"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 7B50D82B77; Sun, 4 Jul 2021 23:15:08 +0200 (CEST) Received: from mail-qv1-xf33.google.com (mail-qv1-xf33.google.com [IPv6:2607:f8b0:4864:20::f33]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 5802782A29 for ; Sun, 4 Jul 2021 23:15:04 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=none (p=none dis=none) header.from=konsulko.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=trini@konsulko.com Received: by mail-qv1-xf33.google.com with SMTP id cn9so5786958qvb.3 for ; Sun, 04 Jul 2021 14:15:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=rUeJsv63XimnUGBdB5/+kj3zjuDbpegLbgeTztwZVaw=; b=HUOq7jZWPqYKP7rraCIjujvwb35BEOpWxKFIIEVPhn8jYO1Y67YQwjw00gmYYlgjsQ osdlHgQpeQ1uE1JrUBu9TaPI37RG5KQbOnjkTc2lHnCxVVp+hLK7xVWneNxQN9r4cNXV bf640oIWkGE8icCtE70PaFsSoIej+r/eUq3N0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=rUeJsv63XimnUGBdB5/+kj3zjuDbpegLbgeTztwZVaw=; b=QoROsaujDtoDYD1imD4RkiN3NqwOn7/xggNWqkTBFVBZ++72hP0b3Eu1/k1PqKi9Fo jSoMKH8sK7fr8QYGOhepob6rrHuFzOeXMv4fco5BlEjVO+c8KfnnNwPtjL3za5JRy2FF SnAGEdpHi+5PJOeZ9CZTrMjKLxw+3bH4zuBX4plZpgv9vCQ6TzyCzSCMxiOPDD8yHUhU PJ+p6LL5CLqrIyYtVWHhmYYIaZHHJdoD+nsBe+IFJPe24SdINtAkoG0krsbxpf+VNNea tl6JMregezNskbLyh9arB6N5uloGRCR7BmulTwgAlPiEkOxD5Qp7etwsSMml/hcRrtgd Tr1A== X-Gm-Message-State: AOAM533gOgqi9ynHjFLKg2sGDzoxNiDoVNKE6tPVvA0MdQrwsv/L7wSN NdoRqvTDh+WaZtCzHcUKqDwN6Q== X-Google-Smtp-Source: ABdhPJzDSI7TGRhYaCcShchI9sl+Jsi2DwoGUQY+qwciFXdX05BO3rtB8lzT75xI48m8NMYHp9vOdg== X-Received: by 2002:a0c:c401:: with SMTP id r1mr10279392qvi.47.1625433302894; Sun, 04 Jul 2021 14:15:02 -0700 (PDT) Received: from bill-the-cat (2603-6081-7b01-cbda-9d20-4c29-3c3e-2cef.res6.spectrum.com. [2603:6081:7b01:cbda:9d20:4c29:3c3e:2cef]) by smtp.gmail.com with ESMTPSA id f25sm3797761qto.13.2021.07.04.14.15.01 (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Sun, 04 Jul 2021 14:15:01 -0700 (PDT) Date: Sun, 4 Jul 2021 17:14:59 -0400 From: Tom Rini To: Alper Nebi Yasak Cc: u-boot@lists.denx.de, Andy Shevchenko , Heinrich Schuchardt , Akashi Takahiro , Simon Glass Subject: Re: [RFC] Start using guestfish for U-Boot fs tests Message-ID: <20210704211459.GR9516@bill-the-cat> References: <20210702190107.GA9516@bill-the-cat> <9e231ca9-4412-f324-3870-2bc59c9dbed2@gmail.com> <20210702202403.GG9516@bill-the-cat> <20210703213807.GQ9516@bill-the-cat> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="vs5BQ8PjXs/IvehQ" Content-Disposition: inline In-Reply-To: <20210703213807.GQ9516@bill-the-cat> X-Clacks-Overhead: GNU Terry Pratchett User-Agent: Mutt/1.9.4 (2018-02-28) X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.2 at phobos.denx.de X-Virus-Status: Clean --vs5BQ8PjXs/IvehQ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jul 03, 2021 at 05:38:07PM -0400, Tom Rini wrote: > On Sat, Jul 03, 2021 at 05:27:44PM +0300, Alper Nebi Yasak wrote: > >=20 > >=20 > > On 02/07/2021 23:24, Tom Rini wrote: > > > On Fri, Jul 02, 2021 at 11:03:52PM +0300, Alper Nebi Yasak wrote: > > >> On 02/07/2021 22:01, Tom Rini wrote: > > >>> Hey all, > > >>> > > >>> I started taking a look at moving to guestfish to see if this resol= ves > > >>> the latest problem I've run in to: > > >>> https://source.denx.de/u-boot/u-boot/-/jobs/284763#L307 > > >>> which I think is due to guestmount not being done in time for the t= est. > > >> > > >> That failing test's setup uses virt-make-fs, different from what you= 're > > >> changing below. I locally only see that failure for the clang build,= and > > >> it still fails after adding time.sleep(300) after its virt-make-fs > > >> calls. I don't think it's an issue in the test setup. > > >=20 > > > Ah good, I need to go catch up on that thread again, thanks for looki= ng. > > >=20 > > >>> So I started converting things to use guestfish directly: > > >>> diff --git a/test/py/tests/test_fs/conftest.py b/test/py/tests/test= _fs/conftest.py > > >>> index 7325486cdb1a..e8899cfdd118 100644 > > >>> --- a/test/py/tests/test_fs/conftest.py > > >>> +++ b/test/py/tests/test_fs/conftest.py > > >>> @@ -265,10 +265,10 @@ def fs_obj_basic(request, u_boot_config): > > >>> fs_ubtype =3D fstype_to_ubname(fs_type) > > >>> check_ubconfig(u_boot_config, fs_ubtype) > > >>> =20 > > >>> - mount_dir =3D u_boot_config.persistent_data_dir + '/mnt' > > >>> + data_dir =3D u_boot_config.persistent_data_dir + '/data' > > >>> =20 > > >>> - small_file =3D mount_dir + '/' + SMALL_FILE > > >>> - big_file =3D mount_dir + '/' + BIG_FILE > > >>> + small_file =3D data_dir + '/' + SMALL_FILE > > >>> + big_file =3D data_dir + '/' + BIG_FILE > > >>> =20 > > >>> try: > > >>> =20 > > >>> @@ -279,26 +279,14 @@ def fs_obj_basic(request, u_boot_config): > > >>> return > > >>> =20 > > >>> try: > > >>> - check_call('mkdir -p %s' % mount_dir, shell=3DTrue) > > >>> + check_call('mkdir -p %s' % data_dir, shell=3DTrue) > > >>> except CalledProcessError as err: > > >>> pytest.skip('Preparing mount folder failed for filesystem:= ' + fs_type + '. {}'.format(err)) > > >>> call('rm -f %s' % fs_img, shell=3DTrue) > > >>> return > > >>> =20 > > >>> try: > > >>> - # Mount the image so we can populate it. > > >>> - mount_fs(fs_type, fs_img, mount_dir) > > >>> - except CalledProcessError as err: > > >>> - pytest.skip('Mounting to folder failed for filesystem: ' += fs_type + '. {}'.format(err)) > > >>> - call('rmdir %s' % mount_dir, shell=3DTrue) > > >>> - call('rm -f %s' % fs_img, shell=3DTrue) > > >>> - return > > >>> - > > >>> - try: > > >>> - # Create a subdirectory. > > >>> - check_call('mkdir %s/SUBDIR' % mount_dir, shell=3DTrue) > > >>> - > > >>> - # Create big file in this image. > > >>> + # Create big file to copy in to the image. > > >>> # Note that we work only on the start 1MB, couple MBs in t= he 2GB range > > >>> # and the last 1 MB of the huge 2.5GB file. > > >>> # So, just put random values only in those areas. > > >>> @@ -309,10 +297,14 @@ def fs_obj_basic(request, u_boot_config): > > >>> check_call('dd if=3D/dev/urandom of=3D%s bs=3D1M count=3D1= seek=3D2499' > > >>> % big_file, shell=3DTrue) > > >>> =20 > > >>> - # Create a small file in this image. > > >>> + # Create a small file to copy in to the image. > > >>> check_call('dd if=3D/dev/urandom of=3D%s bs=3D1M count=3D1' > > >>> % small_file, shell=3DTrue) > > >>> =20 > > >>> + # Copy the files in to the image and add a subdirectory. > > >>> + # Create a subdirectory. > > >>> + check_call('guestfish add %s : run : mount /dev/sda / : mk= dir /SUBDIR : copy-in %s %s /' > > >>> + % (fs_img, big_file, small_file), shell=3DTrue) > > >> > > >> It could be faster to do things within guestfish as much as possible, > > >> instead of preparing the files outside and copying them in. > > >=20 > > > This is, I believe, as much as possible. You can't run arbitrary > > > commands via guestfish, or at least I didn't see how. "dd" isn't rea= lly > > > dd, for example. > > >=20 > > >> Also it looks like python bindings are available as python3-guestfs = on > > >> Debian and Ubuntu, just not on pypi.org. > > >=20 > > > I saw there's some reason why you can't pip install them, and wasn't > > > sure it would be any faster, based on doing these commands outside of > > > pytest first. > > >=20 > > >>> # Delete the small file copies which possibly are written = as part of a > > >>> # previous test. > > >>> # check_call('rm -f "%s.w"' % MB1, shell=3DTrue) > > >>> @@ -357,13 +349,11 @@ def fs_obj_basic(request, u_boot_config): > > >>> =20 > > >>> except CalledProcessError as err: > > >>> pytest.skip('Setup failed for filesystem: ' + fs_type + '.= {}'.format(err)) > > >>> - umount_fs(mount_dir) > > >>> return > > >>> else: > > >>> - umount_fs(mount_dir) > > >>> yield [fs_ubtype, fs_img, md5val] > > >>> finally: > > >>> - call('rmdir %s' % mount_dir, shell=3DTrue) > > >>> + call('rmdir %s' % data_dir, shell=3DTrue) > > >>> call('rm -f %s' % fs_img, shell=3DTrue) > > >>> =20 > > >>> # > > >>> > > >>> The problem here is that a test run went from taking about 5 minute= s to > > >>> taking about 17 minutes. I can reduce this to closer to 15 minutes= with > > >>> LIBGUESTFS_BACKEND=3Ddirect and using libguestfs-make-fixed-applian= ce to > > >>> make an appliance we reuse. But that's still too long to be usable. > > >>> I'm hoping someone has some ideas here on how to improve things. > > >> > > >> If libguestfs is falling back to slow emulation because /dev/kvm isn= 't > > >> available, maybe it's appropriate to check for that and skip the fs = tests... > > >=20 > > > This is, I think, part of the problem. We're jumping through a huge > > > number of hoops to avoid "sudo mount ..." in the tests directly.=20 > >=20 > > Well, whether 'sudo mount' works also depends on CI config. >=20 > Yes. But it's also how the tests can be run outside of CI, and some > level of CI nodes are things we do control. I do want to get this > working in the most friendly and non-privileged way possible, I just > lament a little that this was all working for the case of "just use > sudo", at one point, and was also relatively fast. >=20 > > > thought I was getting kvm used locally, but nope, it's not. I'm going > > > to take another break and see if I can figure out what I'm doing wrong > > > since: > > > sudo docker run --privileged --cap-add SYS_ADMIN --security-opt > > > apparmor=3Dunconfined --rm -v /dev/fuse:/dev/fuse -v /dev/kvm:/dev/kvm > > > isn't doing it. > >=20 > > It should be --device instead of -v (--volume) for /dev/{fuse,kvm}; but > > just --privileged should be enough (no --cap-add, --securtiy-opt, or > > --devices necessary) for Docker itself. Something like: > >=20 > > sudo docker run --privileged -v $(pwd):/home/uboot/u-boot --rm \ > > trini/u-boot-gitlab-ci-runner:focal-20210609-01Jul2021 \ > > bash /home/uboot/u-boot/test.sh >=20 > Ah right. I'll give that another whirl and see if things look > reasonably quick again. JFYI, I managed to, fairly easily, swap the "focal" Dockerfile over instead to Debian Bullseye or Buster "slim" based containers, but in both cases, just doing some simple guestfish commands like: dd if=3D/dev/zero of=3D/tmp/test.img bs=3D1M count=3D4096 mkfs.vfat /tmp/test.img export LIBGUESTFS_DEBUG=3D1 LIBGUESTFS_TRACE=3D1 export LIBGUESTFS_BACKEND=3Duml export LIBGUESTFS_HV=3D/usr/bin/linux guestfish add /tmp/test.img : run : mount /dev/sda / : mkdir /SUBDIR results in UML blowing up in generic looking failures on each. I'm setting UML aside for now. --=20 Tom --vs5BQ8PjXs/IvehQ Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGzBAABCgAdFiEEGjx/cOCPqxcHgJu/FHw5/5Y0tywFAmDiJMkACgkQFHw5/5Y0 tywa7wwAljT9iJGK4A2rbCFuwyPHKFqIUcBpVofmpbDp5rSxiAU8kTkW7/LWnMmT htWX8ge8kJD4YDV7MVNxWejhueJHJZS4gTu2puvgGQcoKBJnkgari4ojR2O0Hb9B eObgcZcNHiR7vztkFAdKhhfJ7wNCJvm7ZpsNsDbLmdai/fO8hlFlkijldN15dHpQ 7LMmhkmXafona/RsZ79V2a8U6Zyemuthn4vxGGJnKoyJyoLgDlp+c5NuFdejlWB3 h0HwP8cvD5o0opr8+YODm+CUX8EYdQHL+fNlKBPNb61NGDhD/1xUgXPjSzvyg+CP XMCD1XTsb4g1qF8sR9y/YigJyqqHl11C8/OJet0YL9ZSwpI3MZPKBMjEIb593Wkg r20YzDIocYlIMhX3t2uXrkhX2DJQHtx54JEMe2wyP6BKOepCyDrBdkTqOgjklnCi 8UQbzsOgCMIEEqHJSpUYGIcqgKvMnGKKE2rEepyYlZc/qnkwqftPnA0bY/YrMgFm MhLxeexX =utPz -----END PGP SIGNATURE----- --vs5BQ8PjXs/IvehQ--