Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.3 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37A12C07E97 for ; Sat, 3 Jul 2021 21:38:21 +0000 (UTC) Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 534AF6128B for ; Sat, 3 Jul 2021 21:38:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 534AF6128B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=konsulko.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 41F1F80C68; Sat, 3 Jul 2021 23:38:18 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=none (p=none dis=none) header.from=konsulko.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=konsulko.com header.i=@konsulko.com header.b="eboXHFsT"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 2186182903; Sat, 3 Jul 2021 23:38:16 +0200 (CEST) Received: from mail-qk1-x730.google.com (mail-qk1-x730.google.com [IPv6:2607:f8b0:4864:20::730]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 6E7E280796 for ; Sat, 3 Jul 2021 23:38:12 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=none (p=none dis=none) header.from=konsulko.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=trini@konsulko.com Received: by mail-qk1-x730.google.com with SMTP id a4so5298972qkn.11 for ; Sat, 03 Jul 2021 14:38:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=Gc4dKTIXNrJUrbUrU3gl3CpjmpC52E89gyKioDh90t8=; b=eboXHFsTSgX9/4zq95zeDxTgpOWB3cdDoxsK00MxEWPlxjVyjtfFXggQRuU2PG6KC5 T3RVAoAt+ti3MZQKXNqRl+8yr+W9Scymh7JYesuZna5vX4LFCfCmyLm09k2tlVs9eQWi hsOlEyH5pyeBwn6LSl9iAMX7LphmRIMg0lWTs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=Gc4dKTIXNrJUrbUrU3gl3CpjmpC52E89gyKioDh90t8=; b=G9KUzQ23XRh5UOLAtjK8X3GHFaTw7quvxiLHlb4zJFLJF94tpDjY7dutUJ0BQabMfT YLBO9+UFu1wCaZvaX9ScxEnHos2rTRHjVAyYp3bgdu2vN1xY6jWdCqpjidcvm2d0A7tg of9cvhX8DSg2viJPhP2Oy9jU22CYt0OhInBaJNDVKL3iLibwvbU8ylB8GYt1A3DWKz9t 9ItvO0klQbAnOTcaBX6o0mXXPo4LD1hSyG3k1+N+9zyeMrCEu3IeHIHo/t5XWwJTLrZH 7CvoyAkOg7RyP0c7F35oJBJti14A3aZsg/hRx7a+YhasBXqm4d+k2Euo/Z/SCDcZjC3l aTxQ== X-Gm-Message-State: AOAM533pXGXu8jTp+Xbb3hCgUU8azNQmW/C0siaamozC34AdchnfVudi LKjCkrX2gMq9trTNOtYUUnz6Rw== X-Google-Smtp-Source: ABdhPJwj0mGpRjdFsWUzQI9YmDvniBuhE3y1L4tR2EsBXGLLRacgJG220kzO0zyuFXCFVSLlnWfdTw== X-Received: by 2002:a05:620a:1491:: with SMTP id w17mr6885959qkj.177.1625348291081; Sat, 03 Jul 2021 14:38:11 -0700 (PDT) Received: from bill-the-cat (2603-6081-7b01-cbda-f813-bee2-01d9-9b4c.res6.spectrum.com. [2603:6081:7b01:cbda:f813:bee2:1d9:9b4c]) by smtp.gmail.com with ESMTPSA id v15sm3388500qkp.96.2021.07.03.14.38.09 (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Sat, 03 Jul 2021 14:38:09 -0700 (PDT) Date: Sat, 3 Jul 2021 17:38:07 -0400 From: Tom Rini To: Alper Nebi Yasak Cc: u-boot@lists.denx.de, Andy Shevchenko , Heinrich Schuchardt , Akashi Takahiro , Simon Glass Subject: Re: [RFC] Start using guestfish for U-Boot fs tests Message-ID: <20210703213807.GQ9516@bill-the-cat> References: <20210702190107.GA9516@bill-the-cat> <9e231ca9-4412-f324-3870-2bc59c9dbed2@gmail.com> <20210702202403.GG9516@bill-the-cat> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="wbokbfg3PEH+QIsf" Content-Disposition: inline In-Reply-To: X-Clacks-Overhead: GNU Terry Pratchett User-Agent: Mutt/1.9.4 (2018-02-28) X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.2 at phobos.denx.de X-Virus-Status: Clean --wbokbfg3PEH+QIsf Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jul 03, 2021 at 05:27:44PM +0300, Alper Nebi Yasak wrote: >=20 >=20 > On 02/07/2021 23:24, Tom Rini wrote: > > On Fri, Jul 02, 2021 at 11:03:52PM +0300, Alper Nebi Yasak wrote: > >> On 02/07/2021 22:01, Tom Rini wrote: > >>> Hey all, > >>> > >>> I started taking a look at moving to guestfish to see if this resolves > >>> the latest problem I've run in to: > >>> https://source.denx.de/u-boot/u-boot/-/jobs/284763#L307 > >>> which I think is due to guestmount not being done in time for the tes= t. > >> > >> That failing test's setup uses virt-make-fs, different from what you're > >> changing below. I locally only see that failure for the clang build, a= nd > >> it still fails after adding time.sleep(300) after its virt-make-fs > >> calls. I don't think it's an issue in the test setup. > >=20 > > Ah good, I need to go catch up on that thread again, thanks for looking. > >=20 > >>> So I started converting things to use guestfish directly: > >>> diff --git a/test/py/tests/test_fs/conftest.py b/test/py/tests/test_f= s/conftest.py > >>> index 7325486cdb1a..e8899cfdd118 100644 > >>> --- a/test/py/tests/test_fs/conftest.py > >>> +++ b/test/py/tests/test_fs/conftest.py > >>> @@ -265,10 +265,10 @@ def fs_obj_basic(request, u_boot_config): > >>> fs_ubtype =3D fstype_to_ubname(fs_type) > >>> check_ubconfig(u_boot_config, fs_ubtype) > >>> =20 > >>> - mount_dir =3D u_boot_config.persistent_data_dir + '/mnt' > >>> + data_dir =3D u_boot_config.persistent_data_dir + '/data' > >>> =20 > >>> - small_file =3D mount_dir + '/' + SMALL_FILE > >>> - big_file =3D mount_dir + '/' + BIG_FILE > >>> + small_file =3D data_dir + '/' + SMALL_FILE > >>> + big_file =3D data_dir + '/' + BIG_FILE > >>> =20 > >>> try: > >>> =20 > >>> @@ -279,26 +279,14 @@ def fs_obj_basic(request, u_boot_config): > >>> return > >>> =20 > >>> try: > >>> - check_call('mkdir -p %s' % mount_dir, shell=3DTrue) > >>> + check_call('mkdir -p %s' % data_dir, shell=3DTrue) > >>> except CalledProcessError as err: > >>> pytest.skip('Preparing mount folder failed for filesystem: '= + fs_type + '. {}'.format(err)) > >>> call('rm -f %s' % fs_img, shell=3DTrue) > >>> return > >>> =20 > >>> try: > >>> - # Mount the image so we can populate it. > >>> - mount_fs(fs_type, fs_img, mount_dir) > >>> - except CalledProcessError as err: > >>> - pytest.skip('Mounting to folder failed for filesystem: ' + f= s_type + '. {}'.format(err)) > >>> - call('rmdir %s' % mount_dir, shell=3DTrue) > >>> - call('rm -f %s' % fs_img, shell=3DTrue) > >>> - return > >>> - > >>> - try: > >>> - # Create a subdirectory. > >>> - check_call('mkdir %s/SUBDIR' % mount_dir, shell=3DTrue) > >>> - > >>> - # Create big file in this image. > >>> + # Create big file to copy in to the image. > >>> # Note that we work only on the start 1MB, couple MBs in the= 2GB range > >>> # and the last 1 MB of the huge 2.5GB file. > >>> # So, just put random values only in those areas. > >>> @@ -309,10 +297,14 @@ def fs_obj_basic(request, u_boot_config): > >>> check_call('dd if=3D/dev/urandom of=3D%s bs=3D1M count=3D1 s= eek=3D2499' > >>> % big_file, shell=3DTrue) > >>> =20 > >>> - # Create a small file in this image. > >>> + # Create a small file to copy in to the image. > >>> check_call('dd if=3D/dev/urandom of=3D%s bs=3D1M count=3D1' > >>> % small_file, shell=3DTrue) > >>> =20 > >>> + # Copy the files in to the image and add a subdirectory. > >>> + # Create a subdirectory. > >>> + check_call('guestfish add %s : run : mount /dev/sda / : mkdi= r /SUBDIR : copy-in %s %s /' > >>> + % (fs_img, big_file, small_file), shell=3DTrue) > >> > >> It could be faster to do things within guestfish as much as possible, > >> instead of preparing the files outside and copying them in. > >=20 > > This is, I believe, as much as possible. You can't run arbitrary > > commands via guestfish, or at least I didn't see how. "dd" isn't really > > dd, for example. > >=20 > >> Also it looks like python bindings are available as python3-guestfs on > >> Debian and Ubuntu, just not on pypi.org. > >=20 > > I saw there's some reason why you can't pip install them, and wasn't > > sure it would be any faster, based on doing these commands outside of > > pytest first. > >=20 > >>> # Delete the small file copies which possibly are written as= part of a > >>> # previous test. > >>> # check_call('rm -f "%s.w"' % MB1, shell=3DTrue) > >>> @@ -357,13 +349,11 @@ def fs_obj_basic(request, u_boot_config): > >>> =20 > >>> except CalledProcessError as err: > >>> pytest.skip('Setup failed for filesystem: ' + fs_type + '. {= }'.format(err)) > >>> - umount_fs(mount_dir) > >>> return > >>> else: > >>> - umount_fs(mount_dir) > >>> yield [fs_ubtype, fs_img, md5val] > >>> finally: > >>> - call('rmdir %s' % mount_dir, shell=3DTrue) > >>> + call('rmdir %s' % data_dir, shell=3DTrue) > >>> call('rm -f %s' % fs_img, shell=3DTrue) > >>> =20 > >>> # > >>> > >>> The problem here is that a test run went from taking about 5 minutes = to > >>> taking about 17 minutes. I can reduce this to closer to 15 minutes w= ith > >>> LIBGUESTFS_BACKEND=3Ddirect and using libguestfs-make-fixed-appliance= to > >>> make an appliance we reuse. But that's still too long to be usable. > >>> I'm hoping someone has some ideas here on how to improve things. > >> > >> If libguestfs is falling back to slow emulation because /dev/kvm isn't > >> available, maybe it's appropriate to check for that and skip the fs te= sts... > >=20 > > This is, I think, part of the problem. We're jumping through a huge > > number of hoops to avoid "sudo mount ..." in the tests directly.=20 >=20 > Well, whether 'sudo mount' works also depends on CI config. Yes. But it's also how the tests can be run outside of CI, and some level of CI nodes are things we do control. I do want to get this working in the most friendly and non-privileged way possible, I just lament a little that this was all working for the case of "just use sudo", at one point, and was also relatively fast. > > thought I was getting kvm used locally, but nope, it's not. I'm going > > to take another break and see if I can figure out what I'm doing wrong > > since: > > sudo docker run --privileged --cap-add SYS_ADMIN --security-opt > > apparmor=3Dunconfined --rm -v /dev/fuse:/dev/fuse -v /dev/kvm:/dev/kvm > > isn't doing it. >=20 > It should be --device instead of -v (--volume) for /dev/{fuse,kvm}; but > just --privileged should be enough (no --cap-add, --securtiy-opt, or > --devices necessary) for Docker itself. Something like: >=20 > sudo docker run --privileged -v $(pwd):/home/uboot/u-boot --rm \ > trini/u-boot-gitlab-ci-runner:focal-20210609-01Jul2021 \ > bash /home/uboot/u-boot/test.sh Ah right. I'll give that another whirl and see if things look reasonably quick again. > where test.sh should be the things in .gitlab-ci.yml or > .azure-pipelines.yml. >=20 > I got gitlab-runner working instead and use e.g.: >=20 > gitlab-runner exec docker --docker-privileged "sandbox test.py" Right, and one of the things that's done for the gitlab runners we control (rather than the free community ones) is to run like that. --=20 Tom --wbokbfg3PEH+QIsf Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGzBAABCgAdFiEEGjx/cOCPqxcHgJu/FHw5/5Y0tywFAmDg2LsACgkQFHw5/5Y0 tyyM5wv/ZyCFJkWUJ2kvufniOV5QIt4u22CsEXSAkWYL5zB75YrzEV8SH2b+Cgqa 6tmjUmh63FNF2Mehu9q9QpUYQk0Jyw8Lu6lq3AbYKLnE80f+0FezNnA703gazpJ1 GsxoZyXhMykcZHhRzqr6EHTdm7Am9EYxyI0EnPc7vPTVzyFx8gyqNvIjd6KnH/ak UWnMoQsBDc3WKx9ezmIorIplU/TQz+YtsJRAoLhZDuzPhUBWieiez7Dyle9cOMgd RF9pI+zPHw4gF9Ci6PGrYUYR+FZjMGIdi5soe9MeYVfT+D6QXk/swbpU7YGn1KnS XyKEcfg9ijm6z4KXr7ZvMJmxh4UI3pUOCSM0BFgWSWSC6VkqoRy15d1g5x1vhm7P r4/jEf465dBEkveECyMQiK58jH3adqL3T3PDo7z2Mct+4+IcFGfw9PKbErH2Uvhx E6Ibctmj+BThV0+8Ge4+c3edoZBpndLS46Kl01uy6OuysiwZGxAEOJLHfelMnpDC jzPN/7fl =4qkL -----END PGP SIGNATURE----- --wbokbfg3PEH+QIsf--