Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.3 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9B13C07E95 for ; Fri, 2 Jul 2021 19:01:20 +0000 (UTC) Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B59EC61410 for ; Fri, 2 Jul 2021 19:01:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B59EC61410 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=konsulko.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 8B42280A22; Fri, 2 Jul 2021 21:01:17 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=none (p=none dis=none) header.from=konsulko.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=konsulko.com header.i=@konsulko.com header.b="fRKhNG22"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id A5C31828A1; Fri, 2 Jul 2021 21:01:15 +0200 (CEST) Received: from mail-qk1-x734.google.com (mail-qk1-x734.google.com [IPv6:2607:f8b0:4864:20::734]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 1FCC380A22 for ; Fri, 2 Jul 2021 21:01:12 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=none (p=none dis=none) header.from=konsulko.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=trini@konsulko.com Received: by mail-qk1-x734.google.com with SMTP id f6so10374963qka.0 for ; Fri, 02 Jul 2021 12:01:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; h=date:from:to:subject:message-id:mime-version:content-disposition :user-agent; bh=0mFcgcyzeeESarSDcVCfxYxMinTGIUzCHAitJDopyS4=; b=fRKhNG22omm3H1qHfij9v0QFlzOEeRUQWNcCImiCnO5pdFf18MDKU0RPGmAtHJKJ1S Lhjfgdh2VL+OLnqy5pXlC1dk48JD4Ymh1CqweQ/KGPMOjC39SIL5f7oLsfi6dEfSGaoL Vk8ARb7R9HVbN7eCfh5HF3LL4sYiJUsgM/dys= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:subject:message-id:mime-version :content-disposition:user-agent; bh=0mFcgcyzeeESarSDcVCfxYxMinTGIUzCHAitJDopyS4=; b=pBKGmln3kUPwtl7w6u1UhQwN+vblm/MAd9TWbocCK2ADYkmZUyc12cddMdXfq3p38r rqCqB2T0hiVnRe7U5RW5pNrRHpGDtS2i8AHR/YHzn2yn4DJ6Kp+4cWG1Xz/BaWC0/kHV zqVYjdrAKDkCZCQlCAHhx7UnhY6yHEnLCN0DYowtMoa6quSd/iYOI8auZv2PaxrpmUxu kZfb66S1LPQ/Jsm2/2k4qFCpUBLAY9RZYI6ejl6Z3DaqfCzqWUJZ4197FsI/SqysK3Qk oHOU6Sj9dwWlHTSWsCgOdUBnNqbzF0VuAJISZ+4c+2cPTlz9nddsnoukgRS1dw+vD61r rf/g== X-Gm-Message-State: AOAM531+tIJYS0U5nK6lbIX1sNiaGW4K4ZCdxLW3+qjnBpyCyYhFYxdd ddAYDks25VzRLeU6RT4cA+cCThvAczGIKA== X-Google-Smtp-Source: ABdhPJzQT3duA0rSlsDdRDH/1D+eNlroqgNQEy85i1fGkf3hd7I0WFHx9QLc1jzBvLlpHo6rmtnGNw== X-Received: by 2002:a37:8345:: with SMTP id f66mr1266769qkd.396.1625252470436; Fri, 02 Jul 2021 12:01:10 -0700 (PDT) Received: from bill-the-cat (2603-6081-7b01-cbda-447a-194c-2139-e81a.res6.spectrum.com. [2603:6081:7b01:cbda:447a:194c:2139:e81a]) by smtp.gmail.com with ESMTPSA id y11sm1691484qkj.48.2021.07.02.12.01.08 (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 02 Jul 2021 12:01:09 -0700 (PDT) Date: Fri, 2 Jul 2021 15:01:07 -0400 From: Tom Rini To: u-boot@lists.denx.de, Andy Shevchenko , Alper Nebi Yasak , Heinrich Schuchardt , Akashi Takahiro , Simon Glass Subject: [RFC] Start using guestfish for U-Boot fs tests Message-ID: <20210702190107.GA9516@bill-the-cat> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="dMZfW4MPVHWzgUqg" Content-Disposition: inline X-Clacks-Overhead: GNU Terry Pratchett User-Agent: Mutt/1.9.4 (2018-02-28) X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.2 at phobos.denx.de X-Virus-Status: Clean --dMZfW4MPVHWzgUqg Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hey all, I started taking a look at moving to guestfish to see if this resolves the latest problem I've run in to: https://source.denx.de/u-boot/u-boot/-/jobs/284763#L307 which I think is due to guestmount not being done in time for the test. So I started converting things to use guestfish directly: diff --git a/test/py/tests/test_fs/conftest.py b/test/py/tests/test_fs/conf= test.py index 7325486cdb1a..e8899cfdd118 100644 --- a/test/py/tests/test_fs/conftest.py +++ b/test/py/tests/test_fs/conftest.py @@ -265,10 +265,10 @@ def fs_obj_basic(request, u_boot_config): fs_ubtype =3D fstype_to_ubname(fs_type) check_ubconfig(u_boot_config, fs_ubtype) =20 - mount_dir =3D u_boot_config.persistent_data_dir + '/mnt' + data_dir =3D u_boot_config.persistent_data_dir + '/data' =20 - small_file =3D mount_dir + '/' + SMALL_FILE - big_file =3D mount_dir + '/' + BIG_FILE + small_file =3D data_dir + '/' + SMALL_FILE + big_file =3D data_dir + '/' + BIG_FILE =20 try: =20 @@ -279,26 +279,14 @@ def fs_obj_basic(request, u_boot_config): return =20 try: - check_call('mkdir -p %s' % mount_dir, shell=3DTrue) + check_call('mkdir -p %s' % data_dir, shell=3DTrue) except CalledProcessError as err: pytest.skip('Preparing mount folder failed for filesystem: ' + fs_= type + '. {}'.format(err)) call('rm -f %s' % fs_img, shell=3DTrue) return =20 try: - # Mount the image so we can populate it. - mount_fs(fs_type, fs_img, mount_dir) - except CalledProcessError as err: - pytest.skip('Mounting to folder failed for filesystem: ' + fs_type= + '. {}'.format(err)) - call('rmdir %s' % mount_dir, shell=3DTrue) - call('rm -f %s' % fs_img, shell=3DTrue) - return - - try: - # Create a subdirectory. - check_call('mkdir %s/SUBDIR' % mount_dir, shell=3DTrue) - - # Create big file in this image. + # Create big file to copy in to the image. # Note that we work only on the start 1MB, couple MBs in the 2GB r= ange # and the last 1 MB of the huge 2.5GB file. # So, just put random values only in those areas. @@ -309,10 +297,14 @@ def fs_obj_basic(request, u_boot_config): check_call('dd if=3D/dev/urandom of=3D%s bs=3D1M count=3D1 seek=3D= 2499' % big_file, shell=3DTrue) =20 - # Create a small file in this image. + # Create a small file to copy in to the image. check_call('dd if=3D/dev/urandom of=3D%s bs=3D1M count=3D1' % small_file, shell=3DTrue) =20 + # Copy the files in to the image and add a subdirectory. + # Create a subdirectory. + check_call('guestfish add %s : run : mount /dev/sda / : mkdir /SUB= DIR : copy-in %s %s /' + % (fs_img, big_file, small_file), shell=3DTrue) # Delete the small file copies which possibly are written as part = of a # previous test. # check_call('rm -f "%s.w"' % MB1, shell=3DTrue) @@ -357,13 +349,11 @@ def fs_obj_basic(request, u_boot_config): =20 except CalledProcessError as err: pytest.skip('Setup failed for filesystem: ' + fs_type + '. {}'.for= mat(err)) - umount_fs(mount_dir) return else: - umount_fs(mount_dir) yield [fs_ubtype, fs_img, md5val] finally: - call('rmdir %s' % mount_dir, shell=3DTrue) + call('rmdir %s' % data_dir, shell=3DTrue) call('rm -f %s' % fs_img, shell=3DTrue) =20 # The problem here is that a test run went from taking about 5 minutes to taking about 17 minutes. I can reduce this to closer to 15 minutes with LIBGUESTFS_BACKEND=3Ddirect and using libguestfs-make-fixed-appliance to make an appliance we reuse. But that's still too long to be usable. I'm hoping someone has some ideas here on how to improve things. --=20 Tom --dMZfW4MPVHWzgUqg Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGzBAABCgAdFiEEGjx/cOCPqxcHgJu/FHw5/5Y0tywFAmDfYm8ACgkQFHw5/5Y0 tyythAv6Aw5SE4To1qnoqkffOtIcZqxsiGbZ9Nx2dtZO+L77cUxMDBtPtTVRoDh6 S6aJ0H37JWEi+SabYyY8jvIOcc6hGQAdYDVuXIvSCR7mWfMgq1d2frFimMcu13nf NjU4HYd0uliNPIEmYNCKJvq3cjkAi/SVj5xf3XeR+YXdMWWPjIlm2flFVUimyLBm We5Jiw0lBOZlX3Wdm9H8fAuRqqcifRPtxVWuVqguKpPhV0ov9BifrmqUcA937X13 RDdsqDbPb/lzIwx7FFv8K6ydGn+S4WDpNMbzQ9U9IpQQfxRPOsHt8PJXxolrs3Z9 YzgDXvktvRcw7o4+z7PL24drTZIWl0iytAoRDktoW/CUmH3zSn2RQihGv/U58m4X HD6thryL8jVkcBgfLTXOYJaAavA6sQlmD5aSfV11NAKdoP7fmuGK7Nk27dg/cHxW ZoJknyfUmmZNqrreR7FWwGM0MeV6WTVZLL26/Vflev0Ay9HVelndDnCRd3YLWJ9o e2u1FX1n =LowL -----END PGP SIGNATURE----- --dMZfW4MPVHWzgUqg--