Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.3 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C177C07E95 for ; Fri, 2 Jul 2021 20:24:15 +0000 (UTC) Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2A8D461164 for ; Fri, 2 Jul 2021 20:24:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2A8D461164 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=konsulko.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id BAFB481D48; Fri, 2 Jul 2021 22:24:11 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=none (p=none dis=none) header.from=konsulko.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=konsulko.com header.i=@konsulko.com header.b="LIrP+H7s"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 5A2E1828A1; Fri, 2 Jul 2021 22:24:10 +0200 (CEST) Received: from mail-qk1-x730.google.com (mail-qk1-x730.google.com [IPv6:2607:f8b0:4864:20::730]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id C4CF780A22 for ; Fri, 2 Jul 2021 22:24:06 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=none (p=none dis=none) header.from=konsulko.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=trini@konsulko.com Received: by mail-qk1-x730.google.com with SMTP id f6so10648663qka.0 for ; Fri, 02 Jul 2021 13:24:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=fpPlGkH/94HAEbgwNbLMkaNEpkb0xcBwAF06wJ32WQo=; b=LIrP+H7sRzM15yvROpsVyhqsbevdc9rD4Jt6/2p2Is658F9CR72XrXqz7j5O7xRcup 4I2YOzYjpX/Zp5OzgBw+xL+WL8aLtMmJ09cnydb/oY4HWQDh5QTqxzQC+YaTEiL9LR7c cqdwtmy36jpComOPRgbjDoME49aVbuvCmz+Nc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=fpPlGkH/94HAEbgwNbLMkaNEpkb0xcBwAF06wJ32WQo=; b=eo6VLB8HnvQfw/tqEf6VtDPdqSH4wrALNFqNwA4P4OTO53afLiUAW2eBaghx7PioIi tMc+LO0alOSBL/qbBHt1o9nfDMgtWXl1D5+CVvjkYbObJ5+rb4DN/zmvgXYIa/NcCik3 ywNpeO50V+sWG69SBqRs902566GWrtlFVLrqzsBcFWYVprnk+Mzdb5KoaKdc4gb4TWnz MgWY0J7s7fwdqhsFRlz+yDt91LispvW2NkZ2Xpp+4qAUokAM6wa/0lOE4pA9RbQuqcL+ wQas3gaLXAIt0qk+5F39Mv1SSkHof9gJtvQOkhHs5VCDAsBF1t6pYwbLm/iY9X440S76 mvYA== X-Gm-Message-State: AOAM5314Mi4My/WU/Tdge0FVZN2PEN1v3GvQa8t5YjBeK937/4YmWTfp POe2JW+BqF6ggtaGl3h3NplOwA== X-Google-Smtp-Source: ABdhPJyXJF+R9QwMDz9kr7N4g3MtYaGYtH+RKuZnHT8WcfewSp6+MhBil9K/kYqZkOZgQFrrphcWuQ== X-Received: by 2002:ae9:f201:: with SMTP id m1mr1539486qkg.381.1625257445665; Fri, 02 Jul 2021 13:24:05 -0700 (PDT) Received: from bill-the-cat (2603-6081-7b01-cbda-447a-194c-2139-e81a.res6.spectrum.com. [2603:6081:7b01:cbda:447a:194c:2139:e81a]) by smtp.gmail.com with ESMTPSA id h13sm1840066qkg.13.2021.07.02.13.24.04 (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 02 Jul 2021 13:24:05 -0700 (PDT) Date: Fri, 2 Jul 2021 16:24:03 -0400 From: Tom Rini To: Alper Nebi Yasak Cc: u-boot@lists.denx.de, Andy Shevchenko , Heinrich Schuchardt , Akashi Takahiro , Simon Glass Subject: Re: [RFC] Start using guestfish for U-Boot fs tests Message-ID: <20210702202403.GG9516@bill-the-cat> References: <20210702190107.GA9516@bill-the-cat> <9e231ca9-4412-f324-3870-2bc59c9dbed2@gmail.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="954UGqs5Wibs31Y1" Content-Disposition: inline In-Reply-To: <9e231ca9-4412-f324-3870-2bc59c9dbed2@gmail.com> X-Clacks-Overhead: GNU Terry Pratchett User-Agent: Mutt/1.9.4 (2018-02-28) X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.2 at phobos.denx.de X-Virus-Status: Clean --954UGqs5Wibs31Y1 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jul 02, 2021 at 11:03:52PM +0300, Alper Nebi Yasak wrote: > On 02/07/2021 22:01, Tom Rini wrote: > > Hey all, > >=20 > > I started taking a look at moving to guestfish to see if this resolves > > the latest problem I've run in to: > > https://source.denx.de/u-boot/u-boot/-/jobs/284763#L307 > > which I think is due to guestmount not being done in time for the test. >=20 > That failing test's setup uses virt-make-fs, different from what you're > changing below. I locally only see that failure for the clang build, and > it still fails after adding time.sleep(300) after its virt-make-fs > calls. I don't think it's an issue in the test setup. Ah good, I need to go catch up on that thread again, thanks for looking. > > So I started converting things to use guestfish directly: > > diff --git a/test/py/tests/test_fs/conftest.py b/test/py/tests/test_fs/= conftest.py > > index 7325486cdb1a..e8899cfdd118 100644 > > --- a/test/py/tests/test_fs/conftest.py > > +++ b/test/py/tests/test_fs/conftest.py > > @@ -265,10 +265,10 @@ def fs_obj_basic(request, u_boot_config): > > fs_ubtype =3D fstype_to_ubname(fs_type) > > check_ubconfig(u_boot_config, fs_ubtype) > > =20 > > - mount_dir =3D u_boot_config.persistent_data_dir + '/mnt' > > + data_dir =3D u_boot_config.persistent_data_dir + '/data' > > =20 > > - small_file =3D mount_dir + '/' + SMALL_FILE > > - big_file =3D mount_dir + '/' + BIG_FILE > > + small_file =3D data_dir + '/' + SMALL_FILE > > + big_file =3D data_dir + '/' + BIG_FILE > > =20 > > try: > > =20 > > @@ -279,26 +279,14 @@ def fs_obj_basic(request, u_boot_config): > > return > > =20 > > try: > > - check_call('mkdir -p %s' % mount_dir, shell=3DTrue) > > + check_call('mkdir -p %s' % data_dir, shell=3DTrue) > > except CalledProcessError as err: > > pytest.skip('Preparing mount folder failed for filesystem: ' += fs_type + '. {}'.format(err)) > > call('rm -f %s' % fs_img, shell=3DTrue) > > return > > =20 > > try: > > - # Mount the image so we can populate it. > > - mount_fs(fs_type, fs_img, mount_dir) > > - except CalledProcessError as err: > > - pytest.skip('Mounting to folder failed for filesystem: ' + fs_= type + '. {}'.format(err)) > > - call('rmdir %s' % mount_dir, shell=3DTrue) > > - call('rm -f %s' % fs_img, shell=3DTrue) > > - return > > - > > - try: > > - # Create a subdirectory. > > - check_call('mkdir %s/SUBDIR' % mount_dir, shell=3DTrue) > > - > > - # Create big file in this image. > > + # Create big file to copy in to the image. > > # Note that we work only on the start 1MB, couple MBs in the 2= GB range > > # and the last 1 MB of the huge 2.5GB file. > > # So, just put random values only in those areas. > > @@ -309,10 +297,14 @@ def fs_obj_basic(request, u_boot_config): > > check_call('dd if=3D/dev/urandom of=3D%s bs=3D1M count=3D1 see= k=3D2499' > > % big_file, shell=3DTrue) > > =20 > > - # Create a small file in this image. > > + # Create a small file to copy in to the image. > > check_call('dd if=3D/dev/urandom of=3D%s bs=3D1M count=3D1' > > % small_file, shell=3DTrue) > > =20 > > + # Copy the files in to the image and add a subdirectory. > > + # Create a subdirectory. > > + check_call('guestfish add %s : run : mount /dev/sda / : mkdir = /SUBDIR : copy-in %s %s /' > > + % (fs_img, big_file, small_file), shell=3DTrue) >=20 > It could be faster to do things within guestfish as much as possible, > instead of preparing the files outside and copying them in. This is, I believe, as much as possible. You can't run arbitrary commands via guestfish, or at least I didn't see how. "dd" isn't really dd, for example. > Also it looks like python bindings are available as python3-guestfs on > Debian and Ubuntu, just not on pypi.org. I saw there's some reason why you can't pip install them, and wasn't sure it would be any faster, based on doing these commands outside of pytest first. > > # Delete the small file copies which possibly are written as p= art of a > > # previous test. > > # check_call('rm -f "%s.w"' % MB1, shell=3DTrue) > > @@ -357,13 +349,11 @@ def fs_obj_basic(request, u_boot_config): > > =20 > > except CalledProcessError as err: > > pytest.skip('Setup failed for filesystem: ' + fs_type + '. {}'= =2Eformat(err)) > > - umount_fs(mount_dir) > > return > > else: > > - umount_fs(mount_dir) > > yield [fs_ubtype, fs_img, md5val] > > finally: > > - call('rmdir %s' % mount_dir, shell=3DTrue) > > + call('rmdir %s' % data_dir, shell=3DTrue) > > call('rm -f %s' % fs_img, shell=3DTrue) > > =20 > > # > >=20 > > The problem here is that a test run went from taking about 5 minutes to > > taking about 17 minutes. I can reduce this to closer to 15 minutes with > > LIBGUESTFS_BACKEND=3Ddirect and using libguestfs-make-fixed-appliance to > > make an appliance we reuse. But that's still too long to be usable. > > I'm hoping someone has some ideas here on how to improve things. >=20 > If libguestfs is falling back to slow emulation because /dev/kvm isn't > available, maybe it's appropriate to check for that and skip the fs tests= =2E.. This is, I think, part of the problem. We're jumping through a huge number of hoops to avoid "sudo mount ..." in the tests directly. I thought I was getting kvm used locally, but nope, it's not. I'm going to take another break and see if I can figure out what I'm doing wrong since: sudo docker run --privileged --cap-add SYS_ADMIN --security-opt apparmor=3Dunconfined --rm -v /dev/fuse:/dev/fuse -v /dev/kvm:/dev/kvm isn't doing it. --=20 Tom --954UGqs5Wibs31Y1 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGzBAABCgAdFiEEGjx/cOCPqxcHgJu/FHw5/5Y0tywFAmDfdeMACgkQFHw5/5Y0 tyxIJwwAriyX7eSPuGTceNNSOiWmSfwcr/2aOf892VlH0hE4jxMrhS+gIM24PeWX k6KjDAArGGBuAcb17xI9+o/LKuFX8hCpkFmKL0O4Ho5tv90wIZiGIT87uSBJgzFS ADXlVlKHGmwnJoUgX4NWmzVdjeMHO8fX7ylJjsyJHuPgeQyQsv5JxTS4g3Pot+vQ 6QnkBTbUOYZL0mqu+/Y87/k8GZRR+srlyl0faGYzuWs2wn8DFSK4ZN8Wr3yKDBas C2ayF48B0G3Np68q2n3vbtBVLTjEycZO3qf8M2XkgONrGzi3fRpqF31iJHKCXrZy 0rCbYQ/o1hSa52eSu/NXB4Umx/xNLbCV2tQ0pymOq048ReWr4fVozEWXA8aBgiOX kOeIOK49ueeV5VcyebTKzfdm6OhpL/w4orq+cVi3Kn3RKBzYZxdFtOhxt0NZR0r1 RKdqB+NF+/HQ7vu8it2eX/EuYY5xcNfkUJP4eE9wk0yHFmMIn/ttqquFjKqkJ+6X RC698ocV =5wBj -----END PGP SIGNATURE----- --954UGqs5Wibs31Y1--