Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.3 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49750C07E9C for ; Mon, 5 Jul 2021 16:34:30 +0000 (UTC) Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8F59261946 for ; Mon, 5 Jul 2021 16:34:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8F59261946 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id A298182BED; Mon, 5 Jul 2021 18:34:27 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="EbECUcPQ"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 5F28D82BF0; Mon, 5 Jul 2021 18:34:26 +0200 (CEST) Received: from mail-ej1-x634.google.com (mail-ej1-x634.google.com [IPv6:2a00:1450:4864:20::634]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id D2DCA82BB4 for ; Mon, 5 Jul 2021 18:34:22 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=alpernebiyasak@gmail.com Received: by mail-ej1-x634.google.com with SMTP id gb6so13112439ejc.5 for ; Mon, 05 Jul 2021 09:34:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=vL+3klxe+w6tMNIAYjW+rBLVgPex55Bmi7iP4eFTSoc=; b=EbECUcPQZG6O39cuhj5RY/r56gw7U2NeTZ8sAgAN0Nmteari7a8SXTlUQFqFF63JDA /OXLqTEfeczcaBJWIPzT7FZ69J26EEP6ew48Ri2eSMVITV05t/ngzvVEkERaMbNUgFRr unQuLuhXfCTIP+rneccUAqrHV8jXQpkwXEtOknZd56fouukuR5ie49Q/P3kMOp041rNe ooCjs2ikVx7Z84cn+niE14HNKxA2LaiXZvKaiMHVswJVPRa3fH9///BBmyTxXXncNptz GXqQfyVjKt1eNwRW6etABJyB6FIHB1OtR9ypojmzDwTdID9ouI944+wpX3WQg/7kWqJ6 kMSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=vL+3klxe+w6tMNIAYjW+rBLVgPex55Bmi7iP4eFTSoc=; b=cWTxSWP9kMAORXctr1bej+/KClYeGTHzYmkYVZQFSF6NVl1HQ5r+IXxRYTnmLvrqjh pvSibVpSDImbZ5UwLabhEPElzOPlHphZezKqQ69b724qgxpoDRBdqEcXeD+3sQ6yj0OT InqiWGGkoFBWbXU1wdw8M8tnrNLw4hiOWmBTUqx+3evtp+VjMxBBKPkSxBu72LQ40DxJ RIDX6WxcZG07iSMiI6vcRKoPLzZxbxVqT0rAAFvJRU1M2wEnk7DTMAO/HVIBTkMSr2yr wNmChVzfa3emcnK7aEoxbiPFpf9+i1bX7GADkYJmYnhMQhdsD5W5rkiq9Mnk9jaC3hRe 8GWQ== X-Gm-Message-State: AOAM530iUkDWofCEPCSIKpqncoOCdwY+ngTUdcdoCpvsUbBPFumx/f8K w/gap0YfLvxv5Tbc7Xy5S0k= X-Google-Smtp-Source: ABdhPJwQiy2kiXYNSpM89JCy9Chs4JmT6hAPRvFxOspaVsqS4rsYrK+XIfTaw12ZNnqew4RY3bs/3Q== X-Received: by 2002:a17:906:d541:: with SMTP id cr1mr11935012ejc.81.1625502862518; Mon, 05 Jul 2021 09:34:22 -0700 (PDT) Received: from [192.168.0.74] ([178.233.26.119]) by smtp.gmail.com with ESMTPSA id c15sm5624218eds.16.2021.07.05.09.34.20 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 05 Jul 2021 09:34:22 -0700 (PDT) Subject: Re: [RFC] Start using guestfish for U-Boot fs tests To: Tom Rini Cc: u-boot@lists.denx.de, Andy Shevchenko , Heinrich Schuchardt , Akashi Takahiro , Simon Glass References: <20210702190107.GA9516@bill-the-cat> <9e231ca9-4412-f324-3870-2bc59c9dbed2@gmail.com> <20210702202403.GG9516@bill-the-cat> <20210703213807.GQ9516@bill-the-cat> <20210704211459.GR9516@bill-the-cat> From: Alper Nebi Yasak Message-ID: <522534ad-ee4c-c40a-79e7-8b69fb864bc1@gmail.com> Date: Mon, 5 Jul 2021 19:34:17 +0300 User-Agent: Mozilla/5.0 (X11; Linux aarch64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <20210704211459.GR9516@bill-the-cat> Content-Type: text/plain; charset=windows-1252 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.2 at phobos.denx.de X-Virus-Status: Clean On 05/07/2021 00:14, Tom Rini wrote: > On Sat, Jul 03, 2021 at 05:38:07PM -0400, Tom Rini wrote: >> On Sat, Jul 03, 2021 at 05:27:44PM +0300, Alper Nebi Yasak wrote: >>> >>> >>> On 02/07/2021 23:24, Tom Rini wrote: >>>> On Fri, Jul 02, 2021 at 11:03:52PM +0300, Alper Nebi Yasak wrote: >>>>> On 02/07/2021 22:01, Tom Rini wrote: >>>>>> Hey all, >>>>>> >>>>>> I started taking a look at moving to guestfish to see if this resolves >>>>>> the latest problem I've run in to: >>>>>> https://source.denx.de/u-boot/u-boot/-/jobs/284763#L307 >>>>>> which I think is due to guestmount not being done in time for the test. >>>>> >>>>> That failing test's setup uses virt-make-fs, different from what you're >>>>> changing below. I locally only see that failure for the clang build, and >>>>> it still fails after adding time.sleep(300) after its virt-make-fs >>>>> calls. I don't think it's an issue in the test setup. >>>> >>>> Ah good, I need to go catch up on that thread again, thanks for looking. >>>> >>>>>> So I started converting things to use guestfish directly: >>>>>> diff --git a/test/py/tests/test_fs/conftest.py b/test/py/tests/test_fs/conftest.py >>>>>> index 7325486cdb1a..e8899cfdd118 100644 >>>>>> --- a/test/py/tests/test_fs/conftest.py >>>>>> +++ b/test/py/tests/test_fs/conftest.py >>>>>> @@ -265,10 +265,10 @@ def fs_obj_basic(request, u_boot_config): >>>>>> fs_ubtype = fstype_to_ubname(fs_type) >>>>>> check_ubconfig(u_boot_config, fs_ubtype) >>>>>> >>>>>> - mount_dir = u_boot_config.persistent_data_dir + '/mnt' >>>>>> + data_dir = u_boot_config.persistent_data_dir + '/data' >>>>>> >>>>>> - small_file = mount_dir + '/' + SMALL_FILE >>>>>> - big_file = mount_dir + '/' + BIG_FILE >>>>>> + small_file = data_dir + '/' + SMALL_FILE >>>>>> + big_file = data_dir + '/' + BIG_FILE >>>>>> >>>>>> try: >>>>>> >>>>>> @@ -279,26 +279,14 @@ def fs_obj_basic(request, u_boot_config): >>>>>> return >>>>>> >>>>>> try: >>>>>> - check_call('mkdir -p %s' % mount_dir, shell=True) >>>>>> + check_call('mkdir -p %s' % data_dir, shell=True) >>>>>> except CalledProcessError as err: >>>>>> pytest.skip('Preparing mount folder failed for filesystem: ' + fs_type + '. {}'.format(err)) >>>>>> call('rm -f %s' % fs_img, shell=True) >>>>>> return >>>>>> >>>>>> try: >>>>>> - # Mount the image so we can populate it. >>>>>> - mount_fs(fs_type, fs_img, mount_dir) >>>>>> - except CalledProcessError as err: >>>>>> - pytest.skip('Mounting to folder failed for filesystem: ' + fs_type + '. {}'.format(err)) >>>>>> - call('rmdir %s' % mount_dir, shell=True) >>>>>> - call('rm -f %s' % fs_img, shell=True) >>>>>> - return >>>>>> - >>>>>> - try: >>>>>> - # Create a subdirectory. >>>>>> - check_call('mkdir %s/SUBDIR' % mount_dir, shell=True) >>>>>> - >>>>>> - # Create big file in this image. >>>>>> + # Create big file to copy in to the image. >>>>>> # Note that we work only on the start 1MB, couple MBs in the 2GB range >>>>>> # and the last 1 MB of the huge 2.5GB file. >>>>>> # So, just put random values only in those areas. >>>>>> @@ -309,10 +297,14 @@ def fs_obj_basic(request, u_boot_config): >>>>>> check_call('dd if=/dev/urandom of=%s bs=1M count=1 seek=2499' >>>>>> % big_file, shell=True) >>>>>> >>>>>> - # Create a small file in this image. >>>>>> + # Create a small file to copy in to the image. >>>>>> check_call('dd if=/dev/urandom of=%s bs=1M count=1' >>>>>> % small_file, shell=True) >>>>>> >>>>>> + # Copy the files in to the image and add a subdirectory. >>>>>> + # Create a subdirectory. >>>>>> + check_call('guestfish add %s : run : mount /dev/sda / : mkdir /SUBDIR : copy-in %s %s /' >>>>>> + % (fs_img, big_file, small_file), shell=True) >>>>> >>>>> It could be faster to do things within guestfish as much as possible, >>>>> instead of preparing the files outside and copying them in. >>>> >>>> This is, I believe, as much as possible. You can't run arbitrary >>>> commands via guestfish, or at least I didn't see how. "dd" isn't really >>>> dd, for example. I guess something like: dd if=/dev/urandom of="rand1.bin" bs=1M count=1 dd if=/dev/urandom of="rand2.bin" bs=1M count=2 dd if=/dev/urandom of="rand3.bin" bs=1M count=1 dd if=/dev/urandom of="rand4.bin" bs=1M count=1 fallocate -l 3GiB disk.img mkfs.vfat disk.img guestfish <>>>> Also it looks like python bindings are available as python3-guestfs on >>>>> Debian and Ubuntu, just not on pypi.org. >>>> >>>> I saw there's some reason why you can't pip install them, and wasn't >>>> sure it would be any faster, based on doing these commands outside of >>>> pytest first. >>>> >>>>>> # Delete the small file copies which possibly are written as part of a >>>>>> # previous test. >>>>>> # check_call('rm -f "%s.w"' % MB1, shell=True) >>>>>> @@ -357,13 +349,11 @@ def fs_obj_basic(request, u_boot_config): >>>>>> >>>>>> except CalledProcessError as err: >>>>>> pytest.skip('Setup failed for filesystem: ' + fs_type + '. {}'.format(err)) >>>>>> - umount_fs(mount_dir) >>>>>> return >>>>>> else: >>>>>> - umount_fs(mount_dir) >>>>>> yield [fs_ubtype, fs_img, md5val] >>>>>> finally: >>>>>> - call('rmdir %s' % mount_dir, shell=True) >>>>>> + call('rmdir %s' % data_dir, shell=True) >>>>>> call('rm -f %s' % fs_img, shell=True) >>>>>> >>>>>> # >>>>>> >>>>>> The problem here is that a test run went from taking about 5 minutes to >>>>>> taking about 17 minutes. I can reduce this to closer to 15 minutes with >>>>>> LIBGUESTFS_BACKEND=direct and using libguestfs-make-fixed-appliance to >>>>>> make an appliance we reuse. But that's still too long to be usable. >>>>>> I'm hoping someone has some ideas here on how to improve things. >>>>> >>>>> If libguestfs is falling back to slow emulation because /dev/kvm isn't >>>>> available, maybe it's appropriate to check for that and skip the fs tests... >>>> >>>> This is, I think, part of the problem. We're jumping through a huge >>>> number of hoops to avoid "sudo mount ..." in the tests directly. >>> >>> Well, whether 'sudo mount' works also depends on CI config. >> >> Yes. But it's also how the tests can be run outside of CI, and some >> level of CI nodes are things we do control. I do want to get this >> working in the most friendly and non-privileged way possible, I just >> lament a little that this was all working for the case of "just use >> sudo", at one point, and was also relatively fast. >> >>>> thought I was getting kvm used locally, but nope, it's not. I'm going >>>> to take another break and see if I can figure out what I'm doing wrong >>>> since: >>>> sudo docker run --privileged --cap-add SYS_ADMIN --security-opt >>>> apparmor=unconfined --rm -v /dev/fuse:/dev/fuse -v /dev/kvm:/dev/kvm >>>> isn't doing it. >>> >>> It should be --device instead of -v (--volume) for /dev/{fuse,kvm}; but >>> just --privileged should be enough (no --cap-add, --securtiy-opt, or >>> --devices necessary) for Docker itself. Something like: >>> >>> sudo docker run --privileged -v $(pwd):/home/uboot/u-boot --rm \ >>> trini/u-boot-gitlab-ci-runner:focal-20210609-01Jul2021 \ >>> bash /home/uboot/u-boot/test.sh >> >> Ah right. I'll give that another whirl and see if things look >> reasonably quick again. > > JFYI, I managed to, fairly easily, swap the "focal" Dockerfile over > instead to Debian Bullseye or Buster "slim" based containers, but in > both cases, just doing some simple guestfish commands like: > dd if=/dev/zero of=/tmp/test.img bs=1M count=4096 > mkfs.vfat /tmp/test.img > export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 > export LIBGUESTFS_BACKEND=uml > export LIBGUESTFS_HV=/usr/bin/linux > guestfish add /tmp/test.img : run : mount /dev/sda / : mkdir /SUBDIR > > results in UML blowing up in generic looking failures on each. I'm > setting UML aside for now. I couldn't get the UML backend to work either, e.g. anything trying to write to a guestmount-ed directory becomes unresponsive.