Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.3 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D35CC07E95 for ; Fri, 2 Jul 2021 20:14:12 +0000 (UTC) Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EC2A261405 for ; Fri, 2 Jul 2021 20:14:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EC2A261405 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 0AC0D829BE; Fri, 2 Jul 2021 22:14:10 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="WYf8oGvj"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id A081F829F0; Fri, 2 Jul 2021 22:14:07 +0200 (CEST) Received: from mail-ed1-x535.google.com (mail-ed1-x535.google.com [IPv6:2a00:1450:4864:20::535]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 3EF43828A4 for ; Fri, 2 Jul 2021 22:14:04 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=alpernebiyasak@gmail.com Received: by mail-ed1-x535.google.com with SMTP id i5so14847058eds.1 for ; Fri, 02 Jul 2021 13:14:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:references:message-id:date:user-agent:mime-version :in-reply-to:content-language:content-transfer-encoding; bh=4LzYo/jnLMfxVrmYxbTwbIqMRNE7QBRUpvW/YtpZGqY=; b=WYf8oGvjXkHVMKwcFLZqg3g4UNDRooSl2GxRaaHbc6IAP82rPtTvLXTRFqIaJ2u/8T RZyw00GZmHzS4GSJAvoxamywX6UIHv0WKNgxB8NVnqJ2lqx3dBsPBcadtKeo3JVkFXAL hm5NOiLf9KHudm2Oyhx1CnndkO0Uynfw4EGzowi/xweP0WS+R6qU5ZnZh//yycSkxlHE er4vY76flW600p/NmYCeM5swFJLmUnMBcGFkFIg++TemJtXHpc/4BcWWZNOpgD30KhBo TAAWLM5+UUOrYOPP4mOXJfbyuJkdpXhWu45xk7eIBDtw8F2u7TyJdA9Mo93cZjGoXhnR 84qQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:references:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=4LzYo/jnLMfxVrmYxbTwbIqMRNE7QBRUpvW/YtpZGqY=; b=SJHPukImdp1iSJXoxrbo6GoEuVfzolaqOsepnEYvxxkEGbWodztErLJbELgCCKqR/X lsovnKwNeMbo1IiqwOS8XsGBfD6fZWItHYnTGxoCQdbRs3WanAFH4QAO7O8xz7Em7KRc vNOlHvSY4lJw1Js0nviP5QaBcfpTVV1EFgjXxlp6MqmQhaRHMzQJm0IVWDAyAUQcRWMD 3Z46Iw6hGq2rsHIMs/p4ITQI0SOj7s8hlh22k4XDOPAJiz/e3ESjTsvmkEOcegYMCj/0 itohmzICMZ1BthD7ivG7J/Jddeym/prG+91wKTVXLdraiZLJJDiYDXDEPqnmVNC47q2B XIHA== X-Gm-Message-State: AOAM5319Yu/dtdwRATDnYbfeXOXlPOGQzk6lEiQRYJy63e62Ct067eU8 tck4GCfs9o84pQGpXtuOgJI= X-Google-Smtp-Source: ABdhPJxtyq5H3ZuqUhxnSUkxSvA7cLBsEHGtEI79CTG0s6jZXt++u1ywbjQ5CVbrQPZpiCn+zj0OPQ== X-Received: by 2002:a05:6402:4243:: with SMTP id g3mr1548752edb.118.1625256843915; Fri, 02 Jul 2021 13:14:03 -0700 (PDT) Received: from [192.168.0.74] ([178.233.26.119]) by smtp.gmail.com with ESMTPSA id u4sm1367902eje.81.2021.07.02.13.14.02 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 02 Jul 2021 13:14:03 -0700 (PDT) Subject: Re: [RFC] Start using guestfish for U-Boot fs tests From: Alper Nebi Yasak To: Tom Rini , u-boot@lists.denx.de, Andy Shevchenko , Heinrich Schuchardt , Akashi Takahiro , Simon Glass References: <20210702190107.GA9516@bill-the-cat> <9e231ca9-4412-f324-3870-2bc59c9dbed2@gmail.com> Message-ID: <7e27bfdf-0e54-904f-6e0d-d5d6f69798f2@gmail.com> Date: Fri, 2 Jul 2021 23:14:00 +0300 User-Agent: Mozilla/5.0 (X11; Linux aarch64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <9e231ca9-4412-f324-3870-2bc59c9dbed2@gmail.com> Content-Type: text/plain; charset=windows-1252 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.2 at phobos.denx.de X-Virus-Status: Clean On 02/07/2021 23:03, Alper Nebi Yasak wrote: > On 02/07/2021 22:01, Tom Rini wrote: >> Hey all, >> >> I started taking a look at moving to guestfish to see if this resolves >> the latest problem I've run in to: >> https://source.denx.de/u-boot/u-boot/-/jobs/284763#L307 >> which I think is due to guestmount not being done in time for the test. > > That failing test's setup uses virt-make-fs, different from what you're > changing below. I locally only see that failure for the clang build, and > it still fails after adding time.sleep(300) after its virt-make-fs > calls. I don't think it's an issue in the test setup. > >> So I started converting things to use guestfish directly: >> diff --git a/test/py/tests/test_fs/conftest.py b/test/py/tests/test_fs/conftest.py >> index 7325486cdb1a..e8899cfdd118 100644 >> --- a/test/py/tests/test_fs/conftest.py >> +++ b/test/py/tests/test_fs/conftest.py >> @@ -265,10 +265,10 @@ def fs_obj_basic(request, u_boot_config): >> fs_ubtype = fstype_to_ubname(fs_type) >> check_ubconfig(u_boot_config, fs_ubtype) >> >> - mount_dir = u_boot_config.persistent_data_dir + '/mnt' >> + data_dir = u_boot_config.persistent_data_dir + '/data' >> >> - small_file = mount_dir + '/' + SMALL_FILE >> - big_file = mount_dir + '/' + BIG_FILE >> + small_file = data_dir + '/' + SMALL_FILE >> + big_file = data_dir + '/' + BIG_FILE >> >> try: >> >> @@ -279,26 +279,14 @@ def fs_obj_basic(request, u_boot_config): >> return >> >> try: >> - check_call('mkdir -p %s' % mount_dir, shell=True) >> + check_call('mkdir -p %s' % data_dir, shell=True) >> except CalledProcessError as err: >> pytest.skip('Preparing mount folder failed for filesystem: ' + fs_type + '. {}'.format(err)) >> call('rm -f %s' % fs_img, shell=True) >> return >> >> try: >> - # Mount the image so we can populate it. >> - mount_fs(fs_type, fs_img, mount_dir) >> - except CalledProcessError as err: >> - pytest.skip('Mounting to folder failed for filesystem: ' + fs_type + '. {}'.format(err)) >> - call('rmdir %s' % mount_dir, shell=True) >> - call('rm -f %s' % fs_img, shell=True) >> - return >> - >> - try: >> - # Create a subdirectory. >> - check_call('mkdir %s/SUBDIR' % mount_dir, shell=True) >> - >> - # Create big file in this image. >> + # Create big file to copy in to the image. >> # Note that we work only on the start 1MB, couple MBs in the 2GB range >> # and the last 1 MB of the huge 2.5GB file. >> # So, just put random values only in those areas. >> @@ -309,10 +297,14 @@ def fs_obj_basic(request, u_boot_config): >> check_call('dd if=/dev/urandom of=%s bs=1M count=1 seek=2499' >> % big_file, shell=True) >> >> - # Create a small file in this image. >> + # Create a small file to copy in to the image. >> check_call('dd if=/dev/urandom of=%s bs=1M count=1' >> % small_file, shell=True) >> >> + # Copy the files in to the image and add a subdirectory. >> + # Create a subdirectory. >> + check_call('guestfish add %s : run : mount /dev/sda / : mkdir /SUBDIR : copy-in %s %s /' >> + % (fs_img, big_file, small_file), shell=True) > > It could be faster to do things within guestfish as much as possible, > instead of preparing the files outside and copying them in. > > Also it looks like python bindings are available as python3-guestfs on > Debian and Ubuntu, just not on pypi.org. > >> # Delete the small file copies which possibly are written as part of a >> # previous test. >> # check_call('rm -f "%s.w"' % MB1, shell=True) >> @@ -357,13 +349,11 @@ def fs_obj_basic(request, u_boot_config): >> >> except CalledProcessError as err: >> pytest.skip('Setup failed for filesystem: ' + fs_type + '. {}'.format(err)) >> - umount_fs(mount_dir) >> return >> else: >> - umount_fs(mount_dir) >> yield [fs_ubtype, fs_img, md5val] >> finally: >> - call('rmdir %s' % mount_dir, shell=True) >> + call('rmdir %s' % data_dir, shell=True) >> call('rm -f %s' % fs_img, shell=True) >> >> # >> >> The problem here is that a test run went from taking about 5 minutes to >> taking about 17 minutes. I can reduce this to closer to 15 minutes with >> LIBGUESTFS_BACKEND=direct and using libguestfs-make-fixed-appliance to >> make an appliance we reuse. But that's still too long to be usable. >> I'm hoping someone has some ideas here on how to improve things. > > If libguestfs is falling back to slow emulation because /dev/kvm isn't > available, maybe it's appropriate to check for that and skip the fs tests... Also the User-Mode Linux backend might be worth exploring.