Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 469AFC47082 for ; Sun, 30 May 2021 00:10:21 +0000 (UTC) Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E9EDB610FA for ; Sun, 30 May 2021 00:10:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E9EDB610FA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id A717182B5B; Sun, 30 May 2021 02:10:10 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Cabq85Lx"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id B063082B1D; Sun, 30 May 2021 02:09:59 +0200 (CEST) Received: from mail-qt1-x82d.google.com (mail-qt1-x82d.google.com [IPv6:2607:f8b0:4864:20::82d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id B9CC682B1D for ; Sun, 30 May 2021 02:09:54 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=jmcosta944@gmail.com Received: by mail-qt1-x82d.google.com with SMTP id h24so5485043qtm.12 for ; Sat, 29 May 2021 17:09:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pjbvxroSt3dRgJ2sfLyBKGmWoQtW6S/LZ3emNy2zaMM=; b=Cabq85LxS9ydr7L2JYqNxblvs1heNmtH/S9aBaJ6Ja8RsESZ6fcIQD6Qfy49S9e8WF Q8o6dSVi5ug3cLG5DITJ2hH0mv1ZaD5GMV/vmH+foNsBd0kkNyz+O4XHGwG1q24GJmh+ xoTENywiMzT0lW4ilSk80A2ZwayEIza6ZVghKNJbvPlpe7579UP5DJfoz2xzuCkE31yH CzeViXkcg2qXu01TU5CbC80w2QamYYRXjEB9n4EQ/WntoHa495PqQGyPy8vDeDU0EDlJ /QW2ydesFpPm110RibOD4O8LRIsldD29pM3gqzcdZbsbT1r98RE8aRUcaA3W1BxlvG9h 78TA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pjbvxroSt3dRgJ2sfLyBKGmWoQtW6S/LZ3emNy2zaMM=; b=WCwrnOgxEOkM36z0fsykUJ397Hd2rLeWYr2h4XWYSBtZhR5B26inUdCA6pZ0M8u5JG tPtfmU1unSnBKJr5sobaUfA1+SaxYlvsWYJOPcuao9SkvcSpNE804B7WxEzoOQsL/J74 KOrr7vziM1NB8k5xteNjApPWNUqaRj1hE8Lw3wraoa0SdB9WXkU5RYDMx7yWIumBsn8/ k9NMJLfGiZVMndtqHgn/YrzaDw9PF24ldTd2L/MLky/r04AwbPapXMMpRF4cFnuG84f8 Dxm/kMkfS4FO4v5hp40DL0S0Wef/fqOSMSnFORWQNGykPlPUMfCccEGv//oXJ5tSFmfT 9khA== X-Gm-Message-State: AOAM530x9Og9dzfVCESXwaKNOq0S8w6ke5lk1lIXDoEx7G2nvVh6iE6Y zRDVO2o8B01oTSTM4QyiwDeZg7INjzMpZP1y X-Google-Smtp-Source: ABdhPJzzNiDDEiT7IGVc9b4HZ0NzhujKTIrLnzjyALkn63dZo2olKkiT2PaGRWfroPeFn7+DKmsmDA== X-Received: by 2002:ac8:65d4:: with SMTP id t20mr9333513qto.15.1622333393228; Sat, 29 May 2021 17:09:53 -0700 (PDT) Received: from localhost.localdomain ([177.194.32.83]) by smtp.googlemail.com with ESMTPSA id u22sm1514300qtw.18.2021.05.29.17.09.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 May 2021 17:09:52 -0700 (PDT) From: Joao Marcos Costa To: u-boot@lists.denx.de Cc: thomas.petazzoni@bootlin.com, miquel.raynal@bootlin.com, richard.genoud@posteo.net, sjg@chromium.org, Joao Marcos Costa Subject: [PATCH v2 1/3] test/py: rewrite common tools for SquashFS tests Date: Sat, 29 May 2021 21:09:36 -0300 Message-Id: <20210530000938.48873-2-jmcosta944@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210530000938.48873-1-jmcosta944@gmail.com> References: <20210530000938.48873-1-jmcosta944@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.4 at phobos.denx.de X-Virus-Status: Clean Remove the previous OOP approach, which was confusing and incomplete. Add more test cases by making SquashFS images with various options, concerning file fragmentation and its compression. Add comments to properly document the code. Signed-off-by: Joao Marcos Costa --- .../test_fs/test_squashfs/sqfs_common.py | 237 +++++++++++++----- 1 file changed, 173 insertions(+), 64 deletions(-) diff --git a/test/py/tests/test_fs/test_squashfs/sqfs_common.py b/test/py/tests/test_fs/test_squashfs/sqfs_common.py index c96f92c1d8..5469b1c017 100644 --- a/test/py/tests/test_fs/test_squashfs/sqfs_common.py +++ b/test/py/tests/test_fs/test_squashfs/sqfs_common.py @@ -3,74 +3,183 @@ # Author: Joao Marcos Costa import os -import random -import string +import shutil import subprocess -def sqfs_get_random_letters(size): - letters = [] - for i in range(0, size): - letters.append(random.choice(string.ascii_letters)) +""" standard test images table: Each table item is a key:value pair +representing the output image name and its respective mksquashfs options. +This table should be modified only when adding support for new compression +algorithms. The 'default' case takes no options but the input and output +names, so it must be assigned with an empty string. +""" +STANDARD_TABLE = { + 'default' : '', + 'lzo_comp_frag' : '', + 'lzo_frag' : '', + 'lzo_no_frag' : '', + 'zstd_comp_frag' : '', + 'zstd_frag' : '', + 'zstd_no_frag' : '', + 'gzip_comp_frag' : '', + 'gzip_frag' : '', + 'gzip_no_frag' : '' +} - return ''.join(letters) +""" EXTRA_TABLE: Set this table's keys and values if you want to make squashfs +images with your own customized options. +""" +EXTRA_TABLE = {} -def sqfs_generate_file(path, size): - content = sqfs_get_random_letters(size) - file = open(path, "w") +# path to source directory used to make squashfs test images +SQFS_SRC_DIR = 'sqfs_src_dir' + +def get_opts_list(): + """ Combines fragmentation and compression options into a list of strings. + + opts_list's firts item is an empty string as STANDARD_TABLE's first item is + the 'default' case. + + Returns: + A list of strings whose items are formed by a compression and a + fragmentation option joined by a whitespace. + """ + # supported compression options only + comp_opts = ['-comp lzo', '-comp zstd', '-comp gzip'] + # file fragmentation options + frag_opts = ['-always-use-fragments', '-always-use-fragments -noF', '-no-fragments'] + + opts_list = [' '] + for comp_opt in comp_opts: + for frag_opt in frag_opts: + opts_list.append(' '.join([comp_opt, frag_opt])) + + return opts_list + +def init_standard_table(): + """ Initializes STANDARD_TABLE values. + + STANDARD_TABLE's keys are pre-defined, and init_standard_table() assigns + the right value for each one of them. + """ + opts_list = get_opts_list() + + for key, value in zip(STANDARD_TABLE.keys(), opts_list): + STANDARD_TABLE[key] = value + +def generate_file(file_name, file_size): + """ Generates a file filled with 'x'. + + Args: + file_name: the file's name. + file_size: the content's length and therefore the file size. + """ + content = 'x' * file_size + + file = open(file_name, 'w') file.write(content) file.close() -class Compression: - def __init__(self, name, files, sizes, block_size = 4096): - self.name = name - self.files = files - self.sizes = sizes - self.mksquashfs_opts = " -b " + str(block_size) + " -comp " + self.name - - def add_opt(self, opt): - self.mksquashfs_opts += " " + opt - - def gen_image(self, build_dir): - src = os.path.join(build_dir, "sqfs_src/") - os.mkdir(src) - for (f, s) in zip(self.files, self.sizes): - sqfs_generate_file(src + f, s) - - # the symbolic link always targets the first file - os.symlink(self.files[0], src + "sym") - - sqfs_img = os.path.join(build_dir, "sqfs-" + self.name) - i_o = src + " " + sqfs_img - opts = self.mksquashfs_opts - try: - subprocess.run(["mksquashfs " + i_o + opts], shell = True, check = True) - except: - print("mksquashfs error. Compression type: " + self.name) - raise RuntimeError - - def clean_source(self, build_dir): - src = os.path.join(build_dir, "sqfs_src/") - for f in self.files: - os.remove(src + f) - os.remove(src + "sym") - os.rmdir(src) - - def cleanup(self, build_dir): - self.clean_source(build_dir) - sqfs_img = os.path.join(build_dir, "sqfs-" + self.name) - os.remove(sqfs_img) - -files = ["blks_only", "blks_frag", "frag_only"] -sizes = [4096, 5100, 100] -gzip = Compression("gzip", files, sizes) -zstd = Compression("zstd", files, sizes) -lzo = Compression("lzo", files, sizes) - -# use fragment blocks for files larger than block_size -gzip.add_opt("-always-use-fragments") -zstd.add_opt("-always-use-fragments") - -# avoid fragments if lzo is used -lzo.add_opt("-no-fragments") - -comp_opts = [gzip, zstd, lzo] +def generate_sqfs_src_dir(build_dir): + """ Generates the source directory used to make the SquashFS images. + + The source directory is generated at build_dir, and it has the following + structure: + sqfs_src_dir/ + ├── empty-dir/ + ├── f1000 + ├── f4096 + ├── f5096 + ├── subdir/ + │   └── subdir-file + └── sym -> subdir + + 3 directories, 4 files + + The files in the root dir. are prefixed with an 'f' followed by its size. + + Args: + build_dir: u-boot's build-sandbox directory. + """ + + root = os.path.join(build_dir, SQFS_SRC_DIR) + # make root directory + os.makedirs(root) + + # 4096: minimum block size + file_name = 'f4096' + generate_file(os.path.join(root, file_name), 4096) + + # 5096: minimum block size + 1000 chars (fragment) + file_name = 'f5096' + generate_file(os.path.join(root, file_name), 5096) + + # 1000: less than minimum block size (fragment only) + file_name = 'f1000' + generate_file(os.path.join(root, file_name), 1000) + + # sub-directory with a single file inside + subdir_path = os.path.join(root, 'subdir') + os.makedirs(subdir_path) + generate_file(os.path.join(subdir_path, 'subdir-file'), 100) + + # symlink (target: sub-directory) + os.symlink('subdir', os.path.join(root, 'sym')) + + # empty directory + os.makedirs(os.path.join(root, 'empty-dir')) + +def mksquashfs(args): + """ Runs mksquashfs command. + + Args: + args: mksquashfs options (e.g.: compression and fragmentation). + """ + subprocess.run(['mksquashfs ' + args], shell=True, check=True, + stdout=subprocess.DEVNULL) + +def make_all_images(build_dir): + """ Makes the SquashFS images used in the test suite. + + The image names and respective mksquashfs options are defined in STANDARD_TABLE + and EXTRA_TABLE. The destination is defined by 'build_dir'. + + Args: + build_dir: u-boot's build-sandbox directory. + """ + + init_standard_table() + input_path = os.path.join(build_dir, SQFS_SRC_DIR) + + # make squashfs images according to STANDARD_TABLE + for out, opts in zip(STANDARD_TABLE.keys(), STANDARD_TABLE.values()): + output_path = os.path.join(build_dir, out) + mksquashfs(' '.join([input_path, output_path, opts])) + + # make squashfs images according to EXTRA_TABLE + for out, opts in zip(EXTRA_TABLE.keys(), EXTRA_TABLE.values()): + output_path = os.path.join(build_dir, out) + mksquashfs(' '.join([input_path, output_path, opts])) + +def clean_all_images(build_dir): + """ Deletes the SquashFS images at build_dir. + + Args: + build_dir: u-boot's build-sandbox directory. + """ + + for image_name in STANDARD_TABLE: + image_path = os.path.join(build_dir, image_name) + os.remove(image_path) + + for image_name in EXTRA_TABLE: + image_path = os.path.join(build_dir, image_name) + os.remove(image_path) + +def clean_sqfs_src_dir(build_dir): + """ Deletes the source directory at build_dir. + + Args: + build_dir: u-boot's build-sandbox directory. + """ + path = os.path.join(build_dir, SQFS_SRC_DIR) + shutil.rmtree(path) -- 2.25.1