Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F0F2DC43334 for ; Tue, 7 Jun 2022 17:45:33 +0000 (UTC) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 48EC0842DF; Tue, 7 Jun 2022 19:45:32 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="VJ0JLoAk"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 90269842FC; Tue, 7 Jun 2022 19:45:30 +0200 (CEST) Received: from mail-ej1-x62b.google.com (mail-ej1-x62b.google.com [IPv6:2a00:1450:4864:20::62b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 9B2E3842C4 for ; Tue, 7 Jun 2022 19:45:27 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=alpernebiyasak@gmail.com Received: by mail-ej1-x62b.google.com with SMTP id h23so25623876ejj.12 for ; Tue, 07 Jun 2022 10:45:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:date:mime-version:user-agent:subject:content-language:to :cc:references:from:in-reply-to:content-transfer-encoding; bh=YS9aDmqFqIOa2x8ly+m5uq0BJD9JBQR73OSmYom8zjw=; b=VJ0JLoAkLX0HorCv6aDcMtBGES5cv9iKCch/nwPDANOF2511J0OsLq4QxBPneVEs/m 3Xl4ENWZwn4DdmhW84ZnXXL8JQ/rAxFTaVetBBNr6I7mVsF0Ik2Swsa1YMcz/a/23IgK x8Kebg33SVNXRIIkLWv4fwr4jTvAwfQVYKQoFvE/cjgNJR4VNoz6luh1kwgqB0tJ7sng v17PIzI1bx7JkF/TZBiJr5u2y3L/8wtE/AdYfmDE6kfmXbH98TvgyQ3icCZgnwhAGJ0H kYFpZ5zhuLVBL8sP0WR6AdYVr6veesc0ehfSteDAbWqdT0+yzM0c+lC1XSceCI3jRt5h 1RcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:subject :content-language:to:cc:references:from:in-reply-to :content-transfer-encoding; bh=YS9aDmqFqIOa2x8ly+m5uq0BJD9JBQR73OSmYom8zjw=; b=R0/tFmSrxwYFIdAR6Zl7nC3crZRLDdiFQ1g+wKJOpf41nmIsvq+p33wezQYYeAKlZ/ bAMgzS4XV5I3dRVsVK2sDB+9DezMHctr/1fI1pB1mDLFj8ANnjRf8EL5oTVuQ3nCeqB8 s7+86PoVI0+5ARH9IpdB/ygCPBLr0u0Swp9fYQKMEp0r+tHHFWwh4buVqKM5/t68Xjo6 qe9EXbdRTl35U5nEjUwCUclrqyYAM2te9V5ka8dy2G1ts/4GG2dU2CjbAVTSHp9gelq0 Pf9f7Nsn78LZT65tGdCU8GCJnk/O+3+2pmAv27bBGG/yW/dHyLod9fUNopFpef7C+ymu sofg== X-Gm-Message-State: AOAM532qncUFzsCTRnWJ8ew0lYbemIfZEcgPY9pIze2kUKqHaGqksBes 9E27/nL3FnmwAKKnCvJ04Y8= X-Google-Smtp-Source: ABdhPJxCfw7LXjsGNXfmZL3W6Uwhjfo7EMP202oiWh65ttxf4BIEaMaS7wMFO0BLSkbFd2jA8SdCPA== X-Received: by 2002:a17:907:1b1e:b0:6d7:31b0:e821 with SMTP id mp30-20020a1709071b1e00b006d731b0e821mr27595161ejc.334.1654623927143; Tue, 07 Jun 2022 10:45:27 -0700 (PDT) Received: from [192.168.0.74] ([178.233.178.185]) by smtp.gmail.com with ESMTPSA id t26-20020a17090605da00b006fe7d269db8sm7923815ejt.104.2022.06.07.10.45.25 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 07 Jun 2022 10:45:26 -0700 (PDT) Message-ID: <16e28d22-c682-00d6-6e0e-2f198fbff067@gmail.com> Date: Tue, 7 Jun 2022 20:45:23 +0300 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux aarch64; rv:91.0) Gecko/20100101 Thunderbird/91.10.0 Subject: Re: binman: Why is the first word the compressed size in compressed data Content-Language: en-US To: Stefan Herbrechtsmeier Cc: Michal Simek , U-Boot Mailing List , Simon Glass References: <77ffc9ce-b781-8930-8556-fa3d8fb9a834@weidmueller.com> From: Alper Nebi Yasak In-Reply-To: <77ffc9ce-b781-8930-8556-fa3d8fb9a834@weidmueller.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.5 at phobos.denx.de X-Virus-Status: Clean On 03/06/2022 18:07, Stefan Herbrechtsmeier wrote: > Am 02.06.2022 um 20:10 schrieb Alper Nebi Yasak: >> On 01/06/2022 11:29, Stefan Herbrechtsmeier wrote: >>> Hi Simon, >>> >>> I want to compress a FPGA Image on the fly via binman but this doesn't >>> work. I have add a bintool implementation for gzip, add gzip support to >>> comp_util.py and set `compress` and `compression` property in the binman >>> node of the u-boot dtsi: >>> >>> fpga-2cg { >>> compatible = "u-boot,fpga-legacy"; >>> description = "FPGA"; >>> type = "fpga"; >>> compression = "gzip"; >>> load = <(CONFIG_SYS_TEXT_BASE + 0x4000000)>; >>> >>> blob-ext { >>> compress = "gzip"; >>> filename = "2cg.bit.bin"; >>> }; >>> }; >>> >>> It works if I remove the `compress` property and use a `2cg.bit.bin.gz` >>> or remove the header for gzip [1]. >>> >>> Regarding the code binman add the compressed size as first word to the >>> data [2]. The code in spl-fit.c doesn't remove the size [3]. Why is this >>> size added? >> >> I looked into this a tiny bit, git blame eventually ends up at: >> >>> commit eb0f4a4cb40264a90a91e10e3ec00d1e0da7fb66 >>> Author: Simon Glass >>> Date: 2019-07-20 12:24:06 -0600 >>> >>> binman: Support replacing data in a cbfs >>> >>> At present binman cannot replace data within a CBFS since it does not >>> allow rewriting of the files in that CBFS. Implement this by using the >>> new WriteData() method to handle the case. >>> >>> Add a header to compressed data so that the amount of compressed data can >>> be determined without reference to the size of the containing entry. This >>> allows the entry to be larger that the contents, without causing errors in >>> decompression. This is necessary to cope with a compressed device tree >>> being updated in such a way that it shrinks after the entry size is >>> already set (an obscure case). It is not used with CBFS since it has its >>> own metadata for this. Increase the number of passes allowed to resolve >>> the position of entries, to handle this case. >>> >>> Add a test for this new logic. >>> >>> Signed-off-by: Simon Glass > > Thanks. It looks like the header was added for internal compress and > decompress. It shouldn't be part of the blob. > >> For the record, I dislike the size header, since as you found out it >> turns the standard formats into custom formats that nothing else knows. > > Maybe I overlook something in the code but I don't find a support for > the header inside the bootm or spl_fit code. > >> But I'm not familiar with binman's compression parts, I don't know what >> would go wrong if the header is removed and how. I'm also not sure what >> happens in the 'obscure case' the commit message mentions. > > The header should be removed or the fit image should point to the offset > behind the header. The compression test passed only because it supports > the header. I agree that the header should be removed completely. I'll try to do it later on, but I don't know when I'll be able to.