Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7425C4332F for ; Wed, 27 Oct 2021 11:31:27 +0000 (UTC) Subject: Re: [PATCH V2] archiver: Configurable tarball compression To: openembedded-core@lists.openembedded.org From: "Martyn Welch" X-Originating-Location: Crieff, Scotland, GB (86.153.39.205) X-Originating-Platform: Linux Chrome 95 User-Agent: GROUPS.IO Web Poster MIME-Version: 1.0 Date: Wed, 27 Oct 2021 04:31:26 -0700 References: In-Reply-To: Message-ID: <31960.1635334286957262862@lists.openembedded.org> Content-Type: multipart/alternative; boundary="raLNUh6ovUQ1MGIhy5W0" List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 27 Oct 2021 11:31:27 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/157511 --raLNUh6ovUQ1MGIhy5W0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable >=20 > So, here the claim that zstd (with default options) is almost as good as > xz in compressed size is not confirmed. However, zstd is a clear winner > in terms of compression speed, and anyway better than gzip. This is > worth switching. >=20 That claim doesn't seem to be confirmed with any of the (admittedly small) = selection of archives I tried, with zstd compression being approx 21 to 69%= less efficient in terms of storage space than xz compression, but still be= ing the best in terms of compression and decompression speeds. However, I think neatly highlights why it may make sense to make this confi= gurable, as which algorithm is "best" is going to depend on whether you're = optimising for (de)compression speed or size. Testing results below, Martyn --- $ time gzip -k linux-5.14.tar real 0m26.807s user 0m26.392s sys 0m0.368s $ time xz -k linux-5.14.tar real 6m42.494s user 6m40.167s sys 0m1.757s $ time zstd -k linux-5.14.tar linux-5.14.tar=C2=A0 =C2=A0 =C2=A0 =C2=A0: 16.28%=C2=A0 =C2=A0(1126737920 = =3D> 183398470 bytes, linux-5.14.tar.zst) real 0m3.531s user 0m3.631s sys 0m0.509s $ ls -la=C2=A0 * -rw-r--r-- 1 martyn martyn 1126737920 Oct 27 10:54 linux-5.14.tar -rw-r--r-- 1 martyn martyn=C2=A0 196107916 Oct 27 10:54 linux-5.14.tar.gz -rw-r--r-- 1 martyn martyn=C2=A0 124724612 Oct 27 10:54 linux-5.14.tar.xz -rw-r--r-- 1 martyn martyn=C2=A0 183398470 Oct 27 10:54 linux-5.14.tar.zst $ time gunzip linux-5.14.tar.gz real 0m5.141s user 0m4.462s sys 0m0.613s $ time xz -d linux-5.14.tar.xz real 0m8.571s user 0m7.739s sys 0m0.820s $ time zstd -d linux-5.14.tar.zst linux-5.14.tar.zst=C2=A0 : 1126737920 bytes real 0m1.906s user 0m1.185s sys 0m0.710s $ time gzip -k coreutils-9.0.tar real 0m1.685s user 0m1.669s sys 0m0.016s $ time xz -k coreutils-9.0.tar real 0m14.891s user 0m14.795s sys 0m0.060s $ time zstd -k coreutils-9.0.tar coreutils-9.0.tar=C2=A0 =C2=A0 : 19.21%=C2=A0 =C2=A0(54394880 =3D> 10447053= bytes, coreutils-9.0.tar.zst) real 0m0.207s user 0m0.215s sys 0m0.029s $ ls -la coreutils-9.0.tar* -rw-r--r-- 1 martyn martyn 54394880 Oct 27 11:16 coreutils-9.0.tar -rw-r--r-- 1 martyn martyn 13595007 Oct 27 11:16 coreutils-9.0.tar.gz -rw-r--r-- 1 martyn martyn=C2=A0 6177372 Oct 27 11:16 coreutils-9.0.tar.xz -rw-r--r-- 1 martyn martyn 10447053 Oct 27 11:16 coreutils-9.0.tar.zst $ time gzip -d coreutils-9.0.tar.gz real 0m0.362s user 0m0.280s sys 0m0.048s $ time xz -d coreutils-9.0.tar.xz real 0m0.444s user 0m0.424s sys 0m0.020s $ time zstd -d coreutils-9.0.tar.zst coreutils-9.0.tar.zst: 54394880 bytes real 0m0.095s user 0m0.044s sys 0m0.052s $ time gzip -k tcp_wrappers_7.6.tar real 0m0.033s user 0m0.033s sys 0m0.000s $ time xz -k tcp_wrappers_7.6.tar real 0m0.116s user 0m0.104s sys 0m0.012s $ time zstd -k tcp_wrappers_7.6.tar tcp_wrappers_7.6.tar : 26.57%=C2=A0 =C2=A0(360448 =3D>=C2=A0 95772 bytes, t= cp_wrappers_7.6.tar.zst) real 0m0.006s user 0m0.003s sys 0m0.003s $ ls -la tcp_wrappers_7.6.tar* -rw-r--r-- 1 martyn martyn 360448 Oct 27 11:15 tcp_wrappers_7.6.tar -rw-r--r-- 1 martyn martyn=C2=A0 99459 Oct 27 11:15 tcp_wrappers_7.6.tar.gz -rw-r--r-- 1 martyn martyn=C2=A0 79316 Oct 27 11:15 tcp_wrappers_7.6.tar.xz -rw-r--r-- 1 martyn martyn=C2=A0 95772 Oct 27 11:15 tcp_wrappers_7.6.tar.zs= t $ time gzip -d tcp_wrappers_7.6.tar.gz real 0m0.008s user 0m0.004s sys 0m0.004s $ time xz -d tcp_wrappers_7.6.tar.xz real 0m0.019s user 0m0.015s sys 0m0.004s $ time zstd -d tcp_wrappers_7.6.tar.zst tcp_wrappers_7.6.tar.zst: 360448 bytes real 0m0.005s user 0m0.000s sys 0m0.005s --raLNUh6ovUQ1MGIhy5W0 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable
So, here the claim that = zstd (with default options) is almost as good as
xz in compressed = size is not confirmed. However, zstd is a clear winner
in terms of= compression speed, and anyway better than gzip. This is
worth swi= tching.
That claim doesn't seem to be confirmed with any of the (admittedly small) = selection of archives I tried, with zstd compression being approx 21 to 69%= less efficient in terms of storage space than xz compression, but still be= ing the best in terms of compression and decompression speeds.

H= owever, I think neatly highlights why it may make sense to make this config= urable, as which algorithm is "best" is going to depend on whether you're o= ptimising for (de)compression speed or size.

Testing results bel= ow,

Martyn

---

$ time gzip -k linux-5.14.tar 
 
real 0m26.807s
user 0m26.392s
sys 0m0.368s
$ time xz -k linux-5.14.tar 
 
real 6m42.494s
user 6m40.167s
sys 0m1.757s
$ time zstd -k linux-5.14.tar 
linux-5.14.tar       : 16.28%   (1126737= 920 =3D> 183398470 bytes, linux-5.14.tar.zst) 
 
real 0m3.531s
user 0m3.631s
sys 0m0.509s
$ ls -la  *
-rw-r--r-- 1 martyn martyn 1126737920 Oct 27 10:54 linux-5.14.tar
-rw-r--r-- 1 martyn martyn  196107916 Oct 27 10:54 linux-5.14.tar= .gz
-rw-r--r-- 1 martyn martyn  124724612 Oct 27 10:54 linux-5.14.tar= .xz
-rw-r--r-- 1 martyn martyn  183398470 Oct 27 10:54 linux-5.14.tar= .zst
$ time gunzip linux-5.14.tar.gz 
 
real 0m5.141s
user 0m4.462s
sys 0m0.613s
$ time xz -d linux-5.14.tar.xz 
 
real 0m8.571s
user 0m7.739s
sys 0m0.820s
$ time zstd -d linux-5.14.tar.zst 
linux-5.14.tar.zst  : 1126737920 bytes       =                     &nbs= p;            
 
real 0m1.906s
user 0m1.185s
sys 0m0.710s
 
 
$ time gzip -k coreutils-9.0.tar
 
real 0m1.685s
user 0m1.669s
sys 0m0.016s
$ time xz -k coreutils-9.0.tar
 
real 0m14.891s
user 0m14.795s
sys 0m0.060s
$ time zstd -k coreutils-9.0.tar
coreutils-9.0.tar    : 19.21%   (54394880 =3D> = 10447053 bytes, coreutils-9.0.tar.zst) 
 
real 0m0.207s
user 0m0.215s
sys 0m0.029s
$ ls -la coreutils-9.0.tar*
-rw-r--r-- 1 martyn martyn 54394880 Oct 27 11:16 coreutils-9.0.tar
-rw-r--r-- 1 martyn martyn 13595007 Oct 27 11:16 coreutils-9.0.tar.gz<= /div>
-rw-r--r-- 1 martyn martyn  6177372 Oct 27 11:16 coreutils-9.0.ta= r.xz
-rw-r--r-- 1 martyn martyn 10447053 Oct 27 11:16 coreutils-9.0.tar.zst=
$ time gzip -d coreutils-9.0.tar.gz 
 
real 0m0.362s
user 0m0.280s
sys 0m0.048s
$ time xz -d coreutils-9.0.tar.xz
 
real 0m0.444s
user 0m0.424s
sys 0m0.020s
$ time zstd -d coreutils-9.0.tar.zst
coreutils-9.0.tar.zst: 54394880 bytes         = ;                     &nb= sp;          
 
real 0m0.095s
user 0m0.044s
sys 0m0.052s
 
 
$ time gzip -k tcp_wrappers_7.6.tar
 
real 0m0.033s
user 0m0.033s
sys 0m0.000s
$ time xz -k tcp_wrappers_7.6.tar
 
real 0m0.116s
user 0m0.104s
sys 0m0.012s
$ time zstd -k tcp_wrappers_7.6.tar
tcp_wrappers_7.6.tar : 26.57%   (360448 =3D>  95772 = bytes, tcp_wrappers_7.6.tar.zst) 
 
real 0m0.006s
user 0m0.003s
sys 0m0.003s
$ ls -la tcp_wrappers_7.6.tar*
-rw-r--r-- 1 martyn martyn 360448 Oct 27 11:15 tcp_wrappers_7.6.tar
-rw-r--r-- 1 martyn martyn  99459 Oct 27 11:15 tcp_wrappers_7.6.t= ar.gz
-rw-r--r-- 1 martyn martyn  79316 Oct 27 11:15 tcp_wrappers_7.6.t= ar.xz
-rw-r--r-- 1 martyn martyn  95772 Oct 27 11:15 tcp_wrappers_7.6.t= ar.zst
$ time gzip -d tcp_wrappers_7.6.tar.gz 
 
real 0m0.008s
user 0m0.004s
sys 0m0.004s
$ time xz -d tcp_wrappers_7.6.tar.xz
 
real 0m0.019s
user 0m0.015s
sys 0m0.004s
$ time zstd -d tcp_wrappers_7.6.tar.zst
tcp_wrappers_7.6.tar.zst: 360448 bytes        &nbs= p;                     &n= bsp;         
 
real 0m0.005s
user 0m0.000s
sys 0m0.005s
--raLNUh6ovUQ1MGIhy5W0--