Smaller compressed kernel source tarballs?
Today as I was watching the linux-2.6.18.tar.bz2 slowly download I thought it would be nice if it could be made smaller.
The 7zip program/algorithm is free software (LGPL) and can be obtained from http://www.7-zip.org/ and it is distributed with several distributions (it is in Fedora Core 6 extras for example).
Here are the numbers:
ls -al
-rw-r--r-- 1 root root 240138240 Sep 21 13:55 linux-2.6.18.tar -rw-r--r-- 1 root root 34180796 Sep 21 13:42 linux-2.6.18.tar.7z -rw-r--r-- 1 root root 41863580 Sep 21 13:45 linux-2.6.18.tar.bz2 -rw-r--r-- 1 root root 52467357 Sep 21 13:13 linux-2.6.18.tar.gz
ls -alh
-rw-r--r-- 1 root root 230M Sep 21 13:55 linux-2.6.18.tar -rw-r--r-- 1 root root 33M Sep 21 13:42 linux-2.6.18.tar.7z -rw-r--r-- 1 root root 40M Sep 21 13:45 linux-2.6.18.tar.bz2 -rw-r--r-- 1 root root 51M Sep 21 13:13 linux-2.6.18.tar.gz
Smaller the better, especially with the international audience.
Dax Kelson
Quoted 21 lines · Dax Kelson
Today as I was watching the linux-2.6.18.tar.bz2 slowly download I thought it would be nice if it could be made smaller.
The 7zip program/algorithm is free software (LGPL) and can be obtained from http://www.7-zip.org/ and it is distributed with several distributions (it is in Fedora Core 6 extras for example).
Here are the numbers:
ls -al
(quoted diff) +0
-4-rw-r--r-- 1 root root 240138240 Sep 21 13:55 linux-2.6.18.tar -rw-r--r-- 1 root root 34180796 Sep 21 13:42 linux-2.6.18.tar.7z -rw-r--r-- 1 root root 41863580 Sep 21 13:45 linux-2.6.18.tar.bz2 -rw-r--r-- 1 root root 52467357 Sep 21 13:13 linux-2.6.18.tar.gzls -alh
(quoted diff) +0
-4-rw-r--r-- 1 root root 230M Sep 21 13:55 linux-2.6.18.tar -rw-r--r-- 1 root root 33M Sep 21 13:42 linux-2.6.18.tar.7z -rw-r--r-- 1 root root 40M Sep 21 13:45 linux-2.6.18.tar.bz2 -rw-r--r-- 1 root root 51M Sep 21 13:13 linux-2.6.18.tar.gz
Quoted 1 line
Smaller the better, especially with the international audience.
But after you download it once, you can just get the diff next time. How is the decompression time on 7zip versus bzip2 and gzip?
--
Len Sorensen
Quoted 3 lines · On Thu, 21 Sep 2006 16:42:50 -0400 Lennart Sorensen…
Quoted 2 lines · Dax Kelson
Today as I was watching the linux-2.6.18.tar.bz2 slowly download I thought it would be nice if it could be made smaller.
[...]
Quoted 2 lines
But after you download it once, you can just get the diff next time. How is the decompression time on 7zip versus bzip2 and gzip?
Not to mention that by using Git it will take care of all that for you. Downloading only the updates with no need for you to manually apply diffs etc..
Sean
Quoted 2 lines · Lennart Sorensen
But after you download it once, you can just get the diff next time. How is the decompression time on 7zip versus bzip2 and gzip?
Decompression times on 2.6.18 are as follows:
gzip: 0m3.509s 7zip: 0m10.012s bzip2: 0m22.703s
Dax Kelson
Quoted 5 lines · Dax Kelson
Decompression times on 2.6.18 are as follows:
gzip: 0m3.509s 7zip: 0m10.012s bzip2: 0m22.703s
Hmm, not bad.
--
Len Sorensen
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 5 lines · Sean
Not to mention that by using Git it will take care of all that for you. Downloading only the updates with no need for you to manually apply diffs etc..
Sean
Git users and tarball users are different audiences.
Dax Kelson
Quoted 1 line · Dax Kelson
Git users and tarball users are different audiences.
Try ketchup then. http://www.selenic.com/ketchup/wiki/
Quoted 3 lines · On Thu, 21 Sep 2006 15:41:15 -0600 Dax Kelson <[email protected]> wrote:
Git users and tarball users are different audiences.
Don't see why that needs to be the case. Git can even produce the tarballs once you've synced up with kernel.org (see git-tar-tree). People interested in conserving bandwidth should really consider the use of Git.
Sean
Quoted 11 lines · Sean
Quoted 3 lines · On Thu, 21 Sep 2006 15:41:15 -0600 Dax Kelson <[email protected]> wrote:
Git users and tarball users are different audiences.
Don't see why that needs to be the case. Git can even produce the tarballs once you've synced up with kernel.org (see git-tar-tree). People interested in conserving bandwidth should really consider the use of Git.
yes,
however git users are people who plan on following every kernel version for a
while, tarball users are people who grab a copy of the kernel once in a while (probably not every version). for the tarball users they would have to grab multiple patches to get from the last thing that they have to whatever is current. and frankly they may not (and probably should not) trust the last thing that they have, as in many cases it's a distro patched kernel that may not be compatable with the vanilla kernel.
people who start downloading every revision should start useing git or patches, but not everyone needs it.
also people could be behind a firewall that prevents git from working properly, for them tarballs and patches are the right way of doing things.
David Lang
Quoted 3 lines · David Lang
for the tarball users they would have to grab
multiple patches to get from the last thing that they have to whatever is current.
ketchup solves that problem. One command brings any tree up to current.
Quoted 2 lines
also people could be behind a firewall that prevents git from working properly, for them tarballs and patches are the right way of doing things.
If they can't git through a firewall, they won't be able to wget a tarball through it either.
Dave
Quoted 7 lines · Dave Jones
Quoted 3 lines · David Lang
for the tarball users they would have to grab
multiple patches to get from the last thing that they have to whatever is current.ketchup solves that problem. One command brings any tree up to current.
so are you saying that ketchup should be used for _all_ access to the vanilla tree that isn't done via git?
if not then tarballs still have a place.
and how does ketchup deal with patched trees to start with?
Quoted 5 lines
Quoted 2 lines
also people could be behind a firewall that prevents git from working properly, for them tarballs and patches are the right way of doing things.
If they can't git through a firewall, they won't be able to wget a tarball through it either.
to work properly git should talk it's own protocol, http/ftp can be allowed (and authenticated) through firewalls that don't allow the git protocol.
David Lang
Quoted 2 lines
Dave
Quoted 8 lines · David Lang
Quoted 6 lines · Dave Jones
Quoted 3 lines · David Lang
for the tarball users they would have to grab
multiple patches to get from the last thing that they have to whatever is current.
Quoted 5 lines
Quoted 1 line
ketchup solves that problem. One command brings any tree up to current.
so are you saying that ketchup should be used for _all_ access to the vanilla tree that isn't done via git?
if not then tarballs still have a place.
I think you have a misunderstanding over what ketchup is/does.
It cannot usurp tarballs by its very nature. It retrieves tarballs (if necessary) and whatever patches are necessary to get to the tree you want.
http://www.selenic.com/ketchup/
Quoted 1 line
and how does ketchup deal with patched trees to start with?
By unpatching if necessary.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
also people could be behind a firewall that prevents git from working properly, for them tarballs and patches are the right way of doing things.
If they can't git through a firewall, they won't be able to wget a tarball through it either.
to work properly git should talk it's own protocol, http/ftp can be allowed (and authenticated) through firewalls that don't allow the git protocol.
'properly' is the wrong word here. optimally, yes, but the firewall argument alone isn't sufficient to claim git can't be used to clone a tree. A tree cloned over http: vs one over git: has exactly the same information in it. All the history, all the changes. Everything.
Dave
Quoted 12 lines · Dave Jones
Quoted 10 lines · David Lang
Quoted 7 lines · Dave Jones
Quoted 3 lines · David Lang
for the tarball users they would have to grab
multiple patches to get from the last thing that they have to whatever is current.ketchup solves that problem. One command brings any tree up to current.
Quoted 8 lines
Quoted 3 lines
so are you saying that ketchup should be used for _all_ access to the vanilla tree that isn't done via git?
if not then tarballs still have a place.I think you have a misunderstanding over what ketchup is/does.
It cannot usurp tarballs by its very nature. It retrieves tarballs (if necessary) and whatever patches are necessary to get to the tree you want.
http://www.selenic.com/ketchup/
in that case the compression of the tarballs is still worth dealing with
Quoted 3 lines
Quoted 1 line
and how does ketchup deal with patched trees to start with?
By unpatching if necessary.
assuming that it knows where to get the patches from, I was refering to things like the debian or redhat tree with their patches.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
also people could be behind a firewall that prevents git from working properly, for them tarballs and patches are the right way of doing things.
If they can't git through a firewall, they won't be able to wget a tarball through it either.
Quoted 7 lines
Quoted 2 lines
to work properly git should talk it's own protocol, http/ftp can be allowed (and authenticated) through firewalls that don't allow the git protocol.
'properly' is the wrong word here. optimally, yes, but the firewall argument alone isn't sufficient to claim git can't be used to clone a tree. A tree cloned over http: vs one over git: has exactly the same information in it. All the history, all the changes. Everything.
in most cases, but there are cases where the dumb transports can make mistakes (there have been several threads on the git list covering these), git is good enough to notice mos of them, but there is still room for problems. Also, installing and configuring git should not be a prerequesite to getting the kernel.
the point being git and ketchup do not eliminate the need to transfer tarballs, and therfor do not eliminate the attractivness of a compression that saves a significant amount of bandwidth.
I was responding to the (apparent) argument that with git and ketchup people should not ever be downloading tarballs, so something that cuts the size of a tarball in half doesn't make any difference.
David Lang
Quoted 3 lines · On Thu, 21 Sep 2006 15:34:53 -0700 (PDT) David Lang <[email protected]>…
I was responding to the (apparent) argument that with git and ketchup people should not ever be downloading tarballs, so something that cuts the size of a tarball in half doesn't make any difference.
Sure there are some cases where tarballs are more appropriate, but with git and maybe some of the other tools it should really be the minority situation. I wonder how many people just use tarballs out of inertia. All said though saving a few bytes of bandwidth by making the tarballs smaller can't hurt.
Sean
Quoted 11 lines · On Thu, 21 Sep 2006 15:00:48 -0700 (PDT) David Lang <[email protected]>…
yes,
however git users are people who plan on following every kernel version for a
while, tarball users are people who grab a copy of the kernel once in a while (probably not every version). for the tarball users they would have to grab multiple patches to get from the last thing that they have to whatever is current. and frankly they may not (and probably should not) trust the last thing that they have, as in many cases it's a distro patched kernel that may not be compatable with the vanilla kernel.people who start downloading every revision should start useing git or patches, but not everyone needs it.
Agreed, but for those people there isn't going to be much need (if any) to worry about if the tar ball is in .gzip or .bzip2 or whatever then either. And that was the case that inspired the suggestion.
Quoted 2 lines
also people could be behind a firewall that prevents git from working properly, for them tarballs and patches are the right way of doing things.
I use git from behind a firewall everyday without a problem. If you've seen such a problem yourself, a bug report would hopefully lead to a solution.
Thanks,
Sean
Quoted 5 lines · Sean
Quoted 2 lines
also people could be behind a firewall that prevents git from working properly, for them tarballs and patches are the right way of doing things.
I use git from behind a firewall everyday without a problem. If you've seen such a problem yourself, a bug report would hopefully lead to a solution.
it's not a bug, it's simply the fact that git (properly) uses it's own port for it's own protocol, and not all firewalls allow access to that port. in some cases even where a person would have the ability to get the firewall changed they may not want to for other (political) reasons.
even if git tunneled over HTTP there would be firewalls that would require authentication that git wouldn't be able to do and would therefor block the access.
David Lang
Quoted 4 lines · Lennart Sorensen
Quoted 3 lines · Dax Kelson
Today as I was watching the linux-2.6.18.tar.bz2 slowly download I thought it would be nice if it could be made smaller.
Quoted 8 lines
Quoted 4 lines
The 7zip program/algorithm is free software (LGPL) and can be obtained from http://www.7-zip.org/ and it is distributed with several distributions (it is in Fedora Core 6 extras for example).
But after you download it once, you can just get the diff next time. How is the decompression time on 7zip versus bzip2 and gzip?
7zip (LZMA) decompresses quickly, and the decompressor text is actually smaller than the equivalent for gzip. Quite nice.
What is not nice is the code for the compressor, which is a total mess.
I have been holding out on implementing LZMA on kernel.org, because
just as zip (deflate) didn't become common in the Unix world until an encapsulation format that handles things expected in the Unix world, e.g. streaming, was created (gzip), I don't think LZMA is going to be widely used until there is an "lzip" which does the same thing. I actually started the work of adding LZMA support to gzip, but then realized it would be better if a new encapsulation format with proper 64-bit support everywhere was created.
-hpa
Quoted 12 lines · H. Peter Anvin
7zip (LZMA) decompresses quickly, and the decompressor text is actually smaller than the equivalent for gzip. Quite nice.
What is not nice is the code for the compressor, which is a total mess.
I have been holding out on implementing LZMA on kernel.org, because
just as zip (deflate) didn't become common in the Unix world until an encapsulation format that handles things expected in the Unix world, e.g. streaming, was created (gzip), I don't think LZMA is going to be widely used until there is an "lzip" which does the same thing. I actually started the work of adding LZMA support to gzip, but then realized it would be better if a new encapsulation format with proper 64-bit support everywhere was created.
It doesn't handle streaming?
So you can't do: tar c dirname | 7zip dirname.tar.7z ?
--
Len Sorensen
RuggedCom
Quoted 11 lines · Lennart Sorensen
Quoted 5 lines · H. Peter Anvin
7zip (LZMA) decompresses quickly, and the decompressor text is actually smaller than the equivalent for gzip. Quite nice.
What is not nice is the code for the compressor, which is a total mess.
I have been holding out on implementing LZMA on kernel.org, because
just as zip (deflate) didn't become common in the Unix world until an encapsulation format that handles things expected in the Unix world, e.g. streaming, was created (gzip), I don't think LZMA is going to be widely used until there is an "lzip" which does the same thing. I actually started the work of adding LZMA support to gzip, but then realized it would be better if a new encapsulation format with proper 64-bit support everywhere was created.It doesn't handle streaming?
So you can't do: tar c dirname | 7zip dirname.tar.7z ?
Nope, and in particular you can't do:
tar cf - dirname | 7zip | ssh ...
This is because 7zip is an archiving format in its own right, much like zip. What we want is something that is to 7zip what gzip is to zip.
-hpa
Quoted 8 lines
Quoted 4 lines
widely used until there is an "lzip" which does the same thing. I actually started the work of adding LZMA support to gzip, but then realized it would be better if a new encapsulation format with proper 64-bit support everywhere was created.
It doesn't handle streaming?
So you can't do: tar c dirname | 7zip dirname.tar.7z ?
man 7z [slightly changed for reasonability]:
-si
Read data from StdIn (eg: tar -c directory | 7z a -si directory.tar.7z)
Jan Engelhardt
Quoted 10 lines · Jan Engelhardt
Quoted 4 lines
Quoted 1 line
widely used until there is an "lzip" which does the same thing. I actually started the work of adding LZMA support to gzip, but then realized it would be better if a new encapsulation format with proper 64-bit support everywhere was created.
It doesn't handle streaming?
So you can't do: tar c dirname | 7zip dirname.tar.7z ?
man 7z [slightly changed for reasonability]:
-si
Read data from StdIn (eg: tar -c directory | 7z a -si directory.tar.7z)
Yes, but you can't make it write to an unseekable stdout.
-hpa
Quoted 9 lines · H. Peter Anvin
Quoted 8 lines · Jan Engelhardt
Quoted 7 lines
Quoted 4 lines
widely used until there is an "lzip" which does the same thing. I actually started the work of adding LZMA support to gzip, but then realized it would be better if a new encapsulation format with proper 64-bit support everywhere was created.
It doesn't handle streaming?
So you can't do: tar c dirname | 7zip dirname.tar.7z ?
Quoted 8 lines
Quoted 6 lines
man 7z [slightly changed for reasonability]:
-si
Read data from StdIn (eg: tar -c directory | 7z a -si directory.tar.7z)Yes, but you can't make it write to an unseekable stdout.
It seems the "lzma" program from LZMA Utils can:
http://tukaani.org/lzma/
"Very similar command line interface than what gzip and bzip2 have."
(Debian sid has this in the "lzma" package.)
Johannes
Quoted 8 lines · Johannes Stezenbach
It seems the "lzma" program from LZMA Utils can:
http://tukaani.org/lzma/
"Very similar command line interface than what gzip and bzip2 have."(Debian sid has this in the "lzma" package.)
Yes, it can. If that's the way things go then I don't mind it, however, my biggest problem with lzma utils is that the command line parsing is done in a shell script wrapper.
Maybe I'll start using it anyway...
-hpa
Quoted 7 lines · H. Peter Anvin
Quoted 6 lines · Johannes Stezenbach
It seems the "lzma" program from LZMA Utils can:
http://tukaani.org/lzma/
"Very similar command line interface than what gzip and bzip2 have."
Quoted 6 lines
Quoted 2 lines
(Debian sid has this in the "lzma" package.)
Yes, it can. If that's the way things go then I don't mind it, however, my biggest problem with lzma utils is that the command line parsing is done in a shell script wrapper.
Well, I don't see any shell code here, in /usr/bin/lzma as in istalled from debian version 4.43-2.
But note that this lzma utility does not have any 'magic number' and does no crc checks. On the site it's said lzma(sdk) is under rewrite to support new format with magic number and crc checks...
After reading this thread I wanted to teach GNU tar to automatically recognize ..tar.lzma archives - and failed, eactly because of the lack of magic number at the start of a file...
/mjt
Quoted 6 lines · Michael Tokarev
Well, I don't see any shell code here, in
/usr/bin/lzmaas in istalled from debian version 4.43-2.But note that this lzma utility does not have any 'magic number' and does no crc checks.
Ah, right, that's a total killer.
Quoted 2 lines
On the site it's said lzma(sdk) is under rewrite to support new format with magic number and crc checks...
That is an absolute must, IMO. I would use the gzip format as a base.
-hpa
Quoted 6 lines · H. Peter Anvin
Quoted 3 lines · Michael Tokarev
[...]
On the site it's said lzma(sdk) is under rewrite to support new format with magic number and crc checks...That is an absolute must, IMO. I would use the gzip format as a base.
If you're suggesting a gzip like format (but with different magic, etc.), that's ok.
However, it has been suggested on similar threads to use the CM field of the gzip format to introduce different compression methods.
While this is the purpose of this field, I find this to be a very bad idea. The worse part of it is that, after "lzma gzip" files start to proliferate, you never know if you can decompress a .gz with your version of gunzip, which is something that you currently have for granted.
If more formats start being supported inside gzip, this only gets worse...
Signature · 4 lines
Paulo Marques - www.grupopie.com "The face of a child can say it all, especially the mouth part of the face."
Quoted 8 lines · Paulo Marques
Quoted 6 lines · H. Peter Anvin
Quoted 3 lines · Michael Tokarev
[...]
On the site it's said lzma(sdk) is under rewrite to support new format with magic number and crc checks...That is an absolute must, IMO. I would use the gzip format as a base.
Quoted 8 lines
If you're suggesting a gzip like format (but with different magic, etc.), that's ok.
However, it has been suggested on similar threads to use the CM field of the gzip format to introduce different compression methods.
While this is the purpose of this field, I find this to be a very bad idea. The worse part of it is that, after "lzma gzip" files start to proliferate, you never know if you can decompress a .gz with your version of gunzip, which is something that you currently have for granted.
If more formats start being supported inside gzip, this only gets worse...
Doesn't mean that one should name the files .gz.
A more significant reason to not do this is that I think there are a lot of programs out where which only check the magic number and not the compression format.
-hpa