Skip to contentSkip to conversation

XZ Migration discussion

DISCUSSION J.H. 35 participants 106 messages Feb 11, 2010 – Feb 23, 2010 2 messages not in the archive

lore
Share & tools
lore

J.H.

author
#0 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hey Everyone,

So as the subject states this is more a centralized discussion on migration plans to using and providing xz for content on kernel.org. Currently we provide gz and bz2, with gz acting as the original content and kernel.org itself generating the resulting bz2 files. There are a couple of possible proposals and wanted to toss them out there, and get feedback from everyone: the kernel community, the mirrors of kernel.org and the direct users of kernel.org.

========================================================================

Option 1)

Leave gz as the master, and migrate bz2 to xz. This will happen in stages obviously. with bz2 ultimately being phased out.

Migration option 1)

All new content would be provided in .bz2 and .xz with an ultimate date set that the .bz2 files would stop being generated with new content. This would leave all existing content alone and it would not be a migration of the current .bz2 files to xz

Migration option 2)

At some point there would be a mass conversion of all existing content to include .bz2 and .xz. These would be run in parallel for a time period until it was determined that .bz2 was no longer needed and it would be removed from the servers leaving .gz and .xz

Option 2)

Convert the master data from gz to bz2 and use xz as the new file format. This has the downside of causing more tool churn as it means the kernel developers will have to eventually convert from gz to bz2, which means for a time there will be nag e-mails if you upload gz instead of bz2 and such. It would also mean that we (kernel.org) would need to be able to support .gz and .bz2 as master data for a time.

Migration options are identical to Option 1 more or less, with either just new content getting converted, or all content getting converted.

========================================================================

I'm personally leaning towards option 1, though personally don't really have a preference on the migration options, as both obviously offer different advantages, and again this e-mail is more to spur on the discussion and come to some general consensus across all of the groups concerned before moving forward with a more specific plan.

So I'm inviting discussion, questions and comments on this so we know which way to ultimately go.

- - John 'Warthog9' Hawley
Chief Kernel.org Administrator
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iEYEARECAAYFAkt0ThMACgkQ/E3kyWU9dicQIwCggCACEHuEViVvnIiv42McbCQh SmUAn049beEHPucEc7X+0hBQkW6A5oTt
=ugaX
-----END PGP SIGNATURE-----

#1 lore raw
Quoted 42 lines · J.H.

Hey Everyone,

So as the subject states this is more a centralized discussion on migration plans to using and providing xz for content on kernel.org. Currently we provide gz and bz2, with gz acting as the original content and kernel.org itself generating the resulting bz2 files. There are a couple of possible proposals and wanted to toss them out there, and get feedback from everyone: the kernel community, the mirrors of kernel.org and the direct users of kernel.org.

========================================================================

Option 1)

Leave gz as the master, and migrate bz2 to xz. This will happen in stages obviously. with bz2 ultimately being phased out.

Migration option 1)

All new content would be provided in .bz2 and .xz with an ultimate date set that the .bz2 files would stop being generated with new content. This would leave all existing content alone and it would not be a migration of the current .bz2 files to xz

Migration option 2)

At some point there would be a mass conversion of all existing content to include .bz2 and .xz. These would be run in parallel for a time period until it was determined that .bz2 was no longer needed and it would be removed from the servers leaving .gz and .xz

Option 2)

Convert the master data from gz to bz2 and use xz as the new file format. This has the downside of causing more tool churn as it means the kernel developers will have to eventually convert from gz to bz2, which means for a time there will be nag e-mails if you upload gz instead of bz2 and such. It would also mean that we (kernel.org) would need to be able to support .gz and .bz2 as master data for a time.

Quoted 2 lines

Migration options are identical to Option 1 more or less, with either just new content getting converted, or all content getting converted.

I thought that the result of the poll that Linus did a few days ago was that there were advantages in having .gz, but that .xz was both smaller and faster than .bz2 and therefor the best thing to do would be to end up with .gz and .xz.

As such your option 2 doesn't seem like a reasonable path to go through.

David Lang

#2 lore raw
In reply to J.H.
Quoted 35 lines · J.H.

Option 1)

Leave gz as the master, and migrate bz2 to xz. This will happen in stages obviously. with bz2 ultimately being phased out.

Migration option 1)

All new content would be provided in .bz2 and .xz with an ultimate date set that the .bz2 files would stop being generated with new content. This would leave all existing content alone and it would not be a migration of the current .bz2 files to xz

Migration option 2)

At some point there would be a mass conversion of all existing content to include .bz2 and .xz. These would be run in parallel for a time period until it was determined that .bz2 was no longer needed and it would be removed from the servers leaving .gz and .xz
Option 2)

Convert the master data from gz to bz2 and use xz as the new file format. This has the downside of causing more tool churn as it means the kernel developers will have to eventually convert from gz to bz2, which means for a time there will be nag e-mails if you upload gz instead of bz2 and such. It would also mean that we (kernel.org) would need to be able to support .gz and .bz2 as master data for a time.

Migration options are identical to Option 1 more or less, with either just new content getting converted, or all content getting converted.

========================================================================

Quoted 8 lines

I'm personally leaning towards option 1, though personally don't really have a preference on the migration options, as both obviously offer different advantages, and again this e-mail is more to spur on the discussion and come to some general consensus across all of the groups concerned before moving forward with a more specific plan.

So I'm inviting discussion, questions and comments on this so we know which way to ultimately go.

My personal recommendation would be for Option 1, Migration option 2.

I think the idea of having two "best" file formats (Migration Option 1) in use indefinitely is rather pointless.

As for the motivations for Option 1:

a) Currently, .gz contents is original, whereas .bz2 contents is automatically generated. Flushing the .gz files would mean flushing original content.

b) .gz is one of the most widely supported compression formats ever created, plus it is very fast, especially on the decompression side. .xz is reasonably fast but very memory intensive; .bz2 is moderately fast and still fairly memory intensive (but not as much as .xz). In other words, .gz provides an option for small, slow or old systems, or systems running inferior operating systems. .xz provides the best compression. .bz2 is in between, but it doesn't serve either purpose as well as the other two.

Realistically speaking, kernel.org itself will carry all three formats for an extended transition time (at least a year); we will probably discontinue the victim format only when we start running shy on disk space. However, we obviously don't want to push this burden onto all the mirrors. Therefore, I would really appreciate feedback from mirror admins as to how you would prefer to see the transition -- either transition -- happen.

-hpa

#3 lore raw

H. Peter Anvin ([email protected]) wrote on 11 February 2010 11:48:

Quoted 31 lines · J.H.
Quoted 30 lines

Option 1)

Leave gz as the master, and migrate bz2 to xz. This will happen in stages obviously. with bz2 ultimately being phased out.

Migration option 1)

All new content would be provided in .bz2 and .xz with an ultimate date set that the .bz2 files would stop being generated with new content. This would leave all existing content alone and it would not be a migration of the current .bz2 files to xz

Migration option 2)

At some point there would be a mass conversion of all existing content to include .bz2 and .xz. These would be run in parallel for a time period until it was determined that .bz2 was no longer needed and it would be removed from the servers leaving .gz and .xz
Option 2)

Convert the master data from gz to bz2 and use xz as the new file format. This has the downside of causing more tool churn as it means the kernel developers will have to eventually convert from gz to bz2, which means for a time there will be nag e-mails if you upload gz instead of bz2 and such. It would also mean that we (kernel.org) would need to be able to support .gz and .bz2 as master data for a time.

Quoted 4 lines
Quoted 2 lines

Migration options are identical to Option 1 more or less, with either just new content getting converted, or all content getting converted.

My personal recommendation would be for Option 1, Migration option 2.

Agreed (speaking for the ftp.br.kernel.org mirror).

#4 lore raw
In reply to J.H.

Hi John,

On Thu, Feb 11, 2010 at 10:36:03AM -0800, J.H. wrote:
(...)

Quoted 5 lines

I'm personally leaning towards option 1, though personally don't really have a preference on the migration options, as both obviously offer different advantages, and again this e-mail is more to spur on the discussion and come to some general consensus across all of the groups concerned before moving forward with a more specific plan.

I too think option 1 is better for various reasons, one of them being that gzip is present on every platform and OS right now, while bzip2 is orders of magnitude less common. Of course on Linux we all have both of them, but I find it important to continue to provide the sources in a standard format. Also, on the platforms where bzip2 is not installed by default, switching to xz should not be that hard, because either bz2 is currently not used and that changes nothing, or the users managed to install it themselves, and they'll simply have to do the same with xz.

I don't have much preference for any of the two migration options though.

Just my 2 cents,
Willy

#5 lore raw
In reply to J.H.

Hi!

Quoted 5 lines

Option 1)

Leave gz as the master, and migrate bz2 to xz. This will happen in stages obviously. with bz2 ultimately being phased out.

Quoted 7 lines

Migration option 1)

All new content would be provided in .bz2 and .xz with an ultimate date set that the .bz2 files would stop being generated with new content. This would leave all existing content alone and it would not be a migration of the current .bz2 files to xz

I believe this is cleanest. gzip has performance advantages, and old files suddenly disappearing would be just weird.

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#6 lore raw
Quoted 9 lines · Pavel Machek

Hi!

Quoted 7 lines

Option 1)

Leave gz as the master, and migrate bz2 to xz. This will happen in stages obviously. with bz2 ultimately being phased out.

Migration option 1)

Quoted 9 lines
Quoted 5 lines

All new content would be provided in .bz2 and .xz with an ultimate date set that the .bz2 files would stop being generated with new content. This would leave all existing content alone and it would not be a migration of the current .bz2 files to xz

I believe this is cleanest. gzip has performance advantages, and old files suddenly disappearing would be just weird.

It would also be "just weird" to:

a) require that the end user knows the particular compression format used by a particular legacy file.
b) having to have the mirror system deal with a mix of .bz2 and .xz files.

-hpa

#7 lore raw
Quoted 4 lines · H. Peter Anvin

It would also be "just weird" to:

a) require that the end user knows the particular compression format used by a particular legacy file.

While we're here... I think it is weird that the testing files for the latest kernel are in testing/ directly, while the older ones have their own subdirectory. I have a script which is greatly confused by this, and I can imagine that other tools (for example ketchup) appreciate this inconsistency moderately too. It is probably not ideal for mirrors either... I don't know if the mirroring system is smart enough to deal with file moves? If not then it generates useless traffic.

So I would like to propose that testing files are placed directly in their final location. This would be both more consistent and easier for everyone.

Thanks,

Signature · 1 line
Jean Delvare
#8 lore raw
In reply to Pavel Machek

Hi Pavel, all,

Quoted 9 lines · Pavel Machek

Hi!

Quoted 7 lines

Option 1)

Leave gz as the master, and migrate bz2 to xz. This will happen in stages obviously. with bz2 ultimately being phased out.

Migration option 1)

Quoted 7 lines
Quoted 5 lines

All new content would be provided in .bz2 and .xz with an ultimate date set that the .bz2 files would stop being generated with new content. This would leave all existing content alone and it would not be a migration of the current .bz2 files to xz

I believe this is cleanest. gzip has performance advantages, (...)

I have been investigating this issue and would like to share my findings as an additional data point for this discussion.

For my testing, I have been using the slowest machine I still have available here: a Pentium 166 MMX, with 64 MB of memory and a slow hard disk drive. I've been writing down the duration of each task it took to boot kernel 2.6.27.45 on this machine. I did this for both .gz and .bz2 formats.

Raw results are as follow (format=min:s):

downloading linux-2.6.27.tar.bz2 5:01
downloading patch-2.6.27.45.bz2 0:02
unpacking linux-2.6.27.tar.bz2 7:28
applying patch-2.6.27.45.bz2 1:21
----------------------------------------------
total for bz2 13:52

downloading linux-2.6.27.tar.gz 6:23
downloading patch-2.6.27.45.gz 0:02
unpacking linux-2.6.27.tar.gz 3:20
applying patch-2.6.27.45.gz 1:10
----------------------------------------------
total for gz 10:55

So the gz option is unsurprisingly faster, setting up the source tree takes almost 3 minutes less (-21%).

Then the (common) build and installation times:

building 117:26
installing modules 0:12
----------------------------------------------
total 117:38

This is a customized kernel, as small as I could do, with almost no features and the minimal set of drivers. As you can see, the build time is one order of magnitude greater than the tree setup time. Comparing the total times from download to install between bz2 and gz:

bz2: 13:52 + 117:38 = 131:30
gz:  10:55 + 117:38 = 128:33

Compared to bz2, gz saves... 2% on the overall time. As a conclusion, I think we can plain discard the argument "I need .gz because my machine is slow" from now on. It simply doesn't hold.

Signature · 1 line
Jean Delvare
#9 lore raw
Quoted 43 lines · Jean Delvare
Quoted 1 line · Pavel Machek

I believe this is cleanest. gzip has performance advantages, (...)

I have been investigating this issue and would like to share my findings as an additional data point for this discussion.

For my testing, I have been using the slowest machine I still have available here: a Pentium 166 MMX, with 64 MB of memory and a slow hard disk drive. I've been writing down the duration of each task it took to boot kernel 2.6.27.45 on this machine. I did this for both .gz and .bz2 formats.

Raw results are as follow (format=min:s):

downloading linux-2.6.27.tar.bz2          5:01
downloading patch-2.6.27.45.bz2           0:02
unpacking linux-2.6.27.tar.bz2            7:28
applying patch-2.6.27.45.bz2              1:21
----------------------------------------------
total for bz2                            13:52

downloading linux-2.6.27.tar.gz           6:23
downloading patch-2.6.27.45.gz            0:02
unpacking linux-2.6.27.tar.gz             3:20
applying patch-2.6.27.45.gz               1:10
----------------------------------------------
total for gz                             10:55

So the gz option is unsurprisingly faster, setting up the source tree takes almost 3 minutes less (-21%).

Then the (common) build and installation times:

building                                117:26
installing modules                        0:12
----------------------------------------------
total                                   117:38

This is a customized kernel, as small as I could do, with almost no features and the minimal set of drivers. As you can see, the build time is one order of magnitude greater than the tree setup time. Comparing the total times from download to install between bz2 and gz:

Quoted 6 lines

bz2: 13:52 + 117:38 = 131:30
gz:  10:55 + 117:38 = 128:33

Compared to bz2, gz saves... 2% on the overall time. As a conclusion, I think we can plain discard the argument "I need .gz because my machine is slow" from now on. It simply doesn't hold.

166 MHz and 64 MB of RAM is still an order of magnitude more than some other machines Linux is capable of running on.

Of course we no longer build kernels on those machines natively, we cross-compile
on much faster machines (and use git to fetch the kernel sources, FWIW ;-).

BTW, who still downloads full kernel tarballs? From my experience, that's mostly distro people who have scripts to build kernels from tarballs + a bunch of patches?
I guess actual developers use git nowadays, even if they're stuck on an old 2.6.x kernel?
Backporting critical fixes is sooo much easier using git cherry-pick...

Do we have statistics about tarball downloads vs. git clone / git pull?

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

#10 lore raw
Quoted 54 lines · Geert Uytterhoeven
Quoted 43 lines · Jean Delvare
Quoted 1 line · Pavel Machek

I believe this is cleanest. gzip has performance advantages, (...)

I have been investigating this issue and would like to share my findings as an additional data point for this discussion.

For my testing, I have been using the slowest machine I still have available here: a Pentium 166 MMX, with 64 MB of memory and a slow hard disk drive. I've been writing down the duration of each task it took to boot kernel 2.6.27.45 on this machine. I did this for both .gz and .bz2 formats.

Raw results are as follow (format=min:s):

downloading linux-2.6.27.tar.bz2          5:01
downloading patch-2.6.27.45.bz2           0:02
unpacking linux-2.6.27.tar.bz2            7:28
applying patch-2.6.27.45.bz2              1:21

----------------------------------------------
total for bz2 13:52

downloading linux-2.6.27.tar.gz           6:23
downloading patch-2.6.27.45.gz            0:02
unpacking linux-2.6.27.tar.gz             3:20
applying patch-2.6.27.45.gz               1:10

----------------------------------------------
total for gz 10:55

So the gz option is unsurprisingly faster, setting up the source tree takes almost 3 minutes less (-21%).

Then the (common) build and installation times:

building 117:26
installing modules 0:12
----------------------------------------------
total 117:38

This is a customized kernel, as small as I could do, with almost no features and the minimal set of drivers. As you can see, the build time is one order of magnitude greater than the tree setup time. Comparing the total times from download to install between bz2 and gz:

Quoted 6 lines
bz2: 13:52 + 117:38 = 131:30
gz:  10:55 + 117:38 = 128:33

Compared to bz2, gz saves... 2% on the overall time. As a conclusion, I think we can plain discard the argument "I need .gz because my machine is slow" from now on. It simply doesn't hold.

166 MHz and 64 MB of RAM is still an order of magnitude more than some other machines Linux is capable of running on.

Quoted 7 lines

Of course we no longer build kernels on those machines natively, we cross-compile
on much faster machines (and use git to fetch the kernel sources, FWIW ;-).

BTW, who still downloads full kernel tarballs? From my experience, that's mostly distro people who have scripts to build kernels from tarballs + a bunch of patches?

All of my daily build & boot testing downloads tarballs. They use full linux-x.y.z for "releases" (like 2.6.32) and they use patch-x.y.z.gz/bz2 for intermediate versions.

I don't mind updating the scripts to use .xz tarballs, but I would really prefer to stick with tarballs instead of git trees.

Quoted 5 lines

I guess actual developers use git nowadays, even if they're stuck on an old 2.6.x kernel?
Backporting critical fixes is sooo much easier using git cherry-pick...

Do we have statistics about tarball downloads vs. git clone / git pull?

That would be good to see.

Signature · 1 line
~Randy
#11 lore raw
In reply to Geert Uytterhoeven
Quoted 3 lines · Geert Uytterhoeven

BTW, who still downloads full kernel tarballs? From my experience, that's mostly distro people who have scripts to build kernels from tarballs + a bunch of patches?

When I download to develop, I use git. When I just want to install the latest kernel on a box (not used for development) I use ketchup (downloads tarballs).

For those that just want the latest kernel release, git is too bloated.

-- Steve

#12 lore raw
In reply to Jean Delvare
Quoted 22 lines · Jean Delvare

For my testing, I have been using the slowest machine I still have available here: a Pentium 166 MMX, with 64 MB of memory and a slow hard disk drive. I've been writing down the duration of each task it took to boot kernel 2.6.27.45 on this machine. I did this for both .gz and .bz2 formats.

Raw results are as follow (format=min:s):

downloading linux-2.6.27.tar.bz2 5:01
downloading patch-2.6.27.45.bz2 0:02
unpacking linux-2.6.27.tar.bz2 7:28
applying patch-2.6.27.45.bz2 1:21
----------------------------------------------
total for bz2 13:52

downloading linux-2.6.27.tar.gz 6:23
downloading patch-2.6.27.45.gz 0:02
unpacking linux-2.6.27.tar.gz 3:20
applying patch-2.6.27.45.gz 1:10
----------------------------------------------
total for gz 10:55

Quoted 2 lines

So the gz option is unsurprisingly faster, setting up the source tree takes almost 3 minutes less (-21%).

If the download link had been slower than about 75 kB/s, the bz2 option would have been faster even on this old machine.

With xz, download would be faster than bz2 and decompression would be somewhere between bz2 and gz --- at least on machines without notable memory constraints. xz's decompressor is more memory hungry than bzip2's one as far as I understand their manual pages. But at the default xz compressor setting of -6, the decompressor will still use just 10 MB and should therefore not cause even your 64 MB machine to swap all the time during decompression.

Quoted 12 lines

Then the (common) build and installation times:

building 117:26
installing modules 0:12
----------------------------------------------
total 117:38

This is a customized kernel, as small as I could do, with almost no features and the minimal set of drivers. As you can see, the build time is one order of magnitude greater than the tree setup time. Comparing the total times from download to install between bz2 and gz:

Quoted 6 lines
bz2: 13:52 + 117:38 = 131:30
gz:  10:55 + 117:38 = 128:33

Compared to bz2, gz saves... 2% on the overall time. As a conclusion, I think we can plain discard the argument "I need .gz because my machine is slow" from now on. It simply doesn't hold.

Yep, whether the target machine is meant to compile the kernel or to be used to browse the source code (with a bare minimum of comfort), the hardware resources required for either task mean that there isn't such a great difference between gz, bz2, xz WRT resource requirements at the receivers, except that xz goes easiest on network bandwidth and disk utilization.

Signature · 3 lines
Stefan Richter
-=====-==-=- --=- -==-=
http://arcgraph.de/sr/
#13 lore raw
Quoted 7 lines · Stefan Richter
Quoted 2 lines · Jean Delvare

So the gz option is unsurprisingly faster, setting up the source tree takes almost 3 minutes less (-21%).

If the download link had been slower than about 75 kB/s, the bz2 option would have been faster even on this old machine.

Quoted 7 lines

With xz, download would be faster than bz2 and decompression would be somewhere between bz2 and gz --- at least on machines without notable memory constraints. xz's decompressor is more memory hungry than bzip2's one as far as I understand their manual pages. But at the default xz compressor setting of -6, the decompressor will still use just 10 MB and should therefore not cause even your 64 MB machine to swap all the time during decompression.

Note that, if memory consumption is really a concern on either end, we could use xz -5, which still achieves much better compression than bz2 but doesn't require more memory for decompression.

Signature · 1 line
Jean Delvare
#14 lore raw
In reply to Jean Delvare
Quoted 5 lines · Jean Delvare

Compared to bz2, gz saves... 2% on the overall time. As a conclusion, I think we can plain discard the argument "I need .gz because my machine is slow" from now on. It simply doesn't hold.

I agree, but, IMHO the main argument for keeping .gz is cross-platform availability and wide language support, not hardware limitations. Doing a quick google brings up .gz interfaces for every language you can think of (C, Java, Perl, Python, TCL etc.), not to mention complete separate implementations in Java and Pascal (not just wrappers on top of the zlib library), and probably more.

With xz you have just one C/C++ implementation with a single library with an undocumented API for C/C++ programmers.

It may be a slight stretch of the imagination, but with with .gz you can conceive programmers writing programs to download a .gz from kernel.org and decompressing/searching it, in almost any language of choice. With the JAVA implementation .gz is genuinely cross platform and you don't need glibc/ C++ compilers, just a Java VM. Contrast with xz, where if the xz utility isn't available, or doesn't do what you want, you're stuck with programming in C/C++ with all the baggage that entails.

Phillip

#15 lore raw
Quoted 15 lines · Phillip Lougher
Quoted 5 lines · Jean Delvare

Compared to bz2, gz saves... 2% on the overall time. As a conclusion, I think we can plain discard the argument "I need .gz because my machine is slow" from now on. It simply doesn't hold.

I agree, but, IMHO the main argument for keeping .gz is cross-platform availability and wide language support, not hardware limitations. Doing a quick google brings up .gz interfaces for every language you can think of (C, Java, Perl, Python, TCL etc.), not to mention complete separate implementations in Java and Pascal (not just wrappers on top of the zlib library), and probably more.

Quoted 2 lines

With xz you have just one C/C++ implementation with a single library with an undocumented API for C/C++ programmers.

This can probably be easily explained. gz is very fast decompressing so it is a very good choice for transparent decompression of files which must be accessible fast but aren't used frequently. Manual pages or printer drivers come to mind. bz2 and lzma, OTOH, are meant for longer term archiving. Their compression ratio benefit is only worth it for larger files that you don't access that frequently.

I am not claiming that gzip is dead. It is very useful and it is there to stay for the years to come, no doubt about that. What I'm saying is that it isn't the best choice for large files to be downloaded from a remote server.

Quoted 7 lines

It may be a slight stretch of the imagination, but with with .gz you can conceive programmers writing programs to download a .gz from kernel.org and decompressing/searching it, in almost any language of choice. With the JAVA implementation .gz is genuinely cross platform and you don't need glibc/ C++ compilers, just a Java VM. Contrast with xz, where if the xz utility isn't available, or doesn't do what you want, you're stuck with programming in C/C++ with all the baggage that entails.

Honestly, I don't think we care at all when it comes to the kernel.org files. Accessing individual files inside a compressed kernel tarball without first expanding it entirely would be horribly slow and unpractical, no matter which compression format was used. I can't think of any case where you won't unpack the tarball first, and for this task an external tool will do just fine.

And, once again, there are several public instances of gitweb and LXR available if you only want to browse the code.

Signature · 1 line
Jean Delvare
#16 lore raw
Quoted 46 lines · Jean Delvare
Quoted 15 lines · Phillip Lougher
Quoted 5 lines · Jean Delvare

Compared to bz2, gz saves... 2% on the overall time. As a conclusion, I think we can plain discard the argument "I need .gz because my machine is slow" from now on. It simply doesn't hold.

I agree, but, IMHO the main argument for keeping .gz is cross-platform availability and wide language support, not hardware limitations. Doing a quick google brings up .gz interfaces for every language you can think of (C, Java, Perl, Python, TCL etc.), not to mention complete separate implementations in Java and Pascal (not just wrappers on top of the zlib library), and probably more.

Quoted 2 lines

With xz you have just one C/C++ implementation with a single library with an undocumented API for C/C++ programmers.

This can probably be easily explained. gz is very fast decompressing so it is a very good choice for transparent decompression of files which must be accessible fast but aren't used frequently. Manual pages or printer drivers come to mind. bz2 and lzma, OTOH, are meant for longer term archiving. Their compression ratio benefit is only worth it for larger files that you don't access that frequently.

I am not claiming that gzip is dead. It is very useful and it is there to stay for the years to come, no doubt about that. What I'm saying is that it isn't the best choice for large files to be downloaded from a remote server.

Quoted 7 lines

It may be a slight stretch of the imagination, but with with .gz you can conceive programmers writing programs to download a .gz from kernel.org and decompressing/searching it, in almost any language of choice. With the JAVA implementation .gz is genuinely cross platform and you don't need glibc/ C++ compilers, just a Java VM. Contrast with xz, where if the xz utility isn't available, or doesn't do what you want, you're stuck with programming in C/C++ with all the baggage that entails.

Honestly, I don't think we care at all when it comes to the kernel.org files. Accessing individual files inside a compressed kernel tarball without first expanding it entirely would be horribly slow and unpractical, no matter which compression format was used. I can't think of any case where you won't unpack the tarball first, and for this task an external tool will do just fine.

Quoted 3 lines

And, once again, there are several public instances of gitweb and LXR available if you only want to browse the code.

just out of curiosity what would happen if by say I take a file and turn it into .gz then turn the .gz into .xz or vice versa?

so at the end of the day you have a list of .gz's(or whatever),
then expending on the type(.gz,.bz2,etc..) unpackage and voila either a tree or some other compressed file(.bz2,xz, or .gz).

just thinking out loud(so don't shoot me please).

Justin P. Mattock

#17 lore raw
In reply to Jean Delvare

Hi Jean,

Quoted 4 lines · Jean Delvare

I am not claiming that gzip is dead. It is very useful and it is there to stay for the years to come, no doubt about that. What I'm saying is that it isn't the best choice for large files to be downloaded from a remote server.

Well, I personally like to be able to simply run "less patch-2.6.27.45.gz" and have it transparently uncompressed and dumped on my terminal. It doesn't do that on bz2. We could find multiple examples.

Another thing comes to mind, because I've been beaten by this in the past. People working in enterprises where the internet access passes via mandatory proxies coupled with anti-virus can often download many things but not binaries that can't be analysed. At this time, I could only download tar.gz but not .bz2. And please don't tell me I have to go to the admin to tell him to change his proxy's configuration, you can't do that when you work as a consultant for a 20000 persons group where products are selected after 6-12 months of testing and managed by different people from those who qualify them.

In my opinion, .xz is a very good option to replace .bz2 as it will bring advantages without downsides. But .gz should stay as it has been available from day 1, at least for all people who may have trouble with .xz for whatever reason. If it has not been a problem for the last 16 years, I don't see why it would suddenly become one.

Regards,
Willy

#18 lore raw

Salut Willy,

Quoted 9 lines · Willy Tarreau
Quoted 4 lines · Jean Delvare

I am not claiming that gzip is dead. It is very useful and it is there to stay for the years to come, no doubt about that. What I'm saying is that it isn't the best choice for large files to be downloaded from a remote server.

Well, I personally like to be able to simply run "less patch-2.6.27.45.gz" and have it transparently uncompressed and dumped on my terminal. It doesn't do that on bz2. We could find multiple examples.

This only underlines what I just wrote: the purpose of gzip is different from the purpose of bzip2 or lzma. Transparent decompression makes sense for the former but little for the latter two. But this doesn't mean everyone must make all their files available in .gz format. Not every file, and not everyone, needs transparent decompression. It makes sense on patch files, but not on tarballs. You have the need, but I don't. Furthermore, the fact that your local usage of patch files is more convenient if they come in .gz format doesn't imply that this is the format in which you have to download them. You are free to repack files after downloading them. This can even be easily automated.

hpa seems to be very reluctant to the idea, but one thing I had in mind was that patches could be in .gz format and tarballs in .xz, for example. Having to stick to a common strategy for all the files seems suboptimal, given their different natures and uses.

(And for completeness: on my distribution, the bzip2 package comes with a little helper script named bzless, which does exactly what you want for bzip2-compressed patches. But I wouldn't recommend using it...)

Quoted 9 lines

Another thing comes to mind, because I've been beaten by this in the past. People working in enterprises where the internet access passes via mandatory proxies coupled with anti-virus can often download many things but not binaries that can't be analysed. At this time, I could only download tar.gz but not .bz2. And please don't tell me I have to go to the admin to tell him to change his proxy's configuration, you can't do that when you work as a consultant for a 20000 persons group where products are selected after 6-12 months of testing and managed by different people from those who qualify them.

I've been working in such environments in the past, so I see what you mean. And I do not disagree that the format in which projects make their files available can be more or less convenient in such situations. For example, I remember being deeply annoyed by projects not releasing tarballs, because I simply couldn't access CVS repositories.

That being said, I also don't think that you can put all the burden of bad company policies on the shoulder of open source software providers. If someone worked in a company where even .gz compressed files can't be downloaded, we wouldn't ask kernel.org to provide uncompressed files on top of .gz and .bz2, would we? There is a trade-off to be found between flexibility of access and disk and network usage.

It should be possible to retrieve the kernel sources using git over HTTP these days anyway, right? Or are firewalls frequently blocking this as well?

Quoted 5 lines

In my opinion, .xz is a very good option to replace .bz2 as it will bring advantages without downsides. But .gz should stay as it has been available from day 1, at least for all people who may have trouble with .xz for whatever reason. If it has not been a problem for the last 16 years, I don't see why it would suddenly become one.

People have been using Windows for the last 16 years, it has not been a problem, so I don't see why it would suddenly become one. Let's stop working on an alternative. That way we don't have to decide which compression format to use! Problem solved ;)

Seriously: things can become a problem over time even if they were not one initially, and needs may evolve as well. The Linux kernel releases have grown very big compared to the early days, and we release more often, and new compression technologies exist and are getting widely adopted. You don't have to stand in front of the wall to declare that there's a problem. Just improving the user experience is worth the change sometimes.

That being all said, I don't fundamentally disagree with you: just replacing .bz2 with .xz would be fine with me. Really, my main concern is the file count in directory v2.6/, much more than the compression formats being used.

Signature · 1 line
Jean Delvare
#19 lore raw
In reply to Willy Tarreau
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"W" == Willy Tarreau <[email protected]> writes:

W> Well, I personally like to be able to simply run "less patch-2.6.27.45.gz" W> and have it transparently uncompressed and dumped on my terminal. It W> doesn't do that on bz2. We could find multiple examples.

It does here, and lzma & xz, too. And has since just days after Lasse annouced that the new name would be xz.

Your LESSOPEN controls that, and can be easily coded to support any archive.

-JimC

Signature · 1 line
James Cloos <[email protected]>         OpenPGP: 1024D/ED7DAEA6
#20 lore raw
Quoted 10 lines · James Cloos
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"W" == Willy Tarreau <[email protected]> writes:

W> Well, I personally like to be able to simply run "less patch-2.6.27.45.gz" W> and have it transparently uncompressed and dumped on my terminal. It W> doesn't do that on bz2. We could find multiple examples.

It does here, and lzma & xz, too. And has since just days after Lasse annouced that the new name would be xz.

Your LESSOPEN controls that, and can be easily coded to support any archive.

Just checked and I found it funny to see that patch-2.6.1.bz2 is not correctly opened while 2.6.27.45.bz2 is. Maybe some things have slightly changed in the tools by that time and the output differs slightly. However bzcat opens them both so the file is not corrupted.

This raises the point of the maturity of the tools BTW. GZ is mature, BZ2 has become mature over the years, XZ is very recent and may still be buggy at times. We'll only know that when people will complain that they cannot open one file from time to time.

Willy

#21 lore raw
Quoted 6 lines · Willy Tarreau

This raises the point of the maturity of the tools BTW. GZ is mature, BZ2 has become mature over the years, XZ is very recent and may still be buggy at times. We'll only know that when people will complain that they cannot open one file from time to time.

This isn't a huge problem. Since it is generated content we can, if necessary, run a robot over the archive and verify and regenerate files that have problems.

-hpa

#22 lore raw
In reply to Willy Tarreau
Quoted 29 lines · Willy Tarreau
Quoted 10 lines · James Cloos
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"W" == Willy Tarreau <[email protected]> writes:

W> Well, I personally like to be able to simply run "less patch-2.6.27.45.gz" W> and have it transparently uncompressed and dumped on my terminal. It W> doesn't do that on bz2. We could find multiple examples.

It does here, and lzma & xz, too. And has since just days after Lasse annouced that the new name would be xz.

Your LESSOPEN controls that, and can be easily coded to support any archive.

Just checked and I found it funny to see that patch-2.6.1.bz2 is not correctly opened while 2.6.27.45.bz2 is. Maybe some things have slightly changed in the tools by that time and the output differs slightly. However bzcat opens them both so the file is not corrupted.

This raises the point of the maturity of the tools BTW. GZ is mature, BZ2 has become mature over the years, XZ is very recent and may still be buggy at times. We'll only know that when people will complain that they cannot open one file from time to time.

Willy

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#23 lore raw
In reply to Willy Tarreau
Quoted 7 lines · Willy Tarreau
Quoted 6 lines · James Cloos
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"W" == Willy Tarreau <[email protected]> writes:

W> Well, I personally like to be able to simply run "less patch-2.6.27.45.gz" W> and have it transparently uncompressed and dumped on my terminal. It W> doesn't do that on bz2. We could find multiple examples.

Quoted 7 lines
Quoted 4 lines

It does here, and lzma & xz, too. And has since just days after Lasse annouced that the new name would be xz.

Your LESSOPEN controls that, and can be easily coded to support any archive.

Just checked and I found it funny to see that patch-2.6.1.bz2 is not correctly opened while 2.6.27.45.bz2 is.

There is a bug in lesspipe.sh, at least here. It confuses it with compresses man pages.

lesspipe.sh 2010-02-17 12:19:09.000000000 +0200 renamed from lesspipe.sh~ 2009-11-06 02:14:58.000000000 +0200 +0 -0
@@ -50,6 +50,7 @@
 

*.1.bz2|*.2.bz2|*.3.bz2|*.4.bz2|*.5.bz2|*.6.bz2|*.7.bz2|*.8.bz2|*.9.bz2|*.n.bz2|*.man.bz2)

# compressed *roff src?
     if bzip2 -dc "$1" | file - | grep roff 1> /dev/null ; then
       bzip2 -dc "$1" | nroff -S -mandoc -
+    else bzip2 -dc "$1"  2>/dev/null
     fi ;;
   *.gz) gzip -dc "$1"  2>/dev/null ;;
   *.bz2) bzip2 -dc "$1" 2>/dev/null ;;
#24 lore raw
In reply to Phillip Lougher
Quoted 2 lines

With xz you have just one C/C++ implementation with a single library with an undocumented API for C/C++ programmers.

... which is not even widely deployed. I did a quick survey and none of my systems (none of which terrible old and not particularly embedded) have it installed and for most them there's only "lzma-utils" in the distribution package repositories which I understand is not compatible.

I would basically need to download the source by hand and install it like back in the bad old "unix with all useful commands missing" HP-UX/Solaris/etc. days.

Please keep .gz and better .bz2

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#25 lore raw
Quoted 11 lines · Andi Kleen
Quoted 2 lines

With xz you have just one C/C++ implementation with a single library with an undocumented API for C/C++ programmers.

... which is not even widely deployed. I did a quick survey and none of my systems (none of which terrible old and not particularly embedded) have it installed and for most them there's only "lzma-utils" in the distribution package repositories which I understand is not compatible.

I would basically need to download the source by hand and install it like back in the bad old "unix with all useful commands missing" HP-UX/Solaris/etc. days.

When was the last time you compiled a Linux kernel on HP-UX or Solaris? I think others did that first (me being along the party), and quite frankly, the plain Solaris without CSW has quickly-reachable limits even for non-kernels.

#26 lore raw
In reply to Phillip Lougher
Quoted 2 lines · On 2010-02-14 Phillip Lougher wrote:

With xz you have just one C/C++ implementation with a single library with an undocumented API for C/C++ programmers.

I completely agree that language support is bad compared to the .gz format, but I think the above sentence makes it sound a bit too bad.

There are three C (no C++ needed) libraries that support the .xz format:
- LZMA SDK (7-zip.org)
- XZ Utils has liblzma (tukaani.org)
- XZ Embedded (tukaani.org, limited support only)

The latter two are more or less based on LZMA SDK, although the code looks very different (different coding style, different APIs etc.).

The liblzma API has reference documentation as Doxygen tags in the API headers. They aren't the best docs and there's no tutorial or example programs yet, but the API certainly isn't undocumented. It has similarities to the zlib API, so those used to zlib shouldn't have trouble learning the basic features of liblzma, which are enough for most people.

There are liblzma bindings for Perl and Python, but I don't know how good they are.

Signature · 1 line
Lasse Collin  |  IRC: Larhzu @ IRCnet & Freenode
#27 lore raw
In reply to J.H.
Quoted 5 lines · J.H.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hey Everyone,

Quoted 7 lines

So as the subject states this is more a centralized discussion on migration plans to using and providing xz for content on kernel.org. Currently we provide gz and bz2, with gz acting as the original content and kernel.org itself generating the resulting bz2 files. There are a couple of possible proposals and wanted to toss them out there, and get feedback from everyone: the kernel community, the mirrors of kernel.org and the direct users of kernel.org.

Don't you have download statistics available? If we knew which compression format is preferred, an by which margin, it would help make an educated decision.

Quoted 37 lines

========================================================================

Option 1)

Leave gz as the master, and migrate bz2 to xz. This will happen in stages obviously. with bz2 ultimately being phased out.

Migration option 1)

All new content would be provided in .bz2 and .xz with an ultimate date set that the .bz2 files would stop being generated with new content. This would leave all existing content alone and it would not be a migration of the current .bz2 files to xz

Migration option 2)

At some point there would be a mass conversion of all existing content to include .bz2 and .xz. These would be run in parallel for a time period until it was determined that .bz2 was no longer needed and it would be removed from the servers leaving .gz and .xz

Option 2)

Convert the master data from gz to bz2 and use xz as the new file format. This has the downside of causing more tool churn as it means the kernel developers will have to eventually convert from gz to bz2, which means for a time there will be nag e-mails if you upload gz instead of bz2 and such. It would also mean that we (kernel.org) would need to be able to support .gz and .bz2 as master data for a time.

Migration options are identical to Option 1 more or less, with either just new content getting converted, or all content getting converted.

========================================================================

Quoted 8 lines

I'm personally leaning towards option 1, though personally don't really have a preference on the migration options, as both obviously offer different advantages, and again this e-mail is more to spur on the discussion and come to some general consensus across all of the groups concerned before moving forward with a more specific plan.

So I'm inviting discussion, questions and comments on this so we know which way to ultimately go.

Maybe that's just me, but my main concern is neither download times nor decompression times. My main concern is the access time to directory indexes when browsing the kernel archive, because there are 5 entries for every patch or tarball: .bz2, .bz2.sign, .gz, .gz.sign and .sign. This is horribly slow. The main directory for 2.6 kernels has an index weighting over 300 kB raw, turning into a ~600 kB document when HTML-ized. Just fetching it takes 3 seconds and then my browser takes a long time to format it. There are 3881 entries in that directory today, and it keeps growing!

So, once we have settled for a compression strategy, I think it would be the right time to discuss the directory structure. With the advent of the stable branches and the new development model - which pretty much implies that we'll live with main version 2.6 forever - the file count is much higher than it used to me.

I can think of several ways to improve the situation here, some of which could be combined.

1* Keep a single compression format. This saves almost 40% of the files.

2* Move one of the compression formats somewhere else, so that it doesn't get in the way but is still available if needed.

3* Create a new subdirectory for every 2.6.x kernel, and move all the related files there. This would shrink the main index drastically, and each subdirectory would have a reasonable size (except maybe 2.6.16 and 2.6.27.) Oddly enough this has been done for the files under testing/ already, so I am curious why we don't do it for the release files (and the testing/incr/ files, while we're at it.)

4* Get rid of the LATEST-IS-* files. This is a small count, won't save much, but these files seem totally useless to me these days. Depending on what you want exactly, there are many versions which can be considered the latest, and there are better ways to know which they are (for example http://www.eu.kernel.org/kdist/finger_banner ). And these files tend to get stuck so you can't rely on them anyway.

I wouldn't worry too much about breaking the current locations. Just give some time for software authors (ketchup comes to mind) to update their code and it shouldn't be a big problem.

Thanks,

Signature · 1 line
Jean Delvare
#28 lore raw
Quoted 3 lines · Jean Delvare

I wouldn't worry too much about breaking the current locations. Just give some time for software authors (ketchup comes to mind) to update their code and it shouldn't be a big problem.

As the new maintainer of ketchup:

git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/ketchup.git

I could write a fix for the new format real quick. My concern is that users of ketchup may not see this update for a long time.

-- Steve

#29 lore raw
Quoted 6 lines · Steven Rostedt
Quoted 3 lines · Jean Delvare

I wouldn't worry too much about breaking the current locations. Just give some time for software authors (ketchup comes to mind) to update their code and it shouldn't be a big problem.

Quoted 6 lines

As the new maintainer of ketchup:

git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/ketchup.git

I could write a fix for the new format real quick. My concern is that users of ketchup may not see this update for a long time.

What would you consider a reasonable turnaround time? 3 month? 6 month? More?

We could make a decision now, you update ketchup immediately, and the actual change happens at the specified date in the future.

Signature · 1 line
Jean Delvare
#30 lore raw
Quoted 6 lines · Jean Delvare

What would you consider a reasonable turnaround time? 3 month? 6 month? More?

We could make a decision now, you update ketchup immediately, and the actual change happens at the specified date in the future.

Usually users don't update ketchup until it breaks ;)

But sure, 3 months probably would be fine. Then Matt Mackall will start getting emails about it not working, and he will forward them to me. Where I can then point them to the new repository.

-- Steve

#31 lore raw
Quoted 9 lines · Steven Rostedt
Quoted 6 lines · Jean Delvare

What would you consider a reasonable turnaround time? 3 month? 6 month? More?

We could make a decision now, you update ketchup immediately, and the actual change happens at the specified date in the future.

Quoted 5 lines

Usually users don't update ketchup until it breaks ;)

But sure, 3 months probably would be fine. Then Matt Mackall will start getting emails about it not working, and he will forward them to me. Where I can then point them to the new repository.

You can probably anticipate this a bit by warning the maintainers of the ketchup package in openSUSE, Fedora etc.

Signature · 1 line
Jean Delvare
#32 lore raw
In reply to Steven Rostedt
Quoted 10 lines · Steven Rostedt
Quoted 6 lines · Jean Delvare

What would you consider a reasonable turnaround time? 3 month? 6 month? More?

We could make a decision now, you update ketchup immediately, and the actual change happens at the specified date in the future.

Usually users don't update ketchup until it breaks ;)

Then the archive location can change immediately after you issue an update ;-)

Would it be an option for a newer version of ketchup to have a fallback codepath to read a machine-readable description of how the archive is structured from a known location? E.g. instead of appending a hard-coded string '/people/akpm/patches/2.6/' when computing latest_mm(), the script would read that "relative path to mm" from such a description file. The logic of doing 'url = "%s/v%s/%s" % (kernel_url, t, f)' in install_nearest would similarly be customizable, without too much hassle, I think.

#33 lore raw
Quoted 16 lines · Junio C Hamano
Quoted 10 lines · Steven Rostedt
Quoted 6 lines · Jean Delvare

What would you consider a reasonable turnaround time? 3 month? 6 month? More?

We could make a decision now, you update ketchup immediately, and the actual change happens at the specified date in the future.

Usually users don't update ketchup until it breaks ;)

Then the archive location can change immediately after you issue an update ;-)

Quoted 7 lines

Would it be an option for a newer version of ketchup to have a fallback codepath to read a machine-readable description of how the archive is structured from a known location? E.g. instead of appending a hard-coded string '/people/akpm/patches/2.6/' when computing latest_mm(), the script would read that "relative path to mm" from such a description file. The logic of doing 'url = "%s/v%s/%s" % (kernel_url, t, f)' in install_nearest would similarly be customizable, without too much hassle, I think.

Are you suggesting that the path be stored at a known location on kernel.org? Such that it can be read and the tools like ketchup can find where a particular version resides?

-- Steve

#34 lore raw

ketchup was Re: [kernel.org users] XZ Migration discussion

In reply to Steven Rostedt

Hi!

Quoted 10 lines
Quoted 3 lines

I wouldn't worry too much about breaking the current locations. Just give some time for software authors (ketchup comes to mind) to update their code and it shouldn't be a big problem.

As the new maintainer of ketchup:

git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/ketchup.git

I could write a fix for the new format real quick. My concern is that users of ketchup may not see this update for a long time.

I'd really like to fix ketchup to behave sanely in late -rc stages:
going from -rc6 to -rc8 is possible using two downloads through --rc7 (like 100KB?) instead of two big ones through full release (like 10MB).

But first step is --only-dl option -- when you want to cache the needed patches but not apply anything yet.

Please apply,
Pavel

ketchup +4 -3
index 0728aec..3249cbc 100755
@@ -405,7 +405,7 @@ def apply_patch(ver, reverse = 0):
         r = " -R"
 
     qprint("Applying %s%s" % (os.path.basename(p), r))
-    if options["dry-run"]:
+    if options["dry-run"] or options["only-dl"]:
         return ver
 
     def cmd(patch, reverse, dry):
@@ -484,7 +484,7 @@ def install_nearest(ver):
         ver = list[0][2]
 
     qprint("Unpacking %s" % os.path.basename(f))
-    if options["dry-run"]: return ver
+    if options["dry-run"] or options["only-dl"]: return ver
     untar(f)
 
     return ver
@@ -658,6 +658,7 @@ opts = [
     ('l', 'list-trees', None, 'list supported trees'),
     ('m', 'show-makefile', None, 'output version in makefile <arg>'),
     ('n', 'dry-run', None, 'don\'t download or apply patches'),
+    ('o', 'only-dl', None, 'don\'t apply patches'),
     ('p', 'show-previous', None, 'output version previous to <arg>'),
     ('q', 'quiet', None, 'reduce output'),
     ('r', 'rename-directory', None, 'rename updated directory to %s<v>'
@@ -750,7 +751,7 @@ if not a and os.listdir("."):
 b = find_ver(args[0])
 qprint("%s -> %s" % (a, b))
 transform(a, b)
-if options["rename-directory"] and not options["dry-run"]:
+if options["rename-directory"] and not options["dry-run"] and not options["only-dl"] :
     rename_dir(b)
 
 if postcommand and os.system(postcommand):
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#35 lore raw

Re: ketchup was Re: [kernel.org users] XZ Migration discussion

Quoted 10 lines · Pavel Machek

I'd really like to fix ketchup to behave sanely in late -rc stages:
going from -rc6 to -rc8 is possible using two downloads through --rc7 (like 100KB?) instead of two big ones through full release (like 10MB).

But first step is --only-dl option -- when you want to cache the needed patches but not apply anything yet.

Please apply,
Pavel

Can you send this to me as an official patch. With SoB and all.

Thanks,

-- Steve

#36 lore raw

Re: ketchup was Re: [kernel.org users] XZ Migration discussion

Add --only-dl option -- when you want to cache the needed patches but not apply anything yet.

ketchup +4 -3
index 0728aec..3249cbc 100755
@@ -405,7 +405,7 @@ def apply_patch(ver, reverse = 0):
         r = " -R"
 
     qprint("Applying %s%s" % (os.path.basename(p), r))
-    if options["dry-run"]:
+    if options["dry-run"] or options["only-dl"]:
         return ver
 
     def cmd(patch, reverse, dry):
@@ -484,7 +484,7 @@ def install_nearest(ver):
         ver = list[0][2]
 
     qprint("Unpacking %s" % os.path.basename(f))
-    if options["dry-run"]: return ver
+    if options["dry-run"] or options["only-dl"]: return ver
     untar(f)
 
     return ver
@@ -658,6 +658,7 @@ opts = [
     ('l', 'list-trees', None, 'list supported trees'),
     ('m', 'show-makefile', None, 'output version in makefile <arg>'),
     ('n', 'dry-run', None, 'don\'t download or apply patches'),
+    ('o', 'only-dl', None, 'don\'t apply patches'),
     ('p', 'show-previous', None, 'output version previous to <arg>'),
     ('q', 'quiet', None, 'reduce output'),
     ('r', 'rename-directory', None, 'rename updated directory to %s<v>'
@@ -750,7 +751,7 @@ if not a and os.listdir("."):
 b = find_ver(args[0])
 qprint("%s -> %s" % (a, b))
 transform(a, b)
-if options["rename-directory"] and not options["dry-run"]:
+if options["rename-directory"] and not options["dry-run"] and not options["only-dl"] :
     rename_dir(b)
 
 if postcommand and os.system(postcommand):
ketchup.1 +6 -1
index 0a313ee..9e5a385 100644
@@ -1,5 +1,5 @@
 .\"                                      Hey, EMACS: -*- nroff -*-
-.TH KETCHUP 1 "April 12, 2006"
+.TH KETCHUP 1 "February 16, 2010"
 .\" Please adjust this date whenever revising the manpage.
 .\"
 .\" Some roff macros, for reference:
@@ -74,6 +74,11 @@ output version in makefile <arg>
 .IP
 don't download or apply patches
 .HP
+.B \-o
+.B \-\-only\-dl
+.IP
+don't apply patches
+.HP
 .B \-p
 .B \-\-show\-previous
 .IP
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#37 lore raw

Re: ketchup was Re: [kernel.org users] XZ Migration discussion

Quoted 2 lines · Pavel Machek

Add --only-dl option -- when you want to cache the needed patches but not apply anything yet.

Actually, you'll probably get a better patch if you relace 'only-dl' with just 'dl'. Mistaking --only-dl and --dl-only is just too easy.

Quoted 68 lines
ketchup +4 -3
index 0728aec..3249cbc 100755
@@ -405,7 +405,7 @@ def apply_patch(ver, reverse = 0):
         r = " -R"
 
     qprint("Applying %s%s" % (os.path.basename(p), r))
-    if options["dry-run"]:
+    if options["dry-run"] or options["only-dl"]:
         return ver
 
     def cmd(patch, reverse, dry):
@@ -484,7 +484,7 @@ def install_nearest(ver):
         ver = list[0][2]
 
     qprint("Unpacking %s" % os.path.basename(f))
-    if options["dry-run"]: return ver
+    if options["dry-run"] or options["only-dl"]: return ver
     untar(f)
 
     return ver
@@ -658,6 +658,7 @@ opts = [
     ('l', 'list-trees', None, 'list supported trees'),
     ('m', 'show-makefile', None, 'output version in makefile <arg>'),
     ('n', 'dry-run', None, 'don\'t download or apply patches'),
+    ('o', 'only-dl', None, 'don\'t apply patches'),
     ('p', 'show-previous', None, 'output version previous to <arg>'),
     ('q', 'quiet', None, 'reduce output'),
     ('r', 'rename-directory', None, 'rename updated directory to %s<v>'
@@ -750,7 +751,7 @@ if not a and os.listdir("."):
 b = find_ver(args[0])
 qprint("%s -> %s" % (a, b))
 transform(a, b)
-if options["rename-directory"] and not options["dry-run"]:
+if options["rename-directory"] and not options["dry-run"] and not options["only-dl"] :
     rename_dir(b)
 
 if postcommand and os.system(postcommand):
ketchup.1 +6 -1
index 0a313ee..9e5a385 100644
@@ -1,5 +1,5 @@
 .\"                                      Hey, EMACS: -*- nroff -*-
-.TH KETCHUP 1 "April 12, 2006"
+.TH KETCHUP 1 "February 16, 2010"
 .\" Please adjust this date whenever revising the manpage.
 .\"
 .\" Some roff macros, for reference:
@@ -74,6 +74,11 @@ output version in makefile <arg>
 .IP
 don't download or apply patches
 .HP
+.B \-o
+.B \-\-only\-dl
+.IP
+don't apply patches
+.HP
 .B \-p
 .B \-\-show\-previous
 .IP
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#38 lore raw

Re: ketchup was Re: [kernel.org users] XZ Migration discussion

Quoted 6 lines · Pavel Machek
Quoted 2 lines · Pavel Machek

Add --only-dl option -- when you want to cache the needed patches but not apply anything yet.

Actually, you'll probably get a better patch if you relace 'only-dl' with just 'dl'. Mistaking --only-dl and --dl-only is just too easy.

Or --download. Acronyms just suck.

Quoted 70 lines
Quoted 69 lines
ketchup +4 -3
index 0728aec..3249cbc 100755
@@ -405,7 +405,7 @@ def apply_patch(ver, reverse = 0):
         r = " -R"
 
     qprint("Applying %s%s" % (os.path.basename(p), r))
-    if options["dry-run"]:
+    if options["dry-run"] or options["only-dl"]:
         return ver
 
     def cmd(patch, reverse, dry):
@@ -484,7 +484,7 @@ def install_nearest(ver):
         ver = list[0][2]
 
     qprint("Unpacking %s" % os.path.basename(f))
-    if options["dry-run"]: return ver
+    if options["dry-run"] or options["only-dl"]: return ver
     untar(f)
 
     return ver
@@ -658,6 +658,7 @@ opts = [
     ('l', 'list-trees', None, 'list supported trees'),
     ('m', 'show-makefile', None, 'output version in makefile <arg>'),
     ('n', 'dry-run', None, 'don\'t download or apply patches'),
+    ('o', 'only-dl', None, 'don\'t apply patches'),
     ('p', 'show-previous', None, 'output version previous to <arg>'),
     ('q', 'quiet', None, 'reduce output'),
     ('r', 'rename-directory', None, 'rename updated directory to %s<v>'
@@ -750,7 +751,7 @@ if not a and os.listdir("."):
 b = find_ver(args[0])
 qprint("%s -> %s" % (a, b))
 transform(a, b)
-if options["rename-directory"] and not options["dry-run"]:
+if options["rename-directory"] and not options["dry-run"] and not options["only-dl"] :
     rename_dir(b)
 
 if postcommand and os.system(postcommand):
ketchup.1 +6 -1
index 0a313ee..9e5a385 100644
@@ -1,5 +1,5 @@
 .\"                                      Hey, EMACS: -*- nroff -*-
-.TH KETCHUP 1 "April 12, 2006"
+.TH KETCHUP 1 "February 16, 2010"
 .\" Please adjust this date whenever revising the manpage.
 .\"
 .\" Some roff macros, for reference:
@@ -74,6 +74,11 @@ output version in makefile <arg>
 .IP
 don't download or apply patches
 .HP
+.B \-o
+.B \-\-only\-dl
+.IP
+don't apply patches
+.HP
 .B \-p
 .B \-\-show\-previous
 .IP
Signature · 1 line
Jean Delvare
#39 lore raw

Re: ketchup was Re: [kernel.org users] XZ Migration discussion

Quoted 9 lines · Jean Delvare
Quoted 6 lines · Pavel Machek
Quoted 2 lines · Pavel Machek

Add --only-dl option -- when you want to cache the needed patches but not apply anything yet.

Actually, you'll probably get a better patch if you relace 'only-dl' with just 'dl'. Mistaking --only-dl and --dl-only is just too easy.

Or --download. Acronyms just suck.

Works for me.

And here's quick patch to show what I'm doing: it teaches ketchup about 2.6.33-rc1-rc2 patches, thus saving lot of bandwidth. Of course, I'll need to get it to work for more than 2.6.A->2.6.B-rcC case, but...

If there are some comments, or maybe better approach, let me know...

ketchup +61 -5
index 3249cbc..dc1bbf8 100755
@@ -107,19 +107,28 @@ local_trees = {}
 # Functions to parse version strings
 
 def tree(ver):
+    """returns 2.6"""
     return float(re.match(r'(\d+\.\d+)', ver).group(1))
 
 def rev(ver):
+    """given 2.6.31 or 2.6.32-rc1 returns 31"""
     p = pre(ver)
     r = int(re.match(r'\d+\.\d+\.(\d+)', ver).group(1))
     if p: r = r - 1
     return r
 
 def pre(ver):
+    """returns rc5"""
     try: return re.match(r'\d+\.\d+\.\d+(\.\d+)?-((rc|pre)\d+)', ver).group(2)
     except: return None
 
+def next_pre(ver):
+    s = pre(ver)
+    i = int(re.match(r'rc(\d+)', s).group(1))
+    return "rc%d" % (i+1)
+
 def post(ver):
+    """given 2.6.27.1 returns 1"""
     try: return re.match(r'\d+\.\d+\.\d+\.(\d+)', ver).group(1)
     except: return None
 
@@ -132,9 +141,15 @@ def prenum(ver):
     except: return None
 
 def prebase(ver):
+    """returns 2.6.13-rc1"""
     return re.match(r'(\d+\.\d+\.\d+((-(rc|pre)|\.)\d+)?)', ver).group(1)
 
+def preincr(ver):
+    """only use when incremental patches are requested"""
+    return re.match(r'(\d+\.\d+\.\d+((-(rc|pre)|\.).+)?)', ver).group(1)
+
 def revbase(ver):
+    """returns 2.6.23 for 2.6.23.15 or 2.6.24-rc5"""
     return "%s.%s" % (tree(ver), rev(ver))
 
 def base(ver):
@@ -283,8 +298,12 @@ def find_info(ver):
     f = forkname(ver)
     p = pre(ver)
 
+    print "find_info (ver) ", ver, "f=", f
+
     s = b
-    if f:
+    if re.match(".*rc.*rc.*", ver):
+        s = "%s-incrc" %b
+    elif f:
         s = "%s-%s" % (b, f)
     elif p:
         s = "%s-pre" % b
@@ -297,11 +316,14 @@ def version_urls(ver):
     if type(i) != type([]):
         i = [i]
 
+    print "version urls i = ", i    
+
     v = {
         'full': ver,
         'tree': tree(ver),
         'base': base(ver),
-        'prebase': prebase(ver)
+        'prebase': prebase(ver),
+        'preincr': preincr(ver),
         }
 
     l = []
@@ -399,6 +421,7 @@ def get_patch(ver):
 def apply_patch(ver, reverse = 0):
     """Find the patch to upgrade from the predecessor of ver to ver and
     apply or reverse it."""
+    print 'apply patch ', ver, ' reverse = ', reverse
     p = get_patch(ver)
     r = ""
     if reverse:
@@ -501,6 +524,15 @@ def find_ver(ver):
     else:
         return ver
 
+def remove_pre(a):
+    print 'remove -pre ' + a
+    apply_patch(a, 1)
+
+def goto_pre(a):
+    print 'goto -pre ' + a
+    apply_patch(base(a)+'-rc1', 0)
+    return base(a)+'-rc1'
+
 def transform(a, b):
     if a == b:
         qprint("Nothing to do!")
@@ -514,9 +546,16 @@ def transform(a, b):
     if fork(a):
         apply_patch(a, 1)
         a = prebase(a)
+
+    if base(a) == base(b):
+        if pre(a) and pre(b) and pre(a) < pre(b):
+            print a, ' to ', b, ' possible small steps'
+
     if prebase(a) != prebase(b):
+        print 'prebase ', a, ' is not prebase ', b
         if pre(a):
-            apply_patch(a, 1)
+            print 'pre (a)', pre(a)
+            remove_pre(a)
             a = base(a)
 
         if post(a) and (post(a) != post(b) or rev(a) != rev(b)):
@@ -536,8 +575,21 @@ def transform(a, b):
             a = base(b)
 
         if pre(b):
-            apply_patch(prebase(b))
-            a = prebase(b)
+            print "Now at ", a, " should go to ", b
+            rb = rev(a)
+            rc1base = "%s.%s" % (t, rb+1)
+            print "... ", rc1base
+            a = goto_pre(rc1base)
+
+            print "Now at ", a
+            
+            while pre(a) < pre(b):
+                s = ("%s-%s-%s" % (rc1base, pre(a), next_pre(a)))
+                print "Applying ", s
+                apply_patch(s, 0)
+                a = "%s-%s" % (rc1base, next_pre(a))
+                print "Should be at ", a
+            
 
     if fork(b):
         a = apply_patch(b)
@@ -573,6 +625,10 @@ version_info = {
             kernel_url + "/v2.6" + "/patch-%(prebase)s.bz2",
             r'patch-(.*?).bz2',
             1, "current stable kernel series"),
+    '2.6-incrc': (latest_dir,
+                kernel_url + "/v2.6" + "/testing/incr/patch-%(preincr)s.bz2",
+                r'patch-(.*?).bz2',
+                1, "current stable kernel series prereleases -- incremental"),
     '2.6-rc': (latest_dir,
                 kernel_url + "/v2.6" + "/testing/patch-%(prebase)s.bz2",
                 r'patch-(.*?).bz2',
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#40 lore raw

Re: ketchup was Re: [kernel.org users] XZ Migration discussion

In reply to Pavel Machek
Quoted 6 lines · Pavel Machek
Quoted 2 lines · Pavel Machek

Add --only-dl option -- when you want to cache the needed patches but not apply anything yet.

Actually, you'll probably get a better patch if you relace 'only-dl' with just 'dl'. Mistaking --only-dl and --dl-only is just too easy.

This should work; save time&bandwidth by using incremental patches between -rcX.

Is there testsuite for ketchup? It would be useful at this point...

Pavel

ketchup renamed from ketchup.orig +76 -9
@@ -107,19 +107,32 @@
 # Functions to parse version strings
 
 def tree(ver):
+    """returns 2.6"""
     return float(re.match(r'(\d+\.\d+)', ver).group(1))
 
+def rawrev(ver):
+    """given 2.6.31 or 2.6.31-rc1 returns 31"""
+    return int(re.match(r'\d+\.\d+\.(\d+)', ver).group(1))
+
 def rev(ver):
+    """given 2.6.31 or 2.6.32-rc1 returns 31"""
     p = pre(ver)
-    r = int(re.match(r'\d+\.\d+\.(\d+)', ver).group(1))
+    r = rawrev(ver)
     if p: r = r - 1
     return r
 
 def pre(ver):
+    """returns rc5"""
     try: return re.match(r'\d+\.\d+\.\d+(\.\d+)?-((rc|pre)\d+)', ver).group(2)
     except: return None
 
+def next_pre(ver, inc = 1):
+    s = pre(ver)
+    i = int(re.match(r'rc(\d+)', s).group(1))
+    return "rc%d" % (i+inc)
+
 def post(ver):
+    """given 2.6.27.1 returns 1"""
     try: return re.match(r'\d+\.\d+\.\d+\.(\d+)', ver).group(1)
     except: return None
 
@@ -132,9 +145,15 @@
     except: return None
 
 def prebase(ver):
+    """returns 2.6.13-rc1"""
     return re.match(r'(\d+\.\d+\.\d+((-(rc|pre)|\.)\d+)?)', ver).group(1)
 
+def preincr(ver):
+    """only use when incremental patches are requested"""
+    return re.match(r'(\d+\.\d+\.\d+((-(rc|pre)|\.).+)?)', ver).group(1)
+
 def revbase(ver):
+    """returns 2.6.23 for 2.6.23.15 or 2.6.24-rc5"""
     return "%s.%s" % (tree(ver), rev(ver))
 
 def base(ver):
@@ -283,8 +302,12 @@
     f = forkname(ver)
     p = pre(ver)
 
+    print "find_info (ver) ", ver, "f=", f
+
     s = b
-    if f:
+    if re.match(".*rc.*rc.*", ver):
+        s = "%s-incrc" %b
+    elif f:
         s = "%s-%s" % (b, f)
     elif p:
         s = "%s-pre" % b
@@ -297,11 +320,14 @@
     if type(i) != type([]):
         i = [i]
 
+    print "version urls i = ", i    
+
     v = {
         'full': ver,
         'tree': tree(ver),
         'base': base(ver),
-        'prebase': prebase(ver)
+        'prebase': prebase(ver),
+        'preincr': preincr(ver),
         }
 
     l = []
@@ -399,6 +425,7 @@
 def apply_patch(ver, reverse = 0):
     """Find the patch to upgrade from the predecessor of ver to ver and
     apply or reverse it."""
+    print 'apply patch ', ver, ' reverse = ', reverse
     p = get_patch(ver)
     r = ""
     if reverse:
@@ -501,6 +528,24 @@
     else:
         return ver
 
+def step_rc(a, b):
+    rc1base = "%s.%s" % (tree(b), rawrev(b))    
+    print 'stepping from ', a, ' to ', b 
+    while pre(a) != pre(b):
+        if pre(a) < pre(b):
+            next = next_pre(a, 1)
+            s = ("%s-%s-%s" % (rc1base, pre(a), next))
+            apply_patch(s)
+        else:
+            next = next_pre(a, -1)
+            s = ("%s-%s-%s" % (rc1base, next, pre(a)))
+            apply_patch(s, 1)
+
+        print "Applying ", s
+
+        a = "%s-%s" % (rc1base, next)
+        print "Should be at ", a
+
 def transform(a, b):
     if a == b:
         qprint("Nothing to do!")
@@ -514,10 +559,25 @@
     if fork(a):
         apply_patch(a, 1)
         a = prebase(a)
+
+    if base(a) == base(b):
+        if pre(a) and pre(b):
+            print a, ' to ', b, ' possible small steps'
+            step_rc(prebase(a), prebase(b))
+            a = prebase(b)
+
     if prebase(a) != prebase(b):
-        if pre(a):
-            apply_patch(a, 1)
-            a = base(a)
+        print 'prebase ', a, ' is not prebase ', b
+        if (rev(a) != rev(b)) and pre(a):
+            print 'stepping to release, first'
+            rc1base = "%s.%s" % (tree(a), rawrev(a))
+            rc1 = rc1base + '-rc1'
+            print "Base is ", rc1base, "now at ", a, " should go to ", rc1
+            step_rc(a, rc1)
+            print "Should now be on -rc1"
+            apply_patch(rc1, 1)
+            a = base(rc1)
+            print "Should now be at", a
 
         if post(a) and (post(a) != post(b) or rev(a) != rev(b)):
             apply_patch(prebase(a), 1)
@@ -536,9 +596,12 @@
             a = base(b)
 
         if pre(b):
-            apply_patch(prebase(b))
-            a = prebase(b)
-
+            rc1base = "%s.%s" % (tree(b), rawrev(b))
+            rc1 = rc1base + '-rc1'
+            print "Base is ", rc1base, "now at ", a, " should go to ", b
+            apply_patch(rc1)
+            step_rc(rc1, b)
+            
     if fork(b):
         a = apply_patch(b)
 
@@ -573,6 +636,10 @@
             kernel_url + "/v2.6" + "/patch-%(prebase)s.bz2",
             r'patch-(.*?).bz2',
             1, "current stable kernel series"),
+    '2.6-incrc': (latest_dir,
+                kernel_url + "/v2.6" + "/testing/incr/patch-%(preincr)s.bz2",
+                r'patch-(.*?).bz2',
+                1, "current stable kernel series prereleases -- incremental"),
     '2.6-rc': (latest_dir,
                 kernel_url + "/v2.6" + "/testing/patch-%(prebase)s.bz2",
                 r'patch-(.*?).bz2',
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#41 lore raw

Re: ketchup was Re: [kernel.org users] XZ Migration discussion

Quoted 5 lines · Pavel Machek
Quoted 4 lines · Pavel Machek
Quoted 2 lines · Pavel Machek

Add --only-dl option -- when you want to cache the needed patches but not apply anything yet.

Quoted 7 lines
Quoted 2 lines

Actually, you'll probably get a better patch if you relace 'only-dl' with just 'dl'. Mistaking --only-dl and --dl-only is just too easy.

This should work; save time&bandwidth by using incremental patches between -rcX.

Is there testsuite for ketchup? It would be useful at this point...

I don't have one, but that would be helpful.

Would you like to take over maintaining ketchup? I took it over because Matt did not want it anymore and I was the last to send him patches.

I've been quite busy working on Ftrace, trace-cmd and kconfig stuff, that I'm willing to hand this over to you, if you want it. My last release of ketchup is my git tree on kernel.org.

-- Steve

#42 lore raw

Re: ketchup was Re: [kernel.org users] XZ Migration discussion

Hi!

Quoted 11 lines
Quoted 10 lines
Quoted 5 lines
Quoted 2 lines

Add --only-dl option -- when you want to cache the needed patches but not apply anything yet.

Actually, you'll probably get a better patch if you relace 'only-dl' with just 'dl'. Mistaking --only-dl and --dl-only is just too easy.

This should work; save time&bandwidth by using incremental patches between -rcX.

Is there testsuite for ketchup? It would be useful at this point...

Quoted 8 lines

I don't have one, but that would be helpful.

Would you like to take over maintaining ketchup? I took it over because Matt did not want it anymore and I was the last to send him patches.

I've been quite busy working on Ftrace, trace-cmd and kconfig stuff, that I'm willing to hand this over to you, if you want it. My last release of ketchup is my git tree on kernel.org.

I'd really prefer not to maintain it -- I'm using it on my zaurus, because git is just too slow there -- but I don't really have time just now.

(Plus, I'm not good enough with git, I'm afraid).

But, if you think the patches are good enough, I can just prepare nice, easy to apply series? (But I'd really like someone to check that the basic idea is right...)

Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#43 lore raw
In reply to Jean Delvare
Quoted 6 lines · Jean Delvare

Maybe that's just me, but my main concern is neither download times nor decompression times. My main concern is the access time to directory indexes when browsing the kernel archive, because there are 5 entries for every patch or tarball: .bz2, .bz2.sign, .gz, .gz.sign and .sign. This is horribly slow.

This was actually the main reason for me personally to ask about just dropping support for .gz files - not because I care deeply about how much disk space kernel.org wastes, but because the long directory listings make it slower for me to mentally index the directory.

Quoted 8 lines

1* Keep a single compression format. This saves almost 40% of the files.

2* Move one of the compression formats somewhere else, so that it doesn't get in the way but is still available if needed.

3* Create a new subdirectory for every 2.6.x kernel, and move all the related files there.

I did 3* for the testing kernels (exactly because the directory listing got to be unreadable), and you just complained about it ;)

Of course, your complaint was that the subdirectory wasn't done immediately, and that the old files get moved to their own subdirectory later as a "archival" thing.

I just didn't want to change the location for the latest kernel.

Quoted 2 lines

4* Get rid of the LATEST-IS-* files. This is a small count, won't save much, but these files seem totally useless to me these days.

Yeah, they also end up continually being stale.

Linus

author
#44 lore raw
Quoted 10 lines · Linus Torvalds
Quoted 6 lines · Jean Delvare

Maybe that's just me, but my main concern is neither download times nor decompression times. My main concern is the access time to directory indexes when browsing the kernel archive, because there are 5 entries for every patch or tarball: .bz2, .bz2.sign, .gz, .gz.sign and .sign. This is horribly slow.

Quoted 4 lines

This was actually the main reason for me personally to ask about just dropping support for .gz files - not because I care deeply about how much disk space kernel.org wastes, but because the long directory listings make it slower for me to mentally index the directory.

It's probably worth keeping things like the .gz files around, if nothing else for older distros, systems, etc that don't have xz yet (since it's still relatively new)

Breaking things out into directories or such might be the easiest way with that

v2.6/
v2.6/2.6.23
v2.6/2.6.32.6

etc

Would clean up the v2.6 directory a lot.

Quoted 16 lines
Quoted 8 lines

1* Keep a single compression format. This saves almost 40% of the files.

2* Move one of the compression formats somewhere else, so that it doesn't get in the way but is still available if needed.

3* Create a new subdirectory for every 2.6.x kernel, and move all the related files there.

I did 3* for the testing kernels (exactly because the directory listing got to be unreadable), and you just complained about it ;)

Of course, your complaint was that the subdirectory wasn't done immediately, and that the old files get moved to their own subdirectory later as a "archival" thing.

Quoted 6 lines

I just didn't want to change the location for the latest kernel.

Quoted 2 lines

4* Get rid of the LATEST-IS-* files. This is a small count, won't save much, but these files seem totally useless to me these days.

Yeah, they also end up continually being stale.

Only thoughts there are that there seem to be a lot of automated processes that rely on LATEST-IS-*. Personally I'd rather see them snag the RSS feed and figure out what they want from there, but that may not be completely feasible. It also gives a quick indication as to what's latest in the directory and a quick search on the page usually gets me what I'm looking for when I do it.

- John 'Warthog9' Hawley

#45 lore raw
Quoted 11 lines · J.H.
Quoted 10 lines · Linus Torvalds
Quoted 6 lines · Jean Delvare

Maybe that's just me, but my main concern is neither download times nor decompression times. My main concern is the access time to directory indexes when browsing the kernel archive, because there are 5 entries for every patch or tarball: .bz2, .bz2.sign, .gz, .gz.sign and .sign. This is horribly slow.

Quoted 8 lines
Quoted 4 lines

This was actually the main reason for me personally to ask about just dropping support for .gz files - not because I care deeply about how much disk space kernel.org wastes, but because the long directory listings make it slower for me to mentally index the directory.

It's probably worth keeping things like the .gz files around, if nothing else for older distros, systems, etc that don't have xz yet (since it's still relatively new)

Hardly a good reason IMHO. xz can be installed on these systems. When we switched to git, nobody had it and it did not stop us.

Quoted 6 lines

Breaking things out into directories or such might be the easiest way with that

v2.6/
v2.6/2.6.23

Yes.

Quoted 1 line

v2.6/2.6.32.6

I'd rather group all 2.6.32.* files together, so that the main index is as small as possible. We're adding one indirection step, so it should be fast.

Quoted 5 lines

etc

Would clean up the v2.6 directory a lot.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

(...)
4* Get rid of the LATEST-IS-* files. This is a small count, won't save much, but these files seem totally useless to me these days.

Yeah, they also end up continually being stale.

Only thoughts there are that there seem to be a lot of automated processes that rely on LATEST-IS-*.

Care to give details? Given how often the old files get stuck, I am surprised any process can really rely on them. And I also can't really think of any automated process that would care.

Quoted 3 lines

Personally I'd rather see them snag
the RSS feed and figure out what they want from there, but that may not be completely feasible.

There's RSS, there's a mailing list and there's a web page. Certainly one of these 3 methods would work.

Quoted 2 lines

It also gives a quick indication as to what's
latest in the directory

Sorting by time works just as well.

Quoted 2 lines

and a quick search on the page usually gets me
what I'm looking for when I do it.

What's your workflow? I normally go to the download directory after either reading the kernel.org main page or some post on the announce mailing list. So I already know which version I am looking for. Having to search for "LATEST-IS" and then again for the version doesn't look terribly efficient.

If we really want a helper to locate the latest version, I'd rather have a "latest" symbolic link pointing to the most recent v2.6.x subdirectory. Or maybe two, "latest-stable" and "latest-devel". Can be discussed...

Signature · 1 line
Jean Delvare
#46 lore raw
Quoted 6 lines · Jean Delvare
Quoted 3 lines

It's probably worth keeping things like the .gz files around, if nothing else for older distros, systems, etc that don't have xz yet (since it's still relatively new)

Hardly a good reason IMHO. xz can be installed on these systems. When we switched to git, nobody had it and it did not stop us.

I don't agree, it's different. Git is only used by developers, and even not all of them. Sources are a reference. Anyone can download them to look for anything. Switching to a specific format which really is not common at all on older distros nor on any system looks a bit like proprietary encoding eventhough it's not the case. But it's a way to tell people that if they want the sources in clear text form, they first have to find a tool capable of decompressing them. Gzip is well defined as a standard, it's even described in an RFC and is present on almost any system (unix or not) now. Any student who wants to take a look at the kernel will have access to gunzip, even from an old Solaris 8 workstation or a Windows XP desktop PC. XZ if far from being there, and the student will not necessarily be able to install it. And Peter raised some valid points about the hardware requirements to run such tools ; I'm not sure the guys running Linux on their old Sparc-2 would like XZ only a lot.

Regards,
Willy

#47 lore raw

Hi!

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

It's probably worth keeping things like the .gz files around, if nothing else for older distros, systems, etc that don't have xz yet (since it's still relatively new)

Hardly a good reason IMHO. xz can be installed on these systems. When we switched to git, nobody had it and it did not stop us.

Quoted 15 lines

I don't agree, it's different. Git is only used by developers, and even not all of them. Sources are a reference. Anyone can download them to look for anything. Switching to a specific format which really is not common at all on older distros nor on any system looks a bit like proprietary encoding eventhough it's not the case. But it's a way to tell people that if they want the sources in clear text form, they first have to find a tool capable of decompressing them. Gzip is well defined as a standard, it's even described in an RFC and is present on almost any system (unix or not) now. Any student who wants to take a look at the kernel will have access to gunzip, even from an old Solaris 8 workstation or a Windows XP desktop PC. XZ if far from being there, and the student will not necessarily be able to install it. And Peter raised some valid points about the hardware requirements to run such tools ; I'm not sure the guys running Linux on their old Sparc-2 would like XZ only a lot.

It is not just student on old workstation. I'm trying to keep Linux working on spitz PDA and kohjinsha subnotebook. Especially zaurus has about power of old sparc two... otoh it runs 4 hours on cellphone battery.
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#48 lore raw
Quoted 7 lines · Pavel Machek
Quoted 3 lines

And Peter raised some valid points about the hardware requirements to run such tools ; I'm not sure the guys running Linux on their old Sparc-2 would like XZ only a lot.

It is not just student on old workstation. I'm trying to keep Linux working on spitz PDA and kohjinsha subnotebook. Especially zaurus has about power of old sparc two...

Do you actually require to download and unpack (and configure, build) the kernel sources from kernel.org to the PDA directly? As Jean wrote, how does unpack time matter compared to build time?

Signature · 3 lines
Stefan Richter
-=====-==-=- --=- -==-=
http://arcgraph.de/sr/
#49 lore raw
Quoted 11 lines · Stefan Richter
Quoted 6 lines · Pavel Machek
Quoted 3 lines

And Peter raised some valid points about the hardware requirements to run such tools ; I'm not sure the guys running Linux on their old Sparc-2 would like XZ only a lot.

It is not just student on old workstation. I'm trying to keep Linux working on spitz PDA and kohjinsha subnotebook. Especially zaurus has about power of old sparc two...

Do you actually require to download and unpack (and configure, build) the kernel sources from kernel.org to the PDA directly? As Jean wrote, how does unpack time matter compared to build time?

PS: It boils down to CPU time requirements. For small memory systems, there is the XZ Embedded decompressor.

Signature · 3 lines
Stefan Richter
-=====-==-=- --=- -==-=
http://arcgraph.de/sr/
#50 lore raw
Quoted 2 lines · On 2010-02-13 Stefan Richter wrote:

PS: It boils down to CPU time requirements. For small memory systems, there is the XZ Embedded decompressor.

XZ Embedded saves only code size. If the file was compressed with settings that make the decompressor require e.g. 65 MiB RAM (xz -9), using XZ Embedded won't help you.

XZ Embedded is also very limited. It cannot decompress all .xz files. It should still be quite useful, since on some architectures it can compress the kernel image better than the current LZMA support in the kernel, and the API should be nice to use e.g. by Squashfs.

Signature · 1 line
Lasse Collin  |  IRC: Larhzu @ IRCnet & Freenode
#51 lore raw
In reply to Willy Tarreau

Salut Willy,

Quoted 8 lines · Willy Tarreau
Quoted 6 lines · Jean Delvare
Quoted 3 lines

It's probably worth keeping things like the .gz files around, if nothing else for older distros, systems, etc that don't have xz yet (since it's still relatively new)

Hardly a good reason IMHO. xz can be installed on these systems. When we switched to git, nobody had it and it did not stop us.

Quoted 7 lines

I don't agree, it's different. Git is only used by developers, and even not all of them. Sources are a reference. Anyone can download them to look for anything. Switching to a specific format which really is not common at all on older distros nor on any system looks a bit like proprietary encoding eventhough it's not the case. But it's a way to tell people that if they want the sources in clear text form, they first have to find a tool capable of decompressing them.

Just like switching to git was a way to tell people that if they wanted to contribute to the kernel, they first had to install the right, at the time uncommon tool. Of course the audience isn't the same, but the principle is similar. And the audience is still fairly limited in both cases. My parents aren't downloading kernel tarballs. I would assume that anyone willing and able to download, read and understand the linux kernel source code wouldn't be frightened by having to install a small tool to extract these sources. And they may not even have to do this:
sources can be read with just a web browser: we have the gitweb interface, and several public LXR installations are deployed as well. These days, web browsers are much more popular than ftp clients and tarballs.

As a matter of fact, I am advocating the use of xz while I don't have it installed on most of my machines. I really don't see this as a blocker.

Quoted 5 lines

Gzip is well
defined as a standard, it's even described in an RFC and is present on almost any system (unix or not) now. Any student who wants to take a look at the kernel will have access to gunzip, even from an old Solaris 8 workstation or a Windows XP desktop PC.

Really? I have a Windows XP laptop at hand and it can't read .gz files. If I ask it to try, it tells me I should install WinZip. I also seem to recall that I had to install GNU gzip myself back when I was working on a Solaris workstation (but I might remember badly.)

Quoted 5 lines

XZ if far from
being there, and the student will not necessarily be able to install it. And Peter raised some valid points about the hardware requirements to run such tools ; I'm not sure the guys running Linux on their old Sparc-2 would like XZ only a lot.

I don't quite buy this argument either. I suspect this is a very limited count of users, and these users have access to other, more powerful machines where they can easily achieve any format conversion they need.

I have an old, slow machine here which I am going to use to perform some real world testing, and I'll post the results when I'm done. But I suspect that building a kernel on this machine, even a small one with just the drivers it needs, will take much longer than unpacking the sources. So anyone worrying about performance would rather rely on cross-compilation, and in turn can afford whatever decompression tool is needed.

Signature · 1 line
Jean Delvare
#52 lore raw
Quoted 11 lines · Jean Delvare
Quoted 5 lines · Willy Tarreau

Gzip is well
defined as a standard, it's even described in an RFC and is present on almost any system (unix or not) now. Any student who wants to take a look at the kernel will have access to gunzip, even from an old Solaris 8 workstation or a Windows XP desktop PC.

Really? I have a Windows XP laptop at hand and it can't read .gz files. If I ask it to try, it tells me I should install WinZip. I also seem to recall that I had to install GNU gzip myself back when I was working on a Solaris workstation (but I might remember badly.)

As a side note: 7zip, a very popular and Free archiving tool for Windows, supports xz since its version 9 which is currently available as a beta. So, WRT the need to get an extra unarchiver, xz is just as accessible to Windows users as gz is.

Signature · 3 lines
Stefan Richter
-=====-==-=- --=- -==-=
http://arcgraph.de/sr/
#53 lore raw
Quoted 15 lines · Stefan Richter
Quoted 8 lines · Jean Delvare
Quoted 6 lines · Willy Tarreau

Gzip is well
defined as a standard, it's even described in an RFC and is present on almost any system (unix or not) now. Any student who wants to take a look at the kernel will have access to gunzip, even from an old Solaris 8 workstation or a Windows XP desktop PC.

Quoted 5 lines

Really? I have a Windows XP laptop at hand and it can't read .gz files. If I ask it to try, it tells me I should install WinZip. I also seem to recall that I had to install GNU gzip myself back when I was working on a Solaris workstation (but I might remember badly.)

Quoted 5 lines

As a side note: 7zip, a very popular and Free archiving tool for Windows, supports xz since its version 9 which is currently available as a beta. So, WRT the need to get an extra unarchiver, xz is just as accessible to Windows users as gz is.

Can you even unpack a kernel tree on Windows? There are some files (e.g. net/ipv4/netfilter/ipt_{ecn,ECN}.c) which conflict on a case-preserving filesystem.

Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
#54 lore raw
Quoted 2 lines · Avi Kivity
Quoted 1 line · Stefan Richter

Jean Delvare wrote:

...snip...

Quoted 6 lines
Quoted 4 lines

As a side note: 7zip, a very popular and Free archiving tool for Windows, supports xz since its version 9 which is currently available as a beta. So, WRT the need to get an extra unarchiver, xz is just as accessible to Windows users as gz is.

Can you even unpack a kernel tree on Windows? There are some files (e.g. net/ipv4/netfilter/ipt_{ecn,ECN}.c) which conflict on a case-preserving filesystem.

What filesystem version are you talking about ? ntfs as of recent windows is not case insensitive . What do you mean by , "a case-preserving filesystem" ?

Now whether or not the tools available are case insensitive is another
matter .

Hth , JimL

Signature · 5 lines
+------------------------------------------------------------------+
| James   W.   Laferriere | System    Techniques | Give me VMS     |
| Network&System Engineer | 3237     Holden Road |  Give me Linux  |
| [email protected] | Fairbanks, AK. 99709 |   only  on  AXP |
+------------------------------------------------------------------+
#55 lore raw
Quoted 1 line · Mr. James W. Laferriere

On Sat, 13 Feb 2010, Avi Kivity wrote:

[...]

Quoted 1 line
Quoted 1 line

Can you even unpack a kernel tree on Windows? There are some files (e.g.

Given a sane filesystem, yes. If (the) native filesystem(s) are sane, is more a flamebait than anything else.

Quoted 3 lines
Quoted 2 lines

net/ipv4/netfilter/ipt_{ecn,ECN}.c) which conflict on a case-preserving filesystem.

What filesystem version are you talking about ?

All of NTFS (and basically FAT*). Are there are different versions? If yes, how do I tell?

Quoted 1 line

ntfs as of recent windows is not case insensitive .

It *is* case insensitive (as it compares filesystem names case-insensitive on the equivalent of an open() syscall).

Quoted 1 line

What do you mean by , "a case-preserving filesystem" ?

Apart from "please google that, thank you": The filesystem (driver)) preserves the case of a filename if you create it (TTBOMK). But it ignores the case of the filename for other file operations like open(), stat(), etc.

Quoted 2 lines

Now whether or not the tools available are case insensitive is another
matter .

ACK. But if you have a case-insensitive or case-preserving filesystem, the tools can´t really do that much.
And given the "understanding" of the monopolist (as such) in that area, most of the "tools" are actually inherent part of the OS (and not just another 3rd party app). SCNR ....

Bernd

Signature · 2 lines
Bernd Petrovitsch                  Email : [email protected]
                     LUGA : http://www.luga.at
#56 lore raw
Quoted 9 lines · Bernd Petrovitsch
Quoted 1 line · Mr. James W. Laferriere

On Sat, 13 Feb 2010, Avi Kivity wrote:

[...]

Quoted 1 line
Quoted 1 line

Can you even unpack a kernel tree on Windows?  There are some files (e.g.

Given a sane filesystem, yes. If (the) native filesystem(s) are sane, is more a flamebait than anything else.

Quoted 2 lines
Quoted 2 lines

net/ipv4/netfilter/ipt_{ecn,ECN}.c) which conflict on a case-preserving filesystem.

Quoted 7 lines
Quoted 1 line

      What filesystem version are you talking about ?

All of NTFS (and basically FAT*). Are there are different versions? If yes, how do I tell?

Quoted 1 line

      ntfs as of recent windows is not case insensitive .

It *is* case insensitive (as it compares filesystem names case-insensitive on the equivalent of an open() syscall).

There are case sensitive Windows subsystems (e.g. SFU/SUA at least when running over NTFS), and the default behavior for Win32 apps even can be changed to be case sensitive via a registry key: ObCaseInsensitive).

More important is how easy it is to install - since XZ is not even available via apt-get install on recent distros (e.g. April 2009 Ubuntu 9.04), this discussion seems about a year premature.

Signature · 3 lines
Thanks,

Steve
#57 lore raw
Quoted 2 lines · Steve French

On Sat, Feb 13, 2010 at 4:39 PM, Bernd Petrovitsch <[email protected]> wrote:

[...]

Quoted 9 lines
Quoted 3 lines
Quoted 1 line

ntfs as of recent windows is not case insensitive .

It *is* case insensitive (as it compares filesystem names case-insensitive on the equivalent of an open() syscall).

There are case sensitive Windows subsystems (e.g. SFU/SUA at least when running over NTFS), and the default behavior for

Win32 apps even can be changed to be case sensitive via a registry key: ObCaseInsensitive).

It's somewhat - ähemm - strange IMHO that the casing is an app-specific feature (and not filesystem specific).
And it is really that implemented that the app can choose in what way the filesystem below - given that it supports that feature - compares two filenames?

Quoted 4 lines

More important is how easy it is to install - since XZ is not even available via apt-get install on recent distros (e.g. April 2009 Ubuntu 9.04), this discussion seems about a year premature.

It's in recent Fedoras so Ubuntu is perhaps just late. And FWIW, it' 2 packages too as the second one is the .lzma support (so probably Debian should be able to fix the clash with whatever the current lzma package is).

Bernd

Signature · 2 lines
Bernd Petrovitsch                  Email : [email protected]
                     LUGA : http://www.luga.at
author
#58 lore raw
Quoted 23 lines · Bernd Petrovitsch
Quoted 2 lines · Steve French

On Sat, Feb 13, 2010 at 4:39 PM, Bernd Petrovitsch <[email protected]> wrote:

[...]

Quoted 9 lines
Quoted 3 lines
Quoted 1 line

ntfs as of recent windows is not case insensitive .

It *is* case insensitive (as it compares filesystem names case-insensitive on the equivalent of an open() syscall).

There are case sensitive Windows subsystems (e.g. SFU/SUA at least when running over NTFS), and the default behavior for

Win32 apps even can be changed to be case sensitive via a registry key: ObCaseInsensitive).

It's somewhat - ähemm - strange IMHO that the casing is an app-specific feature (and not filesystem specific).
And it is really that implemented that the app can choose in what way the filesystem below - given that it supports that feature - compares two filenames?

Quoted 4 lines

More important is how easy it is to install - since XZ is not even available via apt-get install on recent distros (e.g. April 2009 Ubuntu 9.04), this discussion seems about a year premature.

Quoted 6 lines

It's in recent Fedoras so Ubuntu is perhaps just late. And FWIW, it' 2 packages too as the second one is the .lzma support (so probably Debian should be able to fix the clash with whatever the current lzma package is).

Bernd

There's a package on Fedora called 'xz' that provides it on my Fedora 11 & 12 boxes.

- John 'Warthog9' Hawley

#59 lore raw
In reply to Stefan Richter
Quoted 13 lines · Stefan Richter
Quoted 11 lines · Jean Delvare
Quoted 5 lines · Willy Tarreau

Gzip is well
defined as a standard, it's even described in an RFC and is present on almost any system (unix or not) now. Any student who wants to take a look at the kernel will have access to gunzip, even from an old Solaris 8 workstation or a Windows XP desktop PC.

Really? I have a Windows XP laptop at hand and it can't read .gz files. If I ask it to try, it tells me I should install WinZip. I also seem to recall that I had to install GNU gzip myself back when I was working on a Solaris workstation (but I might remember badly.)

Quoted 4 lines

As a side note: 7zip, a very popular and Free archiving tool for Windows, supports xz since its version 9 which is currently available as a beta. So, WRT the need to get an extra unarchiver, xz is just as accessible to Windows users as gz is.

Wow, what a stretch.

There's about milion tools supporting gz for Windows, and many of them are out of beta... xz is not even supported on my zaurus, and I'll need to write some emails to get it installed on school servers...

Anyway, gz+xz should be reasonable combination.
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#60 lore raw
Quoted 10 lines · Pavel Machek
Quoted 4 lines · Stefan Richter

As a side note: 7zip, a very popular and Free archiving tool for Windows, supports xz since its version 9 which is currently available as a beta. So, WRT the need to get an extra unarchiver, xz is just as accessible to Windows users as gz is.

Wow, what a stretch.

There's about milion tools supporting gz for Windows, and many of them are out of beta...

And roughly half of this million archiving/ compression tools use 7zip's backend library, which is just one way of getting xz support in a Windows application. Besides, what one project calls beta is called dot-zero release elsewhere. :-) Seriously, getting xz support on Windows is just as easy or as difficult as getting gz support.

Signature · 3 lines
Stefan Richter
-=====-==-=- --=- -===-
http://arcgraph.de/sr/
#61 lore raw
Quoted 9 lines · Stefan Richter
Quoted 8 lines · Pavel Machek
Quoted 4 lines · Stefan Richter

As a side note: 7zip, a very popular and Free archiving tool for Windows, supports xz since its version 9 which is currently available as a beta. So, WRT the need to get an extra unarchiver, xz is just as accessible to Windows users as gz is.

Wow, what a stretch.

Quoted 8 lines
Quoted 2 lines

There's about milion tools supporting gz for Windows, and many of them are out of beta...

And roughly half of this million archiving/ compression tools use 7zip's backend library, which is just one way of getting xz support in a Windows application. Besides, what one project calls beta is called dot-zero release elsewhere. :-) Seriously, getting xz support on Windows is just as easy or as difficult as getting gz support.

Sorry, but I just don't think that's true. No more windows systems here. On android, gzip is supported, xzip is not. Debian machine supports both, but gzip was preinstalled and I had to pull xzip. (This does not help either:

root@amd:~# apt-cache search xzip
xzip - Interpreter of Infocom-format story-files

) Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#62 lore raw
Quoted 5 lines · Pavel Machek
Quoted 2 lines · Stefan Richter

Seriously, getting xz support on
Windows is just as easy or as difficult as getting gz support.

Sorry, but I just don't think that's true.

Although it is. :-)

Quoted 7 lines

No more windows systems
here. On android, gzip is supported, xzip is not. Debian machine supports both, but gzip was preinstalled and I had to pull xzip. (This does not help either:

root@amd:~# apt-cache search xzip
xzip - Interpreter of Infocom-format story-files

Considering that xz support is available even on niche systems like Amiga OS and BeOS (via p7zip if not by other means) and xz-utils proper build even on DOS, OpenVMS and other systems, how hard can it be to obtain an xz decompressor on Android, Debian, or Ubuntu¹? (¹About which there was a note somewhere else in this thread that there is a conflict between xz-utils and lzma-utils... That's basically because the former supersede the latter.)

The name confusion between xz-utils and xzip can be avoided if you search for the package in a package manager which shows package categories (archivers vs. games).

Signature · 3 lines
Stefan Richter
-=====-==-=- --=- -====
http://arcgraph.de/sr/
#63 lore raw
Quoted 1 line · Stefan Richter

Pavel Machek wrote:

[snip]

Quoted 8 lines
Quoted 7 lines

No more windows systems
here. On android, gzip is supported, xzip is not. Debian machine supports both, but gzip was preinstalled and I had to pull xzip. (This does not help either:

root@amd:~# apt-cache search xzip
xzip - Interpreter of Infocom-format story-files
Quoted 7 lines

Considering that xz support is available even on niche systems like Amiga OS and BeOS (via p7zip if not by other means) and xz-utils proper build even on DOS, OpenVMS and other systems, how hard can it be to obtain an xz decompressor on Android, Debian, or Ubuntu¹? (¹About which there was a note somewhere else in this thread that there is a conflict between xz-utils and lzma-utils... That's basically because the former supersede the latter.)

This, I think gets to one of the problems. You're telling me that the p7zip thing I installed for work is this .xz thing? And it's really all this LZMA algo? That's part of the transition problem, folks quite likely have access, easily, to a decompressor but don't know what it is (.gz->gzip, .bz2 -> bzip2, .xz -> {p7zip, p7zip-full, xz-utils, ???}).

In fact, why not .lzma? I'm assuming we're talking about the format, and xz-utils nad p7zip and others all implement the same format and it's all compatible and it's just a "how do we get there quickest / fastest" kind of thing between the utils.

Quoted 3 lines

The name confusion between xz-utils and xzip can be avoided if you search for the package in a package manager which shows package categories (archivers vs. games).

Actually, no, there's no 'xz-utils' in Debian/Lenny or Ubuntu 9.04, but there is in 9.10. But in 9.10, searching on xzip (a good, but incorrect guess) only gives that. Searching on xz shows xz-utils too. At least with apt-cache, and since the desc doesn't list xzip (since it's not xzip, it's an incorrect but not illogical guess), that wouldn't help.

Signature · 1 line
Tom Rini
#64 lore raw
Quoted 2 lines · Tom Rini

That's part of the transition problem, folks quite
likely have access, easily, to a decompressor but don't know what it is

Naturally. Information on it is quickly found online though.

Signature · 3 lines
Stefan Richter
-=====-==-=- --=- =----
http://arcgraph.de/sr/
#65 lore raw
In reply to Jean Delvare

Hi!

Quoted 3 lines

As a matter of fact, I am advocating the use of xz while I don't have it installed on most of my machines. I really don't see this as a blocker.

Eh?

Making many people around the world install uncommon tool is not something that should be done lightly.

Quoted 7 lines

I have an old, slow machine here which I am going to use to perform some real world testing, and I'll post the results when I'm done. But I suspect that building a kernel on this machine, even a small one with just the drivers it needs, will take much longer than unpacking the sources. So anyone worrying about performance would rather rely on cross-compilation, and in turn can afford whatever decompression tool is needed.

On zaurus, kernel compilation takes 4 hours. (I.e. "one night"). So that one is ... well ... done overnight.

Untar is something I normally wait for, since you need to run (interactive) oldconfig after that.
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#66 lore raw

Pavel Machek wrote:
[Jean Delvare wrote]

Quoted 8 lines
Quoted 7 lines

I have an old, slow machine here which I am going to use to perform some real world testing, and I'll post the results when I'm done. But I suspect that building a kernel on this machine, even a small one with just the drivers it needs, will take much longer than unpacking the sources. So anyone worrying about performance would rather rely on cross-compilation, and in turn can afford whatever decompression tool is needed.

Quoted 5 lines

On zaurus, kernel compilation takes 4 hours. (I.e. "one night"). So that one is ... well ... done overnight.

Untar is something I normally wait for, since you need to run (interactive) oldconfig after that.

If the download takes m minutes and unpacking and unarchiving tar.gz takes another n minutes, what difference does it make for this workflow when instead m + p minutes are spent for download + unpacking and unarchiving tar.bz2 or tar.xz?

Signature · 3 lines
Stefan Richter
-=====-==-=- --=- -===-
http://arcgraph.de/sr/
#67 lore raw
Quoted 13 lines · Stefan Richter

Pavel Machek wrote:
[Jean Delvare wrote]

Quoted 11 lines
Quoted 7 lines

I have an old, slow machine here which I am going to use to perform some real world testing, and I'll post the results when I'm done. But I suspect that building a kernel on this machine, even a small one with just the drivers it needs, will take much longer than unpacking the sources. So anyone worrying about performance would rather rely on cross-compilation, and in turn can afford whatever decompression tool is needed.

On zaurus, kernel compilation takes 4 hours. (I.e. "one night"). So that one is ... well ... done overnight.

Quoted 7 lines
Quoted 2 lines

Untar is something I normally wait for, since you need to run (interactive) oldconfig after that.

If the download takes m minutes and unpacking and unarchiving tar.gz takes another n minutes, what difference does it make for this workflow when instead m + p minutes are spent for download + unpacking and unarchiving tar.bz2 or tar.xz?

The difference is that the user is waiting for the n or m minutes, but goes off and does something else for the p minutes.

So it doesn't really matter if the compile takes 1 hour or 6 hours. As Pavel noted, this is done overnight and it doesn't matter if it gets done at 2am or 6am.

But the uncompression has a user waiting for it (to do the config step), so here it makes a big difference if it takes 6 minutes ot 8 minutes.

David Lang

#68 lore raw
Quoted 17 lines · On Sun, 14 Feb 2010 13:04:45 -0800 (PST), [email protected] wrote:
Quoted 11 lines · Stefan Richter
Quoted 5 lines · Pavel Machek

On zaurus, kernel compilation takes 4 hours. (I.e. "one night"). So that one is ... well ... done overnight.

Untar is something I normally wait for, since you need to run (interactive) oldconfig after that.

If the download takes m minutes and unpacking and unarchiving tar.gz takes another n minutes, what difference does it make for this workflow when instead m + p minutes are spent for download + unpacking and unarchiving tar.bz2 or tar.xz?

The difference is that the user is waiting for the n or m minutes, but goes off and does something else for the p minutes.

Quoted 6 lines

So it doesn't really matter if the compile takes 1 hour or 6 hours. As Pavel noted, this is done overnight and it doesn't matter if it gets done at 2am or 6am.

But the uncompression has a user waiting for it (to do the config step), so here it makes a big difference if it takes 6 minutes ot 8 minutes.

You totally missed Stefan's point. Read again.

Signature · 1 line
Jean Delvare
#69 lore raw
In reply to Pavel Machek
Quoted 11 lines · Pavel Machek

Hi!

Quoted 3 lines

As a matter of fact, I am advocating the use of xz while I don't have it installed on most of my machines. I really don't see this as a blocker.

Eh?

Making many people around the world install uncommon tool is not something that should be done lightly.

Indeed.

Let me make this straight before we waste more breath on this: we're not going to go to xz only at this time. Heck, we were getting pushback on the idea of going to bz2 only *ten years* after it was deployed.

-hpa

#70 lore raw
In reply to Pavel Machek
Quoted 10 lines · Pavel Machek

Hi!

Quoted 3 lines

As a matter of fact, I am advocating the use of xz while I don't have it installed on most of my machines. I really don't see this as a blocker.

Eh?

Making many people around the world install uncommon tool is not something that should be done lightly.

It's pretty obvious that xz will become popular quickly, at least on Linux and BSD systems, much like bz2 is today. I'm not asking people to start using ClearCase ;) xz will supersede bz2, it's only a matter of time. I see no problem in being one of the early adopters.

Quoted 10 lines
Quoted 7 lines

I have an old, slow machine here which I am going to use to perform some real world testing, and I'll post the results when I'm done. But I suspect that building a kernel on this machine, even a small one with just the drivers it needs, will take much longer than unpacking the sources. So anyone worrying about performance would rather rely on cross-compilation, and in turn can afford whatever decompression tool is needed.

On zaurus, kernel compilation takes 4 hours. (I.e. "one night"). So that one is ... well ... done overnight.

Out of curiosity, if it takes that long, why don't you use a cross-compiler?

Quoted 2 lines

Untar is something I normally wait for, since you need to run (interactive) oldconfig after that.

You'll have to wait, no matter what compression format you use (and even if you don't compress the tarball). Judging by the duration of the build on your machine, I'd estimate the decompression time to 7 minutes for gz vs. 15 minutes for bz2 maybe? I doubt you sit in front of the machine for 7 minutes waiting for tar.gz to decompress, right? So I fail to see what difference it makes. You'll just do something else for 15 minutes instead of doing something else for 7 minutes.

Anyway, as I have been saying several times already, nothing prevents you from repacking tarballs to gz before uploading it to your slow system if such is your desire. I can understand the portability argument, but the decompression time, no way.

Signature · 1 line
Jean Delvare
#71 lore raw
Quoted 9 lines · Jean Delvare
Quoted 8 lines · Pavel Machek

Hi!

Quoted 3 lines

As a matter of fact, I am advocating the use of xz while I don't have it installed on most of my machines. I really don't see this as a blocker.

Eh?

Quoted 7 lines
Quoted 2 lines

Making many people around the world install uncommon tool is not something that should be done lightly.

It's pretty obvious that xz will become popular quickly, at least on Linux and BSD systems, much like bz2 is today. I'm not asking people to start using ClearCase ;) xz will supersede bz2, it's only a matter of time. I see no problem in being one of the early adopters.

If by "quickly" you mean 'ten years', sure, maybe. Keep in mind that there are people where who are still using RHEL 3, and some of them might want to download from ftp.kernel.org. So those people who are suggesting that we replace .gz files with .xz on kernel.org are *really* smoking something good.

People who think xz are good should be working to get it installed by default into the community and then enterprise distro's, first....

- Ted

author
#72 lore raw
Quoted 24 lines · On 02/15/2010 07:15 AM, [email protected] wrote:
Quoted 9 lines · Jean Delvare
Quoted 8 lines · Pavel Machek

Hi!

Quoted 3 lines

As a matter of fact, I am advocating the use of xz while I don't have it installed on most of my machines. I really don't see this as a blocker.

Eh?

Quoted 7 lines
Quoted 2 lines

Making many people around the world install uncommon tool is not something that should be done lightly.

It's pretty obvious that xz will become popular quickly, at least on Linux and BSD systems, much like bz2 is today. I'm not asking people to start using ClearCase ;) xz will supersede bz2, it's only a matter of time. I see no problem in being one of the early adopters.

If by "quickly" you mean 'ten years', sure, maybe. Keep in mind that there are people where who are still using RHEL 3, and some of them might want to download from ftp.kernel.org. So those people who are suggesting that we replace .gz files with .xz on kernel.org are *really* smoking something good.

Quoted 4 lines

People who think xz are good should be working to get it installed by default into the community and then enterprise distro's, first....

- Ted

As a note xz is available via EPEL for Redhat Enterprice Linux 5 and anything that's derived from that. Doesn't seem to be available, directly, for 4 or lower. Not sure on Suse, and Debian's already been mentioned. Just trying to do a quick survey of what's out there already.

- John 'Warthog9' Hawley

#73 lore raw
Quoted 4 lines · J.H.

As a note xz is available via EPEL for Redhat Enterprice Linux 5 and anything that's derived from that. Doesn't seem to be available, directly, for 4 or lower. Not sure on Suse, and Debian's already been mentioned. Just trying to do a quick survey of what's out there already.

xz is available since openSUSE 11.2, not before.

Signature · 1 line
Jean Delvare
#74 lore raw
In reply to J.H.
Quoted 8 lines · J.H.
Quoted 2 lines · On 02/15/2010 07:15 AM, [email protected] wrote:

People who think xz are good should be working to get it installed by default into the community and then enterprise distro's, first....

As a note xz is available via EPEL for Redhat Enterprice Linux 5 and anything that's derived from that.  Doesn't seem to be available, directly, for 4 or lower.  Not sure on Suse, and Debian's already been mentioned.  Just trying to do a quick survey of what's out there already.

Something I noticed, too. Hopefully going to work on getting set up as a packager and get xz pushed out to EPEL EL-4 over the next couple weeks unless someone else beats me to it as I still have a handful of EL-4 machines grinding away. :-)

-Dave

#75 lore raw
In reply to Jean Delvare
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

I have an old, slow machine here which I am going to use to perform some real world testing, and I'll post the results when I'm done. But I suspect that building a kernel on this machine, even a small one with just the drivers it needs, will take much longer than unpacking the sources. So anyone worrying about performance would rather rely on cross-compilation, and in turn can afford whatever decompression tool is needed.

Quoted 5 lines
Quoted 2 lines

On zaurus, kernel compilation takes 4 hours. (I.e. "one night"). So that one is ... well ... done overnight.

Out of curiosity, if it takes that long, why don't you use a cross-compiler?

Because I hack it on the go. First build is long and ugly, but subsequent builds only take 5 minutes or so, so development is possible.

(If I crosscompiled it, I'd have to crosscompile even the subsequent builds, which is impossible -- no powerful machine nearby).

Quoted 10 lines
Quoted 2 lines

Untar is something I normally wait for, since you need to run (interactive) oldconfig after that.

You'll have to wait, no matter what compression format you use (and even if you don't compress the tarball). Judging by the duration of the build on your machine, I'd estimate the decompression time to 7 minutes for gz vs. 15 minutes for bz2 maybe? I doubt you sit in front of the machine for 7 minutes waiting for tar.gz to decompress, right? So I fail to see what difference it makes. You'll just do something else for 15 minutes instead of doing something else for 7 minutes.

Actually I do sit in front of the machine, reading mails while it decompresses.

[I'll get some numbers.]

sh-3.2$ time bzip2 -d < ~/.ketchup/l^Ginux-2.6.31.tar.bz2 > delme.tar 485.73user 137.35system 683.32 (11m23.320s) elapsed 91.18%CPU sh-3.2$ df -h^H^H^H^H^Htime cat delme.tar > /usr/src/delme.tar 0.57user 109.03system 381.13 (6m21.133s) elapsed 28.75%CPU sh-3.2$ time zca^G^Gt delme.tar > /u ^H^H ^H^H ^H^H ^H^Hdelme.tar.gz ^H^H ^H^H ^H^H
+^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H.gz > delme.tar 43.97user 106.22system 223.26 (3m43.261s) elapsed 67.27%CPU

So... gzip is actually _faster_ than uncompressed data, while bzip is twice slower. Don't know about xzip.

Anyway, gzip just makes sense. It is both smaller and faster than alternatives, and nearly as portable.

Quoted 4 lines

Anyway, as I have been saying several times already, nothing prevents you from repacking tarballs to gz before uploading it to your slow system if such is your desire. I can understand the portability argument, but the decompression time, no way.

Ok, so lets go by the portability argument.
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#76 lore raw
In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds
Quoted 6 lines · Jean Delvare

Maybe that's just me, but my main concern is neither download times nor decompression times. My main concern is the access time to directory indexes when browsing the kernel archive, because there are 5 entries for every patch or tarball: .bz2, .bz2.sign, .gz, .gz.sign and .sign. This is horribly slow.

Quoted 4 lines

This was actually the main reason for me personally to ask about just dropping support for .gz files - not because I care deeply about how much disk space kernel.org wastes, but because the long directory listings make it slower for me to mentally index the directory.

Can I feature-request that someone reduces the git.kernel.org frontpage? It's almost twice as large as the v2.6 dir index in http-delivered form, so you can already experience what it's like when v2.6/ grows bigger.

author
#77 lore raw
Quoted 10 lines · Jan Engelhardt
Quoted 8 lines · Linus Torvalds
Quoted 6 lines · Jean Delvare

Maybe that's just me, but my main concern is neither download times nor decompression times. My main concern is the access time to directory indexes when browsing the kernel archive, because there are 5 entries for every patch or tarball: .bz2, .bz2.sign, .gz, .gz.sign and .sign. This is horribly slow.

Quoted 8 lines
Quoted 4 lines

This was actually the main reason for me personally to ask about just dropping support for .gz files - not because I care deeply about how much disk space kernel.org wastes, but because the long directory listings make it slower for me to mentally index the directory.

Can I feature-request that someone reduces the git.kernel.org frontpage? It's almost twice as large as the v2.6 dir index in http-delivered form, so you can already experience what it's like when v2.6/ grows bigger.

That's a topic for a completely different thread / audience than this, lets try to keep this a bit more focused since we have the mirrors on this discussion.

- John 'Warthog9' Hawley

#78 lore raw
In reply to Jean Delvare
Quoted 20 lines · Jean Delvare

There are 3881 entries in that directory today,
and it keeps growing!

I can think of several ways to improve the situation here, some of which could be combined.

1* Keep a single compression format. This saves almost 40% of the files.

2* Move one of the compression formats somewhere else, so that it doesn't get in the way but is still available if needed.

3* Create a new subdirectory for every 2.6.x kernel, and move all the related files there. This would shrink the main index drastically, and each subdirectory would have a reasonable size (except maybe 2.6.16 and 2.6.27.) Oddly enough this has been done for the files under testing/ already, so I am curious why we don't do it for the release files (and the testing/incr/ files, while we're at it.)

Quoted 7 lines

4* Get rid of the LATEST-IS-* files. This is a small count, won't save much, but these files seem totally useless to me these days. Depending on what you want exactly, there are many versions which can be considered the latest, and there are better ways to know which they are (for example http://www.eu.kernel.org/kdist/finger_banner ). And these files tend to get stuck so you can't rely on them anyway.

5* Archive all the older 2.6.x files and move them into a separate
directory (e.g. v2.6-pre20). Moving all the pre 2.6.20 files saves 42% of the file listing.

This seems an obvious solution, what am I missing?

Thanks

Phillip

#79 lore raw
Quoted 5 lines · Phillip Lougher

5* Archive all the older 2.6.x files and move them into a separate
directory (e.g. v2.6-pre20). Moving all the pre 2.6.20 files saves 42% of the file listing.

This seems an obvious solution, what am I missing?

This is confusing, inconsistent and unstable. Confusing because 2.6-pre referred so far to the releases immediately preceding 2.6.0. Inconsistent because it requires the downloader to have preliminary knowledge about what the break point is. Unstable because, while you consider pre20 to qualify as "old" today, in 5 years you will want pre30 to qualify as "old" instead, meaning that tools such as ketchup would have to be updated once again.

I think we want to come up with a directory structure which won't change in the future.

Signature · 1 line
Jean Delvare
#80 lore raw
Quoted 9 lines · Jean Delvare
Quoted 5 lines · Phillip Lougher

5* Archive all the older 2.6.x files and move them into a separate
directory (e.g. v2.6-pre20). Moving all the pre 2.6.20 files saves 42% of the file listing.

This seems an obvious solution, what am I missing?

This is confusing, inconsistent and unstable. Confusing because 2.6-pre referred so far to the releases immediately preceding 2.6.0.

I didn't say "2.6-pre", anyway it could be called something different, like 'older-releases'.

Quoted 5 lines

Inconsistent because it requires the downloader to have preliminary knowledge about what the break point is. Unstable because, while you consider pre20 to qualify as "old" today, in 5 years you will want pre30 to qualify as "old" instead, meaning that tools such as ketchup would have to be updated once again.

You yourself said "I wouldn't worry too much about breaking the current locations. Just give some time for software authors (ketchup comes to mind) to update their code and it shouldn't be a big problem."

The major advantage with my suggestion is for the majority of users/tools interested in "recent" kernels, nothing changes at all. Your suggestions break everything for everyone.

Quoted 4 lines

I think we want to come up with a directory structure which won't change in the future.

I think trying to do that is utterly futile.

Phillip

#81 lore raw
Quoted 20 lines · Phillip Lougher
Quoted 9 lines · Jean Delvare
Quoted 5 lines · Phillip Lougher

5* Archive all the older 2.6.x files and move them into a separate
directory (e.g. v2.6-pre20). Moving all the pre 2.6.20 files saves 42% of the file listing.

This seems an obvious solution, what am I missing?

This is confusing, inconsistent and unstable. Confusing because 2.6-pre referred so far to the releases immediately preceding 2.6.0.

I didn't say "2.6-pre", anyway it could be called something different, like 'older-releases'.

Quoted 5 lines

Inconsistent because it requires the downloader to have preliminary knowledge about what the break point is. Unstable because, while you consider pre20 to qualify as "old" today, in 5 years you will want pre30 to qualify as "old" instead, meaning that tools such as ketchup would have to be updated once again.

Quoted 3 lines

You yourself said "I wouldn't worry too much about breaking the current locations. Just give some time for software authors (ketchup comes to mind) to update their code and it shouldn't be a big problem."

Yes, I did say that, and I can repeat it if needed. There's a big difference between breaking an old scheme which used to be valid and happens to no longer be, and designing a new scheme where breakages are bound to happen by design. Even if technically that's the same breakage, its reception by the affected users will be very different, because in the latter case you have no excuse.

Quoted 2 lines

The major advantage with my suggestion is for the majority of users/tools interested in "recent" kernels, nothing changes at all.

Prove it. Many people out there are still working on older trees. I am working on 2.6.5 and 2.6.16 kernels on a weekly basis. If ketchup or other tools break for these trees only and not more recent ones, that won't help me at all, I will still have to update them.

Quoted 1 line

Your suggestions break everything for everyone.

This doesn't make me happy, but at least it is consistent and durable. When you change something for everyone, it has the advantage that the solutions are general, come quickly and are widely documented. Using quirks to limit the effects is a burden for the future, and may not even help that much in practice.

Quoted 4 lines
Quoted 2 lines

I think we want to come up with a directory structure which won't change in the future.

I think trying to do that is utterly futile.

You didn't have to join this discussion in the first place.

Signature · 1 line
Jean Delvare
#82 lore raw
Quoted 6 lines · Jean Delvare
Quoted 3 lines
Quoted 2 lines

I think we want to come up with a directory structure which won't change in the future.

I think trying to do that is utterly futile.

You didn't have to join this discussion in the first place.

And who are you to tell me I shouldn't have? I was trying to be constructive, and it seems as if I've been insulted just because I dared to disagree with you. Grow up.

My comment regarding futility is that things rarely turn out as planned even with the best intentions, and it's rarely worth getting hung up about designing the perfect system for something you can't predict.

Phillip

#83 lore raw
In reply to Jean Delvare
Quoted 4 lines · Jean Delvare

Prove it. Many people out there are still working on older trees. I am working on 2.6.5 and 2.6.16 kernels on a weekly basis. If ketchup or other tools break for these trees only and not more recent ones, that won't help me at all, I will still have to update them.

Why are you still working on 2.6.5 and 2.6.16 kernels on a weekly basis? This could quite useful to know. I could be pedantic and simply turn your "prove it" around, and ask you to prove you're not an isolated case, but that wouldn't be constructive, but irritating, like you.

Phillip

#84 lore raw
Quoted 8 lines · Phillip Lougher
Quoted 4 lines · Jean Delvare

Prove it. Many people out there are still working on older trees. I am working on 2.6.5 and 2.6.16 kernels on a weekly basis. If ketchup or other tools break for these trees only and not more recent ones, that won't help me at all, I will still have to update them.

Why are you still working on 2.6.5 and 2.6.16 kernels on a weekly basis?

Because I am doing support for enterprise customers who are using distributions based on these kernel versions. These are SLE 9 and SLE 10, to name them, but RHEL supporters are in the same situation. And I've heard embedded developers report many times that they had to stick to older 2.6.x kernels too for various reasons. Not everyone is using a recent 2.6.x kernel, which makes it hard to draw a line between what should be considered old ones and what should be considered new ones.

Signature · 3 lines
Jean Delvare
"It is not necessary to hope in order to undertake, nor to succeed in
order to persevere" -- Charles The Bold
#85 lore raw
Quoted 9 lines · Jean Delvare
Quoted 7 lines · Phillip Lougher
Quoted 4 lines · Jean Delvare

Prove it. Many people out there are still working on older trees. I am working on 2.6.5 and 2.6.16 kernels on a weekly basis. If ketchup or other tools break for these trees only and not more recent ones, that won't help me at all, I will still have to update them.

Why are you still working on 2.6.5 and 2.6.16 kernels on a weekly basis?

Quoted 8 lines

Because I am doing support for enterprise customers who are using distributions based on these kernel versions. These are SLE 9 and SLE 10, to name them, but RHEL supporters are in the same situation. And I've heard embedded developers report many times that they had to stick to older 2.6.x kernels too for various reasons. Not everyone is using a recent 2.6.x kernel, which makes it hard to draw a line between what should be considered old ones and what should be considered new ones.

Embedded and enterprise distro users are usually stuck on ancient kernels that were downloaded from kernel.org and patched *years ago*. The reason they're stuck on them is due to local modifications, and so they're not going to be downloading ancient vanilla kernels from kernel.org now.

Phillip

#86 lore raw

Hi Phillip,

Quoted 4 lines · Phillip Lougher

Embedded and enterprise distro users are usually stuck on ancient kernels that were downloaded from kernel.org and patched *years ago*. The reason they're stuck on them is due to local modifications, and so they're not going to be downloading ancient vanilla kernels from kernel.org now.

They perfectly could. This is exactly what we're doing at Suse and I can easily imagine other companies follow the same model. We store our local changes as patches on top of the old kernel version. When a new developer joins the team and needs to setup a working tree, our setup script gets the patches from our internal repository, fetches the relevant kernel tarball from kernel.org, unpacks it and applies all the patches.

This is one of the reasons why others have been claiming in this
discussion: it would be weird if files which were previously available would suddenly disappear. We can discuss the cost and benefits of any change done to the tree structure, compression formats etc. but please do not assume that nobody is downloading the old files from kernel.org.

Personally I wouldn't mind at all if old files would disappear and our tools have to be adjusted accordingly, as long as it happens only once in a long while and not on a regular basis by (broken) design.

Signature · 1 line
Jean Delvare
#87 lore raw
Quoted 22 lines · Jean Delvare

Hi Phillip,

Quoted 4 lines · Phillip Lougher

Embedded and enterprise distro users are usually stuck on ancient kernels that were downloaded from kernel.org and patched *years ago*. The reason they're stuck on them is due to local modifications, and so they're not going to be downloading ancient vanilla kernels from kernel.org now.

They perfectly could. This is exactly what we're doing at Suse and I can easily imagine other companies follow the same model. We store our local changes as patches on top of the old kernel version. When a new developer joins the team and needs to setup a working tree, our setup script gets the patches from our internal repository, fetches the relevant kernel tarball from kernel.org, unpacks it and applies all the patches.

This is one of the reasons why others have been claiming in this
discussion: it would be weird if files which were previously available would suddenly disappear. We can discuss the cost and benefits of any change done to the tree structure, compression formats etc. but please do not assume that nobody is downloading the old files from kernel.org.

Quoted 4 lines

Personally I wouldn't mind at all if old files would disappear and our tools have to be adjusted accordingly, as long as it happens only once in a long while and not on a regular basis by (broken) design.

not trying to cut in, but the best example I can see for this
(hopefully), or a good example of just changing everything
(cut the middle man per say)is libc there is no libc-2.11.90.so .tar.gz(etc..)only through git(but could be wrong).

My system that I built is only handling everything(meaning every package as much as possible)through git.

Justin P. Mattock

A message that is not in the archive (sent off-list or not yet received): 20100212223547.GN5186@tux

#89 lore raw
Quoted 8 lines · Luis R. Rodriguez
Quoted 2 lines · Jean Delvare

Prove it. Many people out there are still working on older trees. I am working on 2.6.5 and 2.6.16 kernels on a weekly basis.

I have to ask, I've read about the reasons why people still using 2.4 [1], why are you using 2.5? Same thing?

/me gets popcorn

Maybe try getting glasses instead? ;)

Signature · 1 line
Jean Delvare
#90 lore raw
In reply to Jean Delvare
Quoted 5 lines · Jean Delvare

1* Keep a single compression format. This saves almost 40% of the files.

2* Move one of the compression formats somewhere else, so that it doesn't get in the way but is still available if needed.

Neither of these are feasible, sorry.

Quoted 6 lines

3* Create a new subdirectory for every 2.6.x kernel, and move all the related files there. This would shrink the main index drastically, and each subdirectory would have a reasonable size (except maybe 2.6.16 and 2.6.27.) Oddly enough this has been done for the files under testing/ already, so I am curious why we don't do it for the release files (and the testing/incr/ files, while we're at it.)

Well, part of the reason why is that we're functionally "stuck" on 2.6; a prefix which really has lost all meaning.

It might open up the question if we shouldn't just do a Solaris and drop the leading 2 (so the next kernel would be 6.33) or call the kernel after that 3.0 instead of 2.6.34, and then 3.1 instead of 2.6.35.

-hpa

#91 lore raw
Quoted 7 lines · H. Peter Anvin
Quoted 6 lines

3* Create a new subdirectory for every 2.6.x kernel, and move all the related files there. This would shrink the main index drastically, and each subdirectory would have a reasonable size (except maybe 2.6.16 and 2.6.27.) Oddly enough this has been done for the files under testing/ already, so I am curious why we don't do it for the release files (and the testing/incr/ files, while we're at it.)

Quoted 6 lines

Well, part of the reason why is that we're functionally "stuck" on 2.6; a prefix which really has lost all meaning.

It might open up the question if we shouldn't just do a Solaris and drop the leading 2 (so the next kernel would be 6.33) or call the kernel after that 3.0 instead of 2.6.34, and then 3.1 instead of 2.6.35.

Damn, we forgot to have that fight at Kernel Summit last year.

I'm in favour of the 3.0 / 3.1 / 3.2 with stable@ being responsible for 3.0.1, 3.0.2, 3.1.1, etc.

Signature · 4 lines
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."
#92 lore raw
Quoted 8 lines · Matthew Wilcox
Quoted 7 lines · H. Peter Anvin
Quoted 6 lines

3* Create a new subdirectory for every 2.6.x kernel, and move all the related files there. This would shrink the main index drastically, and each subdirectory would have a reasonable size (except maybe 2.6.16 and 2.6.27.) Oddly enough this has been done for the files under testing/ already, so I am curious why we don't do it for the release files (and the testing/incr/ files, while we're at it.)

Quoted 8 lines
Quoted 6 lines

Well, part of the reason why is that we're functionally "stuck" on 2.6; a prefix which really has lost all meaning.

It might open up the question if we shouldn't just do a Solaris and drop the leading 2 (so the next kernel would be 6.33) or call the kernel after that 3.0 instead of 2.6.34, and then 3.1 instead of 2.6.35.

Damn, we forgot to have that fight at Kernel Summit last year.

No one wanted to take it on :(

Quoted 2 lines

I'm in favour of the 3.0 / 3.1 / 3.2 with stable@ being responsible for 3.0.1, 3.0.2, 3.1.1, etc.

I'm in favor of almost _anything_ new, the current numbering scheme drives me crazy, but then, I'm the one having to deal with it more than anyone these days...

thanks,

greg k-h

#93 lore raw
In reply to Matthew Wilcox
Quoted 9 lines · Matthew Wilcox
Quoted 3 lines

It might open up the question if we shouldn't just do a Solaris and drop the leading 2 (so the next kernel would be 6.33) or call the kernel after that 3.0 instead of 2.6.34, and then 3.1 instead of 2.6.35.

Damn, we forgot to have that fight at Kernel Summit last year.

I'm in favour of the 3.0 / 3.1 / 3.2 with stable@ being responsible for 3.0.1, 3.0.2, 3.1.1, etc.

At last years LinuxCon conference, I was practically bouncing out of my chair wanting to ask Linus about 3.0 in front of everyone while he was on the panel discussion. But James Bottomley refused to take my question.

-- Steve

#94 lore raw
In reply to Matthew Wilcox
Quoted 11 lines · Matthew Wilcox
Quoted 10 lines · H. Peter Anvin
Quoted 6 lines

3* Create a new subdirectory for every 2.6.x kernel, and move all the related files there. This would shrink the main index drastically, and each subdirectory would have a reasonable size (except maybe 2.6.16 and 2.6.27.) Oddly enough this has been done for the files under testing/ already, so I am curious why we don't do it for the release files (and the testing/incr/ files, while we're at it.)

Well, part of the reason why is that we're functionally "stuck" on 2.6; a prefix which really has lost all meaning.

Quoted 8 lines
Quoted 3 lines

It might open up the question if we shouldn't just do a Solaris and drop the leading 2 (so the next kernel would be 6.33) or call the kernel after that 3.0 instead of 2.6.34, and then 3.1 instead of 2.6.35.

Damn, we forgot to have that fight at Kernel Summit last year.

I'm in favour of the 3.0 / 3.1 / 3.2 with stable@ being responsible for 3.0.1, 3.0.2, 3.1.1, etc.

Like I suggested in October 2008, but it would have been more natural at that time:

<http://marc.info/?l=linux-kernel&m=122418454113793&w=2>

Signature · 1 line
Hilsen Harald.
#95 lore raw
Quoted 8 lines · Harald Arnesen
Quoted 3 lines

I'm in favour of the 3.0 / 3.1 / 3.2 with stable@ being responsible for 3.0.1, 3.0.2, 3.1.1, etc.

Like I suggested in October 2008, but it would have been more natural at that time:

<http://marc.info/?l=linux-kernel&m=122418454113793&w=2>

You and several other people.

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
#96 lore raw
In reply to H. Peter Anvin
Quoted 9 lines
Quoted 1 line
Quoted 1 line · H. Peter Anvin
Quoted 6 lines · Jean Delvare

3* Create a new subdirectory for every 2.6.x kernel, and move all the related files there. This would shrink the main index drastically, and each subdirectory would have a reasonable size (except maybe 2.6.16 and 2.6.27.) Oddly enough this has been done for the files under testing/ already, so I am curious why we don't do it for the release files (and the testing/incr/ files, while we're at it.)

Quoted 6 lines

Well, part of the reason why is that we're functionally "stuck" on 2.6; a prefix which really has lost all meaning.

It might open up the question if we shouldn't just do a Solaris and drop the leading 2 (so the next kernel would be 6.33) or call the kernel after that 3.0 instead of 2.6.34, and then 3.1 instead of 2.6.35.

I remember the whole LKML discussion about this a few years back:
http://lkml.org/lkml/2008/10/15/377

The whole year.version or year/month versioning Greg HK proposed
made a lot of sense to me. It would also solve our problem with the 2.6 directory just growing and growing as the year versioning would make a natural hierarchy which keeps going no matter what.

--
Jeffry Sleddens
Rotterdam University

#97 lore raw
Quoted 16 lines · J.P.G.
Quoted 9 lines
Quoted 1 line
Quoted 1 line · H. Peter Anvin
Quoted 6 lines · Jean Delvare

3* Create a new subdirectory for every 2.6.x kernel, and move all the related files there. This would shrink the main index drastically, and each subdirectory would have a reasonable size (except maybe 2.6.16 and 2.6.27.) Oddly enough this has been done for the files under testing/ already, so I am curious why we don't do it for the release files (and the testing/incr/ files, while we're at it.)

Quoted 6 lines

Well, part of the reason why is that we're functionally "stuck" on 2.6; a prefix which really has lost all meaning.

It might open up the question if we shouldn't just do a Solaris and drop the leading 2 (so the next kernel would be 6.33) or call the kernel after that 3.0 instead of 2.6.34, and then 3.1 instead of 2.6.35.

Quoted 7 lines

I remember the whole LKML discussion about this a few years back:
http://lkml.org/lkml/2008/10/15/377

The whole year.version or year/month versioning Greg HK proposed
made a lot of sense to me. It would also solve our problem with the 2.6 directory just growing and growing as the year versioning would make a natural hierarchy which keeps going no matter what.

Note also that every time this conversation happens it starts to pull away in different directions, and as a result nothing happens.

I'm going to stick my foot in it and state the following: I think incremental numbers work well, and everyone are used to them. It doesn't seem to be the major issue with the current scheme; the issue with the current scheme is that we have one or two levels too much.

-hpa

#98 lore raw
Quoted 25 lines · H. Peter Anvin
Quoted 16 lines · J.P.G.
Quoted 9 lines
Quoted 1 line
Quoted 1 line · H. Peter Anvin
Quoted 6 lines · Jean Delvare

3* Create a new subdirectory for every 2.6.x kernel, and move all the related files there. This would shrink the main index drastically, and each subdirectory would have a reasonable size (except maybe 2.6.16 and 2.6.27.) Oddly enough this has been done for the files under testing/ already, so I am curious why we don't do it for the release files (and the testing/incr/ files, while we're at it.)

Quoted 6 lines

Well, part of the reason why is that we're functionally "stuck" on 2.6; a prefix which really has lost all meaning.

It might open up the question if we shouldn't just do a Solaris and drop the leading 2 (so the next kernel would be 6.33) or call the kernel after that 3.0 instead of 2.6.34, and then 3.1 instead of 2.6.35.

Quoted 7 lines

I remember the whole LKML discussion about this a few years back:
http://lkml.org/lkml/2008/10/15/377

The whole year.version or year/month versioning Greg HK proposed
made a lot of sense to me. It would also solve our problem with the 2.6 directory just growing and growing as the year versioning would make a natural hierarchy which keeps going no matter what.

Quoted 7 lines

Note also that every time this conversation happens it starts to pull away in different directions, and as a result nothing happens.

I'm going to stick my foot in it and state the following: I think incremental numbers work well, and everyone are used to them. It doesn't seem to be the major issue with the current scheme; the issue with the current scheme is that we have one or two levels too much.

Exactly. And for having worked with projects using dates, it's a hell complicated to handle delayed releases... I'd like 3.0, 3.1, ... too, and am using that scheme for my projects, which people easily understand very well, as opposed to x.y.z.t.

But now I think that Linus has a "grep -v 3.0" filter on his inbox, I never got a reply from him on the subject and I know I'm not the only one :-(

Willy

#99 lore raw
In reply to H. Peter Anvin
Quoted 3 lines · H. Peter Anvin

It might open up the question if we shouldn't just do a Solaris and drop the leading 2 (so the next kernel would be 6.33) or call the kernel after that 3.0 instead of 2.6.34, and then 3.1 instead of 2.6.35.

This sounds like a good plan (and since we have so far failed to come up with some new feature in Linux that is so awesome that it warrants bumping the version number to 3.0 ... we might as well manufacture one, and "The HTML for the archive directory on kernel.org has gotten too big" seems a pretty good reason to me).

So the plan could be:
1) Declare 2.6.35 (or so) to be the last in the 2.6 series.
2) Define a better archive directory structure for the 3.x releases (scripts will have to be changed anyway to handle the 3.x change)
3) Keep all the .gz and .bz2 in the old 2.x hierarchy
4) Only have .xz in the new 3.x directories.

This should give time for people to update scripts and for xz compression tools to become widely available.

People on the bleeding edge versions are the most likely ones to update their tools promptly. People still using 2.x series kernels can keep using their old tools forever.

-Tony

#100 lore raw
Quoted 4 lines · Tony Luck

3) Keep all the .gz and .bz2 in the old 2.x hierarchy
4) Only have .xz in the new 3.x directories.

I really dislike this for reasons already stated. Keep in mind that the compression management is global to the entire archive, not just to the kernel, too.

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
#101 lore raw
Quoted 9 lines · H. Peter Anvin
Quoted 4 lines · Tony Luck

3) Keep all the .gz and .bz2 in the old 2.x hierarchy
4) Only have .xz in the new 3.x directories.

I really dislike this for reasons already stated. Keep in mind that the compression management is global to the entire archive, not just to the kernel, too.

How hard a requirement is this? It might be the right time to reconsider... Tony's plan may not be perfect but overall I think I like it.

Signature · 1 line
Jean Delvare
#102 lore raw
Quoted 9 lines · Jean Delvare
Quoted 4 lines

I really dislike this for reasons already stated. Keep in mind that the compression management is global to the entire archive, not just to the kernel, too.

How hard a requirement is this? It might be the right time to reconsider... Tony's plan may not be perfect but overall I think I like it.

It's pretty darn hard. It makes it particularly messy for the mirrors to ignore it.

-hpa

A message that is not in the archive (sent off-list or not yet received): 987664A83D2D224EAE907B061CE93D53123E0ED5@orsmsx505.amr.corp.intel.com

#104 lore raw
Quoted 6 lines · Tony
Quoted 6 lines
Quoted 5 lines

How hard a requirement is this? It might be the right time to reconsider... Tony's plan may not be perfect but overall I think I like it.

Quoted 7 lines
Quoted 2 lines

It's pretty darn hard. It makes it particularly messy for the mirrors to ignore it.

Sometimes hard things are worth the effort though. I would think that there would be considerable advantage to a mirror system that had a mechanism to mark entire subtrees as "archive only - no further changes".

There are a couple of reasons to not do that.

a) the moment we do that, some mirrors will start purging those areas. We have tried to keep our mirror system universally usable by keeping a "whole mirrors only, please" policy, and it has worked really well so far.

b) it would prevent us from *ever* reorganizing those areas, as many be required.

c) it complicates a lot of the machinery, including the bits that allow mirrors to only carry some of the compression formats. That includes mirror-side configuration, which I'd really loathe to make anything more than very simple; mirror admins want things to work as hands-off and automatic as possible, with a minimum of setup. Having to learn the intricacies of each mirror target is just plain hell.

-hpa

#105 lore raw
In reply to J.H.
Quoted 53 lines · J.H.

Hey Everyone,

So as the subject states this is more a centralized discussion on migration plans to using and providing xz for content on kernel.org. Currently we provide gz and bz2, with gz acting as the original content and kernel.org itself generating the resulting bz2 files. There are a couple of possible proposals and wanted to toss them out there, and get feedback from everyone: the kernel community, the mirrors of kernel.org and the direct users of kernel.org.

========================================================================

Option 1)

Leave gz as the master, and migrate bz2 to xz. This will happen in stages obviously. with bz2 ultimately being phased out.

Migration option 1)

All new content would be provided in .bz2 and .xz with an ultimate date set that the .bz2 files would stop being generated with new content. This would leave all existing content alone and it would not be a migration of the current .bz2 files to xz

Migration option 2)

At some point there would be a mass conversion of all existing content to include .bz2 and .xz. These would be run in parallel for a time period until it was determined that .bz2 was no longer needed and it would be removed from the servers leaving .gz and .xz

Option 2)

Convert the master data from gz to bz2 and use xz as the new file format. This has the downside of causing more tool churn as it means the kernel developers will have to eventually convert from gz to bz2, which means for a time there will be nag e-mails if you upload gz instead of bz2 and such. It would also mean that we (kernel.org) would need to be able to support .gz and .bz2 as master data for a time.

Migration options are identical to Option 1 more or less, with either just new content getting converted, or all content getting converted.

========================================================================

I'm personally leaning towards option 1, though personally don't really have a preference on the migration options, as both obviously offer different advantages, and again this e-mail is more to spur on the discussion and come to some general consensus across all of the groups concerned before moving forward with a more specific plan.

Quoted 5 lines

So I'm inviting discussion, questions and comments on this so we know which way to ultimately go.

- John 'Warthog9' Hawley
Chief Kernel.org Administrator

My would go for Option 1.

With respect to migration Option 2. The last time we had a succsessful change of the default compression format from compress to gzip, it required a change from a encumbered format to an unencombered format, and gunzip was able to decompress files created with compress.

Neither xz nor bzip2 can decompress gzip'd files, and appear to be heavy enough that for modest file sizes they give little gain. bzip2 fills a niche but does not act as a generally available, general purpose compressor today. It appears that xz is coming up fast in bzip2's niche, and provides better compression, so would appear to be a good replacement for bzip2.

The role of gzip is to be compatible with the everything, and I would be very sad to see it gzip disappear as that would make files on kernel.org much less accessible.

Migration option 2 sounds like I would could just replace bzip2 support in scripts with xz support so it sounds handy, but I will leave the details to those who have to run the servers.

Right now xz suffers from limited availability. I just checked the systems I have handy and of the 4 distro's I have installed on various machines only 1 provides xz, so I expect it will be a while before xz achieves ubiquity. At the same time the fact that xz reduces the linux-2.6.32 tarball by 10Megs is a welcome benefit, and seems to make the replacement of bzip2 with xz worth doing.

Eric

#106 lore raw
Quoted 6 lines · On Sun, 14 Feb 2010 10:03:31 -0800 [email protected] (Eric W. Biederman)…

Right now xz suffers from limited availability. I just checked the systems I have handy and of the 4 distro's I have installed on various machines only 1 provides xz, so I expect it will be a while before xz achieves ubiquity. At the same time the fact that xz reduces the linux-2.6.32 tarball by 10Megs is a welcome benefit, and seems to make the replacement of bzip2 with xz worth doing.

xv is not standard on Ubuntu.
Installing xv-utils on Ubuntu 9.10 produces big scary warning about package conflict with lzma.

This seems like a premature optimization at this point

#107 lore raw
Quoted 10 lines · Stephen Hemminger
Quoted 6 lines · On Sun, 14 Feb 2010 10:03:31 -0800 [email protected] (Eric W. Biederman)…

Right now xz suffers from limited availability. I just checked the systems I have handy and of the 4 distro's I have installed on various machines only 1 provides xz, so I expect it will be a while before xz achieves ubiquity. At the same time the fact that xz reduces the linux-2.6.32 tarball by 10Megs is a welcome benefit, and seems to make the replacement of bzip2 with xz worth doing.

Quoted 5 lines

xv is not standard on Ubuntu.
Installing xv-utils on Ubuntu 9.10 produces big scary warning about package conflict with lzma.

This seems like a premature optimization at this point

That would be a packaging bug, xz should transparently supersede lzma.

Signature · 1 line
Jean Delvare