Moving zlib so that others may use it
I'm working on a function that uses zlib in the kernel, and I know of other places zlib is used (ppp_deflate, jffs2, mcore). I would expect more users to come along.
I would like to propose putting zlib in the lib directory and making it optionally compile if it is needed. It's pretty easy to move the files around and make a few small changes to the code. However, how do I configure such a thing? I could add something like:
if [ "$CONFIG_PPP_DEFLATE" = "y" ]; then
define_tristate CONFIG_ZLIB y
fi
if [ "$CONFIG_PPP_DEFLATE" = "m" ]; then
if [ "$CONFIG_ZLIB" != "y" ]; then
define_tristate CONFIG_ZLIB m
fi
fi
to every place that uses it, or I could put something like:
if [ "$CONFIG_JFFS2_FS" = "y" \
-o "$CONFIG_PPP_DEFLATE" = "y" ]; then
define_tristate CONFIG_ZLIB y
else
if [ "$CONFIG_JFFS2_FS" = "m" \
-o "$CONFIG_PPP_DEFLATE" = "m" ]; then
define_tristate CONFIG_ZLIB m
fi
fiat the end of the config. I would prefer the latter, but there is not an "end of config" place, you would have to put it at the end of every config.in (about 15 architectures right now). I propose to add a Config.in to the lib directory that is sourced at the end of every config.in for the architectures. I'll do the work if it's willing to be accepted into the kernel.
-Corey
Followup to: <[email protected]> By author: Corey Minyard <[email protected]> In newsgroup: linux.dev.kernel
Quoted 5 lines
I'm working on a function that uses zlib in the kernel, and I know of other places zlib is used (
ppp_deflate, jffs2, mcore). I would expect more users to come along.
CAREFUL. First of all, don't mix up the deflate and inflate functions, second of all, make sure you get the memory management right. It's not trivial to do the latter, since the default zlib memory management is unusable for at least some users in kernelspace.
-hpa
Signature · 3 lines
<[email protected]> at work, <[email protected]> in private! "Unix gives you enough rope to shoot yourself in the foot." http://www.zytor.com/~hpa/puzzle.txt <[email protected]>
Quoted 8 lines · Corey Minyard
I'm working on a function that uses zlib in the kernel, and I know of other places zlib is used (
ppp_deflate, jffs2, mcore). I would expect more users to come along.I would like to propose putting zlib in the lib directory and making it optionally compile if it is needed. It's pretty easy to move the files around and make a few small changes to the code. However, how do I configure such a thing? I could add something like:
I am assuming that you can satisfy hpa's concerns about using a single version of zlib for everybody. Also note that arch/ppc/boot/lib has its own version of zlib which is quite different to the others. First make sure that you can build one version of zlib that works for everybody.
Configuring and linking zlib is a problem. lib/lib.a is an archive which means that the linker will only include zlib from lib.a if some other code refers to zlib. That breaks when all users of zlib are in modules and nothing in vmlinux refers to zlib, the zlib code is not linked into the kernel and modules get unresolved errors. Linking lib.a into the affected modules is not the solution either, the modules would pick up all the code from lib.a instead of using the code in the kernel with undefined results.
The best option is to build zlib.o for the kernel (not module) and store it in lib.a. Compile zlib.o if any consumer of zlib has been selected and add a dummy reference to zlib code in vmlinux to ensure that zlib is pulled from the archive if anybody needs it, even if all the consumers are in modules. Some of the zlib symbols will need to be exported, I will leave that to you.
This should do the trick, completely untested. It assumes that zlib.h has been moved to to include/linux and zlib.c to lib. Against 2.4.17.
Index: 17.1/lib/Makefile
lib/Makefile Thu, 10 Jan 2002 12:26:18 +1100 kaos (linux-2.4/j/28_Makefile 1.1.4.3.3.2 644) renamed from lib/Makefile Tue, 18 Sep 2001 13:43:44 +1000 kaos (linux-2.4/j/28_Makefile 1.1.4.3.3.2 644) +2 -1
@@ -8,12 +8,13 @@ L_TARGET := lib.a -export-objs := cmdline.o dec_and_lock.o rwsem-spinlock.o rwsem.o +export-objs := cmdline.o dec_and_lock.o rwsem-spinlock.o rwsem.o zlib.o obj-y := errno.o ctype.o string.o vsprintf.o brlock.o cmdline.o bust_spinlocks.o rbtree.o obj-$(CONFIG_RWSEM_GENERIC_SPINLOCK) += rwsem-spinlock.o obj-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem.o +obj-$(CONFIG_ZLIB) += zlib.o ifneq ($(CONFIG_HAVE_DEC_LOCK),y) obj-y += dec_and_lock.o
Index: 17.1/init/main.c
init/main.c Thu, 10 Jan 2002 12:24:49 +1100 kaos (linux-2.4/k/11_main.c 1.1.5.1.1.8.1.3.1.8 644) renamed from init/main.c Sat, 01 Dec 2001 11:29:21 +1100 kaos (linux-2.4/k/11_main.c 1.1.5.1.1.8.1.3.1.8 644) +5 -0
@@ -269,6 +269,11 @@ static struct dev_name_struct { { NULL, 0 } }; +#ifdef CONFIG_ZLIB +#include <linux/zlib.h> +static void *dummy_zlib_reference __initdata = &deflate; +#endif + kdev_t __init name_to_kdev_t(char *line) { int base = 0;
Index: 17.1/Makefile
Makefile Thu, 10 Jan 2002 12:42:09 +1100 kaos (linux-2.4/T/c/50_Makefile 1.1.2.15.1.2.2.25.2.2.1.17.1.4.1.29.1.40 644) renamed from Makefile Sat, 22 Dec 2001 12:56:52 +1100 kaos (linux-2.4/T/c/50_Makefile 1.1.2.15.1.2.2.25.2.2.1.17.1.4.1.29.1.40 644) +15 -0
@@ -124,6 +124,21 @@ NETWORKS =net/network.o LIBS =$(TOPDIR)/lib/lib.a SUBDIRS =kernel drivers mm fs net ipc lib +# FIXME: make CONFIG_ZLIB a CML2 derived rule later and remove all references to +# zlib from this file. There is no clean place to handle this in CML1. KAO +export CONFIG_ZLIB +ifneq ($(subst n,,$(CONFIG_JFFS2)$(CONFIG_PPP_DEFLATE)),) + CONFIG_ZLIB := y +endif +# FIXME: Use this if ppc/boot has been changed to use the common zlib. +# ifeq ($(ARCH),ppc) +# CONFIG_ZLIB := y +# endif +# Kludge because CONFIG_ZLIB is not in include/linux/autoconf.h in CML1. +ifeq ($(CONFIG_ZLIB),y) + CFLAGS += -DCONFIG_ZLIB=1 +endif + DRIVERS-n := DRIVERS-y := DRIVERS-m :=
Quoted 4 lines · Keith Owens
Quoted 2 lines · Corey Minyard
I'm working on a function that uses zlib in the kernel, and I know of other places zlib is used (
ppp_deflate, jffs2, mcore). I would expect more users to come along.
Quoted 9 lines
Quoted 2 lines
I would like to propose putting zlib in the lib directory and making it optionally compile if it is needed. It's pretty easy to move the files around and make a few small changes to the code. However, how do I configure such a thing? I could add something like:
I am assuming that you can satisfy hpa's concerns about using a single version of zlib for everybody. Also note that
arch/ppc/boot/libhas its own version of zlib which is quite different to the others. First make sure that you can build one version of zlib that works for everybody.
Every version I found was practically the same (only comment differences and one that variable that was initialized in one and not initialized in the other). I think (actually, I know, because I have tested it) everyone can use the same one. I'm not talking about the ones in the boot directories, nor the one for inflate_fs, either, just the ones in the kernel proper that are named zlib.c and zlib.h.
Quoted 9 lines
Configuring and linking zlib is a problem.
lib/lib.ais an archive which means that the linker will only include zlib from lib.a if some other code refers to zlib. That breaks when all users of zlib are in modules and nothing in vmlinux refers to zlib, the zlib code is not linked into the kernel and modules get unresolved errors. Linking lib.a into the affected modules is not the solution either, the modules would pick up all the code from lib.a instead of using the code in the kernel with undefined result
Quoted 7 lines
The best option is to build zlib.o for the kernel (not module) and store it in lib.a. Compile zlib.o if any consumer of zlib has been selected and add a dummy reference to zlib code in vmlinux to ensure that zlib is pulled from the archive if anybody needs it, even if all the consumers are in modules. Some of the zlib symbols will need to be exported, I will leave that to you.
Why not just create zlib as a module if all the users are modules (so depmod and modprobe load it)? That's what everything else does. And that way, if it's already in the kernel, the module just won't get loaded, but if it's not the module gets loaded. What you are suggesting seems rather convoluted.
I guess one other option would be to have an explicit user-set tristate like CONFIG_ZLIB, and if anything uses zlib, it could only be modules if CONFIG_ZLIB was a module, etc.
Quoted 5 lines
This should do the trick, completely untested. It assumes that
zlib.hhas been moved to toinclude/linuxandzlib.cto lib. Against 2.4.17.
I already have patches to do all this, actually, I was just trying to get some feedback on it and see if the kernel maintainers would be interested in this. I should have said that earlier.
Thanks,
-Corey
Quoted 13 lines · Corey Minyard
Quoted 5 lines · Keith Owens
Quoted 2 lines · Corey Minyard
I would like to propose putting zlib in the lib directory and making it optionally compile if it is needed.
Quoted 7 lines
The best option is to build zlib.o for the kernel (not module) and store it in lib.a. Compile zlib.o if any consumer of zlib has been selected and add a dummy reference to zlib code in vmlinux to ensure that zlib is pulled from the archive if anybody needs it, even if all the consumers are in modules. Some of the zlib symbols will need to be exported, I will leave that to you.
Quoted 5 lines
Why not just create zlib as a module if all the users are modules (so depmod and modprobe load it)? That's what everything else does. And that way, if it's already in the kernel, the module just won't get loaded, but if it's not the module gets loaded. What you are suggesting seems rather convoluted.
If zlib is a module then it cannot be part of lib/lib.a, it has to be separate, with changes to the top level Makefile to conditionally include lib/zlib.o. I did that originally but the changes to lib/Makefile and the top level Makefile were worse. Building zlib as a module guarantees that you cannot use it in a boot loader, forcing you to maintain multiple versions of zlib.c. If you are going to use one version of zlib then you should try to handle bootloaders as well.
What is convoluted about my solution? The derivation of CONFIG_ZLIB in the top level Makefile is ugly but that ugliness is a side effect of CML1. CONFIG_ZLIB has to be derived somewhere, it is a smaller patch to do it in Makefile than to patch 15 arch/*/config.in files. Apart from that, the only other niggle is the dummy reference in init/main.c.
Quoted 3 lines
I guess one other option would be to have an explicit user-set tristate like
CONFIG_ZLIB, and if anything uses zlib, it could only be modules ifCONFIG_ZLIBwas a module, etc.
Don't ask the user, they will not understand the problem. CONFIG_ZLIB is derived from other configs and possibly ARCH variables, users have no direct control over CONFIG_ZLIB.
Quoted 18 lines · Keith Owens
Quoted 6 lines · Corey Minyard
Quoted 4 lines · Keith Owens
Quoted 2 lines · Corey Minyard
I would like to propose putting zlib in the lib directory and making it optionally compile if it is needed.
Quoted 9 lines
Quoted 7 lines
The best option is to build zlib.o for the kernel (not module) and store it in lib.a. Compile zlib.o if any consumer of zlib has been selected and add a dummy reference to zlib code in vmlinux to ensure that zlib is pulled from the archive if anybody needs it, even if all the consumers are in modules. Some of the zlib symbols will need to be exported, I will leave that to you.
Why not just create zlib as a module if all the users are modules (so depmod and modprobe load it)? That's what everything else does. And that way, if it's already in the kernel, the module just won't get loaded, but if it's not the module gets loaded. What you are suggesting seems rather convoluted.
Quoted 8 lines
If zlib is a module then it cannot be part of
lib/lib.a, it has to be separate, with changes to the top levelMakefileto conditionally includelib/zlib.o. I did that originally but the changes to lib/Makefile and the top levelMakefilewere worse. Building zlib as a module guarantees that you cannot use it in a boot loader, forcing you to maintain multiple versions ofzlib.c. If you are going to use one version of zlib then you should try to handle bootloaders as well.
Hmm. It worked fine for me. I made it a module, and it put it into kernel/lib in /lib/modules/2.4.17 and it did not put it in lib/lib.a I make it a non-module, and it gets included in lib/lib.a. My diff was the same as yours for the Makefile.
I don't know about the bootloaders. I'm not sure you can make the requirement to have them compiled the same as the kernel, since they may have different compilation requirements in the boot loader.
Quoted 6 lines
What is convoluted about my solution? The derivation of
CONFIG_ZLIBin the top levelMakefileis ugly but that ugliness is a side effect of CML1.CONFIG_ZLIBhas to be derived somewhere, it is a smaller patch to do it inMakefilethan to patch 15arch/*/config.infiles. Apart from that, the only other niggle is the dummy reference ininit/main.c.
The problem is that if you come along later and compile a new module that needs it, it won't work. That's a fairly common thing I do, I expect other kernel developers do the same. And the dummy ref thing is a little ugly.
Quoted 7 lines
Quoted 2 lines
I guess one other option would be to have an explicit user-set tristate like
CONFIG_ZLIB, and if anything uses zlib, it could only be modules ifCONFIG_ZLIBwas a module, etc.Don't ask the user, they will not understand the problem.
CONFIG_ZLIBis derived from other configs and possibly ARCH variables, users have no direct control overCONFIG_ZLIB.
Yeah, I agree, it was just a random thought that occured to me.
Thanks,
-Corey
Quoted 6 lines · Corey Minyard
Hmm. It worked fine for me. I made it a module, and it put it into kernel/lib in
/lib/modules/2.4.17and it did not put it inlib/lib.aI make it a non-module, and
it gets included inlib/lib.a. My diff was the same as yours for theMakefile.
Worked for me this time as well. I had a typo the first time then did an ugly fix to a non-existent problem :(
Quoted 4 lines
I don't know about the bootloaders. I'm not sure you can make the requirement
to have them compiled the same as the kernel, since they may have different compilation requirements in the boot loader.
Probably true, but I don't want to rule it out completely. In any case it is easily catered for in lib/Makefile.
obj-$(CONFIG_JFFS2_FS) += zlib.o obj-$(CONFIG_PPP_DEFLATE) += zlib.o
# Uncomment these if ppc bootloader can use the common zlib
# ifeq ($(ARCH),ppc) # obj-y += zlib.o # endif
Quoted 4 lines
The problem is that if you come along later and compile a new module that needs it, it won't work. That's a fairly common thing I do, I expect other kernel
developers do the same. And the dummy ref thing is a little ugly.
Any new module that requires zlib requires a change to the zlib selection. Whether you do it in the top level Makefile, in lib/Config.in or lib/Makefile is irrelevant, zlib has to be selected somewhere and the criteria must be updated for new modules.
Since lib/zlib.c works for both built in and modules, there is no need to change the top level Makefile. AFAICT the above lines in lib/Makefile are the minimal change.
Quoted 10 lines · Keith Owens
On Wed, 09 Jan 2002 23:13:28 -0600,
Quoted 6 lines · Corey Minyard
Hmm. It worked fine for me. I made it a module, and it put it into kernel/lib in
/lib/modules/2.4.17and it did not put it inlib/lib.aI make it a non-module, and
it gets included inlib/lib.a. My diff was the same as yours for theMakefile.
Quoted 2 lines
Worked for me this time as well. I had a typo the first time then did an ugly fix to a non-existent problem :(
Could you give a patch for all of this ? either Keith or Corey ?
Thomas
I'll do it, since I already have most of it, unless Keith already has or wants to.
-Corey
Quoted 23 lines · Thomas Capricelli
Quoted 11 lines · Keith Owens
On Wed, 09 Jan 2002 23:13:28 -0600,
Quoted 7 lines · Corey Minyard
Hmm. It worked fine for me. I made it a module, and it put it into kernel/lib in
/lib/modules/2.4.17and it did not put it inlib/lib.aI make it a non-module, and
it gets included inlib/lib.a. My diff was the same as yours for theMakefile.Quoted 3 lines
Worked for me this time as well. I had a typo the first time then did an ugly fix to a non-existent problem :(
Could you give a patch for all of this ? either Keith or Corey ?
Thomas
Ok, the patch is done. It's large, so get it at: http://home.attbi.com/~minyard/linux-genzlib.gz.
-Corey
Quoted 3 lines · Thomas Capricelli
Could you give a patch for all of this ? either Keith or Corey ?
Quoted 5 lines · Corey Minyard
Quoted 3 lines · Keith Owens
Quoted 1 line · Corey Minyard
Keith Owens wrote:
[...]
Quoted 4 lines
Building zlib as a
Quoted 3 lines
module guarantees that you cannot use it in a boot loader, forcing you to maintain multiple versions of
zlib.c. If you are going to use one version of zlib then you should try to handle bootloaders as well.
[...]
Quoted 4 lines
I don't know about the bootloaders. I'm not sure you can make the requirement
to have them compiled the same as the kernel, since they may have different compilation requirements in the boot loader.
Ummm, you can't use an in-kernel anything in a bootloader. How do you uncompress an in-kernel zlib.o without an out-of-kernel zlib.o lying around somewhere?
The closest thing to a zlib.o shared between bootloader and kernel would be to build one zlib.o and then perhaps copy the compiled binary from the kernel to the bootloader (thus having only one zlib.c but two zlib.o) or link it from the bootloader to the kernel once the kernel is uncompressed.
Signature · 2 lines
Zygo Blaxell (Laptop) <[email protected]> GPG = D13D 6651 F446 9787 600B AD1E CCF3 6F93 2823 44AD
Quoted 12 lines · Zygo Blaxell
Quoted 5 lines · Corey Minyard
Quoted 3 lines · Keith Owens
Quoted 1 line · Corey Minyard
Keith Owens wrote:
[...]
Quoted 4 lines
Building zlib as a
Quoted 3 lines
module guarantees that you cannot use it in a boot loader, forcing you to maintain multiple versions of
zlib.c. If you are going to use one version of zlib then you should try to handle bootloaders as well.[...]
Quoted 8 lines
Quoted 4 lines
I don't know about the bootloaders. I'm not sure you can make the requirement
to have them compiled the same as the kernel, since they may have different compilation requirements in the boot loader.Ummm, you can't use an in-kernel anything in a bootloader. How do you uncompress an in-kernel zlib.o without an out-of-kernel zlib.o lying around somewhere?
Well, when your 'bootloader' is in arch/$(ARCH)/boot, you can rather easily. I think there's a few archs which grab lib/lib.a right now.
Signature · 2 lines
Tom Rini (TR1265) http://gate.crashing.org/~trini/
Quoted 21 lines · Keith Owens
Quoted 13 lines · Corey Minyard
Quoted 5 lines · Keith Owens
Quoted 2 lines · Corey Minyard
I would like to propose putting zlib in the lib directory and making it optionally compile if it is needed.
Quoted 7 lines
The best option is to build zlib.o for the kernel (not module) and store it in lib.a. Compile zlib.o if any consumer of zlib has been selected and add a dummy reference to zlib code in vmlinux to ensure that zlib is pulled from the archive if anybody needs it, even if all the consumers are in modules. Some of the zlib symbols will need to be exported, I will leave that to you.
Quoted 5 lines
Why not just create zlib as a module if all the users are modules (so depmod and modprobe load it)? That's what everything else does. And that way, if it's already in the kernel, the module just won't get loaded, but if it's not the module gets loaded. What you are suggesting seems rather convoluted.
Quoted 7 lines
If zlib is a module then it cannot be part of
lib/lib.a, it has to be separate, with changes to the top levelMakefileto conditionally includelib/zlib.o. I did that originally but the changes to lib/Makefile and the top levelMakefilewere worse. Building zlib as a module guarantees that you cannot use it in a boot loader, forcing you to maintain multiple versions ofzlib.c. If you are going to use one version of zlib then you should try to handle bootloaders as well.
It's possible they can share, but the bootloaders (PPC & MIPS) need a slight change to the zlib.c code to to allow using zero as a real address to store the uncompressed data. So we'd want to guard the changes with __BOOTER__ or so, and then cp the file or do #define __BOOTER__
#include "zlib.c"
And do -I$(TOPDIR)/lib, or something along those lines.
Signature · 2 lines
Tom Rini (TR1265) http://gate.crashing.org/~trini/
Quoted 10 lines · Tom Rini
It's possible they can share, but the bootloaders (PPC & MIPS) need a slight change to the
zlib.ccode to to allow using zero as a real address to store the uncompressed data. So we'd want to guard the changes with__BOOTER__or so, and then cp the file or do #define__BOOTER__
#include "zlib.c"And do -I$(TOPDIR)/lib, or something along those lines.
I agree, but I think it would be better to do this one step at a time. Let's get the things in the kernel working first to get rid of the namespace crash, then get the bootloaders to share the code. I can't test out a lot of the code, so I can't really do the bootloader changes, and the bootloader changes are completely independent, anyway, once zlib is moved.
-Corey
Quoted 10 lines · Corey Minyard
Quoted 8 lines · Tom Rini
It's possible they can share, but the bootloaders (PPC & MIPS) need a slight change to the
zlib.ccode to to allow using zero as a real address to store the uncompressed data. So we'd want to guard the changes with__BOOTER__or so, and then cp the file or do #define__BOOTER__
#include "zlib.c"
Quoted 3 lines
Quoted 1 line
And do -I$(TOPDIR)/lib, or something along those lines.
I agree, but I think it would be better to do this one step at a time.
So do I. So it can work, but lets get this in and I'll make the PPC stuff use it (since I've got lots of other updates for it I'm getting ready to push out).
Signature · 2 lines
Tom Rini (TR1265) http://gate.crashing.org/~trini/
Quoted 8 lines · H. Peter Anvin
Quoted 3 lines
I'm working on a function that uses zlib in the kernel, and I know of other places zlib is used (
ppp_deflate, jffs2, mcore). I would expect more users to come along.CAREFUL. First of all, don't mix up the deflate and inflate functions, second of all, make sure you get the memory management right. It's not trivial to do the latter, since the default zlib memory management is unusable for at least some users in kernelspace.
I'm not sure I follow you here. Do you want to completely separate the inflate and deflate stuff (so if something only needs one, it only has to include one)? I'm not sure of the value, and it would be kind of a pain for maintenance (since zlib is from an external source).
As far as memory management, all the versions I am talking about are almost exactly the same, so that shouldn't be a problem.
-Corey
Quoted 9 lines · Corey Minyard
I'm not sure I follow you here. Do you want to completely separate the inflate and deflate stuff (so if something only needs one, it only has to include one)? I'm not sure of the value, and it would be kind of a pain for maintenance (since zlib is from an external source).
As far as memory management, all the versions I am talking about are almost exactly the same, so that shouldn't be a problem.
It is external, but it's quite well separated anyway.
Look at the inflate_fs one (fs/inflate_fs); it has both the memory management dealt with and decompression factored out...
-hpa
Quoted 5 lines · [email protected] said:
I am assuming that you can satisfy hpa's concerns about using a
single version of zlib for everybody. Also note thatarch/ppc/boot/lib has its own version of zlib which is quite different to the others. First make sure that you can build one version of zlib that works for everybody.
I can confirm that the JFFS2 and PPP zlib are compatible - differing in cosmetics only. Moving it to lib/zlib would be a good thing.
We can verify compatibility for other zlib users as an when those other users are converted to use lib/zlib instead of their own private copy.
Quoted 5 lines
The best option is to build zlib.o for the kernel (not module) and store it in lib.a. Compile zlib.o if any consumer of zlib has been selected and add a dummy reference to zlib code in vmlinux to ensure that zlib is pulled from the archive if anybody needs it, even if all the consumers are in modules.
AUIU you've since decided this isn't necessary - which is good. Making the static kernel image differ according to which modules happened to be compiled at the time is not a good thing. Sometimes we do it, but we should avoid it when we can.
If zlib.o is used in modules only, compile it as a module. Don't put it into the kernel.
--
dwmw2