compat-drivers wishlist
Hi,
I've long been fed up with the state of compat's build system, so I figured I'd start by soliciting wishlist items.
FWIW, I think that the whole compat-drivers/scripts/ dir needs to be rewritten for this ... :-)
My personal wishlist is below. I realize that some of these wishes cause contradictions/problems in their implementation, but nonetheless I have them :-)
1) Ability to copy only some drivers.
We're interested in using compat-drivers to ship our driver to
customers for
older kernels. We're not at all interested in shipping DRM, Atheros
drivers,
etc.
2) To make that easier: split all patches per *file* in addition to per change.
That way, it would be possible to apply only patches for those files
that are
actually in the tree. Right now, I heavily patch out those patches
that aren't
really need.
I suppose the best way would be to have directories for each change
and change
the script that applies them to parse the patch header first. Or just
ignore
patches that fail with "no file", but I don't know if that's easy to
determine
from the patch exit status code.
3) Separate the output from the compat-drivers tree.
There's no reason to ship patches/, most of scripts/ etc. in the
resulting
tarball, in fact it's just confusing. Having the output in a separate directory, copying there only the few things that are needed, would
be very
helpful.
4) Kconfig.
This is a bit conflicting with 1) because a valid Kconfig structure
requires
all the sub-Kconfig files exist. Maybe ship all Kconfig files, or
automate a
way to remove "source" statements for files that don't exist
(probably easy)
I'll leave it at that for now, and follow up with some concrete implementation ideas :-)
Any other thoughts?
johannes
Quoted 12 lines · Johannes Berg
Hi,
I've long been fed up with the state of compat's build system, so I figured I'd start by soliciting wishlist items.
FWIW, I think that the whole compat-drivers/scripts/ dir needs to be rewritten for this ... :-)
My personal wishlist is below. I realize that some of these wishes cause contradictions/problems in their implementation, but nonetheless I have them :-)
Quoted 6 lines
1) Ability to copy only some drivers.
We're interested in using compat-drivers to ship our driver to
customers for
older kernels. We're not at all interested in shipping DRM, Atheros
drivers,
etc.
Seems reasonable. Maybe admin-update can read your own .config
Quoted 6 lines
2) To make that easier: split all patches per *file* in addition to per change.
That way, it would be possible to apply only patches for those files
that are
actually in the tree. Right now, I heavily patch out those patches
that aren't
Quoted 8 lines
really need.
I suppose the best way would be to have directories for each change
and change
the script that applies them to parse the patch header first. Or just
ignore
patches that fail with "no file", but I don't know if that's easy to
determine
from the patch exit status code.
GNU patch could use some love. See commit 4a1a94f93 on compat-drivers. Perhaps this is an area we'd be interested in extending functionality as well. Having collateral evolutions split up each into their own directory is a fine approach as well, but lets think long term here, given that we're seeing what short term script/hack solutions can cost us.
Maybe we need a context patch parser that given a .config will only apply patches for the desired files. For this to work we'd need a mapping of CONFIG_FOO symbols to respective driver path files regexps. The Linux kernel MAINTAINERS file comes to mind as something close to get us there, but that does not yet use CONFIG_ symbols, the title is a bit loose.
Kconfig however gives us the config to let us build a target, and the Makefile should let us extract its target dependencies, so I wonder if we can use this to dynamically build kconfig to target maps.
Quoted 7 lines
3) Separate the output from the compat-drivers tree.
There's no reason to ship patches/, most ofscripts/etc. in the
resulting
tarball, in fact it's just confusing. Having the output in a separate directory, copying there only the few things that are needed, would
be very
helpful.
Sure, makes sense.
Quoted 7 lines
4)
Kconfig.
This is a bit conflicting with 1) because a validKconfigstructure
requires
all the sub-Kconfig files exist. Maybe ship allKconfigfiles, or
automate a
way to remove "source" statements for files that don't exist
(probably easy)
I think its possible to remove not desired source code based on reading .config. This can be done as a pass if the Kconfig to Makefile thing is figured out. Then as a secondary step I think we can remove directories that have just a Kconfig / Makefile, but that's theoretical, in practice this may be harder, the biggest issue I think would be to ensure removal of the calling 'source foo/Kconfig' entry, or we extend Kconfig to support Kconfig imports that don't exist, sort of with Makefile and the -include foo.mk trick.
Quoted 4 lines
I'll leave it at that for now, and follow up with some concrete implementation ideas :-)
Any other thoughts?
There, some feedback, thanks for taking the time to dumping your ideas.
Luis
Quoted 10 lines · Luis R. Rodriguez
Quoted 7 lines · Johannes Berg
4)
Kconfig.
This is a bit conflicting with 1) because a validKconfigstructure
requires
all the sub-Kconfig files exist. Maybe ship allKconfigfiles, or
automate a
way to remove "source" statements for files that don't exist
(probably easy)
Quoted 8 lines
I think its possible to remove not desired source code based on reading .config. This can be done as a pass if the
KconfigtoMakefilething is figured out. Then as a secondary step I think we can remove directories that have just aKconfig/Makefile, but that's theoretical, in practice this may be harder, the biggest issue I think would be to ensure removal of the calling 'source foo/Kconfig' entry, or we extendKconfigto supportKconfigimports that don't exist, sort of withMakefileand the -include foo.mk trick.
Another thing I should mention here about Kconfig.
'make localmodconfig' is great and IMHO something which we should strive for as well for end users. The issue comes in when you have to build something for remote systems. For this I wrote a script a while ago to let us extract only the necessary components from the kernel, scp to target, and extract back our .config [0]. This is hacky but something to keep in mind.
[0] https://lkml.org/lkml/2009/11/12/350
Luis
Quoted 8 lines · Luis R. Rodriguez
Another thing I should mention here about
Kconfig.'make localmodconfig' is great and IMHO something which we should strive for as well for end users. The issue comes in when you have to build something for remote systems. For this I wrote a script a while ago to let us extract only the necessary components from the kernel, scp to target, and extract back our .config [0]. This is hacky but something to keep in mind.
Yes, that's another interesting thing to do. I myself tend to think it's less important as even building all of compat shouldn't take all that much time. OTOH, it's clearly a lot more than what you usually want.
The tricky part here is that for those users that have most reason to use compat, those who need it for new driver modules, it won't work because they can't already be loaded on the system.
Anyway I don't see why we couldn't just copy the streamline config perl script, with the Kconfig tree we build it would probably work just fine.
johannes
Quoted 3 lines · Luis R. Rodriguez
Quoted 1 line
1) Ability to copy only some drivers.
Seems reasonable. Maybe admin-update can read your own .config
This is a tricky proposition, see below & my other mail on Kconfig thoughts.
Quoted 1 line
GNU patch could use some love.
I'd hate to have to rely on a modified version of that :-)
FWIW, I think there's a python library to apply (unified) patches, if we want to get fancy.
Quoted 1 line
See commit
4a1a94f93on compat-drivers.
Ah, what's that for? Refreshing patches? I've never seen dependencies in the compat patches so mostly I refresh manually, but yeah, the refresh part is something I'd have to think about. Though, to be honest, when you can just blow away your output directory and try again I'm not too worried about this. Trying to apply the patches could just leave it in the state you're in, and then you modify them until it works?
Quoted 3 lines
Maybe we need a context patch parser that given a .config will only apply patches for the desired files. For this to work we'd need a mapping of
CONFIG_FOOsymbols to respective driver path files regexps.
Even assuming we do have such a mapping, I think it's the wrong approach and causes a chicken-and-egg problem.
As I wrote in my other mail, you don't have a .config file before you know the target kernel. This is because, for example, there's no way to know if PCI is enabled in the target kernel, and thus sometimes you get PCI drivers and sometimes not. Also I think maintenance and driver split is not usually per Kconfig symbol but per directory. There are many more Kconfig symbols, after all.
Also above you said you could do the source copying (or pruning) based on the .config file. This has the exact same issue, I'd rather have the copying or pruning be based on directories.
Basically I'm saying the .config file only exists on the target build system, while the selective copying/pruning needs to be done on the tarball creation system.
Quoted 4 lines
Quoted 1 line
4)
Kconfig.I think its possible to remove not desired source code based on reading .config.
See above.
Oh, another issue I haven't addressed is default configurations, but I think that's easy to do by providing something like the kernel's defconfig.
johannes
Quoted 16 lines · Johannes Berg
Quoted 3 lines · Luis R. Rodriguez
Quoted 1 line
1) Ability to copy only some drivers.
Seems reasonable. Maybe admin-update can read your own .config
This is a tricky proposition, see below & my other mail on
Kconfigthoughts.Quoted 1 line
GNU patch could use some love.
I'd hate to have to rely on a modified version of that :-)
FWIW, I think there's a python library to apply (unified) patches, if we want to get fancy.
Quoted 8 lines
Quoted 1 line
See commit
4a1a94f93on compat-drivers.Ah, what's that for? Refreshing patches? I've never seen dependencies in the compat patches so mostly I refresh manually, but yeah, the refresh part is something I'd have to think about. Though, to be honest, when you can just blow away your output directory and try again I'm not too worried about this. Trying to apply the patches could just leave it in the state you're in, and then you modify them until it works?
Refreshing patches should not be a problem. Normally I use "./scripts/admin-refresh.sh -n -p -c -u refresh" to refresh the patches and this should work with some minor changes with the proposal you made.
Quoted 14 lines
Quoted 3 lines
Maybe we need a context patch parser that given a .config will only apply patches for the desired files. For this to work we'd need a mapping of
CONFIG_FOOsymbols to respective driver path files regexps.Even assuming we do have such a mapping, I think it's the wrong approach and causes a chicken-and-egg problem.
As I wrote in my other mail, you don't have a .config file before you know the target kernel. This is because, for example, there's no way to know if PCI is enabled in the target kernel, and thus sometimes you get PCI drivers and sometimes not. Also I think maintenance and driver split is not usually per
Kconfigsymbol but per directory. There are many moreKconfigsymbols, after all.
Quoted 7 lines
Also above you said you could do the source copying (or pruning) based on the .config file. This has the exact same issue, I'd rather have the copying or pruning be based on directories.
Basically I'm saying the .config file only exists on the target build system, while the selective copying/pruning needs to be done on the tarball creation system.
How do you want to make the tar generation being configurable? The build time Kconfig options should be generated from the Kconfig files provided by the kernel and not by some own files.
I think some manual edited config file with the directories or files to be included sounds good. This function would not be used by the normal user so that should be good enough.
Quoted 12 lines
Quoted 4 lines
Quoted 1 line
4)
Kconfig.I think its possible to remove not desired source code based on reading .config.
See above.
Oh, another issue I haven't addressed is default configurations, but I think that's easy to do by providing something like the kernel's defconfig.
johannes
Quoted 10 lines · Hauke Mehrtens
Quoted 6 lines
Ah, what's that for? Refreshing patches? I've never seen dependencies in the compat patches so mostly I refresh manually, but yeah, the refresh part is something I'd have to think about. Though, to be honest, when you can just blow away your output directory and try again I'm not too worried about this. Trying to apply the patches could just leave it in the state you're in, and then you modify them until it works?
Refreshing patches should not be a problem. Normally I use "
./scripts/admin-refresh.sh-n -p -c -u refresh" to refresh the patches and this should work with some minor changes with the proposal you made.
Ok. I have no idea what that does, since most of the time my "refreshing" isn't really refreshing but making them apply again, and I do that manually.
Quoted 7 lines
Quoted 3 lines
Basically I'm saying the .config file only exists on the target build system, while the selective copying/pruning needs to be done on the tarball creation system.
How do you want to make the tar generation being configurable? The build time
Kconfigoptions should be generated from theKconfigfiles provided by the kernel and not by some own files.
Right, the build time Kconfig should just be copied as other source files.
Quoted 3 lines
I think some manual edited config file with the directories or files to be included sounds good. This function would not be used by the normal user so that should be good enough.
Indeed, that's what I was thinking. Sorry for not making that clear.
johannes
Quoted 3 lines
1) Ability to copy only some drivers.
2) To make that easier: split all patches per *file* in addition to per change.
You could think that an alternative to these would be for me to post-process the sources and remove everything I don't care about. However, that gets tricky because my compat package is created from my own tree, and there the patches don't always apply. Therefore these wishes.
I think for 2), I'd propose the following structure:
patches/collateral-evolutions/network/0001-netdev_ops/
-> INFO -> rndis.patch -> usbnet.patch -> ath6kl.patch
Since the filename doesn't matter all that much, an automated conversion could just split the patch per file and store it in a filename obtained as follows:
drivers/net/usb/rndis_host.c
-> drivers_net_usb_rndis_host.c
Easy :-)
The script to apply them gets a little more difficult, but I think we should start using python for the tree-creation-time scripting which will make this easy enough. Just walk the tree, etc.
Quoted 1 line
3) Separate the output from the compat-drivers tree.
Should be easy, though it probably requires rewriting admin-update. OTOH, admin-refresh etc. all pretty much goes away since you can just rm -rf the output directory.
Quoted 1 line
4)
Kconfig.
So my wishes 1-3 apply at "tree creation" time, which is when the compat source tree (and later tarball) is created and patched for the backports. I've already suggested to rewrite that scripting in python, and I'm willing to invest time in that if it stands a chance of getting accepted :-)
This one is a bit more difficult. On the one hand we need to have all the symbols from the kernel, but we don't want to leak them into the compat namespace. Right now, for example, we do things like renaming CONFIG_USB_NET_RNDIS_HOST to CONFIG_USB_NET_COMPAT_RNDIS_HOST which we really should do for everything -- we get into trouble occasionally because somebody enables something in the base kernel and then tries to disable it in compat, which doesn't work.
I've been thinking a while about this, and I think I pretty much found a solution.
First of all, it's easy to write a small Kconfig program (using the existing Kconfig C code of course) that generates a list of all symbols present in a given Kconfig tree, even disabled ones. Here's the program, you just have to link it with zconf.tab.c and pass a root Kconfig file as the only command line argument:
http://p.sipsolutions.net/3300bc91340ae747.txt
Equipped with a list of config symbols, we can go through the sources and Makefiles and replace CONFIG_* with (e.g.) CPTCFG_* (and also CONFIG_*_MODULE). We can't just replace CONFIG_ with CPTCFG_ because then we get kernel symbols wrong. We could rename all CONFIG_ to CPTCFG_ and import the kernel symbols, which we have to do for Kconfig anyway, but I feel that doing that would make things more brittle. (*)
Now obviously if we have a local Kconfig tree, it's pretty useless. The reason is that we'll have things like
config MYDRIVER_PCI
tristate "my good PCI driver" depends on PCI && HAS_IOMEM
config MYDRIVER_USB
tristate "my good USB driver"
depends on USB
or something like that, and the Kconfig tree we have clearly doesn't know about PCI, HAS_IOMEM or USB. We'll have to teach it, which is surprisingly easy:
path/to/syms-program Kconfig > local-symbol-list
sed -i 's/$/=/' local-symbol-list
grep -v -f local-symbol-list $KLIB_BUILD/.config | dotconf-to-kconf.py > Kconfig.kernel
dotconf-to-kconf.py is this little tool: http://p.sipsolutions.net/986bfbd397ad51c5.txt
(Clearly it has to be rewritten in a different language since this has
to run at build time, not just at tarball creation time, but that's easy to do)
Now we can "source Kconfig.kernel" into our local Kconfig tree, and the "depends on USB" can magically work the right way. For the symbol walk (syms program above) we'll have to include an empty file, but that's not terribly difficult either.
Oh, whenever we execute menuconfig/oldconfig/... we set the environment variable CONFIG_ to CPTCFG_ -- this will make the tool output CPTCFG_ instead of CONFIG_ as the prefix for all the output variables.
This is about as far as I got with the implementation. There are a few issues left.
1) All of the kernel symbols like USB get put into the .config file, e.g. CPTCFG_USB. This doesn't actually matter all that much because they're prefixed CPTCFG_ and the code we compile still looks for CONFIG_ for the kernel symbols -- remember that we only renamed the ones we found in our local Kconfig tree.
2) I have no idea how to go from a .config file to a working Makefile/autoconf.h system.
3) If KLIB_BUILD dir or the kernel config changes, this has to be regenerated and oldconfig must be run. For instance, if USB was now disabled in the base kernel then we shouldn't attempt to build USB drivers, etc. This is a bit tricky because we also don't want to redo everything for each "make". I'm not sure how much we care about this, but we can probably solve it by storing a checksum of $KLIB_BUILD/.config and doing the preparation steps only if that changes, or so.
4) This assumes KLIB_BUILD is configured externally, it's not possible to configure that through menuconfig since the Kconfig tree needs to be built first. I don't think this is a problem, but it looks like Luis wanted to put that into Kconfig so I'm saying it won't really work.
Thoughts?
johannes
(*) post-scriptum: this can and should actually happen at tree creation time, not at build time
Quoted 92 lines · Johannes Berg
Quoted 3 lines
1) Ability to copy only some drivers.
2) To make that easier: split all patches per *file* in addition to per change.You could think that an alternative to these would be for me to post-process the sources and remove everything I don't care about. However, that gets tricky because my compat package is created from my own tree, and there the patches don't always apply. Therefore these wishes.
I think for 2), I'd propose the following structure:
patches/collateral-evolutions/network/0001-netdev_ops/
-> INFO -> rndis.patch -> usbnet.patch -> ath6kl.patchSince the filename doesn't matter all that much, an automated conversion could just split the patch per file and store it in a filename obtained as follows:
drivers/net/usb/rndis_host.c
->drivers_net_usb_rndis_host.cEasy :-)
The script to apply them gets a little more difficult, but I think we should start using python for the tree-creation-time scripting which will make this easy enough. Just walk the tree, etc.
Quoted 1 line
3) Separate the output from the compat-drivers tree.
Should be easy, though it probably requires rewriting admin-update. OTOH, admin-refresh etc. all pretty much goes away since you can just rm -rf the output directory.
Quoted 1 line
4)
Kconfig.So my wishes 1-3 apply at "tree creation" time, which is when the compat source tree (and later tarball) is created and patched for the backports. I've already suggested to rewrite that scripting in python, and I'm willing to invest time in that if it stands a chance of getting accepted :-)
This one is a bit more difficult. On the one hand we need to have all the symbols from the kernel, but we don't want to leak them into the compat namespace. Right now, for example, we do things like renaming
CONFIG_USB_NET_RNDIS_HOSTtoCONFIG_USB_NET_COMPAT_RNDIS_HOSTwhich we really should do for everything -- we get into trouble occasionally because somebody enables something in the base kernel and then tries to disable it in compat, which doesn't work.I've been thinking a while about this, and I think I pretty much found a solution.
First of all, it's easy to write a small
Kconfigprogram (using the existingKconfigC code of course) that generates a list of all symbols present in a givenKconfigtree, even disabled ones. Here's the program, you just have to link it withzconf.tab.cand pass a rootKconfigfile as the only command line argument:
http://p.sipsolutions.net/3300bc91340ae747.txtEquipped with a list of config symbols, we can go through the sources and Makefiles and replace CONFIG_* with (e.g.) CPTCFG_* (and also CONFIG_*_MODULE). We can't just replace
CONFIG_withCPTCFG_because then we get kernel symbols wrong. We could rename allCONFIG_toCPTCFG_and import the kernel symbols, which we have to do forKconfiganyway, but I feel that doing that would make things more brittle. (*)Now obviously if we have a local
Kconfigtree, it's pretty useless. The reason is that we'll have things likeconfig
MYDRIVER_PCItristate "my good PCI driver" depends on PCI && HAS_IOMEMconfig
MYDRIVER_USB
tristate "my good USB driver"
depends on USBor something like that, and the
Kconfigtree we have clearly doesn't know about PCI,HAS_IOMEMor USB. We'll have to teach it, which is surprisingly easy:path/to/syms-program
Kconfig> local-symbol-list
sed -i 's/$/=/' local-symbol-list
grep -v -f local-symbol-list $KLIB_BUILD/.config |dotconf-to-kconf.py>Kconfig.kernel
dotconf-to-kconf.pyis this little tool: http://p.sipsolutions.net/986bfbd397ad51c5.txt
(Clearly it has to be rewritten in a different language since this has
to run at build time, not just at tarball creation time, but that's easy to do)
Quoted 4 lines
Now we can "source
Kconfig.kernel" into our localKconfigtree, and the "depends on USB" can magically work the right way. For the symbol walk (syms program above) we'll have to include an empty file, but that's not terribly difficult either.
We also have to change all the select SOME_KERNEL_OPTION into a depends SOME_KERNEL_OPTION, because the options of the used kernel can not be changed.
Quoted 16 lines
Oh, whenever we execute menuconfig/oldconfig/... we set the environment variable
CONFIG_toCPTCFG_-- this will make the tool outputCPTCFG_instead ofCONFIG_as the prefix for all the output variables.This is about as far as I got with the implementation. There are a few issues left.
1) All of the kernel symbols like USB get put into the .config file, e.g.
CPTCFG_USB. This doesn't actually matter all that much because they're prefixedCPTCFG_and the code we compile still looks forCONFIG_for the kernel symbols -- remember that we only renamed the ones we found in our localKconfigtree.2) I have no idea how to go from a .config file to a working
Makefile/autoconf.hsystem.
Quoted 8 lines
3) If
KLIB_BUILDdir or the kernel config changes, this has to be regenerated and oldconfig must be run. For instance, if USB was now disabled in the base kernel then we shouldn't attempt to build USB drivers, etc. This is a bit tricky because we also don't want to redo everything for each "make". I'm not sure how much we care about this, but we can probably solve it by storing a checksum of $KLIB_BUILD/.config and doing the preparation steps only if that changes, or so.
compat should be changed in this way to not always backport any e.g. all USB stuff and depend on USB stuff if the USB subsystem was not activated. I think this should be modularized to have one *.ko for the USB stuff, one for the PCMCIA stuff and so on.
Currently compat.ko depends on USB for some kernel versions.
This could also be done later.
Quoted 12 lines
4) This assumes
KLIB_BUILDis configured externally, it's not possible to configure that through menuconfig since theKconfigtree needs to be built first. I don't think this is a problem, but it looks like Luis wanted to put that intoKconfigso I'm saying it won't really work.Thoughts?
johannes
(*) post-scriptum: this can and should actually happen at tree creation time, not at build time
Quoted 8 lines · Hauke Mehrtens
Quoted 4 lines
Now we can "source
Kconfig.kernel" into our localKconfigtree, and the "depends on USB" can magically work the right way. For the symbol walk (syms program above) we'll have to include an empty file, but that's not terribly difficult either.We also have to change all the select
SOME_KERNEL_OPTIONinto a dependsSOME_KERNEL_OPTION, because the options of the used kernel can not be changed.
Huh, good point. I didn't think about that, and it probably doesn't even exist? But yeah, I agree, we need to do this. It's not terribly difficult though -- after we have a list of local symbols we can go through and rewrite "select NONLOCAL" to "depends on NONLOCAL".
Quoted 5 lines
compat should be changed in this way to not always backport any e.g. all USB stuff and depend on USB stuff if the USB subsystem was not activated. I think this should be modularized to have one *.ko for the USB stuff, one for the PCMCIA stuff and so on.
Currently compat.ko depends on USB for some kernel versions.
I don't think compat backports USB/PCMCIA/...? I was under the impression that it was necessary to be from the base kernel anyway, but I don't really know. In theory I agree, anything that's backported in compat needs to be included into the Kconfig somehow, particularly things like this.
Actually, I guess we do have things like that now, just not USB/PCMCIA:
CONFIG_COMPAT_FIRMWARE_CLASSCONFIG_COMPAT_NET_SCH_CODELCONFIG_COMPAT_NET_SCH_FQ_CODELCONFIG_COMPAT_KFIFOCONFIG_COMPAT_CORDICCONFIG_COMPAT_CRC8
and something about CONFIG_GENERIC_ATOMIC64 on non-64 bit platforms.
This is interesting! It probably means we need to generate a Kconfig file for the kernel versions as well, at build time:
config COMPAT_KERNEL_2_6_26
def_bool y
Obviously then compat relies more on compat-drivers to actually build? It could have its own Kconfig file, and probably best if the creation of this Kconfig.compat file is done through a script there.
Anyway this is special. So we can treat it specially, and do something like
config COMPAT_CORDIC
bool
default !CORDIC
which should work just fine since we imported the symbols from the base kernel. We'll then get a CPTCFG_COMPAT_CORDIC symbol that we can use to build (or not) the cordic code.
For something that depends on a specific kernel version it's just as simple:
config COMPAT_FIRMWARE_CLASS
tristate default m if COMPAT_KERNEL_2_6_33 && !COMPAT_RHEL_6_1
or so.
johannes
Quoted 10 lines · Johannes Berg
Quoted 3 lines
1) Ability to copy only some drivers.
2) To make that easier: split all patches per *file* in addition to per change.You could think that an alternative to these would be for me to post-process the sources and remove everything I don't care about. However, that gets tricky because my compat package is created from my own tree, and there the patches don't always apply. Therefore these wishes.
Quoted 7 lines
I think for 2), I'd propose the following structure:
patches/collateral-evolutions/network/0001-netdev_ops/
-> INFO -> rndis.patch -> usbnet.patch -> ath6kl.patch
That's fine, and in fact with the other long term things I'm looking at, such as SmPL to exrpress these collateral evolutions, is to categorize the collateral evolutions into different types and see if we can get *all* of them implemented in compat alone. That is, strive to get rid of all patches required.
This requires a study of the different types of collateral evolutions we have. compat already addresses the ones we can implement without requiring changing the upstream drivers. I'm now starting to address -- what is upstream willing to do to help with backports. An example is the static inlines.
https://lkml.org/lkml/2013/3/28/187
If the above patch is accepted I'd then try to send upstream the netdev_attach_ops() one, and that would nuke that entire patch from compat-drivers.
Quoted 7 lines
Since the filename doesn't matter all that much, an automated conversion could just split the patch per file and store it in a filename obtained as follows:
drivers/net/usb/rndis_host.c
->drivers_net_usb_rndis_host.cEasy :-)
True. I welcome this change if it helps you.
Quoted 3 lines
The script to apply them gets a little more difficult, but I think we should start using python for the tree-creation-time scripting which will make this easy enough. Just walk the tree, etc.
ACK.
Quoted 6 lines
Quoted 1 line
3) Separate the output from the compat-drivers tree.
Should be easy, though it probably requires rewriting admin-update. OTOH, admin-refresh etc. all pretty much goes away since you can just rm -rf the output directory.
Quoted 7 lines
Quoted 1 line
4)
Kconfig.So my wishes 1-3 apply at "tree creation" time, which is when the compat source tree (and later tarball) is created and patched for the backports. I've already suggested to rewrite that scripting in python, and I'm willing to invest time in that if it stands a chance of getting accepted :-)
I've long said bring it. I'm on the Python bandwagon now, my first rewrite was ckmake.
Quoted 7 lines
This one is a bit more difficult. On the one hand we need to have all the symbols from the kernel, but we don't want to leak them into the compat namespace. Right now, for example, we do things like renaming
CONFIG_USB_NET_RNDIS_HOSTtoCONFIG_USB_NET_COMPAT_RNDIS_HOSTwhich we really should do for everything -- we get into trouble occasionally because somebody enables something in the base kernel and then tries to disable it in compat, which doesn't work.
True.
Quoted 2 lines
I've been thinking a while about this, and I think I pretty much found a solution.
Yay.
Quoted 24 lines
First of all, it's easy to write a small
Kconfigprogram (using the existingKconfigC code of course) that generates a list of all symbols present in a givenKconfigtree, even disabled ones. Here's the program, you just have to link it withzconf.tab.cand pass a rootKconfigfile as the only command line argument:
http://p.sipsolutions.net/3300bc91340ae747.txtEquipped with a list of config symbols, we can go through the sources and Makefiles and replace CONFIG_* with (e.g.) CPTCFG_* (and also CONFIG_*_MODULE). We can't just replace
CONFIG_withCPTCFG_because then we get kernel symbols wrong. We could rename allCONFIG_toCPTCFG_and import the kernel symbols, which we have to do forKconfiganyway, but I feel that doing that would make things more brittle. (*)Now obviously if we have a local
Kconfigtree, it's pretty useless. The reason is that we'll have things likeconfig
MYDRIVER_PCI
tristate "my good PCI driver"
depends on PCI &&HAS_IOMEM
configMYDRIVER_USB
tristate "my good USB driver"
depends on USB
Quoted 7 lines
or something like that, and the
Kconfigtree we have clearly doesn't know about PCI,HAS_IOMEMor USB. We'll have to teach it, which is surprisingly easy:path/to/syms-program
Kconfig> local-symbol-list
sed -i 's/$/=/' local-symbol-list
grep -v -f local-symbol-list $KLIB_BUILD/.config |dotconf-to-kconf.py>Kconfig.kernel
Neat trick! This doesn't handle the mapping of depdendencies but certainly already makes good assumptions given that you *have* these things built. It does not however take into consideration for example that a user may say -- let me build cfg80211 alone, build it, install it and now have their 802.11 driver also selected (if they didn't).
Quoted 4 lines
dotconf-to-kconf.pyis this little tool: http://p.sipsolutions.net/986bfbd397ad51c5.txt
(Clearly it has to be rewritten in a different language since this has
to run at build time, not just at tarball creation time, but that's easy to do)
I actually like the simplicity of it in Python and prefer it if we can live with it, but given that we follow the kernel build system I'd rather see the kernel first take the bullet on swallowing Python... not sure, anyway doesn't matter much if its simple.
Quoted 11 lines
Now we can "source
Kconfig.kernel" into our localKconfigtree, and the "depends on USB" can magically work the right way. For the symbol walk (syms program above) we'll have to include an empty file, but that's not terribly difficult either.Oh, whenever we execute menuconfig/oldconfig/... we set the environment variable
CONFIG_toCPTCFG_-- this will make the tool outputCPTCFG_instead ofCONFIG_as the prefix for all the output variables.This is about as far as I got with the implementation. There are a few issues left.
Like Kconfig renames that went upstream and mapping them to their older name on the users's older kernels? This likely could be addressed by having these maps laid out for each kernel, but would require manual maintenance. It shouldn't be so hard if we do it *moving forward*.
The other bigger thing I think is the dependency map output not existing so it'd let users break a build / existing dep.
Quoted 8 lines
1) All of the kernel symbols like USB get put into the .config file, e.g.
CPTCFG_USB. This doesn't actually matter all that much because they're prefixedCPTCFG_and the code we compile still looks forCONFIG_for the kernel symbols -- remember that we only renamed the ones we found in our localKconfigtree.2) I have no idea how to go from a .config file to a working
Makefile/autoconf.hsystem.
Hm, we'd already have the Makefile for each target no? Are you trying to avoid having to ship each Makefile for each pulled driver? For autoconf we have our own thingy so far.
Maybe I'm not getting what you mean here.
Quoted 8 lines
3) If
KLIB_BUILDdir or the kernel config changes, this has to be regenerated and oldconfig must be run. For instance, if USB was now disabled in the base kernel then we shouldn't attempt to build USB drivers, etc. This is a bit tricky because we also don't want to redo everything for each "make". I'm not sure how much we care about this, but we can probably solve it by storing a checksum of $KLIB_BUILD/.config and doing the preparation steps only if that changes, or so.
Sure.
Quoted 4 lines
4) This assumes
KLIB_BUILDis configured externally, it's not possible to configure that through menuconfig since theKconfigtree needs to be built first. I don't think this is a problem, but it looks like Luis wanted to put that intoKconfigso I'm saying it won't really work.
I didn't want to put it into Kconfig actually, I was just throwing the variables on the menuconfig display to ensure the user building is aware of the sources of everything.
Luis
Quoted 11 lines · Luis R. Rodriguez
This requires a study of the different types of collateral evolutions we have. compat already addresses the ones we can implement without requiring changing the upstream drivers. I'm now starting to address -- what is upstream willing to do to help with backports. An example is the static inlines.
https://lkml.org/lkml/2013/3/28/187
If the above patch is accepted I'd then try to send upstream the
netdev_attach_ops()one, and that would nuke that entire patch from compat-drivers.
Right, this is a goal in itself, but I don't think we'll get rid of all patches. Consider things like the completely different API for multicast lists, for example.
Quoted 9 lines
Quoted 7 lines
Since the filename doesn't matter all that much, an automated conversion could just split the patch per file and store it in a filename obtained as follows:
drivers/net/usb/rndis_host.c
->drivers_net_usb_rndis_host.cEasy :-)
True. I welcome this change if it helps you.
It would help me quite a bit. I suppose I could actually send you patches for this soon. OTOH, without addressing my wishlist item 1) I probably won't update to a current compat-drivers base ... I heavily patch admin-update.sh now.
Quoted 7 lines
Quoted 3 lines
path/to/syms-program
Kconfig> local-symbol-list
sed -i 's/$/=/' local-symbol-list
grep -v -f local-symbol-list $KLIB_BUILD/.config |dotconf-to-kconf.py>Kconfig.kernelNeat trick! This doesn't handle the mapping of depdendencies but certainly already makes good assumptions given that you *have* these things built.
Are you referring to the "select" issue Hauke pointed out? Or what other handling of dependencies are you thinking of? This entire trick exists to allow compat's Kconfig to know about the base kernel's Kconfig so the user can select the drivers that are actually possible given the base kernel. For example, if building for kernel without PCI, you wouldn't be able to select PCI drivers, etc.
Quoted 3 lines
It does not however take into consideration for example that a user may say -- let me build cfg80211 alone, build it, install it and now have their 802.11 driver also selected (if they didn't).
That ... no I don't think you can do that. Nor do I think you even want to? You can't build cfg80211 from one compat and the driver from another, they won't be compatible anyway. So building cfg80211 first and then installing it would be useless unless they build their driver from the same tree later, but then it doesn't really matter that they still have to select cfg80211? Or maybe I'm misunderstanding?
Quoted 4 lines
I actually like the simplicity of it in Python and prefer it if we can live with it, but given that we follow the kernel build system I'd rather see the kernel first take the bullet on swallowing Python... not sure, anyway doesn't matter much if its simple.
It won't be more difficult even in bash, I'm just faster at writing it in python :-)
Quoted 8 lines
Quoted 2 lines
This is about as far as I got with the implementation. There are a few issues left.
Like
Kconfigrenames that went upstream and mapping them to their older name on the users's older kernels? This likely could be addressed by having these maps laid out for each kernel, but would require manual maintenance. It shouldn't be so hard if we do it *moving forward*.
Hmm, I never thought about that. Does that happen frequently? Do you have any examples? Presumably, an example of this would be if you have e.g. CONFIG_USB_DEVICES in an old kernel, but it's CONFIG_USB in the new one, and all drivers that exist now have "depends on USB" while on an old kernel USB won't exist, so you wouldn't be able to select such a driver?
Anyway it seems like we must already maintain such a mapping in a way, maybe a bit more implicitly? The way here you'd do it is add something like this:
config USB
bool default y depends on USB_DEVICES && COMPAT_KERNEL_2_3_4_5
or something like that?
Quoted 2 lines
The other bigger thing I think is the dependency map output not existing so it'd let users break a build / existing dep.
I think it wouldn't break, worst case you wouldn't be able to select a driver that you should be able to select?
Quoted 5 lines
Quoted 2 lines
2) I have no idea how to go from a .config file to a working
Makefile/autoconf.hsystem.Hm, we'd already have the
Makefilefor each target no? Are you trying to avoid having to ship eachMakefilefor each pulled driver?
Yes, we do have the makefiles of course, as copied from the kernel sources. We want to keep them, obviously. I'm not trying to avoid that. I'm saying I have no idea how to get the CONFIG_* or rather CPTCFG_* symbols into make when it reads those makefiles.
Quoted 1 line
For autoconf we have our own thingy so far.
Yeah, but how do we get the symbols into make? That part isn't clear to me at all, even for the regular kernel.
Quoted 8 lines
Quoted 4 lines
4) This assumes
KLIB_BUILDis configured externally, it's not possible to configure that through menuconfig since theKconfigtree needs to be built first. I don't think this is a problem, but it looks like Luis wanted to put that intoKconfigso I'm saying it won't really work.I didn't want to put it into
Kconfigactually, I was just throwing the variables on the menuconfig display to ensure the user building is aware of the sources of everything.
Ok :-)
Here's another issue I haven't really solved yet: All of compat must be modules, so selecting Y for any symbol, e.g. CFG80211, must be disallowed. This seems a little difficult, maybe the best choice would be to simply hack menuconfig/... tool to disallow setting tristate options to Y.
johannes
Quoted 10 lines · Johannes Berg
Quoted 8 lines · Luis R. Rodriguez
This requires a study of the different types of collateral evolutions we have. compat already addresses the ones we can implement without requiring changing the upstream drivers. I'm now starting to address -- what is upstream willing to do to help with backports. An example is the static inlines.
Quoted 7 lines
Quoted 3 lines
If the above patch is accepted I'd then try to send upstream the
netdev_attach_ops()one, and that would nuke that entire patch from compat-drivers.Right, this is a goal in itself, but I don't think we'll get rid of all patches. Consider things like the completely different API for multicast lists, for example.
I am willing to accept perhaps some changes may not be possible and we may need to live with patches but.. if we do at the very least we can use SmPL grammar to express it. Whether or not we can express this in SmPL that's a separate question but to help with that question we need to start categorizing these types of collateral evolutions.
Julia in case you are interested in exploring the one Johannes mentioned you can see commit 6dc94f7b9 on compat-drivers or just look at the file:
patches/collateral-evolutions/network/25-multicast-list_head.patch
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Since the filename doesn't matter all that much, an automated conversion could just split the patch per file and store it in a filename obtained as follows:
drivers/net/usb/rndis_host.c
->drivers_net_usb_rndis_host.c
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Easy :-)
True. I welcome this change if it helps you.
It would help me quite a bit. I suppose I could actually send you patches for this soon. OTOH, without addressing my wishlist item 1) I probably won't update to a current compat-drivers base ... I heavily patch
admin-update.shnow.
OK.
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
path/to/syms-program
Kconfig> local-symbol-list
sed -i 's/$/=/' local-symbol-list
grep -v -f local-symbol-list $KLIB_BUILD/.config |dotconf-to-kconf.py>Kconfig.kernelNeat trick! This doesn't handle the mapping of depdendencies but certainly already makes good assumptions given that you *have* these things built.
Quoted 6 lines
Are you referring to the "select" issue Hauke pointed out? Or what other handling of dependencies are you thinking of? This entire trick exists to allow compat's
Kconfigto know about the base kernel'sKconfigso the user can select the drivers that are actually possible given the base kernel. For example, if building for kernel without PCI, you wouldn't be able to select PCI drivers, etc.
No I'm concerned about the dependency within compat-drivers itself on allowing users to build an updated driver that breaks existing stock drivers that they didn't enable on compat-drivers. One example: say a user decides to build ath9k, but also has a USB dongle and ath9k_htc but does not build the ath9k_htc module, only ath9k. Loading ath9k_htc will fail after he installs that build.
Quoted 7 lines
Quoted 3 lines
It does not however take into consideration for example that a user may say -- let me build cfg80211 alone, build it, install it and now have their 802.11 driver also selected (if they didn't).
That ... no I don't think you can do that. Nor do I think you even want to? You can't build cfg80211 from one compat and the driver from another, they won't be compatible anyway.
Well of course, I'm just saying -- it'd be possible to do this. But I guess this is possible to day anyway :)
Quoted 4 lines
So building cfg80211 first and
then installing it would be useless unless they build their driver from the same tree later, but then it doesn't really matter that they still have to select cfg80211? Or maybe I'm misunderstanding?
No its fine, its a general build dependency issue but that issue exists today, we just don't allow for huge different set of configurations. We allow:
* build everything
* driver-select
And these are tested (or at least the ones I maintain are).
Kconfig opens the door for users doing complex configurations and these could end up being reported as "issues", flooding the lists, and time on people.
Quoted 7 lines
Quoted 4 lines
I actually like the simplicity of it in Python and prefer it if we can live with it, but given that we follow the kernel build system I'd rather see the kernel first take the bullet on swallowing Python... not sure, anyway doesn't matter much if its simple.
It won't be more difficult even in bash, I'm just faster at writing it in python :-)
OK ;)
Quoted 10 lines
Quoted 8 lines
Quoted 2 lines
This is about as far as I got with the implementation. There are a few issues left.
Like
Kconfigrenames that went upstream and mapping them to their older name on the users's older kernels? This likely could be addressed by having these maps laid out for each kernel, but would require manual maintenance. It shouldn't be so hard if we do it *moving forward*.Hmm, I never thought about that. Does that happen frequently?
Not *that* frequently but handling that is a fucking bitch. At times I wish the maintainer would send me a patch but so far only the contributors to compat have addressed that.
This is not a 'collateral evolution' per se, but certainly something to consider if we can extract logic from git to address it for us.
Quoted 6 lines
Do you
have any examples? Presumably, an example of this would be if you have e.g.CONFIG_USB_DEVICESin an old kernel, but it'sCONFIG_USBin the new one, and all drivers that exist now have "depends on USB" while on an old kernel USB won't exist, so you wouldn't be able to select such a driver?
Typically its not core stuff. Its all driver poo. At times its simply renames from upstream to downstream mapping issues. compat-drivers examples:
c5cfdf2d138ba20a648990d9615c44942eef5fd38c8113109f6916c4926413aa3416cefffaa5d9b8
Quoted 10 lines
Anyway it seems like we must already maintain such a mapping in a way, maybe a bit more implicitly? The way here you'd do it is add something like this:
config USB
bool default y depends on USB_DEVICES && COMPAT_KERNEL_2_3_4_5or something like that?
Sure. I think you're trying to automate this though?
BTW I'm trying to move things to BACKPORT_ prefix for poo given that the kernel does use compat_ already for some poo. Not a big issue, whatever.
Quoted 5 lines
Quoted 2 lines
The other bigger thing I think is the dependency map output not existing so it'd let users break a build / existing dep.
I think it wouldn't break, worst case you wouldn't be able to select a driver that you should be able to select?
Yes perhaps, its wroth for us to try it, likely better than the current unscalable situation.
Quoted 10 lines
Quoted 5 lines
Quoted 2 lines
2) I have no idea how to go from a .config file to a working
Makefile/autoconf.hsystem.Hm, we'd already have the
Makefilefor each target no? Are you trying to avoid having to ship eachMakefilefor each pulled driver?Yes, we do have the makefiles of course, as copied from the kernel sources. We want to keep them, obviously. I'm not trying to avoid that. I'm saying I have no idea how to get the CONFIG_* or rather CPTCFG_* symbols into make when it reads those makefiles.
Ah yes. Can't we assume a direct mapping CONFIG_FOO --> CPTCFG_FOO ?
Quoted 4 lines
Quoted 1 line
For autoconf we have our own thingy so far.
Yeah, but how do we get the symbols into make? That part isn't clear to me at all, even for the regular kernel.
Yeah, never figured that poo out myself... The Makefile somehow has to end up parsing the .config as a Makefile for variables, how that's done no sure. For compat-drivers we just used assignments on config.mk.
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines
4) This assumes
KLIB_BUILDis configured externally, it's not possible to configure that through menuconfig since theKconfigtree needs to be built first. I don't think this is a problem, but it looks like Luis wanted to put that intoKconfigso I'm saying it won't really work.I didn't want to put it into
Kconfigactually, I was just throwing the variables on the menuconfig display to ensure the user building is aware of the sources of everything.
Quoted 7 lines
Ok :-)
Here's another issue I haven't really solved yet: All of compat must be modules, so selecting Y for any symbol, e.g. CFG80211, must be disallowed. This seems a little difficult, maybe the best choice would be to simply hack menuconfig/... tool to disallow setting tristate options to Y.
If that's 100% true, sure! But say you want to disable a feature, or enable a new one you didn't have ?
Luis
Quoted 7 lines · Luis R. Rodriguez
Quoted 6 lines
Are you referring to the "select" issue Hauke pointed out? Or what other handling of dependencies are you thinking of? This entire trick exists to allow compat's
Kconfigto know about the base kernel'sKconfigso the user can select the drivers that are actually possible given the base kernel. For example, if building for kernel without PCI, you wouldn't be able to select PCI drivers, etc.
Quoted 6 lines
No I'm concerned about the dependency within compat-drivers itself on allowing users to build an updated driver that breaks existing stock drivers that they didn't enable on compat-drivers. One example: say a user decides to build ath9k, but also has a USB dongle and
ath9k_htcbut does not build theath9k_htcmodule, only ath9k. Loadingath9k_htcwill fail after he installs that build.
Yes. That's unfortunate, but I don't really think we should _force_ users to enable ath9k_htc? And we can't know what kind of USB devices they're going to plug in, so even "make localmodconfig" won't help all that much. We could default to all the drivers that the user has _installed_, but that'd mean maintaining a Kconfig <-> module mapping (though maybe the localmodconfig script has that?) and it wouldn't be what many users want? Why build USB drivers just because your distro did if you don't own the device, after all...
IOW, I'm not sure what we can do about this.
Quoted 4 lines
No its fine, its a general build dependency issue but that issue exists today, we just don't allow for huge different set of configurations. We allow:
Quoted 8 lines
* build everything
* driver-selectAnd these are tested (or at least the ones I maintain are).
Kconfigopens the door for users doing complex configurations and these could end up being reported as "issues", flooding the lists, and time on people.
I'm not all that worried about that, to be honest. We should probably default to running something like "allmodconfig", just like we do now. Also, the kernel's configuration is much more complex and did you see people complain there? :-)
In addition, we get the advantage that then people are unable to make things that don't build. I for one have in fact gotten plenty of bug reports because of wrong or missing ifdefs in config.mk (granted, through my own fault) -- e.g. config.mk enabling debugfs when the base kernel didn't allow for it, etc. So many issues are also solved by this.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Like
Kconfigrenames that went upstream and mapping them to their older name on the users's older kernels? This likely could be addressed by having these maps laid out for each kernel, but would require manual maintenance. It shouldn't be so hard if we do it *moving forward*.
Quoted 8 lines
Quoted 1 line
Hmm, I never thought about that. Does that happen frequently?
Not *that* frequently but handling that is a fucking bitch. At times I wish the maintainer would send me a patch but so far only the contributors to compat have addressed that.
This is not a 'collateral evolution' per se, but certainly something to consider if we can extract logic from git to address it for us.
I don't really see what we could extract from git. The way I see it, the only way to really address it is to add this to compat/Kconfig:
config NEW_SYMBOL_NAME
tristate
default OLD_SYMBOL_NAME
(or bool, and maybe with "depends on BACKPORT_KERNEL_X_Y_Z" added in)
Quoted 7 lines
Quoted 6 lines
Do you
have any examples? Presumably, an example of this would be if you have e.g.CONFIG_USB_DEVICESin an old kernel, but it'sCONFIG_USBin the new one, and all drivers that exist now have "depends on USB" while on an old kernel USB won't exist, so you wouldn't be able to select such a driver?
Quoted 5 lines
Typically its not core stuff. Its all driver poo. At times its simply renames from upstream to downstream mapping issues. compat-drivers examples:
c5cfdf2d138ba20a648990d9615c44942eef5fd3
Heh, beating me with my own driver, always a good plan :-)
However this is entirely within the backports package so it's not an issue at all?
Quoted 1 line
8c8113109f6916c4926413aa3416cefffaa5d9b8
Ditto here. In fact both of these patches just go away with my Kconfig scheme since I prefix CPTCFG_ instead of CONFIG_ and thus kernel symbols aren't used any more by the Makefile etc. stuff in backports.
Quoted 6 lines
Quoted 6 lines
config USB
bool default y depends on USB_DEVICES && COMPAT_KERNEL_2_3_4_5or something like that?
Heh, almost the same I wrote above...
Quoted 1 line
Sure. I think you're trying to automate this though?
But no, I don't think I can automate this part.
What I can automate is for drivers that are part of the backports package. For *dependencies* those drivers have, like my hypothetical USB example, we'll have to maintain it manually. It shouldn't be too difficult though.
Quoted 3 lines
BTW I'm trying to move things to
BACKPORT_prefix for poo given that the kernel does usecompat_already for some poo. Not a big issue, whatever.
Good plan. I'll make the COMPAT_KERNEL_X_Y_Z symbols BACKPORT_KERNEL_X_Y_Z instead, and also the other symbols with COMPAT in the name. For the automatic ones I picked CPTCFG_ which is hopefully unique enough and has the advantage of having the same length as CONFIG_ (which is selfish of me, but it helps me maintain the code since we work inside a backport package much of the time)
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
2) I have no idea how to go from a .config file to a working
Makefile/autoconf.hsystem.Hm, we'd already have the
Makefilefor each target no? Are you trying to avoid having to ship eachMakefilefor each pulled driver?
Quoted 6 lines
Quoted 4 lines
Yes, we do have the makefiles of course, as copied from the kernel sources. We want to keep them, obviously. I'm not trying to avoid that. I'm saying I have no idea how to get the CONFIG_* or rather CPTCFG_* symbols into make when it reads those makefiles.
Ah yes. Can't we assume a direct mapping
CONFIG_FOO-->CPTCFG_FOO?
Yes, we can mostly do that, and get rid of all the patches that modify symbols.
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
For autoconf we have our own thingy so far.
Yeah, but how do we get the symbols into make? That part isn't clear to me at all, even for the regular kernel.
Yeah, never figured that poo out myself... The
Makefilesomehow has to end up parsing the .config as aMakefilefor variables, how that's done no sure. For compat-drivers we just used assignments on config.mk.
Yeah ... I think the makefile just has to include the .config, I'll have to check it out. Shouldn't be too difficult. I think the difficulty with the backport system is that it has to call the kernel's Makefile, and has to export everything for that ... anyway I'll figure out something, can't be too hard.
Quoted 8 lines
Quoted 5 lines
Here's another issue I haven't really solved yet: All of compat must be modules, so selecting Y for any symbol, e.g. CFG80211, must be disallowed. This seems a little difficult, maybe the best choice would be to simply hack menuconfig/... tool to disallow setting tristate options to Y.
If that's 100% true, sure! But say you want to disable a feature, or enable a new one you didn't have ?
Well everything inside backports that's allowed to be tristate in the kernel, must be N or M in backports, selecting Y for a tristate (module) doesn't make sense. I'm already doing this now.
johannes
Quoted 25 lines · Johannes Berg
Quoted 7 lines · Luis R. Rodriguez
Quoted 6 lines
Are you referring to the "select" issue Hauke pointed out? Or what other handling of dependencies are you thinking of? This entire trick exists to allow compat's
Kconfigto know about the base kernel'sKconfigso the user can select the drivers that are actually possible given the base kernel. For example, if building for kernel without PCI, you wouldn't be able to select PCI drivers, etc.Quoted 6 lines
No I'm concerned about the dependency within compat-drivers itself on allowing users to build an updated driver that breaks existing stock drivers that they didn't enable on compat-drivers. One example: say a user decides to build ath9k, but also has a USB dongle and
ath9k_htcbut does not build theath9k_htcmodule, only ath9k. Loadingath9k_htcwill fail after he installs that build.Yes. That's unfortunate, but I don't really think we should _force_ users to enable
ath9k_htc? And we can't know what kind of USB devices they're going to plug in, so even "make localmodconfig" won't help all that much. We could default to all the drivers that the user has _installed_, but that'd mean maintaining aKconfig<-> module mapping (though maybe the localmodconfig script has that?) and it wouldn't be what many users want? Why build USB drivers just because your distro did if you don't own the device, after all...
Quoted 1 line
IOW, I'm not sure what we can do about this.
See commit 8ae9673098 on compat, specifically the reference to Andi Kleen's module symbol namespace. This would have solved that issue but also allow, for example, iwlwifi to use say your stock mac80211 while perhaps ath9k does need the backported wireless. Some distributions have expressed interest in this as a feature (not specifically this example, just saying any driver). The usage of the updates/ directory would still enable us to let the backport driver supersede the older stock kernel one.
Right now we don't make cfg80211, mac80211 use a prefix for their own symbols but perhaps we should, that would address this. It would not incur cost on RAM given that you'd only load the old module if a device required it, and there is no extra storage cost given that you'd still install the backport drivers as we do today.
Thoughts?
Quoted 9 lines
Quoted 9 lines
No its fine, its a general build dependency issue but that issue exists today, we just don't allow for huge different set of configurations. We allow:
* build everything
* driver-selectAnd these are tested (or at least the ones I maintain are).
Quoted 8 lines
Quoted 3 lines
Kconfigopens the door for users doing complex configurations and these could end up being reported as "issues", flooding the lists, and time on people.I'm not all that worried about that, to be honest. We should probably default to running something like "allmodconfig", just like we do now. Also, the kernel's configuration is much more complex and did you see people complain there? :-)
Actually I do, I believe randconfig is used to test builds by some folks, is how some obscure Kconfig bugs have been found, that is, a driver developer not testing building under certain Kconfig settings.
Asking users to build everything resolves this in part, and having users use menuconfig should then be something we have maintainers for those options. This lets users peg those maintainers on the bug reports found on those kconfig options. Thoughts?
Quoted 5 lines
In addition, we get the advantage that then people are unable to make things that don't build. I for one have in fact gotten plenty of bug reports because of wrong or missing ifdefs in config.mk (granted, through my own fault) -- e.g. config.mk enabling debugfs when the base kernel didn't allow for it, etc. So many issues are also solved by this.
That's indeed awesome, I can't wait!
Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Like
Kconfigrenames that went upstream and mapping them to their older name on the users's older kernels? This likely could be addressed by having these maps laid out for each kernel, but would require manual maintenance. It shouldn't be so hard if we do it *moving forward*.Quoted 8 lines
Quoted 1 line
Hmm, I never thought about that. Does that happen frequently?
Not *that* frequently but handling that is a fucking bitch. At times I wish the maintainer would send me a patch but so far only the contributors to compat have addressed that.
This is not a 'collateral evolution' per se, but certainly something to consider if we can extract logic from git to address it for us.
Quoted 8 lines
I don't really see what we could extract from git. The way I see it, the only way to really address it is to add this to compat/Kconfig:
config
NEW_SYMBOL_NAME
tristate
defaultOLD_SYMBOL_NAME(or bool, and maybe with "depends on
BACKPORT_KERNEL_X_Y_Z" added in)
Well I like the idea :D
It does mean manual work though and upon integration we'd discover this as a build failure, then we'd fix it.
Quoted 11 lines
Quoted 11 lines
Quoted 6 lines
Do you
have any examples? Presumably, an example of this would be if you have e.g.CONFIG_USB_DEVICESin an old kernel, but it'sCONFIG_USBin the new one, and all drivers that exist now have "depends on USB" while on an old kernel USB won't exist, so you wouldn't be able to select such a driver?Typically its not core stuff. Its all driver poo. At times its simply renames from upstream to downstream mapping issues. compat-drivers examples:
Quoted 6 lines
Quoted 1 line
c5cfdf2d138ba20a648990d9615c44942eef5fd3Heh, beating me with my own driver, always a good plan :-)
However this is entirely within the backports package so it's not an issue at all?
Sure, in this case, yes.
Quoted 5 lines
Quoted 1 line
8c8113109f6916c4926413aa3416cefffaa5d9b8Ditto here. In fact both of these patches just go away with my
Kconfigscheme since I prefixCPTCFG_instead ofCONFIG_and thus kernel symbols aren't used any more by theMakefileetc. stuff in backports.
OK neat.
Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
config USB
bool default y depends on USB_DEVICES && COMPAT_KERNEL_2_3_4_5or something like that?
Heh, almost the same I wrote above...
Quoted 8 lines
Quoted 1 line
Sure. I think you're trying to automate this though?
But no, I don't think I can automate this part.
What I can automate is for drivers that are part of the backports package. For *dependencies* those drivers have, like my hypothetical USB example, we'll have to maintain it manually. It shouldn't be too difficult though.
OK. I think this could be automated if we could script the question:
What dependencies did this module have in the original kernel? depmod to the rescue.
/lib/modules/$(uname -r)/modules.dep
But that does the inverse, module to symbol lookup (BTW lets be clear on language, symbol: export symbol, config: CONFIG_ poo), we need something to query the dependencies based on a given CONFIG_ and I believe we can get this, just gotta write it up. This can be done later though for sure :)
Quoted 10 lines
Quoted 3 lines
BTW I'm trying to move things to
BACKPORT_prefix for poo given that the kernel does usecompat_already for some poo. Not a big issue, whatever.Good plan. I'll make the
COMPAT_KERNEL_X_Y_ZsymbolsBACKPORT_KERNEL_X_Y_Zinstead, and also the other symbols with COMPAT in the name. For the automatic ones I pickedCPTCFG_which is hopefully unique enough and has the advantage of having the same length asCONFIG_(which is selfish of me, but it helps me maintain the code since we work inside a backport package much of the time)
Sure, thanks!
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 5 lines
Quoted 2 lines
2) I have no idea how to go from a .config file to a working
Makefile/autoconf.hsystem.Hm, we'd already have the
Makefilefor each target no? Are you trying to avoid having to ship eachMakefilefor each pulled driver?Yes, we do have the makefiles of course, as copied from the kernel sources. We want to keep them, obviously. I'm not trying to avoid that. I'm saying I have no idea how to get the CONFIG_* or rather CPTCFG_* symbols into make when it reads those makefiles.
Quoted 4 lines
Quoted 1 line
Ah yes. Can't we assume a direct mapping
CONFIG_FOO-->CPTCFG_FOO?Yes, we can mostly do that, and get rid of all the patches that modify symbols.
:D
perl -pi -e 's|CONFIG_|CONFIG_BACKPORT_|g' some/Makefile
But this would also end up renaming dependencies the Makefile might have on stuff not provided by compat, an example:
ifeq ($(CONFIG_COMPAT),y)
videodev-objs += v4l2-compat-ioctl32.o
endif
Would end up:
ifeq ($(CONFIG_BACKPORT_COMPAT),y)
videodev-objs += v4l2-compat-ioctl32.o
endif
Maybe we need to script this to not do the mangle for config options that are not present and assigned on the existing config.mk or .config (not sure what these map to exactly on your tree yet).
Quoted 11 lines
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
For autoconf we have our own thingy so far.
Yeah, but how do we get the symbols into make? That part isn't clear to me at all, even for the regular kernel.
Yeah, never figured that poo out myself... The
Makefilesomehow has to end up parsing the .config as aMakefilefor variables, how that's done no sure. For compat-drivers we just used assignments on config.mk.Yeah ... I think the makefile just has to include the .config,
I reached the same conclusion I just never saw it being done, but I also noticed I had to actually call export on these too.
Quoted 5 lines
I'll have
to check it out. Shouldn't be too difficult. I think the difficulty with the backport system is that it has to call the kernel'sMakefile, and has to export everything for that ... anyway I'll figure out something, can't be too hard.
OK cool.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Here's another issue I haven't really solved yet: All of compat must be modules, so selecting Y for any symbol, e.g. CFG80211, must be disallowed. This seems a little difficult, maybe the best choice would be to simply hack menuconfig/... tool to disallow setting tristate options to Y.
Quoted 6 lines
Quoted 2 lines
If that's 100% true, sure! But say you want to disable a feature, or enable a new one you didn't have ?
Well everything inside backports that's allowed to be tristate in the kernel, must be N or M in backports, selecting Y for a tristate (module) doesn't make sense. I'm already doing this now.
Ah tristate, yeah that make sense and would be a good idea to help avoid issues.
Luis
Quoted 9 lines · Luis R. Rodriguez
See commit
8ae9673098on compat, specifically the reference to Andi Kleen's module symbol namespace. This would have solved that issue but also allow, for example, iwlwifi to use say your stock mac80211 while perhaps ath9k does need the backported wireless. Some distributions have expressed interest in this as a feature (not specifically this example, just saying any driver). The usage of the updates/ directory would still enable us to let the backport driver supersede the older stock kernel one.
Quoted 7 lines
Right now we don't make cfg80211, mac80211 use a prefix for their own symbols but perhaps we should, that would address this. It would not incur cost on RAM given that you'd only load the old module if a device required it, and there is no extra storage cost given that you'd still install the backport drivers as we do today.
Thoughts?
Ok yes, this is still an interesting thing that we might consider doing. I consider it orthogonal to my current work though.
Tackling this would be tricky though, you don't want to patch all the different uses of a symbol, so you'd have to do something like module symbol namespaces, and teach the build environment about it, or something like that. Maybe it could be done with the linker's symbol versioning.
Anyway -- this is interesting but not my (primary) goal right now.
Quoted 3 lines
Actually I do, I believe randconfig is used to test builds by some folks, is how some obscure
Kconfigbugs have been found, that is, a driver developer not testing building under certainKconfigsettings.
Yes, of course, this happens all the time, but it's just a bug.
Quoted 4 lines
Asking users to build everything resolves this in part, and having users use menuconfig should then be something we have maintainers for those options. This lets users peg those maintainers on the bug reports found on those kconfig options. Thoughts?
Users can attach their backport .config file to the bug report, so this should be fairly easy to identify in the bug. Also, this is really not much different from getting a "build failed with these options" bug on the kernel. It's different, of course, but many such issues would probably replicate in the kernel as well.
Quoted 10 lines
Quoted 5 lines
config
NEW_SYMBOL_NAME
tristate
defaultOLD_SYMBOL_NAME(or bool, and maybe with "depends on
BACKPORT_KERNEL_X_Y_Z" added in)Well I like the idea :D
It does mean manual work though and upon integration we'd discover this as a build failure, then we'd fix it.
Yes. We don't get to magically avoid porting work :-)
Quoted 10 lines
Quoted 4 lines
What I can automate is for drivers that are part of the backports package. For *dependencies* those drivers have, like my hypothetical USB example, we'll have to maintain it manually. It shouldn't be too difficult though.
OK. I think this could be automated if we could script the question:
What dependencies did this module have in the original kernel? depmod to the rescue.
Quoted 7 lines
/lib/modules/$(uname -r)/modules.dep
But that does the inverse, module to symbol lookup (BTW lets be clear on language, symbol: export symbol, config:
CONFIG_poo), we need something to query the dependencies based on a givenCONFIG_and I believe we can get this, just gotta write it up. This can be done later though for sure :)
Yeah, agree, later.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Ah yes. Can't we assume a direct mapping
CONFIG_FOO-->CPTCFG_FOO?Yes, we can mostly do that, and get rid of all the patches that modify symbols.
:D
perl -pi -e 's|CONFIG_|CONFIG_BACKPORT_|g' some/Makefile
[...]
Quoted 3 lines
Maybe we need to script this to not do the mangle for config options that are not present and assigned on the existing config.mk or .config (not sure what these map to exactly on your tree yet).
Yes. I did pretty much exactly that. There's no config.mk, I only have a .config file and that's generated at build time via the kconfig mechanism.
Quoted 4 lines
Quoted 1 line
Yeah ... I think the makefile just has to include the .config,
I reached the same conclusion I just never saw it being done, but I also noticed I had to actually call export on these too.
include .config
export
works for me. Just "export" means export all.
johannes
OK sweet, so just to keep track of pending work then:
https://backports.wiki.kernel.org/index.php/Todo
Luis
Quoted 3 lines · Johannes Berg
Quoted 3 lines
1) Ability to copy only some drivers.
2) To make that easier: split all patches per *file* in addition to per change.
3) Separate the output from the compat-drivers tree.
done.
Quoted 1 line
Quoted 1 line
4)
Kconfig.
pretty much done -- right now users can get into unbuildable situations, I obviously want to avoid.
I also decided to merge compat and compat-drivers -- right now I see no value in them being combined. I therefore created a new backports.git tree.
Note that this is very very raw. It doesn't even have the ability to *build* the drivers yet, which is kinda the whole point ... But I had to start somewhere, and obviously that was at step 1: building the sources :-)
It's all here, if anyone wants to risk a look.
http://git.sipsolutions.net/?p=backports.git;a=summary
I'll continue next week, most likely.
johannes