Skip to contentSkip to conversation

compat-drivers wishlist

DISCUSSION Johannes Berg 4 participants 18 messages Mar 28, 2013 – Mar 30, 2013

lore
Share & tools
lore
#0 lore raw

Hi,

I've long been fed up with the state of compat's build system, so I figured I'd start by soliciting wishlist items.

FWIW, I think that the whole compat-drivers/scripts/ dir needs to be rewritten for this ... :-)

My personal wishlist is below. I realize that some of these wishes cause contradictions/problems in their implementation, but nonetheless I have them :-)

1) Ability to copy only some drivers.
We're interested in using compat-drivers to ship our driver to
customers for
older kernels. We're not at all interested in shipping DRM, Atheros
drivers,
etc.

2) To make that easier: split all patches per *file* in addition to per change.
That way, it would be possible to apply only patches for those files
that are
actually in the tree. Right now, I heavily patch out those patches
that aren't
really need.
I suppose the best way would be to have directories for each change
and change
the script that applies them to parse the patch header first. Or just
ignore
patches that fail with "no file", but I don't know if that's easy to
determine
from the patch exit status code.

3) Separate the output from the compat-drivers tree.
There's no reason to ship patches/, most of scripts/ etc. in the
resulting
tarball, in fact it's just confusing. Having the output in a separate directory, copying there only the few things that are needed, would
be very
helpful.

4) Kconfig.
This is a bit conflicting with 1) because a valid Kconfig structure
requires
all the sub-Kconfig files exist. Maybe ship all Kconfig files, or
automate a
way to remove "source" statements for files that don't exist
(probably easy)

I'll leave it at that for now, and follow up with some concrete implementation ideas :-)

Any other thoughts?

johannes

#1 lore raw
Quoted 12 lines · Johannes Berg

Hi,

I've long been fed up with the state of compat's build system, so I figured I'd start by soliciting wishlist items.

FWIW, I think that the whole compat-drivers/scripts/ dir needs to be rewritten for this ... :-)

My personal wishlist is below. I realize that some of these wishes cause contradictions/problems in their implementation, but nonetheless I have them :-)

Quoted 6 lines

1) Ability to copy only some drivers.
We're interested in using compat-drivers to ship our driver to
customers for
older kernels. We're not at all interested in shipping DRM, Atheros
drivers,
etc.

Seems reasonable. Maybe admin-update can read your own .config

Quoted 6 lines

2) To make that easier: split all patches per *file* in addition to per change.
That way, it would be possible to apply only patches for those files
that are
actually in the tree. Right now, I heavily patch out those patches
that aren't

Quoted 8 lines

really need.
I suppose the best way would be to have directories for each change
and change
the script that applies them to parse the patch header first. Or just
ignore
patches that fail with "no file", but I don't know if that's easy to
determine
from the patch exit status code.

GNU patch could use some love. See commit 4a1a94f93 on compat-drivers. Perhaps this is an area we'd be interested in extending functionality as well. Having collateral evolutions split up each into their own directory is a fine approach as well, but lets think long term here, given that we're seeing what short term script/hack solutions can cost us.

Maybe we need a context patch parser that given a .config will only apply patches for the desired files. For this to work we'd need a mapping of CONFIG_FOO symbols to respective driver path files regexps. The Linux kernel MAINTAINERS file comes to mind as something close to get us there, but that does not yet use CONFIG_ symbols, the title is a bit loose.

Kconfig however gives us the config to let us build a target, and the Makefile should let us extract its target dependencies, so I wonder if we can use this to dynamically build kconfig to target maps.

Quoted 7 lines

3) Separate the output from the compat-drivers tree.
There's no reason to ship patches/, most of scripts/ etc. in the
resulting
tarball, in fact it's just confusing. Having the output in a separate directory, copying there only the few things that are needed, would
be very
helpful.

Sure, makes sense.

Quoted 7 lines

4) Kconfig.
This is a bit conflicting with 1) because a valid Kconfig structure
requires
all the sub-Kconfig files exist. Maybe ship all Kconfig files, or
automate a
way to remove "source" statements for files that don't exist
(probably easy)

I think its possible to remove not desired source code based on reading .config. This can be done as a pass if the Kconfig to Makefile thing is figured out. Then as a secondary step I think we can remove directories that have just a Kconfig / Makefile, but that's theoretical, in practice this may be harder, the biggest issue I think would be to ensure removal of the calling 'source foo/Kconfig' entry, or we extend Kconfig to support Kconfig imports that don't exist, sort of with Makefile and the -include foo.mk trick.

Quoted 4 lines

I'll leave it at that for now, and follow up with some concrete implementation ideas :-)

Any other thoughts?

There, some feedback, thanks for taking the time to dumping your ideas.

Luis

#2 lore raw
Quoted 10 lines · Luis R. Rodriguez
Quoted 7 lines · Johannes Berg

4) Kconfig.
This is a bit conflicting with 1) because a valid Kconfig structure
requires
all the sub-Kconfig files exist. Maybe ship all Kconfig files, or
automate a
way to remove "source" statements for files that don't exist
(probably easy)

Quoted 8 lines

I think its possible to remove not desired source code based on reading .config. This can be done as a pass if the Kconfig to Makefile thing is figured out. Then as a secondary step I think we can remove directories that have just a Kconfig / Makefile, but that's theoretical, in practice this may be harder, the biggest issue I think would be to ensure removal of the calling 'source foo/Kconfig' entry, or we extend Kconfig to support Kconfig imports that don't exist, sort of with Makefile and the -include foo.mk trick.

Another thing I should mention here about Kconfig.

'make localmodconfig' is great and IMHO something which we should strive for as well for end users. The issue comes in when you have to build something for remote systems. For this I wrote a script a while ago to let us extract only the necessary components from the kernel, scp to target, and extract back our .config [0]. This is hacky but something to keep in mind.

[0] https://lkml.org/lkml/2009/11/12/350

Luis

#3 lore raw
Quoted 8 lines · Luis R. Rodriguez

Another thing I should mention here about Kconfig.

'make localmodconfig' is great and IMHO something which we should strive for as well for end users. The issue comes in when you have to build something for remote systems. For this I wrote a script a while ago to let us extract only the necessary components from the kernel, scp to target, and extract back our .config [0]. This is hacky but something to keep in mind.

Yes, that's another interesting thing to do. I myself tend to think it's less important as even building all of compat shouldn't take all that much time. OTOH, it's clearly a lot more than what you usually want.

The tricky part here is that for those users that have most reason to use compat, those who need it for new driver modules, it won't work because they can't already be loaded on the system.

Anyway I don't see why we couldn't just copy the streamline config perl script, with the Kconfig tree we build it would probably work just fine.

johannes

#4 lore raw
In reply to Luis R. Rodriguez
Quoted 3 lines · Luis R. Rodriguez
Quoted 1 line

1) Ability to copy only some drivers.

Seems reasonable. Maybe admin-update can read your own .config

This is a tricky proposition, see below & my other mail on Kconfig thoughts.

Quoted 1 line

GNU patch could use some love.

I'd hate to have to rely on a modified version of that :-)

FWIW, I think there's a python library to apply (unified) patches, if we want to get fancy.

Quoted 1 line

See commit 4a1a94f93 on compat-drivers.

Ah, what's that for? Refreshing patches? I've never seen dependencies in the compat patches so mostly I refresh manually, but yeah, the refresh part is something I'd have to think about. Though, to be honest, when you can just blow away your output directory and try again I'm not too worried about this. Trying to apply the patches could just leave it in the state you're in, and then you modify them until it works?

Quoted 3 lines

Maybe we need a context patch parser that given a .config will only apply patches for the desired files. For this to work we'd need a mapping of CONFIG_FOO symbols to respective driver path files regexps.

Even assuming we do have such a mapping, I think it's the wrong approach and causes a chicken-and-egg problem.

As I wrote in my other mail, you don't have a .config file before you know the target kernel. This is because, for example, there's no way to know if PCI is enabled in the target kernel, and thus sometimes you get PCI drivers and sometimes not. Also I think maintenance and driver split is not usually per Kconfig symbol but per directory. There are many more Kconfig symbols, after all.

Also above you said you could do the source copying (or pruning) based on the .config file. This has the exact same issue, I'd rather have the copying or pruning be based on directories.

Basically I'm saying the .config file only exists on the target build system, while the selective copying/pruning needs to be done on the tarball creation system.

Quoted 4 lines
Quoted 1 line

4) Kconfig.

I think its possible to remove not desired source code based on reading .config.

See above.

Oh, another issue I haven't addressed is default configurations, but I think that's easy to do by providing something like the kernel's defconfig.

johannes

#5 lore raw
Quoted 16 lines · Johannes Berg
Quoted 3 lines · Luis R. Rodriguez
Quoted 1 line

1) Ability to copy only some drivers.

Seems reasonable. Maybe admin-update can read your own .config

This is a tricky proposition, see below & my other mail on Kconfig thoughts.

Quoted 1 line

GNU patch could use some love.

I'd hate to have to rely on a modified version of that :-)

FWIW, I think there's a python library to apply (unified) patches, if we want to get fancy.

Quoted 8 lines
Quoted 1 line

See commit 4a1a94f93 on compat-drivers.

Ah, what's that for? Refreshing patches? I've never seen dependencies in the compat patches so mostly I refresh manually, but yeah, the refresh part is something I'd have to think about. Though, to be honest, when you can just blow away your output directory and try again I'm not too worried about this. Trying to apply the patches could just leave it in the state you're in, and then you modify them until it works?

Refreshing patches should not be a problem. Normally I use "./scripts/admin-refresh.sh -n -p -c -u refresh" to refresh the patches and this should work with some minor changes with the proposal you made.

Quoted 14 lines
Quoted 3 lines

Maybe we need a context patch parser that given a .config will only apply patches for the desired files. For this to work we'd need a mapping of CONFIG_FOO symbols to respective driver path files regexps.

Even assuming we do have such a mapping, I think it's the wrong approach and causes a chicken-and-egg problem.

As I wrote in my other mail, you don't have a .config file before you know the target kernel. This is because, for example, there's no way to know if PCI is enabled in the target kernel, and thus sometimes you get PCI drivers and sometimes not. Also I think maintenance and driver split is not usually per Kconfig symbol but per directory. There are many more Kconfig symbols, after all.

Quoted 7 lines

Also above you said you could do the source copying (or pruning) based on the .config file. This has the exact same issue, I'd rather have the copying or pruning be based on directories.

Basically I'm saying the .config file only exists on the target build system, while the selective copying/pruning needs to be done on the tarball creation system.

How do you want to make the tar generation being configurable? The build time Kconfig options should be generated from the Kconfig files provided by the kernel and not by some own files.
I think some manual edited config file with the directories or files to be included sounds good. This function would not be used by the normal user so that should be good enough.

Quoted 12 lines
Quoted 4 lines
Quoted 1 line

4) Kconfig.

I think its possible to remove not desired source code based on reading .config.

See above.

Oh, another issue I haven't addressed is default configurations, but I think that's easy to do by providing something like the kernel's defconfig.

johannes

#6 lore raw
Quoted 10 lines · Hauke Mehrtens
Quoted 6 lines

Ah, what's that for? Refreshing patches? I've never seen dependencies in the compat patches so mostly I refresh manually, but yeah, the refresh part is something I'd have to think about. Though, to be honest, when you can just blow away your output directory and try again I'm not too worried about this. Trying to apply the patches could just leave it in the state you're in, and then you modify them until it works?

Refreshing patches should not be a problem. Normally I use "./scripts/admin-refresh.sh -n -p -c -u refresh" to refresh the patches and this should work with some minor changes with the proposal you made.

Ok. I have no idea what that does, since most of the time my "refreshing" isn't really refreshing but making them apply again, and I do that manually.

Quoted 7 lines
Quoted 3 lines

Basically I'm saying the .config file only exists on the target build system, while the selective copying/pruning needs to be done on the tarball creation system.

How do you want to make the tar generation being configurable? The build time Kconfig options should be generated from the Kconfig files provided by the kernel and not by some own files.

Right, the build time Kconfig should just be copied as other source files.

Quoted 3 lines

I think some manual edited config file with the directories or files to be included sounds good. This function would not be used by the normal user so that should be good enough.

Indeed, that's what I was thinking. Sorry for not making that clear.

johannes

#7 lore raw
In reply to Johannes Berg
Quoted 3 lines

1) Ability to copy only some drivers.
2) To make that easier: split all patches per *file* in addition to per change.

You could think that an alternative to these would be for me to post-process the sources and remove everything I don't care about. However, that gets tricky because my compat package is created from my own tree, and there the patches don't always apply. Therefore these wishes.

I think for 2), I'd propose the following structure:

patches/collateral-evolutions/network/0001-netdev_ops/

	-> INFO
	-> rndis.patch
	-> usbnet.patch
	-> ath6kl.patch

Since the filename doesn't matter all that much, an automated conversion could just split the patch per file and store it in a filename obtained as follows:
drivers/net/usb/rndis_host.c
-> drivers_net_usb_rndis_host.c

Easy :-)

The script to apply them gets a little more difficult, but I think we should start using python for the tree-creation-time scripting which will make this easy enough. Just walk the tree, etc.

Quoted 1 line

3) Separate the output from the compat-drivers tree.

Should be easy, though it probably requires rewriting admin-update. OTOH, admin-refresh etc. all pretty much goes away since you can just rm -rf the output directory.

Quoted 1 line

4) Kconfig.

So my wishes 1-3 apply at "tree creation" time, which is when the compat source tree (and later tarball) is created and patched for the backports. I've already suggested to rewrite that scripting in python, and I'm willing to invest time in that if it stands a chance of getting accepted :-)

This one is a bit more difficult. On the one hand we need to have all the symbols from the kernel, but we don't want to leak them into the compat namespace. Right now, for example, we do things like renaming CONFIG_USB_NET_RNDIS_HOST to CONFIG_USB_NET_COMPAT_RNDIS_HOST which we really should do for everything -- we get into trouble occasionally because somebody enables something in the base kernel and then tries to disable it in compat, which doesn't work.

I've been thinking a while about this, and I think I pretty much found a solution.

First of all, it's easy to write a small Kconfig program (using the existing Kconfig C code of course) that generates a list of all symbols present in a given Kconfig tree, even disabled ones. Here's the program, you just have to link it with zconf.tab.c and pass a root Kconfig file as the only command line argument:
http://p.sipsolutions.net/3300bc91340ae747.txt

Equipped with a list of config symbols, we can go through the sources and Makefiles and replace CONFIG_* with (e.g.) CPTCFG_* (and also CONFIG_*_MODULE). We can't just replace CONFIG_ with CPTCFG_ because then we get kernel symbols wrong. We could rename all CONFIG_ to CPTCFG_ and import the kernel symbols, which we have to do for Kconfig anyway, but I feel that doing that would make things more brittle. (*)

Now obviously if we have a local Kconfig tree, it's pretty useless. The reason is that we'll have things like

config MYDRIVER_PCI

	tristate "my good PCI driver"
	depends on PCI && HAS_IOMEM

config MYDRIVER_USB
tristate "my good USB driver"
depends on USB

or something like that, and the Kconfig tree we have clearly doesn't know about PCI, HAS_IOMEM or USB. We'll have to teach it, which is surprisingly easy:

path/to/syms-program Kconfig > local-symbol-list
sed -i 's/$/=/' local-symbol-list
grep -v -f local-symbol-list $KLIB_BUILD/.config | dotconf-to-kconf.py > Kconfig.kernel

dotconf-to-kconf.py is this little tool: http://p.sipsolutions.net/986bfbd397ad51c5.txt
(Clearly it has to be rewritten in a different language since this has
to run at build time, not just at tarball creation time, but that's easy to do)

Now we can "source Kconfig.kernel" into our local Kconfig tree, and the "depends on USB" can magically work the right way. For the symbol walk (syms program above) we'll have to include an empty file, but that's not terribly difficult either.

Oh, whenever we execute menuconfig/oldconfig/... we set the environment variable CONFIG_ to CPTCFG_ -- this will make the tool output CPTCFG_ instead of CONFIG_ as the prefix for all the output variables.

This is about as far as I got with the implementation. There are a few issues left.

1) All of the kernel symbols like USB get put into the .config file, e.g. CPTCFG_USB. This doesn't actually matter all that much because they're prefixed CPTCFG_ and the code we compile still looks for CONFIG_ for the kernel symbols -- remember that we only renamed the ones we found in our local Kconfig tree.

2) I have no idea how to go from a .config file to a working Makefile/autoconf.h system.

3) If KLIB_BUILD dir or the kernel config changes, this has to be regenerated and oldconfig must be run. For instance, if USB was now disabled in the base kernel then we shouldn't attempt to build USB drivers, etc. This is a bit tricky because we also don't want to redo everything for each "make". I'm not sure how much we care about this, but we can probably solve it by storing a checksum of $KLIB_BUILD/.config and doing the preparation steps only if that changes, or so.

4) This assumes KLIB_BUILD is configured externally, it's not possible to configure that through menuconfig since the Kconfig tree needs to be built first. I don't think this is a problem, but it looks like Luis wanted to put that into Kconfig so I'm saying it won't really work.

Thoughts?

johannes

(*) post-scriptum: this can and should actually happen at tree creation time, not at build time

#8 lore raw
Quoted 92 lines · Johannes Berg
Quoted 3 lines

1) Ability to copy only some drivers.
2) To make that easier: split all patches per *file* in addition to per change.

You could think that an alternative to these would be for me to post-process the sources and remove everything I don't care about. However, that gets tricky because my compat package is created from my own tree, and there the patches don't always apply. Therefore these wishes.

I think for 2), I'd propose the following structure:

patches/collateral-evolutions/network/0001-netdev_ops/

	-> INFO
	-> rndis.patch
	-> usbnet.patch
	-> ath6kl.patch

Since the filename doesn't matter all that much, an automated conversion could just split the patch per file and store it in a filename obtained as follows:
drivers/net/usb/rndis_host.c
-> drivers_net_usb_rndis_host.c

Easy :-)

The script to apply them gets a little more difficult, but I think we should start using python for the tree-creation-time scripting which will make this easy enough. Just walk the tree, etc.

Quoted 1 line

3) Separate the output from the compat-drivers tree.

Should be easy, though it probably requires rewriting admin-update. OTOH, admin-refresh etc. all pretty much goes away since you can just rm -rf the output directory.

Quoted 1 line

4) Kconfig.

So my wishes 1-3 apply at "tree creation" time, which is when the compat source tree (and later tarball) is created and patched for the backports. I've already suggested to rewrite that scripting in python, and I'm willing to invest time in that if it stands a chance of getting accepted :-)

This one is a bit more difficult. On the one hand we need to have all the symbols from the kernel, but we don't want to leak them into the compat namespace. Right now, for example, we do things like renaming CONFIG_USB_NET_RNDIS_HOST to CONFIG_USB_NET_COMPAT_RNDIS_HOST which we really should do for everything -- we get into trouble occasionally because somebody enables something in the base kernel and then tries to disable it in compat, which doesn't work.

I've been thinking a while about this, and I think I pretty much found a solution.

First of all, it's easy to write a small Kconfig program (using the existing Kconfig C code of course) that generates a list of all symbols present in a given Kconfig tree, even disabled ones. Here's the program, you just have to link it with zconf.tab.c and pass a root Kconfig file as the only command line argument:
http://p.sipsolutions.net/3300bc91340ae747.txt

Equipped with a list of config symbols, we can go through the sources and Makefiles and replace CONFIG_* with (e.g.) CPTCFG_* (and also CONFIG_*_MODULE). We can't just replace CONFIG_ with CPTCFG_ because then we get kernel symbols wrong. We could rename all CONFIG_ to CPTCFG_ and import the kernel symbols, which we have to do for Kconfig anyway, but I feel that doing that would make things more brittle. (*)

Now obviously if we have a local Kconfig tree, it's pretty useless. The reason is that we'll have things like

config MYDRIVER_PCI

	tristate "my good PCI driver"
	depends on PCI && HAS_IOMEM

config MYDRIVER_USB
tristate "my good USB driver"
depends on USB

or something like that, and the Kconfig tree we have clearly doesn't know about PCI, HAS_IOMEM or USB. We'll have to teach it, which is surprisingly easy:

path/to/syms-program Kconfig > local-symbol-list
sed -i 's/$/=/' local-symbol-list
grep -v -f local-symbol-list $KLIB_BUILD/.config | dotconf-to-kconf.py > Kconfig.kernel

dotconf-to-kconf.py is this little tool: http://p.sipsolutions.net/986bfbd397ad51c5.txt
(Clearly it has to be rewritten in a different language since this has
to run at build time, not just at tarball creation time, but that's easy to do)

Quoted 4 lines

Now we can "source Kconfig.kernel" into our local Kconfig tree, and the "depends on USB" can magically work the right way. For the symbol walk (syms program above) we'll have to include an empty file, but that's not terribly difficult either.

We also have to change all the select SOME_KERNEL_OPTION into a depends SOME_KERNEL_OPTION, because the options of the used kernel can not be changed.

Quoted 16 lines

Oh, whenever we execute menuconfig/oldconfig/... we set the environment variable CONFIG_ to CPTCFG_ -- this will make the tool output CPTCFG_ instead of CONFIG_ as the prefix for all the output variables.

This is about as far as I got with the implementation. There are a few issues left.

1) All of the kernel symbols like USB get put into the .config file, e.g. CPTCFG_USB. This doesn't actually matter all that much because they're prefixed CPTCFG_ and the code we compile still looks for CONFIG_ for the kernel symbols -- remember that we only renamed the ones we found in our local Kconfig tree.

2) I have no idea how to go from a .config file to a working Makefile/autoconf.h system.

Quoted 8 lines

3) If KLIB_BUILD dir or the kernel config changes, this has to be regenerated and oldconfig must be run. For instance, if USB was now disabled in the base kernel then we shouldn't attempt to build USB drivers, etc. This is a bit tricky because we also don't want to redo everything for each "make". I'm not sure how much we care about this, but we can probably solve it by storing a checksum of $KLIB_BUILD/.config and doing the preparation steps only if that changes, or so.

compat should be changed in this way to not always backport any e.g. all USB stuff and depend on USB stuff if the USB subsystem was not activated. I think this should be modularized to have one *.ko for the USB stuff, one for the PCMCIA stuff and so on.
Currently compat.ko depends on USB for some kernel versions.

This could also be done later.

Quoted 12 lines

4) This assumes KLIB_BUILD is configured externally, it's not possible to configure that through menuconfig since the Kconfig tree needs to be built first. I don't think this is a problem, but it looks like Luis wanted to put that into Kconfig so I'm saying it won't really work.

Thoughts?

johannes

(*) post-scriptum: this can and should actually happen at tree creation time, not at build time

#9 lore raw
Quoted 8 lines · Hauke Mehrtens
Quoted 4 lines

Now we can "source Kconfig.kernel" into our local Kconfig tree, and the "depends on USB" can magically work the right way. For the symbol walk (syms program above) we'll have to include an empty file, but that's not terribly difficult either.

We also have to change all the select SOME_KERNEL_OPTION into a depends SOME_KERNEL_OPTION, because the options of the used kernel can not be changed.

Huh, good point. I didn't think about that, and it probably doesn't even exist? But yeah, I agree, we need to do this. It's not terribly difficult though -- after we have a list of local symbols we can go through and rewrite "select NONLOCAL" to "depends on NONLOCAL".

Quoted 5 lines

compat should be changed in this way to not always backport any e.g. all USB stuff and depend on USB stuff if the USB subsystem was not activated. I think this should be modularized to have one *.ko for the USB stuff, one for the PCMCIA stuff and so on.
Currently compat.ko depends on USB for some kernel versions.

I don't think compat backports USB/PCMCIA/...? I was under the impression that it was necessary to be from the base kernel anyway, but I don't really know. In theory I agree, anything that's backported in compat needs to be included into the Kconfig somehow, particularly things like this.

Actually, I guess we do have things like that now, just not USB/PCMCIA:

CONFIG_COMPAT_FIRMWARE_CLASS
CONFIG_COMPAT_NET_SCH_CODEL
CONFIG_COMPAT_NET_SCH_FQ_CODEL
CONFIG_COMPAT_KFIFO
CONFIG_COMPAT_CORDIC
CONFIG_COMPAT_CRC8

and something about CONFIG_GENERIC_ATOMIC64 on non-64 bit platforms.

This is interesting! It probably means we need to generate a Kconfig file for the kernel versions as well, at build time:

config COMPAT_KERNEL_2_6_26
def_bool y

Obviously then compat relies more on compat-drivers to actually build? It could have its own Kconfig file, and probably best if the creation of this Kconfig.compat file is done through a script there.

Anyway this is special. So we can treat it specially, and do something like

config COMPAT_CORDIC
bool
default !CORDIC

which should work just fine since we imported the symbols from the base kernel. We'll then get a CPTCFG_COMPAT_CORDIC symbol that we can use to build (or not) the cordic code.

For something that depends on a specific kernel version it's just as simple:

config COMPAT_FIRMWARE_CLASS

	tristate
	default m if COMPAT_KERNEL_2_6_33 && !COMPAT_RHEL_6_1

or so.

johannes

#10 lore raw
In reply to Johannes Berg
Quoted 10 lines · Johannes Berg
Quoted 3 lines

1) Ability to copy only some drivers.
2) To make that easier: split all patches per *file* in addition to per change.

You could think that an alternative to these would be for me to post-process the sources and remove everything I don't care about. However, that gets tricky because my compat package is created from my own tree, and there the patches don't always apply. Therefore these wishes.

Quoted 7 lines

I think for 2), I'd propose the following structure:

patches/collateral-evolutions/network/0001-netdev_ops/

        -> INFO
        -> rndis.patch
        -> usbnet.patch
        -> ath6kl.patch

That's fine, and in fact with the other long term things I'm looking at, such as SmPL to exrpress these collateral evolutions, is to categorize the collateral evolutions into different types and see if we can get *all* of them implemented in compat alone. That is, strive to get rid of all patches required.

This requires a study of the different types of collateral evolutions we have. compat already addresses the ones we can implement without requiring changing the upstream drivers. I'm now starting to address -- what is upstream willing to do to help with backports. An example is the static inlines.

https://lkml.org/lkml/2013/3/28/187

If the above patch is accepted I'd then try to send upstream the netdev_attach_ops() one, and that would nuke that entire patch from compat-drivers.

Quoted 7 lines

Since the filename doesn't matter all that much, an automated conversion could just split the patch per file and store it in a filename obtained as follows:
drivers/net/usb/rndis_host.c
-> drivers_net_usb_rndis_host.c

Easy :-)

True. I welcome this change if it helps you.

Quoted 3 lines

The script to apply them gets a little more difficult, but I think we should start using python for the tree-creation-time scripting which will make this easy enough. Just walk the tree, etc.

ACK.

Quoted 6 lines
Quoted 1 line

3) Separate the output from the compat-drivers tree.

Should be easy, though it probably requires rewriting admin-update. OTOH, admin-refresh etc. all pretty much goes away since you can just rm -rf the output directory.

Quoted 7 lines
Quoted 1 line

4) Kconfig.

So my wishes 1-3 apply at "tree creation" time, which is when the compat source tree (and later tarball) is created and patched for the backports. I've already suggested to rewrite that scripting in python, and I'm willing to invest time in that if it stands a chance of getting accepted :-)

I've long said bring it. I'm on the Python bandwagon now, my first rewrite was ckmake.

Quoted 7 lines

This one is a bit more difficult. On the one hand we need to have all the symbols from the kernel, but we don't want to leak them into the compat namespace. Right now, for example, we do things like renaming CONFIG_USB_NET_RNDIS_HOST to CONFIG_USB_NET_COMPAT_RNDIS_HOST which we really should do for everything -- we get into trouble occasionally because somebody enables something in the base kernel and then tries to disable it in compat, which doesn't work.

True.

Quoted 2 lines

I've been thinking a while about this, and I think I pretty much found a solution.

Yay.

Quoted 24 lines

First of all, it's easy to write a small Kconfig program (using the existing Kconfig C code of course) that generates a list of all symbols present in a given Kconfig tree, even disabled ones. Here's the program, you just have to link it with zconf.tab.c and pass a root Kconfig file as the only command line argument:
http://p.sipsolutions.net/3300bc91340ae747.txt

Equipped with a list of config symbols, we can go through the sources and Makefiles and replace CONFIG_* with (e.g.) CPTCFG_* (and also CONFIG_*_MODULE). We can't just replace CONFIG_ with CPTCFG_ because then we get kernel symbols wrong. We could rename all CONFIG_ to CPTCFG_ and import the kernel symbols, which we have to do for Kconfig anyway, but I feel that doing that would make things more brittle. (*)

Now obviously if we have a local Kconfig tree, it's pretty useless. The reason is that we'll have things like

config MYDRIVER_PCI
tristate "my good PCI driver"
depends on PCI && HAS_IOMEM
config MYDRIVER_USB
tristate "my good USB driver"
depends on USB

Quoted 7 lines

or something like that, and the Kconfig tree we have clearly doesn't know about PCI, HAS_IOMEM or USB. We'll have to teach it, which is surprisingly easy:

path/to/syms-program Kconfig > local-symbol-list
sed -i 's/$/=/' local-symbol-list
grep -v -f local-symbol-list $KLIB_BUILD/.config | dotconf-to-kconf.py > Kconfig.kernel

Neat trick! This doesn't handle the mapping of depdendencies but certainly already makes good assumptions given that you *have* these things built. It does not however take into consideration for example that a user may say -- let me build cfg80211 alone, build it, install it and now have their 802.11 driver also selected (if they didn't).

Quoted 4 lines

dotconf-to-kconf.py is this little tool: http://p.sipsolutions.net/986bfbd397ad51c5.txt
(Clearly it has to be rewritten in a different language since this has
to run at build time, not just at tarball creation time, but that's easy to do)

I actually like the simplicity of it in Python and prefer it if we can live with it, but given that we follow the kernel build system I'd rather see the kernel first take the bullet on swallowing Python... not sure, anyway doesn't matter much if its simple.

Quoted 11 lines

Now we can "source Kconfig.kernel" into our local Kconfig tree, and the "depends on USB" can magically work the right way. For the symbol walk (syms program above) we'll have to include an empty file, but that's not terribly difficult either.

Oh, whenever we execute menuconfig/oldconfig/... we set the environment variable CONFIG_ to CPTCFG_ -- this will make the tool output CPTCFG_ instead of CONFIG_ as the prefix for all the output variables.

This is about as far as I got with the implementation. There are a few issues left.

Like Kconfig renames that went upstream and mapping them to their older name on the users's older kernels? This likely could be addressed by having these maps laid out for each kernel, but would require manual maintenance. It shouldn't be so hard if we do it *moving forward*.

The other bigger thing I think is the dependency map output not existing so it'd let users break a build / existing dep.

Quoted 8 lines

1) All of the kernel symbols like USB get put into the .config file, e.g. CPTCFG_USB. This doesn't actually matter all that much because they're prefixed CPTCFG_ and the code we compile still looks for CONFIG_ for the kernel symbols -- remember that we only renamed the ones we found in our local Kconfig tree.

2) I have no idea how to go from a .config file to a working Makefile/autoconf.h system.

Hm, we'd already have the Makefile for each target no? Are you trying to avoid having to ship each Makefile for each pulled driver? For autoconf we have our own thingy so far.
Maybe I'm not getting what you mean here.

Quoted 8 lines

3) If KLIB_BUILD dir or the kernel config changes, this has to be regenerated and oldconfig must be run. For instance, if USB was now disabled in the base kernel then we shouldn't attempt to build USB drivers, etc. This is a bit tricky because we also don't want to redo everything for each "make". I'm not sure how much we care about this, but we can probably solve it by storing a checksum of $KLIB_BUILD/.config and doing the preparation steps only if that changes, or so.

Sure.

Quoted 4 lines

4) This assumes KLIB_BUILD is configured externally, it's not possible to configure that through menuconfig since the Kconfig tree needs to be built first. I don't think this is a problem, but it looks like Luis wanted to put that into Kconfig so I'm saying it won't really work.

I didn't want to put it into Kconfig actually, I was just throwing the variables on the menuconfig display to ensure the user building is aware of the sources of everything.

Luis

#11 lore raw
Quoted 11 lines · Luis R. Rodriguez

This requires a study of the different types of collateral evolutions we have. compat already addresses the ones we can implement without requiring changing the upstream drivers. I'm now starting to address -- what is upstream willing to do to help with backports. An example is the static inlines.

https://lkml.org/lkml/2013/3/28/187

If the above patch is accepted I'd then try to send upstream the netdev_attach_ops() one, and that would nuke that entire patch from compat-drivers.

Right, this is a goal in itself, but I don't think we'll get rid of all patches. Consider things like the completely different API for multicast lists, for example.

Quoted 9 lines
Quoted 7 lines

Since the filename doesn't matter all that much, an automated conversion could just split the patch per file and store it in a filename obtained as follows:
drivers/net/usb/rndis_host.c
-> drivers_net_usb_rndis_host.c

Easy :-)

True. I welcome this change if it helps you.

It would help me quite a bit. I suppose I could actually send you patches for this soon. OTOH, without addressing my wishlist item 1) I probably won't update to a current compat-drivers base ... I heavily patch admin-update.sh now.

Quoted 7 lines
Quoted 3 lines

path/to/syms-program Kconfig > local-symbol-list
sed -i 's/$/=/' local-symbol-list
grep -v -f local-symbol-list $KLIB_BUILD/.config | dotconf-to-kconf.py > Kconfig.kernel

Neat trick! This doesn't handle the mapping of depdendencies but certainly already makes good assumptions given that you *have* these things built.

Are you referring to the "select" issue Hauke pointed out? Or what other handling of dependencies are you thinking of? This entire trick exists to allow compat's Kconfig to know about the base kernel's Kconfig so the user can select the drivers that are actually possible given the base kernel. For example, if building for kernel without PCI, you wouldn't be able to select PCI drivers, etc.

Quoted 3 lines

It does not however take into consideration for example that a user may say -- let me build cfg80211 alone, build it, install it and now have their 802.11 driver also selected (if they didn't).

That ... no I don't think you can do that. Nor do I think you even want to? You can't build cfg80211 from one compat and the driver from another, they won't be compatible anyway. So building cfg80211 first and then installing it would be useless unless they build their driver from the same tree later, but then it doesn't really matter that they still have to select cfg80211? Or maybe I'm misunderstanding?

Quoted 4 lines

I actually like the simplicity of it in Python and prefer it if we can live with it, but given that we follow the kernel build system I'd rather see the kernel first take the bullet on swallowing Python... not sure, anyway doesn't matter much if its simple.

It won't be more difficult even in bash, I'm just faster at writing it in python :-)

Quoted 8 lines
Quoted 2 lines

This is about as far as I got with the implementation. There are a few issues left.

Like Kconfig renames that went upstream and mapping them to their older name on the users's older kernels? This likely could be addressed by having these maps laid out for each kernel, but would require manual maintenance. It shouldn't be so hard if we do it *moving forward*.

Hmm, I never thought about that. Does that happen frequently? Do you have any examples? Presumably, an example of this would be if you have e.g. CONFIG_USB_DEVICES in an old kernel, but it's CONFIG_USB in the new one, and all drivers that exist now have "depends on USB" while on an old kernel USB won't exist, so you wouldn't be able to select such a driver?

Anyway it seems like we must already maintain such a mapping in a way, maybe a bit more implicitly? The way here you'd do it is add something like this:

config USB

	bool
	default y
	depends on USB_DEVICES && COMPAT_KERNEL_2_3_4_5

or something like that?

Quoted 2 lines

The other bigger thing I think is the dependency map output not existing so it'd let users break a build / existing dep.

I think it wouldn't break, worst case you wouldn't be able to select a driver that you should be able to select?

Quoted 5 lines
Quoted 2 lines

2) I have no idea how to go from a .config file to a working Makefile/autoconf.h system.

Hm, we'd already have the Makefile for each target no? Are you trying to avoid having to ship each Makefile for each pulled driver?

Yes, we do have the makefiles of course, as copied from the kernel sources. We want to keep them, obviously. I'm not trying to avoid that. I'm saying I have no idea how to get the CONFIG_* or rather CPTCFG_* symbols into make when it reads those makefiles.

Quoted 1 line

For autoconf we have our own thingy so far.

Yeah, but how do we get the symbols into make? That part isn't clear to me at all, even for the regular kernel.

Quoted 8 lines
Quoted 4 lines

4) This assumes KLIB_BUILD is configured externally, it's not possible to configure that through menuconfig since the Kconfig tree needs to be built first. I don't think this is a problem, but it looks like Luis wanted to put that into Kconfig so I'm saying it won't really work.

I didn't want to put it into Kconfig actually, I was just throwing the variables on the menuconfig display to ensure the user building is aware of the sources of everything.

Ok :-)

Here's another issue I haven't really solved yet: All of compat must be modules, so selecting Y for any symbol, e.g. CFG80211, must be disallowed. This seems a little difficult, maybe the best choice would be to simply hack menuconfig/... tool to disallow setting tristate options to Y.

johannes

#12 lore raw
Quoted 10 lines · Johannes Berg
Quoted 8 lines · Luis R. Rodriguez

This requires a study of the different types of collateral evolutions we have. compat already addresses the ones we can implement without requiring changing the upstream drivers. I'm now starting to address -- what is upstream willing to do to help with backports. An example is the static inlines.

https://lkml.org/lkml/2013/3/28/187

Quoted 7 lines
Quoted 3 lines

If the above patch is accepted I'd then try to send upstream the netdev_attach_ops() one, and that would nuke that entire patch from compat-drivers.

Right, this is a goal in itself, but I don't think we'll get rid of all patches. Consider things like the completely different API for multicast lists, for example.

I am willing to accept perhaps some changes may not be possible and we may need to live with patches but.. if we do at the very least we can use SmPL grammar to express it. Whether or not we can express this in SmPL that's a separate question but to help with that question we need to start categorizing these types of collateral evolutions.

Julia in case you are interested in exploring the one Johannes mentioned you can see commit 6dc94f7b9 on compat-drivers or just look at the file:

patches/collateral-evolutions/network/25-multicast-list_head.patch

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Since the filename doesn't matter all that much, an automated conversion could just split the patch per file and store it in a filename obtained as follows:
drivers/net/usb/rndis_host.c
-> drivers_net_usb_rndis_host.c

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Easy :-)

True. I welcome this change if it helps you.

It would help me quite a bit. I suppose I could actually send you patches for this soon. OTOH, without addressing my wishlist item 1) I probably won't update to a current compat-drivers base ... I heavily patch admin-update.sh now.

OK.

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

path/to/syms-program Kconfig > local-symbol-list
sed -i 's/$/=/' local-symbol-list
grep -v -f local-symbol-list $KLIB_BUILD/.config | dotconf-to-kconf.py > Kconfig.kernel

Neat trick! This doesn't handle the mapping of depdendencies but certainly already makes good assumptions given that you *have* these things built.

Quoted 6 lines

Are you referring to the "select" issue Hauke pointed out? Or what other handling of dependencies are you thinking of? This entire trick exists to allow compat's Kconfig to know about the base kernel's Kconfig so the user can select the drivers that are actually possible given the base kernel. For example, if building for kernel without PCI, you wouldn't be able to select PCI drivers, etc.

No I'm concerned about the dependency within compat-drivers itself on allowing users to build an updated driver that breaks existing stock drivers that they didn't enable on compat-drivers. One example: say a user decides to build ath9k, but also has a USB dongle and ath9k_htc but does not build the ath9k_htc module, only ath9k. Loading ath9k_htc will fail after he installs that build.

Quoted 7 lines
Quoted 3 lines

It does not however take into consideration for example that a user may say -- let me build cfg80211 alone, build it, install it and now have their 802.11 driver also selected (if they didn't).

That ... no I don't think you can do that. Nor do I think you even want to? You can't build cfg80211 from one compat and the driver from another, they won't be compatible anyway.

Well of course, I'm just saying -- it'd be possible to do this. But I guess this is possible to day anyway :)

Quoted 4 lines

So building cfg80211 first and
then installing it would be useless unless they build their driver from the same tree later, but then it doesn't really matter that they still have to select cfg80211? Or maybe I'm misunderstanding?

No its fine, its a general build dependency issue but that issue exists today, we just don't allow for huge different set of configurations. We allow:

* build everything
* driver-select

And these are tested (or at least the ones I maintain are).

Kconfig opens the door for users doing complex configurations and these could end up being reported as "issues", flooding the lists, and time on people.

Quoted 7 lines
Quoted 4 lines

I actually like the simplicity of it in Python and prefer it if we can live with it, but given that we follow the kernel build system I'd rather see the kernel first take the bullet on swallowing Python... not sure, anyway doesn't matter much if its simple.

It won't be more difficult even in bash, I'm just faster at writing it in python :-)

OK ;)

Quoted 10 lines
Quoted 8 lines
Quoted 2 lines

This is about as far as I got with the implementation. There are a few issues left.

Like Kconfig renames that went upstream and mapping them to their older name on the users's older kernels? This likely could be addressed by having these maps laid out for each kernel, but would require manual maintenance. It shouldn't be so hard if we do it *moving forward*.

Hmm, I never thought about that. Does that happen frequently?

Not *that* frequently but handling that is a fucking bitch. At times I wish the maintainer would send me a patch but so far only the contributors to compat have addressed that.

This is not a 'collateral evolution' per se, but certainly something to consider if we can extract logic from git to address it for us.

Quoted 6 lines

Do you
have any examples? Presumably, an example of this would be if you have e.g. CONFIG_USB_DEVICES in an old kernel, but it's CONFIG_USB in the new one, and all drivers that exist now have "depends on USB" while on an old kernel USB won't exist, so you wouldn't be able to select such a driver?

Typically its not core stuff. Its all driver poo. At times its simply renames from upstream to downstream mapping issues. compat-drivers examples:

c5cfdf2d138ba20a648990d9615c44942eef5fd3
8c8113109f6916c4926413aa3416cefffaa5d9b8

Quoted 10 lines

Anyway it seems like we must already maintain such a mapping in a way, maybe a bit more implicitly? The way here you'd do it is add something like this:

config USB

        bool
        default y
        depends on USB_DEVICES && COMPAT_KERNEL_2_3_4_5

or something like that?

Sure. I think you're trying to automate this though?

BTW I'm trying to move things to BACKPORT_ prefix for poo given that the kernel does use compat_ already for some poo. Not a big issue, whatever.

Quoted 5 lines
Quoted 2 lines

The other bigger thing I think is the dependency map output not existing so it'd let users break a build / existing dep.

I think it wouldn't break, worst case you wouldn't be able to select a driver that you should be able to select?

Yes perhaps, its wroth for us to try it, likely better than the current unscalable situation.

Quoted 10 lines
Quoted 5 lines
Quoted 2 lines

2) I have no idea how to go from a .config file to a working Makefile/autoconf.h system.

Hm, we'd already have the Makefile for each target no? Are you trying to avoid having to ship each Makefile for each pulled driver?

Yes, we do have the makefiles of course, as copied from the kernel sources. We want to keep them, obviously. I'm not trying to avoid that. I'm saying I have no idea how to get the CONFIG_* or rather CPTCFG_* symbols into make when it reads those makefiles.

Ah yes. Can't we assume a direct mapping CONFIG_FOO --> CPTCFG_FOO ?

Quoted 4 lines
Quoted 1 line

For autoconf we have our own thingy so far.

Yeah, but how do we get the symbols into make? That part isn't clear to me at all, even for the regular kernel.

Yeah, never figured that poo out myself... The Makefile somehow has to end up parsing the .config as a Makefile for variables, how that's done no sure. For compat-drivers we just used assignments on config.mk.

Quoted 9 lines
Quoted 8 lines
Quoted 4 lines

4) This assumes KLIB_BUILD is configured externally, it's not possible to configure that through menuconfig since the Kconfig tree needs to be built first. I don't think this is a problem, but it looks like Luis wanted to put that into Kconfig so I'm saying it won't really work.

I didn't want to put it into Kconfig actually, I was just throwing the variables on the menuconfig display to ensure the user building is aware of the sources of everything.

Quoted 7 lines

Ok :-)

Here's another issue I haven't really solved yet: All of compat must be modules, so selecting Y for any symbol, e.g. CFG80211, must be disallowed. This seems a little difficult, maybe the best choice would be to simply hack menuconfig/... tool to disallow setting tristate options to Y.

If that's 100% true, sure! But say you want to disable a feature, or enable a new one you didn't have ?

Luis

#13 lore raw
Quoted 7 lines · Luis R. Rodriguez
Quoted 6 lines

Are you referring to the "select" issue Hauke pointed out? Or what other handling of dependencies are you thinking of? This entire trick exists to allow compat's Kconfig to know about the base kernel's Kconfig so the user can select the drivers that are actually possible given the base kernel. For example, if building for kernel without PCI, you wouldn't be able to select PCI drivers, etc.

Quoted 6 lines

No I'm concerned about the dependency within compat-drivers itself on allowing users to build an updated driver that breaks existing stock drivers that they didn't enable on compat-drivers. One example: say a user decides to build ath9k, but also has a USB dongle and ath9k_htc but does not build the ath9k_htc module, only ath9k. Loading ath9k_htc will fail after he installs that build.

Yes. That's unfortunate, but I don't really think we should _force_ users to enable ath9k_htc? And we can't know what kind of USB devices they're going to plug in, so even "make localmodconfig" won't help all that much. We could default to all the drivers that the user has _installed_, but that'd mean maintaining a Kconfig <-> module mapping (though maybe the localmodconfig script has that?) and it wouldn't be what many users want? Why build USB drivers just because your distro did if you don't own the device, after all...

IOW, I'm not sure what we can do about this.

Quoted 4 lines

No its fine, its a general build dependency issue but that issue exists today, we just don't allow for huge different set of configurations. We allow:

Quoted 8 lines

* build everything
* driver-select

And these are tested (or at least the ones I maintain are).

Kconfig opens the door for users doing complex configurations and these could end up being reported as "issues", flooding the lists, and time on people.

I'm not all that worried about that, to be honest. We should probably default to running something like "allmodconfig", just like we do now. Also, the kernel's configuration is much more complex and did you see people complain there? :-)
In addition, we get the advantage that then people are unable to make things that don't build. I for one have in fact gotten plenty of bug reports because of wrong or missing ifdefs in config.mk (granted, through my own fault) -- e.g. config.mk enabling debugfs when the base kernel didn't allow for it, etc. So many issues are also solved by this.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

Like Kconfig renames that went upstream and mapping them to their older name on the users's older kernels? This likely could be addressed by having these maps laid out for each kernel, but would require manual maintenance. It shouldn't be so hard if we do it *moving forward*.

Quoted 8 lines
Quoted 1 line

Hmm, I never thought about that. Does that happen frequently?

Not *that* frequently but handling that is a fucking bitch. At times I wish the maintainer would send me a patch but so far only the contributors to compat have addressed that.

This is not a 'collateral evolution' per se, but certainly something to consider if we can extract logic from git to address it for us.

I don't really see what we could extract from git. The way I see it, the only way to really address it is to add this to compat/Kconfig:

config NEW_SYMBOL_NAME
tristate
default OLD_SYMBOL_NAME

(or bool, and maybe with "depends on BACKPORT_KERNEL_X_Y_Z" added in)

Quoted 7 lines
Quoted 6 lines

Do you
have any examples? Presumably, an example of this would be if you have e.g. CONFIG_USB_DEVICES in an old kernel, but it's CONFIG_USB in the new one, and all drivers that exist now have "depends on USB" while on an old kernel USB won't exist, so you wouldn't be able to select such a driver?

Quoted 5 lines

Typically its not core stuff. Its all driver poo. At times its simply renames from upstream to downstream mapping issues. compat-drivers examples:

c5cfdf2d138ba20a648990d9615c44942eef5fd3

Heh, beating me with my own driver, always a good plan :-)

However this is entirely within the backports package so it's not an issue at all?

Quoted 1 line

8c8113109f6916c4926413aa3416cefffaa5d9b8

Ditto here. In fact both of these patches just go away with my Kconfig scheme since I prefix CPTCFG_ instead of CONFIG_ and thus kernel symbols aren't used any more by the Makefile etc. stuff in backports.

Quoted 6 lines
Quoted 6 lines

config USB

        bool
        default y
        depends on USB_DEVICES && COMPAT_KERNEL_2_3_4_5

or something like that?

Heh, almost the same I wrote above...

Quoted 1 line

Sure. I think you're trying to automate this though?

But no, I don't think I can automate this part.

What I can automate is for drivers that are part of the backports package. For *dependencies* those drivers have, like my hypothetical USB example, we'll have to maintain it manually. It shouldn't be too difficult though.

Quoted 3 lines

BTW I'm trying to move things to BACKPORT_ prefix for poo given that the kernel does use compat_ already for some poo. Not a big issue, whatever.

Good plan. I'll make the COMPAT_KERNEL_X_Y_Z symbols BACKPORT_KERNEL_X_Y_Z instead, and also the other symbols with COMPAT in the name. For the automatic ones I picked CPTCFG_ which is hopefully unique enough and has the advantage of having the same length as CONFIG_ (which is selfish of me, but it helps me maintain the code since we work inside a backport package much of the time)

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

2) I have no idea how to go from a .config file to a working Makefile/autoconf.h system.

Hm, we'd already have the Makefile for each target no? Are you trying to avoid having to ship each Makefile for each pulled driver?

Quoted 6 lines
Quoted 4 lines

Yes, we do have the makefiles of course, as copied from the kernel sources. We want to keep them, obviously. I'm not trying to avoid that. I'm saying I have no idea how to get the CONFIG_* or rather CPTCFG_* symbols into make when it reads those makefiles.

Ah yes. Can't we assume a direct mapping CONFIG_FOO --> CPTCFG_FOO ?

Yes, we can mostly do that, and get rid of all the patches that modify symbols.

Quoted 9 lines
Quoted 4 lines
Quoted 1 line

For autoconf we have our own thingy so far.

Yeah, but how do we get the symbols into make? That part isn't clear to me at all, even for the regular kernel.

Yeah, never figured that poo out myself... The Makefile somehow has to end up parsing the .config as a Makefile for variables, how that's done no sure. For compat-drivers we just used assignments on config.mk.

Yeah ... I think the makefile just has to include the .config, I'll have to check it out. Shouldn't be too difficult. I think the difficulty with the backport system is that it has to call the kernel's Makefile, and has to export everything for that ... anyway I'll figure out something, can't be too hard.

Quoted 8 lines
Quoted 5 lines

Here's another issue I haven't really solved yet: All of compat must be modules, so selecting Y for any symbol, e.g. CFG80211, must be disallowed. This seems a little difficult, maybe the best choice would be to simply hack menuconfig/... tool to disallow setting tristate options to Y.

If that's 100% true, sure! But say you want to disable a feature, or enable a new one you didn't have ?

Well everything inside backports that's allowed to be tristate in the kernel, must be N or M in backports, selecting Y for a tristate (module) doesn't make sense. I'm already doing this now.

johannes

#14 lore raw
Quoted 25 lines · Johannes Berg
Quoted 7 lines · Luis R. Rodriguez
Quoted 6 lines

Are you referring to the "select" issue Hauke pointed out? Or what other handling of dependencies are you thinking of? This entire trick exists to allow compat's Kconfig to know about the base kernel's Kconfig so the user can select the drivers that are actually possible given the base kernel. For example, if building for kernel without PCI, you wouldn't be able to select PCI drivers, etc.

Quoted 6 lines

No I'm concerned about the dependency within compat-drivers itself on allowing users to build an updated driver that breaks existing stock drivers that they didn't enable on compat-drivers. One example: say a user decides to build ath9k, but also has a USB dongle and ath9k_htc but does not build the ath9k_htc module, only ath9k. Loading ath9k_htc will fail after he installs that build.

Yes. That's unfortunate, but I don't really think we should _force_ users to enable ath9k_htc? And we can't know what kind of USB devices they're going to plug in, so even "make localmodconfig" won't help all that much. We could default to all the drivers that the user has _installed_, but that'd mean maintaining a Kconfig <-> module mapping (though maybe the localmodconfig script has that?) and it wouldn't be what many users want? Why build USB drivers just because your distro did if you don't own the device, after all...

Quoted 1 line

IOW, I'm not sure what we can do about this.

See commit 8ae9673098 on compat, specifically the reference to Andi Kleen's module symbol namespace. This would have solved that issue but also allow, for example, iwlwifi to use say your stock mac80211 while perhaps ath9k does need the backported wireless. Some distributions have expressed interest in this as a feature (not specifically this example, just saying any driver). The usage of the updates/ directory would still enable us to let the backport driver supersede the older stock kernel one.

Right now we don't make cfg80211, mac80211 use a prefix for their own symbols but perhaps we should, that would address this. It would not incur cost on RAM given that you'd only load the old module if a device required it, and there is no extra storage cost given that you'd still install the backport drivers as we do today.

Thoughts?

Quoted 9 lines
Quoted 9 lines

No its fine, its a general build dependency issue but that issue exists today, we just don't allow for huge different set of configurations. We allow:

* build everything
* driver-select

And these are tested (or at least the ones I maintain are).

Quoted 8 lines
Quoted 3 lines

Kconfig opens the door for users doing complex configurations and these could end up being reported as "issues", flooding the lists, and time on people.

I'm not all that worried about that, to be honest. We should probably default to running something like "allmodconfig", just like we do now. Also, the kernel's configuration is much more complex and did you see people complain there? :-)

Actually I do, I believe randconfig is used to test builds by some folks, is how some obscure Kconfig bugs have been found, that is, a driver developer not testing building under certain Kconfig settings.

Asking users to build everything resolves this in part, and having users use menuconfig should then be something we have maintainers for those options. This lets users peg those maintainers on the bug reports found on those kconfig options. Thoughts?

Quoted 5 lines

In addition, we get the advantage that then people are unable to make things that don't build. I for one have in fact gotten plenty of bug reports because of wrong or missing ifdefs in config.mk (granted, through my own fault) -- e.g. config.mk enabling debugfs when the base kernel didn't allow for it, etc. So many issues are also solved by this.

That's indeed awesome, I can't wait!

Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

Like Kconfig renames that went upstream and mapping them to their older name on the users's older kernels? This likely could be addressed by having these maps laid out for each kernel, but would require manual maintenance. It shouldn't be so hard if we do it *moving forward*.

Quoted 8 lines
Quoted 1 line

Hmm, I never thought about that. Does that happen frequently?

Not *that* frequently but handling that is a fucking bitch. At times I wish the maintainer would send me a patch but so far only the contributors to compat have addressed that.

This is not a 'collateral evolution' per se, but certainly something to consider if we can extract logic from git to address it for us.

Quoted 8 lines

I don't really see what we could extract from git. The way I see it, the only way to really address it is to add this to compat/Kconfig:

config NEW_SYMBOL_NAME
tristate
default OLD_SYMBOL_NAME

(or bool, and maybe with "depends on BACKPORT_KERNEL_X_Y_Z" added in)

Well I like the idea :D

It does mean manual work though and upon integration we'd discover this as a build failure, then we'd fix it.

Quoted 11 lines
Quoted 11 lines
Quoted 6 lines

Do you
have any examples? Presumably, an example of this would be if you have e.g. CONFIG_USB_DEVICES in an old kernel, but it's CONFIG_USB in the new one, and all drivers that exist now have "depends on USB" while on an old kernel USB won't exist, so you wouldn't be able to select such a driver?

Typically its not core stuff. Its all driver poo. At times its simply renames from upstream to downstream mapping issues. compat-drivers examples:

Quoted 6 lines
Quoted 1 line

c5cfdf2d138ba20a648990d9615c44942eef5fd3

Heh, beating me with my own driver, always a good plan :-)

However this is entirely within the backports package so it's not an issue at all?

Sure, in this case, yes.

Quoted 5 lines
Quoted 1 line

8c8113109f6916c4926413aa3416cefffaa5d9b8

Ditto here. In fact both of these patches just go away with my Kconfig scheme since I prefix CPTCFG_ instead of CONFIG_ and thus kernel symbols aren't used any more by the Makefile etc. stuff in backports.

OK neat.

Quoted 9 lines
Quoted 6 lines
Quoted 6 lines

config USB

        bool
        default y
        depends on USB_DEVICES && COMPAT_KERNEL_2_3_4_5

or something like that?

Heh, almost the same I wrote above...

Quoted 8 lines
Quoted 1 line

Sure. I think you're trying to automate this though?

But no, I don't think I can automate this part.

What I can automate is for drivers that are part of the backports package. For *dependencies* those drivers have, like my hypothetical USB example, we'll have to maintain it manually. It shouldn't be too difficult though.

OK. I think this could be automated if we could script the question:

What dependencies did this module have in the original kernel? depmod to the rescue.

/lib/modules/$(uname -r)/modules.dep

But that does the inverse, module to symbol lookup (BTW lets be clear on language, symbol: export symbol, config: CONFIG_ poo), we need something to query the dependencies based on a given CONFIG_ and I believe we can get this, just gotta write it up. This can be done later though for sure :)

Quoted 10 lines
Quoted 3 lines

BTW I'm trying to move things to BACKPORT_ prefix for poo given that the kernel does use compat_ already for some poo. Not a big issue, whatever.

Good plan. I'll make the COMPAT_KERNEL_X_Y_Z symbols BACKPORT_KERNEL_X_Y_Z instead, and also the other symbols with COMPAT in the name. For the automatic ones I picked CPTCFG_ which is hopefully unique enough and has the advantage of having the same length as CONFIG_ (which is selfish of me, but it helps me maintain the code since we work inside a backport package much of the time)

Sure, thanks!

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 5 lines
Quoted 2 lines

2) I have no idea how to go from a .config file to a working Makefile/autoconf.h system.

Hm, we'd already have the Makefile for each target no? Are you trying to avoid having to ship each Makefile for each pulled driver?

Yes, we do have the makefiles of course, as copied from the kernel sources. We want to keep them, obviously. I'm not trying to avoid that. I'm saying I have no idea how to get the CONFIG_* or rather CPTCFG_* symbols into make when it reads those makefiles.

Quoted 4 lines
Quoted 1 line

Ah yes. Can't we assume a direct mapping CONFIG_FOO --> CPTCFG_FOO ?

Yes, we can mostly do that, and get rid of all the patches that modify symbols.

:D

perl -pi -e 's|CONFIG_|CONFIG_BACKPORT_|g' some/Makefile

But this would also end up renaming dependencies the Makefile might have on stuff not provided by compat, an example:

ifeq ($(CONFIG_COMPAT),y)
videodev-objs += v4l2-compat-ioctl32.o
endif

Would end up:

ifeq ($(CONFIG_BACKPORT_COMPAT),y)
videodev-objs += v4l2-compat-ioctl32.o
endif

Maybe we need to script this to not do the mangle for config options that are not present and assigned on the existing config.mk or .config (not sure what these map to exactly on your tree yet).

Quoted 11 lines
Quoted 9 lines
Quoted 4 lines
Quoted 1 line

For autoconf we have our own thingy so far.

Yeah, but how do we get the symbols into make? That part isn't clear to me at all, even for the regular kernel.

Yeah, never figured that poo out myself... The Makefile somehow has to end up parsing the .config as a Makefile for variables, how that's done no sure. For compat-drivers we just used assignments on config.mk.

Yeah ... I think the makefile just has to include the .config,

I reached the same conclusion I just never saw it being done, but I also noticed I had to actually call export on these too.

Quoted 5 lines

I'll have
to check it out. Shouldn't be too difficult. I think the difficulty with the backport system is that it has to call the kernel's Makefile, and has to export everything for that ... anyway I'll figure out something, can't be too hard.

OK cool.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

Here's another issue I haven't really solved yet: All of compat must be modules, so selecting Y for any symbol, e.g. CFG80211, must be disallowed. This seems a little difficult, maybe the best choice would be to simply hack menuconfig/... tool to disallow setting tristate options to Y.

Quoted 6 lines
Quoted 2 lines

If that's 100% true, sure! But say you want to disable a feature, or enable a new one you didn't have ?

Well everything inside backports that's allowed to be tristate in the kernel, must be N or M in backports, selecting Y for a tristate (module) doesn't make sense. I'm already doing this now.

Ah tristate, yeah that make sense and would be a good idea to help avoid issues.

Luis

#15 lore raw
Quoted 9 lines · Luis R. Rodriguez

See commit 8ae9673098 on compat, specifically the reference to Andi Kleen's module symbol namespace. This would have solved that issue but also allow, for example, iwlwifi to use say your stock mac80211 while perhaps ath9k does need the backported wireless. Some distributions have expressed interest in this as a feature (not specifically this example, just saying any driver). The usage of the updates/ directory would still enable us to let the backport driver supersede the older stock kernel one.

Quoted 7 lines

Right now we don't make cfg80211, mac80211 use a prefix for their own symbols but perhaps we should, that would address this. It would not incur cost on RAM given that you'd only load the old module if a device required it, and there is no extra storage cost given that you'd still install the backport drivers as we do today.

Thoughts?

Ok yes, this is still an interesting thing that we might consider doing. I consider it orthogonal to my current work though.

Tackling this would be tricky though, you don't want to patch all the different uses of a symbol, so you'd have to do something like module symbol namespaces, and teach the build environment about it, or something like that. Maybe it could be done with the linker's symbol versioning.

Anyway -- this is interesting but not my (primary) goal right now.

Quoted 3 lines

Actually I do, I believe randconfig is used to test builds by some folks, is how some obscure Kconfig bugs have been found, that is, a driver developer not testing building under certain Kconfig settings.

Yes, of course, this happens all the time, but it's just a bug.

Quoted 4 lines

Asking users to build everything resolves this in part, and having users use menuconfig should then be something we have maintainers for those options. This lets users peg those maintainers on the bug reports found on those kconfig options. Thoughts?

Users can attach their backport .config file to the bug report, so this should be fairly easy to identify in the bug. Also, this is really not much different from getting a "build failed with these options" bug on the kernel. It's different, of course, but many such issues would probably replicate in the kernel as well.

Quoted 10 lines
Quoted 5 lines

config NEW_SYMBOL_NAME
tristate
default OLD_SYMBOL_NAME

(or bool, and maybe with "depends on BACKPORT_KERNEL_X_Y_Z" added in)

Well I like the idea :D

It does mean manual work though and upon integration we'd discover this as a build failure, then we'd fix it.

Yes. We don't get to magically avoid porting work :-)

Quoted 10 lines
Quoted 4 lines

What I can automate is for drivers that are part of the backports package. For *dependencies* those drivers have, like my hypothetical USB example, we'll have to maintain it manually. It shouldn't be too difficult though.

OK. I think this could be automated if we could script the question:

What dependencies did this module have in the original kernel? depmod to the rescue.

Quoted 7 lines

/lib/modules/$(uname -r)/modules.dep

But that does the inverse, module to symbol lookup (BTW lets be clear on language, symbol: export symbol, config: CONFIG_ poo), we need something to query the dependencies based on a given CONFIG_ and I believe we can get this, just gotta write it up. This can be done later though for sure :)

Yeah, agree, later.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Ah yes. Can't we assume a direct mapping CONFIG_FOO --> CPTCFG_FOO ?

Yes, we can mostly do that, and get rid of all the patches that modify symbols.

:D

perl -pi -e 's|CONFIG_|CONFIG_BACKPORT_|g' some/Makefile

[...]

Quoted 3 lines

Maybe we need to script this to not do the mangle for config options that are not present and assigned on the existing config.mk or .config (not sure what these map to exactly on your tree yet).

Yes. I did pretty much exactly that. There's no config.mk, I only have a .config file and that's generated at build time via the kconfig mechanism.

Quoted 4 lines
Quoted 1 line

Yeah ... I think the makefile just has to include the .config,

I reached the same conclusion I just never saw it being done, but I also noticed I had to actually call export on these too.

include .config
export

works for me. Just "export" means export all.

johannes

#17 lore raw
In reply to Johannes Berg
Quoted 3 lines · Johannes Berg
Quoted 3 lines

1) Ability to copy only some drivers.
2) To make that easier: split all patches per *file* in addition to per change.
3) Separate the output from the compat-drivers tree.

done.

Quoted 1 line
Quoted 1 line

4) Kconfig.

pretty much done -- right now users can get into unbuildable situations, I obviously want to avoid.

I also decided to merge compat and compat-drivers -- right now I see no value in them being combined. I therefore created a new backports.git tree.

Note that this is very very raw. It doesn't even have the ability to *build* the drivers yet, which is kinda the whole point ... But I had to start somewhere, and obviously that was at step 1: building the sources :-)

It's all here, if anyone wants to risk a look.
http://git.sipsolutions.net/?p=backports.git;a=summary

I'll continue next week, most likely.

johannes