Skip to contentSkip to conversation

new backports project

ANN Johannes Berg 5 participants 12 messages Apr 2, 2013 – Apr 5, 2013

lore
Share & tools
lore
#0 lore raw

Hey,

I guess just a few days later it's time to announce it ... I've been working on a merge of compat/compat-drivers into a new backports tree. That by itself would be boring, so here's what I also did:

* made driver copying variable
As discussed a few days ago, I wanted this to be able to port only
some
drivers. To really achieve this without breaking all patches, I also
had to
- split patches and make applying them smarter
- remove Makefile patches and mangle the Makefiles in a scripted way
* added Kconfig support
This allows users to better select which drivers they want, with
their
dependencies etc. To achieve this, it was necessary to also mangle
the Kconfig
files in some ways, but that's all scripted as well
* separate output directory
My new script writes in a new output directory, which helps separate
the
compat code creation step from the build step, and end users don't
get
burdened with the scripting needed for the creation.

To be fair, I should say what I broke. I think the system should be generic enough to allow adding it back easily though.
* graphics drivers -- I couldn't even get the patches to apply
* ethernet drivers -- I just wasn't sure which ones were needed etc.
* "crap"/staging/... patches
* plain 'make' -- you now need to do 'make allyesconfig' or so
first :-)
* Redhat kernel backport support -- this should be fairly easy

I'm hoping that the improvements I have are sufficient to get everyone to help out, and have the old compat/compat-drivers die ... :-)

The code is available here (for now):
http://git.sipsolutions.net/?p=backports.git;a=summary

And in case you're wondering ... I got ckmake to build an "allyesconfig" (except for CFG80211_INTERNAL_REGDB which I forced off) against all kernels on Luis's build machine. This took forever, but I got this:

1   2.6.24              [  OK  ]
2   2.6.25              [  OK  ]
3   2.6.26              [  OK  ]
4   2.6.27              [  OK  ]
5   2.6.28              [  OK  ]
6   2.6.29              [  OK  ]
7   2.6.30              [  OK  ]
8   2.6.31              [  OK  ]
9   2.6.32              [  OK  ]
10  2.6.33              [  OK  ]
11  2.6.34              [  OK  ]
12  2.6.35              [  OK  ]
13  2.6.36              [  OK  ]
14  2.6.37              [  OK  ]
15  2.6.38              [  OK  ]
16  2.6.39              [  OK  ]
17  3.0.65              [  OK  ]
18  3.1.10              [  OK  ]
19  3.2.38              [  OK  ]
20  3.3.8               [  OK  ]
21  3.4.32              [  OK  ]
22  3.5.7               [  OK  ]
23  3.6.11              [  OK  ]
24  3.7.9               [  OK  ]
25  3.8.0               [  OK  ]
26  3.9-rc1             [  OK  ]

johannes

#1 lore raw
Quoted 4 lines · Johannes Berg

To be fair, I should say what I broke. I think the system should be generic enough to allow adding it back easily though.
* graphics drivers -- I couldn't even get the patches to apply
* ethernet drivers -- I just wasn't sure which ones were needed etc.

alx please.

Norm

--
Norman Shulman
Sr. Developer/Architect
N-Dimension Solutions Inc.
9030 Leslie St, Unit 300
Richmond Hill, ON L4B 1G2
Canada

Tel: 905 707-8884 x 226
Fax: 905 707-0886

This email and any files transmitted with it are solely intended for the use of the named recipient(s) and may contain information that is privileged and confidential. If you receive this email in error, please immediately notify the sender and delete this message in all its forms.

#2 lore raw
In reply to Johannes Berg
Quoted 2 lines

* Redhat kernel backport support -- this should be fairly easy

I addressed this, but I'm unsure of it and don't have a way to test it, so any testing would be appreciated (as it is in general, of course)

johannes

#3 lore raw
Quoted 5 lines · Johannes Berg
Quoted 2 lines

* Redhat kernel backport support -- this should be fairly easy

I addressed this, but I'm unsure of it and don't have a way to test it, so any testing would be appreciated (as it is in general, of course)

FWIW the LINUX_BACKPORT() stuff replaced RHEL specific changes, that is, LINUX_BACKPORT() stuff extended the work that folks used to add RHEL support making it more generic. This helps try to avoid clashes with *any* Linux distribution trying to cherry pick features from the kernel, so long as they don't use the same prefix for exported symbols. I wonder if there's any more need for RHEL / SUSE specific kernel checks now after LINUX_BACKPORT() stuff.

Luis

#4 lore raw
In reply to Johannes Berg
Quoted 3 lines · Johannes Berg
Quoted 2 lines

* Redhat kernel backport support -- this should be fairly easy

I too am very interested in support for RHEL.

We had support up to date for compat-wireless-3.5 under RHEL-6.3.

Unsurprisingly the release of RHEL-6.4 broke a whole load of stuff given the amount of backporting RH did into their kernel (I believe they backported the wireless stack from ~ kernel-3.5 into their kernel).

I (and a colleague) played with compat-drivers-2013-03-07-u.tar.bz2 to get alx to build under RHEL-6.4 but I never got as far as looking at building everything. The seemingly endless cycle of breakage on every RHEL release and having to fix it kind of left me losing the will to keep fixing it, especially as for the moment the RHEL kernel has a relatively recent wireless driver stack. Any solution that might ease this break and fix cycle would be very welcome here.

Quoted 3 lines

I addressed this, but I'm unsure of it and don't have a way to test it, so any testing would be appreciated (as it is in general, of course)

I've not looked closely at what you've done, but I have an interest in building/packaging compat-drivers for RHEL and can possibly help test. If you have a snapshot that you think should build on RHEL or that I could use as a starting point?

Phil

#5 lore raw

Hi,

Quoted 1 line

I too am very interested in support for RHEL.

:-)
I should say that I have no idea what RHEL does etc.

Quoted 5 lines

We had support up to date for compat-wireless-3.5 under RHEL-6.3.

Unsurprisingly the release of RHEL-6.4 broke a whole load of stuff given the amount of backporting RH did into their kernel (I believe they backported the wireless stack from ~ kernel-3.5 into their kernel).

Makes sense.

Quoted 8 lines

I (and a colleague) played with compat-drivers-2013-03-07-u.tar.bz2 to get alx to build under RHEL-6.4 but I never got as far as looking at building everything. The seemingly endless cycle of breakage on every RHEL release and having to fix it kind of left me losing the will to keep fixing it, especially as for the moment the RHEL kernel has a relatively recent wireless driver stack. Any solution that might ease this break and fix cycle would be very welcome here.

Quoted 8 lines
Quoted 3 lines

I addressed this, but I'm unsure of it and don't have a way to test it, so any testing would be appreciated (as it is in general, of course)

I've not looked closely at what you've done, but I have an interest in building/packaging compat-drivers for RHEL and can possibly help test. If you have a snapshot that you think should build on RHEL or that I could use as a starting point?

I only addressed what compat-drivers-... had a few days ago.

However, I simplified the system a bit. I don't have any snapshot, but I did announce the git tree. Grab that, run ./gentree.py against linux-next and you should get something. Then you will have to hack up the Kconfig (backport/compat/Kconfig) and include files (backport/include/*), presumably.

If you're interested in alx, you'd have to add that -- look at the copy-list.alx file for that.

I hope that soon enough Luis might cut some testing releases from this though.

johannes

#6 lore raw
Quoted 7 lines · Johannes Berg

I only addressed what compat-drivers-... had a few days ago.

However, I simplified the system a bit. I don't have any snapshot, but I did announce the git tree. Grab that, run ./gentree.py against linux-next and you should get something. Then you will have to hack up the Kconfig (backport/compat/Kconfig) and include files (backport/include/*), presumably.

Or you can wait until we merge. At this point I'm freezing development on compat and compat-drivers to merge all this work onto Johannes' tree.

Quoted 2 lines

If you're interested in alx, you'd have to add that -- look at the copy-list.alx file for that.

Johannes already committed a few things here, I'll be sure to test it as well.

Quoted 2 lines

I hope that soon enough Luis might cut some testing releases from this though.

Absolutely, full steam ahead, just give us a few days.

I should note that RHEL / SUSE / Debian all had the same issues -- running into symbols we are exporting with the same upstream name. The move to use LINUX_BACKPORT() should help cure these issues in general moving forward. Additionally if you really want to help in this area please review commit 8ae96730 on compat.git specifically Andi Kleen's work on module namespace. If we can figure out to support module namespace *ourselves*, ie, without Linus having to carry this crap around, then we'd solve all these issues easily, I believe we wouldn't have to rely on LINUX_BACKPORT() and in fact that would also mean renaming our *driver* symbols. This means for instance you can run one drivers with your stock mac80211, another with the backported version, should you ever have to support such crazy environement.

Luis

#7 lore raw
Quoted 23 lines · Luis R. Rodriguez
Quoted 7 lines · Johannes Berg

I only addressed what compat-drivers-... had a few days ago.

However, I simplified the system a bit. I don't have any snapshot, but I did announce the git tree. Grab that, run ./gentree.py against linux-next and you should get something. Then you will have to hack up the Kconfig (backport/compat/Kconfig) and include files (backport/include/*), presumably.

Or you can wait until we merge. At this point I'm freezing development on compat and compat-drivers to merge all this work onto Johannes' tree.

Quoted 2 lines

If you're interested in alx, you'd have to add that -- look at the copy-list.alx file for that.

Johannes already committed a few things here, I'll be sure to test it as well.

Quoted 2 lines

I hope that soon enough Luis might cut some testing releases from this though.

Absolutely, full steam ahead, just give us a few days.

Quoted 13 lines

I should note that RHEL / SUSE / Debian all had the same issues -- running into symbols we are exporting with the same upstream name. The move to use LINUX_BACKPORT() should help cure these issues in general moving forward. Additionally if you really want to help in this area please review commit 8ae96730 on compat.git specifically Andi Kleen's work on module namespace. If we can figure out to support module namespace *ourselves*, ie, without Linus having to carry this crap around, then we'd solve all these issues easily, I believe we wouldn't have to rely on LINUX_BACKPORT() and in fact that would also mean renaming our *driver* symbols. This means for instance you can run one drivers with your stock mac80211, another with the backported version, should you ever have to support such crazy environement.

As I was watching the patches for LINUX_BACKPORT fly past over the last few weeks and making changes to allow RHEL to work with compat, I wondered, "Why not add the LINUX_BACKPORT macro to each one of the symbols backported?"

Of course that seems extremely tedious, but is there a compelling reason why something like this should not be done -- other than possibly doing as Luis suggests and finding a way to resurrect Andi Kleen's proposal for module symbol namespaces -- then I think the option to begin a more liberal use of the LINUX_BACKPORT macro is worth considering.

#8 lore raw
Quoted 15 lines · Andy Gospodarek
Quoted 4 lines · Luis R. Rodriguez

I should note that RHEL / SUSE / Debian all had the same issues -- running into symbols we are exporting with the same upstream name. The move to use LINUX_BACKPORT() should help cure these issues in general moving forward. Additionally if you really want to help in this area

Quoted 9 lines

please review commit 8ae96730 on compat.git specifically Andi Kleen's work on module namespace. If we can figure out to support module namespace *ourselves*, ie, without Linus having to carry this crap around, then we'd solve all these issues easily, I believe we wouldn't have to rely on LINUX_BACKPORT() and in fact that would also mean renaming our *driver* symbols. This means for instance you can run one drivers with your stock mac80211, another with the backported version, should you ever have to support such crazy environement.

Quoted 4 lines

As I was watching the patches for LINUX_BACKPORT fly past over the last few weeks and making changes to allow RHEL to work with compat, I wondered, "Why not add the LINUX_BACKPORT macro to each one of the symbols backported?"

Its sort of already done and committed!

Quoted 5 lines

Of course that seems extremely tedious, but is there a compelling reason why something like this should not be done -- other than possibly doing as Luis suggests and finding a way to resurrect Andi Kleen's proposal for module symbol namespaces -- then I think the option to begin a more liberal use of the LINUX_BACKPORT macro is worth considering.

Johannes some idea to help with the tedious work of doing this moving forward for new code. I'll let him dive in when he awakens.

Luis

#9 lore raw
Quoted 8 lines · Luis R. Rodriguez
Quoted 5 lines

Of course that seems extremely tedious, but is there a compelling reason why something like this should not be done -- other than possibly doing as Luis suggests and finding a way to resurrect Andi Kleen's proposal for module symbol namespaces -- then I think the option to begin a more liberal use of the LINUX_BACKPORT macro is worth considering.

Johannes some idea to help with the tedious work of doing this moving forward for new code. I'll let him dive in when he awakens.

Well my idea here was more related to the fact that right now we copy a lot of code into compat, and it seems a bit tedious to maintain this manually. Also, if using the 'partial backport' where you only put a single driver into the copy-list file, we still carry everything needed to backport any driver etc.

My idea revolves around a few observations I made:
a) we typically copy an entire subsystem (usually a C and H file) from
the
kernel and then add LINUX_BACKPORT() to the symbols
b) we already need to add some configuration for this, in Kconfig (in
the new
system anyway)

So take for example the recent addition of HDMI. Luis did basically three things:

1) add it to Makefile/Kconfig
2) copy kernel/drivers/video/hdmi.c to backport/compat/hdmi.c and kernel/include/linux/hdmi.h to backport/include/linux/hdmi.h
3) modify the hdmi.h file like this:

+#ifndef CPTCFG_BACKPORT_BUILD_HDMI
+#include_next <linux/hdmi.h>
+#else
...
+#define hdmi_spd_infoframe_init LINUX_BACKPORT(hdmi_spd_infoframe_init)
...
+#define hdmi_audio_infoframe_init LINUX_BACKPORT(hdmi_audio_infoframe_init)
(etc)
#endif

I'll note that he forgot to add LINUX_BACKPORT() to hdmi_spd_infoframe_init and hdmi_spd_infoframe_pack :-)

So with these observations, why not do the following instead:

1) add it to Makefile/Kconfig (manually), and include the information
about the
C/H file(s) into e.g. the Kconfig file, or a separate configuration
file. I
wouldn't use the copy-list since we need to do some transformation
steps.

2) At backport generation time, use that information to copy the C/H
files just
as Luis did manually for HDMI, however, copy
kernel/include/linux/hdmi.h to
backport/include/linux/backport_hdmi.h instead.

3) At backport generation time, create include/linux/hdmi.h like this:

#ifndef CPTCFG_BACKPORT_BUILD_HDMI
#include_next <linux/hdmi.h>
#else
#ifndef __BACKPORT_HDMI
#define __BACKPORT_HDMI
#define hdmi_avi_infoframe_init LINUX_BACKPORT(hdmi_avi_infoframe_init) #define hdmi_avi_infoframe_pack LINUX_BACKPORT(hdmi_avi_infoframe_pack)
...
#include <linux/backport_hdmi.h>
#endif /* __BACKPORT_HDMI */
#endif /* CPTCFG_BACKPORT_BUILD_HDMI */

The biggest 'issue' here is how to find the symbols. Luis even missed some, as I noticed, because there's an easy algorithm for this:

sed 's/^EXPORT_SYMBOL\(_GPL\|\)(\([^)]*\)/#define \2 LINUX_BACKPORT(\2)/;t;d' < C file(s)

Basically that was my idea. Maybe we should attempt to do this instead of backporting HDMI? It seems for HDMI this would yield a more correct version that Luis's current patch :-D

If there are subsystems with multiple header files (not counting uapi/, those don't declare symbols and can probably just copied as is) then we might want to stick #undef in there before each #define.

Extending it a bit further, we could do this _after_ we did Makefile/Kconfig "impossible" analysis. This way, HDMI would only be copied if the copy-list included a driver that requires it, otherwise it would be left out.

One way to configure it would be to put this into the Kconfig (as structured comments), maybe like this:

config BACKPORT_BUILD_HDMI

	...
	#backport-C-file drivers/video/hdmi.c
	#backport-H-file include/linux/hdmi.h

That'd be easy to read and easy to see almost everything (except the Makefile snippet) at once. Heck, we could even generate the Makefile line automatically by just writing

	obj-$(CPTCFG_BACKPORT_BUILD_HDMI) += hdmi.o

to the end of backport/compat/Makefile ...

Thoughts?

johannes

#10 lore raw
Quoted 4 lines · Johannes Berg

config BACKPORT_BUILD_HDMI

	...
	#backport-C-file drivers/video/hdmi.c
	#backport-H-file include/linux/hdmi.h

If we generate the makefile line for this automatically, we'll have to be careful, some things should be built-in:

compat-$(CPTCFG_BACKPORT_BUILD_HDMI) += hdmi.o

while others should be modules

obj-$(CPTCFG_BACKPORT_BUILD_AVERAGE) += average.o

I think the way to distinguish this would be to check whether the BACKPORT_BUILD_* option is "bool" (built-in or not present) or tristate (will be made modular only)

johannes

#11 lore raw
In reply to Phil Perry
Quoted 14 lines · Phil Perry
Quoted 3 lines · Johannes Berg
Quoted 2 lines

* Redhat kernel backport support -- this should be fairly easy

I too am very interested in support for RHEL.

We had support up to date for compat-wireless-3.5 under RHEL-6.3.

Unsurprisingly the release of RHEL-6.4 broke a whole load of stuff given the amount of backporting RH did into their kernel (I believe they backported the wireless stack from ~ kernel-3.5 into their kernel).

Quoted 7 lines

I (and a colleague) played with compat-drivers-2013-03-07-u.tar.bz2 to get alx to build under RHEL-6.4 but I never got as far as looking at building everything. The seemingly endless cycle of breakage on every RHEL release and having to fix it kind of left me losing the will to keep fixing it, especially as for the moment the RHEL kernel has a relatively recent wireless driver stack. Any solution that might ease this break and fix cycle would be very welcome here.

I'm glad to hear the use of compat-wireless is at least moderately useful on RHEL6. A few of us have an interest in keeping compat going on RHEL6, but none had the cycles to get compat.git fixed-up before 6.4 was released. I'm actually in the process of posting patches for 6.3.z as someone had created an upstream bugzilla for this, so I would encourage anyone who wants it working on 6.4 to do the same.

Quoted 9 lines
Quoted 3 lines

I addressed this, but I'm unsure of it and don't have a way to test it, so any testing would be appreciated (as it is in general, of course)

I've not looked closely at what you've done, but I have an interest in building/packaging compat-drivers for RHEL and can possibly help test. If you have a snapshot that you think should build on RHEL or that I could use as a starting point?

I would also like to see this -- though I guess I can probably just clone the git repo and check it out.