RFC: what's in a stable series?

So, I was wondering if it is possible to find a consensus on what sorts of changes are ok in a stable series, as it matures. This mainly applies to 2.4 right now, but I am interested in thoughts on 2.6.x also.

The recent "big fuss over a small change", the direct_IO API change, is indicative to me of a larger question: what sort of API changes are acceptable in a stable series?

Linux has traditionally been fairly loose with caring about API changes.
We "try" to make sure major changes all occur during the development
series, but reality doesn't always play out so nicely. The oft-mentioned chicken-n-egg situation: people really only test the stable series' heavily, so problems that should have been caught during development (if the world was perfect) are instead caught during the stable series. New hardware and features appear, that our existing APIs do not take into account.

First example is the direct_IO change. There are valid reasons to include the change, and valid reasons to avoid it. Vendors (and -ac/-aa?) appear to be shipping this non-standard direct_IO API. It's a good change and has definite value, so why not include it in the 2.4.x kernel? The other side of the argument is breakage not immediately visible: the people maintaining drivers that support multiple kernels must deal with N variations of the same API, within the stable series.

I know the gut reaction on lkml is often to trash these people, but let's be realistic. Linux is huge these days. Drivers normally start life outside the tree, and then once acceptable, are merged. Other drivers must be cross-kernel compatible in order to support all the customers that the hardware vendor decides to support. Red Hat, SuSE, Mandrake, etc. distros in the field are not automagically upgraded -- so if a driver for those existing releases is desired, one must deal with all the API changes. API changes in a stable series instantly make life harder on all these people.

The flip side of the coin is progress.

If most vendors are shipping a non-mainstream change, why not make it mainstream?
Linux users buying new, cool hardware would logically want a driver from the stable series to drive their hardware. If API changes are required to support this hardware, why not go ahead and make the change? It will increase Linux acceptable, market penetration, make people happy, etc., etc.

Adding into the mix, until recently, the 2.4.x series was proceeding at a glacial pace. I think Marcelo has demonstrated that this situation is changes, but we still need to consider it's effects. 2.4.21 was a fairly large chunk of changes, because it took so long. 2.4.22 is absorbing the after-effects of those changes. So one would expect that, (a) with such a time lag between 2.4.x releases (previously), and (b) no 2.6.0 kernel yet, it is understandable that driver authors are concentrating on 2.4, because that's where the users are right now.

What do to?

Currently we are really operating on the "hope and pray" system. Nobody (IMO) really pays much attention to the API changes, since the 2.4.x releases were so far apart. Between releases, "features depending on features depending features" situations were created. However, since Marcelo has fixed this problem, I feel that some reflection on what a stable series means is called for.

Does it mean, no API changes except for security (or similarly severe) bugs? Does it mean, no userland ABI changes, but API changes affecting onto the kernel are ok? Does it mean, "just don't break things such that people are pissed off"?

I request the community's input, particularly those CC'd, for some sort of direction or consensus on this.

In any case, I personally feel that increased stability of the API, coupled with the increased frequency of releases, will make the most people happy. I would prefer some sort of 2.4.x API freeze, say post-2.4.22, but maybe that's too radical?

Jeff

s/onto/only/ :)

In my opinion (and you requested input *g*), the kernel userland API can be changed as much as is required to improve/stabilize/bugfix the kernel, unless this change breaks something in userland without an already available update/upgrade/etc ...

To me it makes no sense to keep an API or some kind of compatibilty thing alive, just because it was defined somewhere along the way, and some programmer might have picked it up ...

when (for example) I decide to use devfs at boot time, completely replacing the traditional /dev I will certainly break some scripts/applications, but it is my decision to do so. The same is true for any kernel update, except for security bugfixes, which are pretty well handled by the big distros anyway ...

In general, I would prefer some -preX series, introducing a new feature or API, hanging around some time (two to three weeks) with occasional code cleanups, waiting for somebody shouting "hey guys, this stuff breaks my famous wossname".

In the end either the new feature is silently accepted, or backed out, starting a new cycle of feature add/feature settle ...

security issues should be addressed in any case.

if some change affects one or two driver developer (read only a few people), then they should agree on a change and let the people test whatever they've agreed on ...

I guess this will be the most favored answer from any user ... and at least some developers ...

I guess this depends on the actual 2.6 time frame ...

best,
Herbert

Changing the kernel/userland API and ABI is totally out of question - we _really_ can't do that. This is about inkernel APIs for modules.

In a lot of cases (like O_DIRECT) its cleaner to simply break the API in a way that will spew warnings if people miss changes than mess around with extra methods that instead break drivers that forgot to use C99 initializers.

I plan to carry on breaking the kernel internal API when I have to and its easy to fix up the few affected users. I broke all the audio drivers between 2.4.21->22 but that was worth doing for example.

yeah, I obviously forgot to mention this case ;) and I agree that sometimes this is the best way to do it ...

but this is a little bit different because
a) you do it intentionally
b) you know how to fix it

keep breaking the kernel internal ... 8-)

best,
Herbert

Do that, and we will stagnate.

That sounds acceptable to me.

But maybe I'm just biased, as I'm looking to start backporting some of the USB 2.5 changes to 2.4 to fix real bugs that are in 2.4. This will require changing kernel apis. And in doing so, fixing up all of the in-kernel usages of that api will happen.

In doing this, it just enforces the fact that it really matters if you have a in-kernel driver or not. If you want to keep a driver or any other kernel code outside of the main tree, it is costly, both in time and effort.

My .02

greg k-h

My initial though on some new feature is to avoid it. We are a stable release, after all.

The quota patches have been around for a long time, and Jan Kara has been trying to include for sometime now (since 2.4.20/21). I tried to avoid it.

Now I realized the possible drawbacks of it are minimal (if any) compared to the overall advantage it brings to Linux 2.4.

The same goes for ACPI, people have been trying to include it for a long time, and I prefered to reject it until 2.4.22-pre.

Its a case-by-case problem. You can't have a general rules like "no API changes except for security (or similarly severe) bugs? Does it mean, no userland ABI changes, but API changes affecting onto the kernel are ok?"

I reverted the direct IO patches because hch complained on me that they change the direct IO API, and we really dont want that kind of change, IMHO.

Trond/Christoph/you now have the direct_io2 possibility, which wouldnt break the current API and still get us the desired "feature".

I also plan a 2.4.x API freeze. Maybe latter.

In general, I think each case is a case and has its tradeoffs which need to be discussed and agreed on.

It is. Generally I think we should prefer to do the right thing rather than adhering to the old API out of some principle.

Evaluate the impact on out-of-tree kernel patches (especially vendor kernels) and if it is unacceptable then reject the change or augment the API rather than changing it.

OK, we're on to a specific case. Albeit a very small one.

I think Trond's direct IO change was right. The impact on out-of-tree code is infinitesimal. Stick a #define O_DIRECT_NEEDS_A_FILP in the header and let the XFS guys write a four-line patch. There's no point in mucking up the kernel API to save such a small amount of work.

Or merge XFS.

Oh, we have that patch even without the feature define in say -ac and -aa but it's just horrible to have APIs silently change behind you. Especially when just changing a function arg where you only get one more warning in the forrest of warnings produced by gcc 3.3 on a 2.4 tree..

That's of course a good idea [1] but doesn't really help in this discussion. There's other filesystems like ocfs or opengfs that have the same kind of problems.

[1] and with the new quota code and vmap() we're almost there..

Switching topics (to one of my favourites):

This is a perfect demonstration of why it is utterly unacceptable for compiler developers to add new warnings without also providing a way of turning them off.

Also the quota patches don't change any ABI or API - userland can still use the old ABI in addition to the new one, 16bit ondisk quotas are still supported and filesystems couldn't care less which implementation it plugs into - the API is the same.

Because you hacked v1 support out of Jan Kara's stuff the quota bits are pretty useless to most people because they have v1 format files.

Linus removed from CC because I think he does not care about 2.4.x specific issues - which is what this thread is now about.

So Christoph's quota patch does not support vendors "v1" files?

I must be misunderstanding someone.

There are three species of quota in Linux

v0 (official old Linux)
v1 (most 2.4 vendor trees)
v2 (the 2.5 format)

[Plus other per file system stuff that is abstracted by the quota updates and used by folks like XFS]

The v1 quota definitely doesn't belong in 2.5, but its needed by a lot of 2.4 users who come from vendor trees (and may want to go back again)

Umm, no. You misunderstood.

There's two quota _ondisk_ formats:

v1 old 16bit quota.
v2 new 32bit quota.

2.4 previously only supported v1, 2.5 and 2.4.22-pre4 also support v2.

There's three sys_quotactl ABIs

1) original 16 bit one, supported by all kernels <= 2.4
2) first 32bit one, supported by 2.4-ac any many vendor trees, but never in mainline
3) new 32bit one, supported by 2.4.x (x >= 22-pre4) and 2.5

Unfortunately the second one uses the same constants as the old 16bit one but different structures so there is no way to support both in a single kernel.

Exactly.

Actually at least later SuSE kernels and -ac kernels support 3)...

Right.

Honza

Jan Kara <[email protected]>
SuSE CR Labs

And the latest 2.4.20-based Red Hat kernel (RH 7/8/9 errata) support _only_ 3, and the 32bit ondisk quota format.

Ok that makes a lot more sense

Alan obviously didn't even look at the code. Of course the old quota format is supported as described in the mail I sent you the patch with.

Please take a look at the code before saying something that unqalified. ( or just read the changelog..).

The old quota format is enabled unconditionally, not removed.