RFD: Kernel release numbering

This is an idea that has been brewing for some time: Andrew has mentioned it a couple of times, I've talked to some people about it, and today Davem sent a suggestion along similar lines to me for 2.6.12.

Namely that we could adopt the even/odd numbering scheme that we used to do on a minor number basis, and instead of dropping it entirely like we did, we could have just moved it to the release number, as an indication of what was the intent of the release.

The problem with major development trees like 2.4.x vs 2.5.x was that the release cycles were too long, and that people hated the back- and forward-porting. That said, it did serve a purpose - people kind of knew where they stood, even though we always ended up having to have big changes in the stable tree too, just to keep up with a changing landscape.

So the suggestion on the table would be to go back to even/odd, but do it at the "micro-level" of single releases, rather than make it a two- or three-year release cycle.

In this setup, all kernels would still be _stable_, in the sense that we don't anticipate any real breakage (if we end up having to rip up so much basic stuff that we have to break anything, we'd go back to the 2.7.x kind of numbering scheme). So we should fear odd releases, but track them, to make sure that they are good (if you don't track them, and problems won't be fixed in the even version either)

But we'd basically have stricter concerns for an even release, and in particular the plan would be that the diff files would alternate between bigger ones (the 2.6.10->11 full diff was almost 5MB) and smaller ones (a 2.6.11->12 release would be a "stability only" thing, and hopefully the diff file would be much smaller).

We'd still do the -rcX candidates as we go along in either case, so as a user you wouldn't even _need_ to know, but the numbering would be a rough guide to intentions. Ie I'd expect that distributions would always try to base their stuff off a 2.6.<even> release.

It seems like a sensible approach, and it's not like the 2.4.x vs 2.5.x kind of even/odd thing didn't _work_, the problems really were an issue of too big granularity making it hard for user and developers alike. So I see this as a tweak of the "let's drop the notion althogether for now" decision, and just modify it to "even/odd is meaningful at all levels".

In other words, we'd have an increasing level of instability with an odd release number, depending on how long-term the instability is.

- 2.6.<even>: even at all levels, aim for having had minimally intrusive patches leading up to it (timeframe: a week or two)

with the odd numbers going like:

- 2.6.<odd>: still a stable kernel, but accept bigger changes leading up to it (timeframe: a month or two).
- 2.<odd>.x: aim for big changes that may destabilize the kernel for several releases (timeframe: a year or two)
- <odd>.x.x: Linus went crazy, broke absolutely _everything_, and rewrote the kernel to be a microkernel using a special message-passing version of Visual Basic. (timeframe: "we expect that he will be released from the mental institution in a decade or two").

The reason I put a shorter timeframe on the "all-even" kernel is because I don't want developers to be too itchy and sitting on stuff for too long if they did something slightly bigger. In theory, the longer the better there, but in practice this release numbering is still nothing but a hint of the _intent_ of the developers - it's still not a guarantee of "we fixed all bugs", and anybody who expects that (and tries to avoid all odd release entirely) is just setting himself up for not testing - and thus bugs.

Comments?

Linus

Just rename:

2.<even>.<odd>-rcX  ->  2.<even>.y-preX
2.<even>.<odd>      ->  2.<even>.y-rcX
2.<even>.<even>     ->  2.<even>.y

Dmitry

If the users wouldn't even have to know, why do it? Who will benefit from this, then?

I think a better approach, and one which is already working out well in practice, is to put "more intrusive" features into -mm first, and only migrate them into 2.6.x when they have 'stabilized'.

This could be improved: _All_ new features have to go through -mm first for a period (of whatever length) / one cycle. 2.6.x only directly picks up "obvious" bugfixes, and a select set of features which have ripened in -mm. 2.6.x-pre releases would then basically "only" clean up integration bugs.

-mm would be the 'feature tree'. Of course, features which have matured in other eligible trees might also work; the key point is the two-stage approach and it doesn't matter whether the chaos stage has one or three trees, as long as it's not more than that.

I think that would be natural extension of how things already work and just tightens the process some.

(From a vendor perspective, this would mean we'd be safe picking up any 2.6.x tree + select choices from x+1-pre plus whatever we are force fed by those who pay.)

If one wanted to get fancy, which I'm throwing in just to make everybody lose the core point of the argument: One could associate "points" with each feature / patch in -mm, based on an estimation of how intrusive/well-tested/dangerous/heavenly that patch is, and mandate that only 42 points per 2.6.x release are allowed.

Of course, one could also apply common sense. But, that's not as silly. Or way more so, but less amusing than voting wars.

The numbering scheme is more confusing and unclear, and complexity is the enemy of reliability.

Sincerely,
Lars Marowsky-Brée <[email protected]>

High Availability & Clustering
SUSE Labs, Research and Development
SUSE LINUX Products GmbH - A Novell Business

This is the way things work today already. The only exception being the networking code, but hey, networking's always been special :)

thanks,

greg k-h

That's exactly what I'm saying. It already works this way, w/o misleading the users and attaching confusing meaning to minor revision numbers. It could do with being more clearly advertised and sticked to, but that's about it.

Don't confuse more. People are already way to confused on their own.

If anything, if we expect a release to have ... interesting side-effects, Linus will find just the funny words for the release notes to say so ;-)

Sincerely,
Lars Marowsky-Brée <[email protected]>

High Availability & Clustering
SUSE Labs, Research and Development
SUSE LINUX Products GmbH - A Novell Business

Certainly -mm can be the feature tree, but i've noticed that not that many people run -mm aside from developers. Meaning that a fair number of bugs seep into Linus' tree before they get attended to. It would even be more effective if we could get more -mm user coverage. A Linus based odd number might be closer to that if we hope on people unwittingly running them.

                                               ^^^^^^^^^^^

I think this is a very unhelpful attitude. Don't expect people to do things unwittingly. It won't work.

You should expect people to be more intelligent and more confused than you expect (yes, I know that is recursive).

Being more intelligent means you won't be able to trick them. Being more confused means they will try out less things.

Constantly changing the naming will confuse people, and they will experiment less.
Having some clear and often-stated gaols for different releases - which are adhered to - will make people feel less confused and so more willing to experiment. Asking nicely probably helps too.

--e.g.--
The latest 2.6.X is stable. Feel free to use it for any system. The latest 2.6.X-pre is fairly stable. Please consider using it on
non-"mission critical" system.
The latest 2.6.X-mmY is under development. Please test it if you have
the opportunity.

Agree on that. Stick to it. Get everyone to plaster it on their websites. Stick it at the bottom of all @vger.kernel.org mailing lists. See what happens.

NeilBrown

I think you may be reading too much into that, by unwittingly i meant that people tend to run Linus' releases more than others because they are considered to be more stable.

Thanks,
Zwane

They don't _have_ to know. But both users and developers can take advantage of this to time their patches.

That wouldn't change. But we still have the issue of "they have to be released sometime". This makes it clear to everybody when to merge, and when to calm down.

Linus

So is the problem that people aren't listening when you say "lets slow down" ? Why would this change things for people who obviously ignore what you say ? :)

I'll bet you'll still get flooded with "lets see if Linus takes this despite what he said in his last announcement" patches if we moved to this model.

The only thing that would make a difference afaics, would be you putting your foot down and just ignoring such submissions ?

Dave

ROTFL. You've not watched Linus for a long time have you. I could mention 2.1.102, 1.3.59 and a few other "just ready for release so I added this cool..." cases 8)

If they're feeling itchy they should dig in and help fix the bugs in 2.6.<odd>.

Yep, that's what I'll be doing. :)

~Randy

__Stable__ would be a good thing. The entire 2.6 development has been a disaster from a stability viewpoint. I have to maintain a huge tree of patches in order to ship appliance builds due to the lack of stability for 2.6. I think that the even number releases will take longer but it's worth the wait.

Jeff

Linus's release cycle estimate might be optimistic. :)

I'm seeing lots more bug reports recently than I care to see. :(

~Randy

Yep, but at least he is addressing the issue and taking responsibility and showing good prudent leadership and some reasonable and sound ideas to correct it.

Yep.

Jeff

What form of instability are you referring to?

Where have you been? Read the bug reports and the types of bugs.

Jeff

I'm only emailing to the list, feel free to keep my in CC (this way I'll know what part of the thread was directed towards me)

I agree about the stability of the 2.6 kernels. The system I'm using now has always been 2.6 since I first installed it. I have noticed there were stability issues with 2.6. I remember 2.6.7 was fairly good and was a bit stable. 2.6.8.1 wasn't that stable (I'm sure I had the .1 patch on it, however, attempting to do a lock over NFS caused the system to hard freeze). 2.6.9 was more unstable especially with USB. 2.6.10 (which I'm using now) has to be IMO the most stable 2.6 kernel produced. I'm quite pleased with it (and to all the kernel hackers, thanks for a great kernel).

P.S. System is a dual xeon 2.6ghz on a supermicro x5da8 1gb ram.

 Lab tests show that use of micro$oft causes cancer in lab animals

Remember - each patch you successfully push upstream is one less patch you have to maintain. So re-read Documentation/SubmittingPatches, beat the patches into shape, and send them on their way. ;)

This sounds good, until you realise that some of us have been sitting on about 30 patches for at least the last month, because we where following your guidelines about the -rc's. Things like adding support for new ARM machines and other devices, dynamic tick support for ARM, etc.

If I'm going to have to sit on this stuff for another month, it'll bit rot rather badly, and I might as well throw away all these patches now and ask people not to send stuff other than pure bug fixes.

A slightly bigger problem is that I'm in the middle of merging a large chunk of these for you right now, and at least Andrew has already pulled some of these from my BK repo.

If we are going to do this, can we start it post 2.6.12 please?

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

The fact that this new approach serialises the stable/devel lineation whereas traditionally it was parallel (2.x.y/2.x+1.y) is going to be a real pain for a lot of maintainers.

In short, instead of a single 'merge with linus tree', I'm now going to need a 'merge with linus' and 'merge with linus next time' tree for every tree I maintain.. It's not impossible to maintain, but its extra burden. Burden which a lot of folks may consider not worth it.

Dave

30? Try 310 changesets, in my netdev-2.6 pending queue.

Jeff

Note that I don't think a 2.6.<even> would have problems with things like driver updates.

This was somewhat brought on (at least for me, dunno about Davem) by things like 4-level page tables etc stuff. I don't think most people even realized how _smoothly_ that thing seems to have gone, even if the ppc64 people ended up having some really nasty debugging (and they came through with flying colors, but they probably didn't much enjoy the thing).

I would not keep regular driver updates from a 2.6.<even> thing. But I _would_ try to keep things like all the TSO pain, the 4-level page tables, and in general big merges that have been in CVS trees etc, and can't claim to be "lots of small stuff".

For example, there's some fundamental 16-bit PCMCIA cleanups pending in the -mm tree, meaning that the kernel can work with PCMCIA cards without a "cardmgr" deamon. That would be something that is _not_ appropriate in the first stable version. That doesn't mean that individual PCMCIA device drivers could not get updated.

But hey, you may be right. It might just not be obvious enough which class any particular merge falls under.

Linus

I think this statement proves that the current development situation is working quite well. The nasty breakage and details got worked out in the -mm tree, and then flowed into your tree when they seemed sane.

So, any driver stuff is just fine? Great, I don't have an issue with your proposal then, as it wouldn't affect me that much :)

I do understand what you are trying to achieve here, people don't really test the -rc releases as much as a "real" 2.6.11 release. Getting a week of testing and bugfix only type patches to then release a 2.6.12 makes a lot of sense. For example, see all of the bug reports that came out of the woodwork today on lkml from the 2.6.11 release...

But accepting 310 netdev patches, 250 USB, 50 PCI, 50 I2c, and 100 ALSA patches in one week and expect the tree to stay "stable" might be a pretty unreasonable thing to wish for...

thanks,

greg k-h

Actually, the breakage I was talking about got fixed in _my_ tree.

I'd love for the -mm tree to get more testing, but it doesn't.

I don't know about "any", but yeah.

A large part of it is psychological. On the other hand, it may be that Neil is right and it would just mean that people wouldn't even test the odd releases (..because they want to wait a couple of weeks for the even one), so it may not actually end up helping much.

The thing is, I _do_ believe the current setup is working reasonably well. But I also do know that some people (a fairly small group, but anyway) seem to want an extra level of stability - although those people seem to not talk so much about "it works" kind of stability, but literally a "we can't keep up" kind of stability (ie at least a noticeable percentage of that group is not complaining about crashes, they are complaining about speed of development).

And I suspect that _anything_ I do won't make those people happy.

Linus

Well, that might change if, when I came crying to the list about something thats broken in an -mm release, I wasn't chased off to go run a "more stable" release. Thats occured 2-3 times in the past year.

I'll willingly play the canary as long as I don't wind up with a totally hosed filesystem. So far, knock on wood, I've been fairly lucky and have not had to do a bare metal recovery from amanda.

Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.34% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.

A gentoo view: There are lots of patchsets floating around in the gentoo forum based on either vanilla or mm-kernel, but over the months something has
changed: Previously most patchsets were based on mm, but now are based on vanilla. Why? Very simple: mm became too unstable. I used to go with a mm based kernel just for fun, but it changed as one kernel had some serious issues with reiserfs - and it is really not fun to lose data. (At least I read about it, before testing that kernel.) Since then i never touched a mm-kernel again, in fact now I even feel scared to put on a vanilla-rc kernel. I do it, but I feel like when I use a "stable" mm-kernel from earlier times...

So if you wantpeople to test kernels, they shouldn't be too unstable...

And if you want bug reports, make it easier for the user. I know there is a txt file in the kernel src dir, but it would be better, if there would be a complete script which gets all possible need infos itself. Then user driven input could be minimized.

Furthermore many users are simple afraid to post to lkml. (They report to forums, but don't want to report to lkml.) Even I often tend to think: Ok, I came across a small bug. Should I post it or should I wait till next rc? Usually I wait, because I feel like getting on people's nerves when I post every little tidbit. Esp if bugreports seem to get ignored motivation goes down to report again...

Nevertheless, for a fun loving desktop user, I enjoy the pace the linux kernel is evolving. It was so "boring" the 2.4 days. But on the other hand linux kernel now tends to break various things with each release. I think esp nvidia users now about it...

Cheers

--
Prakash Punnoor

formerly known as Prakash K. Cheemplavam

On Thu, 03 Mar 2005 11:29:07 +0100, Prakash Punnoor <[email protected]> wrote:
[...]

Yup, having an interactive script that produce a bug-report-date.txt file would be great.
Furthemore I don't understand why bugzilla is not used as the main bug reporting tool.

Nvidia binary driver is a totaly different problem.

Paolo <paolo dot ciarrocchi at gmail dot com>
msn: [email protected]
hello: ciarrop
paoloc.blogspot.com

Thanks, it's about time someone said this.

We had a preview of what it would be like if "more users tested -mm" when Ingo's realtime preempt patches were based on it, so the audio oriented distros briefly shipped -mm kernels.

It was a nightmare. Most people (icluding myself) experienced daily lockups. When Ingo rebased the RT patch against mainline there was muich rejoicing on LAU.

-mm is just too unstable for anyone but kernel hackers to run. Anyone who tells you otherwise is smoking something.

Lee

It's not the same thing to test ONE release and test SEVERAL -rc. I take my case as an example. I don't have enough time to keep up, so I try to compile at least each release and test them on one of my systems, maybe for curiosity. I would like to test the -rc, but the problem is that you don't know how much you can expect this or that -rc to be close to usable. You even expect it to fail or not build at all, so when you don't have much time to spend on this, unfortunately, you don't test them.

On the contrary, Marcelo makes a strong difference between -pre and -rc. And when I see a 2.4-rc, I expect it to be a potential candidate, so I try to get some time to compile it just in case I would discover an awful bug, and avoid the 2.6.8 -> 2.6.8.1 situation. Honnestly, I would have reported the 2.6.8 NFS bug earlier if there had been a true -rc before it (=one which does not change except for trivial bug fixes).

The only solution against this is to freeze for real and start the devel tree in parallel. People are not complaining anymore about 2.4 change speed, because every time a folk sends something, we tell him to push that into 2.6 and not 2.4. The same thing must work with 2.6 and 2.7. In the end, there would a general feeling that "2.6 was not as stable as 2.8", etc... That's not a problem, there are always ups and downs in software.

Regards,
Willy

That single sentence sums it up perfectly. When I have given talks about how our current development cycle works, and what's happening with it, people just feel odd seeing all of this change happen and get upset at it. Perhaps it's because they never paid attention before, or that they are new to Linux and like to believe that old-style development models were somehow "better", and that they know we are doing something wrong.

But when pressed about the issue of speed of development, rate of change, feature increase, driver updates, and so on, no one else has any clue of what to do. They respond with, "but only put bugfixes into a stable release." My comeback is explaining how we handle lots of different types of bugfixes, by api changes, real fixes, and driver updates for new hardware. Sometimes these cause other bugs to happen, or just get shaken out where they were previously hiding (acpi is a great example of this issue.) In the end, they usually fall back on muttering, "well, I'm just glad that I'm not a kernel developer..." and back away.

Like I previously said, I think we're doing a great job. The current -mm staging area could use some more testers to help weed out the real issues, and we could do "real" releases a bit faster than every 2 months or so. But other than that, we have adapted over the years to handle this extremely high rate of change in a pretty sane manner.

So don't spend too much time worring that we need to make those types of people happy. As Jeff said, no other OS is accepting changes at such a high rate. But then again, no other OS supports more devices, on more different processors. So we must be doing something right :)

thanks,

greg k-h

The pertinent question for a point release (2.6.X.Y) would simply be "does a 2.6.11 user really need this fix?"

I think Linus's "even/odd" proposal is an admission that 2.6.X releases need some important fixes after it hits kernel.org.

Otherwise 2.6.X is simply a constantly indeterminent state.

We need to serve users, not just make life easier for kernel developers ;-)

Jeff

"need this fix bad enough now, or can it wait until 2.6.12?"

I agree.

Heh, true, but all software is in that state :)

Damm those pesky users. Without them our life would be so much easier...

greg k-h

hehe :)

I do honestly worry that sometimes we lose sight of what best serves our users.

We need to not only produce a useful kernel, but also package it in a way that is useful to the direct consumers of the kernel: distros [large and small] and power users.

What do those direct consumers really want? What best serves them?

Jeff

This comes down to the question "what are we making"? Is it an end product, or is it a technology which can be turned into an end product? Because the two are different things.

I'd say that mainline kernel.org for the past couple of years has been a technology, not a product.

We do put effort into productising that technology every couple of months because that's what a (very important) minority of our users want, and because it is good discipline, but productisation hasn't been the main focus.

If we were to get serious with maintenance of 2.6.x.y streams then that is a 100% productisation activity. It's a very useful activity, and there is demand for it. But it is a very different activity. And a lot of this discussion has been getting these two activities confused.

But there's something else:

I would maintain that we're still fixing stuff faster than we're breaking stuff. If you look at the fixes which are going into the tree (and there are a HUGE number of fixes), many of them are addressing problems which have been there for a long time.

So as long as we remain in this state, we don't need to do anything. The technology gets closer to a product until we reach the stage where the
fixage rate equals the breakage rate. And we're not there yet.

(It's nice that patches are called "fix the frobnozzle gadget", but this analysis would be a lot easier if people would also label their patches "break the frobnozzle gadget" when that's what they do. Oh well).

So I'd suspect that on average, kernel releases are getting more stable. But the big big problem we have is that even though we fixed ten things for each one thing we broke, those single breakages tend to be prominent, and people get upset. It's fairly bad PR that Dell Inspiron keyboards don't work in 2.6.11, for example...

And people will incorrectly (and even wildly) generalise as a result of such silly little isolated bugs. We can wholly address such problems with a 2.6.x.y productisation series.

And something else:

I don't think 2.2 and 2.4 models are applicable any more. There are more of us, we're better (and older) than we used to be, we're better paid (and hence able to work more), our human processes are better and the tools are better. This all adds up to a qualitative shift in the rate and accuracy of development. We need to take this into account when thinking about processes.

It's important to remember that all those kernel developers out there *aren't going to stop typing*. They're just going to keep on spewing out near-production-quality code with the very reasonable expectation that it'll become publically available in less than three years. We need processes which will allow that.

And another else:

Many people on this mailing list want a super-stable kernel as their first (and sometimes only) priority (the product group). But others have other
requirements: to make their code avaialble, or to get their hardware supported, or to fix that scalability problem (the technology group). The product group's interests are in conflict with the technology group's.

There will be no solution to this problem which is completely satisfactory to either party.

That's the primary question, of course.

Except I'd replace the term "consumers" with the more general term "customers". And I'd then say that, for example, the SGI Altix development group is a customer of the kernel development team.

And how do we satisfy that customer? By merging scary patches and hence screwing your "consumers" ;)

oops. Tension..

If you consider mainline a technology and distributions your main users, what is the use of a stable release every months or two months? No distribution is going to updates its release that often. Looking at the Debian kernel packages it took at least a month just to get a single release ready for distro use. Needless to say, Debian (or any other distro) is not going to go through that for every release.

We already saw that with 2.0, 2.2 and 2.4 kernels: distributions rarely used the last mainline release but older released with (a sometimes huge amount of) patches.

So continueing that thought pattern; why not go for something like 6 month release cycles? That seems to fit with a distro release cycles.

same for latitude keyboards after a resume I just discovered :(

Wichert.

Wichert Akkerman <[email protected]>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.

I don't know, because we've never tried it. One can only speculate, and I'd speculate that the kernel would end up pretty crappy because there would never be a point in time when the thing was sufficiently usable.

The need to periodically lock things down and concentrate on stability and generally get our act together every couple of months is good discipline even if we're only making a technology, IMO. And it gets more testers.

That could work. The releases are slowing down as we spend longer wringing out bugs. Six months would be a bit excessive though.

But before solutions are discussed, we need to work out what the question was.

What are we trying to fix, and why?

You know what to do ;)

Correct. That's what -ac and -as kernels try to achieve. Moving those activities into 2.6.x.y would be easier to understand for users.

Ack.

Right, but moving to the even/odd scheme is worse. Even versions will be ignored within no time like the -rc versions are now. The result will be that the "stable" odd releases will be less frequent and the -rc phase of those will be as hard to sell to testers as the current ones. We will end up with 2.6.ODD in the same shape as we do now with 2.6.x releases.

Moving to a -preX / rcX scheme might help to convince users to give the -rc versions a try and help to stabilze for the real release.

What about:
2.6-mm
2.6-devel
2.6.x-stable

-mm bleeding edge
-devel stabilized from -mm and subsystems
-stable stabilized from devel and real bugfixes

In this case the current -rc scheme could work in a real "release candidate" scheme.

The devel tree would not be bound to release cycles and would just contain ongoing development, so the held back pile of changes is not growing up in the way it does now.

tglx

Even-numbered kernels started becoming a product after the next odd-numbered kernel series started.

2.4 might have it's limitations, but today it's a pretty good product.

The point behind this is:

It's not the most important question for a user whether the total number of bugs has decreased or increased.

The most serious problems for users are regressions compared to the kernel before.

If a user who is happily using 2.6.8 learned that "stable" kernel 2.6.9 broke driver A for him and "stable" kernel 2.6.10 broke driver B for him, he will not try any new "stable" kernel simply because it only causes trouble for him.

Bad luck, if he therefore misses some serious security fix.

And yes, there are many people out there who use for many different reasons self-compiled kernels.

The traditional solution was to have a development series for the developers who don't stop typing and a stable series with few regresssions.

Until now, it seems 2.6 has too many regressions in every new released kernel for being a really stable kernel.

And people aren't dumb - you can't fool them with any version number games. People have learned that a -rc by Linus is equivalent to a -pre release by Marcelo. And they will quickly learn that a 2.6.<even> kernel will be equivalent to a -pre releaese by Marcelo.

What about thinking instead how to get a 2.7 cycle that roughly fits everyones needs?

It took two years from 2.5.0 to 2.6.0 .
That was long. Let's try to shorten it.

Make the feature freeze half a year after the start of 2.5 instead of one year as was done in 2.5 . This cuts off half a year of the two years. Yes, it really cuts off half a year, because the first year of 2.5 also included half a year of IDE changes where even the bravest kernel developer didn't dare to test the kernel.

Perhaps there are also ways how the more developers, better tools and processes can help in shortening the one year after halloween.

cu
Adrian

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Rather than mixing problem and solution, let me just define the two problems in this thread:

1) There is no clear, CONSISTENT point where "bugfixes only" begins. Right now, it could be -rc2, -rc3, -rc4... who knows.

We need to send a clear signal to users "this is when you can really start hammering it." A signal that does not change from release to release. A signal that does not require intimate knowledge of the kernel devel process.

This is a key reason why we don't get more pre-release testing.

2) After 2.6.11 release is out, there is no established process for "oh shit, 2.6.11 users will really want that fixed."

--------------------

Linus's even/odd proposal is an example of a solution for problem #2, as is my 2.6.X.Y proposal.

The 2.4.x series -pre/-rc is an example of a solution for problem #1.

Jeff

The problem I see with that is that the majority is going to wait until one or two releases after that point since the first 'stablization point' since that the first such release will not have received much testing yet. Just like the .0 release for a lot of projects turns out to a beta version even though they had a seperate beta test.

Wichert.

Wichert Akkerman <[email protected]>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.

This is exactly how I see it as well. Guess we drank the same koolaid ;-) I don't see the reluctance to use -pre/-rc since that's already what we're doing (just poorly encoded in -rc$x). Also, .x.y does require some release discipline, there may be cases where the .x.y fix should be much simpler than .x+1-pre/rc fix (as in Greg's comment that fixes may include basic API changes).

thanks,
-chris

I think so.

We can do 2.6.11.x scheme. For the last stable kernel, of course (i.e., one additional small patchset). No more 2.6.11.x when 2.6.12 is out.

Yes.
But, to shorten the time between stable kernels, I would limit number of "pre"s. Maybe 1 or even no "pre" kernel is enough? Massive updates would take place before rc1 is released, of course.

Krzysztof Halasa

I'd rather say we don't guarantee there will be one anymore after 2.6.<next>.2 is out. First we need to provide it a little longer because .0 isn't alwaysas stable as we want, second people will fix up older releases when they are used as base kernel for distributions, and there's no point of not marking it official if it is there.

IMHO, Jeff Garzik has made two very useful points in this thread:

1. The number of changesets flowing towards the Linus kernel is accelerating, so the kernel developers should be trying to accelerate the merging process, not introducing delays. Having an extended -rc period that stuffs up merging just creates back pressure and causes changesets that could be getting reviewed, merged, and booted somewhere to instead lie dormant.

2. No matter what one calls it, -rc1, .<odd>, or just 2.6.X these days, intelligent consumers know a "dot-zero" release when they see one. [I've had experience of several boneheaded corporate policies dictating an unpatched kernel.org kernel, but they are uninteresting users.] The class of users that want to use the kernel in production are going to wait days to weeks, no matter what. The trick is in encouraging everyone else to overcome inertia and test new releases.

As part of a solution to the "production kernel" problem, Jeff suggested a 2.6.x.y tree that gets pulled to 2.6.x+1. Neil Brown made a similar point:

For the kernel, I am the "distribution" for my employer and I choose which kernel to use, with which patches. I really don't want to hunt around for all those stablisation patches, or sift through the patches in 2.6.X+1-pre to find things to apply to 2.6.X. I would be really happy there was a central place where maintainers can put suitably reviewed "important bug fix"es for recent releases, and from where kernel maintainers for any distribution (official or not) could pull them.

I'm in the same boat with Neil. Determined to stay reasonably close to mainline, I started in the 2.6.9-bk series to try to nail down a stable production kernel. I spent about two months reading lkml and bk-commits-head, picking through -mm for patches that might be important for my workloads (e.g., vmtrunc), and spending my days with "quilt", merging up a new -bk kernel every few days, backing out "dangerous changes", and retesting. At 2.6.10, I stopped revving up and started to just merge fixes from 2.6.11-bk.

I'm sure Neil and I are not alone. I perceive four groups of users for kernel.org users, with differing requirements:

1. Developers. For them, the Linus kernel is a synchronization point for merging, as well as their personal test environment.

2. "Casual" end-users who like to build their own kernels, and for whom a kernel oops, crash, or driver failure is not a big hassle; they just reboot into their previous kernel. They are content if a new kernel doesn't corrupt their data.

3. "Production" end-users, who need a kernel that is going to run stably, usually on many servers, indefinitely [until a bug or desired feature forces an upgrade/reboot]. Rolling out a new kernel is a hassle, and is usually done to fix a serious kernel bug or driver problem.

4. Vendors, who need a long period of stabilization and testing, as well as a (vendor-internal) mechanism for determining what features, drivers, etc. to support.

As individuals, many of us live in multiple categories, e.g., I'm a (3) at work, and a mix of (2) [laptop] and (3) [file server] at home.

Greg KH complained:

Bug fixes for what? Kernel api changes that fix bugs? That's pretty big. Some driver fixes, but not others? Driver fixes that are in the middle of bigger, subsystem reworks as a series of patches? All of this currently happens today in the main tree in a semi-cohesive manner. To try to split it out is a very difficult task.

Opinions will differ, but I think things are a lot more clear-cut than Greg allows. I certainly don't expect to download, build, and deploy a kernel devoid of patches without expecting at least a few problems. It's the incredible duplication of effort to sort through thousands of changesets in order to cull dozens to hundreds, with the result that everyone is running a subtly different kernel core. And most of us are far less qualified than subsystem maintainers to evaluate the risk of individual changesets.

Folks in categories (3) and (4) care very deeply about subtle corruption [like the recent pty lost bytes], even if rare, as well as easily triggerable oopses, races, deadlock, livelock, resource leaks, massive performance regressions, and serious breakage in the (rapidly evolving) networking stack. These belong in 2.6.x.y. API changes do not, unless they are required to fix one if the above.

Sure, this is going to create situations, such as just occurred, where the change to 4-level tables meant that some later patches require a bit of love before they'll apply to the previous 2.6.X release or vice-versa; but it isn't an everyday occurence.

Driver fixes? For category (3) users, if one doesn't have the hardware, or the driver is not broken with the end-user's hardware, one mostly doesn't care about driver fixes. Vendors, like Dave Jones, are of course in a different position, because a vendor kernel is a different animal that needs to work everywhere, or bug trackers starts filling up quickly.

Dave has been building "unstable" bleeding-edge Fedora kernels from 2.6.x-rcM-bkN, as well as "test" kernels for Fedora updates; they simply aren't receiving enough testing, and/or the bug reports are going to the wrong place. Similarly, Arjan has been building rpms for Alan's kernels; those kernels are "vanilla" -ac.

Part of the problem here is that most users install e.g., Fedora Core, and don't enable "testing" in their package manager; judging by the Fedora lists, many don't even know about it. [Or don't know or care that they could limit updates from "testing" or "unstable" to just the kernel or other packages that allow multiple versions to be installed; the update would simply fail if a new udev or whatever is required, prompting admin intervention]. This contrast with the much slower-moving Debian, where getting useful work done often requires running parts of "testing" or even "unstable".

There is a large universe of desktop and laptop users who reboot their machines every day, and would probably run the most up-to-date kernel when they boot every morning, confident that they can simply reboot into the last working kernel if there is a problem. But it doesn't happen,
because it is not automatic. In order for this to happen we need
new kernels to be installed automatically and made the default, on systems where the admin has elected to do so, and a policy for cleaning up old kernels that are unused (haven't been used for the last N boots or whatever).

So in short, I'm saying that solutions to the stable kernel problem and the testing problem are not necessarily solved with a single mechanism.

The testing situation would be improved if a distro install asked the end-users whether they'd like to participate in kernel testing, explain the importance of it, and then set up their package manager cron scripts accordingly (-linus, -mm, -ac, whatever). I believe the onus here is on the distros to convince their "hobbyist/enthusiast/sysadmin" users to help them test before wider release.

Regards,

Bill Rugolsky

If you accelerate the merging process, you're lowering the review process. The only answer to get regressions fixed up as quickly as possible (because prevention is nigh on impossible at the current rate, so
any faster is just absurd), would be more regular releases, so that
they got spotted quicker.

Actually only rawhide (FC4-to-be) has been getting -rc-bk patches. The FC2/FC3 updates have been release versions only, with -ac patches. (and also some additional patches backing out bits of the -ac) During a test cycle (Ie, after FC4test1 is released, but before FC4 gold) is the only time Fedora users regularly test a -rc-bk tree. This is part of the problem with rebasing the existing releases to new kernels. It's shoving a largely untested codebase into a release that was never tested in that combination. It's expected that some stuff will break, but the volume of breakage is increasing as time goes on. Even if _I_ stopped rebasing the Fedora kernel, some of our users will still want to build and run the latest kernel.org kernel on their FC2 boxes. We shouldn't be expecting them to have to rebuild half of their userspace just because we've been sloppy and broke interfaces.

Dave

Right. My point, and I think Jeff's, was that being extra careful for the 'even' releases and waiting around N days to see whether someone will finally test the -rc and see that it is broken impedes the whole process. Getting more people to test is not necessarily a function of the wait duration.

When Arjan started testing 2.6, he set up a repo, and FC1 users at the time could just pull rpms from that repo. I and many others did it. Currently I'm rolling my own, so I haven't been consistently testing your Fedora kernels, rawhide or update, but I occasionally do the "grab the SRPM from Rawhide and rebuild it on FC3"-thing for my laptop. I think that we should institutionalize that.

I've watched you periodically announce "hey, I'm doing an update for FC3/FC2, please test" on the mail list, and a handful of people go test. If we could convince many of the the less risk-averse but lazy users to grab kernels automatically from updates/3/testing/ or updates/3/unstable/ as part of "yum update", and have a way to manage the plethora of (even daily) kernel updates by removing old unused kernels, then we'd only have to convince them *once* to set up their YUM repos, and then get them to poweroff or reboot [or use a Xen domain] occasionally. :-)

Yes, this is miserable, and exacerbated by the inability of almost all distros to deal with multiple installed versions of packages, or easily roll back changes, which crimps my argument w.r.t. wider testing, since the typical user willing to test while otherwise doing useful work wants to be able to roll back easily if there is a serious problem.

The LVM packaging situation between 2.4 and 2.6 illustrated the problem well; one couldn't boot back into 2.4 unless LVM1 and LVM2 userland could coexist; I spent time rolling packages back then to do just that. In any case, kernel helper packages (udev, device-mapper, iptables, etc.) need to be added to a "kernel+related" package repo.

Users could be encouraged to do more testing if they are provided with a simple mechanism to snapshot, update, test, and then either keep the changes or roll back. I do this in UML with the COW tools. Currently LVM2 has writable snapshots, but no easy way (at system boot) to reintegrate the changes into the base from a snapshot, or "fallback" to a snapshot. Still, using Xen/UML/QEMU, it is not difficult to take advantage of copy-on-write to update the kernel and other packages, then boot the image to start a shell or web/ftp/mail/... daemon(s). At least that would exercise the non-hardware-related, (and for now, non-SMP) parts of the kernel.

Bill Rugolsky

Tangent: I would like to see requests-for-testing for FC kernels on LKML.

If people announce -ac/-as/-aa/-ck/etc. kernels on LKML, why not distro kernels?

Jeff

Debian unstable currently contains only for kernel 2.6.8 (which is AFAIK still the main kernel in Debian unstable although there are also 2.6.10 sources and 2.6.10 kernel images on some architectures) for eight different architectures - many of them containing or depending on their own patches.

How big is the value of a

New 2.6.8-5 s390 kernel images for Debian unstable - please test!

or a

New 2.2.25-4 atari kernel images for Debian unstable - please test!

on LKML?

Note that there are currently 42 (what a number ;-) ) different packages that build 2.2, 2.4 and 2.6 kernels images in Debian unstable plus the packages that build the kernel images for the 11 architectures in Debian stable.
Considering how many other distributions exist, this generates a serious amount of extra traffic on LKML if a security hole gets fixed that affects many kernel versions...

cu
Adrian

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

There's also no other (suitable) place to announce kernel trees. Debian kernels get announced on various debian-related lists; I'd imagine FC kernels have the same thing. The only place to announce non-distro trees is lkml (and I've had requests for an -as specific announce list, I haven't haven't found the time to get something going).

Andres,

considering that your kernels' purpose is to provide a stabler base than mainline to people who still want to stay close to mainline, I think that your announces bring some real value here.

What does not have its place here are the announces of full-featured patchsets not targetted at developpers nor testers.

Cheers,
Willy

Please keep announcing them here - its useful to see all the trees in one place. I certainly look at the others to make sure I don't miss stuff. We might not agree on what should be merged but the cross checking is valuable.

--- Jeff Garzik <[email protected]> a écrit :

Because some people switched to other distribution also because of the distribution kernel ;-).

When i was using redhat (6.x) the only compilable kernel was from kernel.org. I couldn't get the kernel shipped by
redhat to compile on my machine.

--
A mouse is a device used to point at
the xterm you want to type in.
Kim Alm on a.s.r.

Découvrez le nouveau Yahoo! Mail : 250 Mo d'espace de stockage pour vos mails ! Créez votre Yahoo! Mail sur http://fr.mail.yahoo.com/

Nah, I agree with DaveJ -- there are definitely "dev" portions when it comes to driver updates.

Judging from recent posting from Bart, it looks like he has an evil plot to merge the IDE driver with libata. libata will also eventually [perhaps with Bart's changes?] make the SCSI portion optional, as I have long promised. And it's getting other new and destabilizing features.

There will be other changes in SCSI and block too, which want staging...
Some of the stuff I've been putting off until "2.7" will be re-thought
into something that appears in the on-going 2.6 series.

If you don't have driver stability, you don't have a useful kernel...

Jeff

Then the notion of it being stable is bogus, given how many regressions the last few kernels have brought in drivers. Moving from 2.6.9 -> 2.6.10 broke ALSA, USB, parport, firewire, and countless other little bits and pieces that users tend to notice.

The reason that things like 4-level page tables worked out so well was that it was tested in -mm for however long, so by the time it got to your tree, the silly bugs had already been shaken out.

Compare this to random-driver-update. -mm testing is a valuable proving ground, but its no panacea to stability. There's no guarantee that someone with $affected_device even tried a -mm kernel.

For it to truly be a stable kernel, the only patches I'd expect to drivers would be ones fixing blindingly obvious bugs. No cleanups. No new functionality. I'd even question new hardware support if it wasn't just a PCI ID addition.

Dave

Maybe I don't understand? Is someone expecting distro quality/stability from kernel.org kernels?
I don't, but maybe I'm one of those minorities.

~Randy

My complaint is the charade of calling it 'stable' when it clearly wouldn't be anything of the sort, given that a majority of the bugs our users experienced on rebasing were driver related.
The core itself may be rock-solid, but if we're continually pulling in random driver updates of questionable quality with limited testing, the result as a whole isn't stable.

Putting the onus on distributions to make things stable is no excuse for the ever-increasing number of regressions each release.

This might sound over-dramatic, but it's the current state as far as I'm concerned. The 2.6.8->2.6.9 update for Fedora users brought a bunch of carnage that took time to shake out. 2.6.9->2.6.10 I'm still picking up the pieces of. If the 2.6.10->2.6.11 update that I'll do for Fedora in a week or two turns out to have less regressions than the previous releases, I'll be stunned. Really.

Already I'm wondering how many userspace packages are going to randomly stop working as they have done in previous releases. With the clear delineation of stable/development, we were able to say things like "we won't change a user visible interface in a stable series" Now, we don't have that. So we find things ranging from slabtop to alsa-lib completely break unless you update the userspace too.

regressions like this is what I'm bitching about. There's nothing a vendor can do to make such things stable (other than dropping the various patches that introduce the breakage, but at ~4000 csets per release right now, there will be stuff that gets missed). Whilst the slabinfo example was a non-driver related regression, it's a good example of how little care we're taking these days to make sure existing userspace continues to work correctly.

Some may suggest the close tracking of mainline is the problem. Maybe they're right. Maybe we should have stuck with a 2.6.5 kernel until Fedora Core 2 reached end of life, and gone with the old 'have hundreds and hundreds of patches piling up' approach.

But, as someone who has maintained vendor kernels that have tried both methods, the sticking close to mainline approach wins hands down. If something is broken, more often than not, I can bug the upstream developer and ask "hey, this is a wierd problem our fedora users hit, we don't have any patches against this code, can you take a look?" and developers have been very responsive, and helpful on many occasions, ultimatly leading bugs being fixed both in our kernel, and upstream.

If I asked most upstream developers about a problem we've been facing with our 2.6.5 kernels, I'd get a much less helpful response. And rightly so. In their position I'd do exactly the same thing.

Dave

Sure, I'm not trying to put the onus on distros, just saying that they add some real value there, but that doesn't excuse us from trying to make the mainline kernel as good as we can make it.

~Randy

There are few distributors who can sufficiently QA the kernel they ship. I think only Redhat/Fedora, Novell/Suse and Mandrake/Conectiva currently have *good* testing procedures and good QA. Most other distributions basically ship the vanilla Linus kernel (or Alan kernel) with some basic patches applied.

Besides that, a *lot* of admins still prefer to compile their own kernel. I often encounter admins who still don't know about the 2.6 development scheme and blindly compile each shiny new 2.6 kernel released by Linus.

So basically what I'm trying to tell is: it could be time to start the 2.7 series to have some room for experimentation. That is what Linus enjoys doing (*) and that is what Linus is good at. Leave the 2.6 kernel to Alan or someone else.

However, I think a 2.6.x-mm kernel is still a good idea after 2.7 branches. The -mm kernel could be a collection of backports from the 2.7 kernel, waiting to be included in the 2.6 kernel.

The 3.x.y kernel could be the place for very wild experimentation. I'd love to see a kernel which supports a object-relational non-POSIX compatible filesystem, ready for 2010's storage requirements. But that's just me.

(*) I've got a magic Linus brain reader device.

Erik Hensema <[email protected]>

    Hi,

How do you expect a broad user base testing your kernels if "stable" kernel.org kernels aren't to be expected stable?

Just my 2p,

Jochen.

Please place all complaints in this box --> []

Don't say that too loudly or else the embedded people will use it as another excuse not to move on to 2.6 kernels.

But yes, _I_ do expect a certain amount of quality and stability from kernel.org kernels mainly because there _isn't_ a distro to do that for us on ARM. It's fine in x86 land where there are such things and you have One Single Platform(tm), but in ARM land where there's numerious platforms, there's very little sense in having a "distro" maintained kernel.

For instance, I'd like to upgrade the kernel on the main ARM server here from 2.4 to 2.6, but since it requires a number of subsystems to work properly (MD, Promise IDE, Ext3, NFS server) which you normally wouldn't see on ARM embedded systems, I'm not entirely happy to throw a 2.6 kernel on it just yet.

However, coupled with the requirement for quality, some in the ARM community expect a certain amount of development to be going on, since ARM is a relatively fast moving community [*]. Freezing stuff for 6 months isn't really practical when you have support for new platforms or new drivers wanting to be merged all the time. Ignoring these things for 6 months causes someone to spawn a new tree and that's usually the last you see of any contribution from that direction.

So, what I think we need is for core system, subsystem and "service" development to slow down a little while still allowing peripheral low-impact "new stuff" to be added during a "stablisation" period. We also need to qualify what we mean by "stable in relation to the resulting kernel - eg, I don't think we should call the "new stuff" merged during that time "stable".

This actually brings up an interesting point - if new machine support is stablised external to the "stable" tree, and then merged into that tree, should it be marked as "experimental"? Given the previous paragraph, I think the answer is "yes" since this will allow people to distinguish the recently merged stuff.

[*] - depends how you understand "fast". My method of working new features into the ARM code in Linus' kernel is via a series of small steps (or a series of smaller series of steps) spread out over a period of time.

An example of this is the ARM SMP merging, where the low level entry code has had to be rewritten. However, rather than just replace the file, it's getting a series of clean ups first which are aimed in the direction we want to go. We're getting towards the state where the _major_ difference between todays implementation and the SMP version is the code in a small number of macros, rather than virtually every line in the file. However, I do feel like I'm going against the grain when Linus says "slow down" and I'm seen to be doing cleanup after cleanup with no obvious effect or point.

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

I certainly do, and I think many others do as well.

My assumptions/expectations on vanilla kernels has changed a bit over time, and I don't expect as much from them these days as I once did.

I used to expect/assume (and this was back when I had never touched a single line of kernel code nor read a single post on lkml), that 1) the vanilla kernel was the ones with all the latest bugfixes and features 2) the latest stable vanilla kernel had gone through extensive testing (manual and automated) to make sure there were no regressions compared to the previous release 3) I could always go from <stable_series>.X to <stable_series>.X+1 without any features having disappeared 4) that a new vanilla kernel would never be released if there were still known bugs in it (even if those bugs were not regressions).

I've become wiser since then, and I'm more aware of the level of testing actually done, so I expect to be hit by bugs/regressions in vanilla kernels once in a while. But I still expect a very high degree of stability/quality from stable series vanilla kernels in general..

I run vanilla kernels on all my boxes, workstations and servers, since I don't really trust vendor kernels. I don't want to start relying on features only available in a single vendors patched kernel so that I will have trouble if I switch vendor (or want to get some new feature that's in vanilla but has not entered the vendors kernel yet). I've also had bad experiences in the past with patched vendor kernels being more unstable on my boxes than the vanilla kernels.. and then there's the testing issue, if everyone ran vendor kernels then who'd be testing vanilla?

Jesper Juhl

That's a strange statement, I don't think you are aware of the level of testing that goes into a vendor kernel, at least for the 'enterprise' products. There's just no comparison to the vanilla kernels.

Of course fixes get merged back into the vanilla kernels, but these just don't have the lengthy test and maturity period of vendor kernels (by a long stretch).

In fact the 2.6 cycle is specially geared towards vendor stabilization. At some point this will slow down of course, but I don't expect that to happen in the near future.

Jens Axboe

I can't claim to have any detailed knowledge of that, no.

That's true. I guess my lack of trust in vendor kernels is part being bitten by them in the past where my own custom build vanilla kernels have worked fine, and part the fear of getting locked-in to some vendor specific feature... Perhaps things are different these days and I should reevaluate my view on vendor kernels - but that's why I haven't been trusting them.

Jesper

Traditionally the vendor kernels from the big two vendors have been full of crap, for one of them it seems to be slowly changing now..

--- "Randy.Dunlap" <[email protected]> a écrit :

yes. Some people (like me) would like to use from time to time
some _new_ stable kernel. It's annoying to see that sometimes
kernel people say that you have a hardware problem in 2.6.0 release and then the bug is silently fixed in a 2.6.5 release for example.

The 2.6 kernel is not good enough for desktop -- sound skips on a 700Mhz duron when there is a lot (50M of 256) of free memory and mozilla is using CPU. 2.4 with preemptible patches was better in that respect.

What i would like to really have is a stable version not a demo version. I like testing but sometimes i also want stability and it seems that a good enough 2.6 kernel is still away. I don't care how you call it 2.6.x.y.z-rc-kappa i just want to be sure that it will not mess up my system.
(and no, i'm not going to use <insert your favourite distro> kernel )

Just my 2 euro cents (if somebody cares)

P.S. I know this message is late. Sorry if it annoys you.

--
A mouse is a device used to point at
the xterm you want to type in.
Kim Alm on a.s.r.

Découvrez le nouveau Yahoo! Mail : 250 Mo d'espace de stockage pour vos mails ! Créez votre Yahoo! Mail sur http://fr.mail.yahoo.com/

Grump. Have all these regressions received the appropriate level of visibility on this mailing list?

I too am a little put off by the number of regressions which certain (admittedly tricky) subsystems keep on introducing. One does wonder how careful people are being at the development stage. And that's a thing which we can surely fix.

For example, here's today's crop (so far):

include/linux/soundcard.h:195: warning: _PATCHKEY' redefined include/linux/awe_voice.h:33: warning: this is the location of the previous definition drivers/i2c/chips/ds1337.c:60: I2C_DRIVERID_DS1337' undeclared here (not in a function) drivers/i2c/chips/ds1337.c:60: initializer element is not constant drivers/i2c/chips/ds1337.c:60: (near initialization for ds1337_driver.id') drivers/i2c/chips/ds1337.c: In function ds1337_get_datetime':
drivers/i2c/chips/ds1337.c:155: structure has no member named id' drivers/i2c/chips/ds1337.c: In function ds1337_set_datetime':
drivers/i2c/chips/ds1337.c:206: structure has no member named id' drivers/i2c/chips/ds1337.c: In function ds1337_detect':
drivers/i2c/chips/ds1337.c:333: structure has no member named id' drivers/i2c/chips/ds1337.c:343: structure has no member named id'
make[3]: *** [drivers/i2c/chips/ds1337.o] Error 1
make[2]: *** [drivers/i2c/chips] Error 2
make[1]: *** [drivers/i2c] Error 2
make[1]: *** Waiting for unfinished jobs....
sound/pci/als4000.c: In function `snd_als4000_create_gameport':
sound/pci/als4000.c:572: warning: r' might be used uninitialized in this function drivers/char/watchdog/alim1535_wdt.c:319: warning: ali_pci_tbl' defined but not used drivers/char/sx.c:255: warning: sx_pci_tbl' defined but not used drivers/char/applicom.c:68: warning: applicom_pci_tbl' defined but not used
make: *** [drivers] Error 2
make: *** Waiting for unfinished jobs....
sound/pci/trident/trident_main.c: In function `snd_trident_gameport_trigger':
sound/pci/trident/trident_main.c:3125: warning: `return' with a value, in function returning void

Agree.

For the most part these things are usually known about by their upstream authors. To give an example: ALSA update in 2.6.10 broke sound for a bunch of IBM thinkpads. As it turns out there are quite a lot of these out there, so when I released a 2.6.10 update for Fedora, bugzilla lit up like a christmas tree with "Hey, where'd my sound go?" reports. (It was further confused by a load of other sound card problems, but
thats another issue). This got so out of control, Alan asked the ALSA
folks to take a look, and iirc Takashi figured out what had caused the problem, and it got fixed in the ALSA folks tree, and subsequently, in 2.6.11rc.

Now, during all this time, there hadn't been any reports of this problem at all on Linux-kernel. Not even from Linus' tree, let alone -mm. Which amazes me given how widespread the problem was.

compile time regressions we should be able to nail down fairly easily. (someone from OSDL is already doing compile stats and such on each release
[too bad they're mostly incomprehensible to the casual viewer])
The bigger problem is runtime testing. If things aren't getting the exposure they need, we're going to get screwed over by something or other every time Linus bk pull's some random driver repo.

Dave

I'll ding the OSDL release builder about that (I agree with you)....

~Randy

There is not much the ALSA developers can do about this, because we just do not have access to the 1000s of hardware varieties out there, and vendors are inexplicably loath to give out any docs on something as simple as a cheap shitty laptop AC97 codec.

The usual pattern that leads to a regression like the Thinkpad issue is that a user reports a bug, a fix is merged, the user reports the problem is fixed on $FOO hardware, then a week later we are deluged with reports that $BAR hardware is now broken.

Catching a problem like this before it gets out the door would require access to 1000s of laptops, unlimited time, and reasonable hardware documentation. Chances are we would still miss something. IOW it's impossible.

The *only* solution for these kinds of driver regressions is to get more people to test the release candidates. "Being more careful" will not help, because we just don't have access to all of the hardware.

Lee

Part of that problem may be that reporting bugs to the kernel is hard for everybody except kernel hackers. bugzilla.kernel.org is there but not many people look at it (which I understand, using bugzilla is painfull, altough basing all your development strategy around it _is_ rewarding, as happens in gnome/etc, where the release announcement includes a list bugzilla numbers which point to fixed bugs or "new feature" bugs in the case of new features).

That could be very well at last as important as the "release numbering". If there're bugs out there and people are not reporting them, the "release numbering" you choose is irrelevant since bugs are not being reported and you can't fix them. Just my humble opinion.

As one of those who initially thought "great" about bugzilla for kernel stuff, and then got rather annoyed with it, I think I can talk about why bugzilla doesn't work for kernel developers.

* The most obvious problem is that it requires you to go to the website before you can do anything with a bug.
* Bug reporters appear to report a bug and run away - attempting to get more information from them sometimes resorts in silence until you threaten to close the bug, or do close the bug.
* Bug categories aren't explained well enough to allow users to determine the correct category. Eg, PCMCIA network card - should that be networking or PCMCIA (where the bugzilla PCMCIA category is actually *only* the PCMCIA core and not any of the drivers.)

Overall, my experience with the kernel bugzilla has been rather unproductive. Most bugs which came in my direction weren't for things I could resolve.

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

It's possible that there're other bug tracking systems that can fit? The main disadvantage of the (few) bug tracking systems I've seen is that at the end they end up being a "additional task" for the developers (ie: more work) instead of being something that really helps to developers and free them of doing some extra work, so people avoids them when they can. There're some mail-based bug tracking systems like the one from debian but they don't include a alternative web interface for final users (well, I think bugzilla has a "email mode" or something like that but I don't really know)

Bug reporters running away is something that will happen always no matter what system is used, but is it important? If someone submits a bug and he don't answer when he's asked for more details, there's no way you can fix it and the bug should be closed given a reasonable time frame. There's no reason why this can't happen today when bug are reported to the lkml, except that people who reports bugs to the lkml is people who really wants to get their bugs fixed. If the bug is important enought, it'll get reported again by someone else who cares about getting it fixed.

(There're other problems with traditional bug tracking systems, like "ownership" of a bug. Some bugs are reported, discussed, and then they get fixed silently outside of the scope of the bug tracking system, but the bug remains open for _years_ - IMHO all open bugs which have not been discussed and closed in such amount of time should be closed unless there're real reasons to keep them open, new versions could have fixed it and if it remains it could be re-submitted - and it can't get closed because the developer who owns it is not looking at it. Some bug tracking systems (like the one from joel spolsky, aka "joelonsoftware.com") just don't allow to have "ownership" of bugs, because if you want that the system helps to solve problems instead of being an additional task, _everyone_ should be allowed to solve problems - which is specially true in the open source world)

On Thu, 2005-03-03 at 00:21 -0500, Dave Jones wrote:
<snip>

Dave, I'm the "someone from OSDL". I agree that the compile stats and error/warning regresssions can be a little challenging to grock for the casual observer. Is it content or formatting that would help the casual viewer?

John

I don't have one handy to quote from, but personally I find there are two problems with what's currently presented.

1 - information overload. It's just a screenfull of numbers. 2 - the column headings were non-obvious iirc. Or maybe I'm just dumb
and didn't look at them long enough.

So it could just need some formatting tweaks to make it more palatable.

Dave

This is not too difficult to do. I'll tweak the format a bit and see if it is more consumable.

John

I reported once a bug on alsa-devel and cc-ed on lkml The sequencer isn't working with my card cs4239 with alsa.

Taking into account that nobody responded on lkml nor on alsa (the message was awaiting modderator aprouval on alsa-devel) i don't think i will send more bug reports to alsa.

Maybe more people are testing the kernel but they don't have time to report the bugs (for me it takes about 15 minutes to do it properly). And some bugs like "i had a crash last night" are irrelevant if i don't have some logs traces.

And _you_ (most of you) _want_ _people_ _to_ _test_ _2.6_ _kernels_ *and in the mean time* _you_ _recommend_ _using_ _vendor_ _kernels_.

So if i want also a stable kernel i should use two trees (a vendor tree and a "2.6..whatever" tree) to compile the kernel ? How about mm-tree ?
The whole 2.6. tree is strange. On one side you have -mm where all the "experimental" stuff goes and on the other side there is Linus's tree (2.6.x) where also some "experimental" stuff goes. (from a luser/tester point of view)

Also some people need time to test. I have 4 hours free every day.

--
A mouse is a device used to point at
the xterm you want to type in.
Kim Alm on a.s.r.

Yahoo! Mail - Votre e-mail personnel et gratuit qui vous suit partout !
Créez votre Yahoo! Mail sur http://mail.yahoo.fr

I cannot find your report (checked back to the start of the year).

Please send a new one. I'm collection them.

What exactly do you mean by "it isn't working"?

90% of "MIDI does not work" bug reports are from users who expect playing MIDI files to work OOTB like it does on Mac and Windows.

This only works because those OS'es come bundled with a toy softsynth. With ALSA, you either need a supported hardware wavetable synth (emu10k1) or a real soft synth like Timidity or Fluidsynth.

Anyway, please repost your bug report to alsa-devel. There is no point in cc'ing LKML for ALSA problems, unless you find a problem like a regression in ALSA functionality from one kernel release to another.

Lee

CS4239 has a toy synth of sorts (its more "doorbell" than synth admittedly). There are a pile of funnies with IBM laptops and CS423x to watch out for that might be worth mentioning - in particular you need to turn the fast boot stuff -off-.

How long ago was this? alsa-devel has been accepting messages from non-subscribers for at least 6 months.

Lee

Looking at the http://l4x.org/k/ site, it appears that all -mm versions have broken ARM support with the defconfig, while Linus kernels at least build fine. I've no idea why though.

I haven't had any reports of breakage, and I don't think I've missed any heads up on something which needs to be fixed up, except maybe the 4 level page table stuff. (TBH I don't think many ARM people look at -mm, but it is worrying that something which may be merged into Linus' tree from -mm may have the unexpected consequence of breaking something which used to work.)

Unfortunately, http://l4x.org/k/ doesn't save any build logs for investigation.

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

If you click the 'Fail' then it seems to keep the make output etc.

Ian.

Ian Campbell, Senior Design Engineer
                                        Web: http://www.arcom.com
Arcom, Clifton Road,                    Direct: +44 (0)1223 403 465
Cambridge CB1 7EA, United Kingdom       Phone:  +44 (0)1223 411 200

elinks doesn't show any of the "fail" in the matrix as links - this seems to be using javascript.

In fact there doesn't appear to be a reason to use javascript for this - it seems to be implementing a standard link to:

	http://l4x.org/k/?d=<number>

which can be done just as well with standard HTML without need to resort to javascript. Plus it works with a wider range of browsers. Including those used by certain blind developers in this community who are completely unable to use any form of GUI.

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

The links and row headers high-light when you mouse over each cell, which appears to be why there is javascript involved at all.

I don't know if javascript is necessary for that feature, but I agree that using it for the links seems wrong.

Ian.

Ian Campbell, Senior Design Engineer
                                        Web: http://www.arcom.com
Arcom, Clifton Road,                    Direct: +44 (0)1223 403 465
Cambridge CB1 7EA, United Kingdom       Phone:  +44 (0)1223 411 200

No it isn't necessary; the CSS attribute :hover is the correct way to do this. Using Javascript for any kind of design is just wrong, wrong, wrong.

Regards: David Weinehall

 /) David Weinehall <[email protected]> /) Northern lights wander      (\
//  Maintainer of the v2.0 kernel   //  Dance across the winter sky //
\)  http://www.acc.umu.se/~tao/    (/   Full colour fire           (/

It's very much in an arch maintainer's interest to make sure that cross-compilers are easily obtainable. Any hints?

Been trying to achieve that since it's a FAQ on ARM lists. Even gone to the extent of setting up a separate mailing list, getting a volunteer to track what people want and do the hard work to build them. That was about 6 months ago, and I haven't seen any results.

I was thinking at the time "great, this is one problem which should be solved". How silly of me. It seems, yet again, that it comes down to a case of "if rmk doesn't do it, it won't get done." Sad but true. Now, why do I keep feeling that I'm being taken advantage of all the time? Could it be the complete lack of productive help from anyone else.

Anyway, going back to why -mm doesn't work:

arch/arm/kernel/built-in.o(.init.text+0xb64): In function `$a':
: undefined reference to `rd_size'
make[1]: *** [.tmp_vmlinux1] Error 1

So "rd_size" got deleted in -mm kernels without reference to anyone else who's using it. Greeeeaaatttt....

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

hm. That's strange. I'd have thought that 99% of the arm embedded developers cross-build.

Ah. Fixed, thanks.

Yes - I think Dan Kegel's cross-tool gets used a fair bit...

Actually, this highlights another problem - it's ARM binutils again. Jan's cross-binutils for ARM doesn't contain the patches to make the linker resolve addresses to the _correct_ symbol. Don't ask me where they are, but I'm lead to believe that cross-tool knows.

This also means that Jan's compile test is rather worthless for ARM - it might be linking a kernel with undefined symbols due to the other assembler bug.

Thanks.

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Russell King:

I've found the arm cross compiler generated from openembedded (http://openembedded.org) to be very reliable. The big advantage in using oe would be that it is in active use so it is always highly likely to generate a working compiler. Someone just needs to make it generate a toolchain/compiler for external use[1], make it available somewhere and advertise the fact its available. Generation of the toolchain could probably be almost entirely automated.

Fixes for any problems with compiler would be more than welcome for incorporation into oe short term and for submission upstream for "proper" fixing.

[1] I think I've seen reference that it can already do this...

Richard

I'll only believe it when I see it. The problem is this "someone"... Who's going to forfill that space and produce some results?

Something tells me that we'll be very lucky to get a volunteer, let alone see any results.

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Russell King:

As an experiment I ran "bitbake meta-sdk" on my copy of openemedded. A while later I have these in the deploy directory amongst other things.

http://www.rpsys.net/openzaurus/arm-cross/binutils-cross-sdk-2.15.91.0.2-r5.tar.gz (3.8MB)
http://www.rpsys.net/openzaurus/arm-cross/gcc-cross-sdk-3.4.2-r0.tar.gz (17.5MB)

"bitbake binutls-cross-sdk" and "bitbake gcc-cross-sdk" are more efficient as they won't build all the other things in the sdk.

Generating a cross compiler from oe really is as simple as that. Setting up oe in the first place is a bit more tricky but too bad.

An improvement to the current situation might be some instructions and links to oe to tell people they can build a toolchain this way. Better still someone could run the above commands now and again and upload the files somewhere. I don't have anywhere I can host files that size on a permanent basis. I possibly could be persuaded to run the commands now and again if people were going to use the results. I'm in two minds though as generating your own from openembedded isn't difficult. Writing instructions for setting up oe to build it may be the best option.

Richard

Two things - are you sure that openembedded contains the patches to fix the two biggest binutils issues we have, as documented on
http://www.arm.linux.org.uk/developer/toolchain/ ?

Secondly, are you seriously suggesting people like Jan Dittmer, who provide a cross-architecture service should jump through some loops just to get a working toolchain for the ARM architecture?

Jan, how do you feel about this?

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

As long it is documented and it _works_ that's no problem. But it was quite a hassle to get working cross-compilers for all 23 archs to build, because for some there is no real documentation which target is the correct one and upstream gcc and/or binutils sometimes don't compile.
For example where can I find information about the arm26 arch? I suppose it can be build with a normal arm toolchain (and the breakage looks like a real compiler failure), but is this documented somewhere?
It would be nice to have such documentation in kernel under Documentation/$arch/HOW-TO-BUILD for every arch.
How much work is it to set-up an openembedded environment anyways?

Jan

http://l4x.org/k/

Russell King:

I've checked and it contains the tc-arm.c.patch but does not have the ARM mapping symbols fix. As recent kernels have fixes for that, its not so much of a problem as it was however it should be staightforward to add into oe and I will aim to do that.

You said nobody was willing/interested in maintaining a toolchain. I'm saying that a toolchain is maintained within openembedded and that pointing people at that is better than nothing. Maintaining a set of patches to ensure bugs in binutils etc are fixed is easy within oe's framework. (To add the above patch, in theory I just need to add a line to a file).

Jan Dittmer:

This is why openembedded exists - it tracks known working build configurations of every bit of software needed to make complete linux distributions.

I've written a quick and dirty set of instructions on setting up openembedded below. Please note that oe is used for building complete linux distributions so there is a lot of functionality being unused here (as you can imagine from my statement above). bitbake is a known memory hog (it can use up to 1GB of ram) - we all know this is appalling and a rewrite to address this is in progress.

cd /work/
svn co svn://svn.berlios.de/bitbake/trunk/bitbake
export PATH=/work/bitbake/bin:$PATH

bk clone bk://openembedded.bkbits.net/openembeded
export BBPATH=/work/build:/work/openembedded

mkdir build
mkdir build/conf/
cp openembedded/conf/local.conf.sample build/conf/local.conf

Edit local.conf as appropriate. local.conf.sample has details about what the variables do. My local.conf to generate the arm toolchain has:

DL_DIR="/work/sources"
BBFILES="/work/openembedded/packages/*/*.bb"
TARGET_ARCH="arm"
TARGET_OS="linux"
INHERIT="package_tar"

(everything else is left unchanged)

cd build
bitbake binutils-cross-sdk
bitbake gcc-cross-sdk

Output ends up in /work/build/tmp/deploy

Richard

As it happens I was editing that exact page in the wiki t'other day:
http://openembedded.org/cgi-bin/moin.cgi/GettingStarted

I actually only wanted a toolchain but oe and scratchbox[1] seemed the rational alternatives. Scratchbox seems to offer : arm-gcc-3.3.4-glibc-2.3.2 but I've not gotten round to using it yet.

Thanks for the comment on "bitbake binutils-cross-sdk" and "bitbake gcc-cross-sdk". I'll add more notes to the page once I figure it all out.

David

[1] http://www.scratchbox.org/download/scratchbox-1.0/

Bless you. I just built an arm kernel!

That compiler is *fast*. 47 seconds. Weird.

For reference, untar the above in / and use

#!/bin/sh
export ARCH=arm
export CROSS_COMPILE=arm-linux-
W=/usr/local/arm/oe/bin

MAKE="make"

if [ -z "$1" ]
then
WHAT="vmlinux"
else

	WHAT="$1"

fi

nr_cpus=$(grep processor /proc/cpuinfo|wc -l)
j=$(expr $nr_cpus \* 3 / 2)

MAKE_ARGS="ARCH=$ARCH CROSS_COMPILE=$W/arm-linux-"

if [ x"$DISTCC_HOSTS" != x ]
then

	$MAKE -j 12 CC="ccache distcc --ccache-skip $W/$CROSS_COMPILE""gcc" $MAKE_ARGS $WHAT 2>/tmp/log
else
	$MAKE -j $j $MAKE_ARGS CC="$W/$CROSS_COMPILE""gcc" $WHAT 2>/tmp/log

fi
cat /tmp/log

That's now eight architectures I'll compile-test mm kernels on.

Cool, but please check whether this produces an error:

echo "mov r0, #foo" | arm-linux-as -o /dev/null -

you should get:
{standard input}: Assembler messages:
{standard input}:1: Error: undefined symbol foo used as an immediate value

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

I did get that.

Great - this will help ensure that any breakage due to that binutils problem should get caught relatively quickly no matter how it gets in to either your or Linus' kernel. This is a definite plus. Thanks.

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Sorry, this was my fault (I made it static and oversaw the ARM usage when grep'ing for it).

cu
Adrian

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

And guess what? This patch is now in Linus kernel. Greeeeaaaattttt stuff whoever did that. Truely wonderful job.

Thanks for taking notice of me a month ago. I really appreciate it. Especially when stuff gets merged which has already been pointed out as needlessly _BREAKING_ stuff.

Linus - please revert the change which caused this.

  CHK     include/linux/compile.h
  UPD     include/linux/compile.h
  CC      init/version.o
  LD      init/built-in.o
  LD      .tmp_vmlinux1

arch/arm/kernel/built-in.o(.init.text+0x9d4): In function `$a':
: undefined reference to `rd_size'
make: *** [.tmp_vmlinux1] Error 1

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

megamolehill, Russell.

I immediately dropped that patch when the problem was pointed out.

But early this month Adrian resent the same patch and I applied it again and it slipped through.

Stuff happens. That's why we have -rc kernels.

I already have the fix for this queued up.

Sorry for tapping in so late. I don't follow lkml that close. For some time I thought about providing semi-automatic mails of what architectures broke/ got fixed from one version to another, tracking mm/rc/git(?). Would that be useful?

Jan

We have http://armlinux.simtec.co.uk/kautobuild/ setup for ARM - it would be nice if it could generate mail reports and send them out, especially when failures occur.

I think it would need some post-processing and rate limiting to determine if there is a common problem across multiple platforms (eg, 40 builds failing due to rd_size), or just one platform (some local breakage).

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Well I built something like this now which I mail to myself, overlook and then going sent to lkml:

Comparing 2.6.12-rc3-mm2 to 2.6.12-rc3-mm3 (defconfig)

- arm: broke

    AR      arch/arm/lib/lib.a
    GEN     .version
    CHK     include/linux/compile.h
    UPD     include/linux/compile.h
    CC      init/version.o
    LD      init/built-in.o
    LD      .tmp_vmlinux1

arch/arm/kernel/built-in.o(.init.text+0xb64): In function `$a':
: undefined reference to `rd_size'
make[1]: *** [.tmp_vmlinux1] Error 1
make: *** [_all] Error 2
Details: http://l4x.org/k/?d=3476

- m32r: broke
: relocation truncated to fit: R_M32R_26_PCREL_RELA pcibus_to_node drivers/built-in.o(.text+0x46258): In function `init_irq':
: undefined reference to pcibus_to_node' drivers/built-in.o(.text+0x46258): In function init_irq':
: relocation truncated to fit: R_M32R_26_PCREL_RELA pcibus_to_node drivers/built-in.o(.text+0x4a15c): In function `idedisk_attach':
: undefined reference to pcibus_to_node' drivers/built-in.o(.text+0x4a15c): In function idedisk_attach':
: relocation truncated to fit: R_M32R_26_PCREL_RELA pcibus_to_node make[1]: *** [vmlinux] Error 1
make: *** [_all] Error 2
Details: http://l4x.org/k/?d=3483

- ppc: fixed

- um: fixed

- arm26: still broken
Details: http://l4x.org/k/?d=3477

- cris: still broken
Details: http://l4x.org/k/?d=3478

- frv: still broken
Details: http://l4x.org/k/?d=3479

- h8300: still broken
Details: http://l4x.org/k/?d=3480

- ia64: still broken
Details: http://l4x.org/k/?d=3482

- m68k: still broken
Details: http://l4x.org/k/?d=3484

- m68knommu: still broken
Details: http://l4x.org/k/?d=3486

- parisc: still broken
Details: http://l4x.org/k/?d=3488

- s390: still broken
Details: http://l4x.org/k/?d=3491

- sh: still broken
Details: http://l4x.org/k/?d=3492

- sh64: still broken
Details: http://l4x.org/k/?d=3493

- v850: still broken
Details: http://l4x.org/k/?d=3497

Summary: 8 ok, 14 failed
Link to this page: http://l4x.org/k/?diff[v1]=mm

Jan

Seems to be valuable.

Known issue (was my fault).

I'll send separate emails regarding some other issues the output shows.

cu
Adrian

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

arm26, cris, sparc: select of non-existing I2C* symbols:

@ Ian, Mikael, William:
This could be fixed by sourcing drivers/i2c/Kconfig in arch/*/Kconfig, but it would be better to switch to use drivers/Kconfig.

@ Roman:
Shouldn't kconfig exit with an error if a not available symbol gets selected?

cu
Adrian

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

No. There are meny configurations where we select a symbol that is only visible in some configurations.

Several possibilities exists:
1) Silently ignore SELECT SYMBOL when SYMBOL is undefined
2) Warn - as we do today
3) Error out as you suggest

Option 1) is preferable for 'make oldconfig' simply because target group here do not care. But it would be nice to know when one do a typing error in SELECT. So one *config target should continue to warn about it.

Sam

I disagree.

Consider that some developers do not use anything other than "make oldconfig" so having this silently ignore stuff means that these folk never get the warnings. I myself fall into this category.

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

I'd prefer 3) and fix up all problems.

E.g. i the case of the I2C* select's the current warnings seem to indicate possible configurations that do not compile.

cu
Adrian

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

You're telling me I have to futz with the i2c Kconfig just to cope with it not existing?

-- wli

What happens if a user selects one of the options that do themself select one or more of the I2C* options?

This might be solved differently (e.g. via some kind of ARCH_SUPPORTS_I2C), but it should be solved.

cu
Adrian

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

I've got a patch to convert arch/sparc/Kconfig to sourcing drivers/Kconfig. I've checked that defconfig comes out largely the same (the difference seems to be order and comments) but haven't really checked thoroughly.

Maybe if others (e.g. sparc32 users) could look and complain if something goes wrong that would help.

vs. 2.6.12-rc5-mm2

-- wli

Index: mm2-2.6.12-rc5/arch/sparc/Kconfig
=================================--- mm2-2.6.12-rc5.orig/arch/sparc/Kconfig 2005-06-06 13:45:32.000000000 -0700
+++ mm2-2.6.12-rc5/arch/sparc/Kconfig 2005-06-06 16:08:36.859702935 -0700
@@ -264,7 +264,11 @@
want to run SunOS binaries on an Ultra you must also say Y to "Kernel support for 32-bit a.out binaries" above.

-source "drivers/parport/Kconfig"
+source "mm/Kconfig"
+
+endmenu
+
+source "drivers/Kconfig"

config PRINTER
tristate "Parallel printer support"
@@ -291,41 +295,10 @@
If you have more than 8 printers, you need to increase the LP_NO macro in lp.c and the PARPORT_MAX macro in parport.h.

-source "mm/Kconfig"
-
-endmenu
-
-source "drivers/base/Kconfig"
-
-source "drivers/video/Kconfig"
-
-source "drivers/mtd/Kconfig"
-
-source "drivers/serial/Kconfig"
-
if !SUN4
source "drivers/sbus/char/Kconfig"
endif

-source "drivers/block/Kconfig"
-
-# Don't frighten a common SBus user
-if PCI
-
-source "drivers/ide/Kconfig"
-
-endif
-
-source "drivers/isdn/Kconfig"
-
-source "drivers/scsi/Kconfig"
-
-source "drivers/fc4/Kconfig"
-
-source "drivers/md/Kconfig"
-
-source "net/Kconfig"
-
# This one must be before the filesystem configs. -DaveM

menu "Unix98 PTY support"
@@ -374,18 +347,8 @@

endmenu

-source "drivers/input/Kconfig"
-
source "fs/Kconfig"

-source "sound/Kconfig"
-
-source "drivers/usb/Kconfig"
-
-source "drivers/infiniband/Kconfig"
-
-source "drivers/char/watchdog/Kconfig"
-
source "arch/sparc/Kconfig.debug"

source "security/Kconfig"

@ David, Miles:
Could you add defconfig's?

They serve at least two purposes:
- give people a starting point for their kernel configuration
- allow simple cross-compile tests like the one Jan did

A defconfig isn't something special, it's simply one reasonable .config for this architecture.

cu
Adrian

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Trivial fix below.

cu
Adrian

<-- snip -->

This patch fixes a trivial typo in introduced by
make-each-arch-use-mm-kconfig.patch .

@@ -181,6 +181,6 @@
 	bool "Preemptible Kernel"
 	default n
 
-source "mm/Kconfig'
+source "mm/Kconfig"
 
 endmenu

In theory. In practice on some platforms we need special compiler patches which will never be accepted into gcc upstream or are restricted to particular versions of tools. Building crosstools is tricky and yet it seems every moron really has to toll it's own from his private mouldy collection of patches. The whole tools stuff has become very much a battlefield of it's own.

Ralf

/me kills my patchbomb script for now

Ok, that's acceptable to me, but realize that this puts a bigger burden on the maintainers to queue up patches for you. It's not that big of a deal, just something to be aware of.

Speaking of which, does this mean I shouldn't hit you with all of my pending stuff? I know some of the other subsystem maintainers have a lot of stuff queued up too. Should we start this new numbering scheme as of today? Or wait until 2.6.13?

thanks,

greg k-h

Hi Greg,

Not necessaruly, because the rules could be more relaxed during -rc stage in an odd release, and stable releases could be shorter, so at the end, the "patch fridge" would be needed only between the very end of the odd version and the start of the even one.

Another possibility is for developpers to start to submit/merge patches for the next odd release while the even one is still in -rc.

if there is so much pending stuff, why not sacrifice 2.6.12, use 2.6.13 to fix a bit and merge even more, then 2.6.14 would be the real first stable release ?

Regards,
Willy

It's still a lot of work to queue up pending patches. Over a certian limit, it gets pretty unstable, trust me. I'm not complaining, just letting you know that this will push more work on the maintainers, but hey, we scale, so it's not that big of a deal :)

No, that's not ok to do. We want people to focus on getting a -rc out.

thanks,

greg k-h

Random ideas. Why not use the already established 2.6.x.y method ? This allows development to continue on 2.6.x+1, and if needed, you should be able to bk pull the 2.6.x.y[n] tree(s) into 2.6.x+1 no ?

My concern about this 'stable kernel every other release' method is that if a particular development cycle drags on for some reason, and certain bugs never got fixed in the previous release, that's a long time users have to wait until they get things fixed.

It's somewhat akin to the problem with the infrequent out-of-tree merges that some subsystem maintainers have. If the latest push they do doesn't fix your bug, you typically have to wait until the next release until they do another push.

Dave

Hi.

My first response is: this is a recipe for great confusion among users.

I'd far rather see things only make it into your tree when they've been thoroughly tested (in -mm and prior to that). Following that strategy, your tree could always be relied upon to be stable and -rcs would only needed for dealing with the unforeseen interactions between otherwise mature patches.

Regards,

Nigel

Nigel Cunningham
Software Engineer, Canberra, Australia
http://www.cyclades.com
Bus: +61 (2) 6291 9554; Hme: +61 (2) 6292 8028;  Mob: +61 (417) 100 574

Maintainer of Suspend2 Kernel Patches http://softwaresuspend.berlios.de

Hi Linus,

For a long time, I've been hoping/asking for a more frequent stable/unstable cycle, so clearly you can count my vote on this one (eventhough it might count for close to zero). This is a very good step towards a better stability IMHO.

However, I have a comment :

I don't agree with you on this last one (not the fact that you don't want an mk+mp+vb combination :-)). The VERSION number (in the makefile meaning of the term) only gets updated every 10 years or so. So it does not need to jump. PATCHLEVEL increments are rare enough to justify lots of breaking. I certainly can imagine people laughing at your OS when you jump from v2.6.X to v4.0.X, it will have a smell of slowaris (remember 2.6 - 2.7 - 7 - 8 that confused everyone ?). On the other side, the openbsd numbering scheme is far simpler to understand, and I sincerely think that we should enter 3.0 after 2.8. As a side note, I've always wondered why we would not swap odds and evens, so that we can keep the same major number between devel and release (eg: devel 2.8, release 2.9 ; not devel 2.9, release 3.0). Anyway, people have got used to stay away from the '.0' releases of any product on the planet.

Regards
Willy

or

I hate to point out the obvious to someone of longer standing, but according to the current system, since we increment integrally, 2.10.0 would be the successor release to the 2.9.X dev series, unless you know something about a severe and profound internals shakeup scheduled for 2.9 that we don't. :)

OT, 3.0.0 is an even-numbered release, therefore stable. So what do you call the odd-numbered unstable series that produces it? ;)

Matt

Yes, I have too much free time.

3.-1.x

:)

Ryan Anderson
  sometimes Pug Majere

I like it, but what about important security fixes or mistakes like 2.6.8.1? In those situations, would you release that kernel under the next even number and skip the odd number? Or would you release the fix under an odd number and sorta throw off the "meaning"?

I'm not trying to be a pain, I'm really not.

josh

Why be so wishy washy?

You can't say all that good stuff and then end it with " in practice this release numbering is still nothing but a hint of the _intent_ of the developer" and expect this to work out for more than a couple of builds before things fall back to where they are today.

Keep the change sets for the even numbers bug fix and stability changes only.

While your at it schedule a cut off rc number by which any moderately risky bug and stability changes are allowed into the even build cycle, say RC2, after which only stability and bug fixes that are considered to be "safe" changes are accepted.

You could also consider using the RC2 rule for odd releases so that there is plenty of time to hash out the bugs from the bigger changes. Just reject big changes after RC2. The developers will adjust and plan for that after a few releases.

Thrashing your users with regressions is a very bad thing.

--mgross

IMO too confusing.

And it exacerbates an on-going issue: we are moving away from "release early, release often", as this proposal just pushes the list of pending stuff back even further.

Developers right now are sitting on big piles, and pushing that back even further means every odd release means you are creating a 2.4.x/2.5.x backport situation every two releases.

To take a radical position on the other side, I would prefer a weekly snapshot as the release, staging invasive things in -mm.

And I think -mm is not enough, even. We have to come up with new ways to manage this ever-increasing flow of data into our tree.

Jeff

2.6.even: bugfixes only
2.6.odd: bugfixes and features.

That doesn't even confuse me!

No, there is no backporting. If you have a bug, fix it in 2.6.12-pre. There is no need to maintain that bugfix in your 2.6.13-candidate tree.

It's still a linear progression of kernel trees. The only thing which changes is the types of patches which we put into even releases.

You are missing where the backporting is.

If the time between big merges increases, as with this proposal, then the distance between local dev trees and linux-2.6 increases.

With that distance, breakages like the 64-bit resource struct stuff become more painful.

I like my own "ongoing dev tree, ongoing stable tree" proposal a lot better. But then, I'm biased :)

Jeff

The problem is people don't test until 2.6.whatever-final goes out. Nothing will change that.

And the day Linus releases we always get a pile of "missing MODULE_EXPORT()" type bug reports that are one liner fixes. Those fixes will not be seen by users until the next 2.6.x rev comes out and right now that takes months which is rediculious for such simple fixes.

We're talking about a one week "calming" period to collect the brown paper bag fixes for a 2.6.${even} release, that's all.

All this "I have to hold onto my backlog longer, WAHHH!" arguments are bogus IMHO. We're using a week of quiescence to fix the tree for users so they are happy whilst we work on the 2.6.${odd} interesting stuff :-)

So what was broken with the 2.6.8.1 type of 'hotfix kernel' release ? Apart from needing to do the 2.6.8.1 -> 2.6.8 -> 2.6.9 transition which confused some people, it seemed to be easily understood for what it was, and solved the same problem as this new proposal.

Dave

That's an alternative, of course.

But that _is_ a branch, and does need active forward- and (mainly) backward-porting work.

There's nothing wrong with it per-se, but it becomes a "stabilised version of the kernel.org tree" or even a "production version of the kernel.org tree". In other words it's somewhere on the line between the mainline kernel.org tree and a distribution. How far along that line should it be positioned?

In an ideal world, we'd see a single 'y' release of 2.6.x.y, but if x+1 takes too long to be released, bits of x+1 should also appear in x.y+1 The only question in my mind is 'how critical does a bug have to be to justify a .y release. Once a new 'x' release comes out, the previous x.y would likely no longer get any further fixes (unless someone decides the new 'x' sucked so bad).

Linus using bitkeeper should actually remove most of the pain from this multiple branch thing though. If fixes alternatively get checked into x.y and new development goes into x+1, x+1 could do a daily pull of x.y Thus saving the having to check in fixes to two seperate branches. (which really really sucks under some SCMs).

Dave

Ha - a nice big thread. Issues include trivial fixes, testing, and API stability.

-----
About trivial fixes:
davem: the day Linus releases we always get a pile of "missing MODULE_EXPORT()"
type bug reports that are one liner fixes.
davej: So what was broken with the 2.6.8.1 type of release ?
akpm: But that _is_ a branch

I think we all like the 2.6.8.1 model.

davej: The only question in my mind is 'how critical does a bug have to be
to justify a .y release'.

I think that is the wrong question. The question should be 'how trivial should a fix be to be admissible in a .y release'. If something is important but nonobvious, it should cause earlier release of 2.6.x+1.
No regressions in 2.6.x.y.

-----
About testing: users do not test p.q.r-suffix, they just test p.q.r
Solution: Release early, release often. No silly odd/even games.
jgarzik: I would prefer a weekly snapshot as the release

-----
API stability: Stable series like 2.0, 2.2, 2.4 try to maintain the guarantee that user-visible APIs do not change. That goal has disappeared, meaning that anything might break when one upgrades from 2.6.14 to 2.6.16.
This is one of the big disadvantages of the new 2.6 way.

-----
A separate twig is that of when to call a release 'stable'. Since there is no good way to predict, the solution is 'wait and see'. 'Stable' also depends on the type of use. It is easier to leave construction of stable kernels to separate people and trees. With some delay they can have the benefit of hindsight.

Andries

Both 2.4 and 2.6 are IMHO similar. There were API changes in 2.4, and in 2.6 they should be discouraged exactly as in 2.4.

Krzysztof Halasa

It depends on what you call a "user visible API".

Changes which are visible to userspace are treated with great caution nowadays[*]. I keep on rejecting patches...

Changes which are visible to third-party kernel modules are more acceptable, but we're still fairly reluctant to make them.

So I think you exaggerate. And I don't see that the problem which you describe is inevitably a part of the "new 2.6 way".

[*] I don't know any details of the /proc incompatibility which davej
mentions, and I'd like to. That sounds like a screw-up.

We changed the format of /proc/slabinfo. Running slabtop threw up an error message complaining that the format had changed.

It was a graceful failure, but a failure none-the-less.

Other failures have been somewhat more dramatic.
I know ipsec-tools, and alsa-lib have both caused pain on at least one occasion after the last 2-3 kernel updates.

I was keeping track of these breakages, but I can't seem to find the text file right now. Bill, can you remember any other cases worthy of mentioning ?

Dave

alsa-lib is a special case.

alsa-lib exists so that it can mitigate changes between the kernel and userland. If the kernel API changes, alsa-lib may need to be updated, but not all the apps using alsa-lib.

Someone needs to put alsa-lib updating on the list of stuff to do when updating the kernel...

Jeff

The ones I remember were:

- alsa-lib, as mentioned
- ipsec-tools/openswan needed updating when the kernel changed to require forward SAs
- a old script on an earlier release needed fixed when usbdevfs finally went from being deprecated to dead

SELinux changed froom 2.6.early to 2.6.current as well, IIRC.

Bill

alsa-lib: You mean the problem with the emu10k1 based soundcards from Creative? It's because the multichannel devices were added and reorganized. But the basic functionality (sound from front speakers and analog capturing) should be preserved with all 1.0.x alsa-lib versions.

Jaroslav

-----
Jaroslav Kysela <[email protected]>
Linux Kernel Sound Maintainer
ALSA Project, SUSE Labs

Do the statistics on rate of discovery of security problems (generally low priority and by good guys/verification tools) and you are looking at about x.y.8 or so assuming there are no 2.x.y.z releases for post release howlers.

I think there is a case for the "community" providing the most "stable" kernel that it (reasonably) can without depending on "distributions" to do that.

One reason is that (some) distributions are known to have released kernels with quite broken and unreviewed patches, or with new functionality that never ends up appearing in main-line for whatever reason.
Further, it would surely be useful for all distributions to have one central place that 'stablising' patches appear so they can pick-and-choose from them rather than each keeping their own independent set.

For the kernel, I am the "distribution" for my employer and I choose which kernel to use, with which patches. I really don't want to hunt around for all those stablisation patches, or sift through the patches in 2.6.X+1-pre to find things to apply to 2.6.X. I would be really happy there was a central place where maintainers can put suitably reviewed "important bug fix"es for recent releases, and from where kernel maintainers for any distribution (official or not) could pull them.

Having said that, I am not in a position to offer my services to maintain such a really-stable kernel branch, so I'll just cope with whatever is provided.

NeilBrown

The point is that it's happening anyway. See Andres' -as tree which is the basis for the Debian vendor kernel. Getting that up to an official status as 2.6.x.y would be very nice (and having it on linux.bkbits.net)

Christoph wrote in reply to Neil:

Interesting. We already have a pre-Linus tree, in Andrew's *-mm. Currently, each distro adds its own set of patches, on top of some version of a Linus tree. If we had a blessed post-Linus tree, which the 2.6.<even>.y numbering fits, then we could unify this post-Linus tree patching to some degree as well, to the general benefit of most everyone involved. A bit of holy penguin pee, a chosen spot in the official numbering scheme, and a good choice of penguin prince as manager of this release series should be enough to get this going.

                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.650.933.1373, 1.925.600.0401

IMHO it is nowhere near conservative enough (or at times complete enough) to be a 2.6.x.y kernel. In some respects -ac is closer but it also isn't as conservative as a real 2.6.x.y should be.

2.6.x.y needs several people to keep it tight and to ensure there is always cover on a security fix.

Hi!

Eh?

Like you add security fix and then some formatting change to hide it?

Pavel

People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!

Cover has rather too many meanings I guess. Cover as in "there is someone available to handle it"

I agree, I think that's useful and needed. It's possible to get the fixes committed to an effective branch in bk and pull that back into mainline. So at each new release the last release hotfix branch would stop. But does that solve a different problem? I don't think it'll increase people testing intended-stable versions.

thanks,
-chris

In article <[email protected]> (at Wed, 2 Mar 2005 16:58:30 -0800), "David S. Miller" <[email protected]> says:

I'm afarid people start waiting 2.6.${even} releases...

--yoshfuji

I understand the desire and benifit to do this sort of fixing, but I really don't like extending the odd/even thing to other parts of the kernel version numbers

with 2.6.8 a different path was attempted with 2.6.8.1 couldn't we just use that numbering scheme (avoiding adding to the numbering confusion further) and plan on releasing a 2.6.11.1 next tuesday (or so) with the various paper-bag fixes?

If I understand bitkeeper properly Linus wouldn't even need to duplicate the patches, he should be able to fork his tree to do the paper-bag fixes in one fork while continueing development in the other one and then recombining them when either generates a release.

David Lang

There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare

Except more people who think like me. I usually enjoy playing the canary, else I wouldn't do it.

More Canaries I say. I build, and run till the next rc comes out almost every rc & I'm curently up on 2.6.11 for 5h:30m. I've only got this one box thats fast enough to do that and still get anything else done too, so this is the workhorse. I don't yelp about deprecation notices during the build, I figure you people have eyes too.

Basicly on my limited variety of hardware, I at least can play the part of the canary here, and have a couple of times when problems show up in less than a couple of days or during the build itself.

What you really need is a stronger plea for more people to go beat on the rc releases, but I fear thats never going to happen as long as there is nothing but the rc stuffs, and no -pre stuff for testing the -mm stuff a little wider. Back in 2.4 days, even the -pre stuff was often stable enough for extended uptimes, and I mean that I ran

May 1 2003 vmlinuz-2.4.21-pre7 on that 7.3 box until Jan 21 2005.

Its now got a 2.4.29 final on it and:
8:56pm up 23 days, 8:00, 6 users, load average: 2.12, 1.40, 1.14 sailing along just as smoothly as ever since Feb 7.

Not without a reboot over that nearly 2 years of course, but the choice of when to do the reboot was mine in every case. Here on this box, trying to play canary, uptimes are often less than a week, sometimes less than 24 hours, and that isn't anywhere near long enough to be able to say 'its stable'. Stuff is IMO, being shoved out without an adequate amount of time to see what the 'canary' does.

If we could only get more people to 'bang on it, kick the tires etc', AND fix those sorts of problems that do show up in the next sequential rc, a lot of stuff like 2.6.8 could be headed off. Note the caps on the word AND. That bugfix in the next rc release seems to be broken when obviously needed fixes often have to go back to the head of the queue and work their way back to Linus.

So how DO you go about getting the tires kicked by more users? I'm not sure what would work.

One doesn't have to be a code monkey to do this 'canary' scene as long as a bash script can be hacked up to do the majority of the work, I have a couple of them that basicly make a new kernel install a no brainer. Often under 30 minutes to being rebooted to a new rc from going after the patch.

And I ramble, but this is my $0.02. And probably about what its worth in the Grande Scheme of Things...

I agree. Ditto for the 1394 fixes that have been upstream for at least a month, maybe more. I haven't checked to see if kino can run my movie camera yet, if it can, then maybe it made it into 2.6.11. Otherwise I'll have to go nuke that dir in the drivers tree, and again grab the svn from the 1394 site and rebuild/reinstall. Provided the svn server is up, thats not a big deal here. The point is that it is being tested, and it does work.

Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.34% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.

-mm always holds the latest 1394 tree. So you can run -mm, or just snarf bk-ieee1394.patch from the broken-out directory.

Thanks Andrew. If this is the same as I pulled from svn 2 weeks ago, it does need to be pushed on down the line.

Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.34% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.

I'm a casual user, and I'm happy to bang on the latest (-mm) kernels so long as my data doesn't get corrupted. But the process of downloading, patching, configuring, compiling, and installing a new kernel is a bit of a pain, requiring user intervention a number of times. It's enough that I don't update very often.

I've just done a bit of looking for scripts to automate the process of installing a new kernel, and I haven't come up with much of much. So right now I'm writing my own. If there are tools to help automate this they need to be more prominent on www.kernel.org and www.kernelnewbies.org, to make casual testing even easier.

Try ketchup from here: http://www.selenic.com/ketchup/
ketchup 2.6-mm for example will download and patch the newest 2.6-mm kernel (also try ketchup 2.6-pre, ketchup 2.6-bk, ...).

Jan

http://l4x.org/k/

If we want a calming period, we need to do development like 2.4.x is done today. It's sane, understandable and it works.

2.6.x-pre: bugfixes and features
2.6.x-rc: bugfixes only

If you think it will be only a week, you're deluding yourself. It will stretch out to a month or longer, and the backlog problems will be real.

A calming period is fine. But this even/odd mess is just silly.

Jeff

No. It's insane, and the only reason it works is that 2.4.x is a totally different animal. Namely it doesn't have the kind of active development AT ALL any more. It _only_ has the "even" number kind of things, and quite frankly, even those are a lot less than 2.6.x has.

And the reason it does _not_ work is that all the people we want testing sure as _hell_ won't be testing -rc versions.

That's the whole point here, at least to me. I want to have people test things out, but it doesn't matter how many -rc kernels I'd do, it just won't happen. It's not a "real release".

In contrast, making it a real release, and making it clear that it's a release in its own right, might actually get people to use it.

Might. Maybe.

Linus

People don't test 2.6-rc releases because they know they are not "release candidate, with only bug fixes" releases, which is how the rest of the world interprets the phrase.

Making them official releases in the even/odd manner is what neilb implies. You'll just be diminishing the value of releases. A "real release" won't be a real release anymore. You're just renaming the -rc that isn't really an -rc.

Jeff

That's not %100 true. No matter what -rc* really is, people perceive it to be something based upon it's name and whether or not it is an official release. As far as I can see it's %95 perception vs. reality. And IT IS part of the reason the -rc's are not as good as they could be.

The reasons -rcs are not as good as they could be is that they include more than just bug fixes. Users are discouraged from testing because they must scan LKML, or guess, which -rc that Linus/Andrew started getting serious about "bugfixes only."

With the -pre/-rc scheme, it's clear to users.

With the even/odd scheme, you just devalue releases. Previously, all releases were worthy of testing and use. Now, half of them aren't.

Jeff

If Linus/DaveM really don't like -pre/-rc naming, I think 2.6.x.y is preferable to even/odd.

Just create a 2.6.X repo at each release. For bug fixes to 2.6.X, commit to this repo, then pull into linux-2.6. For everything else, pull straight into linux-2.6.

The linux-2.6 repo would be upstream, and linux-2.6.X repo would be where bugfix-only releases come from.

Jeff

All of these arguments are circular. If people think that even/odd will devalue odd releases, guess what 2.6.x.y will do? By that line of reasoning nobody will test 2.6.x just the same as they aren't testing 2.6.x-rc* right now.

I think they will test the odd releases, because as a real release they will get slashdot/lwn.net/etc. announcements.

That's one of the major things the -rc's don't get. Maybe it gets a reference in lwn.net's weekly kernel article, but mostly kernel geeks read those and that's not who we want testing -rc's (such geeks already are doing so).

It has to be a "real" release. That does have an impact. However, I am ambivalent about how to make them real. Even/odd, 2.6.x.y, either is fine with me.

How do you know that they won't stop the announcements if this change is made?

--Russell

Russell Miller - [email protected] - Agoura, CA

Nobody knows such things for sure, let's test it and find out :-)

even/odd means that certain releases (even ones) are more magical than others. That's weird, since users aren't used to that sort of thing in any other project.

2.6.x.y and 2.6.x-rc [where rc == serious bugfixes only!] are understandable to users, because they have seen that sort of thing before.

Users _aren't_ fooled by naming games. The current 2.6-rc proves that.

LWN, Slashdot and others will not be fooled though :) They will note that release 2.6.<odd> is not a real release.

2.6.x.y has a very real engineering benefit: it becomes a stable release branch. That will encourage even more users to test it, over and above a simple release naming change.

Users have been clamoring for a stable release branch in any case, as you see from comments about Alan's -ac and an LKML user's -as kernels.

Jeff

Sure they've been asking for it, but I think they really don't know what it entails. Look at all of the "non-stable" type patches in the -ac and as tree. There's a lot of stuff in there. It's a slippery slope down when trying to say, "I'm only going to accept bug fixes."

Bug fixes for what? Kernel api changes that fix bugs? That's pretty big. Some driver fixes, but not others? Driver fixes that are in the middle of bigger, subsystem reworks as a series of patches? All of this currently happens today in the main tree in a semi-cohesive manner. To try to split it out is a very difficult task.

So, while I like the _idea_ of the 2.6.x.y type releases, having those releases contain anything but a handful of patches will quickly get quite messy.

Not to mention the issue of the need for me as a maintainer to mark "bugfix only" specific patches and pull them out and submit them separately. Due to api changes, and all sorts of other issues, that can get to be a difficult job in itself.

Personally, I like the current, "test it all in -mm and then forward the good bits to Linus" mode we are operating in. My only suggestion would to possibly speed up the release cycle a bit faster than every two months, like we currently are on. Once a month perhaps? That was how we were working at the beginning of 2.6, and it seemed like the backlog was much smaller then.

thanks,

greg k-h

We have all these problems precisely because _nobody_ is saying "I'm only going to accept bug fixes". We _need_ some amount of release engineering. Right now we basically have none.

Easiest to answer with a concrete example:

Linux 2.6.11 is released. Linus then does a

	bk clone linux-2.6 linux-2.6.11

Bug fixes that
(a) 2.6.11 users really should have, or
(b) Linus/Andrew feels are important, or
(c) a subsystem maintainer feels are important [and does the work to split out the fixes]

go into linux-2.6.11 repo, and then is pulled into linux-2.6 repo.

All other changes go into linux-2.6.

There's no need to over-think or over-work this. The goal is to provide a stable 2.6.11 for users, until 2.6.12 is available.

My prediction is that several patches will flow into the linux-2.6.11 repo a week or so after a release, and then the flow will die off to a trickle. Subsystem maintainers that care can submit patches/BK-pulls for the stable release if they so desire.

Only important "oh shit, that should have been in 2.6.11" bug fixes need apply. Bug fixes for reworks, API changes, etc. are -not- applicable to linux-2.6.11 repo.

Since BitKeeper can handle nicely a

	cd linux-2.6
	bk pull ../linux-2.6.11

there is no duplication of bug fixes.

Jeff

Linus says that in -rc releases, are you not paying attention to that? (yes, I know you are...)

Will Linus start accepting 2.6.12 patches right after 2.6.10 is out?

That sounds very sane to me, I can live with it.

The main point is, will people really test the 2.6.odd releases to help us out in making this new scheme work. If so, I'm all for it.

thanks,

greg k-h

The point there is that users, who are conditioned in the software world to think "release candidate means bug fixes only" come to Linux only to learn that

-rc1 isn't a release candidate, it's just the first public test kernel.
-rc2 isn't a release candidate, it's just a signal to kernel developers to slow down -rc3 may be a release candidate, unless Linus does something typical and merges non-bug fixes. -rc4 is probably a release candidate
...

The users ARE paying attention to this -- this is precisely why they do not take -rc releases seriously.

2.6.X.Y is a bit different from 2.6.{odd|even}.

With 2.6.X.Y, you are not limited to single bugfix release.

With 2.6.X.Y, the kernel only gets successively more stable and bugfixed, without having to worry about new breakages.

With 2.6.X.Y, you are giving the users a familiar paradigm for understanding the release mechanism.

With 2.6.X.Y, all releases are "real releases." You're not tricking the users with semantic games.

So there is a strong incentive not only to test... but to _use_ our releases.

Jeff

Sorry Jeff, but that's crap. Go look at the commits list. Every single patch which has gone into the tree for the past two weeks is a bugfix, I think.

The most recent non-fix I see is

ChangeSet 1.2044, 2005/02/22 20:44:37-05:00, [email protected]

[PATCH] r8169: removal of unused #define

Removal of unused #define

So pphhhfftt ;)

And the most recent non-trivial not-completely-a-bugfix patch I see is "USB: Add ioctls to ub", from Feb 17. And even that's an outlier.

It's an an exaggeration, but what do users that don't follow daily kernel development see?

"oh, somewhere between -rc1 and -rc5 they got serious about taking only bugfixes. Andrew Morton says it was the past few weeks... what -rc is that?"

Jeff

I agree that this is one of the main problems.

But look at how to solve it. The _logical_ solution is to have a third line of defense: we have the -mm trees (wild and wacky patches), and we have my tree (hopefully not wacky any more), and it would be good to have a third level tree (which I'm just not interested in, because that one doesn't do any development any more) which only takes the "so totally not wild that it's really boring" patches.

In fact, if somebody maintained that kind of tree, especially in BK, it would be trivial for me to just pull from it every once in a while (like ever _day_ if necessary). But for that to work, then that tree would have to be about so _obviously_ not wild patches that it's a no-brainer.

So what's the problem with this approach? It would seem to make everybody
happy: it would reduce my load, it would give people the alternate "2.6.x base kernel plus fixes only" parallell track, and it would _not_ have the testability issue (because I think a lot of people would be happy to test that tree, and if it was always based on the last 2.6.x release, there would be no issues.

Anybody?

I'll tell you what the problem is: I don't think you'll find anybody to do the parallell "only trivial patches" tree. They'll go crazy in a couple of weeks. Why? Because it's a _damn_ hard problem. Where do you draw the line? What's an acceptable patch? And if you get it wrong, people will complain _very_ loudly, since by now you've "promised" them a kernel that is better than the mainline. In other words: there's almost zero glory, there are no interesting problems, and there will absolutely be people who claim that you're a dick-head and worse, probably on a weekly basis.

That said, I think in theory it's a great idea. It might even be technically feasible if there was some hard technical criteria for each patch that gets accepted, so that you don't have the burn-out problem.

So let's loook at how we could set that up. We need:

- a sucker who wants to do this, or a company that pays for somebody good to do this (and remember: "good" here doesn't necessarily have to mean technical genius, it's about taking abuse and being stable). The whole setup should be such that there can never be any question about the patches for _other_ reasons (to avoid the sucker becoming a target for abuse), so this person really to some degree would be fairly mechanical.

Don't make it automated, though. That just gets us down the path of flaming about the scripts and automation. And I'm not claiming that we should aim for somebody _stupid_, I'm just claiming that it takes a certain kind of person to do something that is not all that glamorous, and that puts you in the spot.

We don't ever want to have that spark of "wouldn't this be cool" in this project.

- some very _technical_ and objective rules on patches. And they should limit the patches severely, so that people can never blame the sucker who does the job. For example, I would suggest that "size" be one hard technical rule. If the patch is more than 100 lines (with context) in size, it's not trivial any more. Really. Two big screenfuls (or four, for people who still use the ISO-ANSI standard 80x24 vt100)

Also, I'd suggest that a _hard_ rule (ie nobody can override it) would also be that the problem causes an oops, a hang, or a real security problem that somebody can come up with an exploit for (ie no "there could be a two-instruction race" crap. Only "there is a race, and here's how you exploit it"). The exploit wouldn't need to be full code that gets root, but an explanation of it, at least.

- a vetting process. You'd have ten people, and five of them would have to sign off on the patch, and even a single veto would shoot it down.

Again, this is really to protect the sucker, and make it possible to
work: I don't think this can work with a creative person (everybody else calls me "flaky", and I much prefer that "creative" word, it sounds so much better), which I personally believe means that we don't _want_ people like Alan, Andrea, Andrew etc etc that have historically maintained their own trees that sometimes have tried to do something like this.

- Finally: this tree never has any history past the "last release". When a new kernel comes, the tree is frozen, and never to be touched again.

If somebody _else_ wants to base things off this special "sucker tree", and make a fourth level tree that is based on the _previous_ stable tree, that's fine, but that's a separate process. He would be totally free to do so, but the rule is that this particular maintenance program _never_ gets stuck on an old kernel, like the vendor trees always are.

This is not a long-range tree, it would _purely_ be about one thing and one thing only: the last stable kernel. The people involved (sucker and vetters all) would never have to remember two different trees, or care about problems that aren't in the top-of-tree. Keep ti simple, and keep the rules clear.

Does this mean that some patches would never go into this tree? Yes. It would mean that patches that some people might feel very _strongly_ are good patches would never ever show up in this tree, but on the other hand, I can see this tree being useful regardless, and I think the lack of flexibility in this case is actually the whole _point_ of the tree. The lack of flexibility is the very thing that makes this be the kind of base that anybody else can then hang their own patches on top of. There should never be a situation where "I'd like that tree, but I think xxxx was done wrong".

Might something like this make people happier? (I wrote "happy" rather than "happier" at first, but let's face it, people are better at whining than they are at being happy ;)

Linus

Well, I'm one person who has said that this would be a very tough problem to solve. And hey, I like tough problems, so I'll volunteer to start this. If I burn out, I'll take the responsibility of finding someone else to take it over.

I really like the rules you've outlined, that makes it almost possible to achieve sanity.

thanks,

greg k-h

Ooh, a sucker!

Seriously, I think Linus's plan makes a lot of sense, as a scalable way of maintaining a 2.6.x.y release strategy.

The other thing which would probably be useful to maintain would be a list of "known regressions" yet to be fixed in 2.6.x.y, and to address the somewhat disturbing assertions that sometimes regressions "light up bugzilla" at distro's like Fedora, but don't get reflected back up to LKML. Maybe we could recruit some other sucker to maintain such a list? It wouldn't be a complete bug list, and it can be discarded every time a new 2.6.x is released, which means it should be more manageable than some previous attempts to maintain LK bug lists.

- Ted

Two of us even :)

I agree, and if Chris and I share the load, it might even make it a bit more robust in that we can cover for each other when one is traveling, etc.

That would be great, and any help from the distro bug-wranglers would be appreciated.

thanks,

greg k-h

How does what Linus outlined differ from splitting to 2.7? Isn't it the same thing with just a different versioning scheme? In the end we have a stable 2.6 tree. What's the difference between the 2.6 development tree and 2.7.

All that aside... why not make the "sucker tree" a breeding ground for new kernel hackers. I've been trying to get more involved with kernel development (as are others) but it's tough because things change so rapidly that once you finally figure it out, it has changed. As the kernel matures into new versions it's getting much harder for guys coming out of school to figure out. Sure you have that exceptional individual but maybe you're committing genocide in the end.

If someone like Greg could lead this band of gypsies then that would be great. Most of the more experienced hackers don't want to slow pace and do all this boring work but people like myself would be honored to "pay our dues". We would feel free to ask some questions and see a slower pace so we can learn. If patches are small and truly only fix bugs then wouldn't you say this would be possible? In fact one criteria could be that if we can't understand just what exactly the patch fixes then it probably doesn't belong in this tree? It's a win - win situation. End users are happy and we start a semi-structured training camp.

--
"Education is what remains after one has forgotten everything he learned in school."
- Albert Einstein

- Each 2.6.x.y series would be abandanded after the next 2.6.x release came out.
- There would be no big development fork.

Those are two ways this differs.

Have you looked into the kernel-janitor project? That's the best "breeding" ground around for people who want to learn the kernel development process. I highly recommend that.

I don't think this "stable/bugfix" release series would be a good place for new hackers, as most first bugfixes are of the janitorial type, which would not be accepted into such a release tree.

thanks,

greg k-h

Andres Salomon (-as patches) and I have been talking about that at least regarding security fixes. It's worth trying in a more complete and formalized way. Guess I can be branded a sucker ;-)

I like this definition. The only remaining question is what determines a 2.6.x.y release? One patch? Sure if it's critical enough.

Heh, maybe people are happiest when they are whining ;-)

thanks,
-chris

Linux Security Modules     http://lsm.immunix.org     http://lsm.bkbits.net

Why should there be one? One of the things I like about this concept is that it's just a moving tree. There could be daily snapshots like the -bkX "releases" of Linus's tree, if there are changes from the day before. It means (hopefully) that no one will "wait for x.y.z.2 because that is really stable".

Jens Axboe

So that means we do the release often, _and_ provide -bkX snapshots daily if we happen to take a few days between releases and patches are pending for some reason.

thanks,

greg k-h

Exactly. Th ewhole point of this tree is that there shouldn't be anything questionable in it. All the patches are independent, and they are all trivial and small.

Which is not to say there couldn't be regressions even from trivial and small patches, and yes, there will be an outcry when there is, but we're talking minimizing the risk, not making it impossible.

Linus

OK. I was aniticpating people's request for an official 2.6.x.y release (since I've already had that request a bunch). But as long as the snapshot can be easily identified in the unlikely case of a regression, then snapshot could be good enough.

Linux Security Modules     http://lsm.immunix.org     http://lsm.bkbits.net

Wait a second though, this tree will be branched from the development
mainline. So it will contain many patches that entered with less
testing. What will be the policy for dealing with regressions relative
to the previous $sucker release caused by huge patches that entered via
the development tree? Is reverting them prohibited because of the patch
size?

Sean

Less testing than what? Remember, we don't have a "test suite" for the kernel, no matter how much propaganda osdl likes to emit :)

We'll deal with them as they come, on a case-by-case basis. Acceptable?

thanks,

greg k-h

Well, since I'd pull basically all the time, all the patches _do_ get testing the the -rc kernels.

But the rules would be that small patches can also be verified independently.

And realize that the 2.6.x tree doesn't "change" any way. It doesn't get more unruly just because we have a side tree that is being anal about things. It's still as stable as it ever is..

Linus

I can see two conflicting desires in this discussion, the desire to continue development to avoid patch backlog, and the desire to slow down and stabilize to provide a sane release-candidate and release scheme. Could the two desires somehow be both resolved simultaneously?

Perhaps instead of forking when 2.6.A is released, Linux could fork earlier, after the 2.6.A-bk series. After the fork, the main tree would become the new 2.6.A+1-bk, and the forked tree would become 2.6.A+1-pre. Then the final stabilization and patches could continue while normal kernel development moves on. The latest kernel could take advantage of patches to the release kernel, but would be able to maintain the steady patch stream. The release kernel could be managed by the previously mentioned "sucker", and could go through a more-stabilizing and better tested Release Candidate series, and then maintain post-release bugfixes. When 2.6.A+1-pre is released, then all upstream development on the forked 2.6.A-post tree would cease.

Cheers,
Kyle Moffett

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$ L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+ PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r !y?(-) ------END GEEK CODE BLOCK------

Sure. Or a patch for 1-2 days, will less critical things.

And probably no .tar* balls for them. Just a patch against 2.6.x.y-1.

Krzysztof Halasa

This way you have a fixup tree for the latest 2.6.x release, but does not resolve the problem of the release quality itself.

It makes more sense to do

2.6.x-pre1 .. preX in your tree until you feel comfortable to move it into the release tree, where it starts from

2.6.x-rc1 .. rcX bugfix only up to the final release.

the real trivial and urgent fixes and produces 2.6.x.y updates. When the next -rc1 appears the previous 2.6.x.y is frozen and never touched again.

This way you have ongoing development in your tree and the -rcX releases might find more testers than now.

I think when the rules are clear and some gratification would be available you might find somebody to jump on this train.

tglx

Isn't this what -ac and -as effectively already are?

Chris

Based on the patches in those trees, no :)

thanks,

greg k-h

I've not found a much smaller set that isn't rootable, trivially DoSable with published tools or leaves users with non-working hardware that got pulled by Linus having a random pissy fit about pwc etc.

-ac is essentially base security fixes + working IDE locking + pwc + fixes for the bugs everyone hit that needed fixing urgently. I consider working locking on my storage essential because I like my data to still be there.

-as is similar although it makes different choices about what matters.

Given 3 or 4 people it ought to be possible to make a much much tighter patch set for this purpose.

Working IDE locking? Does this mean if I have 2 promise cards, a HD on each card and I copy from one to the other it wont all blow up in my face?

Please say yes. :)

    Red herrings strewn hither and yon.

Depends on your PCI bus and also if the are on the same IRQ. In the same IRQ case you may find 2.6.11 is a bit saner as Bartlomiej may have sorted one of the IRQ masking problems now.

Hmm. How can I check on this? What should I look for?

/proc/interrupts reports different irqs for each ide device:

  5:    5018683          XT-PIC  ide4
 11:     167995          XT-PIC  ide2, eth0
 14:      39799          XT-PIC  ide0
 15:      10704          XT-PIC  ide1

The problems were weird. The fs I was copying from decided it was corrupt. Unmounting the partition and trying an fsck reported that it couldn't find the partition. After a reboot all was well and a fsck reported no problems.

    Red herrings strewn hither and yon.

Similar stuff with ac12 if dma is on on both drives.

Mar  4 15:15:55 nessie kernel: hdi: dma_timer_expiry: dma status == 0x21
Mar  4 15:16:10 nessie kernel: hdi: DMA timeout error
Mar  4 15:16:10 nessie kernel: hdi: dma timeout error: status=0x58 { DriveReady SeekComplete DataRequest }
Mar  4 15:16:10 nessie kernel:
Mar  4 15:16:10 nessie kernel: ide: failed opcode was: unknown

Should be noted that hdi does not boot with multisec or dma on.

    Red herrings strewn hither and yon.

If you've got two promise cards on a VIA 133Mhz or early 266Mhz chipset for example then there are large numbers of reports both Linux and other OS about problems.

That indicates the corruption was in memory not on disk. I take it the box passes memtest86 fine however ?

No. They both end up doing a lot of much fancier stuff. There are patches in there that I may not be comfortable with, because they end up doing things like totally re-doing the locking for some subsystem.

Yes, they end up re-doing _broken_ locking, but the point is that they are not obvious. They are just "more careful" versions of the -mm tree.

Linus

This only attacks part of the problem.

Most regressions that annoy users aren't in some core system, they are in drivers.

What if $big_driver_update in 2.6.12 fixed a serious bug for some people but broke the driver for many other people, and both reverting this patch and fixing the driver for the people it's broken for exceeds such size limits?

cu
Adrian

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Then either the patch author splits out the bug if they want to, or we punt and say "wait for 2.6.12". Distros can then decide if they want to take the whole $big_patch in their releases, if they are near a release cycle.

I feel that imposing such limits is the only way this can be done and remain sane.

thanks,

greg k-h

Yes. Exactly.

We're not aiming for "perfect". Just _trying_ to be perfect is what would kill the whole scheme in the first place. We'd be aiming for "known rules".

Whether people _agree_ with those rules is then actually not a huge issue. There will _always_ be things that people don't agree with. Aiming for consistency is worthwhile in itself.

(Of course, the rules _do_ matter in the sense that there has to be some point to the consistency. You can have a consistent rule that "the ChangeLog entries must rhyme", and I think it's a great rule, and I encourage anybody who wants to to set up such a "rhyming kernel tree", but that doesn't mean that it makes a lot of difference to people ;).

So havign strict rules that allow _one_ kind of consistency that people agree is good is a fine idea.

And Adrian, you can always have a different tree that has another set of rules - and if you use BK you can merge the two and have the "combination of the rules" tree. The reason I would _stronly_ urge very tight rules is that if they aren't tight, it ends up having all the problems we've always seen in other trees.

For example, if the "tight rules tree" allowed reverting an otheriwse good patch because it had a bug (instead of trying to fix the bug), then I would never be able to pull that tree into mine. It would take development _backwards_, and thus it might be sensible for a vendor, but it would automatically mean that it's not a good base for the next kernel version.

And if I can't just say "ok, I'll always take the 'tight rules' tree", then we'd get into the forward-and-backward porting hell again, which would make the whole tree totally pointless. See my point?

Linus

It still does not solve the problem of "untested" releases. Users will still ignore the linus-tree-rcX kernels.

So we move the real -rcX phase after the so called stable release.

Doing -rcX from the "sucker" tree up to a stable release makes much more sense and would have more testers and get back lost confidence.

tglx

Nod. I really don't think the key penguins understand that there is even a problem here. Let me attempt to re-state the problem:

There is no flag day when "bugfixes only" mode begins.

There are two component to this problem:

1) Release maintainers need to avoid merging non-bugfixes. Lately, the key penguins _have_ been doing their job here. This manifested in 2.6.11-rc4, 2.6.11-rc5.

2) This "flag day" when bugfixes-only mode starts needs to be completely obvious to _scripts_ and really dumb people. Posting to LKML "with this -rc, I am only taking serious bugfixes" doesn't cut it. There must be a clear, consistent point where testing may begin.

As a result of this problem, people have learned [just re-read this thread if you don't believe me] that "-rc" doesn't really mean Release Candidate until -rc2 or -rc3 or so. And as a result of that, people do less testing.

Jeff

True, but the confidence of users in -rc is gone already. So testing happens when the "stable" release is out, but thats too late.

If you clearly seperate out the -rc process of the development tree then it is no question how dump scripts and people are. The -rc tree is untouchable for anything else than bugfixes.

tglx

.. and maybe that problem is unsolvable. People certainly argued vehemently that anything we do to try to make test releases (renaming etc) won't help.

So what do you do if you find an unsolvable problem? You don't solve it:
you make sure it's not a show-stopper.

So part of the idea of having the "other tree" is that it ends up solving the "hmm, we missed that detail" problem. And by _not_ giving it release numbers or any schedule at all, people can't _wait_ for it.

Sneaky. That's my middle name.

Linus

I disagree it's unsolvable:

1) At some point in the -rc cycle, you put your foot down and say "nothing but bugfixes."

2) Indicate this foot-stomp in a manner that a script can easily notice.

That's all the 2.4.x's -pre/-rc accomplishes. It encourages people to test, by telling them when their testing would be most useful.

Jeff

Linus actually did, as Andrew showed you, and it was actually followed quite well.

You keep ignoring this evidence, why?

You can quiet me up about this by showing counter evidence to what Andrew pointed out to you.

I agree this occurred. I'm not ignoring the evidence, you are ignoring the part you didn't quote -- the key component of the "users don't test" problem.

A user who doesn't follow LKML does not know that Linus put his foot down in 2.6.11-rc3. or 2.6.10-rc2. or 2.6.9-rc1. A user just sees "at some random point, which is never consistent, it becomes bugfixes only."

The lack of consistency is the problem. That is why posters in this thread keep suggesting a -pre/-rc split. That is why posters in this thread talk about "users don't trust -rc to mean -rc."

Marcelo consistently says "-rc means serious bugfixes only" and never deviates from that. And that builds users TRUST.

"putting down your foot" is only half the solution.

The other half is, "users need not follow LKML to know when this occurs."

Jeff

Release candidates are supposed to be bugfix only from -rc1. Everything else can only be called the "ridiculous count".

Correct, but again 2.4 is a different beast as no active development is taking place.

In a active development you should move the -rc step out of the development line into a seperate release line after the -preX steps.

It has two advantages:

1. The release process is encapsulated and secure against non bugfix merges. This can be "sold" to users for testing.

2. The development cycle is more continous. While -rc1 is led to release 2.6.X the preparations for 2.6.X+1-pre1 are moving on. This might shorten the overall release cycles as well.

tglx

As a further elaboration...

The problem with the current 2.6-rc setup is a _human_ _communications_ problem.

Users have been trained in a metaphor that is applied uniformly across all software projects that use the metaphor:

test release: a useful merge/testing point
release candidate: bugfixes only, test test test

Linux does it differently.

It's hard enough to get users to test... now we have raised the barrier even higher by abusing a common metaphor. A metaphor that is used _succesfully_ elsewhere to get users to test.

"release candidate" is a promise to users that the current tree is close to what the release will look like, and only major fixes will appear between -rc and -final.

We broke that promise. In human interface terms, this is like redefining the "garbage can" icon to mean "save your work." ;-)

Jeff

I've seen this comment before too, and I still think it says it best:

The full release s/b the last rc with NO changes other than its name.

Now we are faced with a final that may have another 50k+ of patches applied over what made the rc5. IMO, in the immediately past case, that should not have been final, but an rc6.

I personally vote for going back to the -pre series where we get to test new ideas & maybe morph the neighbors dog/cat. At some point it gets stable enough to call it -rc1, at which point nothing but bugfixes should be allowed in until the final. That way we have a develop, then stabilize, then develop the next, then stabilize it set of stairsteps.

I personally build and run as many of them as I can find the time to do. I just applied the bk-1394-patch and its building because 2.6.11 isn't really comfortable running my movie camera yet.

I also haven't built any more of Andrews -mm's since the last time I got told off about it. I can go back to doing that too, if the results of my testing it aren't going to be thrown out like the babys bath water.

Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.34% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.

Not sure about it. I think last-rc -> final diff should be allowed to contain trivial patches, like those qualified for 2.6.x.y.

Krzysztof Halasa

It wont help, if you keep it in the development line. If you move it out of the development line into the "other tree" then it is likely to be accepted as _real_ release candidates which get a proper testing coverage before declaring it stable release nr. XYZ.

Users _are_ scared off by the current release quality. Your model is ignoring this part of the problem and just provides a pacifier gimmick.

After the _stable_ release, of course no schedule and release numbers are necessary. The proposed scheme will work nicely.

People are clever enough to debunk sneaky gimmicks.

tglx

No, we argued that _if_ you don't do real -rc's no amount of renaming will trick people into believing they are (for long). Note how that's what has happened now. Your -rc's are -pre's, certainly the first ones for each kernel, and that's exactly how people treat them -- that is, many avoid them.

Doing -rc1 to release and beyond from that per-release "sucker tree" as you call it (I prefer "release tree") makes more sense. This gets you real release management and the only thing left to do is appoint the release manager.

It's not unsolveable. Doing real -pre's and -rc's (which means "doing doing real -rc's" in addition to the current situation) would at least to a degree solve it.

Judging by the fact that apparently you know have to few testers "seeing right through Linus" would seem to be everybody else's, though.

Rene.

One thing that's worth pointing out is that sometimes the 20-line patch is sufficient to solve the problem, but is lousy code. That's fine for a tree that will be frozen in a couple of months after only getting little fixes to other files, but mainline should get a real fix, which wouldn't be acceptable in the sucker tree. So 2.6.(x+1) shouldn't automatically get 2.6.x.y. Should reverting a 200-line patch which turned out to need another 200 lines to work be acceptable?

Similar rule for driver patches: you can patch a driver if it makes the device not work (but you couldn't patch the core)?

The good patches will show up in 2.6.(x+1).1, which should be sufficient. It's not the same sucker tree, but it's a sucker tree.

-Daniel
*This .sig left intentionally blank*

It's simple in bk to revert a patch. So in this instance, when the 2.6.x person pulls in the 2.6.x.y tree, they just revert the offending patch, and move on. Actually, the maintainer applying the offending patch would do it, in their tree, before they sent it to Linus. Or they do it by the normal patch way, if they don't use bk.

So, not a problem.

If it's a bugfix for the core, that met the above rules, I wouldn't mind it at all. See the sysfs core patch that made it in between 2.6.11-rc5 and 2.6.11 for such an example.

thanks,

greg k-h

The only problem I see with this -- and its a minor problem -- is that some patches that belong in the 2.6.X.Y tree go straight to you/Andrew, rather than to $sucker.

It's perfectly workable from a BK standpoint to do

	-> linux-2.6 commit
	-> cpcset into linux-2.6.X.Y [see Documentation/BK-usage/cpcset]
	-> pull from linux-2.6.X.Y into linux-2.6 [dups cset, but no
	   real code change]

but that causes dups in the BK changelog and history. Not a big deal, though, just a minor technical nit.

Jeff

That's fine with me to do. As long as someone points out to $sucker that such a patch should go into 2.6.x.y.

thanks,

greg k-h

That's the only way it _can_ work. The maintainer of 2.6.x.y shouldn't be put in a position of having to locate the patches which he needs.

Like it or not, Vojtech is still the maintainer of the input system in 2.6.x.y and he should be the primary guy who keeps an eye out for patches which needs to be applied there.

If we go overboard here, every maintainer will end up maintaining another tree of "stuff which needs to be backported to 2.6.x.y", which is probably more work than we want to do.

As long as we can keep it down to "small and really critical things" then it should work OK.

Ideally, the 2.6.x.y maintainer wouldn't need any particular kernel development skills - it's just patchmonkeying the things which maintainers send him.

Andrew, what about my suggestion of shifting left x.y of 8 bits? ;) Do we risk the magic 2.7 number to get us stuck in unstable mode for 2 years instead of 2 months? Doesn't 2.6.x.y pose the same risk but by also breaking the numbering and the stable kernel identification for no good reason? (ignoring the "2.6." part that carries no useful info anymore ;)

I think this is all a bit of a storm in a teacup, really.

2.6.x is making good progress but there have been a handful of prominent regressions which seem to be making people think that the whole process is bust. I don't believe that this has been proven yet.

There is still potential to make good improvements to the existing processes, and the main way of doing this is for the various subsystems to be a bit more disciplined. That means:

a) Have all your 2.6.n stuff ready to go when 2.6.n-1 is released.

b) Merge your 2.6.n stuff promptly - within two weeks of the 2.6.n-1 release.

c) Then, keep your sticky paws off it. New development and not-completely-trustworthy development goes into your subsystem tree and Linus's tree just gets bugfixes.

That's how it _should_ work, and I'm not sure that we're sufficiently close to it now.

Or, to put it another way, we're getting a small number of irritating regressions, mainly in device drivers which is giving the whole thing a bad rep. Is there some way in which we can fix that problem without reinventing the whole world?

    Hi,

Sorry -- what you (with the vision of a kernel developer) are seeing here surely is interesting, but it's not the point:

The point is what the *users* think. Just in case it still hasn't been made clear enough in this thread: If your user base gets the impression the development process isn't reliable any longer, you won't get your kernel tested as much as you want.

Please remember why this thread started. It was a proposal to get more users testing the rc kernels.

So I hope the latest proposal really helps making releases contain fewer surprises.

Greetings,

Jochen.

All I want is a warm bed and a kind word and unlimited power.
         -- Ashleigh Brilliant

You cannot have it both ways. Either the kernel needs testers, or it is "stable". See how these are opposites?

We don't _need_ people to test stable kernels, because they're stable. (OK, we'll pick up on a few things, but we'd pick up on them if people were testing tip-of-tree, as well).

The 2.6.x.y thing is a service to people who want 2.6.x with kinks ironed out. It's not particularly interesting or useful from a development POV, apart from its potential to attract a few people who are presently stuck on 2.4 or 2.6.crufty.

It won't help that at all. None of these proposals will increase testing of tip-of-tree. In fact the 2.6.x proposal may decrease that level of that testing, although probably not much.

There is no complete answer to all of this, because there are competing needs. It's a question of balance.

I think one of the fundamental problems is "either the kernel needs more features, or it needs stablization". You cannot have it both ways. With the current model, the kernel develops at a faster pace than testers can keep up with, and that's why you feel there aren't enough testers.

Not everyone downloads a kernel every day or even every week. Just once a while. If you roll out a kernel, you need to give some time to people to test it out. However, with the current model the kernel keeps adding features, non-bug fixes, etc, _and completely abandons the previous one and moves on_. So what's the point of testing? When I download 2.6.9, 2.6.11 might have come out. Even if bug reports do not become obosolete, they are outpaced by new bugs.

Agreed we need a balance. The problem is the 2.6 focuses too much on development than stablization. The old "stable release maintainer" model was completely abandoned. Surely that was not an exciting job, but people need it.

Hua

Giving humans a well-known point where bugfixes-only mode starts would help. Such as the -pre/-rc split does in 2.4.x.

Jeff

Yup, sure. I agree that we should be using Marcelo methodology on 2.6.x release numbering. Last time I mentioned it to Linus I didn't understand the reply ;)

I don't see a contradiction. You need testers for release candidates to make them stable. The problem is that Linux release candidates are not release candidates.

I don't see that the releases are stable. They are defined stable by proclamation.

This 2.6.x.y tree will change nothing as long as the underlying problem is not solved.

A clearly defined switch from -preX to -rc will give the avarage user a clear sign where he might jump in and test.

2.6.11-rc5 (which is -pre5 in disguise) would have been the real point for a -rc1 ...-rcX freeze and testing phase.

tglx