Aix7xxx unstable in 2.4.21-rc2? (RE: Linux 2.4.21-rc2)

although I respect your maintainer's responsible and safe position, I'd like to state that version 6.2.28 has been in the latest pre-releases for quite some time, and the reason you invoked for removing it at -rc time was the lockups people still encounter with the version present in -rc3, perhaps to a lesser extent. These lockups *SEEM* to have vanished from 6.2.33 for people who complained previously. Moreover, the lockup I encountered on my systems was fixed and demonstrated by Justin to really be a locking bug, so this was not just a "let's see how it behaves" fix.

I think there's some misunderstanding about what a release candidate is. It's an attempt to see if a particular set of code is viable as the released product. Any bugs reported against a rc that are deemed problems to the release need to be fixed, either by adding a simple and easily verifiable bug fix or by reverting the problem code.

The bksend file on http://people.freebsd.org/~gibbs/linux/SRC/
representing the requested updates is 475k compressed. There's no definition of the phrase "simple and easily verifiable bug fix" I can encompass that could be applied to a chunk of code that size.

In these circumstances, absent a simple fix for the problem, the only choice seems to be reversion and trying to get the code base stable at the beginning of the next -pre, which is the current decision.

James

That's also my point. People were reporting problems till -rc1 which included driver version 6.2.28. So Marcelo reverted to 6.2.8 for -rc2 (74500 lines of code reverted, not including doc nor aic79xx which was kept). Then, people were still reporting problems with -rc2 which they claim are fixed by updating to last driver updates, which were 16000 lines forward from -rc1, so less than one fourth of what Marcelo accepted to change from -rc1 to -rc2. Although I find this big, it's less than the change in fusion/mpt* that has gone from -rc2 to -rc3. So I think it's not a matter of size here.

Most of this is a 1 MB Changelog, files going back to their original place (Marcelo moved aic79xx to a proper directory to keep it), documentation, and initialization code which was exploded in more little functions, then bug fixes.

I wish Justin would have proposed a little patch to fix only the locking bugs in -rc1, but honnestly, why should he fix only these bugs when he knows about others that must be fixed too ? I can understand he gives up. -rc is for bug fixes, and his bug fixes are reverted !

As I said, I really hope that we'll have a quick 2.4.22 with bug fixes taken as a priority. The current pre-releases are as frequent and as big as what used to be full releases in the past.

Regards,
Willy

It is a question of size and provenance. Alan Cox descriped the mpt fusion update as "assorted small fixes" and deletions exceed additions in the patch set by 40%. It's also about user base: aic7xxx is by far the most widely used SCSI chip, I'm not sure how many 2.4 fusion users there are but I speculate its probably orders of magnitude fewer.

The argument isn't about size, it's about safety. No company that wants to stay in business accepts code into release stabilisation unless it's clearly justifiable. Trying to buck the system by including five features plus one critical bug fix is one of the oldest tricks in the Software Engineers book---do this and you get hauled before the release committee whose job will be to pare the addition back to just the bug fix (and send you away with a flea in your ear to boot).

Marcelo reacted exactly as the release committee would at Adaptec:
either provide the bug fix for assessment or we'll push it out into the next release. This is industry standard practice, so I don't see any problem.

I agree. One of the necessary things for a fast release is a good release manager (and thus one prepared to make unpopular decisions--and ones you don't necessarily agree with).

James

Just for clarification. Marcelo never asked me for a fix. The only mail I received from him was an informational message indicating that the code was being backed out. If I had been provided an opportunity to fix the problem, I would have. Considering that the fix has been available long before RC2 was cut (May 1st.), it's not hard to see that getting a proper fix required nothing more than just upgrading the driver or contacting its maintainer to get a paired down fix.

--
Justin

The kernel, as you have been told several times before, follows a push model, not a pull one. Just looking after SCSI, I don't have time to go around asking all the driver writers for updates; likewise Marcelo really doesn't have the time to do this for everything in the 2.4 kernel.

Every maintained piece of the kernel has a listed maintainer to whom the bug reports are supposed to go. The expectation is that these maintainers will see the bug reports and pro-actively provide fixes before they become release issues. The maintainers also do enhancements, *but* these enhancements should follow the proper release cycle (i.e. in at the early -pre stage).

Could you please get with the program? The bug fix vs enhancement issue hasn't previously mattered that much for 2.5, but I anticipate we'll be following a similar model when 2.6 is released.

James

That's nice when someone is working for you. In this case I believe there were multiple bugs, and the viable choices were to ship with the old known bugs or do testing on the new version until it is accepted as tested. Given that the release schedule seems to be measures in generations these days, shipping a driver known to be bad is probably worse than doing one more rc and asking people to beat hell out of the driver.

I bet your release schedule is more frequent, too. There comes a time when doing a whole new QA is better than trying to be sure a fix works without doing full QA anyway. As in both faster and more likely to result in a correct result.

bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

I really think 2.4.x development is becoming almost non-existent lately.

It's 5 or 6 days of silence, nothing happening at all, then a flurry of 10 or 20 checkins and a -rc or -pre release.

If Conectiva needs to task Marcelo to so much work that he can only really put 1 or 2 days a week into 2.4.x, this needs be rethought at either one end (Conectiva finding a way to give him more 2.4.x time) or another (Marcelo splits up the work with someone else or we simply find another 2.4.x maintainer).

I really want something more -ac paced although that may be too extreme for some people. :-)

David S. Miller <[email protected]>

On Monday 26 May 2003 06:25, David S. Miller wrote:

Hi David,

_full_ ack!

Just in case if there will ever be a new election ...

... */ME stretches his hands up*

ciao, Marc

I said publically that this situation will change in 2.4.22-pre.

If you think the situation didnt got better during that time, I'm pleased to giveup 2.4.x maintenance.

Is that fine for you?

From: Marcelo Tosatti <[email protected]>
Date: Mon, 26 May 2003 14:58:35 -0300 (BRT)

I said publically that this situation will change in 2.4.22-pre.

Great.

If you think the situation didnt got better during that time, I'm pleased to giveup 2.4.x maintenance.

Is that fine for you?

I have no desire to take you away from your 2.4.x duties. In fact, I as much as anyone else, want you to work as 2.4.x maintainer.

Therefore, you should not think twice about asking people such as myself for help if you want it.

I fully expect you to resolve the situation. :-)

Splitting up the work with someone is senseless, IMO. As I said before, 2.4.22-pre should be better in that aspect. In case it doesnt, I'm giving up 2.4.x maintenance.

Marcelo,

Reading your words, I have the sad feeling that you take no interest in doing this job, and that you do it only because people ask you to. What a shame :-(

Although it sure can be annoying, aren't you proud of each new release ? Usually, kernel integrators are proud of their new kernels when they get something rock solid ! People like Con Kolivas, J.A.Magallon, Marc-Christian Pettersen are often proud to announce us the few bits they changed in their tree and which stabilized it. It seems you only do this as an obligation, which is sad, really.

I understand that maintaining the stable tree, the one which MUST NOT FAIL, may be frustrating, not being as excitant as playing with kernels which try to get the most of every piece of hardware, as others do (although nobody prevents you from developing your own Wolk). But you don't seem to share much about your feelings, ideas or doubts with others. Alan, for example, exchanges a lot with people testing his kernels, suggesting a few tweaks to help them workaround their problems, and integrating the tweak in the next release if it succeeds. This fast feedback allows him to release more often. It also makes his work more intersting for others. People often prefer "here is -rcxx-acxx, which my EPIA now fully supports" to "here is -rcxx, please test it extensively".

Perhaps you don't feel assurance when you have to blindly integrate hundreds of patches from people you don't always trust, and that may explain why you suddenly announce a new pre-release and keep silent, hoping for patch authors to reply to questions if any ? If this is the case, jump into the train, there's no risk, except of being caught by Rik's troll-o-meter, or having Viro or hch insult you ! And then ? What's the matter ? Every one has his turn. I even risk it with this OT mail. When you started with 2.4.16, you said that you were afraid you lacked some skills, but you proved to be very capable, because the kernel has moved since, and 2.4.21 should be far more stable than 2.4.16 !

This mail is not intended to give you any lesson, but to give a feedback from a Linux 2.4 user who, as many others, feels more and more forgotten by his maintainer. Unfortunately, what David wrote is what many people currently think of 2.4 :-( You threatened to give up, but that would be bad for your image and for Linux. Giving up means no maintainer for a certain amount of time, then a self-proclamed new maintainer (or worse, several ones with a tree fork). Being replaced is cleaner, since you do the job until the new maintainer is ready to start.

If you don't have enough time to do everything, send a source quench, or apply one of David's proposed solutions : ask for some help so that only subsystems maintainers feed you as some already do (eg: David, Jeff, Greg), or ask for a pure replacement. If you're bored, that I could understand, because having to deal with arrogant and sometimes even selfish users is not always pleasant, ask for a replacement. If you're fed up with patches that you don't understand, reject them LOUDLY asking for more documentation. And if you plan to have a rest for two weeks, say it, so that people don't send you patches that will be lost in a full mailbox at your return. Yes, this may be what Linus did before you, when people already complained. But there should be a middle line between how he managed his kernel and how you manage it, and BTW, Linus clearly stated that maintaining 2.4 bored him.

I've just read your mail about -rc[45]. I'm happy we start to see the light at the end of the 2.4.21 tunnel. As others people, I'm now impatient to both 2.4.21 and 2.4.22-pre1. BTW, as discussed perhaps a year or two ago, you could have a preview of 2.4.22-pre1 in parallel with 2.4.21-rc, to feed the impatients, although that may be double work, which you don't necessarily need at the moment.

And remember, please communicate, communicate, communicate. You and only you know what problem you have at a given time. If you don't communicate, people always imagine the worst.

Regards,
Willy

What I said is that if people think I'm not maintaining 2.4.x (quoting Davem, "I really think 2.4.x development is becoming almost non-existent lately.) in a acceptable way, the work should be done by someone else. I WANT to keep maintaining 2.4, but only if people are happy with that. Do you understand ?

I dont understand what you mean.

Yes, I understand. I think it's good that people have complained because now you know what they expect from you, so it's up to you to make them happy :)

John correctly replied for me. Several -pre didn't even compile for most of us, but you let them as they were during weeks. When this happens, only a few people who have the time to follow LKML and grab patches can try these pre-releases. When that happens, please do apply the trivial fixes and send one more just after, even the same day so that people can try them. Alan often does this, and nobody has ever complained about him for releasing 2 kernels the same day, one with a big bug and the next one without.

I already had collegues saying to me "Yesterday, I've tried to compile 2.4.21-preXX, but it doesn't compile". It's always sad to say to them "wait 3 weeks, hoping for the next one to fix it and not to add new bugs !". I'd really prefer to reply "you idiot, return to kernel.org and get the next one or send a patch, Marcelo wouldn't let his kernel in this state".

Regards,
Willy

Please don't give up!

This e-mail is meant as public encouragement and big THANKS for you. Every time I hit a bug in 2.4 and sent you a patch, it was applied after a short time (few days). So far, 2.4 has been working perfectly for me.

There are many users and developers who are happy with the 2.4 kernel and with you as a maintainer, they just don't say it. Something that works is often forgotten, once it breaks or doesn't work as expected, people start crying loudly.

Thank you for maintaining a stable tree,

Carl-Daniel

On Tuesday 27 May 2003 00:16, Carl-Daniel Hailfinger wrote:

Hi Carl-Daniel,

Sorry, but are you kidding?

ciao, Marc

No. It works for me(TM). If it didn't work for you during the 2.4.21 release cycle, my apologies. Marcelo stated he wants to take the patches you complained about for 2.4.22-pre1. That should also fix your issues.

Regards,
Carl-Daniel

Its up to Marcelo. If he wants to hand it on to me now 2.2 is basically a one day a month job he can, or to someone else.

One thing I will say however - I'd have done the *same* thing as Marcelo with aic7xxx during -rc which is to defer it. A maintainer gets a continual stream of "completely tested utterly reliable fixes everything" drivers, none of which prove to be so. The simple truth is that when you give something to 10,000 users instead of 20 something breaks. Its not that authors suck its just another testimony to the fact computer programming is still firmly at the alchemy not the chemistry end of its history.

If the driver works well fine, but maintainers don't have the ability to see into the future either.

Alan

I think you would at least have forwarded problem reports to Justin, expecting him to look into the problem first.

I don't reproach Marcelo of not including the last aic7xxx driver in mainline, but of reverting an enormous amount of code at the last minute without prior asking the maintainer if he has an idea about the problem people encounter. Of course, if he hasn't, the driver has to be removed, but he gave him no chance to fix it, nor any detail about who had problems and what type of problems they had. And reverting to what it was in 2.4.20 is not safer than trying to fix, since other code touched in 2.4.21 may bring side effects (APIC ?) which might explain why it doesn't work for some people.

and that computer makers can't read the specs !

It reminds me that I often worry when one of my programs runs right on the first compilation, because it doesn't give me the opportunity of puting my nose into sensible places where I could find obvious bugs :-)

Regards,
Willy

Justin used to say "use my latest driver" when people reported problems. Read lkml.

Its great if Justins new driver fixes the problems, but as I told him I thought it was too late for it to be included. Thats my bad, too, because if I had included it early in 2.4.21pre it could be in now.

Justin could well have fixed the problems in the current driver instead answering "use my latest driver", couldnt he?

I agree, but honnestly, when a driver author remembers about hundreds of bugs fixed between the version the user complains about and the last one, it's difficult to point the real problem, and then to say to this people "just apply this little fix for this particular bug, and cross your fingers not to be caught by the 99 others".

I think he could have tried to fix the most obvious ones and then say "Marcelo, my old driver is plain buggy, here are a few fixes for the complainers, but it's about to explode, please plan on a full upgrade soon".

Anyway, what is done is done. It's nonsense always talking about the past, we'd all better spend our time fixing bugs.

Regards,
Willy

As the one who started this discussion... I'd simply like to quote this and say:

_FULL_ ack!

[ ... ]

I also changed the whole server (the one which had the aix7xxx problems) in the meantime... Changed the Adaptec 2940, now there is a Adaptec 29160. I switched from a Dual-P3 to a P4. And well, the interessting part, I switched from 2.4.20(-XX) to 2.4.19. EVERYTHING runs faster and stable now!

Best regards,
Oliver

On Tuesday 27 May 2003 10:38, Oliver Pitzeier wrote:

Hi Oliver,

try 2.4.18 and you'll maybe s/faster/"fuck damn fast as speed of light"/

;)

ciao, Marc

Thanks Marc-Christian!

This is what I'll try on my development server. :-) And then I may switch at the production system as well. :-)

Best regards,
Oliver

Oliver,

Does 2.4.21-rc5 work for you?

Hi Marcello!

Marcello Tosatti wrote:
[ ... ]

Blame me for this! I was very busy the last few weeks, that's why I answer this mail soooooo late. Because I have the machine now at home and no longer in a production environment, I'm able to test everything...

(FYI. The P4 machine still runs stable. :-) ).

I'll try the latest kernel 2.4.22-pre1 on the dual-machine and tell you if it runs stable or not. I believe there are not too much people who have a Dual-PIII with an Adaptec controller...(!?)

However...

Keep on going!

Best regards,
Oliver