stable? quality assurance?
Sorry, forgot Cc again.
Am Sonntag 11 Juli 2010 schrieb Willy Tarreau:
Quoted 1 line
Hi Martin,
Hi Willy, hi everyone else reading this,
Quoted 6 lines · Martin Steigerwald
Quoted 2 lines
I hope that someone answers who actually can take some critique. From the current replies I perceive a lack of that ability.
well, I'll try to do then :-)
Quoted 8 lines
There were some threads in the past about kernel releases quality, where Linus explained why it could not be completely black or white.
Among the things he explained, I remember that one of primary concern was the inability to slow down development. I mean, if he waits 2 more weeks for things to stabilize, then there will be two more weeks of crap^H^H^H^Hdevelopment merged in next merge window, so in fact this will just shift dates and not quality.
During bisecting [Bug 16376] random - possibly Radeon DRM KMS related freezes, which goes very slowly due to having lots of unbootable kernels with an ext4 / readahead related backtrace during boot, I had an idea:
I think main problem is that the current development process does not give time for quality work and bug fixing. As I understand it currently its just a constant development of new features with bug fixing and quality work having to be done beneath that development:
- before 2.6.36 is released developers aim at developing new stuff for 2.6.37.
- after 2.6.36 is released developers aim at getting as much stuff into 2.6.37 and then after two weeks at developing new features for 2.6.38.
This process does not take bug fixing into account at all, cause after the merge window has closing, developers hurry to get the stuff ready for the next window.
In that model extending the freeze period after rc1 doesn't help at all, cause as you say more "crap^H^H^H^Hdevelopment" gets collected for the next kernel.
But is that a *given* that no one actually has any influence to? Is collecting changes for next kernel like rain that either pours down or not
- usually pours down in this case like in August in Germany ;)? Who feeds Linus with new stuff during the merge window? From what I understand of the Linux development process its mainly the subsystem maintainers and Andrew Morton.
What if those people stop collecting new stuff for Linus except bugfixes about two or three weeks before the next kernel is relased? This would give the subsystem trees and the mm tree some time to stabilize a bit, so that Linus gets more quality stuff in the first time. And more importantly, since developers know that subsystem maintainers and Andrew only collect bugfixes 2-3 weeks before the release of a stable kernel, they can as well spend some time on quality work.
Of course, developers can still decide: Well if 2.6.37 work is closed already and continue developing for 2.6.38 even earlier, but I still think this would help to slow things down a bit prior to the critical phase before releasing a stable kernel. Cause when I know my subsystem maintainer or Andrew won't be taking my stuff anyway, before the release kernel is released, I can take a little time for other things.
The main idea here is to have a two-staged freeze process and to distribute the "I am only taking bug fixes" work to more people than Linus.
For this to work properly, I think at the time of the release of the stable kernel subsystem maintainers and Andrew should branch their trees. For example when 2.6.36 is released:
- tree
=> 2.6.36-stable-tree
=> tree, where 2.6.37 stuff will be going in
Thus when subsystem maintainers take new stuff during the merge window, it will be for the next kernel release already, not for the current one. Except bugfix work. Whereas I think the criteria for bug fix work should not be that strict than for the stable patches Greg collects.
Thus it needs to be clear: No new stuff for next kernel already two weeks prior to release the current stable kernel.
I think, this could help. Its a bit like the two-staged development process of Debian, but with the freeze period for "unstable" being a fixed time interval of about 2 weeks instead of RC=0 for stable ;). Its a bit of a formal shift of attention to the stable kernel about 2 weeks before its release. Developers might find creative ways to circumvent it, or they understand, that this process serves a purpose of improving kernel quality.
When you think these two weeks cannot be squeezed into the three-monthly development cycle, a four-monthly development cycle might do. But actually I don't see why these two weeks could not be made to fit in there.
Installing and testing next kernel after yet another mail to this thread,
Signature · 2 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7
Hi Martin,
On Sat, Sep 04, 2010 at 06:42:19PM +0200, Martin Steigerwald wrote:
(...)
Quoted 11 lines
The main idea here is to have a two-staged freeze process and to distribute the "I am only taking bug fixes" work to more people than Linus.
For this to work properly, I think at the time of the release of the stable kernel subsystem maintainers and Andrew should branch their trees. For example when 2.6.36 is released:
- tree
=> 2.6.36-stable-tree
=> tree, where 2.6.37 stuff will be going in
Quoted 7 lines
Thus when subsystem maintainers take new stuff during the merge window, it will be for the next kernel release already, not for the current one. Except bugfix work. Whereas I think the criteria for bug fix work should not be that strict than for the stable patches Greg collects.
Thus it needs to be clear: No new stuff for next kernel already two weeks prior to release the current stable kernel.
While I respect your beliefs on this matter (they once were mine too), I now realized I was wrong for several reasons :
- most developers want to create. They (generally) test what they create, they believe it's flawless because it works for them. No need for more testing, go on with new features ; if you refuse to merge their new work for some time, they work on their own tree and push you more work at once next time.
- developers need real world use cases. That means more testers. Developers are bad testers because they don't trigger the unexpected use cases. And how do you get good testers ? by motivating end users to test your code. Most testers will only test a new kernel to get a new feature. If it works for them, no need to push the tests further. So that means that the first RCs are the most tested, and that the later ones are the least tested. Thus at one point you can't hope to get bug reports anymore. When you see an -rc7 or -rc8, you think "hey, -rc4 was OK, let's wait for -final and install it".
- people concerned by stability don't test every release. They test when they can, precisely because they can't impact production. So they don't contribute bug reports in time. And as the 2.4 maintainer, I'm well aware of that, because when I break something, I only know about it 3-4 months later.
For this reason, I think the release rhythm can't much be changed. I think that trying to evaluate and publish quality per developer or maintainer can have a better effect because everyone in the commit chain is responsible. But even doing that is hard because some changes touch everything and it's not obvious to say that Mr X or Y has done some crap.
In my opinion, reporting bugs is the most effective way of improving quality. If you report 10 bugs in a week on the same driver, there are chances that at one point this driver's author will want to take some time to audit his code and find other bugs before you next point your finger at him/her. As you see, the goal is not just to report bugs to get them fixed, but to educate bug authors.
I can tell you that I am an author of quite a number of bugs in another project (haproxy), and I absolutely hate it when a bug is detected in production (especially given the product's goal), to the point that the code is generally reworked 2, 3, 5, 10 times before being committed. Of course it is still not enough to catch all bugs, but since the product has got a widely accepted reputation of being rock solid, I think it works quite well afterall.
Last, developers must not betray their users' trust. When they're not certain of their code, this must be advertised (this is often the case but not always). That helps a lot end users select only reliable features and experience more stability.
Regards,
Willy
Hi again,
Am Samstag 04 September 2010 schrieb Willy Tarreau:
Quoted 33 lines · Martin Steigerwald
(...)
Quoted 13 lines
The main idea here is to have a two-staged freeze process and to distribute the "I am only taking bug fixes" work to more people than Linus.
For this to work properly, I think at the time of the release of the stable kernel subsystem maintainers and Andrew should branch their trees. For example when 2.6.36 is released:
- tree
=> 2.6.36-stable-tree
=> tree, where 2.6.37 stuff will be going inQuoted 8 lines
Thus when subsystem maintainers take new stuff during the merge window, it will be for the next kernel release already, not for the current one. Except bugfix work. Whereas I think the criteria for bug fix work should not be that strict than for the stable patches Greg collects.
Thus it needs to be clear: No new stuff for next kernel already two weeks prior to release the current stable kernel.
While I respect your beliefs on this matter (they once were mine too), I now realized I was wrong for several reasons :
- most developers want to create. They (generally) test what they
create, they believe it's flawless because it works for them. No need for more testing, go on with new features ; if you refuse to merge their new work for some time, they work on their own tree and push you more work at once next time.
Quoted 9 lines
- developers need real world use cases. That means more testers.
Developers are bad testers because they don't trigger the unexpected use cases. And how do you get good testers ? by motivating end users to test your code. Most testers will only test a new kernel to get a new feature. If it works for them, no need to push the tests further. So that means that the first RCs are the most tested, and that the later ones are the least tested. Thus at one point you can't hope to get bug reports anymore. When you see an -rc7 or -rc8, you think "hey, -rc4 was OK, let's wait for -final and install it".
That fits perfectly well. If the first rcs are nicely testing, then ideally all major issues should be done, when rc7 or rc8 are reached. And thus time can be spent on fixing the major remaining open regression. I guess those who reported these regression are interested in testing a fix.
For me features have been number one reason to upgrade kernels as well, but then its not a yes or no decision, but more a tuning on how much new feature stuff each stable kernel release should have and a way to put a little bit more attention to making a stable kernel release stable.
Quoted 5 lines
- people concerned by stability don't test every release. They test
when they can, precisely because they can't impact production. So they don't contribute bug reports in time. And as the 2.4 maintainer, I'm well aware of that, because when I break something, I only know about it 3-4 months later.
How does this affect my suggestion above? If as you say the first rcs are tested better and if as I assume those who reported regressions have an interest in testing their fixes, I think this can work out nicely.
Aside from that, I am not sure whether most people step in with rc1 or rc2 already. When I tested rc kernels - there have been some times - I usually waited to rc3 or rc4 so I could be somewhat confident that really major issues are fixed already.
Quoted 1 line
For this reason, I think the release rhythm can't much be changed.
I still object that for above given reasons. And cause I think that if something does not work out perfectly it still can be improved. But I am interested in your other suggestions as well, cause maybe its not so much the release process but something else the issue here:
Quoted 5 lines
I think that trying to evaluate and publish quality per developer or maintainer can have a better effect because everyone in the commit chain is responsible. But even doing that is hard because some changes touch everything and it's not obvious to say that Mr X or Y has done some crap.
And who judges on what is crap? Build failures could be tracked automatically. Partly maybe even performance regression as the automated tests from Phoronix show. Well boot failures or freezes are even more important. But then, you are probably not judging the quality of the work of the developer but the difficulty of the area he works on.
Nix pointed out that programming ATI Radeon cards can be quite challenging. And I do have lots of respect for the Radeon KMS related work. So I think it would be unfair to point at one of the Radeon KMS developers and say to him "you did crap" for example.
I think crap does happen and am more concerned about how to handle it when it does.
Quoted 6 lines
In my opinion, reporting bugs is the most effective way of improving quality. If you report 10 bugs in a week on the same driver, there are chances that at one point this driver's author will want to take some time to audit his code and find other bugs before you next point your finger at him/her. As you see, the goal is not just to report bugs to get them fixed, but to educate bug authors.
Okay, my contribution then: I report bugs. I reported 4-5 kernels bugs in the last time. I reported some before, but only occassionally. I didn't face that many bugs prior to 2.6.34 which contributed to my admittedly very subjective impression that kernel quality has lowered.
Quoted 7 lines
I can tell you that I am an author of quite a number of bugs in another project (haproxy), and I absolutely hate it when a bug is detected in production (especially given the product's goal), to the point that the code is generally reworked 2, 3, 5, 10 times before being committed. Of course it is still not enough to catch all bugs, but since the product has got a widely accepted reputation of being rock solid, I think it works quite well afterall.
Interesting project, I am implementing a highly available active/passive loadbalancer cluster using Corosync, Pacemaker and the IPVS frontend Ldirectord at the moment currently at work.
Quoted 4 lines
Last, developers must not betray their users' trust. When they're not certain of their code, this must be advertised (this is often the case but not always). That helps a lot end users select only reliable features and experience more stability.
Well for me a balance must be met: A kernel has to work good enough for me to use it regularily. And currently 2.6.34 upto 2.6.36-rc2 on my ThinkPad T42 simply do not fulfil that criterium. What annoys me most: Radeon KMS already works perfectly stable on 2.6.33 for me. So the issue was not in the initial version of Radeon KMS. It has been introduced afterwards. Thus a supposedly more matured and stable version of it is working less stable for me.
2.6.33-tp42-01231-g11b897c has been good to me so far. I am glad it had not frozen yet. I better press send now.
Ciao,
Signature · 2 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7
Quoted 7 lines · Martin Steigerwald
Quoted 3 lines
Thus at one point you can't hope to get bug reports anymore. When you see an -rc7 or -rc8, you think "hey, -rc4 was OK, let's wait for -final and install it".
That fits perfectly well. If the first rcs are nicely testing, then ideally all major issues should be done, when rc7 or rc8 are reached. And thus time can be spent on fixing the major remaining open regression.
OK I see that you're talking about *open* regressions. I thought you were talking about bugs in general. I think (but that's my own feeling) that as soon as the cause of a regression is narrowed down enough to identify the commit that caused it, it gets quickly fixed (though I have no numbers on the subject). But when someone says "I was doing this or that when my kernel froze", it can be anything. Drivers are different because they impact less people than the core. However the developers don't always have access to the hardware combination causing a reproducible error case.
Quoted 2 lines
I guess
those who reported these regression are interested in testing a fix.
I really think that there's good interactivity when the bug is spotted. The hard part is the one before.
Quoted 9 lines
Quoted 5 lines
- people concerned by stability don't test every release. They test
when they can, precisely because they can't impact production. So they don't contribute bug reports in time. And as the 2.4 maintainer, I'm well aware of that, because when I break something, I only know about it 3-4 months later.How does this affect my suggestion above? If as you say the first rcs are tested better and if as I assume those who reported regressions have an interest in testing their fixes, I think this can work out nicely.
But you can't have developer sit on their code for 4 months waiting for bug reports to come in. And if you're talking about open bugs only, each one of them will think the issue is probably in the other one's code. Common problem of development teams.
Quoted 4 lines
Aside from that, I am not sure whether most people step in with rc1 or rc2 already. When I tested rc kernels - there have been some times - I usually waited to rc3 or rc4 so I could be somewhat confident that really major issues are fixed already.
I think that people waiting for a specific feature will immediately jump on rc1 or rc2. People who are curious about what was stuffed in the new kernel will likely wait for rc3/4, hoping to get something they can run a day long.
Quoted 11 lines
Quoted 5 lines
I think that trying to evaluate and publish quality per developer or maintainer can have a better effect because everyone in the commit chain is responsible. But even doing that is hard because some changes touch everything and it's not obvious to say that Mr X or Y has done some crap.
And who judges on what is crap? Build failures could be tracked automatically. Partly maybe even performance regression as the automated tests from Phoronix show. Well boot failures or freezes are even more important. But then, you are probably not judging the quality of the work of the developer but the difficulty of the area he works on.
I agree with you in general on this point, which makes the issue even harder to solve. However, some bugs are definitely caused by crap (look for Al Viro's occasional audit reports, missing locks and thinks like this should not get merged). Every developer starts inexperienced, and may humbly ask for help.
Quoted 4 lines
Nix pointed out that programming ATI Radeon cards can be quite challenging. And I do have lots of respect for the Radeon KMS related work. So I think it would be unfair to point at one of the Radeon KMS developers and say to him "you did crap" for example.
100% agreed. It's the same in my opinion for every piece of code that relies on configs that are hard to obtain. For instance, if a driver breaks on configs with more than 256 CPUs or 1 TB of RAM, we can't necessarily blame the author for not being able to test his code in such situations.
Quoted 2 lines
I think crap does happen and am more concerned about how to handle it when it does.
OK, but when an unusual config is required, sometimes the author cannot help getting his code fixed.
Quoted 2 lines
Okay, my contribution then: I report bugs. I reported 4-5 kernels bugs in the last time. I reported some before, but only occassionally.
That's really nice.
Quoted 3 lines
I didn't
face that many bugs prior to 2.6.34 which contributed to my admittedly very subjective impression that kernel quality has lowered.
Possible, but it's also possible that the new bugs affect an area that you're using much more than the ones affected by bugs in older versions. It's also possible that you became better at noticing bugs.
Quoted 7 lines
Quoted 4 lines
Last, developers must not betray their users' trust. When they're not certain of their code, this must be advertised (this is often the case but not always). That helps a lot end users select only reliable features and experience more stability.
Well for me a balance must be met: A kernel has to work good enough for me to use it regularily.
That's what everyone looks for, and obviously the threshold is not the same for everyone, and the bugs don't affect everyone. You see, while 2.4 is in feature freeze and thought to be very stable by its users (and I occasionally encounter systems with 2 years of uptime under permanent stress), i would not be surprized that some people consider it still not stable enough for their usages. It's just a matter of personal taste.
Quoted 6 lines
And currently 2.6.34 upto 2.6.36-rc2 on my ThinkPad
T42 simply do not fulfil that criterium. What annoys me most: Radeon KMS already works perfectly stable on 2.6.33 for me. So the issue was not in the initial version of Radeon KMS. It has been introduced afterwards. Thus a supposedly more matured and stable version of it is working less stable for me.
That's where you're on the wrong side. 2.6.34 is not supposed to be a more matured and stable version than 2.6.33. It's supposed to be a more *advanced* version. Some issues were fixed, some features were added, some improvements were performed and many bugs were added in that whole process. There's a rule to follow concerning kernel upgrades in my opinion : you should only upgrade for at least one of these 4 reasons :
- test new kernels
- get new features
- fix a known bug
- remain on a supported version
It's very likely that you'll regularly switch between newer and older kernels to switch between the first 2 and the last 2 reasons. But people who upgrade just to be on the edge and who don't even contribute bug reports back are just looking for trouble in my opinion.
Regards,
Willy