XFS problem in 2.6.32
Hi List,
i've massive problems with a machine using 2.6.32 kernel and XFS: I've desribed the problem here:
http://www.spinics.net/lists/xfs/msg04984.html
The reply on the XFS mailinglist was:
http://www.spinics.net/lists/xfs/msg04994.html
So who keeps track on which patches needs to get backported or not? And who will backport XFS fixes back to 2.6.32?
Thanks!
Please CC me i'm NOT on list.
Regards,
Stefan
Quoted 9 lines · Stefan Priebe - Profihost AG
Hi List,
i've massive problems with a machine using 2.6.32 kernel and XFS: I've desribed the problem here:
http://www.spinics.net/lists/xfs/msg04984.htmlThe reply on the XFS mailinglist was:
http://www.spinics.net/lists/xfs/msg04994.htmlSo who keeps track on which patches needs to get backported or not? And who will backport XFS fixes back to 2.6.32?
An interested kernel developer. They can become interested because they personally have the time or interest, or because someone pays them to become interested. Support of the stable kernel series is not something that happens magically, or which is funded by a charity, you know. That's why some companies pay $$$ for a supported distribution kernel.
-- Ted
Quoted 3 lines
Quoted 1 line
So who keeps track on which patches needs to get backported or not? And who will backport XFS fixes back to 2.6.32?
An interested kernel developer. They can become interested because they personally have the time or interest, or because someone pays them to become interested. Support of the stable kernel series is not something that happens magically, or which is funded by a charity, you know. That's why some companies pay $$$ for a supported distribution kernel.
OK so my thought was totally wrong. I thought the longterm stable releases will still get bugfixed by SGI or whoever wrote the stuff. Sorry for that then. But what is then the idea of a longterm stable?
Stefan
Quoted 6 lines · Stefan Priebe - Profihost AG
Date: Tue, 07 Jun 2011 15:49:36 +0200
Quoted 3 lines
Quoted 1 line
So who keeps track on which patches needs to get backported or not? And who will backport XFS fixes back to 2.6.32?
An interested kernel developer. They can become interested because they personally have the time or interest, or because someone pays them to become interested. Support of the stable kernel series is not something that happens magically, or which is funded by a charity, you know. That's why some companies pay $$$ for a supported distribution kernel.
OK so my thought was totally wrong. I thought the longterm stable releases will still get bugfixed by SGI or whoever wrote the stuff. Sorry for that then. But what is then the idea of a longterm stable?
development and bugfixes are done on the latest kernel, if the problem is known to affect old kernels the developers sometimes remember to notify the -stable list that this patch is important and needs to be applied to older kernels.
whoever the maintainer of the -stable/-longterm tree is (be it an individual or a team employeed by some comapny) then looks at the patch and considers backporting it (if it's too hard, or to intrusive, they may decide not to).
the idea of the lonterm kernels is that organizations need to maintain a kernel for a long time due to commitments that they have made (Debian doesn't want to change the kernel it ships in a stable version, RedHat doesn't want to change the kernel version in a RHEL release, etc), and so they publicly announce this so that anyone else wanting to use the same kernel version can share in the work (and therefor everyone can benifit from each other's work)
David Lang
Quoted 4 lines
whoever the maintainer of the -stable/-longterm tree is (be it an individual or a team employeed by some comapny) then looks at the patch and considers backporting it (if it's too hard, or to intrusive, they may decide not to).
So i have to contact Greg Kohan from SUSE directly?
Quoted 7 lines
the idea of the lonterm kernels is that organizations need to maintain a kernel for a long time due to commitments that they have made (Debian doesn't want to change the kernel it ships in a stable version, RedHat doesn't want to change the kernel version in a RHEL release, etc), and so they publicly announce this so that anyone else wanting to use the same kernel version can share in the work (and therefor everyone can benifit from each other's work)
That was what i thoght. So a bug like this should get fixed right? Otherwise this makes no sense. Sadly Redhat has ported the fix back in his RHEL 6 2.6.32 kernel but they haven't send the patch to stable / vanilla team.
Stefan
Quoted 5 lines · Stefan Priebe - Profihost AG
Quoted 4 lines
whoever the maintainer of the -stable/-longterm tree is (be it an individual or a team employeed by some comapny) then looks at the patch and considers backporting it (if it's too hard, or to intrusive, they may decide not to).
So i have to contact Greg Kohan from SUSE directly?
if he's the maintainer of the longterm tree, then yes, you would contact him or the stable mailing list.
Quoted 8 lines
Quoted 7 lines
the idea of the lonterm kernels is that organizations need to maintain a kernel for a long time due to commitments that they have made (Debian doesn't want to change the kernel it ships in a stable version, RedHat doesn't want to change the kernel version in a RHEL release, etc), and so they publicly announce this so that anyone else wanting to use the same kernel version can share in the work (and therefor everyone can benifit from each other's work)
That was what i thoght. So a bug like this should get fixed right? Otherwise this makes no sense. Sadly Redhat has ported the fix back in his RHEL 6 2.6.32 kernel but they haven't send the patch to stable / vanilla team.
it depends on how intrusive/disruptive the patch is.
David Lang
Quoted 5 lines · Stefan Priebe - Profihost AG
Quoted 4 lines
whoever the maintainer of the -stable/-longterm tree is (be it an individual or a team employeed by some comapny) then looks at the patch and considers backporting it (if it's too hard, or to intrusive, they may decide not to).
So i have to contact Greg Kohan from SUSE directly?
Only if you can identity a specific patch you want backported to 2.6.32.
It's not the responsibility of the long-term stable tree maintainer to go looking through potentially tens of thousands of commits to find the one that should be backported. And if the code has changed too much since .2.6.32, and requires detailed reworking before the patch can get integrated into 2.6.32, then a subsystem developer will have to do that work.
Quoted 9 lines
Quoted 8 lines
the idea of the lonterm kernels is that organizations need to maintain a kernel for a long time due to commitments that they have made (Debian doesn't want to change the kernel it ships in a stable version, RedHat doesn't want to change the kernel version in a RHEL release, etc), and so they publicly announce this so that anyone else wanting to use the same kernel version can share in the work (and therefor everyone can benifit from each other's work)
That was what i thoght. So a bug like this should get fixed right? Otherwise this makes no sense. Sadly Redhat has ported the fix back in his RHEL 6 2.6.32 kernel but they haven't send the patch to stable / vanilla team.
Not all distributions will participate in the maintenance stable tree. Red Hat for example is probably worried about people (specifically, Oracle) taking their kernel expertise "for free" and bidding it against them. So it doesn't surprise me that they aren't submitting patches to the stable tree. After all, they would like you to purchase a support contract if you want to get high quality, supported kernel. Why should they give that work away for free? After all, their salaried developers have to get paid somehow. Others will contribute work in the hopes that other people will also contribute fixes back, but of course nothing forces Red Hat to do this.
The thing that you have to understand that this is all a volunteer effort, and a few of your messages sound like you are expecting people to do the work for you. It doesn't work that way. If you ask nicely, maybe one of the XFS developers will help you. But please don't expect free support. That will just annoy people.
-- Ted
Hi Ted,
Quoted 4 lines
Not all distributions will participate in the maintenance stable tree. Red Hat for example is probably worried about people (specifically, Oracle) taking their kernel expertise "for free" and bidding it against them. So it doesn't surprise me that they aren't submitting patches to the stable tree. After all, they would like you to purchase a support contract if you want to get high quality, supported kernel. Why should they give that work away for free? After all, their salaried developers have to get paid somehow. Others will contribute work in the hopes that other people will also contribute fixes back, but of course nothing forces Red Hat to do this.
The thing that you have to understand that this is all a volunteer effort, and a few of your messages sound like you are expecting people to do the work for you. It doesn't work that way. If you ask nicely, maybe one of the XFS developers will help you. But please don't expect free support. That will just annoy people.
i'm sorry this was never my intention. I'm no native speaker so it might be that my word selection was not perfect.
I quite understand that alle people are doing this for free or got paid by a company who have their own interests.
I just thought it would be important to make it public that there is a problem with XFS in the longterm stable kernel and that it would be really nice if someone can take a look at it.
Greets Stefan
Quoted 21 lines · Theodore Tso
Quoted 5 lines · Stefan Priebe - Profihost AG
Quoted 4 lines
whoever the maintainer of the -stable/-longterm tree is (be it an individual or a team employeed by some comapny) then looks at the patch and considers backporting it (if it's too hard, or to intrusive, they may decide not to).
So i have to contact Greg Kohan from SUSE directly?
Only if you can identity a specific patch you want backported to 2.6.32.
It's not the responsibility of the long-term stable tree maintainer to go looking through potentially tens of thousands of commits to find the one that should be backported. And if the code has changed too much since .2.6.32, and requires detailed reworking before the patch can get integrated into 2.6.32, then a subsystem developer will have to do that work.
Quoted 8 lines
Quoted 8 lines
the idea of the lonterm kernels is that organizations need to maintain a kernel for a long time due to commitments that they have made (Debian doesn't want to change the kernel it ships in a stable version, RedHat doesn't want to change the kernel version in a RHEL release, etc), and so they publicly announce this so that anyone else wanting to use the same kernel version can share in the work (and therefor everyone can benifit from each other's work)
Quoted 6 lines
Quoted 1 line
That was what i thoght. So a bug like this should get fixed right? Otherwise this makes no sense. Sadly Redhat has ported the fix back in his RHEL 6 2.6.32 kernel but they haven't send the patch to stable / vanilla team.
Not all distributions will participate in the maintenance stable tree. Red Hat for example is probably worried about people (specifically, Oracle) taking their kernel expertise "for free" and bidding it against them. So it doesn't surprise me that they aren't submitting patches to the stable tree. After all, they would like you to purchase a support contract if you want to get high quality, supported kernel. Why should they give that work away for free? After all, their salaried developers have to get paid somehow. Others will contribute work in the hopes that other people will also contribute fixes back, but of course nothing forces Red Hat to do this.
The thing that you have to understand that this is all a volunteer effort, and a few of your messages sound like you are expecting people to do the work for you. It doesn't work that way. If you ask nicely, maybe one of the XFS developers will help you. But please don't expect free support. That will just annoy people.
Ok, I don't speak for my company, and your point about not expecting people to do this work for you is valid, however I don't see why you need to take potshots at Red Hat, - they are quite active in the stable effort.
wget http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.38.8
grep Author ChangeLog-2.6.38.8 | grep -i redhat | wc -l 9
grep Author ChangeLog-2.6.38.8 | grep -i suse | wc -l 10
grep Author ChangeLog-2.6.38.8 | grep -i canonical | wc -l 5
I'm not even claiming that these are typical stats, but as just a quick check on your statement, the contributions for one stable release are in the same ballpark as everyone else. Don't know if lwn has stable stats.
Thanks
John
Quoted 3 lines · John Kacur
Ok, I don't speak for my company, and your point about not expecting people to do this work for you is valid, however I don't see why you need to take potshots at Red Hat
It wasn't a potshot; if it is true, it is a completely rational economic argument that is completely within the bounds of the requirements of the GPL, and with LKML community standards.
The reason why I say it is because of (a) http://lwn.net/Articles/430098/,
and (b) a few months ago, when I quietly floated starting a new long-term stable kernel series that a number of companies would maintain cooperatively, I was told, privately, that such a proposal would not likely be received positively by Red Hat management because of the reasons behind the policy instituted by (a).
Which is fine, I don't consider that a potshot, no more than I consider the fact that IBM forked the OpenOffice before it was relicensed to the LGPL and made changes which they didn't give back, and seems to be actively supporting Oracle's attempt to get Apache to adopt OpenOffice.org inspite of the currently healthy LibreOffice LGPL3 branch which already has an active community to be a potshot.
Both are completely legal things set by the ground rules of the copyright licenses involvled, and someone no less than Linus has said that the ability to fork is healthy because it keeps people honest.
Quoted 9 lines
- they are quite active in the stable effort.
wget http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.38.8
grep Author ChangeLog-2.6.38.8 | grep -i redhat | wc -l 9grep Author ChangeLog-2.6.38.8 | grep -i suse | wc -l 10
Quoted 6 lines
grep Author ChangeLog-2.6.38.8 | grep -i canonical | wc -l 5
I'm not even claiming that these are typical stats, but as just a quick check on your statement, the contributions for one stable release are in the same ballpark as everyone else.
Nah, that just means that commits which are labelled with "CC:
[email protected]" are automatically accepted into stable kernel series.
If you can point efforts where painful backports of ext4 and xfs bug fixes into RHEL 6.x are making it back into 2.6.32.y, even though in some cases it takes tens of hours of engineering and QA efforts, we can talk. But please note that I wasn't calling out Red Hat as being bad or evil by doing what they are doing; it is completely economically rational and allowed by the GPL rules for them to be doing what they are doing. (Just as what Android has been doing with their constant forward porting of the Wakelocks API is completely within the GPL rules.)
- Ted
Ted,
Quoted 10 lines · Ted Ts'o
Quoted 3 lines · John Kacur
Ok, I don't speak for my company, and your point about not expecting people to do this work for you is valid, however I don't see why you need to take potshots at Red Hat
It wasn't a potshot; if it is true, it is a completely rational economic argument that is completely within the bounds of the requirements of the GPL, and with LKML community standards.
Quoted 6 lines
The reason why I say it is because of (a) http://lwn.net/Articles/430098/,
and (b) a few months ago, when I quietly floated starting a new long-term stable kernel series that a number of companies would maintain cooperatively, I was told, privately, that such a proposal would not likely be received positively by Red Hat management because of the reasons behind the policy instituted by (a).
OK, you seem to think you have it all figured out and that your Red Hat contact knows everything there is to the Red Hat kernel engineering development process.
However, it doesn't jive with what I know or how I (and my coworkers) work on a daily basis.
Since we're referencing lwn.net articles please have a look at these comments that I previously made on this subject of RHEL and stable@:
http://lwn.net/Articles/431317/
http://lwn.net/Articles/431420/
<snip irrelevant tangent about GPL>
Quoted 8 lines
Quoted 3 lines
I'm not even claiming that these are typical stats, but as just a quick check on your statement, the contributions for one stable release are in the same ballpark as everyone else.
Nah, that just means that commits which are labelled with "CC:
[email protected]" are automatically accepted into stable kernel series.
Quoted 9 lines
If you can point efforts where painful backports of ext4 and xfs bug fixes into RHEL 6.x are making it back into 2.6.32.y, even though in some cases it takes tens of hours of engineering and QA efforts, we can talk. But please note that I wasn't calling out Red Hat as being bad or evil by doing what they are doing; it is completely economically rational and allowed by the GPL rules for them to be doing what they are doing. (Just as what Android has been doing with their constant forward porting of the Wakelocks API is completely within the GPL rules.)
And what _exactly_ is Red Hat (not) doing? Red Hat isn't going crazy backporting its upstream > 2.6.32 fixes to 2.6.32.y even when Red Hat doesn't consume 2.6.32.y? *gasp*
Mike
Quoted 4 lines · Mike Snitzer
And what _exactly_ is Red Hat (not) doing? Red Hat isn't going crazy backporting its upstream > 2.6.32 fixes to 2.6.32.y even when Red Hat doesn't consume 2.6.32.y? *gasp*
Well, the original poster was expecting that (unspecified people) would be doing this regularly (in fact he was complaining about how an XFS bug fixed in RHEL wasn't fixed in 2.6.32.y). I was explaining how it wasn't happening, and it was perfectly acceptable for that to be the case.
So I was actually *defending* Red Hat....
- Ted
Quoted 6 lines · Ted Ts'o
Quoted 4 lines · Mike Snitzer
And what _exactly_ is Red Hat (not) doing? Red Hat isn't going crazy backporting its upstream > 2.6.32 fixes to 2.6.32.y even when Red Hat doesn't consume 2.6.32.y? *gasp*
Quoted 7 lines
Well, the original poster was expecting that (unspecified people) would be doing this regularly (in fact he was complaining about how an XFS bug fixed in RHEL wasn't fixed in 2.6.32.y). I was explaining how it wasn't happening, and it was perfectly acceptable for that to be the case.
So I was actually *defending* Red Hat....
Btw, Redhat has a completely different XFS codebase than mainline 2.6.32, here's the diffstat summary between 2.6.32 and rhel6.1:
96 files changed, 5083 insertions(+), 5211 deletions(-)
Which roughly equals the diff between 2.6.32 and 2.6.34. So there's a fairly large chance things simply won't apply as-is anyway. The same is also true for ext4, btrfs and nfs.
Quoted 1 line
Quoted 1 line
That was what i thoght. So a bug like this should get fixed right? Otherwise this makes no sense. Sadly Redhat has ported the fix back in his RHEL 6 2.6.32 kernel but they haven't send the patch to stable / vanilla team.
It's often not that simple. As vendor trees diverge you may find it's not a case of 'sending patch X upstream' because the patches you need may well depend upon other things in the vendor tree not all of which will be in the 'stable' tree. So it's a base at best.
Quoted 1 line
Not all distributions will participate in the maintenance stable tree. Red Hat for example is probably worried about people (specifically, Oracle) taking their kernel expertise "for free" and bidding it against them.
Judging from the commit log that isn't the case.
Besides which I hear a rumour that oracle employees have learned how to download files from other websites and look at the content so it won't make any odds 8)
Alan
On Tue, Jun 07, 2011 at 11:28:59PM -0400, Theodore Tso wrote:
Ted, you need a new MUA, as it reads horribly in mutt.
Quoted 3 lines
Not all distributions will participate in the maintenance stable tree. Red Hat for example is probably worried about people (specifically, Oracle) taking their kernel expertise "for free" and bidding it against them. So it doesn't surprise me that they aren't submitting patches to the stable tree. After all, they would like you to purchase a support contract if you want to get high quality, supported kernel. Why should they give that work away for free? After all, their salaried developers have to get paid somehow. Others will contribute work in the hopes that other people will also contribute fixes back, but of course nothing forces Red Hat to do this.
As a Red Hat employee, I must speak against this. I have *never* been told to keep something from stable. In fact, I've always been encouraged to push to stable. But it is usually the individual engineer's responsibility to get a patch into stable. If something was missed, it was more the fault of that individual engineer that made the fix than Red Hat.
Ask Greg, I'm constantly sending stable patches to him.
-- Steve
Quoted 3 lines · Steven Rostedt
On Tue, Jun 07, 2011 at 11:28:59PM -0400, Theodore Tso wrote:
Ted, you need a new MUA, as it reads horribly in mutt.
Seconded. ;)
Quoted 12 lines
Quoted 11 lines
Not all distributions will participate in the maintenance stable tree. Red Hat for example is probably worried about people (specifically, Oracle) taking their kernel expertise "for free" and bidding it against them. So it doesn't surprise me that they aren't submitting patches to the stable tree. After all, they would like you to purchase a support contract if you want
to get high quality, supported kernel. Why should they give
that work away for free? After all, their salaried developers have to get paid somehow. Others will contribute work in the hopes that other people will also contribute fixes back, but of course nothing forces Red Hat to do this.
Quoted 6 lines
As a Red Hat employee, I must speak against this. I have *never* been told to keep something from stable. In fact, I've always been encouraged to push to stable. But it is usually the individual engineer's responsibility to get a patch into stable. If something was missed, it was more the fault of that individual engineer that made the fix than Red Hat.
If you want someone to blame, then point the fingers at me, not RedHat or RedHat's processes - RedHat is not involved in the processes and decisions as to what fixes
get backported into the community stable trees. How that is done is mainly dictated by available resources, which are generally scarce. We push bug fixes into the lastest kernel release, and if known to be needed for stable series they get pushed back via the stable queues.
This particular case is clearly an oversight - when we fixed the problem I pushed it into the current stable release, but forgot that we'd also backported fixes to .32 that meant we also needed to backport this fix to .32 as well.
If you look at a 2.6.32 kernel the fix is not needed for backport, but somewhere in the 2.6.32.y series, other fixes were backported which introduce the bug that make this fix necessary. That history is not in the mainline git tree and so it's really quite easy to make such a mistake. So the real cause of this oversight is the fact that there are simply too many disjoint community trees and that makes it difficult to determine what fixes need to go where.
As a result of the difficulty tracking stable trees, I don't even bother trying. Instead I use bug reports as triggers for "we need to backport this fix" to a stable kernel because I do not have the time to waste backporting fixes for bugs no-one is actually hitting...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 8 lines
If you want someone to blame, then point the fingers at me, not RedHat or RedHat's processes - RedHat is not involved in the processes and decisions as to what fixes
get backported into the community stable trees. How that is done is mainly dictated by available resources, which are generally scarce. We push bug fixes into the lastest kernel release, and if known to be needed for stable series they get pushed back via the stable queues.
I'm really sorry that my post goes that way. It was never my intention to blame anybody for anything.
I just didn't know the things work and was just wondering that Redhat had a fix which wasn't pushed back to vanilla 2.6.32. But i also didn't know that the xfs codebase is totally different due to other backports.
I'm sorry.
Stefan