Qlogic 2x00 driver
hi,
as the feature freeze of 2.5 comes close, i want to ask if the driver for the qlogic sanblade 2200/2300 series of hba's will be included in 2.5 ...
are there any plan's to do so ? has it been discussed before ?
i ask because i use those hba's together with ibm's fastt500 storage system, and it will be nice to have this driver in the default kernel ...
i use version 5.36.3 of the qlogic 2x00 driver in production (vanilla kernel 2.4.17 + qlogic 2x00 driver v5.36.3) since may 2002 and i never had any problems with this driver ...
(2 lotus domino servers and 1 fileserver all 3 are attached to the ibm fastt500 storage system using qlogic sanblade 2200 cards)
i don't know how many people use those qlogic card's, i got them together with the fastt500 storage system, ...
the current driver is avaiable here (it's GPL't):
http://www.qlogic.com/support/os_detail.asp?productid=112&osid=26
the qlogic 2x00 driver is also in andrea arcangelis "-aa" patches, so possibly he knows better if it would be useful to integrate those drivers into 2.5 or not, ...
thanks for your time,
please CC me, i'm currently not subscribed to lkml, simon.
Quoted 6 lines · Simon Roscic
hi,
as the feature freeze of 2.5 comes close, i want to ask if the driver for the qlogic sanblade 2200/2300 series of hba's will be included in 2.5 ...
are there any plan's to do so ? has it been discussed before ?
Quoted 6 lines
i ask because i use those hba's together with ibm's fastt500 storage system, and it will be nice to have this driver in the default kernel ...
i use version 5.36.3 of the qlogic 2x00 driver in production (vanilla kernel 2.4.17 + qlogic 2x00 driver v5.36.3) since may 2002 and i never had any problems with this driver ...
Oh so you haven't notices how it buffer-overflows the kernel stack, how it has major stack hog issues, how it keeps the io request lock (and interrupts disabled) for a WEEK ?
Quoted 3 lines · Arjan van de Ven
Oh so you haven't notices how it buffer-overflows the kernel stack, how it has major stack hog issues, how it keeps the io request lock (and interrupts disabled) for a WEEK ?
doesn't sound good, ...
as i said, i don't have any problems (= failures, data loss, etc.) with this driver, sounds like i should update to a newer driver version, wich qlogic 2x00 driver
version do you recommend ? or does this affect all versions of this driver ?
(performance on the machines i use is quite good, a dbench 256 gave me approx. 60 mb/s (ibm xseries 342, 1x pentium 3 - 1,2 ghz, 512 - 1024 mb ram))
arjan, thanks for the info, i didn't notice that the driver was that bad, i had much to do in the past months, so i possibly missed disscussions about the qlogic2x00 stuff, sorry,
simon.
(please CC me, i'm not subscribed to lkml)
Version 6.1b5 does appear to be a big improvement from looking at the code (certainly much more readable than version 4.x end earlier).
Although the method for creating the different modules for different hardware is pretty ugly.
in qla2300.c
#define ISP2300
[snip]
#include "qla2x00.c"
in qla2200.c
#define ISP2200
[snip]
#include "qla2x00.c"
I'm sure this would have to go before it got it.
~mc
Quoted 5 lines · Simon Roscic
Quoted 3 lines · Arjan van de Ven
Oh so you haven't notices how it buffer-overflows the kernel stack, how it has major stack hog issues, how it keeps the io request lock (and interrupts disabled) for a WEEK ?
This may have been the cause of problems I had running qla driver with lvm and ext3 - I was getting ooops with what looked like corrupted bufferheads.
This was happening in pretty much all kernels I tried (a variety of redhat kernels and aa kernels). Removing LVM has solved the problem. Although i was blaming LVM - maybe it was a buffer overflow in qla driver.
The rh kernel I tried had quite an old version (4.31) of the driver suffered from problems recovering from LIP resets. The latest 6.x drivers seem to handle this much better.
~mc
You might wanna look at version 6.01 instead. I say this because it's *not* a beta driver.
Quoted 47 lines · Michael Clark
Version 6.1b5 does appear to be a big improvement from looking at the code (certainly much more readable than version 4.x end earlier).
Although the method for creating the different modules for different hardware is pretty ugly.
in
qla2300.c#define ISP2300
[snip]
#include "qla2x00.c"in
qla2200.c#define ISP2200
[snip]
#include "qla2x00.c"I'm sure this would have to go before it got it.
~mc
Quoted 5 lines · Simon Roscic
Quoted 3 lines · Arjan van de Ven
Oh so you haven't notices how it buffer-overflows the kernel stack, how it has major stack hog issues, how it keeps the io request lock (and interrupts disabled) for a WEEK ?
This may have been the cause of problems I had running qla driver with lvm and ext3 - I was getting ooops with what looked like corrupted bufferheads.
This was happening in pretty much all kernels I tried (a variety of redhat kernels and aa kernels). Removing LVM has solved the problem. Although i was blaming LVM - maybe it was a buffer overflow in qla driver.
The rh kernel I tried had quite an old version (4.31) of the driver suffered from problems recovering from LIP resets. The latest 6.x drivers seem to handle this much better.
~mc
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
I doubt it will make a difference. LVM and qlogic drivers seem to be a bad mix. I've already tried the beta5 of 6.01 and same problem exists - ooops about every 5-8 days. Removing LVM and solved the problem.
The changelog only lists small changes since between 6.01 and 6.01b5
Although one entry suggests a fix to a race in qla2x00_done that would allow multiple completions on the same IO. Not sure if this relates to my problem with LVM as this occured with earlier versions of the qlogic driver without the dpc threads.
~mc
Quoted 56 lines · GrandMasterLee
You might wanna look at version 6.01 instead. I say this because it's *not* a beta driver.
Quoted 49 lines · Michael Clark
Version 6.1b5 does appear to be a big improvement from looking at the code (certainly much more readable than version 4.x end earlier).
Although the method for creating the different modules for different hardware is pretty ugly.
in
qla2300.c#define ISP2300
[snip]
#include "qla2x00.c"in
qla2200.c#define ISP2200
[snip]
#include "qla2x00.c"I'm sure this would have to go before it got it.
~mc
Quoted 7 lines · Simon Roscic
Quoted 3 lines · Arjan van de Ven
Oh so you haven't notices how it buffer-overflows the kernel stack, how it has major stack hog issues, how it keeps the io request lock (and interrupts disabled) for a WEEK ?
This may have been the cause of problems I had running qla driver with lvm and ext3 - I was getting ooops with what looked like corrupted bufferheads.
This was happening in pretty much all kernels I tried (a variety of redhat kernels and aa kernels). Removing LVM has solved the problem. Although i was blaming LVM - maybe it was a buffer overflow in qla driver.
The rh kernel I tried had quite an old version (4.31) of the driver suffered from problems recovering from LIP resets. The latest 6.x drivers seem to handle this much better.
~mc
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 6 lines
Michael Clark, . . . . . . . . . . . . . . . [email protected] Managing Director, . . . . . . . . . . . . . . . phone: +65 6395 6277 Metaparadigm Pte. Ltd. . . . . . . . . . . . . . mobile: +65 9645 9612 25F Paterson Road, Singapore 238515 . . . . . . . . fax: +65 6234 4043 I'm successful because I'm lucky. The harder I work, the luckier I get.
Just to make sure we are on the same page,
was that LVM1, LVM2, or EVMS?
Joe
Quoted 4 lines · Michael Clark
I doubt it will make a difference. LVM and qlogic drivers seem to be a bad mix. I've already tried the beta5 of 6.01 and same problem exists - ooops about every 5-8 days. Removing LVM and solved the problem.
My Dell 6650 has been doing this exact behaviour since we got on 5.38.9 and up, using LVM in a production capacity. Both servers we have, have crashed mysteriously, without any kernel dump, etc, but all hardware diags come out clean.
All hardware configuration bits are perfect, as can be anyway, and we still get this behaviour. After 5-6.5 days...the box black screens. So bad so, that all the XFS volumes we have, never enter a shutdown. We must repair them all, today this happened, and we lost one part of the tablespace on our beta db. We're using LVM1, on 2.4.19-aa1.
Quoted 14 lines · J Sloan
Just to make sure we are on the same page,
was that LVM1, LVM2, or EVMS?Joe
Quoted 4 lines · Michael Clark
I doubt it will make a difference. LVM and qlogic drivers seem to be a bad mix. I've already tried the beta5 of 6.01 and same problem exists - ooops about every 5-8 days. Removing LVM and solved the problem.
Quoted 4 lines · GrandMasterLee
My Dell 6650 has been doing this exact behaviour since we got on 5.38.9 and up, using LVM in a production capacity. Both servers we have, have crashed mysteriously, without any kernel dump, etc, but all hardware diags come out clean.
I tell you my honest hunch - remove LVM and try again. This has made my life a little more peaceful lately. Even with a 2-3 minute outages while our cluster automatically fails over - the 100's of users whining about their sessions being disconnected makes you a bit depressed.
Quoted 5 lines
All hardware configuration bits are perfect, as can be anyway, and we still get this behaviour. After 5-6.5 days...the box black screens. So bad so, that all the XFS volumes we have, never enter a shutdown. We must repair them all, today this happened, and we lost one part of the tablespace on our beta db. We're using LVM1, on 2.4.19-aa1.
We had the black screen also until we got the machines oopsing over serial. The oops was actually showing up in ext3 with a corrupted bufferhead. Without LVM, i've measured my longest uptime, 17 days x 4 machines in the cluster (68 days) ie. we only did it 17 days ago.
~mc
Quoted 18 lines · J Sloan
Quoted 13 lines
Just to make sure we are on the same page,
was that LVM1, LVM2, or EVMS?Joe
Quoted 4 lines · Michael Clark
I doubt it will make a difference. LVM and qlogic drivers seem to be a bad mix. I've already tried the beta5 of 6.01 and same problem exists - ooops about every 5-8 days. Removing LVM and solved the problem.
Quoted 10 lines · Michael Clark
Quoted 4 lines · GrandMasterLee
My Dell 6650 has been doing this exact behaviour since we got on 5.38.9 and up, using LVM in a production capacity. Both servers we have, have crashed mysteriously, without any kernel dump, etc, but all hardware diags come out clean.
I tell you my honest hunch - remove LVM and try again. This has made my life a little more peaceful lately. Even with a 2-3 minute outages while our cluster automatically fails over - the 100's of users whining about their sessions being disconnected makes you a bit depressed.
Almost making it to your go-live date, only to have everything come crashing down all around you is quite depressing.
Quoted 6 lines
Quoted 5 lines
All hardware configuration bits are perfect, as can be anyway, and we still get this behaviour. After 5-6.5 days...the box black screens. So bad so, that all the XFS volumes we have, never enter a shutdown. We must repair them all, today this happened, and we lost one part of the tablespace on our beta db. We're using LVM1, on 2.4.19-aa1.
Quoted 6 lines
We had the black screen also until we got the machines oopsing over serial. The oops was actually showing up in ext3 with a corrupted bufferhead. Without LVM, i've measured my longest uptime, 17 days x 4 machines in the cluster (68 days) ie. we only did it 17 days ago.
~mc
I believe you, that was my next thought, but I didn't know if that would really help just to be honest. Thanks for the input there.
I've been going crazy trying to catch any piece of sanity out of this thing to understand if this was what was happening or not. I feel a bit dumb for not trying serial console yet, but I knew either that or KDB should tell us something. I will see what we can do, it will take less time to do this, than to reload everything all over again.
Should I remove LVM all together, or just not use it? In your opinion.
Quoted 4 lines · GrandMasterLee
Quoted 2 lines · Michael Clark
On 10/16/02 12:43, GrandMasterLee wrote:
[snip]
Quoted 14 lines
Quoted 10 lines
Quoted 5 lines
All hardware configuration bits are perfect, as can be anyway, and we still get this behaviour. After 5-6.5 days...the box black screens. So bad so, that all the XFS volumes we have, never enter a shutdown. We must repair them all, today this happened, and we lost one part of the tablespace on our beta db. We're using LVM1, on 2.4.19-aa1.
We had the black screen also until we got the machines oopsing over serial. The oops was actually showing up in ext3 with a corrupted bufferhead. Without LVM, i've measured my longest uptime, 17 days x 4 machines in the cluster (68 days) ie. we only did it 17 days ago.
I believe you, that was my next thought, but I didn't know if that would really help just to be honest. Thanks for the input there.
Quoted 7 lines
I've been going crazy trying to catch any piece of sanity out of this thing to understand if this was what was happening or not. I feel a bit dumb for not trying serial console yet, but I knew either that or KDB should tell us something. I will see what we can do, it will take less time to do this, than to reload everything all over again.
Should I remove LVM all together, or just not use it? In your opinion.
I just didn't load the module after migrating my volumes. If the problem is a stack problem, then its probably not necessarily a bug in LVM
- just the combination of it, ext3 and the qlogic driver don't mix well
- so if its not being used, then it won't be increasing the stack footprint.
~mc
On Wed, 2002-10-16 at 01:40, Michael Clark wrote:
...
Quoted 9 lines
Quoted 1 line
Should I remove LVM all together, or just not use it? In your opinion.
I just didn't load the module after migrating my volumes. If the problem is a stack problem, then its probably not necessarily a bug in LVM
- just the combination of it, ext3 and the qlogic driver don't mix well
- so if its not being used, then it won't be increasing the stack footprint.~mc
Not to be dense, but it's compiled into my kernel, that's why I ask. We try not to use modules where we can help it. So I'm thinking, if no VG are actively used, then LVM won't affect the stack much. I just don't know if that's true or not.
--The GrandMaster
Quoted 10 lines · GrandMasterLee
On Wed, 2002-10-16 at 01:40, Michael Clark wrote:
...Quoted 6 lines
Quoted 1 line
Should I remove LVM all together, or just not use it? In your opinion.
I just didn't load the module after migrating my volumes. If the problem is a stack problem, then its probably not necessarily a bug in LVM
- just the combination of it, ext3 and the qlogic driver don't mix well
- so if its not being used, then it won't be increasing the stack footprint.
Quoted 4 lines
Not to be dense, but it's compiled into my kernel, that's why I ask. We try not to use modules where we can help it. So I'm thinking, if no VG are actively used, then LVM won't affect the stack much. I just don't know if that's true or not.
Correct. Won't effect the stack at all.
~mc
Quoted 7 lines · J Sloan
Just to make sure we are on the same page,
was that LVM1, LVM2, or EVMS?Joe
Michael Clark wrote:
Quick question on this, could this problem be exacerbated, perhaps, by large pagebuf usage that XFS performs, as well as the FS buffers that it allocates, since XFS allocates a lot of read/write buffers for logging?
LVM1 tried in numerous versions of 2.4.x both aa and rh version.
Every one i was getting oops when used with a combination of ext3, LVM1 and qla2x00 driver.
Since taking LVM1 out of the picture, my oopsing problem has gone away. This could of course not be LVM1's fault but the fact that qla driver is a stack hog or something - i don't have enough information to draw any conclusions all at the moment i'm too scared to try LVM again (plus the time it takes to migrate a few hundred gigs of storage).
~mc
Quoted 11 lines · J Sloan
Just to make sure we are on the same page,
was that LVM1, LVM2, or EVMS?Joe
Quoted 4 lines · Michael Clark
I doubt it will make a difference. LVM and qlogic drivers seem to be a bad mix. I've already tried the beta5 of 6.01 and same problem exists - ooops about every 5-8 days. Removing LVM and solved the problem.
Quoted 9 lines · Michael Clark
Every one i was getting oops when used with a combination of ext3, LVM1 and qla2x00 driver.
Since taking LVM1 out of the picture, my oopsing problem has gone away. This could of course not be LVM1's fault but the fact that qla driver is a stack hog or something - i don't have enough information to draw any conclusions all at the moment i'm too scared to try LVM again (plus the time it takes to migrate a few hundred gigs of storage).
Yes, we have seen that ext3 is a stack hog in some cases, and I know there were some fixes in later LVM versions to remove some huge stack allocations. Arjan also reported stack problems with qla2x00, so it is not a surprise that the combination causes problems.
In 2.5 there is the "4k IRQ stack" patch floating around, which would avoid these problems.
Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/
Quoted 11 lines · Andreas Dilger
Quoted 9 lines · Michael Clark
Every one i was getting oops when used with a combination of ext3, LVM1 and qla2x00 driver.
Since taking LVM1 out of the picture, my oopsing problem has gone away. This could of course not be LVM1's fault but the fact that qla driver is a stack hog or something - i don't have enough information to draw any conclusions all at the moment i'm too scared to try LVM again (plus the time it takes to migrate a few hundred gigs of storage).
Quoted 6 lines
Yes, we have seen that ext3 is a stack hog in some cases, and I know there were some fixes in later LVM versions to remove some huge stack allocations. Arjan also reported stack problems with qla2x00, so it is not a surprise that the combination causes problems.
The stack issues were a major problem in the 5.3x series driver. I believe, I can check tomorrow, 5.38.9 (the driver Dell distributes) contains fixes for the stack clobbering -- qla2x00-rh1-3 also contain the fixes.
IAC, I believe the support tech working with MasterLee had asked for additional information regarding the configuration as well as some basic logs. Ideally we'd like to setup a similiar configuration in house and see what's happening...
--
Andrew Vasquez | [email protected] |
DSS: 0x508316BB, FP: 79BD 4FAC 7E82 FF70 6C2B 7E8B 168F 5529 5083 16BB
Quoted 10 lines · Andrew Vasquez
Quoted 6 lines
Yes, we have seen that ext3 is a stack hog in some cases, and I know there were some fixes in later LVM versions to remove some huge stack allocations. Arjan also reported stack problems with qla2x00, so it is not a surprise that the combination causes problems.
The stack issues were a major problem in the 5.3x series driver. I believe, I can check tomorrow, 5.38.9 (the driver Dell distributes) contains fixes for the stack clobbering -- qla2x00-rh1-3 also contain the fixes.
Does this mean that 6.01 will NOT work either? What drivers will be affected? We've already made the move to remove LVM from the mix, but your comments above give me some doubt as to how definite it is, that the stack clobbering will be fixed by doing so.
Quoted 4 lines
IAC, I believe the support tech working with MasterLee had asked for additional information regarding the configuration as well as some basic logs. Ideally we'd like to setup a similiar configuration in house and see what's happening...
In-house? Just curious. What can "I" do to know if our configuration won't get broken, just by removing LVM? TIA.
Quoted 3 lines
--
Andrew Vasquez | [email protected] |
DSS:0x508316BB, FP: 79BD 4FAC 7E82 FF70 6C2B 7E8B 168F 5529 5083 16BB
Quoted 12 lines · GrandMasterLee
Quoted 10 lines · Andrew Vasquez
Quoted 6 lines
Yes, we have seen that ext3 is a stack hog in some cases, and I know there were some fixes in later LVM versions to remove some huge stack allocations. Arjan also reported stack problems with qla2x00, so it is not a surprise that the combination causes problems.
The stack issues were a major problem in the 5.3x series driver. I believe, I can check tomorrow, 5.38.9 (the driver Dell distributes) contains fixes for the stack clobbering -- qla2x00-rh1-3 also contain the fixes.
Quoted 5 lines
Does this mean that 6.01 will NOT work either? What drivers will be affected? We've already made the move to remove LVM from the mix, but your comments above give me some doubt as to how definite it is, that the stack clobbering will be fixed by doing so.
The 6.x series driver basically branched from the 5.x series driver. Changes made, many moons ago, are already in the 6.x series driver. To quell your concerns, yes, stack overflow is not an issue with the 6.x series driver.
I believe if we are to get anywhere regarding this issue, we need to shift focus from stack corruption in early versions of the driver.
Quoted 7 lines
Quoted 4 lines
IAC, I believe the support tech working with MasterLee had asked for additional information regarding the configuration as well as some basic logs. Ideally we'd like to setup a similiar configuration in house and see what's happening...
In-house?
Sorry, short introduction, Andrew Vasquez, Linux driver development at QLogic.
Quoted 3 lines
Just curious. What can "I" do to know if our configuration won't get broken, just by removing LVM? TIA.
I've personally never used LVM before, so I cannot even begin to attempt to answer your question -- please work with the tech on this one, if it's a driver problem, we'd like to fix it.
--
Andrew
On Wed, 2002-10-16 at 22:11, Andrew Vasquez wrote:
...
Quoted 5 lines
Quoted 5 lines
Does this mean that 6.01 will NOT work either? What drivers will be affected? We've already made the move to remove LVM from the mix, but your comments above give me some doubt as to how definite it is, that the stack clobbering will be fixed by doing so.
I was asking because We crashed, while using this driver, AND LVM.
Quoted 7 lines
The 6.x series driver basically branched from the 5.x series driver. Changes made, many moons ago, are already in the 6.x series driver. To quell your concerns, yes, stack overflow is not an issue with the 6.x series driver.
I believe if we are to get anywhere regarding this issue, we need to shift focus from stack corruption in early versions of the driver.
In this way, you mean, that it is not an issue since you guys don't try to use LVM.
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
IAC, I believe the support tech working with MasterLee had asked for additional information regarding the configuration as well as some basic logs. Ideally we'd like to setup a similiar configuration in house and see what's happening...
In-house?
Sorry, short introduction, Andrew Vasquez, Linux driver development at QLogic.
Nice to meet ya. :)
Quoted 5 lines
Quoted 3 lines
Just curious. What can "I" do to know if our configuration won't get broken, just by removing LVM? TIA.
I've personally never used LVM before, so I cannot even begin to attempt to answer your question --
We've removed LVM from the config, per Michael's issue and recommendation, but I'm just scared that we *could* see this issue with XFS and Qlogic. Since you're saying the 6.01 has no stack clobbering issues, then is it XFS, LVM and Qlogic?
Quoted 2 lines
please work with the tech on this
one, if it's a driver problem, we'd like to fix it.
I'm going to try, but we've got to get up and in production ASAP. Since it takes *days* to cause the crash, I don't know how I can cause it and get the stack dump.
Quoted 7 lines
--
Andrew
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 16 lines · Andrew Vasquez
Quoted 7 lines · GrandMasterLee
Quoted 4 lines · Andrew Vasquez
The stack issues were a major problem in the 5.3x series driver. I believe, I can check tomorrow, 5.38.9 (the driver Dell distributes) contains fixes for the stack clobbering -- qla2x00-rh1-3 also contain the fixes.
Quoted 5 lines
Does this mean that 6.01 will NOT work either? What drivers will be affected? We've already made the move to remove LVM from the mix, but your comments above give me some doubt as to how definite it is, that the stack clobbering will be fixed by doing so.
Quoted 5 lines
The 6.x series driver basically branched from the 5.x series driver. Changes made, many moons ago, are already in the 6.x series driver. To quell your concerns, yes, stack overflow is not an issue with the
6.x series driver.I believe if we are to get anywhere regarding this issue, we need to shift focus from stack corruption in early versions of the driver.
Well corruption of bufferheads was happening for me with a potentially stack deep setup (ext3+LVM+qlogic). Maybe it has been fixed in the non-LVM case but is still an issue as I have had it with 6.0.1b3 - The stack fix is listed in 6.0b13 which is quite a few release behind the one i've had the problem with.
I posted the oops to lk about 3 weeks ago. Wasn't sure it was a qlogic problem at the time, and still am not certain - maybe just sum of stack(ext3+lvm+qlogic). Even if qla stack was trimmed for the common case, it may still be a problem when LVM is active as there would be much deeper stacks during block io.
http://marc.theaimsgroup.com/?l=linux-kernel&m=103302016311188&w=2
The oops doesn't show qlogic at all although it is a corrupt bufferhead that is causing the oops so may have been silently corrupted earlier by a qlogic interrupt or block io submission while deep inside lvm and ext3 or some such, ie. the oops is one of those difficult sort that shows up corruption from some earlier event that is not directly traceable from the oops itself.
~mc
Quoted 22 lines · Michael Clark
Quoted 21 lines · Andrew Vasquez
Quoted 7 lines · GrandMasterLee
Quoted 4 lines · Andrew Vasquez
The stack issues were a major problem in the 5.3x series driver. I believe, I can check tomorrow, 5.38.9 (the driver Dell distributes) contains fixes for the stack clobbering -- qla2x00-rh1-3 also contain the fixes.
Quoted 5 lines
Does this mean that 6.01 will NOT work either? What drivers will be affected? We've already made the move to remove LVM from the mix, but your comments above give me some doubt as to how definite it is, that the stack clobbering will be fixed by doing so.
The 6.x series driver basically branched from the 5.x series driver. Changes made, many moons ago, are already in the 6.x series driver. To quell your concerns, yes, stack overflow is not an issue with the 6.x series driver.
Quoted 8 lines
Quoted 2 lines
I believe if we are to get anywhere regarding this issue, we need to shift focus from stack corruption in early versions of the driver.
Well corruption of bufferheads was happening for me with a potentially stack deep setup (ext3+LVM+qlogic). Maybe it has been fixed in the non-LVM case but is still an issue as I have had it with 6.0.1b3 - The stack fix is listed in 6.0b13 which is quite a few release behind the one i've had the problem with.
I don't disagree, but I saw the same things with XFS filesystems on LVM also. This leads me to my next question. Does anyone on this list use XFS plus QLA2300's with 500GB+ mounted by several volumes on Qlogic driver 5.38.x or > and have greater than 20 days uptime to date?
Quoted 6 lines
I posted the oops to lk about 3 weeks ago. Wasn't sure it was a qlogic problem at the time, and still am not certain - maybe just sum of stack(ext3+lvm+qlogic). Even if qla stack was trimmed for the common case, it may still be a problem when LVM is active as there would be much deeper stacks during block io.
Kewl..thanks much.
Quoted 2 lines · Michael Clark
Version 6.1b5 does appear to be a big improvement from looking at the code (certainly much more readable than version 4.x end earlier).
i'll try version 6.01 or so next week and i will see what happens. thanks for your help.
Quoted 3 lines
Although the method for creating the different modules for different hardware is pretty ugly.
...
i see.
Quoted 3 lines
This was happening in pretty much all kernels I tried (a variety of redhat kernels and aa kernels). Removing LVM has solved the problem. Although i was blaming LVM - maybe it was a buffer overflow in qla driver.
looks like i had a lot of luck, because my 3 servers wich are using the qla2x00 5.36.3 driver were running without problems, but i'll update to 6.01 in the next few day's.
i don't use lvm, the filesystem i use is xfs, so it smells like i had a lot of luck for not running into this problem, ...
simon.
(please CC me, i'm not subscribed to lkml)
Quoted 9 lines · Simon Roscic
Quoted 3 lines
This was happening in pretty much all kernels I tried (a variety of redhat kernels and aa kernels). Removing LVM has solved the problem. Although i was blaming LVM - maybe it was a buffer overflow in qla driver.
looks like i had a lot of luck, because my 3 servers wich are using the qla2x00 5.36.3 driver were running without problems, but i'll update to 6.01 in the next few day's.
i don't use lvm, the filesystem i use is xfs, so it smells like i had a lot of luck for not running into this problem, ...
Seems to be the correlation so far. qlogic driver without lvm works okay. qlogic driver with lvm, oopsorama.
~mc
Quoted 6 lines · Michael Clark
Quoted 4 lines · Simon Roscic
Quoted 3 lines
This was happening in pretty much all kernels I tried (a variety of redhat kernels and aa kernels). Removing LVM has solved the problem. Although i was blaming LVM - maybe it was a buffer overflow in qla driver.
Quoted 6 lines
Quoted 6 lines
looks like i had a lot of luck, because my 3 servers wich are using the qla2x00 5.36.3 driver were running without problems, but i'll update to 6.01 in the next few day's.
i don't use lvm, the filesystem i use is xfs, so it smells like i had a lot of luck for not running into this problem, ...
So then, it seems that LVM is adding stress to the system in a way that is bad for the kernel. Perhaps the read-ahead in conjunction with the large buffers from XFS, plus the amount of volumes we run(22 on the latest machine to crash).
Quoted 2 lines
Seems to be the correlation so far. qlogic driver without lvm works okay. qlogic driver with lvm, oopsorama.
Michael, what exactly do your servers do? Are they DB servers with ~1Tb connected, or file-servers with hundreds of gigs, etc?
Quoted 7 lines
~mc
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 7 lines · GrandMasterLee
Quoted 2 lines · Michael Clark
Seems to be the correlation so far. qlogic driver without lvm works okay. qlogic driver with lvm, oopsorama.
Michael, what exactly do your servers do? Are they DB servers with ~1Tb connected, or file-servers with hundreds of gigs, etc?
My customer currently has about 400Gb on this particular 4 node Application cluster (actually 2 x 2 node clusters using kimberlite HA software).
It has 11 logical hosts (services) spread over the 4 nodes with services such as Oracle 8.1.7, Oracle Financials (11i), a busy openldap server, and busy netatalk AppleShare Servers, Cyrus IMAP server. All are on ext3 partitions and were previously using LVM to slice up the storage.
The cluster usually has around 200-300 active users.
We have had oops (in ext3) on differing logical hosts which where running different services. ie. has oopsed on the node mastering the fileserver, and also on the node mastering the oracle database.
Cross fingers, since removing LVM (which was the only change we have made, same kernel) we have had 3 times our longest uptime and still counting.
By the sounds, from earlier emails I had posted, users had responded to me who were also using qlogic and none of them had had any problems, the key factor was none of them were running LVM - this is what made me think to try and remove it (it was really just a hunch). We had gone through months of changing kernel versions, changing GigE network adapters, driver versions, etc, to no avail, then finally the LVM removal.
Due to the potential nature of it being a stack problem. The problem really can't just be pointed at LVM but more the additive effect this would have on some underlying stack problem.
I believe the RedHat kernels i tried (rh7.2 2.4.9-34 errata was the most recent) also had this 'stack' problem. I am currently using 2.4.19pre10aa4.
I would hate to reccomend you remove LVM and it not work, but i must say it has worked for me (i'm just glad i didn't go to XFS instead of removing LVM as i did - as this was the other option i was pondering).
~mc
Quoted 9 lines · Michael Clark
Quoted 7 lines · GrandMasterLee
Quoted 2 lines · Michael Clark
Seems to be the correlation so far. qlogic driver without lvm works okay. qlogic driver with lvm, oopsorama.
Michael, what exactly do your servers do? Are they DB servers with ~1Tb connected, or file-servers with hundreds of gigs, etc?
Quoted 7 lines
My customer currently has about 400Gb on this particular 4 node Application cluster (actually 2 x 2 node clusters using kimberlite HA software).
It has 11 logical hosts (services) spread over the 4 nodes with services such as Oracle 8.1.7, Oracle Financials (11i), a busy openldap server, and busy netatalk AppleShare Servers, Cyrus IMAP server. All are on ext3 partitions and were previously using LVM to slice up the storage.
On each of the Nodes, correct?
Quoted 5 lines
The cluster usually has around 200-300 active users.
We have had oops (in ext3) on differing logical hosts which where running different services. ie. has oopsed on the node mastering the fileserver, and also on the node mastering the oracle database.
And again, each was running LVM in a shared storage mode for failover?
Quoted 9 lines
Cross fingers, since removing LVM (which was the only change we have made, same kernel) we have had 3 times our longest uptime and still counting.
By the sounds, from earlier emails I had posted, users had responded to me who were also using qlogic and none of them had had any problems, the key factor was none of them were running LVM - this is what made me think to try and remove it (it was really just a hunch). We had gone through months of changing kernel versions, changing GigE network adapters, driver versions, etc, to no avail, then finally the LVM removal.
Kewl. That makes me feel much better now too.
Quoted 6 lines
Due to the potential nature of it being a stack problem. The problem really can't just be pointed at LVM but more the additive effect this would have on some underlying stack problem.
I believe the RedHat kernels i tried (rh7.2 2.4.9-34 errata was the most recent) also had this 'stack' problem. I am currently using 2.4.19pre10aa4.
Kewl. I'm using 2.4.19-aa1 (rc5-aa1, but hell, it's the same thing).
Quoted 3 lines
I would hate to reccomend you remove LVM and it not work, but i must say it has worked for me (i'm just glad i didn't go to XFS instead of removing LVM as i did - as this was the other option i was pondering).
I hear you. We were pondering changing to EXT3, and not just EXT3, RHAS also. i.e. more money, unknown kernel config, etc. I was going to be *very* upset. Are you running FC2(qla2300Fs in FC2 config) or FC1?
TIA
Quoted 2 lines
~mc
Quoted 16 lines · GrandMasterLee
Quoted 14 lines · Michael Clark
Quoted 8 lines · GrandMasterLee
Quoted 2 lines · Michael Clark
Seems to be the correlation so far. qlogic driver without lvm works okay. qlogic driver with lvm, oopsorama.
Michael, what exactly do your servers do? Are they DB servers with ~1Tb connected, or file-servers with hundreds of gigs, etc?
My customer currently has about 400Gb on this particular 4 node Application cluster (actually 2 x 2 node clusters using kimberlite HA software).
Quoted 7 lines
Quoted 4 lines
It has 11 logical hosts (services) spread over the 4 nodes with services such as Oracle 8.1.7, Oracle Financials (11i), a busy openldap server, and busy netatalk AppleShare Servers, Cyrus IMAP server. All are on ext3 partitions and were previously using LVM to slice up the storage.
On each of the Nodes, correct?
We had originally planned to split up the storage in the RAID head using individual luns for each cluster logical host - so we could use SCSI reservations - but we encountered problems with the RAID heads device queue.
The RAID head has a global queue depth of 64 and to alleviate queue problems with the RAID heading locking up, we needed to minimise the number of luns, so late in the piece we added LVM to split up the storage.
We are using LVM in a clustered fashion. ie. we export most of the array as one big lun, and slice it into lvs, each one associated with a logical host. All lvs are accessible from all 4 physical hosts in the cluster. Care and application locking ensures only 1 physical host mounts any lv/partition at the same time (except for cluster quorum partitions which need to be accessed concurrently from 2 nodes - and for these we have seperate quorum disks in the array).
lvm metadata changes are made from one node while the others are down (or just have volumes deactivated, unmounted, then lvm-mod removed) to avoid screwing our metadata because lvm is not cluster aware.
We are not using mutlipath put have the cluster arranged in a topology such that the HA RAID Head has 2 controllers with each side of the cluster hanging of a different one ie. L and R. If we have a path failure, we will just loose CPU capacity (25-50% depending). The logical hosts will automatically move onto a physical node which still has connectivity to the RAID head (by the cluster software checking of connectivity to the quorum paritions). This gives us a good level of redundancy without the added cost of 2 paths from each host. ie. after a path failure we run with degraded performance only.
We are using vanilla 2300's.
~mc
Quoted 1 line · Simon Roscic
hi,
...
Quoted 8 lines
i ask because i use those hba's together with ibm's fastt500 storage system, and it will be nice to have this driver in the default kernel ...
i use version 5.36.3 of the qlogic 2x00 driver in production (vanilla kernel 2.4.17 + qlogic 2x00 driver v5.36.3) since may 2002 and i never had any problems with this driver ...
(2 lotus domino servers and 1 fileserver all 3 are attached to the ibm fastt500 storage system using qlogic sanblade 2200 cards)
Do you use LVM, EVMS, MD, other, or none?
TIA
--The GrandMaster
Quoted 2 lines · GrandMasterLee
Do you use LVM, EVMS, MD, other, or none?
none.
it's a XFS filesystem with the folowing mount options:
rw,noatime,logbufs=8,logbsize=32768
(this apply's to all 3 machines)
simon.
(please CC me, i'm not subscribed to lkml)
Do you actually get the lockups then?
Quoted 13 lines · Simon Roscic
Quoted 2 lines · GrandMasterLee
Do you use LVM, EVMS, MD, other, or none?
none.
it's a XFS filesystem with the folowing mount options:
rw,noatime,logbufs=8,logbsize=32768(this apply's to all 3 machines)
simon.
(please CC me, i'm not subscribed to lkml)
Quoted 1 line · GrandMasterLee
Do you actually get the lockups then?
no, i didn't had any lookups, each of the machines currently have an uptime of only 16 day's and that's because we had to shutdown the power in our whole company for a half day.
the best uptime i had was approx 40-50 day's, then i got the following
problem: the lotus domino server processes (not the whole machine) were freezing every week, but that is a known problem for heavy loaded domino servers, you have to increase the ammount of ipc memory for java or something (of the domino server), and since i did this everything works without problems.
the "primary" lotus domino server also got quite swap happy in the last weeks, currently he has to serve almost everything that has to do with notes, the second server isn't realy in use yet ...
if you are interested, procinfo -a shows this on one of the 3 machines:
(all 3 are the same, except that the "primary" lotus domino server has 2 cpu's and 2 gb ram, the other 2, have 1 cpu and 1 gb ram)
---------------- procinfo ----------------
Linux 2.4.17-xfs-smp (root@adam-neu) (gcc 2.95.3 20010315 ) #1 2CPU [adam.]
Memory: Total Used Free Shared Buffers Cached Mem: 2061272 2050784 10488 0 2328 1865288 Swap: 1056124 265652 790472
Bootup: Tue Oct 1 17:42:07 2002 Load average: 0.14 0.08 0.02 1/445 11305
user : 1d 20:15:32.03 5.7% page in :1196633058 disk 1: 1670401r 953006w nice : 0:00:24.69 0.0% page out:261985556 disk 2: 27762380r11039499w system: 13:05:51.19 1.7% swap in : 5870304 disk 3: 4r 0w idle : 29d 18:09:25.15 92.5% swap out: 5099371 disk 4: 4r 0w uptime: 16d 1:45:36.53 context :2810591591
irq 0: 138873653 timer irq 12: 104970 PS/2 Mouse irq 1: 5597 keyboard irq 14: 54 ide0 irq 2: 0 cascade [4] irq 18: 8659653 ips irq 3: 1 irq 20: 421419256 e1000 irq 4: 1 irq 24: 38444870 qla2200 irq 6: 3 irq 28: 17728 e100 irq 8: 2 rtc
Kernel Command Line:
auto BOOT_IMAGE=Linux ro root=803 BOOT_FILE=/boot/vmlinuz
Modules:
24 *sg 6 lp 25 parport 59 *e100
48 *e1000 165 *qla2200
Character Devices: Block Devices:
1 mem 10 misc 2 fd 2 pty 21 sg 3 ide0 3 ttyp 29 fb 8 sd 4 ttyS 128 ptm 65 sd 5 cua 136 pts 66 sd 6 lp 162 raw 7 vcs 254 HbaApiDev
File Systems:
[rootfs] [bdev] [proc] [sockfs] [tmpfs] [pipefs] ext3 ext2 [nfs] [smbfs] [devpts] xfs
---------------- procinfo ----------------
the kernel running on the 3 machines is a "vanilla" 2.4.17 plus XFS, plus ext3-0.9.17, plus intel ether express 100 and and intel ether express 1000 driver (e100 and e1000), and the qlogic qla2x00 5.36.3 driver ...
i think i will wait for 2.4.20 and then make a new kernel for the 3 machines ...
simon.
(please CC me, i'm not subscribed to lkml)
Quoted 7 lines · Simon Roscic
Quoted 1 line · GrandMasterLee
Do you actually get the lockups then?
no, i didn't had any lookups, each of the machines currently have an uptime of only 16 day's and that's because we had to shutdown the power in our whole company for a half day.
...
One question about your config, are you using, on ANY machines, QLA2300's or PCI-X, and 5.38.x or 6.xx qlogic drivers? If so, then you've experienced no lockups with those machines too?
Quoted 6 lines
if you are interested, procinfo -a shows this on one of the 3 machines:
(all 3 are the same, except that the "primary" lotus domino server has 2 cpu's and 2 gb ram, the other 2, have 1 cpu and 1 gb ram)---------------- procinfo ----------------
Linux 2.4.17-xfs-smp (root@adam-neu) (gcc 2.95.3 20010315 ) #1 2CPU [adam.]
Thanks for the info. I will hopefully have >5 days uptime now too. If not, anyone need a Systems Architect? :-D
--The GrandMaster
Quoted 3 lines · GrandMasterLee
One question about your config, are you using, on ANY machines, QLA2300's or PCI-X, and 5.38.x or 6.xx qlogic drivers? If so, then you've experienced no lockups with those machines too?
no, the 3 machines i use, are basically the same (ibm xseries 342), and have the same qlogic cards (qla2200), all 3 machines use the same kernel (2.4.17+xfs+ext3-0.9.17+e100+e1000+qla2x00-5.36.3).
a few details, possibly something help's you:
---------------- lspci ----------------
00:00.0 Host bridge: ServerWorks CNB20HE (rev 23)
00:00.1 Host bridge: ServerWorks CNB20HE (rev 01)
00:00.2 Host bridge: ServerWorks: Unknown device 0006 (rev 01)
00:00.3 Host bridge: ServerWorks: Unknown device 0006 (rev 01)
00:06.0 VGA compatible controller: S3 Inc. Savage 4 (rev 06)
00:0f.0 ISA bridge: ServerWorks OSB4 (rev 51)
00:0f.1 IDE interface: ServerWorks: Unknown device 0211
00:0f.2 USB Controller: ServerWorks: Unknown device 0220 (rev 04)
01:02.0 RAID bus controller: IBM Netfinity ServeRAID controller
01:03.0 Ethernet controller: Intel Corporation 82543GC Gigabit Ethernet Controller (rev 02) 01:07.0 Ethernet controller: Intel Corporation 82557 [Ethernet Pro 100] (rev 0c) 02:05.0 SCSI storage controller: QLogic Corp. QLA2200 (rev 05)
---------------------------------------
--------- dmesg (qla stuff)--------
qla2x00: Found VID=1077 DID=2200 SSVID=1077 SSDID=2
scsi1: Found a QLA2200 @ bus 2, device 0x5, irq 24, iobase 0x2100
scsi(1): Configure NVRAM parameters...
scsi(1): Verifying loaded RISC code...
scsi(1): Verifying chip...
scsi(1): Waiting for LIP to complete...
scsi(1): LIP reset occurred
scsi(1): LIP occurred.
scsi(1): LOOP UP detected
scsi1: Topology - (Loop), Host Loop address 0x7d scsi-qla0-adapter-port=210000e08b064002\; scsi-qla0-tgt-0-di-0-node=200600a0b80c3d8c\; scsi-qla0-tgt-0-di-0-port=200600a0b80c3d8d\;
scsi-qla0-tgt-0-di-0-control=00\; scsi-qla0-tgt-0-di-0-preferred=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff\; scsi1 : QLogic QLA2200 PCI to Fibre Channel Host Adapter: bus 2 device 5 irq 24
Firmware version: 2.01.37, Driver version 5.36.3 Vendor: IBM Model: 3552 Rev: 0401 Type: Direct-Access ANSI SCSI revision: 03 Vendor: IBM Model: 3552 Rev: 0401 Type: Direct-Access ANSI SCSI revision: 03 Vendor: IBM Model: 3552 Rev: 0401 Type: Direct-Access ANSI SCSI revision: 03 Vendor: IBM Model: 3552 Rev: 0401 Type: Direct-Access ANSI SCSI revision: 03
scsi(1:0:0:0): Enabled tagged queuing, queue depth 16. scsi(1:0:0:1): Enabled tagged queuing, queue depth 16. scsi(1:0:0:2): Enabled tagged queuing, queue depth 16. scsi(1:0:0:3): Enabled tagged queuing, queue depth 16. Attached scsi disk sdb at scsi1, channel 0, id 0, lun 0 Attached scsi disk sdc at scsi1, channel 0, id 0, lun 1 Attached scsi disk sdd at scsi1, channel 0, id 0, lun 2 Attached scsi disk sde at scsi1, channel 0, id 0, lun 3 SCSI device sdb: 125829120 512-byte hdwr sectors (64425 MB)
sdb: sdb1
SCSI device sdc: 125829120 512-byte hdwr sectors (64425 MB)
sdc: sdc1
SCSI device sdd: 125829120 512-byte hdwr sectors (64425 MB)
sdd: sdd1
SCSI device sde: 48599040 512-byte hdwr sectors (24883 MB)
sde: sde1
---------------------------------------
alle 3 machines have the same filesystem concept:
internal storage -> ext3 (linux and programs)
storage on fastt500 -> xfs (data only)
except the mount options, because the fileserver also need's quotas, he has:
rw,noatime,quota,usrquota,grpquota,logbufs=8,logbsize=32768
i consider the primary lotus domino server to be the machine wich has to handle the highest load of the 3, because he currently has to handle almost everything that has to do with lotus notes in your company, it's friday afternoon here, so the load isn't realy high, but it's possibly nice for you to know how much load the machine has to handle:
---------------------------------------
4:59pm up 16 days, 23:17, 3 users, load average: 0.43, 0.18, 0.11
445 processes: 441 sleeping, 4 running, 0 zombie, 0 stopped
CPU0 states: 6.13% user, 1.57% system, 0.0% nice, 91.57% idle CPU1 states: 26.19% user, 5.27% system, 0.0% nice, 68.17% idle Mem: 2061272K av, 2049636K used, 11636K free, 0K shrd, 1604K buff Swap: 1056124K av, 262532K used, 793592K free 1856420K cached
---------------------------------------
(/local/notesdata is on the fastt500)
adam:/ # lsof |grep /local/notesdata/ |wc -l
33076
---------------------------------------
adam:/ # lsof |wc -l
84604
---------------------------------------
simon.
(please CC me, i'm not subscribed to lkml)