2.6.6 lockup
Subsystems (2)
Hi all again,
After fixing up the failover issues, I got my pair of Itaniums into
production with 2.6.5 and as soon as the real world load went up, the machines started freezing. No net response, no console, only sysreq keys work.
I updated to 2.6.6 and it doesn't freeze up as often, but it's still really
bad, at least a few times a day. Unfortunately, I can't seem to figure out how to get a decent kernel trace. Apperently sysreq-Crash doesn't work in ia64. And NMI watchdog doesn't work on ia. And I can't find any info on a hardware watchdog on the mobo!
I do have some other info from sysreq on the cpu regs, memory, and processes if anyone would find that interesting.
The work load freezes up the machines under heavy streaming writes from about a 100 processes on at least 60 clients. A combined load of about 80GB/hour is enough to freeze up the machine pretty regularly.
I tried Trond's 2.6.6 patches at his website, but those brokes things considerably. Since I don't have any actual Oops messages, anyone have any experimental deadlock-fixing patches they want me to test? :)
Signature · 2 lines
Garrick Staples, Linux/HPCC Administrator University of Southern California
Re: 2.6.6 lockupy
Quoted 3 lines · Garrick Staples
I tried Trond's 2.6.6 patches at his website, but those brokes things considerably. Since I don't have any actual Oops messages, anyone have any experimental deadlock-fixing patches they want me to test? :)
Note that Trond's patches are client-side only, so shouldn't affect your servers one way or another (unless I'm misunderstanding what your setup).--b.
-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Re: 2.6.6 lockupy
On Wed, May 26, 2004 at 05:37:32PM -0400, J. Bruce Fields alleged:
Quoted 8 lines · Garrick Staples
Quoted 3 lines
I tried Trond's 2.6.6 patches at his website, but those brokes things considerably. Since I don't have any actual Oops messages, anyone have any experimental deadlock-fixing patches they want me to test? :)
Note that Trond's patches are client-side only, so shouldn't affect your servers one way or another (unless I'm misunderstanding what your setup).--b.
You didn't misunderstand... but I was at a complete loss with production machines suddenly dropping like flies. I'm now at the stage where I just try random things =P
Signature · 2 lines
Garrick Staples, Linux/HPCC Administrator University of Southern California
Re: 2.6.6 lockupy
Quoted 3 lines · Garrick Staples
You didn't misunderstand... but I was at a complete loss with production machines suddenly dropping like flies. I'm now at the stage where I just try random things =P
OK. May as well send us what information you have on the lockups, and maybe your .config while you're at it....
(Also, you mention 2.6.6 in your subject line but it sounds like this happens to you under earlier kernels as well? Are there any kernels that are OK?)
--Bruce Fields
-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Re: 2.6.6 lockupy
On Wed, May 26, 2004 at 05:55:56PM -0400, J. Bruce Fields alleged:
Quoted 7 lines · Garrick Staples
Quoted 3 lines
You didn't misunderstand... but I was at a complete loss with production machines suddenly dropping like flies. I'm now at the stage where I just try random things =P
OK. May as well send us what information you have on the lockups, and maybe your .config while you're at it....
It's more info then I feel like posting to the list, but I've got it at:
http://www-rds.usc.edu/~garrick/nfsprobs/
Quoted 3 lines
(Also, you mention 2.6.6 in your subject line but it sounds like this happens to you under earlier kernels as well? Are there any kernels that are OK?)
2.6.6 locks up less often than 2.6.5. I also had 2.6.3, but it had other scsi driver issues. 2.6.3 never saw this kind of load, but I can try it if you want.
Signature · 2 lines
Garrick Staples, Linux/HPCC Administrator University of Southern California
Re: 2.6.6 lockupy
In reply to J. Bruce FieldsOn Wed, May 26, 2004 at 05:55:56PM -0400, J. Bruce Fields alleged:
Quoted 7 lines · Garrick Staples
Quoted 3 lines
You didn't misunderstand... but I was at a complete loss with production machines suddenly dropping like flies. I'm now at the stage where I just try random things =P
OK. May as well send us what information you have on the lockups, and maybe your .config while you're at it....
Filesystem corruption from the repeated lockups finally showed up today. So I managed to get those two machines out of production for now. I'll be able to figure out to trigger the problem and hopefully get you some better info tomorrow.
Btw, the failover capabilities of 2.6 has been very well tested the last few days :) Nearly 15TB of data was swapped back and forth during heavy writes. Good job guys on that!
Btw, reiserfs+lvm2 is very resilient!
(does anyone have advice on triggering a kernel trace on ia64?)
Signature · 2 lines
Garrick Staples, Linux/HPCC Administrator University of Southern California