i686 hang on boot in userspace
Quoted 8 lines · Uwe Bugla
Hi everybody,
first of all thanks to the explanatory hints how a magic Sysrq key works – I've learned a lot.I first pressed ALT + PrintScreen + P, then ALT + PrintScreen + T.
To avoid wordwrapping or other unwanted effects please see the resulting kern.log as outline attachment.Could someone please explain to me what's behind that cryptic code? Hope I could help - still need a booting kernel, and I think I ain´t the only one.
Hmmm... I don't see anything that sticks out in the sysrq info (well, not sure if the do_wp_page() is just trace junk or not - CC'ed Peter just in case). Maybe this is related to the expand files OOM thing that Martin saw?
If you boot w/ init=/bin/bash do you also see the hang? If you execute "date" a few times, does it seem to keep proper track of time?
Also could you enable softlockup detection? (CONFIG_DETECT_SOFTLOCKUP) Which you can find under Kernel debugging in the make menuconfig.
thanks
-john
Hi,
Quoted 7 lines · Uwe Bugla
Hi everybody,
first of all thanks to the explanatory hints how a magic Sysrq key works – I've learned a lot.I first pressed ALT + PrintScreen + P, then ALT + PrintScreen + T.
To avoid wordwrapping or other unwanted effects please see the resulting kern.log as outline attachment.Could someone please explain to me what's behind that cryptic code?
It shows what the kernel is currently is doing and where it's spending the time.
First, your kernel buffer log buffer seems a little small, so not everything is captured. Could you increase the number in the "Kernel log buffer size" option (it's in the "Kernel debugging" part of the "Kernel hacking" menu).
Second, could you press ALT+PrintScreen+P a few more times (maybe around 10 at least) while the kernel hangs? This would should where the cpu is spending its time and whether it's at a single place or at different places.
Thanks.
bye, Roman
Hi,
Quoted 2 lines · Roman Zippel
Quoted 2 lines
I first pressed ALT + PrintScreen + P, then ALT + PrintScreen + T.
To avoid wordwrapping or other unwanted effects please see the resulting kern.log as outline attachment.
I almost forgot, please compress it next time, the list has a size limit.
bye, Roman
-------- Original-Nachricht --------
Datum: Mon, 17 Jul 2006 12:52:28 +0200 (CEST)
Von: Roman Zippel <[email protected]>
An: Uwe Bugla <[email protected]>
Betreff: Re: i686 hang on boot in userspace
Quoted 26 lines
Hi,
Quoted 2 lines · Uwe Bugla
Hi everybody,
first of all thanks to the explanatory hints how a magic Sysrq key works– I've learned a lot.
Quoted 3 lines
I first pressed ALT + PrintScreen + P, then ALT + PrintScreen + T. To avoid wordwrapping or other unwanted effects please see the resulting
kern.log as outline attachment.
Quoted 2 lines
Could someone please explain to me what's behind that cryptic code?
It shows what the kernel is currently is doing and where it's spending the time.
First, your kernel buffer log buffer seems a little small, so not everything is captured. Could you increase the number in the "Kernel log buffer size" option (it's in the "Kernel debugging" part of the "Kernel hacking" menu).
Second, could you press ALT+PrintScreen+P a few more times (maybe around 10 at least) while the kernel hangs? This would should where the cpu is spending its time and whether it's at a single place or at different places.
Thanks.
Quoted 1 line
bye, Roman
Hi Roman, Hi everybody else,
my boot problem is solved within kernel 2.6.18-rc1-mm2!!!
A thousands of thanks for all your efforts!
I have compared 18-rc1-mm1 and 18-rc1-mm2.
mm2 contains a patch for timer.c owning almost twice as many hunks than mm1. In so far I was sure it was a timer.c issue.
Regards
Uwe
Signature · 2 lines
Echte DSL-Flatrate dauerhaft für 0,- Euro*! "Feel free" mit GMX DSL! http://www.gmx.net/de/go/dsl
Hi,
Quoted 3 lines · Uwe Bugla
I have compared 18-rc1-mm1 and 18-rc1-mm2.
mm2 contains a patch fortimer.cowning almost twice as many hunks than mm1. In so far I was sure it was atimer.cissue.
You're still guessing, a lot more things changed between 18-rc1-mm1 and
18-rc1-mm2. It's rather unlikely that the timer changes fixed your
problem. You might want to try to revert ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.18-rc1/2.6.18-rc1-mm2/broken-out/improve-timekeeping-resume-robustness.patch to see whether the problem is back afterwards.
bye, Roman
Quoted 8 lines · Roman Zippel
Hi,
Quoted 3 lines · Uwe Bugla
I have compared 18-rc1-mm1 and 18-rc1-mm2.
mm2 contains a patch fortimer.cowning almost twice as many hunks than mm1. In so far I was sure it was atimer.cissue.
Quoted 5 lines
You're still guessing, a lot more things changed between 18-rc1-mm1 and
18-rc1-mm2. It's rather unlikely that the timer changes fixed your
problem. You might want to try to revertftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.18-rc1/2.6.18-rc1-mm2/broken-out/improve-timekeeping-resume-robustness.patchto see whether the problem is back afterwards.
I was preparing a post to lkml about a similar hang which happens during init. I also saw an error while ntpdate tried to set the time/get the time. this only happens after I've enabled the NX bit on the dual 32bit Xeons installed in the HP Proliant Server. it works flawlessly with 2.6.17.6 (CONFIG_X86_PAE and CONFIG_HIGHMEM_64) but since 2.6.18-rc2-git4 (including 2.6.18-rc2) it hangs late in the init process.
could this be related?
Hi,
Quoted 9 lines · On Mon, 17 Jul 2006, gmu 2k6 wrote:
I was preparing a post to lkml about a similar hang which happens during init. I also saw an error while ntpdate tried to set the time/get the time. this only happens after I've enabled the NX bit on the dual 32bit Xeons installed in the HP Proliant Server. it works flawlessly with 2.6.17.6 (
CONFIG_X86_PAEandCONFIG_HIGHMEM_64) but since 2.6.18-rc2-git4 (including 2.6.18-rc2) it hangs late in the init process.could this be related?
Well, it could, but without further information it's impossible to say. What error did you see with ntpdate? Could you post the kernel messages and also insert a few stack traces as mentioned earlier? Thanks.
bye, Rman
Quoted 12 lines · Roman Zippel
Hi,
Quoted 8 lines · On Mon, 17 Jul 2006, gmu 2k6 wrote:
I was preparing a post to lkml about a similar hang which happens during init. I also saw an error while ntpdate tried to set the time/get the time. this only happens after I've enabled the NX bit on the dual 32bit Xeons installed in the HP Proliant Server. it works flawlessly with 2.6.17.6 (
CONFIG_X86_PAEandCONFIG_HIGHMEM_64) but since 2.6.18-rc2-git4 (including 2.6.18-rc2) it hangs late in the init process.
Quoted 6 lines
Quoted 1 line
could this be related?
Well, it could, but without further information it's impossible to say. What error did you see with ntpdate? Could you post the kernel messages and also insert a few stack traces as mentioned earlier? Thanks.
ok, the error printed from ntpdate has to do with networking:
Running ntpdate to synchronize clockError : Temporary failure in name resolution
it then stops after printing the following tg3 initialization line:
tg3: eth0 Flow control is on for TX and on for RX.
right now I'm trying to get SysRq working (my first try with it) so that I see where it's hanging.
Quoted 20 lines · On 7/17/06, gmu 2k6 <[email protected]> wrote:
Quoted 12 lines · Roman Zippel
Hi,
Quoted 8 lines · On Mon, 17 Jul 2006, gmu 2k6 wrote:
I was preparing a post to lkml about a similar hang which happens during init. I also saw an error while ntpdate tried to set the time/get the time. this only happens after I've enabled the NX bit on the dual 32bit Xeons installed in the HP Proliant Server. it works flawlessly with 2.6.17.6 (
CONFIG_X86_PAEandCONFIG_HIGHMEM_64) but since 2.6.18-rc2-git4 (including 2.6.18-rc2) it hangs late in the init process.Quoted 6 lines
Quoted 1 line
could this be related?
Well, it could, but without further information it's impossible to say. What error did you see with ntpdate? Could you post the kernel messages and also insert a few stack traces as mentioned earlier? Thanks.
Quoted 8 lines
ok, the error printed from ntpdate has to do with networking:
Running ntpdate to synchronize clockError : Temporary failure in name resolutionit then stops after printing the following tg3 initialization line:
tg3: eth0 Flow control is on for TX and on for RX.right now I'm trying to get SysRq working (my first try with it) so that I see where it's hanging.
either I'm too dumb or there is an undocumented way to enable SysRq on bootup or the machine is really hanging hard. I'm not able use Alt+Print as nothing happens besides console showing the typed in characters ^[t.
Quoted 20 lines · On 7/17/06, gmu 2k6 <[email protected]> wrote:
Quoted 12 lines · Roman Zippel
Hi,
Quoted 8 lines · On Mon, 17 Jul 2006, gmu 2k6 wrote:
I was preparing a post to lkml about a similar hang which happens during init. I also saw an error while ntpdate tried to set the time/get the time. this only happens after I've enabled the NX bit on the dual 32bit Xeons installed in the HP Proliant Server. it works flawlessly with 2.6.17.6 (
CONFIG_X86_PAEandCONFIG_HIGHMEM_64) but since 2.6.18-rc2-git4 (including 2.6.18-rc2) it hangs late in the init process.Quoted 6 lines
Quoted 1 line
could this be related?
Well, it could, but without further information it's impossible to say. What error did you see with ntpdate? Could you post the kernel messages and also insert a few stack traces as mentioned earlier? Thanks.
Quoted 8 lines
ok, the error printed from ntpdate has to do with networking:
Running ntpdate to synchronize clockError : Temporary failure in name resolutionit then stops after printing the following tg3 initialization line:
tg3: eth0 Flow control is on for TX and on for RX.right now I'm trying to get SysRq working (my first try with it) so that I see where it's hanging.
ignore the ntpdate message, it also appears when normally booting as the tg3 interfaces are not up yet when ntpdate is run, but this is ok as I've been doing ntpdate @hourly anyway.
therefore, there must be a different issue which I'm not able to debug yet.
Hi,
Quoted 4 lines · On Mon, 17 Jul 2006, gmu 2k6 wrote:
either I'm too dumb or there is an undocumented way to enable SysRq on bootup or the machine is really hanging hard. I'm not able use Alt+Print as nothing happens besides console showing the typed in characters ^[t.
It might be a keyboard problem, try releasing Print, but keeping Alt pressed and then try another key.
bye, Roman
Quoted 11 lines · Roman Zippel
Hi,
Quoted 4 lines · On Mon, 17 Jul 2006, gmu 2k6 wrote:
either I'm too dumb or there is an undocumented way to enable SysRq on bootup or the machine is really hanging hard. I'm not able use Alt+Print as nothing happens besides console showing the typed in characters ^[t.
It might be a keyboard problem, try releasing Print, but keeping Alt pressed and then try another key.
maybe the problem is HP's Integrated Lights Out Java Applet. I will try tomorrow morning in the server room.
enabling HIGHMEM_64 to get NX bit enabled on a Xeon supporting the bit does not make the overall system slower with only 4GiB RAM installed, does it?
Quoted 10 lines · On 7/17/06, gmu 2k6 <[email protected]> wrote:
Quoted 9 lines · Roman Zippel
Hi,
Quoted 4 lines · On Mon, 17 Jul 2006, gmu 2k6 wrote:
either I'm too dumb or there is an undocumented way to enable SysRq on bootup or the machine is really hanging hard. I'm not able use Alt+Print as nothing happens besides console showing the typed in characters ^[t.
Quoted 5 lines
Quoted 2 lines
It might be a keyboard problem, try releasing Print, but keeping Alt pressed and then try another key.
maybe the problem is HP's Integrated Lights Out Java Applet. I will try tomorrow morning in the server room.
yep that Java Applet was the problem. it worked when I was physically connected by keyboard. I got the following by pressing Alt+SysRq+p but I'm not sure it helps as being in cpu_idle looks normal to me:
Pid: 0, comm: swapper
EIP: 0060 [<c0101a57>] CPU: 0
EIP: isat mwait_idle+0x2a/0x34
EAX: 00000000 EBX: c0414008 ECX: 00000000 EDX: 00000000
ESI: c0414000 EDI: c33984e4 EPP: 00004864 DS: 007b ES: 007b
CR0: 8005003b CR2: b7f818cc CR3: 375e73c0 CR4: 000006f0
[<c0101a175>] cpu_idle+0x63/0x79 [<c041a6cf>] start_kernel+0x262/0x393 [<c041a1c3>] unknown_bootoption+0x0/0x25a
Alt+SysRq+s seems to sync and write the logs to kern.log/messages but the logs vanish after reboot. Therefore for the time being I had to write it down by hand but I'm sure there's an elegant way like saving the logfiles before booting up again via a second system or livecd. Maybe there's a better way than that?
Quoted 30 lines · On 7/18/06, gmu 2k6 <[email protected]> wrote:
Quoted 10 lines · On 7/17/06, gmu 2k6 <[email protected]> wrote:
Quoted 9 lines · Roman Zippel
Hi,
Quoted 4 lines · On Mon, 17 Jul 2006, gmu 2k6 wrote:
either I'm too dumb or there is an undocumented way to enable SysRq on bootup or the machine is really hanging hard. I'm not able use Alt+Print as nothing happens besides console showing the typed in characters ^[t.
Quoted 5 lines
Quoted 2 lines
It might be a keyboard problem, try releasing Print, but keeping Alt pressed and then try another key.
maybe the problem is HP's Integrated Lights Out Java Applet. I will try tomorrow morning in the server room.
yep that Java Applet was the problem. it worked when I was physically connected by keyboard. I got the following by pressing Alt+SysRq+p but I'm not sure it helps as being in
cpu_idlelooks normal to me:
Pid: 0, comm: swapper
EIP: 0060 [<c0101a57>] CPU: 0
EIP: isatmwait_idle+0x2a/0x34
EAX: 00000000 EBX:c0414008ECX: 00000000 EDX: 00000000
ESI:c0414000EDI:c33984e4EPP: 00004864 DS: 007b ES: 007b
CR0:8005003bCR2:b7f818ccCR3:375e73c0CR4:000006f0[<c0101a175>] cpu_idle+0x63/0x79 [<c041a6cf>] start_kernel+0x262/0x393 [<c041a1c3>] unknown_bootoption+0x0/0x25a
Quoted 5 lines
Alt+SysRq+s seems to sync and write the logs to kern.log/messages but the logs vanish after reboot. Therefore for the time being I had to write it down by hand but I'm sure there's an elegant way like saving the logfiles before booting up again via a second system or livecd. Maybe there's a better way than that?
same boot problem with 2.6.18-rc1-mm2 btw, did not SysRq for that though.
the problem I have with hangs is related to changes in CFQ and that CFQ is now the default. 2.6.17-git12 had the problem but booting it with elevator=deadline fixes the hang.
symptoms encountered during git-bisecting between v2.6.17 and v2.6.18-rc1:
A hang while starting network services
B hang while trying to login
1 on remote console [not SSH] it hang after typing <uid><CR> 1 via OpenSSH it hang after typing <pwd><CR> when doing slogin root@<IP>
A is the problem I got in the first place and this seems to be the case since 2.6.17-git11 definitely although git-bisect pointed me at the following
changeset which is included since 2.6.17-git12:
caaa5f9f0a75d1dc5e812e69afdbb8720e077fd3
by Jens Axboe
titled "[PATCH] cfq-iosched: many performance fixes"
strange enough it also hangs with 2.6.17-git11 which did not include that one changeset yet.
to sum it up "elevator=deadline" fixes the problem, so CFQ seems to be buggy in the current Torvalds 2.6 tree.
therefore, sorry Roman for bugging you, Jens seems to be the right person to ping about the problem.
Quoted 15 lines · On Mon, Jul 24 2006, gmu 2k6 wrote:
the problem I have with hangs is related to changes in CFQ and that CFQ is now the default. 2.6.17-git12 had the problem but booting it with elevator=deadline fixes the hang.
symptoms encountered during git-bisecting between v2.6.17 and v2.6.18-rc1:
A hang while starting network services
B hang while trying to login
1 on remote console [not SSH] it hang after typing <uid><CR> 1 via OpenSSH it hang after typing <pwd><CR> when doing slogin root@<IP>A is the problem I got in the first place and this seems to be the case since 2.6.17-git11 definitely although git-bisect pointed me at the following
changeset which is included since 2.6.17-git12:
Quoted 6 lines
caaa5f9f0a75d1dc5e812e69afdbb8720e077fd3
by Jens Axboe
titled "[PATCH] cfq-iosched: many performance fixes"strange enough it also hangs with 2.6.17-git11 which did not include that one changeset yet.
So perhaps your bisect isn't 100% trust worthy? Can you do a manual -gitX bisect to see which 2.6.17-gitX introduced the problem?
Also please put a serial console or similar on the machine, so you can log + store the sysrq+t output.
Signature · 1 line
Jens Axboe
Quoted 20 lines · Jens Axboe
Quoted 19 lines · On Mon, Jul 24 2006, gmu 2k6 wrote:
the problem I have with hangs is related to changes in CFQ and that CFQ is now the default. 2.6.17-git12 had the problem but booting it with elevator=deadline fixes the hang.
symptoms encountered during git-bisecting between v2.6.17 and v2.6.18-rc1:
A hang while starting network services
B hang while trying to login
1 on remote console [not SSH] it hang after typing <uid><CR> 1 via OpenSSH it hang after typing <pwd><CR> when doing slogin root@<IP>A is the problem I got in the first place and this seems to be the case since 2.6.17-git11 definitely although git-bisect pointed me at the following
changeset which is included since 2.6.17-git12:
caaa5f9f0a75d1dc5e812e69afdbb8720e077fd3
by Jens Axboe
titled "[PATCH] cfq-iosched: many performance fixes"
Quoted 8 lines
Quoted 2 lines
strange enough it also hangs with 2.6.17-git11 which did not include that one changeset yet.
So perhaps your bisect isn't 100% trust worthy? Can you do a manual -gitX bisect to see which 2.6.17-gitX introduced the problem?
Also please put a serial console or similar on the machine, so you can log + store the sysrq+t output.
well I didn't say that caa....fd3 is the exact change which broke it, just that it's related to 1) CFQ changes and 2) CFQ being the default now.
I have a Remote Serial Console via HP's integrated Lights-Out Java Applet but am not sure how to enable serial console via kernel boot params (will try to find out).
I will first try to find the 2.6.17-git* revision working before bisecting it against -git11 or git12.
Quoted 32 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 22 lines · Jens Axboe
Quoted 5 lines · On Mon, Jul 24 2006, gmu 2k6 wrote:
the problem I have with hangs is related to changes in CFQ and that CFQ is now the default. 2.6.17-git12 had the problem but booting it with elevator=deadline fixes the hang.
symptoms encountered during git-bisecting between v2.6.17 and
v2.6.18-rc1:
Quoted 4 lines
A hang while starting network services
B hang while trying to login
1 on remote console [not SSH] it hang after typing <uid><CR> 1 via OpenSSH it hang after typing <pwd><CR> when doing sloginroot@<IP>
Quoted 10 lines
A is the problem I got in the first place and this seems to be the case since 2.6.17-git11 definitely although git-bisect pointed me at the following
changeset which is included since 2.6.17-git12:
caaa5f9f0a75d1dc5e812e69afdbb8720e077fd3
by Jens Axboe
titled "[PATCH] cfq-iosched: many performance fixes"Quoted 8 lines
Quoted 2 lines
strange enough it also hangs with 2.6.17-git11 which did not include that one changeset yet.
So perhaps your bisect isn't 100% trust worthy? Can you do a manual -gitX bisect to see which 2.6.17-gitX introduced the problem?
Also please put a serial console or similar on the machine, so you can log + store the sysrq+t output.
Quoted 8 lines
well I didn't say that caa....fd3 is the exact change which broke it, just that it's related to 1) CFQ changes and 2) CFQ being the default now.
I have a Remote Serial Console via HP's integrated Lights-Out Java Applet but am not sure how to enable serial console via kernel boot params (will try to find out).
I will first try to find the 2.6.17-git* revision working before bisecting it against -git11 or git12.
Thanks, would be much appreciated to try and narrow it down to a specific fix.
Are you seeing the hang on cciss?
Signature · 1 line
Jens Axboe
Quoted 50 lines · Jens Axboe
Quoted 49 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 47 lines · Jens Axboe
Quoted 19 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 18 lines · Jens Axboe
Quoted 5 lines · On Mon, Jul 24 2006, gmu 2k6 wrote:
the problem I have with hangs is related to changes in CFQ and that CFQ is now the default. 2.6.17-git12 had the problem but booting it with elevator=deadline fixes the hang.
symptoms encountered during git-bisecting between v2.6.17 and
v2.6.18-rc1:
Quoted 4 lines
A hang while starting network services
B hang while trying to login
1 on remote console [not SSH] it hang after typing <uid><CR> 1 via OpenSSH it hang after typing <pwd><CR> when doing sloginroot@<IP>
Quoted 6 lines
A is the problem I got in the first place and this seems to be the case since 2.6.17-git11 definitely although git-bisect pointed me at the following
changeset which is included since 2.6.17-git12:Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
caaa5f9f0a75d1dc5e812e69afdbb8720e077fd3
by Jens Axboe
titled "[PATCH] cfq-iosched: many performance fixes"strange enough it also hangs with 2.6.17-git11 which did not include
that
Quoted 8 lines
Quoted 7 lines
Quoted 1 line
one changeset yet.
So perhaps your bisect isn't 100% trust worthy? Can you do a manual -gitX bisect to see which 2.6.17-gitX introduced the problem?
Also please put a serial console or similar on the machine, so you can log + store the sysrq+t output.
Quoted 8 lines
well I didn't say that caa....fd3 is the exact change which broke it, just that it's related to 1) CFQ changes and 2) CFQ being the default now.
I have a Remote Serial Console via HP's integrated Lights-Out Java Applet but am not sure how to enable serial console via kernel boot params (will try to find out).
I will first try to find the 2.6.17-git* revision working before bisecting it against -git11 or git12.Thanks, would be much appreciated to try and narrow it down to a specific fix.
Are you seeing the hang on cciss?
Quoted 7 lines
Quoted 3 lines
I'm not sure it is in the cciss driver, but the SmartArray is driven by cciss.
starting git<11 boot tests in a minute now.Ok, thanks for confirming it's cciss. The bug is likely an interaction between cciss and cfq I think, so it would be very useful if you can pin point which of the cfq patches make it stall.
is there anything special about cciss or did you just deduce that it must be cciss in that particular box and are suspecting interaction problems with that driver and your CFQ changes?
Quoted 58 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 57 lines · Jens Axboe
Quoted 10 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 9 lines · Jens Axboe
Quoted 8 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 7 lines · Jens Axboe
Quoted 5 lines · On Mon, Jul 24 2006, gmu 2k6 wrote:
the problem I have with hangs is related to changes in CFQ and that CFQ is now the default. 2.6.17-git12 had the problem but booting it with elevator=deadline fixes the hang.
symptoms encountered during git-bisecting between v2.6.17 and
v2.6.18-rc1:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
A hang while starting network services
B hang while trying to login
1 on remote console [not SSH] it hang after typing <uid><CR> 1 via OpenSSH it hang after typing <pwd><CR> when doing sloginroot@<IP>
Quoted 3 lines
A is the problem I got in the first place and this seems to be the case since 2.6.17-git11 definitely although git-bisect pointed me
at
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
the following
changeset which is included since 2.6.17-git12:
caaa5f9f0a75d1dc5e812e69afdbb8720e077fd3
by Jens Axboe
titled "[PATCH] cfq-iosched: many performance fixes"strange enough it also hangs with 2.6.17-git11 which did not
include
Quoted 7 lines
Quoted 7 lines
that
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
one changeset yet.
So perhaps your bisect isn't 100% trust worthy? Can you do a manual -gitX bisect to see which 2.6.17-gitX introduced the problem?
Also please put a serial console or similar on the machine, so you
can
Quoted 19 lines
Quoted 11 lines
Quoted 10 lines
Quoted 1 line
log + store the sysrq+t output.
well I didn't say that caa....fd3 is the exact change which broke it, just that it's related to 1) CFQ changes and 2) CFQ being the default now.
I have a Remote Serial Console via HP's integrated Lights-Out Java Applet but am not sure how to enable serial console via kernel boot params (will try to find out).
I will first try to find the 2.6.17-git* revision working before bisecting it against -git11 or git12.Quoted 4 lines
Thanks, would be much appreciated to try and narrow it down to a specific fix.
Are you seeing the hang on cciss?
I'm not sure it is in the cciss driver, but the SmartArray is driven by cciss.
starting git<11 boot tests in a minute now.
Quoted 7 lines
Quoted 3 lines
Ok, thanks for confirming it's cciss. The bug is likely an interaction between cciss and cfq I think, so it would be very useful if you can pin point which of the cfq patches make it stall.
is there anything special about cciss or did you just deduce that it must be cciss in that particular box and are suspecting interaction problems with that driver and your CFQ changes?
Nothing really special about cciss, but a few months ago I had a similar discussion about cciss and a strange hang.
If possible, please also try a known bad kernel and apply the below patch and see if it still reproduces:
drivers/block/cciss.c +4 -0
@@ -2362,7 +2362,11 @@ static inline void complete_command(ctlr cmd->rq->completion_data = cmd; cmd->rq->errors = status;blk_add_trace_rq (cmd->rq->q, cmd->rq,BLK_TA_COMPLETE ); +#if 1 +cciss_softirq_done (cmd->rq); +#else blk_complete_request (cmd->rq); +#endif }/*
Signature · 1 line
Jens Axboe
Quoted 79 lines · Jens Axboe
Quoted 58 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 57 lines · Jens Axboe
Quoted 10 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 9 lines · Jens Axboe
Quoted 8 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 7 lines · Jens Axboe
Quoted 5 lines · On Mon, Jul 24 2006, gmu 2k6 wrote:
the problem I have with hangs is related to changes in CFQ and that CFQ is now the default. 2.6.17-git12 had the problem but booting it with elevator=deadline fixes the hang.
symptoms encountered during git-bisecting between v2.6.17 and
v2.6.18-rc1:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
A hang while starting network services
B hang while trying to login
1 on remote console [not SSH] it hang after typing <uid><CR> 1 via OpenSSH it hang after typing <pwd><CR> when doing sloginroot@<IP>
Quoted 3 lines
A is the problem I got in the first place and this seems to be the case since 2.6.17-git11 definitely although git-bisect pointed me
at
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
the following
changeset which is included since 2.6.17-git12:
caaa5f9f0a75d1dc5e812e69afdbb8720e077fd3
by Jens Axboe
titled "[PATCH] cfq-iosched: many performance fixes"strange enough it also hangs with 2.6.17-git11 which did not
include
Quoted 7 lines
Quoted 7 lines
that
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
one changeset yet.
So perhaps your bisect isn't 100% trust worthy? Can you do a manual -gitX bisect to see which 2.6.17-gitX introduced the problem?
Also please put a serial console or similar on the machine, so you
can
Quoted 19 lines
Quoted 11 lines
Quoted 10 lines
Quoted 1 line
log + store the sysrq+t output.
well I didn't say that caa....fd3 is the exact change which broke it, just that it's related to 1) CFQ changes and 2) CFQ being the default now.
I have a Remote Serial Console via HP's integrated Lights-Out Java Applet but am not sure how to enable serial console via kernel boot params (will try to find out).
I will first try to find the 2.6.17-git* revision working before bisecting it against -git11 or git12.Quoted 4 lines
Thanks, would be much appreciated to try and narrow it down to a specific fix.
Are you seeing the hang on cciss?
I'm not sure it is in the cciss driver, but the SmartArray is driven by cciss.
starting git<11 boot tests in a minute now.Quoted 7 lines
Quoted 3 lines
Ok, thanks for confirming it's cciss. The bug is likely an interaction between cciss and cfq I think, so it would be very useful if you can pin point which of the cfq patches make it stall.
is there anything special about cciss or did you just deduce that it must be cciss in that particular box and are suspecting interaction problems with that driver and your CFQ changes?
Nothing really special about cciss, but a few months ago I had a similar discussion about cciss and a strange hang.
If possible, please also try a known bad kernel and apply the below patch and see if it still reproduces:
drivers/block/cciss.c +0
-0@@ -2362,7 +2362,11 @@ static inline void complete_command(ctlr cmd->rq->completion_data = cmd;
drivers/block/cciss.c static inline void complete_command(ctlr · line 2372
Quoted 10 lines
drivers/block/cciss.c +4
-0cmd->rq->errors = status;blk_add_trace_rq (cmd->rq->q, cmd->rq,BLK_TA_COMPLETE ); +#if 1 +cciss_softirq_done (cmd->rq); +#else blk_complete_request (cmd->rq); +#endif }/*
manually nailed it down to 2.6.17-git7 being the first broken revision. going to try whether Linus' git tree knows the -git revisions and do a bisect otherwise interdiff and looking for CFQ or cciss changes as best I can.
Quoted 89 lines · On 7/25/06, gmu 2k6 <[email protected]> wrote:
Quoted 88 lines · Jens Axboe
Quoted 58 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 57 lines · Jens Axboe
Quoted 10 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 9 lines · Jens Axboe
Quoted 8 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 7 lines · Jens Axboe
Quoted 5 lines · On Mon, Jul 24 2006, gmu 2k6 wrote:
the problem I have with hangs is related to changes in CFQ and that CFQ is now the default. 2.6.17-git12 had the problem but booting it with elevator=deadline fixes the hang.
symptoms encountered during git-bisecting between v2.6.17 and
v2.6.18-rc1:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
A hang while starting network services
B hang while trying to login
1 on remote console [not SSH] it hang after typing <uid><CR> 1 via OpenSSH it hang after typing <pwd><CR> when doing sloginroot@<IP>
Quoted 3 lines
A is the problem I got in the first place and this seems to be the case since 2.6.17-git11 definitely although git-bisect pointed me
at
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
the following
changeset which is included since 2.6.17-git12:
caaa5f9f0a75d1dc5e812e69afdbb8720e077fd3
by Jens Axboe
titled "[PATCH] cfq-iosched: many performance fixes"strange enough it also hangs with 2.6.17-git11 which did not
include
Quoted 7 lines
Quoted 7 lines
that
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
one changeset yet.
So perhaps your bisect isn't 100% trust worthy? Can you do a manual -gitX bisect to see which 2.6.17-gitX introduced the problem?
Also please put a serial console or similar on the machine, so you
can
Quoted 19 lines
Quoted 11 lines
Quoted 10 lines
Quoted 1 line
log + store the sysrq+t output.
well I didn't say that caa....fd3 is the exact change which broke it, just that it's related to 1) CFQ changes and 2) CFQ being the default now.
I have a Remote Serial Console via HP's integrated Lights-Out Java Applet but am not sure how to enable serial console via kernel boot params (will try to find out).
I will first try to find the 2.6.17-git* revision working before bisecting it against -git11 or git12.Quoted 4 lines
Thanks, would be much appreciated to try and narrow it down to a specific fix.
Are you seeing the hang on cciss?
I'm not sure it is in the cciss driver, but the SmartArray is driven by cciss.
starting git<11 boot tests in a minute now.Quoted 7 lines
Quoted 3 lines
Ok, thanks for confirming it's cciss. The bug is likely an interaction between cciss and cfq I think, so it would be very useful if you can pin point which of the cfq patches make it stall.
is there anything special about cciss or did you just deduce that it must be cciss in that particular box and are suspecting interaction problems with that driver and your CFQ changes?
Nothing really special about cciss, but a few months ago I had a similar discussion about cciss and a strange hang.
If possible, please also try a known bad kernel and apply the below patch and see if it still reproduces:
drivers/block/cciss.c +4
-0@@ -2362,7 +2362,11 @@ static inline void complete_command(ctlr cmd->rq->completion_data = cmd; cmd->rq->errors = status;blk_add_trace_rq (cmd->rq->q, cmd->rq,BLK_TA_COMPLETE ); +#if 1 +cciss_softirq_done (cmd->rq); +#else blk_complete_request (cmd->rq); +#endif }
Quoted 5 lines
drivers/block/cciss.c static inline void complete_command(ctlr · line 2372
Quoted 1 line
drivers/block/cciss.c +0
-0/* manually nailed it down to 2.6.17-git7 being the first broken revision. going to try whether Linus' git tree knows the -git revisions and do a bisect otherwise interdiff and looking for CFQ or cciss changes as best I can.
oops, doing git-status while running 2.6.17-git6 seems to have locked the box again :D, ping works though. *sigh*. Jens I will try your cciss.c change now.
Quoted 96 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 95 lines · Jens Axboe
Quoted 7 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 6 lines · Jens Axboe
Quoted 5 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 4 lines · Jens Axboe
Quoted 3 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 2 lines · Jens Axboe
Quoted 1 line · On Mon, Jul 24 2006, gmu 2k6 wrote:
the problem I have with hangs is related to changes in CFQ and
that
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
CFQ is now the default. 2.6.17-git12 had the problem but booting it with elevator=deadline fixes the hang.
symptoms encountered during git-bisecting between v2.6.17 and
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
v2.6.18-rc1:
Quoted 4 lines
A hang while starting network services
B hang while trying to login
1 on remote console [not SSH] it hang after typing <uid><CR> 1 via OpenSSH it hang after typing <pwd><CR> when doing sloginroot@<IP>
Quoted 2 lines
A is the problem I got in the first place and this seems to be
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
case since 2.6.17-git11 definitely although git-bisect pointed
me
Quoted 8 lines
Quoted 8 lines
at
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
the following
changeset which is included since 2.6.17-git12:
caaa5f9f0a75d1dc5e812e69afdbb8720e077fd3
by Jens Axboe
titled "[PATCH] cfq-iosched: many performance fixes"Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
strange enough it also hangs with 2.6.17-git11 which did not
include
Quoted 4 lines
Quoted 4 lines
that
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
one changeset yet.
So perhaps your bisect isn't 100% trust worthy? Can you do a
manual
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
-gitX bisect to see which 2.6.17-gitX introduced the problem?
Also please put a serial console or similar on the machine, so you
can
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
log + store the sysrq+t output.
well I didn't say that caa....fd3 is the exact change which broke
it,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
just that it's related to 1) CFQ changes and 2) CFQ being the
default
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
now.
I have a Remote Serial Console via HP's integrated Lights-Out Java Applet but am not sure how to enable serial console via kernel boot params (will try to find out).
I will first try to find the 2.6.17-git* revision working before bisecting it against -git11 or git12.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Thanks, would be much appreciated to try and narrow it down to a specific fix.
Are you seeing the hang on cciss?
I'm not sure it is in the cciss driver, but the SmartArray is driven
by
Quoted 10 lines
Quoted 6 lines
Quoted 2 lines
cciss.
starting git<11 boot tests in a minute now.Ok, thanks for confirming it's cciss. The bug is likely an interaction between cciss and cfq I think, so it would be very useful if you can pin point which of the cfq patches make it stall.
is there anything special about cciss or did you just deduce that it must be cciss in that particular box and are suspecting interaction problems with that driver and your CFQ changes?
Nothing really special about cciss, but a few months ago I had a similar discussion about cciss and a strange hang.
If possible, please also try a known bad kernel and apply the below patch and see if it still reproduces:
drivers/block/cciss.c +4
-0@@ -2362,7 +2362,11 @@ static inline void complete_command(ctlr cmd->rq->completion_data = cmd; cmd->rq->errors = status;blk_add_trace_rq (cmd->rq->q, cmd->rq,BLK_TA_COMPLETE ); +#if 1 +cciss_softirq_done (cmd->rq); +#else blk_complete_request (cmd->rq); +#endif }
Quoted 6 lines
drivers/block/cciss.c static inline void complete_command(ctlr · line 2372
Quoted 1 line
drivers/block/cciss.c +0
-0/* manually nailed it down to 2.6.17-git7 being the first broken revision. going to try whether Linus' git tree knows the -git revisions and do a bisect
otherwise interdiff and looking for CFQ or cciss changes as best I can.
Hmm, there are no cfq/cciss changes between git6 and git7. Some SCSI changes, though. Are you using SCSI for anything?
We really need that sysrq-t dump.
Signature · 1 line
Jens Axboe
Quoted 110 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 102 lines · On 7/25/06, gmu 2k6 <[email protected]> wrote:
Quoted 8 lines · Jens Axboe
Quoted 7 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 6 lines · Jens Axboe
Quoted 5 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 4 lines · Jens Axboe
Quoted 3 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 2 lines · Jens Axboe
Quoted 1 line · On Mon, Jul 24 2006, gmu 2k6 wrote:
the problem I have with hangs is related to changes in CFQ
and that
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
CFQ is now the default. 2.6.17-git12 had the problem but
booting
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
it with elevator=deadline fixes the hang.
symptoms encountered during git-bisecting between v2.6.17 and
v2.6.18-rc1:
Quoted 4 lines
A hang while starting network services
B hang while trying to login
1 on remote console [not SSH] it hang after typing <uid><CR> 1 via OpenSSH it hang after typing <pwd><CR> when doingslogin
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
root@<IP>
Quoted 2 lines
A is the problem I got in the first place and this seems to
be the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
case since 2.6.17-git11 definitely although git-bisect
pointed me
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
at
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
the following
changeset which is included since 2.6.17-git12:
caaa5f9f0a75d1dc5e812e69afdbb8720e077fd3
by Jens Axboe
titled "[PATCH] cfq-iosched: many performance fixes"Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
strange enough it also hangs with 2.6.17-git11 which did not
include
Quoted 4 lines
Quoted 4 lines
that
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
one changeset yet.
So perhaps your bisect isn't 100% trust worthy? Can you do a
manual
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
-gitX bisect to see which 2.6.17-gitX introduced the problem?
Also please put a serial console or similar on the machine, so
you
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
can
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
log + store the sysrq+t output.
well I didn't say that caa....fd3 is the exact change which
broke it,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
just that it's related to 1) CFQ changes and 2) CFQ being the
default
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
now.
I have a Remote Serial Console via HP's integrated Lights-OutJava
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Applet but am not sure how to enable serial console via kernel
boot
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
Quoted 3 lines
params (will try to find out).
I will first try to find the 2.6.17-git* revision working before bisecting it against -git11 or git12.Thanks, would be much appreciated to try and narrow it down to a specific fix.
Are you seeing the hang on cciss?
I'm not sure it is in the cciss driver, but the SmartArray is
driven by
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
cciss.
starting git<11 boot tests in a minute now.Ok, thanks for confirming it's cciss. The bug is likely an interaction between cciss and cfq I think, so it would be very useful if you can
pin
Quoted 27 lines
Quoted 5 lines
Quoted 1 line
point which of the cfq patches make it stall.
is there anything special about cciss or did you just deduce that it must be cciss in that particular box and are suspecting interaction problems with that driver and your CFQ changes?
Nothing really special about cciss, but a few months ago I had a similar discussion about cciss and a strange hang.
If possible, please also try a known bad kernel and apply the below patch and see if it still reproduces:
drivers/block/cciss.c +4
-0@@ -2362,7 +2362,11 @@ static inline void complete_command(ctlr cmd->rq->completion_data = cmd; cmd->rq->errors = status;blk_add_trace_rq (cmd->rq->q, cmd->rq,BLK_TA_COMPLETE ); +#if 1 +cciss_softirq_done (cmd->rq); +#else blk_complete_request (cmd->rq); +#endif }Quoted 6 lines
drivers/block/cciss.c static inline void complete_command(ctlr · line 2372
Quoted 1 line
drivers/block/cciss.c +0
-0/* manually nailed it down to 2.6.17-git7 being the first broken revision. going to try whether Linus' git tree knows the -git revisions and do a bisect
otherwise interdiff and looking for CFQ or cciss changes as best I can.
Quoted 4 lines
oops, doing git-status while running 2.6.17-git6 seems to have locked the box
again :D, ping works though. *sigh*. Jens I will try yourcciss.cchange now.
I guess that's a good thing, if it was git7 that introduced it, then things are looking fishy.
Signature · 1 line
Jens Axboe
Quoted 90 lines · On 7/25/06, gmu 2k6 <[email protected]> wrote:
Quoted 89 lines · On 7/25/06, gmu 2k6 <[email protected]> wrote:
Quoted 88 lines · Jens Axboe
Quoted 58 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 57 lines · Jens Axboe
Quoted 10 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 9 lines · Jens Axboe
Quoted 8 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 7 lines · Jens Axboe
Quoted 5 lines · On Mon, Jul 24 2006, gmu 2k6 wrote:
the problem I have with hangs is related to changes in CFQ and that CFQ is now the default. 2.6.17-git12 had the problem but booting it with elevator=deadline fixes the hang.
symptoms encountered during git-bisecting between v2.6.17 and
v2.6.18-rc1:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
A hang while starting network services
B hang while trying to login
1 on remote console [not SSH] it hang after typing <uid><CR> 1 via OpenSSH it hang after typing <pwd><CR> when doing sloginroot@<IP>
Quoted 3 lines
A is the problem I got in the first place and this seems to be the case since 2.6.17-git11 definitely although git-bisect pointed me
at
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
the following
changeset which is included since 2.6.17-git12:
caaa5f9f0a75d1dc5e812e69afdbb8720e077fd3
by Jens Axboe
titled "[PATCH] cfq-iosched: many performance fixes"strange enough it also hangs with 2.6.17-git11 which did not
include
Quoted 7 lines
Quoted 7 lines
that
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
one changeset yet.
So perhaps your bisect isn't 100% trust worthy? Can you do a manual -gitX bisect to see which 2.6.17-gitX introduced the problem?
Also please put a serial console or similar on the machine, so you
can
Quoted 19 lines
Quoted 11 lines
Quoted 10 lines
Quoted 1 line
log + store the sysrq+t output.
well I didn't say that caa....fd3 is the exact change which broke it, just that it's related to 1) CFQ changes and 2) CFQ being the default now.
I have a Remote Serial Console via HP's integrated Lights-Out Java Applet but am not sure how to enable serial console via kernel boot params (will try to find out).
I will first try to find the 2.6.17-git* revision working before bisecting it against -git11 or git12.Quoted 4 lines
Thanks, would be much appreciated to try and narrow it down to a specific fix.
Are you seeing the hang on cciss?
I'm not sure it is in the cciss driver, but the SmartArray is driven by cciss.
starting git<11 boot tests in a minute now.Quoted 7 lines
Quoted 3 lines
Ok, thanks for confirming it's cciss. The bug is likely an interaction between cciss and cfq I think, so it would be very useful if you can pin point which of the cfq patches make it stall.
is there anything special about cciss or did you just deduce that it must be cciss in that particular box and are suspecting interaction problems with that driver and your CFQ changes?
Nothing really special about cciss, but a few months ago I had a similar discussion about cciss and a strange hang.
If possible, please also try a known bad kernel and apply the below patch and see if it still reproduces:
drivers/block/cciss.c +4
-0@@ -2362,7 +2362,11 @@ static inline void complete_command(ctlr cmd->rq->completion_data = cmd; cmd->rq->errors = status;blk_add_trace_rq (cmd->rq->q, cmd->rq,BLK_TA_COMPLETE ); +#if 1 +cciss_softirq_done (cmd->rq); +#else blk_complete_request (cmd->rq); +#endif }
Quoted 8 lines
Quoted 5 lines
drivers/block/cciss.c static inline void complete_command(ctlr · line 2372
Quoted 1 line
drivers/block/cciss.c +0
-0/* manually nailed it down to 2.6.17-git7 being the first broken revision. going to try whether Linus' git tree knows the -git revisions and do a bisect otherwise interdiff and looking for CFQ or cciss changes as best I can.
oops, doing git-status while running 2.6.17-git6 seems to have locked the box again :D, ping works though. *sigh*. Jens I will try your
cciss.cchange now.
ok, let's nail it to 2.6.17-git5 instead as it survived git status compared to -git6
which seems to have correctly booted by accident the lastime. timing issues I guess.
Quoted 121 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 112 lines · On 7/25/06, gmu 2k6 <[email protected]> wrote:
Quoted 9 lines · On 7/25/06, gmu 2k6 <[email protected]> wrote:
Quoted 8 lines · Jens Axboe
Quoted 7 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 6 lines · Jens Axboe
Quoted 5 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 4 lines · Jens Axboe
Quoted 3 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 2 lines · Jens Axboe
Quoted 1 line · On Mon, Jul 24 2006, gmu 2k6 wrote:
the problem I have with hangs is related to changes in CFQ
and that
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
CFQ is now the default. 2.6.17-git12 had the problem but
booting
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
it with elevator=deadline fixes the hang.
symptoms encountered during git-bisecting between v2.6.17
and
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
v2.6.18-rc1:
Quoted 3 lines
A hang while starting network services
B hang while trying to login
1 on remote console [not SSH] it hang after typing<uid><CR>
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
1 via OpenSSH it hang after typing <pwd><CR> when doing
slogin
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
root@<IP>
Quoted 2 lines
A is the problem I got in the first place and this seems to
be the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
case since 2.6.17-git11 definitely although git-bisect
pointed me
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
at
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
the following
changeset which is included since 2.6.17-git12:
caaa5f9f0a75d1dc5e812e69afdbb8720e077fd3
by Jens Axboe
titled "[PATCH] cfq-iosched: many performance fixes"Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
strange enough it also hangs with 2.6.17-git11 which did not
include
Quoted 4 lines
Quoted 4 lines
that
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
one changeset yet.
So perhaps your bisect isn't 100% trust worthy? Can you do a
manual
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
-gitX bisect to see which 2.6.17-gitX introduced the problem?
Also please put a serial console or similar on the machine,
so you
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
can
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
log + store the sysrq+t output.
well I didn't say that caa....fd3 is the exact change which
broke it,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
just that it's related to 1) CFQ changes and 2) CFQ being the
default
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
now.
I have a Remote Serial Console via HP's integrated Lights-OutJava
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Applet but am not sure how to enable serial console via kernel
boot
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
params (will try to find out).
I will first try to find the 2.6.17-git* revision workingbefore
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
bisecting it against -git11 or git12.
Thanks, would be much appreciated to try and narrow it down to a specific fix.
Are you seeing the hang on cciss?
I'm not sure it is in the cciss driver, but the SmartArray is
driven by
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
cciss.
starting git<11 boot tests in a minute now.Ok, thanks for confirming it's cciss. The bug is likely an
interaction
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
between cciss and cfq I think, so it would be very useful if you
can pin
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
point which of the cfq patches make it stall.
is there anything special about cciss or did you just deduce that it must be cciss in that particular box and are suspecting interaction problems with that driver and your CFQ changes?
Nothing really special about cciss, but a few months ago I had a
similar
Quoted 20 lines
Quoted 20 lines
discussion about cciss and a strange hang.
If possible, please also try a known bad kernel and apply the below patch and see if it still reproduces:
drivers/block/cciss.c +4
-0@@ -2362,7 +2362,11 @@ static inline void complete_command(ctlr cmd->rq->completion_data = cmd; cmd->rq->errors = status;blk_add_trace_rq (cmd->rq->q, cmd->rq,BLK_TA_COMPLETE ); +#if 1 +cciss_softirq_done (cmd->rq); +#else blk_complete_request (cmd->rq); +#endif }Quoted 4 lines
drivers/block/cciss.c static inline void complete_command(ctlr · line 2372
Quoted 1 line
drivers/block/cciss.c +0
-0/* manually nailed it down to 2.6.17-git7 being the first broken revision. going to try whether Linus' git tree knows the -git revisions and do a
Quoted 7 lines
bisect
Quoted 1 line
otherwise interdiff and looking for CFQ or cciss changes as best I can.
oops, doing git-status while running 2.6.17-git6 seems to have locked the box
again :D, ping works though. *sigh*. Jens I will try yourcciss.cchange now.
Quoted 4 lines
ok, let's nail it to 2.6.17-git5 instead as it survived git status compared to -git6
which seems to have correctly booted by accident the lastime. timing issues I guess.
Then please also try the cciss patch, as suggested.
Signature · 1 line
Jens Axboe
Quoted 79 lines · Jens Axboe
Quoted 58 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 57 lines · Jens Axboe
Quoted 10 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 9 lines · Jens Axboe
Quoted 8 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 7 lines · Jens Axboe
Quoted 5 lines · On Mon, Jul 24 2006, gmu 2k6 wrote:
the problem I have with hangs is related to changes in CFQ and that CFQ is now the default. 2.6.17-git12 had the problem but booting it with elevator=deadline fixes the hang.
symptoms encountered during git-bisecting between v2.6.17 and
v2.6.18-rc1:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
A hang while starting network services
B hang while trying to login
1 on remote console [not SSH] it hang after typing <uid><CR> 1 via OpenSSH it hang after typing <pwd><CR> when doing sloginroot@<IP>
Quoted 3 lines
A is the problem I got in the first place and this seems to be the case since 2.6.17-git11 definitely although git-bisect pointed me
at
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
the following
changeset which is included since 2.6.17-git12:
caaa5f9f0a75d1dc5e812e69afdbb8720e077fd3
by Jens Axboe
titled "[PATCH] cfq-iosched: many performance fixes"strange enough it also hangs with 2.6.17-git11 which did not
include
Quoted 7 lines
Quoted 7 lines
that
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
one changeset yet.
So perhaps your bisect isn't 100% trust worthy? Can you do a manual -gitX bisect to see which 2.6.17-gitX introduced the problem?
Also please put a serial console or similar on the machine, so you
can
Quoted 19 lines
Quoted 11 lines
Quoted 10 lines
Quoted 1 line
log + store the sysrq+t output.
well I didn't say that caa....fd3 is the exact change which broke it, just that it's related to 1) CFQ changes and 2) CFQ being the default now.
I have a Remote Serial Console via HP's integrated Lights-Out Java Applet but am not sure how to enable serial console via kernel boot params (will try to find out).
I will first try to find the 2.6.17-git* revision working before bisecting it against -git11 or git12.Quoted 4 lines
Thanks, would be much appreciated to try and narrow it down to a specific fix.
Are you seeing the hang on cciss?
I'm not sure it is in the cciss driver, but the SmartArray is driven by cciss.
starting git<11 boot tests in a minute now.Quoted 7 lines
Quoted 3 lines
Ok, thanks for confirming it's cciss. The bug is likely an interaction between cciss and cfq I think, so it would be very useful if you can pin point which of the cfq patches make it stall.
is there anything special about cciss or did you just deduce that it must be cciss in that particular box and are suspecting interaction problems with that driver and your CFQ changes?
Nothing really special about cciss, but a few months ago I had a similar discussion about cciss and a strange hang.
If possible, please also try a known bad kernel and apply the below patch and see if it still reproduces:
drivers/block/cciss.c +0
-0@@ -2362,7 +2362,11 @@ static inline void complete_command(ctlr cmd->rq->completion_data = cmd;
drivers/block/cciss.c static inline void complete_command(ctlr · line 2372
Quoted 10 lines
drivers/block/cciss.c +4
-0cmd->rq->errors = status;blk_add_trace_rq (cmd->rq->q, cmd->rq,BLK_TA_COMPLETE ); +#if 1 +cciss_softirq_done (cmd->rq); +#else blk_complete_request (cmd->rq); +#endif }/*
I patched Linus' HEAD/trunk/master tree with the following and it stuck in cciss init.
I hope I didn't get your diff wrong:
drivers/block/cciss.c +5 -1
@@ -2362,7 +2362,11 @@ static inline void complete_command(ctlr cmd->rq->completion_data = cmd; cmd->rq->errors = status;blk_add_trace_rq (cmd->rq->q, cmd->rq,BLK_TA_COMPLETE ); -blk_complete_request (cmd->rq); +#if 1 +cciss_softirq_done (cmd->rq); +#else +blk_complete_request (cmd->rq); +#endif
}
/*
now I'm really going to try to get the remote serial console working.
Quoted 96 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 95 lines · Jens Axboe
Quoted 7 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 6 lines · Jens Axboe
Quoted 5 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 4 lines · Jens Axboe
Quoted 3 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 2 lines · Jens Axboe
Quoted 1 line · On Mon, Jul 24 2006, gmu 2k6 wrote:
the problem I have with hangs is related to changes in CFQ and
that
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
CFQ is now the default. 2.6.17-git12 had the problem but booting it with elevator=deadline fixes the hang.
symptoms encountered during git-bisecting between v2.6.17 and
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
v2.6.18-rc1:
Quoted 4 lines
A hang while starting network services
B hang while trying to login
1 on remote console [not SSH] it hang after typing <uid><CR> 1 via OpenSSH it hang after typing <pwd><CR> when doing sloginroot@<IP>
Quoted 2 lines
A is the problem I got in the first place and this seems to be
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
case since 2.6.17-git11 definitely although git-bisect pointed
me
Quoted 8 lines
Quoted 8 lines
at
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
the following
changeset which is included since 2.6.17-git12:
caaa5f9f0a75d1dc5e812e69afdbb8720e077fd3
by Jens Axboe
titled "[PATCH] cfq-iosched: many performance fixes"Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
strange enough it also hangs with 2.6.17-git11 which did not
include
Quoted 4 lines
Quoted 4 lines
that
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
one changeset yet.
So perhaps your bisect isn't 100% trust worthy? Can you do a
manual
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
-gitX bisect to see which 2.6.17-gitX introduced the problem?
Also please put a serial console or similar on the machine, so you
can
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
log + store the sysrq+t output.
well I didn't say that caa....fd3 is the exact change which broke
it,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
just that it's related to 1) CFQ changes and 2) CFQ being the
default
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
now.
I have a Remote Serial Console via HP's integrated Lights-Out Java Applet but am not sure how to enable serial console via kernel boot params (will try to find out).
I will first try to find the 2.6.17-git* revision working before bisecting it against -git11 or git12.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Thanks, would be much appreciated to try and narrow it down to a specific fix.
Are you seeing the hang on cciss?
I'm not sure it is in the cciss driver, but the SmartArray is driven
by
Quoted 10 lines
Quoted 6 lines
Quoted 2 lines
cciss.
starting git<11 boot tests in a minute now.Ok, thanks for confirming it's cciss. The bug is likely an interaction between cciss and cfq I think, so it would be very useful if you can pin point which of the cfq patches make it stall.
is there anything special about cciss or did you just deduce that it must be cciss in that particular box and are suspecting interaction problems with that driver and your CFQ changes?
Nothing really special about cciss, but a few months ago I had a similar discussion about cciss and a strange hang.
If possible, please also try a known bad kernel and apply the below patch and see if it still reproduces:
drivers/block/cciss.c +4
-0@@ -2362,7 +2362,11 @@ static inline void complete_command(ctlr cmd->rq->completion_data = cmd; cmd->rq->errors = status;blk_add_trace_rq (cmd->rq->q, cmd->rq,BLK_TA_COMPLETE ); +#if 1 +cciss_softirq_done (cmd->rq); +#else blk_complete_request (cmd->rq); +#endif }
Quoted 5 lines
drivers/block/cciss.c static inline void complete_command(ctlr · line 2372
Quoted 1 line
drivers/block/cciss.c +0
-0/* I patched Linus' HEAD/trunk/master tree with the following and it stuck in cciss init.
I hope I didn't get your diff wrong:
Doh, sorry about that - it needs an unlocking change as well. Try this one.
Quoted 1 line
now I'm really going to try to get the remote serial console working.
Thanks!
drivers/block/cciss.c +4 -2
@@ -1261,10 +1261,8 @@ #ifdef CCISS_DEBUG#endif /* CCISS_DEBUG */ add_disk_randomness (rq->rq_disk); -spin_lock_irqsave (&h->lock, flags);end_that_request_last (rq, rq->errors);cmd_free (h, cmd,1 ); -spin_unlock_irqrestore (&h->lock, flags); }/* This function will check the usage_count of the drive to be updated/added.
@@ -2362,7 +2360,11 @@ static inline void complete_command(ctlr cmd->rq->completion_data = cmd; cmd->rq->errors = status;blk_add_trace_rq (cmd->rq->q, cmd->rq,BLK_TA_COMPLETE ); +#if 1 +cciss_softirq_done (cmd->rq); +#else blk_complete_request (cmd->rq); +#endif }/*
Signature · 1 line
Jens Axboe
Quoted 104 lines · Jens Axboe
Quoted 96 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 95 lines · Jens Axboe
Quoted 7 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 6 lines · Jens Axboe
Quoted 5 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 4 lines · Jens Axboe
Quoted 3 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 2 lines · Jens Axboe
Quoted 1 line · On Mon, Jul 24 2006, gmu 2k6 wrote:
the problem I have with hangs is related to changes in CFQ and
that
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
CFQ is now the default. 2.6.17-git12 had the problem but booting it with elevator=deadline fixes the hang.
symptoms encountered during git-bisecting between v2.6.17 and
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
v2.6.18-rc1:
Quoted 4 lines
A hang while starting network services
B hang while trying to login
1 on remote console [not SSH] it hang after typing <uid><CR> 1 via OpenSSH it hang after typing <pwd><CR> when doing sloginroot@<IP>
Quoted 2 lines
A is the problem I got in the first place and this seems to be
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
case since 2.6.17-git11 definitely although git-bisect pointed
me
Quoted 8 lines
Quoted 8 lines
at
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
the following
changeset which is included since 2.6.17-git12:
caaa5f9f0a75d1dc5e812e69afdbb8720e077fd3
by Jens Axboe
titled "[PATCH] cfq-iosched: many performance fixes"Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
strange enough it also hangs with 2.6.17-git11 which did not
include
Quoted 4 lines
Quoted 4 lines
that
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
one changeset yet.
So perhaps your bisect isn't 100% trust worthy? Can you do a
manual
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
-gitX bisect to see which 2.6.17-gitX introduced the problem?
Also please put a serial console or similar on the machine, so you
can
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
log + store the sysrq+t output.
well I didn't say that caa....fd3 is the exact change which broke
it,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
just that it's related to 1) CFQ changes and 2) CFQ being the
default
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
now.
I have a Remote Serial Console via HP's integrated Lights-Out Java Applet but am not sure how to enable serial console via kernel boot params (will try to find out).
I will first try to find the 2.6.17-git* revision working before bisecting it against -git11 or git12.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Thanks, would be much appreciated to try and narrow it down to a specific fix.
Are you seeing the hang on cciss?
I'm not sure it is in the cciss driver, but the SmartArray is driven
by
Quoted 10 lines
Quoted 6 lines
Quoted 2 lines
cciss.
starting git<11 boot tests in a minute now.Ok, thanks for confirming it's cciss. The bug is likely an interaction between cciss and cfq I think, so it would be very useful if you can pin point which of the cfq patches make it stall.
is there anything special about cciss or did you just deduce that it must be cciss in that particular box and are suspecting interaction problems with that driver and your CFQ changes?
Nothing really special about cciss, but a few months ago I had a similar discussion about cciss and a strange hang.
If possible, please also try a known bad kernel and apply the below patch and see if it still reproduces:
drivers/block/cciss.c +4
-0@@ -2362,7 +2362,11 @@ static inline void complete_command(ctlr cmd->rq->completion_data = cmd; cmd->rq->errors = status;blk_add_trace_rq (cmd->rq->q, cmd->rq,BLK_TA_COMPLETE ); +#if 1 +cciss_softirq_done (cmd->rq); +#else blk_complete_request (cmd->rq); +#endif }Quoted 6 lines
drivers/block/cciss.c static inline void complete_command(ctlr · line 2372
Quoted 1 line
drivers/block/cciss.c +0
-0/* manually nailed it down to 2.6.17-git7 being the first broken revision. going to try whether Linus' git tree knows the -git revisions and do a bisect
otherwise interdiff and looking for CFQ or cciss changes as best I can.
Quoted 2 lines
Hmm, there are no cfq/cciss changes between git6 and git7. Some SCSI changes, though. Are you using SCSI for anything?
I thought cciss (SmartArray) was SCSI, isn't it? I guess you mean "no James Bottomley changes in the SCSI layer".
Quoted 1 line
We really need that sysrq-t dump.
I'm not able to get the virtual remote serial console working, so I will try to go down to the datacenter and do "1) SysRq-t 2) SysRq-S 3) reboot with livecd and get the content of the synced kern.log which should contain the SysRq-t output (hopefully).
Quoted 118 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 117 lines · Jens Axboe
Quoted 9 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 8 lines · Jens Axboe
Quoted 7 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 6 lines · Jens Axboe
Quoted 5 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 4 lines · Jens Axboe
Quoted 3 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 2 lines · Jens Axboe
Quoted 1 line · On Mon, Jul 24 2006, gmu 2k6 wrote:
the problem I have with hangs is related to changes in CFQ
and
Quoted 2 lines
Quoted 2 lines
that
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
CFQ is now the default. 2.6.17-git12 had the problem but
booting
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
it with elevator=deadline fixes the hang.
symptoms encountered during git-bisecting between v2.6.17 and
v2.6.18-rc1:
Quoted 3 lines
A hang while starting network services
B hang while trying to login
1 on remote console [not SSH] it hang after typing<uid><CR>
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
1 via OpenSSH it hang after typing <pwd><CR> when doing
slogin
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
root@<IP>
Quoted 2 lines
A is the problem I got in the first place and this seems to
be
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
case since 2.6.17-git11 definitely although git-bisect
pointed
Quoted 11 lines
Quoted 11 lines
me
Quoted 10 lines
Quoted 10 lines
at
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
the following
changeset which is included since 2.6.17-git12:
caaa5f9f0a75d1dc5e812e69afdbb8720e077fd3
by Jens Axboe
titled "[PATCH] cfq-iosched: many performance fixes"strange enough it also hangs with 2.6.17-git11 which did not
include
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
that
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
one changeset yet.
So perhaps your bisect isn't 100% trust worthy? Can you do a
manual
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
-gitX bisect to see which 2.6.17-gitX introduced the problem?
Also please put a serial console or similar on the machine, so
you
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
can
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
log + store the sysrq+t output.
well I didn't say that caa....fd3 is the exact change which
broke
Quoted 5 lines
Quoted 5 lines
it,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
just that it's related to 1) CFQ changes and 2) CFQ being the
default
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
now.
I have a Remote Serial Console via HP's integrated Lights-OutJava
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Applet but am not sure how to enable serial console via kernel
boot
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
Quoted 3 lines
params (will try to find out).
I will first try to find the 2.6.17-git* revision working before bisecting it against -git11 or git12.Thanks, would be much appreciated to try and narrow it down to a specific fix.
Are you seeing the hang on cciss?
I'm not sure it is in the cciss driver, but the SmartArray is
driven
Quoted 5 lines
Quoted 5 lines
by
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
cciss.
starting git<11 boot tests in a minute now.Ok, thanks for confirming it's cciss. The bug is likely an
interaction
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
between cciss and cfq I think, so it would be very useful if you can
pin
Quoted 33 lines
Quoted 18 lines
Quoted 5 lines
Quoted 1 line
point which of the cfq patches make it stall.
is there anything special about cciss or did you just deduce that it must be cciss in that particular box and are suspecting interaction problems with that driver and your CFQ changes?
Nothing really special about cciss, but a few months ago I had a similar discussion about cciss and a strange hang.
If possible, please also try a known bad kernel and apply the below patch and see if it still reproduces:
drivers/block/cciss.c +0
-0@@ -2362,7 +2362,11 @@ static inline void complete_command(ctlr cmd->rq->completion_data = cmd;drivers/block/cciss.c static inline void complete_command(ctlr · line 2372
Quoted 10 lines
drivers/block/cciss.c +4
-0cmd->rq->errors = status;blk_add_trace_rq (cmd->rq->q, cmd->rq,BLK_TA_COMPLETE ); +#if 1 +cciss_softirq_done (cmd->rq); +#else blk_complete_request (cmd->rq); +#endif }/* manually nailed it down to 2.6.17-git7 being the first broken revision. going to try whether Linus' git tree knows the -git revisions and do a bisect
otherwise interdiff and looking for CFQ or cciss changes as best I can.
Quoted 5 lines
Quoted 2 lines
Hmm, there are no cfq/cciss changes between git6 and git7. Some SCSI changes, though. Are you using SCSI for anything?
I thought cciss (SmartArray) was SCSI, isn't it? I guess you mean "no James Bottomley changes in the SCSI layer".
Nope, cciss doesn't interact with the SCSI layer except for tapes.
Quoted 6 lines
Quoted 1 line
We really need that sysrq-t dump.
I'm not able to get the virtual remote serial console working, so I will try to go down to the datacenter and do "1) SysRq-t 2) SysRq-S 3) reboot with livecd and get the content of the synced kern.log which should contain the SysRq-t output (hopefully).
Ok, hope it works out :)
You can also use netconsole, that might be a lot easier for you. That just requires networking and et netcat at the other end.
Signature · 1 line
Jens Axboe
Quoted 127 lines · Jens Axboe
Quoted 118 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 117 lines · Jens Axboe
Quoted 9 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 8 lines · Jens Axboe
Quoted 7 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 6 lines · Jens Axboe
Quoted 5 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 4 lines · Jens Axboe
Quoted 3 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 2 lines · Jens Axboe
Quoted 1 line · On Mon, Jul 24 2006, gmu 2k6 wrote:
the problem I have with hangs is related to changes in CFQ
and
Quoted 2 lines
Quoted 2 lines
that
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
CFQ is now the default. 2.6.17-git12 had the problem but
booting
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
it with elevator=deadline fixes the hang.
symptoms encountered during git-bisecting between v2.6.17 and
v2.6.18-rc1:
Quoted 3 lines
A hang while starting network services
B hang while trying to login
1 on remote console [not SSH] it hang after typing<uid><CR>
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
1 via OpenSSH it hang after typing <pwd><CR> when doing
slogin
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
root@<IP>
Quoted 2 lines
A is the problem I got in the first place and this seems to
be
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
case since 2.6.17-git11 definitely although git-bisect
pointed
Quoted 11 lines
Quoted 11 lines
me
Quoted 10 lines
Quoted 10 lines
at
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
the following
changeset which is included since 2.6.17-git12:
caaa5f9f0a75d1dc5e812e69afdbb8720e077fd3
by Jens Axboe
titled "[PATCH] cfq-iosched: many performance fixes"strange enough it also hangs with 2.6.17-git11 which did not
include
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
that
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
one changeset yet.
So perhaps your bisect isn't 100% trust worthy? Can you do a
manual
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
-gitX bisect to see which 2.6.17-gitX introduced the problem?
Also please put a serial console or similar on the machine, so
you
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
can
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
log + store the sysrq+t output.
well I didn't say that caa....fd3 is the exact change which
broke
Quoted 5 lines
Quoted 5 lines
it,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
just that it's related to 1) CFQ changes and 2) CFQ being the
default
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
now.
I have a Remote Serial Console via HP's integrated Lights-OutJava
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Applet but am not sure how to enable serial console via kernel
boot
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
Quoted 3 lines
params (will try to find out).
I will first try to find the 2.6.17-git* revision working before bisecting it against -git11 or git12.Thanks, would be much appreciated to try and narrow it down to a specific fix.
Are you seeing the hang on cciss?
I'm not sure it is in the cciss driver, but the SmartArray is
driven
Quoted 5 lines
Quoted 5 lines
by
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
cciss.
starting git<11 boot tests in a minute now.Ok, thanks for confirming it's cciss. The bug is likely an
interaction
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
between cciss and cfq I think, so it would be very useful if you can
pin
Quoted 33 lines
Quoted 18 lines
Quoted 5 lines
Quoted 1 line
point which of the cfq patches make it stall.
is there anything special about cciss or did you just deduce that it must be cciss in that particular box and are suspecting interaction problems with that driver and your CFQ changes?
Nothing really special about cciss, but a few months ago I had a similar discussion about cciss and a strange hang.
If possible, please also try a known bad kernel and apply the below patch and see if it still reproduces:
drivers/block/cciss.c +0
-0@@ -2362,7 +2362,11 @@ static inline void complete_command(ctlr cmd->rq->completion_data = cmd;drivers/block/cciss.c static inline void complete_command(ctlr · line 2372
Quoted 10 lines
drivers/block/cciss.c +4
-0cmd->rq->errors = status;blk_add_trace_rq (cmd->rq->q, cmd->rq,BLK_TA_COMPLETE ); +#if 1 +cciss_softirq_done (cmd->rq); +#else blk_complete_request (cmd->rq); +#endif }/* manually nailed it down to 2.6.17-git7 being the first broken revision. going to try whether Linus' git tree knows the -git revisions and do a bisect
otherwise interdiff and looking for CFQ or cciss changes as best I can.Quoted 5 lines
Quoted 2 lines
Hmm, there are no cfq/cciss changes between git6 and git7. Some SCSI changes, though. Are you using SCSI for anything?
I thought cciss (SmartArray) was SCSI, isn't it? I guess you mean "no James Bottomley changes in the SCSI layer".
Nope, cciss doesn't interact with the SCSI layer except for tapes.
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
We really need that sysrq-t dump.
I'm not able to get the virtual remote serial console working, so I will try to go down to the datacenter and do "1) SysRq-t 2) SysRq-S 3) reboot with livecd and get the content of the synced kern.log which should contain the SysRq-t output (hopefully).
Ok, hope it works out :)
I'm going downstairs now.
Quoted 2 lines
You can also use netconsole, that might be a lot easier for you. That just requires networking and et netcat at the other end.
any howto?
Quoted 4 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 2 lines
You can also use netconsole, that might be a lot easier for you. That just requires networking and et netcat at the other end.
any howto?
Documentation/networking/netconsole.txt
using the modular approach is the easiest.
Signature · 1 line
Jens Axboe
Quoted 4 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
ok, let's nail it to 2.6.17-git5 instead as it survived git status compared to -git6
which seems to have correctly booted by accident the lastime. timing issues I guess.
I will try and reproduce it here now. It seems to be in between commit 271f18f102c789f59644bb6c53a69da1df72b2f4 and commit dd67d051529387f6e44d22d1d5540ef281965fdd where the first one could also be bad.
I'm assuming that acf421755593f7d7bd9352d57eda796c6eb4fa43 should be good, so you can try and verify that dd67d051529387f6e44d22d1d5540ef281965fdd is bad and bisect between the two. It's only about 6 commits, so should be quick enough to do.
Signature · 1 line
Jens Axboe
Quoted 11 lines · Jens Axboe
Quoted 4 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
ok, let's nail it to 2.6.17-git5 instead as it survived git status compared to -git6
which seems to have correctly booted by accident the lastime. timing issues I guess.I will try and reproduce it here now. It seems to be in between commit
271f18f102c789f59644bb6c53a69da1df72b2f4and commitdd67d051529387f6e44d22d1d5540ef281965fddwhere the first one could also be bad.
Quoted 4 lines
I'm assuming that
acf421755593f7d7bd9352d57eda796c6eb4fa43should be good, so you can try and verify thatdd67d051529387f6e44d22d1d5540ef281965fddis bad and bisect between the two. It's only about 6 commits, so should be quick enough to do.
1) no luck with remote serial console
2) netconsole does not work although connecting to the listener with netcat and sending strings works
I'm gonna try via physical rs232 9pins and see how that works. afterwards I will try to bisect the revisions you mentioned.
btw, the issue seems to come and go as I managed to boot log into a .17-git6 kernel or is timing-dependent.
Quoted 25 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 12 lines · Jens Axboe
Quoted 3 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
ok, let's nail it to 2.6.17-git5 instead as it survived git status compared to -git6
which seems to have correctly booted by accident the lastime. timingissues
Quoted 1 line
I guess.
I will try and reproduce it here now. It seems to be in between commit
271f18f102c789f59644bb6c53a69da1df72b2f4and commitdd67d051529387f6e44d22d1d5540ef281965fddwhere the first one could also be bad.Quoted 4 lines
I'm assuming that
acf421755593f7d7bd9352d57eda796c6eb4fa43should be good, so you can try and verify thatdd67d051529387f6e44d22d1d5540ef281965fddis bad and bisect between the two. It's only about 6 commits, so should be quick enough to do.1) no luck with remote serial console
2) netconsole does not work although connecting to the listener with netcat and
sending strings works
I'm gonna try via physical rs232 9pins and see how that works. afterwards I will try to bisect the revisions you mentioned.
Quoted 2 lines
btw, the issue seems to come and go as I managed to boot log into a .17-git6 kernel or is timing-dependent.
I can reproduce it, you don't have to spend more time on bisecting or testing. This should fix it:
drivers/block/cciss.c +4 -0
@@ -1238,6 +1238,7 @@ static void cciss_softirq_done(struct reCommandList_struct *cmd = rq->completion_data;ctlr_info_t *h = hba[cmd->ctlr];unsigned long flags; +request_queue_t *q;u64bit temp64;int i, ddir;
@@ -1260,10 +1261,13 @@ #ifdef CCISS_DEBUGprintk ("Done with %p\n" , rq);#endif /* CCISS_DEBUG */ + q = rq->q; +add_disk_randomness (rq->rq_disk);spin_lock_irqsave (&h->lock, flags);end_that_request_last (rq, rq->errors);cmd_free (h, cmd,1 ); +blk_start_queue (q);spin_unlock_irqrestore (&h->lock, flags); }
A better fix would rework the start_queue logic entirely in the driver, but the above should get you running for now. I'll take a further look.
Signature · 1 line
Jens Axboe
Quoted 59 lines · Jens Axboe
Quoted 25 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 12 lines · Jens Axboe
Quoted 3 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
ok, let's nail it to 2.6.17-git5 instead as it survived git status compared to -git6
which seems to have correctly booted by accident the lastime. timingissues
Quoted 1 line
I guess.
I will try and reproduce it here now. It seems to be in between commit
271f18f102c789f59644bb6c53a69da1df72b2f4and commitdd67d051529387f6e44d22d1d5540ef281965fddwhere the first one could also be bad.Quoted 4 lines
I'm assuming that
acf421755593f7d7bd9352d57eda796c6eb4fa43should be good, so you can try and verify thatdd67d051529387f6e44d22d1d5540ef281965fddis bad and bisect between the two. It's only about 6 commits, so should be quick enough to do.1) no luck with remote serial console
2) netconsole does not work although connecting to the listener with netcat and
sending strings works
I'm gonna try via physical rs232 9pins and see how that works. afterwards I will try to bisect the revisions you mentioned.Quoted 2 lines
btw, the issue seems to come and go as I managed to boot log into a .17-git6 kernel or is timing-dependent.
I can reproduce it, you don't have to spend more time on bisecting or testing. This should fix it:
drivers/block/cciss.c +4
-0@@ -1238,6 +1238,7 @@ static void cciss_softirq_done(struct reCommandList_struct *cmd = rq->completion_data;ctlr_info_t *h = hba[cmd->ctlr];unsigned long flags; +request_queue_t *q;u64bit temp64;int i, ddir;@@ -1260,10 +1261,13 @@ #ifdef CCISS_DEBUGprintk ("Done with %p\n" , rq);#endif /* CCISS_DEBUG */ + q = rq->q; +add_disk_randomness (rq->rq_disk);spin_lock_irqsave (&h->lock, flags);end_that_request_last (rq, rq->errors);cmd_free (h, cmd,1 ); +blk_start_queue (q);spin_unlock_irqrestore (&h->lock, flags); }
Quoted 2 lines
A better fix would rework the
start_queuelogic entirely in the driver, but the above should get you running for now. I'll take a further look.
Something like this matches the current logic better. It's not very good from a cpu efficiency point of view, but it's better than what is there now since at least it's not in hard irq context.
Not tested yet, will do so right now.
drivers/block/cciss.c +45 -41
@@ -1233,6 +1233,50 @@ static inline void complete_buffers(stru } } +static void cciss_check_queues (ctlr_info_t *h) +{ +int start_queue = h->next_to_run; +int i; + +/* check to see if we have maxed out the number of commands that can +* be placed on the queue. If so then exit. We do this check here +* in case the interrupt we serviced was from an ioctl and did not +* free any new commands. +*/ +if ((find_first_zero_bit (h->cmd_pool_bits,NR_CMDS )) ==NR_CMDS ) +return ; + +/* We have room on the queue for more commands. Now we need to queue +* them up. We will also keep track of the next queue to run so +* that every queue gets a chance to be started first. +*/ +for (i =0 ; i < h->highest_lun +1 ; i++) { +int curr_queue = (start_queue + i) % (h->highest_lun +1 ); +/* make sure the disk has been added and the drive is real +* because this can be called from the middle of init_one. +*/ +if (!(h->drv[curr_queue].queue) || !(h->drv[curr_queue].heads)) +continue ; +blk_start_queue (h->gendisk[curr_queue]->queue); + +/* check to see if we have maxed out the number of commands +* that can be placed on the queue. +*/ +if ((find_first_zero_bit (h->cmd_pool_bits,NR_CMDS )) ==NR_CMDS ) { +if (curr_queue == start_queue) { + h->next_to_run = + (start_queue +1 ) % (h->highest_lun +1 ); +break ; + }else { + h->next_to_run = curr_queue; +break ; + } + }else { + curr_queue = (curr_queue +1 ) % (h->highest_lun +1 ); + } + } +} +static void cciss_softirq_done (struct request *rq) {CommandList_struct *cmd = rq->completion_data;
@@ -1264,6 +1308,7 @@ #endif /* CCISS_DEBUG */spin_lock_irqsave (&h->lock, flags);end_that_request_last (rq, rq->errors);cmd_free (h, cmd,1 ); +cciss_check_queues (h);spin_unlock_irqrestore (&h->lock, flags); }
@@ -2528,8 +2573,6 @@ static irqreturn_t do_cciss_intr(int irqCommandList_struct *c;unsigned long flags;__u32 a, a1, a2; -int j; -int start_queue = h->next_to_run;if (interrupt_not_for_us (h))return IRQ_NONE ;
@@ -2588,45 +2631,6 @@ # endif } } -/* check to see if we have maxed out the number of commands that can -* be placed on the queue. If so then exit. We do this check here -* in case the interrupt we serviced was from an ioctl and did not -* free any new commands. -*/ -if ((find_first_zero_bit (h->cmd_pool_bits,NR_CMDS )) ==NR_CMDS ) - goto cleanup; - -/* We have room on the queue for more commands. Now we need to queue -* them up. We will also keep track of the next queue to run so -* that every queue gets a chance to be started first. -*/ -for (j =0 ; j < h->highest_lun +1 ; j++) { -int curr_queue = (start_queue + j) % (h->highest_lun +1 ); -/* make sure the disk has been added and the drive is real -* because this can be called from the middle of init_one. -*/ -if (!(h->drv[curr_queue].queue) || !(h->drv[curr_queue].heads)) -continue ; -blk_start_queue (h->gendisk[curr_queue]->queue); - -/* check to see if we have maxed out the number of commands -* that can be placed on the queue. -*/ -if ((find_first_zero_bit (h->cmd_pool_bits,NR_CMDS )) ==NR_CMDS ) { -if (curr_queue == start_queue) { - h->next_to_run = - (start_queue +1 ) % (h->highest_lun +1 ); - goto cleanup; - }else { - h->next_to_run = curr_queue; - goto cleanup; - } - }else { - curr_queue = (curr_queue +1 ) % (h->highest_lun +1 ); - } - } - - cleanup:spin_unlock_irqrestore (CCISS_LOCK (h->ctlr), flags);return IRQ_HANDLED ; }
Signature · 1 line
Jens Axboe
thanks, so I can stop searching for a terminal app (HyperTerminal was not installed
on this Windows box) and escape the 20°C which feel like winter when it is 35°C outside.
I will test the 2nd patch and let you know.
Quoted 66 lines · Jens Axboe
Quoted 25 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 12 lines · Jens Axboe
Quoted 3 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
ok, let's nail it to 2.6.17-git5 instead as it survived git status compared to -git6
which seems to have correctly booted by accident the lastime. timingissues
Quoted 1 line
I guess.
I will try and reproduce it here now. It seems to be in between commit
271f18f102c789f59644bb6c53a69da1df72b2f4and commitdd67d051529387f6e44d22d1d5540ef281965fddwhere the first one could also be bad.Quoted 4 lines
I'm assuming that
acf421755593f7d7bd9352d57eda796c6eb4fa43should be good, so you can try and verify thatdd67d051529387f6e44d22d1d5540ef281965fddis bad and bisect between the two. It's only about 6 commits, so should be quick enough to do.1) no luck with remote serial console
2) netconsole does not work although connecting to the listener with netcat and
sending strings works
I'm gonna try via physical rs232 9pins and see how that works. afterwards I will try to bisect the revisions you mentioned.Quoted 2 lines
btw, the issue seems to come and go as I managed to boot log into a .17-git6 kernel or is timing-dependent.
I can reproduce it, you don't have to spend more time on bisecting or testing. This should fix it:
drivers/block/cciss.c +4
-0@@ -1238,6 +1238,7 @@ static void cciss_softirq_done(struct reCommandList_struct *cmd = rq->completion_data;ctlr_info_t *h = hba[cmd->ctlr];unsigned long flags; +request_queue_t *q;u64bit temp64;int i, ddir;@@ -1260,10 +1261,13 @@ #ifdef CCISS_DEBUGprintk ("Done with %p\n" , rq);#endif /* CCISS_DEBUG */ + q = rq->q; +add_disk_randomness (rq->rq_disk);spin_lock_irqsave (&h->lock, flags);end_that_request_last (rq, rq->errors);cmd_free (h, cmd,1 ); +blk_start_queue (q);spin_unlock_irqrestore (&h->lock, flags); }A better fix would rework the
start_queuelogic entirely in the driver, but the above should get you running for now. I'll take a further look.--
Jens Axboe
Quoted 177 lines · Jens Axboe
Quoted 59 lines · Jens Axboe
Quoted 25 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 12 lines · Jens Axboe
Quoted 3 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
ok, let's nail it to 2.6.17-git5 instead as it survived git status compared to -git6
which seems to have correctly booted by accident the lastime. timingissues
Quoted 1 line
I guess.
I will try and reproduce it here now. It seems to be in between commit
271f18f102c789f59644bb6c53a69da1df72b2f4and commitdd67d051529387f6e44d22d1d5540ef281965fddwhere the first one could also be bad.Quoted 4 lines
I'm assuming that
acf421755593f7d7bd9352d57eda796c6eb4fa43should be good, so you can try and verify thatdd67d051529387f6e44d22d1d5540ef281965fddis bad and bisect between the two. It's only about 6 commits, so should be quick enough to do.1) no luck with remote serial console
2) netconsole does not work although connecting to the listener with netcat and
sending strings works
I'm gonna try via physical rs232 9pins and see how that works. afterwards I will try to bisect the revisions you mentioned.Quoted 2 lines
btw, the issue seems to come and go as I managed to boot log into a .17-git6 kernel or is timing-dependent.
I can reproduce it, you don't have to spend more time on bisecting or testing. This should fix it:
drivers/block/cciss.c +4
-0@@ -1238,6 +1238,7 @@ static void cciss_softirq_done(struct reCommandList_struct *cmd = rq->completion_data;ctlr_info_t *h = hba[cmd->ctlr];unsigned long flags; +request_queue_t *q;u64bit temp64;int i, ddir;@@ -1260,10 +1261,13 @@ #ifdef CCISS_DEBUGprintk ("Done with %p\n" , rq);#endif /* CCISS_DEBUG */ + q = rq->q; +add_disk_randomness (rq->rq_disk);spin_lock_irqsave (&h->lock, flags);end_that_request_last (rq, rq->errors);cmd_free (h, cmd,1 ); +blk_start_queue (q);spin_unlock_irqrestore (&h->lock, flags); }Quoted 2 lines
A better fix would rework the
start_queuelogic entirely in the driver, but the above should get you running for now. I'll take a further look.Something like this matches the current logic better. It's not very good from a cpu efficiency point of view, but it's better than what is there now since at least it's not in hard irq context.
Not tested yet, will do so right now.
drivers/block/cciss.c +45
-34@@ -1233,6 +1233,50 @@ static inline void complete_buffers(stru } } +static void cciss_check_queues (ctlr_info_t *h) +{ +int start_queue = h->next_to_run; +int i; + +/* check to see if we have maxed out the number of commands that can +* be placed on the queue. If so then exit. We do this check here +* in case the interrupt we serviced was from an ioctl and did not +* free any new commands. +*/ +if ((find_first_zero_bit (h->cmd_pool_bits,NR_CMDS )) ==NR_CMDS ) +return ; + +/* We have room on the queue for more commands. Now we need to queue +* them up. We will also keep track of the next queue to run so +* that every queue gets a chance to be started first. +*/ +for (i =0 ; i < h->highest_lun +1 ; i++) { +int curr_queue = (start_queue + i) % (h->highest_lun +1 ); +/* make sure the disk has been added and the drive is real +* because this can be called from the middle of init_one. +*/ +if (!(h->drv[curr_queue].queue) || !(h->drv[curr_queue].heads)) +continue ; +blk_start_queue (h->gendisk[curr_queue]->queue); + +/* check to see if we have maxed out the number of commands +* that can be placed on the queue. +*/ +if ((find_first_zero_bit (h->cmd_pool_bits,NR_CMDS )) ==NR_CMDS ) { +if (curr_queue == start_queue) { + h->next_to_run = + (start_queue +1 ) % (h->highest_lun +1 ); +break ; + }else { + h->next_to_run = curr_queue; +break ; + } + }else { + curr_queue = (curr_queue +1 ) % (h->highest_lun +1 ); + } + } +} +static void cciss_softirq_done (struct request *rq) {CommandList_struct *cmd = rq->completion_data;@@ -1264,6 +1308,7 @@ #endif /* CCISS_DEBUG */spin_lock_irqsave (&h->lock, flags);end_that_request_last (rq, rq->errors);cmd_free (h, cmd,1 ); +cciss_check_queues (h);spin_unlock_irqrestore (&h->lock, flags); }@@ -2528,8 +2573,6 @@ static irqreturn_t do_cciss_intr(int irqCommandList_struct *c;unsigned long flags;__u32 a, a1, a2; -int j; -int start_queue = h->next_to_run;if (interrupt_not_for_us (h))return IRQ_NONE ;@@ -2588,45 +2631,6 @@ # endif } } -/* check to see if we have maxed out the number of commands that can -* be placed on the queue. If so then exit. We do this check here -* in case the interrupt we serviced was from an ioctl and did not -* free any new commands. -*/ -if ((find_first_zero_bit (h->cmd_pool_bits,NR_CMDS )) ==NR_CMDS ) - goto cleanup; - -/* We have room on the queue for more commands. Now we need to queue -* them up. We will also keep track of the next queue to run so -* that every queue gets a chance to be started first. -*/ -for (j =0 ; j < h->highest_lun +1 ; j++) { -int curr_queue = (start_queue + j) % (h->highest_lun +1 ); -/* make sure the disk has been added and the drive is real -* because this can be called from the middle of init_one. -*/ -if (!(h->drv[curr_queue].queue) || !(h->drv[curr_queue].heads)) -continue ; -blk_start_queue (h->gendisk[curr_queue]->queue); - -/* check to see if we have maxed out the number of commands -* that can be placed on the queue. -*/ -if ((find_first_zero_bit (h->cmd_pool_bits,NR_CMDS )) ==NR_CMDS ) { -if (curr_queue == start_queue) { - h->next_to_run = - (start_queue +1 ) % (h->highest_lun +1 ); - goto cleanup; - }else { - h->next_to_run = curr_queue; - goto cleanup;
drivers/block/cciss.c # endif · line 2636
Quoted 10 lines
drivers/block/cciss.c +0
-7- } - }else { - curr_queue = (curr_queue +1 ) % (h->highest_lun +1 ); - } - } - - cleanup:spin_unlock_irqrestore (CCISS_LOCK (h->ctlr), flags);return IRQ_HANDLED ; }
this makes the cciss init hang.
Quoted 182 lines · On 7/25/06, gmu 2k6 <[email protected]> wrote:
Quoted 181 lines · Jens Axboe
Quoted 59 lines · Jens Axboe
Quoted 25 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 12 lines · Jens Axboe
Quoted 3 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
ok, let's nail it to 2.6.17-git5 instead as it survived git status compared to -git6
which seems to have correctly booted by accident the lastime. timingissues
Quoted 1 line
I guess.
I will try and reproduce it here now. It seems to be in between commit
271f18f102c789f59644bb6c53a69da1df72b2f4and commitdd67d051529387f6e44d22d1d5540ef281965fddwhere the first one could also be bad.Quoted 4 lines
I'm assuming that
acf421755593f7d7bd9352d57eda796c6eb4fa43should be good, so you can try and verify thatdd67d051529387f6e44d22d1d5540ef281965fddis bad and bisect between the two. It's only about 6 commits, so should be quick enough to do.1) no luck with remote serial console
2) netconsole does not work although connecting to the listener with netcat and
sending strings works
I'm gonna try via physical rs232 9pins and see how that works. afterwards I will try to bisect the revisions you mentioned.Quoted 2 lines
btw, the issue seems to come and go as I managed to boot log into a .17-git6 kernel or is timing-dependent.
I can reproduce it, you don't have to spend more time on bisecting or testing. This should fix it:
drivers/block/cciss.c +4
-0@@ -1238,6 +1238,7 @@ static void cciss_softirq_done(struct reCommandList_struct *cmd = rq->completion_data;ctlr_info_t *h = hba[cmd->ctlr];unsigned long flags; +request_queue_t *q;u64bit temp64;int i, ddir;@@ -1260,10 +1261,13 @@ #ifdef CCISS_DEBUGprintk ("Done with %p\n" , rq);#endif /* CCISS_DEBUG */ + q = rq->q; +add_disk_randomness (rq->rq_disk);spin_lock_irqsave (&h->lock, flags);end_that_request_last (rq, rq->errors);cmd_free (h, cmd,1 ); +blk_start_queue (q);spin_unlock_irqrestore (&h->lock, flags); }Quoted 2 lines
A better fix would rework the
start_queuelogic entirely in the driver, but the above should get you running for now. I'll take a further look.Something like this matches the current logic better. It's not very good from a cpu efficiency point of view, but it's better than what is there now since at least it's not in hard irq context.
Not tested yet, will do so right now.
drivers/block/cciss.c +45
-38@@ -1233,6 +1233,50 @@ static inline void complete_buffers(stru } } +static void cciss_check_queues (ctlr_info_t *h) +{ +int start_queue = h->next_to_run; +int i; + +/* check to see if we have maxed out the number of commands that can +* be placed on the queue. If so then exit. We do this check here +* in case the interrupt we serviced was from an ioctl and did not +* free any new commands. +*/ +if ((find_first_zero_bit (h->cmd_pool_bits,NR_CMDS )) ==NR_CMDS ) +return ; + +/* We have room on the queue for more commands. Now we need to queue +* them up. We will also keep track of the next queue to run so +* that every queue gets a chance to be started first. +*/ +for (i =0 ; i < h->highest_lun +1 ; i++) { +int curr_queue = (start_queue + i) % (h->highest_lun +1 ); +/* make sure the disk has been added and the drive is real +* because this can be called from the middle of init_one. +*/ +if (!(h->drv[curr_queue].queue) || !(h->drv[curr_queue].heads)) +continue ; +blk_start_queue (h->gendisk[curr_queue]->queue); + +/* check to see if we have maxed out the number of commands +* that can be placed on the queue. +*/ +if ((find_first_zero_bit (h->cmd_pool_bits,NR_CMDS )) ==NR_CMDS ) { +if (curr_queue == start_queue) { + h->next_to_run = + (start_queue +1 ) % (h->highest_lun +1 ); +break ; + }else { + h->next_to_run = curr_queue; +break ; + } + }else { + curr_queue = (curr_queue +1 ) % (h->highest_lun +1 ); + } + } +} +static void cciss_softirq_done (struct request *rq) {CommandList_struct *cmd = rq->completion_data;@@ -1264,6 +1308,7 @@ #endif /* CCISS_DEBUG */spin_lock_irqsave (&h->lock, flags);end_that_request_last (rq, rq->errors);cmd_free (h, cmd,1 ); +cciss_check_queues (h);spin_unlock_irqrestore (&h->lock, flags); }@@ -2528,8 +2573,6 @@ static irqreturn_t do_cciss_intr(int irqCommandList_struct *c;unsigned long flags;__u32 a, a1, a2; -int j; -int start_queue = h->next_to_run;if (interrupt_not_for_us (h))return IRQ_NONE ;@@ -2588,45 +2631,6 @@ # endif } } -/* check to see if we have maxed out the number of commands that can -* be placed on the queue. If so then exit. We do this check here -* in case the interrupt we serviced was from an ioctl and did not -* free any new commands. -*/ -if ((find_first_zero_bit (h->cmd_pool_bits,NR_CMDS )) ==NR_CMDS ) - goto cleanup; - -/* We have room on the queue for more commands. Now we need to queue -* them up. We will also keep track of the next queue to run so -* that every queue gets a chance to be started first. -*/ -for (j =0 ; j < h->highest_lun +1 ; j++) { -int curr_queue = (start_queue + j) % (h->highest_lun +1 ); -/* make sure the disk has been added and the drive is real -* because this can be called from the middle of init_one. -*/ -if (!(h->drv[curr_queue].queue) || !(h->drv[curr_queue].heads)) -continue ; -blk_start_queue (h->gendisk[curr_queue]->queue); - -/* check to see if we have maxed out the number of commands -* that can be placed on the queue. -*/ -if ((find_first_zero_bit (h->cmd_pool_bits,NR_CMDS )) ==NR_CMDS ) { -if (curr_queue == start_queue) { - h->next_to_run = - (start_queue +1 ) % (h->highest_lun +1 ); - goto cleanup; - }else { - h->next_to_run = curr_queue; - goto cleanup; - } - }else { - curr_queue = (curr_queue +1 ) % (h->highest_lun +1 ); - }
Quoted 8 lines
drivers/block/cciss.c # endif · line 2636
Quoted 6 lines
drivers/block/cciss.c +0
-3- } - - cleanup:spin_unlock_irqrestore (CCISS_LOCK (h->ctlr), flags);return IRQ_HANDLED ; }this makes the cciss init hang.
I've patched Linus' trunk/HEAD with it, btw.
Quoted 59 lines · Jens Axboe
Quoted 25 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 12 lines · Jens Axboe
Quoted 3 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
ok, let's nail it to 2.6.17-git5 instead as it survived git status compared to -git6
which seems to have correctly booted by accident the lastime. timingissues
Quoted 1 line
I guess.
I will try and reproduce it here now. It seems to be in between commit
271f18f102c789f59644bb6c53a69da1df72b2f4and commitdd67d051529387f6e44d22d1d5540ef281965fddwhere the first one could also be bad.Quoted 4 lines
I'm assuming that
acf421755593f7d7bd9352d57eda796c6eb4fa43should be good, so you can try and verify thatdd67d051529387f6e44d22d1d5540ef281965fddis bad and bisect between the two. It's only about 6 commits, so should be quick enough to do.1) no luck with remote serial console
2) netconsole does not work although connecting to the listener with netcat and
sending strings works
I'm gonna try via physical rs232 9pins and see how that works. afterwards I will try to bisect the revisions you mentioned.Quoted 2 lines
btw, the issue seems to come and go as I managed to boot log into a .17-git6 kernel or is timing-dependent.
I can reproduce it, you don't have to spend more time on bisecting or testing. This should fix it:
drivers/block/cciss.c +4
-0@@ -1238,6 +1238,7 @@ static void cciss_softirq_done(struct reCommandList_struct *cmd = rq->completion_data;ctlr_info_t *h = hba[cmd->ctlr];unsigned long flags; +request_queue_t *q;u64bit temp64;int i, ddir;@@ -1260,10 +1261,13 @@ #ifdef CCISS_DEBUGprintk ("Done with %p\n" , rq);#endif /* CCISS_DEBUG */ + q = rq->q; +add_disk_randomness (rq->rq_disk);spin_lock_irqsave (&h->lock, flags);end_that_request_last (rq, rq->errors);cmd_free (h, cmd,1 ); +blk_start_queue (q);spin_unlock_irqrestore (&h->lock, flags); }
Quoted 2 lines
A better fix would rework the
start_queuelogic entirely in the driver, but the above should get you running for now. I'll take a further look.
this four-liner seems to fix it:
- I can boot
- log in
- git-status works
- svn up works
as my last mail said the 2nd patch with the new function introduced did hang cciss
on driver init before printing any drive info.
btw, I assume you have systems with SmartArray 6* at your disposal to test, right?
I mean SuSE should have some as a distro vendor.
thanks Jens,7b30f09245d0e6868819b946b2f6879e5d3d106b
http://kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=7b30f09245d0e6868819b946b2f6879e5d3d106b
has fixed the problem (maybe together with the other 3 changes in HEAD as the 2nd patch in this thread did not work in the first place or maybe it is a little bit different, no time to check right now).
Quoted 197 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 196 lines · Jens Axboe
Quoted 6 lines · Jens Axboe
Quoted 5 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 4 lines · Jens Axboe
Quoted 3 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
ok, let's nail it to 2.6.17-git5 instead as it survived git status compared to -git6
which seems to have correctly booted by accident the lastime. timingQuoted 6 lines
Quoted 6 lines
Quoted 6 lines
issues
Quoted 1 line
I guess.
I will try and reproduce it here now. It seems to be in between commit
271f18f102c789f59644bb6c53a69da1df72b2f4and commitdd67d051529387f6e44d22d1d5540ef281965fddwhere the first one couldalso
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
be bad.
I'm assuming that
acf421755593f7d7bd9352d57eda796c6eb4fa43should be good, so you can try and verify thatdd67d051529387f6e44d22d1d5540ef281965fddis bad and bisect between the two. It's only about 6 commits, so should be quick enough to do.1) no luck with remote serial console
2) netconsole does not work although connecting to the listener withnetcat
Quoted 6 lines
Quoted 6 lines
and
sending strings works
I'm gonna try via physical rs232 9pins and see how that works. afterwards I will try to bisect the revisions you mentioned.btw, the issue seems to come and go as I managed to boot log into a
.17-git6
Quoted 32 lines
Quoted 1 line
kernel or is timing-dependent.
I can reproduce it, you don't have to spend more time on bisecting or testing. This should fix it:
drivers/block/cciss.c +4
-0@@ -1238,6 +1238,7 @@ static void cciss_softirq_done(struct reCommandList_struct *cmd = rq->completion_data;ctlr_info_t *h = hba[cmd->ctlr];unsigned long flags; +request_queue_t *q;u64bit temp64;int i, ddir;@@ -1260,10 +1261,13 @@ #ifdef CCISS_DEBUGprintk ("Done with %p\n" , rq);#endif /* CCISS_DEBUG */ + q = rq->q; +add_disk_randomness (rq->rq_disk);spin_lock_irqsave (&h->lock, flags);end_that_request_last (rq, rq->errors);cmd_free (h, cmd,1 ); +blk_start_queue (q);spin_unlock_irqrestore (&h->lock, flags); }Quoted 2 lines
A better fix would rework the
start_queuelogic entirely in the driver, but the above should get you running for now. I'll take a further look.Something like this matches the current logic better. It's not very good from a cpu efficiency point of view, but it's better than what is there now since at least it's not in hard irq context.
Not tested yet, will do so right now.
drivers/block/cciss.c +6
-0@@ -1233,6 +1233,50 @@ static inline void complete_buffers(stru } } +static void cciss_check_queues (ctlr_info_t *h) +{ +int start_queue = h->next_to_run; +int i; + +/* check to see if we have maxed out the number of commands that can
drivers/block/cciss.c +8
-0+* be placed on the queue. If so then exit. We do this check here +* in case the interrupt we serviced was from an ioctl and did not +* free any new commands. +*/ +if ((find_first_zero_bit (h->cmd_pool_bits,NR_CMDS )) ==NR_CMDS ) +return ; + +/* We have room on the queue for more commands. Now we need to queue
drivers/block/cciss.c +9
-0+* them up. We will also keep track of the next queue to run so +* that every queue gets a chance to be started first. +*/ +for (i =0 ; i < h->highest_lun +1 ; i++) { +int curr_queue = (start_queue + i) % (h->highest_lun +1 ); +/* make sure the disk has been added and the drive is real +* because this can be called from the middle of init_one. +*/ + if (!(h->drv[curr_queue].queue) ||!(h->drv[curr_queue].heads))
drivers/block/cciss.c +7
-0+continue ; +blk_start_queue (h->gendisk[curr_queue]->queue); + +/* check to see if we have maxed out the number of commands +* that can be placed on the queue. +*/ + if ((find_first_zero_bit(h->cmd_pool_bits, NR_CMDS)) ==
NR_CMDS) {drivers/block/cciss.c +3
-0+if (curr_queue == start_queue) { + h->next_to_run = + (start_queue + 1) % (h->highest_lun +1);
drivers/block/cciss.c +7
-0+break ; + }else { + h->next_to_run = curr_queue; +break ; + } + }else { + curr_queue = (curr_queue + 1) % (h->highest_lun +1);
drivers/block/cciss.c +4
-0+ } + } +} +static void
cciss_softirq_done(struct request *rq){ CommandList_struct *cmd = rq->completion_data;drivers/block/cciss.c +1
-3@@ -1264,6 +1308,7 @@ #endif /* CCISS_DEBUG */spin_lock_irqsave (&h->lock, flags);end_that_request_last (rq, rq->errors);cmd_free (h, cmd,1 ); +cciss_check_queues (h);spin_unlock_irqrestore (&h->lock, flags); }@@ -2528,8 +2573,6 @@ static irqreturn_t do_cciss_intr(int irqCommandList_struct *c;unsigned long flags;__u32 a, a1, a2; -int j; -int start_queue = h->next_to_run;if (interrupt_not_for_us (h))return IRQ_NONE ;@@ -2588,45 +2631,6 @@ # endif } } -/* check to see if we have maxed out the number of commands that can
drivers/block/cciss.c +0
-8-* be placed on the queue. If so then exit. We do this check here -* in case the interrupt we serviced was from an ioctl and did not -* free any new commands. -*/ -if ((find_first_zero_bit (h->cmd_pool_bits,NR_CMDS )) ==NR_CMDS ) - goto cleanup; - -/* We have room on the queue for more commands. Now we need to queue
drivers/block/cciss.c +0
-9-* them up. We will also keep track of the next queue to run so -* that every queue gets a chance to be started first. -*/ -for (j =0 ; j < h->highest_lun +1 ; j++) { -int curr_queue = (start_queue + j) % (h->highest_lun +1 ); -/* make sure the disk has been added and the drive is real -* because this can be called from the middle of init_one. -*/ - if (!(h->drv[curr_queue].queue) ||!(h->drv[curr_queue].heads))
drivers/block/cciss.c +0
-7-continue ; -blk_start_queue (h->gendisk[curr_queue]->queue); - -/* check to see if we have maxed out the number of commands -* that can be placed on the queue. -*/ - if ((find_first_zero_bit(h->cmd_pool_bits, NR_CMDS)) ==
NR_CMDS) {drivers/block/cciss.c +0
-3-if (curr_queue == start_queue) { - h->next_to_run = - (start_queue + 1) % (h->highest_lun +1);
drivers/block/cciss.c +0
-7- goto cleanup; - }else { - h->next_to_run = curr_queue; - goto cleanup; - } - }else { - curr_queue = (curr_queue + 1) % (h->highest_lun +1);
- }
Quoted 8 lines
Quoted 6 lines
drivers/block/cciss.c +0
-3- } - - cleanup:spin_unlock_irqrestore (CCISS_LOCK (h->ctlr), flags);return IRQ_HANDLED ;}
this makes the cciss init hang.
hmm strange, it works for me. sysrq-t for the hang, please. just note the top few functions, should be easy enough to write down manually.
Signature · 1 line
Jens Axboe
Quoted 6 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
thanks Jens,
7b30f09245d0e6868819b946b2f6879e5d3d106b
http://kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=7b30f09245d0e6868819b946b2f6879e5d3d106b
has fixed the problem (maybe together with the other 3 changes in HEAD as the 2nd patch in this thread did not work in the first place or maybe it is a little bit different, no time to check right now).
It's an identical change, so the one sent you should work as well. Perhaps you botched that one test? These things happen, it's happened to me as well :-)
The change definitely fixed it for me.
Signature · 1 line
Jens Axboe
Quoted 8 lines · Jens Axboe
Quoted 6 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
thanks Jens,
7b30f09245d0e6868819b946b2f6879e5d3d106b
http://kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=7b30f09245d0e6868819b946b2f6879e5d3d106b
has fixed the problem (maybe together with the other 3 changes in HEAD as the 2nd patch in this thread did not work in the first place or maybe it is a little bit different, no time to check right now).
Quoted 5 lines
It's an identical change, so the one sent you should work as well. Perhaps you botched that one test? These things happen, it's happened to me as well :-)
The change definitely fixed it for me.
here too. right now I'm busy with trying to find out why /dev/hwrng is present but does not pass rngtest checks. I'll post about that soonish but it's a different issue of course.
anyway, thanks a lot Jens.
Quoted 71 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 61 lines · Jens Axboe
Quoted 13 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
Quoted 12 lines · Jens Axboe
Quoted 3 lines · On Tue, Jul 25 2006, gmu 2k6 wrote:
ok, let's nail it to 2.6.17-git5 instead as it survived git status compared to -git6
which seems to have correctly booted by accident the lastime. timingissues
Quoted 1 line
I guess.
I will try and reproduce it here now. It seems to be in between commit
271f18f102c789f59644bb6c53a69da1df72b2f4and commitdd67d051529387f6e44d22d1d5540ef281965fddwhere the first one could also be bad.Quoted 7 lines
Quoted 4 lines
I'm assuming that
acf421755593f7d7bd9352d57eda796c6eb4fa43should be good, so you can try and verify thatdd67d051529387f6e44d22d1d5540ef281965fddis bad and bisect between the two. It's only about 6 commits, so should be quick enough to do.1) no luck with remote serial console
2) netconsole does not work although connecting to the listener withnetcat
Quoted 6 lines
and
sending strings works
I'm gonna try via physical rs232 9pins and see how that works. afterwards I will try to bisect the revisions you mentioned.btw, the issue seems to come and go as I managed to boot log into a
.17-git6
Quoted 1 line
kernel or is timing-dependent.
I can reproduce it, you don't have to spend more time on bisecting or testing. This should fix it:
drivers/block/cciss.c +4
-0@@ -1238,6 +1238,7 @@ static void cciss_softirq_done(struct reCommandList_struct *cmd = rq->completion_data;ctlr_info_t *h = hba[cmd->ctlr];unsigned long flags; +request_queue_t *q;u64bit temp64;int i, ddir;@@ -1260,10 +1261,13 @@ #ifdef CCISS_DEBUGprintk ("Done with %p\n" , rq);#endif /* CCISS_DEBUG */ + q = rq->q; +add_disk_randomness (rq->rq_disk);spin_lock_irqsave (&h->lock, flags);end_that_request_last (rq, rq->errors);cmd_free (h, cmd,1 ); +blk_start_queue (q);spin_unlock_irqrestore (&h->lock, flags); }Quoted 2 lines
A better fix would rework the
start_queuelogic entirely in the driver, but the above should get you running for now. I'll take a further look.this four-liner seems to fix it:
- I can boot
- log in
- git-status works
- svn up works
Quoted 3 lines
as my last mail said the 2nd patch with the new function introduced did hang cciss
on driver init before printing any drive info.
The problem with the 4-liner is that it potentially starves some arrays (in theory). I'll retry the full fix.
Quoted 3 lines
btw, I assume you have systems with SmartArray 6* at your disposal to test, right?
I mean SuSE should have some as a distro vendor.
I do, I have several right here next to me.
Signature · 1 line
Jens Axboe