Skip to contentSkip to conversation

2.6.14-mm1 RAID-1 in D< state

DISCUSSION Chris Boot 3 participants 9 messages Nov 9, 2005 – Nov 10, 2005 MD RAID

lore
Share & tools
lore

Chris Boot

author
#0 lore raw

Hi all,

I haven't noticed this until today...but my load average has been skyrocketing past 3.00 since Monday, which is when I upgraded to 2.6.14-mm1. I've got 3 Software RAID-1 arrays across 4 SATA disks, and all 3 processes are locked in an uninterruptible sleep.

What's interesting, though, is I haven't noticed a degradation of performance at all, and all the arrays work absolutely fine. They aren't rebuilding or doing anything strange that I can see.

Any ideas?

Cheers,
Chris

Signature · 3 lines
Chris Boot
[email protected]
http://www.bootc.net/
#1 lore raw
Quoted 7 lines · Chris Boot

Hi all,

I haven't noticed this until today...but my load average has been skyrocketing past 3.00 since Monday, which is when I upgraded to 2.6.14-mm1. I've got 3 Software RAID-1 arrays across 4 SATA disks, and all 3 processes are locked in an uninterruptible sleep.

Quoted 6 lines

What's interesting, though, is I haven't noticed a degradation of performance at all, and all the arrays work absolutely fine. They aren't rebuilding or doing anything strange that I can see.

Any ideas?

Try this:

http://marc.theaimsgroup.com/?l=linux-scsi&m=113145981728205&w=2

My raid 5 was oopsing till I applied this.

--
J.A. Magallon <jamagallon()able!es> \ Software is like sex:
werewolf!able!es \ It's better when it's free
Mandriva Linux release 2006.1 (Cooker) for i586
Linux 2.6.14-jam1 (gcc 4.0.2 (4.0.2-1mdk for Mandriva Linux release 2006.1))

#2 lore raw
In reply to Chris Boot
Quoted 7 lines · On Wednesday November 9, [email protected] wrote:

Hi all,

I haven't noticed this until today...but my load average has been skyrocketing past 3.00 since Monday, which is when I upgraded to 2.6.14-mm1. I've got 3 Software RAID-1 arrays across 4 SATA disks, and all 3 processes are locked in an uninterruptible sleep.

Quoted 5 lines

What's interesting, though, is I haven't noticed a degradation of performance at all, and all the arrays work absolutely fine. They aren't rebuilding or doing anything strange that I can see.

Any ideas?

Can you
echo t > /proc/sysrq-trigger
dmesg > /tmp/log
and post the log created, possibly removing everything before
SysRq : Show State

If you can't find the 'Show State', then maybe your log buffer isn't big enough. use 'dmesg -s ...' to make it bigger and try again.

NeilBrown

author
#3 lore raw
Quoted 12 lines · Neil Brown
Quoted 11 lines · On Wednesday November 9, [email protected] wrote:

Hi all,

I haven't noticed this until today...but my load average has been skyrocketing past 3.00 since Monday, which is when I upgraded to 2.6.14-mm1. I've got 3 Software RAID-1 arrays across 4 SATA disks, and all 3 processes are locked in an uninterruptible sleep.

What's interesting, though, is I haven't noticed a degradation of performance at all, and all the arrays work absolutely fine. They aren't rebuilding or doing anything strange that I can see.

Quoted 7 lines
Quoted 1 line

Any ideas?

Can you
echo t > /proc/sysrq-trigger
dmesg > /tmp/log
and post the log created, possibly removing everything before
SysRq : Show State

So that's what the sysrq-trigger is for... :-) Certainly easier that way when your system still works!

Quoted 2 lines

If you can't find the 'Show State', then maybe your log buffer isn't big enough. use 'dmesg -s ...' to make it bigger and try again

It was too small, but the serial console got it:

[4329954.200000] md2_raid1     D F7D776E0     0   809      6           810   799 (L-TLB)
[4329954.200000] f7db7f30 f7d2ba8c c02809e0 f7d776e0 c02c14f2 e9924580 c1b48b60 c1b8e200
[4329954.200000]        f7c5bd40 7fffffff f7db7f88 00000000 23c37e00 000f6206 f7d6fa50 f7d6fb78
[4329954.200000]        7fffffff 7fffffff f7db7f88 f7db6000 c0338098 c1b8e200 f7db7f94 f7db7f88
[4329954.200000] Call Trace:
[4329954.200000]  [<c02809e0>] generic_unplug_device+0x10/0x20
[4329954.200000]  [<c02c14f2>] unplug_slaves+0xd2/0xe0
[4329954.200000]  [<c0338098>] schedule_timeout+0x98/0xa0
[4329954.200000]  [<c01295a9>] finish_wait+0x39/0x50
[4329954.200000]  [<c02c9309>] md_thread+0xc9/0x100
[4329954.200000]  [<c01295c0>] autoremove_wake_function+0x0/0x50
[4329954.200000]  [<c01142d7>] __wake_up_common+0x37/0x60
[4329954.200000]  [<c01295c0>] autoremove_wake_function+0x0/0x50
[4329954.200000]  [<c02c9240>] md_thread+0x0/0x100
[4329954.200000]  [<c0129174>] kthread+0xa4/0xe0
[4329954.200000]  [<c01290d0>] kthread+0x0/0xe0
[4329954.200000]  [<c0100f35>] kernel_thread_helper+0x5/0x10
[4329954.200000] md0_raid1     D F7D774A0     0   810      6           812   809 (L-TLB)
[4329954.200000] f7db5f30 f7d2b79c c02809e0 f7d774a0 c02c14f2 c0383bc0 c1b48ae0 c1b8e400
[4329954.200000]        f7c5bb60 7fffffff f7db5f88 00000000 9bd42ec0 000f6211 f7d69090 f7d691b8
[4329954.200000]        7fffffff 7fffffff f7db5f88 f7db4000 c0338098 c1b8e400 00000002 f7db4000
[4329954.200000] Call Trace:
[4329954.200000]  [<c02809e0>] generic_unplug_device+0x10/0x20
[4329954.200000]  [<c02c14f2>] unplug_slaves+0xd2/0xe0
[4329954.200000]  [<c0338098>] schedule_timeout+0x98/0xa0
[4329954.200000]  [<c0129501>] prepare_to_wait+0x41/0x50
[4329954.200000]  [<c02c9309>] md_thread+0xc9/0x100
[4329954.200000]  [<c01295c0>] autoremove_wake_function+0x0/0x50
[4329954.200000]  [<c01142d7>] __wake_up_common+0x37/0x60
[4329954.200000]  [<c01295c0>] autoremove_wake_function+0x0/0x50
[4329954.200000]  [<c02c9240>] md_thread+0x0/0x100
[4329954.200000]  [<c0129174>] kthread+0xa4/0xe0
[4329954.200000]  [<c01290d0>] kthread+0x0/0xe0
[4329954.200000]  [<c0100f35>] kernel_thread_helper+0x5/0x10
[4329954.200000] md1_raid1     D F7D77860     0   812      6           813   810 (L-TLB)
[4329954.200000] f7dbbf30 f7d2bc04 c02809e0 f7d77860 c02c14f2 e9924580 c1b48a60 c1b8e000
[4329954.200000]        f7c5f920 7fffffff f7dbbf88 00000000 2358ae40 000f6206 f7d5b5c0 f7d5b6e8
[4329954.200000]        7fffffff 7fffffff f7dbbf88 f7dba000 c0338098 c1b8e000 f7dbbf88 f7dba000
[4329954.200000] Call Trace:
[4329954.200000]  [<c02809e0>] generic_unplug_device+0x10/0x20
[4329954.200000]  [<c02c14f2>] unplug_slaves+0xd2/0xe0
[4329954.200000]  [<c0338098>] schedule_timeout+0x98/0xa0
[4329954.200000]  [<c02c29ba>] raid1d+0x32a/0x350
[4329954.200000]  [<c02c9309>] md_thread+0xc9/0x100
[4329954.200000]  [<c01295c0>] autoremove_wake_function+0x0/0x50
[4329954.200000]  [<c01142d7>] __wake_up_common+0x37/0x60
[4329954.200000]  [<c01295c0>] autoremove_wake_function+0x0/0x50
[4329954.200000]  [<c02c9240>] md_thread+0x0/0x100
[4329954.200000]  [<c0129174>] kthread+0xa4/0xe0
[4329954.200000]  [<c01290d0>] kthread+0x0/0xe0
[4329954.200000]  [<c0100f35>] kernel_thread_helper+0x5/0x10

Let me know if you need dumps of any other processes.

Quoted 1 line

NeilBrown

Cheers,
Chris

Signature · 3 lines
Chris Boot
[email protected]
http://www.bootc.net/
#4 lore raw

Thanks for the trace. I see what is happening.
I changed
wait_event_timeout_interruptible
in md.c(md_thread) to
wait_event_timeout

as the thread no longer needs to be able to respond the signals. However that has the side-effect of putting the process in the 'D' state and adding to the 'uptime'.

I guess I'll put that back...

NeilBrown

### Diffstat output

 ./drivers/md/md.c |    9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/md/md.c renamed from drivers/md/md.c~current~ +5 -4
diff ./drivers/md/md.c~current~ ./drivers/md/md.c
@@ -3439,10 +3439,11 @@ static int md_thread(void * arg)
 	allow_signal(SIGKILL);
 	while (!kthread_should_stop()) {
 
-		wait_event_timeout(thread->wqueue,
-				   test_bit(THREAD_WAKEUP, &thread->flags)
-				   || kthread_should_stop(),
-				   thread->timeout);
+		wait_event_timeout_interruptible
+			(thread->wqueue,
+			 test_bit(THREAD_WAKEUP, &thread->flags)
+			 || kthread_should_stop(),
+			 thread->timeout);
 		try_to_freeze();
 
 		clear_bit(THREAD_WAKEUP, &thread->flags);
author
#5 lore raw
Quoted 31 lines · Neil Brown

Thanks for the trace. I see what is happening.
I changed
wait_event_timeout_interruptible
in md.c(md_thread) to
wait_event_timeout

as the thread no longer needs to be able to respond the signals. However that has the side-effect of putting the process in the 'D' state and adding to the 'uptime'.

I guess I'll put that back...

NeilBrown

### Diffstat output

 ./drivers/md/md.c |    9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff ./drivers/md/md.c~current~ ./drivers/md/md.c

drivers/md/md.c renamed from drivers/md/md.c~current~ +0 -2
@@ -3439,10 +3439,11 @@ static int md_thread(void * arg)
 	allow_signal(SIGKILL);
 	while (!kthread_should_stop()) {
 
-		wait_event_timeout(thread->wqueue,
-				   test_bit(THREAD_WAKEUP, &thread->flags)

drivers/md/md.c static int md_thread(void * arg) · line 3449

Quoted 10 lines
drivers/md/md.c renamed from drivers/md/md.c~current~ +5 -2
-				   || kthread_should_stop(),
-				   thread->timeout);
+		wait_event_timeout_interruptible
+			(thread->wqueue,
+			 test_bit(THREAD_WAKEUP, &thread->flags)
+			 || kthread_should_stop(),
+			 thread->timeout);
 		try_to_freeze();
 
 		clear_bit(THREAD_WAKEUP, &thread->flags);

Sounds about right but...

drivers/md/md.c: In function `md_thread':
drivers/md/md.c:3441: warning: implicit declaration of function `wait_event_timeout_interruptible' [...]
LD .tmp_vmlinux1
drivers/built-in.o(.text+0x9904f): In function `md_thread':
: undefined reference to wait_event_timeout_interruptible' drivers/built-in.o(.text+0x9908f): In function md_thread':
: undefined reference to `wait_event_timeout_interruptible'
make: *** [.tmp_vmlinux1] Error 1

HTH,
Chris

Signature · 3 lines
Chris Boot
[email protected]
http://www.bootc.net/
#6 lore raw
Quoted 6 lines · On Thursday November 10, [email protected] wrote:

Sounds about right but...

drivers/md/md.c: In function `md_thread':
drivers/md/md.c:3441: warning: implicit declaration of function `wait_event_timeout_interruptible'

should be
wait_event_interruptible_timeout

Sorry.
NeilBrown

author
#7 lore raw
Quoted 8 lines · Neil Brown
Quoted 6 lines · On Thursday November 10, [email protected] wrote:

Sounds about right but...

drivers/md/md.c: In function `md_thread':
drivers/md/md.c:3441: warning: implicit declaration of function `wait_event_timeout_interruptible'

Quoted 5 lines

should be
wait_event_interruptible_timeout

Sorry.
NeilBrown

No problem. Builds, boots, and fixes the problem.

Cheers,
Chris

Signature · 3 lines
Chris Boot
[email protected]
http://www.bootc.net/
#8 lore raw
In reply to Neil Brown
Quoted 31 lines · Neil Brown

Thanks for the trace. I see what is happening.
I changed
wait_event_timeout_interruptible
in md.c(md_thread) to
wait_event_timeout

as the thread no longer needs to be able to respond the signals. However that has the side-effect of putting the process in the 'D' state and adding to the 'uptime'.

I guess I'll put that back...

NeilBrown

### Diffstat output

 ./drivers/md/md.c |    9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff ./drivers/md/md.c~current~ ./drivers/md/md.c

drivers/md/md.c renamed from drivers/md/md.c~current~ +0 -2
@@ -3439,10 +3439,11 @@ static int md_thread(void * arg)
 	allow_signal(SIGKILL);
 	while (!kthread_should_stop()) {
 
-		wait_event_timeout(thread->wqueue,
-				   test_bit(THREAD_WAKEUP, &thread->flags)

drivers/md/md.c static int md_thread(void * arg) · line 3449

Quoted 10 lines
drivers/md/md.c renamed from drivers/md/md.c~current~ +5 -2
-				   || kthread_should_stop(),
-				   thread->timeout);
+		wait_event_timeout_interruptible
+			(thread->wqueue,
+			 test_bit(THREAD_WAKEUP, &thread->flags)
+			 || kthread_should_stop(),
+			 thread->timeout);
 		try_to_freeze();
 
 		clear_bit(THREAD_WAKEUP, &thread->flags);

s/wait_event_timeout_interruptible/wait_event_interruptible_timeout/

;)

--
J.A. Magallon <jamagallon()able!es> \ Software is like sex:
werewolf!able!es \ It's better when it's free
Mandriva Linux release 2006.1 (Cooker) for i586
Linux 2.6.14-jam1 (gcc 4.0.2 (4.0.2-1mdk for Mandriva Linux release 2006.1))