Skip to contentSkip to conversation

hung in raise_barrier() in raid1.c -- any ideas?

DISCUSSION Chris Friesen 2 participants 5 messages Sep 20, 2012 – Sep 21, 2012 MD RAID

lore
Share & tools
lore

Chris Friesen

author
#0 lore raw

Hi,

I've got a fairly beefy (32 cpus, 64GB ram, isci-based SAS disks, etc.) embedded system running 2.6.27.

We're seeing issues where disk operations suddenly seem to stall. In the most recent case we had the hung-task watchdog indicate that md1_resync was stuck for more than 120sec in raise_barrier().

There are a bunch of "normal" tasks also stuck in wait_barrier(), so based on that I assume we're stuck in the second call to wait_event_lock_irq().

Has anyone seen anything like this? Could commit 73d5c38 be related? What about 1d9d524?

Also, what's the meaning of RESYNC_DEPTH?

Thanks,
Chris

Signature · 6 lines
Chris Friesen
Software Designer

3500 Carling Avenue
Ottawa, Ontario K2H 8E9
www.genband.com
author
#1 lore raw
Quoted 10 lines · Chris Friesen

Hi,

I've got a fairly beefy (32 cpus, 64GB ram, isci-based SAS disks, etc.) embedded system running 2.6.27.

We're seeing issues where disk operations suddenly seem to stall. In the most recent case we had the hung-task watchdog indicate that md1_resync was stuck for more than 120sec in raise_barrier().

There are a bunch of "normal" tasks also stuck in wait_barrier(), so based on that I assume we're stuck in the second call to wait_event_lock_irq().

Has anyone seen anything like this? Could commit 73d5c38 be related? What about 1d9d524?

Could d6b42dc be related?

Chris

#2 lore raw
Quoted 11 lines · On Thu, 20 Sep 2012 11:55:02 -0600 Chris Friesen <[email protected]>…
Quoted 10 lines · Chris Friesen

Hi,

I've got a fairly beefy (32 cpus, 64GB ram, isci-based SAS disks, etc.) embedded system running 2.6.27.

We're seeing issues where disk operations suddenly seem to stall. In the most recent case we had the hung-task watchdog indicate that md1_resync was stuck for more than 120sec in raise_barrier().

Quoted 8 lines
Quoted 6 lines

There are a bunch of "normal" tasks also stuck in wait_barrier(), so based on that I assume we're stuck in the second call to wait_event_lock_irq().

Has anyone seen anything like this? Could commit 73d5c38 be related? What about 1d9d524?

Could d6b42dc be related?

That last one seems more likely. Does the scenario fit your config. i.e. is your RAID1 being used under LVM?

If it does, then I would say it is very likely this issue.

Quoted 1 line

Also, what's the meaning of RESYNC_DEPTH?

The maximum number of resync requests that can be concurrently active.

RESYNC_WINDOW should really be RESYNC_BLOCK_SIZE * RESYNC_DEPTH I wonder why it isn't.

NeilBrown

author
#3 lore raw
Quoted 21 lines · NeilBrown
Quoted 18 lines · On Thu, 20 Sep 2012 11:55:02 -0600 Chris Friesen<[email protected]>…
Quoted 16 lines · Chris Friesen

Hi,

I've got a fairly beefy (32 cpus, 64GB ram, isci-based SAS disks, etc.) embedded system running 2.6.27.

We're seeing issues where disk operations suddenly seem to stall. In the most recent case we had the hung-task watchdog indicate that md1_resync was stuck for more than 120sec in raise_barrier().

There are a bunch of "normal" tasks also stuck in wait_barrier(), so based on that I assume we're stuck in the second call to wait_event_lock_irq().

Has anyone seen anything like this? Could commit 73d5c38 be related? What about 1d9d524?

Quoted 6 lines
Quoted 1 line

Could d6b42dc be related?

That last one seems more likely. Does the scenario fit your config. i.e. is your RAID1 being used under LVM?

If it does, then I would say it is very likely this issue.

Yes, we're using it under LVM. I've added some instrumentation to tell if we're hitting that case. The current->bio_list handling is a bit different in 2.6.27 but I think I figured out the equivalent to the patch.

Interesting that it took this long to fix that issue.

Quoted 3 lines
Quoted 1 line

Also, what's the meaning of RESYNC_DEPTH?

The maximum number of resync requests that can be concurrently active.

And each request would resync a single block?

Chris

#4 lore raw
Quoted 34 lines · On Thu, 20 Sep 2012 17:05:23 -0600 Chris Friesen <[email protected]>…
Quoted 21 lines · NeilBrown
Quoted 18 lines · On Thu, 20 Sep 2012 11:55:02 -0600 Chris Friesen<[email protected]>…
Quoted 16 lines · Chris Friesen

Hi,

I've got a fairly beefy (32 cpus, 64GB ram, isci-based SAS disks, etc.) embedded system running 2.6.27.

We're seeing issues where disk operations suddenly seem to stall. In the most recent case we had the hung-task watchdog indicate that md1_resync was stuck for more than 120sec in raise_barrier().

There are a bunch of "normal" tasks also stuck in wait_barrier(), so based on that I assume we're stuck in the second call to wait_event_lock_irq().

Has anyone seen anything like this? Could commit 73d5c38 be related? What about 1d9d524?

Quoted 6 lines
Quoted 1 line

Could d6b42dc be related?

That last one seems more likely. Does the scenario fit your config. i.e. is your RAID1 being used under LVM?

If it does, then I would say it is very likely this issue.

Yes, we're using it under LVM. I've added some instrumentation to tell if we're hitting that case. The current->bio_list handling is a bit different in 2.6.27 but I think I figured out the equivalent to the patch.

Quoted 8 lines

Interesting that it took this long to fix that issue.

Quoted 3 lines
Quoted 1 line

Also, what's the meaning of RESYNC_DEPTH?

The maximum number of resync requests that can be concurrently active.

And each request would resync a single block?

Each request will resync up to RESYNC_BLOCK_SIZE bytes - i.e. up to 64K.

NeilBrown