IBM xSeries stop responding during RAID1 reconstruction
Hi to all,
I have a new IBM xSeries 206m with two SATA drives, I installed a Debian Testing (Etch) and configured a software RAID as shown:
Personalities : [raid1]
md1 : active raid1 sdb5[1] sda5[0]
1951744 blocks [2/2] [UU]
md2 : active raid1 sdb6[1] sda6[0]
2931712 blocks [2/2] [UU]
md3 : active raid1 sdb7[1] sda7[0]
39061952 blocks [2/2] [UU]
md0 : active raid1 sdb1[1] sda1[0]
5855552 blocks [2/2] [UU]
I experience this problem: whenever a volume is reconstructing (syncing), the system stops responding. The machine is alive, because it responds to the ping, the console is responsive but I cannot pass the login prompt. It seems that every disk activity is delayed and blocking.
When the sync is complete, the machine start to respond again perfectly.
Any hints on how to start debugging?
In the log file nothing strange appears:
Jun 13 18:48:04 paros kernel: md: bind<sdb1>
Jun 13 18:48:04 paros kernel: RAID1 conf printout:
Jun 13 18:48:04 paros kernel: --- wd:1 rd:2
Jun 13 18:48:04 paros kernel: disk 0, wo:0, o:1, dev:sda1
Jun 13 18:48:04 paros kernel: disk 1, wo:1, o:1, dev:sdb1
Jun 13 18:48:04 paros kernel: md: syncing RAID array md0
Jun 13 18:48:04 paros kernel: md: minimum _guaranteed_
reconstruction speed: 1000 KB/sec/disc.
Jun 13 18:48:04 paros kernel: md: using maximum available idle IO
bandwidth (but not more than 200000 KB/sec) for
reconstruction.
Jun 13 18:48:04 paros kernel: md: using 128k window, over a total
of 5855552 blocks.
Jun 13 18:57:30 paros kernel: md: md0: sync done.
Jun 13 18:57:30 paros kernel: RAID1 conf printout:
Jun 13 18:57:30 paros kernel: --- wd:2 rd:2
Jun 13 18:57:30 paros kernel: disk 0, wo:0, o:1, dev:sda1
Jun 13 18:57:30 paros kernel: disk 1, wo:0, o:1, dev:sdb1Those are the IDE controllers as identified by lspci:
0000:00:1f.1 IDE interface: Intel Corporation 82801G (ICH7
Family) IDE Controller (rev 01)
0000:00:1f.2 IDE interface: Intel Corporation 82801GB/GR/GH (ICH7
Family) Serial ATA Storage Controllers cc=IDE (rev 01)
And those are the SATA drivers detected by the kernel:
ata1: SATA max UDMA/133 cmd 0x30C8 ctl 0x30BE bmdma 0x3090
irq 233
ata2: SATA max UDMA/133 cmd 0x30C0 ctl 0x30BA bmdma 0x3098
irq 233
ata1: dev 1 cfg 49:2f00 82:346b 83:7fe9 84:4773 85:3469 86:3c01
87:4763 88:207fata1: dev 1 ATA-7, max UDMA/133, 156312576 sectors: LBA48
ata1: dev 1 configured for UDMA/133
scsi0 : ata_piix
ata2: dev 1 cfg 49:2f00 82:346b 83:7fe9 84:4773 85:3469 86:3c01
87:4763 88:207f
ata2: dev 1 ATA-7, max UDMA/133, 156312576 sectors: LBA48
ata2: dev 1 configured for UDMA/133
scsi1 : ata_piix
Vendor: ATA Model: HDS728080PLA380 Rev: PF2O
Type: Direct-Access ANSI SCSI revision: 05
Vendor: ATA Model: HDS728080PLA380 Rev: PF2O
Type: Direct-Access ANSI SCSI revision: 05
SCSI device sda: 156312576 512-byte hdwr sectors (80032 MB) SCSI device sda: drive cache: write back
SCSI device sda: 156312576 512-byte hdwr sectors (80032 MB) SCSI device sda: drive cache: write back
sda: sda1 sda2 < sda5 sda6 sda7 >
sd 0:0:1:0: Attached scsi disk sda
SCSI device sdb: 156312576 512-byte hdwr sectors (80032 MB) SCSI device sdb: drive cache: write back
SCSI device sdb: 156312576 512-byte hdwr sectors (80032 MB) SCSI device sdb: drive cache: write back
sdb: sdb1 sdb2 < sdb5 sdb6 sdb7 >
sd 1:0:1:0: Attached scsi disk sdb
Signature · 4 lines
Niccolo Rigacci Firenze - Italy Iraq, missione di pace: 38355 morti - www.iraqbodycount.net
Quoted 14 lines · Niccolo Rigacci
Hi to all,
I have a new IBM xSeries 206m with two SATA drives, I installed a Debian Testing (Etch) and configured a software RAID as shown:
Personalities : [raid1]
md1 : active raid1 sdb5[1] sda5[0]1951744 blocks [2/2] [UU]md2 : active raid1 sdb6[1] sda6[0]
2931712 blocks [2/2] [UU]md3 : active raid1 sdb7[1] sda7[0]
39061952 blocks [2/2] [UU]
Quoted 10 lines
md0 : active raid1 sdb1[1] sda1[0]
5855552 blocks [2/2] [UU]I experience this problem: whenever a volume is reconstructing (syncing), the system stops responding. The machine is alive, because it responds to the ping, the console is responsive but I cannot pass the login prompt. It seems that every disk activity is delayed and blocking.
When the sync is complete, the machine start to respond again perfectly.
Any hints on how to start debugging?
I ran into a similar problem using kernel 2.6.16.14 on an ASUS motherboard: When I mirrored two SATA drives it seemed to block all other disk I/O until the sync was complete.
My symptoms were the same: all consoles were non-responsive and when I tried to login it just sat there until the sync was complete.
I was able to work around this by lowering /proc/sys/dev/raid/speed_limit_max to a value below my disk thruput value (~ 50 MB/s) as follows:
$ echo 45000 > /proc/sys/dev/raid/speed_limit_max
That kept my system usable but didn't address the underlying problem of the raid resync not being appropriately throttled. I ended up configuring my system differently so this became a moot point for me.
Hope this helps,
Bill
Quoted 10 lines · Bill Cizek
Quoted 3 lines · Niccolo Rigacci
When the sync is complete, the machine start to respond again perfectly.
I was able to work around this by lowering
/proc/sys/dev/raid/speed_limit_maxto a value below my disk thruput value (~ 50 MB/s) as follows:$ echo 45000 > /proc/sys/dev/raid/speed_limit_max
Thanks!
This hack seems to solve my problem too. So it seems that the RAID subsystem does not detect a proper speed to throttle the sync.
Can you please send me some details of your system?
- SATA chipset (or motherboard model)?
- Disks make/model?
- Do you have the config file of the kernel that you was running (look at /boot/config-<version> file)?
I wonder if kernel preemption can be blamed for that, or burst speed of disks can fool the throttle calculation.
Signature · 4 lines
Niccolo Rigacci Firenze - Italy Iraq, missione di pace: 38355 morti - www.iraqbodycount.net
Quoted 10 lines · On Thursday June 15, [email protected] wrote:
Quoted 9 lines · Bill Cizek
Quoted 3 lines · Niccolo Rigacci
When the sync is complete, the machine start to respond again perfectly.
I was able to work around this by lowering
/proc/sys/dev/raid/speed_limit_maxto a value below my disk thruput value (~ 50 MB/s) as follows:
Quoted 7 lines
Quoted 1 line
$ echo 45000 > /proc/sys/dev/raid/speed_limit_maxThanks!
This hack seems to solve my problem too. So it seems that the RAID subsystem does not detect a proper speed to throttle the sync.
The RAID subsystem doesn't try to detect a 'proper' speed. When there is nothing else happening, it just drives the disks as fast as they will go.
If this is causing a lockup, then there is something else wrong, just as any single process should not - by writing constantly to disks - be able to clog up the whole system.
Maybe if you could get the result of
alt-sysrq-P
or even
alt-sysrq-T
while the system seems to hang.
NeilBrown
Quoted 6 lines · you
If this is causing a lockup, then there is something else wrong, just as any single process should not - by writing constantly to disks - be able to clog up the whole system.
Maybe if you could get the result of
alt-sysrq-P
I tried some kernel changes enabling the HyperThreading on the (single) P4 processor and enabling CONFIG_PREEMPT_VOLUNTARY=y, but with no success.
During the lookup Alt-SysRq-P constantly says that:
EIP is at mwait_idle+0x1a/0x2e
While Alt-SysRq-T shows - among other processes - the MD syncing and the bash looked-up; this is the hand-copied call traces:
md3_resync
device_barrier
default_wake_function
sync_request
__generic_unplug_device
md_do_sync
schedule
md_thread
md_thread
kthread
kthread
kernel_thread_helper
bash
io_schedule
sync_buffer
sync_buffer
__wait_on_bit_lock
sync_buffer
out_of_line_wait_on_bit_lock
wake_bit_function
__lock_buffer
do_get_write_access
__ext3_get_inode_loc
jurnal_get_write_access
ext3_reserve_inode_write
ext3_mark_inode_dirty
ext3_dirty_inode
__mark_inode_dirty
update_atime
vfs_readdir
sys_getdents64
filldir64
syscall_call
This is also the top output, which runs regularly during the lookup:
top - 11:40:41 up 7 min, 2 users, load average: 8.70, 4.92, 2.04
Tasks: 70 total, 1 running, 69 sleeping, 0 stopped, 0 zombie Cpu(s): 0.2% us, 0.7% sy, 0.0% ni, 98.7% id, 0.0% wa, 0.0% hi, 0.5% si Mem: 906212k total, 58620k used, 847592k free, 3420k buffers Swap: 1951736k total, 0k used, 1951736k free, 23848k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
829 root 10 -5 0 0 0 S 1 0.0 0:01.70 md3_raid1
2823 root 10 -5 0 0 0 D 1 0.0 0:01.62 md3_resync
1 root 16 0 1956 656 560 S 0 0.1 0:00.52 init
2 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0
5 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1
6 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
7 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/1
8 root 10 -5 0 0 0 S 0 0.0 0:00.01 events/0
9 root 10 -5 0 0 0 S 0 0.0 0:00.01 events/1
10 root 10 -5 0 0 0 S 0 0.0 0:00.00 khelper
11 root 10 -5 0 0 0 S 0 0.0 0:00.00 kthread
14 root 10 -5 0 0 0 S 0 0.0 0:00.00 kblockd/0
15 root 10 -5 0 0 0 S 0 0.0 0:00.00 kblockd/1
16 root 11 -5 0 0 0 S 0 0.0 0:00.00 kacpid
152 root 20 0 0 0 0 S 0 0.0 0:00.00 pdflush
153 root 15 0 0 0 0 D 0 0.0 0:00.00 pdflush
154 root 17 0 0 0 0 S 0 0.0 0:00.00 kswapd0
155 root 11 -5 0 0 0 S 0 0.0 0:00.00 aio/0
156 root 11 -5 0 0 0 S 0 0.0 0:00.00 aio/1
755 root 10 -5 0 0 0 S 0 0.0 0:00.00 kseriod
796 root 10 -5 0 0 0 S 0 0.0 0:00.00 ata/0
797 root 11 -5 0 0 0 S 0 0.0 0:00.00 ata/1
799 root 11 -5 0 0 0 S 0 0.0 0:00.00 scsi_eh_0
800 root 11 -5 0 0 0 S 0 0.0 0:00.00 scsi_eh_1
825 root 15 0 0 0 0 S 0 0.0 0:00.00 kirqd
831 root 10 -5 0 0 0 D 0 0.0 0:00.00 md2_raid1
833 root 10 -5 0 0 0 S 0 0.0 0:00.00 md1_raid1
834 root 10 -5 0 0 0 D 0 0.0 0:00.00 md0_raid1
835 root 15 0 0 0 0 D 0 0.0 0:00.00 kjournald
932 root 18 -4 2192 584 368 S 0 0.1 0:00.19 udevd
1698 root 10 -5 0 0 0 S 0 0.0 0:00.00 khubd
2031 root 22 0 0 0 0 S 0 0.0 0:00.00 kjournald
2032 root 15 0 0 0 0 D 0 0.0 0:00.00 kjournald
2142 daemon 16 0 1708 364 272 S 0 0.0 0:00.00 portmap
2464 root 16 0 2588 932 796 S 0 0.1 0:00.01 syslogdSignature · 4 lines
Niccolo Rigacci Firenze - Italy War against Iraq? Not in my name!
Quoted 3 lines · Bill Cizek
I was able to work around this by lowering
/proc/sys/dev/raid/speed_limit_maxto a value below my disk thruput value (~ 50 MB/s) as follows:
IMHO a much better fix is to use the cfq I/O scheduler during the rebuild. The default anticipatory scheduler gives horrible latencies and can cause the machine to appear as 'locked up' if there is heavy I/O load like a RAID reconstruct or heavy database usage.
The price of cfq is lower throughput (higher RAID rebuild time) than with the anticipatory I/O scheduler.
Gabor
Signature · 4 lines
---------------------------------------------------------
MTA SZTAKI Computer and Automation Research Institute
Hungarian Academy of Sciences
---------------------------------------------------------Quoted 3 lines · Gabor Gombas
IMHO a much better fix is to use the cfq I/O scheduler during the rebuild.
Yes, changing the default I/O to DEFAULT_CFQ solve the problem very well, I get over 40 Mb/s resync speed with no lock-up at all!
Thank you very much, I think we can elaborate a new FAQ entry.
Do you know if it is possible to switch the scheduler at runtime?
Signature · 4 lines
Niccolo Rigacci Firenze - Italy Iraq, missione di pace: 38475 morti - www.iraqbodycount.net
Quoted 1 line · Niccolo Rigacci
Do you know if it is possible to switch the scheduler at runtime?
echo cfq > /sys/block/<disk>/queue/scheduler
Gabor
Signature · 8 lines
---------------------------------------------------------
MTA SZTAKI Computer and Automation Research Institute
Hungarian Academy of Sciences,
Laboratory of Parallel and Distributed Systems
Address : H-1132 Budapest Victor Hugo u. 18-22. Hungary
Phone/Fax : +36 1 329-78-64 (secretary)
W3 : http://www.lpds.sztaki.hu
---------------------------------------------------------Hello Gabor ,
Quoted 3 lines · Gabor Gombas
Quoted 1 line · Niccolo Rigacci
Do you know if it is possible to switch the scheduler at runtime?
echo cfq > /sys/block/<disk>/queue/scheduler
At least one can do a ls of the /sys/block area & then do an automated echo cfq down the tree . Does anyone know of a method to set a default scheduler ? Scanning down a list or manually maintaining a list seems to be a bug in the waiting . Tia , JimL
Signature · 5 lines
+----------------------------------------------------------------------+ | James W. Laferriere | System Techniques | Give me VMS | | Network Engineer | 3600 14th Ave SE #20-103 | Give me Linux | | [email protected] | Olympia , WA. 98501 | only on AXP | +----------------------------------------------------------------------+
Quoted 5 lines
At least one can do a ls of the
/sys/blockarea & then do an automated
echo cfq down the tree . Does anyone know of a method to set a default
scheduler ?
My be I didn't understand the question...
You decide what schedulers are available at kernel compile time, also at kernel compile time you decide which is the default i/o scheduler.
Signature · 4 lines
Niccolo Rigacci Firenze - Italy Iraq, missione di pace: 38475 morti - www.iraqbodycount.net
Quoted 5 lines · Mr. James W. Laferriere
At least one can do a ls of the
/sys/blockarea & then do an automated
echo cfq down the tree . Does anyone know of a method to set a default
scheduler ?
RTFM: Documentation/kernel-parameters.txt in the kernel source.
Gabor
Signature · 4 lines
---------------------------------------------------------
MTA SZTAKI Computer and Automation Research Institute
Hungarian Academy of Sciences
---------------------------------------------------------Quoted 11 lines · Mr. James W. Laferriere
Hello Gabor ,
Quoted 5 lines · Gabor Gombas
Quoted 1 line · Niccolo Rigacci
Do you know if it is possible to switch the scheduler at runtime?
echo cfq > /sys/block/<disk>/queue/scheduler
Quoted 7 lines
At least one can do a ls of the
/sys/blockarea & then do an
automated
echo cfq down the tree . Does anyone know of a method to set a
default
scheduler ? Scanning down a list or manually maintaining a list
seems
to be a bug in the waiting . Tia , JimL
Thought I posted this... it can be set in kernel build or on the bloot parameters from grub/lilo.
2nd thought: set it to cfq by default, then at the END of rc.local, if there are no arrays rebuilding, change to something else if you like.
Signature · 3 lines
bill davidsen <[email protected]> CTO TMR Associates, Inc Doing interesting things with small computers since 1979