Skip to contentSkip to conversation

stuck tasks

DISCUSSION Jeremy Sanders 2 participants 6 messages Apr 12, 2010 MD RAID

lore
Share & tools
lore
#0 lore raw

Hi - I'm not getting any joy with Fedora's bugzilla. Has anyone seen problems like this with Fedora 12? Our systems have recently been getting stuck while rsyncing data onto an MD device:

https://bugzilla.redhat.com/show_bug.cgi?id=578549

INFO: task kthreadd:2 blocked for more than 120 seconds.

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kthreadd D 0000000000000002 0 2 0 0x00000000
ffff88007dbfd4c0 0000000000000046 0000000000000000 0000000a00000000 ffff880000000001 ffff880079f9b800 ffff88007dbfdfd8 ffff88007dbfdfd8 ffff88007dbf1b38 000000000000f980 0000000000015740 ffff88007dbf1b38

Call Trace:
 [<ffffffff8107c30d>] ? ktime_get_ts+0x85/0x8e
 [<ffffffff810d604d>] ? sync_page+0x0/0x4a
 [<ffffffff810d604d>] ? sync_page+0x0/0x4a
 [<ffffffff814546f5>] io_schedule+0x43/0x5d
 [<ffffffff810d6093>] sync_page+0x46/0x4a
 [<ffffffff81454c48>] __wait_on_bit+0x48/0x7b
 ...

Several processes end up stuck in a D state:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         2  0.0  0.0      0     0 ?        D    Mar31   0:07 [kthreadd]
root        14  0.0  0.0      0     0 ?        D    Mar31   9:38 [async/mgr]
root        17  0.0  0.0      0     0 ?        D    Mar31   0:00 [bdi-
default]
root        34  0.0  0.0      0     0 ?        D    Mar31  10:06 [kswapd0]
root      5509  0.0  0.3  50732  7900 ?        D    Apr09   0:03 rsync -

raHSx --stats --whole-file --numeric-ids --link- dest=/xback2_back1/YY/20100407-000501 --exclude=/lost+found -- exclude=.mozilla/*/*/Cache/* XX:/XX_data1/data/YY/ /xback2_back1/YY/20100409-000502/
root 17457 0.0 0.2 61920 5756 ? D Apr11 0:00 python
/data/soft3/backup/diskbackup/diskbackup.py /data/soft3/backup/diskbackup/main.cfg

root     18402  0.0  0.0      0     0 ?        D    Apr09   0:11 [flush-9:0]
root     20259  0.0  0.1   4284  3424 ?        DN   Apr11   0:00 

/usr/sbin/prelink -av -mR -q

It only seems to affect our MD systems. The kernel is 2.6.32.10-90.fc12.x86_64. The systems have 3ware 96xx controllers. This kernel does have the issue when there are lots of aio processes.

The two affected systems have different file systems: xfs and ext3.

Jeremy

#1 lore raw
Quoted 7 lines · Jeremy Sanders

Hi - I'm not getting any joy with Fedora's bugzilla. Has anyone seen problems like this with Fedora 12? Our systems have recently been getting stuck while rsyncing data onto an MD device:

https://bugzilla.redhat.com/show_bug.cgi?id=578549

You need to lower the sync speed by catting a value into /sys/block/md{n}/md/sync_speed_max the value should be about 1/3 lower than the max speed you see (cat /proc/mdstat) now that it's not yet limited. Set up a script to set it at boot.

If it was not happening for you on older kernels might be a good sign: it might mean that the resync is faster now...

What is the sync speed you see (cat /proc/mdstat)? How many drives do you have and what type of raid is that?

#2 lore raw
Quoted 5 lines · MRK

You need to lower the sync speed by catting a value into /sys/block/md{n}/md/sync_speed_max
the value should be about 1/3 lower than the max speed you see (cat /proc/mdstat) now that it's not yet limited.
Set up a script to set it at boot.

Thanks - I'll try it. It didn't happen in a raid sync, just during an rsync run. Would this have any effect on normal operation?

Quoted 5 lines

If it was not happening for you on older kernels might be a good sign:
it might mean that the resync is faster now...

What is the sync speed you see (cat /proc/mdstat)? How many drives do you have and what type of raid is that?

We're only getting 30MB/s. I thought it used to be quite a lot faster. It seems to slow down as the sync progresses:

[root@xback2 ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdb1[0] sdj1[10](S) sdk1[9] sdl1[8] sdi1[7] sdh1[6] sdg1[5] sdf1[4] sde1[3] sdd1[2] sdc1[1]

      8788959360 blocks level 5, 32k chunk, algorithm 2 [10/10] [UUUUUUUUUU]
      [>....................]  resync =  1.4% (14078544/976551040) 

finish=501.4min speed=31990K/sec

unused devices: <none>

The drives are connected to a single 3ware Inc 9650SE SATA-II RAID PCIe card on this particular system. They're SATA 1GB drives.

Jeremy

#3 lore raw
Quoted 12 lines · Jeremy Sanders
Quoted 6 lines · MRK

You need to lower the sync speed by catting a value into /sys/block/md{n}/md/sync_speed_max
the value should be about 1/3 lower than the max speed you see (cat /proc/mdstat) now that it's not yet limited.
Set up a script to set it at boot.

Thanks - I'll try it. It didn't happen in a raid sync, just during an rsync run. Would this have any effect on normal operation?

!?!!

Mistake of mine, but I might have gotten the right answer by chance I had read resyncing but you wrote rsyncing.
But you were in fact also resyncing from what you write below:

Quoted 9 lines
Quoted 6 lines

If it was not happening for you on older kernels might be a good sign:
it might mean that the resync is faster now...

What is the sync speed you see (cat /proc/mdstat)? How many drives do you have and what type of raid is that?

We're only getting 30MB/s. I thought it used to be quite a lot faster. It seems to slow down as the sync progresses:

Quoted 8 lines

[root@xback2 ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdb1[0] sdj1[10](S) sdk1[9] sdl1[8] sdi1[7] sdh1[6] sdg1[5] sdf1[4] sde1[3] sdd1[2] sdc1[1]

       8788959360 blocks level 5, 32k chunk, algorithm 2 [10/10] [UUUUUUUUUU]
       [>....................]  resync =  1.4% (14078544/976551040)

finish=501.4min speed=31990K/sec

Resync speed is indeed quite low if you confirm there is no other disk activity.
Instead if rsync is also running, you need to stop that one to have a proper resync speed measurement (to compute the value to be entered into sync_speed_max as per my previous email). Do you have disk write caches activated? See that with tw_cli (3ware's CLI)
How much is /sys/block/md{n}/md/stripe_cache_size? Pump it up to 32768.

Quoted 7 lines

unused devices:<none>

The drives are connected to a single 3ware Inc 9650SE SATA-II RAID PCIe card on this particular system. They're SATA 1GB drives.

Jeremy

#4 lore raw
Quoted 4 lines · MRK

Mistake of mine, but I might have gotten the right answer by chance I had read resyncing but you wrote rsyncing.
But you were in fact also resyncing from what you write below:

Not at the time of crash! Sorry for the confusion. This happened on reboot as the drive didn't properly unmount.

Quoted 2 lines

Resync speed is indeed quite low if you confirm there is no other disk activity.

It's not doing anything else on that drive. I think it would be faster but the option CONFIG_MULTICORE_RAID456 was switched on in this kernel, so there are lots of async processes fighting each other (190 of them on this system).

Quoted 6 lines

Instead if rsync is also running, you need to stop that one to have a proper resync speed measurement (to compute the value to be entered into sync_speed_max as per my previous email).
Do you have disk write caches activated? See that with tw_cli (3ware's CLI) How much is /sys/block/md{n}/md/stripe_cache_size? Pump it up to 32768.

The disk write caches are on. The raid device is fast if you do bonnie tests to them (>100MBs).

I tried the strip_cache_size option. The speed stays around 10 MB/s on this system. It looks like the sync speed has slowed down a lot since the sync started.

Jeremy

#5 lore raw
Quoted 5 lines · Jeremy Sanders

It's not doing anything else on that drive. I think it would be faster but the option CONFIG_MULTICORE_RAID456 was switched on in this kernel, so there are lots of async processes fighting each other (190 of them on this system).

Oh it's this one then. I read it has already been discussed here... you really need to remove the multicore implementation: it's experimental as of now and *much* slower than single core. I don't know why the your distro maintainer has activated it. What distro is that? Someone should probably tell the maintainers to disable it by default. I think all your problems will go away if you can recompile with multicore off.