stuck tasks
Hi - I'm not getting any joy with Fedora's bugzilla. Has anyone seen problems like this with Fedora 12? Our systems have recently been getting stuck while rsyncing data onto an MD device:
https://bugzilla.redhat.com/show_bug.cgi?id=578549
INFO: task kthreadd:2 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kthreadd D 0000000000000002 0 2 0 0x00000000
ffff88007dbfd4c0 0000000000000046 0000000000000000 0000000a00000000 ffff880000000001 ffff880079f9b800 ffff88007dbfdfd8 ffff88007dbfdfd8 ffff88007dbf1b38 000000000000f980 0000000000015740 ffff88007dbf1b38
Call Trace: [<ffffffff8107c30d>] ? ktime_get_ts+0x85/0x8e [<ffffffff810d604d>] ? sync_page+0x0/0x4a [<ffffffff810d604d>] ? sync_page+0x0/0x4a [<ffffffff814546f5>] io_schedule+0x43/0x5d [<ffffffff810d6093>] sync_page+0x46/0x4a [<ffffffff81454c48>] __wait_on_bit+0x48/0x7b ...
Several processes end up stuck in a D state:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 2 0.0 0.0 0 0 ? D Mar31 0:07 [kthreadd] root 14 0.0 0.0 0 0 ? D Mar31 9:38 [async/mgr] root 17 0.0 0.0 0 0 ? D Mar31 0:00 [bdi- default] root 34 0.0 0.0 0 0 ? D Mar31 10:06 [kswapd0] root 5509 0.0 0.3 50732 7900 ? D Apr09 0:03 rsync -
raHSx --stats --whole-file --numeric-ids --link- dest=/xback2_back1/YY/20100407-000501 --exclude=/lost+found -- exclude=.mozilla/*/*/Cache/* XX:/XX_data1/data/YY/ /xback2_back1/YY/20100409-000502/
root 17457 0.0 0.2 61920 5756 ? D Apr11 0:00 python
/data/soft3/backup/diskbackup/diskbackup.py /data/soft3/backup/diskbackup/main.cfg
root 18402 0.0 0.0 0 0 ? D Apr09 0:11 [flush-9:0] root 20259 0.0 0.1 4284 3424 ? DN Apr11 0:00
/usr/sbin/prelink -av -mR -q
It only seems to affect our MD systems. The kernel is 2.6.32.10-90.fc12.x86_64. The systems have 3ware 96xx controllers. This kernel does have the issue when there are lots of aio processes.
The two affected systems have different file systems: xfs and ext3.
Jeremy
Quoted 7 lines · Jeremy Sanders
Hi - I'm not getting any joy with Fedora's bugzilla. Has anyone seen problems like this with Fedora 12? Our systems have recently been getting stuck while rsyncing data onto an MD device:
You need to lower the sync speed by catting a value into /sys/block/md{n}/md/sync_speed_max the value should be about 1/3 lower than the max speed you see (cat /proc/mdstat) now that it's not yet limited. Set up a script to set it at boot.
If it was not happening for you on older kernels might be a good sign: it might mean that the resync is faster now...
What is the sync speed you see (cat /proc/mdstat)? How many drives do you have and what type of raid is that?
Quoted 5 lines · MRK
You need to lower the sync speed by catting a value into
/sys/block/md{n}/md/sync_speed_max
the value should be about 1/3 lower than the max speed you see (cat/proc/mdstat) now that it's not yet limited.
Set up a script to set it at boot.
Thanks - I'll try it. It didn't happen in a raid sync, just during an rsync run. Would this have any effect on normal operation?
Quoted 5 lines
If it was not happening for you on older kernels might be a good sign:
it might mean that the resync is faster now...What is the sync speed you see (cat
/proc/mdstat)? How many drives do you have and what type of raid is that?
We're only getting 30MB/s. I thought it used to be quite a lot faster. It seems to slow down as the sync progresses:
[root@xback2 ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdb1[0] sdj1[10](S) sdk1[9] sdl1[8] sdi1[7] sdh1[6] sdg1[5] sdf1[4] sde1[3] sdd1[2] sdc1[1]
8788959360 blocks level 5, 32k chunk, algorithm 2 [10/10] [UUUUUUUUUU]
[>....................] resync = 1.4% (14078544/976551040) finish=501.4min speed=31990K/sec
unused devices: <none>
The drives are connected to a single 3ware Inc 9650SE SATA-II RAID PCIe card on this particular system. They're SATA 1GB drives.
Jeremy
Quoted 12 lines · Jeremy Sanders
Quoted 6 lines · MRK
You need to lower the sync speed by catting a value into
/sys/block/md{n}/md/sync_speed_max
the value should be about 1/3 lower than the max speed you see (cat/proc/mdstat) now that it's not yet limited.
Set up a script to set it at boot.Thanks - I'll try it. It didn't happen in a raid sync, just during an rsync run. Would this have any effect on normal operation?
!?!!
Mistake of mine, but I might have gotten the right answer by chance I had read resyncing but you wrote rsyncing.
But you were in fact also resyncing from what you write below:
Quoted 9 lines
Quoted 6 lines
If it was not happening for you on older kernels might be a good sign:
it might mean that the resync is faster now...What is the sync speed you see (cat
/proc/mdstat)? How many drives do you have and what type of raid is that?We're only getting 30MB/s. I thought it used to be quite a lot faster. It seems to slow down as the sync progresses:
Quoted 8 lines
[root@xback2 ~]# cat
/proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdb1[0] sdj1[10](S) sdk1[9] sdl1[8] sdi1[7] sdh1[6] sdg1[5] sdf1[4] sde1[3] sdd1[2] sdc1[1]8788959360 blocks level 5, 32k chunk, algorithm 2 [10/10] [UUUUUUUUUU] [>....................] resync = 1.4% (14078544/976551040)finish=501.4min speed=31990K/sec
Resync speed is indeed quite low if you confirm there is no other disk activity.
Instead if rsync is also running, you need to stop that one to have a proper resync speed measurement (to compute the value to be entered into sync_speed_max as per my previous email). Do you have disk write caches activated? See that with tw_cli (3ware's CLI)
How much is /sys/block/md{n}/md/stripe_cache_size? Pump it up to 32768.
Quoted 7 lines
unused devices:<none>
The drives are connected to a single 3ware Inc 9650SE SATA-II RAID PCIe card on this particular system. They're SATA 1GB drives.
Jeremy
Quoted 4 lines · MRK
Mistake of mine, but I might have gotten the right answer by chance I had read resyncing but you wrote rsyncing.
But you were in fact also resyncing from what you write below:
Not at the time of crash! Sorry for the confusion. This happened on reboot as the drive didn't properly unmount.
Quoted 2 lines
Resync speed is indeed quite low if you confirm there is no other disk activity.
It's not doing anything else on that drive. I think it would be faster but the option CONFIG_MULTICORE_RAID456 was switched on in this kernel, so there are lots of async processes fighting each other (190 of them on this system).
Quoted 6 lines
Instead if rsync is also running, you need to stop that one to have a proper resync speed measurement (to compute the value to be entered into
sync_speed_maxas per my previous email).
Do you have disk write caches activated? See that withtw_cli(3ware's CLI) How much is/sys/block/md{n}/md/stripe_cache_size? Pump it up to 32768.
The disk write caches are on. The raid device is fast if you do bonnie tests to them (>100MBs).
I tried the strip_cache_size option. The speed stays around 10 MB/s on this system. It looks like the sync speed has slowed down a lot since the sync started.
Jeremy
Quoted 5 lines · Jeremy Sanders
It's not doing anything else on that drive. I think it would be faster but the option
CONFIG_MULTICORE_RAID456was switched on in this kernel, so there are lots of async processes fighting each other (190 of them on this system).
Oh it's this one then. I read it has already been discussed here... you really need to remove the multicore implementation: it's experimental as of now and *much* slower than single core. I don't know why the your distro maintainer has activated it. What distro is that? Someone should probably tell the maintainers to disable it by default. I think all your problems will go away if you can recompile with multicore off.