null-pointer in task_rq_lock (2.6.35 to 3.0-rc7)
Hi,
reloading an apache httpd can crash the kernel since 2.6.35. It seems that tasks are removed between creating the task-list and calling wake_up_sem_queue_do in freeary. The pointers to the task_struct elements end up in try_to_wake_up and sometimes contain 0x0 there.
The problem did not exist in 2.6.34. It does not show up on single processor systems. Depending on the apache httpd settings it only takes a few tries to kill the system on our 8-core servers. Dualcore did not want to crash, maybe it really needs more than one real CPU. Various gcc versions (4.1 to 4.6) were used.
If anyone wants to crash a system using an prefork apache httpd:
<IfModule mpm_prefork_module>
ServerLimit 512
StartServers 50
MinSpareServers 50
MaxSpareServers 100
MaxClients 200
MaxRequestsPerChild 500
</IfModule>
(Details do not seem to matter but some settings did not die fast.)
I'm not able to fix or understand this bug myself, its already in bugzilla with the call trace:
https://bugzilla.kernel.org/show_bug.cgi?id=27142
Is there any more useful information I can provide? Anything to test? Does anyone know of changes from 2.6.34 to 2.6.35 that might have broken this? (The diff and the changelog do not enlighten me, too much changed and I understand little of it.)
Thanks,
Harald
Quoted 23 lines · Harald Laabs
Hi,
reloading an apache httpd can crash the kernel since 2.6.35. It seems that tasks are removed between creating the task-list and callingwake_up_sem_queue_doin freeary. The pointers to thetask_structelements end up intry_to_wake_upand sometimes contain0x0there.
The problem did not exist in 2.6.34. It does not show up on single processor systems. Depending on the apache httpd settings it only takes a few tries to kill the system on our 8-core servers. Dualcore did not want to crash, maybe it really needs more than one real CPU. Various gcc versions (4.1 to 4.6) were used.If anyone wants to crash a system using an prefork apache httpd:
<IfModulempm_prefork_module>
ServerLimit 512
StartServers 50
MinSpareServers 50
MaxSpareServers 100
MaxClients 200
MaxRequestsPerChild 500
</IfModule>
(Details do not seem to matter but some settings did not die fast.)
Quoted 8 lines
I'm not able to fix or understand this bug myself, its already in bugzilla with the call trace:
https://bugzilla.kernel.org/show_bug.cgi?id=27142Is there any more useful information I can provide? Anything to test? Does anyone know of changes from 2.6.34 to 2.6.35 that might have broken this? (The diff and the changelog do not enlighten me, too much changed and I understand little of it.)
I feel commit 0a2b9d4c79671b059568 might be the bug origin (ipc/sem.c: move wake_up_process out of the spinlock section)
CC Manfred & Andrew
Quoted 7 lines · Eric Dumazet
Le mardi 19 juillet 2011 à 22:03 +0200, Harald Laabs a écrit :Hi,
Quoted 6 lines
I'm not able to fix or understand this bug myself, its already in bugzilla with the call trace:
https://bugzilla.kernel.org/show_bug.cgi?id=27142Is there any more useful information I can provide? Anything to test?
Could you build a kernel with CONFIG_DEBUG_LIST enabled? Does it report anything?
Quoted 6 lines
Quoted 3 lines
Does anyone know of changes from 2.6.34 to 2.6.35 that might have broken this? (The diff and the changelog do not enlighten me, too much changed and I understand little of it.)
I feel commit
0a2b9d4c79671b059568might be the bug origin (ipc/sem.c: movewake_up_processout of the spinlock section)
I'll try to reproduce the bug tomorrow.
Perhaps a race with multiple processes sleeping, some/all woken up by a signal an a concurrent IPC_RM.
But I don't see the bug yet.
--
Manfred
Quoted 7 lines · Harald Laabs <kernelml <at> dasr.de> writes:
Hi,
reloading an apache httpd can crash the kernel since 2.6.35. It seems that tasks are removed between creating the task-list and callingwake_up_sem_queue_doin freeary. The pointers to thetask_structelements end up intry_to_wake_upand sometimes contain0x0there.
Had the same in production.
Linux version 2.6.35-22-server (buildd@allspice) (gcc version 4.4.5 (Ubuntu/Linaro 4.4.4-14ubuntu4) ) #33-Ubuntu SMP Sun Sep 19 20:48:58 UTC 2010 (Ubuntu 2.6.35-22.33-server 2.6.35.4)
Apache/2.2.16 (Ubuntu) PHP/5.3.3-1ubuntu9 with Suhosin-Patch mod_ssl/2.2.16
OpenSSL/0.9.8o
Appeared 2 times after apache2 reload during cron daily jobs.