Skip to contentSkip to conversation

null-pointer in task_rq_lock (2.6.35 to 3.0-rc7)

BUG Harald Laabs 4 participants 4 messages Jul 19, 2011 – Aug 15, 2011

lore
Share & tools
lore
#0 lore raw

Hi,
reloading an apache httpd can crash the kernel since 2.6.35. It seems that tasks are removed between creating the task-list and calling wake_up_sem_queue_do in freeary. The pointers to the task_struct elements end up in try_to_wake_up and sometimes contain 0x0 there.
The problem did not exist in 2.6.34. It does not show up on single processor systems. Depending on the apache httpd settings it only takes a few tries to kill the system on our 8-core servers. Dualcore did not want to crash, maybe it really needs more than one real CPU. Various gcc versions (4.1 to 4.6) were used.

If anyone wants to crash a system using an prefork apache httpd:
<IfModule mpm_prefork_module>
ServerLimit 512
StartServers 50
MinSpareServers 50
MaxSpareServers 100
MaxClients 200
MaxRequestsPerChild 500
</IfModule>
(Details do not seem to matter but some settings did not die fast.)

I'm not able to fix or understand this bug myself, its already in bugzilla with the call trace:
https://bugzilla.kernel.org/show_bug.cgi?id=27142

Is there any more useful information I can provide? Anything to test? Does anyone know of changes from 2.6.34 to 2.6.35 that might have broken this? (The diff and the changelog do not enlighten me, too much changed and I understand little of it.)

Thanks,
Harald

#1 lore raw
Quoted 23 lines · Harald Laabs

Hi,
reloading an apache httpd can crash the kernel since 2.6.35. It seems that tasks are removed between creating the task-list and calling wake_up_sem_queue_do in freeary. The pointers to the task_struct elements end up in try_to_wake_up and sometimes contain 0x0 there.
The problem did not exist in 2.6.34. It does not show up on single processor systems. Depending on the apache httpd settings it only takes a few tries to kill the system on our 8-core servers. Dualcore did not want to crash, maybe it really needs more than one real CPU. Various gcc versions (4.1 to 4.6) were used.

If anyone wants to crash a system using an prefork apache httpd:
<IfModule mpm_prefork_module>
ServerLimit 512
StartServers 50
MinSpareServers 50
MaxSpareServers 100
MaxClients 200
MaxRequestsPerChild 500
</IfModule>
(Details do not seem to matter but some settings did not die fast.)

Quoted 8 lines

I'm not able to fix or understand this bug myself, its already in bugzilla with the call trace:
https://bugzilla.kernel.org/show_bug.cgi?id=27142

Is there any more useful information I can provide? Anything to test? Does anyone know of changes from 2.6.34 to 2.6.35 that might have broken this? (The diff and the changelog do not enlighten me, too much changed and I understand little of it.)

I feel commit 0a2b9d4c79671b059568 might be the bug origin (ipc/sem.c: move wake_up_process out of the spinlock section)

CC Manfred & Andrew

#2 lore raw
Quoted 7 lines · Eric Dumazet

Le mardi 19 juillet 2011 à 22:03 +0200, Harald Laabs a écrit :Hi,

Quoted 6 lines

I'm not able to fix or understand this bug myself, its already in bugzilla with the call trace:
https://bugzilla.kernel.org/show_bug.cgi?id=27142

Is there any more useful information I can provide? Anything to test?

Could you build a kernel with CONFIG_DEBUG_LIST enabled? Does it report anything?

Quoted 6 lines
Quoted 3 lines

Does anyone know of changes from 2.6.34 to 2.6.35 that might have broken this? (The diff and the changelog do not enlighten me, too much changed and I understand little of it.)

I feel commit 0a2b9d4c79671b059568 might be the bug origin (ipc/sem.c: move wake_up_process out of the spinlock section)

I'll try to reproduce the bug tomorrow.
Perhaps a race with multiple processes sleeping, some/all woken up by a signal an a concurrent IPC_RM.

But I don't see the bug yet.

--
Manfred

#3 lore raw
In reply to Harald Laabs
Quoted 7 lines · Harald Laabs <kernelml <at> dasr.de> writes:

Hi,
reloading an apache httpd can crash the kernel since 2.6.35. It seems that tasks are removed between creating the task-list and calling wake_up_sem_queue_do in freeary. The pointers to the task_struct elements end up in try_to_wake_up and sometimes contain 0x0 there.

Had the same in production.

Linux version 2.6.35-22-server (buildd@allspice) (gcc version 4.4.5 (Ubuntu/Linaro 4.4.4-14ubuntu4) ) #33-Ubuntu SMP Sun Sep 19 20:48:58 UTC 2010 (Ubuntu 2.6.35-22.33-server 2.6.35.4)

Apache/2.2.16 (Ubuntu) PHP/5.3.3-1ubuntu9 with Suhosin-Patch mod_ssl/2.2.16
OpenSSL/0.9.8o

Appeared 2 times after apache2 reload during cron daily jobs.