Scheduling in interrupt BUG.
Subsystems (2)
Hi Guys,
Once upon a time on my
x86 UP box, UP kernel 2.4.4, (64M ram, 260M swap)
http://home.sch.bme.hu/~cell/.config
I hit a reproducable "Scheduling in interrupt" BUG.
Also reproduced with 128M ram and low memory pressure (first I suspected it is related to swapping)
Running lots of pppd version 2.4.0 (pppoe) sessions almost at the same time.
(before the crash the pppoe sessions work fine)
It crashed after 89 sessions, 473 another time.. (depending on the phase of Jupiter moons I guess .. I still have to verify this), usually much before memory is exhausted (30k mem/pppd process). To do this you have to patch ppp_generic.c
http://x-dsl.hu/~cell/ppp_generic_hash/, because
otherwise we hit 'NULL ptr in all_ppp_units list'
BUG much _more likely_ than this 'sched.c line 709 thingy'..
EIP: c010faa4 <schedule+388/394> <===== sched.c schedule(), line 709:
which is ~ printk("Scheduling in interrupt");BUG();
Trace:
0xc01ddac5 <__lock_sock+53>: movl $0x0,0x1c(%esp,1) 0xc01ddacd <__lock_sock+61>: mov %ebx,0x20(%esp,1) 0xc01ddad1 <__lock_sock+65>: movl $0x0,0x24(%esp,1) 0xc01ddad9 <__lock_sock+73>: movl $0x0,0x28(%esp,1) 0xc01ddae1 <__lock_sock+81>: lea 0x1c(%esp,1),%esi 0xc01ddae5 <__lock_sock+85>: lea 0x34(%edi),%eax 0xc01ddae8 <__lock_sock+88>: mov %esi,%edx 0xc01ddaea <__lock_sock+90>: call 0xc0110598 <add_wait_queue_exclusive> 0xc01ddaef <__lock_sock+95>: nop 0xc01ddaf0 <__lock_sock+96>: movl $0x2,(%ebx) 0xc01ddaf6 <__lock_sock+102>: decl 0xc02f75ec 0xc01ddafc <__lock_sock+108>: call 0xc010f71c <schedule> ***************** 0xc01ddb01 <__lock_sock+113>: incl 0xc02f75ec 0xc01ddb07 <__lock_sock+119>: cmpl $0x0,0x30(%edi) 0xc01ddb0b <__lock_sock+123>: jne 0xc01ddaf0 <__lock_sock+96>
-----0xc01a315c <pppoe_backlog_rcv>: push %esi
0xc01a315d <pppoe_backlog_rcv+1>: push %ebx
0xc01a315e <pppoe_backlog_rcv+2>: mov 0xc(%esp,1),%ebx
0xc01a3162 <pppoe_backlog_rcv+6>: incl 0xc02f75ec
0xc01a3168 <pppoe_backlog_rcv+12>: cmpl $0x0,0x30(%ebx)
0xc01a316c <pppoe_backlog_rcv+16>:
je 0xc01a3177 <pppoe_backlog_rcv+27>
0xc01a316e <pppoe_backlog_rcv+18>: push %ebx
0xc01a316f <pppoe_backlog_rcv+19>: call 0xc01dda90
<__lock_sock> ************
0xc01a3174 <pppoe_backlog_rcv+24>: add $0x4,%esp
0xc01a3177 <pppoe_backlog_rcv+27>: movl $0x1,0x30(%ebx)
0xc01a317e <pppoe_backlog_rcv+34>: decl 0xc02f75ec
0xc01a3184 <pppoe_backlog_rcv+40>: mov 0x10(%esp,1),%eax
--------
0xc01ddb2c <__release_sock>: push %esi
0xc01ddb2d <__release_sock+1>: push %ebx
0xc01ddb2e <__release_sock+2>: mov 0xc(%esp,1),%esi
0xc01ddb32 <__release_sock+6>: mov 0xb8(%esi),%eax
0xc01ddb38 <__release_sock+12>: movl $0x0,0xbc(%esi)
0xc01ddb42 <__release_sock+22>: movl $0x0,0xb8(%esi)
0xc01ddb4c <__release_sock+32>: lea 0x0(%esi,1),%esi
0xc01ddb50 <__release_sock+36>: mov (%eax),%ebx
0xc01ddb52 <__release_sock+38>: movl $0x0,(%eax)
0xc01ddb58 <__release_sock+44>: push %eax
0xc01ddb59 <__release_sock+45>: push %esi
0xc01ddb5a <__release_sock+46>: mov 0x31c(%esi),%eax
0xc01ddb60 <__release_sock+52>: call *%eax ********************
0xc01ddb62 <__release_sock+54>: mov %ebx,%eax
0xc01ddb64 <__release_sock+56>: add $0x8,%esp
0xc01ddb67 <__release_sock+59>: test %eax,%eaxint pppoe_backlog_rcv(struct sock *sk, struct sk_buff *skb)
{
lock_sock(sk);
pppoe_rcv_core(sk, skb);
release_sock(sk);
return 0;
}What else should I check? How can we fix it?
PPPoE is more and more frequently used nowadays because of ADSL services. I think this can effect its stability (guess which direction ;-)
even with one session (though probably not that bad as with many sessions).
Have a nice week:
Cell
Signature · 1 line
Alice? WTFIA?
Scheduling in interrupt BUG. [Patch]
Quoted 19 lines · Marcell GAL
Hi Guys,
Once upon a time on my
x86 UP box, UP kernel 2.4.4, (64M ram, 260M swap)
http://home.sch.bme.hu/~cell/.config
I hit a reproducable "Scheduling in interrupt" BUG.
Also reproduced with 128M ram and low memory pressure (first I suspected it is related to swapping)
Running lots of pppd version 2.4.0 (pppoe) sessions almost at the same time.
(before the crash the pppoe sessions work fine)
It crashed after 89 sessions, 473 another time.. (depending on the phase of Jupiter moons I guess .. I still have to verify this), usually much before memory is exhausted (30k mem/pppd process). To do this you have to patchppp_generic.c
http://x-dsl.hu/~cell/ppp_generic_hash/, because
otherwise we hit 'NULL ptr inall_ppp_unitslist'
BUG much_morelikely_than this 'sched.cline 709 thingy'..
Quoted 4 lines
EIP:
c010faa4<schedule+388/394> <=====sched.cschedule(), line 709:
which is ~printk("Scheduling in interrupt");BUG();From what I've seen, you have a call chain consisting of:
__release_sock -> pppoe_backlog_rcv -> __lock_sock
This is going to be bad, because when __release_sock calls sk->backlog_rcv, lock.users is still non-zero and thus the lock_sock operation will block (leading to deadlock). This problem is fixed with the patch that I've added below.
You're seeing the "Scheduling in interrupt" message because the combined effect of the various spin_lock/unlock calls in release_sock and __release_sock at the time of the call to sk->backlog_rcv is to increase the local bh count.
Having looked at the code for locking sockets I am concerned that the
locking procedures for tcp may be wrong. __release_sock releases the
socket spinlock before calling sk->backlog_rcv() (== tcp_v4_do_rcv), however the comments at the top of tcp_v4_do_rcv() assert that the socket's spinlock is held (which is definitely not the case).
Anybody care to comment on this?
Michal Ostrowski
[email protected]
net/pppoe.c renamed from net/pppoe.c~ +4 -3
@@ -5,7 +5,7 @@* PPPoE --- PPP over Ethernet (RFC 2516) * * -* Version: 0.6.5 +* Version: 0.6.6 * * 030700 : Fixed connect logic to allow for disconnect. * 270700 : Fixed potential SMP problems; we must protect against
@@ -19,6 +19,7 @@* 051000 : Initialization cleanup. * 111100 : Fix recvmsg. * 050101 : Fix PADT procesing. +* 140501 : pppoe_backlog_rcv must call bh_lock_sock, not lock_sock. * * Author: Michal Ostrowski <[email protected]> * Contributors:
@@ -384,9 +385,9 @@***********************************************************************/ int pppoe_backlog_rcv (struct sock *sk,struct sk_buff *skb) { -lock_sock (sk); +bh_lock_sock (sk);pppoe_rcv_core (sk, skb); -release_sock (sk); +bh_unlock_sock (sk);return 0 ; }
Quoted 9 lines · Marcell GAL
int
pppoe_backlog_rcv(struct sock *sk, struct sk_buff *skb){ lock_sock(sk); pppoe_rcv_core(sk, skb); release_sock(sk); return 0; }
The backlog_rcv() method is called inside local_bh_disable()
and so cannot call lock_sock(). Definitely a bug in pppoe.
It looks like pppoe_backlog_rcv() should be using bh_lock_sock().
Re: Scheduling in interrupt BUG. [Patch]
In reply to Marcell GALHi,
This patch solved the problem. Should be ready for inclusion in 2.4. No more 'Scheduling in interrupt' under those conditions. Thanx for the thoughts, solution and the amazing speed. You guys are doing a really great job!
I hope we can get the earlier mentioned NULL ptr in all_ppp_units list straight
soon. (I have a simple workaround - the mentioned hash, that even improves speed,
but I a real fix would be more satisfaction. The relevant part of ppp_generic.c
is so simple that it's really strange it is not correct.. ).
thanx:
Cell
Quoted 2 lines · Michal Ostrowski
Anybody care to comment on this?
[email protected]
drivers/net/pppoe.c Mon May 14 22:11:25 2001 renamed from drivers/net/pppoe.c Mon May 14 22:06:44 2001 +4 -3
@@ -4,9 +4,9 @@ * PPPoX --- Generic PPP encapsulation socket family * PPPoE --- PPP over Ethernet (RFC 2516) * * - * Version: 0.6.5 + * Version: 0.6.6 * * 030700 : Fixed connect logic to allow for disconnect. * 270700 : Fixed potential SMP problems; we must protect against * simultaneous invocation of ppp_input
@@ -18,8 +18,9 @@ * in pppoe_release. * 051000 : Initialization cleanup. * 111100 : Fix recvmsg. * 050101 : Fix PADT procesing. + * 140501 : pppoe_backlog_rcv must call bh_lock_sock, not lock_sock. * * Author: Michal Ostrowski <[email protected]> * Contributors: * Arnaldo Carvalho de Melo <[email protected]>
@@ -383,11 +384,11 @@ * ***********************************************************************/ int pppoe_backlog_rcv(struct sock *sk, struct sk_buff *skb) { - lock_sock(sk); + bh_lock_sock(sk); pppoe_rcv_core(sk, skb); - release_sock(sk); + bh_unlock_sock(sk); return 0; }
--
You'll never see all the places, or read all the books, but fortunately, they're not all recommended.
Re: Scheduling in interrupt BUG. [Patch]
Quoted 7 lines · Marcell Gal
Hi,
This patch solved the problem. Should be ready for inclusion in 2.4. No more 'Scheduling in interrupt' under those conditions. Thanx for the thoughts, solution and the amazing speed. You guys are doing a really great job!
Alexey pointed out a much nicer/correct/elegant/efficient solution to this problem and I think that that's the way to go.
New patch below.
Michal Ostrowski
[email protected]
net/pppoe.c renamed from net/pppoe.c~ +3 -18
@@ -5,7 +5,7 @@* PPPoE --- PPP over Ethernet (RFC 2516) * * -* Version: 0.6.5 +* Version: 0.6.6 * * 030700 : Fixed connect logic to allow for disconnect. * 270700 : Fixed potential SMP problems; we must protect against
@@ -19,6 +19,7 @@* 051000 : Initialization cleanup. * 111100 : Fix recvmsg. * 050101 : Fix PADT procesing. +* 140501 : Use pppoe_rcv_core to handle all backlog. (Alexey) * * Author: Michal Ostrowski <[email protected]> * Contributors:
@@ -376,22 +377,6 @@return ret; } - -/************************************************************************ -* -* Receive wrapper called in process context. -* -***********************************************************************/ -int pppoe_backlog_rcv (struct sock *sk,struct sk_buff *skb) -{ -lock_sock (sk); -pppoe_rcv_core (sk, skb); -release_sock (sk); -return 0 ; -} - - -/************************************************************************ * * Receive a PPPoE Discovery frame.
@@ -481,7 +466,7 @@ sk->protocol =PX_PROTO_OE ; sk->family =PF_PPPOX ; - sk->backlog_rcv = pppoe_backlog_rcv; + sk->backlog_rcv = pppoe_rcv_core; sk->next =NULL ; sk->pprev =NULL ; sk->state =PPPOX_NONE ;
Re: Scheduling in interrupt BUG. [Patch]
In reply to Marcell GALQuoted 7 lines · Michal Ostrowski
Having looked at the code for locking sockets I am concerned that the
locking procedures for tcp may be wrong.__release_sockreleases the
socket spinlock before callingsk->backlog_rcv()(==tcp_v4_do_rcv), however the comments at the top oftcp_v4_do_rcv()assert that the socket's spinlock is held (which is definitely not the case).Anybody care to comment on this?
Looks ok for me.
The user socket lock (lock.users>0) is held while __release_sock runs, which is also sufficient to protect it as all new packets will go into backlog. The spinlock comment only applies to bottom halves.
-Andi