System crash in tcp_fragment()
Subsystems (2)
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 6 lines
Incase you can not see this, it appears that the addresses of each skb are alright, so the assumption is that the skb passed to
tcp_fragment()has been unlinked whiletcp_fragment()was doing its thing. This implies a need for locking at some higher level and we don't know enough about the tcp code to divine where this might best be done.
2.4 TCP should in theory already have enough locking to prevent this (the socket lock that is aquired by timers and user context socket users)
-Andi
A message that is not in the archive (sent off-list or not yet received): 20020520222937.A1467@averell
Quoted 8 lines · Andi Kleen
Quoted 7 lines
Incase you can not see this, it appears that the addresses of each skb are alright, so the assumption is that the skb passed to
tcp_fragment()has been unlinked whiletcp_fragment()was doing its thing. This implies a need for locking at some higher level and we don't know enough about the tcp code to divine where this might best be done.
Quoted 4 lines
2.4 TCP should in theory already have enough locking to prevent this (the socket lock that is aquired by timers and user context socket users)
-Andi
Here is another oops, not quite the same, AND with an assert failure ahead of it. I append the whole report and some and some observations:
We had two more panics over the weekend.
Here is the analysis from one of them.
---------comments from Dave Howell-------------- Looking at the sysint4l dump, some observations:
- Panic was due to an Oops (Null pointer dereference kernel incident)
- Full system configuration is in kernel startup logs (memory, disks, chipsets,
etc)
- Last part of kernel log has oops info, follows kernel assertion failed
warning:
<4>KERNRL: assertion (atomic_read(&sk->wmem_alloc) == 0) failed at af_inet.c <============== (174):inet_sock_destruct
<1>Unable to handle kernel NULL pointer dereference at virtual address 00000049
<4> printing eip:
<4>c0255196
<1>*pde = 00000000
<4>Oops: 0000
<4>CPU: 1
<4>EIP: 0010:[<c0255196>] Not tainted
<4>EFLAGS: 00010213
<4>eax: c6ace4c8 ebx: 00000000 ecx: 00000004 edx:
00000000
<4>esi: c6ace538 edi: c6ace460 ebp: 00000026 esp:c1219eb4
<4>ds: 0018 es: 0018 ss: 0018
<4>Process swapper (pid: 0, stackpage=c1219000) <4>Stack: c6ace460 c6ace538 c6ace460 004ec3ef c025de3e c6ace460 00000000c72011a0
<4> c1218050 004ec2d2 c02395b2 c6ace460 c6ace538c1218000 004ec3efc025e056
<4> c6ace460 c1218000 00000046 004ebfe7 00000000c1218000 00cf70a0c0128eaa
<4>Call Trace: [<c025de3e>] [<c02395b2>] [<c025e056>] [<c0128eaa>] [<c025df70>]
<4> [<c0128fad>] [<c012483b>] [<c0109704>] [<c0105490>]
[<c0105490>]
[<c0105490>]
<4> [<c01054bc>] [<c0105542>] [<c011d51b>] [<c011d8ad>]
<4>
<4>Code: 0f b6 4b 49 45 f6 c1 82 74 0c 31 d2 89 96 78 01 00 00 0f b6
- Finally at the bottom of the trace the active backtrace, a bit suspect
because it's on the interrupt
side (not trace but process it's attributed to).
===========================
STACK TRACE OF FAILING TASK
===========================
================================================================
STACK TRACE FOR TASK: 0xc1218000 (swapper)
0 tcp_enter_loss+198 [0xc0255196]
1 tcp_retransmit_timer+473 [0xc025de39]
2 tcp_write_timer+225 [0xc025e051]
3 timer_bh+710 [0xc0128ea6]
4 timer_softirq+40 [0xc0128fa8]
5 do_softirq+185 [0xc0124839]
6 do_IRQ+511 [0xc01096ff]
7 do_IRQ+511 [0xc01096ff]
TRACE ERROR 0x1
================================================================
- In comparison with previous dump looks like the same upstream event occured,
with a timer bottom half running and invoking the tcp_retransmit_timer. Last
one caught it oopsing in the tcp_fragment code, this is a bit different but the
upstream path there is the same.
- Same pile of unknown symbol references bringing up dump manually in lcrash,
must be corrupt or wrong system.0 or kerntypes.0. Needs a look.
- Dumped tcp_enter_loss+0 to tcp_enter_loss+200 to see site at
tcp_enter_loss+198.
Code at this site is:
movzbl 0x49(%ebx),%ecx
%ebx is NULL at this point (see above), hence the oops at
00000049.
Code for function is in net/ipv4/tcp_input.c starting at
line 987.
- The failure is in the loop starting at line 1002:
for_retrans_queue(skb, sk, tp) {
cnt++;
if (TCP_SKB_CB(skb)->sacked&TCPCB_RETRANS)
tp->undo_marker = 0;
TCP_SKB_CB(skb)->sacked &=
(~TCPCB_TAGBITS)|TCPCB_SACKED_ACKED;
if
(!(TCP_SKB_CB(skb)->sacked&TCPCB_SACKED_ACKED) || how) {
TCP_SKB_CB(skb)->sacked &=
~TCPCB_SACKED_ACKED;
TCP_SKB_CB(skb)->sacked |=
TCPCB_LOST;
tp->lost_out++;
} else {
tp->sacked_out++;
tp->fackets_out = cnt;
}
}
I didn't fully map the code but think that the expansion of:
if (TCP_SKB_CB(skb)->sacked&TCPCB_RETRANS)is where the zeroed pointer is used. Looks like the intent is that skp is the
iterater variable to loop through the retrans_queue and it got the zero value
set on some iteration, not the first. So my guess is a corrupted queue element
pointer being picked up and used.
- I still would look upstream at the timer bottom half invocation as in both
of the dumps this upstream trace is present, and it seems
like an exception
path for a timeout that leads to a retransmit.
- Also needs a look is the kernel assertion that failed and likely led to the
oops, looks a lot like an allocation failed and returned a
NULL value, this
would be my top culprit to pursue.
Code from af_net.c at line 174:
void inet_sock_destruct(struct sock *sk)
{
__skb_queue_purge(&sk->receive_queue);
__skb_queue_purge(&sk->error_queue);
if (sk->type == SOCK_STREAM && sk->state !=
TCP_CLOSE) {
printk("Attempt to release TCP socket in
state %d %p\n",
sk->state,
sk);
return;
}
if (!sk->dead) {
printk("Attempt to release alive inet socket
%p\n", sk);
return;
}
BUG_TRAP(atomic_read(&sk->rmem_alloc) == 0);
BUG_TRAP(atomic_read(&sk->wmem_alloc) == 0); <<--
assert reported
here
BUG_TRAP(sk->wmem_queued == 0);
BUG_TRAP(sk->forward_alloc == 0);
if (sk->protinfo.af_inet.opt)
kfree(sk->protinfo.af_inet.opt); Continuing on after this likely led to the oops that
killed us.
Signature · 6 lines
George Anzinger [email protected] High-res-timers: http://sourceforge.net/projects/high-res-timers/ Real time sched: http://sourceforge.net/projects/rtsched/ Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml
Hello!
Quoted 3 lines
Is is possible for the other cpu (or even this one given the ksoftirqd stuff) to remove or alter the skb that
tcp_fragment()is processing?
No.
Quoted 1 line
What locks, if any, are needed to prevent this.
They are already applied.
Looking at the last lines of the bugzilla thread, I see the answer:
Quoted 4 lines
- Observation, the
BUG_TRAPassertion likely should have done something better than just report the condition, wonder if an earlier panic or other corrective action may have kept the TCP stack moving instead of the oops? Looks lazy to me...
Well, add BUG() to BIG_TRAP() to oops it earlier. Maybe this will move you closer to real problem.
And also your reasoning about smp_processor_id() sounds strange, preemption code must reschedule thread preemted in the kernel to the same cpu, it is enough to avoid troubles with this.
Alexey
From: george anzinger <[email protected]>
Date: Mon, 20 May 2002 12:42:08 -0700
I wonder if you could help me squash a bug in the tcp code. Here is what we know thus far:
An SMP (x386 dual) 2.4.17 kernel crashes with an attempt to deference NULL at the end of tcp_fragment() (in net/ipv4/tcp_output.c) while attempting to link in the newly created fragment.
The bugzilla report is:
%99 of all such bug reports turn out to be driver bugs where the net driver frees SKBs improperly or there is some missing internal locking in the net device driver.
I think you efforts are better spent auditing what net drivers are being used on this machine :-)
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 14 lines · [email protected] wrote:
Hello!
Quoted 3 lines
Is is possible for the other cpu (or even this one given the ksoftirqd stuff) to remove or alter the skb that
tcp_fragment()is processing?No.
Quoted 1 line
What locks, if any, are needed to prevent this.
They are already applied.
Looking at the last lines of the bugzilla thread, I see the answer:
Quoted 7 lines
Quoted 4 lines
- Observation, the
BUG_TRAPassertion likely should have done something better than just report the condition, wonder if an earlier panic or other corrective action may have kept the TCP stack moving instead of the oops? Looks lazy to me...Well, add
BUG()toBIG_TRAP()to oops it earlier. Maybe this will move you closer to real problem.
Right. Will do.
Quoted 4 lines
And also your reasoning about
smp_processor_id()sounds strange, preemption code must reschedule thread preemted in the kernel to the same cpu, it is enough to avoid troubles with this.
That (ahem) hack was tried and rejected by the powers that be. I admit that is it more work to find the resulting problems, but the fixes seem to be easy and do not to add to overhead, at least thus far.
Quoted 2 lines
Alexey
Signature · 6 lines
George Anzinger [email protected] High-res-timers: http://sourceforge.net/projects/high-res-timers/ Real time sched: http://sourceforge.net/projects/rtsched/ Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml
Quoted 3 lines
And also your reasoning about
smp_processor_id()sounds strange, preemption code must reschedule thread preemted in the kernel to the same cpu, it is enough to avoid troubles with this.
It's unfortunately the truth. Even in_interrupt() is buggy current on SMP preemptive.
-Andi
Network oops
In reply to a message not in the archiveIs this sort of thing expected when the NIC is out of poop?
<4>eth1: card reports no resources.
<4>KERNEL: assertion ((int)tcp_packets_in_flight(tp) >=
0) failed at tcp_input.c(956):tcp_sacktag_write_queue <7>Leak l=8 4
<1>Unable to handle kernel NULL pointer dereference at
virtual address 00000049
<4> printing eip:
<4>c025b7aa
<1>*pde = 00000000
<4>Oops: 0000
<4>CPU: 0
<4>EIP: 0010:[<c025b7aa>] Not tainted
<4>EFLAGS: 00010207
<4>eax: c40d82c8 ebx: 00000000 ecx: 00000004 edx:
00000000
<4>esi: c40d8338 edi: c40d8260 ebp: c0437eb4 esp:
c0437ea4
<4>ds: 0018 es: 0018 ss: 0018
<4>Process swapper (pid: 0, stackpage=c0437000)
<4>Stack: 0000002e c40d8260 c40d8338 c40d8260 c0437ee4
c0264e8b c40d8260 00000000
<4> 00000001 c73cd944 00000009 c0437f10 c887250f
c40d8260 c40d8338 c0436000
<4> c0437f08 c02650a6 c40d8260 c037edb8 00000094
00000000 c0437f08 c40d8260
<4>Call Trace: [<c0264e8b>] [<c887250f>] [<c02650a6>]
[<c01296a9>] [<c0264fc0>]
<4> [<c01297ac>] [<c0124ebb>] [<c0109ae2>]
[<c01054f0>] [<c0105530>] [<c01055c2>]
<4> [<c0105000>]
<4>
<4>Code: 0f b6 4b 49 f6 c1 82 74 0c 31 d2 89 96 78 01 0000 0f b6 4b
STACK TRACE FOR TASK: 0xc0436000 (swapper)
0 tcp_enter_loss+202 [0xc025b7aa]
1 tcp_retransmit_timer+566 [0xc0264e86]
2 tcp_write_timer+225 [0xc02650a1]
3 timer_bh+710 [0xc01296a6]
4 timer_softirq+39 [0xc01297a7]
5 do_softirq+249 [0xc0124eb9]
6 do_IRQ+525 [0xc0109add]
7 do_IRQ+525 [0xc0109add]
Signature · 6 lines
George Anzinger [email protected] High-res-timers: http://sourceforge.net/projects/high-res-timers/ Real time sched: http://sourceforge.net/projects/rtsched/ Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: Network oops
No mention of what kernel version, what patches applied, etc. so we cannot help you.
Re: Network oops
In reply to a message not in the archiveAlso we need to know what network driver, if preemption is being used (being from mvista I assume you are using preemption, and if so make sure you have the preemption networking patches applied).
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: Network oops
Quoted 3 lines · David S. Miller
No mention of what kernel version, what patches applied, etc. so we cannot help you.
Sorry bout that. It is a 2.4.17 kernel and the test is to verify that we have the preempt code right. The question at hand is if this is a likely preempt problem or just a pure overload. The stress on the network is rather high.
I would expect that the network code would recover from this sort of thing, so we are looking for a preempt issue at the moment. Still, it could just be the way things work in the 2.4.17 kernel so I thought I would ask.
Signature · 6 lines
George Anzinger [email protected] High-res-timers: http://sourceforge.net/projects/high-res-timers/ Real time sched: http://sourceforge.net/projects/rtsched/ Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: Network oops
From: george anzinger <[email protected]>
Date: Sun, 09 Jun 2002 21:48:15 -0700
I would expect that the network code would recover from this sort of thing, so we are looking for a preempt issue at the moment. Still, it could just be the way things work in the 2.4.17 kernel so I thought I would ask.
Even though 2.4.17 is pretty old I still think it's a preempt problem.
Re: Network oops
In reply to a message not in the archiveWe need help from someone who knows the network code. I have tried to give all the relevant information below. The machine that last failed can still be examined with kgdb to answer any further questions.
We are working with a 2.4.17 kernel with all the latest preempt patches as well as the high-res-timres patch (which, by the way has the proposed TIMER_BH conversion to softirq code). Other patches are applied but, when removed, do not affect the below described behavior. The system is an SMP (2) processor: <4>CPU0: Intel(R) Pentium(R) III CPU family 1266MHz stepping 01
A break point placed in deliver_to_old_ones() is never hit.
Failure rate under heavy network stress (4 machines each measuring network performance with all 4 machines in the test) occurs very seldom. The last failure took 32 hours, the prior one 22 hours.
Failure appears to occur because a call is made to a bogus address that is pulled from an skb. Here is a back trace of the latest failure:
Program received signal SIGEMT, Emulation trap.0x00b24d18 in ?? () at af_packet.c:1891
(gdb) bt
#0 0x00b24d18 in ?? () at af_packet.c:1891 #1 0xc0267751 in tcp_v4_destroy_sock (sk=0xc7164260)
at /usr/src/linux-2.4.17-CLT/include/net/tcp.h:1673
#2 0xc02566f1 in tcp_destroy_sock (sk=0xc7164260) at tcp.c:1800
#3 0xc025731a in tcp_close (sk=0xc7164260, timeout=0) at tcp.c:1971
#4 0xc0274f67 in inet_release (sock=0xc2c94160) at af_inet.c:465
#5 0xc02304a2 in sock_release (sock=0xc2c94160) at socket.c:489
#6 0xc0230a50 in sock_close (inode=0xc2c94040, filp=0xc51297e0)
at socket.c:724
#7 0xc014d833 in fput (file=0xc51297e0) at file_table.c:113
#8 0xc014bfe3 in filp_close (filp=0xc51297e0, id=0xc2eb6d60) at open.c:838
#9 0xc014c0b2 in sys_close (fd=4) at open.c:862#10 0xc010782b in system_call () at af_packet.c:1891 #11 0x40043507 in ?? () at af_packet.c:1891 (
kgdb caught this in:0xc0119b3d is in do_page_fault (fault.c:329).
324 * terminate things with extreme prejudice.
325 */
326 #ifdef CONFIG_KGDB
327 if (!user_mode(regs)){
328 kgdb_handle_exception(14,SIGBUS, error_code, regs);
329 return;
330 }
331 #endif
332
333 bust_spinlocks(1);
(
In both failures we found this in the log buffer just prior to the failure:
<5>\n<4>KERNEL: assertion ((int)tcp_packets_in_flight(tp) >= 0) failed at tcp_input.c(956):tcp_sacktag_write_queue\n
On the assumption that preemption is the root cause of this problem we have instrumented the preemption code to keep track of the last 100 preemptions. What follows is an edited log of these preemptions. Each entry consists of two words of time (sec, usec) followed by the
address (hex and symbolic) and the pid of the process. Preemptions clearly unrelated to the network code have been removed. Most of these were in idle or sched_yield.
0xc0368e60 <preempt_log>: 0x3d101941 0xc9868 0xc025f4bb <tcp_transmit_skb+747> 0x15aa 0xc0368e80 <preempt_log+32>: 0x3d101941 0xec247 0xc025d821 <tcp_copy_to_iovec+129> 0x15aa 0xc0368ea0 <preempt_log+64>: 0x3d101942 0xc1f6 0xc0167ee0 <update_atime> 0x15ae 0xc0368ec0 <preempt_log+96>: 0x3d101942 0x24440 0xc025d821 <tcp_copy_to_iovec+129> 0x15aa 0xc0368ee0 <preempt_log+128>: 0x3d101942 0x26eae 0xc025d821 <tcp_copy_to_iovec+129> 0x15a9 0xc0368f00 <preempt_log+160>: 0x3d101942 0x2a4b4 0xc025d821 <tcp_copy_to_iovec+129> 0x15aa 0xc0368f20 <preempt_log+192>: 0x3d101942 0x2c74a 0xc02558b1 <tcp_prequeue_process+305> 0x15aa 0xc0368f40 <preempt_log+224>: 0x3d101942 0x2d5b6 0xc0255741 <tcp_data_wait+705> 0x15aa 0xc0368f60 <preempt_log+256>: 0x3d101942 0x2e634 0xc02558b1 <tcp_prequeue_process+305> 0x15a9 0xc0368f80 <preempt_log+288>: 0x3d101942 0x2fcaa 0xc024db81 <ip_output+401> 0x15aa 0xc0368fc0 <preempt_log+352>: 0x3d101942 0x30c4f 0xc02558b1 <tcp_prequeue_process+305> 0x15a6 0xc0368fe0 <preempt_log+384>: 0x3d101942 0x33603 0xc02558b1 <tcp_prequeue_process+305> 0x15a6 0xc0369000 <preempt_log+416>: 0x3d101942 0xc5137 0xc011d1ec <remove_wait_queue+156> 0x15a8 0xc0369060 <preempt_log+512>: 0x3d101944 0x6f2e4 0xc025561b <tcp_data_wait+411> 0x15a8 0xc0369120 <preempt_log+704>: 0x3d101948 0xae22f 0xc024dc28 <ip_queue_xmit+24> 0x15b2 0xc0369140 <preempt_log+736>: 0x3d101949 0xbe01 0xc02558b1 <tcp_prequeue_process+305> 0x15b2 0xc03691a0 <preempt_log+832>: 0x3d10194b 0xbd567 0xc025d821 <tcp_copy_to_iovec+129> 0x15b3 0xc0369250 <preempt_log+1008>: 0x3d10194b 0xbde39 0xc01054df <default_idle+47> 0x0
Each of these is pinned to source below:
(gdb)
(gdb)l* l* tcp_transmit_skb+7470xc01f4a83 is in tcp_transmit_skb (/usr/src/cvs/hhl-kernel-cambell/linux_test/include/net/tcp.h:1422).
1417 TCPOLEN_TIMESTAMP);
1418 *ptr++ = htonl(tstamp);
1419 *ptr++ = htonl(tp->ts_recent);
1420 }
1421 if (tp->eff_sacks) {
1422 struct tcp_sack_block *sp = tp->dsack ? tp->duplicate_sack : tp->selective_acks;
1423 int this_sack;
1424
1425 *ptr++ = __constant_htonl((TCPOPT_NOP << 24) |
1426 (TCPOPT_NOP << 16) |
(gdb)l * tcp_copy_to_iovec+1290xc01f2eed is in tcp_copy_to_iovec (tcp_input.c:3157).
3152 int chunk = skb->len - hlen;
3153 int err;
3154
3155 local_bh_enable();
3156 if (skb->ip_summed==CHECKSUM_UNNECESSARY)
3157 err = skb_copy_datagram_iovec(skb, hlen, tp->ucopy.iov, chunk);
3158 else
3159 err = skb_copy_and_csum_datagram_iovec(skb, hlen, tp->ucopy.iov);
3160
3161 if (!err) {
(gdb) l *tcp_prequeue_process+3050xc01ebba5 is in tcp_recvmsg (tcp.c:1400).
1395 int err;
1396 int target; /* Read at least this many bytes */
1397 long timeo;
1398 struct task_struct *user_recv = NULL;
1399
1400 lock_sock(sk);
1401
1402 TCP_CHECK_TIMER(sk);
1403
1404 err = -ENOTCONN;
(gdb) l* tcp_data_wait+7050xc01ebb49 is in tcp_prequeue_process (tcp.c:1376).
1371 while ((skb = __skb_dequeue(&tp->ucopy.prequeue)) != NULL)
1372 sk->backlog_rcv(sk, skb);
1373 local_bh_enable();
1374
1375 /* Clear memory counter. */
1376 tp->ucopy.memory = 0;
1377 }
1378
1379 /*
1380 * This routine copies from a sock struct into the user buffer.
(gdb) l *tcp_data_wait+4110xc01eba23 is in tcp_data_wait (tcp.c:1354).
1349 release_sock(sk);
1350
1351 if (skb_queue_empty(&sk->receive_queue))
1352 timeo = schedule_timeout(timeo);
1353
1354 lock_sock(sk);
1355 clear_bit(SOCK_ASYNC_WAITDATA, &sk->socket->flags);
1356
1357 remove_wait_queue(sk->sleep, &wait);
1358 __set_current_state(TASK_RUNNING);
(gdb) (gdb) l *ip_queue_xmit+240xc024dc28 is in ip_queue_xmit (ip_output.c:351).
346 struct iphdr *iph;
347
348 /* Skip all of this if the packet is already routed,
349 * f.e. by something like SCTP.
350 */
351 rt = (struct rtable *) skb->dst;
352 if (rt != NULL)
353 goto packet_routed;
354
355 /* Make sure we can route this packet. */
(gdb)
(gdb) l * ip_output+4010xc01e55d9 is in ip_queue_xmit (ip_output.c:344).
339 }
340
341 int ip_queue_xmit(struct sk_buff *skb)
342 {
343 struct sock *sk = skb->sk;
344 struct ip_options *opt = sk->protinfo.af_inet.opt;
345 struct rtable *rt;
346 struct iphdr *iph;
347
348 /* Skip all of this if the packet is already routed,
Any help would be greatly appreciated. We also can probe the system to answer any further questions. As said above, we are assuming this is related to preemption, however, that assumption may be bad.
Thanks
Signature · 4 lines
George Anzinger [email protected] High-res-timers: http://sourceforge.net/projects/high-res-timers/ Real time sched: http://sourceforge.net/projects/rtsched/ Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: Network oops
I don't understand, you've completely turned the preemption model of the kernel upside down and you want _US_ to debug this for you?
This is a lot of work, work I personally don't have time for. It requires a full audit of the networking in the new preemption environment.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: Network oops
Quoted 3 lines · David S. Miller
I don't understand, you've completely turned the preemption model of the kernel upside down and you want _US_ to debug this for you?
You could look at it this way :)
On the other hand, I prefer to think of the linux community as a group of folks working on making one of the best OSs on the planet even better. Its adoption by more and more companies attests to its place in the world. We who work on it, IMHO, should cooperate with each other in our efforts to make the system even better. Sure we will disagree about some of the things others are doing and thus the need for some central control of the system. For this we are indebted to Linus. And we may not have time to spend working on the system when others would most like us to, but that is the nature of life and we should respect others in this regard.
So while I would very much like your help, I understand that you may be busy with other things at this time and not have the time. Still, I must ask, just as others ask of me.
Quoted 2 lines
This is a lot of work, work I personally don't have time for.
I can understand and accept that.
Quoted 2 lines
It requires a full audit of the networking in the new preemption environment.
Any pointers on what to look for would be welcome.
Signature · 4 lines
George Anzinger [email protected] High-res-timers: http://sourceforge.net/projects/high-res-timers/ Real time sched: http://sourceforge.net/projects/rtsched/ Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: Network oops
From: george anzinger <[email protected]>
Date: Fri, 21 Jun 2002 07:16:24 -0700
[ BTW please start using newlines in your emails, instead of 1,000
character monstrosities lacking them. Thanks ]
On the other hand, I prefer to think of the linux community as a group of folks working on making one of the best OSs on the planet even better.
Right, and I don't think CONFIG_PREEMPT makes the planet better. In fact, now that you mention it, I think CONFIG_PREEMPT is a pile of crap.
Quoted 2 lines
It requires a full audit of the networking in the new preemption environment.
Any pointers on what to look for would be welcome.
And you go right back to asking me to do the work for you. What is wrong with you?
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: Network oops
Quoted 6 lines · David S. Miller
From: george anzinger <[email protected]>
Date: Fri, 21 Jun 2002 07:16:24 -0700[ BTW please start using newlines in your emails, instead of 1,000
character monstrosities lacking them. Thanks ]
Sure. I am trying to find something that works with patches as well as text. Still looking I guess.
Quoted 8 lines
On the other hand, I prefer to think of the linux community as a group of folks working on making one of the best OSs on the planet even better.
Right, and I don't think
CONFIG_PREEMPTmakes the planet better. In fact, now that you mention it, I thinkCONFIG_PREEMPTis a pile of crap.
Can we agree to disagree :)
Quoted 8 lines
Quoted 3 lines
It requires a full audit of the networking in the new preemption environment.
Any pointers on what to look for would be welcome.
And you go right back to asking me to do the work for you. What is wrong with you?
I said they would be welcome, not that you have to reply...
Signature · 6 lines
George Anzinger [email protected] High-res-timers: http://sourceforge.net/projects/high-res-timers/ Real time sched: http://sourceforge.net/projects/rtsched/ Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml
Re: Network oops
In reply to a message not in the archiveQuoted 4 lines
Quoted 2 lines
It requires a full audit of the networking in the new preemption environment.
Any pointers on what to look for would be welcome.
I would look at the driver, especially races in its skb handling.
-Andi
A message that is not in the archive (sent off-list or not yet received): 20020622025551.A1919@averell
Re: Network oops
We finally found the problem. I want to thank the folks who gave any thought and/or feedback to us, particularly Alexey and David.
For what its worth we tracked it down to the kernel memory allocation routines allocating the same buffer to two different network requesters. This in turn was caused by a basic flaw in the preemption code that preempted on spin_unlock, REGARDLESS OF THE STATE OF THE INTERRUPT SYSTEM.
Signature · 6 lines
George Anzinger [email protected] High-res-timers: http://sourceforge.net/projects/high-res-timers/ Real time sched: http://sourceforge.net/projects/rtsched/ Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: Network oops
Quoted 4 lines · David S. Miller
Also we need to know what network driver, if preemption is being used (being from mvista I assume you are using preemption, and if so make sure you have the preemption networking patches applied).
Uh, are you saying there is a set of network patches? If so, where might they be found? And yes, we are testing preemption, trying to wring out the "last" bug :) (Could you be referring to the patches we are developing?)
The driver (from the system log):
<4>eepro100.c:v1.09j-t 9/29/99 Donald Becker http://cesdis.gsfc.nasa.gov/linux/drivers/eepro100.html <4>eepro100.c: $Revision: 1.36 $ 2000/11/17 Modified by Andrey V. Savochkin <[email protected]> and others <6>eth0: OEM i82557/i82558 10/100 Ethernet, 00:03:47:BD:60:86, IRQ 21. <6> Board assembly fab600-000, Physical connectors present: RJ45 <6> Primary interface chip i82555 PHY #1. <6> General self-test: passed. <6> Serial sub-system self-test: passed. <6> Internal registers self-test: passed. <6> ROM checksum self-test: passed (0xb874c1d3). <6>eth1: OEM i82557/i82558 10/100 Ethernet, 00:03:47:BD:60:87, IRQ 20. <6> Board assembly fab600-000, Physical connectors present: RJ45 <6> Primary interface chip i82555 PHY #1. <6> General self-test: passed. <6> Serial sub-system self-test: passed. <6> Internal registers self-test: passed. <6> ROM checksum self-test: passed (0xb874c1d3).
Looks like two NICs.
Signature · 6 lines
George Anzinger [email protected] High-res-timers: http://sourceforge.net/projects/high-res-timers/ Real time sched: http://sourceforge.net/projects/rtsched/ Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: Network oops
From: george anzinger <[email protected]>
Date: Mon, 10 Jun 2002 01:11:01 -0700
Quoted 4 lines · David S. Miller
Also we need to know what network driver, if preemption is being used (being from mvista I assume you are using preemption, and if so make sure you have the preemption networking patches applied).
Uh, are you saying there is a set of network patches? If so, where might they be found? And yes, we are testing preemption, trying to wring out the "last" bug :) (Could you be referring to the patches we are developing?)
An mvista person recently put "make net preempt safe" patches into 2.5.x Basically it amounted to putting a few preempt_disable sections into net/core/skbuff.c
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: Network oops
Quoted 10 lines · David S. Miller
From: george anzinger <[email protected]>
Date: Mon, 10 Jun 2002 01:11:01 -0700Quoted 4 lines · David S. Miller
Also we need to know what network driver, if preemption is being used (being from mvista I assume you are using preemption, and if so make sure you have the preemption networking patches applied).
Quoted 8 lines
Uh, are you saying there is a set of network patches? If so, where might they be found? And yes, we are testing preemption, trying to wring out the "last" bug :) (Could you be referring to the patches we are developing?)
An mvista person recently put "make net preempt safe" patches into 2.5.x Basically it amounted to putting a few
preempt_disablesections intonet/core/skbuff.c
Ah, yes. Those are the ones I have found sofar. Robert Love would be the submitter. It would apear there is at least one more hole :( Thus the query.
Signature · 6 lines
George Anzinger [email protected] High-res-timers: http://sourceforge.net/projects/high-res-timers/ Real time sched: http://sourceforge.net/projects/rtsched/ Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml