New: PPP over SSH tunnel triggers OOPS

Bugme-new Andrew Morton 2 participants 2 messages Jan 28, 2011 1 message not in the archive

Subsystems (2)

Networking Network drivers

lore

Share & tools

lore

Andrew Morton

author

2011-01-28 22:34 UTC #1 lore raw

In reply to a message not in the archive

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 95 lines · On Fri, 28 Jan 2011 21:58:49 GMT [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=27742
           Summary: PPP over SSH tunnel triggers OOPS
           Product: Networking
           Version: 2.5
    Kernel Version: 2.6.30 ?
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: Other
        AssignedTo: [email protected]
        ReportedBy: [email protected]
        Regression: Yes
Created an attachment (id=45412)
--> (https://bugzilla.kernel.org/attachment.cgi?id=45412)
Hand-copied OOPS from 2.6.37 kernel
When creating a VPN connection by using PPP tunneled over SSH, the kernel will OOPS when certain traffic patterns are encountered. (See attached OOPS)
I first created such a VPN connection using kernel 2.6.33, which is affected. Kernel 2.6.27 is not affected. I have not attempted to binary-search for the exact commit, but am merely guessing it is in kernel 2.6.30 (as a variety of ppp-related commits appear in the changelog there).
The VPN tunnel is established by invoking 'pppd' with the 'pty' parameter set to invoke "ssh remotehost.com pppd" which establishes a SSH tunnel over IPv4 to the remote host and then invokes the remote pppd to handle the other end of the point-to-point VPN.
Reproducing this bug is not easy. With the ppp-ssh-ppp tunnel open, I have tried triggering the OOPs by sending PINGs, rsync-ing files in both directions, opening interactive SSH connections. Nothing seems to trigger the OOPS except
one: running Mozilla Thunderbird on the remote end; it opens several IMAP connections over the tunnel simultaneously. Typically, the OOPS will occur within 1 or 2 seconds of invoking Thunderbird.
When the OOPS occurs, usually the console will be scrolling wildly with OOPS after OOPS, making copying impossible. It has taken me two months of repeated tries to get one OOPS that remained on-screen and could be copied. The kernel is in a hard-run state when the OOPS occurs; nothing gets logged to syslog, the keyboard is unresponsive (magic sysrq key does nothing).
skb_over_panic: text:c12a354f len:847 put:847 head:f57e8c00 data:f57e8c00 tail:0xf57e8f4f end:0xf57e8e80 dev:<NULL> kernel BUG at net/core/skbuff.c:127!
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/devices/virtual/net/ppp0/flags
Modules linked in:
Pid: 0, comm: swapper Not tainted 2.6.37 #1 0KH290/OptiPlex GX620
EIP: 0060:[<c1330110>] EFLAGS: 00010282 CPU: 0
EIP is at skb_put+0x82/0x84
EAX: 00000089 EBX: f57e8f4f ECX: c151579c EDX: 00000046
ESI: 00000000 EDI: c1530760 EBP: f67bb384 ESP: f6409d50
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process swapper (pid: 0, ti=f6408000 task=c15114a0 task.ti=c1502000)
Stack:
 c14d4590 c12a354f 0000034f 0000034f f57e8c00 f57e8c00 f57e8f4f f57e8e80
 c14d2509 f67bb380 f454db80 c12a354f 000005e0 00000244 f5f944c2 0000034f
 c1390a0a f67bb3d4 f4648380 f67bb394 f67bb3a4 00000202 f454db80 f67bb000
Call Trace:
 [<c12a354f>] ? ppp_xmit_process+0x45a/0x4e6
 [<c12a354f>] ? ppp_xmit_process+0x45a/0x4e6
 [<c1390a0a>] ? tcp_manip_pkt+0xad/0xcb
 [<c12a36d4>] ? ppp_start_xmit+0xf9/0x175
 [<c133a496>] ? dev_hard_start_xmit+0x2a4/0x5c3
 [<c1347cad>] ? sch_direct_xmit+0xb9/0x184
 [<c134c663>] ? nf_iterate+0x52/0x76
 [<c1362d56>] ? ip_finish_output+0x0/0x294
 [<c133a88e>] ? dev_queue_xmit+0xd9/0x3b0
 [<c1362d56>] ? ip_finish_output+0x0/0x294
 [<c1362f32>] ? ip_finish_output+0x1dc/0x294
 [<c1362d56>] ? ip_finish_output+0x0/0x294
 [<c1360d66>] ? ip_forward_finish+0x36/0x42
 [<c135f8a4>] ? ip_rcv_finish+0x42/0x323
 [<c13384ac>] ? __netif_receive_skb+0x225/0x299
 [<c1048b56>] ? getnstimeofday+0x42/0xe8
 [<c13386ab>] ? netif_receive_skb+0x41/0x64
 [<c1339356>] ? dev_gro_receive+0x146/0x1dd
 [<c133955e>] ? napi_gro_receive+0xa5/0xb3
 [<c129ae2f>] ? tg3_poll_wor+0x5df/0xaca
 [<c1007340>] ? nommu_sync_single_for_device+0x0/0x1
 [<c129b3e2>] ? tg3_poll+0x43/0x19a
 [<c133893b>] ? net_rx_action+0x6c/0xf4
 [<c1031eb5>] ? __do_softirq+0x77/0xf0
 [<c1031e3e>] ? __do_softirq+0x0/0xf0
 <IRQ>
 [<c1031fe6>] ? irq_exit+0x5d/0x5f

David Miller

maintainer

2011-01-28 22:55 UTC #2 lore raw

From: Andrew Morton <[email protected]>
Date: Fri, 28 Jan 2011 14:32:38 -0800

Quoted 2 lines

Quoted 2 lines
skb_over_panic: text:c12a354f len:847 put:847 head:f57e8c00 data:f57e8c00 tail:0xf57e8f4f end:0xf57e8e80 dev:<NULL> kernel BUG at net/core/skbuff.c:127!

...

Quoted 3 lines

Quoted 3 lines
Pid: 0, comm: swapper Not tainted 2.6.37 #1 0KH290/OptiPlex GX620
EIP: 0060:[<c1330110>] EFLAGS: 00010282 CPU: 0
EIP is at skb_put+0x82/0x84

...

Quoted 5 lines

Call Trace:
 [<c12a354f>] ? ppp_xmit_process+0x45a/0x4e6
 [<c12a354f>] ? ppp_xmit_process+0x45a/0x4e6
 [<c1390a0a>] ? tcp_manip_pkt+0xad/0xcb
 [<c12a36d4>] ? ppp_start_xmit+0xf9/0x175

I took a quick look at this, I can surmise that we have a packet we are trying to compress (that's the only way I see in the ppp_xmit_process() code paths that we can get an skb_put() call so large).

And we can see from the skb_over_panic message that we have an SKB which was allocated with 640 bytes of space, but we are trying to "put" 847 bytes into it which is too large and overflows.

Can you run with the following debugging patch and see what it prints out when this happens?

drivers/net/ppp_generic.c +19 -0

index 9f6d670..06c6ea7 100644
@@ -1093,6 +1093,15 @@ pad_compress_skb(struct ppp *ppp, struct sk_buff *skb)
 	if (len > 0 && (ppp->flags & SC_CCP_UP)) {
 		kfree_skb(skb);
 		skb = new_skb;
+#if 1
+		if (len > (skb->end - skb->tail)) {
+			printk(KERN_ERR "pad_compress_skb: Compression overflow ["
+			       "new_skb_size(%d) compressor_skb_size(%d) "
+			       "hard_header_len(%d) len(%d)]\n",
+			       new_skb_size, compressor_skb_size,
+			       ppp->dev->hard_header_len, len);
+		}
+#endif
 		skb_put(skb, len);
 		skb_pull(skb, 2);	/* pull off A/C bytes */
 	} else if (len == 0) {

@@ -1179,6 +1188,9 @@ ppp_send_frame(struct ppp *ppp, struct sk_buff *skb)
 			/* didn't compress */
 			kfree_skb(new_skb);
 		} else {
+#if 1
+			unsigned int orig_skb_len = skb->len;
+#endif
 			if (cp[0] & SL_TYPE_COMPRESSED_TCP) {
 				proto = PPP_VJC_COMP;
 				cp[0] &= ~SL_TYPE_COMPRESSED_TCP;

@@ -1188,6 +1200,13 @@ ppp_send_frame(struct ppp *ppp, struct sk_buff *skb)
 			}
 			kfree_skb(skb);
 			skb = new_skb;
+#if 1
+			if (len > (skb->end - skb->tail)) {
+				printk(KERN_ERR "slhc_compress_skb: Compression overflow ["
+				       "skb->len(%u) hard_header_len(%d) len(%d)]\n",
+				       orig_skb_len, ppp->dev->hard_header_len, len);
+			}
+#endif
 			cp = skb_put(skb, len + 2);
 			cp[0] = 0;
 			cp[1] = proto;