Spontaneous LCP ConfReq after connection made
G'day,
pppd tears down a connection and starts a new one, without the modem hanging up, in response to an LCP ConfReq that arrives out of the blue from the peer.
I'm thinking it's a peer bug, but I'm not sure, and I can't do anything about the peer, as it's a consumer service by an apparently disinterested service provider who refuses to resolve problems if "unsupported" software is used.
Can anyone suggest anything?
Is there somewhere in pppd source where I can easily disable this response, in case I can continue the existing connection?
The debug and dump output is here;
http://quozl.linux.org.au/mm-5100/2005-08-12/rtt-fail.log
(the SIGHUP after the restart is generated by my ip-down script)
The pppdump output is here;
http://quozl.linux.org.au/mm-5100/2005-08-12/rtt.pppdump
The service I'm using is a CDMA 1x RTT modem attached by USB;
http://quozl.linux.org.au/mm-5100/
pppd is from Debian GNU/Linux sarge, 2.4.3-20050321 Linux kernel 2.6.11.7 on i686.
Problem is reproducible; most easily if there is a PPTP session active on another interface and packets begin to be sent over the new interface as a result of a route change, but also during normal operation without any PPTP session active. Sometimes it will operate for minutes or hours, sometimes only seconds. The trouble always begins with an LCP ConfReq out of the blue, and generally in apparent response to some packet sent by my end.
Signature · 2 lines
James Cameron http://ftp.hp.com.au/sigs/jc/
Quoted 3 lines · James Cameron
pppd tears down a connection and starts a new one, without the modem hanging up, in response to an LCP ConfReq that arrives out of the blue from the peer.
That's exactly what the standards require pppd to do.
When an LCP Configure-Request arrives, the LCP state machine leaves Opened state. This causes all the other state machines to get lower-layer-down, and they all come down as well.
Quoted 6 lines
I'm thinking it's a peer bug, but I'm not sure, and I can't do anything about the peer, as it's a consumer service by an apparently disinterested service provider who refuses to resolve problems if "unsupported" software is used.
Can anyone suggest anything?
I'd strongly suggest finding a different service provider. Giving your money to a "service" provider that won't even accept a valid bug report doesn't strike me as a good plan.
Quoted 2 lines
Is there somewhere in pppd source where I can easily disable this response, in case I can continue the existing connection?
You'd probably have to modify the link_down() routine so that it doesn't do upper_layers_down.
Note that by doing so, you're essentially making pppd non-compliant with the standards. I'm also not really sure it would fix the problem. How do you really know that this is an out-of-the-blue packet, and *not* the peer attempting to renegotiate the link for some reason? If it's the latter, which I suspect is the case, then just ignoring the out-of-the-blue packet will cause the link to go dead.
In other words, you can do it, but I think it'd be wrong and self-defeating to do so.
Quoted 3 lines
The debug and dump output is here;
http://quozl.linux.org.au/mm-5100/2005-08-12/rtt-fail.log
(the SIGHUP after the restart is generated by my ip-down script)
In reading this log, it looks to me like the peer just wanted to renegotiate the link.
How do you know that's not what it wanted to do? Does this behavior happen with other PPP implementations? If not, then what's different with them?
Assuming that it's a peer bug (and the peer doesn't actually *want* to renegotiate the link), you might be able to find out what causes the peer to behave this way and avoid that, rather than breaking pppd's state machine.
Sometimes, bug-ridden peers (which this seems to be) are tickled by seemingly innocuous behaviors. Other implementations that don't have trouble with this problematic peer might ask for options you don't ask for or exhibit timing behaviors that you don't. Getting to the bottom of such issues is essentially debugging that remote peer from afar. It's not easy at all.
One other bit of concern is that the peer says "mru 1500." That's the default, and there's just never any reason to ask for the well-known default. That means the peer is probably an idiot.
Signature · 1 line
James Carlson 42.703N 71.076W <[email protected]>
Quoted 3 lines · James Carlson
I'd strongly suggest finding a different service provider. Giving your money to a "service" provider that won't even accept a valid bug report doesn't strike me as a good plan.
Unfortunately, there is no other service provider; all transmission towers within radio range are owned by the one provider, and no comparable service is available. DSL is not available, and copper pairs can only do about 24kbit/sec. It's a 24-month contract, I'm up to the ninth month, and the contract terms include a "supported software" clause. It was working reasonably well before now.
Quoted 6 lines
In reading this log, it looks to me like the peer just wanted to renegotiate the link.
How do you know that's not what it wanted to do? Does this behavior happen with other PPP implementations? If not, then what's different with them?
Apart from the IP address change, there doesn't seem to be any reason for the renegotiation. These renegotiations happen in the middle of the cell phone call; the cell call is not terminated. But the provider charges based on their PPP records, not their cell call records. As a result of the renegotiation, a new call record is placed in the service provider's billing system, costing me $USD 0.385 each renegotiation. I've had it renegotiate 30 times in a minute, before I added an if-down script to SIGHUP it. The service provider has reversed these charges, admitting a fault condition exists, but they haven't been able to progress the fault since Feb '05, so I thought to seek additional help.
You raise an excellent point with regard to other PPP implementations. Perhaps there is something negotiated or implied that isn't being done by Linux PPP. (Nor should it do so, of course).
I've not tried the PPP implementation that has been provided by the modem manufacturer, because it is Windows specific, I don't run Windows here, and as the modem is USB connected instead of serial port connected I don't know a way to catch the stream for analysis. But thanks for the idea, I'll see if I can get the manufacturer to provide information. (So far they have been silent since I told them I am using Linux).
Quoted 2 lines
Getting to the bottom of such issues is essentially debugging that remote peer from afar. It's not easy at all.
Agreed. So in addition to whinging I could try to find what innocuous behaviour Linux PPP is doing that apparently causes the peer to renegotiate. Have you any suggestions for options to try blindly in case they have an effect?
I've tried "asyncmap ffffffff" and "escape 00,01,02,03,04,05,06,07,08,09,0a,0b,0c,0d,0e,0f,10,11,12,13,14,15,16,17, 18,19,1a,1b,1c,1d,1e,1f,80,81,82,83,84,85,86,87,88,89,8a,8b,8c,8d,8e,8f,90,91,92 ,93,94,95,96,97,98,99,9a,9b,9c,9d,9e,9f" so far, with no apparent change.
Have you any suggestions for things to look for in the data sent to the peer that seems to trigger the event?
Signature · 2 lines
James Cameron http://ftp.hp.com.au/sigs/jc/
----- Original Message -----
From: "James Cameron" <[email protected]>
To: <[email protected]>
Sent: Saturday, August 13, 2005 1:32 AM
Subject: Re: Spontaneous LCP ConfReq after connection made
Quoted 5 lines
Quoted 3 lines · James Carlson
I'd strongly suggest finding a different service provider. Giving your money to a "service" provider that won't even accept a valid bug report doesn't strike me as a good plan.
[ snip ]
Quoted 6 lines
I've not tried the PPP implementation that has been provided by the modem manufacturer, because it is Windows specific, I don't run Windows here, and as the modem is USB connected instead of serial port connected I don't know a way to catch the stream for analysis. But thanks for the idea, I'll see if I can get the manufacturer to provide information. (So far they have been silent since I told them I am using Linux).
there is an open source tool to sniff usb from windows
http://sourceforge.net/projects/usbsnoop
and a sligthlty modified version available at
http://eciadsl.flashtux.org/download.php?lang=en
(search for sniffer)
Gilles
Quoted 6 lines · James Cameron
Unfortunately, there is no other service provider; all transmission towers within radio range are owned by the one provider, and no comparable service is available. DSL is not available, and copper pairs can only do about 24kbit/sec. It's a 24-month contract, I'm up to the ninth month, and the contract terms include a "supported software" clause. It was working reasonably well before now.
Ugh. Sounds like an expensive experience.
If it's not for a mobile station (assuming so, since you mentioned DSL), the one means you didn't suggest was cable. I take it that's not an option.
(There's also satellite, but I certainly wouldn't recommend that due to the latency issues. Though I suppose that any functioning connection is better than none.)
Quoted 3 lines
Apart from the IP address change, there doesn't seem to be any reason for the renegotiation. These renegotiations happen in the middle of the cell phone call; the cell call is not terminated. But the provider
There are a few things that could be causing it, even if there doesn't appear to be a cause.
In general, PPP will want to renegotiate if the lower layer appears to have gone down and back up. If the RF layer is telling their system that your telephone has disconnected and come back, it may well trigger a renegotiation on that glitch. Another possibility (since you mention cellular service) is that something about your location or the phone itself is causing it to hand off frequently between sites. (That's not _supposed_ to happen, but it's at least theoretically possible.)
And since it gets them extra cash to have this bug and most people won't notice, they probably aren't too highly inclined to fix it.
Here's another possibility: I once saw a bug in a particular embedded implementation where they apparently accidentally left a timer running after setting LCP to Opened state that caused renegotiation. It was triggered when the 'wrong' side sent the first LCP message -- if I recall correctly, they were expecting that the answerer (the "server") would send the first packet, and somehow had managed to code their implementation to depend on that. (Pretty hard to do, I think, when the RFC has the state machine right in it ...)
Obviously, of course, LCP is completely symmetric, and there's just no reason for such a bug (and it's hard for me to imagine how you could possibly get it that wrong), but bugs are funny that way.
Using the "silent" option might help, if the problem is similar.
Quoted 5 lines
charges based on their PPP records, not their cell call records. As a result of the renegotiation, a new call record is placed in the service provider's billing system, costing me $USD 0.385 each renegotiation. I've had it renegotiate 30 times in a minute, before I added an if-down script to SIGHUP it. The service provider has reversed these charges,
There's a fine line, I think, between deliberately foolish billing practices and just outright fraud.
Quoted 6 lines
I've not tried the PPP implementation that has been provided by the modem manufacturer, because it is Windows specific, I don't run Windows here, and as the modem is USB connected instead of serial port connected I don't know a way to catch the stream for analysis. But thanks for the idea, I'll see if I can get the manufacturer to provide information. (So far they have been silent since I told them I am using Linux).
Is there any chance you could borrow someone else's machine for a day to troubleshoot the problem? One of the unfortunately good things about Windows is that it's often pretty easily available.
Even if it's USB, there's a PPP-level "advanced" debug log buried in the Windows client, and that might provide enough clues to go on. Plus, in the good case, it fails just as miserably as Linux does, proving to them that they have a problem and owe you a fix, and that it's not your "unsupported" software that's at fault.
Quoted 7 lines
Quoted 2 lines
Getting to the bottom of such issues is essentially debugging that remote peer from afar. It's not easy at all.
Agreed. So in addition to whinging I could try to find what innocuous behaviour Linux PPP is doing that apparently causes the peer to renegotiate. Have you any suggestions for options to try blindly in case they have an effect?
As above, "silent" is one possibility.
Another would be to start refusing some of the options, such as "nopcomp" and "noaccomp".
Quoted 1 line
I've tried "asyncmap ffffffff" and "escape
Instead of ffffffff, I'd use "default-asyncmap". I doubt it'll help, though.
Quoted 2 lines
Have you any suggestions for things to look for in the data sent to the peer that seems to trigger the event?
Well, if varying the options doesn't help, and borrowing a Windows machine doesn't get to it, and you still really want to debug this, then the next step would probably be to start varying the timing. Fortunately, pppd's negotiation mechanisms are all in user space, so a few well-placed "sleep(1);" hacks might either avoid the problem or open the hole wide enough so that you can see it happen every time.
Do they support anything besides Windows? If they support either standalone routers or perhaps Macs you might be able to get something cheap on ebay to use as a gateway or NAT. A solution like that (if it's supported and actually works) might be cheaper than trying to fix whatever's wrong here ... don't forget that your time is worth something, too. ;-}
Signature · 1 line
James Carlson 42.703N 71.076W <[email protected]>
Quoted 3 lines · James Carlson
If it's not for a mobile station (assuming so, since you mentioned DSL), the one means you didn't suggest was cable. I take it that's not an option.
Yes. I'm six hours drive from the nearest city with cable. I'm using satellite too, but it has data limits (500Mb per month) whereas the CDMA service has none. They only have a time limit (50 hours per month).
Quoted 3 lines
Here's another possibility: I once saw a bug in a particular embedded implementation where they apparently accidentally left a timer running after setting LCP to Opened state that caused renegotiation.
I've excluded this based on the apparent random timing, but I will give "silent" a try.
One thing I've noticed with testing past few days ... the renegotiation is very probable if a GRE packet for PPTP is sent over the PPP link.
The peer provides a NAT service, and supports the use of PPTP. PPTP over NAT requires the peer implement stateful inspection and connection tracking.
Normally a PPTP tunnel runs inside an OpenVPN tunnel over the satellite service. When the CDMA service comes up, my if-up scripts change the route to the PPTP tunnel server so that packets for the active tunnel go via the link in question. Breaks the tunnel, of course.
The presumably buggy peer receives a GRE packet out of the blue that it cannot relate to any active connection. It might be crashing and restarting.
It occurs to me that this might be causing problems for other users. The service provider did say that other users had reported a similar problem. If I can reproduce it reliably, I'll let the service provider know.
Thanks for the suggestion to capture the Windows client negotiation; first I'll see if I can isolate the problem to orphan PPTP frames.
Signature · 2 lines
James Cameron http://ftp.hp.com.au/sigs/jc/
Providing a touch more closure to this thread; since blocking certain packets from being sent through the link, the symptom no longer occurs, so I think the peer is indeed broken.
The ip-up script now does this;
# eth1 satellite service is 192.168.x.x
iptables --insert OUTPUT 1 --source 192.168.0.0/255.255.0.0 \
--destination 0.0.0.0/0.0.0.0 --jump DROP \
--out-interface ${PPP_IFACE}# eth0 internal network is 10.0.x.x
iptables --insert OUTPUT 1 --source 10.0.0.0/255.255.0.0 \
--destination 0.0.0.0/0.0.0.0 --jump DROP \
--out-interface ${PPP_IFACE}# block any PPTP VPN traffic
iptables --insert OUTPUT 1 --protocol GRE --jump DROP \
--out-interface ${PPP_IFACE}Signature · 2 lines
James Cameron http://ftp.hp.com.au/sigs/jc/