Kernel memory leak with PPPoE
I'm not sure if this is the right list or perhaps linux-kernel. But I figure people here will at least be interested even if it's not strictly in your bailiwick.
The other day my ISP went down for a few days (They hadn't paid their bills to the ILEC, doh). I discovered that there's a kernel memory leak in the pppoe module somewhere.
Since there was no config option I have my pppd patched to keep retrying PPPoE forever rather than giving up after a few iterations. (Why do people write code with hard coded limitations like that anyways? argh!).
After a few hours of PPPoE interface creation and teardown on PAP authentication failures my router box (a poor 486 with 24M of memory) started swapping. By the end of the first day it was swapping madly and eventually became totally unresponsive.
The same thing happened later after I rebooted it, except I stopped pppd completely. The problem did not go away. I'm pretty convinced it's a kernel memory leak by the fact that the machine kept swapping after pppd died. I've seen the same symptoms in the past too in the same circumstances (it's not a very reliable ISP).
This isn't totally up to date source code, but I'm pretty sure the PPPoE kernel module hasn't undergone much work in the past year or so has it?
This is Linux 2.4.21 with the ppp_generic, pppox, pppoe modules. pppd version 2.4.2b2 which I believe I actually built from CVS sources.
I know it's suboptimal to report a bug report without reproducing it with a clean unpatched, preferably recent, version. However this is my router box and I upgrading it to linux 2.6 would be a major project (and I believe 2.4 suits the 486 better). And with the unpatched pppd I would have to script restarting pppd repeatedly (and deal with the consequences of the interface disappearing and reappearing) since the pppoe module has a hard coded constant number of retries before it aborts.
Signature · 1 line
greg
Quoted 4 lines · Greg Stark
I'm not sure if this is the right list or perhaps linux-kernel. But I figure people here will at least be interested even if it's not strictly in your bailiwick.
Quoted 8 lines
The other day my ISP went down for a few days (They hadn't paid their bills to the ILEC, doh). I discovered that there's a kernel memory leak in the pppoe module somewhere.
Since there was no config option I have my pppd patched to keep retrying PPPoE forever rather than giving up after a few iterations. (Why do people write code with hard coded limitations like that anyways? argh!).
Can you try doing this another way? When pppd fails,let pppd die and start a new one.
My concern is that by trying to stick within one pppd process you may be leaking something in pppd (e.g. not closing file-descriptors). If the leak does not occur when you run new pppd processes, then I'd be inclined to think that it's a problem within pppd.
Quoted 11 lines
After a few hours of PPPoE interface creation and teardown on PAP authentication failures my router box (a poor 486 with 24M of memory) started swapping. By the end of the first day it was swapping madly and eventually became totally unresponsive.
The same thing happened later after I rebooted it, except I stopped pppd completely. The problem did not go away. I'm pretty convinced it's a kernel memory leak by the fact that the machine kept swapping after pppd died. I've seen the same symptoms in the past too in the same circumstances (it's not a very reliable ISP).
This is not the same thing as killing pppd every time you have a failure. By this point who knows what state your system is in.
Quoted 6 lines
This isn't totally up to date source code, but I'm pretty sure the PPPoE kernel module hasn't undergone much work in the past year or so has it?
This is Linux 2.4.21 with the
ppp_generic, pppox, pppoe modules. pppd version 2.4.2b2 which I believe I actually built from CVS sources.
Quoted 8 lines
I know it's suboptimal to report a bug report without reproducing it with a clean unpatched, preferably recent, version. However this is my router box and I upgrading it to linux 2.6 would be a major project (and I believe 2.4 suits the 486 better). And with the unpatched pppd I would have to script restarting pppd repeatedly (and deal with the consequences of the interface disappearing and reappearing) since the pppoe module has a hard coded constant number of retries before it aborts.
You can change that if you'd like. I just decided on something that seemed sane.
I've looked through the 2.4.21 code, and in pppoe code there are few
allocations to begin with. The ones that could be a leak I've verified
are not, or would only be a leak if the leak was actually in core net code (in which case it probably would have manifested itself in other ways and been dealt with already).
Signature · 1 line
Michal Ostrowski <[email protected]>
Quoted 2 lines · Michal Ostrowski
Can you try doing this another way? When pppd fails,let pppd die and start a new one.
I could do it for debugging. It's not acceptable for a real solution since then the interface would appear and disappear and clients would get networking errors for transient network reconnections.
Quoted 4 lines
My concern is that by trying to stick within one pppd process you may be leaking something in pppd (e.g. not closing file-descriptors). If the leak does not occur when you run new pppd processes, then I'd be inclined to think that it's a problem within pppd.
Well fds would normally get cleaned up when pppd exits.
Quoted 11 lines
Quoted 8 lines
I know it's suboptimal to report a bug report without reproducing it with a clean unpatched, preferably recent, version. However this is my router box and I upgrading it to linux 2.6 would be a major project (and I believe 2.4 suits the 486 better). And with the unpatched pppd I would have to script restarting pppd repeatedly (and deal with the consequences of the interface disappearing and reappearing) since the pppoe module has a hard coded constant number of retries before it aborts.
You can change that if you'd like. I just decided on something that seemed sane.
Well really the PAD* retries ought to be a configurable parameter, along with the timeouts, just like the ppp level retries. I tried adding parameters for these but never finished the job.
For users manually dialing out it ought to fail after a few retries. But for a router box that is supposed to maintain a 24x7 connection it really ought to do exponential backoff at first and then keep retrying about once a minute or so forever.
Quoted 5 lines
I've looked through the 2.4.21 code, and in pppoe code there are few
allocations to begin with. The ones that could be a leak I've verified
are not, or would only be a leak if the leak was actually in core net code (in which case it probably would have manifested itself in other ways and been dealt with already).
Hm. Perhaps it's elsewhere in the kernel but only triggered by the way pppox creates and destroys interfaces dynamically?
Signature · 1 line
greg