2.6.8-rc4-bk1 problem: unregister_netdevice: waiting for ppp0 to become free. Usage count = 1

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi!

With 2.6.8-rc4-bk1 I get "Aug 12 17:33:10 puma kernel:
unregister_netdevice: waiting for ppp0 to become free. Usage count = 1" in the logs after pppd exit.

Also, the box won't reboot and print that message forever in the console. sysrq-U && sysrq-R did it :-)

The last version I tried was 2.6.8-rc2-bk11 and, wrt this prob, is running fine. So, the problem is in that window and the changelog for rc4 mentions something about ppp:
http://kernel.org/pub/linux/kernel/v2.6/testing/ChangeLog-2.6.8-rc4

If someone requires more information or tests feel free to ask!

Regards,
Nuno Silva

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBG8KDOPig54MP17wRArTNAKCkY2p0NqKYesxDAUvA7JraQO607QCgoDwo 4v/jnMy2G4jRL9AwALjUzLs=
=f0cy
-----END PGP SIGNATURE-----

I saw this too, with 2.6.7-rc3-mm1.
I have discovered that it happens because of idle TCP socket holds a reference to a network device.
After killing associated process, device was freed immediately.

~
:wq
With best regards,
Vladimir Savkin.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Vladimir B. Savkin wrote:

| On Thu, Aug 12, 2004 at 08:18:28PM +0100, Nuno Silva wrote:
|
|>Hi!
|>
|>With 2.6.8-rc4-bk1 I get "Aug 12 17:33:10 puma kernel:
|>unregister_netdevice: waiting for ppp0 to become free. Usage count = 1"
|>in the logs after pppd exit.
|>
|>Also, the box won't reboot and print that message forever in the
|>console. sysrq-U && sysrq-R did it :-)
|>
|>The last version I tried was 2.6.8-rc2-bk11 and, wrt this prob, is
|>running fine. So, the problem is in that window and the changelog for
|>rc4 mentions something about ppp:
|>http://kernel.org/pub/linux/kernel/v2.6/testing/ChangeLog-2.6.8-rc4
|>
|>If someone requires more information or tests feel free to ask!
|
|
| I saw this too, with 2.6.7-rc3-mm1.
| I have discovered that it happens because of idle TCP socket
| holds a reference to a network device.
| After killing associated process, device was freed immediately.
|

I waited for 5 mins before sysrq-U && sysrq-R.
Anyway, if I 'killall pppd' and then issue 'ifconfig -a' the ifconfig command will hang.

This didn't happen with 2.6.8-rc2-bk11 (the one I'm running now), so something changed... I'm I the only one with ppp/pppd/pppoe who tried 2.6.8-rc4-bk1? :-) Any success reports?

Regards,
Nuno Silva

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBHSYlOPig54MP17wRAs1NAJ0dMRd7tqnpRk/tnxZj7xSeoUL3UgCgrk50 tnzeT6sy3CauOI5WK7HF16o=
=mkWf
-----END PGP SIGNATURE-----

OK, I just tested again and the problem persists. However this time I had some spare moments so I rebooted a few times to isolate the problem.

The problem is in the QoS code. If I start ppp whithout the /etc/ppp/ip-up.d/wshaper script everything is fine. If I try the wshaper.htb it's also fine. So, I'd say that the problem is in the CBQ section. This time I waited for 1 hour and got hundreds of "unregister_netdevice: waiting for ppp0 to become free. Usage count = 1" in the console/syslog. pppd eats 99% CPU. ifconfig freezes. Even reboot isn't possible without sysrq's help.

This problem was introduced between 2.6.8-rc2-bk11 and 2.6.8-rc4-bk1 and always happens. Right now I'm testing with 2.6.8.1 with a patch from Mr. Miller -- "cacheline-align qdisc data in qdisc_create()" (attached).

If someone needs more details feel free to ask!

Regards,
Nuno Silva

# This is a BitKeeper generated diff -Nru style patch. #
# ChangeSet
# 2004/08/15 19:33:16-07:00 [email protected]
# [PKT_SCHED]: cacheline-align qdisc data in qdisc_create()
#
# Signed-off-by: Patrick McHardy <[email protected]>
# Signed-off-by: David S. Miller <[email protected]>
#

# net/sched/sch_api.c
#   2004/08/15 19:32:59-07:00 [email protected] +13 -8
#   [PKT_SCHED]: cacheline-align qdisc data in qdisc_create()
#   
#   Signed-off-by: Patrick McHardy <[email protected]>
#   Signed-off-by: David S. Miller <[email protected]>
# 
diff -Nru a/net/sched/sch_api.c b/net/sched/sch_api.c
--- a/net/sched/sch_api.c	2004-08-19 07:33:10 -07:00
+++ b/net/sched/sch_api.c	2004-08-19 07:33:10 -07:00
@@ -389,7 +389,8 @@
 {
 	int err;
 	struct rtattr *kind = tca[TCA_KIND-1];
-	struct Qdisc *sch = NULL;
+	void *p = NULL;
+	struct Qdisc *sch;
 	struct Qdisc_ops *ops;
 	int size;

@@ -407,12 +408,18 @@
if (ops == NULL)
goto err_out;

- size = sizeof(*sch) + ops->priv_size;
+ /* ensure that the Qdisc and the private data are 32-byte aligned */
+ size = ((sizeof(*sch) + QDISC_ALIGN_CONST) & ~QDISC_ALIGN_CONST);
+ size += ops->priv_size + QDISC_ALIGN_CONST;

- sch = kmalloc(size, GFP_KERNEL);
+ p = kmalloc(size, GFP_KERNEL);
err = -ENOBUFS;
- if (!sch)
+ if (!p)
goto err_out;
+ memset(p, 0, size);
+ sch = (struct Qdisc *)(((unsigned long)p + QDISC_ALIGN_CONST)
+ & ~QDISC_ALIGN_CONST);
+ sch->padded = (char *)sch - (char *)p;

/* Grrr... Resolve race condition with module unload */

@@ -420,8 +427,6 @@
if (ops != qdisc_lookup_ops(kind))
goto err_out;

- memset(sch, 0, size);
-
INIT_LIST_HEAD(&sch->list);
skb_queue_head_init(&sch->q);

@@ -470,8 +475,8 @@

err_out:
*errp = err;
- if (sch)
- kfree(sch);
+ if (p)
+ kfree(p);
return NULL;
}

Does that patch fix the problem?

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

David S. Miller wrote:

| On Sun, 22 Aug 2004 03:13:55 +0100
| Nuno Silva <[email protected]> wrote:
|
|
|>This problem was introduced between 2.6.8-rc2-bk11 and 2.6.8-rc4-bk1 and
|>always happens. Right now I'm testing with 2.6.8.1 with a patch from Mr.
|>Miller -- "cacheline-align qdisc data in qdisc_create()" (attached).
|
|
| Does that patch fix the problem?
|

Hi!

No, but that patch is necessary for me to use QoS at all, or the kernel oopses.

I was refering to that patch for you to know that's not the same problem. Sorry for not being 100% clear.

If you want to freeze your networking :-) you can do this:

remove pppd (or adjust the ppp0 interface name later)

# killall pppd

# sleep 10

make a fony connection to 1.1.1.2 using /dev/null #pppd /dev/null connect /bin/true persist demand 1.1.1.1:1.1.1.2

# sleep 1

add a CBQ root to ppp0 (or ppp1, if you didn't killall and have another running)

# tc qdisc add dev ppp0 root handle 1: cbq avpkt 1000 bandwidth 10mbit

ping the fony remote:

# ping 1.1.1.2

kill pppd
#killall pppd

Now check dmesg, you should get "unregister_netdevice: waiting for ppp0 to become free. Usage count = 1"

Another remark: In single user mode pppd died and system load is 0. In initlevel 3, pppd eats all my CPU... Don't know why.

Right now I'm testing 2.6.8.1-ck3 (because of that can't-burn-cdrom thing and other stuff) so the attached /proc/config.gz is somewhat non-vanilla, but this happens with 2.6.8-rc4-bk1[*] and above.

[*] this version is "bad" and 2.6.8-rc2-bk11 is "good". Didn't test with rc3...

As you can see in the .config I'm using the OLD POLICER.

As always, please request more info if you need it! :-)

Regards,
Nuno Silva
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBKFL5OPig54MP17wRAsHeAJ4nZFSW7nat0/aahUokzriHIueO5wCgqPCM EogRAFF/jofq4kN609j6dFc=
=1HPc
-----END PGP SIGNATURE-----

OK, this appears to be due to the changeset titled

[PKT_SCHED]: Refcount qdisc->dev for __qdisc_destroy rcu-callback

It adds a reference to dev.

I don't see any code that cleans up that reference when the dev goes down. So someone needs to add that similar to the code in net/core/dst.c.

Patrick, could you please have a look at this?

Thanks,

Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

The reference is dropped in __qdisc_destroy. The problem lies in the CBQ qdisc, it doesn't destroy the root-class and leaks the inner qdisc. These two patches for 2.4 and 2.6 fix the problem.

Regards
Patrick

# This is a BitKeeper generated diff -Nru style patch. #
# ChangeSet
# 2004/08/22 14:37:58+02:00 [email protected]
# [PKT_SCHED]: Fix class leak in CBQ scheduler
#
# Signed-off-by: Patrick McHardy <[email protected]>
#

# net/sched/sch_cbq.c
#   2004/08/22 14:37:27+02:00 [email protected] +8 -6
#   [PKT_SCHED]: Fix class leak in CBQ scheduler
# 
diff -Nru a/net/sched/sch_cbq.c b/net/sched/sch_cbq.c
--- a/net/sched/sch_cbq.c	2004-08-22 14:38:18 +02:00
+++ b/net/sched/sch_cbq.c	2004-08-22 14:38:18 +02:00
@@ -1712,15 +1712,18 @@
 	}
 }

-static void cbq_destroy_class(struct cbq_class *cl)
+static void cbq_destroy_class(struct Qdisc *sch, struct cbq_class *cl)
{
+ struct cbq_sched_data *q = (struct cbq_sched_data *)sch->data;
+
cbq_destroy_filters(cl);
qdisc_destroy(cl->q);
qdisc_put_rtab(cl->R_tab);
#ifdef CONFIG_NET_ESTIMATOR
qdisc_kill_estimator(&cl->stats);
#endif
- kfree(cl);
+ if (cl != &q->link)
+ kfree(cl);
}

static void
@@ -1743,8 +1746,7 @@

for (cl = q->classes[h]; cl; cl = next) {
next = cl->next;
- if (cl != &q->link)
- cbq_destroy_class(cl);
+ cbq_destroy_class(sch, cl);
}
}

@@ -1766,7 +1768,7 @@
spin_unlock_bh(&sch->dev->queue_lock);
#endif

- cbq_destroy_class(cl);
+ cbq_destroy_class(sch, cl);
}
}

@@ -2000,7 +2002,7 @@
sch_tree_unlock(sch);

if (--cl->refcnt == 0)
- cbq_destroy_class(cl);
+ cbq_destroy_class(sch, cl);

return 0;
}

# This is a BitKeeper generated diff -Nru style patch. #
# ChangeSet
# 2004/08/22 14:30:32+02:00 [email protected]
# [PKT_SCHED]: Fix class leak in CBQ scheduler
#
# Signed-off-by: Patrick McHardy <[email protected]>
#

# net/sched/sch_cbq.c
#   2004/08/22 14:30:13+02:00 [email protected] +8 -6
#   [PKT_SCHED]: Fix class leak in CBQ scheduler
# 
diff -Nru a/net/sched/sch_cbq.c b/net/sched/sch_cbq.c
--- a/net/sched/sch_cbq.c	2004-08-22 14:33:59 +02:00
+++ b/net/sched/sch_cbq.c	2004-08-22 14:33:59 +02:00
@@ -1746,15 +1746,18 @@
 	}
 }

-static void cbq_destroy_class(struct cbq_class *cl)
+static void cbq_destroy_class(struct Qdisc *sch, struct cbq_class *cl)
{
+ struct cbq_sched_data *q = qdisc_priv(sch);
+
cbq_destroy_filters(cl);
qdisc_destroy(cl->q);
qdisc_put_rtab(cl->R_tab);
#ifdef CONFIG_NET_ESTIMATOR
qdisc_kill_estimator(&cl->stats);
#endif
- kfree(cl);
+ if (cl != &q->link)
+ kfree(cl);
}

static void
@@ -1777,8 +1780,7 @@

for (cl = q->classes[h]; cl; cl = next) {
next = cl->next;
- if (cl != &q->link)
- cbq_destroy_class(cl);
+ cbq_destroy_class(sch, cl);
}
}

@@ -1799,7 +1801,7 @@
spin_unlock_bh(&sch->dev->queue_lock);
#endif

- cbq_destroy_class(cl);
+ cbq_destroy_class(sch, cl);
}
}

@@ -2035,7 +2037,7 @@
sch_tree_unlock(sch);

if (--cl->refcnt == 0)
- cbq_destroy_class(cl);
+ cbq_destroy_class(sch, cl);

return 0;
}

Awesome, good detective work guys.

Patch applied, thanks.

From: David S. Miller <[email protected]>
Date: Sun, Aug 22, 2004 at 09:47:46PM -0700

I get the same error with 2.6.9-rc1-mm1, but now for my ipv6-over-ipv4 tunnel.

unregister_netdevice: waiting for xs6all to become free

etc. when trying to reboot or shutdown. Usage count = 1

I'm pretty sure I didn't see this in 2.6.8.1 nor in 2.6.8.1-mm1.

This is a plain ipv6-over-ipv4 tunnel like this:

/etc/network/interfaces:
auto xs6all
iface xs6all inet6 v4tunnel
endpoint xxx.xxx.xxx.xxx
up ip route add 2000::0/3 via xxx:xxx:xxx:xxx:xxx:xxx address xxx:xxx:xxx:xxx:xxx:xxx
netmask 64
up ip tunnel change xs6all ttl 64

I'm running Debian Unstable with all the latest updates as of today.

Thanks for any hints,
Jurriaan

"At least she's on our side," said Lindholm.
Corbie looked at him. "Investigators aren't on anybody's side."
	Simon R Green - Hellworld
Debian (Unstable) GNU/Linux 2.6.9-rc1 2x6078 bogomips load 0.46

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Patrick McHardy wrote:

| Herbert Xu wrote:
|
|> Nuno Silva <[email protected]> wrote:
|>
|>
|>> The problem is in the QoS code. If I start ppp whithout the
|>
|>
|> OK, this appears to be due to the changeset titled
|>
|> [PKT_SCHED]: Refcount qdisc->dev for __qdisc_destroy rcu-callback
|>
|> It adds a reference to dev.
|>
|> I don't see any code that cleans up that reference when the dev goes
|> down.  So someone needs to add that similar to the code in
|> net/core/dst.c.
|>
|> Patrick, could you please have a look at this?
|>
|>
| The reference is dropped in __qdisc_destroy. The problem lies in the CBQ
| qdisc, it doesn't destroy the root-class and leaks the inner qdisc. These
| two patches for 2.4 and 2.6 fix the problem.

Hi!

Just to give some feedback: IT WORKS! Thanks!

Didn't try with 2.4, but it works very well with 2.6.8.1. Thanks again,
Nuno Silva

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBKi/FOPig54MP17wRAiYBAJ41ZGzauhY6dDVtylWkLSD3V+vx9QCgteNF 21sEmv0wqP+9hdnXEc4DNBE=
=ByPY
-----END PGP SIGNATURE-----