Skip to contentSkip to conversation

sungem hangs in atomic if netconsole enabled but no carrier

DISCUSSION Johannes Berg 4 participants 5 messages Dec 20, 2005

lore
Share & tools
lore
#0 lore raw

I've been debugging some issues and wondered why I got hangs in random places in the code. It turns out that the problem is that I still had netconsole enabled even though I have no network at the moment. So what I had was:
* sungem compiled in
* netconsole=.... as command line
* no network cable plugged in

sungem does recognise this situation and says that waiting for carrier timed out. However, later, when I printk() in with interrupts disabled, the system hangs after printing out a few lines to the console (I think it's more than one, not sure though, might be just a single one).

Turns out that if I remove the netconsole=... option to my kernel, all works fine and the system no longer hangs. Obviously not plugging in a network cable is pretty useless when netconsole is turned on, but I think it should not hang the system completely. So far I haven't been able to figure out where it actually hangs and don't even know how to do so -- I'm open for suggestions on how to find out why/where it hangs or even fixes.

johannes

#1 lore raw
Quoted 8 lines

Turns out that if I remove the netconsole=... option to my kernel, all works fine and the system no longer hangs. Obviously not plugging in a network cable is pretty useless when netconsole is turned on, but I think it should not hang the system completely. So far I haven't been able to figure out where it actually hangs and don't even know how to do so -- I'm open for suggestions on how to find out why/where it hangs or even fixes.

Hrm... I've heard various reports about problems with netconsole... I've never tried it myself so far though. One thing I remember to beware of is if sungem does a printk while holding its lock...

Ben.

#2 lore raw

From: Benjamin Herrenschmidt <[email protected]>
Date: Wed, 21 Dec 2005 08:18:40 +1100

Quoted 9 lines
Quoted 8 lines

Turns out that if I remove the netconsole=... option to my kernel, all works fine and the system no longer hangs. Obviously not plugging in a network cable is pretty useless when netconsole is turned on, but I think it should not hang the system completely. So far I haven't been able to figure out where it actually hangs and don't even know how to do so -- I'm open for suggestions on how to find out why/where it hangs or even fixes.

Quoted 3 lines

Hrm... I've heard various reports about problems with netconsole... I've never tried it myself so far though. One thing I remember to beware of is if sungem does a printk while holding its lock...

I bet that's it, doing a printk while already in the netconsole printk output path.

#3 lore raw
In reply to Johannes Berg

Johannes Berg <[email protected]> :
[...]

Quoted 4 lines

think it should not hang the system completely. So far I haven't been able to figure out where it actually hangs and don't even know how to do so -- I'm open for suggestions on how to find out why/where it hangs or even fixes.

See the thread "Netconsole violates dev->hard_start_xmit synch rules" started the 06/09/2005 on [email protected] for some interesting background.

(the innocent hero slowly fades into the swamps of netpolling...)

Still with us ?

Were you using sundance.c, you would probably bug on the first timeout:

[net/sched/sch_generic.c]
static void dev_watchdog(unsigned long arg)

{
        struct net_device *dev = (struct net_device *)arg;

        spin_lock(&dev->xmit_lock);
        ^^^^^^^^^
        if (dev->qdisc != &noop_qdisc) {
                if (netif_device_present(dev) &&
                    netif_running(dev) &&
                    netif_carrier_ok(dev)) {
                        if (netif_queue_stopped(dev) &&
                            (jiffies - dev->trans_start) > dev->watchdog_timeo) {
                                printk(KERN_INFO "NETDEV WATCHDOG: %s: transmit timed out\n", dev->name);
                                dev->tx_timeout(dev);
                                ^^^^^^^^^^^^^^^
[net/core/netpoll.c]
static void netpoll_send_skb(struct netpoll *np, struct sk_buff *skb)
{
        int status;
        struct netpoll_info *npinfo;

        if (!np || !np->dev || !netif_running(np->dev)) {
                __kfree_skb(skb);
                return;
        }

        npinfo = np->dev->npinfo;

        /* avoid recursion */
        if (npinfo->poll_owner == smp_processor_id() ||
            np->dev->xmit_lock_owner == smp_processor_id()) {
                if (np->drop)
                        np->drop(skb);
                else
                        __kfree_skb(skb);
                return;
        }

        do {
                npinfo->tries--;
                spin_lock(&np->dev->xmit_lock);
                ^^^^^^^^^

A quick glance shows no netif_carrier_{on/off} in the sundance driver. It would be a good candidate.

However you are using sungem.c and despite the fact that I should really have something for dinner *now*, you are protected by netif_carrier_off.

But (drums roll):

[drivers/net/sungem.c]

#define DEFAULT_MSG     (NETIF_MSG_DRV          | \
                         NETIF_MSG_PROBE        | \
                         NETIF_MSG_LINK)

Thus gem_link_timer() will periodically complain that the link is down.

So gem_start_xmit() is issued.

Repeat until the TX ring is full: netif_stop_queue() is called.

gem_link_timer() printks.

net/core/netpoll.c::netpoll_send_skb() notices that the queue is stopped and decides to try the usual NAPI poll(). A few function calls later, the driver ends in drivers/net/sungem.c::gem_poll() where it takes so many (irq-)locks that I do not even want to verify that it has a chance to play nice with the pending gem_link_timer().

--
Ueimor

#4 lore raw
Quoted 5 lines · Francois Romieu

net/core/netpoll.c::netpoll_send_skb() notices that the queue is stopped and decides to try the usual NAPI poll(). A few function calls later, the driver ends in drivers/net/sungem.c::gem_poll() where it takes so many (irq-)locks that I do not even want to verify that it has a chance to play nice with the pending gem_link_timer().

I'm not fan of the locking in sungem, I think I wrote a big fat comment about it and why it is like that for now, better ideas are welcome :)

Ben.