Skip to contentSkip to conversation

netpoll trapped question

DISCUSSION Jeff Moyer 3 participants 11 messages Aug 31, 2004 – Sep 8, 2004

lore
Share & tools
lore

Jeff Moyer

author
#0 lore raw

Hi, Matt,

This part of the netpoll trapped logic seems suspect to me, from include/linux/netdevice.h:

static inline void netif_wake_queue(struct net_device *dev) {

#ifdef CONFIG_NETPOLL_TRAP
	if (netpoll_trap())
		return;
#endif
	if (test_and_clear_bit(__LINK_STATE_XOFF, &dev->state))
		__netif_schedule(dev);
}

static inline void netif_stop_queue(struct net_device *dev) {

#ifdef CONFIG_NETPOLL_TRAP
	if (netpoll_trap())
		return;
#endif
	set_bit(__LINK_STATE_XOFF, &dev->state);
}

This looks buggy. Network drivers are now not able to stop the queue when they run out of Tx descriptors. I think the __netif_schedule is okay to do in the context of netpoll, and certainly a set_bit is okay. Why are these hooks in place? I've tested alt-sysrq-t over netconsole and also netdump with these #ifdef's removed, and things work correctly. Compare this with alt-sysrq-t hanging the system with these tests in place. If I run netdump with this logic still in place, I get the following messages from the tg3 driver:

eth0: BUG! Tx Ring full when queue awake!

Shall I send a patch, or have I missed something?

-Jeff

#1 lore raw
Quoted 33 lines · Jeff Moyer

Hi, Matt,

This part of the netpoll trapped logic seems suspect to me, from include/linux/netdevice.h:

static inline void netif_wake_queue(struct net_device *dev) {

#ifdef CONFIG_NETPOLL_TRAP
	if (netpoll_trap())
		return;
#endif
	if (test_and_clear_bit(__LINK_STATE_XOFF, &dev->state))
		__netif_schedule(dev);
}

static inline void netif_stop_queue(struct net_device *dev) {

#ifdef CONFIG_NETPOLL_TRAP
	if (netpoll_trap())
		return;
#endif
	set_bit(__LINK_STATE_XOFF, &dev->state);
}

This looks buggy. Network drivers are now not able to stop the queue when they run out of Tx descriptors. I think the __netif_schedule is okay to do in the context of netpoll, and certainly a set_bit is okay. Why are these hooks in place? I've tested alt-sysrq-t over netconsole and also netdump with these #ifdef's removed, and things work correctly. Compare this with alt-sysrq-t hanging the system with these tests in place. If I run netdump with this logic still in place, I get the following messages from the tg3 driver:

Quoted 3 lines

eth0: BUG! Tx Ring full when queue awake!

Shall I send a patch, or have I missed something?

I don't remember the origin or motivation of this bit, so I'm not sure at the moment. Shoot me a patch and I'll poke at it.

Btw, did I send you my thoughts on your recursion patch?

Signature · 1 line
Mathematics is the supreme nostalgia of our time.
author
#2 lore raw

==> Regarding Re: netpoll trapped question; Matt Mackall <[email protected]> adds:

Quoted 14 lines · Jeff Moyer
Quoted 14 lines

Hi, Matt,

This part of the netpoll trapped logic seems suspect to me, from include/linux/netdevice.h:

static inline void netif_wake_queue(struct net_device *dev) { #ifdef CONFIG_NETPOLL_TRAP if (netpoll_trap()) return; #endif if (test_and_clear_bit(__LINK_STATE_XOFF, &dev->state)) __netif_schedule(dev); }

static inline void netif_stop_queue(struct net_device *dev) { #ifdef CONFIG_NETPOLL_TRAP if (netpoll_trap()) return; #endif set_bit(__LINK_STATE_XOFF, &dev->state); }

Quoted 8 lines
Quoted 8 lines

This looks buggy. Network drivers are now not able to stop the queue when they run out of Tx descriptors. I think the __netif_schedule is okay to do in the context of netpoll, and certainly a set_bit is okay. Why are these hooks in place? I've tested alt-sysrq-t over netconsole and also netdump with these #ifdef's removed, and things work correctly. Compare this with alt-sysrq-t hanging the system with these tests in place. If I run netdump with this logic still in place, I get the following messages from the tg3

driver>

Quoted 1 line
Quoted 1 line

eth0> BUG! Tx Ring full when queue awake!

Quoted 1 line
Quoted 1 line

Shall I send a patch, or have I missed something?

mpm> I don't remember the origin or motivation of this bit, so I'm not sure mpm> at the moment. Shoot me a patch and I'll poke at it.

What tree would you like me to base the patch on? In absence of response, it will be the latest -mm.

mpm> Btw, did I send you my thoughts on your recursion patch?

Yes, from your mail:

mpm> But I still don't like this. dev->poll() is liable to attempt to mpm> recursively take its own driver lock again internally and we still mpm> deadlock. Have we already seen recursion here? If we do, I think we mpm> need to fix that in drivers. Meanwhile we should just bail here and mpm> maybe set a "something bad happened" flag.

I'm not sure I follow. Which lock are you concerned about deadlocking on?

Quoted 1 line

From an earlier message, you say this:

mpm> It can [recurse] if the poll function does a printk or the like and mpm> wants to recurse via netconsole.

So I think it's worth addressing in netpoll, as opposed to every driver.

-Jeff

#3 lore raw
Quoted 27 lines · Jeff Moyer

==> Regarding Re: netpoll trapped question; Matt Mackall <[email protected]> adds:

Quoted 14 lines · Jeff Moyer
Quoted 14 lines

Hi, Matt,

This part of the netpoll trapped logic seems suspect to me, from include/linux/netdevice.h:

static inline void netif_wake_queue(struct net_device *dev) { #ifdef CONFIG_NETPOLL_TRAP if (netpoll_trap()) return; #endif if (test_and_clear_bit(__LINK_STATE_XOFF, &dev->state)) __netif_schedule(dev); }

static inline void netif_stop_queue(struct net_device *dev) { #ifdef CONFIG_NETPOLL_TRAP if (netpoll_trap()) return; #endif set_bit(__LINK_STATE_XOFF, &dev->state); }

Quoted 8 lines
Quoted 8 lines

This looks buggy. Network drivers are now not able to stop the queue when they run out of Tx descriptors. I think the __netif_schedule is okay to do in the context of netpoll, and certainly a set_bit is okay. Why are these hooks in place? I've tested alt-sysrq-t over netconsole and also netdump with these #ifdef's removed, and things work correctly. Compare this with alt-sysrq-t hanging the system with these tests in place. If I run netdump with this logic still in place, I get the following messages from the tg3

driver>

Quoted 1 line
Quoted 1 line
Quoted 8 lines

eth0> BUG! Tx Ring full when queue awake!

Quoted 1 line
Quoted 1 line

Shall I send a patch, or have I missed something?

mpm> I don't remember the origin or motivation of this bit, so I'm not sure mpm> at the moment. Shoot me a patch and I'll poke at it.

What tree would you like me to base the patch on? In absence of response, it will be the latest -mm.

-mm is fine.

Quoted 11 lines

mpm> Btw, did I send you my thoughts on your recursion patch?

Yes, from your mail:

mpm> But I still don't like this. dev->poll() is liable to attempt to mpm> recursively take its own driver lock again internally and we still mpm> deadlock. Have we already seen recursion here? If we do, I think we mpm> need to fix that in drivers. Meanwhile we should just bail here and mpm> maybe set a "something bad happened" flag.

I'm not sure I follow. Which lock are you concerned about deadlocking on?

A random lock private to a given driver, for instance one taken on entry to the IRQ handler. If said driver tries to do a printk inside that lock and we recursively call the handler in netconsole, we're in trouble. These are the issues that will have to be cleaned up in individual drivers. So far, I haven't seen any reports, but I'm pretty sure such cases exist. I suppose it's also possible for us to disable recursion in netconsole instead of at the netpoll level.

Signature · 1 line
Mathematics is the supreme nostalgia of our time.
author
#4 lore raw

==> Regarding Re: netpoll trapped question; Matt Mackall <[email protected]> adds:

Quoted 3 lines · Jeff Moyer
Quoted 3 lines

==> Regarding Re: netpoll trapped question; Matt Mackall <[email protected]> adds:

Quoted 14 lines · Jeff Moyer
Quoted 14 lines
Quoted 3 lines
Quoted 3 lines

Hi, Matt,

This part of the netpoll trapped logic seems suspect to me, from >>

include/linux/netdevice.h:

Quoted 3 lines
Quoted 3 lines

static inline void netif_wake_queue(struct net_device *dev) { #ifdef CONFIG_NETPOLL_TRAP if (netpoll_trap()) return; #endif if >>

(test_and_clear_bit(__LINK_STATE_XOFF, &dev->state)) >> __netif_schedule(dev); }

Quoted 3 lines
Quoted 3 lines

static inline void netif_stop_queue(struct net_device *dev) { #ifdef CONFIG_NETPOLL_TRAP if (netpoll_trap()) return; #endif >>

set_bit(__LINK_STATE_XOFF, &dev->state); }

Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

This looks buggy. Network drivers are now not able to stop the queue when they run out of Tx descriptors. I think the __netif_schedule is okay to do in the context of netpoll, and certainly a set_bit is

okay. >> Why are these hooks in place? I've tested alt-sysrq-t over netconsole >> and also netdump with these #ifdef's removed, and things work correctly. >> Compare this with alt-sysrq-t hanging the system with these tests in >> place. If I run netdump with this logic still in place, I get the >> following messages from the tg3

driver>

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

eth0> BUG! Tx Ring full when queue awake!

Quoted 2 lines
Quoted 2 lines
Quoted 1 line
Quoted 1 line

Shall I send a patch, or have I missed something?

mpm> I don't remember the origin or motivation of this bit, so I'm not sure mpm> at the moment. Shoot me a patch and I'll poke at it.

Quoted 2 lines
Quoted 2 lines

What tree would you like me to base the patch on? In absence of response, it will be the latest -mm.

mpm> -mm is fine.

mpm> Btw, did I send you my thoughts on your recursion patch?

Quoted 2 lines
Quoted 2 lines

Yes, from your mail:

mpm> But I still don't like this. dev->poll() is liable to attempt to mpm> recursively take its own driver lock again internally and we still mpm> deadlock. Have we already seen recursion here? If we do, I think we mpm> need to fix that in drivers. Meanwhile we should just bail here and mpm> maybe set a "something bad happened" flag.

Quoted 2 lines
Quoted 2 lines

I'm not sure I follow. Which lock are you concerned about deadlocking on?

mpm> A random lock private to a given driver, for instance one taken on mpm> entry to the IRQ handler. If said driver tries to do a printk inside mpm> that lock and we recursively call the handler in netconsole, we're in mpm> trouble. These are the issues that will have to be cleaned up in mpm> individual drivers. So far, I haven't seen any reports, but I'm pretty mpm> sure such cases exist. I suppose it's also possible for us to disable mpm> recursion in netconsole instead of at the netpoll level.

Recursion in netconsole is protected by the console semaphore.

-Jeff

#5 lore raw
Quoted 9 lines · Jeff Moyer

mpm> A random lock private to a given driver, for instance one taken on mpm> entry to the IRQ handler. If said driver tries to do a printk inside mpm> that lock and we recursively call the handler in netconsole, we're in mpm> trouble. These are the issues that will have to be cleaned up in mpm> individual drivers. So far, I haven't seen any reports, but I'm pretty mpm> sure such cases exist. I suppose it's also possible for us to disable mpm> recursion in netconsole instead of at the netpoll level.

Recursion in netconsole is protected by the console semaphore.

Yes, true. But we're still in trouble if we have nic irq handler -> take private lock -> printk -> netconsole -> nic irq handler -> take private lock. See?

Signature · 1 line
Mathematics is the supreme nostalgia of our time.
author
#6 lore raw

==> Regarding Re: netpoll trapped question; Matt Mackall <[email protected]> adds:

mpm> On Tue, Sep 07, 2004 at 12:53:17PM -0400, Jeff Moyer wrote: A random mpm> lock private to a given driver, for instance one taken on entry to the mpm> IRQ handler. If said driver tries to do a printk inside that lock and mpm> we recursively call the handler in netconsole, we're in trouble. mpm> These are the issues that will have to be cleaned up in individual mpm> drivers. So far, I haven't seen any reports, but I'm pretty sure such mpm> cases exist. I suppose it's also possible for us to disable recursion mpm> in netconsole instead of at the netpoll level.

Quoted 1 line
Quoted 1 line

Recursion in netconsole is protected by the console semaphore.

mpm> Yes, true. But we're still in trouble if we have nic irq handler -> mpm> take private lock -> printk -> netconsole -> nic irq handler -> take mpm> private lock. See?

Okay, so that one has to be addressed on a per-driver basis. There's no way for us to detect that situation. And how do drivers address this? Simply don't printk inside the lock? I think that's reasonable.

-Jeff

#7 lore raw
Quoted 12 lines · Jeff Moyer

==> Regarding Re: netpoll trapped question; Matt Mackall <[email protected]> adds:

mpm> On Tue, Sep 07, 2004 at 12:53:17PM -0400, Jeff Moyer wrote: A random mpm> lock private to a given driver, for instance one taken on entry to the mpm> IRQ handler. If said driver tries to do a printk inside that lock and mpm> we recursively call the handler in netconsole, we're in trouble. mpm> These are the issues that will have to be cleaned up in individual mpm> drivers. So far, I haven't seen any reports, but I'm pretty sure such mpm> cases exist. I suppose it's also possible for us to disable recursion mpm> in netconsole instead of at the netpoll level.

Quoted 1 line
Quoted 1 line

Recursion in netconsole is protected by the console semaphore.

Quoted 7 lines

mpm> Yes, true. But we're still in trouble if we have nic irq handler -> mpm> take private lock -> printk -> netconsole -> nic irq handler -> take mpm> private lock. See?

Okay, so that one has to be addressed on a per-driver basis. There's no way for us to detect that situation. And how do drivers address this? Simply don't printk inside the lock? I think that's reasonable.

That or drop the lock where possible.

Signature · 1 line
Mathematics is the supreme nostalgia of our time.
#8 lore raw
In reply to Jeff Moyer
Quoted 8 lines · Jeff Moyer

mpm> Yes, true. But we're still in trouble if we have nic irq handler -> mpm> take private lock -> printk -> netconsole -> nic irq handler -> take mpm> private lock. See?

Okay, so that one has to be addressed on a per-driver basis. There's no way for us to detect that situation. And how do drivers address this? Simply don't printk inside the lock? I think that's reasonable.

Why not queue the message whenever you're in IRQ context, and only print when you are not?

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#9 lore raw
Quoted 6 lines · Herbert Xu
Quoted 5 lines · Jeff Moyer

mpm> Yes, true. But we're still in trouble if we have nic irq handler -> mpm> take private lock -> printk -> netconsole -> nic irq handler -> take mpm> private lock. See?

Quoted 6 lines
Quoted 3 lines

Okay, so that one has to be addressed on a per-driver basis. There's no way for us to detect that situation. And how do drivers address this? Simply don't printk inside the lock? I think that's reasonable.

Why not queue the message whenever you're in IRQ context, and only print when you are not?

Actually how can this happen at all? The IRQ handler should've disabled local IRQs which prevents the second handler from occuring.

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
author
#10 lore raw

==> Regarding Re: netpoll trapped question; Herbert Xu <[email protected]> adds:

Quoted 2 lines · herbert> Herbert Xu <[email protected]> wrote:
Quoted 2 lines
Quoted 1 line · Jeff Moyer

mpm> Yes, true. But we're still in trouble if we have nic irq handler -> mpm> take private lock -> printk -> netconsole -> nic irq handler -> take mpm> private lock. See?

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

Okay, so that one has to be addressed on a per-driver basis. There's no way for us to detect that situation. And how do drivers address this? Simply don't printk inside the lock? I think that's reasonable.

Why not queue the message whenever you're in IRQ context, and only print when you are not?

This question has been answered before. We don't want to defer the output of say an oops or panic message, since we may never get back to process context.

herbert> Actually how can this happen at all? The IRQ handler should've herbert> disabled local IRQs which prevents the second handler from herbert> occuring.

The netpoll infrastructure calls into the device's netpoll routine. This will, in turn, call the interrupt routine for the device.

-Jeff