drivers/net/8139too.c: deadlock?
the lock validator i'm working on found the following scenario in the rtl8139 driver, which it flagged as a deadlock:
----------------------------------------> NETDEV WATCHDOG: eth0: transmit timed out
eth0: Transmit timeout, status 0d 0000 c07f media 80.
eth0: Tx queue start entry 164281 dirty entry 164277.
eth0: Tx descriptor 0 is 000805ea. eth0: Tx descriptor 1 is 00080042. (queue head) eth0: Tx descriptor 2 is 00080042. eth0: Tx descriptor 3 is 000805ea.
============================================
[ BUG: circular locking deadlock detected! ]
--------------------------------------------
hackbench/9560 [2] is trying to acquire lock {&tp->rx_lock} at:
[<c033e460>] rtl8139_tx_timeout+0x110/0x1f0
but task is already holding lock {&dev->xmit_lock}, acquired at:
[<c045294b>] dev_watchdog+0x1b/0xc0
which lock already depends on the new lock,
which could lead to circular deadlocks! the dependency chain (in reverse order) is:
-> #4 {&dev->xmit_lock}: [<c045294b>] dev_watchdog+0x1b/0xc0 -> #3 {&dev->queue_lock}: [<c0447154>] dev_queue_xmit+0x64/0x290 -> #2 {&((sk)->sk_lock.slock)}: [<c043eb66>] sk_clone+0x66/0x200 -> #1 {&((sk)->sk_lock.slock)}: [<c047a116>] tcp_v4_rcv+0x726/0x9d0 -> #0 {&tp->rx_lock}: [<c033e460>] rtl8139_tx_timeout+0x110/0x1f0
other info that might help us debug this:
locks held by hackbench/9560:
#0: {net/unix/af_unix.c:&u->readlock} [<c0490e6f>] unix_stream_recvmsg+0xbf/0x4f0 #1: {&dev->xmit_lock} [<c045294b>] dev_watchdog+0x1b/0xc0
stack backtrace: [<c010432d>] show_trace+0xd/0x10 [<c0104347>] dump_stack+0x17/0x20 [<c0137d60>] print_circular_bug_tail+0x40/0x50 [<c013922f>] debug_lock_chain+0x74f/0xd40 [<c013985d>] debug_lock_chain_spin+0x3d/0x60 [<c0266add>] _raw_spin_lock+0x2d/0x90 [<c04d9d18>] _spin_lock+0x8/0x10 [<c033e460>] rtl8139_tx_timeout+0x110/0x1f0 [<c04529e8>] dev_watchdog+0xb8/0xc0 [<c0127615>] run_timer_softirq+0xf5/0x1f0 [<c0122f67>] __do_softirq+0x97/0x130 [<c0105519>] do_softirq+0x69/0x100 ======================= [<c0122c19>] irq_exit+0x39/0x50 [<c010f4cc>] smp_apic_timer_interrupt+0x4c/0x50 [<c010393b>] apic_timer_interrupt+0x27/0x2c [<c0441829>] skb_release_data+0x59/0xa0 [<c0441b43>] kfree_skbmem+0x13/0xe0 [<c0441c58>] __kfree_skb+0x48/0xc0 [<c0490f7c>] unix_stream_recvmsg+0x1cc/0x4f0 [<c043d2d5>] do_sock_read+0x95/0xd0 [<c043d475>] sock_aio_read+0x75/0x80 [<c016499b>] do_sync_read+0xbb/0x110 [<c0164e88>] vfs_read+0x148/0x150 [<c01658bd>] sys_read+0x3d/0x70 [<c0102df7>] sysenter_past_esp+0x54/0x8d eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1 <----------------------------------------------
i'm wondering, is this a genuine deadlock, or a false positive? The dependency chain is quite complex, but looks realistic:
-> #4 {&dev->xmit_lock}: [<c045294b>] dev_watchdog+0x1b/0xc0 -> #3 {&dev->queue_lock}: [<c0447154>] dev_queue_xmit+0x64/0x290 -> #2 {&((sk)->sk_lock.slock)}: [<c043eb66>] sk_clone+0x66/0x200 -> #1 {&((sk)->sk_lock.slock)}: [<c047a116>] tcp_v4_rcv+0x726/0x9d0 -> #0 {&tp->rx_lock}: [<c033e460>] rtl8139_tx_timeout+0x110/0x1f0
and rtl8139_tx_timeout() is rare enough to not cause real lockups in practice all that often.
explanation of the validator output: the above dependency chain does not mean it actually occured in one single call sequence - it is a comulative (and full) depdency graph the validator is maintaining, to prove locking correctness. So it can easily be multiple tasks, at distinct points in time, on different CPUs, which built this dependency chain. The first (#0) and the last (#4) entry is the current locking sequence's fingerprint - so do not understand the above to be an actual locking stack - it cannot possibly occur in this order.
Ingo
Ingo Molnar <[email protected]> :
[...]
Quoted 8 lines
i'm wondering, is this a genuine deadlock, or a false positive? The dependency chain is quite complex, but looks realistic:
-> #4 {
&dev->xmit_lock}: [<c045294b>]dev_watchdog+0x1b/0xc0-> #3 {&dev->queue_lock}: [<c0447154>]dev_queue_xmit+0x64/0x290-> #2 {&((sk)->sk_lock.slock)}: [<c043eb66>]sk_clone+0x66/0x200-> #1 {&((sk)->sk_lock.slock)}: [<c047a116>]tcp_v4_rcv+0x726/0x9d0-> #0 {&tp->rx_lock}: [<c033e460>]rtl8139_tx_timeout+0x110/0x1f0
It looks like watchdog racing against rx_poll (which should/can not happen). Do you have something specific in mind ?
--
Ueimor
Quoted 5 lines · Francois Romieu
Ingo Molnar <[email protected]> :
[...]Quoted 3 lines
i'm wondering, is this a genuine deadlock, or a false positive? The dependency chain is quite complex, but looks realistic:
Quoted 8 lines
Quoted 5 lines
-> #4 {
&dev->xmit_lock}: [<c045294b>]dev_watchdog+0x1b/0xc0-> #3 {&dev->queue_lock}: [<c0447154>]dev_queue_xmit+0x64/0x290-> #2 {&((sk)->sk_lock.slock)}: [<c043eb66>]sk_clone+0x66/0x200-> #1 {&((sk)->sk_lock.slock)}: [<c047a116>]tcp_v4_rcv+0x726/0x9d0-> #0 {&tp->rx_lock}: [<c033e460>]rtl8139_tx_timeout+0x110/0x1f0It looks like watchdog racing against
rx_poll(which should/can not happen). Do you have something specific in mind ?
You've got it.
rx_poll => rtl8139_rx => netif_receive_skb => ... => tcp_v4_rcv
In fact once we're at netif_receive_skb it's easy to see how we'll grab xmit_lock again.
Prescription: Move TX timeout handling into a work queue.
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtQuoted 8 lines · Herbert Xu
You've got it.
rx_poll=>rtl8139_rx=>netif_receive_skb=> ... =>tcp_v4_rcvIn fact once we're at
netif_receive_skbit's easy to see how we'll grabxmit_lockagain.Prescription: Move TX timeout handling into a work queue.
This was my recommendation as well.
Using a work queue not only solves this locking problem, but it also enables the possibility of sleeping, a common operation during the hardware resets that often occur during ->tx_timeout()
Anyone who wants to make this change in a driver, don't forget that this could race with other operations, notably dev->stop() or suspend.
Jeff
PATCH 1/1 8139too: fix a TX timeout watchdog thread against NAPI softirq race
Ingo's stealth lock validator detected that both thread acquire dev->xmit_lock and tp->rx_lock in reverse order.
471819cf98a56751ae0387400542fe2997855327
drivers/net/8139too.c +26 -12
@@ -586,6 +586,7 @@ struct rtl8139_private {dma_addr_t tx_bufs_dma;signed char phys[4 ];/* MII device addresses. */ char twistie, twist_row, twist_col;/* Twister tune state. */ + unsigned int watchdog_fired : 1; unsigned int default_port : 4; /* Last dev->if_port value. */ unsigned int have_thread : 1;spinlock_t lock;
@@ -638,6 +639,7 @@ static void rtl8139_set_rx_mode (structstatic void __set_rx_mode (struct net_device *dev);static void rtl8139_hw_start (struct net_device *dev);static void rtl8139_thread (void *_data); +static void rtl8139_tx_timeout_task (void *_data);static struct ethtool_ops rtl8139_ethtool_ops;/* write MMIO register, with flush */
@@ -1598,13 +1600,14 @@ static void rtl8139_thread (void *_data) {struct net_device *dev = _data;struct rtl8139_private *tp =netdev_priv (dev); -unsigned long thr_delay; +unsigned long thr_delay = next_tick; -if (rtnl_shlock_nowait () ==0 ) { +if (tp->watchdog_fired) { + tp->watchdog_fired =0 ; +rtl8139_tx_timeout_task (_data); + }else if (rtnl_shlock_nowait () ==0 ) {rtl8139_thread_iter (dev, tp, tp->mmio_addr);rtnl_unlock (); - - thr_delay = next_tick; }else {/* unlikely race. mitigate with fast poll. */ thr_delay =HZ /2 ;
@@ -1631,7 +1634,8 @@ static void rtl8139_stop_thread(struct rif (tp->have_thread) {cancel_rearming_delayed_work (&tp->thread); tp->have_thread =0 ; - } + }else +flush_scheduled_work (); }static inline void rtl8139_tx_clear (struct rtl8139_private *tp)
@@ -1642,14 +1646,13 @@ static inline void rtl8139_tx_clear (str/* XXX account for unsent Tx packets in tp->stats.tx_dropped */ } - -static void rtl8139_tx_timeout (struct net_device *dev) +static void rtl8139_tx_timeout_task (void *_data) { +struct net_device *dev = _data;struct rtl8139_private *tp =netdev_priv (dev);void __iomem *ioaddr = tp->mmio_addr;int i;u8 tmp8; -unsigned long flags; printk (KERN_DEBUG "%s: Transmit timeout, status %2.2x %4.4x %4.4x " "media %2.2x.\n", dev->name, RTL_R8 (ChipCmd),
@@ -1670,23 +1673,34 @@ static void rtl8139_tx_timeout (struct nif (tmp8 & CmdTxEnb)RTL_W8 (ChipCmd, CmdRxEnb); -spin_lock (&tp->rx_lock); +spin_lock_bh (&tp->rx_lock);/* Disable interrupts by clearing the interrupt mask. */ RTL_W16 (IntrMask,0x0000 );/* Stop a shared interrupt from scavenging while we are. */ -spin_lock_irqsave (&tp->lock, flags); +spin_lock_irq (&tp->lock);rtl8139_tx_clear (tp); -spin_unlock_irqrestore (&tp->lock, flags); +spin_unlock_irq (&tp->lock);/* ...and finally, reset everything */ if (netif_running (dev)) {rtl8139_hw_start (dev);netif_wake_queue (dev); } -spin_unlock (&tp->rx_lock); +spin_unlock_bh (&tp->rx_lock); } +static void rtl8139_tx_timeout (struct net_device *dev) +{ +struct rtl8139_private *tp =netdev_priv (dev); + +if (!tp->have_thread) { +INIT_WORK (&tp->thread, rtl8139_tx_timeout_task, dev); +schedule_delayed_work (&tp->thread, next_tick); + }else + tp->watchdog_fired =1 ; + +}static int rtl8139_start_xmit (struct sk_buff *skb,struct net_device *dev) {
Signature · 1 line
1.1.3
Re: [PATCH] 8139too: fix a TX timeout watchdog thread against NAPI softirq race
Quoted 4 lines · Francois Romieu
Ingo's stealth lock validator detected that both thread acquire
dev->xmit_lockandtp->rx_lockin reverse order.Signed-off-by Francois Romieu <[email protected]>
i've ported this to -mm4 (see the attached patch), but i'm also getting a new deadlock message. Seems to be a separate issue, not introduced by your patch - but still needs fixing i guess.
Ingo
=========================================
[ BUG: irq lock inversion bug detected! ]
-----------------------------------------
rcu_torture_rea/263 just changed the state of lock:
(&mc->mca_lock){-+}, at: [<c04b4c0f>] mld_ifc_timer_expire+0x17f/0x2c0
but this lock took another, softirq-unsafe lock in the past:
(&tp->lock){--}, at: [<c03406d2>] rtl8139_set_rx_mode+0x22/0x180
and interrupts could create an inverse lock dependency between them, which could lead to deadlocks!
other info that might help us debug this:
no locks held by rcu_torture_rea/263.
-> (&tp->lock){--} {
used at: [<c03413da>] rtl8139_interrupt+0x2a/0x5e0
softirq-on at: [<c044b5fa>] dev_mc_upload+0x2a/0x40 hardirq-on at: [<c0160a28>] kmem_cache_alloc+0x98/0xd0
}
... key at: [<c033fffb>] rtl8139_init_one+0x2ab/0x960
... acquired at: [<c03406d2>] rtl8139_set_rx_mode+0x22/0x180
the first lock's dependencies:
-> (&mc->mca_lock){-+} {
used at: [<c04b3440>] igmp6_group_added+0x20/0x170
in-softirq at: [<c04b4c0f>] mld_ifc_timer_expire+0x17f/0x2c0 hardirq-on at: [<c04dcaa5>] _spin_unlock_irqrestore+0x25/0x30
}
... key at: [<c04b3edd>] ipv6_dev_mc_inc+0x17d/0x3c0
... acquired at: [<c04b4c0f>] mld_ifc_timer_expire+0x17f/0x2c0
-> (&dev->xmit_lock){-.} {
used at: [<c044b5ec>] dev_mc_upload+0x1c/0x40
hardirq-on at: [<c04dba6a>] __mutex_lock_slowpath+0x27a/0x4e0
}
... key at: [<c0449fdd>] register_netdevice+0x5d/0x3a0
... acquired at: [<c044b724>] dev_mc_add+0x34/0x140 -> (&tp->lock){--} {
used at: [<c03413da>] rtl8139_interrupt+0x2a/0x5e0
softirq-on at: [<c044b5fa>] dev_mc_upload+0x2a/0x40
hardirq-on at: [<c0160a28>] kmem_cache_alloc+0x98/0xd0
}
... key at: [<c033fffb>] rtl8139_init_one+0x2ab/0x960
... acquired at: [<c03406d2>] rtl8139_set_rx_mode+0x22/0x180
-> (&base->t_base.lock){++} {
used at: [<c0126df3>] lock_timer_base+0x23/0x50
in-hardirq at: [<c0126df3>] lock_timer_base+0x23/0x50
in-softirq at: [<c01277f2>] run_timer_softirq+0x42/0x1f0
}
... key at: [<c1fd2ef0>] 0xc1fd2ef0
... acquired at: [<c0126df3>] lock_timer_base+0x23/0x50the second lock's dependencies:
-> (&tp->lock){--} {
used at: [<c03413da>] rtl8139_interrupt+0x2a/0x5e0
softirq-on at: [<c044b5fa>] dev_mc_upload+0x2a/0x40 hardirq-on at: [<c0160a28>] kmem_cache_alloc+0x98/0xd0
}
... key at: [<c033fffb>] rtl8139_init_one+0x2ab/0x960
... acquired at: [<c03406d2>] rtl8139_set_rx_mode+0x22/0x180
stack backtrace: [<c010437d>] show_trace+0xd/0x10 [<c0104397>] dump_stack+0x17/0x20 [<c0137db2>] print_irq_inversion_bug+0x142/0x1b0 [<c0137f91>] check_usage_forwards+0x41/0x50 [<c0139520>] mark_lock+0x180/0x350 [<c0139b83>] debug_lock_chain+0x493/0x1090 [<c013a7bd>] debug_lock_chain_spin+0x3d/0x60 [<c026777d>] _raw_spin_lock+0x2d/0x90 [<c04dc942>] _spin_lock_bh+0x12/0x20 [<c04b4c0f>] mld_ifc_timer_expire+0x17f/0x2c0 [<c01278a5>] run_timer_softirq+0xf5/0x1f0 [<c01233a7>] __do_softirq+0x97/0x130 [<c01054d9>] do_softirq+0x69/0x100 ======================= [<c0123065>] irq_exit+0x45/0x50 [<c010f534>] smp_apic_timer_interrupt+0x54/0x60 [<c010395b>] apic_timer_interrupt+0x27/0x2c [<c0265f0c>] __delay+0xc/0x10 [<c0265f71>] __udelay+0x31/0x40 [<c0143333>] rcu_torture_reader+0x83/0x140 [<c0131dea>] kthread+0xca/0xd0 [<c0100ef5>] kernel_thread_helper+0x5/0x10
eth0: no IPv6 routers present
Ingo's stealth lock validator detected that both thread acquire dev->xmit_lock and tp->rx_lock in reverse order.
----
drivers/net/8139too.c | 36 ++++++++++++++++++++++++++---------- 1 files changed, 26 insertions(+), 10 deletions(-)
drivers/net/8139too.c +26 -10
@@ -586,6 +586,7 @@ struct rtl8139_private {dma_addr_t tx_bufs_dma;signed char phys[4 ];/* MII device addresses. */ char twistie, twist_row, twist_col;/* Twister tune state. */ + unsigned int watchdog_fired : 1; unsigned int default_port : 4; /* Last dev->if_port value. */ unsigned int have_thread : 1;spinlock_t lock;
@@ -638,6 +639,7 @@ static void rtl8139_set_rx_mode (structstatic void __set_rx_mode (struct net_device *dev);static void rtl8139_hw_start (struct net_device *dev);static void rtl8139_thread (void *_data); +static void rtl8139_tx_timeout_task (void *_data);static struct ethtool_ops rtl8139_ethtool_ops;/* write MMIO register, with flush */
@@ -1598,9 +1600,12 @@ static void rtl8139_thread (void *_data) {struct net_device *dev = _data;struct rtl8139_private *tp =netdev_priv (dev); -unsigned long thr_delay; +unsigned long thr_delay = next_tick; -if (rtnl_shlock_nowait () !=0 ) { +if (tp->watchdog_fired) { + tp->watchdog_fired =0 ; +rtl8139_tx_timeout_task (_data); + }else if (rtnl_shlock_nowait () !=0 ) {rtl8139_thread_iter (dev, tp, tp->mmio_addr);rtnl_unlock ();
@@ -1631,7 +1636,8 @@ static void rtl8139_stop_thread(struct rif (tp->have_thread) {cancel_rearming_delayed_work (&tp->thread); tp->have_thread =0 ; - } + }else +flush_scheduled_work (); }static inline void rtl8139_tx_clear (struct rtl8139_private *tp)
@@ -1642,14 +1648,13 @@ static inline void rtl8139_tx_clear (str/* XXX account for unsent Tx packets in tp->stats.tx_dropped */ } - -static void rtl8139_tx_timeout (struct net_device *dev) +static void rtl8139_tx_timeout_task (void *_data) { +struct net_device *dev = _data;struct rtl8139_private *tp =netdev_priv (dev);void __iomem *ioaddr = tp->mmio_addr;int i;u8 tmp8; -unsigned long flags; printk (KERN_DEBUG "%s: Transmit timeout, status %2.2x %4.4x %4.4x " "media %2.2x.\n", dev->name, RTL_R8 (ChipCmd),
@@ -1670,23 +1675,34 @@ static void rtl8139_tx_timeout (struct nif (tmp8 & CmdTxEnb)RTL_W8 (ChipCmd, CmdRxEnb); -spin_lock (&tp->rx_lock); +spin_lock_bh (&tp->rx_lock);/* Disable interrupts by clearing the interrupt mask. */ RTL_W16 (IntrMask,0x0000 );/* Stop a shared interrupt from scavenging while we are. */ -spin_lock_irqsave (&tp->lock, flags); +spin_lock_irq (&tp->lock);rtl8139_tx_clear (tp); -spin_unlock_irqrestore (&tp->lock, flags); +spin_unlock_irq (&tp->lock);/* ...and finally, reset everything */ if (netif_running (dev)) {rtl8139_hw_start (dev);netif_wake_queue (dev); } -spin_unlock (&tp->rx_lock); +spin_unlock_bh (&tp->rx_lock); } +static void rtl8139_tx_timeout (struct net_device *dev) +{ +struct rtl8139_private *tp =netdev_priv (dev); + +if (!tp->have_thread) { +INIT_WORK (&tp->thread, rtl8139_tx_timeout_task, dev); +schedule_delayed_work (&tp->thread, next_tick); + }else + tp->watchdog_fired =1 ; + +}static int rtl8139_start_xmit (struct sk_buff *skb,struct net_device *dev) {
Re: [PATCH] 8139too: fix a TX timeout watchdog thread against NAPI softirq race
Quoted 8 lines · Ingo Molnar
Quoted 4 lines
Ingo's stealth lock validator detected that both thread acquire
dev->xmit_lockandtp->rx_lockin reverse order.Signed-off-by Francois Romieu <[email protected]>i've ported this to -mm4 (see the attached patch), but i'm also getting a new deadlock message. Seems to be a separate issue, not introduced by your patch - but still needs fixing i guess.
sorry - this is a false alarm. (caused by the lock validator not properly handling DEBUG_SHIRQ)
Ingo
Re: [PATCH] 8139too: fix a TX timeout watchdog thread against NAPI softirq race
In reply to Francois Romieudrivers/net/8139too.c struct rtl8139_private { · line 591
Quoted 4 lines · Francois Romieu
drivers/net/8139too.c +1
-0+ unsigned int watchdog_fired : 1; unsigned int default_port : 4; /* Last dev->if_port value. */ unsigned int have_thread : 1;
Bear in mind that the compiler will put these three fields into the same word and will use non-atomic RMWs when they are modified.
In this particualr case it's hard to see how an SMP or IRQ race can occur,
but it's a trap. Which won't show up on x86.
struct x {
unsigned int a:1;
unsigned int b:1;
};void foo(struct x *a)
{
a->a = 1;
}void bar(struct x *a)
{
a->b = 1;
}On ppc:
foo:
lwz 0,0(3)
oris 0,0,0x8000
stw 0,0(3)
blr
.size foo, .-foo
.align 2
.globl bar
.type bar, @function
bar:
lwz 0,0(3)
oris 0,0,0x4000
stw 0,0(3)
blr