New: RTNL: assertion failed at net/ipv6/addrconf.c (2164)/RTNL: assertion failed at net/ipv4/devinet.c (1055)

Bugme-new Andrew Morton 5 participants 24 messages Dec 11, 2007 – Jan 7, 2008 1 message not in the archive

Subsystems (3)

Networking Network drivers Bonding driver

lore

Share & tools

lore

A message that is not in the archive (sent off-list or not yet received): [email protected]/

Andrew Morton

author

2007-12-11 11:46 UTC #1 lore raw

Quoted 48 lines · On Tue, 11 Dec 2007 03:20:48 -0800 (PST) [email protected] wrote:

http://bugzilla.kernel.org/show_bug.cgi?id=9543

           Summary: RTNL: assertion failed at net/ipv6/addrconf.c
                    (2164)/RTNL: assertion failed at net/ipv4/devinet.c
                    (1055)
           Product: Drivers
           Version: 2.5
     KernelVersion: 2.6.24-rc4-git7
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: Network
        AssignedTo: [email protected]
        ReportedBy: [email protected]

Most recent kernel where this bug did not occur: 2.6.23
Distribution: Gentoo

Problem Description:
ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
RTNL: assertion failed at net/ipv6/addrconf.c (2164)
Pid: 9, comm: events/0 Not tainted 2.6.24-rc4-git7 #1

 [<78402cfb>] addrconf_notify+0x5b4/0x7b7
 [<7812203a>] finish_task_switch+0x0/0x8c
 [<781346ff>] worker_thread+0x0/0x85
 [<78438e23>] schedule+0x545/0x55f
 [<781408d1>] print_lock_contention_bug+0x11/0xd2
 [<783bfa72>] rt_run_flush+0x43/0x8b
 [<783bfa93>] rt_run_flush+0x64/0x8b
 [<7813ac54>] notifier_call_chain+0x2a/0x52
 [<7813ac9e>] raw_notifier_call_chain+0x17/0x1a
 [<783a3471>] netdev_state_change+0x18/0x29
 [<783ac6a9>] __linkwatch_run_queue+0x150/0x17e
 [<783ac6f4>] linkwatch_event+0x1d/0x22
 [<78133cdf>] run_workqueue+0xdb/0x1b6
 [<78133c8b>] run_workqueue+0x87/0x1b6
 [<783ac6d7>] linkwatch_event+0x0/0x22
 [<781346ff>] worker_thread+0x0/0x85
 [<78134778>] worker_thread+0x79/0x85
 [<781371ad>] autoremove_wake_function+0x0/0x35
 [<781370f6>] kthread+0x38/0x5e
 [<781370be>] kthread+0x0/0x5e
 [<78104baf>] kernel_thread_helper+0x7/0x10
 =======================

Quoted 1 line

RTNL: assertion failed at net/ipv6/addrconf.c (1610)

Hopefully this is due to the bug you reported in bug #9542.

Does this patch fix both issues?

From: Andrew Morton <[email protected]>

Remove stray rtnl_unlock().

Addresses http://bugzilla.kernel.org/show_bug.cgi?id=9542

Cc "David S. Miller" <[email protected]>

Cc Stephen Hemminger <[email protected]>

Cc Krzysztof Oledzki <[email protected]>

Signed-off-by Andrew Morton <[email protected]>

---

 drivers/net/bonding/bond_sysfs.c |    2 --
 1 file changed, 2 deletions(-)

drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -2

diff -puN drivers/net/bonding/bond_sysfs.c~bonding-locking-fix drivers/net/bonding/bond_sysfs.c
@@ -1111,8 +1111,6 @@ static ssize_t bonding_store_primary(str
 out:
 	write_unlock_bh(&bond->lock);
 
-	rtnl_unlock();
-
 	return count;
 }
 static DEVICE_ATTR(primary, S_IRUGO | S_IWUSR, bonding_show_primary, bonding_store_primary);

Krzysztof Oledzki

2007-12-11 15:05 UTC #2 lore raw

Quoted 53 lines · Andrew Morton

Quoted 48 lines · On Tue, 11 Dec 2007 03:20:48 -0800 (PST) [email protected] wrote:

http://bugzilla.kernel.org/show_bug.cgi?id=9543

           Summary: RTNL: assertion failed at net/ipv6/addrconf.c
                    (2164)/RTNL: assertion failed at net/ipv4/devinet.c
                    (1055)
           Product: Drivers
           Version: 2.5
     KernelVersion: 2.6.24-rc4-git7
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: Network
        AssignedTo: [email protected]
        ReportedBy: [email protected]

Most recent kernel where this bug did not occur: 2.6.23
Distribution: Gentoo

Problem Description:
ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
RTNL: assertion failed at net/ipv6/addrconf.c (2164)
Pid: 9, comm: events/0 Not tainted 2.6.24-rc4-git7 #1

 [<78402cfb>] addrconf_notify+0x5b4/0x7b7
 [<7812203a>] finish_task_switch+0x0/0x8c
 [<781346ff>] worker_thread+0x0/0x85
 [<78438e23>] schedule+0x545/0x55f
 [<781408d1>] print_lock_contention_bug+0x11/0xd2
 [<783bfa72>] rt_run_flush+0x43/0x8b
 [<783bfa93>] rt_run_flush+0x64/0x8b
 [<7813ac54>] notifier_call_chain+0x2a/0x52
 [<7813ac9e>] raw_notifier_call_chain+0x17/0x1a
 [<783a3471>] netdev_state_change+0x18/0x29
 [<783ac6a9>] __linkwatch_run_queue+0x150/0x17e
 [<783ac6f4>] linkwatch_event+0x1d/0x22
 [<78133cdf>] run_workqueue+0xdb/0x1b6
 [<78133c8b>] run_workqueue+0x87/0x1b6
 [<783ac6d7>] linkwatch_event+0x0/0x22
 [<781346ff>] worker_thread+0x0/0x85
 [<78134778>] worker_thread+0x79/0x85
 [<781371ad>] autoremove_wake_function+0x0/0x35
 [<781370f6>] kthread+0x38/0x5e
 [<781370be>] kthread+0x0/0x5e
 [<78104baf>] kernel_thread_helper+0x7/0x10
 =======================

Quoted 1 line

RTNL: assertion failed at net/ipv6/addrconf.c (1610)

Quoted 3 lines

Hopefully this is due to the bug you reported in bug #9542.
Does this patch fix both issues?

Unfortunately not. I just updated bugzilla.

Best regards,

Krzysztof Olędzki

Andrew Morton

author

2007-12-11 20:30 UTC #3 lore raw

Quoted 20 lines · On Tue, 11 Dec 2007 16:04:27 +0100 (CET) Krzysztof Oledzki <[email protected]> wrote:

Quoted 20 lines

Quoted 18 lines

 [<783bfa72>] rt_run_flush+0x43/0x8b
 [<783bfa93>] rt_run_flush+0x64/0x8b
 [<7813ac54>] notifier_call_chain+0x2a/0x52
 [<7813ac9e>] raw_notifier_call_chain+0x17/0x1a
 [<783a3471>] netdev_state_change+0x18/0x29
 [<783ac6a9>] __linkwatch_run_queue+0x150/0x17e
 [<783ac6f4>] linkwatch_event+0x1d/0x22
 [<78133cdf>] run_workqueue+0xdb/0x1b6
 [<78133c8b>] run_workqueue+0x87/0x1b6
 [<783ac6d7>] linkwatch_event+0x0/0x22
 [<781346ff>] worker_thread+0x0/0x85
 [<78134778>] worker_thread+0x79/0x85
 [<781371ad>] autoremove_wake_function+0x0/0x35
 [<781370f6>] kthread+0x38/0x5e
 [<781370be>] kthread+0x0/0x5e
 [<78104baf>] kernel_thread_helper+0x7/0x10
 =======================

RTNL: assertion failed at net/ipv6/addrconf.c (1610)

Quoted 5 lines

Quoted 3 lines
Hopefully this is due to the bug you reported in bug #9542.
Does this patch fix both issues?
Unfortunately not. I just updated bugzilla.

argh. Please conduct these discussions via emailed reply-to-all, not via the bugzilla web interface.

I think my patch _did_ fix http://bugzilla.kernel.org/show_bug.cgi?id=9542
But then you hit a new bonding locking bug.

Rafael, please track 9542 and 9543 as post-2.6.23 regressions, thanks.

Herbert Xu

2007-12-12 13:53 UTC #4 lore raw

In reply to Andrew Morton

Quoted 6 lines · Andrew Morton

From: Andrew Morton <[email protected]>
Remove stray rtnl_unlock().
Addresses http://bugzilla.kernel.org/show_bug.cgi?id=9542

Adnrew, please cc Jay Vosburgh <[email protected]> on bonding issues.

drivers/net/bonding/bond_sysfs.c static ssize_t bonding_store_primary(str · line 1115

Quoted 9 lines

diff -puN drivers/net/bonding/bond_sysfs.c~bonding-locking-fix drivers/net/bonding/bond_sysfs.c
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -0
@@ -1111,8 +1111,6 @@ static ssize_t bonding_store_primary(str
out:
       write_unlock_bh(&bond->lock);
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -2
-       rtnl_unlock();
-

Looking at the changeset that added this perhaps the intention is to hold the lock? If so we should add an rtnl_lock to the start of the function.

Thanks,

Signature · 4 lines

Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Jay Vosburgh

2007-12-12 17:47 UTC #5 lore raw

Quoted 7 lines · Herbert Xu

Quoted 7 lines
diff -puN drivers/net/bonding/bond_sysfs.c~bonding-locking-fix drivers/net/bonding/bond_sysfs.c
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -0
@@ -1111,8 +1111,6 @@ static ssize_t bonding_store_primary(str
out:
       write_unlock_bh(&bond->lock);

Quoted 6 lines

drivers/net/bonding/bond_sysfs.c
Quoted 2 lines
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -2
-       rtnl_unlock();
-
Looking at the changeset that added this perhaps the intention is to hold the lock? If so we should add an rtnl_lock to the start of the function.

Yes, this function needs to hold locks, and more than just
what's there now. I believe the following should be correct; I haven't tested it, though (I'm supposedly on vacation right now).

The following change should be correct for the
bonding_store_primary case discussed in this thread, and also corrects the bonding_store_active case which performs similar functions.

The bond_change_active_slave and bond_select_active_slave
functions both require rtnl, bond->lock for read and curr_slave_lock for write_bh, and no other locks. This is so that the lower level mode-specific functions can release locks down to just rtnl in order to call, e.g., dev_set_mac_address with the locks it expects (rtnl only).

Signed-off-by Jay Vosburgh <[email protected]>

drivers/net/bonding/bond_sysfs.c +10 -5

index 11b76b3..28a2d80 100644
@@ -1075,7 +1075,10 @@ static ssize_t bonding_store_primary(struct device *d,
 	struct slave *slave;
 	struct bonding *bond = to_bond(d);
 
-	write_lock_bh(&bond->lock);
+	rtnl_lock();
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
+
 	if (!USES_PRIMARY(bond->params.mode)) {
 		printk(KERN_INFO DRV_NAME
 		       ": %s: Unable to set primary slave; %s is in mode %d\n",

@@ -1109,8 +1112,8 @@ static ssize_t bonding_store_primary(struct device *d,
 		}
 	}
 out:
-	write_unlock_bh(&bond->lock);
-
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
 	rtnl_unlock();
 
 	return count;

@@ -1190,7 +1193,8 @@ static ssize_t bonding_store_active_slave(struct device *d,
 	struct bonding *bond = to_bond(d);
 
 	rtnl_lock();
-	write_lock_bh(&bond->lock);
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
 
 	if (!USES_PRIMARY(bond->params.mode)) {
 		printk(KERN_INFO DRV_NAME

@@ -1247,7 +1251,8 @@ static ssize_t bonding_store_active_slave(struct device *d,
 		}
 	}
 out:
-	write_unlock_bh(&bond->lock);
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
 	rtnl_unlock();
 
 	return count;

Andy Gospodarek

2007-12-12 19:07 UTC #6 lore raw

Quoted 24 lines · Jay Vosburgh

Quoted 7 lines · Herbert Xu
Quoted 7 lines
diff -puN drivers/net/bonding/bond_sysfs.c~bonding-locking-fix drivers/net/bonding/bond_sysfs.c
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -0
@@ -1111,8 +1111,6 @@ static ssize_t bonding_store_primary(str
out:
       write_unlock_bh(&bond->lock);
Quoted 6 lines
drivers/net/bonding/bond_sysfs.c
Quoted 2 lines
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -2
-       rtnl_unlock();
-
Looking at the changeset that added this perhaps the intention is to hold the lock? If so we should add an rtnl_lock to the start of the function.
Yes, this function needs to hold locks, and more than just
what's there now. I believe the following should be correct; I haven't tested it, though (I'm supposedly on vacation right now).
The following change should be correct for the
bonding_store_primary case discussed in this thread, and also corrects the bonding_store_active case which performs similar functions.

Quoted 7 lines

The bond_change_active_slave and bond_select_active_slave
functions both require rtnl, bond->lock for read and curr_slave_lock for write_bh, and no other locks. This is so that the lower level mode-specific functions can release locks down to just rtnl in order to call, e.g., dev_set_mac_address with the locks it expects (rtnl only).
Signed-off-by Jay Vosburgh <[email protected]>

This looks good to me as well....

Signed-off-by Andy Gospodarek <[email protected]>

Krzysztof Oledzki

2007-12-14 16:46 UTC #7 lore raw

In reply to Jay Vosburgh

Quoted 76 lines · Jay Vosburgh

Quoted 7 lines · Herbert Xu
Quoted 7 lines
diff -puN drivers/net/bonding/bond_sysfs.c~bonding-locking-fix drivers/net/bonding/bond_sysfs.c
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -0
@@ -1111,8 +1111,6 @@ static ssize_t bonding_store_primary(str
out:
       write_unlock_bh(&bond->lock);
Quoted 6 lines
drivers/net/bonding/bond_sysfs.c
Quoted 2 lines
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -2
-       rtnl_unlock();
-
Looking at the changeset that added this perhaps the intention is to hold the lock? If so we should add an rtnl_lock to the start of the function.
Yes, this function needs to hold locks, and more than just
what's there now. I believe the following should be correct; I haven't tested it, though (I'm supposedly on vacation right now).
The following change should be correct for the
bonding_store_primary case discussed in this thread, and also corrects the bonding_store_active case which performs similar functions.
The bond_change_active_slave and bond_select_active_slave
functions both require rtnl, bond->lock for read and curr_slave_lock for write_bh, and no other locks. This is so that the lower level mode-specific functions can release locks down to just rtnl in order to call, e.g., dev_set_mac_address with the locks it expects (rtnl only).
Signed-off-by Jay Vosburgh <[email protected]>
drivers/net/bonding/bond_sysfs.c +4 -1
index 11b76b3..28a2d80 100644
@@ -1075,7 +1075,10 @@ static ssize_t bonding_store_primary(struct device *d,
 	struct slave *slave;
 	struct bonding *bond = to_bond(d);
 
-	write_lock_bh(&bond->lock);
+	rtnl_lock();
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
+
 	if (!USES_PRIMARY(bond->params.mode)) {
 		printk(KERN_INFO DRV_NAME
 		       ": %s: Unable to set primary slave; %s is in mode %d\n",
@@ -1109,8 +1112,8 @@ static ssize_t bonding_store_primary(struct device *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +2 -2
-	write_unlock_bh(&bond->lock);
-
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
	rtnl_unlock();

	return count;
drivers/net/bonding/bond_sysfs.c +2 -1
@@ -1190,7 +1193,8 @@ static ssize_t bonding_store_active_slave(struct device *d,
 	struct bonding *bond = to_bond(d);
 
 	rtnl_lock();
-	write_lock_bh(&bond->lock);
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
 
 	if (!USES_PRIMARY(bond->params.mode)) {
 		printk(KERN_INFO DRV_NAME
@@ -1247,7 +1251,8 @@ static ssize_t bonding_store_active_slave(struct device *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +2 -1
-	write_unlock_bh(&bond->lock);
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);

Quoted 3 lines

	rtnl_unlock();

	return count;

Vanilla 2.6.24-rc5 plus this patch:

=========================================================
[ INFO: possible irq lock inversion dependency detected ] 2.6.24-rc5 #1
---------------------------------------------------------
events/0/9 just changed the state of lock:
(&mc->mca_lock){-+..}, at: [<c0411c7a>] mld_ifc_timer_expire+0x130/0x1fb
but this lock took another, soft-read-irq-unsafe lock in the past:
(&bond->lock){-.--}

and interrupts could create inverse lock ordering between them.

other info that might help us debug this:
4 locks held by events/0/9:
#0: (events){--..}, at: [<c0133c57>] run_workqueue+0x87/0x1b6 #1: ((linkwatch_work).work){--..}, at: [<c0133c57>]

run_workqueue+0x87/0x1b6
 #2:  (rtnl_mutex){--..}, at: [<c03abd50>] linkwatch_event+0x5/0x22
 #3:  (&ndev->lock){-.-+}, at: [<c0411b61>] 
mld_ifc_timer_expire+0x17/0x1fb

the first lock's dependencies:
-> (&mc->mca_lock){-+..} ops: 10 {
initial-use at:

                        [<c0104ee2>] dump_trace+0x83/0x8d
                        [<c014289c>] __lock_acquire+0x4ba/0xc07
                        [<c0109ef2>] save_stack_trace+0x20/0x3a
                        [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                        [<c0412452>] ipv6_dev_mc_inc+0x24d/0x31c
                        [<c0143062>] lock_acquire+0x79/0x93
                        [<c04120d6>] igmp6_group_added+0x18/0x11d
                        [<c0439d62>] _spin_lock_bh+0x3b/0x64
                        [<c04120d6>] igmp6_group_added+0x18/0x11d
                        [<c04120d6>] igmp6_group_added+0x18/0x11d
                        [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                        [<c04124a8>] ipv6_dev_mc_inc+0x2a3/0x31c
                        [<c0412452>] ipv6_dev_mc_inc+0x24d/0x31c
                        [<c04124dd>] ipv6_dev_mc_inc+0x2d8/0x31c
                        [<c0412205>] ipv6_dev_mc_inc+0x0/0x31c
                        [<c0401834>] ipv6_add_dev+0x21c/0x24b
                        [<c040b07d>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
                        [<c05c5b40>] addrconf_init+0x13/0x193
                        [<c0199f63>] proc_net_fops_create+0x10/0x21
                        [<c0419b38>] ip6_flowlabel_init+0x1e/0x20
                        [<c05c5a20>] inet6_init+0x1f0/0x2ad
                        [<c05a9499>] kernel_init+0x150/0x2b7
                        [<c05a9349>] kernel_init+0x0/0x2b7
                        [<c05a9349>] kernel_init+0x0/0x2b7
                        [<c0104baf>] kernel_thread_helper+0x7/0x10
                        [<ffffffff>] 0xffffffff
   in-softirq-W at:
                        [<c0142822>] __lock_acquire+0x440/0xc07
                        [<c013e0f3>] clockevents_program_event+0xe0/0xee
                        [<c0143062>] lock_acquire+0x79/0x93
                        [<c0411c7a>] mld_ifc_timer_expire+0x130/0x1fb
                        [<c0411b4a>] mld_ifc_timer_expire+0x0/0x1fb
                        [<c0439d62>] _spin_lock_bh+0x3b/0x64
                        [<c0411c7a>] mld_ifc_timer_expire+0x130/0x1fb
                        [<c0411c7a>] mld_ifc_timer_expire+0x130/0x1fb
                        [<c0411b4a>] mld_ifc_timer_expire+0x0/0x1fb
                        [<c0141f89>] trace_hardirqs_on+0x10c/0x14c
                        [<c0411b4a>] mld_ifc_timer_expire+0x0/0x1fb
                        [<c012df52>] run_timer_softirq+0xfa/0x15d
                        [<c012a8a6>] __do_softirq+0x56/0xdb
                        [<c0141f89>] trace_hardirqs_on+0x10c/0x14c
                        [<c012a8b8>] __do_softirq+0x68/0xdb
                        [<c012a961>] do_softirq+0x36/0x51
                        [<c012ae4a>] local_bh_enable_ip+0xad/0xed
                        [<c03bf107>] rt_run_flush+0x64/0x8b
                        [<c03e9296>] fib_netdev_event+0x61/0x65
                        [<c013ac20>] notifier_call_chain+0x2a/0x52
                        [<c013ac6a>] raw_notifier_call_chain+0x17/0x1a
                        [<c03a2ae5>] netdev_state_change+0x18/0x29
                        [<c03abd1d>] __linkwatch_run_queue+0x150/0x17e
                        [<c03abd68>] linkwatch_event+0x1d/0x22
                        [<c0133cab>] run_workqueue+0xdb/0x1b6
                        [<c0133c57>] run_workqueue+0x87/0x1b6
                        [<c03abd4b>] linkwatch_event+0x0/0x22
                        [<c01346cb>] worker_thread+0x0/0x85
                        [<c0134744>] worker_thread+0x79/0x85
                        [<c0137179>] autoremove_wake_function+0x0/0x35
                        [<c01370c2>] kthread+0x38/0x5e
                        [<c013708a>] kthread+0x0/0x5e
                        [<c0104baf>] kernel_thread_helper+0x7/0x10
                        [<ffffffff>] 0xffffffff
   hardirq-on-W at:
                        [<c01417ee>] find_usage_backwards+0xbb/0xe2
                        [<c0104ee2>] dump_trace+0x83/0x8d
                        [<c014286a>] __lock_acquire+0x488/0xc07
                        [<c0109ef2>] save_stack_trace+0x20/0x3a
                        [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                        [<c0412452>] ipv6_dev_mc_inc+0x24d/0x31c
                        [<c0143062>] lock_acquire+0x79/0x93
                        [<c04120d6>] igmp6_group_added+0x18/0x11d
                        [<c0439d62>] _spin_lock_bh+0x3b/0x64
                        [<c04120d6>] igmp6_group_added+0x18/0x11d
                        [<c04120d6>] igmp6_group_added+0x18/0x11d
                        [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                        [<c04124a8>] ipv6_dev_mc_inc+0x2a3/0x31c
                        [<c0412452>] ipv6_dev_mc_inc+0x24d/0x31c
                        [<c04124dd>] ipv6_dev_mc_inc+0x2d8/0x31c
                        [<c0412205>] ipv6_dev_mc_inc+0x0/0x31c
                        [<c0401834>] ipv6_add_dev+0x21c/0x24b
                        [<c040b07d>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
                        [<c05c5b40>] addrconf_init+0x13/0x193
                        [<c0199f63>] proc_net_fops_create+0x10/0x21
                        [<c0419b38>] ip6_flowlabel_init+0x1e/0x20
                        [<c05c5a20>] inet6_init+0x1f0/0x2ad
                        [<c05a9499>] kernel_init+0x150/0x2b7
                        [<c05a9349>] kernel_init+0x0/0x2b7
                        [<c05a9349>] kernel_init+0x0/0x2b7
                        [<c0104baf>] kernel_thread_helper+0x7/0x10
                        [<ffffffff>] 0xffffffff
 }
 ... key      at: [<c087e2d8>] __key.30798+0x0/0x8
 -> (_xmit_ETHER){-...} ops: 8 {
    initial-use  at:
                          [<c014289c>] __lock_acquire+0x4ba/0xc07
                          [<c0143062>] lock_acquire+0x79/0x93
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c0439d62>] _spin_lock_bh+0x3b/0x64
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c0412114>] igmp6_group_added+0x56/0x11d
                          [<c04124a8>] ipv6_dev_mc_inc+0x2a3/0x31c
                          [<c0410100>] igmp6_mc_seq_start+0xde/0x138
                          [<c04124dd>] ipv6_dev_mc_inc+0x2d8/0x31c
                          [<c0412205>] ipv6_dev_mc_inc+0x0/0x31c
                          [<c0401834>] ipv6_add_dev+0x21c/0x24b
                          [<c040b07d>] 
ndisc_ifinfo_sysctl_change+0x0/0x1ef
                          [<c0401e17>] addrconf_notify+0x60/0x7b7
                          [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                          [<c0141dac>] mark_held_locks+0x39/0x53
                          [<c0439066>] mutex_lock_nested+0x286/0x2ac
                          [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                          [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
                          [<c03a3f4d>] 
register_netdevice_notifier+0xe/0x126
                          [<c03a3f4d>] 
register_netdevice_notifier+0xe/0x126
                          [<c03a3f88>] 
register_netdevice_notifier+0x49/0x126
                          [<c05c5bda>] addrconf_init+0xad/0x193
                          [<c05c5b48>] addrconf_init+0x1b/0x193
                          [<c05c5a20>] inet6_init+0x1f0/0x2ad
                          [<c05a9499>] kernel_init+0x150/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c0104baf>] kernel_thread_helper+0x7/0x10
                          [<ffffffff>] 0xffffffff
    hardirq-on-W at:
                          [<c0141986>] mark_lock+0x64/0x451
                          [<c014286a>] __lock_acquire+0x488/0xc07
                          [<c0143062>] lock_acquire+0x79/0x93
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c0439d62>] _spin_lock_bh+0x3b/0x64
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c0412114>] igmp6_group_added+0x56/0x11d
                          [<c04124a8>] ipv6_dev_mc_inc+0x2a3/0x31c
                          [<c0410100>] igmp6_mc_seq_start+0xde/0x138
                          [<c04124dd>] ipv6_dev_mc_inc+0x2d8/0x31c
                          [<c0412205>] ipv6_dev_mc_inc+0x0/0x31c
                          [<c0401834>] ipv6_add_dev+0x21c/0x24b
                          [<c040b07d>] 
ndisc_ifinfo_sysctl_change+0x0/0x1ef
                          [<c0401e17>] addrconf_notify+0x60/0x7b7
                          [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                          [<c0141dac>] mark_held_locks+0x39/0x53
                          [<c0439066>] mutex_lock_nested+0x286/0x2ac
                          [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                          [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
                          [<c03a3f4d>] 
register_netdevice_notifier+0xe/0x126
                          [<c03a3f4d>] 
register_netdevice_notifier+0xe/0x126
                          [<c03a3f88>] 
register_netdevice_notifier+0x49/0x126
                          [<c05c5bda>] addrconf_init+0xad/0x193
                          [<c05c5b48>] addrconf_init+0x1b/0x193
                          [<c05c5a20>] inet6_init+0x1f0/0x2ad
                          [<c05a9499>] kernel_init+0x150/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c0104baf>] kernel_thread_helper+0x7/0x10
                          [<ffffffff>] 0xffffffff
  }
  ... key      at: [<c087adc8>] netdev_xmit_lock_key+0x8/0x1c0
 ... acquired at:
   [<c0142dff>] __lock_acquire+0xa1d/0xc07
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c0143062>] lock_acquire+0x79/0x93
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c0439d62>] _spin_lock_bh+0x3b/0x64
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c0412114>] igmp6_group_added+0x56/0x11d
   [<c04124a8>] ipv6_dev_mc_inc+0x2a3/0x31c
   [<c0410100>] igmp6_mc_seq_start+0xde/0x138
   [<c04124dd>] ipv6_dev_mc_inc+0x2d8/0x31c
   [<c0412205>] ipv6_dev_mc_inc+0x0/0x31c
   [<c0401834>] ipv6_add_dev+0x21c/0x24b
   [<c040b07d>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
   [<c0401e17>] addrconf_notify+0x60/0x7b7
   [<c0142fa1>] __lock_acquire+0xbbf/0xc07
   [<c0141dac>] mark_held_locks+0x39/0x53
   [<c0439066>] mutex_lock_nested+0x286/0x2ac
   [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
   [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
   [<c03a3f4d>] register_netdevice_notifier+0xe/0x126
   [<c03a3f4d>] register_netdevice_notifier+0xe/0x126
   [<c03a3f88>] register_netdevice_notifier+0x49/0x126
   [<c05c5bda>] addrconf_init+0xad/0x193
   [<c05c5b48>] addrconf_init+0x1b/0x193
   [<c05c5a20>] inet6_init+0x1f0/0x2ad
   [<c05a9499>] kernel_init+0x150/0x2b7
   [<c05a9349>] kernel_init+0x0/0x2b7
   [<c05a9349>] kernel_init+0x0/0x2b7
   [<c0104baf>] kernel_thread_helper+0x7/0x10
   [<ffffffff>] 0xffffffff

 -> (&bonding_netdev_xmit_lock_key){-...} ops: 6 {
    initial-use  at:
                          [<c014289c>] __lock_acquire+0x4ba/0xc07
                          [<c0143062>] lock_acquire+0x79/0x93
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c0439d62>] _spin_lock_bh+0x3b/0x64
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c0412114>] igmp6_group_added+0x56/0x11d
                          [<c04124a8>] ipv6_dev_mc_inc+0x2a3/0x31c
                          [<c0410100>] igmp6_mc_seq_start+0xde/0x138
                          [<c04124dd>] ipv6_dev_mc_inc+0x2d8/0x31c
                          [<c0412205>] ipv6_dev_mc_inc+0x0/0x31c
                          [<c0401834>] ipv6_add_dev+0x21c/0x24b
                          [<c040b07d>] 
ndisc_ifinfo_sysctl_change+0x0/0x1ef
                          [<c0401e17>] addrconf_notify+0x60/0x7b7
                          [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                          [<c0141dac>] mark_held_locks+0x39/0x53
                          [<c0439066>] mutex_lock_nested+0x286/0x2ac
                          [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                          [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
                          [<c03a3f4d>] 
register_netdevice_notifier+0xe/0x126
                          [<c03a3f4d>] 
register_netdevice_notifier+0xe/0x126
                          [<c03a3f88>] 
register_netdevice_notifier+0x49/0x126
                          [<c05c5bda>] addrconf_init+0xad/0x193
                          [<c05c5b48>] addrconf_init+0x1b/0x193
                          [<c05c5a20>] inet6_init+0x1f0/0x2ad
                          [<c05a9499>] kernel_init+0x150/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c0104baf>] kernel_thread_helper+0x7/0x10
                          [<ffffffff>] 0xffffffff
    hardirq-on-W at:
                          [<c0141986>] mark_lock+0x64/0x451
                          [<c014286a>] __lock_acquire+0x488/0xc07
                          [<c0143062>] lock_acquire+0x79/0x93
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c0439d62>] _spin_lock_bh+0x3b/0x64
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c0412114>] igmp6_group_added+0x56/0x11d
                          [<c04124a8>] ipv6_dev_mc_inc+0x2a3/0x31c
                          [<c0410100>] igmp6_mc_seq_start+0xde/0x138
                          [<c04124dd>] ipv6_dev_mc_inc+0x2d8/0x31c
                          [<c0412205>] ipv6_dev_mc_inc+0x0/0x31c
                          [<c0401834>] ipv6_add_dev+0x21c/0x24b
                          [<c040b07d>] 
ndisc_ifinfo_sysctl_change+0x0/0x1ef
                          [<c0401e17>] addrconf_notify+0x60/0x7b7
                          [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                          [<c0141dac>] mark_held_locks+0x39/0x53
                          [<c0439066>] mutex_lock_nested+0x286/0x2ac
                          [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                          [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
                          [<c03a3f4d>] 
register_netdevice_notifier+0xe/0x126
                          [<c03a3f4d>] 
register_netdevice_notifier+0xe/0x126
                          [<c03a3f88>] 
register_netdevice_notifier+0x49/0x126
                          [<c05c5bda>] addrconf_init+0xad/0x193
                          [<c05c5b48>] addrconf_init+0x1b/0x193
                          [<c05c5a20>] inet6_init+0x1f0/0x2ad
                          [<c05a9499>] kernel_init+0x150/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c0104baf>] kernel_thread_helper+0x7/0x10
                          [<ffffffff>] 0xffffffff
  }
  ... key      at: [<c0877804>] bonding_netdev_xmit_lock_key+0x0/0x8
  -> (&bond->lock){-.--} ops: 99 {
     initial-use  at:
                            [<c013fe35>] put_lock_stats+0xa/0x1e
                            [<c014289c>] __lock_acquire+0x4ba/0xc07
                            [<c0141dac>] mark_held_locks+0x39/0x53
                            [<c043a3c5>] _spin_unlock_irqrestore+0x40/0x58
                            [<c0143062>] lock_acquire+0x79/0x93
                            [<c02edcc1>] bond_get_stats+0x28/0xd0
                            [<c0439eee>] _read_lock_bh+0x3b/0x64
                            [<c02edcc1>] bond_get_stats+0x28/0xd0
                            [<c02edcc1>] bond_get_stats+0x28/0xd0
                            [<c03aa3e7>] rtnl_fill_ifinfo+0x2bf/0x563
                            [<c03aa965>] rtmsg_ifinfo+0x5d/0xdf
                            [<c03aaa26>] rtnetlink_event+0x3f/0x42
                            [<c013ac20>] notifier_call_chain+0x2a/0x52
                            [<c013ac6a>] raw_notifier_call_chain+0x17/0x1a
                            [<c03a31de>] register_netdevice+0x2a7/0x2e7
                            [<c02ed862>] bond_create+0x1f2/0x26a
                            [<c05bedcd>] bonding_init+0x761/0x7ea
                            [<c05be635>] e1000_init_module+0x45/0x7c
                            [<c05a9499>] kernel_init+0x150/0x2b7
                            [<c05a9349>] kernel_init+0x0/0x2b7
                            [<c05a9349>] kernel_init+0x0/0x2b7
                            [<c0104baf>] kernel_thread_helper+0x7/0x10
                            [<ffffffff>] 0xffffffff
     hardirq-on-W at:
                            [<c014286a>] __lock_acquire+0x488/0xc07
                            [<c012093c>] try_to_wake_up+0x2ce/0x2d8
                            [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                            [<c0143062>] lock_acquire+0x79/0x93
                            [<c02eda75>] 
bond_set_multicast_list+0x1d/0x241
                            [<c0439e25>] _write_lock_bh+0x3b/0x64
                            [<c02eda75>] 
bond_set_multicast_list+0x1d/0x241
                            [<c02eda75>] 
bond_set_multicast_list+0x1d/0x241
                            [<c013fe35>] put_lock_stats+0xa/0x1e
                            [<c03a14db>] __dev_set_rx_mode+0x7b/0x7d
                            [<c03a1675>] dev_set_rx_mode+0x23/0x36
                            [<c03a3d50>] dev_open+0x5e/0x77
                            [<c03a2a1f>] dev_change_flags+0x9d/0x14b
                            [<c03a1823>] __dev_get_by_name+0x68/0x73
                            [<c03e3850>] devinet_ioctl+0x22b/0x536
                            [<c03a3b45>] dev_ioctl+0x46f/0x5b7
                            [<c0399c78>] sock_ioctl+0x167/0x18b
                            [<c0399b11>] sock_ioctl+0x0/0x18b
                            [<c01725f7>] do_ioctl+0x1f/0x62
                            [<c0172867>] vfs_ioctl+0x22d/0x23f
                            [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                            [<c01728ac>] sys_ioctl+0x33/0x4b
                            [<c0103e92>] sysenter_past_esp+0x5f/0xa5
                            [<ffffffff>] 0xffffffff
     softirq-on-R at:
                            [<c0141986>] mark_lock+0x64/0x451
                            [<c013575e>] __kernel_text_address+0x5/0xe
                            [<c0104ee2>] dump_trace+0x83/0x8d
                            [<c0142889>] __lock_acquire+0x4a7/0xc07
                            [<c013fc76>] save_trace+0x37/0x89
                            [<c0133c57>] run_workqueue+0x87/0x1b6
                            [<c0143062>] lock_acquire+0x79/0x93
                            [<c02eee5d>] bond_mii_monitor+0x19/0x85
                            [<c0439f4d>] _read_lock+0x36/0x5f
                            [<c02eee5d>] bond_mii_monitor+0x19/0x85
                            [<c02eee5d>] bond_mii_monitor+0x19/0x85
                            [<c0133cab>] run_workqueue+0xdb/0x1b6
                            [<c0133c57>] run_workqueue+0x87/0x1b6
                            [<c02eee44>] bond_mii_monitor+0x0/0x85
                            [<c01346cb>] worker_thread+0x0/0x85
                            [<c0134744>] worker_thread+0x79/0x85
                            [<c0137179>] autoremove_wake_function+0x0/0x35
                            [<c01370c2>] kthread+0x38/0x5e
                            [<c013708a>] kthread+0x0/0x5e
                            [<c0104baf>] kernel_thread_helper+0x7/0x10
                            [<ffffffff>] 0xffffffff
     hardirq-on-R at:
                            [<c013fe0a>] get_lock_stats+0xd/0x2e
                            [<c013fe35>] put_lock_stats+0xa/0x1e
                            [<c0142844>] __lock_acquire+0x462/0xc07
                            [<c0141dac>] mark_held_locks+0x39/0x53
                            [<c043a3c5>] _spin_unlock_irqrestore+0x40/0x58
                            [<c0143062>] lock_acquire+0x79/0x93
                            [<c02edcc1>] bond_get_stats+0x28/0xd0
                            [<c0439eee>] _read_lock_bh+0x3b/0x64
                            [<c02edcc1>] bond_get_stats+0x28/0xd0
                            [<c02edcc1>] bond_get_stats+0x28/0xd0
                            [<c03aa3e7>] rtnl_fill_ifinfo+0x2bf/0x563
                            [<c03aa965>] rtmsg_ifinfo+0x5d/0xdf
                            [<c03aaa26>] rtnetlink_event+0x3f/0x42
                            [<c013ac20>] notifier_call_chain+0x2a/0x52
                            [<c013ac6a>] raw_notifier_call_chain+0x17/0x1a
                            [<c03a31de>] register_netdevice+0x2a7/0x2e7
                            [<c02ed862>] bond_create+0x1f2/0x26a
                            [<c05bedcd>] bonding_init+0x761/0x7ea
                            [<c05be635>] e1000_init_module+0x45/0x7c
                            [<c05a9499>] kernel_init+0x150/0x2b7
                            [<c05a9349>] kernel_init+0x0/0x2b7
                            [<c05a9349>] kernel_init+0x0/0x2b7
                            [<c0104baf>] kernel_thread_helper+0x7/0x10
                            [<ffffffff>] 0xffffffff
   }
   ... key      at: [<c08777d0>] __key.32969+0x0/0x8
   -> (_xmit_ETHER){-...} ops: 8 {
      initial-use  at:
                              [<c014289c>] __lock_acquire+0x4ba/0xc07
                              [<c0143062>] lock_acquire+0x79/0x93
                              [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                              [<c0439d62>] _spin_lock_bh+0x3b/0x64
                              [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                              [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                              [<c0412114>] igmp6_group_added+0x56/0x11d
                              [<c04124a8>] ipv6_dev_mc_inc+0x2a3/0x31c
                              [<c0410100>] igmp6_mc_seq_start+0xde/0x138
                              [<c04124dd>] ipv6_dev_mc_inc+0x2d8/0x31c
                              [<c0412205>] ipv6_dev_mc_inc+0x0/0x31c
                              [<c0401834>] ipv6_add_dev+0x21c/0x24b
                              [<c040b07d>] 
ndisc_ifinfo_sysctl_change+0x0/0x1ef
                              [<c0401e17>] addrconf_notify+0x60/0x7b7
                              [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                              [<c0141dac>] mark_held_locks+0x39/0x53
                              [<c0439066>] mutex_lock_nested+0x286/0x2ac
                              [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                              [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
                              [<c03a3f4d>] 
register_netdevice_notifier+0xe/0x126
                              [<c03a3f4d>] 
register_netdevice_notifier+0xe/0x126
                              [<c03a3f88>] 
register_netdevice_notifier+0x49/0x126
                              [<c05c5bda>] addrconf_init+0xad/0x193
                              [<c05c5b48>] addrconf_init+0x1b/0x193
                              [<c05c5a20>] inet6_init+0x1f0/0x2ad
                              [<c05a9499>] kernel_init+0x150/0x2b7
                              [<c05a9349>] kernel_init+0x0/0x2b7
                              [<c05a9349>] kernel_init+0x0/0x2b7
                              [<c0104baf>] kernel_thread_helper+0x7/0x10
                              [<ffffffff>] 0xffffffff
      hardirq-on-W at:
                              [<c0141986>] mark_lock+0x64/0x451
                              [<c014286a>] __lock_acquire+0x488/0xc07
                              [<c0143062>] lock_acquire+0x79/0x93
                              [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                              [<c0439d62>] _spin_lock_bh+0x3b/0x64
                              [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                              [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                              [<c0412114>] igmp6_group_added+0x56/0x11d
                              [<c04124a8>] ipv6_dev_mc_inc+0x2a3/0x31c
                              [<c0410100>] igmp6_mc_seq_start+0xde/0x138
                              [<c04124dd>] ipv6_dev_mc_inc+0x2d8/0x31c
                              [<c0412205>] ipv6_dev_mc_inc+0x0/0x31c
                              [<c0401834>] ipv6_add_dev+0x21c/0x24b
                              [<c040b07d>] 
ndisc_ifinfo_sysctl_change+0x0/0x1ef
                              [<c0401e17>] addrconf_notify+0x60/0x7b7
                              [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                              [<c0141dac>] mark_held_locks+0x39/0x53
                              [<c0439066>] mutex_lock_nested+0x286/0x2ac
                              [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                              [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
                              [<c03a3f4d>] 
register_netdevice_notifier+0xe/0x126
                              [<c03a3f4d>] 
register_netdevice_notifier+0xe/0x126
                              [<c03a3f88>] 
register_netdevice_notifier+0x49/0x126
                              [<c05c5bda>] addrconf_init+0xad/0x193
                              [<c05c5b48>] addrconf_init+0x1b/0x193
                              [<c05c5a20>] inet6_init+0x1f0/0x2ad
                              [<c05a9499>] kernel_init+0x150/0x2b7
                              [<c05a9349>] kernel_init+0x0/0x2b7
                              [<c05a9349>] kernel_init+0x0/0x2b7
                              [<c0104baf>] kernel_thread_helper+0x7/0x10
                              [<ffffffff>] 0xffffffff
    }
    ... key      at: [<c087adc8>] netdev_xmit_lock_key+0x8/0x1c0
   ... acquired at:
   [<c0142dff>] __lock_acquire+0xa1d/0xc07
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c043a3c5>] _spin_unlock_irqrestore+0x40/0x58
   [<c0109ef2>] save_stack_trace+0x20/0x3a
   [<c0143062>] lock_acquire+0x79/0x93
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c0439d62>] _spin_lock_bh+0x3b/0x64
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c02ee492>] bond_change_active_slave+0x1a9/0x3bf
   [<c02ec7c3>] bond_update_speed_duplex+0x26/0x65
   [<c02ee9af>] bond_select_active_slave+0x95/0xcd
   [<c02ed22b>] bond_compute_features+0x45/0x84
   [<c02ef9be>] bond_enslave+0x6a7/0x884
   [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
   [<c02f6197>] bonding_store_slaves+0x1ae/0x2fb
   [<c02f5fe9>] bonding_store_slaves+0x0/0x2fb
   [<c02ce8d7>] dev_attr_store+0x27/0x2c
   [<c019bcb9>] sysfs_write_file+0xad/0xe0
   [<c019bc0c>] sysfs_write_file+0x0/0xe0
   [<c0168ddc>] vfs_write+0x8a/0x10c
   [<c0118566>] do_page_fault+0x0/0x54a
   [<c0169361>] sys_write+0x41/0x67
   [<c0103e92>] sysenter_past_esp+0x5f/0xa5
   [<ffffffff>] 0xffffffff

   -> (lweventlist_lock){.+..} ops: 10 {
      initial-use  at:
                              [<c0141986>] mark_lock+0x64/0x451
                              [<c014289c>] __lock_acquire+0x4ba/0xc07
                              [<c02e365c>] e1000_read_phy_reg+0x1c7/0x1d3
                              [<c02e348b>] e1000_write_phy_reg+0xb9/0xc3
                              [<c024a7de>] delay_tsc+0x25/0x3b
                              [<c0143062>] lock_acquire+0x79/0x93
                              [<c03abaf7>] linkwatch_add_event+0xd/0x2c
                              [<c043a07f>] _spin_lock_irqsave+0x3f/0x6c
                              [<c03abaf7>] linkwatch_add_event+0xd/0x2c
                              [<c03abaf7>] linkwatch_add_event+0xd/0x2c
                              [<c03abbbb>] linkwatch_fire_event+0x25/0x37
                              [<c02e1c43>] e1000_probe+0xad1/0xbe8
                              [<c0257f3f>] pci_device_probe+0x36/0x57
                              [<c02d0e5f>] driver_probe_device+0xe1/0x15f
                              [<c043a2f9>] _spin_unlock+0x25/0x3b
                              [<c04375b2>] klist_next+0x58/0x6d
                              [<c02d0f6f>] __driver_attach+0x0/0x7f
                              [<c02d0fb8>] __driver_attach+0x49/0x7f
                              [<c02d0403>] bus_for_each_dev+0x36/0x58
                              [<c02d0cb7>] driver_attach+0x16/0x18
                              [<c02d0f6f>] __driver_attach+0x0/0x7f
                              [<c02d06fa>] bus_add_driver+0x6d/0x18d
                              [<c0258089>] __pci_register_driver+0x53/0x7f
                              [<c05be635>] e1000_init_module+0x45/0x7c
                              [<c05a9499>] kernel_init+0x150/0x2b7
                              [<c05a9349>] kernel_init+0x0/0x2b7
                              [<c05a9349>] kernel_init+0x0/0x2b7
                              [<c0104baf>] kernel_thread_helper+0x7/0x10
                              [<ffffffff>] 0xffffffff
      in-softirq-W at:
                              [<c011d20a>] __wake_up_common+0x32/0x5c
                              [<c0142822>] __lock_acquire+0x440/0xc07
                              [<c043a3c5>] 
_spin_unlock_irqrestore+0x40/0x58
                              [<c0143062>] lock_acquire+0x79/0x93
                              [<c03abaf7>] linkwatch_add_event+0xd/0x2c
                              [<c02dff01>] e1000_watchdog+0x0/0x5c9
                              [<c043a07f>] _spin_lock_irqsave+0x3f/0x6c
                              [<c03abaf7>] linkwatch_add_event+0xd/0x2c
                              [<c03abaf7>] linkwatch_add_event+0xd/0x2c
                              [<c03abbbb>] linkwatch_fire_event+0x25/0x37
                              [<c03aeb42>] netif_carrier_on+0x16/0x27
                              [<c02e0156>] e1000_watchdog+0x255/0x5c9
                              [<c02dff01>] e1000_watchdog+0x0/0x5c9
                              [<c012df52>] run_timer_softirq+0xfa/0x15d
                              [<c012a8a6>] __do_softirq+0x56/0xdb
                              [<c0141f89>] trace_hardirqs_on+0x10c/0x14c
                              [<c012a8b8>] __do_softirq+0x68/0xdb
                              [<c012a961>] do_softirq+0x36/0x51
                              [<c012ab07>] irq_exit+0x43/0x4e
                              [<c0114122>] 
smp_apic_timer_interrupt+0x74/0x80
                              [<c0104a01>] apic_timer_interrupt+0x29/0x38
                              [<c0104a0b>] apic_timer_interrupt+0x33/0x38
                              [<c01600d8>] sys_swapon+0x29c/0x9aa
                              [<c01021a6>] mwait_idle_with_hints+0x3b/0x3f
                              [<c0102447>] mwait_idle+0x0/0xf
                              [<c0102581>] cpu_idle+0x99/0xc6
                              [<c05a98c7>] start_kernel+0x2c7/0x2cf
                              [<c05a90e0>] unknown_bootoption+0x0/0x195
                              [<ffffffff>] 0xffffffff
    }
    ... key      at: [<c058a194>] lweventlist_lock+0x14/0x40
   ... acquired at:
   [<c0142dff>] __lock_acquire+0xa1d/0xc07
   [<c03abaf7>] linkwatch_add_event+0xd/0x2c
   [<c0142fa1>] __lock_acquire+0xbbf/0xc07
   [<c0143062>] lock_acquire+0x79/0x93
   [<c03abaf7>] linkwatch_add_event+0xd/0x2c
   [<c043a07f>] _spin_lock_irqsave+0x3f/0x6c
   [<c03abaf7>] linkwatch_add_event+0xd/0x2c
   [<c03abaf7>] linkwatch_add_event+0xd/0x2c
   [<c03abbbb>] linkwatch_fire_event+0x25/0x37
   [<c03aeb42>] netif_carrier_on+0x16/0x27
   [<c02ede2c>] bond_set_carrier+0x31/0x55
   [<c02ee9b6>] bond_select_active_slave+0x9c/0xcd
   [<c02ed22b>] bond_compute_features+0x45/0x84
   [<c02ef9be>] bond_enslave+0x6a7/0x884
   [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
   [<c02f6197>] bonding_store_slaves+0x1ae/0x2fb
   [<c02f5fe9>] bonding_store_slaves+0x0/0x2fb
   [<c02ce8d7>] dev_attr_store+0x27/0x2c
   [<c019bcb9>] sysfs_write_file+0xad/0xe0
   [<c019bc0c>] sysfs_write_file+0x0/0xe0
   [<c0168ddc>] vfs_write+0x8a/0x10c
   [<c0118566>] do_page_fault+0x0/0x54a
   [<c0169361>] sys_write+0x41/0x67
   [<c0103e92>] sysenter_past_esp+0x5f/0xa5
   [<ffffffff>] 0xffffffff

... acquired at:

   [<c0142dff>] __lock_acquire+0xa1d/0xc07
   [<c02eda75>] bond_set_multicast_list+0x1d/0x241
   [<c0143062>] lock_acquire+0x79/0x93
   [<c02eda75>] bond_set_multicast_list+0x1d/0x241
   [<c0439e25>] _write_lock_bh+0x3b/0x64
   [<c02eda75>] bond_set_multicast_list+0x1d/0x241
   [<c02eda75>] bond_set_multicast_list+0x1d/0x241
   [<c013fe35>] put_lock_stats+0xa/0x1e
   [<c03a14db>] __dev_set_rx_mode+0x7b/0x7d
   [<c03a1675>] dev_set_rx_mode+0x23/0x36
   [<c03a3d50>] dev_open+0x5e/0x77
   [<c03a2a1f>] dev_change_flags+0x9d/0x14b
   [<c03a1823>] __dev_get_by_name+0x68/0x73
   [<c03e3850>] devinet_ioctl+0x22b/0x536
   [<c03a3b45>] dev_ioctl+0x46f/0x5b7
   [<c0399c78>] sock_ioctl+0x167/0x18b
   [<c0399b11>] sock_ioctl+0x0/0x18b
   [<c01725f7>] do_ioctl+0x1f/0x62
   [<c0172867>] vfs_ioctl+0x22d/0x23f
   [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
   [<c01728ac>] sys_ioctl+0x33/0x4b
   [<c0103e92>] sysenter_past_esp+0x5f/0xa5
   [<ffffffff>] 0xffffffff

... acquired at:

   [<c0142dff>] __lock_acquire+0xa1d/0xc07
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c0143062>] lock_acquire+0x79/0x93
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c0439d62>] _spin_lock_bh+0x3b/0x64
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c0412114>] igmp6_group_added+0x56/0x11d
   [<c04124a8>] ipv6_dev_mc_inc+0x2a3/0x31c
   [<c0410100>] igmp6_mc_seq_start+0xde/0x138
   [<c04124dd>] ipv6_dev_mc_inc+0x2d8/0x31c
   [<c0412205>] ipv6_dev_mc_inc+0x0/0x31c
   [<c0401834>] ipv6_add_dev+0x21c/0x24b
   [<c040b07d>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
   [<c0401e17>] addrconf_notify+0x60/0x7b7
   [<c0142fa1>] __lock_acquire+0xbbf/0xc07
   [<c0141dac>] mark_held_locks+0x39/0x53
   [<c0439066>] mutex_lock_nested+0x286/0x2ac
   [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
   [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
   [<c03a3f4d>] register_netdevice_notifier+0xe/0x126
   [<c03a3f4d>] register_netdevice_notifier+0xe/0x126
   [<c03a3f88>] register_netdevice_notifier+0x49/0x126
   [<c05c5bda>] addrconf_init+0xad/0x193
   [<c05c5b48>] addrconf_init+0x1b/0x193
   [<c05c5a20>] inet6_init+0x1f0/0x2ad
   [<c05a9499>] kernel_init+0x150/0x2b7
   [<c05a9349>] kernel_init+0x0/0x2b7
   [<c05a9349>] kernel_init+0x0/0x2b7
   [<c0104baf>] kernel_thread_helper+0x7/0x10
   [<ffffffff>] 0xffffffff

the second lock's dependencies:
-> (&bond->lock){-.--} ops: 99 {
initial-use at:

                        [<c013fe35>] put_lock_stats+0xa/0x1e
                        [<c014289c>] __lock_acquire+0x4ba/0xc07
                        [<c0141dac>] mark_held_locks+0x39/0x53
                        [<c043a3c5>] _spin_unlock_irqrestore+0x40/0x58
                        [<c0143062>] lock_acquire+0x79/0x93
                        [<c02edcc1>] bond_get_stats+0x28/0xd0
                        [<c0439eee>] _read_lock_bh+0x3b/0x64
                        [<c02edcc1>] bond_get_stats+0x28/0xd0
                        [<c02edcc1>] bond_get_stats+0x28/0xd0
                        [<c03aa3e7>] rtnl_fill_ifinfo+0x2bf/0x563
                        [<c03aa965>] rtmsg_ifinfo+0x5d/0xdf
                        [<c03aaa26>] rtnetlink_event+0x3f/0x42
                        [<c013ac20>] notifier_call_chain+0x2a/0x52
                        [<c013ac6a>] raw_notifier_call_chain+0x17/0x1a
                        [<c03a31de>] register_netdevice+0x2a7/0x2e7
                        [<c02ed862>] bond_create+0x1f2/0x26a
                        [<c05bedcd>] bonding_init+0x761/0x7ea
                        [<c05be635>] e1000_init_module+0x45/0x7c
                        [<c05a9499>] kernel_init+0x150/0x2b7
                        [<c05a9349>] kernel_init+0x0/0x2b7
                        [<c05a9349>] kernel_init+0x0/0x2b7
                        [<c0104baf>] kernel_thread_helper+0x7/0x10
                        [<ffffffff>] 0xffffffff
   hardirq-on-W at:
                        [<c014286a>] __lock_acquire+0x488/0xc07
                        [<c012093c>] try_to_wake_up+0x2ce/0x2d8
                        [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                        [<c0143062>] lock_acquire+0x79/0x93
                        [<c02eda75>] bond_set_multicast_list+0x1d/0x241
                        [<c0439e25>] _write_lock_bh+0x3b/0x64
                        [<c02eda75>] bond_set_multicast_list+0x1d/0x241
                        [<c02eda75>] bond_set_multicast_list+0x1d/0x241
                        [<c013fe35>] put_lock_stats+0xa/0x1e
                        [<c03a14db>] __dev_set_rx_mode+0x7b/0x7d
                        [<c03a1675>] dev_set_rx_mode+0x23/0x36
                        [<c03a3d50>] dev_open+0x5e/0x77
                        [<c03a2a1f>] dev_change_flags+0x9d/0x14b
                        [<c03a1823>] __dev_get_by_name+0x68/0x73
                        [<c03e3850>] devinet_ioctl+0x22b/0x536
                        [<c03a3b45>] dev_ioctl+0x46f/0x5b7
                        [<c0399c78>] sock_ioctl+0x167/0x18b
                        [<c0399b11>] sock_ioctl+0x0/0x18b
                        [<c01725f7>] do_ioctl+0x1f/0x62
                        [<c0172867>] vfs_ioctl+0x22d/0x23f
                        [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                        [<c01728ac>] sys_ioctl+0x33/0x4b
                        [<c0103e92>] sysenter_past_esp+0x5f/0xa5
                        [<ffffffff>] 0xffffffff
   softirq-on-R at:
                        [<c0141986>] mark_lock+0x64/0x451
                        [<c013575e>] __kernel_text_address+0x5/0xe
                        [<c0104ee2>] dump_trace+0x83/0x8d
                        [<c0142889>] __lock_acquire+0x4a7/0xc07
                        [<c013fc76>] save_trace+0x37/0x89
                        [<c0133c57>] run_workqueue+0x87/0x1b6
                        [<c0143062>] lock_acquire+0x79/0x93
                        [<c02eee5d>] bond_mii_monitor+0x19/0x85
                        [<c0439f4d>] _read_lock+0x36/0x5f
                        [<c02eee5d>] bond_mii_monitor+0x19/0x85
                        [<c02eee5d>] bond_mii_monitor+0x19/0x85
                        [<c0133cab>] run_workqueue+0xdb/0x1b6
                        [<c0133c57>] run_workqueue+0x87/0x1b6
                        [<c02eee44>] bond_mii_monitor+0x0/0x85
                        [<c01346cb>] worker_thread+0x0/0x85
                        [<c0134744>] worker_thread+0x79/0x85
                        [<c0137179>] autoremove_wake_function+0x0/0x35
                        [<c01370c2>] kthread+0x38/0x5e
                        [<c013708a>] kthread+0x0/0x5e
                        [<c0104baf>] kernel_thread_helper+0x7/0x10
                        [<ffffffff>] 0xffffffff
   hardirq-on-R at:
                        [<c013fe0a>] get_lock_stats+0xd/0x2e
                        [<c013fe35>] put_lock_stats+0xa/0x1e
                        [<c0142844>] __lock_acquire+0x462/0xc07
                        [<c0141dac>] mark_held_locks+0x39/0x53
                        [<c043a3c5>] _spin_unlock_irqrestore+0x40/0x58
                        [<c0143062>] lock_acquire+0x79/0x93
                        [<c02edcc1>] bond_get_stats+0x28/0xd0
                        [<c0439eee>] _read_lock_bh+0x3b/0x64
                        [<c02edcc1>] bond_get_stats+0x28/0xd0
                        [<c02edcc1>] bond_get_stats+0x28/0xd0
                        [<c03aa3e7>] rtnl_fill_ifinfo+0x2bf/0x563
                        [<c03aa965>] rtmsg_ifinfo+0x5d/0xdf
                        [<c03aaa26>] rtnetlink_event+0x3f/0x42
                        [<c013ac20>] notifier_call_chain+0x2a/0x52
                        [<c013ac6a>] raw_notifier_call_chain+0x17/0x1a
                        [<c03a31de>] register_netdevice+0x2a7/0x2e7
                        [<c02ed862>] bond_create+0x1f2/0x26a
                        [<c05bedcd>] bonding_init+0x761/0x7ea
                        [<c05be635>] e1000_init_module+0x45/0x7c
                        [<c05a9499>] kernel_init+0x150/0x2b7
                        [<c05a9349>] kernel_init+0x0/0x2b7
                        [<c05a9349>] kernel_init+0x0/0x2b7
                        [<c0104baf>] kernel_thread_helper+0x7/0x10
                        [<ffffffff>] 0xffffffff
 }
 ... key      at: [<c08777d0>] __key.32969+0x0/0x8
 -> (_xmit_ETHER){-...} ops: 8 {
    initial-use  at:
                          [<c014289c>] __lock_acquire+0x4ba/0xc07
                          [<c0143062>] lock_acquire+0x79/0x93
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c0439d62>] _spin_lock_bh+0x3b/0x64
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c0412114>] igmp6_group_added+0x56/0x11d
                          [<c04124a8>] ipv6_dev_mc_inc+0x2a3/0x31c
                          [<c0410100>] igmp6_mc_seq_start+0xde/0x138
                          [<c04124dd>] ipv6_dev_mc_inc+0x2d8/0x31c
                          [<c0412205>] ipv6_dev_mc_inc+0x0/0x31c
                          [<c0401834>] ipv6_add_dev+0x21c/0x24b
                          [<c040b07d>] 
ndisc_ifinfo_sysctl_change+0x0/0x1ef
                          [<c0401e17>] addrconf_notify+0x60/0x7b7
                          [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                          [<c0141dac>] mark_held_locks+0x39/0x53
                          [<c0439066>] mutex_lock_nested+0x286/0x2ac
                          [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                          [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
                          [<c03a3f4d>] 
register_netdevice_notifier+0xe/0x126
                          [<c03a3f4d>] 
register_netdevice_notifier+0xe/0x126
                          [<c03a3f88>] 
register_netdevice_notifier+0x49/0x126
                          [<c05c5bda>] addrconf_init+0xad/0x193
                          [<c05c5b48>] addrconf_init+0x1b/0x193
                          [<c05c5a20>] inet6_init+0x1f0/0x2ad
                          [<c05a9499>] kernel_init+0x150/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c0104baf>] kernel_thread_helper+0x7/0x10
                          [<ffffffff>] 0xffffffff
    hardirq-on-W at:
                          [<c0141986>] mark_lock+0x64/0x451
                          [<c014286a>] __lock_acquire+0x488/0xc07
                          [<c0143062>] lock_acquire+0x79/0x93
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c0439d62>] _spin_lock_bh+0x3b/0x64
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c0412114>] igmp6_group_added+0x56/0x11d
                          [<c04124a8>] ipv6_dev_mc_inc+0x2a3/0x31c
                          [<c0410100>] igmp6_mc_seq_start+0xde/0x138
                          [<c04124dd>] ipv6_dev_mc_inc+0x2d8/0x31c
                          [<c0412205>] ipv6_dev_mc_inc+0x0/0x31c
                          [<c0401834>] ipv6_add_dev+0x21c/0x24b
                          [<c040b07d>] 
ndisc_ifinfo_sysctl_change+0x0/0x1ef
                          [<c0401e17>] addrconf_notify+0x60/0x7b7
                          [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                          [<c0141dac>] mark_held_locks+0x39/0x53
                          [<c0439066>] mutex_lock_nested+0x286/0x2ac
                          [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                          [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
                          [<c03a3f4d>] 
register_netdevice_notifier+0xe/0x126
                          [<c03a3f4d>] 
register_netdevice_notifier+0xe/0x126
                          [<c03a3f88>] 
register_netdevice_notifier+0x49/0x126
                          [<c05c5bda>] addrconf_init+0xad/0x193
                          [<c05c5b48>] addrconf_init+0x1b/0x193
                          [<c05c5a20>] inet6_init+0x1f0/0x2ad
                          [<c05a9499>] kernel_init+0x150/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c0104baf>] kernel_thread_helper+0x7/0x10
                          [<ffffffff>] 0xffffffff
  }
  ... key      at: [<c087adc8>] netdev_xmit_lock_key+0x8/0x1c0
 ... acquired at:
   [<c0142dff>] __lock_acquire+0xa1d/0xc07
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c043a3c5>] _spin_unlock_irqrestore+0x40/0x58
   [<c0109ef2>] save_stack_trace+0x20/0x3a
   [<c0143062>] lock_acquire+0x79/0x93
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c0439d62>] _spin_lock_bh+0x3b/0x64
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c02ee492>] bond_change_active_slave+0x1a9/0x3bf
   [<c02ec7c3>] bond_update_speed_duplex+0x26/0x65
   [<c02ee9af>] bond_select_active_slave+0x95/0xcd
   [<c02ed22b>] bond_compute_features+0x45/0x84
   [<c02ef9be>] bond_enslave+0x6a7/0x884
   [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
   [<c02f6197>] bonding_store_slaves+0x1ae/0x2fb
   [<c02f5fe9>] bonding_store_slaves+0x0/0x2fb
   [<c02ce8d7>] dev_attr_store+0x27/0x2c
   [<c019bcb9>] sysfs_write_file+0xad/0xe0
   [<c019bc0c>] sysfs_write_file+0x0/0xe0
   [<c0168ddc>] vfs_write+0x8a/0x10c
   [<c0118566>] do_page_fault+0x0/0x54a
   [<c0169361>] sys_write+0x41/0x67
   [<c0103e92>] sysenter_past_esp+0x5f/0xa5
   [<ffffffff>] 0xffffffff

 -> (lweventlist_lock){.+..} ops: 10 {
    initial-use  at:
                          [<c0141986>] mark_lock+0x64/0x451
                          [<c014289c>] __lock_acquire+0x4ba/0xc07
                          [<c02e365c>] e1000_read_phy_reg+0x1c7/0x1d3
                          [<c02e348b>] e1000_write_phy_reg+0xb9/0xc3
                          [<c024a7de>] delay_tsc+0x25/0x3b
                          [<c0143062>] lock_acquire+0x79/0x93
                          [<c03abaf7>] linkwatch_add_event+0xd/0x2c
                          [<c043a07f>] _spin_lock_irqsave+0x3f/0x6c
                          [<c03abaf7>] linkwatch_add_event+0xd/0x2c
                          [<c03abaf7>] linkwatch_add_event+0xd/0x2c
                          [<c03abbbb>] linkwatch_fire_event+0x25/0x37
                          [<c02e1c43>] e1000_probe+0xad1/0xbe8
                          [<c0257f3f>] pci_device_probe+0x36/0x57
                          [<c02d0e5f>] driver_probe_device+0xe1/0x15f
                          [<c043a2f9>] _spin_unlock+0x25/0x3b
                          [<c04375b2>] klist_next+0x58/0x6d
                          [<c02d0f6f>] __driver_attach+0x0/0x7f
                          [<c02d0fb8>] __driver_attach+0x49/0x7f
                          [<c02d0403>] bus_for_each_dev+0x36/0x58
                          [<c02d0cb7>] driver_attach+0x16/0x18
                          [<c02d0f6f>] __driver_attach+0x0/0x7f
                          [<c02d06fa>] bus_add_driver+0x6d/0x18d
                          [<c0258089>] __pci_register_driver+0x53/0x7f
                          [<c05be635>] e1000_init_module+0x45/0x7c
                          [<c05a9499>] kernel_init+0x150/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c0104baf>] kernel_thread_helper+0x7/0x10
                          [<ffffffff>] 0xffffffff
    in-softirq-W at:
                          [<c011d20a>] __wake_up_common+0x32/0x5c
                          [<c0142822>] __lock_acquire+0x440/0xc07
                          [<c043a3c5>] _spin_unlock_irqrestore+0x40/0x58
                          [<c0143062>] lock_acquire+0x79/0x93
                          [<c03abaf7>] linkwatch_add_event+0xd/0x2c
                          [<c02dff01>] e1000_watchdog+0x0/0x5c9
                          [<c043a07f>] _spin_lock_irqsave+0x3f/0x6c
                          [<c03abaf7>] linkwatch_add_event+0xd/0x2c
                          [<c03abaf7>] linkwatch_add_event+0xd/0x2c
                          [<c03abbbb>] linkwatch_fire_event+0x25/0x37
                          [<c03aeb42>] netif_carrier_on+0x16/0x27
                          [<c02e0156>] e1000_watchdog+0x255/0x5c9
                          [<c02dff01>] e1000_watchdog+0x0/0x5c9
                          [<c012df52>] run_timer_softirq+0xfa/0x15d
                          [<c012a8a6>] __do_softirq+0x56/0xdb
                          [<c0141f89>] trace_hardirqs_on+0x10c/0x14c
                          [<c012a8b8>] __do_softirq+0x68/0xdb
                          [<c012a961>] do_softirq+0x36/0x51
                          [<c012ab07>] irq_exit+0x43/0x4e
                          [<c0114122>] smp_apic_timer_interrupt+0x74/0x80
                          [<c0104a01>] apic_timer_interrupt+0x29/0x38
                          [<c0104a0b>] apic_timer_interrupt+0x33/0x38
                          [<c01600d8>] sys_swapon+0x29c/0x9aa
                          [<c01021a6>] mwait_idle_with_hints+0x3b/0x3f
                          [<c0102447>] mwait_idle+0x0/0xf
                          [<c0102581>] cpu_idle+0x99/0xc6
                          [<c05a98c7>] start_kernel+0x2c7/0x2cf
                          [<c05a90e0>] unknown_bootoption+0x0/0x195
                          [<ffffffff>] 0xffffffff
  }
  ... key      at: [<c058a194>] lweventlist_lock+0x14/0x40
 ... acquired at:
   [<c0142dff>] __lock_acquire+0xa1d/0xc07
   [<c03abaf7>] linkwatch_add_event+0xd/0x2c
   [<c0142fa1>] __lock_acquire+0xbbf/0xc07
   [<c0143062>] lock_acquire+0x79/0x93
   [<c03abaf7>] linkwatch_add_event+0xd/0x2c
   [<c043a07f>] _spin_lock_irqsave+0x3f/0x6c
   [<c03abaf7>] linkwatch_add_event+0xd/0x2c
   [<c03abaf7>] linkwatch_add_event+0xd/0x2c
   [<c03abbbb>] linkwatch_fire_event+0x25/0x37
   [<c03aeb42>] netif_carrier_on+0x16/0x27
   [<c02ede2c>] bond_set_carrier+0x31/0x55
   [<c02ee9b6>] bond_select_active_slave+0x9c/0xcd
   [<c02ed22b>] bond_compute_features+0x45/0x84
   [<c02ef9be>] bond_enslave+0x6a7/0x884
   [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
   [<c02f6197>] bonding_store_slaves+0x1ae/0x2fb
   [<c02f5fe9>] bonding_store_slaves+0x0/0x2fb
   [<c02ce8d7>] dev_attr_store+0x27/0x2c
   [<c019bcb9>] sysfs_write_file+0xad/0xe0
   [<c019bc0c>] sysfs_write_file+0x0/0xe0
   [<c0168ddc>] vfs_write+0x8a/0x10c
   [<c0118566>] do_page_fault+0x0/0x54a
   [<c0169361>] sys_write+0x41/0x67
   [<c0103e92>] sysenter_past_esp+0x5f/0xa5
   [<ffffffff>] 0xffffffff


stack backtrace:
Pid: 9, comm: events/0 Not tainted 2.6.24-rc5 #1
 [<c0140b38>] print_irq_inversion_bug+0x108/0x112
 [<c014191d>] check_usage_forwards+0x3c/0x41
 [<c0141b09>] mark_lock+0x1e7/0x451
 [<c0142822>] __lock_acquire+0x440/0xc07
 [<c013e0f3>] clockevents_program_event+0xe0/0xee
 [<c0143062>] lock_acquire+0x79/0x93
 [<c0411c7a>] mld_ifc_timer_expire+0x130/0x1fb
 [<c0411b4a>] mld_ifc_timer_expire+0x0/0x1fb
 [<c0439d62>] _spin_lock_bh+0x3b/0x64
 [<c0411c7a>] mld_ifc_timer_expire+0x130/0x1fb
 [<c0411c7a>] mld_ifc_timer_expire+0x130/0x1fb
 [<c0411b4a>] mld_ifc_timer_expire+0x0/0x1fb
 [<c0141f89>] trace_hardirqs_on+0x10c/0x14c
 [<c0411b4a>] mld_ifc_timer_expire+0x0/0x1fb
 [<c012df52>] run_timer_softirq+0xfa/0x15d
 [<c012a8a6>] __do_softirq+0x56/0xdb
 [<c0141f89>] trace_hardirqs_on+0x10c/0x14c
 [<c012a8b8>] __do_softirq+0x68/0xdb
 [<c012a961>] do_softirq+0x36/0x51
 [<c012ae4a>] local_bh_enable_ip+0xad/0xed
 [<c03bf107>] rt_run_flush+0x64/0x8b
 [<c03e9296>] fib_netdev_event+0x61/0x65
 [<c013ac20>] notifier_call_chain+0x2a/0x52
 [<c013ac6a>] raw_notifier_call_chain+0x17/0x1a
 [<c03a2ae5>] netdev_state_change+0x18/0x29
 [<c03abd1d>] __linkwatch_run_queue+0x150/0x17e
 [<c03abd68>] linkwatch_event+0x1d/0x22
 [<c0133cab>] run_workqueue+0xdb/0x1b6
 [<c0133c57>] run_workqueue+0x87/0x1b6
 [<c03abd4b>] linkwatch_event+0x0/0x22
 [<c01346cb>] worker_thread+0x0/0x85
 [<c0134744>] worker_thread+0x79/0x85
 [<c0137179>] autoremove_wake_function+0x0/0x35
 [<c01370c2>] kthread+0x38/0x5e
 [<c013708a>] kthread+0x0/0x5e
 [<c0104baf>] kernel_thread_helper+0x7/0x10
 =======================

Best regards,

Krzysztof Olędzki

Andy Gospodarek

2007-12-14 18:27 UTC #8 lore raw

Quoted 102 lines · Krzysztof Oledzki

Quoted 83 lines · Jay Vosburgh
Quoted 8 lines · Herbert Xu
Quoted 8 lines
diff -puN drivers/net/bonding/bond_sysfs.c~bonding-locking-fix drivers/net/bonding/bond_sysfs.c
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -0
@@ -1111,8 +1111,6 @@ static ssize_t bonding_store_primary(str
out:
      write_unlock_bh(&bond->lock);
Quoted 6 lines
drivers/net/bonding/bond_sysfs.c
Quoted 2 lines
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -2
-       rtnl_unlock();
-
Looking at the changeset that added this perhaps the intention is to hold the lock? If so we should add an rtnl_lock to the start of the function.
Yes, this function needs to hold locks, and more than just
what's there now. I believe the following should be correct; I haven't tested it, though (I'm supposedly on vacation right now).
The following change should be correct for the
bonding_store_primary case discussed in this thread, and also corrects the bonding_store_active case which performs similar functions.
The bond_change_active_slave and bond_select_active_slave
functions both require rtnl, bond->lock for read and curr_slave_lock for write_bh, and no other locks. This is so that the lower level mode-specific functions can release locks down to just rtnl in order to call, e.g., dev_set_mac_address with the locks it expects (rtnl only).
Signed-off-by Jay Vosburgh <[email protected]>
drivers/net/bonding/bond_sysfs.c +4 -1
index 11b76b3..28a2d80 100644
@@ -1075,7 +1075,10 @@ static ssize_t bonding_store_primary(struct device 
 *d,
 	struct slave *slave;
 	struct bonding *bond = to_bond(d);
 
-	write_lock_bh(&bond->lock);
+	rtnl_lock();
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
+
 	if (!USES_PRIMARY(bond->params.mode)) {
 		printk(KERN_INFO DRV_NAME
		       ": %s: Unable to set primary slave; %s is in mode 
		       %d\n",
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1109,8 +1112,8 @@ static ssize_t bonding_store_primary(struct device 
 *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +2 -2
-	write_unlock_bh(&bond->lock);
-
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
	rtnl_unlock();

	return count;
drivers/net/bonding/bond_sysfs.c +2 -1
@@ -1190,7 +1193,8 @@ static ssize_t bonding_store_active_slave(struct 
 device *d,
 	struct bonding *bond = to_bond(d);
 
 	rtnl_lock();
-	write_lock_bh(&bond->lock);
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
 
 	if (!USES_PRIMARY(bond->params.mode)) {
printk(KERN_INFO DRV_NAME
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1247,7 +1251,8 @@ static ssize_t bonding_store_active_slave(struct 
 device *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +2 -1
-	write_unlock_bh(&bond->lock);
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
Quoted 3 lines
	rtnl_unlock();

	return count;
Vanilla 2.6.24-rc5 plus this patch:
=========================================================
[ INFO: possible irq lock inversion dependency detected ] 2.6.24-rc5 #1
---------------------------------------------------------
events/0/9 just changed the state of lock:
(&mc->mca_lock){-+..}, at: [<c0411c7a>] mld_ifc_timer_expire+0x130/0x1fb
but this lock took another, soft-read-irq-unsafe lock in the past:
(&bond->lock){-.--}

Quoted 3 lines

and interrupts could create inverse lock ordering between them.

Grrr, I should have seen that -- sorry. Try your luck with this instead:

drivers/net/bonding/bond_sysfs.c +10 -5

index 11b76b3..0694254 100644
@@ -1075,7 +1075,10 @@ static ssize_t bonding_store_primary(struct device *d,
 	struct slave *slave;
 	struct bonding *bond = to_bond(d);
 
-	write_lock_bh(&bond->lock);
+	rtnl_lock();
+	read_lock_bh(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
+
 	if (!USES_PRIMARY(bond->params.mode)) {
 		printk(KERN_INFO DRV_NAME
 		       ": %s: Unable to set primary slave; %s is in mode %d\n",

@@ -1109,8 +1112,8 @@ static ssize_t bonding_store_primary(struct device *d,
 		}
 	}
 out:
-	write_unlock_bh(&bond->lock);
-
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock_bh(&bond->lock);
 	rtnl_unlock();
 
 	return count;

@@ -1190,7 +1193,8 @@ static ssize_t bonding_store_active_slave(struct device *d,
 	struct bonding *bond = to_bond(d);
 
 	rtnl_lock();
-	write_lock_bh(&bond->lock);
+	read_lock_bh(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
 
 	if (!USES_PRIMARY(bond->params.mode)) {
 		printk(KERN_INFO DRV_NAME

@@ -1247,7 +1251,8 @@ static ssize_t bonding_store_active_slave(struct device *d,
 		}
 	}
 out:
-	write_unlock_bh(&bond->lock);
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock_bh(&bond->lock);
 	rtnl_unlock();
 
 	return count;

Krzysztof Oledzki

2007-12-14 18:58 UTC #9 lore raw

Quoted 103 lines · Andy Gospodarek

Quoted 102 lines · Krzysztof Oledzki
Quoted 83 lines · Jay Vosburgh
Quoted 8 lines · Herbert Xu
Quoted 8 lines
diff -puN drivers/net/bonding/bond_sysfs.c~bonding-locking-fix drivers/net/bonding/bond_sysfs.c
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -0
@@ -1111,8 +1111,6 @@ static ssize_t bonding_store_primary(str
out:
      write_unlock_bh(&bond->lock);
Quoted 6 lines
drivers/net/bonding/bond_sysfs.c
Quoted 2 lines
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -2
-       rtnl_unlock();
-
Looking at the changeset that added this perhaps the intention is to hold the lock? If so we should add an rtnl_lock to the start of the function.
Yes, this function needs to hold locks, and more than just
what's there now. I believe the following should be correct; I haven't tested it, though (I'm supposedly on vacation right now).
The following change should be correct for the
bonding_store_primary case discussed in this thread, and also corrects the bonding_store_active case which performs similar functions.
The bond_change_active_slave and bond_select_active_slave
functions both require rtnl, bond->lock for read and curr_slave_lock for write_bh, and no other locks. This is so that the lower level mode-specific functions can release locks down to just rtnl in order to call, e.g., dev_set_mac_address with the locks it expects (rtnl only).
Signed-off-by Jay Vosburgh <[email protected]>
drivers/net/bonding/bond_sysfs.c +4 -1
index 11b76b3..28a2d80 100644
@@ -1075,7 +1075,10 @@ static ssize_t bonding_store_primary(struct device
 *d,
 	struct slave *slave;
 	struct bonding *bond = to_bond(d);
 
-	write_lock_bh(&bond->lock);
+	rtnl_lock();
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
+
 	if (!USES_PRIMARY(bond->params.mode)) {
 		printk(KERN_INFO DRV_NAME
		       ": %s: Unable to set primary slave; %s is in mode
		       %d\n",
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1109,8 +1112,8 @@ static ssize_t bonding_store_primary(struct device
 *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +2 -2
-	write_unlock_bh(&bond->lock);
-
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
	rtnl_unlock();

	return count;
drivers/net/bonding/bond_sysfs.c +2 -1
@@ -1190,7 +1193,8 @@ static ssize_t bonding_store_active_slave(struct
 device *d,
 	struct bonding *bond = to_bond(d);
 
 	rtnl_lock();
-	write_lock_bh(&bond->lock);
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
 
 	if (!USES_PRIMARY(bond->params.mode)) {
printk(KERN_INFO DRV_NAME
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1247,7 +1251,8 @@ static ssize_t bonding_store_active_slave(struct
 device *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +2 -1
-	write_unlock_bh(&bond->lock);
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
Quoted 3 lines
	rtnl_unlock();

	return count;
Vanilla 2.6.24-rc5 plus this patch:
=========================================================
[ INFO: possible irq lock inversion dependency detected ] 2.6.24-rc5 #1
---------------------------------------------------------
events/0/9 just changed the state of lock:
(&mc->mca_lock){-+..}, at: [<c0411c7a>] mld_ifc_timer_expire+0x130/0x1fb
but this lock took another, soft-read-irq-unsafe lock in the past:
(&bond->lock){-.--}

Quoted 5 lines

Quoted 3 lines
and interrupts could create inverse lock ordering between them.
Grrr, I should have seen that -- sorry. Try your luck with this instead:

<CUT>

No luck.

bonding: bond0: setting mode to active-backup (1).
bonding: bond0: Setting MII monitoring interval to 100. ADDRCONF(NETDEV_UP): bond0: link is not ready
bonding: bond0: Adding slave eth0.
e1000: eth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
bonding: bond0: making interface eth0 the new active one.
bonding: bond0: first active interface up!
bonding: bond0: enslaving eth0 as an active interface with an up link.
bonding: bond0: Adding slave eth1.
ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready

and interrupts could create inverse lock ordering between them.

other info that might help us debug this:
4 locks held by events/0/9:

 #0:  (events){--..}, at: [<c0133c57>] run_workqueue+0x87/0x1b6
 #1:  ((linkwatch_work).work){--..}, at: [<c0133c57>] run_workqueue+0x87/0x1b6
 #2:  (rtnl_mutex){--..}, at: [<c03abd50>] linkwatch_event+0x5/0x22
 #3:  (&ndev->lock){-.-+}, at: [<c0411b61>] mld_ifc_timer_expire+0x17/0x1fb

the first lock's dependencies:
-> (&mc->mca_lock){-+..} ops: 10 {
initial-use at:

                        [<c0104ee2>] dump_trace+0x83/0x8d
                        [<c014289c>] __lock_acquire+0x4ba/0xc07
                        [<c0109ef2>] save_stack_trace+0x20/0x3a
                        [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                        [<c0412452>] ipv6_dev_mc_inc+0x24d/0x31c
                        [<c0143062>] lock_acquire+0x79/0x93
                        [<c04120d6>] igmp6_group_added+0x18/0x11d
                        [<c0439d62>] _spin_lock_bh+0x3b/0x64
                        [<c04120d6>] igmp6_group_added+0x18/0x11d
                        [<c04120d6>] igmp6_group_added+0x18/0x11d
                        [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                        [<c04124a8>] ipv6_dev_mc_inc+0x2a3/0x31c
                        [<c0412452>] ipv6_dev_mc_inc+0x24d/0x31c
                        [<c04124dd>] ipv6_dev_mc_inc+0x2d8/0x31c
                        [<c0412205>] ipv6_dev_mc_inc+0x0/0x31c
                        [<c0401834>] ipv6_add_dev+0x21c/0x24b
                        [<c040b07d>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
                        [<c05c5b40>] addrconf_init+0x13/0x193
                        [<c0199f63>] proc_net_fops_create+0x10/0x21
                        [<c0419b38>] ip6_flowlabel_init+0x1e/0x20
                        [<c05c5a20>] inet6_init+0x1f0/0x2ad
                        [<c05a9499>] kernel_init+0x150/0x2b7
                        [<c05a9349>] kernel_init+0x0/0x2b7
                        [<c05a9349>] kernel_init+0x0/0x2b7
                        [<c0104baf>] kernel_thread_helper+0x7/0x10
                        [<ffffffff>] 0xffffffff
   in-softirq-W at:
                        [<c0142822>] __lock_acquire+0x440/0xc07
                        [<c013fe0a>] get_lock_stats+0xd/0x2e
                        [<c013fe35>] put_lock_stats+0xa/0x1e
                        [<c0143062>] lock_acquire+0x79/0x93
                        [<c0411c7a>] mld_ifc_timer_expire+0x130/0x1fb
                        [<c0411b4a>] mld_ifc_timer_expire+0x0/0x1fb
                        [<c0439d62>] _spin_lock_bh+0x3b/0x64
                        [<c0411c7a>] mld_ifc_timer_expire+0x130/0x1fb
                        [<c0411c7a>] mld_ifc_timer_expire+0x130/0x1fb
                        [<c0411b4a>] mld_ifc_timer_expire+0x0/0x1fb
                        [<c0141f89>] trace_hardirqs_on+0x10c/0x14c
                        [<c0411b4a>] mld_ifc_timer_expire+0x0/0x1fb
                        [<c012df52>] run_timer_softirq+0xfa/0x15d
                        [<c012a8a6>] __do_softirq+0x56/0xdb
                        [<c0141f89>] trace_hardirqs_on+0x10c/0x14c
                        [<c012a8b8>] __do_softirq+0x68/0xdb
                        [<c012a961>] do_softirq+0x36/0x51
                        [<c012ae4a>] local_bh_enable_ip+0xad/0xed
                        [<c03bf107>] rt_run_flush+0x64/0x8b
                        [<c03e9296>] fib_netdev_event+0x61/0x65
                        [<c013ac20>] notifier_call_chain+0x2a/0x52
                        [<c013ac6a>] raw_notifier_call_chain+0x17/0x1a
                        [<c03a2ae5>] netdev_state_change+0x18/0x29
                        [<c03abd1d>] __linkwatch_run_queue+0x150/0x17e
                        [<c03abd68>] linkwatch_event+0x1d/0x22
                        [<c0133cab>] run_workqueue+0xdb/0x1b6
                        [<c0133c57>] run_workqueue+0x87/0x1b6
                        [<c03abd4b>] linkwatch_event+0x0/0x22
                        [<c01346cb>] worker_thread+0x0/0x85
                        [<c0134744>] worker_thread+0x79/0x85
                        [<c0137179>] autoremove_wake_function+0x0/0x35
                        [<c01370c2>] kthread+0x38/0x5e
                        [<c013708a>] kthread+0x0/0x5e
                        [<c0104baf>] kernel_thread_helper+0x7/0x10
                        [<ffffffff>] 0xffffffff
   hardirq-on-W at:
                        [<c01417ee>] find_usage_backwards+0xbb/0xe2
                        [<c0104ee2>] dump_trace+0x83/0x8d
                        [<c014286a>] __lock_acquire+0x488/0xc07
                        [<c0109ef2>] save_stack_trace+0x20/0x3a
                        [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                        [<c0412452>] ipv6_dev_mc_inc+0x24d/0x31c
                        [<c0143062>] lock_acquire+0x79/0x93
                        [<c04120d6>] igmp6_group_added+0x18/0x11d
                        [<c0439d62>] _spin_lock_bh+0x3b/0x64
                        [<c04120d6>] igmp6_group_added+0x18/0x11d
                        [<c04120d6>] igmp6_group_added+0x18/0x11d
                        [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                        [<c04124a8>] ipv6_dev_mc_inc+0x2a3/0x31c
                        [<c0412452>] ipv6_dev_mc_inc+0x24d/0x31c
                        [<c04124dd>] ipv6_dev_mc_inc+0x2d8/0x31c
                        [<c0412205>] ipv6_dev_mc_inc+0x0/0x31c
                        [<c0401834>] ipv6_add_dev+0x21c/0x24b
                        [<c040b07d>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
                        [<c05c5b40>] addrconf_init+0x13/0x193
                        [<c0199f63>] proc_net_fops_create+0x10/0x21
                        [<c0419b38>] ip6_flowlabel_init+0x1e/0x20
                        [<c05c5a20>] inet6_init+0x1f0/0x2ad
                        [<c05a9499>] kernel_init+0x150/0x2b7
                        [<c05a9349>] kernel_init+0x0/0x2b7
                        [<c05a9349>] kernel_init+0x0/0x2b7
                        [<c0104baf>] kernel_thread_helper+0x7/0x10
                        [<ffffffff>] 0xffffffff
 }
 ... key      at: [<c087e2d8>] __key.30798+0x0/0x8
 -> (_xmit_ETHER){-...} ops: 8 {
    initial-use  at:
                          [<c014289c>] __lock_acquire+0x4ba/0xc07
                          [<c0143062>] lock_acquire+0x79/0x93
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c0439d62>] _spin_lock_bh+0x3b/0x64
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c0412114>] igmp6_group_added+0x56/0x11d
                          [<c04124a8>] ipv6_dev_mc_inc+0x2a3/0x31c
                          [<c0410100>] igmp6_mc_seq_start+0xde/0x138
                          [<c04124dd>] ipv6_dev_mc_inc+0x2d8/0x31c
                          [<c0412205>] ipv6_dev_mc_inc+0x0/0x31c
                          [<c0401834>] ipv6_add_dev+0x21c/0x24b
                          [<c040b07d>] 
ndisc_ifinfo_sysctl_change+0x0/0x1ef
                          [<c0401e17>] addrconf_notify+0x60/0x7b7
                          [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                          [<c0141dac>] mark_held_locks+0x39/0x53
                          [<c0439066>] mutex_lock_nested+0x286/0x2ac
                          [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                          [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
                          [<c03a3f4d>] register_netdevice_notifier+0xe/0x126
                          [<c03a3f4d>] register_netdevice_notifier+0xe/0x126
                          [<c03a3f88>] register_netdevice_notifier+0x49/0x126
                          [<c05c5bda>] addrconf_init+0xad/0x193
                          [<c05c5b48>] addrconf_init+0x1b/0x193
                          [<c05c5a20>] inet6_init+0x1f0/0x2ad
                          [<c05a9499>] kernel_init+0x150/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c0104baf>] kernel_thread_helper+0x7/0x10
                          [<ffffffff>] 0xffffffff
    hardirq-on-W at:
                          [<c0141986>] mark_lock+0x64/0x451
                          [<c014286a>] __lock_acquire+0x488/0xc07
                          [<c0143062>] lock_acquire+0x79/0x93
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c0439d62>] _spin_lock_bh+0x3b/0x64
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c0412114>] igmp6_group_added+0x56/0x11d
                          [<c04124a8>] ipv6_dev_mc_inc+0x2a3/0x31c
                          [<c0410100>] igmp6_mc_seq_start+0xde/0x138
                          [<c04124dd>] ipv6_dev_mc_inc+0x2d8/0x31c
                          [<c0412205>] ipv6_dev_mc_inc+0x0/0x31c
                          [<c0401834>] ipv6_add_dev+0x21c/0x24b
                          [<c040b07d>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
                          [<c0401e17>] addrconf_notify+0x60/0x7b7
                          [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                          [<c0141dac>] mark_held_locks+0x39/0x53
                          [<c0439066>] mutex_lock_nested+0x286/0x2ac
                          [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                          [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
                          [<c03a3f4d>] register_netdevice_notifier+0xe/0x126
                          [<c03a3f4d>] register_netdevice_notifier+0xe/0x126
                          [<c03a3f88>] register_netdevice_notifier+0x49/0x126
                          [<c05c5bda>] addrconf_init+0xad/0x193
                          [<c05c5b48>] addrconf_init+0x1b/0x193
                          [<c05c5a20>] inet6_init+0x1f0/0x2ad
                          [<c05a9499>] kernel_init+0x150/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c0104baf>] kernel_thread_helper+0x7/0x10
                          [<ffffffff>] 0xffffffff
  }
  ... key      at: [<c087adc8>] netdev_xmit_lock_key+0x8/0x1c0
 ... acquired at:
   [<c0142dff>] __lock_acquire+0xa1d/0xc07
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c0143062>] lock_acquire+0x79/0x93
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c0439d62>] _spin_lock_bh+0x3b/0x64
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c0412114>] igmp6_group_added+0x56/0x11d
   [<c04124a8>] ipv6_dev_mc_inc+0x2a3/0x31c
   [<c0410100>] igmp6_mc_seq_start+0xde/0x138
   [<c04124dd>] ipv6_dev_mc_inc+0x2d8/0x31c
   [<c0412205>] ipv6_dev_mc_inc+0x0/0x31c
   [<c0401834>] ipv6_add_dev+0x21c/0x24b
   [<c040b07d>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
   [<c0401e17>] addrconf_notify+0x60/0x7b7
   [<c0142fa1>] __lock_acquire+0xbbf/0xc07
   [<c0141dac>] mark_held_locks+0x39/0x53
   [<c0439066>] mutex_lock_nested+0x286/0x2ac
   [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
   [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
   [<c03a3f4d>] register_netdevice_notifier+0xe/0x126
   [<c03a3f4d>] register_netdevice_notifier+0xe/0x126
   [<c03a3f88>] register_netdevice_notifier+0x49/0x126
   [<c05c5bda>] addrconf_init+0xad/0x193
   [<c05c5b48>] addrconf_init+0x1b/0x193
   [<c05c5a20>] inet6_init+0x1f0/0x2ad
   [<c05a9499>] kernel_init+0x150/0x2b7
   [<c05a9349>] kernel_init+0x0/0x2b7
   [<c05a9349>] kernel_init+0x0/0x2b7
   [<c0104baf>] kernel_thread_helper+0x7/0x10
   [<ffffffff>] 0xffffffff

 -> (&bonding_netdev_xmit_lock_key){-...} ops: 6 {
    initial-use  at:
                          [<c014289c>] __lock_acquire+0x4ba/0xc07
                          [<c0143062>] lock_acquire+0x79/0x93
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c0439d62>] _spin_lock_bh+0x3b/0x64
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c0412114>] igmp6_group_added+0x56/0x11d
                          [<c04124a8>] ipv6_dev_mc_inc+0x2a3/0x31c
                          [<c0410100>] igmp6_mc_seq_start+0xde/0x138
                          [<c04124dd>] ipv6_dev_mc_inc+0x2d8/0x31c
                          [<c0412205>] ipv6_dev_mc_inc+0x0/0x31c
                          [<c0401834>] ipv6_add_dev+0x21c/0x24b
                          [<c040b07d>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
                          [<c0401e17>] addrconf_notify+0x60/0x7b7
                          [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                          [<c0141dac>] mark_held_locks+0x39/0x53
                          [<c0439066>] mutex_lock_nested+0x286/0x2ac
                          [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                          [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
                          [<c03a3f4d>] register_netdevice_notifier+0xe/0x126
                          [<c03a3f4d>] register_netdevice_notifier+0xe/0x126
                          [<c03a3f88>] register_netdevice_notifier+0x49/0x126
                          [<c05c5bda>] addrconf_init+0xad/0x193
                          [<c05c5b48>] addrconf_init+0x1b/0x193
                          [<c05c5a20>] inet6_init+0x1f0/0x2ad
                          [<c05a9499>] kernel_init+0x150/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c0104baf>] kernel_thread_helper+0x7/0x10
                          [<ffffffff>] 0xffffffff
    hardirq-on-W at:
                          [<c0141986>] mark_lock+0x64/0x451
                          [<c014286a>] __lock_acquire+0x488/0xc07
                          [<c0143062>] lock_acquire+0x79/0x93
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c0439d62>] _spin_lock_bh+0x3b/0x64
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c0412114>] igmp6_group_added+0x56/0x11d
                          [<c04124a8>] ipv6_dev_mc_inc+0x2a3/0x31c
                          [<c0410100>] igmp6_mc_seq_start+0xde/0x138
                          [<c04124dd>] ipv6_dev_mc_inc+0x2d8/0x31c
                          [<c0412205>] ipv6_dev_mc_inc+0x0/0x31c
                          [<c0401834>] ipv6_add_dev+0x21c/0x24b
                          [<c040b07d>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
                          [<c0401e17>] addrconf_notify+0x60/0x7b7
                          [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                          [<c0141dac>] mark_held_locks+0x39/0x53
                          [<c0439066>] mutex_lock_nested+0x286/0x2ac
                          [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                          [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
                          [<c03a3f4d>] register_netdevice_notifier+0xe/0x126
                          [<c03a3f4d>] register_netdevice_notifier+0xe/0x126
                          [<c03a3f88>] register_netdevice_notifier+0x49/0x126
                          [<c05c5bda>] addrconf_init+0xad/0x193
                          [<c05c5b48>] addrconf_init+0x1b/0x193
                          [<c05c5a20>] inet6_init+0x1f0/0x2ad
                          [<c05a9499>] kernel_init+0x150/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c0104baf>] kernel_thread_helper+0x7/0x10
                          [<ffffffff>] 0xffffffff
  }
  ... key      at: [<c0877804>] bonding_netdev_xmit_lock_key+0x0/0x8
  -> (&bond->lock){-.--} ops: 98 {
     initial-use  at:
                            [<c013fe35>] put_lock_stats+0xa/0x1e
                            [<c014289c>] __lock_acquire+0x4ba/0xc07
                            [<c0141dac>] mark_held_locks+0x39/0x53
                            [<c043a3c5>] _spin_unlock_irqrestore+0x40/0x58
                            [<c0143062>] lock_acquire+0x79/0x93
                            [<c02edcc1>] bond_get_stats+0x28/0xd0
                            [<c0439eee>] _read_lock_bh+0x3b/0x64
                            [<c02edcc1>] bond_get_stats+0x28/0xd0
                            [<c02edcc1>] bond_get_stats+0x28/0xd0
                            [<c03aa3e7>] rtnl_fill_ifinfo+0x2bf/0x563
                            [<c03aa965>] rtmsg_ifinfo+0x5d/0xdf
                            [<c03aaa26>] rtnetlink_event+0x3f/0x42
                            [<c013ac20>] notifier_call_chain+0x2a/0x52
                            [<c013ac6a>] raw_notifier_call_chain+0x17/0x1a
                            [<c03a31de>] register_netdevice+0x2a7/0x2e7
                            [<c02ed862>] bond_create+0x1f2/0x26a
                            [<c05bedcd>] bonding_init+0x761/0x7ea
                            [<c05be635>] e1000_init_module+0x45/0x7c
                            [<c05a9499>] kernel_init+0x150/0x2b7
                            [<c05a9349>] kernel_init+0x0/0x2b7
                            [<c05a9349>] kernel_init+0x0/0x2b7
                            [<c0104baf>] kernel_thread_helper+0x7/0x10
                            [<ffffffff>] 0xffffffff
     hardirq-on-W at:
                            [<c014286a>] __lock_acquire+0x488/0xc07
                            [<c012093c>] try_to_wake_up+0x2ce/0x2d8
                            [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                            [<c0143062>] lock_acquire+0x79/0x93
                            [<c02eda75>] bond_set_multicast_list+0x1d/0x241
                            [<c0439e25>] _write_lock_bh+0x3b/0x64
                            [<c02eda75>] bond_set_multicast_list+0x1d/0x241
                            [<c02eda75>] bond_set_multicast_list+0x1d/0x241
                            [<c013fe35>] put_lock_stats+0xa/0x1e
                            [<c03a14db>] __dev_set_rx_mode+0x7b/0x7d
                            [<c03a1675>] dev_set_rx_mode+0x23/0x36
                            [<c03a3d50>] dev_open+0x5e/0x77
                            [<c03a2a1f>] dev_change_flags+0x9d/0x14b
                            [<c03a1823>] __dev_get_by_name+0x68/0x73
                            [<c03e3850>] devinet_ioctl+0x22b/0x536
                            [<c03a3b45>] dev_ioctl+0x46f/0x5b7
                            [<c0399c78>] sock_ioctl+0x167/0x18b
                            [<c0399b11>] sock_ioctl+0x0/0x18b
                            [<c01725f7>] do_ioctl+0x1f/0x62
                            [<c0172867>] vfs_ioctl+0x22d/0x23f
                            [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                            [<c01728ac>] sys_ioctl+0x33/0x4b
                            [<c0103e92>] sysenter_past_esp+0x5f/0xa5
                            [<ffffffff>] 0xffffffff
     softirq-on-R at:
                            [<c0141986>] mark_lock+0x64/0x451
                            [<c013575e>] __kernel_text_address+0x5/0xe
                            [<c0104ee2>] dump_trace+0x83/0x8d
                            [<c0142889>] __lock_acquire+0x4a7/0xc07
                            [<c013fc76>] save_trace+0x37/0x89
                            [<c0133c57>] run_workqueue+0x87/0x1b6
                            [<c0143062>] lock_acquire+0x79/0x93
                            [<c02eee5d>] bond_mii_monitor+0x19/0x85
                            [<c0439f4d>] _read_lock+0x36/0x5f
                            [<c02eee5d>] bond_mii_monitor+0x19/0x85
                            [<c02eee5d>] bond_mii_monitor+0x19/0x85
                            [<c0133cab>] run_workqueue+0xdb/0x1b6
                            [<c0133c57>] run_workqueue+0x87/0x1b6
                            [<c02eee44>] bond_mii_monitor+0x0/0x85
                            [<c01346cb>] worker_thread+0x0/0x85
                            [<c0134744>] worker_thread+0x79/0x85
                            [<c0137179>] autoremove_wake_function+0x0/0x35
                            [<c01370c2>] kthread+0x38/0x5e
                            [<c013708a>] kthread+0x0/0x5e
                            [<c0104baf>] kernel_thread_helper+0x7/0x10
                            [<ffffffff>] 0xffffffff
     hardirq-on-R at:
                            [<c013fe0a>] get_lock_stats+0xd/0x2e
                            [<c013fe35>] put_lock_stats+0xa/0x1e
                            [<c0142844>] __lock_acquire+0x462/0xc07
                            [<c0141dac>] mark_held_locks+0x39/0x53
                            [<c043a3c5>] _spin_unlock_irqrestore+0x40/0x58
                            [<c0143062>] lock_acquire+0x79/0x93
                            [<c02edcc1>] bond_get_stats+0x28/0xd0
                            [<c0439eee>] _read_lock_bh+0x3b/0x64
                            [<c02edcc1>] bond_get_stats+0x28/0xd0
                            [<c02edcc1>] bond_get_stats+0x28/0xd0
                            [<c03aa3e7>] rtnl_fill_ifinfo+0x2bf/0x563
                            [<c03aa965>] rtmsg_ifinfo+0x5d/0xdf
                            [<c03aaa26>] rtnetlink_event+0x3f/0x42
                            [<c013ac20>] notifier_call_chain+0x2a/0x52
                            [<c013ac6a>] raw_notifier_call_chain+0x17/0x1a
                            [<c03a31de>] register_netdevice+0x2a7/0x2e7
                            [<c02ed862>] bond_create+0x1f2/0x26a
                            [<c05bedcd>] bonding_init+0x761/0x7ea
                            [<c05be635>] e1000_init_module+0x45/0x7c
                            [<c05a9499>] kernel_init+0x150/0x2b7
                            [<c05a9349>] kernel_init+0x0/0x2b7
                            [<c05a9349>] kernel_init+0x0/0x2b7
                            [<c0104baf>] kernel_thread_helper+0x7/0x10
                            [<ffffffff>] 0xffffffff
   }
   ... key      at: [<c08777d0>] __key.32969+0x0/0x8
   -> (_xmit_ETHER){-...} ops: 8 {
      initial-use  at:
                              [<c014289c>] __lock_acquire+0x4ba/0xc07
                              [<c0143062>] lock_acquire+0x79/0x93
                              [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                              [<c0439d62>] _spin_lock_bh+0x3b/0x64
                              [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                              [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                              [<c0412114>] igmp6_group_added+0x56/0x11d
                              [<c04124a8>] ipv6_dev_mc_inc+0x2a3/0x31c
                              [<c0410100>] igmp6_mc_seq_start+0xde/0x138
                              [<c04124dd>] ipv6_dev_mc_inc+0x2d8/0x31c
                              [<c0412205>] ipv6_dev_mc_inc+0x0/0x31c
                              [<c0401834>] ipv6_add_dev+0x21c/0x24b
                              [<c040b07d>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
                              [<c0401e17>] addrconf_notify+0x60/0x7b7
                              [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                              [<c0141dac>] mark_held_locks+0x39/0x53
                              [<c0439066>] mutex_lock_nested+0x286/0x2ac
                              [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                              [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
                              [<c03a3f4d>] register_netdevice_notifier+0xe/0x126
                              [<c03a3f4d>] register_netdevice_notifier+0xe/0x126
                              [<c03a3f88>] register_netdevice_notifier+0x49/0x126
                              [<c05c5bda>] addrconf_init+0xad/0x193
                              [<c05c5b48>] addrconf_init+0x1b/0x193
                              [<c05c5a20>] inet6_init+0x1f0/0x2ad
                              [<c05a9499>] kernel_init+0x150/0x2b7
                              [<c05a9349>] kernel_init+0x0/0x2b7
                              [<c05a9349>] kernel_init+0x0/0x2b7
                              [<c0104baf>] kernel_thread_helper+0x7/0x10
                              [<ffffffff>] 0xffffffff
      hardirq-on-W at:
                              [<c0141986>] mark_lock+0x64/0x451
                              [<c014286a>] __lock_acquire+0x488/0xc07
                              [<c0143062>] lock_acquire+0x79/0x93
                              [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                              [<c0439d62>] _spin_lock_bh+0x3b/0x64
                              [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                              [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                              [<c0412114>] igmp6_group_added+0x56/0x11d
                              [<c04124a8>] ipv6_dev_mc_inc+0x2a3/0x31c
                              [<c0410100>] igmp6_mc_seq_start+0xde/0x138
                              [<c04124dd>] ipv6_dev_mc_inc+0x2d8/0x31c
                              [<c0412205>] ipv6_dev_mc_inc+0x0/0x31c
                              [<c0401834>] ipv6_add_dev+0x21c/0x24b
                              [<c040b07d>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
                              [<c0401e17>] addrconf_notify+0x60/0x7b7
                              [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                              [<c0141dac>] mark_held_locks+0x39/0x53
                              [<c0439066>] mutex_lock_nested+0x286/0x2ac
                              [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                              [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
                              [<c03a3f4d>] register_netdevice_notifier+0xe/0x126
                              [<c03a3f4d>] register_netdevice_notifier+0xe/0x126
                              [<c03a3f88>] register_netdevice_notifier+0x49/0x126
                              [<c05c5bda>] addrconf_init+0xad/0x193
                              [<c05c5b48>] addrconf_init+0x1b/0x193
                              [<c05c5a20>] inet6_init+0x1f0/0x2ad
                              [<c05a9499>] kernel_init+0x150/0x2b7
                              [<c05a9349>] kernel_init+0x0/0x2b7
                              [<c05a9349>] kernel_init+0x0/0x2b7
                              [<c0104baf>] kernel_thread_helper+0x7/0x10
                              [<ffffffff>] 0xffffffff
    }
    ... key      at: [<c087adc8>] netdev_xmit_lock_key+0x8/0x1c0
   ... acquired at:
   [<c0142dff>] __lock_acquire+0xa1d/0xc07
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c043a3c5>] _spin_unlock_irqrestore+0x40/0x58
   [<c0109ef2>] save_stack_trace+0x20/0x3a
   [<c0143062>] lock_acquire+0x79/0x93
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c0439d62>] _spin_lock_bh+0x3b/0x64
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c02ee492>] bond_change_active_slave+0x1a9/0x3bf
   [<c02ec7c3>] bond_update_speed_duplex+0x26/0x65
   [<c02ee9af>] bond_select_active_slave+0x95/0xcd
   [<c02ed22b>] bond_compute_features+0x45/0x84
   [<c02ef9be>] bond_enslave+0x6a7/0x884
   [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
   [<c02f6197>] bonding_store_slaves+0x1ae/0x2fb
   [<c02f5fe9>] bonding_store_slaves+0x0/0x2fb
   [<c02ce8d7>] dev_attr_store+0x27/0x2c
   [<c019bcb9>] sysfs_write_file+0xad/0xe0
   [<c019bc0c>] sysfs_write_file+0x0/0xe0
   [<c0168ddc>] vfs_write+0x8a/0x10c
   [<c0118566>] do_page_fault+0x0/0x54a
   [<c0169361>] sys_write+0x41/0x67
   [<c0103e92>] sysenter_past_esp+0x5f/0xa5
   [<ffffffff>] 0xffffffff

   -> (lweventlist_lock){.+..} ops: 10 {
      initial-use  at:
                              [<c0141986>] mark_lock+0x64/0x451
                              [<c014289c>] __lock_acquire+0x4ba/0xc07
                              [<c02e365c>] e1000_read_phy_reg+0x1c7/0x1d3
                              [<c02e348b>] e1000_write_phy_reg+0xb9/0xc3
                              [<c024a7de>] delay_tsc+0x25/0x3b
                              [<c0143062>] lock_acquire+0x79/0x93
                              [<c03abaf7>] linkwatch_add_event+0xd/0x2c
                              [<c043a07f>] _spin_lock_irqsave+0x3f/0x6c
                              [<c03abaf7>] linkwatch_add_event+0xd/0x2c
                              [<c03abaf7>] linkwatch_add_event+0xd/0x2c
                              [<c03abbbb>] linkwatch_fire_event+0x25/0x37
                              [<c02e1c43>] e1000_probe+0xad1/0xbe8
                              [<c0257f3f>] pci_device_probe+0x36/0x57
                              [<c02d0e5f>] driver_probe_device+0xe1/0x15f
                              [<c043a2f9>] _spin_unlock+0x25/0x3b
                              [<c04375b2>] klist_next+0x58/0x6d
                              [<c02d0f6f>] __driver_attach+0x0/0x7f
                              [<c02d0fb8>] __driver_attach+0x49/0x7f
                              [<c02d0403>] bus_for_each_dev+0x36/0x58
                              [<c02d0cb7>] driver_attach+0x16/0x18
                              [<c02d0f6f>] __driver_attach+0x0/0x7f
                              [<c02d06fa>] bus_add_driver+0x6d/0x18d
                              [<c0258089>] __pci_register_driver+0x53/0x7f
                              [<c05be635>] e1000_init_module+0x45/0x7c
                              [<c05a9499>] kernel_init+0x150/0x2b7
                              [<c05a9349>] kernel_init+0x0/0x2b7
                              [<c05a9349>] kernel_init+0x0/0x2b7
                              [<c0104baf>] kernel_thread_helper+0x7/0x10
                              [<ffffffff>] 0xffffffff
      in-softirq-W at:
                              [<c011d20a>] __wake_up_common+0x32/0x5c
                              [<c0142822>] __lock_acquire+0x440/0xc07
                              [<c043a3c5>] _spin_unlock_irqrestore+0x40/0x58
                              [<c0143062>] lock_acquire+0x79/0x93
                              [<c03abaf7>] linkwatch_add_event+0xd/0x2c
                              [<c02dff01>] e1000_watchdog+0x0/0x5c9
                              [<c043a07f>] _spin_lock_irqsave+0x3f/0x6c
                              [<c03abaf7>] linkwatch_add_event+0xd/0x2c
                              [<c03abaf7>] linkwatch_add_event+0xd/0x2c
                              [<c03abbbb>] linkwatch_fire_event+0x25/0x37
                              [<c03aeb42>] netif_carrier_on+0x16/0x27
                              [<c02e0156>] e1000_watchdog+0x255/0x5c9
                              [<c02dff01>] e1000_watchdog+0x0/0x5c9
                              [<c012df52>] run_timer_softirq+0xfa/0x15d
                              [<c012a8a6>] __do_softirq+0x56/0xdb
                              [<c0141f89>] trace_hardirqs_on+0x10c/0x14c
                              [<c012a8b8>] __do_softirq+0x68/0xdb
                              [<c012a961>] do_softirq+0x36/0x51
                              [<c012ab07>] irq_exit+0x43/0x4e
                              [<c0114122>] smp_apic_timer_interrupt+0x74/0x80
                              [<c0104a01>] apic_timer_interrupt+0x29/0x38
                              [<c0104a0b>] apic_timer_interrupt+0x33/0x38
                              [<c01600d8>] sys_swapon+0x29c/0x9aa
                              [<c01021a6>] mwait_idle_with_hints+0x3b/0x3f
                              [<c0102447>] mwait_idle+0x0/0xf
                              [<c0102581>] cpu_idle+0x99/0xc6
                              [<c05a98c7>] start_kernel+0x2c7/0x2cf
                              [<c05a90e0>] unknown_bootoption+0x0/0x195
                              [<ffffffff>] 0xffffffff
    }
    ... key      at: [<c058a194>] lweventlist_lock+0x14/0x40
   ... acquired at:
   [<c0142dff>] __lock_acquire+0xa1d/0xc07
   [<c03abaf7>] linkwatch_add_event+0xd/0x2c
   [<c0142fa1>] __lock_acquire+0xbbf/0xc07
   [<c0143062>] lock_acquire+0x79/0x93
   [<c03abaf7>] linkwatch_add_event+0xd/0x2c
   [<c043a07f>] _spin_lock_irqsave+0x3f/0x6c
   [<c03abaf7>] linkwatch_add_event+0xd/0x2c
   [<c03abaf7>] linkwatch_add_event+0xd/0x2c
   [<c03abbbb>] linkwatch_fire_event+0x25/0x37
   [<c03aeb42>] netif_carrier_on+0x16/0x27
   [<c02ede2c>] bond_set_carrier+0x31/0x55
   [<c02ee9b6>] bond_select_active_slave+0x9c/0xcd
   [<c02ed22b>] bond_compute_features+0x45/0x84
   [<c02ef9be>] bond_enslave+0x6a7/0x884
   [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
   [<c02f6197>] bonding_store_slaves+0x1ae/0x2fb
   [<c02f5fe9>] bonding_store_slaves+0x0/0x2fb
   [<c02ce8d7>] dev_attr_store+0x27/0x2c
   [<c019bcb9>] sysfs_write_file+0xad/0xe0
   [<c019bc0c>] sysfs_write_file+0x0/0xe0
   [<c0168ddc>] vfs_write+0x8a/0x10c
   [<c0118566>] do_page_fault+0x0/0x54a
   [<c0169361>] sys_write+0x41/0x67
   [<c0103e92>] sysenter_past_esp+0x5f/0xa5
   [<ffffffff>] 0xffffffff

... acquired at:

   [<c0142dff>] __lock_acquire+0xa1d/0xc07
   [<c02eda75>] bond_set_multicast_list+0x1d/0x241
   [<c0143062>] lock_acquire+0x79/0x93
   [<c02eda75>] bond_set_multicast_list+0x1d/0x241
   [<c0439e25>] _write_lock_bh+0x3b/0x64
   [<c02eda75>] bond_set_multicast_list+0x1d/0x241
   [<c02eda75>] bond_set_multicast_list+0x1d/0x241
   [<c013fe35>] put_lock_stats+0xa/0x1e
   [<c03a14db>] __dev_set_rx_mode+0x7b/0x7d
   [<c03a1675>] dev_set_rx_mode+0x23/0x36
   [<c03a3d50>] dev_open+0x5e/0x77
   [<c03a2a1f>] dev_change_flags+0x9d/0x14b
   [<c03a1823>] __dev_get_by_name+0x68/0x73
   [<c03e3850>] devinet_ioctl+0x22b/0x536
   [<c03a3b45>] dev_ioctl+0x46f/0x5b7
   [<c0399c78>] sock_ioctl+0x167/0x18b
   [<c0399b11>] sock_ioctl+0x0/0x18b
   [<c01725f7>] do_ioctl+0x1f/0x62
   [<c0172867>] vfs_ioctl+0x22d/0x23f
   [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
   [<c01728ac>] sys_ioctl+0x33/0x4b
   [<c0103e92>] sysenter_past_esp+0x5f/0xa5
   [<ffffffff>] 0xffffffff

... acquired at:

   [<c0142dff>] __lock_acquire+0xa1d/0xc07
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c0143062>] lock_acquire+0x79/0x93
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c0439d62>] _spin_lock_bh+0x3b/0x64
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c0412114>] igmp6_group_added+0x56/0x11d
   [<c04124a8>] ipv6_dev_mc_inc+0x2a3/0x31c
   [<c0410100>] igmp6_mc_seq_start+0xde/0x138
   [<c04124dd>] ipv6_dev_mc_inc+0x2d8/0x31c
   [<c0412205>] ipv6_dev_mc_inc+0x0/0x31c
   [<c0401834>] ipv6_add_dev+0x21c/0x24b
   [<c040b07d>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
   [<c0401e17>] addrconf_notify+0x60/0x7b7
   [<c0142fa1>] __lock_acquire+0xbbf/0xc07
   [<c0141dac>] mark_held_locks+0x39/0x53
   [<c0439066>] mutex_lock_nested+0x286/0x2ac
   [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
   [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
   [<c03a3f4d>] register_netdevice_notifier+0xe/0x126
   [<c03a3f4d>] register_netdevice_notifier+0xe/0x126
   [<c03a3f88>] register_netdevice_notifier+0x49/0x126
   [<c05c5bda>] addrconf_init+0xad/0x193
   [<c05c5b48>] addrconf_init+0x1b/0x193
   [<c05c5a20>] inet6_init+0x1f0/0x2ad
   [<c05a9499>] kernel_init+0x150/0x2b7
   [<c05a9349>] kernel_init+0x0/0x2b7
   [<c05a9349>] kernel_init+0x0/0x2b7
   [<c0104baf>] kernel_thread_helper+0x7/0x10
   [<ffffffff>] 0xffffffff

the second lock's dependencies:
-> (&bond->lock){-.--} ops: 98 {
initial-use at:

                        [<c013fe35>] put_lock_stats+0xa/0x1e
                        [<c014289c>] __lock_acquire+0x4ba/0xc07
                        [<c0141dac>] mark_held_locks+0x39/0x53
                        [<c043a3c5>] _spin_unlock_irqrestore+0x40/0x58
                        [<c0143062>] lock_acquire+0x79/0x93
                        [<c02edcc1>] bond_get_stats+0x28/0xd0
                        [<c0439eee>] _read_lock_bh+0x3b/0x64
                        [<c02edcc1>] bond_get_stats+0x28/0xd0
                        [<c02edcc1>] bond_get_stats+0x28/0xd0
                        [<c03aa3e7>] rtnl_fill_ifinfo+0x2bf/0x563
                        [<c03aa965>] rtmsg_ifinfo+0x5d/0xdf
                        [<c03aaa26>] rtnetlink_event+0x3f/0x42
                        [<c013ac20>] notifier_call_chain+0x2a/0x52
                        [<c013ac6a>] raw_notifier_call_chain+0x17/0x1a
                        [<c03a31de>] register_netdevice+0x2a7/0x2e7
                        [<c02ed862>] bond_create+0x1f2/0x26a
                        [<c05bedcd>] bonding_init+0x761/0x7ea
                        [<c05be635>] e1000_init_module+0x45/0x7c
                        [<c05a9499>] kernel_init+0x150/0x2b7
                        [<c05a9349>] kernel_init+0x0/0x2b7
                        [<c05a9349>] kernel_init+0x0/0x2b7
                        [<c0104baf>] kernel_thread_helper+0x7/0x10
                        [<ffffffff>] 0xffffffff
   hardirq-on-W at:
                        [<c014286a>] __lock_acquire+0x488/0xc07
                        [<c012093c>] try_to_wake_up+0x2ce/0x2d8
                        [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                        [<c0143062>] lock_acquire+0x79/0x93
                        [<c02eda75>] bond_set_multicast_list+0x1d/0x241
                        [<c0439e25>] _write_lock_bh+0x3b/0x64
                        [<c02eda75>] bond_set_multicast_list+0x1d/0x241
                        [<c02eda75>] bond_set_multicast_list+0x1d/0x241
                        [<c013fe35>] put_lock_stats+0xa/0x1e
                        [<c03a14db>] __dev_set_rx_mode+0x7b/0x7d
                        [<c03a1675>] dev_set_rx_mode+0x23/0x36
                        [<c03a3d50>] dev_open+0x5e/0x77
                        [<c03a2a1f>] dev_change_flags+0x9d/0x14b
                        [<c03a1823>] __dev_get_by_name+0x68/0x73
                        [<c03e3850>] devinet_ioctl+0x22b/0x536
                        [<c03a3b45>] dev_ioctl+0x46f/0x5b7
                        [<c0399c78>] sock_ioctl+0x167/0x18b
                        [<c0399b11>] sock_ioctl+0x0/0x18b
                        [<c01725f7>] do_ioctl+0x1f/0x62
                        [<c0172867>] vfs_ioctl+0x22d/0x23f
                        [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                        [<c01728ac>] sys_ioctl+0x33/0x4b
                        [<c0103e92>] sysenter_past_esp+0x5f/0xa5
                        [<ffffffff>] 0xffffffff
   softirq-on-R at:
                        [<c0141986>] mark_lock+0x64/0x451
                        [<c013575e>] __kernel_text_address+0x5/0xe
                        [<c0104ee2>] dump_trace+0x83/0x8d
                        [<c0142889>] __lock_acquire+0x4a7/0xc07
                        [<c013fc76>] save_trace+0x37/0x89
                        [<c0133c57>] run_workqueue+0x87/0x1b6
                        [<c0143062>] lock_acquire+0x79/0x93
                        [<c02eee5d>] bond_mii_monitor+0x19/0x85
                        [<c0439f4d>] _read_lock+0x36/0x5f
                        [<c02eee5d>] bond_mii_monitor+0x19/0x85
                        [<c02eee5d>] bond_mii_monitor+0x19/0x85
                        [<c0133cab>] run_workqueue+0xdb/0x1b6
                        [<c0133c57>] run_workqueue+0x87/0x1b6
                        [<c02eee44>] bond_mii_monitor+0x0/0x85
                        [<c01346cb>] worker_thread+0x0/0x85
                        [<c0134744>] worker_thread+0x79/0x85
                        [<c0137179>] autoremove_wake_function+0x0/0x35
                        [<c01370c2>] kthread+0x38/0x5e
                        [<c013708a>] kthread+0x0/0x5e
                        [<c0104baf>] kernel_thread_helper+0x7/0x10
                        [<ffffffff>] 0xffffffff
   hardirq-on-R at:
                        [<c013fe0a>] get_lock_stats+0xd/0x2e
                        [<c013fe35>] put_lock_stats+0xa/0x1e
                        [<c0142844>] __lock_acquire+0x462/0xc07
                        [<c0141dac>] mark_held_locks+0x39/0x53
                        [<c043a3c5>] _spin_unlock_irqrestore+0x40/0x58
                        [<c0143062>] lock_acquire+0x79/0x93
                        [<c02edcc1>] bond_get_stats+0x28/0xd0
                        [<c0439eee>] _read_lock_bh+0x3b/0x64
                        [<c02edcc1>] bond_get_stats+0x28/0xd0
                        [<c02edcc1>] bond_get_stats+0x28/0xd0
                        [<c03aa3e7>] rtnl_fill_ifinfo+0x2bf/0x563
                        [<c03aa965>] rtmsg_ifinfo+0x5d/0xdf
                        [<c03aaa26>] rtnetlink_event+0x3f/0x42
                        [<c013ac20>] notifier_call_chain+0x2a/0x52
                        [<c013ac6a>] raw_notifier_call_chain+0x17/0x1a
                        [<c03a31de>] register_netdevice+0x2a7/0x2e7
                        [<c02ed862>] bond_create+0x1f2/0x26a
                        [<c05bedcd>] bonding_init+0x761/0x7ea
                        [<c05be635>] e1000_init_module+0x45/0x7c
                        [<c05a9499>] kernel_init+0x150/0x2b7
                        [<c05a9349>] kernel_init+0x0/0x2b7
                        [<c05a9349>] kernel_init+0x0/0x2b7
                        [<c0104baf>] kernel_thread_helper+0x7/0x10
                        [<ffffffff>] 0xffffffff
 }
 ... key      at: [<c08777d0>] __key.32969+0x0/0x8
 -> (_xmit_ETHER){-...} ops: 8 {
    initial-use  at:
                          [<c014289c>] __lock_acquire+0x4ba/0xc07
                          [<c0143062>] lock_acquire+0x79/0x93
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c0439d62>] _spin_lock_bh+0x3b/0x64
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c0412114>] igmp6_group_added+0x56/0x11d
                          [<c04124a8>] ipv6_dev_mc_inc+0x2a3/0x31c
                          [<c0410100>] igmp6_mc_seq_start+0xde/0x138
                          [<c04124dd>] ipv6_dev_mc_inc+0x2d8/0x31c
                          [<c0412205>] ipv6_dev_mc_inc+0x0/0x31c
                          [<c0401834>] ipv6_add_dev+0x21c/0x24b
                          [<c040b07d>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
                          [<c0401e17>] addrconf_notify+0x60/0x7b7
                          [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                          [<c0141dac>] mark_held_locks+0x39/0x53
                          [<c0439066>] mutex_lock_nested+0x286/0x2ac
                          [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                          [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
                          [<c03a3f4d>] register_netdevice_notifier+0xe/0x126
                          [<c03a3f4d>] register_netdevice_notifier+0xe/0x126
                          [<c03a3f88>] register_netdevice_notifier+0x49/0x126
                          [<c05c5bda>] addrconf_init+0xad/0x193
                          [<c05c5b48>] addrconf_init+0x1b/0x193
                          [<c05c5a20>] inet6_init+0x1f0/0x2ad
                          [<c05a9499>] kernel_init+0x150/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c0104baf>] kernel_thread_helper+0x7/0x10
                          [<ffffffff>] 0xffffffff
    hardirq-on-W at:
                          [<c0141986>] mark_lock+0x64/0x451
                          [<c014286a>] __lock_acquire+0x488/0xc07
                          [<c0143062>] lock_acquire+0x79/0x93
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c0439d62>] _spin_lock_bh+0x3b/0x64
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c03a5aa1>] dev_mc_add+0x1a/0x6a
                          [<c0412114>] igmp6_group_added+0x56/0x11d
                          [<c04124a8>] ipv6_dev_mc_inc+0x2a3/0x31c
                          [<c0410100>] igmp6_mc_seq_start+0xde/0x138
                          [<c04124dd>] ipv6_dev_mc_inc+0x2d8/0x31c
                          [<c0412205>] ipv6_dev_mc_inc+0x0/0x31c
                          [<c0401834>] ipv6_add_dev+0x21c/0x24b
                          [<c040b07d>] 
ndisc_ifinfo_sysctl_change+0x0/0x1ef
                          [<c0401e17>] addrconf_notify+0x60/0x7b7
                          [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                          [<c0141dac>] mark_held_locks+0x39/0x53
                          [<c0439066>] mutex_lock_nested+0x286/0x2ac
                          [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                          [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
                          [<c03a3f4d>] register_netdevice_notifier+0xe/0x126
                          [<c03a3f4d>] register_netdevice_notifier+0xe/0x126
                          [<c03a3f88>] register_netdevice_notifier+0x49/0x126
                          [<c05c5bda>] addrconf_init+0xad/0x193
                          [<c05c5b48>] addrconf_init+0x1b/0x193
                          [<c05c5a20>] inet6_init+0x1f0/0x2ad
                          [<c05a9499>] kernel_init+0x150/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c0104baf>] kernel_thread_helper+0x7/0x10
                          [<ffffffff>] 0xffffffff
  }
  ... key      at: [<c087adc8>] netdev_xmit_lock_key+0x8/0x1c0
 ... acquired at:
   [<c0142dff>] __lock_acquire+0xa1d/0xc07
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c043a3c5>] _spin_unlock_irqrestore+0x40/0x58
   [<c0109ef2>] save_stack_trace+0x20/0x3a
   [<c0143062>] lock_acquire+0x79/0x93
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c0439d62>] _spin_lock_bh+0x3b/0x64
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c03a5aa1>] dev_mc_add+0x1a/0x6a
   [<c02ee492>] bond_change_active_slave+0x1a9/0x3bf
   [<c02ec7c3>] bond_update_speed_duplex+0x26/0x65
   [<c02ee9af>] bond_select_active_slave+0x95/0xcd
   [<c02ed22b>] bond_compute_features+0x45/0x84
   [<c02ef9be>] bond_enslave+0x6a7/0x884
   [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
   [<c02f6197>] bonding_store_slaves+0x1ae/0x2fb
   [<c02f5fe9>] bonding_store_slaves+0x0/0x2fb
   [<c02ce8d7>] dev_attr_store+0x27/0x2c
   [<c019bcb9>] sysfs_write_file+0xad/0xe0
   [<c019bc0c>] sysfs_write_file+0x0/0xe0
   [<c0168ddc>] vfs_write+0x8a/0x10c
   [<c0118566>] do_page_fault+0x0/0x54a
   [<c0169361>] sys_write+0x41/0x67
   [<c0103e92>] sysenter_past_esp+0x5f/0xa5
   [<ffffffff>] 0xffffffff

 -> (lweventlist_lock){.+..} ops: 10 {
    initial-use  at:
                          [<c0141986>] mark_lock+0x64/0x451
                          [<c014289c>] __lock_acquire+0x4ba/0xc07
                          [<c02e365c>] e1000_read_phy_reg+0x1c7/0x1d3
                          [<c02e348b>] e1000_write_phy_reg+0xb9/0xc3
                          [<c024a7de>] delay_tsc+0x25/0x3b
                          [<c0143062>] lock_acquire+0x79/0x93
                          [<c03abaf7>] linkwatch_add_event+0xd/0x2c
                          [<c043a07f>] _spin_lock_irqsave+0x3f/0x6c
                          [<c03abaf7>] linkwatch_add_event+0xd/0x2c
                          [<c03abaf7>] linkwatch_add_event+0xd/0x2c
                          [<c03abbbb>] linkwatch_fire_event+0x25/0x37
                          [<c02e1c43>] e1000_probe+0xad1/0xbe8
                          [<c0257f3f>] pci_device_probe+0x36/0x57
                          [<c02d0e5f>] driver_probe_device+0xe1/0x15f
                          [<c043a2f9>] _spin_unlock+0x25/0x3b
                          [<c04375b2>] klist_next+0x58/0x6d
                          [<c02d0f6f>] __driver_attach+0x0/0x7f
                          [<c02d0fb8>] __driver_attach+0x49/0x7f
                          [<c02d0403>] bus_for_each_dev+0x36/0x58
                          [<c02d0cb7>] driver_attach+0x16/0x18
                          [<c02d0f6f>] __driver_attach+0x0/0x7f
                          [<c02d06fa>] bus_add_driver+0x6d/0x18d
                          [<c0258089>] __pci_register_driver+0x53/0x7f
                          [<c05be635>] e1000_init_module+0x45/0x7c
                          [<c05a9499>] kernel_init+0x150/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c0104baf>] kernel_thread_helper+0x7/0x10
                          [<ffffffff>] 0xffffffff
    in-softirq-W at:
                          [<c011d20a>] __wake_up_common+0x32/0x5c
                          [<c0142822>] __lock_acquire+0x440/0xc07
                          [<c043a3c5>] _spin_unlock_irqrestore+0x40/0x58
                          [<c0143062>] lock_acquire+0x79/0x93
                          [<c03abaf7>] linkwatch_add_event+0xd/0x2c
                          [<c02dff01>] e1000_watchdog+0x0/0x5c9
                          [<c043a07f>] _spin_lock_irqsave+0x3f/0x6c
                          [<c03abaf7>] linkwatch_add_event+0xd/0x2c
                          [<c03abaf7>] linkwatch_add_event+0xd/0x2c
                          [<c03abbbb>] linkwatch_fire_event+0x25/0x37
                          [<c03aeb42>] netif_carrier_on+0x16/0x27
                          [<c02e0156>] e1000_watchdog+0x255/0x5c9
                          [<c02dff01>] e1000_watchdog+0x0/0x5c9
                          [<c012df52>] run_timer_softirq+0xfa/0x15d
                          [<c012a8a6>] __do_softirq+0x56/0xdb
                          [<c0141f89>] trace_hardirqs_on+0x10c/0x14c
                          [<c012a8b8>] __do_softirq+0x68/0xdb
                          [<c012a961>] do_softirq+0x36/0x51
                          [<c012ab07>] irq_exit+0x43/0x4e
                          [<c0114122>] smp_apic_timer_interrupt+0x74/0x80
                          [<c0104a01>] apic_timer_interrupt+0x29/0x38
                          [<c0104a0b>] apic_timer_interrupt+0x33/0x38
                          [<c01600d8>] sys_swapon+0x29c/0x9aa
                          [<c01021a6>] mwait_idle_with_hints+0x3b/0x3f
                          [<c0102447>] mwait_idle+0x0/0xf
                          [<c0102581>] cpu_idle+0x99/0xc6
                          [<c05a98c7>] start_kernel+0x2c7/0x2cf
                          [<c05a90e0>] unknown_bootoption+0x0/0x195
                          [<ffffffff>] 0xffffffff
  }
  ... key      at: [<c058a194>] lweventlist_lock+0x14/0x40
 ... acquired at:
   [<c0142dff>] __lock_acquire+0xa1d/0xc07
   [<c03abaf7>] linkwatch_add_event+0xd/0x2c
   [<c0142fa1>] __lock_acquire+0xbbf/0xc07
   [<c0143062>] lock_acquire+0x79/0x93
   [<c03abaf7>] linkwatch_add_event+0xd/0x2c
   [<c043a07f>] _spin_lock_irqsave+0x3f/0x6c
   [<c03abaf7>] linkwatch_add_event+0xd/0x2c
   [<c03abaf7>] linkwatch_add_event+0xd/0x2c
   [<c03abbbb>] linkwatch_fire_event+0x25/0x37
   [<c03aeb42>] netif_carrier_on+0x16/0x27
   [<c02ede2c>] bond_set_carrier+0x31/0x55
   [<c02ee9b6>] bond_select_active_slave+0x9c/0xcd
   [<c02ed22b>] bond_compute_features+0x45/0x84
   [<c02ef9be>] bond_enslave+0x6a7/0x884
   [<c0439084>] mutex_lock_nested+0x2a4/0x2ac
   [<c02f6197>] bonding_store_slaves+0x1ae/0x2fb
   [<c02f5fe9>] bonding_store_slaves+0x0/0x2fb
   [<c02ce8d7>] dev_attr_store+0x27/0x2c
   [<c019bcb9>] sysfs_write_file+0xad/0xe0
   [<c019bc0c>] sysfs_write_file+0x0/0xe0
   [<c0168ddc>] vfs_write+0x8a/0x10c
   [<c0118566>] do_page_fault+0x0/0x54a
   [<c0169361>] sys_write+0x41/0x67
   [<c0103e92>] sysenter_past_esp+0x5f/0xa5
   [<ffffffff>] 0xffffffff


stack backtrace:
Pid: 9, comm: events/0 Not tainted 2.6.24-rc5 #1
 [<c0140b38>] print_irq_inversion_bug+0x108/0x112
 [<c014191d>] check_usage_forwards+0x3c/0x41
 [<c0141b09>] mark_lock+0x1e7/0x451
 [<c0142822>] __lock_acquire+0x440/0xc07
 [<c013fe0a>] get_lock_stats+0xd/0x2e
 [<c013fe35>] put_lock_stats+0xa/0x1e
 [<c0143062>] lock_acquire+0x79/0x93
 [<c0411c7a>] mld_ifc_timer_expire+0x130/0x1fb
 [<c0411b4a>] mld_ifc_timer_expire+0x0/0x1fb
 [<c0439d62>] _spin_lock_bh+0x3b/0x64
 [<c0411c7a>] mld_ifc_timer_expire+0x130/0x1fb
 [<c0411c7a>] mld_ifc_timer_expire+0x130/0x1fb
 [<c0411b4a>] mld_ifc_timer_expire+0x0/0x1fb
 [<c0141f89>] trace_hardirqs_on+0x10c/0x14c
 [<c0411b4a>] mld_ifc_timer_expire+0x0/0x1fb
 [<c012df52>] run_timer_softirq+0xfa/0x15d
 [<c012a8a6>] __do_softirq+0x56/0xdb
 [<c0141f89>] trace_hardirqs_on+0x10c/0x14c
 [<c012a8b8>] __do_softirq+0x68/0xdb
 [<c012a961>] do_softirq+0x36/0x51
 [<c012ae4a>] local_bh_enable_ip+0xad/0xed
 [<c03bf107>] rt_run_flush+0x64/0x8b
 [<c03e9296>] fib_netdev_event+0x61/0x65
 [<c013ac20>] notifier_call_chain+0x2a/0x52
 [<c013ac6a>] raw_notifier_call_chain+0x17/0x1a
 [<c03a2ae5>] netdev_state_change+0x18/0x29
 [<c03abd1d>] __linkwatch_run_queue+0x150/0x17e
 [<c03abd68>] linkwatch_event+0x1d/0x22
 [<c0133cab>] run_workqueue+0xdb/0x1b6
 [<c0133c57>] run_workqueue+0x87/0x1b6
 [<c03abd4b>] linkwatch_event+0x0/0x22
 [<c01346cb>] worker_thread+0x0/0x85
 [<c0134744>] worker_thread+0x79/0x85
 [<c0137179>] autoremove_wake_function+0x0/0x35
 [<c01370c2>] kthread+0x38/0x5e
 [<c013708a>] kthread+0x0/0x5e
 [<c0104baf>] kernel_thread_helper+0x7/0x10
 =======================

bonding: bond0: enslaving eth1 as a backup interface with a down link.
bonding: bond0: Setting eth0 as primary slave.
bond0: no IPv6 routers present

Best regards,

Krzysztof Olędzki

Andy Gospodarek

2007-12-14 22:15 UTC #10 lore raw

Quoted 52 lines · Krzysztof Oledzki

Quoted 48 lines · Andy Gospodarek
Quoted 47 lines · Krzysztof Oledzki
Quoted 43 lines · Jay Vosburgh
Quoted 8 lines · Herbert Xu
Quoted 8 lines
diff -puN drivers/net/bonding/bond_sysfs.c~bonding-locking-fix drivers/net/bonding/bond_sysfs.c
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -0
@@ -1111,8 +1111,6 @@ static ssize_t bonding_store_primary(str
out:
     write_unlock_bh(&bond->lock);
Quoted 6 lines
drivers/net/bonding/bond_sysfs.c
Quoted 2 lines
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -2
-       rtnl_unlock();
-
Looking at the changeset that added this perhaps the intention is to hold the lock? If so we should add an rtnl_lock to the start of the function.
Yes, this function needs to hold locks, and more than just
what's there now. I believe the following should be correct; I haven't tested it, though (I'm supposedly on vacation right now).
The following change should be correct for the
bonding_store_primary case discussed in this thread, and also corrects the bonding_store_active case which performs similar functions.
The bond_change_active_slave and bond_select_active_slave
functions both require rtnl, bond->lock for read and curr_slave_lock for write_bh, and no other locks. This is so that the lower level mode-specific functions can release locks down to just rtnl in order to call, e.g., dev_set_mac_address with the locks it expects (rtnl only).
Signed-off-by Jay Vosburgh <[email protected]>
drivers/net/bonding/bond_sysfs.c +0 -0
index 11b76b3..28a2d80 100644
@@ -1075,7 +1075,10 @@ static ssize_t bonding_store_primary(struct device
 *d,
 	struct slave *slave;
 	struct bonding *bond = to_bond(d);
 

Quoted 4 lines

Quoted 4 lines
Quoted 4 lines
drivers/net/bonding/bond_sysfs.c static ssize_t bonding_store_primary(struct device *d, · line 1080
Quoted 4 lines
drivers/net/bonding/bond_sysfs.c +3 -1
-	write_lock_bh(&bond->lock);
+	rtnl_lock();
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);

Quoted 71 lines

Quoted 51 lines
Quoted 51 lines
Quoted 36 lines
drivers/net/bonding/bond_sysfs.c +1 -0
+
 	if (!USES_PRIMARY(bond->params.mode)) {
 		printk(KERN_INFO DRV_NAME
		       ": %s: Unable to set primary slave; %s is in mode
		       %d\n",
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1109,8 +1112,8 @@ static ssize_t bonding_store_primary(struct device
 *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +2 -2
-	write_unlock_bh(&bond->lock);
-
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
	rtnl_unlock();

	return count;
drivers/net/bonding/bond_sysfs.c +2 -1
@@ -1190,7 +1193,8 @@ static ssize_t bonding_store_active_slave(struct
 device *d,
 	struct bonding *bond = to_bond(d);
 
 	rtnl_lock();
-	write_lock_bh(&bond->lock);
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
 
 	if (!USES_PRIMARY(bond->params.mode)) {
printk(KERN_INFO DRV_NAME
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1247,7 +1251,8 @@ static ssize_t bonding_store_active_slave(struct
 device *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +2 -1
-	write_unlock_bh(&bond->lock);
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
Quoted 3 lines
	rtnl_unlock();

	return count;
Vanilla 2.6.24-rc5 plus this patch:
=========================================================
[ INFO: possible irq lock inversion dependency detected ] 2.6.24-rc5 #1
---------------------------------------------------------
events/0/9 just changed the state of lock:
(&mc->mca_lock){-+..}, at: [<c0411c7a>] mld_ifc_timer_expire+0x130/0x1fb but this lock took another, soft-read-irq-unsafe lock in the past:
(&bond->lock){-.--}
Quoted 5 lines
Quoted 3 lines
and interrupts could create inverse lock ordering between them.
Grrr, I should have seen that -- sorry. Try your luck with this instead:
<CUT>
No luck.
bonding: bond0: setting mode to active-backup (1).
bonding: bond0: Setting MII monitoring interval to 100. ADDRCONF(NETDEV_UP): bond0: link is not ready
bonding: bond0: Adding slave eth0.
e1000: eth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex, Flow
Control: RX/TX
bonding: bond0: making interface eth0 the new active one.
bonding: bond0: first active interface up!
bonding: bond0: enslaving eth0 as an active interface with an up link.
bonding: bond0: Adding slave eth1.
ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready

<SNIP>

Quoted 4 lines

bonding: bond0: enslaving eth1 as a backup interface with a down link.
bonding: bond0: Setting eth0 as primary slave.
bond0: no IPv6 routers present

Based on the console log, I'm guessing your initialization scripts use sysfs to set eth0 as the primary interface for bond0? Can you confirm?

If you did somehow use sysfs to set the primary device as eth0, I'm guessing you never see this issue without that line or without this patch. Please confirm this as well.

Thanks,

-andy

Krzysztof Oledzki

2007-12-14 22:17 UTC #11 lore raw

Quoted 132 lines · Andy Gospodarek

Quoted 52 lines · Krzysztof Oledzki
Quoted 48 lines · Andy Gospodarek
Quoted 47 lines · Krzysztof Oledzki
Quoted 43 lines · Jay Vosburgh
Quoted 8 lines · Herbert Xu
Quoted 8 lines
diff -puN drivers/net/bonding/bond_sysfs.c~bonding-locking-fix drivers/net/bonding/bond_sysfs.c
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -0
@@ -1111,8 +1111,6 @@ static ssize_t bonding_store_primary(str
out:
     write_unlock_bh(&bond->lock);
Quoted 6 lines
drivers/net/bonding/bond_sysfs.c
Quoted 2 lines
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -2
-       rtnl_unlock();
-
Looking at the changeset that added this perhaps the intention is to hold the lock? If so we should add an rtnl_lock to the start of the function.
Yes, this function needs to hold locks, and more than just
what's there now. I believe the following should be correct; I haven't tested it, though (I'm supposedly on vacation right now).
The following change should be correct for the
bonding_store_primary case discussed in this thread, and also corrects the bonding_store_active case which performs similar functions.
The bond_change_active_slave and bond_select_active_slave
functions both require rtnl, bond->lock for read and curr_slave_lock for write_bh, and no other locks. This is so that the lower level mode-specific functions can release locks down to just rtnl in order to call, e.g., dev_set_mac_address with the locks it expects (rtnl only).
Signed-off-by Jay Vosburgh <[email protected]>
drivers/net/bonding/bond_sysfs.c +0 -0
index 11b76b3..28a2d80 100644
@@ -1075,7 +1075,10 @@ static ssize_t bonding_store_primary(struct device
 *d,
 	struct slave *slave;
 	struct bonding *bond = to_bond(d);
 
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/net/bonding/bond_sysfs.c static ssize_t bonding_store_primary(struct device *d, · line 1080
Quoted 4 lines
drivers/net/bonding/bond_sysfs.c +3 -1
-	write_lock_bh(&bond->lock);
+	rtnl_lock();
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
F
Quoted 71 lines
Quoted 51 lines
Quoted 51 lines
Quoted 36 lines
drivers/net/bonding/bond_sysfs.c +1 -0
+
 	if (!USES_PRIMARY(bond->params.mode)) {
 		printk(KERN_INFO DRV_NAME
		       ": %s: Unable to set primary slave; %s is in mode
		       %d\n",
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1109,8 +1112,8 @@ static ssize_t bonding_store_primary(struct device
 *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +2 -2
-	write_unlock_bh(&bond->lock);
-
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
	rtnl_unlock();

	return count;
drivers/net/bonding/bond_sysfs.c +2 -1
@@ -1190,7 +1193,8 @@ static ssize_t bonding_store_active_slave(struct
 device *d,
 	struct bonding *bond = to_bond(d);
 
 	rtnl_lock();
-	write_lock_bh(&bond->lock);
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
 
 	if (!USES_PRIMARY(bond->params.mode)) {
printk(KERN_INFO DRV_NAME
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1247,7 +1251,8 @@ static ssize_t bonding_store_active_slave(struct
 device *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +2 -1
-	write_unlock_bh(&bond->lock);
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
Quoted 3 lines
	rtnl_unlock();

	return count;
Vanilla 2.6.24-rc5 plus this patch:
=========================================================
[ INFO: possible irq lock inversion dependency detected ] 2.6.24-rc5 #1
---------------------------------------------------------
events/0/9 just changed the state of lock:
(&mc->mca_lock){-+..}, at: [<c0411c7a>] mld_ifc_timer_expire+0x130/0x1fb but this lock took another, soft-read-irq-unsafe lock in the past:
(&bond->lock){-.--}
Quoted 5 lines
Quoted 3 lines
and interrupts could create inverse lock ordering between them.
Grrr, I should have seen that -- sorry. Try your luck with this instead:
<CUT>
No luck.
bonding: bond0: setting mode to active-backup (1).
bonding: bond0: Setting MII monitoring interval to 100. ADDRCONF(NETDEV_UP): bond0: link is not ready
bonding: bond0: Adding slave eth0.
e1000: eth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex, Flow
Control: RX/TX
bonding: bond0: making interface eth0 the new active one.
bonding: bond0: first active interface up!
bonding: bond0: enslaving eth0 as an active interface with an up link.
bonding: bond0: Adding slave eth1.
ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
<SNIP>

Quoted 7 lines

Quoted 4 lines
bonding: bond0: enslaving eth1 as a backup interface with a down link.
bonding: bond0: Setting eth0 as primary slave.
bond0: no IPv6 routers present
Based on the console log, I'm guessing your initialization scripts use sysfs to set eth0 as the primary interface for bond0? Can you confirm?

Yep, that's correct:

postup() {
        if [[ ${IFACE} == "bond0" ]] ; then
                echo -n +eth0 > /sys/class/net/${IFACE}/bonding/slaves
                echo -n +eth1 > /sys/class/net/${IFACE}/bonding/slaves
                echo -n  eth0 > /sys/class/net/${IFACE}/bonding/primary
        fi
}

Quoted 3 lines

If you did somehow use sysfs to set the primary device as eth0, I'm guessing you never see this issue without that line or without this patch. Please confirm this as well.

Without this patch I get another error, of course. Unfortunately checking what happens without using sysfs to set the primary device as eth0, have to wait, as I am not able to test it before Monday.

Best regards,

Krzysztof Olędzki

Andy Gospodarek

2007-12-14 22:32 UTC #12 lore raw

Quoted 149 lines · Krzysztof Oledzki

Quoted 135 lines · Andy Gospodarek
Quoted 54 lines · Krzysztof Oledzki
Quoted 50 lines · Andy Gospodarek
Quoted 49 lines · Krzysztof Oledzki
Quoted 45 lines · Jay Vosburgh
Quoted 8 lines · Herbert Xu
Quoted 8 lines
diff -puN drivers/net/bonding/bond_sysfs.c~bonding-locking-fix drivers/net/bonding/bond_sysfs.c
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -0
@@ -1111,8 +1111,6 @@ static ssize_t bonding_store_primary(str
out:
    write_unlock_bh(&bond->lock);
Quoted 6 lines
drivers/net/bonding/bond_sysfs.c
Quoted 2 lines
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -2
-       rtnl_unlock();
-
Looking at the changeset that added this perhaps the intention is to hold the lock? If so we should add an rtnl_lock to the start of the function.
Yes, this function needs to hold locks, and more than just
what's there now. I believe the following should be correct; I haven't tested it, though (I'm supposedly on vacation right now).
The following change should be correct for the
bonding_store_primary case discussed in this thread, and also corrects the bonding_store_active case which performs similar functions.
The bond_change_active_slave and bond_select_active_slave
functions both require rtnl, bond->lock for read and curr_slave_lock for
write_bh, and no other locks. This is so that the lower level mode-specific functions can release locks down to just rtnl in order to call, e.g., dev_set_mac_address with the locks it expects (rtnl only).
Signed-off-by Jay Vosburgh <[email protected]>
drivers/net/bonding/bond_sysfs.c +0 -0
index 11b76b3..28a2d80 100644
@@ -1075,7 +1075,10 @@ static ssize_t bonding_store_primary(struct 
device
*d,
	struct slave *slave;
	struct bonding *bond = to_bond(d);
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/net/bonding/bond_sysfs.c static ssize_t bonding_store_primary(struct device *d, · line 1080
Quoted 4 lines
drivers/net/bonding/bond_sysfs.c +3 -1
-	write_lock_bh(&bond->lock);
+	rtnl_lock();
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
F
Quoted 72 lines
Quoted 52 lines
Quoted 52 lines
Quoted 37 lines
+
	if (!USES_PRIMARY(bond->params.mode)) {
		printk(KERN_INFO DRV_NAME
		       ": %s: Unable to set primary slave; %s is in mode
		       %d\n",
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1109,8 +1112,8 @@ static ssize_t bonding_store_primary(struct 
device
*d,
		}
	}
out:
drivers/net/bonding/bond_sysfs.c +2 -2
-	write_unlock_bh(&bond->lock);
-
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
	rtnl_unlock();

	return count;
drivers/net/bonding/bond_sysfs.c +2 -1
@@ -1190,7 +1193,8 @@ static ssize_t bonding_store_active_slave(struct
 device *d,
 	struct bonding *bond = to_bond(d);
 
 	rtnl_lock();
-	write_lock_bh(&bond->lock);
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
 
 	if (!USES_PRIMARY(bond->params.mode)) {
printk(KERN_INFO DRV_NAME
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1247,7 +1251,8 @@ static ssize_t bonding_store_active_slave(struct
 device *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +2 -1
-	write_unlock_bh(&bond->lock);
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
Quoted 3 lines
	rtnl_unlock();

	return count;
Vanilla 2.6.24-rc5 plus this patch:
=========================================================
[ INFO: possible irq lock inversion dependency detected ] 2.6.24-rc5 #1
---------------------------------------------------------
events/0/9 just changed the state of lock:
(&mc->mca_lock){-+..}, at: [<c0411c7a>] mld_ifc_timer_expire+0x130/0x1fb but this lock took another, soft-read-irq-unsafe lock in the past:
(&bond->lock){-.--}
Quoted 5 lines
Quoted 3 lines
and interrupts could create inverse lock ordering between them.
Grrr, I should have seen that -- sorry. Try your luck with this instead:
<CUT>
No luck.
bonding: bond0: setting mode to active-backup (1).
bonding: bond0: Setting MII monitoring interval to 100. ADDRCONF(NETDEV_UP): bond0: link is not ready
bonding: bond0: Adding slave eth0.
e1000: eth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex, Flow
Control: RX/TX
bonding: bond0: making interface eth0 the new active one.
bonding: bond0: first active interface up!
bonding: bond0: enslaving eth0 as an active interface with an up link.
bonding: bond0: Adding slave eth1.
ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
<SNIP>
Quoted 7 lines
Quoted 4 lines
bonding: bond0: enslaving eth1 as a backup interface with a down link.
bonding: bond0: Setting eth0 as primary slave.
bond0: no IPv6 routers present
Based on the console log, I'm guessing your initialization scripts use sysfs to set eth0 as the primary interface for bond0? Can you confirm?
Yep, that's correct:

Quoted 8 lines

postup() {
        if [[ ${IFACE} == "bond0" ]] ; then
                echo -n +eth0 > /sys/class/net/${IFACE}/bonding/slaves
                echo -n +eth1 > /sys/class/net/${IFACE}/bonding/slaves
                echo -n  eth0 > /sys/class/net/${IFACE}/bonding/primary
        fi
}

Good. Thanks for the confirmation.

Quoted 5 lines

Quoted 3 lines
If you did somehow use sysfs to set the primary device as eth0, I'm guessing you never see this issue without that line or without this patch. Please confirm this as well.
Without this patch I get another error, of course. Unfortunately checking

:-/

Quoted 3 lines

what happens without using sysfs to set the primary device as eth0, have to wait, as I am not able to test it before Monday.

No problem, thanks for the feedback and quick response.

Krzysztof Oledzki

2007-12-18 20:03 UTC #13 lore raw

In reply to Andy Gospodarek

Quoted 137 lines · Andy Gospodarek

Quoted 52 lines · Krzysztof Oledzki
Quoted 48 lines · Andy Gospodarek
Quoted 47 lines · Krzysztof Oledzki
Quoted 43 lines · Jay Vosburgh
Quoted 8 lines · Herbert Xu
Quoted 8 lines
diff -puN drivers/net/bonding/bond_sysfs.c~bonding-locking-fix drivers/net/bonding/bond_sysfs.c
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -0
@@ -1111,8 +1111,6 @@ static ssize_t bonding_store_primary(str
out:
     write_unlock_bh(&bond->lock);
Quoted 6 lines
drivers/net/bonding/bond_sysfs.c
Quoted 2 lines
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -2
-       rtnl_unlock();
-
Looking at the changeset that added this perhaps the intention is to hold the lock? If so we should add an rtnl_lock to the start of the function.
Yes, this function needs to hold locks, and more than just
what's there now. I believe the following should be correct; I haven't tested it, though (I'm supposedly on vacation right now).
The following change should be correct for the
bonding_store_primary case discussed in this thread, and also corrects the bonding_store_active case which performs similar functions.
The bond_change_active_slave and bond_select_active_slave
functions both require rtnl, bond->lock for read and curr_slave_lock for write_bh, and no other locks. This is so that the lower level mode-specific functions can release locks down to just rtnl in order to call, e.g., dev_set_mac_address with the locks it expects (rtnl only).
Signed-off-by Jay Vosburgh <[email protected]>
drivers/net/bonding/bond_sysfs.c +0 -0
index 11b76b3..28a2d80 100644
@@ -1075,7 +1075,10 @@ static ssize_t bonding_store_primary(struct device
 *d,
 	struct slave *slave;
 	struct bonding *bond = to_bond(d);
 
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/net/bonding/bond_sysfs.c static ssize_t bonding_store_primary(struct device *d, · line 1080
Quoted 4 lines
drivers/net/bonding/bond_sysfs.c +3 -1
-	write_lock_bh(&bond->lock);
+	rtnl_lock();
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
F
Quoted 71 lines
Quoted 51 lines
Quoted 51 lines
Quoted 36 lines
drivers/net/bonding/bond_sysfs.c +1 -0
+
 	if (!USES_PRIMARY(bond->params.mode)) {
 		printk(KERN_INFO DRV_NAME
		       ": %s: Unable to set primary slave; %s is in mode
		       %d\n",
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1109,8 +1112,8 @@ static ssize_t bonding_store_primary(struct device
 *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +2 -2
-	write_unlock_bh(&bond->lock);
-
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
	rtnl_unlock();

	return count;
drivers/net/bonding/bond_sysfs.c +2 -1
@@ -1190,7 +1193,8 @@ static ssize_t bonding_store_active_slave(struct
 device *d,
 	struct bonding *bond = to_bond(d);
 
 	rtnl_lock();
-	write_lock_bh(&bond->lock);
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
 
 	if (!USES_PRIMARY(bond->params.mode)) {
printk(KERN_INFO DRV_NAME
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1247,7 +1251,8 @@ static ssize_t bonding_store_active_slave(struct
 device *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +2 -1
-	write_unlock_bh(&bond->lock);
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
Quoted 3 lines
	rtnl_unlock();

	return count;
Vanilla 2.6.24-rc5 plus this patch:
=========================================================
[ INFO: possible irq lock inversion dependency detected ] 2.6.24-rc5 #1
---------------------------------------------------------
events/0/9 just changed the state of lock:
(&mc->mca_lock){-+..}, at: [<c0411c7a>] mld_ifc_timer_expire+0x130/0x1fb but this lock took another, soft-read-irq-unsafe lock in the past:
(&bond->lock){-.--}
Quoted 5 lines
Quoted 3 lines
and interrupts could create inverse lock ordering between them.
Grrr, I should have seen that -- sorry. Try your luck with this instead:
<CUT>
No luck.
bonding: bond0: setting mode to active-backup (1).
bonding: bond0: Setting MII monitoring interval to 100. ADDRCONF(NETDEV_UP): bond0: link is not ready
bonding: bond0: Adding slave eth0.
e1000: eth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex, Flow
Control: RX/TX
bonding: bond0: making interface eth0 the new active one.
bonding: bond0: first active interface up!
bonding: bond0: enslaving eth0 as an active interface with an up link.
bonding: bond0: Adding slave eth1.
ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
<SNIP>
Quoted 4 lines
bonding: bond0: enslaving eth1 as a backup interface with a down link.
bonding: bond0: Setting eth0 as primary slave.
bond0: no IPv6 routers present

Quoted 6 lines

Based on the console log, I'm guessing your initialization scripts use sysfs to set eth0 as the primary interface for bond0? Can you confirm?
If you did somehow use sysfs to set the primary device as eth0, I'm guessing you never see this issue without that line or without this patch. Please confirm this as well.

Unpatched 2.6.24-rc5 with "echo -n eth0 > /sys/class/net/${IFACE}/bonding/primary" removed, still has this issue. I also removed:
# echo -n 1 > /sys/class/net/${IFACE}/bonding/mode
# echo -n 100 > /sys/class/net/${IFACE}/bonding/miimon
but this did not help:

=========================================================
[ INFO: possible irq lock inversion dependency detected ] 2.6.24-rc5 #1
---------------------------------------------------------
events/0/9 just changed the state of lock:
(&mc->mca_lock){-+..}, at: [<c0411c52>] mld_ifc_timer_expire+0x130/0x1fb
but this lock took another, soft-read-irq-unsafe lock in the past:
(&bond->lock){-.--}

and interrupts could create inverse lock ordering between them.

other info that might help us debug this:
4 locks held by events/0/9:

 #0:  (events){--..}, at: [<c0133c57>] run_workqueue+0x87/0x1b6
 #1:  ((linkwatch_work).work){--..}, at: [<c0133c57>] run_workqueue+0x87/0x1b6
 #2:  (rtnl_mutex){--..}, at: [<c03abd28>] linkwatch_event+0x5/0x22
 #3:  (&ndev->lock){-.-+}, at: [<c0411b39>] mld_ifc_timer_expire+0x17/0x1fb

the first lock's dependencies:
-> (&mc->mca_lock){-+..} ops: 10 {
initial-use at:

                        [<c0104ee2>] dump_trace+0x83/0x8d
                        [<c014289c>] __lock_acquire+0x4ba/0xc07
                        [<c0109ef2>] save_stack_trace+0x20/0x3a
                        [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                        [<c041242a>] ipv6_dev_mc_inc+0x24d/0x31c
                        [<c0143062>] lock_acquire+0x79/0x93
                        [<c04120ae>] igmp6_group_added+0x18/0x11d
                        [<c0439d3a>] _spin_lock_bh+0x3b/0x64
                        [<c04120ae>] igmp6_group_added+0x18/0x11d
                        [<c04120ae>] igmp6_group_added+0x18/0x11d
                        [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                        [<c0412480>] ipv6_dev_mc_inc+0x2a3/0x31c
                        [<c041242a>] ipv6_dev_mc_inc+0x24d/0x31c
                        [<c04124b5>] ipv6_dev_mc_inc+0x2d8/0x31c
                        [<c04121dd>] ipv6_dev_mc_inc+0x0/0x31c
                        [<c040180c>] ipv6_add_dev+0x21c/0x24b
                        [<c040b055>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
                        [<c05c5b40>] addrconf_init+0x13/0x193
                        [<c0199f63>] proc_net_fops_create+0x10/0x21
                        [<c0419b10>] ip6_flowlabel_init+0x1e/0x20
                        [<c05c5a20>] inet6_init+0x1f0/0x2ad
                        [<c05a9499>] kernel_init+0x150/0x2b7
                        [<c05a9349>] kernel_init+0x0/0x2b7
                        [<c05a9349>] kernel_init+0x0/0x2b7
                        [<c0104baf>] kernel_thread_helper+0x7/0x10
                        [<ffffffff>] 0xffffffff
   in-softirq-W at:
                        [<c0142822>] __lock_acquire+0x440/0xc07
                        [<c011d686>] update_curr+0x52/0xc4
                        [<c013f175>] tick_sched_timer+0x129/0x165
                        [<c0143062>] lock_acquire+0x79/0x93
                        [<c0411c52>] mld_ifc_timer_expire+0x130/0x1fb
                        [<c0411b22>] mld_ifc_timer_expire+0x0/0x1fb
                        [<c0439d3a>] _spin_lock_bh+0x3b/0x64
                        [<c0411c52>] mld_ifc_timer_expire+0x130/0x1fb
                        [<c0411c52>] mld_ifc_timer_expire+0x130/0x1fb
                        [<c0411b22>] mld_ifc_timer_expire+0x0/0x1fb
                        [<c0141f89>] trace_hardirqs_on+0x10c/0x14c
                        [<c0411b22>] mld_ifc_timer_expire+0x0/0x1fb
                        [<c012df52>] run_timer_softirq+0xfa/0x15d
                        [<c012a8a6>] __do_softirq+0x56/0xdb
                        [<c0141f89>] trace_hardirqs_on+0x10c/0x14c
                        [<c012a8b8>] __do_softirq+0x68/0xdb
                        [<c012a961>] do_softirq+0x36/0x51
                        [<c012ae4a>] local_bh_enable_ip+0xad/0xed
                        [<c03bf0df>] rt_run_flush+0x64/0x8b
                        [<c03e926e>] fib_netdev_event+0x61/0x65
                        [<c013ac20>] notifier_call_chain+0x2a/0x52
                        [<c013ac6a>] raw_notifier_call_chain+0x17/0x1a
                        [<c03a2abd>] netdev_state_change+0x18/0x29
                        [<c03abcf5>] __linkwatch_run_queue+0x150/0x17e
                        [<c03abd40>] linkwatch_event+0x1d/0x22
                        [<c0133cab>] run_workqueue+0xdb/0x1b6
                        [<c0133c57>] run_workqueue+0x87/0x1b6
                        [<c03abd23>] linkwatch_event+0x0/0x22
                        [<c01346cb>] worker_thread+0x0/0x85
                        [<c0134744>] worker_thread+0x79/0x85
                        [<c0137179>] autoremove_wake_function+0x0/0x35
                        [<c01370c2>] kthread+0x38/0x5e
                        [<c013708a>] kthread+0x0/0x5e
                        [<c0104baf>] kernel_thread_helper+0x7/0x10
                        [<ffffffff>] 0xffffffff
   hardirq-on-W at:
                        [<c01417ee>] find_usage_backwards+0xbb/0xe2
                        [<c0104ee2>] dump_trace+0x83/0x8d
                        [<c014286a>] __lock_acquire+0x488/0xc07
                        [<c0109ef2>] save_stack_trace+0x20/0x3a
                        [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                        [<c041242a>] ipv6_dev_mc_inc+0x24d/0x31c
                        [<c0143062>] lock_acquire+0x79/0x93
                        [<c04120ae>] igmp6_group_added+0x18/0x11d
                        [<c0439d3a>] _spin_lock_bh+0x3b/0x64
                        [<c04120ae>] igmp6_group_added+0x18/0x11d
                        [<c04120ae>] igmp6_group_added+0x18/0x11d
                        [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                        [<c0412480>] ipv6_dev_mc_inc+0x2a3/0x31c
                        [<c041242a>] ipv6_dev_mc_inc+0x24d/0x31c
                        [<c04124b5>] ipv6_dev_mc_inc+0x2d8/0x31c
                        [<c04121dd>] ipv6_dev_mc_inc+0x0/0x31c
                        [<c040180c>] ipv6_add_dev+0x21c/0x24b
                        [<c040b055>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
                        [<c05c5b40>] addrconf_init+0x13/0x193
                        [<c0199f63>] proc_net_fops_create+0x10/0x21
                        [<c0419b10>] ip6_flowlabel_init+0x1e/0x20
                        [<c05c5a20>] inet6_init+0x1f0/0x2ad
                        [<c05a9499>] kernel_init+0x150/0x2b7
                        [<c05a9349>] kernel_init+0x0/0x2b7
                        [<c05a9349>] kernel_init+0x0/0x2b7
                        [<c0104baf>] kernel_thread_helper+0x7/0x10
                        [<ffffffff>] 0xffffffff
 }
 ... key      at: [<c087e2d8>] __key.30798+0x0/0x8
 -> (_xmit_ETHER){-...} ops: 8 {
    initial-use  at:
                          [<c014289c>] __lock_acquire+0x4ba/0xc07
                          [<c0143062>] lock_acquire+0x79/0x93
                          [<c03a5a79>] dev_mc_add+0x1a/0x6a
                          [<c0439d3a>] _spin_lock_bh+0x3b/0x64
                          [<c03a5a79>] dev_mc_add+0x1a/0x6a
                          [<c03a5a79>] dev_mc_add+0x1a/0x6a
                          [<c04120ec>] igmp6_group_added+0x56/0x11d
                          [<c0412480>] ipv6_dev_mc_inc+0x2a3/0x31c
                          [<c0410100>] igmp6_mc_seq_start+0x106/0x138
                          [<c04124b5>] ipv6_dev_mc_inc+0x2d8/0x31c
                          [<c04121dd>] ipv6_dev_mc_inc+0x0/0x31c
                          [<c040180c>] ipv6_add_dev+0x21c/0x24b
                          [<c040b055>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
                          [<c0401def>] addrconf_notify+0x60/0x7b7
                          [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                          [<c0141dac>] mark_held_locks+0x39/0x53
                          [<c043903e>] mutex_lock_nested+0x286/0x2ac
                          [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                          [<c043905c>] mutex_lock_nested+0x2a4/0x2ac
                          [<c03a3f25>] register_netdevice_notifier+0xe/0x126
                          [<c03a3f25>] register_netdevice_notifier+0xe/0x126
                          [<c03a3f60>] register_netdevice_notifier+0x49/0x126
                          [<c05c5bda>] addrconf_init+0xad/0x193
                          [<c05c5b48>] addrconf_init+0x1b/0x193
                          [<c05c5a20>] inet6_init+0x1f0/0x2ad
                          [<c05a9499>] kernel_init+0x150/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c0104baf>] kernel_thread_helper+0x7/0x10
                          [<ffffffff>] 0xffffffff
    hardirq-on-W at:
                          [<c0141986>] mark_lock+0x64/0x451
                          [<c014286a>] __lock_acquire+0x488/0xc07
                          [<c0143062>] lock_acquire+0x79/0x93
                          [<c03a5a79>] dev_mc_add+0x1a/0x6a
                          [<c0439d3a>] _spin_lock_bh+0x3b/0x64
                          [<c03a5a79>] dev_mc_add+0x1a/0x6a
                          [<c03a5a79>] dev_mc_add+0x1a/0x6a
                          [<c04120ec>] igmp6_group_added+0x56/0x11d
                          [<c0412480>] ipv6_dev_mc_inc+0x2a3/0x31c
                          [<c0410100>] igmp6_mc_seq_start+0x106/0x138
                          [<c04124b5>] ipv6_dev_mc_inc+0x2d8/0x31c
                          [<c04121dd>] ipv6_dev_mc_inc+0x0/0x31c
                          [<c040180c>] ipv6_add_dev+0x21c/0x24b
                          [<c040b055>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
                          [<c0401def>] addrconf_notify+0x60/0x7b7
                          [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                          [<c0141dac>] mark_held_locks+0x39/0x53
                          [<c043903e>] mutex_lock_nested+0x286/0x2ac
                          [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                          [<c043905c>] mutex_lock_nested+0x2a4/0x2ac
                          [<c03a3f25>] register_netdevice_notifier+0xe/0x126
                          [<c03a3f25>] register_netdevice_notifier+0xe/0x126
                          [<c03a3f60>] register_netdevice_notifier+0x49/0x126
                          [<c05c5bda>] addrconf_init+0xad/0x193
                          [<c05c5b48>] addrconf_init+0x1b/0x193
                          [<c05c5a20>] inet6_init+0x1f0/0x2ad
                          [<c05a9499>] kernel_init+0x150/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c0104baf>] kernel_thread_helper+0x7/0x10
                          [<ffffffff>] 0xffffffff
  }
  ... key      at: [<c087adc8>] netdev_xmit_lock_key+0x8/0x1c0
 ... acquired at:
   [<c0142dff>] __lock_acquire+0xa1d/0xc07
   [<c03a5a79>] dev_mc_add+0x1a/0x6a
   [<c0143062>] lock_acquire+0x79/0x93
   [<c03a5a79>] dev_mc_add+0x1a/0x6a
   [<c0439d3a>] _spin_lock_bh+0x3b/0x64
   [<c03a5a79>] dev_mc_add+0x1a/0x6a
   [<c03a5a79>] dev_mc_add+0x1a/0x6a
   [<c04120ec>] igmp6_group_added+0x56/0x11d
   [<c0412480>] ipv6_dev_mc_inc+0x2a3/0x31c
   [<c0410100>] igmp6_mc_seq_start+0x106/0x138
   [<c04124b5>] ipv6_dev_mc_inc+0x2d8/0x31c
   [<c04121dd>] ipv6_dev_mc_inc+0x0/0x31c
   [<c040180c>] ipv6_add_dev+0x21c/0x24b
   [<c040b055>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
   [<c0401def>] addrconf_notify+0x60/0x7b7
   [<c0142fa1>] __lock_acquire+0xbbf/0xc07
   [<c0141dac>] mark_held_locks+0x39/0x53
   [<c043903e>] mutex_lock_nested+0x286/0x2ac
   [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
   [<c043905c>] mutex_lock_nested+0x2a4/0x2ac
   [<c03a3f25>] register_netdevice_notifier+0xe/0x126
   [<c03a3f25>] register_netdevice_notifier+0xe/0x126
   [<c03a3f60>] register_netdevice_notifier+0x49/0x126
   [<c05c5bda>] addrconf_init+0xad/0x193
   [<c05c5b48>] addrconf_init+0x1b/0x193
   [<c05c5a20>] inet6_init+0x1f0/0x2ad
   [<c05a9499>] kernel_init+0x150/0x2b7
   [<c05a9349>] kernel_init+0x0/0x2b7
   [<c05a9349>] kernel_init+0x0/0x2b7
   [<c0104baf>] kernel_thread_helper+0x7/0x10
   [<ffffffff>] 0xffffffff

 -> (&bonding_netdev_xmit_lock_key){-...} ops: 6 {
    initial-use  at:
                          [<c014289c>] __lock_acquire+0x4ba/0xc07
                          [<c0143062>] lock_acquire+0x79/0x93
                          [<c03a5a79>] dev_mc_add+0x1a/0x6a
                          [<c0439d3a>] _spin_lock_bh+0x3b/0x64
                          [<c03a5a79>] dev_mc_add+0x1a/0x6a
                          [<c03a5a79>] dev_mc_add+0x1a/0x6a
                          [<c04120ec>] igmp6_group_added+0x56/0x11d
                          [<c0412480>] ipv6_dev_mc_inc+0x2a3/0x31c
                          [<c0410100>] igmp6_mc_seq_start+0x106/0x138
                          [<c04124b5>] ipv6_dev_mc_inc+0x2d8/0x31c
                          [<c04121dd>] ipv6_dev_mc_inc+0x0/0x31c
                          [<c040180c>] ipv6_add_dev+0x21c/0x24b
                          [<c040b055>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
                          [<c0401def>] addrconf_notify+0x60/0x7b7
                          [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                          [<c0141dac>] mark_held_locks+0x39/0x53
                          [<c043903e>] mutex_lock_nested+0x286/0x2ac
                          [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                          [<c043905c>] mutex_lock_nested+0x2a4/0x2ac
                          [<c03a3f25>] register_netdevice_notifier+0xe/0x126
                          [<c03a3f25>] register_netdevice_notifier+0xe/0x126
                          [<c03a3f60>] register_netdevice_notifier+0x49/0x126
                          [<c05c5bda>] addrconf_init+0xad/0x193
                          [<c05c5b48>] addrconf_init+0x1b/0x193
                          [<c05c5a20>] inet6_init+0x1f0/0x2ad
                          [<c05a9499>] kernel_init+0x150/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c0104baf>] kernel_thread_helper+0x7/0x10
                          [<ffffffff>] 0xffffffff
    hardirq-on-W at:
                          [<c0141986>] mark_lock+0x64/0x451
                          [<c014286a>] __lock_acquire+0x488/0xc07
                          [<c0143062>] lock_acquire+0x79/0x93
                          [<c03a5a79>] dev_mc_add+0x1a/0x6a
                          [<c0439d3a>] _spin_lock_bh+0x3b/0x64
                          [<c03a5a79>] dev_mc_add+0x1a/0x6a
                          [<c03a5a79>] dev_mc_add+0x1a/0x6a
                          [<c04120ec>] igmp6_group_added+0x56/0x11d
                          [<c0412480>] ipv6_dev_mc_inc+0x2a3/0x31c
                          [<c0410100>] igmp6_mc_seq_start+0x106/0x138
                          [<c04124b5>] ipv6_dev_mc_inc+0x2d8/0x31c
                          [<c04121dd>] ipv6_dev_mc_inc+0x0/0x31c
                          [<c040180c>] ipv6_add_dev+0x21c/0x24b
                          [<c040b055>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
                          [<c0401def>] addrconf_notify+0x60/0x7b7
                          [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                          [<c0141dac>] mark_held_locks+0x39/0x53
                          [<c043903e>] mutex_lock_nested+0x286/0x2ac
                          [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                          [<c043905c>] mutex_lock_nested+0x2a4/0x2ac
                          [<c03a3f25>] register_netdevice_notifier+0xe/0x126
                          [<c03a3f25>] register_netdevice_notifier+0xe/0x126
                          [<c03a3f60>] register_netdevice_notifier+0x49/0x126
                          [<c05c5bda>] addrconf_init+0xad/0x193
                          [<c05c5b48>] addrconf_init+0x1b/0x193
                          [<c05c5a20>] inet6_init+0x1f0/0x2ad
                          [<c05a9499>] kernel_init+0x150/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c0104baf>] kernel_thread_helper+0x7/0x10
                          [<ffffffff>] 0xffffffff
  }
  ... key      at: [<c0877804>] bonding_netdev_xmit_lock_key+0x0/0x8
  -> (&bond->lock){-.--} ops: 99 {
     initial-use  at:
                            [<c013fe35>] put_lock_stats+0xa/0x1e
                            [<c014289c>] __lock_acquire+0x4ba/0xc07
                            [<c0141dac>] mark_held_locks+0x39/0x53
                            [<c043a39d>] _spin_unlock_irqrestore+0x40/0x58
                            [<c0143062>] lock_acquire+0x79/0x93
                            [<c02edcc1>] bond_get_stats+0x28/0xd0
                            [<c0439ec6>] _read_lock_bh+0x3b/0x64
                            [<c02edcc1>] bond_get_stats+0x28/0xd0
                            [<c02edcc1>] bond_get_stats+0x28/0xd0
                            [<c03aa3bf>] rtnl_fill_ifinfo+0x2bf/0x563
                            [<c03aa93d>] rtmsg_ifinfo+0x5d/0xdf
                            [<c03aa9fe>] rtnetlink_event+0x3f/0x42
                            [<c013ac20>] notifier_call_chain+0x2a/0x52
                            [<c013ac6a>] raw_notifier_call_chain+0x17/0x1a
                            [<c03a31b6>] register_netdevice+0x2a7/0x2e7
                            [<c02ed862>] bond_create+0x1f2/0x26a
                            [<c05bedcd>] bonding_init+0x761/0x7ea
                            [<c05be635>] e1000_init_module+0x45/0x7c
                            [<c05a9499>] kernel_init+0x150/0x2b7
                            [<c05a9349>] kernel_init+0x0/0x2b7
                            [<c05a9349>] kernel_init+0x0/0x2b7
                            [<c0104baf>] kernel_thread_helper+0x7/0x10
                            [<ffffffff>] 0xffffffff
     hardirq-on-W at:
                            [<c014286a>] __lock_acquire+0x488/0xc07
                            [<c0141dac>] mark_held_locks+0x39/0x53
                            [<c043a3d5>] _spin_unlock_irq+0x20/0x41
                            [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                            [<c043a3e0>] _spin_unlock_irq+0x2b/0x41
                            [<c012208a>] finish_task_switch+0x50/0x8c
                            [<c0143062>] lock_acquire+0x79/0x93
                            [<c02eda75>] bond_set_multicast_list+0x1d/0x241
                            [<c0439dfd>] _write_lock_bh+0x3b/0x64
                            [<c02eda75>] bond_set_multicast_list+0x1d/0x241
                            [<c02eda75>] bond_set_multicast_list+0x1d/0x241
                            [<c013fe35>] put_lock_stats+0xa/0x1e
                            [<c03a14b3>] __dev_set_rx_mode+0x7b/0x7d
                            [<c03a164d>] dev_set_rx_mode+0x23/0x36
                            [<c03a3d28>] dev_open+0x5e/0x77
                            [<c03a29f7>] dev_change_flags+0x9d/0x14b
                            [<c03a17fb>] __dev_get_by_name+0x68/0x73
                            [<c03e3828>] devinet_ioctl+0x22b/0x536
                            [<c03a3b1d>] dev_ioctl+0x46f/0x5b7
                            [<c0399c50>] sock_ioctl+0x167/0x18b
                            [<c0399ae9>] sock_ioctl+0x0/0x18b
                            [<c01725f7>] do_ioctl+0x1f/0x62
                            [<c0172867>] vfs_ioctl+0x22d/0x23f
                            [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                            [<c01728ac>] sys_ioctl+0x33/0x4b
                            [<c0103e92>] sysenter_past_esp+0x5f/0xa5
                            [<ffffffff>] 0xffffffff
     softirq-on-R at:
                            [<c0141986>] mark_lock+0x64/0x451
                            [<c013575e>] __kernel_text_address+0x5/0xe
                            [<c0104ee2>] dump_trace+0x83/0x8d
                            [<c0142889>] __lock_acquire+0x4a7/0xc07
                            [<c013fc76>] save_trace+0x37/0x89
                            [<c0133c57>] run_workqueue+0x87/0x1b6
                            [<c0143062>] lock_acquire+0x79/0x93
                            [<c02eee5d>] bond_mii_monitor+0x19/0x85
                            [<c0439f25>] _read_lock+0x36/0x5f
                            [<c02eee5d>] bond_mii_monitor+0x19/0x85
                            [<c02eee5d>] bond_mii_monitor+0x19/0x85
                            [<c0133cab>] run_workqueue+0xdb/0x1b6
                            [<c0133c57>] run_workqueue+0x87/0x1b6
                            [<c02eee44>] bond_mii_monitor+0x0/0x85
                            [<c01346cb>] worker_thread+0x0/0x85
                            [<c0134744>] worker_thread+0x79/0x85
                            [<c0137179>] autoremove_wake_function+0x0/0x35
                            [<c01370c2>] kthread+0x38/0x5e
                            [<c013708a>] kthread+0x0/0x5e
                            [<c0104baf>] kernel_thread_helper+0x7/0x10
                            [<ffffffff>] 0xffffffff
     hardirq-on-R at:
                            [<c013fe0a>] get_lock_stats+0xd/0x2e
                            [<c013fe35>] put_lock_stats+0xa/0x1e
                            [<c0142844>] __lock_acquire+0x462/0xc07
                            [<c0141dac>] mark_held_locks+0x39/0x53
                            [<c043a39d>] _spin_unlock_irqrestore+0x40/0x58
                            [<c0143062>] lock_acquire+0x79/0x93
                            [<c02edcc1>] bond_get_stats+0x28/0xd0
                            [<c0439ec6>] _read_lock_bh+0x3b/0x64
                            [<c02edcc1>] bond_get_stats+0x28/0xd0
                            [<c02edcc1>] bond_get_stats+0x28/0xd0
                            [<c03aa3bf>] rtnl_fill_ifinfo+0x2bf/0x563
                            [<c03aa93d>] rtmsg_ifinfo+0x5d/0xdf
                            [<c03aa9fe>] rtnetlink_event+0x3f/0x42
                            [<c013ac20>] notifier_call_chain+0x2a/0x52
                            [<c013ac6a>] raw_notifier_call_chain+0x17/0x1a
                            [<c03a31b6>] register_netdevice+0x2a7/0x2e7
                            [<c02ed862>] bond_create+0x1f2/0x26a
                            [<c05bedcd>] bonding_init+0x761/0x7ea
                            [<c05be635>] e1000_init_module+0x45/0x7c
                            [<c05a9499>] kernel_init+0x150/0x2b7
                            [<c05a9349>] kernel_init+0x0/0x2b7
                            [<c05a9349>] kernel_init+0x0/0x2b7
                            [<c0104baf>] kernel_thread_helper+0x7/0x10
                            [<ffffffff>] 0xffffffff
   }
   ... key      at: [<c08777d0>] __key.32969+0x0/0x8
   -> (_xmit_ETHER){-...} ops: 8 {
      initial-use  at:
                              [<c014289c>] __lock_acquire+0x4ba/0xc07
                              [<c0143062>] lock_acquire+0x79/0x93
                              [<c03a5a79>] dev_mc_add+0x1a/0x6a
                              [<c0439d3a>] _spin_lock_bh+0x3b/0x64
                              [<c03a5a79>] dev_mc_add+0x1a/0x6a
                              [<c03a5a79>] dev_mc_add+0x1a/0x6a
                              [<c04120ec>] igmp6_group_added+0x56/0x11d
                              [<c0412480>] ipv6_dev_mc_inc+0x2a3/0x31c
                              [<c0410100>] igmp6_mc_seq_start+0x106/0x138
                              [<c04124b5>] ipv6_dev_mc_inc+0x2d8/0x31c
                              [<c04121dd>] ipv6_dev_mc_inc+0x0/0x31c
                              [<c040180c>] ipv6_add_dev+0x21c/0x24b
                              [<c040b055>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
                              [<c0401def>] addrconf_notify+0x60/0x7b7
                              [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                              [<c0141dac>] mark_held_locks+0x39/0x53
                              [<c043903e>] mutex_lock_nested+0x286/0x2ac
                              [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                              [<c043905c>] mutex_lock_nested+0x2a4/0x2ac
                              [<c03a3f25>] register_netdevice_notifier+0xe/0x126
                              [<c03a3f25>] register_netdevice_notifier+0xe/0x126
                              [<c03a3f60>] register_netdevice_notifier+0x49/0x126
                              [<c05c5bda>] addrconf_init+0xad/0x193
                              [<c05c5b48>] addrconf_init+0x1b/0x193
                              [<c05c5a20>] inet6_init+0x1f0/0x2ad
                              [<c05a9499>] kernel_init+0x150/0x2b7
                              [<c05a9349>] kernel_init+0x0/0x2b7
                              [<c05a9349>] kernel_init+0x0/0x2b7
                              [<c0104baf>] kernel_thread_helper+0x7/0x10
                              [<ffffffff>] 0xffffffff
      hardirq-on-W at:
                              [<c0141986>] mark_lock+0x64/0x451
                              [<c014286a>] __lock_acquire+0x488/0xc07
                              [<c0143062>] lock_acquire+0x79/0x93
                              [<c03a5a79>] dev_mc_add+0x1a/0x6a
                              [<c0439d3a>] _spin_lock_bh+0x3b/0x64
                              [<c03a5a79>] dev_mc_add+0x1a/0x6a
                              [<c03a5a79>] dev_mc_add+0x1a/0x6a
                              [<c04120ec>] igmp6_group_added+0x56/0x11d
                              [<c0412480>] ipv6_dev_mc_inc+0x2a3/0x31c
                              [<c0410100>] igmp6_mc_seq_start+0x106/0x138
                              [<c04124b5>] ipv6_dev_mc_inc+0x2d8/0x31c
                              [<c04121dd>] ipv6_dev_mc_inc+0x0/0x31c
                              [<c040180c>] ipv6_add_dev+0x21c/0x24b
                              [<c040b055>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
                              [<c0401def>] addrconf_notify+0x60/0x7b7
                              [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                              [<c0141dac>] mark_held_locks+0x39/0x53
                              [<c043903e>] mutex_lock_nested+0x286/0x2ac
                              [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                              [<c043905c>] mutex_lock_nested+0x2a4/0x2ac
                              [<c03a3f25>] register_netdevice_notifier+0xe/0x126
                              [<c03a3f25>] register_netdevice_notifier+0xe/0x126
                              [<c03a3f60>] register_netdevice_notifier+0x49/0x126
                              [<c05c5bda>] addrconf_init+0xad/0x193
                              [<c05c5b48>] addrconf_init+0x1b/0x193
                              [<c05c5a20>] inet6_init+0x1f0/0x2ad
                              [<c05a9499>] kernel_init+0x150/0x2b7
                              [<c05a9349>] kernel_init+0x0/0x2b7
                              [<c05a9349>] kernel_init+0x0/0x2b7
                              [<c0104baf>] kernel_thread_helper+0x7/0x10
                              [<ffffffff>] 0xffffffff
    }
    ... key      at: [<c087adc8>] netdev_xmit_lock_key+0x8/0x1c0
   ... acquired at:
   [<c0142dff>] __lock_acquire+0xa1d/0xc07
   [<c03a5a79>] dev_mc_add+0x1a/0x6a
   [<c043a39d>] _spin_unlock_irqrestore+0x40/0x58
   [<c0109ef2>] save_stack_trace+0x20/0x3a
   [<c0143062>] lock_acquire+0x79/0x93
   [<c03a5a79>] dev_mc_add+0x1a/0x6a
   [<c0439d3a>] _spin_lock_bh+0x3b/0x64
   [<c03a5a79>] dev_mc_add+0x1a/0x6a
   [<c03a5a79>] dev_mc_add+0x1a/0x6a
   [<c02ee492>] bond_change_active_slave+0x1a9/0x3bf
   [<c02ec7c3>] bond_update_speed_duplex+0x26/0x65
   [<c02ee9af>] bond_select_active_slave+0x95/0xcd
   [<c02ed22b>] bond_compute_features+0x45/0x84
   [<c02ef9be>] bond_enslave+0x6a7/0x884
   [<c043905c>] mutex_lock_nested+0x2a4/0x2ac
   [<c02f616d>] bonding_store_slaves+0x1ae/0x2fb
   [<c02f5fbf>] bonding_store_slaves+0x0/0x2fb
   [<c02ce8d7>] dev_attr_store+0x27/0x2c
   [<c019bcb9>] sysfs_write_file+0xad/0xe0
   [<c019bc0c>] sysfs_write_file+0x0/0xe0
   [<c0168ddc>] vfs_write+0x8a/0x10c
   [<c0118566>] do_page_fault+0x0/0x54a
   [<c0169361>] sys_write+0x41/0x67
   [<c0103e92>] sysenter_past_esp+0x5f/0xa5
   [<ffffffff>] 0xffffffff

   -> (lweventlist_lock){.+..} ops: 10 {
      initial-use  at:
                              [<c0141986>] mark_lock+0x64/0x451
                              [<c014289c>] __lock_acquire+0x4ba/0xc07
                              [<c02e365c>] e1000_read_phy_reg+0x1c7/0x1d3
                              [<c02e348b>] e1000_write_phy_reg+0xb9/0xc3
                              [<c024a7de>] delay_tsc+0x25/0x3b
                              [<c0143062>] lock_acquire+0x79/0x93
                              [<c03abacf>] linkwatch_add_event+0xd/0x2c
                              [<c043a057>] _spin_lock_irqsave+0x3f/0x6c
                              [<c03abacf>] linkwatch_add_event+0xd/0x2c
                              [<c03abacf>] linkwatch_add_event+0xd/0x2c
                              [<c03abb93>] linkwatch_fire_event+0x25/0x37
                              [<c02e1c43>] e1000_probe+0xad1/0xbe8
                              [<c0257f3f>] pci_device_probe+0x36/0x57
                              [<c02d0e5f>] driver_probe_device+0xe1/0x15f
                              [<c043a2d1>] _spin_unlock+0x25/0x3b
                              [<c043758a>] klist_next+0x58/0x6d
                              [<c02d0f6f>] __driver_attach+0x0/0x7f
                              [<c02d0fb8>] __driver_attach+0x49/0x7f
                              [<c02d0403>] bus_for_each_dev+0x36/0x58
                              [<c02d0cb7>] driver_attach+0x16/0x18
                              [<c02d0f6f>] __driver_attach+0x0/0x7f
                              [<c02d06fa>] bus_add_driver+0x6d/0x18d
                              [<c0258089>] __pci_register_driver+0x53/0x7f
                              [<c05be635>] e1000_init_module+0x45/0x7c
                              [<c05a9499>] kernel_init+0x150/0x2b7
                              [<c05a9349>] kernel_init+0x0/0x2b7
                              [<c05a9349>] kernel_init+0x0/0x2b7
                              [<c0104baf>] kernel_thread_helper+0x7/0x10
                              [<ffffffff>] 0xffffffff
      in-softirq-W at:
                              [<c011d20a>] __wake_up_common+0x32/0x5c
                              [<c0142822>] __lock_acquire+0x440/0xc07
                              [<c043a39d>] _spin_unlock_irqrestore+0x40/0x58
                              [<c0143062>] lock_acquire+0x79/0x93
                              [<c03abacf>] linkwatch_add_event+0xd/0x2c
                              [<c02dff01>] e1000_watchdog+0x0/0x5c9
                              [<c043a057>] _spin_lock_irqsave+0x3f/0x6c
                              [<c03abacf>] linkwatch_add_event+0xd/0x2c
                              [<c03abacf>] linkwatch_add_event+0xd/0x2c
                              [<c03abb93>] linkwatch_fire_event+0x25/0x37
                              [<c03aeb1a>] netif_carrier_on+0x16/0x27
                              [<c02e0156>] e1000_watchdog+0x255/0x5c9
                              [<c02dff01>] e1000_watchdog+0x0/0x5c9
                              [<c012df52>] run_timer_softirq+0xfa/0x15d
                              [<c012a8a6>] __do_softirq+0x56/0xdb
                              [<c0141f89>] trace_hardirqs_on+0x10c/0x14c
                              [<c012a8b8>] __do_softirq+0x68/0xdb
                              [<c012a961>] do_softirq+0x36/0x51
                              [<c012ae4a>] local_bh_enable_ip+0xad/0xed
                              [<c03bf0df>] rt_run_flush+0x64/0x8b
                              [<c03e7924>] ip_mc_inc_group+0x184/0x1c1
                              [<c03e79a2>] ip_mc_up+0x41/0x59
                              [<c03e32dc>] inetdev_event+0x257/0x465
                              [<c03aa8da>] rtnl_notify+0x3a/0x40
                              [<c03aa997>] rtmsg_ifinfo+0xb7/0xdf
                              [<c013ac20>] notifier_call_chain+0x2a/0x52
                              [<c013ac6a>] raw_notifier_call_chain+0x17/0x1a
                              [<c03a3d3b>] dev_open+0x71/0x77
                              [<c02ef626>] bond_enslave+0x30f/0x884
                              [<c043905c>] mutex_lock_nested+0x2a4/0x2ac
                              [<c02f6161>] bonding_store_slaves+0x1a2/0x2fb
                              [<c02f616d>] bonding_store_slaves+0x1ae/0x2fb
                              [<c02f5fbf>] bonding_store_slaves+0x0/0x2fb
                              [<c02ce8d7>] dev_attr_store+0x27/0x2c
                              [<c019bcb9>] sysfs_write_file+0xad/0xe0
                              [<c019bc0c>] sysfs_write_file+0x0/0xe0
                              [<c0168ddc>] vfs_write+0x8a/0x10c
                              [<c0118566>] do_page_fault+0x0/0x54a
                              [<c0169361>] sys_write+0x41/0x67
                              [<c0103e92>] sysenter_past_esp+0x5f/0xa5
                              [<ffffffff>] 0xffffffff
    }
    ... key      at: [<c058a194>] lweventlist_lock+0x14/0x40
   ... acquired at:
   [<c0142dff>] __lock_acquire+0xa1d/0xc07
   [<c03abacf>] linkwatch_add_event+0xd/0x2c
   [<c0142fa1>] __lock_acquire+0xbbf/0xc07
   [<c0143062>] lock_acquire+0x79/0x93
   [<c03abacf>] linkwatch_add_event+0xd/0x2c
   [<c043a057>] _spin_lock_irqsave+0x3f/0x6c
   [<c03abacf>] linkwatch_add_event+0xd/0x2c
   [<c03abacf>] linkwatch_add_event+0xd/0x2c
   [<c03abb93>] linkwatch_fire_event+0x25/0x37
   [<c03aeb1a>] netif_carrier_on+0x16/0x27
   [<c02ede2c>] bond_set_carrier+0x31/0x55
   [<c02ee9b6>] bond_select_active_slave+0x9c/0xcd
   [<c02ed22b>] bond_compute_features+0x45/0x84
   [<c02ef9be>] bond_enslave+0x6a7/0x884
   [<c043905c>] mutex_lock_nested+0x2a4/0x2ac
   [<c02f616d>] bonding_store_slaves+0x1ae/0x2fb
   [<c02f5fbf>] bonding_store_slaves+0x0/0x2fb
   [<c02ce8d7>] dev_attr_store+0x27/0x2c
   [<c019bcb9>] sysfs_write_file+0xad/0xe0
   [<c019bc0c>] sysfs_write_file+0x0/0xe0
   [<c0168ddc>] vfs_write+0x8a/0x10c
   [<c0118566>] do_page_fault+0x0/0x54a
   [<c0169361>] sys_write+0x41/0x67
   [<c0103e92>] sysenter_past_esp+0x5f/0xa5
   [<ffffffff>] 0xffffffff

... acquired at:

   [<c0142dff>] __lock_acquire+0xa1d/0xc07
   [<c02eda75>] bond_set_multicast_list+0x1d/0x241
   [<c0141dac>] mark_held_locks+0x39/0x53
   [<c0143062>] lock_acquire+0x79/0x93
   [<c02eda75>] bond_set_multicast_list+0x1d/0x241
   [<c0439dfd>] _write_lock_bh+0x3b/0x64
   [<c02eda75>] bond_set_multicast_list+0x1d/0x241
   [<c02eda75>] bond_set_multicast_list+0x1d/0x241
   [<c013fe35>] put_lock_stats+0xa/0x1e
   [<c03a14b3>] __dev_set_rx_mode+0x7b/0x7d
   [<c03a164d>] dev_set_rx_mode+0x23/0x36
   [<c03a3d28>] dev_open+0x5e/0x77
   [<c03a29f7>] dev_change_flags+0x9d/0x14b
   [<c03a17fb>] __dev_get_by_name+0x68/0x73
   [<c03e3828>] devinet_ioctl+0x22b/0x536
   [<c03a3b1d>] dev_ioctl+0x46f/0x5b7
   [<c0399c50>] sock_ioctl+0x167/0x18b
   [<c0399ae9>] sock_ioctl+0x0/0x18b
   [<c01725f7>] do_ioctl+0x1f/0x62
   [<c0172867>] vfs_ioctl+0x22d/0x23f
   [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
   [<c01728ac>] sys_ioctl+0x33/0x4b
   [<c0103e92>] sysenter_past_esp+0x5f/0xa5
   [<ffffffff>] 0xffffffff

... acquired at:

   [<c0142dff>] __lock_acquire+0xa1d/0xc07
   [<c03a5a79>] dev_mc_add+0x1a/0x6a
   [<c0143062>] lock_acquire+0x79/0x93
   [<c03a5a79>] dev_mc_add+0x1a/0x6a
   [<c0439d3a>] _spin_lock_bh+0x3b/0x64
   [<c03a5a79>] dev_mc_add+0x1a/0x6a
   [<c03a5a79>] dev_mc_add+0x1a/0x6a
   [<c04120ec>] igmp6_group_added+0x56/0x11d
   [<c0412480>] ipv6_dev_mc_inc+0x2a3/0x31c
   [<c0410100>] igmp6_mc_seq_start+0x106/0x138
   [<c04124b5>] ipv6_dev_mc_inc+0x2d8/0x31c
   [<c04121dd>] ipv6_dev_mc_inc+0x0/0x31c
   [<c040180c>] ipv6_add_dev+0x21c/0x24b
   [<c040b055>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
   [<c0401def>] addrconf_notify+0x60/0x7b7
   [<c0142fa1>] __lock_acquire+0xbbf/0xc07
   [<c0141dac>] mark_held_locks+0x39/0x53
   [<c043903e>] mutex_lock_nested+0x286/0x2ac
   [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
   [<c043905c>] mutex_lock_nested+0x2a4/0x2ac
   [<c03a3f25>] register_netdevice_notifier+0xe/0x126
   [<c03a3f25>] register_netdevice_notifier+0xe/0x126
   [<c03a3f60>] register_netdevice_notifier+0x49/0x126
   [<c05c5bda>] addrconf_init+0xad/0x193
   [<c05c5b48>] addrconf_init+0x1b/0x193
   [<c05c5a20>] inet6_init+0x1f0/0x2ad
   [<c05a9499>] kernel_init+0x150/0x2b7
   [<c05a9349>] kernel_init+0x0/0x2b7
   [<c05a9349>] kernel_init+0x0/0x2b7
   [<c0104baf>] kernel_thread_helper+0x7/0x10
   [<ffffffff>] 0xffffffff

the second lock's dependencies:
-> (&bond->lock){-.--} ops: 99 {
initial-use at:

                        [<c013fe35>] put_lock_stats+0xa/0x1e
                        [<c014289c>] __lock_acquire+0x4ba/0xc07
                        [<c0141dac>] mark_held_locks+0x39/0x53
                        [<c043a39d>] _spin_unlock_irqrestore+0x40/0x58
                        [<c0143062>] lock_acquire+0x79/0x93
                        [<c02edcc1>] bond_get_stats+0x28/0xd0
                        [<c0439ec6>] _read_lock_bh+0x3b/0x64
                        [<c02edcc1>] bond_get_stats+0x28/0xd0
                        [<c02edcc1>] bond_get_stats+0x28/0xd0
                        [<c03aa3bf>] rtnl_fill_ifinfo+0x2bf/0x563
                        [<c03aa93d>] rtmsg_ifinfo+0x5d/0xdf
                        [<c03aa9fe>] rtnetlink_event+0x3f/0x42
                        [<c013ac20>] notifier_call_chain+0x2a/0x52
                        [<c013ac6a>] raw_notifier_call_chain+0x17/0x1a
                        [<c03a31b6>] register_netdevice+0x2a7/0x2e7
                        [<c02ed862>] bond_create+0x1f2/0x26a
                        [<c05bedcd>] bonding_init+0x761/0x7ea
                        [<c05be635>] e1000_init_module+0x45/0x7c
                        [<c05a9499>] kernel_init+0x150/0x2b7
                        [<c05a9349>] kernel_init+0x0/0x2b7
                        [<c05a9349>] kernel_init+0x0/0x2b7
                        [<c0104baf>] kernel_thread_helper+0x7/0x10
                        [<ffffffff>] 0xffffffff
   hardirq-on-W at:
                        [<c014286a>] __lock_acquire+0x488/0xc07
                        [<c0141dac>] mark_held_locks+0x39/0x53
                        [<c043a3d5>] _spin_unlock_irq+0x20/0x41
                        [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                        [<c043a3e0>] _spin_unlock_irq+0x2b/0x41
                        [<c012208a>] finish_task_switch+0x50/0x8c
                        [<c0143062>] lock_acquire+0x79/0x93
                        [<c02eda75>] bond_set_multicast_list+0x1d/0x241
                        [<c0439dfd>] _write_lock_bh+0x3b/0x64
                        [<c02eda75>] bond_set_multicast_list+0x1d/0x241
                        [<c02eda75>] bond_set_multicast_list+0x1d/0x241
                        [<c013fe35>] put_lock_stats+0xa/0x1e
                        [<c03a14b3>] __dev_set_rx_mode+0x7b/0x7d
                        [<c03a164d>] dev_set_rx_mode+0x23/0x36
                        [<c03a3d28>] dev_open+0x5e/0x77
                        [<c03a29f7>] dev_change_flags+0x9d/0x14b
                        [<c03a17fb>] __dev_get_by_name+0x68/0x73
                        [<c03e3828>] devinet_ioctl+0x22b/0x536
                        [<c03a3b1d>] dev_ioctl+0x46f/0x5b7
                        [<c0399c50>] sock_ioctl+0x167/0x18b
                        [<c0399ae9>] sock_ioctl+0x0/0x18b
                        [<c01725f7>] do_ioctl+0x1f/0x62
                        [<c0172867>] vfs_ioctl+0x22d/0x23f
                        [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                        [<c01728ac>] sys_ioctl+0x33/0x4b
                        [<c0103e92>] sysenter_past_esp+0x5f/0xa5
                        [<ffffffff>] 0xffffffff
   softirq-on-R at:
                        [<c0141986>] mark_lock+0x64/0x451
                        [<c013575e>] __kernel_text_address+0x5/0xe
                        [<c0104ee2>] dump_trace+0x83/0x8d
                        [<c0142889>] __lock_acquire+0x4a7/0xc07
                        [<c013fc76>] save_trace+0x37/0x89
                        [<c0133c57>] run_workqueue+0x87/0x1b6
                        [<c0143062>] lock_acquire+0x79/0x93
                        [<c02eee5d>] bond_mii_monitor+0x19/0x85
                        [<c0439f25>] _read_lock+0x36/0x5f
                        [<c02eee5d>] bond_mii_monitor+0x19/0x85
                        [<c02eee5d>] bond_mii_monitor+0x19/0x85
                        [<c0133cab>] run_workqueue+0xdb/0x1b6
                        [<c0133c57>] run_workqueue+0x87/0x1b6
                        [<c02eee44>] bond_mii_monitor+0x0/0x85
                        [<c01346cb>] worker_thread+0x0/0x85
                        [<c0134744>] worker_thread+0x79/0x85
                        [<c0137179>] autoremove_wake_function+0x0/0x35
                        [<c01370c2>] kthread+0x38/0x5e
                        [<c013708a>] kthread+0x0/0x5e
                        [<c0104baf>] kernel_thread_helper+0x7/0x10
                        [<ffffffff>] 0xffffffff
   hardirq-on-R at:
                        [<c013fe0a>] get_lock_stats+0xd/0x2e
                        [<c013fe35>] put_lock_stats+0xa/0x1e
                        [<c0142844>] __lock_acquire+0x462/0xc07
                        [<c0141dac>] mark_held_locks+0x39/0x53
                        [<c043a39d>] _spin_unlock_irqrestore+0x40/0x58
                        [<c0143062>] lock_acquire+0x79/0x93
                        [<c02edcc1>] bond_get_stats+0x28/0xd0
                        [<c0439ec6>] _read_lock_bh+0x3b/0x64
                        [<c02edcc1>] bond_get_stats+0x28/0xd0
                        [<c02edcc1>] bond_get_stats+0x28/0xd0
                        [<c03aa3bf>] rtnl_fill_ifinfo+0x2bf/0x563
                        [<c03aa93d>] rtmsg_ifinfo+0x5d/0xdf
                        [<c03aa9fe>] rtnetlink_event+0x3f/0x42
                        [<c013ac20>] notifier_call_chain+0x2a/0x52
                        [<c013ac6a>] raw_notifier_call_chain+0x17/0x1a
                        [<c03a31b6>] register_netdevice+0x2a7/0x2e7
                        [<c02ed862>] bond_create+0x1f2/0x26a
                        [<c05bedcd>] bonding_init+0x761/0x7ea
                        [<c05be635>] e1000_init_module+0x45/0x7c
                        [<c05a9499>] kernel_init+0x150/0x2b7
                        [<c05a9349>] kernel_init+0x0/0x2b7
                        [<c05a9349>] kernel_init+0x0/0x2b7
                        [<c0104baf>] kernel_thread_helper+0x7/0x10
                        [<ffffffff>] 0xffffffff
 }
 ... key      at: [<c08777d0>] __key.32969+0x0/0x8
 -> (_xmit_ETHER){-...} ops: 8 {
    initial-use  at:
                          [<c014289c>] __lock_acquire+0x4ba/0xc07
                          [<c0143062>] lock_acquire+0x79/0x93
                          [<c03a5a79>] dev_mc_add+0x1a/0x6a
                          [<c0439d3a>] _spin_lock_bh+0x3b/0x64
                          [<c03a5a79>] dev_mc_add+0x1a/0x6a
                          [<c03a5a79>] dev_mc_add+0x1a/0x6a
                          [<c04120ec>] igmp6_group_added+0x56/0x11d
                          [<c0412480>] ipv6_dev_mc_inc+0x2a3/0x31c
                          [<c0410100>] igmp6_mc_seq_start+0x106/0x138
                          [<c04124b5>] ipv6_dev_mc_inc+0x2d8/0x31c
                          [<c04121dd>] ipv6_dev_mc_inc+0x0/0x31c
                          [<c040180c>] ipv6_add_dev+0x21c/0x24b
                          [<c040b055>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
                          [<c0401def>] addrconf_notify+0x60/0x7b7
                          [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                          [<c0141dac>] mark_held_locks+0x39/0x53
                          [<c043903e>] mutex_lock_nested+0x286/0x2ac
                          [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                          [<c043905c>] mutex_lock_nested+0x2a4/0x2ac
                          [<c03a3f25>] register_netdevice_notifier+0xe/0x126
                          [<c03a3f25>] register_netdevice_notifier+0xe/0x126
                          [<c03a3f60>] register_netdevice_notifier+0x49/0x126
                          [<c05c5bda>] addrconf_init+0xad/0x193
                          [<c05c5b48>] addrconf_init+0x1b/0x193
                          [<c05c5a20>] inet6_init+0x1f0/0x2ad
                          [<c05a9499>] kernel_init+0x150/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c0104baf>] kernel_thread_helper+0x7/0x10
                          [<ffffffff>] 0xffffffff
    hardirq-on-W at:
                          [<c0141986>] mark_lock+0x64/0x451
                          [<c014286a>] __lock_acquire+0x488/0xc07
                          [<c0143062>] lock_acquire+0x79/0x93
                          [<c03a5a79>] dev_mc_add+0x1a/0x6a
                          [<c0439d3a>] _spin_lock_bh+0x3b/0x64
                          [<c03a5a79>] dev_mc_add+0x1a/0x6a
                          [<c03a5a79>] dev_mc_add+0x1a/0x6a
                          [<c04120ec>] igmp6_group_added+0x56/0x11d
                          [<c0412480>] ipv6_dev_mc_inc+0x2a3/0x31c
                          [<c0410100>] igmp6_mc_seq_start+0x106/0x138
                          [<c04124b5>] ipv6_dev_mc_inc+0x2d8/0x31c
                          [<c04121dd>] ipv6_dev_mc_inc+0x0/0x31c
                          [<c040180c>] ipv6_add_dev+0x21c/0x24b
                          [<c040b055>] ndisc_ifinfo_sysctl_change+0x0/0x1ef
                          [<c0401def>] addrconf_notify+0x60/0x7b7
                          [<c0142fa1>] __lock_acquire+0xbbf/0xc07
                          [<c0141dac>] mark_held_locks+0x39/0x53
                          [<c043903e>] mutex_lock_nested+0x286/0x2ac
                          [<c0141f9f>] trace_hardirqs_on+0x122/0x14c
                          [<c043905c>] mutex_lock_nested+0x2a4/0x2ac
                          [<c03a3f25>] register_netdevice_notifier+0xe/0x126
                          [<c03a3f25>] register_netdevice_notifier+0xe/0x126
                          [<c03a3f60>] register_netdevice_notifier+0x49/0x126
                          [<c05c5bda>] addrconf_init+0xad/0x193
                          [<c05c5b48>] addrconf_init+0x1b/0x193
                          [<c05c5a20>] inet6_init+0x1f0/0x2ad
                          [<c05a9499>] kernel_init+0x150/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c0104baf>] kernel_thread_helper+0x7/0x10
                          [<ffffffff>] 0xffffffff
  }
  ... key      at: [<c087adc8>] netdev_xmit_lock_key+0x8/0x1c0
 ... acquired at:
   [<c0142dff>] __lock_acquire+0xa1d/0xc07
   [<c03a5a79>] dev_mc_add+0x1a/0x6a
   [<c043a39d>] _spin_unlock_irqrestore+0x40/0x58
   [<c0109ef2>] save_stack_trace+0x20/0x3a
   [<c0143062>] lock_acquire+0x79/0x93
   [<c03a5a79>] dev_mc_add+0x1a/0x6a
   [<c0439d3a>] _spin_lock_bh+0x3b/0x64
   [<c03a5a79>] dev_mc_add+0x1a/0x6a
   [<c03a5a79>] dev_mc_add+0x1a/0x6a
   [<c02ee492>] bond_change_active_slave+0x1a9/0x3bf
   [<c02ec7c3>] bond_update_speed_duplex+0x26/0x65
   [<c02ee9af>] bond_select_active_slave+0x95/0xcd
   [<c02ed22b>] bond_compute_features+0x45/0x84
   [<c02ef9be>] bond_enslave+0x6a7/0x884
   [<c043905c>] mutex_lock_nested+0x2a4/0x2ac
   [<c02f616d>] bonding_store_slaves+0x1ae/0x2fb
   [<c02f5fbf>] bonding_store_slaves+0x0/0x2fb
   [<c02ce8d7>] dev_attr_store+0x27/0x2c
   [<c019bcb9>] sysfs_write_file+0xad/0xe0
   [<c019bc0c>] sysfs_write_file+0x0/0xe0
   [<c0168ddc>] vfs_write+0x8a/0x10c
   [<c0118566>] do_page_fault+0x0/0x54a
   [<c0169361>] sys_write+0x41/0x67
   [<c0103e92>] sysenter_past_esp+0x5f/0xa5
   [<ffffffff>] 0xffffffff

 -> (lweventlist_lock){.+..} ops: 10 {
    initial-use  at:
                          [<c0141986>] mark_lock+0x64/0x451
                          [<c014289c>] __lock_acquire+0x4ba/0xc07
                          [<c02e365c>] e1000_read_phy_reg+0x1c7/0x1d3
                          [<c02e348b>] e1000_write_phy_reg+0xb9/0xc3
                          [<c024a7de>] delay_tsc+0x25/0x3b
                          [<c0143062>] lock_acquire+0x79/0x93
                          [<c03abacf>] linkwatch_add_event+0xd/0x2c
                          [<c043a057>] _spin_lock_irqsave+0x3f/0x6c
                          [<c03abacf>] linkwatch_add_event+0xd/0x2c
                          [<c03abacf>] linkwatch_add_event+0xd/0x2c
                          [<c03abb93>] linkwatch_fire_event+0x25/0x37
                          [<c02e1c43>] e1000_probe+0xad1/0xbe8
                          [<c0257f3f>] pci_device_probe+0x36/0x57
                          [<c02d0e5f>] driver_probe_device+0xe1/0x15f
                          [<c043a2d1>] _spin_unlock+0x25/0x3b
                          [<c043758a>] klist_next+0x58/0x6d
                          [<c02d0f6f>] __driver_attach+0x0/0x7f
                          [<c02d0fb8>] __driver_attach+0x49/0x7f
                          [<c02d0403>] bus_for_each_dev+0x36/0x58
                          [<c02d0cb7>] driver_attach+0x16/0x18
                          [<c02d0f6f>] __driver_attach+0x0/0x7f
                          [<c02d06fa>] bus_add_driver+0x6d/0x18d
                          [<c0258089>] __pci_register_driver+0x53/0x7f
                          [<c05be635>] e1000_init_module+0x45/0x7c
                          [<c05a9499>] kernel_init+0x150/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c05a9349>] kernel_init+0x0/0x2b7
                          [<c0104baf>] kernel_thread_helper+0x7/0x10
                          [<ffffffff>] 0xffffffff
    in-softirq-W at:
                          [<c011d20a>] __wake_up_common+0x32/0x5c
                          [<c0142822>] __lock_acquire+0x440/0xc07
                          [<c043a39d>] _spin_unlock_irqrestore+0x40/0x58
                          [<c0143062>] lock_acquire+0x79/0x93
                          [<c03abacf>] linkwatch_add_event+0xd/0x2c
                          [<c02dff01>] e1000_watchdog+0x0/0x5c9
                          [<c043a057>] _spin_lock_irqsave+0x3f/0x6c
                          [<c03abacf>] linkwatch_add_event+0xd/0x2c
                          [<c03abacf>] linkwatch_add_event+0xd/0x2c
                          [<c03abb93>] linkwatch_fire_event+0x25/0x37
                          [<c03aeb1a>] netif_carrier_on+0x16/0x27
                          [<c02e0156>] e1000_watchdog+0x255/0x5c9
                          [<c02dff01>] e1000_watchdog+0x0/0x5c9
                          [<c012df52>] run_timer_softirq+0xfa/0x15d
                          [<c012a8a6>] __do_softirq+0x56/0xdb
                          [<c0141f89>] trace_hardirqs_on+0x10c/0x14c
                          [<c012a8b8>] __do_softirq+0x68/0xdb
                          [<c012a961>] do_softirq+0x36/0x51
                          [<c012ae4a>] local_bh_enable_ip+0xad/0xed
                          [<c03bf0df>] rt_run_flush+0x64/0x8b
                          [<c03e7924>] ip_mc_inc_group+0x184/0x1c1
                          [<c03e79a2>] ip_mc_up+0x41/0x59
                          [<c03e32dc>] inetdev_event+0x257/0x465
                          [<c03aa8da>] rtnl_notify+0x3a/0x40
                          [<c03aa997>] rtmsg_ifinfo+0xb7/0xdf
                          [<c013ac20>] notifier_call_chain+0x2a/0x52
                          [<c013ac6a>] raw_notifier_call_chain+0x17/0x1a
                          [<c03a3d3b>] dev_open+0x71/0x77
                          [<c02ef626>] bond_enslave+0x30f/0x884
                          [<c043905c>] mutex_lock_nested+0x2a4/0x2ac
                          [<c02f6161>] bonding_store_slaves+0x1a2/0x2fb
                          [<c02f616d>] bonding_store_slaves+0x1ae/0x2fb
                          [<c02f5fbf>] bonding_store_slaves+0x0/0x2fb
                          [<c02ce8d7>] dev_attr_store+0x27/0x2c
                          [<c019bcb9>] sysfs_write_file+0xad/0xe0
                          [<c019bc0c>] sysfs_write_file+0x0/0xe0
                          [<c0168ddc>] vfs_write+0x8a/0x10c
                          [<c0118566>] do_page_fault+0x0/0x54a
                          [<c0169361>] sys_write+0x41/0x67
                          [<c0103e92>] sysenter_past_esp+0x5f/0xa5
                          [<ffffffff>] 0xffffffff
  }
  ... key      at: [<c058a194>] lweventlist_lock+0x14/0x40
 ... acquired at:
   [<c0142dff>] __lock_acquire+0xa1d/0xc07
   [<c03abacf>] linkwatch_add_event+0xd/0x2c
   [<c0142fa1>] __lock_acquire+0xbbf/0xc07
   [<c0143062>] lock_acquire+0x79/0x93
   [<c03abacf>] linkwatch_add_event+0xd/0x2c
   [<c043a057>] _spin_lock_irqsave+0x3f/0x6c
   [<c03abacf>] linkwatch_add_event+0xd/0x2c
   [<c03abacf>] linkwatch_add_event+0xd/0x2c
   [<c03abb93>] linkwatch_fire_event+0x25/0x37
   [<c03aeb1a>] netif_carrier_on+0x16/0x27
   [<c02ede2c>] bond_set_carrier+0x31/0x55
   [<c02ee9b6>] bond_select_active_slave+0x9c/0xcd
   [<c02ed22b>] bond_compute_features+0x45/0x84
   [<c02ef9be>] bond_enslave+0x6a7/0x884
   [<c043905c>] mutex_lock_nested+0x2a4/0x2ac
   [<c02f616d>] bonding_store_slaves+0x1ae/0x2fb
   [<c02f5fbf>] bonding_store_slaves+0x0/0x2fb
   [<c02ce8d7>] dev_attr_store+0x27/0x2c
   [<c019bcb9>] sysfs_write_file+0xad/0xe0
   [<c019bc0c>] sysfs_write_file+0x0/0xe0
   [<c0168ddc>] vfs_write+0x8a/0x10c
   [<c0118566>] do_page_fault+0x0/0x54a
   [<c0169361>] sys_write+0x41/0x67
   [<c0103e92>] sysenter_past_esp+0x5f/0xa5
   [<ffffffff>] 0xffffffff


stack backtrace:
Pid: 9, comm: events/0 Not tainted 2.6.24-rc5 #1
 [<c0140b38>] print_irq_inversion_bug+0x108/0x112
 [<c014191d>] check_usage_forwards+0x3c/0x41
 [<c0141b09>] mark_lock+0x1e7/0x451
 [<c0142822>] __lock_acquire+0x440/0xc07
 [<c011d686>] update_curr+0x52/0xc4
 [<c013f175>] tick_sched_timer+0x129/0x165
 [<c0143062>] lock_acquire+0x79/0x93
 [<c0411c52>] mld_ifc_timer_expire+0x130/0x1fb
 [<c0411b22>] mld_ifc_timer_expire+0x0/0x1fb
 [<c0439d3a>] _spin_lock_bh+0x3b/0x64
 [<c0411c52>] mld_ifc_timer_expire+0x130/0x1fb
 [<c0411c52>] mld_ifc_timer_expire+0x130/0x1fb
 [<c0411b22>] mld_ifc_timer_expire+0x0/0x1fb
 [<c0141f89>] trace_hardirqs_on+0x10c/0x14c
 [<c0411b22>] mld_ifc_timer_expire+0x0/0x1fb
 [<c012df52>] run_timer_softirq+0xfa/0x15d
 [<c012a8a6>] __do_softirq+0x56/0xdb
 [<c0141f89>] trace_hardirqs_on+0x10c/0x14c
 [<c012a8b8>] __do_softirq+0x68/0xdb
 [<c012a961>] do_softirq+0x36/0x51
 [<c012ae4a>] local_bh_enable_ip+0xad/0xed
 [<c03bf0df>] rt_run_flush+0x64/0x8b
 [<c03e926e>] fib_netdev_event+0x61/0x65
 [<c013ac20>] notifier_call_chain+0x2a/0x52
 [<c013ac6a>] raw_notifier_call_chain+0x17/0x1a
 [<c03a2abd>] netdev_state_change+0x18/0x29
 [<c03abcf5>] __linkwatch_run_queue+0x150/0x17e
 [<c03abd40>] linkwatch_event+0x1d/0x22
 [<c0133cab>] run_workqueue+0xdb/0x1b6
 [<c0133c57>] run_workqueue+0x87/0x1b6
 [<c03abd23>] linkwatch_event+0x0/0x22
 [<c01346cb>] worker_thread+0x0/0x85
 [<c0134744>] worker_thread+0x79/0x85
 [<c0137179>] autoremove_wake_function+0x0/0x35
 [<c01370c2>] kthread+0x38/0x5e
 [<c013708a>] kthread+0x0/0x5e
 [<c0104baf>] kernel_thread_helper+0x7/0x10
 =======================

bonding: bond0: enslaving eth1 as a backup interface with a down link.
bond0: no IPv6 routers present

Best regards,

Krzysztof Olędzki

Andy Gospodarek

2007-12-14 22:47 UTC #14 lore raw

In reply to Krzysztof Oledzki

Quoted 107 lines · Krzysztof Oledzki

Quoted 103 lines · Andy Gospodarek
Quoted 102 lines · Krzysztof Oledzki
Quoted 83 lines · Jay Vosburgh
Quoted 8 lines · Herbert Xu
Quoted 8 lines
diff -puN drivers/net/bonding/bond_sysfs.c~bonding-locking-fix drivers/net/bonding/bond_sysfs.c
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -0
@@ -1111,8 +1111,6 @@ static ssize_t bonding_store_primary(str
out:
     write_unlock_bh(&bond->lock);
Quoted 6 lines
drivers/net/bonding/bond_sysfs.c
Quoted 2 lines
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -2
-       rtnl_unlock();
-
Looking at the changeset that added this perhaps the intention is to hold the lock? If so we should add an rtnl_lock to the start of the function.
Yes, this function needs to hold locks, and more than just
what's there now. I believe the following should be correct; I haven't tested it, though (I'm supposedly on vacation right now).
The following change should be correct for the
bonding_store_primary case discussed in this thread, and also corrects the bonding_store_active case which performs similar functions.
The bond_change_active_slave and bond_select_active_slave
functions both require rtnl, bond->lock for read and curr_slave_lock for write_bh, and no other locks. This is so that the lower level mode-specific functions can release locks down to just rtnl in order to call, e.g., dev_set_mac_address with the locks it expects (rtnl only).
Signed-off-by Jay Vosburgh <[email protected]>
drivers/net/bonding/bond_sysfs.c +4 -1
index 11b76b3..28a2d80 100644
@@ -1075,7 +1075,10 @@ static ssize_t bonding_store_primary(struct device
 *d,
 	struct slave *slave;
 	struct bonding *bond = to_bond(d);
 
-	write_lock_bh(&bond->lock);
+	rtnl_lock();
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
+
 	if (!USES_PRIMARY(bond->params.mode)) {
 		printk(KERN_INFO DRV_NAME
		       ": %s: Unable to set primary slave; %s is in mode
		       %d\n",
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1109,8 +1112,8 @@ static ssize_t bonding_store_primary(struct device
 *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +2 -2
-	write_unlock_bh(&bond->lock);
-
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
	rtnl_unlock();

	return count;
drivers/net/bonding/bond_sysfs.c +2 -1
@@ -1190,7 +1193,8 @@ static ssize_t bonding_store_active_slave(struct
 device *d,
 	struct bonding *bond = to_bond(d);
 
 	rtnl_lock();
-	write_lock_bh(&bond->lock);
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
 
 	if (!USES_PRIMARY(bond->params.mode)) {
printk(KERN_INFO DRV_NAME
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1247,7 +1251,8 @@ static ssize_t bonding_store_active_slave(struct
 device *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +2 -1
-	write_unlock_bh(&bond->lock);
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
Quoted 3 lines
	rtnl_unlock();

	return count;
Vanilla 2.6.24-rc5 plus this patch:
=========================================================
[ INFO: possible irq lock inversion dependency detected ] 2.6.24-rc5 #1
---------------------------------------------------------
events/0/9 just changed the state of lock:
(&mc->mca_lock){-+..}, at: [<c0411c7a>] mld_ifc_timer_expire+0x130/0x1fb but this lock took another, soft-read-irq-unsafe lock in the past:
(&bond->lock){-.--}

Quoted 9 lines

Quoted 5 lines
Quoted 3 lines
and interrupts could create inverse lock ordering between them.
Grrr, I should have seen that -- sorry. Try your luck with this instead:
<CUT>
No luck.

I'm guessing if we go back to using a write-lock for bond->lock this will go back to working again, but I'm not totally convinced since there are plenty of places where we used a read-lock with it.

drivers/net/bonding/bond_sysfs.c +6 -1

index 11b76b3..635b857 100644
@@ -1075,7 +1075,10 @@ static ssize_t bonding_store_primary(struct device *d,
 	struct slave *slave;
 	struct bonding *bond = to_bond(d);
 
+	rtnl_lock();
 	write_lock_bh(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
+
 	if (!USES_PRIMARY(bond->params.mode)) {
 		printk(KERN_INFO DRV_NAME
 		       ": %s: Unable to set primary slave; %s is in mode %d\n",

@@ -1109,8 +1112,8 @@ static ssize_t bonding_store_primary(struct device *d,
 		}
 	}
 out:
+	write_unlock_bh(&bond->curr_slave_lock);
 	write_unlock_bh(&bond->lock);
-
 	rtnl_unlock();
 
 	return count;

@@ -1191,6 +1194,7 @@ static ssize_t bonding_store_active_slave(struct device *d,
 
 	rtnl_lock();
 	write_lock_bh(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
 
 	if (!USES_PRIMARY(bond->params.mode)) {
 		printk(KERN_INFO DRV_NAME

@@ -1247,6 +1251,7 @@ static ssize_t bonding_store_active_slave(struct device *d,
 		}
 	}
 out:
+	write_unlock_bh(&bond->curr_slave_lock);
 	write_unlock_bh(&bond->lock);
 	rtnl_unlock();

Herbert Xu

2007-12-15 04:11 UTC #15 lore raw

Quoted 4 lines · Andy Gospodarek

I'm guessing if we go back to using a write-lock for bond->lock this will go back to working again, but I'm not totally convinced since there are plenty of places where we used a read-lock with it.

Sorry I forgot to cc you earlier Andy.

But to fix this you need make sure that all read locks on bond->lock in process context disable BH. This is because at least one write lock can be taken from BH context.

You don't need to turn the read locks into write locks however.

This is also something that we can undo once the set_multicast interface has been fixed to not take the tx lock.

Cheers,

Signature · 4 lines

Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Andy Gospodarek

2007-12-15 15:09 UTC #16 lore raw

Quoted 11 lines · Herbert Xu

Quoted 4 lines · Andy Gospodarek
I'm guessing if we go back to using a write-lock for bond->lock this will go back to working again, but I'm not totally convinced since there are plenty of places where we used a read-lock with it.
Sorry I forgot to cc you earlier Andy.
But to fix this you need make sure that all read locks on bond->lock in process context disable BH. This is because at least one write lock can be taken from BH context.

I agree with you completely, Herbet, which is why I was surprised that my first apparently did not resolve the issue. I felt it should have....

Quoted 12 lines

You don't need to turn the read locks into write locks however.
This is also something that we can undo once the set_multicast interface has been fixed to not take the tx lock.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Herbert Xu

2007-12-16 02:28 UTC #17 lore raw

Quoted 4 lines · Andy Gospodarek

I agree with you completely, Herbet, which is why I was surprised that my first apparently did not resolve the issue. I felt it should have....

Did it change all occurrences of read_lock(&bond->lock) to read_lock_bh? If so I better look at the lockdep output again.

Cheers,

Signature · 4 lines

Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Andy Gospodarek

2007-12-16 03:17 UTC #18 lore raw

Quoted 9 lines · Herbert Xu

Quoted 4 lines · Andy Gospodarek
I agree with you completely, Herbet, which is why I was surprised that my first apparently did not resolve the issue. I felt it should have....
Did it change all occurrences of read_lock(&bond->lock) to read_lock_bh? If so I better look at the lockdep output again.

Not all of them in the bonding code, but all two of them in the small patch.

Herbert Xu

2007-12-16 03:23 UTC #19 lore raw

Quoted 2 lines · Andy Gospodarek

Not all of them in the bonding code, but all two of them in the small patch.

OK, we need to change all of the ones that may be called from process context with BH on.

Cheers,

Signature · 4 lines

Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Krzysztof Oledzki

2007-12-18 20:01 UTC #20 lore raw

In reply to Andy Gospodarek

Quoted 115 lines · Andy Gospodarek

Quoted 114 lines · Krzysztof Oledzki
Quoted 108 lines · Andy Gospodarek
Quoted 102 lines · Krzysztof Oledzki
Quoted 83 lines · Jay Vosburgh
Quoted 8 lines · Herbert Xu
Quoted 8 lines
diff -puN drivers/net/bonding/bond_sysfs.c~bonding-locking-fix drivers/net/bonding/bond_sysfs.c
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -0
@@ -1111,8 +1111,6 @@ static ssize_t bonding_store_primary(str
out:
     write_unlock_bh(&bond->lock);
Quoted 6 lines
drivers/net/bonding/bond_sysfs.c
Quoted 2 lines
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -2
-       rtnl_unlock();
-
Looking at the changeset that added this perhaps the intention is to hold the lock? If so we should add an rtnl_lock to the start of the function.
Yes, this function needs to hold locks, and more than just
what's there now. I believe the following should be correct; I haven't tested it, though (I'm supposedly on vacation right now).
The following change should be correct for the
bonding_store_primary case discussed in this thread, and also corrects the bonding_store_active case which performs similar functions.
The bond_change_active_slave and bond_select_active_slave
functions both require rtnl, bond->lock for read and curr_slave_lock for write_bh, and no other locks. This is so that the lower level mode-specific functions can release locks down to just rtnl in order to call, e.g., dev_set_mac_address with the locks it expects (rtnl only).
Signed-off-by Jay Vosburgh <[email protected]>
drivers/net/bonding/bond_sysfs.c +4 -1
index 11b76b3..28a2d80 100644
@@ -1075,7 +1075,10 @@ static ssize_t bonding_store_primary(struct device
 *d,
 	struct slave *slave;
 	struct bonding *bond = to_bond(d);
 
-	write_lock_bh(&bond->lock);
+	rtnl_lock();
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
+
 	if (!USES_PRIMARY(bond->params.mode)) {
 		printk(KERN_INFO DRV_NAME
		       ": %s: Unable to set primary slave; %s is in mode
		       %d\n",
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1109,8 +1112,8 @@ static ssize_t bonding_store_primary(struct device
 *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +2 -2
-	write_unlock_bh(&bond->lock);
-
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
	rtnl_unlock();

	return count;
drivers/net/bonding/bond_sysfs.c +2 -1
@@ -1190,7 +1193,8 @@ static ssize_t bonding_store_active_slave(struct
 device *d,
 	struct bonding *bond = to_bond(d);
 
 	rtnl_lock();
-	write_lock_bh(&bond->lock);
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
 
 	if (!USES_PRIMARY(bond->params.mode)) {
printk(KERN_INFO DRV_NAME
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1247,7 +1251,8 @@ static ssize_t bonding_store_active_slave(struct
 device *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +2 -1
-	write_unlock_bh(&bond->lock);
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
Quoted 3 lines
	rtnl_unlock();

	return count;
Vanilla 2.6.24-rc5 plus this patch:
=========================================================
[ INFO: possible irq lock inversion dependency detected ] 2.6.24-rc5 #1
---------------------------------------------------------
events/0/9 just changed the state of lock:
(&mc->mca_lock){-+..}, at: [<c0411c7a>] mld_ifc_timer_expire+0x130/0x1fb but this lock took another, soft-read-irq-unsafe lock in the past:
(&bond->lock){-.--}
Quoted 3 lines
and interrupts could create inverse lock ordering between them.
Grrr, I should have seen that -- sorry. Try your luck with this instead:
<CUT>

Quoted 7 lines

Quoted 2 lines
No luck.
I'm guessing if we go back to using a write-lock for bond->lock this will go back to working again, but I'm not totally convinced since there are plenty of places where we used a read-lock with it.

Should I check this patch or rather, based on a future discussion, wait for another version?

Quoted 39 lines

drivers/net/bonding/bond_sysfs.c +3 -0

index 11b76b3..635b857 100644
@@ -1075,7 +1075,10 @@ static ssize_t bonding_store_primary(struct device *d,
 	struct slave *slave;
 	struct bonding *bond = to_bond(d);
 
+	rtnl_lock();
 	write_lock_bh(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
+
 	if (!USES_PRIMARY(bond->params.mode)) {
 		printk(KERN_INFO DRV_NAME
 		       ": %s: Unable to set primary slave; %s is in mode %d\n",

@@ -1109,8 +1112,8 @@ static ssize_t bonding_store_primary(struct device *d,
 		}
 	}

out:

drivers/net/bonding/bond_sysfs.c +1 -0

+	write_unlock_bh(&bond->curr_slave_lock);

	write_unlock_bh(&bond->lock);
-
	rtnl_unlock();

	return count;

drivers/net/bonding/bond_sysfs.c +1 -0

@@ -1191,6 +1194,7 @@ static ssize_t bonding_store_active_slave(struct device *d,
 
 	rtnl_lock();
 	write_lock_bh(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
 
 	if (!USES_PRIMARY(bond->params.mode)) {
 		printk(KERN_INFO DRV_NAME

@@ -1247,6 +1251,7 @@ static ssize_t bonding_store_active_slave(struct device *d,
 		}
 	}

out:

drivers/net/bonding/bond_sysfs.c +1 -0

+	write_unlock_bh(&bond->curr_slave_lock);

Quoted 3 lines

	write_unlock_bh(&bond->lock);
	rtnl_unlock();

Best regards,

Krzysztof Olędzki

Andy Gospodarek

2007-12-19 14:42 UTC #21 lore raw

Quoted 185 lines · Krzysztof Oledzki

Quoted 118 lines · Andy Gospodarek
Quoted 117 lines · Krzysztof Oledzki
Quoted 111 lines · Andy Gospodarek
Quoted 105 lines · Krzysztof Oledzki
Quoted 86 lines · Jay Vosburgh
Quoted 8 lines · Herbert Xu
Quoted 8 lines
diff -puN drivers/net/bonding/bond_sysfs.c~bonding-locking-fix drivers/net/bonding/bond_sysfs.c
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -0
@@ -1111,8 +1111,6 @@ static ssize_t bonding_store_primary(str
out:
    write_unlock_bh(&bond->lock);
Quoted 6 lines
drivers/net/bonding/bond_sysfs.c
Quoted 2 lines
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -2
-       rtnl_unlock();
-
Looking at the changeset that added this perhaps the intention is to hold the lock? If so we should add an rtnl_lock to the start of the function.
Yes, this function needs to hold locks, and more than just
what's there now. I believe the following should be correct; I haven't tested it, though (I'm supposedly on vacation right now).
The following change should be correct for the
bonding_store_primary case discussed in this thread, and also corrects the bonding_store_active case which performs similar functions.
The bond_change_active_slave and bond_select_active_slave
functions both require rtnl, bond->lock for read and curr_slave_lock for
write_bh, and no other locks. This is so that the lower level mode-specific functions can release locks down to just rtnl in order to call, e.g., dev_set_mac_address with the locks it expects (rtnl only).
Signed-off-by Jay Vosburgh <[email protected]>
drivers/net/bonding/bond_sysfs.c +0 -0
index 11b76b3..28a2d80 100644
@@ -1075,7 +1075,10 @@ static ssize_t bonding_store_primary(struct 
device
*d,
	struct slave *slave;
	struct bonding *bond = to_bond(d);
drivers/net/bonding/bond_sysfs.c +4 -1
-	write_lock_bh(&bond->lock);
+	rtnl_lock();
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
+
	if (!USES_PRIMARY(bond->params.mode)) {
		printk(KERN_INFO DRV_NAME
		       ": %s: Unable to set primary slave; %s is in mode
		       %d\n",
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1109,8 +1112,8 @@ static ssize_t bonding_store_primary(struct 
device
*d,
		}
	}
out:
drivers/net/bonding/bond_sysfs.c +2 -2
-	write_unlock_bh(&bond->lock);
-
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
	rtnl_unlock();

	return count;
drivers/net/bonding/bond_sysfs.c +2 -1
@@ -1190,7 +1193,8 @@ static ssize_t bonding_store_active_slave(struct
 device *d,
 	struct bonding *bond = to_bond(d);
 
 	rtnl_lock();
-	write_lock_bh(&bond->lock);
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
 
 	if (!USES_PRIMARY(bond->params.mode)) {
printk(KERN_INFO DRV_NAME
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1247,7 +1251,8 @@ static ssize_t bonding_store_active_slave(struct
 device *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +2 -1
-	write_unlock_bh(&bond->lock);
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
Quoted 3 lines
	rtnl_unlock();

	return count;
Vanilla 2.6.24-rc5 plus this patch:
=========================================================
[ INFO: possible irq lock inversion dependency detected ] 2.6.24-rc5 #1
---------------------------------------------------------
events/0/9 just changed the state of lock:
(&mc->mca_lock){-+..}, at: [<c0411c7a>] mld_ifc_timer_expire+0x130/0x1fb but this lock took another, soft-read-irq-unsafe lock in the past:
(&bond->lock){-.--}
Quoted 3 lines
and interrupts could create inverse lock ordering between them.
Grrr, I should have seen that -- sorry. Try your luck with this instead:
<CUT>
Quoted 7 lines
Quoted 2 lines
No luck.
I'm guessing if we go back to using a write-lock for bond->lock this will go back to working again, but I'm not totally convinced since there are plenty of places where we used a read-lock with it.
Should I check this patch or rather, based on a future discussion, wait for another version?
Quoted 45 lines
drivers/net/bonding/bond_sysfs.c +3 -0
index 11b76b3..635b857 100644
@@ -1075,7 +1075,10 @@ static ssize_t bonding_store_primary(struct device 
 *d,
 	struct slave *slave;
 	struct bonding *bond = to_bond(d);
 
+	rtnl_lock();
 	write_lock_bh(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
+
 	if (!USES_PRIMARY(bond->params.mode)) {
 		printk(KERN_INFO DRV_NAME
		       ": %s: Unable to set primary slave; %s is in mode 
		       %d\n",
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1109,8 +1112,8 @@ static ssize_t bonding_store_primary(struct device 
 *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +1 -0
+	write_unlock_bh(&bond->curr_slave_lock);
	write_unlock_bh(&bond->lock);
-
	rtnl_unlock();

	return count;
drivers/net/bonding/bond_sysfs.c +1 -0
@@ -1191,6 +1194,7 @@ static ssize_t bonding_store_active_slave(struct 
 device *d,
 
 	rtnl_lock();
 	write_lock_bh(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
 
 	if (!USES_PRIMARY(bond->params.mode)) {
printk(KERN_INFO DRV_NAME
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1247,6 +1251,7 @@ static ssize_t bonding_store_active_slave(struct 
 device *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +1 -0
+	write_unlock_bh(&bond->curr_slave_lock);
Quoted 3 lines
	write_unlock_bh(&bond->lock);
	rtnl_unlock();
Best regards,

Quoted 1 line

Krzysztof Olędzki

For now, I prefer Jay's original patch -- with the read_locks (rather than read/write_lock_bh) and the added rtnl_lock. There is still a lockdep issue that we need to sort-out, but this patch is needed first.

Krzysztof Oledzki

2008-01-07 18:14 UTC #22 lore raw

Quoted 186 lines · Andy Gospodarek

Quoted 185 lines · Krzysztof Oledzki
Quoted 118 lines · Andy Gospodarek
Quoted 117 lines · Krzysztof Oledzki
Quoted 111 lines · Andy Gospodarek
Quoted 105 lines · Krzysztof Oledzki
Quoted 86 lines · Jay Vosburgh
Quoted 8 lines · Herbert Xu
Quoted 8 lines
diff -puN drivers/net/bonding/bond_sysfs.c~bonding-locking-fix drivers/net/bonding/bond_sysfs.c
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -0
@@ -1111,8 +1111,6 @@ static ssize_t bonding_store_primary(str
out:
    write_unlock_bh(&bond->lock);
Quoted 6 lines
drivers/net/bonding/bond_sysfs.c
Quoted 2 lines
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -2
-       rtnl_unlock();
-
Looking at the changeset that added this perhaps the intention is to hold the lock? If so we should add an rtnl_lock to the start of the function.
Yes, this function needs to hold locks, and more than just
what's there now. I believe the following should be correct; I haven't tested it, though (I'm supposedly on vacation right now).
The following change should be correct for the
bonding_store_primary case discussed in this thread, and also corrects the bonding_store_active case which performs similar functions.
The bond_change_active_slave and bond_select_active_slave
functions both require rtnl, bond->lock for read and curr_slave_lock for
write_bh, and no other locks. This is so that the lower level mode-specific functions can release locks down to just rtnl in order to call, e.g., dev_set_mac_address with the locks it expects (rtnl only).
Signed-off-by Jay Vosburgh <[email protected]>
drivers/net/bonding/bond_sysfs.c +0 -0
index 11b76b3..28a2d80 100644
@@ -1075,7 +1075,10 @@ static ssize_t bonding_store_primary(struct
device
*d,
	struct slave *slave;
	struct bonding *bond = to_bond(d);
drivers/net/bonding/bond_sysfs.c +4 -1
-	write_lock_bh(&bond->lock);
+	rtnl_lock();
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
+
	if (!USES_PRIMARY(bond->params.mode)) {
		printk(KERN_INFO DRV_NAME
		       ": %s: Unable to set primary slave; %s is in mode
		       %d\n",
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1109,8 +1112,8 @@ static ssize_t bonding_store_primary(struct
device
*d,
		}
	}
out:
drivers/net/bonding/bond_sysfs.c +2 -2
-	write_unlock_bh(&bond->lock);
-
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
	rtnl_unlock();

	return count;
drivers/net/bonding/bond_sysfs.c +2 -1
@@ -1190,7 +1193,8 @@ static ssize_t bonding_store_active_slave(struct
 device *d,
 	struct bonding *bond = to_bond(d);
 
 	rtnl_lock();
-	write_lock_bh(&bond->lock);
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
 
 	if (!USES_PRIMARY(bond->params.mode)) {
printk(KERN_INFO DRV_NAME
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1247,7 +1251,8 @@ static ssize_t bonding_store_active_slave(struct
 device *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +2 -1
-	write_unlock_bh(&bond->lock);
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
Quoted 3 lines
	rtnl_unlock();

	return count;
Vanilla 2.6.24-rc5 plus this patch:
=========================================================
[ INFO: possible irq lock inversion dependency detected ] 2.6.24-rc5 #1
---------------------------------------------------------
events/0/9 just changed the state of lock:
(&mc->mca_lock){-+..}, at: [<c0411c7a>] mld_ifc_timer_expire+0x130/0x1fb but this lock took another, soft-read-irq-unsafe lock in the past:
(&bond->lock){-.--}
Quoted 3 lines
and interrupts could create inverse lock ordering between them.
Grrr, I should have seen that -- sorry. Try your luck with this instead:
<CUT>
Quoted 7 lines
Quoted 2 lines
No luck.
I'm guessing if we go back to using a write-lock for bond->lock this will go back to working again, but I'm not totally convinced since there are plenty of places where we used a read-lock with it.
Should I check this patch or rather, based on a future discussion, wait for another version?
Quoted 45 lines
drivers/net/bonding/bond_sysfs.c +3 -0
index 11b76b3..635b857 100644
@@ -1075,7 +1075,10 @@ static ssize_t bonding_store_primary(struct device
 *d,
 	struct slave *slave;
 	struct bonding *bond = to_bond(d);
 
+	rtnl_lock();
 	write_lock_bh(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
+
 	if (!USES_PRIMARY(bond->params.mode)) {
 		printk(KERN_INFO DRV_NAME
		       ": %s: Unable to set primary slave; %s is in mode
		       %d\n",
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1109,8 +1112,8 @@ static ssize_t bonding_store_primary(struct device
 *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +1 -0
+	write_unlock_bh(&bond->curr_slave_lock);
	write_unlock_bh(&bond->lock);
-
	rtnl_unlock();

	return count;
drivers/net/bonding/bond_sysfs.c +1 -0
@@ -1191,6 +1194,7 @@ static ssize_t bonding_store_active_slave(struct
 device *d,
 
 	rtnl_lock();
 	write_lock_bh(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
 
 	if (!USES_PRIMARY(bond->params.mode)) {
printk(KERN_INFO DRV_NAME
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1247,6 +1251,7 @@ static ssize_t bonding_store_active_slave(struct
 device *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +1 -0
+	write_unlock_bh(&bond->curr_slave_lock);
Quoted 3 lines
	write_unlock_bh(&bond->lock);
	rtnl_unlock();
Best regards,

Quoted 5 lines

Quoted 1 line
Krzysztof Olędzki
For now, I prefer Jay's original patch -- with the read_locks (rather than read/write_lock_bh) and the added rtnl_lock. There is still a lockdep issue that we need to sort-out, but this patch is needed first.

This bug has not been fixed yet as it still exists in 2.6.24-rc7. Any chances to cure it before 2.6.24-final?

Best regards,

Krzysztof Olędzki

Andy Gospodarek

2008-01-07 20:30 UTC #23 lore raw

Quoted 205 lines · Krzysztof Oledzki

Quoted 195 lines · Andy Gospodarek
Quoted 194 lines · Krzysztof Oledzki
Quoted 127 lines · Andy Gospodarek
Quoted 126 lines · Krzysztof Oledzki
Quoted 120 lines · Andy Gospodarek
Quoted 113 lines · Krzysztof Oledzki
Quoted 93 lines · Jay Vosburgh
Quoted 8 lines · Herbert Xu
Quoted 8 lines
diff -puN drivers/net/bonding/bond_sysfs.c~bonding-locking-fix drivers/net/bonding/bond_sysfs.c
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -0
@@ -1111,8 +1111,6 @@ static ssize_t bonding_store_primary(str
out:
   write_unlock_bh(&bond->lock);
Quoted 6 lines
drivers/net/bonding/bond_sysfs.c
Quoted 2 lines
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -2
-       rtnl_unlock();
-
Looking at the changeset that added this perhaps the intention is to hold the lock? If so we should add an rtnl_lock to the start of the function.
Yes, this function needs to hold locks, and more than just
what's there now. I believe the following should be correct; I haven't
tested it, though (I'm supposedly on vacation right now).
The following change should be correct for the
bonding_store_primary case discussed in this thread, and also corrects
the bonding_store_active case which performs similar functions.
The bond_change_active_slave and bond_select_active_slave
functions both require rtnl, bond->lock for read and curr_slave_lock for
write_bh, and no other locks. This is so that the lower level mode-specific functions can release locks down to just rtnl in order to
call, e.g., dev_set_mac_address with the locks it expects (rtnl only).
Signed-off-by Jay Vosburgh <[email protected]>
drivers/net/bonding/bond_sysfs.c +0 -0
index 11b76b3..28a2d80 100644
@@ -1075,7 +1075,10 @@ static ssize_t bonding_store_primary(struct
device
*d,
	struct slave *slave;
	struct bonding *bond = to_bond(d);
drivers/net/bonding/bond_sysfs.c +4 -1
-	write_lock_bh(&bond->lock);
+	rtnl_lock();
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
+
	if (!USES_PRIMARY(bond->params.mode)) {
		printk(KERN_INFO DRV_NAME
		       ": %s: Unable to set primary slave; %s is in 
		       mode
		       %d\n",
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1109,8 +1112,8 @@ static ssize_t bonding_store_primary(struct
device
*d,
		}
	}
out:
drivers/net/bonding/bond_sysfs.c +2 -2
-	write_unlock_bh(&bond->lock);
-
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
	rtnl_unlock();

	return count;
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1190,7 +1193,8 @@ static ssize_t 
bonding_store_active_slave(struct
device *d,
	struct bonding *bond = to_bond(d);

	rtnl_lock();
drivers/net/bonding/bond_sysfs.c +2 -1
-	write_lock_bh(&bond->lock);
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
	if (!USES_PRIMARY(bond->params.mode)) {
		printk(KERN_INFO DRV_NAME
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1247,7 +1251,8 @@ static ssize_t 
bonding_store_active_slave(struct
device *d,
		}
	}
out:
drivers/net/bonding/bond_sysfs.c +2 -1
-	write_unlock_bh(&bond->lock);
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
Quoted 3 lines
	rtnl_unlock();

	return count;
Vanilla 2.6.24-rc5 plus this patch:
=========================================================
[ INFO: possible irq lock inversion dependency detected ] 2.6.24-rc5 #1
---------------------------------------------------------
events/0/9 just changed the state of lock:
(&mc->mca_lock){-+..}, at: [<c0411c7a>] mld_ifc_timer_expire+0x130/0x1fb
but this lock took another, soft-read-irq-unsafe lock in the past:
(&bond->lock){-.--}
Quoted 3 lines
and interrupts could create inverse lock ordering between them.
Grrr, I should have seen that -- sorry. Try your luck with this instead:
<CUT>
Quoted 7 lines
Quoted 2 lines
No luck.
I'm guessing if we go back to using a write-lock for bond->lock this will go back to working again, but I'm not totally convinced since there are plenty of places where we used a read-lock with it.
Should I check this patch or rather, based on a future discussion, wait for another version?
Quoted 45 lines
drivers/net/bonding/bond_sysfs.c +3 -0
index 11b76b3..635b857 100644
@@ -1075,7 +1075,10 @@ static ssize_t bonding_store_primary(struct device
 *d,
 	struct slave *slave;
 	struct bonding *bond = to_bond(d);
 
+	rtnl_lock();
 	write_lock_bh(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
+
 	if (!USES_PRIMARY(bond->params.mode)) {
 		printk(KERN_INFO DRV_NAME
		       ": %s: Unable to set primary slave; %s is in mode
		       %d\n",
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1109,8 +1112,8 @@ static ssize_t bonding_store_primary(struct device
 *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +1 -0
+	write_unlock_bh(&bond->curr_slave_lock);
	write_unlock_bh(&bond->lock);
-
	rtnl_unlock();

	return count;
drivers/net/bonding/bond_sysfs.c +1 -0
@@ -1191,6 +1194,7 @@ static ssize_t bonding_store_active_slave(struct
 device *d,
 
 	rtnl_lock();
 	write_lock_bh(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
 
 	if (!USES_PRIMARY(bond->params.mode)) {
printk(KERN_INFO DRV_NAME
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1247,6 +1251,7 @@ static ssize_t bonding_store_active_slave(struct
 device *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +1 -0
+	write_unlock_bh(&bond->curr_slave_lock);
Quoted 3 lines
	write_unlock_bh(&bond->lock);
	rtnl_unlock();
Best regards,
Quoted 5 lines
Quoted 1 line
Krzysztof Olędzki
For now, I prefer Jay's original patch -- with the read_locks (rather than read/write_lock_bh) and the added rtnl_lock. There is still a lockdep issue that we need to sort-out, but this patch is needed first.

Quoted 6 lines

This bug has not been fixed yet as it still exists in 2.6.24-rc7. Any chances to cure it before 2.6.24-final?
Best regards,
Krzysztof Olędzki

Krzysztof,

I doubt the lockdep issue will be fixed, but the patch Jay posted and I acked needs to be included in 2.6.24.

I played around with the locking when setting the multicast list and I can make the lockdep issue go away, but I need to be sure that it's OK to switch it to a read-lock from a write-lock (and I don't really think it is).

-andy

Jay Vosburgh

2008-01-07 20:40 UTC #24 lore raw

Quoted 209 lines · Andy Gospodarek

Quoted 208 lines · Krzysztof Oledzki
Quoted 195 lines · Andy Gospodarek
Quoted 194 lines · Krzysztof Oledzki
Quoted 127 lines · Andy Gospodarek
Quoted 126 lines · Krzysztof Oledzki
Quoted 120 lines · Andy Gospodarek
Quoted 113 lines · Krzysztof Oledzki
Quoted 93 lines · Jay Vosburgh
Quoted 8 lines · Herbert Xu
Quoted 8 lines
diff -puN drivers/net/bonding/bond_sysfs.c~bonding-locking-fix drivers/net/bonding/bond_sysfs.c
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -0
@@ -1111,8 +1111,6 @@ static ssize_t bonding_store_primary(str
out:
   write_unlock_bh(&bond->lock);
Quoted 6 lines
drivers/net/bonding/bond_sysfs.c
Quoted 2 lines
drivers/net/bonding/bond_sysfs.c renamed from drivers/net/bonding/bond_sysfs.c~bonding-locking-fix +0 -2
-       rtnl_unlock();
-
Looking at the changeset that added this perhaps the intention is to hold the lock? If so we should add an rtnl_lock to the start of the function.
Yes, this function needs to hold locks, and more than just
what's there now. I believe the following should be correct; I haven't
tested it, though (I'm supposedly on vacation right now).
The following change should be correct for the
bonding_store_primary case discussed in this thread, and also corrects
the bonding_store_active case which performs similar functions.
The bond_change_active_slave and bond_select_active_slave
functions both require rtnl, bond->lock for read and curr_slave_lock for
write_bh, and no other locks. This is so that the lower level mode-specific functions can release locks down to just rtnl in order to
call, e.g., dev_set_mac_address with the locks it expects (rtnl only).
Signed-off-by Jay Vosburgh <[email protected]>
drivers/net/bonding/bond_sysfs.c +0 -0
index 11b76b3..28a2d80 100644
@@ -1075,7 +1075,10 @@ static ssize_t bonding_store_primary(struct
device
*d,
	struct slave *slave;
	struct bonding *bond = to_bond(d);
drivers/net/bonding/bond_sysfs.c +4 -1
-	write_lock_bh(&bond->lock);
+	rtnl_lock();
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
+
	if (!USES_PRIMARY(bond->params.mode)) {
		printk(KERN_INFO DRV_NAME
		       ": %s: Unable to set primary slave; %s is in 
		       mode
		       %d\n",
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1109,8 +1112,8 @@ static ssize_t bonding_store_primary(struct
device
*d,
		}
	}
out:
drivers/net/bonding/bond_sysfs.c +2 -2
-	write_unlock_bh(&bond->lock);
-
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
	rtnl_unlock();

	return count;
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1190,7 +1193,8 @@ static ssize_t 
bonding_store_active_slave(struct
device *d,
	struct bonding *bond = to_bond(d);

	rtnl_lock();
drivers/net/bonding/bond_sysfs.c +2 -1
-	write_lock_bh(&bond->lock);
+	read_lock(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
	if (!USES_PRIMARY(bond->params.mode)) {
		printk(KERN_INFO DRV_NAME
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1247,7 +1251,8 @@ static ssize_t 
bonding_store_active_slave(struct
device *d,
		}
	}
out:
drivers/net/bonding/bond_sysfs.c +2 -1
-	write_unlock_bh(&bond->lock);
+	write_unlock_bh(&bond->curr_slave_lock);
+	read_unlock(&bond->lock);
Quoted 3 lines
	rtnl_unlock();

	return count;
Vanilla 2.6.24-rc5 plus this patch:
=========================================================
[ INFO: possible irq lock inversion dependency detected ] 2.6.24-rc5 #1
---------------------------------------------------------
events/0/9 just changed the state of lock:
(&mc->mca_lock){-+..}, at: [<c0411c7a>] mld_ifc_timer_expire+0x130/0x1fb
but this lock took another, soft-read-irq-unsafe lock in the past:
(&bond->lock){-.--}
Quoted 3 lines
and interrupts could create inverse lock ordering between them.
Grrr, I should have seen that -- sorry. Try your luck with this instead:
<CUT>
Quoted 7 lines
Quoted 2 lines
No luck.
I'm guessing if we go back to using a write-lock for bond->lock this will go back to working again, but I'm not totally convinced since there are plenty of places where we used a read-lock with it.
Should I check this patch or rather, based on a future discussion, wait for another version?
Quoted 45 lines
drivers/net/bonding/bond_sysfs.c +3 -0
index 11b76b3..635b857 100644
@@ -1075,7 +1075,10 @@ static ssize_t bonding_store_primary(struct device
 *d,
 	struct slave *slave;
 	struct bonding *bond = to_bond(d);
 
+	rtnl_lock();
 	write_lock_bh(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
+
 	if (!USES_PRIMARY(bond->params.mode)) {
 		printk(KERN_INFO DRV_NAME
		       ": %s: Unable to set primary slave; %s is in mode
		       %d\n",
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1109,8 +1112,8 @@ static ssize_t bonding_store_primary(struct device
 *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +1 -0
+	write_unlock_bh(&bond->curr_slave_lock);
	write_unlock_bh(&bond->lock);
-
	rtnl_unlock();

	return count;
drivers/net/bonding/bond_sysfs.c +1 -0
@@ -1191,6 +1194,7 @@ static ssize_t bonding_store_active_slave(struct
 device *d,
 
 	rtnl_lock();
 	write_lock_bh(&bond->lock);
+	write_lock_bh(&bond->curr_slave_lock);
 
 	if (!USES_PRIMARY(bond->params.mode)) {
printk(KERN_INFO DRV_NAME
drivers/net/bonding/bond_sysfs.c +0 -0
@@ -1247,6 +1251,7 @@ static ssize_t bonding_store_active_slave(struct
 device *d,
 		}
 	}
out:
drivers/net/bonding/bond_sysfs.c +1 -0
+	write_unlock_bh(&bond->curr_slave_lock);
Quoted 3 lines
	write_unlock_bh(&bond->lock);
	rtnl_unlock();
Best regards,
Quoted 5 lines
Quoted 1 line
Krzysztof Olędzki
For now, I prefer Jay's original patch -- with the read_locks (rather than read/write_lock_bh) and the added rtnl_lock. There is still a lockdep issue that we need to sort-out, but this patch is needed first.
This bug has not been fixed yet as it still exists in 2.6.24-rc7. Any chances to cure it before 2.6.24-final?

Quoted 8 lines

Quoted 3 lines
Best regards,
Krzysztof Olędzki
Krzysztof,
I doubt the lockdep issue will be fixed, but the patch Jay posted and I acked needs to be included in 2.6.24.

I'm (finally) back from vacation and am working on the lock
problem right now; there are a couple of other changes that need to go in (in addition to what was posted previously). One is a spurious RTNL warning, the other is a similar 'wrong lock' type of problem that arises during module unload.

I should have a patch set for this posted in a couple of hours.

Quoted 4 lines

I played around with the locking when setting the multicast list and I can make the lockdep issue go away, but I need to be sure that it's OK to switch it to a read-lock from a write-lock (and I don't really think it is).

I haven't looked at the lockdep problem yet. If you want to be
brave and post your working patch for the lockdep thing, I might be able to crush your hopes that it's ok.

-J

---
-Jay Vosburgh, IBM Linux Technology Center, [email protected]