bonding: change spinlocks and remove timers in favor of workqueues
Subsystems (3)
PATCH 1/1 bonding: change spinlocks and remove timers in favor of workqueues
The main purpose of this patch is to clean-up the bonding code so that several important operations are not done in the incorrect (softirq) context. Whenever a kernel is compiled with CONFIG_DEBUG_SPINLOCK_SLEEP all sorts of backtraces are spewed to the log since might_sleep will kindly remind us we are doing something in a atomic context when we probably should not.
In order to resolve this, the spin_[un]lock_bh needed to be converted to spin_[un]lock and to do that the timers needed to be dropped in favor of workqueues. Since there isn't the chance that this work will be done in a softirq context, the bh-locks aren't needed since we should not be preempted to service the workqueue. Both of those changes are included in this patch.
I've done a bit of testing switching between modes and changing some of the important values through sysfs, so I feel that creating and canceling the work is working fine. This code could use some quick testing with 802.3ad since I didn't have access to a switch with that capability, so if someone can verify it I would appreciate it.
---
bond_3ad.c | 9 +- bond_3ad.h | 2 bond_alb.c | 17 +++- bond_alb.h | 2 bond_main.c | 215 ++++++++++++++++++++++++++++++++++++++--------------------- bond_sysfs.c | 78 ++++++++++----------- bonding.h | 21 +++-- 7 files changed, 212 insertions(+), 132 deletions(-)
drivers/net/bonding/bond_3ad.c +7 -2
@@ -2097,8 +2097,10 @@ void bond_3ad_unbind_slave(struct slave* times out, and it selects an aggregator for the ports that are yet not * related to any aggregator, and selects the active aggregator for a bond. */ -void bond_3ad_state_machine_handler (struct bonding *bond) +void bond_3ad_state_machine_handler (void *work_data) { +struct net_device *bond_dev = (struct net_device *)work_data; +struct bonding *bond = bond_dev->priv;struct port *port;struct aggregator *aggregator;
@@ -2149,7 +2151,10 @@ void bond_3ad_state_machine_handler(stru } re_arm: -mod_timer (&(BOND_AD_INFO (bond).ad_timer), jiffies + ad_delta_in_ticks); +bond_work_create (bond_dev, + bond_3ad_state_machine_handler, + &bond->ad_work, + ad_delta_in_ticks); out:read_unlock (&bond->lock); }
drivers/net/bonding/bond_3ad.h +1 -1
@@ -276,7 +276,7 @@ struct ad_slave_info {void bond_3ad_initialize (struct bonding *bond,u16 tick_resolution,int lacp_fast);int bond_3ad_bind_slave (struct slave *slave);void bond_3ad_unbind_slave (struct slave *slave); -void bond_3ad_state_machine_handler (struct bonding *bond); +void bond_3ad_state_machine_handler (void *work_data);void bond_3ad_adapter_speed_changed (struct slave *slave);void bond_3ad_adapter_duplex_changed (struct slave *slave);void bond_3ad_handle_link_change (struct slave *slave,char link);
drivers/net/bonding/bond_alb.c +11 -6
@@ -1367,8 +1367,10 @@ out:return 0 ; } -void bond_alb_monitor (struct bonding *bond) +void bond_alb_monitor (void *work_data) { +struct net_device *bond_dev = (struct net_device *)work_data; +struct bonding *bond = bond_dev->priv;struct alb_bond_info *bond_info = &(BOND_ALB_INFO (bond));struct slave *slave;int i;
@@ -1433,7 +1435,7 @@ void bond_alb_monitor(struct bonding *bo* write lock to protect from other code that also * sets the promiscuity. */ -write_lock_bh (&bond->curr_slave_lock); +write_lock (&bond->curr_slave_lock);if (bond_info->primary_is_promisc && (++bond_info->rlb_promisc_timeout_counter >=RLB_PROMISC_TIMEOUT )) {
@@ -1448,7 +1450,7 @@ void bond_alb_monitor(struct bonding *bo bond_info->primary_is_promisc =0 ; } -write_unlock_bh (&bond->curr_slave_lock); +write_unlock (&bond->curr_slave_lock);if (bond_info->rlb_rebalance) { bond_info->rlb_rebalance =0 ;
@@ -1471,7 +1473,10 @@ void bond_alb_monitor(struct bonding *bo } re_arm: -mod_timer (&(bond_info->alb_timer), jiffies + alb_delta_in_ticks); +bond_work_create (bond_dev, + bond_alb_monitor, + &bond->alb_work, + alb_delta_in_ticks); out:read_unlock (&bond->lock); }
@@ -1492,11 +1497,11 @@ int bond_alb_init_slave(struct bonding */* caller must hold the bond lock for write since the mac addresses * are compared and may be swapped. */ -write_lock_bh (&bond->lock); +write_lock (&bond->lock); res =alb_handle_addr_collision_on_attach (bond, slave); -write_unlock_bh (&bond->lock); +write_unlock (&bond->lock);if (res) {return res;
drivers/net/bonding/bond_alb.h +1 -1
@@ -125,7 +125,7 @@ void bond_alb_deinit_slave(struct bondinvoid bond_alb_handle_link_change (struct bonding *bond,struct slave *slave,char link);void bond_alb_handle_active_change (struct bonding *bond,struct slave *new_slave);int bond_alb_xmit (struct sk_buff *skb,struct net_device *bond_dev); -void bond_alb_monitor (struct bonding *bond); +void bond_alb_monitor (void *work_data);int bond_alb_set_mac_address (struct net_device *bond_dev,void *addr);void bond_alb_clear_vlan (struct bonding *bond,unsigned short vlan_id);#endif /* __BOND_ALB_H__ */
drivers/net/bonding/bond_main.c +139 -76
520 lines not on this page · raw message
drivers/net/bonding/bond_sysfs.c +39 -39
164 lines not on this page · raw message
drivers/net/bonding/bonding.h +14 -7
@@ -22,8 +22,8 @@ #include <linux/kobject.h>#include "bond_3ad.h" #include "bond_alb.h" -#define DRV_VERSION "3.1.1" -#define DRV_RELDATE "September 26, 2006" +#define DRV_VERSION "3.2.0" +#define DRV_RELDATE "November 21, 2006"#define DRV_NAME "bonding"#define DRV_DESCRIPTION "Ethernet Channel Bonding Driver"
@@ -182,8 +182,6 @@ struct bonding {s32 slave_cnt;/* never change this value outside the attach/detach wrappers */ rwlock_t lock;rwlock_t curr_slave_lock; -struct timer_list mii_timer; -struct timer_list arp_timer;s8 kill_timers;struct net_device_stats stats;#ifdef CONFIG_PROC_FS
@@ -201,6 +199,11 @@ #endif /* CONFIG_PROC_FS */struct list_head vlan_list;struct vlan_group *vlgrp;struct packet_type arp_mon_pt; +struct workqueue_struct *wq; +struct work_struct mii_work; +struct work_struct arp_work; +struct work_struct alb_work; +struct work_struct ad_work; };/**
@@ -300,9 +303,9 @@ void bond_destroy_slave_symlinks(structint bond_enslave (struct net_device *bond_dev,struct net_device *slave_dev);int bond_release (struct net_device *bond_dev,struct net_device *slave_dev);int bond_sethwaddr (struct net_device *bond_dev,struct net_device *slave_dev); -void bond_mii_monitor (struct net_device *bond_dev); -void bond_loadbalance_arp_mon (struct net_device *bond_dev); -void bond_activebackup_arp_mon (struct net_device *bond_dev); +void bond_mii_monitor (void *work_data); +void bond_loadbalance_arp_mon (void *work_data); +void bond_activebackup_arp_mon (void *work_data);void bond_set_mode_ops (struct bonding *bond,int mode);int bond_parse_parm (char *mode_arg,struct bond_parm_tbl *tbl);const char *bond_mode_name (int mode);
@@ -310,6 +313,10 @@ void bond_select_active_slave(struct bonvoid bond_change_active_slave (struct bonding *bond,struct slave *new_active);void bond_register_arp (struct bonding *);void bond_unregister_arp (struct bonding *); +void bond_work_create (struct net_device *bond_dev,void (*fn)(void *),struct work_struct *work,unsigned long delay); +void bond_work_cancel (struct work_struct *work); +void bond_work_reschedule (struct bonding *bond,struct work_struct *work); +void bond_wq_destroy (struct net_device *bond_dev);#endif /* _LINUX_BONDING_H */
Quoted 6 lines · Andy Gospodarek
The main purpose of this patch is to clean-up the bonding code so that several important operations are not done in the incorrect (softirq) context. Whenever a kernel is compiled with
CONFIG_DEBUG_SPINLOCK_SLEEPall sorts of backtraces are spewed to the log sincemight_sleepwill kindly remind us we are doing something in a atomic context when we probably should not.
[...]
I'll look at the patch in detail in a bit (and I have 802.3ad
switches to test on), but on first glance, does this not still hold a lock during failover operations in balance-alb mode? I.e., this doesn't change the locking model, it just moves the timers to workqueues and relaxes the _bh locking.
The really problematic case calls dev_set_mac_address() with a
lock held, and I don't see that this patch changes that behavior. Do you still get the lock warnings during link fail / recovery in balance-alb mode?
Also, on an CONFIG_PREEMPT kernel, it'll still get the sleep
warnings, since in_atomic() will trip __might_sleep() for any lock (if I'm reading things correctly).
Don't get me wrong, this (switching to workqueues, etc) needs to
be done, but I don't think this patch really resolves the underlying problem that causes the warnings.
Let me see if I can dust off the extensive patch that does
change the locking model; I'll see if I can bring it up to the current git and post it.
-J
---
-Jay Vosburgh, IBM Linux Technology Center, [email protected]
Quoted 8 lines · Jay Vosburgh
Quoted 6 lines · Andy Gospodarek
The main purpose of this patch is to clean-up the bonding code so that several important operations are not done in the incorrect (softirq) context. Whenever a kernel is compiled with
CONFIG_DEBUG_SPINLOCK_SLEEPall sorts of backtraces are spewed to the log sincemight_sleepwill kindly remind us we are doing something in a atomic context when we probably should not.
Quoted 7 lines
[...]
I'll look at the patch in detail in a bit (and I have 802.3ad
switches to test on), but on first glance, does this not still hold a lock during failover operations in balance-alb mode? I.e., this doesn't change the locking model, it just moves the timers to workqueues and relaxes the_bhlocking.
Jay,
Thanks for the response. You are correct. This patch really doesn't change functionality -- in fact that was one of goals of this patch. I wanted to simply start the conversion since it seemed like 'the right way' to do things going forward.
Quoted 5 lines
The really problematic case calls
dev_set_mac_address()with a
lock held, and I don't see that this patch changes that behavior. Do you still get the lock warnings during link fail / recovery in balance-alb mode?
I no longer get lock warnings indicating that I'm taking a lock in an invalid context, but lately I've been seeing rtnl lock assertion failures when in balance-alb mode and whenever a call to dev_set_mac_address is made. It seems to be expected that the rtnl lock is taken and that isn't the case anymore.
Quoted 4 lines
Also, on an
CONFIG_PREEMPTkernel, it'll still get the sleep
warnings, sincein_atomic()will trip__might_sleep()for any lock (if I'm reading things correctly).
Based on my reading you will still only get these warnings if CONFIG_DEBUG_SPINLOCK_SLEEP=y and CONFIG_PREEMPT=y. Since most never try with CONFIG_DEBUG_SPINLOCK_SLEEP=y they don't see these.
Quoted 3 lines
Don't get me wrong, this (switching to workqueues, etc) needs to
be done, but I don't think this patch really resolves the underlying problem that causes the warnings.
Agreed. I didn't want to tackle too many of the issues with one giant patch. Doing them in smallish steps seemed like a better way to go.
Quoted 5 lines
Let me see if I can dust off the extensive patch that does
change the locking model; I'll see if I can bring it up to the current git and post it.
It would seem ideal if we could combine the two into one big patch.
-andy
Quoted 5 lines
-J---
-Jay Vosburgh, IBM Linux Technology Center, [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 7 lines · Jay Vosburgh
Andy Gospodarek <[email protected]> wrote:
[...]Quoted 5 lines
Quoted 4 lines
Let me see if I can dust off the extensive patch that does
change the locking model; I'll see if I can bring it up to the current git and post it.
Quoted 8 lines
Quoted 1 line
It would seem ideal if we could combine the two into one big patch.
Appended is my working development patch for rearranging a bunch
of stuff in bonding. This is a work in progress, and is not likely to be very stable. This largely reimplements the entire link monitor scheme, along with associated locking changes. It's not split into functional pieces, and is just a big kitchen sink blob at the moment. It's still very experimental.
Jay,
Thanks for sending it. I've finally got some time to digest it. I didn't get a chance to test it, but I did have a chance to take a look at it in detail.
I certainly agree with the design points you laid out in your description. I like the fact that you've restructured a lot of the code and moved the monitoring to a common place. It is also nice to have it in one spot so you can report link status and bond membership. I don't think the bh-locking is necessary since the workqueues eliminate the possibility that you will be preempted by the bonding driver -- I dropped them all from my patch.
Overall this seems like a good way to go, but it [obviously] seems like it would be good to work towards this functionality rather than simply dumping this huge blob all at once. I would like to use a smaller patch to switch to workqueues (my patch or a partial of yours -- it makes no difference to me) as a start with the intent to let that soak-in for a while and follow it with chunks that will eventually get the entire driver closer to the functionality that your big patch has.
Thoughts?
-andy
Quoted 1 line · Andy Gospodarek
On 12/4/06, Jay Vosburgh <[email protected]> wrote:
[...]
Quoted 6 lines
Quoted 6 lines
Appended is my working development patch for rearranging a bunch
of stuff in bonding. This is a work in progress, and is not likely to be very stable. This largely reimplements the entire link monitor scheme, along with associated locking changes. It's not split into functional pieces, and is just a big kitchen sink blob at the moment. It's still very experimental.
[...]
Quoted 3 lines
Thanks for sending it. I've finally got some time to digest it. I didn't get a chance to test it, but I did have a chance to take a look at it in detail.
I think the patch might have been over the size limit for
netdev, as I never saw it appear on the list. For interested readers, the description Andy mentions in his next paragraph is:
By way of description, in the patch, all monitors (link state
and mode specific bits for ALB and 802.3ad) are all dispatched from a single workqueue (rather than each on a separate timer), and the updelay / downdelay parameters are done as special monitors. The monitors can be paused for purposes of conflict avoidance, although some mutexing with TX activity is still necessary. It also allows for the ARP and MII link monitors to run simultaneously (which isn't allowed in the current mainline driver).
This also splits up the slave lists into an "all" and an
"active" list, rearranges the release processing for bonding (consolidates duplicated code from bond_release and bond_release_all into a common function). I'm still dithering as to whether or not this is a good idea; it adds a little complexity to link state changes and add/remove of slaves, but simplifies most of the transmit processing.
The locking still needs some adjustment; there's some tracking
of whether or not RTNL is held, and right now most locking is of the _bh variety, which may be too strict.
Quoted 7 lines
I certainly agree with the design points you laid out in your description. I like the fact that you've restructured a lot of the code and moved the monitoring to a common place. It is also nice to have it in one spot so you can report link status and bond membership. I don't think the bh-locking is necessary since the workqueues eliminate the possibility that you will be preempted by the bonding driver -- I dropped them all from my patch.
The only concern I had in that regard is whether it's possible
to get into a bonding transmit function from someplace that requires _bh level locking to mutex against. I'm thinking in particular of packet receive processing somewhere turning around to do a transmit right away.
Quoted 3 lines
Overall this seems like a good way to go, but it [obviously] seems like it would be good to work towards this functionality rather than simply dumping this huge blob all at once.
I absolutely agree that moving in steps to a final resolution is
much better than dropping a big blob; the patch ended up this way since it's just a big diff from my experimental tree.
[...] I would like to use a
Quoted 5 lines
smaller patch to switch to workqueues (my patch or a partial of yours -- it makes no difference to me) as a start with the intent to let that soak-in for a while and follow it with chunks that will eventually get the entire driver closer to the functionality that your big patch has.
I don't see a problem in starting with your patch; the end state
will be sufficiently different (e.g., the four workqueues would ultimately be consolidated into one) that it's still a good testbed to start with.
The only other minor issue is that after this week, I will be
off for the holidays (and far, far away from email) for the next month.
-J
---
-Jay Vosburgh, IBM Linux Technology Center, [email protected]
Quoted 5 lines · Jay Vosburgh
I don't see a problem in starting with your patch; the end state
will be sufficiently different (e.g., the four workqueues would ultimately be consolidated into one) that it's still a good testbed to start with.
My patch actually only uses a single workqueue for each bond (almost exactly like yours in-fact) with 4 different types of work that can be placed on it.
Quoted 4 lines
The only other minor issue is that after this week, I will be
off for the holidays (and far, far away from email) for the next month.
Good to know.
Quoted 5 lines
-J---
-Jay Vosburgh, IBM Linux Technology Center, [email protected]
Andy Gospodarek <[email protected]> wrote:
[...]
Quoted 3 lines
My patch actually only uses a single workqueue for each bond (almost exactly like yours in-fact) with 4 different types of work that can be placed on it.
Well, what I was trying to say was that there are still multiple
independently scheduled thingies on the workqueue (although you can't ever have all of them going at the same time), whereas in the Big Patch, there's only one scheduled thingie on the workqueue, and everything is dispatched from within its single handler (making it less complicated to pause all of the "timers" for mutual exclusion purposes).
-J
---
-Jay Vosburgh, IBM Linux Technology Center, [email protected]