bonding: fix slave stuck in BOND_LINK_FAIL state
Subsystems (3)
PATCH 1/1 bonding: fix slave stuck in BOND_LINK_FAIL state
The bonding miimon logic has a flaw, in that a failure of thertnl_trylock can cause a slave to become permanently stuck in BOND_LINK_FAIL state.
The sequence of events to cause this is as follows:
1) bond_miimon_inspect finds that a slave's link is down, and so
calls bond_propose_link_state, setting slave->new_link_state to BOND_LINK_FAIL, then sets slave->new_link to BOND_LINK_DOWN and returns non-zero.
2) In bond_mii_monitor, the rtnl_trylock fails, and the timer is
rescheduled. No change is committed.
3) bond_miimon_inspect is called again, but this time the slave
from step 1 has recovered. slave->new_link is reset to NOCHANGE, and, as slave->link was never changed, the switch enters the BOND_LINK_UP case, and does nothing. The pending BOND_LINK_FAIL state from step 1 remains pending, as new_link_state is not reset.
4) The state from step 3 persists until another slave changes link
state and causes bond_miimon_inspect to return non-zero. At this point, the BOND_LINK_FAIL state change on the slave from steps 1-3 is committed, and the slave will remain stuck in BOND_LINK_FAIL state even though it is actually link up.
The remedy for this is to initialize new_link_state on each entry
to bond_miimon_inspect, as is already done with new_link.
--- drivers/net/bonding/bond_main.c | 1 + 1 file changed, 1 insertion(+)
drivers/net/bonding/bond_main.c +1 -0
@@ -2042,6 +2042,7 @@ static int bond_miimon_inspect(struct bonding *bond)bond_for_each_slave_rcu (bond, slave, iter) { slave->new_link =BOND_LINK_NOCHANGE ; + slave->link_new_state = slave->link; link_state =bond_check_dev_link (bond, slave->dev,0 );
Signature · 1 line
2.14.1
Quoted 26 lines · Jay Vosburgh
The bonding miimon logic has a flaw, in that a failure of the
rtnl_trylockcan cause a slave to become permanently stuck inBOND_LINK_FAILstate.The sequence of events to cause this is as follows:
1)
bond_miimon_inspectfinds that a slave's link is down, and so
callsbond_propose_link_state, settingslave->new_link_statetoBOND_LINK_FAIL, then setsslave->new_linktoBOND_LINK_DOWNand returns non-zero.2) In
bond_mii_monitor, thertnl_trylockfails, and the timer is
rescheduled. No change is committed.3)
bond_miimon_inspectis called again, but this time the slave
from step 1 has recovered.slave->new_linkis reset to NOCHANGE, and, asslave->linkwas never changed, the switch enters theBOND_LINK_UPcase, and does nothing. The pendingBOND_LINK_FAILstate from step 1 remains pending, asnew_link_stateis not reset.4) The state from step 3 persists until another slave changes link
state and causesbond_miimon_inspectto return non-zero. At this point, theBOND_LINK_FAILstate change on the slave from steps 1-3 is committed, and the slave will remain stuck inBOND_LINK_FAILstate even though it is actually link up.
Quoted 7 lines
The remedy for this is to initialize
new_link_stateon each entry
tobond_miimon_inspect, as is already done withnew_link.Reported-by Alex Sidorenko <[email protected]>Reviewed-by Jarod Wilson <[email protected]>Signed-off-by Jay Vosburgh <[email protected]>Fixes fb9eb899a6dc ("bonding: handle link transition from FAIL to UP correctly")
Quoted 19 lines
--- drivers/net/bonding/bond_main.c | 1 + 1 file changed, 1 insertion(+)drivers/net/bonding/bond_main.c +1
-1@@ -2042,6 +2042,7 @@ static int bond_miimon_inspect(struct bonding *bond)bond_for_each_slave_rcu (bond, slave, iter) { slave->new_link =BOND_LINK_NOCHANGE ; + slave->link_new_state = slave->link; link_state =bond_check_dev_link (bond, slave->dev,0 ); --2.14.1
David Miller
maintainerFrom: Jay Vosburgh <[email protected]>
Date: Tue, 07 Nov 2017 19:50:07 +0900
Quoted 26 lines
The bonding miimon logic has a flaw, in that a failure of the
rtnl_trylockcan cause a slave to become permanently stuck inBOND_LINK_FAILstate.The sequence of events to cause this is as follows:
1)
bond_miimon_inspectfinds that a slave's link is down, and so
callsbond_propose_link_state, settingslave->new_link_statetoBOND_LINK_FAIL, then setsslave->new_linktoBOND_LINK_DOWNand returns non-zero.2) In
bond_mii_monitor, thertnl_trylockfails, and the timer is
rescheduled. No change is committed.3)
bond_miimon_inspectis called again, but this time the slave
from step 1 has recovered.slave->new_linkis reset to NOCHANGE, and, asslave->linkwas never changed, the switch enters theBOND_LINK_UPcase, and does nothing. The pendingBOND_LINK_FAILstate from step 1 remains pending, asnew_link_stateis not reset.4) The state from step 3 persists until another slave changes link
state and causesbond_miimon_inspectto return non-zero. At this point, theBOND_LINK_FAILstate change on the slave from steps 1-3 is committed, and the slave will remain stuck inBOND_LINK_FAILstate even though it is actually link up.
Quoted 7 lines
The remedy for this is to initialize
new_link_stateon each entry
tobond_miimon_inspect, as is already done withnew_link.Reported-by Alex Sidorenko <[email protected]>Reviewed-by Jarod Wilson <[email protected]>Signed-off-by Jay Vosburgh <[email protected]>Fixes fb9eb899a6dc ("bonding: handle link transition from FAIL to UP correctly")
Applied and queued up for -stable.
As discussed with some others here at netdev... rtnl_trylock() really needs to be re-evaluated if not removed completely.