Skip to contentSkip to conversation

bonding: fix active-backup transition after link failure

PATCH In review Mahesh Bandewar 3 participants 9 messages Dec 6, 2019 – Dec 15, 2019

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 bonding: fix active-backup transition after link failure

After the recent fix 1899bb325149 ("bonding: fix state transition issue in link monitoring"), the active-backup mode with miimon initially come-up fine but after a link-failure, both members transition into backup state.

Following steps to reproduce the scenario (eth1 and eth2 are the slaves of the bond):

ip link set eth1 up
ip link set eth2 down
sleep 1
ip link set eth2 up
ip link set eth1 down
cat /sys/class/net/eth1/bonding_slave/state
cat /sys/class/net/eth2/bonding_slave/state

Fixes 1899bb325149 ("bonding: fix state transition issue in link monitoring")
CC Jay Vosburgh <[email protected]>
---
 drivers/net/bonding/bond_main.c | 3 ---
 1 file changed, 3 deletions(-)
drivers/net/bonding/bond_main.c +0 -3
index fcb7c2f7f001..ad9906c102b4 100644
@@ -2272,9 +2272,6 @@ static void bond_miimon_commit(struct bonding *bond)
 			} else if (BOND_MODE(bond) != BOND_MODE_ACTIVEBACKUP) {
 				/* make it immediately active */
 				bond_set_active_slave(slave);
-			} else if (slave != primary) {
-				/* prevent it from being the active one */
-				bond_set_backup_slave(slave);
 			}
 
 			slave_info(bond->dev, slave->dev, "link status definitely up, %u Mbps %s duplex\n",
Signature · 1 line
2.24.0.393.g34dc348eaf-goog
#1 lore raw
Quoted 28 lines · Mahesh Bandewar

After the recent fix 1899bb325149 ("bonding: fix state transition issue in link monitoring"), the active-backup mode with miimon initially come-up fine but after a link-failure, both members transition into backup state.

Following steps to reproduce the scenario (eth1 and eth2 are the slaves of the bond):

ip link set eth1 up
ip link set eth2 down
sleep 1
ip link set eth2 up
ip link set eth1 down
cat /sys/class/net/eth1/bonding_slave/state
cat /sys/class/net/eth2/bonding_slave/state

Fixes 1899bb325149 ("bonding: fix state transition issue in link monitoring")
CC Jay Vosburgh <[email protected]>

---
drivers/net/bonding/bond_main.c | 3 ---

1 file changed, 3 deletions(-)
drivers/net/bonding/bond_main.c +0 -0
index fcb7c2f7f001..ad9906c102b4 100644

drivers/net/bonding/bond_main.c static void bond_miimon_commit(struct bonding *bond) · line 2277

Quoted 6 lines
drivers/net/bonding/bond_main.c +0 -3
@@ -2272,9 +2272,6 @@ static void bond_miimon_commit(struct bonding *bond)
 			} else if (BOND_MODE(bond) != BOND_MODE_ACTIVEBACKUP) {
 				/* make it immediately active */
 				bond_set_active_slave(slave);
-			} else if (slave != primary) {
-				/* prevent it from being the active one */
-				bond_set_backup_slave(slave);
	How does this fix things?  Doesn't bond_select_active_slave() ->

bond_change_active_slave() set the backup flag correctly via a call to bond_set_slave_active_flags() when it sets a slave to be the active slave? If this change resolves the problem, I'm not sure how this ever worked correctly, even prior to 1899bb325149.

	-J
Quoted 6 lines
drivers/net/bonding/bond_main.c +0 -0
 			}
 
 			slave_info(bond->dev, slave->dev, "link status definitely up, %u Mbps %s duplex\n",
Signature · 1 line
2.24.0.393.g34dc348eaf-goog

---
-Jay Vosburgh, [email protected]

#2 lore raw
Quoted 39 lines · Jay Vosburgh
Quoted 28 lines · Mahesh Bandewar

After the recent fix 1899bb325149 ("bonding: fix state transition issue in link monitoring"), the active-backup mode with miimon initially come-up fine but after a link-failure, both members transition into backup state.

Following steps to reproduce the scenario (eth1 and eth2 are the slaves of the bond):

ip link set eth1 up
ip link set eth2 down
sleep 1
ip link set eth2 up
ip link set eth1 down
cat /sys/class/net/eth1/bonding_slave/state
cat /sys/class/net/eth2/bonding_slave/state

Fixes 1899bb325149 ("bonding: fix state transition issue in link monitoring")
CC Jay Vosburgh <[email protected]>

---
drivers/net/bonding/bond_main.c | 3 ---

1 file changed, 3 deletions(-)
drivers/net/bonding/bond_main.c +0 -0
index fcb7c2f7f001..ad9906c102b4 100644

drivers/net/bonding/bond_main.c static void bond_miimon_commit(struct bonding *bond) · line 2277

Quoted 6 lines
drivers/net/bonding/bond_main.c +0 -3
@@ -2272,9 +2272,6 @@ static void bond_miimon_commit(struct bonding *bond)
                      } else if (BOND_MODE(bond) != BOND_MODE_ACTIVEBACKUP) {
                              /* make it immediately active */
                              bond_set_active_slave(slave);
-                      } else if (slave != primary) {
-                              /* prevent it from being the active one */
-                              bond_set_backup_slave(slave);
        How does this fix things?  Doesn't bond_select_active_slave() ->
Quoted 5 lines

bond_change_active_slave() set the backup flag correctly via a call to bond_set_slave_active_flags() when it sets a slave to be the active slave? If this change resolves the problem, I'm not sure how this ever worked correctly, even prior to 1899bb325149.

Hi Jay, I used kprobes to figure out the brokenness this patch fixes. Prior to your patch this call would not happen but with the patch, this extra call will put the master into the backup mode erroneously (in fact both members would be in backup state). The mechanics you have mentioned works correctly except that in the prior case, the switch statement was using new_link which was not same as link_new_state. The miimon_inspect will update new_link which is what was used in miimon_commit code. The link_new_state was used only to mitigate the rtnl-lock issue which would update the "link". Hence in the prior code, this path would never get executed.

The steps to reproduce this issue is straightforward and happens 100% of the time (I used two mlx interfaces but that shouldn't matter).

thanks,
--mahesh..

Quoted 11 lines
        -J
Quoted 6 lines
drivers/net/bonding/bond_main.c +0 -1
                      }
 
                      slave_info(bond->dev, slave->dev, "link status definitely up, %u Mbps %s duplex\n",
--

2.24.0.393.g34dc348eaf-goog

---
-Jay Vosburgh, [email protected]

#3 lore raw
Quoted 57 lines · Mahesh Bandewar (महेश बंडेवार)
Quoted 39 lines · Jay Vosburgh
Quoted 28 lines · Mahesh Bandewar

After the recent fix 1899bb325149 ("bonding: fix state transition issue in link monitoring"), the active-backup mode with miimon initially come-up fine but after a link-failure, both members transition into backup state.

Following steps to reproduce the scenario (eth1 and eth2 are the slaves of the bond):

ip link set eth1 up
ip link set eth2 down
sleep 1
ip link set eth2 up
ip link set eth1 down
cat /sys/class/net/eth1/bonding_slave/state
cat /sys/class/net/eth2/bonding_slave/state

Fixes 1899bb325149 ("bonding: fix state transition issue in link monitoring")
CC Jay Vosburgh <[email protected]>

---
drivers/net/bonding/bond_main.c | 3 ---

1 file changed, 3 deletions(-)
drivers/net/bonding/bond_main.c +0 -0
index fcb7c2f7f001..ad9906c102b4 100644

drivers/net/bonding/bond_main.c static void bond_miimon_commit(struct bonding *bond) · line 2277

Quoted 6 lines
drivers/net/bonding/bond_main.c +0 -3
@@ -2272,9 +2272,6 @@ static void bond_miimon_commit(struct bonding *bond)
                      } else if (BOND_MODE(bond) != BOND_MODE_ACTIVEBACKUP) {
                              /* make it immediately active */
                              bond_set_active_slave(slave);
-                      } else if (slave != primary) {
-                              /* prevent it from being the active one */
-                              bond_set_backup_slave(slave);
        How does this fix things?  Doesn't bond_select_active_slave() ->
Quoted 5 lines

bond_change_active_slave() set the backup flag correctly via a call to bond_set_slave_active_flags() when it sets a slave to be the active slave? If this change resolves the problem, I'm not sure how this ever worked correctly, even prior to 1899bb325149.

Hi Jay, I used kprobes to figure out the brokenness this patch fixes. Prior to your patch this call would not happen but with the patch, this extra call will put the master into the backup mode erroneously (in fact both members would be in backup state). The mechanics you have mentioned works correctly except that in the prior case, the switch statement was using new_link which was not same as link_new_state. The miimon_inspect will update new_link which is what was used in miimon_commit code. The link_new_state was used only to mitigate the rtnl-lock issue which would update the "link". Hence in the prior code, this path would never get executed.

Quoted 3 lines

The steps to reproduce this issue is straightforward and happens 100% of the time (I used two mlx interfaces but that shouldn't matter).

A friendly ping.

Quoted 13 lines

thanks,
--mahesh..

Quoted 11 lines
        -J
Quoted 6 lines
drivers/net/bonding/bond_main.c +0 -1
                      }
 
                      slave_info(bond->dev, slave->dev, "link status definitely up, %u Mbps %s duplex\n",
--

2.24.0.393.g34dc348eaf-goog

---
-Jay Vosburgh, [email protected]

#4 lore raw
In reply to Mahesh Bandewar (महेश बंडेवार)
Quoted 45 lines · Mahesh Bandewar (महेश बंडेवार)
Quoted 39 lines · Jay Vosburgh
Quoted 28 lines · Mahesh Bandewar

After the recent fix 1899bb325149 ("bonding: fix state transition issue in link monitoring"), the active-backup mode with miimon initially come-up fine but after a link-failure, both members transition into backup state.

Following steps to reproduce the scenario (eth1 and eth2 are the slaves of the bond):

ip link set eth1 up
ip link set eth2 down
sleep 1
ip link set eth2 up
ip link set eth1 down
cat /sys/class/net/eth1/bonding_slave/state
cat /sys/class/net/eth2/bonding_slave/state

Fixes 1899bb325149 ("bonding: fix state transition issue in link monitoring")
CC Jay Vosburgh <[email protected]>

---
drivers/net/bonding/bond_main.c | 3 ---

1 file changed, 3 deletions(-)
drivers/net/bonding/bond_main.c +0 -0
index fcb7c2f7f001..ad9906c102b4 100644

drivers/net/bonding/bond_main.c static void bond_miimon_commit(struct bonding *bond) · line 2277

Quoted 6 lines
drivers/net/bonding/bond_main.c +0 -3
@@ -2272,9 +2272,6 @@ static void bond_miimon_commit(struct bonding *bond)
                      } else if (BOND_MODE(bond) != BOND_MODE_ACTIVEBACKUP) {
                              /* make it immediately active */
                              bond_set_active_slave(slave);
-                      } else if (slave != primary) {
-                              /* prevent it from being the active one */
-                              bond_set_backup_slave(slave);
        How does this fix things?  Doesn't bond_select_active_slave() ->
Quoted 5 lines

bond_change_active_slave() set the backup flag correctly via a call to bond_set_slave_active_flags() when it sets a slave to be the active slave? If this change resolves the problem, I'm not sure how this ever worked correctly, even prior to 1899bb325149.

Quoted 10 lines

Hi Jay, I used kprobes to figure out the brokenness this patch fixes. Prior to your patch this call would not happen but with the patch, this extra call will put the master into the backup mode erroneously (in fact both members would be in backup state). The mechanics you have mentioned works correctly except that in the prior case, the switch statement was using new_link which was not same as link_new_state. The miimon_inspect will update new_link which is what was used in miimon_commit code. The link_new_state was used only to mitigate the rtnl-lock issue which would update the "link". Hence in the prior code, this path would never get executed.

I'm looking at the old code (prior to 1899bb325149), and I don't
see a path to what you're describing for the down to up transition in active-backup mode.

bond_miimon_inspect enters switch, slave->link == BOND_LINK_DOWN.

link_state is nonzero, call bond_propose_link_state(BOND_LINK_BACK), which sets slave->link_new_state to _BACK.

Fall through to BOND_LINK_BACK case, set slave->new_link = BOND_LINK_UP

bond_mii_monitor then calls bond_commit_link_state, which sets slave->link to BOND_LINK_BACK

Enter bond_miimon_commit switch (new_link), which is BOND_LINK_UP

In "case BOND_LINK_UP:" there is no way out of this block, and it should proceed to call bond_set_backup_slave for active-backup mode every time.

Quoted 2 lines

The steps to reproduce this issue is straightforward and happens 100% of the time (I used two mlx interfaces but that shouldn't matter).

Yes, I've been able to reproduce it locally (with igb, FWIW). I
think the patch is likely ok, I'm just mystified as to how the backup setting could have worked prior to 1899bb325149, so perhaps the Fixes tag doesn't go back far enough.

	-J
Quoted 9 lines

thanks,
--mahesh..

Quoted 7 lines
        -J
Quoted 5 lines
drivers/net/bonding/bond_main.c +0 -1
                      }
 
                      slave_info(bond->dev, slave->dev, "link status definitely up, %u Mbps %s duplex\n",
--

2.24.0.393.g34dc348eaf-goog

---
-Jay Vosburgh, [email protected]

#5 lore raw
Quoted 59 lines · Jay Vosburgh
Quoted 45 lines · Mahesh Bandewar (महेश बंडेवार)
Quoted 39 lines · Jay Vosburgh
Quoted 28 lines · Mahesh Bandewar

After the recent fix 1899bb325149 ("bonding: fix state transition issue in link monitoring"), the active-backup mode with miimon initially come-up fine but after a link-failure, both members transition into backup state.

Following steps to reproduce the scenario (eth1 and eth2 are the slaves of the bond):

ip link set eth1 up
ip link set eth2 down
sleep 1
ip link set eth2 up
ip link set eth1 down
cat /sys/class/net/eth1/bonding_slave/state
cat /sys/class/net/eth2/bonding_slave/state

Fixes 1899bb325149 ("bonding: fix state transition issue in link monitoring")
CC Jay Vosburgh <[email protected]>

---
drivers/net/bonding/bond_main.c | 3 ---

1 file changed, 3 deletions(-)
drivers/net/bonding/bond_main.c +0 -0
index fcb7c2f7f001..ad9906c102b4 100644

drivers/net/bonding/bond_main.c static void bond_miimon_commit(struct bonding *bond) · line 2277

Quoted 6 lines
drivers/net/bonding/bond_main.c +0 -3
@@ -2272,9 +2272,6 @@ static void bond_miimon_commit(struct bonding *bond)
                      } else if (BOND_MODE(bond) != BOND_MODE_ACTIVEBACKUP) {
                              /* make it immediately active */
                              bond_set_active_slave(slave);
-                      } else if (slave != primary) {
-                              /* prevent it from being the active one */
-                              bond_set_backup_slave(slave);
        How does this fix things?  Doesn't bond_select_active_slave() ->
Quoted 5 lines

bond_change_active_slave() set the backup flag correctly via a call to bond_set_slave_active_flags() when it sets a slave to be the active slave? If this change resolves the problem, I'm not sure how this ever worked correctly, even prior to 1899bb325149.

Quoted 10 lines

Hi Jay, I used kprobes to figure out the brokenness this patch fixes. Prior to your patch this call would not happen but with the patch, this extra call will put the master into the backup mode erroneously (in fact both members would be in backup state). The mechanics you have mentioned works correctly except that in the prior case, the switch statement was using new_link which was not same as link_new_state. The miimon_inspect will update new_link which is what was used in miimon_commit code. The link_new_state was used only to mitigate the rtnl-lock issue which would update the "link". Hence in the prior code, this path would never get executed.

Quoted 4 lines

I'm looking at the old code (prior to 1899bb325149), and I don't
see a path to what you're describing for the down to up transition in active-backup mode.

I was referring to the code where bond_miimon_inspect() switches using bond->link and bond_miimon_commit() (which happens after inspect) switches using bond->new_link. inspect doesn't touch new_link unless delay is set which is a corner case and probably ignore for this purpose since it's just postponing the behavior. bond->link_new_state was brought in to mitigate RTNL issue and affects only bond->link, if it can acquire RTNL. So irrespective of what bond_miimon_inspect() does for bond->link or bond->link_new_state the bond->new_link was maintained and then used in the bond_miimon_commit. Because of this the wrong transition wouldn't happen.

Once the new_link and link_new_state is merged, the state that bond_miimon_inspect() sets for bond->link_new_state *is* used in bond_miimon_commit() (which is after the fact) and hence (I believe) the erroneous transition.

Having said that, the fix that you put in is necessary to close the window between link_propose() and link_commit() but the side effect of that was the situation that I explained
above which is what this patch fixes it.

Quoted 15 lines

bond_miimon_inspect enters switch, slave->link == BOND_LINK_DOWN.

link_state is nonzero, call bond_propose_link_state(BOND_LINK_BACK), which sets slave->link_new_state to _BACK.

Fall through to BOND_LINK_BACK case, set slave->new_link = BOND_LINK_UP

bond_mii_monitor then calls bond_commit_link_state, which sets slave->link to BOND_LINK_BACK

Enter bond_miimon_commit switch (new_link), which is BOND_LINK_UP

In "case BOND_LINK_UP:" there is no way out of this block, and it should proceed to call bond_set_backup_slave for active-backup mode every time.

Quoted 8 lines
Quoted 2 lines

The steps to reproduce this issue is straightforward and happens 100% of the time (I used two mlx interfaces but that shouldn't matter).

Yes, I've been able to reproduce it locally (with igb, FWIW). I
think the patch is likely ok, I'm just mystified as to how the backup setting could have worked prior to 1899bb325149, so perhaps the Fixes tag doesn't go back far enough.

Well, I added fixes-tag since the behavior started as soon as the 1899bb325149 was added. I don't see the issue if I revert 1899bb325149.

Quoted 14 lines
        -J
Quoted 9 lines

thanks,
--mahesh..

Quoted 7 lines
        -J
Quoted 5 lines
drivers/net/bonding/bond_main.c +0 -1
                      }
 
                      slave_info(bond->dev, slave->dev, "link status definitely up, %u Mbps %s duplex\n",
--

2.24.0.393.g34dc348eaf-goog

---
-Jay Vosburgh, [email protected]

Acked-by
#6 lore raw
Quoted 81 lines · Mahesh Bandewar (महेश बंडेवार)
Quoted 59 lines · Jay Vosburgh
Quoted 45 lines · Mahesh Bandewar (महेश बंडेवार)
Quoted 39 lines · Jay Vosburgh
Quoted 28 lines · Mahesh Bandewar

After the recent fix 1899bb325149 ("bonding: fix state transition issue in link monitoring"), the active-backup mode with miimon initially come-up fine but after a link-failure, both members transition into backup state.

Following steps to reproduce the scenario (eth1 and eth2 are the slaves of the bond):

ip link set eth1 up
ip link set eth2 down
sleep 1
ip link set eth2 up
ip link set eth1 down
cat /sys/class/net/eth1/bonding_slave/state
cat /sys/class/net/eth2/bonding_slave/state

Fixes 1899bb325149 ("bonding: fix state transition issue in link monitoring")
CC Jay Vosburgh <[email protected]>

---
drivers/net/bonding/bond_main.c | 3 ---

1 file changed, 3 deletions(-)
drivers/net/bonding/bond_main.c +0 -0
index fcb7c2f7f001..ad9906c102b4 100644

drivers/net/bonding/bond_main.c static void bond_miimon_commit(struct bonding *bond) · line 2277

Quoted 6 lines
drivers/net/bonding/bond_main.c +0 -3
@@ -2272,9 +2272,6 @@ static void bond_miimon_commit(struct bonding *bond)
                      } else if (BOND_MODE(bond) != BOND_MODE_ACTIVEBACKUP) {
                              /* make it immediately active */
                              bond_set_active_slave(slave);
-                      } else if (slave != primary) {
-                              /* prevent it from being the active one */
-                              bond_set_backup_slave(slave);
        How does this fix things?  Doesn't bond_select_active_slave() ->
Quoted 5 lines

bond_change_active_slave() set the backup flag correctly via a call to bond_set_slave_active_flags() when it sets a slave to be the active slave? If this change resolves the problem, I'm not sure how this ever worked correctly, even prior to 1899bb325149.

Quoted 10 lines

Hi Jay, I used kprobes to figure out the brokenness this patch fixes. Prior to your patch this call would not happen but with the patch, this extra call will put the master into the backup mode erroneously (in fact both members would be in backup state). The mechanics you have mentioned works correctly except that in the prior case, the switch statement was using new_link which was not same as link_new_state. The miimon_inspect will update new_link which is what was used in miimon_commit code. The link_new_state was used only to mitigate the rtnl-lock issue which would update the "link". Hence in the prior code, this path would never get executed.

Quoted 4 lines

I'm looking at the old code (prior to 1899bb325149), and I don't
see a path to what you're describing for the down to up transition in active-backup mode.

I was referring to the code where bond_miimon_inspect() switches using bond->link and bond_miimon_commit() (which happens after inspect) switches using bond->new_link. inspect doesn't touch new_link unless delay is set which is a corner case and probably ignore for this purpose since it's just postponing the behavior. bond->link_new_state was brought in to mitigate RTNL issue and affects only bond->link, if it can acquire RTNL. So irrespective of what bond_miimon_inspect() does for bond->link or bond->link_new_state the bond->new_link was maintained and then used in the bond_miimon_commit. Because of this the wrong transition wouldn't happen.

Once the new_link and link_new_state is merged, the state that bond_miimon_inspect() sets for bond->link_new_state *is* used in bond_miimon_commit() (which is after the fact) and hence (I believe) the erroneous transition.

Quoted 4 lines

Having said that, the fix that you put in is necessary to close the window between link_propose() and link_commit() but the side effect of that was the situation that I explained
above which is what this patch fixes it.

Ok, I think I understand, and am fine with the patch as-is.

	-J
Quoted 18 lines
Quoted 18 lines

bond_miimon_inspect enters switch, slave->link == BOND_LINK_DOWN.

link_state is nonzero, call bond_propose_link_state(BOND_LINK_BACK), which sets slave->link_new_state to _BACK.

Fall through to BOND_LINK_BACK case, set slave->new_link = BOND_LINK_UP

bond_mii_monitor then calls bond_commit_link_state, which sets slave->link to BOND_LINK_BACK

Enter bond_miimon_commit switch (new_link), which is BOND_LINK_UP

In "case BOND_LINK_UP:" there is no way out of this block, and it should proceed to call bond_set_backup_slave for active-backup mode every time.

Quoted 2 lines

The steps to reproduce this issue is straightforward and happens 100% of the time (I used two mlx interfaces but that shouldn't matter).

Quoted 21 lines
Quoted 5 lines

Yes, I've been able to reproduce it locally (with igb, FWIW). I
think the patch is likely ok, I'm just mystified as to how the backup setting could have worked prior to 1899bb325149, so perhaps the Fixes tag doesn't go back far enough.

Well, I added fixes-tag since the behavior started as soon as the 1899bb325149 was added. I don't see the issue if I revert 1899bb325149.

Quoted 11 lines
        -J
Quoted 9 lines

thanks,
--mahesh..

Quoted 7 lines
        -J
Quoted 5 lines
drivers/net/bonding/bond_main.c +0 -1
                      }
 
                      slave_info(bond->dev, slave->dev, "link status definitely up, %u Mbps %s duplex\n",
--

2.24.0.393.g34dc348eaf-goog

---
-Jay Vosburgh, [email protected]

#7 lore raw
Quoted 3 lines · Jay Vosburgh

Ok, I think I understand, and am fine with the patch as-is.

Okay, applied then, and queued for 4.14+ stable.

Mahesh, I reworded the commit message slightly, checkpatch insists that the word "commit" occurs before the hash when quoting.

Thanks!

#8 lore raw
Quoted 11 lines · Jakub Kicinski
Quoted 3 lines · Jay Vosburgh

Ok, I think I understand, and am fine with the patch as-is.

Okay, applied then, and queued for 4.14+ stable.

Mahesh, I reworded the commit message slightly, checkpatch insists that the word "commit" occurs before the hash when quoting.

Thanks and good to know, I'll keep that in mind.

Quoted 1 line

Thanks!