Skip to contentSkip to conversation

New: Bonding Driver(version : 3.5.0) - Problem with ARP monitoring in active backup mode

Bugme-new Andrew Morton 5 participants 7 messages Feb 24, 2011 – Mar 4, 2011 3 messages not in the archive

Subsystems (3)
Networking Network drivers Bonding driver
lore
Share & tools
lore

Andrew Morton

author
#1 lore raw
In reply to a message not in the archive

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 7 lines · On Wed, 23 Feb 2011 10:41:34 GMT [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=29712

Summary: Bonding Driver(version : 3.5.0) - Problem with ARP
monitoring in active backup mode
Product: Drivers
Version: 2.5
Kernel Version: 2.6.32

That's a paleolithic kernel you have there. This problem might have been fixed already. Can you test a more recent kernel?

Quoted 118 lines

Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Network
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: No

We are facing an issue with arp_monitoring in active_backup mode when two network interfaces of two systems are connected back to back (point to point connected without switch connection) and bond is created on either systems with point-to-point connected interfaces as slaves.

Steps to reproduce :

1. Initially the bond was created with two interfaces eth2 and eth3, having eth2 as primary

    # modprobe bonding primary=eth2 mode=1 arp_interval=500
    arp_ip_target=192.168.4.61
    # ifconfig bond0 192.168.2.63 netmask 255.255.255.0
    # ifenslave bond0 eth2 eth3
    # ifconfig bond0 up
    # cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)

Bonding Mode: fault-tolerance (active-backup)

Primary Slave: eth2
Currently Active Slave: eth2
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
ARP Polling Interval (ms): 500
ARP IP target/s (n.n.n.n form): 192.168.4.61

Slave Interface: eth2
MII Status: up
Link Failure Count: 1
Permanent HW addr: 00:26:55:27:88:52

Slave Interface: eth3
MII Status: down
Link Failure Count: 1
Permanent HW addr: 00:26:55:27:88:54

2. The primary interface was made down, and fail over happened

    # ifconfig down
    # cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)

Bonding Mode: fault-tolerance (active-backup) Primary Slave: eth2
Currently Active Slave: eth3 <-- As expected --> MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
ARP Polling Interval (ms): 500
ARP IP target/s (n.n.n.n form): 192.168.4.61

Slave Interface: eth2
MII Status: down
Link Failure Count: 2
Permanent HW addr: 00:26:55:27:88:52

Slave Interface: eth3
MII Status: up
Link Failure Count: 1
Permanent HW addr: 00:26:55:27:88:54

3. The primary interface was brought up again and we did not see failover happening back to primary

    ned1g6:~# ifconfig eth2 up
    ned1g6:~# cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: eth2
Currently Active Slave: eth3 <-- Ideally this should have been eth2 --> MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
ARP Polling Interval (ms): 500
ARP IP target/s (n.n.n.n form): 192.168.4.61

Slave Interface: eth2
MII Status: down
Link Failure Count: 2
Permanent HW addr: 00:26:55:27:88:52

Slave Interface: eth3
MII Status: up
Link Failure Count: 1
Permanent HW addr: 00:26:55:27:88:54

The problem is that when the primary_slave comes up from the down state it won't get selected as the currently active slave for the bond.

Best Regards,
Harsha

#2 lore raw
Quoted 8 lines · Andrew Morton

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 2 lines · On Wed, 23 Feb 2011 10:41:34 GMT [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=29712

Quoted 8 lines
Quoted 5 lines

Summary: Bonding Driver(version : 3.5.0) - Problem with ARP
monitoring in active backup mode
Product: Drivers
Version: 2.5
Kernel Version: 2.6.32

That's a paleolithic kernel you have there. This problem might have been fixed already. Can you test a more recent kernel?

I can add some more info since I originally looked at the problem. This happens on 2.6.38 as well, and on this 2.6.32 kernel with a backported 3.7.0 bonding driver (with the primary_reselect option). Harsha has a prototype patch that's being tested, but wanted to log the bug to see if one of the bonding maintainers had a better solution.

I'll let him respond as I'm now out of the loop...

Thanks,

-Brian

#3 lore raw

Attached patch resolves the issue. Failover happened back to primary when it was up again in both the point to point and switch configuration.

Please let us know if this change can be included.

Thanks,

- Harsha

-----Original Message-----
From: Brian Haley [mailto:[email protected]]
Sent: Friday, February 25, 2011 9:12 AM
To: Andrew Morton
Cc: Harsha R02; [email protected]; [email protected]; [email protected]; Jay Vosburgh
Subject: Re: [Bugme-new] [Bug 29712] New: Bonding Driver(version :
3.5.0) - Problem with ARP monitoring in active backup mode

Quoted 1 line · Andrew Morton

(switched to email. Please respond via emailed reply-to-all, not via

the

Quoted 8 lines

bugzilla web interface).

Quoted 3 lines · On Wed, 23 Feb 2011 10:41:34 GMT [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=29712

Summary: Bonding Driver(version : 3.5.0) - Problem with

ARP

Quoted 7 lines
Quoted 4 lines
                    monitoring in active backup mode
           Product: Drivers
           Version: 2.5
    Kernel Version: 2.6.32

That's a paleolithic kernel you have there. This problem might have been fixed already. Can you test a more recent kernel?

I can add some more info since I originally looked at the problem. This happens on 2.6.38 as well, and on this 2.6.32 kernel with a backported 3.7.0 bonding driver (with the primary_reselect option). Harsha has a prototype patch that's being tested, but wanted to log the bug to see if one of the bonding maintainers had a better solution.

I'll let him respond as I'm now out of the loop...

Thanks,

-Brian

Information transmitted by this e-mail is proprietary to MphasiS, its associated companies and/ or its customers and is intended for use only by the individual or entity to which it is addressed, and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly prohibited. In such cases, please notify us immediately at [email protected] and delete this mail from your records.

drivers/net/bonding/bond_main.c +5 -0
index 40fb5ee..0413917 100644
@@ -3020,11 +3020,16 @@ static void bond_ab_arp_probe(struct bonding *bond)
                       bond->curr_active_slave->dev->name);
 
        if (bond->curr_active_slave) {
+                if((bond->curr_active_slave != bond->primary_slave) &&
+                   (IS_UP(bond->primary_slave->dev)))
+                        goto failover;
+
                bond_arp_send_all(bond, bond->curr_active_slave);
                read_unlock(&bond->curr_slave_lock);
                return;
        }
 
+failover:
        read_unlock(&bond->curr_slave_lock);
 
        /* if we don't have a curr_active_slave, search for the next available
maintainer
#4 lore raw

From: "Harsha R02" <[email protected]>
Date: Fri, 25 Feb 2011 18:14:32 +0530

Quoted 5 lines

Attached patch resolves the issue. Failover happened back to primary when it was up again in both the point to point and switch configuration.

Please let us know if this change can be included.

Please don't base64 encode your patches, that makes them harder to read for some people. It's just plain text.

#7 lore raw
In reply to a message not in the archive
Quoted 8 lines · Harsha R02 <[email protected]> wrote:
drivers/net/bonding/bond_main.c +1 -0
index 40fb5ee..0413917 100644
@@ -3020,11 +3020,16 @@ static void bond_ab_arp_probe(struct bonding *bond)
                      bond->curr_active_slave->dev->name);
       if (bond->curr_active_slave) {
+                if((bond->curr_active_slave != bond->primary_slave) &&

drivers/net/bonding/bond_main.c static void bond_ab_arp_probe(struct bonding *bond) · line 3032

Quoted 10 lines
drivers/net/bonding/bond_main.c +4 -0
+                   (IS_UP(bond->primary_slave->dev)))
+                        goto failover;
+
               bond_arp_send_all(bond, bond->curr_active_slave);
               read_unlock(&bond->curr_slave_lock);
               return;
       }
+failover:
       read_unlock(&bond->curr_slave_lock);
       /* if we don't have a curr_active_slave, search for the next available

I'm not sure this is the proper place to put the "failover:"
label, as it will go through the "search for any peer" logic that's normally used when there are no available slaves. That will likely take longer than simply switching to the primary.

It should be possible to simply call bond_change_active_slave
with the appropriate arguments; did you try this?

	-J
Quoted 51 lines
drivers/net/bonding/bond_main.c +0 -1
-------------------------------------------------------------------------------

From: Harsha R02
Sent: Fri 2/25/2011 6:14 PM
To: Brian Haley; Andrew Morton
Cc: [email protected]; [email protected]; [email protected]; Jay Vosburgh
Subject: RE: [Bugme-new] [Bug 29712] New: Bonding Driver(version : 3.5.0) - Problem with ARP monitoring in active backup mode

Attached patch resolves the issue. Failover happened back to primary when it was up again in both the point to point and switch configuration.

Please let us know if this change can be included.

Thanks,

- Harsha

-----Original Message-----
From: Brian Haley [mailto:[email protected]]
Sent: Friday, February 25, 2011 9:12 AM
To: Andrew Morton
Cc: Harsha R02; [email protected]; [email protected]; [email protected]; Jay Vosburgh
Subject: Re: [Bugme-new] [Bug 29712] New: Bonding Driver(version : 3.5.0) - Problem with ARP monitoring in active backup mode

Quoted 8 lines · Andrew Morton

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 2 lines · On Wed, 23 Feb 2011 10:41:34 GMT [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=29712

Quoted 8 lines
Quoted 5 lines

Summary: Bonding Driver(version : 3.5.0) - Problem with ARP
monitoring in active backup mode
Product: Drivers
Version: 2.5
Kernel Version: 2.6.32

That's a paleolithic kernel you have there. This problem might have been fixed already. Can you test a more recent kernel?

I can add some more info since I originally looked at the problem. This happens on 2.6.38 as well, and on this 2.6.32 kernel with a backported 3.7.0 bonding driver (with the primary_reselect option). Harsha has a prototype patch that's being tested, but wanted to log the bug to see if one of the bonding maintainers had a better solution.

Quoted 5 lines

I'll let him respond as I'm now out of the loop...

Thanks,

-Brian

---
-Jay Vosburgh, IBM Linux Technology Center, [email protected]

#8 lore raw

Hi Jay,

We found that the patch that is presented here has some issues and we cannot go with this solution.

In function "bond_ab_arp_probe" in addition to sending arp probes for the currently active slave we should also
be sending arp probes for the primary_slave if the link status of the primary slave is up correct ?

I have made changes as below :

static void bond_ab_arp_probe(struct bonding *bond)

{
        struct slave *slave;
        int i;

        read_lock(&bond->curr_slave_lock);

        if (bond->current_arp_slave && bond->curr_active_slave)
                pr_info(DRV_NAME "PROBE: c_arp %s && cas %s BAD\n",
                       bond->current_arp_slave->dev->name,
                       bond->curr_active_slave->dev->name);

        if (bond->curr_active_slave) {

+ if((bond->curr_active_slave != bond->primary_slave) &&
+ (IS_UP(bond->primary_slave->dev))) {
+ bond_arp_send_all(bond, bond->primary_slave);
+ }

                bond_arp_send_all(bond, bond->curr_active_slave);
                read_unlock(&bond->curr_slave_lock);
                return;
        }

Please let us know if this can help us ? or if you see any side effects ?

Thanks,
Harsha

-----Original Message-----
From: Jay Vosburgh [mailto:[email protected]]
Sent: Saturday, February 26, 2011 12:33 AM
To: Harsha R02
Cc: Brian Haley; Andrew Morton; [email protected]; [email protected]; [email protected]
Subject: Re: [Bugme-new] [Bug 29712] New: Bonding Driver(version :
3.5.0) - Problem with ARP monitoring in active backup mode

Review

Quoted 1 line · Harsha R02 <[email protected]> wrote:
(quoted diff) +0 -0

b/drivers/net/bonding/bond_main.c

drivers/net/bonding/bond_main.c static void bond_ab_arp_probe(struct bonding

Quoted 4 lines

index 40fb5ee..0413917 100644

drivers/net/bonding/bond_main.c +0 -0
@@ -3020,11 +3020,16 @@ static void bond_ab_arp_probe(struct bonding

*bond)

drivers/net/bonding/bond_main.c static void bond_ab_arp_probe(struct bonding · line 3032

Quoted 13 lines
drivers/net/bonding/bond_main.c +5 -0
                      bond->curr_active_slave->dev->name);
       if (bond->curr_active_slave) {
+                if((bond->curr_active_slave != bond->primary_slave) &&
+                   (IS_UP(bond->primary_slave->dev)))
+                        goto failover;
+
               bond_arp_send_all(bond, bond->curr_active_slave);
               read_unlock(&bond->curr_slave_lock);
               return;
       }
+failover:
       read_unlock(&bond->curr_slave_lock);
       /* if we don't have a curr_active_slave, search for the next

available

I'm not sure this is the proper place to put the "failover:"
label, as it will go through the "search for any peer" logic that's normally used when there are no available slaves. That will likely take longer than simply switching to the primary.

It should be possible to simply call bond_change_active_slave
with the appropriate arguments; did you try this?

	-J

drivers/net/bonding/bond_main.c static void bond_ab_arp_probe(struct bonding · line 3028

Quoted 1 line
drivers/net/bonding/bond_main.c +0 -1
-----------------------------------------------------------------------

--------

Quoted 4 lines

From: Harsha R02
Sent: Fri 2/25/2011 6:14 PM
To: Brian Haley; Andrew Morton
Cc: [email protected];

[email protected];

Quoted 2 lines

[email protected]; Jay Vosburgh
Subject: RE: [Bugme-new] [Bug 29712] New: Bonding Driver(version :

3.5.0) -

Quoted 3 lines

Problem with ARP monitoring in active backup mode

Attached patch resolves the issue. Failover happened back to primary

when it

Quoted 8 lines

was up again in both the point to point and switch configuration.

Please let us know if this change can be included.

Thanks,

- Harsha

Quoted 7 lines

-----Original Message-----
From: Brian Haley [mailto:[email protected]]
Sent: Friday, February 25, 2011 9:12 AM
To: Andrew Morton
Cc: Harsha R02; [email protected]; [email protected]; [email protected]; Jay Vosburgh
Subject: Re: [Bugme-new] [Bug 29712] New: Bonding Driver(version :

3.5.0) -

Quoted 4 lines

Problem with ARP monitoring in active backup mode

Quoted 1 line · Andrew Morton

(switched to email. Please respond via emailed reply-to-all, not via

the

Quoted 8 lines
Quoted 8 lines

bugzilla web interface).

Quoted 3 lines · On Wed, 23 Feb 2011 10:41:34 GMT [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=29712

Summary: Bonding Driver(version : 3.5.0) - Problem with

ARP

Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
                    monitoring in active backup mode
           Product: Drivers
           Version: 2.5
    Kernel Version: 2.6.32

That's a paleolithic kernel you have there. This problem might have been fixed already. Can you test a more recent kernel?

I can add some more info since I originally looked at the problem.

This

Quoted 10 lines

happens on 2.6.38 as well, and on this 2.6.32 kernel with a backported 3.7.0 bonding driver (with the primary_reselect option). Harsha has a prototype patch that's being tested, but wanted to log the bug to see if one of the bonding maintainers had a better solution.

I'll let him respond as I'm now out of the loop...

Thanks,

-Brian

---
-Jay Vosburgh, IBM Linux Technology Center, [email protected]

Information transmitted by this e-mail is proprietary to MphasiS, its associated companies and/ or its customers and is intended for use only by the individual or entity to which it is addressed, and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly prohibited. In such cases, please notify us immediately at [email protected] and delete this mail from your records.

#9 lore raw
Quoted 23 lines · Harsha R02 <[email protected]> wrote:

We found that the patch that is presented here has some issues and we cannot go with this solution.

In function "bond_ab_arp_probe" in addition to sending arp probes for the currently active slave we should also
be sending arp probes for the primary_slave if the link status of the primary slave is up correct ?

I have made changes as below :

static void bond_ab_arp_probe(struct bonding *bond)

{
       struct slave *slave;
       int i;

       read_lock(&bond->curr_slave_lock);

       if (bond->current_arp_slave && bond->curr_active_slave)
               pr_info(DRV_NAME "PROBE: c_arp %s && cas %s BAD\n",
                      bond->current_arp_slave->dev->name,
                      bond->curr_active_slave->dev->name);

       if (bond->curr_active_slave) {

Review

Quoted 6 lines
(quoted diff) +4 -0
+                if((bond->curr_active_slave != bond->primary_slave) &&
+                   (IS_UP(bond->primary_slave->dev))) {
+                    bond_arp_send_all(bond, bond->primary_slave);
+                }
               bond_arp_send_all(bond, bond->curr_active_slave);
               read_unlock(&bond->curr_slave_lock);

No, we can't do this; if we send ARP probes out from an inactive
slave (which the primary would be at this point) it will confuse switches that snoop traffic to determine the switch port's MAC addresses (the switches will believe that the "primary" slave is the port to use to reach the bond's MAC address).

I think your problem is that your configuration (two systems,
back to back, no switch) is not a configuration the ARP monitor is designed to work with.

The ARP monitor determines the availability of backup slaves
based on traffic received by the backup slaves. The usual source of this traffic is the ARP broadcast requests being sent out the active slave and then forwarded by the switch to all switch ports, including the backup slave's port. I'm guessing that your system isn't forwarding these packets like a switch would, and so the primary slave isn't seeing any incoming packets at all.

If your primary slave (which is an inactive slave at the moment)
is not receiving traffic, bonding will never believe it is available.

I've never experimented with using the ARP monitor in a
back-to-back confguration; I'm thinking through how the ARP monitor functions, and I'm not sure it can be reliable when set up like this.

	-J

---
-Jay Vosburgh, IBM Linux Technology Center, [email protected]