Skip to contentSkip to conversation

lost gARP after live migration

PATCH Superseded Laszlo Ersek 5 participants 10 messages Jun 28, 2011 – Jul 14, 2011

Subsystems (3)

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

Hi,

with reference to RHBZ#713585:

It seems when a RHEL-6.1 or F-15 Xen PV guest is live migrated, the gratuitous ARP packet is not forwarded to the affected "networking equipment". The netback vif is added to a routed bridge in the host(s) and external hosts are expeted to have connection to the guest at all times, no matter the current Xen host.

I experimented a bit with tcpdump, and the gARP does appear on the netfront interface. It also appears on the host bridge if sufficient time passes between completing the xenbus handshake and sending the gARP.

When the guest queues eg. three gARPs in rapid succession, a variable number of them gets lost. (When all such packets disappear, then the migrated guest becomes invisible to the outside world, until it initiates network traffic on its own.)

When the guest waits for about half a second before sending (queueing), the very first gARP packet successfully appears on the host bridge.

I suspect it's a timing race against the netback vif being added to the host bridge. What would be a good countermeasure?

- Adding two modparams to xen-netfront (gARP requeue count & number of msecs to wait between queueing the gARPs).
- (Paolo's idea:) watching the "hotplug-status" xenstore node and sending a single gARP when the watch fires with "connected". This node belongs to the backend xenstore subtree, thus watching it from the guest doesn't please the architecture astronaut in me.
- Something else.

Sorry for the naivety / verbiage.

Thanks,
lacos

#1 lore raw
Quoted 4 lines · Laszlo Ersek

- (Paolo's idea:) watching the "hotplug-status" xenstore node and sending a single gARP when the watch fires with "connected". This node belongs to the backend xenstore subtree, thus watching it from the guest doesn't please the architecture astronaut in me.

Note that watching the backend and reading its information is quite common. In fact, that's how the state of the backend is observed in the first place. Of course you cannot write to the backend tree, but you do not have to do that.

Paolo

#2 lore raw
In reply to Laszlo Ersek
Quoted 25 lines · Laszlo Ersek

Hi,

with reference to RHBZ#713585:

It seems when a RHEL-6.1 or F-15 Xen PV guest is live migrated, the gratuitous ARP packet is not forwarded to the affected "networking equipment". The netback vif is added to a routed bridge in the host(s) and external hosts are expeted to have connection to the guest at all times, no matter the current Xen host.

I experimented a bit with tcpdump, and the gARP does appear on the netfront interface. It also appears on the host bridge if sufficient time passes between completing the xenbus handshake and sending the gARP.

When the guest queues eg. three gARPs in rapid succession, a variable number of them gets lost. (When all such packets disappear, then the migrated guest becomes invisible to the outside world, until it initiates network traffic on its own.)

When the guest waits for about half a second before sending (queueing), the very first gARP packet successfully appears on the host bridge.

I suspect it's a timing race against the netback vif being added to the host bridge. What would be a good countermeasure?

Quoted 6 lines

- Adding two modparams to xen-netfront (gARP requeue count & number of msecs to wait between queueing the gARPs).
- (Paolo's idea:) watching the "hotplug-status" xenstore node and sending a single gARP when the watch fires with "connected". This node belongs to the backend xenstore subtree, thus watching it from the guest doesn't please the architecture astronaut in me.

netback already waits (or should...) for hotplug-status to fire with "connected" before moving to state XenbusStateConnected. See hotplug_status_changed in drivers/net/xen-netback/xenbus.c. You need either the netback in upstream or something newer than 43223efd9bfd (C Feb 2010) if you are using e.g. xen.git#xen/next-2.6.32. That commit fixes pretty much the issue you describe.

I expected that netfront waited for the backend to hit XenbusStateConnected before sending the grat ARP but instead I find it happens when the backend hits XenbusStateInitWait. I'm not sure if that is a problem -- it appears to have been done this way since forever (even back in the classic Xen kernels) and I've never noticed a gARP go missing in the way you describe, but perhaps something isn't quite matching up any more.

Ian.

Quoted 11 lines

- Something else.

Sorry for the naivety / verbiage.

Thanks,
lacos

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#3 lore raw

PATCH 1/1 xen-netfront: delay gARP until backend switches to Connected

Quoted 7 lines · Ian Campbell

netback already waits (or should...) for hotplug-status to fire with "connected" before moving to state XenbusStateConnected. See hotplug_status_changed in drivers/net/xen-netback/xenbus.c. You need either the netback in upstream or something newer than 43223efd9bfd (C Feb 2010) if you are using e.g. xen.git#xen/next-2.6.32. That commit fixes pretty much the issue you describe.

Quoted 7 lines

I expected that netfront waited for the backend to hit XenbusStateConnected before sending the grat ARP but instead I find it happens when the backend hits XenbusStateInitWait. I'm not sure if that is a problem -- it appears to have been done this way since forever (even back in the classic Xen kernels) and I've never noticed a gARP go missing in the way you describe, but perhaps something isn't quite matching up any more.

In addition to backporting 43223efd9bfd to the RHEL-5 host side, we needed the following in the RHEL-6 guest, in order to fix the network outage after live migration. The patch is for xen/next-2.6.32. I also tested a Fedora-15 guest (without the patch), and the backport to the host side alone was not sufficient there either.

Thanks for the help and for considering this one.

---
 drivers/net/xen-netfront.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)
drivers/net/xen-netfront.c +3 -1
index 3f71199..214c3a4 100644
@@ -1732,7 +1732,6 @@ static void netback_changed(struct xenbus_device *dev,
 	case XenbusStateInitialised:
 	case XenbusStateReconfiguring:
 	case XenbusStateReconfigured:
-	case XenbusStateConnected:
 	case XenbusStateUnknown:
 	case XenbusStateClosed:
 		break;
@@ -1743,6 +1742,9 @@ static void netback_changed(struct xenbus_device *dev,
 		if (xennet_connect(netdev) != 0)
 			break;
 		xenbus_switch_state(dev, XenbusStateConnected);
+		break;
+
+	case XenbusStateConnected:
 		netif_notify_peers(netdev);
 		break;
 
Signature · 1 line
1.7.4.4
#4 lore raw

Re: [PATCH] xen-netfront: delay gARP until backend switches to Connected

Quoted 3 lines

migration. The patch is for xen/next-2.6.32. I also tested a Fedora-15 guest (without the patch), and the backport to the host side alone was not sufficient there either.

I am having a hard time parsing that. It sounds like this patch is also neccessary for upstream kernel? Or do you have another one for the upstream kernel that you are going to submit soon?
(If so, please CC David Miller on it).

Quoted 38 lines

Thanks for the help and for considering this one.

---
 drivers/net/xen-netfront.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)
drivers/net/xen-netfront.c +3 -1
index 3f71199..214c3a4 100644
@@ -1732,7 +1732,6 @@ static void netback_changed(struct xenbus_device *dev,
 	case XenbusStateInitialised:
 	case XenbusStateReconfiguring:
 	case XenbusStateReconfigured:
-	case XenbusStateConnected:
 	case XenbusStateUnknown:
 	case XenbusStateClosed:
 		break;
@@ -1743,6 +1742,9 @@ static void netback_changed(struct xenbus_device *dev,
 		if (xennet_connect(netdev) != 0)
 			break;
 		xenbus_switch_state(dev, XenbusStateConnected);
+		break;
+
+	case XenbusStateConnected:
 		netif_notify_peers(netdev);
 		break;
 
Signature · 7 lines
1.7.4.4


_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
#5 lore raw

PATCH v2 1/1 xen-netfront: delay gARP until backend switches to Connected

In addition to backporting 43223efd9bfd to the RHEL-5 host side, we needed the following in the RHEL-6 guest, in order to fix the network outage after live migration. I also tested a Fedora-15 guest (without the patch), and the backport to the host side alone was not sufficient there either. Therefore I think it could be useful to apply this guest-side patch to upstream as well.

Reference: https://bugzilla.redhat.com/show_bug.cgi?id=713585

v1->v2: "port" to Linus's tree; add RHBZ ref.

---
 drivers/net/xen-netfront.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)
drivers/net/xen-netfront.c +3 -1
index d29365a..f033656 100644
@@ -1646,7 +1646,6 @@ static void netback_changed(struct xenbus_device *dev,
 	case XenbusStateInitialised:
 	case XenbusStateReconfiguring:
 	case XenbusStateReconfigured:
-	case XenbusStateConnected:
 	case XenbusStateUnknown:
 	case XenbusStateClosed:
 		break;
@@ -1657,6 +1656,9 @@ static void netback_changed(struct xenbus_device *dev,
 		if (xennet_connect(netdev) != 0)
 			break;
 		xenbus_switch_state(dev, XenbusStateConnected);
+		break;
+
+	case XenbusStateConnected:
 		netif_notify_peers(netdev);
 		break;
 
Signature · 1 line
1.7.4.4
#6 lore raw

Re: [PATCH v2] xen-netfront: delay gARP until backend switches to Connected

Quoted 3 lines · Laszlo Ersek

In addition to backporting 43223efd9bfd to the RHEL-5 host side, we needed the following in the RHEL-6 guest, in order to fix the network outage after live migration. I also tested a Fedora-15 guest (without the patch), and the

Laszlo,

This description is .. well, pointless for upstream patches. Just succinctly describe the problem, how to reproduce it, and what this patch does.

And do mention more about 43223efd9bfd, like "this is the front counterpart of patch xen: netback: wait for hotplug scripts to complete before signalling connected to frontend (43223efd9bfd)"

Quoted 42 lines

backport to the host side alone was not sufficient there either. Therefore I think it could be useful to apply this guest-side patch to upstream as well.

Reference: https://bugzilla.redhat.com/show_bug.cgi?id=713585

v1->v2: "port" to Linus's tree; add RHBZ ref.

---
 drivers/net/xen-netfront.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)
drivers/net/xen-netfront.c +3 -1
index d29365a..f033656 100644
@@ -1646,7 +1646,6 @@ static void netback_changed(struct xenbus_device *dev,
 	case XenbusStateInitialised:
 	case XenbusStateReconfiguring:
 	case XenbusStateReconfigured:
-	case XenbusStateConnected:
 	case XenbusStateUnknown:
 	case XenbusStateClosed:
 		break;
@@ -1657,6 +1656,9 @@ static void netback_changed(struct xenbus_device *dev,
 		if (xennet_connect(netdev) != 0)
 			break;
 		xenbus_switch_state(dev, XenbusStateConnected);
+		break;
+
+	case XenbusStateConnected:
 		netif_notify_peers(netdev);
 		break;
 
Signature · 7 lines
1.7.4.4


_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
#7 lore raw

PATCH v3 1/1 xen-netfront: delay gARP until backend switches to Connected

Quoted 9 lines · Konrad Rzeszutek Wilk
Quoted 3 lines · Laszlo Ersek

In addition to backporting 43223efd9bfd to the RHEL-5 host side, we needed the following in the RHEL-6 guest, in order to fix the network outage after live migration. I also tested a Fedora-15 guest (without the patch), and the

Laszlo,

This description is .. well, pointless for upstream patches. Just succinctly describe the problem, how to reproduce it, and what this patch does.

I avoided writing up the commit message myself because in my first posting [1] I quoted two paragraphs from Ian Campbell [2]. I intended those two cited paragraphs verbatim as the commit message (along with my short remarks), because they describe the problem exactly. Anyway:

After a guest is live migrated, the xen-netfront driver emits a gratuitous ARP message, so that networking hardware on the target host's subnet can take notice, and public routing to the guest is re-established. However, if the packet appears on the backend interface before the backend is added to the target host's bridge, the packet is lost, and the migrated guest's peers become unable to talk to the guest.

A sufficient two-parts condition to prevent the above is:

(1) ensure that the backend only moves to Connected xenbus state after its hotplug scripts completed, ie. the netback interface got added to the bridge; and

(2) ensure the frontend only queues the gARP when it sees the backend move to Connected.

These two together provide complete ordering. Sub-condition (1) is satisfied by pvops commit 43223efd9bfd.

In general, the full condition is sufficient, not necessary, because, according to [2], live migration has been working for a long time without satisfying sub-condition (2). However, after 43223efd9bfd was backported to the RHEL-5 host to ensure (1), (2) still proved necessary in the RHEL-6 guest. This patch intends to provide (2) for upstream.

[1] http://lists.xensource.com/archives/html/xen-devel/2011-07/msg00327.html
[2] http://lists.xensource.com/archives/html/xen-devel/2011-06/msg01969.html

---
 drivers/net/xen-netfront.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)
drivers/net/xen-netfront.c +3 -1
index d29365a..f033656 100644
@@ -1646,7 +1646,6 @@ static void netback_changed(struct xenbus_device *dev,
 	case XenbusStateInitialised:
 	case XenbusStateReconfiguring:
 	case XenbusStateReconfigured:
-	case XenbusStateConnected:
 	case XenbusStateUnknown:
 	case XenbusStateClosed:
 		break;
@@ -1657,6 +1656,9 @@ static void netback_changed(struct xenbus_device *dev,
 		if (xennet_connect(netdev) != 0)
 			break;
 		xenbus_switch_state(dev, XenbusStateConnected);
+		break;
+
+	case XenbusStateConnected:
 		netif_notify_peers(netdev);
 		break;
 
Signature · 1 line
1.7.4.4
Reviewed-by on v3 1/1
#8 lore raw

Re: [PATCH v3] xen-netfront: delay gARP until backend switches to Connected

Quoted 8 lines · Laszlo Ersek
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 3 lines · Laszlo Ersek

In addition to backporting 43223efd9bfd to the RHEL-5 host side, we needed the following in the RHEL-6 guest, in order to fix the network outage after live migration. I also tested a Fedora-15 guest (without the patch), and the

Laszlo,

Quoted 7 lines
Quoted 2 lines

This description is .. well, pointless for upstream patches. Just succinctly describe the problem, how to reproduce it, and what this patch does.

I avoided writing up the commit message myself because in my first posting [1] I quoted two paragraphs from Ian Campbell [2]. I intended those two cited paragraphs verbatim as the commit message (along with my short remarks), because they describe the problem exactly.

Please don't expect/require that maintainers trawl around and construct a commit message for you, always propose the full text you would like to see committed in each patch posting.

Quoted 18 lines

Anyway:

After a guest is live migrated, the xen-netfront driver emits a gratuitous ARP message, so that networking hardware on the target host's subnet can take notice, and public routing to the guest is re-established. However, if the packet appears on the backend interface before the backend is added to the target host's bridge, the packet is lost, and the migrated guest's peers become unable to talk to the guest.

A sufficient two-parts condition to prevent the above is:

(1) ensure that the backend only moves to Connected xenbus state after its hotplug scripts completed, ie. the netback interface got added to the bridge; and

(2) ensure the frontend only queues the gARP when it sees the backend move to Connected.

Quoted 8 lines

These two together provide complete ordering. Sub-condition (1) is satisfied by pvops commit 43223efd9bfd.

In general, the full condition is sufficient, not necessary, because, according to [2], live migration has been working for a long time without satisfying sub-condition (2). However, after 43223efd9bfd was backported to the RHEL-5 host to ensure (1), (2) still proved necessary in the RHEL-6 guest. This patch intends to provide (2) for upstream.

I expect that 43223efd9bfd just reduces the window (enough to prevent it occuring most of the time) but doesn't actually close it. This change seems good to me.

Quoted 4 lines

[1] http://lists.xensource.com/archives/html/xen-devel/2011-07/msg00327.html
[2] http://lists.xensource.com/archives/html/xen-devel/2011-06/msg01969.html

Quoted 26 lines
---
 drivers/net/xen-netfront.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)
drivers/net/xen-netfront.c +3 -1
index d29365a..f033656 100644
@@ -1646,7 +1646,6 @@ static void netback_changed(struct xenbus_device *dev,
 	case XenbusStateInitialised:
 	case XenbusStateReconfiguring:
 	case XenbusStateReconfigured:
-	case XenbusStateConnected:
 	case XenbusStateUnknown:
 	case XenbusStateClosed:
 		break;
@@ -1657,6 +1656,9 @@ static void netback_changed(struct xenbus_device *dev,
 		if (xennet_connect(netdev) != 0)
 			break;
 		xenbus_switch_state(dev, XenbusStateConnected);
+		break;
+
+	case XenbusStateConnected:
 		netif_notify_peers(netdev);
 		break;
 
#9 lore raw
In reply to Laszlo Ersek

On Tue, 2011-06-28 at 15:01 +0200, Laszlo Ersek wrote:
[...]

Quoted 8 lines

When the guest waits for about half a second before sending (queueing), the very first gARP packet successfully appears on the host bridge.

I suspect it's a timing race against the netback vif being added to the host bridge. What would be a good countermeasure?

- Adding two modparams to xen-netfront (gARP requeue count & number of msecs to wait between queueing the gARPs).

Note that peer notifications are indirected through netdev notifiers and now include IPv6 NAs as well as ARPs. If repeated notifications are commonly necessary then this should probably be handled in the protocol (or in the networking core). However this sounds like a workaround whereas your other option would be a proper fix:

Quoted 4 lines

- (Paolo's idea:) watching the "hotplug-status" xenstore node and sending a single gARP when the watch fires with "connected". This node belongs to the backend xenstore subtree, thus watching it from the guest doesn't please the architecture astronaut in me.

[...]

Signature · 3 lines
Ben Hutchings, Senior Software Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.