lost gARP after live migration
Subsystems (3)
Hi,
with reference to RHBZ#713585:
It seems when a RHEL-6.1 or F-15 Xen PV guest is live migrated, the gratuitous ARP packet is not forwarded to the affected "networking equipment". The netback vif is added to a routed bridge in the host(s) and external hosts are expeted to have connection to the guest at all times, no matter the current Xen host.
I experimented a bit with tcpdump, and the gARP does appear on the netfront interface. It also appears on the host bridge if sufficient time passes between completing the xenbus handshake and sending the gARP.
When the guest queues eg. three gARPs in rapid succession, a variable number of them gets lost. (When all such packets disappear, then the migrated guest becomes invisible to the outside world, until it initiates network traffic on its own.)
When the guest waits for about half a second before sending (queueing), the very first gARP packet successfully appears on the host bridge.
I suspect it's a timing race against the netback vif being added to the host bridge. What would be a good countermeasure?
- Adding two modparams to xen-netfront (gARP requeue count & number of msecs to wait between queueing the gARPs).
- (Paolo's idea:) watching the "hotplug-status" xenstore node and sending a single gARP when the watch fires with "connected". This node belongs to the backend xenstore subtree, thus watching it from the guest doesn't please the architecture astronaut in me.
- Something else.
Sorry for the naivety / verbiage.
Thanks,
lacos
Quoted 4 lines · Laszlo Ersek
- (Paolo's idea:) watching the "hotplug-status" xenstore node and sending a single gARP when the watch fires with "connected". This node belongs to the backend xenstore subtree, thus watching it from the guest doesn't please the architecture astronaut in me.
Note that watching the backend and reading its information is quite common. In fact, that's how the state of the backend is observed in the first place. Of course you cannot write to the backend tree, but you do not have to do that.
Paolo
Quoted 25 lines · Laszlo Ersek
Hi,
with reference to RHBZ#713585:
It seems when a RHEL-6.1 or F-15 Xen PV guest is live migrated, the gratuitous ARP packet is not forwarded to the affected "networking equipment". The netback vif is added to a routed bridge in the host(s) and external hosts are expeted to have connection to the guest at all times, no matter the current Xen host.
I experimented a bit with tcpdump, and the gARP does appear on the netfront interface. It also appears on the host bridge if sufficient time passes between completing the xenbus handshake and sending the gARP.
When the guest queues eg. three gARPs in rapid succession, a variable number of them gets lost. (When all such packets disappear, then the migrated guest becomes invisible to the outside world, until it initiates network traffic on its own.)
When the guest waits for about half a second before sending (queueing), the very first gARP packet successfully appears on the host bridge.
I suspect it's a timing race against the netback vif being added to the host bridge. What would be a good countermeasure?
Quoted 6 lines
- Adding two modparams to xen-netfront (gARP requeue count & number of msecs to wait between queueing the gARPs).
- (Paolo's idea:) watching the "hotplug-status" xenstore node and sending a single gARP when the watch fires with "connected". This node belongs to the backend xenstore subtree, thus watching it from the guest doesn't please the architecture astronaut in me.
netback already waits (or should...) for hotplug-status to fire with "connected" before moving to state XenbusStateConnected. See hotplug_status_changed in drivers/net/xen-netback/xenbus.c. You need either the netback in upstream or something newer than 43223efd9bfd (C Feb 2010) if you are using e.g. xen.git#xen/next-2.6.32. That commit fixes pretty much the issue you describe.
I expected that netfront waited for the backend to hit XenbusStateConnected before sending the grat ARP but instead I find it happens when the backend hits XenbusStateInitWait. I'm not sure if that is a problem -- it appears to have been done this way since forever (even back in the classic Xen kernels) and I've never noticed a gARP go missing in the way you describe, but perhaps something isn't quite matching up any more.
Ian.
Quoted 11 lines
- Something else.
Sorry for the naivety / verbiage.
Thanks,
lacos_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
On Tue, 2011-06-28 at 15:01 +0200, Laszlo Ersek wrote:
[...]
Quoted 8 lines
When the guest waits for about half a second before sending (queueing), the very first gARP packet successfully appears on the host bridge.
I suspect it's a timing race against the netback vif being added to the host bridge. What would be a good countermeasure?
- Adding two modparams to xen-netfront (gARP requeue count & number of msecs to wait between queueing the gARPs).
Note that peer notifications are indirected through netdev notifiers and now include IPv6 NAs as well as ARPs. If repeated notifications are commonly necessary then this should probably be handled in the protocol (or in the networking core). However this sounds like a workaround whereas your other option would be a proper fix:
Quoted 4 lines
- (Paolo's idea:) watching the "hotplug-status" xenstore node and sending a single gARP when the watch fires with "connected". This node belongs to the backend xenstore subtree, thus watching it from the guest doesn't please the architecture astronaut in me.
[...]
Signature · 3 lines
Ben Hutchings, Senior Software Engineer, Solarflare Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked.
PATCH 1/1 xen-netfront: delay gARP until backend switches to Connected
In reply to Ian CampbellQuoted 7 lines · Ian Campbell
netback already waits (or should...) for hotplug-status to fire with "connected" before moving to state XenbusStateConnected. See
hotplug_status_changedindrivers/net/xen-netback/xenbus.c. You need either the netback in upstream or something newer than43223efd9bfd(C Feb 2010) if you are using e.g. xen.git#xen/next-2.6.32. That commit fixes pretty much the issue you describe.
Quoted 7 lines
I expected that netfront waited for the backend to hit XenbusStateConnected before sending the grat ARP but instead I find it happens when the backend hits XenbusStateInitWait. I'm not sure if that is a problem -- it appears to have been done this way since forever (even back in the classic Xen kernels) and I've never noticed a gARP go missing in the way you describe, but perhaps something isn't quite matching up any more.
In addition to backporting 43223efd9bfd to the RHEL-5 host side, we needed the following in the RHEL-6 guest, in order to fix the network outage after live migration. The patch is for xen/next-2.6.32. I also tested a Fedora-15 guest (without the patch), and the backport to the host side alone was not sufficient there either.
Thanks for the help and for considering this one.
--- drivers/net/xen-netfront.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-)
drivers/net/xen-netfront.c +3 -1
@@ -1732,7 +1732,6 @@ static void netback_changed(struct xenbus_device *dev,case XenbusStateInitialised:case XenbusStateReconfiguring:case XenbusStateReconfigured: -case XenbusStateConnected:case XenbusStateUnknown:case XenbusStateClosed:break ;
@@ -1743,6 +1742,9 @@ static void netback_changed(struct xenbus_device *dev,if (xennet_connect (netdev) !=0 )break ;xenbus_switch_state (dev, XenbusStateConnected); +break ; + +case XenbusStateConnected:netif_notify_peers (netdev);break ;
Signature · 1 line
1.7.4.4
Re: [PATCH] xen-netfront: delay gARP until backend switches to Connected
Quoted 3 lines
migration. The patch is for xen/next-2.6.32. I also tested a Fedora-15 guest (without the patch), and the backport to the host side alone was not sufficient there either.
I am having a hard time parsing that. It sounds like this patch is also neccessary for upstream kernel? Or do you have another one for the upstream kernel that you are going to submit soon?
(If so, please CC David Miller on it).
Quoted 38 lines
Thanks for the help and for considering this one.
Signed-off-by Laszlo Ersek <[email protected]>--- drivers/net/xen-netfront.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-)drivers/net/xen-netfront.c +3
-1@@ -1732,7 +1732,6 @@ static void netback_changed(struct xenbus_device *dev,case XenbusStateInitialised:case XenbusStateReconfiguring:case XenbusStateReconfigured: -case XenbusStateConnected:case XenbusStateUnknown:case XenbusStateClosed:break ;@@ -1743,6 +1742,9 @@ static void netback_changed(struct xenbus_device *dev,if (xennet_connect (netdev) !=0 )break ;xenbus_switch_state (dev, XenbusStateConnected); +break ; + +case XenbusStateConnected:netif_notify_peers (netdev);break ;Signature · 7 lines
1.7.4.4 _______________________________________________ Xen-devel mailing list [email protected] http://lists.xensource.com/xen-devel
PATCH v2 1/1 xen-netfront: delay gARP until backend switches to Connected
In addition to backporting 43223efd9bfd to the RHEL-5 host side, we needed the following in the RHEL-6 guest, in order to fix the network outage after live migration. I also tested a Fedora-15 guest (without the patch), and the backport to the host side alone was not sufficient there either. Therefore I think it could be useful to apply this guest-side patch to upstream as well.
Reference: https://bugzilla.redhat.com/show_bug.cgi?id=713585
v1->v2: "port" to Linus's tree; add RHBZ ref.
--- drivers/net/xen-netfront.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-)
drivers/net/xen-netfront.c +3 -1
@@ -1646,7 +1646,6 @@ static void netback_changed(struct xenbus_device *dev,case XenbusStateInitialised:case XenbusStateReconfiguring:case XenbusStateReconfigured: -case XenbusStateConnected:case XenbusStateUnknown:case XenbusStateClosed:break ;
@@ -1657,6 +1656,9 @@ static void netback_changed(struct xenbus_device *dev,if (xennet_connect (netdev) !=0 )break ;xenbus_switch_state (dev, XenbusStateConnected); +break ; + +case XenbusStateConnected:netif_notify_peers (netdev);break ;
Signature · 1 line
1.7.4.4
Re: [PATCH v2] xen-netfront: delay gARP until backend switches to Connected
Quoted 3 lines · Laszlo Ersek
In addition to backporting
43223efd9bfdto the RHEL-5 host side, we needed the following in the RHEL-6 guest, in order to fix the network outage after live migration. I also tested a Fedora-15 guest (without the patch), and the
Laszlo,
This description is .. well, pointless for upstream patches. Just succinctly describe the problem, how to reproduce it, and what this patch does.
And do mention more about 43223efd9bfd, like "this is the front counterpart of patch xen: netback: wait for hotplug scripts to complete before signalling connected to frontend (43223efd9bfd)"
Quoted 42 lines
backport to the host side alone was not sufficient there either. Therefore I think it could be useful to apply this guest-side patch to upstream as well.
Reference: https://bugzilla.redhat.com/show_bug.cgi?id=713585
v1->v2: "port" to Linus's tree; add RHBZ ref.Signed-off-by Laszlo Ersek <[email protected]>--- drivers/net/xen-netfront.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-)drivers/net/xen-netfront.c +3
-1@@ -1646,7 +1646,6 @@ static void netback_changed(struct xenbus_device *dev,case XenbusStateInitialised:case XenbusStateReconfiguring:case XenbusStateReconfigured: -case XenbusStateConnected:case XenbusStateUnknown:case XenbusStateClosed:break ;@@ -1657,6 +1656,9 @@ static void netback_changed(struct xenbus_device *dev,if (xennet_connect (netdev) !=0 )break ;xenbus_switch_state (dev, XenbusStateConnected); +break ; + +case XenbusStateConnected:netif_notify_peers (netdev);break ;Signature · 7 lines
1.7.4.4 _______________________________________________ Xen-devel mailing list [email protected] http://lists.xensource.com/xen-devel
PATCH v3 1/1 xen-netfront: delay gARP until backend switches to Connected
Quoted 9 lines · Konrad Rzeszutek Wilk
Quoted 3 lines · Laszlo Ersek
In addition to backporting
43223efd9bfdto the RHEL-5 host side, we needed the following in the RHEL-6 guest, in order to fix the network outage after live migration. I also tested a Fedora-15 guest (without the patch), and theLaszlo,
This description is .. well, pointless for upstream patches. Just succinctly describe the problem, how to reproduce it, and what this patch does.
I avoided writing up the commit message myself because in my first posting [1] I quoted two paragraphs from Ian Campbell [2]. I intended those two cited paragraphs verbatim as the commit message (along with my short remarks), because they describe the problem exactly. Anyway:
After a guest is live migrated, the xen-netfront driver emits a gratuitous ARP message, so that networking hardware on the target host's subnet can take notice, and public routing to the guest is re-established. However, if the packet appears on the backend interface before the backend is added to the target host's bridge, the packet is lost, and the migrated guest's peers become unable to talk to the guest.
A sufficient two-parts condition to prevent the above is:
(1) ensure that the backend only moves to Connected xenbus state after its hotplug scripts completed, ie. the netback interface got added to the bridge; and
(2) ensure the frontend only queues the gARP when it sees the backend move to Connected.
These two together provide complete ordering. Sub-condition (1) is satisfied by pvops commit 43223efd9bfd.
In general, the full condition is sufficient, not necessary, because, according to [2], live migration has been working for a long time without satisfying sub-condition (2). However, after 43223efd9bfd was backported to the RHEL-5 host to ensure (1), (2) still proved necessary in the RHEL-6 guest. This patch intends to provide (2) for upstream.
[1] http://lists.xensource.com/archives/html/xen-devel/2011-07/msg00327.html
[2] http://lists.xensource.com/archives/html/xen-devel/2011-06/msg01969.html
--- drivers/net/xen-netfront.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-)
drivers/net/xen-netfront.c +3 -1
@@ -1646,7 +1646,6 @@ static void netback_changed(struct xenbus_device *dev,case XenbusStateInitialised:case XenbusStateReconfiguring:case XenbusStateReconfigured: -case XenbusStateConnected:case XenbusStateUnknown:case XenbusStateClosed:break ;
@@ -1657,6 +1656,9 @@ static void netback_changed(struct xenbus_device *dev,if (xennet_connect (netdev) !=0 )break ;xenbus_switch_state (dev, XenbusStateConnected); +break ; + +case XenbusStateConnected:netif_notify_peers (netdev);break ;
Signature · 1 line
1.7.4.4
Re: [PATCH v3] xen-netfront: delay gARP until backend switches to Connected
Quoted 8 lines · Laszlo Ersek
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 3 lines · Laszlo Ersek
In addition to backporting
43223efd9bfdto the RHEL-5 host side, we needed the following in the RHEL-6 guest, in order to fix the network outage after live migration. I also tested a Fedora-15 guest (without the patch), and theLaszlo,
Quoted 7 lines
Quoted 2 lines
This description is .. well, pointless for upstream patches. Just succinctly describe the problem, how to reproduce it, and what this patch does.
I avoided writing up the commit message myself because in my first posting [1] I quoted two paragraphs from Ian Campbell [2]. I intended those two cited paragraphs verbatim as the commit message (along with my short remarks), because they describe the problem exactly.
Please don't expect/require that maintainers trawl around and construct a commit message for you, always propose the full text you would like to see committed in each patch posting.
Quoted 18 lines
Anyway:
After a guest is live migrated, the xen-netfront driver emits a gratuitous ARP message, so that networking hardware on the target host's subnet can take notice, and public routing to the guest is re-established. However, if the packet appears on the backend interface before the backend is added to the target host's bridge, the packet is lost, and the migrated guest's peers become unable to talk to the guest.
A sufficient two-parts condition to prevent the above is:
(1) ensure that the backend only moves to Connected xenbus state after its hotplug scripts completed, ie. the netback interface got added to the bridge; and
(2) ensure the frontend only queues the gARP when it sees the backend move to Connected.
Quoted 8 lines
These two together provide complete ordering. Sub-condition (1) is satisfied by pvops commit
43223efd9bfd.In general, the full condition is sufficient, not necessary, because, according to [2], live migration has been working for a long time without satisfying sub-condition (2). However, after
43223efd9bfdwas backported to the RHEL-5 host to ensure (1), (2) still proved necessary in the RHEL-6 guest. This patch intends to provide (2) for upstream.
I expect that 43223efd9bfd just reduces the window (enough to prevent it occuring most of the time) but doesn't actually close it. This change seems good to me.
Quoted 4 lines
[1] http://lists.xensource.com/archives/html/xen-devel/2011-07/msg00327.html
[2] http://lists.xensource.com/archives/html/xen-devel/2011-06/msg01969.htmlSigned-off-by Laszlo Ersek <[email protected]>
Quoted 26 lines
--- drivers/net/xen-netfront.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-)drivers/net/xen-netfront.c +3
-1@@ -1646,7 +1646,6 @@ static void netback_changed(struct xenbus_device *dev,case XenbusStateInitialised:case XenbusStateReconfiguring:case XenbusStateReconfigured: -case XenbusStateConnected:case XenbusStateUnknown:case XenbusStateClosed:break ;@@ -1657,6 +1656,9 @@ static void netback_changed(struct xenbus_device *dev,if (xennet_connect (netdev) !=0 )break ;xenbus_switch_state (dev, XenbusStateConnected); +break ; + +case XenbusStateConnected:netif_notify_peers (netdev);break ;