Skip to contentSkip to conversation

xen: fix frontend driver disconnected from xenbus on removal

PATCH New Oleksandr Andrushchenko 2 participants 11 messages Feb 1, 2018 – Feb 2, 2018 Xen hypervisor interface

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 xen: fix frontend driver disconnected from xenbus on removal

From: Oleksandr Andrushchenko <[email protected]>

Hi, all!

While working on DRM PV frontend driver I faced an issue with driver removal, e.g. when driver's .remove callback is called the driver is already disconnected form the xenbus and it is not possible to synchronize the process of removal with the backend. Backend in my case is a user-space application, so this may explain why backends which are kernel modules do not suffer from this:
it seems that user-space is clumsy and its states are not delivered through xenbus fast enough.

The obvious way to fix this behavior is to disconnect frontend drivers from xenbus _after_ their .remove callback is called.

Thank you,
Oleksandr Andrushchenko

Oleksandr Andrushchenko (1):
xen: fix frontend driver disconnected from xenbus on removal

 drivers/xen/xenbus/xenbus_probe.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Signature · 1 line
2.7.4
#1 lore raw

PATCH 1/1 xen: fix frontend driver disconnected from xenbus on removal

From: Oleksandr Andrushchenko <[email protected]>

Current xenbus frontend driver removal flow first disconnects the driver from xenbus and then calls driver's remove callback. This makes it impossible for the driver to listen to backend's state changes and synchronize the removal procedure.

Fix this by removing other end XenBus watches after the driver's remove callback is called.

---
 drivers/xen/xenbus/xenbus_probe.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/xen/xenbus/xenbus_probe.c +2 -2
index 74888cacd0b0..9c63cd3f416b 100644
@@ -258,11 +258,11 @@ int xenbus_dev_remove(struct device *_dev)
 
 	DPRINTK("%s", dev->nodename);
 
-	free_otherend_watch(dev);
-
 	if (drv->remove)
 		drv->remove(dev);
 
+	free_otherend_watch(dev);
+
 	free_otherend_details(dev);
 
 	xenbus_switch_state(dev, XenbusStateClosed);
Signature · 7 lines
2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#2 lore raw

PATCH 1/1 xen: fix frontend driver disconnected from xenbus on removal

In reply to Oleksandr Andrushchenko

From: Oleksandr Andrushchenko <[email protected]>

Current xenbus frontend driver removal flow first disconnects the driver from xenbus and then calls driver's remove callback. This makes it impossible for the driver to listen to backend's state changes and synchronize the removal procedure.

Fix this by removing other end XenBus watches after the driver's remove callback is called.

---
 drivers/xen/xenbus/xenbus_probe.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/xen/xenbus/xenbus_probe.c +2 -2
index 74888cacd0b0..9c63cd3f416b 100644
@@ -258,11 +258,11 @@ int xenbus_dev_remove(struct device *_dev)
 
 	DPRINTK("%s", dev->nodename);
 
-	free_otherend_watch(dev);
-
 	if (drv->remove)
 		drv->remove(dev);
 
+	free_otherend_watch(dev);
+
 	free_otherend_details(dev);
 
 	xenbus_switch_state(dev, XenbusStateClosed);
Signature · 1 line
2.7.4
#3 lore raw
Quoted 20 lines · Oleksandr Andrushchenko

From: Oleksandr Andrushchenko <[email protected]>

Current xenbus frontend driver removal flow first disconnects the driver from xenbus and then calls driver's remove callback. This makes it impossible for the driver to listen to backend's state changes and synchronize the removal procedure.

Fix this by removing other end XenBus watches after the driver's remove callback is called.

---
 drivers/xen/xenbus/xenbus_probe.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/xen/xenbus/xenbus_probe.c +0 -0
index 74888cacd0b0..9c63cd3f416b 100644

drivers/xen/xenbus/xenbus_probe.c int xenbus_dev_remove(struct device *_dev) · line 262

Quoted 7 lines
drivers/xen/xenbus/xenbus_probe.c +0 -2
@@ -258,11 +258,11 @@ int xenbus_dev_remove(struct device *_dev)
 
 	DPRINTK("%s", dev->nodename);
 
-	free_otherend_watch(dev);
-
 	if (drv->remove)
 		drv->remove(dev);

Is it possible for the watch to fire here?

-boris

drivers/xen/xenbus/xenbus_probe.c int xenbus_dev_remove(struct device *_dev) · line 268

Quoted 6 lines
drivers/xen/xenbus/xenbus_probe.c +2 -0
+	free_otherend_watch(dev);
+
 	free_otherend_details(dev);
 
 	xenbus_switch_state(dev, XenbusStateClosed);

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#4 lore raw
In reply to Oleksandr Andrushchenko
Quoted 20 lines · Oleksandr Andrushchenko

From: Oleksandr Andrushchenko <[email protected]>

Current xenbus frontend driver removal flow first disconnects the driver from xenbus and then calls driver's remove callback. This makes it impossible for the driver to listen to backend's state changes and synchronize the removal procedure.

Fix this by removing other end XenBus watches after the driver's remove callback is called.

---
 drivers/xen/xenbus/xenbus_probe.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/xen/xenbus/xenbus_probe.c +0 -0
index 74888cacd0b0..9c63cd3f416b 100644

drivers/xen/xenbus/xenbus_probe.c int xenbus_dev_remove(struct device *_dev) · line 262

Quoted 7 lines
drivers/xen/xenbus/xenbus_probe.c +0 -2
@@ -258,11 +258,11 @@ int xenbus_dev_remove(struct device *_dev)
 
 	DPRINTK("%s", dev->nodename);
 
-	free_otherend_watch(dev);
-
 	if (drv->remove)
 		drv->remove(dev);

Is it possible for the watch to fire here?

-boris

Quoted 6 lines
drivers/xen/xenbus/xenbus_probe.c +2 -0
+	free_otherend_watch(dev);
+
 	free_otherend_details(dev);
 
 	xenbus_switch_state(dev, XenbusStateClosed);
#5 lore raw
Quoted 22 lines · Boris Ostrovsky
Quoted 21 lines · Oleksandr Andrushchenko

From: Oleksandr Andrushchenko <[email protected]>

Current xenbus frontend driver removal flow first disconnects the driver from xenbus and then calls driver's remove callback. This makes it impossible for the driver to listen to backend's state changes and synchronize the removal procedure.

Fix this by removing other end XenBus watches after the driver's remove callback is called.

---
  drivers/xen/xenbus/xenbus_probe.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
drivers/xen/xenbus/xenbus_probe.c +0 -0
index 74888cacd0b0..9c63cd3f416b 100644
@@ -258,11 +258,11 @@ int xenbus_dev_remove(struct device *_dev)
 
Quoted 8 lines

drivers/xen/xenbus/xenbus_probe.c int xenbus_dev_remove(struct device *_dev) · line 262

Quoted 6 lines
drivers/xen/xenbus/xenbus_probe.c +0 -2
  	DPRINTK("%s", dev->nodename);
 
-	free_otherend_watch(dev);
-
  	if (drv->remove)
  		drv->remove(dev);

Is it possible for the watch to fire here?

Indeed. Yes, It is possible, so we have to somehow protect the removed driver from being called, e.g. the driver cleans up in its .remove, but watch may still trigger .otherend_changed callback.
Is this what you mean?
If so, do you have something neat on your mind how to solve this?

Quoted 8 lines

-boris

drivers/xen/xenbus/xenbus_probe.c int xenbus_dev_remove(struct device *_dev) · line 268

Quoted 6 lines
drivers/xen/xenbus/xenbus_probe.c +2 -0
+	free_otherend_watch(dev);
+
  	free_otherend_details(dev);
 
  	xenbus_switch_state(dev, XenbusStateClosed);

Thank you,
Oleksandr

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#6 lore raw
In reply to Boris Ostrovsky
Quoted 22 lines · Boris Ostrovsky
Quoted 21 lines · Oleksandr Andrushchenko

From: Oleksandr Andrushchenko <[email protected]>

Current xenbus frontend driver removal flow first disconnects the driver from xenbus and then calls driver's remove callback. This makes it impossible for the driver to listen to backend's state changes and synchronize the removal procedure.

Fix this by removing other end XenBus watches after the driver's remove callback is called.

---
  drivers/xen/xenbus/xenbus_probe.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
drivers/xen/xenbus/xenbus_probe.c +0 -0
index 74888cacd0b0..9c63cd3f416b 100644
@@ -258,11 +258,11 @@ int xenbus_dev_remove(struct device *_dev)
 
Quoted 8 lines

drivers/xen/xenbus/xenbus_probe.c int xenbus_dev_remove(struct device *_dev) · line 262

Quoted 6 lines
drivers/xen/xenbus/xenbus_probe.c +0 -2
  	DPRINTK("%s", dev->nodename);
 
-	free_otherend_watch(dev);
-
  	if (drv->remove)
  		drv->remove(dev);

Is it possible for the watch to fire here?

Indeed. Yes, It is possible, so we have to somehow protect the removed driver from being called, e.g. the driver cleans up in its .remove, but watch may still trigger .otherend_changed callback.
Is this what you mean?
If so, do you have something neat on your mind how to solve this?

Quoted 8 lines

-boris

drivers/xen/xenbus/xenbus_probe.c int xenbus_dev_remove(struct device *_dev) · line 268

Quoted 6 lines
drivers/xen/xenbus/xenbus_probe.c +2 -0
+	free_otherend_watch(dev);
+
  	free_otherend_details(dev);
 
  	xenbus_switch_state(dev, XenbusStateClosed);

Thank you,
Oleksandr

#7 lore raw
Quoted 33 lines · Oleksandr Andrushchenko
Quoted 17 lines · Boris Ostrovsky
Quoted 16 lines · Oleksandr Andrushchenko

From: Oleksandr Andrushchenko <[email protected]>

Current xenbus frontend driver removal flow first disconnects the driver from xenbus and then calls driver's remove callback. This makes it impossible for the driver to listen to backend's state changes and synchronize the removal procedure.

Fix this by removing other end XenBus watches after the driver's remove callback is called.

Signed-off-by: Oleksandr Andrushchenko
<[email protected]>

---
  drivers/xen/xenbus/xenbus_probe.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
Quoted 13 lines

drivers/xen/xenbus/xenbus_probe.c int xenbus_dev_remove(struct device *_dev) · line 261

Quoted 11 lines
drivers/xen/xenbus/xenbus_probe.c +0 -1
index 74888cacd0b0..9c63cd3f416b 100644
@@ -258,11 +258,11 @@ int xenbus_dev_remove(struct device *_dev)
        DPRINTK("%s", dev->nodename);
  -    free_otherend_watch(dev);
-
      if (drv->remove)
          drv->remove(dev);

Is it possible for the watch to fire here?

Quoted 4 lines

Indeed. Yes, It is possible, so we have to somehow protect the removed driver from being called, e.g. the driver cleans up in its .remove, but watch may still trigger .otherend_changed callback.
Is this what you mean?

(-David who is not at Citrix anymore)

Exactly.

That's why otherend cleanup is split into free_otherend_watch() and free_otherend_details().

Quoted 1 line

If so, do you have something neat on your mind how to solve this?

Not necessarily "neat" but perhaps you can use xenbus_read_otherend_details() in both front and back ends. After all, IIUIC you are doing something synchronously so you don't really need a watch.

-boris

Quoted 9 lines
Quoted 7 lines

-boris

Quoted 4 lines
drivers/xen/xenbus/xenbus_probe.c +1 -0
  +    free_otherend_watch(dev);
+
      free_otherend_details(dev);
        xenbus_switch_state(dev, XenbusStateClosed);

Thank you,
Oleksandr

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#8 lore raw
In reply to Oleksandr Andrushchenko
Quoted 33 lines · Oleksandr Andrushchenko
Quoted 17 lines · Boris Ostrovsky
Quoted 16 lines · Oleksandr Andrushchenko

From: Oleksandr Andrushchenko <[email protected]>

Current xenbus frontend driver removal flow first disconnects the driver from xenbus and then calls driver's remove callback. This makes it impossible for the driver to listen to backend's state changes and synchronize the removal procedure.

Fix this by removing other end XenBus watches after the driver's remove callback is called.

Signed-off-by: Oleksandr Andrushchenko
<[email protected]>

---
  drivers/xen/xenbus/xenbus_probe.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
Quoted 13 lines

drivers/xen/xenbus/xenbus_probe.c int xenbus_dev_remove(struct device *_dev) · line 261

Quoted 11 lines
drivers/xen/xenbus/xenbus_probe.c +0 -1
index 74888cacd0b0..9c63cd3f416b 100644
@@ -258,11 +258,11 @@ int xenbus_dev_remove(struct device *_dev)
        DPRINTK("%s", dev->nodename);
  -    free_otherend_watch(dev);
-
      if (drv->remove)
          drv->remove(dev);

Is it possible for the watch to fire here?

Quoted 4 lines

Indeed. Yes, It is possible, so we have to somehow protect the removed driver from being called, e.g. the driver cleans up in its .remove, but watch may still trigger .otherend_changed callback.
Is this what you mean?

(-David who is not at Citrix anymore)

Exactly.

That's why otherend cleanup is split into free_otherend_watch() and free_otherend_details().

Quoted 1 line

If so, do you have something neat on your mind how to solve this?

Not necessarily "neat" but perhaps you can use xenbus_read_otherend_details() in both front and back ends. After all, IIUIC you are doing something synchronously so you don't really need a watch.

-boris

Quoted 9 lines
Quoted 7 lines

-boris

Quoted 4 lines
drivers/xen/xenbus/xenbus_probe.c +1 -0
  +    free_otherend_watch(dev);
+
      free_otherend_details(dev);
        xenbus_switch_state(dev, XenbusStateClosed);

Thank you,
Oleksandr

#9 lore raw
Quoted 36 lines · Boris Ostrovsky
Quoted 31 lines · Oleksandr Andrushchenko
Quoted 17 lines · Boris Ostrovsky
Quoted 16 lines · Oleksandr Andrushchenko

From: Oleksandr Andrushchenko <[email protected]>

Current xenbus frontend driver removal flow first disconnects the driver from xenbus and then calls driver's remove callback. This makes it impossible for the driver to listen to backend's state changes and synchronize the removal procedure.

Fix this by removing other end XenBus watches after the driver's remove callback is called.

Signed-off-by: Oleksandr Andrushchenko
<[email protected]>

---
   drivers/xen/xenbus/xenbus_probe.c | 4 ++--
   1 file changed, 2 insertions(+), 2 deletions(-)
Quoted 12 lines

drivers/xen/xenbus/xenbus_probe.c int xenbus_dev_remove(struct device *_dev) · line 261

Quoted 11 lines
drivers/xen/xenbus/xenbus_probe.c +0 -1
index 74888cacd0b0..9c63cd3f416b 100644
@@ -258,11 +258,11 @@ int xenbus_dev_remove(struct device *_dev)
         DPRINTK("%s", dev->nodename);
   -    free_otherend_watch(dev);
-
       if (drv->remove)
           drv->remove(dev);

Is it possible for the watch to fire here?

Quoted 4 lines

Indeed. Yes, It is possible, so we have to somehow protect the removed driver from being called, e.g. the driver cleans up in its .remove, but watch may still trigger .otherend_changed callback.
Is this what you mean?

Quoted 6 lines

(-David who is not at Citrix anymore)

Exactly.

That's why otherend cleanup is split into free_otherend_watch() and free_otherend_details().

Understood, thank you
Confusion came because of the patch [1]: in .remove we wait for the backend to change its states in .otherend_changed callback and wake us, but I am not sure how those state changes may occur if during .remove the driver has already watches freed. So, this is why I tried to play around with free_otherend_watch()...

Quoted 6 lines
Quoted 2 lines

If so, do you have something neat on your mind how to solve this?

Not necessarily "neat" but perhaps you can use xenbus_read_otherend_details() in both front and back ends. After all, IIUIC you are doing something synchronously so you don't really need a watch.

Yes, I will implement a dedicated flow in the .remove instead of relying on .otherend_changed

Quoted 10 lines

-boris

Quoted 8 lines
Quoted 6 lines

-boris

Quoted 4 lines
drivers/xen/xenbus/xenbus_probe.c +1 -0
   +    free_otherend_watch(dev);
+
       free_otherend_details(dev);
         xenbus_switch_state(dev, XenbusStateClosed);

Thank you,
Oleksandr

Thank you,
Oleksandr

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/net/xen-netfront.c?id=5b5971df3bc2775107ddad164018a8a8db633b81

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#10 lore raw
In reply to Boris Ostrovsky
Quoted 36 lines · Boris Ostrovsky
Quoted 31 lines · Oleksandr Andrushchenko
Quoted 17 lines · Boris Ostrovsky
Quoted 16 lines · Oleksandr Andrushchenko

From: Oleksandr Andrushchenko <[email protected]>

Current xenbus frontend driver removal flow first disconnects the driver from xenbus and then calls driver's remove callback. This makes it impossible for the driver to listen to backend's state changes and synchronize the removal procedure.

Fix this by removing other end XenBus watches after the driver's remove callback is called.

Signed-off-by: Oleksandr Andrushchenko
<[email protected]>

---
   drivers/xen/xenbus/xenbus_probe.c | 4 ++--
   1 file changed, 2 insertions(+), 2 deletions(-)
Quoted 12 lines

drivers/xen/xenbus/xenbus_probe.c int xenbus_dev_remove(struct device *_dev) · line 261

Quoted 11 lines
drivers/xen/xenbus/xenbus_probe.c +0 -1
index 74888cacd0b0..9c63cd3f416b 100644
@@ -258,11 +258,11 @@ int xenbus_dev_remove(struct device *_dev)
         DPRINTK("%s", dev->nodename);
   -    free_otherend_watch(dev);
-
       if (drv->remove)
           drv->remove(dev);

Is it possible for the watch to fire here?

Quoted 4 lines

Indeed. Yes, It is possible, so we have to somehow protect the removed driver from being called, e.g. the driver cleans up in its .remove, but watch may still trigger .otherend_changed callback.
Is this what you mean?

Quoted 6 lines

(-David who is not at Citrix anymore)

Exactly.

That's why otherend cleanup is split into free_otherend_watch() and free_otherend_details().

Understood, thank you
Confusion came because of the patch [1]: in .remove we wait for the backend to change its states in .otherend_changed callback and wake us, but I am not sure how those state changes may occur if during .remove the driver has already watches freed. So, this is why I tried to play around with free_otherend_watch()...

Quoted 6 lines
Quoted 2 lines

If so, do you have something neat on your mind how to solve this?

Not necessarily "neat" but perhaps you can use xenbus_read_otherend_details() in both front and back ends. After all, IIUIC you are doing something synchronously so you don't really need a watch.

Yes, I will implement a dedicated flow in the .remove instead of relying on .otherend_changed

Quoted 10 lines

-boris

Quoted 8 lines
Quoted 6 lines

-boris

Quoted 4 lines
drivers/xen/xenbus/xenbus_probe.c +1 -0
   +    free_otherend_watch(dev);
+
       free_otherend_details(dev);
         xenbus_switch_state(dev, XenbusStateClosed);

Thank you,
Oleksandr

Thank you,
Oleksandr

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/net/xen-netfront.c?id=5b5971df3bc2775107ddad164018a8a8db633b81