Skip to contentSkip to conversation

PCI: hv: add hard timeout to wait_for_response()

PATCH In review Hamza Mahfooz 5 participants 6 messages Jun 12 – Jun 17

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 PCI: hv: add hard timeout to wait_for_response()

It is possible that we never receive a rescind event, in which case we will wait indefinitely for a device that will never show up. So, assume a device is gone if have been polling for more than 5 seconds.

Fixes c3635da2a336 ("PCI: hv: Do not wait forever on a device that has disappeared")
---
 drivers/pci/controller/pci-hyperv.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
drivers/pci/controller/pci-hyperv.c +9 -1
index cfc8fa403dad..bd63efc4a210 100644
@@ -52,6 +52,7 @@
 #include <linux/acpi.h>
 #include <linux/sizes.h>
 #include <linux/of_irq.h>
+#include <linux/jiffies.h>
 #include <asm/mshyperv.h>
 
 /*
@@ -1038,6 +1039,8 @@ static void put_pcichild(struct hv_pci_dev *hpdev)
 		kfree(hpdev);
 }
 
+#define TIMEOUT_MS 5000
+
 /*
  * There is no good way to get notified from vmbus_onoffer_rescind(),
  * so let's use polling here, since this is not a hot path.
@@ -1045,8 +1048,13 @@ static void put_pcichild(struct hv_pci_dev *hpdev)
 static int wait_for_response(struct hv_device *hdev,
 			     struct completion *comp)
 {
+	unsigned long timeout = get_jiffies_64() + msecs_to_jiffies(TIMEOUT_MS);
+	unsigned long now;
+
 	while (true) {
-		if (hdev->channel->rescind) {
+		now = get_jiffies_64();
+		if (hdev->channel->rescind ||
+		    time_after(now, timeout)) {
 			dev_warn_once(&hdev->device, "The device is gone.\n");
 			return -ENODEV;
 		}
Signature · 1 line
2.54.0
#1 lore raw
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Critical] Use-After-Free and Stack Corruption due to abandoned VMBus requests on timeout. - [Medium]…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] Use-After-Free and Stack Corruption due to abandoned VMBus requests on timeout.
- [Medium] Race condition in wait_for_response() polling loop can falsely trigger timeout. --

commit 18b7ab3e09172b3f0651ad176d0ab6367f1e43bb
Author: Hamza Mahfooz <[email protected]>

PCI: hv: add hard timeout to wait_for_response()

This commit adds a hard 5-second timeout to wait_for_response() in the hyper-V PCI controller driver. It addresses an issue where the driver could wait indefinitely for a device to appear if a rescind event is never received.

Quoted 4 lines
drivers/pci/controller/pci-hyperv.c +0 -0
index cfc8fa403dad6..bd63efc4a2103 100644

[ ... ]

drivers/pci/controller/pci-hyperv.c static void put_pcichild(struct hv_pci_dev *hpdev) · line 1057

Quoted 12 lines
drivers/pci/controller/pci-hyperv.c +6 -1
@@ -1045,8 +1048,13 @@ static void put_pcichild(struct hv_pci_dev *hpdev)
 static int wait_for_response(struct hv_device *hdev,
 			     struct completion *comp)
 {
+	unsigned long timeout = get_jiffies_64() + msecs_to_jiffies(TIMEOUT_MS);
+	unsigned long now;
+
 	while (true) {
-		if (hdev->channel->rescind) {
+		now = get_jiffies_64();
+		if (hdev->channel->rescind ||
+		    time_after(now, timeout)) {

[Severity: Medium]
Is there a race condition if the thread is preempted right before updating the time?

If the thread is preempted just before calling get_jiffies_64() and the delay crosses the 5-second boundary, this will evaluate to true and immediately return -ENODEV.

Since there is no final check of the completion status after the timeout check, could this falsely assume a timeout even if the Hyper-V host successfully replied during the preemption window?

drivers/pci/controller/pci-hyperv.c static void put_pcichild(struct hv_pci_dev *hpdev) · line 1059

Quoted 2 lines
drivers/pci/controller/pci-hyperv.c +0 -0
 			dev_warn_once(&hdev->device, "The device is gone.\n");
 			return -ENODEV;

[Severity: Critical]
By adding this timeout, do we risk a use-after-free and stack corruption if the VMBus request is never explicitly cancelled?

When we return -ENODEV early here, callers like hv_send_resources_allocated() will free their heap-allocated request packets and return, which pops the local completion structure off the stack.

However, since the VMBus channel remains active and the pending request is not removed from the requestor array, what happens if the Hyper-V host eventually replies?

Wouldn't hv_pci_onchannelcallback() fetch the dangling packet pointer and invoke comp_packet->completion_func(...) on freed memory?

It seems this could also corrupt the stack when writing to the destroyed completion structure. Additionally, in hv_pci_query_relations(), returning on timeout leaves hbus->survey_event pointing to a local stack variable that will later be completed by survey_child_resources().

Does this also leak a slot in the VMBus requestor array for each timed-out request?

Quoted 3 lines
drivers/pci/controller/pci-hyperv.c +0 -0
 		}

if (wait_for_completion_timeout(comp, HZ / 10))

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#2 lore raw
In reply to Hamza Mahfooz
Quoted 5 lines

Subject: [PATCH] PCI: hv: add hard timeout to wait_for_response()

It is possible that we never receive a rescind event, in which case we will wait indefinitely for a device that will never show up. So, assume a device is gone if have been polling for more than 5 seconds.

Can you explain in what situation we never receive a rescind event? If this for dealing a device unload when the vmbus is dead? Please provide more context. A kernel trace is helpful.

Does this patch handle the case where the rescind event comes in right after the timeout?

Quoted 34 lines
Fixes c3635da2a336 ("PCI: hv: Do not wait forever on a device that has

disappeared")

---
 drivers/pci/controller/pci-hyperv.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
drivers/pci/controller/pci- renamed from drivers/pci/controller/pci-hyperv.c +0 -0

hyperv.c
index cfc8fa403dad..bd63efc4a210 100644

drivers/pci/controller/pci-hyperv.c +3 -0
@@ -52,6 +52,7 @@
 #include <linux/acpi.h>
 #include <linux/sizes.h>
 #include <linux/of_irq.h>
+#include <linux/jiffies.h>
 #include <asm/mshyperv.h>
 
 /*
@@ -1038,6 +1039,8 @@ static void put_pcichild(struct hv_pci_dev *hpdev)
 		kfree(hpdev);
 }
 
+#define TIMEOUT_MS 5000
+
 /*
  * There is no good way to get notified from vmbus_onoffer_rescind(),
  * so let's use polling here, since this is not a hot path.
@@ -1045,8 +1048,13 @@ static void put_pcichild(struct hv_pci_dev *hpdev)
 static int wait_for_response(struct hv_device *hdev,
 			     struct completion *comp)

drivers/pci/controller/pci-hyperv.c static void put_pcichild(struct hv_pci_dev *hpdev) · line 1058

Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +6 -1
 {
+	unsigned long timeout = get_jiffies_64() +
 msecs_to_jiffies(TIMEOUT_MS);
+	unsigned long now;
+
 	while (true) {
-		if (hdev->channel->rescind) {
+		now = get_jiffies_64();
+		if (hdev->channel->rescind ||
+		    time_after(now, timeout)) {

What if the VMBUS response comes in right after this check? The completion is allocated on the caller stack, and it will cause kernel OOP.

How do you test this patch?

#3 lore raw
Quoted 1 line · Long Li

What if the VMBUS response comes in right after this check? The completion is allocated on the caller stack, and it will cause kernel OOP.

That is a fair point, I'll try to see if there is a better way to handle this error case.

#4 lore raw
In reply to Hamza Mahfooz
Quoted 4 lines · Hamza Mahfooz

It is possible that we never receive a rescind event, in which case we will wait indefinitely for a device that will never show up. So, assume a device is gone if have been polling for more than 5 seconds.

Echo Long's request for more context on where this was seen.

Quoted 20 lines
Fixes c3635da2a336 ("PCI: hv: Do not wait forever on a device that has disappeared")
---
 drivers/pci/controller/pci-hyperv.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
drivers/pci/controller/pci-hyperv.c +1 -0
index cfc8fa403dad..bd63efc4a210 100644
@@ -52,6 +52,7 @@
 #include <linux/acpi.h>
 #include <linux/sizes.h>
 #include <linux/of_irq.h>
+#include <linux/jiffies.h>
 #include <asm/mshyperv.h>
 
 /*

drivers/pci/controller/pci-hyperv.c static void put_pcichild(struct hv_pci_dev *hpdev) · line 1043

Quoted 5 lines
drivers/pci/controller/pci-hyperv.c +2 -0
@@ -1038,6 +1039,8 @@ static void put_pcichild(struct hv_pci_dev *hpdev)
 		kfree(hpdev);
 }
 
+#define TIMEOUT_MS 5000
+

This can be in seconds, see below.

drivers/pci/controller/pci-hyperv.c static void put_pcichild(struct hv_pci_dev *hpdev) · line 1051

Quoted 8 lines
drivers/pci/controller/pci-hyperv.c +1 -0
 /*
  * There is no good way to get notified from vmbus_onoffer_rescind(),
  * so let's use polling here, since this is not a hot path.
@@ -1045,8 +1048,13 @@ static void put_pcichild(struct hv_pci_dev *hpdev)
 static int wait_for_response(struct hv_device *hdev,
 			     struct completion *comp)
 {
+	unsigned long timeout = get_jiffies_64() + msecs_to_jiffies(TIMEOUT_MS);

You can use secs_to_jiffies() here instead.

Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +5 -1
+	unsigned long now;
+
 	while (true) {
-		if (hdev->channel->rescind) {
+		now = get_jiffies_64();
+		if (hdev->channel->rescind ||
+		    time_after(now, timeout)) {
 			dev_warn_once(&hdev->device, "The device is gone.\n");
 			return -ENODEV;
 		}
#5 lore raw
In reply to Hamza Mahfooz

From: Hamza Mahfooz <[email protected]> Sent: Friday, June 12, 2026 10:40 AM

Quoted 1 line

The addressees for this patch include the linux-hyperv and linux-pci mailing lists, but not the broad linux-kernel mailing list. Any reason not to include the latter? In Documentation/process/submitting-patches.rst, in the section "Select the recipients for your patch", it says

[email protected] should be used by default for all patches

I subscribe to the broad linux-kernel list and do my own filtering, instead of subscribing to individual subsystem lists. So I didn't see this patch until I glanced at https://lore.kernel.org/linux-hyperv/. Following the guidance to always include linux-kernel would help me, for one. :-)

Thx,

Michael

Quoted 50 lines

It is possible that we never receive a rescind event, in which case we will wait indefinitely for a device that will never show up. So, assume a device is gone if have been polling for more than 5 seconds.

Fixes c3635da2a336 ("PCI: hv: Do not wait forever on a device that has disappeared")
---
 drivers/pci/controller/pci-hyperv.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
drivers/pci/controller/pci-hyperv.c +9 -2
index cfc8fa403dad..bd63efc4a210 100644
@@ -52,6 +52,7 @@
 #include <linux/acpi.h>
 #include <linux/sizes.h>
 #include <linux/of_irq.h>
+#include <linux/jiffies.h>
 #include <asm/mshyperv.h>
 
 /*
@@ -1038,6 +1039,8 @@ static void put_pcichild(struct hv_pci_dev *hpdev)
 		kfree(hpdev);
 }
 
+#define TIMEOUT_MS 5000
+
 /*
  * There is no good way to get notified from vmbus_onoffer_rescind(),
  * so let's use polling here, since this is not a hot path.
@@ -1045,8 +1048,13 @@ static void put_pcichild(struct hv_pci_dev *hpdev)
 static int wait_for_response(struct hv_device *hdev,
 			     struct completion *comp)
 {
+	unsigned long timeout = get_jiffies_64() + msecs_to_jiffies(TIMEOUT_MS);
+	unsigned long now;
+
 	while (true) {
-		if (hdev->channel->rescind) {
+		now = get_jiffies_64();
+		if (hdev->channel->rescind ||
+		    time_after(now, timeout)) {
 			dev_warn_once(&hdev->device, "The device is gone.\n");
 			return -ENODEV;
 		}
--

2.54.0