pci-hyperv: fix memory leak and add pci_destroy_slot()
Subsystems (5)
Dexuan Cui
maintainercover pci-hyperv: fix memory leak and add pci_destroy_slot()
Patch #1 fixes a memory leak caused by incorrectly-maintained hpdev->refs.
Patch #2 and #3 make sure the "slot" is removed in all the scenarios. Without them, in the quick hot-add/hot-remove test, systemd-dev may easily crash when trying to access a dangling sys file in /sys/bus/pci/slots/:
"BUG: unable to handle kernel paging request".
BTW, Patch #2 was posted on Feb 7, 2019, and this is the v2: the change to hv_eject_device_work() in v1 is removed, as the change is only needed when we hot-remove the device and remove the pci-hyperv driver at the same time. It looks more work is required to make this scenaro work correctly, and since removing the driver is not really a "usual" usage, we can address this scenario in the future.
Please review the patchset.
Dexuan Cui (3):
PCI: hv: Fix a memory leak in hv_eject_device_work()
PCI: hv: Add hv_pci_remove_slots() when we unload the driver
PCI: hv: Add pci_destroy_slot() in pci_devices_present_work(), if
necessary
drivers/pci/controller/pci-hyperv.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
Signature · 1 line
2.19.1
Dexuan Cui
maintainerPATCH 2/3 PCI: hv: Add hv_pci_remove_slots() when we unload the driver
When we unload pci-hyperv, the host doesn't send us a PCI_EJECT message. In this case we also need to make sure the sysfs pci slot directory is removed, otherwise "cat /sys/bus/pci/slots/2/address" will trigger "BUG: unable to handle kernel paging request" (I noticed the issue when systemd-dev crashed for me when I unloaded the driver). And, if we unload/reload the driver several times, we'll have multiple pci slot directories in /sys/bus/pci/slots/ like this:
root@localhost:~# ls -rtl /sys/bus/pci/slots/ total 0 drwxr-xr-x 2 root root 0 Feb 7 10:49 2 drwxr-xr-x 2 root root 0 Feb 7 10:49 2-1 drwxr-xr-x 2 root root 0 Feb 7 10:51 2-2
The patch adds the missing code.
--- drivers/pci/controller/pci-hyperv.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
drivers/pci/controller/pci-hyperv.c +16 -0
@@ -1486,6 +1486,21 @@ static void hv_pci_assign_slots(struct hv_pcibus_device *hbus) } } +/* +* Remove entries in sysfs pci slot directory. +*/ +static void hv_pci_remove_slots (struct hv_pcibus_device *hbus) +{ +struct hv_pci_dev *hpdev; + +list_for_each_entry (hpdev, &hbus->children, list_entry) { +if (!hpdev->pci_slot) +continue ; +pci_destroy_slot (hpdev->pci_slot); + hpdev->pci_slot =NULL ; + } +} +/** * create_root_hv_pci_bus() - Expose a new root PCI bus * @hbus: Root PCI bus, as understood by this driver
@@ -2680,6 +2695,7 @@ static int hv_pci_remove(struct hv_device *hdev)pci_lock_rescan_remove ();pci_stop_root_bus (hbus->pci_bus);pci_remove_root_bus (hbus->pci_bus); +hv_pci_remove_slots (hbus);pci_unlock_rescan_remove (); hbus->state = hv_pcibus_removed; }
Signature · 1 line
2.19.1
Dexuan Cui
maintainerPATCH 1/3 PCI: hv: Fix a memory leak in hv_eject_device_work()
In reply to Dexuan CuiAfter a device is just created in new_pcichild_device(), hpdev->refs is set to 2 (i.e. the initial value of 1 plus the get_pcichild()).
When we hot remove the device from the host, in Linux VM we first call hv_pci_eject_device(), which increases hpdev->refs by get_pcichild() and then schedules a work of hv_eject_device_work(), so hpdev->refs becomes 3 (let's ignore the paired get/put_pcichild() in other places). But in hv_eject_device_work(), currently we only call put_pcichild() twice, meaning the 'hpdev' struct can't be freed in put_pcichild(). This patch adds one put_pcichild() to fix the memory leak.
BTW, the device can also be removed when we run "rmmod pci-hyperv". On this path (hv_pci_remove() -> hv_pci_bus_exit() -> hv_pci_devices_present()), hpdev->refs is 2, and we do correctly call put_pcichild() twice in pci_devices_present_work().
--- drivers/pci/controller/pci-hyperv.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/pci/controller/pci-hyperv.c +3 -0
@@ -1900,6 +1900,9 @@ static void hv_eject_device_work(struct work_struct *work)sizeof (*ejct_pkt), (unsigned long )&ctxt.pkt, VM_PKT_DATA_INBAND, 0); +/* For the get_pcichild() in hv_pci_eject_device() */ +put_pcichild (hpdev); +/* For the two refs got in new_pcichild_device() */ put_pcichild (hpdev);put_pcichild (hpdev);put_hvpcibus (hpdev->hbus);
Signature · 1 line
2.19.1
Dexuan Cui
maintainerPATCH 3/3 PCI: hv: Add pci_destroy_slot() in pci_devices_present_work(), if necessary
In reply to Dexuan CuiWhen we hot-remove a device, usually the host sends us a PCI_EJECT message, and a PCI_BUS_RELATIONS message with bus_rel->device_count == 0. But when we do the quick hot-add/hot-remove test, the host may not send us the PCI_EJECT message, if the guest has not fully finished the initialization by sending the PCI_RESOURCES_ASSIGNED* message to the host, so it's potentially unsafe to only depend on the pci_destroy_slot() in hv_eject_device_work(), though create_root_hv_pci_bus() -> hv_pci_assign_slots() is not called in this case. Note: in this case, the host still sends the guest a PCI_BUS_RELATIONS message with bus_rel->device_count == 0.
And, in the quick hot-add/hot-remove test, we can have such a race: before pci_devices_present_work() -> new_pcichild_device() adds the new device into hbus->children, we may have already received the PCI_EJECT message, and hence the taklet handler hv_pci_onchannelcallback() may fail to find the "hpdev" by get_pcichild_wslot(hbus, dev_message->wslot.slot), so hv_pci_eject_device() is NOT called; later create_root_hv_pci_bus() -> hv_pci_assign_slots() creates the slot, and the PCI_BUS_RELATIONS message with bus_rel->device_count == 0 removes the device from hbus->children, and we end up being unable to remove the slot in hv_pci_remove() -> hv_pci_remove_slots().
The patch removes the slot in pci_devices_present_work() when the device is removed. This can address the above race. Note 1:pci_devices_present_work() and hv_eject_device_work() run in the singled-threaded hbus->wq, so there is not a double-remove issue for the slot. Note 2: we can't offload hv_pci_eject_device() from hv_pci_onchannelcallback() to the workqueue, because we need hv_pci_onchannelcallback() synchronously call hv_pci_eject_device() to poll the channel's ringbuffer to work around the
"hangs in hv_compose_msi_msg()" issue: see
commit de0aa7b2f97d ("PCI: hv: Fix 2 hang issues in hv_compose_msi_msg()")
--- drivers/pci/controller/pci-hyperv.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/pci/controller/pci-hyperv.c +4 -0
@@ -1776,6 +1776,10 @@ static void pci_devices_present_work(struct work_struct *work) hpdev = list_first_entry(&removed, struct hv_pci_dev, list_entry);list_del (&hpdev->list_entry); + +if (hpdev->pci_slot) +pci_destroy_slot (hpdev->pci_slot); +put_pcichild (hpdev); }
Signature · 1 line
2.19.1
Quoted 16 lines · On Mon, 4 Mar 2019 21:34:47 +0000 Dexuan Cui <[email protected]> wrote:
Patch #1 fixes a memory leak caused by incorrectly-maintained
hpdev->refs.Patch #2 and #3 make sure the "slot" is removed in all the scenarios. Without them, in the quick hot-add/hot-remove test, systemd-dev may easily crash when trying to access a dangling sys file in
/sys/bus/pci/slots/:
"BUG: unable to handle kernel paging request".BTW, Patch #2 was posted on Feb 7, 2019, and this is the v2: the change to
hv_eject_device_work()in v1 is removed, as the change is only needed when we hot-remove the device and remove the pci-hyperv driver at the same time. It looks more work is required to make this scenaro work correctly, and since removing the driver is not really a "usual" usage, we can address this scenario in the future.Please review the patchset.
Quoted 8 lines
Dexuan Cui (3):
PCI: hv: Fix a memory leak inhv_eject_device_work()
PCI: hv: Addhv_pci_remove_slots()when we unload the driver
PCI: hv: Addpci_destroy_slot()inpci_devices_present_work(), if
necessarydrivers/pci/controller/pci-hyperv.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
Thanks for fixing this.
RE: [PATCH 1/3] PCI: hv: Fix a memory leak in hv_eject_device_work()
In reply to Dexuan CuiFrom: Dexuan Cui <[email protected]>
Quoted 12 lines
After a device is just created in
new_pcichild_device(),hpdev->refsis set to 2 (i.e. the initial value of 1 plus theget_pcichild()).When we hot remove the device from the host, in Linux VM we first call
hv_pci_eject_device(), which increaseshpdev->refsbyget_pcichild()and then schedules a work ofhv_eject_device_work(), sohpdev->refsbecomes 3 (let's ignore the paired get/put_pcichild() in other places). But inhv_eject_device_work(), currently we only callput_pcichild()twice, meaning the 'hpdev' struct can't be freed input_pcichild(). This patch adds oneput_pcichild()to fix the memory leak.
Quoted 8 lines
BTW, the device can also be removed when we run "rmmod pci-hyperv". On this path (
hv_pci_remove()->hv_pci_bus_exit()->hv_pci_devices_present()),hpdev->refsis 2, and we do correctly callput_pcichild()twice inpci_devices_present_work().Fixes 4daace0d8ce8 ("PCI: hv: Add paravirtual PCI front-end for Microsoft Hyper-V VMs")Signed-off-by Dexuan Cui <[email protected]>Cc <[email protected]>
Exiting new_pcichild_device() with hpdev->refs set to 2 seems OK to me. There is the reference in the hbus->children list, and there is the reference that is returned to the caller. But what is strange is that pci_devices_present_work() overwrites the reference returned in local variable hpdev without doing a put_pcichild(). It seems like the "normal" reference count should be 1 when the child device is not being manipulated, not 2. The fix would be to add a call to put_pcichild() when the return value from new_pcichild_device() is overwritten. Then remove the call to put_pcichild() in pci_device_present_work() when missing children are moved to the local list. The children have been moved from one list to another, so there's no need to decrement the reference count. Then when everything in the local list is deleted, the reference is correctly decremented, presumably freeing the memory.
With this approach, the code in hv_eject_device_work() is correct. There's one call to put_pcichild() to reflect removing the child device from the hbus-> children list, and one call to put_pcichild() to pair with the get_pcichild() in hv_pci_eject_device().
Your patch works, but to me it leaves the ref count in an unnatural state most of the time.
Michael
RE: [PATCH 2/3] PCI: hv: Add hv_pci_remove_slots() when we unload the driver
In reply to Dexuan CuiFrom: Dexuan Cui <[email protected]> Sent: Monday, March 4, 2019 1:35 PM
Quoted 15 lines
When we unload pci-hyperv, the host doesn't send us a
PCI_EJECTmessage. In this case we also need to make sure the sysfs pci slot directory is removed, otherwise "cat/sys/bus/pci/slots/2/address" will trigger "BUG: unable to handle kernel paging request" (I noticed the issue when systemd-dev crashed for me when I unloaded the driver). And, if we unload/reload the driver several times, we'll have multiple pci slot directories in/sys/bus/pci/slots/like this:root@localhost:~# ls -rtl /sys/bus/pci/slots/ total 0 drwxr-xr-x 2 root root 0 Feb 7 10:49 2 drwxr-xr-x 2 root root 0 Feb 7 10:49 2-1 drwxr-xr-x 2 root root 0 Feb 7 10:51 2-2
Quoted 6 lines
The patch adds the missing code.
Fixes a15f2c08c708 ("PCI: hv: support reporting serial number as slot information")Signed-off-by Dexuan Cui <[email protected]>
RE: [PATCH 3/3] PCI: hv: Add pci_destroy_slot() in pci_devices_present_work(), if necessary
In reply to Dexuan CuiFrom: Dexuan Cui <[email protected]> Sent: Monday, March 4, 2019 1:35 PM
drivers/pci/controller/pci-hyperv.c static void pci_devices_present_work(struct work_struct *work) · line 1781
Quoted 12 lines
drivers/pci/controller/pci-hyperv.c +3
-0@@ -1776,6 +1776,10 @@ static void pci_devices_present_work(struct work_struct *work) hpdev = list_first_entry(&removed, struct hv_pci_dev, list_entry);list_del (&hpdev->list_entry); + +if (hpdev->pci_slot) +pci_destroy_slot (hpdev->pci_slot);
The code is inconsistent in whether hpdev->pci_slot is set to NULL after calling pci_destory_slot(). Patch 2 in this series does set it to NULL, but this code does not. And the code in hv_eject_device_work() does not set it to NULL.
It looks like all the places that test the value of hpdev->pci_slot or call pci_destroy_slot() are serialized, so it looks like it really doesn't matter. But when the code is inconsistent about setting to NULL, it always makes me wonder if there is a reason.
Michael
Quoted 6 lines
drivers/pci/controller/pci-hyperv.c +1
-1+put_pcichild (hpdev); } --2.19.1
Dexuan Cui
maintainerRE: [PATCH 1/3] PCI: hv: Fix a memory leak in hv_eject_device_work()
In reply to Michael KelleyQuoted 17 lines
From: Michael Kelley <[email protected]>
Sent: Wednesday, March 20, 2019 2:38 PMFrom: Dexuan Cui <[email protected]>
Quoted 12 lines
After a device is just created in
new_pcichild_device(),hpdev->refsis set to 2 (i.e. the initial value of 1 plus theget_pcichild()).When we hot remove the device from the host, in Linux VM we first call
hv_pci_eject_device(), which increaseshpdev->refsbyget_pcichild()and then schedules a work ofhv_eject_device_work(), sohpdev->refsbecomes 3 (let's ignore the paired get/put_pcichild() in other places). But inhv_eject_device_work(), currently we only callput_pcichild()twice, meaning the 'hpdev' struct can't be freed input_pcichild(). This patch adds oneput_pcichild()to fix the memory leak.Quoted 1 line
BTW, the device can also be removed when we run "rmmod pci-hyperv". On
Quoted 8 lines
this
Quoted 3 lines
path (
hv_pci_remove()->hv_pci_bus_exit()->hv_pci_devices_present()),hpdev->refsis 2, and we do correctly callput_pcichild()twice inpci_devices_present_work().Exiting
new_pcichild_device()withhpdev->refsset to 2 seems OK to me. There is the reference in thehbus->childrenlist, and there is the reference that is returned to the caller.
So IMO the "normal" reference count should be 2. :-) IMO only when a hv_pci_dev device is about to be destroyed, its reference count can drop to less than 2, i.e. first temporarily drop to 1 (meaning the hv_pci_dev device is removed from hbus->children), and then drop to zero (meaning kfree(hpdev) is called).
Quoted 3 lines
But what is strange is that
pci_devices_present_work()overwrites the reference returned in local variable hpdev without doing aput_pcichild().
I suppose you mean:
/* First, mark all existing children as reported missing. */
spin_lock_irqsave(&hbus->device_list_lock, flags);
list_for_each_entry(hpdev, &hbus->children, list_entry) {
hpdev->reported_missing = true;
}
spin_unlock_irqrestore(&hbus->device_list_lock, flags)This is not strange to me, because, in pci_devices_present_work(), at first we don't know which devices are about to disappear, so we pre-mark all devices to be potentially missing like that; if a device is still on the bus, we'll mark its hpdev->reported_missing to false later; only after we know exactly which devices are missing, we should call put_pcichild() against them. All these seem natural to me.
Quoted 2 lines
It seems like the "normal" reference count should be 1 when the child device is not being manipulated, not 2.
What does "not being manipulated" mean?
Quoted 3 lines
The fix would be to add a call to
put_pcichild()when the return value fromnew_pcichild_device()is overwritten.
In pci_devices_present_work(), we NEVER "overwrite" the "hpdev" returned from new_pcichild_device(): the "reported_missing" field of the new hpdev is implicitly initialized to false in new_pcichild_device().
Quoted 8 lines
Then remove the call to
put_pcichild()inpci_device_present_work()when missing
children are moved to the local list. The children have been moved from one list
to another, so there's no need to decrement the reference count. Then when everything in the local list is deleted, the reference is correctly decremented, presumably freeing the memory.
Quoted 4 lines
With this approach, the code in
hv_eject_device_work()is correct. There's one call toput_pcichild()to reflect removing the child device from the hbus-> children list, and one call toput_pcichild()to pair with theget_pcichild()inhv_pci_eject_device().
Please refer to my replies above. IMO we should fix hv_eject_device_work() rather than pci_devices_present_work().
Thanks
-- Dexuan
Quoted 4 lines
Your patch works, but to me it leaves the ref count in an unnatural state most of the time.
Michael
Dexuan Cui
maintainerRE: [PATCH 3/3] PCI: hv: Add pci_destroy_slot() in pci_devices_present_work(), if necessary
In reply to Michael KelleyQuoted 8 lines
From: Michael Kelley <[email protected]>
Sent: Wednesday, March 20, 2019 2:44 PM
To: Dexuan Cui <[email protected]>; [email protected]; [email protected]; [email protected]; KY SrinivasanReview static void pci_devices_present_work(struct
Quoted 3 lines
...
(quoted diff) +0
-0@@ -1776,6 +1776,10 @@ static void pci_devices_present_work(struct
work_struct*work)
Quoted 9 lines
drivers/pci/controller/pci-hyperv.c static void pci_devices_present_work(struct work_struct *work) · line 1781
Quoted 6 lines
drivers/pci/controller/pci-hyperv.c +3
-0hpdev = list_first_entry(&removed, struct hv_pci_dev, list_entry);list_del (&hpdev->list_entry); + +if (hpdev->pci_slot) +pci_destroy_slot (hpdev->pci_slot);The code is inconsistent in whether
hpdev->pci_slotis set to NULL after callingpci_destory_slot().
Here, in pci_devices_present_work(), it's unnecessary to set it to NULL, Because:
1) the "hpdev" is removed from hbus->children and it can not be seen elsewhere;
2) the "hpdev" struct is freed in the below put_pcichild():
while (!list_empty(&removed)) {
hpdev = list_first_entry(&removed, struct hv_pci_dev,
list_entry);
list_del(&hpdev->list_entry);
if (hpdev->pci_slot)
pci_destroy_slot(hpdev->pci_slot);
put_pcichild(hpdev);
}Quoted 1 line
Patch 2 in this series does set it to NULL, but this code does not.
In Patch2, i.e. in the code path hv_pci_remove() -> hv_pci_remove_slots(), we must set hpdev->pci_slot to NULL, otherwise, later, due to hv_pci_remove() -> hv_pci_bus_exit() -> hv_pci_devices_present() with the zero "relations", we'll double-free the "hpdev" struct in pci_devices_present_work() -- see the above.
Quoted 1 line
And the code in
hv_eject_device_work()does not set it to NULL.
It's unnecessary to set hpdev->pci_slot to NULL in hv_eject_device_work(), Because in hv_eject_device_work():
1) the "hpdev" is removed from hbus->children and it can not be seen elsewhere;
2) the "hpdev" struct is freed at the end of hv_eject_device_work() with my first patch: [PATCH 1/3] PCI: hv: Fix a memory leak in hv_eject_device_work().
Quoted 8 lines
It looks like all the places that test the value of
hpdev->pci_slotor callpci_destroy_slot()are serialized, so it looks like it really doesn't matter. But when
the code is inconsistent about setting to NULL, it always makes me wonder if there
is a reason.Michael
Thanks,
-- Dexuan
Dexuan Cui
maintainerRE: [PATCH 3/3] PCI: hv: Add pci_destroy_slot() in pci_devices_present_work(), if necessary
Quoted 9 lines
From: [email protected] <[email protected]> On Behalf Of Dexuan Cui
Quoted 2 lines
...
Patch 2 in this series does set it to NULL, but this code does not.In Patch2, i.e. in the code path
hv_pci_remove()->hv_pci_remove_slots(), we must sethpdev->pci_slotto NULL, otherwise, later, due tohv_pci_remove()->hv_pci_bus_exit()->hv_pci_devices_present()with the zero "relations", we'll double-free the "hpdev" struct inpci_devices_present_work()-- see the above.
A minor correction: I meant we'll call pci_destroy_slot(hpdev->pci_slot) twice, not "double-free hpdev".
Thanks,
-- Dexuan
Re: [PATCH 1/3] PCI: hv: Fix a memory leak in hv_eject_device_work()
In reply to Dexuan CuiQuoted 68 lines · Dexuan Cui
Quoted 17 lines
From: Michael Kelley <[email protected]>
Sent: Wednesday, March 20, 2019 2:38 PMFrom: Dexuan Cui <[email protected]>
Quoted 12 lines
After a device is just created in
new_pcichild_device(),hpdev->refsis set to 2 (i.e. the initial value of 1 plus theget_pcichild()).When we hot remove the device from the host, in Linux VM we first call
hv_pci_eject_device(), which increaseshpdev->refsbyget_pcichild()and then schedules a work ofhv_eject_device_work(), sohpdev->refsbecomes 3 (let's ignore the paired get/put_pcichild() in other places). But inhv_eject_device_work(), currently we only callput_pcichild()twice, meaning the 'hpdev' struct can't be freed input_pcichild(). This patch adds oneput_pcichild()to fix the memory leak.Quoted 1 line
BTW, the device can also be removed when we run "rmmod pci-hyperv". On
Quoted 8 lines
this
Quoted 3 lines
path (
hv_pci_remove()->hv_pci_bus_exit()->hv_pci_devices_present()),hpdev->refsis 2, and we do correctly callput_pcichild()twice inpci_devices_present_work().Exiting
new_pcichild_device()withhpdev->refsset to 2 seems OK to me. There is the reference in thehbus->childrenlist, and there is the reference that is returned to the caller.So IMO the "normal" reference count should be 2. :-) IMO only when a
hv_pci_devdevice is about to be destroyed, its reference count can drop to less than 2, i.e. first temporarily drop to 1 (meaning thehv_pci_devdevice is removed fromhbus->children), and then drop to zero (meaning kfree(hpdev) is called).Quoted 3 lines
But what is strange is that
pci_devices_present_work()overwrites the reference returned in local variable hpdev without doing aput_pcichild().I suppose you mean:
/* First, mark all existing children as reported missing. */ spin_lock_irqsave(&hbus->device_list_lock, flags); list_for_each_entry(hpdev, &hbus->children, list_entry) { hpdev->reported_missing = true; } spin_unlock_irqrestore(&hbus->device_list_lock, flags)This is not strange to me, because, in
pci_devices_present_work(), at first we don't know which devices are about to disappear, so we pre-mark all devices to be potentially missing like that; if a device is still on the bus, we'll mark itshpdev->reported_missingto false later; only after we know exactly which devices are missing, we should callput_pcichild()against them. All these seem natural to me.Quoted 2 lines
It seems like the "normal" reference count should be 1 when the child device is not being manipulated, not 2.
What does "not being manipulated" mean?
Quoted 3 lines
The fix would be to add a call to
put_pcichild()when the return value fromnew_pcichild_device()is overwritten.In
pci_devices_present_work(), we NEVER "overwrite" the "hpdev" returned fromnew_pcichild_device(): the "reported_missing" field of the new hpdev is implicitly initialized to false innew_pcichild_device().Quoted 8 lines
Then remove the call to
put_pcichild()inpci_device_present_work()when missing
children are moved to the local list. The children have been moved from one list
to another, so there's no need to decrement the reference count. Then when everything in the local list is deleted, the reference is correctly decremented, presumably freeing the memory.
Quoted 6 lines
Quoted 4 lines
With this approach, the code in
hv_eject_device_work()is correct. There's one call toput_pcichild()to reflect removing the child device from the hbus-> children list, and one call toput_pcichild()to pair with theget_pcichild()inhv_pci_eject_device().Please refer to my replies above. IMO we should fix
hv_eject_device_work()rather thanpci_devices_present_work().
Have we reached a conclusion on this ? I would like to merge this series given that it is fixing bugs and it has hung in the balance for quite a while but it looks like Michael is not too happy about these patches and I need a maintainer ACK to merge them.
Thanks,
Lorenzo
Quoted 8 lines
Thanks
-- DexuanQuoted 4 lines
Your patch works, but to me it leaves the ref count in an unnatural state most of the time.
Michael
Michael Kelley
Reviewed-byRE: [PATCH 1/3] PCI: hv: Fix a memory leak in hv_eject_device_work()
From: Lorenzo Pieralisi <[email protected]> Sent: Tuesday, March 26, 2019 10:09 AM
Quoted 76 lines · Dexuan Cui
Quoted 68 lines
Quoted 17 lines
From: Michael Kelley <[email protected]>
Sent: Wednesday, March 20, 2019 2:38 PMFrom: Dexuan Cui <[email protected]>
Quoted 12 lines
After a device is just created in
new_pcichild_device(),hpdev->refsis set to 2 (i.e. the initial value of 1 plus theget_pcichild()).When we hot remove the device from the host, in Linux VM we first call
hv_pci_eject_device(), which increaseshpdev->refsbyget_pcichild()and then schedules a work ofhv_eject_device_work(), sohpdev->refsbecomes 3 (let's ignore the paired get/put_pcichild() in other places). But inhv_eject_device_work(), currently we only callput_pcichild()twice, meaning the 'hpdev' struct can't be freed input_pcichild(). This patch adds oneput_pcichild()to fix the memory leak.Quoted 1 line
BTW, the device can also be removed when we run "rmmod pci-hyperv". On
Quoted 8 lines
this
Quoted 3 lines
path (
hv_pci_remove()->hv_pci_bus_exit()->hv_pci_devices_present()),hpdev->refsis 2, and we do correctly callput_pcichild()twice inpci_devices_present_work().Exiting
new_pcichild_device()withhpdev->refsset to 2 seems OK to me. There is the reference in thehbus->childrenlist, and there is the reference that is returned to the caller.So IMO the "normal" reference count should be 2. :-) IMO only when a
hv_pci_devdevice is about to be destroyed, its reference count can drop to less than 2, i.e. first temporarily drop to 1 (meaning thehv_pci_devdevice is removed fromhbus->children), and then drop to zero (meaning kfree(hpdev) is called).Quoted 3 lines
But what is strange is that
pci_devices_present_work()overwrites the reference returned in local variable hpdev without doing aput_pcichild().I suppose you mean:
/* First, mark all existing children as reported missing. */ spin_lock_irqsave(&hbus->device_list_lock, flags); list_for_each_entry(hpdev, &hbus->children, list_entry) { hpdev->reported_missing = true; } spin_unlock_irqrestore(&hbus->device_list_lock, flags)This is not strange to me, because, in
pci_devices_present_work(), at first we don't know which devices are about to disappear, so we pre-mark all devices to be potentially missing like that; if a device is still on the bus, we'll mark itshpdev->reported_missingto false later; only after we know exactly which devices are missing, we should callput_pcichild()against them. All these seem natural to me.Quoted 2 lines
It seems like the "normal" reference count should be 1 when the child device is not being manipulated, not 2.
What does "not being manipulated" mean?
Quoted 3 lines
The fix would be to add a call to
put_pcichild()when the return value fromnew_pcichild_device()is overwritten.In
pci_devices_present_work(), we NEVER "overwrite" the "hpdev" returned fromnew_pcichild_device(): the "reported_missing" field of the new hpdev is implicitly initialized to false innew_pcichild_device().Quoted 8 lines
Then remove the call to
put_pcichild()inpci_device_present_work()when missing
children are moved to the local list. The children have been moved from one list
to another, so there's no need to decrement the reference count. Then when everything in the local list is deleted, the reference is correctly decremented, presumably freeing the memory.Quoted 6 lines
Quoted 4 lines
With this approach, the code in
hv_eject_device_work()is correct. There's one call toput_pcichild()to reflect removing the child device from the hbus-> children list, and one call toput_pcichild()to pair with theget_pcichild()inhv_pci_eject_device().Please refer to my replies above. IMO we should fix
hv_eject_device_work()rather thanpci_devices_present_work().
Quoted 7 lines
Have we reached a conclusion on this ? I would like to merge this series given that it is fixing bugs and it has hung in the balance for quite a while but it looks like Michael is not too happy about these patches and I need a maintainer ACK to merge them.
Thanks,
Lorenzo
Dexuan and I have discussed the topic extensively offline. The patch works in its current form, and I'll agree to it.
Michael Kelley
Reviewed-byRE: [PATCH 3/3] PCI: hv: Add pci_destroy_slot() in pci_devices_present_work(), if necessary
In reply to Dexuan CuiFrom: Dexuan Cui <[email protected]> Sent: Wednesday, March 20, 2019 5:36 PM
Quoted 39 lines
Quoted 6 lines
From: Michael Kelley <[email protected]>
Review static void pci_devices_present_work(struct
Quoted 3 lines
...
(quoted diff) +0
-0@@ -1776,6 +1776,10 @@ static void pci_devices_present_work(struct
work_struct*work)Quoted 9 lines
drivers/pci/controller/pci-hyperv.c static void pci_devices_present_work(struct work_struct *work) · line 1781
Quoted 6 lines
drivers/pci/controller/pci-hyperv.c +3
-0hpdev = list_first_entry(&removed, struct hv_pci_dev, list_entry);list_del (&hpdev->list_entry); + +if (hpdev->pci_slot) +pci_destroy_slot (hpdev->pci_slot);The code is inconsistent in whether
hpdev->pci_slotis set to NULL after callingpci_destory_slot().Here, in
pci_devices_present_work(), it's unnecessary to set it to NULL, Because:
1) the "hpdev" is removed fromhbus->childrenand it can not be seen elsewhere;
2) the "hpdev" struct is freed in the belowput_pcichild():while (!list_empty(&removed)) { hpdev = list_first_entry(&removed, struct hv_pci_dev, list_entry); list_del(&hpdev->list_entry); if (hpdev->pci_slot) pci_destroy_slot(hpdev->pci_slot); put_pcichild(hpdev); }Quoted 1 line
Patch 2 in this series does set it to NULL, but this code does not.
In Patch2, i.e. in the code path
hv_pci_remove()->hv_pci_remove_slots(), we must sethpdev->pci_slotto NULL, otherwise, later, due tohv_pci_remove()->hv_pci_bus_exit()->hv_pci_devices_present()with the zero "relations", we'll double-free the "hpdev" struct inpci_devices_present_work()-- see the above.
Quoted 17 lines
Quoted 1 line
And the code in
hv_eject_device_work()does not set it to NULL.It's unnecessary to set
hpdev->pci_slotto NULL inhv_eject_device_work(), Because inhv_eject_device_work():
1) the "hpdev" is removed fromhbus->childrenand it can not be seen elsewhere;
2) the "hpdev" struct is freed at the end ofhv_eject_device_work()with my first patch: [PATCH 1/3] PCI: hv: Fix a memory leak inhv_eject_device_work().Quoted 8 lines
It looks like all the places that test the value of
hpdev->pci_slotor callpci_destroy_slot()are serialized, so it looks like it really doesn't matter. But when
the code is inconsistent about setting to NULL, it always makes me wonder if there
is a reason.Michael
Dexuan Cui
maintainerRE: [PATCH 1/3] PCI: hv: Fix a memory leak in hv_eject_device_work()
In reply to Michael KelleyQuoted 113 lines
From: Michael Kelley <[email protected]>
Sent: Tuesday, March 26, 2019 10:47 AM
To: Lorenzo Pieralisi <[email protected]>; Dexuan Cui <[email protected]>
Cc: [email protected]; [email protected]; KY Srinivasan <[email protected]>; Stephen Hemminger <[email protected]>; Sasha Levin <[email protected]>; [email protected]; [email protected]; [email protected]; Haiyang Zhang <[email protected]>; [email protected]; [email protected]; [email protected]; vkuznets <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 1/3] PCI: hv: Fix a memory leak inhv_eject_device_work()From: Lorenzo Pieralisi <[email protected]> Sent: Tuesday, March 26, 2019 10:09 AM
Quoted 7 lines · Dexuan Cui
Quoted 6 lines
Quoted 6 lines
From: Michael Kelley <[email protected]>
Sent: Wednesday, March 20, 2019 2:38 PMFrom: Dexuan Cui <[email protected]>
Quoted 2 lines
After a device is just created in
new_pcichild_device(),hpdev->refsisset
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
to 2 (i.e. the initial value of 1 plus the
get_pcichild()).When we hot remove the device from the host, in Linux VM we first call
hv_pci_eject_device(), which increaseshpdev->refsbyget_pcichild()and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
then schedules a work of
hv_eject_device_work(), sohpdev->refsbecomes 3
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
(let's ignore the paired get/put_pcichild() in other places). But in
hv_eject_device_work(), currently we only callput_pcichild()twice, meaning the 'hpdev' struct can't be freed input_pcichild(). This patch adds oneput_pcichild()to fix the memory leak.BTW, the device can also be removed when we run "rmmod pci-hyperv".
On
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
this
Quoted 1 line
path (
hv_pci_remove()->hv_pci_bus_exit()->
hv_pci_devices_present()),Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
hpdev->refsis 2, and we do correctly callput_pcichild()twice inpci_devices_present_work().Exiting
new_pcichild_device()withhpdev->refsset to 2 seems OK to me. There is the reference in thehbus->childrenlist, and there is thereference that
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
is returned to the caller.
So IMO the "normal" reference count should be 2. :-) IMO only when a
hv_pci_devQuoted 2 lines
Quoted 2 lines
device is about to be destroyed, its reference count can drop to less than 2, i.e. first temporarily drop to 1 (meaning the
hv_pci_devdevice is removedfrom
Quoted 7 lines
Quoted 7 lines
hbus->children), and then drop to zero (meaning kfree(hpdev) is called).Quoted 3 lines
But what is strange is that
pci_devices_present_work()overwrites the reference returned in local variable hpdev without doing aput_pcichild().I suppose you mean:
Quoted 8 lines
Quoted 8 lines
/* First, mark all existing children as reported missing. */ spin_lock_irqsave(&hbus->device_list_lock, flags); list_for_each_entry(hpdev, &hbus->children, list_entry) { hpdev->reported_missing = true; } spin_unlock_irqrestore(&hbus->device_list_lock, flags)This is not strange to me, because, in
pci_devices_present_work(), at firstwe
Quoted 1 line
Quoted 1 line
don't know which devices are about to disappear, so we pre-mark all
devices to
Quoted 5 lines
Quoted 5 lines
be potentially missing like that; if a device is still on the bus, we'll mark its
hpdev->reported_missingto false later; only after we know exactly which devices are missing, we should callput_pcichild()against them. All these seem natural to me.Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
It seems like the "normal" reference count should be 1 when the child device is not being manipulated, not 2.
What does "not being manipulated" mean?
Quoted 3 lines
The fix would be to add a call to
put_pcichild()when the return value fromnew_pcichild_device()is overwritten.In
pci_devices_present_work(), we NEVER "overwrite" the "hpdev"returned
Quoted 4 lines
Quoted 4 lines
from
new_pcichild_device(): the "reported_missing" field of the new hpdev is implicitly initialized to false innew_pcichild_device().Quoted 1 line
Then remove the call to
put_pcichild()inpci_device_present_work()when
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
missing
children are moved to the local list. The children have been moved fromone
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
list
to another, so there's no need to decrement the reference count. Thenwhen
Quoted 1 line
Quoted 1 line
Quoted 1 line
everything in the local list is deleted, the reference is correctly
decremented,
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
presumably freeing the memory.
With this approach, the code in
hv_eject_device_work()is correct.There's
Quoted 1 line
Quoted 1 line
Quoted 1 line
one call to
put_pcichild()to reflect removing the child device from thehbus->
Quoted 10 lines
Quoted 4 lines
Quoted 2 lines
children list, and one call to
put_pcichild()to pair with theget_pcichild()inhv_pci_eject_device().Please refer to my replies above. IMO we should fix
hv_eject_device_work()rather thanpci_devices_present_work().Have we reached a conclusion on this ? I would like to merge this series given that it is fixing bugs and it has hung in the balance for quite a while but it looks like Michael is not too happy about these patches and I need a maintainer ACK to merge them.
Quoted 7 lines
Quoted 2 lines
Thanks,
LorenzoDexuan and I have discussed the topic extensively offline. The patch works in its current form, and I'll agree to it.
Reviewed-by Michael Kelley <[email protected]>
Thanks, Michael!
Hi Lorenzo,
All the 3 patches have got Michael's Reviewed-by.
Previously, Stephen Hemminger, one of the Hyper-V driver maintainers, provided his Reviewed-by in the " [PATCH 0/3]" mail:
https://lkml.org/lkml/2019/3/5/521
Thanks,
--Dexuan
Re: [PATCH 1/3] PCI: hv: Fix a memory leak in hv_eject_device_work()
On Tue, Mar 26, 2019 at 06:01:32PM +0000, Dexuan Cui wrote:
[...]
Quoted 11 lines
Quoted 11 lines
Quoted 7 lines
Have we reached a conclusion on this ? I would like to merge this series given that it is fixing bugs and it has hung in the balance for quite a while but it looks like Michael is not too happy about these patches and I need a maintainer ACK to merge them.
Thanks,
LorenzoDexuan and I have discussed the topic extensively offline. The patch works in its current form, and I'll agree to it.
Quoted 6 lines
Quoted 1 line
Reviewed-by Michael Kelley <[email protected]>Thanks, Michael!
Hi Lorenzo,
All the 3 patches have got Michael's Reviewed-by.
Good, I asked because I do not want to merge patches with review questions open, thanks for the clarifications.
Quoted 3 lines
Previously, Stephen Hemminger, one of the Hyper-V driver maintainers, provided his Reviewed-by in the " [PATCH 0/3]" mail:
https://lkml.org/lkml/2019/3/5/521
I will reformat/rewrite the logs and notify you when queued.
Thanks,
Lorenzo
Re: [PATCH 3/3] PCI: hv: Add pci_destroy_slot() in pci_devices_present_work(), if necessary
In reply to Dexuan CuiQuoted 22 lines · Dexuan Cui
When we hot-remove a device, usually the host sends us a
PCI_EJECTmessage, and aPCI_BUS_RELATIONSmessage withbus_rel->device_count== 0. But when we do the quick hot-add/hot-remove test, the host may not send us thePCI_EJECTmessage, if the guest has not fully finished the initialization by sending the PCI_RESOURCES_ASSIGNED* message to the host, so it's potentially unsafe to only depend on thepci_destroy_slot()inhv_eject_device_work(), thoughcreate_root_hv_pci_bus()->hv_pci_assign_slots()is not called in this case. Note: in this case, the host still sends the guest aPCI_BUS_RELATIONSmessage withbus_rel->device_count== 0.And, in the quick hot-add/hot-remove test, we can have such a race: before
pci_devices_present_work()->new_pcichild_device()adds the new device intohbus->children, we may have already received thePCI_EJECTmessage, and hence the taklet handlerhv_pci_onchannelcallback()may fail to find the "hpdev" byget_pcichild_wslot(hbus, dev_message->wslot.slot), sohv_pci_eject_device()is NOT called; latercreate_root_hv_pci_bus()->hv_pci_assign_slots()creates the slot, and thePCI_BUS_RELATIONSmessage withbus_rel->device_count== 0 removes the device fromhbus->children, and we end up being unable to remove the slot inhv_pci_remove()->hv_pci_remove_slots().
Quoted 10 lines
The patch removes the slot in
pci_devices_present_work()when the device is removed. This can address the above race. Note 1:pci_devices_present_work()andhv_eject_device_work()run in the singled-threadedhbus->wq, so there is not a double-remove issue for the slot. Note 2: we can't offloadhv_pci_eject_device()fromhv_pci_onchannelcallback()to the workqueue, because we needhv_pci_onchannelcallback()synchronously callhv_pci_eject_device()to poll the channel's ringbuffer to work around the
"hangs inhv_compose_msi_msg()" issue: see
commitde0aa7b2f97d("PCI: hv: Fix 2 hang issues inhv_compose_msi_msg()")
This commit log is unreadable, sorry. Indentation, punctuation and formatting are just a mess, try to read it, you will notice by yourself.
I basically reformatted it completely and pushed the series to pci/controller-fixes but that's the last time I do it since I am not an editor, next time I won't merge it.
More importantly, these patches are marked for stable, given the series of fixes that triggered this series please ensure it was tested thoroughly because it is honestly complicate to understand and I do not want to backport further fixes to stable kernels on top of this.
Please have a look and report back.
Thanks,
Lorenzo
Quoted 25 lines
Fixes a15f2c08c708 ("PCI: hv: support reporting serial number as slot information")Signed-off-by Dexuan Cui <[email protected]>--- drivers/pci/controller/pci-hyperv.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/pci/controller/pci-hyperv.c +4
-0@@ -1776,6 +1776,10 @@ static void pci_devices_present_work(struct work_struct *work) hpdev = list_first_entry(&removed, struct hv_pci_dev, list_entry);list_del (&hpdev->list_entry); + +if (hpdev->pci_slot) +pci_destroy_slot (hpdev->pci_slot); +put_pcichild (hpdev); }Signature · 1 line
2.19.1
Dexuan Cui
maintainerRE: [PATCH 3/3] PCI: hv: Add pci_destroy_slot() in pci_devices_present_work(), if necessary
Quoted 40 lines
From: Lorenzo Pieralisi <[email protected]>
Sent: Tuesday, March 26, 2019 12:55 PMQuoted 1 line · Dexuan Cui
When we hot-remove a device, usually the host sends us a
PCI_EJECTmessage,
Quoted 1 line
and a
PCI_BUS_RELATIONSmessage withbus_rel->device_count== 0. Butwhen
Quoted 9 lines
we do the quick hot-add/hot-remove test, the host may not send us the
PCI_EJECTmessage, if the guest has not fully finished the initialization by sending the PCI_RESOURCES_ASSIGNED* message to the host, so it's potentially unsafe to only depend on thepci_destroy_slot()inhv_eject_device_work(), thoughcreate_root_hv_pci_bus()->hv_pci_assign_slots()is not called in this case. Note: in this case, the host still sends the guest aPCI_BUS_RELATIONSmessage withbus_rel->device_count== 0.Quoted 8 lines
And, in the quick hot-add/hot-remove test, we can have such a race: before
pci_devices_present_work()->new_pcichild_device()adds the new device intohbus->children, we may have already received thePCI_EJECTmessage, and hence the taklet handlerhv_pci_onchannelcallback()may fail to find the "hpdev" byget_pcichild_wslot(hbus, dev_message->wslot.slot), sohv_pci_eject_device()is NOT called; latercreate_root_hv_pci_bus()->hv_pci_assign_slots()creates the slot, and thePCI_BUS_RELATIONSmessage withbus_rel->device_count== 0 removes the device fromhbus->children,and
Quoted 13 lines
we end up being unable to remove the slot in
hv_pci_remove()->hv_pci_remove_slots().The patch removes the slot in
pci_devices_present_work()when the device is removed. This can address the above race. Note 1:pci_devices_present_work()andhv_eject_device_work()run in the singled-threadedhbus->wq, so there is not a double-remove issue for the slot. Note 2: we can't offloadhv_pci_eject_device()fromhv_pci_onchannelcallback()to the workqueue, because we needhv_pci_onchannelcallback()synchronously callhv_pci_eject_device()to poll the channel's ringbuffer to work around the
"hangs inhv_compose_msi_msg()" issue: see
commitde0aa7b2f97d("PCI: hv: Fix 2 hang issues in
hv_compose_msi_msg()")
Quoted 7 lines
This commit log is unreadable, sorry. Indentation, punctuation and formatting are just a mess, try to read it, you will notice by yourself.
I basically reformatted it completely and pushed the series to pci/controller-fixes but that's the last time I do it since I am not an editor, next time I won't merge it.
Hi Lorenzo,
Thank you for helping improve my changelogs! I did learn a lot after carefully comparing the improved version with my original version. :-)
I'll try my best to write a good changelog for my future patches.
Quoted 4 lines
More importantly, these patches are marked for stable, given the series of fixes that triggered this series please ensure it was tested thoroughly because it is honestly complicate to understand and I do not want to backport further fixes to stable kernels on top of this.
I did the hot-add/hot-remove test in a loop for several thousand times, and the patchset worked as expected and didn't show any issue.
Quoted 4 lines
Please have a look and report back.
Thanks,
Lorenzo
Thanks again!
Thanks,
-- Dexuan