Support Hyper-V vPCI controller reset method
Subsystems (5)
[email protected]
maintainercover Support Hyper-V vPCI controller reset method
From: Wei Liu <[email protected]>
Hi,
On Azure, there are devices that don't support function level or PM reset. They can only be reset by sending a VMBus message to the host.
This series is RFC because I modelled the reset method at the controller level. If there is a better way to do it, I'm happy to change the code.
Thanks,
Wei
Wei Liu (2):
PCI: Add controller reset method
PCI: hv: Add vPCI device reset support
Documentation/virt/hyperv/vpci.rst | 2 +- drivers/pci/controller/pci-hyperv.c | 61 +++++++++++++++++++++++++++++ drivers/pci/pci.c | 22 +++++++++++ include/linux/pci.h | 3 +- 4 files changed, 86 insertions(+), 2 deletions(-)
Signature · 1 line
2.53.0
[email protected]
maintainerRFC PATCH 1/2 PCI: Add controller reset method
From: Wei Liu <[email protected]>
Some PCI controllers provide a function reset mechanism that is not advertised in PCI configuration space. Allow them to expose it through an optional pci_ops callback.
Add the controller reset method to the standard reset_method interface. Prefer FLR and AF FLR by default, and use the controller operation before PM and bus reset fallbacks.
--- drivers/pci/pci.c | 22 ++++++++++++++++++++++ include/linux/pci.h | 3 ++- 2 files changed, 24 insertions(+), 1 deletion(-)
drivers/pci/pci.c +22 -0
@@ -4962,6 +4962,27 @@ static int pci_reset_bus_function(struct pci_dev *dev, bool probe)return rc; } +static int pci_controller_reset (struct pci_dev *dev,bool probe) +{ +int rc; + +if (!dev->bus->ops->reset) +return -ENOTTY ; + +if (probe) +return dev->bus->ops->reset (dev, probe); + + rc =pci_dev_reset_iommu_prepare (dev); +if (rc) { +pci_err (dev,"failed to stop IOMMU for a PCI reset: %d\n" , rc); +return rc; + } + + rc = dev->bus->ops->reset (dev, probe); +pci_dev_reset_iommu_done (dev); +return rc; +} +static int cxl_reset_bus_function (struct pci_dev *dev,bool probe) {struct pci_dev *bridge;
@@ -5094,6 +5115,7 @@ const struct pci_reset_fn_method pci_reset_fn_methods[] = { { pci_dev_acpi_reset, .name = "acpi" }, { pcie_reset_flr, .name = "flr" }, { pci_af_flr, .name = "af_flr" }, + { pci_controller_reset, .name = "controller" }, { pci_pm_reset, .name = "pm" }, { pci_reset_bus_function, .name = "bus" }, { cxl_reset_bus_function, .name = "cxl_bus" },
include/linux/pci.h +2 -1
@@ -52,7 +52,7 @@ PCI_STATUS_PARITY)/* Number of reset methods used in pci_reset_fn_methods array in pci.c */ -#define PCI_NUM_RESET_METHODS 8 +#define PCI_NUM_RESET_METHODS 9#define PCI_RESET_PROBE true#define PCI_RESET_DO_RESET false
@@ -875,6 +875,7 @@ struct pci_ops {void __iomem *(*map_bus)(struct pci_bus *bus,unsigned int devfn,int where);int (*read)(struct pci_bus *bus,unsigned int devfn,int where,int size,u32 *val);int (*write)(struct pci_bus *bus,unsigned int devfn,int where,int size,u32 val); +int (*reset)(struct pci_dev *dev,bool probe); };/*
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH RFC 1/2] PCI: Add controller reset method
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
From: Wei Liu <[email protected]>
Some PCI controllers provide a function reset mechanism that is not advertised in PCI configuration space. Allow them to expose it through an optional
pci_opscallback.Add the controller reset method to the standard
reset_methodinterface. Prefer FLR and AF FLR by default, and use the controller operation before PM and bus reset fallbacks.Signed-off-by Wei Liu <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Wei Liu
maintainerRe: [PATCH RFC 1/2] PCI: Add controller reset method
In reply to [email protected]Hi Bjorn,
Since this is adding to the PCI reset framework, this patch needs your approval.
Please see below for my questions. I'm happy to change the code however you see fit.
Quoted 35 lines · On Fri, Jul 24, 2026 at 04:08:42PM -0700, [email protected] wrote:
From: Wei Liu <[email protected]>
Some PCI controllers provide a function reset mechanism that is not advertised in PCI configuration space. Allow them to expose it through an optional
pci_opscallback.Add the controller reset method to the standard
reset_methodinterface. Prefer FLR and AF FLR by default, and use the controller operation before PM and bus reset fallbacks.Signed-off-by Wei Liu <[email protected]>--- drivers/pci/pci.c | 22 ++++++++++++++++++++++ include/linux/pci.h | 3 ++- 2 files changed, 24 insertions(+), 1 deletion(-)drivers/pci/pci.c +11
-0@@ -4962,6 +4962,27 @@ static int pci_reset_bus_function(struct pci_dev *dev, bool probe)return rc; } +static int pci_controller_reset (struct pci_dev *dev,bool probe) +{ +int rc; + +if (!dev->bus->ops->reset) +return -ENOTTY ; + +if (probe) +return dev->bus->ops->reset (dev, probe); + + rc =pci_dev_reset_iommu_prepare (dev);
drivers/pci/pci.c static int pci_reset_bus_function(struct pci_dev *dev, bool probe) · line 4985
Quoted 10 lines
drivers/pci/pci.c +10
-0+if (rc) { +pci_err (dev,"failed to stop IOMMU for a PCI reset: %d\n" , rc); +return rc; + } + + rc = dev->bus->ops->reset (dev, probe); +pci_dev_reset_iommu_done (dev); +return rc; +} +
The first question is whether modelling this on the controller level is the correct approach.
Please refer to the second patch for the intended usage in the Hyper-V vPCI code.
Whatever is added here, a new reset_method value will be added to the table. I chose "controller" to reflect the decision above.
drivers/pci/pci.c const struct pci_reset_fn_method pci_reset_fn_methods[] = { · line 5118
Quoted 8 lines
drivers/pci/pci.c +1
-0static int cxl_reset_bus_function (struct pci_dev *dev,bool probe) {struct pci_dev *bridge;@@ -5094,6 +5115,7 @@ const struct pci_reset_fn_method pci_reset_fn_methods[] = { { pci_dev_acpi_reset, .name = "acpi" }, { pcie_reset_flr, .name = "flr" }, { pci_af_flr, .name = "af_flr" }, + { pci_controller_reset, .name = "controller" },
The second question is whether this ordering is okay.
Thanks,
Wei
Quoted 27 lines
drivers/pci/pci.c +0
-0{ pci_pm_reset, .name = "pm" }, { pci_reset_bus_function, .name = "bus" }, { cxl_reset_bus_function, .name = "cxl_bus" },include/linux/pci.h +2
-1@@ -52,7 +52,7 @@ PCI_STATUS_PARITY)/* Number of reset methods used in pci_reset_fn_methods array in pci.c */ -#define PCI_NUM_RESET_METHODS 8 +#define PCI_NUM_RESET_METHODS 9#define PCI_RESET_PROBE true#define PCI_RESET_DO_RESET false@@ -875,6 +875,7 @@ struct pci_ops {void __iomem *(*map_bus)(struct pci_bus *bus,unsigned int devfn,int where);int (*read)(struct pci_bus *bus,unsigned int devfn,int where,int size,u32 *val);int (*write)(struct pci_bus *bus,unsigned int devfn,int where,int size,u32 val); +int (*reset)(struct pci_dev *dev,bool probe); };/* Signature · 1 line
2.53.0
Manivannan Sadhasivam
maintainerRe: [PATCH RFC 1/2] PCI: Add controller reset method
Quoted 58 lines · Wei Liu
Hi Bjorn,
Since this is adding to the PCI reset framework, this patch needs your approval.
Please see below for my questions. I'm happy to change the code however you see fit.
Quoted 35 lines · On Fri, Jul 24, 2026 at 04:08:42PM -0700, [email protected] wrote:
From: Wei Liu <[email protected]>
Some PCI controllers provide a function reset mechanism that is not advertised in PCI configuration space. Allow them to expose it through an optional
pci_opscallback.Add the controller reset method to the standard
reset_methodinterface. Prefer FLR and AF FLR by default, and use the controller operation before PM and bus reset fallbacks.Signed-off-by Wei Liu <[email protected]>--- drivers/pci/pci.c | 22 ++++++++++++++++++++++ include/linux/pci.h | 3 ++- 2 files changed, 24 insertions(+), 1 deletion(-)drivers/pci/pci.c +11
-0@@ -4962,6 +4962,27 @@ static int pci_reset_bus_function(struct pci_dev *dev, bool probe)return rc; } +static int pci_controller_reset (struct pci_dev *dev,bool probe) +{ +int rc; + +if (!dev->bus->ops->reset) +return -ENOTTY ; + +if (probe) +return dev->bus->ops->reset (dev, probe); + + rc =pci_dev_reset_iommu_prepare (dev);drivers/pci/pci.c static int pci_reset_bus_function(struct pci_dev *dev, bool probe) · line 4985
Quoted 10 lines
drivers/pci/pci.c +10
-0+if (rc) { +pci_err (dev,"failed to stop IOMMU for a PCI reset: %d\n" , rc); +return rc; + } + + rc = dev->bus->ops->reset (dev, probe); +pci_dev_reset_iommu_done (dev); +return rc; +} +The first question is whether modelling this on the controller level is the correct approach.
Quoted 6 lines
Please refer to the second patch for the intended usage in the Hyper-V vPCI code.
Whatever is added here, a new
reset_methodvalue will be added to the table. I chose "controller" to reflect the decision above.
I recently added a hook for resetting the Root Ports in a platform specific way [1]. This new hook is called in the SBR reset path. Will it work for you?
- Mani
[1] https://lore.kernel.org/all/[email protected]/
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Wei Liu
maintainerRe: [PATCH RFC 1/2] PCI: Add controller reset method
Quoted 8 lines · Manivannan Sadhasivam
I recently added a hook for resetting the Root Ports in a platform specific way [1]. This new hook is called in the SBR reset path. Will it work for you?
- Mani
Thank you. I will check that out.
Wei
[email protected]
maintainerRFC PATCH 2/2 PCI: hv: Add vPCI device reset support
In reply to [email protected]From: Wei Liu <[email protected]>
Hyper-V vPCI protocol version 1.5 adds a RESET_DEVICE request for projected PCI functions. Negotiate the new protocol version and issue the request through the vPCI VMBus channel from the PCI controller reset callback.
Use the existing VMBus response path. Return -ENOTTY when the host reports STATUS_NOT_SUPPORTED so PCI core may try another reset method.
--- Documentation/virt/hyperv/vpci.rst | 2 +- drivers/pci/controller/pci-hyperv.c | 61 +++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+), 1 deletion(-)
Documentation/virt/hyperv/vpci.rst +1 -1
@@ -65,7 +65,7 @@ exchange messages with the vPCI VSP for the purpose of setting up and configuring the vPCI device in Linux. Once the device is fully configured in Linux as a PCI device, the VMBus channel is used only if Linux changes the vCPU to be interrupted -in the guest, or if the vPCI device is removed from +in the guest, or if the vPCI device is reset or removed from the VM while the VM is running. The ongoing operation of the device happens directly between the Linux device driver for the device and the hardware, with VMBus and the VMBus channel
drivers/pci/controller/pci-hyperv.c +61 -0
101 lines not on this page · raw message
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH RFC 2/2] PCI: hv: Add vPCI device reset support
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 10 lines
From: Wei Liu <[email protected]>
Hyper-V vPCI protocol version 1.5 adds a
RESET_DEVICErequest for projected PCI functions. Negotiate the new protocol version and issue the request through the vPCI VMBus channel from the PCI controller reset callback.Use the existing VMBus response path. Return
-ENOTTYwhen the host reportsSTATUS_NOT_SUPPORTEDso PCI core may try another reset method.Signed-off-by Wei Liu <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
RE: [PATCH RFC 2/2] PCI: hv: Add vPCI device reset support
In reply to [email protected]From: [email protected] <[email protected]> Sent: Friday, July 24, 2026 4:09 PM
Quoted 32 lines
Hyper-V vPCI protocol version 1.5 adds a
RESET_DEVICErequest for projected PCI functions. Negotiate the new protocol version and issue the request through the vPCI VMBus channel from the PCI controller reset callback.Use the existing VMBus response path. Return
-ENOTTYwhen the host reportsSTATUS_NOT_SUPPORTEDso PCI core may try another reset method.Signed-off-by Wei Liu <[email protected]>--- Documentation/virt/hyperv/vpci.rst | 2 +- drivers/pci/controller/pci-hyperv.c | 61 +++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+), 1 deletion(-)Documentation/virt/hyperv/vpci.rst +1
-1@@ -65,7 +65,7 @@ exchange messages with the vPCI VSP for the purpose of setting up and configuring the vPCI device in Linux. Once the device is fully configured in Linux as a PCI device, the VMBus channel is used only if Linux changes the vCPU to be interrupted -in the guest, or if the vPCI device is removed from +in the guest, or if the vPCI device is reset or removed from the VM while the VM is running. The ongoing operation of the device happens directly between the Linux device driver for the device and the hardware, with VMBus and the VMBus channeldrivers/pci/controller/pci-hyperv.c +0
-0
drivers/pci/controller/pci-hyperv.c enum pci_protocol_version_t { · line 71
Quoted 4 lines
drivers/pci/controller/pci-hyperv.c +1
-0@@ -68,6 +68,7 @@ enum pci_protocol_version_t {PCI_PROTOCOL_VERSION_1_2 =PCI_MAKE_VERSION (1 ,2 ),/* RS1 */ PCI_PROTOCOL_VERSION_1_3 =PCI_MAKE_VERSION (1 ,3 ),/* Vibranium */ PCI_PROTOCOL_VERSION_1_4 =PCI_MAKE_VERSION (1 ,4 ),/* WS2022 */ + PCI_PROTOCOL_VERSION_1_5 = PCI_MAKE_VERSION(1, 5), /* GE, device reset */
I wish there were a better way to identify the Hyper-V version than internal code names that mean nothing to the Linux community. "GE" refers to Germanium, which would be the Build 26100 series, right?
Quoted 67 lines
drivers/pci/controller/pci-hyperv.c +38
-0};#define CPU_AFFINITY_ALL -1ULL@@ -77,6 +78,7 @@ enum pci_protocol_version_t {* first. */ static enum pci_protocol_version_t pci_protocol_versions[] = { +PCI_PROTOCOL_VERSION_1_5 ,PCI_PROTOCOL_VERSION_1_4 ,PCI_PROTOCOL_VERSION_1_3 ,PCI_PROTOCOL_VERSION_1_2 ,@@ -90,6 +92,7 @@ static enum pci_protocol_version_t pci_protocol_versions[] = {#define MAX_SUPPORTED_MSI_MESSAGES 0x400#define STATUS_REVISION_MISMATCH 0xC0000059 +#define STATUS_NOT_SUPPORTED 0xC00000BB/* space for 32bit serial number as string */ #define SLOT_NAME_SIZE 11@@ -136,6 +139,7 @@ enum pci_message_type {PCI_BUS_RELATIONS2 =PCI_MESSAGE_BASE +0x19 ,PCI_RESOURCES_ASSIGNED3 =PCI_MESSAGE_BASE +0x1A ,PCI_CREATE_INTERRUPT_MESSAGE3 =PCI_MESSAGE_BASE +0x1B , +PCI_RESET_DEVICE =PCI_MESSAGE_BASE +0x1C , PCI_MESSAGE_MAXIMUM };@@ -1397,10 +1401,66 @@ static int hv_pcifront_write_config(struct pci_bus *bus,unsigned int devfn, return PCIBIOS_SUCCESSFUL; } +static int hv_pcifront_reset (struct pci_dev *pdev,bool probe) +{ +struct hv_pcibus_device *hbus = + container_of(pdev->bus->sysdata, struct hv_pcibus_device, sysdata); +struct pci_child_message reset = {}; +struct hv_pci_compl comp_pkt; +struct pci_packet pkt = { + .completion_func = hv_pci_generic_compl, + .compl_ctxt = &comp_pkt, + }; +enum hv_pcibus_state state; +int ret; + +/* Device reset was added in vPCI protocol version 1.5. */ +if (hbus->protocol_version <PCI_PROTOCOL_VERSION_1_5 ) +return -ENOTTY ; + +/* Hyper-V exposes projected functions directly on the root bus. */ +if (!pci_is_root_bus (pdev->bus)) +return -ENOTTY ; + +if (probe) +return 0 ; + +/* Do not take state_lock: eject holds it while removing/locking pdev. */ + state =READ_ONCE (hbus->state); +if (state != hv_pcibus_probed && state != hv_pcibus_installed) +return -ENODEV ; + +init_completion (&comp_pkt.host_event); + reset.message_type.type =PCI_RESET_DEVICE ; + reset.wslot.slot =devfn_to_wslot (pdev->devfn); + + ret = vmbus_sendpacket(hbus->hdev->channel, &reset, sizeof(reset), + (unsigned long)&pkt, VM_PKT_DATA_INBAND,
drivers/pci/controller/pci-hyperv.c static int hv_pcifront_write_config(struct pci_bus *bus, · line 1449
Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +10
-0+ VMBUS_DATA_PACKET_FLAG_COMPLETION_REQUESTED); +if (ret) +return ret; + + ret =wait_for_response (hbus->hdev, &comp_pkt.host_event); +if (ret) +return ret; + +if (comp_pkt.completion_status ==STATUS_NOT_SUPPORTED ) +return -ENOTTY ;
I tried this patch series in a linux-next20260726 build, and running on a D16lds v6 VM in Azure. The host hypervisor version is 10.0.26100.1652-1-0, and the VM has a paravisor with HvLite.
This VM has an NVMe OS disk, two NVMe temp disks, and a MANA network controller. Absent this patch set, the temp disks and MANA report the "reset_method" as "flr", while the NVMe OS disk reports no reset methods. With this patch set, "controller" is added as a reset method for all. The NVMe disks and MANA network controller are probed with PCI protocol 1.5. That's all good and as expected (though I'm not sure why the NVMe OS disk doesn't support flr).
I then did "echo 1 >reset" for the NVMe OS disk. This returns a -ENOTTY error from the above line of code. So the host hypervisor (or paravisor?) is saying that the new PCI_RESET_DEVICE message isn't supported. Presumably this is new functionality that hasn’t been rolled out to where I'm running this Azure VM. That's fine too.
But interestingly, the NVMe OS disk did a Linux-side reset anyway. That's because of this stack trace from the "echo 1 >reset" command:
[ 99.642483] nvme_try_sched_reset+0x25/0x60 [nvme_core] [ 99.642495] nvme_reset_done+0x1c/0x40 [nvme] [ 99.642499] pci_dev_restore+0x35/0x70 [ 99.642503] pci_reset_function+0x100/0x140 [ 99.642506] reset_store+0x5a/0xa0 [ 99.642508] dev_attr_store+0x16/0x30 [ 99.642512] sysfs_kf_write+0x71/0x80 [ 99.642516] kernfs_fop_write_iter+0x140/0x1d0 [ 99.642518] vfs_write+0x313/0x420 [ 99.642522] ksys_write+0x68/0xe0 [ 99.642525] __x64_sys_write+0x18/0x20 [ 99.642527] x64_sys_call+0x1700/0x21c0 [ 99.642530] do_syscall_64+0x8d/0x440 [ 99.642532] entry_SYSCALL_64_after_hwframe+0x76/0x7e
Even though __pci_reset_function_locked() failed, the subsequent code in pci_reset_function() calls nvme_try_sched_reset(), which puts nvme_reset_work() on a workqueue. nvme_reset_work() tears things down on the Linux side and rebuilds, and the message
nvme nvme0: 16/0/0 default/read/poll queues
is output.
It's not immediately clear to me how to resolve this issue, so I'm just pointing it out. :-(
Michael
Quoted 30 lines
drivers/pci/controller/pci-hyperv.c +11
-0+ +if (comp_pkt.completion_status) { +pci_err (pdev,"Hyper-V device reset failed: %#x\n" , + comp_pkt.completion_status); +return -EIO ; + } + +return 0 ; +} +/* PCIe operations */ static struct pci_ops hv_pcifront_ops = { .read = hv_pcifront_read_config, .write = hv_pcifront_write_config, + .reset = hv_pcifront_reset, };/*
drivers/pci/controller/pci-hyperv.c +1
-0@@ -1996,6 +2056,7 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg) break;case PCI_PROTOCOL_VERSION_1_4 : +case PCI_PROTOCOL_VERSION_1_5 : size = hv_compose_msi_req_v3(&ctxt.int_pkts.v3, cpu,
hpdev->desc.win_slot.slot,
--
2.53.0
Wei Liu
maintainerRe: [PATCH RFC 2/2] PCI: hv: Add vPCI device reset support
Quoted 33 lines · Michael Kelley
From: [email protected] <[email protected]> Sent: Friday, July 24, 2026 4:09 PM
Quoted 32 lines
Hyper-V vPCI protocol version 1.5 adds a
RESET_DEVICErequest for projected PCI functions. Negotiate the new protocol version and issue the request through the vPCI VMBus channel from the PCI controller reset callback.Use the existing VMBus response path. Return
-ENOTTYwhen the host reportsSTATUS_NOT_SUPPORTEDso PCI core may try another reset method.Signed-off-by Wei Liu <[email protected]>--- Documentation/virt/hyperv/vpci.rst | 2 +- drivers/pci/controller/pci-hyperv.c | 61 +++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+), 1 deletion(-)Documentation/virt/hyperv/vpci.rst +1
-1@@ -65,7 +65,7 @@ exchange messages with the vPCI VSP for the purpose of setting up and configuring the vPCI device in Linux. Once the device is fully configured in Linux as a PCI device, the VMBus channel is used only if Linux changes the vCPU to be interrupted -in the guest, or if the vPCI device is removed from +in the guest, or if the vPCI device is reset or removed from the VM while the VM is running. The ongoing operation of the device happens directly between the Linux device driver for the device and the hardware, with VMBus and the VMBus channeldrivers/pci/controller/pci-hyperv.c +0
-0@@ -68,6 +68,7 @@ enum pci_protocol_version_t {
Quoted 9 lines
drivers/pci/controller/pci-hyperv.c enum pci_protocol_version_t { · line 71
Quoted 4 lines
drivers/pci/controller/pci-hyperv.c +1
-0PCI_PROTOCOL_VERSION_1_2 =PCI_MAKE_VERSION (1 ,2 ),/* RS1 */ PCI_PROTOCOL_VERSION_1_3 =PCI_MAKE_VERSION (1 ,3 ),/* Vibranium */ PCI_PROTOCOL_VERSION_1_4 =PCI_MAKE_VERSION (1 ,4 ),/* WS2022 */ + PCI_PROTOCOL_VERSION_1_5 = PCI_MAKE_VERSION(1, 5), /* GE, device reset */I wish there were a better way to identify the Hyper-V version than internal code names that mean nothing to the Linux community. "GE" refers to Germanium, which would be the Build 26100 series, right?
Yes, you're right about Germanium. I see the numbers also match, though I don't have a clear idea that the numbering will stay the same.
Quoted 89 lines
Quoted 67 lines
drivers/pci/controller/pci-hyperv.c +38
-0};#define CPU_AFFINITY_ALL -1ULL@@ -77,6 +78,7 @@ enum pci_protocol_version_t {* first. */ static enum pci_protocol_version_t pci_protocol_versions[] = { +PCI_PROTOCOL_VERSION_1_5 ,PCI_PROTOCOL_VERSION_1_4 ,PCI_PROTOCOL_VERSION_1_3 ,PCI_PROTOCOL_VERSION_1_2 ,@@ -90,6 +92,7 @@ static enum pci_protocol_version_t pci_protocol_versions[] = {#define MAX_SUPPORTED_MSI_MESSAGES 0x400#define STATUS_REVISION_MISMATCH 0xC0000059 +#define STATUS_NOT_SUPPORTED 0xC00000BB/* space for 32bit serial number as string */ #define SLOT_NAME_SIZE 11@@ -136,6 +139,7 @@ enum pci_message_type {PCI_BUS_RELATIONS2 =PCI_MESSAGE_BASE +0x19 ,PCI_RESOURCES_ASSIGNED3 =PCI_MESSAGE_BASE +0x1A ,PCI_CREATE_INTERRUPT_MESSAGE3 =PCI_MESSAGE_BASE +0x1B , +PCI_RESET_DEVICE =PCI_MESSAGE_BASE +0x1C , PCI_MESSAGE_MAXIMUM };@@ -1397,10 +1401,66 @@ static int hv_pcifront_write_config(struct pci_bus *bus,unsigned int devfn, return PCIBIOS_SUCCESSFUL; } +static int hv_pcifront_reset (struct pci_dev *pdev,bool probe) +{ +struct hv_pcibus_device *hbus = + container_of(pdev->bus->sysdata, struct hv_pcibus_device, sysdata); +struct pci_child_message reset = {}; +struct hv_pci_compl comp_pkt; +struct pci_packet pkt = { + .completion_func = hv_pci_generic_compl, + .compl_ctxt = &comp_pkt, + }; +enum hv_pcibus_state state; +int ret; + +/* Device reset was added in vPCI protocol version 1.5. */ +if (hbus->protocol_version <PCI_PROTOCOL_VERSION_1_5 ) +return -ENOTTY ; + +/* Hyper-V exposes projected functions directly on the root bus. */ +if (!pci_is_root_bus (pdev->bus)) +return -ENOTTY ; + +if (probe) +return 0 ; + +/* Do not take state_lock: eject holds it while removing/locking pdev. */ + state =READ_ONCE (hbus->state); +if (state != hv_pcibus_probed && state != hv_pcibus_installed) +return -ENODEV ; + +init_completion (&comp_pkt.host_event); + reset.message_type.type =PCI_RESET_DEVICE ; + reset.wslot.slot =devfn_to_wslot (pdev->devfn); + + ret = vmbus_sendpacket(hbus->hdev->channel, &reset, sizeof(reset), + (unsigned long)&pkt, VM_PKT_DATA_INBAND,drivers/pci/controller/pci-hyperv.c static int hv_pcifront_write_config(struct pci_bus *bus, · line 1449
Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +10
-0+ VMBUS_DATA_PACKET_FLAG_COMPLETION_REQUESTED); +if (ret) +return ret; + + ret =wait_for_response (hbus->hdev, &comp_pkt.host_event); +if (ret) +return ret; + +if (comp_pkt.completion_status ==STATUS_NOT_SUPPORTED ) +return -ENOTTY ;I tried this patch series in a linux-next20260726 build, and running on a D16lds v6 VM in Azure. The host hypervisor version is 10.0.26100.1652-1-0, and the VM has a paravisor with HvLite.
This VM has an NVMe OS disk, two NVMe temp disks, and a MANA network controller. Absent this patch set, the temp disks and MANA report the "
reset_method" as "flr", while the NVMe OS disk reports no reset methods. With this patch set, "controller" is added as a reset method for all. The NVMe disks and MANA network controller are probed with PCI protocol 1.5. That's all good and as expected (though I'm not sure why the NVMe OS disk doesn't support flr).
Quoted 6 lines
I then did "echo 1 >reset" for the NVMe OS disk. This returns a
-ENOTTYerror from the above line of code. So the host hypervisor (or paravisor?) is saying that the newPCI_RESET_DEVICEmessage isn't supported. Presumably this is new functionality that hasn’t been rolled out to where I'm running this Azure VM. That's fine too.
That's right. It is not yet rolled out.
Quoted 23 lines
But interestingly, the NVMe OS disk did a Linux-side reset anyway. That's because of this stack trace from the "echo 1 >reset" command:
[ 99.642483] nvme_try_sched_reset+0x25/0x60 [nvme_core] [ 99.642495] nvme_reset_done+0x1c/0x40 [nvme] [ 99.642499] pci_dev_restore+0x35/0x70 [ 99.642503] pci_reset_function+0x100/0x140 [ 99.642506] reset_store+0x5a/0xa0 [ 99.642508] dev_attr_store+0x16/0x30 [ 99.642512] sysfs_kf_write+0x71/0x80 [ 99.642516] kernfs_fop_write_iter+0x140/0x1d0 [ 99.642518] vfs_write+0x313/0x420 [ 99.642522] ksys_write+0x68/0xe0 [ 99.642525] __x64_sys_write+0x18/0x20 [ 99.642527] x64_sys_call+0x1700/0x21c0 [ 99.642530] do_syscall_64+0x8d/0x440 [ 99.642532] entry_SYSCALL_64_after_hwframe+0x76/0x7eEven though
__pci_reset_function_locked()failed, the subsequent code inpci_reset_function()callsnvme_try_sched_reset(), which putsnvme_reset_work()on a workqueue.nvme_reset_work()tears things down on the Linux side and rebuilds, and the message
Quoted 7 lines
nvme nvme0: 16/0/0 default/read/poll queues
is output.
It's not immediately clear to me how to resolve this issue, so I'm just pointing it out. :-(
Is resetting the OS disk a real use case?
I'm not sure what magic is happening behind this.
Wei
Quoted 33 lines
Michael
Quoted 30 lines
drivers/pci/controller/pci-hyperv.c +11
-0+ +if (comp_pkt.completion_status) { +pci_err (pdev,"Hyper-V device reset failed: %#x\n" , + comp_pkt.completion_status); +return -EIO ; + } + +return 0 ; +} +/* PCIe operations */ static struct pci_ops hv_pcifront_ops = { .read = hv_pcifront_read_config, .write = hv_pcifront_write_config, + .reset = hv_pcifront_reset, };/*
drivers/pci/controller/pci-hyperv.c +1
-0@@ -1996,6 +2056,7 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg) break;case PCI_PROTOCOL_VERSION_1_4 : +case PCI_PROTOCOL_VERSION_1_5 : size = hv_compose_msi_req_v3(&ctxt.int_pkts.v3, cpu,
hpdev->desc.win_slot.slot,
--
2.53.0
RE: [PATCH RFC 2/2] PCI: hv: Add vPCI device reset support
From: Wei Liu <[email protected]> Sent: Tuesday, August 11, 2026 10:37 AM
Quoted 170 lines · Michael Kelley
Quoted 33 lines
From: [email protected] <[email protected]> Sent: Friday, July 24, 2026 4:09 PM
Quoted 32 lines
Hyper-V vPCI protocol version 1.5 adds a
RESET_DEVICErequest for projected PCI functions. Negotiate the new protocol version and issue the request through the vPCI VMBus channel from the PCI controller reset callback.Use the existing VMBus response path. Return
-ENOTTYwhen the host reportsSTATUS_NOT_SUPPORTEDso PCI core may try another reset method.Signed-off-by Wei Liu <[email protected]>--- Documentation/virt/hyperv/vpci.rst | 2 +- drivers/pci/controller/pci-hyperv.c | 61 +++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+), 1 deletion(-)Documentation/virt/hyperv/vpci.rst +1
-1@@ -65,7 +65,7 @@ exchange messages with the vPCI VSP for the purpose of setting up and configuring the vPCI device in Linux. Once the device is fully configured in Linux as a PCI device, the VMBus channel is used only if Linux changes the vCPU to be interrupted -in the guest, or if the vPCI device is removed from +in the guest, or if the vPCI device is reset or removed from the VM while the VM is running. The ongoing operation of the device happens directly between the Linux device driver for the device and the hardware, with VMBus and the VMBus channeldrivers/pci/controller/pci-hyperv.c +0
-0@@ -68,6 +68,7 @@ enum pci_protocol_version_t {Quoted 9 lines
drivers/pci/controller/pci-hyperv.c enum pci_protocol_version_t { · line 71
Quoted 4 lines
drivers/pci/controller/pci-hyperv.c +1
-0PCI_PROTOCOL_VERSION_1_2 =PCI_MAKE_VERSION (1 ,2 ),/* RS1 */ PCI_PROTOCOL_VERSION_1_3 =PCI_MAKE_VERSION (1 ,3 ),/* Vibranium */ PCI_PROTOCOL_VERSION_1_4 =PCI_MAKE_VERSION (1 ,4 ),/* WS2022 */ + PCI_PROTOCOL_VERSION_1_5 = PCI_MAKE_VERSION(1, 5), /* GE, device reset */I wish there were a better way to identify the Hyper-V version than internal code names that mean nothing to the Linux community. "GE" refers to Germanium, which would be the Build 26100 series, right?
Yes, you're right about Germanium. I see the numbers also match, though I don't have a clear idea that the numbering will stay the same.
Quoted 88 lines
Quoted 66 lines
drivers/pci/controller/pci-hyperv.c +38
-0};#define CPU_AFFINITY_ALL -1ULL@@ -77,6 +78,7 @@ enum pci_protocol_version_t {* first. */ static enum pci_protocol_version_t pci_protocol_versions[] = { +PCI_PROTOCOL_VERSION_1_5 ,PCI_PROTOCOL_VERSION_1_4 ,PCI_PROTOCOL_VERSION_1_3 ,PCI_PROTOCOL_VERSION_1_2 ,@@ -90,6 +92,7 @@ static enum pci_protocol_version_t pci_protocol_versions[] = {#define MAX_SUPPORTED_MSI_MESSAGES 0x400#define STATUS_REVISION_MISMATCH 0xC0000059 +#define STATUS_NOT_SUPPORTED 0xC00000BB/* space for 32bit serial number as string */ #define SLOT_NAME_SIZE 11@@ -136,6 +139,7 @@ enum pci_message_type {PCI_BUS_RELATIONS2 =PCI_MESSAGE_BASE +0x19 ,PCI_RESOURCES_ASSIGNED3 =PCI_MESSAGE_BASE +0x1A ,PCI_CREATE_INTERRUPT_MESSAGE3 =PCI_MESSAGE_BASE +0x1B , +PCI_RESET_DEVICE =PCI_MESSAGE_BASE +0x1C , PCI_MESSAGE_MAXIMUM };@@ -1397,10 +1401,66 @@ static int hv_pcifront_write_config(struct pci_bus *bus, unsigned int devfn,return PCIBIOS_SUCCESSFUL ; } +static int hv_pcifront_reset (struct pci_dev *pdev,bool probe) +{ +struct hv_pcibus_device *hbus = + container_of(pdev->bus->sysdata, struct hv_pcibus_device, sysdata); +struct pci_child_message reset = {}; +struct hv_pci_compl comp_pkt; +struct pci_packet pkt = { + .completion_func = hv_pci_generic_compl, + .compl_ctxt = &comp_pkt, + }; +enum hv_pcibus_state state; +int ret; + +/* Device reset was added in vPCI protocol version 1.5. */ +if (hbus->protocol_version <PCI_PROTOCOL_VERSION_1_5 ) +return -ENOTTY ; + +/* Hyper-V exposes projected functions directly on the root bus. */ +if (!pci_is_root_bus (pdev->bus)) +return -ENOTTY ; + +if (probe) +return 0 ; + +/* Do not take state_lock: eject holds it while removing/locking pdev. */ + state =READ_ONCE (hbus->state); +if (state != hv_pcibus_probed && state != hv_pcibus_installed) +return -ENODEV ; + +init_completion (&comp_pkt.host_event); + reset.message_type.type =PCI_RESET_DEVICE ; + reset.wslot.slot =devfn_to_wslot (pdev->devfn); + + ret = vmbus_sendpacket(hbus->hdev->channel, &reset, sizeof(reset), + (unsigned long)&pkt, VM_PKT_DATA_INBAND,drivers/pci/controller/pci-hyperv.c static int hv_pcifront_write_config(struct pci_bus *bus, unsigned int devfn, · line 1448
Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +10
-0+ VMBUS_DATA_PACKET_FLAG_COMPLETION_REQUESTED); +if (ret) +return ret; + + ret =wait_for_response (hbus->hdev, &comp_pkt.host_event); +if (ret) +return ret; + +if (comp_pkt.completion_status ==STATUS_NOT_SUPPORTED ) +return -ENOTTY ;I tried this patch series in a linux-next20260726 build, and running on a D16lds v6 VM in Azure. The host hypervisor version is 10.0.26100.1652-1-0, and the VM has a paravisor with HvLite.
This VM has an NVMe OS disk, two NVMe temp disks, and a MANA network controller. Absent this patch set, the temp disks and MANA report the "
reset_method" as "flr", while the NVMe OS disk reports no reset methods. With this patch set, "controller" is added as a reset method for all. The NVMe disks and MANA network controller are probed with PCI protocol 1.5. That's all good and as expected (though I'm not sure why the NVMe OS disk doesn't support flr).Quoted 6 lines
I then did "echo 1 >reset" for the NVMe OS disk. This returns a
-ENOTTYerror from the above line of code. So the host hypervisor (or paravisor?) is saying that the newPCI_RESET_DEVICEmessage isn't supported. Presumably this is new functionality that hasn’t been rolled out to where I'm running this Azure VM. That's fine too.That's right. It is not yet rolled out.
Quoted 25 lines
But interestingly, the NVMe OS disk did a Linux-side reset anyway. That's because of this stack trace from the "echo 1 >reset" command:
[ 99.642483] nvme_try_sched_reset+0x25/0x60 [nvme_core] [ 99.642495] nvme_reset_done+0x1c/0x40 [nvme] [ 99.642499] pci_dev_restore+0x35/0x70 [ 99.642503] pci_reset_function+0x100/0x140 [ 99.642506] reset_store+0x5a/0xa0 [ 99.642508] dev_attr_store+0x16/0x30 [ 99.642512] sysfs_kf_write+0x71/0x80 [ 99.642516] kernfs_fop_write_iter+0x140/0x1d0 [ 99.642518] vfs_write+0x313/0x420 [ 99.642522] ksys_write+0x68/0xe0 [ 99.642525] __x64_sys_write+0x18/0x20 [ 99.642527] x64_sys_call+0x1700/0x21c0 [ 99.642530] do_syscall_64+0x8d/0x440 [ 99.642532] entry_SYSCALL_64_after_hwframe+0x76/0x7eEven though
__pci_reset_function_locked()failed, the subsequent code inpci_reset_function()callsnvme_try_sched_reset(), which putsnvme_reset_work()on a workqueue.nvme_reset_work()tears things down on the Linux side and rebuilds, and the messagenvme nvme0: 16/0/0 default/read/poll queues
Quoted 7 lines
Quoted 5 lines
is output.
It's not immediately clear to me how to resolve this issue, so I'm just pointing it out. :-(
Is resetting the OS disk a real use case?
Perhaps not. However, the same behavior occurs with the NVMe temp disks if "flr" is removed from "reset_modes" so that the "controller" method is used. I was just trying to see how this new controller method behaves, and the result seemed anomalous. Using the "controller" method against the MANA device (again presumably with a "not supported" reply from Hyper-V) produced a hung MANA that I couldn't recover without rebooting the VM. I did not try to sort out what when wrong there.
Quoted 3 lines
I'm not sure what magic is happening behind this.
FWIW, I don't think anything is happening on the Hyper-V host side. Hyper-V says PCI_RESET_DEVICE isn't supported and then just goes on normally. But the Linux path through pci_reset_function() and the NVMe driver does its reset handling even if none of the available reset methods succeed. Arguably that's independent of your patch set, though the existence of Hyper-V versions that support vPCI protocol 1.5 but not PCI_RESET_DEVICE could make the problem more likely.
Michael
RE: [EXTERNAL] [PATCH RFC 2/2] PCI: hv: Add vPCI device reset support
In reply to [email protected]Quoted 20 lines
-----Original Message-----
From: [email protected] <[email protected]>
Sent: Friday, July 24, 2026 4:09 PM
To: Linux on Hyper-V List <[email protected]>; linux- [email protected]
Cc: Wei Liu <[email protected]>; KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; Dexuan Cui <[email protected]>; Long Li <[email protected]>; Jonathan Corbet <[email protected]>; Shuah Khan <[email protected]>; Lorenzo Pieralisi <[email protected]>; Krzysztof Wilczyński <[email protected]>; Manivannan Sadhasivam <[email protected]>; Rob Herring <[email protected]>; Bjorn Helgaas <[email protected]>; open list:DOCUMENTATION <[email protected]>; open list <linux- [email protected]>
Subject: [EXTERNAL] [PATCH RFC 2/2] PCI: hv: Add vPCI device reset supportFrom: Wei Liu <[email protected]>
Quoted 8 lines
Hyper-V vPCI protocol version 1.5 adds a
RESET_DEVICErequest for projected PCI functions. Negotiate the new protocol version and issue the request through the vPCI VMBus channel from the PCI controller reset callback.Use the existing VMBus response path. Return
-ENOTTYwhen the host reportsSTATUS_NOT_SUPPORTEDso PCI core may try another reset method.Signed-off-by Wei Liu <[email protected]>
Quoted 139 lines
---
Documentation/virt/hyperv/vpci.rst| 2 +-drivers/pci/controller/pci-hyperv.c| 61 +++++++++++++++++++++++++++++2 files changed, 62 insertions(+), 1 deletion(-)Documentation/virt/hyperv/vpci.rst +0
-0@@ -65,7 +65,7 @@ exchange messages with the vPCI VSP for the purposeof setting up and configuring the vPCI device in Linux. Once the device is fully configured in Linux as a PCI device, the VMBus channel is used only if Linux changes the vCPU to be interrupted -in the guest, or if the vPCI device is removed from
+in the guest, or if the vPCI device is reset or removed from
the VM while the VM is running. The ongoing operation of the device
happens directly between the Linux device driver for the device and the hardware, with VMBus and the VMBus channel diff --gita/drivers/pci/controller/pci-hyperv.cb/drivers/pci/controller/pci-hyperv.cindexcfc8fa403dad..d3c0fd5e1d8e100644drivers/pci/controller/pci-hyperv.c +0
-0@@ -68,6 +68,7 @@ enum pci_protocol_version_t { PCI_PROTOCOL_VERSION_1_2 = PCI_MAKE_VERSION(1, 2), /*RS1 */
PCI_PROTOCOL_VERSION_1_3=PCI_MAKE_VERSION(1, 3), /*
Vibranium */
PCI_PROTOCOL_VERSION_1_4=PCI_MAKE_VERSION(1, 4), /*
WS2022 */drivers/pci/controller/pci-hyperv.c +1
-0+ PCI_PROTOCOL_VERSION_1_5 = PCI_MAKE_VERSION(1, 5), /* GE,device reset */
};#define
CPU_AFFINITY_ALL-1ULLdrivers/pci/controller/pci-hyperv.c +1
-0@@ -77,6 +78,7 @@ enum pci_protocol_version_t {* first. */ static enum pci_protocol_version_t pci_protocol_versions[] = { +PCI_PROTOCOL_VERSION_1_5 ,PCI_PROTOCOL_VERSION_1_4 ,PCI_PROTOCOL_VERSION_1_3 ,PCI_PROTOCOL_VERSION_1_2 ,@@ -90,6 +92,7 @@ static enum pci_protocol_version_tpci_protocol_versions[] = { #define
MAX_SUPPORTED_MSI_MESSAGES0x400#define
STATUS_REVISION_MISMATCH0xC0000059drivers/pci/controller/pci-hyperv.c +58
-0+#define STATUS_NOT_SUPPORTED 0xC00000BB/* space for 32bit serial number as string */ #define SLOT_NAME_SIZE 11@@ -136,6 +139,7 @@ enum pci_message_type {PCI_BUS_RELATIONS2 =PCI_MESSAGE_BASE +0x19 ,PCI_RESOURCES_ASSIGNED3 =PCI_MESSAGE_BASE +0x1A ,PCI_CREATE_INTERRUPT_MESSAGE3 =PCI_MESSAGE_BASE +0x1B , +PCI_RESET_DEVICE =PCI_MESSAGE_BASE +0x1C , PCI_MESSAGE_MAXIMUM };@@ -1397,10 +1401,66 @@ static int hv_pcifront_write_config(struct pci_bus *bus, unsigned int devfn, return PCIBIOS_SUCCESSFUL; } +static int hv_pcifront_reset (struct pci_dev *pdev,bool probe) { +struct hv_pcibus_device *hbus = + container_of(pdev->bus->sysdata, struct hv_pcibus_device, sysdata); +struct pci_child_message reset = {}; +struct hv_pci_compl comp_pkt; +struct pci_packet pkt = { + .completion_func = hv_pci_generic_compl, + .compl_ctxt = &comp_pkt, + }; +enum hv_pcibus_state state; +int ret; + +/* Device reset was added in vPCI protocol version 1.5. */ +if (hbus->protocol_version <PCI_PROTOCOL_VERSION_1_5 ) +return -ENOTTY ; + +/* Hyper-V exposes projected functions directly on the root bus. */ +if (!pci_is_root_bus (pdev->bus)) +return -ENOTTY ; + +if (probe) +return 0 ; + +/* Do not take state_lock: eject holds it while removing/locking pdev. */ + state =READ_ONCE (hbus->state); +if (state != hv_pcibus_probed && state != hv_pcibus_installed) +return -ENODEV ; + +init_completion (&comp_pkt.host_event); + reset.message_type.type =PCI_RESET_DEVICE ; + reset.wslot.slot =devfn_to_wslot (pdev->devfn); + + ret =vmbus_sendpacket (hbus->hdev->channel, &reset,sizeof (reset), + (unsigned long )&pkt,VM_PKT_DATA_INBAND , +VMBUS_DATA_PACKET_FLAG_COMPLETION_REQUESTED ); +if (ret) +return ret; + + ret =wait_for_response (hbus->hdev, &comp_pkt.host_event); +if (ret) +return ret; + +if (comp_pkt.completion_status ==STATUS_NOT_SUPPORTED ) +return -ENOTTY ; + +if (comp_pkt.completion_status) { +pci_err (pdev,"Hyper-V device reset failed: %#x\n" , + comp_pkt.completion_status); +return -EIO ; + } + +return 0 ; +} +/* PCIe operations */ static struct pci_ops hv_pcifront_ops = { .read = hv_pcifront_read_config, .write = hv_pcifront_write_config, + .reset = hv_pcifront_reset, };/* @@ -1996,6 +2056,7 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)break ;case PCI_PROTOCOL_VERSION_1_4 : +case PCI_PROTOCOL_VERSION_1_5 : size = hv_compose_msi_req_v3(&ctxt.int_pkts.v3, cpu,
hpdev->desc.win_slot.slot,
--
2.53.0
Wei Liu
maintainerRe: [EXTERNAL] [PATCH RFC 2/2] PCI: hv: Add vPCI device reset support
Quoted 24 lines · Long Li
Quoted 24 lines
-----Original Message-----
From: [email protected] <[email protected]>
Sent: Friday, July 24, 2026 4:09 PM
To: Linux on Hyper-V List <[email protected]>; linux- [email protected]
Cc: Wei Liu <[email protected]>; KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; Dexuan Cui <[email protected]>; Long Li <[email protected]>; Jonathan Corbet <[email protected]>; Shuah Khan <[email protected]>; Lorenzo Pieralisi <[email protected]>; Krzysztof Wilczyński <[email protected]>; Manivannan Sadhasivam <[email protected]>; Rob Herring <[email protected]>; Bjorn Helgaas <[email protected]>; open list:DOCUMENTATION <[email protected]>; open list <linux- [email protected]>
Subject: [EXTERNAL] [PATCH RFC 2/2] PCI: hv: Add vPCI device reset supportFrom: Wei Liu <[email protected]>
Hyper-V vPCI protocol version 1.5 adds a
RESET_DEVICErequest for projected PCI functions. Negotiate the new protocol version and issue the request through the vPCI VMBus channel from the PCI controller reset callback.
Quoted 6 lines
Quoted 4 lines
Use the existing VMBus response path. Return
-ENOTTYwhen the host reportsSTATUS_NOT_SUPPORTEDso PCI core may try another reset method.Signed-off-by Wei Liu <[email protected]>Reviewed-by Long Li <[email protected]>
Thanks for the review!
I still need to find some copious free time to test the new hook suggested in patch one.
I don't think changes in patch 1 will result in major changes in this patch, so I intend to retain this tag when I post next time.
Thanks,
Wei