vfio/pci: expose device's PASID capability to VMs
Subsystems (2)
cover vfio/pci: expose device's PASID capability to VMs
From: Liu Yi L <[email protected]>
Shared Virtual Addressing (SVA), a.k.a, Shared Virtual Memory (SVM) on Intel platforms allows address space sharing between device DMA and applications. SVA can reduce programming complexity and enhance security.
To enable SVA, device needs to have PASID capability, which is a key capability for SVA. This patchset exposes the device's PASID capability to guest instead of hiding it from guest.
The second patch emulates PASID capability for VFs (Virtual Function) since VFs don't implement such capability per PCIe spec. This patch emulates such capability and expose to VM if the capability is enabled in PF (Physical Function).
However, there is an open for PASID emulation. If PF driver disables PASID capability at runtime, then it may be an issue. e.g. PF should not disable PASID capability if there is guest using this capability on any VF related to this PF. To solve it, may need to introduce a generic communication framework between vfio-pci driver and PF drivers. Please feel free to give your suggestions on it.
Regards,
Yi Liu
Changelog:
- RFC v1 -> Patch v1:
Add CONFIG_PCI_ATS #ifdef control to avoid compiling error.
Liu Yi L (2):
vfio/pci: Expose PCIe PASID capability to guest
vfio/pci: Emulate PASID/PRI capability for VFs
drivers/vfio/pci/vfio_pci_config.c | 327 ++++++++++++++++++++++++++++++++++++- 1 file changed, 324 insertions(+), 3 deletions(-)
Signature · 6 lines
2.7.4 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
PATCH 1/2 vfio/pci: Expose PCIe PASID capability to guest
From: Liu Yi L <[email protected]>
This patch exposes PCIe PASID capability to guest. Existing vfio_pci driver hides it from guest by setting the capability length as 0 in pci_ext_cap_length[].
This capability is required for vSVA enabling on pass-through PCIe devices.
--- drivers/vfio/pci/vfio_pci_config.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/vfio/pci/vfio_pci_config.c +1 -1
@@ -95,7 +95,7 @@ static const u16 pci_ext_cap_length[PCI_EXT_CAP_ID_MAX + 1] = { [PCI_EXT_CAP_ID_LTR] = PCI_EXT_CAP_LTR_SIZEOF, [PCI_EXT_CAP_ID_SECPCI] = 0, /* not yet */ [PCI_EXT_CAP_ID_PMUX] = 0, /* not yet */ - [PCI_EXT_CAP_ID_PASID] = 0, /* not yet */ + [PCI_EXT_CAP_ID_PASID] = PCI_EXT_CAP_PASID_SIZEOF, };/*
Signature · 1 line
2.7.4
RE: [PATCH v1 1/2] vfio/pci: Expose PCIe PASID capability to guest
Quoted 11 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:33 PMFrom: Liu Yi L <[email protected]>
This patch exposes PCIe PASID capability to guest. Existing
vfio_pcidriver hides it from guest by setting the capability length as 0 in pci_ext_cap_length[].This capability is required for vSVA enabling on pass-through PCIe devices.
should this be [PATCH 2/2], after you have the emulation in place?
and it might be worthy of noting that PRI is already exposed, to avoid confusion from one like me that why two capabilities are emulated in this series while only one is being exposed. 😊
Quoted 28 lines
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/pci/vfio_pci_config.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/vfio/pci/vfio_pci_config.c +1
-1@@ -95,7 +95,7 @@ static const u16 pci_ext_cap_length[PCI_EXT_CAP_ID_MAX + 1] = { [PCI_EXT_CAP_ID_LTR] = PCI_EXT_CAP_LTR_SIZEOF, [PCI_EXT_CAP_ID_SECPCI] = 0, /* not yet */ [PCI_EXT_CAP_ID_PMUX] = 0, /* not yet */ - [PCI_EXT_CAP_ID_PASID] = 0, /* not yet */ + [PCI_EXT_CAP_ID_PASID] = PCI_EXT_CAP_PASID_SIZEOF, };/*
--
2.7.4
RE: [PATCH v1 1/2] vfio/pci: Expose PCIe PASID capability to guest
Quoted 10 lines
From: Tian, Kevin <[email protected]>
Sent: Tuesday, March 31, 2020 2:39 PM
To: Liu, Yi L <[email protected]>; [email protected];
Subject: RE: [PATCH v1 1/2] vfio/pci: Expose PCIe PASID capability to guestQuoted 5 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:33 PMFrom: Liu Yi L <[email protected]>
Quoted 8 lines
Quoted 6 lines
This patch exposes PCIe PASID capability to guest. Existing
vfio_pcidriver hides it from guest by setting the capability length as 0 in pci_ext_cap_length[].This capability is required for vSVA enabling on pass-through PCIe devices.
should this be [PATCH 2/2], after you have the emulation in place?
oh, yes, I can re-sequence it.
Quoted 3 lines
and it might be worthy of noting that PRI is already exposed, to avoid confusion from one like me that why two capabilities are emulated in this series while only one is being exposed. 😊
got it. it would be helpful. thanks.
Regards,
Yi Liu
PATCH 2/2 vfio/pci: Emulate PASID/PRI capability for VFs
In reply to Liu, Yi LFrom: Liu Yi L <[email protected]>
Per PCIe r5.0, sec 9.3.7.14, if a PF implements the PASID Capability, the PF PASID configuration is shared by its VFs. VFs must not implement their own PASID Capability.
Per PCIe r5.0, sec 9.3.7.11, VFs must not implement the PRI Capability. If the PF implements PRI, it is shared by the VFs.
On bare metal, it has been fixed by below efforts. to PASID/PRI are
https://lkml.org/lkml/2019/9/5/996
https://lkml.org/lkml/2019/9/5/995
This patch adds emulated PASID/PRI capabilities for VFs when assigned to VMs via vfio-pci driver. This is required for enabling vSVA on pass-through VFs as VFs have no PASID/PRI capability structure in its configure space.
--- drivers/vfio/pci/vfio_pci_config.c | 325 ++++++++++++++++++++++++++++++++++++- 1 file changed, 323 insertions(+), 2 deletions(-)
drivers/vfio/pci/vfio_pci_config.c +323 -2
@@ -1509,11 +1509,304 @@ static int vfio_cap_init(struct vfio_pci_device *vdev)return 0 ; } +static int vfio_fill_custom_vconfig_bytes (struct vfio_pci_device *vdev, +int offset,uint8_t *data,int size) +{ +int ret =0 , data_offset =0 ; + +while (size) { +int filled; + +if (size >=4 && !(offset %4 )) { +__le32 *dwordp = (__le32 *)&vdev->vconfig[offset]; +u32 dword; + +memcpy (&dword, data + data_offset,4 ); + *dwordp =cpu_to_le32 (dword); + filled =4 ; + }else if (size >=2 && !(offset %2 )) { +__le16 *wordp = (__le16 *)&vdev->vconfig[offset]; +u16 word; + +memcpy (&word, data + data_offset,2 ); + *wordp =cpu_to_le16 (word); + filled =2 ; + }else { +u8 *byte = &vdev->vconfig[offset]; + +memcpy (byte, data + data_offset,1 ); + filled =1 ; + } + + offset += filled; + data_offset += filled; + size -= filled; + } + +return ret; +} + +static int vfio_pci_get_ecap_content (struct pci_dev *pdev, +int cap,int cap_len,u8 *content) +{ +int pos, offset, len = cap_len, ret =0 ; + + pos =pci_find_ext_capability (pdev, cap); +if (!pos) +return -EINVAL ; + + offset =0 ; +while (len) { +int fetched; + +if (len >=4 && !(pos %4 )) { +u32 *dwordp = (u32 *) (content + offset); +u32 dword; +__le32 *dwptr = (__le32 *) &dword; + + ret =pci_read_config_dword (pdev, pos, &dword); +if (ret) +return ret; + *dwordp =le32_to_cpu (*dwptr); + fetched =4 ; + }else if (len >=2 && !(pos %2 )) { +u16 *wordp = (u16 *) (content + offset); +u16 word; +__le16 *wptr = (__le16 *) &word; + + ret =pci_read_config_word (pdev, pos, &word); +if (ret) +return ret; + *wordp =le16_to_cpu (*wptr); + fetched =2 ; + }else { +u8 *byte = (u8 *) (content + offset); + + ret =pci_read_config_byte (pdev, pos, byte); +if (ret) +return ret; + fetched =1 ; + } + + pos += fetched; + offset += fetched; + len -= fetched; + } + +return ret; +} + +struct vfio_pci_pasid_cap_data { +u32 id:16 ; +u32 version:4 ; +u32 next:12 ; +union { +u16 cap_reg_val; +struct { +u16 rsv1:1 ; +u16 execs:1 ; +u16 prvs:1 ; +u16 rsv2:5 ; +u16 pasid_bits:5 ; +u16 rsv3:3 ; + }; + } cap_reg; +union { +u16 control_reg_val; +struct { +u16 paside:1 ; +u16 exece:1 ; +u16 prve:1 ; +u16 rsv4:13 ; + }; + } control_reg; +}; + +static int vfio_pci_add_pasid_cap (struct vfio_pci_device *vdev, +struct pci_dev *pdev, +u16 epos,u16 *next,__le32 **prevp) +{ +u8 *map = vdev->pci_config_map; +int ecap =PCI_EXT_CAP_ID_PASID ; +int len = pci_ext_cap_length[ecap]; +struct vfio_pci_pasid_cap_data pasid_cap; +struct vfio_pci_pasid_cap_data vpasid_cap; +int ret; + +/* +* If no cap filled in this function, should make sure the next +* pointer points to current epos. +*/ + *next = epos; + +if (!len) { +pr_info ("%s: VF %s hiding PASID capability\n" , + __func__,dev_name (&vdev->pdev->dev)); + ret =0 ; + goto out; + } + +/* Add PASID capability*/ + ret =vfio_pci_get_ecap_content (pdev, ecap, + len, (u8 *)&pasid_cap); +if (ret) + goto out; + +if (!pasid_cap.control_reg.paside) { +pr_debug ("%s: its PF's PASID capability is not enabled\n" , +dev_name (&vdev->pdev->dev)); + ret =0 ; + goto out; + } + +memcpy (&vpasid_cap, &pasid_cap, len); + + vpasid_cap.id =0x18 ; + vpasid_cap.next =0 ; +/* clear the control reg for guest */ +memset (&vpasid_cap.control_reg,0x0 , +sizeof (vpasid_cap.control_reg)); + +memset (map + epos, vpasid_cap.id, len); + ret =vfio_fill_custom_vconfig_bytes (vdev, epos, + (u8 *)&vpasid_cap, len); +if (!ret) { +/* +* Successfully filled in PASID cap, update +* the next offset in previous cap header, +* and also update caller about the offset +* of next cap if any. +*/ +u32 val = epos; + **prevp &=cpu_to_le32 (~(0xffcU <<20 )); + **prevp |=cpu_to_le32 (val <<20 ); + *prevp = (__le32 *)&vdev->vconfig[epos]; + *next = epos + len; + } + +out: +return ret; +} + +struct vfio_pci_pri_cap_data { +u32 id:16 ; +u32 version:4 ; +u32 next:12 ; +union { +u16 control_reg_val; +struct { +u16 enable:1 ; +u16 reset:1 ; +u16 rsv1:14 ; + }; + } control_reg; +union { +u16 status_reg_val; +struct { +u16 rf:1 ; +u16 uprgi:1 ; +u16 rsv2:6 ; +u16 stop:1 ; +u16 rsv3:6 ; +u16 pasid_required:1 ; + }; + } status_reg; +u32 prq_capacity; +u32 prq_quota; +}; + +static int vfio_pci_add_pri_cap (struct vfio_pci_device *vdev, +struct pci_dev *pdev, +u16 epos,u16 *next,__le32 **prevp) +{ +u8 *map = vdev->pci_config_map; +int ecap =PCI_EXT_CAP_ID_PRI ; +int len = pci_ext_cap_length[ecap]; +struct vfio_pci_pri_cap_data pri_cap; +struct vfio_pci_pri_cap_data vpri_cap; +int ret; + +/* +* If no cap filled in this function, should make sure the next +* pointer points to current epos. +*/ + *next = epos; + +if (!len) { +pr_info ("%s: VF %s hiding PRI capability\n" , + __func__,dev_name (&vdev->pdev->dev)); + ret =0 ; + goto out; + } + +/* Add PASID capability*/ + ret =vfio_pci_get_ecap_content (pdev, ecap, + len, (u8 *)&pri_cap); +if (ret) + goto out; + +if (!pri_cap.control_reg.enable) { +pr_debug ("%s: its PF's PRI capability is not enabled\n" , +dev_name (&vdev->pdev->dev)); + ret =0 ; + goto out; + } + +memcpy (&vpri_cap, &pri_cap, len); + + vpri_cap.id =0x19 ; + vpri_cap.next =0 ; +/* clear the control reg for guest */ +memset (&vpri_cap.control_reg,0x0 , +sizeof (vpri_cap.control_reg)); + +memset (map + epos, vpri_cap.id, len); + ret =vfio_fill_custom_vconfig_bytes (vdev, epos, + (u8 *)&vpri_cap, len); +if (!ret) { +/* +* Successfully filled in PASID cap, update +* the next offset in previous cap header, +* and also update caller about the offset +* of next cap if any. +*/ +u32 val = epos; + **prevp &=cpu_to_le32 (~(0xffcU <<20 )); + **prevp |=cpu_to_le32 (val <<20 ); + *prevp = (__le32 *)&vdev->vconfig[epos]; + *next = epos + len; + } + +out: +return ret; +} + +static int vfio_pci_add_emulated_cap_for_vf (struct vfio_pci_device *vdev, +struct pci_dev *pdev,u16 start_epos,__le32 *prev) +{ +__le32 *__prev = prev; +u16 epos = start_epos, epos_next = start_epos; +int ret =0 ; + +/* Add PASID capability*/ + ret =vfio_pci_add_pasid_cap (vdev, pdev, epos, + &epos_next, &__prev); +if (ret) +return ret; + +/* Add PRI capability */ + epos = epos_next; + ret =vfio_pci_add_pri_cap (vdev, pdev, epos, + &epos_next, &__prev); + +return ret; +} +static int vfio_ecap_init (struct vfio_pci_device *vdev) {struct pci_dev *pdev = vdev->pdev;u8 *map = vdev->pci_config_map; -u16 epos; +u16 epos, epos_max;__le32 *prev =NULL ;int loops, ret, ecaps =0 ;
@@ -1521,6 +1814,7 @@ static int vfio_ecap_init(struct vfio_pci_device *vdev)return 0 ; epos =PCI_CFG_SPACE_SIZE ; + epos_max =PCI_CFG_SPACE_SIZE ; loops = (pdev->cfg_size -PCI_CFG_SPACE_SIZE ) /PCI_CAP_SIZEOF ;
@@ -1545,6 +1839,9 @@ static int vfio_ecap_init(struct vfio_pci_device *vdev) } } +if (epos_max <= (epos + len)) + epos_max = epos + len; +if (!len) {pci_info (pdev,"%s: hiding ecap %#x@%#x\n" , __func__, ecap, epos);
@@ -1604,6 +1901,18 @@ static int vfio_ecap_init(struct vfio_pci_device *vdev)if (!ecaps) *(u32 *)&vdev->vconfig[PCI_CFG_SPACE_SIZE ] =0 ; +#ifdef CONFIG_PCI_ATS +if (pdev->is_virtfn) { +struct pci_dev *physfn = pdev->physfn; + + ret =vfio_pci_add_emulated_cap_for_vf (vdev, + physfn, epos_max, prev); +if (ret) +pr_info ("%s, failed to add special caps for VF %s\n" , + __func__,dev_name (&vdev->pdev->dev)); + } +#endif +return 0 ; }
@@ -1748,6 +2057,17 @@ static size_t vfio_pci_cap_remaining_dword(struct vfio_pci_device *vdev,return i; } +static bool vfio_pci_need_virt_perm (struct pci_dev *pdev,u8 cap_id) +{ +#ifdef CONFIG_PCI_ATS +return (pdev->is_virtfn && + (cap_id ==PCI_EXT_CAP_ID_PASID || + cap_id ==PCI_EXT_CAP_ID_PRI )); +#else +return false; +#endif +} +static ssize_t vfio_config_do_rw (struct vfio_pci_device *vdev,char __user *buf,size_t count,loff_t *ppos,bool iswrite) {
@@ -1781,7 +2101,8 @@ static ssize_t vfio_config_do_rw(struct vfio_pci_device *vdev, char __user *buf,if (cap_id ==PCI_CAP_ID_INVALID ) { perm = &unassigned_perms; cap_start = *ppos; - }else if (cap_id ==PCI_CAP_ID_INVALID_VIRT ) { + }else if (cap_id ==PCI_CAP_ID_INVALID_VIRT || +vfio_pci_need_virt_perm (pdev, cap_id)) { perm = &virt_perms; cap_start = *ppos; }else {
Signature · 6 lines
2.7.4 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
Quoted 40 lines · Yi L
From: Liu Yi L <[email protected]>
Per PCIe r5.0, sec 9.3.7.14, if a PF implements the PASID Capability, the PF PASID configuration is shared by its VFs. VFs must not implement their own PASID Capability.
Per PCIe r5.0, sec 9.3.7.11, VFs must not implement the PRI Capability. If the PF implements PRI, it is shared by the VFs.
On bare metal, it has been fixed by below efforts. to PASID/PRI are
https://lkml.org/lkml/2019/9/5/996
https://lkml.org/lkml/2019/9/5/995This patch adds emulated PASID/PRI capabilities for VFs when assigned to VMs via vfio-pci driver. This is required for enabling vSVA on pass-through VFs as VFs have no PASID/PRI capability structure in its configure space.
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/pci/vfio_pci_config.c | 325 ++++++++++++++++++++++++++++++++++++- 1 file changed, 323 insertions(+), 2 deletions(-)drivers/vfio/pci/vfio_pci_config.c +4
-0@@ -1509,11 +1509,304 @@ static int vfio_cap_init(struct vfio_pci_device *vdev)return 0 ; } +static int vfio_fill_custom_vconfig_bytes (struct vfio_pci_device *vdev, +int offset,uint8_t *data,int size) +{ +int ret =0 , data_offset =0 ;
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1525
Quoted 10 lines
drivers/vfio/pci/vfio_pci_config.c +10
-0+ +while (size) { +int filled; + +if (size >=4 && !(offset %4 )) { +__le32 *dwordp = (__le32 *)&vdev->vconfig[offset]; +u32 dword; + +memcpy (&dword, data + data_offset,4 ); + *dwordp =cpu_to_le32 (dword);
Why wouldn't we do:
*dwordp = cpu_to_le32(*(u32 *)(data + data_offset));
or better yet, increment data on each iteration for:
*dwordp = cpu_to_le32(*(u32 *)data);
vfio_fill_vconfig_bytes() does almost this same thing, getting the data from config space rather than a buffer, so please figure out how to avoid duplicating the logic.
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1537
Quoted 12 lines
drivers/vfio/pci/vfio_pci_config.c +12
-0+ filled =4 ; + }else if (size >=2 && !(offset %2 )) { +__le16 *wordp = (__le16 *)&vdev->vconfig[offset]; +u16 word; + +memcpy (&word, data + data_offset,2 ); + *wordp =cpu_to_le16 (word); + filled =2 ; + }else { +u8 *byte = &vdev->vconfig[offset]; + +memcpy (byte, data + data_offset,1 );
This one is really silly.
vdev->vconfig[offset] = *data;
Quoted 23 lines
drivers/vfio/pci/vfio_pci_config.c +23
-0+ filled =1 ; + } + + offset += filled; + data_offset += filled; + size -= filled; + } + +return ret; +} + +static int vfio_pci_get_ecap_content (struct pci_dev *pdev, +int cap,int cap_len,u8 *content) +{ +int pos, offset, len = cap_len, ret =0 ; + + pos =pci_find_ext_capability (pdev, cap); +if (!pos) +return -EINVAL ; + + offset =0 ; +while (len) { +int fetched;
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1570
Quoted 10 lines
drivers/vfio/pci/vfio_pci_config.c +10
-0+ +if (len >=4 && !(pos %4 )) { +u32 *dwordp = (u32 *) (content + offset); +u32 dword; +__le32 *dwptr = (__le32 *) &dword; + + ret =pci_read_config_dword (pdev, pos, &dword); +if (ret) +return ret; + *dwordp =le32_to_cpu (*dwptr);
WHAT??? pci_read_config_dword() returns cpu endian data!
Quoted 67 lines
drivers/vfio/pci/vfio_pci_config.c +67
-0+ fetched =4 ; + }else if (len >=2 && !(pos %2 )) { +u16 *wordp = (u16 *) (content + offset); +u16 word; +__le16 *wptr = (__le16 *) &word; + + ret =pci_read_config_word (pdev, pos, &word); +if (ret) +return ret; + *wordp =le16_to_cpu (*wptr); + fetched =2 ; + }else { +u8 *byte = (u8 *) (content + offset); + + ret =pci_read_config_byte (pdev, pos, byte); +if (ret) +return ret; + fetched =1 ; + } + + pos += fetched; + offset += fetched; + len -= fetched; + } + +return ret; +} + +struct vfio_pci_pasid_cap_data { +u32 id:16 ; +u32 version:4 ; +u32 next:12 ; +union { +u16 cap_reg_val; +struct { +u16 rsv1:1 ; +u16 execs:1 ; +u16 prvs:1 ; +u16 rsv2:5 ; +u16 pasid_bits:5 ; +u16 rsv3:3 ; + }; + } cap_reg; +union { +u16 control_reg_val; +struct { +u16 paside:1 ; +u16 exece:1 ; +u16 prve:1 ; +u16 rsv4:13 ; + }; + } control_reg; +}; + +static int vfio_pci_add_pasid_cap (struct vfio_pci_device *vdev, +struct pci_dev *pdev, +u16 epos,u16 *next,__le32 **prevp) +{ +u8 *map = vdev->pci_config_map; +int ecap =PCI_EXT_CAP_ID_PASID ; +int len = pci_ext_cap_length[ecap]; +struct vfio_pci_pasid_cap_data pasid_cap; +struct vfio_pci_pasid_cap_data vpasid_cap; +int ret; + +/* +* If no cap filled in this function, should make sure the next
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1647
Quoted 10 lines
drivers/vfio/pci/vfio_pci_config.c +10
-0+* pointer points to current epos. +*/ + *next = epos; + +if (!len) { +pr_info ("%s: VF %s hiding PASID capability\n" , + __func__,dev_name (&vdev->pdev->dev)); + ret =0 ; + goto out; + }
No! Why? This is dead code.
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1651
Quoted 4 lines
drivers/vfio/pci/vfio_pci_config.c +4
-0+ +/* Add PASID capability*/ + ret =vfio_pci_get_ecap_content (pdev, ecap, + len, (u8 *)&pasid_cap);
Why wouldn't we just use vfio_fill_config_bytes() rather than this ridiculous approach of coping it to a buffer, modifying it and copying it out?
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1660
Quoted 9 lines
drivers/vfio/pci/vfio_pci_config.c +9
-0+if (ret) + goto out; + +if (!pasid_cap.control_reg.paside) { +pr_debug ("%s: its PF's PASID capability is not enabled\n" , +dev_name (&vdev->pdev->dev)); + ret =0 ; + goto out; + }
What happens if the PF's PASID gets disabled while we're using it??
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1664
Quoted 4 lines
drivers/vfio/pci/vfio_pci_config.c +4
-0+ +memcpy (&vpasid_cap, &pasid_cap, len); + + vpasid_cap.id =0x18 ;
What is going on here?
#define PCI_EXT_CAP_ID_LTR 0x18 /* Latency Tolerance Reporting */
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1668
Quoted 4 lines
drivers/vfio/pci/vfio_pci_config.c +4
-0+ vpasid_cap.next =0 ; +/* clear the control reg for guest */ +memset (&vpasid_cap.control_reg,0x0 , +sizeof (vpasid_cap.control_reg));
So we zero a couple registers and that's why we need a structure to define the entire capability and this crazy copy to a cpu endian buffer. No.
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1670
Quoted 2 lines
drivers/vfio/pci/vfio_pci_config.c +2
-0+ +memset (map + epos, vpasid_cap.id, len);
See below.
Quoted 4 lines
drivers/vfio/pci/vfio_pci_config.c +4
-0+ ret =vfio_fill_custom_vconfig_bytes (vdev, epos, + (u8 *)&vpasid_cap, len); +if (!ret) { +/*
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1684
Quoted 10 lines
drivers/vfio/pci/vfio_pci_config.c +10
-0+* Successfully filled in PASID cap, update +* the next offset in previous cap header, +* and also update caller about the offset +* of next cap if any. +*/ +u32 val = epos; + **prevp &=cpu_to_le32 (~(0xffcU <<20 )); + **prevp |=cpu_to_le32 (val <<20 ); + *prevp = (__le32 *)&vdev->vconfig[epos]; + *next = epos + len;
Could we make this any more complicated?
Quoted 63 lines
drivers/vfio/pci/vfio_pci_config.c +63
-0+ } + +out: +return ret; +} + +struct vfio_pci_pri_cap_data { +u32 id:16 ; +u32 version:4 ; +u32 next:12 ; +union { +u16 control_reg_val; +struct { +u16 enable:1 ; +u16 reset:1 ; +u16 rsv1:14 ; + }; + } control_reg; +union { +u16 status_reg_val; +struct { +u16 rf:1 ; +u16 uprgi:1 ; +u16 rsv2:6 ; +u16 stop:1 ; +u16 rsv3:6 ; +u16 pasid_required:1 ; + }; + } status_reg; +u32 prq_capacity; +u32 prq_quota; +}; + +static int vfio_pci_add_pri_cap (struct vfio_pci_device *vdev, +struct pci_dev *pdev, +u16 epos,u16 *next,__le32 **prevp) +{ +u8 *map = vdev->pci_config_map; +int ecap =PCI_EXT_CAP_ID_PRI ; +int len = pci_ext_cap_length[ecap]; +struct vfio_pci_pri_cap_data pri_cap; +struct vfio_pci_pri_cap_data vpri_cap; +int ret; + +/* +* If no cap filled in this function, should make sure the next +* pointer points to current epos. +*/ + *next = epos; + +if (!len) { +pr_info ("%s: VF %s hiding PRI capability\n" , + __func__,dev_name (&vdev->pdev->dev)); + ret =0 ; + goto out; + } + +/* Add PASID capability*/ + ret =vfio_pci_get_ecap_content (pdev, ecap, + len, (u8 *)&pri_cap); +if (ret) + goto out; +
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1757
Quoted 10 lines
drivers/vfio/pci/vfio_pci_config.c +10
-0+if (!pri_cap.control_reg.enable) { +pr_debug ("%s: its PF's PRI capability is not enabled\n" , +dev_name (&vdev->pdev->dev)); + ret =0 ; + goto out; + } + +memcpy (&vpri_cap, &pri_cap, len); + + vpri_cap.id =0x19 ;
#define PCI_EXT_CAP_ID_SECPCI 0x19 /* Secondary PCIe Capability */
???
Quoted 79 lines
drivers/vfio/pci/vfio_pci_config.c +53
-1+ vpri_cap.next =0 ; +/* clear the control reg for guest */ +memset (&vpri_cap.control_reg,0x0 , +sizeof (vpri_cap.control_reg)); + +memset (map + epos, vpri_cap.id, len); + ret =vfio_fill_custom_vconfig_bytes (vdev, epos, + (u8 *)&vpri_cap, len); +if (!ret) { +/* +* Successfully filled in PASID cap, update +* the next offset in previous cap header, +* and also update caller about the offset +* of next cap if any. +*/ +u32 val = epos; + **prevp &=cpu_to_le32 (~(0xffcU <<20 )); + **prevp |=cpu_to_le32 (val <<20 ); + *prevp = (__le32 *)&vdev->vconfig[epos]; + *next = epos + len; + } + +out: +return ret; +} + +static int vfio_pci_add_emulated_cap_for_vf (struct vfio_pci_device *vdev, +struct pci_dev *pdev,u16 start_epos,__le32 *prev) +{ +__le32 *__prev = prev; +u16 epos = start_epos, epos_next = start_epos; +int ret =0 ; + +/* Add PASID capability*/ + ret =vfio_pci_add_pasid_cap (vdev, pdev, epos, + &epos_next, &__prev); +if (ret) +return ret; + +/* Add PRI capability */ + epos = epos_next; + ret =vfio_pci_add_pri_cap (vdev, pdev, epos, + &epos_next, &__prev); + +return ret; +} +static int vfio_ecap_init (struct vfio_pci_device *vdev) {struct pci_dev *pdev = vdev->pdev;u8 *map = vdev->pci_config_map; -u16 epos; +u16 epos, epos_max;__le32 *prev =NULL ;int loops, ret, ecaps =0 ;@@ -1521,6 +1814,7 @@ static int vfio_ecap_init(struct vfio_pci_device *vdev)return 0 ; epos =PCI_CFG_SPACE_SIZE ; + epos_max =PCI_CFG_SPACE_SIZE ; loops = (pdev->cfg_size -PCI_CFG_SPACE_SIZE ) /PCI_CAP_SIZEOF ;@@ -1545,6 +1839,9 @@ static int vfio_ecap_init(struct vfio_pci_device *vdev) } } +if (epos_max <= (epos + len)) + epos_max = epos + len; +if (!len) {pci_info (pdev,"%s: hiding ecap %#x@%#x\n" , __func__, ecap, epos);@@ -1604,6 +1901,18 @@ static int vfio_ecap_init(struct vfio_pci_device *vdev)if (!ecaps) *(u32 *)&vdev->vconfig[PCI_CFG_SPACE_SIZE ] =0 ; +#ifdef CONFIG_PCI_ATS
drivers/vfio/pci/vfio_pci_config.c static int vfio_ecap_init(struct vfio_pci_device *vdev) · line 1914
Quoted 10 lines
drivers/vfio/pci/vfio_pci_config.c +10
-0+if (pdev->is_virtfn) { +struct pci_dev *physfn = pdev->physfn; + + ret =vfio_pci_add_emulated_cap_for_vf (vdev, + physfn, epos_max, prev); +if (ret) +pr_info ("%s, failed to add special caps for VF %s\n" , + __func__,dev_name (&vdev->pdev->dev)); + } +#endif
I can only imagine that we should place the caps at the same location they exist on the PF, we don't know what hidden registers might be hiding in config space.
Quoted 23 lines
drivers/vfio/pci/vfio_pci_config.c +12
-0+return 0 ; }@@ -1748,6 +2057,17 @@ static size_t vfio_pci_cap_remaining_dword(struct vfio_pci_device *vdev,return i; } +static bool vfio_pci_need_virt_perm (struct pci_dev *pdev,u8 cap_id) +{ +#ifdef CONFIG_PCI_ATS +return (pdev->is_virtfn && + (cap_id ==PCI_EXT_CAP_ID_PASID || + cap_id ==PCI_EXT_CAP_ID_PRI )); +#else +return false; +#endif +} +static ssize_t vfio_config_do_rw (struct vfio_pci_device *vdev,char __user *buf,size_t count,loff_t *ppos,bool iswrite) {
drivers/vfio/pci/vfio_pci_config.c static ssize_t vfio_config_do_rw(struct vfio_pci_device *vdev, char __user *buf, · line 2105
Quoted 6 lines
drivers/vfio/pci/vfio_pci_config.c +2
-1@@ -1781,7 +2101,8 @@ static ssize_t vfio_config_do_rw(struct vfio_pci_device *vdev, char __user *buf,if (cap_id ==PCI_CAP_ID_INVALID ) { perm = &unassigned_perms; cap_start = *ppos; - }else if (cap_id ==PCI_CAP_ID_INVALID_VIRT ) { + }else if (cap_id ==PCI_CAP_ID_INVALID_VIRT || +vfio_pci_need_virt_perm (pdev, cap_id)) {
Why not simply fill the map with PCI_CAP_ID_INVALID_VIRT?
drivers/vfio/pci/vfio_pci_config.c static ssize_t vfio_config_do_rw(struct vfio_pci_device *vdev, char __user *buf, · line 2108
Quoted 3 lines
drivers/vfio/pci/vfio_pci_config.c +0
-0perm = &virt_perms; cap_start = *ppos; }else {
This is really not close to acceptable as is. Sorry. Thanks,
Alex
RE: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
Hi Alex,
Quoted 62 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 7:00 AM
To: Liu, Yi L <[email protected]>
Subject: Re: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFsQuoted 18 lines · Yi L
From: Liu Yi L <[email protected]>
Per PCIe r5.0, sec 9.3.7.14, if a PF implements the PASID Capability, the PF PASID configuration is shared by its VFs. VFs must not implement their own PASID Capability.
Per PCIe r5.0, sec 9.3.7.11, VFs must not implement the PRI Capability. If the PF implements PRI, it is shared by the VFs.
On bare metal, it has been fixed by below efforts. to PASID/PRI are
https://lkml.org/lkml/2019/9/5/996
https://lkml.org/lkml/2019/9/5/995This patch adds emulated PASID/PRI capabilities for VFs when assigned to VMs via vfio-pci driver. This is required for enabling vSVA on pass-through VFs as VFs have no PASID/PRI capability structure in its configure space.
Quoted 8 lines
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/pci/vfio_pci_config.c | 325++++++++++++++++++++++++++++++++++++-
Quoted 14 lines
1 file changed, 323 insertions(+), 2 deletions(-)drivers/vfio/pci/vfio_pci_config.c +4
-0@@ -1509,11 +1509,304 @@ static int vfio_cap_init(struct vfio_pci_device *vdev)return 0 ; } +static int vfio_fill_custom_vconfig_bytes (struct vfio_pci_device *vdev, +int offset,uint8_t *data,int size) +{ +int ret =0 , data_offset =0 ;drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1525
Quoted 10 lines
drivers/vfio/pci/vfio_pci_config.c +10
-0+ +while (size) { +int filled; + +if (size >=4 && !(offset %4 )) { +__le32 *dwordp = (__le32 *)&vdev->vconfig[offset]; +u32 dword; + +memcpy (&dword, data + data_offset,4 ); + *dwordp =cpu_to_le32 (dword);Why wouldn't we do:
Quoted 5 lines
*dwordp =
cpu_to_le32(*(u32 *)(data + data_offset));or better yet, increment data on each iteration for:
*dwordp =
cpu_to_le32(*(u32 *)data);
I'll refine it.
Quoted 4 lines
vfio_fill_vconfig_bytes()does almost this same thing, getting the data
from config space rather than a buffer, so please figure out how to avoid duplicating the logic.
This patch is to emulate the PASID/PRI capability for VF. And per my understanding, the cap data from PF's config space cannot be filled to VF's vconfig directly. Take the control register in PASID capability structure as an example. If PASID cap is enabled in PF, the PASID enable bit in the control register will be set. If the cap data is filled to VF's vconfig directly, then guest will see a default enabled PASID capability in the VF. I guess it is not good. But, I may be wrong. If no such requirement, I can use vfio_fill_vconfig_bytes() directly, no need to do copy and modify and then copy.
Also, if still needs to do the copy and modification, I may modify the vfio_fill_vconfig_bytes() to pass in the buffer and an indicator to tell vfio_fill_vconfig_bytes() whether fetch data from buffer or from config space.
Quoted 15 lines
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1537
Quoted 12 lines
drivers/vfio/pci/vfio_pci_config.c +12
-0+ filled =4 ; + }else if (size >=2 && !(offset %2 )) { +__le16 *wordp = (__le16 *)&vdev->vconfig[offset]; +u16 word; + +memcpy (&word, data + data_offset,2 ); + *wordp =cpu_to_le16 (word); + filled =2 ; + }else { +u8 *byte = &vdev->vconfig[offset]; + +memcpy (byte, data + data_offset,1 );This one is really silly.
Quoted 1 line
vdev->vconfig[offset] = *data;
got it.
Quoted 27 lines
Quoted 27 lines
drivers/vfio/pci/vfio_pci_config.c +27
-0+ filled =1 ; + } + + offset += filled; + data_offset += filled; + size -= filled; + } + +return ret; +} + +static int vfio_pci_get_ecap_content (struct pci_dev *pdev, +int cap,int cap_len,u8 *content) +{ +int pos, offset, len = cap_len, ret =0 ; + + pos =pci_find_ext_capability (pdev, cap); +if (!pos) +return -EINVAL ; + + offset =0 ; +while (len) { +int fetched; + +if (len >=4 && !(pos %4 )) { +u32 *dwordp = (u32 *) (content + offset); +u32 dword;
Quoted 8 lines
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1570
Quoted 6 lines
drivers/vfio/pci/vfio_pci_config.c +6
-0+__le32 *dwptr = (__le32 *) &dword; + + ret =pci_read_config_dword (pdev, pos, &dword); +if (ret) +return ret; + *dwordp =le32_to_cpu (*dwptr);WHAT???
pci_read_config_dword()returns cpu endian data!
my bad. will remove the silly le32_to_cpu() convert.
Quoted 71 lines
Quoted 71 lines
drivers/vfio/pci/vfio_pci_config.c +71
-0+ fetched =4 ; + }else if (len >=2 && !(pos %2 )) { +u16 *wordp = (u16 *) (content + offset); +u16 word; +__le16 *wptr = (__le16 *) &word; + + ret =pci_read_config_word (pdev, pos, &word); +if (ret) +return ret; + *wordp =le16_to_cpu (*wptr); + fetched =2 ; + }else { +u8 *byte = (u8 *) (content + offset); + + ret =pci_read_config_byte (pdev, pos, byte); +if (ret) +return ret; + fetched =1 ; + } + + pos += fetched; + offset += fetched; + len -= fetched; + } + +return ret; +} + +struct vfio_pci_pasid_cap_data { +u32 id:16 ; +u32 version:4 ; +u32 next:12 ; +union { +u16 cap_reg_val; +struct { +u16 rsv1:1 ; +u16 execs:1 ; +u16 prvs:1 ; +u16 rsv2:5 ; +u16 pasid_bits:5 ; +u16 rsv3:3 ; + }; + } cap_reg; +union { +u16 control_reg_val; +struct { +u16 paside:1 ; +u16 exece:1 ; +u16 prve:1 ; +u16 rsv4:13 ; + }; + } control_reg; +}; + +static int vfio_pci_add_pasid_cap (struct vfio_pci_device *vdev, +struct pci_dev *pdev, +u16 epos,u16 *next,__le32 **prevp) +{ +u8 *map = vdev->pci_config_map; +int ecap =PCI_EXT_CAP_ID_PASID ; +int len = pci_ext_cap_length[ecap]; +struct vfio_pci_pasid_cap_data pasid_cap; +struct vfio_pci_pasid_cap_data vpasid_cap; +int ret; + +/* +* If no cap filled in this function, should make sure the next +* pointer points to current epos. +*/ + *next = epos; +
Quoted 8 lines
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1647
Quoted 6 lines
drivers/vfio/pci/vfio_pci_config.c +6
-0+if (!len) { +pr_info ("%s: VF %s hiding PASID capability\n" , + __func__,dev_name (&vdev->pdev->dev)); + ret =0 ; + goto out; + }No! Why? This is dead code.
will remove it. thanks.
Quoted 9 lines
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1651
Quoted 4 lines
drivers/vfio/pci/vfio_pci_config.c +4
-0+ +/* Add PASID capability*/ + ret =vfio_pci_get_ecap_content (pdev, ecap, + len, (u8 *)&pasid_cap);Why wouldn't we just use
vfio_fill_config_bytes()rather than this ridiculous approach of coping it to a buffer, modifying it and copying it out?
As the above reply, if copying PF's config space data to VF's vconfig directly is fine, then I can drop them.
Quoted 11 lines
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1660
Quoted 9 lines
drivers/vfio/pci/vfio_pci_config.c +9
-0+if (ret) + goto out; + +if (!pasid_cap.control_reg.paside) { +pr_debug ("%s: its PF's PASID capability is not enabled\n" , +dev_name (&vdev->pdev->dev)); + ret =0 ; + goto out; + }What happens if the PF's PASID gets disabled while we're using it??
This is actually the open I highlighted in cover letter. Per the reply from Baolu, this seems to be an open for bare-metal all the same.
https://lkml.org/lkml/2020/3/31/95
Quoted 8 lines
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1664
Quoted 4 lines
drivers/vfio/pci/vfio_pci_config.c +4
-0+ +memcpy (&vpasid_cap, &pasid_cap, len); + + vpasid_cap.id =0x18 ;What is going on here?
#define
PCI_EXT_CAP_ID_LTR0x18/* Latency Tolerance Reporting */
my bad... This one was used in development phase. not needed in formal patch. really sorry for the mislead.. will remove it.
Quoted 8 lines
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1668
Quoted 4 lines
drivers/vfio/pci/vfio_pci_config.c +4
-0+ vpasid_cap.next =0 ; +/* clear the control reg for guest */ +memset (&vpasid_cap.control_reg,0x0 , +sizeof (vpasid_cap.control_reg));So we zero a couple registers and that's why we need a structure to define the entire capability and this crazy copy to a cpu endian buffer. No.
there are two reasons for define a structure for the capability. For one, to check if PASID capability is enabled in PF. For two, to zero the control registers to ensure the guest see a clean control register. But if it is not necessary to do the two operations, it can be dropped.
Quoted 5 lines
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1670
Quoted 2 lines
drivers/vfio/pci/vfio_pci_config.c +2
-0+ +memset (map + epos, vpasid_cap.id, len);See below.
Quoted 16 lines
Quoted 4 lines
drivers/vfio/pci/vfio_pci_config.c +4
-0+ ret =vfio_fill_custom_vconfig_bytes (vdev, epos, + (u8 *)&vpasid_cap, len); +if (!ret) { +/* drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1684
Quoted 10 lines
drivers/vfio/pci/vfio_pci_config.c +10
-0+* Successfully filled in PASID cap, update +* the next offset in previous cap header, +* and also update caller about the offset +* of next cap if any. +*/ +u32 val = epos; + **prevp &=cpu_to_le32 (~(0xffcU <<20 )); + **prevp |=cpu_to_le32 (val <<20 ); + *prevp = (__le32 *)&vdev->vconfig[epos]; + *next = epos + len;Could we make this any more complicated?
yeah, I should have added some comments.
/* update the next field of prior cap structure */ **prevp &= cpu_to_le32(~(0xffcU << 20)); **prevp |= cpu_to_le32(val << 20); /* update the prevp pointer to point to the current cap */ *prevp = (__le32 *)&vdev->vconfig[epos]; *next = epos + len;
Quoted 76 lines
Quoted 63 lines
drivers/vfio/pci/vfio_pci_config.c +63
-0+ } + +out: +return ret; +} + +struct vfio_pci_pri_cap_data { +u32 id:16 ; +u32 version:4 ; +u32 next:12 ; +union { +u16 control_reg_val; +struct { +u16 enable:1 ; +u16 reset:1 ; +u16 rsv1:14 ; + }; + } control_reg; +union { +u16 status_reg_val; +struct { +u16 rf:1 ; +u16 uprgi:1 ; +u16 rsv2:6 ; +u16 stop:1 ; +u16 rsv3:6 ; +u16 pasid_required:1 ; + }; + } status_reg; +u32 prq_capacity; +u32 prq_quota; +}; + +static int vfio_pci_add_pri_cap (struct vfio_pci_device *vdev, +struct pci_dev *pdev, +u16 epos,u16 *next,__le32 **prevp) +{ +u8 *map = vdev->pci_config_map; +int ecap =PCI_EXT_CAP_ID_PRI ; +int len = pci_ext_cap_length[ecap]; +struct vfio_pci_pri_cap_data pri_cap; +struct vfio_pci_pri_cap_data vpri_cap; +int ret; + +/* +* If no cap filled in this function, should make sure the next +* pointer points to current epos. +*/ + *next = epos; + +if (!len) { +pr_info ("%s: VF %s hiding PRI capability\n" , + __func__,dev_name (&vdev->pdev->dev)); + ret =0 ; + goto out; + } + +/* Add PASID capability*/ + ret =vfio_pci_get_ecap_content (pdev, ecap, + len, (u8 *)&pri_cap); +if (ret) + goto out; +drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1757
Quoted 10 lines
drivers/vfio/pci/vfio_pci_config.c +10
-0+if (!pri_cap.control_reg.enable) { +pr_debug ("%s: its PF's PRI capability is not enabled\n" , +dev_name (&vdev->pdev->dev)); + ret =0 ; + goto out; + } + +memcpy (&vpri_cap, &pri_cap, len); + + vpri_cap.id =0x19 ;#define
PCI_EXT_CAP_ID_SECPCI0x19/* Secondary PCIe Capability */
Quoted 1 line
???
my bad... This line was used in development phase. not needed in formal patch. really sorry for the mislead.. will remove it.
Quoted 78 lines
Quoted 78 lines
drivers/vfio/pci/vfio_pci_config.c +52
-1+ vpri_cap.next =0 ; +/* clear the control reg for guest */ +memset (&vpri_cap.control_reg,0x0 , +sizeof (vpri_cap.control_reg)); + +memset (map + epos, vpri_cap.id, len); + ret =vfio_fill_custom_vconfig_bytes (vdev, epos, + (u8 *)&vpri_cap, len); +if (!ret) { +/* +* Successfully filled in PASID cap, update +* the next offset in previous cap header, +* and also update caller about the offset +* of next cap if any. +*/ +u32 val = epos; + **prevp &=cpu_to_le32 (~(0xffcU <<20 )); + **prevp |=cpu_to_le32 (val <<20 ); + *prevp = (__le32 *)&vdev->vconfig[epos]; + *next = epos + len; + } + +out: +return ret; +} + +static int vfio_pci_add_emulated_cap_for_vf (struct vfio_pci_device *vdev, +struct pci_dev *pdev,u16 start_epos,__le32 *prev) +{ +__le32 *__prev = prev; +u16 epos = start_epos, epos_next = start_epos; +int ret =0 ; + +/* Add PASID capability*/ + ret =vfio_pci_add_pasid_cap (vdev, pdev, epos, + &epos_next, &__prev); +if (ret) +return ret; + +/* Add PRI capability */ + epos = epos_next; + ret =vfio_pci_add_pri_cap (vdev, pdev, epos, + &epos_next, &__prev); + +return ret; +} +static int vfio_ecap_init (struct vfio_pci_device *vdev) {struct pci_dev *pdev = vdev->pdev;u8 *map = vdev->pci_config_map; -u16 epos; +u16 epos, epos_max;__le32 *prev =NULL ;int loops, ret, ecaps =0 ;@@ -1521,6 +1814,7 @@ static int vfio_ecap_init(struct vfio_pci_device *vdev)return 0 ; epos =PCI_CFG_SPACE_SIZE ; + epos_max =PCI_CFG_SPACE_SIZE ; loops = (pdev->cfg_size -PCI_CFG_SPACE_SIZE ) /PCI_CAP_SIZEOF ;@@ -1545,6 +1839,9 @@ static int vfio_ecap_init(struct vfio_pci_device *vdev) } } +if (epos_max <= (epos + len)) + epos_max = epos + len; +if (!len) {pci_info (pdev,"%s: hiding ecap %#x@%#x\n" , __func__, ecap, epos);@@ -1604,6 +1901,18 @@ static int vfio_ecap_init(struct vfio_pci_device *vdev)if (!ecaps) *(u32 *)&vdev->vconfig[PCI_CFG_SPACE_SIZE ] =0 ;
Quoted 15 lines
drivers/vfio/pci/vfio_pci_config.c static int vfio_ecap_init(struct vfio_pci_device *vdev) · line 1914
Quoted 11 lines
drivers/vfio/pci/vfio_pci_config.c +11
-0+#ifdef CONFIG_PCI_ATS +if (pdev->is_virtfn) { +struct pci_dev *physfn = pdev->physfn; + + ret =vfio_pci_add_emulated_cap_for_vf (vdev, + physfn, epos_max, prev); +if (ret) +pr_info ("%s, failed to add special caps for VF %s\n" , + __func__,dev_name (&vdev->pdev->dev)); + } +#endif I can only imagine that we should place the caps at the same location they exist on the PF, we don't know what hidden registers might be hiding in config space.
but we are not sure whether the same location is available on VF. In this patch, it actually places the emulated cap physically behind the cap which lays farthest (its offset is largest) within VF's config space as the PCIe caps are linked in a chain.
Quoted 25 lines
drivers/vfio/pci/vfio_pci_config.c static size_t vfio_pci_cap_remaining_dword(struct
Quoted 5 lines
drivers/vfio/pci/vfio_pci_config.c +1
-0+return 0 ; }@@ -1748,6 +2057,17 @@ static size_t vfio_pci_cap_remaining_dword(struct
vfio_pci_device*vdev,drivers/vfio/pci/vfio_pci_config.c static ssize_t vfio_config_do_rw(struct vfio_pci_device
Quoted 18 lines
drivers/vfio/pci/vfio_pci_config.c +11
-0return i; } +static bool vfio_pci_need_virt_perm (struct pci_dev *pdev,u8 cap_id) +{ +#ifdef CONFIG_PCI_ATS +return (pdev->is_virtfn && + (cap_id ==PCI_EXT_CAP_ID_PASID || + cap_id ==PCI_EXT_CAP_ID_PRI )); +#else +return false; +#endif +} +static ssize_t vfio_config_do_rw (struct vfio_pci_device *vdev,char __user *buf,size_t count,loff_t *ppos,bool iswrite) {@@ -1781,7 +2101,8 @@ static ssize_t vfio_config_do_rw(struct vfio_pci_device*vdev, char
__user*buf,
Quoted 8 lines
drivers/vfio/pci/vfio_pci_config.c static ssize_t vfio_config_do_rw(struct vfio_pci_device *vdev, char __user *buf, · line 2105
Quoted 6 lines
drivers/vfio/pci/vfio_pci_config.c +2
-1if (cap_id ==PCI_CAP_ID_INVALID ) { perm = &unassigned_perms; cap_start = *ppos; - }else if (cap_id ==PCI_CAP_ID_INVALID_VIRT ) { + }else if (cap_id ==PCI_CAP_ID_INVALID_VIRT || +vfio_pci_need_virt_perm (pdev, cap_id)) {Why not simply fill the map with
PCI_CAP_ID_INVALID_VIRT?
good idea. thanks.
Quoted 5 lines
drivers/vfio/pci/vfio_pci_config.c static ssize_t vfio_config_do_rw(struct vfio_pci_device *vdev, char __user *buf, · line 2108
Quoted 3 lines
drivers/vfio/pci/vfio_pci_config.c +0
-0perm = &virt_perms; cap_start = *ppos; }else {This is really not close to acceptable as is. Sorry. Thanks,
really sorry for the silly mistakes, I can try to make a new version after we got a decision on whether needs to do data copy and medication before filling into vconfig.
Thanks,
Yi Liu
Re: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
Quoted 87 lines · Yi L
Hi Alex,
Quoted 62 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 7:00 AM
To: Liu, Yi L <[email protected]>
Subject: Re: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFsQuoted 18 lines · Yi L
From: Liu Yi L <[email protected]>
Per PCIe r5.0, sec 9.3.7.14, if a PF implements the PASID Capability, the PF PASID configuration is shared by its VFs. VFs must not implement their own PASID Capability.
Per PCIe r5.0, sec 9.3.7.11, VFs must not implement the PRI Capability. If the PF implements PRI, it is shared by the VFs.
On bare metal, it has been fixed by below efforts. to PASID/PRI are
https://lkml.org/lkml/2019/9/5/996
https://lkml.org/lkml/2019/9/5/995This patch adds emulated PASID/PRI capabilities for VFs when assigned to VMs via vfio-pci driver. This is required for enabling vSVA on pass-through VFs as VFs have no PASID/PRI capability structure in its configure space.
Quoted 8 lines
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/pci/vfio_pci_config.c | 325++++++++++++++++++++++++++++++++++++-
Quoted 14 lines
1 file changed, 323 insertions(+), 2 deletions(-)drivers/vfio/pci/vfio_pci_config.c +4
-0@@ -1509,11 +1509,304 @@ static int vfio_cap_init(struct vfio_pci_device *vdev)return 0 ; } +static int vfio_fill_custom_vconfig_bytes (struct vfio_pci_device *vdev, +int offset,uint8_t *data,int size) +{ +int ret =0 , data_offset =0 ;drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1525
Quoted 10 lines
drivers/vfio/pci/vfio_pci_config.c +10
-0+ +while (size) { +int filled; + +if (size >=4 && !(offset %4 )) { +__le32 *dwordp = (__le32 *)&vdev->vconfig[offset]; +u32 dword; + +memcpy (&dword, data + data_offset,4 ); + *dwordp =cpu_to_le32 (dword);Why wouldn't we do:
Quoted 5 lines
*dwordp =
cpu_to_le32(*(u32 *)(data + data_offset));or better yet, increment data on each iteration for:
*dwordp =
cpu_to_le32(*(u32 *)data);I'll refine it.
Quoted 4 lines
vfio_fill_vconfig_bytes()does almost this same thing, getting the data
from config space rather than a buffer, so please figure out how to avoid duplicating the logic.This patch is to emulate the PASID/PRI capability for VF. And per my understanding, the cap data from PF's config space cannot be filled to VF's vconfig directly. Take the control register in PASID capability structure as an example. If PASID cap is enabled in PF, the PASID enable bit in the control register will be set. If the cap data is filled to VF's vconfig directly, then guest will see a default enabled PASID capability in the VF. I guess it is not good. But, I may be wrong. If no such requirement, I can use
vfio_fill_vconfig_bytes()directly, no need to do copy and modify and then copy.
Quoted 4 lines
Also, if still needs to do the copy and modification, I may modify the
vfio_fill_vconfig_bytes()to pass in the buffer and an indicator to tellvfio_fill_vconfig_bytes()whether fetch data from buffer or from config space.
Why is vconfig not your buffer? We're building config space emulation before the user has access to the device, so it's really not an issue to copy the PF config into the VF vconfig, then modify the enable bit in the vconfig space.
Quoted 164 lines
Quoted 15 lines
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1537
Quoted 12 lines
drivers/vfio/pci/vfio_pci_config.c +12
-0+ filled =4 ; + }else if (size >=2 && !(offset %2 )) { +__le16 *wordp = (__le16 *)&vdev->vconfig[offset]; +u16 word; + +memcpy (&word, data + data_offset,2 ); + *wordp =cpu_to_le16 (word); + filled =2 ; + }else { +u8 *byte = &vdev->vconfig[offset]; + +memcpy (byte, data + data_offset,1 );This one is really silly.
Quoted 1 line
vdev->vconfig[offset] = *data;
got it.
Quoted 27 lines
Quoted 27 lines
drivers/vfio/pci/vfio_pci_config.c +27
-0+ filled =1 ; + } + + offset += filled; + data_offset += filled; + size -= filled; + } + +return ret; +} + +static int vfio_pci_get_ecap_content (struct pci_dev *pdev, +int cap,int cap_len,u8 *content) +{ +int pos, offset, len = cap_len, ret =0 ; + + pos =pci_find_ext_capability (pdev, cap); +if (!pos) +return -EINVAL ; + + offset =0 ; +while (len) { +int fetched; + +if (len >=4 && !(pos %4 )) { +u32 *dwordp = (u32 *) (content + offset); +u32 dword;Quoted 8 lines
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1570
Quoted 6 lines
drivers/vfio/pci/vfio_pci_config.c +6
-0+__le32 *dwptr = (__le32 *) &dword; + + ret =pci_read_config_dword (pdev, pos, &dword); +if (ret) +return ret; + *dwordp =le32_to_cpu (*dwptr);WHAT???
pci_read_config_dword()returns cpu endian data!my bad. will remove the silly
le32_to_cpu()convert.Quoted 71 lines
Quoted 71 lines
drivers/vfio/pci/vfio_pci_config.c +71
-0+ fetched =4 ; + }else if (len >=2 && !(pos %2 )) { +u16 *wordp = (u16 *) (content + offset); +u16 word; +__le16 *wptr = (__le16 *) &word; + + ret =pci_read_config_word (pdev, pos, &word); +if (ret) +return ret; + *wordp =le16_to_cpu (*wptr); + fetched =2 ; + }else { +u8 *byte = (u8 *) (content + offset); + + ret =pci_read_config_byte (pdev, pos, byte); +if (ret) +return ret; + fetched =1 ; + } + + pos += fetched; + offset += fetched; + len -= fetched; + } + +return ret; +} + +struct vfio_pci_pasid_cap_data { +u32 id:16 ; +u32 version:4 ; +u32 next:12 ; +union { +u16 cap_reg_val; +struct { +u16 rsv1:1 ; +u16 execs:1 ; +u16 prvs:1 ; +u16 rsv2:5 ; +u16 pasid_bits:5 ; +u16 rsv3:3 ; + }; + } cap_reg; +union { +u16 control_reg_val; +struct { +u16 paside:1 ; +u16 exece:1 ; +u16 prve:1 ; +u16 rsv4:13 ; + }; + } control_reg; +}; + +static int vfio_pci_add_pasid_cap (struct vfio_pci_device *vdev, +struct pci_dev *pdev, +u16 epos,u16 *next,__le32 **prevp) +{ +u8 *map = vdev->pci_config_map; +int ecap =PCI_EXT_CAP_ID_PASID ; +int len = pci_ext_cap_length[ecap]; +struct vfio_pci_pasid_cap_data pasid_cap; +struct vfio_pci_pasid_cap_data vpasid_cap; +int ret; + +/* +* If no cap filled in this function, should make sure the next +* pointer points to current epos. +*/ + *next = epos; +Quoted 8 lines
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1647
Quoted 6 lines
drivers/vfio/pci/vfio_pci_config.c +6
-0+if (!len) { +pr_info ("%s: VF %s hiding PASID capability\n" , + __func__,dev_name (&vdev->pdev->dev)); + ret =0 ; + goto out; + }No! Why? This is dead code.
will remove it. thanks.
Quoted 9 lines
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1651
Quoted 4 lines
drivers/vfio/pci/vfio_pci_config.c +4
-0+ +/* Add PASID capability*/ + ret =vfio_pci_get_ecap_content (pdev, ecap, + len, (u8 *)&pasid_cap);Why wouldn't we just use
vfio_fill_config_bytes()rather than this ridiculous approach of coping it to a buffer, modifying it and copying it out?As the above reply, if copying PF's config space data to VF's vconfig directly is fine, then I can drop them.
Quoted 11 lines
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1660
Quoted 9 lines
drivers/vfio/pci/vfio_pci_config.c +9
-0+if (ret) + goto out; + +if (!pasid_cap.control_reg.paside) { +pr_debug ("%s: its PF's PASID capability is not enabled\n" , +dev_name (&vdev->pdev->dev)); + ret =0 ; + goto out; + }What happens if the PF's PASID gets disabled while we're using it??
Quoted 3 lines
This is actually the open I highlighted in cover letter. Per the reply from Baolu, this seems to be an open for bare-metal all the same.
https://lkml.org/lkml/2020/3/31/95
Seems that needs to get sorted out before we can expose this. Maybe some sort of registration with the PF driver that PASID is being used by a VF so it cannot be disabled?
Quoted 21 lines
Quoted 8 lines
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1664
Quoted 4 lines
drivers/vfio/pci/vfio_pci_config.c +4
-0+ +memcpy (&vpasid_cap, &pasid_cap, len); + + vpasid_cap.id =0x18 ;What is going on here?
#define
PCI_EXT_CAP_ID_LTR0x18/* Latency Tolerance Reporting */my bad... This one was used in development phase. not needed in formal patch. really sorry for the mislead.. will remove it.
Quoted 8 lines
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1668
Quoted 4 lines
drivers/vfio/pci/vfio_pci_config.c +4
-0+ vpasid_cap.next =0 ; +/* clear the control reg for guest */ +memset (&vpasid_cap.control_reg,0x0 , +sizeof (vpasid_cap.control_reg));So we zero a couple registers and that's why we need a structure to define the entire capability and this crazy copy to a cpu endian buffer. No.
Quoted 4 lines
there are two reasons for define a structure for the capability. For one, to check if PASID capability is enabled in PF. For two, to zero the control registers to ensure the guest see a clean control register. But if it is not necessary to do the two operations, it can be dropped.
Neither of these require the structures you've defined and I haven't convinced myself that the bit fields don't have their own issues with endianness. Please drop them.
Quoted 22 lines
Quoted 5 lines
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1670
Quoted 2 lines
drivers/vfio/pci/vfio_pci_config.c +2
-0+ +memset (map + epos, vpasid_cap.id, len);See below.
Quoted 16 lines
Quoted 4 lines
drivers/vfio/pci/vfio_pci_config.c +4
-0+ ret =vfio_fill_custom_vconfig_bytes (vdev, epos, + (u8 *)&vpasid_cap, len); +if (!ret) { +/* drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1684
Quoted 10 lines
drivers/vfio/pci/vfio_pci_config.c +10
-0+* Successfully filled in PASID cap, update +* the next offset in previous cap header, +* and also update caller about the offset +* of next cap if any. +*/ +u32 val = epos; + **prevp &=cpu_to_le32 (~(0xffcU <<20 )); + **prevp |=cpu_to_le32 (val <<20 ); + *prevp = (__le32 *)&vdev->vconfig[epos]; + *next = epos + len;Could we make this any more complicated?
Quoted 8 lines
yeah, I should have added some comments.
/* update the next field of prior cap structure */ **prevp &= cpu_to_le32(~(0xffcU << 20)); **prevp |= cpu_to_le32(val << 20); /* update the prevp pointer to point to the current cap */ *prevp = (__le32 *)&vdev->vconfig[epos]; *next = epos + len;
The main loop when processing capabilities already has this sort of logic. Please figure out a way to not duplicate it.
Quoted 175 lines
Quoted 76 lines
Quoted 63 lines
drivers/vfio/pci/vfio_pci_config.c +63
-0+ } + +out: +return ret; +} + +struct vfio_pci_pri_cap_data { +u32 id:16 ; +u32 version:4 ; +u32 next:12 ; +union { +u16 control_reg_val; +struct { +u16 enable:1 ; +u16 reset:1 ; +u16 rsv1:14 ; + }; + } control_reg; +union { +u16 status_reg_val; +struct { +u16 rf:1 ; +u16 uprgi:1 ; +u16 rsv2:6 ; +u16 stop:1 ; +u16 rsv3:6 ; +u16 pasid_required:1 ; + }; + } status_reg; +u32 prq_capacity; +u32 prq_quota; +}; + +static int vfio_pci_add_pri_cap (struct vfio_pci_device *vdev, +struct pci_dev *pdev, +u16 epos,u16 *next,__le32 **prevp) +{ +u8 *map = vdev->pci_config_map; +int ecap =PCI_EXT_CAP_ID_PRI ; +int len = pci_ext_cap_length[ecap]; +struct vfio_pci_pri_cap_data pri_cap; +struct vfio_pci_pri_cap_data vpri_cap; +int ret; + +/* +* If no cap filled in this function, should make sure the next +* pointer points to current epos. +*/ + *next = epos; + +if (!len) { +pr_info ("%s: VF %s hiding PRI capability\n" , + __func__,dev_name (&vdev->pdev->dev)); + ret =0 ; + goto out; + } + +/* Add PASID capability*/ + ret =vfio_pci_get_ecap_content (pdev, ecap, + len, (u8 *)&pri_cap); +if (ret) + goto out; +drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1757
Quoted 10 lines
drivers/vfio/pci/vfio_pci_config.c +10
-0+if (!pri_cap.control_reg.enable) { +pr_debug ("%s: its PF's PRI capability is not enabled\n" , +dev_name (&vdev->pdev->dev)); + ret =0 ; + goto out; + } + +memcpy (&vpri_cap, &pri_cap, len); + + vpri_cap.id =0x19 ;#define
PCI_EXT_CAP_ID_SECPCI0x19/* Secondary PCIe Capability */Quoted 1 line
???
my bad... This line was used in development phase. not needed in formal patch. really sorry for the mislead.. will remove it.
Quoted 78 lines
Quoted 78 lines
drivers/vfio/pci/vfio_pci_config.c +52
-1+ vpri_cap.next =0 ; +/* clear the control reg for guest */ +memset (&vpri_cap.control_reg,0x0 , +sizeof (vpri_cap.control_reg)); + +memset (map + epos, vpri_cap.id, len); + ret =vfio_fill_custom_vconfig_bytes (vdev, epos, + (u8 *)&vpri_cap, len); +if (!ret) { +/* +* Successfully filled in PASID cap, update +* the next offset in previous cap header, +* and also update caller about the offset +* of next cap if any. +*/ +u32 val = epos; + **prevp &=cpu_to_le32 (~(0xffcU <<20 )); + **prevp |=cpu_to_le32 (val <<20 ); + *prevp = (__le32 *)&vdev->vconfig[epos]; + *next = epos + len; + } + +out: +return ret; +} + +static int vfio_pci_add_emulated_cap_for_vf (struct vfio_pci_device *vdev, +struct pci_dev *pdev,u16 start_epos,__le32 *prev) +{ +__le32 *__prev = prev; +u16 epos = start_epos, epos_next = start_epos; +int ret =0 ; + +/* Add PASID capability*/ + ret =vfio_pci_add_pasid_cap (vdev, pdev, epos, + &epos_next, &__prev); +if (ret) +return ret; + +/* Add PRI capability */ + epos = epos_next; + ret =vfio_pci_add_pri_cap (vdev, pdev, epos, + &epos_next, &__prev); + +return ret; +} +static int vfio_ecap_init (struct vfio_pci_device *vdev) {struct pci_dev *pdev = vdev->pdev;u8 *map = vdev->pci_config_map; -u16 epos; +u16 epos, epos_max;__le32 *prev =NULL ;int loops, ret, ecaps =0 ;@@ -1521,6 +1814,7 @@ static int vfio_ecap_init(struct vfio_pci_device *vdev)return 0 ; epos =PCI_CFG_SPACE_SIZE ; + epos_max =PCI_CFG_SPACE_SIZE ; loops = (pdev->cfg_size -PCI_CFG_SPACE_SIZE ) /PCI_CAP_SIZEOF ;@@ -1545,6 +1839,9 @@ static int vfio_ecap_init(struct vfio_pci_device *vdev) } } +if (epos_max <= (epos + len)) + epos_max = epos + len; +if (!len) {pci_info (pdev,"%s: hiding ecap %#x@%#x\n" , __func__, ecap, epos);@@ -1604,6 +1901,18 @@ static int vfio_ecap_init(struct vfio_pci_device *vdev)if (!ecaps) *(u32 *)&vdev->vconfig[PCI_CFG_SPACE_SIZE ] =0 ;Quoted 15 lines
drivers/vfio/pci/vfio_pci_config.c static int vfio_ecap_init(struct vfio_pci_device *vdev) · line 1914
Quoted 11 lines
drivers/vfio/pci/vfio_pci_config.c +11
-0+#ifdef CONFIG_PCI_ATS +if (pdev->is_virtfn) { +struct pci_dev *physfn = pdev->physfn; + + ret =vfio_pci_add_emulated_cap_for_vf (vdev, + physfn, epos_max, prev); +if (ret) +pr_info ("%s, failed to add special caps for VF %s\n" , + __func__,dev_name (&vdev->pdev->dev)); + } +#endif I can only imagine that we should place the caps at the same location they exist on the PF, we don't know what hidden registers might be hiding in config space.
Quoted 4 lines
but we are not sure whether the same location is available on VF. In this patch, it actually places the emulated cap physically behind the cap which lays farthest (its offset is largest) within VF's config space as the PCIe caps are linked in a chain.
But, as we've found on Broadcom NICs (iirc), hardware developers have a nasty habit of hiding random registers in PCI config space, outside of defined capabilities. I feel like IGD might even do this too, is that true? So I don't think we can guarantee that just because a section of config space isn't part of a defined capability that its unused. It only means that it's unused by common code, but it might have device specific purposes. So of the PCIe spec indicates that VFs cannot include these capabilities and virtialization software needs to emulate them, we need somewhere safe to place them in config space, and simply placing them off the end of known capabilities doesn't give me any confidence. Also, hardware has no requirement to make compact use of extended config space. The first capability must be at 0x100, the very next capability could consume all the way to the last byte of the 4K extended range, and the next link in the chain could be somewhere in the middle. Thanks,
Alex
Quoted 49 lines
Quoted 25 lines
drivers/vfio/pci/vfio_pci_config.c static size_t vfio_pci_cap_remaining_dword(struct
Quoted 5 lines
drivers/vfio/pci/vfio_pci_config.c +1
-0+return 0 ; }@@ -1748,6 +2057,17 @@ static size_t vfio_pci_cap_remaining_dword(struct
vfio_pci_device*vdev,drivers/vfio/pci/vfio_pci_config.c static ssize_t vfio_config_do_rw(struct vfio_pci_device
Quoted 18 lines
drivers/vfio/pci/vfio_pci_config.c +11
-0return i; } +static bool vfio_pci_need_virt_perm (struct pci_dev *pdev,u8 cap_id) +{ +#ifdef CONFIG_PCI_ATS +return (pdev->is_virtfn && + (cap_id ==PCI_EXT_CAP_ID_PASID || + cap_id ==PCI_EXT_CAP_ID_PRI )); +#else +return false; +#endif +} +static ssize_t vfio_config_do_rw (struct vfio_pci_device *vdev,char __user *buf,size_t count,loff_t *ppos,bool iswrite) {@@ -1781,7 +2101,8 @@ static ssize_t vfio_config_do_rw(struct vfio_pci_device*vdev, char
__user*buf,Quoted 8 lines
drivers/vfio/pci/vfio_pci_config.c static ssize_t vfio_config_do_rw(struct vfio_pci_device *vdev, char __user *buf, · line 2105
Quoted 6 lines
drivers/vfio/pci/vfio_pci_config.c +2
-1if (cap_id ==PCI_CAP_ID_INVALID ) { perm = &unassigned_perms; cap_start = *ppos; - }else if (cap_id ==PCI_CAP_ID_INVALID_VIRT ) { + }else if (cap_id ==PCI_CAP_ID_INVALID_VIRT || +vfio_pci_need_virt_perm (pdev, cap_id)) {Why not simply fill the map with
PCI_CAP_ID_INVALID_VIRT?good idea. thanks.
Quoted 5 lines
drivers/vfio/pci/vfio_pci_config.c static ssize_t vfio_config_do_rw(struct vfio_pci_device *vdev, char __user *buf, · line 2108
Quoted 3 lines
drivers/vfio/pci/vfio_pci_config.c +0
-0perm = &virt_perms; cap_start = *ppos; }else {This is really not close to acceptable as is. Sorry. Thanks,
really sorry for the silly mistakes, I can try to make a new version after we got a decision on whether needs to do data copy and medication before filling into vconfig.
Thanks,
Yi Liu
RE: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
Quoted 2 lines
From: Alex Williamson <[email protected]>
Sent: Saturday, April 4, 2020 1:26 AM
[...]
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1660
Quoted 6 lines
drivers/vfio/pci/vfio_pci_config.c +6
-0+if (!pasid_cap.control_reg.paside) { +pr_debug ("%s: its PF's PASID capability is not enabled\n" , +dev_name (&vdev->pdev->dev)); + ret =0 ; + goto out; + }What happens if the PF's PASID gets disabled while we're using it??
Quoted 7 lines
Quoted 3 lines
This is actually the open I highlighted in cover letter. Per the reply from Baolu, this seems to be an open for bare-metal all the same.
https://lkml.org/lkml/2020/3/31/95Seems that needs to get sorted out before we can expose this. Maybe some sort of registration with the PF driver that PASID is being used by a VF so it cannot be disabled?
I guess we may do vSVA for PF first, and then adding VF vSVA later given above additional need. It's not necessarily to enable both in one step.
[...]
Quoted 20 lines
Quoted 1 line
Quoted 1 line
Review static int vfio_ecap_init(struct
Quoted 1 line
(quoted diff) +0
-0@@ -1604,6 +1901,18 @@ static int vfio_ecap_init(struct
vfio_pci_device*vdev)Quoted 18 lines
Quoted 18 lines
Quoted 4 lines
drivers/vfio/pci/vfio_pci_config.c +1
-0if (!ecaps) *(u32 *)&vdev->vconfig[PCI_CFG_SPACE_SIZE ] =0 ; +#ifdef CONFIG_PCI_ATS drivers/vfio/pci/vfio_pci_config.c static int vfio_ecap_init(struct vfio_pci_device *vdev) · line 1914
Quoted 10 lines
drivers/vfio/pci/vfio_pci_config.c +10
-0+if (pdev->is_virtfn) { +struct pci_dev *physfn = pdev->physfn; + + ret =vfio_pci_add_emulated_cap_for_vf (vdev, + physfn, epos_max, prev); +if (ret) +pr_info ("%s, failed to add special caps for VF %s\n" , + __func__,dev_name (&vdev->pdev->dev)); + } +#endif I can only imagine that we should place the caps at the same location they exist on the PF, we don't know what hidden registers might be hiding in config space.
Is there vendor guarantee that hidden registers will locate at the same offset between PF and VF config space?
Quoted 6 lines
Quoted 5 lines
but we are not sure whether the same location is available on VF. In this patch, it actually places the emulated cap physically behind the cap which lays farthest (its offset is largest) within VF's config space as the PCIe caps are linked in a chain.
Quoted 16 lines
But, as we've found on Broadcom NICs (iirc), hardware developers have a nasty habit of hiding random registers in PCI config space, outside of defined capabilities. I feel like IGD might even do this too, is that true? So I don't think we can guarantee that just because a section of config space isn't part of a defined capability that its unused. It only means that it's unused by common code, but it might have device specific purposes. So of the PCIe spec indicates that VFs cannot include these capabilities and virtialization software needs to emulate them, we need somewhere safe to place them in config space, and simply placing them off the end of known capabilities doesn't give me any confidence. Also, hardware has no requirement to make compact use of extended config space. The first capability must be at
0x100, the very next capability could consume all the way to the last byte of the 4K extended range, and the next link in the chain could be somewhere in the middle. Thanks,
Then what would be a viable option? Vendor nasty habit implies no standard, thus I don't see how VFIO can find a safe location by itself. Also curious how those hidden registers are identified by VFIO and employed with proper r/w policy today. If sort of quirks are used, then could such quirk way be extended to also carry the information about vendor specific safe location? When no such quirk info is provided (the majority case), VFIO then finds out a free location to carry the new cap.
Thanks
Kevin
Re: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
Quoted 46 lines · Kevin
Quoted 2 lines
From: Alex Williamson <[email protected]>
Sent: Saturday, April 4, 2020 1:26 AM[...]
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1660
Quoted 6 lines
drivers/vfio/pci/vfio_pci_config.c +6
-0+if (!pasid_cap.control_reg.paside) { +pr_debug ("%s: its PF's PASID capability is not enabled\n" , +dev_name (&vdev->pdev->dev)); + ret =0 ; + goto out; + }What happens if the PF's PASID gets disabled while we're using it??
Quoted 7 lines
Quoted 3 lines
This is actually the open I highlighted in cover letter. Per the reply from Baolu, this seems to be an open for bare-metal all the same.
https://lkml.org/lkml/2020/3/31/95Seems that needs to get sorted out before we can expose this. Maybe some sort of registration with the PF driver that PASID is being used by a VF so it cannot be disabled?
I guess we may do vSVA for PF first, and then adding VF vSVA later given above additional need. It's not necessarily to enable both in one step.
[...]
Quoted 20 lines
Quoted 1 line
Quoted 1 line
Review static int vfio_ecap_init(struct
Quoted 1 line
(quoted diff) +0
-0@@ -1604,6 +1901,18 @@ static int vfio_ecap_init(struct
vfio_pci_device*vdev)Quoted 18 lines
Quoted 18 lines
Quoted 4 lines
drivers/vfio/pci/vfio_pci_config.c +1
-0if (!ecaps) *(u32 *)&vdev->vconfig[PCI_CFG_SPACE_SIZE ] =0 ; +#ifdef CONFIG_PCI_ATS drivers/vfio/pci/vfio_pci_config.c static int vfio_ecap_init(struct vfio_pci_device *vdev) · line 1914
Quoted 10 lines
drivers/vfio/pci/vfio_pci_config.c +10
-0+if (pdev->is_virtfn) { +struct pci_dev *physfn = pdev->physfn; + + ret =vfio_pci_add_emulated_cap_for_vf (vdev, + physfn, epos_max, prev); +if (ret) +pr_info ("%s, failed to add special caps for VF %s\n" , + __func__,dev_name (&vdev->pdev->dev)); + } +#endif I can only imagine that we should place the caps at the same location they exist on the PF, we don't know what hidden registers might be hiding in config space.
Quoted 2 lines
Is there vendor guarantee that hidden registers will locate at the same offset between PF and VF config space?
I'm not sure if the spec really precludes hidden registers, but the fact that these registers are explicitly outside of the capability chain implies they're only intended for device specific use, so I'd say there are no guarantees about anything related to these registers.
FWIW, vfio started out being more strict about restricting config space access to defined capabilities, until...
commit a7d1ea1c11b33bda2691f3294b4d735ed635535a
Author: Alex Williamson <[email protected]> Date: Mon Apr 1 09:04:12 2013 -0600
vfio-pci: Enable raw access to unassigned config space
Devices like be2net hide registers between the gaps in capabilities and architected regions of PCI config space. Our choices to support such devices is to either build an ever growing and unmanageable white list or rely on hardware isolation to protect us. These registers are really no different than MMIO or I/O port space registers, which we don't attempt to regulate, so treat PCI config space in the same way.
Quoted 22 lines
Quoted 5 lines
Quoted 4 lines
but we are not sure whether the same location is available on VF. In this patch, it actually places the emulated cap physically behind the cap which lays farthest (its offset is largest) within VF's config space as the PCIe caps are linked in a chain.
Quoted 16 lines
But, as we've found on Broadcom NICs (iirc), hardware developers have a nasty habit of hiding random registers in PCI config space, outside of defined capabilities. I feel like IGD might even do this too, is that true? So I don't think we can guarantee that just because a section of config space isn't part of a defined capability that its unused. It only means that it's unused by common code, but it might have device specific purposes. So of the PCIe spec indicates that VFs cannot include these capabilities and virtialization software needs to emulate them, we need somewhere safe to place them in config space, and simply placing them off the end of known capabilities doesn't give me any confidence. Also, hardware has no requirement to make compact use of extended config space. The first capability must be at
0x100, the very next capability could consume all the way to the last byte of the 4K extended range, and the next link in the chain could be somewhere in the middle. Thanks,
Quoted 8 lines
Then what would be a viable option? Vendor nasty habit implies no standard, thus I don't see how VFIO can find a safe location by itself. Also curious how those hidden registers are identified by VFIO and employed with proper r/w policy today. If sort of quirks are used, then could such quirk way be extended to also carry the information about vendor specific safe location? When no such quirk info is provided (the majority case), VFIO then finds out a free location to carry the new cap.
See above commit, rather than quirks we allow raw access to any config space outside of the capability chain. My preference for trying to place virtual capabilities at the same offset as the capability exists on the PF is my impression that the PF config space is often a template for the VF config space. The PF and VF are clearly not independent devices, they share design aspects, and sometimes drivers. Therefore if I was a lazy engineer trying to find a place to hide a register in config space (and ignoring vendor capabilities*), I'd probably put it in the same place on both devices. Thus if we maintain the same capability footprint as the PF, we have a better chance of avoiding them. It's a gamble and maybe we're overthinking it, but this has always been a concern when adding virtual capabilities to a physical device. We can always fail over to an approach where we simply find free space. Thanks,
Alex
* ISTR the Broadcom device implemented the hidden register in standard config space, which was otherwise entirely packed, ie. there was no room for the register to be implemented as a vendor cap.
RE: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
Quoted 112 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, April 7, 2020 11:58 PMQuoted 5 lines · Kevin
Quoted 2 lines
From: Alex Williamson <[email protected]>
Sent: Saturday, April 4, 2020 1:26 AM[...]
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ if (!pasid_cap.control_reg.paside) { + pr_debug("%s: its PF's PASID capability is notenabled\n",
Quoted 25 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1660
Quoted 4 lines
drivers/vfio/pci/vfio_pci_config.c +4
-0+ dev_name(&vdev->pdev->dev)); + ret =0 ; + goto out; + }What happens if the PF's PASID gets disabled while we're using it??
Quoted 7 lines
Quoted 3 lines
This is actually the open I highlighted in cover letter. Per the reply from Baolu, this seems to be an open for bare-metal all the same.
https://lkml.org/lkml/2020/3/31/95Seems that needs to get sorted out before we can expose this. Maybe some sort of registration with the PF driver that PASID is being used by a VF so it cannot be disabled?
I guess we may do vSVA for PF first, and then adding VF vSVA later given above additional need. It's not necessarily to enable both in one step.
[...]
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Review static int vfio_ecap_init(struct
Quoted 1 line
(quoted diff) +0
-0@@ -1604,6 +1901,18 @@ static int vfio_ecap_init(struct
vfio_pci_device*vdev)Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
drivers/vfio/pci/vfio_pci_config.c +0
-0if (!ecaps) *(u32 *)&vdev->vconfig[PCI_CFG_SPACE_SIZE ] =0 ;Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/vfio/pci/vfio_pci_config.c static int vfio_ecap_init(struct vfio_pci_device *vdev)
Quoted 8 lines
drivers/vfio/pci/vfio_pci_config.c +8
-0+#ifdef CONFIG_PCI_ATS +if (pdev->is_virtfn) { +struct pci_dev *physfn = pdev->physfn; + + ret =vfio_pci_add_emulated_cap_for_vf (vdev, + physfn, epos_max, prev); +if (ret) + pr_info("%s, failed to add special caps forVF %s\n",
Quoted 11 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Review static int vfio_ecap_init(struct · line 1912
Quoted 1 line
(quoted diff) +1
-0+ __func__, dev_name(&vdev->pdev-dev));
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
drivers/vfio/pci/vfio_pci_config.c static int vfio_ecap_init(struct vfio_pci_device *vdev) · line 1914
Quoted 2 lines
drivers/vfio/pci/vfio_pci_config.c +2
-0+ } +#endif I can only imagine that we should place the caps at the same location they exist on the PF, we don't know what hidden registers might be hiding in config space.
Is there vendor guarantee that hidden registers will locate at the same offset between PF and VF config space?
I'm not sure if the spec really precludes hidden registers, but the fact that these registers are explicitly outside of the capability chain implies they're only intended for device specific use, so I'd say there are no guarantees about anything related to these registers.
FWIW, vfio started out being more strict about restricting config space access to defined capabilities, until...
commit
a7d1ea1c11b33bda2691f3294b4d735ed635535aAuthor: Alex Williamson <[email protected]> Date: Mon Apr 1 09:04:12 2013 -0600vfio-pci: Enable raw access to unassigned config space
Devices like be2net hide registers between the gaps in capabilities and architected regions of PCI config space. Our choices to support such devices is to either build an ever growing and unmanageable white list or rely on hardware isolation to protect us. These registers are really no different than MMIO or I/O port space registers, which we don't attempt to regulate, so treat PCI config space in the same way.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
but we are not sure whether the same location is available on VF. In this patch, it actually places the emulated cap physically behind the cap which lays farthest (its offset is largest) within VF's config space as the PCIe caps are linked in a chain.
Quoted 9 lines
Quoted 9 lines
But, as we've found on Broadcom NICs (iirc), hardware developers have a nasty habit of hiding random registers in PCI config space, outside of defined capabilities. I feel like IGD might even do this too, is that true? So I don't think we can guarantee that just because a section of config space isn't part of a defined capability that its unused. It only means that it's unused by common code, but it might have device specific purposes. So of the PCIe spec indicates that VFs cannot include these capabilities and virtialization software needs to emulate them, we need somewhere safe to place them in config space,
and
Quoted 2 lines
Quoted 2 lines
simply placing them off the end of known capabilities doesn't give me any confidence. Also, hardware has no requirement to make compact
use
Quoted 6 lines
Quoted 5 lines
of extended config space. The first capability must be at
0x100, the very next capability could consume all the way to the last byte of the 4K extended range, and the next link in the chain could be somewhere in the middle. Thanks,Quoted 8 lines
Then what would be a viable option? Vendor nasty habit implies no standard, thus I don't see how VFIO can find a safe location by itself. Also curious how those hidden registers are identified by VFIO and employed with proper r/w policy today. If sort of quirks are used, then could such quirk way be extended to also carry the information about vendor specific safe location? When no such quirk info is provided (the majority case), VFIO then finds out a free location to carry the new cap.
Quoted 9 lines
See above commit, rather than quirks we allow raw access to any config space outside of the capability chain. My preference for trying to place virtual capabilities at the same offset as the capability exists on the PF is my impression that the PF config space is often a template for the VF config space. The PF and VF are clearly not independent devices, they share design aspects, and sometimes drivers. Therefore if I was a lazy engineer trying to find a place to hide a register in config space (and ignoring vendor capabilities*), I'd probably put it in the same place on both devices. Thus if we maintain the same
We are checking internally whether this assumption makes sense at least for Intel devices which are PASID-capable.
Quoted 5 lines
capability footprint as the PF, we have a better chance of avoiding them. It's a gamble and maybe we're overthinking it, but this has always been a concern when adding virtual capabilities to a physical device. We can always fail over to an approach where we simply find free space. Thanks,
Curious how failover could be triggered in your mind. It's easy to detect conflict with other PCI caps, but not for conflict with hidden registers. The latter can be identified only with device specific knowledge. Possibly in the end we may leverage Yan's vendor ops to find a safe location...
Quoted 6 lines
Alex
* ISTR the Broadcom device implemented the hidden register in standard config space, which was otherwise entirely packed, ie. there was no room for the register to be implemented as a vendor cap.
I suppose such packed design is mostly for PF. Ideally VF is much simpler and the requirement of hidden registers should be much fewer. Otherwise even using same PF offset doesn't work. Long-term it is better for PCISIG to add some recommendations, e.g. for capabilities that are shared between PF/VF, VF config space should still reserve a range at the same location/size as of the PF ones.
Thanks
Kevin
Re: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
In reply to Alex WilliamsonHi Alex
+ Bjorn
FWIW I can't understand why PCI SIG went different ways with ATS, where its enumerated on PF and VF. But for PASID and PRI its only in PF.
I'm checking with our internal SIG reps to followup on that.
Quoted 7 lines · Alex Williamson
Quoted 2 lines
Is there vendor guarantee that hidden registers will locate at the same offset between PF and VF config space?
I'm not sure if the spec really precludes hidden registers, but the fact that these registers are explicitly outside of the capability chain implies they're only intended for device specific use, so I'd say there are no guarantees about anything related to these registers.
As you had suggested in the other thread, we could consider using the same offset as in PF, but even that's a better guess still not reliable.
The other option is to maybe extend driver ops in the PF to expose where the offsets should be. Sort of adding the quirk in the implementation.
I'm not sure how prevalent are PASID and PRI in VF devices. If SIG is resisting making VF's first class citizen, we might ask them to add some verbiage to suggest leave the same offsets as PF open to help emulation software.
Quoted 8 lines
FWIW, vfio started out being more strict about restricting config space access to defined capabilities, until...
commit
a7d1ea1c11b33bda2691f3294b4d735ed635535aAuthor: Alex Williamson <[email protected]> Date: Mon Apr 1 09:04:12 2013 -0600
Cheers,
Ashok
Re: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
Quoted 3 lines · Ashok
Hi Alex
+ Bjorn
+ Don
Quoted 19 lines
FWIW I can't understand why PCI SIG went different ways with ATS, where its enumerated on PF and VF. But for PASID and PRI its only in PF.
I'm checking with our internal SIG reps to followup on that.
Quoted 7 lines · Alex Williamson
Quoted 2 lines
Is there vendor guarantee that hidden registers will locate at the same offset between PF and VF config space?
I'm not sure if the spec really precludes hidden registers, but the fact that these registers are explicitly outside of the capability chain implies they're only intended for device specific use, so I'd say there are no guarantees about anything related to these registers.
As you had suggested in the other thread, we could consider using the same offset as in PF, but even that's a better guess still not reliable.
Quoted 7 lines
The other option is to maybe extend driver ops in the PF to expose where the offsets should be. Sort of adding the quirk in the implementation.
I'm not sure how prevalent are PASID and PRI in VF devices. If SIG is resisting making VF's first class citizen, we might ask them to add some verbiage to suggest leave the same offsets as PF open to help emulation software.
Even if we know where to expose these capabilities on the VF, it's not clear to me how we can actually virtualize the capability itself. If the spec defines, for example, an enable bit as r/w then software that interacts with that register expects the bit is settable. There's no protocol for "try to set the bit and re-read it to see if the hardware accepted it". Therefore a capability with a fixed enable bit representing the state of the PF, not settable by the VF, is disingenuous to the spec.
If what we're trying to do is expose that PASID and PRI are enabled on the PF to a VF driver, maybe duplicating the PF capabilities on the VF without the ability to control it is not the right approach. Maybe we need new capabilities exposing these as slave features that cannot be controlled? We could define our own vendor capability for this, but of course we have both the where to put it in config space issue, as well as the issue of trying to push an ad-hoc standard. vfio could expose these as device features rather than emulating capabilities, but that still leaves a big gap between vfio in the hypervisor and the driver in the guest VM. That might still help push the responsibility and policy for how to expose it to the VM as a userspace problem though.
I agree though, I don't know why the SIG would preclude implementing per VF control of these features. Thanks,
Alex
Quoted 11 lines
Quoted 7 lines
FWIW, vfio started out being more strict about restricting config space access to defined capabilities, until...
commit
a7d1ea1c11b33bda2691f3294b4d735ed635535aAuthor: Alex Williamson <[email protected]> Date: Mon Apr 1 09:04:12 2013 -0600Cheers,
Ashok
Re: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
Hi Alex
Quoted 45 lines · Alex Williamson
Quoted 3 lines · Ashok
Hi Alex
+ Bjorn
+ Don
Quoted 19 lines
FWIW I can't understand why PCI SIG went different ways with ATS, where its enumerated on PF and VF. But for PASID and PRI its only in PF.
I'm checking with our internal SIG reps to followup on that.
Quoted 7 lines · Alex Williamson
Quoted 2 lines
Is there vendor guarantee that hidden registers will locate at the same offset between PF and VF config space?
I'm not sure if the spec really precludes hidden registers, but the fact that these registers are explicitly outside of the capability chain implies they're only intended for device specific use, so I'd say there are no guarantees about anything related to these registers.
As you had suggested in the other thread, we could consider using the same offset as in PF, but even that's a better guess still not reliable.
Quoted 7 lines
The other option is to maybe extend driver ops in the PF to expose where the offsets should be. Sort of adding the quirk in the implementation.
I'm not sure how prevalent are PASID and PRI in VF devices. If SIG is resisting making VF's first class citizen, we might ask them to add some verbiage to suggest leave the same offsets as PF open to help emulation software.
Even if we know where to expose these capabilities on the VF, it's not clear to me how we can actually virtualize the capability itself. If the spec defines, for example, an enable bit as r/w then software that interacts with that register expects the bit is settable. There's no protocol for "try to set the bit and re-read it to see if the hardware accepted it". Therefore a capability with a fixed enable bit representing the state of the PF, not settable by the VF, is disingenuous to the spec.
Quoted 5 lines
If what we're trying to do is expose that PASID and PRI are enabled on the PF to a VF driver, maybe duplicating the PF capabilities on the VF without the ability to control it is not the right approach. Maybe we need new capabilities exposing these as slave features that cannot be controlled? We could define our own vendor capability for this, but of
The other option is to say, VFIO would never emulate these fake capablities. If exposing a VF with PASID/PRI is required the PF driver would simply wrap it into a VDCM like model which we do today for Scalable IOV devices. So PF handles all aspects of this interface.
I also like the suggestion you propose, maybe an offset where these capabilities are exposed to VF's. Maybe have an architected DEVCAPx which exposes these RO capabilities. No control, and the offset should be preserved by the SIG, so VMM can have a safe place to stash them.
Quoted 9 lines
course we have both the where to put it in config space issue, as well as the issue of trying to push an ad-hoc standard. vfio could expose these as device features rather than emulating capabilities, but that still leaves a big gap between vfio in the hypervisor and the driver in the guest VM. That might still help push the responsibility and policy for how to expose it to the VM as a userspace problem though.
I agree though, I don't know why the SIG would preclude implementing per VF control of these features. Thanks,
Even if we ask SIG for clarification, it might affect today's devices So might not be useful to solve our current situation.
Ashok
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
In reply to Alex WilliamsonQuoted 36 lines · Alex Williamson
Quoted 3 lines · Ashok
Hi Alex
+ Bjorn
+ Don
Quoted 19 lines
FWIW I can't understand why PCI SIG went different ways with ATS, where its enumerated on PF and VF. But for PASID and PRI its only in PF.
I'm checking with our internal SIG reps to followup on that.
Quoted 7 lines · Alex Williamson
Quoted 2 lines
Is there vendor guarantee that hidden registers will locate at the same offset between PF and VF config space?
I'm not sure if the spec really precludes hidden registers, but the fact that these registers are explicitly outside of the capability chain implies they're only intended for device specific use, so I'd say there are no guarantees about anything related to these registers.
As you had suggested in the other thread, we could consider using the same offset as in PF, but even that's a better guess still not reliable.
Quoted 7 lines
The other option is to maybe extend driver ops in the PF to expose where the offsets should be. Sort of adding the quirk in the implementation.
I'm not sure how prevalent are PASID and PRI in VF devices. If SIG is resisting making VF's first class citizen, we might ask them to add some verbiage to suggest leave the same offsets as PF open to help emulation software.
Quoted 8 lines
Even if we know where to expose these capabilities on the VF, it's not clear to me how we can actually virtualize the capability itself. If the spec defines, for example, an enable bit as r/w then software that interacts with that register expects the bit is settable. There's no protocol for "try to set the bit and re-read it to see if the hardware accepted it". Therefore a capability with a fixed enable bit representing the state of the PF, not settable by the VF, is disingenuous to the spec.
Would it be OK to implement a lock down mechanism for the PF PASID capability, preventing changes to the PF cap when the VF is in use by VFIO? The emulation would still break the spec: since the PF cap would always be enabled the VF configuration bits would have no effect, but it seems preferable to having the Enable bit not enable anything.
Quoted 12 lines
If what we're trying to do is expose that PASID and PRI are enabled on the PF to a VF driver, maybe duplicating the PF capabilities on the VF without the ability to control it is not the right approach. Maybe we need new capabilities exposing these as slave features that cannot be controlled? We could define our own vendor capability for this, but of course we have both the where to put it in config space issue, as well as the issue of trying to push an ad-hoc standard. vfio could expose these as device features rather than emulating capabilities, but that still leaves a big gap between vfio in the hypervisor and the driver in the guest VM. That might still help push the responsibility and policy for how to expose it to the VM as a userspace problem though.
Userspace might have more difficulty working around the issues mentioned in this thread. They would still need a guarantee that the PF PASID configuration doesn't change at runtime, and they wouldn't have the ability to talk to a vendor driver to figure out where to place the fake PASID capability.
Thanks,
Jean
Quoted 18 lines
I agree though, I don't know why the SIG would preclude implementing per VF control of these features. Thanks,
Alex
Quoted 11 lines
Quoted 7 lines
FWIW, vfio started out being more strict about restricting config space access to defined capabilities, until...
commit
a7d1ea1c11b33bda2691f3294b4d735ed635535aAuthor: Alex Williamson <[email protected]> Date: Mon Apr 1 09:04:12 2013 -0600Cheers,
Ashok
Re: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
Quoted 46 lines · On Thu, 9 Apr 2020 09:35:33 +0200 Jean-Philippe Brucker…
Quoted 36 lines · Alex Williamson
Quoted 3 lines · Ashok
Hi Alex
+ Bjorn
+ Don
Quoted 19 lines
FWIW I can't understand why PCI SIG went different ways with ATS, where its enumerated on PF and VF. But for PASID and PRI its only in PF.
I'm checking with our internal SIG reps to followup on that.
Quoted 7 lines · Alex Williamson
Quoted 2 lines
Is there vendor guarantee that hidden registers will locate at the same offset between PF and VF config space?
I'm not sure if the spec really precludes hidden registers, but the fact that these registers are explicitly outside of the capability chain implies they're only intended for device specific use, so I'd say there are no guarantees about anything related to these registers.
As you had suggested in the other thread, we could consider using the same offset as in PF, but even that's a better guess still not reliable.
Quoted 7 lines
The other option is to maybe extend driver ops in the PF to expose where the offsets should be. Sort of adding the quirk in the implementation.
I'm not sure how prevalent are PASID and PRI in VF devices. If SIG is resisting making VF's first class citizen, we might ask them to add some verbiage to suggest leave the same offsets as PF open to help emulation software.
Quoted 8 lines
Even if we know where to expose these capabilities on the VF, it's not clear to me how we can actually virtualize the capability itself. If the spec defines, for example, an enable bit as r/w then software that interacts with that register expects the bit is settable. There's no protocol for "try to set the bit and re-read it to see if the hardware accepted it". Therefore a capability with a fixed enable bit representing the state of the PF, not settable by the VF, is disingenuous to the spec.
Quoted 5 lines
Would it be OK to implement a lock down mechanism for the PF PASID capability, preventing changes to the PF cap when the VF is in use by VFIO? The emulation would still break the spec: since the PF cap would always be enabled the VF configuration bits would have no effect, but it seems preferable to having the Enable bit not enable anything.
I think we absolutely need some mechanism to make sure the PF driver doesn't change the PASID enable bit while we're using it. A PASID user registration perhaps. And yes, that doesn't necessarily map to being able to actually disable PASID, but it sounds like Kevin and Ashok have some ideas how the emulation could use the IOMMU settings to achieve that more precisely.
Quoted 12 lines
Quoted 11 lines
If what we're trying to do is expose that PASID and PRI are enabled on the PF to a VF driver, maybe duplicating the PF capabilities on the VF without the ability to control it is not the right approach. Maybe we need new capabilities exposing these as slave features that cannot be controlled? We could define our own vendor capability for this, but of course we have both the where to put it in config space issue, as well as the issue of trying to push an ad-hoc standard. vfio could expose these as device features rather than emulating capabilities, but that still leaves a big gap between vfio in the hypervisor and the driver in the guest VM. That might still help push the responsibility and policy for how to expose it to the VM as a userspace problem though.
Quoted 5 lines
Userspace might have more difficulty working around the issues mentioned in this thread. They would still need a guarantee that the PF PASID configuration doesn't change at runtime, and they wouldn't have the ability to talk to a vendor driver to figure out where to place the fake PASID capability.
Couldn't we do this with the DEVICE_FEATURE ioctl we just added? A user could set a PASID user or get a capability offset, where vfio-pci would plumb these through to whatever PF/vendor driver provides that interface, just as if it was implemented in the vfio-pci driver. But that still lets us leave the policy of inserting a capability and using the IOMMU to implement the bits to the user/QEMU. Thanks,
Alex
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
In reply to Alex WilliamsonQuoted 36 lines · Alex Williamson
Quoted 3 lines · Ashok
Hi Alex
+ Bjorn
+ Don
Quoted 19 lines
FWIW I can't understand why PCI SIG went different ways with ATS, where its enumerated on PF and VF. But for PASID and PRI its only in PF.
I'm checking with our internal SIG reps to followup on that.
Quoted 7 lines · Alex Williamson
Quoted 2 lines
Is there vendor guarantee that hidden registers will locate at the same offset between PF and VF config space?
I'm not sure if the spec really precludes hidden registers, but the fact that these registers are explicitly outside of the capability chain implies they're only intended for device specific use, so I'd say there are no guarantees about anything related to these registers.
As you had suggested in the other thread, we could consider using the same offset as in PF, but even that's a better guess still not reliable.
Quoted 7 lines
The other option is to maybe extend driver ops in the PF to expose where the offsets should be. Sort of adding the quirk in the implementation.
I'm not sure how prevalent are PASID and PRI in VF devices. If SIG is resisting making VF's first class citizen, we might ask them to add some verbiage to suggest leave the same offsets as PF open to help emulation software.
Quoted 8 lines
Even if we know where to expose these capabilities on the VF, it's not clear to me how we can actually virtualize the capability itself. If the spec defines, for example, an enable bit as r/w then software that interacts with that register expects the bit is settable. There's no protocol for "try to set the bit and re-read it to see if the hardware accepted it". Therefore a capability with a fixed enable bit representing the state of the PF, not settable by the VF, is disingenuous to the spec.
I think we are all in violent agreement. A lot of times the pci spec gets defined several years ahead of real products and no one remembers the justification on why they restricted things the way they did.
Maybe someone early product wasn't quite exposing these features to the VF and hence the spec is bug compatible :-)
Quoted 4 lines
If what we're trying to do is expose that PASID and PRI are enabled on the PF to a VF driver, maybe duplicating the PF capabilities on the VF without the ability to control it is not the right approach. Maybe we
As long as the capability enable is only provided when the PF has enabled the feature. Then it seems the hardware seems to do the right thing.
Assume we expose PASID/PRI only when PF has enabled it. It will be the case since the PF driver needs to exist, and IOMMU would have set the PASID/PRI/ATS on PF.
If the emulation is purely spoofing the capability. Once vIOMMU driver enables PASID, the context entries for the VF are completely independent from the PF context entries.
vIOMMU would enable PASID, and we just spoof the PASID capability.
If vIOMMU or guest for some reason does disable_pasid(), then the vIOMMU driver can disaable PASID on the VF context entries. So the VF although the capability is blanket enabled on PF, IOMMU gaurantees the transactions are blocked.
In the interim, it seems like the intent of the virtual capability can be honored via help from the IOMMU for the controlling aspect..
Did i miss anything?
Quoted 8 lines
need new capabilities exposing these as slave features that cannot be controlled? We could define our own vendor capability for this, but of course we have both the where to put it in config space issue, as well as the issue of trying to push an ad-hoc standard. vfio could expose these as device features rather than emulating capabilities, but that still leaves a big gap between vfio in the hypervisor and the driver in the guest VM. That might still help push the responsibility and policy for how to expose it to the VM as a userspace problem though.
I think this is a good long term solution, but if the vIOMMU implenentations can carry us for the time being, we can probably defer them unless we are stuck.
Quoted 4 lines
I agree though, I don't know why the SIG would preclude implementing per VF control of these features. Thanks,
Cheers,
Ashok
Re: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
Hi Alex
Going through the PCIe Spec, there seems a lot of such capabilities that are different between PF and VF. Some that make sense and some don't.
Quoted 5 lines · Ashok
Quoted 5 lines
I agree though, I don't know why the SIG would preclude implementing per VF control of these features. Thanks,
For e.g.
VF doesn't have I/O and Mem space enables, but has BME Interrupt Status
Correctable Error Reporting
Almost all of Device Control Register.
So it seems like there is a ton of them we have to deal with today for VF's. How do we manage to emulate them without any support for them in VF's?
Quoted 3 lines
Cheers,
Ashok
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
Quoted 10 lines · Ashok
Hi Alex
Going through the PCIe Spec, there seems a lot of such capabilities that are different between PF and VF. Some that make sense and some don't.
Quoted 2 lines · Ashok
Quoted 2 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
I agree though, I don't know why the SIG would preclude implementing per VF control of these features. Thanks,
For e.g.
VF doesn't have I/O and Mem space enables, but has BME
VFs don't have I/O, so I/O enable is irrelevant. The memory enable bit is emulated, so it doesn't really do anything from the VM perspective. The hypervisor could provide more emulation around this, but it hasn't proven necessary.
Quoted 1 line
Interrupt Status
VFs don't have INTx, so this is irrelevant.
Quoted 2 lines
Correctable Error Reporting
Almost all of Device Control Register.
Are we doing anything to virtualize these for VFs? I think we've addressed access control to these for PFs, but I don't see that we try to virtualize them for the VF.
Quoted 3 lines
So it seems like there is a ton of them we have to deal with today for VF's. How do we manage to emulate them without any support for them in VF's?
The memory enable bit is just access to the MMIO space of the device, the hypervisor could choose to do more, but currently emulating the bit itself is sufficient. This doesn't really affect the device, just access to the device. The device control registers, I don't think we've had a need to virtualize them yet and I think we'd run into many of the same questions. If your point is that there exists gaps in the spec that make things difficult to virtualize, I won't argue with you there. MPS is a nearby one that's difficult to virtualize on the PF since its setting needs to take entire communication channels into account.
So far though we aren't inventing new capabilities to add to VF config space and pretending they work, we're just stumbling on what the VF exposes whether on bare metal or in a VM. Thanks,
Alex
RE: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
In reply to Raj, AshokQuoted 82 lines
From: Raj, Ashok <[email protected]>
Sent: Monday, April 13, 2020 11:11 AMQuoted 33 lines · Alex Williamson
Quoted 3 lines · Ashok
Hi Alex
+ Bjorn
+ Don
Quoted 24 lines
FWIW I can't understand why PCI SIG went different ways with ATS, where its enumerated on PF and VF. But for PASID and PRI its only in PF.
I'm checking with our internal SIG reps to followup on that.
Quoted 7 lines · Alex Williamson
Quoted 2 lines
Is there vendor guarantee that hidden registers will locate at the same offset between PF and VF config space?
I'm not sure if the spec really precludes hidden registers, but the fact that these registers are explicitly outside of the capability chain implies they're only intended for device specific use, so I'd say there are no guarantees about anything related to these registers.
As you had suggested in the other thread, we could consider using the same offset as in PF, but even that's a better guess still not reliable.
The other option is to maybe extend driver ops in the PF to expose where the offsets should be. Sort of adding the quirk in the implementation.
I'm not sure how prevalent are PASID and PRI in VF devices. If SIG is
resisting
Quoted 11 lines
Quoted 2 lines
making VF's first class citizen, we might ask them to add some verbiage to suggest leave the same offsets as PF open to help emulation software.
Even if we know where to expose these capabilities on the VF, it's not clear to me how we can actually virtualize the capability itself. If the spec defines, for example, an enable bit as r/w then software that interacts with that register expects the bit is settable. There's no protocol for "try to set the bit and re-read it to see if the hardware accepted it". Therefore a capability with a fixed enable bit representing the state of the PF, not settable by the VF, is disingenuous to the spec.
I think we are all in violent agreement. A lot of times the pci spec gets defined several years ahead of real products and no one remembers the justification on why they restricted things the way they did.
Maybe someone early product wasn't quite exposing these features to the VF
and hence the spec is bug compatible :-)Quoted 4 lines
If what we're trying to do is expose that PASID and PRI are enabled on the PF to a VF driver, maybe duplicating the PF capabilities on the VF without the ability to control it is not the right approach. Maybe we
As long as the capability enable is only provided when the PF has enabled the feature. Then it seems the hardware seems to do the right thing.
Assume we expose PASID/PRI only when PF has enabled it. It will be the case since the PF driver needs to exist, and IOMMU would have set the PASID/PRI/ATS on PF.
If the emulation is purely spoofing the capability. Once vIOMMU driver enables PASID, the context entries for the VF are completely independent from the PF context entries.
vIOMMU would enable PASID, and we just spoof the PASID capability.
If vIOMMU or guest for some reason does
disable_pasid(), then the vIOMMU driver can disaable PASID on the VF context entries. So the VF although the capability is blanket enabled on PF, IOMMU gaurantees the transactions are blocked.
Quoted 4 lines
In the interim, it seems like the intent of the virtual capability can be honored via help from the IOMMU for the controlling aspect..
Did i miss anything?
Above works for emulating the enable bit (under the assumption that PF driver won't disable pasid when vf is assigned). However, there are also "Execute permission enable" and "Privileged mode enable" bits in PASID control registers. I don't know how those bits could be cleanly emulated when the guest writes a value different from PF's...
Similar problem also exists when talking about PRI emulation, e.g. to enable PRI the software usually waits until the 'stopped' bit is set (indicating all previously issued requests have completed). How to emulate this bit accurately when one guest toggles the enable bit while the PF and other VFs are actively issuing page requests through the shared page request interface? from pcie spec I didn't find a way to catch when all previously-issued requests from a specific VF have completed. Can a conservative big-enough timeout value help here? I don't know... similar puzzle also exists for emulating the 'reset' control bit which is supposed to clear the pending request state for the whole page request interface.
I feel the main problem in pcie spec is that, while they invent SR-IOV to address I/O virtualization requirement (where strict isolation is required), they blurred the boundary by leaving some shared resource cross PF and VFs which imply sort of cooperation between PF and VF drivers. On bare metal such cooperation is easy to build, by enabling/ disabling a capability en mass, by using the same set of setting, etc. However it doesn't consider the virtualization case where a VF is assigned to the guest which considers the VF as a standard PCI/PCIe endpoint thus such cooperation is missing. A vendor capability could help fix the gap here but making it adopted by major guest OSes will take time. But honestly speaking I don't know other good alternative now... :/
Quoted 15 lines
Quoted 9 lines
need new capabilities exposing these as slave features that cannot be controlled? We could define our own vendor capability for this, but of course we have both the where to put it in config space issue, as well as the issue of trying to push an ad-hoc standard. vfio could expose these as device features rather than emulating capabilities, but that still leaves a big gap between vfio in the hypervisor and the driver in the guest VM. That might still help push the responsibility and policy for how to expose it to the VM as a userspace problem though.
I think this is a good long term solution, but if the vIOMMU implenentations can carry us for the time being, we can probably defer them unless we are stuck.
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
I agree though, I don't know why the SIG would preclude implementing per VF control of these features. Thanks,
Cheers,
Ashok
Thanks
Kevin
RE: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
In reply to Raj, AshokQuoted 88 lines
From: Tian, Kevin
Sent: Monday, April 13, 2020 3:55 PMQuoted 85 lines
From: Raj, Ashok <[email protected]>
Sent: Monday, April 13, 2020 11:11 AMQuoted 33 lines · Alex Williamson
Quoted 3 lines · Ashok
Hi Alex
+ Bjorn
+ Don
Quoted 24 lines
FWIW I can't understand why PCI SIG went different ways with ATS, where its enumerated on PF and VF. But for PASID and PRI its only in PF.
I'm checking with our internal SIG reps to followup on that.
Quoted 7 lines · Alex Williamson
Quoted 2 lines
Is there vendor guarantee that hidden registers will locate at the same offset between PF and VF config space?
I'm not sure if the spec really precludes hidden registers, but the fact that these registers are explicitly outside of the capability chain implies they're only intended for device specific use, so I'd say there are no guarantees about anything related to these registers.
As you had suggested in the other thread, we could consider using the same offset as in PF, but even that's a better guess still not reliable.
The other option is to maybe extend driver ops in the PF to expose where the offsets should be. Sort of adding the quirk in the implementation.
I'm not sure how prevalent are PASID and PRI in VF devices. If SIG is
resisting
Quoted 11 lines
Quoted 2 lines
making VF's first class citizen, we might ask them to add some verbiage to suggest leave the same offsets as PF open to help emulation software.
Even if we know where to expose these capabilities on the VF, it's not clear to me how we can actually virtualize the capability itself. If the spec defines, for example, an enable bit as r/w then software that interacts with that register expects the bit is settable. There's no protocol for "try to set the bit and re-read it to see if the hardware accepted it". Therefore a capability with a fixed enable bit representing the state of the PF, not settable by the VF, is disingenuous to the spec.
I think we are all in violent agreement. A lot of times the pci spec gets defined several years ahead of real products and no one remembers the justification on why they restricted things the way they did.
Maybe someone early product wasn't quite exposing these features to the VF
and hence the spec is bug compatible :-)Quoted 4 lines
If what we're trying to do is expose that PASID and PRI are enabled on the PF to a VF driver, maybe duplicating the PF capabilities on the VF without the ability to control it is not the right approach. Maybe we
As long as the capability enable is only provided when the PF has enabled the feature. Then it seems the hardware seems to do the right thing.
Assume we expose PASID/PRI only when PF has enabled it. It will be the case since the PF driver needs to exist, and IOMMU would have set the PASID/PRI/ATS on PF.
If the emulation is purely spoofing the capability. Once vIOMMU driver enables PASID, the context entries for the VF are completely independent from the PF context entries.
vIOMMU would enable PASID, and we just spoof the PASID capability.
If vIOMMU or guest for some reason does
disable_pasid(), then the vIOMMU driver can disaable PASID on the VF context entries. So the VF although the capability is blanket enabled on PF, IOMMU gaurantees the transactions are blocked.In the interim, it seems like the intent of the virtual capability can be honored via help from the IOMMU for the controlling aspect..
Quoted 7 lines
Quoted 1 line
Did i miss anything?
Above works for emulating the enable bit (under the assumption that PF driver won't disable pasid when vf is assigned). However, there are also "Execute permission enable" and "Privileged mode enable" bits in PASID control registers. I don't know how those bits could be cleanly emulated when the guest writes a value different from PF's...
sent too quick. the IOMMU also includes control bits for allowing/ blocking execute requests and supervisor requests. We can rely on IOMMU to block those requests to emulate the disabled cases of all three control bits in the pasid cap.
Thanks
Kevin
Quoted 45 lines
Similar problem also exists when talking about PRI emulation, e.g. to enable PRI the software usually waits until the 'stopped' bit is set (indicating all previously issued requests have completed). How to emulate this bit accurately when one guest toggles the enable bit while the PF and other VFs are actively issuing page requests through the shared page request interface? from pcie spec I didn't find a way to catch when all previously-issued requests from a specific VF have completed. Can a conservative big-enough timeout value help here? I don't know... similar puzzle also exists for emulating the 'reset' control bit which is supposed to clear the pending request state for the whole page request interface.
I feel the main problem in pcie spec is that, while they invent SR-IOV to address I/O virtualization requirement (where strict isolation is required), they blurred the boundary by leaving some shared resource cross PF and VFs which imply sort of cooperation between PF and VF drivers. On bare metal such cooperation is easy to build, by enabling/ disabling a capability en mass, by using the same set of setting, etc. However it doesn't consider the virtualization case where a VF is assigned to the guest which considers the VF as a standard PCI/PCIe endpoint thus such cooperation is missing. A vendor capability could help fix the gap here but making it adopted by major guest OSes will take time. But honestly speaking I don't know other good alternative now... :/
Quoted 19 lines
Quoted 9 lines
need new capabilities exposing these as slave features that cannot be controlled? We could define our own vendor capability for this, but of course we have both the where to put it in config space issue, as well as the issue of trying to push an ad-hoc standard. vfio could expose these as device features rather than emulating capabilities, but that still leaves a big gap between vfio in the hypervisor and the driver in the guest VM. That might still help push the responsibility and policy for how to expose it to the VM as a userspace problem though.
I think this is a good long term solution, but if the vIOMMU implenentations can carry us for the time being, we can probably defer them unless we are stuck.
Quoted 4 lines
I agree though, I don't know why the SIG would preclude implementing per VF control of these features. Thanks,
Quoted 5 lines
Quoted 2 lines
Cheers,
AshokThanks
Kevin
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
Quoted 96 lines · Kevin
Quoted 88 lines
From: Tian, Kevin
Sent: Monday, April 13, 2020 3:55 PMQuoted 85 lines
From: Raj, Ashok <[email protected]>
Sent: Monday, April 13, 2020 11:11 AMQuoted 33 lines · Alex Williamson
Quoted 3 lines · Ashok
Hi Alex
+ Bjorn
+ Don
Quoted 24 lines
FWIW I can't understand why PCI SIG went different ways with ATS, where its enumerated on PF and VF. But for PASID and PRI its only in PF.
I'm checking with our internal SIG reps to followup on that.
Quoted 7 lines · Alex Williamson
Quoted 2 lines
Is there vendor guarantee that hidden registers will locate at the same offset between PF and VF config space?
I'm not sure if the spec really precludes hidden registers, but the fact that these registers are explicitly outside of the capability chain implies they're only intended for device specific use, so I'd say there are no guarantees about anything related to these registers.
As you had suggested in the other thread, we could consider using the same offset as in PF, but even that's a better guess still not reliable.
The other option is to maybe extend driver ops in the PF to expose where the offsets should be. Sort of adding the quirk in the implementation.
I'm not sure how prevalent are PASID and PRI in VF devices. If SIG is
resisting
Quoted 11 lines
Quoted 2 lines
making VF's first class citizen, we might ask them to add some verbiage to suggest leave the same offsets as PF open to help emulation software.
Even if we know where to expose these capabilities on the VF, it's not clear to me how we can actually virtualize the capability itself. If the spec defines, for example, an enable bit as r/w then software that interacts with that register expects the bit is settable. There's no protocol for "try to set the bit and re-read it to see if the hardware accepted it". Therefore a capability with a fixed enable bit representing the state of the PF, not settable by the VF, is disingenuous to the spec.
I think we are all in violent agreement. A lot of times the pci spec gets defined several years ahead of real products and no one remembers the justification on why they restricted things the way they did.
Maybe someone early product wasn't quite exposing these features to the VF
and hence the spec is bug compatible :-)Quoted 4 lines
If what we're trying to do is expose that PASID and PRI are enabled on the PF to a VF driver, maybe duplicating the PF capabilities on the VF without the ability to control it is not the right approach. Maybe we
As long as the capability enable is only provided when the PF has enabled the feature. Then it seems the hardware seems to do the right thing.
Assume we expose PASID/PRI only when PF has enabled it. It will be the case since the PF driver needs to exist, and IOMMU would have set the PASID/PRI/ATS on PF.
If the emulation is purely spoofing the capability. Once vIOMMU driver enables PASID, the context entries for the VF are completely independent from the PF context entries.
vIOMMU would enable PASID, and we just spoof the PASID capability.
If vIOMMU or guest for some reason does
disable_pasid(), then the vIOMMU driver can disaable PASID on the VF context entries. So the VF although the capability is blanket enabled on PF, IOMMU gaurantees the transactions are blocked.In the interim, it seems like the intent of the virtual capability can be honored via help from the IOMMU for the controlling aspect..
Quoted 7 lines
Quoted 1 line
Did i miss anything?
Above works for emulating the enable bit (under the assumption that PF driver won't disable pasid when vf is assigned). However, there are also "Execute permission enable" and "Privileged mode enable" bits in PASID control registers. I don't know how those bits could be cleanly emulated when the guest writes a value different from PF's...
Quoted 4 lines
sent too quick. the IOMMU also includes control bits for allowing/ blocking execute requests and supervisor requests. We can rely on IOMMU to block those requests to emulate the disabled cases of all three control bits in the pasid cap.
So if the emulation of the PASID capability takes into account the IOMMU configuration to back that emulation, shouldn't we do that emulation in the hypervisor, ie. QEMU, rather than the kernel vfio layer? Thanks,
Alex
Quoted 50 lines
Quoted 49 lines
Similar problem also exists when talking about PRI emulation, e.g. to enable PRI the software usually waits until the 'stopped' bit is set (indicating all previously issued requests have completed). How to emulate this bit accurately when one guest toggles the enable bit while the PF and other VFs are actively issuing page requests through the shared page request interface? from pcie spec I didn't find a way to catch when all previously-issued requests from a specific VF have completed. Can a conservative big-enough timeout value help here? I don't know... similar puzzle also exists for emulating the 'reset' control bit which is supposed to clear the pending request state for the whole page request interface.
I feel the main problem in pcie spec is that, while they invent SR-IOV to address I/O virtualization requirement (where strict isolation is required), they blurred the boundary by leaving some shared resource cross PF and VFs which imply sort of cooperation between PF and VF drivers. On bare metal such cooperation is easy to build, by enabling/ disabling a capability en mass, by using the same set of setting, etc. However it doesn't consider the virtualization case where a VF is assigned to the guest which considers the VF as a standard PCI/PCIe endpoint thus such cooperation is missing. A vendor capability could help fix the gap here but making it adopted by major guest OSes will take time. But honestly speaking I don't know other good alternative now... :/
Quoted 15 lines
Quoted 9 lines
need new capabilities exposing these as slave features that cannot be controlled? We could define our own vendor capability for this, but of course we have both the where to put it in config space issue, as well as the issue of trying to push an ad-hoc standard. vfio could expose these as device features rather than emulating capabilities, but that still leaves a big gap between vfio in the hypervisor and the driver in the guest VM. That might still help push the responsibility and policy for how to expose it to the VM as a userspace problem though.
I think this is a good long term solution, but if the vIOMMU implenentations can carry us for the time being, we can probably defer them unless we are stuck.
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
I agree though, I don't know why the SIG would preclude implementing per VF control of these features. Thanks,
Cheers,
AshokThanks
Kevin
RE: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
Quoted 115 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, April 14, 2020 3:21 AMQuoted 22 lines · Kevin
Quoted 22 lines
From: Tian, Kevin
Sent: Monday, April 13, 2020 3:55 PMQuoted 19 lines
From: Raj, Ashok <[email protected]>
Sent: Monday, April 13, 2020 11:11 AMQuoted 15 lines · Alex Williamson
Quoted 3 lines · Ashok
Hi Alex
+ Bjorn
+ Don
Quoted 6 lines
FWIW I can't understand why PCI SIG went different ways with ATS, where its enumerated on PF and VF. But for PASID and PRI its only in PF.
I'm checking with our internal SIG reps to followup on that.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines · Alex Williamson
Quoted 6 lines
Quoted 2 lines
Is there vendor guarantee that hidden registers will locate at the same offset between PF and VF config space?
I'm not sure if the spec really precludes hidden registers, but the fact that these registers are explicitly outside of the capability chain implies they're only intended for device specific use, so I'd
say
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
there are no guarantees about anything related to these registers.
As you had suggested in the other thread, we could consider using the same offset as in PF, but even that's a better guess still not reliable.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
The other option is to maybe extend driver ops in the PF to expose where the offsets should be. Sort of adding the quirk in the implementation.
I'm not sure how prevalent are PASID and PRI in VF devices. If SIG is
resisting
Quoted 1 line
Quoted 1 line
making VF's first class citizen, we might ask them to add some
verbiage
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to suggest leave the same offsets as PF open to help emulation
software.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Even if we know where to expose these capabilities on the VF, it's not clear to me how we can actually virtualize the capability itself. If the spec defines, for example, an enable bit as r/w then software that interacts with that register expects the bit is settable. There's no protocol for "try to set the bit and re-read it to see if the hardware accepted it". Therefore a capability with a fixed enable bit representing the state of the PF, not settable by the VF, is disingenuous to the spec.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
I think we are all in violent agreement. A lot of times the pci spec gets defined several years ahead of real products and no one remembers the justification on why they restricted things the way they did.
Maybe someone early product wasn't quite exposing these features to
the
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
VF
and hence the spec is bug compatible :-)Quoted 4 lines
If what we're trying to do is expose that PASID and PRI are enabled on the PF to a VF driver, maybe duplicating the PF capabilities on the VF without the ability to control it is not the right approach. Maybe we
As long as the capability enable is only provided when the PF has
enabled
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
the feature. Then it seems the hardware seems to do the right thing.
Assume we expose PASID/PRI only when PF has enabled it. It will be the case since the PF driver needs to exist, and IOMMU would have set the PASID/PRI/ATS on PF.
If the emulation is purely spoofing the capability. Once vIOMMU driver enables PASID, the context entries for the VF are completely
independent
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
from the PF context entries.
vIOMMU would enable PASID, and we just spoof the PASID capability.
If vIOMMU or guest for some reason does
disable_pasid(), then the vIOMMU driver can disaable PASID on the VF context entries. So the VF although the capability is blanket enabled on PF, IOMMU gauranteesthe
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
transactions are blocked.
In the interim, it seems like the intent of the virtual capability can be honored via help from the IOMMU for the controlling aspect..
Quoted 7 lines
Quoted 1 line
Did i miss anything?
Above works for emulating the enable bit (under the assumption that PF driver won't disable pasid when vf is assigned). However, there are also "Execute permission enable" and "Privileged mode enable" bits in PASID control registers. I don't know how those bits could be cleanly emulated when the guest writes a value different from PF's...
Quoted 4 lines
sent too quick. the IOMMU also includes control bits for allowing/ blocking execute requests and supervisor requests. We can rely on IOMMU to block those requests to emulate the disabled cases of all three control bits in the pasid cap.
Quoted 6 lines
So if the emulation of the PASID capability takes into account the IOMMU configuration to back that emulation, shouldn't we do that emulation in the hypervisor, ie. QEMU, rather than the kernel vfio layer? Thanks,
Alex
We need enforce it in physical IOMMU, to ensure that even the VF may send requests which violate the guest expectation those requests are always blocked by IOMMU. Kernel vfio identifies such need when emulating the pasid cap and then forward the request to host iommu driver.
Thanks
Kevin
Quoted 37 lines
Quoted 27 lines
Quoted 27 lines
Similar problem also exists when talking about PRI emulation, e.g. to enable PRI the software usually waits until the 'stopped' bit is set (indicating all previously issued requests have completed). How to emulate this bit accurately when one guest toggles the enable bit while the PF and other VFs are actively issuing page requests through the shared page request interface? from pcie spec I didn't find a way to catch when all previously-issued requests from a specific VF have completed. Can a conservative big-enough timeout value help here? I don't know... similar puzzle also exists for emulating the 'reset' control bit which is supposed to clear the pending request state for the whole page request interface.
I feel the main problem in pcie spec is that, while they invent SR-IOV to address I/O virtualization requirement (where strict isolation is required), they blurred the boundary by leaving some shared resource cross PF and VFs which imply sort of cooperation between PF and VF drivers. On bare metal such cooperation is easy to build, by enabling/ disabling a capability en mass, by using the same set of setting, etc. However it doesn't consider the virtualization case where a VF is assigned to the guest which considers the VF as a standard PCI/PCIe endpoint thus such cooperation is missing. A vendor capability could help fix the gap here but making it adopted by major guest OSes will take time. But honestly speaking I don't know other good alternative now... :/
Quoted 1 line
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
need new capabilities exposing these as slave features that cannot be controlled? We could define our own vendor capability for this, but
of
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
course we have both the where to put it in config space issue, as well as the issue of trying to push an ad-hoc standard. vfio could expose these as device features rather than emulating capabilities, but that still leaves a big gap between vfio in the hypervisor and the driver in the guest VM. That might still help push the responsibility and policy for how to expose it to the VM as a userspace problem though.
Quoted 17 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
I think this is a good long term solution, but if the vIOMMU
implenentations
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
can carry us for the time being, we can probably defer them unless we are stuck.
Quoted 2 lines
I agree though, I don't know why the SIG would preclude
implementing
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
per VF control of these features. Thanks,
Cheers,
AshokThanks
Kevin
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
Quoted 120 lines · Kevin
Quoted 120 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, April 14, 2020 3:21 AMQuoted 22 lines · Kevin
Quoted 22 lines
From: Tian, Kevin
Sent: Monday, April 13, 2020 3:55 PMQuoted 19 lines
From: Raj, Ashok <[email protected]>
Sent: Monday, April 13, 2020 11:11 AMQuoted 15 lines · Alex Williamson
Quoted 3 lines · Ashok
Hi Alex
+ Bjorn
+ Don
Quoted 6 lines
FWIW I can't understand why PCI SIG went different ways with ATS, where its enumerated on PF and VF. But for PASID and PRI its only in PF.
I'm checking with our internal SIG reps to followup on that.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines · Alex Williamson
Quoted 6 lines
Quoted 2 lines
Is there vendor guarantee that hidden registers will locate at the same offset between PF and VF config space?
I'm not sure if the spec really precludes hidden registers, but the fact that these registers are explicitly outside of the capability chain implies they're only intended for device specific use, so I'd
say
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
there are no guarantees about anything related to these registers.
As you had suggested in the other thread, we could consider using the same offset as in PF, but even that's a better guess still not reliable.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
The other option is to maybe extend driver ops in the PF to expose where the offsets should be. Sort of adding the quirk in the implementation.
I'm not sure how prevalent are PASID and PRI in VF devices. If SIG is
resisting
Quoted 1 line
Quoted 1 line
making VF's first class citizen, we might ask them to add some
verbiage
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to suggest leave the same offsets as PF open to help emulation
software.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Even if we know where to expose these capabilities on the VF, it's not clear to me how we can actually virtualize the capability itself. If the spec defines, for example, an enable bit as r/w then software that interacts with that register expects the bit is settable. There's no protocol for "try to set the bit and re-read it to see if the hardware accepted it". Therefore a capability with a fixed enable bit representing the state of the PF, not settable by the VF, is disingenuous to the spec.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
I think we are all in violent agreement. A lot of times the pci spec gets defined several years ahead of real products and no one remembers the justification on why they restricted things the way they did.
Maybe someone early product wasn't quite exposing these features to
the
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
VF
and hence the spec is bug compatible :-)Quoted 4 lines
If what we're trying to do is expose that PASID and PRI are enabled on the PF to a VF driver, maybe duplicating the PF capabilities on the VF without the ability to control it is not the right approach. Maybe we
As long as the capability enable is only provided when the PF has
enabled
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
the feature. Then it seems the hardware seems to do the right thing.
Assume we expose PASID/PRI only when PF has enabled it. It will be the case since the PF driver needs to exist, and IOMMU would have set the PASID/PRI/ATS on PF.
If the emulation is purely spoofing the capability. Once vIOMMU driver enables PASID, the context entries for the VF are completely
independent
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
from the PF context entries.
vIOMMU would enable PASID, and we just spoof the PASID capability.
If vIOMMU or guest for some reason does
disable_pasid(), then the vIOMMU driver can disaable PASID on the VF context entries. So the VF although the capability is blanket enabled on PF, IOMMU gauranteesthe
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
transactions are blocked.
In the interim, it seems like the intent of the virtual capability can be honored via help from the IOMMU for the controlling aspect..
Quoted 7 lines
Quoted 1 line
Did i miss anything?
Above works for emulating the enable bit (under the assumption that PF driver won't disable pasid when vf is assigned). However, there are also "Execute permission enable" and "Privileged mode enable" bits in PASID control registers. I don't know how those bits could be cleanly emulated when the guest writes a value different from PF's...
Quoted 4 lines
sent too quick. the IOMMU also includes control bits for allowing/ blocking execute requests and supervisor requests. We can rely on IOMMU to block those requests to emulate the disabled cases of all three control bits in the pasid cap.
So if the emulation of the PASID capability takes into account the IOMMU configuration to back that emulation, shouldn't we do that emulation in the hypervisor, ie. QEMU, rather than the kernel vfio layer? Thanks,
Quoted 7 lines
Quoted 1 line
Alex
We need enforce it in physical IOMMU, to ensure that even the VF may send requests which violate the guest expectation those requests are always blocked by IOMMU. Kernel vfio identifies such need when emulating the pasid cap and then forward the request to host iommu driver.
Implementing this in the kernel would be necessary if we needed to protect from the guest device doing something bad to the host or other devices. Making sure the physical IOMMU is configured to meet guest expectations doesn't sound like it necessarily falls into that category. We do that on a regular basis to program the DMA mappings. Tell me more about why the hypervisor can't handle this piece of guest/host synchronization on top of all the other things it synchronizes to make a VM. Thanks,
Alex
RE: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
Quoted 156 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, April 14, 2020 11:29 AMQuoted 16 lines · Kevin
Quoted 16 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, April 14, 2020 3:21 AMQuoted 10 lines · Kevin
Quoted 10 lines
From: Tian, Kevin
Sent: Monday, April 13, 2020 3:55 PMQuoted 7 lines
From: Raj, Ashok <[email protected]>
Sent: Monday, April 13, 2020 11:11 AMQuoted 3 lines · Alex Williamson
On Tue, 7 Apr 2020 21:00:21 -0700
"Raj, Ashok" <[email protected]> wrote:Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Hi Alex
+ Bjorn
+ Don
Quoted 1 line
FWIW I can't understand why PCI SIG went different ways with
ATS,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
where its enumerated on PF and VF. But for PASID and PRI its
only
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
in PF.
I'm checking with our internal SIG reps to followup on that.
On Tue, Apr 07, 2020 at 09:58:01AM -0600, Alex Williamson
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Is there vendor guarantee that hidden registers will locate at
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
same offset between PF and VF config space?
I'm not sure if the spec really precludes hidden registers, but
the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
fact that these registers are explicitly outside of the capability chain implies they're only intended for device specific use, so
I'd
Quoted 2 lines
Quoted 2 lines
say
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
there are no guarantees about anything related to these
registers.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
As you had suggested in the other thread, we could consider using the same offset as in PF, but even that's a better guess still not reliable.
The other option is to maybe extend driver ops in the PF to
expose
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
where the offsets should be. Sort of adding the quirk in the implementation.
I'm not sure how prevalent are PASID and PRI in VF devices. If
SIG is
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
resisting
Quoted 1 line
Quoted 1 line
making VF's first class citizen, we might ask them to add some
verbiage
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to suggest leave the same offsets as PF open to help emulation
software.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Even if we know where to expose these capabilities on the VF, it's
not
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
clear to me how we can actually virtualize the capability itself. If the spec defines, for example, an enable bit as r/w then software
that
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
interacts with that register expects the bit is settable. There's no protocol for "try to set the bit and re-read it to see if the hardware accepted it". Therefore a capability with a fixed enable bit representing the state of the PF, not settable by the VF, is disingenuous to the spec.
I think we are all in violent agreement. A lot of times the pci spec
gets
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
defined several years ahead of real products and no one
remembers
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
the justification on why they restricted things the way they did.
Maybe someone early product wasn't quite exposing these features
to
Quoted 6 lines
Quoted 6 lines
the
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
VF
and hence the spec is bug compatible :-)Quoted 2 lines
If what we're trying to do is expose that PASID and PRI are enabled
on
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the PF to a VF driver, maybe duplicating the PF capabilities on the
VF
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
without the ability to control it is not the right approach. Maybe
we
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
As long as the capability enable is only provided when the PF has
enabled
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
the feature. Then it seems the hardware seems to do the right thing.
Assume we expose PASID/PRI only when PF has enabled it. It will be
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
case since the PF driver needs to exist, and IOMMU would have set
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
PASID/PRI/ATS on PF.
If the emulation is purely spoofing the capability. Once vIOMMU
driver
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
enables PASID, the context entries for the VF are completely
independent
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
from the PF context entries.
vIOMMU would enable PASID, and we just spoof the PASID
capability.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
If vIOMMU or guest for some reason does
disable_pasid(), then the vIOMMU driver can disaable PASID on the VF context entries. So theVF
Quoted 27 lines
Quoted 27 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
although the capability is blanket enabled on PF, IOMMU gaurantees
the
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
transactions are blocked.
In the interim, it seems like the intent of the virtual capability can be honored via help from the IOMMU for the controlling aspect..
Quoted 7 lines
Quoted 1 line
Did i miss anything?
Above works for emulating the enable bit (under the assumption that PF driver won't disable pasid when vf is assigned). However, there are also "Execute permission enable" and "Privileged mode enable" bits in PASID control registers. I don't know how those bits could be cleanly emulated when the guest writes a value different from PF's...
Quoted 4 lines
sent too quick. the IOMMU also includes control bits for allowing/ blocking execute requests and supervisor requests. We can rely on IOMMU to block those requests to emulate the disabled cases of all three control bits in the pasid cap.
So if the emulation of the PASID capability takes into account the IOMMU configuration to back that emulation, shouldn't we do that emulation in the hypervisor, ie. QEMU, rather than the kernel vfio layer? Thanks,
Quoted 7 lines
Quoted 1 line
Alex
We need enforce it in physical IOMMU, to ensure that even the VF may send requests which violate the guest expectation those requests are always blocked by IOMMU. Kernel vfio identifies such need when emulating the pasid cap and then forward the request to host iommu driver.
Quoted 9 lines
Implementing this in the kernel would be necessary if we needed to protect from the guest device doing something bad to the host or other devices. Making sure the physical IOMMU is configured to meet guest expectations doesn't sound like it necessarily falls into that category. We do that on a regular basis to program the DMA mappings. Tell me more about why the hypervisor can't handle this piece of guest/host synchronization on top of all the other things it synchronizes to make a VM. Thanks,
I care more about "execution permission" and "privileged mode". It might be dangerous when the guest disallows the VF from sending DMA requests which have the execute or privileged bit set while the VF can still do so w/o proper protection in the IOMMU. This is all about vSVA for vfio devices, where the guest page table is directly linked in IOMMU and vfio doesn't participate in the specific DMA mappings. if an emulated device includes a pasid cap, Qemu vIOMMU will handle it for sure.
Thanks
Kevin
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
Quoted 166 lines · Kevin
Quoted 156 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, April 14, 2020 11:29 AMQuoted 16 lines · Kevin
Quoted 16 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, April 14, 2020 3:21 AMQuoted 10 lines · Kevin
Quoted 10 lines
From: Tian, Kevin
Sent: Monday, April 13, 2020 3:55 PMQuoted 7 lines
From: Raj, Ashok <[email protected]>
Sent: Monday, April 13, 2020 11:11 AMQuoted 3 lines · Alex Williamson
On Tue, 7 Apr 2020 21:00:21 -0700
"Raj, Ashok" <[email protected]> wrote:Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Hi Alex
+ Bjorn
+ Don
Quoted 1 line
FWIW I can't understand why PCI SIG went different ways with
ATS,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
where its enumerated on PF and VF. But for PASID and PRI its
only
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
in PF.
I'm checking with our internal SIG reps to followup on that.
On Tue, Apr 07, 2020 at 09:58:01AM -0600, Alex Williamson
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Is there vendor guarantee that hidden registers will locate at
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
same offset between PF and VF config space?
I'm not sure if the spec really precludes hidden registers, but
the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
fact that these registers are explicitly outside of the capability chain implies they're only intended for device specific use, so
I'd
Quoted 2 lines
Quoted 2 lines
say
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
there are no guarantees about anything related to these
registers.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
As you had suggested in the other thread, we could consider using the same offset as in PF, but even that's a better guess still not reliable.
The other option is to maybe extend driver ops in the PF to
expose
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
where the offsets should be. Sort of adding the quirk in the implementation.
I'm not sure how prevalent are PASID and PRI in VF devices. If
SIG is
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
resisting
Quoted 1 line
Quoted 1 line
making VF's first class citizen, we might ask them to add some
verbiage
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to suggest leave the same offsets as PF open to help emulation
software.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Even if we know where to expose these capabilities on the VF, it's
not
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
clear to me how we can actually virtualize the capability itself. If the spec defines, for example, an enable bit as r/w then software
that
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
interacts with that register expects the bit is settable. There's no protocol for "try to set the bit and re-read it to see if the hardware accepted it". Therefore a capability with a fixed enable bit representing the state of the PF, not settable by the VF, is disingenuous to the spec.
I think we are all in violent agreement. A lot of times the pci spec
gets
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
defined several years ahead of real products and no one
remembers
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
the justification on why they restricted things the way they did.
Maybe someone early product wasn't quite exposing these features
to
Quoted 6 lines
Quoted 6 lines
the
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
VF
and hence the spec is bug compatible :-)Quoted 2 lines
If what we're trying to do is expose that PASID and PRI are enabled
on
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the PF to a VF driver, maybe duplicating the PF capabilities on the
VF
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
without the ability to control it is not the right approach. Maybe
we
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
As long as the capability enable is only provided when the PF has
enabled
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
the feature. Then it seems the hardware seems to do the right thing.
Assume we expose PASID/PRI only when PF has enabled it. It will be
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
case since the PF driver needs to exist, and IOMMU would have set
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
PASID/PRI/ATS on PF.
If the emulation is purely spoofing the capability. Once vIOMMU
driver
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
enables PASID, the context entries for the VF are completely
independent
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
from the PF context entries.
vIOMMU would enable PASID, and we just spoof the PASID
capability.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
If vIOMMU or guest for some reason does
disable_pasid(), then the vIOMMU driver can disaable PASID on the VF context entries. So theVF
Quoted 27 lines
Quoted 27 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
although the capability is blanket enabled on PF, IOMMU gaurantees
the
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
transactions are blocked.
In the interim, it seems like the intent of the virtual capability can be honored via help from the IOMMU for the controlling aspect..
Quoted 7 lines
Quoted 1 line
Did i miss anything?
Above works for emulating the enable bit (under the assumption that PF driver won't disable pasid when vf is assigned). However, there are also "Execute permission enable" and "Privileged mode enable" bits in PASID control registers. I don't know how those bits could be cleanly emulated when the guest writes a value different from PF's...
Quoted 4 lines
sent too quick. the IOMMU also includes control bits for allowing/ blocking execute requests and supervisor requests. We can rely on IOMMU to block those requests to emulate the disabled cases of all three control bits in the pasid cap.
So if the emulation of the PASID capability takes into account the IOMMU configuration to back that emulation, shouldn't we do that emulation in the hypervisor, ie. QEMU, rather than the kernel vfio layer? Thanks,
Quoted 7 lines
Quoted 1 line
Alex
We need enforce it in physical IOMMU, to ensure that even the VF may send requests which violate the guest expectation those requests are always blocked by IOMMU. Kernel vfio identifies such need when emulating the pasid cap and then forward the request to host iommu driver.
Quoted 9 lines
Implementing this in the kernel would be necessary if we needed to protect from the guest device doing something bad to the host or other devices. Making sure the physical IOMMU is configured to meet guest expectations doesn't sound like it necessarily falls into that category. We do that on a regular basis to program the DMA mappings. Tell me more about why the hypervisor can't handle this piece of guest/host synchronization on top of all the other things it synchronizes to make a VM. Thanks,
Quoted 2 lines
I care more about "execution permission" and "privileged mode". It might be dangerous when the guest disallows the VF from sending
"Dangerous" how? We're generally ok with the user managing their own consistency, it's when the user can affect other users/devices that we require vfio in the kernel to actively manage something. There's a very different scope to the vfio-pci kernel module implementing a fake capability and trying to make it behave indistinguishably from the real capability versus a userspace driver piecing together an emulation that's good enough for their purposes. Thanks,
Alex
Quoted 10 lines
DMA requests which have the execute or privileged bit set while the VF can still do so w/o proper protection in the IOMMU. This is all about vSVA for vfio devices, where the guest page table is directly linked in IOMMU and vfio doesn't participate in the specific DMA mappings. if an emulated device includes a pasid cap, Qemu vIOMMU will handle it for sure.
Thanks
Kevin
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
Quoted 203 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, April 14, 2020 11:24 PMQuoted 19 lines · Kevin
Quoted 19 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, April 14, 2020 11:29 AMQuoted 13 lines · Kevin
Quoted 13 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, April 14, 2020 3:21 AMQuoted 7 lines · Kevin
Quoted 7 lines
From: Tian, Kevin
Sent: Monday, April 13, 2020 3:55 PMQuoted 4 lines
From: Raj, Ashok <[email protected]>
Sent: Monday, April 13, 2020 11:11 AMOn Wed, Apr 08, 2020 at 10:19:40AM -0600, Alex Williamson
Quoted 10 lines · wrote:
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 3 lines · Ashok
Hi Alex
+ Bjorn
+ Don
Quoted 1 line
FWIW I can't understand why PCI SIG went different ways
with
Quoted 10 lines
Quoted 10 lines
ATS,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
where its enumerated on PF and VF. But for PASID and PRI its
only
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
in PF.
I'm checking with our internal SIG reps to followup on that.
On Tue, Apr 07, 2020 at 09:58:01AM -0600, Alex Williamson
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Is there vendor guarantee that hidden registers will locate
at
Quoted 4 lines
Quoted 4 lines
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
same offset between PF and VF config space?
I'm not sure if the spec really precludes hidden registers,
but
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
fact that these registers are explicitly outside of the
capability
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
chain implies they're only intended for device specific use,
so
Quoted 10 lines
Quoted 10 lines
I'd
Quoted 2 lines
Quoted 2 lines
say
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
there are no guarantees about anything related to these
registers.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
As you had suggested in the other thread, we could consider using the same offset as in PF, but even that's a better guess still not reliable.
The other option is to maybe extend driver ops in the PF to
Quoted 8 lines
Quoted 8 lines
expose
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
where the offsets should be. Sort of adding the quirk in the implementation.
I'm not sure how prevalent are PASID and PRI in VF devices. If
SIG is
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
resisting
Quoted 1 line
Quoted 1 line
making VF's first class citizen, we might ask them to add
some
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
verbiage
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to suggest leave the same offsets as PF open to help
emulation
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
software.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Even if we know where to expose these capabilities on the VF,
it's
Quoted 2 lines
Quoted 2 lines
not
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
clear to me how we can actually virtualize the capability itself.
If
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the spec defines, for example, an enable bit as r/w then
software
Quoted 2 lines
Quoted 2 lines
that
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
interacts with that register expects the bit is settable. There's
no
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
protocol for "try to set the bit and re-read it to see if the
hardware
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
accepted it". Therefore a capability with a fixed enable bit representing the state of the PF, not settable by the VF, is disingenuous to the spec.
I think we are all in violent agreement. A lot of times the pci spec
gets
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
defined several years ahead of real products and no one
remembers
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
the justification on why they restricted things the way they did.
Maybe someone early product wasn't quite exposing these
features
Quoted 7 lines
Quoted 7 lines
to
Quoted 6 lines
Quoted 6 lines
the
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
VF
and hence the spec is bug compatible :-)Quoted 2 lines
If what we're trying to do is expose that PASID and PRI are
enabled
Quoted 2 lines
Quoted 2 lines
on
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the PF to a VF driver, maybe duplicating the PF capabilities on
the
Quoted 2 lines
Quoted 2 lines
VF
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
without the ability to control it is not the right approach.
Maybe
Quoted 5 lines
Quoted 5 lines
we
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
As long as the capability enable is only provided when the PF has
enabled
Quoted 1 line
Quoted 1 line
Quoted 1 line
the feature. Then it seems the hardware seems to do the right
thing.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Assume we expose PASID/PRI only when PF has enabled it. It will
be
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
case since the PF driver needs to exist, and IOMMU would have
set
Quoted 5 lines
Quoted 5 lines
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
PASID/PRI/ATS on PF.
If the emulation is purely spoofing the capability. Once vIOMMU
driver
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
enables PASID, the context entries for the VF are completely
independent
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
from the PF context entries.
vIOMMU would enable PASID, and we just spoof the PASID
capability.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
If vIOMMU or guest for some reason does
disable_pasid(), thenthe
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
vIOMMU driver can disaable PASID on the VF context entries. So
the
Quoted 2 lines
Quoted 2 lines
VF
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
although the capability is blanket enabled on PF, IOMMU
gaurantees
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
the
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
transactions are blocked.
In the interim, it seems like the intent of the virtual capability can be honored via help from the IOMMU for the controlling
aspect..
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Did i miss anything?
Above works for emulating the enable bit (under the assumption
that
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
PF driver won't disable pasid when vf is assigned). However, there
are
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
also "Execute permission enable" and "Privileged mode enable"
bits in
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
PASID control registers. I don't know how those bits could be
cleanly
Quoted 31 lines
Quoted 21 lines
Quoted 13 lines
Quoted 13 lines
Quoted 6 lines
Quoted 1 line
emulated when the guest writes a value different from PF's...
sent too quick. the IOMMU also includes control bits for allowing/ blocking execute requests and supervisor requests. We can rely on IOMMU to block those requests to emulate the disabled cases of all three control bits in the pasid cap.
So if the emulation of the PASID capability takes into account the IOMMU configuration to back that emulation, shouldn't we do that emulation in the hypervisor, ie. QEMU, rather than the kernel vfio layer? Thanks,
Quoted 7 lines
Quoted 1 line
Alex
We need enforce it in physical IOMMU, to ensure that even the VF may send requests which violate the guest expectation those requests are always blocked by IOMMU. Kernel vfio identifies such need when emulating the pasid cap and then forward the request to host iommu driver.
Quoted 9 lines
Implementing this in the kernel would be necessary if we needed to protect from the guest device doing something bad to the host or other devices. Making sure the physical IOMMU is configured to meet guest expectations doesn't sound like it necessarily falls into that category. We do that on a regular basis to program the DMA mappings. Tell me more about why the hypervisor can't handle this piece of guest/host synchronization on top of all the other things it synchronizes to make a VM. Thanks,
Quoted 2 lines
I care more about "execution permission" and "privileged mode". It might be dangerous when the guest disallows the VF from sending
Quoted 8 lines
"Dangerous" how? We're generally ok with the user managing their own consistency, it's when the user can affect other users/devices that we require vfio in the kernel to actively manage something. There's a very different scope to the vfio-pci kernel module implementing a fake capability and trying to make it behave indistinguishably from the real capability versus a userspace driver piecing together an emulation that's good enough for their purposes. Thanks,
How could emulation fix this gap when the VF DMAs don't go through the vIOMMU? What you explained all makes sense before talking about the emulation of PASID capability, i.e. vfio only cares about isolation between assigned devices. However now vfio exposes a capability which is shared by PF/VF while pure software emulation may break the guest expectation, and now the only viable mitigation is to get the help from physical IOMMU. then why cannot vfio include such mitigation in its emulation of the PASID capability?
Thanks
Kevin
Quoted 11 lines
Quoted 11 lines
DMA requests which have the execute or privileged bit set while the VF can still do so w/o proper protection in the IOMMU. This is all about vSVA for vfio devices, where the guest page table is directly linked in IOMMU and vfio doesn't participate in the specific DMA mappings. if an emulated device includes a pasid cap, Qemu vIOMMU will handle it for sure.
Thanks
Kevin
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
Quoted 212 lines · Kevin
Quoted 203 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, April 14, 2020 11:24 PMQuoted 19 lines · Kevin
Quoted 19 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, April 14, 2020 11:29 AMQuoted 13 lines · Kevin
Quoted 13 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, April 14, 2020 3:21 AMQuoted 7 lines · Kevin
Quoted 7 lines
From: Tian, Kevin
Sent: Monday, April 13, 2020 3:55 PMQuoted 4 lines
From: Raj, Ashok <[email protected]>
Sent: Monday, April 13, 2020 11:11 AMOn Wed, Apr 08, 2020 at 10:19:40AM -0600, Alex Williamson
Quoted 10 lines · wrote:
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 3 lines · Ashok
Hi Alex
+ Bjorn
+ Don
Quoted 1 line
FWIW I can't understand why PCI SIG went different ways
with
Quoted 10 lines
Quoted 10 lines
ATS,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
where its enumerated on PF and VF. But for PASID and PRI its
only
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
in PF.
I'm checking with our internal SIG reps to followup on that.
On Tue, Apr 07, 2020 at 09:58:01AM -0600, Alex Williamson
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Is there vendor guarantee that hidden registers will locate
at
Quoted 4 lines
Quoted 4 lines
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
same offset between PF and VF config space?
I'm not sure if the spec really precludes hidden registers,
but
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
fact that these registers are explicitly outside of the
capability
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
chain implies they're only intended for device specific use,
so
Quoted 10 lines
Quoted 10 lines
I'd
Quoted 2 lines
Quoted 2 lines
say
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
there are no guarantees about anything related to these
registers.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
As you had suggested in the other thread, we could consider using the same offset as in PF, but even that's a better guess still not reliable.
The other option is to maybe extend driver ops in the PF to
Quoted 8 lines
Quoted 8 lines
expose
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
where the offsets should be. Sort of adding the quirk in the implementation.
I'm not sure how prevalent are PASID and PRI in VF devices. If
SIG is
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
resisting
Quoted 1 line
Quoted 1 line
making VF's first class citizen, we might ask them to add
some
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
verbiage
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to suggest leave the same offsets as PF open to help
emulation
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
software.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Even if we know where to expose these capabilities on the VF,
it's
Quoted 2 lines
Quoted 2 lines
not
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
clear to me how we can actually virtualize the capability itself.
If
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the spec defines, for example, an enable bit as r/w then
software
Quoted 2 lines
Quoted 2 lines
that
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
interacts with that register expects the bit is settable. There's
no
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
protocol for "try to set the bit and re-read it to see if the
hardware
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
accepted it". Therefore a capability with a fixed enable bit representing the state of the PF, not settable by the VF, is disingenuous to the spec.
I think we are all in violent agreement. A lot of times the pci spec
gets
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
defined several years ahead of real products and no one
remembers
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
the justification on why they restricted things the way they did.
Maybe someone early product wasn't quite exposing these
features
Quoted 7 lines
Quoted 7 lines
to
Quoted 6 lines
Quoted 6 lines
the
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
VF
and hence the spec is bug compatible :-)Quoted 2 lines
If what we're trying to do is expose that PASID and PRI are
enabled
Quoted 2 lines
Quoted 2 lines
on
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the PF to a VF driver, maybe duplicating the PF capabilities on
the
Quoted 2 lines
Quoted 2 lines
VF
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
without the ability to control it is not the right approach.
Maybe
Quoted 5 lines
Quoted 5 lines
we
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
As long as the capability enable is only provided when the PF has
enabled
Quoted 1 line
Quoted 1 line
Quoted 1 line
the feature. Then it seems the hardware seems to do the right
thing.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Assume we expose PASID/PRI only when PF has enabled it. It will
be
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
case since the PF driver needs to exist, and IOMMU would have
set
Quoted 5 lines
Quoted 5 lines
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
PASID/PRI/ATS on PF.
If the emulation is purely spoofing the capability. Once vIOMMU
driver
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
enables PASID, the context entries for the VF are completely
independent
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
from the PF context entries.
vIOMMU would enable PASID, and we just spoof the PASID
capability.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
If vIOMMU or guest for some reason does
disable_pasid(), thenthe
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
vIOMMU driver can disaable PASID on the VF context entries. So
the
Quoted 2 lines
Quoted 2 lines
VF
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
although the capability is blanket enabled on PF, IOMMU
gaurantees
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
the
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
transactions are blocked.
In the interim, it seems like the intent of the virtual capability can be honored via help from the IOMMU for the controlling
aspect..
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Did i miss anything?
Above works for emulating the enable bit (under the assumption
that
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
PF driver won't disable pasid when vf is assigned). However, there
are
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
also "Execute permission enable" and "Privileged mode enable"
bits in
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
PASID control registers. I don't know how those bits could be
cleanly
Quoted 31 lines
Quoted 21 lines
Quoted 13 lines
Quoted 13 lines
Quoted 6 lines
Quoted 1 line
emulated when the guest writes a value different from PF's...
sent too quick. the IOMMU also includes control bits for allowing/ blocking execute requests and supervisor requests. We can rely on IOMMU to block those requests to emulate the disabled cases of all three control bits in the pasid cap.
So if the emulation of the PASID capability takes into account the IOMMU configuration to back that emulation, shouldn't we do that emulation in the hypervisor, ie. QEMU, rather than the kernel vfio layer? Thanks,
Quoted 7 lines
Quoted 1 line
Alex
We need enforce it in physical IOMMU, to ensure that even the VF may send requests which violate the guest expectation those requests are always blocked by IOMMU. Kernel vfio identifies such need when emulating the pasid cap and then forward the request to host iommu driver.
Quoted 9 lines
Implementing this in the kernel would be necessary if we needed to protect from the guest device doing something bad to the host or other devices. Making sure the physical IOMMU is configured to meet guest expectations doesn't sound like it necessarily falls into that category. We do that on a regular basis to program the DMA mappings. Tell me more about why the hypervisor can't handle this piece of guest/host synchronization on top of all the other things it synchronizes to make a VM. Thanks,
Quoted 2 lines
I care more about "execution permission" and "privileged mode". It might be dangerous when the guest disallows the VF from sending
Quoted 8 lines
"Dangerous" how? We're generally ok with the user managing their own consistency, it's when the user can affect other users/devices that we require vfio in the kernel to actively manage something. There's a very different scope to the vfio-pci kernel module implementing a fake capability and trying to make it behave indistinguishably from the real capability versus a userspace driver piecing together an emulation that's good enough for their purposes. Thanks,
Quoted 8 lines
How could emulation fix this gap when the VF DMAs don't go through the vIOMMU? What you explained all makes sense before talking about the emulation of PASID capability, i.e. vfio only cares about isolation between assigned devices. However now vfio exposes a capability which is shared by PF/VF while pure software emulation may break the guest expectation, and now the only viable mitigation is to get the help from physical IOMMU. then why cannot vfio include such mitigation in its emulation of the PASID capability?
DMA never actually goes "through" the vIOMMU. I'm not suggesting that vfio doesn't participate some how, but I don't know that emulating a capability that doesn't exist and involves policy should be done in the kernel, versus providing userspace with an interface to control what they need to implement that emulation. Thanks,
Alex
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
Quoted 257 lines
From: Alex Williamson <[email protected]>
Sent: Wednesday, April 15, 2020 8:36 AMQuoted 31 lines · Kevin
Quoted 31 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, April 14, 2020 11:24 PMQuoted 19 lines · Kevin
Quoted 19 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, April 14, 2020 11:29 AMQuoted 13 lines · Kevin
Quoted 13 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, April 14, 2020 3:21 AMQuoted 7 lines · Kevin
Quoted 7 lines
From: Tian, Kevin
Sent: Monday, April 13, 2020 3:55 PMQuoted 4 lines
From: Raj, Ashok <[email protected]>
Sent: Monday, April 13, 2020 11:11 AMOn Wed, Apr 08, 2020 at 10:19:40AM -0600, Alex Williamson
Quoted 5 lines · wrote:
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines · Ashok
Hi Alex
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
+ Bjorn
+ Don
Quoted 1 line
FWIW I can't understand why PCI SIG went different ways
with
Quoted 2 lines
Quoted 2 lines
ATS,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
where its enumerated on PF and VF. But for PASID and
PRI its
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
only
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
in PF.
I'm checking with our internal SIG reps to followup on
that.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
On Tue, Apr 07, 2020 at 09:58:01AM -0600, Alex
Williamson
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Is there vendor guarantee that hidden registers will
locate
Quoted 10 lines
Quoted 10 lines
at
Quoted 4 lines
Quoted 4 lines
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
same offset between PF and VF config space?
I'm not sure if the spec really precludes hidden registers,
but
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
fact that these registers are explicitly outside of the
capability
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
chain implies they're only intended for device specific
use,
Quoted 7 lines
Quoted 7 lines
so
Quoted 6 lines
Quoted 6 lines
I'd
Quoted 2 lines
Quoted 2 lines
say
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
there are no guarantees about anything related to these
registers.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
As you had suggested in the other thread, we could
consider
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
using the same offset as in PF, but even that's a better
guess
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
still not reliable.
The other option is to maybe extend driver ops in the PF
to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
expose
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
where the offsets should be. Sort of adding the quirk in
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
implementation.
I'm not sure how prevalent are PASID and PRI in VF
devices. If
Quoted 10 lines
Quoted 10 lines
Quoted 3 lines
Quoted 3 lines
SIG is
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
resisting
Quoted 1 line
Quoted 1 line
making VF's first class citizen, we might ask them to add
some
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
verbiage
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to suggest leave the same offsets as PF open to help
emulation
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
software.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Even if we know where to expose these capabilities on the
VF,
Quoted 3 lines
Quoted 3 lines
it's
Quoted 2 lines
Quoted 2 lines
not
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
clear to me how we can actually virtualize the capability
itself.
Quoted 5 lines
Quoted 5 lines
If
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the spec defines, for example, an enable bit as r/w then
software
Quoted 2 lines
Quoted 2 lines
that
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
interacts with that register expects the bit is settable.
There's
Quoted 8 lines
Quoted 8 lines
no
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
protocol for "try to set the bit and re-read it to see if the
hardware
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
accepted it". Therefore a capability with a fixed enable bit representing the state of the PF, not settable by the VF, is disingenuous to the spec.
I think we are all in violent agreement. A lot of times the pci
spec
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
gets
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
defined several years ahead of real products and no one
remembers
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the justification on why they restricted things the way they
did.
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Maybe someone early product wasn't quite exposing these
features
Quoted 2 lines
Quoted 2 lines
to
Quoted 1 line
Quoted 1 line
the
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
VF
and hence the spec is bug compatible :-)Quoted 2 lines
If what we're trying to do is expose that PASID and PRI are
enabled
Quoted 2 lines
Quoted 2 lines
on
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the PF to a VF driver, maybe duplicating the PF capabilities
on
Quoted 7 lines
Quoted 7 lines
the
Quoted 2 lines
Quoted 2 lines
VF
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
without the ability to control it is not the right approach.
Maybe
Quoted 3 lines
Quoted 3 lines
we
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
As long as the capability enable is only provided when the PF
has
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
enabled
Quoted 1 line
Quoted 1 line
Quoted 1 line
the feature. Then it seems the hardware seems to do the
right
Quoted 3 lines
Quoted 3 lines
thing.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Assume we expose PASID/PRI only when PF has enabled it. It
will
Quoted 3 lines
Quoted 3 lines
be
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
case since the PF driver needs to exist, and IOMMU would
have
Quoted 5 lines
Quoted 5 lines
set
Quoted 4 lines
Quoted 4 lines
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
PASID/PRI/ATS on PF.
If the emulation is purely spoofing the capability. Once
vIOMMU
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
driver
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
enables PASID, the context entries for the VF are completely
independent
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
from the PF context entries.
vIOMMU would enable PASID, and we just spoof the PASID
capability.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
If vIOMMU or guest for some reason does
disable_pasid(),then
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
vIOMMU driver can disaable PASID on the VF context entries.
So
Quoted 8 lines
Quoted 8 lines
the
Quoted 2 lines
Quoted 2 lines
VF
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
although the capability is blanket enabled on PF, IOMMU
gaurantees
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
transactions are blocked.
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
In the interim, it seems like the intent of the virtual capability can be honored via help from the IOMMU for the controlling
aspect..
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Did i miss anything?
Above works for emulating the enable bit (under the
assumption
Quoted 2 lines
Quoted 2 lines
that
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
PF driver won't disable pasid when vf is assigned). However,
there
Quoted 2 lines
Quoted 2 lines
are
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
also "Execute permission enable" and "Privileged mode
enable"
Quoted 6 lines
Quoted 6 lines
bits in
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
PASID control registers. I don't know how those bits could be
cleanly
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
emulated when the guest writes a value different from PF's...
sent too quick. the IOMMU also includes control bits for
allowing/
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
blocking execute requests and supervisor requests. We can rely
on
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
IOMMU to block those requests to emulate the disabled cases of all three control bits in the pasid cap.
So if the emulation of the PASID capability takes into account the IOMMU configuration to back that emulation, shouldn't we do
that
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 4 lines
emulation in the hypervisor, ie. QEMU, rather than the kernel vfio layer? Thanks,
Alex
We need enforce it in physical IOMMU, to ensure that even the VF may send requests which violate the guest expectation those requests are always blocked by IOMMU. Kernel vfio identifies such need when emulating the pasid cap and then forward the request to host iommu driver.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Implementing this in the kernel would be necessary if we needed to protect from the guest device doing something bad to the host or other devices. Making sure the physical IOMMU is configured to meet guest expectations doesn't sound like it necessarily falls into that category. We do that on a regular basis to program the DMA
mappings.
Quoted 17 lines
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Tell me more about why the hypervisor can't handle this piece of guest/host synchronization on top of all the other things it synchronizes to make a VM. Thanks,
I care more about "execution permission" and "privileged mode". It might be dangerous when the guest disallows the VF from sending
Quoted 8 lines
"Dangerous" how? We're generally ok with the user managing their own consistency, it's when the user can affect other users/devices that we require vfio in the kernel to actively manage something. There's a very different scope to the vfio-pci kernel module implementing a fake capability and trying to make it behave indistinguishably from the real capability versus a userspace driver piecing together an emulation that's good enough for their purposes. Thanks,
Quoted 8 lines
How could emulation fix this gap when the VF DMAs don't go through the vIOMMU? What you explained all makes sense before talking about the emulation of PASID capability, i.e. vfio only cares about isolation between assigned devices. However now vfio exposes a capability which is shared by PF/VF while pure software emulation may break the guest expectation, and now the only viable mitigation is to get the help from physical IOMMU. then why cannot vfio include such mitigation in its emulation of the PASID capability?
Quoted 6 lines
DMA never actually goes "through" the vIOMMU. I'm not suggesting that vfio doesn't participate some how, but I don't know that emulating a capability that doesn't exist and involves policy should be done in the kernel, versus providing userspace with an interface to control what they need to implement that emulation. Thanks,
OK, I see your point. We'll think about the latter option (e.g. through DEVICE_FEATURE) and put a proposal for further discussion.
btw as I commented earlier, let's focus on PF first to enable vSVA given that there are already many flying bits. After that we can then extend the vSVA support to VF based on discussions in this thread.
Thanks
Kevin
RE: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
In reply to Alex WilliamsonHi Alex,
Quoted 64 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 3, 2020 7:00 AM
To: Liu, Yi L <[email protected]>
Subject: Re: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFsQuoted 18 lines · Yi L
From: Liu Yi L <[email protected]>
Per PCIe r5.0, sec 9.3.7.14, if a PF implements the PASID Capability, the PF PASID configuration is shared by its VFs. VFs must not implement their own PASID Capability.
Per PCIe r5.0, sec 9.3.7.11, VFs must not implement the PRI Capability. If the PF implements PRI, it is shared by the VFs.
On bare metal, it has been fixed by below efforts. to PASID/PRI are
https://lkml.org/lkml/2019/9/5/996
https://lkml.org/lkml/2019/9/5/995This patch adds emulated PASID/PRI capabilities for VFs when assigned to VMs via vfio-pci driver. This is required for enabling vSVA on pass-through VFs as VFs have no PASID/PRI capability structure in its configure space.
Quoted 8 lines
Cc Kevin Tian <[email protected]>CC Jacob Pan <[email protected]>Cc Alex Williamson <[email protected]>Cc Eric Auger <[email protected]>Cc Jean-Philippe Brucker <[email protected]>Signed-off-by Liu Yi L <[email protected]>--- drivers/vfio/pci/vfio_pci_config.c | 325++++++++++++++++++++++++++++++++++++-
Quoted 14 lines
1 file changed, 323 insertions(+), 2 deletions(-)drivers/vfio/pci/vfio_pci_config.c +4
-0@@ -1509,11 +1509,304 @@ static int vfio_cap_init(struct vfio_pci_device *vdev)return 0 ; } +static int vfio_fill_custom_vconfig_bytes (struct vfio_pci_device *vdev, +int offset,uint8_t *data,int size) +{ +int ret =0 , data_offset =0 ;drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1525
Quoted 10 lines
drivers/vfio/pci/vfio_pci_config.c +10
-0+ +while (size) { +int filled; + +if (size >=4 && !(offset %4 )) { +__le32 *dwordp = (__le32 *)&vdev->vconfig[offset]; +u32 dword; + +memcpy (&dword, data + data_offset,4 ); + *dwordp =cpu_to_le32 (dword);Why wouldn't we do:
*dwordp =
cpu_to_le32(*(u32 *)(data + data_offset));
Quoted 7 lines
or better yet, increment data on each iteration for:
*dwordp =
cpu_to_le32(*(u32 *)data);
vfio_fill_vconfig_bytes()does almost this same thing, getting the data from config space rather than a buffer, so please figure out how to avoid duplicating the logic.
Got another alternative. I may use the vfio_fill_vconfig_bytes() to fill the cap data from PF's config space into VF's vconfig. And after that, I can further modify the data in vconfig. e.g. zero the control reg of pasid cap. would it make more sense?
Quoted 12 lines
Quoted 12 lines
drivers/vfio/pci/vfio_pci_config.c +12
-0+ filled =4 ; + }else if (size >=2 && !(offset %2 )) { +__le16 *wordp = (__le16 *)&vdev->vconfig[offset]; +u16 word; + +memcpy (&word, data + data_offset,2 ); + *wordp =cpu_to_le16 (word); + filled =2 ; + }else { +u8 *byte = &vdev->vconfig[offset]; + +memcpy (byte, data + data_offset,1 );
[...]
Quoted 6 lines
drivers/vfio/pci/vfio_pci_config.c
Quoted 3 lines
drivers/vfio/pci/vfio_pci_config.c +2
-0+ +memset (map + epos, vpasid_cap.id, len);See below.
Quoted 16 lines
Quoted 4 lines
drivers/vfio/pci/vfio_pci_config.c +4
-0+ ret =vfio_fill_custom_vconfig_bytes (vdev, epos, + (u8 *)&vpasid_cap, len); +if (!ret) { +/* drivers/vfio/pci/vfio_pci_config.c static int vfio_cap_init(struct vfio_pci_device *vdev) · line 1684
Quoted 10 lines
drivers/vfio/pci/vfio_pci_config.c +10
-0+* Successfully filled in PASID cap, update +* the next offset in previous cap header, +* and also update caller about the offset +* of next cap if any. +*/ +u32 val = epos; + **prevp &=cpu_to_le32 (~(0xffcU <<20 )); + **prevp |=cpu_to_le32 (val <<20 ); + *prevp = (__le32 *)&vdev->vconfig[epos]; + *next = epos + len;Could we make this any more complicated?
I'm not sure if adding comments addressed this comment. After adding new cap in vconfig, it needs to update the cap.next field of prior cap. And in case of further add other cap, this function needs to update the prevp pointer to the address of the newly added cap.
Regards,
Yi Liu
Quoted 18 lines
From: Liu, Yi L <[email protected]>
Sent: Sunday, March 22, 2020 8:33 PMFrom: Liu Yi L <[email protected]>
Shared Virtual Addressing (SVA), a.k.a, Shared Virtual Memory (SVM) on Intel platforms allows address space sharing between device DMA and applications. SVA can reduce programming complexity and enhance security.
To enable SVA, device needs to have PASID capability, which is a key capability for SVA. This patchset exposes the device's PASID capability to guest instead of hiding it from guest.
The second patch emulates PASID capability for VFs (Virtual Function) since VFs don't implement such capability per PCIe spec. This patch emulates such capability and expose to VM if the capability is enabled in PF (Physical Function).
Quoted 6 lines
However, there is an open for PASID emulation. If PF driver disables PASID capability at runtime, then it may be an issue. e.g. PF should not disable PASID capability if there is guest using this capability on any VF related to this PF. To solve it, may need to introduce a generic communication framework between vfio-pci driver and PF drivers. Please feel free to give your suggestions on it.
I'm not sure how this is addressed on bate metal today, i.e. between normal kernel PF and VF drivers. I look at pasid enable/disable code in intel-iommu.c. There is no check on PF/VF dependency so far. The cap is toggled when attaching/detaching the PF to its domain. Let's see how IOMMU guys respond, and if there is a way for VF driver to block PF driver from disabling the pasid cap when it's being actively used by VF driver, then we may leverage the same trick in VFIO when emulation is provided to guest.
Thanks
Kevin
Quoted 18 lines
Regards,
Yi LiuChangelog:
- RFC v1 -> Patch v1:
AddCONFIG_PCI_ATS#ifdef control to avoid compiling error.Liu Yi L (2):
vfio/pci: Expose PCIe PASID capability to guest
vfio/pci: Emulate PASID/PRI capability for VFsdrivers/vfio/pci/vfio_pci_config.c | 327 ++++++++++++++++++++++++++++++++++++- 1 file changed, 324 insertions(+), 3 deletions(-)--
2.7.4
Quoted 24 lines · Kevin
Quoted 18 lines
From: Liu, Yi L<[email protected]>
Sent: Sunday, March 22, 2020 8:33 PMFrom: Liu Yi L<[email protected]>
Shared Virtual Addressing (SVA), a.k.a, Shared Virtual Memory (SVM) on Intel platforms allows address space sharing between device DMA and applications. SVA can reduce programming complexity and enhance security.
To enable SVA, device needs to have PASID capability, which is a key capability for SVA. This patchset exposes the device's PASID capability to guest instead of hiding it from guest.
The second patch emulates PASID capability for VFs (Virtual Function) since VFs don't implement such capability per PCIe spec. This patch emulates such capability and expose to VM if the capability is enabled in PF (Physical Function).
Quoted 6 lines
However, there is an open for PASID emulation. If PF driver disables PASID capability at runtime, then it may be an issue. e.g. PF should not disable PASID capability if there is guest using this capability on any VF related to this PF. To solve it, may need to introduce a generic communication framework between vfio-pci driver and PF drivers. Please feel free to give your suggestions on it.
Quoted 7 lines
I'm not sure how this is addressed on bate metal today, i.e. between normal kernel PF and VF drivers. I look at pasid enable/disable code in
intel-iommu.c. There is no check on PF/VF dependency so far. The cap is toggled when attaching/detaching the PF to its domain. Let's see how IOMMU guys respond, and if there is a way for VF driver to block PF driver from disabling the pasid cap when it's being actively used by VF driver, then we may leverage the same trick in VFIO when emulation is provided to guest.
IOMMU subsystem doesn't expose any APIs for pasid enabling/disabling. The PCI subsystem does. It handles VF/PF like below.
/**
* pci_enable_pasid - Enable the PASID capability
* @pdev: PCI device structure
* @features: Features to enable
*
* Returns 0 on success, negative value on error. This function checks
* whether the features are actually supported by the device and returns
* an error if not.
*/
int pci_enable_pasid(struct pci_dev *pdev, int features)
{
u16 control, supported;
int pasid = pdev->pasid_cap;
/*
* VFs must not implement the PASID Capability, but if a PF
* supports PASID, its VFs share the PF PASID configuration.
*/
if (pdev->is_virtfn) {
if (pci_physfn(pdev)->pasid_enabled)
return 0;
return -EINVAL;
}/**
* pci_disable_pasid - Disable the PASID capability
* @pdev: PCI device structure
*/
void pci_disable_pasid(struct pci_dev *pdev)
{
u16 control = 0;
int pasid = pdev->pasid_cap;
/* VFs share the PF PASID configuration */
if (pdev->is_virtfn)
return;It doesn't block disabling PASID on PF even VFs are possibly using it.
Best regards,
baolu
Quoted 74 lines · Baolu
Quoted 24 lines · Kevin
Quoted 18 lines
From: Liu, Yi L<[email protected]>
Sent: Sunday, March 22, 2020 8:33 PMFrom: Liu Yi L<[email protected]>
Shared Virtual Addressing (SVA), a.k.a, Shared Virtual Memory (SVM) on Intel platforms allows address space sharing between device DMA and applications. SVA can reduce programming complexity and enhance security.
To enable SVA, device needs to have PASID capability, which is a key capability for SVA. This patchset exposes the device's PASID capability to guest instead of hiding it from guest.
The second patch emulates PASID capability for VFs (Virtual Function) since VFs don't implement such capability per PCIe spec. This patch emulates such capability and expose to VM if the capability is enabled in PF (Physical Function).
Quoted 6 lines
However, there is an open for PASID emulation. If PF driver disables PASID capability at runtime, then it may be an issue. e.g. PF should not disable PASID capability if there is guest using this capability on any VF related to this PF. To solve it, may need to introduce a generic communication framework between vfio-pci driver and PF drivers. Please feel free to give your suggestions on it.
Quoted 7 lines
I'm not sure how this is addressed on bate metal today, i.e. between normal kernel PF and VF drivers. I look at pasid enable/disable code in
intel-iommu.c. There is no check on PF/VF dependency so far. The cap is toggled when attaching/detaching the PF to its domain. Let's see how IOMMU guys respond, and if there is a way for VF driver to block PF driver from disabling the pasid cap when it's being actively used by VF driver, then we may leverage the same trick in VFIO when emulation is provided to guest.IOMMU subsystem doesn't expose any APIs for pasid enabling/disabling. The PCI subsystem does. It handles VF/PF like below.
/**
*pci_enable_pasid- Enable the PASID capability
* @pdev: PCI device structure
* @features: Features to enable
*
* Returns 0 on success, negative value on error. This function checks
* whether the features are actually supported by the device and returns
* an error if not.
*/
intpci_enable_pasid(struct pci_dev *pdev, int features){ u16 control, supported; int pasid = pdev->pasid_cap; /* * VFs must not implement the PASID Capability, but if a PF * supports PASID, its VFs share the PF PASID configuration. */ if (pdev->is_virtfn) { if (pci_physfn(pdev)->pasid_enabled) return 0; return -EINVAL; }/**
*pci_disable_pasid- Disable the PASID capability
* @pdev: PCI device structure
*/
voidpci_disable_pasid(struct pci_dev *pdev){ u16 control = 0; int pasid = pdev->pasid_cap; /* VFs share the PF PASID configuration */ if (pdev->is_virtfn) return;
Quoted 2 lines
It doesn't block disabling PASID on PF even VFs are possibly using it.
hi
I'm not sure, but is it possible for pci_enable_pasid() and pci_disable_pasid() to do the same thing as pdev->driver->sriov_configure, e.g. pci_sriov_configure_simple() below.
It checks whether there are VFs are assigned in pci_vfs_assigned(dev). and we can set the VF in assigned status if vfio_pci_open() is performed on the VF.
int pci_sriov_configure_simple(struct pci_dev *dev, int nr_virtfn)
{
int rc;
might_sleep();
if (!dev->is_physfn)
return -ENODEV;
if (pci_vfs_assigned(dev)) {
pci_warn(dev, "Cannot modify SR-IOV while VFs are assigned\n");
return -EPERM;
}
if (nr_virtfn == 0) {
sriov_disable(dev);
return 0;
}
rc = sriov_enable(dev, nr_virtfn);
if (rc < 0)
return rc;
return nr_virtfn;
}Thanks
Yan
Hi Zhao
Quoted 77 lines · Yan Zhao
Quoted 74 lines · Baolu
Quoted 24 lines · Kevin
Quoted 18 lines
From: Liu, Yi L<[email protected]>
Sent: Sunday, March 22, 2020 8:33 PMFrom: Liu Yi L<[email protected]>
Shared Virtual Addressing (SVA), a.k.a, Shared Virtual Memory (SVM) on Intel platforms allows address space sharing between device DMA and applications. SVA can reduce programming complexity and enhance security.
To enable SVA, device needs to have PASID capability, which is a key capability for SVA. This patchset exposes the device's PASID capability to guest instead of hiding it from guest.
The second patch emulates PASID capability for VFs (Virtual Function) since VFs don't implement such capability per PCIe spec. This patch emulates such capability and expose to VM if the capability is enabled in PF (Physical Function).
Quoted 6 lines
However, there is an open for PASID emulation. If PF driver disables PASID capability at runtime, then it may be an issue. e.g. PF should not disable PASID capability if there is guest using this capability on any VF related to this PF. To solve it, may need to introduce a generic communication framework between vfio-pci driver and PF drivers. Please feel free to give your suggestions on it.
Quoted 7 lines
I'm not sure how this is addressed on bate metal today, i.e. between normal kernel PF and VF drivers. I look at pasid enable/disable code in
intel-iommu.c. There is no check on PF/VF dependency so far. The cap is toggled when attaching/detaching the PF to its domain. Let's see how IOMMU guys respond, and if there is a way for VF driver to block PF driver from disabling the pasid cap when it's being actively used by VF driver, then we may leverage the same trick in VFIO when emulation is provided to guest.IOMMU subsystem doesn't expose any APIs for pasid enabling/disabling. The PCI subsystem does. It handles VF/PF like below.
/**
*pci_enable_pasid- Enable the PASID capability
* @pdev: PCI device structure
* @features: Features to enable
*
* Returns 0 on success, negative value on error. This function checks
* whether the features are actually supported by the device and returns
* an error if not.
*/
intpci_enable_pasid(struct pci_dev *pdev, int features){ u16 control, supported; int pasid = pdev->pasid_cap; /* * VFs must not implement the PASID Capability, but if a PF * supports PASID, its VFs share the PF PASID configuration. */ if (pdev->is_virtfn) { if (pci_physfn(pdev)->pasid_enabled) return 0; return -EINVAL; }/**
*pci_disable_pasid- Disable the PASID capability
* @pdev: PCI device structure
*/
voidpci_disable_pasid(struct pci_dev *pdev){ u16 control = 0; int pasid = pdev->pasid_cap; /* VFs share the PF PASID configuration */ if (pdev->is_virtfn) return;Quoted 2 lines
It doesn't block disabling PASID on PF even VFs are possibly using it.
Quoted 8 lines
hi
I'm not sure, but is it possible forpci_enable_pasid()andpci_disable_pasid()to do the same thing aspdev->driver->sriov_configure, e.g.pci_sriov_configure_simple()below.It checks whether there are VFs are assigned in
pci_vfs_assigned(dev). and we can set the VF in assigned status ifvfio_pci_open()is performed on the VF.
But you can still unbind the PF driver that magically causes the VF's to be removed from the guest image too correct?
Only the IOMMU mucks with pasid_enable/disable. And it doesn't look like we have a path to disable without tearing down the PF binding.
We originally had some refcounts and such and would do the real disable only when the refcount drops to 0, but we found it wasn't actually necessary to protect these resources like that.
Quoted 30 lines
int
pci_sriov_configure_simple(struct pci_dev *dev, int nr_virtfn){ int rc; might_sleep(); if (!dev->is_physfn) return -ENODEV; if (pci_vfs_assigned(dev)) { pci_warn(dev, "Cannot modify SR-IOV while VFs are assigned\n"); return -EPERM; } if (nr_virtfn == 0) { sriov_disable(dev); return 0; } rc = sriov_enable(dev, nr_virtfn); if (rc < 0) return rc; return nr_virtfn; }Thanks
Yan
Quoted 93 lines · Ashok
Hi Zhao
Quoted 77 lines · Yan Zhao
Quoted 74 lines · Baolu
Quoted 24 lines · Kevin
Quoted 18 lines
From: Liu, Yi L<[email protected]>
Sent: Sunday, March 22, 2020 8:33 PMFrom: Liu Yi L<[email protected]>
Shared Virtual Addressing (SVA), a.k.a, Shared Virtual Memory (SVM) on Intel platforms allows address space sharing between device DMA and applications. SVA can reduce programming complexity and enhance security.
To enable SVA, device needs to have PASID capability, which is a key capability for SVA. This patchset exposes the device's PASID capability to guest instead of hiding it from guest.
The second patch emulates PASID capability for VFs (Virtual Function) since VFs don't implement such capability per PCIe spec. This patch emulates such capability and expose to VM if the capability is enabled in PF (Physical Function).
Quoted 6 lines
However, there is an open for PASID emulation. If PF driver disables PASID capability at runtime, then it may be an issue. e.g. PF should not disable PASID capability if there is guest using this capability on any VF related to this PF. To solve it, may need to introduce a generic communication framework between vfio-pci driver and PF drivers. Please feel free to give your suggestions on it.
Quoted 7 lines
I'm not sure how this is addressed on bate metal today, i.e. between normal kernel PF and VF drivers. I look at pasid enable/disable code in
intel-iommu.c. There is no check on PF/VF dependency so far. The cap is toggled when attaching/detaching the PF to its domain. Let's see how IOMMU guys respond, and if there is a way for VF driver to block PF driver from disabling the pasid cap when it's being actively used by VF driver, then we may leverage the same trick in VFIO when emulation is provided to guest.IOMMU subsystem doesn't expose any APIs for pasid enabling/disabling. The PCI subsystem does. It handles VF/PF like below.
/**
*pci_enable_pasid- Enable the PASID capability
* @pdev: PCI device structure
* @features: Features to enable
*
* Returns 0 on success, negative value on error. This function checks
* whether the features are actually supported by the device and returns
* an error if not.
*/
intpci_enable_pasid(struct pci_dev *pdev, int features){ u16 control, supported; int pasid = pdev->pasid_cap; /* * VFs must not implement the PASID Capability, but if a PF * supports PASID, its VFs share the PF PASID configuration. */ if (pdev->is_virtfn) { if (pci_physfn(pdev)->pasid_enabled) return 0; return -EINVAL; }/**
*pci_disable_pasid- Disable the PASID capability
* @pdev: PCI device structure
*/
voidpci_disable_pasid(struct pci_dev *pdev){ u16 control = 0; int pasid = pdev->pasid_cap; /* VFs share the PF PASID configuration */ if (pdev->is_virtfn) return;Quoted 2 lines
It doesn't block disabling PASID on PF even VFs are possibly using it.
Quoted 8 lines
hi
I'm not sure, but is it possible forpci_enable_pasid()andpci_disable_pasid()to do the same thing aspdev->driver->sriov_configure, e.g.pci_sriov_configure_simple()below.It checks whether there are VFs are assigned in
pci_vfs_assigned(dev). and we can set the VF in assigned status ifvfio_pci_open()is performed on the VF.But you can still unbind the PF driver that magically causes the VF's to be removed from the guest image too correct?
Quoted 7 lines
Only the IOMMU mucks with pasid_enable/disable. And it doesn't look like we have a path to disable without tearing down the PF binding.
We originally had some refcounts and such and would do the real disable only when the refcount drops to 0, but we found it wasn't actually necessary to protect these resources like that.
right. now unbinding PF driver would cause VFs unplugged from guest. if we modify vfio_pci and set VFs to be assigned, then VFs could remain appearing in guest but it cannot function well as PF driver has been unbound.
thanks for explanation :)
Quoted 30 lines
Quoted 30 lines
int
pci_sriov_configure_simple(struct pci_dev *dev, int nr_virtfn){ int rc; might_sleep(); if (!dev->is_physfn) return -ENODEV; if (pci_vfs_assigned(dev)) { pci_warn(dev, "Cannot modify SR-IOV while VFs are assigned\n"); return -EPERM; } if (nr_virtfn == 0) { sriov_disable(dev); return 0; } rc = sriov_enable(dev, nr_virtfn); if (rc < 0) return rc; return nr_virtfn; }Thanks
Yan