Rebase ATS onto lastest Qemu mailing list state
cover Rebase ATS onto lastest Qemu mailing list state
From: Clément Mathieu--Drif <[email protected]>
This series belongs to a list of series that add SVM support for VT-d.
As a starting point, we use the series called
'intel_iommu: Enable stage-1 translation for emulated device' by Zhenzhong Duan and Yi Liu.
Ref: https://github.com/yiliu1765/qemu/tree/zhenzhong/iommufd_stage1_emu_v5
Based-on: [email protected]
Here we focus on implementing ATS support in the IOMMU and adding a PCI-level API to be used by virtual devices.
This work is based on the VT-d specification version 4.1 (March 2023).
Here is a link to our GitHub repository where you can find the following elements:
- Qemu with all the patches for SVM
- ATS
- PRI
- Device IOTLB invalidations
- Requests with already pre-translated addresses
- A demo device
- A simple driver for the demo device
- A userspace program (for testing and demonstration purposes)
https://github.com/BullSequana/Qemu-in-guest-SVM-demo
===============
Context and design notes
''''''''''''''''''''''''
The main purpose of this work is to enable vVT-d users to make translation requests to the vIOMMU as described in the PCIe Gen 5.0 specification (section 10). Moreover, we aim to implement a PCI/Memory-level framework that could be used by other vIOMMUs to implement the same features.
What is ATS?
''''''''''''
ATS (Address Translation Service) is a PCIe-level protocol that enables PCIe devices to query an IOMMU for virtual to physical address translations in a specific address space (such as a userland process address space). When a device receives translation responses from an IOMMU, it may decide to store them in an internal cache, often known as "ATC" (Address Translation Cache) or "Device IOTLB". To keep page tables and caches consistent, the IOMMU is allowed to send asynchronous invalidation requests to its client devices.
To avoid introducing an unnecessarily complicated API, this series simply exposes 3 functions. The first 2 are a pair of setup functions that are called to install and remove the ATS invalidation callback during the initialization phase of a process. The third one will be used to request translations. The callback setup API introduced in this series calls the IOMMUNotifier API under the hood.
API design
''''''''''
- int pci_register_iommu_tlb_event_notifier(PCIDevice *dev,
uint32_t pasid,
IOMMUNotifier *n);- int pci_unregister_iommu_tlb_event_notifier(PCIDevice *dev, uint32_t pasid,
IOMMUNotifier *n);
- ssize_t pci_ats_request_translation_pasid(PCIDevice *dev, uint32_t pasid, bool priv_req, bool exec_req, hwaddr addr, size_t length, bool no_write, IOMMUTLBEntry *result, size_t result_length, uint32_t *err_count);
Although device developers may want to implement custom ATC for testing or performance measurement purposes, we provide a generic implementation as a utility module.
Overview
''''''''
Here are the interactions between an ATS-capable PCIe device and the vVT-d:
┌───────────┐ ┌────────────┐ ┌─────────────────────────┐
│Device │ │PCI / Memory│ │vVT-d │
│ │ pci_ats_request_│abstraction │ iommu_ats_ │ │
│ │ translation_ │ │ request_ │ │
│┌─────────┐│ pasid │ AS lookup │ translation │ │
││Logic ││────────────────>│╶╶╶╶╶╶╶╶╶╶╶>│───────────────>│────────────────┐ │
│└─────────┘│<────────────────│<╶╶╶╶╶╶╶╶╶╶╶│<───────────────│<─────┐ ∨ │
│┌─────────┐│ │ │ │┌───────────────────────┐│
││inv func ││<───────┐ │ │ ││Translation logic ││
│└─────────┘│ │ │ │ │└───────────────────────┘│
│ │ │ │ │ │ ┌─────────│<───────────┐ │
│ ∨ │ │ │ │ │ │ │ │
│┌─────────┐│ │ │ │ │ │┌───────────────────────┐│
││ATC ││ │ │ │ │ ││ Invalidation queue ││
│└─────────┘│ │ │ │ │ │└───────────∧───────────┘│
└───────────┘ │ └────────────┘ │ └────────────┼────────────┘
│ │ │
└────────────────────────────┘ │
│
┌────────────────────────┐
│Kernel driver │
│ │
└────────────────────────┘Clément Mathieu--Drif (19):
memory: Add permissions in IOMMUAccessFlags
intel_iommu: Declare supported PASID size
memory: Allow to store the PASID in IOMMUTLBEntry
intel_iommu: Fill the PASID field when creating an IOMMUTLBEntry
pcie: Add helper to declare PASID capability for a pcie device
pcie: Helper functions to check if PASID is enabled
pcie: Helper function to check if ATS is enabled
pci: Cache the bus mastering status in the device
pci: Add IOMMU operations to get memory regions with PASID
intel_iommu: Implement the get_memory_region_pasid iommu operation
memory: Store user data pointer in the IOMMU notifiers
pci: Add a pci-level initialization function for iommu notifiers
atc: Generic ATC that can be used by PCIe devices that support SVM
atc: Add unit tests
memory: Add an API for ATS support
pci: Add a pci-level API for ATS
intel_iommu: Set address mask when a translation fails and adjust W
permission
intel_iommu: Return page walk level even when the translation fails
intel_iommu: Add support for ATS
hw/i386/intel_iommu.c | 128 ++++++-- hw/i386/intel_iommu_internal.h | 2 + hw/pci/pci.c | 111 ++++++- hw/pci/pcie.c | 42 +++ include/exec/memory.h | 51 +++- include/hw/i386/intel_iommu.h | 2 +- include/hw/pci/pci.h | 83 ++++++ include/hw/pci/pci_device.h | 1 + include/hw/pci/pcie.h | 9 +- include/hw/pci/pcie_regs.h | 5 + system/memory.c | 20 ++ tests/unit/meson.build | 1 + tests/unit/test-atc.c | 527 +++++++++++++++++++++++++++++++++ util/atc.c | 211 +++++++++++++ util/atc.h | 117 ++++++++ util/meson.build | 1 + 16 files changed, 1280 insertions(+), 31 deletions(-) create mode 100644 tests/unit/test-atc.c create mode 100644 util/atc.c create mode 100644 util/atc.h
Signature · 1 line
2.47.0
PATCH 1/19 memory: Add permissions in IOMMUAccessFlags
From: Clément Mathieu--Drif <[email protected]>
This will be necessary for devices implementing ATS.
We also define a new macro IOMMU_ACCESS_FLAG_FULL in addition to IOMMU_ACCESS_FLAG to support more access flags. IOMMU_ACCESS_FLAG is kept for convenience and backward compatibility.
Here are the flags added (defined by the PCIe 5 specification) :
- Execute Requested
- Privileged Mode Requested
- Global
- Untranslated Only
IOMMU_ACCESS_FLAG sets the additional flags to 0
--- include/exec/memory.h | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-)
include/exec/memory.h +21 -2
@@ -110,15 +110,34 @@ struct MemoryRegionSection {typedef struct IOMMUTLBEntry IOMMUTLBEntry ; -/* See address_space_translate: bit 0 is read, bit 1 is write. */ +/* +* See address_space_translate: +* - bit 0 : read +* - bit 1 : write +* - bit 2 : exec +* - bit 3 : priv +* - bit 4 : global +* - bit 5 : untranslated only +*/ typedef enum {IOMMU_NONE =0 ,IOMMU_RO =1 ,IOMMU_WO =2 ,IOMMU_RW =3 , +IOMMU_EXEC =4 , +IOMMU_PRIV =8 , +IOMMU_GLOBAL =16 , +IOMMU_UNTRANSLATED_ONLY =32 , }IOMMUAccessFlags ; -#define IOMMU_ACCESS_FLAG (r, w) (((r) ? IOMMU_RO : 0) | ((w) ? IOMMU_WO : 0)) +#define IOMMU_ACCESS_FLAG (r, w) (((r) ? IOMMU_RO : 0) | \ + ((w) ? IOMMU_WO : 0)) +#define IOMMU_ACCESS_FLAG_FULL (r, w, x, p, g, uo) \ + (IOMMU_ACCESS_FLAG(r, w) | \ + ((x) ? IOMMU_EXEC : 0) | \ + ((p) ? IOMMU_PRIV : 0) | \ + ((g) ? IOMMU_GLOBAL : 0) | \ + ((uo) ? IOMMU_UNTRANSLATED_ONLY : 0))struct IOMMUTLBEntry {AddressSpace *target_as;
Signature · 1 line
2.47.0
PATCH 2/19 intel_iommu: Declare supported PASID size
In reply to CLEMENT MATHIEU--DRIFFrom: Clément Mathieu--Drif <[email protected]>
PSS field of the ecap register stores the supported PASID size minus 1. Thus, this commit adds support for 20bits PASIDs.
--- hw/i386/intel_iommu.c | 2 +- hw/i386/intel_iommu_internal.h | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)
hw/i386/intel_iommu.c +1 -1
@@ -4575,7 +4575,7 @@ static void vtd_cap_init(IntelIOMMUState *s) }if (s->pasid) { - s->ecap |=VTD_ECAP_PASID ; + s->ecap |=VTD_ECAP_PASID |VTD_ECAP_PSS ; } }
hw/i386/intel_iommu_internal.h +1 -0
@@ -192,6 +192,7 @@#define VTD_ECAP_SC (1ULL << 7)#define VTD_ECAP_MHMV (15ULL << 20)#define VTD_ECAP_SRS (1ULL << 31) +#define VTD_ECAP_PSS (19ULL << 35)#define VTD_ECAP_PASID (1ULL << 40)#define VTD_ECAP_SMTS (1ULL << 43)#define VTD_ECAP_SLTS (1ULL << 46)
Signature · 1 line
2.47.0
PATCH 3/19 memory: Allow to store the PASID in IOMMUTLBEntry
In reply to CLEMENT MATHIEU--DRIFFrom: Clément Mathieu--Drif <[email protected]>
This will be useful for devices that support ATS and need to store entries in an ATC (device IOTLB).
--- include/exec/memory.h | 1 + 1 file changed, 1 insertion(+)
include/exec/memory.h +1 -0
@@ -145,6 +145,7 @@ struct IOMMUTLBEntry {hwaddr translated_addr;hwaddr addr_mask;/* 0xfff = 4k translation */ IOMMUAccessFlags perm; +uint32_t pasid; };/*
Signature · 1 line
2.47.0
PATCH 4/19 intel_iommu: Fill the PASID field when creating an IOMMUTLBEntry
In reply to CLEMENT MATHIEU--DRIFFrom: Clément Mathieu--Drif <[email protected]>
PASID value must be used by devices as a key (or part of a key) when populating their ATC with the IOTLB entries returned by the IOMMU.
--- hw/i386/intel_iommu.c | 9 +++++++++ 1 file changed, 9 insertions(+)
hw/i386/intel_iommu.c +9 -0
@@ -2119,6 +2119,9 @@ static bool vtd_do_iommu_translate(VTDAddressSpace *vtd_as, PCIBus *bus,vtd_iommu_lock (s); +/* fill the pasid before getting rid2pasid */ + entry->pasid = pasid; + cc_entry = &vtd_as->context_cache_entry;/* Try to fetch pte from IOTLB, we don't need RID2PASID logic */
@@ -2260,6 +2263,7 @@ out: entry->translated_addr =vtd_get_pte_addr (pte, s->aw_bits) & page_mask; entry->addr_mask = ~page_mask; entry->perm = access_flags; +/* pasid already set */ return true; error:
@@ -2268,6 +2272,7 @@ error: entry->translated_addr =0 ; entry->addr_mask =0 ; entry->perm =IOMMU_NONE ; + entry->pasid =PCI_NO_PASID ;return false; }
@@ -2511,6 +2516,7 @@ static void vtd_iotlb_page_invalidate_notify(IntelIOMMUState *s, .translated_addr = 0, .addr_mask = size - 1, .perm = IOMMU_NONE, + .pasid = pasid, }, };memory_region_notify_iommu (&vtd_as->iommu,0 , event);
@@ -3098,6 +3104,7 @@ static void do_invalidate_device_tlb(VTDAddressSpace *vtd_dev_as, event.entry.iova = addr; event.entry.perm =IOMMU_NONE ; event.entry.translated_addr =0 ; + event.entry.pasid = vtd_dev_as->pasid;memory_region_notify_iommu (&vtd_dev_as->iommu,0 , event); }
@@ -3680,6 +3687,7 @@ static IOMMUTLBEntry vtd_iommu_translate(IOMMUMemoryRegion *iommu, hwaddr addr,IOMMUTLBEntry iotlb = {/* We'll fill in the rest later. */ .target_as = &address_space_memory, + .pasid = vtd_as->pasid, };bool success;
@@ -3692,6 +3700,7 @@ static IOMMUTLBEntry vtd_iommu_translate(IOMMUMemoryRegion *iommu, hwaddr addr, iotlb.translated_addr = addr &VTD_PAGE_MASK_4K ; iotlb.addr_mask = ~VTD_PAGE_MASK_4K ; iotlb.perm =IOMMU_RW ; + iotlb.pasid =PCI_NO_PASID ; success = true; }
Signature · 1 line
2.47.0
RE: [PATCH v1 04/19] intel_iommu: Fill the PASID field when creating an IOMMUTLBEntry
Hi Clement,
Quoted 20 lines
-----Original Message-----
From: CLEMENT MATHIEU--DRIF <[email protected]>
Subject: [PATCH v1 04/19]intel_iommu: Fill the PASID field when creating an IOMMUTLBEntryFrom: Clément Mathieu--Drif <[email protected]>
PASID value must be used by devices as a key (or part of a key) when populating their ATC with the IOTLB entries returned by the IOMMU.
Signed-off-by Clément Mathieu--Drif <[email protected]>---
hw/i386/intel_iommu.c| 9 +++++++++1 file changed, 9 insertions(+)hw/i386/intel_iommu.c +0
-0
hw/i386/intel_iommu.c static bool vtd_do_iommu_translate(VTDAddressSpace · line 2124
Quoted 6 lines
hw/i386/intel_iommu.c +2
-0@@ -2119,6 +2119,9 @@ static bool vtd_do_iommu_translate(VTDAddressSpace *vtd_as, PCIBus *bus,vtd_iommu_lock (s); +/* fill the pasid before getting rid2pasid */ + entry->pasid = pasid;
Seems unnecessary because vtd_iommu_translate() already assigned it.
Quoted 13 lines
hw/i386/intel_iommu.c +1
-0+ cc_entry = &vtd_as->context_cache_entry;/* Try to fetch pte from IOTLB, we don't need RID2PASID logic */hw/i386/intel_iommu.c +1
-0@@ -2260,6 +2263,7 @@ out: entry->translated_addr =vtd_get_pte_addr (pte, s->aw_bits) & page_mask; entry->addr_mask = ~page_mask; entry->perm = access_flags; +/* pasid already set */ return true;error:
hw/i386/intel_iommu.c error: · line 2275
Quoted 4 lines
hw/i386/intel_iommu.c +1
-0@@ -2268,6 +2272,7 @@ error: entry->translated_addr =0 ; entry->addr_mask =0 ; entry->perm =IOMMU_NONE ; + entry->pasid =PCI_NO_PASID ;
Shouldn't we keep original pasid value?
hw/i386/intel_iommu.c static void · line 2520
Quoted 9 lines
hw/i386/intel_iommu.c +1
-0return false; }@@ -2511,6 +2516,7 @@ static void vtd_iotlb_page_invalidate_notify(IntelIOMMUState *s, .translated_addr = 0, .addr_mask = size - 1, .perm = IOMMU_NONE, + .pasid = pasid,
Shouldn't we assign vtd_dev_as->pasid?
Quoted 22 lines
hw/i386/intel_iommu.c +0
-0}, };memory_region_notify_iommu(&vtd_as->iommu, 0, event);hw/i386/intel_iommu.c +2
-0@@ -3098,6 +3104,7 @@ static void do_invalidate_device_tlb(VTDAddressSpace *vtd_dev_as, event.entry.iova = addr; event.entry.perm =IOMMU_NONE ; event.entry.translated_addr =0 ; + event.entry.pasid = vtd_dev_as->pasid;memory_region_notify_iommu (&vtd_dev_as->iommu,0 , event); }@@ -3680,6 +3687,7 @@ static IOMMUTLBEntry vtd_iommu_translate(IOMMUMemoryRegion *iommu, hwaddr addr, IOMMUTLBEntry iotlb = {/* We'll fill in the rest later. */ .target_as = &address_space_memory, + .pasid = vtd_as->pasid, }; bool success;
hw/i386/intel_iommu.c static IOMMUTLBEntry · line 3704
Quoted 5 lines
hw/i386/intel_iommu.c +1
-0@@ -3692,6 +3700,7 @@ static IOMMUTLBEntry vtd_iommu_translate(IOMMUMemoryRegion *iommu, hwaddr addr, iotlb.translated_addr = addr & VTD_PAGE_MASK_4K; iotlb.addr_mask = ~VTD_PAGE_MASK_4K ; iotlb.perm =IOMMU_RW ; + iotlb.pasid =PCI_NO_PASID ;
Shouldn't we keep earlier assigned pasid value as above?
Thanks
Zhenzhong
Re: [PATCH v1 04/19] intel_iommu: Fill the PASID field when creating an IOMMUTLBEntry
Hi zhenzhong,
Quoted 27 lines · Zhenzhong
Caution: External email. Do not open attachments or click links, unless this email comes from a known sender and you know the content is safe.
Hi Clement,
Quoted 22 lines
-----Original Message-----
From: CLEMENT MATHIEU--DRIF <[email protected]>
Subject: [PATCH v1 04/19]intel_iommu: Fill the PASID field when creating an IOMMUTLBEntryFrom: Clément Mathieu--Drif <[email protected]>
PASID value must be used by devices as a key (or part of a key) when populating their ATC with the IOTLB entries returned by the IOMMU.
Signed-off-by Clément Mathieu--Drif <[email protected]>---
hw/i386/intel_iommu.c| 9 +++++++++1 file changed, 9 insertions(+)hw/i386/intel_iommu.c +0
-0@@ -2119,6 +2119,9 @@ static bool vtd_do_iommu_translate(VTDAddressSpace *vtd_as, PCIBus *bus,
Quoted 5 lines
hw/i386/intel_iommu.c static bool vtd_do_iommu_translate(VTDAddressSpace *vtd_as, PCIBus *bus, · line 2123
Quoted 4 lines
hw/i386/intel_iommu.c +2
-0vtd_iommu_lock (s); +/* fill the pasid before getting rid2pasid */ + entry->pasid = pasid;Seems unnecessary because
vtd_iommu_translate()already assigned it.
You are right
Quoted 12 lines
Quoted 12 lines
hw/i386/intel_iommu.c +1
-0+ cc_entry = &vtd_as->context_cache_entry;/* Try to fetch pte from IOTLB, we don't need RID2PASID logic */hw/i386/intel_iommu.c +1
-0@@ -2260,6 +2263,7 @@ out: entry->translated_addr =vtd_get_pte_addr (pte, s->aw_bits) & page_mask; entry->addr_mask = ~page_mask; entry->perm = access_flags; +/* pasid already set */ return true;
Quoted 7 lines
hw/i386/intel_iommu.c error: · line 2275
Quoted 6 lines
error:
hw/i386/intel_iommu.c +1
-0@@ -2268,6 +2272,7 @@ error: entry->translated_addr =0 ; entry->addr_mask =0 ; entry->perm =IOMMU_NONE ; + entry->pasid =PCI_NO_PASID ;Shouldn't we keep original pasid value?
will do
Quoted 11 lines
hw/i386/intel_iommu.c static void · line 2520
Quoted 10 lines
hw/i386/intel_iommu.c +1
-0return false; }@@ -2511,6 +2516,7 @@ static void vtd_iotlb_page_invalidate_notify(IntelIOMMUState *s, .translated_addr = 0, .addr_mask = size - 1, .perm = IOMMU_NONE, + .pasid = pasid,Shouldn't we assign
vtd_dev_as->pasid?
will do
Quoted 22 lines
Quoted 22 lines
hw/i386/intel_iommu.c +0
-0}, };memory_region_notify_iommu(&vtd_as->iommu, 0, event);hw/i386/intel_iommu.c +2
-0@@ -3098,6 +3104,7 @@ static void do_invalidate_device_tlb(VTDAddressSpace *vtd_dev_as, event.entry.iova = addr; event.entry.perm =IOMMU_NONE ; event.entry.translated_addr =0 ; + event.entry.pasid = vtd_dev_as->pasid;memory_region_notify_iommu (&vtd_dev_as->iommu,0 , event); }@@ -3680,6 +3687,7 @@ static IOMMUTLBEntry vtd_iommu_translate(IOMMUMemoryRegion *iommu, hwaddr addr, IOMMUTLBEntry iotlb = {/* We'll fill in the rest later. */ .target_as = &address_space_memory, + .pasid = vtd_as->pasid, }; bool success;
Quoted 7 lines
hw/i386/intel_iommu.c static IOMMUTLBEntry · line 3704
Quoted 6 lines
hw/i386/intel_iommu.c +1
-0@@ -3692,6 +3700,7 @@ static IOMMUTLBEntry vtd_iommu_translate(IOMMUMemoryRegion *iommu, hwaddr addr, iotlb.translated_addr = addr & VTD_PAGE_MASK_4K; iotlb.addr_mask = ~VTD_PAGE_MASK_4K ; iotlb.perm =IOMMU_RW ; + iotlb.pasid =PCI_NO_PASID ;Shouldn't we keep earlier assigned pasid value as above?
will do
Quoted 3 lines
Thanks
Zhenzhong
I'll wait for more reviews before sending a v2
Thanks
cmd
PATCH 5/19 pcie: Add helper to declare PASID capability for a pcie device
In reply to CLEMENT MATHIEU--DRIFFrom: Clément Mathieu--Drif <[email protected]>
--- hw/pci/pcie.c | 24 ++++++++++++++++++++++++ include/hw/pci/pcie.h | 6 +++++- include/hw/pci/pcie_regs.h | 5 +++++ 3 files changed, 34 insertions(+), 1 deletion(-)
hw/pci/pcie.c +24 -0
@@ -1210,3 +1210,27 @@ void pcie_acs_reset(PCIDevice *dev)pci_set_word (dev->config + dev->exp.acs_cap +PCI_ACS_CTRL ,0 ); } } + +/* PASID */ +void pcie_pasid_init (PCIDevice *dev,uint16_t offset,uint8_t pasid_width, +bool exec_perm,bool priv_mod) +{ +assert (pasid_width <=PCI_EXT_CAP_PASID_MAX_WIDTH ); +static const uint16_t control_reg_rw_mask =0x07 ; +uint16_t capability_reg = pasid_width; + +pcie_add_capability (dev,PCI_EXT_CAP_ID_PASID ,PCI_PASID_VER , offset, +PCI_EXT_CAP_PASID_SIZEOF ); + + capability_reg <<=PCI_PASID_CAP_WIDTH_SHIFT ; + capability_reg |= exec_perm ?PCI_PASID_CAP_EXEC :0 ; + capability_reg |= priv_mod ?PCI_PASID_CAP_PRIV :0 ; +pci_set_word (dev->config + offset +PCI_PASID_CAP , capability_reg); + +/* Everything is disabled by default */ +pci_set_word (dev->config + offset +PCI_PASID_CTRL ,0 ); + +pci_set_word (dev->wmask + offset +PCI_PASID_CTRL , control_reg_rw_mask); + + dev->exp.pasid_cap = offset; +}
include/hw/pci/pcie.h +5 -1
@@ -72,8 +72,9 @@ struct PCIExpressDevice {uint16_t aer_cap;PCIEAERLog aer_log; -/* Offset of ATS capability in config space */ +/* Offset of ATS and PASID capabilities in config space */ uint16_t ats_cap; +uint16_t pasid_cap;/* ACS */ uint16_t acs_cap;
@@ -152,4 +153,7 @@ void pcie_cap_slot_unplug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp);void pcie_cap_slot_unplug_request_cb (HotplugHandler *hotplug_dev,DeviceState *dev,Error **errp); + +void pcie_pasid_init (PCIDevice *dev,uint16_t offset,uint8_t pasid_width, +bool exec_perm,bool priv_mod);#endif /* QEMU_PCIE_H */
include/hw/pci/pcie_regs.h +5 -0
@@ -86,6 +86,11 @@ typedef enum PCIExpLinkWidth {#define PCI_ARI_VER 1#define PCI_ARI_SIZEOF 8 +/* PASID */ +#define PCI_PASID_VER 1 +#define PCI_EXT_CAP_PASID_MAX_WIDTH 20 +#define PCI_PASID_CAP_WIDTH_SHIFT 8 +/* AER */ #define PCI_ERR_VER 2#define PCI_ERR_SIZEOF 0x48
Signature · 1 line
2.47.0
PATCH 6/19 pcie: Helper functions to check if PASID is enabled
In reply to CLEMENT MATHIEU--DRIFFrom: Clément Mathieu--Drif <[email protected]>
pasid_enabled checks whether the capability is present or not. If so, we read the configuration space to get the status of the feature (enabled or not).
--- hw/pci/pcie.c | 9 +++++++++ include/hw/pci/pcie.h | 2 ++ 2 files changed, 11 insertions(+)
hw/pci/pcie.c +9 -0
@@ -1234,3 +1234,12 @@ void pcie_pasid_init(PCIDevice *dev, uint16_t offset, uint8_t pasid_width, dev->exp.pasid_cap = offset; } + +bool pcie_pasid_enabled (const PCIDevice *dev) +{ +if (!pci_is_express (dev) || !dev->exp.pasid_cap) { +return false; + } +return (pci_get_word (dev->config + dev->exp.pasid_cap +PCI_PASID_CTRL ) & +PCI_PASID_CTRL_ENABLE ) !=0 ; +}
include/hw/pci/pcie.h +2 -0
@@ -156,4 +156,6 @@ void pcie_cap_slot_unplug_request_cb(HotplugHandler *hotplug_dev,void pcie_pasid_init (PCIDevice *dev,uint16_t offset,uint8_t pasid_width,bool exec_perm,bool priv_mod); + +bool pcie_pasid_enabled (const PCIDevice *dev);#endif /* QEMU_PCIE_H */
Signature · 1 line
2.47.0
PATCH 7/19 pcie: Helper function to check if ATS is enabled
In reply to CLEMENT MATHIEU--DRIFFrom: Clément Mathieu--Drif <[email protected]>
ats_enabled checks whether the capability is present or not. If so, we read the configuration space to get the status of the feature (enabled or not).
--- hw/pci/pcie.c | 9 +++++++++ include/hw/pci/pcie.h | 1 + 2 files changed, 10 insertions(+)
hw/pci/pcie.c +9 -0
@@ -1243,3 +1243,12 @@ bool pcie_pasid_enabled(const PCIDevice *dev)return (pci_get_word (dev->config + dev->exp.pasid_cap +PCI_PASID_CTRL ) &PCI_PASID_CTRL_ENABLE ) !=0 ; } + +bool pcie_ats_enabled (const PCIDevice *dev) +{ +if (!pci_is_express (dev) || !dev->exp.ats_cap) { +return false; + } +return (pci_get_word (dev->config + dev->exp.ats_cap +PCI_ATS_CTRL ) & +PCI_ATS_CTRL_ENABLE ) !=0 ; +}
include/hw/pci/pcie.h +1 -0
@@ -158,4 +158,5 @@ void pcie_pasid_init(PCIDevice *dev, uint16_t offset, uint8_t pasid_width, bool exec_perm, bool priv_mod);bool pcie_pasid_enabled (const PCIDevice *dev); +bool pcie_ats_enabled (const PCIDevice *dev);#endif /* QEMU_PCIE_H */
Signature · 1 line
2.47.0
PATCH 8/19 pci: Cache the bus mastering status in the device
In reply to CLEMENT MATHIEU--DRIFFrom: Clément Mathieu--Drif <[email protected]>
The cached is_master value is necessary to know if a device is allowed to issue ATS requests or not.
This behavior is implemented in an upcoming patch.
--- hw/pci/pci.c | 25 +++++++++++++++---------- include/hw/pci/pci_device.h | 1 + 2 files changed, 16 insertions(+), 10 deletions(-)
hw/pci/pci.c +15 -10
@@ -134,6 +134,12 @@ static GSequence *pci_acpi_index_list(void)return used_acpi_index_list; } +static void pci_set_master (PCIDevice *d,bool enable) +{ +memory_region_set_enabled (&d->bus_master_enable_region, enable); + d->is_master = enable;/* cache the status */ +} +static void pci_init_bus_master (PCIDevice *pci_dev) {AddressSpace *dma_as =pci_device_iommu_address_space (pci_dev);
@@ -141,7 +147,7 @@ static void pci_init_bus_master(PCIDevice *pci_dev)memory_region_init_alias (&pci_dev->bus_master_enable_region,OBJECT (pci_dev),"bus master" , dma_as->root,0 ,memory_region_size (dma_as->root)); -memory_region_set_enabled (&pci_dev->bus_master_enable_region, false); +pci_set_master (pci_dev, false);memory_region_add_subregion (&pci_dev->bus_master_container_region,0 , &pci_dev->bus_master_enable_region); }
@@ -675,9 +681,8 @@ static int get_pci_config_device(QEMUFile *f, void *pv, size_t size,pci_bridge_update_mappings (PCI_BRIDGE (s)); } -memory_region_set_enabled (&s->bus_master_enable_region, -pci_get_word (s->config +PCI_COMMAND ) - &PCI_COMMAND_MASTER ); +pci_set_master (s,pci_get_word (s->config +PCI_COMMAND ) + &PCI_COMMAND_MASTER );g_free (config);return 0 ;
@@ -1632,9 +1637,10 @@ void pci_default_write_config(PCIDevice *d, uint32_t addr, uint32_t val_in, intif (ranges_overlap (addr, l,PCI_COMMAND ,2 )) {pci_update_irq_disabled (d, was_irq_disabled); -memory_region_set_enabled (&d->bus_master_enable_region, - (pci_get_word (d->config +PCI_COMMAND ) - &PCI_COMMAND_MASTER ) && d->has_power); +pci_set_master (d, + (pci_get_word (d->config +PCI_COMMAND ) & +PCI_COMMAND_MASTER ) && + d->has_power); }msi_write_config (d, addr, val_in, l);
@@ -2917,9 +2923,8 @@ void pci_set_power(PCIDevice *d, bool state) d->has_power = state;pci_update_mappings (d); -memory_region_set_enabled (&d->bus_master_enable_region, - (pci_get_word (d->config +PCI_COMMAND ) - &PCI_COMMAND_MASTER ) && d->has_power); +pci_set_master (d, (pci_get_word (d->config +PCI_COMMAND ) + &PCI_COMMAND_MASTER ) && d->has_power);if (!d->has_power) {pci_device_reset (d); }
include/hw/pci/pci_device.h +1 -0
@@ -88,6 +88,7 @@ struct PCIDevice {char name[64 ];PCIIORegion io_regions[PCI_NUM_REGIONS ];AddressSpace bus_master_as; +bool is_master;MemoryRegion bus_master_container_region;MemoryRegion bus_master_enable_region;
Signature · 1 line
2.47.0
PATCH 9/19 pci: Add IOMMU operations to get memory regions with PASID
In reply to CLEMENT MATHIEU--DRIFFrom: Clément Mathieu--Drif <[email protected]>
The region returned by this operation will be used as the input region for ATS.
--- include/hw/pci/pci.h | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
include/hw/pci/pci.h +16 -0
@@ -388,6 +388,22 @@ typedef struct PCIIOMMUOps {* @devfn: device and function number */ AddressSpace * (*get_address_space)(PCIBus *bus,void *opaque,int devfn); +/** +* @get_memory_region_pasid: get the iommu memory region for a given +* device and pasid +* +* @bus: the #PCIBus being accessed. +* +* @opaque: the data passed to pci_setup_iommu(). +* +* @devfn: device and function number +* +* @pasid: the pasid associated with the requested memory region +*/ +IOMMUMemoryRegion * (*get_memory_region_pasid)(PCIBus *bus, +void *opaque, +int devfn, +uint32_t pasid);/** * @set_iommu_device: attach a HostIOMMUDevice to a vIOMMU *
Signature · 1 line
2.47.0
PATCH 10/19 intel_iommu: Implement the get_memory_region_pasid iommu operation
In reply to CLEMENT MATHIEU--DRIFFrom: Clément Mathieu--Drif <[email protected]>
--- hw/i386/intel_iommu.c | 17 ++++++++++++++++- include/hw/i386/intel_iommu.h | 2 +- 2 files changed, 17 insertions(+), 2 deletions(-)
hw/i386/intel_iommu.c +16 -1
@@ -4209,7 +4209,7 @@ static const MemoryRegionOps vtd_mem_ir_fault_ops = { };VTDAddressSpace *vtd_find_add_as (IntelIOMMUState *s,PCIBus *bus, -int devfn,unsigned int pasid) +int devfn,uint32_t pasid) {/* * We can't simply use sid here since the bus number might not be
@@ -4726,8 +4726,23 @@ static AddressSpace *vtd_host_dma_iommu(PCIBus *bus, void *opaque, int devfn)return &vtd_as->as; } +static IOMMUMemoryRegion *vtd_get_memory_region_pasid (PCIBus *bus, +void *opaque, +int devfn, +uint32_t pasid) +{ +IntelIOMMUState *s = opaque; +VTDAddressSpace *vtd_as; + +assert (0 <= devfn && devfn <PCI_DEVFN_MAX ); + + vtd_as =vtd_find_add_as (s, bus, devfn, pasid); +return &vtd_as->iommu; +} +static PCIIOMMUOps vtd_iommu_ops = { .get_address_space = vtd_host_dma_iommu, + .get_memory_region_pasid = vtd_get_memory_region_pasid, .set_iommu_device = vtd_dev_set_iommu_device, .unset_iommu_device = vtd_dev_unset_iommu_device, };
include/hw/i386/intel_iommu.h +1 -1
@@ -324,6 +324,6 @@ struct IntelIOMMUState {* create a new one if none exists */ VTDAddressSpace *vtd_find_add_as (IntelIOMMUState *s,PCIBus *bus, -int devfn,unsigned int pasid); +int devfn,uint32_t pasid);#endif
Signature · 1 line
2.47.0
PATCH 11/19 memory: Store user data pointer in the IOMMU notifiers
In reply to CLEMENT MATHIEU--DRIFFrom: Clément Mathieu--Drif <[email protected]>
This will help developers of svm devices to track a state
--- include/exec/memory.h | 1 + 1 file changed, 1 insertion(+)
include/exec/memory.h +1 -0
@@ -204,6 +204,7 @@ struct IOMMUNotifier {hwaddr start;hwaddr end;int iommu_idx; +void *opaque;QLIST_ENTRY (IOMMUNotifier) node; };typedef struct IOMMUNotifier IOMMUNotifier ;
Signature · 1 line
2.47.0
PATCH 12/19 pci: Add a pci-level initialization function for iommu notifiers
In reply to CLEMENT MATHIEU--DRIFFrom: Clément Mathieu--Drif <[email protected]>
We add a convenient way to initialize an device-iotlb notifier. This is meant to be used by ATS-capable devices.
pci_device_iommu_memory_region_pasid is introduces in this commit and will be used in several other SVM-related functions exposed in the PCI API.
--- hw/pci/pci.c | 40 ++++++++++++++++++++++++++++++++++++++++ include/hw/pci/pci.h | 15 +++++++++++++++ 2 files changed, 55 insertions(+)
hw/pci/pci.c +40 -0
@@ -2773,6 +2773,46 @@ AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)return &address_space_memory; } +static IOMMUMemoryRegion *pci_device_iommu_memory_region_pasid (PCIDevice *dev, +uint32_t pasid) +{ +PCIBus *bus; +PCIBus *iommu_bus; +int devfn; + +/* +* This function is for internal use in the module, +* we can call it with PCI_NO_PASID +*/ +if (!dev->is_master || + ((pasid !=PCI_NO_PASID ) && !pcie_pasid_enabled (dev))) { +return NULL ; + } + +pci_device_get_iommu_bus_devfn (dev, &bus, &iommu_bus, &devfn); +if (iommu_bus && iommu_bus->iommu_ops->get_memory_region_pasid) { +return iommu_bus->iommu_ops->get_memory_region_pasid (bus, + iommu_bus->iommu_opaque, devfn, pasid); + } +return NULL ; +} + +bool pci_iommu_init_iotlb_notifier (PCIDevice *dev,uint32_t pasid, +IOMMUNotifier *n,IOMMUNotify fn, +void *opaque) +{ +IOMMUMemoryRegion *iommu_mr =pci_device_iommu_memory_region_pasid (dev, + pasid); +if (!iommu_mr) { +return false; + } +iommu_notifier_init (n, fn,IOMMU_NOTIFIER_DEVIOTLB_EVENTS ,0 ,HWADDR_MAX , +memory_region_iommu_attrs_to_index (iommu_mr, +MEMTXATTRS_UNSPECIFIED )); + n->opaque = opaque; +return true; +} +bool pci_device_set_iommu_device (PCIDevice *dev,HostIOMMUDevice *hiod,Error **errp) {
include/hw/pci/pci.h +15 -0
@@ -443,6 +443,21 @@ bool pci_device_set_iommu_device(PCIDevice *dev, HostIOMMUDevice *hiod, Error **errp);void pci_device_unset_iommu_device (PCIDevice *dev); +/** +* pci_iommu_init_iotlb_notifier: initialize an IOMMU notifier +* +* This function is used by devices before registering an IOTLB notifier +* +* @dev: the device +* @pasid: the pasid of the address space to watch +* @n: the notifier to initialize +* @fn: the callback to be installed +* @opaque: user pointer that can be used to store a state +*/ +bool pci_iommu_init_iotlb_notifier (PCIDevice *dev,uint32_t pasid, +IOMMUNotifier *n,IOMMUNotify fn, +void *opaque); +/** * pci_setup_iommu: Initialize specific IOMMU handlers for a PCIBus *
Signature · 1 line
2.47.0
PATCH 13/19 atc: Generic ATC that can be used by PCIe devices that support SVM
In reply to CLEMENT MATHIEU--DRIFFrom: Clément Mathieu--Drif <[email protected]>
As the SVM-capable devices will need to cache translations, we provide an first implementation.
This cache uses a two-level design based on hash tables.
The first level is indexed by a PASID and the second by a virtual addresse.
--- util/atc.c | 211 +++++++++++++++++++++++++++++++++++++++++++++++ util/atc.h | 117 ++++++++++++++++++++++++++ util/meson.build | 1 + 3 files changed, 329 insertions(+) create mode 100644 util/atc.c create mode 100644 util/atc.h
util/atc.c new file +211 -0
@@ -0,0 +1,211 @@ +/* +* QEMU emulation of an ATC +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License, or +* (at your option) any later version. + +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. + +* You should have received a copy of the GNU General Public License along +* with this program; if not, see <http://www.gnu.org/licenses/>. +*/ + +#include "util/atc.h" + + +#define PAGE_TABLE_ENTRY_SIZE 8 + +/* a pasid is hashed using the identity function */ +static guint atc_pasid_key_hash (gconstpointer v) +{ +return (guint )(uintptr_t )v;/* pasid */ +} + +/* pasid equality */ +static gboolean atc_pasid_key_equal (gconstpointer v1,gconstpointer v2) +{ +return v1 == v2; +} + +/* Hash function for IOTLB entries */ +static guint atc_addr_key_hash (gconstpointer v) +{ +hwaddr addr = (hwaddr )v; +return (guint)((addr >>32 ) ^ (addr &0xffffffffU )); +} + +/* Equality test for IOTLB entries */ +static gboolean atc_addr_key_equal (gconstpointer v1,gconstpointer v2) +{ +return (hwaddr )v1 == (hwaddr )v2; +} + +static void atc_address_space_free (void *as) +{ +g_hash_table_unref (as); +} + +/* return log2(val), or UINT8_MAX if val is not a power of 2 */ +static uint8_t ilog2 (uint64_t val) +{ +uint8_t result =0 ; +while (val !=1 ) { +if (val &1 ) { +return UINT8_MAX ; + } + + val >>=1 ; + result +=1 ; + } +return result; +} + +ATC *atc_new (uint64_t page_size,uint8_t address_width) +{ +ATC *atc; +uint8_t log_page_size =ilog2 (page_size); +/* number of bits each used to store all the intermediate indexes */ +uint64_t addr_lookup_indexes_size; + +if (log_page_size ==UINT8_MAX ) { +return NULL ; + } +/* +* We only support page table entries of 8 (PAGE_TABLE_ENTRY_SIZE) bytes +* log2(page_size / 8) = log2(page_size) - 3 +* is the level offset +*/ +if (log_page_size <=3 ) { +return NULL ; + } + + atc =g_new0 (ATC ,1 ); + atc->address_spaces =g_hash_table_new_full (atc_pasid_key_hash, + atc_pasid_key_equal, +NULL , atc_address_space_free); + atc->level_offset = log_page_size -3 ; +/* at this point, we know that page_size is a power of 2 */ + atc->min_addr_mask = page_size -1 ; + addr_lookup_indexes_size = address_width - log_page_size; +if ((addr_lookup_indexes_size % atc->level_offset) !=0 ) { + goto error; + } + atc->levels = addr_lookup_indexes_size / atc->level_offset; + atc->page_size = page_size; +return atc; + +error: +g_free (atc); +return NULL ; +} + +static inline GHashTable *atc_get_address_space_cache (ATC *atc,uint32_t pasid) +{ +return g_hash_table_lookup (atc->address_spaces, + (gconstpointer )(uintptr_t )pasid); +} + +void atc_create_address_space_cache (ATC *atc,uint32_t pasid) +{ +GHashTable *as_cache; + + as_cache =atc_get_address_space_cache (atc, pasid); +if (!as_cache) { + as_cache =g_hash_table_new_full (atc_addr_key_hash, + atc_addr_key_equal, +NULL , g_free); +g_hash_table_replace (atc->address_spaces, + (gpointer )(uintptr_t )pasid, as_cache); + } +} + +void atc_delete_address_space_cache (ATC *atc,uint32_t pasid) +{ +g_hash_table_remove (atc->address_spaces, (gpointer )(uintptr_t )pasid); +} + +int atc_update (ATC *atc,IOMMUTLBEntry *entry) +{ +IOMMUTLBEntry *value; +GHashTable *as_cache =atc_get_address_space_cache (atc, entry->pasid); +if (!as_cache) { +return -ENODEV ; + } + value =g_memdup2 (entry,sizeof (*value)); +g_hash_table_replace (as_cache, (gpointer)(entry->iova), value); +return 0 ; +} + +IOMMUTLBEntry *atc_lookup (ATC *atc,uint32_t pasid,hwaddr addr) +{ +IOMMUTLBEntry *entry; +hwaddr mask = atc->min_addr_mask; +hwaddr key = addr & (~mask); +GHashTable *as_cache =atc_get_address_space_cache (atc, pasid); + +if (!as_cache) { +return NULL ; + } + +/* +* Iterate over the possible page sizes and try to find a hit +*/ +for (uint8_t level =0 ; level < atc->levels; ++level) { + entry =g_hash_table_lookup (as_cache, (gconstpointer )key); +if (entry && (mask == entry->addr_mask)) { +return entry; + } + mask = (mask << atc->level_offset) | ((1 << atc->level_offset) -1 ); + key = addr & (~mask); + } + +return NULL ; +} + +static gboolean atc_invalidate_entry_predicate (gpointer key,gpointer value, +gpointer user_data) +{ +IOMMUTLBEntry *entry = (IOMMUTLBEntry *)value; +IOMMUTLBEntry *target = (IOMMUTLBEntry *)user_data; +hwaddr target_mask = ~target->addr_mask; +hwaddr entry_mask = ~entry->addr_mask; +return ((target->iova & target_mask) == (entry->iova & target_mask)) || + ((target->iova & entry_mask) == (entry->iova & entry_mask)); +} + +void atc_invalidate (ATC *atc,IOMMUTLBEntry *entry) +{ +GHashTable *as_cache =atc_get_address_space_cache (atc, entry->pasid); +if (!as_cache) { +return ; + } +g_hash_table_foreach_remove (as_cache, + atc_invalidate_entry_predicate, + entry); +} + +void atc_destroy (ATC *atc) +{ +g_hash_table_unref (atc->address_spaces); +} + +size_t atc_get_max_number_of_pages (ATC *atc,hwaddr addr,size_t length) +{ +hwaddr page_mask = ~(atc->min_addr_mask); +size_t result = (length / atc->page_size); +if ((((addr & page_mask) + length -1 ) & page_mask) != + ((addr + length -1 ) & page_mask)) { + result +=1 ; + } +return result + (length % atc->page_size !=0 ?1 :0 ); +} + +void atc_reset (ATC *atc) +{ +g_hash_table_remove_all (atc->address_spaces); +}
util/atc.h new file +117 -0
117 lines not on this page · raw message
util/meson.build +1 -0
@@ -93,6 +93,7 @@ if have_block util_ss.add(files('hbitmap.c')) util_ss.add(files('hexdump.c')) util_ss.add(files('iova-tree.c')) + util_ss.add(files('atc.c')) util_ss.add(files('iov.c')) util_ss.add(files('nvdimm-utils.c')) util_ss.add(files('block-helpers.c'))
Signature · 1 line
2.47.0
PATCH 14/19 atc: Add unit tests
In reply to CLEMENT MATHIEU--DRIFFrom: Clément Mathieu--Drif <[email protected]>
--- tests/unit/meson.build | 1 + tests/unit/test-atc.c | 527 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 528 insertions(+) create mode 100644 tests/unit/test-atc.c
tests/unit/meson.build +1 -0
@@ -48,6 +48,7 @@ tests = { 'test-qapi-util': [], 'test-interval-tree': [], 'test-fifo': [], + 'test-atc': [], } if have_system or have_tools
tests/unit/test-atc.c new file +527 -0
527 lines not on this page · raw message
Signature · 1 line
2.47.0
PATCH 15/19 memory: Add an API for ATS support
In reply to CLEMENT MATHIEU--DRIFFrom: Clément Mathieu--Drif <[email protected]>
IOMMU have to implement iommu_ats_request_translation to support ATS.
Devices can use IOMMU_TLB_ENTRY_TRANSLATION_ERROR to check the tlb entries returned by a translation request.
We decided not to use the existing translation operation for 2 reasons. First, ATS is designed to translate ranges and not isolated addresses. Second, we need ATS-specific parameters.
--- include/exec/memory.h | 26 ++++++++++++++++++++++++++ system/memory.c | 20 ++++++++++++++++++++ 2 files changed, 46 insertions(+)
include/exec/memory.h +26 -0
@@ -148,6 +148,10 @@ struct IOMMUTLBEntry {uint32_t pasid; }; +/* Check if an IOMMU TLB entry indicates a translation error */ +#define IOMMU_TLB_ENTRY_TRANSLATION_ERROR (entry) ((((entry)->perm) & IOMMU_RW) \ + == IOMMU_NONE) +/* * Bitmap for different IOMMUNotifier capabilities. Each notifier can * register with one or multiple IOMMU Notifier capability bit(s).
@@ -525,6 +529,20 @@ struct IOMMUMemoryRegionClass {* @iommu: the IOMMUMemoryRegion */ int (*num_indexes)(IOMMUMemoryRegion *iommu); + +/** +* @iommu_ats_request_translation: +* This method must be implemented if the IOMMU has ATS enabled +* +* @see pci_ats_request_translation_pasid +*/ +ssize_t (*iommu_ats_request_translation)(IOMMUMemoryRegion *iommu, +bool priv_req,bool exec_req, +hwaddr addr,size_t length, +bool no_write, +IOMMUTLBEntry *result, +size_t result_length, +uint32_t *err_count); };typedef struct RamDiscardListener RamDiscardListener ;
@@ -1883,6 +1901,14 @@ void memory_region_iommu_replay(IOMMUMemoryRegion *iommu_mr, IOMMUNotifier *n);void memory_region_unregister_iommu_notifier (MemoryRegion *mr,IOMMUNotifier *n); +ssize_t memory_region_iommu_ats_request_translation (IOMMUMemoryRegion *iommu_mr, +bool priv_req,bool exec_req, +hwaddr addr,size_t length, +bool no_write, +IOMMUTLBEntry *result, +size_t result_length, +uint32_t *err_count); +/** * memory_region_iommu_get_attr: return an IOMMU attr if get_attr() is * defined on the IOMMU.
system/memory.c +20 -0
@@ -2011,6 +2011,26 @@ void memory_region_unregister_iommu_notifier(MemoryRegion *mr,memory_region_update_iommu_notify_flags (iommu_mr,NULL ); } +ssize_t memory_region_iommu_ats_request_translation (IOMMUMemoryRegion *iommu_mr, +bool priv_req, +bool exec_req, +hwaddr addr,size_t length, +bool no_write, +IOMMUTLBEntry *result, +size_t result_length, +uint32_t *err_count) +{ +IOMMUMemoryRegionClass *imrc =memory_region_get_iommu_class_nocheck (iommu_mr); + +if (!imrc->iommu_ats_request_translation) { +return -ENODEV ; + } + +return imrc->iommu_ats_request_translation (iommu_mr, priv_req, exec_req, + addr, length, no_write, result, + result_length, err_count); +} +void memory_region_notify_iommu_one (IOMMUNotifier *notifier,const IOMMUTLBEvent *event) {
Signature · 1 line
2.47.0
Re: [PATCH v1 15/19] memory: Add an API for ATS support
Quoted 77 lines · CLEMENT MATHIEU--DRIF
From: Clément Mathieu--Drif <[email protected]>
IOMMU have to implement
iommu_ats_request_translationto support ATS.Devices can use
IOMMU_TLB_ENTRY_TRANSLATION_ERRORto check the tlb entries returned by a translation request.We decided not to use the existing translation operation for 2 reasons. First, ATS is designed to translate ranges and not isolated addresses. Second, we need ATS-specific parameters.
Signed-off-by Clément Mathieu--Drif <[email protected]>--- include/exec/memory.h | 26 ++++++++++++++++++++++++++ system/memory.c | 20 ++++++++++++++++++++ 2 files changed, 46 insertions(+)include/exec/memory.h +26
-0@@ -148,6 +148,10 @@ struct IOMMUTLBEntry {uint32_t pasid; }; +/* Check if an IOMMU TLB entry indicates a translation error */ +#define IOMMU_TLB_ENTRY_TRANSLATION_ERROR (entry) ((((entry)->perm) & IOMMU_RW) \ + == IOMMU_NONE) +/* * Bitmap for different IOMMUNotifier capabilities. Each notifier can * register with one or multiple IOMMU Notifier capability bit(s). @@ -525,6 +529,20 @@ struct IOMMUMemoryRegionClass {* @iommu: the IOMMUMemoryRegion */ int (*num_indexes)(IOMMUMemoryRegion *iommu); + +/** +* @iommu_ats_request_translation: +* This method must be implemented if the IOMMU has ATS enabled +* +* @see pci_ats_request_translation_pasid +*/ +ssize_t (*iommu_ats_request_translation)(IOMMUMemoryRegion *iommu, +bool priv_req,bool exec_req, +hwaddr addr,size_t length, +bool no_write, +IOMMUTLBEntry *result, +size_t result_length, +uint32_t *err_count); };typedef struct RamDiscardListener RamDiscardListener ;@@ -1883,6 +1901,14 @@ void memory_region_iommu_replay(IOMMUMemoryRegion *iommu_mr, IOMMUNotifier *n);void memory_region_unregister_iommu_notifier (MemoryRegion *mr,IOMMUNotifier *n); +ssize_t memory_region_iommu_ats_request_translation (IOMMUMemoryRegion *iommu_mr, +bool priv_req,bool exec_req, +hwaddr addr,size_t length, +bool no_write, +IOMMUTLBEntry *result, +size_t result_length, +uint32_t *err_count); +/** * memory_region_iommu_get_attr: return an IOMMU attr if get_attr() is * defined on the IOMMU. system/memory.c +1
-0@@ -2011,6 +2011,26 @@ void memory_region_unregister_iommu_notifier(MemoryRegion *mr,memory_region_update_iommu_notify_flags (iommu_mr,NULL ); } +ssize_t memory_region_iommu_ats_request_translation(IOMMUMemoryRegion *iommu_mr,
system/memory.c void memory_region_unregister_iommu_notifier(MemoryRegion *mr, · line 2024
Quoted 10 lines
system/memory.c +10
-0+bool priv_req, + bool exec_req, + hwaddr addr, size_t length, + bool no_write, + IOMMUTLBEntry *result, + size_t result_length, + uint32_t *err_count) +{ + IOMMUMemoryRegionClass *imrc = memory_region_get_iommu_class_nocheck(iommu_mr); +
line too long - just call the variable mr.
Quoted 14 lines
system/memory.c +9
-0+if (!imrc->iommu_ats_request_translation) { +return -ENODEV ; + } + +return imrc->iommu_ats_request_translation (iommu_mr, priv_req, exec_req, + addr, length, no_write, result, + result_length, err_count); +} +void memory_region_notify_iommu_one (IOMMUNotifier *notifier,const IOMMUTLBEvent *event) {Signature · 1 line
2.47.0
Re: [PATCH v1 15/19] memory: Add an API for ATS support
Quoted 80 lines · Michael S. Tsirkin
Caution: External email. Do not open attachments or click links, unless this email comes from a known sender and you know the content is safe.
Quoted 76 lines · CLEMENT MATHIEU--DRIF
From: Clément Mathieu--Drif <[email protected]>
IOMMU have to implement
iommu_ats_request_translationto support ATS.Devices can use
IOMMU_TLB_ENTRY_TRANSLATION_ERRORto check the tlb entries returned by a translation request.We decided not to use the existing translation operation for 2 reasons. First, ATS is designed to translate ranges and not isolated addresses. Second, we need ATS-specific parameters.
Signed-off-by Clément Mathieu--Drif <[email protected]>--- include/exec/memory.h | 26 ++++++++++++++++++++++++++ system/memory.c | 20 ++++++++++++++++++++ 2 files changed, 46 insertions(+)include/exec/memory.h +26
-0@@ -148,6 +148,10 @@ struct IOMMUTLBEntry {uint32_t pasid; }; +/* Check if an IOMMU TLB entry indicates a translation error */ +#define IOMMU_TLB_ENTRY_TRANSLATION_ERROR (entry) ((((entry)->perm) & IOMMU_RW) \ + == IOMMU_NONE) +/* * Bitmap for different IOMMUNotifier capabilities. Each notifier can * register with one or multiple IOMMU Notifier capability bit(s). @@ -525,6 +529,20 @@ struct IOMMUMemoryRegionClass {* @iommu: the IOMMUMemoryRegion */ int (*num_indexes)(IOMMUMemoryRegion *iommu); + +/** +* @iommu_ats_request_translation: +* This method must be implemented if the IOMMU has ATS enabled +* +* @see pci_ats_request_translation_pasid +*/ +ssize_t (*iommu_ats_request_translation)(IOMMUMemoryRegion *iommu, +bool priv_req,bool exec_req, +hwaddr addr,size_t length, +bool no_write, +IOMMUTLBEntry *result, +size_t result_length, +uint32_t *err_count); };typedef struct RamDiscardListener RamDiscardListener ;@@ -1883,6 +1901,14 @@ void memory_region_iommu_replay(IOMMUMemoryRegion *iommu_mr, IOMMUNotifier *n);void memory_region_unregister_iommu_notifier (MemoryRegion *mr,IOMMUNotifier *n); +ssize_t memory_region_iommu_ats_request_translation (IOMMUMemoryRegion *iommu_mr, +bool priv_req,bool exec_req, +hwaddr addr,size_t length, +bool no_write, +IOMMUTLBEntry *result, +size_t result_length, +uint32_t *err_count); +/** * memory_region_iommu_get_attr: return an IOMMU attr if get_attr() is * defined on the IOMMU. system/memory.c +0
-0@@ -2011,6 +2011,26 @@ void memory_region_unregister_iommu_notifier(MemoryRegion *mr,memory_region_update_iommu_notify_flags (iommu_mr,NULL ); }
Quoted 13 lines
system/memory.c void memory_region_unregister_iommu_notifier(MemoryRegion *mr, · line 2024
Quoted 11 lines
system/memory.c +11
-0+ssize_t memory_region_iommu_ats_request_translation (IOMMUMemoryRegion *iommu_mr, +bool priv_req, +bool exec_req, +hwaddr addr,size_t length, +bool no_write, +IOMMUTLBEntry *result, +size_t result_length, +uint32_t *err_count) +{ +IOMMUMemoryRegionClass *imrc =memory_region_get_iommu_class_nocheck (iommu_mr); +line too long - just call the variable mr.
It's still too long with mr, will do this instead:
IOMMUMemoryRegionClass *imrc =
memory_region_get_iommu_class_nocheck(iommu_mr);
By the way, I think I figured out why patch encoding is not correct. Will fix this in the next spin.
Quoted 15 lines
Quoted 15 lines
system/memory.c +9
-1+if (!imrc->iommu_ats_request_translation) { +return -ENODEV ; + } + +return imrc->iommu_ats_request_translation (iommu_mr, priv_req, exec_req, + addr, length, no_write, result, + result_length, err_count); +} +void memory_region_notify_iommu_one (IOMMUNotifier *notifier,const IOMMUTLBEvent *event) { --2.47.0
PATCH 16/19 pci: Add a pci-level API for ATS
In reply to CLEMENT MATHIEU--DRIFFrom: Clément Mathieu--Drif <[email protected]>
Devices implementing ATS can send translation requests using pci_ats_request_translation_pasid.
The invalidation events are sent back to the device using the iommu notifier managed with pci_register_iommu_tlb_event_notifier and pci_unregister_iommu_tlb_event_notifier
--- hw/pci/pci.c | 46 +++++++++++++++++++++++++++++++++++++++ include/hw/pci/pci.h | 52 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 98 insertions(+)
hw/pci/pci.c +46 -0
@@ -2844,6 +2844,52 @@ void pci_device_unset_iommu_device(PCIDevice *dev) } } +ssize_t pci_ats_request_translation_pasid (PCIDevice *dev,uint32_t pasid, +bool priv_req,bool exec_req, +hwaddr addr,size_t length, +bool no_write,IOMMUTLBEntry *result, +size_t result_length, +uint32_t *err_count) +{ +IOMMUMemoryRegion *iommu_mr =pci_device_iommu_memory_region_pasid (dev, + pasid); + +assert (result_length); + +if (!iommu_mr || !pcie_ats_enabled (dev)) { +return -EPERM ; + } +return memory_region_iommu_ats_request_translation (iommu_mr, priv_req, + exec_req, addr, length, + no_write, result, + result_length, + err_count); +} + +int pci_register_iommu_tlb_event_notifier (PCIDevice *dev,uint32_t pasid, +IOMMUNotifier *n) +{ +IOMMUMemoryRegion *iommu_mr =pci_device_iommu_memory_region_pasid (dev, + pasid); +if (!iommu_mr) { +return -EPERM ; + } +return memory_region_register_iommu_notifier (MEMORY_REGION (iommu_mr), n, + &error_fatal); +} + +int pci_unregister_iommu_tlb_event_notifier (PCIDevice *dev,uint32_t pasid, +IOMMUNotifier *n) +{ +IOMMUMemoryRegion *iommu_mr =pci_device_iommu_memory_region_pasid (dev, + pasid); +if (!iommu_mr) { +return -EPERM ; + } +memory_region_unregister_iommu_notifier (MEMORY_REGION (iommu_mr), n); +return 0 ; +} +void pci_setup_iommu (PCIBus *bus,const PCIIOMMUOps *ops,void *opaque) {/*
include/hw/pci/pci.h +52 -0
58 lines not on this page · raw message
Signature · 1 line
2.47.0
PATCH 17/19 intel_iommu: Set address mask when a translation fails and adjust W permission
In reply to CLEMENT MATHIEU--DRIFFrom: Clément Mathieu--Drif <[email protected]>
Implements the behavior defined in section 10.2.3.5 of PCIe spec rev 5. This is needed by devices that support ATS.
--- hw/i386/intel_iommu.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
hw/i386/intel_iommu.c +9 -3
@@ -2100,7 +2100,8 @@ static bool vtd_do_iommu_translate(VTDAddressSpace *vtd_as, PCIBus *bus,uint8_t bus_num =pci_bus_num (bus);VTDContextCacheEntry *cc_entry;uint64_t pte, page_mask; -uint32_t level, pasid = vtd_as->pasid; +uint32_t level =UINT32_MAX ; +uint32_t pasid = vtd_as->pasid;uint16_t source_id =PCI_BUILD_BDF (bus_num, devfn);int ret_fr;bool is_fpd_set = false;
@@ -2262,7 +2263,7 @@ out: entry->iova = addr & page_mask; entry->translated_addr =vtd_get_pte_addr (pte, s->aw_bits) & page_mask; entry->addr_mask = ~page_mask; - entry->perm = access_flags; + entry->perm = (is_write ? access_flags : (access_flags & (~IOMMU_WO )));/* pasid already set */ return true;
@@ -2270,7 +2271,12 @@ error:vtd_iommu_unlock (s); entry->iova =0 ; entry->translated_addr =0 ; - entry->addr_mask =0 ; +/* +* Set the mask for ATS (the range must be present even when the +* translation fails : PCIe rev 5 10.2.3.5) +*/ + entry->addr_mask = (level !=UINT32_MAX ) ? + (~vtd_pt_level_page_mask (level)) : (~VTD_PAGE_MASK_4K ); entry->perm =IOMMU_NONE ; entry->pasid =PCI_NO_PASID ;return false;
Signature · 1 line
2.47.0
PATCH 18/19 intel_iommu: Return page walk level even when the translation fails
In reply to CLEMENT MATHIEU--DRIFFrom: Clément Mathieu--Drif <[email protected]>
We use this information in vtd_do_iommu_translate to populate the IOMMUTLBEntry and indicate the correct page mask. This prevents ATS devices from sending many useless translation requests when a megapage or gigapage iova is not mapped to a physical address.
--- hw/i386/intel_iommu.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-)
hw/i386/intel_iommu.c +8 -9
@@ -1995,9 +1995,9 @@ static int vtd_iova_to_flpte(IntelIOMMUState *s, VTDContextEntry *ce, uint32_t pasid) {dma_addr_t addr =vtd_get_iova_pgtbl_base (s, ce, pasid); -uint32_t level =vtd_get_iova_level (s, ce, pasid);uint32_t offset;uint64_t flpte, flag_ad =VTD_FL_A ; + *flpte_level =vtd_get_iova_level (s, ce, pasid);if (!vtd_iova_fl_check_canonical (s, iova, ce, pasid)) { error_report_once("%s: detected non canonical IOVA (iova=0x%" PRIx64 ","
@@ -2006,11 +2006,11 @@ static int vtd_iova_to_flpte(IntelIOMMUState *s, VTDContextEntry *ce, }while (true) { - offset =vtd_iova_level_offset (iova, level); + offset =vtd_iova_level_offset (iova, *flpte_level); flpte =vtd_get_pte (addr, offset);if (flpte == (uint64_t )-1 ) { -if (level ==vtd_get_iova_level (s, ce, pasid)) { +if (*flpte_level ==vtd_get_iova_level (s, ce, pasid)) {/* Invalid programming of pasid-entry */ return -VTD_FR_PASID_ENTRY_FSPTPTR_INV ; }else {
@@ -2036,15 +2036,15 @@ static int vtd_iova_to_flpte(IntelIOMMUState *s, VTDContextEntry *ce,if (is_write && !(flpte &VTD_FL_RW )) {return -VTD_FR_SM_WRITE ; } -if (vtd_flpte_nonzero_rsvd (flpte, level)) { +if (vtd_flpte_nonzero_rsvd (flpte, *flpte_level)) {error_report_once ("%s: detected flpte reserved non-zero " "iova=0x%" PRIx64", level=0x%" PRIx32"flpte=0x%" PRIx64", pasid=0x%" PRIX32 ")" , - __func__, iova, level, flpte, pasid); + __func__, iova, *flpte_level, flpte, pasid);return -VTD_FR_FS_PAGING_ENTRY_RSVD ; } -if (vtd_is_last_pte (flpte, level) && is_write) { +if (vtd_is_last_pte (flpte, *flpte_level) && is_write) { flag_ad |=VTD_FL_D ; }
@@ -2052,14 +2052,13 @@ static int vtd_iova_to_flpte(IntelIOMMUState *s, VTDContextEntry *ce,return -VTD_FR_FS_BIT_UPDATE_FAILED ; } -if (vtd_is_last_pte (flpte, level)) { +if (vtd_is_last_pte (flpte, *flpte_level)) { *flptep = flpte; - *flpte_level = level;return 0 ; } addr =vtd_get_pte_addr (flpte, aw_bits); - level--; + (*flpte_level)--; } }
Signature · 1 line
2.47.0
PATCH 19/19 intel_iommu: Add support for ATS
In reply to CLEMENT MATHIEU--DRIFFrom: Clément Mathieu--Drif <[email protected]>
--- hw/i386/intel_iommu.c | 71 ++++++++++++++++++++++++++++++++-- hw/i386/intel_iommu_internal.h | 1 + 2 files changed, 69 insertions(+), 3 deletions(-)
hw/i386/intel_iommu.c +68 -3
@@ -4166,12 +4166,10 @@ static void vtd_report_ir_illegal_access(VTDAddressSpace *vtd_as,bool is_fpd_set = false;VTDContextEntry ce; -assert (vtd_as->pasid !=PCI_NO_PASID ); -/* Try out best to fetch FPD, we can't do anything more */ if (vtd_dev_to_context_entry (s, bus_n, vtd_as->devfn, &ce) ==0 ) { is_fpd_set = ce.lo &VTD_CONTEXT_ENTRY_FPD ; -if (!is_fpd_set && s->root_scalable) { +if (!is_fpd_set && s->root_scalable && vtd_as->pasid !=PCI_NO_PASID ) {vtd_ce_get_pasid_fpd (s, &ce, &is_fpd_set, vtd_as->pasid); } }
@@ -4745,6 +4743,71 @@ static IOMMUMemoryRegion *vtd_get_memory_region_pasid(PCIBus *bus,return &vtd_as->iommu; } +static IOMMUTLBEntry vtd_iommu_ats_do_translate (IOMMUMemoryRegion *iommu, +hwaddr addr, +IOMMUAccessFlags flags, +int iommu_idx) +{ +IOMMUTLBEntry entry; +VTDAddressSpace *vtd_as =container_of (iommu, VTDAddressSpace, iommu); + +if (vtd_is_interrupt_addr (addr)) { +vtd_report_ir_illegal_access (vtd_as, addr, flags &IOMMU_WO ); + entry.iova =0 ; + entry.translated_addr =0 ; + entry.addr_mask = ~VTD_PAGE_MASK_4K ; + entry.perm =IOMMU_NONE ; + entry.pasid =PCI_NO_PASID ; + }else { + entry =vtd_iommu_translate (iommu, addr, flags, iommu_idx); + } +return entry; +} + +static ssize_t vtd_iommu_ats_request_translation (IOMMUMemoryRegion *iommu, +bool priv_req,bool exec_req, +hwaddr addr,size_t length, +bool no_write, +IOMMUTLBEntry *result, +size_t result_length, +uint32_t *err_count) +{ +IOMMUAccessFlags flags =IOMMU_ACCESS_FLAG_FULL (true, !no_write, exec_req, + priv_req, false, false); +ssize_t res_index =0 ; +hwaddr target_address = addr + length; +IOMMUTLBEntry entry; + + *err_count =0 ; + +while ((addr < target_address) && (res_index < result_length)) { + entry =vtd_iommu_ats_do_translate (iommu, addr, flags,0 ); +if (!IOMMU_TLB_ENTRY_TRANSLATION_ERROR (&entry)) {/* Translation done */ +/* +* 4.1.2 : Global Mapping (G) : Remapping hardware provides a value +* of 0 in this field +*/ + entry.perm &= ~IOMMU_GLOBAL ; + }else { + *err_count +=1 ; + } + result[res_index] = entry; + res_index +=1 ; + addr = (addr & (~entry.addr_mask)) + (entry.addr_mask +1 ); + } + +/* Buffer too small */ +if (addr < target_address) { +return -ENOMEM ; + } +return res_index; +} + +static uint64_t vtd_get_min_page_size (IOMMUMemoryRegion *iommu) +{ +return VTD_PAGE_SIZE ; +} +static PCIIOMMUOps vtd_iommu_ops = { .get_address_space = vtd_host_dma_iommu, .get_memory_region_pasid = vtd_get_memory_region_pasid,
@@ -4922,6 +4985,8 @@ static void vtd_iommu_memory_region_class_init(ObjectClass *klass, imrc->translate = vtd_iommu_translate; imrc->notify_flag_changed = vtd_iommu_notify_flag_changed; imrc->replay = vtd_iommu_replay; + imrc->iommu_ats_request_translation = vtd_iommu_ats_request_translation; + imrc->get_min_page_size = vtd_get_min_page_size; }static const TypeInfo vtd_iommu_memory_region_info = {
hw/i386/intel_iommu_internal.h +1 -0
@@ -192,6 +192,7 @@#define VTD_ECAP_SC (1ULL << 7)#define VTD_ECAP_MHMV (15ULL << 20)#define VTD_ECAP_SRS (1ULL << 31) +#define VTD_ECAP_NWFS (1ULL << 33)#define VTD_ECAP_PSS (19ULL << 35)#define VTD_ECAP_PASID (1ULL << 40)#define VTD_ECAP_SMTS (1ULL << 43)
Signature · 1 line
2.47.0