ARM SMMUv3 Emulation Support
v4 cover ARM SMMUv3 Emulation Support
This series introduces the emulation code for ARM SMMUv3. This is the continuation of Prem's work [1].
This v4 is yet another visibility step as many restrictions apply to the model at the moment:
- only VMSAv8-64 is supported (no VMSAv8-32)
- only stage 1 has been tested (Block PTE still needs to be implemented and tested though)
- no integration with VFIO. We are missing some quirks to force the guest to invalidate TLBs when updating the page tables. But replay will be easily implemented upon new page table scan.
- stage 2 is not supported. There is partial support when decoding configuration information but no support at page table walk level. This is my next step.
- at the moment I don't plan to support nested S1 + S2 as I understand no SW stack uses it at the moment.
- no support for HYP mappings
- register fine emulation, commands, interrupts and errors were not accurately tested. Handling is sufficient to run a guest with a virtio-net-pci device using dma ops.
- At the moment no change to the PCIe instantiation (further discussions about SMMUv2 TBUs versus SMMUv3 TLBs needed).
- In ACPI mode, the guest must feature the IOMMU probe deferral series wich fixes streamid multiple lookup
The smmu is instantiated when passing the smmu option to machvirt:
"-M virt-2.10,smmu"
Best Regards
Eric
This series can be found at:
v4: https://github.com/eauger/qemu/tree/v2.9-SMMU-v4
Testing:
- booted a 4.11 guest in dt and acpi mode with an iommu_platform virtio-net-pci device (using dma ops). Tested with the following guest combinations: 4K page - 39 bit VA, 4K - 48b, 64K - 39b, 64K - 48b.
- Also Prem's unit test environment was rebased (available upon demand)
References:
[1] Prem's last iteration:
- https://lists.gnu.org/archive/html/qemu-devel/2016-08/msg03531.html
History:
v3 -> v4 [Eric]:
- page table walk rewritten to allow scan of the page table within a range of IOVA. This prepares for VFIO integration and replay.
- configuration parsing partially reworked.
- do not advertise unsupported/untested features: S2, S1 + S2, HYP, PRI, ATS, ..
- added ACPI table generation
- migrated to dynamic traces
- mingw compilation fix
v2 -> v3 [Eric]:
- rebased on 2.9
- mostly code and patch reorganization to ease the review process
- optional patches removed. They may be handled separately. I am currently working on ACPI enablement.
- optional instantiation of the smmu in mach-virt
- removed [2/9] (fdt functions) since not mandated
- start splitting main patch into base and derived object
- no new function feature added
v1 -> v2 [Prem]:
- Adopted review comments from Eric Auger
- Make SMMU_DPRINTF to internally call qemu_log (since translation requests are too many, we need control on the type of log we want)
- SMMUTransCfg modified to suite simplicity
- Change RegInfo to uint64 register array
- Code cleanup
- Test cleanups
- Reshuffled patches
v0 -> v1 [Prem]:
- As per SMMUv3 spec 16.0 (only is_ste_consistant() is noticeable)
- Reworked register access/update logic
- Factored out translation code for
- single point bug fix
- sharing/removal in future
- (optional) Unit tests added, with PCI test device
- S1 with 4k/64k, S1+S2 with 4k/64k
- (S1 or S2) only can be verified by Linux 4.7 driver
- (optional) Priliminary ACPI support
v0 [Prem]:
- Implements SMMUv3 spec 11.0
- Supported for PCIe devices,
- Command Queue and Event Queue supported
- LPAE only, S1 is supported and Tested, S2 not tested
- BE mode Translation not supported
- IRQ support (legacy, no MSI)
- Tested with DPDK and e1000
Eric Auger (2):
hw/arm/smmu-common: smmu base class
hw/arm/virt: Add 2.10 machine type
Prem Mallappa (3):
hw/arm/smmuv3: smmuv3 emulation model hw/arm/virt: Add SMMUv3 to the virt board hw/arm/virt-acpi-build: add smmuv3 node in IORT table
default-configs/aarch64-softmmu.mak | 1 + hw/arm/Makefile.objs | 1 + hw/arm/smmu-common.c | 419 +++++++++++++ hw/arm/smmu-internal.h | 97 +++ hw/arm/smmuv3-internal.h | 603 +++++++++++++++++++ hw/arm/smmuv3.c | 1134 +++++++++++++++++++++++++++++++++++ hw/arm/trace-events | 44 ++ hw/arm/virt-acpi-build.c | 56 +- hw/arm/virt.c | 109 +++- include/hw/acpi/acpi-defs.h | 15 + include/hw/arm/smmu-common.h | 125 ++++ include/hw/arm/smmuv3.h | 87 +++ include/hw/arm/virt.h | 5 + 13 files changed, 2687 insertions(+), 9 deletions(-) create mode 100644 hw/arm/smmu-common.c create mode 100644 hw/arm/smmu-internal.h create mode 100644 hw/arm/smmuv3-internal.h create mode 100644 hw/arm/smmuv3.c create mode 100644 include/hw/arm/smmu-common.h create mode 100644 include/hw/arm/smmuv3.h
Signature · 1 line
2.5.5
RFC v4 1/5 hw/arm/smmu-common: smmu base class
Introduces the base device and class for the ARM smmu. Implements VMSAv8-64 table lookup and translation. VMSAv8-32 is not yet implemented.
For VFIO integration we will need to notify mapping changes of an input range and skipped unmapped regions. table walk helper allows.
---
v3 -> v4:
- reworked page table walk to prepare for VFIO integration (capability to scan a range of IOVA). Same function is used for translate for a single iova. This is largely inspired from intel_iommu.c
- as the translate function was not straightforward to me, I tried to stick more closely to the VMSA spec.
- remove support of nested stage (kernel driver does not support it anyway)
- introduce smmu-internal.h to put page table definitions
- added smmu_find_as_from_bus_num
- SMMU_PCI_BUS_MAX and SMMU_PCI_DEVFN_MAX in smmu-common header
- new fields in SMMUState:
- iommu_ops, smmu_as_by_busptr, smmu_as_by_bus_num
- use error_report and trace events
- add aa64[] field in SMMUTransCfg
v3:
- moved the base code in a separate patch to ease the review.
- clearer separation between base class and smmuv3 class
- translate_* only implemented as class methods
--- default-configs/aarch64-softmmu.mak | 1 + hw/arm/Makefile.objs | 1 + hw/arm/smmu-common.c | 419 ++++++++++++++++++++++++++++++++++++ hw/arm/smmu-internal.h | 97 +++++++++ hw/arm/trace-events | 12 ++ include/hw/arm/smmu-common.h | 125 +++++++++++ 6 files changed, 655 insertions(+) create mode 100644 hw/arm/smmu-common.c create mode 100644 hw/arm/smmu-internal.h create mode 100644 include/hw/arm/smmu-common.h
default-configs/aarch64-softmmu.mak +1 -0
@@ -7,3 +7,4 @@ CONFIG_AUX=y CONFIG_DDC=y CONFIG_DPCD=y CONFIG_XLNX_ZYNQMP=y +CONFIG_ARM_SMMUV3=y
hw/arm/Makefile.objs +1 -0
@@ -18,3 +18,4 @@ obj-$(CONFIG_FSL_IMX25) += fsl-imx25.o imx25_pdk.o obj-$(CONFIG_FSL_IMX31) += fsl-imx31.o kzm.o obj-$(CONFIG_FSL_IMX6) += fsl-imx6.o sabrelite.o obj-$(CONFIG_ASPEED_SOC) += aspeed_soc.o aspeed.o +obj-$(CONFIG_ARM_SMMUV3) += smmu-common.o
hw/arm/smmu-common.c new file +419 -0
419 lines not on this page · raw message
hw/arm/smmu-internal.h new file +97 -0
97 lines not on this page · raw message
hw/arm/trace-events +12 -0
@@ -2,3 +2,15 @@ # hw/arm/virt-acpi-build.c virt_acpi_setup(void) "No fw cfg or ACPI disabled. Bailing out." + +# hw/arm/smmu-common.c + +smmu_page_walk(int stage, uint64_t baseaddr, int first_level, uint64_t start, uint64_t end) "stage=%d, baseaddr=0x%"PRIx64", first level=%d, start=0x%"PRIx64", end=0x%"PRIx64 +smmu_page_walk_level_in(int level, uint64_t baseaddr, int granule_sz, uint64_t start, uint64_t end, uint64_t subpage_size) "level=%d baseaddr=0x%"PRIx64" granule=%d, start=0x%"PRIx64" end=0x%"PRIx64", subpage_size=0x%lx" +smmu_page_walk_level(int level, uint64_t iova, uint64_t baseaddr, uint32_t offset, uint64_t pte) "level=%d iova=0x%lx baseaddr=0x%"PRIx64" offset=0x%x => pte=0x%lx" +smmu_page_walk_level_res_invalid_pte(uint64_t baseaddr, uint32_t offset, uint64_t pte) "baseaddr=0x%"PRIx64" offset=0x%x pte=0x%lx" +smmu_page_walk_level_page_pte(uint64_t pte, uint64_t address) "pte=0x%"PRIx64" page address = 0x%"PRIx64 +smmu_page_walk_level_block_pte(uint64_t pte, uint64_t address) "pte=0x%"PRIx64" block address = 0x%"PRIx64 +smmu_page_walk_level_table_pte(uint64_t pte, uint64_t address) "pte=0x%"PRIx64" next table address = 0x%"PRIx64 +smmu_get_pte(uint64_t baseaddr, int index, uint64_t pteaddr, uint64_t pte) "baseaddr=0x%"PRIx64" index=0x%x, pteaddr=0x%"PRIx64", pte=0x%"PRIx64 +smmu_set_translated_address(hwaddr iova, hwaddr va) "iova = 0x%"PRIx64" -> pa = 0x%"PRIx64
include/hw/arm/smmu-common.h new file +125 -0
125 lines not on this page · raw message
Signature · 1 line
2.5.5
Re: [Qemu-arm] [RFC v4 1/5] hw/arm/smmu-common: smmu base class
Quoted 45 lines · Eric Auger
Introduces the base device and class for the ARM smmu. Implements VMSAv8-64 table lookup and translation. VMSAv8-32 is not yet implemented.
For VFIO integration we will need to notify mapping changes of an input range and skipped unmapped regions. table walk helper allows.
Signed-off-by Eric Auger <[email protected]>Signed-off-by Prem Mallappa <[email protected]>hw/arm/smmu-common.c new file +33
-0+/** +* smmu_page_walk_level_64 - Walk an IOVA range from a specific level +* @baseaddr: table base address corresponding to @level +* @level: level +* @cfg: translation config +* @start: end of the IOVA range +* @end: end of the IOVA range +* @hook_fn: the hook that to be called for each detected area +* @private: private data for the hook function +* @read: whether parent level has read permission +* @write: whether parent level has write permission +* @must_translate: indicates whether each iova of the range +* must be translated or whether failure is allowed +* @notify_unmap: whether we should notify invalid entries +* +* Return 0 on success, < 0 on errors not related to translation +* process, > 1 on errors related to translation process (only +* if must_translate is set) +*/ +static int +smmu_page_walk_level_64 (dma_addr_t baseaddr,int level, +SMMUTransCfg *cfg,uint64_t start,uint64_t end, +smmu_page_walk_hook hook_fn,void *private, +bool read,bool write,bool must_translate, +bool notify_unmap) +{ +/* table pte */ + next_table_baseaddr =get_table_pte_address (pte, granule_sz); +trace_smmu_page_walk_level_table_pte (pte, next_table_baseaddr); + ret =smmu_page_walk_level_64 (next_table_baseaddr, level +1 , cfg, + iova,MIN (iova_next, end), + hook_fn, private, read_cur, write_cur, + must_translate, notify_unmap);
hw/arm/smmu-common.c line 275
Quoted 10 lines
hw/arm/smmu-common.c new file +10
-0+if (!ret) { +return ret; + } + +next: + iova = iova_next; + } + +return SMMU_TRANS_ERR_NONE ; +}
It seems a bit odd that this function is recursive -- I was expecting it just to loop through for each successive level until it hits a block/page entry, as the LPAE table walk code in target/arm/helper.c does...
thanks
-- PMM
Re: [Qemu-arm] [RFC v4 1/5] hw/arm/smmu-common: smmu base class
Hi Peter,
Quoted 53 lines · Peter Maydell
Quoted 52 lines · Eric Auger
Introduces the base device and class for the ARM smmu. Implements VMSAv8-64 table lookup and translation. VMSAv8-32 is not yet implemented.
For VFIO integration we will need to notify mapping changes of an input range and skipped unmapped regions. table walk helper allows.
Signed-off-by Eric Auger <[email protected]>Signed-off-by Prem Mallappa <[email protected]>hw/arm/smmu-common.c new file +40
-0+/** +* smmu_page_walk_level_64 - Walk an IOVA range from a specific level +* @baseaddr: table base address corresponding to @level +* @level: level +* @cfg: translation config +* @start: end of the IOVA range +* @end: end of the IOVA range +* @hook_fn: the hook that to be called for each detected area +* @private: private data for the hook function +* @read: whether parent level has read permission +* @write: whether parent level has write permission +* @must_translate: indicates whether each iova of the range +* must be translated or whether failure is allowed +* @notify_unmap: whether we should notify invalid entries +* +* Return 0 on success, < 0 on errors not related to translation +* process, > 1 on errors related to translation process (only +* if must_translate is set) +*/ +static int +smmu_page_walk_level_64 (dma_addr_t baseaddr,int level, +SMMUTransCfg *cfg,uint64_t start,uint64_t end, +smmu_page_walk_hook hook_fn,void *private, +bool read,bool write,bool must_translate, +bool notify_unmap) +{ +/* table pte */ + next_table_baseaddr =get_table_pte_address (pte, granule_sz); +trace_smmu_page_walk_level_table_pte (pte, next_table_baseaddr); + ret =smmu_page_walk_level_64 (next_table_baseaddr, level +1 , cfg, + iova,MIN (iova_next, end), + hook_fn, private, read_cur, write_cur, + must_translate, notify_unmap); +if (!ret) { +return ret; + } + +next: + iova = iova_next; + }
Quoted 8 lines
hw/arm/smmu-common.c line 275
Quoted 3 lines
hw/arm/smmu-common.c new file +3
-0+ +return SMMU_TRANS_ERR_NONE ; +}It seems a bit odd that this function is recursive -- I was expecting it just to loop through for each successive level until it hits a block/page entry, as the LPAE table walk code in
target/arm/helper.cdoes...
welcome back ;-)
Thanks for the pointer to get_phys_addr_lpae(). Actually I got largely inspired of vtd_page_walk and vtd_page_walk_level of hw/i386/intel_iommu.c which also use the same recursive form.
The peculiarity compared to get_phys_addr_lpae() is that we scan a range of iovas and do not translate a single iova. In case a level entry is not implemented we want to jump to the next one at the same level and in case the entry matches a table entry we want to scan the next level entries matching that iova range.
Does that make more sense?
Thanks
Eric
Quoted 4 lines
thanks
-- PMM
RFC v4 2/5 hw/arm/smmuv3: smmuv3 emulation model
In reply to Eric AugerFrom: Prem Mallappa <[email protected]>
Introduces the SMMUv3 derived model. This is based on System MMUv3 specification (v17).
---
v3 -> v4
- smmu_irq_update
- fix hash key allocation
- set smmu_iommu_ops
- set SMMU_REG_CR0,
- smmuv3_translate: ret.perm not set in bypass mode
- use trace events
- renamed STM2U64 into L1STD_L2PTR and STMSPAN into L1STD_SPAN
- rework smmu_find_ste
- fix tg2granule in TT0/0b10 corresponds to 16kB
v2 -> v3:
- move creation of include/hw/arm/smmuv3.h to this patch to fix compil issue
- compilation allowed
- fix sbus allocation in smmu_init_pci_iommu
- restructure code into headers
- misc cleanups
--- hw/arm/Makefile.objs | 2 +- hw/arm/smmuv3-internal.h | 603 ++++++++++++++++++++++++ hw/arm/smmuv3.c | 1134 ++++++++++++++++++++++++++++++++++++++++++++++ hw/arm/trace-events | 32 ++ include/hw/arm/smmuv3.h | 87 ++++ 5 files changed, 1857 insertions(+), 1 deletion(-) create mode 100644 hw/arm/smmuv3-internal.h create mode 100644 hw/arm/smmuv3.c create mode 100644 include/hw/arm/smmuv3.h
hw/arm/Makefile.objs +1 -1
@@ -18,4 +18,4 @@ obj-$(CONFIG_FSL_IMX25) += fsl-imx25.o imx25_pdk.o obj-$(CONFIG_FSL_IMX31) += fsl-imx31.o kzm.o obj-$(CONFIG_FSL_IMX6) += fsl-imx6.o sabrelite.o obj-$(CONFIG_ASPEED_SOC) += aspeed_soc.o aspeed.o -obj-$(CONFIG_ARM_SMMUV3) += smmu-common.o +obj-$(CONFIG_ARM_SMMUV3) += smmu-common.o smmuv3.o
hw/arm/smmuv3-internal.h new file +603 -0
603 lines not on this page · raw message
hw/arm/smmuv3.c new file +1134 -0
1,134 lines not on this page · raw message
hw/arm/trace-events +32 -0
@@ -14,3 +14,35 @@ smmu_page_walk_level_block_pte(uint64_t pte, uint64_t address) "pte=0x%"PRIx64" smmu_page_walk_level_table_pte(uint64_t pte, uint64_t address) "pte=0x%"PRIx64" next table address = 0x%"PRIx64 smmu_get_pte(uint64_t baseaddr, int index, uint64_t pteaddr, uint64_t pte) "baseaddr=0x%"PRIx64" index=0x%x, pteaddr=0x%"PRIx64", pte=0x%"PRIx64 smmu_set_translated_address(hwaddr iova, hwaddr va) "iova = 0x%"PRIx64" -> pa = 0x%"PRIx64 + +#hw/arm/smmuv3.c +smmuv3_irq_update(uint32_t error, uint32_t gerror, uint32_t gerrorn) "<<<< error:0x%x gerror:0x%x gerrorn:0x%x" +smmuv3_irq_raise(int irq) "irq:%d" +smmuv3_unhandled_cmd(uint32_t type) "Unhandled command type=%d" +smmuv3_cmdq_consume(int error) "CMDQ_ERR: %d" +smmuv3_cmdq_consume_details(hwaddr base, uint32_t cons, uint32_t prod, uint32_t word, uint8_t wrap_cons) "CMDQ base: 0x%"PRIx64" cons:%d prod:%d val:0x%x wrap:%d" +smmuv3_cmdq_consume_sev(void) "CMD_SYNC CS=SEV not supported, ignoring" +smmuv3_cmdq_consume_out(uint8_t prod_wrap, uint32_t prod, uint8_t cons_wrap, uint32_t cons) "prod_wrap:%d, prod:0x%x cons_wrap:%d cons:0x%x" +smmuv3_update(bool is_empty, uint32_t prod, uint32_t cons, uint8_t prod_wrap, uint8_t cons_wrap) "q empty:%d prod:%d cons:%d p.wrap:%d p.cons:%d" +smmuv3_update_check_cmd(int error) "cmdq not enabled or error :0x%x" +smmuv3_update_irq(bool is_pending, uint32_t gerror, uint32_t gerrorn) "irq pend: %d gerror:0x%x gerrorn:0x%x" +smmuv3_is_ste_consistent(bool cfg, bool granule_supported, bool addr_oor, uint32_t aa64, int s2ha, int s2hd, uint64_t s2ttb ) "config[1]:%d gran:%d addr:%d aa64:%d s2ha:%d s2hd:%d s2ttb:0x%"PRIx64 +smmuv3_find_ste(uint16_t sid, uint32_t features, uint16_t sid_split) "SID:0x%x features:0x%x, sid_split:0x%x" +smmuv3_find_ste_2lvl(uint64_t strtab_base, hwaddr l1ptr, int l1_ste_offset, hwaddr l2ptr, int l2_ste_offset, int max_l2_ste) "strtab_base:%lx l1ptr:0x%"PRIx64" l1_off:0x%x, l2ptr:0x%"PRIx64" l2_off:0x%x max_l2_ste:%d" +smmuv3_get_ste(hwaddr addr) "STE addr: 0x%"PRIx64 +smmuv3_walk_pgtable(hwaddr iova, bool is_write) "Input addr: 0x%"PRIx64", is_write=%d" +smmuv3_walk_pgtable_out(hwaddr addr, uint32_t mask, int perm) "DONE: o/p addr:0x%"PRIx64" mask:0x%x perm:%d" +smmuv3_translate_bypass(const char *n, uint16_t sid, hwaddr addr, bool is_write) "%s sid=%d bypass iova:0x%"PRIx64" is_write=%d" +smmuv3_translate_in(uint16_t sid, int pci_bus_num, hwaddr strtab_base) "SID:0x%x bus:%d strtab_base:0x%"PRIx64 +smmuv3_get_cd(hwaddr addr) "CD addr: 0x%"PRIx64 +smmuv3_translate_ok(const char *n, uint16_t sid, hwaddr iova, hwaddr translated, int perm) "%s sid=%d iova=0x%"PRIx64" translated=0x%"PRIx64" perm=0x%x" +smmuv3_update_qreg(uint32_t cons, uint64_t val) "cons written : %d val:0x%"PRIx64 +smmuv3_write_mmio(hwaddr addr, uint64_t val) "addr: 0x%"PRIx64" val:0x%"PRIx64 +smmuv3_write_mmio_idr(hwaddr addr, uint64_t val) "write to RO/Unimpl reg %lx val64:%lx" +smmuv3_write_mmio_evtq_cons_bef_clear(uint32_t prod, uint32_t cons, uint8_t prod_wrap, uint8_t cons_wrap) "Before clearing interrupt prod:0x%x cons:0x%x prod.w:%d cons.w:%d" +smmuv3_write_mmio_evtq_cons_after_clear(uint32_t prod, uint32_t cons, uint8_t prod_wrap, uint8_t cons_wrap) "after clearing interrupt prod:0x%x cons:0x%x prod.w:%d cons.w:%d" +smmuv3_read_mmio(hwaddr addr, uint64_t val, uint32_t cons) "addr: 0x%"PRIx64" val:0x%"PRIx64" cmdq cons:%d" +smmuv3_dump_ste(int i, uint32_t word0, int j, uint32_t word1) "STE[%2d]: %#010x\t STE[%2d]: %#010x" +smmuv3_dump_cd(int i, uint32_t word0, int j, uint32_t word1) "CD[%2d]: %#010x\t CD[%2d]: %#010x" +smmuv3_dump_cmd(int i, uint32_t word0, int j, uint32_t word1) "CMD[%2d]: %#010x\t CMD[%2d]: %#010x" +smmuv3_cfg_stage(int s, uint32_t oas, uint32_t tsz, uint64_t ttbr, bool aa64, uint32_t granule_sz) "TransCFG stage:%d oas:%d tsz:%d ttbr:0x%"PRIx64" aa64:%d granule_sz:%d"
include/hw/arm/smmuv3.h new file +87 -0
87 lines not on this page · raw message
Signature · 1 line
2.5.5
Re: [Qemu-devel] [RFC v4 2/5] hw/arm/smmuv3: smmuv3 emulation model
Quoted 26 lines · Eric Auger
From: Prem Mallappa <[email protected]>
Introduces the SMMUv3 derived model. This is based on System MMUv3 specification (v17).
Signed-off-by Prem Mallappa <[email protected]>Signed-off-by Eric Auger <[email protected]>---
v3 -> v4
-smmu_irq_update
- fix hash key allocation
- setsmmu_iommu_ops
- setSMMU_REG_CR0,
-smmuv3_translate: ret.perm not set in bypass mode
- use trace events
- renamed STM2U64 intoL1STD_L2PTRand STMSPAN intoL1STD_SPAN
- reworksmmu_find_ste
- fix tg2granule in TT0/0b10 corresponds to 16kBv2 -> v3:
- move creation ofinclude/hw/arm/smmuv3.hto this patch to fix compil issue
- compilation allowed
- fix sbus allocation insmmu_init_pci_iommu
- restructure code into headers
- misc cleanups
Quoted 10 lines
--- hw/arm/Makefile.objs | 2 +- hw/arm/smmuv3-internal.h | 603 ++++++++++++++++++++++++ hw/arm/smmuv3.c | 1134 ++++++++++++++++++++++++++++++++++++++++++++++ hw/arm/trace-events | 32 ++ include/hw/arm/smmuv3.h | 87 ++++ 5 files changed, 1857 insertions(+), 1 deletion(-) create mode 100644 hw/arm/smmuv3-internal.h create mode 100644 hw/arm/smmuv3.c create mode 100644 include/hw/arm/smmuv3.h
This is a bit of a big patch for review for my taste -- are there some easy splits into multiple patches possible?
include/hw/arm/smmuv3.h line 37
Quoted 11 lines
include/hw/arm/smmuv3.h new file +11
-0+typedef struct SMMUQueue { + hwaddr base; + uint32_t prod; + uint32_t cons; + union { + struct { + uint8_t prod:1; + uint8_t cons:1; + }; + uint8_t unused; + } wrap;
Use of bitfields here seems a bit odd but I haven't looked at the code really.
thanks
-- PMM
Re: [Qemu-arm] [RFC v4 2/5] hw/arm/smmuv3: smmuv3 emulation model
Hi Peter,
Quoted 38 lines · Peter Maydell
Quoted 26 lines · Eric Auger
From: Prem Mallappa <[email protected]>
Introduces the SMMUv3 derived model. This is based on System MMUv3 specification (v17).
Signed-off-by Prem Mallappa <[email protected]>Signed-off-by Eric Auger <[email protected]>---
v3 -> v4
-smmu_irq_update
- fix hash key allocation
- setsmmu_iommu_ops
- setSMMU_REG_CR0,
-smmuv3_translate: ret.perm not set in bypass mode
- use trace events
- renamed STM2U64 intoL1STD_L2PTRand STMSPAN intoL1STD_SPAN
- reworksmmu_find_ste
- fix tg2granule in TT0/0b10 corresponds to 16kBv2 -> v3:
- move creation ofinclude/hw/arm/smmuv3.hto this patch to fix compil issue
- compilation allowed
- fix sbus allocation insmmu_init_pci_iommu
- restructure code into headers
- misc cleanupsQuoted 10 lines
--- hw/arm/Makefile.objs | 2 +- hw/arm/smmuv3-internal.h | 603 ++++++++++++++++++++++++ hw/arm/smmuv3.c | 1134 ++++++++++++++++++++++++++++++++++++++++++++++ hw/arm/trace-events | 32 ++ include/hw/arm/smmuv3.h | 87 ++++ 5 files changed, 1857 insertions(+), 1 deletion(-) create mode 100644 hw/arm/smmuv3-internal.h create mode 100644 hw/arm/smmuv3.c create mode 100644 include/hw/arm/smmuv3.h
Quoted 2 lines
This is a bit of a big patch for review for my taste -- are there some easy splits into multiple patches possible?
Yes I fully understand. I will try to split it to ease the review. Maybe I first need to reach a decent level of functionality first and then split it.
Quoted 13 lines
include/hw/arm/smmuv3.h line 37
Quoted 12 lines
include/hw/arm/smmuv3.h new file +11
-0+typedef struct SMMUQueue { + hwaddr base; + uint32_t prod; + uint32_t cons; + union { + struct { + uint8_t prod:1; + uint8_t cons:1; + }; + uint8_t unused; + } wrap;
Quoted 2 lines
Use of bitfields here seems a bit odd but I haven't looked at the code really.
OK. I will further look at this part.
Thanks
Eric
Quoted 4 lines
thanks
-- PMM
RFC v4 3/5 hw/arm/virt: Add SMMUv3 to the virt board
In reply to Eric AugerFrom: Prem Mallappa <[email protected]>
Add code to instantiate an smmu-v3 in mach-virt. A new boolean flag is introduced in VirtMachineState to allow this instantiation. It is currently false.
---
v2 -> v3:
- vbi was removed. Use vms instead
- migrate to new smmu binding format (iommu-map)
- don't use appendprop anymore
- add vms->smmu and guard instantiation with this latter
- interrupts type changed to edge
--- hw/arm/virt.c | 58 +++++++++++++++++++++++++++++++++++++++++++++++++++ include/hw/arm/virt.h | 4 ++++ 2 files changed, 62 insertions(+)
hw/arm/virt.c +58 -0
@@ -56,6 +56,7 @@#include "hw/smbios/smbios.h" #include "qapi/visitor.h" #include "standard-headers/linux/input.h" +#include "hw/arm/smmuv3.h" #define DEFINE_VIRT_MACHINE_LATEST (major, minor, latest) \ static void virt_##major##_##minor##_class_init(ObjectClass *oc, \
@@ -139,6 +140,7 @@ static const MemMapEntry a15memmap[] = { [VIRT_FW_CFG] = { 0x09020000, 0x00000018 }, [VIRT_GPIO] = { 0x09030000, 0x00001000 }, [VIRT_SECURE_UART] = { 0x09040000, 0x00001000 }, + [VIRT_SMMU] = { 0x09050000, 0x00020000 }, /* 128K, needed */ [VIRT_MMIO] = { 0x0a000000, 0x00000200 }, /* ...repeating for a total of NUM_VIRTIO_TRANSPORTS, each of that size */ [VIRT_PLATFORM_BUS] = { 0x0c000000, 0x02000000 },
@@ -159,6 +161,7 @@ static const int a15irqmap[] = { [VIRT_SECURE_UART] = 8, [VIRT_MMIO] = 16, /* ...to 16 + NUM_VIRTIO_TRANSPORTS - 1 */ [VIRT_GIC_V2M] = 48, /* ...to 48 + NUM_GICV2M_SPIS - 1 */ + [VIRT_SMMU] = 74, /* ...to 74 + NUM_SMMU_IRQS - 1 */ [VIRT_PLATFORM_BUS] = 112, /* ...to 112 + PLATFORM_BUS_NUM_IRQS -1 */ };
@@ -969,6 +972,52 @@ static void create_pcie_irq_map(const VirtMachineState *vms, 0x7 /* PCI irq */); } +static void alloc_smmu_phandle (VirtMachineState *vms) +{ +if (vms->smmu && !vms->smmu_phandle) { + vms->smmu_phandle =qemu_fdt_alloc_phandle (vms->fdt); + } +} + +static void create_smmu (const VirtMachineState *vms,qemu_irq *pic) +{ +char *smmu; +const char compat[] ="arm,smmu-v3" ; +int irq = vms->irqmap[VIRT_SMMU ]; +hwaddr base = vms->memmap[VIRT_SMMU ].base; +hwaddr size = vms->memmap[VIRT_SMMU ].size; +const char irq_names[] ="eventq\0priq\0cmdq-sync\0gerror" ; + +if (!vms->smmu) { +return ; + } + +sysbus_create_varargs ("smmuv3" , base, pic[irq], pic[irq +1 ], + pic[irq +2 ], pic[irq +3 ],NULL ); + + smmu =g_strdup_printf ("/smmuv3@%" PRIx64, base); +qemu_fdt_add_subnode (vms->fdt, smmu); +qemu_fdt_setprop (vms->fdt, smmu,"compatible" , compat,sizeof (compat)); +qemu_fdt_setprop_sized_cells (vms->fdt, smmu,"reg" ,2 , base,2 , size); + +qemu_fdt_setprop_cells (vms->fdt, smmu,"interrupts" , +GIC_FDT_IRQ_TYPE_SPI , irq ,GIC_FDT_IRQ_FLAGS_EDGE_LO_HI , +GIC_FDT_IRQ_TYPE_SPI , irq +1 ,GIC_FDT_IRQ_FLAGS_EDGE_LO_HI , +GIC_FDT_IRQ_TYPE_SPI , irq +2 ,GIC_FDT_IRQ_FLAGS_EDGE_LO_HI , +GIC_FDT_IRQ_TYPE_SPI , irq +3 ,GIC_FDT_IRQ_FLAGS_EDGE_LO_HI ); + +qemu_fdt_setprop (vms->fdt, smmu,"interrupt-names" , irq_names, +sizeof (irq_names)); + +qemu_fdt_setprop_cell (vms->fdt, smmu,"clocks" , vms->clock_phandle); +qemu_fdt_setprop_string (vms->fdt, smmu,"clock-names" ,"apb_pclk" ); + +qemu_fdt_setprop_cell (vms->fdt, smmu,"#iommu-cells" ,1 ); + +qemu_fdt_setprop_cell (vms->fdt, smmu,"phandle" , vms->smmu_phandle); +g_free (smmu); +} +static void create_pcie (const VirtMachineState *vms,qemu_irq *pic) {hwaddr base_mmio = vms->memmap[VIRT_PCIE_MMIO ].base;
@@ -1081,6 +1130,11 @@ static void create_pcie(const VirtMachineState *vms, qemu_irq *pic)qemu_fdt_setprop_cell (vms->fdt, nodename,"#interrupt-cells" ,1 );create_pcie_irq_map (vms, vms->gic_phandle, irq, nodename); +if (vms->smmu) { +qemu_fdt_setprop_cells (vms->fdt, nodename,"iommu-map" , +0x0 , vms->smmu_phandle,0x0 ,0x10000 ); + } +g_free (nodename); }
@@ -1402,8 +1456,12 @@ static void machvirt_init(MachineState *machine)create_rtc (vms, pic); +alloc_smmu_phandle (vms); +create_pcie (vms, pic); +create_smmu (vms, pic); +create_gpio (vms, pic);/* Create mmio transports, so the user can create virtio backends
include/hw/arm/virt.h +4 -0
@@ -38,6 +38,7 @@#define NUM_GICV2M_SPIS 64#define NUM_VIRTIO_TRANSPORTS 32 +#define NUM_SMMU_IRQS 4#define ARCH_GICV3_MAINT_IRQ 9
@@ -59,6 +60,7 @@ enum {VIRT_GIC_V2M ,VIRT_GIC_ITS ,VIRT_GIC_REDIST , +VIRT_SMMU ,VIRT_UART ,VIRT_MMIO , VIRT_RTC,
@@ -95,6 +97,7 @@ typedef struct {bool highmem;bool its;bool virt; +bool smmu;int32_t gic_version;struct arm_boot_info bootinfo;const MemMapEntry *memmap;
@@ -105,6 +108,7 @@ typedef struct {uint32_t clock_phandle;uint32_t gic_phandle;uint32_t msi_phandle; +uint32_t smmu_phandle;int psci_conduit; } VirtMachineState;
Signature · 1 line
2.5.5
RFC v4 4/5 hw/arm/virt: Add 2.10 machine type
In reply to Eric AugerThe new machine type allows smmuv3 instantiation. A new option is introduced to turn the feature on/off (off by default).
---
Another alternative would be to use the -device option as done on x86. As the smmu is a sysbus device, we would need to use the platform bus framework. This would work fine for the dt generation. However the feasibility needs to be studied for ACPI table generation.
a Veuillez saisir le message de validation pour vos modifications. Les lignes
--- hw/arm/virt.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++-- include/hw/arm/virt.h | 1 + 2 files changed, 50 insertions(+), 2 deletions(-)
hw/arm/virt.c +49 -2
@@ -1496,6 +1496,20 @@ static void machvirt_init(MachineState *machine)create_platform_bus (vms, pic); } +static bool virt_get_smmu (Object *obj,Error **errp) +{ +VirtMachineState *vms =VIRT_MACHINE (obj); + +return vms->smmu; +} + +static void virt_set_smmu (Object *obj,bool value,Error **errp) +{ +VirtMachineState *vms =VIRT_MACHINE (obj); + + vms->smmu = value; +} +static bool virt_get_secure (Object *obj,Error **errp) {VirtMachineState *vms =VIRT_MACHINE (obj);
@@ -1609,7 +1623,7 @@ static void machvirt_machine_init(void) }type_init (machvirt_machine_init); -static void virt_2_9_instance_init (Object *obj) +static void virt_2_10_instance_init (Object *obj) {VirtMachineState *vms =VIRT_MACHINE (obj);VirtMachineClass *vmc =VIRT_MACHINE_GET_CLASS (vms);
@@ -1665,14 +1679,47 @@ static void virt_2_9_instance_init(Object *obj) NULL); } +if (vmc->no_smmu) { + vms->smmu = false; + }else { +/* Default disallows smmu instantiation */ + vms->smmu = false; +object_property_add_bool (obj,"smmu" , virt_get_smmu, + virt_set_smmu,NULL ); +object_property_set_description (obj,"smmu" , +"Set on/off to enable/disable " +"smmu instantiation (default off)" , +NULL ); + } + vms->memmap = a15memmap; vms->irqmap = a15irqmap; } +static void virt_machine_2_10_options (MachineClass *mc) +{ +} +DEFINE_VIRT_MACHINE_AS_LATEST(2, 10) + +#define VIRT_COMPAT_2_9 \ + HW_COMPAT_2_9 + +static void virt_2_9_instance_init (Object *obj) +{ +virt_2_10_instance_init (obj); +} +static void virt_machine_2_9_options (MachineClass *mc) { +VirtMachineClass *vmc =VIRT_MACHINE_CLASS (OBJECT_CLASS (mc)); + +virt_machine_2_10_options (mc); +SET_MACHINE_COMPAT (mc,VIRT_COMPAT_2_9 ); + + vmc->no_smmu = true; } -DEFINE_VIRT_MACHINE_AS_LATEST(2, 9) +DEFINE_VIRT_MACHINE(2, 9) +#define VIRT_COMPAT_2_8 \ HW_COMPAT_2_8
include/hw/arm/virt.h +1 -0
@@ -86,6 +86,7 @@ typedef struct {bool disallow_affinity_adjustment;bool no_its;bool no_pmu; +bool no_smmu;bool claim_edge_triggered_timers; } VirtMachineClass;
Signature · 1 line
2.5.5
Re: [Qemu-arm] [RFC v4 4/5] hw/arm/virt: Add 2.10 machine type
Quoted 2 lines · Eric Auger
The new machine type allows smmuv3 instantiation. A new option is introduced to turn the feature on/off (off by default).
Should we go for default-on, or would that break guests? For other things added to the virt board I think the approach we've taken has been:
* if this is just an extra device, simply provide it in the new virt-n.nn machine by default
* if this is something that changes how the machine behaves even for code that doesn't care about that feature (eg EL2, EL3, since they change what mode you start in on boot) then default it to off
* if the feature only works with TCG and not KVM then maybe default it to off
thanks
-- PMM
Re: [Qemu-arm] [RFC v4 4/5] hw/arm/virt: Add 2.10 machine type
Hi Peter,
Quoted 4 lines · Peter Maydell
Quoted 2 lines · Eric Auger
The new machine type allows smmuv3 instantiation. A new option is introduced to turn the feature on/off (off by default).
Quoted 11 lines
Should we go for default-on, or would that break guests? For other things added to the virt board I think the approach we've taken has been:
* if this is just an extra device, simply provide it in the new virt-n.nn machine by default
* if this is something that changes how the machine behaves even for code that doesn't care about that feature (eg EL2, EL3, since they change what mode you start in on boot) then default it to off
* if the feature only works with TCG and not KVM then maybe default it to off
OK thanks for the guidelines. I put it off by default since it can largely degrade the performance in some use cases. Also instantiating the smmu can also induce some option changes at other levels: for instance virtio-pci devices may be used along with iommu_platform option for the guest to use dma ops.
Thanks
Eric
Quoted 4 lines
thanks
-- PMM
RFC v4 5/5 hw/arm/virt-acpi-build: add smmuv3 node in IORT table
In reply to Eric AugerFrom: Prem Mallappa <[email protected]>
This patch builds the smmuv3 node in the ACPI IORT table.
The RID space of the root complex, which spans 0x0-0x10000 maps to streamid space 0x0-0x10000 in smmuv3, which in turn maps to deviceid space 0x0-0x10000 in the ITS group.
The guest must feature the IOMMU probe deferral series
(https://lkml.org/lkml/2017/4/10/214) wich fixes streamid multiple lookup. This bug is not related to the SMMU emulation.
---
v2 -> v3:
- integrate into the existing IORT table made up of ITS, RC nodes
- take into account vms->smmu
- match linux actbl2.h acpi_iort_smmu_v3 field names
--- hw/arm/virt-acpi-build.c | 56 +++++++++++++++++++++++++++++++++++++++------ include/hw/acpi/acpi-defs.h | 15 ++++++++++++ 2 files changed, 64 insertions(+), 7 deletions(-)
hw/arm/virt-acpi-build.c +49 -7
@@ -393,19 +393,26 @@ build_rsdp(GArray *rsdp_table, BIOSLinker *linker, unsigned rsdt_tbl_offset) }static void -build_iort (GArray *table_data,BIOSLinker *linker) +build_iort (GArray *table_data,BIOSLinker *linker,VirtMachineState *vms) { -int iort_start = table_data->len; +int nb_nodes, iort_start = table_data->len;AcpiIortIdMapping *idmap;AcpiIortItsGroup *its;AcpiIortTable *iort; -size_t node_size, iort_length; +AcpiIortSmmu3 *smmu; +size_t node_size, iort_length, smmu_offset =0 ;AcpiIortRC *rc; iort =acpi_data_push (table_data,sizeof (*iort)); +if (vms->smmu) { + nb_nodes =3 ;/* RC, ITS, SMMUv3 */ + }else { + nb_nodes =2 ;/* RC, ITS */ + } + iort_length =sizeof (*iort); - iort->node_count =cpu_to_le32 (2 );/* RC and ITS nodes */ + iort->node_count =cpu_to_le32 (nb_nodes); iort->node_offset =cpu_to_le32 (sizeof (*iort));/* ITS group node */
@@ -418,6 +425,35 @@ build_iort(GArray *table_data, BIOSLinker *linker) its->its_count =cpu_to_le32 (1 ); its->identifiers[0 ] =0 ;/* MADT translation_id */ +if (vms->smmu) { +int irq = vms->irqmap[VIRT_SMMU ]; + +/* SMMUv3 node */ + smmu_offset =cpu_to_le32 (iort->node_offset + node_size); + node_size =sizeof (*smmu) +sizeof (*idmap); + iort_length += node_size; + smmu =acpi_data_push (table_data, node_size); + + + smmu->type =ACPI_IORT_NODE_SMMU_V3 ; + smmu->length =cpu_to_le16 (node_size); + smmu->mapping_count =cpu_to_le32 (1 ); + smmu->mapping_offset =cpu_to_le32 (sizeof (*smmu)); + smmu->base_address =cpu_to_le64 (vms->memmap[VIRT_SMMU ].base); + smmu->event_gsiv =cpu_to_le32 (irq); + smmu->pri_gsiv =cpu_to_le32 (irq +1 ); + smmu->gerr_gsiv =cpu_to_le32 (irq +2 ); + smmu->sync_gsiv =cpu_to_le32 (irq +3 ); + +/* Identity RID mapping covering the whole input RID range */ + idmap = &smmu->id_mapping_array[0 ]; + idmap->input_base =0 ; + idmap->id_count =cpu_to_le32 (0xFFFF ); + idmap->output_base =0 ; +/* output IORT node is the ITS group node (the first node) */ + idmap->output_reference =cpu_to_le32 (iort->node_offset); + } +/* Root Complex Node */ node_size =sizeof (*rc) +sizeof (*idmap); iort_length += node_size;
@@ -438,8 +474,14 @@ build_iort(GArray *table_data, BIOSLinker *linker) idmap->input_base =0 ; idmap->id_count =cpu_to_le32 (0xFFFF ); idmap->output_base =0 ; -/* output IORT node is the ITS group node (the first node) */ - idmap->output_reference =cpu_to_le32 (iort->node_offset); + +if (vms->smmu) { +/* output IORT node is the smmuv3 node */ + idmap->output_reference =cpu_to_le32 (smmu_offset); + }else { +/* output IORT node is the ITS group node (the first node) */ + idmap->output_reference =cpu_to_le32 (iort->node_offset); + } iort->length =cpu_to_le32 (iort_length);
@@ -785,7 +827,7 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)if (its_class_name () && !vmc->no_its) {acpi_add_table (table_offsets, tables_blob); -build_iort (tables_blob, tables->linker); +build_iort (tables_blob, tables->linker, vms); }/* RSDT is pointed to by RSDP */
include/hw/acpi/acpi-defs.h +15 -0
@@ -696,6 +696,21 @@ struct AcpiIortItsGroup { }QEMU_PACKED ;typedef struct AcpiIortItsGroup AcpiIortItsGroup ; +struct AcpiIortSmmu3 { +ACPI_IORT_NODE_HEADER_DEF + uint64_t base_address; +uint32_t flags; +uint32_t reserved2; +uint64_t vatos_address; +uint32_t model; +uint32_t event_gsiv; +uint32_t pri_gsiv; +uint32_t gerr_gsiv; +uint32_t sync_gsiv; +AcpiIortIdMapping id_mapping_array[0 ]; +}QEMU_PACKED ; +typedef struct AcpiIortSmmu3 AcpiIortSmmu3 ; +struct AcpiIortRC {ACPI_IORT_NODE_HEADER_DEF AcpiIortMemoryAccess memory_properties;
Signature · 1 line
2.5.5
Quoted 5 lines · Eric Auger
This series introduces the emulation code for ARM SMMUv3. This is the continuation of Prem's work [1].
This v4 is yet another visibility step as many restrictions apply to the model at the moment:
I had a quick scan through to see if anything leapt out as odd, and have sent a few comments, but I'm assuming you don't want in-depth review at this time (and hoping that somebody else will provide it when you do want it ;-)).
thanks
-- PMM
Hi Peter,
Quoted 11 lines · Peter Maydell
Quoted 5 lines · Eric Auger
This series introduces the emulation code for ARM SMMUv3. This is the continuation of Prem's work [1].
This v4 is yet another visibility step as many restrictions apply to the model at the moment:
I had a quick scan through to see if anything leapt out as odd, and have sent a few comments, but I'm assuming you don't want in-depth review at this time (and hoping that somebody else will provide it when you do want it ;-)).
Thank you for this quick scan. Indeed this is what I expect at the moment since the model clearly misses maturity and features. I will ping people for proper review when this makes sense.
At the moment people can start testing it for stage 1 only. I was reported and reproduced boot problems with a guest using virtio-blk-pci. This is currently under debug and then my next step is to complete the VFIO integration.
Thanks
Eric
Quoted 4 lines
thanks
-- PMM