Skip to content

iommu/riscv: Add hardware dirty tracking for second-stage domains

[email protected] kvm-riscv linux-iommu linux-riscv lkml 7 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v7 Latest10 patches 9 repliesIn review
  2. v610 patches 3 repliesSuperseded
  3. v5 RFC10 patches 13 repliesSuperseded
Earlier versions · 4
VersionPostedPatchesRepliesStatus
v4 RFC10 patches15 repliesSuperseded
v3 RFC9 of 10 patches4 repliesSuperseded
v2 RFC10 patches6 repliesSuperseded
v1 RFC11 patches18 repliesSuperseded

Changes in v7 from the cover letter

- Rebased on top of v7.4-rc1 (6628e7d4c6b2).
- Return -EINVAL instead of -ENODEV when the IOMMU does not support the iohgatp mode of a second-stage domain, following the attach_dev errno semantics and allowing callers such as QEMU to try another domain or allocate a new compatible one.
- Assign dirty_ops only when IOMMU_HWPT_ALLOC_DIRTY_TRACKING is requested, and reject attaching a dirty-tracking domain to an IOMMU without AMO_HWAD with -EINVAL. (Per Gong Shuai)
- Link to v6:
https://lore.kernel.org/linux-iommu/[email protected]/

Patches in v7 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/10iommu/generic_pt: Fix the RISC-V supported feature maskdrivers/iommu/generic_pt/fmt/iommu_riscv64.c+2 -10 replies
2/10iommupt: Add RISC-V Second-stage (iohgatp) page table supportdrivers/iommu/generic_pt/fmt/iommu_riscv64.c drivers/iommu/generic_pt/fmt/riscv.h include/linux/generic_pt/common.h +1 more+102 -240 replies
3/10iommupt: Add RISC-V dirty tracking PTE opsdrivers/iommu/generic_pt/fmt/riscv.h+49 -00 replies
4/10iommu/riscv: use data structure instead of individual valuesdrivers/iommu/riscv/iommu.c+17 -100 replies
5/10iommu/riscv: support GSCID and GVMA invalidation commanddrivers/iommu/riscv/iommu-bits.h drivers/iommu/riscv/iommu.c+58 -112 replies
6/10RISC-V: KVM: Enable KVM_VFIO interfaces on RISC-V archarch/riscv/kvm/Kconfig+2 -02 replies
7/10iommu/riscv: Add domain_alloc_paging_flags for second-stage domaindrivers/iommu/riscv/iommu.c+84 -230 replies
8/10iommu/riscv: Pre-enable GADE for second-stage domainsdrivers/iommu/riscv/iommu.c+4 -00 replies
9/10iommu/riscv: Add dirty tracking support for second-stage domainsdrivers/iommu/riscv/iommu.c+30 -22 replies
10/10iommu/riscv: Add IOTINVAL.GVMA invalidationsdrivers/iommu/riscv/iommu.c+15 -30 replies

Reviews trailers per patch

9 of 10 patches have a Reviewed-by or Acked-by.

PatchFangyu YuNutty LiuJason Gunthorpe
1/10T-b R-b R-b
2/10R-b
3/10R-b
4/10R-b R-b
5/10R-b
6/10
7/10R-b
8/10R-b
9/10R-b
10/10R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.

Read the v7 thread

Cover letter [PATCH v7 00/10] iommu/riscv: Add hardware dirty tracking for second-stage domains

From: Fangyu Yu <[email protected]>

The RISC-V IOMMU architecture defines an AMO_HWAD capability (Hardware Access/Dirty update) that allows the IOMMU to atomically set the A/D bits in second-stage PTEs on DMA access. When DC.tc.GADE is asserted, the IOMMU autonomously sets D on the first write to a page mapped by an iohgatp domain. This series wires that capability up to the iommufd dirty-tracking interface (IOMMU_HWPT_SET_DIRTY_TRACKING / IOMMU_HWPT_GET_DIRTY_BITMAP) and reports IOMMU_CAP_DIRTY_TRACKING.

Design notes
============

* The feature is scoped to second-stage (iohgatp) domains only; these are the domains created for KVM / VFIO device pass-through when userspace allocates an HWPT with IOMMU_HWPT_ALLOC_NEST_PARENT or IOMMU_HWPT_ALLOC_DIRTY_TRACKING. First-stage (iosatp) domains are not touched by this series.

* The page-table side plugs into the existing generic_pt dirty hook framework (amdv1 / vtdss style). RISC-V adds the three required PTE ops - is_write_dirty / make_write_clean / make_write_dirty.

Testing
=======

* Test on QEMU RISC-V, a nvme and an e1000e device was passed through to an L2 guest via vfio-pci + iommufd.

* generic_pt KUnit: the existing test_dirty case now runs and passes for the RISC-V 64-bit format.

Follow-up work
==============
* Build a dedicated end-to-end test case that drives the full flow (HWPT_ALLOC with DIRTY_TRACKING -> attach -> IOAS_MAP -> generate real DMA -> SET_DIRTY_TRACKING -> GET_DIRTY_BITMAP -> verify bitmap against expected IOVA footprint) so that the behaviour can be regression-tested beyond the KUnit PTE-level coverage.

LLM-based coding assistants were used for code exploration, patch review, test execution, and drafting/editing messages and this cover letter. I reviewed and finalized all resulting code and text.

This series functionally depends on [1], which has been posted but is not yet in mainline.

---
Changes in v7:
- Rebased on top of v7.4-rc1 (6628e7d4c6b2).
- Return -EINVAL instead of -ENODEV when the IOMMU does not support the iohgatp mode of a second-stage domain, following the attach_dev errno semantics and allowing callers such as QEMU to try another domain or allocate a new compatible one.
- Assign dirty_ops only when IOMMU_HWPT_ALLOC_DIRTY_TRACKING is requested, and reject attaching a dirty-tracking domain to an IOMMU without AMO_HWAD with -EINVAL. (Per Gong Shuai)
- Link to v6:
https://lore.kernel.org/linux-iommu/[email protected]/
Changes in v6:
- Drop the RFC tag.
- Rebased on top of mainline (390febb5805d).
- Carry the unmerged prerequisite [2] as patch 1.
- Drop "iommu/riscv: report iommu capabilities"; mainline commit 3069e8f3e1e9 ("iommu/riscv: Report cache coherency capability") already adds riscv_iommu_capable().
- domain_alloc_paging_flags: drop the redundant flags pre-check and supported_flags, and split the per-stage setup into riscv_iommu_init_domain_s1() / riscv_iommu_init_domain_s2(). (Per Jason)
- Link to v5:
https://lore.kernel.org/linux-iommu/[email protected]/
Changes in v5:
- Propagate the GADE bit from the new device context into the hardware DC in riscv_iommu_iodir_update(), so the pre-enabled GADE actually reaches the hardware. (Per Gong Shuai)
- Issue a global IOTINVAL.GVMA in riscv_iommu_iodir_set_mode(), so that second-stage translations cached before a DDTP mode change (e.g. by a previous kernel after kexec) are flushed as the specification requires. (Per Tan Chi)
- Rebased on top of v7.3, with [1] and [2] applied.
- Link to v4:
https://lore.kernel.org/linux-riscv/[email protected]/
Changes in v4 (Jason's suggestions):
- Rebased the series on top of [1] and [2].
- Drop the RISC-V-specific pt_num_items_lg2() top-level special case.
- Validate page-table address widths by translation stage: accept only Sv39/Sv48/Sv57 widths for first-stage tables and only Sv39x4/Sv48x4/ Sv57x4 widths for second-stage tables.
- Replace the aliased first-stage/second-stage MODE union with independent FSC/IOSATP and IOHGATP MODE fields.
- Add pt_dirty_supported() to restrict RISC-V generic_pt dirty tracking to second-stage tables.
- Link to v3:
https://lore.kernel.org/linux-riscv/[email protected]/
Changes in v3 (Andrew Jones's suggestions):
- Rebased the series on top of Andrew Jones' generic_pt RISC-V supported feature mask fix, which adds PT_FEAT_RISCV_SVPBMT to the supported feature set.
- Added PT_FEAT_RISCV_S2 to the RISC-V generic_pt supported feature mask and KUnit feature matrix.
- Kept PT_FEAT_SIGN_EXTEND only for first-stage RISC-V KUnit configs; second stage configs now use PT_FEAT_RISCV_S2 without sign-extension semantics.
- Split RISC-V IOMMU capability checks into first-stage FSC and second-stage IOHGATP helpers.
- Updated second-stage paging domain setup to use GPA widths 41/50/59 and to avoid PT_FEAT_SIGN_EXTEND.
- Link to v2:
https://lore.kernel.org/linux-riscv/[email protected]/
Changes in v2 (Jason's suggestions):
- Introduced a single PT_FEAT_RISCV_S2: second-stage selection is driven purely by this feature bit.
- Switched from dynamic DC.tc.GADE toggling to static pre-enable.
- domain_alloc_paging_flags: follow the switch/case design from other drivers.
- Drop IOMMU_CAP_DEFERRED_FLUSH in riscv_iommu_capable.
- Remove the .hw_info-related patch.
- Link to v1:
https://lore.kernel.org/linux-riscv/[email protected]/

[1] https://lore.kernel.org/linux-iommu/[email protected]/
[2] https://lore.kernel.org/linux-iommu/[email protected]/

Andrew Jones (1):
iommu/generic_pt: Fix the RISC-V supported feature mask

Fangyu Yu (6):
iommupt: Add RISC-V Second-stage (iohgatp) page table support
iommupt: Add RISC-V dirty tracking PTE ops
iommu/riscv: Add domain_alloc_paging_flags for second-stage domain iommu/riscv: Pre-enable GADE for second-stage domains iommu/riscv: Add dirty tracking support for second-stage domains iommu/riscv: Add IOTINVAL.GVMA invalidations

Tomasz Jeznach (1):
RISC-V: KVM: Enable KVM_VFIO interfaces on RISC-V arch

Zong Li (2):
iommu/riscv: use data structure instead of individual values iommu/riscv: support GSCID and GVMA invalidation command

 arch/riscv/kvm/Kconfig                       |   2 +
 drivers/iommu/generic_pt/fmt/iommu_riscv64.c |   3 +-
 drivers/iommu/generic_pt/fmt/riscv.h         | 158 ++++++++++--
 drivers/iommu/riscv/iommu-bits.h             |   7 +
 drivers/iommu/riscv/iommu.c                  | 244 +++++++++++++++----
 include/linux/generic_pt/common.h            |   4 +
 include/linux/generic_pt/iommu.h             |  11 +
 7 files changed, 359 insertions(+), 70 deletions(-)

base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4

Signature · 7 lines
2.50.1


-- 
kvm-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kvm-riscv