iommu/rockchip: implement .flush_iotlb_all
Subsystems (2)
PATCH 1/1 iommu/rockchip: implement .flush_iotlb_all
rk_iommu invalidates the IOTLB synchronously in .unmap (via rk_iommu_zap_iova), so it stays consistent without a deferred .iotlb_sync and does not advertise IOMMU_CAP_DEFERRED_FLUSH.
It never implemented the optional .flush_iotlb_all op, though, and iommu_flush_iotlb_all() is a no-op when that op is absent -- so on Rockchip a driver that owns an rk_iommu domain and calls it to flush the whole domain's TLB (e.g. before reusing a mapping, without unmapping it) gets nothing, even though the hardware can do it: rk_iommu already issues ZAP_CACHE for range shootdowns.
Implement it with that same primitive: ZAP_CACHE every bank of every IOMMU on the domain, under the runtime-PM + clk guard already used by rk_iommu_zap_iova().
Found while bringing up an out-of-tree RK3576 NPU (accel/rocket) driver that flushes before each submit; on Rockchip that flush did nothing.
---
drivers/iommu/rockchip-iommu.c +25 -0
@@ -1202,6 +1202,30 @@return 0 ; } +/* shootdown the entire tlb on all iommus using this domain */ +static void rk_iommu_flush_iotlb_all (struct iommu_domain *domain) +{ +struct rk_iommu_domain *rk_domain =to_rk_domain (domain); +struct list_head *pos; +unsigned long flags; + +spin_lock_irqsave (&rk_domain->iommus_lock, flags); +list_for_each (pos, &rk_domain->iommus) { + struct rk_iommu *iommu = list_entry(pos, struct rk_iommu, node); +int ret =pm_runtime_get_if_in_use (iommu->dev); + +if (WARN_ON_ONCE (ret <0 )) +continue ; +if (ret) { +WARN_ON (clk_bulk_enable (iommu->num_clocks, iommu->clocks)); +rk_iommu_command (iommu,RK_MMU_CMD_ZAP_CACHE ); +clk_bulk_disable (iommu->num_clocks, iommu->clocks); +pm_runtime_put (iommu->dev); + } + } +spin_unlock_irqrestore (&rk_domain->iommus_lock, flags); +} +static const struct iommu_ops rk_iommu_ops = { .identity_domain = &rk_identity_domain, .domain_alloc_paging = rk_iommu_domain_alloc_paging,
@@ -1214,6 +1245,7 @@ .map_pages = rk_iommu_map, .unmap_pages = rk_iommu_unmap, .iova_to_phys = rk_iommu_iova_to_phys, + .flush_iotlb_all = rk_iommu_flush_iotlb_all, .free = rk_iommu_domain_free, } };
Will Deacon
maintainerQuoted 15 lines · Jiaxing Hu
rk_iommuinvalidates the IOTLB synchronously in .unmap (viark_iommu_zap_iova), so it stays consistent without a deferred .iotlb_sync and does not advertiseIOMMU_CAP_DEFERRED_FLUSH.It never implemented the optional .flush_iotlb_all op, though, and
iommu_flush_iotlb_all()is a no-op when that op is absent -- so on Rockchip a driver that owns anrk_iommudomain and calls it to flush the whole domain's TLB (e.g. before reusing a mapping, without unmapping it) gets nothing, even though the hardware can do it:rk_iommualready issuesZAP_CACHEfor range shootdowns.Implement it with that same primitive:
ZAP_CACHEevery bank of every IOMMU on the domain, under the runtime-PM + clk guard already used byrk_iommu_zap_iova().
Quoted 5 lines
Found while bringing up an out-of-tree RK3576 NPU (accel/rocket) driver that flushes before each submit; on Rockchip that flush did nothing.
Signed-off-by Jiaxing Hu <[email protected]>---
We tried to get rid of iommu_flush_iotlb_all() in 69e5a17511f6 ("iommu: Remove useless flush from iommu_create_device_direct_mappings()") but we had to revert that in a33bf8d8ce7e ("iommu: Restore iommu_flush_iotlb_all()") because of a user in the MSM driver.
So I think this is the wrong direction. We should try harder to remove the function rather than add new implementations of it. I'm assuming you're only adding it for some out-of-tree code that calls it?
Will
Robin Murphy
reviewerQuoted 19 lines · Will Deacon
Quoted 18 lines · Jiaxing Hu
rk_iommuinvalidates the IOTLB synchronously in .unmap (viark_iommu_zap_iova), so it stays consistent without a deferred .iotlb_sync and does not advertiseIOMMU_CAP_DEFERRED_FLUSH.It never implemented the optional .flush_iotlb_all op, though, and
iommu_flush_iotlb_all()is a no-op when that op is absent -- so on Rockchip a driver that owns anrk_iommudomain and calls it to flush the whole domain's TLB (e.g. before reusing a mapping, without unmapping it) gets nothing, even though the hardware can do it:rk_iommualready issuesZAP_CACHEfor range shootdowns.Implement it with that same primitive:
ZAP_CACHEevery bank of every IOMMU on the domain, under the runtime-PM + clk guard already used byrk_iommu_zap_iova().Found while bringing up an out-of-tree RK3576 NPU (accel/rocket) driver that flushes before each submit; on Rockchip that flush did nothing.
Quoted 7 lines
Quoted 2 lines
Signed-off-by Jiaxing Hu <[email protected]>---
We tried to get rid of
iommu_flush_iotlb_all()in69e5a17511f6("iommu: Remove useless flush fromiommu_create_device_direct_mappings()") but we had to revert that ina33bf8d8ce7e("iommu: Restoreiommu_flush_iotlb_all()") because of a user in the MSM driver.
We wanted rid of the public API (indeed partly due to the prospect of client drivers abusing it), but the actual ->flush_tlb_all driver op remains crucial to the flush queue mechanism, so there's no harm in drivers implementing it *if* they also implement the relevant optimisations to avoid synchronous flushing in unmap_pages to make CAP_DEFERRED_FLUSH worthwhile.
Thanks,
Robin.
Quoted 5 lines
So I think this is the wrong direction. We should try harder to remove the function rather than add new implementations of it. I'm assuming you're only adding it for some out-of-tree code that calls it?
Will
Quoted 3 lines · Will Deacon
So I think this is the wrong direction. We should try harder to remove the function rather than add new implementations of it. I'm assuming you're only adding it for some out-of-tree code that calls it?
Yes, that's right. The only caller is the out-of-tree RK3576 NPU (accel/rocket) bring-up I mentioned in the commit message. There's no in-tree rk_iommu user, since the in-tree consumers all invalidate through .unmap and none of them call iommu_flush_iotlb_all().
So if the plan is to remove the function rather than add new implementations of it, then I've got this backwards. Please drop the patch. If the NPU driver ever goes upstream I'll make it stop relying on iommu_flush_iotlb_all() rather than bring this one back.
Thanks for taking a look, and for the pointers to those two commits.
Jiaxing
Heiko Stübner
maintainerHi,
Am Freitag, 10. Juli 2026, 13:15:08 Mitteleuropäische Sommerzeit schrieb Jiaxing Hu:
Quoted 9 lines · Will Deacon
Quoted 3 lines
So I think this is the wrong direction. We should try harder to remove the function rather than add new implementations of it. I'm assuming you're only adding it for some out-of-tree code that calls it?
Yes, that's right. The only caller is the out-of-tree RK3576 NPU (accel/rocket) bring-up I mentioned in the commit message. There's no in-tree
rk_iommuuser, since the in-tree consumers all invalidate through .unmap and none of them calliommu_flush_iotlb_all().
accel/rocket is in mainline, so I guess the interesting question is, why are you expecting your changes to be (stay?) out of tree?
When you show your changes - even as a RFC - on the lists you might get helpful comments - like the ones here for the iommu change. That might also prevent you from going too far in a suboptimal direction and thus could save time.
Additionally, you would show you're working on that, so that could prevent someone else starting a duplicate of your work - or at least pool resources.
There are a bunch of people working on the RK3576 for different projects so there is a pretty high chance of getting valuable feedback. (like what happened with this patch ;-) )
Heiko
Quoted 2 lines · Heiko Stübner
accel/rocket is in mainline, so I guess the interesting question is, why are you expecting your changes to be (stay?) out of tree?
Thanks. I had been treating it as exploratory: single-layer inference runs on the open stack (rocket + Teflon) on RK3576, but chained inference does not, and I narrowed the failure to below the software surface (fixed-function hardware I cannot reach from an open board), so it felt too unfinished to send.
But you are right that an RFC is the better way to handle it. I will clean up the RK3576 bring-up (power domains, clocks, IOMMU, DPU/RDMA, DTS) into an RFC against current mainline and post it, with the chained-task limitation described honestly. If you can point me at who else is working on the RK3576 NPU, I would be glad to compare notes.
Thanks,
Jiaxing
Quoted 3 lines · Robin Murphy
there's no harm in drivers implementing it *if* they also implement the relevant optimisations to avoid synchronous flushing in
unmap_pagesto makeCAP_DEFERRED_FLUSHworthwhile.
Understood, thanks. I will drop the standalone patch. If I come back to rk_iommu it will be the full version: move invalidation out of unmap_pages into an iotlb_sync/gather path and advertise CAP_DEFERRED_FLUSH with numbers, rather than flush_iotlb_all on its own.
Thanks,
Jiaxing