iommu/amd: Remove iommu_group_mutex_assert() from amd_iommu_clear_gcr3()
Subsystems (2)
Vasant Hegde
maintainerPATCH 1/1 iommu/amd: Remove iommu_group_mutex_assert() from amd_iommu_clear_gcr3()
amd_iommu_clear_gcr3() is called from sva_mn_release() during the mmu_notifier release path to detach a PASIDs when a process exits. This path does not hold the iommu group mutex, since it is invoked asynchronously from mm teardown context rather than through the normal IOMMU group/attach APIs.
As a result, the iommu_group_mutex_assert() check in amd_iommu_clear_gcr3() logs a kernel warning.
[62863.761300] ------------[ cut here ]------------ [62863.761320] WARNING: drivers/iommu/iommu.c:1361 at iommu_group_mutex_assert+0x3a/0x50, CPU#14: kworker/14:2/621 ... [62863.761706] <TASK> [62863.761712] amd_iommu_clear_gcr3+0x1e/0x50 [62863.761724] sva_mn_release+0x68/0xd0 [62863.761739] __mmu_notifier_release+0xb9/0x2d0 [62863.761763] exit_mmap+0x427/0x460 [62863.761836] __mmput+0x41/0x120 [62863.761844] mmput_async_fn+0x15/0x20 [62863.761852] process_one_work+0x22e/0x780 [62863.761876] worker_thread+0x1b5/0x380 [62863.761892] kthread+0x10d/0x150 [62863.761913] ret_from_fork+0x346/0x3c0 [62863.761930] ret_from_fork_asm+0x1a/0x30 [62863.761964] </TASK>
Remove the iommu_group_mutex_assert() call from amd_iommu_clear_gcr3(), since PASID teardown via the mmu_notifier release path is a legitimate caller that does not and need not hold the group mutex.
--- drivers/iommu/amd/iommu.c | 2 -- 1 file changed, 2 deletions(-)
drivers/iommu/amd/iommu.c +0 -2
@@ -2063,8 +2063,6 @@ int amd_iommu_clear_gcr3(struct iommu_dev_data *dev_data, ioasid_t pasid)struct gcr3_tbl_info *gcr3_info = &dev_data->gcr3_info;int ret; -iommu_group_mutex_assert (dev_data->dev); - ret =update_gcr3 (dev_data, pasid,0 , false);if (ret)return ret;
Signature · 1 line
2.31.1
Quoted 4 lines · Vasant Hegde
Remove the
iommu_group_mutex_assert()call fromamd_iommu_clear_gcr3(), since PASID teardown via themmu_notifierrelease path is a legitimate caller that does not and need not hold the group mutex.
Why doesn't it?
Right below this line is:
gcr3_info->pasid_cnt--;
Which certainly needs the lock.
In the ARM driver this async release notifier was pretty tricky to lock properly. It just updates the equivilent to the GCR3 entry to make it disabled and doesn't touch anything else. This is locked in a way so that concurrent domain detach doesn't face any races.
AMD should have the same basic design here too.
Jason
Vasant Hegde
maintainerHi Jason,
Quoted 7 lines · Jason Gunthorpe
Quoted 4 lines · Vasant Hegde
Remove the
iommu_group_mutex_assert()call fromamd_iommu_clear_gcr3(), since PASID teardown via themmu_notifierrelease path is a legitimate caller that does not and need not hold the group mutex.
Quoted 7 lines
Why doesn't it?
Right below this line is:
gcr3_info->pasid_cnt--;Which certainly needs the lock.
yeah. I overlooked it. Its part of dev_data and domain lock is not enough.
Quoted 5 lines
In the ARM driver this async release notifier was pretty tricky to lock properly. It just updates the equivilent to the GCR3 entry to make it disabled and doesn't touch anything else. This is locked in a way so that concurrent domain detach doesn't face any races.
How about sva_mn_release() path just cleared GCR[PASID]/flush cache and then iommu_sva_unbind_device() path (amd_iommu_remove_dev_pasid), decrements count and removes pdom_dev_data structure. something like below?
---<---
drivers/iommu/amd/amd_iommu.h +1 -0
@@ -77,6 +77,7 @@ void amd_iommu_iopf_remove_device(struct amd_iommu *iommu,int amd_iommu_set_gcr3 (struct iommu_dev_data *dev_data,ioasid_t pasid,unsigned long gcr3);int amd_iommu_clear_gcr3 (struct iommu_dev_data *dev_data,ioasid_t pasid); +int amd_iommu_disable_gcr3 (struct iommu_dev_data *dev_data,ioasid_t pasid);/* PPR */ int __init amd_iommu_alloc_ppr_log (struct amd_iommu *iommu);
drivers/iommu/amd/iommu.c +0 -0
@@ -2070,6 +2070,15 @@ int amd_iommu_set_gcr3(struct iommu_dev_data *dev_data,
ioasid_t pasid,
return ret;
}
+/*
+ * Disable a GCR3 entry without detaching its PASID from the IOMMU core.
+ * This is used from the SVA mmu notifier release callback.
+ */
+int amd_iommu_disable_gcr3(struct iommu_dev_data *dev_data, ioasid_t pasid) +{
+ return update_gcr3(dev_data, pasid, 0, false);
+}
+
int amd_iommu_clear_gcr3(struct iommu_dev_data *dev_data, ioasid_t pasid) {
struct gcr3_tbl_info *gcr3_info = &dev_data->gcr3_info;
@@ -2077,7 +2086,7 @@ int amd_iommu_clear_gcr3(struct iommu_dev_data *dev_data, ioasid_t pasid)
iommu_group_mutex_assert(dev_data->dev);
- ret = update_gcr3(dev_data, pasid, 0, false);
+ ret = amd_iommu_disable_gcr3(dev_data, pasid);
if (ret)
return ret;
drivers/iommu/amd/pasid.c +0 -0
@@ -79,7 +79,7 @@ static void sva_arch_invalidate_secondary_tlbs(struct
mmu_notifier *mn,
static void sva_mn_release(struct mmu_notifier *mn, struct mm_struct *mm) {
- struct pdom_dev_data *pdom_dev_data, *next;
+ struct pdom_dev_data *pdom_dev_data;
struct protection_domain *sva_pdom;
unsigned long flags;
@@ -87,9 +87,14 @@ static void sva_mn_release(struct mmu_notifier *mn, struct mm_struct *mm)
spin_lock_irqsave(&sva_pdom->lock, flags);
- /* Assume dev_data_list contains same PASID with different devices */
- for_each_pdom_dev_data_safe(pdom_dev_data, next, sva_pdom)
- remove_dev_pasid(pdom_dev_data);
+ /*
+ * Disable the GCR3 entry before the mm page tables are freed. Leave
+ * the PASID attachment and pdom_dev_data for the IOMMU core to
+ * remove through the normal detach path.
+ */
+ for_each_pdom_dev_data(pdom_dev_data, sva_pdom)
+ amd_iommu_disable_gcr3(pdom_dev_data->dev_data,
+ pdom_dev_data->pasid);
spin_unlock_irqrestore(&sva_pdom->lock, flags);
}
Quoted 22 lines · Vasant Hegde
Hi Jason,
Quoted 7 lines · Jason Gunthorpe
Quoted 4 lines · Vasant Hegde
Remove the
iommu_group_mutex_assert()call fromamd_iommu_clear_gcr3(), since PASID teardown via themmu_notifierrelease path is a legitimate caller that does not and need not hold the group mutex.Quoted 7 lines
Why doesn't it?
Right below this line is:
gcr3_info->pasid_cnt--;Which certainly needs the lock.
yeah. I overlooked it. Its part of
dev_dataand domain lock is not enough.Quoted 1 line
Quoted 8 lines
Quoted 4 lines
In the ARM driver this async release notifier was pretty tricky to lock properly. It just updates the equivilent to the GCR3 entry to make it disabled and doesn't touch anything else. This is locked in a way so that concurrent domain detach doesn't face any races.
How about
sva_mn_release()path just cleared GCR[PASID]/flush cache and theniommu_sva_unbind_device()path (amd_iommu_remove_dev_pasid), decrements count and removespdom_dev_datastructure. something like below?
One of the considerations here is the release of the SVA should not trigger errors from the iommu driver if the device continues to use it.
Every device access should continue to be captured as page fault and run the SVA page fault path. It will naturally end in an error.
Not sure this:
Review
Quoted 8 lines
(quoted diff) +8
-0+/* + * Disable a GCR3 entry without detaching its PASID from the IOMMU core. + * This is used from the SVA mmu notifier release callback. + */ +int amd_iommu_disable_gcr3(struct iommu_dev_data *dev_data, ioasid_t pasid) +{ + return update_gcr3(dev_data, pasid, 0, false); +}
Gets there? Or at least explain it in the comment?
Jason
Vasant Hegde
maintainerQuoted 28 lines · Jason Gunthorpe
Quoted 27 lines · Vasant Hegde
Hi Jason,
Quoted 7 lines · Jason Gunthorpe
Quoted 4 lines · Vasant Hegde
Remove the
iommu_group_mutex_assert()call fromamd_iommu_clear_gcr3(), since PASID teardown via themmu_notifierrelease path is a legitimate caller that does not and need not hold the group mutex.Quoted 7 lines
Why doesn't it?
Right below this line is:
gcr3_info->pasid_cnt--;Which certainly needs the lock.
yeah. I overlooked it. Its part of
dev_dataand domain lock is not enough.Quoted 5 lines
In the ARM driver this async release notifier was pretty tricky to lock properly. It just updates the equivilent to the GCR3 entry to make it disabled and doesn't touch anything else. This is locked in a way so that concurrent domain detach doesn't face any races.
Quoted 7 lines
Quoted 3 lines
How about
sva_mn_release()path just cleared GCR[PASID]/flush cache and theniommu_sva_unbind_device()path (amd_iommu_remove_dev_pasid), decrements count and removespdom_dev_datastructure. something like below?One of the considerations here is the release of the SVA should not trigger errors from the iommu driver if the device continues to use it.
I am clearing only the GCR3 table entry. It will block page table walk for this particular PASID. All inflight requests and new request for other PASIDs will continue to be processed.
Quoted 5 lines
Every device access should continue to be captured as page fault and run the SVA page fault path. It will naturally end in an error.
Quoted 1 line
Not sure this:
Quoted 10 lines
Review
Quoted 8 lines
(quoted diff) +8
-0+/* + * Disable a GCR3 entry without detaching its PASID from the IOMMU core. + * This is used from the SVA mmu notifier release callback. + */ +int amd_iommu_disable_gcr3(struct iommu_dev_data *dev_data, ioasid_t pasid) +{ + return update_gcr3(dev_data, pasid, 0, false); +}Gets there? Or at least explain it in the comment?
Sure. I will add a comment. I realized AMD SVA needs lot more fixes. I will send V2 along with few other fixes to handle device reset scenarios.
-Vasant
Vasant Hegde
maintainerQuoted 31 lines · Vasant Hegde
Quoted 28 lines · Jason Gunthorpe
Quoted 27 lines · Vasant Hegde
Hi Jason,
Quoted 7 lines · Jason Gunthorpe
Quoted 4 lines · Vasant Hegde
Remove the
iommu_group_mutex_assert()call fromamd_iommu_clear_gcr3(), since PASID teardown via themmu_notifierrelease path is a legitimate caller that does not and need not hold the group mutex.Quoted 7 lines
Why doesn't it?
Right below this line is:
gcr3_info->pasid_cnt--;Which certainly needs the lock.
yeah. I overlooked it. Its part of
dev_dataand domain lock is not enough.Quoted 5 lines
In the ARM driver this async release notifier was pretty tricky to lock properly. It just updates the equivilent to the GCR3 entry to make it disabled and doesn't touch anything else. This is locked in a way so that concurrent domain detach doesn't face any races.
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
How about
sva_mn_release()path just cleared GCR[PASID]/flush cache and theniommu_sva_unbind_device()path (amd_iommu_remove_dev_pasid), decrements count and removespdom_dev_datastructure. something like below?One of the considerations here is the release of the SVA should not trigger errors from the iommu driver if the device continues to use it.
Actually we will hit IO_PAGE_FAULT if device tries to do the DMA. One way to solve this is to have dummy page table. During sva_mn_release() will switch to dummy page table instead of clearing GCR3[PASID] and then in iommu_sva_unbind_device() path clear GCR3[PASID]. That way it will not produce the IO_PAGE_FAULT. Does this makese sense?
-Vasant
Quoted 43 lines · Vasant Hegde
Quoted 31 lines · Vasant Hegde
Quoted 28 lines · Jason Gunthorpe
Quoted 27 lines · Vasant Hegde
Hi Jason,
Quoted 7 lines · Jason Gunthorpe
Quoted 4 lines · Vasant Hegde
Remove the
iommu_group_mutex_assert()call fromamd_iommu_clear_gcr3(), since PASID teardown via themmu_notifierrelease path is a legitimate caller that does not and need not hold the group mutex.Quoted 7 lines
Why doesn't it?
Right below this line is:
gcr3_info->pasid_cnt--;Which certainly needs the lock.
yeah. I overlooked it. Its part of
dev_dataand domain lock is not enough.Quoted 5 lines
In the ARM driver this async release notifier was pretty tricky to lock properly. It just updates the equivilent to the GCR3 entry to make it disabled and doesn't touch anything else. This is locked in a way so that concurrent domain detach doesn't face any races.
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
How about
sva_mn_release()path just cleared GCR[PASID]/flush cache and theniommu_sva_unbind_device()path (amd_iommu_remove_dev_pasid), decrements count and removespdom_dev_datastructure. something like below?One of the considerations here is the release of the SVA should not trigger errors from the iommu driver if the device continues to use it.
Quoted 5 lines
Actually we will hit
IO_PAGE_FAULTif device tries to do the DMA. One way to solve this is to have dummy page table. Duringsva_mn_release()will switch to dummy page table instead of clearing GCR3[PASID] and then iniommu_sva_unbind_device()path clear GCR3[PASID]. That way it will not produce theIO_PAGE_FAULT. Does this makese sense?
Yeah, ARM was going to do something like that too, but it turned out there were bits that disabled walking.
When this happens PRI is still enabled, so we need clean logless PRI not-present responses.
Jason
Vasant Hegde
maintainerJason,
Quoted 44 lines · Jason Gunthorpe
Quoted 43 lines · Vasant Hegde
Quoted 38 lines · Vasant Hegde
Quoted 35 lines · Jason Gunthorpe
Quoted 22 lines · Vasant Hegde
Hi Jason,
Quoted 7 lines · Jason Gunthorpe
Quoted 4 lines · Vasant Hegde
Remove the
iommu_group_mutex_assert()call fromamd_iommu_clear_gcr3(), since PASID teardown via themmu_notifierrelease path is a legitimate caller that does not and need not hold the group mutex.Quoted 7 lines
Why doesn't it?
Right below this line is:
gcr3_info->pasid_cnt--;Which certainly needs the lock.
yeah. I overlooked it. Its part of
dev_dataand domain lock is not enough.Quoted 1 line
Quoted 8 lines
Quoted 4 lines
In the ARM driver this async release notifier was pretty tricky to lock properly. It just updates the equivilent to the GCR3 entry to make it disabled and doesn't touch anything else. This is locked in a way so that concurrent domain detach doesn't face any races.
How about
sva_mn_release()path just cleared GCR[PASID]/flush cache and theniommu_sva_unbind_device()path (amd_iommu_remove_dev_pasid), decrements count and removespdom_dev_datastructure. something like below?One of the considerations here is the release of the SVA should not trigger errors from the iommu driver if the device continues to use it.
Quoted 8 lines
Quoted 5 lines
Actually we will hit
IO_PAGE_FAULTif device tries to do the DMA. One way to solve this is to have dummy page table. Duringsva_mn_release()will switch to dummy page table instead of clearing GCR3[PASID] and then iniommu_sva_unbind_device()path clear GCR3[PASID]. That way it will not produce theIO_PAGE_FAULT. Does this makese sense?Yeah, ARM was going to do something like that too, but it turned out there were bits that disabled walking.
Ok. Let me run some tests and see if we have dummy page table with Present=0, whether we hit PRI fault or IO_PAGE_FAULT.
Quoted 3 lines
When this happens PRI is still enabled, so we need clean logless PRI not-present responses.
if dummy page table doesn't work then should we skip clearing GCR3[PASID] in mn_release path?
-Vasant
Quoted 9 lines · Vasant Hegde
Quoted 8 lines
Quoted 5 lines
Actually we will hit
IO_PAGE_FAULTif device tries to do the DMA. One way to solve this is to have dummy page table. Duringsva_mn_release()will switch to dummy page table instead of clearing GCR3[PASID] and then iniommu_sva_unbind_device()path clear GCR3[PASID]. That way it will not produce theIO_PAGE_FAULT. Does this makese sense?Yeah, ARM was going to do something like that too, but it turned out there were bits that disabled walking.
Quoted 8 lines
Ok. Let me run some tests and see if we have dummy page table with Present=0, whether we hit PRI fault or
IO_PAGE_FAULT.Quoted 2 lines
When this happens PRI is still enabled, so we need clean logless PRI not-present responses.
if dummy page table doesn't work then should we skip clearing GCR3[PASID] in
mn_releasepath?
No, that would UAF things.
It has to work or SVA is broken already!
Jason
Vasant Hegde
maintainerQuoted 16 lines · Jason Gunthorpe
Quoted 15 lines · Vasant Hegde
Quoted 8 lines
Quoted 5 lines
Actually we will hit
IO_PAGE_FAULTif device tries to do the DMA. One way to solve this is to have dummy page table. Duringsva_mn_release()will switch to dummy page table instead of clearing GCR3[PASID] and then iniommu_sva_unbind_device()path clear GCR3[PASID]. That way it will not produce theIO_PAGE_FAULT. Does this makese sense?Yeah, ARM was going to do something like that too, but it turned out there were bits that disabled walking.
Ok. Let me run some tests and see if we have dummy page table with Present=0, whether we hit PRI fault or
IO_PAGE_FAULT.Quoted 2 lines
When this happens PRI is still enabled, so we need clean logless PRI not-present responses.
Quoted 6 lines
Quoted 2 lines
if dummy page table doesn't work then should we skip clearing GCR3[PASID] in
mn_releasepath?No, that would UAF things.
It has to work or SVA is broken already!
We did some experiment. Setting up dummy page table works fine. I will fix it in v2.
-Vasant
Jörg Rödel
maintainerQuoted 4 lines · Vasant Hegde
Remove the
iommu_group_mutex_assert()call fromamd_iommu_clear_gcr3(), since PASID teardown via themmu_notifierrelease path is a legitimate caller that does not and need not hold the group mutex.
Jason is right, there is a race around pasid_cnt. Either make that atomic, or, since it is never read, remove it.
-Joerg