Skip to contentSkip to conversation

iommu/amd: Remove iommu_group_mutex_assert() from amd_iommu_clear_gcr3()

PATCH In review Vasant Hegde 3 participants 11 messages Aug 14 – Nov 5

Subsystems (2)
lore
Share & tools
lore

Vasant Hegde

author maintainer
#0 lore raw

PATCH 1/1 iommu/amd: Remove iommu_group_mutex_assert() from amd_iommu_clear_gcr3()

amd_iommu_clear_gcr3() is called from sva_mn_release() during the mmu_notifier release path to detach a PASIDs when a process exits. This path does not hold the iommu group mutex, since it is invoked asynchronously from mm teardown context rather than through the normal IOMMU group/attach APIs.

As a result, the iommu_group_mutex_assert() check in amd_iommu_clear_gcr3() logs a kernel warning.

[62863.761300] ------------[ cut here ]------------
[62863.761320] WARNING: drivers/iommu/iommu.c:1361 at iommu_group_mutex_assert+0x3a/0x50, CPU#14: kworker/14:2/621
...
[62863.761706]  <TASK>
[62863.761712]  amd_iommu_clear_gcr3+0x1e/0x50
[62863.761724]  sva_mn_release+0x68/0xd0
[62863.761739]  __mmu_notifier_release+0xb9/0x2d0
[62863.761763]  exit_mmap+0x427/0x460
[62863.761836]  __mmput+0x41/0x120
[62863.761844]  mmput_async_fn+0x15/0x20
[62863.761852]  process_one_work+0x22e/0x780
[62863.761876]  worker_thread+0x1b5/0x380
[62863.761892]  kthread+0x10d/0x150
[62863.761913]  ret_from_fork+0x346/0x3c0
[62863.761930]  ret_from_fork_asm+0x1a/0x30
[62863.761964]  </TASK>

Remove the iommu_group_mutex_assert() call from amd_iommu_clear_gcr3(), since PASID teardown via the mmu_notifier release path is a legitimate caller that does not and need not hold the group mutex.

Fixes 1af95763e0a3 ("iommu/amd: Initial SVA support for AMD IOMMU")
---
 drivers/iommu/amd/iommu.c | 2 --
 1 file changed, 2 deletions(-)
drivers/iommu/amd/iommu.c +0 -2
index 29dc18d3d22e..1fe99fb98793 100644
@@ -2063,8 +2063,6 @@ int amd_iommu_clear_gcr3(struct iommu_dev_data *dev_data, ioasid_t pasid)
 	struct gcr3_tbl_info *gcr3_info = &dev_data->gcr3_info;
 	int ret;
 
-	iommu_group_mutex_assert(dev_data->dev);
-
 	ret = update_gcr3(dev_data, pasid, 0, false);
 	if (ret)
 		return ret;
Signature · 1 line
2.31.1
#1 lore raw
Quoted 4 lines · Vasant Hegde

Remove the iommu_group_mutex_assert() call from amd_iommu_clear_gcr3(), since PASID teardown via the mmu_notifier release path is a legitimate caller that does not and need not hold the group mutex.

Why doesn't it?

Right below this line is:

	gcr3_info->pasid_cnt--;

Which certainly needs the lock.

In the ARM driver this async release notifier was pretty tricky to lock properly. It just updates the equivilent to the GCR3 entry to make it disabled and doesn't touch anything else. This is locked in a way so that concurrent domain detach doesn't face any races.

AMD should have the same basic design here too.

Jason

author maintainer
#2 lore raw

Hi Jason,

Quoted 7 lines · Jason Gunthorpe
Quoted 4 lines · Vasant Hegde

Remove the iommu_group_mutex_assert() call from amd_iommu_clear_gcr3(), since PASID teardown via the mmu_notifier release path is a legitimate caller that does not and need not hold the group mutex.

Quoted 7 lines

Why doesn't it?

Right below this line is:

	gcr3_info->pasid_cnt--;

Which certainly needs the lock.

yeah. I overlooked it. Its part of dev_data and domain lock is not enough.

Quoted 5 lines

In the ARM driver this async release notifier was pretty tricky to lock properly. It just updates the equivilent to the GCR3 entry to make it disabled and doesn't touch anything else. This is locked in a way so that concurrent domain detach doesn't face any races.

How about sva_mn_release() path just cleared GCR[PASID]/flush cache and then iommu_sva_unbind_device() path (amd_iommu_remove_dev_pasid), decrements count and removes pdom_dev_data structure. something like below?

---<---

drivers/iommu/amd/amd_iommu.h +1 -0
index 4546f8be34ef..81da797b3c55 100644
@@ -77,6 +77,7 @@ void amd_iommu_iopf_remove_device(struct amd_iommu *iommu,
 int amd_iommu_set_gcr3(struct iommu_dev_data *dev_data,
 		       ioasid_t pasid, unsigned long gcr3);
 int amd_iommu_clear_gcr3(struct iommu_dev_data *dev_data, ioasid_t pasid);
+int amd_iommu_disable_gcr3(struct iommu_dev_data *dev_data, ioasid_t pasid);
 
 /* PPR */
 int __init amd_iommu_alloc_ppr_log(struct amd_iommu *iommu);
drivers/iommu/amd/iommu.c +0 -0
index a0f149a8f825..2e7a84b614b5 100644
@@ -2070,6 +2070,15 @@ int amd_iommu_set_gcr3(struct iommu_dev_data *dev_data,

ioasid_t pasid,
return ret;
}

+/*
+ * Disable a GCR3 entry without detaching its PASID from the IOMMU core.
+ * This is used from the SVA mmu notifier release callback.
+ */
+int amd_iommu_disable_gcr3(struct iommu_dev_data *dev_data, ioasid_t pasid) +{
+ return update_gcr3(dev_data, pasid, 0, false);
+}
+
int amd_iommu_clear_gcr3(struct iommu_dev_data *dev_data, ioasid_t pasid) {
struct gcr3_tbl_info *gcr3_info = &dev_data->gcr3_info;
@@ -2077,7 +2086,7 @@ int amd_iommu_clear_gcr3(struct iommu_dev_data *dev_data, ioasid_t pasid)

iommu_group_mutex_assert(dev_data->dev);

- ret = update_gcr3(dev_data, pasid, 0, false);
+ ret = amd_iommu_disable_gcr3(dev_data, pasid);
if (ret)
return ret;

drivers/iommu/amd/pasid.c +0 -0
index 40be5902087c..43575cf742bb 100644
@@ -79,7 +79,7 @@ static void sva_arch_invalidate_secondary_tlbs(struct

mmu_notifier *mn,

static void sva_mn_release(struct mmu_notifier *mn, struct mm_struct *mm) {
- struct pdom_dev_data *pdom_dev_data, *next;
+ struct pdom_dev_data *pdom_dev_data;
struct protection_domain *sva_pdom;
unsigned long flags;

@@ -87,9 +87,14 @@ static void sva_mn_release(struct mmu_notifier *mn, struct mm_struct *mm)

spin_lock_irqsave(&sva_pdom->lock, flags);

- /* Assume dev_data_list contains same PASID with different devices */
- for_each_pdom_dev_data_safe(pdom_dev_data, next, sva_pdom)
- remove_dev_pasid(pdom_dev_data);
+ /*
+ * Disable the GCR3 entry before the mm page tables are freed. Leave
+ * the PASID attachment and pdom_dev_data for the IOMMU core to
+ * remove through the normal detach path.
+ */
+ for_each_pdom_dev_data(pdom_dev_data, sva_pdom)
+ amd_iommu_disable_gcr3(pdom_dev_data->dev_data,
+ pdom_dev_data->pasid);

spin_unlock_irqrestore(&sva_pdom->lock, flags);
}

#3 lore raw
Quoted 22 lines · Vasant Hegde

Hi Jason,

Quoted 7 lines · Jason Gunthorpe
Quoted 4 lines · Vasant Hegde

Remove the iommu_group_mutex_assert() call from amd_iommu_clear_gcr3(), since PASID teardown via the mmu_notifier release path is a legitimate caller that does not and need not hold the group mutex.

Quoted 7 lines

Why doesn't it?

Right below this line is:

	gcr3_info->pasid_cnt--;

Which certainly needs the lock.

yeah. I overlooked it. Its part of dev_data and domain lock is not enough.

Quoted 1 line
Quoted 8 lines
Quoted 4 lines

In the ARM driver this async release notifier was pretty tricky to lock properly. It just updates the equivilent to the GCR3 entry to make it disabled and doesn't touch anything else. This is locked in a way so that concurrent domain detach doesn't face any races.

How about sva_mn_release() path just cleared GCR[PASID]/flush cache and then iommu_sva_unbind_device() path (amd_iommu_remove_dev_pasid), decrements count and removes pdom_dev_data structure. something like below?

One of the considerations here is the release of the SVA should not trigger errors from the iommu driver if the device continues to use it.

Every device access should continue to be captured as page fault and run the SVA page fault path. It will naturally end in an error.

Not sure this:

Review

Quoted 8 lines
(quoted diff) +8 -0
+/*
+ * Disable a GCR3 entry without detaching its PASID from the IOMMU core.
+ * This is used from the SVA mmu notifier release callback.
+ */
+int amd_iommu_disable_gcr3(struct iommu_dev_data *dev_data, ioasid_t pasid)
+{
+	return update_gcr3(dev_data, pasid, 0, false);
+}

Gets there? Or at least explain it in the comment?

Jason

author maintainer
#4 lore raw
Quoted 28 lines · Jason Gunthorpe
Quoted 27 lines · Vasant Hegde

Hi Jason,

Quoted 7 lines · Jason Gunthorpe
Quoted 4 lines · Vasant Hegde

Remove the iommu_group_mutex_assert() call from amd_iommu_clear_gcr3(), since PASID teardown via the mmu_notifier release path is a legitimate caller that does not and need not hold the group mutex.

Quoted 7 lines

Why doesn't it?

Right below this line is:

	gcr3_info->pasid_cnt--;

Which certainly needs the lock.

yeah. I overlooked it. Its part of dev_data and domain lock is not enough.

Quoted 5 lines

In the ARM driver this async release notifier was pretty tricky to lock properly. It just updates the equivilent to the GCR3 entry to make it disabled and doesn't touch anything else. This is locked in a way so that concurrent domain detach doesn't face any races.

Quoted 7 lines
Quoted 3 lines

How about sva_mn_release() path just cleared GCR[PASID]/flush cache and then iommu_sva_unbind_device() path (amd_iommu_remove_dev_pasid), decrements count and removes pdom_dev_data structure. something like below?

One of the considerations here is the release of the SVA should not trigger errors from the iommu driver if the device continues to use it.

I am clearing only the GCR3 table entry. It will block page table walk for this particular PASID. All inflight requests and new request for other PASIDs will continue to be processed.

Quoted 5 lines

Every device access should continue to be captured as page fault and run the SVA page fault path. It will naturally end in an error.

Quoted 1 line

Not sure this:

Quoted 10 lines

Review

Quoted 8 lines
(quoted diff) +8 -0
+/*
+ * Disable a GCR3 entry without detaching its PASID from the IOMMU core.
+ * This is used from the SVA mmu notifier release callback.
+ */
+int amd_iommu_disable_gcr3(struct iommu_dev_data *dev_data, ioasid_t pasid)
+{
+	return update_gcr3(dev_data, pasid, 0, false);
+}

Gets there? Or at least explain it in the comment?

Sure. I will add a comment. I realized AMD SVA needs lot more fixes. I will send V2 along with few other fixes to handle device reset scenarios.

-Vasant

author maintainer
#5 lore raw
Quoted 31 lines · Vasant Hegde
Quoted 28 lines · Jason Gunthorpe
Quoted 27 lines · Vasant Hegde

Hi Jason,

Quoted 7 lines · Jason Gunthorpe
Quoted 4 lines · Vasant Hegde

Remove the iommu_group_mutex_assert() call from amd_iommu_clear_gcr3(), since PASID teardown via the mmu_notifier release path is a legitimate caller that does not and need not hold the group mutex.

Quoted 7 lines

Why doesn't it?

Right below this line is:

	gcr3_info->pasid_cnt--;

Which certainly needs the lock.

yeah. I overlooked it. Its part of dev_data and domain lock is not enough.

Quoted 5 lines

In the ARM driver this async release notifier was pretty tricky to lock properly. It just updates the equivilent to the GCR3 entry to make it disabled and doesn't touch anything else. This is locked in a way so that concurrent domain detach doesn't face any races.

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

How about sva_mn_release() path just cleared GCR[PASID]/flush cache and then iommu_sva_unbind_device() path (amd_iommu_remove_dev_pasid), decrements count and removes pdom_dev_data structure. something like below?

One of the considerations here is the release of the SVA should not trigger errors from the iommu driver if the device continues to use it.

Actually we will hit IO_PAGE_FAULT if device tries to do the DMA. One way to solve this is to have dummy page table. During sva_mn_release() will switch to dummy page table instead of clearing GCR3[PASID] and then in iommu_sva_unbind_device() path clear GCR3[PASID]. That way it will not produce the IO_PAGE_FAULT. Does this makese sense?

-Vasant

#6 lore raw
Quoted 43 lines · Vasant Hegde
Quoted 31 lines · Vasant Hegde
Quoted 28 lines · Jason Gunthorpe
Quoted 27 lines · Vasant Hegde

Hi Jason,

Quoted 7 lines · Jason Gunthorpe
Quoted 4 lines · Vasant Hegde

Remove the iommu_group_mutex_assert() call from amd_iommu_clear_gcr3(), since PASID teardown via the mmu_notifier release path is a legitimate caller that does not and need not hold the group mutex.

Quoted 7 lines

Why doesn't it?

Right below this line is:

	gcr3_info->pasid_cnt--;

Which certainly needs the lock.

yeah. I overlooked it. Its part of dev_data and domain lock is not enough.

Quoted 5 lines

In the ARM driver this async release notifier was pretty tricky to lock properly. It just updates the equivilent to the GCR3 entry to make it disabled and doesn't touch anything else. This is locked in a way so that concurrent domain detach doesn't face any races.

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

How about sva_mn_release() path just cleared GCR[PASID]/flush cache and then iommu_sva_unbind_device() path (amd_iommu_remove_dev_pasid), decrements count and removes pdom_dev_data structure. something like below?

One of the considerations here is the release of the SVA should not trigger errors from the iommu driver if the device continues to use it.

Quoted 5 lines

Actually we will hit IO_PAGE_FAULT if device tries to do the DMA. One way to solve this is to have dummy page table. During sva_mn_release() will switch to dummy page table instead of clearing GCR3[PASID] and then in iommu_sva_unbind_device() path clear GCR3[PASID]. That way it will not produce the IO_PAGE_FAULT. Does this makese sense?

Yeah, ARM was going to do something like that too, but it turned out there were bits that disabled walking.

When this happens PRI is still enabled, so we need clean logless PRI not-present responses.

Jason

author maintainer
#7 lore raw

Jason,

Quoted 44 lines · Jason Gunthorpe
Quoted 43 lines · Vasant Hegde
Quoted 38 lines · Vasant Hegde
Quoted 35 lines · Jason Gunthorpe
Quoted 22 lines · Vasant Hegde

Hi Jason,

Quoted 7 lines · Jason Gunthorpe
Quoted 4 lines · Vasant Hegde

Remove the iommu_group_mutex_assert() call from amd_iommu_clear_gcr3(), since PASID teardown via the mmu_notifier release path is a legitimate caller that does not and need not hold the group mutex.

Quoted 7 lines

Why doesn't it?

Right below this line is:

	gcr3_info->pasid_cnt--;

Which certainly needs the lock.

yeah. I overlooked it. Its part of dev_data and domain lock is not enough.

Quoted 1 line
Quoted 8 lines
Quoted 4 lines

In the ARM driver this async release notifier was pretty tricky to lock properly. It just updates the equivilent to the GCR3 entry to make it disabled and doesn't touch anything else. This is locked in a way so that concurrent domain detach doesn't face any races.

How about sva_mn_release() path just cleared GCR[PASID]/flush cache and then iommu_sva_unbind_device() path (amd_iommu_remove_dev_pasid), decrements count and removes pdom_dev_data structure. something like below?

One of the considerations here is the release of the SVA should not trigger errors from the iommu driver if the device continues to use it.

Quoted 8 lines
Quoted 5 lines

Actually we will hit IO_PAGE_FAULT if device tries to do the DMA. One way to solve this is to have dummy page table. During sva_mn_release() will switch to dummy page table instead of clearing GCR3[PASID] and then in iommu_sva_unbind_device() path clear GCR3[PASID]. That way it will not produce the IO_PAGE_FAULT. Does this makese sense?

Yeah, ARM was going to do something like that too, but it turned out there were bits that disabled walking.

Ok. Let me run some tests and see if we have dummy page table with Present=0, whether we hit PRI fault or IO_PAGE_FAULT.

Quoted 3 lines

When this happens PRI is still enabled, so we need clean logless PRI not-present responses.

if dummy page table doesn't work then should we skip clearing GCR3[PASID] in mn_release path?

-Vasant

#8 lore raw
Quoted 9 lines · Vasant Hegde
Quoted 8 lines
Quoted 5 lines

Actually we will hit IO_PAGE_FAULT if device tries to do the DMA. One way to solve this is to have dummy page table. During sva_mn_release() will switch to dummy page table instead of clearing GCR3[PASID] and then in iommu_sva_unbind_device() path clear GCR3[PASID]. That way it will not produce the IO_PAGE_FAULT. Does this makese sense?

Yeah, ARM was going to do something like that too, but it turned out there were bits that disabled walking.

Quoted 8 lines

Ok. Let me run some tests and see if we have dummy page table with Present=0, whether we hit PRI fault or IO_PAGE_FAULT.

Quoted 2 lines

When this happens PRI is still enabled, so we need clean logless PRI not-present responses.

if dummy page table doesn't work then should we skip clearing GCR3[PASID] in mn_release path?

No, that would UAF things.

It has to work or SVA is broken already!

Jason

author maintainer
#9 lore raw
Quoted 16 lines · Jason Gunthorpe
Quoted 15 lines · Vasant Hegde
Quoted 8 lines
Quoted 5 lines

Actually we will hit IO_PAGE_FAULT if device tries to do the DMA. One way to solve this is to have dummy page table. During sva_mn_release() will switch to dummy page table instead of clearing GCR3[PASID] and then in iommu_sva_unbind_device() path clear GCR3[PASID]. That way it will not produce the IO_PAGE_FAULT. Does this makese sense?

Yeah, ARM was going to do something like that too, but it turned out there were bits that disabled walking.

Ok. Let me run some tests and see if we have dummy page table with Present=0, whether we hit PRI fault or IO_PAGE_FAULT.

Quoted 2 lines

When this happens PRI is still enabled, so we need clean logless PRI not-present responses.

Quoted 6 lines
Quoted 2 lines

if dummy page table doesn't work then should we skip clearing GCR3[PASID] in mn_release path?

No, that would UAF things.

It has to work or SVA is broken already!

We did some experiment. Setting up dummy page table works fine. I will fix it in v2.

-Vasant

maintainer
#10 lore raw
In reply to Vasant Hegde
Quoted 4 lines · Vasant Hegde

Remove the iommu_group_mutex_assert() call from amd_iommu_clear_gcr3(), since PASID teardown via the mmu_notifier release path is a legitimate caller that does not and need not hold the group mutex.

Jason is right, there is a race around pasid_cnt. Either make that atomic, or, since it is never read, remove it.

-Joerg