Skip to contentSkip to conversation

iommupt/vtd: Support dirty tracking on PASID

PATCH Superseded Zhenzhong Duan 5 participants 13 messages Feb 5 – Mar 30

Subsystems (4)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover iommupt/vtd: Support dirty tracking on PASID

Hi,

When we add pasid support in QEMU for passthrough device, we found PASID attachment to a nested parent domain with dirty tracking failed.

It's because PASID-level dirty tracking is not there yet, by adding it, we can enable PASID attachment to such domain.

Thanks
Zhenzhong

Zhenzhong Duan (3):
iommupt/vtd: Pass dmar_domain pointer to device_set_dirty_tracking() iommupt/vtd: Support dirty tracking on PASID iommufd/selftest: Test dirty tracking on PASID

 drivers/iommu/intel/iommu.c             | 33 ++++++++++++++-----------
 tools/testing/selftests/iommu/iommufd.c | 27 ++++++++++++++++++++
 2 files changed, 45 insertions(+), 15 deletions(-)
Signature · 1 line
2.47.3
#1 lore raw

PATCH 1/3 iommupt/vtd: Pass dmar_domain pointer to device_set_dirty_tracking()

Pass dmar_domain pointer to device_set_dirty_tracking() instead of device list in the domain.

This is a prerequisite patch to support dirty tracking on PASIDs.

No functional changes intended.

---
 drivers/iommu/intel/iommu.c | 21 +++++++++------------
 1 file changed, 9 insertions(+), 12 deletions(-)
drivers/iommu/intel/iommu.c +9 -12
index 134302fbcd92..acaa6eaf7e2f 100644
@@ -3687,16 +3687,15 @@ static void *intel_iommu_hw_info(struct device *dev, u32 *length,
 	return vtd;
 }
 
-/*
- * Set dirty tracking for the device list of a domain. The caller must
- * hold the domain->lock when calling it.
- */
-static int device_set_dirty_tracking(struct list_head *devices, bool enable)
+/* Set dirty tracking for the devices that the domain has been attached. */
+static int device_set_dirty_tracking(struct dmar_domain *domain, bool enable)
 {
 	struct device_domain_info *info;
 	int ret = 0;
 
-	list_for_each_entry(info, devices, link) {
+	lockdep_assert_held(&domain->lock);
+
+	list_for_each_entry(info, &domain->devices, link) {
 		ret = intel_pasid_setup_dirty_tracking(info->iommu, info->dev,
 						       IOMMU_NO_PASID, enable);
 		if (ret)
@@ -3716,7 +3715,7 @@ static int parent_domain_set_dirty_tracking(struct dmar_domain *domain,
 	spin_lock(&domain->s1_lock);
 	list_for_each_entry(s1_domain, &domain->s1_domains, s2_link) {
 		spin_lock_irqsave(&s1_domain->lock, flags);
-		ret = device_set_dirty_tracking(&s1_domain->devices, enable);
+		ret = device_set_dirty_tracking(s1_domain, enable);
 		spin_unlock_irqrestore(&s1_domain->lock, flags);
 		if (ret)
 			goto err_unwind;
@@ -3727,8 +3726,7 @@ static int parent_domain_set_dirty_tracking(struct dmar_domain *domain,
 err_unwind:
 	list_for_each_entry(s1_domain, &domain->s1_domains, s2_link) {
 		spin_lock_irqsave(&s1_domain->lock, flags);
-		device_set_dirty_tracking(&s1_domain->devices,
-					  domain->dirty_tracking);
+		device_set_dirty_tracking(s1_domain, domain->dirty_tracking);
 		spin_unlock_irqrestore(&s1_domain->lock, flags);
 	}
 	spin_unlock(&domain->s1_lock);
@@ -3745,7 +3743,7 @@ static int intel_iommu_set_dirty_tracking(struct iommu_domain *domain,
 	if (dmar_domain->dirty_tracking == enable)
 		goto out_unlock;
 
-	ret = device_set_dirty_tracking(&dmar_domain->devices, enable);
+	ret = device_set_dirty_tracking(dmar_domain, enable);
 	if (ret)
 		goto err_unwind;
 
@@ -3762,8 +3760,7 @@ static int intel_iommu_set_dirty_tracking(struct iommu_domain *domain,
 	return 0;
 
 err_unwind:
-	device_set_dirty_tracking(&dmar_domain->devices,
-				  dmar_domain->dirty_tracking);
+	device_set_dirty_tracking(dmar_domain, dmar_domain->dirty_tracking);
 	spin_unlock(&dmar_domain->lock);
 	return ret;
 }
Signature · 1 line
2.47.3
#2 lore raw

PATCH 2/3 iommupt/vtd: Support dirty tracking on PASID

In reply to Zhenzhong Duan

In order to support passthrough device with PASID capability in QEMU, e.g., DSA device, kernel needs to support attaching PASID to a domain.

But attaching is not allowed if the domain is a second stage domain with dirty tracking, this is the use case when guest configures 'iommu=pt'.

The reason is kernel lacking support for dirty tracking on such domain attached to PASID. Same for nested domain even though there is no such checking.

By adding dirty tracking on PASID, the check can be removed.

---
 drivers/iommu/intel/iommu.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
drivers/iommu/intel/iommu.c +9 -3
index acaa6eaf7e2f..5a9ed91109a2 100644
@@ -3621,9 +3621,6 @@ static int intel_iommu_set_dev_pasid(struct iommu_domain *domain,
 	if (!pasid_supported(iommu) || dev_is_real_dma_subdevice(dev))
 		return -EOPNOTSUPP;
 
-	if (domain->dirty_ops)
-		return -EINVAL;
-
 	if (context_copied(iommu, info->bus, info->devfn))
 		return -EBUSY;
 
@@ -3691,6 +3688,7 @@ static void *intel_iommu_hw_info(struct device *dev, u32 *length,
 static int device_set_dirty_tracking(struct dmar_domain *domain, bool enable)
 {
 	struct device_domain_info *info;
+	struct dev_pasid_info *dev_pasid;
 	int ret = 0;
 
 	lockdep_assert_held(&domain->lock);
@@ -3698,6 +3696,14 @@ static int device_set_dirty_tracking(struct dmar_domain *domain, bool enable)
 	list_for_each_entry(info, &domain->devices, link) {
 		ret = intel_pasid_setup_dirty_tracking(info->iommu, info->dev,
 						       IOMMU_NO_PASID, enable);
+		if (ret)
+			return ret;
+	}
+
+	list_for_each_entry(dev_pasid, &domain->dev_pasids, link_domain) {
+		info = dev_iommu_priv_get(dev_pasid->dev);
+		ret = intel_pasid_setup_dirty_tracking(info->iommu, info->dev,
+						       dev_pasid->pasid, enable);
 		if (ret)
 			break;
 	}
Signature · 1 line
2.47.3
maintainer
#3 lore raw
Quoted 6 lines

From: Duan, Zhenzhong <[email protected]>
Sent: Thursday, February 5, 2026 10:34 AM

In order to support passthrough device with PASID capability in QEMU, e.g., DSA device, kernel needs to support attaching PASID to a domain.

Quoted 6 lines

But attaching is not allowed if the domain is a second stage domain with dirty tracking, this is the use case when guest configures 'iommu=pt'.

The reason is kernel lacking support for dirty tracking on such domain attached to PASID. Same for nested domain even though there is no such checking.

no check for nested domain is a bug. We should first add a check and backport it to stable kernel. Then have this series to allow it in new kernels.

#4 lore raw
Quoted 15 lines

-----Original Message-----
From: Tian, Kevin <[email protected]>
Subject: RE: [PATCH 2/3] iommupt/vtd: Support dirty tracking on PASID

Quoted 9 lines

From: Duan, Zhenzhong <[email protected]>
Sent: Thursday, February 5, 2026 10:34 AM

In order to support passthrough device with PASID capability in QEMU, e.g., DSA device, kernel needs to support attaching PASID to a domain.

But attaching is not allowed if the domain is a second stage domain with dirty tracking, this is the use case when guest configures 'iommu=pt'.

Quoted 7 lines
Quoted 3 lines

The reason is kernel lacking support for dirty tracking on such domain attached to PASID. Same for nested domain even though there is no such checking.

no check for nested domain is a bug. We should first add a check and backport it to stable kernel. Then have this series to allow it in new kernels.

Make sense, will send v2 with the bug fix as first patch.

Thanks
Zhenzhong

#5 lore raw

PATCH 3/3 iommufd/selftest: Test dirty tracking on PASID

In reply to Zhenzhong Duan

Add test case for dirty tracking on a domain attached to PASID, also confirm attachment to PASID fail if device doesn't support dirty tracking.

---
 tools/testing/selftests/iommu/iommufd.c | 27 +++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
tools/testing/selftests/iommu/iommufd.c +27 -0
index dadad277f4eb..d1fe5dbc2813 100644
@@ -2275,6 +2275,33 @@ TEST_F(iommufd_dirty_tracking, set_dirty_tracking)
 	test_ioctl_destroy(hwpt_id);
 }
 
+TEST_F(iommufd_dirty_tracking, pasid_set_dirty_tracking)
+{
+	uint32_t stddev_id, ioas_id, hwpt_id, pasid = 100;
+	uint32_t dev_flags = MOCK_FLAGS_DEVICE_PASID;
+
+	/* Regular case */
+	test_cmd_hwpt_alloc(self->idev_id, self->ioas_id,
+			    IOMMU_HWPT_ALLOC_PASID | IOMMU_HWPT_ALLOC_DIRTY_TRACKING,
+			    &hwpt_id);
+	test_cmd_mock_domain_flags(hwpt_id, dev_flags, &stddev_id, NULL, NULL);
+	ASSERT_EQ(0, _test_cmd_pasid_attach(self->fd, stddev_id, pasid, hwpt_id));
+	test_cmd_set_dirty_tracking(hwpt_id, true);
+	test_cmd_set_dirty_tracking(hwpt_id, false);
+	ASSERT_EQ(0, _test_cmd_pasid_detach(self->fd, stddev_id, pasid));
+
+	test_ioctl_destroy(stddev_id);
+
+	/* IOMMU device does not support dirty tracking */
+	dev_flags |= MOCK_FLAGS_DEVICE_NO_DIRTY;
+	test_ioctl_ioas_alloc(&ioas_id);
+	test_cmd_mock_domain_flags(ioas_id, dev_flags, &stddev_id, NULL, NULL);
+	EXPECT_ERRNO(EINVAL, _test_cmd_pasid_attach(self->fd, stddev_id, pasid, hwpt_id));
+
+	test_ioctl_destroy(stddev_id);
+	test_ioctl_destroy(hwpt_id);
+}
+
 TEST_F(iommufd_dirty_tracking, device_dirty_capability)
 {
 	uint32_t caps = 0;
Signature · 1 line
2.47.3
#6 lore raw
In reply to Zhenzhong Duan
Quoted 8 lines · Zhenzhong Duan

Hi,

When we add pasid support in QEMU for passthrough device, we found PASID attachment to a nested parent domain with dirty tracking failed.

It's because PASID-level dirty tracking is not there yet, by adding it, we can enable PASID attachment to such domain.

Quoted 8 lines

Thanks
Zhenzhong

Zhenzhong Duan (3):
iommupt/vtd: Pass dmar_domain pointer to device_set_dirty_tracking() iommupt/vtd: Support dirty tracking on PASID iommufd/selftest: Test dirty tracking on PASID

I'm not sure why you marked this s iommupt? It isn't right?

Also only the second stage table supports dirty tracking so I'm confused what is the issue? Is it trying to say that if a second stage is written to a pasid entry on a !=0 pasid path it doesn't work right? But it does work on a 0 pasid?

Huh?

That sounds like a systemic problem, the two flows shouldn't actually be any different.

Jason

#7 lore raw
Quoted 19 lines

-----Original Message-----
From: Jason Gunthorpe <[email protected]>
Subject: Re: [PATCH 0/3] iommupt/vtd: Support dirty tracking on PASID

Quoted 12 lines · Zhenzhong Duan

Hi,

When we add pasid support in QEMU for passthrough device, we found PASID attachment to a nested parent domain with dirty tracking failed.

It's because PASID-level dirty tracking is not there yet, by adding it, we can enable PASID attachment to such domain.

Thanks
Zhenzhong

Quoted 6 lines
Quoted 4 lines

Zhenzhong Duan (3):
iommupt/vtd: Pass dmar_domain pointer to device_set_dirty_tracking() iommupt/vtd: Support dirty tracking on PASID iommufd/selftest: Test dirty tracking on PASID

I'm not sure why you marked this s iommupt? It isn't right?

Oops, should be iommu/vtd.

Quoted 4 lines

Also only the second stage table supports dirty tracking so I'm confused what is the issue? Is it trying to say that if a second stage is written to a pasid entry on a !=0 pasid path it doesn't work right?

Yes, more strictly, if a second stage domain flagged with IOMMU_HWPT_ALLOC_DIRTY_TRACKING is written to a pasid entry on a !=0 pasid, it's not allowed currently, by below check:

static int intel_iommu_set_dev_pasid()
{
...

        if (domain->dirty_ops)
                return -EINVAL;
Quoted 1 line

But it does work on a 0 pasid?

More strictly, it works on IOMMU_NO_PASID.
For IOMMU_NO_PASID, we trigger attach_dev(), for other pasids, We trigger set_dev_pasid().

Quoted 5 lines

Huh?

That sounds like a systemic problem, the two flows shouldn't actually be any different.

Yes, just because we have code to setup dirty tracking on 0 pasid entry but lack the same code for !=0 pasid entry in VTD.

0 pasid is special that it can also be IOMMU_NO_PASID if PASID isn't supported.

Thanks
Zhenzhong

#8 lore raw
In reply to Jason Gunthorpe

Hi,

Kindly ping, any comments?

Thanks
Zhenzhong

Quoted 16 lines · Zhenzhong Duan

Hi,

When we add pasid support in QEMU for passthrough device, we found PASID attachment to a nested parent domain with dirty tracking failed.

It's because PASID-level dirty tracking is not there yet, by adding it, we can enable PASID attachment to such domain.

Thanks
Zhenzhong

Zhenzhong Duan (3):
iommupt/vtd: Pass dmar_domain pointer to device_set_dirty_tracking() iommupt/vtd: Support dirty tracking on PASID iommufd/selftest: Test dirty tracking on PASID

maintainer
#9 lore raw
Quoted 3 lines · Zhenzhong

Hi,

Kindly ping, any comments?

This series looks fine to me. I will queue it (with the typo in the subject fixed) for iommu/next if no further concerns. Thanks!

Quoted 21 lines

Thanks
Zhenzhong

Quoted 16 lines · Zhenzhong Duan

Hi,

When we add pasid support in QEMU for passthrough device, we found PASID attachment to a nested parent domain with dirty tracking failed.

It's because PASID-level dirty tracking is not there yet, by adding it, we can enable PASID attachment to such domain.

Thanks
Zhenzhong

Zhenzhong Duan (3):
iommupt/vtd: Pass dmar_domain pointer to device_set_dirty_tracking() iommupt/vtd: Support dirty tracking on PASID iommufd/selftest: Test dirty tracking on PASID

Reviewed-by
#10 lore raw

Hi Baolu,

Quoted 7 lines · Baolu Lu
Quoted 3 lines · Zhenzhong

Hi,

Kindly ping, any comments?

This series looks fine to me. I will queue it (with the typo in the subject fixed) for iommu/next if no further concerns. Thanks!

The series also looks good to me with one more nit besides the typo in the subjects.

I think it's better to name device_set_dirty_tracking() to be domain_set_dirty_tracking() in patch 01.

Feel free to add:

Regards,
Yi Liu

#11 lore raw

Hi Baolu,

Quoted 12 lines

-----Original Message-----
From: Liu, Yi L <[email protected]>
Subject: Re: [PATCH 0/3] iommupt/vtd: Support dirty tracking on PASID

Hi Baolu,

Quoted 5 lines · Baolu Lu
Quoted 3 lines · Zhenzhong

Hi,

Kindly ping, any comments?

Quoted 7 lines
Quoted 2 lines

This series looks fine to me. I will queue it (with the typo in the subject fixed) for iommu/next if no further concerns. Thanks!

The series also looks good to me with one more nit besides the typo in the subjects.

I think it's better to name device_set_dirty_tracking() to be domain_set_dirty_tracking() in patch 01.

May I ask your preference so I could have it in next respin, device_set_dirty_tracking vs domain_set_dirty_tracking?

Thanks
Zhenzhong

Quoted 7 lines

Feel free to add:

Regards,
Yi Liu

maintainer
#12 lore raw
Quoted 13 lines · Zhenzhong

Hi Baolu,

Quoted 11 lines

-----Original Message-----
From: Liu, Yi L<[email protected]>
Subject: Re: [PATCH 0/3] iommupt/vtd: Support dirty tracking on PASID

Hi Baolu,

Quoted 4 lines · Baolu Lu
Quoted 3 lines · Zhenzhong

Hi,

Kindly ping, any comments?

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

This series looks fine to me. I will queue it (with the typo in the subject fixed) for iommu/next if no further concerns. Thanks!

The series also looks good to me with one more nit besides the typo in the subjects.

I think it's better to name device_set_dirty_tracking() to be domain_set_dirty_tracking() in patch 01.

May I ask your preference so I could have it in next respin, device_set_dirty_tracking vs domain_set_dirty_tracking?

I prefer domain_set_dirty_tracking() according to what the helper does.

Thanks,
baolu