iommu/vt-d: Remove caching mode check before device TLB flush
Subsystems (2)
Lu Baolu
maintainerPATCH v2 1/1 iommu/vt-d: Remove caching mode check before device TLB flush
The Caching Mode (CM) of the Intel IOMMU indicates if the hardware implementation caches not-present or erroneous translation-structure entries except the first-stage translation. The caching mode is irrelevant to the device TLB , therefore there is no need to check it before a device TLB invalidation operation.
iommu_flush_iotlb_psi() is called in map and unmap paths. The caching mode check before device TLB invalidation will cause device TLB invalidation always issued if IOMMU is not running in caching mode. This is wrong and causes unnecessary performance overhead.
The removal of caching mode check in intel_flush_iotlb_all() doesn't impact anything no matter the IOMMU is working in caching mode or not. Commit <29b32839725f> ("iommu/vt-d: Do not use flush-queue when caching-mode is on") has already disabled flush-queue for caching mode, hence caching mode will never call intel_flush_iotlb_all().
--- drivers/iommu/intel/iommu.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)
Change log:
v2:
- Squash two patches into a single one.
- No functionality changes.
v1: https://lore.kernel.org/linux-iommu/[email protected]/
drivers/iommu/intel/iommu.c +2 -7
@@ -1501,11 +1501,7 @@ static void iommu_flush_iotlb_psi(struct intel_iommu *iommu,else __iommu_flush_iotlb_psi (iommu, did, pfn, pages, ih); -/* -* In caching mode, changes of pages from non-present to present require -* flush. However, device IOTLB doesn't need to be flushed in this case. -*/ -if (!cap_caching_mode (iommu->cap) || !map) +if (!map)iommu_flush_dev_iotlb (domain, addr, mask); }
@@ -1579,8 +1575,7 @@ static void intel_flush_iotlb_all(struct iommu_domain *domain) iommu->flush.flush_iotlb (iommu, did,0 ,0 ,DMA_TLB_DSI_FLUSH ); -if (!cap_caching_mode (iommu->cap)) -iommu_flush_dev_iotlb (dmar_domain,0 ,MAX_AGAW_PFN_WIDTH ); +iommu_flush_dev_iotlb (dmar_domain,0 ,MAX_AGAW_PFN_WIDTH ); }if (dmar_domain->nested_parent)
Signature · 1 line
2.34.1
Quoted 20 lines
From: Lu Baolu <[email protected]>
Sent: Wednesday, April 10, 2024 1:58 PMThe Caching Mode (CM) of the Intel IOMMU indicates if the hardware implementation caches not-present or erroneous translation-structure entries except the first-stage translation. The caching mode is irrelevant to the device TLB , therefore there is no need to check it before a device TLB invalidation operation.
iommu_flush_iotlb_psi()is called in map and unmap paths. The caching mode check before device TLB invalidation will cause device TLB invalidation always issued if IOMMU is not running in caching mode. This is wrong and causes unnecessary performance overhead.The removal of caching mode check in
intel_flush_iotlb_all()doesn't impact anything no matter the IOMMU is working in caching mode or not. Commit <29b32839725f> ("iommu/vt-d: Do not use flush-queue when caching-mode is on") has already disabled flush-queue for caching mode, hence caching mode will never callintel_flush_iotlb_all().
Quoted 3 lines
Fixes bf92df30df90 ("intel-iommu: Only avoid flushing device IOTLB fordomain ID 0 in caching mode")
Signed-off-by Lu Baolu <[email protected]>
Quoted 10 lines · Lu Baolu
The Caching Mode (CM) of the Intel IOMMU indicates if the hardware implementation caches not-present or erroneous translation-structure entries except the first-stage translation. The caching mode is irrelevant to the device TLB , therefore there is no need to check it before a device TLB invalidation operation.
iommu_flush_iotlb_psi()is called in map and unmap paths. The caching mode check before device TLB invalidation will cause device TLB invalidation always issued if IOMMU is not running in caching mode. This is wrong and causes unnecessary performance overhead.
I don't think the original code is wrong. As I replied before, if CM==0, the iommu_flush_iotlb_psi() is only called in unmap path, in which the @map is false. [1] The reason to make the change is to make the logic simpler. :)
[1] https://lore.kernel.org/linux-iommu/[email protected]/
Quoted 38 lines
The removal of caching mode check in
intel_flush_iotlb_all()doesn't impact anything no matter the IOMMU is working in caching mode or not. Commit <29b32839725f> ("iommu/vt-d: Do not use flush-queue when caching-mode is on") has already disabled flush-queue for caching mode, hence caching mode will never callintel_flush_iotlb_all().Fixes bf92df30df90 ("intel-iommu: Only avoid flushing device IOTLB for domain ID 0 in caching mode")Signed-off-by Lu Baolu <[email protected]>--- drivers/iommu/intel/iommu.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)Change log:
v2:
- Squash two patches into a single one.
- No functionality changes.v1: https://lore.kernel.org/linux-iommu/[email protected]/
drivers/iommu/intel/iommu.c +1
-5@@ -1501,11 +1501,7 @@ static void iommu_flush_iotlb_psi(struct intel_iommu *iommu,else __iommu_flush_iotlb_psi (iommu, did, pfn, pages, ih); -/* -* In caching mode, changes of pages from non-present to present require -* flush. However, device IOTLB doesn't need to be flushed in this case. -*/ -if (!cap_caching_mode (iommu->cap) || !map) +if (!map)iommu_flush_dev_iotlb (domain, addr, mask); }
drivers/iommu/intel/iommu.c static void intel_flush_iotlb_all(struct iommu_domain *domain) · line 1578
Quoted 6 lines
drivers/iommu/intel/iommu.c +1
-2@@ -1579,8 +1575,7 @@ static void intel_flush_iotlb_all(struct iommu_domain *domain) iommu->flush.flush_iotlb (iommu, did,0 ,0 ,DMA_TLB_DSI_FLUSH ); -if (!cap_caching_mode (iommu->cap)) -iommu_flush_dev_iotlb (dmar_domain,0 ,MAX_AGAW_PFN_WIDTH ); +iommu_flush_dev_iotlb (dmar_domain,0 ,MAX_AGAW_PFN_WIDTH );
why not one more step to move it out of the iommu_array loop? There is no more reason to keep it in the loop.
Quoted 3 lines
drivers/iommu/intel/iommu.c +0
-0}if (dmar_domain->nested_parent)
Signature · 2 lines
Regards, Yi Liu
Baolu Lu
maintainerQuoted 12 lines · Yi Liu
Quoted 10 lines · Lu Baolu
The Caching Mode (CM) of the Intel IOMMU indicates if the hardware implementation caches not-present or erroneous translation-structure entries except the first-stage translation. The caching mode is irrelevant to the device TLB , therefore there is no need to check it before a device TLB invalidation operation.
iommu_flush_iotlb_psi()is called in map and unmap paths. The caching mode check before device TLB invalidation will cause device TLB invalidation always issued if IOMMU is not running in caching mode. This is wrong and causes unnecessary performance overhead.
Quoted 4 lines
I don't think the original code is wrong. As I replied before, if CM==0, the
iommu_flush_iotlb_psi()is only called in unmap path, in which the @map is false. [1] The reason to make the change is to make the logic simpler. 🙂
Oh, I see. There is a magic
if (cap_caching_mode(iommu->cap) && !domain->use_first_level)
iommu_flush_iotlb_psi(iommu, domain, pfn, pages, 0, 1);in __mapping_notify_one().
So if it's caching mode, then
- iommu_flush_iotlb_psi() will be called with @map=1 from __mapping_notify_one(), "!cap_caching_mode(iommu->cap) || !map" is not true, and device TLB is not invalidated.
- iommu_flush_iotlb_psi() will also be called with @map=0 from intel_iommu_tlb_sync(), device TLB is issued there.
That's the expected behavior for caching mode.
If it's not the caching mode, then
- iommu_flush_iotlb_psi() will be called with @map=0 from intel_iommu_tlb_sync(), device TLB is issued there.
That's also the expected behavior.
So the existing code is correct but obscure and difficult to understand, right? If so, we should make this patch as a cleanup rather than a fix.
Best regards,
baolu
Quoted 37 lines · Baolu Lu
Quoted 12 lines · Yi Liu
Quoted 10 lines · Lu Baolu
The Caching Mode (CM) of the Intel IOMMU indicates if the hardware implementation caches not-present or erroneous translation-structure entries except the first-stage translation. The caching mode is irrelevant to the device TLB , therefore there is no need to check it before a device TLB invalidation operation.
iommu_flush_iotlb_psi()is called in map and unmap paths. The caching mode check before device TLB invalidation will cause device TLB invalidation always issued if IOMMU is not running in caching mode. This is wrong and causes unnecessary performance overhead.Quoted 4 lines
I don't think the original code is wrong. As I replied before, if CM==0, the
iommu_flush_iotlb_psi()is only called in unmap path, in which the @map is false. [1] The reason to make the change is to make the logic simpler. 🙂Oh, I see. There is a magic
if (
cap_caching_mode(iommu->cap)&& !domain->use_first_level)iommu_flush_iotlb_psi(iommu, domain, pfn, pages, 0, 1);in
__mapping_notify_one().So if it's caching mode, then
-
iommu_flush_iotlb_psi()will be called with @map=1 from__mapping_notify_one(), "!cap_caching_mode(iommu->cap) || !map" is not true, and device TLB is not invalidated.
-iommu_flush_iotlb_psi()will also be called with @map=0 fromintel_iommu_tlb_sync(), device TLB is issued there.That's the expected behavior for caching mode.
If it's not the caching mode, then
Quoted 7 lines
-
iommu_flush_iotlb_psi()will be called with @map=0 fromintel_iommu_tlb_sync(), device TLB is issued there.That's also the expected behavior.
So the existing code is correct but obscure and difficult to understand, right? If so, we should make this patch as a cleanup rather than a fix.
aha, yes. As the below table, iommu_flush_iotlb_psi() does flush device TLB as expected. But there is a NA case. When CM==0, it should not be possible to call iommu_flush_iotlb_psi() with @map==1 as cache invalidation is not required when CM==0. So the existing code logic is really confusing, checking @map is enough and clearer. Since the old code works, so perhaps no fix tag is needed. :)
+----+------+-----------+------------+ | \ | | | | \ @map | | | | CM \ | 0 | 1 | | \ | | | +------+---+------------+------------+ | | | | | 0 | Y | NA | +----------+------------+------------+ | | | | | 1 | Y | N | +----------+------------+------------+
Y means flush dev-TLB please
N means no need to flush dev-TLB
NA means not applied
BTW. I think it is better to have the below change in a separate patch. The below change does fix a improper dev-TLB flushing behavior. Also how about Kevin's concern in the end of [1]. I didn't see your respond about it.
@@ -1579,8 +1575,7 @@ static void intel_flush_iotlb_all(struct iommu_domain *domain)
iommu->flush.flush_iotlb(iommu, did, 0, 0,
DMA_TLB_DSI_FLUSH);
- if (!cap_caching_mode(iommu->cap))
- iommu_flush_dev_iotlb(dmar_domain, 0, MAX_AGAW_PFN_WIDTH);
+ iommu_flush_dev_iotlb(dmar_domain, 0, MAX_AGAW_PFN_WIDTH);
}
if (dmar_domain->nested_parent)
Signature · 2 lines
Regards, Yi Liu
Baolu Lu
maintainerQuoted 68 lines · Yi Liu
Quoted 37 lines · Baolu Lu
Quoted 12 lines · Yi Liu
Quoted 10 lines · Lu Baolu
The Caching Mode (CM) of the Intel IOMMU indicates if the hardware implementation caches not-present or erroneous translation-structure entries except the first-stage translation. The caching mode is irrelevant to the device TLB , therefore there is no need to check it before a device TLB invalidation operation.
iommu_flush_iotlb_psi()is called in map and unmap paths. The caching mode check before device TLB invalidation will cause device TLB invalidation always issued if IOMMU is not running in caching mode. This is wrong and causes unnecessary performance overhead.Quoted 4 lines
I don't think the original code is wrong. As I replied before, if CM==0, the
iommu_flush_iotlb_psi()is only called in unmap path, in which the @map is false. [1] The reason to make the change is to make the logic simpler. 🙂Oh, I see. There is a magic
if (
cap_caching_mode(iommu->cap)&& !domain->use_first_level)iommu_flush_iotlb_psi(iommu, domain, pfn, pages, 0, 1);in
__mapping_notify_one().So if it's caching mode, then
-
iommu_flush_iotlb_psi()will be called with @map=1 from__mapping_notify_one(), "!cap_caching_mode(iommu->cap) || !map" is not true, and device TLB is not invalidated.
-iommu_flush_iotlb_psi()will also be called with @map=0 fromintel_iommu_tlb_sync(), device TLB is issued there.That's the expected behavior for caching mode.
If it's not the caching mode, then
Quoted 7 lines
-
iommu_flush_iotlb_psi()will be called with @map=0 fromintel_iommu_tlb_sync(), device TLB is issued there.That's also the expected behavior.
So the existing code is correct but obscure and difficult to understand, right? If so, we should make this patch as a cleanup rather than a fix.
aha, yes. As the below table,
iommu_flush_iotlb_psi()does flush device TLB as expected. But there is a NA case. When CM==0, it should not be possible to calliommu_flush_iotlb_psi()with @map==1 as cache invalidation is not required when CM==0. So the existing code logic is really confusing, checking @map is enough and clearer. Since the old code works, so perhaps no fix tag is needed. :)+----+------+-----------+------------+ | \ | | | | \ @map | | | | CM \ | 0 | 1 | | \ | | | +------+---+------------+------------+ | | | | | 0 | Y | NA | +----------+------------+------------+ | | | | | 1 | Y | N | +----------+------------+------------+
Quoted 3 lines
Y means flush dev-TLB please
N means no need to flush dev-TLB
NA means not applied
Yes. We have the same understanding now. :-)
Quoted 5 lines
BTW. I think it is better to have the below change in a separate patch. The below change does fix a improper dev-TLB flushing behavior. Also how about Kevin's concern in the end of [1]. I didn't see your respond about it.
I had an offline discussion with him and I included the conclusion in the commit message of this patch.
Best regards,
baolu
Yi Liu
Reviewed-byQuoted 69 lines · Baolu Lu
Quoted 68 lines · Yi Liu
Quoted 37 lines · Baolu Lu
Quoted 12 lines · Yi Liu
Quoted 10 lines · Lu Baolu
The Caching Mode (CM) of the Intel IOMMU indicates if the hardware implementation caches not-present or erroneous translation-structure entries except the first-stage translation. The caching mode is irrelevant to the device TLB , therefore there is no need to check it before a device TLB invalidation operation.
iommu_flush_iotlb_psi()is called in map and unmap paths. The caching mode check before device TLB invalidation will cause device TLB invalidation always issued if IOMMU is not running in caching mode. This is wrong and causes unnecessary performance overhead.Quoted 4 lines
I don't think the original code is wrong. As I replied before, if CM==0, the
iommu_flush_iotlb_psi()is only called in unmap path, in which the @map is false. [1] The reason to make the change is to make the logic simpler. 🙂Oh, I see. There is a magic
if (
cap_caching_mode(iommu->cap)&& !domain->use_first_level)iommu_flush_iotlb_psi(iommu, domain, pfn, pages, 0, 1);in
__mapping_notify_one().So if it's caching mode, then
-
iommu_flush_iotlb_psi()will be called with @map=1 from__mapping_notify_one(), "!cap_caching_mode(iommu->cap) || !map" is not true, and device TLB is not invalidated.
-iommu_flush_iotlb_psi()will also be called with @map=0 fromintel_iommu_tlb_sync(), device TLB is issued there.That's the expected behavior for caching mode.
If it's not the caching mode, then
Quoted 7 lines
-
iommu_flush_iotlb_psi()will be called with @map=0 fromintel_iommu_tlb_sync(), device TLB is issued there.That's also the expected behavior.
So the existing code is correct but obscure and difficult to understand, right? If so, we should make this patch as a cleanup rather than a fix.
aha, yes. As the below table,
iommu_flush_iotlb_psi()does flush device TLB as expected. But there is a NA case. When CM==0, it should not be possible to calliommu_flush_iotlb_psi()with @map==1 as cache invalidation is not required when CM==0. So the existing code logic is really confusing, checking @map is enough and clearer. Since the old code works, so perhaps no fix tag is needed. :)+----+------+-----------+------------+ | \ | | | | \ @map | | | | CM \ | 0 | 1 | | \ | | | +------+---+------------+------------+ | | | | | 0 | Y | NA | +----------+------------+------------+ | | | | | 1 | Y | N | +----------+------------+------------+
Quoted 5 lines
Quoted 3 lines
Y means flush dev-TLB please
N means no need to flush dev-TLB
NA means not appliedYes. We have the same understanding now. :-)
yeah, would be helpful to refine the commit message w.r.t. this change.
Quoted 8 lines
Quoted 5 lines
BTW. I think it is better to have the below change in a separate patch. The below change does fix a improper dev-TLB flushing behavior. Also how about Kevin's concern in the end of [1]. I didn't see your respond about it.
I had an offline discussion with him and I included the conclusion in the commit message of this patch.
ok. So the key reason is that intel_flush_iotlb_all() is not called at all if CM==1 after that commit. So there is nothing fixed by this change. right? This just makes the logic cleaner. Perhaps no need for a fix tag.
Signature · 2 lines
Regards, Yi Liu
Robin Murphy
reviewerQuoted 11 lines · Lu Baolu
The Caching Mode (CM) of the Intel IOMMU indicates if the hardware implementation caches not-present or erroneous translation-structure entries except the first-stage translation. The caching mode is irrelevant to the device TLB , therefore there is no need to check it before a device TLB invalidation operation.
iommu_flush_iotlb_psi()is called in map and unmap paths. The caching mode check before device TLB invalidation will cause device TLB invalidation always issued if IOMMU is not running in caching mode. This is wrong and causes unnecessary performance overhead.
Quoted 5 lines
The removal of caching mode check in
intel_flush_iotlb_all()doesn't impact anything no matter the IOMMU is working in caching mode or not. Commit <29b32839725f> ("iommu/vt-d: Do not use flush-queue when caching-mode is on") has already disabled flush-queue for caching mode, hence caching mode will never callintel_flush_iotlb_all().
Well, technically it might still, at domain creation via iommu_create_device_direct_mappings(), but domain->has_iotlb_device should definitely be false at that point :)
Cheers,
Robin.
Quoted 41 lines
Fixes bf92df30df90 ("intel-iommu: Only avoid flushing device IOTLB for domain ID 0 in caching mode")Signed-off-by Lu Baolu <[email protected]>--- drivers/iommu/intel/iommu.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)Change log:
v2:
- Squash two patches into a single one.
- No functionality changes.v1: https://lore.kernel.org/linux-iommu/[email protected]/
drivers/iommu/intel/iommu.c +2
-7@@ -1501,11 +1501,7 @@ static void iommu_flush_iotlb_psi(struct intel_iommu *iommu,else __iommu_flush_iotlb_psi (iommu, did, pfn, pages, ih); -/* -* In caching mode, changes of pages from non-present to present require -* flush. However, device IOTLB doesn't need to be flushed in this case. -*/ -if (!cap_caching_mode (iommu->cap) || !map) +if (!map)iommu_flush_dev_iotlb (domain, addr, mask); }@@ -1579,8 +1575,7 @@ static void intel_flush_iotlb_all(struct iommu_domain *domain) iommu->flush.flush_iotlb (iommu, did,0 ,0 ,DMA_TLB_DSI_FLUSH ); -if (!cap_caching_mode (iommu->cap)) -iommu_flush_dev_iotlb (dmar_domain,0 ,MAX_AGAW_PFN_WIDTH ); +iommu_flush_dev_iotlb (dmar_domain,0 ,MAX_AGAW_PFN_WIDTH ); }if (dmar_domain->nested_parent)
Baolu Lu
maintainerQuoted 12 lines · Robin Murphy
Quoted 11 lines · Lu Baolu
The Caching Mode (CM) of the Intel IOMMU indicates if the hardware implementation caches not-present or erroneous translation-structure entries except the first-stage translation. The caching mode is irrelevant to the device TLB , therefore there is no need to check it before a device TLB invalidation operation.
iommu_flush_iotlb_psi()is called in map and unmap paths. The caching mode check before device TLB invalidation will cause device TLB invalidation always issued if IOMMU is not running in caching mode. This is wrong and causes unnecessary performance overhead.
Quoted 7 lines
Quoted 5 lines
The removal of caching mode check in
intel_flush_iotlb_all()doesn't impact anything no matter the IOMMU is working in caching mode or not. Commit <29b32839725f> ("iommu/vt-d: Do not use flush-queue when caching-mode is on") has already disabled flush-queue for caching mode, hence caching mode will never callintel_flush_iotlb_all().Well, technically it might still, at domain creation via
iommu_create_device_direct_mappings(), butdomain->has_iotlb_deviceshould definitely be false at that point 🙂
Oh! I overlooked that path. :-)
Yes. iommu_create_device_direct_mappings() is called before setting the domain to device for intel iommu driver, hence in practice the domain->has_iotlb_device is always false.
Best regards,
baolu