iommu/vt-d: Fix potential lockup if qi_submit_sync called with 0 count
Subsystems (2)
PATCH v2 1/1 iommu/vt-d: Fix potential lockup if qi_submit_sync called with 0 count
From: Sanjay K Kumar <[email protected]>
If qi_submit_sync() is invoked with 0 invalidation descriptors (for instance, for DMA draining purposes), we can run into a bug where a submitting thread fails to detect the completion of invalidation_wait. Subsequently, this led to a soft lockup. Currently, there is no impact by this bug on the existing users because no callers are submitting invalidations with 0 descriptors. This fix will enable future users (such as DMA drain) calling qi_submit_sync() with 0 count.
Suppose thread T1 invokes qi_submit_sync() with non-zero descriptors, while concurrently, thread T2 calls qi_submit_sync() with zero descriptors. Both threads then enter a while loop, waiting for their respective descriptors to complete. T1 detects its completion (i.e., T1's invalidation_wait status changes to QI_DONE by HW) and proceeds to call reclaim_free_desc() to reclaim all descriptors, potentially including adjacent ones of other threads that are also marked as QI_DONE.
During this time, while T2 is waiting to acquire the qi->q_lock, the IOMMU hardware may complete the invalidation for T2, setting its status to QI_DONE. However, if T1's execution of reclaim_free_desc() frees T2's invalidation_wait descriptor and changes its status to QI_FREE, T2 will not observe the QI_DONE status for its invalidation_wait and will indefinitely remain stuck.
This soft lockup does not occur when only non-zero descriptors are submitted.In such cases, invalidation descriptors are interspersed among wait descriptors with the status QI_IN_USE, acting as barriers. These barriers prevent the reclaim code from mistakenly freeing descriptors belonging to other submitters.
Considered the following example timeline:
T1 T2 ======================================== ID1 WD1 while(WD1!=QI_DONE) unlock lock WD1=QI_DONE* WD2 while(WD2!=QI_DONE) unlock lock WD1==QI_DONE? ID1=QI_DONE WD2=DONE* reclaim() ID1=FREE WD1=FREE WD2=FREE unlock soft lockup! T2 never sees QI_DONE in WD2
Where:
ID = invalidation descriptor
WD = wait descriptor
* Written by hardware
The root of the problem is that the descriptor status QI_DONE flag is used for two conflicting purposes:
1. signal a descriptor is ready for reclaim (to be freed)
2. signal by the hardware that a wait descriptor is complete
The solution (in this patch) is state separation by using QI_FREE flag for #1.
Once a thread's invalidation descriptors are complete, their status would be set to QI_FREE. The reclaim_free_desc() function would then only free descriptors marked as QI_FREE instead of those marked as QI_DONE. This change ensures that T2 (from the previous example) will correctly observe the completion of its invalidation_wait (marked as QI_DONE).
---
v2: (Kevin)
- Reuse QI_FREE flag instead of a new QI_TO_BE_FREED flag
- Clarify the impact of the bug on existing users
--- drivers/iommu/intel/dmar.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)
drivers/iommu/intel/dmar.c +11 -5
@@ -1204,9 +1204,7 @@ static void free_iommu(struct intel_iommu *iommu)*/ static inline void reclaim_free_desc (struct q_inval *qi) { -while (qi->desc_status[qi->free_tail] ==QI_DONE || - qi->desc_status[qi->free_tail] ==QI_ABORT ) { - qi->desc_status[qi->free_tail] =QI_FREE ; +while (qi->desc_status[qi->free_tail] ==QI_FREE && qi->free_tail != qi->free_head) { qi->free_tail = (qi->free_tail +1 ) %QI_LENGTH ; qi->free_cnt++; }
@@ -1463,8 +1461,16 @@ int qi_submit_sync(struct intel_iommu *iommu, struct qi_desc *desc,raw_spin_lock (&qi->q_lock); } -for (i =0 ; i < count; i++) - qi->desc_status[(index + i) %QI_LENGTH ] =QI_DONE ; +/* +* The reclaim code can free descriptors from multiple submissions +* starting from the tail of the queue. When count == 0, the +* status of the standalone wait descriptor at the tail of the queue +* must be set to QI_FREE to allow the reclaim code to proceed. +* It is also possible that descriptors from one of the previous +* submissions has to be reclaimed by a subsequent submission. +*/ +for (i =0 ; i <= count; i++) + qi->desc_status[(index + i) %QI_LENGTH ] =QI_FREE ;reclaim_free_desc (qi);raw_spin_unlock_irqrestore (&qi->q_lock, flags);
Signature · 1 line
2.25.1
Quoted 76 lines
From: Jacob Pan <[email protected]>
Sent: Monday, July 29, 2024 5:01 AMFrom: Sanjay K Kumar <[email protected]>
If
qi_submit_sync()is invoked with 0 invalidation descriptors (for instance, for DMA draining purposes), we can run into a bug where a submitting thread fails to detect the completion ofinvalidation_wait. Subsequently, this led to a soft lockup. Currently, there is no impact by this bug on the existing users because no callers are submitting invalidations with 0 descriptors. This fix will enable future users (such as DMA drain) callingqi_submit_sync()with 0 count.Suppose thread T1 invokes
qi_submit_sync()with non-zero descriptors, while
concurrently, thread T2 callsqi_submit_sync()with zero descriptors. Both threads then enter a while loop, waiting for their respective descriptors to complete. T1 detects its completion (i.e., T1'sinvalidation_waitstatus changes toQI_DONEby HW) and proceeds to callreclaim_free_desc()to reclaim all descriptors, potentially including adjacent ones of other threads that are also marked asQI_DONE.During this time, while T2 is waiting to acquire the
qi->q_lock, the IOMMU hardware may complete the invalidation for T2, setting its status toQI_DONE. However, if T1's execution ofreclaim_free_desc()frees T2'sinvalidation_waitdescriptor and changes its status toQI_FREE, T2 will not observe theQI_DONEstatus for itsinvalidation_waitand will indefinitely remain stuck.This soft lockup does not occur when only non-zero descriptors are submitted.In such cases, invalidation descriptors are interspersed among wait descriptors with the status
QI_IN_USE, acting as barriers. These barriers prevent the reclaim code from mistakenly freeing descriptors belonging to other submitters.Considered the following example timeline:
T1 T2 ======================================== ID1 WD1 while(WD1!=QI_DONE) unlock lock WD1=QI_DONE* WD2 while(WD2!=QI_DONE) unlock lock WD1==QI_DONE? ID1=QI_DONE WD2=DONE* reclaim() ID1=FREE WD1=FREE WD2=FREE unlock soft lockup! T2 never sees QI_DONE in WD2Where:
ID = invalidation descriptor
WD = wait descriptor
* Written by hardwareThe root of the problem is that the descriptor status
QI_DONEflag is used for two conflicting purposes:
1. signal a descriptor is ready for reclaim (to be freed)
2. signal by the hardware that a wait descriptor is completeThe solution (in this patch) is state separation by using
QI_FREEflag for #1.Once a thread's invalidation descriptors are complete, their status would be set to
QI_FREE. Thereclaim_free_desc()function would then only free descriptors marked asQI_FREEinstead of those marked asQI_DONE. This change ensures that T2 (from the previous example) will correctly observe the completion of itsinvalidation_wait(marked asQI_DONE).
Quoted 3 lines
Signed-off-by Sanjay K Kumar <[email protected]>Signed-off-by Jacob Pan <[email protected]>
Baolu Lu
maintainerQuoted 72 lines · Jacob Pan
From: Sanjay K Kumar<[email protected]>
If
qi_submit_sync()is invoked with 0 invalidation descriptors (for instance, for DMA draining purposes), we can run into a bug where a submitting thread fails to detect the completion ofinvalidation_wait. Subsequently, this led to a soft lockup. Currently, there is no impact by this bug on the existing users because no callers are submitting invalidations with 0 descriptors. This fix will enable future users (such as DMA drain) callingqi_submit_sync()with 0 count.Suppose thread T1 invokes
qi_submit_sync()with non-zero descriptors, while concurrently, thread T2 callsqi_submit_sync()with zero descriptors. Both threads then enter a while loop, waiting for their respective descriptors to complete. T1 detects its completion (i.e., T1'sinvalidation_waitstatus changes toQI_DONEby HW) and proceeds to callreclaim_free_desc()to reclaim all descriptors, potentially including adjacent ones of other threads that are also marked asQI_DONE.During this time, while T2 is waiting to acquire the
qi->q_lock, the IOMMU hardware may complete the invalidation for T2, setting its status toQI_DONE. However, if T1's execution ofreclaim_free_desc()frees T2'sinvalidation_waitdescriptor and changes its status toQI_FREE, T2 will not observe theQI_DONEstatus for itsinvalidation_waitand will indefinitely remain stuck.This soft lockup does not occur when only non-zero descriptors are submitted.In such cases, invalidation descriptors are interspersed among wait descriptors with the status
QI_IN_USE, acting as barriers. These barriers prevent the reclaim code from mistakenly freeing descriptors belonging to other submitters.Considered the following example timeline:
T1 T2 ======================================== ID1 WD1 while(WD1!=QI_DONE) unlock lock WD1=QI_DONE* WD2 while(WD2!=QI_DONE) unlock lock WD1==QI_DONE? ID1=QI_DONE WD2=DONE* reclaim() ID1=FREE WD1=FREE WD2=FREE unlock soft lockup! T2 never sees QI_DONE in WD2Where:
ID = invalidation descriptor
WD = wait descriptor
* Written by hardwareThe root of the problem is that the descriptor status
QI_DONEflag is used for two conflicting purposes:
1. signal a descriptor is ready for reclaim (to be freed)
2. signal by the hardware that a wait descriptor is completeThe solution (in this patch) is state separation by using
QI_FREEflag for #1.Once a thread's invalidation descriptors are complete, their status would be set to
QI_FREE. Thereclaim_free_desc()function would then only free descriptors marked asQI_FREEinstead of those marked asQI_DONE. This change ensures that T2 (from the previous example) will correctly observe the completion of itsinvalidation_wait(marked asQI_DONE).
Quoted 2 lines
Signed-off-by Sanjay K Kumar<[email protected]>Signed-off-by Jacob Pan<[email protected]>
Queued for v6.12-rc1.
Thanks,
baolu