iommu/vt-d: Fix potential soft lockup due to reclaim
Subsystems (2)
PATCH 1/1 iommu/vt-d: Fix potential soft lockup due to reclaim
From: Sanjay K Kumar <[email protected]>
If qi_submit_sync() is invoked with 0 invalidation descriptors (for instance, for DMA draining purposes), we can run into a bug where a submitting thread fails to detect the completion of invalidation_wait. Subsequently, this led to a soft lockup.
Suppose thread T1 invokes qi_submit_sync() with non-zero descriptors, while concurrently, thread T2 calls qi_submit_sync() with zero descriptors. Both threads then enter a while loop, waiting for their respective descriptors to complete. T1 detects its completion (i.e., T1's invalidation_wait status changes to QI_DONE by HW) and proceeds to call reclaim_free_desc() to reclaim all descriptors, potentially including adjacent ones of other threads that are also marked as QI_DONE.
During this time, while T2 is waiting to acquire the qi->q_lock, the IOMMU hardware may complete the invalidation for T2, setting its status to QI_DONE. However, if T1's execution of reclaim_free_desc() frees T2's invalidation_wait descriptor and changes its status to QI_FREE, T2 will not observe the QI_DONE status for its invalidation_wait and will indefinitely remain stuck.
This soft lockup does not occur when only non-zero descriptors are submitted.In such cases, invalidation descriptors are interspersed among wait descriptors with the status QI_IN_USE, acting as barriers. These barriers prevent the reclaim code from mistakenly freeing descriptors belonging to other submitters.
Considered the following example timeline:
T1 T2 ======================================== ID1 WD1 while(WD1!=QI_DONE) unlock lock WD1=QI_DONE* WD2 while(WD2!=QI_DONE) unlock lock WD1==QI_DONE? ID1=QI_DONE WD2=DONE* reclaim() ID1=FREE WD1=FREE WD2=FREE unlock soft lockup! T2 never sees QI_DONE in WD2
Where:
ID = invalidation descriptor
WD = wait descriptor
* Written by hardware
The root of the problem is that the descriptor status QI_DONE flag is used for two conflicting purposes:
1. signal a descriptor is ready for reclaim (to be freed)
2. signal by the hardware that a wait descriptor is complete
The solution (in this patch) is state separation by introducing a new flag for the descriptors called QI_TO_BE_FREED.
Once a thread's invalidation descriptors are complete, their status would be set to QI_TO_BE_FREED. The reclaim_free_desc() function would then only free descriptors marked as QI_TO_BE_FREED instead of those marked as QI_DONE. This change ensures that T2 (from the previous example) will correctly observe the completion of its invalidation_wait (marked as QI_DONE).
Currently, there is no impact by this bug on the existing users because no callers are submitting invalidations with 0 descriptors.
--- drivers/iommu/intel/dmar.c | 13 +++++++++---- drivers/iommu/intel/iommu.h | 3 ++- 2 files changed, 11 insertions(+), 5 deletions(-)
drivers/iommu/intel/dmar.c +9 -4
@@ -1204,8 +1204,7 @@ static void free_iommu(struct intel_iommu *iommu)*/ static inline void reclaim_free_desc (struct q_inval *qi) { -while (qi->desc_status[qi->free_tail] ==QI_DONE || - qi->desc_status[qi->free_tail] ==QI_ABORT ) { +while (qi->desc_status[qi->free_tail] ==QI_TO_BE_FREED ) { qi->desc_status[qi->free_tail] =QI_FREE ; qi->free_tail = (qi->free_tail +1 ) %QI_LENGTH ; qi->free_cnt++;
@@ -1463,8 +1462,14 @@ int qi_submit_sync(struct intel_iommu *iommu, struct qi_desc *desc,raw_spin_lock (&qi->q_lock); } -for (i =0 ; i < count; i++) - qi->desc_status[(index + i) %QI_LENGTH ] =QI_DONE ; +/* +* The reclaim code can free descriptors from multiple submissions +* starting from the tail of the queue. When count == 0, the +* status of the standalone wait descriptor at the tail of the queue +* must be set to QI_TO_BE_FREED to allow the reclaim code to proceed. +*/ +for (i =0 ; i <= count; i++) + qi->desc_status[(index + i) %QI_LENGTH ] =QI_TO_BE_FREED ;reclaim_free_desc (qi);raw_spin_unlock_irqrestore (&qi->q_lock, flags);
drivers/iommu/intel/iommu.h +2 -1
@@ -382,7 +382,8 @@ enum {QI_FREE ,QI_IN_USE ,QI_DONE , - QI_ABORT +QI_ABORT , + QI_TO_BE_FREED };#define QI_CC_TYPE 0x1
Signature · 1 line
2.25.1
Quoted 74 lines
From: Jacob Pan <[email protected]>
Sent: Saturday, July 20, 2024 2:17 AMFrom: Sanjay K Kumar <[email protected]>
If
qi_submit_sync()is invoked with 0 invalidation descriptors (for instance, for DMA draining purposes), we can run into a bug where a submitting thread fails to detect the completion ofinvalidation_wait. Subsequently, this led to a soft lockup.Suppose thread T1 invokes
qi_submit_sync()with non-zero descriptors, while
concurrently, thread T2 callsqi_submit_sync()with zero descriptors. Both threads then enter a while loop, waiting for their respective descriptors to complete. T1 detects its completion (i.e., T1'sinvalidation_waitstatus changes toQI_DONEby HW) and proceeds to callreclaim_free_desc()to reclaim all descriptors, potentially including adjacent ones of other threads that are also marked asQI_DONE.During this time, while T2 is waiting to acquire the
qi->q_lock, the IOMMU hardware may complete the invalidation for T2, setting its status toQI_DONE. However, if T1's execution ofreclaim_free_desc()frees T2'sinvalidation_waitdescriptor and changes its status toQI_FREE, T2 will not observe theQI_DONEstatus for itsinvalidation_waitand will indefinitely remain stuck.This soft lockup does not occur when only non-zero descriptors are submitted.In such cases, invalidation descriptors are interspersed among wait descriptors with the status
QI_IN_USE, acting as barriers. These barriers prevent the reclaim code from mistakenly freeing descriptors belonging to other submitters.Considered the following example timeline:
T1 T2 ======================================== ID1 WD1 while(WD1!=QI_DONE) unlock lock WD1=QI_DONE* WD2 while(WD2!=QI_DONE) unlock lock WD1==QI_DONE? ID1=QI_DONE WD2=DONE* reclaim() ID1=FREE WD1=FREE WD2=FREE unlock soft lockup! T2 never sees QI_DONE in WD2Where:
ID = invalidation descriptor
WD = wait descriptor
* Written by hardwareThe root of the problem is that the descriptor status
QI_DONEflag is used for two conflicting purposes:
1. signal a descriptor is ready for reclaim (to be freed)
2. signal by the hardware that a wait descriptor is completeThe solution (in this patch) is state separation by introducing a new flag for the descriptors called
QI_TO_BE_FREED.Once a thread's invalidation descriptors are complete, their status would be set to
QI_TO_BE_FREED. Thereclaim_free_desc()function would then only
free descriptors marked asQI_TO_BE_FREEDinstead of those marked asQI_DONE. This change ensures that T2 (from the previous example) will correctly observe the completion of itsinvalidation_wait(marked asQI_DONE).
Quoted 2 lines
Currently, there is no impact by this bug on the existing users because no callers are submitting invalidations with 0 descriptors.
bug fix is for existing users. Please revise the subject line and this msg to make it clear that it's for preparation of a new usage.
Quoted 36 lines
Signed-off-by Sanjay K Kumar <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- drivers/iommu/intel/dmar.c | 13 +++++++++---- drivers/iommu/intel/iommu.h | 3 ++- 2 files changed, 11 insertions(+), 5 deletions(-)drivers/iommu/intel/dmar.c +1
-2@@ -1204,8 +1204,7 @@ static void free_iommu(struct intel_iommu*iommu) */ static inline void reclaim_free_desc (struct q_inval *qi) { -while (qi->desc_status[qi->free_tail] ==QI_DONE || - qi->desc_status[qi->free_tail] ==QI_ABORT ) { +while (qi->desc_status[qi->free_tail] ==QI_TO_BE_FREED ) { qi->desc_status[qi->free_tail] =QI_FREE ; qi->free_tail = (qi->free_tail +1 ) %QI_LENGTH ;drivers/iommu/intel/dmar.c +5
-2qi->free_cnt++;@@ -1463,8 +1462,14 @@ int qi_submit_sync(struct intel_iommu *iommu,struct qi_desc *desc, raw_spin_lock(&qi->q_lock); } -for (i =0 ; i < count; i++) - qi->desc_status[(index + i) %QI_LENGTH ] =QI_DONE ; +/* +* The reclaim code can free descriptors from multiple submissions +* starting from the tail of the queue. When count == 0, the +* status of the standalone wait descriptor at the tail of the queue +* must be set to QI_TO_BE_FREED to allow the reclaim code to proceed.
drivers/iommu/intel/dmar.c int qi_submit_sync(struct intel_iommu *iommu, struct qi_desc *desc, · line 1472
Quoted 3 lines
drivers/iommu/intel/dmar.c +3
-0+*/ +for (i =0 ; i <= count; i++) + qi->desc_status[(index + i) %QI_LENGTH ] =QI_TO_BE_FREED ;
We don't really need a new flag. Just set them to QI_FREE and then reclaim QI_FREE slots until hitting qi->head in reclaim_free_desc().
Quoted 20 lines
drivers/iommu/intel/dmar.c +0
-0reclaim_free_desc (qi);raw_spin_unlock_irqrestore (&qi->q_lock, flags);drivers/iommu/intel/iommu.h +2
-2@@ -382,7 +382,8 @@ enum {QI_FREE ,QI_IN_USE ,QI_DONE , -QI_ABORT +QI_ABORT , + QI_TO_BE_FREED };#define QI_CC_TYPE 0x1 --2.25.1
Quoted 75 lines · Kevin
Quoted 75 lines
From: Jacob Pan <[email protected]>
Sent: Saturday, July 20, 2024 2:17 AMFrom: Sanjay K Kumar <[email protected]>
If
qi_submit_sync()is invoked with 0 invalidation descriptors (for instance, for DMA draining purposes), we can run into a bug where a submitting thread fails to detect the completion ofinvalidation_wait. Subsequently, this led to a soft lockup.Suppose thread T1 invokes
qi_submit_sync()with non-zero descriptors, while
concurrently, thread T2 callsqi_submit_sync()with zero descriptors. Both threads then enter a while loop, waiting for their respective descriptors to complete. T1 detects its completion (i.e., T1'sinvalidation_waitstatus changes toQI_DONEby HW) and proceeds to callreclaim_free_desc()to reclaim all descriptors, potentially including adjacent ones of other threads that are also marked asQI_DONE.During this time, while T2 is waiting to acquire the
qi->q_lock, the IOMMU hardware may complete the invalidation for T2, setting its status toQI_DONE. However, if T1's execution ofreclaim_free_desc()frees T2'sinvalidation_waitdescriptor and changes its status toQI_FREE, T2 will not observe theQI_DONEstatus for itsinvalidation_waitand will indefinitely remain stuck.This soft lockup does not occur when only non-zero descriptors are submitted.In such cases, invalidation descriptors are interspersed among wait descriptors with the status
QI_IN_USE, acting as barriers. These barriers prevent the reclaim code from mistakenly freeing descriptors belonging to other submitters.Considered the following example timeline:
T1 T2 ======================================== ID1 WD1 while(WD1!=QI_DONE) unlock lock WD1=QI_DONE* WD2 while(WD2!=QI_DONE) unlock lock WD1==QI_DONE? ID1=QI_DONE WD2=DONE* reclaim() ID1=FREE WD1=FREE WD2=FREE unlock soft lockup! T2 never sees QI_DONE inWD2
Where:
ID = invalidation descriptor
WD = wait descriptor
* Written by hardwareThe root of the problem is that the descriptor status
QI_DONEflag is used for two conflicting purposes:
1. signal a descriptor is ready for reclaim (to be freed)
2. signal by the hardware that a wait descriptor is completeThe solution (in this patch) is state separation by introducing a new flag for the descriptors called
QI_TO_BE_FREED.Once a thread's invalidation descriptors are complete, their status would be set to
QI_TO_BE_FREED. Thereclaim_free_desc()function would then only
free descriptors marked asQI_TO_BE_FREEDinstead of those marked asQI_DONE. This change ensures that T2 (from the previous example) will correctly observe the completion of itsinvalidation_wait(marked asQI_DONE).
Quoted 5 lines
Quoted 2 lines
Currently, there is no impact by this bug on the existing users because no callers are submitting invalidations with 0 descriptors.
bug fix is for existing users. Please revise the subject line and this msg to make it clear that it's for preparation of a new usage.
The bug is in the qi_submit_sync function itself since it permits callers to give 0 as count. It is a bug regardless of users.
I put "potential" in the subject line to indicate, perhaps it is too vague. How about just stating what it is fixing:
"Fix potential lockup if qi_submit_sync called with 0 count"
Also change this paragraph to:
"Currently, there is no impact by this bug on the existing users because
no callers are submitting invalidations with 0 descriptors. This fix will enable future users (such as DMA drain) calling qi_submit_sync() with 0 count."
Quoted 37 lines
Quoted 37 lines
Signed-off-by Sanjay K Kumar <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- drivers/iommu/intel/dmar.c | 13 +++++++++---- drivers/iommu/intel/iommu.h | 3 ++- 2 files changed, 11 insertions(+), 5 deletions(-)drivers/iommu/intel/dmar.c +1
-2@@ -1204,8 +1204,7 @@ static void free_iommu(struct intel_iommu*iommu) */ static inline void reclaim_free_desc (struct q_inval *qi) { -while (qi->desc_status[qi->free_tail] ==QI_DONE || - qi->desc_status[qi->free_tail] ==QI_ABORT ) { +while (qi->desc_status[qi->free_tail] ==QI_TO_BE_FREED ) { qi->desc_status[qi->free_tail] =QI_FREE ; qi->free_tail = (qi->free_tail +1 ) %QI_LENGTH ;drivers/iommu/intel/dmar.c +2
-2qi->free_cnt++;@@ -1463,8 +1462,14 @@ int qi_submit_sync(struct intel_iommu *iommu,struct qi_desc *desc, raw_spin_lock(&qi->q_lock); } -for (i =0 ; i < count; i++) - qi->desc_status[(index + i) %QI_LENGTH ] =QI_DONE ; +/* +* The reclaim code can free descriptors from multiple submissions
drivers/iommu/intel/dmar.c +2
-0+* starting from the tail of the queue. When count == 0, the +* status of the standalone wait descriptor at the tail of the queue
drivers/iommu/intel/dmar.c +1
-0+* must be set to QI_TO_BE_FREED to allow the reclaim code to
Quoted 8 lines
Quoted 5 lines
proceed.
drivers/iommu/intel/dmar.c +3
-0+*/ +for (i =0 ; i <= count; i++) + qi->desc_status[(index + i) % QI_LENGTH] =
QI_TO_BE_FREED;We don't really need a new flag. Just set them to
QI_FREEand then reclaimQI_FREEslots until hittingqi->headinreclaim_free_desc().
We do need to have a separate state for descriptors pending to be freed. Otherwise, reclaim code will advance pass the intended range.
Quoted 16 lines
Quoted 16 lines
drivers/iommu/intel/dmar.c +0
-0reclaim_free_desc (qi);raw_spin_unlock_irqrestore (&qi->q_lock, flags);drivers/iommu/intel/iommu.h +2
-1@@ -382,7 +382,8 @@ enum {QI_FREE ,QI_IN_USE ,QI_DONE , - QI_ABORT +QI_ABORT , + QI_TO_BE_FREED };
Quoted 5 lines
Quoted 4 lines
drivers/iommu/intel/iommu.h +0
-1#define QI_CC_TYPE 0x1 --2.25.1
Thanks,
Jacob
Quoted 21 lines
From: Jacob Pan <[email protected]>
Sent: Thursday, July 25, 2024 12:26 AMQuoted 8 lines · Kevin
Quoted 5 lines
From: Jacob Pan <[email protected]>
Sent: Saturday, July 20, 2024 2:17 AMCurrently, there is no impact by this bug on the existing users because no callers are submitting invalidations with 0 descriptors.
bug fix is for existing users. Please revise the subject line and this msg to make it clear that it's for preparation of a new usage.
The bug is in the
qi_submit_syncfunction itself since it permits callers to give 0 as count. It is a bug regardless of users.I put "potential" in the subject line to indicate, perhaps it is too vague. How about just stating what it is fixing:
"Fix potential lockup ifqi_submit_synccalled with 0 count"
Quoted 5 lines
Also change this paragraph to:
"Currently, there is no impact by this bug on the existing users because
no callers are submitting invalidations with 0 descriptors. This fix will enable future users (such as DMA drain) callingqi_submit_sync()with 0 count."
Then please move it to the start.
Quoted 23 lines
Quoted 1 line
Review int qi_submit_sync(struct intel_iommu
Quoted 1 line
(quoted diff) +0
-0@@ -1463,8 +1462,14 @@ int qi_submit_sync(struct intel_iommu*iommu,
Quoted 13 lines
Quoted 13 lines
drivers/iommu/intel/dmar.c +2
-2struct qi_desc *desc, raw_spin_lock(&qi->q_lock); } -for (i =0 ; i < count; i++) - qi->desc_status[(index + i) %QI_LENGTH ] =QI_DONE ; +/* +* The reclaim code can free descriptors from multiple submissions
(quoted diff) +2
-0+ * starting from the tail of the queue. When count == 0, the + * status of the standalone wait descriptor at the tail of thequeue
(quoted diff) +1
-0+ * must be set to QI_TO_BE_FREED to allow the reclaim code toQuoted 8 lines
Quoted 5 lines
proceed.
drivers/iommu/intel/dmar.c +3
-0+*/ +for (i =0 ; i <= count; i++) + qi->desc_status[(index + i) % QI_LENGTH] =
QI_TO_BE_FREED;We don't really need a new flag. Just set them to
QI_FREEand then reclaimQI_FREEslots until hittingqi->headinreclaim_free_desc().
Quoted 3 lines
We do need to have a separate state for descriptors pending to be freed. Otherwise, reclaim code will advance pass the intended range.
The commit msg said that QI_DONE is currently used for conflicting purpose.
Using QI_FREE keeps it only for signaling that a wait desc is completed.
The key is that reclaim() should not change a desc's state before it's consumed by the owner. Instead we always let the owner to change the state and reclaim() only does scan and adjust the tracking fields then such race condition disappears.
In this example T2's slots are changed to QI_FREE by T2 after it completes all the checks. Only at this point those slots can be reclaimed.
Quoted 63 lines · Kevin
Quoted 22 lines
From: Jacob Pan <[email protected]>
Sent: Thursday, July 25, 2024 12:26 AMQuoted 9 lines · Kevin
Quoted 6 lines
From: Jacob Pan <[email protected]>
Sent: Saturday, July 20, 2024 2:17 AMCurrently, there is no impact by this bug on the existing users because no callers are submitting invalidations with 0 descriptors.
bug fix is for existing users. Please revise the subject line and this msg to make it clear that it's for preparation of a new usage.
The bug is in the
qi_submit_syncfunction itself since it permits callers to give 0 as count. It is a bug regardless of users.I put "potential" in the subject line to indicate, perhaps it is too vague. How about just stating what it is fixing:
"Fix potential lockup ifqi_submit_synccalled with 0 count"Quoted 5 lines
Also change this paragraph to:
"Currently, there is no impact by this bug on the existing users because
no callers are submitting invalidations with 0 descriptors. This fix
will enable future users (such as DMA drain) callingqi_submit_sync()with 0 count."Then please move it to the start.
Quoted 24 lines
Quoted 1 line
Review int qi_submit_sync(struct intel_iommu
Quoted 1 line
(quoted diff) +0
-0@@ -1463,8 +1462,14 @@ int qi_submit_sync(struct intel_iommu*iommu,
Quoted 14 lines
Quoted 14 lines
(quoted diff) +2
-2struct qi_desc *desc, raw_spin_lock(&qi->q_lock); } - for (i = 0; i < count; i++) - qi->desc_status[(index + i) % QI_LENGTH] = QI_DONE; + /* + * The reclaim code can free descriptors from multiplesubmissions
(quoted diff) +1
-0+ * starting from the tail of the queue. When count == 0,the
(quoted diff) +1
-0+ * status of the standalone wait descriptor at the tail ofthe queue
(quoted diff) +1
-0+ * must be set to QI_TO_BE_FREED to allow the reclaim codeQuoted 8 lines
Quoted 5 lines
to proceed.
(quoted diff) +3
-0+ */ + for (i = 0; i <= count; i++) + qi->desc_status[(index + i) % QI_LENGTH] =
QI_TO_BE_FREED;We don't really need a new flag. Just set them to
QI_FREEand then reclaimQI_FREEslots until hittingqi->headinreclaim_free_desc().Quoted 3 lines
We do need to have a separate state for descriptors pending to be freed. Otherwise, reclaim code will advance pass the intended range.
The commit msg said that
QI_DONEis currently used for conflicting purpose.Using
QI_FREEkeeps it only for signaling that a wait desc is completed.
Quoted 7 lines
The key is that
reclaim()should not change a desc's state before it's consumed by the owner. Instead we always let the owner to change the state andreclaim()only does scan and adjust the tracking fields then such race condition disappears.In this example T2's slots are changed to
QI_FREEby T2 after it completes all the checks. Only at this point those slots can be reclaimed.
The problem is that without the TO_BE_FREED state, the reclaim code would have no way of knowing which ones are to be reclaimed and which ones are currently free. Therefore, it cannot track free_cnt.
The current reclaim code is not aware of owners nor how many to reclaim.
If I make the following changes and run, free_cnt will keep going up and system cannot boot. Perhaps you meant another way?
drivers/iommu/intel/dmar.c +3 -4
@@ -1204,8 +1204,7 @@ static void free_iommu(struct intel_iommu *iommu)*/ static inline void reclaim_free_desc (struct q_inval *qi) { -while (qi->desc_status[qi->free_tail] ==QI_TO_BE_FREED ) { - qi->desc_status[qi->free_tail] =QI_FREE ; +while (qi->desc_status[qi->free_tail] ==QI_FREE ) { qi->free_tail = (qi->free_tail +1 ) %QI_LENGTH ; qi->free_cnt++; }
@@ -1466,10 +1465,10 @@ int qi_submit_sync(struct intel_iommu *iommu, struct qi_desc *desc,* The reclaim code can free descriptors from multiple submissions * starting from the tail of the queue. When count == 0, the * status of the standalone wait descriptor at the tail of the queue -* must be set to QI_TO_BE_FREED to allow the reclaim code to proceed. +* must be set to QI_FREE to allow the reclaim code to proceed. */ for (i =0 ; i <= count; i++) - qi->desc_status[(index + i) %QI_LENGTH ] =QI_TO_BE_FREED ; + qi->desc_status[(index + i) %QI_LENGTH ] =QI_FREE ;reclaim_free_desc (qi);raw_spin_unlock_irqrestore (&qi->q_lock, flags);
drivers/iommu/intel/iommu.h +1 -2
@@ -382,8 +382,7 @@ enum {QI_FREE ,QI_IN_USE ,QI_DONE , -QI_ABORT , - QI_TO_BE_FREED + QI_ABORT };
Thanks,
Jacob
Quoted 55 lines
From: Jacob Pan <[email protected]>
Sent: Friday, July 26, 2024 5:11 AMQuoted 33 lines
Quoted 24 lines
Quoted 1 line
Review int qi_submit_sync(struct intel_iommu
Quoted 1 line
(quoted diff) +0
-0@@ -1463,8 +1462,14 @@ int qi_submit_sync(struct intel_iommu*iommu,
Quoted 14 lines
Quoted 14 lines
(quoted diff) +2
-2struct qi_desc *desc, raw_spin_lock(&qi->q_lock); } - for (i = 0; i < count; i++) - qi->desc_status[(index + i) % QI_LENGTH] = QI_DONE; + /* + * The reclaim code can free descriptors from multiplesubmissions
(quoted diff) +1
-0+ * starting from the tail of the queue. When count == 0,the
(quoted diff) +1
-0+ * status of the standalone wait descriptor at the tail ofthe queue
(quoted diff) +1
-0+ * must be set to QI_TO_BE_FREED to allow the reclaim codeQuoted 8 lines
Quoted 5 lines
to proceed.
(quoted diff) +3
-0+ */ + for (i = 0; i <= count; i++) + qi->desc_status[(index + i) % QI_LENGTH] =
QI_TO_BE_FREED;We don't really need a new flag. Just set them to
QI_FREEand then reclaimQI_FREEslots until hittingqi->headinreclaim_free_desc().Quoted 3 lines
We do need to have a separate state for descriptors pending to be freed. Otherwise, reclaim code will advance pass the intended range.
The commit msg said that
QI_DONEis currently used for conflicting purpose.Using
QI_FREEkeeps it only for signaling that a wait desc is completed.Quoted 7 lines
The key is that
reclaim()should not change a desc's state before it's consumed by the owner. Instead we always let the owner to change the state andreclaim()only does scan and adjust the tracking fields then such race condition disappears.In this example T2's slots are changed to
QI_FREEby T2 after it completes all the checks. Only at this point those slots can be reclaimed.The problem is that without the
TO_BE_FREEDstate, the reclaim code would have no way of knowing which ones are to be reclaimed and which ones are currently free. Therefore, it cannot trackfree_cnt.The current reclaim code is not aware of owners nor how many to reclaim.
If I make the following changes and run,
free_cntwill keep going up and system cannot boot. Perhaps you meant another way?
drivers/iommu/intel/dmar.c static void free_iommu(struct intel_iommu · line 1210
Quoted 9 lines
drivers/iommu/intel/dmar.c +1
-2@@ -1204,8 +1204,7 @@ static void free_iommu(struct intel_iommu*iommu) */ static inline void reclaim_free_desc (struct q_inval *qi) { -while (qi->desc_status[qi->free_tail] ==QI_TO_BE_FREED ) { - qi->desc_status[qi->free_tail] =QI_FREE ; +while (qi->desc_status[qi->free_tail] ==QI_FREE ) { qi->free_tail = (qi->free_tail +1 ) %QI_LENGTH ; qi->free_cnt++;
Here miss a check to prevent reclaiming unused slots:
if (qi->free_tail == qi->free_head) break;
In the example flow reclaim_free_desc() in T1 will only reclaim slots used by T1 as slots of T2 are either QI_IN_USE or QI_DONE. T2 slots will be reclaimed when T2 calls reclaim_free_desc() after setting them to QI_FREE, and reclaim will stop at qi->free_head.
If for some reason T2 completes earlier than T1. reclaim_free_desc() in T2 does nothing as the first slot qi->free_tail belongs to T1 still IN_USE. T2's slots will then wait until reclaim is triggered by T1 later.
Quoted 31 lines
}drivers/iommu/intel/dmar.c +2
-2@@ -1466,10 +1465,10 @@ int qi_submit_sync(struct intel_iommu *iommu,struct qi_desc *desc, * The reclaim code can free descriptors from multiple submissions * starting from the tail of the queue. When count == 0, the * status of the standalone wait descriptor at the tail of the queue -* must be set to QI_TO_BE_FREED to allow the reclaim code to proceed. +* must be set to QI_FREE to allow the reclaim code to proceed. */ for (i =0 ; i <= count; i++) - qi->desc_status[(index + i) %QI_LENGTH ] =QI_TO_BE_FREED ; + qi->desc_status[(index + i) %QI_LENGTH ] =QI_FREE ;reclaim_free_desc (qi);raw_spin_unlock_irqrestore(&qi->q_lock, flags);drivers/iommu/intel/iommu.h +1
-2@@ -382,8 +382,7 @@ enum {QI_FREE ,QI_IN_USE ,QI_DONE , -QI_ABORT , -QI_TO_BE_FREED +QI_ABORT }; Thanks,Jacob
Quoted 74 lines · Kevin
Quoted 59 lines
From: Jacob Pan <[email protected]>
Sent: Friday, July 26, 2024 5:11 AMQuoted 36 lines
Quoted 26 lines
Quoted 1 line
Review int qi_submit_sync(struct intel_iommu
Quoted 1 line
(quoted diff) +0
-0@@ -1463,8 +1462,14 @@ int qi_submit_sync(struct intel_iommu*iommu,
Quoted 16 lines
Quoted 16 lines
(quoted diff) +2
-2struct qi_desc *desc, raw_spin_lock(&qi->q_lock); } - for (i = 0; i < count; i++) - qi->desc_status[(index + i) % QI_LENGTH] = QI_DONE; + /* + * The reclaim code can free descriptors from multiplesubmissions
(quoted diff) +1
-0+ * starting from the tail of the queue. When count ==0, the
(quoted diff) +1
-0+ * status of the standalone wait descriptor at thetail of the queue
(quoted diff) +1
-0+ * must be set to QI_TO_BE_FREED to allow the reclaimcode to proceed.
Quoted 8 lines
Quoted 4 lines
(quoted diff) +3
-0+ */ + for (i = 0; i <= count; i++) + qi->desc_status[(index + i) % QI_LENGTH] =
QI_TO_BE_FREED;We don't really need a new flag. Just set them to
QI_FREEand then reclaimQI_FREEslots until hittingqi->headinreclaim_free_desc().Quoted 3 lines
We do need to have a separate state for descriptors pending to be freed. Otherwise, reclaim code will advance pass the intended range.
The commit msg said that
QI_DONEis currently used for conflicting purpose.Using
QI_FREEkeeps it only for signaling that a wait desc is completed.Quoted 8 lines
The key is that
reclaim()should not change a desc's state before it's consumed by the owner. Instead we always let the owner to change the state andreclaim()only does scan and adjust the tracking fields then such race condition disappears.In this example T2's slots are changed to
QI_FREEby T2 after it completes all the checks. Only at this point those slots can be reclaimed.The problem is that without the
TO_BE_FREEDstate, the reclaim code would have no way of knowing which ones are to be reclaimed and which ones are currently free. Therefore, it cannot trackfree_cnt.The current reclaim code is not aware of owners nor how many to reclaim.
If I make the following changes and run,
free_cntwill keep going up and system cannot boot. Perhaps you meant another way?drivers/iommu/intel/dmar.c static void free_iommu(struct intel_iommu · line 1210
Quoted 9 lines
drivers/iommu/intel/dmar.c +1
-2@@ -1204,8 +1204,7 @@ static void free_iommu(struct intel_iommu*iommu) */ static inline void reclaim_free_desc (struct q_inval *qi) { -while (qi->desc_status[qi->free_tail] ==QI_TO_BE_FREED ) { - qi->desc_status[qi->free_tail] =QI_FREE ; +while (qi->desc_status[qi->free_tail] ==QI_FREE ) { qi->free_tail = (qi->free_tail +1 ) %QI_LENGTH ; qi->free_cnt++;Here miss a check to prevent reclaiming unused slots:
if (qi->free_tail == qi->free_head) break;
Quoted 9 lines
In the example flow
reclaim_free_desc()in T1 will only reclaim slots used by T1 as slots of T2 are eitherQI_IN_USEorQI_DONE. T2 slots will be reclaimed when T2 callsreclaim_free_desc()after setting them toQI_FREE, and reclaim will stop atqi->free_head.If for some reason T2 completes earlier than T1.
reclaim_free_desc()in T2 does nothing as the first slotqi->free_tailbelongs to T1 stillIN_USE. T2's slots will then wait until reclaim is triggered by T1 later.
This makes sense. Unlike the original code, we now only allow freeing the submitter's own descriptors.
Quoted 32 lines
Quoted 32 lines
}drivers/iommu/intel/dmar.c +0
-0@@ -1466,10 +1465,10 @@ int qi_submit_sync(struct intel_iommu *iommu,struct qi_desc *desc, * The reclaim code can free descriptors from multiplesubmissions
* starting from the tail of the queue. When count == 0, the
* status of the standalone wait descriptor at the tail of the
queue
- * must be set toQI_TO_BE_FREEDto allow the reclaim code to
proceed.drivers/iommu/intel/dmar.c +1
-1+* must be set to QI_FREE to allow the reclaim code to proceed. */ for (i =0 ; i <= count; i++) - qi->desc_status[(index + i) % QI_LENGTH] =
QI_TO_BE_FREED;drivers/iommu/intel/dmar.c +1
-0+ qi->desc_status[(index + i) %QI_LENGTH ] =QI_FREE ;reclaim_free_desc (qi);raw_spin_unlock_irqrestore (&qi->q_lock, flags);drivers/iommu/intel/iommu.h +1
-2@@ -382,8 +382,7 @@ enum {QI_FREE ,QI_IN_USE ,QI_DONE , -QI_ABORT , - QI_TO_BE_FREED + QI_ABORT };
Quoted 4 lines
Quoted 3 lines
drivers/iommu/intel/iommu.h +0
-0Thanks,Jacob
Thanks,
Jacob
Sorry, just catching up with emails.
I noticed another thing.
In qi_submit_sync() input parameters, when the count is 0, the expectation should be that desc should be NULL right? In that case, the below code will cause a problem.
type = desc->qw0 & GENMASK_ULL(3, 0);
The above line requires caller (when calling with count = 0) to pass a fake non-NULL desc pointer. Should we fix this as well? Alternatively, we can fix it whenever we create the use case of caller calling with count=0.
Thanks,
Sanjay
-----Original Message-----
From: Jacob Pan <[email protected]>
Sent: Thursday, July 25, 2024 7:42 PM
To: Tian, Kevin <[email protected]>
Cc: [email protected]; LKML <[email protected]>; Lu Baolu <[email protected]>; Liu, Yi L <[email protected]>; Zhang, Tina <[email protected]>; Kumar, Sanjay K <[email protected]>; [email protected]
Subject: Re: [PATCH] iommu/vt-d: Fix potential soft lockup due to reclaim
Quoted 73 lines · Kevin
Quoted 59 lines
From: Jacob Pan <[email protected]>
Sent: Friday, July 26, 2024 5:11 AMQuoted 36 lines
Quoted 26 lines
Quoted 1 line
Review int qi_submit_sync(struct intel_iommu
Quoted 1 line
(quoted diff) +0
-0@@ -1463,8 +1462,14 @@ int qi_submit_sync(struct intel_iommu*iommu,
Quoted 16 lines
Quoted 16 lines
(quoted diff) +2
-2struct qi_desc *desc, raw_spin_lock(&qi->q_lock); } - for (i = 0; i < count; i++) - qi->desc_status[(index + i) % QI_LENGTH] = QI_DONE; + /* + * The reclaim code can free descriptors from multiplesubmissions
(quoted diff) +1
-0+ * starting from the tail of the queue. When count ==0, the
(quoted diff) +1
-0+ * status of the standalone wait descriptor at thetail of the queue
(quoted diff) +1
-0+ * must be set to QI_TO_BE_FREED to allow the reclaimcode to proceed.
Quoted 8 lines
Quoted 4 lines
(quoted diff) +3
-0+ */ + for (i = 0; i <= count; i++) + qi->desc_status[(index + i) % QI_LENGTH] =
QI_TO_BE_FREED;We don't really need a new flag. Just set them to
QI_FREEand then reclaimQI_FREEslots until hittingqi->headinreclaim_free_desc().Quoted 3 lines
We do need to have a separate state for descriptors pending to be freed. Otherwise, reclaim code will advance pass the intended range.
The commit msg said that
QI_DONEis currently used for conflicting purpose.Using
QI_FREEkeeps it only for signaling that a wait desc is completed.Quoted 8 lines
The key is that
reclaim()should not change a desc's state before it's consumed by the owner. Instead we always let the owner to change the state andreclaim()only does scan and adjust the tracking fields then such race condition disappears.In this example T2's slots are changed to
QI_FREEby T2 after it completes all the checks. Only at this point those slots can be reclaimed.The problem is that without the
TO_BE_FREEDstate, the reclaim code would have no way of knowing which ones are to be reclaimed and which ones are currently free. Therefore, it cannot trackfree_cnt.The current reclaim code is not aware of owners nor how many to reclaim.
If I make the following changes and run,
free_cntwill keep going up and system cannot boot. Perhaps you meant another way?drivers/iommu/intel/dmar.c static void free_iommu(struct intel_iommu · line 1209
Quoted 8 lines
drivers/iommu/intel/dmar.c +1
-2@@ -1204,8 +1204,7 @@ static void free_iommu(struct intel_iommu*iommu) */ static inline void reclaim_free_desc (struct q_inval *qi) { -while (qi->desc_status[qi->free_tail] ==QI_TO_BE_FREED ) { - qi->desc_status[qi->free_tail] =QI_FREE ; +while (qi->desc_status[qi->free_tail] ==QI_FREE ) { qi->free_tail = (qi->free_tail +1 ) %QI_LENGTH ; qi->free_cnt++;Here miss a check to prevent reclaiming unused slots:
if (qi->free_tail == qi->free_head) break;
Quoted 9 lines
In the example flow
reclaim_free_desc()in T1 will only reclaim slots used by T1 as slots of T2 are eitherQI_IN_USEorQI_DONE. T2 slots will be reclaimed when T2 callsreclaim_free_desc()after setting them toQI_FREE, and reclaim will stop atqi->free_head.If for some reason T2 completes earlier than T1.
reclaim_free_desc()in T2 does nothing as the first slotqi->free_tailbelongs to T1 stillIN_USE. T2's slots will then wait until reclaim is triggered by T1 later.
This makes sense. Unlike the original code, we now only allow freeing the submitter's own descriptors.
Quoted 32 lines
Quoted 32 lines
drivers/iommu/intel/dmar.c +0
-0}@@ -1466,10 +1465,10 @@ int qi_submit_sync(struct intel_iommu *iommu, struct qi_desc *desc, * The reclaim code can free descriptors from multiplesubmissions
* starting from the tail of the queue. When count == 0, the
* status of the standalone wait descriptor at the tail of
the queue
- * must be set toQI_TO_BE_FREEDto allow the reclaim code to
proceed.drivers/iommu/intel/dmar.c +1
-1+* must be set to QI_FREE to allow the reclaim code to proceed. */ for (i =0 ; i <= count; i++) - qi->desc_status[(index + i) % QI_LENGTH] =
QI_TO_BE_FREED;
+ qi->desc_status[(index + i) %QI_LENGTH] =QI_FREE;reclaim_free_desc(qi); raw_spin_unlock_irqrestore(&qi->q_lock, flags); diff --git
a/drivers/iommu/intel/iommu.hb/drivers/iommu/intel/iommu.hindex1ab39f9145f2..eaf015b4353b100644drivers/iommu/intel/iommu.h +1
-2@@ -382,8 +382,7 @@ enum {QI_FREE ,QI_IN_USE ,QI_DONE , -QI_ABORT , - QI_TO_BE_FREED + QI_ABORT };
Quoted 4 lines
Quoted 3 lines
drivers/iommu/intel/iommu.h +0
-0Thanks,Jacob
Thanks,
Jacob
Baolu Lu
maintainerQuoted 8 lines · Sanjay K
Sorry, just catching up with emails.
I noticed another thing.
Inqi_submit_sync()input parameters, when the count is 0, the expectation should be that desc should be NULL right? In that case, the below code will cause a problem.type =
desc->qw0&GENMASK_ULL(3, 0);The above line requires caller (when calling with count = 0) to pass a fake non-NULL desc pointer. Should we fix this as well? Alternatively, we can fix it whenever we create the use case of caller calling with count=0.
No worries, Sanjay. I will take care of this later. This is actually not a fix, but rather an extension of the helper to support a new use case (count=0).
Thanks,
baolu
Quoted 141 lines
Thanks,
Sanjay-----Original Message-----
From: Jacob Pan <[email protected]>
Sent: Thursday, July 25, 2024 7:42 PM
To: Tian, Kevin <[email protected]>
Cc: [email protected]; LKML <[email protected]>; Lu Baolu <[email protected]>; Liu, Yi L <[email protected]>; Zhang, Tina <[email protected]>; Kumar, Sanjay K <[email protected]>; [email protected]
Subject: Re: [PATCH] iommu/vt-d: Fix potential soft lockup due to reclaimQuoted 73 lines · Kevin
Quoted 59 lines
From: Jacob Pan <[email protected]>
Sent: Friday, July 26, 2024 5:11 AMQuoted 36 lines
Quoted 26 lines
Quoted 1 line
Review int qi_submit_sync(struct intel_iommu
Quoted 1 line
(quoted diff) +0
-0@@ -1463,8 +1462,14 @@ int qi_submit_sync(struct intel_iommu*iommu,
Quoted 16 lines
Quoted 16 lines
(quoted diff) +2
-2struct qi_desc *desc, raw_spin_lock(&qi->q_lock); } - for (i = 0; i < count; i++) - qi->desc_status[(index + i) % QI_LENGTH] = QI_DONE; + /* + * The reclaim code can free descriptors from multiplesubmissions
(quoted diff) +1
-0+ * starting from the tail of the queue. When count ==0, the
(quoted diff) +1
-0+ * status of the standalone wait descriptor at thetail of the queue
(quoted diff) +1
-0+ * must be set to QI_TO_BE_FREED to allow the reclaimcode to proceed.
Quoted 8 lines
Quoted 4 lines
(quoted diff) +3
-0+ */ + for (i = 0; i <= count; i++) + qi->desc_status[(index + i) % QI_LENGTH] =
QI_TO_BE_FREED;We don't really need a new flag. Just set them to
QI_FREEand then reclaimQI_FREEslots until hittingqi->headinreclaim_free_desc().Quoted 3 lines
We do need to have a separate state for descriptors pending to be freed. Otherwise, reclaim code will advance pass the intended range.
The commit msg said that
QI_DONEis currently used for conflicting purpose.Using
QI_FREEkeeps it only for signaling that a wait desc is completed.Quoted 8 lines
The key is that
reclaim()should not change a desc's state before it's consumed by the owner. Instead we always let the owner to change the state andreclaim()only does scan and adjust the tracking fields then such race condition disappears.In this example T2's slots are changed to
QI_FREEby T2 after it completes all the checks. Only at this point those slots can be reclaimed.The problem is that without the
TO_BE_FREEDstate, the reclaim code would have no way of knowing which ones are to be reclaimed and which ones are currently free. Therefore, it cannot trackfree_cnt.The current reclaim code is not aware of owners nor how many to reclaim.
If I make the following changes and run,
free_cntwill keep going up and system cannot boot. Perhaps you meant another way?drivers/iommu/intel/dmar.c static void free_iommu(struct intel_iommu · line 1209
Quoted 8 lines
drivers/iommu/intel/dmar.c +1
-2@@ -1204,8 +1204,7 @@ static void free_iommu(struct intel_iommu*iommu) */ static inline void reclaim_free_desc (struct q_inval *qi) { -while (qi->desc_status[qi->free_tail] ==QI_TO_BE_FREED ) { - qi->desc_status[qi->free_tail] =QI_FREE ; +while (qi->desc_status[qi->free_tail] ==QI_FREE ) { qi->free_tail = (qi->free_tail +1 ) %QI_LENGTH ; qi->free_cnt++;Here miss a check to prevent reclaiming unused slots:
if (qi->free_tail == qi->free_head) break;Quoted 9 lines
In the example flow
reclaim_free_desc()in T1 will only reclaim slots used by T1 as slots of T2 are eitherQI_IN_USEorQI_DONE. T2 slots will be reclaimed when T2 callsreclaim_free_desc()after setting them toQI_FREE, and reclaim will stop atqi->free_head.If for some reason T2 completes earlier than T1.
reclaim_free_desc()in T2 does nothing as the first slotqi->free_tailbelongs to T1 stillIN_USE. T2's slots will then wait until reclaim is triggered by T1 later.This makes sense. Unlike the original code, we now only allow freeing the submitter's own descriptors.
Quoted 32 lines
Quoted 32 lines
drivers/iommu/intel/dmar.c +0
-0}@@ -1466,10 +1465,10 @@ int qi_submit_sync(struct intel_iommu *iommu, struct qi_desc *desc, * The reclaim code can free descriptors from multiplesubmissions
* starting from the tail of the queue. When count == 0, the
* status of the standalone wait descriptor at the tail of
the queue
- * must be set toQI_TO_BE_FREEDto allow the reclaim code to
proceed.drivers/iommu/intel/dmar.c +1
-1+* must be set to QI_FREE to allow the reclaim code to proceed. */ for (i =0 ; i <= count; i++) - qi->desc_status[(index + i) % QI_LENGTH] =
QI_TO_BE_FREED;
+ qi->desc_status[(index + i) %QI_LENGTH] =QI_FREE;reclaim_free_desc(qi); raw_spin_unlock_irqrestore(&qi->q_lock, flags); diff --git
a/drivers/iommu/intel/iommu.hb/drivers/iommu/intel/iommu.hindex1ab39f9145f2..eaf015b4353b100644drivers/iommu/intel/iommu.h +1
-2@@ -382,8 +382,7 @@ enum {QI_FREE ,QI_IN_USE ,QI_DONE , -QI_ABORT , - QI_TO_BE_FREED + QI_ABORT };Quoted 4 lines
Quoted 3 lines
drivers/iommu/intel/iommu.h +0
-0Thanks,Jacob
Thanks,
Jacob