idxd: avoid deadlock in process_misc_interrupts()
Subsystems (2)
PATCH 1/1 idxd: avoid deadlock in process_misc_interrupts()
idxd_device_clear_state() now grabs the idxd->dev_lock itself, so don't grab the lock prior to calling it.
This was seen in testing after dmar fault occurred on system, resulting in lockup stack traces.
--- drivers/dma/idxd/irq.c | 2 -- 1 file changed, 2 deletions(-)
drivers/dma/idxd/irq.c +0 -2
@@ -324,13 +324,11 @@ static int process_misc_interrupts(struct idxd_device *idxd, u32 cause) idxd->state =IDXD_DEV_HALTED ;idxd_wqs_quiesce (idxd);idxd_wqs_unmap_portal (idxd); -spin_lock (&idxd->dev_lock);idxd_device_clear_state (idxd);dev_err (&idxd->pdev->dev,"idxd halted, need %s.\n" , gensts.reset_type ==IDXD_DEVICE_RESET_FLR ?"FLR" :"system reset" ); -spin_unlock (&idxd->dev_lock);return -ENXIO ; } }
Signature · 1 line
2.37.2
PATCH v2 1/1 dmaengine: idxd: avoid deadlock in process_misc_interrupts()
idxd_device_clear_state() now grabs the idxd->dev_lock itself, so don't grab the lock prior to calling it.
This was seen in testing after dmar fault occurred on system, resulting in lockup stack traces.
---
v2: add Fixes tag, and add subsystem to summary
drivers/dma/idxd/irq.c | 2 -- 1 file changed, 2 deletions(-)
drivers/dma/idxd/irq.c +0 -2
@@ -324,13 +324,11 @@ static int process_misc_interrupts(struct idxd_device *idxd, u32 cause) idxd->state =IDXD_DEV_HALTED ;idxd_wqs_quiesce (idxd);idxd_wqs_unmap_portal (idxd); -spin_lock (&idxd->dev_lock);idxd_device_clear_state (idxd);dev_err (&idxd->pdev->dev,"idxd halted, need %s.\n" , gensts.reset_type ==IDXD_DEVICE_RESET_FLR ?"FLR" :"system reset" ); -spin_unlock (&idxd->dev_lock);return -ENXIO ; } }
Signature · 1 line
2.37.2
Re: [PATCH v2] dmaengine: idxd: avoid deadlock in process_misc_interrupts()
Quoted 6 lines · Jerry Snitselaar
idxd_device_clear_state()now grabs theidxd->dev_lockitself, so don't grab the lock prior to calling it.This was seen in testing after dmar fault occurred on system, resulting in lockup stack traces.
Quoted 6 lines
Cc Fenghua Yu <[email protected]>Cc Dave Jiang <[email protected]>Cc Vinod Koul <[email protected]>Fixes cf4ac3fef338 ("dmaengine: idxd: fix lockdep warning on device driver removal")Signed-off-by Jerry Snitselaar <[email protected]>
Thanks Jerry!
Quoted 24 lines
---
v2: add Fixes tag, and add subsystem to summarydrivers/dma/idxd/irq.c | 2 -- 1 file changed, 2 deletions(-)drivers/dma/idxd/irq.c +0
-2@@ -324,13 +324,11 @@ static int process_misc_interrupts(struct idxd_device *idxd, u32 cause) idxd->state =IDXD_DEV_HALTED ;idxd_wqs_quiesce (idxd);idxd_wqs_unmap_portal (idxd); -spin_lock (&idxd->dev_lock);idxd_device_clear_state (idxd);dev_err (&idxd->pdev->dev,"idxd halted, need %s.\n" , gensts.reset_type ==IDXD_DEVICE_RESET_FLR ?"FLR" :"system reset" ); -spin_unlock (&idxd->dev_lock);return -ENXIO ; } }
Re: [PATCH v2] dmaengine: idxd: avoid deadlock in process_misc_interrupts()
Quoted 15 lines · Dave Jiang
Quoted 6 lines · Jerry Snitselaar
idxd_device_clear_state()now grabs theidxd->dev_lockitself, so don't grab the lock prior to calling it.This was seen in testing after dmar fault occurred on system, resulting in lockup stack traces.
Quoted 6 lines
Cc Fenghua Yu <[email protected]>Cc Dave Jiang <[email protected]>Cc Vinod Koul <[email protected]>Fixes cf4ac3fef338 ("dmaengine: idxd: fix lockdep warning on device driver removal")Signed-off-by Jerry Snitselaar <[email protected]>
Quoted 4 lines
Thanks Jerry!
Reviewed-by Dave Jiang <[email protected]>
I noticed another problem while looking at this. When the device ends up in the halted state, and needs an flr or system reset, it calls idxd_wqs_unmap_portal(). Then if you do a modprobe -r idxd, you hit the WARN_ON in devm_iounmap(), because the remove code path calls idxd_wq_portal_unmap(), and wq->portal is null. I'm not sure if it just needs a simple sanity check in drv_disable_wq() to avoid the call in the case that it has already been unmapped, or if more cleanup needs to be done, and possibly a state to differentiate between halted + soft reset possible, versus halted + flr or system reset needed. You get multiple "Device is HALTED" messages during the removal as well.
Regards,
Jerry
Dave Jiang
reviewerRe: [PATCH v2] dmaengine: idxd: avoid deadlock in process_misc_interrupts()
Quoted 18 lines · Jerry Snitselaar
Quoted 13 lines · Dave Jiang
Quoted 6 lines · Jerry Snitselaar
idxd_device_clear_state()now grabs theidxd->dev_lockitself, so don't grab the lock prior to calling it.This was seen in testing after dmar fault occurred on system, resulting in lockup stack traces.
Quoted 6 lines
Cc Fenghua Yu <[email protected]>Cc Dave Jiang <[email protected]>Cc Vinod Koul <[email protected]>Fixes cf4ac3fef338 ("dmaengine: idxd: fix lockdep warning on device driver removal")Signed-off-by Jerry Snitselaar <[email protected]>Quoted 4 lines
Thanks Jerry!
Reviewed-by Dave Jiang <[email protected]>
Quoted 11 lines
I noticed another problem while looking at this. When the device ends up in the halted state, and needs an flr or system reset, it calls
idxd_wqs_unmap_portal(). Then if you do a modprobe -r idxd, you hit theWARN_ONindevm_iounmap(), because the remove code path callsidxd_wq_portal_unmap(), andwq->portalis null. I'm not sure if it just needs a simple sanity check indrv_disable_wq()to avoid the call in the case that it has already been unmapped, or if more cleanup needs to be done, and possibly a state to differentiate between halted + soft reset possible, versus halted + flr or system reset needed. You get multiple "Device is HALTED" messages during the removal as well.
Thanks!
Fenghua, can you please take a look at this when you have a chance? Thank you!
Quoted 4 lines
Regards,
Jerry
Re: [PATCH v2] dmaengine: idxd: avoid deadlock in process_misc_interrupts()
Quoted 35 lines · Dave Jiang
Quoted 19 lines · Jerry Snitselaar
Quoted 14 lines · Dave Jiang
Quoted 6 lines · Jerry Snitselaar
idxd_device_clear_state()now grabs theidxd->dev_lockitself, so don't grab the lock prior to calling it.This was seen in testing after dmar fault occurred on system, resulting in lockup stack traces.
Quoted 7 lines
Cc Fenghua Yu <[email protected]>Cc Dave Jiang <[email protected]>Cc Vinod Koul <[email protected]>Fixes cf4ac3fef338 ("dmaengine: idxd: fix lockdep warning ondevice driver removal")
Signed-off-by Jerry Snitselaar <[email protected]>Quoted 4 lines
Thanks Jerry!
Reviewed-by Dave Jiang <[email protected]>Quoted 13 lines
I noticed another problem while looking at this. When the device ends
up in the halted state, and needs an flr or system reset, it callsidxd_wqs_unmap_portal(). Then if you do a modprobe -r idxd, you hit theWARN_ONindevm_iounmap(), because the remove code path callsidxd_wq_portal_unmap(), andwq->portalis null. I'm not sure if it just needs a simple sanity check indrv_disable_wq()to avoid the call
in the case that it has already been unmapped, or if more cleanup needs to be done, and possibly a state to differentiate between halted + soft reset possible, versus halted + flr or system reset needed. You get multiple "Device is HALTED" messages during the removal as well.
Quoted 6 lines
Thanks!
Fenghua, can you please take a look at this when you have a chance? Thank you!
Fenghua,
I see another potential issue. If a software reset is attempted idxd_device_reinit() will be called which walks the wqs, and if a wq has the state IDXD_WQ_ENABLED it calls idxd_wq_enable(), but the first thing idxd_wq_enable() does is see that the state is IDXD_WQ_ENABLED and returns 0. Without the wq enable command being sent, it will not be re-enabled, yes?
Regards,
Jerry
Quoted 5 lines
Quoted 4 lines
Regards,
Jerry
RE: [PATCH v2] dmaengine: idxd: avoid deadlock in process_misc_interrupts()
Hi, Jerry,
Quoted 5 lines
I see another potential issue. If a software reset is attempted
idxd_device_reinit()will be called which walks the wqs, and if a wq has the stateIDXD_WQ_ENABLEDit callsidxd_wq_enable(), but the first thingidxd_wq_enable()does is see that the state isIDXD_WQ_ENABLEDand returns 0. Without the wq enable command being sent, it will not be re-enabled, yes?
Could you please describe how to reproduce the issues and test case?
Thanks.
-Fenghua
Re: [PATCH v2] dmaengine: idxd: avoid deadlock in process_misc_interrupts()
Quoted 11 lines · Fenghua
Hi, Jerry,
Quoted 8 lines
I see another potential issue. If a software reset is attempted
idxd_device_reinit()
will be called which walks the wqs, and if a wq has the stateIDXD_WQ_ENABLEDit callsidxd_wq_enable(), but the first thingidxd_wq_enable()does is see that the state isIDXD_WQ_ENABLEDand returns 0.
Without the wq enable command being sent, it will not be re- enabled, yes?
Quoted 1 line
Could you please describe how to reproduce the issues and test case?
Nothing special, just running dsa_user_test_runner.sh from idxd-config (3.4.6.3) with intel_iommu=on,sm_on (default is lazy dma domain) on an Intel supplied SPR system. During the run there is a dmar fault, halts the device with it needing an flr reset. That is where the deadlock was noticed. Then after the dsa_user_runner.sh fails, if you do modprobe -r idxd, you hit the warning in devm_iounmap().
The idxd_device_reinit() case I haven't reproduced, I'm just looking at the code.
I should note that I've only run into an issue with that model of system. The other models of SPR systems don't hit an issue. This is with a 6.0-rc1 kernel, and our kernel. I'm guessing there is an issue with that model of system/fw. I've seen both fault reasons 0x50 and 0x58 when it faults.
Regards,
Jerry
Quoted 4 lines
Thanks.
-Fenghua
Re: [PATCH v2] dmaengine: idxd: avoid deadlock in process_misc_interrupts()
In reply to Jerry SnitselaarQuoted 5 lines · Jerry Snitselaar
idxd_device_clear_state()now grabs theidxd->dev_lockitself, so don't grab the lock prior to calling it.This was seen in testing after dmar fault occurred on system, resulting in lockup stack traces.
Applied, thanks
Signature · 1 line
~Vinod
Hi, Jerry,
Quoted 6 lines
idxd_device_clear_state()now grabs theidxd->dev_lockitself, so don't grab the lock prior to calling it.This was seen in testing after dmar fault occurred on system, resulting in lockup stack traces.
Please add Fixes: cf4ac3fef338 ...
Quoted 27 lines
Cc Fenghua Yu <[email protected]>Cc Dave Jiang <[email protected]>Cc Vinod Koul <[email protected]>Signed-off-by Jerry Snitselaar <[email protected]>--- drivers/dma/idxd/irq.c | 2 -- 1 file changed, 2 deletions(-)drivers/dma/idxd/irq.c index renamed from drivers/dma/idxd/irq.c +0
-0
743ead5ebc57..5b9921475be6100644drivers/dma/idxd/irq.c +0
-2@@ -324,13 +324,11 @@ static int process_misc_interrupts(struct idxd_device *idxd, u32 cause) idxd->state =IDXD_DEV_HALTED ;idxd_wqs_quiesce (idxd);idxd_wqs_unmap_portal (idxd); -spin_lock (&idxd->dev_lock);idxd_device_clear_state (idxd);dev_err (&idxd->pdev->dev,"idxd halted, need %s.\n" , gensts.reset_type ==IDXD_DEVICE_RESET_FLR ?"FLR" :"system reset" ); -spin_unlock (&idxd->dev_lock);return -ENXIO ; }
Quoted 3 lines
}
--
2.37.2
Thanks.
-Fenghua
Quoted 48 lines · Fenghua
Hi, Jerry,
Quoted 6 lines
idxd_device_clear_state()now grabs theidxd->dev_lockitself, so don't grab the lock prior to calling it.This was seen in testing after dmar fault occurred on system, resulting in lockup stack traces.
Please add Fixes:
cf4ac3fef338...Quoted 28 lines
Cc Fenghua Yu <[email protected]>Cc Dave Jiang <[email protected]>Cc Vinod Koul <[email protected]>Signed-off-by Jerry Snitselaar <[email protected]>---
drivers/dma/idxd/irq.c| 2 --1 file changed, 2 deletions(-)drivers/dma/idxd/irq.c index renamed from drivers/dma/idxd/irq.c +0
-0
743ead5ebc57..5b9921475be6100644drivers/dma/idxd/irq.c +0
-2@@ -324,13 +324,11 @@ static int process_misc_interrupts(struct idxd_device *idxd, u32 cause) idxd->state =IDXD_DEV_HALTED ;idxd_wqs_quiesce (idxd);idxd_wqs_unmap_portal (idxd); -spin_lock (&idxd->dev_lock);idxd_device_clear_state (idxd);dev_err (&idxd->pdev->dev,"idxd halted, need %s.\n" , gensts.reset_type ==IDXD_DEVICE_RESET_FLR ?"FLR" :"system reset" ); -spin_unlock (&idxd->dev_lock);return -ENXIO ; }}Quoted 2 lines
--
2.37.2Thanks.
Quoted 2 lines
-Fenghua
Hi Fenghua,
I think this was merged back in August. Not at my system to get the exact date.
Regards,
Jerry