Skip to contentSkip to conversation

idxd: avoid deadlock in process_misc_interrupts()

PATCH Superseded Jerry Snitselaar 4 participants 11 messages Aug 23, 2022 – Dec 12, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 idxd: avoid deadlock in process_misc_interrupts()

idxd_device_clear_state() now grabs the idxd->dev_lock itself, so don't grab the lock prior to calling it.

This was seen in testing after dmar fault occurred on system, resulting in lockup stack traces.

---
 drivers/dma/idxd/irq.c | 2 --
 1 file changed, 2 deletions(-)
drivers/dma/idxd/irq.c +0 -2
index 743ead5ebc57..5b9921475be6 100644
@@ -324,13 +324,11 @@ static int process_misc_interrupts(struct idxd_device *idxd, u32 cause)
 			idxd->state = IDXD_DEV_HALTED;
 			idxd_wqs_quiesce(idxd);
 			idxd_wqs_unmap_portal(idxd);
-			spin_lock(&idxd->dev_lock);
 			idxd_device_clear_state(idxd);
 			dev_err(&idxd->pdev->dev,
 				"idxd halted, need %s.\n",
 				gensts.reset_type == IDXD_DEVICE_RESET_FLR ?
 				"FLR" : "system reset");
-			spin_unlock(&idxd->dev_lock);
 			return -ENXIO;
 		}
 	}
Signature · 1 line
2.37.2
#1 lore raw

PATCH v2 1/1 dmaengine: idxd: avoid deadlock in process_misc_interrupts()

idxd_device_clear_state() now grabs the idxd->dev_lock itself, so don't grab the lock prior to calling it.

This was seen in testing after dmar fault occurred on system, resulting in lockup stack traces.

Cc Fenghua Yu <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Vinod Koul <[email protected]>
Fixes cf4ac3fef338 ("dmaengine: idxd: fix lockdep warning on device driver removal")

---
v2: add Fixes tag, and add subsystem to summary

 drivers/dma/idxd/irq.c | 2 --
 1 file changed, 2 deletions(-)
drivers/dma/idxd/irq.c +0 -2
index 743ead5ebc57..5b9921475be6 100644
@@ -324,13 +324,11 @@ static int process_misc_interrupts(struct idxd_device *idxd, u32 cause)
 			idxd->state = IDXD_DEV_HALTED;
 			idxd_wqs_quiesce(idxd);
 			idxd_wqs_unmap_portal(idxd);
-			spin_lock(&idxd->dev_lock);
 			idxd_device_clear_state(idxd);
 			dev_err(&idxd->pdev->dev,
 				"idxd halted, need %s.\n",
 				gensts.reset_type == IDXD_DEVICE_RESET_FLR ?
 				"FLR" : "system reset");
-			spin_unlock(&idxd->dev_lock);
 			return -ENXIO;
 		}
 	}
Signature · 1 line
2.37.2
reviewer Reviewed-by on v2 1/1
#2 lore raw

Re: [PATCH v2] dmaengine: idxd: avoid deadlock in process_misc_interrupts()

Quoted 6 lines · Jerry Snitselaar

idxd_device_clear_state() now grabs the idxd->dev_lock itself, so don't grab the lock prior to calling it.

This was seen in testing after dmar fault occurred on system, resulting in lockup stack traces.

Quoted 6 lines
Cc Fenghua Yu <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Vinod Koul <[email protected]>
Fixes cf4ac3fef338 ("dmaengine: idxd: fix lockdep warning on device driver removal")

Thanks Jerry!

Quoted 24 lines

---
v2: add Fixes tag, and add subsystem to summary

  drivers/dma/idxd/irq.c | 2 --
  1 file changed, 2 deletions(-)
drivers/dma/idxd/irq.c +0 -2
index 743ead5ebc57..5b9921475be6 100644
@@ -324,13 +324,11 @@ static int process_misc_interrupts(struct idxd_device *idxd, u32 cause)
  			idxd->state = IDXD_DEV_HALTED;
  			idxd_wqs_quiesce(idxd);
  			idxd_wqs_unmap_portal(idxd);
-			spin_lock(&idxd->dev_lock);
  			idxd_device_clear_state(idxd);
  			dev_err(&idxd->pdev->dev,
  				"idxd halted, need %s.\n",
  				gensts.reset_type == IDXD_DEVICE_RESET_FLR ?
  				"FLR" : "system reset");
-			spin_unlock(&idxd->dev_lock);
  			return -ENXIO;
  		}
  	}
#3 lore raw

Re: [PATCH v2] dmaengine: idxd: avoid deadlock in process_misc_interrupts()

Quoted 15 lines · Dave Jiang
Quoted 6 lines · Jerry Snitselaar

idxd_device_clear_state() now grabs the idxd->dev_lock itself, so don't grab the lock prior to calling it.

This was seen in testing after dmar fault occurred on system, resulting in lockup stack traces.

Quoted 6 lines
Cc Fenghua Yu <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Vinod Koul <[email protected]>
Fixes cf4ac3fef338 ("dmaengine: idxd: fix lockdep warning on device driver removal")
Quoted 4 lines

Thanks Jerry!

I noticed another problem while looking at this. When the device ends up in the halted state, and needs an flr or system reset, it calls idxd_wqs_unmap_portal(). Then if you do a modprobe -r idxd, you hit the WARN_ON in devm_iounmap(), because the remove code path calls idxd_wq_portal_unmap(), and wq->portal is null. I'm not sure if it just needs a simple sanity check in drv_disable_wq() to avoid the call in the case that it has already been unmapped, or if more cleanup needs to be done, and possibly a state to differentiate between halted + soft reset possible, versus halted + flr or system reset needed. You get multiple "Device is HALTED" messages during the removal as well.

Regards,
Jerry

reviewer
#4 lore raw

Re: [PATCH v2] dmaengine: idxd: avoid deadlock in process_misc_interrupts()

Quoted 18 lines · Jerry Snitselaar
Quoted 13 lines · Dave Jiang
Quoted 6 lines · Jerry Snitselaar

idxd_device_clear_state() now grabs the idxd->dev_lock itself, so don't grab the lock prior to calling it.

This was seen in testing after dmar fault occurred on system, resulting in lockup stack traces.

Quoted 6 lines
Cc Fenghua Yu <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Vinod Koul <[email protected]>
Fixes cf4ac3fef338 ("dmaengine: idxd: fix lockdep warning on device driver removal")
Quoted 4 lines

Thanks Jerry!

Quoted 11 lines

I noticed another problem while looking at this. When the device ends up in the halted state, and needs an flr or system reset, it calls idxd_wqs_unmap_portal(). Then if you do a modprobe -r idxd, you hit the WARN_ON in devm_iounmap(), because the remove code path calls idxd_wq_portal_unmap(), and wq->portal is null. I'm not sure if it just needs a simple sanity check in drv_disable_wq() to avoid the call in the case that it has already been unmapped, or if more cleanup needs to be done, and possibly a state to differentiate between halted + soft reset possible, versus halted + flr or system reset needed. You get multiple "Device is HALTED" messages during the removal as well.

Thanks!

Fenghua, can you please take a look at this when you have a chance? Thank you!

Quoted 4 lines

Regards,
Jerry

#5 lore raw

Re: [PATCH v2] dmaengine: idxd: avoid deadlock in process_misc_interrupts()

Quoted 35 lines · Dave Jiang
Quoted 19 lines · Jerry Snitselaar
Quoted 14 lines · Dave Jiang
Quoted 6 lines · Jerry Snitselaar

idxd_device_clear_state() now grabs the idxd->dev_lock itself, so don't grab the lock prior to calling it.

This was seen in testing after dmar fault occurred on system, resulting in lockup stack traces.

Quoted 7 lines
Cc Fenghua Yu <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Vinod Koul <[email protected]>
Fixes cf4ac3fef338 ("dmaengine: idxd: fix lockdep warning on

device driver removal")

Quoted 4 lines

Thanks Jerry!

Quoted 13 lines

I noticed another problem while looking at this. When the device ends
up in the halted state, and needs an flr or system reset, it calls idxd_wqs_unmap_portal(). Then if you do a modprobe -r idxd, you hit the WARN_ON in devm_iounmap(), because the remove code path calls idxd_wq_portal_unmap(), and wq->portal is null. I'm not sure if it just needs a simple sanity check in drv_disable_wq() to avoid the call
in the case that it has already been unmapped, or if more cleanup needs to be done, and possibly a state to differentiate between halted + soft reset possible, versus halted + flr or system reset needed.  You get multiple "Device is HALTED" messages during the removal as well.

Quoted 6 lines

Thanks!

Fenghua, can you please take a look at this when you have a chance? Thank you!

Fenghua,

I see another potential issue. If a software reset is attempted idxd_device_reinit() will be called which walks the wqs, and if a wq has the state IDXD_WQ_ENABLED it calls idxd_wq_enable(), but the first thing idxd_wq_enable() does is see that the state is IDXD_WQ_ENABLED and returns 0. Without the wq enable command being sent, it will not be re-enabled, yes?

Regards,
Jerry

Quoted 5 lines
Quoted 4 lines

Regards,
Jerry

#6 lore raw

RE: [PATCH v2] dmaengine: idxd: avoid deadlock in process_misc_interrupts()

Hi, Jerry,

Quoted 5 lines

I see another potential issue. If a software reset is attempted idxd_device_reinit() will be called which walks the wqs, and if a wq has the state IDXD_WQ_ENABLED it calls idxd_wq_enable(), but the first thing idxd_wq_enable() does is see that the state is IDXD_WQ_ENABLED and returns 0. Without the wq enable command being sent, it will not be re-enabled, yes?

Could you please describe how to reproduce the issues and test case?

Thanks.

-Fenghua

#7 lore raw

Re: [PATCH v2] dmaengine: idxd: avoid deadlock in process_misc_interrupts()

Quoted 11 lines · Fenghua

Hi, Jerry,

Quoted 8 lines

I see another potential issue. If a software reset is attempted idxd_device_reinit()
will be called which walks the wqs, and if a wq has the state IDXD_WQ_ENABLED it calls idxd_wq_enable(), but the first thing idxd_wq_enable() does is see that the state is IDXD_WQ_ENABLED and returns 0.
Without the wq enable command being sent, it will not be re- enabled, yes?

Quoted 1 line

Could you please describe how to reproduce the issues and test case?

Nothing special, just running dsa_user_test_runner.sh from idxd-config (3.4.6.3) with intel_iommu=on,sm_on (default is lazy dma domain) on an Intel supplied SPR system. During the run there is a dmar fault, halts the device with it needing an flr reset. That is where the deadlock was noticed. Then after the dsa_user_runner.sh fails, if you do modprobe -r idxd, you hit the warning in devm_iounmap().

The idxd_device_reinit() case I haven't reproduced, I'm just looking at the code.

I should note that I've only run into an issue with that model of system. The other models of SPR systems don't hit an issue. This is with a 6.0-rc1 kernel, and our kernel. I'm guessing there is an issue with that model of system/fw. I've seen both fault reasons 0x50 and 0x58 when it faults.

Regards,
Jerry

Quoted 4 lines

Thanks.

-Fenghua

maintainer Applied
#8 lore raw

Re: [PATCH v2] dmaengine: idxd: avoid deadlock in process_misc_interrupts()

In reply to Jerry Snitselaar
Quoted 5 lines · Jerry Snitselaar

idxd_device_clear_state() now grabs the idxd->dev_lock itself, so don't grab the lock prior to calling it.

This was seen in testing after dmar fault occurred on system, resulting in lockup stack traces.

Applied, thanks

Signature · 1 line
~Vinod
#9 lore raw
In reply to Jerry Snitselaar

Hi, Jerry,

Quoted 6 lines

idxd_device_clear_state() now grabs the idxd->dev_lock itself, so don't grab the lock prior to calling it.

This was seen in testing after dmar fault occurred on system, resulting in lockup stack traces.

Please add Fixes: cf4ac3fef338 ...

Quoted 27 lines
---
 drivers/dma/idxd/irq.c | 2 --
 1 file changed, 2 deletions(-)
drivers/dma/idxd/irq.c index renamed from drivers/dma/idxd/irq.c +0 -0

743ead5ebc57..5b9921475be6 100644

drivers/dma/idxd/irq.c +0 -2
@@ -324,13 +324,11 @@ static int process_misc_interrupts(struct idxd_device
 *idxd, u32 cause)
 			idxd->state = IDXD_DEV_HALTED;
 			idxd_wqs_quiesce(idxd);
 			idxd_wqs_unmap_portal(idxd);
-			spin_lock(&idxd->dev_lock);
 			idxd_device_clear_state(idxd);
 			dev_err(&idxd->pdev->dev,
 				"idxd halted, need %s.\n",
 				gensts.reset_type == IDXD_DEVICE_RESET_FLR ?
 				"FLR" : "system reset");
-			spin_unlock(&idxd->dev_lock);
 			return -ENXIO;
 		}
Quoted 3 lines

}
--
2.37.2

Thanks.

-Fenghua

#10 lore raw
Quoted 48 lines · Fenghua

Hi, Jerry,

Quoted 6 lines

idxd_device_clear_state() now grabs the idxd->dev_lock itself, so don't grab the lock prior to calling it.

This was seen in testing after dmar fault occurred on system, resulting in lockup stack traces.

Please add Fixes: cf4ac3fef338 ...

Quoted 28 lines

---
drivers/dma/idxd/irq.c | 2 --

1 file changed, 2 deletions(-)
drivers/dma/idxd/irq.c index renamed from drivers/dma/idxd/irq.c +0 -0

743ead5ebc57..5b9921475be6 100644

drivers/dma/idxd/irq.c +0 -2
@@ -324,13 +324,11 @@ static int process_misc_interrupts(struct idxd_device
 *idxd, u32 cause)
           idxd->state = IDXD_DEV_HALTED;
           idxd_wqs_quiesce(idxd);
           idxd_wqs_unmap_portal(idxd);
-            spin_lock(&idxd->dev_lock);
           idxd_device_clear_state(idxd);
           dev_err(&idxd->pdev->dev,
               "idxd halted, need %s.\n",
               gensts.reset_type == IDXD_DEVICE_RESET_FLR ?
               "FLR" : "system reset");
-            spin_unlock(&idxd->dev_lock);
           return -ENXIO;
       }
   }
Quoted 2 lines

--
2.37.2

Thanks.

Quoted 2 lines

-Fenghua

Hi Fenghua,

I think this was merged back in August. Not at my system to get the exact date.

Regards,
Jerry