Skip to contentSkip to conversation

dmaengine: idxd: Set workqueue state to disabled before trying to re-enable

PATCH In review Jerry Snitselaar 2 participants 10 messages Aug 24, 2022 – Sep 19, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 dmaengine: idxd: Set workqueue state to disabled before trying to re-enable

For a software reset idxd_device_reinit() is called, which will walk the device workqueues to see which ones were enabled, and try to re-enable them. It keys off wq->state being iDXD_WQ_ENABLED, but the first thing idxd_enable_wq() will do is see that the state of the workqueue is enabled, and return 0 instead of attempting to issue a command to enable the workqueue.

So once a workqueue is found that needs to be re-enabled, set the state to disabled prior to calling idxd_enable_wq(). This would accurately reflect the state if the enable fails as well.

Cc Fenghua Yu <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Vinod Koul <[email protected]>
Fixes bfe1d56091c1 ("dmaengine: idxd: Init and probe for Intel data accelerators")
---
 drivers/dma/idxd/irq.c | 1 +
 1 file changed, 1 insertion(+)
drivers/dma/idxd/irq.c +1 -0
index 743ead5ebc57..723eeb5328d6 100644
@@ -52,6 +52,7 @@ static void idxd_device_reinit(struct work_struct *work)
 		struct idxd_wq *wq = idxd->wqs[i];
 
 		if (wq->state == IDXD_WQ_ENABLED) {
+			wq->state = IDXD_WQ_DISABLED;
 			rc = idxd_wq_enable(wq);
 			if (rc < 0) {
 				dev_warn(dev, "Unable to re-enable wq %s\n",
Signature · 1 line
2.37.2
reviewer
#1 lore raw
Quoted 27 lines · Jerry Snitselaar

For a software reset idxd_device_reinit() is called, which will walk the device workqueues to see which ones were enabled, and try to re-enable them. It keys off wq->state being iDXD_WQ_ENABLED, but the first thing idxd_enable_wq() will do is see that the state of the workqueue is enabled, and return 0 instead of attempting to issue a command to enable the workqueue.

So once a workqueue is found that needs to be re-enabled, set the state to disabled prior to calling idxd_enable_wq(). This would accurately reflect the state if the enable fails as well.

Cc Fenghua Yu <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Vinod Koul <[email protected]>
Fixes bfe1d56091c1 ("dmaengine: idxd: Init and probe for Intel data accelerators")
---
  drivers/dma/idxd/irq.c | 1 +
  1 file changed, 1 insertion(+)
drivers/dma/idxd/irq.c +0 -0
index 743ead5ebc57..723eeb5328d6 100644

drivers/dma/idxd/irq.c static void idxd_device_reinit(struct work_struct *work) · line 55

Quoted 4 lines
drivers/dma/idxd/irq.c +1 -0
@@ -52,6 +52,7 @@ static void idxd_device_reinit(struct work_struct *work)
  		struct idxd_wq *wq = idxd->wqs[i];
 
  		if (wq->state == IDXD_WQ_ENABLED) {
+			wq->state = IDXD_WQ_DISABLED;

Might be better off to insert this line in idxd_wq_disable_cleanup(). I think that should put it in sane state.

Quoted 3 lines
drivers/dma/idxd/irq.c +0 -0
  			rc = idxd_wq_enable(wq);
  			if (rc < 0) {
  				dev_warn(dev, "Unable to re-enable wq %s\n",
#2 lore raw
Quoted 24 lines · Dave Jiang
Quoted 22 lines · Jerry Snitselaar

For a software reset idxd_device_reinit() is called, which will walk the device workqueues to see which ones were enabled, and try to re-enable them. It keys off wq->state being iDXD_WQ_ENABLED, but the first thing idxd_enable_wq() will do is see that the state of the workqueue is enabled, and return 0 instead of attempting to issue a command to enable the workqueue.

So once a workqueue is found that needs to be re-enabled, set the state to disabled prior to calling idxd_enable_wq(). This would accurately reflect the state if the enable fails as well.

Cc Fenghua Yu <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Vinod Koul <[email protected]>
Fixes bfe1d56091c1 ("dmaengine: idxd: Init and probe for Intel data accelerators")
---
  drivers/dma/idxd/irq.c | 1 +
  1 file changed, 1 insertion(+)
Quoted 10 lines

drivers/dma/idxd/irq.c static void idxd_device_reinit(struct work_struct *work) · line 54

Quoted 8 lines
drivers/dma/idxd/irq.c +1 -0
index 743ead5ebc57..723eeb5328d6 100644
@@ -52,6 +52,7 @@ static void idxd_device_reinit(struct work_struct *work)
  		struct idxd_wq *wq = idxd->wqs[i];
  		if (wq->state == IDXD_WQ_ENABLED) {
+			wq->state = IDXD_WQ_DISABLED;

Might be better off to insert this line in idxd_wq_disable_cleanup(). I think that should put it in sane state.

I don't think that is called in the code path that I was lookng at. I've been looking at this bit of process_misc_interrupts():

halt:

	gensts.bits = ioread32(idxd->reg_base + IDXD_GENSTATS_OFFSET);
	if (gensts.state == IDXD_DEVICE_STATE_HALT) {
		idxd->state = IDXD_DEV_HALTED;
		if (gensts.reset_type == IDXD_DEVICE_RESET_SOFTWARE) {
			/*
			 * If we need a software reset, we will throw the work
			 * on a system workqueue in order to allow interrupts
			 * for the device command completions.
			 */
			INIT_WORK(&idxd->work, idxd_device_reinit);
			queue_work(idxd->wq, &idxd->work);
		} else {
			idxd->state = IDXD_DEV_HALTED;
			idxd_wqs_quiesce(idxd);
			idxd_wqs_unmap_portal(idxd);
			spin_lock(&idxd->dev_lock);
			idxd_device_clear_state(idxd);
			dev_err(&idxd->pdev->dev,
				"idxd halted, need %s.\n",
				gensts.reset_type == IDXD_DEVICE_RESET_FLR ?
				"FLR" : "system reset");
			spin_unlock(&idxd->dev_lock);
			return -ENXIO;
		}
	}

	return 0;
}

So it sees that the device is halted, and sticks idxd_device_reinint() on that workqueue. The idxd_device_reinit() has this loop to re-enable the idxd wqs:

	for (i = 0; i < idxd->max_wqs; i++) {
		struct idxd_wq *wq = idxd->wqs[i];

		if (wq->state == IDXD_WQ_ENABLED) {
			wq->state = IDXD_WQ_DISABLED;
			rc = idxd_wq_enable(wq);
			if (rc < 0) {
				dev_warn(dev, "Unable to re-enable wq %s\n",
					 dev_name(wq_confdev(wq)));
			}
		}
	}

Once you go into idxd_wq_enable() though you get this check at the beginning:

 	if (wq->state == IDXD_WQ_ENABLED) {
		dev_dbg(dev, "WQ %d already enabled\n", wq->id);
		return 0;
	}

So IIUC it sees the device is halted, goes to reset it, figures out a wq should be re-enabled, calls idxd_wq_enable() which hits the check, returns 0 and the wq is never really re-enabled, though it will still have wq state set to IDXD_WQ_ENABLED.

Or am I missing something?

Regards,
Jerry

Quoted 3 lines
Quoted 3 lines
drivers/dma/idxd/irq.c +0 -0
  			rc = idxd_wq_enable(wq);
  			if (rc < 0) {
  				dev_warn(dev, "Unable to re-enable wq %s\n",
reviewer
#3 lore raw
Quoted 67 lines · Jerry Snitselaar
Quoted 23 lines · Dave Jiang
Quoted 22 lines · Jerry Snitselaar

For a software reset idxd_device_reinit() is called, which will walk the device workqueues to see which ones were enabled, and try to re-enable them. It keys off wq->state being iDXD_WQ_ENABLED, but the first thing idxd_enable_wq() will do is see that the state of the workqueue is enabled, and return 0 instead of attempting to issue a command to enable the workqueue.

So once a workqueue is found that needs to be re-enabled, set the state to disabled prior to calling idxd_enable_wq(). This would accurately reflect the state if the enable fails as well.

Cc Fenghua Yu <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Vinod Koul <[email protected]>
Fixes bfe1d56091c1 ("dmaengine: idxd: Init and probe for Intel data accelerators")
---
   drivers/dma/idxd/irq.c | 1 +
   1 file changed, 1 insertion(+)
Quoted 10 lines

drivers/dma/idxd/irq.c static void idxd_device_reinit(struct work_struct *work) · line 54

Quoted 8 lines
drivers/dma/idxd/irq.c +1 -0
index 743ead5ebc57..723eeb5328d6 100644
@@ -52,6 +52,7 @@ static void idxd_device_reinit(struct work_struct *work)
   		struct idxd_wq *wq = idxd->wqs[i];
   		if (wq->state == IDXD_WQ_ENABLED) {
+			wq->state = IDXD_WQ_DISABLED;

Might be better off to insert this line in idxd_wq_disable_cleanup(). I think that should put it in sane state.

I don't think that is called in the code path that I was lookng at. I've been looking at this bit of process_misc_interrupts():

halt:

	gensts.bits = ioread32(idxd->reg_base + IDXD_GENSTATS_OFFSET);
	if (gensts.state == IDXD_DEVICE_STATE_HALT) {
		idxd->state = IDXD_DEV_HALTED;
		if (gensts.reset_type == IDXD_DEVICE_RESET_SOFTWARE) {
			/*
			 * If we need a software reset, we will throw the work
			 * on a system workqueue in order to allow interrupts
			 * for the device command completions.
			 */
			INIT_WORK(&idxd->work, idxd_device_reinit);
			queue_work(idxd->wq, &idxd->work);
		} else {
			idxd->state = IDXD_DEV_HALTED;
			idxd_wqs_quiesce(idxd);
			idxd_wqs_unmap_portal(idxd);
			spin_lock(&idxd->dev_lock);
			idxd_device_clear_state(idxd);
			dev_err(&idxd->pdev->dev,
				"idxd halted, need %s.\n",
				gensts.reset_type == IDXD_DEVICE_RESET_FLR ?
				"FLR" : "system reset");
			spin_unlock(&idxd->dev_lock);
			return -ENXIO;
		}
	}

	return 0;
}
Quoted 2 lines

So it sees that the device is halted, and sticks idxd_device_reinint() on that workqueue. The idxd_device_reinit() has this loop to re-enable the idxd wqs:

idxd_device_reinit() should called idxd_device_reset() first. And that should at some point call idxd_wq_disable_cleanup() and clean up the states.

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/irq.c#L42

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/device.c#L725

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/device.c#L711

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/device.c#L376

So if we stick the wq state reset in there, it should show up as "disabled" by the time we try to enable the WQs again. Does that look reasonable?

Quoted 34 lines
	for (i = 0; i < idxd->max_wqs; i++) {
		struct idxd_wq *wq = idxd->wqs[i];

		if (wq->state == IDXD_WQ_ENABLED) {
			wq->state = IDXD_WQ_DISABLED;
			rc = idxd_wq_enable(wq);
			if (rc < 0) {
				dev_warn(dev, "Unable to re-enable wq %s\n",
					 dev_name(wq_confdev(wq)));
			}
		}
	}

Once you go into idxd_wq_enable() though you get this check at the beginning:

  	if (wq->state == IDXD_WQ_ENABLED) {
		dev_dbg(dev, "WQ %d already enabled\n", wq->id);
		return 0;
	}

So IIUC it sees the device is halted, goes to reset it, figures out a wq should be re-enabled, calls idxd_wq_enable() which hits the check, returns 0 and the wq is never really re-enabled, though it will still have wq state set to IDXD_WQ_ENABLED.

Or am I missing something?

Regards,
Jerry

Quoted 3 lines
Quoted 3 lines
drivers/dma/idxd/irq.c +0 -0
   			rc = idxd_wq_enable(wq);
   			if (rc < 0) {
   				dev_warn(dev, "Unable to re-enable wq %s\n",
#4 lore raw
Quoted 97 lines · Dave Jiang
Quoted 78 lines · Jerry Snitselaar
Quoted 29 lines · Dave Jiang
Quoted 28 lines · Jerry Snitselaar

For a software reset idxd_device_reinit() is called, which will walk
the device workqueues to see which ones were enabled, and try to
re-enable them. It keys off wq->state being iDXD_WQ_ENABLED, but the
first thing idxd_enable_wq() will do is see that the state of the
workqueue is enabled, and return 0 instead of attempting to issue
a command to enable the workqueue.

So once a workqueue is found that needs to be re-enabled, set the state to disabled prior to calling idxd_enable_wq(). This would accurately reflect the state if the enable fails as well.

Cc Fenghua Yu <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Vinod Koul <[email protected]>
Fixes bfe1d56091c1 ("dmaengine: idxd: Init and probe for Intel

data accelerators")

---
   drivers/dma/idxd/irq.c | 1 +

   1 file changed, 1 insertion(+)
Quoted 12 lines

drivers/dma/idxd/irq.c static void idxd_device_reinit(struct · line 55

Quoted 9 lines
drivers/dma/idxd/irq.c +1 -0
index 743ead5ebc57..723eeb5328d6 100644
@@ -52,6 +52,7 @@ static void idxd_device_reinit(struct
 work_struct *work)
                 struct idxd_wq *wq = idxd->wqs[i];
                 if (wq->state == IDXD_WQ_ENABLED) {
+                       wq->state = IDXD_WQ_DISABLED;

Might be better off to insert this line in
idxd_wq_disable_cleanup(). I
think that should put it in sane state.

I don't think that is called in the code path that I was lookng at. I've been
looking at this bit of process_misc_interrupts():

halt:
        gensts.bits = ioread32(idxd->reg_base + IDXD_GENSTATS_OFFSET);
        if (gensts.state == IDXD_DEVICE_STATE_HALT) { idxd->state = IDXD_DEV_HALTED;
                if (gensts.reset_type == IDXD_DEVICE_RESET_SOFTWARE) {
                        /*
                         * If we need a software reset, we will throw the work
                         * on a system workqueue in order to allow interrupts
                         * for the device command completions. */
                        INIT_WORK(&idxd->work, idxd_device_reinit); queue_work(idxd->wq, &idxd->work); } else {
                        idxd->state = IDXD_DEV_HALTED; idxd_wqs_quiesce(idxd); idxd_wqs_unmap_portal(idxd); spin_lock(&idxd->dev_lock); idxd_device_clear_state(idxd); dev_err(&idxd->pdev->dev,
                                "idxd halted, need %s.\n", gensts.reset_type == IDXD_DEVICE_RESET_FLR ?
                                "FLR" : "system reset"); spin_unlock(&idxd->dev_lock); return -ENXIO;
                }
        }

Quoted 7 lines

        return 0;
}

So it sees that the device is halted, and sticks idxd_device_reinint() on that
workqueue. The idxd_device_reinit() has this loop to re-enable the idxd wqs:

idxd_device_reinit() should called idxd_device_reset() first. And that
should at some point call idxd_wq_disable_cleanup() and clean up the states.

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/irq.c#L42

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/device.c#L725

Quoted 8 lines

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/device.c#L711

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/device.c#L376

So if we stick the wq state reset in there, it should show up as "disabled" by the time we try to enable the WQs again. Does that look reasonable?

Ah, yeah I see that now. So, if it does set the state to disabled in idxd_wq_disable_cleanup(), does it have another means to track which wqs need to be re-enabled for that loop that happens after the idxd_device_reset() call?

Quoted 42 lines
Quoted 41 lines

        for (i = 0; i < idxd->max_wqs; i++) { struct idxd_wq *wq = idxd->wqs[i];

                if (wq->state == IDXD_WQ_ENABLED) { wq->state = IDXD_WQ_DISABLED;
                        rc = idxd_wq_enable(wq);
                        if (rc < 0) {
                                dev_warn(dev, "Unable to re-enable wq %s\n",
                                         dev_name(wq_confdev(wq))); }
                }
        }

Once you go into idxd_wq_enable() though you get this check at the beginning:

        if (wq->state == IDXD_WQ_ENABLED) { dev_dbg(dev, "WQ %d already enabled\n", wq->id); return 0;
        }

So IIUC it sees the device is halted, goes to reset it, figures out a wq
should be re-enabled, calls idxd_wq_enable() which hits the check, returns
0 and the wq is never really re-enabled, though it will still have wq state
set to IDXD_WQ_ENABLED.

Or am I missing something?

Regards,
Jerry

Quoted 4 lines
Quoted 4 lines
drivers/dma/idxd/irq.c +0 -0
                         rc = idxd_wq_enable(wq);
                         if (rc < 0) {

                                dev_warn(dev, "Unable to re- enable wq %s\n",

reviewer
#5 lore raw
Quoted 100 lines · Jerry Snitselaar
Quoted 99 lines · Dave Jiang
Quoted 78 lines · Jerry Snitselaar
Quoted 29 lines · Dave Jiang
Quoted 28 lines · Jerry Snitselaar

For a software reset idxd_device_reinit() is called, which will walk
the device workqueues to see which ones were enabled, and try to
re-enable them. It keys off wq->state being iDXD_WQ_ENABLED, but the
first thing idxd_enable_wq() will do is see that the state of the
workqueue is enabled, and return 0 instead of attempting to issue
a command to enable the workqueue.

So once a workqueue is found that needs to be re-enabled, set the state to disabled prior to calling idxd_enable_wq(). This would accurately reflect the state if the enable fails as well.

Cc Fenghua Yu <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Vinod Koul <[email protected]>
Fixes bfe1d56091c1 ("dmaengine: idxd: Init and probe for Intel

data accelerators")

---
    drivers/dma/idxd/irq.c | 1 +
    1 file changed, 1 insertion(+)
Quoted 12 lines

drivers/dma/idxd/irq.c static void idxd_device_reinit(struct · line 55

Quoted 9 lines
drivers/dma/idxd/irq.c +1 -0
index 743ead5ebc57..723eeb5328d6 100644
@@ -52,6 +52,7 @@ static void idxd_device_reinit(struct
 work_struct *work)
                 struct idxd_wq *wq = idxd->wqs[i];
                 if (wq->state == IDXD_WQ_ENABLED) {
+                       wq->state = IDXD_WQ_DISABLED;

Might be better off to insert this line in
idxd_wq_disable_cleanup(). I
think that should put it in sane state.

I don't think that is called in the code path that I was lookng at. I've been
looking at this bit of process_misc_interrupts():

halt:
        gensts.bits = ioread32(idxd->reg_base +
IDXD_GENSTATS_OFFSET);
        if (gensts.state == IDXD_DEVICE_STATE_HALT) { idxd->state = IDXD_DEV_HALTED;
                if (gensts.reset_type ==
IDXD_DEVICE_RESET_SOFTWARE) {
                        /*
                         * If we need a software reset, we will
throw the work
                         * on a system workqueue in order to allow
interrupts
                         * for the device command completions. */
                        INIT_WORK(&idxd->work, idxd_device_reinit); queue_work(idxd->wq, &idxd->work); } else {
                        idxd->state = IDXD_DEV_HALTED; idxd_wqs_quiesce(idxd); idxd_wqs_unmap_portal(idxd); spin_lock(&idxd->dev_lock); idxd_device_clear_state(idxd); dev_err(&idxd->pdev->dev,
                                "idxd halted, need %s.\n", gensts.reset_type ==
IDXD_DEVICE_RESET_FLR ?
                                "FLR" : "system reset"); spin_unlock(&idxd->dev_lock); return -ENXIO;
                }
        }

Quoted 7 lines

        return 0;
}

So it sees that the device is halted, and sticks idxd_device_reinint() on that
workqueue. The idxd_device_reinit() has this loop to re-enable the idxd wqs:

idxd_device_reinit() should called idxd_device_reset() first. And that
should at some point call idxd_wq_disable_cleanup() and clean up the states.

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/irq.c#L42

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/device.c#L725

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/device.c#L711

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/device.c#L376

Quoted 8 lines
Quoted 4 lines

So if we stick the wq state reset in there, it should show up as "disabled" by the time we try to enable the WQs again. Does that look reasonable?

Ah, yeah I see that now. So, if it does set the state to disabled in idxd_wq_disable_cleanup(), does it have another means to track which wqs need to be re-enabled for that loop that happens after the idxd_device_reset() call?

Oh I see what you mean... So we can either do what you did or create a mask and mark the WQ that are "enabled" before reset. Maybe that's cleaner rather than relying on the side effect of the WQ state isn't cleared? Thoughts?

Quoted 40 lines
Quoted 40 lines
Quoted 40 lines

        for (i = 0; i < idxd->max_wqs; i++) { struct idxd_wq *wq = idxd->wqs[i];

                if (wq->state == IDXD_WQ_ENABLED) { wq->state = IDXD_WQ_DISABLED;
                        rc = idxd_wq_enable(wq);
                        if (rc < 0) {
                                dev_warn(dev, "Unable to re-enable
wq %s\n",
                                         dev_name(wq_confdev(wq))); }
                }
        }

Once you go into idxd_wq_enable() though you get this check at the beginning:

        if (wq->state == IDXD_WQ_ENABLED) { dev_dbg(dev, "WQ %d already enabled\n", wq->id); return 0;
        }

So IIUC it sees the device is halted, goes to reset it, figures out a wq
should be re-enabled, calls idxd_wq_enable() which hits the check, returns
0 and the wq is never really re-enabled, though it will still have wq state
set to IDXD_WQ_ENABLED.

Or am I missing something?

Regards,
Jerry

Quoted 4 lines
Quoted 4 lines
drivers/dma/idxd/irq.c +0 -0
                         rc = idxd_wq_enable(wq);
                         if (rc < 0) {
                                 dev_warn(dev, "Unable to re-

enable wq %s\n",

#6 lore raw
Quoted 127 lines · Dave Jiang
Quoted 118 lines · Jerry Snitselaar
Quoted 108 lines · Dave Jiang
Quoted 89 lines · Jerry Snitselaar
Quoted 35 lines · Dave Jiang
Quoted 34 lines · Jerry Snitselaar

For a software reset idxd_device_reinit() is called, which will
walk
the device workqueues to see which ones were enabled, and try
to
re-enable them. It keys off wq->state being iDXD_WQ_ENABLED,
but the
first thing idxd_enable_wq() will do is see that the state of
the
workqueue is enabled, and return 0 instead of attempting to issue
a command to enable the workqueue.

So once a workqueue is found that needs to be re-enabled, set the state to disabled prior to calling idxd_enable_wq().
This would accurately reflect the state if the enable fails as well.

Cc Fenghua Yu <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Vinod Koul <[email protected]>
Fixes bfe1d56091c1 ("dmaengine: idxd: Init and probe for

Intel
data accelerators")

---
    drivers/dma/idxd/irq.c | 1 +

    1 file changed, 1 insertion(+)
Quoted 13 lines

drivers/dma/idxd/irq.c static void idxd_device_reinit(struct · line 55

Quoted 10 lines
drivers/dma/idxd/irq.c +1 -0
index 743ead5ebc57..723eeb5328d6 100644
@@ -52,6 +52,7 @@ static void idxd_device_reinit(struct
 work_struct *work)
                  struct idxd_wq *wq = idxd->wqs[i];
                  if (wq->state == IDXD_WQ_ENABLED) {
+                       wq->state = IDXD_WQ_DISABLED;

Might be better off to insert this line in
idxd_wq_disable_cleanup(). I
think that should put it in sane state.

I don't think that is called in the code path that I was lookng at.
I've been
looking at this bit of process_misc_interrupts():

halt:
         gensts.bits = ioread32(idxd->reg_base + IDXD_GENSTATS_OFFSET);
         if (gensts.state == IDXD_DEVICE_STATE_HALT) { idxd->state = IDXD_DEV_HALTED;
                 if (gensts.reset_type == IDXD_DEVICE_RESET_SOFTWARE) {
                         /*
                          * If we need a software reset, we will
throw the work
                          * on a system workqueue in order to allow
interrupts
                          * for the device command completions. */
                         INIT_WORK(&idxd->work, idxd_device_reinit);
                         queue_work(idxd->wq, &idxd->work); } else {
                         idxd->state = IDXD_DEV_HALTED; idxd_wqs_quiesce(idxd); idxd_wqs_unmap_portal(idxd); spin_lock(&idxd->dev_lock); idxd_device_clear_state(idxd); dev_err(&idxd->pdev->dev, "idxd halted, need %s.\n", gensts.reset_type == IDXD_DEVICE_RESET_FLR ?
                                 "FLR" : "system reset"); spin_unlock(&idxd->dev_lock); return -ENXIO;
                 }
         }

Quoted 8 lines

         return 0;
}

So it sees that the device is halted, and sticks idxd_device_reinint() on that
workqueue. The idxd_device_reinit() has this loop to re-enable the
idxd wqs:

idxd_device_reinit() should called idxd_device_reset() first. And that
should at some point call idxd_wq_disable_cleanup() and clean up the
states.

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/irq.c#L42

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/device.c#L725

Quoted 9 lines

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/device.c#L711

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/device.c#L376

So if we stick the wq state reset in there, it should show up as "disabled" by the time we try to enable the WQs again. Does that look
reasonable?

Quoted 6 lines

Ah, yeah I see that now. So, if it does set the state to disabled in
idxd_wq_disable_cleanup(), does it have another means to track which
wqs need to be re-enabled for that loop that happens after the idxd_device_reset() call?

Quoted 6 lines

Oh I see what you mean... So we can either do what you did or create a
mask and mark the WQ that are "enabled" before reset. Maybe that's cleaner rather than relying on the side effect of the WQ state isn't cleared? Thoughts?

That would be better I think.

Quoted 50 lines
Quoted 49 lines
Quoted 49 lines
Quoted 49 lines

         for (i = 0; i < idxd->max_wqs; i++) { struct idxd_wq *wq = idxd->wqs[i];

                 if (wq->state == IDXD_WQ_ENABLED) { wq->state = IDXD_WQ_DISABLED; rc = idxd_wq_enable(wq); if (rc < 0) { dev_warn(dev, "Unable to re- enable
wq %s\n",

dev_name(wq_confdev(wq)));
                         }
                 }
         }

Once you go into idxd_wq_enable() though you get this check at the
beginning:

         if (wq->state == IDXD_WQ_ENABLED) { dev_dbg(dev, "WQ %d already enabled\n", wq-

Quoted 1 line

id);

                 return 0;
         }

So IIUC it sees the device is halted, goes to reset it, figures out
a wq
should be re-enabled, calls idxd_wq_enable() which hits the check,
returns
0 and the wq is never really re-enabled, though it will still have
wq state
set to IDXD_WQ_ENABLED.

Or am I missing something?

Regards,
Jerry

Quoted 5 lines
Quoted 5 lines
drivers/dma/idxd/irq.c +0 -0
                          rc = idxd_wq_enable(wq);
                          if (rc < 0) {

                                 dev_warn(dev, "Unable to re-
enable wq %s\n",

#7 lore raw
In reply to Dave Jiang
Quoted 106 lines · Dave Jiang
Quoted 104 lines · Jerry Snitselaar
Quoted 103 lines · Dave Jiang
Quoted 78 lines · Jerry Snitselaar
Quoted 29 lines · Dave Jiang
Quoted 28 lines · Jerry Snitselaar

For a software reset idxd_device_reinit() is called, which will walk
the device workqueues to see which ones were enabled, and try to
re-enable them. It keys off wq->state being iDXD_WQ_ENABLED, but the
first thing idxd_enable_wq() will do is see that the state of the
workqueue is enabled, and return 0 instead of attempting to issue
a command to enable the workqueue.

So once a workqueue is found that needs to be re-enabled, set the state to disabled prior to calling idxd_enable_wq(). This would accurately reflect the state if the enable fails as well.

Cc Fenghua Yu <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Vinod Koul <[email protected]>
Fixes bfe1d56091c1 ("dmaengine: idxd: Init and probe for Intel

data accelerators")

---
    drivers/dma/idxd/irq.c | 1 +
    1 file changed, 1 insertion(+)
Quoted 12 lines

drivers/dma/idxd/irq.c static void idxd_device_reinit(struct · line 55

Quoted 9 lines
drivers/dma/idxd/irq.c +1 -0
index 743ead5ebc57..723eeb5328d6 100644
@@ -52,6 +52,7 @@ static void idxd_device_reinit(struct
 work_struct *work)
                 struct idxd_wq *wq = idxd->wqs[i];
                 if (wq->state == IDXD_WQ_ENABLED) {
+                       wq->state = IDXD_WQ_DISABLED;

Might be better off to insert this line in
idxd_wq_disable_cleanup(). I
think that should put it in sane state.

I don't think that is called in the code path that I was lookng at. I've been
looking at this bit of process_misc_interrupts():

halt:
        gensts.bits = ioread32(idxd->reg_base +
IDXD_GENSTATS_OFFSET);
        if (gensts.state == IDXD_DEVICE_STATE_HALT) { idxd->state = IDXD_DEV_HALTED;
                if (gensts.reset_type ==
IDXD_DEVICE_RESET_SOFTWARE) {
                        /*
                         * If we need a software reset, we will
throw the work
                         * on a system workqueue in order to allow
interrupts
                         * for the device command completions. */
                        INIT_WORK(&idxd->work, idxd_device_reinit); queue_work(idxd->wq, &idxd->work); } else {
                        idxd->state = IDXD_DEV_HALTED; idxd_wqs_quiesce(idxd); idxd_wqs_unmap_portal(idxd); spin_lock(&idxd->dev_lock); idxd_device_clear_state(idxd); dev_err(&idxd->pdev->dev,
                                "idxd halted, need %s.\n", gensts.reset_type ==
IDXD_DEVICE_RESET_FLR ?
                                "FLR" : "system reset"); spin_unlock(&idxd->dev_lock); return -ENXIO;
                }
        }

Quoted 7 lines

        return 0;
}

So it sees that the device is halted, and sticks idxd_device_reinint() on that
workqueue. The idxd_device_reinit() has this loop to re-enable the idxd wqs:

idxd_device_reinit() should called idxd_device_reset() first. And that
should at some point call idxd_wq_disable_cleanup() and clean up the states.

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/irq.c#L42

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/device.c#L725

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/device.c#L711

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/device.c#L376

So if we stick the wq state reset in there, it should show up as "disabled" by the time we try to enable the WQs again. Does that look reasonable?

Quoted 9 lines
Quoted 4 lines

Ah, yeah I see that now. So, if it does set the state to disabled in idxd_wq_disable_cleanup(), does it have another means to track which wqs need to be re-enabled for that loop that happens after the idxd_device_reset() call?

Oh I see what you mean... So we can either do what you did or create a mask and mark the WQ that are "enabled" before reset. Maybe that's cleaner rather than relying on the side effect of the WQ state isn't cleared? Thoughts?

Circling back to this. Since max_wqs could theoretically go up to 2^8, I guess this would need to be done with the bitmap_* functions?

Regards,
Jerry

Quoted 41 lines
Quoted 41 lines
Quoted 41 lines
Quoted 41 lines

        for (i = 0; i < idxd->max_wqs; i++) { struct idxd_wq *wq = idxd->wqs[i];

                if (wq->state == IDXD_WQ_ENABLED) { wq->state = IDXD_WQ_DISABLED;
                        rc = idxd_wq_enable(wq);
                        if (rc < 0) {
                                dev_warn(dev, "Unable to re-enable
wq %s\n",
                                         dev_name(wq_confdev(wq))); }
                }
        }

Once you go into idxd_wq_enable() though you get this check at the beginning:

        if (wq->state == IDXD_WQ_ENABLED) { dev_dbg(dev, "WQ %d already enabled\n", wq->id); return 0;
        }

So IIUC it sees the device is halted, goes to reset it, figures out a wq
should be re-enabled, calls idxd_wq_enable() which hits the check, returns
0 and the wq is never really re-enabled, though it will still have wq state
set to IDXD_WQ_ENABLED.

Or am I missing something?

Regards,
Jerry

Quoted 4 lines
Quoted 4 lines
drivers/dma/idxd/irq.c +0 -0
                         rc = idxd_wq_enable(wq);
                         if (rc < 0) {
                                 dev_warn(dev, "Unable to re-

enable wq %s\n",

reviewer
#8 lore raw
Quoted 110 lines · Jerry Snitselaar
Quoted 109 lines · Dave Jiang
Quoted 108 lines · Jerry Snitselaar
Quoted 95 lines · Dave Jiang
Quoted 78 lines · Jerry Snitselaar
Quoted 29 lines · Dave Jiang
Quoted 28 lines · Jerry Snitselaar

For a software reset idxd_device_reinit() is called, which will walk
the device workqueues to see which ones were enabled, and try to
re-enable them. It keys off wq->state being iDXD_WQ_ENABLED, but the
first thing idxd_enable_wq() will do is see that the state of the
workqueue is enabled, and return 0 instead of attempting to issue
a command to enable the workqueue.

So once a workqueue is found that needs to be re-enabled, set the state to disabled prior to calling idxd_enable_wq(). This would accurately reflect the state if the enable fails as well.

Cc Fenghua Yu <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Vinod Koul <[email protected]>
Fixes bfe1d56091c1 ("dmaengine: idxd: Init and probe for Intel

data accelerators")

---
     drivers/dma/idxd/irq.c | 1 +
     1 file changed, 1 insertion(+)
Quoted 12 lines

drivers/dma/idxd/irq.c static void idxd_device_reinit(struct · line 55

Quoted 9 lines
drivers/dma/idxd/irq.c +1 -0
index 743ead5ebc57..723eeb5328d6 100644
@@ -52,6 +52,7 @@ static void idxd_device_reinit(struct
 work_struct *work)
                  struct idxd_wq *wq = idxd->wqs[i];
                  if (wq->state == IDXD_WQ_ENABLED) {
+                       wq->state = IDXD_WQ_DISABLED;

Might be better off to insert this line in
idxd_wq_disable_cleanup(). I
think that should put it in sane state.

I don't think that is called in the code path that I was lookng at. I've been
looking at this bit of process_misc_interrupts():

halt:
        gensts.bits = ioread32(idxd->reg_base +
IDXD_GENSTATS_OFFSET);
        if (gensts.state == IDXD_DEVICE_STATE_HALT) { idxd->state = IDXD_DEV_HALTED;
                if (gensts.reset_type ==
IDXD_DEVICE_RESET_SOFTWARE) {
                        /*
                         * If we need a software reset, we will
throw the work
                         * on a system workqueue in order to allow
interrupts
                         * for the device command completions. */
                        INIT_WORK(&idxd->work, idxd_device_reinit); queue_work(idxd->wq, &idxd->work); } else {
                        idxd->state = IDXD_DEV_HALTED; idxd_wqs_quiesce(idxd); idxd_wqs_unmap_portal(idxd); spin_lock(&idxd->dev_lock); idxd_device_clear_state(idxd); dev_err(&idxd->pdev->dev,
                                "idxd halted, need %s.\n", gensts.reset_type ==
IDXD_DEVICE_RESET_FLR ?
                                "FLR" : "system reset"); spin_unlock(&idxd->dev_lock); return -ENXIO;
                }
        }

Quoted 7 lines

        return 0;
}

So it sees that the device is halted, and sticks idxd_device_reinint() on that
workqueue. The idxd_device_reinit() has this loop to re-enable the idxd wqs:

idxd_device_reinit() should called idxd_device_reset() first. And that
should at some point call idxd_wq_disable_cleanup() and clean up the states.

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/irq.c#L42

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/device.c#L725

Quoted 8 lines

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/device.c#L711

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/device.c#L376

So if we stick the wq state reset in there, it should show up as "disabled" by the time we try to enable the WQs again. Does that look reasonable?

Ah, yeah I see that now. So, if it does set the state to disabled in idxd_wq_disable_cleanup(), does it have another means to track which wqs need to be re-enabled for that loop that happens after the idxd_device_reset() call?

Quoted 6 lines
Quoted 4 lines

Oh I see what you mean... So we can either do what you did or create a mask and mark the WQ that are "enabled" before reset. Maybe that's cleaner rather than relying on the side effect of the WQ state isn't cleared? Thoughts?

Circling back to this. Since max_wqs could theoretically go up to 2^8, I guess this would need to be done with the bitmap_* functions?

Hi Jerry,

I wouldn't say never but I doubt any time soon for 2^8. DSA 1.0 has 8 WQs, and 2.0 (spec just went public) has 16. But yes we can use bitmap to be future proof. Are you currently working on a fix for this? Just don't want to duplicate effort if you already have something going. Thank you!

Quoted 43 lines

Regards,
Jerry

Quoted 39 lines
Quoted 39 lines
Quoted 39 lines
Quoted 39 lines
          for (i = 0; i < idxd->max_wqs; i++) {
                  struct idxd_wq *wq = idxd->wqs[i];

                if (wq->state == IDXD_WQ_ENABLED) { wq->state = IDXD_WQ_DISABLED;
                        rc = idxd_wq_enable(wq);
                        if (rc < 0) {
                                dev_warn(dev, "Unable to re-enable
wq %s\n",
                                         dev_name(wq_confdev(wq))); }
                }
        }

Once you go into idxd_wq_enable() though you get this check at the beginning:

          if (wq->state == IDXD_WQ_ENABLED) {
                  dev_dbg(dev, "WQ %d already enabled\n", wq->id);
                  return 0;
          }

So IIUC it sees the device is halted, goes to reset it, figures out a wq
should be re-enabled, calls idxd_wq_enable() which hits the check, returns
0 and the wq is never really re-enabled, though it will still have wq state
set to IDXD_WQ_ENABLED.

Or am I missing something?

Regards,
Jerry

Quoted 4 lines
Quoted 4 lines
drivers/dma/idxd/irq.c +0 -0
                          rc = idxd_wq_enable(wq);
                          if (rc < 0) {
                                  dev_warn(dev, "Unable to re-

enable wq %s\n",

#9 lore raw
Quoted 119 lines · Dave Jiang
Quoted 110 lines · Jerry Snitselaar
Quoted 109 lines · Dave Jiang
Quoted 108 lines · Jerry Snitselaar
Quoted 95 lines · Dave Jiang
Quoted 78 lines · Jerry Snitselaar
Quoted 29 lines · Dave Jiang
Quoted 28 lines · Jerry Snitselaar

For a software reset idxd_device_reinit() is called, which will walk
the device workqueues to see which ones were enabled, and try to
re-enable them. It keys off wq->state being iDXD_WQ_ENABLED, but the
first thing idxd_enable_wq() will do is see that the state of the
workqueue is enabled, and return 0 instead of attempting to issue
a command to enable the workqueue.

So once a workqueue is found that needs to be re-enabled, set the state to disabled prior to calling idxd_enable_wq(). This would accurately reflect the state if the enable fails as well.

Cc Fenghua Yu <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Vinod Koul <[email protected]>
Fixes bfe1d56091c1 ("dmaengine: idxd: Init and probe for Intel

data accelerators")

---
     drivers/dma/idxd/irq.c | 1 +
     1 file changed, 1 insertion(+)
Quoted 12 lines

drivers/dma/idxd/irq.c static void idxd_device_reinit(struct · line 55

Quoted 9 lines
drivers/dma/idxd/irq.c +1 -0
index 743ead5ebc57..723eeb5328d6 100644
@@ -52,6 +52,7 @@ static void idxd_device_reinit(struct
 work_struct *work)
                  struct idxd_wq *wq = idxd->wqs[i];
                  if (wq->state == IDXD_WQ_ENABLED) {
+                       wq->state = IDXD_WQ_DISABLED;

Might be better off to insert this line in
idxd_wq_disable_cleanup(). I
think that should put it in sane state.

I don't think that is called in the code path that I was lookng at. I've been
looking at this bit of process_misc_interrupts():

halt:
        gensts.bits = ioread32(idxd->reg_base +
IDXD_GENSTATS_OFFSET);
        if (gensts.state == IDXD_DEVICE_STATE_HALT) { idxd->state = IDXD_DEV_HALTED;
                if (gensts.reset_type ==
IDXD_DEVICE_RESET_SOFTWARE) {
                        /*
                         * If we need a software reset, we will
throw the work
                         * on a system workqueue in order to allow
interrupts
                         * for the device command completions. */
                        INIT_WORK(&idxd->work, idxd_device_reinit); queue_work(idxd->wq, &idxd->work); } else {
                        idxd->state = IDXD_DEV_HALTED; idxd_wqs_quiesce(idxd); idxd_wqs_unmap_portal(idxd); spin_lock(&idxd->dev_lock); idxd_device_clear_state(idxd); dev_err(&idxd->pdev->dev,
                                "idxd halted, need %s.\n", gensts.reset_type ==
IDXD_DEVICE_RESET_FLR ?
                                "FLR" : "system reset"); spin_unlock(&idxd->dev_lock); return -ENXIO;
                }
        }

Quoted 7 lines

        return 0;
}

So it sees that the device is halted, and sticks idxd_device_reinint() on that
workqueue. The idxd_device_reinit() has this loop to re-enable the idxd wqs:

idxd_device_reinit() should called idxd_device_reset() first. And that
should at some point call idxd_wq_disable_cleanup() and clean up the states.

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/irq.c#L42

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/device.c#L725

Quoted 8 lines

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/device.c#L711

https://elixir.bootlin.com/linux/v6.0-rc2/source/drivers/dma/idxd/device.c#L376

So if we stick the wq state reset in there, it should show up as "disabled" by the time we try to enable the WQs again. Does that look reasonable?

Ah, yeah I see that now. So, if it does set the state to disabled in idxd_wq_disable_cleanup(), does it have another means to track which wqs need to be re-enabled for that loop that happens after the idxd_device_reset() call?

Quoted 6 lines
Quoted 4 lines

Oh I see what you mean... So we can either do what you did or create a mask and mark the WQ that are "enabled" before reset. Maybe that's cleaner rather than relying on the side effect of the WQ state isn't cleared? Thoughts?

Circling back to this. Since max_wqs could theoretically go up to 2^8, I guess this would need to be done with the bitmap_* functions?

Quoted 7 lines

Hi Jerry,

I wouldn't say never but I doubt any time soon for 2^8. DSA 1.0 has 8 WQs, and 2.0 (spec just went public) has 16. But yes we can use bitmap to be future proof. Are you currently working on a fix for this? Just don't want to duplicate effort if you already have something going. Thank you!

Hi Dave,

Yes, I will try to send it this afternoon.

Regards,
Jerry

Quoted 44 lines
Quoted 44 lines

Regards,
Jerry

Quoted 39 lines
Quoted 39 lines
Quoted 39 lines
Quoted 39 lines
          for (i = 0; i < idxd->max_wqs; i++) {
                  struct idxd_wq *wq = idxd->wqs[i];

                if (wq->state == IDXD_WQ_ENABLED) { wq->state = IDXD_WQ_DISABLED;
                        rc = idxd_wq_enable(wq);
                        if (rc < 0) {
                                dev_warn(dev, "Unable to re-enable
wq %s\n",
                                         dev_name(wq_confdev(wq))); }
                }
        }

Once you go into idxd_wq_enable() though you get this check at the beginning:

          if (wq->state == IDXD_WQ_ENABLED) {
                  dev_dbg(dev, "WQ %d already enabled\n", wq->id);
                  return 0;
          }

So IIUC it sees the device is halted, goes to reset it, figures out a wq
should be re-enabled, calls idxd_wq_enable() which hits the check, returns
0 and the wq is never really re-enabled, though it will still have wq state
set to IDXD_WQ_ENABLED.

Or am I missing something?

Regards,
Jerry

Quoted 4 lines
Quoted 4 lines
drivers/dma/idxd/irq.c +0 -0
                          rc = idxd_wq_enable(wq);
                          if (rc < 0) {
                                  dev_warn(dev, "Unable to re-

enable wq %s\n",