Skip to contentSkip to conversation

fpga: dfl: afu: harden port enable logic

PATCH Superseded Russ Weight 6 participants 14 messages Sep 17, 2020 – Feb 4, 2021

Subsystems (2)

v2 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 fpga: dfl: afu: harden port enable logic

Port enable is not complete until ACK = 0. Change __afu_port_enable() to guarantee that the enable process is complete by polling for ACK == 0.

---
 drivers/fpga/dfl-afu-error.c |  2 +-
 drivers/fpga/dfl-afu-main.c  | 29 +++++++++++++++++++++--------
 drivers/fpga/dfl-afu.h       |  2 +-
 3 files changed, 23 insertions(+), 10 deletions(-)
drivers/fpga/dfl-afu-error.c +1 -1
index c4691187cca9..0806532a3e9f 100644
@@ -103,7 +103,7 @@ static int afu_port_err_clear(struct device *dev, u64 err)
 	__afu_port_err_mask(dev, false);
 
 	/* Enable the Port by clear the reset */
-	__afu_port_enable(pdev);
+	ret = __afu_port_enable(pdev);
 
 done:
 	mutex_unlock(&pdata->lock);
drivers/fpga/dfl-afu-main.c +21 -8
index 753cda4b2568..f73b06cdf13c 100644
@@ -21,6 +21,9 @@
 
 #include "dfl-afu.h"
 
+#define RST_POLL_INVL 10 /* us */
+#define RST_POLL_TIMEOUT 1000 /* us */
+
 /**
  * __afu_port_enable - enable a port by clear reset
  * @pdev: port platform device.
@@ -32,7 +35,7 @@
  *
  * The caller needs to hold lock for protection.
  */
-void __afu_port_enable(struct platform_device *pdev)
+int __afu_port_enable(struct platform_device *pdev)
 {
 	struct dfl_feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
 	void __iomem *base;
@@ -41,7 +44,7 @@ void __afu_port_enable(struct platform_device *pdev)
 	WARN_ON(!pdata->disable_count);
 
 	if (--pdata->disable_count != 0)
-		return;
+		return 0;
 
 	base = dfl_get_feature_ioaddr_by_id(&pdev->dev, PORT_FEATURE_ID_HEADER);
 
@@ -49,10 +52,20 @@ void __afu_port_enable(struct platform_device *pdev)
 	v = readq(base + PORT_HDR_CTRL);
 	v &= ~PORT_CTRL_SFTRST;
 	writeq(v, base + PORT_HDR_CTRL);
-}
 
-#define RST_POLL_INVL 10 /* us */
-#define RST_POLL_TIMEOUT 1000 /* us */
+	/*
+	 * HW clears the ack bit to indicate that the port is fully out
+	 * of reset.
+	 */
+	if (readq_poll_timeout(base + PORT_HDR_CTRL, v,
+			       !(v & PORT_CTRL_SFTRST_ACK),
+			       RST_POLL_INVL, RST_POLL_TIMEOUT)) {
+		dev_err(&pdev->dev, "timeout, failure to enable device\n");
+		return -ETIMEDOUT;
+	}
+
+	return 0;
+}
 
 /**
  * __afu_port_disable - disable a port by hold reset
@@ -111,7 +124,7 @@ static int __port_reset(struct platform_device *pdev)
 
 	ret = __afu_port_disable(pdev);
 	if (!ret)
-		__afu_port_enable(pdev);
+		ret = __afu_port_enable(pdev);
 
 	return ret;
 }
@@ -872,11 +885,11 @@ static int afu_dev_destroy(struct platform_device *pdev)
 static int port_enable_set(struct platform_device *pdev, bool enable)
 {
 	struct dfl_feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
-	int ret = 0;
+	int ret;
 
 	mutex_lock(&pdata->lock);
 	if (enable)
-		__afu_port_enable(pdev);
+		ret = __afu_port_enable(pdev);
 	else
 		ret = __afu_port_disable(pdev);
 	mutex_unlock(&pdata->lock);
drivers/fpga/dfl-afu.h +1 -1
index 576e94960086..e5020e2b1f3d 100644
@@ -80,7 +80,7 @@ struct dfl_afu {
 };
 
 /* hold pdata->lock when call __afu_port_enable/disable */
-void __afu_port_enable(struct platform_device *pdev);
+int __afu_port_enable(struct platform_device *pdev);
 int __afu_port_disable(struct platform_device *pdev);
 
 void afu_mmio_region_init(struct dfl_feature_platform_data *pdata);
Signature · 1 line
2.17.1
reviewer
#1 lore raw
Quoted 16 lines · Russ Weight

Port enable is not complete until ACK = 0. Change __afu_port_enable() to guarantee that the enable process is complete by polling for ACK == 0.

---
 drivers/fpga/dfl-afu-error.c |  2 +-
 drivers/fpga/dfl-afu-main.c  | 29 +++++++++++++++++++++--------
 drivers/fpga/dfl-afu.h       |  2 +-
 3 files changed, 23 insertions(+), 10 deletions(-)
drivers/fpga/dfl-afu-error.c +0 -0
index c4691187cca9..0806532a3e9f 100644

drivers/fpga/dfl-afu-error.c static int afu_port_err_clear(struct device *dev, u64 err) · line 104

Quoted 2 lines
drivers/fpga/dfl-afu-error.c +0 -0
@@ -103,7 +103,7 @@ static int afu_port_err_clear(struct device *dev, u64 err)
 	__afu_port_err_mask(dev, false);
 

There is an earlier bit that sets ret = -EINVAL.

This error will be lost or not handled well.

Right now it doesn't seem to be handled.

Quoted 30 lines
drivers/fpga/dfl-afu-error.c +1 -1
 	/* Enable the Port by clear the reset */
-	__afu_port_enable(pdev);
+	ret = __afu_port_enable(pdev);
 
 done:
 	mutex_unlock(&pdata->lock);
drivers/fpga/dfl-afu-main.c +4 -1
index 753cda4b2568..f73b06cdf13c 100644
@@ -21,6 +21,9 @@
 
 #include "dfl-afu.h"
 
+#define RST_POLL_INVL 10 /* us */
+#define RST_POLL_TIMEOUT 1000 /* us */
+
 /**
  * __afu_port_enable - enable a port by clear reset
  * @pdev: port platform device.
@@ -32,7 +35,7 @@
  *
  * The caller needs to hold lock for protection.
  */
-void __afu_port_enable(struct platform_device *pdev)
+int __afu_port_enable(struct platform_device *pdev)
 {
 	struct dfl_feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
 	void __iomem *base;

drivers/fpga/dfl-afu-main.c void __afu_port_enable(struct platform_device *pdev) · line 47

Quoted 5 lines
drivers/fpga/dfl-afu-main.c +1 -1
@@ -41,7 +44,7 @@ void __afu_port_enable(struct platform_device *pdev)
 	WARN_ON(!pdata->disable_count);
 
 	if (--pdata->disable_count != 0)
-		return;
+		return 0;

Is this really a success ? Maybe -EBUSY ?

Quoted 55 lines
drivers/fpga/dfl-afu-main.c +16 -6
 	base = dfl_get_feature_ioaddr_by_id(&pdev->dev, PORT_FEATURE_ID_HEADER);
 
@@ -49,10 +52,20 @@ void __afu_port_enable(struct platform_device *pdev)
 	v = readq(base + PORT_HDR_CTRL);
 	v &= ~PORT_CTRL_SFTRST;
 	writeq(v, base + PORT_HDR_CTRL);
-}
 
-#define RST_POLL_INVL 10 /* us */
-#define RST_POLL_TIMEOUT 1000 /* us */
+	/*
+	 * HW clears the ack bit to indicate that the port is fully out
+	 * of reset.
+	 */
+	if (readq_poll_timeout(base + PORT_HDR_CTRL, v,
+			       !(v & PORT_CTRL_SFTRST_ACK),
+			       RST_POLL_INVL, RST_POLL_TIMEOUT)) {
+		dev_err(&pdev->dev, "timeout, failure to enable device\n");
+		return -ETIMEDOUT;
+	}
+
+	return 0;
+}
 
 /**
  * __afu_port_disable - disable a port by hold reset
@@ -111,7 +124,7 @@ static int __port_reset(struct platform_device *pdev)
 
 	ret = __afu_port_disable(pdev);
 	if (!ret)
-		__afu_port_enable(pdev);
+		ret = __afu_port_enable(pdev);
 
 	return ret;
 }
@@ -872,11 +885,11 @@ static int afu_dev_destroy(struct platform_device *pdev)
 static int port_enable_set(struct platform_device *pdev, bool enable)
 {
 	struct dfl_feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
-	int ret = 0;
+	int ret;
 
 	mutex_lock(&pdata->lock);
 	if (enable)
-		__afu_port_enable(pdev);
+		ret = __afu_port_enable(pdev);
 	else
 		ret = __afu_port_disable(pdev);
 	mutex_unlock(&pdata->lock);
drivers/fpga/dfl-afu.h +0 -0
index 576e94960086..e5020e2b1f3d 100644

drivers/fpga/dfl-afu.h struct dfl_afu { · line 84

Quoted 6 lines
drivers/fpga/dfl-afu.h +1 -1
@@ -80,7 +80,7 @@ struct dfl_afu {
 };
 
 /* hold pdata->lock when call __afu_port_enable/disable */
-void __afu_port_enable(struct platform_device *pdev);
+int __afu_port_enable(struct platform_device *pdev);
 int __afu_port_disable(struct platform_device *pdev);

The other functions in this file have afu_*  since the __afu_port_enable/disable

are used other places would it make sense to remove the '__' prefix ?

If you think so, maybe a cleanup patch later.

Tom

Quoted 2 lines
drivers/fpga/dfl-afu.h +0 -0
 void afu_mmio_region_init(struct dfl_feature_platform_data *pdata);
maintainer
#2 lore raw
Quoted 7 lines · Tom Rix
Quoted 5 lines · Russ Weight

Port enable is not complete until ACK = 0. Change __afu_port_enable() to guarantee that the enable process is complete by polling for ACK == 0.

General note: Please keep a changelog if you send updated versions of a patch. This can be added here with an extra '---' + Text between Signed-off and diffstat:

---
Changes from v1:
- FOo
- Bar

Quoted 16 lines

drivers/fpga/dfl-afu-error.c static int afu_port_err_clear(struct device *dev, u64 err) · line 104

Quoted 13 lines
---
 drivers/fpga/dfl-afu-error.c |  2 +-
 drivers/fpga/dfl-afu-main.c  | 29 +++++++++++++++++++++--------
 drivers/fpga/dfl-afu.h       |  2 +-
 3 files changed, 23 insertions(+), 10 deletions(-)
drivers/fpga/dfl-afu-error.c +0 -0
index c4691187cca9..0806532a3e9f 100644
@@ -103,7 +103,7 @@ static int afu_port_err_clear(struct device *dev, u64 err)
 	__afu_port_err_mask(dev, false);
 

There is an earlier bit that sets ret = -EINVAL.

Quoted 3 lines

This error will be lost or not handled well.

Right now it doesn't seem to be handled.

Ultimately you'd want to report *at least* one of them, the current code seems to continue and enable the port either case. Is that what it should be doing?

Is the timeout more severe than the invalid value? Do you want to print a warning?

Either way a comment explaining why this is ok would be appreciated :)

Quoted 33 lines
Quoted 33 lines
drivers/fpga/dfl-afu-error.c +1 -1
 	/* Enable the Port by clear the reset */
-	__afu_port_enable(pdev);
+	ret = __afu_port_enable(pdev);
 
 done:
 	mutex_unlock(&pdata->lock);
drivers/fpga/dfl-afu-main.c +4 -1
index 753cda4b2568..f73b06cdf13c 100644
@@ -21,6 +21,9 @@
 
 #include "dfl-afu.h"
 
+#define RST_POLL_INVL 10 /* us */
+#define RST_POLL_TIMEOUT 1000 /* us */
+
 /**
  * __afu_port_enable - enable a port by clear reset
  * @pdev: port platform device.
@@ -32,7 +35,7 @@
  *
  * The caller needs to hold lock for protection.
  */
-void __afu_port_enable(struct platform_device *pdev)
+int __afu_port_enable(struct platform_device *pdev)
 {
 	struct dfl_feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
 	void __iomem *base;
@@ -41,7 +44,7 @@ void __afu_port_enable(struct platform_device *pdev)
 	WARN_ON(!pdata->disable_count);
 
Quoted 4 lines

drivers/fpga/dfl-afu-main.c void __afu_port_enable(struct platform_device *pdev) · line 47

Quoted 3 lines
drivers/fpga/dfl-afu-main.c +1 -1
 	if (--pdata->disable_count != 0)
-		return;
+		return 0;

Is this really a success ? Maybe -EBUSY ?

Seems like if it's severe enough for a warning you'd probably want to return an error.

Quoted 57 lines
Quoted 57 lines
drivers/fpga/dfl-afu-main.c +16 -6
 	base = dfl_get_feature_ioaddr_by_id(&pdev->dev, PORT_FEATURE_ID_HEADER);
 
@@ -49,10 +52,20 @@ void __afu_port_enable(struct platform_device *pdev)
 	v = readq(base + PORT_HDR_CTRL);
 	v &= ~PORT_CTRL_SFTRST;
 	writeq(v, base + PORT_HDR_CTRL);
-}
 
-#define RST_POLL_INVL 10 /* us */
-#define RST_POLL_TIMEOUT 1000 /* us */
+	/*
+	 * HW clears the ack bit to indicate that the port is fully out
+	 * of reset.
+	 */
+	if (readq_poll_timeout(base + PORT_HDR_CTRL, v,
+			       !(v & PORT_CTRL_SFTRST_ACK),
+			       RST_POLL_INVL, RST_POLL_TIMEOUT)) {
+		dev_err(&pdev->dev, "timeout, failure to enable device\n");
+		return -ETIMEDOUT;
+	}
+
+	return 0;
+}
 
 /**
  * __afu_port_disable - disable a port by hold reset
@@ -111,7 +124,7 @@ static int __port_reset(struct platform_device *pdev)
 
 	ret = __afu_port_disable(pdev);
 	if (!ret)
-		__afu_port_enable(pdev);
+		ret = __afu_port_enable(pdev);
 
 	return ret;
 }
@@ -872,11 +885,11 @@ static int afu_dev_destroy(struct platform_device *pdev)
 static int port_enable_set(struct platform_device *pdev, bool enable)
 {
 	struct dfl_feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
-	int ret = 0;
+	int ret;
 
 	mutex_lock(&pdata->lock);
 	if (enable)
-		__afu_port_enable(pdev);
+		ret = __afu_port_enable(pdev);
 	else
 		ret = __afu_port_disable(pdev);
 	mutex_unlock(&pdata->lock);
drivers/fpga/dfl-afu.h +0 -0
index 576e94960086..e5020e2b1f3d 100644
@@ -80,7 +80,7 @@ struct dfl_afu {
 };
 
Quoted 8 lines

drivers/fpga/dfl-afu.h struct dfl_afu { · line 84

Quoted 4 lines
drivers/fpga/dfl-afu.h +1 -1
 /* hold pdata->lock when call __afu_port_enable/disable */
-void __afu_port_enable(struct platform_device *pdev);
+int __afu_port_enable(struct platform_device *pdev);
 int __afu_port_disable(struct platform_device *pdev);

The other functions in this file have afu_*  since the __afu_port_enable/disable

are used other places would it make sense to remove the '__' prefix ?

The idea on those is to indicate that the caller need to be cautious (often a lock / mutex) is required. I think keeping them as is is fine.

Quoted 8 lines

If you think so, maybe a cleanup patch later.

Tom

Quoted 2 lines
drivers/fpga/dfl-afu.h +0 -0
 void afu_mmio_region_init(struct dfl_feature_platform_data *pdata);

Thanks,
Moritz

maintainer
#3 lore raw
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

/**
* __afu_port_enable - enable a port by clear reset
* @pdev: port platform device.

drivers/fpga/dfl-afu-main.c +1 -1
@@ -32,7 +35,7 @@
  *
  * The caller needs to hold lock for protection.
  */
-void __afu_port_enable(struct platform_device *pdev)
+int __afu_port_enable(struct platform_device *pdev)
 {
 	struct dfl_feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
 	void __iomem *base;
@@ -41,7 +44,7 @@ void __afu_port_enable(struct platform_device *pdev)
 	WARN_ON(!pdata->disable_count);
 
Quoted 6 lines
Quoted 4 lines

drivers/fpga/dfl-afu-main.c void __afu_port_enable(struct platform_device *pdev) · line 47

Quoted 3 lines
drivers/fpga/dfl-afu-main.c +1 -1
 	if (--pdata->disable_count != 0)
-		return;
+		return 0;

Is this really a success ? Maybe -EBUSY ?

Seems like if it's severe enough for a warning you'd probably want to return an error.

This code is to handle the port enable/disable request from multiple users. This is a voting mechanism, the port would not be physically enabled if there is still an disable vote. The --diable_count != 0 works for this purpose. So I think it should be OK here since the voting mechanism is working as expected.

Thanks,
Yilun

#4 lore raw
In reply to Moritz Fischer
Quoted 36 lines

Subject: Re: [PATCH v2 1/1] fpga: dfl: afu: harden port enable logic

Quoted 7 lines · Tom Rix
Quoted 5 lines · Russ Weight

Port enable is not complete until ACK = 0. Change __afu_port_enable() to guarantee that the enable process is complete by polling for ACK == 0.

General note: Please keep a changelog if you send updated versions of a patch. This can be added here with an extra '---' + Text between Signed-off and
diffstat:

---
Changes from v1:
- FOo
- Bar

Quoted 11 lines

drivers/fpga/dfl-afu-error.c static int afu_port_err_clear(struct device *dev,

Quoted 11 lines
---
 drivers/fpga/dfl-afu-error.c |  2 +-
 drivers/fpga/dfl-afu-main.c  | 29 +++++++++++++++++++++--------
 drivers/fpga/dfl-afu.h       |  2 +-
 3 files changed, 23 insertions(+), 10 deletions(-)
drivers/fpga/dfl-afu-error.c +0 -0
index c4691187cca9..0806532a3e9f 100644
@@ -103,7 +103,7 @@ static int afu_port_err_clear(struct device *dev,

u64 err)

Quoted 5 lines

drivers/fpga/dfl-afu-error.c static int afu_port_err_clear(struct device *dev, u64 err) · line 104

Quoted 2 lines
drivers/fpga/dfl-afu-error.c +0 -0
 	__afu_port_err_mask(dev, false);
 

There is an earlier bit that sets ret = -EINVAL.

Quoted 7 lines
Quoted 3 lines

This error will be lost or not handled well.

Right now it doesn't seem to be handled.

Ultimately you'd want to report *at least* one of them, the current code seems to continue and enable the port either case. Is that what it should be doing?

In order to do error clear, we have to put port into reset firstly and then clear port after error clearing is done. If we see failure during error clearing that we still want to get the port back to work at least. As we know, if port is still in reset, then the accelerator connected to the port won't work.

Quoted 3 lines

Is the timeout more severe than the invalid value? Do you want to print a warning?

Yes, It's a very bad case if port can not be enabled any more (accelerator may not be accessible any more), hardware should already be in error, it's better we have some warning messages here.

Quoted 35 lines

Either way a comment explaining why this is ok would be appreciated :)

Quoted 30 lines
Quoted 22 lines
drivers/fpga/dfl-afu-error.c +1 -1
 	/* Enable the Port by clear the reset */
-	__afu_port_enable(pdev);
+	ret = __afu_port_enable(pdev);
 
 done:
 	mutex_unlock(&pdata->lock);
drivers/fpga/dfl-afu-main.c +3 -0
index 753cda4b2568..f73b06cdf13c 100644
@@ -21,6 +21,9 @@
 
 #include "dfl-afu.h"
 
+#define RST_POLL_INVL 10 /* us */
+#define RST_POLL_TIMEOUT 1000 /* us */
+
 /**
  * __afu_port_enable - enable a port by clear reset
  * @pdev: port platform device.

drivers/fpga/dfl-afu-main.c line 40

Quoted 7 lines
drivers/fpga/dfl-afu-main.c +1 -1
@@ -32,7 +35,7 @@
  *
  * The caller needs to hold lock for protection.
  */
-void __afu_port_enable(struct platform_device *pdev)
+int __afu_port_enable(struct platform_device *pdev)
 {
 	struct dfl_feature_platform_data *pdata = dev_get_platdata(&pdev-

dev);

Quoted 2 lines

drivers/fpga/dfl-afu-main.c void __afu_port_enable(struct platform_device

Quoted 2 lines
drivers/fpga/dfl-afu-main.c +0 -0
 	void __iomem *base;
@@ -41,7 +44,7 @@ void __afu_port_enable(struct platform_device

*pdev)

Quoted 8 lines
Quoted 6 lines

drivers/fpga/dfl-afu-main.c void __afu_port_enable(struct platform_device *pdev) · line 47

Quoted 5 lines
drivers/fpga/dfl-afu-main.c +1 -1
 	WARN_ON(!pdata->disable_count);
 
 	if (--pdata->disable_count != 0)
-		return;
+		return 0;

Is this really a success ? Maybe -EBUSY ?

Seems like if it's severe enough for a warning you'd probably want to return an error.

As Yilun mentioned, this is just a reference count operation, we don't need to return error code.

Quoted 68 lines
Quoted 2 lines

drivers/fpga/dfl-afu-main.c void __afu_port_enable(struct platform_device · line 48

Quoted 2 lines
drivers/fpga/dfl-afu-main.c +0 -0
 	base = dfl_get_feature_ioaddr_by_id(&pdev->dev,

PORT_FEATURE_ID_HEADER);

Quoted 2 lines

drivers/fpga/dfl-afu-main.c void __afu_port_enable(struct platform_device

Quoted 2 lines
drivers/fpga/dfl-afu-main.c +0 -0
@@ -49,10 +52,20 @@ void __afu_port_enable(struct platform_device

*pdev)

Quoted 21 lines
Quoted 21 lines
drivers/fpga/dfl-afu-main.c +13 -3
 	v = readq(base + PORT_HDR_CTRL);
 	v &= ~PORT_CTRL_SFTRST;
 	writeq(v, base + PORT_HDR_CTRL);
-}
 
-#define RST_POLL_INVL 10 /* us */
-#define RST_POLL_TIMEOUT 1000 /* us */
+	/*
+	 * HW clears the ack bit to indicate that the port is fully out
+	 * of reset.
+	 */
+	if (readq_poll_timeout(base + PORT_HDR_CTRL, v,
+			       !(v & PORT_CTRL_SFTRST_ACK),
+			       RST_POLL_INVL, RST_POLL_TIMEOUT)) {
+		dev_err(&pdev->dev, "timeout, failure to enable device\n");
+		return -ETIMEDOUT;
+	}
+
+	return 0;
+}
 
Quoted 3 lines

drivers/fpga/dfl-afu-main.c static int __port_reset(struct platform_device

Quoted 3 lines
drivers/fpga/dfl-afu-main.c +0 -0
 /**
  * __afu_port_disable - disable a port by hold reset
@@ -111,7 +124,7 @@ static int __port_reset(struct platform_device

*pdev)

Quoted 9 lines

drivers/fpga/dfl-afu-main.c static int afu_dev_destroy(struct

Quoted 9 lines
drivers/fpga/dfl-afu-main.c +1 -1
 	ret = __afu_port_disable(pdev);
 	if (!ret)
-		__afu_port_enable(pdev);
+		ret = __afu_port_enable(pdev);
 
 	return ret;
 }
@@ -872,11 +885,11 @@ static int afu_dev_destroy(struct

platform_device *pdev)

Quoted 21 lines

drivers/fpga/dfl-afu-main.c static int afu_dev_destroy(struct platform_device *pdev)

Quoted 3 lines
drivers/fpga/dfl-afu-main.c +0 -0
 static int port_enable_set(struct platform_device *pdev, bool enable)
 {
 	struct dfl_feature_platform_data *pdata = dev_get_platdata(&pdev-

dev);

Quoted 17 lines
drivers/fpga/dfl-afu-main.c +2 -2
-	int ret = 0;
+	int ret;
 
 	mutex_lock(&pdata->lock);
 	if (enable)
-		__afu_port_enable(pdev);
+		ret = __afu_port_enable(pdev);
 	else
 		ret = __afu_port_disable(pdev);
 	mutex_unlock(&pdata->lock);
drivers/fpga/dfl-afu.h +0 -0
index 576e94960086..e5020e2b1f3d 100644
@@ -80,7 +80,7 @@ struct dfl_afu {
 };
 
Quoted 6 lines

drivers/fpga/dfl-afu.h struct dfl_afu { · line 84

Quoted 4 lines
drivers/fpga/dfl-afu.h +1 -1
 /* hold pdata->lock when call __afu_port_enable/disable */
-void __afu_port_enable(struct platform_device *pdev);
+int __afu_port_enable(struct platform_device *pdev);
 int __afu_port_disable(struct platform_device *pdev);

The other functions in this file have afu_*  since the

Quoted 6 lines

__afu_port_enable/disable

Quoted 2 lines

are used other places would it make sense to remove the '__' prefix ?

The idea on those is to indicate that the caller need to be cautious (often a lock / mutex) is required. I think keeping them as is is fine.

Yes. That's why we add the prefix for these functions.

Thanks
Hao

Quoted 12 lines
Quoted 9 lines

If you think so, maybe a cleanup patch later.

Tom

Quoted 2 lines
drivers/fpga/dfl-afu.h +0 -0
 void afu_mmio_region_init(struct dfl_feature_platform_data *pdata);

Thanks,
Moritz

#5 lore raw
In reply to Moritz Fischer
Quoted 7 lines · Moritz Fischer
Quoted 6 lines · Tom Rix
Quoted 5 lines · Russ Weight

Port enable is not complete until ACK = 0. Change __afu_port_enable() to guarantee that the enable process is complete by polling for ACK == 0.

Quoted 8 lines

General note: Please keep a changelog if you send updated versions of a patch. This can be added here with an extra '---' + Text between Signed-off and diffstat:

---
Changes from v1:
- FOo
- Bar

Yes - I'll do that on future patch updates. In this case v2 just fixed a typo in the commit message, so the patch was essentially the same as v1.

Quoted 18 lines
Quoted 15 lines

drivers/fpga/dfl-afu-error.c static int afu_port_err_clear(struct device *dev, u64 err) · line 104

Quoted 13 lines
---
 drivers/fpga/dfl-afu-error.c |  2 +-
 drivers/fpga/dfl-afu-main.c  | 29 +++++++++++++++++++++--------
 drivers/fpga/dfl-afu.h       |  2 +-
 3 files changed, 23 insertions(+), 10 deletions(-)
drivers/fpga/dfl-afu-error.c +0 -0
index c4691187cca9..0806532a3e9f 100644
@@ -103,7 +103,7 @@ static int afu_port_err_clear(struct device *dev, u64 err)
 	__afu_port_err_mask(dev, false);
 

There is an earlier bit that sets ret = -EINVAL.

Quoted 3 lines

This error will be lost or not handled well.

Right now it doesn't seem to be handled.

Quoted 8 lines

Ultimately you'd want to report *at least* one of them, the current code seems to continue and enable the port either case. Is that what it should be doing?

Is the timeout more severe than the invalid value? Do you want to print a warning?

Either way a comment explaining why this is ok would be appreciated :)

Yes - I'll add a comment explaining how the errors arebeing prioritized. I'll give priority to the timeout, asit is likely a HW failure.

Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
drivers/fpga/dfl-afu-error.c +1 -1
 	/* Enable the Port by clear the reset */
-	__afu_port_enable(pdev);
+	ret = __afu_port_enable(pdev);
 
 done:
 	mutex_unlock(&pdata->lock);
drivers/fpga/dfl-afu-main.c +4 -1
index 753cda4b2568..f73b06cdf13c 100644
@@ -21,6 +21,9 @@
 
 #include "dfl-afu.h"
 
+#define RST_POLL_INVL 10 /* us */
+#define RST_POLL_TIMEOUT 1000 /* us */
+
 /**
  * __afu_port_enable - enable a port by clear reset
  * @pdev: port platform device.
@@ -32,7 +35,7 @@
  *
  * The caller needs to hold lock for protection.
  */
-void __afu_port_enable(struct platform_device *pdev)
+int __afu_port_enable(struct platform_device *pdev)
 {
 	struct dfl_feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
 	void __iomem *base;
@@ -41,7 +44,7 @@ void __afu_port_enable(struct platform_device *pdev)
 	WARN_ON(!pdata->disable_count);
 
Quoted 6 lines
Quoted 4 lines

drivers/fpga/dfl-afu-main.c void __afu_port_enable(struct platform_device *pdev) · line 47

Quoted 3 lines
drivers/fpga/dfl-afu-main.c +1 -1
 	if (--pdata->disable_count != 0)
-		return;
+		return 0;

Is this really a success ? Maybe -EBUSY ?

Seems like if it's severe enough for a warning you'd probably want to return an error.

As mentioned by Hao and Yilun, the disable_count is a reference count.The WARN_ON() is checking for a different condition - an invalid reference count. We should never call port_enable if the port is not disabled. Do you think a comment is needed here?

Thanks,
- Russ

Quoted 75 lines
Quoted 57 lines
Quoted 57 lines
drivers/fpga/dfl-afu-main.c +16 -6
 	base = dfl_get_feature_ioaddr_by_id(&pdev->dev, PORT_FEATURE_ID_HEADER);
 
@@ -49,10 +52,20 @@ void __afu_port_enable(struct platform_device *pdev)
 	v = readq(base + PORT_HDR_CTRL);
 	v &= ~PORT_CTRL_SFTRST;
 	writeq(v, base + PORT_HDR_CTRL);
-}
 
-#define RST_POLL_INVL 10 /* us */
-#define RST_POLL_TIMEOUT 1000 /* us */
+	/*
+	 * HW clears the ack bit to indicate that the port is fully out
+	 * of reset.
+	 */
+	if (readq_poll_timeout(base + PORT_HDR_CTRL, v,
+			       !(v & PORT_CTRL_SFTRST_ACK),
+			       RST_POLL_INVL, RST_POLL_TIMEOUT)) {
+		dev_err(&pdev->dev, "timeout, failure to enable device\n");
+		return -ETIMEDOUT;
+	}
+
+	return 0;
+}
 
 /**
  * __afu_port_disable - disable a port by hold reset
@@ -111,7 +124,7 @@ static int __port_reset(struct platform_device *pdev)
 
 	ret = __afu_port_disable(pdev);
 	if (!ret)
-		__afu_port_enable(pdev);
+		ret = __afu_port_enable(pdev);
 
 	return ret;
 }
@@ -872,11 +885,11 @@ static int afu_dev_destroy(struct platform_device *pdev)
 static int port_enable_set(struct platform_device *pdev, bool enable)
 {
 	struct dfl_feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
-	int ret = 0;
+	int ret;
 
 	mutex_lock(&pdata->lock);
 	if (enable)
-		__afu_port_enable(pdev);
+		ret = __afu_port_enable(pdev);
 	else
 		ret = __afu_port_disable(pdev);
 	mutex_unlock(&pdata->lock);
drivers/fpga/dfl-afu.h +0 -0
index 576e94960086..e5020e2b1f3d 100644
@@ -80,7 +80,7 @@ struct dfl_afu {
 };
 
Quoted 7 lines

drivers/fpga/dfl-afu.h struct dfl_afu { · line 84

Quoted 4 lines
drivers/fpga/dfl-afu.h +1 -1
 /* hold pdata->lock when call __afu_port_enable/disable */
-void __afu_port_enable(struct platform_device *pdev);
+int __afu_port_enable(struct platform_device *pdev);
 int __afu_port_disable(struct platform_device *pdev);

The other functions in this file have afu_*  since the __afu_port_enable/disable

are used other places would it make sense to remove the '__' prefix ?

The idea on those is to indicate that the caller need to be cautious (often a lock / mutex) is required. I think keeping them as is is fine.

Quoted 6 lines

If you think so, maybe a cleanup patch later.

Tom

Quoted 2 lines
drivers/fpga/dfl-afu.h +0 -0
 void afu_mmio_region_init(struct dfl_feature_platform_data *pdata);

Thanks,
Moritz

#6 lore raw
In reply to Tom Rix
Quoted 21 lines · Tom Rix
Quoted 16 lines · Russ Weight

Port enable is not complete until ACK = 0. Change __afu_port_enable() to guarantee that the enable process is complete by polling for ACK == 0.

---
 drivers/fpga/dfl-afu-error.c |  2 +-
 drivers/fpga/dfl-afu-main.c  | 29 +++++++++++++++++++++--------
 drivers/fpga/dfl-afu.h       |  2 +-
 3 files changed, 23 insertions(+), 10 deletions(-)
drivers/fpga/dfl-afu-error.c +0 -0
index c4691187cca9..0806532a3e9f 100644

drivers/fpga/dfl-afu-error.c static int afu_port_err_clear(struct device *dev, u64 err) · line 104

Quoted 2 lines
drivers/fpga/dfl-afu-error.c +0 -0
@@ -103,7 +103,7 @@ static int afu_port_err_clear(struct device *dev, u64 err)
 	__afu_port_err_mask(dev, false);
 

There is an earlier bit that sets ret = -EINVAL.

Quoted 3 lines

This error will be lost or not handled well.

Right now it doesn't seem to be handled.

Good catch. I'll give priority to -EINVAL in the next version of the patch, as it is more informative in the context of this function.

Quoted 33 lines
Quoted 33 lines
drivers/fpga/dfl-afu-error.c +1 -1
 	/* Enable the Port by clear the reset */
-	__afu_port_enable(pdev);
+	ret = __afu_port_enable(pdev);
 
 done:
 	mutex_unlock(&pdata->lock);
drivers/fpga/dfl-afu-main.c +4 -1
index 753cda4b2568..f73b06cdf13c 100644
@@ -21,6 +21,9 @@
 
 #include "dfl-afu.h"
 
+#define RST_POLL_INVL 10 /* us */
+#define RST_POLL_TIMEOUT 1000 /* us */
+
 /**
  * __afu_port_enable - enable a port by clear reset
  * @pdev: port platform device.
@@ -32,7 +35,7 @@
  *
  * The caller needs to hold lock for protection.
  */
-void __afu_port_enable(struct platform_device *pdev)
+int __afu_port_enable(struct platform_device *pdev)
 {
 	struct dfl_feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
 	void __iomem *base;
@@ -41,7 +44,7 @@ void __afu_port_enable(struct platform_device *pdev)
 	WARN_ON(!pdata->disable_count);
 
Quoted 4 lines

drivers/fpga/dfl-afu-main.c void __afu_port_enable(struct platform_device *pdev) · line 47

Quoted 3 lines
drivers/fpga/dfl-afu-main.c +1 -1
 	if (--pdata->disable_count != 0)
-		return;
+		return 0;

Is this really a success ? Maybe -EBUSY ?

Yilun addressed this question in his previous response. This isessentially a reference count for nested disable calls. Weonly do the enable if the disable count has gone to zero, so this isn't an error condition.

Quoted 63 lines
Quoted 55 lines
drivers/fpga/dfl-afu-main.c +16 -6
 	base = dfl_get_feature_ioaddr_by_id(&pdev->dev, PORT_FEATURE_ID_HEADER);
 
@@ -49,10 +52,20 @@ void __afu_port_enable(struct platform_device *pdev)
 	v = readq(base + PORT_HDR_CTRL);
 	v &= ~PORT_CTRL_SFTRST;
 	writeq(v, base + PORT_HDR_CTRL);
-}
 
-#define RST_POLL_INVL 10 /* us */
-#define RST_POLL_TIMEOUT 1000 /* us */
+	/*
+	 * HW clears the ack bit to indicate that the port is fully out
+	 * of reset.
+	 */
+	if (readq_poll_timeout(base + PORT_HDR_CTRL, v,
+			       !(v & PORT_CTRL_SFTRST_ACK),
+			       RST_POLL_INVL, RST_POLL_TIMEOUT)) {
+		dev_err(&pdev->dev, "timeout, failure to enable device\n");
+		return -ETIMEDOUT;
+	}
+
+	return 0;
+}
 
 /**
  * __afu_port_disable - disable a port by hold reset
@@ -111,7 +124,7 @@ static int __port_reset(struct platform_device *pdev)
 
 	ret = __afu_port_disable(pdev);
 	if (!ret)
-		__afu_port_enable(pdev);
+		ret = __afu_port_enable(pdev);
 
 	return ret;
 }
@@ -872,11 +885,11 @@ static int afu_dev_destroy(struct platform_device *pdev)
 static int port_enable_set(struct platform_device *pdev, bool enable)
 {
 	struct dfl_feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
-	int ret = 0;
+	int ret;
 
 	mutex_lock(&pdata->lock);
 	if (enable)
-		__afu_port_enable(pdev);
+		ret = __afu_port_enable(pdev);
 	else
 		ret = __afu_port_disable(pdev);
 	mutex_unlock(&pdata->lock);
drivers/fpga/dfl-afu.h +0 -0
index 576e94960086..e5020e2b1f3d 100644

drivers/fpga/dfl-afu.h struct dfl_afu { · line 84

Quoted 6 lines
drivers/fpga/dfl-afu.h +1 -1
@@ -80,7 +80,7 @@ struct dfl_afu {
 };
 
 /* hold pdata->lock when call __afu_port_enable/disable */
-void __afu_port_enable(struct platform_device *pdev);
+int __afu_port_enable(struct platform_device *pdev);
 int __afu_port_disable(struct platform_device *pdev);

The other functions in this file have afu_*  since the __afu_port_enable/disable

Quoted 3 lines

are used other places would it make sense to remove the '__' prefix ?

If you think so, maybe a cleanup patch later.

Yilun and Hao addressed this comment in their previous responses. We are using the '__' prefix to indicate highlight the fact caller needs to use care in managing the locking associated with these functions.

Thanks,
- Russ

Quoted 5 lines

Tom

Quoted 2 lines
drivers/fpga/dfl-afu.h +0 -0
 void afu_mmio_region_init(struct dfl_feature_platform_data *pdata);
#7 lore raw
Quoted 23 lines · Russ Weight
Quoted 21 lines · Tom Rix
Quoted 16 lines · Russ Weight

Port enable is not complete until ACK = 0. Change __afu_port_enable() to guarantee that the enable process is complete by polling for ACK == 0.

---
 drivers/fpga/dfl-afu-error.c |  2 +-
 drivers/fpga/dfl-afu-main.c  | 29 +++++++++++++++++++++--------
 drivers/fpga/dfl-afu.h       |  2 +-
 3 files changed, 23 insertions(+), 10 deletions(-)
drivers/fpga/dfl-afu-error.c +0 -0
index c4691187cca9..0806532a3e9f 100644

drivers/fpga/dfl-afu-error.c static int afu_port_err_clear(struct device *dev, u64 err) · line 104

Quoted 2 lines
drivers/fpga/dfl-afu-error.c +0 -0
@@ -103,7 +103,7 @@ static int afu_port_err_clear(struct device *dev, u64 err)
 	__afu_port_err_mask(dev, false);
 

There is an earlier bit that sets ret = -EINVAL.

Quoted 5 lines
Quoted 3 lines

This error will be lost or not handled well.

Right now it doesn't seem to be handled.

Good catch. I'll give priority to -EINVAL in the next version of the patch, as it is more informative in the context of this function.

Actually - Hao pointed out in his response that the falure to re-enable the port is a more serious error, so the code flow OK, but needs a comment.

- Russ

Quoted 115 lines
Quoted 32 lines
Quoted 32 lines
drivers/fpga/dfl-afu-error.c +1 -1
 	/* Enable the Port by clear the reset */
-	__afu_port_enable(pdev);
+	ret = __afu_port_enable(pdev);
 
 done:
 	mutex_unlock(&pdata->lock);
drivers/fpga/dfl-afu-main.c +4 -1
index 753cda4b2568..f73b06cdf13c 100644
@@ -21,6 +21,9 @@
 
 #include "dfl-afu.h"
 
+#define RST_POLL_INVL 10 /* us */
+#define RST_POLL_TIMEOUT 1000 /* us */
+
 /**
  * __afu_port_enable - enable a port by clear reset
  * @pdev: port platform device.
@@ -32,7 +35,7 @@
  *
  * The caller needs to hold lock for protection.
  */
-void __afu_port_enable(struct platform_device *pdev)
+int __afu_port_enable(struct platform_device *pdev)
 {
 	struct dfl_feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
 	void __iomem *base;
@@ -41,7 +44,7 @@ void __afu_port_enable(struct platform_device *pdev)
 	WARN_ON(!pdata->disable_count);
 
Quoted 4 lines

drivers/fpga/dfl-afu-main.c void __afu_port_enable(struct platform_device *pdev) · line 47

Quoted 3 lines
drivers/fpga/dfl-afu-main.c +1 -1
 	if (--pdata->disable_count != 0)
-		return;
+		return 0;

Is this really a success ? Maybe -EBUSY ?

Yilun addressed this question in his previous response. This isessentially a reference count for nested disable calls. Weonly do the enable if the disable count has gone to zero, so this isn't an error condition.

Quoted 63 lines
Quoted 55 lines
drivers/fpga/dfl-afu-main.c +16 -6
 	base = dfl_get_feature_ioaddr_by_id(&pdev->dev, PORT_FEATURE_ID_HEADER);
 
@@ -49,10 +52,20 @@ void __afu_port_enable(struct platform_device *pdev)
 	v = readq(base + PORT_HDR_CTRL);
 	v &= ~PORT_CTRL_SFTRST;
 	writeq(v, base + PORT_HDR_CTRL);
-}
 
-#define RST_POLL_INVL 10 /* us */
-#define RST_POLL_TIMEOUT 1000 /* us */
+	/*
+	 * HW clears the ack bit to indicate that the port is fully out
+	 * of reset.
+	 */
+	if (readq_poll_timeout(base + PORT_HDR_CTRL, v,
+			       !(v & PORT_CTRL_SFTRST_ACK),
+			       RST_POLL_INVL, RST_POLL_TIMEOUT)) {
+		dev_err(&pdev->dev, "timeout, failure to enable device\n");
+		return -ETIMEDOUT;
+	}
+
+	return 0;
+}
 
 /**
  * __afu_port_disable - disable a port by hold reset
@@ -111,7 +124,7 @@ static int __port_reset(struct platform_device *pdev)
 
 	ret = __afu_port_disable(pdev);
 	if (!ret)
-		__afu_port_enable(pdev);
+		ret = __afu_port_enable(pdev);
 
 	return ret;
 }
@@ -872,11 +885,11 @@ static int afu_dev_destroy(struct platform_device *pdev)
 static int port_enable_set(struct platform_device *pdev, bool enable)
 {
 	struct dfl_feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
-	int ret = 0;
+	int ret;
 
 	mutex_lock(&pdata->lock);
 	if (enable)
-		__afu_port_enable(pdev);
+		ret = __afu_port_enable(pdev);
 	else
 		ret = __afu_port_disable(pdev);
 	mutex_unlock(&pdata->lock);
drivers/fpga/dfl-afu.h +0 -0
index 576e94960086..e5020e2b1f3d 100644

drivers/fpga/dfl-afu.h struct dfl_afu { · line 84

Quoted 6 lines
drivers/fpga/dfl-afu.h +1 -1
@@ -80,7 +80,7 @@ struct dfl_afu {
 };
 
 /* hold pdata->lock when call __afu_port_enable/disable */
-void __afu_port_enable(struct platform_device *pdev);
+int __afu_port_enable(struct platform_device *pdev);
 int __afu_port_disable(struct platform_device *pdev);

The other functions in this file have afu_*  since the __afu_port_enable/disable

Quoted 3 lines

are used other places would it make sense to remove the '__' prefix ?

If you think so, maybe a cleanup patch later.

Yilun and Hao addressed this comment in their previous responses. We are using the '__' prefix to indicate highlight the fact caller needs to use care in managing the locking associated with these functions.

Thanks,
- Russ

Quoted 4 lines

Tom

Quoted 2 lines
drivers/fpga/dfl-afu.h +0 -0
 void afu_mmio_region_init(struct dfl_feature_platform_data *pdata);
#8 lore raw
In reply to Russ Weight
Quoted 11 lines

-----Original Message-----
From: Russ Weight <[email protected]>
Sent: Friday, September 18, 2020 2:32 AM
To: [email protected]; [email protected]; linux- [email protected]
Cc: [email protected]; [email protected]; Xu, Yilun <[email protected]>; Wu, Hao <[email protected]>; Gerlach, Matthew <[email protected]>; Weight, Russell H <[email protected]>
Subject: [PATCH v2 1/1] fpga: dfl: afu: harden port enable logic

Quoted 3 lines

Port enable is not complete until ACK = 0. Change __afu_port_enable() to guarantee that the enable process is complete by polling for ACK == 0.

The description of this port reset ack bit is

" After initiating a Port soft reset, SW should monitor this bit. HW will set this bit when all outstanding requests initiated by this port have been drained, and the minimum soft reset pulse width has elapsed. "

But no description about what to do when clearing a Port soft reset to enable the port.

So we need to understand clearly on why we need this change (e.g. what may happen without this change), and will it apply for all existing DFL devices and future ones, or just for one specific card. Could you please help? : )

Quoted 82 lines
---
 drivers/fpga/dfl-afu-error.c |  2 +-
 drivers/fpga/dfl-afu-main.c  | 29 +++++++++++++++++++++--------
 drivers/fpga/dfl-afu.h       |  2 +-
 3 files changed, 23 insertions(+), 10 deletions(-)
drivers/fpga/dfl-afu-error.c +1 -1
index c4691187cca9..0806532a3e9f 100644
@@ -103,7 +103,7 @@ static int afu_port_err_clear(struct device *dev, u64
 err)
 	__afu_port_err_mask(dev, false);
 
 	/* Enable the Port by clear the reset */
-	__afu_port_enable(pdev);
+	ret = __afu_port_enable(pdev);
 
 done:
drivers/fpga/dfl-afu-error.c +0 -0
 	mutex_unlock(&pdata->lock);
drivers/fpga/dfl-afu-main.c +4 -1
index 753cda4b2568..f73b06cdf13c 100644
@@ -21,6 +21,9 @@
 
 #include "dfl-afu.h"
 
+#define RST_POLL_INVL 10 /* us */
+#define RST_POLL_TIMEOUT 1000 /* us */
+
 /**
  * __afu_port_enable - enable a port by clear reset
  * @pdev: port platform device.
@@ -32,7 +35,7 @@
  *
  * The caller needs to hold lock for protection.
  */
-void __afu_port_enable(struct platform_device *pdev)
+int __afu_port_enable(struct platform_device *pdev)
 {
 	struct dfl_feature_platform_data *pdata = dev_get_platdata(&pdev-
 >dev);
drivers/fpga/dfl-afu-main.c +14 -4
 	void __iomem *base;
@@ -41,7 +44,7 @@ void __afu_port_enable(struct platform_device *pdev)
 	WARN_ON(!pdata->disable_count);
 
 	if (--pdata->disable_count != 0)
-		return;
+		return 0;
 
 	base = dfl_get_feature_ioaddr_by_id(&pdev->dev,
 PORT_FEATURE_ID_HEADER);
@@ -49,10 +52,20 @@ void __afu_port_enable(struct platform_device
 *pdev)
 	v = readq(base + PORT_HDR_CTRL);
 	v &= ~PORT_CTRL_SFTRST;
 	writeq(v, base + PORT_HDR_CTRL);
-}
 
-#define RST_POLL_INVL 10 /* us */
-#define RST_POLL_TIMEOUT 1000 /* us */
+	/*
+	 * HW clears the ack bit to indicate that the port is fully out
+	 * of reset.
+	 */
+	if (readq_poll_timeout(base + PORT_HDR_CTRL, v,
+			       !(v & PORT_CTRL_SFTRST_ACK),
+			       RST_POLL_INVL, RST_POLL_TIMEOUT)) {
+		dev_err(&pdev->dev, "timeout, failure to enable device\n");
+		return -ETIMEDOUT;
+	}
+
+	return 0;
+}
 
 /**

* __afu_port_disable - disable a port by hold reset

drivers/fpga/dfl-afu-main.c static int __port_reset(struct platform_device *pdev) · line 129

Quoted 7 lines
drivers/fpga/dfl-afu-main.c +1 -1
@@ -111,7 +124,7 @@ static int __port_reset(struct platform_device *pdev)
 
 	ret = __afu_port_disable(pdev);
 	if (!ret)
-		__afu_port_enable(pdev);
+		ret = __afu_port_enable(pdev);
 
 	return ret;

What about:

	ret = __afu_port_disable(pdev);
	if (ret)
		return ret;

	return __afu_port_enable(pdev);

Thanks
Hao

Quoted 32 lines
drivers/fpga/dfl-afu-main.c +2 -2
 }
@@ -872,11 +885,11 @@ static int afu_dev_destroy(struct platform_device
 *pdev)
 static int port_enable_set(struct platform_device *pdev, bool enable)
 {
 	struct dfl_feature_platform_data *pdata = dev_get_platdata(&pdev-
 >dev);
-	int ret = 0;
+	int ret;
 
 	mutex_lock(&pdata->lock);
 	if (enable)
-		__afu_port_enable(pdev);
+		ret = __afu_port_enable(pdev);
 	else
drivers/fpga/dfl-afu-main.c +0 -0
 		ret = __afu_port_disable(pdev);
 	mutex_unlock(&pdata->lock);
drivers/fpga/dfl-afu.h +1 -2
index 576e94960086..e5020e2b1f3d 100644
@@ -80,7 +80,7 @@ struct dfl_afu {
 };
 
 /* hold pdata->lock when call __afu_port_enable/disable */
-void __afu_port_enable(struct platform_device *pdev);
+int __afu_port_enable(struct platform_device *pdev);
 int __afu_port_disable(struct platform_device *pdev);
 
 void afu_mmio_region_init(struct dfl_feature_platform_data *pdata);
--

2.17.1

#9 lore raw

Sorry for the delay on this patch. It seemed like a lower priority patch than others, since we haven't seen any issues with current products. Please my responses inline.

Quoted 21 lines · Hao
Quoted 11 lines

-----Original Message-----
From: Russ Weight <[email protected]>
Sent: Friday, September 18, 2020 2:32 AM
To: [email protected]; [email protected]; linux- [email protected]
Cc: [email protected]; [email protected]; Xu, Yilun <[email protected]>; Wu, Hao <[email protected]>; Gerlach, Matthew <[email protected]>; Weight, Russell H <[email protected]>
Subject: [PATCH v2 1/1] fpga: dfl: afu: harden port enable logic

Quoted 3 lines

Port enable is not complete until ACK = 0. Change __afu_port_enable() to guarantee that the enable process is complete by polling for ACK == 0.

The description of this port reset ack bit is

" After initiating a Port soft reset, SW should monitor this bit. HW will set this bit when all outstanding requests initiated by this port have been drained, and the minimum soft reset pulse width has elapsed. "

Quoted 7 lines

But no description about what to do when clearing a Port soft reset to enable the port.

So we need to understand clearly on why we need this change (e.g. what may happen without this change), and will it apply for all existing DFL devices and future ones, or just for one specific card. Could you please help? : )

I touched bases with the hardware engineers. The recommendation to wait for ACK to be cleared is new with OFS and is documented in the latest OFS specification as follows (see step #4):

Quoted 9 lines

3.7.1 AFU Soft Resets
Software may cause a soft reset to be issued to the AFU as follows:
1. Assert the PortSoftReset field of the PORT_CONTROL register
2. Wait for the Port to acknowledge the soft reset by monitoring the PortSoftResetAck field of the PORT_CONTROL register, i.e. PortSoftResetAck=1
3. Deasserting the PortSoftReset field
4. Wait for the Port to acknowledge the soft reset de-assertion by monitoring the PortSoftResetAck field of the PORT_CONTROL register, i.e. PortSoftResetAck=0

Quoted 3 lines

This sequence ensures that outstanding transactions are suitably flushed and that the FIM minimum reset pulse width is respected. Failing to follow this sequence leaves the AFU in an undefined state.

The OFS specification has not been posted publicly, yet.

Also, this is how it was explained to me:

Quoted 8 lines

In most scenario, port will be able to get out of reset soon enough when SW releases the port reset, especially on all the PAC products which have been verified before release.

Polling for HW to clear the ACK is meant to handle the following scenarios:

* Different platform can take variable period of time to get out of reset
* Bug in the HW that hold the port in reset

So this change is not required for the currently released PAC cards, but it is needed for OFS based products. I don't think there is any reason to hold off on the patch, as it is still valid for current products.

Quoted 87 lines
Quoted 87 lines
---
 drivers/fpga/dfl-afu-error.c |  2 +-
 drivers/fpga/dfl-afu-main.c  | 29 +++++++++++++++++++++--------
 drivers/fpga/dfl-afu.h       |  2 +-
 3 files changed, 23 insertions(+), 10 deletions(-)
drivers/fpga/dfl-afu-error.c +1 -1
index c4691187cca9..0806532a3e9f 100644
@@ -103,7 +103,7 @@ static int afu_port_err_clear(struct device *dev, u64
 err)
 __afu_port_err_mask(dev, false);
 
 /* Enable the Port by clear the reset */
-__afu_port_enable(pdev);
+ret = __afu_port_enable(pdev);
 
 done:

mutex_unlock(&pdata->lock);

drivers/fpga/dfl-afu-main.c +4 -1
index 753cda4b2568..f73b06cdf13c 100644
@@ -21,6 +21,9 @@
 
 #include "dfl-afu.h"
 
+#define RST_POLL_INVL 10 /* us */
+#define RST_POLL_TIMEOUT 1000 /* us */
+
 /**
  * __afu_port_enable - enable a port by clear reset
  * @pdev: port platform device.
@@ -32,7 +35,7 @@
  *
  * The caller needs to hold lock for protection.
  */
-void __afu_port_enable(struct platform_device *pdev)
+int __afu_port_enable(struct platform_device *pdev)
 {
 struct dfl_feature_platform_data *pdata = dev_get_platdata(&pdev-
 > dev);

void __iomem *base;

drivers/fpga/dfl-afu-main.c +14 -4
@@ -41,7 +44,7 @@ void __afu_port_enable(struct platform_device *pdev)
 WARN_ON(!pdata->disable_count);
 
 if (--pdata->disable_count != 0)
-return;
+return 0;
 
 base = dfl_get_feature_ioaddr_by_id(&pdev->dev,
 PORT_FEATURE_ID_HEADER);
@@ -49,10 +52,20 @@ void __afu_port_enable(struct platform_device
 *pdev)
 v = readq(base + PORT_HDR_CTRL);
 v &= ~PORT_CTRL_SFTRST;
 writeq(v, base + PORT_HDR_CTRL);
-}
 
-#define RST_POLL_INVL 10 /* us */
-#define RST_POLL_TIMEOUT 1000 /* us */
+/*
+ * HW clears the ack bit to indicate that the port is fully out
+ * of reset.
+ */
+if (readq_poll_timeout(base + PORT_HDR_CTRL, v,
+       !(v & PORT_CTRL_SFTRST_ACK),
+       RST_POLL_INVL, RST_POLL_TIMEOUT)) {
+dev_err(&pdev->dev, "timeout, failure to enable device\n");
+return -ETIMEDOUT;
+}
+
+return 0;
+}
 
 /**

* __afu_port_disable - disable a port by hold reset

drivers/fpga/dfl-afu-main.c +1 -1
@@ -111,7 +124,7 @@ static int __port_reset(struct platform_device *pdev)
 
 ret = __afu_port_disable(pdev);
 if (!ret)
-__afu_port_enable(pdev);
+ret = __afu_port_enable(pdev);
 
Quoted 8 lines

drivers/fpga/dfl-afu-main.c static int __port_reset(struct platform_device *pdev) · line 129

Quoted 1 line
drivers/fpga/dfl-afu-main.c +0 -0
 return ret;

What about:

ret = __afu_port_disable(pdev);
if (ret)
return ret;

return __afu_port_enable(pdev);

Sure - I'll make this change.

Thanks,
- Russ

Quoted 36 lines

Thanks
Hao

Quoted 32 lines
drivers/fpga/dfl-afu-main.c +2 -2
 }
@@ -872,11 +885,11 @@ static int afu_dev_destroy(struct platform_device
 *pdev)
 static int port_enable_set(struct platform_device *pdev, bool enable)
 {
 struct dfl_feature_platform_data *pdata = dev_get_platdata(&pdev-
 > dev);
-int ret = 0;
+int ret;
 
 mutex_lock(&pdata->lock);
 if (enable)
-__afu_port_enable(pdev);
+ret = __afu_port_enable(pdev);
 else

ret = __afu_port_disable(pdev);
mutex_unlock(&pdata->lock);

drivers/fpga/dfl-afu.h +1 -2
index 576e94960086..e5020e2b1f3d 100644
@@ -80,7 +80,7 @@ struct dfl_afu {
 };
 
 /* hold pdata->lock when call __afu_port_enable/disable */
-void __afu_port_enable(struct platform_device *pdev);
+int __afu_port_enable(struct platform_device *pdev);
 int __afu_port_disable(struct platform_device *pdev);
 
 void afu_mmio_region_init(struct dfl_feature_platform_data *pdata);
--

2.17.1

#10 lore raw
Quoted 63 lines

Subject: Re: [PATCH v2 1/1] fpga: dfl: afu: harden port enable logic

Sorry for the delay on this patch. It seemed like a lower priority patch than others, since we haven't seen any issues with current products. Please my responses inline.

Quoted 21 lines · Hao
Quoted 11 lines

-----Original Message-----
From: Russ Weight <[email protected]>
Sent: Friday, September 18, 2020 2:32 AM
To: [email protected]; [email protected]; linux- [email protected]
Cc: [email protected]; [email protected]; Xu, Yilun <[email protected]>; Wu, Hao <[email protected]>; Gerlach, Matthew <[email protected]>; Weight, Russell H <[email protected]>
Subject: [PATCH v2 1/1] fpga: dfl: afu: harden port enable logic

Quoted 3 lines

Port enable is not complete until ACK = 0. Change __afu_port_enable() to guarantee that the enable process is complete by polling for ACK == 0.

The description of this port reset ack bit is

" After initiating a Port soft reset, SW should monitor this bit. HW will set this bit when all outstanding requests initiated by this port have been drained, and the minimum soft reset pulse width has elapsed. "

Quoted 7 lines

But no description about what to do when clearing a Port soft reset to enable the port.

So we need to understand clearly on why we need this change (e.g. what may happen without this change), and will it apply for all existing DFL devices and future ones, or just for one specific card. Could you please help? : )

I touched bases with the hardware engineers. The recommendation to wait for ACK to be cleared is new with OFS and is documented in the latest OFS specification as follows (see step #4):

Quoted 5 lines

3.7.1 AFU Soft Resets
Software may cause a soft reset to be issued to the AFU as follows:
1. Assert the PortSoftReset field of the PORT_CONTROL register
2. Wait for the Port to acknowledge the soft reset by monitoring the PortSoftResetAck field of the PORT_CONTROL register, i.e.

PortSoftResetAck=1

Quoted 2 lines

3. Deasserting the PortSoftReset field
4. Wait for the Port to acknowledge the soft reset de-assertion by monitoring

the

Quoted 1 line

PortSoftResetAck field of the PORT_CONTROL register, i.e.

PortSoftResetAck=0

Quoted 4 lines

This sequence ensures that outstanding transactions are suitably flushed and that the FIM minimum reset pulse width is respected. Failing to follow this sequence leaves the AFU in an undefined state.

The OFS specification has not been posted publicly, yet.

Also, this is how it was explained to me:

Quoted 4 lines

In most scenario, port will be able to get out of reset soon enough when SW releases the port reset, especially on all the PAC products which have been verified before release.

Quoted 8 lines
Quoted 4 lines

Polling for HW to clear the ACK is meant to handle the following scenarios:

* Different platform can take variable period of time to get out of reset
* Bug in the HW that hold the port in reset

So this change is not required for the currently released PAC cards, but it is needed for OFS based products. I don't think there is any reason to hold off on the patch, as it is still valid for current products.

As you know, this driver is used for different cards, and we need to make sure new changes introduced in new version spec, don't break old products as we are sharing the same driver. and we are not sure if in the future some new products but still uses old specs, and then things may be broken if the driver which always perform new flow. Another method is that introduce 1 bit in hardware register to tell the driver to perform the additional steps, then it can avoid impacts to the old products. If this can't be done, then we at least need to verify this change on all existing hardware and suggest users to follow new spec only.

Hao

#11 lore raw
Quoted 68 lines · Hao
Quoted 61 lines

Subject: Re: [PATCH v2 1/1] fpga: dfl: afu: harden port enable logic

Sorry for the delay on this patch. It seemed like a lower priority patch than others, since we haven't seen any issues with current products. Please my responses inline.

Quoted 21 lines · Hao
Quoted 11 lines

-----Original Message-----
From: Russ Weight <[email protected]>
Sent: Friday, September 18, 2020 2:32 AM
To: [email protected]; [email protected]; linux- [email protected]
Cc: [email protected]; [email protected]; Xu, Yilun <[email protected]>; Wu, Hao <[email protected]>; Gerlach, Matthew <[email protected]>; Weight, Russell H <[email protected]>
Subject: [PATCH v2 1/1] fpga: dfl: afu: harden port enable logic

Quoted 3 lines

Port enable is not complete until ACK = 0. Change __afu_port_enable() to guarantee that the enable process is complete by polling for ACK == 0.

The description of this port reset ack bit is

" After initiating a Port soft reset, SW should monitor this bit. HW will set this bit when all outstanding requests initiated by this port have been drained, and the minimum soft reset pulse width has elapsed. "

Quoted 7 lines

But no description about what to do when clearing a Port soft reset to enable the port.

So we need to understand clearly on why we need this change (e.g. what may happen without this change), and will it apply for all existing DFL devices and future ones, or just for one specific card. Could you please help? : )

I touched bases with the hardware engineers. The recommendation to wait for ACK to be cleared is new with OFS and is documented in the latest OFS specification as follows (see step #4):

Quoted 5 lines

3.7.1 AFU Soft Resets
Software may cause a soft reset to be issued to the AFU as follows:
1. Assert the PortSoftReset field of the PORT_CONTROL register
2. Wait for the Port to acknowledge the soft reset by monitoring the PortSoftResetAck field of the PORT_CONTROL register, i.e.

PortSoftResetAck=1

Quoted 2 lines

3. Deasserting the PortSoftReset field
4. Wait for the Port to acknowledge the soft reset de-assertion by monitoring

the

Quoted 1 line

PortSoftResetAck field of the PORT_CONTROL register, i.e.

PortSoftResetAck=0

Quoted 3 lines

This sequence ensures that outstanding transactions are suitably flushed and that the FIM minimum reset pulse width is respected. Failing to follow this sequence leaves the AFU in an undefined state.

The OFS specification has not been posted publicly, yet.

Also, this is how it was explained to me:

Quoted 4 lines

In most scenario, port will be able to get out of reset soon enough when SW releases the port reset, especially on all the PAC products which have been verified before release.

Quoted 7 lines
Quoted 4 lines

Polling for HW to clear the ACK is meant to handle the following scenarios:

* Different platform can take variable period of time to get out of reset
* Bug in the HW that hold the port in reset

So this change is not required for the currently released PAC cards, but it is needed for OFS based products. I don't think there is any reason to hold off on the patch, as it is still valid for current products.

Quoted 9 lines

As you know, this driver is used for different cards, and we need to make sure new changes introduced in new version spec, don't break old products as we are sharing the same driver. and we are not sure if in the future some new products but still uses old specs, and then things may be broken if the driver which always perform new flow. Another method is that introduce 1 bit in hardware register to tell the driver to perform the additional steps, then it can avoid impacts to the old products. If this can't be done, then we at least need to verify this change on all existing hardware and suggest users to follow new spec only.

According to the HW engineers, the RTL implementation has not changed; it is the same as the RTL in the current PAC products. Polling for HW to clear the ACK is something we could have (should have?) been doing all along. The timing hasn't been an issue for the current PAC products, as proven by our testing. However, with OFS we cannot anticipate what the timing will be for customer designed products, so the specification is calling out this requirement as a precaution.

I am using a development machine that has the older PAC devices installed. I cleared port errors on these cards as a quick check, and the reset completes without hanging - which indicates that the ACK bit is in fact getting cleared. So there is not need for any device-specific conditional statements here.

- Russ

Quoted 2 lines

Hao

#12 lore raw
Quoted 81 lines · Russ Weight
Quoted 68 lines · Hao
Quoted 61 lines

Subject: Re: [PATCH v2 1/1] fpga: dfl: afu: harden port enable logic

Sorry for the delay on this patch. It seemed like a lower priority patch than others, since we haven't seen any issues with current products. Please my responses inline.

Quoted 21 lines · Hao
Quoted 11 lines

-----Original Message-----
From: Russ Weight <[email protected]>
Sent: Friday, September 18, 2020 2:32 AM
To: [email protected]; [email protected]; linux- [email protected]
Cc: [email protected]; [email protected]; Xu, Yilun <[email protected]>; Wu, Hao <[email protected]>; Gerlach, Matthew <[email protected]>; Weight, Russell H <[email protected]>
Subject: [PATCH v2 1/1] fpga: dfl: afu: harden port enable logic

Quoted 3 lines

Port enable is not complete until ACK = 0. Change __afu_port_enable() to guarantee that the enable process is complete by polling for ACK == 0.

The description of this port reset ack bit is

" After initiating a Port soft reset, SW should monitor this bit. HW will set this bit when all outstanding requests initiated by this port have been drained, and the minimum soft reset pulse width has elapsed. "

Quoted 7 lines

But no description about what to do when clearing a Port soft reset to enable the port.

So we need to understand clearly on why we need this change (e.g. what may happen without this change), and will it apply for all existing DFL devices and future ones, or just for one specific card. Could you please help? : )

I touched bases with the hardware engineers. The recommendation to wait for ACK to be cleared is new with OFS and is documented in the latest OFS specification as follows (see step #4):

Quoted 5 lines

3.7.1 AFU Soft Resets
Software may cause a soft reset to be issued to the AFU as follows:
1. Assert the PortSoftReset field of the PORT_CONTROL register
2. Wait for the Port to acknowledge the soft reset by monitoring the PortSoftResetAck field of the PORT_CONTROL register, i.e.

PortSoftResetAck=1

Quoted 2 lines

3. Deasserting the PortSoftReset field
4. Wait for the Port to acknowledge the soft reset de-assertion by monitoring

the

Quoted 1 line

PortSoftResetAck field of the PORT_CONTROL register, i.e.

PortSoftResetAck=0

Quoted 3 lines

This sequence ensures that outstanding transactions are suitably flushed and that the FIM minimum reset pulse width is respected. Failing to follow this sequence leaves the AFU in an undefined state.

The OFS specification has not been posted publicly, yet.

Also, this is how it was explained to me:

Quoted 4 lines

In most scenario, port will be able to get out of reset soon enough when SW releases the port reset, especially on all the PAC products which have been verified before release.

Quoted 7 lines
Quoted 4 lines

Polling for HW to clear the ACK is meant to handle the following scenarios:

* Different platform can take variable period of time to get out of reset
* Bug in the HW that hold the port in reset

So this change is not required for the currently released PAC cards, but it is needed for OFS based products. I don't think there is any reason to hold off on the patch, as it is still valid for current products.

Quoted 9 lines

As you know, this driver is used for different cards, and we need to make sure new changes introduced in new version spec, don't break old products as we are sharing the same driver. and we are not sure if in the future some new products but still uses old specs, and then things may be broken if the driver which always perform new flow. Another method is that introduce 1 bit in hardware register to tell the driver to perform the additional steps, then it can avoid impacts to the old products. If this can't be done, then we at least need to verify this change on all existing hardware and suggest users to follow new spec only.

Quoted 3 lines

According to the HW engineers, the RTL implementation has not changed; it is the same as the RTL in the current PAC products. Polling for HW to clear the ACK is something we could have (should have?) been doing all along. The timing

I also confirmed with HW engineers. The original specification was not precise. The code should have been doing this all along.

Matthew Gerlach

Quoted 16 lines

hasn't been an issue for the current PAC products, as proven by our testing. However, with OFS we cannot anticipate what the timing will be for customer designed products, so the specification is calling out this requirement as a precaution.

I am using a development machine that has the older PAC devices installed. I cleared port errors on these cards as a quick check, and the reset completes without hanging - which indicates that the ACK bit is in fact getting cleared. So there is not need for any device-specific conditional statements here.

- Russ

Quoted 2 lines

Hao

#13 lore raw
Quoted 86 lines · Russ Weight
Quoted 10 lines · Hao
Quoted 6 lines
Quoted 6 lines

Subject: Re: [PATCH v2 1/1] fpga: dfl: afu: harden port enable logic

Sorry for the delay on this patch. It seemed like a lower priority patch than others, since we haven't seen any issues with current products. Please my responses inline.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines · Hao
Quoted 6 lines
Quoted 6 lines

-----Original Message-----
From: Russ Weight <[email protected]>
Sent: Friday, September 18, 2020 2:32 AM
To: [email protected]; [email protected]; linux- [email protected]
Cc: [email protected]; [email protected]; Xu, Yilun

<[email protected]>;

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 15 lines
Quoted 8 lines

Wu, Hao <[email protected]>; Gerlach, Matthew <[email protected]>; Weight, Russell H <[email protected]>
Subject: [PATCH v2 1/1] fpga: dfl: afu: harden port enable logic

Port enable is not complete until ACK = 0. Change __afu_port_enable() to guarantee that the enable process is complete by polling for ACK == 0.

The description of this port reset ack bit is

" After initiating a Port soft reset, SW should monitor this bit. HW will set this bit when all outstanding requests initiated by this port have been drained, and the minimum soft reset pulse width has elapsed. "

Quoted 7 lines

But no description about what to do when clearing a Port soft reset to enable the port.

So we need to understand clearly on why we need this change (e.g. what may happen without this change), and will it apply for all existing DFL devices and future ones, or just for one specific card. Could you please help? : )

I touched bases with the hardware engineers. The recommendation to wait for ACK to be cleared is new with OFS and is documented in the latest OFS specification as follows (see step #4):

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

3.7.1 AFU Soft Resets
Software may cause a soft reset to be issued to the AFU as follows:
1. Assert the PortSoftReset field of the PORT_CONTROL register
2. Wait for the Port to acknowledge the soft reset by monitoring the PortSoftResetAck field of the PORT_CONTROL register, i.e.

PortSoftResetAck=1

Quoted 2 lines

3. Deasserting the PortSoftReset field
4. Wait for the Port to acknowledge the soft reset de-assertion by

monitoring

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

the

Quoted 1 line

PortSoftResetAck field of the PORT_CONTROL register, i.e.

PortSoftResetAck=0

Quoted 1 line

This sequence ensures that outstanding transactions are suitably flushed

and

Quoted 27 lines
Quoted 26 lines
Quoted 10 lines
Quoted 2 lines

that the FIM minimum reset pulse width is respected. Failing to follow this sequence leaves the AFU in an undefined state.

The OFS specification has not been posted publicly, yet.

Also, this is how it was explained to me:

Quoted 4 lines

In most scenario, port will be able to get out of reset soon enough when SW releases the port reset, especially on all the PAC products which have been verified before release.

Quoted 7 lines
Quoted 4 lines

Polling for HW to clear the ACK is meant to handle the following scenarios:

* Different platform can take variable period of time to get out of reset
* Bug in the HW that hold the port in reset

So this change is not required for the currently released PAC cards, but it is needed for OFS based products. I don't think there is any reason to hold off on the patch, as it is still valid for current products.

As you know, this driver is used for different cards, and we need to make sure new changes introduced in new version spec, don't break old products as we are sharing the same driver. and we are not sure if in the future some new products but still uses old specs, and then things may be broken if the driver which always perform new flow. Another method is that introduce 1 bit in hardware register to tell the driver to perform the additional steps, then it can avoid impacts to the old products. If this can't be done, then we at least need to verify this change on all existing hardware and suggest users to follow new spec only.

Quoted 7 lines
Quoted 3 lines

According to the HW engineers, the RTL implementation has not changed; it is the same as the RTL in the current PAC products. Polling for HW to clear the ACK is something we could have (should have?) been doing all along. The

timing

I also confirmed with HW engineers. The original specification was not precise. The code should have been doing this all along.

Thanks for this confirmation, then it sounds good to me. I think only Intel hardware is using this driver now, so if this is confirmed from hardware side, then we should be safe to take this one.

Hao

Quoted 19 lines

Matthew Gerlach

Quoted 16 lines

hasn't been an issue for the current PAC products, as proven by our testing. However, with OFS we cannot anticipate what the timing will be for customer designed products, so the specification is calling out this requirement as a precaution.

I am using a development machine that has the older PAC devices installed. I cleared port errors on these cards as a quick check, and the reset completes without hanging - which indicates that the ACK bit is in fact getting cleared. So there is not need for any device-specific conditional statements here.

- Russ

Quoted 2 lines

Hao