fpga: dfl: afu: harden port enable logic
Subsystems (2)
PATCH v2 1/1 fpga: dfl: afu: harden port enable logic
Port enable is not complete until ACK = 0. Change __afu_port_enable() to guarantee that the enable process is complete by polling for ACK == 0.
--- drivers/fpga/dfl-afu-error.c | 2 +- drivers/fpga/dfl-afu-main.c | 29 +++++++++++++++++++++-------- drivers/fpga/dfl-afu.h | 2 +- 3 files changed, 23 insertions(+), 10 deletions(-)
drivers/fpga/dfl-afu-error.c +1 -1
@@ -103,7 +103,7 @@ static int afu_port_err_clear(struct device *dev, u64 err)__afu_port_err_mask (dev, false);/* Enable the Port by clear the reset */ -__afu_port_enable (pdev); + ret =__afu_port_enable (pdev); done:mutex_unlock (&pdata->lock);
drivers/fpga/dfl-afu-main.c +21 -8
@@ -21,6 +21,9 @@#include "dfl-afu.h" +#define RST_POLL_INVL 10/* us */ +#define RST_POLL_TIMEOUT 1000/* us */ +/** * __afu_port_enable - enable a port by clear reset * @pdev: port platform device.
@@ -32,7 +35,7 @@* * The caller needs to hold lock for protection. */ -void __afu_port_enable (struct platform_device *pdev) +int __afu_port_enable (struct platform_device *pdev) {struct dfl_feature_platform_data *pdata =dev_get_platdata (&pdev->dev);void __iomem *base;
@@ -41,7 +44,7 @@ void __afu_port_enable(struct platform_device *pdev)WARN_ON (!pdata->disable_count);if (--pdata->disable_count !=0 ) -return ; +return 0 ; base =dfl_get_feature_ioaddr_by_id (&pdev->dev,PORT_FEATURE_ID_HEADER );
@@ -49,10 +52,20 @@ void __afu_port_enable(struct platform_device *pdev) v =readq (base +PORT_HDR_CTRL ); v &= ~PORT_CTRL_SFTRST ;writeq (v, base +PORT_HDR_CTRL ); -} -#define RST_POLL_INVL 10/* us */ -#define RST_POLL_TIMEOUT 1000/* us */ +/* +* HW clears the ack bit to indicate that the port is fully out +* of reset. +*/ +if (readq_poll_timeout (base +PORT_HDR_CTRL , v, + !(v &PORT_CTRL_SFTRST_ACK ), +RST_POLL_INVL ,RST_POLL_TIMEOUT )) { +dev_err (&pdev->dev,"timeout, failure to enable device\n" ); +return -ETIMEDOUT ; + } + +return 0 ; +}/** * __afu_port_disable - disable a port by hold reset
@@ -111,7 +124,7 @@ static int __port_reset(struct platform_device *pdev) ret =__afu_port_disable (pdev);if (!ret) -__afu_port_enable (pdev); + ret =__afu_port_enable (pdev);return ret; }
@@ -872,11 +885,11 @@ static int afu_dev_destroy(struct platform_device *pdev)static int port_enable_set (struct platform_device *pdev,bool enable) {struct dfl_feature_platform_data *pdata =dev_get_platdata (&pdev->dev); -int ret =0 ; +int ret;mutex_lock (&pdata->lock);if (enable) -__afu_port_enable (pdev); + ret =__afu_port_enable (pdev);else ret =__afu_port_disable (pdev);mutex_unlock (&pdata->lock);
drivers/fpga/dfl-afu.h +1 -1
@@ -80,7 +80,7 @@ struct dfl_afu { };/* hold pdata->lock when call __afu_port_enable/disable */ -void __afu_port_enable (struct platform_device *pdev); +int __afu_port_enable (struct platform_device *pdev);int __afu_port_disable (struct platform_device *pdev);void afu_mmio_region_init (struct dfl_feature_platform_data *pdata);
Signature · 1 line
2.17.1
Tom Rix
reviewerQuoted 16 lines · Russ Weight
Port enable is not complete until ACK = 0. Change
__afu_port_enable()to guarantee that the enable process is complete by polling for ACK == 0.Signed-off-by Russ Weight <[email protected]>--- drivers/fpga/dfl-afu-error.c | 2 +- drivers/fpga/dfl-afu-main.c | 29 +++++++++++++++++++++-------- drivers/fpga/dfl-afu.h | 2 +- 3 files changed, 23 insertions(+), 10 deletions(-)drivers/fpga/dfl-afu-error.c +0
-0
drivers/fpga/dfl-afu-error.c static int afu_port_err_clear(struct device *dev, u64 err) · line 104
Quoted 2 lines
drivers/fpga/dfl-afu-error.c +0
-0@@ -103,7 +103,7 @@ static int afu_port_err_clear(struct device *dev, u64 err)__afu_port_err_mask (dev, false);
There is an earlier bit that sets ret = -EINVAL.
This error will be lost or not handled well.
Right now it doesn't seem to be handled.
Quoted 30 lines
drivers/fpga/dfl-afu-error.c +1
-1/* Enable the Port by clear the reset */ -__afu_port_enable (pdev); + ret =__afu_port_enable (pdev); done:mutex_unlock (&pdata->lock);drivers/fpga/dfl-afu-main.c +4
-1@@ -21,6 +21,9 @@#include "dfl-afu.h" +#define RST_POLL_INVL 10/* us */ +#define RST_POLL_TIMEOUT 1000/* us */ +/** * __afu_port_enable - enable a port by clear reset * @pdev: port platform device. @@ -32,7 +35,7 @@* * The caller needs to hold lock for protection. */ -void __afu_port_enable (struct platform_device *pdev) +int __afu_port_enable (struct platform_device *pdev) {struct dfl_feature_platform_data *pdata =dev_get_platdata (&pdev->dev);void __iomem *base;
drivers/fpga/dfl-afu-main.c void __afu_port_enable(struct platform_device *pdev) · line 47
Quoted 5 lines
drivers/fpga/dfl-afu-main.c +1
-1@@ -41,7 +44,7 @@ void __afu_port_enable(struct platform_device *pdev)WARN_ON (!pdata->disable_count);if (--pdata->disable_count !=0 ) -return ; +return 0 ;
Is this really a success ? Maybe -EBUSY ?
Quoted 55 lines
drivers/fpga/dfl-afu-main.c +16
-6base =dfl_get_feature_ioaddr_by_id (&pdev->dev,PORT_FEATURE_ID_HEADER );@@ -49,10 +52,20 @@ void __afu_port_enable(struct platform_device *pdev) v =readq (base +PORT_HDR_CTRL ); v &= ~PORT_CTRL_SFTRST ;writeq (v, base +PORT_HDR_CTRL ); -} -#define RST_POLL_INVL 10/* us */ -#define RST_POLL_TIMEOUT 1000/* us */ +/* +* HW clears the ack bit to indicate that the port is fully out +* of reset. +*/ +if (readq_poll_timeout (base +PORT_HDR_CTRL , v, + !(v &PORT_CTRL_SFTRST_ACK ), +RST_POLL_INVL ,RST_POLL_TIMEOUT )) { +dev_err (&pdev->dev,"timeout, failure to enable device\n" ); +return -ETIMEDOUT ; + } + +return 0 ; +}/** * __afu_port_disable - disable a port by hold reset @@ -111,7 +124,7 @@ static int __port_reset(struct platform_device *pdev) ret =__afu_port_disable (pdev);if (!ret) -__afu_port_enable (pdev); + ret =__afu_port_enable (pdev);return ret; }@@ -872,11 +885,11 @@ static int afu_dev_destroy(struct platform_device *pdev)static int port_enable_set (struct platform_device *pdev,bool enable) {struct dfl_feature_platform_data *pdata =dev_get_platdata (&pdev->dev); -int ret =0 ; +int ret;mutex_lock (&pdata->lock);if (enable) -__afu_port_enable (pdev); + ret =__afu_port_enable (pdev);else ret =__afu_port_disable (pdev);mutex_unlock (&pdata->lock);drivers/fpga/dfl-afu.h +0
-0
drivers/fpga/dfl-afu.h struct dfl_afu { · line 84
Quoted 6 lines
drivers/fpga/dfl-afu.h +1
-1@@ -80,7 +80,7 @@ struct dfl_afu { };/* hold pdata->lock when call __afu_port_enable/disable */ -void __afu_port_enable (struct platform_device *pdev); +int __afu_port_enable (struct platform_device *pdev);int __afu_port_disable (struct platform_device *pdev);
The other functions in this file have afu_* since the __afu_port_enable/disable
are used other places would it make sense to remove the '__' prefix ?
If you think so, maybe a cleanup patch later.
Tom
Quoted 2 lines
drivers/fpga/dfl-afu.h +0
-0void afu_mmio_region_init (struct dfl_feature_platform_data *pdata);
Moritz Fischer
maintainerQuoted 7 lines · Tom Rix
Quoted 5 lines · Russ Weight
Port enable is not complete until ACK = 0. Change
__afu_port_enable()to guarantee that the enable process is complete by polling for ACK == 0.Signed-off-by Russ Weight <[email protected]>
General note: Please keep a changelog if you send updated versions of a patch. This can be added here with an extra '---' + Text between Signed-off and diffstat:
---
Changes from v1:
- FOo
- Bar
Quoted 16 lines
drivers/fpga/dfl-afu-error.c static int afu_port_err_clear(struct device *dev, u64 err) · line 104
Quoted 13 lines
--- drivers/fpga/dfl-afu-error.c | 2 +- drivers/fpga/dfl-afu-main.c | 29 +++++++++++++++++++++-------- drivers/fpga/dfl-afu.h | 2 +- 3 files changed, 23 insertions(+), 10 deletions(-)drivers/fpga/dfl-afu-error.c +0
-0@@ -103,7 +103,7 @@ static int afu_port_err_clear(struct device *dev, u64 err)__afu_port_err_mask (dev, false);There is an earlier bit that sets ret =
-EINVAL.
Quoted 3 lines
This error will be lost or not handled well.
Right now it doesn't seem to be handled.
Ultimately you'd want to report *at least* one of them, the current code seems to continue and enable the port either case. Is that what it should be doing?
Is the timeout more severe than the invalid value? Do you want to print a warning?
Either way a comment explaining why this is ok would be appreciated :)
Quoted 33 lines
Quoted 33 lines
drivers/fpga/dfl-afu-error.c +1
-1/* Enable the Port by clear the reset */ -__afu_port_enable (pdev); + ret =__afu_port_enable (pdev); done:mutex_unlock (&pdata->lock);drivers/fpga/dfl-afu-main.c +4
-1@@ -21,6 +21,9 @@#include "dfl-afu.h" +#define RST_POLL_INVL 10/* us */ +#define RST_POLL_TIMEOUT 1000/* us */ +/** * __afu_port_enable - enable a port by clear reset * @pdev: port platform device. @@ -32,7 +35,7 @@* * The caller needs to hold lock for protection. */ -void __afu_port_enable (struct platform_device *pdev) +int __afu_port_enable (struct platform_device *pdev) {struct dfl_feature_platform_data *pdata =dev_get_platdata (&pdev->dev);void __iomem *base;@@ -41,7 +44,7 @@ void __afu_port_enable(struct platform_device *pdev)WARN_ON (!pdata->disable_count);
Quoted 4 lines
drivers/fpga/dfl-afu-main.c void __afu_port_enable(struct platform_device *pdev) · line 47
Quoted 3 lines
drivers/fpga/dfl-afu-main.c +1
-1if (--pdata->disable_count !=0 ) -return ; +return 0 ;Is this really a success ? Maybe
-EBUSY?
Seems like if it's severe enough for a warning you'd probably want to return an error.
Quoted 57 lines
Quoted 57 lines
drivers/fpga/dfl-afu-main.c +16
-6base =dfl_get_feature_ioaddr_by_id (&pdev->dev,PORT_FEATURE_ID_HEADER );@@ -49,10 +52,20 @@ void __afu_port_enable(struct platform_device *pdev) v =readq (base +PORT_HDR_CTRL ); v &= ~PORT_CTRL_SFTRST ;writeq (v, base +PORT_HDR_CTRL ); -} -#define RST_POLL_INVL 10/* us */ -#define RST_POLL_TIMEOUT 1000/* us */ +/* +* HW clears the ack bit to indicate that the port is fully out +* of reset. +*/ +if (readq_poll_timeout (base +PORT_HDR_CTRL , v, + !(v &PORT_CTRL_SFTRST_ACK ), +RST_POLL_INVL ,RST_POLL_TIMEOUT )) { +dev_err (&pdev->dev,"timeout, failure to enable device\n" ); +return -ETIMEDOUT ; + } + +return 0 ; +}/** * __afu_port_disable - disable a port by hold reset @@ -111,7 +124,7 @@ static int __port_reset(struct platform_device *pdev) ret =__afu_port_disable (pdev);if (!ret) -__afu_port_enable (pdev); + ret =__afu_port_enable (pdev);return ret; }@@ -872,11 +885,11 @@ static int afu_dev_destroy(struct platform_device *pdev)static int port_enable_set (struct platform_device *pdev,bool enable) {struct dfl_feature_platform_data *pdata =dev_get_platdata (&pdev->dev); -int ret =0 ; +int ret;mutex_lock (&pdata->lock);if (enable) -__afu_port_enable (pdev); + ret =__afu_port_enable (pdev);else ret =__afu_port_disable (pdev);mutex_unlock (&pdata->lock);drivers/fpga/dfl-afu.h +0
-0@@ -80,7 +80,7 @@ struct dfl_afu { };
Quoted 8 lines
drivers/fpga/dfl-afu.h struct dfl_afu { · line 84
Quoted 4 lines
drivers/fpga/dfl-afu.h +1
-1/* hold pdata->lock when call __afu_port_enable/disable */ -void __afu_port_enable (struct platform_device *pdev); +int __afu_port_enable (struct platform_device *pdev);int __afu_port_disable (struct platform_device *pdev);The other functions in this file have afu_* since the __afu_port_enable/disable
are used other places would it make sense to remove the '__' prefix ?
The idea on those is to indicate that the caller need to be cautious (often a lock / mutex) is required. I think keeping them as is is fine.
Quoted 8 lines
If you think so, maybe a cleanup patch later.
Tom
Quoted 2 lines
drivers/fpga/dfl-afu.h +0
-0void afu_mmio_region_init (struct dfl_feature_platform_data *pdata);
Thanks,
Moritz
Xu Yilun
maintainerQuoted 15 lines
Quoted 15 lines
Quoted 15 lines
/**
*__afu_port_enable- enable a port by clear reset
* @pdev: port platform device.drivers/fpga/dfl-afu-main.c +1
-1@@ -32,7 +35,7 @@* * The caller needs to hold lock for protection. */ -void __afu_port_enable (struct platform_device *pdev) +int __afu_port_enable (struct platform_device *pdev) {struct dfl_feature_platform_data *pdata =dev_get_platdata (&pdev->dev);void __iomem *base;@@ -41,7 +44,7 @@ void __afu_port_enable(struct platform_device *pdev)WARN_ON (!pdata->disable_count);
Quoted 6 lines
Quoted 4 lines
drivers/fpga/dfl-afu-main.c void __afu_port_enable(struct platform_device *pdev) · line 47
Quoted 3 lines
drivers/fpga/dfl-afu-main.c +1
-1if (--pdata->disable_count !=0 ) -return ; +return 0 ;Is this really a success ? Maybe
-EBUSY?Seems like if it's severe enough for a warning you'd probably want to return an error.
This code is to handle the port enable/disable request from multiple users. This is a voting mechanism, the port would not be physically enabled if there is still an disable vote. The --diable_count != 0 works for this purpose. So I think it should be OK here since the voting mechanism is working as expected.
Thanks,
Yilun
Quoted 36 lines
Subject: Re: [PATCH v2 1/1] fpga: dfl: afu: harden port enable logic
Quoted 7 lines · Tom Rix
Quoted 5 lines · Russ Weight
Port enable is not complete until ACK = 0. Change
__afu_port_enable()to guarantee that the enable process is complete by polling for ACK == 0.Signed-off-by Russ Weight <[email protected]>General note: Please keep a changelog if you send updated versions of a patch. This can be added here with an extra '---' + Text between Signed-off and
diffstat:---
Changes from v1:
- FOo
- BarQuoted 11 lines
drivers/fpga/dfl-afu-error.c static int afu_port_err_clear(struct device *dev,
Quoted 11 lines
--- drivers/fpga/dfl-afu-error.c | 2 +- drivers/fpga/dfl-afu-main.c | 29 +++++++++++++++++++++-------- drivers/fpga/dfl-afu.h | 2 +- 3 files changed, 23 insertions(+), 10 deletions(-)drivers/fpga/dfl-afu-error.c +0
-0@@ -103,7 +103,7 @@ static int afu_port_err_clear(struct device *dev,u64 err)
Quoted 5 lines
drivers/fpga/dfl-afu-error.c static int afu_port_err_clear(struct device *dev, u64 err) · line 104
Quoted 2 lines
drivers/fpga/dfl-afu-error.c +0
-0__afu_port_err_mask (dev, false);There is an earlier bit that sets ret =
-EINVAL.
Quoted 7 lines
Quoted 3 lines
This error will be lost or not handled well.
Right now it doesn't seem to be handled.
Ultimately you'd want to report *at least* one of them, the current code seems to continue and enable the port either case. Is that what it should be doing?
In order to do error clear, we have to put port into reset firstly and then clear port after error clearing is done. If we see failure during error clearing that we still want to get the port back to work at least. As we know, if port is still in reset, then the accelerator connected to the port won't work.
Quoted 3 lines
Is the timeout more severe than the invalid value? Do you want to print a warning?
Yes, It's a very bad case if port can not be enabled any more (accelerator may not be accessible any more), hardware should already be in error, it's better we have some warning messages here.
Quoted 35 lines
Either way a comment explaining why this is ok would be appreciated :)
Quoted 30 lines
Quoted 22 lines
drivers/fpga/dfl-afu-error.c +1
-1/* Enable the Port by clear the reset */ -__afu_port_enable (pdev); + ret =__afu_port_enable (pdev); done:mutex_unlock (&pdata->lock);drivers/fpga/dfl-afu-main.c +3
-0@@ -21,6 +21,9 @@#include "dfl-afu.h" +#define RST_POLL_INVL 10/* us */ +#define RST_POLL_TIMEOUT 1000/* us */ +/** * __afu_port_enable - enable a port by clear reset * @pdev: port platform device. drivers/fpga/dfl-afu-main.c line 40
Quoted 7 lines
drivers/fpga/dfl-afu-main.c +1
-1@@ -32,7 +35,7 @@* * The caller needs to hold lock for protection. */ -void __afu_port_enable (struct platform_device *pdev) +int __afu_port_enable (struct platform_device *pdev) { struct dfl_feature_platform_data *pdata = dev_get_platdata(&pdev-dev);
Quoted 2 lines
drivers/fpga/dfl-afu-main.c void __afu_port_enable(struct platform_device
Quoted 2 lines
drivers/fpga/dfl-afu-main.c +0
-0void __iomem *base;@@ -41,7 +44,7 @@ void __afu_port_enable(struct platform_device*pdev)
Quoted 8 lines
Quoted 6 lines
drivers/fpga/dfl-afu-main.c void __afu_port_enable(struct platform_device *pdev) · line 47
Quoted 5 lines
drivers/fpga/dfl-afu-main.c +1
-1WARN_ON (!pdata->disable_count);if (--pdata->disable_count !=0 ) -return ; +return 0 ;Is this really a success ? Maybe
-EBUSY?Seems like if it's severe enough for a warning you'd probably want to return an error.
As Yilun mentioned, this is just a reference count operation, we don't need to return error code.
Quoted 68 lines
Quoted 2 lines
drivers/fpga/dfl-afu-main.c void __afu_port_enable(struct platform_device · line 48
Quoted 2 lines
drivers/fpga/dfl-afu-main.c +0
-0base = dfl_get_feature_ioaddr_by_id(&pdev->dev,
PORT_FEATURE_ID_HEADER);Quoted 2 lines
drivers/fpga/dfl-afu-main.c void __afu_port_enable(struct platform_device
Quoted 2 lines
drivers/fpga/dfl-afu-main.c +0
-0@@ -49,10 +52,20 @@ void __afu_port_enable(struct platform_device*pdev)
Quoted 21 lines
Quoted 21 lines
drivers/fpga/dfl-afu-main.c +13
-3v =readq (base +PORT_HDR_CTRL ); v &= ~PORT_CTRL_SFTRST ;writeq (v, base +PORT_HDR_CTRL ); -} -#define RST_POLL_INVL 10/* us */ -#define RST_POLL_TIMEOUT 1000/* us */ +/* +* HW clears the ack bit to indicate that the port is fully out +* of reset. +*/ +if (readq_poll_timeout (base +PORT_HDR_CTRL , v, + !(v &PORT_CTRL_SFTRST_ACK ), +RST_POLL_INVL ,RST_POLL_TIMEOUT )) { +dev_err (&pdev->dev,"timeout, failure to enable device\n" ); +return -ETIMEDOUT ; + } + +return 0 ; +}Quoted 3 lines
drivers/fpga/dfl-afu-main.c static int __port_reset(struct platform_device
Quoted 3 lines
drivers/fpga/dfl-afu-main.c +0
-0/** * __afu_port_disable - disable a port by hold reset @@ -111,7 +124,7 @@ static int __port_reset(struct platform_device*pdev)
Quoted 9 lines
drivers/fpga/dfl-afu-main.c static int afu_dev_destroy(struct
Quoted 9 lines
drivers/fpga/dfl-afu-main.c +1
-1ret =__afu_port_disable (pdev);if (!ret) -__afu_port_enable (pdev); + ret =__afu_port_enable (pdev);return ret; }@@ -872,11 +885,11 @@ static int afu_dev_destroy(struct
platform_device*pdev)Quoted 21 lines
drivers/fpga/dfl-afu-main.c static int afu_dev_destroy(struct platform_device *pdev)
Quoted 3 lines
drivers/fpga/dfl-afu-main.c +0
-0static int port_enable_set (struct platform_device *pdev,bool enable) { struct dfl_feature_platform_data *pdata = dev_get_platdata(&pdev-dev);
Quoted 17 lines
drivers/fpga/dfl-afu-main.c +2
-2-int ret =0 ; +int ret;mutex_lock (&pdata->lock);if (enable) -__afu_port_enable (pdev); + ret =__afu_port_enable (pdev);else ret =__afu_port_disable (pdev);mutex_unlock (&pdata->lock);drivers/fpga/dfl-afu.h +0
-0@@ -80,7 +80,7 @@ struct dfl_afu { };Quoted 6 lines
drivers/fpga/dfl-afu.h struct dfl_afu { · line 84
Quoted 4 lines
drivers/fpga/dfl-afu.h +1
-1/* hold pdata->lock when call __afu_port_enable/disable */ -void __afu_port_enable (struct platform_device *pdev); +int __afu_port_enable (struct platform_device *pdev);int __afu_port_disable (struct platform_device *pdev);The other functions in this file have afu_* since the
Quoted 6 lines
__afu_port_enable/disable
Quoted 2 lines
are used other places would it make sense to remove the '__' prefix ?
The idea on those is to indicate that the caller need to be cautious (often a lock / mutex) is required. I think keeping them as is is fine.
Yes. That's why we add the prefix for these functions.
Thanks
Hao
Quoted 12 lines
Quoted 9 lines
If you think so, maybe a cleanup patch later.
Tom
Quoted 2 lines
drivers/fpga/dfl-afu.h +0
-0void afu_mmio_region_init (struct dfl_feature_platform_data *pdata);Thanks,
Moritz
Quoted 7 lines · Moritz Fischer
Quoted 6 lines · Tom Rix
Quoted 5 lines · Russ Weight
Port enable is not complete until ACK = 0. Change
__afu_port_enable()to guarantee that the enable process is complete by polling for ACK == 0.Signed-off-by Russ Weight <[email protected]>
Quoted 8 lines
General note: Please keep a changelog if you send updated versions of a patch. This can be added here with an extra '---' + Text between Signed-off and diffstat:
---
Changes from v1:
- FOo
- Bar
Yes - I'll do that on future patch updates. In this case v2 just fixed a typo in the commit message, so the patch was essentially the same as v1.
Quoted 18 lines
Quoted 15 lines
drivers/fpga/dfl-afu-error.c static int afu_port_err_clear(struct device *dev, u64 err) · line 104
Quoted 13 lines
--- drivers/fpga/dfl-afu-error.c | 2 +- drivers/fpga/dfl-afu-main.c | 29 +++++++++++++++++++++-------- drivers/fpga/dfl-afu.h | 2 +- 3 files changed, 23 insertions(+), 10 deletions(-)drivers/fpga/dfl-afu-error.c +0
-0@@ -103,7 +103,7 @@ static int afu_port_err_clear(struct device *dev, u64 err)__afu_port_err_mask (dev, false);There is an earlier bit that sets ret =
-EINVAL.Quoted 3 lines
This error will be lost or not handled well.
Right now it doesn't seem to be handled.
Quoted 8 lines
Ultimately you'd want to report *at least* one of them, the current code seems to continue and enable the port either case. Is that what it should be doing?
Is the timeout more severe than the invalid value? Do you want to print a warning?
Either way a comment explaining why this is ok would be appreciated :)
Yes - I'll add a comment explaining how the errors arebeing prioritized. I'll give priority to the timeout, asit is likely a HW failure.
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
drivers/fpga/dfl-afu-error.c +1
-1/* Enable the Port by clear the reset */ -__afu_port_enable (pdev); + ret =__afu_port_enable (pdev); done:mutex_unlock (&pdata->lock);drivers/fpga/dfl-afu-main.c +4
-1@@ -21,6 +21,9 @@#include "dfl-afu.h" +#define RST_POLL_INVL 10/* us */ +#define RST_POLL_TIMEOUT 1000/* us */ +/** * __afu_port_enable - enable a port by clear reset * @pdev: port platform device. @@ -32,7 +35,7 @@* * The caller needs to hold lock for protection. */ -void __afu_port_enable (struct platform_device *pdev) +int __afu_port_enable (struct platform_device *pdev) {struct dfl_feature_platform_data *pdata =dev_get_platdata (&pdev->dev);void __iomem *base;@@ -41,7 +44,7 @@ void __afu_port_enable(struct platform_device *pdev)WARN_ON (!pdata->disable_count);
Quoted 6 lines
Quoted 4 lines
drivers/fpga/dfl-afu-main.c void __afu_port_enable(struct platform_device *pdev) · line 47
Quoted 3 lines
drivers/fpga/dfl-afu-main.c +1
-1if (--pdata->disable_count !=0 ) -return ; +return 0 ;Is this really a success ? Maybe
-EBUSY?Seems like if it's severe enough for a warning you'd probably want to return an error.
As mentioned by Hao and Yilun, the disable_count is a reference count.The WARN_ON() is checking for a different condition - an invalid reference count. We should never call port_enable if the port is not disabled. Do you think a comment is needed here?
Thanks,
- Russ
Quoted 75 lines
Quoted 57 lines
Quoted 57 lines
drivers/fpga/dfl-afu-main.c +16
-6base =dfl_get_feature_ioaddr_by_id (&pdev->dev,PORT_FEATURE_ID_HEADER );@@ -49,10 +52,20 @@ void __afu_port_enable(struct platform_device *pdev) v =readq (base +PORT_HDR_CTRL ); v &= ~PORT_CTRL_SFTRST ;writeq (v, base +PORT_HDR_CTRL ); -} -#define RST_POLL_INVL 10/* us */ -#define RST_POLL_TIMEOUT 1000/* us */ +/* +* HW clears the ack bit to indicate that the port is fully out +* of reset. +*/ +if (readq_poll_timeout (base +PORT_HDR_CTRL , v, + !(v &PORT_CTRL_SFTRST_ACK ), +RST_POLL_INVL ,RST_POLL_TIMEOUT )) { +dev_err (&pdev->dev,"timeout, failure to enable device\n" ); +return -ETIMEDOUT ; + } + +return 0 ; +}/** * __afu_port_disable - disable a port by hold reset @@ -111,7 +124,7 @@ static int __port_reset(struct platform_device *pdev) ret =__afu_port_disable (pdev);if (!ret) -__afu_port_enable (pdev); + ret =__afu_port_enable (pdev);return ret; }@@ -872,11 +885,11 @@ static int afu_dev_destroy(struct platform_device *pdev)static int port_enable_set (struct platform_device *pdev,bool enable) {struct dfl_feature_platform_data *pdata =dev_get_platdata (&pdev->dev); -int ret =0 ; +int ret;mutex_lock (&pdata->lock);if (enable) -__afu_port_enable (pdev); + ret =__afu_port_enable (pdev);else ret =__afu_port_disable (pdev);mutex_unlock (&pdata->lock);drivers/fpga/dfl-afu.h +0
-0@@ -80,7 +80,7 @@ struct dfl_afu { };Quoted 7 lines
drivers/fpga/dfl-afu.h struct dfl_afu { · line 84
Quoted 4 lines
drivers/fpga/dfl-afu.h +1
-1/* hold pdata->lock when call __afu_port_enable/disable */ -void __afu_port_enable (struct platform_device *pdev); +int __afu_port_enable (struct platform_device *pdev);int __afu_port_disable (struct platform_device *pdev);The other functions in this file have afu_* since the __afu_port_enable/disable
are used other places would it make sense to remove the '__' prefix ?
The idea on those is to indicate that the caller need to be cautious (often a lock / mutex) is required. I think keeping them as is is fine.
Quoted 6 lines
If you think so, maybe a cleanup patch later.
Tom
Quoted 2 lines
drivers/fpga/dfl-afu.h +0
-0void afu_mmio_region_init (struct dfl_feature_platform_data *pdata);Thanks,
Moritz
Quoted 21 lines · Tom Rix
Quoted 16 lines · Russ Weight
Port enable is not complete until ACK = 0. Change
__afu_port_enable()to guarantee that the enable process is complete by polling for ACK == 0.Signed-off-by Russ Weight <[email protected]>--- drivers/fpga/dfl-afu-error.c | 2 +- drivers/fpga/dfl-afu-main.c | 29 +++++++++++++++++++++-------- drivers/fpga/dfl-afu.h | 2 +- 3 files changed, 23 insertions(+), 10 deletions(-)drivers/fpga/dfl-afu-error.c +0
-0drivers/fpga/dfl-afu-error.c static int afu_port_err_clear(struct device *dev, u64 err) · line 104
Quoted 2 lines
drivers/fpga/dfl-afu-error.c +0
-0@@ -103,7 +103,7 @@ static int afu_port_err_clear(struct device *dev, u64 err)__afu_port_err_mask (dev, false);There is an earlier bit that sets ret =
-EINVAL.
Quoted 3 lines
This error will be lost or not handled well.
Right now it doesn't seem to be handled.
Good catch. I'll give priority to -EINVAL in the next version of the patch, as it is more informative in the context of this function.
Quoted 33 lines
Quoted 33 lines
drivers/fpga/dfl-afu-error.c +1
-1/* Enable the Port by clear the reset */ -__afu_port_enable (pdev); + ret =__afu_port_enable (pdev); done:mutex_unlock (&pdata->lock);drivers/fpga/dfl-afu-main.c +4
-1@@ -21,6 +21,9 @@#include "dfl-afu.h" +#define RST_POLL_INVL 10/* us */ +#define RST_POLL_TIMEOUT 1000/* us */ +/** * __afu_port_enable - enable a port by clear reset * @pdev: port platform device. @@ -32,7 +35,7 @@* * The caller needs to hold lock for protection. */ -void __afu_port_enable (struct platform_device *pdev) +int __afu_port_enable (struct platform_device *pdev) {struct dfl_feature_platform_data *pdata =dev_get_platdata (&pdev->dev);void __iomem *base;@@ -41,7 +44,7 @@ void __afu_port_enable(struct platform_device *pdev)WARN_ON (!pdata->disable_count);
Quoted 4 lines
drivers/fpga/dfl-afu-main.c void __afu_port_enable(struct platform_device *pdev) · line 47
Quoted 3 lines
drivers/fpga/dfl-afu-main.c +1
-1if (--pdata->disable_count !=0 ) -return ; +return 0 ;Is this really a success ? Maybe
-EBUSY?
Yilun addressed this question in his previous response. This isessentially a reference count for nested disable calls. Weonly do the enable if the disable count has gone to zero, so this isn't an error condition.
Quoted 63 lines
Quoted 55 lines
drivers/fpga/dfl-afu-main.c +16
-6base =dfl_get_feature_ioaddr_by_id (&pdev->dev,PORT_FEATURE_ID_HEADER );@@ -49,10 +52,20 @@ void __afu_port_enable(struct platform_device *pdev) v =readq (base +PORT_HDR_CTRL ); v &= ~PORT_CTRL_SFTRST ;writeq (v, base +PORT_HDR_CTRL ); -} -#define RST_POLL_INVL 10/* us */ -#define RST_POLL_TIMEOUT 1000/* us */ +/* +* HW clears the ack bit to indicate that the port is fully out +* of reset. +*/ +if (readq_poll_timeout (base +PORT_HDR_CTRL , v, + !(v &PORT_CTRL_SFTRST_ACK ), +RST_POLL_INVL ,RST_POLL_TIMEOUT )) { +dev_err (&pdev->dev,"timeout, failure to enable device\n" ); +return -ETIMEDOUT ; + } + +return 0 ; +}/** * __afu_port_disable - disable a port by hold reset @@ -111,7 +124,7 @@ static int __port_reset(struct platform_device *pdev) ret =__afu_port_disable (pdev);if (!ret) -__afu_port_enable (pdev); + ret =__afu_port_enable (pdev);return ret; }@@ -872,11 +885,11 @@ static int afu_dev_destroy(struct platform_device *pdev)static int port_enable_set (struct platform_device *pdev,bool enable) {struct dfl_feature_platform_data *pdata =dev_get_platdata (&pdev->dev); -int ret =0 ; +int ret;mutex_lock (&pdata->lock);if (enable) -__afu_port_enable (pdev); + ret =__afu_port_enable (pdev);else ret =__afu_port_disable (pdev);mutex_unlock (&pdata->lock);drivers/fpga/dfl-afu.h +0
-0drivers/fpga/dfl-afu.h struct dfl_afu { · line 84
Quoted 6 lines
drivers/fpga/dfl-afu.h +1
-1@@ -80,7 +80,7 @@ struct dfl_afu { };/* hold pdata->lock when call __afu_port_enable/disable */ -void __afu_port_enable (struct platform_device *pdev); +int __afu_port_enable (struct platform_device *pdev);int __afu_port_disable (struct platform_device *pdev);The other functions in this file have afu_* since the __afu_port_enable/disable
Quoted 3 lines
are used other places would it make sense to remove the '__' prefix ?
If you think so, maybe a cleanup patch later.
Yilun and Hao addressed this comment in their previous responses. We are using the '__' prefix to indicate highlight the fact caller needs to use care in managing the locking associated with these functions.
Thanks,
- Russ
Quoted 5 lines
Tom
Quoted 2 lines
drivers/fpga/dfl-afu.h +0
-0void afu_mmio_region_init (struct dfl_feature_platform_data *pdata);
Quoted 23 lines · Russ Weight
Quoted 21 lines · Tom Rix
Quoted 16 lines · Russ Weight
Port enable is not complete until ACK = 0. Change
__afu_port_enable()to guarantee that the enable process is complete by polling for ACK == 0.Signed-off-by Russ Weight <[email protected]>--- drivers/fpga/dfl-afu-error.c | 2 +- drivers/fpga/dfl-afu-main.c | 29 +++++++++++++++++++++-------- drivers/fpga/dfl-afu.h | 2 +- 3 files changed, 23 insertions(+), 10 deletions(-)drivers/fpga/dfl-afu-error.c +0
-0drivers/fpga/dfl-afu-error.c static int afu_port_err_clear(struct device *dev, u64 err) · line 104
Quoted 2 lines
drivers/fpga/dfl-afu-error.c +0
-0@@ -103,7 +103,7 @@ static int afu_port_err_clear(struct device *dev, u64 err)__afu_port_err_mask (dev, false);There is an earlier bit that sets ret =
-EINVAL.
Quoted 5 lines
Quoted 3 lines
This error will be lost or not handled well.
Right now it doesn't seem to be handled.
Good catch. I'll give priority to
-EINVALin the next version of the patch, as it is more informative in the context of this function.
Actually - Hao pointed out in his response that the falure to re-enable the port is a more serious error, so the code flow OK, but needs a comment.
- Russ
Quoted 115 lines
Quoted 32 lines
Quoted 32 lines
drivers/fpga/dfl-afu-error.c +1
-1/* Enable the Port by clear the reset */ -__afu_port_enable (pdev); + ret =__afu_port_enable (pdev); done:mutex_unlock (&pdata->lock);drivers/fpga/dfl-afu-main.c +4
-1@@ -21,6 +21,9 @@#include "dfl-afu.h" +#define RST_POLL_INVL 10/* us */ +#define RST_POLL_TIMEOUT 1000/* us */ +/** * __afu_port_enable - enable a port by clear reset * @pdev: port platform device. @@ -32,7 +35,7 @@* * The caller needs to hold lock for protection. */ -void __afu_port_enable (struct platform_device *pdev) +int __afu_port_enable (struct platform_device *pdev) {struct dfl_feature_platform_data *pdata =dev_get_platdata (&pdev->dev);void __iomem *base;@@ -41,7 +44,7 @@ void __afu_port_enable(struct platform_device *pdev)WARN_ON (!pdata->disable_count);Quoted 4 lines
drivers/fpga/dfl-afu-main.c void __afu_port_enable(struct platform_device *pdev) · line 47
Quoted 3 lines
drivers/fpga/dfl-afu-main.c +1
-1if (--pdata->disable_count !=0 ) -return ; +return 0 ;Is this really a success ? Maybe
-EBUSY?Yilun addressed this question in his previous response. This isessentially a reference count for nested disable calls. Weonly do the enable if the disable count has gone to zero, so this isn't an error condition.
Quoted 63 lines
Quoted 55 lines
drivers/fpga/dfl-afu-main.c +16
-6base =dfl_get_feature_ioaddr_by_id (&pdev->dev,PORT_FEATURE_ID_HEADER );@@ -49,10 +52,20 @@ void __afu_port_enable(struct platform_device *pdev) v =readq (base +PORT_HDR_CTRL ); v &= ~PORT_CTRL_SFTRST ;writeq (v, base +PORT_HDR_CTRL ); -} -#define RST_POLL_INVL 10/* us */ -#define RST_POLL_TIMEOUT 1000/* us */ +/* +* HW clears the ack bit to indicate that the port is fully out +* of reset. +*/ +if (readq_poll_timeout (base +PORT_HDR_CTRL , v, + !(v &PORT_CTRL_SFTRST_ACK ), +RST_POLL_INVL ,RST_POLL_TIMEOUT )) { +dev_err (&pdev->dev,"timeout, failure to enable device\n" ); +return -ETIMEDOUT ; + } + +return 0 ; +}/** * __afu_port_disable - disable a port by hold reset @@ -111,7 +124,7 @@ static int __port_reset(struct platform_device *pdev) ret =__afu_port_disable (pdev);if (!ret) -__afu_port_enable (pdev); + ret =__afu_port_enable (pdev);return ret; }@@ -872,11 +885,11 @@ static int afu_dev_destroy(struct platform_device *pdev)static int port_enable_set (struct platform_device *pdev,bool enable) {struct dfl_feature_platform_data *pdata =dev_get_platdata (&pdev->dev); -int ret =0 ; +int ret;mutex_lock (&pdata->lock);if (enable) -__afu_port_enable (pdev); + ret =__afu_port_enable (pdev);else ret =__afu_port_disable (pdev);mutex_unlock (&pdata->lock);drivers/fpga/dfl-afu.h +0
-0drivers/fpga/dfl-afu.h struct dfl_afu { · line 84
Quoted 6 lines
drivers/fpga/dfl-afu.h +1
-1@@ -80,7 +80,7 @@ struct dfl_afu { };/* hold pdata->lock when call __afu_port_enable/disable */ -void __afu_port_enable (struct platform_device *pdev); +int __afu_port_enable (struct platform_device *pdev);int __afu_port_disable (struct platform_device *pdev);The other functions in this file have afu_* since the __afu_port_enable/disable
Quoted 3 lines
are used other places would it make sense to remove the '__' prefix ?
If you think so, maybe a cleanup patch later.
Yilun and Hao addressed this comment in their previous responses. We are using the '__' prefix to indicate highlight the fact caller needs to use care in managing the locking associated with these functions.
Thanks,
- RussQuoted 4 lines
Tom
Quoted 2 lines
drivers/fpga/dfl-afu.h +0
-0void afu_mmio_region_init (struct dfl_feature_platform_data *pdata);
Quoted 11 lines
-----Original Message-----
From: Russ Weight <[email protected]>
Sent: Friday, September 18, 2020 2:32 AM
To: [email protected]; [email protected]; linux- [email protected]
Cc: [email protected]; [email protected]; Xu, Yilun <[email protected]>; Wu, Hao <[email protected]>; Gerlach, Matthew <[email protected]>; Weight, Russell H <[email protected]>
Subject: [PATCH v2 1/1] fpga: dfl: afu: harden port enable logic
Quoted 3 lines
Port enable is not complete until ACK = 0. Change
__afu_port_enable()to guarantee that the enable process is complete by polling for ACK == 0.
The description of this port reset ack bit is
" After initiating a Port soft reset, SW should monitor this bit. HW will set this bit when all outstanding requests initiated by this port have been drained, and the minimum soft reset pulse width has elapsed. "
But no description about what to do when clearing a Port soft reset to enable the port.
So we need to understand clearly on why we need this change (e.g. what may happen without this change), and will it apply for all existing DFL devices and future ones, or just for one specific card. Could you please help? : )
Quoted 82 lines
Signed-off-by Russ Weight <[email protected]>--- drivers/fpga/dfl-afu-error.c | 2 +- drivers/fpga/dfl-afu-main.c | 29 +++++++++++++++++++++-------- drivers/fpga/dfl-afu.h | 2 +- 3 files changed, 23 insertions(+), 10 deletions(-)drivers/fpga/dfl-afu-error.c +1
-1@@ -103,7 +103,7 @@ static int afu_port_err_clear(struct device *dev, u64 err)__afu_port_err_mask (dev ,false );/* Enable the Port by clear the reset */ -__afu_port_enable (pdev); + ret =__afu_port_enable (pdev); done:drivers/fpga/dfl-afu-error.c +0
-0mutex_unlock (&pdata->lock);drivers/fpga/dfl-afu-main.c +4
-1@@ -21,6 +21,9 @@#include "dfl-afu.h" +#define RST_POLL_INVL 10/* us */ +#define RST_POLL_TIMEOUT 1000/* us */ +/** * __afu_port_enable - enable a port by clear reset * @pdev: port platform device. @@ -32,7 +35,7 @@* * The caller needs to hold lock for protection. */ -void __afu_port_enable (struct platform_device *pdev) +int __afu_port_enable (struct platform_device *pdev) {struct dfl_feature_platform_data *pdata =dev_get_platdata (&pdev- >dev);drivers/fpga/dfl-afu-main.c +14
-4void __iomem *base;@@ -41,7 +44,7 @@ void __afu_port_enable(struct platform_device *pdev)WARN_ON (!pdata->disable_count);if (--pdata->disable_count !=0 ) -return ; +return 0 ; base =dfl_get_feature_ioaddr_by_id (&pdev->dev,PORT_FEATURE_ID_HEADER );@@ -49,10 +52,20 @@ void __afu_port_enable(struct platform_device *pdev) v =readq (base +PORT_HDR_CTRL ); v &= ~PORT_CTRL_SFTRST ;writeq (v, base +PORT_HDR_CTRL ); -} -#define RST_POLL_INVL 10/* us */ -#define RST_POLL_TIMEOUT 1000/* us */ +/* +* HW clears the ack bit to indicate that the port is fully out +* of reset. +*/ +if (readq_poll_timeout (base +PORT_HDR_CTRL , v, + !(v &PORT_CTRL_SFTRST_ACK ), +RST_POLL_INVL ,RST_POLL_TIMEOUT )) { +dev_err (&pdev->dev,"timeout, failure to enable device\n" ); +return -ETIMEDOUT ; + } + +return 0 ; +}/** *
__afu_port_disable- disable a port by hold reset
drivers/fpga/dfl-afu-main.c static int __port_reset(struct platform_device *pdev) · line 129
Quoted 7 lines
drivers/fpga/dfl-afu-main.c +1
-1@@ -111,7 +124,7 @@ static int __port_reset(struct platform_device *pdev) ret =__afu_port_disable (pdev);if (!ret) -__afu_port_enable (pdev); + ret =__afu_port_enable (pdev);return ret;
What about:
ret = __afu_port_disable(pdev); if (ret) return ret; return __afu_port_enable(pdev);
Thanks
Hao
Quoted 32 lines
drivers/fpga/dfl-afu-main.c +2
-2}@@ -872,11 +885,11 @@ static int afu_dev_destroy(struct platform_device *pdev)static int port_enable_set (struct platform_device *pdev,bool enable) {struct dfl_feature_platform_data *pdata =dev_get_platdata (&pdev- >dev); -int ret =0 ; +int ret;mutex_lock (&pdata->lock);if (enable) -__afu_port_enable (pdev); + ret =__afu_port_enable (pdev);else drivers/fpga/dfl-afu-main.c +0
-0ret =__afu_port_disable (pdev);mutex_unlock (&pdata->lock);drivers/fpga/dfl-afu.h +1
-2@@ -80,7 +80,7 @@ struct dfl_afu { };/* hold pdata->lock when call __afu_port_enable/disable */ -void __afu_port_enable (struct platform_device *pdev); +int __afu_port_enable (struct platform_device *pdev);int __afu_port_disable (struct platform_device *pdev);void afu_mmio_region_init (struct dfl_feature_platform_data *pdata); --2.17.1
Sorry for the delay on this patch. It seemed like a lower priority patch than others, since we haven't seen any issues with current products. Please my responses inline.
Quoted 21 lines · Hao
Quoted 11 lines
-----Original Message-----
From: Russ Weight <[email protected]>
Sent: Friday, September 18, 2020 2:32 AM
To: [email protected]; [email protected]; linux- [email protected]
Cc: [email protected]; [email protected]; Xu, Yilun <[email protected]>; Wu, Hao <[email protected]>; Gerlach, Matthew <[email protected]>; Weight, Russell H <[email protected]>
Subject: [PATCH v2 1/1] fpga: dfl: afu: harden port enable logicQuoted 3 lines
Port enable is not complete until ACK = 0. Change
__afu_port_enable()to guarantee that the enable process is complete by polling for ACK == 0.The description of this port reset ack bit is
" After initiating a Port soft reset, SW should monitor this bit. HW will set this bit when all outstanding requests initiated by this port have been drained, and the minimum soft reset pulse width has elapsed. "
Quoted 7 lines
But no description about what to do when clearing a Port soft reset to enable the port.
So we need to understand clearly on why we need this change (e.g. what may happen without this change), and will it apply for all existing DFL devices and future ones, or just for one specific card. Could you please help? : )
I touched bases with the hardware engineers. The recommendation to wait for ACK to be cleared is new with OFS and is documented in the latest OFS specification as follows (see step #4):
Quoted 9 lines
3.7.1 AFU Soft Resets
Software may cause a soft reset to be issued to the AFU as follows:
1. Assert the PortSoftReset field of thePORT_CONTROLregister
2. Wait for the Port to acknowledge the soft reset by monitoring the PortSoftResetAck field of thePORT_CONTROLregister, i.e. PortSoftResetAck=1
3. Deasserting the PortSoftReset field
4. Wait for the Port to acknowledge the soft reset de-assertion by monitoring the PortSoftResetAck field of thePORT_CONTROLregister, i.e. PortSoftResetAck=0
Quoted 3 lines
This sequence ensures that outstanding transactions are suitably flushed and that the FIM minimum reset pulse width is respected. Failing to follow this sequence leaves the AFU in an undefined state.
The OFS specification has not been posted publicly, yet.
Also, this is how it was explained to me:
Quoted 8 lines
In most scenario, port will be able to get out of reset soon enough when SW releases the port reset, especially on all the PAC products which have been verified before release.
Polling for HW to clear the ACK is meant to handle the following scenarios:
* Different platform can take variable period of time to get out of reset
* Bug in the HW that hold the port in reset
So this change is not required for the currently released PAC cards, but it is needed for OFS based products. I don't think there is any reason to hold off on the patch, as it is still valid for current products.
Quoted 87 lines
Quoted 87 lines
Signed-off-by Russ Weight <[email protected]>--- drivers/fpga/dfl-afu-error.c | 2 +- drivers/fpga/dfl-afu-main.c | 29 +++++++++++++++++++++-------- drivers/fpga/dfl-afu.h | 2 +- 3 files changed, 23 insertions(+), 10 deletions(-)drivers/fpga/dfl-afu-error.c +1
-1@@ -103,7 +103,7 @@ static int afu_port_err_clear(struct device *dev, u64 err)__afu_port_err_mask (dev ,false );/* Enable the Port by clear the reset */ -__afu_port_enable (pdev); +ret =__afu_port_enable (pdev); done:
mutex_unlock(&pdata->lock);drivers/fpga/dfl-afu-main.c +4
-1@@ -21,6 +21,9 @@#include "dfl-afu.h" +#define RST_POLL_INVL 10/* us */ +#define RST_POLL_TIMEOUT 1000/* us */ +/** * __afu_port_enable - enable a port by clear reset * @pdev: port platform device. @@ -32,7 +35,7 @@* * The caller needs to hold lock for protection. */ -void __afu_port_enable (struct platform_device *pdev) +int __afu_port_enable (struct platform_device *pdev) {struct dfl_feature_platform_data *pdata =dev_get_platdata (&pdev- > dev);void
__iomem*base;drivers/fpga/dfl-afu-main.c +14
-4@@ -41,7 +44,7 @@ void __afu_port_enable(struct platform_device *pdev)WARN_ON (!pdata->disable_count);if (--pdata->disable_count !=0 ) -return ; +return 0 ; base =dfl_get_feature_ioaddr_by_id (&pdev->dev,PORT_FEATURE_ID_HEADER );@@ -49,10 +52,20 @@ void __afu_port_enable(struct platform_device *pdev) v =readq (base +PORT_HDR_CTRL ); v &= ~PORT_CTRL_SFTRST ;writeq (v, base +PORT_HDR_CTRL ); -} -#define RST_POLL_INVL 10/* us */ -#define RST_POLL_TIMEOUT 1000/* us */ +/* +* HW clears the ack bit to indicate that the port is fully out +* of reset. +*/ +if (readq_poll_timeout (base +PORT_HDR_CTRL , v, + !(v &PORT_CTRL_SFTRST_ACK ), +RST_POLL_INVL ,RST_POLL_TIMEOUT )) { +dev_err (&pdev->dev,"timeout, failure to enable device\n" ); +return -ETIMEDOUT ; +} + +return 0 ; +}/** *
__afu_port_disable- disable a port by hold resetdrivers/fpga/dfl-afu-main.c +1
-1@@ -111,7 +124,7 @@ static int __port_reset(struct platform_device *pdev) ret =__afu_port_disable (pdev);if (!ret) -__afu_port_enable (pdev); +ret =__afu_port_enable (pdev);
Quoted 8 lines
drivers/fpga/dfl-afu-main.c static int __port_reset(struct platform_device *pdev) · line 129
Quoted 1 line
drivers/fpga/dfl-afu-main.c +0
-0return ret;What about:
ret =
__afu_port_disable(pdev);
if (ret)
return ret;return
__afu_port_enable(pdev);
Sure - I'll make this change.
Thanks,
- Russ
Quoted 36 lines
Thanks
HaoQuoted 32 lines
drivers/fpga/dfl-afu-main.c +2
-2}@@ -872,11 +885,11 @@ static int afu_dev_destroy(struct platform_device *pdev)static int port_enable_set (struct platform_device *pdev,bool enable) {struct dfl_feature_platform_data *pdata =dev_get_platdata (&pdev- > dev); -int ret =0 ; +int ret;mutex_lock (&pdata->lock);if (enable) -__afu_port_enable (pdev); +ret =__afu_port_enable (pdev);else ret =
__afu_port_disable(pdev);
mutex_unlock(&pdata->lock);drivers/fpga/dfl-afu.h +1
-2@@ -80,7 +80,7 @@ struct dfl_afu { };/* hold pdata->lock when call __afu_port_enable/disable */ -void __afu_port_enable (struct platform_device *pdev); +int __afu_port_enable (struct platform_device *pdev);int __afu_port_disable (struct platform_device *pdev);void afu_mmio_region_init (struct dfl_feature_platform_data *pdata); --2.17.1
Quoted 63 lines
Subject: Re: [PATCH v2 1/1] fpga: dfl: afu: harden port enable logic
Sorry for the delay on this patch. It seemed like a lower priority patch than others, since we haven't seen any issues with current products. Please my responses inline.
Quoted 21 lines · Hao
Quoted 11 lines
-----Original Message-----
From: Russ Weight <[email protected]>
Sent: Friday, September 18, 2020 2:32 AM
To: [email protected]; [email protected]; linux- [email protected]
Cc: [email protected]; [email protected]; Xu, Yilun <[email protected]>; Wu, Hao <[email protected]>; Gerlach, Matthew <[email protected]>; Weight, Russell H <[email protected]>
Subject: [PATCH v2 1/1] fpga: dfl: afu: harden port enable logicQuoted 3 lines
Port enable is not complete until ACK = 0. Change
__afu_port_enable()to guarantee that the enable process is complete by polling for ACK == 0.The description of this port reset ack bit is
" After initiating a Port soft reset, SW should monitor this bit. HW will set this bit when all outstanding requests initiated by this port have been drained, and the minimum soft reset pulse width has elapsed. "
Quoted 7 lines
But no description about what to do when clearing a Port soft reset to enable the port.
So we need to understand clearly on why we need this change (e.g. what may happen without this change), and will it apply for all existing DFL devices and future ones, or just for one specific card. Could you please help? : )
I touched bases with the hardware engineers. The recommendation to wait for ACK to be cleared is new with OFS and is documented in the latest OFS specification as follows (see step #4):
Quoted 5 lines
3.7.1 AFU Soft Resets
Software may cause a soft reset to be issued to the AFU as follows:
1. Assert the PortSoftReset field of thePORT_CONTROLregister
2. Wait for the Port to acknowledge the soft reset by monitoring the PortSoftResetAck field of thePORT_CONTROLregister, i.e.PortSoftResetAck=1
Quoted 2 lines
3. Deasserting the PortSoftReset field
4. Wait for the Port to acknowledge the soft reset de-assertion by monitoringthe
Quoted 1 line
PortSoftResetAck field of the
PORT_CONTROLregister, i.e.PortSoftResetAck=0
Quoted 4 lines
This sequence ensures that outstanding transactions are suitably flushed and that the FIM minimum reset pulse width is respected. Failing to follow this sequence leaves the AFU in an undefined state.
The OFS specification has not been posted publicly, yet.
Also, this is how it was explained to me:
Quoted 4 lines
In most scenario, port will be able to get out of reset soon enough when SW releases the port reset, especially on all the PAC products which have been verified before release.
Quoted 8 lines
Quoted 4 lines
Polling for HW to clear the ACK is meant to handle the following scenarios:
* Different platform can take variable period of time to get out of reset
* Bug in the HW that hold the port in resetSo this change is not required for the currently released PAC cards, but it is needed for OFS based products. I don't think there is any reason to hold off on the patch, as it is still valid for current products.
As you know, this driver is used for different cards, and we need to make sure new changes introduced in new version spec, don't break old products as we are sharing the same driver. and we are not sure if in the future some new products but still uses old specs, and then things may be broken if the driver which always perform new flow. Another method is that introduce 1 bit in hardware register to tell the driver to perform the additional steps, then it can avoid impacts to the old products. If this can't be done, then we at least need to verify this change on all existing hardware and suggest users to follow new spec only.
Hao
Quoted 68 lines · Hao
Quoted 61 lines
Subject: Re: [PATCH v2 1/1] fpga: dfl: afu: harden port enable logic
Sorry for the delay on this patch. It seemed like a lower priority patch than others, since we haven't seen any issues with current products. Please my responses inline.
Quoted 21 lines · Hao
Quoted 11 lines
-----Original Message-----
From: Russ Weight <[email protected]>
Sent: Friday, September 18, 2020 2:32 AM
To: [email protected]; [email protected]; linux- [email protected]
Cc: [email protected]; [email protected]; Xu, Yilun <[email protected]>; Wu, Hao <[email protected]>; Gerlach, Matthew <[email protected]>; Weight, Russell H <[email protected]>
Subject: [PATCH v2 1/1] fpga: dfl: afu: harden port enable logicQuoted 3 lines
Port enable is not complete until ACK = 0. Change
__afu_port_enable()to guarantee that the enable process is complete by polling for ACK == 0.The description of this port reset ack bit is
" After initiating a Port soft reset, SW should monitor this bit. HW will set this bit when all outstanding requests initiated by this port have been drained, and the minimum soft reset pulse width has elapsed. "
Quoted 7 lines
But no description about what to do when clearing a Port soft reset to enable the port.
So we need to understand clearly on why we need this change (e.g. what may happen without this change), and will it apply for all existing DFL devices and future ones, or just for one specific card. Could you please help? : )
I touched bases with the hardware engineers. The recommendation to wait for ACK to be cleared is new with OFS and is documented in the latest OFS specification as follows (see step #4):
Quoted 5 lines
3.7.1 AFU Soft Resets
Software may cause a soft reset to be issued to the AFU as follows:
1. Assert the PortSoftReset field of thePORT_CONTROLregister
2. Wait for the Port to acknowledge the soft reset by monitoring the PortSoftResetAck field of thePORT_CONTROLregister, i.e.PortSoftResetAck=1
Quoted 2 lines
3. Deasserting the PortSoftReset field
4. Wait for the Port to acknowledge the soft reset de-assertion by monitoringthe
Quoted 1 line
PortSoftResetAck field of the
PORT_CONTROLregister, i.e.PortSoftResetAck=0
Quoted 3 lines
This sequence ensures that outstanding transactions are suitably flushed and that the FIM minimum reset pulse width is respected. Failing to follow this sequence leaves the AFU in an undefined state.
The OFS specification has not been posted publicly, yet.
Also, this is how it was explained to me:
Quoted 4 lines
In most scenario, port will be able to get out of reset soon enough when SW releases the port reset, especially on all the PAC products which have been verified before release.
Quoted 7 lines
Quoted 4 lines
Polling for HW to clear the ACK is meant to handle the following scenarios:
* Different platform can take variable period of time to get out of reset
* Bug in the HW that hold the port in resetSo this change is not required for the currently released PAC cards, but it is needed for OFS based products. I don't think there is any reason to hold off on the patch, as it is still valid for current products.
Quoted 9 lines
As you know, this driver is used for different cards, and we need to make sure new changes introduced in new version spec, don't break old products as we are sharing the same driver. and we are not sure if in the future some new products but still uses old specs, and then things may be broken if the driver which always perform new flow. Another method is that introduce 1 bit in hardware register to tell the driver to perform the additional steps, then it can avoid impacts to the old products. If this can't be done, then we at least need to verify this change on all existing hardware and suggest users to follow new spec only.
According to the HW engineers, the RTL implementation has not changed; it is the same as the RTL in the current PAC products. Polling for HW to clear the ACK is something we could have (should have?) been doing all along. The timing hasn't been an issue for the current PAC products, as proven by our testing. However, with OFS we cannot anticipate what the timing will be for customer designed products, so the specification is calling out this requirement as a precaution.
I am using a development machine that has the older PAC devices installed. I cleared port errors on these cards as a quick check, and the reset completes without hanging - which indicates that the ACK bit is in fact getting cleared. So there is not need for any device-specific conditional statements here.
- Russ
Quoted 2 lines
Hao
Quoted 81 lines · Russ Weight
Quoted 68 lines · Hao
Quoted 61 lines
Subject: Re: [PATCH v2 1/1] fpga: dfl: afu: harden port enable logic
Sorry for the delay on this patch. It seemed like a lower priority patch than others, since we haven't seen any issues with current products. Please my responses inline.
Quoted 21 lines · Hao
Quoted 11 lines
-----Original Message-----
From: Russ Weight <[email protected]>
Sent: Friday, September 18, 2020 2:32 AM
To: [email protected]; [email protected]; linux- [email protected]
Cc: [email protected]; [email protected]; Xu, Yilun <[email protected]>; Wu, Hao <[email protected]>; Gerlach, Matthew <[email protected]>; Weight, Russell H <[email protected]>
Subject: [PATCH v2 1/1] fpga: dfl: afu: harden port enable logicQuoted 3 lines
Port enable is not complete until ACK = 0. Change
__afu_port_enable()to guarantee that the enable process is complete by polling for ACK == 0.The description of this port reset ack bit is
" After initiating a Port soft reset, SW should monitor this bit. HW will set this bit when all outstanding requests initiated by this port have been drained, and the minimum soft reset pulse width has elapsed. "
Quoted 7 lines
But no description about what to do when clearing a Port soft reset to enable the port.
So we need to understand clearly on why we need this change (e.g. what may happen without this change), and will it apply for all existing DFL devices and future ones, or just for one specific card. Could you please help? : )
I touched bases with the hardware engineers. The recommendation to wait for ACK to be cleared is new with OFS and is documented in the latest OFS specification as follows (see step #4):
Quoted 5 lines
3.7.1 AFU Soft Resets
Software may cause a soft reset to be issued to the AFU as follows:
1. Assert the PortSoftReset field of thePORT_CONTROLregister
2. Wait for the Port to acknowledge the soft reset by monitoring the PortSoftResetAck field of thePORT_CONTROLregister, i.e.PortSoftResetAck=1
Quoted 2 lines
3. Deasserting the PortSoftReset field
4. Wait for the Port to acknowledge the soft reset de-assertion by monitoringthe
Quoted 1 line
PortSoftResetAck field of the
PORT_CONTROLregister, i.e.PortSoftResetAck=0
Quoted 3 lines
This sequence ensures that outstanding transactions are suitably flushed and that the FIM minimum reset pulse width is respected. Failing to follow this sequence leaves the AFU in an undefined state.
The OFS specification has not been posted publicly, yet.
Also, this is how it was explained to me:
Quoted 4 lines
In most scenario, port will be able to get out of reset soon enough when SW releases the port reset, especially on all the PAC products which have been verified before release.
Quoted 7 lines
Quoted 4 lines
Polling for HW to clear the ACK is meant to handle the following scenarios:
* Different platform can take variable period of time to get out of reset
* Bug in the HW that hold the port in resetSo this change is not required for the currently released PAC cards, but it is needed for OFS based products. I don't think there is any reason to hold off on the patch, as it is still valid for current products.
Quoted 9 lines
As you know, this driver is used for different cards, and we need to make sure new changes introduced in new version spec, don't break old products as we are sharing the same driver. and we are not sure if in the future some new products but still uses old specs, and then things may be broken if the driver which always perform new flow. Another method is that introduce 1 bit in hardware register to tell the driver to perform the additional steps, then it can avoid impacts to the old products. If this can't be done, then we at least need to verify this change on all existing hardware and suggest users to follow new spec only.
Quoted 3 lines
According to the HW engineers, the RTL implementation has not changed; it is the same as the RTL in the current PAC products. Polling for HW to clear the ACK is something we could have (should have?) been doing all along. The timing
I also confirmed with HW engineers. The original specification was not precise. The code should have been doing this all along.
Matthew Gerlach
Quoted 16 lines
hasn't been an issue for the current PAC products, as proven by our testing. However, with OFS we cannot anticipate what the timing will be for customer designed products, so the specification is calling out this requirement as a precaution.
I am using a development machine that has the older PAC devices installed. I cleared port errors on these cards as a quick check, and the reset completes without hanging - which indicates that the ACK bit is in fact getting cleared. So there is not need for any device-specific conditional statements here.
- Russ
Quoted 2 lines
Hao
Quoted 86 lines · Russ Weight
Quoted 10 lines · Hao
Quoted 6 lines
Quoted 6 lines
Subject: Re: [PATCH v2 1/1] fpga: dfl: afu: harden port enable logic
Sorry for the delay on this patch. It seemed like a lower priority patch than others, since we haven't seen any issues with current products. Please my responses inline.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines · Hao
Quoted 6 lines
Quoted 6 lines
-----Original Message-----
From: Russ Weight <[email protected]>
Sent: Friday, September 18, 2020 2:32 AM
To: [email protected]; [email protected]; linux- [email protected]
Cc: [email protected]; [email protected]; Xu, YilunQuoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 15 lines
Quoted 8 lines
Wu, Hao <[email protected]>; Gerlach, Matthew <[email protected]>; Weight, Russell H <[email protected]>
Subject: [PATCH v2 1/1] fpga: dfl: afu: harden port enable logicPort enable is not complete until ACK = 0. Change
__afu_port_enable()to guarantee that the enable process is complete by polling for ACK == 0.The description of this port reset ack bit is
" After initiating a Port soft reset, SW should monitor this bit. HW will set this bit when all outstanding requests initiated by this port have been drained, and the minimum soft reset pulse width has elapsed. "
Quoted 7 lines
But no description about what to do when clearing a Port soft reset to enable the port.
So we need to understand clearly on why we need this change (e.g. what may happen without this change), and will it apply for all existing DFL devices and future ones, or just for one specific card. Could you please help? : )
I touched bases with the hardware engineers. The recommendation to wait for ACK to be cleared is new with OFS and is documented in the latest OFS specification as follows (see step #4):
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
3.7.1 AFU Soft Resets
Software may cause a soft reset to be issued to the AFU as follows:
1. Assert the PortSoftReset field of thePORT_CONTROLregister
2. Wait for the Port to acknowledge the soft reset by monitoring the PortSoftResetAck field of thePORT_CONTROLregister, i.e.PortSoftResetAck=1
Quoted 2 lines
3. Deasserting the PortSoftReset field
4. Wait for the Port to acknowledge the soft reset de-assertion bymonitoring
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
the
Quoted 1 line
PortSoftResetAck field of the
PORT_CONTROLregister, i.e.PortSoftResetAck=0
Quoted 1 line
This sequence ensures that outstanding transactions are suitably flushed
and
Quoted 27 lines
Quoted 26 lines
Quoted 10 lines
Quoted 2 lines
that the FIM minimum reset pulse width is respected. Failing to follow this sequence leaves the AFU in an undefined state.
The OFS specification has not been posted publicly, yet.
Also, this is how it was explained to me:
Quoted 4 lines
In most scenario, port will be able to get out of reset soon enough when SW releases the port reset, especially on all the PAC products which have been verified before release.
Quoted 7 lines
Quoted 4 lines
Polling for HW to clear the ACK is meant to handle the following scenarios:
* Different platform can take variable period of time to get out of reset
* Bug in the HW that hold the port in resetSo this change is not required for the currently released PAC cards, but it is needed for OFS based products. I don't think there is any reason to hold off on the patch, as it is still valid for current products.
As you know, this driver is used for different cards, and we need to make sure new changes introduced in new version spec, don't break old products as we are sharing the same driver. and we are not sure if in the future some new products but still uses old specs, and then things may be broken if the driver which always perform new flow. Another method is that introduce 1 bit in hardware register to tell the driver to perform the additional steps, then it can avoid impacts to the old products. If this can't be done, then we at least need to verify this change on all existing hardware and suggest users to follow new spec only.
Quoted 7 lines
Quoted 3 lines
According to the HW engineers, the RTL implementation has not changed; it is the same as the RTL in the current PAC products. Polling for HW to clear the ACK is something we could have (should have?) been doing all along. The
timing
I also confirmed with HW engineers. The original specification was not precise. The code should have been doing this all along.
Thanks for this confirmation, then it sounds good to me. I think only Intel hardware is using this driver now, so if this is confirmed from hardware side, then we should be safe to take this one.
Hao
Quoted 19 lines
Matthew Gerlach
Quoted 16 lines
hasn't been an issue for the current PAC products, as proven by our testing. However, with OFS we cannot anticipate what the timing will be for customer designed products, so the specification is calling out this requirement as a precaution.
I am using a development machine that has the older PAC devices installed. I cleared port errors on these cards as a quick check, and the reset completes without hanging - which indicates that the ACK bit is in fact getting cleared. So there is not need for any device-specific conditional statements here.
- Russ
Quoted 2 lines
Hao