cxl/mbox: Robustify handling of mbox_cmd.return_code
Davidlohr Bueso
maintainercover cxl/mbox: Robustify handling of mbox_cmd.return_code
Hello,
These are some patchlets that came up while going through the code.
Currently the return_code from a completed mbox command is handled as either successful or not. This series teaches the driver to better deal with the different returns from the hardware, allowing better debugging and mapping to proper kernel errno semantics (which are left unchanged for now) as well as more ad-hoc handling.
Patches 1 and 2 are small nits.
Patch 3, 4 and 5 implement and use the new calls.
Applies on top of linux-cxl's next branch but have not really been very tested as I'm not sure how to actually send mbox commands but thought I'd post regardless.
Davidlohr Bueso (5):
cxl/mbox: Move mbox_mutex usage comment
cxl/pci: Use CXL_MBOX_SUCCESS to check against mbox_cmd return code cxl/mbox: Improve handling of mbox_cmd return codes cxl/mbox: Use new return_code handling
cxl/mbox: Retry sending mbox command upon RETRY return_code
drivers/cxl/core/mbox.c | 23 +++++++---- drivers/cxl/cxlmem.h | 90 ++++++++++++++++++++++++++++++++++++++++- drivers/cxl/pci.c | 7 ++-- 3 files changed, 108 insertions(+), 12 deletions(-)
--
2.26.2
Davidlohr Bueso
maintainerPATCH 1/5 cxl/mbox: Move mbox_mutex usage comment
... this is better served in the callback that actually grabs the lock.
--- drivers/cxl/core/mbox.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/cxl/core/mbox.c +2 -1
@@ -136,7 +136,7 @@ static struct cxl_mem_command *cxl_mem_find_command(u16 opcode)* @out: Caller allocated buffer for the output. * @out_size: Expected size of output. * -* Context: Any context. Will acquire and release mbox_mutex. +* Context: Any context. * Return: * * %>=0 - Number of bytes returned in @out. * * %-E2BIG - Payload is too large for hardware.
@@ -165,6 +165,7 @@ int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in,if (out_size > cxlds->payload_size)return -E2BIG ; +/* acquire and releases the mbox_mutex */ rc = cxlds->mbox_send (cxlds, &mbox_cmd);if (rc)return rc;
Signature · 1 line
2.26.2
Re: [PATCH 1/5] cxl/mbox: Move mbox_mutex usage comment
Quoted 33 lines · Davidlohr Bueso
... this is better served in the callback that actually grabs the lock.
Signed-off-by Davidlohr Bueso <[email protected]>--- drivers/cxl/core/mbox.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/cxl/core/mbox.c +2
-1@@ -136,7 +136,7 @@ static struct cxl_mem_command *cxl_mem_find_command(u16 opcode)* @out: Caller allocated buffer for the output. * @out_size: Expected size of output. * -* Context: Any context. Will acquire and release mbox_mutex. +* Context: Any context. * Return: * * %>=0 - Number of bytes returned in @out. * * %-E2BIG - Payload is too large for hardware. @@ -165,6 +165,7 @@ int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in,if (out_size > cxlds->payload_size)return -E2BIG ; +/* acquire and releases the mbox_mutex */ rc = cxlds->mbox_send (cxlds, &mbox_cmd);if (rc)return rc;Signature · 2 lines
-- 2.26.2
Reviewed by: Adam Manzanares <[email protected]>
Re: [PATCH 1/5] cxl/mbox: Move mbox_mutex usage comment
In reply to Davidlohr BuesoQuoted 23 lines · Davidlohr Bueso
... this is better served in the callback that actually grabs the lock.
Signed-off-by Davidlohr Bueso <[email protected]>--- drivers/cxl/core/mbox.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/cxl/core/mbox.c +1
-1@@ -136,7 +136,7 @@ static struct cxl_mem_command *cxl_mem_find_command(u16 opcode)* @out: Caller allocated buffer for the output. * @out_size: Expected size of output. * -* Context: Any context. Will acquire and release mbox_mutex. +* Context: Any context. * Return: * * %>=0 - Number of bytes returned in @out. * * %-E2BIG - Payload is too large for hardware.
drivers/cxl/core/mbox.c int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in, · line 168
Quoted 4 lines
drivers/cxl/core/mbox.c +1
-0@@ -165,6 +165,7 @@ int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in,if (out_size > cxlds->payload_size)return -E2BIG ; +/* acquire and releases the mbox_mutex */
I'm not even sure this comment is necessary, because that's what lockdep is for, and it's also not true for cxl_mock_mbox_send(). Nothing outside of the ->mbox_send() implementation should be assuming that ->mbox_send() is enforcing serialization. So let's just drop this second hunk.
Davidlohr Bueso
maintainerPATCH 2/5 cxl/pci: Use CXL_MBOX_SUCCESS to check against mbox_cmd return code
In reply to Davidlohr BuesoAlso mention the need for the caller to check against any errors from the hardware in return_code.
--- drivers/cxl/pci.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/cxl/pci.c +2 -2
@@ -177,9 +177,9 @@ static int __cxl_pci_mbox_send_cmd(struct cxl_dev_state *cxlds, mbox_cmd->return_code =FIELD_GET (CXLDEV_MBOX_STATUS_RET_CODE_MASK , status_reg); -if (mbox_cmd->return_code !=0 ) { +if (mbox_cmd->return_code !=CXL_MBOX_SUCCESS ) {dev_dbg (dev,"Mailbox operation had an error\n" ); -return 0 ; +return 0 ;/* completed but caller must check return_code */ }/* #7 */
Signature · 1 line
2.26.2
Re: [PATCH 2/5] cxl/pci: Use CXL_MBOX_SUCCESS to check against mbox_cmd return code
Quoted 28 lines · Davidlohr Bueso
Also mention the need for the caller to check against any errors from the hardware in
return_code.Signed-off-by Davidlohr Bueso <[email protected]>--- drivers/cxl/pci.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/cxl/pci.c +2
-2@@ -177,9 +177,9 @@ static int __cxl_pci_mbox_send_cmd(struct cxl_dev_state *cxlds, mbox_cmd->return_code =FIELD_GET (CXLDEV_MBOX_STATUS_RET_CODE_MASK , status_reg); -if (mbox_cmd->return_code !=0 ) { +if (mbox_cmd->return_code !=CXL_MBOX_SUCCESS ) {dev_dbg (dev,"Mailbox operation had an error\n" ); -return 0 ; +return 0 ;/* completed but caller must check return_code */ }/* #7 */ Signature · 2 lines
-- 2.26.2
Reviewed by: Adam Manzanares <[email protected]>
Dan Williams
AppliedRe: [PATCH 2/5] cxl/pci: Use CXL_MBOX_SUCCESS to check against mbox_cmd return code
In reply to Davidlohr BuesoQuoted 14 lines · Davidlohr Bueso
Also mention the need for the caller to check against any errors from the hardware in
return_code.Signed-off-by Davidlohr Bueso <[email protected]>--- drivers/cxl/pci.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/cxl/pci.c +0
-0
drivers/cxl/pci.c static int __cxl_pci_mbox_send_cmd(struct cxl_dev_state *cxlds, · line 182
Quoted 8 lines
drivers/cxl/pci.c +2
-2@@ -177,9 +177,9 @@ static int __cxl_pci_mbox_send_cmd(struct cxl_dev_state *cxlds, mbox_cmd->return_code =FIELD_GET (CXLDEV_MBOX_STATUS_RET_CODE_MASK , status_reg); -if (mbox_cmd->return_code !=0 ) { +if (mbox_cmd->return_code !=CXL_MBOX_SUCCESS ) {dev_dbg (dev,"Mailbox operation had an error\n" ); -return 0 ; +return 0 ;/* completed but caller must check return_code */
Looks like a good clarification to me.
Applied.
...but note it won't appear anywhere until after v5.18-rc1 is out.
Re: [PATCH 2/5] cxl/pci: Use CXL_MBOX_SUCCESS to check against mbox_cmd return code
Quoted 27 lines · Dan Williams
Quoted 14 lines · Davidlohr Bueso
Also mention the need for the caller to check against any errors from the hardware in
return_code.Signed-off-by Davidlohr Bueso <[email protected]>--- drivers/cxl/pci.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/cxl/pci.c +0
-0drivers/cxl/pci.c static int __cxl_pci_mbox_send_cmd(struct cxl_dev_state *cxlds, · line 182
Quoted 8 lines
drivers/cxl/pci.c +2
-2@@ -177,9 +177,9 @@ static int __cxl_pci_mbox_send_cmd(struct cxl_dev_state *cxlds, mbox_cmd->return_code =FIELD_GET (CXLDEV_MBOX_STATUS_RET_CODE_MASK , status_reg); -if (mbox_cmd->return_code !=0 ) { +if (mbox_cmd->return_code !=CXL_MBOX_SUCCESS ) {dev_dbg (dev,"Mailbox operation had an error\n" ); -return 0 ; +return 0 ;/* completed but caller must check return_code */ Looks like a good clarification to me.
Quoted 3 lines
Applied.
...but note it won't appear anywhere until after v5.18-rc1 is out.
...although given there are already feedback comments on patch 1 and 3 I also don't mind if you resend the full series for v2.
Davidlohr Bueso
maintainerPATCH 3/5 cxl/mbox: Improve handling of mbox_cmd return codes
In reply to Davidlohr BuesoUpon a completed command the caller is still expected to check the actual return_code register for to ensure it succeed. This adds, per the spec, the potential command return codes. It maps the hardware return code with the kernel's errno style, and by default continues to use -ENXIO (Command completed, but device reported an error).
--- drivers/cxl/core/mbox.c | 2 +- drivers/cxl/cxlmem.h | 90 ++++++++++++++++++++++++++++++++++++++++- drivers/cxl/pci.c | 2 +- 3 files changed, 91 insertions(+), 3 deletions(-)
drivers/cxl/core/mbox.c +1 -1
@@ -171,7 +171,7 @@ int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in,return rc;/* TODO: Map return code to proper kernel style errno */ -if (mbox_cmd.return_code !=CXL_MBOX_SUCCESS ) +if (mbox_cmd.return_code !=CXL_MBOX_CMD_RC_SUCCESS )return -ENXIO ;/*
drivers/cxl/cxlmem.h +89 -1
98 lines not on this page · raw message
drivers/cxl/pci.c +1 -1
@@ -177,7 +177,7 @@ static int __cxl_pci_mbox_send_cmd(struct cxl_dev_state *cxlds, mbox_cmd->return_code =FIELD_GET (CXLDEV_MBOX_STATUS_RET_CODE_MASK , status_reg); -if (mbox_cmd->return_code !=CXL_MBOX_SUCCESS ) { +if (mbox_cmd->return_code !=CXL_MBOX_CMD_RC_SUCCESS ) {dev_dbg (dev,"Mailbox operation had an error\n" );return 0 ;/* completed but caller must check return_code */ }
Signature · 1 line
2.26.2
Re: [PATCH 3/5] cxl/mbox: Improve handling of mbox_cmd return codes
Quoted 2 lines · Davidlohr Bueso
Upon a completed command the caller is still expected to check the actual
return_coderegister for to ensure it succeed. This
Is MB Status register == return_code register here. s/for to/to
Quoted 42 lines
adds, per the spec, the potential command return codes. It maps the hardware return code with the kernel's errno style, and by default continues to use
-ENXIO(Command completed, but device reported an error).Signed-off-by Davidlohr Bueso <[email protected]>--- drivers/cxl/core/mbox.c | 2 +- drivers/cxl/cxlmem.h | 90 ++++++++++++++++++++++++++++++++++++++++- drivers/cxl/pci.c | 2 +- 3 files changed, 91 insertions(+), 3 deletions(-)drivers/cxl/core/mbox.c +1
-1@@ -171,7 +171,7 @@ int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in,return rc;/* TODO: Map return code to proper kernel style errno */ -if (mbox_cmd.return_code !=CXL_MBOX_SUCCESS ) +if (mbox_cmd.return_code !=CXL_MBOX_CMD_RC_SUCCESS )return -ENXIO ;/* drivers/cxl/cxlmem.h +6
-1@@ -85,9 +85,97 @@ struct cxl_mbox_cmd {size_t size_in;size_t size_out;u16 return_code; -#define CXL_MBOX_SUCCESS 0 }; +/* +* Per CXL 2.0 Section 8.2.8.4.5.1 +*/ +enum { +CXL_MBOX_CMD_RC_SUCCESS =0 , +CXL_MBOX_CMD_RC_BACKGROUND ,
drivers/cxl/cxlmem.h struct cxl_mbox_cmd { · line 105
Quoted 10 lines
drivers/cxl/cxlmem.h +10
-0+CXL_MBOX_CMD_RC_INVINPUT , +CXL_MBOX_CMD_RC_UNSUPPORTED , +CXL_MBOX_CMD_RC_INTERNAL , +CXL_MBOX_CMD_RC_RETRY , +CXL_MBOX_CMD_RC_BUSY , +CXL_MBOX_CMD_RC_MEDIADISABLED , +CXL_MBOX_CMD_RC_FWINPROGRESS , +CXL_MBOX_CMD_RC_FWOOO , +CXL_MBOX_CMD_RC_FWAUTH , + CXL_MBOX_CMD_RC_INVSLOT,
Should we keep the FW prefix for INVSLOT, since this is still related to a FW operation?
drivers/cxl/cxlmem.h struct cxl_mbox_cmd { · line 110
Quoted 5 lines
drivers/cxl/cxlmem.h +5
-0+CXL_MBOX_CMD_RC_FWROLLBACK , +CXL_MBOX_CMD_RC_FWRESET , +CXL_MBOX_CMD_RC_INVHANDLE , +CXL_MBOX_CMD_RC_INVPA , + CXL_MBOX_CMD_RC_INJPOISON,
We should indicate that we hit the poison limit, something like the following POISONLMT
drivers/cxl/cxlmem.h struct cxl_mbox_cmd { · line 116
Quoted 6 lines
drivers/cxl/cxlmem.h +6
-0+CXL_MBOX_CMD_RC_MEDIAFAILURE , +CXL_MBOX_CMD_RC_ABORT , +CXL_MBOX_CMD_RC_INVSECURITY , +CXL_MBOX_CMD_RC_PASSPHRASE , +CXL_MBOX_CMD_RC_MBOXUNSUPPORTED , + CXL_MBOX_CMD_RC_INVPAYLOAD,
Might as well be more descriptive here PAYLOADLEN, looking at this more now maybe we drop all of the INV prefixes since most of the return codes are error codes.
Quoted 31 lines
drivers/cxl/cxlmem.h +31
-0+}; + +#define CXL_CMD_RC (_value, errno, str) \ + [CXL_MBOX_CMD_RC_##_value] = { \ + .err = errno, \ + .desc = str, \ + } + +/* +* mbox cmd hardware return_code handling +*/ +struct cxl_mbox_cmd_rc { +int err;/* proper kernel style errno */ +const char *desc; +}; + +static const struct cxl_mbox_cmd_rc cxl_mbox_cmd_rctable[] = { +CXL_CMD_RC (SUCCESS ,0 ,NULL ), +CXL_CMD_RC (BACKGROUND ,ENXIO ,"background cmd started successfully" ), +/* errors here on out */ +CXL_CMD_RC (INVINPUT ,ENXIO ,"cmd input was invalid" ), +CXL_CMD_RC (UNSUPPORTED ,ENXIO ,"cmd is not supported" ), +CXL_CMD_RC (INTERNAL ,ENXIO ,"internal device error" ), +CXL_CMD_RC (RETRY ,ENXIO ,"temporary error, retry once" ), +CXL_CMD_RC (BUSY ,ENXIO ,"ongoing background operation" ), +CXL_CMD_RC (MEDIADISABLED ,ENXIO ,"media access is disabled" ), +CXL_CMD_RC (FWINPROGRESS ,ENXIO , +"only one FW package can be transferred at a time" ), +CXL_CMD_RC (FWOOO ,ENXIO , +"FW package content was transferred out of order" ), +CXL_CMD_RC (FWAUTH ,ENXIO ,"FW package authentication failed" ),
drivers/cxl/cxlmem.h struct cxl_mbox_cmd { · line 157
Quoted 10 lines
drivers/cxl/cxlmem.h +10
-0+CXL_CMD_RC (INVSLOT ,ENXIO , +"FW slot specified is not supported for requested op" ), +CXL_CMD_RC (FWROLLBACK ,ENXIO , +"rolled back to the previous active FW" ), +CXL_CMD_RC (FWRESET ,ENXIO ,"FW failed to activate, needs cold reset" ), +CXL_CMD_RC (INVHANDLE ,ENXIO , +"one or more Event Record Handles were invalid" ), +CXL_CMD_RC (INVPA ,ENXIO ,"physical address specified is invalid" ), +CXL_CMD_RC (INJPOISON ,ENXIO , + "allowed poison injection has been reached"),
injection limit
drivers/cxl/cxlmem.h struct cxl_mbox_cmd { · line 163
Quoted 6 lines
drivers/cxl/cxlmem.h +6
-0+CXL_CMD_RC (MEDIAFAILURE ,ENXIO ,"permanent issue with the media" ), +CXL_CMD_RC (ABORT ,ENXIO ,"background cmd was aborted by device" ), +CXL_CMD_RC (INVSECURITY ,ENXIO , +"not valid in the current security state" ), +CXL_CMD_RC (PASSPHRASE ,ENXIO , + "passphrase doesn't match current passphrase"),
current set passphrase
Quoted 35 lines
drivers/cxl/cxlmem.h +15
-0+CXL_CMD_RC (MBOXUNSUPPORTED ,ENXIO , +"not supported on the mailbox it was issued on" ), + CXL_CMD_RC(INVPAYLOAD, ENXIO, "invalid payload length"), +}; + +static inline const char *cxl_mbox_cmd_rc2str (struct cxl_mbox_cmd *mbox_cmd) +{ +return cxl_mbox_cmd_rctable[mbox_cmd->return_code].desc; +} + +static inline int cxl_mbox_cmd_rc2errno (struct cxl_mbox_cmd *mbox_cmd) +{ +return cxl_mbox_cmd_rctable[mbox_cmd->return_code].err; +} +/* * CXL 2.0 - Memory capacity multiplier * See Section 8.2.9.5 drivers/cxl/pci.c +1
-1@@ -177,7 +177,7 @@ static int __cxl_pci_mbox_send_cmd(struct cxl_dev_state *cxlds, mbox_cmd->return_code =FIELD_GET (CXLDEV_MBOX_STATUS_RET_CODE_MASK , status_reg); -if (mbox_cmd->return_code !=CXL_MBOX_SUCCESS ) { +if (mbox_cmd->return_code !=CXL_MBOX_CMD_RC_SUCCESS ) {dev_dbg (dev,"Mailbox operation had an error\n" );return 0 ;/* completed but caller must check return_code */ }Signature · 2 lines
-- 2.26.2
Other than the minor nits.
Looks good.
Reviewed by: Adam Manzanares <[email protected]>
Davidlohr Bueso
maintainerRe: [PATCH 3/5] cxl/mbox: Improve handling of mbox_cmd return codes
Quoted 70 lines · Adam Manzanares
Quoted 2 lines · Davidlohr Bueso
Upon a completed command the caller is still expected to check the actual
return_coderegister for to ensure it succeed. ThisIs MB Status register ==
return_coderegister here. s/for to/toQuoted 42 lines
adds, per the spec, the potential command return codes. It maps the hardware return code with the kernel's errno style, and by default continues to use
-ENXIO(Command completed, but device reported an error).Signed-off-by Davidlohr Bueso <[email protected]>--- drivers/cxl/core/mbox.c | 2 +- drivers/cxl/cxlmem.h | 90 ++++++++++++++++++++++++++++++++++++++++- drivers/cxl/pci.c | 2 +- 3 files changed, 91 insertions(+), 3 deletions(-)drivers/cxl/core/mbox.c +1
-1@@ -171,7 +171,7 @@ int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in,return rc;/* TODO: Map return code to proper kernel style errno */ -if (mbox_cmd.return_code !=CXL_MBOX_SUCCESS ) +if (mbox_cmd.return_code !=CXL_MBOX_CMD_RC_SUCCESS )return -ENXIO ;/* drivers/cxl/cxlmem.h +6
-1@@ -85,9 +85,97 @@ struct cxl_mbox_cmd {size_t size_in;size_t size_out;u16 return_code; -#define CXL_MBOX_SUCCESS 0 }; +/* +* Per CXL 2.0 Section 8.2.8.4.5.1 +*/ +enum { +CXL_MBOX_CMD_RC_SUCCESS =0 , +CXL_MBOX_CMD_RC_BACKGROUND ,drivers/cxl/cxlmem.h struct cxl_mbox_cmd { · line 105
Quoted 10 lines
drivers/cxl/cxlmem.h +10
-0+CXL_MBOX_CMD_RC_INVINPUT , +CXL_MBOX_CMD_RC_UNSUPPORTED , +CXL_MBOX_CMD_RC_INTERNAL , +CXL_MBOX_CMD_RC_RETRY , +CXL_MBOX_CMD_RC_BUSY , +CXL_MBOX_CMD_RC_MEDIADISABLED , +CXL_MBOX_CMD_RC_FWINPROGRESS , +CXL_MBOX_CMD_RC_FWOOO , +CXL_MBOX_CMD_RC_FWAUTH , + CXL_MBOX_CMD_RC_INVSLOT,Should we keep the FW prefix for INVSLOT, since this is still related to a FW operation?
drivers/cxl/cxlmem.h struct cxl_mbox_cmd { · line 110
Quoted 5 lines
drivers/cxl/cxlmem.h +5
-0+CXL_MBOX_CMD_RC_FWROLLBACK , +CXL_MBOX_CMD_RC_FWRESET , +CXL_MBOX_CMD_RC_INVHANDLE , +CXL_MBOX_CMD_RC_INVPA , + CXL_MBOX_CMD_RC_INJPOISON,We should indicate that we hit the poison limit, something like the following POISONLMT
Quoted 10 lines
drivers/cxl/cxlmem.h struct cxl_mbox_cmd { · line 116
Quoted 6 lines
drivers/cxl/cxlmem.h +6
-0+CXL_MBOX_CMD_RC_MEDIAFAILURE , +CXL_MBOX_CMD_RC_ABORT , +CXL_MBOX_CMD_RC_INVSECURITY , +CXL_MBOX_CMD_RC_PASSPHRASE , +CXL_MBOX_CMD_RC_MBOXUNSUPPORTED , + CXL_MBOX_CMD_RC_INVPAYLOAD,Might as well be more descriptive here PAYLOADLEN, looking at this more now maybe we drop all of the INV prefixes since most of the return codes are error codes.
Yes to all the above.
Quoted 54 lines
Quoted 32 lines
drivers/cxl/cxlmem.h +31
-0+}; + +#define CXL_CMD_RC (_value, errno, str) \ + [CXL_MBOX_CMD_RC_##_value] = { \ + .err = errno, \ + .desc = str, \ + } + +/* +* mbox cmd hardware return_code handling +*/ +struct cxl_mbox_cmd_rc { +int err;/* proper kernel style errno */ +const char *desc; +}; + +static const struct cxl_mbox_cmd_rc cxl_mbox_cmd_rctable[] = { +CXL_CMD_RC (SUCCESS ,0 ,NULL ), +CXL_CMD_RC (BACKGROUND ,ENXIO ,"background cmd started successfully" ), +/* errors here on out */ +CXL_CMD_RC (INVINPUT ,ENXIO ,"cmd input was invalid" ), +CXL_CMD_RC (UNSUPPORTED ,ENXIO ,"cmd is not supported" ), +CXL_CMD_RC (INTERNAL ,ENXIO ,"internal device error" ), +CXL_CMD_RC (RETRY ,ENXIO ,"temporary error, retry once" ), +CXL_CMD_RC (BUSY ,ENXIO ,"ongoing background operation" ), +CXL_CMD_RC (MEDIADISABLED ,ENXIO ,"media access is disabled" ), +CXL_CMD_RC (FWINPROGRESS ,ENXIO , +"only one FW package can be transferred at a time" ), +CXL_CMD_RC (FWOOO ,ENXIO , +"FW package content was transferred out of order" ), +CXL_CMD_RC (FWAUTH ,ENXIO ,"FW package authentication failed" ),drivers/cxl/cxlmem.h struct cxl_mbox_cmd { · line 157
Quoted 10 lines
drivers/cxl/cxlmem.h +10
-0+CXL_CMD_RC (INVSLOT ,ENXIO , +"FW slot specified is not supported for requested op" ), +CXL_CMD_RC (FWROLLBACK ,ENXIO , +"rolled back to the previous active FW" ), +CXL_CMD_RC (FWRESET ,ENXIO ,"FW failed to activate, needs cold reset" ), +CXL_CMD_RC (INVHANDLE ,ENXIO , +"one or more Event Record Handles were invalid" ), +CXL_CMD_RC (INVPA ,ENXIO ,"physical address specified is invalid" ), +CXL_CMD_RC (INJPOISON ,ENXIO , + "allowed poison injection has been reached"),injection limit
drivers/cxl/cxlmem.h struct cxl_mbox_cmd { · line 163
Quoted 6 lines
drivers/cxl/cxlmem.h +6
-0+CXL_CMD_RC (MEDIAFAILURE ,ENXIO ,"permanent issue with the media" ), +CXL_CMD_RC (ABORT ,ENXIO ,"background cmd was aborted by device" ), +CXL_CMD_RC (INVSECURITY ,ENXIO , +"not valid in the current security state" ), +CXL_CMD_RC (PASSPHRASE ,ENXIO , + "passphrase doesn't match current passphrase"),current set passphrase
Quoted 7 lines
drivers/cxl/cxlmem.h struct cxl_mbox_cmd { · line 167
Quoted 4 lines
drivers/cxl/cxlmem.h +4
-0+CXL_CMD_RC (MBOXUNSUPPORTED ,ENXIO , +"not supported on the mailbox it was issued on" ), + CXL_CMD_RC(INVPAYLOAD, ENXIO, "invalid payload length"), +};Other than the minor nits.
Looks good.
Thanks for reviewing. I'll add the changes in a v2 but will wait a bit for any further feedback.
Re: [PATCH 3/5] cxl/mbox: Improve handling of mbox_cmd return codes
In reply to Davidlohr BuesoQuoted 107 lines · Davidlohr Bueso
Upon a completed command the caller is still expected to check the actual
return_coderegister for to ensure it succeed. This adds, per the spec, the potential command return codes. It maps the hardware return code with the kernel's errno style, and by default continues to use-ENXIO(Command completed, but device reported an error).Signed-off-by Davidlohr Bueso <[email protected]>--- drivers/cxl/core/mbox.c | 2 +- drivers/cxl/cxlmem.h | 90 ++++++++++++++++++++++++++++++++++++++++- drivers/cxl/pci.c | 2 +- 3 files changed, 91 insertions(+), 3 deletions(-)drivers/cxl/core/mbox.c +1
-1@@ -171,7 +171,7 @@ int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in,return rc;/* TODO: Map return code to proper kernel style errno */ -if (mbox_cmd.return_code !=CXL_MBOX_SUCCESS ) +if (mbox_cmd.return_code !=CXL_MBOX_CMD_RC_SUCCESS )return -ENXIO ;/* drivers/cxl/cxlmem.h +68
-1@@ -85,9 +85,97 @@ struct cxl_mbox_cmd {size_t size_in;size_t size_out;u16 return_code; -#define CXL_MBOX_SUCCESS 0 }; +/* +* Per CXL 2.0 Section 8.2.8.4.5.1 +*/ +enum { +CXL_MBOX_CMD_RC_SUCCESS =0 , +CXL_MBOX_CMD_RC_BACKGROUND , +CXL_MBOX_CMD_RC_INVINPUT , +CXL_MBOX_CMD_RC_UNSUPPORTED , +CXL_MBOX_CMD_RC_INTERNAL , +CXL_MBOX_CMD_RC_RETRY , +CXL_MBOX_CMD_RC_BUSY , +CXL_MBOX_CMD_RC_MEDIADISABLED , +CXL_MBOX_CMD_RC_FWINPROGRESS , +CXL_MBOX_CMD_RC_FWOOO , +CXL_MBOX_CMD_RC_FWAUTH , +CXL_MBOX_CMD_RC_INVSLOT , +CXL_MBOX_CMD_RC_FWROLLBACK , +CXL_MBOX_CMD_RC_FWRESET , +CXL_MBOX_CMD_RC_INVHANDLE , +CXL_MBOX_CMD_RC_INVPA , +CXL_MBOX_CMD_RC_INJPOISON , +CXL_MBOX_CMD_RC_MEDIAFAILURE , +CXL_MBOX_CMD_RC_ABORT , +CXL_MBOX_CMD_RC_INVSECURITY , +CXL_MBOX_CMD_RC_PASSPHRASE , +CXL_MBOX_CMD_RC_MBOXUNSUPPORTED , +CXL_MBOX_CMD_RC_INVPAYLOAD , +}; + +#define CXL_CMD_RC (_value, errno, str) \ + [CXL_MBOX_CMD_RC_##_value] = { \ + .err = errno, \ + .desc = str, \ + } + +/* +* mbox cmd hardware return_code handling +*/ +struct cxl_mbox_cmd_rc { +int err;/* proper kernel style errno */ +const char *desc; +}; + +static const struct cxl_mbox_cmd_rc cxl_mbox_cmd_rctable[] = { +CXL_CMD_RC (SUCCESS ,0 ,NULL ), +CXL_CMD_RC (BACKGROUND ,ENXIO ,"background cmd started successfully" ), +/* errors here on out */ +CXL_CMD_RC (INVINPUT ,ENXIO ,"cmd input was invalid" ), +CXL_CMD_RC (UNSUPPORTED ,ENXIO ,"cmd is not supported" ), +CXL_CMD_RC (INTERNAL ,ENXIO ,"internal device error" ), +CXL_CMD_RC (RETRY ,ENXIO ,"temporary error, retry once" ), +CXL_CMD_RC (BUSY ,ENXIO ,"ongoing background operation" ), +CXL_CMD_RC (MEDIADISABLED ,ENXIO ,"media access is disabled" ), +CXL_CMD_RC (FWINPROGRESS ,ENXIO , +"only one FW package can be transferred at a time" ), +CXL_CMD_RC (FWOOO ,ENXIO , +"FW package content was transferred out of order" ), +CXL_CMD_RC (FWAUTH ,ENXIO ,"FW package authentication failed" ), +CXL_CMD_RC (INVSLOT ,ENXIO , +"FW slot specified is not supported for requested op" ), +CXL_CMD_RC (FWROLLBACK ,ENXIO , +"rolled back to the previous active FW" ), +CXL_CMD_RC (FWRESET ,ENXIO ,"FW failed to activate, needs cold reset" ), +CXL_CMD_RC (INVHANDLE ,ENXIO , +"one or more Event Record Handles were invalid" ), +CXL_CMD_RC (INVPA ,ENXIO ,"physical address specified is invalid" ), +CXL_CMD_RC (INJPOISON ,ENXIO , +"allowed poison injection has been reached" ),
drivers/cxl/cxlmem.h struct cxl_mbox_cmd { · line 167
Quoted 10 lines
drivers/cxl/cxlmem.h +10
-0+CXL_CMD_RC (MEDIAFAILURE ,ENXIO ,"permanent issue with the media" ), +CXL_CMD_RC (ABORT ,ENXIO ,"background cmd was aborted by device" ), +CXL_CMD_RC (INVSECURITY ,ENXIO , +"not valid in the current security state" ), +CXL_CMD_RC (PASSPHRASE ,ENXIO , +"passphrase doesn't match current passphrase" ), +CXL_CMD_RC (MBOXUNSUPPORTED ,ENXIO , +"not supported on the mailbox it was issued on" ), + CXL_CMD_RC(INVPAYLOAD, ENXIO, "invalid payload length"), +};
You can define the table and the enums together with Steven's trick [1]
#define CMD_CMD_RC_TABLE \
C(SUCCESS, 0, NULL), \
C(BACKGROUND, -ENXIO, "background cmd started successfully"), \
...
C(PAYLOADLEN, -ENXIO, "invalid payload length")#undef C
#define C(a, b, c) CXL_CMD_RC_##a
enum { CMD_CMD_RC_TABLE };
#undef C
#define C(a, b, c) { b, c }
static struct {
int err;
const char *desc;
} cxl_mbox_cmd_rctable[] = { OPS };#undef C
Ideally you could align the columns so it's easy to read which hardware code relates to which errno.
[1]: See "cpp tricks and treats": https://lwn.net/Articles/383362/
Davidlohr Bueso
maintainerPATCH 4/5 cxl/mbox: Use new return_code handling
In reply to Davidlohr BuesoUse the global cxl_mbox_cmd_rc table to improve debug messaging in __cxl_pci_mbox_send_cmd() and allow cxl_mbox_send_cmd() to map to proper kernel style errno codes - this patch continues to use -ENXIO only so no change in semantics.
--- drivers/cxl/core/mbox.c | 7 ++++--- drivers/cxl/pci.c | 3 ++- 2 files changed, 6 insertions(+), 4 deletions(-)
drivers/cxl/core/mbox.c +4 -3
@@ -170,9 +170,10 @@ int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in,if (rc)return rc; -/* TODO: Map return code to proper kernel style errno */ -if (mbox_cmd.return_code !=CXL_MBOX_CMD_RC_SUCCESS ) -return -ENXIO ; +if (mbox_cmd.return_code !=CXL_MBOX_CMD_RC_SUCCESS ) { +int err =cxl_mbox_cmd_rc2errno (&mbox_cmd); +return -err; + }/* * Variable sized commands can't be validated and so it's up to the
drivers/cxl/pci.c +2 -1
@@ -178,7 +178,8 @@ static int __cxl_pci_mbox_send_cmd(struct cxl_dev_state *cxlds,FIELD_GET (CXLDEV_MBOX_STATUS_RET_CODE_MASK , status_reg);if (mbox_cmd->return_code !=CXL_MBOX_CMD_RC_SUCCESS ) { -dev_dbg (dev,"Mailbox operation had an error\n" ); +dev_dbg (dev,"Mailbox operation had an error: %s\n" , +cxl_mbox_cmd_rc2str (mbox_cmd));return 0 ;/* completed but caller must check return_code */ }
Signature · 1 line
2.26.2
Re: [PATCH 4/5] cxl/mbox: Use new return_code handling
Quoted 47 lines · Davidlohr Bueso
Use the global
cxl_mbox_cmd_rctable to improve debug messaging in__cxl_pci_mbox_send_cmd()and allowcxl_mbox_send_cmd()to map to proper kernel style errno codes - this patch continues to use-ENXIOonly so no change in semantics.Signed-off-by Davidlohr Bueso <[email protected]>--- drivers/cxl/core/mbox.c | 7 ++++--- drivers/cxl/pci.c | 3 ++- 2 files changed, 6 insertions(+), 4 deletions(-)drivers/cxl/core/mbox.c +4
-3@@ -170,9 +170,10 @@ int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in,if (rc)return rc; -/* TODO: Map return code to proper kernel style errno */ -if (mbox_cmd.return_code !=CXL_MBOX_CMD_RC_SUCCESS ) -return -ENXIO ; +if (mbox_cmd.return_code !=CXL_MBOX_CMD_RC_SUCCESS ) { +int err =cxl_mbox_cmd_rc2errno (&mbox_cmd); +return -err; + }/* * Variable sized commands can't be validated and so it's up to the drivers/cxl/pci.c +2
-1@@ -178,7 +178,8 @@ static int __cxl_pci_mbox_send_cmd(struct cxl_dev_state *cxlds,FIELD_GET (CXLDEV_MBOX_STATUS_RET_CODE_MASK , status_reg);if (mbox_cmd->return_code !=CXL_MBOX_CMD_RC_SUCCESS ) { -dev_dbg (dev,"Mailbox operation had an error\n" ); +dev_dbg (dev,"Mailbox operation had an error: %s\n" , +cxl_mbox_cmd_rc2str (mbox_cmd));return 0 ;/* completed but caller must check return_code */ }Signature · 2 lines
-- 2.26.2
Looks good.
Reviewed by: Adam Manzanares <[email protected]>
Re: [PATCH 4/5] cxl/mbox: Use new return_code handling
In reply to Davidlohr BuesoQuoted 17 lines · Davidlohr Bueso
Use the global
cxl_mbox_cmd_rctable to improve debug messaging in__cxl_pci_mbox_send_cmd()and allowcxl_mbox_send_cmd()to map to proper kernel style errno codes - this patch continues to use-ENXIOonly so no change in semantics.Signed-off-by Davidlohr Bueso <[email protected]>--- drivers/cxl/core/mbox.c | 7 ++++--- drivers/cxl/pci.c | 3 ++- 2 files changed, 6 insertions(+), 4 deletions(-)drivers/cxl/core/mbox.c +0
-0
drivers/cxl/core/mbox.c int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in, · line 175
Quoted 9 lines
drivers/cxl/core/mbox.c +3
-3@@ -170,9 +170,10 @@ int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in,if (rc)return rc; -/* TODO: Map return code to proper kernel style errno */ -if (mbox_cmd.return_code !=CXL_MBOX_CMD_RC_SUCCESS ) -return -ENXIO ; +if (mbox_cmd.return_code !=CXL_MBOX_CMD_RC_SUCCESS ) { +int err =cxl_mbox_cmd_rc2errno (&mbox_cmd); +return -err;
I don't like the need for a local variable and adding the negation. Just store the negative value directly and call the helper cxl_mbox_cmd_rc2err() so that this can just do "return cxl_mbox_cmd_rc2err(&mbox_cmd);".
Quoted 9 lines
drivers/cxl/core/mbox.c +1
-0+ }/* * Variable sized commands can't be validated and so it's up to the drivers/cxl/pci.c +0
-0
drivers/cxl/pci.c static int __cxl_pci_mbox_send_cmd(struct cxl_dev_state *cxlds, · line 182
Quoted 6 lines
drivers/cxl/pci.c +2
-1@@ -178,7 +178,8 @@ static int __cxl_pci_mbox_send_cmd(struct cxl_dev_state *cxlds,FIELD_GET (CXLDEV_MBOX_STATUS_RET_CODE_MASK , status_reg);if (mbox_cmd->return_code !=CXL_MBOX_CMD_RC_SUCCESS ) { -dev_dbg (dev,"Mailbox operation had an error\n" ); +dev_dbg (dev,"Mailbox operation had an error: %s\n" , +cxl_mbox_cmd_rc2str (mbox_cmd));
Looks good.
Davidlohr Bueso
maintainerPATCH 5/5 cxl/mbox: Retry sending mbox command upon RETRY return_code
In reply to Davidlohr BuesoAs mentioned in CXL 2.0 specs: the hardware can set a Retry Required if the the command was not completed due to a temporary error. An optional single retry may resolve the issue. This is the only retry case as in general they are not recommended for commands that return an error.
--- drivers/cxl/core/mbox.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-)
drivers/cxl/core/mbox.c +14 -9
@@ -160,20 +160,25 @@ int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in, .size_out = out_size, .payload_out = out, }; -int rc; +int rc, retry =1 ;if (out_size > cxlds->payload_size)return -E2BIG ; -/* acquire and releases the mbox_mutex */ - rc = cxlds->mbox_send (cxlds, &mbox_cmd); -if (rc) -return rc; +do { +/* acquire and releases the mbox_mutex */ + rc = cxlds->mbox_send (cxlds, &mbox_cmd); +if (rc) +return rc; -if (mbox_cmd.return_code !=CXL_MBOX_CMD_RC_SUCCESS ) { -int err =cxl_mbox_cmd_rc2errno (&mbox_cmd); -return -err; - } +if (mbox_cmd.return_code ==CXL_MBOX_CMD_RC_SUCCESS ) +break ; + +if (mbox_cmd.return_code !=CXL_MBOX_CMD_RC_RETRY || !retry) { +int err =cxl_mbox_cmd_rc2errno (&mbox_cmd); +return -err; + } + }while (retry--);/* * Variable sized commands can't be validated and so it's up to the
Signature · 1 line
2.26.2
Re: [PATCH 5/5] cxl/mbox: Retry sending mbox command upon RETRY return_code
drivers/cxl/core/mbox.c int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in,
Quoted 16 lines · Davidlohr Bueso
As mentioned in CXL 2.0 specs: the hardware can set a Retry Required if the the command was not completed due to a temporary error. An optional single retry may resolve the issue. This is the only retry case as in general they are not recommended for commands that return an error.
Signed-off-by Davidlohr Bueso <[email protected]>--- drivers/cxl/core/mbox.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-)drivers/cxl/core/mbox.c +0
-0@@ -160,20 +160,25 @@ int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in,
Can you update the documentation cxl_mbox_send_cmd that the function now retries commands if the RC indicates it should be done?
Quoted 38 lines
drivers/cxl/core/mbox.c +14
-9.size_out = out_size, .payload_out = out, }; -int rc; +int rc, retry =1 ;if (out_size > cxlds->payload_size)return -E2BIG ; -/* acquire and releases the mbox_mutex */ - rc = cxlds->mbox_send (cxlds, &mbox_cmd); -if (rc) -return rc; +do { +/* acquire and releases the mbox_mutex */ + rc = cxlds->mbox_send (cxlds, &mbox_cmd); +if (rc) +return rc; -if (mbox_cmd.return_code !=CXL_MBOX_CMD_RC_SUCCESS ) { -int err =cxl_mbox_cmd_rc2errno (&mbox_cmd); -return -err; - } +if (mbox_cmd.return_code ==CXL_MBOX_CMD_RC_SUCCESS ) +break ; + +if (mbox_cmd.return_code !=CXL_MBOX_CMD_RC_RETRY || !retry) { +int err =cxl_mbox_cmd_rc2errno (&mbox_cmd); +return -err; + } + }while (retry--);/* * Variable sized commands can't be validated and so it's up to the Signature · 2 lines
-- 2.26.2
Other than the doc change for the cxl_mbox_send_cmd function. LGTM
Reviewed by: Adam Manzanares <[email protected]>
Re: [PATCH 5/5] cxl/mbox: Retry sending mbox command upon RETRY return_code
In reply to Davidlohr BuesoQuoted 6 lines · Davidlohr Bueso
As mentioned in CXL 2.0 specs: the hardware can set a Retry Required if the the command was not completed due to a temporary error. An optional single retry may resolve the issue. This is the only retry case as in general they are not recommended for commands that return an error.
So I disagree that the driver should be deploying this behaviour by default. This was also tried for the ACPI NVDIMM commands and it was helpful for Linux to not play along and get the firmware implementers to get their act together and detail when a retry is appropriate. So, this policy is up to the submitter. If that agent has information that the kernel does not have about a retry being effective, it is free to attempt the retry after the failure. Otherwise, the spec is overly hopeful in expecting an implementation to attempt a speculative / optional retry. Hopefully, a default Linux policy to fail commands that end in a retry encourages fixes to devices / firmwares that want to use optional retry as a crutch... or it encourages the discipline to come back to the specification working group with explicit per-command updates to define the scenarios when a retry is mandatory and not optional.