Skip to contentSkip to conversation

CXL UE RAS Multiple Header Logging support

RFC PATCH In review Jonathan Cameron 2 participants 4 messages Jan 13, 2023 – Jan 17, 2023 CXL

lore
Share & tools
lore
#0 lore raw

cover CXL UE RAS Multiple Header Logging support

CXL UE RAS Error reporting allows an EP to report the capability of recording Multiple Header Logs for uncorrectable errors. Unlike equivalent feature in PCIe, there is no enable control for this feature, so a supporting device may be expecting a more complex software flow than that necessary for devices that do not support this feature. Documentation of this feature is sparse, with assumption it works the same as PCIe.

There are hardware implementation choices allowed in the equivalent PCIe r6.0 base spec section (6.4.2.4) that could be safely used with the existing code, even with Multiple Header Recording support but there are others that cannot.

The issue is what happens when the EP is doing Multiple Header Recording but then the software writes 1 to clear more than one status bit at the time (PCIe spec warns against doing this
- but it is what the current kernel code will do):
Option 1)
It does the nice thing and clears all matching errors. Note this is a bit strange for the case where the device supports logging multiple instances of a given error - so the two can't be combined cleanly. With that feature I can't see how anyone could implement hardware that coped cleanly with the wrong software flow.
Option 2)
It clears only the first error bit leaving a bunch of error bits set (note that if it has recorded multiple errors of same type it might not even do that). These are sticky across resets, so you will probably end up coming back up and immediately seeing an error.

So whilst you can design an EP to safe against non MH recording aware software, it isn't generally the case. As we don't have an explicit enable on CXL we have to handle anything reporting the capability in a MH safe fashion.

This feature was developed against emulation in QEMU. The relevant patches have not yet been posted but can be found on
https://gitlab.com/jic23/qemu/-/commits/cxl-2023-01-11
along with description of how to inject errors in the patch descriptions. I'll post them for review for QEMU inclusion shortly.

RFC simply because the lack of specification detail means I am less sure on this code than I would normally be. Unfortunately it could be argued that the first patch is a fix for the current upstream CXL RAS support. If we want a simpler fix one option would be to just fail to enable RAS support if Multiple Header recording capability bit is set. Or we decide that it doesn't matter for now and add support for this feature via the normal merge cycle.

Second patch is just there to make this easier to test as no additional software is needed to print the header log.

Base is rather messy due to a clash between multiple cxl tree branches.
cxl/fixes with the trace move on cxl/next cherry picked on top as it moves the code that was fixed.

Jonathan Cameron (2):
cxl: RAS: Multiple header recording support
cxl: Add tprintk support for header log hex dump

 drivers/cxl/core/pci.c   | 17 ++++++++++++-----
 drivers/cxl/core/trace.h |  7 +++++--
 drivers/cxl/cxl.h        |  1 +
 3 files changed, 18 insertions(+), 7 deletions(-)
Signature · 1 line
2.37.2
#1 lore raw

RFC PATCH 1/2 cxl: RAS: Multiple header recording support

Similar to PCIe, CXL devices may support logging multiple headers corresponding to multiple errors as reported via the CXL RAS capability.

Unlike PCIe, in CXL there is no Multiple Header Recording Enable bit and the CXL r3.0 specification is sparse on details. As such, the kernel should allow for any reasonable interpretation including endpoints for which the capability bit is set that behave as per the PCIe equivalent definitions (with assumption that the missing 'enable bit' is set). Note that behaving as if Multiple Headers are being logged is also valid behavior when they are not so this approach should be safe with all sensible specification interpretations.

By repeatedly attempting to clear a single bit corresponding to the reported First Error (may need multiple goes if multiple records of same type are tracked by the hardware) the additional header logs may be obtained.

Note that each trace record only records the FE in the status. We could record them all as done without Multi header recording capability but that seemed less intuitive to me.

---
 drivers/cxl/core/pci.c | 17 ++++++++++++-----
 drivers/cxl/cxl.h      |  1 +
 2 files changed, 13 insertions(+), 5 deletions(-)
drivers/cxl/core/pci.c +12 -5
index 184ead6a2796..6fd311e313c6 100644
@@ -673,10 +673,13 @@ static bool cxl_report_and_clear(struct cxl_dev_state *cxlds)
 	void __iomem *addr;
 	u32 status;
 	u32 fe;
+	bool mh;
 
 	if (!cxlds->regs.ras)
 		return false;
 
+next_record:
+	mh = false;
 	addr = cxlds->regs.ras + CXL_RAS_UNCORRECTABLE_STATUS_OFFSET;
 	status = readl(addr);
 	if (!(status & CXL_RAS_UNCORRECTABLE_STATUS_MASK))
@@ -684,11 +687,13 @@ static bool cxl_report_and_clear(struct cxl_dev_state *cxlds)
 
 	/* If multiple errors, log header points to first error from ctrl reg */
 	if (hweight32(status) > 1) {
-		void __iomem *rcc_addr =
-			cxlds->regs.ras + CXL_RAS_CAP_CONTROL_OFFSET;
-
-		fe = BIT(FIELD_GET(CXL_RAS_CAP_CONTROL_FE_MASK,
-				   readl(rcc_addr)));
+		u32 capctrl = readl(cxlds->regs.ras + CXL_RAS_CAP_CONTROL_OFFSET);
+		fe = BIT(FIELD_GET(CXL_RAS_CAP_CONTROL_FE_MASK, capctrl));
+		if (FIELD_GET(CXL_RAS_CAP_CONTROL_MH_REC_CAP, capctrl)) {
+			mh = true;
+			/* Report and clear only first error */
+			status = fe;
+		}
 	} else {
 		fe = status;
 	}
@@ -696,6 +701,8 @@ static bool cxl_report_and_clear(struct cxl_dev_state *cxlds)
 	header_log_copy(cxlds, hl);
 	trace_cxl_aer_uncorrectable_error(dev, status, fe, hl);
 	writel(status & CXL_RAS_UNCORRECTABLE_STATUS_MASK, addr);
+	if (mh)
+		goto next_record;
 
 	return true;
 }
drivers/cxl/cxl.h +1 -0
index aa3af3bb73b2..ee31a99073c2 100644
@@ -138,6 +138,7 @@ static inline int ways_to_eiw(unsigned int ways, u8 *eiw)
 #define   CXL_RAS_CORRECTABLE_MASK_MASK GENMASK(6, 0)
 #define CXL_RAS_CAP_CONTROL_OFFSET 0x14
 #define CXL_RAS_CAP_CONTROL_FE_MASK GENMASK(5, 0)
+#define CXL_RAS_CAP_CONTROL_MH_REC_CAP BIT(9)
 #define CXL_RAS_HEADER_LOG_OFFSET 0x18
 #define CXL_RAS_CAPABILITY_LENGTH 0x58
 #define CXL_HEADERLOG_SIZE SZ_512
Signature · 1 line
2.37.2
#2 lore raw
In reply to Jonathan Cameron

On Fri, 13 Jan 2023 15:40:09 +0000
Jonathan Cameron <[email protected]> wrote:

Missed Dave Jiang off cc so resent
(I thought I'd hit cancel fast enough but apparently not) Sorry for the noise!

Quoted 69 lines

CXL UE RAS Error reporting allows an EP to report the capability of recording Multiple Header Logs for uncorrectable errors. Unlike equivalent feature in PCIe, there is no enable control for this feature, so a supporting device may be expecting a more complex software flow than that necessary for devices that do not support this feature. Documentation of this feature is sparse, with assumption it works the same as PCIe.

There are hardware implementation choices allowed in the equivalent PCIe r6.0 base spec section (6.4.2.4) that could be safely used with the existing code, even with Multiple Header Recording support but there are others that cannot.

The issue is what happens when the EP is doing Multiple Header Recording but then the software writes 1 to clear more than one status bit at the time (PCIe spec warns against doing this
- but it is what the current kernel code will do):
Option 1)
It does the nice thing and clears all matching errors. Note this is a bit strange for the case where the device supports logging multiple instances of a given error - so the two can't be combined cleanly. With that feature I can't see how anyone could implement hardware that coped cleanly with the wrong software flow.
Option 2)
It clears only the first error bit leaving a bunch of error bits set (note that if it has recorded multiple errors of same type it might not even do that). These are sticky across resets, so you will probably end up coming back up and immediately seeing an error.

So whilst you can design an EP to safe against non MH recording aware software, it isn't generally the case. As we don't have an explicit enable on CXL we have to handle anything reporting the capability in a MH safe fashion.

This feature was developed against emulation in QEMU. The relevant patches have not yet been posted but can be found on
https://gitlab.com/jic23/qemu/-/commits/cxl-2023-01-11
along with description of how to inject errors in the patch descriptions. I'll post them for review for QEMU inclusion shortly.

RFC simply because the lack of specification detail means I am less sure on this code than I would normally be. Unfortunately it could be argued that the first patch is a fix for the current upstream CXL RAS support. If we want a simpler fix one option would be to just fail to enable RAS support if Multiple Header recording capability bit is set. Or we decide that it doesn't matter for now and add support for this feature via the normal merge cycle.

Second patch is just there to make this easier to test as no additional software is needed to print the header log.

Base is rather messy due to a clash between multiple cxl tree branches.
cxl/fixes with the trace move on cxl/next cherry picked on top as it moves the code that was fixed.

Jonathan Cameron (2):
cxl: RAS: Multiple header recording support
cxl: Add tprintk support for header log hex dump

 drivers/cxl/core/pci.c   | 17 ++++++++++++-----
 drivers/cxl/core/trace.h |  7 +++++--
 drivers/cxl/cxl.h        |  1 +
 3 files changed, 18 insertions(+), 7 deletions(-)
maintainer
#3 lore raw
Quoted 6 lines · Jonathan Cameron

On Fri, 13 Jan 2023 15:40:09 +0000
Jonathan Cameron <[email protected]> wrote:

Missed Dave Jiang off cc so resent
(I thought I'd hit cancel fast enough but apparently not) Sorry for the noise!

No worries. I see them from the list either way.

Quoted 71 lines
Quoted 70 lines

CXL UE RAS Error reporting allows an EP to report the capability of recording Multiple Header Logs for uncorrectable errors. Unlike equivalent feature in PCIe, there is no enable control for this feature, so a supporting device may be expecting a more complex software flow than that necessary for devices that do not support this feature. Documentation of this feature is sparse, with assumption it works the same as PCIe.

There are hardware implementation choices allowed in the equivalent PCIe r6.0 base spec section (6.4.2.4) that could be safely used with the existing code, even with Multiple Header Recording support but there are others that cannot.

The issue is what happens when the EP is doing Multiple Header Recording but then the software writes 1 to clear more than one status bit at the time (PCIe spec warns against doing this
- but it is what the current kernel code will do):
Option 1)
It does the nice thing and clears all matching errors. Note this is a bit strange for the case where the device supports logging multiple instances of a given error - so the two can't be combined cleanly. With that feature I can't see how anyone could implement hardware that coped cleanly with the wrong software flow.
Option 2)
It clears only the first error bit leaving a bunch of error bits set (note that if it has recorded multiple errors of same type it might not even do that). These are sticky across resets, so you will probably end up coming back up and immediately seeing an error.

So whilst you can design an EP to safe against non MH recording aware software, it isn't generally the case. As we don't have an explicit enable on CXL we have to handle anything reporting the capability in a MH safe fashion.

This feature was developed against emulation in QEMU. The relevant patches have not yet been posted but can be found on
https://gitlab.com/jic23/qemu/-/commits/cxl-2023-01-11
along with description of how to inject errors in the patch descriptions. I'll post them for review for QEMU inclusion shortly.

RFC simply because the lack of specification detail means I am less sure on this code than I would normally be. Unfortunately it could be argued that the first patch is a fix for the current upstream CXL RAS support. If we want a simpler fix one option would be to just fail to enable RAS support if Multiple Header recording capability bit is set. Or we decide that it doesn't matter for now and add support for this feature via the normal merge cycle.

Second patch is just there to make this easier to test as no additional software is needed to print the header log.

Base is rather messy due to a clash between multiple cxl tree branches.
cxl/fixes with the trace move on cxl/next cherry picked on top as it moves the code that was fixed.

Jonathan Cameron (2):
cxl: RAS: Multiple header recording support
cxl: Add tprintk support for header log hex dump

  drivers/cxl/core/pci.c   | 17 ++++++++++++-----
  drivers/cxl/core/trace.h |  7 +++++--
  drivers/cxl/cxl.h        |  1 +
  3 files changed, 18 insertions(+), 7 deletions(-)