EDAC, dmc520:: add DMC520 EDAC driver
Subsystems (2)
PATCH 1/1 EDAC, dmc520:: add DMC520 EDAC driver
From: Rui Zhao <[email protected]>
New driver supports DRAM error detection and correction on DMC520 controller.
Validated on actual hardware: DRAM errors showed up once the DDR core voltage was lowered down by 200+mV using test tool.
--- MAINTAINERS | 6 + drivers/edac/Kconfig | 7 + drivers/edac/Makefile | 1 + drivers/edac/dmc520_edac.c | 495 +++++++++++++++++++++++++++++++++++++ 4 files changed, 509 insertions(+) create mode 100644 drivers/edac/dmc520_edac.c
MAINTAINERS +6 -0
@@ -5429,6 +5429,12 @@ F: Documentation/driver-api/edac.rst F: drivers/edac/ F: include/linux/edac.h +EDAC-DMC520 +M: Rui Zhao <[email protected]> +L: [email protected] +S: Supported +F: drivers/edac/dmc520_edac.c + EDAC-E752X M: Mark Gross <[email protected]> L: [email protected]
drivers/edac/Kconfig +7 -0
@@ -475,4 +475,11 @@ config EDAC_QCOM For debugging issues having to do with stability and overall system health, you should probably say 'Y' here. +config EDAC_DMC520 + tristate "ARM DMC-520 ECC" + depends on ARM64 + help + Support for error detection and correction on + SoCs with the ARM DMC-520 DRAM controller. + endif # EDAC
drivers/edac/Makefile +1 -0
@@ -78,3 +78,4 @@ obj-$(CONFIG_EDAC_SYNOPSYS) += synopsys_edac.o obj-$(CONFIG_EDAC_XGENE) += xgene_edac.o obj-$(CONFIG_EDAC_TI) += ti_edac.o obj-$(CONFIG_EDAC_QCOM) += qcom_edac.o +obj-$(CONFIG_EDAC_DMC520) += dmc520_edac.o
drivers/edac/dmc520_edac.c new file +495 -0
495 lines not on this page · raw message
Borislav Petkov
maintainerQuoted 4 lines · Sasha Levin
From: Rui Zhao <[email protected]>
New driver supports DRAM error detection and correction on DMC520 controller.
That's this thing, right?
https://developer.arm.com/products/system-ip/memory-controllers/corelink-dmc-520
Quoted 1 line
Validated on actual hardware:
Which is what exactly?
This looks like a driver for the memory controller IP and that could get integrated in other platforms so I'd prefer if this driver was called <your_platform>_edac and the DMC520 was a generic piece of functionality like the FSL memory controller IP:
mpc85xx_edac_mod-y := fsl_ddr_edac.o mpc85xx_edac.o
obj-$(CONFIG_EDAC_MPC85XX) += mpc85xx_edac_mod.o
layerscape_edac_mod-y := fsl_ddr_edac.o layerscape_edac.o
obj-$(CONFIG_EDAC_LAYERSCAPE) += layerscape_edac_mod.o
Thx.
Quoted 9 lines · Borislav Petkov
Quoted 4 lines · Sasha Levin
From: Rui Zhao <[email protected]>
New driver supports DRAM error detection and correction on DMC520 controller.
That's this thing, right?
https://developer.arm.com/products/system-ip/memory-controllers/corelink-dmc-520
Yup!
Quoted 3 lines
Quoted 1 line
Validated on actual hardware:
Which is what exactly?
A variant of a Broadcom's SST100 board.
---
Thanks,
Sasha
Borislav Petkov
maintainerQuoted 1 line · Sasha Levin
A variant of a Broadcom's SST100 board.
Is that some platform which people will use and run linux on and thus would make sense to have an EDAC driver for or is this something devel-only toy thing?
Searching a bit doesn't tell me a whole lot except some enablement for some stingray SOC reference board...
Quoted 6 lines · Borislav Petkov
Quoted 1 line · Sasha Levin
A variant of a Broadcom's SST100 board.
Is that some platform which people will use and run linux on and thus would make sense to have an EDAC driver for or is this something devel-only toy thing?
It will have (a lot of) use in prod, and is not just a toy :)
Quoted 2 lines
Searching a bit doesn't tell me a whole lot except some enablement for some stingray SOC reference board...
Right, they hardware we're working on is nothing more than reference boards at this point.
---
Thanks,
Sasha
Hi Sasha, Rui,
Quoted 6 lines · Sasha Levin
From: Rui Zhao <[email protected]>
New driver supports DRAM error detection and correction on DMC520 controller.Validated on actual hardware: DRAM errors showed up once the DDR core voltage was lowered down by 200+mV using test tool.
That's quite cool!
Quoted 7 lines
--- MAINTAINERS | 6 + drivers/edac/Kconfig | 7 + drivers/edac/Makefile | 1 + drivers/edac/dmc520_edac.c | 495 +++++++++++++++++++++++++++++++++++++ 4 files changed, 509 insertions(+) create mode 100644 drivers/edac/dmc520_edac.c
Where do I find the dt-binding for this?
It would be good if we can make this generic, so it works on all platforms with a DMC520, possibly along with other components. (e.g. the a15 L2 driver posted recently).
This will mostly be getting the DT right, as we can refactor the code when a second user comes along, but can't change the DT format.
The TRM describes 'a set of interrupts', which ones does the binding anticipate are wired up for this? There are separate status bits for corrected and uncorrected, and one pair for dram versus ram. (not sure what this corresponds with).
Do we care about the link-error interrupt?
It looks you're platform has wired corrected and uncorrected dram up as separate SPI, but not touched the ram interrupts. These are choices the soc designers made, we need to capture this stuff in the binding.
A system may have multiple memory controllers, they may share the interrupts.
(It looks like you've folded these corners out by not using IRQF_SHARED: this driver only works for independent interrupts, which can run concurrently, but work fine because the only register they both touch is interrupt_clr, which is write-only.)
For these pre-v8.2-RAS things the expectation is firmware handles all this stuff. I'm surprised your platform hasn't made the memory-controller 'secure-only', so only platform-firmware can touch it.
I can't see how this could be used on v8/aarch64, it would imply there is no firmware, which suggests this is a UP system. (or firmware trusts linux not to mess it up!)
I'm guessing this is for 32bit, where on your platform linux is running in 'secure'. This is a significant platform policy, to make this generic we'd need to find a way of describing it. i.e., other platforms may have a dmc520, the DT may describe where it is, but linux can't touch it.
I believe that today this depends on the bootloader knowing which nodes to remove when starting linux.
I think this policy-bit can be done by having a soc-family/vendor specific driver that knows which of the edac components can be used on this platform. The altera driver does something along these lines in altr_sdram_probe(). Obviously if we can have all the data come from DT that is better.
drivers/edac/dmc520_edac.c line 68
Quoted 12 lines
drivers/edac/dmc520_edac.c new file +5
-0@@ -0,0 +1,495 @@ +/* Driver settings */ +#define DMC520_EDAC_CHANS 1 +#define DMC520_EDAC_ERR_GRAIN 1 +#define DMC520_EDAC_INT_COUNT 2 +#define DMC520_EDAC_BUS_WIDTH 8
Should these be in the DT?
If someone else has a dmc520 configured slightly differently, how can we get both systems going, without having to change your system's DT?
drivers/edac/dmc520_edac.c line 176
Quoted 6 lines
drivers/edac/dmc520_edac.c new file +6
-0+static bool dmc520_is_ecc_enabled (struct dmc520_edac *edac) +{ +u32 reg_val =dmc520_read_reg (edac,REG_OFFSET_FEATURE_CONFIG ); + +return (FIELD_GET (REG_FIELD_DRAM_ECC_ENABLED , reg_val) !=0 ); +}
Ah, so there is firmware that sets this up and enables it ...
Quoted 11 lines
drivers/edac/dmc520_edac.c new file +11
-0+static int dmc520_edac_probe (struct platform_device *pdev) +{ +struct dmc520_edac *edac; +struct mem_ctl_info *mci; +struct edac_mc_layer layers[2 ]; +int ret, irq; +struct resource *res; +void __iomem *reg_base; + + res =platform_get_resource (pdev,IORESOURCE_MEM ,0 ); + reg_base =devm_ioremap_resource (&pdev->dev, res);
drivers/edac/dmc520_edac.c line 367
Quoted 10 lines
drivers/edac/dmc520_edac.c new file +10
-0+if (IS_ERR (reg_base)) +return PTR_ERR (reg_base); + + layers[0 ].type =EDAC_MC_LAYER_CHIP_SELECT ; + layers[0 ].size =dmc520_get_rank_count (reg_base); + layers[0 ].is_virt_csrow = true; + + layers[1 ].type =EDAC_MC_LAYER_CHANNEL ; + layers[1 ].size =DMC520_EDAC_CHANS ; + layers[1 ].is_virt_csrow = false;
If you can read the rank count, why hard code the bank?
(which is what I assume channels corresponds to ... although this has confused me before [0]).
drivers/edac/dmc520_edac.c line 392
Quoted 8 lines
drivers/edac/dmc520_edac.c new file +7
-0+platform_set_drvdata (pdev, mci); + + mci->pdev = &pdev->dev; + mci->mtype_cap =MEM_FLAG_DDR3 |MEM_FLAG_DDR4 ; + mci->edac_ctl_cap =EDAC_FLAG_NONE |EDAC_FLAG_SECDED ; + mci->scrub_cap =SCRUB_HW_SRC ; + mci->scrub_mode =SCRUB_NONE ;
Is this saying the device supports error scrubbing, but its disabled? Do we know that?
Can the user try and turn it on? (I can't find anything that reads this!)
It doesn't look like we can configured scrubbing if it wasn't done at boot. 3.3.245 scrub_control0_now of the TRM has "Cannot be written to and only updated when in CONFIG or LOW-POWER states". I assume we can't put this thing back into config mode when we're running.
[...]
drivers/edac/dmc520_edac.c line 440
Quoted 3 lines
drivers/edac/dmc520_edac.c new file +3
-0+/* Check ECC CE/UE errors */ +dmc520_handle_ecc_errors (mci, true, false); +dmc520_handle_ecc_errors (mci, false, false);
Do we know overflow=false?
drivers/edac/dmc520_edac.c line 445
Quoted 4 lines
drivers/edac/dmc520_edac.c new file +4
-0+/* Enable interrupts */ +dmc520_write_reg (edac, +DRAM_ECC_INT_CE_MASK |DRAM_ECC_INT_UE_MASK , +REG_OFFSET_INTERRUPT_CONTROL );
What if they were enabled before? (e.g. enabled by firmware, the bootloader or kdump). If they're already enabled, can we race with the interrupt handler on another CPU and get double reporting of the error counter?
Thanks,
James
A message that is not in the archive (sent off-list or not yet received): MW2PR2101MB09701C705250DAC025AFC251B3990@MW2PR2101MB0970.namprd21.prod.outlook.com
Hi Rui,
Quoted 11 lines · Rui Zhao
Quoted 6 lines · James Morse
It would be good if we can make this generic, so it works on all platforms with a DMC520, possibly along with other components. (e.g. the a15 L2 driver posted recently).
This will mostly be getting the DT right, as we can refactor the code when a second user comes along, but can't change the DT format.
Agreed. It'd be good if we can move all platform specific settings to DT such that a second user can update the code and won't break the DT for original device. I'll think about the DT format to make it more generic.
When the time comes, could you post a dt-binding as the first patch? These add the documentation under Documentation/device-tree/bindings, and need to be CC'd to the DT folks.
Quoted 8 lines
Quoted 6 lines
The TRM describes 'a set of interrupts', which ones does the binding anticipate are wired up for this? There are separate status bits for corrected and uncorrected, and one pair for dram versus ram. (not sure what this corresponds with).
Do we care about the link-error interrupt?
Will add ram interrupts handling, and make it configurable in DT.
I don't think the code needs to change, we just need to make it clear these are the dram interrupts, and they are different numbers.
This means someone can add the ram interrupts later, and not have to disambiguate them to keep your platform going.
(its this stuff that the binding document describes)
Quoted 3 lines
Could you please
share a bit more info on link-error interrupt? TRM doesn't have detailed info about it.
I only have the TRM too! The section titled 'RAS' on page 2-23 talks about 'link error protection', so it might be relevant so someone. We don't care about this now, but if someone does in the future, we will need to have a way of adding it to the DT.
Quoted 2 lines
Would like to know what's the impact if this error happens, and how to fit it with current reporting in EDAC core.
At a guess the interrupt triggers when link_err_count increases. (link_err has an overflow bit, so the interrupt must be related to a counter).
If we could associate a link with a layer in edac, we could report errors against that point. But I've no idea how 'links' correspond with 'ranks and banks'!
Quoted 10 lines
Quoted 8 lines
Quoted 3 lines
+ layers[1].type =
EDAC_MC_LAYER_CHANNEL;
+ layers[1].size =DMC520_EDAC_CHANS;
+ layers[1].is_virt_csrow = false;If you can read the rank count, why hard code the bank?
(which is what I assume channels corresponds to ... although this has confused me before [0]).
I misunderstood what channel meant. For bank, we can read config from the register.
Its an assumption. Channel seems to be an overloaded term with different meanings. Fortunately edac drivers get to pick what their layers mean. It looks like all this controls is the names that come out of sysfs.
rank/bank are part of the address decoding, as we can read both sizes I think it makes sense to use both as it should further localise the error.
Quoted 9 lines
Quoted 7 lines
Quoted 2 lines
+
mci->scrub_cap=SCRUB_HW_SRC;
+mci->scrub_mode=SCRUB_NONE;Is this saying the device supports error scrubbing, but its disabled? Do we know that?
Can the user try and turn it on? (I can't find anything that reads this!)
We don’t want to change what’s already configured.
I don't think we could if we wanted to. This thing has 'READY' and 'CONFIG' modes, the scrubbing can only be written to in 'CONFIG' mode. I'm willing to bet we need it to be in 'READY' to be executing the kernel from dram.
Quoted 1 line
Will change
scrub_modeto HW.
Do we know its enabled? This is something firmware has to set up, someone else's platform may do it differently.
I think should read one of the scrub control registers to find out if its turned on.
But, I can't find what uses this value ...
Quoted 11 lines
Quoted 8 lines
Quoted 4 lines
+ /* Enable interrupts */
+ dmc520_write_reg(edac,
+DRAM_ECC_INT_CE_MASK|DRAM_ECC_INT_UE_MASK, +REG_OFFSET_INTERRUPT_CONTROL);What if they were enabled before? (e.g. enabled by firmware, the bootloader or kdump). If they're already enabled, can we race with the interrupt handler on another CPU and get double reporting of the error counter?
We don't expect interrupts to be enabled by firmware or bootloader if this driver is enabled.
What about a previous instance of this driver? Linux supports kexec, kdump and hibernate, all of which cause us to inherit a slightly used platform.
Quoted 1 line
If firmware enables it, they're suppose to handle the interrupt.
Ah, so you still have resident firmware!
How come your firmware trusts linux not to turn off the memory controller?! These things are usually protected by trust zone so the OS can't pull the memory from under firmware's feet.
Thanks,
James
Borislav Petkov
maintainerQuoted 8 lines · James Morse
Quoted 2 lines
Would like to know what's the impact if this error happens, and how to fit it with current reporting in EDAC core.
At a guess the interrupt triggers when
link_err_countincreases. (link_errhas an overflow bit, so the interrupt must be related to a counter).If we could associate a link with a layer in edac, we could report errors against that point. But I've no idea how 'links' correspond with 'ranks and banks'!
Well, I have no clue what kind of links you guys are talking but if those are per-chance coherent links used by cores to communicate in a coherent fabric, or cores and devices, what would showing those errors to the user bring ya?
Or are ya talking about different kinds of links?
In any case, the first question to ask would be, can some agent or the user do something with the information that X or Y link errors happened?
If not, then why bother?
If yes, then that's a different story.
Hi Boris,
Quoted 7 lines · Borislav Petkov
Quoted 6 lines · James Morse
Quoted 2 lines
Would like to know what's the impact if this error happens, and how to fit it with current reporting in EDAC core.
At a guess the interrupt triggers when
link_err_countincreases. (link_errhas an overflow bit, so the interrupt must be related to a counter).
Quoted 8 lines
Quoted 2 lines
If we could associate a link with a layer in edac, we could report errors against that point. But I've no idea how 'links' correspond with 'ranks and banks'!
Well, I have no clue what kind of links you guys are talking but if those are per-chance coherent links used by cores to communicate in a coherent fabric, or cores and devices, what would showing those errors to the user bring ya?
(I mentioned this because its the next interrupt in the register, its an example of something that may be added for another platform in the future, which affects the DT and probing)
Quoted 1 line
Or are ya talking about different kinds of links?
... whatever the manual means by 'link', good point, it could be the interconnect side.
'alert_mode_next', in the feature control register talks about DIMM training, and says 'dfi_err' is treated a a link error. DFI is defined earlier as the 'DDR PHY interface', so these must be links between the DMC520 and DDR.
Quoted 5 lines
In any case, the first question to ask would be, can some agent or the user do something with the information that X or Y link errors happened?
If not, then why bother?
If yes, then that's a different story.
I agree. Surely if the DIMMs are socketed link-errors are another reason to replace the DIMM.
It looks like this doesn't matter on Rui's platform,
Thanks,
James
Hi James,
Quoted 1 line · James Morse
When the time comes, could you post a dt-binding as the first patch? These add the documentation under
Documentation/device-tree/bindings, and need to be CC'd to the DT folks.
Sure, will do.
Quoted 5 lines
Quoted 1 line
Will change
scrub_modeto HW.Do we know its enabled? This is something firmware has to set up, someone else's platform may do it differently. I think should read one of the scrub control registers to find out if its turned on.
But, I can't find what uses this value ...
Looks like if scrub_mode is set to SCRUB_SW_SRC, the MC core will do arch specific sw scrub on the page with ce error. We can config the mode base on register settings.
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/edac/dmc520_edac.c new file +4
-0+/* Enable interrupts */ +dmc520_write_reg (edac, +DRAM_ECC_INT_CE_MASK |DRAM_ECC_INT_UE_MASK , +REG_OFFSET_INTERRUPT_CONTROL );
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
What if they were enabled before? (e.g. enabled by firmware, the bootloader or kdump). If they're already enabled, can we race with
the interrupt handler on another CPU and get double reporting of the error counter?We don't expect interrupts to be enabled by firmware or bootloader if this driver is enabled.
What about a previous instance of this driver? Linux supports kexec, kdump and hibernate, all of which cause us to inherit a slightly used platform.
Agreed. We shouldn't make assumption on the register value when driver starts.
Quoted 5 lines
Quoted 1 line
If firmware enables it, they're suppose to handle the interrupt.
Ah, so you still have resident firmware!
How come your firmware trusts linux not to turn off the memory controller?!
These things are usually protected by trust zone so the OS can't pull the memory from under firmware's feet.
We have firmware to config the memory controller and want to have an EDAC driver to report ECC status. Could you please elaborate a bit on the security concern on this approach? Like some malicious app/driver can access memory controller registers can cause issue?
What's the recommend approach if Linux won't be able to access memory controller registers? Have firmware do the ECC status monitoring and some sort of driver to query ECC status from firmware?
Appreciate your insights on this.
Thanks,
Rui
Hi Rui,
Quoted 11 lines · Rui Zhao
Quoted 5 lines · James Morse
Quoted 1 line
If firmware enables it, they're suppose to handle the interrupt.
Ah, so you still have resident firmware!
How come your firmware trusts linux not to turn off the memory controller?!
These things are usually protected by trust zone so the OS can't pull the memory from under firmware's feet.We have firmware to config the memory controller and want to have an EDAC driver to report ECC status.
Could you please elaborate a bit on the security concern on this approach? Like some malicious app/driver can access memory controller registers can cause issue?
I'm remembering this:
https://lore.kernel.org/linux-arm-kernel/[email protected]/
Robin Murphy wrote:
| [ For anyone interested, it puts the DRAM controller into sleep mode. | The kernel can't even panic if all the memory suddenly disappears :D ]
This would be a problem if you need your Secure-world software needs to keep working, and depends on the memory behind this controller.
It might be that your secure-world software only uses some other memory, in which case this wouldn't matter.
It may be linux _is_ your secure-world software, in which case it wouldn't matter either.
Quoted 3 lines
What's the recommend approach if Linux won't be able to access memory controller registers? Have firmware do the ECC
status monitoring and some sort of driver to query ECC status from firmware?
If Linux runs in the normal-world, can't you use trust-zone to prevent Linux from accessing the memory controller?
If you did this, you'd need to handle the UE interrupts in firmware, and wouldn't be able to use this driver in linux. Your platform hasn't gone this way, so I guess one of the above cases applies.
Thanks,
James
Hi James,
Quoted 10 lines · James Morse
Quoted 5 lines
We have firmware to config the memory controller and want to have an EDAC driver to report ECC status.
Could you please elaborate a bit on the security concern on this
approach? Like some malicious app/driver can access memory controller registers can cause issue?
Quoted 8 lines
Robin Murphy wrote:
| [ For anyone interested, it puts the DRAM controller into sleep mode. | The kernel can't even panic if all the memory suddenly disappears :D ]This would be a problem if you need your Secure-world software needs to keep working, and depends on the memory behind this controller.
It might be that your secure-world software only uses some other memory, in which case this wouldn't matter. It may be linux _is_ your secure-world software, in which case it wouldn't matter either.
We had internal discussion with our security team and in our product we do trust Linux. I'll send an updated patch to move platform specific settings like interrupt config to DT and include DT bindings doc for this driver.
Thanks,
Rui