ECC memory of BMC
Hi,
we are trying to enable ECC on BMC memory.
is there any exist solution on BMC?
we can also share our proposal to review if there was anyone interested in it.
BRs,
Will
Quoted 6 lines · Will Liang (梁永鉉)
Hi,
we are trying to enable ECC on BMC memory.
is there any exist solution on BMC?
Well, for what it's worth a kernel driver has been submitted upstream and is queued for the 5.1 merge window:
The patch does say it needs bootloader support though, and I'm not exactly sure what that involves. I've added Stefan to Cc, maybe he can provide some insight.
Andrew
Hi Andrew,
Thanks for your response.
We have also found EDAC driver.
What we want to do is to record the ECC events to SEL.
we are considering to create new dbus and a service.
Do you have any suggestions?
BRs,
Will
________________________________
寄件者: Andrew Jeffery <[email protected]>
寄件日期: 2019年2月21日 下午 03:10
收件者: [email protected]; Will Liang (梁永鉉)
副本: Stefan M Schaeckeler
主旨: Re: ECC memory of BMC
Quoted 6 lines · Will Liang (梁永鉉)
Hi,
we are trying to enable ECC on BMC memory.
is there any exist solution on BMC?
Well, for what it's worth a kernel driver has been submitted upstream and is queued for the 5.1 merge window:
The patch does say it needs bootloader support though, and I'm not exactly sure what that involves. I've added Stefan to Cc, maybe he can provide some insight.
Andrew
Hi all,
ECC needs to be enabled in u-boot. The reason is explained in my first upload https://patchwork.kernel.org/patch/10732769/
Our u-boot engineer did it for us. I see u-boot code is full of #ifdef CONFIG_DRAM_ECC blocks. In theory, enabling this option should be enough.
- A good indication of having ECC successfully enabled is linux actually booting.
- Another indication is available memory being 8/9th of what it used to be (check /proc/meminfo)
- Then, my driver should not log any errors in its probe function (check dmesg).
Stefan
On 2/20/19, 11:27 PM, "Will Liang (梁永鉉)" <[email protected]> wrote:
Hi Andrew,
Thanks for your response.
We have also found EDAC driver.
What we want to do is to record the ECC events to SEL.
we are considering to create new dbus and a service.
Do you have any suggestions?
BRs,
Will
________________________________________
寄件者: Andrew Jeffery <[email protected]> 寄件日期: 2019年2月21日 下午 03:10
收件者: [email protected]; Will Liang (梁永鉉) 副本: Stefan M Schaeckeler
主旨: Re: ECC memory of BMC
On Thu, 21 Feb 2019, at 17:16, Will Liang (梁永鉉) wrote:
>
> Hi,
>
> we are trying to enable ECC on BMC memory. >
> is there any exist solution on BMC?
Well, for what it's worth a kernel driver has been submitted upstream and is queued for the 5.1 merge window:
The patch does say it needs bootloader support though, and I'm not exactly sure what that involves. I've added Stefan to Cc, maybe he can provide some insight.
Andrew
Hi Will,
As a note, please send text-only email to the list in the future, and try to avoid top-posting (try to reply "inline" underneath the paragraph of interest, this way the context is always adjacent to your response)
I understand text-only email and inline replies can be hard to do with some mail clients, but that's the list preference.
Quoted 10 lines · Will Liang (梁永鉉)
Hi Andrew,
Thanks for your response.
We have also found EDAC driver.
What we want to do is to record the ECC events to SEL.
we are considering to create new dbus and a service.
Right; I think you need to create a new service that polls the sysfs interface for the EDAC device, and then use phosphor-logging to create error logs. I'm not much of a userspace guy, so I've Cc'ed Deepak who might be able to help on that front.
Andrew
Quoted 7 lines · Will Liang (梁永鉉)
Quoted 5 lines
Hi Andrew,
Thanks for your response.
Quoted 8 lines
Quoted 5 lines
We have also found EDAC driver.
What we want to do is to record the ECC events to SEL.
we are considering to create new dbus and a service.
Right; I think you need to create a new service that polls the sysfs interface for the EDAC device, and then use phosphor-logging to create error logs.
We consider creating the following objects for D-Bus:
-bus name : /xyz/openbmc_project/ECC
-object path : /xyz/openbmc_project/ECC/status -interface : xyz.openbmc_project.Memory.MemoryECC
and error types for xyz::openbmc_project::Memory::Ecc::Error::ceCount and "ueCount" and "isLoggingLimitReached" for phosphor-logging error message.
I'm not
Quoted 5 lines
much of a userspace guy, so I've Cc'ed Deepak who might be able to help on that front.
Andrew
Quoted 35 lines · Stefan Schaeckeler (sschaeck)
Hi Will,
Quoted 12 lines · Will Liang (梁永鉉)
Quoted 6 lines
Quoted 3 lines
What we want to do is to record the ECC events to SEL.
we are considering to create new dbus and a service.
Right; I think you need to create a new service that polls the sysfs interface for the EDAC device, and then use phosphor-logging to create error logs.
We consider creating the following objects for D-Bus:
(quoted diff) +0
-3-bus name : /xyz/openbmc_project/ECC -object path : /xyz/openbmc_project/ECC/status -interface : xyz.openbmc_project.Memory.MemoryECCQuoted 2 lines
and error types for
xyz::openbmc_project::Memory::Ecc::Error::ceCountand "ueCount" and "isLoggingLimitReached" for phosphor-logging error message.Note, the driver also logs the addresses of the recoverable and un-recoverable errors. Perhaps you want to expose them, too?
The edac framework is unfortunately not exposing them through sysfs. They get printed through "
edac_mc_handle_error()" asprintk(KERN_WARNING, ...)and look likeroot@aspeed-arm:# dmesg | grep EDAC [ 1718.900000] EDAC MC0: 1 CE address(es) not available on mc#0csrow#0channel#0 (csrow:0 channel:0 page:0x0 offset:0x0 grain:0 syndrome:0x0) [ 1718.900000] EDAC MC0: 1 CE on mc#0csrow#0channel#0 (csrow:0 channel:0 page:0x80000 offset:0x0 grain:0 syndrome:0x0)
Quoted 3 lines
I'm not sure if there is an elegant way for userspace to retrieve messages from the kernel ring buffer.
Lets not start scraping dmesg. It's not considered part of the kernel ABI.
Hi Will,
Quoted 12 lines · Will Liang (梁永鉉)
Quoted 6 lines
Quoted 3 lines
What we want to do is to record the ECC events to SEL.
we are considering to create new dbus and a service.
Right; I think you need to create a new service that polls the sysfs interface for the EDAC device, and then use phosphor-logging to create error logs.
We consider creating the following objects for D-Bus:
(quoted diff) +0
-3-bus name : /xyz/openbmc_project/ECC -object path : /xyz/openbmc_project/ECC/status -interface : xyz.openbmc_project.Memory.MemoryECC
Quoted 2 lines
and error types for
xyz::openbmc_project::Memory::Ecc::Error::ceCountand "ueCount" and "isLoggingLimitReached" for phosphor-logging error message.
Note, the driver also logs the addresses of the recoverable and un-recoverable errors. Perhaps you want to expose them, too?
The edac framework is unfortunately not exposing them through sysfs. They get printed through "edac_mc_handle_error()" as printk(KERN_WARNING, ...) and look like
root@aspeed-arm:# dmesg | grep EDAC [ 1718.900000] EDAC MC0: 1 CE address(es) not available on mc#0csrow#0channel#0 (csrow:0 channel:0 page:0x0 offset:0x0 grain:0 syndrome:0x0) [ 1718.900000] EDAC MC0: 1 CE on mc#0csrow#0channel#0 (csrow:0 channel:0 page:0x80000 offset:0x0 grain:0 syndrome:0x0)
I'm not sure if there is an elegant way for userspace to retrieve messages from the kernel ring buffer.
Stefan
Hello Will,
One more thing about enabling ECC:
Quoted 10 lines
ECC needs to be enabled in u-boot. The reason is explained in my first upload
https://patchwork.kernel.org/patch/10732769/Our u-boot engineer did it for us. I see u-boot code is full of #ifdef
CONFIG_DRAM_ECCblocks. In theory, enabling this option should be enough.- A good indication of having ECC successfully enabled is linux actually booting.
- Another indication is available memory being 8/9th of what it used to be (check/proc/meminfo)
- Then, my driver should not log any errors in its probe function (check dmesg).
While you have not set up ECC mode in u-boot yet, you might be able to use the edac driver by removing the following code in aspeed_probe()
/* bail out if ECC mode is not configured */
regmap_read(aspeed_regmap, ASPEED_MCR_CONF, ®04);
if (!(reg04 & ASPEED_MCR_CONF_ECC)) {
dev_err(&pdev->dev, "ECC mode is not configured in u-boot\n");
return -EPERM;
}This will run the probe function hopefully to the end and expose the sysfs nodes. I have not tested it. Maybe you need to further butcher down the driver?
You can then inject fake ecc errors via the kernel debug framework:
root@aspeed-arm:/# echo 42 > /sys/kernel/debug/edac/mc0/fake_inject
root@aspeed-arm:/# dmesg | tail -1 [ 293.020000] EDAC MC0: 1 CE FAKE ERROR on mc#0csrow#0channel#0 (csrow:0 channel:0 page:0x0 offset:0x0 grain:0 syndrome:0x0 - for EDAC testing only)
root@aspeed-arm:/# cat /sys/devices/system/edac/mc/mc0/ce* 1 0
BTW, here are my notes on EDAC and ECC on Aspeed which you might find
interesting: http://students.engr.scu.edu/~sschaeck/misc/aspeed-edac.html
Hope that helps, Stefan
Hi,
Quoted 47 lines
-----Original Message-----
From: Andrew Jeffery [mailto:[email protected]]
Sent: Friday, February 22, 2019 1:55 PM
To: Will Liang (梁永鉉) <[email protected]>; [email protected]; Stefan M Schaeckeler <[email protected]>
Cc: [email protected]
Subject: Re: ECC memory of BMCQuoted 3 lines · Stefan Schaeckeler (sschaeck)
Hi Will,
On 2/21/19, 6:00 PM, "Will Liang (梁永鉉)" <[email protected]>
Quoted 7 lines · wrote:
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
What we want to do is to record the ECC events to SEL.
we are considering to create new dbus and a service.
Right; I think you need to create a new service that polls the sysfs interface for the EDAC device, and then use phosphor-logging to
create error logs.
Quoted 7 lines
Quoted 7 lines
We consider creating the following objects for D-Bus:
-bus name : /xyz/openbmc_project/ECC -object path :
/xyz/openbmc_project/ECC/status -interface :
xyz.openbmc_project.Memory.MemoryECCand error types for
xyz::openbmc_project::Memory::Ecc::Error::ceCountand "ueCount"
Quoted 18 lines
Quoted 1 line
and "isLoggingLimitReached" for phosphor-logging error message.
Note, the driver also logs the addresses of the recoverable and un-recoverable errors. Perhaps you want to expose them, too?
The edac framework is unfortunately not exposing them through sysfs. They get printed through "
edac_mc_handle_error()" asprintk(KERN_WARNING, ...)and look likeroot@aspeed-arm:# dmesg | grep EDAC [ 1718.900000] EDAC MC0: 1 CE address(es) not available on mc#0csrow#0channel#0 (csrow:0 channel:0 page:0x0 offset:0x0 grain:0 syndrome:0x0) [ 1718.900000] EDAC MC0: 1 CE on mc#0csrow#0channel#0 (csrow:0 channel:0 page:0x80000 offset:0x0 grain:0 syndrome:0x0)
Quoted 6 lines
Quoted 1 line
I'm not sure if there is an elegant way for userspace to retrieve messages
from
Quoted 2 lines
the kernel ring buffer.
Lets not start scraping dmesg. It's not considered part of the kernel ABI.
We do not expose error message from EDAC driver.
We only want to fetch recoverable/un-recoverable counts and record the ECC log. Therefore, we need a service to poll the EDAC driver.
Will
Quoted 52 lines
Hello Will,
One more thing about enabling ECC:
Quoted 4 lines
ECC needs to be enabled in u-boot. The reason is explained in my first upload https://patchwork.kernel.org/patch/10732769/
Our u-boot engineer did it for us. I see u-boot code is full of #ifdef
CONFIG_DRAM_ECCblocks.Quoted 4 lines
In theory, enabling this option should be enough.
- A good indication of having ECC successfully enabled is linux actually
booting.
Quoted 3 lines
- Another indication is available memory being 8/9th of what it used to be (check
/proc/meminfo)
- Then, my driver should not log any errors in its probe function (checkdmesg).
While you have not set up ECC mode in u-boot yet, you might be able to use the edac driver by removing the following code in
aspeed_probe()/* bail out if ECC mode is not configured */ regmap_read(aspeed_regmap, ASPEED_MCR_CONF, ®04); if (!(reg04 & ASPEED_MCR_CONF_ECC)) { dev_err(&pdev->dev, "ECC mode is not configured in u-boot\n"); return -EPERM; }This will run the probe function hopefully to the end and expose the sysfs nodes.
I have not tested it. Maybe you need to further butcher down the driver?You can then inject fake ecc errors via the kernel debug framework:
root@aspeed-arm:/# echo 42 > /sys/kernel/debug/edac/mc0/fake_injectroot@aspeed-arm:/# dmesg | tail -1 [ 293.020000] EDAC MC0: 1 CE FAKE ERROR on mc#0csrow#0channel#0 (csrow:0 channel:0 page:0x0 offset:0x0 grain:0 syndrome:0x0 - for EDAC testing only)root@aspeed-arm:/# cat /sys/devices/system/edac/mc/mc0/ce* 1 0
Quoted 6 lines
BTW, here are my notes on EDAC and ECC on Aspeed which you might find
interesting: http://students.engr.scu.edu/~sschaeck/misc/aspeed-edac.htmlHope that helps, Stefan
Hi Stefan,
Thanks for the updated information.
Instead of extending the kernel space, we are developing services in user space.
Will