Skip to contentSkip to conversation

ECC memory of BMC

DISCUSSION Will Liang (梁永鉉) 3 participants 11 messages Feb 21, 2019 – Feb 22, 2019

lore
Share & tools
lore
#0 lore raw

Hi,

we are trying to enable ECC on BMC memory.

is there any exist solution on BMC?

we can also share our proposal to review if there was anyone interested in it.

BRs,

Will

#1 lore raw
Quoted 6 lines · Will Liang (梁永鉉)

Hi,

we are trying to enable ECC on BMC memory.

is there any exist solution on BMC?

Well, for what it's worth a kernel driver has been submitted upstream and is queued for the 5.1 merge window:

https://git.kernel.org/pub/scm/linux/kernel/git/bp/bp.git/commit/?h=for-next&id=9b7e6242ee4efcd7f9ef699bf1965e3a5343f216

The patch does say it needs bootloader support though, and I'm not exactly sure what that involves. I've added Stefan to Cc, maybe he can provide some insight.

Andrew

#2 lore raw

Hi Andrew,

Thanks for your response.

We have also found EDAC driver.

What we want to do is to record the ECC events to SEL.

we are considering to create new dbus and a service.

Do you have any suggestions?

BRs,

Will

________________________________
寄件者: Andrew Jeffery <[email protected]>
寄件日期: 2019年2月21日 下午 03:10
收件者: [email protected]; Will Liang (梁永鉉)
副本: Stefan M Schaeckeler
主旨: Re: ECC memory of BMC

Quoted 6 lines · Will Liang (梁永鉉)

Hi,

we are trying to enable ECC on BMC memory.

is there any exist solution on BMC?

Well, for what it's worth a kernel driver has been submitted upstream and is queued for the 5.1 merge window:

https://git.kernel.org/pub/scm/linux/kernel/git/bp/bp.git/commit/?h=for-next&id=9b7e6242ee4efcd7f9ef699bf1965e3a5343f216

The patch does say it needs bootloader support though, and I'm not exactly sure what that involves. I've added Stefan to Cc, maybe he can provide some insight.

Andrew

#3 lore raw

Hi all,

ECC needs to be enabled in u-boot. The reason is explained in my first upload https://patchwork.kernel.org/patch/10732769/

Our u-boot engineer did it for us. I see u-boot code is full of #ifdef CONFIG_DRAM_ECC blocks. In theory, enabling this option should be enough.

- A good indication of having ECC successfully enabled is linux actually booting.
- Another indication is available memory being 8/9th of what it used to be (check /proc/meminfo)
- Then, my driver should not log any errors in its probe function (check dmesg).

Stefan

On 2/20/19, 11:27 PM, "Will Liang (梁永鉉)" <[email protected]> wrote:

Hi Andrew,

Thanks for your response.

We have also found EDAC driver.
What we want to do is to record the ECC events to SEL.

we are considering to create new dbus and a service.

Do you have any suggestions?

BRs,
Will

________________________________________
寄件者: Andrew Jeffery <[email protected]> 寄件日期: 2019年2月21日 下午 03:10
收件者: [email protected]; Will Liang (梁永鉉) 副本: Stefan M Schaeckeler
主旨: Re: ECC memory of BMC

On Thu, 21 Feb 2019, at 17:16, Will Liang (梁永鉉) wrote:
>
> Hi,
>
> we are trying to enable ECC on BMC memory. >
> is there any exist solution on BMC?

Well, for what it's worth a kernel driver has been submitted upstream and is queued for the 5.1 merge window:

https://git.kernel.org/pub/scm/linux/kernel/git/bp/bp.git/commit/?h=for-next&id=9b7e6242ee4efcd7f9ef699bf1965e3a5343f216

The patch does say it needs bootloader support though, and I'm not exactly sure what that involves. I've added Stefan to Cc, maybe he can provide some insight.

Andrew

#6 lore raw
In reply to Will Liang (梁永鉉)

Hi Will,

As a note, please send text-only email to the list in the future, and try to avoid top-posting (try to reply "inline" underneath the paragraph of interest, this way the context is always adjacent to your response)

I understand text-only email and inline replies can be hard to do with some mail clients, but that's the list preference.

Quoted 10 lines · Will Liang (梁永鉉)

Hi Andrew,

Thanks for your response.

We have also found EDAC driver.

What we want to do is to record the ECC events to SEL.

we are considering to create new dbus and a service.

Right; I think you need to create a new service that polls the sysfs interface for the EDAC device, and then use phosphor-logging to create error logs. I'm not much of a userspace guy, so I've Cc'ed Deepak who might be able to help on that front.

Andrew

#7 lore raw
Quoted 7 lines · Will Liang (梁永鉉)
Quoted 5 lines

Hi Andrew,

Thanks for your response.

Quoted 8 lines
Quoted 5 lines

We have also found EDAC driver.

What we want to do is to record the ECC events to SEL.

we are considering to create new dbus and a service.

Right; I think you need to create a new service that polls the sysfs interface for the EDAC device, and then use phosphor-logging to create error logs.

We consider creating the following objects for D-Bus:
-bus name : /xyz/openbmc_project/ECC
-object path : /xyz/openbmc_project/ECC/status -interface : xyz.openbmc_project.Memory.MemoryECC

and error types for xyz::openbmc_project::Memory::Ecc::Error::ceCount and "ueCount" and "isLoggingLimitReached" for phosphor-logging error message.

I'm not

Quoted 5 lines

much of a userspace guy, so I've Cc'ed Deepak who might be able to help on that front.

Andrew

#9 lore raw
In reply to Stefan Schaeckeler (sschaeck)
Quoted 35 lines · Stefan Schaeckeler (sschaeck)

Hi Will,

Quoted 12 lines · Will Liang (梁永鉉)
Quoted 6 lines
Quoted 3 lines

What we want to do is to record the ECC events to SEL.

we are considering to create new dbus and a service.

Right; I think you need to create a new service that polls the sysfs interface for the EDAC device, and then use phosphor-logging to create error logs.

We consider creating the following objects for D-Bus:

(quoted diff) +0 -3
-bus name : /xyz/openbmc_project/ECC
-object path : /xyz/openbmc_project/ECC/status
-interface : xyz.openbmc_project.Memory.MemoryECC
Quoted 2 lines

and error types for xyz::openbmc_project::Memory::Ecc::Error::ceCount and "ueCount" and "isLoggingLimitReached" for phosphor-logging error message.

Note, the driver also logs the addresses of the recoverable and un-recoverable errors. Perhaps you want to expose them, too?

The edac framework is unfortunately not exposing them through sysfs. They get printed through "edac_mc_handle_error()" as printk(KERN_WARNING, ...) and look like

root@aspeed-arm:# dmesg | grep EDAC
[ 1718.900000] EDAC MC0: 1 CE address(es) not available on 
mc#0csrow#0channel#0 (csrow:0 channel:0 page:0x0 offset:0x0 grain:0 
syndrome:0x0)
[ 1718.900000] EDAC MC0: 1 CE on mc#0csrow#0channel#0 (csrow:0 
channel:0 page:0x80000 offset:0x0 grain:0 syndrome:0x0)
Quoted 3 lines

I'm not sure if there is an elegant way for userspace to retrieve messages from the kernel ring buffer.

Lets not start scraping dmesg. It's not considered part of the kernel ABI.

#8 lore raw
In reply to Will Liang (梁永鉉)

Hi Will,

Quoted 12 lines · Will Liang (梁永鉉)
Quoted 6 lines
Quoted 3 lines

What we want to do is to record the ECC events to SEL.

we are considering to create new dbus and a service.

Right; I think you need to create a new service that polls the sysfs interface for the EDAC device, and then use phosphor-logging to create error logs.

We consider creating the following objects for D-Bus:

(quoted diff) +0 -3
-bus name : /xyz/openbmc_project/ECC
-object path : /xyz/openbmc_project/ECC/status
-interface : xyz.openbmc_project.Memory.MemoryECC
Quoted 2 lines

and error types for xyz::openbmc_project::Memory::Ecc::Error::ceCount and "ueCount" and "isLoggingLimitReached" for phosphor-logging error message.

Note, the driver also logs the addresses of the recoverable and un-recoverable errors. Perhaps you want to expose them, too?

The edac framework is unfortunately not exposing them through sysfs. They get printed through "edac_mc_handle_error()" as printk(KERN_WARNING, ...) and look like

root@aspeed-arm:# dmesg | grep EDAC
[ 1718.900000] EDAC MC0: 1 CE address(es) not available on mc#0csrow#0channel#0 (csrow:0 channel:0 page:0x0 offset:0x0 grain:0 syndrome:0x0)
[ 1718.900000] EDAC MC0: 1 CE on mc#0csrow#0channel#0 (csrow:0 channel:0 page:0x80000 offset:0x0 grain:0 syndrome:0x0)

I'm not sure if there is an elegant way for userspace to retrieve messages from the kernel ring buffer.

Stefan

#4 lore raw
In reply to Stefan Schaeckeler (sschaeck)

Hello Will,

One more thing about enabling ECC:

Quoted 10 lines

ECC needs to be enabled in u-boot. The reason is explained in my first upload
https://patchwork.kernel.org/patch/10732769/

Our u-boot engineer did it for us. I see u-boot code is full of #ifdef CONFIG_DRAM_ECC blocks. In theory, enabling this option should be enough.

- A good indication of having ECC successfully enabled is linux actually booting.
- Another indication is available memory being 8/9th of what it used to be (check /proc/meminfo)
- Then, my driver should not log any errors in its probe function (check dmesg).

While you have not set up ECC mode in u-boot yet, you might be able to use the edac driver by removing the following code in aspeed_probe()

	/* bail out if ECC mode is not configured */
	regmap_read(aspeed_regmap, ASPEED_MCR_CONF, &reg04);
	if (!(reg04 & ASPEED_MCR_CONF_ECC)) {
		dev_err(&pdev->dev, "ECC mode is not configured in u-boot\n");
		return -EPERM;
	}

This will run the probe function hopefully to the end and expose the sysfs nodes. I have not tested it. Maybe you need to further butcher down the driver?

You can then inject fake ecc errors via the kernel debug framework:

root@aspeed-arm:/# echo 42 > /sys/kernel/debug/edac/mc0/fake_inject  
root@aspeed-arm:/# dmesg | tail -1
[  293.020000] EDAC MC0: 1 CE FAKE ERROR on mc#0csrow#0channel#0 (csrow:0 channel:0 page:0x0 offset:0x0 grain:0 syndrome:0x0 - for EDAC testing only)
root@aspeed-arm:/# cat /sys/devices/system/edac/mc/mc0/ce*
1
0

BTW, here are my notes on EDAC and ECC on Aspeed which you might find
interesting: http://students.engr.scu.edu/~sschaeck/misc/aspeed-edac.html

Hope that helps, Stefan

#10 lore raw
In reply to Andrew Jeffery

Hi,

Quoted 47 lines

-----Original Message-----
From: Andrew Jeffery [mailto:[email protected]]
Sent: Friday, February 22, 2019 1:55 PM
To: Will Liang (梁永鉉) <[email protected]>; [email protected]; Stefan M Schaeckeler <[email protected]>
Cc: [email protected]
Subject: Re: ECC memory of BMC

Quoted 3 lines · Stefan Schaeckeler (sschaeck)

Hi Will,

On 2/21/19, 6:00 PM, "Will Liang (梁永鉉)" <[email protected]>

Quoted 7 lines · wrote:
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

What we want to do is to record the ECC events to SEL.

we are considering to create new dbus and a service.

Right; I think you need to create a new service that polls the sysfs interface for the EDAC device, and then use phosphor-logging to

create error logs.

Quoted 7 lines
Quoted 7 lines

We consider creating the following objects for D-Bus:
-bus name : /xyz/openbmc_project/ECC -object path :
/xyz/openbmc_project/ECC/status -interface :
xyz.openbmc_project.Memory.MemoryECC

and error types for xyz::openbmc_project::Memory::Ecc::Error::ceCount

and "ueCount"

Quoted 18 lines
Quoted 1 line

and "isLoggingLimitReached" for phosphor-logging error message.

Note, the driver also logs the addresses of the recoverable and un-recoverable errors. Perhaps you want to expose them, too?

The edac framework is unfortunately not exposing them through sysfs. They get printed through "edac_mc_handle_error()" as printk(KERN_WARNING, ...) and look like

root@aspeed-arm:# dmesg | grep EDAC
[ 1718.900000] EDAC MC0: 1 CE address(es) not available on
mc#0csrow#0channel#0 (csrow:0 channel:0 page:0x0 offset:0x0 grain:0
syndrome:0x0)
[ 1718.900000] EDAC MC0: 1 CE on mc#0csrow#0channel#0 (csrow:0
channel:0 page:0x80000 offset:0x0 grain:0 syndrome:0x0)
Quoted 6 lines
Quoted 1 line

I'm not sure if there is an elegant way for userspace to retrieve messages

from

Quoted 2 lines

the kernel ring buffer.

Lets not start scraping dmesg. It's not considered part of the kernel ABI.

We do not expose error message from EDAC driver.

We only want to fetch recoverable/un-recoverable counts and record the ECC log. Therefore, we need a service to poll the EDAC driver.

Will

#5 lore raw
In reply to Stefan Schaeckeler (sschaeck)
Quoted 52 lines

Hello Will,

One more thing about enabling ECC:

Quoted 4 lines

ECC needs to be enabled in u-boot. The reason is explained in my first upload https://patchwork.kernel.org/patch/10732769/

Our u-boot engineer did it for us. I see u-boot code is full of #ifdef

CONFIG_DRAM_ECC blocks.

Quoted 4 lines

In theory, enabling this option should be enough.

- A good indication of having ECC successfully enabled is linux actually

booting.

Quoted 3 lines

- Another indication is available memory being 8/9th of what it used to be (check /proc/meminfo)
- Then, my driver should not log any errors in its probe function (check

dmesg).

While you have not set up ECC mode in u-boot yet, you might be able to use the edac driver by removing the following code in aspeed_probe()

	/* bail out if ECC mode is not configured */
	regmap_read(aspeed_regmap, ASPEED_MCR_CONF, &reg04);
	if (!(reg04 & ASPEED_MCR_CONF_ECC)) {
		dev_err(&pdev->dev, "ECC mode is not configured in u-boot\n");
		return -EPERM;
	}

This will run the probe function hopefully to the end and expose the sysfs nodes.
I have not tested it. Maybe you need to further butcher down the driver?

You can then inject fake ecc errors via the kernel debug framework:

root@aspeed-arm:/# echo 42 > /sys/kernel/debug/edac/mc0/fake_inject
root@aspeed-arm:/# dmesg | tail -1
[  293.020000] EDAC MC0: 1 CE FAKE ERROR on mc#0csrow#0channel#0
(csrow:0 channel:0 page:0x0 offset:0x0 grain:0 syndrome:0x0 - for EDAC
testing only)
root@aspeed-arm:/# cat /sys/devices/system/edac/mc/mc0/ce*
1
0
Quoted 6 lines

BTW, here are my notes on EDAC and ECC on Aspeed which you might find
interesting: http://students.engr.scu.edu/~sschaeck/misc/aspeed-edac.html

Hope that helps, Stefan

Hi Stefan,

Thanks for the updated information.
Instead of extending the kernel space, we are developing services in user space.

Will