Skip to contentSkip to conversation

EDAC and the sysfs

RFC Doug Thompson 4 participants 12 messages Nov 14, 2005 – Nov 17, 2005

lore
Share & tools
lore
#0 lore raw

I am trying to design the sysfs interface tree for the new set of EDAC modules that are waiting for this interface, before being put into the kernel.

Currently the original EDAC (bluesmoke) has its own /proc directory (/proc/mc) with files and a directory (0,1,2,...)for each memory controller on the system. This will be removed and the new information interface will be placed in the sysfs.

One proposal is to place the information in /sys/devices/system in the following directories:

For EDAC general memory ECC controls and information files:

/sys/devices/systems/edac/mc/

For PCI Parity Error detection controls and information files:

/sys/devices/system/edac/pci

In addition /sys/devices/system/edac/mc/ would have directories:

mc0/
mc1/
...

for each memory controller's specific controls and information.

Currently the similiar error detection device /sys/devices/system/machinecheck resides here are well.

The alternative layout would be to use the /sys/class directory when nested-classes become available:

/sys/class/edac/mc/...

and

/sys/class/edac/pci/...

But edac doesn't quite seem to fit here.

I have failed to date to really find a policy or set of rules of use for the sysfs as to what goes where for such items as EDAC. After searching the web, articles and thinking about this for some time now, I am requesting comments on the sysfs model for where EDAC would fit best.

I currently favor the /sys/devices/system/edac placement at the moment, but would welcome input.

thanks

doug thompson

"If you think Education is expensive, just try Ignorance"

"Don't tell people HOW to do things, tell them WHAT you want and they will surprise you with their ingenuity."
Gen George Patton

#1 lore raw
Quoted 5 lines · Doug Thompson

I am trying to design the sysfs interface tree for the new set of EDAC modules that are waiting for this interface, before being put into the kernel.

Quoted 8 lines

Currently the original EDAC (bluesmoke) has its own /proc directory (/proc/mc) with files and a directory (0,1,2,...)for each memory controller on the system. This will be removed and the new information interface will be placed in the sysfs.

One proposal is to place the information in /sys/devices/system in the following directories:

Why not use /sys/firmware/ instead?

Or do you want to use the struct device stuff?

Quoted 4 lines

For EDAC general memory ECC controls and information files:

/sys/devices/systems/edac/mc/

What kind of controls and files?

Quoted 5 lines

For PCI Parity Error detection controls and information files:

/sys/devices/system/edac/pci

That kind of controls and files?

Quoted 9 lines

In addition /sys/devices/system/edac/mc/ would have directories:

mc0/
mc1/
...

for each memory controller's specific controls and information.

Again, what kind of controls and information?

Quoted 7 lines

Currently the similiar error detection device /sys/devices/system/machinecheck resides here are well.

The alternative layout would be to use the /sys/class directory when nested-classes become available:

They are in 2.6.15-rc1, but you _really_ don't want to use them, they are a huge pain, and I will be getting rid of them, along with all struct class_device stuff in the near future. See the archives for details, or it's summarized here:
http://www.kroah.com/log/linux/driver_model_changes.html

Quoted 8 lines

/sys/class/edac/mc/...

and

/sys/class/edac/pci/...

But edac doesn't quite seem to fit here.

I agree.

Quoted 6 lines

I have failed to date to really find a policy or set of rules of use for the sysfs as to what goes where for such items as EDAC. After searching the web, articles and thinking about this for some time now, I am requesting comments on the sysfs model for where EDAC would fit best.

What exactly does EDAC do (and what does it stand for anyway?)

thanks,

greg k-h

#2 lore raw
Quoted 12 lines · Greg Kroah-Hartman
Quoted 11 lines · Doug Thompson

I am trying to design the sysfs interface tree for the new set of EDAC modules that are waiting for this interface, before being put into the kernel.

Currently the original EDAC (bluesmoke) has its own /proc directory (/proc/mc) with files and a directory (0,1,2,...)for each memory controller on the system. This will be removed and the new information interface will be placed in the sysfs.

Quoted 4 lines
Quoted 2 lines

One proposal is to place the information in /sys/devices/system in the following directories:

Why not use /sys/firmware/ instead?

Probably the same reason we don't have the cpufreq (for eg) stuff under /sys/firmware. Because it's poking hardware, not manipulating firmware.

/sys/devices/system makes a lot more sense, as thats where the cpu level machine check stuff is (amongst other similar things).

Quoted 8 lines
Quoted 6 lines

I have failed to date to really find a policy or set of rules of use for the sysfs as to what goes where for such items as EDAC. After searching the web, articles and thinking about this for some time now, I am requesting comments on the sysfs model for where EDAC would fit best.

What exactly does EDAC do (and what does it stand for anyway?)

Reports hardware events read from chipset specific registers. Similar to /sys/devices/system/machinecheck/, but from chipset instead of CPU. (That's grossly simplified, but hopefully gets the idea across).

Dave

#3 lore raw
Quoted 18 lines · Dave Jones
Quoted 12 lines · Greg Kroah-Hartman
Quoted 11 lines · Doug Thompson

I am trying to design the sysfs interface tree for the new set of EDAC modules that are waiting for this interface, before being put into the kernel.

Currently the original EDAC (bluesmoke) has its own /proc directory (/proc/mc) with files and a directory (0,1,2,...)for each memory controller on the system. This will be removed and the new information interface will be placed in the sysfs.

Quoted 4 lines
Quoted 2 lines

One proposal is to place the information in /sys/devices/system in the following directories:

Why not use /sys/firmware/ instead?

Quoted 7 lines

Probably the same reason we don't have the cpufreq (for eg) stuff under /sys/firmware. Because it's poking hardware, not manipulating firmware.

/sys/devices/system makes a lot more sense, as thats where the cpu level machine check stuff is (amongst other similar things).

Ok, that does make sense, thanks for explaining it.

greg k-h

#4 lore raw
In reply to Greg KH

What is EDAC and what does it stand for?

EDAC= Error Detection And Correction.

It currently is in the -mm2 tree, but with the older files and controls location.

The primary purpose of 'edac' (formerly blusmoke.sourceforge.net) is to provide a DETECTOR module of various errors detected by the hardware. The main detector has been the detecting of ECC memory errors of memory controllers. PCI Parity scanning was recently added.

Uncorrected ECC Errors (UE) are detected if the machinecheck is not configured in the kernel, since the machinecheck will occur synchronous with the error, edac polss. edac can log UEs and panic if the control is set for panic_on_ue. edac logs Corrected ECC Errors (CE) to the sys concole and it is output.

edac is a two component system. edac_mc is the core and then one of several memory controller (mc) modules is used as the mc driver. edac_k8 is the opteron/athlon64 modules. The mc driver can then extract the mc specific information and abstract it and send the information to the edac_mc core.

The current information file in /proc/mc/0 (for CPU 0's mc) is as follows:

Check PCI Parity:       1
Panic PCI Parity:       1
Panic UE:               1
Log UE:                 1
Log CE:                 1
Poll msec:              1000

MC Core: bluesmoke_mc Ver: 2.0.3 Nov
11 2005
MC Module: bluesmoke_k8 Ver: 2.0.2 Nov
11 2005

Memory Controller:      Athlon64/Opteron
PCI Bus ID:             0000:00:18.2 (0000:00:18.2)
EDAC capability:        None SECDED S4ECD4ED
Current EDAC capability:        None SECDED S4ECD4ED
Supported Mem Types:    Unbuffered-DDR Registered-DDR
0:H0_DIMM0|H0_DIMM1:Memory Size:        1024 MiB
0:H0_DIMM0|H0_DIMM1:Mem Type:           Registered-DDR
0:H0_DIMM0|H0_DIMM1:Dev Type:           x4
0:H0_DIMM0|H0_DIMM1:EDAC Mode:          S4ECD4ED
0:H0_DIMM0|H0_DIMM1:UE:                 0
0:H0_DIMM0|H0_DIMM1:CE:                 0

0.0:H0_DIMM0:CE: 0
0.1:H0_DIMM1:CE: 0

1:H0_DIMM0|H0_DIMM1:Memory Size:        1024 MiB
1:H0_DIMM0|H0_DIMM1:Mem Type:           Registered-DDR
1:H0_DIMM0|H0_DIMM1:Dev Type:           x4
1:H0_DIMM0|H0_DIMM1:EDAC Mode:          S4ECD4ED
1:H0_DIMM0|H0_DIMM1:UE:                 0
1:H0_DIMM0|H0_DIMM1:CE:                 0

1.0:H0_DIMM0:CE: 0
1.1:H0_DIMM1:CE: 0

Total Memory Size:      2048 MiB
Seconds since reset:    270160
UE No Info:             0
CE No Info:             0
Total UE:               0
Total CE:               0
Total PCI Parity:       0

<end paste>

Yes, this output is way too monolithic and needs to be refactored. I aim to move this output to the new sysfs destination, but in a different format via different files.

The output show UE and CE counts arranged by CSROW and by Channel 0 or Channel 1. These are also tagged with the motherboard silk screen labels (Arima's HDAMA mobo is the example: H0_DIM0, etc). This allows the adminstrator to harvest this data and map to the node AND DIMM slot for replacement.

The above also has PCI device parity scanning information, which will be broken out as mentioned in my original posting. We have found flaws in PCI riser cards with this scanning process, which caused unreported data corruption in high speed interconnects. BUT not all devices conform to the PCI spec on parity generation. (as usual)

Quoted 14 lines · --- Greg KH
Quoted 2 lines · Doug Thompson

I am trying to design the sysfs interface tree for

the

Quoted 4 lines

new set of EDAC modules that are waiting for this
interface, before being put into the kernel.

Currently the original EDAC (bluesmoke) has its

own

Quoted 1 line

/proc directory (/proc/mc) with files and a

directory

Quoted 1 line

(0,1,2,...)for each memory controller on the

system.

Quoted 8 lines
Quoted 1 line

This will be removed and the new information

interface

Quoted 4 lines

will be placed in the sysfs.

One proposal is to place the information in
/sys/devices/system in the following directories:

Why not use /sys/firmware/ instead?

I guess my initial explaination was not clear enough. This doesn't fit, and I assume you see this now from the above explaination.

Quoted 10 lines

Or do you want to use the struct device stuff

Quoted 1 line

For EDAC general memory ECC controls and

information

Quoted 3 lines

files:

/sys/devices/systems/edac/mc/

What kind of controls and files?

Abstracted mc files, taken from the above older monolithic output:

mc_core_version
mc_driver_version
memory_controller
device_bus_id (symlink)
edac_capability
current_edac_capability
supported_mem_types
seconds_since_counter_reset
total_memory_size
total_ue_noinfo_count
total_ce_noinfo_count
total_ue_count
total_ce_count

[the no_info counts are respective errors, but the edac mc drivers could not determine more information on it, hence a no_info count]

Controls:

panic_on_ue
log_ue
log_ce
poll_interval_msec

These controls are also set via module load options.

Quoted 8 lines
Quoted 6 lines

For PCI Parity Error detection controls and
information files:

/sys/devices/system/edac/pci

That kind of controls and files?

Controls:

check_pci_parity
panic_on_pci_parity

info files:

total_pci_parity_count

Quoted 6 lines
Quoted 3 lines

In addition /sys/devices/system/edac/mc/ would

have

Quoted 2 lines

directories:

Quoted 8 lines
Quoted 6 lines

mc0/
mc1/
...

for each memory controller's specific controls and information.

Again, what kind of controls and information?

For each Chip-Select Row (csrow) there would be information. I am still trying to determine if each csrow would be in its own directory or all cwrows just flat in the mc0, mc1, ... directories.

Assuming each csrow is in its own directory (which is the way I am leaning) below:

csrow0/
csrow1/
csrow2/
csrow3/
...

info files in the above directories:

memory_size
memory_type
device_type
edac_mode
ue_count
ce_count
ce_count_channel_0
ce_count_channel_1
dimm_label
dimm_label_channel_0
dimm_label_channel_1

controls:

none at this time

--------------

Quoted 1 line

From this data, a reaper/harvester can determine the

CE rate, which is the main real value in EDAC at this time, and notify the admin of preventative maintainance work.

Quoted 10 lines
Quoted 7 lines

Currently the similiar error detection device
/sys/devices/system/machinecheck resides here are
well.

The alternative layout would be to use the

/sys/class

Quoted 1 line

directory when nested-classes become available:

Quoted 7 lines

They are in 2.6.15-rc1, but you _really_ don't want to use them, they
are a huge pain, and I will be getting rid of them, along with all
struct class_device stuff in the near future. See the archives for
details, or it's summarized here:

interesting. That info is what I was looking for. BTW, thanks for starting the 'HOWTO do kernel development'

Quoted 2 lines

http://www.kroah.com/log/linux/driver_model_changes.html

Quoted 19 lines
Quoted 10 lines

/sys/class/edac/mc/...

and

/sys/class/edac/pci/...

But edac doesn't quite seem to fit here.

I agree.

Quoted 1 line

I have failed to date to really find a policy or

set

Quoted 1 line

of rules of use for the sysfs as to what goes

where

Quoted 2 lines

for such items as EDAC. After searching the web,
articles and thinking about this for some time

Quoted 7 lines

now, I

Quoted 1 line

am requesting comments on the sysfs model for

where

Quoted 1 line

EDAC would fit best.

What exactly does EDAC do (and what does it stand
for anyway?)

see beginning of this post

Quoted 5 lines

thanks,

greg k-h

thank you greg, for the specific questions to clarify my request.

doug t

"If you think Education is expensive, just try Ignorance"

"Don't tell people HOW to do things, tell them WHAT you want and they will surprise you with their ingenuity."
Gen George Patton

#5 lore raw
Quoted 10 lines · --- Doug Thompson

What is EDAC and what does it stand for?

EDAC= Error Detection And Correction.

Quoted 4 lines

For PCI Parity Error detection controls and
information files:

/sys/devices/system/edac/pci

That kind of controls and files?

I left out new PCI whitelist/blacklist control files I am also working on:

pci_parity_whitelist
pci_parity_blacklist

Since some unmentioned (and expensive) pci boards fail to conform with the PCI spec when dealing with PCI parity status reporting (or they just plain have bugs), the pci scanning feature needs a whitelist or a blacklist of "vendor_id:device_id" to specificly scan or not scan.

If there is a whitelist, no blacklist occurs. When a blacklist is written to, the whitelist is erased and devices on blacklist are skipped on the scanning.

format of info to write to these controls, in hex:

vendor_id:device_id[,vendor_id:device_id...]

I have timed both ECC scanning and PCI parity scanning.

ECC scanning on a dual opteron is 170 TSC clocks.

PCI Parity for 24 devices is 65000 TSC clocks. Ouch! When all devices are blacklisted, the iterator is 2700 TSC clocks.

doug t

"If you think Education is expensive, just try Ignorance"

"Don't tell people HOW to do things, tell them WHAT you want and they will surprise you with their ingenuity."
Gen George Patton

#6 lore raw
In reply to Doug Thompson
Quoted 16 lines · Doug Thompson

For each Chip-Select Row (csrow) there would be information. I am still trying to determine if each csrow would be in its own directory or all cwrows just flat in the mc0, mc1, ... directories.

Assuming each csrow is in its own directory (which is the way I am leaning) below:

csrow0/
csrow1/
csrow2/
csrow3/
...

info files in the above directories:

Quoted 12 lines

memory_size
memory_type
device_type
edac_mode
ue_count
ce_count
ce_count_channel_0
ce_count_channel_1
dimm_label
dimm_label_channel_0
dimm_label_channel_1

Ok, thanks for the details, it makes more sense now. Your heirachy seems sane, have you implemented it to see if it works properly?

thanks,

greg k-h

#7 lore raw
Quoted 34 lines · --- Greg KH
Quoted 2 lines · Doug Thompson

For each Chip-Select Row (csrow) there would be
information. I am still trying to determine if

each

Quoted 1 line

csrow would be in its own directory or all cwrows

just

Quoted 3 lines

flat in the mc0, mc1, ... directories.

Assuming each csrow is in its own directory (which

is

Quoted 10 lines

the way I am leaning) below:

csrow0/
csrow1/
csrow2/
csrow3/
...

info files in the above directories:

Quoted 12 lines

memory_size
memory_type
device_type
edac_mode
ue_count
ce_count
ce_count_channel_0
ce_count_channel_1
dimm_label
dimm_label_channel_0
dimm_label_channel_1

Quoted 4 lines

Ok, thanks for the details, it makes more sense now.
Your heirachy
seems sane, have you implemented it to see if it works properly?

I began implementing first in /sys/classes and that is when I ran into the nested class issue. I then looked at the /sys/devices/system interface point and then sought more information and then ASKED for RFC from the list.

I will now use the /sys/devices/system/edac as my root for my files and controls.

Speaking of controls, edac has them currently in /proc/sys/mc. I have proposed to have them in /sys/devices/system/edac/mc and friends.

My question is: Should I remove entirely my old /proc/sys/mc sysctl tree? Or still maintain the aliases there (which seems weird)?

If I do that, then /etc/sysctl.conf will no longer allow for setting things up there.

Is there going to be a similiar functionality as /etc/sysctl.conf for those items we place in sysfs, in the future?

thanks

doug t

PS. These questions on sysfs seem a perfect food stream for your 'HOWTO do kernel development'. Trying to do new entries in sysfs has been a painstaking adventure. After googling the web for info, it definitely has been a bit thin on information on sysfs at the level I am seeking.

Quoted 5 lines

thanks,

greg k-h

"If you think Education is expensive, just try Ignorance"

"Don't tell people HOW to do things, tell them WHAT you want and they will surprise you with their ingenuity."
Gen George Patton

#8 lore raw
Quoted 7 lines · Doug Thompson

Speaking of controls, edac has them currently in /proc/sys/mc. I have proposed to have them in /sys/devices/system/edac/mc and friends.

My question is: Should I remove entirely my old /proc/sys/mc sysctl tree? Or still maintain the aliases there (which seems weird)?

That would be wierd, just drop the sysctl stuff.

Quoted 2 lines

If I do that, then /etc/sysctl.conf will no longer allow for setting things up there.

True.

Quoted 3 lines

Is there going to be a similiar functionality as /etc/sysctl.conf for those items we place in sysfs, in the future?

If you want to write one, sure :)

But you can just probably use a udev rule to initialize your things properly, that's what all of the distros are now using.

Quoted 6 lines

PS. These questions on sysfs seem a perfect food stream for your 'HOWTO do kernel development'. Trying to do new entries in sysfs has been a painstaking adventure. After googling the web for info, it definitely has been a bit thin on information on sysfs at the level I am seeking.

sysfs and the driver model are woefully underdocumented. Right now, I'd recommend the Linux Device Drivers, third edition, free online if you don't want to buy it, for anyone doing any driver core stuff. It has a whole chapter that is the most up-to-date and the best description I've seen so far.

But even then, it is out of date, due to api changes, sorry.

thanks,

greg k-h

#9 lore raw
Quoted 12 lines · --- Greg KH
Quoted 3 lines · Doug Thompson

My question is: Should I remove entirely my old /proc/sys/mc sysctl tree? Or still maintain the aliases there (which seems weird)?

That would be wierd, just drop the sysctl stuff.

Quoted 2 lines

If I do that, then /etc/sysctl.conf will no longer allow for setting things up there.

Quoted 8 lines

True.

Quoted 2 lines

Is there going to be a similiar functionality as /etc/sysctl.conf for those items we place in

sysfs, in

Quoted 1 line

the future?

If you want to write one, sure :)

Good idea. Seems I have hit the edge of current features. That's is good for now there is another food item for a TODO list.

Quoted 5 lines

But you can just probably use a udev rule to
initialize your things
properly, that's what all of the distros are now
using.

Ok. That's another area for me to research. edac does not have any /dev/ entries, just the files and controls previous mentioned.

So, from your comment then, udev has some mechanism to set controls in sysfs?

Quoted 10 lines
Quoted 3 lines

PS. These questions on sysfs seem a perfect food stream for your 'HOWTO do kernel development'.

Trying

Quoted 3 lines

to do new entries in sysfs has been a painstaking adventure. After googling the web for info, it definitely has been a bit thin on information on

sysfs

Quoted 1 line

at the level I am seeking.

Quoted 9 lines

sysfs and the driver model are woefully underdocumented. Right now, I'd
recommend the Linux Device Drivers, third edition, free online if you
don't want to buy it, for anyone doing any driver core stuff. It has a
whole chapter that is the most up-to-date and the best description I've
seen so far.

Yes, I have LDD 3rd and it is good. I have also have Robert Love's book and it has some good stuff on the device model and sysfs, but I assume since the whole feature is fairly new its documentation and understanding are still in the nursery.

I also came across Patrick Mochel's paper given at Linux Symposium in June 2005. That helped.

Quoted 1 line

From the src on the machinecheck currently in sysfs, I

see it implements the 'subsystem' feature of sysfs.

Quoted 10 lines

From that code I see the pattern I can use.

But even then, it is out of date, due to api
changes, sorry.

thanks,

greg k-h

at least there is some docs AND some people to ask questions to.

thanks

doug t

"If you think Education is expensive, just try Ignorance"

"Don't tell people HOW to do things, tell them WHAT you want and they will surprise you with their ingenuity."
Gen George Patton

#10 lore raw
Quoted 11 lines · Doug Thompson
Quoted 4 lines

But you can just probably use a udev rule to
initialize your things
properly, that's what all of the distros are now
using.

Ok. That's another area for me to research. edac does not have any /dev/ entries, just the files and controls previous mentioned.

So, from your comment then, udev has some mechanism to set controls in sysfs?

udev gets called whenever you add a kobject to the system. You can then do whatever you want in udev when this happens. As an example, on one distro, when a bluetooth device is created by the kernel, a bluetooth startup script is run by udev.

thanks,

greg k-h

#11 lore raw
Quoted 10 lines · Greg KH
Quoted 9 lines · Doug Thompson
Quoted 4 lines

But you can just probably use a udev rule to
initialize your things
properly, that's what all of the distros are now
using.

Ok. That's another area for me to research. edac does not have any /dev/ entries, just the files and controls previous mentioned.

Quoted 7 lines
Quoted 2 lines

So, from your comment then, udev has some mechanism to set controls in sysfs?

udev gets called whenever you add a kobject to the system. You can then do whatever you want in udev when this happens. As an example, on one distro, when a bluetooth device is created by the kernel, a bluetooth startup script is run by udev.

We do things like this:
ACTION=="add", SUBSYSTEM="scsi", SYSFS{type}=="1", RUN+="/bin/sh -c 'echo 900 > /sys/$DEVPATH/timeout'"

There are only very few users now, that set values in sysfs. If that is a common need to change values with udev rules we may integrate that into udev itself, instead of calling a shell, but that works fine so far.

Kay