Skip to contentSkip to conversation

amd64_edac: add module registration routines

PATCH Doug Thompson 1 participant 1 message May 8, 2009

lore
Share & tools
lore
#0 lore raw
Quoted 35 lines · Ingo Molnar

From: Ingo Molnar <[email protected]>
Subject: Re: [PATCH 21/21] amd64_edac: add module registration routines
To: "Borislav Petkov" <[email protected]>
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], "Doug Thompson" <[email protected]>
Date: Thursday, May 7, 2009, 3:58 PM

Quoted 1 line · Borislav Petkov

+      Recent Opterons (Family 10h

and later) provide for Memory Error

Quoted 1 line

+      Injection into the ECC

detection circuits. The amd64_edac module

Quoted 1 line

+      allows the operator/user to

inject Uncorrectable and Correctable

Quoted 3 lines

+      errors into DRAM.
+
+      When enabled, in each of

the respective memory controller directories

Quoted 1 line

+

(/sys/devices/system/edac/mc/mcX), there are 3 input files:

Quoted 2 lines

+
+      - z_inject_section (0..3,

16-byte section of 64-byte cacheline),

Quoted 1 line

+      - z_inject_word (0..8,

16-bit word of 16-byte section),

Quoted 1 line

+      - z_inject_bit_map (hex

bitmap vector: mask bits of 16 bit word to

Quoted 3 lines

+        error-out)
+
+      In addition, there are two

control files, z_inject_read and

Quoted 1 line

+      z_inject_write, which

trigger the Read and Write errors respectively.

Quoted 9 lines

I think the file names should follow existing EDAC driver practices,
and be named according to the existing inject_data_* pattern?

There's nothing more annoying to users (and tools) than inconsistent
driver namespaces.

originally the injection was enabled only for development debug, but
was changed to a .config switch. The 'z-*' use was to sort the directory list with those debug files to the bottom of the list.

So now, yes they should have the 'z-*' prefix removed.

Quoted 13 lines
Quoted 1 line

+    /* Exam the capabilities of the

northbridge in order to reflect them

Quoted 1 line

+     * in the presentation

via sysfs attributes, etc

Quoted 1 line

+     */

please use the customary comment style:

  /*
   * Comment .....
   * ...... goes here:
   */

Quoted 5 lines

also described in Documentation/CodingStyle.

This is a continuing observation for a number of other cases where
you introduce half-winged comments.

yeah, we missed some of those

Quoted 33 lines
Quoted 2 lines

+
+    /* IMPORTANT: Set the polling

'check' function in this module */

Quoted 4 lines

+    mci->edac_check = amd64_check;
+
+    /* memory scrubber interface */
+    mci->set_sdram_scrub_rate =

amd64_set_scrub_rate;

Quoted 1 line

+    mci->get_sdram_scrub_rate =

amd64_get_scrub_rate;

Review

Quoted 6 lines
(quoted diff) +6 -0
+}
+
+/*
+ * amd64_probe_one_instance
+ *
+ *    probe function to determine if there

is a DRAM Controller device is

Quoted 1 line

+ *    present and to construct data tables

for it.

Quoted 2 lines

+ *
+ *    Due to a hardware feature on Family

10H cpus, the Enable Extended

Quoted 1 line

+ *    Configuration Space feature MUST be

enabled on ALL Processors prior to

Quoted 1 line

+ *    actually reading from the ECS

registers. Since the loading of the module

Quoted 1 line

+ *    can occur on any 'core', and cores

don't 'see' all the other processors

Quoted 1 line

+ *    ECS data when the others are NOT

enabled. Our solution is to first

Quoted 1 line

+ *    enable ECS access in this routine on

all processors, gather some data in

Quoted 1 line

+ *    a amd64_pvt structure and later come

Quoted 9 lines

back in a 'finishup_setup' function

Quoted 1 line

+ *    to perform that final

initialization.

Quoted 3 lines

+ *
+ *    See also amd64_init_2nd_stage().
+ */

Nice comment!

IBM guys created that one with their patch after they found the 'issue'

Quoted 25 lines
Quoted 1 line

+static int amd64_probe_one_instance(struct pci_dev

*dram_f2_ctl,

Quoted 1 line

+

            int
mc_type_index)

Review

Quoted 6 lines
(quoted diff) +6 -0
+{
+    struct amd64_pvt *pvt;
+    int err;
+    int rc;
+
+    pvt = kzalloc(sizeof(struct

amd64_pvt), GFP_KERNEL);

Quoted 4 lines

+    if (pvt == NULL) {
+        rc = -ENOMEM;
+        goto exit_now;
+    }

The customary pattern is:

    ret = -ENOMEM;
    pvt = kzalloc(sizeof(struct amd64_pvt),
GFP_KERNEL);
    if (!pvt)
        goto err;

Quoted 7 lines

Note the four differences:

- use 'ret' instead of 'rc' for integer returns
- pre-initialize 'ret', not in the exception branch
- !pvt instead of 'pvt == NULL'
- use 'err*' naming for error labels

ok, thanks

Quoted 6 lines
Quoted 2 lines

+    pvt->mc_node_id =

get_mc_node_id_from_pdev(dram_f2_ctl);

Quoted 2 lines

+
+    debugf0("=========== %s(Instance=

%d) ===========\n",

Quoted 8 lines
Quoted 1 line

+        __func__,

pvt->mc_node_id);

please use something like trace_printk() instead. That will allow
high-performance tracing of this code, should the need arise. Also
'debugf0' says nothing - what does it mean?

debugfX() is a set of macros in the EDAC CORE (already in the kernel) that has 4 levels of verbosity, X=(0..3). Increase the number increases the output. This output is only enabled via the DEBUG switch. In normal operation, DEBUG should be off. It accesses a debug variable in sysfs which allows the developer to increase error reporting granularity.

Quoted 9 lines
Quoted 2 lines

+
+    rc =

amd64_check_ecc_enabled(pvt);

Quoted 2 lines

+    if (rc)
+        goto

exit_release_devices;

Bug: in the error case there's now a memory leak of 'pvt'.

ow, yeah. Thanks for that catch

Quoted 24 lines
Quoted 3 lines

+
+    /*
+     * Save the pointer to

the private data for use in 2nd initialization

Quoted 3 lines

+     * stage
+     */
+    pvt_lookup[pvt->mc_node_id] =

pvt;

Quoted 2 lines

+
+    debugf0("%s(): init 1st stage done

pvt-%d\n", __func__,

Quoted 1 line

+

pvt->mc_node_id);

Review

Quoted 6 lines

+    return 0;

(quoted diff) +5 -0
+
+
+exit_release_devices:
+    pci_dev_put(pvt->addr_f1_ctl);
+    pci_dev_put(pvt->misc_f3_ctl);

this teardown is correct but a bit unclean: it open-codes the
reverse direction of:

Quoted 7 lines

    err = amd64_reserve_mc_sibling_devices(pvt, mc_type_index);

It would be better to add a
amd64_free_mc_sibling_devices(pvt)
method.

ah, good point

Quoted 10 lines
Quoted 3 lines

+
+    /*
+     * transfer the info

from the interium pvt area to the private area of

Quoted 1 line

+     * the MC instance

structure

Quoted 3 lines

+     */
+    pvt = mci->pvt_info;
+    *pvt = *pvt_temp;

Quoted 5 lines

hm, such copying can be dangerous. If pvt_temp ever grows something
like a list_head, then the copy can corrupt the list. Could this be
avoided?

Yeah, it was a misgiving to me as well. It came from the 1st/2nd stage patch set. Needed to 'save' various harvested information during the 1st stage (after all the CPU's had been setup).

The constructor of the MCI object (in EDAC CORE) is a single memory allocated area to hold all the info, but it was dependent on the how many CSROWS and CHANNELS there were to save memory. So the actual 'private' area is within the bounds of the MCI allocation. The temporary allocation was a temp storage area until the MCI structure was finally allocated, which occurs in the 2nd stage pass.

The 'private' area is defined differently in each of the various EDAC modules. The amd64 module definitely has an involved twist on initialization.

Quoted 46 lines
Quoted 2 lines

+
+

kfree(pvt_lookup[pvt->mc_node_id]);

Review

Quoted 13 lines

+    pvt_lookup[node_id] = NULL;

(quoted diff) +12 -0
+
+    return rc;
+}
+
+
+/*
+ * amd64_init_one_instance
+ *
+ *    initialize just one device
+ *
+ *    returns:
+ *   

     count (>= 0), or

Quoted 1 line

+ *        negative on

error

Quoted 2 lines

+ */
+static int __devinit amd64_init_one_instance(struct

pci_dev *pdev,

Quoted 1 line

+

         const
struct pci_device_id *mc_type)

Review

Quoted 4 lines
(quoted diff) +4 -0
+{
+    int rc;
+
+    debugf0("%s(MC

node=%d,mc_type='%s')\n",

Quoted 2 lines

+        __func__,
+

get_mc_node_id_from_pdev(pdev),

Quoted 1 line

+

get_amd_family_name(mc_type->driver_data));

Quoted 7 lines

+
+    /* wake up and enable device */
+    rc = pci_enable_device(pdev);
+    if (rc < 0)
+        rc = -EIO;
+    else
+        rc =

amd64_probe_one_instance(pdev, mc_type->driver_data);

Quoted 3 lines

+
+    if (rc < 0)
+        debugf0("%s()

Quoted 7 lines

rc=%d\n", __func__, rc);

there's way too many debugf*() calls in the whole code, often
obscuring the real structure. I think we'll be far better off with
having just a few key ones.

Probably, but the reverse mapping of the amd64 uses many data fields. I added so many debug output to harvest that data so as to debug the code. I added some lines to be able to partition the output to isolate various data output in order to "see" it faster in the log.

But there ARE some lines that are remnants that could be removed.

Quoted 139 lines

Review

Quoted 7 lines
(quoted diff) +7 -0
+};
+
+
+/*
+ * amd64_setup_pci_device
+ *
+ *    setup the PCI Device Driver for

monitoring PCI errors

Review

Quoted 7 lines

+ */

(quoted diff) +6 -0
+static void amd64_setup_pci_device(void)
+{
+    struct mem_ctl_info *mci;
+    struct amd64_pvt *pvt;
+
+    if (!amd64_ctl_pci) {

this branch goes on:

Quoted 2 lines

+
+        mci =

mci_lookup[0];

Quoted 2 lines

+        if (mci) {
+

    debugf1("%s(): Registering ONE PCI
control\n",

Quoted 1 line

+

        __func__);

and on:

Quoted 2 lines

+
+

    pvt = mci->pvt_info;

Quoted 1 line

+

    amd64_ctl_pci =
edac_pci_create_generic_ctl(

Quoted 1 line

+


    &pvt->dram_f2_ctl->dev,

Quoted 1 line

+


    EDAC_MOD_STR);

Quoted 1 line

+

    if (!amd64_ctl_pci) {

and on:

Quoted 1 line

+

        printk(KERN_WARNING

Quoted 1 line

+


"%s(): Unable to create PCI control\n",

Quoted 1 line

+


__func__);

And here you have too many tabs (five of them), that break up that
printk in an ugly way.

The better solution is to realize that the branch could be inverted
and a full level of indentation won by doing:

    if (amd64_ctl_pci)
        return;

And winning another level of indentation later on by doing:

    if (mci) {
        debugf1("%s(): ONE
PCI control already registered\n",

    __func__);
        return;
    }

Btw., please change all instances of:

   printk(KERN_WARNING,
   => pr_warning(
   printk(KERN_INFO,
    => pr_info(

etc.

Quoted 1 line

+

        printk(KERN_WARNING

Quoted 1 line

+


"%s(): PCI error report via EDAC "

Quoted 1 line

+


"not setup\n",

Quoted 1 line

+


__func__);

Quoted 1 line

+

    }

Quoted 2 lines

+        } else {
+

    debugf1("%s(): ONE PCI control already
registered\n",

Quoted 1 line

+

        __func__);

Review

Quoted 10 lines

+        }
+    }

(quoted diff) +8 -0
+}
+
+static int __init amd64_edac_init(void)
+{
+    int err;
+    int node;
+
+    edac_printk(KERN_INFO,

EDAC_MOD_STR, EDAC_AMD64_VERSION "\n");

Review

Quoted 4 lines
(quoted diff) +4 -0
+
+    opstate_init();
+
+    debugf0("%s()

******************  ENTRY
**********************\n",

Quoted 1 line

+        __func__);

if you want a debug facility, please shorten it to something like:

    dbg(" **** ENTRY ****\n")

and make the printing of __func__ implicit in the dbg() method.
This will shorten these things quite a bit. But please get rid of
most of them - there's no excuse for having such things in the
kernel beyond the first 1-2 days of the development of this

Quoted 2 lines

function.

Good idea to fold the __func__ into the macro.

For each CPU socket this gets called and I used these to isolate each instance in a 2, 4 or 8 socket based system True, used for ease of debugging

Quoted 20 lines
Quoted 1 line

+    /* Attempt to register drivers for

instances

Quoted 1 line

+     * DUE to the failure of

some 'cores' to access Extended Config Space

Quoted 1 line

+     * prior to all memory

controllers having their ECS register enabled,

Quoted 1 line

+     * the initialization

has been created into 2 stages. Here we

Quoted 1 line

+     * call for the 1st

stage. After all have been enabled, then we

Quoted 1 line

+     * do the 2nd stage to

finishup setup.

Quoted 2 lines

+     */
+    err =

pci_register_driver(&amd64_pci_driver);

Quoted 2 lines

+
+    /* At this point, the array

'pvt_lookup[]' contains pointers to

Quoted 1 line

+     * allocated struct

amd64_pvt control structures. These will be used

Quoted 8 lines
Quoted 1 line

+     * in the 2nd stage init

function to finish initialization of

Quoted 2 lines

+     * the MC instances.
+     */

after using good comment style for a while, bad comment style shows
up again.

yeah, again, didn't get all of them yet

Quoted 6 lines
Quoted 3 lines

+
+    /* if no error occurred on first

pass init, then do 2nd pass init */

Quoted 1 line

+    if (!err) {

Quoted 9 lines

Again, by doing:

    if (err)
        goto err;

You can save an identation level below, and avoid line-wrap

artifacts.

ok

Quoted 127 lines
Quoted 1 line

+

for_each_online_node(node) {

Quoted 1 line

+

    if (!pvt_lookup[node])

Quoted 1 line

+

        continue;

Quoted 2 lines

+
+

    /* If any failure then need to clean up
*/

Quoted 1 line

+

    err =
amd64_init_2nd_stage(pvt_lookup[node]);

Quoted 1 line

+

    if (err) {

Quoted 1 line

+

        debugf0("%s()
'finish_setup' stage failed\n",

Quoted 1 line

+


__func__);

Quoted 2 lines

+
+

        /* undo prior
instances' registrations

Quoted 1 line

+

         * and leave
as failed

Quoted 1 line

+

         */

Quoted 1 line

+


pci_unregister_driver(&amd64_pci_driver);

Quoted 1 line

+

        goto error_exit;

Quoted 1 line

+

    }

Quoted 2 lines

+        }
+

amd64_setup_pci_device();

Review

Quoted 10 lines

+    }

(quoted diff) +9 -0
+
+error_exit:
+    return err;
+}
+
+static void __exit amd64_edac_exit(void)
+{
+    if (amd64_ctl_pci)
+       

edac_pci_release_generic_ctl(amd64_ctl_pci);

Quoted 2 lines

+
+

pci_unregister_driver(&amd64_pci_driver);

Review

Quoted 8 lines
(quoted diff) +8 -0
+}
+
+module_init(amd64_edac_init);
+module_exit(amd64_edac_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("SoftwareBitMaker: Doug Thompson, "
+        "Dave Peterson,

Thayne Harbaugh");

Quoted 1 line

+MODULE_DESCRIPTION("MC support for AMD64 memory

controllers - "

Quoted 1 line

+

EDAC_AMD64_VERSION);

Review

Quoted 3 lines
(quoted diff) +3 -0
+
+module_param(edac_op_state, int, 0444);
+MODULE_PARM_DESC(edac_op_state, "EDAC Error Reporting

state: 0=Poll,1=NMI");

Review

Quoted 1 line
(quoted diff) +0 -0

b/drivers/edac/amd64_edac.h

drivers/edac/amd64_edac.h

Quoted 10 lines

index d435ace..a3603f7 100644

drivers/edac/amd64_edac.h +0 -0
@@ -891,6 +891,11 @@ extern const char *ii_msgs[4];
   extern const char *ext_msgs[32];
   extern const char *htlink_msgs[8];

drivers/edac/amd64_edac.h +3 -0
+#define NUM_DBG_ATTRS  9
+#define NUM_INJ_ATTRS  5
+extern struct mcidev_sysfs_attribute

amd64_dbg_attrs[NUM_DBG_ATTRS],

Quoted 1 line

+


   amd64_inj_attrs[NUM_INJ_ATTRS];

Quoted 3 lines

+
  /*
   * Each of the PCI Device IDs types

have their own set of hardware

Quoted 1 line

   * accessor function and per device

encoding/decoding logic.

Review

Quoted 1 line
(quoted diff) +0 -0

b/drivers/edac/amd64_edac_dbg.c

drivers/edac/amd64_edac_dbg.c static ssize_t

Quoted 4 lines

index 3741af9..f538129 100644

drivers/edac/amd64_edac_dbg.c +0 -0
@@ -211,6 +211,9 @@ static ssize_t

amd64_hole_show(struct mem_ctl_info *mci, char *data)

Quoted 1 line


     hole_size);

Quoted 6 lines

  }

+/*
+ * update NUM_DBG_ATTRS in case you add new members
+ */
  struct mcidev_sysfs_attribute amd64_dbg_attrs[]

= {

Review

Quoted 3 lines


      {

(quoted diff) +0 -0

b/drivers/edac/amd64_edac_inj.c

drivers/edac/amd64_edac_inj.c static ssize_t

Quoted 4 lines

index 7f978cb..8706954 100644

drivers/edac/amd64_edac_inj.c +0 -0
@@ -155,6 +155,9 @@ static ssize_t

amd64_inject_write_store(struct mem_ctl_info *mci,

Quoted 7 lines
drivers/edac/amd64_edac_inj.c +0 -0
       return 0;
   }


+/*
+ * update NUM_INJ_ATTRS in case you add new members
+ */
  struct mcidev_sysfs_attribute amd64_inj_attrs[]

= {

Quoted 9 lines
Quoted 2 lines


      {

Looks like these fixlets dont really belong in this patch and should
either be backmerged or put into a separate patch?

    Ingo

thanks for the diligence Ingo