Decode dimms on dual socket machines

Hi all,

I have a Dual-Socket Machine with Supermicro Motherboard (X8DTH-iF) and I'd like to use decode-dimms, but it just decodes the modules on one socket (dmidecode -t 17 and ipmitool sdr show all 6 modules on both sockets).

Does anybody have a clue how to access those other 3 modules?

Regards,
Michael!

this is what I can see right now:
test24:/media/i2c-tools # ./tools/i2cdetect 0
WARNING! This program can confuse your I2C bus, cause data loss and worse! I will probe file /dev/i2c-0.
I will probe address range 0x03-0x77.
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- 08 -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- UU -- UU -- UU -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 2e 2f
30: 30 -- 32 -- 34 -- -- -- -- -- -- -- -- -- 3e --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: UU -- UU -- UU -- -- -- -- -- -- -- -- -- -- --
60: -- 61 62 -- -- -- -- -- -- 69 -- -- -- -- -- --
70: 70 -- -- -- -- -- -- --

18,1a,1c: jc42 temperature sensors
50,52,54: eeproms

test24:~ # ll /sys/bus/i2c/devices/
total 0
lrwxrwxrwx 1 root root 0 Apr 10 13:45 0-0018 -> ../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-0018 lrwxrwxrwx 1 root root 0 Apr 10 13:45 0-001a -> ../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-001a lrwxrwxrwx 1 root root 0 Apr 10 13:45 0-001c -> ../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-001c lrwxrwxrwx 1 root root 0 Apr 10 13:48 0-0050 -> ../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-0050 lrwxrwxrwx 1 root root 0 Apr 10 13:48 0-0052 -> ../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-0052 lrwxrwxrwx 1 root root 0 Apr 10 13:48 0-0054 -> ../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-0054 lrwxrwxrwx 1 root root 0 Apr 7 17:12 i2c-0 -> ../../../devices/pci0000:00/0000:00:1f.3/i2c-0

Hallo Michael,

Odds are that your SMBus is multiplexed and the other memory sockets are connected to a different bus segment. Unfortunately there are various ways to achieve multiplexing, this can't be detected automatically and it is rarely documented in the board manual. So you will have to ask the vendor for technical details on this point. Multiplexing can be done through GPIO pins (either on the south bridge or the Super-I/O chip) or with a chip on the SMBus itself.

Note that you will need a recent enough kernel (>= 2.6.36) to have support for SMBus multiplexing.

FWIW, address 0x70 is used by the PCA954x multiplexers, so maybe this is what you have (and a driver is available).

At 0x2f is probably a W83793 or W83795 monitoring chip, supported by the w83793 and w83795 drivers, respectively. Only use these dedicated drivers if you do not intend to use the IPMI features of the board.

Jean Delvare
http://khali.linux-fr.org/wishlist.html

On 04/12/2011 01:26 PM, Jean Delvare wrote:
Hi all,

currently I use OpenSUSE11.4 with 2.6.37.1-1.2-default

Now I loaded the module

test24:/media/i2c-tools/tools # lsmod |grep pca

pca954x                 3288  0
i2c_mux                 2672  1 pca954x

test24:/media/i2c-tools/tools # ll /sys/bus/i2c/devices/
total 0
lrwxrwxrwx 1 root root 0 Apr 12 14:26 0-0050 -> ../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-0050 lrwxrwxrwx 1 root root 0 Apr 12 14:26 0-0052 -> ../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-0052 lrwxrwxrwx 1 root root 0 Apr 12 14:26 0-0054 -> ../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-0054 lrwxrwxrwx 1 root root 0 Apr 12 13:54 i2c-0 -> ../../../devices/pci0000:00/0000:00:1f.3/i2c-0

Seems like this is what I need to know, but I don't know how to interpret it. Is it multiplexed on the GPIO Pin 52 and 53 at the W83795ADG?

ftp://ftp.supermicro.com/utility/SuperoDoctorINI/AllMEMDIMM.ini

Regards,
Michael!

This is OK then.

Loading the pca954x driver doesn't do anything. It needs an instantiated device with proper platform data before it will do anything. This can only be done from within the kernel, using some sort of platform initialization code.

Indeed, good finding. This shows that the vendor is using GPIOs to switch between the SMBus segments. This rules out the chip at I2C address 0x70. Unfortunately the document doesn't say of which chip the GPIO 52 and 53 are used. It's not even clear if "52" means "pin 52" or "GPIO group 5, pin 2". I'm certain it's not related to the W83795ADG though, as this chip has only 48 pins and 4 GPIO pins max.

My first suspect would be the Intel ICH south bridge. The ICH10 has GPIO pins numbered 52 and 53. I don't think there is any driver for these GPIO pins though, so one will have to be written first. Only then, board-specific glue code can be written to link between this new driver, i2c-i801 and gpio-i2cmux.

Jean Delvare
http://khali.linux-fr.org/wishlist.html

Jean Delvare said the following:

I would expect a
#> echo "<deviceName> 0x70" > /sys/bus/i2c/devices/i2c-0/new_device to span new buses, their numbers taken by random, even without platform data (not tried by myself yet, but will have to do it soon). The challenge will be to detect which of the 8 supported devices ( pca_9540, pca_9542, pca_9543, pca_9544, pca_9545, pca_9546, pca_9547, pca_9548) is actually to be used.

JM2C

KR
Michael

Hmm, you're right, my memory that we has made platform data mandatory was wrong, sorry.

In Michael's case it is irrelevant anyway, as the information he found meanwhile clearly indicates that bus multiplexing is achieved by GPIOs (most certainly on the ICH10) and not an PCA954x chip. The chip at 0x70 is probably the Intel 5500 IOH.

Jean Delvare

Supermicro just gave me the following information:

To access all SPDs, you have to pull low GPIO49 (IO_0x528 bit 17) first. Then pull low GPIO52 (IO_0x528 bit 20) to access first CPU memory(0x50~0x55); pull high GPIO52 to access 2nd CPU memory(0x50~0x55).

I can't find anything about GPIO49/52,53 in Datasheet for 5500/5520 chipset
http://www.intel.com/assets/pdf/datasheet/321328.pdf

So it is probably connected to ICH10
http://www.intel.com/Assets/PDF/datasheet/319973.pdf

but I don't know how to use this information

Regards,
Michael!

Hallo Michael,

I am surprised because AllMEMDIMM.ini suggests that GPIO 53 and 52 are used, not 47 and 52.

My own workstation board happens to have a similar (but slightly different) GPIO-based SMBus multiplexing scheme. I wanted to look into it but could never find the time. Your request gave me an opportunity to look into it again. As a first step, I've improved isadump and isaset to support 32-bit I/O access:
http://marc.info/?l=lm-sensors&m=130264185604706&w=2
http://marc.info/?l=lm-sensors&m=130264189004750&w=2

With these two patches applied, you should be able to control the multiplexer from user-space. First of all, check the base I/O address of the GPIO registers, because the values provided by Supermicro are off by 16 compared to what I have:

# setpci -s 00:1f.0 48.l

It returns 00000501 for me, for I/O base 0x500. If it is the same for you, you can dump the GPIO registers with:

# isadump -f -L 0x500 64

Preliminary note: wrong setup of GPIO could have disastrous effects on your hardware. If at any point you are unsure of what you're doing, you better just stop and ask.

The operations described by Supermicro would then be achieved with the following commands:

: Pull GPIO49 low

# isaset -f -L 0x538 0x00000000 0x00020000
: Pull GPIO52 low: 
# isaset -f -L 0x538 0x00000000 0x00100000
: Pull GPIO52 up: 
# isaset -f -L 0x538 0x00100000 0x00100000

Note the 0x538 when your Supermicro contact said 0x528. 0x528 would be correct only if the I/O base is 0x4f0 instead of 0x500... but I'd be surprised if it is the case.

Of course this is all a hack and the kernel drivers won't notice the multiplexer change happening in their backs. So if you happen to have the same number of memory modules for both CPUs, plugged in symmetrical sockets, "sensors" (and "decode-dimms") will nicely report the values from the currently selected bus segment, but you can't see all modules at once. For this, proper multiplexing support in the kernel will be needed, which is a lot more work. Patience...

Jean Delvare
http://khali.linux-fr.org/wishlist.html

Hi Michael,

I've written a kernel driver to gain access to the GPIO pins of the ICH10 (and previous ICH chips too.) It's available here:
http://khali.linux-fr.org/devel/misc/i801_gpio/

Generic instructions for standalone drivers are here:
http://khali.linux-fr.org/devel/misc/INSTALL

I would appreciate if you could give it a try on your system and report everything in the kernel log after loading the driver. By default it won't change the GPIO values so it should be totally safe.

Thanks,

Jean Delvare
http://khali.linux-fr.org/wishlist.html

Jean Delvare said the following:

Well, if he implements a multiplexer code for it, he is done :-)

KR
Michael