Skip to contentSkip to conversation

Opteron box and 4Gb memory

DISCUSSION J.A. Magallon 6 participants 11 messages Oct 25, 2007 – Nov 8, 2007

lore
Share & tools
lore
#0 lore raw

Hi...

I have some Quad-Opteron boxes with 4Gb memory and two of them are running two different Linux distros.

Box one sees 4Gb of memory, but box two just sees 3. Their mtrr setups are different:

one:

[    0.000000] Linux version 2.6.20...
...
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
[    0.000000]  BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
[    0.000000]  BIOS-e820: 00000000000e6000 - 0000000000100000 (reserved)
[    0.000000]  BIOS-e820: 0000000000100000 - 00000000b7fd0000 (usable)
[    0.000000]  BIOS-e820: 00000000b7fd0000 - 00000000b7fde000 (ACPI data)
[    0.000000]  BIOS-e820: 00000000b7fde000 - 00000000b8000000 (ACPI NVS)
[    0.000000]  BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
[    0.000000]  BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
[    0.000000]  BIOS-e820: 00000000ff780000 - 0000000100000000 (reserved)
[    0.000000]  BIOS-e820: 0000000100000000 - 0000000145000000 (usable)
[    0.000000] Entering add_active_range(0, 0, 159) 0 entries of 3200 used
[    0.000000] Entering add_active_range(0, 256, 753616) 1 entries of 3200 used
[    0.000000] Entering add_active_range(0, 1048576, 1331200) 2 entries of 3200 
used
[    0.000000] end_pfn_map = 1331200

cat /proc/mtrr:
reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg02: base=0x140000000 (5120MB), size= 64MB: write-back, count=1
reg03: base=0x144000000 (5184MB), size= 16MB: write-back, count=1
reg04: base=0xb8000000 (2944MB), size= 128MB: uncachable, count=1
reg05: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1

two:
Linux version 2.6.22.9...
...
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000e6000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 00000000b8fd0000 (usable)
BIOS-e820: 00000000b8fd0000 - 00000000b8fde000 (ACPI data)
BIOS-e820: 00000000b8fde000 - 00000000b9000000 (ACPI NVS)
BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
BIOS-e820: 00000000ff780000 - 0000000100000000 (reserved)
Entering add_active_range(0, 0, 159) 0 entries of 3200 used Entering add_active_range(0, 256, 757712) 1 entries of 3200 used end_pfn_map = 1048576

cicely:~# cat /proc/mtrr
reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
reg01: base=0x80000000 (2048MB), size= 512MB: write-back, count=1
reg02: base=0xa0000000 (2560MB), size= 256MB: write-back, count=1
reg03: base=0xb0000000 (2816MB), size= 128MB: write-back, count=1
reg04: base=0xb8000000 (2944MB), size=  16MB: write-back, count=1

Why ? Is it a bios setup problem ? A kernel problem ? grep HIGHMEN in configs for both kernels does not give anything, so I still understand less this thing...

Signature · 3 lines
J.A. Magallon <magallon()unizar!es>     \               Software is like sex:
                                         \         It's better when it's free
MacOS X 10.4.8 Tiger - Darwin Kernel Version 8.8.0 PPC
#1 lore raw
Quoted 5 lines · J.A. Magallon

Hi...

I have some Quad-Opteron boxes with 4Gb memory and two of them are running two different Linux distros.

Quoted 7 lines

Box one sees 4Gb of memory, but box two just sees 3. Their mtrr setups are different:

Why ? Is it a bios setup problem ? A kernel problem ? grep HIGHMEN in configs for both kernels does not give anything, so I still understand less this thing...

It would depend on how the BIOS programmed the memory controllers. For 32-bit (and lots of device) compatibility, a memory hole is required below 4 GB. Not all memory controllers can remap memory in the 3-4 GB range above the 4 GB memory; I'm not sure if that varies with the different Opteron processors.

Also, if you run a 32-bit distribution, you need to have HIGHMEM_64G enabled in the kernel.

-hpa

#2 lore raw
Quoted 14 lines · H. Peter Anvin
Quoted 5 lines · J.A. Magallon

Hi...

I have some Quad-Opteron boxes with 4Gb memory and two of them are running two different Linux distros.

Quoted 7 lines

Box one sees 4Gb of memory, but box two just sees 3. Their mtrr setups are different:

Why ? Is it a bios setup problem ? A kernel problem ? grep HIGHMEN in configs for both kernels does not give anything, so I still understand less this thing...

Quoted 5 lines

It would depend on how the BIOS programmed the memory controllers. For 32-bit (and lots of device) compatibility, a memory hole is required below 4 GB. Not all memory controllers can remap memory in the 3-4 GB range above the 4 GB memory; I'm not sure if that varies with the different Opteron processors.

It shouldn't, AFAICS.

Greetings,
Rafael

#3 lore raw
In reply to H. Peter Anvin
Quoted 14 lines · H. Peter Anvin
Quoted 5 lines · J.A. Magallon

Hi...

I have some Quad-Opteron boxes with 4Gb memory and two of them are running two different Linux distros.

Quoted 7 lines

Box one sees 4Gb of memory, but box two just sees 3. Their mtrr setups are different:

Why ? Is it a bios setup problem ? A kernel problem ? grep HIGHMEN in configs for both kernels does not give anything, so I still understand less this thing...

Quoted 5 lines

It would depend on how the BIOS programmed the memory controllers. For 32-bit (and lots of device) compatibility, a memory hole is required below 4 GB. Not all memory controllers can remap memory in the 3-4 GB range above the 4 GB memory; I'm not sure if that varies with the different Opteron processors.

I have collected several pieces of info around the internet...

- Some people uses this options in the BIOS:
Node interleave: off
Bank interleave: auto
SW memory hole: disable
HW memory hole: enable
MTRR: Continuous

- Node Memory Interleaving DISABLES NUMA and generally is a bad thing

- MTRR setting -should be set to "discrete" for Linux, and probably for Windows too.

- This is what SuperMicro's tech support said about 2.96GB vs. 4GB.

"This is as expected, as soon as you set "software memory hole" to disabled, you also disable option ROM remapping functionality, this option normally remaps used option rom (option rom= raid bios, lan pxe ; usb legacy, bioses on add-on cards, etc) in the 4GB region, so no basis memory is lost, while this feature is now disabled the option rom space occupies the space between 3 and 4 GB which results in lower main memory availability. There is no solution or work around for this phenomenon"

so software memory hole enabled might be needed to get all 4GB to show up

Quoted 1 line

From mobo manual:

Software Memory Hole
When "Enabled", allows software memory remapping around the memory hole. Options are Enabled and Disabled.

Hardware Memory Hole
When "Enabled", allows software memory remapping around the memory hole. Options are Enabled and Disabled. Note: this is only supported by Rev E0 processors and above.
( I have two Opteron 275 processors, no idea about revision)

So _my_ conclussion is:

Node interleave: off (numa mode)
Bank interleave: auto
SW memory hole: disable |
HW memory hole: enable | allow remapping
MTRR: Discrete |

But then, do I need to enable NUMA options in the kernel ?

Quoted 4 lines

Also, if you run a 32-bit distribution, you need to have HIGHMEM_64G enabled in the kernel.

I run a 64 bit one, then I don't need anything, isn't it ? That's why I don't see any _HIGHMEM in the kernel configs...

Some day I will understand this crappy BIOS thing (or burn a photo of its inventor...).
Why can't we have OpenFirmware PC's, like my MacPro and Sparcs ?

--
J.A. Magallon <jamagallon()ono!com> \ Software is like sex:
\ It's better when it's free
Mandriva Linux release 2008.1 (Cooker) for i586
Linux 2.6.23-jam01 (gcc 4.2.2 20070909 (4.2.2-0.RC.1mdv2008.0)) SMP PREEMPT 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0

#4 lore raw
Quoted 9 lines · J.A. Magallón

Software Memory Hole
When "Enabled", allows software memory remapping around the memory hole. Options are Enabled and Disabled.

Hardware Memory Hole
When "Enabled", allows software memory remapping around the memory hole. Options are Enabled and Disabled. Note: this is only supported by Rev E0 processors and above.
( I have two Opteron 275 processors, no idea about revision)

The configuration register used to to reclaim DRAM lost to an MMIO hole was introduced with revision E of the gen1 Opterons. (This feature is supposed to work both in interleaved and non-interleaved mode.) What does /proc/cpuinfo say? (On both?)

(Still, even without this your BIOS should be able to map your memory so that you are able to use all 4G. Provided you disable interleaving, I can't see that there's anything stopping the BIOS from mapping the memory from node 1 to 0-2G and the memory from node 2 to 4-6G, leaving a 2G hole for MMIO and other junk. Whether your BIOS actually supports this is another matter.)

Signature · 1 line
								Arne.
#5 lore raw
In reply to H. Peter Anvin
Quoted 14 lines · H. Peter Anvin
Quoted 5 lines · J.A. Magallon

Hi...

I have some Quad-Opteron boxes with 4Gb memory and two of them are running two different Linux distros.

Quoted 7 lines

Box one sees 4Gb of memory, but box two just sees 3. Their mtrr setups are different:

Why ? Is it a bios setup problem ? A kernel problem ? grep HIGHMEN in configs for both kernels does not give anything, so I still understand less this thing...

Quoted 6 lines

It would depend on how the BIOS programmed the memory controllers. For 32-bit (and lots of device) compatibility, a memory hole is required below 4 GB. Not all memory controllers can remap memory in the 3-4 GB range above the 4 GB memory; I'm not sure if that varies with the different Opteron processors.

Well, I was able to get about 3 Gb with MTRR=discrete in the BIOS, but I'm still in the process to find the 'software hole' option to get the rest of the 4Gb...

But now another (perhaps related) question has arised... I like all those 5-line progams to test system performance...;). I just wrote a simple program that sums/muls int/float vectors with scalar/sse operations. And my opteron box looks terribly slow.

This is my MacPro, Xeon 5130:

belly:~/bn> bn

	proc: 4 x MacPro1,1 @ 2000 MHz
	ram:  2048 Mb
	os:   unx, Darwin, 9.0.0
	cc:   gcc-4.0.1
vector size   : 8 x 1024 x 1024
allocation:     0.01 ms
int scl add: ..........   36.78 ms,  228.07 Mips   |  114.03 Mips  /GHz
int scl mul: ..........   34.30 ms,  244.60 Mips   |  122.30 Mips  /GHz
flt scl add: ..........   34.28 ms,  244.73 Mflops |  122.37 Mflops/GHz
flt vec add: ..........    7.89 ms, 1063.15 Mflops |  531.58 Mflops/GHz
flt scl mul: ..........   34.20 ms,  245.28 Mflops |  122.64 Mflops/GHz
flt vec mul: ..........    7.90 ms, 1061.77 Mflops |  530.89 Mflops/GHz
total:       3322.19 ms

This is a normal (I think) opteron box (Opteron 846):

selene:~/bn> g

	proc: 4 x x86_64 @ 2004 MHz
	ram:  3496 Mb
	os:   unx, Linux, 2.6.9-42.0.10.ELsmp
	cc:   gcc-4.0.2
vector size   : 8 x 1024 x 1024
allocation:     0.05 ms
int scl add: ..........   45.98 ms,  182.42 Mips   |   91.03 Mips  /GHz
int scl mul: ..........   44.31 ms,  189.30 Mips   |   94.46 Mips  /GHz
flt scl add: ..........   44.52 ms,  188.41 Mflops |   94.02 Mflops/GHz
flt vec add: ..........   10.03 ms,  836.70 Mflops |  417.52 Mflops/GHz
flt scl mul: ..........   43.32 ms,  193.63 Mflops |   96.62 Mflops/GHz
flt vec mul: ..........   10.02 ms,  836.98 Mflops |  417.65 Mflops/GHz
total:       4705.07 ms

And this is my opteron (Opteron 275)

cicely:~/bn> g

	proc: 4 x x86_64 @ 2200 MHz
	ram:  2914 Mb
	os:   unx, Linux, 2.6.23.1-desktop-1mdv
	cc:   gcc-4.0.2
vector size   : 8 x 1024 x 1024
allocation:     0.03 ms
int scl add: ..........   87.67 ms,   95.68 Mips   |   43.49 Mips  /GHz
int scl mul: ..........   85.48 ms,   98.13 Mips   |   44.61 Mips  /GHz
flt scl add: ..........   85.90 ms,   97.66 Mflops |   44.39 Mflops/GHz
flt vec add: ..........   19.51 ms,  429.96 Mflops |  195.44 Mflops/GHz
flt scl mul: ..........   85.86 ms,   97.70 Mflops |   44.41 Mflops/GHz
flt vec mul: ..........   19.50 ms,  430.11 Mflops |  195.50 Mflops/GHz
total:       6334.96 ms

As I read in AMD site, the only difference that matters in models is the xx5 vx xx6, related to fequency, but the processors should be just the same.

As this only does intensive memory/fp operations, I'm not going to blame gcc nor kernel versions here (I have compared gcc 3.4, 4.0, 4.1, and 4.2 on one of the boxes and results are very similar, the code is really stupid and not very suitable for compiler smartness...).
I suspect it is a memory problem. It can be hardware or caused by incorrect BIOS/kernel-mtrr setup:

selene:~> cat /proc/mtrr
reg00: base=0x00000000 ( 0MB), size=16384MB: write-back, count=1
reg01: base=0xf0000000 (3840MB), size= 256MB: uncachable, count=1

cicely:~> cat /proc/mtrr
reg00: base=0x00000000 ( 0MB), size=2048MB: write-back, count=1
reg01: base=0x80000000 (2048MB), size= 512MB: write-back, count=1
reg02: base=0xa0000000 (2560MB), size= 256MB: write-back, count=1
reg03: base=0xb0000000 (2816MB), size= 128MB: write-back, count=1
reg04: base=0xb8000000 (2944MB), size= 16MB: write-back, count=1

Any idea on what can be going on here ? I have asked the 'good opteron' admin info about the mobo an memory of the box.

Any help will be _very_ appreciated.

TIA

--
J.A. Magallon <jamagallon()ono!com> \ Software is like sex:
\ It's better when it's free
Mandriva Linux release 2008.1 (Cooker) for i586
Linux 2.6.23-jam01 (gcc 4.2.2 20070909 (4.2.2-0.RC.1mdv2008.0)) SMP PREEMPT 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0

#6 lore raw
Quoted 83 lines · J.A. Magall?n

Well, I was able to get about 3 Gb with MTRR=discrete in the BIOS, but I'm still in the process to find the 'software hole' option to get the rest of the 4Gb...

But now another (perhaps related) question has arised... I like all those 5-line progams to test system performance...;). I just wrote a simple program that sums/muls int/float vectors with scalar/sse operations. And my opteron box looks terribly slow.

This is my MacPro, Xeon 5130:

belly:~/bn> bn

	proc: 4 x MacPro1,1 @ 2000 MHz
	ram:  2048 Mb
	os:   unx, Darwin, 9.0.0
	cc:   gcc-4.0.1
vector size   : 8 x 1024 x 1024
allocation:     0.01 ms
int scl add: ..........   36.78 ms,  228.07 Mips   |  114.03 Mips  /GHz
int scl mul: ..........   34.30 ms,  244.60 Mips   |  122.30 Mips  /GHz
flt scl add: ..........   34.28 ms,  244.73 Mflops |  122.37 Mflops/GHz
flt vec add: ..........    7.89 ms, 1063.15 Mflops |  531.58 Mflops/GHz
flt scl mul: ..........   34.20 ms,  245.28 Mflops |  122.64 Mflops/GHz
flt vec mul: ..........    7.90 ms, 1061.77 Mflops |  530.89 Mflops/GHz
total:       3322.19 ms

This is a normal (I think) opteron box (Opteron 846):

selene:~/bn> g

	proc: 4 x x86_64 @ 2004 MHz
	ram:  3496 Mb
	os:   unx, Linux, 2.6.9-42.0.10.ELsmp
	cc:   gcc-4.0.2
vector size   : 8 x 1024 x 1024
allocation:     0.05 ms
int scl add: ..........   45.98 ms,  182.42 Mips   |   91.03 Mips  /GHz
int scl mul: ..........   44.31 ms,  189.30 Mips   |   94.46 Mips  /GHz
flt scl add: ..........   44.52 ms,  188.41 Mflops |   94.02 Mflops/GHz
flt vec add: ..........   10.03 ms,  836.70 Mflops |  417.52 Mflops/GHz
flt scl mul: ..........   43.32 ms,  193.63 Mflops |   96.62 Mflops/GHz
flt vec mul: ..........   10.02 ms,  836.98 Mflops |  417.65 Mflops/GHz
total:       4705.07 ms

And this is my opteron (Opteron 275)

cicely:~/bn> g

	proc: 4 x x86_64 @ 2200 MHz
	ram:  2914 Mb
	os:   unx, Linux, 2.6.23.1-desktop-1mdv
	cc:   gcc-4.0.2
vector size   : 8 x 1024 x 1024
allocation:     0.03 ms
int scl add: ..........   87.67 ms,   95.68 Mips   |   43.49 Mips  /GHz
int scl mul: ..........   85.48 ms,   98.13 Mips   |   44.61 Mips  /GHz
flt scl add: ..........   85.90 ms,   97.66 Mflops |   44.39 Mflops/GHz
flt vec add: ..........   19.51 ms,  429.96 Mflops |  195.44 Mflops/GHz
flt scl mul: ..........   85.86 ms,   97.70 Mflops |   44.41 Mflops/GHz
flt vec mul: ..........   19.50 ms,  430.11 Mflops |  195.50 Mflops/GHz
total:       6334.96 ms

As I read in AMD site, the only difference that matters in models is the xx5 vx xx6, related to fequency, but the processors should be just the same.

As this only does intensive memory/fp operations, I'm not going to blame gcc nor kernel versions here (I have compared gcc 3.4, 4.0, 4.1, and 4.2 on one of the boxes and results are very similar, the code is really stupid and not very suitable for compiler smartness...).
I suspect it is a memory problem. It can be hardware or caused by incorrect BIOS/kernel-mtrr setup:

selene:~> cat /proc/mtrr
reg00: base=0x00000000 ( 0MB), size=16384MB: write-back, count=1
reg01: base=0xf0000000 (3840MB), size= 256MB: uncachable, count=1

cicely:~> cat /proc/mtrr
reg00: base=0x00000000 ( 0MB), size=2048MB: write-back, count=1
reg01: base=0x80000000 (2048MB), size= 512MB: write-back, count=1
reg02: base=0xa0000000 (2560MB), size= 256MB: write-back, count=1
reg03: base=0xb0000000 (2816MB), size= 128MB: write-back, count=1
reg04: base=0xb8000000 (2944MB), size= 16MB: write-back, count=1

Quoted 4 lines

Any idea on what can be going on here ? I have asked the 'good opteron' admin info about the mobo an memory of the box.

Any help will be _very_ appreciated.

Well what revisions are the two opterons? Is one running dual channel memory while the other isn't perhaps? What speed and type is the ram on the two opterons?

--
Len Sorensen

#7 lore raw
Quoted 84 lines · On Mon, 5 Nov 2007 13:10:46 -0500, [email protected] (Lennart…
Quoted 83 lines · J.A. Magall?n

Well, I was able to get about 3 Gb with MTRR=discrete in the BIOS, but I'm still in the process to find the 'software hole' option to get the rest of the 4Gb...

But now another (perhaps related) question has arised... I like all those 5-line progams to test system performance...;). I just wrote a simple program that sums/muls int/float vectors with scalar/sse operations. And my opteron box looks terribly slow.

This is my MacPro, Xeon 5130:

belly:~/bn> bn

	proc: 4 x MacPro1,1 @ 2000 MHz
	ram:  2048 Mb
	os:   unx, Darwin, 9.0.0
	cc:   gcc-4.0.1
vector size   : 8 x 1024 x 1024
allocation:     0.01 ms
int scl add: ..........   36.78 ms,  228.07 Mips   |  114.03 Mips  /GHz
int scl mul: ..........   34.30 ms,  244.60 Mips   |  122.30 Mips  /GHz
flt scl add: ..........   34.28 ms,  244.73 Mflops |  122.37 Mflops/GHz
flt vec add: ..........    7.89 ms, 1063.15 Mflops |  531.58 Mflops/GHz
flt scl mul: ..........   34.20 ms,  245.28 Mflops |  122.64 Mflops/GHz
flt vec mul: ..........    7.90 ms, 1061.77 Mflops |  530.89 Mflops/GHz
total:       3322.19 ms

This is a normal (I think) opteron box (Opteron 846):

selene:~/bn> g

	proc: 4 x x86_64 @ 2004 MHz
	ram:  3496 Mb
	os:   unx, Linux, 2.6.9-42.0.10.ELsmp
	cc:   gcc-4.0.2
vector size   : 8 x 1024 x 1024
allocation:     0.05 ms
int scl add: ..........   45.98 ms,  182.42 Mips   |   91.03 Mips  /GHz
int scl mul: ..........   44.31 ms,  189.30 Mips   |   94.46 Mips  /GHz
flt scl add: ..........   44.52 ms,  188.41 Mflops |   94.02 Mflops/GHz
flt vec add: ..........   10.03 ms,  836.70 Mflops |  417.52 Mflops/GHz
flt scl mul: ..........   43.32 ms,  193.63 Mflops |   96.62 Mflops/GHz
flt vec mul: ..........   10.02 ms,  836.98 Mflops |  417.65 Mflops/GHz
total:       4705.07 ms

And this is my opteron (Opteron 275)

cicely:~/bn> g

	proc: 4 x x86_64 @ 2200 MHz
	ram:  2914 Mb
	os:   unx, Linux, 2.6.23.1-desktop-1mdv
	cc:   gcc-4.0.2
vector size   : 8 x 1024 x 1024
allocation:     0.03 ms
int scl add: ..........   87.67 ms,   95.68 Mips   |   43.49 Mips  /GHz
int scl mul: ..........   85.48 ms,   98.13 Mips   |   44.61 Mips  /GHz
flt scl add: ..........   85.90 ms,   97.66 Mflops |   44.39 Mflops/GHz
flt vec add: ..........   19.51 ms,  429.96 Mflops |  195.44 Mflops/GHz
flt scl mul: ..........   85.86 ms,   97.70 Mflops |   44.41 Mflops/GHz
flt vec mul: ..........   19.50 ms,  430.11 Mflops |  195.50 Mflops/GHz
total:       6334.96 ms

As I read in AMD site, the only difference that matters in models is the xx5 vx xx6, related to fequency, but the processors should be just the same.

As this only does intensive memory/fp operations, I'm not going to blame gcc nor kernel versions here (I have compared gcc 3.4, 4.0, 4.1, and 4.2 on one of the boxes and results are very similar, the code is really stupid and not very suitable for compiler smartness...).
I suspect it is a memory problem. It can be hardware or caused by incorrect BIOS/kernel-mtrr setup:

selene:~> cat /proc/mtrr
reg00: base=0x00000000 ( 0MB), size=16384MB: write-back, count=1
reg01: base=0xf0000000 (3840MB), size= 256MB: uncachable, count=1

cicely:~> cat /proc/mtrr
reg00: base=0x00000000 ( 0MB), size=2048MB: write-back, count=1
reg01: base=0x80000000 (2048MB), size= 512MB: write-back, count=1
reg02: base=0xa0000000 (2560MB), size= 256MB: write-back, count=1
reg03: base=0xb0000000 (2816MB), size= 128MB: write-back, count=1
reg04: base=0xb8000000 (2944MB), size= 16MB: write-back, count=1

Quoted 9 lines
Quoted 4 lines

Any idea on what can be going on here ? I have asked the 'good opteron' admin info about the mobo an memory of the box.

Any help will be _very_ appreciated.

Well what revisions are the two opterons? Is one running dual channel memory while the other isn't perhaps? What speed and type is the ram on the two opterons?

Well, problem solved...

I'm going to kill all pc assemblers in the world... Someone should teach them to learn mauals before assembling anything but a power chord.

The memory was not paired, so the motherboard was not interleaving the access. With no inter-node but with inter-module interleaving, and a couple 1Gb sticks for each processor now I get something like:

cicely:~/bn> bn

	name: cicely.cps.unizar.es
	arch: x86-64
	proc: 4 x x86_64 @ 2200 MHz
	ram:  3555 Mb
	os:   unx, Linux, 2.6.23.1-desktop-1mdv
	cc:   gcc-4.3.0
vector size   : 8 x 1024 x 1024
allocation:     0.02 ms
int scl add: ..........   60.56 ms,  138.52 Mips   |   62.96 Mips  /GHz
int scl mul: ..........   59.34 ms,  141.36 Mips   |   64.26 Mips  /GHz
flt scl add: ..........   59.01 ms,  142.16 Mflops |   64.62 Mflops/GHz
flt vec add: ..........   14.79 ms,  567.06 Mflops |  257.75 Mflops/GHz
flt scl mul: ..........   59.02 ms,  142.12 Mflops |   64.60 Mflops/GHz
flt vec mul: ..........   14.82 ms,  566.19 Mflops |  257.36 Mflops/GHz
total:       5019.86 ms

Much better, but not like the other opteron box.

My processors are higher than Rev E0, because the BIOS does not let me choose the 'software' hole. If I activate the 'hardware hole', I see al the memory I can:

cicely:~/bn> free
total used free shared buffers cached

Mem:       3640628     214496    3426132          0      21240      84184
-/+ buffers/cache:     109072    3531556
Swap:      4200988          0    4200988

3.64 Gb. The rest is eaten by the graphics card, as I could read in the AMD site. Don't know if mem=4096 to boot the kernel would help, even if it is possible (don't think so, as it looks like a BIOS mis-feature). The ram is DDR 400.

Anyways, can I trust what dmidecode says ? I installed the ram as the board manual said in banks 1A+1B (not 2A+2B) for each processor, but this program says this:

BANK0   64Mb            BANK4   64Mb
BANK1   64Mb            BANK5   64Mb
BANK2 1024Mb            BANK6 1024Mb
BANK3 1024Mb            BANK7 1024Mb

I would always have thought that BANK0 would be slot 1A in first processor, but it looks like not...
And where do the 64 Mb blocks come from ?

Really strange...

--
J.A. Magallon <jamagallon()ono!com> \ Software is like sex:
\ It's better when it's free
Mandriva Linux release 2008.1 (Cooker) for i586
Linux 2.6.23-jam01 (gcc 4.2.2 20070909 (4.2.2-0.RC.1mdv2008.0)) SMP PREEMPT 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0

#8 lore raw
Quoted 34 lines · J.A. Magall?n

Well, problem solved...

I'm going to kill all pc assemblers in the world... Someone should teach them to learn mauals before assembling anything but a power chord.

The memory was not paired, so the motherboard was not interleaving the access. With no inter-node but with inter-module interleaving, and a couple 1Gb sticks for each processor now I get something like:

cicely:~/bn> bn

	name: cicely.cps.unizar.es
	arch: x86-64
	proc: 4 x x86_64 @ 2200 MHz
	ram:  3555 Mb
	os:   unx, Linux, 2.6.23.1-desktop-1mdv
	cc:   gcc-4.3.0
vector size   : 8 x 1024 x 1024
allocation:     0.02 ms
int scl add: ..........   60.56 ms,  138.52 Mips   |   62.96 Mips  /GHz
int scl mul: ..........   59.34 ms,  141.36 Mips   |   64.26 Mips  /GHz
flt scl add: ..........   59.01 ms,  142.16 Mflops |   64.62 Mflops/GHz
flt vec add: ..........   14.79 ms,  567.06 Mflops |  257.75 Mflops/GHz
flt scl mul: ..........   59.02 ms,  142.12 Mflops |   64.60 Mflops/GHz
flt vec mul: ..........   14.82 ms,  566.19 Mflops |  257.36 Mflops/GHz
total:       5019.86 ms

Much better, but not like the other opteron box.

My processors are higher than Rev E0, because the BIOS does not let me choose the 'software' hole. If I activate the 'hardware hole', I see al the memory I can:

cicely:~/bn> free
total used free shared buffers cached

Quoted 8 lines
Mem:       3640628     214496    3426132          0      21240      84184
-/+ buffers/cache:     109072    3531556
Swap:      4200988          0    4200988

3.64 Gb. The rest is eaten by the graphics card, as I could read in the AMD site. Don't know if mem=4096 to boot the kernel would help, even if it is possible (don't think so, as it looks like a BIOS mis-feature). The ram is DDR 400.

The video card is stealing 300MB of ram? What for? What does the mtrr and e820 map look like with the hardware hole enabled?

Quoted 4 lines

Anyways, can I trust what dmidecode says ? I installed the ram as the board manual said in banks 1A+1B (not 2A+2B) for each processor, but this program says this:

Quoted 8 lines
BANK0   64Mb            BANK4   64Mb
BANK1   64Mb            BANK5   64Mb
BANK2 1024Mb            BANK6 1024Mb
BANK3 1024Mb            BANK7 1024Mb

I would always have thought that BANK0 would be slot 1A in first processor, but it looks like not...
And where do the 64 Mb blocks come from ?

Well if you ahve 4 sticks of 1GB, then I would hope they are installed as a pair for each CPU so that both CPUs can have dual channel ram directly connected.

I have no idea where the 64Mb comes from.

--
Len Sorensen

#9 lore raw
Quoted 39 lines · On Mon, 5 Nov 2007 13:50:40 -0500, [email protected] (Lennart…
Quoted 38 lines · J.A. Magall?n

Well, problem solved...

I'm going to kill all pc assemblers in the world... Someone should teach them to learn mauals before assembling anything but a power chord.

The memory was not paired, so the motherboard was not interleaving the access. With no inter-node but with inter-module interleaving, and a couple 1Gb sticks for each processor now I get something like:

cicely:~/bn> bn

	name: cicely.cps.unizar.es
	arch: x86-64
	proc: 4 x x86_64 @ 2200 MHz
	ram:  3555 Mb
	os:   unx, Linux, 2.6.23.1-desktop-1mdv
	cc:   gcc-4.3.0
vector size   : 8 x 1024 x 1024
allocation:     0.02 ms
int scl add: ..........   60.56 ms,  138.52 Mips   |   62.96 Mips  /GHz
int scl mul: ..........   59.34 ms,  141.36 Mips   |   64.26 Mips  /GHz
flt scl add: ..........   59.01 ms,  142.16 Mflops |   64.62 Mflops/GHz
flt vec add: ..........   14.79 ms,  567.06 Mflops |  257.75 Mflops/GHz
flt scl mul: ..........   59.02 ms,  142.12 Mflops |   64.60 Mflops/GHz
flt vec mul: ..........   14.82 ms,  566.19 Mflops |  257.36 Mflops/GHz
total:       5019.86 ms

Much better, but not like the other opteron box.

My processors are higher than Rev E0, because the BIOS does not let me choose the 'software' hole. If I activate the 'hardware hole', I see al the memory I can:

cicely:~/bn> free
total used free shared buffers cached

Mem:       3640628     214496    3426132          0      21240      84184
-/+ buffers/cache:     109072    3531556
Swap:      4200988          0    4200988
Quoted 8 lines
Quoted 4 lines

3.64 Gb. The rest is eaten by the graphics card, as I could read in the AMD site. Don't know if mem=4096 to boot the kernel would help, even if it is possible (don't think so, as it looks like a BIOS mis-feature). The ram is DDR 400.

The video card is stealing 300MB of ram? What for? What does the mtrr and e820 map look like with the hardware hole enabled?

(correction, this was not AMD website but SuperMicro's)

I just said the same... Board is a SuperMicro H8DCE. From the FAQ section at supermicro:

Question
I have an H8DCE motherboard with 4 x 1GB DIMMS installed but the amount of memory displayed in the BIOS is 3.865GB and in 64-bit XP is 3.76GB. I have the hardware memory hole option enabled in BIOS (rev 1.1a). How I can get the board to count the full 4GB of memory?

Answer
The total available size depends on the PCI-e card you are using; some high-end cards may occupy more memory. For example, with a Quadro FX4500 on the H8DCE with the memory hole enabled, 4GB memory will show up as 3728MB in BIOS and 3.64GB in Windows. For some low-end PCI-e VGA cards, it may show up as 4048MB in BIOS.

Why ? Who knows...
Chipset is all nVidia. I have a GeForce 8800GTX with 768 Mb. It eats up 400Mb.

This are my settings:

BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000e6000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000009ffd0000 (usable)
BIOS-e820: 000000009ffd0000 - 000000009ffde000 (ACPI data)
BIOS-e820: 000000009ffde000 - 00000000a0000000 (ACPI NVS)
BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
BIOS-e820: 00000000ff780000 - 0000000100000000 (reserved)
BIOS-e820: 0000000100000000 - 0000000147000000 (usable)

cicely:~# cat /proc/mtrr
reg00: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg01: base=0x140000000 (5120MB), size=  64MB: write-back, count=1
reg02: base=0x144000000 (5184MB), size=  32MB: write-back, count=1
reg03: base=0x146000000 (5216MB), size=  16MB: write-back, count=1
reg04: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
reg05: base=0x80000000 (2048MB), size= 512MB: write-back, count=1

This is with BIOS set for MTRR=Discrete.
With MTRR=Continuous, the mtrr's are simpler, a full range and a non-usable hole. Which is better for Linux ? Many separate usable zones or one big zone and an un-usable hole ?

BTW, mtrr formatting should be set to 0x%013lx000, to get them aligned with nowadays memory amounts and similar to e820 map, 16 hex digits...

Quoted 9 lines
Quoted 9 lines

Anyways, can I trust what dmidecode says ? I installed the ram as the board manual said in banks 1A+1B (not 2A+2B) for each processor, but this program says this:

BANK0   64Mb            BANK4   64Mb
BANK1   64Mb            BANK5   64Mb
BANK2 1024Mb            BANK6 1024Mb
BANK3 1024Mb            BANK7 1024Mb
Quoted 8 lines
Quoted 3 lines

I would always have thought that BANK0 would be slot 1A in first processor, but it looks like not...
And where do the 64 Mb blocks come from ?

Well if you ahve 4 sticks of 1GB, then I would hope they are installed as a pair for each CPU so that both CPUs can have dual channel ram directly connected.

That's how they are plugged. The strange thing is that I filled the 1st and 2nd slot (by mobo manual numbering), but dmidecode (BIOS?) thinks they are 3rd and 4th. I don't know if it matters, probably not, but who knows... This s***t of PC architecture is a Pandora's Box.

Quoted 2 lines

I have no idea where the 64Mb comes from.

Nor I do...

--
J.A. Magallon <jamagallon()ono!com> \ Software is like sex:
\ It's better when it's free
Mandriva Linux release 2008.1 (Cooker) for i586
Linux 2.6.23-jam01 (gcc 4.2.2 20070909 (4.2.2-0.RC.1mdv2008.0)) SMP PREEMPT 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0

#10 lore raw
Quoted 37 lines · J.A. Magall?n

(correction, this was not AMD website but SuperMicro's)

I just said the same... Board is a SuperMicro H8DCE. From the FAQ section at supermicro:

Question
I have an H8DCE motherboard with 4 x 1GB DIMMS installed but the amount of memory displayed in the BIOS is 3.865GB and in 64-bit XP is 3.76GB. I have the hardware memory hole option enabled in BIOS (rev 1.1a). How I can get the board to count the full 4GB of memory?

Answer
The total available size depends on the PCI-e card you are using; some high-end cards may occupy more memory. For example, with a Quadro FX4500 on the H8DCE with the memory hole enabled, 4GB memory will show up as 3728MB in BIOS and 3.64GB in Windows. For some low-end PCI-e VGA cards, it may show up as 4048MB in BIOS.

Why ? Who knows...
Chipset is all nVidia. I have a GeForce 8800GTX with 768 Mb. It eats up 400Mb.

This are my settings:

BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000e6000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000009ffd0000 (usable)
BIOS-e820: 000000009ffd0000 - 000000009ffde000 (ACPI data)
BIOS-e820: 000000009ffde000 - 00000000a0000000 (ACPI NVS)
BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
BIOS-e820: 00000000ff780000 - 0000000100000000 (reserved)
BIOS-e820: 0000000100000000 - 0000000147000000 (usable)

cicely:~# cat /proc/mtrr
reg00: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg01: base=0x140000000 (5120MB), size=  64MB: write-back, count=1
reg02: base=0x144000000 (5184MB), size=  32MB: write-back, count=1
reg03: base=0x146000000 (5216MB), size=  16MB: write-back, count=1
reg04: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
reg05: base=0x80000000 (2048MB), size= 512MB: write-back, count=1
Quoted 4 lines

This is with BIOS set for MTRR=Discrete.
With MTRR=Continuous, the mtrr's are simpler, a full range and a non-usable hole. Which is better for Linux ? Many separate usable zones or one big zone and an un-usable hole ?

Seems odd if they can't just map memory as:
2GB at 0
512MB at 2GB
1GB at 4GB
512MB at 5GB

Or for that matter:
2GB at 0
2GB at 4GB
Why can't they do things as simple as that? If you have a 64bit OS that would be perfectly fine.

It appears they have
2GB at 0
512MB at 2GB
1GB at 4GB
64MB at 5GB
32MB at 5GB+64MB
16MB at 5GB+64MB+32MB

Where did they map the rest of the ram? All I can think there is that they messed up. Of course some think that having as much ram for XP 32bit as possible is more important than sane systems, so they will map more in the first 4GB where XP32 can use it, and then due to alignments and rounding of the mapping they can't get all the remaining ram mapped above 4GB.

--
Len Sorensen