NUMA support for dual core Opteron
Someone mentioned that NUMA support for dual core opteron need acpi support in LinuxBIOS.
there may be some other solution for that.
1. PowerPC already support dual core and it should support NUMA, So the Open Firmware must have some NUMA entry definition.
Can we make x86-64 kernel support OpenFirmware interface so we can use OpenBIOS as payload of LinuxBIOS.
2. enable acpi and add the NUMA entries into it, the Linux Kernel will be happy.
3. If we are trying to use ADLO to load Windows/Solaris/FreeBSD, We need to pass related acpi info to ADLO....
Solution 1 will be ideal one, and can make Solaris for X86-64 use OpenFirmware interface too.....
which one is better?
YH
Quoted 12 lines · yhlu
Someone mentioned that NUMA support for dual core opteron need acpi support in LinuxBIOS.
there may be some other solution for that.
1. PowerPC already support dual core and it should support NUMA, So the Open Firmware must have some NUMA entry definition.
Can we make x86-64 kernel support OpenFirmware interface so we can use OpenBIOS as payload of LinuxBIOS.
2. enable acpi and add the NUMA entries into it, the Linux Kernel will be happy.
3. If we are trying to use ADLO to load Windows/Solaris/FreeBSD, We need to pass related acpi info to ADLO....
Quoted 5 lines
Solution 1 will be ideal one, and can make Solaris for X86-64 use OpenFirmware interface too.....
which one is better?
AFIAK, for x86_64 kernel, it will try to read NUMA configuration from HW directory. We don't have to export any ACPI table.
Signature · 2 lines
Li-Ta Lo <[email protected]> Los Alamos National Lab
Quoted 2 lines
AFIAK, for
x86_64kernel, it will try to read NUMA configuration from HW directory. We don't have to export any ACPI table.
It doesn't work for dual core or 8 sockets for some reason. Since the SRAT code works fine and is in general more future proof we never tracked down why. Patches welcome.
However you'll likely need ACPI for other reasons anyways, e.g. for better power saving.
-Andi
For S2895, with 1Gx8 Installed and E stepping dual core opteron with 1G mem hole emable, got
Bootdata ok (command line is apic=debug ramdisk_size=65536 root=/dev/ram0 rw console=tty0 console=ttyS0,115200n8 ) Linux version 2.6.12-rc5 (root@tst288xsuse9) (gcc version 3.3.3 (SuSE Linux)) #26 SMP Thu Jun 2 18:15:44 PDT 2005
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 0000000000000dd0 (reserved)
BIOS-e820: 0000000000000dd0 - 00000000000a0000 (usable)
BIOS-e820: 00000000000f0000 - 00000000000f0400 (reserved)
BIOS-e820: 0000000000100000 - 00000000c0000000 (usable)
BIOS-e820: 0000000100000000 - 0000000240000000 (usable)
ACPI: Unable to locate RSDP
Scanning NUMA topology in Northbridge 24
Number of nodes 2
Node 0 MemBase 0000000000000000 Limit 000000013fffffff Node 1 MemBase 0000000140000000 Limit 000000023fffffff node 1 shift 24 addr 140000000 conflict 0
node 1 shift 25 addr 1fe000000 conflict 0
Using node hash shift of 26
Bootmem setup node 0 0000000000000000-000000013fffffff Bootmem setup node 1 0000000140000000-000000023fffffff
~ # cat /proc/mt~ # cat /proc/mtrr
reg00: base=0x00000000 ( 0MB), size=8192MB: write-back, count=1
reg01: base=0x200000000 (8192MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
with suse kernel and LinuxBIOS + S2882 I can get
linux:~ # cat /proc/meminfo
total: used: free: shared: buffers: cached:
Mem: 8131170304 145051648 7986118656 0 15220736 66543616 Swap: 2154979328 0 2154979328 MemTotal: 7940596 kB MemFree: 7798944 kB MemShared: 0 kB Buffers: 14864 kB Cached: 64984 kB SwapCached: 0 kB Active: 39056 kB Inactive: 40844 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 7940596 kB LowFree: 7798944 kB SwapTotal: 2104472 kB SwapFree: 2104472 kB BigFree: 0 kB
Node 0 MemTotal: 4194300 kB Node 0 MemFree: 3793788 kB Node 0 MemUsed: 400512 kB Node 0 HighTotal: 0 kB Node 0 HighFree: 0 kB Node 0 LowTotal: 4194300 kB Node 0 LowFree: 3793788 kB
Node 1 MemTotal: 4194300 kB Node 1 MemFree: 4005156 kB Node 1 MemUsed: 189144 kB Node 1 HighTotal: 0 kB Node 1 HighFree: 0 kB Node 1 LowTotal: 4194300 kB Node 1 LowFree: 4005156 kB
I wonder if suse kernel have some special code to show that. in Kernel.org, I can not Node0....Node 1...
YH
Quoted 28 lines · Li-Ta Lo
Quoted 12 lines · yhlu
Someone mentioned that NUMA support for dual core opteron need acpi support in LinuxBIOS.
there may be some other solution for that.
1. PowerPC already support dual core and it should support NUMA, So the Open Firmware must have some NUMA entry definition.
Can we make x86-64 kernel support OpenFirmware interface so we can use OpenBIOS as payload of LinuxBIOS.
2. enable acpi and add the NUMA entries into it, the Linux Kernel will be happy.
3. If we are trying to use ADLO to load Windows/Solaris/FreeBSD, We need to pass related acpi info to ADLO....Quoted 5 lines
Solution 1 will be ideal one, and can make Solaris for X86-64 use OpenFirmware interface too.....
which one is better?
AFIAK, for
x86_64kernel, it will try to read NUMA configuration from HW directory. We don't have to export any ACPI table.--
Li-Ta Lo <[email protected]>
Los Alamos National Lab
P.S.: It is very nasty to cc closed mailing lists when posting to open ones. Please don't do that in the future.
-Andi
if there is any chance of getting along without ACPI entries that is best. Linux did do this once already, for SMP K8: K8 can boot and run NUMA without an SRAT table. What more is needed for dual core, and could Linux support in this area be extended?
ron
FYI in Tyan S4881 (8 ways dual core 875 cpu ) with LinuxBIOS I got also the 1G mem hole is enabled.
So the kernel should be OK with read NUMA directly from HW.
YH
Firmware type: LinuxBIOS
old bootloader convention, maybe loadlin?
Bootdata ok (command line is apic=debug ramdisk_size=65536 root=/dev/ram0 rw console=tty0 console=ttyS0,115200n8 ) Linux version 2.6.12 (root@tst288xsuse9) (gcc version 3.3.3 (SuSE Linux)) #8 SMP Fri Jun 24 12:41:43 PDT 2005 BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 0000000000000e48 (reserved)
BIOS-e820: 0000000000000e48 - 00000000000a0000 (usable)
BIOS-e820: 00000000000f0000 - 00000000000f0400 (reserved)
BIOS-e820: 0000000000100000 - 00000000c0000000 (usable)
BIOS-e820: 0000000100000000 - 0000000840000000 (usable)
Scanning NUMA topology in Northbridge 24
Number of nodes 8
Node 0 MemBase 0000000000000000 Limit 000000013fffffff Node 1 MemBase 0000000140000000 Limit 000000023fffffff Node 2 MemBase 0000000240000000 Limit 000000033fffffff Node 3 MemBase 0000000340000000 Limit 000000043fffffff Node 4 MemBase 0000000440000000 Limit 000000053fffffff Node 5 MemBase 0000000540000000 Limit 000000063fffffff Node 6 MemBase 0000000640000000 Limit 000000073fffffff Node 7 MemBase 0000000740000000 Limit 000000083fffffff node 1 shift 24 addr 140000000 conflict 0
node 1 shift 25 addr 1fe000000 conflict 0
node 3 shift 26 addr 3fc000000 conflict 0
node 7 shift 27 addr 7f8000000 conflict 0
Using node hash shift of 28
Bootmem setup node 0 0000000000000000-000000013fffffff Bootmem setup node 1 0000000140000000-000000023fffffff Bootmem setup node 2 0000000240000000-000000033fffffff Bootmem setup node 3 0000000340000000-000000043fffffff Bootmem setup node 4 0000000440000000-000000053fffffff Bootmem setup node 5 0000000540000000-000000063fffffff Bootmem setup node 6 0000000640000000-000000073fffffff Bootmem setup node 7 0000000740000000-000000083fffffff
in LB
Setting variable MTRR 0, base: 0MB, range: 32768MB, type WB
Setting variable MTRR 1, base: 32768MB, range: 1024MB, type WB Setting variable MTRR 2, base: 3072MB, range: 1024MB, type UC
Quoted 13 lines · Andi Kleen
Quoted 2 lines
AFIAK, for
x86_64kernel, it will try to read NUMA configuration from HW directory. We don't have to export any ACPI table.It doesn't work for dual core or 8 sockets for some reason. Since the SRAT code works fine and is in general more future proof we never tracked down why. Patches welcome.
However you'll likely need ACPI for other reasons anyways, e.g. for better power saving.
-Andi
Quoted 2 lines · Andi Kleen
However you'll likely need ACPI for other reasons anyways, e.g. for better power saving.
bummer. What the BIOS vendors are doing (to lock in proprietary BIOS, some say) is making ACPI tables copyright the BIOS vendor, not the motherboard vendor. So LinuxBIOS will have to reverse engineer their own, somehow.
Shame we can't free ourselves of ACIP a bit. Oh well.
ron
[closed mailing list dropped. Sorry I have no plans to argue with your mailbots]
Quoted 4 lines · Ronald G. Minnich
if there is any chance of getting along without ACPI entries that is best. Linux did do this once already, for SMP K8: K8 can boot and run NUMA without an SRAT table. What more is needed for dual core, and could Linux support in this area be extended?
The dual core NUMA parsing problem could be probably fixed. I personally have no plans to work on it though, since the ACPI method works fine.
Feel free to submit patches.
However with 90+W CPUs I would strongly recommend having support for PowerNow! and the old style PST table doesn't support dual core or SMP, so you need ACPI for that anyways.
-Andi
Quoted 10 lines · Ronald G. Minnich
Quoted 2 lines · Andi Kleen
However you'll likely need ACPI for other reasons anyways, e.g. for better power saving.
bummer. What the BIOS vendors are doing (to lock in proprietary BIOS, some say) is making ACPI tables copyright the BIOS vendor, not the motherboard vendor. So LinuxBIOS will have to reverse engineer their own, somehow.
You don't need full support, many of it is optional and will fall back to the old methods if not available. e.g. you can probably leave out most of the PCI support if you don't want to support PCI hotplug. Longer term it might be needed again for power management though.
Doing PST objects for power saving shouldn't be that difficult, but you need knowledge of the CPUs from their data sheet (and some testing if the power regulators on the mobo can take all the transitions) But it shouldn't be very motherboard specific.
However that said there is a lot of useful information
in the FADT and some other tables and I definitely plan to use more of it in the future.
-Andi
I didn't see any problem about NUMA with LinuxBIOS + 8way dual core system. of couse the acpi support in Kernel is disabled.
p.s. can you use powernow when acpi is disabled?
p.s.s Is powerpc64 support ACPI? or ACPI is only can be used by x86?
YH
Quoted 20 lines · Andi Kleen
[closed mailing list dropped. Sorry I have no plans to argue with your mailbots]
Quoted 4 lines · Ronald G. Minnich
if there is any chance of getting along without ACPI entries that is best. Linux did do this once already, for SMP K8: K8 can boot and run NUMA without an SRAT table. What more is needed for dual core, and could Linux support in this area be extended?
The dual core NUMA parsing problem could be probably fixed. I personally have no plans to work on it though, since the ACPI method works fine.
Feel free to submit patches.
However with 90+W CPUs I would strongly recommend having support for PowerNow! and the old style PST table doesn't support dual core or SMP, so you need ACPI for that anyways.
-Andi
Quoted 1 line · yhlu
p.s. can you use powernow when acpi is disabled?
Only on uniprocessor machines.
Quoted 1 line
p.s.s Is powerpc64 support ACPI? or ACPI is only can be used by x86?
powerpc64 uses openfirmware, not ACPI.
-Andi
Andi,
How do yo think about make x86-64 kernel support openfirmware interface?
Can we borrow some code from ppc64 arch?
YH
Quoted 11 lines · Andi Kleen
Quoted 1 line · yhlu
p.s. can you use powernow when acpi is disabled?
Only on uniprocessor machines.
Quoted 1 line
p.s.s Is powerpc64 support ACPI? or ACPI is only can be used by x86?
powerpc64 uses openfirmware, not ACPI.
-Andi
Quoted 3 lines · On Thu, 14 Jul 2005 20:52:58 -0700 yhlu <[email protected]> wrote:
Andi,
How do yo think about make x86-64 kernel support openfirmware interface?
I don't like it. We already have the old x86 BIOS interfaces and ACPI and at some point EFI. No need for more.
-Andi
EFI support in x86-64?
Is EFI only support IA64?
Is acpi in EFI?
YH
Quoted 12 lines · Andi Kleen
Quoted 3 lines · On Thu, 14 Jul 2005 20:52:58 -0700 yhlu <[email protected]> wrote:
Andi,
How do yo think about make x86-64 kernel support openfirmware interface?
I don't like it. We already have the old x86 BIOS interfaces and ACPI and at some point EFI. No need for more.
-Andi
Quoted 1 line · Andi Kleen
Only on uniprocessor machines.
Question for the AMD guys: is there a chance of getting non-proprietary-bios ACPI tables from AMD directly? I.e. ACPI tables as needed for power-now etc. could be released under GPL, making inclusion into linuxbios a bit simpler. Right now, the only ACPI table's I've seen all bear "IF I COPY THIS PLEASE SUE ME UNDER THE DMCA" notices :-)
If such tables are available, and I'm just out of touch, I'd be very happy to hear that; please send me a URL.
It makes no sense at all to me that ACPI would be copyright anyone, since they merely describe hardware, and even the OS guys might want to copy them around from node to node in some cases. But that's the problem right now.
ron
Quoted 3 lines · Andi Kleen
However with 90+W CPUs I would strongly recommend having support for PowerNow! and the old style PST table doesn't support dual core or SMP, so you need ACPI for that anyways.
Do opterons even support powernow? The proc or sysfs control file never shows up on mine and the cpu flags don't list it. Then again, neither does my athlon64. They're all in 32bit mode.
Signature · 2 lines
Tom Vier <[email protected]> DSA Key ID 0x15741ECE
Quoted 8 lines · Tom Vier
Quoted 3 lines · Andi Kleen
However with 90+W CPUs I would strongly recommend having support for PowerNow! and the old style PST table doesn't support dual core or SMP, so you need ACPI for that anyways.
Do opterons even support powernow? The proc or sysfs control file never shows up on mine and the cpu flags don't list it. Then again, neither does my athlon64. They're all in 32bit mode.
New enough ones when the BIOS supports it too: yes. Older ones didn't.
-Andi