What about...
Quoted 2 lines · Igor Loncarevic
What about Linux for Origin2000 SGI?
Is there any possibility to run Linux on Origin?
I would say there's almost no chance of running Linux on an Origin 2000 anytime this century.
- Alex
Alex is right.
Linux on Origin 2000 would be a huge project - not just a port.
We have much better chances getting Linux up on SGIs yet-to-be-revealed Intel-based platforms.
g
Signature · 1 line
Greg Chesson
Quoted 2 lines · Greg Chesson
Alex is right.
Linux on Origin 2000 would be a huge project - not just a port.
...
Note that the reason it would be a big project is not so much the
hardware itself, but rather the scale of the system. IRIX needed a lot of infrastructure work to be useful on very large system with a very large number of I/O buses and devices. That is, a toy port would not be all that giant a project, but it would not be particularly useful. If there were a good linux on some other very large ccNUMA machine, then an Origin port would be much simpler. By "good", I mean a linux which scale well for large (greater than 32) processor and I/O count (many I/O buses and thousands of disk). It expect such a linux will happen eventually, but not yet.
Quoted 6 lines
that giant a project, but it would not be particularly useful. If there were a good linux on some other very large ccNUMA machine, then an Origin port would be much simpler. By "good", I mean a linux which scale well for large (greater than 32) processor and I/O count (many I/O buses and thousands of disk). It expect such a linux will happen eventually, but not yet.
The obvious starting point would probably be the older (386/486) sequent boxes. I almost got somewhere with this but one bit of sequent wasnt willing to be helpful and counted its bus arbitrators as staying NDA.
That was 2 years ago so I ought to chase them again I guess
Quoted 5 lines · William J. Earl
Quoted 2 lines · Greg Chesson
Alex is right.
Linux on Origin 2000 would be a huge project - not just a port....
Quoted 10 lines
Note that the reason it would be a big project is not so much the
hardware itself, but rather the scale of the system. IRIX needed a lot of infrastructure work to be useful on very large system with a very large number of I/O buses and devices. That is, a toy port would not be all that giant a project, but it would not be particularly useful. If there were a good linux on some other very large ccNUMA machine, then an Origin port would be much simpler. By "good", I mean a linux which scale well for large (greater than 32) processor and I/O count (many I/O buses and thousands of disk). It expect such a linux will happen eventually, but not yet.
Actually some people begin to be interested in Linux for huge installations. For example a bank recently offer us some machine time on a maximum configuration (64 CPU, 64GB RAM) E10000 for porting Linux. This didn't happen (yet?) for other reasons, but it shows that there is interest.
For the time being most large Linux installations are based on the Beowulf clustering software - and I think place 318 on the list on the top 500 list of supercomputer installations is quite remarkable for something which's OS software is being shipped for USD 29.95 by RedHat.
Anyway, as far as our MIPS port goes we first need to implement SMP at all before we should think about scaling. More about Linux in general - most people are currentl using Linux/SMP on dual processors, few on quad CPU systems and even fewer on larger systems. This is about where the market of Linux and it's limits are. But the tradition of Linux shows that technology for which a demand exists will be developed sooner or later. Where Linux technology exists, usually a market will develop. So far I see a demand for older SGI SMP systems, Origin 200 systems (quite some inquiries) but not yet for Origin 2000 systems.
Ralf
Quoted 7 lines · Alan Cox
Quoted 6 lines
that giant a project, but it would not be particularly useful. If there were a good linux on some other very large ccNUMA machine, then an Origin port would be much simpler. By "good", I mean a linux which scale well for large (greater than 32) processor and I/O count (many I/O buses and thousands of disk). It expect such a linux will happen eventually, but not yet.
Quoted 5 lines
The obvious starting point would probably be the older (386/486) sequent boxes. I almost got somewhere with this but one bit of sequent wasnt willing to be helpful and counted its bus arbitrators as staying NDA.
That was 2 years ago so I ought to chase them again I guess
Do you have hardware? I know an installation which might want to get rid of their 486 based 32 processor Sequent system if they didn't already.
Ralf
Bill is, of course, quite correct.
In addition to the observations about the scale of the system, realize also that a ccNUMA machine has a memory system for each cpu node in the system. The physical base addresses of these blocks of memory are aligned on multi-gigabyte boundaries. The high-order bits of the address designate the cpu node, the rest address the physical memory, etc etc. What this means is that physical memory space has many "holes"... The idea of a simple buddy-system allocator as is ingrained in the Linux kernel falls apart completely in the face of
this kind of architecture. I suppose you could run a copy of Linux
on every node, but I consider that an excuse rather than a solution.
g
Signature · 1 line
Greg Chesson
Quoted 1 line
Do you have hardware? I know an installation which might want to get rid
I dont. I had an offer at the time, and I did some digging on the older boxes also recently on the new ones - did you know Sequent Numa-Q appears to use non standard microcode ?
Quoted 1 line
of their 486 based 32 processor Sequent system if they didn't already.
I can just imagine the shipping cost ;)
Ralf,
Beowolf machines don't have adequate IO for applications other than things that resemble Linpack. The teraflop machine at Sandia would like to generate output every 5 or 10 minutes, but it takes an hour to get the computation result out of the machine. Beowolf machines tend to not have adequate bisection bandwidth more many kinds of codes. Anything that requires a cornerturn (transpose) is a good example. Also, Beowolf machines tend to run a single code for many hours: they don't multiplex well, or sometimes at all. But multiplexing and adapting to a dynamic load is expected of many large scale machines and that requires some OS support not found in the single-cpu or small-cpu-count OS or existing middleware.
We cannot afford to build
machines that are good only for Linpack.... it is better
to build a Beowolf cluster if that is the best fit for the job. It is rather ignorant to think that because something like Beowolf can do some some types of applications well, that some software work can make it do all jobs well. That's like saying a Volkswagon could go 300 kph if you drive carefully.
Linux has proven to be worthwhile as a node controller in an MPP architecture - that's what a Beowolf is. But that does not make it ready for SMP nodes that scale to large numbers. It seems wasteful to program a large scale ccNUMA machine the same way as a Beowolf cluster: you'd be throwing away most of the capabilities of the hardware. That's why I don't think it is interesting or particularly useful... unless a massive amount of work went into rewriting the io and memory management subsystems, not to mention scheduling, administration, etc.
g
Signature · 1 line
Greg Chesson
Quoted 4 lines
many "holes"... The idea of a simple buddy-system allocator as is ingrained in the Linux kernel falls apart completely in the face of
this kind of architecture. I suppose you could run a copy of Linux
on every node, but I consider that an excuse rather than a solution.
Actually the Linux buddy stuff is quite happy with holes. Its still completely inappropriate. From the above I deduce we'd have to do mips64 before we even considerd it anyway
Greg Chesson writes:
...
Quoted 8 lines
Linux has proven to be worthwhile as a node controller in an MPP architecture - that's what a Beowolf is. But that does not make it ready for SMP nodes that scale to large numbers. It seems wasteful to program a large scale ccNUMA machine the same way as a Beowolf cluster: you'd be throwing away most of the capabilities of the hardware. That's why I don't think it is interesting or particularly useful... unless a massive amount of work went into rewriting the io and memory management subsystems, not to mention scheduling, administration, etc.
...
I expect that much of the work will gradually happen in linux, once
the remaining small-CPU-count issues are resolved. Right now, there is no shortage of interesting problems to attack. :-)
One possible way to approach the large-CPU-count space with linux
is to indeed run multiple linux kernels, one per node in a ccNUMA machine, and add a distributed OS layer at a fairly high level. If it is not underway already somewhere, I would expect someone to take up the project as a graduate school project. Some systems of this sort have been built or attempted, with varying degrees of success. Given the linux bias toward small and simple, a linux-based distributed OS might actually work.
One important ingredient in such a system, which would be
valuable immediately for clusters, would be a efficient distributed volume manager and file system.
In any case, trying to port linux straight to a large ccNUMA
(or even a large SMP) system would be a lot of effort for limited return at present.
Quoted 8 lines · Alan Cox
Quoted 4 lines
many "holes"... The idea of a simple buddy-system allocator as is ingrained in the Linux kernel falls apart completely in the face of
this kind of architecture. I suppose you could run a copy of Linux
on every node, but I consider that an excuse rather than a solution.Actually the Linux buddy stuff is quite happy with holes. Its still completely inappropriate. From the above I deduce we'd have to do mips64 before we even considerd it anyway
Yes, the address space is very large, even in a single rack.
Quoted 6 lines · Alan Cox
Quoted 6 lines
that giant a project, but it would not be particularly useful. If there were a good linux on some other very large ccNUMA machine, then an Origin port would be much simpler. By "good", I mean a linux which scale well for large (greater than 32) processor and I/O count (many I/O buses and thousands of disk). It expect such a linux will happen eventually, but not yet.
Boy would it be fun to work on that port. Origin2000s are seriously cool. I am getting 9 new Origin2000 systems in about 6 weeks. I wonder if Sprint would mind if I used one for Linux porting... :-)
J.
o-----------------------------------o
| Jeffrey Watts | | [email protected] o-------------------------------------o | Systems Analyst | "Proprietary system advocates | | Sprint - Systems Management | aren't evil or stupid. They are | o-------------------------------| the victims. They have a disease | | and they need help." | | -- Donald B. Marti Jr. | o-------------------------------------o
Quoted 8 lines · Alan Cox
Quoted 4 lines
many "holes"... The idea of a simple buddy-system allocator as is ingrained in the Linux kernel falls apart completely in the face of
this kind of architecture. I suppose you could run a copy of Linux
on every node, but I consider that an excuse rather than a solution.Actually the Linux buddy stuff is quite happy with holes. Its still completely inappropriate. From the above I deduce we'd have to do mips64 before we even considerd it anyway
At least in Vger CVS we alredy have code to efficiently deal with non-dense memory architectures with buddy.
I think the memory allocator design as presented by Rik van Riel looks as if it is a good base to deal much more efficiently with such an memory architecture. And then there is Sct with his big iron experience working on something better than the buddy system.
I completly agree that we first have to go to MIPS64 before we really can attack big iron problems. The current Linux/MIPS kernel design limits the memory to the size of KSEG0 which is 512mb. Putting Linux into a 64bit universe we'd extend that design limit to the size of XKSEG0, or 1TB for current silicon.
Ralf
Quoted 3 lines · William J. Earl
I expect that much of the work will gradually happen in linux, once
the remaining small-CPU-count issues are resolved. Right now, there is no shortage of interesting problems to attack. :-)
Actually I'd hate it if Linux'd be ``finished'' ...
Quoted 8 lines
One possible way to approach the large-CPU-count space with linux
is to indeed run multiple linux kernels, one per node in a ccNUMA machine, and add a distributed OS layer at a fairly high level. If it is not underway already somewhere, I would expect someone to take up the project as a graduate school project. Some systems of this sort have been built or attempted, with varying degrees of success. Given the linux bias toward small and simple, a linux-based distributed OS might actually work.
I don't know the exact technical details but something similar has already been done back in '95 (?) when some guys in Australia ported Linux to Fujitsu's AP1000. That's of course a different architecture and a multikernel approach makes much more sense there.
Quoted 3 lines
One important ingredient in such a system, which would be
valuable immediately for clusters, would be a efficient distributed volume manager and file system.
Hans Reiser is currently working on a new filesystem with alot of fresh ideas. In some aspects what he is aiming at is similar to XFS, in some aspects not. He basically started on a white sheet of paper with his design, so his team's code isn't contaminated by old ideas. His work looks pretty promising. Among his plans is also the implementation of a distributed filesystem. Current benchmarks are looking pretty good, in fact in some cases extremly good. The URL to checkout for interested people is
http://idiom.com/~beverly/reiserfs.html.
Quoted 3 lines
In any case, trying to port linux straight to a large ccNUMA
(or even a large SMP) system would be a lot of effort for limited return at present.
Explecitly not talking about the MIPS port - I think it makes sense in working on porting to machines beyond what we currently scale to. But I agree about the ``large'' thing.
Ralf
I think everyone is suggesting the same general path, and I'm gratified to know that the buddy system code anticipates non-contiguous physical memory.
I think you'd want to extend the design limit (XKSEG0) beyond 1 TB to handle the next rev of silicon. I'd suggest 64 TB (48 bits) as the appropriate goal.
g
Signature · 1 line
Greg Chesson
Quoted 3 lines · On Jul 18, 3:57am, [email protected] wrote:
Actually I'd hate it if Linux'd be ``finished'' ...
Ancient Chinese proverb: "House finish; man die."
g
Signature · 1 line
Greg Chesson
Quoted 7 lines · Greg Chesson
I think everyone is suggesting the same general path, and I'm gratified to know that the buddy system code anticipates non-contiguous physical memory.
I think you'd want to extend the design limit (XKSEG0) beyond 1 TB to handle the next rev of silicon. I'd suggest 64 TB (48 bits) as the appropriate goal.
As far as the maximum amount of RAM goes, no problem. The only assumption which I make is that the entire available memory is visible in XKSEG0, whatever it's size is. That means the actual design limit is the maximum possible size of XKSEG0 for the MIPS 64bit architecture which is 2^62 bytes.
I assume this also means the size limit of the XKUSEG has been extended to 48 bits. There we actually a somewhat more limiting design limit since we only have three level page tables for 64 bit architectures which limits us to a maximum mappable size of
PAGE_SIZE * (PAGE_SIZE / sizeof(pte_t)) ^ 3
which for PAGE_SIZE = 4 kbytes and sizeof(pte_t) = 8 bytes is 512gb.
(And actually things are somewhat more complex.)
Ralf
Quoted 10 lines · On Sat, Jul 18, 1998 at 04:44:03AM +0200, [email protected] wrote:
Quoted 3 lines · Greg Chesson
I think you'd want to extend the design limit (XKSEG0) beyond 1 TB to handle the next rev of silicon. I'd suggest 64 TB (48 bits) as the appropriate goal.
As far as the maximum amount of RAM goes, no problem. The only assumption which I make is that the entire available memory is visible in XKSEG0, whatever it's size is. That means the actual design limit is the maximum possible size of XKSEG0 for the MIPS 64bit architecture which is 2^62 bytes.
Ooops, the real limit is of course how much memory can be presented in XKPHYS which is a bit less, just 2^58 bytes or 4 exabytes. XKSEG or XKSSEG would be used for kernel virtual memory which usually isn't used very much under Linux - it's virtual, but not swappable.
Ralf