One little, two little, three little endian...

Hello list,

I want a big endian version of Linux for IPF. Just how crazy am I?

I'm working with a group of customers and software developers who are migrating away from HP's proprietary MPE/iX operating system on the HPe3000 systems that HP announced the discontinuance of one year ago today. Many of these people are already using HP-UX to some degree, and most are interested in the possibility of using Linux in the future.

Of course both MPE/iX and HP-UX are big-endian environments, and many of the people I talk to would be very interested in having an endian-compatible Linux that would run on big (i.e. IA-64) HP servers. For this group, x86 compatibility is probably a non-issue.

I'm interested in any comments that come to mind. I'm most interested in just how complex the task would be from a technical point of view, either to make the system buildable either way, or possibly supporting a per-process endian bit (which would be cool, but probably a lot more work).

I'm reasonably familiar with the IA-64 architecture and know most of what there is to know about PA-RISC.

Thanks,

Gavin
--
Gavin Scott
Vice President
Allegro Consultants, Inc.
[email protected]

Hi Gavin,

Gavin> Hello list, I want a big endian version of Linux for IPF. Gavin> Just how crazy am I?

Very! ;-)

Seriously though: running big-endian processes is the easy part, the hard part is developing (& maintaining) big-endian versions of all libraries. Not only is it a lot of work, but it would also fragment the ia64 linux market, so I'd consider it a Bad Thing (TM).

Gavin> Of course both MPE/iX and HP-UX are big-endian environments, Gavin> and many of the people I talk to would be very interested in Gavin> having an endian-compatible Linux that would run on big Gavin> (i.e. IA-64) HP servers. For this group, x86 compatibility Gavin> is probably a non-issue.

It would be a lot easier to add an hp-ux compatibility layer on top of ia64 linux. The layer would emulate the (big-endian) hp-ux system calls and then you would install the big-endian hp-ux libraries on top of that (e.g., in the /emul/ia64-hpux/ tree). Of course, I'm ignoring legal issues here as copying the hp-ux libraries would presumably require some sort of license. But legal issues aside, this is doable and has been done many times before (e.g., alpha linux can run (some) Tru64 binaries, sparc linux can run sunos/solaris(?) binaries, etc.).

Gavin> I'm interested in any comments that come to mind. I'm most Gavin> interested in just how complex the task would be from a Gavin> technical point of view, either to make the system buildable Gavin> either way, or possibly supporting a per-process endian bit Gavin> (which would be cool, but probably a lot more work).

Actually, adding a per-process endian bit would be easy. In fact, even without, a little-endian program can turn on the big-endian bit as long as it's prepared to deal with the consequences. The hard part is really the libraries.

	--david

My reaction is why?

If the HP-UX people are converting to Linux you're talking about porting the applications anyway. Why don't you just bite the bullet and port the applications to little endian Linux and be done with it. I know there's a tendency to shudder when contemplating an endian port but it's really not that difficult these days. GCC is pretty good about flagging most of the issues that get you in trouble so if you resolve all warnings carefully most of the job is done. It's actually easier to port from big endian to little than vice versa so you'd be faced with the simpler problem.

If the apps have assembly language it's going to be a complete port anyway so porting to little endian doesn't add any work there.

Data compatibility is the only real gotcha I can think of and that's just a translation issue. Potentially creating the translation utilities will be a problem but I predict it's a smaller problem than coming up with a big endian IA64 Linux.

Don Dugger
"Censeo Toto nos in Kansa esse decisse." - D. Gale
[email protected]

how about running parisc-linux? :^)

randolph

Randolph Chung
Debian GNU/Linux Developer, hppa/ia64 ports
http://www.tausq.org/

Have you ever tried to import wrong-endian data?

I thought people gave up trying to do that on 16-bit machines. I've been in more endian flip-flops than I care to remember. (Olivetti flip-flopped about every 2 years in the early 90s, talk about customer nightmares...)

Data conversion has to be re-adressed by *every* application because of padding. I think the work mostly cannot be amortized over lots of customers.

I agree with david. User space libs need to be migrated from the desired environment. See how far a per-process approach gets first.

grant

Two points:

1) I didn't say data conversion was easy just that the one time effort of converting would be easier in the long run than the on-going effort of creating and maintaining, for ever, big endian user libraries.

2) I thought PA-RISC was a 32-bit processor which means porting to IA64, assuming you're porting to 64-bit addressing (and if not why are you porting to IA64 in the first place), will entail data incompatibilities independent of endian-ness so you still have to do the data conversion.

It comes down to engineering trade offs and I still say that the one time port will be easier in the long run.

Don Dugger
"Censeo Toto nos in Kansa esse decisse." - D. Gale
[email protected]

this is not true, many (most?) pa20 processors are 64-bit.

randolph

Randolph Chung
Debian GNU/Linux Developer, hppa/ia64 ports
http://www.tausq.org/

all PA2.0 processors are 64-bit. it's possible, though not recommended, to run a 64-bit Linux kernel on all PA2.0 boxes HP produced. however there's no 64-bit userspace for the Linux/PA port yet, so this is a rather academic discussion.

Revolutions do not require corporate support.

David a quick question. Even though the library problem is overcomed, how would signal handling be accomplished by an application running in big-endian. Doesn't the kernel fill in the siginfo and context while runing in little endian?

- Mario.

Mario> David a quick question. Even though the library problem is Mario> overcomed, how would signal handling be accomplished by an Mario> application running in big-endian. Doesn't the kernel fill in Mario> the siginfo and context while runing in little endian?

Yes, signal-handling would have to be adjusted, just like we do for an x86 process (the signal stack format is somewhat different for hp-ux at any rate, so it's more than just endianness).

	--david

FYI,
the Intel Fortran compiler runtime
has features for endianness conversion.
See the environment variable F_UFMTENDIAN.
See Intel Fortran Compiler Users's Guide, Customizing Compilation Environment, Environment Variables, Little-endian to Big-endian Conversion.

Of course ISVs today have faced the endianness problem for many years. So most aps have already designed in features for data conversion.
You should push back very hard as to exactly what application really has a problem here. Clearly Fortran formatted data is not an issue, for e.g. It would be pretty surprising if data generated for enterprise applications were not already routinely interchanging with the little endian Windows environment.

To create an entirely new ABI is not a small deal.

Hopefully this is helpful,
Rich Altmaier, SGI

The largest problem is the fuzziness of the "one time". Customers performing these migrations are likely to want to run in parallel through their development and testing processes, and these can take years in commercial environments.

During this period the issue of data compatibility can be quite important, especially if they are doing any kid of periodic or real-time replication of transactions between the old and new environments.

It's (somewhat) easier to migrate binary data for an application if you don't have to identify all fields and their types in order to byte-juggle them during conversion. At least in cases where storage-compatible compilers and architecture exist.

And then there's the technically-less-relevant but the-customer-is-always-right issue that many people ask for, or feel better with, or simply demand, a target system that matches their own personal endian bigotry[1].

Gavin

[1] For example preferring "big endian" (a.k.a. God's Own Byte Order) over "little endian" (a.k.a. Satan's Humorous Little Joke)

Thanks for all the comments. They have been quite helpful.

Gavin