TSCs are a no-no on i386

This small patch fixes a long-standing problem for bare i386 CPUs. These don't have TSCs and so, a recent 2.4.x kernel will simply halt the

machine leaving a text like "This CPU has no TSC feature (ie was compiled for Pentium+) but I do depend on it".

Please apply. Worst to say, even Debian seems to start using i486+ features (ie. libstdc++5 is SIGILLed on Am386 because there's no "lock" insn available)...

Quoted 2 lines

This small patch fixes a long-standing problem for bare i386 CPUs. These don't have TSCs and so, a recent 2.4.x kernel will simply halt the

Actually i486 processors and some Pentium-like clones have none, too.

Please apply. Worst to say, even Debian seems to start using i486+ features (ie. libstdc++5 is SIGILLed on Am386 because there's no "lock" insn available)...

g++ >=3.2 use 486-specific instructions in order to do atomic operations in C++ code. 3.3 includes a 386 version of the same code, but it's not possible to use a mixture of the two (ie, code compiled with a "486" g++ will not work on the "386" version), and so to keep ABI compatibility with the other distributions it's necessary to break 386. The "obvious" solution (dragging this back on topic) is a kernel patch to emulate 486 instructions on a 386.

...
Please apply. Worst to say, even Debian seems to start using i486+ features (ie. libstdc++5 is SIGILLed on Am386 because there's no "lock" insn available)...

MfG, JBG
...

Quoted 4 lines · Jan-Benedict Glaw

...
Please apply. Worst to say, even Debian seems to start using i486+ features (ie. libstdc++5 is SIGILLed on Am386 because there's no "lock" insn available)...

Shouldn't the 486 emulation in the latest 386 kernel images in Debian unstable take care of this?

Quoted 8 lines · Adrian Bunk

Quoted 4 lines · Jan-Benedict Glaw

...
Please apply. Worst to say, even Debian seems to start using i486+ features (ie. libstdc++5 is SIGILLed on Am386 because there's no "lock" insn available)...

Shouldn't the 486 emulation in the latest 386 kernel images in Debian unstable take care of this?

What emulation?

Note that this can also allow Step-A 486's to correctly run multi-thread applications since cmpxchg has a wrong opcode on this early CPU.

Don't use this to enable multi-threading on an SMP machine, the lock atomicity can't be guaranted!

Quoted 5 lines

Note that this can also allow Step-A 486's to correctly run multi-thread applications since cmpxchg has a wrong opcode on this early CPU.

Don't use this to enable multi-threading on an SMP machine, the lock atomicity can't be guaranted!

The bigger problem (and certainly with some of the cmov emulation hacks I've seen) is getting the security checking right when you need to reprocess the user data - another reason to do it in userspace with a preload 8)

...and IFF those new opcodes bring _that_ much performance, then we should think about another Debian distribution for i686-linux. Up to now, I was really proud of having _one_ distribution that's basically capable of running on all and any machines I own...

[...]

Quoted 4 lines

...and IFF those new opcodes bring _that_ much performance, then we should think about another Debian distribution for i686-linux. Up to now, I was really proud of having _one_ distribution that's basically capable of running on all and any machines I own...

Do as Mandrake (and I suppose another distros), and put optimized libraries in /lib/i686, /lib/mmx, /lib/sse, /usr/lib/i686, etc. New ld.so supports mapping different versions of library depending on arch.

Well... For sure, I can write a LD_PRELOAD lib dealing with SIGILL, but how do I enable it for the whole system. That is, I'd need to give LD_PRELOAD=xxx at the kernel's boot prompt to have it as en environment variable for each and every process?

That sounds a tad inelegant to me. Really, I'd prefer to see libstdc++ be compiled for i386 ...

Quoted 4 lines · Jan-Benedict Glaw

Well... For sure, I can write a LD_PRELOAD lib dealing with SIGILL, but how do I enable it for the whole system. That is, I'd need to give LD_PRELOAD=xxx at the kernel's boot prompt to have it as en environment variable for each and every process?

/etc/ld.preload

Quoted 2 lines

That sounds a tad inelegant to me. Really, I'd prefer to see libstdc++ be compiled for i386 ...

True

Quoted 7 lines · Jan-Benedict Glaw

Quoted 4 lines

Well... For sure, I can write a LD_PRELOAD lib dealing with SIGILL, but how do I enable it for the whole system. That is, I'd need to give LD_PRELOAD=xxx at the kernel's boot prompt to have it as en environment variable for each and every process?

Quoted 4 lines

/etc/ld.preload

Quoted 2 lines

That sounds a tad inelegant to me. Really, I'd prefer to see libstdc++ be compiled for i386 ...

What is a runtime library doing with a TSC? That's the basic problem. These things are for operating systems and, last time I checked, the 'C' runtime libraries weren't (but maybe GNU changed that definition, no?)

Quoted 4 lines

Well... For sure, I can write a LD_PRELOAD lib dealing with SIGILL, but how do I enable it for the whole system. That is, I'd need to give LD_PRELOAD=xxx at the kernel's boot prompt to have it as en environment variable for each and every process?

/etc/ld.preload

...
That sounds a tad inelegant to me. Really, I'd prefer to see libstdc++ be compiled for i386 ...

...and IFF those new opcodes bring _that_ much performance, then we should think about another Debian distribution for i686-linux. Up to now, I was really proud of having _one_ distribution that's basically capable of running on all and any machines I own...

MfG, JBG

Quoted 4 lines

...
That sounds a tad inelegant to me. Really, I'd prefer to see libstdc++ be compiled for i386 ...

Quoted 4 lines

...and IFF those new opcodes bring _that_ much performance, then we should think about another Debian distribution for i686-linux. Up to now, I was really proud of having _one_ distribution that's basically capable of running on all and any machines I own...

The 486 emlation patch for 386 is the way to still allow 386's to run Debian.

To compile libstdc++ for 486 wasn't a performance question - a libstdc++.so.5 compiled for 386 would have meant that C++ binaries compiled on Debian wouldn't run on other Linux distributions and vice versa [1] (it's a bug in libstdc++ that will AFAIR be fixed in gcc 3.4).

Quoted 2 lines

The 486 emlation patch for 386 is the way to still allow 386's to run Debian.

Okay, I'll have a look at it. Where's the 2.6.x version?

Quoted 4 lines

Quoted 2 lines

The 486 emlation patch for 386 is the way to still allow 386's to run Debian.

Okay, I'll have a look at it. Where's the 2.6.x version?

It doesn't exist, but could certainly easily be ported from 2.4.

So to resume, everything can be done through emulation, but that's probably not what we want as a standard for performance reasons. When I have time, I may port it to 2.6, but that's not no my priority list.

Thanks for that. In the meantime, I've started to give a try to the userspace version (using a LD_PRELOAD lib). My current Problem:

amtus:~/sigill_catcher# LD_PRELOAD=./libsigill.so ls
sigill.c:_init():69: sigill started, sigaction() = 0
build.sh  intercept.h  libsigill.so  run.sh  sigill.c  sigill.o
amtus:~/sigill_catcher# LD_PRELOAD=./libsigill.so apt-get update
Illegal instruction

See? It's loaded at the "ls" call, but it seems to be not loaded for apt-get.

Quoted 9 lines

Thanks for that. In the meantime, I've started to give a try to the userspace version (using a LD_PRELOAD lib). My current Problem:

amtus:~/sigill_catcher# LD_PRELOAD=./libsigill.so ls
sigill.c:_init():69: sigill started, sigaction() = 0
build.sh  intercept.h  libsigill.so  run.sh  sigill.c  sigill.o
amtus:~/sigill_catcher# LD_PRELOAD=./libsigill.so apt-get update
Illegal instruction

Quoted 2 lines

See? It's loaded at the "ls" call, but it seems to be not loaded for apt-get.

Remember you need to overload signal setting functions like sigaction. My guess is apt decided to disable your signal and you didnt stop it

On Thu, 2003-07-31 12:51:09 +0100, Alan Cox <[email protected]> wrote in message <[email protected]>:

Quoted 12 lines · Jan-Benedict Glaw

Quoted 11 lines

Thanks for that. In the meantime, I've started to give a try to the userspace version (using a LD_PRELOAD lib). My current Problem:

amtus:~/sigill_catcher# LD_PRELOAD=./libsigill.so ls
sigill.c:_init():69: sigill started, sigaction() = 0
build.sh  intercept.h  libsigill.so  run.sh  sigill.c  sigill.o
amtus:~/sigill_catcher# LD_PRELOAD=./libsigill.so apt-get update
Illegal instruction

See? It's loaded at the "ls" call, but it seems to be not loaded for apt-get.

On Thu, 2003-07-31 14:14:48 +0200, Jan-Benedict Glaw <[email protected]> wrote in message <[email protected]>:

Quoted 14 lines

On Thu, 2003-07-31 12:51:09 +0100, Alan Cox <[email protected]> wrote in message <[email protected]>:

Quoted 12 lines · Jan-Benedict Glaw

Quoted 11 lines

Thanks for that. In the meantime, I've started to give a try to the userspace version (using a LD_PRELOAD lib). My current Problem:

amtus:~/sigill_catcher# LD_PRELOAD=./libsigill.so ls
sigill.c:_init():69: sigill started, sigaction() = 0
build.sh  intercept.h  libsigill.so  run.sh  sigill.c  sigill.o
amtus:~/sigill_catcher# LD_PRELOAD=./libsigill.so apt-get update
Illegal instruction

See? It's loaded at the "ls" call, but it seems to be not loaded for apt-get.

So there we are. Thanks to someone who pointed me to LD_DEBUG et al., I see that my _init() is called too late:

amtus:~/sigill_catcher# LD_DEBUG=all LD_VERBOSE=1
LD_PRELOAD=./libsigill.so apt-get update 2> ld_infos                                 
Illegal instruction
amtus:~/sigill_catcher# grep 'calling init' ld_infos 
00691:  calling init: /lib/libc.so.6
00691:  calling init: /lib/libdl.so.2
00691:  calling init: /lib/libgcc_s.so.1
00691:  calling init: /lib/libm.so.6
00691:  calling init: /usr/lib/libstdc++.so.5

As I guessed, libstdc++5's _init() is called before mine (LD_PRELOADed) _init() function and thus, I cannot intercept this SIGILL, as it seems...

Quoted 18 lines

On Thu, 2003-07-31 14:14:48 +0200, Jan-Benedict Glaw <[email protected]> wrote in message <[email protected]>:

Quoted 3 lines

On Thu, 2003-07-31 12:51:09 +0100, Alan Cox <[email protected]> wrote in message <[email protected]>:

Quoted 1 line · Jan-Benedict Glaw

So there we are. Thanks to someone who pointed me to LD_DEBUG et al., I see that my _init() is called too late:

amtus:~/sigill_catcher# LD_DEBUG=all LD_VERBOSE=1
LD_PRELOAD=./libsigill.so apt-get update 2> ld_infos                                 
Illegal instruction
amtus:~/sigill_catcher# grep 'calling init' ld_infos 
00691:  calling init: /lib/libc.so.6
00691:  calling init: /lib/libdl.so.2
00691:  calling init: /lib/libgcc_s.so.1
00691:  calling init: /lib/libm.so.6
00691:  calling init: /usr/lib/libstdc++.so.5

Quoted 4 lines

As I guessed, libstdc++5's _init() is called before mine (LD_PRELOADed) _init() function and thus, I cannot intercept this SIGILL, as it seems...

You could do what we do with our path sandbox - basically you use a wrapper that setup the environment (prob not needed in your case), and then spawn bash with our sandbox lib preloaded, which then calls whatever should be 'path sandboxed'.

On Thu, 2003-07-31 17:09:19 +0200, Martin Schlemmer <[email protected]> wrote in message <[email protected]>:

Quoted 24 lines · Jan-Benedict Glaw

Quoted 22 lines

On Thu, 2003-07-31 14:14:48 +0200, Jan-Benedict Glaw <[email protected]> wrote in message <[email protected]>:

Quoted 3 lines

On Thu, 2003-07-31 12:51:09 +0100, Alan Cox <[email protected]> wrote in message <[email protected]>:

Quoted 1 line · Jan-Benedict Glaw

So there we are. Thanks to someone who pointed me to LD_DEBUG et al., I see that my _init() is called too late:

amtus:~/sigill_catcher# LD_DEBUG=all LD_VERBOSE=1
LD_PRELOAD=./libsigill.so apt-get update 2> ld_infos                                 
Illegal instruction
amtus:~/sigill_catcher# grep 'calling init' ld_infos 
00691:  calling init: /lib/libc.so.6
00691:  calling init: /lib/libdl.so.2
00691:  calling init: /lib/libgcc_s.so.1
00691:  calling init: /lib/libm.so.6
00691:  calling init: /usr/lib/libstdc++.so.5

As I guessed, libstdc++5's _init() is called before mine (LD_PRELOADed) _init() function and thus, I cannot intercept this SIGILL, as it seems...

Quoted 4 lines

You could do what we do with our path sandbox - basically you use a wrapper that setup the environment (prob not needed in your case), and then spawn bash with our sandbox lib preloaded, which then calls whatever should be 'path sandboxed'.

Care to send some URL? I haven't found it ad hoc by Google...

Quoted 9 lines · Jan-Benedict Glaw

Thanks for that. In the meantime, I've started to give a try to the userspace version (using a LD_PRELOAD lib). My current Problem:

amtus:~/sigill_catcher# LD_PRELOAD=./libsigill.so ls
sigill.c:_init():69: sigill started, sigaction() = 0
build.sh  intercept.h  libsigill.so  run.sh  sigill.c  sigill.o
amtus:~/sigill_catcher# LD_PRELOAD=./libsigill.so apt-get update
Illegal instruction

Quoted 2 lines

See? It's loaded at the "ls" call, but it seems to be not loaded for apt-get.

Remember you need to overload signal setting functions like sigaction. My guess is apt decided to disable your signal and you didnt stop it

Quoted 4 lines · Alan Cox

Quoted 2 lines · Jan-Benedict Glaw

See? It's loaded at the "ls" call, but it seems to be not loaded for apt-get.

Quoted 2 lines

Remember you need to overload signal setting functions like sigaction. My guess is apt decided to disable your signal and you didnt stop it

An application might install its own SIGILL handler to emulate or trap _other_ instructions. To do it properly, you have to chain the handlers.

Not sure how to do this, when you get to the stage of two LD_PRELOAD libraries each wanting to overload sigaction.

The other problem lies with the lock :
When a 486 executes "LOCK ; CMPXCHG", it locks the bus during the whole cmpxchg instruction. If a 386 executes the same code, it will get an exception which will be caught by the emulator. I don't see how we can do such an atomic operation while holding a lock. At best, we would think about a global memory based shared lock during the operation (eg: int bus_lock;), but it's not implemented at the moment, and will only be compatible with processors sharing the same code. Add-on processors, such as co-processors, transputer cards, or DSPs, will know nothing about such a lock emulation. And it would result in even poorer performance of course !

Of course this is not a problem when "lock;cmpxchg" is used only for thread synchronisation on uniprocessor 386s... The lock prefix is irrelevant then.

Perhaps the emulation should refuse to pretend to work on an SMP 386 :)

Quoted 10 lines · Willy Tarreau

The other problem lies with the lock :
When a 486 executes "LOCK ; CMPXCHG", it locks the bus during the whole cmpxchg instruction. If a 386 executes the same code, it will get an exception which will be caught by the emulator. I don't see how we can do such an atomic operation while holding a lock. At best, we would think about a global memory based shared lock during the operation (eg: int bus_lock;), but it's not implemented at the moment, and will only be compatible with processors sharing the same code. Add-on processors, such as co-processors, transputer cards, or DSPs, will know nothing about such a lock emulation. And it would result in even poorer performance of course !

Of course this is not a problem when "lock;cmpxchg" is used only for thread synchronisation on uniprocessor 386s... The lock prefix is irrelevant then.

Perhaps the emulation should refuse to pretend to work on an SMP 386 :)

-- Jamie
-

Quoted 1 line

Willy Tarreau wrote:

The bad op-code for the i386 is cmpxchg. This is what triggers the trap. This can be emulated, although the emulation is not SMP compatible. You worry about this when somebody makes a dual '386 machine ;^). Also, the best performing emulation for any op-codes should be done within the kernel. That way, the invalid-opcode trap works just like the math emulator. You don't need the overhead of calling a user-mode handler. If this is emulation is implimented, then one should also emulate BSWAP and XADD. This makes '486 code compatible with '386 machines.

Quoted 5 lines

Note that this can also allow Step-A 486's to correctly run multi-thread applications since cmpxchg has a wrong opcode on this early CPU.

Don't use this to enable multi-threading on an SMP machine, the lock atomicity can't be guaranted!

That is of course the other problem with this approach - you can't really get the intended results without some extremely heavyweight code (using an IPI to halt all CPU's, doing the access and then resuming them)

Quoted 6 lines

Quoted 5 lines

Note that this can also allow Step-A 486's to correctly run multi-thread applications since cmpxchg has a wrong opcode on this early CPU.

Don't use this to enable multi-threading on an SMP machine, the lock atomicity can't be guaranted!

Quoted 4 lines

That is of course the other problem with this approach - you can't really get the intended results without some extremely heavyweight code (using an IPI to halt all CPU's, doing the access and then resuming them)

Hopefully there are not too many SMP-486 machines out there ;-).

Quoted 10 lines · Adrian Bunk

Quoted 7 lines · Jan-Benedict Glaw

Quoted 3 lines

Please apply. Worst to say, even Debian seems to start using i486+ features (ie. libstdc++5 is SIGILLed on Am386 because there's no "lock" insn available)...

Shouldn't the 486 emulation in the latest 386 kernel images in Debian unstable take care of this?

What emulation?

486 emulation
CONFIG_CPU_EMU486
When used on a 386, Linux can emulate 3 instructions from the 486 set. This allows user space programs compiled for 486 to run on a 386 without crashing with a SIGILL. As any emulation, performance will be
very low, but since these instruction are not often used, this might
not hurt. The emulated instructions are:
- bswap (does the same as htonl())
- cmpxchg (used in multi-threading, mutex locking)
- xadd (rarely used)

On Wed, 2003-07-30 20:45:29 +0200, Adrian Bunk <[email protected]>

Quoted 8 lines

When used on a 386, Linux can emulate 3 instructions from the 486 set. This allows user space programs compiled for 486 to run on a 386 without crashing with a SIGILL. As any emulation, performance will be
very low, but since these instruction are not often used, this might
not hurt. The emulated instructions are:
- bswap (does the same as htonl())
- cmpxchg (used in multi-threading, mutex locking)
- xadd (rarely used)

libstdc++ (and it's main user, apt-get) break at a LOCK insn.

There is no such instruction. Skip LOCK prefix and decode again, you'll get either cmpxchg or xadd (or cmpxchg8b, but then it does not work on i486 too).

Quoted 3 lines · Petr Vandrovec

There is no such instruction. Skip LOCK prefix and decode again, you'll get either cmpxchg or xadd (or cmpxchg8b, but then it does not work on i486 too).

And if the byte you are looking at was patched by another thread you've blown it. Your emulation can only be so good 8) People do stuff like patching instructions under software decode as a robustness check - its normally pretty amusing

Quoted 4 lines

And if the byte you are looking at was patched by another thread you've blown it. Your emulation can only be so good 8) People do stuff like patching instructions under software decode as a robustness check - its normally pretty amusing

On a uniprocessor 386, this is not a problem. Just disable preemption in the kernel decoder.

Quoted 5 lines

Quoted 4 lines

And if the byte you are looking at was patched by another thread you've blown it. Your emulation can only be so good 8) People do stuff like patching instructions under software decode as a robustness check - its normally pretty amusing

Quoted 2 lines

On a uniprocessor 386, this is not a problem. Just disable preemption in the kernel decoder.

Wrong again. Thats a common myth but you see I can put the instruction on a page so that its executed from a page I just did a read() into in another process. If I'm really bored I'll use O_DIRECT but thats mostly for makign life really bad for non cache coherent setups 8)

I wouldn't be surprised if even real CPUs mis-decode when there's a concurrent DMA into the area they are reading instructions from! :)

Quoted 1 line

On Wed, 2003-07-30 20:45:29 +0200, Adrian Bunk <[email protected]>

And yes, it speeds some workloads a lot. Best to look at code, with these instructions you can do couple of operations without doing IPC to synchronize with other threads.

Quoted 4 lines

...
Please apply. Worst to say, even Debian seems to start using i486+ features (ie. libstdc++5 is SIGILLed on Am386 because there's no "lock" insn available)...

Shouldn't the 486 emulation in the latest 386 kernel images in Debian unstable take care of this?