Why Plan 9 C compilers don't have asm("")

Because it's messy and unnecessary. Break this into asmlinkbuild, asmlink.c, asmlink.h and asmlink.S, chmod +x asmlinkbuild, run it, and behold a 6.
__________________________________________________________________

#..........................................................

# asmlinkbuild

gcc -c asmlink.S
gcc -o asmlinked asmlink.c asmlink.o
asmlinked

cat asmlinkbuild asmlink.S asmlink.c > asmlink.post

/* ***************************************************
asmlink.S

int bla (int ha, int hahaha, int uh) ;

That does...

push uh
push hahaha
push ha

*/

.globl bla
bla:

        add 4(%esp), %eax
        add 8(%esp), %eax
        add 12(%esp), %eax
        ret

/* ******************************************** asmlink.c */
#include "asmlink.h"

int main () {
        printf("%d\n", bla(1, 2 , 3 ) ) ;

}

_________________________________________________________________

That's with the GNU tools, without asm(), and without proper declaration of printf, as is my tendency. I don't actually return an int either, do I? LAAETTR.

In other words, if you know the push sequence of your C compiler's function calls, you don't need asm("");. x86 Gcc is "push last declared first, return in EAX". Plan 9 guys, not surprisingly, seem to prefer to keep C as C, and asm as asm. I encountered this while trying to build Linux 1.2.13 with current GNU tools. It breaks on changes in GNU C asm()'s. Rather a silly thing to break on, eh?

I don't think this is much less clear than the : "=r" $0; stuff, if at all. This thing didn't take as long to code as it did to construct this post. Perhaps the C-labels-in-asms optimizes better. I doubt if it's by much, or if it's worth it.

Oops. I didn't include asmlink.h in the above, except as a comment in asmlink.S. Here it is by itself...

/* ********************************************asmlink.h*/ int bla (int ha, int hahaha, int uh) ;

Another easy win from Plan 9 that's related to this but that is not in evidence here is that this thing on Plan 9 could build asmlinkbuild for itself on the fly based on #pragma's in the headers that simply state what library they are the header for. This to me is so obviously an improvement to the usual state of affairs, an ornate system of dead-ends, as to be depressing. The guys that wrote UNIX don't do such things to themselves anymore.

Rick Hohensee
:; cLIeNUX /dev/tty11 11:00:14 /
:;d

ABOUT        LGPL         boot         device       log      subroutine
ABOUT.Linux  Linux        command      floppy       mounts       suite
GPL          README       configure    guest        owner        temp
H3nix        RIGHTS       dev          help         source

:; cLIeNUX /dev/tty11 22:44:25 /
:;

You are very much forgetting _inline_ asm. And if you think that's unimportant for performance, well, as Al would say, go back playing with Hurd.

OG.

There isn't such a crippling difference between straight-line and code with unconditional branches in it with modern processors. In fact, there's very little measurable difference.

If you're looking for something to blame hurd performance on I'd suggest the entire design of Mach, not inline asm vs procedure calls. Tossing a few context switches into calls is a lot more expensive.

} > In other words, if you know the push sequence of your C compiler's } > function calls, you don't need asm("");.
}
} You are very much forgetting _inline_ asm. And if you think that's } unimportant for performance, well, as Al would say, go back playing } with Hurd.

What are advantages of this approach ????

Andrey Panin            | Embedded systems software engineer
[email protected]        | PGP key: http://www.orbita1.ru/~pazke/AndreyPanin.asc

Followup to:  <[email protected]>
By author:    Cort Dougan <[email protected]>
In newsgroup: linux.dev.kernel

That's not where the bulk of the penalty of a function call comes in (and it's a call/return, not an unconditional branch.) The penalty comes in because of the additional need to obey the calling convention, and from the icache discontinuity.

Not to mention that certain things simply cannot be done that way.

-hpa

<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt

In article <[email protected]>,

Oh, the small details get to you eventually.

And it's not just the "call" and "ret" instructions. They _do_ hurt, even on modern CPU's, btw. They tend to break up the prefetching, and often mean that you cannot do as good of a instruction mix.

But there's an even more serious problem: a function call in C is a VERY heavy operation as far as the compiler is concerned. It's a major sequence point, and the compiler doesn't know what memory locations are potentially dead etc.

Which means that the compiler has to save everything that might be relevant to memory, and depending on the calling convention has to assume that registers are trashed. And when you come back, you have to re-load everything again, on the assumption that the function might have changed state.

You also often have issues like reloading the gp pointer on many 64-bit architectures, where functions can be in different "domains", and returning from an unknown function means that you have to do other nasty setup in order to get at your global data.

And trust me, it's noticeable. On alpha, a fast function call _should_ be a a simple two-cycle thing - branch and return. But because of practical linker issues, what the compiler ends up having to generate for calls to targets that it doesn't know where they are is

- load a 64-bit address off the GP area that the linker will have fixed up.
- do an indirect branch to that address
- the callee re-loads the GP with _its_ copy of the GP if it needs any global data or needs to call anybody else.
- we return to the caller
- the caller reloads its GP.

Your theoretical two cycles that the CPU could follow in the front end and speculate around turns into multiple loads, a indirect branch and about 10 instructions. And that's without any of the other effects even being taken into account. No matter _how_ good the CPU is, that's going to be slower than not doing it.

[ And yes, I know there are optimizing linkers for the alpha around that
improve this and notice when they don't need to change GP and can do a straight branch etc. I don't think GNU ld _still_ does that, but who knows. Even the "good" Digital compilers tended to nop out unnecessary instructions rather than remove them, causing more icache pressure on a CPU that was already famous for needing tons of icache ]

Now, you could get around a bit of this by allowing for special calling conventions. Gcc actually has this for some details - namely the "register arguments" part, which actually makes for much more readable code (that's my main personal use for it - never mind the fact that it is probably faster _too_).

But gcc doesn't have a good "you can re-order this call wrt other stuff" setup, and gcc lacks the ability to change the calling convention on-the-fly ("this function will not clobber any registers").

Try it and see. There are good reasons for "inline asm", not the least of which is that it often makes the produced code much more readable.

And if you never look at the produced assembler code, then you'll never have a fast system. Really. Compilers can do only so much. People who understand what the end result is makes for a difference.

Now, you could probably argue that instead of inline asms we should have more flexibility in doing a per-callee calling convention. That would be good too, no question about it.

Linus

Hey folks,

Just a quick reminder: don't feed the troll. He's very hungry.

-ben

Under ISO C rules, this is illegal, since you must have a proper prototype in scope when calling variable argument functions. In fact, I have worked on several GCC ports, where the compiler uses a different calling sequence for variable argument functions than it does for normal functions. For example, on the Mips, if the first argument is floating point and the number of arguments is not variable, it is passed in a FP register, instead of an integer register. For variable argument functions, everything is passed in the integer registers.

Michael Meissner, Red Hat, Inc.  (GCC group)
PMB 198, 174 Littleton Road #3, Westford, Massachusetts 01886, USA
Work:	  [email protected]		phone: +1 978-486-9304
Non-work: [email protected]	fax:   +1 978-692-4482

I didn't know that, but...

You seem to be saying the use of assumptions about args passing is non-standard. I know. It's more standard than GNU extensions to C though, C_labels_in_asms in particular, and even in your examples it appears that the particular function abusing these tenets will know what it can expect from a particular compiler, since it knows what it's arguments are. It can't know what it can expect from any compiler. This perhaps is where #ifdef comes in, or similar. Well, it's not more standard than GNU, but the differences would be less detailed in the case of just dealing with various args passing schemes, and there may be some compiler-to-compiler overlap, where there won't be any with stuff like C_labels_in_asms.

It's illegal to not declare main() as int. I don't know of a unix that actually passes anything but a byte to the calling process. I got flamed mightily for this in comp.unix.programmer until people ran some checks on thier big Real Unix(TM) boxes of various types. Linux won't pass void either, you have to get a 0 at least. Compliance is subjective. It's easier when things make sense.

Rick Hohensee
www.clienux.com

Doing this is a losing game. How many different platforms does Linux currently run on? Do you know exactly what the ABI is for each of the machines? What happens when Linux is ported to a new machine? My point is:

	extern int printf (const char *, ...);
	printf ("%d %d\n", 1, 2);

and

	extern int my_printf (const char *, int, int);
	my_printf ("%d %d\n", 1, 2);

under some ABIs will pass arguments completely differently and as I said, I have worked on various GCC ports that did this, so it is not a theoretical possibility.

Yes, that is an artifact of the original UNIX implementation on the PDP-11 (16 bit ints, signal number is passed back in one byte, and the return value in another byte).

Michael Meissner, Red Hat, Inc.  (GCC group)
PMB 198, 174 Littleton Road #3, Westford, Massachusetts 01886, USA
Work:	  [email protected]		phone: +1 978-486-9304
Non-work: [email protected]	fax:   +1 978-692-4482

I'm talking about _modern_ processors, not processors that dominate the modern age. This isn't x86. I don't believe that even aggressive re-ordering will cause a serious hit in performance on function calls. Unconditional branches are definitely predictable so icache pre-fetches are not more complicated that straight-line code.

Measurement is more important, though. I've rejected a number of optimizations from people (including many of my own) that were "obvious enhancements" because of what they showed in real-world measurements. If it doesn't run faster, despite the theory being "right", it's worthless.

Linus mentioned Alpha specifically. I don't see how any of the things he said were x86-centric in any way shape or form.

All of his examples are entirely accurate on sparc64 for example, and to even moreso his Alpha commentary can nearly directly be applied to the MIPS.

Calls suck ass, even on modern cpus. I've seen several hundreds of cycles go out of the fault path by eliminating them. If you can kill a leaf level call, you can avoid saving the whole frame, and on Sparc (for example) this means saving a potential window spill trap which can be quite costly.

Calls are less simple than branches to do (via prediction etc.) at "zero cost" because usually there is a write port necessary (to write the call instruction's address into the "return" register).

Let's not even start talking about calls in PIC code :-)

Later,
David S. Miller
[email protected]

In article <[email protected]>,

NONE of my examples were about the x86.

I gave the alpha as a specific example. The same issues are true on ia64, sparc64, and mips64. How more "modern" can you get? Name _one_ reasonably important high-end CPU that is more modern than alpha and ia64..

On ia64, you probably end up with function calls costing even more than alpha, because not only does the function call end up being a synchronization point for the compiler, it also means that the compiler cannot expose any parallelism, so you get an added hit from there. At least with other CPU's that find the parallelism dynamically they can do out-of-order stuff across function calls.

Did you READ my mail at all?

Most of these "unconditional branches" are indirect, because rather few 64-bit architectures have a full 64-bit branch. That means that in order to predict them, you either have to do data-prediction (pretty much nobody does this), or you have a branch target prediction cache, which works very well indeed but has the problem that it only works for stuff in the cache, and the cache tends to be fairly limited (because you need to cache the whole address - it's more than a "which direction do we go in").

There are lots of good arguments for function calls: they improve icache when done right, but if you have some non-C-semantics assembler sequence like "cli" or a spinlock that you use a function call for, that would _decrease_ icache effectiveness simply because the call itself is bigger than the instruction (and it breaks up the instruction sequence so you get padding issues).

Linus

That seems amazingly dumb. You'd think a new processor design would optimize parallel computation over calls, but what do I know?

This is something I don't get: I never understood why 32bit risc designers were so damn obstinate about "every instruction fits in 32 bits" and refused to have "call 32 bit immediate given in next word" not to mention a "load 32bit immediate given in next word".
Note, the superior x86 instruction set has a 5 byte call immediate.

I think anywhere that you have inner loop or often used operations that are short assembler sequences, inline asm is a win - it's easy to show for example, that the Linux asm x86 macro semaphore down is three times as fast as
a called version. I wish, however
that GCC did not use a horrible overly complex lisplike syntax and that there was a way to inline functions written in .S files.

And the feature is way too easy to abuse - same argument here as in the threads argument.
It's a far better thing to not need a semaphore at all than to rely on handcoded semaphore down to make your poorly synchronized design sort-of perform.

They try to. Take a look at the trace cache on the Pentium IV. They certainly seem to have badly screwed the chip design up elsewhere but the trace cache has some very clever ideas in it

Some do. After all there is nothing unrisc about

	call [ip]++

Maybe the idea died with the PDP-11 designers 8)

Alan

Sparc has such an instruction, in fact the instruction and the 32-bit immediate fit in a single 32-bit instruction (since the immediate is guarenteed to be modulo 4 you have some extra bits for the instruction opcode itself).

Later,
David S. Miller
[email protected]

the 32 bit MIPS (R3K series, at least) has a 32 bit instruction which loads a 16 bit immediate (which fits within the instruction itself). thus to load a 32 bit number takes two instructions. since the instructions are all 32 bits and must live on a multiple of 4 bytes, this is as compact as you can get given the alignment constraint.

note that x86 is also fussy about alignment in various cases, e.g., double-precision floats.

lisp syntax is extremely simple. i am not sure what GCC does to make it complex.

J o h a n  K u l l s t a m
[[email protected]]
Don't Fear the Penguin!

Yes, that was not easy to miss. I was simply being clear. The plan9 compiler, thus its take on inline asm, doesn't run on ia64 and alpha as far as I can see from the latest release.

} NONE of my examples were about the x86.
}
} I gave the alpha as a specific example. The same issues are true on } ia64, sparc64, and mips64. How more "modern" can you get? Name _one_ } reasonably important high-end CPU that is more modern than alpha and } ia64..
}
} On ia64, you probably end up with function calls costing even more than } alpha, because not only does the function call end up being a } synchronization point for the compiler, it also means that the compiler } cannot expose any parallelism, so you get an added hit from there. At } least with other CPU's that find the parallelism dynamically they can do } out-of-order stuff across function calls.

Yes, that's how I saw it didn't relate to the topic at hand. I suggested measurement rather than theory to determine whether the branch washes out or not. "Everyone knows" is a much weaker statement than "I can show".

} Did you READ my mail at all?

I definitely agree there. If you need an instruction or two that the compiler doesn't offer then it's a loss to call a function and inline asm is worthwhile. If this is a common enough case it's worth the compiler adding inline asm support. I'm not sure how often that is. My own subjective experience has been that calls to such code are rare enough that they fall well into the realm of optimization the uncommon case.

I've used inline asm gratuitously in linux (it's peppered all over the ppc code) because I had the feature. I don't think that's a strong argument for adding it to a compiler that doesn't support it, though.

} There are lots of good arguments for function calls: they improve icache } when done right, but if you have some non-C-semantics assembler sequence } like "cli" or a spinlock that you use a function call for, that would } _decrease_ icache effectiveness simply because the call itself is bigger } than the instruction (and it breaks up the instruction sequence so you } get padding issues).

GNU ld does it with the "-relax" flag.

But you're absolutely right about the nopping -- removing the nops would require debug info and EH info to be re-coded. The later being a matter of correctness. This is a bit nastier than I ever cared to deal with.

r~

I don't see it as being all that nasty. It's only nasty if you make the compiler default to the long form - because it's so hard to reduce the size later (branches around call-sites etc). But if you make the compiler default to the short form, you're ok - you can trivially expand the short form later without any of the same problems.

Isn't this what mips32 used to do too - it didn't have the GP reload issue, but it has 16-bit branch offsets if I remember correctly. So they ended up adding trampoline branches on demand later. And a 16-bit branch offset is a lot more constraining than a 20-bit one. 128kB is not a huge jump, but 2MB is getting to be pretty far in most applications..

So:
- you _always_ generate the fast case. A call is always considered to be a short one, simple "bsr", no GP change, no nothing.
- you generate a trampoline as well, and teach the linker to go through the trampoline if it has to do a far call (one trampoline per target, not per caller). Think of it as a "overflow" case for a .rel20.
- If it's not a weak reference and you can satisfy it at link-time, you can obviously just get rid of the trampoline then and there. This takes care of all the normal "intra-GP" things.

Sure, if you want to be fancy, you also drop unused GOT entries for anything that ends up not having a trampoline.

So the above takes care of correctness. For bonus points, you allow the user to specify "this will be a far call (or weak)" as an attribute, which you use on intra-modules code. Which is almost entirely library interfaces, so you'd have the system header files use this so that shared library calls don't get the hit of the trampoline.

As far as I can see, this should take care of about 99% of all static jumps. Most applications have less than 2MB code-space, and the only real reason for the long form is for intra-module calls which tend to be fairly well specified (ie they are declared in standard headers etc).

Sure, you could sometimes get the slower case: more than a 2MB offset within a module, so that you'd have to use the trampoline, or if you're lazy and don't update the headers for dynamically linked libraries. But even then there would be the potential for icache win. And you could always have a "-mlarge-model" compiler option for those cases, so if you notice that you lose on this optimization, you just disable it.

No?

Linus

Why do I feel like I just read a description on the various real-mode x86 memory models?

mrc

     Mike Castle      [email protected]      www.netcom.com/~dalgoda/
    We are all of us living in the shadow of Manhattan.  -- Watchmen
fatal ("You are in a maze of twisty compiler features, all different"); -- gcc

This is all well and good if one is designing an ABI from scratch. You can't retrofit it onto the current ABI, however. Not without pain anyway.

The call-clobbered GP means that your trampoline has to play games in order to get the GP restored when coming back from an intra module call. Which means a new stack frame. Which is a tad more than you bargined for, really. I can't see that kind of heavyweight solution being any better than the nops.

r~

Ahh, only if you do my optimization of sharing trampolines among users. And you're right, that won't work.

But if you don't do that, you don't need a stack frame. You just reload GP and jump back to the caller.

And assuming most calls don't need the trampoline (and hey, they really shouldn't), you're still way ahead. The only thing you lost was the icache win of re-using the trampoline (and a few cycles for scheduling and the extra short branch).

Think of it as nothing more than a branch prediction thing - you predict that you can take a short branch, and emit the long-branch code out-of-line.

So the code would be roughly (this is not how the compiler would see it, this is the very last stage of outputting the actual assembly. Nothing else needs to know):

	...
	bsr $26,trampoline	// linker overflow case
retpoint:
	....

trampoline:

	ldq $27,fn($gp)		// load the full address
	jsr $26,($27)		// branch to it
	ldgp $29,($26)		// reload our GP
	jsr $31,retpoint	// and go back to where we came from.

And the linker can just use the special .rel20 thing to turn the bsr into a direct call when it can.

Overhead when it cannot: one extra "bsr", one extra "jsr" back, and the lack of scheduling. You lost two cycles and maybe a pipeline stall or something (branching around is never nice, even if it's unconditional).

But you only lose this on misprects. And you can have a pretty high prediction accuracy, even with just static knowledge.

And when you _do_ predict right, you're going to win in icache footprint, code size (and because you can drop the trampoline for non-weak symbolds the executable size also goes down) and cycles.

That still doesn't look complicated to me. Of course, it clearly does depend on whether I'm right that you can fairly easily get 99% prediction accuracy. And I could just be full of sh*t.

Linus

Hmm. Yes, that could work. We'd still be changing the ABI, since the original source "bsr foo" would really mean "bsr foo+skip ldgp". But perhaps one that wouldn't matter for all practical purposes.

This would need to be done with a new relocation type so that old linkers that didn't handle this sort of thing choke, but that's not a big deal.

If I find the time I may give it a shot and see what sort of effect it has on typical code.

r~

Ahh.. Well, that would be more of a linker relocation ABI change, not really a run-time ABI change. And as it needs a new relocation type anyway, because of the overflow case (I assume the current .rel20 complains loudly when it overflows, right?), this should be ok.

And the code to jump over ldgp obviously exists already if there's a "-relax" linker option.

Linus