Generic light-weight syscall.

pa,

Any thoughts about how one should implement some type of lightweight syscall for our glibc to use?

We already have the makings of a simple system for SET_THREAD_SELF (e.g. seting cr27, the thread register, from userpsace), and I want to extend this to:

exchange_and_add (volatile uint32_t *mem, int val)
atomic_add (volatile uint32_t *mem, int val)
compare_and_swap (volatile long int *p, long int oldval, long int newval)

o---> libc
--> exchange_and_add
==> Params into kernel
==> disable interrupts on the current processor

    ==> take a semaphore to keep other cpu's out
    ==> do work 
    ==> release semaphore

==> reenable i-bit
--> back into userspace and done.

This _must_ be very very fast, and appear atomic to userspace.

Perhaps adding other pages instead of 0xE0 for set_thread_self might be the simplest way to do this? The area after the linux gateway page perhaps? Which seems to be the start of the next 4k page? Would this be possible? e.g. branch to 0x1000 (not 0x100 which is the current syscall branch).

c.

I have lots of thoughts ;-)

Sure. Sounds like a great idea.

I'd say a spinlock rather than a semaphore. And likely a special-cased one too.

I'd say we should keep doing stuff on our existing gateway page until we exhaust it. We've got plenty of space -- 248 instruction slots left before 0xE0, and a lot of space left after the syscall handler.

On a related subject, fast gettimeofday is always a popular idea. I'm not sure of all the ramifications of, for example, mapping a user-read-only, system-writable data page after the gateway page (can't put the data on the existing gateway page; a page that can do privilege promotion isn't readable/writable). If we have only one CPU update the data on that page, time shouldn't go backwards ... right?

"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

Spill the beans!

Okay, we talked about this, choose a random 4-bits from the incoming address and hash this to select one of 16 spinlocks that keep other cpu's out. This, as you indicated, should scale to more than 4 CPU's and make LaMont happy.

I'll see if I can fit _all_ the operations into jumps in that area. If I can't then I'll see what the performance of implementing "one" operation in kernel and using that to do the rest atomically.

If you map the page and make CPU 0 update the date, then I'll write the userspace interface for gettimeofday.

I'm not sure how we would do the check for 'do we see fast gettimeofday' but it might be that we include a magic value there and check for it? Other arches must have solved this.

c.

Talked to Rik Van Riel about fast gettimeofday and he indicated that it's not doable since you can't guarantee your process will get scheduled on another CPU whose clock is out of sync by more than X and get negative time. Though I imagine you were talking about having one CPU update one page with time on it... and then other CPU's read this? LaMont notes that there is no requirement from the PA design that CPU's clock at _exactly_ the same frequency or have monotonically incrementing clocks at the right rate. Could you explain the idea you have a bit more?

c.

Yes we can. We can sync CR16 across CPUs within a few CPU cycles. I've described this before on parisc-linux.

"sync" means figure out the difference between CR16 on several CPUs and using CPU 0 as the reference.

correct. IIRC 9000/870 have seperate clock sources.
But all the boxes we support to date have exactly one clock source. The multi-cell boxes (like superdome) will have multiple sources and I don't know how to handle those - maybe a "not quite so fast" gettimeofday().

hth,
grant

It might be too costly to do the sync'ing all the time, and too costly for a fast gettimeofday to do a sync at the polling point.

The whole point behind fast gettimeofday is that userspace apps that want to do timestamping on a _very_ accurate granularity (e.g. nanosecondes) can get monotically incrementing numbers on each gettimeofday. Do we even have such a fast clock on PA? What is the fastest clock across the most boxes?

c.

You know, you don't even need kernel help for this. According to page 2-5 of the Kane book, the Interval Timer is accessible by non-privileged instructions.

"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

Isn't this going to be different for all CPU's? Which means if you get scheduled around you might see negative moving time?

c.

We discussed this in the Black Thorn ... record what value you last returned to the user and never return less than that.

"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

Not very nice if the difference between CPUs is significant. You could find yourself checking time on CPU A, doing some work, getting moved to CPU B, checking time, and finding you apparently did all that in 0ns.

Richard

Willy and myself talked about this, you just return 1ns in the case where you know that _something_ must have taken _some_ amount of time (e.g. the insns that make up fast gettimeofday) so you couldn't have done it in zero time :)

I also just noticed (thanks to jejb) that cr16 is readable from userspace since we leave PSW-S cleared. So the following works:

        unsigned long cr16;
        asm("mfctl %%cr16, %0" : "=r" (cr16) : );
        printf("cr16=%lu\n",cr16);

---

carlos@firin:~$ ./test_cr16_read 
cr16=3544734914
carlos@firin:~$ ./test_cr16_read 
cr16=4230470557
carlos@firin:~$ ./test_cr16_read 
cr16=2337868642
carlos@firin:~$ 
---

It overlaps pretty fast though. Userspace could do the translation and hold on to a 'last_tick' value and then return a really small value if we get scheduled onto a negatively-drifted CPU? Do all CPU's update cr16?

Willy, I just realized that the following trail of function calls doesn't really work under smp:

glibc ->
gettimeofday ->
syscall + syscall table lookup ->
sys32_gettimeofday ->
do_gettimeofday ->
gettimeoffset ( returns 0 in smp)

With the comment I don't quite understand:
linux-2.4/arch/parisc/kernel/time.c
/*
* FIXME: This won't work on smp because jiffies are updated by cpu
* 0.
* Once parisc-linux learns the cr16 difference between
* processors,
* this could be made to work.
*/

Does that mean that our time granularity drops drastically in the SMP case? We loose usec! (2.4 and 2.5 kernels)

c.

this only gets you the CR16 for *that* CPU.
Not useful since CR16 isn't syncronized at powerup between CPUs. At least I don't think that's the case...maybe Kirk Bresniker knows how that works. We've talked about this before on parisc-linux.

...

This is what I've been trying to explain.

not by much.

grant

Depends what people want to use it for. I couldn't use it to time how long some syscall took, for example. But if we zap the microseconds part on smp anyway, that's irrelevant.

Richard

Well there was talk of, on irc, of exporting the CPU# through cr26.

- Read cr16
- Read cr26
- Read cr16

If you weren't rescheduled the delta should be a minimal number of ticks (not taking into account nearness to unsigned overflow). You then use this tick value to calculate a delta via a table of CPU specific offsets. If the tick is far out from the last read, then you assume a reschedule and loop. Perhaps terminating on the third try with a default delta?

I mean the easiest way, as willy notes is to jump into the kernel with a fast syscall, disable interrupts, get the CPU# and index into a table of cpu vs. last known good tick.

However, since we _can_ read cr16 (as willy wrote in an email I totally failed to read, sorry willy!) from userspace on most systems (not on 705's and 710's but they aren't SMP anyway and we can change the method there) we are trying to make good use of that.

So there are a variety of ways:

1. Fast-syscall similar to set_thread_register, clears interrupt bit, indexes into cpu# table to get last good known tick and returns it to userspace (after cleaning up the mess).

2. Userspace does triple read and loop until it looks like we (in a lockless fashion) have both the right CPU# and latest tick which we can use to update our CPU/tick table.

3. Export a page with '(tick_val & mask) | CPU#' or 'tick_val xor CPU#'. You then use this to determine the CPU and tick atomically in a single read.

We know #1 works. We don't know if #2 is faster than #1 (or stable), anyone wishing to comment please do :) Number 3 would loose resolution in, first by loping off bits, or by having near tick values that overlap and you aren't able to find the CPU# from the xor'd quantity.

To add a datapoint to #2, on a PA8700 650Mhz I see:
Anyone wishing to run this test on another box, please do...
---
#include <stdio.h>
#include <time.h>

#define LOOPS 1000

int main(void){
double avg_diff=0.0;
unsigned long cr16a, cr16b, cr26;
int i=LOOPS;

   while(i>0){
      asm("mfctl %%cr16, %%r26   \n\
           mfctl %%cr26, %%r24   \n\
           mfctl %%cr16, %%r25   \n\
           stw %%r26, %0         \n\
           stw %%r25, %1         \n\
           stw %%r24, %2" : "=m" (cr16a), "=m" (cr16b), "=m" (cr26) :);
      printf("cr26=%lu\ncr16a=%lu\ncr16b=%lu\ndiff=%lu\n",
		cr26,cr16a,cr16b,cr16b-cr16a);
      avg_diff+=(double)(cr16b-cr16a);
      printf("avg_diff=%f\n",avg_diff);
      i--;

}

printf("Average ticks per back/back cr16 read (%lu loops) = %f\n",

		LOOPS,avg_diff/(double)LOOPS);

exit(0);
}
---
<snip>
cr26=4294967295
cr16a=3348684869
cr16b=3348684881
diff=12
avg_diff=11435.000000
Average ticks per back/back cr16 read (1000 loops) = 11.435000 carlos@firin:/mnt/flaire/src/linux-2.5/arch/parisc/kernel$
---

c.

A500-5X (PA8600@550Mhz):

cr26=4294934527
cr16a=4175244475
cr16b=4175244488
diff=13
avg_diff=12993.000000
Average ticks per back/back cr16 read (1000 loops) = 12.993000

HTH,

Thibaut VARENE
The PA/Linux ESIEE Team
http://pateam.esiee.fr/

useconds gets zapped in gettimeofday on SMP systems. I'm working on trying to fix this :)

c.

On Mon, Jul 28, 2003 at 06:45:51PM +0100, Matthew Wilcox wrote:
...

If they all use the same clock source they won't drift.

cr16

One needs kernel help in determining the difference between CPUs and handling CR16 rollover (mostly a problem for 32-bit machines). But then user space can read CR16 and "normalize" the time.

grant

Note - some multiple cpu hardware intentionally clocks the cpu's at slightly different rates to limit RFI generation.
Not sure if any HP-PARISC machines do such clocking. Mike

Why not add a flag to syscall() which indicates whether this is a "fast" syscall or a "slow" syscall, and based on this, decide whether to do all the register spilling, etc when entering the kernel? Then we can implement the atomic ops as additional "syscalls"....

I would think that there is at least some amount of logic that needs to be there everytime you enter/exit the kernel, irregardless of whether you are doing a "fast syscall" (i.e. no need to save the processor state, etc) or a regular one... i would hope we don't need to have two copies of that logic.

Carlos had some concerns that this means fast syscalls (or regular ones perhaps) will always incur a mispredicted branch and/or extra stack manipulations that may not be needed..... but i'm not yet convinced that there is enough overhead to make this a problem. what do others think?

thanks,
randolph

Randolph Chung
Debian GNU/Linux Developer, hppa/ia64 ports
http://www.tausq.org/

I don't think enough syscalls are "fast" to make this worth doing. Most of the things people are talking about for lightweight syscalls are things that could/should be done in userspace ... except that you need to be privileged to use them. So the optimum way to solve them is to have a special calling convention and todo the absolute minimum amount of work. Setting cr26 is a great example because we would do it in userspace if the architecture let us.

Does anyone have a better definition for lightweight syscall? How about "Cannot sleep, called frequently, must be fast"?

"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk