i386: vgetcpu(), take 2
RFC PATCH 1/1 i386: vgetcpu(), take 2
Use a GDT entry's limit field to store per-cpu data for fast access from userspace, and provide a vsyscall to access the current CPU number stored there.
Questions:
1. Will the vdso relocation patch break this?
2. Should the version number of the vsyscall .so be incremented?
Test program using the new call:
/* vgetcpu.c: get CPU number we are running on.
* build kernel with vgetcpu patch first, then:
* gcc -o vgetcpu vgetcpu.c <srcpath>/arch/i386/kernel/vsyscall-sysenter.so */
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
extern int __vgetcpu(void);
int main(int argc, char * const argv[])
{
printf("cpu: %u\n", __vgetcpu());
return 0;
}--- arch/i386/kernel/cpu/common.c | 3 +++ arch/i386/kernel/head.S | 8 +++++++- arch/i386/kernel/vsyscall-getcpu.S | 25 +++++++++++++++++++++++++ arch/i386/kernel/vsyscall-int80.S | 2 ++ arch/i386/kernel/vsyscall-sysenter.S | 2 ++ arch/i386/kernel/vsyscall.lds.S | 1 + 6 files changed, 40 insertions(+), 1 deletion(-)
arch/i386/kernel/cpu/common.c +3 -0
@@ -642,6 +642,9 @@ void __cpuinit cpu_init(void) ((((__u64 )stk16_off) <<32 ) &0xff00000000000000ULL ) | (CPU_16BIT_STACK_SIZE -1 ); +/* Set up GDT entry for per-cpu data */ + *(__u64 *)(&gdt[27 ]) |= cpu; + cpu_gdt_descr->size =GDT_SIZE -1 ; cpu_gdt_descr->address = (unsigned long )gdt;
arch/i386/kernel/head.S +7 -1
@@ -525,7 +525,13 @@ ENTRY(cpu_gdt_table) .quad 0x004092000000ffff /* 0xc8 APM DS data */ .quad 0x0000920000000000 /* 0xd0 - ESPFIX 16-bit SS */ - .quad 0x0000000000000000 /* 0xd8 - unused */ + + /* + * Use a GDT entry to store per-cpu data for user space (DPL 3.) + * 32-bit data segment, byte granularity, base 0, limit set at runtime. + */ + .quad 0x0040f20000000000 /* 0xd8 - for per-cpu user data */ + .quad 0x0000000000000000 /* 0xe0 - unused */ .quad 0x0000000000000000 /* 0xe8 - unused */ .quad 0x0000000000000000 /* 0xf0 - unused */
arch/i386/kernel/vsyscall-getcpu.S new file +25 -0
@@ -0,0 +1,25 @@ +/* + * vgetcpu + * This file is #include'd by vsyscall-*.S to define them after the + * vsyscall entry point. The kernel assumes that the addresses of these + * routines are constant for all vsyscall implementations. + */ + +#include <linux/errno.h> + + .text + .org __kernel_rt_sigreturn+32,0x90 + .globl __vgetcpu + .type __vgetcpu,@function +__vgetcpu: +.LSTART_vgetcpu: + movl $-EFAULT,%eax + movl $((27<<3)|3),%edx + lsll %edx,%eax + jnz 1f + andl $0xff,%eax +1: + ret +.LEND_vgetcpu: + .size __vgetcpu,.-.LSTART_vgetcpu +
arch/i386/kernel/vsyscall-int80.S +2 -0
@@ -51,3 +51,5 @@ __kernel_vsyscall: * Get the common code for the sigreturn entry points. */ #include "vsyscall-sigreturn.S" + +#include "vsyscall-getcpu.S"
arch/i386/kernel/vsyscall-sysenter.S +2 -0
@@ -120,3 +120,5 @@ SYSENTER_RETURN: * Get the common code for the sigreturn entry points. */ #include "vsyscall-sigreturn.S" + +#include "vsyscall-getcpu.S"
arch/i386/kernel/vsyscall.lds.S +1 -0
@@ -57,6 +57,7 @@ VERSION __kernel_vsyscall; __kernel_sigreturn; __kernel_rt_sigreturn; + __vgetcpu; local: *; };
Signature · 2 lines
Chuck "You can't read a newspaper if you can't read." --George W. Bush
Quoted 3 lines · Chuck Ebbert
Use a GDT entry's limit field to store per-cpu data for fast access from userspace, and provide a vsyscall to access the current CPU number stored there.
very nice idea! I thought of doing sys_get_cpu() too, but my idea was to use the scheduler to keep a writable [and permanently pinned, per-thread] VDSO data page uptodate with the current CPU# [and other interesting data]. Btw., do we know how fast LSL is on modern CPUs?
Quoted 2 lines
Questions:
1. Will the vdso relocation patch break this?
no - why should it?
Quoted 1 line
2. Should the version number of the vsyscall .so be incremented?
i've Cc:-ed the glibc folks.
but my gut feeling is that we should add a proper sys_get_cpu() syscall as well, and thus make this a transparent syscall, not dependent on the availability of the vDSO.
arch/i386/kernel/vsyscall-getcpu.S line 22
Quoted 9 lines
arch/i386/kernel/vsyscall-getcpu.S new file +9
-0+__vgetcpu: +.LSTART_vgetcpu: + movl $-EFAULT,%eax + movl $((27<<3)|3),%edx + lsll %edx,%eax + jnz 1f + andl $0xff,%eax +1: + ret
this needs unwinder annotations as well to make this a proper DSO, so that for example a breakpoint here does not confuse gdb.
also, would be nice to do something like this in 64-bit mode too.
Ingo
Quoted 5 lines · Ingo Molnar
Quoted 1 line · Chuck Ebbert
Use a GDT entry's limit field to store per-cpu data for fast access from userspace, and provide a vsyscall to access the current CPU number stored there.
very nice idea! I thought of doing
sys_get_cpu()too, but my idea was to use the scheduler to keep a writable [and permanently pinned, per-thread] VDSO data page uptodate with the current CPU# [and other interesting data]. Btw., do we know how fast LSL is on modern CPUs?
a quick check on two p2/p4 boxes gives the cycle numbers below. syscall/io times for comparison. Not that cheap, but still only ~1/4 of a syscall...
P4 P2
123 39 {movl $-47,%%eax ; movl $((27<<3)|3),%%edx ; lsll %%edx,%%eax ; jnz 1f ; andl $0xff,%%eax ; 1: ;} (average: 155)
959 287 {movl $20,%%eax ; int $0x80 ; # getpid() } (average: 983) 475 153 {movl $20,%%eax ; call *vsyscall ; # getpid() } (average: 519)
333 586 {outb %%al,$0x80;} (average: 369)
3572 1181 {outb %%al,$0x80;outb %%al,$0x80;} (average: 3628) 6755 1557 {outb %%al,$0x80;outb %%al,$0x80;outb %%al,$0x80;} (average: 6866)
P2:
cpu family : 6
model : 5
model name : Pentium II (Deschutes)
stepping : 2
cpu MHz : 400.982
cache size : 512 KB
P4:
cpu family : 15
model : 4
model name : Intel(R) Celeron(R) CPU 2.53GHz
stepping : 1
cpu MHz : 2533.270
cache size : 256 KB
Quoted 1 line
but my gut feeling is that we should add a proper
sys_get_cpu()syscall
Yes - this should be for more or less all arch's.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401Quoted 3 lines · Paul Jackson
Quoted 1 line
but my gut feeling is that we should add a proper
sys_get_cpu()syscallYes - this should be for more or less all arch's.
The whole point of the original implementation is to do a fast architecture specific call. A slow generic call isn't very useful.
-Andi
Quoted 7 lines · Andi Kleen
Quoted 3 lines · Paul Jackson
Quoted 1 line
but my gut feeling is that we should add a proper
sys_get_cpu()syscallYes - this should be for more or less all arch's.
The whole point of the original implementation is to do a fast architecture specific call. A slow generic call isn't very useful.
it's useful in terms of userspace uniformity. It's alot easier to expose such APIs via glibc if there's a generic implementation everywhere. Obviously every arch is encouraged to optimize it into a vsyscall.
Ingo
Quoted 1 line · Ingo
it's useful in terms of userspace uniformity.
Yes. It's an important property of Linux that it provides a common, portable API for all arch's, except where the obvious semantics (not performance) of a call are necessarily arch-specific.
Just coding up system calls for those arch's that happen to run a particular call super-fast, even though the call makes logical sense on all arch's, would lead to API chaos and impede application portability.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401Quoted 3 lines · Chuck Ebbert
Use a GDT entry's limit field to store per-cpu data for fast access from userspace, and provide a vsyscall to access the current CPU number stored there.
Just the CPU alone is useless - you want at least the node too in many cases. Best you use the prototype I proposed earlier for x86-64.
-Andi
Quoted 8 lines · Andi Kleen
Quoted 3 lines · Chuck Ebbert
Use a GDT entry's limit field to store per-cpu data for fast access from userspace, and provide a vsyscall to access the current CPU number stored there.
Just the CPU alone is useless - you want at least the node too in many cases. Best you use the prototype I proposed earlier for x86-64.
just the CPU is fine already in many cases [and the node ID derives from the linear CPU id anyway] - but i agree that in the API we want to include the node-ID too, for NUMA-aware userspace allocators, etc.
Ingo
Quoted 6 lines · Andi Kleen
Quoted 3 lines · Chuck Ebbert
Use a GDT entry's limit field to store per-cpu data for fast access from userspace, and provide a vsyscall to access the current CPU number stored there.
Very clever.
Quoted 3 lines
Just the CPU alone is useless - you want at least the node too in many cases. Best you use the prototype I proposed earlier for x86-64.
Can we use similar mechanism to access pda in vsyscall in x86_64 (by storing the address of pda there). That way the useful variables like cpunumber, nodenumber can be accessed easily without doing cpuid (and without tcache). The system call can take a flag like GET_CPUNUMBER or GET_NODENUMBER or GET_NMICOUNT or if anything new gets added in this structure.
-rohit
Quoted 3 lines
Can we use similar mechanism to access pda in vsyscall in
x86_64(by storing the address of pda there).
You mean in the kernel? %gs prefix is a lot faster than this.
Also the limit is only 20bit, not enough for a full address.
For user space it's useful though, but I don't see any immediate uses other than cpu number and node number. For most purposes glibc TLS (which uses %fs) is probably sufficient.
-Andi
Quoted 6 lines · Andi Kleen
Quoted 2 lines
Can we use similar mechanism to access pda in vsyscall in
x86_64(by storing the address of pda there).You mean in the kernel? %gs prefix is a lot faster than this.
Yes it is. And will work if we are okay to swap to kernel gs in vsyscall code.
Quoted 2 lines
Also the limit is only 20bit, not enough for a full address.
I was thinking of storing it is base address part of the descriptor and then using the memory load to read it in vsyscall. (Keeping the p bit to zero in the descriptor).
Quoted 3 lines
For user space it's useful though, but I don't see any immediate uses other than cpu number and node number. For most purposes glibc TLS (which uses %fs) is probably sufficient.
cpu and node number are really important (for the reasons that you mentioned in your initial mail on vgetcpu). In addition to that I was thinking in terms of having some counters like nmi_count that is already there and per cpu specific.
Besides, not having to use the tcache part in the proposed system call seems to just make the interface cleaner.
-rohit
Quoted 10 lines · Rohit Seth
Quoted 6 lines · Andi Kleen
Quoted 2 lines
Can we use similar mechanism to access pda in vsyscall in
x86_64(by storing the address of pda there).You mean in the kernel? %gs prefix is a lot faster than this.
Yes it is. And will work if we are okay to swap to kernel gs in vsyscall code.
swapgs is only allowed in ring 0.
Quoted 7 lines
Quoted 3 lines
Also the limit is only 20bit, not enough for a full address.
I was thinking of storing it is base address part of the descriptor and then using the memory load to read it in vsyscall. (Keeping the p bit to zero in the descriptor).
I'm still not sure where and for what you want to use this. In user space or in kernel space? And what information should be stored in there?
Quoted 9 lines
Quoted 4 lines
For user space it's useful though, but I don't see any immediate uses other than cpu number and node number. For most purposes glibc TLS (which uses %fs) is probably sufficient.
cpu and node number are really important (for the reasons that you mentioned in your initial mail on vgetcpu). In addition to that I was thinking in terms of having some counters like
nmi_countthat is already there and per cpu specific.
For what would you need nmi count in user space?
Quoted 2 lines
Besides, not having to use the tcache part in the proposed system call seems to just make the interface cleaner.
tcache is still far faster than LSL (which is slower than RDTSCP)
-Andi
Quoted 9 lines · Andi Kleen
Quoted 3 lines · Rohit Seth
I was thinking of storing it is base address part of the descriptor and then using the memory load to read it in vsyscall. (Keeping the p bit to zero in the descriptor).
I'm still not sure where and for what you want to use this. In user space or in kernel space? And what information should be stored in there?
Store the kernel virtual pointer in gdt to access pda in (proposed) vgetcpu in vsyscall. Using this pointer we can easily reach the cpu and node numbers and any other information that is there in pda. For the cpu and node numbers this will get rid of the need to do a serializing operation cpuid.
Does it make any sense?
Quoted 4 lines
Quoted 2 lines
Besides, not having to use the tcache part in the proposed system call seems to just make the interface cleaner.
tcache is still far faster than LSL (which is slower than RDTSCP)
Since we are not using the limits part of the descriptor so lsl will not be needed. Though an indirect load from gdt page will be made.
-rohit
Quoted 11 lines · Rohit Seth
Quoted 9 lines · Andi Kleen
Quoted 3 lines · Rohit Seth
I was thinking of storing it is base address part of the descriptor and then using the memory load to read it in vsyscall. (Keeping the p bit to zero in the descriptor).
I'm still not sure where and for what you want to use this. In user space or in kernel space? And what information should be stored in there?
Quoted 8 lines
Store the kernel virtual pointer in gdt to access pda in (proposed) vgetcpu in vsyscall.
Using this pointer we can easily reach the cpu and
node numbers and any other information that is there in pda. For the cpu and node numbers this will get rid of the need to do a serializing operation cpuid.Does it make any sense?
Ok to spell it out (please correct me if I misinterpreted you). You want to:
- Split PDA into kernel part and user exportable part
- Export user exportable part to ring 3
- Put base address of user exportable part into GDT
- Access it using that.
I don't think it can work because the GDT only supports 32bit base addresses for code/data segments in long mode and you can't put a kernel virtual address into 32bit (only user space there)
And you can't get at at the base address anyways because they are ignored in long mode (except for fs/gs). For fs/gs you would need to save/restore them to reuse them which would be slow.
You can't also just put them into fs/gs because those are already reserved for user space.
Also I don't know what other information other than cpu/node would be useful, so just using the 20 bits of limit seems plenty to me.
-Andi
Quoted 19 lines · Andi Kleen
Quoted 18 lines · Rohit Seth
Quoted 9 lines · Andi Kleen
Quoted 3 lines · Rohit Seth
I was thinking of storing it is base address part of the descriptor and then using the memory load to read it in vsyscall. (Keeping the p bit to zero in the descriptor).
I'm still not sure where and for what you want to use this. In user space or in kernel space? And what information should be stored in there?
Store the kernel virtual pointer in gdt to access pda in (proposed) vgetcpu in vsyscall.
Using this pointer we can easily reach the cpu and
node numbers and any other information that is there in pda. For the cpu and node numbers this will get rid of the need to do a serializing operation cpuid.
Quoted 5 lines
Quoted 1 line
Does it make any sense?
Ok to spell it out (please correct me if I misinterpreted you). You want to:
- Split PDA into kernel part and user exportable part
yes.
Quoted 1 line
- Export user exportable part to ring 3
yes for vsyscall purposes.
Quoted 3 lines
- Put base address of user exportable part into GDT
- Access it using that.
These are the steps that I'm proposing in vgetcpu:
Read the GDT pointer in vgetcpu code path. This is the base of gdt table.
Read descriptor #20 from base.
This is the pointer to user visible part of per cpu data structure.
Please let me know if I'm missing something here.
Just a side note, in your vgetcpu patch, would it be better to return the logical CPU number (as printed in /proc/cpuinfo). Also, I think applications would be interested in knowing the physical package id for cores sharing caches.
Quoted 4 lines
I don't think it can work because the GDT only supports 32bit base addresses for code/data segments in long mode and you can't put a kernel virtual address into 32bit (only user space there)
Really not using the GDT descriptor in terms of loading it in any segment register.
Quoted 7 lines
And you can't get at at the base address anyways because they are ignored in long mode (except for fs/gs). For fs/gs you would need to save/restore them to reuse them which would be slow.
You can't also just put them into fs/gs because those are already reserved for user space.
That is the reason I'm not proposing to alter existing fs/gs.
Quoted 3 lines
Also I don't know what other information other than cpu/node would be useful, so just using the 20 bits of limit seems plenty to me.
physical id (of the package for exmpale) is another useful field. I would also like to see number of interrupts serviced by this cpu, page faults etc. But I think that is a separate discussion.
Thanks,
-rohit
Quoted 11 lines · Rohit Seth
Quoted 2 lines
- Put base address of user exportable part into GDT
- Access it using that.These are the steps that I'm proposing in vgetcpu:
Read the GDT pointer in vgetcpu code path. This is the base of gdt table.
Read descriptor #20 from base.
This is the pointer to user visible part of per cpu data structure.Please let me know if I'm missing something here.
Ok that would probably work, but you would need to export the GDT too.
I still don't see why we should do it - limit should be enough.
Quoted 2 lines
Just a side note, in your vgetcpu patch, would it be better to return the logical CPU number (as printed in
/proc/cpuinfo).
The latest code does that already - i dropped the cpuid code completely and replaced it with LSL.
Quoted 3 lines
Also, I think
applications would be interested in knowing the physical package id for cores sharing caches.
They can always map that themselves using cpuinfo. I would prefer to not overload the single call too much.
Quoted 7 lines
Quoted 6 lines
And you can't get at at the base address anyways because they are ignored in long mode (except for fs/gs). For fs/gs you would need to save/restore them to reuse them which would be slow.
You can't also just put them into fs/gs because those are already reserved for user space.
Quoted 6 lines
That is the reason I'm not proposing to alter existing fs/gs.
Quoted 2 lines
Also I don't know what other information other than cpu/node would be useful, so just using the 20 bits of limit seems plenty to me.
physical id (of the package for exmpale) is another useful field.
Ok I see that, but it could be as well done by a small user space library that reads cpuinfo once and maps given vgetcpu()
On the other hand I got people complaining who need some more topology information (like number of cores/cpus), but /proc/cpuinfo is quite slow and adds a lot of overhead to fast starting programs.
I've been pondering to put some more information about that in the ELF aux vector, but exporting might work too. I suppose exporting would require the vDSO first to give a sane interface.
Quoted 2 lines
would also like to see number of interrupts serviced by this cpu, page faults etc. But I think that is a separate discussion.
Well, the complex mechanism you're proposing above only makes sense if it is established more fields are needed (and cannot be satisfied by reserving a few more segment selectors) I admit I'm not quite convinced yet.
-Andi
Quoted 8 lines · Andi Kleen
Quoted 5 lines · Rohit Seth
Quoted 2 lines
- Put base address of user exportable part into GDT
- Access it using that.These are the steps that I'm proposing in vgetcpu:
Quoted 9 lines
Quoted 6 lines
Read the GDT pointer in vgetcpu code path. This is the base of gdt table.
Read descriptor #20 from base.
This is the pointer to user visible part of per cpu data structure.Please let me know if I'm missing something here.
Ok that would probably work, but you would need to export the GDT too.
Would sgdt not be sufficient? I agree that we will have to end up giving RO access to user for the gdt page.
Quoted 8 lines
I still don't see why we should do it - limit should be enough.
Quoted 2 lines
Just a side note, in your vgetcpu patch, would it be better to return the logical CPU number (as printed in
/proc/cpuinfo).The latest code does that already - i dropped the cpuid code completely and replaced it with LSL.
Ah that is good.
Quoted 6 lines
Quoted 3 lines
Also, I think
applications would be interested in knowing the physical package id for cores sharing caches.They can always map that themselves using cpuinfo. I would prefer to not overload the single call too much.
Yes they can map using /proc/cpuinfo. But if there is any easier mechanism then using /proc then that would help.
I agree that we should not overload a single call (though cpu, package and node numbers do belong in one category IMO). We can have multiple calls if that is required as long as there is an efficient mechanism to provide that information.
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
And you can't get at at the base address anyways because they are ignored in long mode (except for fs/gs). For fs/gs you would need to save/restore them to reuse them which would be slow.
You can't also just put them into fs/gs because those are already reserved for user space.
That is the reason I'm not proposing to alter existing fs/gs.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Also I don't know what other information other than cpu/node would be useful, so just using the 20 bits of limit seems plenty to me.
physical id (of the package for exmpale) is another useful field.
Ok I see that, but it could be as well done by a small user space library that reads cpuinfo once and maps given
vgetcpu()
Why maintain that extra logic in user space when kernel can easily give that information.
Quoted 4 lines
On the other hand I got people complaining who need some more topology information (like number of cores/cpus), but
/proc/cpuinfois quite slow and adds a lot of overhead to fast starting programs.
This is an excellent point.
Quoted 4 lines
I've been pondering to put some more information about that in the ELF aux vector, but exporting might work too. I suppose exporting would require the vDSO first to give a sane interface.
Can you please tell me what more information you are thinking of putting in aux vector?
Quoted 4 lines
Quoted 2 lines
would also like to see number of interrupts serviced by this cpu, page faults etc. But I think that is a separate discussion.
Well, the complex mechanism you're proposing above only makes
complex---no. But sure that it is not as simple as lsl.
Quoted 3 lines
sense if it is established more fields are needed (and cannot be satisfied by reserving a few more segment selectors) I admit I'm not quite convinced yet.
You are absolutely right that the mechanism I'm proposing makes sense only if we have more fields AND if any of those fields are dynamically changing. But this is a generic mechanism that could be extended to share any user visible information in efficient way. Once we have this in place then information like whole cpuinfo, percpu interrupts etc. can be retrieved easily.
-rohit
Quoted 3 lines
Would sgdt not be sufficient? I agree that we will have to end up giving RO access to user for the gdt page.
I meant exporting the GDT page
Quoted 4 lines
I agree that we should not overload a single call (though cpu, package and node numbers do belong in one category IMO). We can have multiple calls if that is required as long as there is an efficient mechanism to provide that information.
The current mechanism doesn't scale to much more calls, but I guess i'll have to do a vDSO sooner or later.
Quoted 2 lines
Why maintain that extra logic in user space when kernel can easily give that information.
It already does.
Quoted 6 lines
Quoted 4 lines
I've been pondering to put some more information about that in the ELF aux vector, but exporting might work too. I suppose exporting would require the vDSO first to give a sane interface.
Can you please tell me what more information you are thinking of putting in aux vector?
One proposal (not fully fleshed out was) number of siblings / sockets / nodes I don't think bitmaps would work well there (and if someone really needs those they can read cpuinfo again)
This is mostly for OpenMP and tuning of a few functions (e.g. on AMD the memory latencies varies with the number of nodes so some functions can be tuned in different ways based on that)
Quoted 6 lines
You are absolutely right that the mechanism I'm proposing makes sense only if we have more fields AND if any of those fields are dynamically changing. But this is a generic mechanism that could be extended to share any user visible information in efficient way. Once we have this in place then information like whole cpuinfo, percpu interrupts etc. can be retrieved easily.
The problem with exposing too much is that it might be a nightmare to guarantee a stable ABI for this. At least it would constrain the kernel internally. Probably less is better here.
Also I'm still not sure why user space should care about interrupts?
-Andi
Quoted 5 lines · Andi Kleen
Quoted 2 lines
Would sgdt not be sufficient? I agree that we will have to end up giving RO access to user for the gdt page.
I meant exporting the GDT page
Yes indeed. That shouldn't be an issue though.
Quoted 5 lines
Quoted 4 lines
I agree that we should not overload a single call (though cpu, package and node numbers do belong in one category IMO). We can have multiple calls if that is required as long as there is an efficient mechanism to provide that information.
Quoted 8 lines
The current mechanism doesn't scale to much more calls, but I guess i'll have to do a vDSO sooner or later.
Quoted 2 lines
Why maintain that extra logic in user space when kernel can easily give that information.
It already does.
I'm missing your point here. How and where?
Quoted 11 lines
Quoted 6 lines
Quoted 4 lines
I've been pondering to put some more information about that in the ELF aux vector, but exporting might work too. I suppose exporting would require the vDSO first to give a sane interface.
Can you please tell me what more information you are thinking of putting in aux vector?
One proposal (not fully fleshed out was) number of siblings / sockets / nodes I don't think bitmaps would work well there (and if someone really needs those they can read cpuinfo again)
This is exactly the point, why do that expensive /proc operation when you can do a quick vsyscall and get all of that information. I'm not sure if Aux is the right direction.
Quoted 11 lines
This is mostly for OpenMP and tuning of a few functions (e.g. on AMD the memory latencies varies with the number of nodes so some functions can be tuned in different ways based on that)
Quoted 6 lines
You are absolutely right that the mechanism I'm proposing makes sense only if we have more fields AND if any of those fields are dynamically changing. But this is a generic mechanism that could be extended to share any user visible information in efficient way. Once we have this in place then information like whole cpuinfo, percpu interrupts etc. can be retrieved easily.
Quoted 4 lines
The problem with exposing too much is that it might be a nightmare to guarantee a stable ABI for this. At least it would constrain the kernel internally. Probably less is better here.
There will be (in all probability) requests to include as much as possible, but I think that should be manageable with sensible API.
Quoted 2 lines
Also I'm still not sure why user space should care about interrupts?
Okay. I just cooked that example for some monitoring process to find out the interrupts /sec on that CPU. But as you mentioned above sibling, sockets, nodes, flags, and even other characteristics like current p-state are all important information that will help applications sitting in user land (even if some of them will be used only couple of times in the life of a process).
Side note: I don't want to delay the vgetcpu call into mainline because of this discussion (as long as there is no cpuid and tcache in that call).
-rohit
Quoted 8 lines · Rohit Seth
Quoted 8 lines
Quoted 4 lines
I agree that we should not overload a single call (though cpu, package and node numbers do belong in one category IMO). We can have multiple calls if that is required as long as there is an efficient mechanism to provide that information.
The current mechanism doesn't scale to much more calls, but I guess i'll have to do a vDSO sooner or later.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Why maintain that extra logic in user space when kernel can easily give that information.
It already does.
I'm missing your point here. How and where?
In /proc/cpuinfo.
Suresh and others even put a lot of thought into how to present the information there.
Or did you just refer to the overhead of writing a /proc parser?
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
I've been pondering to put some more information about that in the ELF aux vector, but exporting might work too. I suppose exporting would require the vDSO first to give a sane interface.
Can you please tell me what more information you are thinking of putting in aux vector?
Quoted 8 lines
Quoted 4 lines
One proposal (not fully fleshed out was) number of siblings / sockets / nodes I don't think bitmaps would work well there (and if someone really needs those they can read cpuinfo again)
This is exactly the point, why do that expensive /proc operation when you can do a quick vsyscall and get all of that information. I'm not sure if Aux is the right direction.
It's already used for this at least (hwcap etc.)
vDSO might be better too, but I haven't thought too much about it yet
Quoted 12 lines
Quoted 12 lines
This is mostly for OpenMP and tuning of a few functions (e.g. on AMD the memory latencies varies with the number of nodes so some functions can be tuned in different ways based on that)
Quoted 6 lines
You are absolutely right that the mechanism I'm proposing makes sense only if we have more fields AND if any of those fields are dynamically changing. But this is a generic mechanism that could be extended to share any user visible information in efficient way. Once we have this in place then information like whole cpuinfo, percpu interrupts etc. can be retrieved easily.
Quoted 7 lines
Quoted 4 lines
The problem with exposing too much is that it might be a nightmare to guarantee a stable ABI for this. At least it would constrain the kernel internally. Probably less is better here.
There will be (in all probability) requests to include as much as possible,
Yes but that doesn't mean all these requests make sense and should be actually followed :)
Quoted 1 line
but I think that should be manageable with sensible API.
Not sure. Leaner interfaces are really better here.
It's one of the lessons I learned from libnuma - i provide a lot of tools, but nearly all people are perfectly satisfied with the total basics. So it's better to start small and only add stuff when there is really a clear use case.
Quoted 6 lines
Okay. I just cooked that example for some monitoring process to find out the interrupts /sec on that CPU. But as you mentioned above sibling, sockets, nodes, flags, and even other characteristics like current p-state are all important information that will help applications sitting in user land (even if some of them will be used only couple of times in the life of a process).
Ok you want faster monitoring applications? Some faster way than /proc for some stuff probably makes sense - but I don't think shared mappings are the right way for it.
There's still a lot of other possibilities for this like relayfs or binary /proc files
Quoted 2 lines
Side note: I don't want to delay the vgetcpu call into mainline because of this discussion
I'll probably delay it after 2.6.18
Quoted 2 lines
(as long as there is no cpuid and tcache in that
call).
What do you not like about tcache?
-Andi
Quoted 16 lines · Andi Kleen
Quoted 14 lines · Rohit Seth
Quoted 13 lines
Quoted 4 lines
I agree that we should not overload a single call (though cpu, package and node numbers do belong in one category IMO). We can have multiple calls if that is required as long as there is an efficient mechanism to provide that information.
The current mechanism doesn't scale to much more calls, but I guess i'll have to do a vDSO sooner or later.
Quoted 2 lines
Why maintain that extra logic in user space when kernel can easily give that information.
It already does.
Quoted 7 lines
Quoted 1 line
I'm missing your point here. How and where?
In
/proc/cpuinfo.Suresh and others even put a lot of thought into how to present the information there.
That part I know very well :)
Quoted 1 line
Or did you just refer to the overhead of writing a /proc parser?
Yes exactly.
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 7 lines
Quoted 5 lines
I've been pondering to put some more information about that in the ELF aux vector, but exporting might work too. I suppose exporting would require the vDSO first to give a sane interface.
Can you please tell me what more information you are thinking of putting in aux vector?
One proposal (not fully fleshed out was) number of siblings / sockets / nodes I don't think bitmaps would work well there (and if someone really needs those they can read cpuinfo again)
Quoted 8 lines
Quoted 3 lines
This is exactly the point, why do that expensive /proc operation when you can do a quick vsyscall and get all of that information. I'm not sure if Aux is the right direction.
It's already used for this at least (hwcap etc.)
vDSO might be better too, but I haven't thought too much about it yet
I like the vDSO interface.
Quoted 4 lines
Quoted 3 lines
There will be (in all probability) requests to include as much as possible,
Quoted 8 lines
Yes but that doesn't mean all these requests make sense and should be actually followed :)
Quoted 1 line
but I think that should be manageable with sensible API.
Not sure. Leaner interfaces are really better here.
yes. some sample implementation will be good here. I'll try.
Quoted 5 lines
It's one of the lessons I learned from libnuma - i provide a lot of tools, but nearly all people are perfectly satisfied with the total basics. So it's better to start small and only add stuff when there is really a clear use case.
I'm not sure smaller libnuma would have gone too far at that time. Though in this case you are starting small by just giving the vgetcpu system call. And then we are going to look at other exportable data.
Quoted 8 lines
Quoted 7 lines
Okay. I just cooked that example for some monitoring process to find out the interrupts /sec on that CPU. But as you mentioned above sibling, sockets, nodes, flags, and even other characteristics like current p-state are all important information that will help applications sitting in user land (even if some of them will be used only couple of times in the life of a process).
Quoted 6 lines
Ok you want faster monitoring applications? Some faster way than /proc for some stuff probably makes sense - but I don't think shared mappings are the right way for it.
There's still a lot of other possibilities for this like relayfs or binary /proc files
/proc et.al just add overhead in retrieving this information. For things that are easily available in kernel vDSO will help.
Quoted 6 lines
Quoted 3 lines
Side note: I don't want to delay the vgetcpu call into mainline because of this discussion
I'll probably delay it after 2.6.18
Why?
Quoted 5 lines
Quoted 2 lines
(as long as there is no cpuid and tcache in that
call).What do you not like about tcache?
It just does not sound like a right interface. Why should an app be giving the last time value that it asked for the same information. User wants cpu, package and node numbers and those are the three parameters that should be there. Besides if we are using lsl then the latency part of cpuid is already gone so no need to optimize this any more.
Though this will be good interface to export jiffies ;-)
-rohit
Quoted 3 lines
It just does not sound like a right interface. Why should an app be giving the last time value that it asked for the same information.
First this information comes with a good-before date stamp so it's natural. Otherwise the application will never pick up when the scheduler decides to schedule it somewhere else, which would be bad.
And that came from conversation with application developers.
A: We want something to get the current node
me: how fast does it need to be?
B: we will cache it anyways.
Problem is that normally the application can't do a good job at doing the cache because it doesn't have a fast way to do time stamping (gettimeofday would be too slow and it's the fastest timer available short of having a second thread that sleeps and updates a counter)
But the vsyscall incidentially knows this because of it sharing data with vgettimeofday(), so it can do the job for the application
Quoted 6 lines
User
wants cpu, package and node numbers and those are the three parameters that should be there. Besides if we are using lsl then the latency part of cpuid is already gone so no need to optimize this any more.Though this will be good interface to export jiffies ;-)
No - jiffies don't have a defined unit and might even go away on a fully tickless kernel.
If we just exported jiffies you would get lots of HZ dependent programs.
-Andi
Quoted 8 lines · Andi Kleen
Quoted 2 lines
It just does not sound like a right interface. Why should an app be giving the last time value that it asked for the same information.
First this information comes with a good-before date stamp so it's natural. Otherwise the application will never pick up when the scheduler decides to schedule it somewhere else, which would be bad.
Though the rescheduling can happen any time. I'm not sure how is tcache going to track rescheduling deterministically. In theory there are always going to be those pathological cases which will be very difficult to get right (with or without tcache).
Quoted 12 lines
And that came from conversation with application developers.
A: We want something to get the current node
me: how fast does it need to be?
B: we will cache it anyways.Problem is that normally the application can't do a good job at doing the cache because it doesn't have a fast way to do time stamping (gettimeofday would be too slow and it's the fastest timer available short of having a second thread that sleeps and updates a counter)
Quoted 4 lines
But the vsyscall incidentially knows this because of it sharing data with
vgettimeofday(), so it can do the job for the application
I think I probably read your patch wrong or an earlier version where user was sending the tcache down to vsyscall. Is user sending the tcache parameter containing the last jiffies down to vsyscall in your latest patch? Could you please point me to latest patch.
I think the system call is going to come with caveat that the information provided by vgetcpu could be stale as the process could have moved to different CPU before returning information.
Quoted 6 lines
Quoted 5 lines
User
wants cpu, package and node numbers and those are the three parameters that should be there. Besides if we are using lsl then the latency part of cpuid is already gone so no need to optimize this any more.
Quoted 7 lines
Quoted 1 line
Though this will be good interface to export jiffies ;-)
No - jiffies don't have a defined unit and might even go away on a fully tickless kernel.
If we just exported jiffies you would get lots of HZ dependent programs.
I agree and that is why I don't think we should export anything relating to jiffies to external world (may be my smiley selection wasn't right). This goes back to my understanding that tcache is user visible.
-rohit