Add a sysconf syscall

From: Andi Kleen <[email protected]>

During testing we found some cases where a library wants to know the number of CPUs for internal tuning, and calls sysconf for that. glibc then reads /proc/stat which is very slow and scales poorly, when the program is executed often.

For example sleepycat DB has this problem.

This patch adds a sysconf system call to avoid this problem. This adds very little code to the kernel, but gives a large speedup.

It is intended to be called from glibc.

It is not a 100% POSIX sysconf -- some values in there are only known to the C library, but supplies all values usefully known to the kernel.

In some cases it is more accurate than glibc can do because it doesn't have to guess. So when some value changes in the kernel it can return the current value.

---
 include/linux/sysconf.h |   23 ++++++++++++++
 kernel/Makefile         |    2 +-
 kernel/sysconf.c        |   77 +++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 101 insertions(+), 1 deletions(-)
 create mode 100644 include/linux/sysconf.h
 create mode 100644 kernel/sysconf.c

What glibc does (opening /proc/stat) is rather stupid, but i think your syscall

Note that these are mostly constant or semi-constant values that are updated very rarely:

If glibc is stupid and reads /proc/stat to receive something it could cache or mmap() itself then hey, did you consider fixing glibc or creating a sane libc?

If we *really* want to offer kernel help for these values even then your solution is still wrong: then the proper solution would be to define a standard *data* structure and map it as a vsyscall *data* page - essentially a kernel-guaranteed data mmap(), with no extra syscall needed!

That could have other uses as well in the future.

That way it would much faster than your current code btw.

So unless there are some compelling arguments in favor of sys_sysconf() that i missed, i have to NAK this:

Quoted 2 lines

What glibc does (opening /proc/stat) is rather stupid, but i think your syscall

I don't think it has any other choice today. [...]

[...] So if anything is "stupid" it is the kernel for not providing efficient interfaces for this.

Quoted 2 lines

Note that these are mostly constant or semi-constant values that are updated very rarely:

That's not true. Most of them are dynamic. Take a look at the patch. Also several of those have changed recently.

Quoted 3 lines

If glibc is stupid and reads /proc/stat to receive something it could cache or mmap() itself then hey, did you consider fixing glibc or creating a sane libc?

Caching doesn't help when you have a workload that exec()s a lot. Also some of these values can change at runtime.

Quoted 4 lines

If we *really* want to offer kernel help for these values even then your solution is still wrong: then the proper solution would be to define a standard *data* structure and map it as a vsyscall *data* page - essentially a kernel-guaranteed data mmap(), with no extra syscall needed!

That's quite complicted because several of those are dynamically computed based on other values. Sometimes they are also not tied to the mm_struct -- like the vsyscall is. For example some of the rlimits are per task, not VM. Basically your proposal doesn't interact well with clone().

Even if we ignored that semantic problem it would need another writable page per task because the values cannot be shared.

Also I never liked the idea of having more writable pages per task, [...]

[...] It increases the memory footprint of a single process more. Given a 4K page is not a lot, but lots of 4K pages add up. Some workloads like to have lots of small processes and I think that's a valuable use case Linux should stay lean and efficient at.

[OK in theory one could do COW for the page and share it but that would get really complicated]

I also don't think it's THAT performance critical to justify the vsyscall.

The simple syscall is already orders of magnitude faster than /proc, and seems to solve the performance problems we've seen completely.

It's also simple and straight forward and simple to userstand and maintain. I doubt any of that would apply to a vsyscall solution.

I don't think the additional effort for a vsyscall would be worth it at this point, unless there's some concrete example that would justify it. Even then it wouldn't work for some of the values.

Also a vsyscall doesn't help on non x86 anyways.

As for efficiency: I thought about doing a batched interface where the user could pass in an array of values to fill in. But at least for the workloads I looked at the application usually calls sysconf() where the array size would be always 1. And that's the standard interface. This might be still worth doing, but I would like to see a concrete use case first.

Quoted 1 line

That could have other uses as well in the future.

Hmm for what?

Note we already have a fast mechanism to pass some thing to glibc in the aux vector.

Quoted 4 lines

That way it would much faster than your current code btw.

So unless there are some compelling arguments in favor of sys_sysconf() that i missed, i have to NAK this:

Well see above for lots of reasons you missed. They are understandable mistakes for someone who first looks at the problem though. I'll attempt to improve the documentation next time.

Quoted 4 lines · Andi Kleen

Quoted 2 lines

What glibc does (opening /proc/stat) is rather stupid, but i think your syscall

I don't think it has any other choice today. [...]

Sure it has a choice: most of the sysconf values are global so creating a permanent mmap()-able data in /tmp or elsewhere and mapping it unless it's inaccessible isnt that particularly hard to cache most of the system-wide constants, now is it?

The CPU count could be updated from CPU hotplug events.

rlimits can be queried using getrlimit().

If at that point glibc said to us: "hey, lets work in making it even faster" then there would be no reason for us to criticise glibc. Right now gibc does not even *try* to be smart about it.

Quoted 8 lines

[...] So if anything is "stupid" it is the kernel for not providing efficient interfaces for this.

Quoted 2 lines

Note that these are mostly constant or semi-constant values that are updated very rarely:

That's not true. Most of them are dynamic. Take a look at the patch. Also several of those have changed recently.

As i said they are mostly constant or semi-constant values that are updated very rarely.

If you think that i am wrong then do me the basic courtesy of mentioning the examples that you think disprove my claim, instead of broadly pointing me to your patch ...

Quoted 6 lines

Quoted 3 lines

If glibc is stupid and reads /proc/stat to receive something it could cache or mmap() itself then hey, did you consider fixing glibc or creating a sane libc?

Caching doesn't help when you have a workload that exec()s a lot. Also some of these values can change at runtime.

Here you modify your claim. Now it's not 'dynamic' anymore but 'can change'?

Which one is it now, "mostly constant or semi-constant values that are updated very rarely" as i claim, "dynamic" as you first claimed or "can change" as you claim here (which is also pretty ambiguous)?

Quoted 9 lines

Quoted 4 lines

If we *really* want to offer kernel help for these values even then your solution is still wrong: then the proper solution would be to define a standard *data* structure and map it as a vsyscall *data* page - essentially a kernel-guaranteed data mmap(), with no extra syscall needed!

That's quite complicted because several of those are dynamically computed based on other values. Sometimes they are also not tied to the mm_struct -- like the vsyscall is. For example some of the rlimits are per task, not VM. Basically your proposal doesn't interact well with clone().

Threads with different rlimits but shared VM are extreme rarities.

Could we please concentrate on the common case? A '-1' in the data page can let the code fall back to some slow path.

Also note that rlimit itself already has an interface to query them:
getrlimit(). So if you do not want the complexity of caching rlimits in the data page you do not have to start with that complexity.

[ But it can be done: modifying the rlimit (which predominantly only happens in
the login process) is rare and happens in a parent task. ]

Quoted 2 lines

Even if we ignored that semantic problem it would need another writable page per task because the values cannot be shared.

Sure most of the values can be shared.

Most of them are exactly one of a low number of variants for all tasks in the system, for typical Linux bootups. I suspect if the data page was COW-ed but inherited across exec() it would behave exactly the way it should be: inherited by all tasks but privatized if a task modifies it for itself and all children.

Also, the first step could be simplified by not exposing rlimits - as rlimits are already exposed via other channels.

Quoted 1 line

Also I never liked the idea of having more writable pages per task, [...]

If you limit it to your own faulty implementation then i'm not surprised that you do not like it.

Quoted 7 lines

[...] It increases the memory footprint of a single process more. Given a 4K page is not a lot, but lots of 4K pages add up. Some workloads like to have lots of small processes and I think that's a valuable use case Linux should stay lean and efficient at.

[OK in theory one could do COW for the page and share it but that would get really complicated]

Why would it get complicated? It's not writable to user-space, that's all that is special about it.

Quoted 1 line

I also don't think it's THAT performance critical to justify the vsyscall.

You apparently did not understand the gist of my point: it's the CONCEPT of adding a crappy catch-all sysconf() interface that sucks IMHO. It's simply bad taste.

If you want to expose data then expose the data intelligently, not some poor system call interface that is also slower.

Quoted 2 lines

The simple syscall is already orders of magnitude faster than /proc, and seems to solve the performance problems we've seen completely.

A handful of horse manure is less stinky than a big pile of it, still i wouldnt want to eat either.

Quoted 2 lines

It's also simple and straight forward and simple to userstand and maintain. I doubt any of that would apply to a vsyscall solution.

Note: i did not suggest a vsyscall, but a vsyscall *data area*. There's a big difference between the two!

It could be passed down to user-space using a new auxiliary entry (see fs/binfmt_elf.c), as it's really part of a task's environment conceptually.

Quoted 5 lines

I don't think the additional effort for a vsyscall would be worth it at this point, unless there's some concrete example that would justify it. Even then it wouldn't work for some of the values.

Also a vsyscall doesn't help on non x86 anyways.

There's nothing x86 about aux entries.

Quoted 10 lines

As for efficiency: I thought about doing a batched interface where the user could pass in an array of values to fill in. But at least for the workloads I looked at the application usually calls sysconf() where the array size would be always 1. And that's the standard interface. This might be still worth doing, but I would like to see a concrete use case first.

Quoted 1 line

That could have other uses as well in the future.

Hmm for what?

*Iff* the concensus is that we are fine with a new page per task/thread then we could use it for all sorts of nifty things like the current CPU id for example.

Quoted 2 lines

Note we already have a fast mechanism to pass some thing to glibc in the aux vector.

So when you got so far in your reply why did you not delete your above (false) outrage about the vsyscall, which i never suggested and which you thus forced me to reply to?

Quoted 8 lines

Quoted 4 lines

That way it would much faster than your current code btw.

So unless there are some compelling arguments in favor of sys_sysconf() that i missed, i have to NAK this:

Well see above for lots of reasons you missed. They are understandable mistakes for someone who first looks at the problem though. I'll attempt to improve the documentation next time.

I don't think your condescending tone towards me is warranted or fair, i offered a fair technical criticism of your patch series. Why are you attacking me like this?

Thanks,

Ingo

Quoted 1 line

What glibc does (opening /proc/stat) is rather stupid, but i think your syscall

I don't think it has any other choice today. So if anything is "stupid" it is the kernel for not providing efficient interfaces for this.

Quoted 2 lines

Note that these are mostly constant or semi-constant values that are updated very rarely:

That's not true. Most of them are dynamic. Take a look at the patch. Also several of those have changed recently.

Quoted 2 lines

If glibc is stupid and reads /proc/stat to receive something it could cache or mmap() itself then hey, did you consider fixing glibc or creating a sane libc?

Caching doesn't help when you have a workload that exec()s a lot. Also some of these values can change at runtime.

Quoted 4 lines

If we *really* want to offer kernel help for these values even then your solution is still wrong: then the proper solution would be to define a standard *data* structure and map it as a vsyscall *data* page - essentially a kernel-guaranteed data mmap(), with no extra syscall needed!

That's quite complicted because several of those are dynamically computed based on other values. Sometimes they are also not tied to the mm_struct -- like the vsyscall is. For example some of the rlimits are per task, not VM. Basically your proposal doesn't interact well with clone().

Even if we ignored that semantic problem it would need another writable page per task because the values cannot be shared.

Also I never liked the idea of having more writable pages per task, It increases the memory footprint of a single process more. Given a 4K page is not a lot, but lots of 4K pages add up. Some workloads like to have lots of small processes and I think that's a valuable use case Linux should stay lean and efficient at.

[OK in theory one could do COW for the page and share it but that would get really complicated]

I also don't think it's THAT performance critical to justify the vsyscall. The simple syscall is already orders of magnitude faster than /proc, and seems to solve the performance problems we've seen completely.

It's also simple and straight forward and simple to userstand and maintain. I doubt any of that would apply to a vsyscall solution.

I don't think the additional effort for a vsyscall would be worth it at this point, unless there's some concrete example that would justify it. Even then it wouldn't work for some of the values.

Also a vsyscall doesn't help on non x86 anyways.

You could also add a vsyscall function and have that function pull the values from vvar data. It would be very fast (i.e. almost as fast as mmaping some data) and it would be more portable.

Quoted 3 lines

You could also add a vsyscall function and have that function pull the values from vvar data.  It would be very fast (i.e. almost as fast as mmaping some data) and it would be more portable.

vvar is a global. The variable data in sysconf is per thread (and not even per mm)

Even if you ignore the semantic problem of thread versus mm, you would need a new writable page per process and update it any time the rlimits and other information is updated.

On the other hand my solution is dead simple and already solves the problem.

-Andi

Quoted 2 lines

Auxv is simpler yet.

Yes simple and doesn't work.

kernel/sysconf.c new file +48 -0

+/*
+ * POSIX sysconf subset. Some programs need this in relatively fast paths
+ * and /proc is too slow for them.
+ *
+ * Note this is only a subset of the values supported by POSIX.
+ * We assume the C library handles the others.
+ */
+SYSCALL_DEFINE1(sysconf, int, name)
+{
+       switch (name) {
+       case _SC_ARG_MAX:
+               return rlimit_or(RLIMIT_STACK, ARG_MAX*ARG_MAX_FACTOR) /
+                               ARG_MAX_FACTOR;
+
+       case _SC_CHILD_MAX:
+               return rlimit_or(RLIMIT_NPROC, max_threads);
+
+       case _SC_CLK_TCK:
+               return HZ;
+
+       case _SC_SEM_NSEMS_MAX:
+               return current->nsproxy->ipc_ns->sem_ctls[1];
+
+       case _SC_OPEN_MAX:
+               return rlimit_or(RLIMIT_NOFILE, sysctl_nr_open);
+
+       case _SC_SIGQUEUE_MAX:
+               /* or fallback based on memory? */
+               return rlimit_or(RLIMIT_SIGPENDING, INT_MAX);
+
+       case _SC_UIO_MAXIOV:
+               return UIO_MAXIOV;
+
+       case _SC_PAGESIZE:
+               return PAGE_SIZE;
+
+       case _SC_SYMLOOP_MAX:
+               return SYMLOOP_MAX;
+
+       case _SC_PHYS_PAGES:
+               return totalram_pages;
+
+       case _SC_AVPHYS_PAGES:
+               return nr_free_pages();
+
+       case _SC_NPROCESSORS_CONF:
+               return num_possible_cpus();
+

...and libc will start making many such calls in a row in order to retrieve a dozen of such values.

It's rather inefficient to return just one word. Try to return more data per call.

Pass a pointer to the result struct and its length. Future-proof API, such as using generously wide data types, passing "version of struct" input parameter to facilitate incompatible future changes, etc.

fills in some sysconf values from the auxv and has infrastructure for that. And the perform conscious hacker can access the auxv directly if they want.

The syscall is only current if you call it frequently (i.e. every time), but there is tendency to go local (static memory) caching to avoid the syscall overhead. Something like _SC_NPROCESSORS_ONLN that is used to set up OpenMP/auto-parallel threading for major loops is likely to be cached.

But if you look at the gettimeofday/vdso implementation, the time_base to time conversion is semi-dynamic for the NTPd case and we handle that in VDSO. There is shared memory access between the VDSO and the kernel for this case.

Is problem with rlimit that it is process or session dependent? But is it really dynamic once a process starts?

It is not clear one size fits all here.

Quoted 2 lines

...and libc will start making many such calls in a row in order to retrieve a dozen of such values.

It doesn't because the user interface is sysconf(). So the user program just asks for it piece by piece.

Quoted 2 lines

It's rather inefficient to return just one word.
Try to return more data per call.

I considered that, but is there a concrete use case? I didn't want to code it up without concrete use case.

From: Andi Kleen <[email protected]>
Date: Fri, 13 May 2011 16:24:19 -0700

Quoted 10 lines

arch/sparc/include/asm/unistd.h +1 -0

index 9d897b6..0b0b362 100644
@@ -404,8 +404,9 @@
 #define __NR_open_by_handle_at	333
 #define __NR_clock_adjtime	334
 #define __NR_syncfs		335
+#define __NR_sysconf		336
 

arch/sparc/include/asm/unistd.h line 410

Quoted 3 lines

arch/sparc/include/asm/unistd.h +1 -1

-#define NR_syscalls		336
+#define NR_syscalls		337
 

x86, powerpc, and sparc entries will conflict with already assigned slots for sys_sendmmsg() in networking tree.

 arch/blackfin/include/asm/unistd.h     |    3 ++- arch/blackfin/mach-common/entry.S      |    1 +

Quoted 2 lines · Andi Kleen

 arch/blackfin/include/asm/unistd.h     |    3 ++- arch/blackfin/mach-common/entry.S      |    1 +

i wish more people would wire up new syscalls for arches ;)

i dont recall off the top of my head acking any other pending new syscall which would cause a conflict, so:

Acked-by Mike Frysinger <[email protected]>

i generally wait for syscalls to hit mainline before i wire them up myself

Quoted 10 lines · Mike Frysinger

Quoted 2 lines · Andi Kleen

 arch/blackfin/include/asm/unistd.h     |    3 ++- arch/blackfin/mach-common/entry.S      |    1 +

i wish more people would wire up new syscalls for arches ;)

i dont recall off the top of my head acking any other pending new syscall which would cause a conflict, so:

Acked-by Mike Frysinger <[email protected]>

Quoted 1 line

i generally wait for syscalls to hit mainline before i wire them up myself

i'm wiring up sendmmsg on Blackfin systems and will be pushing in a day or two, so this will need a little tweaking after that hits mainline

ok, i lied about that. i kept the patch local so that when your merge goes through, there wont be any conflicts. you are still planning on pushing this for 2.6.40-rc1 right ? if not, i'll just send out my sendmmsg wiring now.