Skip to contentSkip to conversation

AKT - Automatic Kernel Tunables

PATCH In review [email protected] 3 participants 10 messages Jan 30, 2007 – Feb 13, 2007

Subsystems (5)
Documentation VFS MM core Scheduler +1 more subsystems
lore
Share & tools
lore
#0 lore raw

cover AKT - Automatic Kernel Tunables

Re-sending the series of patches for the automatic kernel tunables feature:
have done some fixes after the remarks sent back by Andrew and Randy.

1) All the type independent macros have been removed, except for the automatic tuning routine: it manages pointers to the tunable and to the value to be checked against that tunable, so it should rmain type independent IMHO. Now, I only left the auto-tuning routines for types int and size_t since these are the types of the tunables the framework is applied to.
It will be easy to add the other types as needed in the future. This makes the code much lighter.

2) CONFIG_AKT has been moved from the FS menu to the "general setup" one.

+ all the other minor changes.

--- Reminder

This is a series of patches that introduces a feature that makes the kernel automatically change the tunables values as it sees resources running out.

The AKT framework is made of 2 parts:

1) Kernel part:
Interfaces are provided to the kernel subsystems, to (un)register the tunables that might be automatically tuned in the future.

Registering a tunable consists in the following steps:
- a structure is declared and filled by the kernel subsystem for the registered tunable
- that tunable structure is registered into sysfs

Registration should be done during the kernel subsystem initialization step.

Another interface is provided to the kernel subsystems, to activate the automatic tuning for a registered tunable. It can be called during resource allocation to tune up, and during resource freeing to tune down the registered tunable. The automatic tuning routine is called only if the tunable has been enabled to be automatically tuning in sysfs.

2) User part:

AKT uses sysfs to enable the tunables management from the user world (mainly making them automatic or manual).

akt uses sysfs in the following way:
- a tunables subsystem (tunables_subsys) is declared and registered during akt initialization.
- registering a tunable is equivalent to registering the corresponding kobject within that subsystem.
- each tunable kobject has 3 associated attributes, all with a RW mode (i.e. the show() and store() methods are provided for them):
. autotune: enables to (de)activate automatic tuning for the tunable . max: enables to set a new maximum value for the tunable . min: enables to set a new minimum value for the tunable

The only way to activate automatic tuning is from user side:
- the directory /sys/tunables is created during the init phase.
- each time a tunable is registered by a kernel subsystem, a directory is created for it under /sys/tunables.
- This directory contains 1 file for each tunable kobject attribute

These patches should be applied to 2.6.20-rc4, in the following order:

[PATCH 1/6]: tunables_registration.patch
[PATCH 2/6]: auto_tuning_activation.patch
[PATCH 3/6]: auto_tuning_kobjects.patch
[PATCH 4/6]: tunable_min_max_kobjects.patch
[PATCH 5/6]: per_namespace_tunables.patch
[PATCH 6/6]: auto_tune_applied.patch

--

#1 lore raw

PATCH 1/6 AKT - Tunable structure and registration routines

[PATCH 01/06]

Defines the auto_tune structure: this is the structure that contains the information needed by the adjustment routine for a given tunable. Also defines the registration routines.

The fork kernel component defines a tunable structure for the threads-max tunable and registers it.

---
 Documentation/00-INDEX      |    2 
 Documentation/auto_tune.txt |  333 ++++++++++++++++++++++++++++++++++++++++++++
 include/linux/akt.h         |  166 +++++++++++++++++++++
 include/linux/akt_ops.h     |  109 ++++++++++++++
 init/Kconfig                |    2 
 init/main.c                 |    2 
 kernel/Makefile             |    1 
 kernel/autotune/Kconfig     |   26 +++
 kernel/autotune/Makefile    |    7 
 kernel/autotune/akt.c       |  119 +++++++++++++++
 kernel/fork.c               |   18 ++
 11 files changed, 785 insertions(+)
Documentation/auto_tune.txt new file +333 -0
Index: linux-2.6.20-rc4/Documentation/auto_tune.txt
===================================================================
@@ -0,0 +1,333 @@
+			Automatic Kernel Tunables
+                        =========================
+
+		   Nadia Derbey ([email protected])
+
+
+
+This feature aims at making the kernel automatically change the tunables
+values as it sees resources running out.
+
+The AKT framework is made of 2 parts:
+
+1) Kernel part:
+Interfaces are provided to the kernel subsystems, to (un)register the
+tunables that might be automatically tuned in the future.
+
+Registering a tunable consists of the following steps:
+- a structure is declared and filled by the kernel subsystem for the
+registered tunable
+- that tunable structure is registered into sysfs
+
+Registration should be done during the kernel subsystem initialization step.
+
+Unregistering a tunable is the reverse operation. It should not be necessary
+for the kernel subsystems: it is only useful when unloading modules that would
+have registered a tunable during their loading step.
+
+The routines interfaces are the following:
+
+1.1) Declaring a tunable:
+
+A tunable structure should be declared and defined by the kernel subsystems as
+follows:
+
+DEFINE_TUNABLE(structure_name, threshold, min, max,
+		tunable_variable_ptr, checked_variable_ptr,
+		tunable_variable_type);
+
+Parameters:
+- structure_name: this is the name of the tunable structure
+
+- threshold: percentage to apply to the tunable value to detect if adjustment
+is needed
+
+- min: minimum value the tunable can ever reach (needed when adjusting down
+the tunable)
+
+- max: maximum value the tunable can ever reach (needed when adjusting up the
+tunable)
+
+- tunable_variable_ptr: address of the tunable that will be adjusted if
+needed.
+(ex: in kernel/fork.c it is max_threads's address)
+
+- checked_variable_ptr: address of the variable that is controlled by the
+tunable. This is the calling subsystem's object counter.
+(ex: in kernel/fork.c it is nr_threads's address: nr_threads should
+always remain < max_threads)
+
+- tunable_variable_type: this type is important since it helps choosing the
+appropriate automatic tuning routine.
+Presently, it can be one of int / size_t and this can easily be enhanced.
+
+The automatic tuning routine (i.e. the routine that should be called when
+automatic tuning is activated) is set to the default one:
+default_auto_tuning_<type>().
+<type> is chosen according to the tunable_variable_type parameters.
+All the previously listed parameters are useful to this routine.
+Refer to the description of the automatic adjustment routine to see how
+these parameters are actually used.
+
+Refer to "Updating the auto-tuning function pointer" to know how to set
+this routine to another one.
+
+
+1.2) Updating a tunable's characteristics
+
+1.2.1) Updating min / max values:
+
+Sometimes, when calling DEFINE_TUNABLE(), the min and max values are not
+exactly known, yet. In that case, the following routine should be called
+once these values are known:
+
+set_tunable_min_max(structure_name, new_min, new_max)
+
+Parameters:
+- structure_name: this is the name of the tunable structure
+
+- new_min: minimum value the tunable can ever reach
+
+- new_max: maximum value the tunable can ever reach
+
+1.2.2) Updating the auto-tuning function pointer:
+
+If the default auto-tuning routine doesn't fit your needs, you can define
+another one and associate it to the tunable using the following routine:
+
+set_autotuning_routine(structure_name, auto_tune)
+
+Parameters:
+- structure_name: this is the name of the tunable structure
+
+- auto_tune: routine that should be called when automatic tuning is activated.
+If this parameter is not NULL, it should be set to a function pointer defined
+by the kernel subsystem caller. See 1.5) for the routine prototype. See also
+maxfiles_auto_tuning() in fs/file_table.c for an example.
+
+
+1.3) Registering a tunable:
+
+Once declared and its min / max / auto_tuning routine updated, the tunable
+structure should be registered using the following routine:
+
+int register_tunable(struct auto_tune *tunable_addr);
+
+Parameters:
+- tunable_addr: address of the tunable structure previsouly declared.
+
+Return value:
+- 0 : successful
+- < 0 : failure
+
+
+Registering a tunable makes it potentially automatically adjustable:
+the tunable is viewed as a kobject with 3 attributes (i.e. 3 files at sysfs
+level):
+- autotune (rw): enables to (de)activate the auto tuning for that tunable
+- min (rw): enables to play with the min tunable value
+- max (rw): enables to play with the max tunable value
+
+The only way to make a registered tunable automatically adjustable is through
+sysfs (see the sysfs part for more details).
+
+
+
+1.4) Unregistering a tunable:
+
+int unregister_tunable(struct auto_tune *reg_tun_addr);
+
+Parameters:
+- reg_tun_addr: address of the tunable structure to unregister
+
+
+This routine is only useful for modules: when unloading, they should
+unregister any previously registered tunable.
+
+
+
+1.5) Automatic tuning routine:
+
+The 2nd main service provided by the kernel part is a function pointer
+(auto_tune_func): it points to the routine that actually automatically
+adjusts the tunable passed in as a parameter.
+
+This is accomplished by one of the following:
+- if an automatic tuning routine has been provided during the tunable
+declaration, that routine will actually be called.
+- if no automatic tuning routine has been provided, the default one is called.
+NOTE: it can process one of the following types, depending on the type used
+	when declaring the tunable (see DEFINE_TUNABLE above): int, size_t.
+
+
+If the automatic tuning routine is provided by the kernel subsystem caller,
+it should be declared as follows:
+
+int <routine_name>(int cmd, struct auto_tune *params);
+
+Parameters:
+- cmd: tuning direction
+	. AKT_UP: the tunable will be adjusted upwards (i.e. its value is
+		increased if needed)
+	. AKT_DOWN: the tunable is adjusted downwards (i.e. its value is
+		decreased if needed)
+- params: pointer to the previously registered tunable structure
+
+
+Any kernel subsystem that has registered a tunable should call
+auto_tune_func() as follows:
+
++-------------------------+--------------------------------------------+
+| Step                    | Routine to call                            |
++-------------------------+--------------------------------------------+
+| Declaration phase       | DEFINE_TUNABLE(name, values...);           |
++-------------------------+--------------------------------------------+
+| Initialization routine  | set_tunable_min_max(name, min, max);       |
+|                         | set_autotuning_routine(name, routine);     |
+|                         | register_tunable(&name);                   |
+| Note: the 1st 2 calls   |                                            |
+|       are optional      |                                            |
++-------------------------+--------------------------------------------+
+| Alloc                   | activate_auto_tuning(AKT_UP, &name);       |
++-------------------------+--------------------------------------------+
+| Free                    | activate_auto_tuning(AKT_DOWN, &name);     |
++-------------------------+--------------------------------------------+
+| module_exit() routine   | unregister_tunable(&name);                 |
++-------------------------+--------------------------------------------+
+
+activate_auto_tuning is a static inline defined in akt.h, that does the
+following:
+. if <tunable is registered> and <auto tuning is allowed for tunable>
+.   call the routine stored in tunable->auto_tune
+
+
+The effect of the default automatic tuning routine is the following:
+
+           +----------------------------------------------------------------+
+           |                 Tunable automatically adjustable               |
+           +---------------+------------------------------------------------+
+           |      NO       |                      YES                       |
++----------+---------------+------------------------------------------------+
+| AKT_UP   | No effect     | If the tunable value exceeds the specified     |
+|          |               | threshold, that value is increased up to a     |
+|          |               | maximum value.                                 |
+|          |               | The maximum value is specified during the      |
+|          |               | tunable declaration and can be changed at any  |
+|          |               | time through sysfs                             |
++----------+---------------+------------------------------------------------+
+| AKT_DOWN | No effect     | If the tunable value falls under the specified |
+|          |               | threshold, that value is decreased down to a   |
+|          |               | minimum value.                                 |
+|          |               | The minimum value is specified during the      |
+|          |               | tunable declaration and can be changed at any  |
+|          |               | time through sysfs                             |
++----------+---------------+------------------------------------------------+
+
+
+1.6. Default automatic adjustment routine
+
+The last service provided by AKT at the kernel level is the default automatic
+adjustment routine. As seen, above, this routine supports various tunables
+types. It works as follows (only the AKT_UP direction is described here -
+AKT_DOWN does the reverse operation):
+
+The 2nd parameter passed in to this routine is a pointer to a previously
+registered tunable structure. That structure contains the following fields
+(see 1.1 for the detailed description):
+- threshold
+- key
+- min
+- max
+- tunable
+- checked
+
+When this routine is entered, it does the following:
+1. <*checked> is compared to <*tunable> * threshold
+2. if <*checked> is greater, <*tunable> is set to:
+	<*tunable> + (<*tunable> * (100 - threshold) / 100)
+
+
+
+1.6) akt and sysfs:
+
+AKT uses sysfs to enable the tunables management from the user world (mainly
+making them automatic or manual).
+
+akt uses sysfs in the following way:
+- a tunables subsystem (tunables_subsys) is declared and registered during akt
+initialization.
+- registering a tunable is equivalent to registering the corresponding kobject
+within that subsystem.
+- each tunable kobject has 3 associated attributes, all with a RW mode (i.e.
+the show() and store() methods are provided for them):
+	. autotune: enables to (de)activate automatic tuning for the tunable
+	. max: enables to set a new maximum value for the tunable
+	. min: enables to set a new minimum value for the tunable
+
+
+1.7) tunables that are namespace dependent
+
+In this paragraph, the particular case of tunables that are namespace
+dependent is presented.
+
+1.7.1) Declaring a tunable:
+
+The tunable structure for such tunables should be declared in the namespace
+structure that contains the associated tunable (ex: the tunable structure for
+msg_ctlmni should be declared in the ipc_namespace structure).
+
+The tunable structure should be declared as follows:
+
+DECLARE_TUNABLE(structure_name);
+
+Parameters:
+- structure_name: this is the name of the tunable structure
+
+1.7.2) Initializing the tunable structure
+
+Then the tunable structure should be initialized by calling the following
+routine:
+
+init_tunable_ipcns(namespace_ptr, structure_name, threshold, min, max,
+		tunable_variable_ptr, checked_variable_ptr,
+		tunable_variable_type);
+
+Parameters:
+- namespace_ptr: pointer to the namespace the tunable belongs to.
+
+See DEFINE_TUNABLE for the other parameters.
+
+1.7.3) Registering the tunable structure
+
+register_tunable should be called, giving it the tunable structure address
+that belongs to the init namespace.
+
+This applies to activate_auto_tuning too.
+
+All the routines that show/store attributes or that do the auto tuning are
+namespace dependent.
+
+
+2) User part:
+
+As seen above, the only way to activate automatic tuning is from user side:
+- the directory /sys/tunables is created during the init phase.
+- each time a tunable is registered by a kernel subsystem, a directory is
+created for it under /sys/tunables.
+- This directory contains 1 file for each tunable kobject attribute:
++-----------+---------------+-------------------+--------------------------+
+| attribute | default value | how to set it     | effect                   |
++-----------+---------------+-------------------+--------------------------+
+| autotune  | 0             | echo 1 > autotune | makes the tunable        |
+|           |               |                   | automatic                |
+|           |               | echo 0 > autotune | makes the tunable manual |
++-----------+---------------+-------------------+--------------------------+
+| max       | max value set | echo <M> > max    | sets the tunable max     |
+|           | during tunable|                   | value to <M>             |
+|           | definition    |                   |                          |
++-----------+---------------+-------------------+--------------------------+
+| min       | min value set | echo <m> > min    | sets the tunable min     |
+|           | during tunable|                   | value to <m>             |
+|           | definition    |                   |                          |
++-----------+---------------+-------------------+--------------------------+
+
Documentation/00-INDEX +2 -0
Index: linux-2.6.20-rc4/Documentation/00-INDEX
===================================================================
@@ -52,6 +52,8 @@ applying-patches.txt
 	- description of various trees and how to apply their patches.
 arm/
 	- directory with info about Linux on the ARM architecture.
+auto_tune.txt
+	- info on the Automatic Kernel Tunables (AKT) feature.
 basic_profiling.txt
 	- basic instructions for those who wants to profile Linux kernel.
 binfmt_misc.txt
include/linux/akt.h new file +166 -0

166 lines not on this page · raw message

include/linux/akt_ops.h new file +109 -0

109 lines not on this page · raw message

init/Kconfig +2 -0
Index: linux-2.6.20-rc4/init/Kconfig
===================================================================
@@ -466,6 +466,8 @@ config VM_EVENT_COUNTERS
 	  on EMBEDDED systems.  /proc/vmstat will only show page counts
 	  if VM event counters are disabled.
 
+source "kernel/autotune/Kconfig"
+
 endmenu		# General setup
 
 config RT_MUTEXES
init/main.c +2 -0
Index: linux-2.6.20-rc4/init/main.c
===================================================================
@@ -54,6 +54,7 @@
 #include <linux/pid_namespace.h>
 #include <linux/compile.h>
 #include <linux/device.h>
+#include <linux/akt.h>
 
 #include <asm/io.h>
 #include <asm/bugs.h>
@@ -613,6 +614,7 @@ asmlinkage void __init start_kernel(void
 	signals_init();
 	/* rootfs populating might need page-writeback */
 	page_writeback_init();
+	fork_late_init();
 #ifdef CONFIG_PROC_FS
 	proc_root_init();
 #endif
kernel/Makefile +1 -0
Index: linux-2.6.20-rc4/kernel/Makefile
===================================================================
@@ -50,6 +50,7 @@ obj-$(CONFIG_RELAY) += relay.o
 obj-$(CONFIG_UTS_NS) += utsname.o
 obj-$(CONFIG_TASK_DELAY_ACCT) += delayacct.o
 obj-$(CONFIG_TASKSTATS) += taskstats.o tsacct.o
+obj-$(CONFIG_AKT) += autotune/
 
 ifneq ($(CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER),y)
 # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is
kernel/autotune/Kconfig new file +26 -0
Index: linux-2.6.20-rc4/kernel/autotune/Kconfig
===================================================================
@@ -0,0 +1,26 @@
+#
+# Automatic Kernel Tunables
+#
+
+config AKT
+	bool "Automatic kernel tunables support (AKT)"
+	depends on PROC_FS && SYSFS
+	help
+	  This is a functionality that enables automatic adjustment of kernel
+	  tunables: when this feature is enabled the kernel can automatically
+	  change the tunables values as it sees resources running out.
+
+	  The list of kernel tunables that can potentially be automatically
+	  adjusted can found under /sys/tunables.
+
+	  In order to make a tunable actually automatic, issue the following
+	  command:
+	  echo 1 > /sys/tunables/<tunable_name>/autotune
+
+	  In order to make it manual, issue the following command:
+	  echo 0 > /sys/tunables/<tunable_name>/autotune
+
+	  See Documentation/auto_tune.txt for more details.
+
+	  If unsure, say N.
+
kernel/autotune/akt.c new file +119 -0

119 lines not on this page · raw message

kernel/autotune/Makefile new file +7 -0
Index: linux-2.6.20-rc4/kernel/autotune/Makefile
===================================================================
@@ -0,0 +1,7 @@
+#
+# Makefile for akt
+#
+
+obj-y := akt.o
+
+
kernel/fork.c +18 -0
Index: linux-2.6.20-rc4/kernel/fork.c
===================================================================
@@ -49,6 +49,8 @@
 #include <linux/delayacct.h>
 #include <linux/taskstats_kern.h>
 #include <linux/random.h>
+#include <linux/akt.h>
+#include <linux/akt_ops.h>
 
 #include <asm/pgtable.h>
 #include <asm/pgalloc.h>
@@ -65,6 +67,13 @@ int nr_threads; 		/* The idle threads do
 
 int max_threads;		/* tunable limit on nr_threads */
 
+#define THREADTHRESH 80
+/*
+ * The actual values for min and max will be known during fork_init
+ */
+DEFINE_TUNABLE(max_threads_akt, THREADTHRESH, 0, 0, &max_threads,
+		&nr_threads, int);
+
 DEFINE_PER_CPU(unsigned long, process_counts) = 0;
 
 __cacheline_aligned DEFINE_RWLOCK(tasklist_lock);  /* outer */
@@ -152,12 +161,21 @@ void __init fork_init(unsigned long memp
 	if(max_threads < 20)
 		max_threads = 20;
 
+	set_tunable_min_max(max_threads_akt, max_threads, mempages / 2);
+
 	init_task.signal->rlim[RLIMIT_NPROC].rlim_cur = max_threads/2;
 	init_task.signal->rlim[RLIMIT_NPROC].rlim_max = max_threads/2;
 	init_task.signal->rlim[RLIMIT_SIGPENDING] =
 		init_task.signal->rlim[RLIMIT_NPROC];
 }
 
+void __init fork_late_init(void)
+{
+	if (register_tunable(&max_threads_akt))
+		printk(KERN_WARNING
+			"AKT: Failed registering tunable max_threads\n");
+}
+
 static struct task_struct *dup_task_struct(struct task_struct *orig)
 {
 	struct task_struct *tsk;

--

#2 lore raw

Re: [PATCH 1/6] AKT - Tunable structure and registration routines

Documentation/auto_tune.txt line 9

Quoted 3 lines · [email protected] writes:
Documentation/auto_tune.txt new file +3 -0
+
+This feature aims at making the kernel automatically change the tunables
+values as it sees resources running out.

The only reason we have resource limit is to avoid DOS when one resource consumes too much memory. When there is no such danger then there isn't any reason to have a limit at all and it could be just eliminated (or set to unlimited by default)

Your feature doesn't address the DOS and without that there isn't any reason to have limits at all. So what's the point?

I agree that some of the default limits we have are not very useful on modern machines. I guess you're trying to address that.

I would suggest the following strategy:

- Review any limits we have and make sure they make sense.

- Figure out if they actually serve a useful purpose
e.g. what happens when they are exceeded, is there a DOS?.
If yes can the DOS be addressed in a better way (e.g. by allowing to shrink the resource by a shrinker callback).

Some of the existing limits are clearly bogus, e.g. the limit on shared memory.

For others i don't see a good alternative. e.g. if you don't limit the number of files allocated the only alternative would be to kill processes when they allocate too many files. Is that really preferable to a errno?

- If they serve a useful purpose then check if the default is useful on a modern machine. Or make them scale with the amount of memory like many limits already do.

-Andi

#3 lore raw

Re: [PATCH 1/6] AKT - Tunable structure and registration routines

Quoted 11 lines · Andi Kleen

Documentation/auto_tune.txt line 9

Quoted 3 lines · [email protected] writes:
Documentation/auto_tune.txt new file +3 -0
+
+This feature aims at making the kernel automatically change the tunables
+values as it sees resources running out.

The only reason we have resource limit is to avoid DOS when one resource consumes too much memory. When there is no such danger then there isn't any reason to have a limit at all and it could be just eliminated (or set to unlimited by default)

Automatic tuning is a way to set the limit to unlimited, in a sense, doesn't it? With this feature, we can leave the default limits as they are for an "every-day" usage, and when a particular application runs on the machine, authorize the limit to grow up as needed.

Quoted 3 lines

Your feature doesn't address the DOS and without that there isn't any reason to have limits at all. So what's the point?

As I told Eric Biederman in another mail, DoS in ensured in AKT by exporting the min and max values for each tunable to sysfs (actually Eric complained about these min and max :-( ). These are RW atrributes that make it possible for a sysadmin to set the max value a tunable can ever reach, instead of letting it grow up to huge values.

Quoted 3 lines

I agree that some of the default limits we have are not very useful on modern machines. I guess you're trying to address that.

Yep

Quoted 9 lines

I would suggest the following strategy:

- Review any limits we have and make sure they make sense.

- Figure out if they actually serve a useful purpose
e.g. what happens when they are exceeded, is there a DOS?. If yes can the DOS be addressed in a better way (e.g. by allowing to shrink the resource by a shrinker callback).

Quoted 7 lines

Some of the existing limits are clearly bogus, e.g. the limit on shared memory.

For others i don't see a good alternative. e.g. if you don't limit the number of files allocated the only alternative would be to kill processes when they allocate too many files. Is that really preferable to a errno?

Agree with you, BUT between the default max_files and the "too many files" situation, there is a gap that can be crossed by automatically tuning max_files, isn't there? e.g. max_file default value is NR_FILE (0x2000), while Oracle expects to have it set to 0x10000.

Quoted 7 lines

- If they serve a useful purpose then check if the default is useful on a modern machine. Or make them scale with the amount of memory like many limits already do.

-Andi

Regards,
Nadia

#4 lore raw

Re: [PATCH 1/6] AKT - Tunable structure and registration routines

Quoted 13 lines · Nadia Derbey
Quoted 11 lines · Andi Kleen

Documentation/auto_tune.txt line 9

Quoted 3 lines · [email protected] writes:
Documentation/auto_tune.txt new file +3 -0
+
+This feature aims at making the kernel automatically change the tunables
+values as it sees resources running out.

The only reason we have resource limit is to avoid DOS when one resource consumes too much memory. When there is no such danger then there isn't any reason to have a limit at all and it could be just eliminated (or set to unlimited by default)

Quoted 4 lines

Automatic tuning is a way to set the limit to unlimited, in a sense, doesn't it? With this feature, we can leave the default limits as they are for an "every-day" usage, and when a particular application runs on the machine, authorize the limit to grow up as needed.

That would be effectively no limit, so why not just do away with them completely? You have to solve the DOS issues first of course, either way.

Quoted 8 lines
Quoted 2 lines

Your feature doesn't address the DOS and without that there isn't any reason to have limits at all. So what's the point?

As I told Eric Biederman in another mail, DoS in ensured in AKT by exporting the min and max values for each tunable to sysfs (actually Eric complained about these min and max :-( ). These are RW atrributes that make it possible for a sysadmin to set the max value a tunable can ever reach, instead of letting it grow up to huge values.

Then you could just set the limit always to that max value and would get the same effect.

Min limit doesn't make much sense to me because near all Linux data structures are on demand only anyways (excluding mempools and a few special cases) and min is defined as the current number of used resources.

Quoted 4 lines

Agree with you, BUT between the default max_files and the "too many files" situation, there is a gap that can be crossed by automatically tuning max_files, isn't there? e.g. max_file default value is NR_FILE (0x2000), while Oracle expects to have it set to 0x10000.

The reason NR_FILE is by default relatively low is mostly because the existing ulimits suck :-) .i.e. you want to limit the memory consumption
of a user you limit the number of the user's processes and the number of files per process -- then the max memory they can allocate in files is process ulimit*nr_files [in theory in practice there are holes like in flight fds in unix socket fd passing]

But of course that ends up with either NR_FILE being far too low or process far too low or too much memory anyways. In practice it doesn't work particularly well.

The real solution for that is "beancounter" which has been posted in several forms recently (from OpenVZ and from Google). Basically it adds real limits (for much more than just file descriptors) per uid or container.

With such infrastructure in place NR_FILES could be set to unlimited, as long as you have a reasonable (large) default per uid.

Similar reasoning applies to other resources which are currently limited.

I think you're just attacking the symptoms here instead of the basic issue.

-Andi

#5 lore raw

PATCH 2/6 AKT - auto_tuning activation

In reply to [email protected]

[PATCH 02/06]

Introduces the auto-tuning activation routine

The auto-tuning routine is called by the fork kernel component

---
 include/linux/akt.h |   51 +++++++++++++++++++++++++++++++++++++++++++++++++++
 kernel/exit.c       |   11 +++++++++++
 kernel/fork.c       |    2 ++
 3 files changed, 64 insertions(+)
include/linux/akt.h +51 -0
Index: linux-2.6.20-rc4/include/linux/akt.h
===================================================================
@@ -102,12 +102,22 @@ struct auto_tune {
 /*
  * Flags for a registered tunable
  */
+#define AUTO_TUNE_ENABLE  0x01
 #define TUNABLE_REGISTERED  0x02
 
 
 /*
  * When calling this routine the tunable lock should be held
  */
+static inline int is_auto_tune_enabled(struct auto_tune *tunable)
+{
+	return (tunable->flags & AUTO_TUNE_ENABLE) == AUTO_TUNE_ENABLE;
+}
+
+
+/*
+ * When calling this routine the tunable lock should be held
+ */
 static inline int is_tunable_registered(struct auto_tune *tunable)
 {
 	return (tunable->flags & TUNABLE_REGISTERED) == TUNABLE_REGISTERED;
@@ -146,6 +156,44 @@ static inline int is_tunable_registered(
 	} while (0)
 
 
+static inline void set_autotuning_routine(struct auto_tune *tunable,
+					auto_tune_fn fn)
+{
+	if (fn != NULL)
+		tunable->auto_tune = fn;
+}
+
+
+/*
+ * direction may be one of:
+ *    AKT_UP: adjust up (i.e. increase tunable value when needed)
+ *    AKT_DOWN: adjust down (i.e. decrease tunable value when needed)
+ */
+static inline int activate_auto_tuning(int direction,
+					struct auto_tune *tunable)
+{
+	int ret = 0;
+
+	BUG_ON(direction != AKT_UP && direction != AKT_DOWN);
+
+	if (tunable == NULL)
+		return 0;
+
+	spin_lock(&tunable->tunable_lck);
+
+	if (!is_auto_tune_enabled(tunable) ||
+					!is_tunable_registered(tunable)) {
+		spin_unlock(&tunable->tunable_lck);
+		return 0;
+	}
+
+	ret = tunable->auto_tune(direction, tunable);
+
+	spin_unlock(&tunable->tunable_lck);
+	return ret;
+}
+
+
 extern int register_tunable(struct auto_tune *);
 extern int unregister_tunable(struct auto_tune *);
 
@@ -155,6 +203,9 @@ extern int unregister_tunable(struct aut
 
 #define DEFINE_TUNABLE(s, thresh, min, max, tun, chk, type)
 #define set_tunable_min_max(s, min, max)         do { } while (0)
+#define set_autotuning_routine(s, fn)            do { } while (0)
+
+#define activate_auto_tuning(direction, tunable) ( { 0; } )
 
 #define register_tunable(a)                 0
 #define unregister_tunable(a)               0
kernel/exit.c +11 -0
Index: linux-2.6.20-rc4/kernel/exit.c
===================================================================
@@ -42,12 +42,15 @@
 #include <linux/audit.h> /* for audit_free() */
 #include <linux/resource.h>
 #include <linux/blkdev.h>
+#include <linux/akt.h>
 
 #include <asm/uaccess.h>
 #include <asm/unistd.h>
 #include <asm/pgtable.h>
 #include <asm/mmu_context.h>
 
+extern struct auto_tune max_threads_akt;
+
 extern void sem_exit (void);
 
 static void exit_mm(struct task_struct * tsk);
@@ -172,6 +175,14 @@ repeat:
 
 	sched_exit(p);
 	write_unlock_irq(&tasklist_lock);
+
+	/*
+	 * nr_threads has been decremented in __unhash_process: adjust
+	 * max_threads down if needed
+	 * We do it here to avoid calling activate_auto_tuning under lock
+	 */
+	activate_auto_tuning(AKT_DOWN, &max_threads_akt);
+
 	proc_flush_task(p);
 	release_thread(p);
 	call_rcu(&p->rcu, delayed_put_task_struct);
kernel/fork.c +2 -0
Index: linux-2.6.20-rc4/kernel/fork.c
===================================================================
@@ -995,6 +995,8 @@ static struct task_struct *copy_process(
 	if ((clone_flags & CLONE_SIGHAND) && !(clone_flags & CLONE_VM))
 		return ERR_PTR(-EINVAL);
 
+	activate_auto_tuning(AKT_UP, &max_threads_akt);
+
 	retval = security_task_create(clone_flags);
 	if (retval)
 		goto fork_out;

--

#6 lore raw

PATCH 3/6 AKT - tunables associated kobjects

In reply to [email protected]

[PATCH 03/06]

Introduces the kobjects associated to each tunable and the sysfs registration

---
 include/linux/akt.h         |   25 ++++-
 init/main.c                 |    1 
 kernel/autotune/Makefile    |    2 
 kernel/autotune/akt.c       |   91 ++++++++++++++++++
 kernel/autotune/akt_sysfs.c |  214 ++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 330 insertions(+), 3 deletions(-)
include/linux/akt.h +24 -1
Index: linux-2.6.20-rc4/include/linux/akt.h
===================================================================
@@ -48,6 +48,16 @@ typedef int (*auto_tune_fn)(int, struct 
 
 
 /*
+ * for sysfs support
+ */
+struct tunable_kobject {
+	struct kobject kobj;
+	struct auto_tune *tun;
+};
+
+
+
+/*
  * Structure used to describe the min / max values for a tunable inside the
  * auto_tune structure.
  */
@@ -62,7 +72,12 @@ struct tunable_limit {
  * allocated for each registered tunable, and the associated kobject exported
  * via sysfs.
  *
- * The structure lock (tunable_lck) protects
+ * This structure may be accessed in 2 ways:
+ *   . directly from inside the kernel susbsystem that uses it (during tunable
+ *     automatic adjustment)
+ *   . from sysfs, while updating the kobject attributes
+ *
+ * In both cases, the structure lock (tunable_lck) is taken: it protects
  * against concurrent accesses to tunable and checked pointers
  *
  * A pointer to this structure is passed in to  the automatic adjustment
@@ -92,6 +107,7 @@ struct auto_tune {
 					/* reach */
 	struct tunable_limit max;	/* max value the tunable can ever */
 					/* reach */
+	struct tunable_kobject    tun_kobj;	/* used for sysfs support */
 	void *tunable;	/* address of the tunable to adjust */
 	void *checked;	/* address of the variable that is controlled by */
 			/* the tunable. This is the calling subsystem's */
@@ -141,6 +157,7 @@ static inline int is_tunable_registered(
 		.max	= {						\
 			.value		= (_max),			\
 		},							\
+		.tun_kobj	= { .tun = NULL, },			\
 		.tunable	= (_tun),				\
 		.checked	= (_chk),				\
 	}
@@ -194,8 +211,12 @@ static inline int activate_auto_tuning(i
 }
 
 
+extern void init_auto_tuning(void);
 extern int register_tunable(struct auto_tune *);
 extern int unregister_tunable(struct auto_tune *);
+extern int tunable_sysfs_setup(struct auto_tune *);
+extern ssize_t show_tuning_mode(struct auto_tune *, char *);
+extern ssize_t store_tuning_mode(struct auto_tune *, const char *, size_t);
 
 
 #else	/* CONFIG_AKT */
@@ -210,6 +231,8 @@ extern int unregister_tunable(struct aut
 #define register_tunable(a)                 0
 #define unregister_tunable(a)               0
 
+static inline void init_auto_tuning(void)   { }
+
 #endif	/* CONFIG_AKT */
 
 extern void fork_late_init(void);
init/main.c +1 -0
Index: linux-2.6.20-rc4/init/main.c
===================================================================
@@ -614,6 +614,7 @@ asmlinkage void __init start_kernel(void
 	signals_init();
 	/* rootfs populating might need page-writeback */
 	page_writeback_init();
+	init_auto_tuning();
 	fork_late_init();
 #ifdef CONFIG_PROC_FS
 	proc_root_init();
kernel/autotune/Makefile +1 -1
Index: linux-2.6.20-rc4/kernel/autotune/Makefile
===================================================================
@@ -2,6 +2,6 @@
 # Makefile for akt
 #
 
-obj-y := akt.o
+obj-y := akt.o akt_sysfs.o
 
 
kernel/autotune/akt.c +90 -1

118 lines not on this page · raw message

kernel/autotune/akt_sysfs.c new file +214 -0

214 lines not on this page · raw message

--

#7 lore raw

PATCH 4/6 AKT - min and max kobjects

In reply to [email protected]

[PATCH 04/06]

Introduces the kobjects associated to each tunable min and max value

---
 include/linux/akt.h         |   14 ++++
 kernel/autotune/akt.c       |  148 ++++++++++++++++++++++++++++++++++++++++++++
 kernel/autotune/akt_sysfs.c |   16 ++++
 3 files changed, 177 insertions(+), 1 deletion(-)
include/linux/akt.h +14 -0
Index: linux-2.6.20-rc4/include/linux/akt.h
===================================================================
@@ -60,9 +60,15 @@ struct tunable_kobject {
 /*
  * Structure used to describe the min / max values for a tunable inside the
  * auto_tune structure.
+ * The abs_value field is used to check that we are not:
+ *   . falling under the very 1st min value when updating the min value
+ *     through sysfs
+ *   . going over the very 1st max value when updating the max value
+ *     through sysfs
  */
 struct tunable_limit {
 	ulong value;
+	ulong abs_value;
 };
 
 
@@ -153,9 +159,11 @@ static inline int is_tunable_registered(
 		.threshold	= (_thresh),				\
 		.min	= {						\
 			.value		= (_min),			\
+			.abs_value	= (_min),			\
 		},							\
 		.max	= {						\
 			.value		= (_max),			\
+			.abs_value	= (_max),			\
 		},							\
 		.tun_kobj	= { .tun = NULL, },			\
 		.tunable	= (_tun),				\
@@ -169,7 +177,9 @@ static inline int is_tunable_registered(
 #define set_tunable_min_max(s, _min, _max)	\
 	do {					\
 		(s).min.value = _min;		\
+		(s).min.abs_value = _min;	\
 		(s).max.value = _max;		\
+		(s).max.abs_value = _max;	\
 	} while (0)
 
 
@@ -217,6 +227,10 @@ extern int unregister_tunable(struct aut
 extern int tunable_sysfs_setup(struct auto_tune *);
 extern ssize_t show_tuning_mode(struct auto_tune *, char *);
 extern ssize_t store_tuning_mode(struct auto_tune *, const char *, size_t);
+extern ssize_t show_tunable_min(struct auto_tune *, char *);
+extern ssize_t store_tunable_min(struct auto_tune *, const char *, size_t);
+extern ssize_t show_tunable_max(struct auto_tune *, char *);
+extern ssize_t store_tunable_max(struct auto_tune *, const char *, size_t);
 
 
 #else	/* CONFIG_AKT */
kernel/autotune/akt.c +148 -0

157 lines not on this page · raw message

kernel/autotune/akt_sysfs.c +15 -1
Index: linux-2.6.20-rc4/kernel/autotune/akt_sysfs.c
===================================================================
@@ -54,8 +54,16 @@ struct tunable_attribute tun_attr_##_nam
 static TUNABLE_ATTR(autotune, S_IWUSR | S_IRUGO, show_tuning_mode,
 		store_tuning_mode);
 
+static TUNABLE_ATTR(min, S_IWUSR | S_IRUGO, show_tunable_min,
+		store_tunable_min);
+
+static TUNABLE_ATTR(max, S_IWUSR | S_IRUGO, show_tunable_max,
+		store_tunable_max);
+
 static struct tunable_attribute *tunable_sysfs_attrs[] = {
 	&tun_attr_autotune,	/* to (de)activate auto tuning */
+	&tun_attr_min,		/* to play with the tunable min value */
+	&tun_attr_max,		/* to play with the tunable max value */
 	NULL,
 };
 
@@ -75,6 +83,8 @@ static int add_tunable_attrs(struct auto
  * @kobj:	tunable associated kobject
  * @attr:	tunable attribute to read. Can be one of:
  *			tun_attr_autotune
+ *			tun_attr_min
+ *			tun_attr_max
  * @buf:	output buffer
  *
  * Forwards any read call to the show method of the owning attribute
@@ -102,6 +112,8 @@ static ssize_t tunable_attr_show(struct 
  * @kobj:	tunable associated kobject
  * @attr:	tunable attribute to update. Can be one of:
  *			tun_attr_autotune
+ *			tun_attr_min
+ *			tun_attr_max
  * @buf:	input buffer
  * @count:	input buffer length (including the '\0')
  *
@@ -145,8 +157,10 @@ decl_subsys(tunables, &tunables_ktype, N
  * @tunable:	tunable structure to be registered
  *
  * Called by register_tunable()
- * The tunable is a kobject with 1 attributes:
+ * The tunable is a kobject with 3 attributes:
  *	autotune (rw): enables to (de)activate the auto tuning for the tunable
+ *	min (rw): enables to play with the min tunable value
+ *	max (rw): enables to play with the max tunable value
  *
  * Returns:	0 - successful
  *		<0 - failure

--

#8 lore raw

PATCH 5/6 AKT - per namespace tunables

In reply to [email protected]

[PATCH 05/06]

This patch introduces all that is needed to process per namespace tunables.

---
 include/linux/akt.h   |   12 ++++++
 kernel/autotune/akt.c |   94 +++++++++++++++++++++++++++++++++++++-------------
 2 files changed, 83 insertions(+), 23 deletions(-)
include/linux/akt.h +12 -0
Index: linux-2.6.20-rc4/include/linux/akt.h
===================================================================
@@ -126,6 +126,7 @@ struct auto_tune {
  */
 #define AUTO_TUNE_ENABLE  0x01
 #define TUNABLE_REGISTERED  0x02
+#define TUNABLE_IPC_NS      0x04
 
 
 /*
@@ -171,6 +172,8 @@ static inline int is_tunable_registered(
 	}
 
 
+#define DECLARE_TUNABLE(s)	struct auto_tune s;
+
 #define DEFINE_TUNABLE(s, thr, min, max, tun, chk, type)		\
 	struct auto_tune s = TUNABLE_INIT(#s, thr, min, max, tun, chk, type)
 
@@ -182,6 +185,13 @@ static inline int is_tunable_registered(
 		(s).max.abs_value = _max;	\
 	} while (0)
 
+#define init_tunable_ipcns(ns, s, thr, min, max, tun, chk, type)	\
+	do {								\
+		DEFINE_TUNABLE(s, thr, min, max, tun, chk, type);	\
+		s.flags |= TUNABLE_IPC_NS;				\
+		ns->s = s;						\
+	} while (0)
+
 
 static inline void set_autotuning_routine(struct auto_tune *tunable,
 					auto_tune_fn fn)
@@ -236,7 +246,9 @@ extern ssize_t store_tunable_max(struct 
 #else	/* CONFIG_AKT */
 
 
+#define DECLARE_TUNABLE(s)
 #define DEFINE_TUNABLE(s, thresh, min, max, tun, chk, type)
+#define init_tunable_ipcns(ns, s, th, m, M, tun, chk, type)  do { } while (0)
 #define set_tunable_min_max(s, min, max)         do { } while (0)
 #define set_autotuning_routine(s, fn)            do { } while (0)
 
kernel/autotune/akt.c +71 -23

202 lines not on this page · raw message

--

#9 lore raw

PATCH 6/6 AKT - automatic tuning applied to some kernel components

In reply to [email protected]

[PATCH 06/06]

The following kernel components register a tunable structure and call the auto-tuning routine:
. file system
. shared memory (per namespace)
. semaphore (per namespace)
. message queues (per namespace)

---
 fs/file_table.c     |   82 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/akt.h |    1 
 include/linux/ipc.h |    6 +++
 init/main.c         |    1 
 ipc/msg.c           |   20 ++++++++++++
 ipc/sem.c           |   43 +++++++++++++++++++++++++++
 ipc/shm.c           |   76 +++++++++++++++++++++++++++++++++++++++++++++---
 7 files changed, 224 insertions(+), 5 deletions(-)
include/linux/akt.h +1 -0
Index: linux-2.6.20-rc4/include/linux/akt.h
===================================================================
@@ -262,5 +262,6 @@ static inline void init_auto_tuning(void
 #endif	/* CONFIG_AKT */
 
 extern void fork_late_init(void);
+extern void files_late_init(void);
 
 #endif /* AKT_H */
include/linux/ipc.h +6 -0
Index: linux-2.6.20-rc4/include/linux/ipc.h
===================================================================
@@ -52,6 +52,7 @@ struct ipc_perm
 #ifdef __KERNEL__
 
 #include <linux/kref.h>
+#include <linux/akt.h>
 
 #define IPCMNI 32768  /* <= MAX_INT limit for ipc arrays (including sysctl changes) */
 
@@ -77,15 +78,20 @@ struct ipc_namespace {
 
 	int		sem_ctls[4];
 	int		used_sems;
+	DECLARE_TUNABLE(semmni_akt);
+	DECLARE_TUNABLE(semmns_akt);
 
 	int		msg_ctlmax;
 	int		msg_ctlmnb;
 	int		msg_ctlmni;
+	DECLARE_TUNABLE(msgmni_akt);
 
 	size_t		shm_ctlmax;
 	size_t		shm_ctlall;
 	int		shm_ctlmni;
 	int		shm_tot;
+	DECLARE_TUNABLE(shmmni_akt);
+	DECLARE_TUNABLE(shmall_akt);
 };
 
 extern struct ipc_namespace init_ipc_ns;
init/main.c +1 -0
Index: linux-2.6.20-rc4/init/main.c
===================================================================
@@ -616,6 +616,7 @@ asmlinkage void __init start_kernel(void
 	page_writeback_init();
 	init_auto_tuning();
 	fork_late_init();
+	files_late_init();
 #ifdef CONFIG_PROC_FS
 	proc_root_init();
 #endif
fs/file_table.c +82 -0

112 lines not on this page · raw message

ipc/msg.c +20 -0
Index: linux-2.6.20-rc4/ipc/msg.c
===================================================================
@@ -36,6 +36,8 @@
 #include <linux/seq_file.h>
 #include <linux/mutex.h>
 #include <linux/nsproxy.h>
+#include <linux/akt.h>
+#include <linux/akt_ops.h>
 
 #include <asm/current.h>
 #include <asm/uaccess.h>
@@ -94,6 +96,11 @@ static void __ipc_init __msg_init_ns(str
 	ns->msg_ctlmnb = MSGMNB;
 	ns->msg_ctlmni = MSGMNI;
 	ipc_init_ids(ids, ns->msg_ctlmni);
+
+#define MSGTHRESH 80
+
+	init_tunable_ipcns(ns, msgmni_akt, MSGTHRESH, MSGMNI, IPCMNI,
+		&ns->msg_ctlmni, &ids->in_use, int);
 }
 
 #ifdef CONFIG_IPC_NS
@@ -133,6 +140,11 @@ void msg_exit_ns(struct ipc_namespace *n
 void __init msg_init(void)
 {
 	__msg_init_ns(&init_ipc_ns, &init_msg_ids);
+
+	if (register_tunable(&init_ipc_ns.msgmni_akt))
+		printk(KERN_WARNING
+			"AKT: Failed registering tunable msgmni\n");
+
 	ipc_init_proc_interface("sysvipc/msg",
 				"       key      msqid perms      cbytes       qnum lspid lrpid   uid   gid  cuid  cgid      stime      rtime      ctime\n",
 				IPC_MSG_IDS, sysvipc_msg_proc_show);
@@ -262,6 +274,8 @@ asmlinkage long sys_msgget(key_t key, in
 
 	ns = current->nsproxy->ipc_ns;
 	
+	activate_auto_tuning(AKT_UP, &ns->msgmni_akt);
+
 	mutex_lock(&msg_ids(ns).mutex);
 	if (key == IPC_PRIVATE) 
 		ret = newque(ns, key, msgflg);
@@ -391,6 +405,7 @@ asmlinkage long sys_msgctl(int msqid, in
 	struct msg_queue *msq;
 	int err, version;
 	struct ipc_namespace *ns;
+	int destroyed = 0;
 
 	if (msqid < 0 || cmd < 0)
 		return -EINVAL;
@@ -555,11 +570,16 @@ asmlinkage long sys_msgctl(int msqid, in
 	}
 	case IPC_RMID:
 		freeque(ns, msq, msqid);
+		destroyed = 1;
 		break;
 	}
 	err = 0;
 out_up:
 	mutex_unlock(&msg_ids(ns).mutex);
+
+	if (destroyed)
+		activate_auto_tuning(AKT_DOWN, &ns->msgmni_akt);
+
 	return err;
 out_unlock_up:
 	msg_unlock(msq);
ipc/shm.c +71 -5

140 lines not on this page · raw message

ipc/sem.c +43 -0

78 lines not on this page · raw message

--