fork: free vmapped stacks in cache when cpus are offline
Subsystems (3)
PATCH v3 1/1 fork: free vmapped stacks in cache when cpus are offline
Using virtually mapped stack, kernel stacks are allocated via vmalloc.
In the current implementation, two stacks per cpu can be cached when tasks are freed and the cached stacks are used again in task duplications. but the cached stacks may remain unfreed even when cpu are offline.
By adding a cpu hotplug callback to free the cached stacks when a cpu
goes offline, the pages of the cached stacks are not wasted.
---
Changes in v3:
fix misuse of per-cpu api
fix location of function definition within CONFIG_VMAP_STACK
Changes in v2:
remove cpuhp callback for startup, only teardown callback is installed.
kernel/fork.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
kernel/fork.c +23 -0
@@ -168,6 +168,24 @@ void __weak arch_release_thread_stack(unsigned long *stack)*/ #define NR_CACHED_STACKS 2 static DEFINE_PER_CPU(struct vm_struct *, cached_stacks[NR_CACHED_STACKS]); + +static int free_vm_stack_cache (unsigned int cpu) +{ +struct vm_struct **cached_vm_stacks =per_cpu_ptr (cached_stacks, cpu); +int i; + +for (i =0 ; i <NR_CACHED_STACKS ; i++) { +struct vm_struct **vm_stack = &cached_vm_stacks[i]; + +if (*vm_stack ==NULL ) +continue ; + +vfree ((*vm_stack)->addr); + *vm_stack =NULL ; + } + +return 0 ; +}#endif static unsigned long *alloc_thread_stack_node (struct task_struct *tsk,int node)
@@ -456,6 +474,11 @@ void __init fork_init(void)for (i =0 ; i <UCOUNT_COUNTS ; i++) { init_user_ns.ucount_max[i] = max_threads/2 ; } + +#ifdef CONFIG_VMAP_STACK +cpuhp_setup_state (CPUHP_AP_ONLINE_DYN ,"vm_stack_cache" , +NULL , free_vm_stack_cache); +#endif } int __weak arch_dup_task_struct(struct task_struct *dst,
Signature · 1 line
2.7.4
On Fri 10-02-17 17:32:07, Hoeun Ryu wrote:
[...]
kernel/fork.c void __weak arch_release_thread_stack(unsigned long *stack) · line 184
Quoted 13 lines
kernel/fork.c +13
-0+static int free_vm_stack_cache (unsigned int cpu) +{ +struct vm_struct **cached_vm_stacks =per_cpu_ptr (cached_stacks, cpu); +int i; + +for (i =0 ; i <NR_CACHED_STACKS ; i++) { +struct vm_struct **vm_stack = &cached_vm_stacks[i]; + +if (*vm_stack ==NULL ) +continue ; + +vfree ((*vm_stack)->addr); + *vm_stack =NULL ;
this seems more obscure than necessary. Probably a matter of taste but I would find the following easier to read
struct vm_struct *vm_stack = cached_vm_stacks[i]; if (!vm_stack) continue; vfree(vm_stack); cached_vm_stacks[i] = NULL;
Quoted 8 lines
kernel/fork.c +4
-0+ } + +return 0 ; +}#endif static unsigned long *alloc_thread_stack_node (struct task_struct *tsk,int node)
kernel/fork.c void __init fork_init(void) · line 481
Quoted 8 lines
kernel/fork.c +5
-0@@ -456,6 +474,11 @@ void __init fork_init(void)for (i =0 ; i <UCOUNT_COUNTS ; i++) { init_user_ns.ucount_max[i] = max_threads/2 ; } + +#ifdef CONFIG_VMAP_STACK +cpuhp_setup_state (CPUHP_AP_ONLINE_DYN ,"vm_stack_cache" , +NULL , free_vm_stack_cache); +#endif
I am not familiar the new hotplug infrastructure so I might be missing something. CPUHP_AP_ONLINE_DYN will allocate a state which is has only 30 slots available. The name also suggests this will be called on an online event. Why doesn't this have its own state like other users. The name should also reflect offline event CPUHP_STACK_CACHE_DEAD or something like that.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 18 lines · Michal Hocko
On Fri 10-02-17 17:32:07, Hoeun Ryu wrote:
[...]kernel/fork.c void __weak arch_release_thread_stack(unsigned long *stack) · line 184
Quoted 13 lines
kernel/fork.c +13
-0+static int free_vm_stack_cache (unsigned int cpu) +{ +struct vm_struct **cached_vm_stacks =per_cpu_ptr (cached_stacks, cpu); +int i; + +for (i =0 ; i <NR_CACHED_STACKS ; i++) { +struct vm_struct **vm_stack = &cached_vm_stacks[i]; + +if (*vm_stack ==NULL ) +continue ; + +vfree ((*vm_stack)->addr); + *vm_stack =NULL ;this seems more obscure than necessary. Probably a matter of taste but I would find the following easier to read
Quoted 8 lines
struct vm_struct *vm_stack = cached_vm_stacks[i];
if (!vm_stack)
continue;
vfree(vm_stack);
cached_vm_stacks[i] = NULL;OK, it's easier to read, I'll fix it.
Quoted 15 lines
Quoted 15 lines
kernel/fork.c +8
-0+ } + +return 0 ; +}#endif static unsigned long *alloc_thread_stack_node (struct task_struct *tsk,int node)@@ -456,6 +474,11 @@ void __init fork_init(void)for (i =0 ; i <UCOUNT_COUNTS ; i++) { init_user_ns.ucount_max[i] = max_threads/2 ; } + +#ifdef CONFIG_VMAP_STACK +cpuhp_setup_state (CPUHP_AP_ONLINE_DYN ,"vm_stack_cache" , +NULL , free_vm_stack_cache);
Quoted 8 lines
kernel/fork.c void __init fork_init(void) · line 481
Quoted 1 line
kernel/fork.c +1
-0+#endif I am not familiar the new hotplug infrastructure so I might be missing something.
CPUHP_AP_ONLINE_DYNwill allocate a state which is has only 30 slots available. The name also suggests this will be called on an online event. Why doesn't this have its own state like other users. The name should also reflect offline eventCPUHP_STACK_CACHE_DEADor something like that.
I'll define CPUHP_VMSTACK_CACHE_DEAD before CPUHP_BP_PREPARE_DYN in cpuhotplug.h.
Do you think the change is made in a separate patch or not ?
Thank you for your review anyway.
Quoted 4 lines
--
Michal Hocko
SUSE Labs
Quoted 2 lines · Hoeun Ryu
Quoted 1 line · Michal Hocko
On Fri 10-02-17 17:32:07, Hoeun Ryu wrote:
[...]
Quoted 18 lines
Quoted 9 lines
Quoted 9 lines
static unsigned long *alloc_thread_stack_node(struct
task_struct*tsk, int node)kernel/fork.c +4
-0@@ -456,6 +474,11 @@ void __init fork_init(void)for (i =0 ; i <UCOUNT_COUNTS ; i++) { init_user_ns.ucount_max[i] = max_threads/2 ; } + +#ifdef CONFIG_VMAP_STACK +cpuhp_setup_state (CPUHP_AP_ONLINE_DYN ,"vm_stack_cache" , +NULL , free_vm_stack_cache);Quoted 8 lines
kernel/fork.c void __init fork_init(void) · line 481
Quoted 1 line
kernel/fork.c +1
-0+#endif I am not familiar the new hotplug infrastructure so I might be missing something.
CPUHP_AP_ONLINE_DYNwill allocate a state which is has only 30 slots available. The name also suggests this will be called on an online event. Why doesn't this have its own state like other users. The name should also reflect offline eventCPUHP_STACK_CACHE_DEADor something like that.
Quoted 3 lines
I'll define
CPUHP_VMSTACK_CACHE_DEADbeforeCPUHP_BP_PREPARE_DYNincpuhotplug.h.
Do you think the change is made in a separate patch or not ?
I think it should be in a single patch. I am not sure what are the rules to define a new state though. Let's CC Thomas.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 22 lines · Michal Hocko
Quoted 2 lines · Hoeun Ryu
Quoted 1 line · Michal Hocko
On Fri 10-02-17 17:32:07, Hoeun Ryu wrote:
[...]
Quoted 18 lines
Quoted 17 lines
kernel/fork.c void __init fork_init(void) · line 481
Quoted 10 lines
static unsigned long *alloc_thread_stack_node(struct
task_struct*tsk, int node)kernel/fork.c +5
-0@@ -456,6 +474,11 @@ void __init fork_init(void)for (i =0 ; i <UCOUNT_COUNTS ; i++) { init_user_ns.ucount_max[i] = max_threads/2 ; } + +#ifdef CONFIG_VMAP_STACK +cpuhp_setup_state (CPUHP_AP_ONLINE_DYN ,"vm_stack_cache" , +NULL , free_vm_stack_cache); +#endif I am not familiar the new hotplug infrastructure so I might be missing something.
CPUHP_AP_ONLINE_DYNwill allocate a state which is has only 30 slots available. The name also suggests this will be called on an online event. Why doesn't this have its own state like other users. The name should also reflect offline eventCPUHP_STACK_CACHE_DEADor something like that.
Quoted 6 lines
Quoted 3 lines
I'll define
CPUHP_VMSTACK_CACHE_DEADbeforeCPUHP_BP_PREPARE_DYNincpuhotplug.h.
Do you think the change is made in a separate patch or not ?I think it should be in a single patch. I am not sure what are the rules to define a new state though. Let's CC Thomas.
OK, I will.
Quoted 3 lines
--
Michal Hocko
SUSE Labs
Quoted 5 lines · Hoeun Ryu
Quoted 4 lines · Michal Hocko
Quoted 2 lines · Hoeun Ryu
Quoted 1 line · Michal Hocko
On Fri 10-02-17 17:32:07, Hoeun Ryu wrote:
[...]
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
static unsigned long *alloc_thread_stack_node(struct
task_struct*tsk, int node)kernel/fork.c +1
-0@@ -456,6 +474,11 @@ void __init fork_init(void)for (i =0 ; i <UCOUNT_COUNTS ; i++) { init_user_ns.ucount_max[i] = max_threads/2 ; } +Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
kernel/fork.c void __init fork_init(void) · line 481
Quoted 4 lines
kernel/fork.c +4
-0+#ifdef CONFIG_VMAP_STACK +cpuhp_setup_state (CPUHP_AP_ONLINE_DYN ,"vm_stack_cache" , +NULL , free_vm_stack_cache); +#endif I am not familiar the new hotplug infrastructure so I might be missing something.
CPUHP_AP_ONLINE_DYNwill allocate a state which is has only 30 slots available.
That's enough and when we hit that limit we just up it.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
The name also suggests this will be called on an online event.
The states are symmetric.
Quoted 22 lines · Michal Hocko
Quoted 2 lines · Hoeun Ryu
Quoted 1 line · Michal Hocko
On Fri 10-02-17 17:32:07, Hoeun Ryu wrote:
[...]
Quoted 18 lines
Quoted 17 lines
kernel/fork.c void __init fork_init(void) · line 481
Quoted 10 lines
static unsigned long *alloc_thread_stack_node(struct
task_struct*tsk, int node)kernel/fork.c +5
-0@@ -456,6 +474,11 @@ void __init fork_init(void)for (i =0 ; i <UCOUNT_COUNTS ; i++) { init_user_ns.ucount_max[i] = max_threads/2 ; } + +#ifdef CONFIG_VMAP_STACK +cpuhp_setup_state (CPUHP_AP_ONLINE_DYN ,"vm_stack_cache" , +NULL , free_vm_stack_cache); +#endif I am not familiar the new hotplug infrastructure so I might be missing something.
CPUHP_AP_ONLINE_DYNwill allocate a state which is has only 30 slots available. The name also suggests this will be called on an online event. Why doesn't this have its own state like other users. The name should also reflect offline eventCPUHP_STACK_CACHE_DEADor something like that.
Quoted 6 lines
Quoted 3 lines
I'll define
CPUHP_VMSTACK_CACHE_DEADbeforeCPUHP_BP_PREPARE_DYNincpuhotplug.h.
Do you think the change is made in a separate patch or not ?I think it should be in a single patch. I am not sure what are the rules to define a new state though. Let's CC Thomas.
So the first question is where do you want that to be called? i.e. in which section:
CPU up CPU down
PREPARE DEAD <- called on some other CPU
ONLINE DOWN <- called on the hotplugged CPU
And then the next question is whether you have ordering constraints, i.e. it must be called before or after some other callback. Only in that case you want to have an explicit state. If not, just use a dynamically allocated one.
Thanks,
tglx
Quoted 30 lines · Thomas Gleixner
Quoted 22 lines · Michal Hocko
Quoted 2 lines · Hoeun Ryu
Quoted 1 line · Michal Hocko
On Fri 10-02-17 17:32:07, Hoeun Ryu wrote:
[...]
Quoted 18 lines
Quoted 17 lines
kernel/fork.c void __init fork_init(void) · line 481
Quoted 10 lines
static unsigned long *alloc_thread_stack_node(struct
task_struct*tsk, int node)kernel/fork.c +5
-0@@ -456,6 +474,11 @@ void __init fork_init(void)for (i =0 ; i <UCOUNT_COUNTS ; i++) { init_user_ns.ucount_max[i] = max_threads/2 ; } + +#ifdef CONFIG_VMAP_STACK +cpuhp_setup_state (CPUHP_AP_ONLINE_DYN ,"vm_stack_cache" , +NULL , free_vm_stack_cache); +#endif I am not familiar the new hotplug infrastructure so I might be missing something.
CPUHP_AP_ONLINE_DYNwill allocate a state which is has only 30 slots available. The name also suggests this will be called on an online event. Why doesn't this have its own state like other users. The name should also reflect offline eventCPUHP_STACK_CACHE_DEADor something like that.Quoted 6 lines
Quoted 3 lines
I'll define
CPUHP_VMSTACK_CACHE_DEADbeforeCPUHP_BP_PREPARE_DYNincpuhotplug.h.
Do you think the change is made in a separate patch or not ?I think it should be in a single patch. I am not sure what are the rules to define a new state though. Let's CC Thomas.
Quoted 8 lines
So the first question is where do you want that to be called? i.e. in which section:
CPU up CPU down
PREPARE DEAD <- called on some other CPU ONLINE DOWN <- called on the hotplugged CPU
It doesn't matter whether the callback is called on the hotplugged CPU or other CPUs.
Quoted 4 lines
And then the next question is whether you have ordering constraints, i.e. it must be called before or after some other callback. Only in that case you want to have an explicit state. If not, just use a dynamically allocated one.
The cache is for virtually mapped kernel stacks. so I think the callback should be called after all tasks are migrated to other CPUs. It must reside before CPUHP_AP_SCHED_STARTING or CPUHP_BRINGUP_CPU.
Quoted 4 lines
Thanks,
tglx
Quoted 43 lines · Hoeun Ryu
Quoted 30 lines · Thomas Gleixner
Quoted 22 lines · Michal Hocko
Quoted 2 lines · Hoeun Ryu
Quoted 1 line · Michal Hocko
On Fri 10-02-17 17:32:07, Hoeun Ryu wrote:
[...]
Quoted 18 lines
Quoted 17 lines
kernel/fork.c void __init fork_init(void) · line 481
Quoted 10 lines
static unsigned long *alloc_thread_stack_node(struct
task_struct*tsk, int node)kernel/fork.c +5
-0@@ -456,6 +474,11 @@ void __init fork_init(void)for (i =0 ; i <UCOUNT_COUNTS ; i++) { init_user_ns.ucount_max[i] = max_threads/2 ; } + +#ifdef CONFIG_VMAP_STACK +cpuhp_setup_state (CPUHP_AP_ONLINE_DYN ,"vm_stack_cache" , +NULL , free_vm_stack_cache); +#endif I am not familiar the new hotplug infrastructure so I might be missing something.
CPUHP_AP_ONLINE_DYNwill allocate a state which is has only 30 slots available. The name also suggests this will be called on an online event. Why doesn't this have its own state like other users. The name should also reflect offline eventCPUHP_STACK_CACHE_DEADor something like that.Quoted 6 lines
Quoted 3 lines
I'll define
CPUHP_VMSTACK_CACHE_DEADbeforeCPUHP_BP_PREPARE_DYNincpuhotplug.h.
Do you think the change is made in a separate patch or not ?I think it should be in a single patch. I am not sure what are the rules to define a new state though. Let's CC Thomas.
Quoted 8 lines
So the first question is where do you want that to be called? i.e. in which section:
CPU up CPU down
PREPARE DEAD <- called on some other CPU ONLINE DOWN <- called on the hotplugged CPUIt doesn't matter whether the callback is called on the hotplugged CPU or other CPUs.
Quoted 8 lines
Quoted 4 lines
And then the next question is whether you have ordering constraints, i.e. it must be called before or after some other callback. Only in that case you want to have an explicit state. If not, just use a dynamically allocated one.
The cache is for virtually mapped kernel stacks. so I think the callback should be called after all tasks are migrated to other CPUs. It must reside before
CPUHP_AP_SCHED_STARTINGorCPUHP_BRINGUP_CPU.
Between AP_OFFLINE and AP_SCHED_STARTING does not work because those states path are called on the hotplugged CPU with interrupts disabled and after the CPU has been taken out from the scheduler.
So the proper place is the dynamic states CPUHP_BP_PREPARE_DYN. You do not have a prepare callback, but then it's called on an still online CPU _AFTER_ the hotplugged CPU vanished from the system.
Thanks,
tglx
Quoted 53 lines · Thomas Gleixner
Quoted 43 lines · Hoeun Ryu
Quoted 30 lines · Thomas Gleixner
Quoted 22 lines · Michal Hocko
Quoted 2 lines · Hoeun Ryu
Quoted 1 line · Michal Hocko
On Fri 10-02-17 17:32:07, Hoeun Ryu wrote:
[...]
Quoted 18 lines
Quoted 17 lines
Review void __init fork_init(void) · line 481
Quoted 10 lines
static unsigned long *alloc_thread_stack_node(struct
task_struct*tsk, int node)(quoted diff) +5
-0@@ -456,6 +474,11 @@ void __init fork_init(void) for (i = 0; i < UCOUNT_COUNTS; i++) { init_user_ns.ucount_max[i] = max_threads/2; } + +#ifdef CONFIG_VMAP_STACK + cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "vm_stack_cache", + NULL, free_vm_stack_cache); +#endifI am not familiar the new hotplug infrastructure so I might be missing something.
CPUHP_AP_ONLINE_DYNwill allocate a state which is has only 30 slots available. The name also suggests this will be called on an online event. Why doesn't this have its own state like other users. The name should also reflect offline eventCPUHP_STACK_CACHE_DEADor something like that.Quoted 6 lines
Quoted 3 lines
I'll define
CPUHP_VMSTACK_CACHE_DEADbeforeCPUHP_BP_PREPARE_DYNincpuhotplug.h.
Do you think the change is made in a separate patch or not ?I think it should be in a single patch. I am not sure what are the rules to define a new state though. Let's CC Thomas.
Quoted 8 lines
So the first question is where do you want that to be called? i.e. in which section:
CPU up CPU down
PREPARE DEAD <- called on some other CPU ONLINE DOWN <- called on the hotplugged CPUIt doesn't matter whether the callback is called on the hotplugged CPU or other CPUs.
Quoted 8 lines
Quoted 4 lines
And then the next question is whether you have ordering constraints, i.e. it must be called before or after some other callback. Only in that case you want to have an explicit state. If not, just use a dynamically allocated one.
The cache is for virtually mapped kernel stacks. so I think the callback should be called after all tasks are migrated to other CPUs. It must reside before
CPUHP_AP_SCHED_STARTINGorCPUHP_BRINGUP_CPU.
Quoted 8 lines
Between
AP_OFFLINEandAP_SCHED_STARTINGdoes not work because those states path are called on the hotplugged CPU with interrupts disabled and after the CPU has been taken out from the scheduler.So the proper place is the dynamic states
CPUHP_BP_PREPARE_DYN. You do not have a prepare callback, but then it's called on an still online CPU _AFTER_ the hotplugged CPU vanished from the system.
OK, I agree.
Quoted 3 lines
Thanks,
tglx